2024-09-20T22:26:18.874 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-20T22:26:18.896 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-20T22:26:19.012 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-09-20_21:08:02-orch-squid-distro-default-smithi/7913018 branch: squid description: orch/cephadm/thrash/{0-distro/centos_9.stream 1-start 2-thrash 3-tasks/snaps-few-objects fixed-2 msgr/async-v2only root} email: ceph-qa@ceph.com first_in_suite: false job_id: '7913018' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: teuthology-2024-09-20_21:08:02-orch-squid-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: squid ceph: conf: global: ms bind msgr1: false ms bind msgr2: true ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 paxos service trim min: 10 osd: debug ms: 1 debug osd: 20 osd debug reject backfill probability: 0.3 osd delete sleep: 1 osd max backfills: 3 osd scrub max interval: 120 osd scrub min interval: 60 osd shutdown pgref assert: true osd snap trim sleep: 2 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - CEPHADM_STRAY_DAEMON - CEPHADM_FAILED_DAEMON - but it is still running - objects unfound and apparently lost - \(POOL_APP_NOT_ENABLED\) log-only-match: - CEPHADM_ sha1: 9e2ac70bb7aea67d34f0d50002344abbe7803736 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} cephadm: cephadm_mode: root install: ceph: flavor: default sha1: 9e2ac70bb7aea67d34f0d50002344abbe7803736 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 workunit: branch: squid sha1: 4e45e6b490e6cbe082cdf155fea780438ec53fc7 owner: scheduled_teuthology@teuthology priority: 100 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - ceph.rgw.foo.a - node-exporter.a - alertmanager.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - grafana.a - node-exporter.b - ceph.iscsi.iscsi.a seed: 9344 sha1: 9e2ac70bb7aea67d34f0d50002344abbe7803736 sleep_before_teardown: 0 subset: 56/64 suite: orch suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_4e45e6b490e6cbe082cdf155fea780438ec53fc7/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: 4e45e6b490e6cbe082cdf155fea780438ec53fc7 targets: smithi084.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNbkBIxhKGH4qWslLm9ZE0lSYZQ0feo5ljTSD49ktN3mB3PgaeWZ7yiQeTmdLnFRDzOQKPiwYZsrIR3PpdtA5t8= smithi154.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOzuhbgWtP9EMLE+3kGuWOHHgrgH64AePzzPAVy+eoOziYPEvdHo+ADzcrafKev+/2WcJZ8c882I9TDqhXNtUvo= tasks: - pexec: all: - sudo dnf remove nvme-cli -y - sudo dnf install nvmetcli nvme-cli -y - install: null - cephadm: conf: mgr: debug mgr: 20 debug ms: 1 - thrashosds: chance_pgnum_grow: 1 chance_pgnum_shrink: 1 chance_pgpnum_fix: 1 timeout: 1200 - rados: clients: - client.0 objects: 50 op_weights: copy_from: 50 delete: 50 read: 100 rollback: 50 snap_create: 50 snap_remove: 50 write: 100 ops: 4000 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-20_21:08:02 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2902 2024-09-20T22:26:19.012 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_4e45e6b490e6cbe082cdf155fea780438ec53fc7/qa; will attempt to use it 2024-09-20T22:26:19.013 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_4e45e6b490e6cbe082cdf155fea780438ec53fc7/qa/tasks 2024-09-20T22:26:19.013 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-20T22:26:19.015 INFO:teuthology.task.internal:Checking packages... 2024-09-20T22:26:19.036 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '9e2ac70bb7aea67d34f0d50002344abbe7803736' 2024-09-20T22:26:19.036 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-20T22:26:19.037 INFO:teuthology.packaging:ref: None 2024-09-20T22:26:19.037 INFO:teuthology.packaging:tag: None 2024-09-20T22:26:19.037 INFO:teuthology.packaging:branch: squid 2024-09-20T22:26:19.037 INFO:teuthology.packaging:sha1: 9e2ac70bb7aea67d34f0d50002344abbe7803736 2024-09-20T22:26:19.037 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-09-20T22:26:19.256 INFO:teuthology.task.internal:Found packages for ceph version 19.1.1-373.g4e45e6b4 2024-09-20T22:26:19.258 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-20T22:26:19.264 INFO:teuthology.task.internal:no buildpackages task found 2024-09-20T22:26:19.264 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-20T22:26:19.270 INFO:teuthology.task.internal:Saving configuration 2024-09-20T22:26:19.283 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-20T22:26:19.290 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-20T22:26:19.315 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi084.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-20_21:08:02-orch-squid-distro-default-smithi/7913018', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-20 22:21:21.678129', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNbkBIxhKGH4qWslLm9ZE0lSYZQ0feo5ljTSD49ktN3mB3PgaeWZ7yiQeTmdLnFRDzOQKPiwYZsrIR3PpdtA5t8='} 2024-09-20T22:26:19.339 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi154.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-20_21:08:02-orch-squid-distro-default-smithi/7913018', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-20 22:21:21.676190', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOzuhbgWtP9EMLE+3kGuWOHHgrgH64AePzzPAVy+eoOziYPEvdHo+ADzcrafKev+/2WcJZ8c882I9TDqhXNtUvo='} 2024-09-20T22:26:19.339 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-20T22:26:19.345 INFO:teuthology.task.internal:roles: ubuntu@smithi084.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'ceph.rgw.foo.a', 'node-exporter.a', 'alertmanager.a'] 2024-09-20T22:26:19.345 INFO:teuthology.task.internal:roles: ubuntu@smithi154.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b', 'ceph.iscsi.iscsi.a'] 2024-09-20T22:26:19.345 INFO:teuthology.run_tasks:Running task console_log... 2024-09-20T22:26:19.407 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f25ab9e0160>, signals=[15]) 2024-09-20T22:26:19.407 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-20T22:26:19.412 INFO:teuthology.task.internal:Opening connections... 2024-09-20T22:26:19.412 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi084.front.sepia.ceph.com 2024-09-20T22:26:19.413 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi084.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-20T22:26:19.489 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi154.front.sepia.ceph.com 2024-09-20T22:26:19.490 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi154.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-20T22:26:19.572 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-20T22:26:19.578 DEBUG:teuthology.orchestra.run.smithi084:> uname -m 2024-09-20T22:26:19.595 INFO:teuthology.orchestra.run.smithi084.stdout:x86_64 2024-09-20T22:26:19.596 DEBUG:teuthology.orchestra.run.smithi084:> cat /etc/os-release 2024-09-20T22:26:19.653 INFO:teuthology.orchestra.run.smithi084.stdout:NAME="CentOS Stream" 2024-09-20T22:26:19.653 INFO:teuthology.orchestra.run.smithi084.stdout:VERSION="9" 2024-09-20T22:26:19.654 INFO:teuthology.orchestra.run.smithi084.stdout:ID="centos" 2024-09-20T22:26:19.654 INFO:teuthology.orchestra.run.smithi084.stdout:ID_LIKE="rhel fedora" 2024-09-20T22:26:19.654 INFO:teuthology.orchestra.run.smithi084.stdout:VERSION_ID="9" 2024-09-20T22:26:19.654 INFO:teuthology.orchestra.run.smithi084.stdout:PLATFORM_ID="platform:el9" 2024-09-20T22:26:19.654 INFO:teuthology.orchestra.run.smithi084.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-20T22:26:19.654 INFO:teuthology.orchestra.run.smithi084.stdout:ANSI_COLOR="0;31" 2024-09-20T22:26:19.654 INFO:teuthology.orchestra.run.smithi084.stdout:LOGO="fedora-logo-icon" 2024-09-20T22:26:19.654 INFO:teuthology.orchestra.run.smithi084.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-20T22:26:19.655 INFO:teuthology.orchestra.run.smithi084.stdout:HOME_URL="https://centos.org/" 2024-09-20T22:26:19.655 INFO:teuthology.orchestra.run.smithi084.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-20T22:26:19.655 INFO:teuthology.orchestra.run.smithi084.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-20T22:26:19.655 INFO:teuthology.orchestra.run.smithi084.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-20T22:26:19.656 INFO:teuthology.lock.ops:Updating smithi084.front.sepia.ceph.com on lock server 2024-09-20T22:26:19.684 DEBUG:teuthology.orchestra.run.smithi154:> uname -m 2024-09-20T22:26:19.702 INFO:teuthology.orchestra.run.smithi154.stdout:x86_64 2024-09-20T22:26:19.702 DEBUG:teuthology.orchestra.run.smithi154:> cat /etc/os-release 2024-09-20T22:26:19.759 INFO:teuthology.orchestra.run.smithi154.stdout:NAME="CentOS Stream" 2024-09-20T22:26:19.759 INFO:teuthology.orchestra.run.smithi154.stdout:VERSION="9" 2024-09-20T22:26:19.759 INFO:teuthology.orchestra.run.smithi154.stdout:ID="centos" 2024-09-20T22:26:19.759 INFO:teuthology.orchestra.run.smithi154.stdout:ID_LIKE="rhel fedora" 2024-09-20T22:26:19.759 INFO:teuthology.orchestra.run.smithi154.stdout:VERSION_ID="9" 2024-09-20T22:26:19.759 INFO:teuthology.orchestra.run.smithi154.stdout:PLATFORM_ID="platform:el9" 2024-09-20T22:26:19.759 INFO:teuthology.orchestra.run.smithi154.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-20T22:26:19.760 INFO:teuthology.orchestra.run.smithi154.stdout:ANSI_COLOR="0;31" 2024-09-20T22:26:19.760 INFO:teuthology.orchestra.run.smithi154.stdout:LOGO="fedora-logo-icon" 2024-09-20T22:26:19.760 INFO:teuthology.orchestra.run.smithi154.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-20T22:26:19.760 INFO:teuthology.orchestra.run.smithi154.stdout:HOME_URL="https://centos.org/" 2024-09-20T22:26:19.760 INFO:teuthology.orchestra.run.smithi154.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-20T22:26:19.760 INFO:teuthology.orchestra.run.smithi154.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-20T22:26:19.760 INFO:teuthology.orchestra.run.smithi154.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-20T22:26:19.760 INFO:teuthology.lock.ops:Updating smithi154.front.sepia.ceph.com on lock server 2024-09-20T22:26:19.793 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-20T22:26:19.801 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-20T22:26:19.805 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-20T22:26:19.805 DEBUG:teuthology.orchestra.run.smithi084:> test '!' -e /home/ubuntu/cephtest 2024-09-20T22:26:19.808 DEBUG:teuthology.orchestra.run.smithi154:> test '!' -e /home/ubuntu/cephtest 2024-09-20T22:26:19.825 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-20T22:26:19.842 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-20T22:26:19.843 DEBUG:teuthology.orchestra.run.smithi084:> test -z $(ls -A /var/lib/ceph) 2024-09-20T22:26:19.865 DEBUG:teuthology.orchestra.run.smithi154:> test -z $(ls -A /var/lib/ceph) 2024-09-20T22:26:19.985 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-20T22:26:20.044 INFO:teuthology.run_tasks:Running task kernel... 2024-09-20T22:26:20.059 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-20T22:26:20.059 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-20T22:26:20.059 DEBUG:teuthology.orchestra.run.smithi084:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-20T22:26:20.060 DEBUG:teuthology.orchestra.run.smithi154:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-20T22:26:20.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:26:20.077 DEBUG:teuthology.orchestra.run.smithi084:> uname -r 2024-09-20T22:26:20.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:26:20.078 DEBUG:teuthology.orchestra.run.smithi154:> uname -r 2024-09-20T22:26:20.135 INFO:teuthology.orchestra.run.smithi084.stdout:5.14.0-503.el9.x86_64 2024-09-20T22:26:20.135 INFO:teuthology.task.kernel:Running kernel on smithi084: 5.14.0-503.el9.x86_64 2024-09-20T22:26:20.135 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum install -y kernel 2024-09-20T22:26:20.136 INFO:teuthology.orchestra.run.smithi154.stdout:5.14.0-503.el9.x86_64 2024-09-20T22:26:20.137 INFO:teuthology.task.kernel:Running kernel on smithi154: 5.14.0-503.el9.x86_64 2024-09-20T22:26:20.137 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum install -y kernel 2024-09-20T22:26:22.555 INFO:teuthology.orchestra.run.smithi084.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.3 MB 00:00 2024-09-20T22:26:22.672 INFO:teuthology.orchestra.run.smithi154.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.3 MB 00:00 2024-09-20T22:26:24.456 INFO:teuthology.orchestra.run.smithi084.stdout:CentOS Stream 9 - AppStream 25 MB/s | 20 MB 00:00 2024-09-20T22:26:28.865 INFO:teuthology.orchestra.run.smithi084.stdout:CentOS Stream 9 - CRB 11 MB/s | 6.5 MB 00:00 2024-09-20T22:26:30.967 INFO:teuthology.orchestra.run.smithi084.stdout:CentOS Stream 9 - Extras packages 21 kB/s | 19 kB 00:00 2024-09-20T22:26:31.780 INFO:teuthology.orchestra.run.smithi154.stdout:CentOS Stream 9 - AppStream 2.5 MB/s | 20 MB 00:08 2024-09-20T22:26:31.870 INFO:teuthology.orchestra.run.smithi084.stdout:Extra Packages for Enterprise Linux 29 MB/s | 23 MB 00:00 2024-09-20T22:26:36.373 INFO:teuthology.orchestra.run.smithi154.stdout:CentOS Stream 9 - CRB 7.9 MB/s | 6.5 MB 00:00 2024-09-20T22:26:37.225 INFO:teuthology.orchestra.run.smithi084.stdout:lab-extras 37 kB/s | 1.7 kB 00:00 2024-09-20T22:26:38.681 INFO:teuthology.orchestra.run.smithi154.stdout:CentOS Stream 9 - Extras packages 18 kB/s | 19 kB 00:01 2024-09-20T22:26:39.131 INFO:teuthology.orchestra.run.smithi084.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-20T22:26:39.131 INFO:teuthology.orchestra.run.smithi084.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-20T22:26:39.221 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-09-20T22:26:39.228 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-09-20T22:26:39.229 INFO:teuthology.orchestra.run.smithi084.stdout: Package Architecture Version Repository Size 2024-09-20T22:26:39.229 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-09-20T22:26:39.229 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2024-09-20T22:26:39.229 INFO:teuthology.orchestra.run.smithi084.stdout: kernel x86_64 5.14.0-508.el9 baseos 36 k 2024-09-20T22:26:39.229 INFO:teuthology.orchestra.run.smithi084.stdout:Installing dependencies: 2024-09-20T22:26:39.229 INFO:teuthology.orchestra.run.smithi084.stdout: kernel-core x86_64 5.14.0-508.el9 baseos 16 M 2024-09-20T22:26:39.229 INFO:teuthology.orchestra.run.smithi084.stdout: kernel-modules x86_64 5.14.0-508.el9 baseos 35 M 2024-09-20T22:26:39.229 INFO:teuthology.orchestra.run.smithi084.stdout: kernel-modules-core x86_64 5.14.0-508.el9 baseos 29 M 2024-09-20T22:26:39.229 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-20T22:26:39.229 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2024-09-20T22:26:39.229 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-09-20T22:26:39.229 INFO:teuthology.orchestra.run.smithi084.stdout:Install 4 Packages 2024-09-20T22:26:39.230 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-20T22:26:39.230 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 79 M 2024-09-20T22:26:39.230 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 126 M 2024-09-20T22:26:39.230 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2024-09-20T22:26:39.810 INFO:teuthology.orchestra.run.smithi154.stdout:Extra Packages for Enterprise Linux 23 MB/s | 23 MB 00:01 2024-09-20T22:26:39.902 INFO:teuthology.orchestra.run.smithi084.stdout:(1/4): kernel-5.14.0-508.el9.x86_64.rpm 63 kB/s | 36 kB 00:00 2024-09-20T22:26:40.718 INFO:teuthology.orchestra.run.smithi084.stdout:(2/4): kernel-core-5.14.0-508.el9.x86_64.rpm 11 MB/s | 16 MB 00:01 2024-09-20T22:26:41.477 INFO:teuthology.orchestra.run.smithi084.stdout:(3/4): kernel-modules-5.14.0-508.el9.x86_64.rpm 16 MB/s | 35 MB 00:02 2024-09-20T22:26:42.527 INFO:teuthology.orchestra.run.smithi084.stdout:(4/4): kernel-modules-core-5.14.0-508.el9.x86_6 11 MB/s | 29 MB 00:02 2024-09-20T22:26:42.527 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2024-09-20T22:26:42.528 INFO:teuthology.orchestra.run.smithi084.stdout:Total 24 MB/s | 79 MB 00:03 2024-09-20T22:26:43.205 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2024-09-20T22:26:43.551 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2024-09-20T22:26:43.551 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2024-09-20T22:26:44.523 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2024-09-20T22:26:44.523 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2024-09-20T22:26:45.275 INFO:teuthology.orchestra.run.smithi154.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-09-20T22:26:46.951 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2024-09-20T22:26:47.206 INFO:teuthology.orchestra.run.smithi154.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-20T22:26:47.206 INFO:teuthology.orchestra.run.smithi154.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-20T22:26:47.300 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2024-09-20T22:26:47.312 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-09-20T22:26:47.313 INFO:teuthology.orchestra.run.smithi154.stdout: Package Architecture Version Repository Size 2024-09-20T22:26:47.313 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-09-20T22:26:47.313 INFO:teuthology.orchestra.run.smithi154.stdout:Installing: 2024-09-20T22:26:47.313 INFO:teuthology.orchestra.run.smithi154.stdout: kernel x86_64 5.14.0-508.el9 baseos 36 k 2024-09-20T22:26:47.313 INFO:teuthology.orchestra.run.smithi154.stdout:Installing dependencies: 2024-09-20T22:26:47.313 INFO:teuthology.orchestra.run.smithi154.stdout: kernel-core x86_64 5.14.0-508.el9 baseos 16 M 2024-09-20T22:26:47.313 INFO:teuthology.orchestra.run.smithi154.stdout: kernel-modules x86_64 5.14.0-508.el9 baseos 35 M 2024-09-20T22:26:47.313 INFO:teuthology.orchestra.run.smithi154.stdout: kernel-modules-core x86_64 5.14.0-508.el9 baseos 29 M 2024-09-20T22:26:47.313 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-20T22:26:47.313 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2024-09-20T22:26:47.313 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-09-20T22:26:47.314 INFO:teuthology.orchestra.run.smithi154.stdout:Install 4 Packages 2024-09-20T22:26:47.314 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-20T22:26:47.314 INFO:teuthology.orchestra.run.smithi154.stdout:Total download size: 79 M 2024-09-20T22:26:47.314 INFO:teuthology.orchestra.run.smithi154.stdout:Installed size: 126 M 2024-09-20T22:26:47.314 INFO:teuthology.orchestra.run.smithi154.stdout:Downloading Packages: 2024-09-20T22:26:47.588 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : kernel-modules-core-5.14.0-508.el9.x86_64 1/4 2024-09-20T22:26:47.676 INFO:teuthology.orchestra.run.smithi154.stdout:(1/4): kernel-5.14.0-508.el9.x86_64.rpm 434 kB/s | 36 kB 00:00 2024-09-20T22:26:47.679 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-20T22:26:49.168 INFO:teuthology.orchestra.run.smithi154.stdout:(2/4): kernel-core-5.14.0-508.el9.x86_64.rpm 9.9 MB/s | 16 MB 00:01 2024-09-20T22:26:49.801 INFO:teuthology.orchestra.run.smithi154.stdout:(3/4): kernel-modules-5.14.0-508.el9.x86_64.rpm 16 MB/s | 35 MB 00:02 2024-09-20T22:26:50.218 INFO:teuthology.orchestra.run.smithi154.stdout:(4/4): kernel-modules-core-5.14.0-508.el9.x86_6 11 MB/s | 29 MB 00:02 2024-09-20T22:26:50.218 INFO:teuthology.orchestra.run.smithi154.stdout:-------------------------------------------------------------------------------- 2024-09-20T22:26:50.219 INFO:teuthology.orchestra.run.smithi154.stdout:Total 27 MB/s | 79 MB 00:02 2024-09-20T22:26:50.413 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-20T22:26:50.539 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-20T22:26:50.900 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2024-09-20T22:26:51.252 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2024-09-20T22:26:51.301 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2024-09-20T22:26:52.222 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2024-09-20T22:26:52.222 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2024-09-20T22:26:54.650 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2024-09-20T22:26:55.332 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : kernel-modules-core-5.14.0-508.el9.x86_64 1/4 2024-09-20T22:26:55.359 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-20T22:26:55.436 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-20T22:26:55.480 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : kernel-5.14.0-508.el9.x86_64 4/4 2024-09-20T22:26:58.203 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-20T22:26:58.333 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-20T22:27:00.321 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-20T22:27:03.147 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-20T22:27:03.266 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : kernel-5.14.0-508.el9.x86_64 4/4 2024-09-20T22:27:08.097 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-20T22:27:38.453 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 4/4 2024-09-20T22:27:38.493 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 4/4 2024-09-20T22:27:38.832 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: kernel-5.14.0-508.el9.x86_64 4/4 2024-09-20T22:27:38.832 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : kernel-5.14.0-508.el9.x86_64 1/4 2024-09-20T22:27:38.832 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-20T22:27:38.832 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-20T22:27:39.682 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-20T22:27:39.682 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-20T22:27:39.682 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2024-09-20T22:27:39.682 INFO:teuthology.orchestra.run.smithi084.stdout: kernel-5.14.0-508.el9.x86_64 2024-09-20T22:27:39.682 INFO:teuthology.orchestra.run.smithi084.stdout: kernel-core-5.14.0-508.el9.x86_64 2024-09-20T22:27:39.682 INFO:teuthology.orchestra.run.smithi084.stdout: kernel-modules-5.14.0-508.el9.x86_64 2024-09-20T22:27:39.682 INFO:teuthology.orchestra.run.smithi084.stdout: kernel-modules-core-5.14.0-508.el9.x86_64 2024-09-20T22:27:39.683 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-20T22:27:39.683 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-09-20T22:27:40.044 DEBUG:teuthology.orchestra.run.smithi084:> rpm -q kernel | sort -rV | head -n 1 2024-09-20T22:27:40.140 INFO:teuthology.orchestra.run.smithi084.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-20T22:27:40.141 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-20T22:27:40.141 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-508.el9.x86_64 2024-09-20T22:27:40.141 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-20T22:27:40.141 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-20T22:27:40.141 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-20T22:27:40.141 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi084.front.sepia.ceph.com, path=None, version=distro) 2024-09-20T22:27:40.141 DEBUG:teuthology.orchestra.run.smithi084:> rpm -q kernel | sort -rV | head -n 1 2024-09-20T22:27:40.200 INFO:teuthology.orchestra.run.smithi084.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-20T22:27:40.200 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-20T22:27:40.200 DEBUG:teuthology.orchestra.run.smithi084:> sudo rpm -qi grub2-tools 2024-09-20T22:27:40.232 INFO:teuthology.orchestra.run.smithi084.stdout:Name : grub2-tools 2024-09-20T22:27:40.232 INFO:teuthology.orchestra.run.smithi084.stdout:Epoch : 1 2024-09-20T22:27:40.232 INFO:teuthology.orchestra.run.smithi084.stdout:Version : 2.06 2024-09-20T22:27:40.232 INFO:teuthology.orchestra.run.smithi084.stdout:Release : 82.el9 2024-09-20T22:27:40.232 INFO:teuthology.orchestra.run.smithi084.stdout:Architecture: x86_64 2024-09-20T22:27:40.232 INFO:teuthology.orchestra.run.smithi084.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-20T22:27:40.232 INFO:teuthology.orchestra.run.smithi084.stdout:Group : Unspecified 2024-09-20T22:27:40.232 INFO:teuthology.orchestra.run.smithi084.stdout:Size : 8274670 2024-09-20T22:27:40.232 INFO:teuthology.orchestra.run.smithi084.stdout:License : GPLv3+ 2024-09-20T22:27:40.232 INFO:teuthology.orchestra.run.smithi084.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-20T22:27:40.233 INFO:teuthology.orchestra.run.smithi084.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-20T22:27:40.233 INFO:teuthology.orchestra.run.smithi084.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-20T22:27:40.233 INFO:teuthology.orchestra.run.smithi084.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-20T22:27:40.233 INFO:teuthology.orchestra.run.smithi084.stdout:Packager : builder@centos.org 2024-09-20T22:27:40.233 INFO:teuthology.orchestra.run.smithi084.stdout:Vendor : CentOS 2024-09-20T22:27:40.233 INFO:teuthology.orchestra.run.smithi084.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-20T22:27:40.233 INFO:teuthology.orchestra.run.smithi084.stdout:Summary : Support tools for GRUB. 2024-09-20T22:27:40.233 INFO:teuthology.orchestra.run.smithi084.stdout:Description : 2024-09-20T22:27:40.233 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-20T22:27:40.233 INFO:teuthology.orchestra.run.smithi084.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-20T22:27:40.233 INFO:teuthology.orchestra.run.smithi084.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-20T22:27:40.233 INFO:teuthology.orchestra.run.smithi084.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-20T22:27:40.233 INFO:teuthology.orchestra.run.smithi084.stdout:hardware devices. 2024-09-20T22:27:40.233 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-20T22:27:40.233 INFO:teuthology.orchestra.run.smithi084.stdout:This subpackage provides tools for support of all platforms. 2024-09-20T22:27:40.234 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-20T22:27:40.234 INFO:teuthology.task.kernel:Updating grub on smithi084 to boot 5.14.0-508.el9.x86_64 2024-09-20T22:27:40.235 DEBUG:teuthology.orchestra.run.smithi084:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-20T22:27:40.865 INFO:teuthology.orchestra.run.smithi084.stderr:Generating grub configuration file ... 2024-09-20T22:27:42.215 INFO:teuthology.orchestra.run.smithi084.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-20T22:27:42.252 INFO:teuthology.orchestra.run.smithi084.stderr:done 2024-09-20T22:27:42.257 DEBUG:teuthology.orchestra.run.smithi084:> mktemp 2024-09-20T22:27:42.274 INFO:teuthology.orchestra.run.smithi084.stdout:/tmp/tmp.ANenLSe2ob 2024-09-20T22:27:42.274 DEBUG:teuthology.orchestra.run.smithi084:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.ANenLSe2ob 2024-09-20T22:27:42.340 DEBUG:teuthology.orchestra.run.smithi084:> sudo chmod 0666 /tmp/tmp.ANenLSe2ob 2024-09-20T22:27:42.477 DEBUG:teuthology.orchestra.remote:smithi084:/tmp/tmp.ANenLSe2ob is 6KB 2024-09-20T22:27:42.525 DEBUG:teuthology.orchestra.run.smithi084:> rm -fr /tmp/tmp.ANenLSe2ob 2024-09-20T22:27:42.540 DEBUG:teuthology.orchestra.run.smithi084:> sudo /bin/ls /boot/loader/entries || true 2024-09-20T22:27:42.604 INFO:teuthology.orchestra.run.smithi084.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-20T22:27:42.604 INFO:teuthology.orchestra.run.smithi084.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-20T22:27:42.604 INFO:teuthology.orchestra.run.smithi084.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-20T22:27:42.604 INFO:teuthology.orchestra.run.smithi084.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64.conf 2024-09-20T22:27:42.606 DEBUG:teuthology.orchestra.run.smithi084:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64 2024-09-20T22:27:42.762 DEBUG:teuthology.orchestra.run.smithi084:> sudo shutdown -r now 2024-09-20T22:27:46.523 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 4/4 2024-09-20T22:27:46.562 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 4/4 2024-09-20T22:27:46.885 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: kernel-5.14.0-508.el9.x86_64 4/4 2024-09-20T22:27:46.885 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : kernel-5.14.0-508.el9.x86_64 1/4 2024-09-20T22:27:46.885 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-20T22:27:46.885 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-20T22:27:47.778 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-20T22:27:47.778 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-20T22:27:47.778 INFO:teuthology.orchestra.run.smithi154.stdout:Installed: 2024-09-20T22:27:47.778 INFO:teuthology.orchestra.run.smithi154.stdout: kernel-5.14.0-508.el9.x86_64 2024-09-20T22:27:47.778 INFO:teuthology.orchestra.run.smithi154.stdout: kernel-core-5.14.0-508.el9.x86_64 2024-09-20T22:27:47.778 INFO:teuthology.orchestra.run.smithi154.stdout: kernel-modules-5.14.0-508.el9.x86_64 2024-09-20T22:27:47.779 INFO:teuthology.orchestra.run.smithi154.stdout: kernel-modules-core-5.14.0-508.el9.x86_64 2024-09-20T22:27:47.779 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-20T22:27:47.779 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2024-09-20T22:27:48.124 DEBUG:teuthology.orchestra.run.smithi154:> rpm -q kernel | sort -rV | head -n 1 2024-09-20T22:27:48.216 INFO:teuthology.orchestra.run.smithi154.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-20T22:27:48.216 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-20T22:27:48.216 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-508.el9.x86_64 2024-09-20T22:27:48.216 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-20T22:27:48.216 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-20T22:27:48.217 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-09-20T22:27:48.217 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi154.front.sepia.ceph.com, path=None, version=distro) 2024-09-20T22:27:48.217 DEBUG:teuthology.orchestra.run.smithi154:> rpm -q kernel | sort -rV | head -n 1 2024-09-20T22:27:48.275 INFO:teuthology.orchestra.run.smithi154.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-20T22:27:48.276 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-20T22:27:48.276 DEBUG:teuthology.orchestra.run.smithi154:> sudo rpm -qi grub2-tools 2024-09-20T22:27:48.315 INFO:teuthology.orchestra.run.smithi154.stdout:Name : grub2-tools 2024-09-20T22:27:48.315 INFO:teuthology.orchestra.run.smithi154.stdout:Epoch : 1 2024-09-20T22:27:48.315 INFO:teuthology.orchestra.run.smithi154.stdout:Version : 2.06 2024-09-20T22:27:48.315 INFO:teuthology.orchestra.run.smithi154.stdout:Release : 82.el9 2024-09-20T22:27:48.316 INFO:teuthology.orchestra.run.smithi154.stdout:Architecture: x86_64 2024-09-20T22:27:48.316 INFO:teuthology.orchestra.run.smithi154.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-20T22:27:48.316 INFO:teuthology.orchestra.run.smithi154.stdout:Group : Unspecified 2024-09-20T22:27:48.316 INFO:teuthology.orchestra.run.smithi154.stdout:Size : 8274670 2024-09-20T22:27:48.316 INFO:teuthology.orchestra.run.smithi154.stdout:License : GPLv3+ 2024-09-20T22:27:48.316 INFO:teuthology.orchestra.run.smithi154.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-20T22:27:48.316 INFO:teuthology.orchestra.run.smithi154.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-20T22:27:48.316 INFO:teuthology.orchestra.run.smithi154.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-20T22:27:48.316 INFO:teuthology.orchestra.run.smithi154.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-20T22:27:48.316 INFO:teuthology.orchestra.run.smithi154.stdout:Packager : builder@centos.org 2024-09-20T22:27:48.316 INFO:teuthology.orchestra.run.smithi154.stdout:Vendor : CentOS 2024-09-20T22:27:48.316 INFO:teuthology.orchestra.run.smithi154.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-20T22:27:48.316 INFO:teuthology.orchestra.run.smithi154.stdout:Summary : Support tools for GRUB. 2024-09-20T22:27:48.316 INFO:teuthology.orchestra.run.smithi154.stdout:Description : 2024-09-20T22:27:48.316 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-20T22:27:48.317 INFO:teuthology.orchestra.run.smithi154.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-20T22:27:48.317 INFO:teuthology.orchestra.run.smithi154.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-20T22:27:48.317 INFO:teuthology.orchestra.run.smithi154.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-20T22:27:48.317 INFO:teuthology.orchestra.run.smithi154.stdout:hardware devices. 2024-09-20T22:27:48.317 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-20T22:27:48.317 INFO:teuthology.orchestra.run.smithi154.stdout:This subpackage provides tools for support of all platforms. 2024-09-20T22:27:48.318 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-20T22:27:48.318 INFO:teuthology.task.kernel:Updating grub on smithi154 to boot 5.14.0-508.el9.x86_64 2024-09-20T22:27:48.318 DEBUG:teuthology.orchestra.run.smithi154:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-20T22:27:48.963 INFO:teuthology.orchestra.run.smithi154.stderr:Generating grub configuration file ... 2024-09-20T22:27:50.296 INFO:teuthology.orchestra.run.smithi154.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-20T22:27:50.323 INFO:teuthology.orchestra.run.smithi154.stderr:done 2024-09-20T22:27:50.329 DEBUG:teuthology.orchestra.run.smithi154:> mktemp 2024-09-20T22:27:50.344 INFO:teuthology.orchestra.run.smithi154.stdout:/tmp/tmp.pU6ZZeE72Q 2024-09-20T22:27:50.345 DEBUG:teuthology.orchestra.run.smithi154:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.pU6ZZeE72Q 2024-09-20T22:27:50.412 DEBUG:teuthology.orchestra.run.smithi154:> sudo chmod 0666 /tmp/tmp.pU6ZZeE72Q 2024-09-20T22:27:50.550 DEBUG:teuthology.orchestra.remote:smithi154:/tmp/tmp.pU6ZZeE72Q is 6KB 2024-09-20T22:27:50.570 DEBUG:teuthology.orchestra.run.smithi154:> rm -fr /tmp/tmp.pU6ZZeE72Q 2024-09-20T22:27:50.586 DEBUG:teuthology.orchestra.run.smithi154:> sudo /bin/ls /boot/loader/entries || true 2024-09-20T22:27:50.648 INFO:teuthology.orchestra.run.smithi154.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-20T22:27:50.648 INFO:teuthology.orchestra.run.smithi154.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-20T22:27:50.649 INFO:teuthology.orchestra.run.smithi154.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-20T22:27:50.649 INFO:teuthology.orchestra.run.smithi154.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64.conf 2024-09-20T22:27:50.650 DEBUG:teuthology.orchestra.run.smithi154:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64 2024-09-20T22:27:50.801 DEBUG:teuthology.orchestra.run.smithi154:> sudo shutdown -r now 2024-09-20T22:28:12.781 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-20T22:28:12.782 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi084.front.sepia.ceph.com' 2024-09-20T22:28:12.782 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi084.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-20T22:28:20.810 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-09-20T22:28:20.811 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi154.front.sepia.ceph.com' 2024-09-20T22:28:20.811 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi154.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-20T22:28:55.407 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.154 2024-09-20T22:29:04.416 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi154.front.sepia.ceph.com' 2024-09-20T22:29:04.417 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi154.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-20T22:29:12.788 DEBUG:teuthology.orchestra.remote:timed out 2024-09-20T22:29:21.797 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi084.front.sepia.ceph.com' 2024-09-20T22:29:21.797 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi084.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-20T22:29:22.113 DEBUG:teuthology.orchestra.run.smithi084:> true 2024-09-20T22:29:22.432 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi084.front.sepia.ceph.com' 2024-09-20T22:29:22.432 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-508.el9.x86_64"... 2024-09-20T22:29:22.432 DEBUG:teuthology.orchestra.run.smithi084:> uname -r 2024-09-20T22:29:22.446 INFO:teuthology.orchestra.run.smithi084.stdout:5.14.0-508.el9.x86_64 2024-09-20T22:29:22.447 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-508.el9.x86_64 vs 5.14.0-508.el9.x86_64 2024-09-20T22:29:22.447 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-20T22:29:22.447 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-20T22:29:22.799 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.154 2024-09-20T22:29:23.448 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-20T22:29:23.448 DEBUG:teuthology.orchestra.run.smithi084:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-20T22:29:23.545 INFO:teuthology.orchestra.run.smithi084.stdout:ttyS1 2024-09-20T22:29:23.569 DEBUG:teuthology.parallel:result is None 2024-09-20T22:29:34.800 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi154.front.sepia.ceph.com' 2024-09-20T22:29:34.801 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi154.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-20T22:29:35.253 DEBUG:teuthology.orchestra.run.smithi154:> true 2024-09-20T22:29:35.668 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi154.front.sepia.ceph.com' 2024-09-20T22:29:35.668 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-508.el9.x86_64"... 2024-09-20T22:29:35.668 DEBUG:teuthology.orchestra.run.smithi154:> uname -r 2024-09-20T22:29:35.682 INFO:teuthology.orchestra.run.smithi154.stdout:5.14.0-508.el9.x86_64 2024-09-20T22:29:35.682 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-508.el9.x86_64 vs 5.14.0-508.el9.x86_64 2024-09-20T22:29:35.682 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-20T22:29:35.683 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-20T22:29:36.684 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-09-20T22:29:36.684 DEBUG:teuthology.orchestra.run.smithi154:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-20T22:29:36.765 INFO:teuthology.orchestra.run.smithi154.stdout:ttyS1 2024-09-20T22:29:36.801 DEBUG:teuthology.parallel:result is None 2024-09-20T22:29:36.802 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-20T22:29:36.809 INFO:teuthology.task.internal:Creating test directory... 2024-09-20T22:29:36.810 DEBUG:teuthology.orchestra.run.smithi084:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-20T22:29:36.813 DEBUG:teuthology.orchestra.run.smithi154:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-20T22:29:36.829 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-20T22:29:36.836 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-20T22:29:36.842 INFO:teuthology.task.internal:Creating archive directory... 2024-09-20T22:29:36.842 DEBUG:teuthology.orchestra.run.smithi084:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-20T22:29:36.872 DEBUG:teuthology.orchestra.run.smithi154:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-20T22:29:36.992 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-20T22:29:36.998 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-20T22:29:36.999 DEBUG:teuthology.orchestra.run.smithi084:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-20T22:29:37.001 DEBUG:teuthology.orchestra.run.smithi154:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-20T22:29:37.028 INFO:teuthology.orchestra.run.smithi084.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-20T22:29:37.038 INFO:teuthology.orchestra.run.smithi154.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-20T22:29:37.040 INFO:teuthology.orchestra.run.smithi084.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-20T22:29:37.050 INFO:teuthology.orchestra.run.smithi154.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-20T22:29:37.051 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-20T22:29:37.058 INFO:teuthology.task.internal:Configuring sudo... 2024-09-20T22:29:37.058 DEBUG:teuthology.orchestra.run.smithi084:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-20T22:29:37.085 DEBUG:teuthology.orchestra.run.smithi154:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-20T22:29:37.118 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-20T22:29:37.127 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-20T22:29:37.127 DEBUG:teuthology.orchestra.run.smithi084:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-20T22:29:37.152 DEBUG:teuthology.orchestra.run.smithi154:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-20T22:29:37.174 DEBUG:teuthology.orchestra.run.smithi084:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-20T22:29:37.248 DEBUG:teuthology.orchestra.run.smithi084:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-20T22:29:37.302 DEBUG:teuthology.orchestra.run.smithi084:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-20T22:29:37.383 DEBUG:teuthology.orchestra.run.smithi084:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-20T22:29:37.424 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-09-20T22:29:37.424 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-20T22:29:37.489 DEBUG:teuthology.orchestra.run.smithi154:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-20T22:29:37.527 DEBUG:teuthology.orchestra.run.smithi154:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-20T22:29:37.579 DEBUG:teuthology.orchestra.run.smithi154:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-20T22:29:37.658 DEBUG:teuthology.orchestra.run.smithi154:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-20T22:29:37.693 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-09-20T22:29:37.693 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-20T22:29:37.755 DEBUG:teuthology.orchestra.run.smithi084:> sudo service rsyslog restart 2024-09-20T22:29:37.757 DEBUG:teuthology.orchestra.run.smithi154:> sudo service rsyslog restart 2024-09-20T22:29:37.803 INFO:teuthology.orchestra.run.smithi084.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-20T22:29:37.841 INFO:teuthology.orchestra.run.smithi154.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-20T22:29:38.146 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-20T22:29:38.154 INFO:teuthology.task.internal:Starting timer... 2024-09-20T22:29:38.154 INFO:teuthology.run_tasks:Running task pcp... 2024-09-20T22:29:38.163 INFO:teuthology.run_tasks:Running task selinux... 2024-09-20T22:29:38.172 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-09-20T22:29:38.172 DEBUG:teuthology.orchestra.run.smithi084:> sudo service auditd rotate 2024-09-20T22:29:38.232 INFO:teuthology.orchestra.run.smithi084.stdout:Rotating logs: 2024-09-20T22:29:38.234 DEBUG:teuthology.orchestra.run.smithi154:> sudo service auditd rotate 2024-09-20T22:29:38.322 INFO:teuthology.orchestra.run.smithi154.stdout:Rotating logs: 2024-09-20T22:29:38.324 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-20T22:29:38.325 DEBUG:teuthology.orchestra.run.smithi084:> /usr/sbin/getenforce 2024-09-20T22:29:38.369 INFO:teuthology.orchestra.run.smithi084.stdout:Permissive 2024-09-20T22:29:38.370 DEBUG:teuthology.orchestra.run.smithi154:> /usr/sbin/getenforce 2024-09-20T22:29:38.389 INFO:teuthology.orchestra.run.smithi154.stdout:Permissive 2024-09-20T22:29:38.389 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi084.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi154.front.sepia.ceph.com': 'permissive'} 2024-09-20T22:29:38.389 DEBUG:teuthology.orchestra.run.smithi084:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-20T22:29:38.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:29:38.414 DEBUG:teuthology.orchestra.run.smithi154:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-20T22:29:38.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:29:38.452 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-20T22:29:38.453 DEBUG:teuthology.orchestra.run.smithi084:> sudo /usr/sbin/setenforce permissive 2024-09-20T22:29:38.497 DEBUG:teuthology.orchestra.run.smithi154:> sudo /usr/sbin/setenforce permissive 2024-09-20T22:29:38.527 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-20T22:29:38.537 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-20T22:29:38.547 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-20T22:29:38.547 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi084.front.sepia.ceph.com,smithi154.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-20T22:33:16.867 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi084.front.sepia.ceph.com'), Remote(name='ubuntu@smithi154.front.sepia.ceph.com')] 2024-09-20T22:33:16.868 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi084.front.sepia.ceph.com' 2024-09-20T22:33:16.868 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi084.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-20T22:33:16.942 DEBUG:teuthology.orchestra.run.smithi084:> true 2024-09-20T22:33:17.017 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi084.front.sepia.ceph.com' 2024-09-20T22:33:17.018 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi154.front.sepia.ceph.com' 2024-09-20T22:33:17.018 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi154.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-20T22:33:17.093 DEBUG:teuthology.orchestra.run.smithi154:> true 2024-09-20T22:33:17.178 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi154.front.sepia.ceph.com' 2024-09-20T22:33:17.178 INFO:teuthology.run_tasks:Running task clock... 2024-09-20T22:33:17.189 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-20T22:33:17.189 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-20T22:33:17.190 DEBUG:teuthology.orchestra.run.smithi084:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-20T22:33:17.192 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-20T22:33:17.192 DEBUG:teuthology.orchestra.run.smithi154:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-20T22:33:17.219 INFO:teuthology.orchestra.run.smithi084.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-20T22:33:17.233 INFO:teuthology.orchestra.run.smithi084.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-20T22:33:17.248 INFO:teuthology.orchestra.run.smithi154.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-20T22:33:17.257 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: ntpd: command not found 2024-09-20T22:33:17.262 INFO:teuthology.orchestra.run.smithi154.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-20T22:33:17.268 INFO:teuthology.orchestra.run.smithi084.stdout:506 Cannot talk to daemon 2024-09-20T22:33:17.281 INFO:teuthology.orchestra.run.smithi084.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-20T22:33:17.287 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: ntpd: command not found 2024-09-20T22:33:17.295 INFO:teuthology.orchestra.run.smithi084.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-20T22:33:17.298 INFO:teuthology.orchestra.run.smithi154.stdout:506 Cannot talk to daemon 2024-09-20T22:33:17.313 INFO:teuthology.orchestra.run.smithi154.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-20T22:33:17.327 INFO:teuthology.orchestra.run.smithi154.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-20T22:33:17.353 INFO:teuthology.orchestra.run.smithi084.stderr:bash: line 1: ntpq: command not found 2024-09-20T22:33:17.357 INFO:teuthology.orchestra.run.smithi084.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-20T22:33:17.357 INFO:teuthology.orchestra.run.smithi084.stdout:=============================================================================== 2024-09-20T22:33:17.357 INFO:teuthology.orchestra.run.smithi084.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-20T22:33:17.357 INFO:teuthology.orchestra.run.smithi084.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-20T22:33:17.357 INFO:teuthology.orchestra.run.smithi084.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-20T22:33:17.357 INFO:teuthology.orchestra.run.smithi084.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-20T22:33:17.389 INFO:teuthology.orchestra.run.smithi154.stderr:bash: line 1: ntpq: command not found 2024-09-20T22:33:17.393 INFO:teuthology.orchestra.run.smithi154.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-20T22:33:17.393 INFO:teuthology.orchestra.run.smithi154.stdout:=============================================================================== 2024-09-20T22:33:17.393 INFO:teuthology.orchestra.run.smithi154.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-20T22:33:17.393 INFO:teuthology.orchestra.run.smithi154.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-20T22:33:17.393 INFO:teuthology.orchestra.run.smithi154.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-20T22:33:17.393 INFO:teuthology.orchestra.run.smithi154.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-20T22:33:17.394 INFO:teuthology.run_tasks:Running task pexec... 2024-09-20T22:33:17.403 INFO:teuthology.task.pexec:Executing custom commands... 2024-09-20T22:33:17.403 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi084.front.sepia.ceph.com 2024-09-20T22:33:17.403 DEBUG:teuthology.orchestra.run.smithi084:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-09-20T22:33:17.404 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi154.front.sepia.ceph.com 2024-09-20T22:33:17.404 DEBUG:teuthology.orchestra.run.smithi154:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-09-20T22:33:17.755 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-09-20T22:33:17.757 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-09-20T22:33:17.757 INFO:teuthology.orchestra.run.smithi084.stdout: Package Architecture Version Repository Size 2024-09-20T22:33:17.757 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-09-20T22:33:17.757 INFO:teuthology.orchestra.run.smithi084.stdout:Removing: 2024-09-20T22:33:17.757 INFO:teuthology.orchestra.run.smithi084.stdout: nvme-cli x86_64 2.9.1-6.el9 @baseos 5.3 M 2024-09-20T22:33:17.758 INFO:teuthology.orchestra.run.smithi084.stdout:Removing unused dependencies: 2024-09-20T22:33:17.758 INFO:teuthology.orchestra.run.smithi084.stdout: libnvme x86_64 1.9-3.el9 @baseos 273 k 2024-09-20T22:33:17.758 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-20T22:33:17.758 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2024-09-20T22:33:17.758 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-09-20T22:33:17.758 INFO:teuthology.orchestra.run.smithi084.stdout:Remove 2 Packages 2024-09-20T22:33:17.758 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-20T22:33:17.758 INFO:teuthology.orchestra.run.smithi084.stdout:Freed space: 5.5 M 2024-09-20T22:33:17.758 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2024-09-20T22:33:17.762 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2024-09-20T22:33:17.762 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2024-09-20T22:33:17.786 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2024-09-20T22:33:17.795 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-09-20T22:33:17.795 INFO:teuthology.orchestra.run.smithi154.stdout: Package Architecture Version Repository Size 2024-09-20T22:33:17.795 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-09-20T22:33:17.795 INFO:teuthology.orchestra.run.smithi154.stdout:Removing: 2024-09-20T22:33:17.795 INFO:teuthology.orchestra.run.smithi154.stdout: nvme-cli x86_64 2.9.1-6.el9 @baseos 5.3 M 2024-09-20T22:33:17.795 INFO:teuthology.orchestra.run.smithi154.stdout:Removing unused dependencies: 2024-09-20T22:33:17.795 INFO:teuthology.orchestra.run.smithi154.stdout: libnvme x86_64 1.9-3.el9 @baseos 273 k 2024-09-20T22:33:17.795 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-20T22:33:17.795 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2024-09-20T22:33:17.795 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-09-20T22:33:17.795 INFO:teuthology.orchestra.run.smithi154.stdout:Remove 2 Packages 2024-09-20T22:33:17.795 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-20T22:33:17.795 INFO:teuthology.orchestra.run.smithi154.stdout:Freed space: 5.5 M 2024-09-20T22:33:17.795 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2024-09-20T22:33:17.799 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2024-09-20T22:33:17.799 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2024-09-20T22:33:17.829 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2024-09-20T22:33:17.830 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2024-09-20T22:33:17.865 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2024-09-20T22:33:17.865 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2024-09-20T22:33:17.965 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2024-09-20T22:33:18.003 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2024-09-20T22:33:18.051 INFO:teuthology.orchestra.run.smithi084.stdout: Erasing : nvme-cli-2.9.1-6.el9.x86_64 1/2 2024-09-20T22:33:18.101 INFO:teuthology.orchestra.run.smithi084.stdout: Erasing : libnvme-1.9-3.el9.x86_64 2/2 2024-09-20T22:33:18.114 INFO:teuthology.orchestra.run.smithi154.stdout: Erasing : nvme-cli-2.9.1-6.el9.x86_64 1/2 2024-09-20T22:33:18.181 INFO:teuthology.orchestra.run.smithi154.stdout: Erasing : libnvme-1.9-3.el9.x86_64 2/2 2024-09-20T22:33:18.590 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: libnvme-1.9-3.el9.x86_64 2/2 2024-09-20T22:33:18.590 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : libnvme-1.9-3.el9.x86_64 1/2 2024-09-20T22:33:18.792 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: libnvme-1.9-3.el9.x86_64 2/2 2024-09-20T22:33:18.792 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libnvme-1.9-3.el9.x86_64 1/2 2024-09-20T22:33:18.809 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : nvme-cli-2.9.1-6.el9.x86_64 2/2 2024-09-20T22:33:18.809 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-20T22:33:18.809 INFO:teuthology.orchestra.run.smithi084.stdout:Removed: 2024-09-20T22:33:18.809 INFO:teuthology.orchestra.run.smithi084.stdout: libnvme-1.9-3.el9.x86_64 nvme-cli-2.9.1-6.el9.x86_64 2024-09-20T22:33:18.810 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-20T22:33:18.810 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-09-20T22:33:19.098 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : nvme-cli-2.9.1-6.el9.x86_64 2/2 2024-09-20T22:33:19.098 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-20T22:33:19.098 INFO:teuthology.orchestra.run.smithi154.stdout:Removed: 2024-09-20T22:33:19.098 INFO:teuthology.orchestra.run.smithi154.stdout: libnvme-1.9-3.el9.x86_64 nvme-cli-2.9.1-6.el9.x86_64 2024-09-20T22:33:19.098 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-20T22:33:19.098 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2024-09-20T22:33:19.638 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:00:46 ago on Fri 20 Sep 2024 10:32:33 PM UTC. 2024-09-20T22:33:19.771 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-09-20T22:33:19.772 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-09-20T22:33:19.773 INFO:teuthology.orchestra.run.smithi084.stdout: Package Architecture Version Repository Size 2024-09-20T22:33:19.773 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-09-20T22:33:19.773 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2024-09-20T22:33:19.773 INFO:teuthology.orchestra.run.smithi084.stdout: nvme-cli x86_64 2.9.1-6.el9 baseos 906 k 2024-09-20T22:33:19.773 INFO:teuthology.orchestra.run.smithi084.stdout: nvmetcli noarch 0.7-3.el9 baseos 44 k 2024-09-20T22:33:19.773 INFO:teuthology.orchestra.run.smithi084.stdout:Installing dependencies: 2024-09-20T22:33:19.773 INFO:teuthology.orchestra.run.smithi084.stdout: libnvme x86_64 1.9-3.el9 baseos 106 k 2024-09-20T22:33:19.773 INFO:teuthology.orchestra.run.smithi084.stdout: python3-configshell noarch 1:1.1.30-1.el9 baseos 72 k 2024-09-20T22:33:19.773 INFO:teuthology.orchestra.run.smithi084.stdout: python3-kmod x86_64 0.9-32.el9 baseos 84 k 2024-09-20T22:33:19.773 INFO:teuthology.orchestra.run.smithi084.stdout: python3-urwid x86_64 2.1.2-4.el9 baseos 837 k 2024-09-20T22:33:19.773 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-20T22:33:19.773 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2024-09-20T22:33:19.773 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-09-20T22:33:19.773 INFO:teuthology.orchestra.run.smithi084.stdout:Install 6 Packages 2024-09-20T22:33:19.773 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-20T22:33:19.782 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 2.0 M 2024-09-20T22:33:19.782 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 9.2 M 2024-09-20T22:33:19.782 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2024-09-20T22:33:19.947 INFO:teuthology.orchestra.run.smithi084.stdout:(1/6): nvmetcli-0.7-3.el9.noarch.rpm 422 kB/s | 44 kB 00:00 2024-09-20T22:33:19.968 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:00:46 ago on Fri 20 Sep 2024 10:32:33 PM UTC. 2024-09-20T22:33:19.981 INFO:teuthology.orchestra.run.smithi084.stdout:(2/6): libnvme-1.9-3.el9.x86_64.rpm 763 kB/s | 106 kB 00:00 2024-09-20T22:33:20.006 INFO:teuthology.orchestra.run.smithi084.stdout:(3/6): python3-configshell-1.1.30-1.el9.noarch. 1.2 MB/s | 72 kB 00:00 2024-09-20T22:33:20.048 INFO:teuthology.orchestra.run.smithi084.stdout:(4/6): python3-kmod-0.9-32.el9.x86_64.rpm 1.2 MB/s | 84 kB 00:00 2024-09-20T22:33:20.082 INFO:teuthology.orchestra.run.smithi084.stdout:(5/6): nvme-cli-2.9.1-6.el9.x86_64.rpm 3.7 MB/s | 906 kB 00:00 2024-09-20T22:33:20.103 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2024-09-20T22:33:20.104 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-09-20T22:33:20.104 INFO:teuthology.orchestra.run.smithi154.stdout: Package Architecture Version Repository Size 2024-09-20T22:33:20.104 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-09-20T22:33:20.104 INFO:teuthology.orchestra.run.smithi154.stdout:Installing: 2024-09-20T22:33:20.104 INFO:teuthology.orchestra.run.smithi154.stdout: nvme-cli x86_64 2.9.1-6.el9 baseos 906 k 2024-09-20T22:33:20.104 INFO:teuthology.orchestra.run.smithi154.stdout: nvmetcli noarch 0.7-3.el9 baseos 44 k 2024-09-20T22:33:20.104 INFO:teuthology.orchestra.run.smithi154.stdout:Installing dependencies: 2024-09-20T22:33:20.105 INFO:teuthology.orchestra.run.smithi154.stdout: libnvme x86_64 1.9-3.el9 baseos 106 k 2024-09-20T22:33:20.105 INFO:teuthology.orchestra.run.smithi154.stdout: python3-configshell noarch 1:1.1.30-1.el9 baseos 72 k 2024-09-20T22:33:20.105 INFO:teuthology.orchestra.run.smithi154.stdout: python3-kmod x86_64 0.9-32.el9 baseos 84 k 2024-09-20T22:33:20.105 INFO:teuthology.orchestra.run.smithi154.stdout: python3-urwid x86_64 2.1.2-4.el9 baseos 837 k 2024-09-20T22:33:20.105 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-20T22:33:20.105 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2024-09-20T22:33:20.105 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-09-20T22:33:20.105 INFO:teuthology.orchestra.run.smithi154.stdout:Install 6 Packages 2024-09-20T22:33:20.105 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-20T22:33:20.113 INFO:teuthology.orchestra.run.smithi154.stdout:Total download size: 2.0 M 2024-09-20T22:33:20.113 INFO:teuthology.orchestra.run.smithi154.stdout:Installed size: 9.2 M 2024-09-20T22:33:20.113 INFO:teuthology.orchestra.run.smithi154.stdout:Downloading Packages: 2024-09-20T22:33:20.124 INFO:teuthology.orchestra.run.smithi084.stdout:(6/6): python3-urwid-2.1.2-4.el9.x86_64.rpm 7.0 MB/s | 837 kB 00:00 2024-09-20T22:33:20.124 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2024-09-20T22:33:20.124 INFO:teuthology.orchestra.run.smithi084.stdout:Total 5.9 MB/s | 2.0 MB 00:00 2024-09-20T22:33:20.197 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2024-09-20T22:33:20.211 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2024-09-20T22:33:20.211 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2024-09-20T22:33:20.518 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2024-09-20T22:33:20.518 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2024-09-20T22:33:20.753 INFO:teuthology.orchestra.run.smithi154.stdout:(1/6): nvmetcli-0.7-3.el9.noarch.rpm 76 kB/s | 44 kB 00:00 2024-09-20T22:33:20.928 INFO:teuthology.orchestra.run.smithi154.stdout:(2/6): libnvme-1.9-3.el9.x86_64.rpm 139 kB/s | 106 kB 00:00 2024-09-20T22:33:20.938 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2024-09-20T22:33:21.018 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-urwid-2.1.2-4.el9.x86_64 1/6 2024-09-20T22:33:21.061 INFO:teuthology.orchestra.run.smithi154.stdout:(3/6): nvme-cli-2.9.1-6.el9.x86_64.rpm 1.0 MB/s | 906 kB 00:00 2024-09-20T22:33:21.062 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-configshell-1:1.1.30-1.el9.noarch 2/6 2024-09-20T22:33:21.097 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-kmod-0.9-32.el9.x86_64 3/6 2024-09-20T22:33:21.308 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : libnvme-1.9-3.el9.x86_64 4/6 2024-09-20T22:33:21.353 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : nvme-cli-2.9.1-6.el9.x86_64 5/6 2024-09-20T22:33:21.395 INFO:teuthology.orchestra.run.smithi154.stdout:(4/6): python3-configshell-1.1.30-1.el9.noarch. 112 kB/s | 72 kB 00:00 2024-09-20T22:33:21.628 INFO:teuthology.orchestra.run.smithi154.stdout:(5/6): python3-kmod-0.9-32.el9.x86_64.rpm 120 kB/s | 84 kB 00:00 2024-09-20T22:33:21.791 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: nvme-cli-2.9.1-6.el9.x86_64 5/6 2024-09-20T22:33:21.815 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : nvmetcli-0.7-3.el9.noarch 6/6 2024-09-20T22:33:21.954 INFO:teuthology.orchestra.run.smithi154.stdout:(6/6): python3-urwid-2.1.2-4.el9.x86_64.rpm 940 kB/s | 837 kB 00:00 2024-09-20T22:33:21.954 INFO:teuthology.orchestra.run.smithi154.stdout:-------------------------------------------------------------------------------- 2024-09-20T22:33:21.954 INFO:teuthology.orchestra.run.smithi154.stdout:Total 1.1 MB/s | 2.0 MB 00:01 2024-09-20T22:33:22.030 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2024-09-20T22:33:22.045 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2024-09-20T22:33:22.045 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2024-09-20T22:33:22.315 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2024-09-20T22:33:22.315 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2024-09-20T22:33:22.746 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2024-09-20T22:33:22.840 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-urwid-2.1.2-4.el9.x86_64 1/6 2024-09-20T22:33:22.853 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: nvmetcli-0.7-3.el9.noarch 6/6 2024-09-20T22:33:22.853 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : libnvme-1.9-3.el9.x86_64 1/6 2024-09-20T22:33:22.854 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : nvme-cli-2.9.1-6.el9.x86_64 2/6 2024-09-20T22:33:22.854 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : nvmetcli-0.7-3.el9.noarch 3/6 2024-09-20T22:33:22.854 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-configshell-1:1.1.30-1.el9.noarch 4/6 2024-09-20T22:33:22.854 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-kmod-0.9-32.el9.x86_64 5/6 2024-09-20T22:33:22.874 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-configshell-1:1.1.30-1.el9.noarch 2/6 2024-09-20T22:33:22.902 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-kmod-0.9-32.el9.x86_64 3/6 2024-09-20T22:33:23.127 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : libnvme-1.9-3.el9.x86_64 4/6 2024-09-20T22:33:23.167 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : nvme-cli-2.9.1-6.el9.x86_64 5/6 2024-09-20T22:33:23.241 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-urwid-2.1.2-4.el9.x86_64 6/6 2024-09-20T22:33:23.241 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-20T22:33:23.241 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2024-09-20T22:33:23.241 INFO:teuthology.orchestra.run.smithi084.stdout: libnvme-1.9-3.el9.x86_64 nvme-cli-2.9.1-6.el9.x86_64 2024-09-20T22:33:23.241 INFO:teuthology.orchestra.run.smithi084.stdout: nvmetcli-0.7-3.el9.noarch python3-configshell-1:1.1.30-1.el9.noarch 2024-09-20T22:33:23.241 INFO:teuthology.orchestra.run.smithi084.stdout: python3-kmod-0.9-32.el9.x86_64 python3-urwid-2.1.2-4.el9.x86_64 2024-09-20T22:33:23.241 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-20T22:33:23.241 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-09-20T22:33:23.577 DEBUG:teuthology.parallel:result is None 2024-09-20T22:33:23.608 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: nvme-cli-2.9.1-6.el9.x86_64 5/6 2024-09-20T22:33:23.636 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : nvmetcli-0.7-3.el9.noarch 6/6 2024-09-20T22:33:24.597 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: nvmetcli-0.7-3.el9.noarch 6/6 2024-09-20T22:33:24.597 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libnvme-1.9-3.el9.x86_64 1/6 2024-09-20T22:33:24.597 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : nvme-cli-2.9.1-6.el9.x86_64 2/6 2024-09-20T22:33:24.597 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : nvmetcli-0.7-3.el9.noarch 3/6 2024-09-20T22:33:24.597 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-configshell-1:1.1.30-1.el9.noarch 4/6 2024-09-20T22:33:24.597 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-kmod-0.9-32.el9.x86_64 5/6 2024-09-20T22:33:24.915 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-urwid-2.1.2-4.el9.x86_64 6/6 2024-09-20T22:33:24.915 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-20T22:33:24.915 INFO:teuthology.orchestra.run.smithi154.stdout:Installed: 2024-09-20T22:33:24.915 INFO:teuthology.orchestra.run.smithi154.stdout: libnvme-1.9-3.el9.x86_64 nvme-cli-2.9.1-6.el9.x86_64 2024-09-20T22:33:24.915 INFO:teuthology.orchestra.run.smithi154.stdout: nvmetcli-0.7-3.el9.noarch python3-configshell-1:1.1.30-1.el9.noarch 2024-09-20T22:33:24.915 INFO:teuthology.orchestra.run.smithi154.stdout: python3-kmod-0.9-32.el9.x86_64 python3-urwid-2.1.2-4.el9.x86_64 2024-09-20T22:33:24.915 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-20T22:33:24.915 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2024-09-20T22:33:25.251 DEBUG:teuthology.parallel:result is None 2024-09-20T22:33:25.251 INFO:teuthology.run_tasks:Running task install... 2024-09-20T22:33:25.261 DEBUG:teuthology.task.install:project ceph 2024-09-20T22:33:25.261 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '9e2ac70bb7aea67d34f0d50002344abbe7803736'}} 2024-09-20T22:33:25.261 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '9e2ac70bb7aea67d34f0d50002344abbe7803736'} 2024-09-20T22:33:25.261 INFO:teuthology.task.install:Using flavor: default 2024-09-20T22:33:25.266 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-20T22:33:25.266 INFO:teuthology.task.install:extra packages: [] 2024-09-20T22:33:25.266 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '9e2ac70bb7aea67d34f0d50002344abbe7803736', 'tag': None, 'wait_for_package': False} 2024-09-20T22:33:25.267 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=9e2ac70bb7aea67d34f0d50002344abbe7803736 2024-09-20T22:33:25.268 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '9e2ac70bb7aea67d34f0d50002344abbe7803736', 'tag': None, 'wait_for_package': False} 2024-09-20T22:33:25.268 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=9e2ac70bb7aea67d34f0d50002344abbe7803736 2024-09-20T22:33:25.410 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/squid/9e2ac70bb7aea67d34f0d50002344abbe7803736/centos/9/flavors/default/ 2024-09-20T22:33:25.410 INFO:teuthology.task.install.rpm:Package version is 19.1.1-367.g9e2ac70b 2024-09-20T22:33:25.414 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/squid/9e2ac70bb7aea67d34f0d50002344abbe7803736/centos/9/flavors/default/ 2024-09-20T22:33:25.414 INFO:teuthology.task.install.rpm:Package version is 19.1.1-367.g9e2ac70b 2024-09-20T22:33:25.518 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/squid/9e2ac70bb7aea67d34f0d50002344abbe7803736/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/squid/9e2ac70bb7aea67d34f0d50002344abbe7803736/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/squid/9e2ac70bb7aea67d34f0d50002344abbe7803736/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-20T22:33:25.518 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-09-20T22:33:25.519 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-20T22:33:25.532 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/squid/9e2ac70bb7aea67d34f0d50002344abbe7803736/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/squid/9e2ac70bb7aea67d34f0d50002344abbe7803736/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/squid/9e2ac70bb7aea67d34f0d50002344abbe7803736/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-20T22:33:25.532 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-09-20T22:33:25.532 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-20T22:33:25.552 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-20T22:33:25.552 DEBUG:teuthology.orchestra.run.smithi084:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/9e2ac70bb7aea67d34f0d50002344abbe7803736/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-20T22:33:25.563 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-20T22:33:25.564 DEBUG:teuthology.orchestra.run.smithi154:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/9e2ac70bb7aea67d34f0d50002344abbe7803736/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-20T22:33:25.625 DEBUG:teuthology.orchestra.run.smithi084:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-20T22:33:25.631 DEBUG:teuthology.orchestra.run.smithi154:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-20T22:33:25.704 DEBUG:teuthology.orchestra.run.smithi084:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-20T22:33:25.708 DEBUG:teuthology.orchestra.run.smithi154:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-20T22:33:25.770 INFO:teuthology.orchestra.run.smithi084.stdout:check_obsoletes = 1 2024-09-20T22:33:25.771 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum clean all 2024-09-20T22:33:25.785 INFO:teuthology.orchestra.run.smithi154.stdout:check_obsoletes = 1 2024-09-20T22:33:25.786 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum clean all 2024-09-20T22:33:25.990 INFO:teuthology.orchestra.run.smithi084.stdout:45 files removed 2024-09-20T22:33:26.017 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install ceph-radosgw 2024-09-20T22:33:26.021 INFO:teuthology.orchestra.run.smithi154.stdout:45 files removed 2024-09-20T22:33:26.048 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install ceph-radosgw 2024-09-20T22:33:26.570 INFO:teuthology.orchestra.run.smithi084.stdout:ceph packages for x86_64 308 kB/s | 84 kB 00:00 2024-09-20T22:33:26.601 INFO:teuthology.orchestra.run.smithi154.stdout:ceph packages for x86_64 310 kB/s | 84 kB 00:00 2024-09-20T22:33:26.862 INFO:teuthology.orchestra.run.smithi084.stdout:ceph noarch packages 47 kB/s | 12 kB 00:00 2024-09-20T22:33:26.885 INFO:teuthology.orchestra.run.smithi154.stdout:ceph noarch packages 49 kB/s | 12 kB 00:00 2024-09-20T22:33:27.137 INFO:teuthology.orchestra.run.smithi084.stdout:ceph source packages 7.9 kB/s | 1.9 kB 00:00 2024-09-20T22:33:27.169 INFO:teuthology.orchestra.run.smithi154.stdout:ceph source packages 7.6 kB/s | 1.9 kB 00:00 2024-09-20T22:33:27.712 INFO:teuthology.orchestra.run.smithi084.stdout:CentOS Stream 9 - BaseOS 15 MB/s | 8.3 MB 00:00 2024-09-20T22:33:27.869 INFO:teuthology.orchestra.run.smithi154.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.3 MB 00:00 2024-09-20T22:33:29.712 INFO:teuthology.orchestra.run.smithi154.stdout:CentOS Stream 9 - AppStream 26 MB/s | 20 MB 00:00 2024-09-20T22:33:31.013 INFO:teuthology.orchestra.run.smithi084.stdout:CentOS Stream 9 - AppStream 9.1 MB/s | 20 MB 00:02 2024-09-20T22:33:34.971 INFO:teuthology.orchestra.run.smithi154.stdout:CentOS Stream 9 - CRB 4.4 MB/s | 6.5 MB 00:01 2024-09-20T22:33:36.588 INFO:teuthology.orchestra.run.smithi084.stdout:CentOS Stream 9 - CRB 3.7 MB/s | 6.5 MB 00:01 2024-09-20T22:33:36.638 INFO:teuthology.orchestra.run.smithi154.stdout:CentOS Stream 9 - Extras packages 45 kB/s | 19 kB 00:00 2024-09-20T22:33:37.726 INFO:teuthology.orchestra.run.smithi154.stdout:Extra Packages for Enterprise Linux 24 MB/s | 23 MB 00:00 2024-09-20T22:33:38.197 INFO:teuthology.orchestra.run.smithi084.stdout:CentOS Stream 9 - Extras packages 72 kB/s | 19 kB 00:00 2024-09-20T22:33:39.076 INFO:teuthology.orchestra.run.smithi084.stdout:Extra Packages for Enterprise Linux 30 MB/s | 23 MB 00:00 2024-09-20T22:33:43.098 INFO:teuthology.orchestra.run.smithi154.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-09-20T22:33:44.474 INFO:teuthology.orchestra.run.smithi084.stdout:lab-extras 20 kB/s | 1.7 kB 00:00 2024-09-20T22:33:45.343 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2024-09-20T22:33:45.345 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-09-20T22:33:45.345 INFO:teuthology.orchestra.run.smithi154.stdout: Package Arch Version Repository Size 2024-09-20T22:33:45.345 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-09-20T22:33:45.345 INFO:teuthology.orchestra.run.smithi154.stdout:Installing: 2024-09-20T22:33:45.345 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-radosgw x86_64 2:19.1.1-367.g9e2ac70b.el9 ceph 11 M 2024-09-20T22:33:45.345 INFO:teuthology.orchestra.run.smithi154.stdout:Upgrading: 2024-09-20T22:33:45.345 INFO:teuthology.orchestra.run.smithi154.stdout: librados2 x86_64 2:19.1.1-367.g9e2ac70b.el9 ceph 3.4 M 2024-09-20T22:33:45.346 INFO:teuthology.orchestra.run.smithi154.stdout: librbd1 x86_64 2:19.1.1-367.g9e2ac70b.el9 ceph 3.2 M 2024-09-20T22:33:45.346 INFO:teuthology.orchestra.run.smithi154.stdout:Installing dependencies: 2024-09-20T22:33:45.346 INFO:teuthology.orchestra.run.smithi154.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-20T22:33:45.346 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-base x86_64 2:19.1.1-367.g9e2ac70b.el9 ceph 5.4 M 2024-09-20T22:33:45.346 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-common x86_64 2:19.1.1-367.g9e2ac70b.el9 ceph 21 M 2024-09-20T22:33:45.346 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-selinux x86_64 2:19.1.1-367.g9e2ac70b.el9 ceph 25 k 2024-09-20T22:33:45.346 INFO:teuthology.orchestra.run.smithi154.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-20T22:33:45.346 INFO:teuthology.orchestra.run.smithi154.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-20T22:33:45.346 INFO:teuthology.orchestra.run.smithi154.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-20T22:33:45.346 INFO:teuthology.orchestra.run.smithi154.stdout: libcephfs2 x86_64 2:19.1.1-367.g9e2ac70b.el9 ceph 752 k 2024-09-20T22:33:45.346 INFO:teuthology.orchestra.run.smithi154.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-20T22:33:45.346 INFO:teuthology.orchestra.run.smithi154.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-20T22:33:45.346 INFO:teuthology.orchestra.run.smithi154.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-20T22:33:45.346 INFO:teuthology.orchestra.run.smithi154.stdout: libradosstriper1 x86_64 2:19.1.1-367.g9e2ac70b.el9 ceph 506 k 2024-09-20T22:33:45.347 INFO:teuthology.orchestra.run.smithi154.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-20T22:33:45.347 INFO:teuthology.orchestra.run.smithi154.stdout: librgw2 x86_64 2:19.1.1-367.g9e2ac70b.el9 ceph 5.3 M 2024-09-20T22:33:45.347 INFO:teuthology.orchestra.run.smithi154.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-20T22:33:45.347 INFO:teuthology.orchestra.run.smithi154.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-20T22:33:45.347 INFO:teuthology.orchestra.run.smithi154.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-20T22:33:45.347 INFO:teuthology.orchestra.run.smithi154.stdout: python3-ceph-argparse x86_64 2:19.1.1-367.g9e2ac70b.el9 ceph 45 k 2024-09-20T22:33:45.347 INFO:teuthology.orchestra.run.smithi154.stdout: python3-ceph-common x86_64 2:19.1.1-367.g9e2ac70b.el9 ceph 136 k 2024-09-20T22:33:45.347 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cephfs x86_64 2:19.1.1-367.g9e2ac70b.el9 ceph 161 k 2024-09-20T22:33:45.347 INFO:teuthology.orchestra.run.smithi154.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-20T22:33:45.347 INFO:teuthology.orchestra.run.smithi154.stdout: python3-rados x86_64 2:19.1.1-367.g9e2ac70b.el9 ceph 320 k 2024-09-20T22:33:45.347 INFO:teuthology.orchestra.run.smithi154.stdout: python3-rbd x86_64 2:19.1.1-367.g9e2ac70b.el9 ceph 301 k 2024-09-20T22:33:45.347 INFO:teuthology.orchestra.run.smithi154.stdout: python3-rgw x86_64 2:19.1.1-367.g9e2ac70b.el9 ceph 99 k 2024-09-20T22:33:45.348 INFO:teuthology.orchestra.run.smithi154.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-20T22:33:45.348 INFO:teuthology.orchestra.run.smithi154.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-20T22:33:45.348 INFO:teuthology.orchestra.run.smithi154.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-20T22:33:45.348 INFO:teuthology.orchestra.run.smithi154.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-20T22:33:45.348 INFO:teuthology.orchestra.run.smithi154.stdout:Installing weak dependencies: 2024-09-20T22:33:45.348 INFO:teuthology.orchestra.run.smithi154.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-20T22:33:45.348 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-20T22:33:45.348 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2024-09-20T22:33:45.348 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-09-20T22:33:45.348 INFO:teuthology.orchestra.run.smithi154.stdout:Install 30 Packages 2024-09-20T22:33:45.348 INFO:teuthology.orchestra.run.smithi154.stdout:Upgrade 2 Packages 2024-09-20T22:33:45.348 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-20T22:33:45.349 INFO:teuthology.orchestra.run.smithi154.stdout:Total download size: 60 M 2024-09-20T22:33:45.349 INFO:teuthology.orchestra.run.smithi154.stdout:Downloading Packages: 2024-09-20T22:33:45.974 INFO:teuthology.orchestra.run.smithi154.stdout:(1/32): ceph-base-19.1.1-367.g9e2ac70b.el9.x86_ 9.9 MB/s | 5.4 MB 00:00 2024-09-20T22:33:46.033 INFO:teuthology.orchestra.run.smithi154.stdout:(2/32): ceph-selinux-19.1.1-367.g9e2ac70b.el9.x 432 kB/s | 25 kB 00:00 2024-09-20T22:33:46.125 INFO:teuthology.orchestra.run.smithi154.stdout:(3/32): libcephfs2-19.1.1-367.g9e2ac70b.el9.x86 8.0 MB/s | 752 kB 00:00 2024-09-20T22:33:46.242 INFO:teuthology.orchestra.run.smithi154.stdout:(4/32): ceph-radosgw-19.1.1-367.g9e2ac70b.el9.x 13 MB/s | 11 MB 00:00 2024-09-20T22:33:46.275 INFO:teuthology.orchestra.run.smithi154.stdout:(5/32): libradosstriper1-19.1.1-367.g9e2ac70b.e 3.3 MB/s | 506 kB 00:00 2024-09-20T22:33:46.325 INFO:teuthology.orchestra.run.smithi154.stdout:(6/32): python3-ceph-argparse-19.1.1-367.g9e2ac 907 kB/s | 45 kB 00:00 2024-09-20T22:33:46.392 INFO:teuthology.orchestra.run.smithi154.stdout:(7/32): python3-ceph-common-19.1.1-367.g9e2ac70 2.0 MB/s | 136 kB 00:00 2024-09-20T22:33:46.460 INFO:teuthology.orchestra.run.smithi154.stdout:(8/32): python3-cephfs-19.1.1-367.g9e2ac70b.el9 2.4 MB/s | 161 kB 00:00 2024-09-20T22:33:46.535 INFO:teuthology.orchestra.run.smithi154.stdout:(9/32): librgw2-19.1.1-367.g9e2ac70b.el9.x86_64 18 MB/s | 5.3 MB 00:00 2024-09-20T22:33:46.569 INFO:teuthology.orchestra.run.smithi154.stdout:(10/32): python3-rados-19.1.1-367.g9e2ac70b.el9 2.9 MB/s | 320 kB 00:00 2024-09-20T22:33:46.594 INFO:teuthology.orchestra.run.smithi154.stdout:(11/32): python3-rbd-19.1.1-367.g9e2ac70b.el9.x 5.0 MB/s | 301 kB 00:00 2024-09-20T22:33:46.619 INFO:teuthology.orchestra.run.smithi154.stdout:(12/32): python3-rgw-19.1.1-367.g9e2ac70b.el9.x 1.9 MB/s | 99 kB 00:00 2024-09-20T22:33:46.684 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-09-20T22:33:46.687 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-09-20T22:33:46.687 INFO:teuthology.orchestra.run.smithi084.stdout: Package Arch Version Repository Size 2024-09-20T22:33:46.687 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-09-20T22:33:46.687 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2024-09-20T22:33:46.687 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-radosgw x86_64 2:19.1.1-367.g9e2ac70b.el9 ceph 11 M 2024-09-20T22:33:46.687 INFO:teuthology.orchestra.run.smithi084.stdout:Upgrading: 2024-09-20T22:33:46.687 INFO:teuthology.orchestra.run.smithi084.stdout: librados2 x86_64 2:19.1.1-367.g9e2ac70b.el9 ceph 3.4 M 2024-09-20T22:33:46.687 INFO:teuthology.orchestra.run.smithi084.stdout: librbd1 x86_64 2:19.1.1-367.g9e2ac70b.el9 ceph 3.2 M 2024-09-20T22:33:46.687 INFO:teuthology.orchestra.run.smithi084.stdout:Installing dependencies: 2024-09-20T22:33:46.687 INFO:teuthology.orchestra.run.smithi084.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-20T22:33:46.687 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-base x86_64 2:19.1.1-367.g9e2ac70b.el9 ceph 5.4 M 2024-09-20T22:33:46.688 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-common x86_64 2:19.1.1-367.g9e2ac70b.el9 ceph 21 M 2024-09-20T22:33:46.688 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-selinux x86_64 2:19.1.1-367.g9e2ac70b.el9 ceph 25 k 2024-09-20T22:33:46.688 INFO:teuthology.orchestra.run.smithi084.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-20T22:33:46.688 INFO:teuthology.orchestra.run.smithi084.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-20T22:33:46.688 INFO:teuthology.orchestra.run.smithi084.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-20T22:33:46.688 INFO:teuthology.orchestra.run.smithi084.stdout: libcephfs2 x86_64 2:19.1.1-367.g9e2ac70b.el9 ceph 752 k 2024-09-20T22:33:46.688 INFO:teuthology.orchestra.run.smithi084.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-20T22:33:46.688 INFO:teuthology.orchestra.run.smithi084.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-20T22:33:46.688 INFO:teuthology.orchestra.run.smithi084.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-20T22:33:46.688 INFO:teuthology.orchestra.run.smithi084.stdout: libradosstriper1 x86_64 2:19.1.1-367.g9e2ac70b.el9 ceph 506 k 2024-09-20T22:33:46.688 INFO:teuthology.orchestra.run.smithi084.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-20T22:33:46.688 INFO:teuthology.orchestra.run.smithi084.stdout: librgw2 x86_64 2:19.1.1-367.g9e2ac70b.el9 ceph 5.3 M 2024-09-20T22:33:46.688 INFO:teuthology.orchestra.run.smithi084.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-20T22:33:46.689 INFO:teuthology.orchestra.run.smithi084.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-20T22:33:46.689 INFO:teuthology.orchestra.run.smithi084.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-20T22:33:46.689 INFO:teuthology.orchestra.run.smithi084.stdout: python3-ceph-argparse x86_64 2:19.1.1-367.g9e2ac70b.el9 ceph 45 k 2024-09-20T22:33:46.689 INFO:teuthology.orchestra.run.smithi084.stdout: python3-ceph-common x86_64 2:19.1.1-367.g9e2ac70b.el9 ceph 136 k 2024-09-20T22:33:46.689 INFO:teuthology.orchestra.run.smithi084.stdout: python3-cephfs x86_64 2:19.1.1-367.g9e2ac70b.el9 ceph 161 k 2024-09-20T22:33:46.689 INFO:teuthology.orchestra.run.smithi084.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-20T22:33:46.689 INFO:teuthology.orchestra.run.smithi084.stdout: python3-rados x86_64 2:19.1.1-367.g9e2ac70b.el9 ceph 320 k 2024-09-20T22:33:46.689 INFO:teuthology.orchestra.run.smithi084.stdout: python3-rbd x86_64 2:19.1.1-367.g9e2ac70b.el9 ceph 301 k 2024-09-20T22:33:46.689 INFO:teuthology.orchestra.run.smithi084.stdout: python3-rgw x86_64 2:19.1.1-367.g9e2ac70b.el9 ceph 99 k 2024-09-20T22:33:46.689 INFO:teuthology.orchestra.run.smithi084.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-20T22:33:46.689 INFO:teuthology.orchestra.run.smithi084.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-20T22:33:46.689 INFO:teuthology.orchestra.run.smithi084.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-20T22:33:46.689 INFO:teuthology.orchestra.run.smithi084.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-20T22:33:46.690 INFO:teuthology.orchestra.run.smithi084.stdout:Installing weak dependencies: 2024-09-20T22:33:46.690 INFO:teuthology.orchestra.run.smithi084.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-20T22:33:46.690 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-20T22:33:46.690 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2024-09-20T22:33:46.690 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-09-20T22:33:46.690 INFO:teuthology.orchestra.run.smithi084.stdout:Install 30 Packages 2024-09-20T22:33:46.690 INFO:teuthology.orchestra.run.smithi084.stdout:Upgrade 2 Packages 2024-09-20T22:33:46.690 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-20T22:33:46.690 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 60 M 2024-09-20T22:33:46.690 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2024-09-20T22:33:46.820 INFO:teuthology.orchestra.run.smithi154.stdout:(13/32): ceph-common-19.1.1-367.g9e2ac70b.el9.x 15 MB/s | 21 MB 00:01 2024-09-20T22:33:46.970 INFO:teuthology.orchestra.run.smithi154.stdout:(14/32): boost-program-options-1.75.0-8.el9.x86 285 kB/s | 107 kB 00:00 2024-09-20T22:33:46.987 INFO:teuthology.orchestra.run.smithi154.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 272 kB/s | 45 kB 00:00 2024-09-20T22:33:47.012 INFO:teuthology.orchestra.run.smithi154.stdout:(16/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 408 kB/s | 160 kB 00:00 2024-09-20T22:33:47.062 INFO:teuthology.orchestra.run.smithi154.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 846 kB/s | 42 kB 00:00 2024-09-20T22:33:47.104 INFO:teuthology.orchestra.run.smithi154.stdout:(18/32): librdkafka-1.6.1-102.el9.x86_64.rpm 4.8 MB/s | 662 kB 00:00 2024-09-20T22:33:47.138 INFO:teuthology.orchestra.run.smithi154.stdout:(19/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.9 MB/s | 292 kB 00:00 2024-09-20T22:33:47.163 INFO:teuthology.orchestra.run.smithi154.stdout:(20/32): qatlib-service-24.02.0-1.el9.x86_64.rp 613 kB/s | 36 kB 00:00 2024-09-20T22:33:47.188 INFO:teuthology.orchestra.run.smithi154.stdout:(21/32): qatlib-24.02.0-1.el9.x86_64.rpm 1.7 MB/s | 221 kB 00:00 2024-09-20T22:33:47.214 INFO:teuthology.orchestra.run.smithi154.stdout:(22/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 625 kB/s | 47 kB 00:00 2024-09-20T22:33:47.216 INFO:teuthology.orchestra.run.smithi084.stdout:(1/32): ceph-base-19.1.1-367.g9e2ac70b.el9.x86_ 12 MB/s | 5.4 MB 00:00 2024-09-20T22:33:47.258 INFO:teuthology.orchestra.run.smithi084.stdout:(2/32): ceph-selinux-19.1.1-367.g9e2ac70b.el9.x 607 kB/s | 25 kB 00:00 2024-09-20T22:33:47.272 INFO:teuthology.orchestra.run.smithi154.stdout:(23/32): gperftools-libs-2.9.1-3.el9.x86_64.rpm 2.8 MB/s | 308 kB 00:00 2024-09-20T22:33:47.297 INFO:teuthology.orchestra.run.smithi154.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 2.1 MB/s | 49 kB 00:00 2024-09-20T22:33:47.323 INFO:teuthology.orchestra.run.smithi154.stdout:(25/32): libarrow-doc-9.0.0-11.el9.noarch.rpm 276 kB/s | 26 kB 00:00 2024-09-20T22:33:47.348 INFO:teuthology.orchestra.run.smithi154.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-09-20T22:33:47.382 INFO:teuthology.orchestra.run.smithi154.stdout:(27/32): re2-20211101-20.el9.x86_64.rpm 5.6 MB/s | 191 kB 00:00 2024-09-20T22:33:47.391 INFO:teuthology.orchestra.run.smithi084.stdout:(3/32): libcephfs2-19.1.1-367.g9e2ac70b.el9.x86 5.5 MB/s | 752 kB 00:00 2024-09-20T22:33:47.440 INFO:teuthology.orchestra.run.smithi154.stdout:(28/32): parquet-libs-9.0.0-11.el9.x86_64.rpm 7.0 MB/s | 839 kB 00:00 2024-09-20T22:33:47.483 INFO:teuthology.orchestra.run.smithi084.stdout:(4/32): libradosstriper1-19.1.1-367.g9e2ac70b.e 5.4 MB/s | 506 kB 00:00 2024-09-20T22:33:47.491 INFO:teuthology.orchestra.run.smithi154.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 15 MB/s | 1.6 MB 00:00 2024-09-20T22:33:47.566 INFO:teuthology.orchestra.run.smithi154.stdout:(30/32): libarrow-9.0.0-11.el9.x86_64.rpm 12 MB/s | 4.4 MB 00:00 2024-09-20T22:33:47.667 INFO:teuthology.orchestra.run.smithi084.stdout:(5/32): ceph-radosgw-19.1.1-367.g9e2ac70b.el9.x 12 MB/s | 11 MB 00:00 2024-09-20T22:33:47.709 INFO:teuthology.orchestra.run.smithi084.stdout:(6/32): python3-ceph-argparse-19.1.1-367.g9e2ac 1.1 MB/s | 45 kB 00:00 2024-09-20T22:33:47.817 INFO:teuthology.orchestra.run.smithi084.stdout:(7/32): librgw2-19.1.1-367.g9e2ac70b.el9.x86_64 16 MB/s | 5.3 MB 00:00 2024-09-20T22:33:47.843 INFO:teuthology.orchestra.run.smithi084.stdout:(8/32): python3-ceph-common-19.1.1-367.g9e2ac70 1.0 MB/s | 136 kB 00:00 2024-09-20T22:33:47.866 INFO:teuthology.orchestra.run.smithi154.stdout:(31/32): librbd1-19.1.1-367.g9e2ac70b.el9.x86_6 8.4 MB/s | 3.2 MB 00:00 2024-09-20T22:33:47.995 INFO:teuthology.orchestra.run.smithi154.stdout:(32/32): librados2-19.1.1-367.g9e2ac70b.el9.x86 6.2 MB/s | 3.4 MB 00:00 2024-09-20T22:33:48.001 INFO:teuthology.orchestra.run.smithi154.stdout:-------------------------------------------------------------------------------- 2024-09-20T22:33:48.001 INFO:teuthology.orchestra.run.smithi154.stdout:Total 23 MB/s | 60 MB 00:02 2024-09-20T22:33:48.043 INFO:teuthology.orchestra.run.smithi084.stdout:(9/32): ceph-common-19.1.1-367.g9e2ac70b.el9.x8 17 MB/s | 21 MB 00:01 2024-09-20T22:33:48.068 INFO:teuthology.orchestra.run.smithi084.stdout:(10/32): python3-cephfs-19.1.1-367.g9e2ac70b.el 645 kB/s | 161 kB 00:00 2024-09-20T22:33:48.093 INFO:teuthology.orchestra.run.smithi084.stdout:(11/32): python3-rados-19.1.1-367.g9e2ac70b.el9 1.2 MB/s | 320 kB 00:00 2024-09-20T22:33:48.112 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2024-09-20T22:33:48.119 INFO:teuthology.orchestra.run.smithi084.stdout:(12/32): python3-rbd-19.1.1-367.g9e2ac70b.el9.x 3.9 MB/s | 301 kB 00:00 2024-09-20T22:33:48.144 INFO:teuthology.orchestra.run.smithi084.stdout:(13/32): python3-rgw-19.1.1-367.g9e2ac70b.el9.x 1.3 MB/s | 99 kB 00:00 2024-09-20T22:33:48.148 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2024-09-20T22:33:48.148 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2024-09-20T22:33:48.244 INFO:teuthology.orchestra.run.smithi084.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 452 kB/s | 45 kB 00:00 2024-09-20T22:33:48.261 INFO:teuthology.orchestra.run.smithi084.stdout:(15/32): boost-program-options-1.75.0-8.el9.x86 639 kB/s | 107 kB 00:00 2024-09-20T22:33:48.286 INFO:teuthology.orchestra.run.smithi084.stdout:(16/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 958 kB/s | 160 kB 00:00 2024-09-20T22:33:48.328 INFO:teuthology.orchestra.run.smithi084.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 1.0 MB/s | 42 kB 00:00 2024-09-20T22:33:48.353 INFO:teuthology.orchestra.run.smithi084.stdout:(18/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.1 MB/s | 292 kB 00:00 2024-09-20T22:33:48.387 INFO:teuthology.orchestra.run.smithi084.stdout:(19/32): librdkafka-1.6.1-102.el9.x86_64.rpm 4.5 MB/s | 662 kB 00:00 2024-09-20T22:33:48.412 INFO:teuthology.orchestra.run.smithi084.stdout:(20/32): qatlib-service-24.02.0-1.el9.x86_64.rp 613 kB/s | 36 kB 00:00 2024-09-20T22:33:48.438 INFO:teuthology.orchestra.run.smithi084.stdout:(21/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 947 kB/s | 47 kB 00:00 2024-09-20T22:33:48.463 INFO:teuthology.orchestra.run.smithi084.stdout:(22/32): qatlib-24.02.0-1.el9.x86_64.rpm 1.6 MB/s | 221 kB 00:00 2024-09-20T22:33:48.513 INFO:teuthology.orchestra.run.smithi084.stdout:(23/32): gperftools-libs-2.9.1-3.el9.x86_64.rpm 3.0 MB/s | 308 kB 00:00 2024-09-20T22:33:48.539 INFO:teuthology.orchestra.run.smithi084.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 2.1 MB/s | 49 kB 00:00 2024-09-20T22:33:48.564 INFO:teuthology.orchestra.run.smithi084.stdout:(25/32): libarrow-doc-9.0.0-11.el9.noarch.rpm 304 kB/s | 26 kB 00:00 2024-09-20T22:33:48.589 INFO:teuthology.orchestra.run.smithi084.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-09-20T22:33:48.623 INFO:teuthology.orchestra.run.smithi084.stdout:(27/32): re2-20211101-20.el9.x86_64.rpm 5.6 MB/s | 191 kB 00:00 2024-09-20T22:33:48.673 INFO:teuthology.orchestra.run.smithi084.stdout:(28/32): parquet-libs-9.0.0-11.el9.x86_64.rpm 7.5 MB/s | 839 kB 00:00 2024-09-20T22:33:48.685 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2024-09-20T22:33:48.685 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2024-09-20T22:33:48.715 INFO:teuthology.orchestra.run.smithi084.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 17 MB/s | 1.6 MB 00:00 2024-09-20T22:33:48.790 INFO:teuthology.orchestra.run.smithi084.stdout:(30/32): libarrow-9.0.0-11.el9.x86_64.rpm 13 MB/s | 4.4 MB 00:00 2024-09-20T22:33:48.899 INFO:teuthology.orchestra.run.smithi084.stdout:(31/32): librados2-19.1.1-367.g9e2ac70b.el9.x86 15 MB/s | 3.4 MB 00:00 2024-09-20T22:33:48.965 INFO:teuthology.orchestra.run.smithi084.stdout:(32/32): librbd1-19.1.1-367.g9e2ac70b.el9.x86_6 13 MB/s | 3.2 MB 00:00 2024-09-20T22:33:48.971 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2024-09-20T22:33:48.971 INFO:teuthology.orchestra.run.smithi084.stdout:Total 26 MB/s | 60 MB 00:02 2024-09-20T22:33:49.076 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2024-09-20T22:33:49.113 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2024-09-20T22:33:49.113 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2024-09-20T22:33:49.631 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2024-09-20T22:33:49.632 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2024-09-20T22:33:50.131 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2024-09-20T22:33:50.217 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-09-20T22:33:50.246 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-09-20T22:33:50.272 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-09-20T22:33:50.294 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-09-20T22:33:50.316 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-09-20T22:33:50.556 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-09-20T22:33:50.609 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-09-20T22:33:51.018 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-09-20T22:33:51.074 INFO:teuthology.orchestra.run.smithi154.stdout: Upgrading : librados2-2:19.1.1-367.g9e2ac70b.el9.x86_64 7/34 2024-09-20T22:33:51.112 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2024-09-20T22:33:51.167 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: librados2-2:19.1.1-367.g9e2ac70b.el9.x86_64 7/34 2024-09-20T22:33:51.200 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-09-20T22:33:51.237 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-09-20T22:33:51.254 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-rados-2:19.1.1-367.g9e2ac70b.el9.x86_64 8/34 2024-09-20T22:33:51.270 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-09-20T22:33:51.283 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : libcephfs2-2:19.1.1-367.g9e2ac70b.el9.x86_64 9/34 2024-09-20T22:33:51.293 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-09-20T22:33:51.312 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-09-20T22:33:51.329 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: libcephfs2-2:19.1.1-367.g9e2ac70b.el9.x86_64 9/34 2024-09-20T22:33:51.358 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-09-20T22:33:51.426 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-ceph-argparse-2:19.1.1-367.g9e2ac70b.el9.x 11/34 2024-09-20T22:33:51.478 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-cephfs-2:19.1.1-367.g9e2ac70b.el9.x86_64 12/34 2024-09-20T22:33:51.502 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : libradosstriper1-2:19.1.1-367.g9e2ac70b.el9.x86_64 13/34 2024-09-20T22:33:51.504 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-09-20T22:33:51.550 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: libradosstriper1-2:19.1.1-367.g9e2ac70b.el9.x86_64 13/34 2024-09-20T22:33:51.557 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-09-20T22:33:51.584 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-09-20T22:33:51.693 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/34 2024-09-20T22:33:51.726 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/34 2024-09-20T22:33:51.777 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-09-20T22:33:51.809 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 18/34 2024-09-20T22:33:51.838 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/34 2024-09-20T22:33:51.886 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-09-20T22:33:51.927 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-09-20T22:33:51.988 INFO:teuthology.orchestra.run.smithi084.stdout: Upgrading : librados2-2:19.1.1-367.g9e2ac70b.el9.x86_64 7/34 2024-09-20T22:33:52.073 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: librados2-2:19.1.1-367.g9e2ac70b.el9.x86_64 7/34 2024-09-20T22:33:52.169 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-rados-2:19.1.1-367.g9e2ac70b.el9.x86_64 8/34 2024-09-20T22:33:52.197 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-09-20T22:33:52.231 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : libcephfs2-2:19.1.1-367.g9e2ac70b.el9.x86_64 9/34 2024-09-20T22:33:52.241 INFO:teuthology.orchestra.run.smithi154.stdout: Upgrading : librbd1-2:19.1.1-367.g9e2ac70b.el9.x86_64 22/34 2024-09-20T22:33:52.279 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: libcephfs2-2:19.1.1-367.g9e2ac70b.el9.x86_64 9/34 2024-09-20T22:33:52.315 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-09-20T22:33:52.332 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: librbd1-2:19.1.1-367.g9e2ac70b.el9.x86_64 22/34 2024-09-20T22:33:52.358 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-ceph-argparse-2:19.1.1-367.g9e2ac70b.el9.x 11/34 2024-09-20T22:33:52.363 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-rbd-2:19.1.1-367.g9e2ac70b.el9.x86_64 23/34 2024-09-20T22:33:52.398 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-09-20T22:33:52.438 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-cephfs-2:19.1.1-367.g9e2ac70b.el9.x86_64 12/34 2024-09-20T22:33:52.466 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : libradosstriper1-2:19.1.1-367.g9e2ac70b.el9.x86_64 13/34 2024-09-20T22:33:52.531 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: libradosstriper1-2:19.1.1-367.g9e2ac70b.el9.x86_64 13/34 2024-09-20T22:33:52.565 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-09-20T22:33:52.684 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/34 2024-09-20T22:33:52.715 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/34 2024-09-20T22:33:52.765 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-09-20T22:33:52.797 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 18/34 2024-09-20T22:33:52.826 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/34 2024-09-20T22:33:52.859 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-09-20T22:33:52.879 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 25/34 2024-09-20T22:33:52.910 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : librgw2-2:19.1.1-367.g9e2ac70b.el9.x86_64 26/34 2024-09-20T22:33:52.965 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: librgw2-2:19.1.1-367.g9e2ac70b.el9.x86_64 26/34 2024-09-20T22:33:53.052 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-rgw-2:19.1.1-367.g9e2ac70b.el9.x86_64 27/34 2024-09-20T22:33:53.096 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-ceph-common-2:19.1.1-367.g9e2ac70b.el9.x86 28/34 2024-09-20T22:33:53.179 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-09-20T22:33:53.213 INFO:teuthology.orchestra.run.smithi084.stdout: Upgrading : librbd1-2:19.1.1-367.g9e2ac70b.el9.x86_64 22/34 2024-09-20T22:33:53.304 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: librbd1-2:19.1.1-367.g9e2ac70b.el9.x86_64 22/34 2024-09-20T22:33:53.344 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-rbd-2:19.1.1-367.g9e2ac70b.el9.x86_64 23/34 2024-09-20T22:33:53.387 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-09-20T22:33:53.878 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 25/34 2024-09-20T22:33:53.907 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : librgw2-2:19.1.1-367.g9e2ac70b.el9.x86_64 26/34 2024-09-20T22:33:53.987 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: librgw2-2:19.1.1-367.g9e2ac70b.el9.x86_64 26/34 2024-09-20T22:33:54.065 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-rgw-2:19.1.1-367.g9e2ac70b.el9.x86_64 27/34 2024-09-20T22:33:54.093 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-ceph-common-2:19.1.1-367.g9e2ac70b.el9.x86 28/34 2024-09-20T22:33:55.319 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-common-2:19.1.1-367.g9e2ac70b.el9.x86_64 29/34 2024-09-20T22:33:55.367 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-common-2:19.1.1-367.g9e2ac70b.el9.x86_64 29/34 2024-09-20T22:33:55.935 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-common-2:19.1.1-367.g9e2ac70b.el9.x86_64 29/34 2024-09-20T22:33:55.970 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-base-2:19.1.1-367.g9e2ac70b.el9.x86_64 30/34 2024-09-20T22:33:56.052 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-base-2:19.1.1-367.g9e2ac70b.el9.x86_64 30/34 2024-09-20T22:33:56.052 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-20T22:33:56.052 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-20T22:33:56.053 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-20T22:33:56.080 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-selinux-2:19.1.1-367.g9e2ac70b.el9.x86_64 31/34 2024-09-20T22:33:56.359 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-common-2:19.1.1-367.g9e2ac70b.el9.x86_64 29/34 2024-09-20T22:33:56.506 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-common-2:19.1.1-367.g9e2ac70b.el9.x86_64 29/34 2024-09-20T22:33:57.073 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-common-2:19.1.1-367.g9e2ac70b.el9.x86_64 29/34 2024-09-20T22:33:57.109 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-base-2:19.1.1-367.g9e2ac70b.el9.x86_64 30/34 2024-09-20T22:33:57.199 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-base-2:19.1.1-367.g9e2ac70b.el9.x86_64 30/34 2024-09-20T22:33:57.199 INFO:teuthology.orchestra.run.smithi084.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-20T22:33:57.199 INFO:teuthology.orchestra.run.smithi084.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-20T22:33:57.200 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-20T22:33:57.235 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-selinux-2:19.1.1-367.g9e2ac70b.el9.x86_64 31/34 2024-09-20T22:34:06.310 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-selinux-2:19.1.1-367.g9e2ac70b.el9.x86_64 31/34 2024-09-20T22:34:06.310 INFO:teuthology.orchestra.run.smithi154.stdout:skipping the directory /sys 2024-09-20T22:34:06.310 INFO:teuthology.orchestra.run.smithi154.stdout:skipping the directory /proc 2024-09-20T22:34:06.310 INFO:teuthology.orchestra.run.smithi154.stdout:skipping the directory /mnt 2024-09-20T22:34:06.310 INFO:teuthology.orchestra.run.smithi154.stdout:skipping the directory /var/tmp 2024-09-20T22:34:06.310 INFO:teuthology.orchestra.run.smithi154.stdout:skipping the directory /home 2024-09-20T22:34:06.310 INFO:teuthology.orchestra.run.smithi154.stdout:skipping the directory /root 2024-09-20T22:34:06.310 INFO:teuthology.orchestra.run.smithi154.stdout:skipping the directory /tmp 2024-09-20T22:34:06.310 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-20T22:34:07.095 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-selinux-2:19.1.1-367.g9e2ac70b.el9.x86_64 31/34 2024-09-20T22:34:07.095 INFO:teuthology.orchestra.run.smithi084.stdout:skipping the directory /sys 2024-09-20T22:34:07.095 INFO:teuthology.orchestra.run.smithi084.stdout:skipping the directory /proc 2024-09-20T22:34:07.096 INFO:teuthology.orchestra.run.smithi084.stdout:skipping the directory /mnt 2024-09-20T22:34:07.096 INFO:teuthology.orchestra.run.smithi084.stdout:skipping the directory /var/tmp 2024-09-20T22:34:07.096 INFO:teuthology.orchestra.run.smithi084.stdout:skipping the directory /home 2024-09-20T22:34:07.096 INFO:teuthology.orchestra.run.smithi084.stdout:skipping the directory /root 2024-09-20T22:34:07.096 INFO:teuthology.orchestra.run.smithi084.stdout:skipping the directory /tmp 2024-09-20T22:34:07.096 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-20T22:34:07.349 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-radosgw-2:19.1.1-367.g9e2ac70b.el9.x86_64 32/34 2024-09-20T22:34:07.376 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-radosgw-2:19.1.1-367.g9e2ac70b.el9.x86_64 32/34 2024-09-20T22:34:07.376 INFO:teuthology.orchestra.run.smithi154.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-20T22:34:07.376 INFO:teuthology.orchestra.run.smithi154.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-20T22:34:07.376 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-20T22:34:07.376 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-20T22:34:07.376 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-20T22:34:07.378 INFO:teuthology.orchestra.run.smithi154.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-09-20T22:34:07.469 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-09-20T22:34:07.469 INFO:teuthology.orchestra.run.smithi154.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-09-20T22:34:08.092 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-radosgw-2:19.1.1-367.g9e2ac70b.el9.x86_64 32/34 2024-09-20T22:34:08.117 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-radosgw-2:19.1.1-367.g9e2ac70b.el9.x86_64 32/34 2024-09-20T22:34:08.117 INFO:teuthology.orchestra.run.smithi084.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-20T22:34:08.117 INFO:teuthology.orchestra.run.smithi084.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-20T22:34:08.117 INFO:teuthology.orchestra.run.smithi084.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-20T22:34:08.118 INFO:teuthology.orchestra.run.smithi084.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-20T22:34:08.118 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-20T22:34:08.119 INFO:teuthology.orchestra.run.smithi084.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-09-20T22:34:08.186 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-09-20T22:34:08.186 INFO:teuthology.orchestra.run.smithi084.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-09-20T22:34:08.636 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-09-20T22:34:08.636 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-base-2:19.1.1-367.g9e2ac70b.el9.x86_64 1/34 2024-09-20T22:34:08.637 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-common-2:19.1.1-367.g9e2ac70b.el9.x86_64 2/34 2024-09-20T22:34:08.637 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-radosgw-2:19.1.1-367.g9e2ac70b.el9.x86_64 3/34 2024-09-20T22:34:08.637 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-selinux-2:19.1.1-367.g9e2ac70b.el9.x86_64 4/34 2024-09-20T22:34:08.637 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libcephfs2-2:19.1.1-367.g9e2ac70b.el9.x86_64 5/34 2024-09-20T22:34:08.637 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libradosstriper1-2:19.1.1-367.g9e2ac70b.el9.x86_64 6/34 2024-09-20T22:34:08.637 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : librgw2-2:19.1.1-367.g9e2ac70b.el9.x86_64 7/34 2024-09-20T22:34:08.637 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-ceph-argparse-2:19.1.1-367.g9e2ac70b.el9.x 8/34 2024-09-20T22:34:08.637 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-ceph-common-2:19.1.1-367.g9e2ac70b.el9.x86 9/34 2024-09-20T22:34:08.637 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-cephfs-2:19.1.1-367.g9e2ac70b.el9.x86_64 10/34 2024-09-20T22:34:08.637 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-rados-2:19.1.1-367.g9e2ac70b.el9.x86_64 11/34 2024-09-20T22:34:08.637 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-rbd-2:19.1.1-367.g9e2ac70b.el9.x86_64 12/34 2024-09-20T22:34:08.638 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-rgw-2:19.1.1-367.g9e2ac70b.el9.x86_64 13/34 2024-09-20T22:34:08.638 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-09-20T22:34:08.638 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-09-20T22:34:08.638 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-09-20T22:34:08.638 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-09-20T22:34:08.638 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-09-20T22:34:08.638 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-09-20T22:34:08.638 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-09-20T22:34:08.638 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-09-20T22:34:08.638 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-09-20T22:34:08.638 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 23/34 2024-09-20T22:34:08.638 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 24/34 2024-09-20T22:34:08.638 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 25/34 2024-09-20T22:34:08.639 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-09-20T22:34:08.639 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-09-20T22:34:08.640 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 28/34 2024-09-20T22:34:08.640 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : re2-1:20211101-20.el9.x86_64 29/34 2024-09-20T22:34:08.640 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-09-20T22:34:08.640 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : librados2-2:19.1.1-367.g9e2ac70b.el9.x86_64 31/34 2024-09-20T22:34:08.640 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-09-20T22:34:08.640 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : librbd1-2:19.1.1-367.g9e2ac70b.el9.x86_64 33/34 2024-09-20T22:34:09.345 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-09-20T22:34:09.345 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-base-2:19.1.1-367.g9e2ac70b.el9.x86_64 1/34 2024-09-20T22:34:09.345 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-common-2:19.1.1-367.g9e2ac70b.el9.x86_64 2/34 2024-09-20T22:34:09.345 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-radosgw-2:19.1.1-367.g9e2ac70b.el9.x86_64 3/34 2024-09-20T22:34:09.345 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-selinux-2:19.1.1-367.g9e2ac70b.el9.x86_64 4/34 2024-09-20T22:34:09.346 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : libcephfs2-2:19.1.1-367.g9e2ac70b.el9.x86_64 5/34 2024-09-20T22:34:09.346 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : libradosstriper1-2:19.1.1-367.g9e2ac70b.el9.x86_64 6/34 2024-09-20T22:34:09.346 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : librgw2-2:19.1.1-367.g9e2ac70b.el9.x86_64 7/34 2024-09-20T22:34:09.346 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-ceph-argparse-2:19.1.1-367.g9e2ac70b.el9.x 8/34 2024-09-20T22:34:09.346 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-ceph-common-2:19.1.1-367.g9e2ac70b.el9.x86 9/34 2024-09-20T22:34:09.346 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-cephfs-2:19.1.1-367.g9e2ac70b.el9.x86_64 10/34 2024-09-20T22:34:09.346 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-rados-2:19.1.1-367.g9e2ac70b.el9.x86_64 11/34 2024-09-20T22:34:09.347 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-rbd-2:19.1.1-367.g9e2ac70b.el9.x86_64 12/34 2024-09-20T22:34:09.347 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-rgw-2:19.1.1-367.g9e2ac70b.el9.x86_64 13/34 2024-09-20T22:34:09.347 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-09-20T22:34:09.347 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-09-20T22:34:09.347 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-09-20T22:34:09.347 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-09-20T22:34:09.347 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-09-20T22:34:09.347 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-09-20T22:34:09.348 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-09-20T22:34:09.348 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-09-20T22:34:09.348 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-09-20T22:34:09.348 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 23/34 2024-09-20T22:34:09.348 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 24/34 2024-09-20T22:34:09.348 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 25/34 2024-09-20T22:34:09.350 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-09-20T22:34:09.350 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-09-20T22:34:09.350 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 28/34 2024-09-20T22:34:09.350 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : re2-1:20211101-20.el9.x86_64 29/34 2024-09-20T22:34:09.351 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-09-20T22:34:09.351 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : librados2-2:19.1.1-367.g9e2ac70b.el9.x86_64 31/34 2024-09-20T22:34:09.351 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-09-20T22:34:09.351 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : librbd1-2:19.1.1-367.g9e2ac70b.el9.x86_64 33/34 2024-09-20T22:34:09.553 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-09-20T22:34:09.553 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-20T22:34:09.554 INFO:teuthology.orchestra.run.smithi154.stdout:Upgraded: 2024-09-20T22:34:09.554 INFO:teuthology.orchestra.run.smithi154.stdout: librados2-2:19.1.1-367.g9e2ac70b.el9.x86_64 2024-09-20T22:34:09.554 INFO:teuthology.orchestra.run.smithi154.stdout: librbd1-2:19.1.1-367.g9e2ac70b.el9.x86_64 2024-09-20T22:34:09.554 INFO:teuthology.orchestra.run.smithi154.stdout:Installed: 2024-09-20T22:34:09.554 INFO:teuthology.orchestra.run.smithi154.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-20T22:34:09.554 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-base-2:19.1.1-367.g9e2ac70b.el9.x86_64 2024-09-20T22:34:09.554 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-common-2:19.1.1-367.g9e2ac70b.el9.x86_64 2024-09-20T22:34:09.554 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-radosgw-2:19.1.1-367.g9e2ac70b.el9.x86_64 2024-09-20T22:34:09.554 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-selinux-2:19.1.1-367.g9e2ac70b.el9.x86_64 2024-09-20T22:34:09.554 INFO:teuthology.orchestra.run.smithi154.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-20T22:34:09.554 INFO:teuthology.orchestra.run.smithi154.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-20T22:34:09.554 INFO:teuthology.orchestra.run.smithi154.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-20T22:34:09.554 INFO:teuthology.orchestra.run.smithi154.stdout: libcephfs2-2:19.1.1-367.g9e2ac70b.el9.x86_64 2024-09-20T22:34:09.555 INFO:teuthology.orchestra.run.smithi154.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-20T22:34:09.555 INFO:teuthology.orchestra.run.smithi154.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-20T22:34:09.555 INFO:teuthology.orchestra.run.smithi154.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-20T22:34:09.555 INFO:teuthology.orchestra.run.smithi154.stdout: libradosstriper1-2:19.1.1-367.g9e2ac70b.el9.x86_64 2024-09-20T22:34:09.555 INFO:teuthology.orchestra.run.smithi154.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-20T22:34:09.555 INFO:teuthology.orchestra.run.smithi154.stdout: librgw2-2:19.1.1-367.g9e2ac70b.el9.x86_64 2024-09-20T22:34:09.555 INFO:teuthology.orchestra.run.smithi154.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-20T22:34:09.555 INFO:teuthology.orchestra.run.smithi154.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-20T22:34:09.555 INFO:teuthology.orchestra.run.smithi154.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-20T22:34:09.555 INFO:teuthology.orchestra.run.smithi154.stdout: python3-ceph-argparse-2:19.1.1-367.g9e2ac70b.el9.x86_64 2024-09-20T22:34:09.555 INFO:teuthology.orchestra.run.smithi154.stdout: python3-ceph-common-2:19.1.1-367.g9e2ac70b.el9.x86_64 2024-09-20T22:34:09.555 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cephfs-2:19.1.1-367.g9e2ac70b.el9.x86_64 2024-09-20T22:34:09.555 INFO:teuthology.orchestra.run.smithi154.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-20T22:34:09.555 INFO:teuthology.orchestra.run.smithi154.stdout: python3-rados-2:19.1.1-367.g9e2ac70b.el9.x86_64 2024-09-20T22:34:09.556 INFO:teuthology.orchestra.run.smithi154.stdout: python3-rbd-2:19.1.1-367.g9e2ac70b.el9.x86_64 2024-09-20T22:34:09.556 INFO:teuthology.orchestra.run.smithi154.stdout: python3-rgw-2:19.1.1-367.g9e2ac70b.el9.x86_64 2024-09-20T22:34:09.556 INFO:teuthology.orchestra.run.smithi154.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-20T22:34:09.556 INFO:teuthology.orchestra.run.smithi154.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-20T22:34:09.556 INFO:teuthology.orchestra.run.smithi154.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-20T22:34:09.556 INFO:teuthology.orchestra.run.smithi154.stdout: re2-1:20211101-20.el9.x86_64 2024-09-20T22:34:09.556 INFO:teuthology.orchestra.run.smithi154.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-20T22:34:09.556 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-20T22:34:09.556 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2024-09-20T22:34:09.928 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install ceph-test 2024-09-20T22:34:10.271 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-09-20T22:34:10.271 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-20T22:34:10.271 INFO:teuthology.orchestra.run.smithi084.stdout:Upgraded: 2024-09-20T22:34:10.271 INFO:teuthology.orchestra.run.smithi084.stdout: librados2-2:19.1.1-367.g9e2ac70b.el9.x86_64 2024-09-20T22:34:10.271 INFO:teuthology.orchestra.run.smithi084.stdout: librbd1-2:19.1.1-367.g9e2ac70b.el9.x86_64 2024-09-20T22:34:10.271 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2024-09-20T22:34:10.271 INFO:teuthology.orchestra.run.smithi084.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-20T22:34:10.271 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-base-2:19.1.1-367.g9e2ac70b.el9.x86_64 2024-09-20T22:34:10.272 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-common-2:19.1.1-367.g9e2ac70b.el9.x86_64 2024-09-20T22:34:10.272 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-radosgw-2:19.1.1-367.g9e2ac70b.el9.x86_64 2024-09-20T22:34:10.272 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-selinux-2:19.1.1-367.g9e2ac70b.el9.x86_64 2024-09-20T22:34:10.272 INFO:teuthology.orchestra.run.smithi084.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-20T22:34:10.272 INFO:teuthology.orchestra.run.smithi084.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-20T22:34:10.272 INFO:teuthology.orchestra.run.smithi084.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-20T22:34:10.272 INFO:teuthology.orchestra.run.smithi084.stdout: libcephfs2-2:19.1.1-367.g9e2ac70b.el9.x86_64 2024-09-20T22:34:10.272 INFO:teuthology.orchestra.run.smithi084.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-20T22:34:10.272 INFO:teuthology.orchestra.run.smithi084.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-20T22:34:10.272 INFO:teuthology.orchestra.run.smithi084.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-20T22:34:10.272 INFO:teuthology.orchestra.run.smithi084.stdout: libradosstriper1-2:19.1.1-367.g9e2ac70b.el9.x86_64 2024-09-20T22:34:10.272 INFO:teuthology.orchestra.run.smithi084.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-20T22:34:10.272 INFO:teuthology.orchestra.run.smithi084.stdout: librgw2-2:19.1.1-367.g9e2ac70b.el9.x86_64 2024-09-20T22:34:10.272 INFO:teuthology.orchestra.run.smithi084.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-20T22:34:10.273 INFO:teuthology.orchestra.run.smithi084.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-20T22:34:10.273 INFO:teuthology.orchestra.run.smithi084.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-20T22:34:10.273 INFO:teuthology.orchestra.run.smithi084.stdout: python3-ceph-argparse-2:19.1.1-367.g9e2ac70b.el9.x86_64 2024-09-20T22:34:10.273 INFO:teuthology.orchestra.run.smithi084.stdout: python3-ceph-common-2:19.1.1-367.g9e2ac70b.el9.x86_64 2024-09-20T22:34:10.273 INFO:teuthology.orchestra.run.smithi084.stdout: python3-cephfs-2:19.1.1-367.g9e2ac70b.el9.x86_64 2024-09-20T22:34:10.273 INFO:teuthology.orchestra.run.smithi084.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-20T22:34:10.273 INFO:teuthology.orchestra.run.smithi084.stdout: python3-rados-2:19.1.1-367.g9e2ac70b.el9.x86_64 2024-09-20T22:34:10.273 INFO:teuthology.orchestra.run.smithi084.stdout: python3-rbd-2:19.1.1-367.g9e2ac70b.el9.x86_64 2024-09-20T22:34:10.273 INFO:teuthology.orchestra.run.smithi084.stdout: python3-rgw-2:19.1.1-367.g9e2ac70b.el9.x86_64 2024-09-20T22:34:10.273 INFO:teuthology.orchestra.run.smithi084.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-20T22:34:10.273 INFO:teuthology.orchestra.run.smithi084.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-20T22:34:10.273 INFO:teuthology.orchestra.run.smithi084.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-20T22:34:10.273 INFO:teuthology.orchestra.run.smithi084.stdout: re2-1:20211101-20.el9.x86_64 2024-09-20T22:34:10.273 INFO:teuthology.orchestra.run.smithi084.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-20T22:34:10.273 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-20T22:34:10.274 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-09-20T22:34:10.506 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:00:27 ago on Fri 20 Sep 2024 10:33:43 PM UTC. 2024-09-20T22:34:10.644 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2024-09-20T22:34:10.644 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-09-20T22:34:10.644 INFO:teuthology.orchestra.run.smithi154.stdout: Package Arch Version Repository Size 2024-09-20T22:34:10.645 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-09-20T22:34:10.645 INFO:teuthology.orchestra.run.smithi154.stdout:Installing: 2024-09-20T22:34:10.645 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-test x86_64 2:19.1.1-367.g9e2ac70b.el9 ceph 49 M 2024-09-20T22:34:10.645 INFO:teuthology.orchestra.run.smithi154.stdout:Installing dependencies: 2024-09-20T22:34:10.645 INFO:teuthology.orchestra.run.smithi154.stdout: libcephsqlite x86_64 2:19.1.1-367.g9e2ac70b.el9 ceph 163 k 2024-09-20T22:34:10.645 INFO:teuthology.orchestra.run.smithi154.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-20T22:34:10.645 INFO:teuthology.orchestra.run.smithi154.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-20T22:34:10.645 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-20T22:34:10.645 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2024-09-20T22:34:10.645 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-09-20T22:34:10.645 INFO:teuthology.orchestra.run.smithi154.stdout:Install 4 Packages 2024-09-20T22:34:10.646 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-20T22:34:10.646 INFO:teuthology.orchestra.run.smithi154.stdout:Total download size: 49 M 2024-09-20T22:34:10.646 INFO:teuthology.orchestra.run.smithi154.stdout:Installed size: 209 M 2024-09-20T22:34:10.646 INFO:teuthology.orchestra.run.smithi154.stdout:Downloading Packages: 2024-09-20T22:34:10.663 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install ceph-test 2024-09-20T22:34:10.980 INFO:teuthology.orchestra.run.smithi154.stdout:(1/4): socat-1.7.4.1-6.el9.x86_64.rpm 1.4 MB/s | 304 kB 00:00 2024-09-20T22:34:11.005 INFO:teuthology.orchestra.run.smithi154.stdout:(2/4): libcephsqlite-19.1.1-367.g9e2ac70b.el9.x 706 kB/s | 163 kB 00:00 2024-09-20T22:34:11.039 INFO:teuthology.orchestra.run.smithi154.stdout:(3/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.1 MB/s | 64 kB 00:00 2024-09-20T22:34:11.238 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:00:27 ago on Fri 20 Sep 2024 10:33:44 PM UTC. 2024-09-20T22:34:11.375 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-09-20T22:34:11.376 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-09-20T22:34:11.376 INFO:teuthology.orchestra.run.smithi084.stdout: Package Arch Version Repository Size 2024-09-20T22:34:11.376 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-09-20T22:34:11.376 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2024-09-20T22:34:11.376 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-test x86_64 2:19.1.1-367.g9e2ac70b.el9 ceph 49 M 2024-09-20T22:34:11.377 INFO:teuthology.orchestra.run.smithi084.stdout:Installing dependencies: 2024-09-20T22:34:11.377 INFO:teuthology.orchestra.run.smithi084.stdout: libcephsqlite x86_64 2:19.1.1-367.g9e2ac70b.el9 ceph 163 k 2024-09-20T22:34:11.377 INFO:teuthology.orchestra.run.smithi084.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-20T22:34:11.377 INFO:teuthology.orchestra.run.smithi084.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-20T22:34:11.377 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-20T22:34:11.377 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2024-09-20T22:34:11.377 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-09-20T22:34:11.377 INFO:teuthology.orchestra.run.smithi084.stdout:Install 4 Packages 2024-09-20T22:34:11.377 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-20T22:34:11.378 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 49 M 2024-09-20T22:34:11.378 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 209 M 2024-09-20T22:34:11.378 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2024-09-20T22:34:11.758 INFO:teuthology.orchestra.run.smithi084.stdout:(1/4): socat-1.7.4.1-6.el9.x86_64.rpm 1.2 MB/s | 304 kB 00:00 2024-09-20T22:34:11.809 INFO:teuthology.orchestra.run.smithi084.stdout:(2/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 64 kB 00:00 2024-09-20T22:34:11.850 INFO:teuthology.orchestra.run.smithi084.stdout:(3/4): libcephsqlite-19.1.1-367.g9e2ac70b.el9.x 480 kB/s | 163 kB 00:00 2024-09-20T22:34:12.273 INFO:teuthology.orchestra.run.smithi154.stdout:(4/4): ceph-test-19.1.1-367.g9e2ac70b.el9.x86_6 32 MB/s | 49 MB 00:01 2024-09-20T22:34:12.275 INFO:teuthology.orchestra.run.smithi154.stdout:-------------------------------------------------------------------------------- 2024-09-20T22:34:12.276 INFO:teuthology.orchestra.run.smithi154.stdout:Total 30 MB/s | 49 MB 00:01 2024-09-20T22:34:12.303 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2024-09-20T22:34:12.323 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2024-09-20T22:34:12.323 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2024-09-20T22:34:12.708 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2024-09-20T22:34:12.708 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2024-09-20T22:34:13.244 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2024-09-20T22:34:13.342 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-20T22:34:13.389 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-20T22:34:13.425 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : libcephsqlite-2:19.1.1-367.g9e2ac70b.el9.x86_64 3/4 2024-09-20T22:34:13.942 INFO:teuthology.orchestra.run.smithi084.stdout:(4/4): ceph-test-19.1.1-367.g9e2ac70b.el9.x86_6 20 MB/s | 49 MB 00:02 2024-09-20T22:34:13.944 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2024-09-20T22:34:13.944 INFO:teuthology.orchestra.run.smithi084.stdout:Total 19 MB/s | 49 MB 00:02 2024-09-20T22:34:13.972 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2024-09-20T22:34:13.992 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2024-09-20T22:34:13.992 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2024-09-20T22:34:14.380 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2024-09-20T22:34:14.381 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2024-09-20T22:34:14.921 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2024-09-20T22:34:15.019 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-20T22:34:15.066 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-20T22:34:15.094 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : libcephsqlite-2:19.1.1-367.g9e2ac70b.el9.x86_64 3/4 2024-09-20T22:34:18.027 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: libcephsqlite-2:19.1.1-367.g9e2ac70b.el9.x86_64 3/4 2024-09-20T22:34:18.107 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-test-2:19.1.1-367.g9e2ac70b.el9.x86_64 4/4 2024-09-20T22:34:19.716 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: libcephsqlite-2:19.1.1-367.g9e2ac70b.el9.x86_64 3/4 2024-09-20T22:34:19.800 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-test-2:19.1.1-367.g9e2ac70b.el9.x86_64 4/4 2024-09-20T22:34:20.501 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-test-2:19.1.1-367.g9e2ac70b.el9.x86_64 4/4 2024-09-20T22:34:20.501 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-test-2:19.1.1-367.g9e2ac70b.el9.x86_64 1/4 2024-09-20T22:34:20.502 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libcephsqlite-2:19.1.1-367.g9e2ac70b.el9.x86_64 2/4 2024-09-20T22:34:20.502 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-20T22:34:20.831 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-20T22:34:20.831 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-20T22:34:20.831 INFO:teuthology.orchestra.run.smithi154.stdout:Installed: 2024-09-20T22:34:20.831 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-test-2:19.1.1-367.g9e2ac70b.el9.x86_64 2024-09-20T22:34:20.832 INFO:teuthology.orchestra.run.smithi154.stdout: libcephsqlite-2:19.1.1-367.g9e2ac70b.el9.x86_64 2024-09-20T22:34:20.832 INFO:teuthology.orchestra.run.smithi154.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-20T22:34:20.832 INFO:teuthology.orchestra.run.smithi154.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-20T22:34:20.832 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-20T22:34:20.832 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2024-09-20T22:34:21.186 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install ceph 2024-09-20T22:34:21.812 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:00:38 ago on Fri 20 Sep 2024 10:33:43 PM UTC. 2024-09-20T22:34:21.962 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2024-09-20T22:34:21.965 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-09-20T22:34:21.965 INFO:teuthology.orchestra.run.smithi154.stdout: Package Arch Version Repository Size 2024-09-20T22:34:21.965 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-09-20T22:34:21.965 INFO:teuthology.orchestra.run.smithi154.stdout:Installing: 2024-09-20T22:34:21.965 INFO:teuthology.orchestra.run.smithi154.stdout: ceph x86_64 2:19.1.1-367.g9e2ac70b.el9 ceph 6.5 k 2024-09-20T22:34:21.965 INFO:teuthology.orchestra.run.smithi154.stdout:Installing dependencies: 2024-09-20T22:34:21.965 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mds x86_64 2:19.1.1-367.g9e2ac70b.el9 ceph 2.4 M 2024-09-20T22:34:21.965 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr x86_64 2:19.1.1-367.g9e2ac70b.el9 ceph 1.5 M 2024-09-20T22:34:21.965 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-modules-core noarch 2:19.1.1-367.g9e2ac70b.el9 ceph-noarch 249 k 2024-09-20T22:34:21.965 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mon x86_64 2:19.1.1-367.g9e2ac70b.el9 ceph 4.7 M 2024-09-20T22:34:21.965 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-osd x86_64 2:19.1.1-367.g9e2ac70b.el9 ceph 17 M 2024-09-20T22:34:21.965 INFO:teuthology.orchestra.run.smithi154.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-20T22:34:21.965 INFO:teuthology.orchestra.run.smithi154.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-20T22:34:21.965 INFO:teuthology.orchestra.run.smithi154.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-20T22:34:21.966 INFO:teuthology.orchestra.run.smithi154.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-20T22:34:21.966 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-20T22:34:21.966 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-20T22:34:21.966 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-20T22:34:21.966 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-20T22:34:21.966 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-20T22:34:21.966 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-20T22:34:21.966 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-09-20T22:34:21.966 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-20T22:34:21.966 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-20T22:34:21.966 INFO:teuthology.orchestra.run.smithi154.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-20T22:34:21.966 INFO:teuthology.orchestra.run.smithi154.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-20T22:34:21.966 INFO:teuthology.orchestra.run.smithi154.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-20T22:34:21.966 INFO:teuthology.orchestra.run.smithi154.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-20T22:34:21.967 INFO:teuthology.orchestra.run.smithi154.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-20T22:34:21.967 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-20T22:34:21.967 INFO:teuthology.orchestra.run.smithi154.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-20T22:34:21.967 INFO:teuthology.orchestra.run.smithi154.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-20T22:34:21.967 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-20T22:34:21.967 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-20T22:34:21.967 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-20T22:34:21.967 INFO:teuthology.orchestra.run.smithi154.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-20T22:34:21.967 INFO:teuthology.orchestra.run.smithi154.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-20T22:34:21.967 INFO:teuthology.orchestra.run.smithi154.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-20T22:34:21.967 INFO:teuthology.orchestra.run.smithi154.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-20T22:34:21.967 INFO:teuthology.orchestra.run.smithi154.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-20T22:34:21.967 INFO:teuthology.orchestra.run.smithi154.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-20T22:34:21.967 INFO:teuthology.orchestra.run.smithi154.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-20T22:34:21.968 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-20T22:34:21.968 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2024-09-20T22:34:21.968 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-09-20T22:34:21.968 INFO:teuthology.orchestra.run.smithi154.stdout:Install 37 Packages 2024-09-20T22:34:21.968 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-20T22:34:21.968 INFO:teuthology.orchestra.run.smithi154.stdout:Total download size: 30 M 2024-09-20T22:34:21.968 INFO:teuthology.orchestra.run.smithi154.stdout:Installed size: 108 M 2024-09-20T22:34:21.968 INFO:teuthology.orchestra.run.smithi154.stdout:Downloading Packages: 2024-09-20T22:34:22.333 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-test-2:19.1.1-367.g9e2ac70b.el9.x86_64 4/4 2024-09-20T22:34:22.333 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-test-2:19.1.1-367.g9e2ac70b.el9.x86_64 1/4 2024-09-20T22:34:22.333 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : libcephsqlite-2:19.1.1-367.g9e2ac70b.el9.x86_64 2/4 2024-09-20T22:34:22.333 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-20T22:34:22.340 INFO:teuthology.orchestra.run.smithi154.stdout:(1/37): ceph-19.1.1-367.g9e2ac70b.el9.x86_64.rp 54 kB/s | 6.5 kB 00:00 2024-09-20T22:34:22.524 INFO:teuthology.orchestra.run.smithi154.stdout:(2/37): ceph-mgr-19.1.1-367.g9e2ac70b.el9.x86_6 5.0 MB/s | 1.5 MB 00:00 2024-09-20T22:34:22.574 INFO:teuthology.orchestra.run.smithi154.stdout:(3/37): ceph-mds-19.1.1-367.g9e2ac70b.el9.x86_6 6.7 MB/s | 2.4 MB 00:00 2024-09-20T22:34:22.630 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-20T22:34:22.630 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-20T22:34:22.630 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2024-09-20T22:34:22.631 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-test-2:19.1.1-367.g9e2ac70b.el9.x86_64 2024-09-20T22:34:22.631 INFO:teuthology.orchestra.run.smithi084.stdout: libcephsqlite-2:19.1.1-367.g9e2ac70b.el9.x86_64 2024-09-20T22:34:22.631 INFO:teuthology.orchestra.run.smithi084.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-20T22:34:22.631 INFO:teuthology.orchestra.run.smithi084.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-20T22:34:22.631 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-20T22:34:22.631 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-09-20T22:34:22.674 INFO:teuthology.orchestra.run.smithi154.stdout:(4/37): ceph-mgr-modules-core-19.1.1-367.g9e2ac 2.4 MB/s | 249 kB 00:00 2024-09-20T22:34:22.758 INFO:teuthology.orchestra.run.smithi154.stdout:(5/37): ceph-mon-19.1.1-367.g9e2ac70b.el9.x86_6 11 MB/s | 4.7 MB 00:00 2024-09-20T22:34:22.883 INFO:teuthology.orchestra.run.smithi154.stdout:(6/37): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.2 MB/s | 253 kB 00:00 2024-09-20T22:34:22.925 INFO:teuthology.orchestra.run.smithi154.stdout:(7/37): python3-ply-3.11-14.el9.noarch.rpm 2.5 MB/s | 106 kB 00:00 2024-09-20T22:34:22.945 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install ceph 2024-09-20T22:34:22.967 INFO:teuthology.orchestra.run.smithi154.stdout:(8/37): python3-cryptography-36.0.1-4.el9.x86_6 5.9 MB/s | 1.2 MB 00:00 2024-09-20T22:34:22.992 INFO:teuthology.orchestra.run.smithi154.stdout:(9/37): python3-pycparser-2.20-6.el9.noarch.rpm 2.0 MB/s | 135 kB 00:00 2024-09-20T22:34:23.018 INFO:teuthology.orchestra.run.smithi154.stdout:(10/37): python3-requests-2.25.1-8.el9.noarch.r 2.4 MB/s | 125 kB 00:00 2024-09-20T22:34:23.043 INFO:teuthology.orchestra.run.smithi154.stdout:(11/37): python3-urllib3-1.26.5-6.el9.noarch.rp 4.2 MB/s | 215 kB 00:00 2024-09-20T22:34:23.227 INFO:teuthology.orchestra.run.smithi154.stdout:(12/37): ceph-osd-19.1.1-367.g9e2ac70b.el9.x86_ 24 MB/s | 17 MB 00:00 2024-09-20T22:34:23.327 INFO:teuthology.orchestra.run.smithi154.stdout:(13/37): python3-markupsafe-1.1.1-12.el9.x86_64 348 kB/s | 35 kB 00:00 2024-09-20T22:34:23.369 INFO:teuthology.orchestra.run.smithi154.stdout:(14/37): lua-5.4.4-4.el9.x86_64.rpm 537 kB/s | 188 kB 00:00 2024-09-20T22:34:23.394 INFO:teuthology.orchestra.run.smithi154.stdout:(15/37): python3-mako-1.1.4-6.el9.noarch.rpm 491 kB/s | 172 kB 00:00 2024-09-20T22:34:23.420 INFO:teuthology.orchestra.run.smithi154.stdout:(16/37): python3-packaging-20.9-5.el9.noarch.rp 838 kB/s | 77 kB 00:00 2024-09-20T22:34:23.447 INFO:teuthology.orchestra.run.smithi154.stdout:(17/37): python3-pytz-2021.1-5.el9.noarch.rpm 658 kB/s | 51 kB 00:00 2024-09-20T22:34:23.472 INFO:teuthology.orchestra.run.smithi154.stdout:(18/37): python3-toml-0.10.2-6.el9.noarch.rpm 536 kB/s | 42 kB 00:00 2024-09-20T22:34:23.497 INFO:teuthology.orchestra.run.smithi154.stdout:(19/37): lua-devel-5.4.4-4.el9.x86_64.rpm 287 kB/s | 22 kB 00:00 2024-09-20T22:34:23.531 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:00:39 ago on Fri 20 Sep 2024 10:33:44 PM UTC. 2024-09-20T22:34:23.548 INFO:teuthology.orchestra.run.smithi154.stdout:(20/37): luarocks-3.9.2-5.el9.noarch.rpm 1.5 MB/s | 151 kB 00:00 2024-09-20T22:34:23.573 INFO:teuthology.orchestra.run.smithi154.stdout:(21/37): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 432 kB/s | 43 kB 00:00 2024-09-20T22:34:23.598 INFO:teuthology.orchestra.run.smithi154.stdout:(22/37): python3-jaraco-8.2.1-3.el9.noarch.rpm 432 kB/s | 11 kB 00:00 2024-09-20T22:34:23.624 INFO:teuthology.orchestra.run.smithi154.stdout:(23/37): python3-jaraco-classes-3.2.1-5.el9.noa 711 kB/s | 18 kB 00:00 2024-09-20T22:34:23.649 INFO:teuthology.orchestra.run.smithi154.stdout:(24/37): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.2 MB/s | 172 kB 00:00 2024-09-20T22:34:23.671 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-09-20T22:34:23.674 INFO:teuthology.orchestra.run.smithi154.stdout:(25/37): python3-jaraco-collections-3.0.0-8.el9 460 kB/s | 23 kB 00:00 2024-09-20T22:34:23.674 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-09-20T22:34:23.674 INFO:teuthology.orchestra.run.smithi084.stdout: Package Arch Version Repository Size 2024-09-20T22:34:23.675 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-09-20T22:34:23.675 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2024-09-20T22:34:23.675 INFO:teuthology.orchestra.run.smithi084.stdout: ceph x86_64 2:19.1.1-367.g9e2ac70b.el9 ceph 6.5 k 2024-09-20T22:34:23.675 INFO:teuthology.orchestra.run.smithi084.stdout:Installing dependencies: 2024-09-20T22:34:23.675 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mds x86_64 2:19.1.1-367.g9e2ac70b.el9 ceph 2.4 M 2024-09-20T22:34:23.675 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mgr x86_64 2:19.1.1-367.g9e2ac70b.el9 ceph 1.5 M 2024-09-20T22:34:23.675 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mgr-modules-core noarch 2:19.1.1-367.g9e2ac70b.el9 ceph-noarch 249 k 2024-09-20T22:34:23.675 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mon x86_64 2:19.1.1-367.g9e2ac70b.el9 ceph 4.7 M 2024-09-20T22:34:23.675 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-osd x86_64 2:19.1.1-367.g9e2ac70b.el9 ceph 17 M 2024-09-20T22:34:23.675 INFO:teuthology.orchestra.run.smithi084.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-20T22:34:23.675 INFO:teuthology.orchestra.run.smithi084.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-20T22:34:23.675 INFO:teuthology.orchestra.run.smithi084.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-20T22:34:23.675 INFO:teuthology.orchestra.run.smithi084.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-20T22:34:23.676 INFO:teuthology.orchestra.run.smithi084.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-20T22:34:23.676 INFO:teuthology.orchestra.run.smithi084.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-20T22:34:23.676 INFO:teuthology.orchestra.run.smithi084.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-20T22:34:23.676 INFO:teuthology.orchestra.run.smithi084.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-20T22:34:23.676 INFO:teuthology.orchestra.run.smithi084.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-20T22:34:23.676 INFO:teuthology.orchestra.run.smithi084.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-20T22:34:23.676 INFO:teuthology.orchestra.run.smithi084.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-09-20T22:34:23.676 INFO:teuthology.orchestra.run.smithi084.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-20T22:34:23.676 INFO:teuthology.orchestra.run.smithi084.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-20T22:34:23.676 INFO:teuthology.orchestra.run.smithi084.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-20T22:34:23.676 INFO:teuthology.orchestra.run.smithi084.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-20T22:34:23.676 INFO:teuthology.orchestra.run.smithi084.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-20T22:34:23.676 INFO:teuthology.orchestra.run.smithi084.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-20T22:34:23.676 INFO:teuthology.orchestra.run.smithi084.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-20T22:34:23.677 INFO:teuthology.orchestra.run.smithi084.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-20T22:34:23.677 INFO:teuthology.orchestra.run.smithi084.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-20T22:34:23.677 INFO:teuthology.orchestra.run.smithi084.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-20T22:34:23.677 INFO:teuthology.orchestra.run.smithi084.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-20T22:34:23.677 INFO:teuthology.orchestra.run.smithi084.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-20T22:34:23.677 INFO:teuthology.orchestra.run.smithi084.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-20T22:34:23.677 INFO:teuthology.orchestra.run.smithi084.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-20T22:34:23.677 INFO:teuthology.orchestra.run.smithi084.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-20T22:34:23.677 INFO:teuthology.orchestra.run.smithi084.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-20T22:34:23.677 INFO:teuthology.orchestra.run.smithi084.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-20T22:34:23.677 INFO:teuthology.orchestra.run.smithi084.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-20T22:34:23.677 INFO:teuthology.orchestra.run.smithi084.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-20T22:34:23.677 INFO:teuthology.orchestra.run.smithi084.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-20T22:34:23.677 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-20T22:34:23.678 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2024-09-20T22:34:23.678 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-09-20T22:34:23.678 INFO:teuthology.orchestra.run.smithi084.stdout:Install 37 Packages 2024-09-20T22:34:23.678 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-20T22:34:23.678 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 30 M 2024-09-20T22:34:23.679 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 108 M 2024-09-20T22:34:23.679 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2024-09-20T22:34:23.699 INFO:teuthology.orchestra.run.smithi154.stdout:(26/37): python3-jaraco-functools-3.5.0-2.el9.n 387 kB/s | 19 kB 00:00 2024-09-20T22:34:23.725 INFO:teuthology.orchestra.run.smithi154.stdout:(27/37): python3-jaraco-text-3.2.0-6.el9.noarch 391 kB/s | 20 kB 00:00 2024-09-20T22:34:23.750 INFO:teuthology.orchestra.run.smithi154.stdout:(28/37): python3-logutils-0.3.5-21.el9.noarch.r 922 kB/s | 46 kB 00:00 2024-09-20T22:34:23.783 INFO:teuthology.orchestra.run.smithi154.stdout:(29/37): python3-cherrypy-18.6.1-2.el9.noarch.r 1.5 MB/s | 358 kB 00:00 2024-09-20T22:34:23.808 INFO:teuthology.orchestra.run.smithi154.stdout:(30/37): python3-more-itertools-8.12.0-2.el9.no 941 kB/s | 79 kB 00:00 2024-09-20T22:34:23.826 INFO:teuthology.orchestra.run.smithi154.stdout:(31/37): python3-portend-3.1.0-2.el9.noarch.rpm 393 kB/s | 16 kB 00:00 2024-09-20T22:34:23.842 INFO:teuthology.orchestra.run.smithi154.stdout:(32/37): python3-tempora-5.0.0-2.el9.noarch.rpm 2.1 MB/s | 36 kB 00:00 2024-09-20T22:34:23.876 INFO:teuthology.orchestra.run.smithi154.stdout:(33/37): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.3 MB/s | 90 kB 00:00 2024-09-20T22:34:23.901 INFO:teuthology.orchestra.run.smithi154.stdout:(34/37): python3-pecan-1.4.2-3.el9.noarch.rpm 1.8 MB/s | 272 kB 00:00 2024-09-20T22:34:23.935 INFO:teuthology.orchestra.run.smithi154.stdout:(35/37): python3-webob-1.8.8-2.el9.noarch.rpm 2.4 MB/s | 230 kB 00:00 2024-09-20T22:34:23.952 INFO:teuthology.orchestra.run.smithi154.stdout:(36/37): python3-zc-lockfile-2.0-10.el9.noarch. 397 kB/s | 20 kB 00:00 2024-09-20T22:34:23.977 INFO:teuthology.orchestra.run.smithi154.stdout:(37/37): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.1 MB/s | 427 kB 00:00 2024-09-20T22:34:23.981 INFO:teuthology.orchestra.run.smithi154.stdout:-------------------------------------------------------------------------------- 2024-09-20T22:34:23.982 INFO:teuthology.orchestra.run.smithi154.stdout:Total 15 MB/s | 30 MB 00:02 2024-09-20T22:34:24.132 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2024-09-20T22:34:24.173 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2024-09-20T22:34:24.173 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2024-09-20T22:34:24.291 INFO:teuthology.orchestra.run.smithi084.stdout:(1/37): ceph-19.1.1-367.g9e2ac70b.el9.x86_64.rp 53 kB/s | 6.5 kB 00:00 2024-09-20T22:34:24.458 INFO:teuthology.orchestra.run.smithi084.stdout:(2/37): ceph-mgr-19.1.1-367.g9e2ac70b.el9.x86_6 5.2 MB/s | 1.5 MB 00:00 2024-09-20T22:34:24.500 INFO:teuthology.orchestra.run.smithi084.stdout:(3/37): ceph-mds-19.1.1-367.g9e2ac70b.el9.x86_6 7.2 MB/s | 2.4 MB 00:00 2024-09-20T22:34:24.575 INFO:teuthology.orchestra.run.smithi084.stdout:(4/37): ceph-mgr-modules-core-19.1.1-367.g9e2ac 3.3 MB/s | 249 kB 00:00 2024-09-20T22:34:24.638 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2024-09-20T22:34:24.638 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2024-09-20T22:34:24.792 INFO:teuthology.orchestra.run.smithi084.stdout:(5/37): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.1 MB/s | 253 kB 00:00 2024-09-20T22:34:25.001 INFO:teuthology.orchestra.run.smithi084.stdout:(6/37): ceph-osd-19.1.1-367.g9e2ac70b.el9.x86_6 31 MB/s | 17 MB 00:00 2024-09-20T22:34:25.068 INFO:teuthology.orchestra.run.smithi084.stdout:(7/37): ceph-mon-19.1.1-367.g9e2ac70b.el9.x86_6 6.1 MB/s | 4.7 MB 00:00 2024-09-20T22:34:25.101 INFO:teuthology.orchestra.run.smithi084.stdout:(8/37): python3-cryptography-36.0.1-4.el9.x86_6 4.0 MB/s | 1.2 MB 00:00 2024-09-20T22:34:25.151 INFO:teuthology.orchestra.run.smithi084.stdout:(9/37): python3-requests-2.25.1-8.el9.noarch.rp 2.4 MB/s | 125 kB 00:00 2024-09-20T22:34:25.202 INFO:teuthology.orchestra.run.smithi084.stdout:(10/37): python3-ply-3.11-14.el9.noarch.rpm 531 kB/s | 106 kB 00:00 2024-09-20T22:34:25.229 INFO:teuthology.orchestra.run.smithi084.stdout:(11/37): python3-urllib3-1.26.5-6.el9.noarch.rp 2.7 MB/s | 215 kB 00:00 2024-09-20T22:34:25.254 INFO:teuthology.orchestra.run.smithi084.stdout:(12/37): python3-pycparser-2.20-6.el9.noarch.rp 725 kB/s | 135 kB 00:00 2024-09-20T22:34:25.413 INFO:teuthology.orchestra.run.smithi084.stdout:(13/37): python3-markupsafe-1.1.1-12.el9.x86_64 219 kB/s | 35 kB 00:00 2024-09-20T22:34:25.455 INFO:teuthology.orchestra.run.smithi084.stdout:(14/37): lua-5.4.4-4.el9.x86_64.rpm 746 kB/s | 188 kB 00:00 2024-09-20T22:34:25.531 INFO:teuthology.orchestra.run.smithi084.stdout:(15/37): python3-pytz-2021.1-5.el9.noarch.rpm 678 kB/s | 51 kB 00:00 2024-09-20T22:34:25.555 INFO:teuthology.orchestra.run.smithi084.stdout:(16/37): python3-mako-1.1.4-6.el9.noarch.rpm 528 kB/s | 172 kB 00:00 2024-09-20T22:34:25.606 INFO:teuthology.orchestra.run.smithi084.stdout:(17/37): python3-toml-0.10.2-6.el9.noarch.rpm 553 kB/s | 42 kB 00:00 2024-09-20T22:34:25.648 INFO:teuthology.orchestra.run.smithi084.stdout:(18/37): python3-packaging-20.9-5.el9.noarch.rp 329 kB/s | 77 kB 00:00 2024-09-20T22:34:25.673 INFO:teuthology.orchestra.run.smithi084.stdout:(19/37): luarocks-3.9.2-5.el9.noarch.rpm 2.2 MB/s | 151 kB 00:00 2024-09-20T22:34:25.707 INFO:teuthology.orchestra.run.smithi084.stdout:(20/37): python3-cheroot-8.6.0-4.el9.noarch.rpm 6.5 MB/s | 172 kB 00:00 2024-09-20T22:34:25.732 INFO:teuthology.orchestra.run.smithi084.stdout:(21/37): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 519 kB/s | 43 kB 00:00 2024-09-20T22:34:25.757 INFO:teuthology.orchestra.run.smithi084.stdout:(22/37): python3-jaraco-8.2.1-3.el9.noarch.rpm 429 kB/s | 11 kB 00:00 2024-09-20T22:34:25.782 INFO:teuthology.orchestra.run.smithi084.stdout:(23/37): python3-cherrypy-18.6.1-2.el9.noarch.r 4.7 MB/s | 358 kB 00:00 2024-09-20T22:34:25.808 INFO:teuthology.orchestra.run.smithi084.stdout:(24/37): python3-jaraco-classes-3.2.1-5.el9.noa 353 kB/s | 18 kB 00:00 2024-09-20T22:34:25.841 INFO:teuthology.orchestra.run.smithi084.stdout:(25/37): python3-jaraco-collections-3.0.0-8.el9 395 kB/s | 23 kB 00:00 2024-09-20T22:34:25.866 INFO:teuthology.orchestra.run.smithi084.stdout:(26/37): python3-jaraco-functools-3.5.0-2.el9.n 381 kB/s | 19 kB 00:00 2024-09-20T22:34:25.883 INFO:teuthology.orchestra.run.smithi084.stdout:(27/37): python3-jaraco-text-3.2.0-6.el9.noarch 473 kB/s | 20 kB 00:00 2024-09-20T22:34:25.917 INFO:teuthology.orchestra.run.smithi084.stdout:(28/37): python3-logutils-0.3.5-21.el9.noarch.r 923 kB/s | 46 kB 00:00 2024-09-20T22:34:25.942 INFO:teuthology.orchestra.run.smithi084.stdout:(29/37): python3-more-itertools-8.12.0-2.el9.no 1.3 MB/s | 79 kB 00:00 2024-09-20T22:34:25.954 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2024-09-20T22:34:25.959 INFO:teuthology.orchestra.run.smithi084.stdout:(30/37): python3-portend-3.1.0-2.el9.noarch.rpm 1.0 MB/s | 16 kB 00:00 2024-09-20T22:34:25.984 INFO:teuthology.orchestra.run.smithi084.stdout:(31/37): python3-pecan-1.4.2-3.el9.noarch.rpm 4.0 MB/s | 272 kB 00:00 2024-09-20T22:34:26.010 INFO:teuthology.orchestra.run.smithi084.stdout:(32/37): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.7 MB/s | 90 kB 00:00 2024-09-20T22:34:26.021 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/37 2024-09-20T22:34:26.035 INFO:teuthology.orchestra.run.smithi084.stdout:(33/37): python3-tempora-5.0.0-2.el9.noarch.rpm 712 kB/s | 36 kB 00:00 2024-09-20T22:34:26.061 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/37 2024-09-20T22:34:26.069 INFO:teuthology.orchestra.run.smithi084.stdout:(34/37): python3-webob-1.8.8-2.el9.noarch.rpm 3.9 MB/s | 230 kB 00:00 2024-09-20T22:34:26.090 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/37 2024-09-20T22:34:26.094 INFO:teuthology.orchestra.run.smithi084.stdout:(35/37): python3-zc-lockfile-2.0-10.el9.noarch. 806 kB/s | 20 kB 00:00 2024-09-20T22:34:26.136 INFO:teuthology.orchestra.run.smithi084.stdout:(36/37): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.2 MB/s | 427 kB 00:00 2024-09-20T22:34:26.164 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/37 2024-09-20T22:34:26.177 INFO:teuthology.orchestra.run.smithi084.stdout:(37/37): lua-devel-5.4.4-4.el9.x86_64.rpm 36 kB/s | 22 kB 00:00 2024-09-20T22:34:26.182 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2024-09-20T22:34:26.182 INFO:teuthology.orchestra.run.smithi084.stdout:Total 12 MB/s | 30 MB 00:02 2024-09-20T22:34:26.214 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/37 2024-09-20T22:34:26.264 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/37 2024-09-20T22:34:26.337 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2024-09-20T22:34:26.364 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/37 2024-09-20T22:34:26.377 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2024-09-20T22:34:26.377 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2024-09-20T22:34:26.451 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/37 2024-09-20T22:34:26.491 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 9/37 2024-09-20T22:34:26.526 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/37 2024-09-20T22:34:26.581 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/37 2024-09-20T22:34:26.616 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/37 2024-09-20T22:34:26.649 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/37 2024-09-20T22:34:26.690 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/37 2024-09-20T22:34:26.733 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/37 2024-09-20T22:34:26.776 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/37 2024-09-20T22:34:26.836 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2024-09-20T22:34:26.836 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2024-09-20T22:34:26.841 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/37 2024-09-20T22:34:26.872 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/37 2024-09-20T22:34:26.955 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/37 2024-09-20T22:34:27.094 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/37 2024-09-20T22:34:27.173 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/37 2024-09-20T22:34:27.235 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 22/37 2024-09-20T22:34:27.281 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 23/37 2024-09-20T22:34:27.337 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/37 2024-09-20T22:34:27.392 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/37 2024-09-20T22:34:27.582 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/37 2024-09-20T22:34:27.635 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 27/37 2024-09-20T22:34:27.713 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/37 2024-09-20T22:34:27.801 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 29/37 2024-09-20T22:34:27.842 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/37 2024-09-20T22:34:27.959 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/37 2024-09-20T22:34:28.113 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-mgr-modules-core-2:19.1.1-367.g9e2ac70b.el9.n 32/37 2024-09-20T22:34:28.144 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-mgr-2:19.1.1-367.g9e2ac70b.el9.x86_64 33/37 2024-09-20T22:34:28.158 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2024-09-20T22:34:28.169 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-mgr-2:19.1.1-367.g9e2ac70b.el9.x86_64 33/37 2024-09-20T22:34:28.169 INFO:teuthology.orchestra.run.smithi154.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-20T22:34:28.170 INFO:teuthology.orchestra.run.smithi154.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-20T22:34:28.170 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-20T22:34:28.170 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-20T22:34:28.170 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-20T22:34:28.230 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/37 2024-09-20T22:34:28.271 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/37 2024-09-20T22:34:28.307 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/37 2024-09-20T22:34:28.381 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/37 2024-09-20T22:34:28.431 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/37 2024-09-20T22:34:28.464 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/37 2024-09-20T22:34:28.556 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/37 2024-09-20T22:34:28.660 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/37 2024-09-20T22:34:28.717 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 9/37 2024-09-20T22:34:28.752 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/37 2024-09-20T22:34:28.782 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/37 2024-09-20T22:34:28.816 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/37 2024-09-20T22:34:28.849 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/37 2024-09-20T22:34:28.890 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/37 2024-09-20T22:34:28.942 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/37 2024-09-20T22:34:28.976 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/37 2024-09-20T22:34:29.025 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/37 2024-09-20T22:34:29.065 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/37 2024-09-20T22:34:29.147 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/37 2024-09-20T22:34:29.293 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/37 2024-09-20T22:34:29.372 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/37 2024-09-20T22:34:29.426 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 22/37 2024-09-20T22:34:29.464 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 23/37 2024-09-20T22:34:29.521 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/37 2024-09-20T22:34:29.585 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/37 2024-09-20T22:34:29.605 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-osd-2:19.1.1-367.g9e2ac70b.el9.x86_64 34/37 2024-09-20T22:34:29.630 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-osd-2:19.1.1-367.g9e2ac70b.el9.x86_64 34/37 2024-09-20T22:34:29.630 INFO:teuthology.orchestra.run.smithi154.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-20T22:34:29.630 INFO:teuthology.orchestra.run.smithi154.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-20T22:34:29.631 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-20T22:34:29.631 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-20T22:34:29.631 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-20T22:34:29.788 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/37 2024-09-20T22:34:29.835 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 27/37 2024-09-20T22:34:29.912 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/37 2024-09-20T22:34:30.002 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 29/37 2024-09-20T22:34:30.050 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/37 2024-09-20T22:34:30.070 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-mon-2:19.1.1-367.g9e2ac70b.el9.x86_64 35/37 2024-09-20T22:34:30.094 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-mon-2:19.1.1-367.g9e2ac70b.el9.x86_64 35/37 2024-09-20T22:34:30.095 INFO:teuthology.orchestra.run.smithi154.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-20T22:34:30.095 INFO:teuthology.orchestra.run.smithi154.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-20T22:34:30.095 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-20T22:34:30.095 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-20T22:34:30.095 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-20T22:34:30.176 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/37 2024-09-20T22:34:30.321 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-mds-2:19.1.1-367.g9e2ac70b.el9.x86_64 36/37 2024-09-20T22:34:30.338 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-mgr-modules-core-2:19.1.1-367.g9e2ac70b.el9.n 32/37 2024-09-20T22:34:30.345 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-mds-2:19.1.1-367.g9e2ac70b.el9.x86_64 36/37 2024-09-20T22:34:30.345 INFO:teuthology.orchestra.run.smithi154.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-20T22:34:30.345 INFO:teuthology.orchestra.run.smithi154.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-20T22:34:30.345 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-20T22:34:30.345 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-20T22:34:30.345 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-20T22:34:30.369 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-mgr-2:19.1.1-367.g9e2ac70b.el9.x86_64 33/37 2024-09-20T22:34:30.395 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-mgr-2:19.1.1-367.g9e2ac70b.el9.x86_64 33/37 2024-09-20T22:34:30.396 INFO:teuthology.orchestra.run.smithi084.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-20T22:34:30.396 INFO:teuthology.orchestra.run.smithi084.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-20T22:34:30.396 INFO:teuthology.orchestra.run.smithi084.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-20T22:34:30.396 INFO:teuthology.orchestra.run.smithi084.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-20T22:34:30.396 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-20T22:34:30.409 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-2:19.1.1-367.g9e2ac70b.el9.x86_64 37/37 2024-09-20T22:34:31.820 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-osd-2:19.1.1-367.g9e2ac70b.el9.x86_64 34/37 2024-09-20T22:34:31.848 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-osd-2:19.1.1-367.g9e2ac70b.el9.x86_64 34/37 2024-09-20T22:34:31.848 INFO:teuthology.orchestra.run.smithi084.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-20T22:34:31.848 INFO:teuthology.orchestra.run.smithi084.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-20T22:34:31.848 INFO:teuthology.orchestra.run.smithi084.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-20T22:34:31.849 INFO:teuthology.orchestra.run.smithi084.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-20T22:34:31.849 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-20T22:34:32.261 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-2:19.1.1-367.g9e2ac70b.el9.x86_64 37/37 2024-09-20T22:34:32.261 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-2:19.1.1-367.g9e2ac70b.el9.x86_64 1/37 2024-09-20T22:34:32.261 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-mds-2:19.1.1-367.g9e2ac70b.el9.x86_64 2/37 2024-09-20T22:34:32.261 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-mgr-2:19.1.1-367.g9e2ac70b.el9.x86_64 3/37 2024-09-20T22:34:32.261 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-mon-2:19.1.1-367.g9e2ac70b.el9.x86_64 4/37 2024-09-20T22:34:32.262 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-osd-2:19.1.1-367.g9e2ac70b.el9.x86_64 5/37 2024-09-20T22:34:32.262 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-mgr-modules-core-2:19.1.1-367.g9e2ac70b.el9.n 6/37 2024-09-20T22:34:32.262 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/37 2024-09-20T22:34:32.262 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/37 2024-09-20T22:34:32.262 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/37 2024-09-20T22:34:32.262 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/37 2024-09-20T22:34:32.262 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/37 2024-09-20T22:34:32.262 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/37 2024-09-20T22:34:32.263 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/37 2024-09-20T22:34:32.263 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/37 2024-09-20T22:34:32.263 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/37 2024-09-20T22:34:32.263 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 16/37 2024-09-20T22:34:32.263 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 17/37 2024-09-20T22:34:32.263 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 18/37 2024-09-20T22:34:32.263 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 19/37 2024-09-20T22:34:32.263 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 20/37 2024-09-20T22:34:32.264 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 21/37 2024-09-20T22:34:32.264 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 22/37 2024-09-20T22:34:32.264 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 23/37 2024-09-20T22:34:32.264 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 24/37 2024-09-20T22:34:32.264 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 25/37 2024-09-20T22:34:32.264 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 26/37 2024-09-20T22:34:32.264 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 27/37 2024-09-20T22:34:32.264 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 28/37 2024-09-20T22:34:32.264 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 29/37 2024-09-20T22:34:32.265 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 30/37 2024-09-20T22:34:32.265 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 31/37 2024-09-20T22:34:32.265 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 32/37 2024-09-20T22:34:32.265 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 33/37 2024-09-20T22:34:32.265 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 34/37 2024-09-20T22:34:32.265 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 35/37 2024-09-20T22:34:32.265 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 36/37 2024-09-20T22:34:32.287 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-mon-2:19.1.1-367.g9e2ac70b.el9.x86_64 35/37 2024-09-20T22:34:32.311 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-mon-2:19.1.1-367.g9e2ac70b.el9.x86_64 35/37 2024-09-20T22:34:32.311 INFO:teuthology.orchestra.run.smithi084.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-20T22:34:32.311 INFO:teuthology.orchestra.run.smithi084.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-20T22:34:32.311 INFO:teuthology.orchestra.run.smithi084.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-20T22:34:32.311 INFO:teuthology.orchestra.run.smithi084.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-20T22:34:32.311 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-20T22:34:32.529 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-mds-2:19.1.1-367.g9e2ac70b.el9.x86_64 36/37 2024-09-20T22:34:32.553 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-mds-2:19.1.1-367.g9e2ac70b.el9.x86_64 36/37 2024-09-20T22:34:32.553 INFO:teuthology.orchestra.run.smithi084.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-20T22:34:32.553 INFO:teuthology.orchestra.run.smithi084.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-20T22:34:32.553 INFO:teuthology.orchestra.run.smithi084.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-20T22:34:32.554 INFO:teuthology.orchestra.run.smithi084.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-20T22:34:32.554 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-20T22:34:32.618 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-2:19.1.1-367.g9e2ac70b.el9.x86_64 37/37 2024-09-20T22:34:33.240 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 37/37 2024-09-20T22:34:33.240 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-20T22:34:33.240 INFO:teuthology.orchestra.run.smithi154.stdout:Installed: 2024-09-20T22:34:33.241 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-2:19.1.1-367.g9e2ac70b.el9.x86_64 2024-09-20T22:34:33.241 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mds-2:19.1.1-367.g9e2ac70b.el9.x86_64 2024-09-20T22:34:33.241 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-2:19.1.1-367.g9e2ac70b.el9.x86_64 2024-09-20T22:34:33.241 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-modules-core-2:19.1.1-367.g9e2ac70b.el9.noarch 2024-09-20T22:34:33.241 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mon-2:19.1.1-367.g9e2ac70b.el9.x86_64 2024-09-20T22:34:33.241 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-osd-2:19.1.1-367.g9e2ac70b.el9.x86_64 2024-09-20T22:34:33.241 INFO:teuthology.orchestra.run.smithi154.stdout: lua-5.4.4-4.el9.x86_64 2024-09-20T22:34:33.241 INFO:teuthology.orchestra.run.smithi154.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-20T22:34:33.241 INFO:teuthology.orchestra.run.smithi154.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-20T22:34:33.241 INFO:teuthology.orchestra.run.smithi154.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-20T22:34:33.241 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-20T22:34:33.241 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-20T22:34:33.241 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-20T22:34:33.241 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-20T22:34:33.241 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-20T22:34:33.242 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-20T22:34:33.242 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-20T22:34:33.242 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-20T22:34:33.242 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-20T22:34:33.242 INFO:teuthology.orchestra.run.smithi154.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-20T22:34:33.242 INFO:teuthology.orchestra.run.smithi154.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-20T22:34:33.242 INFO:teuthology.orchestra.run.smithi154.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-20T22:34:33.242 INFO:teuthology.orchestra.run.smithi154.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-20T22:34:33.242 INFO:teuthology.orchestra.run.smithi154.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-20T22:34:33.242 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-20T22:34:33.242 INFO:teuthology.orchestra.run.smithi154.stdout: python3-ply-3.11-14.el9.noarch 2024-09-20T22:34:33.242 INFO:teuthology.orchestra.run.smithi154.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-20T22:34:33.242 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-20T22:34:33.242 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-20T22:34:33.243 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-20T22:34:33.243 INFO:teuthology.orchestra.run.smithi154.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-20T22:34:33.243 INFO:teuthology.orchestra.run.smithi154.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-20T22:34:33.243 INFO:teuthology.orchestra.run.smithi154.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-20T22:34:33.243 INFO:teuthology.orchestra.run.smithi154.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-20T22:34:33.243 INFO:teuthology.orchestra.run.smithi154.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-20T22:34:33.243 INFO:teuthology.orchestra.run.smithi154.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-20T22:34:33.243 INFO:teuthology.orchestra.run.smithi154.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-20T22:34:33.243 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-20T22:34:33.243 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2024-09-20T22:34:33.664 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install ceph-base 2024-09-20T22:34:34.245 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:00:51 ago on Fri 20 Sep 2024 10:33:43 PM UTC. 2024-09-20T22:34:34.293 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-2:19.1.1-367.g9e2ac70b.el9.x86_64 37/37 2024-09-20T22:34:34.294 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-2:19.1.1-367.g9e2ac70b.el9.x86_64 1/37 2024-09-20T22:34:34.294 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-mds-2:19.1.1-367.g9e2ac70b.el9.x86_64 2/37 2024-09-20T22:34:34.294 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-mgr-2:19.1.1-367.g9e2ac70b.el9.x86_64 3/37 2024-09-20T22:34:34.294 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-mon-2:19.1.1-367.g9e2ac70b.el9.x86_64 4/37 2024-09-20T22:34:34.294 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-osd-2:19.1.1-367.g9e2ac70b.el9.x86_64 5/37 2024-09-20T22:34:34.294 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-mgr-modules-core-2:19.1.1-367.g9e2ac70b.el9.n 6/37 2024-09-20T22:34:34.295 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/37 2024-09-20T22:34:34.295 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/37 2024-09-20T22:34:34.295 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/37 2024-09-20T22:34:34.295 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/37 2024-09-20T22:34:34.295 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/37 2024-09-20T22:34:34.295 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/37 2024-09-20T22:34:34.295 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/37 2024-09-20T22:34:34.296 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/37 2024-09-20T22:34:34.296 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/37 2024-09-20T22:34:34.296 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 16/37 2024-09-20T22:34:34.296 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 17/37 2024-09-20T22:34:34.296 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 18/37 2024-09-20T22:34:34.296 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 19/37 2024-09-20T22:34:34.296 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 20/37 2024-09-20T22:34:34.297 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 21/37 2024-09-20T22:34:34.297 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 22/37 2024-09-20T22:34:34.297 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 23/37 2024-09-20T22:34:34.297 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 24/37 2024-09-20T22:34:34.297 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 25/37 2024-09-20T22:34:34.297 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 26/37 2024-09-20T22:34:34.298 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 27/37 2024-09-20T22:34:34.298 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 28/37 2024-09-20T22:34:34.298 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 29/37 2024-09-20T22:34:34.298 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 30/37 2024-09-20T22:34:34.298 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 31/37 2024-09-20T22:34:34.298 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 32/37 2024-09-20T22:34:34.298 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 33/37 2024-09-20T22:34:34.299 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 34/37 2024-09-20T22:34:34.299 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 35/37 2024-09-20T22:34:34.299 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 36/37 2024-09-20T22:34:34.334 INFO:teuthology.orchestra.run.smithi154.stdout:Package ceph-base-2:19.1.1-367.g9e2ac70b.el9.x86_64 is already installed. 2024-09-20T22:34:34.377 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2024-09-20T22:34:34.378 INFO:teuthology.orchestra.run.smithi154.stdout:Nothing to do. 2024-09-20T22:34:34.378 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2024-09-20T22:34:34.461 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install cephadm 2024-09-20T22:34:35.042 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:00:52 ago on Fri 20 Sep 2024 10:33:43 PM UTC. 2024-09-20T22:34:35.172 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2024-09-20T22:34:35.173 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-09-20T22:34:35.173 INFO:teuthology.orchestra.run.smithi154.stdout: Package Arch Version Repository Size 2024-09-20T22:34:35.173 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-09-20T22:34:35.173 INFO:teuthology.orchestra.run.smithi154.stdout:Installing: 2024-09-20T22:34:35.173 INFO:teuthology.orchestra.run.smithi154.stdout: cephadm noarch 2:19.1.1-367.g9e2ac70b.el9 ceph-noarch 765 k 2024-09-20T22:34:35.174 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-20T22:34:35.174 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2024-09-20T22:34:35.174 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-09-20T22:34:35.174 INFO:teuthology.orchestra.run.smithi154.stdout:Install 1 Package 2024-09-20T22:34:35.174 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-20T22:34:35.174 INFO:teuthology.orchestra.run.smithi154.stdout:Total download size: 765 k 2024-09-20T22:34:35.174 INFO:teuthology.orchestra.run.smithi154.stdout:Installed size: 771 k 2024-09-20T22:34:35.174 INFO:teuthology.orchestra.run.smithi154.stdout:Downloading Packages: 2024-09-20T22:34:35.213 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 37/37 2024-09-20T22:34:35.213 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-20T22:34:35.214 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2024-09-20T22:34:35.214 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-2:19.1.1-367.g9e2ac70b.el9.x86_64 2024-09-20T22:34:35.214 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mds-2:19.1.1-367.g9e2ac70b.el9.x86_64 2024-09-20T22:34:35.214 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mgr-2:19.1.1-367.g9e2ac70b.el9.x86_64 2024-09-20T22:34:35.214 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mgr-modules-core-2:19.1.1-367.g9e2ac70b.el9.noarch 2024-09-20T22:34:35.214 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mon-2:19.1.1-367.g9e2ac70b.el9.x86_64 2024-09-20T22:34:35.214 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-osd-2:19.1.1-367.g9e2ac70b.el9.x86_64 2024-09-20T22:34:35.214 INFO:teuthology.orchestra.run.smithi084.stdout: lua-5.4.4-4.el9.x86_64 2024-09-20T22:34:35.214 INFO:teuthology.orchestra.run.smithi084.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-20T22:34:35.214 INFO:teuthology.orchestra.run.smithi084.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-20T22:34:35.214 INFO:teuthology.orchestra.run.smithi084.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-20T22:34:35.214 INFO:teuthology.orchestra.run.smithi084.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-20T22:34:35.215 INFO:teuthology.orchestra.run.smithi084.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-20T22:34:35.215 INFO:teuthology.orchestra.run.smithi084.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-20T22:34:35.215 INFO:teuthology.orchestra.run.smithi084.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-20T22:34:35.215 INFO:teuthology.orchestra.run.smithi084.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-20T22:34:35.215 INFO:teuthology.orchestra.run.smithi084.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-20T22:34:35.215 INFO:teuthology.orchestra.run.smithi084.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-20T22:34:35.215 INFO:teuthology.orchestra.run.smithi084.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-20T22:34:35.215 INFO:teuthology.orchestra.run.smithi084.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-20T22:34:35.215 INFO:teuthology.orchestra.run.smithi084.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-20T22:34:35.215 INFO:teuthology.orchestra.run.smithi084.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-20T22:34:35.215 INFO:teuthology.orchestra.run.smithi084.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-20T22:34:35.215 INFO:teuthology.orchestra.run.smithi084.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-20T22:34:35.215 INFO:teuthology.orchestra.run.smithi084.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-20T22:34:35.215 INFO:teuthology.orchestra.run.smithi084.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-20T22:34:35.216 INFO:teuthology.orchestra.run.smithi084.stdout: python3-ply-3.11-14.el9.noarch 2024-09-20T22:34:35.216 INFO:teuthology.orchestra.run.smithi084.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-20T22:34:35.216 INFO:teuthology.orchestra.run.smithi084.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-20T22:34:35.216 INFO:teuthology.orchestra.run.smithi084.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-20T22:34:35.216 INFO:teuthology.orchestra.run.smithi084.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-20T22:34:35.216 INFO:teuthology.orchestra.run.smithi084.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-20T22:34:35.216 INFO:teuthology.orchestra.run.smithi084.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-20T22:34:35.216 INFO:teuthology.orchestra.run.smithi084.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-20T22:34:35.216 INFO:teuthology.orchestra.run.smithi084.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-20T22:34:35.216 INFO:teuthology.orchestra.run.smithi084.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-20T22:34:35.216 INFO:teuthology.orchestra.run.smithi084.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-20T22:34:35.216 INFO:teuthology.orchestra.run.smithi084.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-20T22:34:35.216 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-20T22:34:35.216 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-09-20T22:34:35.453 INFO:teuthology.orchestra.run.smithi154.stdout:cephadm-19.1.1-367.g9e2ac70b.el9.noarch.rpm 2.7 MB/s | 765 kB 00:00 2024-09-20T22:34:35.454 INFO:teuthology.orchestra.run.smithi154.stdout:-------------------------------------------------------------------------------- 2024-09-20T22:34:35.454 INFO:teuthology.orchestra.run.smithi154.stdout:Total 2.7 MB/s | 765 kB 00:00 2024-09-20T22:34:35.454 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2024-09-20T22:34:35.459 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2024-09-20T22:34:35.459 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2024-09-20T22:34:35.466 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2024-09-20T22:34:35.466 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2024-09-20T22:34:35.541 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2024-09-20T22:34:35.651 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install ceph-base 2024-09-20T22:34:36.019 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: cephadm-2:19.1.1-367.g9e2ac70b.el9.noarch 1/1 2024-09-20T22:34:36.106 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : cephadm-2:19.1.1-367.g9e2ac70b.el9.noarch 1/1 2024-09-20T22:34:36.229 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:00:52 ago on Fri 20 Sep 2024 10:33:44 PM UTC. 2024-09-20T22:34:36.319 INFO:teuthology.orchestra.run.smithi084.stdout:Package ceph-base-2:19.1.1-367.g9e2ac70b.el9.x86_64 is already installed. 2024-09-20T22:34:36.361 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-09-20T22:34:36.362 INFO:teuthology.orchestra.run.smithi084.stdout:Nothing to do. 2024-09-20T22:34:36.362 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-09-20T22:34:36.418 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install cephadm 2024-09-20T22:34:36.460 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: cephadm-2:19.1.1-367.g9e2ac70b.el9.noarch 1/1 2024-09-20T22:34:36.699 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : cephadm-2:19.1.1-367.g9e2ac70b.el9.noarch 1/1 2024-09-20T22:34:36.699 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-20T22:34:36.699 INFO:teuthology.orchestra.run.smithi154.stdout:Installed: 2024-09-20T22:34:36.699 INFO:teuthology.orchestra.run.smithi154.stdout: cephadm-2:19.1.1-367.g9e2ac70b.el9.noarch 2024-09-20T22:34:36.699 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-20T22:34:36.699 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2024-09-20T22:34:36.934 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install ceph-immutable-object-cache 2024-09-20T22:34:36.996 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:00:52 ago on Fri 20 Sep 2024 10:33:44 PM UTC. 2024-09-20T22:34:37.127 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-09-20T22:34:37.127 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-09-20T22:34:37.128 INFO:teuthology.orchestra.run.smithi084.stdout: Package Arch Version Repository Size 2024-09-20T22:34:37.128 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-09-20T22:34:37.128 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2024-09-20T22:34:37.128 INFO:teuthology.orchestra.run.smithi084.stdout: cephadm noarch 2:19.1.1-367.g9e2ac70b.el9 ceph-noarch 765 k 2024-09-20T22:34:37.128 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-20T22:34:37.128 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2024-09-20T22:34:37.128 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-09-20T22:34:37.128 INFO:teuthology.orchestra.run.smithi084.stdout:Install 1 Package 2024-09-20T22:34:37.128 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-20T22:34:37.128 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 765 k 2024-09-20T22:34:37.128 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 771 k 2024-09-20T22:34:37.128 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2024-09-20T22:34:37.395 INFO:teuthology.orchestra.run.smithi084.stdout:cephadm-19.1.1-367.g9e2ac70b.el9.noarch.rpm 2.8 MB/s | 765 kB 00:00 2024-09-20T22:34:37.395 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2024-09-20T22:34:37.396 INFO:teuthology.orchestra.run.smithi084.stdout:Total 2.8 MB/s | 765 kB 00:00 2024-09-20T22:34:37.396 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2024-09-20T22:34:37.401 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2024-09-20T22:34:37.401 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2024-09-20T22:34:37.407 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2024-09-20T22:34:37.407 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2024-09-20T22:34:37.481 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2024-09-20T22:34:37.524 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:00:54 ago on Fri 20 Sep 2024 10:33:43 PM UTC. 2024-09-20T22:34:37.659 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2024-09-20T22:34:37.659 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-09-20T22:34:37.659 INFO:teuthology.orchestra.run.smithi154.stdout: Package Arch Version Repo Size 2024-09-20T22:34:37.660 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-09-20T22:34:37.660 INFO:teuthology.orchestra.run.smithi154.stdout:Installing: 2024-09-20T22:34:37.660 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-immutable-object-cache x86_64 2:19.1.1-367.g9e2ac70b.el9 ceph 145 k 2024-09-20T22:34:37.660 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-20T22:34:37.660 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2024-09-20T22:34:37.660 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-09-20T22:34:37.660 INFO:teuthology.orchestra.run.smithi154.stdout:Install 1 Package 2024-09-20T22:34:37.660 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-20T22:34:37.660 INFO:teuthology.orchestra.run.smithi154.stdout:Total download size: 145 k 2024-09-20T22:34:37.660 INFO:teuthology.orchestra.run.smithi154.stdout:Installed size: 436 k 2024-09-20T22:34:37.660 INFO:teuthology.orchestra.run.smithi154.stdout:Downloading Packages: 2024-09-20T22:34:37.867 INFO:teuthology.orchestra.run.smithi154.stdout:ceph-immutable-object-cache-19.1.1-367.g9e2ac70 702 kB/s | 145 kB 00:00 2024-09-20T22:34:37.868 INFO:teuthology.orchestra.run.smithi154.stdout:-------------------------------------------------------------------------------- 2024-09-20T22:34:37.868 INFO:teuthology.orchestra.run.smithi154.stdout:Total 697 kB/s | 145 kB 00:00 2024-09-20T22:34:37.868 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2024-09-20T22:34:37.876 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2024-09-20T22:34:37.876 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2024-09-20T22:34:37.938 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2024-09-20T22:34:37.938 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2024-09-20T22:34:37.977 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: cephadm-2:19.1.1-367.g9e2ac70b.el9.noarch 1/1 2024-09-20T22:34:38.048 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : cephadm-2:19.1.1-367.g9e2ac70b.el9.noarch 1/1 2024-09-20T22:34:38.081 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2024-09-20T22:34:38.152 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-immutable-object-cache-2:19.1.1-367.g9e2ac70b.e 1/1 2024-09-20T22:34:38.178 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.1.1-367.g9e2ac70b.e 1/1 2024-09-20T22:34:38.178 INFO:teuthology.orchestra.run.smithi154.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-20T22:34:38.178 INFO:teuthology.orchestra.run.smithi154.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-20T22:34:38.179 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-20T22:34:38.392 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: cephadm-2:19.1.1-367.g9e2ac70b.el9.noarch 1/1 2024-09-20T22:34:38.633 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : cephadm-2:19.1.1-367.g9e2ac70b.el9.noarch 1/1 2024-09-20T22:34:38.634 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-20T22:34:38.634 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2024-09-20T22:34:38.634 INFO:teuthology.orchestra.run.smithi084.stdout: cephadm-2:19.1.1-367.g9e2ac70b.el9.noarch 2024-09-20T22:34:38.634 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-20T22:34:38.634 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-09-20T22:34:38.824 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install ceph-immutable-object-cache 2024-09-20T22:34:39.117 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-immutable-object-cache-2:19.1.1-367.g9e2ac70b.e 1/1 2024-09-20T22:34:39.117 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-20T22:34:39.117 INFO:teuthology.orchestra.run.smithi154.stdout:Installed: 2024-09-20T22:34:39.117 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-immutable-object-cache-2:19.1.1-367.g9e2ac70b.el9.x86_64 2024-09-20T22:34:39.117 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-20T22:34:39.117 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2024-09-20T22:34:39.357 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install ceph-mgr 2024-09-20T22:34:39.409 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:00:55 ago on Fri 20 Sep 2024 10:33:44 PM UTC. 2024-09-20T22:34:39.543 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-09-20T22:34:39.544 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-09-20T22:34:39.544 INFO:teuthology.orchestra.run.smithi084.stdout: Package Arch Version Repo Size 2024-09-20T22:34:39.544 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-09-20T22:34:39.544 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2024-09-20T22:34:39.544 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-immutable-object-cache x86_64 2:19.1.1-367.g9e2ac70b.el9 ceph 145 k 2024-09-20T22:34:39.544 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-20T22:34:39.544 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2024-09-20T22:34:39.544 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-09-20T22:34:39.544 INFO:teuthology.orchestra.run.smithi084.stdout:Install 1 Package 2024-09-20T22:34:39.544 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-20T22:34:39.545 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 145 k 2024-09-20T22:34:39.545 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 436 k 2024-09-20T22:34:39.545 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2024-09-20T22:34:39.750 INFO:teuthology.orchestra.run.smithi084.stdout:ceph-immutable-object-cache-19.1.1-367.g9e2ac70 710 kB/s | 145 kB 00:00 2024-09-20T22:34:39.750 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2024-09-20T22:34:39.751 INFO:teuthology.orchestra.run.smithi084.stdout:Total 705 kB/s | 145 kB 00:00 2024-09-20T22:34:39.751 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2024-09-20T22:34:39.758 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2024-09-20T22:34:39.758 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2024-09-20T22:34:39.820 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2024-09-20T22:34:39.820 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2024-09-20T22:34:39.949 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:00:56 ago on Fri 20 Sep 2024 10:33:43 PM UTC. 2024-09-20T22:34:39.969 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2024-09-20T22:34:40.040 INFO:teuthology.orchestra.run.smithi154.stdout:Package ceph-mgr-2:19.1.1-367.g9e2ac70b.el9.x86_64 is already installed. 2024-09-20T22:34:40.042 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-immutable-object-cache-2:19.1.1-367.g9e2ac70b.e 1/1 2024-09-20T22:34:40.067 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.1.1-367.g9e2ac70b.e 1/1 2024-09-20T22:34:40.067 INFO:teuthology.orchestra.run.smithi084.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-20T22:34:40.067 INFO:teuthology.orchestra.run.smithi084.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-20T22:34:40.068 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-20T22:34:40.081 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2024-09-20T22:34:40.082 INFO:teuthology.orchestra.run.smithi154.stdout:Nothing to do. 2024-09-20T22:34:40.082 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2024-09-20T22:34:40.159 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install ceph-mgr-dashboard 2024-09-20T22:34:40.733 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:00:57 ago on Fri 20 Sep 2024 10:33:43 PM UTC. 2024-09-20T22:34:40.866 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2024-09-20T22:34:40.867 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-09-20T22:34:40.868 INFO:teuthology.orchestra.run.smithi154.stdout: Package Arch Version Repository Size 2024-09-20T22:34:40.868 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-09-20T22:34:40.868 INFO:teuthology.orchestra.run.smithi154.stdout:Installing: 2024-09-20T22:34:40.868 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-dashboard noarch 2:19.1.1-367.g9e2ac70b.el9 ceph-noarch 3.6 M 2024-09-20T22:34:40.868 INFO:teuthology.orchestra.run.smithi154.stdout:Installing dependencies: 2024-09-20T22:34:40.868 INFO:teuthology.orchestra.run.smithi154.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-20T22:34:40.868 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-grafana-dashboards noarch 2:19.1.1-367.g9e2ac70b.el9 ceph-noarch 32 k 2024-09-20T22:34:40.868 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-prometheus-alerts noarch 2:19.1.1-367.g9e2ac70b.el9 ceph-noarch 16 k 2024-09-20T22:34:40.868 INFO:teuthology.orchestra.run.smithi154.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-20T22:34:40.868 INFO:teuthology.orchestra.run.smithi154.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-20T22:34:40.868 INFO:teuthology.orchestra.run.smithi154.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-20T22:34:40.868 INFO:teuthology.orchestra.run.smithi154.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-20T22:34:40.868 INFO:teuthology.orchestra.run.smithi154.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-20T22:34:40.869 INFO:teuthology.orchestra.run.smithi154.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-20T22:34:40.869 INFO:teuthology.orchestra.run.smithi154.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-20T22:34:40.869 INFO:teuthology.orchestra.run.smithi154.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-20T22:34:40.869 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-20T22:34:40.869 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2024-09-20T22:34:40.869 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-09-20T22:34:40.869 INFO:teuthology.orchestra.run.smithi154.stdout:Install 12 Packages 2024-09-20T22:34:40.869 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-20T22:34:40.869 INFO:teuthology.orchestra.run.smithi154.stdout:Total download size: 8.7 M 2024-09-20T22:34:40.870 INFO:teuthology.orchestra.run.smithi154.stdout:Installed size: 105 M 2024-09-20T22:34:40.870 INFO:teuthology.orchestra.run.smithi154.stdout:Downloading Packages: 2024-09-20T22:34:40.993 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-immutable-object-cache-2:19.1.1-367.g9e2ac70b.e 1/1 2024-09-20T22:34:40.993 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-20T22:34:40.993 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2024-09-20T22:34:40.993 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-immutable-object-cache-2:19.1.1-367.g9e2ac70b.el9.x86_64 2024-09-20T22:34:40.993 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-20T22:34:40.993 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-09-20T22:34:41.202 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install ceph-mgr 2024-09-20T22:34:41.379 INFO:teuthology.orchestra.run.smithi154.stdout:(1/12): ceph-prometheus-alerts-19.1.1-367.g9e2a 100 kB/s | 16 kB 00:00 2024-09-20T22:34:41.396 INFO:teuthology.orchestra.run.smithi154.stdout:(2/12): ceph-grafana-dashboards-19.1.1-367.g9e2 176 kB/s | 32 kB 00:00 2024-09-20T22:34:41.522 INFO:teuthology.orchestra.run.smithi154.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 2.1 MB/s | 269 kB 00:00 2024-09-20T22:34:41.563 INFO:teuthology.orchestra.run.smithi154.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 5.5 MB/s | 1.0 MB 00:00 2024-09-20T22:34:41.639 INFO:teuthology.orchestra.run.smithi154.stdout:(5/12): ceph-mgr-dashboard-19.1.1-367.g9e2ac70b 8.4 MB/s | 3.6 MB 00:00 2024-09-20T22:34:41.681 INFO:teuthology.orchestra.run.smithi154.stdout:(6/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 4.6 MB/s | 548 kB 00:00 2024-09-20T22:34:41.706 INFO:teuthology.orchestra.run.smithi154.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 298 kB/s | 19 kB 00:00 2024-09-20T22:34:41.765 INFO:teuthology.orchestra.run.smithi154.stdout:(8/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 25 MB/s | 2.0 MB 00:00 2024-09-20T22:34:41.790 INFO:teuthology.orchestra.run.smithi154.stdout:(9/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 1.7 MB/s | 144 kB 00:00 2024-09-20T22:34:41.800 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:00:57 ago on Fri 20 Sep 2024 10:33:44 PM UTC. 2024-09-20T22:34:41.815 INFO:teuthology.orchestra.run.smithi154.stdout:(10/12): python3-repoze-lru-0.7-16.el9.noarch.r 613 kB/s | 31 kB 00:00 2024-09-20T22:34:41.840 INFO:teuthology.orchestra.run.smithi154.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 3.7 MB/s | 188 kB 00:00 2024-09-20T22:34:41.890 INFO:teuthology.orchestra.run.smithi084.stdout:Package ceph-mgr-2:19.1.1-367.g9e2ac70b.el9.x86_64 is already installed. 2024-09-20T22:34:41.932 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-09-20T22:34:41.933 INFO:teuthology.orchestra.run.smithi084.stdout:Nothing to do. 2024-09-20T22:34:41.934 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-09-20T22:34:41.997 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install ceph-mgr-dashboard 2024-09-20T22:34:42.374 INFO:teuthology.orchestra.run.smithi154.stdout:(12/12): protobuf-compiler-3.14.0-13.el9.x86_64 1.0 MB/s | 863 kB 00:00 2024-09-20T22:34:42.378 INFO:teuthology.orchestra.run.smithi154.stdout:-------------------------------------------------------------------------------- 2024-09-20T22:34:42.378 INFO:teuthology.orchestra.run.smithi154.stdout:Total 5.7 MB/s | 8.7 MB 00:01 2024-09-20T22:34:42.427 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2024-09-20T22:34:42.443 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2024-09-20T22:34:42.444 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2024-09-20T22:34:42.567 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:00:58 ago on Fri 20 Sep 2024 10:33:44 PM UTC. 2024-09-20T22:34:42.648 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2024-09-20T22:34:42.648 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2024-09-20T22:34:42.698 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-09-20T22:34:42.699 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-09-20T22:34:42.700 INFO:teuthology.orchestra.run.smithi084.stdout: Package Arch Version Repository Size 2024-09-20T22:34:42.700 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-09-20T22:34:42.700 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2024-09-20T22:34:42.700 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mgr-dashboard noarch 2:19.1.1-367.g9e2ac70b.el9 ceph-noarch 3.6 M 2024-09-20T22:34:42.700 INFO:teuthology.orchestra.run.smithi084.stdout:Installing dependencies: 2024-09-20T22:34:42.700 INFO:teuthology.orchestra.run.smithi084.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-20T22:34:42.700 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-grafana-dashboards noarch 2:19.1.1-367.g9e2ac70b.el9 ceph-noarch 32 k 2024-09-20T22:34:42.700 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-prometheus-alerts noarch 2:19.1.1-367.g9e2ac70b.el9 ceph-noarch 16 k 2024-09-20T22:34:42.700 INFO:teuthology.orchestra.run.smithi084.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-20T22:34:42.700 INFO:teuthology.orchestra.run.smithi084.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-20T22:34:42.700 INFO:teuthology.orchestra.run.smithi084.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-20T22:34:42.700 INFO:teuthology.orchestra.run.smithi084.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-20T22:34:42.701 INFO:teuthology.orchestra.run.smithi084.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-20T22:34:42.701 INFO:teuthology.orchestra.run.smithi084.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-20T22:34:42.701 INFO:teuthology.orchestra.run.smithi084.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-20T22:34:42.701 INFO:teuthology.orchestra.run.smithi084.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-20T22:34:42.701 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-20T22:34:42.701 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2024-09-20T22:34:42.701 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-09-20T22:34:42.701 INFO:teuthology.orchestra.run.smithi084.stdout:Install 12 Packages 2024-09-20T22:34:42.701 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-20T22:34:42.702 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 8.7 M 2024-09-20T22:34:42.702 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 105 M 2024-09-20T22:34:42.702 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2024-09-20T22:34:43.045 INFO:teuthology.orchestra.run.smithi084.stdout:(1/12): ceph-prometheus-alerts-19.1.1-367.g9e2a 102 kB/s | 16 kB 00:00 2024-09-20T22:34:43.069 INFO:teuthology.orchestra.run.smithi084.stdout:(2/12): ceph-grafana-dashboards-19.1.1-367.g9e2 171 kB/s | 32 kB 00:00 2024-09-20T22:34:43.182 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2024-09-20T22:34:43.237 INFO:teuthology.orchestra.run.smithi084.stdout:(3/12): ceph-mgr-dashboard-19.1.1-367.g9e2ac70b 10 MB/s | 3.6 MB 00:00 2024-09-20T22:34:43.274 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-09-20T22:34:43.310 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-09-20T22:34:43.357 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-09-20T22:34:43.370 INFO:teuthology.orchestra.run.smithi084.stdout:(4/12): python3-protobuf-3.14.0-13.el9.noarch.r 897 kB/s | 269 kB 00:00 2024-09-20T22:34:43.404 INFO:teuthology.orchestra.run.smithi084.stdout:(5/12): protobuf-3.14.0-13.el9.x86_64.rpm 2.8 MB/s | 1.0 MB 00:00 2024-09-20T22:34:43.411 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-09-20T22:34:43.454 INFO:teuthology.orchestra.run.smithi084.stdout:(6/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 6.4 MB/s | 548 kB 00:00 2024-09-20T22:34:43.479 INFO:teuthology.orchestra.run.smithi084.stdout:(7/12): protobuf-compiler-3.14.0-13.el9.x86_64. 3.5 MB/s | 863 kB 00:00 2024-09-20T22:34:43.505 INFO:teuthology.orchestra.run.smithi084.stdout:(8/12): grpc-data-1.46.7-10.el9.noarch.rpm 208 kB/s | 19 kB 00:00 2024-09-20T22:34:43.513 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-09-20T22:34:43.538 INFO:teuthology.orchestra.run.smithi084.stdout:(9/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 2.4 MB/s | 144 kB 00:00 2024-09-20T22:34:43.572 INFO:teuthology.orchestra.run.smithi084.stdout:(10/12): python3-repoze-lru-0.7-16.el9.noarch.r 459 kB/s | 31 kB 00:00 2024-09-20T22:34:43.597 INFO:teuthology.orchestra.run.smithi084.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 3.2 MB/s | 188 kB 00:00 2024-09-20T22:34:43.639 INFO:teuthology.orchestra.run.smithi084.stdout:(12/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 11 MB/s | 2.0 MB 00:00 2024-09-20T22:34:43.642 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2024-09-20T22:34:43.643 INFO:teuthology.orchestra.run.smithi084.stdout:Total 9.2 MB/s | 8.7 MB 00:00 2024-09-20T22:34:43.650 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-09-20T22:34:43.690 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2024-09-20T22:34:43.707 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2024-09-20T22:34:43.707 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2024-09-20T22:34:43.742 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-09-20T22:34:43.793 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-09-20T22:34:43.820 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-09-20T22:34:43.851 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-prometheus-alerts-2:19.1.1-367.g9e2ac70b.el9. 10/12 2024-09-20T22:34:43.920 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2024-09-20T22:34:43.921 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2024-09-20T22:34:44.448 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2024-09-20T22:34:44.523 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-09-20T22:34:44.566 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-09-20T22:34:44.620 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-09-20T22:34:44.660 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-09-20T22:34:44.753 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-09-20T22:34:44.894 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-09-20T22:34:44.922 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-grafana-dashboards-2:19.1.1-367.g9e2ac70b.el9 11/12 2024-09-20T22:34:44.960 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-mgr-dashboard-2:19.1.1-367.g9e2ac70b.el9.noar 12/12 2024-09-20T22:34:44.987 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-09-20T22:34:45.031 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-09-20T22:34:45.059 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-09-20T22:34:45.091 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-prometheus-alerts-2:19.1.1-367.g9e2ac70b.el9. 10/12 2024-09-20T22:34:46.153 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-grafana-dashboards-2:19.1.1-367.g9e2ac70b.el9 11/12 2024-09-20T22:34:46.192 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-mgr-dashboard-2:19.1.1-367.g9e2ac70b.el9.noar 12/12 2024-09-20T22:34:46.411 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.1.1-367.g9e2ac70b.el9.noar 12/12 2024-09-20T22:34:46.411 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-grafana-dashboards-2:19.1.1-367.g9e2ac70b.el9 1/12 2024-09-20T22:34:46.411 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-mgr-dashboard-2:19.1.1-367.g9e2ac70b.el9.noar 2/12 2024-09-20T22:34:46.411 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-prometheus-alerts-2:19.1.1-367.g9e2ac70b.el9. 3/12 2024-09-20T22:34:46.411 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-09-20T22:34:46.411 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-09-20T22:34:46.411 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-09-20T22:34:46.411 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-09-20T22:34:46.411 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-09-20T22:34:46.411 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-09-20T22:34:46.411 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-09-20T22:34:46.411 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-09-20T22:34:46.748 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-09-20T22:34:46.748 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-20T22:34:46.748 INFO:teuthology.orchestra.run.smithi154.stdout:Installed: 2024-09-20T22:34:46.748 INFO:teuthology.orchestra.run.smithi154.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-20T22:34:46.748 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-grafana-dashboards-2:19.1.1-367.g9e2ac70b.el9.noarch 2024-09-20T22:34:46.748 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-dashboard-2:19.1.1-367.g9e2ac70b.el9.noarch 2024-09-20T22:34:46.748 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-prometheus-alerts-2:19.1.1-367.g9e2ac70b.el9.noarch 2024-09-20T22:34:46.748 INFO:teuthology.orchestra.run.smithi154.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-20T22:34:46.748 INFO:teuthology.orchestra.run.smithi154.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-20T22:34:46.748 INFO:teuthology.orchestra.run.smithi154.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-20T22:34:46.748 INFO:teuthology.orchestra.run.smithi154.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-20T22:34:46.749 INFO:teuthology.orchestra.run.smithi154.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-20T22:34:46.749 INFO:teuthology.orchestra.run.smithi154.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-20T22:34:46.749 INFO:teuthology.orchestra.run.smithi154.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-20T22:34:46.749 INFO:teuthology.orchestra.run.smithi154.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-20T22:34:46.749 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-20T22:34:46.749 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2024-09-20T22:34:46.995 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-20T22:34:47.496 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.1.1-367.g9e2ac70b.el9.noar 12/12 2024-09-20T22:34:47.496 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-grafana-dashboards-2:19.1.1-367.g9e2ac70b.el9 1/12 2024-09-20T22:34:47.496 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-mgr-dashboard-2:19.1.1-367.g9e2ac70b.el9.noar 2/12 2024-09-20T22:34:47.496 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-prometheus-alerts-2:19.1.1-367.g9e2ac70b.el9. 3/12 2024-09-20T22:34:47.496 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-09-20T22:34:47.497 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-09-20T22:34:47.497 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-09-20T22:34:47.497 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-09-20T22:34:47.497 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-09-20T22:34:47.497 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-09-20T22:34:47.497 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-09-20T22:34:47.497 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-09-20T22:34:47.575 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:01:04 ago on Fri 20 Sep 2024 10:33:43 PM UTC. 2024-09-20T22:34:47.707 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2024-09-20T22:34:47.708 INFO:teuthology.orchestra.run.smithi154.stdout:====================================================================================== 2024-09-20T22:34:47.708 INFO:teuthology.orchestra.run.smithi154.stdout: Package Arch Version Repository Size 2024-09-20T22:34:47.708 INFO:teuthology.orchestra.run.smithi154.stdout:====================================================================================== 2024-09-20T22:34:47.708 INFO:teuthology.orchestra.run.smithi154.stdout:Installing: 2024-09-20T22:34:47.708 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-diskprediction-local noarch 2:19.1.1-367.g9e2ac70b.el9 ceph-noarch 7.4 M 2024-09-20T22:34:47.709 INFO:teuthology.orchestra.run.smithi154.stdout:Installing dependencies: 2024-09-20T22:34:47.709 INFO:teuthology.orchestra.run.smithi154.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-20T22:34:47.709 INFO:teuthology.orchestra.run.smithi154.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-20T22:34:47.709 INFO:teuthology.orchestra.run.smithi154.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-20T22:34:47.709 INFO:teuthology.orchestra.run.smithi154.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-20T22:34:47.709 INFO:teuthology.orchestra.run.smithi154.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-20T22:34:47.709 INFO:teuthology.orchestra.run.smithi154.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-20T22:34:47.709 INFO:teuthology.orchestra.run.smithi154.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-20T22:34:47.709 INFO:teuthology.orchestra.run.smithi154.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-20T22:34:47.709 INFO:teuthology.orchestra.run.smithi154.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-20T22:34:47.709 INFO:teuthology.orchestra.run.smithi154.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-20T22:34:47.709 INFO:teuthology.orchestra.run.smithi154.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-20T22:34:47.709 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-20T22:34:47.709 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2024-09-20T22:34:47.710 INFO:teuthology.orchestra.run.smithi154.stdout:====================================================================================== 2024-09-20T22:34:47.710 INFO:teuthology.orchestra.run.smithi154.stdout:Install 12 Packages 2024-09-20T22:34:47.710 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-20T22:34:47.710 INFO:teuthology.orchestra.run.smithi154.stdout:Total download size: 38 M 2024-09-20T22:34:47.710 INFO:teuthology.orchestra.run.smithi154.stdout:Installed size: 212 M 2024-09-20T22:34:47.710 INFO:teuthology.orchestra.run.smithi154.stdout:Downloading Packages: 2024-09-20T22:34:47.807 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-09-20T22:34:47.807 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-20T22:34:47.807 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2024-09-20T22:34:47.807 INFO:teuthology.orchestra.run.smithi084.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-20T22:34:47.807 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-grafana-dashboards-2:19.1.1-367.g9e2ac70b.el9.noarch 2024-09-20T22:34:47.807 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mgr-dashboard-2:19.1.1-367.g9e2ac70b.el9.noarch 2024-09-20T22:34:47.807 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-prometheus-alerts-2:19.1.1-367.g9e2ac70b.el9.noarch 2024-09-20T22:34:47.808 INFO:teuthology.orchestra.run.smithi084.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-20T22:34:47.808 INFO:teuthology.orchestra.run.smithi084.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-20T22:34:47.808 INFO:teuthology.orchestra.run.smithi084.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-20T22:34:47.808 INFO:teuthology.orchestra.run.smithi084.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-20T22:34:47.808 INFO:teuthology.orchestra.run.smithi084.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-20T22:34:47.808 INFO:teuthology.orchestra.run.smithi084.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-20T22:34:47.808 INFO:teuthology.orchestra.run.smithi084.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-20T22:34:47.808 INFO:teuthology.orchestra.run.smithi084.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-20T22:34:47.808 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-20T22:34:47.808 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-09-20T22:34:48.048 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-20T22:34:48.248 INFO:teuthology.orchestra.run.smithi154.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 804 kB/s | 191 kB 00:00 2024-09-20T22:34:48.348 INFO:teuthology.orchestra.run.smithi154.stdout:(2/12): libgfortran-11.5.0-2.el9.x86_64.rpm 2.3 MB/s | 801 kB 00:00 2024-09-20T22:34:48.507 INFO:teuthology.orchestra.run.smithi154.stdout:(3/12): ceph-mgr-diskprediction-local-19.1.1-36 15 MB/s | 7.4 MB 00:00 2024-09-20T22:34:48.620 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:01:04 ago on Fri 20 Sep 2024 10:33:44 PM UTC. 2024-09-20T22:34:48.752 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-09-20T22:34:48.753 INFO:teuthology.orchestra.run.smithi084.stdout:====================================================================================== 2024-09-20T22:34:48.754 INFO:teuthology.orchestra.run.smithi084.stdout: Package Arch Version Repository Size 2024-09-20T22:34:48.754 INFO:teuthology.orchestra.run.smithi084.stdout:====================================================================================== 2024-09-20T22:34:48.754 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2024-09-20T22:34:48.754 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mgr-diskprediction-local noarch 2:19.1.1-367.g9e2ac70b.el9 ceph-noarch 7.4 M 2024-09-20T22:34:48.754 INFO:teuthology.orchestra.run.smithi084.stdout:Installing dependencies: 2024-09-20T22:34:48.754 INFO:teuthology.orchestra.run.smithi084.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-20T22:34:48.754 INFO:teuthology.orchestra.run.smithi084.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-20T22:34:48.754 INFO:teuthology.orchestra.run.smithi084.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-20T22:34:48.754 INFO:teuthology.orchestra.run.smithi084.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-20T22:34:48.754 INFO:teuthology.orchestra.run.smithi084.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-20T22:34:48.754 INFO:teuthology.orchestra.run.smithi084.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-20T22:34:48.754 INFO:teuthology.orchestra.run.smithi084.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-20T22:34:48.754 INFO:teuthology.orchestra.run.smithi084.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-20T22:34:48.754 INFO:teuthology.orchestra.run.smithi084.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-20T22:34:48.755 INFO:teuthology.orchestra.run.smithi084.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-20T22:34:48.755 INFO:teuthology.orchestra.run.smithi084.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-20T22:34:48.755 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-20T22:34:48.755 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2024-09-20T22:34:48.755 INFO:teuthology.orchestra.run.smithi084.stdout:====================================================================================== 2024-09-20T22:34:48.755 INFO:teuthology.orchestra.run.smithi084.stdout:Install 12 Packages 2024-09-20T22:34:48.755 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-20T22:34:48.755 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 38 M 2024-09-20T22:34:48.755 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 212 M 2024-09-20T22:34:48.756 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2024-09-20T22:34:48.916 INFO:teuthology.orchestra.run.smithi154.stdout:(4/12): flexiblas-3.0.4-8.el9.x86_64.rpm 49 kB/s | 33 kB 00:00 2024-09-20T22:34:48.999 INFO:teuthology.orchestra.run.smithi154.stdout:(5/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 37 kB/s | 18 kB 00:00 2024-09-20T22:34:49.479 INFO:teuthology.orchestra.run.smithi084.stdout:(1/12): ceph-mgr-diskprediction-local-19.1.1-36 16 MB/s | 7.4 MB 00:00 2024-09-20T22:34:49.491 INFO:teuthology.orchestra.run.smithi154.stdout:(6/12): openblas-0.3.26-2.el9.x86_64.rpm 67 kB/s | 38 kB 00:00 2024-09-20T22:34:49.804 INFO:teuthology.orchestra.run.smithi084.stdout:(2/12): flexiblas-3.0.4-8.el9.x86_64.rpm 101 kB/s | 33 kB 00:00 2024-09-20T22:34:49.921 INFO:teuthology.orchestra.run.smithi084.stdout:(3/12): libquadmath-11.5.0-2.el9.x86_64.rpm 209 kB/s | 191 kB 00:00 2024-09-20T22:34:50.222 INFO:teuthology.orchestra.run.smithi084.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 61 kB/s | 18 kB 00:00 2024-09-20T22:34:50.358 INFO:teuthology.orchestra.run.smithi154.stdout:(7/12): python3-devel-3.9.19-8.el9.x86_64.rpm 283 kB/s | 245 kB 00:00 2024-09-20T22:34:50.422 INFO:teuthology.orchestra.run.smithi084.stdout:(5/12): openblas-0.3.26-2.el9.x86_64.rpm 193 kB/s | 38 kB 00:00 2024-09-20T22:34:50.597 INFO:teuthology.orchestra.run.smithi084.stdout:(6/12): libgfortran-11.5.0-2.el9.x86_64.rpm 503 kB/s | 801 kB 00:01 2024-09-20T22:34:50.839 INFO:teuthology.orchestra.run.smithi084.stdout:(7/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 2.9 MB/s | 3.0 MB 00:01 2024-09-20T22:34:51.247 INFO:teuthology.orchestra.run.smithi084.stdout:(8/12): python3-devel-3.9.19-8.el9.x86_64.rpm 376 kB/s | 245 kB 00:00 2024-09-20T22:34:51.447 INFO:teuthology.orchestra.run.smithi084.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 1.2 MB/s | 250 kB 00:00 2024-09-20T22:34:51.623 INFO:teuthology.orchestra.run.smithi084.stdout:(10/12): openblas-openmp-0.3.26-2.el9.x86_64.rp 4.1 MB/s | 4.9 MB 00:01 2024-09-20T22:34:51.849 INFO:teuthology.orchestra.run.smithi154.stdout:(8/12): openblas-openmp-0.3.26-2.el9.x86_64.rpm 1.7 MB/s | 4.9 MB 00:02 2024-09-20T22:34:52.281 INFO:teuthology.orchestra.run.smithi084.stdout:(11/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 3.5 MB/s | 5.1 MB 00:01 2024-09-20T22:34:52.550 INFO:teuthology.orchestra.run.smithi154.stdout:(9/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 729 kB/s | 3.0 MB 00:04 2024-09-20T22:34:52.675 INFO:teuthology.orchestra.run.smithi154.stdout:(10/12): python3-numpy-f2py-1.20.1-5.el9.x86_64 304 kB/s | 250 kB 00:00 2024-09-20T22:34:52.956 INFO:teuthology.orchestra.run.smithi084.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 11 MB/s | 16 MB 00:01 2024-09-20T22:34:52.957 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2024-09-20T22:34:52.958 INFO:teuthology.orchestra.run.smithi084.stdout:Total 9.1 MB/s | 38 MB 00:04 2024-09-20T22:34:53.309 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2024-09-20T22:34:53.335 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2024-09-20T22:34:53.335 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2024-09-20T22:34:53.755 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2024-09-20T22:34:53.756 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2024-09-20T22:34:54.530 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2024-09-20T22:34:54.587 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-20T22:34:54.644 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-20T22:34:54.744 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-20T22:34:54.780 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-20T22:34:54.964 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-20T22:34:55.004 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-20T22:34:55.080 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-20T22:34:55.721 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-20T22:34:55.840 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-20T22:34:55.900 INFO:teuthology.orchestra.run.smithi154.stdout:(11/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 4.9 MB/s | 16 MB 00:03 2024-09-20T22:34:57.062 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-20T22:34:57.116 INFO:teuthology.orchestra.run.smithi154.stdout:(12/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 772 kB/s | 5.1 MB 00:06 2024-09-20T22:34:57.118 INFO:teuthology.orchestra.run.smithi154.stdout:-------------------------------------------------------------------------------- 2024-09-20T22:34:57.118 INFO:teuthology.orchestra.run.smithi154.stdout:Total 4.1 MB/s | 38 MB 00:09 2024-09-20T22:34:57.478 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2024-09-20T22:34:57.505 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2024-09-20T22:34:57.505 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2024-09-20T22:34:57.933 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2024-09-20T22:34:57.934 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2024-09-20T22:34:58.118 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-20T22:34:58.144 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-mgr-diskprediction-local-2:19.1.1-367.g9e2ac7 12/12 2024-09-20T22:34:58.690 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2024-09-20T22:34:58.772 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-20T22:34:58.818 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-20T22:34:58.914 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-20T22:34:58.966 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-20T22:34:59.150 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-20T22:34:59.181 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-20T22:34:59.258 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-20T22:34:59.924 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-20T22:35:00.044 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-20T22:35:00.813 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.1.1-367.g9e2ac7 12/12 2024-09-20T22:35:00.813 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.1.1-367.g9e2ac7 1/12 2024-09-20T22:35:00.813 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-20T22:35:00.813 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-20T22:35:00.813 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-20T22:35:00.813 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-20T22:35:00.813 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-20T22:35:00.813 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-20T22:35:00.814 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-20T22:35:00.814 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-20T22:35:00.814 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-20T22:35:00.814 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-20T22:35:01.223 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-20T22:35:01.223 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-20T22:35:01.223 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2024-09-20T22:35:01.223 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mgr-diskprediction-local-2:19.1.1-367.g9e2ac70b.el9.noarch 2024-09-20T22:35:01.223 INFO:teuthology.orchestra.run.smithi084.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-20T22:35:01.224 INFO:teuthology.orchestra.run.smithi084.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-20T22:35:01.224 INFO:teuthology.orchestra.run.smithi084.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-20T22:35:01.224 INFO:teuthology.orchestra.run.smithi084.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-20T22:35:01.224 INFO:teuthology.orchestra.run.smithi084.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-20T22:35:01.224 INFO:teuthology.orchestra.run.smithi084.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-20T22:35:01.224 INFO:teuthology.orchestra.run.smithi084.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-20T22:35:01.224 INFO:teuthology.orchestra.run.smithi084.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-20T22:35:01.224 INFO:teuthology.orchestra.run.smithi084.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-20T22:35:01.224 INFO:teuthology.orchestra.run.smithi084.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-20T22:35:01.224 INFO:teuthology.orchestra.run.smithi084.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-20T22:35:01.224 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-20T22:35:01.224 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-09-20T22:35:01.287 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-20T22:35:01.565 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install ceph-mgr-rook 2024-09-20T22:35:02.156 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:01:18 ago on Fri 20 Sep 2024 10:33:44 PM UTC. 2024-09-20T22:35:02.290 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-09-20T22:35:02.292 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-09-20T22:35:02.292 INFO:teuthology.orchestra.run.smithi084.stdout: Package Arch Version Repository Size 2024-09-20T22:35:02.292 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-09-20T22:35:02.292 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2024-09-20T22:35:02.292 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mgr-rook noarch 2:19.1.1-367.g9e2ac70b.el9 ceph-noarch 49 k 2024-09-20T22:35:02.292 INFO:teuthology.orchestra.run.smithi084.stdout:Installing dependencies: 2024-09-20T22:35:02.292 INFO:teuthology.orchestra.run.smithi084.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-20T22:35:02.292 INFO:teuthology.orchestra.run.smithi084.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-20T22:35:02.292 INFO:teuthology.orchestra.run.smithi084.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-20T22:35:02.292 INFO:teuthology.orchestra.run.smithi084.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-20T22:35:02.292 INFO:teuthology.orchestra.run.smithi084.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-20T22:35:02.292 INFO:teuthology.orchestra.run.smithi084.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-20T22:35:02.293 INFO:teuthology.orchestra.run.smithi084.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-20T22:35:02.293 INFO:teuthology.orchestra.run.smithi084.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-20T22:35:02.293 INFO:teuthology.orchestra.run.smithi084.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-20T22:35:02.293 INFO:teuthology.orchestra.run.smithi084.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-20T22:35:02.293 INFO:teuthology.orchestra.run.smithi084.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-20T22:35:02.293 INFO:teuthology.orchestra.run.smithi084.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-20T22:35:02.293 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-20T22:35:02.293 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2024-09-20T22:35:02.293 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-09-20T22:35:02.293 INFO:teuthology.orchestra.run.smithi084.stdout:Install 13 Packages 2024-09-20T22:35:02.293 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-20T22:35:02.294 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 2.2 M 2024-09-20T22:35:02.294 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 26 M 2024-09-20T22:35:02.294 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2024-09-20T22:35:02.321 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-20T22:35:02.355 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-mgr-diskprediction-local-2:19.1.1-367.g9e2ac7 12/12 2024-09-20T22:35:02.594 INFO:teuthology.orchestra.run.smithi084.stdout:(1/13): ceph-mgr-rook-19.1.1-367.g9e2ac70b.el9. 289 kB/s | 49 kB 00:00 2024-09-20T22:35:02.944 INFO:teuthology.orchestra.run.smithi084.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 37 kB/s | 19 kB 00:00 2024-09-20T22:35:02.994 INFO:teuthology.orchestra.run.smithi084.stdout:(3/13): python3-jsonpatch-1.21-16.el9.noarch.rp 45 kB/s | 26 kB 00:00 2024-09-20T22:35:03.420 INFO:teuthology.orchestra.run.smithi084.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 269 kB/s | 222 kB 00:00 2024-09-20T22:35:03.653 INFO:teuthology.orchestra.run.smithi084.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 224 kB/s | 159 kB 00:00 2024-09-20T22:35:03.703 INFO:teuthology.orchestra.run.smithi084.stdout:(6/13): python3-cachetools-4.2.4-1.el9.noarch.r 762 kB/s | 32 kB 00:00 2024-09-20T22:35:03.728 INFO:teuthology.orchestra.run.smithi084.stdout:(7/13): python3-certifi-2023.05.07-4.el9.noarch 565 kB/s | 14 kB 00:00 2024-09-20T22:35:03.754 INFO:teuthology.orchestra.run.smithi084.stdout:(8/13): python3-google-auth-2.34.0-1.el9.noarch 8.7 MB/s | 220 kB 00:00 2024-09-20T22:35:03.779 INFO:teuthology.orchestra.run.smithi084.stdout:(9/13): python3-pyasn1-modules-0.4.8-6.el9.noar 355 kB/s | 279 kB 00:00 2024-09-20T22:35:03.829 INFO:teuthology.orchestra.run.smithi084.stdout:(10/13): python3-kubernetes-26.1.0-2.el9.noarch 14 MB/s | 1.0 MB 00:00 2024-09-20T22:35:03.855 INFO:teuthology.orchestra.run.smithi084.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 3.8 MB/s | 90 kB 00:00 2024-09-20T22:35:03.871 INFO:teuthology.orchestra.run.smithi084.stdout:(12/13): python3-rsa-4.9-2.el9.noarch.rpm 642 kB/s | 59 kB 00:00 2024-09-20T22:35:04.072 INFO:teuthology.orchestra.run.smithi084.stdout:(13/13): python3-requests-oauthlib-1.3.0-12.el9 82 kB/s | 54 kB 00:00 2024-09-20T22:35:04.075 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2024-09-20T22:35:04.076 INFO:teuthology.orchestra.run.smithi084.stdout:Total 1.2 MB/s | 2.2 MB 00:01 2024-09-20T22:35:04.144 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2024-09-20T22:35:04.160 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2024-09-20T22:35:04.160 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2024-09-20T22:35:04.252 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2024-09-20T22:35:04.253 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2024-09-20T22:35:04.756 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2024-09-20T22:35:04.910 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-20T22:35:04.950 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.1.1-367.g9e2ac7 12/12 2024-09-20T22:35:04.950 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.1.1-367.g9e2ac7 1/12 2024-09-20T22:35:04.950 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-20T22:35:04.950 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-20T22:35:04.950 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-20T22:35:04.950 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-20T22:35:04.950 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-20T22:35:04.951 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-20T22:35:04.951 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-20T22:35:04.951 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-20T22:35:04.951 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-20T22:35:04.951 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-20T22:35:04.964 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-20T22:35:05.028 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-20T22:35:05.065 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-20T22:35:05.107 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-20T22:35:05.217 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-20T22:35:05.328 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-20T22:35:05.373 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-20T22:35:05.373 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-20T22:35:05.373 INFO:teuthology.orchestra.run.smithi154.stdout:Installed: 2024-09-20T22:35:05.373 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-diskprediction-local-2:19.1.1-367.g9e2ac70b.el9.noarch 2024-09-20T22:35:05.373 INFO:teuthology.orchestra.run.smithi154.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-20T22:35:05.373 INFO:teuthology.orchestra.run.smithi154.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-20T22:35:05.373 INFO:teuthology.orchestra.run.smithi154.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-20T22:35:05.373 INFO:teuthology.orchestra.run.smithi154.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-20T22:35:05.373 INFO:teuthology.orchestra.run.smithi154.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-20T22:35:05.373 INFO:teuthology.orchestra.run.smithi154.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-20T22:35:05.373 INFO:teuthology.orchestra.run.smithi154.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-20T22:35:05.373 INFO:teuthology.orchestra.run.smithi154.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-20T22:35:05.374 INFO:teuthology.orchestra.run.smithi154.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-20T22:35:05.374 INFO:teuthology.orchestra.run.smithi154.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-20T22:35:05.374 INFO:teuthology.orchestra.run.smithi154.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-20T22:35:05.374 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-20T22:35:05.374 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2024-09-20T22:35:05.375 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-20T22:35:05.720 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install ceph-mgr-rook 2024-09-20T22:35:06.094 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-20T22:35:06.147 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-20T22:35:06.189 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-20T22:35:06.245 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-20T22:35:06.275 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-mgr-rook-2:19.1.1-367.g9e2ac70b.el9.noarch 13/13 2024-09-20T22:35:06.321 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:01:23 ago on Fri 20 Sep 2024 10:33:43 PM UTC. 2024-09-20T22:35:06.455 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2024-09-20T22:35:06.456 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-09-20T22:35:06.457 INFO:teuthology.orchestra.run.smithi154.stdout: Package Arch Version Repository Size 2024-09-20T22:35:06.457 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-09-20T22:35:06.457 INFO:teuthology.orchestra.run.smithi154.stdout:Installing: 2024-09-20T22:35:06.457 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-rook noarch 2:19.1.1-367.g9e2ac70b.el9 ceph-noarch 49 k 2024-09-20T22:35:06.457 INFO:teuthology.orchestra.run.smithi154.stdout:Installing dependencies: 2024-09-20T22:35:06.457 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-20T22:35:06.457 INFO:teuthology.orchestra.run.smithi154.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-20T22:35:06.457 INFO:teuthology.orchestra.run.smithi154.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-20T22:35:06.457 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-20T22:35:06.457 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-20T22:35:06.457 INFO:teuthology.orchestra.run.smithi154.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-20T22:35:06.457 INFO:teuthology.orchestra.run.smithi154.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-20T22:35:06.458 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-20T22:35:06.458 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-20T22:35:06.458 INFO:teuthology.orchestra.run.smithi154.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-20T22:35:06.458 INFO:teuthology.orchestra.run.smithi154.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-20T22:35:06.458 INFO:teuthology.orchestra.run.smithi154.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-20T22:35:06.458 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-20T22:35:06.458 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2024-09-20T22:35:06.458 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-09-20T22:35:06.458 INFO:teuthology.orchestra.run.smithi154.stdout:Install 13 Packages 2024-09-20T22:35:06.458 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-20T22:35:06.459 INFO:teuthology.orchestra.run.smithi154.stdout:Total download size: 2.2 M 2024-09-20T22:35:06.459 INFO:teuthology.orchestra.run.smithi154.stdout:Installed size: 26 M 2024-09-20T22:35:06.459 INFO:teuthology.orchestra.run.smithi154.stdout:Downloading Packages: 2024-09-20T22:35:06.707 INFO:teuthology.orchestra.run.smithi154.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 267 kB/s | 26 kB 00:00 2024-09-20T22:35:06.732 INFO:teuthology.orchestra.run.smithi154.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 158 kB/s | 19 kB 00:00 2024-09-20T22:35:06.790 INFO:teuthology.orchestra.run.smithi154.stdout:(3/13): ceph-mgr-rook-19.1.1-367.g9e2ac70b.el9. 274 kB/s | 49 kB 00:00 2024-09-20T22:35:06.824 INFO:teuthology.orchestra.run.smithi154.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.9 MB/s | 222 kB 00:00 2024-09-20T22:35:06.849 INFO:teuthology.orchestra.run.smithi154.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.3 MB/s | 159 kB 00:00 2024-09-20T22:35:06.883 INFO:teuthology.orchestra.run.smithi154.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 915 kB/s | 54 kB 00:00 2024-09-20T22:35:06.916 INFO:teuthology.orchestra.run.smithi154.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 480 kB/s | 32 kB 00:00 2024-09-20T22:35:06.942 INFO:teuthology.orchestra.run.smithi154.stdout:(8/13): python3-certifi-2023.05.07-4.el9.noarch 249 kB/s | 14 kB 00:00 2024-09-20T22:35:06.967 INFO:teuthology.orchestra.run.smithi154.stdout:(9/13): python3-google-auth-2.34.0-1.el9.noarch 4.4 MB/s | 220 kB 00:00 2024-09-20T22:35:06.992 INFO:teuthology.orchestra.run.smithi154.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 2.3 MB/s | 59 kB 00:00 2024-09-20T22:35:07.018 INFO:teuthology.orchestra.run.smithi154.stdout:(11/13): python3-pyasn1-modules-0.4.8-6.el9.noa 1.2 MB/s | 279 kB 00:00 2024-09-20T22:35:07.034 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-mgr-rook-2:19.1.1-367.g9e2ac70b.el9.noarch 13/13 2024-09-20T22:35:07.034 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-mgr-rook-2:19.1.1-367.g9e2ac70b.el9.noarch 1/13 2024-09-20T22:35:07.034 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-20T22:35:07.034 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-20T22:35:07.034 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-20T22:35:07.034 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-20T22:35:07.034 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-20T22:35:07.034 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-20T22:35:07.035 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-20T22:35:07.035 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-20T22:35:07.035 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-20T22:35:07.035 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-20T22:35:07.035 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-20T22:35:07.043 INFO:teuthology.orchestra.run.smithi154.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.7 MB/s | 90 kB 00:00 2024-09-20T22:35:07.084 INFO:teuthology.orchestra.run.smithi154.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 7.2 MB/s | 1.0 MB 00:00 2024-09-20T22:35:07.087 INFO:teuthology.orchestra.run.smithi154.stdout:-------------------------------------------------------------------------------- 2024-09-20T22:35:07.087 INFO:teuthology.orchestra.run.smithi154.stdout:Total 3.5 MB/s | 2.2 MB 00:00 2024-09-20T22:35:07.151 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2024-09-20T22:35:07.166 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2024-09-20T22:35:07.166 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2024-09-20T22:35:07.254 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2024-09-20T22:35:07.261 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2024-09-20T22:35:07.355 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-20T22:35:07.355 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-20T22:35:07.355 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2024-09-20T22:35:07.355 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mgr-rook-2:19.1.1-367.g9e2ac70b.el9.noarch 2024-09-20T22:35:07.355 INFO:teuthology.orchestra.run.smithi084.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-20T22:35:07.355 INFO:teuthology.orchestra.run.smithi084.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-20T22:35:07.355 INFO:teuthology.orchestra.run.smithi084.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-20T22:35:07.355 INFO:teuthology.orchestra.run.smithi084.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-20T22:35:07.355 INFO:teuthology.orchestra.run.smithi084.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-20T22:35:07.355 INFO:teuthology.orchestra.run.smithi084.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-20T22:35:07.356 INFO:teuthology.orchestra.run.smithi084.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-20T22:35:07.356 INFO:teuthology.orchestra.run.smithi084.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-20T22:35:07.356 INFO:teuthology.orchestra.run.smithi084.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-20T22:35:07.356 INFO:teuthology.orchestra.run.smithi084.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-20T22:35:07.356 INFO:teuthology.orchestra.run.smithi084.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-20T22:35:07.356 INFO:teuthology.orchestra.run.smithi084.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-20T22:35:07.356 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-20T22:35:07.356 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-09-20T22:35:07.608 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install ceph-mgr-cephadm 2024-09-20T22:35:07.770 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2024-09-20T22:35:07.938 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-20T22:35:08.003 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-20T22:35:08.074 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-20T22:35:08.112 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-20T22:35:08.161 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-20T22:35:08.185 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:01:24 ago on Fri 20 Sep 2024 10:33:44 PM UTC. 2024-09-20T22:35:08.262 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-20T22:35:08.323 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-09-20T22:35:08.324 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-09-20T22:35:08.324 INFO:teuthology.orchestra.run.smithi084.stdout: Package Arch Version Repository Size 2024-09-20T22:35:08.324 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-09-20T22:35:08.324 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2024-09-20T22:35:08.324 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mgr-cephadm noarch 2:19.1.1-367.g9e2ac70b.el9 ceph-noarch 146 k 2024-09-20T22:35:08.324 INFO:teuthology.orchestra.run.smithi084.stdout:Installing dependencies: 2024-09-20T22:35:08.324 INFO:teuthology.orchestra.run.smithi084.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-20T22:35:08.324 INFO:teuthology.orchestra.run.smithi084.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-20T22:35:08.324 INFO:teuthology.orchestra.run.smithi084.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-20T22:35:08.324 INFO:teuthology.orchestra.run.smithi084.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-20T22:35:08.324 INFO:teuthology.orchestra.run.smithi084.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-20T22:35:08.324 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-20T22:35:08.324 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2024-09-20T22:35:08.325 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-09-20T22:35:08.325 INFO:teuthology.orchestra.run.smithi084.stdout:Install 6 Packages 2024-09-20T22:35:08.325 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-20T22:35:08.325 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 7.0 M 2024-09-20T22:35:08.325 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 33 M 2024-09-20T22:35:08.325 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2024-09-20T22:35:08.364 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-20T22:35:08.412 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-20T22:35:08.872 INFO:teuthology.orchestra.run.smithi084.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.7 MB/s | 248 kB 00:00 2024-09-20T22:35:08.931 INFO:teuthology.orchestra.run.smithi084.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 9.2 MB/s | 548 kB 00:00 2024-09-20T22:35:08.956 INFO:teuthology.orchestra.run.smithi084.stdout:(3/6): ceph-mgr-cephadm-19.1.1-367.g9e2ac70b.el 649 kB/s | 146 kB 00:00 2024-09-20T22:35:08.981 INFO:teuthology.orchestra.run.smithi084.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 1.1 MB/s | 58 kB 00:00 2024-09-20T22:35:09.032 INFO:teuthology.orchestra.run.smithi084.stdout:(5/6): python3-typing-extensions-4.12.2-2.el9.n 989 kB/s | 74 kB 00:00 2024-09-20T22:35:09.124 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-20T22:35:09.177 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-20T22:35:09.211 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-20T22:35:09.223 INFO:teuthology.orchestra.run.smithi084.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 12 MB/s | 6.0 MB 00:00 2024-09-20T22:35:09.226 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2024-09-20T22:35:09.226 INFO:teuthology.orchestra.run.smithi084.stdout:Total 7.8 MB/s | 7.0 MB 00:00 2024-09-20T22:35:09.259 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-20T22:35:09.289 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-mgr-rook-2:19.1.1-367.g9e2ac70b.el9.noarch 13/13 2024-09-20T22:35:09.297 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2024-09-20T22:35:09.308 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2024-09-20T22:35:09.309 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2024-09-20T22:35:09.409 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2024-09-20T22:35:09.410 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2024-09-20T22:35:09.736 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2024-09-20T22:35:09.906 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-20T22:35:09.958 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-20T22:35:10.078 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-mgr-rook-2:19.1.1-367.g9e2ac70b.el9.noarch 13/13 2024-09-20T22:35:10.078 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-mgr-rook-2:19.1.1-367.g9e2ac70b.el9.noarch 1/13 2024-09-20T22:35:10.078 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-20T22:35:10.078 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-20T22:35:10.078 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-20T22:35:10.078 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-20T22:35:10.079 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-20T22:35:10.079 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-20T22:35:10.079 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-20T22:35:10.079 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-20T22:35:10.079 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-20T22:35:10.079 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-20T22:35:10.079 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-20T22:35:10.420 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-20T22:35:10.420 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-20T22:35:10.420 INFO:teuthology.orchestra.run.smithi154.stdout:Installed: 2024-09-20T22:35:10.420 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-rook-2:19.1.1-367.g9e2ac70b.el9.noarch 2024-09-20T22:35:10.420 INFO:teuthology.orchestra.run.smithi154.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-20T22:35:10.420 INFO:teuthology.orchestra.run.smithi154.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-20T22:35:10.420 INFO:teuthology.orchestra.run.smithi154.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-20T22:35:10.420 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-20T22:35:10.420 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-20T22:35:10.420 INFO:teuthology.orchestra.run.smithi154.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-20T22:35:10.421 INFO:teuthology.orchestra.run.smithi154.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-20T22:35:10.421 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-20T22:35:10.421 INFO:teuthology.orchestra.run.smithi154.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-20T22:35:10.421 INFO:teuthology.orchestra.run.smithi154.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-20T22:35:10.421 INFO:teuthology.orchestra.run.smithi154.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-20T22:35:10.421 INFO:teuthology.orchestra.run.smithi154.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-20T22:35:10.421 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-20T22:35:10.421 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2024-09-20T22:35:10.495 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-20T22:35:10.578 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-20T22:35:10.641 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-20T22:35:10.670 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-mgr-cephadm-2:19.1.1-367.g9e2ac70b.el9.noarch 6/6 2024-09-20T22:35:10.700 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install ceph-mgr-cephadm 2024-09-20T22:35:11.282 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:01:28 ago on Fri 20 Sep 2024 10:33:43 PM UTC. 2024-09-20T22:35:11.418 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2024-09-20T22:35:11.418 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-09-20T22:35:11.419 INFO:teuthology.orchestra.run.smithi154.stdout: Package Arch Version Repository Size 2024-09-20T22:35:11.419 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-09-20T22:35:11.419 INFO:teuthology.orchestra.run.smithi154.stdout:Installing: 2024-09-20T22:35:11.419 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-cephadm noarch 2:19.1.1-367.g9e2ac70b.el9 ceph-noarch 146 k 2024-09-20T22:35:11.419 INFO:teuthology.orchestra.run.smithi154.stdout:Installing dependencies: 2024-09-20T22:35:11.419 INFO:teuthology.orchestra.run.smithi154.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-20T22:35:11.419 INFO:teuthology.orchestra.run.smithi154.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-20T22:35:11.419 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-20T22:35:11.419 INFO:teuthology.orchestra.run.smithi154.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-20T22:35:11.419 INFO:teuthology.orchestra.run.smithi154.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-20T22:35:11.419 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-20T22:35:11.419 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2024-09-20T22:35:11.419 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-09-20T22:35:11.419 INFO:teuthology.orchestra.run.smithi154.stdout:Install 6 Packages 2024-09-20T22:35:11.420 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-20T22:35:11.420 INFO:teuthology.orchestra.run.smithi154.stdout:Total download size: 7.0 M 2024-09-20T22:35:11.420 INFO:teuthology.orchestra.run.smithi154.stdout:Installed size: 33 M 2024-09-20T22:35:11.420 INFO:teuthology.orchestra.run.smithi154.stdout:Downloading Packages: 2024-09-20T22:35:11.466 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.1.1-367.g9e2ac70b.el9.noarch 6/6 2024-09-20T22:35:11.467 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-mgr-cephadm-2:19.1.1-367.g9e2ac70b.el9.noarch 1/6 2024-09-20T22:35:11.467 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-20T22:35:11.467 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-20T22:35:11.467 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-20T22:35:11.467 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-20T22:35:11.744 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-20T22:35:11.744 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-20T22:35:11.744 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2024-09-20T22:35:11.744 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-mgr-cephadm-2:19.1.1-367.g9e2ac70b.el9.noarch 2024-09-20T22:35:11.744 INFO:teuthology.orchestra.run.smithi084.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-20T22:35:11.744 INFO:teuthology.orchestra.run.smithi084.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-20T22:35:11.744 INFO:teuthology.orchestra.run.smithi084.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-20T22:35:11.744 INFO:teuthology.orchestra.run.smithi084.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-20T22:35:11.744 INFO:teuthology.orchestra.run.smithi084.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-20T22:35:11.744 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-20T22:35:11.745 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-09-20T22:35:11.761 INFO:teuthology.orchestra.run.smithi154.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.1 MB/s | 248 kB 00:00 2024-09-20T22:35:11.787 INFO:teuthology.orchestra.run.smithi154.stdout:(2/6): ceph-mgr-cephadm-19.1.1-367.g9e2ac70b.el 579 kB/s | 146 kB 00:00 2024-09-20T22:35:11.837 INFO:teuthology.orchestra.run.smithi154.stdout:(3/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 7.1 MB/s | 548 kB 00:00 2024-09-20T22:35:11.862 INFO:teuthology.orchestra.run.smithi154.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 765 kB/s | 58 kB 00:00 2024-09-20T22:35:11.888 INFO:teuthology.orchestra.run.smithi154.stdout:(5/6): python3-typing-extensions-4.12.2-2.el9.n 1.5 MB/s | 74 kB 00:00 2024-09-20T22:35:12.011 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install ceph-fuse 2024-09-20T22:35:12.603 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:01:28 ago on Fri 20 Sep 2024 10:33:44 PM UTC. 2024-09-20T22:35:12.735 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-09-20T22:35:12.736 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-09-20T22:35:12.736 INFO:teuthology.orchestra.run.smithi084.stdout: Package Arch Version Repository Size 2024-09-20T22:35:12.736 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-09-20T22:35:12.736 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2024-09-20T22:35:12.736 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-fuse x86_64 2:19.1.1-367.g9e2ac70b.el9 ceph 896 k 2024-09-20T22:35:12.736 INFO:teuthology.orchestra.run.smithi084.stdout:Installing dependencies: 2024-09-20T22:35:12.736 INFO:teuthology.orchestra.run.smithi084.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-20T22:35:12.736 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-20T22:35:12.736 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2024-09-20T22:35:12.737 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-09-20T22:35:12.737 INFO:teuthology.orchestra.run.smithi084.stdout:Install 2 Packages 2024-09-20T22:35:12.737 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-20T22:35:12.737 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 976 k 2024-09-20T22:35:12.737 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 2.9 M 2024-09-20T22:35:12.737 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2024-09-20T22:35:12.804 INFO:teuthology.orchestra.run.smithi154.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 4.7 MB/s | 6.0 MB 00:01 2024-09-20T22:35:12.807 INFO:teuthology.orchestra.run.smithi154.stdout:-------------------------------------------------------------------------------- 2024-09-20T22:35:12.807 INFO:teuthology.orchestra.run.smithi154.stdout:Total 5.1 MB/s | 7.0 MB 00:01 2024-09-20T22:35:12.877 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2024-09-20T22:35:12.889 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2024-09-20T22:35:12.889 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2024-09-20T22:35:12.988 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2024-09-20T22:35:12.988 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2024-09-20T22:35:13.108 INFO:teuthology.orchestra.run.smithi084.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 854 kB/s | 80 kB 00:00 2024-09-20T22:35:13.308 INFO:teuthology.orchestra.run.smithi084.stdout:(2/2): ceph-fuse-19.1.1-367.g9e2ac70b.el9.x86_6 3.0 MB/s | 896 kB 00:00 2024-09-20T22:35:13.309 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2024-09-20T22:35:13.309 INFO:teuthology.orchestra.run.smithi084.stdout:Total 1.7 MB/s | 976 kB 00:00 2024-09-20T22:35:13.310 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2024-09-20T22:35:13.320 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2024-09-20T22:35:13.343 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2024-09-20T22:35:13.343 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2024-09-20T22:35:13.425 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2024-09-20T22:35:13.425 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2024-09-20T22:35:13.489 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-20T22:35:13.540 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-20T22:35:13.611 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2024-09-20T22:35:13.743 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-20T22:35:13.815 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-fuse-2:19.1.1-367.g9e2ac70b.el9.x86_64 2/2 2024-09-20T22:35:14.060 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-20T22:35:14.151 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-20T22:35:14.214 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-20T22:35:14.243 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-mgr-cephadm-2:19.1.1-367.g9e2ac70b.el9.noarch 6/6 2024-09-20T22:35:14.486 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-fuse-2:19.1.1-367.g9e2ac70b.el9.x86_64 2/2 2024-09-20T22:35:14.486 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-fuse-2:19.1.1-367.g9e2ac70b.el9.x86_64 1/2 2024-09-20T22:35:14.779 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-20T22:35:14.779 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-20T22:35:14.779 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2024-09-20T22:35:14.779 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-fuse-2:19.1.1-367.g9e2ac70b.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-20T22:35:14.780 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-20T22:35:14.780 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-09-20T22:35:15.020 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install ceph-volume 2024-09-20T22:35:15.063 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.1.1-367.g9e2ac70b.el9.noarch 6/6 2024-09-20T22:35:15.064 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-mgr-cephadm-2:19.1.1-367.g9e2ac70b.el9.noarch 1/6 2024-09-20T22:35:15.064 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-20T22:35:15.064 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-20T22:35:15.064 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-20T22:35:15.064 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-20T22:35:15.343 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-20T22:35:15.343 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-20T22:35:15.343 INFO:teuthology.orchestra.run.smithi154.stdout:Installed: 2024-09-20T22:35:15.343 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-mgr-cephadm-2:19.1.1-367.g9e2ac70b.el9.noarch 2024-09-20T22:35:15.343 INFO:teuthology.orchestra.run.smithi154.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-20T22:35:15.343 INFO:teuthology.orchestra.run.smithi154.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-20T22:35:15.343 INFO:teuthology.orchestra.run.smithi154.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-20T22:35:15.343 INFO:teuthology.orchestra.run.smithi154.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-20T22:35:15.343 INFO:teuthology.orchestra.run.smithi154.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-20T22:35:15.343 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-20T22:35:15.343 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2024-09-20T22:35:15.605 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:01:31 ago on Fri 20 Sep 2024 10:33:44 PM UTC. 2024-09-20T22:35:15.621 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install ceph-fuse 2024-09-20T22:35:15.743 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-09-20T22:35:15.743 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-09-20T22:35:15.743 INFO:teuthology.orchestra.run.smithi084.stdout: Package Arch Version Repository Size 2024-09-20T22:35:15.743 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-09-20T22:35:15.744 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2024-09-20T22:35:15.744 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-volume noarch 2:19.1.1-367.g9e2ac70b.el9 ceph-noarch 285 k 2024-09-20T22:35:15.744 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-20T22:35:15.744 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2024-09-20T22:35:15.744 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-09-20T22:35:15.744 INFO:teuthology.orchestra.run.smithi084.stdout:Install 1 Package 2024-09-20T22:35:15.744 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-20T22:35:15.744 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 285 k 2024-09-20T22:35:15.744 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 1.3 M 2024-09-20T22:35:15.744 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2024-09-20T22:35:15.979 INFO:teuthology.orchestra.run.smithi084.stdout:ceph-volume-19.1.1-367.g9e2ac70b.el9.noarch.rpm 1.2 MB/s | 285 kB 00:00 2024-09-20T22:35:15.979 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2024-09-20T22:35:15.980 INFO:teuthology.orchestra.run.smithi084.stdout:Total 1.2 MB/s | 285 kB 00:00 2024-09-20T22:35:15.980 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2024-09-20T22:35:15.988 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2024-09-20T22:35:15.988 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2024-09-20T22:35:16.020 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2024-09-20T22:35:16.020 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2024-09-20T22:35:16.202 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:01:33 ago on Fri 20 Sep 2024 10:33:43 PM UTC. 2024-09-20T22:35:16.235 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2024-09-20T22:35:16.305 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : ceph-volume-2:19.1.1-367.g9e2ac70b.el9.noarch 1/1 2024-09-20T22:35:16.321 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: ceph-volume-2:19.1.1-367.g9e2ac70b.el9.noarch 1/1 2024-09-20T22:35:16.321 INFO:teuthology.orchestra.run.smithi084.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-20T22:35:16.321 INFO:teuthology.orchestra.run.smithi084.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-20T22:35:16.321 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-20T22:35:16.335 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2024-09-20T22:35:16.336 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-09-20T22:35:16.336 INFO:teuthology.orchestra.run.smithi154.stdout: Package Arch Version Repository Size 2024-09-20T22:35:16.336 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-09-20T22:35:16.336 INFO:teuthology.orchestra.run.smithi154.stdout:Installing: 2024-09-20T22:35:16.336 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-fuse x86_64 2:19.1.1-367.g9e2ac70b.el9 ceph 896 k 2024-09-20T22:35:16.336 INFO:teuthology.orchestra.run.smithi154.stdout:Installing dependencies: 2024-09-20T22:35:16.336 INFO:teuthology.orchestra.run.smithi154.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-20T22:35:16.336 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-20T22:35:16.336 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2024-09-20T22:35:16.336 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-09-20T22:35:16.337 INFO:teuthology.orchestra.run.smithi154.stdout:Install 2 Packages 2024-09-20T22:35:16.337 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-20T22:35:16.337 INFO:teuthology.orchestra.run.smithi154.stdout:Total download size: 976 k 2024-09-20T22:35:16.337 INFO:teuthology.orchestra.run.smithi154.stdout:Installed size: 2.9 M 2024-09-20T22:35:16.337 INFO:teuthology.orchestra.run.smithi154.stdout:Downloading Packages: 2024-09-20T22:35:16.531 INFO:teuthology.orchestra.run.smithi154.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 554 kB/s | 80 kB 00:00 2024-09-20T22:35:16.656 INFO:teuthology.orchestra.run.smithi154.stdout:(2/2): ceph-fuse-19.1.1-367.g9e2ac70b.el9.x86_6 3.3 MB/s | 896 kB 00:00 2024-09-20T22:35:16.657 INFO:teuthology.orchestra.run.smithi154.stdout:-------------------------------------------------------------------------------- 2024-09-20T22:35:16.657 INFO:teuthology.orchestra.run.smithi154.stdout:Total 3.0 MB/s | 976 kB 00:00 2024-09-20T22:35:16.669 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2024-09-20T22:35:16.691 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2024-09-20T22:35:16.691 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2024-09-20T22:35:16.772 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2024-09-20T22:35:16.772 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2024-09-20T22:35:16.955 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2024-09-20T22:35:17.091 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-20T22:35:17.155 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-fuse-2:19.1.1-367.g9e2ac70b.el9.x86_64 2/2 2024-09-20T22:35:17.318 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : ceph-volume-2:19.1.1-367.g9e2ac70b.el9.noarch 1/1 2024-09-20T22:35:17.318 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-20T22:35:17.318 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2024-09-20T22:35:17.319 INFO:teuthology.orchestra.run.smithi084.stdout: ceph-volume-2:19.1.1-367.g9e2ac70b.el9.noarch 2024-09-20T22:35:17.319 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-20T22:35:17.319 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-09-20T22:35:17.622 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install librados-devel 2024-09-20T22:35:17.862 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-fuse-2:19.1.1-367.g9e2ac70b.el9.x86_64 2/2 2024-09-20T22:35:17.862 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-fuse-2:19.1.1-367.g9e2ac70b.el9.x86_64 1/2 2024-09-20T22:35:18.144 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-20T22:35:18.144 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-20T22:35:18.144 INFO:teuthology.orchestra.run.smithi154.stdout:Installed: 2024-09-20T22:35:18.144 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-fuse-2:19.1.1-367.g9e2ac70b.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-20T22:35:18.144 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-20T22:35:18.144 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2024-09-20T22:35:18.222 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:01:34 ago on Fri 20 Sep 2024 10:33:44 PM UTC. 2024-09-20T22:35:18.355 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-09-20T22:35:18.355 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-09-20T22:35:18.355 INFO:teuthology.orchestra.run.smithi084.stdout: Package Arch Version Repo Size 2024-09-20T22:35:18.355 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-09-20T22:35:18.356 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2024-09-20T22:35:18.356 INFO:teuthology.orchestra.run.smithi084.stdout: librados-devel x86_64 2:19.1.1-367.g9e2ac70b.el9 ceph 127 k 2024-09-20T22:35:18.356 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-20T22:35:18.356 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2024-09-20T22:35:18.356 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-09-20T22:35:18.356 INFO:teuthology.orchestra.run.smithi084.stdout:Install 1 Package 2024-09-20T22:35:18.356 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-20T22:35:18.356 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 127 k 2024-09-20T22:35:18.356 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 456 k 2024-09-20T22:35:18.357 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2024-09-20T22:35:18.446 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install ceph-volume 2024-09-20T22:35:18.591 INFO:teuthology.orchestra.run.smithi084.stdout:librados-devel-19.1.1-367.g9e2ac70b.el9.x86_64. 540 kB/s | 127 kB 00:00 2024-09-20T22:35:18.592 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2024-09-20T22:35:18.592 INFO:teuthology.orchestra.run.smithi084.stdout:Total 536 kB/s | 127 kB 00:00 2024-09-20T22:35:18.592 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2024-09-20T22:35:18.597 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2024-09-20T22:35:18.598 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2024-09-20T22:35:18.662 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2024-09-20T22:35:18.662 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2024-09-20T22:35:18.825 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2024-09-20T22:35:18.922 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : librados-devel-2:19.1.1-367.g9e2ac70b.el9.x86_64 1/1 2024-09-20T22:35:19.048 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:01:36 ago on Fri 20 Sep 2024 10:33:43 PM UTC. 2024-09-20T22:35:19.187 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2024-09-20T22:35:19.188 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-09-20T22:35:19.188 INFO:teuthology.orchestra.run.smithi154.stdout: Package Arch Version Repository Size 2024-09-20T22:35:19.188 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-09-20T22:35:19.189 INFO:teuthology.orchestra.run.smithi154.stdout:Installing: 2024-09-20T22:35:19.189 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-volume noarch 2:19.1.1-367.g9e2ac70b.el9 ceph-noarch 285 k 2024-09-20T22:35:19.189 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-20T22:35:19.189 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2024-09-20T22:35:19.189 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-09-20T22:35:19.189 INFO:teuthology.orchestra.run.smithi154.stdout:Install 1 Package 2024-09-20T22:35:19.189 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-20T22:35:19.189 INFO:teuthology.orchestra.run.smithi154.stdout:Total download size: 285 k 2024-09-20T22:35:19.189 INFO:teuthology.orchestra.run.smithi154.stdout:Installed size: 1.3 M 2024-09-20T22:35:19.189 INFO:teuthology.orchestra.run.smithi154.stdout:Downloading Packages: 2024-09-20T22:35:19.359 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: librados-devel-2:19.1.1-367.g9e2ac70b.el9.x86_64 1/1 2024-09-20T22:35:19.420 INFO:teuthology.orchestra.run.smithi154.stdout:ceph-volume-19.1.1-367.g9e2ac70b.el9.noarch.rpm 1.2 MB/s | 285 kB 00:00 2024-09-20T22:35:19.420 INFO:teuthology.orchestra.run.smithi154.stdout:-------------------------------------------------------------------------------- 2024-09-20T22:35:19.421 INFO:teuthology.orchestra.run.smithi154.stdout:Total 1.2 MB/s | 285 kB 00:00 2024-09-20T22:35:19.421 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2024-09-20T22:35:19.429 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2024-09-20T22:35:19.429 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2024-09-20T22:35:19.459 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2024-09-20T22:35:19.459 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2024-09-20T22:35:19.574 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : librados-devel-2:19.1.1-367.g9e2ac70b.el9.x86_64 1/1 2024-09-20T22:35:19.574 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-20T22:35:19.574 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2024-09-20T22:35:19.575 INFO:teuthology.orchestra.run.smithi084.stdout: librados-devel-2:19.1.1-367.g9e2ac70b.el9.x86_64 2024-09-20T22:35:19.575 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-20T22:35:19.575 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-09-20T22:35:19.681 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2024-09-20T22:35:19.747 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : ceph-volume-2:19.1.1-367.g9e2ac70b.el9.noarch 1/1 2024-09-20T22:35:19.764 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: ceph-volume-2:19.1.1-367.g9e2ac70b.el9.noarch 1/1 2024-09-20T22:35:19.764 INFO:teuthology.orchestra.run.smithi154.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-20T22:35:19.764 INFO:teuthology.orchestra.run.smithi154.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-20T22:35:19.764 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-20T22:35:19.867 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install libcephfs2 2024-09-20T22:35:20.462 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:01:36 ago on Fri 20 Sep 2024 10:33:44 PM UTC. 2024-09-20T22:35:20.552 INFO:teuthology.orchestra.run.smithi084.stdout:Package libcephfs2-2:19.1.1-367.g9e2ac70b.el9.x86_64 is already installed. 2024-09-20T22:35:20.596 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-09-20T22:35:20.597 INFO:teuthology.orchestra.run.smithi084.stdout:Nothing to do. 2024-09-20T22:35:20.597 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-09-20T22:35:20.662 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install libcephfs-devel 2024-09-20T22:35:20.856 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : ceph-volume-2:19.1.1-367.g9e2ac70b.el9.noarch 1/1 2024-09-20T22:35:20.857 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-20T22:35:20.857 INFO:teuthology.orchestra.run.smithi154.stdout:Installed: 2024-09-20T22:35:20.857 INFO:teuthology.orchestra.run.smithi154.stdout: ceph-volume-2:19.1.1-367.g9e2ac70b.el9.noarch 2024-09-20T22:35:20.857 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-20T22:35:20.857 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2024-09-20T22:35:21.178 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install librados-devel 2024-09-20T22:35:21.244 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:01:37 ago on Fri 20 Sep 2024 10:33:44 PM UTC. 2024-09-20T22:35:21.375 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-09-20T22:35:21.376 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-09-20T22:35:21.376 INFO:teuthology.orchestra.run.smithi084.stdout: Package Arch Version Repo Size 2024-09-20T22:35:21.376 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-09-20T22:35:21.376 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2024-09-20T22:35:21.376 INFO:teuthology.orchestra.run.smithi084.stdout: libcephfs-devel x86_64 2:19.1.1-367.g9e2ac70b.el9 ceph 32 k 2024-09-20T22:35:21.377 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-20T22:35:21.377 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2024-09-20T22:35:21.377 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-09-20T22:35:21.377 INFO:teuthology.orchestra.run.smithi084.stdout:Install 1 Package 2024-09-20T22:35:21.377 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-20T22:35:21.377 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 32 k 2024-09-20T22:35:21.377 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 142 k 2024-09-20T22:35:21.377 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2024-09-20T22:35:21.519 INFO:teuthology.orchestra.run.smithi084.stdout:libcephfs-devel-19.1.1-367.g9e2ac70b.el9.x86_64 224 kB/s | 32 kB 00:00 2024-09-20T22:35:21.519 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2024-09-20T22:35:21.520 INFO:teuthology.orchestra.run.smithi084.stdout:Total 222 kB/s | 32 kB 00:00 2024-09-20T22:35:21.520 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2024-09-20T22:35:21.523 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2024-09-20T22:35:21.523 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2024-09-20T22:35:21.539 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2024-09-20T22:35:21.539 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2024-09-20T22:35:21.622 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2024-09-20T22:35:21.724 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : libcephfs-devel-2:19.1.1-367.g9e2ac70b.el9.x86_64 1/1 2024-09-20T22:35:21.773 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:01:38 ago on Fri 20 Sep 2024 10:33:43 PM UTC. 2024-09-20T22:35:21.907 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2024-09-20T22:35:21.907 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-09-20T22:35:21.907 INFO:teuthology.orchestra.run.smithi154.stdout: Package Arch Version Repo Size 2024-09-20T22:35:21.908 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-09-20T22:35:21.908 INFO:teuthology.orchestra.run.smithi154.stdout:Installing: 2024-09-20T22:35:21.908 INFO:teuthology.orchestra.run.smithi154.stdout: librados-devel x86_64 2:19.1.1-367.g9e2ac70b.el9 ceph 127 k 2024-09-20T22:35:21.908 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-20T22:35:21.908 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2024-09-20T22:35:21.908 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-09-20T22:35:21.908 INFO:teuthology.orchestra.run.smithi154.stdout:Install 1 Package 2024-09-20T22:35:21.908 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-20T22:35:21.908 INFO:teuthology.orchestra.run.smithi154.stdout:Total download size: 127 k 2024-09-20T22:35:21.908 INFO:teuthology.orchestra.run.smithi154.stdout:Installed size: 456 k 2024-09-20T22:35:21.908 INFO:teuthology.orchestra.run.smithi154.stdout:Downloading Packages: 2024-09-20T22:35:22.081 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: libcephfs-devel-2:19.1.1-367.g9e2ac70b.el9.x86_64 1/1 2024-09-20T22:35:22.125 INFO:teuthology.orchestra.run.smithi154.stdout:librados-devel-19.1.1-367.g9e2ac70b.el9.x86_64. 584 kB/s | 127 kB 00:00 2024-09-20T22:35:22.126 INFO:teuthology.orchestra.run.smithi154.stdout:-------------------------------------------------------------------------------- 2024-09-20T22:35:22.126 INFO:teuthology.orchestra.run.smithi154.stdout:Total 580 kB/s | 127 kB 00:00 2024-09-20T22:35:22.126 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2024-09-20T22:35:22.131 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2024-09-20T22:35:22.131 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2024-09-20T22:35:22.197 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2024-09-20T22:35:22.197 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2024-09-20T22:35:22.305 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : libcephfs-devel-2:19.1.1-367.g9e2ac70b.el9.x86_64 1/1 2024-09-20T22:35:22.305 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-20T22:35:22.305 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2024-09-20T22:35:22.305 INFO:teuthology.orchestra.run.smithi084.stdout: libcephfs-devel-2:19.1.1-367.g9e2ac70b.el9.x86_64 2024-09-20T22:35:22.305 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-20T22:35:22.305 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-09-20T22:35:22.355 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2024-09-20T22:35:22.464 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : librados-devel-2:19.1.1-367.g9e2ac70b.el9.x86_64 1/1 2024-09-20T22:35:22.506 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install librados2 2024-09-20T22:35:22.921 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: librados-devel-2:19.1.1-367.g9e2ac70b.el9.x86_64 1/1 2024-09-20T22:35:23.093 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:01:39 ago on Fri 20 Sep 2024 10:33:44 PM UTC. 2024-09-20T22:35:23.175 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : librados-devel-2:19.1.1-367.g9e2ac70b.el9.x86_64 1/1 2024-09-20T22:35:23.175 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-20T22:35:23.175 INFO:teuthology.orchestra.run.smithi154.stdout:Installed: 2024-09-20T22:35:23.175 INFO:teuthology.orchestra.run.smithi154.stdout: librados-devel-2:19.1.1-367.g9e2ac70b.el9.x86_64 2024-09-20T22:35:23.175 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-20T22:35:23.175 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2024-09-20T22:35:23.182 INFO:teuthology.orchestra.run.smithi084.stdout:Package librados2-2:19.1.1-367.g9e2ac70b.el9.x86_64 is already installed. 2024-09-20T22:35:23.225 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-09-20T22:35:23.226 INFO:teuthology.orchestra.run.smithi084.stdout:Nothing to do. 2024-09-20T22:35:23.226 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-09-20T22:35:23.285 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install librbd1 2024-09-20T22:35:23.512 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install libcephfs2 2024-09-20T22:35:23.867 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:01:39 ago on Fri 20 Sep 2024 10:33:44 PM UTC. 2024-09-20T22:35:23.957 INFO:teuthology.orchestra.run.smithi084.stdout:Package librbd1-2:19.1.1-367.g9e2ac70b.el9.x86_64 is already installed. 2024-09-20T22:35:24.001 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-09-20T22:35:24.002 INFO:teuthology.orchestra.run.smithi084.stdout:Nothing to do. 2024-09-20T22:35:24.002 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-09-20T22:35:24.057 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install python3-rados 2024-09-20T22:35:24.123 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:01:41 ago on Fri 20 Sep 2024 10:33:43 PM UTC. 2024-09-20T22:35:24.215 INFO:teuthology.orchestra.run.smithi154.stdout:Package libcephfs2-2:19.1.1-367.g9e2ac70b.el9.x86_64 is already installed. 2024-09-20T22:35:24.261 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2024-09-20T22:35:24.262 INFO:teuthology.orchestra.run.smithi154.stdout:Nothing to do. 2024-09-20T22:35:24.262 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2024-09-20T22:35:24.354 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install libcephfs-devel 2024-09-20T22:35:24.641 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:01:40 ago on Fri 20 Sep 2024 10:33:44 PM UTC. 2024-09-20T22:35:24.731 INFO:teuthology.orchestra.run.smithi084.stdout:Package python3-rados-2:19.1.1-367.g9e2ac70b.el9.x86_64 is already installed. 2024-09-20T22:35:24.774 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-09-20T22:35:24.775 INFO:teuthology.orchestra.run.smithi084.stdout:Nothing to do. 2024-09-20T22:35:24.775 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-09-20T22:35:24.833 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install python3-rgw 2024-09-20T22:35:24.932 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:01:41 ago on Fri 20 Sep 2024 10:33:43 PM UTC. 2024-09-20T22:35:25.067 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2024-09-20T22:35:25.067 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-09-20T22:35:25.067 INFO:teuthology.orchestra.run.smithi154.stdout: Package Arch Version Repo Size 2024-09-20T22:35:25.067 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-09-20T22:35:25.067 INFO:teuthology.orchestra.run.smithi154.stdout:Installing: 2024-09-20T22:35:25.067 INFO:teuthology.orchestra.run.smithi154.stdout: libcephfs-devel x86_64 2:19.1.1-367.g9e2ac70b.el9 ceph 32 k 2024-09-20T22:35:25.067 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-20T22:35:25.067 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2024-09-20T22:35:25.068 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-09-20T22:35:25.068 INFO:teuthology.orchestra.run.smithi154.stdout:Install 1 Package 2024-09-20T22:35:25.068 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-20T22:35:25.068 INFO:teuthology.orchestra.run.smithi154.stdout:Total download size: 32 k 2024-09-20T22:35:25.068 INFO:teuthology.orchestra.run.smithi154.stdout:Installed size: 142 k 2024-09-20T22:35:25.068 INFO:teuthology.orchestra.run.smithi154.stdout:Downloading Packages: 2024-09-20T22:35:25.212 INFO:teuthology.orchestra.run.smithi154.stdout:libcephfs-devel-19.1.1-367.g9e2ac70b.el9.x86_64 219 kB/s | 32 kB 00:00 2024-09-20T22:35:25.213 INFO:teuthology.orchestra.run.smithi154.stdout:-------------------------------------------------------------------------------- 2024-09-20T22:35:25.213 INFO:teuthology.orchestra.run.smithi154.stdout:Total 217 kB/s | 32 kB 00:00 2024-09-20T22:35:25.213 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2024-09-20T22:35:25.217 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2024-09-20T22:35:25.217 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2024-09-20T22:35:25.248 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2024-09-20T22:35:25.248 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2024-09-20T22:35:25.334 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2024-09-20T22:35:25.409 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:01:41 ago on Fri 20 Sep 2024 10:33:44 PM UTC. 2024-09-20T22:35:25.460 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : libcephfs-devel-2:19.1.1-367.g9e2ac70b.el9.x86_64 1/1 2024-09-20T22:35:25.500 INFO:teuthology.orchestra.run.smithi084.stdout:Package python3-rgw-2:19.1.1-367.g9e2ac70b.el9.x86_64 is already installed. 2024-09-20T22:35:25.543 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-09-20T22:35:25.544 INFO:teuthology.orchestra.run.smithi084.stdout:Nothing to do. 2024-09-20T22:35:25.544 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-09-20T22:35:25.601 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install python3-cephfs 2024-09-20T22:35:25.826 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: libcephfs-devel-2:19.1.1-367.g9e2ac70b.el9.x86_64 1/1 2024-09-20T22:35:26.065 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : libcephfs-devel-2:19.1.1-367.g9e2ac70b.el9.x86_64 1/1 2024-09-20T22:35:26.065 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-20T22:35:26.065 INFO:teuthology.orchestra.run.smithi154.stdout:Installed: 2024-09-20T22:35:26.066 INFO:teuthology.orchestra.run.smithi154.stdout: libcephfs-devel-2:19.1.1-367.g9e2ac70b.el9.x86_64 2024-09-20T22:35:26.066 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-20T22:35:26.066 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2024-09-20T22:35:26.189 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:01:42 ago on Fri 20 Sep 2024 10:33:44 PM UTC. 2024-09-20T22:35:26.268 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install librados2 2024-09-20T22:35:26.279 INFO:teuthology.orchestra.run.smithi084.stdout:Package python3-cephfs-2:19.1.1-367.g9e2ac70b.el9.x86_64 is already installed. 2024-09-20T22:35:26.323 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-09-20T22:35:26.324 INFO:teuthology.orchestra.run.smithi084.stdout:Nothing to do. 2024-09-20T22:35:26.324 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-09-20T22:35:26.395 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install python3-rbd 2024-09-20T22:35:26.855 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:01:43 ago on Fri 20 Sep 2024 10:33:43 PM UTC. 2024-09-20T22:35:26.944 INFO:teuthology.orchestra.run.smithi154.stdout:Package librados2-2:19.1.1-367.g9e2ac70b.el9.x86_64 is already installed. 2024-09-20T22:35:26.970 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:01:42 ago on Fri 20 Sep 2024 10:33:44 PM UTC. 2024-09-20T22:35:26.987 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2024-09-20T22:35:26.989 INFO:teuthology.orchestra.run.smithi154.stdout:Nothing to do. 2024-09-20T22:35:26.989 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2024-09-20T22:35:27.060 INFO:teuthology.orchestra.run.smithi084.stdout:Package python3-rbd-2:19.1.1-367.g9e2ac70b.el9.x86_64 is already installed. 2024-09-20T22:35:27.077 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install librbd1 2024-09-20T22:35:27.104 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-09-20T22:35:27.105 INFO:teuthology.orchestra.run.smithi084.stdout:Nothing to do. 2024-09-20T22:35:27.105 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-09-20T22:35:27.165 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install rbd-fuse 2024-09-20T22:35:27.664 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:01:44 ago on Fri 20 Sep 2024 10:33:43 PM UTC. 2024-09-20T22:35:27.740 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:01:43 ago on Fri 20 Sep 2024 10:33:44 PM UTC. 2024-09-20T22:35:27.754 INFO:teuthology.orchestra.run.smithi154.stdout:Package librbd1-2:19.1.1-367.g9e2ac70b.el9.x86_64 is already installed. 2024-09-20T22:35:27.798 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2024-09-20T22:35:27.799 INFO:teuthology.orchestra.run.smithi154.stdout:Nothing to do. 2024-09-20T22:35:27.799 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2024-09-20T22:35:27.872 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-09-20T22:35:27.872 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-09-20T22:35:27.873 INFO:teuthology.orchestra.run.smithi084.stdout: Package Architecture Version Repository Size 2024-09-20T22:35:27.873 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-09-20T22:35:27.873 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2024-09-20T22:35:27.873 INFO:teuthology.orchestra.run.smithi084.stdout: rbd-fuse x86_64 2:19.1.1-367.g9e2ac70b.el9 ceph 85 k 2024-09-20T22:35:27.873 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-20T22:35:27.873 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2024-09-20T22:35:27.873 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-09-20T22:35:27.873 INFO:teuthology.orchestra.run.smithi084.stdout:Install 1 Package 2024-09-20T22:35:27.873 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-20T22:35:27.873 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 85 k 2024-09-20T22:35:27.873 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 231 k 2024-09-20T22:35:27.873 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2024-09-20T22:35:27.885 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install python3-rados 2024-09-20T22:35:28.063 INFO:teuthology.orchestra.run.smithi084.stdout:rbd-fuse-19.1.1-367.g9e2ac70b.el9.x86_64.rpm 450 kB/s | 85 kB 00:00 2024-09-20T22:35:28.063 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2024-09-20T22:35:28.064 INFO:teuthology.orchestra.run.smithi084.stdout:Total 447 kB/s | 85 kB 00:00 2024-09-20T22:35:28.064 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2024-09-20T22:35:28.070 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2024-09-20T22:35:28.071 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2024-09-20T22:35:28.135 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2024-09-20T22:35:28.135 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2024-09-20T22:35:28.283 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2024-09-20T22:35:28.385 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : rbd-fuse-2:19.1.1-367.g9e2ac70b.el9.x86_64 1/1 2024-09-20T22:35:28.466 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:01:45 ago on Fri 20 Sep 2024 10:33:43 PM UTC. 2024-09-20T22:35:28.555 INFO:teuthology.orchestra.run.smithi154.stdout:Package python3-rados-2:19.1.1-367.g9e2ac70b.el9.x86_64 is already installed. 2024-09-20T22:35:28.598 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2024-09-20T22:35:28.599 INFO:teuthology.orchestra.run.smithi154.stdout:Nothing to do. 2024-09-20T22:35:28.599 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2024-09-20T22:35:28.686 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install python3-rgw 2024-09-20T22:35:28.775 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: rbd-fuse-2:19.1.1-367.g9e2ac70b.el9.x86_64 1/1 2024-09-20T22:35:29.015 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : rbd-fuse-2:19.1.1-367.g9e2ac70b.el9.x86_64 1/1 2024-09-20T22:35:29.015 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-20T22:35:29.015 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2024-09-20T22:35:29.015 INFO:teuthology.orchestra.run.smithi084.stdout: rbd-fuse-2:19.1.1-367.g9e2ac70b.el9.x86_64 2024-09-20T22:35:29.015 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-20T22:35:29.015 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-09-20T22:35:29.283 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install rbd-mirror 2024-09-20T22:35:29.337 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:01:46 ago on Fri 20 Sep 2024 10:33:43 PM UTC. 2024-09-20T22:35:29.427 INFO:teuthology.orchestra.run.smithi154.stdout:Package python3-rgw-2:19.1.1-367.g9e2ac70b.el9.x86_64 is already installed. 2024-09-20T22:35:29.470 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2024-09-20T22:35:29.471 INFO:teuthology.orchestra.run.smithi154.stdout:Nothing to do. 2024-09-20T22:35:29.471 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2024-09-20T22:35:29.561 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install python3-cephfs 2024-09-20T22:35:29.895 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:01:45 ago on Fri 20 Sep 2024 10:33:44 PM UTC. 2024-09-20T22:35:30.030 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-09-20T22:35:30.031 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-09-20T22:35:30.031 INFO:teuthology.orchestra.run.smithi084.stdout: Package Arch Version Repo Size 2024-09-20T22:35:30.031 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-09-20T22:35:30.031 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2024-09-20T22:35:30.031 INFO:teuthology.orchestra.run.smithi084.stdout: rbd-mirror x86_64 2:19.1.1-367.g9e2ac70b.el9 ceph 3.1 M 2024-09-20T22:35:30.031 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-20T22:35:30.031 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2024-09-20T22:35:30.032 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-09-20T22:35:30.032 INFO:teuthology.orchestra.run.smithi084.stdout:Install 1 Package 2024-09-20T22:35:30.032 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-20T22:35:30.032 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 3.1 M 2024-09-20T22:35:30.032 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 13 M 2024-09-20T22:35:30.032 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2024-09-20T22:35:30.143 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:01:47 ago on Fri 20 Sep 2024 10:33:43 PM UTC. 2024-09-20T22:35:30.233 INFO:teuthology.orchestra.run.smithi154.stdout:Package python3-cephfs-2:19.1.1-367.g9e2ac70b.el9.x86_64 is already installed. 2024-09-20T22:35:30.276 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2024-09-20T22:35:30.277 INFO:teuthology.orchestra.run.smithi154.stdout:Nothing to do. 2024-09-20T22:35:30.278 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2024-09-20T22:35:30.374 INFO:teuthology.orchestra.run.smithi084.stdout:rbd-mirror-19.1.1-367.g9e2ac70b.el9.x86_64.rpm 9.1 MB/s | 3.1 MB 00:00 2024-09-20T22:35:30.375 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2024-09-20T22:35:30.376 INFO:teuthology.orchestra.run.smithi084.stdout:Total 9.0 MB/s | 3.1 MB 00:00 2024-09-20T22:35:30.376 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2024-09-20T22:35:30.378 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install python3-rbd 2024-09-20T22:35:30.384 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2024-09-20T22:35:30.384 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2024-09-20T22:35:30.466 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2024-09-20T22:35:30.467 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2024-09-20T22:35:30.896 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2024-09-20T22:35:30.950 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:01:47 ago on Fri 20 Sep 2024 10:33:43 PM UTC. 2024-09-20T22:35:30.958 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : rbd-mirror-2:19.1.1-367.g9e2ac70b.el9.x86_64 1/1 2024-09-20T22:35:30.984 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: rbd-mirror-2:19.1.1-367.g9e2ac70b.el9.x86_64 1/1 2024-09-20T22:35:30.985 INFO:teuthology.orchestra.run.smithi084.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-20T22:35:30.985 INFO:teuthology.orchestra.run.smithi084.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-20T22:35:30.985 INFO:teuthology.orchestra.run.smithi084.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-20T22:35:30.985 INFO:teuthology.orchestra.run.smithi084.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-20T22:35:30.985 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-20T22:35:31.041 INFO:teuthology.orchestra.run.smithi154.stdout:Package python3-rbd-2:19.1.1-367.g9e2ac70b.el9.x86_64 is already installed. 2024-09-20T22:35:31.084 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2024-09-20T22:35:31.085 INFO:teuthology.orchestra.run.smithi154.stdout:Nothing to do. 2024-09-20T22:35:31.085 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2024-09-20T22:35:31.178 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install rbd-fuse 2024-09-20T22:35:31.761 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:01:48 ago on Fri 20 Sep 2024 10:33:43 PM UTC. 2024-09-20T22:35:31.895 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2024-09-20T22:35:31.896 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-09-20T22:35:31.896 INFO:teuthology.orchestra.run.smithi154.stdout: Package Architecture Version Repository Size 2024-09-20T22:35:31.896 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-09-20T22:35:31.896 INFO:teuthology.orchestra.run.smithi154.stdout:Installing: 2024-09-20T22:35:31.896 INFO:teuthology.orchestra.run.smithi154.stdout: rbd-fuse x86_64 2:19.1.1-367.g9e2ac70b.el9 ceph 85 k 2024-09-20T22:35:31.896 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-20T22:35:31.896 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2024-09-20T22:35:31.896 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-09-20T22:35:31.896 INFO:teuthology.orchestra.run.smithi154.stdout:Install 1 Package 2024-09-20T22:35:31.896 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-20T22:35:31.897 INFO:teuthology.orchestra.run.smithi154.stdout:Total download size: 85 k 2024-09-20T22:35:31.897 INFO:teuthology.orchestra.run.smithi154.stdout:Installed size: 231 k 2024-09-20T22:35:31.897 INFO:teuthology.orchestra.run.smithi154.stdout:Downloading Packages: 2024-09-20T22:35:32.065 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : rbd-mirror-2:19.1.1-367.g9e2ac70b.el9.x86_64 1/1 2024-09-20T22:35:32.065 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-20T22:35:32.065 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2024-09-20T22:35:32.065 INFO:teuthology.orchestra.run.smithi084.stdout: rbd-mirror-2:19.1.1-367.g9e2ac70b.el9.x86_64 2024-09-20T22:35:32.065 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-20T22:35:32.065 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-09-20T22:35:32.090 INFO:teuthology.orchestra.run.smithi154.stdout:rbd-fuse-19.1.1-367.g9e2ac70b.el9.x86_64.rpm 443 kB/s | 85 kB 00:00 2024-09-20T22:35:32.091 INFO:teuthology.orchestra.run.smithi154.stdout:-------------------------------------------------------------------------------- 2024-09-20T22:35:32.091 INFO:teuthology.orchestra.run.smithi154.stdout:Total 438 kB/s | 85 kB 00:00 2024-09-20T22:35:32.091 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2024-09-20T22:35:32.098 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2024-09-20T22:35:32.098 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2024-09-20T22:35:32.162 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2024-09-20T22:35:32.162 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2024-09-20T22:35:32.251 DEBUG:teuthology.orchestra.run.smithi084:> sudo yum -y install rbd-nbd 2024-09-20T22:35:32.306 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2024-09-20T22:35:32.421 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : rbd-fuse-2:19.1.1-367.g9e2ac70b.el9.x86_64 1/1 2024-09-20T22:35:32.839 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: rbd-fuse-2:19.1.1-367.g9e2ac70b.el9.x86_64 1/1 2024-09-20T22:35:32.851 INFO:teuthology.orchestra.run.smithi084.stdout:Last metadata expiration check: 0:01:48 ago on Fri 20 Sep 2024 10:33:44 PM UTC. 2024-09-20T22:35:32.983 INFO:teuthology.orchestra.run.smithi084.stdout:Dependencies resolved. 2024-09-20T22:35:32.984 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-09-20T22:35:32.984 INFO:teuthology.orchestra.run.smithi084.stdout: Package Architecture Version Repository Size 2024-09-20T22:35:32.984 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-09-20T22:35:32.984 INFO:teuthology.orchestra.run.smithi084.stdout:Installing: 2024-09-20T22:35:32.984 INFO:teuthology.orchestra.run.smithi084.stdout: rbd-nbd x86_64 2:19.1.1-367.g9e2ac70b.el9 ceph 170 k 2024-09-20T22:35:32.984 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-20T22:35:32.984 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction Summary 2024-09-20T22:35:32.984 INFO:teuthology.orchestra.run.smithi084.stdout:================================================================================ 2024-09-20T22:35:32.984 INFO:teuthology.orchestra.run.smithi084.stdout:Install 1 Package 2024-09-20T22:35:32.984 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-20T22:35:32.984 INFO:teuthology.orchestra.run.smithi084.stdout:Total download size: 170 k 2024-09-20T22:35:32.985 INFO:teuthology.orchestra.run.smithi084.stdout:Installed size: 494 k 2024-09-20T22:35:32.985 INFO:teuthology.orchestra.run.smithi084.stdout:Downloading Packages: 2024-09-20T22:35:33.075 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : rbd-fuse-2:19.1.1-367.g9e2ac70b.el9.x86_64 1/1 2024-09-20T22:35:33.075 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-20T22:35:33.075 INFO:teuthology.orchestra.run.smithi154.stdout:Installed: 2024-09-20T22:35:33.075 INFO:teuthology.orchestra.run.smithi154.stdout: rbd-fuse-2:19.1.1-367.g9e2ac70b.el9.x86_64 2024-09-20T22:35:33.075 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-20T22:35:33.076 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2024-09-20T22:35:33.194 INFO:teuthology.orchestra.run.smithi084.stdout:rbd-nbd-19.1.1-367.g9e2ac70b.el9.x86_64.rpm 816 kB/s | 170 kB 00:00 2024-09-20T22:35:33.194 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2024-09-20T22:35:33.195 INFO:teuthology.orchestra.run.smithi084.stdout:Total 810 kB/s | 170 kB 00:00 2024-09-20T22:35:33.195 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction check 2024-09-20T22:35:33.202 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction check succeeded. 2024-09-20T22:35:33.202 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction test 2024-09-20T22:35:33.268 INFO:teuthology.orchestra.run.smithi084.stdout:Transaction test succeeded. 2024-09-20T22:35:33.268 INFO:teuthology.orchestra.run.smithi084.stdout:Running transaction 2024-09-20T22:35:33.372 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install rbd-mirror 2024-09-20T22:35:33.425 INFO:teuthology.orchestra.run.smithi084.stdout: Preparing : 1/1 2024-09-20T22:35:33.534 INFO:teuthology.orchestra.run.smithi084.stdout: Installing : rbd-nbd-2:19.1.1-367.g9e2ac70b.el9.x86_64 1/1 2024-09-20T22:35:33.977 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:01:50 ago on Fri 20 Sep 2024 10:33:43 PM UTC. 2024-09-20T22:35:33.983 INFO:teuthology.orchestra.run.smithi084.stdout: Running scriptlet: rbd-nbd-2:19.1.1-367.g9e2ac70b.el9.x86_64 1/1 2024-09-20T22:35:34.116 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2024-09-20T22:35:34.117 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-09-20T22:35:34.117 INFO:teuthology.orchestra.run.smithi154.stdout: Package Arch Version Repo Size 2024-09-20T22:35:34.117 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-09-20T22:35:34.117 INFO:teuthology.orchestra.run.smithi154.stdout:Installing: 2024-09-20T22:35:34.117 INFO:teuthology.orchestra.run.smithi154.stdout: rbd-mirror x86_64 2:19.1.1-367.g9e2ac70b.el9 ceph 3.1 M 2024-09-20T22:35:34.117 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-20T22:35:34.117 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2024-09-20T22:35:34.117 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-09-20T22:35:34.117 INFO:teuthology.orchestra.run.smithi154.stdout:Install 1 Package 2024-09-20T22:35:34.118 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-20T22:35:34.118 INFO:teuthology.orchestra.run.smithi154.stdout:Total download size: 3.1 M 2024-09-20T22:35:34.118 INFO:teuthology.orchestra.run.smithi154.stdout:Installed size: 13 M 2024-09-20T22:35:34.118 INFO:teuthology.orchestra.run.smithi154.stdout:Downloading Packages: 2024-09-20T22:35:34.256 INFO:teuthology.orchestra.run.smithi084.stdout: Verifying : rbd-nbd-2:19.1.1-367.g9e2ac70b.el9.x86_64 1/1 2024-09-20T22:35:34.256 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-20T22:35:34.256 INFO:teuthology.orchestra.run.smithi084.stdout:Installed: 2024-09-20T22:35:34.257 INFO:teuthology.orchestra.run.smithi084.stdout: rbd-nbd-2:19.1.1-367.g9e2ac70b.el9.x86_64 2024-09-20T22:35:34.257 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-20T22:35:34.257 INFO:teuthology.orchestra.run.smithi084.stdout:Complete! 2024-09-20T22:35:34.452 INFO:teuthology.orchestra.run.smithi154.stdout:rbd-mirror-19.1.1-367.g9e2ac70b.el9.x86_64.rpm 9.3 MB/s | 3.1 MB 00:00 2024-09-20T22:35:34.453 INFO:teuthology.orchestra.run.smithi154.stdout:-------------------------------------------------------------------------------- 2024-09-20T22:35:34.453 INFO:teuthology.orchestra.run.smithi154.stdout:Total 9.2 MB/s | 3.1 MB 00:00 2024-09-20T22:35:34.453 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2024-09-20T22:35:34.462 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2024-09-20T22:35:34.462 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2024-09-20T22:35:34.489 DEBUG:teuthology.parallel:result is None 2024-09-20T22:35:34.544 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2024-09-20T22:35:34.545 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2024-09-20T22:35:34.982 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2024-09-20T22:35:35.053 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : rbd-mirror-2:19.1.1-367.g9e2ac70b.el9.x86_64 1/1 2024-09-20T22:35:35.079 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: rbd-mirror-2:19.1.1-367.g9e2ac70b.el9.x86_64 1/1 2024-09-20T22:35:35.079 INFO:teuthology.orchestra.run.smithi154.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-20T22:35:35.080 INFO:teuthology.orchestra.run.smithi154.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-20T22:35:35.080 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-20T22:35:35.080 INFO:teuthology.orchestra.run.smithi154.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-20T22:35:35.080 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-20T22:35:36.347 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : rbd-mirror-2:19.1.1-367.g9e2ac70b.el9.x86_64 1/1 2024-09-20T22:35:36.347 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-20T22:35:36.347 INFO:teuthology.orchestra.run.smithi154.stdout:Installed: 2024-09-20T22:35:36.347 INFO:teuthology.orchestra.run.smithi154.stdout: rbd-mirror-2:19.1.1-367.g9e2ac70b.el9.x86_64 2024-09-20T22:35:36.347 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-20T22:35:36.347 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2024-09-20T22:35:36.572 DEBUG:teuthology.orchestra.run.smithi154:> sudo yum -y install rbd-nbd 2024-09-20T22:35:37.178 INFO:teuthology.orchestra.run.smithi154.stdout:Last metadata expiration check: 0:01:54 ago on Fri 20 Sep 2024 10:33:43 PM UTC. 2024-09-20T22:35:37.310 INFO:teuthology.orchestra.run.smithi154.stdout:Dependencies resolved. 2024-09-20T22:35:37.310 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-09-20T22:35:37.311 INFO:teuthology.orchestra.run.smithi154.stdout: Package Architecture Version Repository Size 2024-09-20T22:35:37.311 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-09-20T22:35:37.311 INFO:teuthology.orchestra.run.smithi154.stdout:Installing: 2024-09-20T22:35:37.311 INFO:teuthology.orchestra.run.smithi154.stdout: rbd-nbd x86_64 2:19.1.1-367.g9e2ac70b.el9 ceph 170 k 2024-09-20T22:35:37.311 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-20T22:35:37.311 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction Summary 2024-09-20T22:35:37.311 INFO:teuthology.orchestra.run.smithi154.stdout:================================================================================ 2024-09-20T22:35:37.311 INFO:teuthology.orchestra.run.smithi154.stdout:Install 1 Package 2024-09-20T22:35:37.311 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-20T22:35:37.311 INFO:teuthology.orchestra.run.smithi154.stdout:Total download size: 170 k 2024-09-20T22:35:37.311 INFO:teuthology.orchestra.run.smithi154.stdout:Installed size: 494 k 2024-09-20T22:35:37.311 INFO:teuthology.orchestra.run.smithi154.stdout:Downloading Packages: 2024-09-20T22:35:37.522 INFO:teuthology.orchestra.run.smithi154.stdout:rbd-nbd-19.1.1-367.g9e2ac70b.el9.x86_64.rpm 810 kB/s | 170 kB 00:00 2024-09-20T22:35:37.522 INFO:teuthology.orchestra.run.smithi154.stdout:-------------------------------------------------------------------------------- 2024-09-20T22:35:37.523 INFO:teuthology.orchestra.run.smithi154.stdout:Total 804 kB/s | 170 kB 00:00 2024-09-20T22:35:37.523 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction check 2024-09-20T22:35:37.530 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction check succeeded. 2024-09-20T22:35:37.530 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction test 2024-09-20T22:35:37.594 INFO:teuthology.orchestra.run.smithi154.stdout:Transaction test succeeded. 2024-09-20T22:35:37.595 INFO:teuthology.orchestra.run.smithi154.stdout:Running transaction 2024-09-20T22:35:37.757 INFO:teuthology.orchestra.run.smithi154.stdout: Preparing : 1/1 2024-09-20T22:35:37.869 INFO:teuthology.orchestra.run.smithi154.stdout: Installing : rbd-nbd-2:19.1.1-367.g9e2ac70b.el9.x86_64 1/1 2024-09-20T22:35:38.288 INFO:teuthology.orchestra.run.smithi154.stdout: Running scriptlet: rbd-nbd-2:19.1.1-367.g9e2ac70b.el9.x86_64 1/1 2024-09-20T22:35:38.548 INFO:teuthology.orchestra.run.smithi154.stdout: Verifying : rbd-nbd-2:19.1.1-367.g9e2ac70b.el9.x86_64 1/1 2024-09-20T22:35:38.548 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-20T22:35:38.548 INFO:teuthology.orchestra.run.smithi154.stdout:Installed: 2024-09-20T22:35:38.548 INFO:teuthology.orchestra.run.smithi154.stdout: rbd-nbd-2:19.1.1-367.g9e2ac70b.el9.x86_64 2024-09-20T22:35:38.548 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-20T22:35:38.548 INFO:teuthology.orchestra.run.smithi154.stdout:Complete! 2024-09-20T22:35:38.785 DEBUG:teuthology.parallel:result is None 2024-09-20T22:35:38.785 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=9e2ac70bb7aea67d34f0d50002344abbe7803736 2024-09-20T22:35:38.932 DEBUG:teuthology.orchestra.run.smithi084:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-20T22:35:38.957 INFO:teuthology.orchestra.run.smithi084.stdout:19.1.1-367.g9e2ac70b.el9 2024-09-20T22:35:38.957 INFO:teuthology.packaging:The installed version of ceph is 19.1.1-367.g9e2ac70b.el9 2024-09-20T22:35:38.957 INFO:teuthology.task.install:The correct ceph version 19.1.1-367.g9e2ac70b is installed. 2024-09-20T22:35:38.959 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=9e2ac70bb7aea67d34f0d50002344abbe7803736 2024-09-20T22:35:39.104 DEBUG:teuthology.orchestra.run.smithi154:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-20T22:35:39.126 INFO:teuthology.orchestra.run.smithi154.stdout:19.1.1-367.g9e2ac70b.el9 2024-09-20T22:35:39.127 INFO:teuthology.packaging:The installed version of ceph is 19.1.1-367.g9e2ac70b.el9 2024-09-20T22:35:39.127 INFO:teuthology.task.install:The correct ceph version 19.1.1-367.g9e2ac70b is installed. 2024-09-20T22:35:39.128 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-20T22:35:39.129 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-09-20T22:35:39.129 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-20T22:35:39.159 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-09-20T22:35:39.159 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-20T22:35:39.193 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-20T22:35:39.193 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-09-20T22:35:39.193 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd of=/usr/bin/daemon-helper 2024-09-20T22:35:39.228 DEBUG:teuthology.orchestra.run.smithi084:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-20T22:35:39.293 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-09-20T22:35:39.293 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd of=/usr/bin/daemon-helper 2024-09-20T22:35:39.318 DEBUG:teuthology.orchestra.run.smithi154:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-20T22:35:39.382 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-20T22:35:39.383 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-09-20T22:35:39.383 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-20T22:35:39.409 DEBUG:teuthology.orchestra.run.smithi084:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-20T22:35:39.473 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-09-20T22:35:39.473 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-20T22:35:39.497 DEBUG:teuthology.orchestra.run.smithi154:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-20T22:35:39.562 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-20T22:35:39.563 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-09-20T22:35:39.563 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd of=/usr/bin/stdin-killer 2024-09-20T22:35:39.587 DEBUG:teuthology.orchestra.run.smithi084:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-20T22:35:39.652 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-09-20T22:35:39.652 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd of=/usr/bin/stdin-killer 2024-09-20T22:35:39.677 DEBUG:teuthology.orchestra.run.smithi154:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-20T22:35:39.740 INFO:teuthology.run_tasks:Running task cephadm... 2024-09-20T22:35:39.822 INFO:tasks.cephadm:Config: {'conf': {'mgr': {'debug mgr': 20, 'debug ms': 1}, 'global': {'ms bind msgr1': False, 'ms bind msgr2': True, 'ms type': 'async'}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd max backfills': 3, 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'CEPHADM_STRAY_DAEMON', 'CEPHADM_FAILED_DAEMON', 'but it is still running', 'objects unfound and apparently lost', '\\(POOL_APP_NOT_ENABLED\\)'], 'log-only-match': ['CEPHADM_'], 'sha1': '9e2ac70bb7aea67d34f0d50002344abbe7803736', 'cephadm_mode': 'root'} 2024-09-20T22:35:39.822 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 2024-09-20T22:35:39.822 INFO:tasks.cephadm:Cluster fsid is a9c0bc70-77a0-11ef-baf5-efdc52797490 2024-09-20T22:35:39.823 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-09-20T22:35:39.823 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.84', 'mon.c': '[v2:172.21.15.84:3301,v1:172.21.15.84:6790]', 'mon.b': '172.21.15.154'} 2024-09-20T22:35:39.823 INFO:tasks.cephadm:First mon is mon.a on smithi084 2024-09-20T22:35:39.823 INFO:tasks.cephadm:First mgr is y 2024-09-20T22:35:39.823 INFO:tasks.cephadm:Normalizing hostnames... 2024-09-20T22:35:39.823 DEBUG:teuthology.orchestra.run.smithi084:> sudo hostname $(hostname -s) 2024-09-20T22:35:39.849 DEBUG:teuthology.orchestra.run.smithi154:> sudo hostname $(hostname -s) 2024-09-20T22:35:39.874 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-09-20T22:35:39.874 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=9e2ac70bb7aea67d34f0d50002344abbe7803736 2024-09-20T22:35:40.015 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '9e2ac70bb7aea67d34f0d50002344abbe7803736', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/56118/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.1.1-367-g9e2ac70b', 'node_name': '172.21.5.36+adami06', 'job_name': 'ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.1.1-367.g9e2ac70b'}, 'url': 'https://1.chacra.ceph.com/r/ceph/squid/9e2ac70bb7aea67d34f0d50002344abbe7803736/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-09-19 19:23:07.775447', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'squid', 'chacra_url': 'https://1.chacra.ceph.com/repos/ceph/squid/9e2ac70bb7aea67d34f0d50002344abbe7803736/centos/9/flavors/default/', 'archs': ['x86_64', 'source'], 'distro': 'centos'}] 2024-09-20T22:35:40.171 INFO:tasks.util.chacra:got chacra host 1.chacra.ceph.com, ref squid, sha1 9e2ac70bb7aea67d34f0d50002344abbe7803736 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=9e2ac70bb7aea67d34f0d50002344abbe7803736 2024-09-20T22:35:40.172 INFO:tasks.cephadm:Discovered cachra url: https://1.chacra.ceph.com/binaries/ceph/squid/9e2ac70bb7aea67d34f0d50002344abbe7803736/centos/9/x86_64/flavors/default/cephadm 2024-09-20T22:35:40.172 INFO:tasks.cephadm:Downloading cephadm from url: https://1.chacra.ceph.com/binaries/ceph/squid/9e2ac70bb7aea67d34f0d50002344abbe7803736/centos/9/x86_64/flavors/default/cephadm 2024-09-20T22:35:40.172 DEBUG:teuthology.orchestra.run.smithi084:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/squid/9e2ac70bb7aea67d34f0d50002344abbe7803736/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-09-20T22:35:40.596 INFO:teuthology.orchestra.run.smithi084.stdout:-rw-r--r--. 1 ubuntu ubuntu 784381 Sep 20 22:35 /home/ubuntu/cephtest/cephadm 2024-09-20T22:35:40.596 DEBUG:teuthology.orchestra.run.smithi154:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/squid/9e2ac70bb7aea67d34f0d50002344abbe7803736/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-09-20T22:35:40.974 INFO:teuthology.orchestra.run.smithi154.stdout:-rw-r--r--. 1 ubuntu ubuntu 784381 Sep 20 22:35 /home/ubuntu/cephtest/cephadm 2024-09-20T22:35:40.975 DEBUG:teuthology.orchestra.run.smithi084:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-09-20T22:35:40.991 DEBUG:teuthology.orchestra.run.smithi154:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-09-20T22:35:41.045 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 on all hosts... 2024-09-20T22:35:41.045 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 pull 2024-09-20T22:35:41.047 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 pull 2024-09-20T22:35:42.580 INFO:teuthology.orchestra.run.smithi084.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736... 2024-09-20T22:35:42.769 INFO:teuthology.orchestra.run.smithi154.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736... 2024-09-20T22:36:11.489 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-20T22:36:11.489 INFO:teuthology.orchestra.run.smithi084.stdout: "ceph_version": "ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)", 2024-09-20T22:36:11.489 INFO:teuthology.orchestra.run.smithi084.stdout: "image_id": "cc9251b744a3faafca7c3095f79bfff81873afe823f31a03a11e015a34abcdbd", 2024-09-20T22:36:11.489 INFO:teuthology.orchestra.run.smithi084.stdout: "repo_digests": [ 2024-09-20T22:36:11.489 INFO:teuthology.orchestra.run.smithi084.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5" 2024-09-20T22:36:11.489 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:36:11.490 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-20T22:36:12.277 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-20T22:36:12.278 INFO:teuthology.orchestra.run.smithi154.stdout: "ceph_version": "ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)", 2024-09-20T22:36:12.278 INFO:teuthology.orchestra.run.smithi154.stdout: "image_id": "cc9251b744a3faafca7c3095f79bfff81873afe823f31a03a11e015a34abcdbd", 2024-09-20T22:36:12.278 INFO:teuthology.orchestra.run.smithi154.stdout: "repo_digests": [ 2024-09-20T22:36:12.278 INFO:teuthology.orchestra.run.smithi154.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5" 2024-09-20T22:36:12.278 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:36:12.278 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-20T22:36:12.306 DEBUG:teuthology.orchestra.run.smithi084:> sudo mkdir -p /etc/ceph 2024-09-20T22:36:12.338 DEBUG:teuthology.orchestra.run.smithi154:> sudo mkdir -p /etc/ceph 2024-09-20T22:36:12.366 DEBUG:teuthology.orchestra.run.smithi084:> sudo chmod 777 /etc/ceph 2024-09-20T22:36:12.406 DEBUG:teuthology.orchestra.run.smithi154:> sudo chmod 777 /etc/ceph 2024-09-20T22:36:12.431 INFO:tasks.cephadm:Writing seed config... 2024-09-20T22:36:12.432 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-09-20T22:36:12.432 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-09-20T22:36:12.432 INFO:tasks.cephadm: override: [global] ms bind msgr1 = False 2024-09-20T22:36:12.432 INFO:tasks.cephadm: override: [global] ms bind msgr2 = True 2024-09-20T22:36:12.432 INFO:tasks.cephadm: override: [global] ms type = async 2024-09-20T22:36:12.432 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-09-20T22:36:12.432 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-09-20T22:36:12.432 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-09-20T22:36:12.432 INFO:tasks.cephadm: override: [mon] mon min osdmap epochs = 50 2024-09-20T22:36:12.433 INFO:tasks.cephadm: override: [mon] mon osdmap full prune interval = 2 2024-09-20T22:36:12.433 INFO:tasks.cephadm: override: [mon] mon osdmap full prune min = 15 2024-09-20T22:36:12.433 INFO:tasks.cephadm: override: [mon] mon osdmap full prune txsize = 2 2024-09-20T22:36:12.433 INFO:tasks.cephadm: override: [mon] paxos service trim min = 10 2024-09-20T22:36:12.433 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-09-20T22:36:12.433 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-09-20T22:36:12.433 INFO:tasks.cephadm: override: [osd] osd debug reject backfill probability = 0.3 2024-09-20T22:36:12.433 INFO:tasks.cephadm: override: [osd] osd delete sleep = 1 2024-09-20T22:36:12.433 INFO:tasks.cephadm: override: [osd] osd max backfills = 3 2024-09-20T22:36:12.433 INFO:tasks.cephadm: override: [osd] osd scrub max interval = 120 2024-09-20T22:36:12.433 INFO:tasks.cephadm: override: [osd] osd scrub min interval = 60 2024-09-20T22:36:12.433 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-09-20T22:36:12.433 INFO:tasks.cephadm: override: [osd] osd snap trim sleep = 2 2024-09-20T22:36:12.434 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-09-20T22:36:12.434 DEBUG:teuthology.orchestra.run.smithi084:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-09-20T22:36:12.463 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = a9c0bc70-77a0-11ef-baf5-efdc52797490 ms bind msgr1 = False ms bind msgr2 = True ms type = async [osd] osd scrub load threshold = 5.0 osd scrub max interval = 120 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 osd debug reject backfill probability = 0.3 osd delete sleep = 1 osd max backfills = 3 osd scrub min interval = 60 osd snap trim sleep = 2 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon min osdmap epochs = 50 mon osdmap full prune interval = 2 mon osdmap full prune min = 15 mon osdmap full prune txsize = 2 paxos service trim min = 10 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true 2024-09-20T22:36:12.464 DEBUG:teuthology.orchestra.run.smithi084:mon.a> sudo journalctl -f -n 0 -u ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490@mon.a.service 2024-09-20T22:36:12.506 DEBUG:teuthology.orchestra.run.smithi084:mgr.y> sudo journalctl -f -n 0 -u ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490@mgr.y.service 2024-09-20T22:36:12.549 INFO:tasks.cephadm:Bootstrapping... 2024-09-20T22:36:12.550 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 -v bootstrap --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.84 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-09-20T22:36:12.753 INFO:teuthology.orchestra.run.smithi084.stdout:-------------------------------------------------------------------------------- 2024-09-20T22:36:12.754 INFO:teuthology.orchestra.run.smithi084.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736', '-v', 'bootstrap', '--fsid', 'a9c0bc70-77a0-11ef-baf5-efdc52797490', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.84', '--skip-admin-label'] 2024-09-20T22:36:12.754 INFO:teuthology.orchestra.run.smithi084.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-09-20T22:36:12.754 INFO:teuthology.orchestra.run.smithi084.stdout:Verifying podman|docker is present... 2024-09-20T22:36:12.777 INFO:teuthology.orchestra.run.smithi084.stdout:/bin/podman: stdout 5.2.2 2024-09-20T22:36:12.777 INFO:teuthology.orchestra.run.smithi084.stdout:Verifying lvm2 is present... 2024-09-20T22:36:12.777 INFO:teuthology.orchestra.run.smithi084.stdout:Verifying time synchronization is in place... 2024-09-20T22:36:12.785 INFO:teuthology.orchestra.run.smithi084.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-09-20T22:36:12.785 INFO:teuthology.orchestra.run.smithi084.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-09-20T22:36:12.792 INFO:teuthology.orchestra.run.smithi084.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-09-20T22:36:12.792 INFO:teuthology.orchestra.run.smithi084.stdout:systemctl: stdout inactive 2024-09-20T22:36:12.799 INFO:teuthology.orchestra.run.smithi084.stdout:systemctl: stdout enabled 2024-09-20T22:36:12.805 INFO:teuthology.orchestra.run.smithi084.stdout:systemctl: stdout active 2024-09-20T22:36:12.805 INFO:teuthology.orchestra.run.smithi084.stdout:Unit chronyd.service is enabled and running 2024-09-20T22:36:12.805 INFO:teuthology.orchestra.run.smithi084.stdout:Repeating the final host check... 2024-09-20T22:36:12.827 INFO:teuthology.orchestra.run.smithi084.stdout:/bin/podman: stdout 5.2.2 2024-09-20T22:36:12.827 INFO:teuthology.orchestra.run.smithi084.stdout:podman (/bin/podman) version 5.2.2 is present 2024-09-20T22:36:12.827 INFO:teuthology.orchestra.run.smithi084.stdout:systemctl is present 2024-09-20T22:36:12.827 INFO:teuthology.orchestra.run.smithi084.stdout:lvcreate is present 2024-09-20T22:36:12.833 INFO:teuthology.orchestra.run.smithi084.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-09-20T22:36:12.834 INFO:teuthology.orchestra.run.smithi084.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-09-20T22:36:12.840 INFO:teuthology.orchestra.run.smithi084.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-09-20T22:36:12.840 INFO:teuthology.orchestra.run.smithi084.stdout:systemctl: stdout inactive 2024-09-20T22:36:12.846 INFO:teuthology.orchestra.run.smithi084.stdout:systemctl: stdout enabled 2024-09-20T22:36:12.852 INFO:teuthology.orchestra.run.smithi084.stdout:systemctl: stdout active 2024-09-20T22:36:12.853 INFO:teuthology.orchestra.run.smithi084.stdout:Unit chronyd.service is enabled and running 2024-09-20T22:36:12.853 INFO:teuthology.orchestra.run.smithi084.stdout:Host looks OK 2024-09-20T22:36:12.853 INFO:teuthology.orchestra.run.smithi084.stdout:Cluster fsid: a9c0bc70-77a0-11ef-baf5-efdc52797490 2024-09-20T22:36:12.853 INFO:teuthology.orchestra.run.smithi084.stdout:Acquiring lock 139931774362624 on /run/cephadm/a9c0bc70-77a0-11ef-baf5-efdc52797490.lock 2024-09-20T22:36:12.853 INFO:teuthology.orchestra.run.smithi084.stdout:Lock 139931774362624 acquired on /run/cephadm/a9c0bc70-77a0-11ef-baf5-efdc52797490.lock 2024-09-20T22:36:12.853 INFO:teuthology.orchestra.run.smithi084.stdout:Verifying IP 172.21.15.84 port 3300 ... 2024-09-20T22:36:12.854 INFO:teuthology.orchestra.run.smithi084.stdout:Verifying IP 172.21.15.84 port 6789 ... 2024-09-20T22:36:12.854 INFO:teuthology.orchestra.run.smithi084.stdout:Base mon IP(s) is [172.21.15.84:3300, 172.21.15.84:6789], mon addrv is [v2:172.21.15.84:3300,v1:172.21.15.84:6789] 2024-09-20T22:36:12.857 INFO:teuthology.orchestra.run.smithi084.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.84 metric 100 2024-09-20T22:36:12.857 INFO:teuthology.orchestra.run.smithi084.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.84 metric 100 2024-09-20T22:36:12.859 INFO:teuthology.orchestra.run.smithi084.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-09-20T22:36:12.859 INFO:teuthology.orchestra.run.smithi084.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 1024 pref medium 2024-09-20T22:36:12.859 INFO:teuthology.orchestra.run.smithi084.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 100 pref medium 2024-09-20T22:36:12.862 INFO:teuthology.orchestra.run.smithi084.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-09-20T22:36:12.862 INFO:teuthology.orchestra.run.smithi084.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-09-20T22:36:12.862 INFO:teuthology.orchestra.run.smithi084.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-09-20T22:36:12.862 INFO:teuthology.orchestra.run.smithi084.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-09-20T22:36:12.862 INFO:teuthology.orchestra.run.smithi084.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:fe8f:d751/64 scope link noprefixroute 2024-09-20T22:36:12.862 INFO:teuthology.orchestra.run.smithi084.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-09-20T22:36:12.863 INFO:teuthology.orchestra.run.smithi084.stdout:Mon IP `172.21.15.84` is in CIDR network `172.21.0.0/20` 2024-09-20T22:36:12.863 INFO:teuthology.orchestra.run.smithi084.stdout:Mon IP `172.21.15.84` is in CIDR network `172.21.0.0/20` 2024-09-20T22:36:12.863 INFO:teuthology.orchestra.run.smithi084.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-09-20T22:36:12.863 INFO:teuthology.orchestra.run.smithi084.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-09-20T22:36:12.863 INFO:teuthology.orchestra.run.smithi084.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736... 2024-09-20T22:36:13.291 INFO:teuthology.orchestra.run.smithi084.stdout:/bin/podman: stdout cc9251b744a3faafca7c3095f79bfff81873afe823f31a03a11e015a34abcdbd 2024-09-20T22:36:13.291 INFO:teuthology.orchestra.run.smithi084.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736... 2024-09-20T22:36:13.291 INFO:teuthology.orchestra.run.smithi084.stdout:/bin/podman: stderr Getting image source signatures 2024-09-20T22:36:13.291 INFO:teuthology.orchestra.run.smithi084.stdout:/bin/podman: stderr Copying blob sha256:0031407101cecb3ed6ba394cb16e83fe515e11366eae1f45bd4759e577ea92fc 2024-09-20T22:36:13.291 INFO:teuthology.orchestra.run.smithi084.stdout:/bin/podman: stderr Copying blob sha256:e8b54c863393b7a8216a71737771b73b16a8e43ec7acf927c7faa175c255e551 2024-09-20T22:36:13.291 INFO:teuthology.orchestra.run.smithi084.stdout:/bin/podman: stderr Copying config sha256:cc9251b744a3faafca7c3095f79bfff81873afe823f31a03a11e015a34abcdbd 2024-09-20T22:36:13.292 INFO:teuthology.orchestra.run.smithi084.stdout:/bin/podman: stderr Writing manifest to image destination 2024-09-20T22:36:14.221 INFO:teuthology.orchestra.run.smithi084.stdout:ceph: stdout ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable) 2024-09-20T22:36:14.221 INFO:teuthology.orchestra.run.smithi084.stdout:Ceph version: ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable) 2024-09-20T22:36:14.222 INFO:teuthology.orchestra.run.smithi084.stdout:Extracting ceph user uid/gid from container image... 2024-09-20T22:36:15.110 INFO:teuthology.orchestra.run.smithi084.stdout:stat: stdout 167 167 2024-09-20T22:36:15.110 INFO:teuthology.orchestra.run.smithi084.stdout:Creating initial keys... 2024-09-20T22:36:15.983 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph-authtool: stdout AQDf+O1mKMLSGxAAVrX0+KledznSDfZt/iQt1g== 2024-09-20T22:36:16.913 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph-authtool: stdout AQDg+O1mz2ztEhAAt7tDocuBbPFKTH3qEOSNyw== 2024-09-20T22:36:17.827 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph-authtool: stdout AQDh+O1m7tYsCxAA9hykSqSc3B4aFdprMBqx8w== 2024-09-20T22:36:17.828 INFO:teuthology.orchestra.run.smithi084.stdout:Creating initial monmap... 2024-09-20T22:36:18.875 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-09-20T22:36:18.875 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-09-20T22:36:18.875 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to a9c0bc70-77a0-11ef-baf5-efdc52797490 2024-09-20T22:36:18.875 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-09-20T22:36:18.876 INFO:teuthology.orchestra.run.smithi084.stdout:monmaptool for a [v2:172.21.15.84:3300,v1:172.21.15.84:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-09-20T22:36:18.876 INFO:teuthology.orchestra.run.smithi084.stdout:setting min_mon_release = quincy 2024-09-20T22:36:18.876 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/monmaptool: set fsid to a9c0bc70-77a0-11ef-baf5-efdc52797490 2024-09-20T22:36:18.876 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-09-20T22:36:18.876 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-20T22:36:18.876 INFO:teuthology.orchestra.run.smithi084.stdout:Creating mon... 2024-09-20T22:36:28.085 INFO:teuthology.orchestra.run.smithi084.stdout:create mon.a on 2024-09-20T22:36:28.543 INFO:teuthology.orchestra.run.smithi084.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-09-20T22:36:28.931 INFO:teuthology.orchestra.run.smithi084.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-09-20T22:36:29.368 INFO:teuthology.orchestra.run.smithi084.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490.target → /etc/systemd/system/ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490.target. 2024-09-20T22:36:29.368 INFO:teuthology.orchestra.run.smithi084.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490.target → /etc/systemd/system/ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490.target. 2024-09-20T22:36:29.594 INFO:teuthology.orchestra.run.smithi084.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490@mon.a 2024-09-20T22:36:29.594 INFO:teuthology.orchestra.run.smithi084.stdout:systemctl: stderr Failed to reset failed state of unit ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490@mon.a.service: Unit ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490@mon.a.service not loaded. 2024-09-20T22:36:29.758 INFO:teuthology.orchestra.run.smithi084.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490.target.wants/ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490@mon.a.service → /etc/systemd/system/ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490@.service. 2024-09-20T22:36:30.244 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:30 smithi084 podman[29389]: 2024-09-20 22:36:30.202049077 +0000 UTC m=+0.328356027 container start 38958d68b88d0e225fb517998fe8853d55c50f05cc717839d03821ecd97af5f0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=squid-9e2ac70, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731) 2024-09-20T22:36:30.244 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:30 smithi084 ceph-mon[29408]: mkfs a9c0bc70-77a0-11ef-baf5-efdc52797490 2024-09-20T22:36:30.289 INFO:teuthology.orchestra.run.smithi084.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-20T22:36:30.289 INFO:teuthology.orchestra.run.smithi084.stdout:systemctl: stdout disabled 2024-09-20T22:36:30.296 INFO:teuthology.orchestra.run.smithi084.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-20T22:36:30.296 INFO:teuthology.orchestra.run.smithi084.stdout:systemctl: stdout inactive 2024-09-20T22:36:30.296 INFO:teuthology.orchestra.run.smithi084.stdout:firewalld.service is not enabled 2024-09-20T22:36:30.296 INFO:teuthology.orchestra.run.smithi084.stdout:Not possible to enable service . firewalld.service is not available 2024-09-20T22:36:30.296 INFO:teuthology.orchestra.run.smithi084.stdout:Waiting for mon to start... 2024-09-20T22:36:30.296 INFO:teuthology.orchestra.run.smithi084.stdout:Waiting for mon... 2024-09-20T22:36:30.521 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:30 smithi084 ceph-mon[29408]: mon.a is new leader, mons a in quorum (ranks 0) 2024-09-20T22:36:30.521 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:30 smithi084 bash[29389]: 38958d68b88d0e225fb517998fe8853d55c50f05cc717839d03821ecd97af5f0 2024-09-20T22:36:30.521 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:30 smithi084 systemd[1]: Started Ceph mon.a for a9c0bc70-77a0-11ef-baf5-efdc52797490. 2024-09-20T22:36:31.340 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:31 smithi084 ceph-mon[29408]: mon.a is new leader, mons a in quorum (ranks 0) 2024-09-20T22:36:31.340 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:31 smithi084 ceph-mon[29408]: monmap epoch 1 2024-09-20T22:36:31.340 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:31 smithi084 ceph-mon[29408]: fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 2024-09-20T22:36:31.340 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:31 smithi084 ceph-mon[29408]: last_changed 2024-09-20T22:36:18.149782+0000 2024-09-20T22:36:31.340 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:31 smithi084 ceph-mon[29408]: created 2024-09-20T22:36:18.149782+0000 2024-09-20T22:36:31.340 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:31 smithi084 ceph-mon[29408]: min_mon_release 19 (squid) 2024-09-20T22:36:31.340 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:31 smithi084 ceph-mon[29408]: election_strategy: 1 2024-09-20T22:36:31.340 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:31 smithi084 ceph-mon[29408]: 0: [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] mon.a 2024-09-20T22:36:31.341 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:31 smithi084 ceph-mon[29408]: fsmap 2024-09-20T22:36:31.341 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:31 smithi084 ceph-mon[29408]: osdmap e1: 0 total, 0 up, 0 in 2024-09-20T22:36:31.341 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:31 smithi084 ceph-mon[29408]: mgrmap e1: no daemons active 2024-09-20T22:36:31.341 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:31 smithi084 ceph-mon[29408]: from='client.? 172.21.15.84:0/3178109713' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-09-20T22:36:31.343 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout cluster: 2024-09-20T22:36:31.343 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout id: a9c0bc70-77a0-11ef-baf5-efdc52797490 2024-09-20T22:36:31.343 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-09-20T22:36:31.344 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout 2024-09-20T22:36:31.344 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout services: 2024-09-20T22:36:31.344 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.530089s) 2024-09-20T22:36:31.344 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-09-20T22:36:31.344 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-09-20T22:36:31.344 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout 2024-09-20T22:36:31.344 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout data: 2024-09-20T22:36:31.344 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-09-20T22:36:31.344 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-09-20T22:36:31.344 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-09-20T22:36:31.344 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout pgs: 2024-09-20T22:36:31.344 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout 2024-09-20T22:36:31.344 INFO:teuthology.orchestra.run.smithi084.stdout:mon is available 2024-09-20T22:36:31.344 INFO:teuthology.orchestra.run.smithi084.stdout:Assimilating anything we can from ceph.conf... 2024-09-20T22:36:32.337 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:32 smithi084 ceph-mon[29408]: from='client.? 172.21.15.84:0/410590870' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-09-20T22:36:32.338 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:32 smithi084 ceph-mon[29408]: from='client.? 172.21.15.84:0/410590870' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-09-20T22:36:32.341 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout 2024-09-20T22:36:32.342 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout [global] 2024-09-20T22:36:32.342 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout fsid = a9c0bc70-77a0-11ef-baf5-efdc52797490 2024-09-20T22:36:32.342 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.84:3300,v1:172.21.15.84:6789] 2024-09-20T22:36:32.342 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-09-20T22:36:32.342 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-09-20T22:36:32.342 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-09-20T22:36:32.342 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-09-20T22:36:32.342 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout 2024-09-20T22:36:32.342 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout [mgr] 2024-09-20T22:36:32.342 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-09-20T22:36:32.342 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout 2024-09-20T22:36:32.342 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout [osd] 2024-09-20T22:36:32.342 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-09-20T22:36:32.343 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-09-20T22:36:32.343 INFO:teuthology.orchestra.run.smithi084.stdout:Generating new minimal ceph.conf... 2024-09-20T22:36:33.364 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:33 smithi084 ceph-mon[29408]: from='client.? 172.21.15.84:0/3200236107' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:36:33.374 INFO:teuthology.orchestra.run.smithi084.stdout:Restarting the monitor... 2024-09-20T22:36:33.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:33 smithi084 systemd[1]: Stopping Ceph mon.a for a9c0bc70-77a0-11ef-baf5-efdc52797490... 2024-09-20T22:36:34.273 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:34 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mon-a[29404]: 2024-09-20T22:36:34.015+0000 7f48d49c9640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-20T22:36:34.274 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:34 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mon-a[29404]: 2024-09-20T22:36:34.015+0000 7f48d49c9640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-09-20T22:36:34.614 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:34 smithi084 podman[29650]: 2024-09-20 22:36:34.272545767 +0000 UTC m=+0.728781609 container died 38958d68b88d0e225fb517998fe8853d55c50f05cc717839d03821ecd97af5f0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mon-a, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=squid-9e2ac70, GIT_CLEAN=True) 2024-09-20T22:36:36.864 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:36 smithi084 podman[29650]: 2024-09-20 22:36:36.403174113 +0000 UTC m=+2.859409957 container cleanup 38958d68b88d0e225fb517998fe8853d55c50f05cc717839d03821ecd97af5f0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mon-a, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=squid-9e2ac70, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS) 2024-09-20T22:36:36.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:36 smithi084 bash[29650]: ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mon-a 2024-09-20T22:36:37.517 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:37 smithi084 podman[29662]: 2024-09-20 22:36:37.265856967 +0000 UTC m=+2.983306837 container remove 38958d68b88d0e225fb517998fe8853d55c50f05cc717839d03821ecd97af5f0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mon-a, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-9e2ac70, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-09-20T22:36:37.849 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:37 smithi084 systemd[1]: ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490@mon.a.service: Deactivated successfully. 2024-09-20T22:36:37.849 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:37 smithi084 systemd[1]: Stopped Ceph mon.a for a9c0bc70-77a0-11ef-baf5-efdc52797490. 2024-09-20T22:36:37.849 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:37 smithi084 systemd[1]: Starting Ceph mon.a for a9c0bc70-77a0-11ef-baf5-efdc52797490... 2024-09-20T22:36:37.849 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:37 smithi084 podman[29754]: 2024-09-20 22:36:37.651965209 +0000 UTC m=+0.016148385 image pull cc9251b744a3faafca7c3095f79bfff81873afe823f31a03a11e015a34abcdbd quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 2024-09-20T22:36:37.849 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:37 smithi084 podman[29754]: 2024-09-20 22:36:37.749747563 +0000 UTC m=+0.113930725 container create 318fc1d3b260ddf478f7c5eae8aeb5b15890ed113f21be937bb46cbddb726aed (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mon-a, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, RELEASE=squid-9e2ac70, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-20T22:36:38.074 INFO:teuthology.orchestra.run.smithi084.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-09-20T22:36:38.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:37 smithi084 podman[29754]: 2024-09-20 22:36:37.993817612 +0000 UTC m=+0.358000776 container init 318fc1d3b260ddf478f7c5eae8aeb5b15890ed113f21be937bb46cbddb726aed (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=squid-9e2ac70, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-09-20T22:36:38.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:37 smithi084 podman[29754]: 2024-09-20 22:36:37.996852548 +0000 UTC m=+0.361035711 container start 318fc1d3b260ddf478f7c5eae8aeb5b15890ed113f21be937bb46cbddb726aed (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mon-a, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-9e2ac70) 2024-09-20T22:36:38.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: set uid:gid to 167:167 (ceph:ceph) 2024-09-20T22:36:38.116 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable), process ceph-mon, pid 2 2024-09-20T22:36:38.116 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: pidfile_write: ignore empty --pid-file 2024-09-20T22:36:38.116 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: load: jerasure load: lrc 2024-09-20T22:36:38.116 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: RocksDB version: 7.9.2 2024-09-20T22:36:38.116 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Git sha 0 2024-09-20T22:36:38.116 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Compile date 2024-09-19 18:20:48 2024-09-20T22:36:38.116 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: DB SUMMARY 2024-09-20T22:36:38.116 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: DB Session ID: 9V3R73NOD7R63J2ZRS9F 2024-09-20T22:36:38.116 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: CURRENT file: CURRENT 2024-09-20T22:36:38.116 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: IDENTITY file: IDENTITY 2024-09-20T22:36:38.116 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-09-20T22:36:38.116 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-09-20T22:36:38.116 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 94535 ; 2024-09-20T22:36:38.116 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.error_if_exists: 0 2024-09-20T22:36:38.117 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.create_if_missing: 0 2024-09-20T22:36:38.117 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.paranoid_checks: 1 2024-09-20T22:36:38.117 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.flush_verify_memtable_count: 1 2024-09-20T22:36:38.117 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-20T22:36:38.117 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-09-20T22:36:38.117 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.env: 0x5589926aa940 2024-09-20T22:36:38.117 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.fs: PosixFileSystem 2024-09-20T22:36:38.117 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.info_log: 0x55899344d6e0 2024-09-20T22:36:38.117 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.max_file_opening_threads: 16 2024-09-20T22:36:38.117 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.statistics: (nil) 2024-09-20T22:36:38.117 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.use_fsync: 0 2024-09-20T22:36:38.117 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.max_log_file_size: 0 2024-09-20T22:36:38.117 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-20T22:36:38.117 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-20T22:36:38.118 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.keep_log_file_num: 1000 2024-09-20T22:36:38.118 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.recycle_log_file_num: 0 2024-09-20T22:36:38.118 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.allow_fallocate: 1 2024-09-20T22:36:38.118 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.allow_mmap_reads: 0 2024-09-20T22:36:38.118 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.allow_mmap_writes: 0 2024-09-20T22:36:38.118 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.use_direct_reads: 0 2024-09-20T22:36:38.118 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-20T22:36:38.118 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.create_missing_column_families: 0 2024-09-20T22:36:38.118 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.db_log_dir: 2024-09-20T22:36:38.118 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.wal_dir: 2024-09-20T22:36:38.118 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-20T22:36:38.118 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-20T22:36:38.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-20T22:36:38.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-20T22:36:38.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-20T22:36:38.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-20T22:36:38.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.advise_random_on_open: 1 2024-09-20T22:36:38.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.db_write_buffer_size: 0 2024-09-20T22:36:38.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.write_buffer_manager: 0x55899345cdc0 2024-09-20T22:36:38.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-20T22:36:38.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-20T22:36:38.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-20T22:36:38.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.rate_limiter: (nil) 2024-09-20T22:36:38.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-20T22:36:38.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.wal_recovery_mode: 2 2024-09-20T22:36:38.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.enable_thread_tracking: 0 2024-09-20T22:36:38.119 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.enable_pipelined_write: 0 2024-09-20T22:36:38.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.unordered_write: 0 2024-09-20T22:36:38.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-20T22:36:38.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-20T22:36:38.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-20T22:36:38.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-20T22:36:38.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.row_cache: None 2024-09-20T22:36:38.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.wal_filter: None 2024-09-20T22:36:38.120 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-20T22:36:38.121 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.allow_ingest_behind: 0 2024-09-20T22:36:38.121 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.two_write_queues: 0 2024-09-20T22:36:38.121 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.manual_wal_flush: 0 2024-09-20T22:36:38.121 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.wal_compression: 0 2024-09-20T22:36:38.121 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.atomic_flush: 0 2024-09-20T22:36:38.121 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-20T22:36:38.121 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-20T22:36:38.121 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-20T22:36:38.122 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.log_readahead_size: 0 2024-09-20T22:36:38.122 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-20T22:36:38.122 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.best_efforts_recovery: 0 2024-09-20T22:36:38.122 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-20T22:36:38.122 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-20T22:36:38.122 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.allow_data_in_errors: 0 2024-09-20T22:36:38.122 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.db_host_id: __hostname__ 2024-09-20T22:36:38.122 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.enforce_single_del_contracts: true 2024-09-20T22:36:38.122 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.max_background_jobs: 2 2024-09-20T22:36:38.122 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.max_background_compactions: -1 2024-09-20T22:36:38.122 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.max_subcompactions: 1 2024-09-20T22:36:38.122 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-20T22:36:38.122 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-20T22:36:38.122 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-20T22:36:38.123 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.max_total_wal_size: 0 2024-09-20T22:36:38.123 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-20T22:36:38.123 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-20T22:36:38.123 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-20T22:36:38.123 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-20T22:36:38.123 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.max_open_files: -1 2024-09-20T22:36:38.123 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.bytes_per_sync: 0 2024-09-20T22:36:38.123 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-20T22:36:38.123 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-20T22:36:38.123 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.compaction_readahead_size: 0 2024-09-20T22:36:38.123 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.max_background_flushes: -1 2024-09-20T22:36:38.123 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Compression algorithms supported: 2024-09-20T22:36:38.123 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: kZSTD supported: 0 2024-09-20T22:36:38.123 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: kXpressCompression supported: 0 2024-09-20T22:36:38.124 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: kBZip2Compression supported: 0 2024-09-20T22:36:38.124 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-20T22:36:38.124 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: kLZ4Compression supported: 1 2024-09-20T22:36:38.124 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: kZlibCompression supported: 1 2024-09-20T22:36:38.124 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: kLZ4HCCompression supported: 1 2024-09-20T22:36:38.124 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: kSnappyCompression supported: 1 2024-09-20T22:36:38.124 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-20T22:36:38.124 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: DMutex implementation: pthread_mutex_t 2024-09-20T22:36:38.124 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-09-20T22:36:38.124 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-09-20T22:36:38.124 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-20T22:36:38.124 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.merge_operator: 2024-09-20T22:36:38.124 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.compaction_filter: None 2024-09-20T22:36:38.124 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.compaction_filter_factory: None 2024-09-20T22:36:38.124 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.sst_partitioner_factory: None 2024-09-20T22:36:38.125 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-20T22:36:38.125 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-20T22:36:38.125 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55899344d320) 2024-09-20T22:36:38.125 INFO:journalctl@ceph.mon.a.smithi084.stdout: cache_index_and_filter_blocks: 1 2024-09-20T22:36:38.125 INFO:journalctl@ceph.mon.a.smithi084.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-20T22:36:38.125 INFO:journalctl@ceph.mon.a.smithi084.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-20T22:36:38.125 INFO:journalctl@ceph.mon.a.smithi084.stdout: pin_top_level_index_and_filter: 1 2024-09-20T22:36:38.125 INFO:journalctl@ceph.mon.a.smithi084.stdout: index_type: 0 2024-09-20T22:36:38.125 INFO:journalctl@ceph.mon.a.smithi084.stdout: data_block_index_type: 0 2024-09-20T22:36:38.125 INFO:journalctl@ceph.mon.a.smithi084.stdout: index_shortening: 1 2024-09-20T22:36:38.125 INFO:journalctl@ceph.mon.a.smithi084.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-20T22:36:38.125 INFO:journalctl@ceph.mon.a.smithi084.stdout: checksum: 4 2024-09-20T22:36:38.125 INFO:journalctl@ceph.mon.a.smithi084.stdout: no_block_cache: 0 2024-09-20T22:36:38.125 INFO:journalctl@ceph.mon.a.smithi084.stdout: block_cache: 0x5589934411f0 2024-09-20T22:36:38.126 INFO:journalctl@ceph.mon.a.smithi084.stdout: block_cache_name: BinnedLRUCache 2024-09-20T22:36:38.126 INFO:journalctl@ceph.mon.a.smithi084.stdout: block_cache_options: 2024-09-20T22:36:38.126 INFO:journalctl@ceph.mon.a.smithi084.stdout: capacity : 536870912 2024-09-20T22:36:38.126 INFO:journalctl@ceph.mon.a.smithi084.stdout: num_shard_bits : 4 2024-09-20T22:36:38.126 INFO:journalctl@ceph.mon.a.smithi084.stdout: strict_capacity_limit : 0 2024-09-20T22:36:38.126 INFO:journalctl@ceph.mon.a.smithi084.stdout: high_pri_pool_ratio: 0.000 2024-09-20T22:36:38.126 INFO:journalctl@ceph.mon.a.smithi084.stdout: block_cache_compressed: (nil) 2024-09-20T22:36:38.126 INFO:journalctl@ceph.mon.a.smithi084.stdout: persistent_cache: (nil) 2024-09-20T22:36:38.126 INFO:journalctl@ceph.mon.a.smithi084.stdout: block_size: 4096 2024-09-20T22:36:38.126 INFO:journalctl@ceph.mon.a.smithi084.stdout: block_size_deviation: 10 2024-09-20T22:36:38.126 INFO:journalctl@ceph.mon.a.smithi084.stdout: block_restart_interval: 16 2024-09-20T22:36:38.126 INFO:journalctl@ceph.mon.a.smithi084.stdout: index_block_restart_interval: 1 2024-09-20T22:36:38.126 INFO:journalctl@ceph.mon.a.smithi084.stdout: metadata_block_size: 4096 2024-09-20T22:36:38.126 INFO:journalctl@ceph.mon.a.smithi084.stdout: partition_filters: 0 2024-09-20T22:36:38.126 INFO:journalctl@ceph.mon.a.smithi084.stdout: use_delta_encoding: 1 2024-09-20T22:36:38.127 INFO:journalctl@ceph.mon.a.smithi084.stdout: filter_policy: bloomfilter 2024-09-20T22:36:38.127 INFO:journalctl@ceph.mon.a.smithi084.stdout: whole_key_filtering: 1 2024-09-20T22:36:38.127 INFO:journalctl@ceph.mon.a.smithi084.stdout: verify_compression: 0 2024-09-20T22:36:38.127 INFO:journalctl@ceph.mon.a.smithi084.stdout: read_amp_bytes_per_bit: 0 2024-09-20T22:36:38.127 INFO:journalctl@ceph.mon.a.smithi084.stdout: format_version: 5 2024-09-20T22:36:38.127 INFO:journalctl@ceph.mon.a.smithi084.stdout: enable_index_compression: 1 2024-09-20T22:36:38.127 INFO:journalctl@ceph.mon.a.smithi084.stdout: block_align: 0 2024-09-20T22:36:38.127 INFO:journalctl@ceph.mon.a.smithi084.stdout: max_auto_readahead_size: 262144 2024-09-20T22:36:38.127 INFO:journalctl@ceph.mon.a.smithi084.stdout: prepopulate_block_cache: 0 2024-09-20T22:36:38.127 INFO:journalctl@ceph.mon.a.smithi084.stdout: initial_auto_readahead_size: 8192 2024-09-20T22:36:38.127 INFO:journalctl@ceph.mon.a.smithi084.stdout: num_file_reads_for_auto_readahead: 2 2024-09-20T22:36:38.127 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.write_buffer_size: 33554432 2024-09-20T22:36:38.127 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.max_write_buffer_number: 2 2024-09-20T22:36:38.127 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.compression: NoCompression 2024-09-20T22:36:38.128 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.bottommost_compression: Disabled 2024-09-20T22:36:38.128 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.prefix_extractor: nullptr 2024-09-20T22:36:38.128 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-20T22:36:38.128 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.num_levels: 7 2024-09-20T22:36:38.128 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-20T22:36:38.128 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-20T22:36:38.128 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-20T22:36:38.128 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-20T22:36:38.128 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-20T22:36:38.128 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-20T22:36:38.128 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-20T22:36:38.128 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-20T22:36:38.128 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-20T22:36:38.129 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-20T22:36:38.129 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-09-20T22:36:38.129 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-09-20T22:36:38.129 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-20T22:36:38.129 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.compression_opts.level: 32767 2024-09-20T22:36:38.129 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.compression_opts.strategy: 0 2024-09-20T22:36:38.129 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-20T22:36:38.129 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-20T22:36:38.129 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-09-20T22:36:38.129 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-20T22:36:38.129 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.compression_opts.enabled: false 2024-09-20T22:36:38.129 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-09-20T22:36:38.129 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-20T22:36:38.129 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-20T22:36:38.129 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-20T22:36:38.129 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.target_file_size_base: 67108864 2024-09-20T22:36:38.130 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-20T22:36:38.130 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-20T22:36:38.130 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-20T22:36:38.130 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-20T22:36:38.130 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-20T22:36:38.130 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-20T22:36:38.130 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-20T22:36:38.130 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-20T22:36:38.130 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-20T22:36:38.130 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-20T22:36:38.130 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-20T22:36:38.131 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-20T22:36:38.131 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-20T22:36:38.131 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-09-20T22:36:38.131 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.arena_block_size: 1048576 2024-09-20T22:36:38.131 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-20T22:36:38.131 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-20T22:36:38.131 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.disable_auto_compactions: 0 2024-09-20T22:36:38.131 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-20T22:36:38.131 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-20T22:36:38.131 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-20T22:36:38.131 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-20T22:36:38.131 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-20T22:36:38.131 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-20T22:36:38.131 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-20T22:36:38.132 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-20T22:36:38.132 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-20T22:36:38.132 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-20T22:36:38.132 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-20T22:36:38.132 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.inplace_update_support: 0 2024-09-20T22:36:38.132 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-20T22:36:38.132 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-20T22:36:38.132 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-20T22:36:38.132 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-20T22:36:38.132 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.bloom_locality: 0 2024-09-20T22:36:38.132 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.max_successive_merges: 0 2024-09-20T22:36:38.132 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-20T22:36:38.132 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.paranoid_file_checks: 0 2024-09-20T22:36:38.132 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.force_consistency_checks: 1 2024-09-20T22:36:38.132 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.report_bg_io_stats: 0 2024-09-20T22:36:38.133 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.ttl: 2592000 2024-09-20T22:36:38.133 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-20T22:36:38.133 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-09-20T22:36:38.133 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-09-20T22:36:38.133 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.enable_blob_files: false 2024-09-20T22:36:38.133 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.min_blob_size: 0 2024-09-20T22:36:38.133 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.blob_file_size: 268435456 2024-09-20T22:36:38.133 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-20T22:36:38.133 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-20T22:36:38.133 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-20T22:36:38.133 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-09-20T22:36:38.133 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-09-20T22:36:38.134 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.blob_file_starting_level: 0 2024-09-20T22:36:38.134 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-09-20T22:36:38.134 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-09-20T22:36:38.134 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-09-20T22:36:38.134 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: fe07a67e-442a-447e-8780-1323bbf16115 2024-09-20T22:36:38.134 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726871798023109, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-09-20T22:36:38.134 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-09-20T22:36:38.135 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726871798024067, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 90519, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 282, "table_properties": {"data_size": 88582, "index_size": 257, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 645, "raw_key_size": 11643, "raw_average_key_size": 48, "raw_value_size": 82184, "raw_average_value_size": 341, "num_data_blocks": 11, "num_entries": 241, "num_filter_entries": 241, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1726871798, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "fe07a67e-442a-447e-8780-1323bbf16115", "db_session_id": "9V3R73NOD7R63J2ZRS9F", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-09-20T22:36:38.135 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726871798024162, "job": 1, "event": "recovery_finished"} 2024-09-20T22:36:38.135 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-09-20T22:36:38.135 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-20T22:36:38.135 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55899346ee00 2024-09-20T22:36:38.135 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: DB pointer 0x55899356a000 2024-09-20T22:36:38.135 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-09-20T22:36:38.136 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-09-20T22:36:38.136 INFO:journalctl@ceph.mon.a.smithi084.stdout: ** DB Stats ** 2024-09-20T22:36:38.136 INFO:journalctl@ceph.mon.a.smithi084.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-20T22:36:38.136 INFO:journalctl@ceph.mon.a.smithi084.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-20T22:36:38.136 INFO:journalctl@ceph.mon.a.smithi084.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-20T22:36:38.136 INFO:journalctl@ceph.mon.a.smithi084.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-20T22:36:38.136 INFO:journalctl@ceph.mon.a.smithi084.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-20T22:36:38.136 INFO:journalctl@ceph.mon.a.smithi084.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-20T22:36:38.136 INFO:journalctl@ceph.mon.a.smithi084.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-20T22:36:38.136 INFO:journalctl@ceph.mon.a.smithi084.stdout: 2024-09-20T22:36:38.136 INFO:journalctl@ceph.mon.a.smithi084.stdout: ** Compaction Stats [default] ** 2024-09-20T22:36:38.136 INFO:journalctl@ceph.mon.a.smithi084.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-20T22:36:38.136 INFO:journalctl@ceph.mon.a.smithi084.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-09-20T22:36:38.136 INFO:journalctl@ceph.mon.a.smithi084.stdout: L0 2/0 90.26 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 135.3 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-20T22:36:38.136 INFO:journalctl@ceph.mon.a.smithi084.stdout: Sum 2/0 90.26 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 135.3 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-20T22:36:38.137 INFO:journalctl@ceph.mon.a.smithi084.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 135.3 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-20T22:36:38.137 INFO:journalctl@ceph.mon.a.smithi084.stdout: 2024-09-20T22:36:38.137 INFO:journalctl@ceph.mon.a.smithi084.stdout: ** Compaction Stats [default] ** 2024-09-20T22:36:38.137 INFO:journalctl@ceph.mon.a.smithi084.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-20T22:36:38.137 INFO:journalctl@ceph.mon.a.smithi084.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-20T22:36:38.137 INFO:journalctl@ceph.mon.a.smithi084.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 135.3 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-20T22:36:38.137 INFO:journalctl@ceph.mon.a.smithi084.stdout: 2024-09-20T22:36:38.137 INFO:journalctl@ceph.mon.a.smithi084.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-09-20T22:36:38.137 INFO:journalctl@ceph.mon.a.smithi084.stdout: 2024-09-20T22:36:38.137 INFO:journalctl@ceph.mon.a.smithi084.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-20T22:36:38.137 INFO:journalctl@ceph.mon.a.smithi084.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-09-20T22:36:38.137 INFO:journalctl@ceph.mon.a.smithi084.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-20T22:36:38.137 INFO:journalctl@ceph.mon.a.smithi084.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-20T22:36:38.137 INFO:journalctl@ceph.mon.a.smithi084.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-20T22:36:38.137 INFO:journalctl@ceph.mon.a.smithi084.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-20T22:36:38.137 INFO:journalctl@ceph.mon.a.smithi084.stdout: Cumulative compaction: 0.00 GB write, 28.49 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-20T22:36:38.138 INFO:journalctl@ceph.mon.a.smithi084.stdout: Interval compaction: 0.00 GB write, 28.49 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-20T22:36:38.138 INFO:journalctl@ceph.mon.a.smithi084.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-20T22:36:38.138 INFO:journalctl@ceph.mon.a.smithi084.stdout: Block cache BinnedLRUCache@0x5589934411f0#2 capacity: 512.00 MB usage: 1.30 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1e-05 secs_since: 0 2024-09-20T22:36:38.138 INFO:journalctl@ceph.mon.a.smithi084.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.83 KB,0.000157952%) IndexBlock(2,0.47 KB,8.9407e-05%) Misc(1,0.00 KB,0%) 2024-09-20T22:36:38.138 INFO:journalctl@ceph.mon.a.smithi084.stdout: 2024-09-20T22:36:38.138 INFO:journalctl@ceph.mon.a.smithi084.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-20T22:36:38.138 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: starting mon.a rank 0 at public addrs [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] at bind addrs [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 2024-09-20T22:36:38.138 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: mon.a@-1(???) e1 preinit fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 2024-09-20T22:36:38.138 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: mon.a@-1(???).mds e1 new map 2024-09-20T22:36:38.138 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: mon.a@-1(???).mds e1 print_map 2024-09-20T22:36:38.138 INFO:journalctl@ceph.mon.a.smithi084.stdout: e1 2024-09-20T22:36:38.138 INFO:journalctl@ceph.mon.a.smithi084.stdout: btime 2024-09-20T22:36:30:240441+0000 2024-09-20T22:36:38.138 INFO:journalctl@ceph.mon.a.smithi084.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-20T22:36:38.138 INFO:journalctl@ceph.mon.a.smithi084.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-09-20T22:36:38.138 INFO:journalctl@ceph.mon.a.smithi084.stdout: legacy client fscid: -1 2024-09-20T22:36:38.139 INFO:journalctl@ceph.mon.a.smithi084.stdout: 2024-09-20T22:36:38.139 INFO:journalctl@ceph.mon.a.smithi084.stdout: No filesystems configured 2024-09-20T22:36:38.139 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-09-20T22:36:38.139 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-20T22:36:38.139 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-20T22:36:38.139 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-20T22:36:38.139 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-09-20T22:36:38.139 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 bash[29754]: 318fc1d3b260ddf478f7c5eae8aeb5b15890ed113f21be937bb46cbddb726aed 2024-09-20T22:36:38.139 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 systemd[1]: Started Ceph mon.a for a9c0bc70-77a0-11ef-baf5-efdc52797490. 2024-09-20T22:36:38.139 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: mon.a is new leader, mons a in quorum (ranks 0) 2024-09-20T22:36:38.139 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: monmap epoch 1 2024-09-20T22:36:38.139 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 2024-09-20T22:36:38.139 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: last_changed 2024-09-20T22:36:18.149782+0000 2024-09-20T22:36:38.139 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: created 2024-09-20T22:36:18.149782+0000 2024-09-20T22:36:38.139 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: min_mon_release 19 (squid) 2024-09-20T22:36:38.140 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: election_strategy: 1 2024-09-20T22:36:38.140 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: 0: [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] mon.a 2024-09-20T22:36:38.140 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: fsmap 2024-09-20T22:36:38.140 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: osdmap e1: 0 total, 0 up, 0 in 2024-09-20T22:36:38.140 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:38 smithi084 ceph-mon[29774]: mgrmap e1: no daemons active 2024-09-20T22:36:39.230 INFO:teuthology.orchestra.run.smithi084.stdout:Wrote config to /etc/ceph/ceph.conf 2024-09-20T22:36:39.251 INFO:teuthology.orchestra.run.smithi084.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-09-20T22:36:39.251 INFO:teuthology.orchestra.run.smithi084.stdout:Creating mgr... 2024-09-20T22:36:39.252 INFO:teuthology.orchestra.run.smithi084.stdout:Verifying port 0.0.0.0:9283 ... 2024-09-20T22:36:39.252 INFO:teuthology.orchestra.run.smithi084.stdout:Verifying port 0.0.0.0:8765 ... 2024-09-20T22:36:39.505 INFO:teuthology.orchestra.run.smithi084.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490@mgr.y 2024-09-20T22:36:39.505 INFO:teuthology.orchestra.run.smithi084.stdout:systemctl: stderr Failed to reset failed state of unit ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490@mgr.y.service: Unit ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490@mgr.y.service not loaded. 2024-09-20T22:36:39.704 INFO:teuthology.orchestra.run.smithi084.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490.target.wants/ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490@mgr.y.service → /etc/systemd/system/ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490@.service. 2024-09-20T22:36:39.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:39 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/353055513' entity='client.admin' 2024-09-20T22:36:39.865 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:36:39 smithi084 systemd[1]: Starting Ceph mgr.y for a9c0bc70-77a0-11ef-baf5-efdc52797490... 2024-09-20T22:36:40.248 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:36:39 smithi084 podman[29985]: 2024-09-20 22:36:39.836376497 +0000 UTC m=+0.015696864 image pull cc9251b744a3faafca7c3095f79bfff81873afe823f31a03a11e015a34abcdbd quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 2024-09-20T22:36:40.248 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:36:39 smithi084 podman[29985]: 2024-09-20 22:36:39.953093824 +0000 UTC m=+0.132414233 container create 5780dd378cf0d7032d08f9f77dd852f2e85a2c7fdf3f902478f46e1e457e829d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, RELEASE=squid-9e2ac70, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, ceph=True) 2024-09-20T22:36:40.249 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:36:40 smithi084 podman[29985]: 2024-09-20 22:36:40.155331704 +0000 UTC m=+0.334652067 container init 5780dd378cf0d7032d08f9f77dd852f2e85a2c7fdf3f902478f46e1e457e829d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, RELEASE=squid-9e2ac70, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-20T22:36:40.249 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:36:40 smithi084 podman[29985]: 2024-09-20 22:36:40.158107238 +0000 UTC m=+0.337427604 container start 5780dd378cf0d7032d08f9f77dd852f2e85a2c7fdf3f902478f46e1e457e829d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y, CEPH_POINT_RELEASE=, ceph=True, RELEASE=squid-9e2ac70, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, GIT_CLEAN=True) 2024-09-20T22:36:40.261 INFO:teuthology.orchestra.run.smithi084.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-20T22:36:40.262 INFO:teuthology.orchestra.run.smithi084.stdout:systemctl: stdout disabled 2024-09-20T22:36:40.269 INFO:teuthology.orchestra.run.smithi084.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-20T22:36:40.269 INFO:teuthology.orchestra.run.smithi084.stdout:systemctl: stdout inactive 2024-09-20T22:36:40.269 INFO:teuthology.orchestra.run.smithi084.stdout:firewalld.service is not enabled 2024-09-20T22:36:40.269 INFO:teuthology.orchestra.run.smithi084.stdout:Not possible to enable service . firewalld.service is not available 2024-09-20T22:36:40.278 INFO:teuthology.orchestra.run.smithi084.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-20T22:36:40.278 INFO:teuthology.orchestra.run.smithi084.stdout:systemctl: stdout disabled 2024-09-20T22:36:40.287 INFO:teuthology.orchestra.run.smithi084.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-20T22:36:40.287 INFO:teuthology.orchestra.run.smithi084.stdout:systemctl: stdout inactive 2024-09-20T22:36:40.287 INFO:teuthology.orchestra.run.smithi084.stdout:firewalld.service is not enabled 2024-09-20T22:36:40.287 INFO:teuthology.orchestra.run.smithi084.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-09-20T22:36:40.287 INFO:teuthology.orchestra.run.smithi084.stdout:Waiting for mgr to start... 2024-09-20T22:36:40.287 INFO:teuthology.orchestra.run.smithi084.stdout:Waiting for mgr... 2024-09-20T22:36:40.525 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:36:40 smithi084 bash[29985]: 5780dd378cf0d7032d08f9f77dd852f2e85a2c7fdf3f902478f46e1e457e829d 2024-09-20T22:36:40.525 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:36:40 smithi084 systemd[1]: Started Ceph mgr.y for a9c0bc70-77a0-11ef-baf5-efdc52797490. 2024-09-20T22:36:40.526 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:36:40 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: 2024-09-20T22:36:40.280+0000 7fa97de90040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-20T22:36:40.526 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:36:40 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: 2024-09-20T22:36:40.342+0000 7fa97de90040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-20T22:36:40.526 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:36:40 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: 2024-09-20T22:36:40.491+0000 7fa97de90040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-20T22:36:40.814 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:36:40 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: 2024-09-20T22:36:40.561+0000 7fa97de90040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-20T22:36:40.814 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:36:40 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: 2024-09-20T22:36:40.710+0000 7fa97de90040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-20T22:36:41.114 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:40 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/2067523449' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-09-20T22:36:41.115 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:36:40 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: 2024-09-20T22:36:40.860+0000 7fa97de90040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-20T22:36:41.395 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout 2024-09-20T22:36:41.395 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout { 2024-09-20T22:36:41.395 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "fsid": "a9c0bc70-77a0-11ef-baf5-efdc52797490", 2024-09-20T22:36:41.396 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "health": { 2024-09-20T22:36:41.396 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-09-20T22:36:41.396 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-09-20T22:36:41.396 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-09-20T22:36:41.396 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout }, 2024-09-20T22:36:41.396 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-09-20T22:36:41.396 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-09-20T22:36:41.396 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout 0 2024-09-20T22:36:41.396 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout ], 2024-09-20T22:36:41.396 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-09-20T22:36:41.396 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "a" 2024-09-20T22:36:41.396 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout ], 2024-09-20T22:36:41.396 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-09-20T22:36:41.396 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "monmap": { 2024-09-20T22:36:41.396 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-20T22:36:41.396 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-09-20T22:36:41.397 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-09-20T22:36:41.397 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout }, 2024-09-20T22:36:41.397 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-09-20T22:36:41.397 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-20T22:36:41.397 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-09-20T22:36:41.397 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-09-20T22:36:41.397 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-09-20T22:36:41.397 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-09-20T22:36:41.397 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-09-20T22:36:41.397 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-09-20T22:36:41.397 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout }, 2024-09-20T22:36:41.397 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-09-20T22:36:41.397 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-09-20T22:36:41.397 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-09-20T22:36:41.397 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-09-20T22:36:41.397 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-09-20T22:36:41.398 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-09-20T22:36:41.398 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-09-20T22:36:41.398 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-09-20T22:36:41.398 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-09-20T22:36:41.398 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout }, 2024-09-20T22:36:41.398 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-09-20T22:36:41.398 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-20T22:36:41.398 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "btime": "2024-09-20T22:36:30:240441+0000", 2024-09-20T22:36:41.398 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-09-20T22:36:41.398 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-09-20T22:36:41.398 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout }, 2024-09-20T22:36:41.398 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-09-20T22:36:41.398 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "available": false, 2024-09-20T22:36:41.398 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-09-20T22:36:41.398 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "modules": [ 2024-09-20T22:36:41.399 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "iostat", 2024-09-20T22:36:41.399 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "nfs", 2024-09-20T22:36:41.399 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "restful" 2024-09-20T22:36:41.399 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout ], 2024-09-20T22:36:41.399 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-20T22:36:41.399 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout }, 2024-09-20T22:36:41.399 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-09-20T22:36:41.399 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-20T22:36:41.399 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "modified": "2024-09-20T22:36:30.241076+0000", 2024-09-20T22:36:41.399 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-20T22:36:41.399 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout }, 2024-09-20T22:36:41.399 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-09-20T22:36:41.399 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout } 2024-09-20T22:36:41.399 INFO:teuthology.orchestra.run.smithi084.stdout:mgr not available, waiting (1/15)... 2024-09-20T22:36:41.615 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:36:41 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: 2024-09-20T22:36:41.178+0000 7fa97de90040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-20T22:36:41.615 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:36:41 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: 2024-09-20T22:36:41.238+0000 7fa97de90040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-20T22:36:41.615 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:36:41 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: 2024-09-20T22:36:41.423+0000 7fa97de90040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-20T22:36:42.615 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:36:42 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: 2024-09-20T22:36:42.314+0000 7fa97de90040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-20T22:36:42.615 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:36:42 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: 2024-09-20T22:36:42.383+0000 7fa97de90040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-20T22:36:43.364 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:36:43 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: 2024-09-20T22:36:43.111+0000 7fa97de90040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-20T22:36:43.365 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:36:43 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: 2024-09-20T22:36:43.224+0000 7fa97de90040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-20T22:36:43.620 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:36:43 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: 2024-09-20T22:36:43.363+0000 7fa97de90040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-20T22:36:43.620 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:36:43 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: 2024-09-20T22:36:43.428+0000 7fa97de90040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-20T22:36:43.620 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:36:43 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: 2024-09-20T22:36:43.578+0000 7fa97de90040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-20T22:36:43.882 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:36:43 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: 2024-09-20T22:36:43.650+0000 7fa97de90040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-20T22:36:43.882 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:36:43 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: 2024-09-20T22:36:43.711+0000 7fa97de90040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-20T22:36:44.135 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:43 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/3160400798' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-09-20T22:36:44.136 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:36:43 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: 2024-09-20T22:36:43.915+0000 7fa97de90040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-20T22:36:44.136 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:36:43 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: 2024-09-20T22:36:43.982+0000 7fa97de90040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-20T22:36:44.136 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:36:44 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: 2024-09-20T22:36:44.048+0000 7fa97de90040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-20T22:36:44.436 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:36:44 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: 2024-09-20T22:36:44.133+0000 7fa97de90040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-20T22:36:44.437 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:36:44 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: 2024-09-20T22:36:44.193+0000 7fa97de90040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-20T22:36:44.437 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:36:44 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: 2024-09-20T22:36:44.252+0000 7fa97de90040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-20T22:36:44.444 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout 2024-09-20T22:36:44.444 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout { 2024-09-20T22:36:44.445 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "fsid": "a9c0bc70-77a0-11ef-baf5-efdc52797490", 2024-09-20T22:36:44.445 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "health": { 2024-09-20T22:36:44.445 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-09-20T22:36:44.445 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-09-20T22:36:44.445 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-09-20T22:36:44.445 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout }, 2024-09-20T22:36:44.445 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-09-20T22:36:44.445 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-09-20T22:36:44.445 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout 0 2024-09-20T22:36:44.445 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout ], 2024-09-20T22:36:44.445 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-09-20T22:36:44.445 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "a" 2024-09-20T22:36:44.445 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout ], 2024-09-20T22:36:44.445 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "quorum_age": 5, 2024-09-20T22:36:44.445 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "monmap": { 2024-09-20T22:36:44.446 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-20T22:36:44.446 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-09-20T22:36:44.446 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-09-20T22:36:44.446 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout }, 2024-09-20T22:36:44.446 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-09-20T22:36:44.446 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-20T22:36:44.446 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-09-20T22:36:44.446 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-09-20T22:36:44.446 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-09-20T22:36:44.446 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-09-20T22:36:44.446 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-09-20T22:36:44.446 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-09-20T22:36:44.446 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout }, 2024-09-20T22:36:44.446 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-09-20T22:36:44.447 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-09-20T22:36:44.447 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-09-20T22:36:44.447 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-09-20T22:36:44.447 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-09-20T22:36:44.447 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-09-20T22:36:44.447 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-09-20T22:36:44.447 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-09-20T22:36:44.447 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-09-20T22:36:44.447 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout }, 2024-09-20T22:36:44.447 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-09-20T22:36:44.447 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-20T22:36:44.447 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "btime": "2024-09-20T22:36:30:240441+0000", 2024-09-20T22:36:44.447 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-09-20T22:36:44.447 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-09-20T22:36:44.448 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout }, 2024-09-20T22:36:44.448 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-09-20T22:36:44.448 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "available": false, 2024-09-20T22:36:44.448 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-09-20T22:36:44.448 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "modules": [ 2024-09-20T22:36:44.448 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "iostat", 2024-09-20T22:36:44.448 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "nfs", 2024-09-20T22:36:44.448 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "restful" 2024-09-20T22:36:44.448 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout ], 2024-09-20T22:36:44.448 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-20T22:36:44.448 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout }, 2024-09-20T22:36:44.448 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-09-20T22:36:44.448 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-20T22:36:44.448 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "modified": "2024-09-20T22:36:30.241076+0000", 2024-09-20T22:36:44.448 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-20T22:36:44.449 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout }, 2024-09-20T22:36:44.449 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-09-20T22:36:44.449 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout } 2024-09-20T22:36:44.449 INFO:teuthology.orchestra.run.smithi084.stdout:mgr not available, waiting (2/15)... 2024-09-20T22:36:45.364 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:36:45 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: 2024-09-20T22:36:45.036+0000 7fa97de90040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-20T22:36:45.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:45 smithi084 ceph-mon[29774]: Activating manager daemon y 2024-09-20T22:36:45.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:45 smithi084 ceph-mon[29774]: mgrmap e2: y(active, starting, since 0.00285359s) 2024-09-20T22:36:45.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:45 smithi084 ceph-mon[29774]: from='mgr.14100 172.21.15.84:0/3766563069' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-20T22:36:45.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:45 smithi084 ceph-mon[29774]: from='mgr.14100 172.21.15.84:0/3766563069' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-20T22:36:45.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:45 smithi084 ceph-mon[29774]: from='mgr.14100 172.21.15.84:0/3766563069' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-20T22:36:45.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:45 smithi084 ceph-mon[29774]: from='mgr.14100 172.21.15.84:0/3766563069' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-20T22:36:45.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:45 smithi084 ceph-mon[29774]: from='mgr.14100 172.21.15.84:0/3766563069' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-20T22:36:45.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:45 smithi084 ceph-mon[29774]: Manager daemon y is now available 2024-09-20T22:36:45.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:45 smithi084 ceph-mon[29774]: from='mgr.14100 172.21.15.84:0/3766563069' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-20T22:36:45.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:45 smithi084 ceph-mon[29774]: from='mgr.14100 172.21.15.84:0/3766563069' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-20T22:36:45.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:45 smithi084 ceph-mon[29774]: from='mgr.14100 172.21.15.84:0/3766563069' entity='mgr.y' 2024-09-20T22:36:45.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:45 smithi084 ceph-mon[29774]: from='mgr.14100 172.21.15.84:0/3766563069' entity='mgr.y' 2024-09-20T22:36:45.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:45 smithi084 ceph-mon[29774]: from='mgr.14100 172.21.15.84:0/3766563069' entity='mgr.y' 2024-09-20T22:36:47.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:47 smithi084 ceph-mon[29774]: mgrmap e3: y(active, since 1.00667s) 2024-09-20T22:36:47.541 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout 2024-09-20T22:36:47.541 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout { 2024-09-20T22:36:47.541 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "fsid": "a9c0bc70-77a0-11ef-baf5-efdc52797490", 2024-09-20T22:36:47.541 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "health": { 2024-09-20T22:36:47.542 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-09-20T22:36:47.542 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-09-20T22:36:47.542 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-09-20T22:36:47.542 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout }, 2024-09-20T22:36:47.542 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-09-20T22:36:47.542 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-09-20T22:36:47.542 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout 0 2024-09-20T22:36:47.542 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout ], 2024-09-20T22:36:47.542 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-09-20T22:36:47.542 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "a" 2024-09-20T22:36:47.542 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout ], 2024-09-20T22:36:47.542 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "quorum_age": 9, 2024-09-20T22:36:47.542 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "monmap": { 2024-09-20T22:36:47.543 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-20T22:36:47.543 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-09-20T22:36:47.543 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-09-20T22:36:47.543 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout }, 2024-09-20T22:36:47.543 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-09-20T22:36:47.543 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-20T22:36:47.543 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-09-20T22:36:47.543 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-09-20T22:36:47.543 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-09-20T22:36:47.543 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-09-20T22:36:47.543 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-09-20T22:36:47.543 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-09-20T22:36:47.543 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout }, 2024-09-20T22:36:47.543 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-09-20T22:36:47.543 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-09-20T22:36:47.544 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-09-20T22:36:47.544 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-09-20T22:36:47.544 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-09-20T22:36:47.544 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-09-20T22:36:47.544 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-09-20T22:36:47.544 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-09-20T22:36:47.544 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-09-20T22:36:47.544 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout }, 2024-09-20T22:36:47.544 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-09-20T22:36:47.544 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-20T22:36:47.544 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "btime": "2024-09-20T22:36:30:240441+0000", 2024-09-20T22:36:47.544 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-09-20T22:36:47.544 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-09-20T22:36:47.544 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout }, 2024-09-20T22:36:47.544 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-09-20T22:36:47.544 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-20T22:36:47.545 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-09-20T22:36:47.545 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "modules": [ 2024-09-20T22:36:47.545 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "iostat", 2024-09-20T22:36:47.545 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "nfs", 2024-09-20T22:36:47.545 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "restful" 2024-09-20T22:36:47.545 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout ], 2024-09-20T22:36:47.545 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-20T22:36:47.545 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout }, 2024-09-20T22:36:47.545 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-09-20T22:36:47.545 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-20T22:36:47.545 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "modified": "2024-09-20T22:36:30.241076+0000", 2024-09-20T22:36:47.545 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-20T22:36:47.545 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout }, 2024-09-20T22:36:47.545 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-09-20T22:36:47.545 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout } 2024-09-20T22:36:47.545 INFO:teuthology.orchestra.run.smithi084.stdout:mgr is available 2024-09-20T22:36:48.090 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:48 smithi084 ceph-mon[29774]: mgrmap e4: y(active, since 2s) 2024-09-20T22:36:48.090 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:48 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/1343986212' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-09-20T22:36:48.652 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout 2024-09-20T22:36:48.653 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout [global] 2024-09-20T22:36:48.653 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout fsid = a9c0bc70-77a0-11ef-baf5-efdc52797490 2024-09-20T22:36:48.653 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.84:3300,v1:172.21.15.84:6789] 2024-09-20T22:36:48.653 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-09-20T22:36:48.653 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-09-20T22:36:48.653 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-09-20T22:36:48.653 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-09-20T22:36:48.653 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout 2024-09-20T22:36:48.653 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout [mgr] 2024-09-20T22:36:48.653 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-09-20T22:36:48.653 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout 2024-09-20T22:36:48.653 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout [osd] 2024-09-20T22:36:48.653 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-09-20T22:36:48.653 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-09-20T22:36:48.653 INFO:teuthology.orchestra.run.smithi084.stdout:Enabling cephadm module... 2024-09-20T22:36:49.364 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:49 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/2640092117' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-09-20T22:36:50.363 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:50 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/3768507922' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-09-20T22:36:50.364 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:36:50 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: ignoring --setuser ceph since I am not root 2024-09-20T22:36:50.364 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:36:50 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: ignoring --setgroup ceph since I am not root 2024-09-20T22:36:50.364 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:36:50 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: 2024-09-20T22:36:50.190+0000 7f6750759040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-20T22:36:50.364 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:36:50 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: 2024-09-20T22:36:50.247+0000 7f6750759040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-20T22:36:50.614 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:36:50 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: 2024-09-20T22:36:50.398+0000 7f6750759040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-20T22:36:50.615 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:36:50 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: 2024-09-20T22:36:50.466+0000 7f6750759040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-20T22:36:50.891 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:36:50 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: 2024-09-20T22:36:50.616+0000 7f6750759040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-20T22:36:50.891 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:36:50 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: 2024-09-20T22:36:50.760+0000 7f6750759040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-20T22:36:51.151 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:51 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/3768507922' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-09-20T22:36:51.151 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:51 smithi084 ceph-mon[29774]: mgrmap e5: y(active, since 5s) 2024-09-20T22:36:51.151 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:36:51 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: 2024-09-20T22:36:51.087+0000 7f6750759040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-20T22:36:51.494 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:36:51 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: 2024-09-20T22:36:51.149+0000 7f6750759040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-20T22:36:51.494 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:36:51 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: 2024-09-20T22:36:51.341+0000 7f6750759040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-20T22:36:51.773 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout { 2024-09-20T22:36:51.774 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-09-20T22:36:51.774 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-20T22:36:51.774 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-09-20T22:36:51.774 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-09-20T22:36:51.774 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout } 2024-09-20T22:36:51.774 INFO:teuthology.orchestra.run.smithi084.stdout:Waiting for the mgr to restart... 2024-09-20T22:36:51.774 INFO:teuthology.orchestra.run.smithi084.stdout:Waiting for mgr epoch 5... 2024-09-20T22:36:52.322 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:52 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/3469939428' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-09-20T22:36:52.322 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:36:52 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: 2024-09-20T22:36:52.251+0000 7f6750759040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-20T22:36:52.323 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:36:52 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: 2024-09-20T22:36:52.320+0000 7f6750759040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-20T22:36:53.298 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:36:53 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: 2024-09-20T22:36:53.032+0000 7f6750759040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-20T22:36:53.298 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:36:53 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: 2024-09-20T22:36:53.153+0000 7f6750759040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-20T22:36:53.578 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:36:53 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: 2024-09-20T22:36:53.296+0000 7f6750759040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-20T22:36:53.578 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:36:53 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: 2024-09-20T22:36:53.358+0000 7f6750759040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-20T22:36:53.578 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:36:53 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: 2024-09-20T22:36:53.508+0000 7f6750759040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-20T22:36:53.578 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:36:53 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: 2024-09-20T22:36:53.576+0000 7f6750759040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-20T22:36:53.863 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:36:53 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: 2024-09-20T22:36:53.637+0000 7f6750759040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-20T22:36:53.863 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:36:53 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: 2024-09-20T22:36:53.796+0000 7f6750759040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-20T22:36:53.863 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:36:53 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: 2024-09-20T22:36:53.861+0000 7f6750759040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-20T22:36:54.114 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:36:53 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: 2024-09-20T22:36:53.927+0000 7f6750759040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-20T22:36:54.115 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:36:54 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: 2024-09-20T22:36:54.008+0000 7f6750759040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-20T22:36:54.115 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:36:54 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: 2024-09-20T22:36:54.068+0000 7f6750759040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-20T22:36:54.614 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:36:54 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: 2024-09-20T22:36:54.128+0000 7f6750759040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-20T22:36:55.364 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:36:54 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: 2024-09-20T22:36:54.915+0000 7f6750759040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-20T22:36:55.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:54 smithi084 ceph-mon[29774]: Active manager daemon y restarted 2024-09-20T22:36:55.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:54 smithi084 ceph-mon[29774]: Activating manager daemon y 2024-09-20T22:36:55.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:54 smithi084 ceph-mon[29774]: osdmap e2: 0 total, 0 up, 0 in 2024-09-20T22:36:55.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:54 smithi084 ceph-mon[29774]: mgrmap e6: y(active, starting, since 0.00322476s) 2024-09-20T22:36:55.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:54 smithi084 ceph-mon[29774]: from='mgr.14118 172.21.15.84:0/341853492' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-20T22:36:55.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:54 smithi084 ceph-mon[29774]: from='mgr.14118 172.21.15.84:0/341853492' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-20T22:36:55.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:54 smithi084 ceph-mon[29774]: from='mgr.14118 172.21.15.84:0/341853492' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-20T22:36:55.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:54 smithi084 ceph-mon[29774]: from='mgr.14118 172.21.15.84:0/341853492' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-20T22:36:55.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:54 smithi084 ceph-mon[29774]: from='mgr.14118 172.21.15.84:0/341853492' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-20T22:36:55.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:54 smithi084 ceph-mon[29774]: Manager daemon y is now available 2024-09-20T22:36:55.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:54 smithi084 ceph-mon[29774]: from='mgr.14118 172.21.15.84:0/341853492' entity='mgr.y' 2024-09-20T22:36:55.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:54 smithi084 ceph-mon[29774]: from='mgr.14118 172.21.15.84:0/341853492' entity='mgr.y' 2024-09-20T22:36:55.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:54 smithi084 ceph-mon[29774]: from='mgr.14118 172.21.15.84:0/341853492' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:36:55.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:54 smithi084 ceph-mon[29774]: from='mgr.14118 172.21.15.84:0/341853492' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:36:55.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:54 smithi084 ceph-mon[29774]: from='mgr.14118 172.21.15.84:0/341853492' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-20T22:36:55.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:54 smithi084 ceph-mon[29774]: from='mgr.14118 172.21.15.84:0/341853492' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-20T22:36:56.226 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:55 smithi084 ceph-mon[29774]: Found migration_current of "None". Setting to last migration. 2024-09-20T22:36:56.226 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:55 smithi084 ceph-mon[29774]: from='mgr.14118 172.21.15.84:0/341853492' entity='mgr.y' 2024-09-20T22:36:56.226 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:55 smithi084 ceph-mon[29774]: from='mgr.14118 172.21.15.84:0/341853492' entity='mgr.y' 2024-09-20T22:36:56.226 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:55 smithi084 ceph-mon[29774]: mgrmap e7: y(active, since 1.00507s) 2024-09-20T22:36:56.471 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout { 2024-09-20T22:36:56.472 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-09-20T22:36:56.472 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "initialized": true 2024-09-20T22:36:56.472 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout } 2024-09-20T22:36:56.472 INFO:teuthology.orchestra.run.smithi084.stdout:mgr epoch 5 is available 2024-09-20T22:36:56.472 INFO:teuthology.orchestra.run.smithi084.stdout:Setting orchestrator backend to cephadm... 2024-09-20T22:36:57.356 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:56 smithi084 ceph-mon[29774]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-09-20T22:36:57.357 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:56 smithi084 ceph-mon[29774]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-09-20T22:36:57.357 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:56 smithi084 ceph-mon[29774]: from='mgr.14118 172.21.15.84:0/341853492' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:36:57.357 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:36:56 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: 2024-09-20T22:36:56.901+0000 7f66b7ab3640 -1 log_channel(cephadm) log [ERR] : [20/Sep/2024:22:36:56] ENGINE Error in HTTPServer.serve 2024-09-20T22:36:57.357 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:36:56 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: Traceback (most recent call last): 2024-09-20T22:36:57.357 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:36:56 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-20T22:36:57.357 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:36:56 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: self._connections.run(self.expiration_interval) 2024-09-20T22:36:57.357 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:36:56 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-20T22:36:57.357 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:36:56 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: self._run(expiration_interval) 2024-09-20T22:36:57.357 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:36:56 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-20T22:36:57.358 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:36:56 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: new_conn = self._from_server_socket(self.server.socket) 2024-09-20T22:36:57.358 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:36:56 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-20T22:36:57.358 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:36:56 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-20T22:36:57.358 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:36:56 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-20T22:36:57.358 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:36:56 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: s = self.context.wrap_socket( 2024-09-20T22:36:57.358 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:36:56 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-20T22:36:57.358 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:36:56 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: return self.sslsocket_class._create( 2024-09-20T22:36:57.358 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:36:56 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-20T22:36:57.358 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:36:56 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: self.do_handshake() 2024-09-20T22:36:57.358 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:36:56 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-20T22:36:57.358 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:36:56 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: self._sslobj.do_handshake() 2024-09-20T22:36:57.358 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:36:56 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-20T22:36:57.358 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:36:56 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: 2024-09-20T22:36:58.364 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:57 smithi084 ceph-mon[29774]: [20/Sep/2024:22:36:56] ENGINE Bus STARTING 2024-09-20T22:36:58.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:57 smithi084 ceph-mon[29774]: [20/Sep/2024:22:36:56] ENGINE Serving on http://172.21.15.84:8765 2024-09-20T22:36:58.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:57 smithi084 ceph-mon[29774]: [20/Sep/2024:22:36:56] ENGINE Serving on https://172.21.15.84:7150 2024-09-20T22:36:58.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:57 smithi084 ceph-mon[29774]: [20/Sep/2024:22:36:56] ENGINE Bus STARTED 2024-09-20T22:36:58.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:57 smithi084 ceph-mon[29774]: [20/Sep/2024:22:36:56] ENGINE Error in HTTPServer.serve 2024-09-20T22:36:58.365 INFO:journalctl@ceph.mon.a.smithi084.stdout: Traceback (most recent call last): 2024-09-20T22:36:58.365 INFO:journalctl@ceph.mon.a.smithi084.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-20T22:36:58.365 INFO:journalctl@ceph.mon.a.smithi084.stdout: self._connections.run(self.expiration_interval) 2024-09-20T22:36:58.365 INFO:journalctl@ceph.mon.a.smithi084.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-20T22:36:58.365 INFO:journalctl@ceph.mon.a.smithi084.stdout: self._run(expiration_interval) 2024-09-20T22:36:58.365 INFO:journalctl@ceph.mon.a.smithi084.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-20T22:36:58.365 INFO:journalctl@ceph.mon.a.smithi084.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-20T22:36:58.366 INFO:journalctl@ceph.mon.a.smithi084.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-20T22:36:58.366 INFO:journalctl@ceph.mon.a.smithi084.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-20T22:36:58.366 INFO:journalctl@ceph.mon.a.smithi084.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-20T22:36:58.366 INFO:journalctl@ceph.mon.a.smithi084.stdout: s = self.context.wrap_socket( 2024-09-20T22:36:58.366 INFO:journalctl@ceph.mon.a.smithi084.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-20T22:36:58.366 INFO:journalctl@ceph.mon.a.smithi084.stdout: return self.sslsocket_class._create( 2024-09-20T22:36:58.366 INFO:journalctl@ceph.mon.a.smithi084.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-20T22:36:58.366 INFO:journalctl@ceph.mon.a.smithi084.stdout: self.do_handshake() 2024-09-20T22:36:58.366 INFO:journalctl@ceph.mon.a.smithi084.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-20T22:36:58.366 INFO:journalctl@ceph.mon.a.smithi084.stdout: self._sslobj.do_handshake() 2024-09-20T22:36:58.366 INFO:journalctl@ceph.mon.a.smithi084.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-20T22:36:58.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:57 smithi084 ceph-mon[29774]: mgrmap e8: y(active, since 2s) 2024-09-20T22:36:58.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:57 smithi084 ceph-mon[29774]: from='client.14130 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:36:58.367 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:57 smithi084 ceph-mon[29774]: from='mgr.14118 172.21.15.84:0/341853492' entity='mgr.y' 2024-09-20T22:36:58.367 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:57 smithi084 ceph-mon[29774]: from='mgr.14118 172.21.15.84:0/341853492' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:36:58.658 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout value unchanged 2024-09-20T22:36:58.658 INFO:teuthology.orchestra.run.smithi084.stdout:Generating ssh key... 2024-09-20T22:36:59.300 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:36:59 smithi084 ceph-mon[29774]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:36:59.595 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:36:59 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: Generating public/private rsa key pair. 2024-09-20T22:36:59.595 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:36:59 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: Your identification has been saved in /tmp/tmpw3h82bmz/key 2024-09-20T22:36:59.595 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:36:59 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: Your public key has been saved in /tmp/tmpw3h82bmz/key.pub 2024-09-20T22:36:59.595 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:36:59 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: The key fingerprint is: 2024-09-20T22:36:59.595 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:36:59 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: SHA256:FCMZ98+Gm62WRYCeag/xQT0mu4fE3ZcW6kqRIBC+B/I ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490 2024-09-20T22:36:59.595 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:36:59 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: The key's randomart image is: 2024-09-20T22:36:59.595 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:36:59 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: +---[RSA 3072]----+ 2024-09-20T22:36:59.595 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:36:59 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: | oooo+o | 2024-09-20T22:36:59.595 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:36:59 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: | . oo=+= . | 2024-09-20T22:36:59.596 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:36:59 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: | . o =.B.= . o | 2024-09-20T22:36:59.596 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:36:59 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: | o o..B +++ + | 2024-09-20T22:36:59.596 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:36:59 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: | E .=S+.++o | 2024-09-20T22:36:59.596 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:36:59 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: | .+ + o=o | 2024-09-20T22:36:59.596 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:36:59 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: | . o oo+. | 2024-09-20T22:36:59.596 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:36:59 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: | . +. | 2024-09-20T22:36:59.596 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:36:59 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: | .. | 2024-09-20T22:36:59.596 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:36:59 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: +----[SHA256]-----+ 2024-09-20T22:37:00.441 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:00 smithi084 ceph-mon[29774]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:37:00.442 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:00 smithi084 ceph-mon[29774]: Generating ssh key... 2024-09-20T22:37:00.442 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:00 smithi084 ceph-mon[29774]: from='mgr.14118 172.21.15.84:0/341853492' entity='mgr.y' 2024-09-20T22:37:00.442 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:00 smithi084 ceph-mon[29774]: from='mgr.14118 172.21.15.84:0/341853492' entity='mgr.y' 2024-09-20T22:37:00.996 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCbxQWY5ggD0Q31Py9zN8ggan/SG8UQgqJ9nDMd96yPErTmsLqzVcL5LlCBQpKbgte7KOoL5KNxLL/zsvG39ZrwZgpdtZWBHa0LkQj456h4EmzDFLjpKFm4NduObbnDCSii4CyQ/r92zanPkFJVywx1lMLaidSUPTUJb9yN/QDgA/k3K1mkisw9Xdfx+qQ4Q3xV3cSmu9sQYPd11FFUdcYAf4TWj0rKRp8NI+z43yq5QwUkDWxGVkBv1wgfFGOEOnShj8Wwv9LXSegAlUnKVZN0I+xNtA63hWBS9AzE9T15fjJ61u1Dj2nGYXpz0eWOWAQKc6DfsRBrhQmsZent4Bs987H3ak0UTJRINtV0IuSzLJURd8PA2TbTd7b6+KAvtLmJBwpJirOHfncPYBCikDoTk7c5572ar7LNa19ruwTwr/DA2vsndn3d0P6Fzi9luJr+MRUHVEKFRIJqQF4TrwGbBHxbaPIEfItkyK/Tj7HHUkAK+GeM7boDAuMP6Zen/QM= ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490 2024-09-20T22:37:00.997 INFO:teuthology.orchestra.run.smithi084.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-09-20T22:37:00.997 INFO:teuthology.orchestra.run.smithi084.stdout:Adding key to root@localhost authorized_keys... 2024-09-20T22:37:01.003 INFO:teuthology.orchestra.run.smithi084.stdout:Adding host smithi084... 2024-09-20T22:37:01.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:01 smithi084 ceph-mon[29774]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:37:02.362 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:02 smithi084 ceph-mon[29774]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi084", "addr": "172.21.15.84", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:37:03.614 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:03 smithi084 ceph-mon[29774]: Deploying cephadm binary to smithi084 2024-09-20T22:37:05.312 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout Added host 'smithi084' with addr '172.21.15.84' 2024-09-20T22:37:05.312 INFO:teuthology.orchestra.run.smithi084.stdout:Deploying unmanaged mon service... 2024-09-20T22:37:05.864 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:05 smithi084 ceph-mon[29774]: from='mgr.14118 172.21.15.84:0/341853492' entity='mgr.y' 2024-09-20T22:37:05.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:05 smithi084 ceph-mon[29774]: Added host smithi084 2024-09-20T22:37:05.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:05 smithi084 ceph-mon[29774]: from='mgr.14118 172.21.15.84:0/341853492' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:37:06.662 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-09-20T22:37:06.662 INFO:teuthology.orchestra.run.smithi084.stdout:Deploying unmanaged mgr service... 2024-09-20T22:37:07.114 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:06 smithi084 ceph-mon[29774]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:37:07.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:06 smithi084 ceph-mon[29774]: Saving service mon spec with placement count:5 2024-09-20T22:37:07.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:06 smithi084 ceph-mon[29774]: from='mgr.14118 172.21.15.84:0/341853492' entity='mgr.y' 2024-09-20T22:37:08.614 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:08 smithi084 ceph-mon[29774]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:37:08.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:08 smithi084 ceph-mon[29774]: Saving service mgr spec with placement count:2 2024-09-20T22:37:08.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:08 smithi084 ceph-mon[29774]: from='mgr.14118 172.21.15.84:0/341853492' entity='mgr.y' 2024-09-20T22:37:08.666 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-09-20T22:37:09.603 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:09 smithi084 ceph-mon[29774]: from='mgr.14118 172.21.15.84:0/341853492' entity='mgr.y' 2024-09-20T22:37:09.603 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:09 smithi084 ceph-mon[29774]: from='mgr.14118 172.21.15.84:0/341853492' entity='mgr.y' 2024-09-20T22:37:10.614 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:10 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/2408588549' entity='client.admin' 2024-09-20T22:37:11.499 INFO:teuthology.orchestra.run.smithi084.stdout:Enabling the dashboard module... 2024-09-20T22:37:12.114 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:11 smithi084 ceph-mon[29774]: from='mgr.14118 172.21.15.84:0/341853492' entity='mgr.y' 2024-09-20T22:37:12.114 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:11 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/4226514694' entity='client.admin' 2024-09-20T22:37:12.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:11 smithi084 ceph-mon[29774]: from='mgr.14118 172.21.15.84:0/341853492' entity='mgr.y' 2024-09-20T22:37:13.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:12 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/3711879136' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-09-20T22:37:13.115 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:37:12 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: ignoring --setuser ceph since I am not root 2024-09-20T22:37:13.115 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:37:12 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: ignoring --setgroup ceph since I am not root 2024-09-20T22:37:13.115 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:37:12 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: 2024-09-20T22:37:12.985+0000 7f366d8f1040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-20T22:37:13.115 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:37:13 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: 2024-09-20T22:37:13.043+0000 7f366d8f1040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-20T22:37:13.503 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:37:13 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: 2024-09-20T22:37:13.193+0000 7f366d8f1040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-20T22:37:13.503 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:37:13 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: 2024-09-20T22:37:13.259+0000 7f366d8f1040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-20T22:37:13.503 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:37:13 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: 2024-09-20T22:37:13.406+0000 7f366d8f1040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-20T22:37:13.758 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:37:13 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: 2024-09-20T22:37:13.556+0000 7f366d8f1040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-20T22:37:14.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:13 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/3711879136' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-09-20T22:37:14.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:13 smithi084 ceph-mon[29774]: mgrmap e9: y(active, since 17s) 2024-09-20T22:37:14.115 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:37:13 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: 2024-09-20T22:37:13.878+0000 7f366d8f1040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-20T22:37:14.115 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:37:13 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: 2024-09-20T22:37:13.942+0000 7f366d8f1040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-20T22:37:14.458 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:37:14 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: 2024-09-20T22:37:14.133+0000 7f366d8f1040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-20T22:37:14.745 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout { 2024-09-20T22:37:14.745 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-09-20T22:37:14.745 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-20T22:37:14.746 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-09-20T22:37:14.746 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-09-20T22:37:14.746 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout } 2024-09-20T22:37:14.746 INFO:teuthology.orchestra.run.smithi084.stdout:Waiting for the mgr to restart... 2024-09-20T22:37:14.746 INFO:teuthology.orchestra.run.smithi084.stdout:Waiting for mgr epoch 9... 2024-09-20T22:37:15.039 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:14 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/1312819563' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-09-20T22:37:15.364 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:37:15 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: 2024-09-20T22:37:15.037+0000 7f366d8f1040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-20T22:37:15.365 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:37:15 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: 2024-09-20T22:37:15.109+0000 7f366d8f1040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-20T22:37:16.081 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:37:15 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: 2024-09-20T22:37:15.822+0000 7f366d8f1040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-20T22:37:16.081 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:37:15 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: 2024-09-20T22:37:15.937+0000 7f366d8f1040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-20T22:37:16.357 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:37:16 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: 2024-09-20T22:37:16.079+0000 7f366d8f1040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-20T22:37:16.358 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:37:16 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: 2024-09-20T22:37:16.140+0000 7f366d8f1040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-20T22:37:16.358 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:37:16 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: 2024-09-20T22:37:16.287+0000 7f366d8f1040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-20T22:37:16.358 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:37:16 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: 2024-09-20T22:37:16.355+0000 7f366d8f1040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-20T22:37:16.615 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:37:16 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: 2024-09-20T22:37:16.416+0000 7f366d8f1040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-20T22:37:16.615 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:37:16 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: 2024-09-20T22:37:16.574+0000 7f366d8f1040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-20T22:37:16.906 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:37:16 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: 2024-09-20T22:37:16.640+0000 7f366d8f1040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-20T22:37:16.906 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:37:16 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: 2024-09-20T22:37:16.706+0000 7f366d8f1040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-20T22:37:16.906 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:37:16 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: 2024-09-20T22:37:16.788+0000 7f366d8f1040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-20T22:37:16.906 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:37:16 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: 2024-09-20T22:37:16.846+0000 7f366d8f1040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-20T22:37:16.906 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:37:16 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: 2024-09-20T22:37:16.904+0000 7f366d8f1040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-20T22:37:18.114 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:37:17 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: 2024-09-20T22:37:17.688+0000 7f366d8f1040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-20T22:37:18.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:17 smithi084 ceph-mon[29774]: Active manager daemon y restarted 2024-09-20T22:37:18.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:17 smithi084 ceph-mon[29774]: Activating manager daemon y 2024-09-20T22:37:18.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:17 smithi084 ceph-mon[29774]: osdmap e3: 0 total, 0 up, 0 in 2024-09-20T22:37:18.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:17 smithi084 ceph-mon[29774]: mgrmap e10: y(active, starting, since 0.0035108s) 2024-09-20T22:37:18.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:17 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-20T22:37:18.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:17 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-20T22:37:18.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:17 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-20T22:37:18.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:17 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-20T22:37:18.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:17 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-20T22:37:18.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:17 smithi084 ceph-mon[29774]: Manager daemon y is now available 2024-09-20T22:37:18.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:17 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:37:18.116 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:17 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-20T22:37:18.116 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:17 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-20T22:37:19.115 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:37:18 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: 2024-09-20T22:37:18.813+0000 7f35d33ba640 -1 log_channel(cephadm) log [ERR] : [20/Sep/2024:22:37:18] ENGINE Error in HTTPServer.serve 2024-09-20T22:37:19.115 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:37:18 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: Traceback (most recent call last): 2024-09-20T22:37:19.115 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:37:18 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-20T22:37:19.115 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:37:18 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: self._connections.run(self.expiration_interval) 2024-09-20T22:37:19.115 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:37:18 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-20T22:37:19.116 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:37:18 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: self._run(expiration_interval) 2024-09-20T22:37:19.116 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:37:18 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-20T22:37:19.116 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:37:18 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: new_conn = self._from_server_socket(self.server.socket) 2024-09-20T22:37:19.116 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:37:18 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-20T22:37:19.116 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:37:18 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-20T22:37:19.116 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:37:18 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-20T22:37:19.116 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:37:18 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: s = self.context.wrap_socket( 2024-09-20T22:37:19.116 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:37:18 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-20T22:37:19.116 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:37:18 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: return self.sslsocket_class._create( 2024-09-20T22:37:19.116 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:37:18 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-20T22:37:19.116 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:37:18 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: self.do_handshake() 2024-09-20T22:37:19.116 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:37:18 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-20T22:37:19.116 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:37:18 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: self._sslobj.do_handshake() 2024-09-20T22:37:19.116 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:37:18 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-20T22:37:19.117 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:37:18 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: 2024-09-20T22:37:19.387 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout { 2024-09-20T22:37:19.388 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-09-20T22:37:19.388 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout "initialized": true 2024-09-20T22:37:19.388 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout } 2024-09-20T22:37:19.388 INFO:teuthology.orchestra.run.smithi084.stdout:mgr epoch 9 is available 2024-09-20T22:37:19.388 INFO:teuthology.orchestra.run.smithi084.stdout:Generating a dashboard self-signed certificate... 2024-09-20T22:37:19.771 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:19 smithi084 ceph-mon[29774]: [20/Sep/2024:22:37:18] ENGINE Bus STARTING 2024-09-20T22:37:19.772 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:19 smithi084 ceph-mon[29774]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-09-20T22:37:19.772 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:19 smithi084 ceph-mon[29774]: [20/Sep/2024:22:37:18] ENGINE Serving on http://172.21.15.84:8765 2024-09-20T22:37:19.772 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:19 smithi084 ceph-mon[29774]: mgrmap e11: y(active, since 1.00572s) 2024-09-20T22:37:19.772 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:19 smithi084 ceph-mon[29774]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-09-20T22:37:19.772 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:19 smithi084 ceph-mon[29774]: [20/Sep/2024:22:37:18] ENGINE Serving on https://172.21.15.84:7150 2024-09-20T22:37:19.772 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:19 smithi084 ceph-mon[29774]: [20/Sep/2024:22:37:18] ENGINE Bus STARTED 2024-09-20T22:37:19.772 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:19 smithi084 ceph-mon[29774]: [20/Sep/2024:22:37:18] ENGINE Error in HTTPServer.serve 2024-09-20T22:37:19.772 INFO:journalctl@ceph.mon.a.smithi084.stdout: Traceback (most recent call last): 2024-09-20T22:37:19.772 INFO:journalctl@ceph.mon.a.smithi084.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-20T22:37:19.772 INFO:journalctl@ceph.mon.a.smithi084.stdout: self._connections.run(self.expiration_interval) 2024-09-20T22:37:19.773 INFO:journalctl@ceph.mon.a.smithi084.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-20T22:37:19.773 INFO:journalctl@ceph.mon.a.smithi084.stdout: self._run(expiration_interval) 2024-09-20T22:37:19.773 INFO:journalctl@ceph.mon.a.smithi084.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-20T22:37:19.773 INFO:journalctl@ceph.mon.a.smithi084.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-20T22:37:19.773 INFO:journalctl@ceph.mon.a.smithi084.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-20T22:37:19.773 INFO:journalctl@ceph.mon.a.smithi084.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-20T22:37:19.773 INFO:journalctl@ceph.mon.a.smithi084.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-20T22:37:19.773 INFO:journalctl@ceph.mon.a.smithi084.stdout: s = self.context.wrap_socket( 2024-09-20T22:37:19.773 INFO:journalctl@ceph.mon.a.smithi084.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-20T22:37:19.773 INFO:journalctl@ceph.mon.a.smithi084.stdout: return self.sslsocket_class._create( 2024-09-20T22:37:19.773 INFO:journalctl@ceph.mon.a.smithi084.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-20T22:37:19.773 INFO:journalctl@ceph.mon.a.smithi084.stdout: self.do_handshake() 2024-09-20T22:37:19.773 INFO:journalctl@ceph.mon.a.smithi084.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-20T22:37:19.773 INFO:journalctl@ceph.mon.a.smithi084.stdout: self._sslobj.do_handshake() 2024-09-20T22:37:19.774 INFO:journalctl@ceph.mon.a.smithi084.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-20T22:37:20.586 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-09-20T22:37:20.586 INFO:teuthology.orchestra.run.smithi084.stdout:Creating initial admin user... 2024-09-20T22:37:20.785 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:20 smithi084 ceph-mon[29774]: mgrmap e12: y(active, since 2s) 2024-09-20T22:37:20.785 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:20 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:37:20.785 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:20 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:37:20.785 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:20 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:37:20.785 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:20 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:37:22.108 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:21 smithi084 ceph-mon[29774]: from='client.14162 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:37:22.109 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:21 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:37:22.146 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$.2VaDxsIHREp9tGPzoqJWOz1D9v0YXZkq8JWvusIz3PEIKCslNLCK", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1726871841, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-09-20T22:37:22.147 INFO:teuthology.orchestra.run.smithi084.stdout:Fetching dashboard port number... 2024-09-20T22:37:22.840 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:22 smithi084 ceph-mon[29774]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:37:23.797 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stdout 8443 2024-09-20T22:37:23.808 INFO:teuthology.orchestra.run.smithi084.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-20T22:37:23.808 INFO:teuthology.orchestra.run.smithi084.stdout:systemctl: stdout disabled 2024-09-20T22:37:23.815 INFO:teuthology.orchestra.run.smithi084.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-20T22:37:23.815 INFO:teuthology.orchestra.run.smithi084.stdout:systemctl: stdout inactive 2024-09-20T22:37:23.815 INFO:teuthology.orchestra.run.smithi084.stdout:firewalld.service is not enabled 2024-09-20T22:37:23.816 INFO:teuthology.orchestra.run.smithi084.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-09-20T22:37:23.817 INFO:teuthology.orchestra.run.smithi084.stdout:Ceph Dashboard is now available at: 2024-09-20T22:37:23.817 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-20T22:37:23.817 INFO:teuthology.orchestra.run.smithi084.stdout: URL: https://smithi084.front.sepia.ceph.com:8443/ 2024-09-20T22:37:23.817 INFO:teuthology.orchestra.run.smithi084.stdout: User: admin 2024-09-20T22:37:23.817 INFO:teuthology.orchestra.run.smithi084.stdout: Password: q2eq3k8ehp 2024-09-20T22:37:23.817 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-20T22:37:23.818 INFO:teuthology.orchestra.run.smithi084.stdout:Saving cluster configuration to /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/config directory 2024-09-20T22:37:24.019 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:23 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/1786769922' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-09-20T22:37:24.784 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:24 smithi084 ceph-mon[29774]: mgrmap e13: y(active, since 6s) 2024-09-20T22:37:24.784 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:24 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/3462377453' entity='client.admin' 2024-09-20T22:37:25.004 INFO:teuthology.orchestra.run.smithi084.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-09-20T22:37:25.005 INFO:teuthology.orchestra.run.smithi084.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-09-20T22:37:25.005 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-20T22:37:25.005 INFO:teuthology.orchestra.run.smithi084.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-09-20T22:37:25.005 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-20T22:37:25.005 INFO:teuthology.orchestra.run.smithi084.stdout:Or, if you are only running a single cluster on this host: 2024-09-20T22:37:25.005 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-20T22:37:25.005 INFO:teuthology.orchestra.run.smithi084.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-09-20T22:37:25.005 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-20T22:37:25.005 INFO:teuthology.orchestra.run.smithi084.stdout:Please consider enabling telemetry to help improve Ceph: 2024-09-20T22:37:25.005 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-20T22:37:25.005 INFO:teuthology.orchestra.run.smithi084.stdout: ceph telemetry on 2024-09-20T22:37:25.005 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-20T22:37:25.006 INFO:teuthology.orchestra.run.smithi084.stdout:For more information see: 2024-09-20T22:37:25.006 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-20T22:37:25.006 INFO:teuthology.orchestra.run.smithi084.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-09-20T22:37:25.006 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-20T22:37:25.006 INFO:teuthology.orchestra.run.smithi084.stdout:Bootstrap complete. 2024-09-20T22:37:25.045 INFO:tasks.cephadm:Fetching config... 2024-09-20T22:37:25.046 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-09-20T22:37:25.046 DEBUG:teuthology.orchestra.run.smithi084:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-09-20T22:37:25.062 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-09-20T22:37:25.062 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-09-20T22:37:25.062 DEBUG:teuthology.orchestra.run.smithi084:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-09-20T22:37:25.117 INFO:tasks.cephadm:Fetching mon keyring... 2024-09-20T22:37:25.117 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-09-20T22:37:25.117 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd if=/var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/keyring of=/dev/stdout 2024-09-20T22:37:25.181 INFO:tasks.cephadm:Fetching pub ssh key... 2024-09-20T22:37:25.181 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-09-20T22:37:25.181 DEBUG:teuthology.orchestra.run.smithi084:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-09-20T22:37:25.236 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-09-20T22:37:25.237 DEBUG:teuthology.orchestra.run.smithi084:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCbxQWY5ggD0Q31Py9zN8ggan/SG8UQgqJ9nDMd96yPErTmsLqzVcL5LlCBQpKbgte7KOoL5KNxLL/zsvG39ZrwZgpdtZWBHa0LkQj456h4EmzDFLjpKFm4NduObbnDCSii4CyQ/r92zanPkFJVywx1lMLaidSUPTUJb9yN/QDgA/k3K1mkisw9Xdfx+qQ4Q3xV3cSmu9sQYPd11FFUdcYAf4TWj0rKRp8NI+z43yq5QwUkDWxGVkBv1wgfFGOEOnShj8Wwv9LXSegAlUnKVZN0I+xNtA63hWBS9AzE9T15fjJ61u1Dj2nGYXpz0eWOWAQKc6DfsRBrhQmsZent4Bs987H3ak0UTJRINtV0IuSzLJURd8PA2TbTd7b6+KAvtLmJBwpJirOHfncPYBCikDoTk7c5572ar7LNa19ruwTwr/DA2vsndn3d0P6Fzi9luJr+MRUHVEKFRIJqQF4TrwGbBHxbaPIEfItkyK/Tj7HHUkAK+GeM7boDAuMP6Zen/QM= ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-09-20T22:37:25.310 INFO:teuthology.orchestra.run.smithi084.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCbxQWY5ggD0Q31Py9zN8ggan/SG8UQgqJ9nDMd96yPErTmsLqzVcL5LlCBQpKbgte7KOoL5KNxLL/zsvG39ZrwZgpdtZWBHa0LkQj456h4EmzDFLjpKFm4NduObbnDCSii4CyQ/r92zanPkFJVywx1lMLaidSUPTUJb9yN/QDgA/k3K1mkisw9Xdfx+qQ4Q3xV3cSmu9sQYPd11FFUdcYAf4TWj0rKRp8NI+z43yq5QwUkDWxGVkBv1wgfFGOEOnShj8Wwv9LXSegAlUnKVZN0I+xNtA63hWBS9AzE9T15fjJ61u1Dj2nGYXpz0eWOWAQKc6DfsRBrhQmsZent4Bs987H3ak0UTJRINtV0IuSzLJURd8PA2TbTd7b6+KAvtLmJBwpJirOHfncPYBCikDoTk7c5572ar7LNa19ruwTwr/DA2vsndn3d0P6Fzi9luJr+MRUHVEKFRIJqQF4TrwGbBHxbaPIEfItkyK/Tj7HHUkAK+GeM7boDAuMP6Zen/QM= ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490 2024-09-20T22:37:25.321 DEBUG:teuthology.orchestra.run.smithi154:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCbxQWY5ggD0Q31Py9zN8ggan/SG8UQgqJ9nDMd96yPErTmsLqzVcL5LlCBQpKbgte7KOoL5KNxLL/zsvG39ZrwZgpdtZWBHa0LkQj456h4EmzDFLjpKFm4NduObbnDCSii4CyQ/r92zanPkFJVywx1lMLaidSUPTUJb9yN/QDgA/k3K1mkisw9Xdfx+qQ4Q3xV3cSmu9sQYPd11FFUdcYAf4TWj0rKRp8NI+z43yq5QwUkDWxGVkBv1wgfFGOEOnShj8Wwv9LXSegAlUnKVZN0I+xNtA63hWBS9AzE9T15fjJ61u1Dj2nGYXpz0eWOWAQKc6DfsRBrhQmsZent4Bs987H3ak0UTJRINtV0IuSzLJURd8PA2TbTd7b6+KAvtLmJBwpJirOHfncPYBCikDoTk7c5572ar7LNa19ruwTwr/DA2vsndn3d0P6Fzi9luJr+MRUHVEKFRIJqQF4TrwGbBHxbaPIEfItkyK/Tj7HHUkAK+GeM7boDAuMP6Zen/QM= ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-09-20T22:37:25.404 INFO:teuthology.orchestra.run.smithi154.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCbxQWY5ggD0Q31Py9zN8ggan/SG8UQgqJ9nDMd96yPErTmsLqzVcL5LlCBQpKbgte7KOoL5KNxLL/zsvG39ZrwZgpdtZWBHa0LkQj456h4EmzDFLjpKFm4NduObbnDCSii4CyQ/r92zanPkFJVywx1lMLaidSUPTUJb9yN/QDgA/k3K1mkisw9Xdfx+qQ4Q3xV3cSmu9sQYPd11FFUdcYAf4TWj0rKRp8NI+z43yq5QwUkDWxGVkBv1wgfFGOEOnShj8Wwv9LXSegAlUnKVZN0I+xNtA63hWBS9AzE9T15fjJ61u1Dj2nGYXpz0eWOWAQKc6DfsRBrhQmsZent4Bs987H3ak0UTJRINtV0IuSzLJURd8PA2TbTd7b6+KAvtLmJBwpJirOHfncPYBCikDoTk7c5572ar7LNa19ruwTwr/DA2vsndn3d0P6Fzi9luJr+MRUHVEKFRIJqQF4TrwGbBHxbaPIEfItkyK/Tj7HHUkAK+GeM7boDAuMP6Zen/QM= ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490 2024-09-20T22:37:25.419 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-09-20T22:37:25.664 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:37:27.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:27 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/4192433489' entity='client.admin' 2024-09-20T22:37:27.807 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-09-20T22:37:27.807 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-09-20T22:37:28.473 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:37:29.816 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:29 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:37:29.816 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:29 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:37:29.816 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:29 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi084", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:37:29.816 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:29 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:37:29.816 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:29 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:37:29.817 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:29 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:37:29.817 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:29 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:37:29.817 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:29 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:37:29.817 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:29 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:37:29.817 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:29 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:37:29.847 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi154 2024-09-20T22:37:29.847 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-09-20T22:37:29.847 DEBUG:teuthology.orchestra.run.smithi154:> dd of=/etc/ceph/ceph.conf 2024-09-20T22:37:29.866 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-09-20T22:37:29.866 DEBUG:teuthology.orchestra.run.smithi154:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-20T22:37:29.925 INFO:tasks.cephadm:Adding host smithi154 to orchestrator... 2024-09-20T22:37:29.925 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph orch host add smithi154 2024-09-20T22:37:30.165 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:37:30.594 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:30 smithi084 ceph-mon[29774]: from='client.14172 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:37:30.594 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:30 smithi084 ceph-mon[29774]: Updating smithi084:/etc/ceph/ceph.conf 2024-09-20T22:37:30.594 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:30 smithi084 ceph-mon[29774]: Updating smithi084:/var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T22:37:30.594 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:30 smithi084 ceph-mon[29774]: Updating smithi084:/etc/ceph/ceph.client.admin.keyring 2024-09-20T22:37:30.594 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:30 smithi084 ceph-mon[29774]: Updating smithi084:/var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-20T22:37:30.594 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:30 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:37:30.594 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:30 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:37:30.595 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:30 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:37:31.864 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:31 smithi084 ceph-mon[29774]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi154", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:37:32.864 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:32 smithi084 ceph-mon[29774]: Deploying cephadm binary to smithi154 2024-09-20T22:37:33.922 INFO:teuthology.orchestra.run.smithi084.stdout:Added host 'smithi154' with addr '172.21.15.154' 2024-09-20T22:37:34.566 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph orch host ls --format=json 2024-09-20T22:37:34.801 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:37:35.218 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:34 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:37:35.219 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:34 smithi084 ceph-mon[29774]: Added host smithi154 2024-09-20T22:37:35.219 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:34 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:37:35.219 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:34 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:37:35.219 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:34 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:37:35.403 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-20T22:37:35.403 INFO:teuthology.orchestra.run.smithi084.stdout:[{"addr": "172.21.15.84", "hostname": "smithi084", "labels": [], "status": ""}, {"addr": "172.21.15.154", "hostname": "smithi154", "labels": [], "status": ""}] 2024-09-20T22:37:36.026 INFO:tasks.cephadm:Setting crush tunables to default 2024-09-20T22:37:36.027 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd crush tunables default 2024-09-20T22:37:36.265 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:37:36.591 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:36 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:37:36.592 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:36 smithi084 ceph-mon[29774]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T22:37:37.275 INFO:teuthology.orchestra.run.smithi084.stderr:adjusted tunables profile to default 2024-09-20T22:37:37.562 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:37 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/2465465792' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-09-20T22:37:37.847 INFO:tasks.cephadm:Adding mon.a on smithi084 2024-09-20T22:37:37.848 INFO:tasks.cephadm:Adding mon.c on smithi084 2024-09-20T22:37:37.848 INFO:tasks.cephadm:Adding mon.b on smithi154 2024-09-20T22:37:37.848 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph orch apply mon '3;smithi084:172.21.15.84=a;smithi084:[v2:172.21.15.84:3301,v1:172.21.15.84:6790]=c;smithi154:172.21.15.154=b' 2024-09-20T22:37:38.125 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-20T22:37:38.173 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-20T22:37:38.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:38 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/2465465792' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-09-20T22:37:38.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:38 smithi084 ceph-mon[29774]: osdmap e4: 0 total, 0 up, 0 in 2024-09-20T22:37:38.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:38 smithi084 ceph-mon[29774]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T22:37:38.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:38 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:37:38.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:38 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:37:38.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:38 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:37:38.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:38 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:37:38.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:38 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi154", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:37:38.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:38 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:37:38.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:38 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:37:38.893 INFO:teuthology.orchestra.run.smithi154.stdout:Scheduled mon update... 2024-09-20T22:37:39.569 DEBUG:teuthology.orchestra.run.smithi084:mon.c> sudo journalctl -f -n 0 -u ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490@mon.c.service 2024-09-20T22:37:39.571 DEBUG:teuthology.orchestra.run.smithi154:mon.b> sudo journalctl -f -n 0 -u ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490@mon.b.service 2024-09-20T22:37:39.573 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-20T22:37:39.574 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph mon dump -f json 2024-09-20T22:37:40.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:39 smithi084 ceph-mon[29774]: Updating smithi154:/etc/ceph/ceph.conf 2024-09-20T22:37:40.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:39 smithi084 ceph-mon[29774]: Updating smithi154:/var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T22:37:40.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:39 smithi084 ceph-mon[29774]: Updating smithi154:/etc/ceph/ceph.client.admin.keyring 2024-09-20T22:37:40.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:39 smithi084 ceph-mon[29774]: Updating smithi154:/var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-20T22:37:40.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:39 smithi084 ceph-mon[29774]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi084:172.21.15.84=a;smithi084:[v2:172.21.15.84:3301,v1:172.21.15.84:6790]=c;smithi154:172.21.15.154=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:37:40.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:39 smithi084 ceph-mon[29774]: Saving service mon spec with placement smithi084:172.21.15.84=a;smithi084:[v2:172.21.15.84:3301,v1:172.21.15.84:6790]=c;smithi154:172.21.15.154=b;count:3 2024-09-20T22:37:40.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:39 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:37:40.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:39 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:37:40.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:39 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:37:40.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:39 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:37:40.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:39 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-20T22:37:40.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:39 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:37:40.572 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T22:37:41.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:40 smithi084 ceph-mon[29774]: Deploying daemon mon.b on smithi154 2024-09-20T22:37:41.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:40 smithi084 ceph-mon[29774]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T22:37:41.747 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-20T22:37:41.747 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":1,"fsid":"a9c0bc70-77a0-11ef-baf5-efdc52797490","modified":"2024-09-20T22:36:18.149782Z","created":"2024-09-20T22:36:18.149782Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-09-20T22:37:41.747 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 1 2024-09-20T22:37:42.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:41 smithi084 ceph-mon[29774]: from='client.? 172.21.15.154:0/164256320' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-20T22:37:43.364 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:42 smithi084 ceph-mon[29774]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T22:37:43.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:37:43 smithi154 systemd[1]: Started Ceph mon.b for a9c0bc70-77a0-11ef-baf5-efdc52797490. 2024-09-20T22:37:43.759 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-20T22:37:43.760 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph mon dump -f json 2024-09-20T22:37:44.010 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:37:49.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:37:48 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-20T22:37:49.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:37:48 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T22:37:49.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:37:48 smithi154 ceph-mon[33264]: mon.a calling monitor election 2024-09-20T22:37:49.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:37:48 smithi154 ceph-mon[33264]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T22:37:49.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:37:48 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T22:37:49.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:37:48 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T22:37:49.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:37:48 smithi154 ceph-mon[33264]: mon.b calling monitor election 2024-09-20T22:37:49.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:37:48 smithi154 ceph-mon[33264]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T22:37:49.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:37:48 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T22:37:49.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:37:48 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T22:37:49.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:37:48 smithi154 ceph-mon[33264]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T22:37:49.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:37:48 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T22:37:49.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:37:48 smithi154 ceph-mon[33264]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-09-20T22:37:49.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:37:48 smithi154 ceph-mon[33264]: monmap epoch 2 2024-09-20T22:37:49.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:37:48 smithi154 ceph-mon[33264]: fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 2024-09-20T22:37:49.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:37:48 smithi154 ceph-mon[33264]: last_changed 2024-09-20T22:37:43.563545+0000 2024-09-20T22:37:49.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:37:48 smithi154 ceph-mon[33264]: created 2024-09-20T22:36:18.149782+0000 2024-09-20T22:37:49.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:37:48 smithi154 ceph-mon[33264]: min_mon_release 19 (squid) 2024-09-20T22:37:49.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:37:48 smithi154 ceph-mon[33264]: election_strategy: 1 2024-09-20T22:37:49.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:37:48 smithi154 ceph-mon[33264]: 0: [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] mon.a 2024-09-20T22:37:49.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:37:48 smithi154 ceph-mon[33264]: 1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-20T22:37:49.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:37:48 smithi154 ceph-mon[33264]: fsmap 2024-09-20T22:37:49.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:37:48 smithi154 ceph-mon[33264]: osdmap e4: 0 total, 0 up, 0 in 2024-09-20T22:37:49.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:37:48 smithi154 ceph-mon[33264]: mgrmap e13: y(active, since 30s) 2024-09-20T22:37:49.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:37:48 smithi154 ceph-mon[33264]: overall HEALTH_OK 2024-09-20T22:37:49.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:37:48 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:37:49.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:37:48 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:37:49.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:37:48 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:37:49.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:37:48 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-20T22:37:49.007 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:37:48 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:37:49.012 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:48 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-20T22:37:49.012 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:48 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T22:37:49.012 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:48 smithi084 ceph-mon[29774]: mon.a calling monitor election 2024-09-20T22:37:49.012 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:48 smithi084 ceph-mon[29774]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T22:37:49.012 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:48 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T22:37:49.012 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:48 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T22:37:49.012 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:48 smithi084 ceph-mon[29774]: mon.b calling monitor election 2024-09-20T22:37:49.012 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:48 smithi084 ceph-mon[29774]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T22:37:49.012 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:48 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T22:37:49.012 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:48 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T22:37:49.012 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:48 smithi084 ceph-mon[29774]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T22:37:49.012 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:48 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T22:37:49.012 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:48 smithi084 ceph-mon[29774]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-09-20T22:37:49.013 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:48 smithi084 ceph-mon[29774]: monmap epoch 2 2024-09-20T22:37:49.013 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:48 smithi084 ceph-mon[29774]: fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 2024-09-20T22:37:49.013 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:48 smithi084 ceph-mon[29774]: last_changed 2024-09-20T22:37:43.563545+0000 2024-09-20T22:37:49.013 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:48 smithi084 ceph-mon[29774]: created 2024-09-20T22:36:18.149782+0000 2024-09-20T22:37:49.013 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:48 smithi084 ceph-mon[29774]: min_mon_release 19 (squid) 2024-09-20T22:37:49.013 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:48 smithi084 ceph-mon[29774]: election_strategy: 1 2024-09-20T22:37:49.013 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:48 smithi084 ceph-mon[29774]: 0: [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] mon.a 2024-09-20T22:37:49.013 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:48 smithi084 ceph-mon[29774]: 1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-20T22:37:49.013 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:48 smithi084 ceph-mon[29774]: fsmap 2024-09-20T22:37:49.013 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:48 smithi084 ceph-mon[29774]: osdmap e4: 0 total, 0 up, 0 in 2024-09-20T22:37:49.013 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:48 smithi084 ceph-mon[29774]: mgrmap e13: y(active, since 30s) 2024-09-20T22:37:49.013 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:48 smithi084 ceph-mon[29774]: overall HEALTH_OK 2024-09-20T22:37:49.013 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:48 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:37:49.013 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:48 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:37:49.013 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:48 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:37:49.014 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:48 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-20T22:37:49.014 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:48 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:37:49.068 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-20T22:37:49.068 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":2,"fsid":"a9c0bc70-77a0-11ef-baf5-efdc52797490","modified":"2024-09-20T22:37:43.563545Z","created":"2024-09-20T22:36:18.149782Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-09-20T22:37:49.068 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 2 2024-09-20T22:37:49.666 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:37:49 smithi154 ceph-mon[33264]: Deploying daemon mon.c on smithi084 2024-09-20T22:37:49.666 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:37:49 smithi154 ceph-mon[33264]: from='client.? 172.21.15.154:0/4139868267' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-20T22:37:49.666 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:37:49 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T22:37:49.934 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:49 smithi084 ceph-mon[29774]: Deploying daemon mon.c on smithi084 2024-09-20T22:37:49.934 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:49 smithi084 ceph-mon[29774]: from='client.? 172.21.15.154:0/4139868267' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-20T22:37:49.934 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:49 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T22:37:50.694 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-20T22:37:50.695 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph mon dump -f json 2024-09-20T22:37:50.915 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:50 smithi084 ceph-mon[29774]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T22:37:50.946 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:37:51.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:37:50 smithi154 ceph-mon[33264]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T22:37:51.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 podman[40005]: 2024-09-20 22:37:51.507890968 +0000 UTC m=+0.143620943 container create 3b6b6457d59519c55f4d6253c507f9feca4a312c74fe6f8d808bd2a8d0325339 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mon-c, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, RELEASE=squid-9e2ac70, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-09-20T22:37:51.638 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-20T22:37:51.638 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":2,"fsid":"a9c0bc70-77a0-11ef-baf5-efdc52797490","modified":"2024-09-20T22:37:43.563545Z","created":"2024-09-20T22:36:18.149782Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-09-20T22:37:51.638 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 2 2024-09-20T22:37:52.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:37:51 smithi154 ceph-mon[33264]: from='client.? 172.21.15.154:0/1578152305' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-20T22:37:52.114 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[29774]: from='client.? 172.21.15.154:0/1578152305' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-20T22:37:52.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 podman[40005]: 2024-09-20 22:37:51.735511341 +0000 UTC m=+0.371241335 container init 3b6b6457d59519c55f4d6253c507f9feca4a312c74fe6f8d808bd2a8d0325339 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mon-c, GIT_BRANCH=HEAD, RELEASE=squid-9e2ac70, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2) 2024-09-20T22:37:52.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 podman[40005]: 2024-09-20 22:37:51.739105064 +0000 UTC m=+0.374835040 container start 3b6b6457d59519c55f4d6253c507f9feca4a312c74fe6f8d808bd2a8d0325339 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mon-c, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-9e2ac70, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-20T22:37:52.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: set uid:gid to 167:167 (ceph:ceph) 2024-09-20T22:37:52.116 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable), process ceph-mon, pid 2 2024-09-20T22:37:52.116 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: pidfile_write: ignore empty --pid-file 2024-09-20T22:37:52.116 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: load: jerasure load: lrc 2024-09-20T22:37:52.116 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: RocksDB version: 7.9.2 2024-09-20T22:37:52.116 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Git sha 0 2024-09-20T22:37:52.116 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Compile date 2024-09-19 18:20:48 2024-09-20T22:37:52.116 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: DB SUMMARY 2024-09-20T22:37:52.116 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: DB Session ID: TJ2IYAOR79QG8S3HU0EF 2024-09-20T22:37:52.116 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: CURRENT file: CURRENT 2024-09-20T22:37:52.117 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: IDENTITY file: IDENTITY 2024-09-20T22:37:52.117 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-09-20T22:37:52.117 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-09-20T22:37:52.117 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 636 ; 2024-09-20T22:37:52.117 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.error_if_exists: 0 2024-09-20T22:37:52.117 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.create_if_missing: 0 2024-09-20T22:37:52.117 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.paranoid_checks: 1 2024-09-20T22:37:52.118 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.flush_verify_memtable_count: 1 2024-09-20T22:37:52.118 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-20T22:37:52.118 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-09-20T22:37:52.118 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.env: 0x55c8de19f940 2024-09-20T22:37:52.118 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.fs: PosixFileSystem 2024-09-20T22:37:52.118 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.info_log: 0x55c8df96b6e0 2024-09-20T22:37:52.118 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.max_file_opening_threads: 16 2024-09-20T22:37:52.120 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.statistics: (nil) 2024-09-20T22:37:52.120 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.use_fsync: 0 2024-09-20T22:37:52.120 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.max_log_file_size: 0 2024-09-20T22:37:52.120 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-20T22:37:52.120 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-20T22:37:52.120 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.keep_log_file_num: 1000 2024-09-20T22:37:52.121 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.recycle_log_file_num: 0 2024-09-20T22:37:52.121 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.allow_fallocate: 1 2024-09-20T22:37:52.121 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.allow_mmap_reads: 0 2024-09-20T22:37:52.121 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.allow_mmap_writes: 0 2024-09-20T22:37:52.121 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.use_direct_reads: 0 2024-09-20T22:37:52.121 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-20T22:37:52.121 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.create_missing_column_families: 0 2024-09-20T22:37:52.121 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.db_log_dir: 2024-09-20T22:37:52.121 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.wal_dir: 2024-09-20T22:37:52.122 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-20T22:37:52.122 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-20T22:37:52.122 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-20T22:37:52.122 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-20T22:37:52.122 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-20T22:37:52.122 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-20T22:37:52.122 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.advise_random_on_open: 1 2024-09-20T22:37:52.122 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.db_write_buffer_size: 0 2024-09-20T22:37:52.122 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.write_buffer_manager: 0x55c8df97adc0 2024-09-20T22:37:52.122 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-20T22:37:52.122 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-20T22:37:52.122 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-20T22:37:52.122 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.rate_limiter: (nil) 2024-09-20T22:37:52.123 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-20T22:37:52.123 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.wal_recovery_mode: 2 2024-09-20T22:37:52.123 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.enable_thread_tracking: 0 2024-09-20T22:37:52.123 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.enable_pipelined_write: 0 2024-09-20T22:37:52.123 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.unordered_write: 0 2024-09-20T22:37:52.123 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-20T22:37:52.123 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-20T22:37:52.123 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-20T22:37:52.123 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-20T22:37:52.123 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.row_cache: None 2024-09-20T22:37:52.123 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.wal_filter: None 2024-09-20T22:37:52.123 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-20T22:37:52.124 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.allow_ingest_behind: 0 2024-09-20T22:37:52.124 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.two_write_queues: 0 2024-09-20T22:37:52.124 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.manual_wal_flush: 0 2024-09-20T22:37:52.124 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.wal_compression: 0 2024-09-20T22:37:52.124 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.atomic_flush: 0 2024-09-20T22:37:52.124 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-20T22:37:52.124 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-20T22:37:52.124 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-20T22:37:52.124 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.log_readahead_size: 0 2024-09-20T22:37:52.124 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-20T22:37:52.124 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.best_efforts_recovery: 0 2024-09-20T22:37:52.124 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-20T22:37:52.125 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-20T22:37:52.125 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.allow_data_in_errors: 0 2024-09-20T22:37:52.125 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.db_host_id: __hostname__ 2024-09-20T22:37:52.125 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.enforce_single_del_contracts: true 2024-09-20T22:37:52.125 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.max_background_jobs: 2 2024-09-20T22:37:52.125 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.max_background_compactions: -1 2024-09-20T22:37:52.125 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.max_subcompactions: 1 2024-09-20T22:37:52.125 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-20T22:37:52.125 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-20T22:37:52.125 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-20T22:37:52.125 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.max_total_wal_size: 0 2024-09-20T22:37:52.125 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-20T22:37:52.125 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-20T22:37:52.126 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-20T22:37:52.126 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-20T22:37:52.126 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.max_open_files: -1 2024-09-20T22:37:52.126 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.bytes_per_sync: 0 2024-09-20T22:37:52.126 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-20T22:37:52.126 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-20T22:37:52.126 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.compaction_readahead_size: 0 2024-09-20T22:37:52.126 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.max_background_flushes: -1 2024-09-20T22:37:52.126 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Compression algorithms supported: 2024-09-20T22:37:52.126 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: kZSTD supported: 0 2024-09-20T22:37:52.126 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: kXpressCompression supported: 0 2024-09-20T22:37:52.126 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: kBZip2Compression supported: 0 2024-09-20T22:37:52.126 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-20T22:37:52.126 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: kLZ4Compression supported: 1 2024-09-20T22:37:52.127 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: kZlibCompression supported: 1 2024-09-20T22:37:52.127 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: kLZ4HCCompression supported: 1 2024-09-20T22:37:52.127 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: kSnappyCompression supported: 1 2024-09-20T22:37:52.127 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-20T22:37:52.127 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: DMutex implementation: pthread_mutex_t 2024-09-20T22:37:52.127 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2024-09-20T22:37:52.127 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-09-20T22:37:52.127 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-20T22:37:52.127 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.merge_operator: 2024-09-20T22:37:52.127 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.compaction_filter: None 2024-09-20T22:37:52.127 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.compaction_filter_factory: None 2024-09-20T22:37:52.127 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.sst_partitioner_factory: None 2024-09-20T22:37:52.127 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-20T22:37:52.128 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-20T22:37:52.128 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55c8df96b320) 2024-09-20T22:37:52.128 INFO:journalctl@ceph.mon.c.smithi084.stdout: cache_index_and_filter_blocks: 1 2024-09-20T22:37:52.128 INFO:journalctl@ceph.mon.c.smithi084.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-20T22:37:52.128 INFO:journalctl@ceph.mon.c.smithi084.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-20T22:37:52.128 INFO:journalctl@ceph.mon.c.smithi084.stdout: pin_top_level_index_and_filter: 1 2024-09-20T22:37:52.128 INFO:journalctl@ceph.mon.c.smithi084.stdout: index_type: 0 2024-09-20T22:37:52.128 INFO:journalctl@ceph.mon.c.smithi084.stdout: data_block_index_type: 0 2024-09-20T22:37:52.128 INFO:journalctl@ceph.mon.c.smithi084.stdout: index_shortening: 1 2024-09-20T22:37:52.128 INFO:journalctl@ceph.mon.c.smithi084.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-20T22:37:52.128 INFO:journalctl@ceph.mon.c.smithi084.stdout: checksum: 4 2024-09-20T22:37:52.128 INFO:journalctl@ceph.mon.c.smithi084.stdout: no_block_cache: 0 2024-09-20T22:37:52.128 INFO:journalctl@ceph.mon.c.smithi084.stdout: block_cache: 0x55c8df95f1f0 2024-09-20T22:37:52.128 INFO:journalctl@ceph.mon.c.smithi084.stdout: block_cache_name: BinnedLRUCache 2024-09-20T22:37:52.129 INFO:journalctl@ceph.mon.c.smithi084.stdout: block_cache_options: 2024-09-20T22:37:52.129 INFO:journalctl@ceph.mon.c.smithi084.stdout: capacity : 536870912 2024-09-20T22:37:52.129 INFO:journalctl@ceph.mon.c.smithi084.stdout: num_shard_bits : 4 2024-09-20T22:37:52.129 INFO:journalctl@ceph.mon.c.smithi084.stdout: strict_capacity_limit : 0 2024-09-20T22:37:52.129 INFO:journalctl@ceph.mon.c.smithi084.stdout: high_pri_pool_ratio: 0.000 2024-09-20T22:37:52.129 INFO:journalctl@ceph.mon.c.smithi084.stdout: block_cache_compressed: (nil) 2024-09-20T22:37:52.129 INFO:journalctl@ceph.mon.c.smithi084.stdout: persistent_cache: (nil) 2024-09-20T22:37:52.129 INFO:journalctl@ceph.mon.c.smithi084.stdout: block_size: 4096 2024-09-20T22:37:52.129 INFO:journalctl@ceph.mon.c.smithi084.stdout: block_size_deviation: 10 2024-09-20T22:37:52.129 INFO:journalctl@ceph.mon.c.smithi084.stdout: block_restart_interval: 16 2024-09-20T22:37:52.129 INFO:journalctl@ceph.mon.c.smithi084.stdout: index_block_restart_interval: 1 2024-09-20T22:37:52.129 INFO:journalctl@ceph.mon.c.smithi084.stdout: metadata_block_size: 4096 2024-09-20T22:37:52.129 INFO:journalctl@ceph.mon.c.smithi084.stdout: partition_filters: 0 2024-09-20T22:37:52.130 INFO:journalctl@ceph.mon.c.smithi084.stdout: use_delta_encoding: 1 2024-09-20T22:37:52.130 INFO:journalctl@ceph.mon.c.smithi084.stdout: filter_policy: bloomfilter 2024-09-20T22:37:52.130 INFO:journalctl@ceph.mon.c.smithi084.stdout: whole_key_filtering: 1 2024-09-20T22:37:52.130 INFO:journalctl@ceph.mon.c.smithi084.stdout: verify_compression: 0 2024-09-20T22:37:52.130 INFO:journalctl@ceph.mon.c.smithi084.stdout: read_amp_bytes_per_bit: 0 2024-09-20T22:37:52.130 INFO:journalctl@ceph.mon.c.smithi084.stdout: format_version: 5 2024-09-20T22:37:52.130 INFO:journalctl@ceph.mon.c.smithi084.stdout: enable_index_compression: 1 2024-09-20T22:37:52.130 INFO:journalctl@ceph.mon.c.smithi084.stdout: block_align: 0 2024-09-20T22:37:52.130 INFO:journalctl@ceph.mon.c.smithi084.stdout: max_auto_readahead_size: 262144 2024-09-20T22:37:52.130 INFO:journalctl@ceph.mon.c.smithi084.stdout: prepopulate_block_cache: 0 2024-09-20T22:37:52.130 INFO:journalctl@ceph.mon.c.smithi084.stdout: initial_auto_readahead_size: 8192 2024-09-20T22:37:52.130 INFO:journalctl@ceph.mon.c.smithi084.stdout: num_file_reads_for_auto_readahead: 2 2024-09-20T22:37:52.130 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.write_buffer_size: 33554432 2024-09-20T22:37:52.130 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.max_write_buffer_number: 2 2024-09-20T22:37:52.131 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.compression: NoCompression 2024-09-20T22:37:52.131 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.bottommost_compression: Disabled 2024-09-20T22:37:52.131 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.prefix_extractor: nullptr 2024-09-20T22:37:52.131 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-20T22:37:52.131 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.num_levels: 7 2024-09-20T22:37:52.131 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-20T22:37:52.131 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-20T22:37:52.131 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-20T22:37:52.131 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-20T22:37:52.131 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-20T22:37:52.132 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-20T22:37:52.132 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-20T22:37:52.132 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-20T22:37:52.132 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-20T22:37:52.132 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-20T22:37:52.132 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-09-20T22:37:52.132 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-09-20T22:37:52.132 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-20T22:37:52.132 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.compression_opts.level: 32767 2024-09-20T22:37:52.132 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.compression_opts.strategy: 0 2024-09-20T22:37:52.132 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-20T22:37:52.133 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-20T22:37:52.133 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-09-20T22:37:52.133 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-20T22:37:52.133 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.compression_opts.enabled: false 2024-09-20T22:37:52.133 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-09-20T22:37:52.133 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-20T22:37:52.133 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-20T22:37:52.133 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-20T22:37:52.133 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.target_file_size_base: 67108864 2024-09-20T22:37:52.133 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-20T22:37:52.133 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-20T22:37:52.133 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-20T22:37:52.133 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-20T22:37:52.133 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-20T22:37:52.133 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-20T22:37:52.134 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-20T22:37:52.134 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-20T22:37:52.134 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-20T22:37:52.134 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-20T22:37:52.134 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-20T22:37:52.134 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-20T22:37:52.134 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-20T22:37:52.134 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-09-20T22:37:52.134 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.arena_block_size: 1048576 2024-09-20T22:37:52.134 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-20T22:37:52.134 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-20T22:37:52.135 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.disable_auto_compactions: 0 2024-09-20T22:37:52.135 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-20T22:37:52.135 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-20T22:37:52.135 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-20T22:37:52.135 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-20T22:37:52.135 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-20T22:37:52.135 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-20T22:37:52.135 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-20T22:37:52.135 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-20T22:37:52.135 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-20T22:37:52.136 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-20T22:37:52.136 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-20T22:37:52.136 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.inplace_update_support: 0 2024-09-20T22:37:52.136 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-20T22:37:52.136 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-20T22:37:52.136 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-20T22:37:52.136 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-20T22:37:52.136 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.bloom_locality: 0 2024-09-20T22:37:52.136 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.max_successive_merges: 0 2024-09-20T22:37:52.136 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-20T22:37:52.137 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.paranoid_file_checks: 0 2024-09-20T22:37:52.137 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.force_consistency_checks: 1 2024-09-20T22:37:52.137 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.report_bg_io_stats: 0 2024-09-20T22:37:52.137 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.ttl: 2592000 2024-09-20T22:37:52.137 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-20T22:37:52.137 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-09-20T22:37:52.137 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-09-20T22:37:52.137 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.enable_blob_files: false 2024-09-20T22:37:52.137 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.min_blob_size: 0 2024-09-20T22:37:52.137 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.blob_file_size: 268435456 2024-09-20T22:37:52.137 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-20T22:37:52.137 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-20T22:37:52.138 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-20T22:37:52.138 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-09-20T22:37:52.138 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-09-20T22:37:52.138 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.blob_file_starting_level: 0 2024-09-20T22:37:52.138 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-09-20T22:37:52.138 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-09-20T22:37:52.138 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-09-20T22:37:52.138 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: e492b357-0bee-4037-9d52-e871f3b59d72 2024-09-20T22:37:52.138 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726871871765592, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-09-20T22:37:52.138 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-09-20T22:37:52.138 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726871871765987, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1773, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1726871871, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "e492b357-0bee-4037-9d52-e871f3b59d72", "db_session_id": "TJ2IYAOR79QG8S3HU0EF", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-09-20T22:37:52.138 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726871871766082, "job": 1, "event": "recovery_finished"} 2024-09-20T22:37:52.139 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-09-20T22:37:52.139 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-20T22:37:52.139 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55c8df98ce00 2024-09-20T22:37:52.139 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: DB pointer 0x55c8dfa88000 2024-09-20T22:37:52.139 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-09-20T22:37:52.139 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-09-20T22:37:52.140 INFO:journalctl@ceph.mon.c.smithi084.stdout: ** DB Stats ** 2024-09-20T22:37:52.140 INFO:journalctl@ceph.mon.c.smithi084.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-20T22:37:52.140 INFO:journalctl@ceph.mon.c.smithi084.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-20T22:37:52.140 INFO:journalctl@ceph.mon.c.smithi084.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-20T22:37:52.140 INFO:journalctl@ceph.mon.c.smithi084.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-20T22:37:52.140 INFO:journalctl@ceph.mon.c.smithi084.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-20T22:37:52.140 INFO:journalctl@ceph.mon.c.smithi084.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-20T22:37:52.140 INFO:journalctl@ceph.mon.c.smithi084.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-20T22:37:52.140 INFO:journalctl@ceph.mon.c.smithi084.stdout: 2024-09-20T22:37:52.140 INFO:journalctl@ceph.mon.c.smithi084.stdout: ** Compaction Stats [default] ** 2024-09-20T22:37:52.140 INFO:journalctl@ceph.mon.c.smithi084.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-20T22:37:52.140 INFO:journalctl@ceph.mon.c.smithi084.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-09-20T22:37:52.140 INFO:journalctl@ceph.mon.c.smithi084.stdout: L0 1/0 1.73 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.5 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-20T22:37:52.140 INFO:journalctl@ceph.mon.c.smithi084.stdout: Sum 1/0 1.73 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.5 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-20T22:37:52.141 INFO:journalctl@ceph.mon.c.smithi084.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.5 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-20T22:37:52.141 INFO:journalctl@ceph.mon.c.smithi084.stdout: 2024-09-20T22:37:52.141 INFO:journalctl@ceph.mon.c.smithi084.stdout: ** Compaction Stats [default] ** 2024-09-20T22:37:52.141 INFO:journalctl@ceph.mon.c.smithi084.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-20T22:37:52.141 INFO:journalctl@ceph.mon.c.smithi084.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-20T22:37:52.141 INFO:journalctl@ceph.mon.c.smithi084.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.5 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-20T22:37:52.141 INFO:journalctl@ceph.mon.c.smithi084.stdout: 2024-09-20T22:37:52.141 INFO:journalctl@ceph.mon.c.smithi084.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-09-20T22:37:52.141 INFO:journalctl@ceph.mon.c.smithi084.stdout: 2024-09-20T22:37:52.141 INFO:journalctl@ceph.mon.c.smithi084.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-20T22:37:52.141 INFO:journalctl@ceph.mon.c.smithi084.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-09-20T22:37:52.141 INFO:journalctl@ceph.mon.c.smithi084.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-20T22:37:52.142 INFO:journalctl@ceph.mon.c.smithi084.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-20T22:37:52.142 INFO:journalctl@ceph.mon.c.smithi084.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-20T22:37:52.142 INFO:journalctl@ceph.mon.c.smithi084.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-20T22:37:52.142 INFO:journalctl@ceph.mon.c.smithi084.stdout: Cumulative compaction: 0.00 GB write, 0.66 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-20T22:37:52.142 INFO:journalctl@ceph.mon.c.smithi084.stdout: Interval compaction: 0.00 GB write, 0.66 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-20T22:37:52.142 INFO:journalctl@ceph.mon.c.smithi084.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-20T22:37:52.142 INFO:journalctl@ceph.mon.c.smithi084.stdout: Block cache BinnedLRUCache@0x55c8df95f1f0#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1e-05 secs_since: 0 2024-09-20T22:37:52.142 INFO:journalctl@ceph.mon.c.smithi084.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-09-20T22:37:52.142 INFO:journalctl@ceph.mon.c.smithi084.stdout: 2024-09-20T22:37:52.142 INFO:journalctl@ceph.mon.c.smithi084.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-20T22:37:52.143 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-09-20T22:37:52.143 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: using public_addrv [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] 2024-09-20T22:37:52.143 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: starting mon.c rank -1 at public addrs [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] at bind addrs [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] mon_data /var/lib/ceph/mon/ceph-c fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 2024-09-20T22:37:52.143 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: mon.c@-1(???) e0 preinit fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 2024-09-20T22:37:52.143 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: mon.c@-1(synchronizing).mds e1 new map 2024-09-20T22:37:52.143 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: mon.c@-1(synchronizing).mds e1 print_map 2024-09-20T22:37:52.143 INFO:journalctl@ceph.mon.c.smithi084.stdout: e1 2024-09-20T22:37:52.143 INFO:journalctl@ceph.mon.c.smithi084.stdout: btime 2024-09-20T22:36:30:240441+0000 2024-09-20T22:37:52.143 INFO:journalctl@ceph.mon.c.smithi084.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-20T22:37:52.143 INFO:journalctl@ceph.mon.c.smithi084.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-09-20T22:37:52.143 INFO:journalctl@ceph.mon.c.smithi084.stdout: legacy client fscid: -1 2024-09-20T22:37:52.143 INFO:journalctl@ceph.mon.c.smithi084.stdout: 2024-09-20T22:37:52.143 INFO:journalctl@ceph.mon.c.smithi084.stdout: No filesystems configured 2024-09-20T22:37:52.143 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-09-20T22:37:52.144 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-09-20T22:37:52.144 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-09-20T22:37:52.144 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-09-20T22:37:52.144 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-09-20T22:37:52.144 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-09-20T22:37:52.144 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-09-20T22:37:52.144 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-20T22:37:52.144 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-20T22:37:52.144 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-20T22:37:52.144 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: Deploying daemon mon.b on smithi154 2024-09-20T22:37:52.144 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T22:37:52.144 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: from='client.? 172.21.15.154:0/164256320' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-20T22:37:52.144 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T22:37:52.144 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-20T22:37:52.144 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T22:37:52.145 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: mon.a calling monitor election 2024-09-20T22:37:52.145 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T22:37:52.145 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T22:37:52.145 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T22:37:52.145 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: mon.b calling monitor election 2024-09-20T22:37:52.145 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T22:37:52.145 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T22:37:52.145 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T22:37:52.145 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T22:37:52.145 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T22:37:52.145 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-09-20T22:37:52.145 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: monmap epoch 2 2024-09-20T22:37:52.145 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 2024-09-20T22:37:52.145 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: last_changed 2024-09-20T22:37:43.563545+0000 2024-09-20T22:37:52.146 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: created 2024-09-20T22:36:18.149782+0000 2024-09-20T22:37:52.146 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: min_mon_release 19 (squid) 2024-09-20T22:37:52.146 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: election_strategy: 1 2024-09-20T22:37:52.146 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: 0: [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] mon.a 2024-09-20T22:37:52.146 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: 1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-20T22:37:52.146 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: fsmap 2024-09-20T22:37:52.146 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: osdmap e4: 0 total, 0 up, 0 in 2024-09-20T22:37:52.146 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: mgrmap e13: y(active, since 30s) 2024-09-20T22:37:52.146 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: overall HEALTH_OK 2024-09-20T22:37:52.146 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:37:52.146 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:37:52.146 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:37:52.146 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-20T22:37:52.146 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:37:52.147 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: Deploying daemon mon.c on smithi084 2024-09-20T22:37:52.147 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: from='client.? 172.21.15.154:0/4139868267' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-20T22:37:52.147 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T22:37:52.147 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T22:37:52.147 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: from='client.? 172.21.15.154:0/1578152305' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-20T22:37:52.147 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 ceph-mon[40024]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-09-20T22:37:52.147 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 bash[40005]: 3b6b6457d59519c55f4d6253c507f9feca4a312c74fe6f8d808bd2a8d0325339 2024-09-20T22:37:52.147 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:51 smithi084 systemd[1]: Started Ceph mon.c for a9c0bc70-77a0-11ef-baf5-efdc52797490. 2024-09-20T22:37:53.149 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-20T22:37:53.149 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph mon dump -f json 2024-09-20T22:37:53.391 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:37:57.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:56 smithi084 ceph-mon[29774]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T22:37:57.162 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:56 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-20T22:37:57.162 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:56 smithi084 ceph-mon[29774]: mon.a calling monitor election 2024-09-20T22:37:57.162 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:56 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T22:37:57.162 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:56 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T22:37:57.163 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:56 smithi084 ceph-mon[29774]: mon.b calling monitor election 2024-09-20T22:37:57.163 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:56 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T22:37:57.163 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:56 smithi084 ceph-mon[29774]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T22:37:57.163 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:56 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T22:37:57.163 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:56 smithi084 ceph-mon[29774]: mon.c calling monitor election 2024-09-20T22:37:57.163 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:56 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T22:37:57.163 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:56 smithi084 ceph-mon[29774]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T22:37:57.163 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:56 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T22:37:57.163 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:56 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T22:37:57.163 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:56 smithi084 ceph-mon[29774]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-20T22:37:57.163 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:56 smithi084 ceph-mon[29774]: monmap epoch 3 2024-09-20T22:37:57.163 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:56 smithi084 ceph-mon[29774]: fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 2024-09-20T22:37:57.163 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:56 smithi084 ceph-mon[29774]: last_changed 2024-09-20T22:37:51.789289+0000 2024-09-20T22:37:57.163 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:56 smithi084 ceph-mon[29774]: created 2024-09-20T22:36:18.149782+0000 2024-09-20T22:37:57.164 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:56 smithi084 ceph-mon[29774]: min_mon_release 19 (squid) 2024-09-20T22:37:57.164 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:56 smithi084 ceph-mon[29774]: election_strategy: 1 2024-09-20T22:37:57.164 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:56 smithi084 ceph-mon[29774]: 0: [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] mon.a 2024-09-20T22:37:57.164 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:56 smithi084 ceph-mon[29774]: 1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-20T22:37:57.164 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:56 smithi084 ceph-mon[29774]: 2: [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] mon.c 2024-09-20T22:37:57.164 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:56 smithi084 ceph-mon[29774]: fsmap 2024-09-20T22:37:57.164 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:56 smithi084 ceph-mon[29774]: osdmap e4: 0 total, 0 up, 0 in 2024-09-20T22:37:57.164 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:56 smithi084 ceph-mon[29774]: mgrmap e13: y(active, since 39s) 2024-09-20T22:37:57.164 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:56 smithi084 ceph-mon[29774]: overall HEALTH_OK 2024-09-20T22:37:57.164 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:56 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:37:57.164 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:56 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:37:57.164 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:56 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:37:57.164 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:56 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:37:57.164 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:56 smithi084 ceph-mon[40024]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T22:37:57.165 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:56 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-20T22:37:57.165 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:56 smithi084 ceph-mon[40024]: mon.a calling monitor election 2024-09-20T22:37:57.165 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:56 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T22:37:57.165 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:56 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T22:37:57.165 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:56 smithi084 ceph-mon[40024]: mon.b calling monitor election 2024-09-20T22:37:57.165 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:56 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T22:37:57.165 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:56 smithi084 ceph-mon[40024]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T22:37:57.165 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:56 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T22:37:57.165 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:56 smithi084 ceph-mon[40024]: mon.c calling monitor election 2024-09-20T22:37:57.165 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:56 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T22:37:57.165 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:56 smithi084 ceph-mon[40024]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T22:37:57.165 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:56 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T22:37:57.165 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:56 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T22:37:57.165 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:56 smithi084 ceph-mon[40024]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-20T22:37:57.166 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:56 smithi084 ceph-mon[40024]: monmap epoch 3 2024-09-20T22:37:57.166 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:56 smithi084 ceph-mon[40024]: fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 2024-09-20T22:37:57.166 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:56 smithi084 ceph-mon[40024]: last_changed 2024-09-20T22:37:51.789289+0000 2024-09-20T22:37:57.166 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:56 smithi084 ceph-mon[40024]: created 2024-09-20T22:36:18.149782+0000 2024-09-20T22:37:57.166 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:56 smithi084 ceph-mon[40024]: min_mon_release 19 (squid) 2024-09-20T22:37:57.166 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:56 smithi084 ceph-mon[40024]: election_strategy: 1 2024-09-20T22:37:57.166 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:56 smithi084 ceph-mon[40024]: 0: [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] mon.a 2024-09-20T22:37:57.166 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:56 smithi084 ceph-mon[40024]: 1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-20T22:37:57.166 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:56 smithi084 ceph-mon[40024]: 2: [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] mon.c 2024-09-20T22:37:57.166 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:56 smithi084 ceph-mon[40024]: fsmap 2024-09-20T22:37:57.166 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:56 smithi084 ceph-mon[40024]: osdmap e4: 0 total, 0 up, 0 in 2024-09-20T22:37:57.166 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:56 smithi084 ceph-mon[40024]: mgrmap e13: y(active, since 39s) 2024-09-20T22:37:57.166 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:56 smithi084 ceph-mon[40024]: overall HEALTH_OK 2024-09-20T22:37:57.166 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:56 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:37:57.166 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:56 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:37:57.166 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:56 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:37:57.167 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:56 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:37:57.249 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:37:56 smithi154 ceph-mon[33264]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T22:37:57.249 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:37:56 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-20T22:37:57.249 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:37:56 smithi154 ceph-mon[33264]: mon.a calling monitor election 2024-09-20T22:37:57.249 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:37:56 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T22:37:57.250 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:37:56 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T22:37:57.250 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:37:56 smithi154 ceph-mon[33264]: mon.b calling monitor election 2024-09-20T22:37:57.250 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:37:56 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T22:37:57.250 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:37:56 smithi154 ceph-mon[33264]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T22:37:57.250 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:37:56 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T22:37:57.250 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:37:56 smithi154 ceph-mon[33264]: mon.c calling monitor election 2024-09-20T22:37:57.250 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:37:56 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T22:37:57.250 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:37:56 smithi154 ceph-mon[33264]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T22:37:57.250 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:37:56 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T22:37:57.250 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:37:56 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T22:37:57.250 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:37:56 smithi154 ceph-mon[33264]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-20T22:37:57.250 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:37:56 smithi154 ceph-mon[33264]: monmap epoch 3 2024-09-20T22:37:57.250 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:37:56 smithi154 ceph-mon[33264]: fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 2024-09-20T22:37:57.250 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:37:56 smithi154 ceph-mon[33264]: last_changed 2024-09-20T22:37:51.789289+0000 2024-09-20T22:37:57.250 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:37:56 smithi154 ceph-mon[33264]: created 2024-09-20T22:36:18.149782+0000 2024-09-20T22:37:57.250 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:37:56 smithi154 ceph-mon[33264]: min_mon_release 19 (squid) 2024-09-20T22:37:57.251 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:37:56 smithi154 ceph-mon[33264]: election_strategy: 1 2024-09-20T22:37:57.251 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:37:56 smithi154 ceph-mon[33264]: 0: [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] mon.a 2024-09-20T22:37:57.251 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:37:56 smithi154 ceph-mon[33264]: 1: [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] mon.b 2024-09-20T22:37:57.251 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:37:56 smithi154 ceph-mon[33264]: 2: [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] mon.c 2024-09-20T22:37:57.251 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:37:56 smithi154 ceph-mon[33264]: fsmap 2024-09-20T22:37:57.251 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:37:56 smithi154 ceph-mon[33264]: osdmap e4: 0 total, 0 up, 0 in 2024-09-20T22:37:57.251 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:37:56 smithi154 ceph-mon[33264]: mgrmap e13: y(active, since 39s) 2024-09-20T22:37:57.251 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:37:56 smithi154 ceph-mon[33264]: overall HEALTH_OK 2024-09-20T22:37:57.251 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:37:56 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:37:57.251 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:37:56 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:37:57.251 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:37:56 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:37:57.251 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:37:56 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:37:57.255 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-20T22:37:57.255 INFO:teuthology.orchestra.run.smithi154.stdout:{"epoch":3,"fsid":"a9c0bc70-77a0-11ef-baf5-efdc52797490","modified":"2024-09-20T22:37:51.789289Z","created":"2024-09-20T22:36:18.149782Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3300","nonce":0},{"type":"v1","addr":"172.21.15.84:6789","nonce":0}]},"addr":"172.21.15.84:6789/0","public_addr":"172.21.15.84:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:3300","nonce":0},{"type":"v1","addr":"172.21.15.154:6789","nonce":0}]},"addr":"172.21.15.154:6789/0","public_addr":"172.21.15.154:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:3301","nonce":0},{"type":"v1","addr":"172.21.15.84:6790","nonce":0}]},"addr":"172.21.15.84:6790/0","public_addr":"172.21.15.84:6790/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1,2]} 2024-09-20T22:37:57.255 INFO:teuthology.orchestra.run.smithi154.stderr:dumped monmap epoch 3 2024-09-20T22:37:57.908 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-09-20T22:37:57.909 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph config generate-minimal-conf 2024-09-20T22:37:58.142 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:37:57 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:37:58.142 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:37:57 smithi154 ceph-mon[33264]: from='client.? 172.21.15.154:0/3632785901' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-20T22:37:58.142 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:37:57 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:37:58.142 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:37:57 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T22:37:58.150 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:37:58.271 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:57 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:37:58.271 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:57 smithi084 ceph-mon[29774]: from='client.? 172.21.15.154:0/3632785901' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-20T22:37:58.271 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:57 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:37:58.271 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:57 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T22:37:58.271 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:57 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:37:58.271 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:57 smithi084 ceph-mon[40024]: from='client.? 172.21.15.154:0/3632785901' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-20T22:37:58.271 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:57 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:37:58.271 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:57 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T22:37:58.777 INFO:teuthology.orchestra.run.smithi084.stdout:# minimal ceph.conf for a9c0bc70-77a0-11ef-baf5-efdc52797490 2024-09-20T22:37:58.777 INFO:teuthology.orchestra.run.smithi084.stdout:[global] 2024-09-20T22:37:58.777 INFO:teuthology.orchestra.run.smithi084.stdout: fsid = a9c0bc70-77a0-11ef-baf5-efdc52797490 2024-09-20T22:37:58.778 INFO:teuthology.orchestra.run.smithi084.stdout: mon_host = [v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0] [v2:172.21.15.154:3300/0,v1:172.21.15.154:6789/0] [v2:172.21.15.84:3301/0,v1:172.21.15.84:6790/0] 2024-09-20T22:37:59.129 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:58 smithi084 ceph-mon[29774]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T22:37:59.129 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:58 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:37:59.129 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:58 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:37:59.129 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:58 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:37:59.129 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:58 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:37:59.129 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:58 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:37:59.129 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:58 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:37:59.129 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:58 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/3149917142' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:37:59.130 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:58 smithi084 ceph-mon[40024]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T22:37:59.130 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:58 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:37:59.130 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:58 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:37:59.130 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:58 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:37:59.130 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:58 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:37:59.130 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:58 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:37:59.130 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:58 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:37:59.130 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:58 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/3149917142' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:37:59.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:37:58 smithi154 ceph-mon[33264]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T22:37:59.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:37:58 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:37:59.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:37:58 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:37:59.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:37:58 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:37:59.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:37:58 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:37:59.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:37:58 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:37:59.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:37:58 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:37:59.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:37:58 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/3149917142' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:37:59.474 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-09-20T22:37:59.475 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-09-20T22:37:59.475 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd of=/etc/ceph/ceph.conf 2024-09-20T22:37:59.501 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-09-20T22:37:59.501 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-20T22:37:59.565 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-09-20T22:37:59.565 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd of=/etc/ceph/ceph.conf 2024-09-20T22:37:59.593 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-09-20T22:37:59.593 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-20T22:37:59.658 INFO:tasks.cephadm:Adding mgr.y on smithi084 2024-09-20T22:37:59.658 INFO:tasks.cephadm:Adding mgr.x on smithi154 2024-09-20T22:37:59.658 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph orch apply mgr '2;smithi084=y;smithi154=x' 2024-09-20T22:37:59.935 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:38:00.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:59 smithi084 ceph-mon[40024]: Updating smithi084:/etc/ceph/ceph.conf 2024-09-20T22:38:00.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:59 smithi084 ceph-mon[40024]: Updating smithi154:/etc/ceph/ceph.conf 2024-09-20T22:38:00.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:59 smithi084 ceph-mon[40024]: Updating smithi154:/var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T22:38:00.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:59 smithi084 ceph-mon[40024]: Updating smithi084:/var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T22:38:00.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:59 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:38:00.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:59 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:38:00.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:59 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:38:00.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:59 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:38:00.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:59 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:38:00.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:59 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:38:00.116 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:59 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:38:00.116 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:59 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:38:00.116 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:59 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:38:00.116 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:59 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-20T22:38:00.116 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:59 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-20T22:38:00.116 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:37:59 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:38:00.116 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:59 smithi084 ceph-mon[29774]: Updating smithi084:/etc/ceph/ceph.conf 2024-09-20T22:38:00.116 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:59 smithi084 ceph-mon[29774]: Updating smithi154:/etc/ceph/ceph.conf 2024-09-20T22:38:00.116 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:59 smithi084 ceph-mon[29774]: Updating smithi154:/var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T22:38:00.116 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:59 smithi084 ceph-mon[29774]: Updating smithi084:/var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T22:38:00.116 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:59 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:38:00.117 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:59 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:38:00.117 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:59 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:38:00.117 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:59 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:38:00.117 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:59 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:38:00.117 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:59 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:38:00.117 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:59 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:38:00.117 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:59 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:38:00.117 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:59 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:38:00.117 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:59 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-20T22:38:00.117 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:59 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-20T22:38:00.117 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:37:59 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:38:00.161 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:37:59 smithi154 ceph-mon[33264]: Updating smithi084:/etc/ceph/ceph.conf 2024-09-20T22:38:00.161 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:37:59 smithi154 ceph-mon[33264]: Updating smithi154:/etc/ceph/ceph.conf 2024-09-20T22:38:00.161 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:37:59 smithi154 ceph-mon[33264]: Updating smithi154:/var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T22:38:00.161 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:37:59 smithi154 ceph-mon[33264]: Updating smithi084:/var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T22:38:00.161 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:37:59 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:38:00.162 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:37:59 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:38:00.162 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:37:59 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:38:00.162 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:37:59 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:38:00.162 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:37:59 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:38:00.162 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:37:59 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:38:00.162 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:37:59 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:38:00.162 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:37:59 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:38:00.162 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:37:59 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:38:00.162 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:37:59 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-20T22:38:00.162 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:37:59 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-20T22:38:00.162 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:37:59 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:38:00.620 INFO:teuthology.orchestra.run.smithi154.stdout:Scheduled mgr update... 2024-09-20T22:38:01.133 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:00 smithi084 ceph-mon[40024]: Reconfiguring mon.a (unknown last config time)... 2024-09-20T22:38:01.133 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:00 smithi084 ceph-mon[40024]: Reconfiguring daemon mon.a on smithi084 2024-09-20T22:38:01.133 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:00 smithi084 ceph-mon[40024]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T22:38:01.133 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:00 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:38:01.133 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:00 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:38:01.133 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:00 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-20T22:38:01.133 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:00 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-20T22:38:01.133 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:00 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:38:01.133 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:00 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:38:01.133 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:00 smithi084 ceph-mon[29774]: Reconfiguring mon.a (unknown last config time)... 2024-09-20T22:38:01.133 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:00 smithi084 ceph-mon[29774]: Reconfiguring daemon mon.a on smithi084 2024-09-20T22:38:01.134 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:00 smithi084 ceph-mon[29774]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T22:38:01.134 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:00 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:38:01.134 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:00 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:38:01.134 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:00 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-20T22:38:01.134 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:00 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-20T22:38:01.134 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:00 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:38:01.134 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:00 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:38:01.208 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:00 smithi154 ceph-mon[33264]: Reconfiguring mon.a (unknown last config time)... 2024-09-20T22:38:01.208 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:00 smithi154 ceph-mon[33264]: Reconfiguring daemon mon.a on smithi084 2024-09-20T22:38:01.208 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:00 smithi154 ceph-mon[33264]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T22:38:01.208 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:00 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:38:01.209 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:00 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:38:01.209 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:00 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-20T22:38:01.209 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:00 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-20T22:38:01.209 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:00 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:38:01.209 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:00 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:38:01.236 DEBUG:teuthology.orchestra.run.smithi154:mgr.x> sudo journalctl -f -n 0 -u ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490@mgr.x.service 2024-09-20T22:38:01.238 INFO:tasks.cephadm:Deploying OSDs... 2024-09-20T22:38:01.238 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-09-20T22:38:01.238 DEBUG:teuthology.orchestra.run.smithi084:> dd if=/scratch_devs of=/dev/stdout 2024-09-20T22:38:01.254 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-20T22:38:01.254 DEBUG:teuthology.orchestra.run.smithi084:> stat /dev/vg_nvme/lv_1 2024-09-20T22:38:01.310 INFO:teuthology.orchestra.run.smithi084.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-20T22:38:01.310 INFO:teuthology.orchestra.run.smithi084.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-20T22:38:01.310 INFO:teuthology.orchestra.run.smithi084.stdout:Device: 5h/5d Inode: 967 Links: 1 2024-09-20T22:38:01.310 INFO:teuthology.orchestra.run.smithi084.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-20T22:38:01.311 INFO:teuthology.orchestra.run.smithi084.stdout:Context: system_u:object_r:device_t:s0 2024-09-20T22:38:01.311 INFO:teuthology.orchestra.run.smithi084.stdout:Access: 2024-09-20 22:37:51.207150710 +0000 2024-09-20T22:38:01.311 INFO:teuthology.orchestra.run.smithi084.stdout:Modify: 2024-09-20 22:33:21.894730460 +0000 2024-09-20T22:38:01.311 INFO:teuthology.orchestra.run.smithi084.stdout:Change: 2024-09-20 22:33:21.894730460 +0000 2024-09-20T22:38:01.311 INFO:teuthology.orchestra.run.smithi084.stdout: Birth: 2024-09-20 22:33:21.894730460 +0000 2024-09-20T22:38:01.311 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-20T22:38:01.375 INFO:teuthology.orchestra.run.smithi084.stderr:1+0 records in 2024-09-20T22:38:01.375 INFO:teuthology.orchestra.run.smithi084.stderr:1+0 records out 2024-09-20T22:38:01.375 INFO:teuthology.orchestra.run.smithi084.stderr:512 bytes copied, 0.000208396 s, 2.5 MB/s 2024-09-20T22:38:01.376 DEBUG:teuthology.orchestra.run.smithi084:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-20T22:38:01.433 DEBUG:teuthology.orchestra.run.smithi084:> stat /dev/vg_nvme/lv_2 2024-09-20T22:38:01.489 INFO:teuthology.orchestra.run.smithi084.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-20T22:38:01.489 INFO:teuthology.orchestra.run.smithi084.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-20T22:38:01.490 INFO:teuthology.orchestra.run.smithi084.stdout:Device: 5h/5d Inode: 978 Links: 1 2024-09-20T22:38:01.490 INFO:teuthology.orchestra.run.smithi084.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-20T22:38:01.490 INFO:teuthology.orchestra.run.smithi084.stdout:Context: system_u:object_r:device_t:s0 2024-09-20T22:38:01.490 INFO:teuthology.orchestra.run.smithi084.stdout:Access: 2024-09-20 22:37:51.207150710 +0000 2024-09-20T22:38:01.490 INFO:teuthology.orchestra.run.smithi084.stdout:Modify: 2024-09-20 22:33:21.896730425 +0000 2024-09-20T22:38:01.490 INFO:teuthology.orchestra.run.smithi084.stdout:Change: 2024-09-20 22:33:21.896730425 +0000 2024-09-20T22:38:01.490 INFO:teuthology.orchestra.run.smithi084.stdout: Birth: 2024-09-20 22:33:21.896730425 +0000 2024-09-20T22:38:01.490 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-20T22:38:01.553 INFO:teuthology.orchestra.run.smithi084.stderr:1+0 records in 2024-09-20T22:38:01.553 INFO:teuthology.orchestra.run.smithi084.stderr:1+0 records out 2024-09-20T22:38:01.554 INFO:teuthology.orchestra.run.smithi084.stderr:512 bytes copied, 0.000225201 s, 2.3 MB/s 2024-09-20T22:38:01.554 DEBUG:teuthology.orchestra.run.smithi084:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-20T22:38:01.611 DEBUG:teuthology.orchestra.run.smithi084:> stat /dev/vg_nvme/lv_3 2024-09-20T22:38:01.667 INFO:teuthology.orchestra.run.smithi084.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-20T22:38:01.667 INFO:teuthology.orchestra.run.smithi084.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-20T22:38:01.667 INFO:teuthology.orchestra.run.smithi084.stdout:Device: 5h/5d Inode: 966 Links: 1 2024-09-20T22:38:01.667 INFO:teuthology.orchestra.run.smithi084.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-20T22:38:01.667 INFO:teuthology.orchestra.run.smithi084.stdout:Context: system_u:object_r:device_t:s0 2024-09-20T22:38:01.667 INFO:teuthology.orchestra.run.smithi084.stdout:Access: 2024-09-20 22:37:51.207150710 +0000 2024-09-20T22:38:01.667 INFO:teuthology.orchestra.run.smithi084.stdout:Modify: 2024-09-20 22:33:21.894730460 +0000 2024-09-20T22:38:01.667 INFO:teuthology.orchestra.run.smithi084.stdout:Change: 2024-09-20 22:33:21.894730460 +0000 2024-09-20T22:38:01.667 INFO:teuthology.orchestra.run.smithi084.stdout: Birth: 2024-09-20 22:33:21.894730460 +0000 2024-09-20T22:38:01.668 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-20T22:38:01.731 INFO:teuthology.orchestra.run.smithi084.stderr:1+0 records in 2024-09-20T22:38:01.731 INFO:teuthology.orchestra.run.smithi084.stderr:1+0 records out 2024-09-20T22:38:01.731 INFO:teuthology.orchestra.run.smithi084.stderr:512 bytes copied, 0.000196941 s, 2.6 MB/s 2024-09-20T22:38:01.732 DEBUG:teuthology.orchestra.run.smithi084:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-20T22:38:01.789 DEBUG:teuthology.orchestra.run.smithi084:> stat /dev/vg_nvme/lv_4 2024-09-20T22:38:01.844 INFO:teuthology.orchestra.run.smithi084.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-20T22:38:01.844 INFO:teuthology.orchestra.run.smithi084.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-20T22:38:01.844 INFO:teuthology.orchestra.run.smithi084.stdout:Device: 5h/5d Inode: 961 Links: 1 2024-09-20T22:38:01.844 INFO:teuthology.orchestra.run.smithi084.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-20T22:38:01.844 INFO:teuthology.orchestra.run.smithi084.stdout:Context: system_u:object_r:device_t:s0 2024-09-20T22:38:01.845 INFO:teuthology.orchestra.run.smithi084.stdout:Access: 2024-09-20 22:37:51.208150693 +0000 2024-09-20T22:38:01.845 INFO:teuthology.orchestra.run.smithi084.stdout:Modify: 2024-09-20 22:33:21.894730460 +0000 2024-09-20T22:38:01.845 INFO:teuthology.orchestra.run.smithi084.stdout:Change: 2024-09-20 22:33:21.894730460 +0000 2024-09-20T22:38:01.845 INFO:teuthology.orchestra.run.smithi084.stdout: Birth: 2024-09-20 22:33:21.894730460 +0000 2024-09-20T22:38:01.845 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-20T22:38:01.909 INFO:teuthology.orchestra.run.smithi084.stderr:1+0 records in 2024-09-20T22:38:01.910 INFO:teuthology.orchestra.run.smithi084.stderr:1+0 records out 2024-09-20T22:38:01.910 INFO:teuthology.orchestra.run.smithi084.stderr:512 bytes copied, 0.000180324 s, 2.8 MB/s 2024-09-20T22:38:01.911 DEBUG:teuthology.orchestra.run.smithi084:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-20T22:38:01.967 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-09-20T22:38:01.967 DEBUG:teuthology.orchestra.run.smithi154:> dd if=/scratch_devs of=/dev/stdout 2024-09-20T22:38:01.982 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-20T22:38:01.982 DEBUG:teuthology.orchestra.run.smithi154:> stat /dev/vg_nvme/lv_1 2024-09-20T22:38:02.038 INFO:teuthology.orchestra.run.smithi154.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-20T22:38:02.038 INFO:teuthology.orchestra.run.smithi154.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-20T22:38:02.038 INFO:teuthology.orchestra.run.smithi154.stdout:Device: 5h/5d Inode: 966 Links: 1 2024-09-20T22:38:02.038 INFO:teuthology.orchestra.run.smithi154.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-20T22:38:02.039 INFO:teuthology.orchestra.run.smithi154.stdout:Context: system_u:object_r:device_t:s0 2024-09-20T22:38:02.039 INFO:teuthology.orchestra.run.smithi154.stdout:Access: 2024-09-20 22:37:42.985875184 +0000 2024-09-20T22:38:02.039 INFO:teuthology.orchestra.run.smithi154.stdout:Modify: 2024-09-20 22:33:23.712374679 +0000 2024-09-20T22:38:02.039 INFO:teuthology.orchestra.run.smithi154.stdout:Change: 2024-09-20 22:33:23.712374679 +0000 2024-09-20T22:38:02.039 INFO:teuthology.orchestra.run.smithi154.stdout: Birth: 2024-09-20 22:33:23.712374679 +0000 2024-09-20T22:38:02.039 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-20T22:38:02.103 INFO:teuthology.orchestra.run.smithi154.stderr:1+0 records in 2024-09-20T22:38:02.103 INFO:teuthology.orchestra.run.smithi154.stderr:1+0 records out 2024-09-20T22:38:02.103 INFO:teuthology.orchestra.run.smithi154.stderr:512 bytes copied, 0.000214765 s, 2.4 MB/s 2024-09-20T22:38:02.104 DEBUG:teuthology.orchestra.run.smithi154:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-20T22:38:02.162 DEBUG:teuthology.orchestra.run.smithi154:> stat /dev/vg_nvme/lv_2 2024-09-20T22:38:02.217 INFO:teuthology.orchestra.run.smithi154.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-20T22:38:02.217 INFO:teuthology.orchestra.run.smithi154.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-20T22:38:02.218 INFO:teuthology.orchestra.run.smithi154.stdout:Device: 5h/5d Inode: 964 Links: 1 2024-09-20T22:38:02.218 INFO:teuthology.orchestra.run.smithi154.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-20T22:38:02.218 INFO:teuthology.orchestra.run.smithi154.stdout:Context: system_u:object_r:device_t:s0 2024-09-20T22:38:02.218 INFO:teuthology.orchestra.run.smithi154.stdout:Access: 2024-09-20 22:37:42.985875184 +0000 2024-09-20T22:38:02.218 INFO:teuthology.orchestra.run.smithi154.stdout:Modify: 2024-09-20 22:33:23.712374679 +0000 2024-09-20T22:38:02.218 INFO:teuthology.orchestra.run.smithi154.stdout:Change: 2024-09-20 22:33:23.712374679 +0000 2024-09-20T22:38:02.218 INFO:teuthology.orchestra.run.smithi154.stdout: Birth: 2024-09-20 22:33:23.712374679 +0000 2024-09-20T22:38:02.219 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-20T22:38:02.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:01 smithi154 ceph-mon[33264]: Reconfiguring mon.c (monmap changed)... 2024-09-20T22:38:02.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:01 smithi154 ceph-mon[33264]: Reconfiguring daemon mon.c on smithi084 2024-09-20T22:38:02.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:01 smithi154 ceph-mon[33264]: from='client.24100 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi084=y;smithi154=x", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:38:02.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:01 smithi154 ceph-mon[33264]: Saving service mgr spec with placement smithi084=y;smithi154=x;count:2 2024-09-20T22:38:02.278 INFO:teuthology.orchestra.run.smithi154.stderr:1+0 records in 2024-09-20T22:38:02.278 INFO:teuthology.orchestra.run.smithi154.stderr:1+0 records out 2024-09-20T22:38:02.278 INFO:teuthology.orchestra.run.smithi154.stderr:512 bytes copied, 0.000191883 s, 2.7 MB/s 2024-09-20T22:38:02.280 DEBUG:teuthology.orchestra.run.smithi154:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-20T22:38:02.335 DEBUG:teuthology.orchestra.run.smithi154:> stat /dev/vg_nvme/lv_3 2024-09-20T22:38:02.364 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:01 smithi084 ceph-mon[29774]: Reconfiguring mon.c (monmap changed)... 2024-09-20T22:38:02.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:01 smithi084 ceph-mon[29774]: Reconfiguring daemon mon.c on smithi084 2024-09-20T22:38:02.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:01 smithi084 ceph-mon[29774]: from='client.24100 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi084=y;smithi154=x", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:38:02.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:01 smithi084 ceph-mon[29774]: Saving service mgr spec with placement smithi084=y;smithi154=x;count:2 2024-09-20T22:38:02.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:01 smithi084 ceph-mon[40024]: Reconfiguring mon.c (monmap changed)... 2024-09-20T22:38:02.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:01 smithi084 ceph-mon[40024]: Reconfiguring daemon mon.c on smithi084 2024-09-20T22:38:02.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:01 smithi084 ceph-mon[40024]: from='client.24100 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi084=y;smithi154=x", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:38:02.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:01 smithi084 ceph-mon[40024]: Saving service mgr spec with placement smithi084=y;smithi154=x;count:2 2024-09-20T22:38:02.391 INFO:teuthology.orchestra.run.smithi154.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-20T22:38:02.391 INFO:teuthology.orchestra.run.smithi154.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-20T22:38:02.391 INFO:teuthology.orchestra.run.smithi154.stdout:Device: 5h/5d Inode: 973 Links: 1 2024-09-20T22:38:02.391 INFO:teuthology.orchestra.run.smithi154.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-20T22:38:02.391 INFO:teuthology.orchestra.run.smithi154.stdout:Context: system_u:object_r:device_t:s0 2024-09-20T22:38:02.392 INFO:teuthology.orchestra.run.smithi154.stdout:Access: 2024-09-20 22:37:42.986875159 +0000 2024-09-20T22:38:02.392 INFO:teuthology.orchestra.run.smithi154.stdout:Modify: 2024-09-20 22:33:23.712374679 +0000 2024-09-20T22:38:02.392 INFO:teuthology.orchestra.run.smithi154.stdout:Change: 2024-09-20 22:33:23.712374679 +0000 2024-09-20T22:38:02.392 INFO:teuthology.orchestra.run.smithi154.stdout: Birth: 2024-09-20 22:33:23.712374679 +0000 2024-09-20T22:38:02.392 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-20T22:38:02.455 INFO:teuthology.orchestra.run.smithi154.stderr:1+0 records in 2024-09-20T22:38:02.455 INFO:teuthology.orchestra.run.smithi154.stderr:1+0 records out 2024-09-20T22:38:02.455 INFO:teuthology.orchestra.run.smithi154.stderr:512 bytes copied, 0.000197118 s, 2.6 MB/s 2024-09-20T22:38:02.456 DEBUG:teuthology.orchestra.run.smithi154:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-20T22:38:02.512 DEBUG:teuthology.orchestra.run.smithi154:> stat /dev/vg_nvme/lv_4 2024-09-20T22:38:02.568 INFO:teuthology.orchestra.run.smithi154.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-20T22:38:02.568 INFO:teuthology.orchestra.run.smithi154.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-20T22:38:02.568 INFO:teuthology.orchestra.run.smithi154.stdout:Device: 5h/5d Inode: 983 Links: 1 2024-09-20T22:38:02.568 INFO:teuthology.orchestra.run.smithi154.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-20T22:38:02.568 INFO:teuthology.orchestra.run.smithi154.stdout:Context: system_u:object_r:device_t:s0 2024-09-20T22:38:02.569 INFO:teuthology.orchestra.run.smithi154.stdout:Access: 2024-09-20 22:37:42.986875159 +0000 2024-09-20T22:38:02.569 INFO:teuthology.orchestra.run.smithi154.stdout:Modify: 2024-09-20 22:33:23.713374654 +0000 2024-09-20T22:38:02.569 INFO:teuthology.orchestra.run.smithi154.stdout:Change: 2024-09-20 22:33:23.713374654 +0000 2024-09-20T22:38:02.569 INFO:teuthology.orchestra.run.smithi154.stdout: Birth: 2024-09-20 22:33:23.713374654 +0000 2024-09-20T22:38:02.569 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-20T22:38:02.632 INFO:teuthology.orchestra.run.smithi154.stderr:1+0 records in 2024-09-20T22:38:02.632 INFO:teuthology.orchestra.run.smithi154.stderr:1+0 records out 2024-09-20T22:38:02.632 INFO:teuthology.orchestra.run.smithi154.stderr:512 bytes copied, 0.000200885 s, 2.5 MB/s 2024-09-20T22:38:02.633 DEBUG:teuthology.orchestra.run.smithi154:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-20T22:38:02.689 INFO:tasks.cephadm:Deploying osd.0 on smithi084 with /dev/vg_nvme/lv_4... 2024-09-20T22:38:02.689 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- lvm zap /dev/vg_nvme/lv_4 2024-09-20T22:38:02.931 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:38:03.052 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:02 smithi084 ceph-mon[40024]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T22:38:03.052 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:02 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:38:03.052 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:02 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:38:03.052 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:02 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-20T22:38:03.052 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:02 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-20T22:38:03.052 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:02 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:38:03.052 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:02 smithi084 ceph-mon[29774]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T22:38:03.052 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:02 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:38:03.053 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:02 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:38:03.053 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:02 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-20T22:38:03.053 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:02 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-20T22:38:03.053 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:02 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:38:03.249 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:02 smithi154 ceph-mon[33264]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T22:38:03.249 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:02 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:38:03.249 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:02 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:38:03.249 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:02 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-20T22:38:03.249 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:02 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-20T22:38:03.249 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:02 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:38:03.985 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:03 smithi154 ceph-mon[33264]: Reconfiguring mon.b (monmap changed)... 2024-09-20T22:38:03.985 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:03 smithi154 ceph-mon[33264]: Reconfiguring daemon mon.b on smithi154 2024-09-20T22:38:03.985 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:03 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:38:03.986 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:03 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:38:03.986 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:03 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:38:03.986 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:03 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:38:03.986 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:03 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:38:03.986 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:03 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:38:03.986 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:03 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-20T22:38:03.986 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:03 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-20T22:38:03.986 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:03 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-20T22:38:03.986 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:03 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:38:04.184 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:03 smithi084 ceph-mon[29774]: Reconfiguring mon.b (monmap changed)... 2024-09-20T22:38:04.184 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:03 smithi084 ceph-mon[29774]: Reconfiguring daemon mon.b on smithi154 2024-09-20T22:38:04.184 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:03 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:38:04.184 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:03 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:38:04.184 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:03 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:38:04.184 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:03 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:38:04.184 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:03 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:38:04.184 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:03 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:38:04.185 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:03 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-20T22:38:04.185 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:03 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-20T22:38:04.185 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:03 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-20T22:38:04.185 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:03 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:38:04.185 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:03 smithi084 ceph-mon[40024]: Reconfiguring mon.b (monmap changed)... 2024-09-20T22:38:04.185 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:03 smithi084 ceph-mon[40024]: Reconfiguring daemon mon.b on smithi154 2024-09-20T22:38:04.185 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:03 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:38:04.185 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:03 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:38:04.185 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:03 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:38:04.185 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:03 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:38:04.185 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:03 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:38:04.185 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:03 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:38:04.186 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:03 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-20T22:38:04.186 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:03 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-20T22:38:04.186 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:03 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-20T22:38:04.186 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:03 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:38:05.166 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:04 smithi084 ceph-mon[29774]: Deploying daemon mgr.x on smithi154 2024-09-20T22:38:05.166 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:04 smithi084 ceph-mon[29774]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T22:38:05.166 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:04 smithi084 ceph-mon[40024]: Deploying daemon mgr.x on smithi154 2024-09-20T22:38:05.166 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:04 smithi084 ceph-mon[40024]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T22:38:05.169 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-20T22:38:05.194 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph orch daemon add osd smithi084:vg_nvme/lv_4 2024-09-20T22:38:05.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:04 smithi154 ceph-mon[33264]: Deploying daemon mgr.x on smithi154 2024-09-20T22:38:05.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:04 smithi154 ceph-mon[33264]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T22:38:05.254 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Sep 20 22:38:05 smithi154 systemd[1]: Starting Ceph mgr.x for a9c0bc70-77a0-11ef-baf5-efdc52797490... 2024-09-20T22:38:05.434 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:38:05.522 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Sep 20 22:38:05 smithi154 podman[35062]: 2024-09-20 22:38:05.17258463 +0000 UTC m=+0.016142700 image pull cc9251b744a3faafca7c3095f79bfff81873afe823f31a03a11e015a34abcdbd quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5 2024-09-20T22:38:05.522 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Sep 20 22:38:05 smithi154 podman[35062]: 2024-09-20 22:38:05.283868853 +0000 UTC m=+0.127426956 container create d0c4a10d6a7d4a02405b22afc9bea57dc7977dc16bf4e7a62d18f41beb29469e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-x, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, RELEASE=squid-9e2ac70, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2) 2024-09-20T22:38:05.522 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Sep 20 22:38:05 smithi154 podman[35062]: 2024-09-20 22:38:05.458643988 +0000 UTC m=+0.302202050 container init d0c4a10d6a7d4a02405b22afc9bea57dc7977dc16bf4e7a62d18f41beb29469e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-x, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=squid-9e2ac70, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-20T22:38:05.522 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Sep 20 22:38:05 smithi154 podman[35062]: 2024-09-20 22:38:05.461579326 +0000 UTC m=+0.305137395 container start d0c4a10d6a7d4a02405b22afc9bea57dc7977dc16bf4e7a62d18f41beb29469e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-x, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, RELEASE=squid-9e2ac70, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD) 2024-09-20T22:38:05.522 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Sep 20 22:38:05 smithi154 ceph-mgr[35081]: -- 172.21.15.154:0/29234871 <== mon.1 v2:172.21.15.154:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55a11f8710e0 con 0x55a11f84f000 2024-09-20T22:38:05.522 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Sep 20 22:38:05 smithi154 bash[35062]: d0c4a10d6a7d4a02405b22afc9bea57dc7977dc16bf4e7a62d18f41beb29469e 2024-09-20T22:38:05.522 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Sep 20 22:38:05 smithi154 systemd[1]: Started Ceph mgr.x for a9c0bc70-77a0-11ef-baf5-efdc52797490. 2024-09-20T22:38:05.811 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Sep 20 22:38:05 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-x[35077]: 2024-09-20T22:38:05.594+0000 7fb310ab0040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-20T22:38:05.811 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Sep 20 22:38:05 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-x[35077]: 2024-09-20T22:38:05.652+0000 7fb310ab0040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-20T22:38:05.811 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Sep 20 22:38:05 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-x[35077]: 2024-09-20T22:38:05.809+0000 7fb310ab0040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-20T22:38:06.192 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Sep 20 22:38:05 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-x[35077]: 2024-09-20T22:38:05.877+0000 7fb310ab0040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-20T22:38:06.192 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Sep 20 22:38:06 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-x[35077]: 2024-09-20T22:38:06.042+0000 7fb310ab0040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-20T22:38:06.192 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Sep 20 22:38:06 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-x[35077]: 2024-09-20T22:38:06.190+0000 7fb310ab0040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-20T22:38:06.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Sep 20 22:38:06 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-x[35077]: 2024-09-20T22:38:06.515+0000 7fb310ab0040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-20T22:38:06.776 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Sep 20 22:38:06 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-x[35077]: 2024-09-20T22:38:06.578+0000 7fb310ab0040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-20T22:38:06.776 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:06 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:38:06.777 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:06 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:38:06.777 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:06 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:38:06.777 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:06 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:38:06.777 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:06 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:38:06.777 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:06 smithi154 ceph-mon[33264]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T22:38:06.777 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:06 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-20T22:38:06.777 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:06 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-20T22:38:06.777 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:06 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:38:06.781 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:06 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:38:06.781 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:06 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:38:06.781 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:06 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:38:06.781 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:06 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:38:06.781 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:06 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:38:06.781 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:06 smithi084 ceph-mon[40024]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T22:38:06.781 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:06 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-20T22:38:06.781 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:06 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-20T22:38:06.781 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:06 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:38:06.781 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:06 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:38:06.782 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:06 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:38:06.782 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:06 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:38:06.782 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:06 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:38:06.782 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:06 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:38:06.782 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:06 smithi084 ceph-mon[29774]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T22:38:06.782 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:06 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-20T22:38:06.782 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:06 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-20T22:38:06.782 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:06 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:38:07.254 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Sep 20 22:38:06 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-x[35077]: 2024-09-20T22:38:06.774+0000 7fb310ab0040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-20T22:38:07.985 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:07 smithi084 ceph-mon[40024]: from='client.24113 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi084:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:38:07.985 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:07 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:38:07.985 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:07 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:38:07.985 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:07 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:38:07.986 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:07 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:38:07.986 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:07 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:38:07.986 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:07 smithi084 ceph-mon[40024]: Reconfiguring mgr.y (unknown last config time)... 2024-09-20T22:38:07.986 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:07 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-20T22:38:07.986 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:07 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-20T22:38:07.986 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:07 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:38:07.986 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:07 smithi084 ceph-mon[40024]: Reconfiguring daemon mgr.y on smithi084 2024-09-20T22:38:07.986 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:07 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:38:07.986 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:07 smithi084 ceph-mon[29774]: from='client.24113 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi084:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:38:07.986 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:07 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:38:07.986 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:07 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:38:07.986 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:07 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:38:07.987 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:07 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:38:07.987 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:07 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:38:07.987 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:07 smithi084 ceph-mon[29774]: Reconfiguring mgr.y (unknown last config time)... 2024-09-20T22:38:07.987 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:07 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-20T22:38:07.987 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:07 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-20T22:38:07.987 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:07 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:38:07.987 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:07 smithi084 ceph-mon[29774]: Reconfiguring daemon mgr.y on smithi084 2024-09-20T22:38:07.987 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:07 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:38:08.004 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Sep 20 22:38:07 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-x[35077]: 2024-09-20T22:38:07.654+0000 7fb310ab0040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-20T22:38:08.004 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Sep 20 22:38:07 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-x[35077]: 2024-09-20T22:38:07.724+0000 7fb310ab0040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-20T22:38:08.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:07 smithi154 ceph-mon[33264]: from='client.24113 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi084:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:38:08.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:07 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:38:08.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:07 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:38:08.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:07 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:38:08.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:07 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:38:08.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:07 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:38:08.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:07 smithi154 ceph-mon[33264]: Reconfiguring mgr.y (unknown last config time)... 2024-09-20T22:38:08.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:07 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-20T22:38:08.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:07 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-20T22:38:08.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:07 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:38:08.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:07 smithi154 ceph-mon[33264]: Reconfiguring daemon mgr.y on smithi084 2024-09-20T22:38:08.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:07 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:38:08.591 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Sep 20 22:38:08 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-x[35077]: 2024-09-20T22:38:08.473+0000 7fb310ab0040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-20T22:38:08.591 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Sep 20 22:38:08 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-x[35077]: 2024-09-20T22:38:08.589+0000 7fb310ab0040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-20T22:38:08.945 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Sep 20 22:38:08 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-x[35077]: 2024-09-20T22:38:08.732+0000 7fb310ab0040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-20T22:38:08.945 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Sep 20 22:38:08 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-x[35077]: 2024-09-20T22:38:08.794+0000 7fb310ab0040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-20T22:38:08.945 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Sep 20 22:38:08 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-x[35077]: 2024-09-20T22:38:08.943+0000 7fb310ab0040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-20T22:38:08.945 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:08 smithi154 ceph-mon[33264]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T22:38:08.945 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:08 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/284304423' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ea36a489-6f91-4b2f-b784-b90506b9805d"}]: dispatch 2024-09-20T22:38:08.945 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:08 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/284304423' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ea36a489-6f91-4b2f-b784-b90506b9805d"}]': finished 2024-09-20T22:38:08.945 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:08 smithi154 ceph-mon[33264]: osdmap e5: 1 total, 0 up, 1 in 2024-09-20T22:38:08.945 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:08 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-20T22:38:09.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:08 smithi084 ceph-mon[29774]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T22:38:09.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:08 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/284304423' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ea36a489-6f91-4b2f-b784-b90506b9805d"}]: dispatch 2024-09-20T22:38:09.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:08 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/284304423' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ea36a489-6f91-4b2f-b784-b90506b9805d"}]': finished 2024-09-20T22:38:09.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:08 smithi084 ceph-mon[29774]: osdmap e5: 1 total, 0 up, 1 in 2024-09-20T22:38:09.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:08 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-20T22:38:09.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:08 smithi084 ceph-mon[40024]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T22:38:09.116 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:08 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/284304423' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ea36a489-6f91-4b2f-b784-b90506b9805d"}]: dispatch 2024-09-20T22:38:09.116 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:08 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/284304423' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ea36a489-6f91-4b2f-b784-b90506b9805d"}]': finished 2024-09-20T22:38:09.116 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:08 smithi084 ceph-mon[40024]: osdmap e5: 1 total, 0 up, 1 in 2024-09-20T22:38:09.116 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:08 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-20T22:38:09.229 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Sep 20 22:38:09 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-x[35077]: 2024-09-20T22:38:09.011+0000 7fb310ab0040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-20T22:38:09.229 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Sep 20 22:38:09 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-x[35077]: 2024-09-20T22:38:09.070+0000 7fb310ab0040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-20T22:38:09.229 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Sep 20 22:38:09 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-x[35077]: 2024-09-20T22:38:09.227+0000 7fb310ab0040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-20T22:38:09.499 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Sep 20 22:38:09 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-x[35077]: 2024-09-20T22:38:09.292+0000 7fb310ab0040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-20T22:38:09.499 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Sep 20 22:38:09 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-x[35077]: 2024-09-20T22:38:09.357+0000 7fb310ab0040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-20T22:38:09.499 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Sep 20 22:38:09 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-x[35077]: 2024-09-20T22:38:09.438+0000 7fb310ab0040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-20T22:38:09.754 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Sep 20 22:38:09 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-x[35077]: 2024-09-20T22:38:09.497+0000 7fb310ab0040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-20T22:38:09.754 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Sep 20 22:38:09 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-x[35077]: 2024-09-20T22:38:09.556+0000 7fb310ab0040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-20T22:38:10.114 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:09 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/395808460' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-20T22:38:10.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:09 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/395808460' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-20T22:38:10.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:09 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/395808460' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-20T22:38:10.754 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Sep 20 22:38:10 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-x[35077]: 2024-09-20T22:38:10.341+0000 7fb310ab0040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-20T22:38:11.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:10 smithi084 ceph-mon[40024]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T22:38:11.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:10 smithi084 ceph-mon[40024]: Standby manager daemon x started 2024-09-20T22:38:11.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:10 smithi084 ceph-mon[40024]: from='mgr.? 172.21.15.154:0/737142281' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-20T22:38:11.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:10 smithi084 ceph-mon[40024]: from='mgr.? 172.21.15.154:0/737142281' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-20T22:38:11.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:10 smithi084 ceph-mon[40024]: from='mgr.? 172.21.15.154:0/737142281' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-20T22:38:11.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:10 smithi084 ceph-mon[40024]: from='mgr.? 172.21.15.154:0/737142281' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-20T22:38:11.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:10 smithi084 ceph-mon[29774]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T22:38:11.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:10 smithi084 ceph-mon[29774]: Standby manager daemon x started 2024-09-20T22:38:11.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:10 smithi084 ceph-mon[29774]: from='mgr.? 172.21.15.154:0/737142281' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-20T22:38:11.116 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:10 smithi084 ceph-mon[29774]: from='mgr.? 172.21.15.154:0/737142281' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-20T22:38:11.116 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:10 smithi084 ceph-mon[29774]: from='mgr.? 172.21.15.154:0/737142281' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-20T22:38:11.116 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:10 smithi084 ceph-mon[29774]: from='mgr.? 172.21.15.154:0/737142281' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-20T22:38:11.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:10 smithi154 ceph-mon[33264]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T22:38:11.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:10 smithi154 ceph-mon[33264]: Standby manager daemon x started 2024-09-20T22:38:11.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:10 smithi154 ceph-mon[33264]: from='mgr.? 172.21.15.154:0/737142281' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-20T22:38:11.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:10 smithi154 ceph-mon[33264]: from='mgr.? 172.21.15.154:0/737142281' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-20T22:38:11.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:10 smithi154 ceph-mon[33264]: from='mgr.? 172.21.15.154:0/737142281' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-20T22:38:11.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:10 smithi154 ceph-mon[33264]: from='mgr.? 172.21.15.154:0/737142281' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-20T22:38:12.105 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:11 smithi084 ceph-mon[29774]: mgrmap e14: y(active, since 53s), standbys: x 2024-09-20T22:38:12.106 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:11 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-20T22:38:12.106 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:11 smithi084 ceph-mon[40024]: mgrmap e14: y(active, since 53s), standbys: x 2024-09-20T22:38:12.106 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:11 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-20T22:38:12.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:11 smithi154 ceph-mon[33264]: mgrmap e14: y(active, since 53s), standbys: x 2024-09-20T22:38:12.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:11 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-20T22:38:13.059 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:12 smithi084 ceph-mon[40024]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T22:38:13.059 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:12 smithi084 ceph-mon[29774]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T22:38:13.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:12 smithi154 ceph-mon[33264]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T22:38:14.957 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:14 smithi084 ceph-mon[40024]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T22:38:14.957 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:14 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:38:14.957 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:14 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:38:14.957 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:14 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:38:14.957 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:14 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:38:14.957 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:14 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:38:14.957 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:14 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:38:14.958 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:14 smithi084 ceph-mon[29774]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T22:38:14.958 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:14 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:38:14.958 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:14 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:38:14.958 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:14 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:38:14.958 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:14 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:38:14.958 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:14 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:38:14.958 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:14 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:38:15.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:14 smithi154 ceph-mon[33264]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T22:38:15.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:14 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:38:15.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:14 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:38:15.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:14 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:38:15.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:14 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:38:15.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:14 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:38:15.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:14 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:38:16.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:16 smithi084 ceph-mon[29774]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T22:38:16.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:16 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-09-20T22:38:16.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:16 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:38:16.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:16 smithi084 ceph-mon[29774]: Deploying daemon osd.0 on smithi084 2024-09-20T22:38:16.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:16 smithi084 ceph-mon[40024]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T22:38:16.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:16 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-09-20T22:38:16.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:16 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:38:16.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:16 smithi084 ceph-mon[40024]: Deploying daemon osd.0 on smithi084 2024-09-20T22:38:17.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:16 smithi154 ceph-mon[33264]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T22:38:17.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:16 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-09-20T22:38:17.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:16 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:38:17.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:16 smithi154 ceph-mon[33264]: Deploying daemon osd.0 on smithi084 2024-09-20T22:38:19.095 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:18 smithi084 ceph-mon[40024]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T22:38:19.095 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:18 smithi084 ceph-mon[29774]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T22:38:19.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:18 smithi154 ceph-mon[33264]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T22:38:21.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:20 smithi084 ceph-mon[29774]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T22:38:21.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:20 smithi084 ceph-mon[40024]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T22:38:21.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:20 smithi154 ceph-mon[33264]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T22:38:23.066 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:22 smithi084 ceph-mon[29774]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T22:38:23.066 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:22 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:38:23.066 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:22 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:38:23.066 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:22 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:38:23.066 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:22 smithi084 ceph-mon[40024]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T22:38:23.067 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:22 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:38:23.067 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:22 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:38:23.067 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:22 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:38:23.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:22 smithi154 ceph-mon[33264]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T22:38:23.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:22 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:38:23.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:22 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:38:23.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:22 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:38:25.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:24 smithi084 ceph-mon[29774]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T22:38:25.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:24 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:38:25.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:24 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:38:25.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:24 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:38:25.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:24 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:38:25.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:24 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:38:25.116 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:24 smithi084 ceph-mon[40024]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T22:38:25.116 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:24 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:38:25.116 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:24 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:38:25.116 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:24 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:38:25.116 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:24 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:38:25.116 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:24 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:38:25.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:24 smithi154 ceph-mon[33264]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T22:38:25.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:24 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:38:25.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:24 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:38:25.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:24 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:38:25.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:24 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:38:25.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:24 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:38:25.962 INFO:teuthology.orchestra.run.smithi084.stdout:Created osd(s) 0 on host 'smithi084' 2024-09-20T22:38:26.540 DEBUG:teuthology.orchestra.run.smithi084:osd.0> sudo journalctl -f -n 0 -u ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490@osd.0.service 2024-09-20T22:38:26.542 INFO:tasks.cephadm:Deploying osd.1 on smithi084 with /dev/vg_nvme/lv_3... 2024-09-20T22:38:26.542 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- lvm zap /dev/vg_nvme/lv_3 2024-09-20T22:38:27.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:26 smithi084 ceph-mon[40024]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T22:38:27.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:26 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:38:27.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:26 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:38:27.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:26 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:38:27.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:26 smithi084 ceph-mon[29774]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T22:38:27.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:26 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:38:27.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:26 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:38:27.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:26 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:38:27.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:26 smithi154 ceph-mon[33264]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T22:38:27.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:26 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:38:27.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:26 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:38:27.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:26 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:38:27.261 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:38:27.535 INFO:journalctl@ceph.osd.0.smithi084.stdout:Sep 20 22:38:27 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-0[43832]: 2024-09-20T22:38:27.264+0000 7f13e550e640 -1 osd.0 0 log_to_monitors true 2024-09-20T22:38:28.114 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:27 smithi084 ceph-mon[40024]: from='osd.0 v2:172.21.15.84:6801/268084626' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-20T22:38:28.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:27 smithi084 ceph-mon[29774]: from='osd.0 v2:172.21.15.84:6801/268084626' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-20T22:38:28.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:27 smithi154 ceph-mon[33264]: from='osd.0 v2:172.21.15.84:6801/268084626' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-20T22:38:29.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:28 smithi084 ceph-mon[29774]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T22:38:29.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:28 smithi084 ceph-mon[29774]: from='osd.0 v2:172.21.15.84:6801/268084626' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-20T22:38:29.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:28 smithi084 ceph-mon[29774]: osdmap e6: 1 total, 0 up, 1 in 2024-09-20T22:38:29.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:28 smithi084 ceph-mon[29774]: from='osd.0 v2:172.21.15.84:6801/268084626' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi084", "root=default"]}]: dispatch 2024-09-20T22:38:29.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:28 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-20T22:38:29.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:28 smithi084 ceph-mon[40024]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T22:38:29.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:28 smithi084 ceph-mon[40024]: from='osd.0 v2:172.21.15.84:6801/268084626' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-20T22:38:29.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:28 smithi084 ceph-mon[40024]: osdmap e6: 1 total, 0 up, 1 in 2024-09-20T22:38:29.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:28 smithi084 ceph-mon[40024]: from='osd.0 v2:172.21.15.84:6801/268084626' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi084", "root=default"]}]: dispatch 2024-09-20T22:38:29.116 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:28 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-20T22:38:29.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:28 smithi154 ceph-mon[33264]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T22:38:29.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:28 smithi154 ceph-mon[33264]: from='osd.0 v2:172.21.15.84:6801/268084626' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-20T22:38:29.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:28 smithi154 ceph-mon[33264]: osdmap e6: 1 total, 0 up, 1 in 2024-09-20T22:38:29.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:28 smithi154 ceph-mon[33264]: from='osd.0 v2:172.21.15.84:6801/268084626' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi084", "root=default"]}]: dispatch 2024-09-20T22:38:29.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:28 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-20T22:38:29.400 INFO:journalctl@ceph.osd.0.smithi084.stdout:Sep 20 22:38:29 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-0[43832]: 2024-09-20T22:38:29.306+0000 7f13e1491640 -1 osd.0 0 waiting for initial osdmap 2024-09-20T22:38:29.400 INFO:journalctl@ceph.osd.0.smithi084.stdout:Sep 20 22:38:29 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-0[43832]: 2024-09-20T22:38:29.308+0000 7f13d92b1640 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-20T22:38:30.165 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:29 smithi084 ceph-mon[29774]: from='osd.0 v2:172.21.15.84:6801/268084626' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi084", "root=default"]}]': finished 2024-09-20T22:38:30.166 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:29 smithi084 ceph-mon[29774]: osdmap e7: 1 total, 0 up, 1 in 2024-09-20T22:38:30.166 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:29 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-20T22:38:30.166 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:29 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-20T22:38:30.166 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:29 smithi084 ceph-mon[29774]: from='osd.0 v2:172.21.15.84:6801/268084626' entity='osd.0' 2024-09-20T22:38:30.166 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:29 smithi084 ceph-mon[29774]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T22:38:30.166 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:29 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-20T22:38:30.166 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:29 smithi084 ceph-mon[40024]: from='osd.0 v2:172.21.15.84:6801/268084626' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi084", "root=default"]}]': finished 2024-09-20T22:38:30.166 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:29 smithi084 ceph-mon[40024]: osdmap e7: 1 total, 0 up, 1 in 2024-09-20T22:38:30.166 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:29 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-20T22:38:30.166 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:29 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-20T22:38:30.166 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:29 smithi084 ceph-mon[40024]: from='osd.0 v2:172.21.15.84:6801/268084626' entity='osd.0' 2024-09-20T22:38:30.166 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:29 smithi084 ceph-mon[40024]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T22:38:30.167 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:29 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-20T22:38:30.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:29 smithi154 ceph-mon[33264]: from='osd.0 v2:172.21.15.84:6801/268084626' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi084", "root=default"]}]': finished 2024-09-20T22:38:30.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:29 smithi154 ceph-mon[33264]: osdmap e7: 1 total, 0 up, 1 in 2024-09-20T22:38:30.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:29 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-20T22:38:30.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:29 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-20T22:38:30.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:29 smithi154 ceph-mon[33264]: from='osd.0 v2:172.21.15.84:6801/268084626' entity='osd.0' 2024-09-20T22:38:30.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:29 smithi154 ceph-mon[33264]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T22:38:30.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:29 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-20T22:38:31.473 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:31 smithi084 ceph-mon[40024]: purged_snaps scrub starts 2024-09-20T22:38:31.473 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:31 smithi084 ceph-mon[40024]: purged_snaps scrub ok 2024-09-20T22:38:31.473 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:31 smithi084 ceph-mon[40024]: Detected new or changed devices on smithi084 2024-09-20T22:38:31.473 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:31 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:38:31.473 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:31 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:38:31.474 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:31 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:38:31.474 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:31 smithi084 ceph-mon[40024]: Adjusting osd_memory_target on smithi084 to 16064M 2024-09-20T22:38:31.474 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:31 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:38:31.474 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:31 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:38:31.474 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:31 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:38:31.474 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:31 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:38:31.474 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:31 smithi084 ceph-mon[40024]: osd.0 v2:172.21.15.84:6801/268084626 boot 2024-09-20T22:38:31.474 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:31 smithi084 ceph-mon[40024]: osdmap e8: 1 total, 1 up, 1 in 2024-09-20T22:38:31.474 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:31 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-20T22:38:31.474 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:31 smithi084 ceph-mon[29774]: purged_snaps scrub starts 2024-09-20T22:38:31.474 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:31 smithi084 ceph-mon[29774]: purged_snaps scrub ok 2024-09-20T22:38:31.474 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:31 smithi084 ceph-mon[29774]: Detected new or changed devices on smithi084 2024-09-20T22:38:31.475 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:31 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:38:31.475 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:31 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:38:31.475 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:31 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:38:31.475 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:31 smithi084 ceph-mon[29774]: Adjusting osd_memory_target on smithi084 to 16064M 2024-09-20T22:38:31.475 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:31 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:38:31.475 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:31 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:38:31.475 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:31 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:38:31.475 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:31 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:38:31.475 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:31 smithi084 ceph-mon[29774]: osd.0 v2:172.21.15.84:6801/268084626 boot 2024-09-20T22:38:31.475 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:31 smithi084 ceph-mon[29774]: osdmap e8: 1 total, 1 up, 1 in 2024-09-20T22:38:31.475 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:31 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-20T22:38:31.476 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-20T22:38:31.501 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph orch daemon add osd smithi084:vg_nvme/lv_3 2024-09-20T22:38:31.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:31 smithi154 ceph-mon[33264]: purged_snaps scrub starts 2024-09-20T22:38:31.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:31 smithi154 ceph-mon[33264]: purged_snaps scrub ok 2024-09-20T22:38:31.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:31 smithi154 ceph-mon[33264]: Detected new or changed devices on smithi084 2024-09-20T22:38:31.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:31 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:38:31.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:31 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:38:31.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:31 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:38:31.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:31 smithi154 ceph-mon[33264]: Adjusting osd_memory_target on smithi084 to 16064M 2024-09-20T22:38:31.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:31 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:38:31.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:31 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:38:31.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:31 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:38:31.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:31 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:38:31.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:31 smithi154 ceph-mon[33264]: osd.0 v2:172.21.15.84:6801/268084626 boot 2024-09-20T22:38:31.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:31 smithi154 ceph-mon[33264]: osdmap e8: 1 total, 1 up, 1 in 2024-09-20T22:38:31.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:31 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-20T22:38:31.746 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:38:32.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:32 smithi084 ceph-mon[29774]: osdmap e9: 1 total, 1 up, 1 in 2024-09-20T22:38:32.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:32 smithi084 ceph-mon[29774]: pgmap v36: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-20T22:38:32.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:32 smithi084 ceph-mon[40024]: osdmap e9: 1 total, 1 up, 1 in 2024-09-20T22:38:32.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:32 smithi084 ceph-mon[40024]: pgmap v36: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-20T22:38:32.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:32 smithi154 ceph-mon[33264]: osdmap e9: 1 total, 1 up, 1 in 2024-09-20T22:38:32.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:32 smithi154 ceph-mon[33264]: pgmap v36: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-20T22:38:33.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:33 smithi084 ceph-mon[29774]: from='client.24140 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi084:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:38:33.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:33 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-20T22:38:33.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:33 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-20T22:38:33.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:33 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:38:33.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:33 smithi084 ceph-mon[40024]: from='client.24140 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi084:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:38:33.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:33 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-20T22:38:33.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:33 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-20T22:38:33.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:33 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:38:33.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:33 smithi154 ceph-mon[33264]: from='client.24140 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi084:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:38:33.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:33 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-20T22:38:33.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:33 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-20T22:38:33.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:33 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:38:34.614 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:34 smithi084 ceph-mon[29774]: pgmap v37: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-20T22:38:34.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:34 smithi084 ceph-mon[40024]: pgmap v37: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-20T22:38:34.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:34 smithi154 ceph-mon[33264]: pgmap v37: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-20T22:38:35.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:35 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/1778490363' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c8649d2f-27ef-47f8-bc94-f1cd9e62bdb8"}]: dispatch 2024-09-20T22:38:35.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:35 smithi084 ceph-mon[40024]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c8649d2f-27ef-47f8-bc94-f1cd9e62bdb8"}]: dispatch 2024-09-20T22:38:35.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:35 smithi084 ceph-mon[40024]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c8649d2f-27ef-47f8-bc94-f1cd9e62bdb8"}]': finished 2024-09-20T22:38:35.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:35 smithi084 ceph-mon[40024]: osdmap e10: 2 total, 1 up, 2 in 2024-09-20T22:38:35.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:35 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-20T22:38:35.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:35 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/2489821968' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-20T22:38:35.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:35 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/1778490363' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c8649d2f-27ef-47f8-bc94-f1cd9e62bdb8"}]: dispatch 2024-09-20T22:38:35.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:35 smithi084 ceph-mon[29774]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c8649d2f-27ef-47f8-bc94-f1cd9e62bdb8"}]: dispatch 2024-09-20T22:38:35.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:35 smithi084 ceph-mon[29774]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c8649d2f-27ef-47f8-bc94-f1cd9e62bdb8"}]': finished 2024-09-20T22:38:35.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:35 smithi084 ceph-mon[29774]: osdmap e10: 2 total, 1 up, 2 in 2024-09-20T22:38:35.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:35 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-20T22:38:35.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:35 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/2489821968' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-20T22:38:35.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:35 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/1778490363' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c8649d2f-27ef-47f8-bc94-f1cd9e62bdb8"}]: dispatch 2024-09-20T22:38:35.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:35 smithi154 ceph-mon[33264]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c8649d2f-27ef-47f8-bc94-f1cd9e62bdb8"}]: dispatch 2024-09-20T22:38:35.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:35 smithi154 ceph-mon[33264]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c8649d2f-27ef-47f8-bc94-f1cd9e62bdb8"}]': finished 2024-09-20T22:38:35.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:35 smithi154 ceph-mon[33264]: osdmap e10: 2 total, 1 up, 2 in 2024-09-20T22:38:35.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:35 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-20T22:38:35.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:35 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/2489821968' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-20T22:38:36.614 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:36 smithi084 ceph-mon[29774]: pgmap v39: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-20T22:38:36.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:36 smithi084 ceph-mon[40024]: pgmap v39: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-20T22:38:36.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:36 smithi154 ceph-mon[33264]: pgmap v39: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-20T22:38:39.098 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:38 smithi084 ceph-mon[29774]: pgmap v40: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-20T22:38:39.099 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:38 smithi084 ceph-mon[40024]: pgmap v40: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-20T22:38:39.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:38 smithi154 ceph-mon[33264]: pgmap v40: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-20T22:38:40.946 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:40 smithi084 ceph-mon[29774]: pgmap v41: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-20T22:38:40.946 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:40 smithi084 ceph-mon[40024]: pgmap v41: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-20T22:38:41.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:40 smithi154 ceph-mon[33264]: pgmap v41: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-20T22:38:43.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:42 smithi084 ceph-mon[29774]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-20T22:38:43.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:42 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-09-20T22:38:43.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:42 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:38:43.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:42 smithi084 ceph-mon[40024]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-20T22:38:43.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:42 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-09-20T22:38:43.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:42 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:38:43.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:42 smithi154 ceph-mon[33264]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-20T22:38:43.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:42 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-09-20T22:38:43.258 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:42 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:38:44.089 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:43 smithi084 ceph-mon[29774]: Deploying daemon osd.1 on smithi084 2024-09-20T22:38:44.089 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:43 smithi084 ceph-mon[40024]: Deploying daemon osd.1 on smithi084 2024-09-20T22:38:44.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:43 smithi154 ceph-mon[33264]: Deploying daemon osd.1 on smithi084 2024-09-20T22:38:45.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:44 smithi084 ceph-mon[29774]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-20T22:38:45.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:44 smithi084 ceph-mon[40024]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-20T22:38:45.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:44 smithi154 ceph-mon[33264]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-20T22:38:47.083 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:46 smithi084 ceph-mon[29774]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-20T22:38:47.083 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:46 smithi084 ceph-mon[40024]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-20T22:38:47.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:46 smithi154 ceph-mon[33264]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-20T22:38:48.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:48 smithi084 ceph-mon[29774]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-20T22:38:48.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:48 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:38:48.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:48 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:38:48.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:48 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:38:48.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:48 smithi084 ceph-mon[40024]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-20T22:38:48.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:48 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:38:48.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:48 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:38:48.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:48 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:38:49.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:48 smithi154 ceph-mon[33264]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-20T22:38:49.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:48 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:38:49.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:48 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:38:49.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:48 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:38:51.033 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:50 smithi084 ceph-mon[29774]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-20T22:38:51.034 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:50 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:38:51.034 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:50 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:38:51.034 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:50 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:38:51.034 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:50 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:38:51.034 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:50 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:38:51.034 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:50 smithi084 ceph-mon[40024]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-20T22:38:51.034 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:50 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:38:51.034 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:50 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:38:51.035 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:50 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:38:51.035 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:50 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:38:51.035 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:50 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:38:51.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:50 smithi154 ceph-mon[33264]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-20T22:38:51.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:50 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:38:51.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:50 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:38:51.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:50 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:38:51.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:50 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:38:51.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:50 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:38:51.629 INFO:teuthology.orchestra.run.smithi084.stdout:Created osd(s) 1 on host 'smithi084' 2024-09-20T22:38:51.864 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:51 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:38:51.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:51 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:38:51.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:51 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:38:51.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:51 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:38:51.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:51 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:38:51.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:51 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:38:52.249 DEBUG:teuthology.orchestra.run.smithi084:osd.1> sudo journalctl -f -n 0 -u ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490@osd.1.service 2024-09-20T22:38:52.252 INFO:tasks.cephadm:Deploying osd.2 on smithi084 with /dev/vg_nvme/lv_2... 2024-09-20T22:38:52.252 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- lvm zap /dev/vg_nvme/lv_2 2024-09-20T22:38:52.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:51 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:38:52.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:51 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:38:52.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:51 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:38:52.908 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:38:53.051 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:52 smithi084 ceph-mon[29774]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-20T22:38:53.052 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:52 smithi084 ceph-mon[29774]: from='osd.1 v2:172.21.15.84:6805/418374136' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-20T22:38:53.052 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:52 smithi084 ceph-mon[29774]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-20T22:38:53.052 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:52 smithi084 ceph-mon[40024]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-20T22:38:53.052 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:52 smithi084 ceph-mon[40024]: from='osd.1 v2:172.21.15.84:6805/418374136' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-20T22:38:53.052 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:52 smithi084 ceph-mon[40024]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-20T22:38:53.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:52 smithi154 ceph-mon[33264]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-20T22:38:53.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:52 smithi154 ceph-mon[33264]: from='osd.1 v2:172.21.15.84:6805/418374136' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-20T22:38:53.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:52 smithi154 ceph-mon[33264]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-20T22:38:54.050 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:53 smithi084 ceph-mon[29774]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-20T22:38:54.050 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:53 smithi084 ceph-mon[29774]: osdmap e11: 2 total, 1 up, 2 in 2024-09-20T22:38:54.050 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:53 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-20T22:38:54.050 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:53 smithi084 ceph-mon[29774]: from='osd.1 v2:172.21.15.84:6805/418374136' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi084", "root=default"]}]: dispatch 2024-09-20T22:38:54.050 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:53 smithi084 ceph-mon[29774]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi084", "root=default"]}]: dispatch 2024-09-20T22:38:54.050 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:53 smithi084 ceph-mon[40024]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-20T22:38:54.051 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:53 smithi084 ceph-mon[40024]: osdmap e11: 2 total, 1 up, 2 in 2024-09-20T22:38:54.051 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:53 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-20T22:38:54.051 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:53 smithi084 ceph-mon[40024]: from='osd.1 v2:172.21.15.84:6805/418374136' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi084", "root=default"]}]: dispatch 2024-09-20T22:38:54.051 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:53 smithi084 ceph-mon[40024]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi084", "root=default"]}]: dispatch 2024-09-20T22:38:54.051 INFO:journalctl@ceph.osd.1.smithi084.stdout:Sep 20 22:38:53 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-1[49440]: 2024-09-20T22:38:53.797+0000 7f194a21f640 -1 osd.1 0 waiting for initial osdmap 2024-09-20T22:38:54.051 INFO:journalctl@ceph.osd.1.smithi084.stdout:Sep 20 22:38:53 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-1[49440]: 2024-09-20T22:38:53.799+0000 7f1946047640 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-20T22:38:54.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:53 smithi154 ceph-mon[33264]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-20T22:38:54.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:53 smithi154 ceph-mon[33264]: osdmap e11: 2 total, 1 up, 2 in 2024-09-20T22:38:54.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:53 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-20T22:38:54.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:53 smithi154 ceph-mon[33264]: from='osd.1 v2:172.21.15.84:6805/418374136' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi084", "root=default"]}]: dispatch 2024-09-20T22:38:54.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:53 smithi154 ceph-mon[33264]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi084", "root=default"]}]: dispatch 2024-09-20T22:38:55.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:54 smithi084 ceph-mon[29774]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-20T22:38:55.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:54 smithi084 ceph-mon[29774]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi084", "root=default"]}]': finished 2024-09-20T22:38:55.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:54 smithi084 ceph-mon[29774]: osdmap e12: 2 total, 1 up, 2 in 2024-09-20T22:38:55.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:54 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-20T22:38:55.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:54 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-20T22:38:55.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:54 smithi084 ceph-mon[40024]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-20T22:38:55.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:54 smithi084 ceph-mon[40024]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi084", "root=default"]}]': finished 2024-09-20T22:38:55.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:54 smithi084 ceph-mon[40024]: osdmap e12: 2 total, 1 up, 2 in 2024-09-20T22:38:55.116 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:54 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-20T22:38:55.116 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:54 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-20T22:38:55.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:54 smithi154 ceph-mon[33264]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-20T22:38:55.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:54 smithi154 ceph-mon[33264]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi084", "root=default"]}]': finished 2024-09-20T22:38:55.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:54 smithi154 ceph-mon[33264]: osdmap e12: 2 total, 1 up, 2 in 2024-09-20T22:38:55.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:54 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-20T22:38:55.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:54 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-20T22:38:55.879 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:55 smithi084 ceph-mon[29774]: purged_snaps scrub starts 2024-09-20T22:38:55.879 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:55 smithi084 ceph-mon[29774]: purged_snaps scrub ok 2024-09-20T22:38:55.879 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:55 smithi084 ceph-mon[29774]: osd.1 v2:172.21.15.84:6805/418374136 boot 2024-09-20T22:38:55.879 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:55 smithi084 ceph-mon[29774]: osdmap e13: 2 total, 2 up, 2 in 2024-09-20T22:38:55.879 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:55 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-20T22:38:55.879 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:55 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:38:55.879 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:55 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:38:55.879 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:55 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:38:55.879 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:55 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:38:55.880 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:55 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:38:55.880 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:55 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:38:55.880 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:55 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:38:55.880 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:55 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:38:55.880 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:55 smithi084 ceph-mon[40024]: purged_snaps scrub starts 2024-09-20T22:38:55.880 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:55 smithi084 ceph-mon[40024]: purged_snaps scrub ok 2024-09-20T22:38:55.880 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:55 smithi084 ceph-mon[40024]: osd.1 v2:172.21.15.84:6805/418374136 boot 2024-09-20T22:38:55.880 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:55 smithi084 ceph-mon[40024]: osdmap e13: 2 total, 2 up, 2 in 2024-09-20T22:38:55.880 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:55 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-20T22:38:55.880 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:55 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:38:55.880 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:55 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:38:55.880 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:55 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:38:55.881 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:55 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:38:55.881 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:55 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:38:55.881 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:55 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:38:55.881 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:55 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:38:55.881 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:55 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:38:56.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:55 smithi154 ceph-mon[33264]: purged_snaps scrub starts 2024-09-20T22:38:56.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:55 smithi154 ceph-mon[33264]: purged_snaps scrub ok 2024-09-20T22:38:56.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:55 smithi154 ceph-mon[33264]: osd.1 v2:172.21.15.84:6805/418374136 boot 2024-09-20T22:38:56.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:55 smithi154 ceph-mon[33264]: osdmap e13: 2 total, 2 up, 2 in 2024-09-20T22:38:56.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:55 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-20T22:38:56.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:55 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:38:56.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:55 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:38:56.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:55 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:38:56.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:55 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:38:56.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:55 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:38:56.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:55 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:38:56.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:55 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:38:56.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:55 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:38:56.785 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-20T22:38:56.810 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph orch daemon add osd smithi084:vg_nvme/lv_2 2024-09-20T22:38:57.053 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:38:57.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:56 smithi084 ceph-mon[40024]: Detected new or changed devices on smithi084 2024-09-20T22:38:57.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:56 smithi084 ceph-mon[40024]: Adjusting osd_memory_target on smithi084 to 8032M 2024-09-20T22:38:57.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:56 smithi084 ceph-mon[40024]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-20T22:38:57.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:56 smithi084 ceph-mon[40024]: osdmap e14: 2 total, 2 up, 2 in 2024-09-20T22:38:57.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:56 smithi084 ceph-mon[29774]: Detected new or changed devices on smithi084 2024-09-20T22:38:57.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:56 smithi084 ceph-mon[29774]: Adjusting osd_memory_target on smithi084 to 8032M 2024-09-20T22:38:57.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:56 smithi084 ceph-mon[29774]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-20T22:38:57.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:56 smithi084 ceph-mon[29774]: osdmap e14: 2 total, 2 up, 2 in 2024-09-20T22:38:57.155 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:56 smithi154 ceph-mon[33264]: Detected new or changed devices on smithi084 2024-09-20T22:38:57.155 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:56 smithi154 ceph-mon[33264]: Adjusting osd_memory_target on smithi084 to 8032M 2024-09-20T22:38:57.155 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:56 smithi154 ceph-mon[33264]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-20T22:38:57.156 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:56 smithi154 ceph-mon[33264]: osdmap e14: 2 total, 2 up, 2 in 2024-09-20T22:38:58.068 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:57 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-20T22:38:58.069 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:57 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-20T22:38:58.069 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:57 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:38:58.069 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:57 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-20T22:38:58.069 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:57 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-20T22:38:58.069 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:57 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:38:58.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:57 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-20T22:38:58.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:57 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-20T22:38:58.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:57 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:38:58.975 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:58 smithi084 ceph-mon[29774]: from='client.14271 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi084:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:38:58.975 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:38:58 smithi084 ceph-mon[29774]: pgmap v54: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-20T22:38:58.975 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:58 smithi084 ceph-mon[40024]: from='client.14271 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi084:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:38:58.975 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:38:58 smithi084 ceph-mon[40024]: pgmap v54: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-20T22:38:59.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:58 smithi154 ceph-mon[33264]: from='client.14271 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi084:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:38:59.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:38:58 smithi154 ceph-mon[33264]: pgmap v54: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-20T22:39:01.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:00 smithi084 ceph-mon[29774]: pgmap v55: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-20T22:39:01.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:00 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/1751580365' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c728eb48-a0f0-4de4-a651-7f3bec867280"}]: dispatch 2024-09-20T22:39:01.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:00 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/1751580365' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c728eb48-a0f0-4de4-a651-7f3bec867280"}]': finished 2024-09-20T22:39:01.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:00 smithi084 ceph-mon[29774]: osdmap e15: 3 total, 2 up, 3 in 2024-09-20T22:39:01.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:00 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-20T22:39:01.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:00 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/1091033406' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-20T22:39:01.116 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:00 smithi084 ceph-mon[40024]: pgmap v55: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-20T22:39:01.116 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:00 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/1751580365' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c728eb48-a0f0-4de4-a651-7f3bec867280"}]: dispatch 2024-09-20T22:39:01.116 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:00 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/1751580365' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c728eb48-a0f0-4de4-a651-7f3bec867280"}]': finished 2024-09-20T22:39:01.116 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:00 smithi084 ceph-mon[40024]: osdmap e15: 3 total, 2 up, 3 in 2024-09-20T22:39:01.116 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:00 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-20T22:39:01.116 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:00 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/1091033406' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-20T22:39:01.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:00 smithi154 ceph-mon[33264]: pgmap v55: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-20T22:39:01.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:00 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/1751580365' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c728eb48-a0f0-4de4-a651-7f3bec867280"}]: dispatch 2024-09-20T22:39:01.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:00 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/1751580365' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c728eb48-a0f0-4de4-a651-7f3bec867280"}]': finished 2024-09-20T22:39:01.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:00 smithi154 ceph-mon[33264]: osdmap e15: 3 total, 2 up, 3 in 2024-09-20T22:39:01.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:00 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-20T22:39:01.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:00 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/1091033406' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-20T22:39:03.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:02 smithi084 ceph-mon[29774]: pgmap v57: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-20T22:39:03.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:02 smithi084 ceph-mon[40024]: pgmap v57: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-20T22:39:03.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:02 smithi154 ceph-mon[33264]: pgmap v57: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-20T22:39:05.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:04 smithi084 ceph-mon[29774]: pgmap v58: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-20T22:39:05.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:04 smithi084 ceph-mon[40024]: pgmap v58: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-20T22:39:05.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:04 smithi154 ceph-mon[33264]: pgmap v58: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-20T22:39:07.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:06 smithi084 ceph-mon[29774]: pgmap v59: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-20T22:39:07.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:06 smithi084 ceph-mon[40024]: pgmap v59: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-20T22:39:07.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:06 smithi154 ceph-mon[33264]: pgmap v59: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-20T22:39:08.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:07 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-09-20T22:39:08.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:07 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:39:08.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:07 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-09-20T22:39:08.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:07 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:39:08.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:07 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-09-20T22:39:08.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:07 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:39:09.071 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:08 smithi084 ceph-mon[29774]: Deploying daemon osd.2 on smithi084 2024-09-20T22:39:09.071 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:08 smithi084 ceph-mon[29774]: pgmap v60: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-20T22:39:09.071 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:08 smithi084 ceph-mon[40024]: Deploying daemon osd.2 on smithi084 2024-09-20T22:39:09.071 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:08 smithi084 ceph-mon[40024]: pgmap v60: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-20T22:39:09.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:08 smithi154 ceph-mon[33264]: Deploying daemon osd.2 on smithi084 2024-09-20T22:39:09.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:08 smithi154 ceph-mon[33264]: pgmap v60: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-20T22:39:10.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:10 smithi084 ceph-mon[29774]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-20T22:39:10.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:10 smithi084 ceph-mon[40024]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-20T22:39:11.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:10 smithi154 ceph-mon[33264]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-20T22:39:12.952 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:12 smithi084 ceph-mon[40024]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-20T22:39:12.952 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:12 smithi084 ceph-mon[29774]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-20T22:39:13.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:12 smithi154 ceph-mon[33264]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-20T22:39:13.968 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:13 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:39:13.968 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:13 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:39:13.968 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:13 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:39:13.968 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:13 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:39:13.968 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:13 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:39:13.969 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:13 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:39:14.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:13 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:39:14.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:13 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:39:14.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:13 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:39:14.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:14 smithi084 ceph-mon[29774]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-20T22:39:14.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:14 smithi084 ceph-mon[40024]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-20T22:39:15.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:14 smithi154 ceph-mon[33264]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-20T22:39:16.501 INFO:teuthology.orchestra.run.smithi084.stdout:Created osd(s) 2 on host 'smithi084' 2024-09-20T22:39:16.864 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:16 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:39:16.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:16 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:39:16.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:16 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:39:16.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:16 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:39:16.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:16 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:39:16.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:16 smithi084 ceph-mon[40024]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-20T22:39:16.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:16 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:39:16.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:16 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:39:16.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:16 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:39:16.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:16 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:39:16.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:16 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:39:16.866 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:16 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:39:16.866 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:16 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:39:16.866 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:16 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:39:16.866 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:16 smithi084 ceph-mon[29774]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-20T22:39:16.866 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:16 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:39:16.866 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:16 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:39:16.866 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:16 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:39:17.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:16 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:39:17.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:16 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:39:17.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:16 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:39:17.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:16 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:39:17.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:16 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:39:17.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:16 smithi154 ceph-mon[33264]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-20T22:39:17.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:16 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:39:17.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:16 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:39:17.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:16 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:39:17.159 DEBUG:teuthology.orchestra.run.smithi084:osd.2> sudo journalctl -f -n 0 -u ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490@osd.2.service 2024-09-20T22:39:17.161 INFO:tasks.cephadm:Deploying osd.3 on smithi084 with /dev/vg_nvme/lv_1... 2024-09-20T22:39:17.161 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- lvm zap /dev/vg_nvme/lv_1 2024-09-20T22:39:18.271 INFO:journalctl@ceph.osd.2.smithi084.stdout:Sep 20 22:39:17 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-2[55375]: 2024-09-20T22:39:17.941+0000 7fd496de9640 -1 osd.2 0 log_to_monitors true 2024-09-20T22:39:18.306 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:39:18.864 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:18 smithi084 ceph-mon[40024]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-20T22:39:18.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:18 smithi084 ceph-mon[40024]: from='osd.2 v2:172.21.15.84:6809/3977168611' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-20T22:39:18.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:18 smithi084 ceph-mon[40024]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-20T22:39:18.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:18 smithi084 ceph-mon[29774]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-20T22:39:18.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:18 smithi084 ceph-mon[29774]: from='osd.2 v2:172.21.15.84:6809/3977168611' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-20T22:39:18.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:18 smithi084 ceph-mon[29774]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-20T22:39:19.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:18 smithi154 ceph-mon[33264]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-20T22:39:19.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:18 smithi154 ceph-mon[33264]: from='osd.2 v2:172.21.15.84:6809/3977168611' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-20T22:39:19.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:18 smithi154 ceph-mon[33264]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-20T22:39:20.114 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:19 smithi084 ceph-mon[29774]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-20T22:39:20.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:19 smithi084 ceph-mon[29774]: osdmap e16: 3 total, 2 up, 3 in 2024-09-20T22:39:20.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:19 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-20T22:39:20.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:19 smithi084 ceph-mon[29774]: from='osd.2 v2:172.21.15.84:6809/3977168611' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi084", "root=default"]}]: dispatch 2024-09-20T22:39:20.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:19 smithi084 ceph-mon[29774]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi084", "root=default"]}]: dispatch 2024-09-20T22:39:20.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:19 smithi084 ceph-mon[40024]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-20T22:39:20.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:19 smithi084 ceph-mon[40024]: osdmap e16: 3 total, 2 up, 3 in 2024-09-20T22:39:20.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:19 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-20T22:39:20.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:19 smithi084 ceph-mon[40024]: from='osd.2 v2:172.21.15.84:6809/3977168611' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi084", "root=default"]}]: dispatch 2024-09-20T22:39:20.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:19 smithi084 ceph-mon[40024]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi084", "root=default"]}]: dispatch 2024-09-20T22:39:20.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:19 smithi154 ceph-mon[33264]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-20T22:39:20.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:19 smithi154 ceph-mon[33264]: osdmap e16: 3 total, 2 up, 3 in 2024-09-20T22:39:20.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:19 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-20T22:39:20.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:19 smithi154 ceph-mon[33264]: from='osd.2 v2:172.21.15.84:6809/3977168611' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi084", "root=default"]}]: dispatch 2024-09-20T22:39:20.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:19 smithi154 ceph-mon[33264]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi084", "root=default"]}]: dispatch 2024-09-20T22:39:20.615 INFO:journalctl@ceph.osd.2.smithi084.stdout:Sep 20 22:39:20 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-2[55375]: 2024-09-20T22:39:20.217+0000 7fd492d6c640 -1 osd.2 0 waiting for initial osdmap 2024-09-20T22:39:20.615 INFO:journalctl@ceph.osd.2.smithi084.stdout:Sep 20 22:39:20 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-2[55375]: 2024-09-20T22:39:20.220+0000 7fd48ab8c640 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-20T22:39:20.985 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:20 smithi084 ceph-mon[40024]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-20T22:39:20.985 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:20 smithi084 ceph-mon[40024]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi084", "root=default"]}]': finished 2024-09-20T22:39:20.986 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:20 smithi084 ceph-mon[40024]: osdmap e17: 3 total, 2 up, 3 in 2024-09-20T22:39:20.986 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:20 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-20T22:39:20.986 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:20 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-20T22:39:20.986 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:20 smithi084 ceph-mon[40024]: from='osd.2 ' entity='osd.2' 2024-09-20T22:39:20.986 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:20 smithi084 ceph-mon[29774]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-20T22:39:20.987 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:20 smithi084 ceph-mon[29774]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi084", "root=default"]}]': finished 2024-09-20T22:39:20.987 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:20 smithi084 ceph-mon[29774]: osdmap e17: 3 total, 2 up, 3 in 2024-09-20T22:39:20.987 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:20 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-20T22:39:20.987 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:20 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-20T22:39:20.987 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:20 smithi084 ceph-mon[29774]: from='osd.2 ' entity='osd.2' 2024-09-20T22:39:21.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:20 smithi154 ceph-mon[33264]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-20T22:39:21.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:20 smithi154 ceph-mon[33264]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi084", "root=default"]}]': finished 2024-09-20T22:39:21.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:20 smithi154 ceph-mon[33264]: osdmap e17: 3 total, 2 up, 3 in 2024-09-20T22:39:21.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:20 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-20T22:39:21.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:20 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-20T22:39:21.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:20 smithi154 ceph-mon[33264]: from='osd.2 ' entity='osd.2' 2024-09-20T22:39:22.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:21 smithi154 ceph-mon[33264]: purged_snaps scrub starts 2024-09-20T22:39:22.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:21 smithi154 ceph-mon[33264]: purged_snaps scrub ok 2024-09-20T22:39:22.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:21 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-20T22:39:22.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:21 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:39:22.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:21 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:39:22.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:21 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:39:22.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:21 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:39:22.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:21 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:39:22.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:21 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:39:22.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:21 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:39:22.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:21 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:39:22.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:21 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:39:22.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:21 smithi154 ceph-mon[33264]: osd.2 v2:172.21.15.84:6809/3977168611 boot 2024-09-20T22:39:22.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:21 smithi154 ceph-mon[33264]: osdmap e18: 3 total, 3 up, 3 in 2024-09-20T22:39:22.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:21 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-20T22:39:22.309 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:21 smithi084 ceph-mon[40024]: purged_snaps scrub starts 2024-09-20T22:39:22.310 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:21 smithi084 ceph-mon[40024]: purged_snaps scrub ok 2024-09-20T22:39:22.310 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:21 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-20T22:39:22.310 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:21 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:39:22.310 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:21 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:39:22.310 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:21 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:39:22.310 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:21 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:39:22.310 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:21 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:39:22.310 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:21 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:39:22.310 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:21 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:39:22.310 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:21 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:39:22.310 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:21 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:39:22.310 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:21 smithi084 ceph-mon[40024]: osd.2 v2:172.21.15.84:6809/3977168611 boot 2024-09-20T22:39:22.310 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:21 smithi084 ceph-mon[40024]: osdmap e18: 3 total, 3 up, 3 in 2024-09-20T22:39:22.310 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:21 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-20T22:39:22.311 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:21 smithi084 ceph-mon[29774]: purged_snaps scrub starts 2024-09-20T22:39:22.311 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:21 smithi084 ceph-mon[29774]: purged_snaps scrub ok 2024-09-20T22:39:22.311 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:21 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-20T22:39:22.311 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:21 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:39:22.311 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:21 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:39:22.311 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:21 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:39:22.311 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:21 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:39:22.311 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:21 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:39:22.311 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:21 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:39:22.311 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:21 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:39:22.311 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:21 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:39:22.311 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:21 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:39:22.311 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:21 smithi084 ceph-mon[29774]: osd.2 v2:172.21.15.84:6809/3977168611 boot 2024-09-20T22:39:22.312 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:21 smithi084 ceph-mon[29774]: osdmap e18: 3 total, 3 up, 3 in 2024-09-20T22:39:22.312 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:21 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-20T22:39:22.313 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-20T22:39:22.337 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph orch daemon add osd smithi084:vg_nvme/lv_1 2024-09-20T22:39:22.579 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:39:23.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:22 smithi154 ceph-mon[33264]: Detected new or changed devices on smithi084 2024-09-20T22:39:23.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:22 smithi154 ceph-mon[33264]: Adjusting osd_memory_target on smithi084 to 5354M 2024-09-20T22:39:23.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:22 smithi154 ceph-mon[33264]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-20T22:39:23.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:22 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-09-20T22:39:23.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:22 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-09-20T22:39:23.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:22 smithi154 ceph-mon[33264]: osdmap e19: 3 total, 3 up, 3 in 2024-09-20T22:39:23.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:22 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-09-20T22:39:23.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:22 smithi084 ceph-mon[29774]: Detected new or changed devices on smithi084 2024-09-20T22:39:23.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:22 smithi084 ceph-mon[29774]: Adjusting osd_memory_target on smithi084 to 5354M 2024-09-20T22:39:23.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:22 smithi084 ceph-mon[29774]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-20T22:39:23.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:22 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-09-20T22:39:23.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:22 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-09-20T22:39:23.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:22 smithi084 ceph-mon[29774]: osdmap e19: 3 total, 3 up, 3 in 2024-09-20T22:39:23.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:22 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-09-20T22:39:23.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:22 smithi084 ceph-mon[40024]: Detected new or changed devices on smithi084 2024-09-20T22:39:23.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:22 smithi084 ceph-mon[40024]: Adjusting osd_memory_target on smithi084 to 5354M 2024-09-20T22:39:23.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:22 smithi084 ceph-mon[40024]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-20T22:39:23.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:22 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-09-20T22:39:23.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:22 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-09-20T22:39:23.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:22 smithi084 ceph-mon[40024]: osdmap e19: 3 total, 3 up, 3 in 2024-09-20T22:39:23.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:22 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-09-20T22:39:23.802 INFO:journalctl@ceph.osd.0.smithi084.stdout:Sep 20 22:39:23 smithi084 sudo[59493]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-20T22:39:23.802 INFO:journalctl@ceph.osd.0.smithi084.stdout:Sep 20 22:39:23 smithi084 sudo[59493]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-20T22:39:23.802 INFO:journalctl@ceph.osd.0.smithi084.stdout:Sep 20 22:39:23 smithi084 sudo[59493]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-20T22:39:23.802 INFO:journalctl@ceph.osd.0.smithi084.stdout:Sep 20 22:39:23 smithi084 sudo[59493]: pam_unix(sudo:session): session closed for user root 2024-09-20T22:39:24.115 INFO:journalctl@ceph.osd.0.smithi084.stdout:Sep 20 22:39:23 smithi084 sudo[59501]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-20T22:39:24.115 INFO:journalctl@ceph.osd.0.smithi084.stdout:Sep 20 22:39:23 smithi084 sudo[59501]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-20T22:39:24.115 INFO:journalctl@ceph.osd.0.smithi084.stdout:Sep 20 22:39:23 smithi084 sudo[59501]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-20T22:39:24.115 INFO:journalctl@ceph.osd.0.smithi084.stdout:Sep 20 22:39:23 smithi084 sudo[59501]: pam_unix(sudo:session): session closed for user root 2024-09-20T22:39:24.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:23 smithi084 ceph-mon[40024]: from='client.24194 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi084:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:39:24.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:23 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-20T22:39:24.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:23 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-20T22:39:24.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:23 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:39:24.116 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:23 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-20T22:39:24.116 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:23 smithi084 ceph-mon[40024]: osdmap e20: 3 total, 3 up, 3 in 2024-09-20T22:39:24.116 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:23 smithi084 ceph-mon[40024]: pgmap v73: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-20T22:39:24.116 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:23 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:39:24.116 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:23 smithi084 ceph-mon[29774]: from='client.24194 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi084:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:39:24.116 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:23 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-20T22:39:24.116 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:23 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-20T22:39:24.116 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:23 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:39:24.116 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:23 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-20T22:39:24.116 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:23 smithi084 ceph-mon[29774]: osdmap e20: 3 total, 3 up, 3 in 2024-09-20T22:39:24.116 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:23 smithi084 ceph-mon[29774]: pgmap v73: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-20T22:39:24.116 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:23 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:39:24.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:23 smithi154 ceph-mon[33264]: from='client.24194 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi084:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:39:24.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:23 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-20T22:39:24.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:23 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-20T22:39:24.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:23 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:39:24.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:23 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-20T22:39:24.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:23 smithi154 ceph-mon[33264]: osdmap e20: 3 total, 3 up, 3 in 2024-09-20T22:39:24.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:23 smithi154 ceph-mon[33264]: pgmap v73: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-20T22:39:24.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:23 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:39:24.369 INFO:journalctl@ceph.osd.1.smithi084.stdout:Sep 20 22:39:24 smithi084 sudo[59517]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-20T22:39:24.369 INFO:journalctl@ceph.osd.1.smithi084.stdout:Sep 20 22:39:24 smithi084 sudo[59517]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-20T22:39:24.369 INFO:journalctl@ceph.osd.1.smithi084.stdout:Sep 20 22:39:24 smithi084 sudo[59517]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-20T22:39:24.370 INFO:journalctl@ceph.osd.1.smithi084.stdout:Sep 20 22:39:24 smithi084 sudo[59517]: pam_unix(sudo:session): session closed for user root 2024-09-20T22:39:24.625 INFO:journalctl@ceph.osd.1.smithi084.stdout:Sep 20 22:39:24 smithi084 sudo[59521]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-20T22:39:24.626 INFO:journalctl@ceph.osd.1.smithi084.stdout:Sep 20 22:39:24 smithi084 sudo[59521]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-20T22:39:24.626 INFO:journalctl@ceph.osd.1.smithi084.stdout:Sep 20 22:39:24 smithi084 sudo[59521]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-20T22:39:24.626 INFO:journalctl@ceph.osd.1.smithi084.stdout:Sep 20 22:39:24 smithi084 sudo[59521]: pam_unix(sudo:session): session closed for user root 2024-09-20T22:39:24.888 INFO:journalctl@ceph.osd.2.smithi084.stdout:Sep 20 22:39:24 smithi084 sudo[59527]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-20T22:39:24.888 INFO:journalctl@ceph.osd.2.smithi084.stdout:Sep 20 22:39:24 smithi084 sudo[59527]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-20T22:39:24.888 INFO:journalctl@ceph.osd.2.smithi084.stdout:Sep 20 22:39:24 smithi084 sudo[59527]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-20T22:39:24.888 INFO:journalctl@ceph.osd.2.smithi084.stdout:Sep 20 22:39:24 smithi084 sudo[59527]: pam_unix(sudo:session): session closed for user root 2024-09-20T22:39:25.142 INFO:journalctl@ceph.osd.2.smithi084.stdout:Sep 20 22:39:24 smithi084 sudo[59553]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-20T22:39:25.143 INFO:journalctl@ceph.osd.2.smithi084.stdout:Sep 20 22:39:24 smithi084 sudo[59553]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-20T22:39:25.143 INFO:journalctl@ceph.osd.2.smithi084.stdout:Sep 20 22:39:24 smithi084 sudo[59553]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-20T22:39:25.143 INFO:journalctl@ceph.osd.2.smithi084.stdout:Sep 20 22:39:24 smithi084 sudo[59553]: pam_unix(sudo:session): session closed for user root 2024-09-20T22:39:25.407 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:25 smithi084 sudo[59561]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-20T22:39:25.408 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:25 smithi084 sudo[59561]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-20T22:39:25.408 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:25 smithi084 sudo[59561]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-20T22:39:25.408 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:25 smithi084 sudo[59561]: pam_unix(sudo:session): session closed for user root 2024-09-20T22:39:25.669 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:25 smithi154 ceph-mon[33264]: mgrmap e15: y(active, since 2m), standbys: x 2024-09-20T22:39:25.670 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:25 smithi154 ceph-mon[33264]: osdmap e21: 3 total, 3 up, 3 in 2024-09-20T22:39:25.670 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:25 smithi154 ceph-mon[33264]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-20T22:39:25.670 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:25 smithi154 sudo[35732]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-20T22:39:25.670 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:25 smithi154 sudo[35732]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-20T22:39:25.670 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:25 smithi154 sudo[35732]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-20T22:39:25.670 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:25 smithi154 sudo[35732]: pam_unix(sudo:session): session closed for user root 2024-09-20T22:39:25.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:25 smithi084 ceph-mon[40024]: mgrmap e15: y(active, since 2m), standbys: x 2024-09-20T22:39:25.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:25 smithi084 ceph-mon[40024]: osdmap e21: 3 total, 3 up, 3 in 2024-09-20T22:39:25.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:25 smithi084 ceph-mon[40024]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-20T22:39:25.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:25 smithi084 sudo[59591]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-20T22:39:25.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:25 smithi084 sudo[59591]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-20T22:39:25.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:25 smithi084 sudo[59591]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-20T22:39:25.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:25 smithi084 sudo[59591]: pam_unix(sudo:session): session closed for user root 2024-09-20T22:39:25.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:25 smithi084 ceph-mon[29774]: mgrmap e15: y(active, since 2m), standbys: x 2024-09-20T22:39:25.866 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:25 smithi084 ceph-mon[29774]: osdmap e21: 3 total, 3 up, 3 in 2024-09-20T22:39:25.866 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:25 smithi084 ceph-mon[29774]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-20T22:39:26.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:25 smithi154 sudo[35735]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-20T22:39:26.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:25 smithi154 sudo[35735]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-20T22:39:26.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:25 smithi154 sudo[35735]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-20T22:39:26.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:25 smithi154 sudo[35735]: pam_unix(sudo:session): session closed for user root 2024-09-20T22:39:26.443 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:26 smithi084 sudo[59621]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-20T22:39:26.444 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:26 smithi084 sudo[59621]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-20T22:39:26.444 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:26 smithi084 sudo[59621]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-20T22:39:26.444 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:26 smithi084 sudo[59621]: pam_unix(sudo:session): session closed for user root 2024-09-20T22:39:26.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:26 smithi154 ceph-mon[33264]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-20T22:39:26.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:26 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-20T22:39:26.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:26 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T22:39:26.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:26 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T22:39:26.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:26 smithi154 ceph-mon[33264]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-20T22:39:26.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:26 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-20T22:39:26.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:26 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T22:39:26.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:26 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T22:39:26.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:26 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/869648386' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "573f44eb-0853-4630-a7d8-9caaeb256f9b"}]: dispatch 2024-09-20T22:39:26.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:26 smithi154 ceph-mon[33264]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "573f44eb-0853-4630-a7d8-9caaeb256f9b"}]: dispatch 2024-09-20T22:39:26.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:26 smithi154 ceph-mon[33264]: pgmap v75: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-20T22:39:26.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:26 smithi154 ceph-mon[33264]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-20T22:39:26.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:26 smithi154 ceph-mon[33264]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "573f44eb-0853-4630-a7d8-9caaeb256f9b"}]': finished 2024-09-20T22:39:26.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:26 smithi154 ceph-mon[33264]: osdmap e22: 4 total, 3 up, 4 in 2024-09-20T22:39:26.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:26 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T22:39:26.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:26 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:39:26.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:26 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-20T22:39:26.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:26 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T22:39:26.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:26 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T22:39:26.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:26 smithi084 ceph-mon[29774]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-20T22:39:26.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:26 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-20T22:39:26.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:26 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T22:39:26.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:26 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T22:39:26.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:26 smithi084 ceph-mon[29774]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-20T22:39:26.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:26 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-20T22:39:26.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:26 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T22:39:26.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:26 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T22:39:26.866 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:26 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/869648386' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "573f44eb-0853-4630-a7d8-9caaeb256f9b"}]: dispatch 2024-09-20T22:39:26.866 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:26 smithi084 ceph-mon[29774]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "573f44eb-0853-4630-a7d8-9caaeb256f9b"}]: dispatch 2024-09-20T22:39:26.866 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:26 smithi084 ceph-mon[29774]: pgmap v75: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-20T22:39:26.866 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:26 smithi084 ceph-mon[29774]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-20T22:39:26.866 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:26 smithi084 ceph-mon[29774]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "573f44eb-0853-4630-a7d8-9caaeb256f9b"}]': finished 2024-09-20T22:39:26.866 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:26 smithi084 ceph-mon[29774]: osdmap e22: 4 total, 3 up, 4 in 2024-09-20T22:39:26.866 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:26 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T22:39:26.866 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:26 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:39:26.866 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:26 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-20T22:39:26.866 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:26 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T22:39:26.866 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:26 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T22:39:26.866 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:26 smithi084 sudo[59643]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-20T22:39:26.867 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:26 smithi084 sudo[59643]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-20T22:39:26.867 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:26 smithi084 sudo[59643]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-20T22:39:26.867 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:26 smithi084 sudo[59643]: pam_unix(sudo:session): session closed for user root 2024-09-20T22:39:26.867 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:26 smithi084 ceph-mon[40024]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-20T22:39:26.867 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:26 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-20T22:39:26.867 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:26 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T22:39:26.867 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:26 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T22:39:26.867 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:26 smithi084 ceph-mon[40024]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-20T22:39:26.867 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:26 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-20T22:39:26.867 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:26 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T22:39:26.867 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:26 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T22:39:26.867 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:26 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/869648386' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "573f44eb-0853-4630-a7d8-9caaeb256f9b"}]: dispatch 2024-09-20T22:39:26.867 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:26 smithi084 ceph-mon[40024]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "573f44eb-0853-4630-a7d8-9caaeb256f9b"}]: dispatch 2024-09-20T22:39:26.867 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:26 smithi084 ceph-mon[40024]: pgmap v75: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-20T22:39:26.868 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:26 smithi084 ceph-mon[40024]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-20T22:39:26.868 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:26 smithi084 ceph-mon[40024]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "573f44eb-0853-4630-a7d8-9caaeb256f9b"}]': finished 2024-09-20T22:39:26.868 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:26 smithi084 ceph-mon[40024]: osdmap e22: 4 total, 3 up, 4 in 2024-09-20T22:39:26.868 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:26 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T22:39:26.868 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:26 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:39:26.868 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:26 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-20T22:39:26.868 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:26 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T22:39:26.868 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:26 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T22:39:27.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:27 smithi154 ceph-mon[33264]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-20T22:39:27.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:27 smithi154 ceph-mon[33264]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-20T22:39:27.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:27 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/3707481097' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-20T22:39:27.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:27 smithi084 ceph-mon[29774]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-20T22:39:27.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:27 smithi084 ceph-mon[29774]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-20T22:39:27.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:27 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/3707481097' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-20T22:39:27.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:27 smithi084 ceph-mon[40024]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-20T22:39:27.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:27 smithi084 ceph-mon[40024]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-20T22:39:27.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:27 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/3707481097' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-20T22:39:28.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:28 smithi154 ceph-mon[33264]: pgmap v77: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail 2024-09-20T22:39:28.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:28 smithi084 ceph-mon[40024]: pgmap v77: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail 2024-09-20T22:39:28.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:28 smithi084 ceph-mon[29774]: pgmap v77: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail 2024-09-20T22:39:31.114 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:30 smithi084 ceph-mon[29774]: pgmap v78: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail 2024-09-20T22:39:31.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:30 smithi084 ceph-mon[40024]: pgmap v78: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail 2024-09-20T22:39:31.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:30 smithi154 ceph-mon[33264]: pgmap v78: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail 2024-09-20T22:39:33.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:32 smithi084 ceph-mon[29774]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 3.1 KiB/s rd, 135 KiB/s wr, 9 op/s 2024-09-20T22:39:33.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:32 smithi084 ceph-mon[40024]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 3.1 KiB/s rd, 135 KiB/s wr, 9 op/s 2024-09-20T22:39:33.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:32 smithi154 ceph-mon[33264]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 3.1 KiB/s rd, 135 KiB/s wr, 9 op/s 2024-09-20T22:39:33.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:33 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-09-20T22:39:33.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:33 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:39:33.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:33 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-09-20T22:39:33.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:33 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:39:34.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:33 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-09-20T22:39:34.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:33 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:39:34.858 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:34 smithi084 ceph-mon[40024]: Deploying daemon osd.3 on smithi084 2024-09-20T22:39:34.858 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:34 smithi084 ceph-mon[40024]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 2.3 KiB/s rd, 100 KiB/s wr, 7 op/s 2024-09-20T22:39:34.859 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:34 smithi084 ceph-mon[29774]: Deploying daemon osd.3 on smithi084 2024-09-20T22:39:34.859 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:34 smithi084 ceph-mon[29774]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 2.3 KiB/s rd, 100 KiB/s wr, 7 op/s 2024-09-20T22:39:35.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:34 smithi154 ceph-mon[33264]: Deploying daemon osd.3 on smithi084 2024-09-20T22:39:35.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:34 smithi154 ceph-mon[33264]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 2.3 KiB/s rd, 100 KiB/s wr, 7 op/s 2024-09-20T22:39:36.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:36 smithi084 ceph-mon[29774]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 80 KiB/s wr, 5 op/s 2024-09-20T22:39:36.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:36 smithi084 ceph-mon[40024]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 80 KiB/s wr, 5 op/s 2024-09-20T22:39:37.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:36 smithi154 ceph-mon[33264]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 80 KiB/s wr, 5 op/s 2024-09-20T22:39:39.046 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:38 smithi084 ceph-mon[40024]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 66 KiB/s wr, 4 op/s 2024-09-20T22:39:39.046 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:38 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:39:39.046 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:38 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:39:39.046 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:38 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:39:39.046 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:38 smithi084 ceph-mon[29774]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 66 KiB/s wr, 4 op/s 2024-09-20T22:39:39.047 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:38 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:39:39.047 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:38 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:39:39.047 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:38 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:39:39.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:38 smithi154 ceph-mon[33264]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 66 KiB/s wr, 4 op/s 2024-09-20T22:39:39.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:38 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:39:39.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:38 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:39:39.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:38 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:39:41.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:40 smithi084 ceph-mon[29774]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-09-20T22:39:41.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:40 smithi084 ceph-mon[40024]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-09-20T22:39:41.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:40 smithi154 ceph-mon[33264]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-09-20T22:39:41.561 INFO:teuthology.orchestra.run.smithi084.stdout:Created osd(s) 3 on host 'smithi084' 2024-09-20T22:39:42.165 DEBUG:teuthology.orchestra.run.smithi084:osd.3> sudo journalctl -f -n 0 -u ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490@osd.3.service 2024-09-20T22:39:42.168 INFO:tasks.cephadm:Deploying osd.4 on smithi154 with /dev/vg_nvme/lv_4... 2024-09-20T22:39:42.168 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- lvm zap /dev/vg_nvme/lv_4 2024-09-20T22:39:42.422 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:39:42.799 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:42 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:39:42.799 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:42 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:39:42.799 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:42 smithi154 ceph-mon[33264]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-09-20T22:39:42.799 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:42 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:39:42.799 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:42 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:39:42.864 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:42 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:39:42.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:42 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:39:42.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:42 smithi084 ceph-mon[40024]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-09-20T22:39:42.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:42 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:39:42.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:42 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:39:42.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:42 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:39:42.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:42 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:39:42.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:42 smithi084 ceph-mon[29774]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-09-20T22:39:42.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:42 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:39:42.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:42 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:39:43.522 INFO:journalctl@ceph.osd.3.smithi084.stdout:Sep 20 22:39:43 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-3[61309]: 2024-09-20T22:39:43.330+0000 7f468f149640 -1 osd.3 0 log_to_monitors true 2024-09-20T22:39:44.364 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:44 smithi084 ceph-mon[29774]: from='osd.3 v2:172.21.15.84:6813/2192239945' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-20T22:39:44.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:44 smithi084 ceph-mon[29774]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-20T22:39:44.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:44 smithi084 ceph-mon[29774]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-20T22:39:44.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:44 smithi084 ceph-mon[40024]: from='osd.3 v2:172.21.15.84:6813/2192239945' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-20T22:39:44.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:44 smithi084 ceph-mon[40024]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-20T22:39:44.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:44 smithi084 ceph-mon[40024]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-20T22:39:44.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:44 smithi154 ceph-mon[33264]: from='osd.3 v2:172.21.15.84:6813/2192239945' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-20T22:39:44.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:44 smithi154 ceph-mon[33264]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-20T22:39:44.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:44 smithi154 ceph-mon[33264]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-20T22:39:44.871 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-20T22:39:44.896 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph orch daemon add osd smithi154:vg_nvme/lv_4 2024-09-20T22:39:45.145 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:39:45.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:45 smithi154 ceph-mon[33264]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-20T22:39:45.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:45 smithi154 ceph-mon[33264]: osdmap e23: 4 total, 3 up, 4 in 2024-09-20T22:39:45.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:45 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T22:39:45.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:45 smithi154 ceph-mon[33264]: from='osd.3 v2:172.21.15.84:6813/2192239945' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi084", "root=default"]}]: dispatch 2024-09-20T22:39:45.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:45 smithi154 ceph-mon[33264]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi084", "root=default"]}]: dispatch 2024-09-20T22:39:45.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:45 smithi084 ceph-mon[40024]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-20T22:39:45.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:45 smithi084 ceph-mon[40024]: osdmap e23: 4 total, 3 up, 4 in 2024-09-20T22:39:45.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:45 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T22:39:45.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:45 smithi084 ceph-mon[40024]: from='osd.3 v2:172.21.15.84:6813/2192239945' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi084", "root=default"]}]: dispatch 2024-09-20T22:39:45.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:45 smithi084 ceph-mon[40024]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi084", "root=default"]}]: dispatch 2024-09-20T22:39:45.365 INFO:journalctl@ceph.osd.3.smithi084.stdout:Sep 20 22:39:45 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-3[61309]: 2024-09-20T22:39:45.073+0000 7f468b0cc640 -1 osd.3 0 waiting for initial osdmap 2024-09-20T22:39:45.365 INFO:journalctl@ceph.osd.3.smithi084.stdout:Sep 20 22:39:45 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-3[61309]: 2024-09-20T22:39:45.076+0000 7f4686ef4640 -1 osd.3 24 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-20T22:39:45.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:45 smithi084 ceph-mon[29774]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-20T22:39:45.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:45 smithi084 ceph-mon[29774]: osdmap e23: 4 total, 3 up, 4 in 2024-09-20T22:39:45.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:45 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T22:39:45.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:45 smithi084 ceph-mon[29774]: from='osd.3 v2:172.21.15.84:6813/2192239945' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi084", "root=default"]}]: dispatch 2024-09-20T22:39:45.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:45 smithi084 ceph-mon[29774]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi084", "root=default"]}]: dispatch 2024-09-20T22:39:46.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:46 smithi084 ceph-mon[29774]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi084", "root=default"]}]': finished 2024-09-20T22:39:46.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:46 smithi084 ceph-mon[29774]: osdmap e24: 4 total, 3 up, 4 in 2024-09-20T22:39:46.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:46 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T22:39:46.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:46 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T22:39:46.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:46 smithi084 ceph-mon[29774]: Detected new or changed devices on smithi084 2024-09-20T22:39:46.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:46 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:39:46.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:46 smithi084 ceph-mon[29774]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-20T22:39:46.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:46 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:39:46.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:46 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:39:46.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:46 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:39:46.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:46 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:39:46.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:46 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:39:46.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:46 smithi084 ceph-mon[29774]: Adjusting osd_memory_target on smithi084 to 4016M 2024-09-20T22:39:46.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:46 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:39:46.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:46 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:39:46.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:46 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:39:46.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:46 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:39:46.367 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:46 smithi084 ceph-mon[29774]: from='client.24196 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi154:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:39:46.367 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:46 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-20T22:39:46.367 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:46 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-20T22:39:46.367 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:46 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:39:46.367 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:46 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:39:46.367 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:46 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:39:46.367 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:46 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:39:46.367 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:46 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:39:46.367 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:46 smithi084 ceph-mon[40024]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi084", "root=default"]}]': finished 2024-09-20T22:39:46.367 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:46 smithi084 ceph-mon[40024]: osdmap e24: 4 total, 3 up, 4 in 2024-09-20T22:39:46.367 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:46 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T22:39:46.367 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:46 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T22:39:46.368 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:46 smithi084 ceph-mon[40024]: Detected new or changed devices on smithi084 2024-09-20T22:39:46.368 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:46 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:39:46.368 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:46 smithi084 ceph-mon[40024]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-20T22:39:46.368 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:46 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:39:46.368 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:46 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:39:46.368 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:46 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:39:46.368 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:46 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:39:46.368 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:46 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:39:46.368 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:46 smithi084 ceph-mon[40024]: Adjusting osd_memory_target on smithi084 to 4016M 2024-09-20T22:39:46.368 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:46 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:39:46.368 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:46 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:39:46.368 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:46 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:39:46.368 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:46 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:39:46.368 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:46 smithi084 ceph-mon[40024]: from='client.24196 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi154:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:39:46.368 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:46 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-20T22:39:46.368 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:46 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-20T22:39:46.369 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:46 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:39:46.369 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:46 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:39:46.369 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:46 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:39:46.369 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:46 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:39:46.369 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:46 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:39:46.391 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:46 smithi154 ceph-mon[33264]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi084", "root=default"]}]': finished 2024-09-20T22:39:46.391 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:46 smithi154 ceph-mon[33264]: osdmap e24: 4 total, 3 up, 4 in 2024-09-20T22:39:46.391 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:46 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T22:39:46.391 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:46 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T22:39:46.391 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:46 smithi154 ceph-mon[33264]: Detected new or changed devices on smithi084 2024-09-20T22:39:46.391 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:46 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:39:46.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:46 smithi154 ceph-mon[33264]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-20T22:39:46.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:46 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:39:46.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:46 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:39:46.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:46 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:39:46.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:46 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:39:46.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:46 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:39:46.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:46 smithi154 ceph-mon[33264]: Adjusting osd_memory_target on smithi084 to 4016M 2024-09-20T22:39:46.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:46 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:39:46.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:46 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:39:46.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:46 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:39:46.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:46 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:39:46.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:46 smithi154 ceph-mon[33264]: from='client.24196 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi154:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:39:46.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:46 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-20T22:39:46.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:46 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-20T22:39:46.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:46 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:39:46.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:46 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:39:46.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:46 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:39:46.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:46 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:39:46.393 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:46 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:39:47.347 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:47 smithi154 ceph-mon[33264]: purged_snaps scrub starts 2024-09-20T22:39:47.347 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:47 smithi154 ceph-mon[33264]: purged_snaps scrub ok 2024-09-20T22:39:47.347 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:47 smithi154 ceph-mon[33264]: osd.3 v2:172.21.15.84:6813/2192239945 boot 2024-09-20T22:39:47.347 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:47 smithi154 ceph-mon[33264]: osdmap e25: 4 total, 4 up, 4 in 2024-09-20T22:39:47.347 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:47 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T22:39:47.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:47 smithi084 ceph-mon[29774]: purged_snaps scrub starts 2024-09-20T22:39:47.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:47 smithi084 ceph-mon[29774]: purged_snaps scrub ok 2024-09-20T22:39:47.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:47 smithi084 ceph-mon[29774]: osd.3 v2:172.21.15.84:6813/2192239945 boot 2024-09-20T22:39:47.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:47 smithi084 ceph-mon[29774]: osdmap e25: 4 total, 4 up, 4 in 2024-09-20T22:39:47.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:47 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T22:39:47.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:47 smithi084 ceph-mon[40024]: purged_snaps scrub starts 2024-09-20T22:39:47.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:47 smithi084 ceph-mon[40024]: purged_snaps scrub ok 2024-09-20T22:39:47.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:47 smithi084 ceph-mon[40024]: osd.3 v2:172.21.15.84:6813/2192239945 boot 2024-09-20T22:39:47.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:47 smithi084 ceph-mon[40024]: osdmap e25: 4 total, 4 up, 4 in 2024-09-20T22:39:47.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:47 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T22:39:48.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:48 smithi084 ceph-mon[29774]: osdmap e26: 4 total, 4 up, 4 in 2024-09-20T22:39:48.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:48 smithi084 ceph-mon[29774]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-20T22:39:48.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:48 smithi084 ceph-mon[29774]: from='client.? 172.21.15.154:0/335106079' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a52c3b94-c5f3-490f-8ac8-09061b9b2abb"}]: dispatch 2024-09-20T22:39:48.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:48 smithi084 ceph-mon[29774]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a52c3b94-c5f3-490f-8ac8-09061b9b2abb"}]: dispatch 2024-09-20T22:39:48.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:48 smithi084 ceph-mon[29774]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a52c3b94-c5f3-490f-8ac8-09061b9b2abb"}]': finished 2024-09-20T22:39:48.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:48 smithi084 ceph-mon[29774]: osdmap e27: 5 total, 4 up, 5 in 2024-09-20T22:39:48.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:48 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-20T22:39:48.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:48 smithi084 ceph-mon[40024]: osdmap e26: 4 total, 4 up, 4 in 2024-09-20T22:39:48.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:48 smithi084 ceph-mon[40024]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-20T22:39:48.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:48 smithi084 ceph-mon[40024]: from='client.? 172.21.15.154:0/335106079' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a52c3b94-c5f3-490f-8ac8-09061b9b2abb"}]: dispatch 2024-09-20T22:39:48.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:48 smithi084 ceph-mon[40024]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a52c3b94-c5f3-490f-8ac8-09061b9b2abb"}]: dispatch 2024-09-20T22:39:48.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:48 smithi084 ceph-mon[40024]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a52c3b94-c5f3-490f-8ac8-09061b9b2abb"}]': finished 2024-09-20T22:39:48.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:48 smithi084 ceph-mon[40024]: osdmap e27: 5 total, 4 up, 5 in 2024-09-20T22:39:48.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:48 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-20T22:39:48.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:48 smithi154 ceph-mon[33264]: osdmap e26: 4 total, 4 up, 4 in 2024-09-20T22:39:48.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:48 smithi154 ceph-mon[33264]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-20T22:39:48.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:48 smithi154 ceph-mon[33264]: from='client.? 172.21.15.154:0/335106079' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a52c3b94-c5f3-490f-8ac8-09061b9b2abb"}]: dispatch 2024-09-20T22:39:48.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:48 smithi154 ceph-mon[33264]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a52c3b94-c5f3-490f-8ac8-09061b9b2abb"}]: dispatch 2024-09-20T22:39:48.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:48 smithi154 ceph-mon[33264]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a52c3b94-c5f3-490f-8ac8-09061b9b2abb"}]': finished 2024-09-20T22:39:48.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:48 smithi154 ceph-mon[33264]: osdmap e27: 5 total, 4 up, 5 in 2024-09-20T22:39:48.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:48 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-20T22:39:49.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:49 smithi084 ceph-mon[29774]: from='client.? 172.21.15.154:0/2899730626' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-20T22:39:49.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:49 smithi084 ceph-mon[40024]: from='client.? 172.21.15.154:0/2899730626' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-20T22:39:49.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:49 smithi154 ceph-mon[33264]: from='client.? 172.21.15.154:0/2899730626' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-20T22:39:50.364 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:50 smithi084 ceph-mon[29774]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-20T22:39:50.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:50 smithi084 ceph-mon[40024]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-20T22:39:50.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:50 smithi154 ceph-mon[33264]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-20T22:39:53.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:52 smithi154 ceph-mon[33264]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-20T22:39:53.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:52 smithi084 ceph-mon[29774]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-20T22:39:53.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:52 smithi084 ceph-mon[40024]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-20T22:39:54.046 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:53 smithi154 ceph-mon[33264]: purged_snaps scrub starts 2024-09-20T22:39:54.046 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:53 smithi154 ceph-mon[33264]: purged_snaps scrub ok 2024-09-20T22:39:54.046 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:53 smithi154 ceph-mon[33264]: osdmap e28: 5 total, 4 up, 5 in 2024-09-20T22:39:54.046 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:53 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-20T22:39:54.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:53 smithi084 ceph-mon[29774]: purged_snaps scrub starts 2024-09-20T22:39:54.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:53 smithi084 ceph-mon[29774]: purged_snaps scrub ok 2024-09-20T22:39:54.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:53 smithi084 ceph-mon[29774]: osdmap e28: 5 total, 4 up, 5 in 2024-09-20T22:39:54.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:53 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-20T22:39:54.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:53 smithi084 ceph-mon[40024]: purged_snaps scrub starts 2024-09-20T22:39:54.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:53 smithi084 ceph-mon[40024]: purged_snaps scrub ok 2024-09-20T22:39:54.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:53 smithi084 ceph-mon[40024]: osdmap e28: 5 total, 4 up, 5 in 2024-09-20T22:39:54.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:53 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-20T22:39:55.035 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:54 smithi154 ceph-mon[33264]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-20T22:39:55.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:54 smithi084 ceph-mon[29774]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-20T22:39:55.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:54 smithi084 ceph-mon[40024]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-20T22:39:56.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:55 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-09-20T22:39:56.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:55 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:39:56.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:55 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-09-20T22:39:56.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:55 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:39:56.146 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:55 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-09-20T22:39:56.146 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:55 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:39:57.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:56 smithi154 ceph-mon[33264]: Deploying daemon osd.4 on smithi154 2024-09-20T22:39:57.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:56 smithi154 ceph-mon[33264]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-20T22:39:57.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:56 smithi084 ceph-mon[29774]: Deploying daemon osd.4 on smithi154 2024-09-20T22:39:57.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:56 smithi084 ceph-mon[29774]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-20T22:39:57.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:56 smithi084 ceph-mon[40024]: Deploying daemon osd.4 on smithi154 2024-09-20T22:39:57.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:56 smithi084 ceph-mon[40024]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-20T22:39:59.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:39:58 smithi084 ceph-mon[29774]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-20T22:39:59.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:39:58 smithi084 ceph-mon[40024]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-20T22:39:59.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:39:58 smithi154 ceph-mon[33264]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-20T22:40:01.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:40:00 smithi154 ceph-mon[33264]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-20T22:40:01.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:40:00 smithi154 ceph-mon[33264]: overall HEALTH_OK 2024-09-20T22:40:01.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:40:00 smithi084 ceph-mon[29774]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-20T22:40:01.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:40:00 smithi084 ceph-mon[29774]: overall HEALTH_OK 2024-09-20T22:40:01.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:40:00 smithi084 ceph-mon[40024]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-20T22:40:01.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:40:00 smithi084 ceph-mon[40024]: overall HEALTH_OK 2024-09-20T22:40:01.958 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:40:01 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:40:01.958 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:40:01 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:40:01.958 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:40:01 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:40:02.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:40:01 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:40:02.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:40:01 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:40:02.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:40:01 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:40:02.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:40:01 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:40:02.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:40:01 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:40:02.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:40:01 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:40:03.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:40:02 smithi154 ceph-mon[33264]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-20T22:40:03.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:40:02 smithi084 ceph-mon[29774]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-20T22:40:03.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:40:02 smithi084 ceph-mon[40024]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-20T22:40:05.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:40:04 smithi084 ceph-mon[29774]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-20T22:40:05.156 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:40:04 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:40:05.156 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:40:04 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:40:05.156 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:40:04 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:40:05.156 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:40:04 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:40:05.156 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:40:04 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:40:05.156 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:40:04 smithi084 ceph-mon[40024]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-20T22:40:05.156 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:40:04 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:40:05.156 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:40:04 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:40:05.157 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:40:04 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:40:05.157 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:40:04 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:40:05.157 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:40:04 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:40:05.248 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:40:04 smithi154 ceph-mon[33264]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-20T22:40:05.248 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:40:04 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:40:05.248 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:40:04 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:40:05.249 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:40:04 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:40:05.249 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:40:04 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:40:05.249 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:40:04 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:40:05.523 INFO:teuthology.orchestra.run.smithi154.stdout:Created osd(s) 4 on host 'smithi154' 2024-09-20T22:40:06.073 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:40:05 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:40:06.073 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:40:05 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:40:06.074 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:40:05 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:40:06.114 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:40:05 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:40:06.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:40:05 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:40:06.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:40:05 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:40:06.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:40:05 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:40:06.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:40:05 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:40:06.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:40:05 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:40:06.204 DEBUG:teuthology.orchestra.run.smithi154:osd.4> sudo journalctl -f -n 0 -u ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490@osd.4.service 2024-09-20T22:40:06.206 INFO:tasks.cephadm:Deploying osd.5 on smithi154 with /dev/vg_nvme/lv_3... 2024-09-20T22:40:06.206 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- lvm zap /dev/vg_nvme/lv_3 2024-09-20T22:40:06.956 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:40:07.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:40:06 smithi154 ceph-mon[33264]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-20T22:40:07.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:40:06 smithi154 ceph-mon[33264]: from='osd.4 v2:172.21.15.154:6800/1793136002' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-20T22:40:07.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:40:06 smithi154 ceph-mon[33264]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-20T22:40:07.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:40:06 smithi084 ceph-mon[29774]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-20T22:40:07.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:40:06 smithi084 ceph-mon[29774]: from='osd.4 v2:172.21.15.154:6800/1793136002' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-20T22:40:07.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:40:06 smithi084 ceph-mon[29774]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-20T22:40:07.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:40:06 smithi084 ceph-mon[40024]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-20T22:40:07.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:40:06 smithi084 ceph-mon[40024]: from='osd.4 v2:172.21.15.154:6800/1793136002' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-20T22:40:07.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:40:06 smithi084 ceph-mon[40024]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-20T22:40:08.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:40:07 smithi084 ceph-mon[40024]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-20T22:40:08.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:40:07 smithi084 ceph-mon[40024]: osdmap e29: 5 total, 4 up, 5 in 2024-09-20T22:40:08.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:40:07 smithi084 ceph-mon[40024]: from='osd.4 v2:172.21.15.154:6800/1793136002' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2024-09-20T22:40:08.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:40:07 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-20T22:40:08.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:40:07 smithi084 ceph-mon[40024]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2024-09-20T22:40:08.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:40:07 smithi084 ceph-mon[40024]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]': finished 2024-09-20T22:40:08.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:40:07 smithi084 ceph-mon[29774]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-20T22:40:08.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:40:07 smithi084 ceph-mon[29774]: osdmap e29: 5 total, 4 up, 5 in 2024-09-20T22:40:08.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:40:07 smithi084 ceph-mon[29774]: from='osd.4 v2:172.21.15.154:6800/1793136002' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2024-09-20T22:40:08.116 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:40:07 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-20T22:40:08.116 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:40:07 smithi084 ceph-mon[29774]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2024-09-20T22:40:08.116 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:40:07 smithi084 ceph-mon[29774]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]': finished 2024-09-20T22:40:08.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:40:07 smithi154 ceph-mon[33264]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-20T22:40:08.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:40:07 smithi154 ceph-mon[33264]: osdmap e29: 5 total, 4 up, 5 in 2024-09-20T22:40:08.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:40:07 smithi154 ceph-mon[33264]: from='osd.4 v2:172.21.15.154:6800/1793136002' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2024-09-20T22:40:08.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:40:07 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-20T22:40:08.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:40:07 smithi154 ceph-mon[33264]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2024-09-20T22:40:08.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:40:07 smithi154 ceph-mon[33264]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]': finished 2024-09-20T22:40:08.255 INFO:journalctl@ceph.osd.4.smithi154.stdout:Sep 20 22:40:07 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-4[38043]: 2024-09-20T22:40:07.815+0000 7f44dea78640 -1 osd.4 0 waiting for initial osdmap 2024-09-20T22:40:08.255 INFO:journalctl@ceph.osd.4.smithi154.stdout:Sep 20 22:40:07 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-4[38043]: 2024-09-20T22:40:07.820+0000 7f44da8a0640 -1 osd.4 30 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-20T22:40:09.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:40:08 smithi154 ceph-mon[33264]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-20T22:40:09.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:40:08 smithi154 ceph-mon[33264]: osdmap e30: 5 total, 4 up, 5 in 2024-09-20T22:40:09.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:40:08 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-20T22:40:09.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:40:08 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-20T22:40:09.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:40:08 smithi154 ceph-mon[33264]: osd.4 v2:172.21.15.154:6800/1793136002 boot 2024-09-20T22:40:09.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:40:08 smithi154 ceph-mon[33264]: osdmap e31: 5 total, 5 up, 5 in 2024-09-20T22:40:09.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:40:08 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-20T22:40:09.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:40:08 smithi084 ceph-mon[29774]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-20T22:40:09.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:40:08 smithi084 ceph-mon[29774]: osdmap e30: 5 total, 4 up, 5 in 2024-09-20T22:40:09.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:40:08 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-20T22:40:09.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:40:08 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-20T22:40:09.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:40:08 smithi084 ceph-mon[29774]: osd.4 v2:172.21.15.154:6800/1793136002 boot 2024-09-20T22:40:09.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:40:08 smithi084 ceph-mon[29774]: osdmap e31: 5 total, 5 up, 5 in 2024-09-20T22:40:09.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:40:08 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-20T22:40:09.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:40:08 smithi084 ceph-mon[40024]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-20T22:40:09.116 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:40:08 smithi084 ceph-mon[40024]: osdmap e30: 5 total, 4 up, 5 in 2024-09-20T22:40:09.116 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:40:08 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-20T22:40:09.116 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:40:08 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-20T22:40:09.116 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:40:08 smithi084 ceph-mon[40024]: osd.4 v2:172.21.15.154:6800/1793136002 boot 2024-09-20T22:40:09.116 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:40:08 smithi084 ceph-mon[40024]: osdmap e31: 5 total, 5 up, 5 in 2024-09-20T22:40:09.116 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:40:08 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-20T22:40:10.191 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:40:10 smithi154 ceph-mon[33264]: purged_snaps scrub starts 2024-09-20T22:40:10.192 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:40:10 smithi154 ceph-mon[33264]: purged_snaps scrub ok 2024-09-20T22:40:10.192 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:40:10 smithi154 ceph-mon[33264]: purged_snaps scrub starts 2024-09-20T22:40:10.192 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:40:10 smithi154 ceph-mon[33264]: purged_snaps scrub ok 2024-09-20T22:40:10.192 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:40:10 smithi154 ceph-mon[33264]: osdmap e32: 5 total, 5 up, 5 in 2024-09-20T22:40:10.192 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:40:10 smithi154 ceph-mon[33264]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-20T22:40:10.192 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:40:10 smithi154 ceph-mon[33264]: Detected new or changed devices on smithi154 2024-09-20T22:40:10.192 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:40:10 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:40:10.193 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:40:10 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:40:10.193 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:40:10 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:40:10.194 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:40:10 smithi154 ceph-mon[33264]: Adjusting osd_memory_target on smithi154 to 17088M 2024-09-20T22:40:10.194 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:40:10 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:40:10.194 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:40:10 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:40:10.194 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:40:10 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:40:10.194 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:40:10 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:40:10.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:40:10 smithi084 ceph-mon[40024]: purged_snaps scrub starts 2024-09-20T22:40:10.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:40:10 smithi084 ceph-mon[40024]: purged_snaps scrub ok 2024-09-20T22:40:10.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:40:10 smithi084 ceph-mon[40024]: purged_snaps scrub starts 2024-09-20T22:40:10.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:40:10 smithi084 ceph-mon[40024]: purged_snaps scrub ok 2024-09-20T22:40:10.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:40:10 smithi084 ceph-mon[40024]: osdmap e32: 5 total, 5 up, 5 in 2024-09-20T22:40:10.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:40:10 smithi084 ceph-mon[40024]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-20T22:40:10.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:40:10 smithi084 ceph-mon[40024]: Detected new or changed devices on smithi154 2024-09-20T22:40:10.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:40:10 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:40:10.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:40:10 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:40:10.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:40:10 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:40:10.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:40:10 smithi084 ceph-mon[40024]: Adjusting osd_memory_target on smithi154 to 17088M 2024-09-20T22:40:10.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:40:10 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:40:10.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:40:10 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:40:10.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:40:10 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:40:10.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:40:10 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:40:10.367 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:40:10 smithi084 ceph-mon[29774]: purged_snaps scrub starts 2024-09-20T22:40:10.367 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:40:10 smithi084 ceph-mon[29774]: purged_snaps scrub ok 2024-09-20T22:40:10.367 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:40:10 smithi084 ceph-mon[29774]: purged_snaps scrub starts 2024-09-20T22:40:10.367 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:40:10 smithi084 ceph-mon[29774]: purged_snaps scrub ok 2024-09-20T22:40:10.367 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:40:10 smithi084 ceph-mon[29774]: osdmap e32: 5 total, 5 up, 5 in 2024-09-20T22:40:10.367 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:40:10 smithi084 ceph-mon[29774]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-20T22:40:10.367 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:40:10 smithi084 ceph-mon[29774]: Detected new or changed devices on smithi154 2024-09-20T22:40:10.367 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:40:10 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:40:10.367 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:40:10 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:40:10.367 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:40:10 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:40:10.367 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:40:10 smithi084 ceph-mon[29774]: Adjusting osd_memory_target on smithi154 to 17088M 2024-09-20T22:40:10.367 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:40:10 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:40:10.367 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:40:10 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:40:10.367 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:40:10 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:40:10.368 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:40:10 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:40:11.230 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:40:11 smithi154 ceph-mon[33264]: osdmap e33: 5 total, 5 up, 5 in 2024-09-20T22:40:11.233 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-20T22:40:11.259 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph orch daemon add osd smithi154:vg_nvme/lv_3 2024-09-20T22:40:11.364 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:40:11 smithi084 ceph-mon[29774]: osdmap e33: 5 total, 5 up, 5 in 2024-09-20T22:40:11.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:40:11 smithi084 ceph-mon[40024]: osdmap e33: 5 total, 5 up, 5 in 2024-09-20T22:40:11.503 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:40:12.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:40:12 smithi154 ceph-mon[33264]: osdmap e34: 5 total, 5 up, 5 in 2024-09-20T22:40:12.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:40:12 smithi154 ceph-mon[33264]: pgmap v111: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 147 KiB/s, 0 objects/s recovering 2024-09-20T22:40:12.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:40:12 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-20T22:40:12.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:40:12 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-20T22:40:12.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:40:12 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:40:12.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:40:12 smithi084 ceph-mon[29774]: osdmap e34: 5 total, 5 up, 5 in 2024-09-20T22:40:12.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:40:12 smithi084 ceph-mon[29774]: pgmap v111: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 147 KiB/s, 0 objects/s recovering 2024-09-20T22:40:12.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:40:12 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-20T22:40:12.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:40:12 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-20T22:40:12.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:40:12 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:40:12.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:40:12 smithi084 ceph-mon[40024]: osdmap e34: 5 total, 5 up, 5 in 2024-09-20T22:40:12.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:40:12 smithi084 ceph-mon[40024]: pgmap v111: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 147 KiB/s, 0 objects/s recovering 2024-09-20T22:40:12.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:40:12 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-20T22:40:12.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:40:12 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-20T22:40:12.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:40:12 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:40:13.342 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:40:13 smithi154 ceph-mon[33264]: from='client.24223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi154:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:40:13.342 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:40:13 smithi154 ceph-mon[33264]: osdmap e35: 5 total, 5 up, 5 in 2024-09-20T22:40:13.364 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:40:13 smithi084 ceph-mon[29774]: from='client.24223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi154:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:40:13.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:40:13 smithi084 ceph-mon[29774]: osdmap e35: 5 total, 5 up, 5 in 2024-09-20T22:40:13.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:40:13 smithi084 ceph-mon[40024]: from='client.24223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi154:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:40:13.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:40:13 smithi084 ceph-mon[40024]: osdmap e35: 5 total, 5 up, 5 in 2024-09-20T22:40:14.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:40:14 smithi154 ceph-mon[33264]: pgmap v113: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 124 KiB/s, 0 objects/s recovering 2024-09-20T22:40:14.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:40:14 smithi084 ceph-mon[29774]: pgmap v113: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 124 KiB/s, 0 objects/s recovering 2024-09-20T22:40:14.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:40:14 smithi084 ceph-mon[40024]: pgmap v113: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 124 KiB/s, 0 objects/s recovering 2024-09-20T22:40:15.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:40:15 smithi154 ceph-mon[33264]: from='client.? 172.21.15.154:0/4259281084' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4fa0dc8e-25cc-46cd-96ff-33f20b6faec3"}]: dispatch 2024-09-20T22:40:15.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:40:15 smithi154 ceph-mon[33264]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4fa0dc8e-25cc-46cd-96ff-33f20b6faec3"}]: dispatch 2024-09-20T22:40:15.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:40:15 smithi154 ceph-mon[33264]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4fa0dc8e-25cc-46cd-96ff-33f20b6faec3"}]': finished 2024-09-20T22:40:15.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:40:15 smithi154 ceph-mon[33264]: osdmap e36: 6 total, 5 up, 6 in 2024-09-20T22:40:15.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:40:15 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-20T22:40:15.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:40:15 smithi154 ceph-mon[33264]: from='client.? 172.21.15.154:0/747866478' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-20T22:40:15.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:40:15 smithi084 ceph-mon[29774]: from='client.? 172.21.15.154:0/4259281084' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4fa0dc8e-25cc-46cd-96ff-33f20b6faec3"}]: dispatch 2024-09-20T22:40:15.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:40:15 smithi084 ceph-mon[29774]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4fa0dc8e-25cc-46cd-96ff-33f20b6faec3"}]: dispatch 2024-09-20T22:40:15.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:40:15 smithi084 ceph-mon[29774]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4fa0dc8e-25cc-46cd-96ff-33f20b6faec3"}]': finished 2024-09-20T22:40:15.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:40:15 smithi084 ceph-mon[29774]: osdmap e36: 6 total, 5 up, 6 in 2024-09-20T22:40:15.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:40:15 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-20T22:40:15.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:40:15 smithi084 ceph-mon[29774]: from='client.? 172.21.15.154:0/747866478' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-20T22:40:15.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:40:15 smithi084 ceph-mon[40024]: from='client.? 172.21.15.154:0/4259281084' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4fa0dc8e-25cc-46cd-96ff-33f20b6faec3"}]: dispatch 2024-09-20T22:40:15.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:40:15 smithi084 ceph-mon[40024]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4fa0dc8e-25cc-46cd-96ff-33f20b6faec3"}]: dispatch 2024-09-20T22:40:15.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:40:15 smithi084 ceph-mon[40024]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4fa0dc8e-25cc-46cd-96ff-33f20b6faec3"}]': finished 2024-09-20T22:40:15.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:40:15 smithi084 ceph-mon[40024]: osdmap e36: 6 total, 5 up, 6 in 2024-09-20T22:40:15.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:40:15 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-20T22:40:15.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:40:15 smithi084 ceph-mon[40024]: from='client.? 172.21.15.154:0/747866478' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-20T22:40:16.364 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:40:16 smithi084 ceph-mon[29774]: pgmap v115: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 136 MiB used, 447 GiB / 447 GiB avail; 102 KiB/s, 0 objects/s recovering 2024-09-20T22:40:16.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:40:16 smithi084 ceph-mon[40024]: pgmap v115: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 136 MiB used, 447 GiB / 447 GiB avail; 102 KiB/s, 0 objects/s recovering 2024-09-20T22:40:16.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:40:16 smithi154 ceph-mon[33264]: pgmap v115: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 136 MiB used, 447 GiB / 447 GiB avail; 102 KiB/s, 0 objects/s recovering 2024-09-20T22:40:18.115 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:40:17 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: 2024-09-20T22:40:17.704+0000 7f3606def640 -1 calc_pg_upmaps abort due to max <= 0 2024-09-20T22:40:19.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:40:18 smithi154 ceph-mon[33264]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-20T22:40:19.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:40:18 smithi084 ceph-mon[29774]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-20T22:40:19.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:40:18 smithi084 ceph-mon[40024]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-20T22:40:21.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:40:20 smithi154 ceph-mon[33264]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-20T22:40:21.114 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:40:20 smithi084 ceph-mon[40024]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-20T22:40:21.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:40:20 smithi084 ceph-mon[29774]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-20T22:40:21.939 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:40:21 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-09-20T22:40:21.940 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:40:21 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:40:22.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:40:21 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-09-20T22:40:22.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:40:21 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:40:22.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:40:21 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-09-20T22:40:22.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:40:21 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:40:22.945 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:40:22 smithi154 ceph-mon[33264]: Deploying daemon osd.5 on smithi154 2024-09-20T22:40:22.945 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:40:22 smithi154 ceph-mon[33264]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-20T22:40:23.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:40:22 smithi084 ceph-mon[29774]: Deploying daemon osd.5 on smithi154 2024-09-20T22:40:23.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:40:22 smithi084 ceph-mon[29774]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-20T22:40:23.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:40:22 smithi084 ceph-mon[40024]: Deploying daemon osd.5 on smithi154 2024-09-20T22:40:23.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:40:22 smithi084 ceph-mon[40024]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-20T22:40:24.930 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:40:24 smithi154 ceph-mon[33264]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-20T22:40:25.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:40:24 smithi084 ceph-mon[29774]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-20T22:40:25.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:40:24 smithi084 ceph-mon[40024]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-20T22:40:26.891 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:40:26 smithi154 ceph-mon[33264]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-20T22:40:27.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:40:26 smithi084 ceph-mon[29774]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-20T22:40:27.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:40:26 smithi084 ceph-mon[40024]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-20T22:40:28.073 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:40:27 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:40:28.073 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:40:27 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:40:28.073 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:40:27 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:40:28.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:40:27 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:40:28.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:40:27 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:40:28.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:40:27 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:40:28.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:40:27 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:40:28.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:40:27 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:40:28.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:40:27 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:40:29.105 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:40:28 smithi154 ceph-mon[33264]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-20T22:40:29.114 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:40:28 smithi084 ceph-mon[40024]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-20T22:40:29.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:40:28 smithi084 ceph-mon[29774]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-20T22:40:30.937 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:40:30 smithi154 ceph-mon[33264]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-20T22:40:30.937 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:40:30 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:40:30.937 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:40:30 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:40:30.937 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:40:30 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:40:30.937 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:40:30 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:40:30.937 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:40:30 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:40:31.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:40:30 smithi084 ceph-mon[29774]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-20T22:40:31.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:40:30 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:40:31.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:40:30 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:40:31.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:40:30 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:40:31.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:40:30 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:40:31.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:40:30 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:40:31.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:40:30 smithi084 ceph-mon[40024]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-20T22:40:31.116 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:40:30 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:40:31.116 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:40:30 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:40:31.116 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:40:30 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:40:31.116 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:40:30 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:40:31.116 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:40:30 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:40:31.195 INFO:teuthology.orchestra.run.smithi154.stdout:Created osd(s) 5 on host 'smithi154' 2024-09-20T22:40:31.818 DEBUG:teuthology.orchestra.run.smithi154:osd.5> sudo journalctl -f -n 0 -u ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490@osd.5.service 2024-09-20T22:40:31.821 INFO:tasks.cephadm:Deploying osd.6 on smithi154 with /dev/vg_nvme/lv_2... 2024-09-20T22:40:31.821 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- lvm zap /dev/vg_nvme/lv_2 2024-09-20T22:40:32.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:40:31 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:40:32.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:40:31 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:40:32.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:40:31 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:40:32.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:40:31 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:40:32.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:40:31 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:40:32.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:40:31 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:40:32.154 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:40:31 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:40:32.154 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:40:31 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:40:32.154 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:40:31 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:40:32.629 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:40:32.804 INFO:journalctl@ceph.osd.5.smithi154.stdout:Sep 20 22:40:32 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-5[43596]: 2024-09-20T22:40:32.534+0000 7fb12623e640 -1 osd.5 0 log_to_monitors true 2024-09-20T22:40:33.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:40:32 smithi084 ceph-mon[29774]: purged_snaps scrub starts 2024-09-20T22:40:33.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:40:32 smithi084 ceph-mon[29774]: purged_snaps scrub ok 2024-09-20T22:40:33.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:40:32 smithi084 ceph-mon[29774]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-20T22:40:33.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:40:32 smithi084 ceph-mon[29774]: osdmap e37: 6 total, 5 up, 6 in 2024-09-20T22:40:33.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:40:32 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-20T22:40:33.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:40:32 smithi084 ceph-mon[29774]: from='osd.5 v2:172.21.15.154:6804/493899927' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-20T22:40:33.116 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:40:32 smithi084 ceph-mon[29774]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-20T22:40:33.116 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:40:32 smithi084 ceph-mon[40024]: purged_snaps scrub starts 2024-09-20T22:40:33.116 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:40:32 smithi084 ceph-mon[40024]: purged_snaps scrub ok 2024-09-20T22:40:33.116 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:40:32 smithi084 ceph-mon[40024]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-20T22:40:33.116 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:40:32 smithi084 ceph-mon[40024]: osdmap e37: 6 total, 5 up, 6 in 2024-09-20T22:40:33.116 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:40:32 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-20T22:40:33.116 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:40:32 smithi084 ceph-mon[40024]: from='osd.5 v2:172.21.15.154:6804/493899927' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-20T22:40:33.116 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:40:32 smithi084 ceph-mon[40024]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-20T22:40:33.167 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:40:32 smithi154 ceph-mon[33264]: purged_snaps scrub starts 2024-09-20T22:40:33.167 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:40:32 smithi154 ceph-mon[33264]: purged_snaps scrub ok 2024-09-20T22:40:33.167 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:40:32 smithi154 ceph-mon[33264]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-20T22:40:33.167 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:40:32 smithi154 ceph-mon[33264]: osdmap e37: 6 total, 5 up, 6 in 2024-09-20T22:40:33.167 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:40:32 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-20T22:40:33.167 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:40:32 smithi154 ceph-mon[33264]: from='osd.5 v2:172.21.15.154:6804/493899927' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-20T22:40:33.167 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:40:32 smithi154 ceph-mon[33264]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-20T22:40:34.504 INFO:journalctl@ceph.osd.5.smithi154.stdout:Sep 20 22:40:34 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-5[43596]: 2024-09-20T22:40:34.217+0000 7fb1221c1640 -1 osd.5 0 waiting for initial osdmap 2024-09-20T22:40:34.504 INFO:journalctl@ceph.osd.5.smithi154.stdout:Sep 20 22:40:34 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-5[43596]: 2024-09-20T22:40:34.221+0000 7fb11dfe9640 -1 osd.5 39 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-20T22:40:34.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:40:34 smithi154 ceph-mon[33264]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-20T22:40:34.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:40:34 smithi154 ceph-mon[33264]: from='osd.5 v2:172.21.15.154:6804/493899927' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2024-09-20T22:40:34.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:40:34 smithi154 ceph-mon[33264]: osdmap e38: 6 total, 5 up, 6 in 2024-09-20T22:40:34.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:40:34 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-20T22:40:34.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:40:34 smithi154 ceph-mon[33264]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2024-09-20T22:40:34.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:40:34 smithi154 ceph-mon[33264]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-20T22:40:34.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:40:34 smithi084 ceph-mon[29774]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-20T22:40:34.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:40:34 smithi084 ceph-mon[29774]: from='osd.5 v2:172.21.15.154:6804/493899927' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2024-09-20T22:40:34.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:40:34 smithi084 ceph-mon[29774]: osdmap e38: 6 total, 5 up, 6 in 2024-09-20T22:40:34.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:40:34 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-20T22:40:34.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:40:34 smithi084 ceph-mon[29774]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2024-09-20T22:40:34.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:40:34 smithi084 ceph-mon[29774]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-20T22:40:34.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:40:34 smithi084 ceph-mon[40024]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-20T22:40:34.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:40:34 smithi084 ceph-mon[40024]: from='osd.5 v2:172.21.15.154:6804/493899927' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2024-09-20T22:40:34.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:40:34 smithi084 ceph-mon[40024]: osdmap e38: 6 total, 5 up, 6 in 2024-09-20T22:40:34.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:40:34 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-20T22:40:34.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:40:34 smithi084 ceph-mon[40024]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2024-09-20T22:40:34.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:40:34 smithi084 ceph-mon[40024]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-20T22:40:35.440 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:40:35 smithi154 ceph-mon[33264]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]': finished 2024-09-20T22:40:35.440 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:40:35 smithi154 ceph-mon[33264]: osdmap e39: 6 total, 5 up, 6 in 2024-09-20T22:40:35.440 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:40:35 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-20T22:40:35.440 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:40:35 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-20T22:40:35.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:40:35 smithi084 ceph-mon[29774]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]': finished 2024-09-20T22:40:35.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:40:35 smithi084 ceph-mon[29774]: osdmap e39: 6 total, 5 up, 6 in 2024-09-20T22:40:35.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:40:35 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-20T22:40:35.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:40:35 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-20T22:40:35.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:40:35 smithi084 ceph-mon[40024]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]': finished 2024-09-20T22:40:35.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:40:35 smithi084 ceph-mon[40024]: osdmap e39: 6 total, 5 up, 6 in 2024-09-20T22:40:35.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:40:35 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-20T22:40:35.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:40:35 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-20T22:40:36.493 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:40:36 smithi154 ceph-mon[33264]: purged_snaps scrub starts 2024-09-20T22:40:36.493 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:40:36 smithi154 ceph-mon[33264]: purged_snaps scrub ok 2024-09-20T22:40:36.493 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:40:36 smithi154 ceph-mon[33264]: osd.5 v2:172.21.15.154:6804/493899927 boot 2024-09-20T22:40:36.493 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:40:36 smithi154 ceph-mon[33264]: osdmap e40: 6 total, 6 up, 6 in 2024-09-20T22:40:36.493 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:40:36 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-20T22:40:36.493 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:40:36 smithi154 ceph-mon[33264]: Detected new or changed devices on smithi154 2024-09-20T22:40:36.493 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:40:36 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:40:36.493 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:40:36 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:40:36.493 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:40:36 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:40:36.493 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:40:36 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:40:36.493 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:40:36 smithi154 ceph-mon[33264]: Adjusting osd_memory_target on smithi154 to 8544M 2024-09-20T22:40:36.494 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:40:36 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:40:36.494 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:40:36 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:40:36.494 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:40:36 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:40:36.494 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:40:36 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:40:36.494 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:40:36 smithi154 ceph-mon[33264]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-20T22:40:36.494 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:40:36 smithi154 ceph-mon[33264]: osdmap e41: 6 total, 6 up, 6 in 2024-09-20T22:40:36.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:40:36 smithi084 ceph-mon[29774]: purged_snaps scrub starts 2024-09-20T22:40:36.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:40:36 smithi084 ceph-mon[29774]: purged_snaps scrub ok 2024-09-20T22:40:36.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:40:36 smithi084 ceph-mon[29774]: osd.5 v2:172.21.15.154:6804/493899927 boot 2024-09-20T22:40:36.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:40:36 smithi084 ceph-mon[29774]: osdmap e40: 6 total, 6 up, 6 in 2024-09-20T22:40:36.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:40:36 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-20T22:40:36.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:40:36 smithi084 ceph-mon[29774]: Detected new or changed devices on smithi154 2024-09-20T22:40:36.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:40:36 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:40:36.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:40:36 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:40:36.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:40:36 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:40:36.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:40:36 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:40:36.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:40:36 smithi084 ceph-mon[29774]: Adjusting osd_memory_target on smithi154 to 8544M 2024-09-20T22:40:36.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:40:36 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:40:36.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:40:36 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:40:36.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:40:36 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:40:36.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:40:36 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:40:36.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:40:36 smithi084 ceph-mon[29774]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-20T22:40:36.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:40:36 smithi084 ceph-mon[29774]: osdmap e41: 6 total, 6 up, 6 in 2024-09-20T22:40:36.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:40:36 smithi084 ceph-mon[40024]: purged_snaps scrub starts 2024-09-20T22:40:36.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:40:36 smithi084 ceph-mon[40024]: purged_snaps scrub ok 2024-09-20T22:40:36.617 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:40:36 smithi084 ceph-mon[40024]: osd.5 v2:172.21.15.154:6804/493899927 boot 2024-09-20T22:40:36.617 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:40:36 smithi084 ceph-mon[40024]: osdmap e40: 6 total, 6 up, 6 in 2024-09-20T22:40:36.617 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:40:36 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-20T22:40:36.617 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:40:36 smithi084 ceph-mon[40024]: Detected new or changed devices on smithi154 2024-09-20T22:40:36.617 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:40:36 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:40:36.617 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:40:36 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:40:36.617 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:40:36 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:40:36.617 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:40:36 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:40:36.617 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:40:36 smithi084 ceph-mon[40024]: Adjusting osd_memory_target on smithi154 to 8544M 2024-09-20T22:40:36.617 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:40:36 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:40:36.617 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:40:36 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:40:36.617 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:40:36 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:40:36.617 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:40:36 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:40:36.617 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:40:36 smithi084 ceph-mon[40024]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-20T22:40:36.618 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:40:36 smithi084 ceph-mon[40024]: osdmap e41: 6 total, 6 up, 6 in 2024-09-20T22:40:36.692 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-20T22:40:36.718 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph orch daemon add osd smithi154:vg_nvme/lv_2 2024-09-20T22:40:36.960 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:40:38.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:40:38 smithi154 ceph-mon[33264]: osdmap e42: 6 total, 6 up, 6 in 2024-09-20T22:40:38.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:40:38 smithi154 ceph-mon[33264]: from='client.24250 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi154:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:40:38.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:40:38 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-20T22:40:38.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:40:38 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-20T22:40:38.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:40:38 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:40:38.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:40:38 smithi154 ceph-mon[33264]: pgmap v132: 1 pgs: 1 remapped; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-20T22:40:38.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:40:38 smithi154 ceph-mon[33264]: osdmap e43: 6 total, 6 up, 6 in 2024-09-20T22:40:38.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:40:38 smithi084 ceph-mon[29774]: osdmap e42: 6 total, 6 up, 6 in 2024-09-20T22:40:38.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:40:38 smithi084 ceph-mon[29774]: from='client.24250 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi154:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:40:38.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:40:38 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-20T22:40:38.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:40:38 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-20T22:40:38.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:40:38 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:40:38.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:40:38 smithi084 ceph-mon[29774]: pgmap v132: 1 pgs: 1 remapped; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-20T22:40:38.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:40:38 smithi084 ceph-mon[29774]: osdmap e43: 6 total, 6 up, 6 in 2024-09-20T22:40:38.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:40:38 smithi084 ceph-mon[40024]: osdmap e42: 6 total, 6 up, 6 in 2024-09-20T22:40:38.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:40:38 smithi084 ceph-mon[40024]: from='client.24250 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi154:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:40:38.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:40:38 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-20T22:40:38.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:40:38 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-20T22:40:38.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:40:38 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:40:38.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:40:38 smithi084 ceph-mon[40024]: pgmap v132: 1 pgs: 1 remapped; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-20T22:40:38.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:40:38 smithi084 ceph-mon[40024]: osdmap e43: 6 total, 6 up, 6 in 2024-09-20T22:40:40.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:40:40 smithi084 ceph-mon[29774]: osdmap e44: 6 total, 6 up, 6 in 2024-09-20T22:40:40.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:40:40 smithi084 ceph-mon[29774]: pgmap v135: 1 pgs: 1 remapped; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-20T22:40:40.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:40:40 smithi084 ceph-mon[29774]: from='client.? 172.21.15.154:0/1882920976' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "56788d74-810e-4827-a615-958c12f8a368"}]: dispatch 2024-09-20T22:40:40.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:40:40 smithi084 ceph-mon[29774]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "56788d74-810e-4827-a615-958c12f8a368"}]: dispatch 2024-09-20T22:40:40.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:40:40 smithi084 ceph-mon[29774]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "56788d74-810e-4827-a615-958c12f8a368"}]': finished 2024-09-20T22:40:40.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:40:40 smithi084 ceph-mon[29774]: osdmap e45: 7 total, 6 up, 7 in 2024-09-20T22:40:40.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:40:40 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-20T22:40:40.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:40:40 smithi084 ceph-mon[40024]: osdmap e44: 6 total, 6 up, 6 in 2024-09-20T22:40:40.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:40:40 smithi084 ceph-mon[40024]: pgmap v135: 1 pgs: 1 remapped; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-20T22:40:40.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:40:40 smithi084 ceph-mon[40024]: from='client.? 172.21.15.154:0/1882920976' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "56788d74-810e-4827-a615-958c12f8a368"}]: dispatch 2024-09-20T22:40:40.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:40:40 smithi084 ceph-mon[40024]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "56788d74-810e-4827-a615-958c12f8a368"}]: dispatch 2024-09-20T22:40:40.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:40:40 smithi084 ceph-mon[40024]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "56788d74-810e-4827-a615-958c12f8a368"}]': finished 2024-09-20T22:40:40.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:40:40 smithi084 ceph-mon[40024]: osdmap e45: 7 total, 6 up, 7 in 2024-09-20T22:40:40.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:40:40 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-20T22:40:40.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:40:40 smithi154 ceph-mon[33264]: osdmap e44: 6 total, 6 up, 6 in 2024-09-20T22:40:40.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:40:40 smithi154 ceph-mon[33264]: pgmap v135: 1 pgs: 1 remapped; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-20T22:40:40.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:40:40 smithi154 ceph-mon[33264]: from='client.? 172.21.15.154:0/1882920976' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "56788d74-810e-4827-a615-958c12f8a368"}]: dispatch 2024-09-20T22:40:40.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:40:40 smithi154 ceph-mon[33264]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "56788d74-810e-4827-a615-958c12f8a368"}]: dispatch 2024-09-20T22:40:40.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:40:40 smithi154 ceph-mon[33264]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "56788d74-810e-4827-a615-958c12f8a368"}]': finished 2024-09-20T22:40:40.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:40:40 smithi154 ceph-mon[33264]: osdmap e45: 7 total, 6 up, 7 in 2024-09-20T22:40:40.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:40:40 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-20T22:40:41.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:40:41 smithi084 ceph-mon[29774]: from='client.? 172.21.15.154:0/891008137' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-20T22:40:41.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:40:41 smithi084 ceph-mon[40024]: from='client.? 172.21.15.154:0/891008137' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-20T22:40:41.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:40:41 smithi154 ceph-mon[33264]: from='client.? 172.21.15.154:0/891008137' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-20T22:40:42.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:40:42 smithi084 ceph-mon[29774]: 1.0 scrub starts 2024-09-20T22:40:42.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:40:42 smithi084 ceph-mon[29774]: 1.0 scrub ok 2024-09-20T22:40:42.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:40:42 smithi084 ceph-mon[29774]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 128 KiB/s, 0 objects/s recovering 2024-09-20T22:40:42.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:40:42 smithi084 ceph-mon[40024]: 1.0 scrub starts 2024-09-20T22:40:42.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:40:42 smithi084 ceph-mon[40024]: 1.0 scrub ok 2024-09-20T22:40:42.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:40:42 smithi084 ceph-mon[40024]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 128 KiB/s, 0 objects/s recovering 2024-09-20T22:40:42.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:40:42 smithi154 ceph-mon[33264]: 1.0 scrub starts 2024-09-20T22:40:42.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:40:42 smithi154 ceph-mon[33264]: 1.0 scrub ok 2024-09-20T22:40:42.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:40:42 smithi154 ceph-mon[33264]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 128 KiB/s, 0 objects/s recovering 2024-09-20T22:40:44.364 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:40:44 smithi084 ceph-mon[29774]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-20T22:40:44.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:40:44 smithi084 ceph-mon[40024]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-20T22:40:44.381 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:40:44 smithi154 ceph-mon[33264]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 163 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-20T22:40:47.104 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:40:46 smithi154 ceph-mon[33264]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 75 KiB/s, 0 objects/s recovering 2024-09-20T22:40:47.114 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:40:46 smithi084 ceph-mon[29774]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 75 KiB/s, 0 objects/s recovering 2024-09-20T22:40:47.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:40:46 smithi084 ceph-mon[40024]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 75 KiB/s, 0 objects/s recovering 2024-09-20T22:40:48.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:40:47 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-09-20T22:40:48.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:40:47 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:40:48.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:40:47 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-09-20T22:40:48.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:40:47 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:40:48.116 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:40:47 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-09-20T22:40:48.116 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:40:47 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:40:48.993 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:40:48 smithi154 ceph-mon[33264]: Deploying daemon osd.6 on smithi154 2024-09-20T22:40:48.993 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:40:48 smithi154 ceph-mon[33264]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 67 KiB/s, 0 objects/s recovering 2024-09-20T22:40:49.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:40:48 smithi084 ceph-mon[29774]: Deploying daemon osd.6 on smithi154 2024-09-20T22:40:49.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:40:48 smithi084 ceph-mon[29774]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 67 KiB/s, 0 objects/s recovering 2024-09-20T22:40:49.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:40:48 smithi084 ceph-mon[40024]: Deploying daemon osd.6 on smithi154 2024-09-20T22:40:49.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:40:48 smithi084 ceph-mon[40024]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 67 KiB/s, 0 objects/s recovering 2024-09-20T22:40:50.899 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:40:50 smithi154 ceph-mon[33264]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-20T22:40:51.114 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:40:50 smithi084 ceph-mon[29774]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-20T22:40:51.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:40:50 smithi084 ceph-mon[40024]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-20T22:40:52.902 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:40:52 smithi154 ceph-mon[33264]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 49 KiB/s, 0 objects/s recovering 2024-09-20T22:40:53.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:40:52 smithi084 ceph-mon[29774]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 49 KiB/s, 0 objects/s recovering 2024-09-20T22:40:53.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:40:52 smithi084 ceph-mon[40024]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 49 KiB/s, 0 objects/s recovering 2024-09-20T22:40:53.960 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:40:53 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:40:53.960 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:40:53 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:40:53.960 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:40:53 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:40:54.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:40:53 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:40:54.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:40:53 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:40:54.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:40:53 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:40:54.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:40:53 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:40:54.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:40:53 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:40:54.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:40:53 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:40:54.870 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:40:54 smithi154 ceph-mon[33264]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-20T22:40:55.114 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:40:54 smithi084 ceph-mon[29774]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-20T22:40:55.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:40:54 smithi084 ceph-mon[40024]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-20T22:40:56.836 INFO:teuthology.orchestra.run.smithi154.stdout:Created osd(s) 6 on host 'smithi154' 2024-09-20T22:40:57.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:40:56 smithi084 ceph-mon[29774]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-20T22:40:57.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:40:56 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:40:57.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:40:56 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:40:57.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:40:56 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:40:57.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:40:56 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:40:57.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:40:56 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:40:57.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:40:56 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:40:57.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:40:56 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:40:57.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:40:56 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:40:57.116 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:40:56 smithi084 ceph-mon[40024]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-20T22:40:57.116 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:40:56 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:40:57.116 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:40:56 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:40:57.116 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:40:56 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:40:57.116 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:40:56 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:40:57.116 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:40:56 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:40:57.116 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:40:56 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:40:57.116 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:40:56 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:40:57.116 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:40:56 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:40:57.161 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:40:56 smithi154 ceph-mon[33264]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-20T22:40:57.161 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:40:56 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:40:57.161 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:40:56 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:40:57.161 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:40:56 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:40:57.161 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:40:56 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:40:57.161 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:40:56 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:40:57.161 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:40:56 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:40:57.161 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:40:56 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:40:57.161 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:40:56 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:40:57.488 DEBUG:teuthology.orchestra.run.smithi154:osd.6> sudo journalctl -f -n 0 -u ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490@osd.6.service 2024-09-20T22:40:57.490 INFO:tasks.cephadm:Deploying osd.7 on smithi154 with /dev/vg_nvme/lv_1... 2024-09-20T22:40:57.491 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- lvm zap /dev/vg_nvme/lv_1 2024-09-20T22:40:58.504 INFO:journalctl@ceph.osd.6.smithi154.stdout:Sep 20 22:40:58 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-6[49255]: 2024-09-20T22:40:58.162+0000 7f2ffc0ff640 -1 osd.6 0 log_to_monitors true 2024-09-20T22:40:58.598 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:40:59.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:40:58 smithi084 ceph-mon[29774]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-20T22:40:59.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:40:58 smithi084 ceph-mon[29774]: from='osd.6 v2:172.21.15.154:6808/2375091033' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-20T22:40:59.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:40:58 smithi084 ceph-mon[29774]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-20T22:40:59.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:40:58 smithi084 ceph-mon[40024]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-20T22:40:59.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:40:58 smithi084 ceph-mon[40024]: from='osd.6 v2:172.21.15.154:6808/2375091033' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-20T22:40:59.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:40:58 smithi084 ceph-mon[40024]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-20T22:40:59.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:40:58 smithi154 ceph-mon[33264]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-20T22:40:59.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:40:58 smithi154 ceph-mon[33264]: from='osd.6 v2:172.21.15.154:6808/2375091033' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-20T22:40:59.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:40:58 smithi154 ceph-mon[33264]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-20T22:41:00.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:00 smithi084 ceph-mon[29774]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-20T22:41:00.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:00 smithi084 ceph-mon[29774]: osdmap e46: 7 total, 6 up, 7 in 2024-09-20T22:41:00.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:00 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-20T22:41:00.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:00 smithi084 ceph-mon[29774]: from='osd.6 v2:172.21.15.154:6808/2375091033' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2024-09-20T22:41:00.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:00 smithi084 ceph-mon[29774]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2024-09-20T22:41:00.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:00 smithi084 ceph-mon[29774]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-20T22:41:00.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:00 smithi084 ceph-mon[40024]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-20T22:41:00.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:00 smithi084 ceph-mon[40024]: osdmap e46: 7 total, 6 up, 7 in 2024-09-20T22:41:00.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:00 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-20T22:41:00.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:00 smithi084 ceph-mon[40024]: from='osd.6 v2:172.21.15.154:6808/2375091033' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2024-09-20T22:41:00.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:00 smithi084 ceph-mon[40024]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2024-09-20T22:41:00.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:00 smithi084 ceph-mon[40024]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-20T22:41:00.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:00 smithi154 ceph-mon[33264]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-20T22:41:00.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:00 smithi154 ceph-mon[33264]: osdmap e46: 7 total, 6 up, 7 in 2024-09-20T22:41:00.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:00 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-20T22:41:00.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:00 smithi154 ceph-mon[33264]: from='osd.6 v2:172.21.15.154:6808/2375091033' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2024-09-20T22:41:00.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:00 smithi154 ceph-mon[33264]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2024-09-20T22:41:00.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:00 smithi154 ceph-mon[33264]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-20T22:41:01.004 INFO:journalctl@ceph.osd.6.smithi154.stdout:Sep 20 22:41:00 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-6[49255]: 2024-09-20T22:41:00.537+0000 7f2ff8896640 -1 osd.6 0 waiting for initial osdmap 2024-09-20T22:41:01.004 INFO:journalctl@ceph.osd.6.smithi154.stdout:Sep 20 22:41:00 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-6[49255]: 2024-09-20T22:41:00.546+0000 7f2fefea2640 -1 osd.6 47 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-20T22:41:01.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:01 smithi084 ceph-mon[29774]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]': finished 2024-09-20T22:41:01.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:01 smithi084 ceph-mon[29774]: osdmap e47: 7 total, 6 up, 7 in 2024-09-20T22:41:01.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:01 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-20T22:41:01.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:01 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-20T22:41:01.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:01 smithi084 ceph-mon[29774]: from='osd.6 ' entity='osd.6' 2024-09-20T22:41:01.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:01 smithi084 ceph-mon[40024]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]': finished 2024-09-20T22:41:01.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:01 smithi084 ceph-mon[40024]: osdmap e47: 7 total, 6 up, 7 in 2024-09-20T22:41:01.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:01 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-20T22:41:01.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:01 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-20T22:41:01.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:01 smithi084 ceph-mon[40024]: from='osd.6 ' entity='osd.6' 2024-09-20T22:41:01.391 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:01 smithi154 ceph-mon[33264]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]': finished 2024-09-20T22:41:01.391 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:01 smithi154 ceph-mon[33264]: osdmap e47: 7 total, 6 up, 7 in 2024-09-20T22:41:01.391 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:01 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-20T22:41:01.391 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:01 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-20T22:41:01.391 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:01 smithi154 ceph-mon[33264]: from='osd.6 ' entity='osd.6' 2024-09-20T22:41:02.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:02 smithi084 ceph-mon[29774]: purged_snaps scrub starts 2024-09-20T22:41:02.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:02 smithi084 ceph-mon[29774]: purged_snaps scrub ok 2024-09-20T22:41:02.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:02 smithi084 ceph-mon[29774]: osd.6 v2:172.21.15.154:6808/2375091033 boot 2024-09-20T22:41:02.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:02 smithi084 ceph-mon[29774]: osdmap e48: 7 total, 7 up, 7 in 2024-09-20T22:41:02.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:02 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-20T22:41:02.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:02 smithi084 ceph-mon[29774]: purged_snaps scrub starts 2024-09-20T22:41:02.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:02 smithi084 ceph-mon[29774]: purged_snaps scrub ok 2024-09-20T22:41:02.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:02 smithi084 ceph-mon[29774]: Detected new or changed devices on smithi154 2024-09-20T22:41:02.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:02 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:41:02.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:02 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:41:02.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:02 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:41:02.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:02 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:41:02.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:02 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:41:02.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:02 smithi084 ceph-mon[29774]: Adjusting osd_memory_target on smithi154 to 5696M 2024-09-20T22:41:02.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:02 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:41:02.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:02 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:41:02.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:02 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:41:02.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:02 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:41:02.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:02 smithi084 ceph-mon[29774]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 626 GiB / 626 GiB avail 2024-09-20T22:41:02.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:02 smithi084 ceph-mon[40024]: purged_snaps scrub starts 2024-09-20T22:41:02.367 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:02 smithi084 ceph-mon[40024]: purged_snaps scrub ok 2024-09-20T22:41:02.367 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:02 smithi084 ceph-mon[40024]: osd.6 v2:172.21.15.154:6808/2375091033 boot 2024-09-20T22:41:02.367 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:02 smithi084 ceph-mon[40024]: osdmap e48: 7 total, 7 up, 7 in 2024-09-20T22:41:02.367 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:02 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-20T22:41:02.367 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:02 smithi084 ceph-mon[40024]: purged_snaps scrub starts 2024-09-20T22:41:02.367 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:02 smithi084 ceph-mon[40024]: purged_snaps scrub ok 2024-09-20T22:41:02.367 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:02 smithi084 ceph-mon[40024]: Detected new or changed devices on smithi154 2024-09-20T22:41:02.367 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:02 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:41:02.367 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:02 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:41:02.367 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:02 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:41:02.367 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:02 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:41:02.367 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:02 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:41:02.367 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:02 smithi084 ceph-mon[40024]: Adjusting osd_memory_target on smithi154 to 5696M 2024-09-20T22:41:02.367 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:02 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:41:02.367 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:02 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:41:02.368 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:02 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:41:02.368 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:02 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:41:02.368 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:02 smithi084 ceph-mon[40024]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 626 GiB / 626 GiB avail 2024-09-20T22:41:02.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:02 smithi154 ceph-mon[33264]: purged_snaps scrub starts 2024-09-20T22:41:02.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:02 smithi154 ceph-mon[33264]: purged_snaps scrub ok 2024-09-20T22:41:02.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:02 smithi154 ceph-mon[33264]: osd.6 v2:172.21.15.154:6808/2375091033 boot 2024-09-20T22:41:02.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:02 smithi154 ceph-mon[33264]: osdmap e48: 7 total, 7 up, 7 in 2024-09-20T22:41:02.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:02 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-20T22:41:02.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:02 smithi154 ceph-mon[33264]: purged_snaps scrub starts 2024-09-20T22:41:02.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:02 smithi154 ceph-mon[33264]: purged_snaps scrub ok 2024-09-20T22:41:02.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:02 smithi154 ceph-mon[33264]: Detected new or changed devices on smithi154 2024-09-20T22:41:02.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:02 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:41:02.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:02 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:41:02.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:02 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:41:02.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:02 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:41:02.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:02 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:41:02.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:02 smithi154 ceph-mon[33264]: Adjusting osd_memory_target on smithi154 to 5696M 2024-09-20T22:41:02.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:02 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:41:02.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:02 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:41:02.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:02 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:41:02.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:02 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:41:02.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:02 smithi154 ceph-mon[33264]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 626 GiB / 626 GiB avail 2024-09-20T22:41:02.852 INFO:teuthology.orchestra.run.smithi154.stdout: 2024-09-20T22:41:02.877 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph orch daemon add osd smithi154:vg_nvme/lv_1 2024-09-20T22:41:03.120 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:41:03.359 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:03 smithi154 ceph-mon[33264]: osdmap e49: 7 total, 7 up, 7 in 2024-09-20T22:41:03.364 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:03 smithi084 ceph-mon[29774]: osdmap e49: 7 total, 7 up, 7 in 2024-09-20T22:41:03.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:03 smithi084 ceph-mon[40024]: osdmap e49: 7 total, 7 up, 7 in 2024-09-20T22:41:04.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:04 smithi084 ceph-mon[29774]: osdmap e50: 7 total, 7 up, 7 in 2024-09-20T22:41:04.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:04 smithi084 ceph-mon[29774]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 626 GiB / 626 GiB avail 2024-09-20T22:41:04.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:04 smithi084 ceph-mon[29774]: from='client.24277 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi154:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:41:04.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:04 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-20T22:41:04.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:04 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-20T22:41:04.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:04 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:41:04.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:04 smithi084 ceph-mon[40024]: osdmap e50: 7 total, 7 up, 7 in 2024-09-20T22:41:04.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:04 smithi084 ceph-mon[40024]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 626 GiB / 626 GiB avail 2024-09-20T22:41:04.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:04 smithi084 ceph-mon[40024]: from='client.24277 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi154:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:41:04.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:04 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-20T22:41:04.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:04 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-20T22:41:04.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:04 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:41:04.390 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:04 smithi154 ceph-mon[33264]: osdmap e50: 7 total, 7 up, 7 in 2024-09-20T22:41:04.390 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:04 smithi154 ceph-mon[33264]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 626 GiB / 626 GiB avail 2024-09-20T22:41:04.390 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:04 smithi154 ceph-mon[33264]: from='client.24277 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi154:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:41:04.390 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:04 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-20T22:41:04.390 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:04 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-20T22:41:04.390 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:04 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:41:05.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:05 smithi084 ceph-mon[29774]: osdmap e51: 7 total, 7 up, 7 in 2024-09-20T22:41:05.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:05 smithi084 ceph-mon[40024]: osdmap e51: 7 total, 7 up, 7 in 2024-09-20T22:41:05.409 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:05 smithi154 ceph-mon[33264]: osdmap e51: 7 total, 7 up, 7 in 2024-09-20T22:41:06.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:06 smithi154 ceph-mon[33264]: osdmap e52: 7 total, 7 up, 7 in 2024-09-20T22:41:06.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:06 smithi154 ceph-mon[33264]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 626 GiB / 626 GiB avail 2024-09-20T22:41:06.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:06 smithi084 ceph-mon[40024]: osdmap e52: 7 total, 7 up, 7 in 2024-09-20T22:41:06.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:06 smithi084 ceph-mon[40024]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 626 GiB / 626 GiB avail 2024-09-20T22:41:06.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:06 smithi084 ceph-mon[29774]: osdmap e52: 7 total, 7 up, 7 in 2024-09-20T22:41:06.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:06 smithi084 ceph-mon[29774]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 261 MiB used, 626 GiB / 626 GiB avail 2024-09-20T22:41:07.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:07 smithi154 ceph-mon[33264]: from='client.? 172.21.15.154:0/689505538' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "83503727-afc1-4b95-8b96-ecb63f3338c2"}]: dispatch 2024-09-20T22:41:07.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:07 smithi154 ceph-mon[33264]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "83503727-afc1-4b95-8b96-ecb63f3338c2"}]: dispatch 2024-09-20T22:41:07.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:07 smithi154 ceph-mon[33264]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "83503727-afc1-4b95-8b96-ecb63f3338c2"}]': finished 2024-09-20T22:41:07.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:07 smithi154 ceph-mon[33264]: osdmap e53: 8 total, 7 up, 8 in 2024-09-20T22:41:07.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:07 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-20T22:41:07.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:07 smithi154 ceph-mon[33264]: from='client.? 172.21.15.154:0/4145823629' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-20T22:41:07.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:07 smithi084 ceph-mon[29774]: from='client.? 172.21.15.154:0/689505538' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "83503727-afc1-4b95-8b96-ecb63f3338c2"}]: dispatch 2024-09-20T22:41:07.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:07 smithi084 ceph-mon[29774]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "83503727-afc1-4b95-8b96-ecb63f3338c2"}]: dispatch 2024-09-20T22:41:07.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:07 smithi084 ceph-mon[29774]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "83503727-afc1-4b95-8b96-ecb63f3338c2"}]': finished 2024-09-20T22:41:07.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:07 smithi084 ceph-mon[29774]: osdmap e53: 8 total, 7 up, 8 in 2024-09-20T22:41:07.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:07 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-20T22:41:07.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:07 smithi084 ceph-mon[29774]: from='client.? 172.21.15.154:0/4145823629' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-20T22:41:07.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:07 smithi084 ceph-mon[40024]: from='client.? 172.21.15.154:0/689505538' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "83503727-afc1-4b95-8b96-ecb63f3338c2"}]: dispatch 2024-09-20T22:41:07.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:07 smithi084 ceph-mon[40024]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "83503727-afc1-4b95-8b96-ecb63f3338c2"}]: dispatch 2024-09-20T22:41:07.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:07 smithi084 ceph-mon[40024]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "83503727-afc1-4b95-8b96-ecb63f3338c2"}]': finished 2024-09-20T22:41:07.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:07 smithi084 ceph-mon[40024]: osdmap e53: 8 total, 7 up, 8 in 2024-09-20T22:41:07.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:07 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-20T22:41:07.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:07 smithi084 ceph-mon[40024]: from='client.? 172.21.15.154:0/4145823629' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-20T22:41:08.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:08 smithi154 ceph-mon[33264]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 124 KiB/s, 0 objects/s recovering 2024-09-20T22:41:08.614 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:08 smithi084 ceph-mon[29774]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 124 KiB/s, 0 objects/s recovering 2024-09-20T22:41:08.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:08 smithi084 ceph-mon[40024]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 124 KiB/s, 0 objects/s recovering 2024-09-20T22:41:10.701 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:10 smithi154 ceph-mon[33264]: purged_snaps scrub starts 2024-09-20T22:41:10.701 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:10 smithi154 ceph-mon[33264]: purged_snaps scrub ok 2024-09-20T22:41:10.701 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:10 smithi154 ceph-mon[33264]: osdmap e54: 8 total, 7 up, 8 in 2024-09-20T22:41:10.701 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:10 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-20T22:41:10.701 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:10 smithi154 ceph-mon[33264]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 102 KiB/s, 0 objects/s recovering 2024-09-20T22:41:10.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:10 smithi084 ceph-mon[29774]: purged_snaps scrub starts 2024-09-20T22:41:10.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:10 smithi084 ceph-mon[29774]: purged_snaps scrub ok 2024-09-20T22:41:10.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:10 smithi084 ceph-mon[29774]: osdmap e54: 8 total, 7 up, 8 in 2024-09-20T22:41:10.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:10 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-20T22:41:10.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:10 smithi084 ceph-mon[29774]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 102 KiB/s, 0 objects/s recovering 2024-09-20T22:41:10.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:10 smithi084 ceph-mon[40024]: purged_snaps scrub starts 2024-09-20T22:41:10.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:10 smithi084 ceph-mon[40024]: purged_snaps scrub ok 2024-09-20T22:41:10.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:10 smithi084 ceph-mon[40024]: osdmap e54: 8 total, 7 up, 8 in 2024-09-20T22:41:10.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:10 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-20T22:41:10.866 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:10 smithi084 ceph-mon[40024]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 102 KiB/s, 0 objects/s recovering 2024-09-20T22:41:11.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:11 smithi154 ceph-mon[33264]: purged_snaps scrub starts 2024-09-20T22:41:11.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:11 smithi154 ceph-mon[33264]: purged_snaps scrub ok 2024-09-20T22:41:11.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:11 smithi154 ceph-mon[33264]: osdmap e55: 8 total, 7 up, 8 in 2024-09-20T22:41:11.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:11 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-20T22:41:11.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:11 smithi084 ceph-mon[29774]: purged_snaps scrub starts 2024-09-20T22:41:11.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:11 smithi084 ceph-mon[29774]: purged_snaps scrub ok 2024-09-20T22:41:11.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:11 smithi084 ceph-mon[29774]: osdmap e55: 8 total, 7 up, 8 in 2024-09-20T22:41:11.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:11 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-20T22:41:11.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:11 smithi084 ceph-mon[40024]: purged_snaps scrub starts 2024-09-20T22:41:11.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:11 smithi084 ceph-mon[40024]: purged_snaps scrub ok 2024-09-20T22:41:11.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:11 smithi084 ceph-mon[40024]: osdmap e55: 8 total, 7 up, 8 in 2024-09-20T22:41:11.866 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:11 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-20T22:41:12.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:12 smithi154 ceph-mon[33264]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-20T22:41:12.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:12 smithi084 ceph-mon[29774]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-20T22:41:12.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:12 smithi084 ceph-mon[40024]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-20T22:41:14.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:13 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-09-20T22:41:14.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:13 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:41:14.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:13 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-09-20T22:41:14.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:13 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:41:14.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:13 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-09-20T22:41:14.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:13 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:41:15.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:14 smithi154 ceph-mon[33264]: Deploying daemon osd.7 on smithi154 2024-09-20T22:41:15.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:14 smithi154 ceph-mon[33264]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-09-20T22:41:15.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:14 smithi084 ceph-mon[29774]: Deploying daemon osd.7 on smithi154 2024-09-20T22:41:15.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:14 smithi084 ceph-mon[29774]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-09-20T22:41:15.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:14 smithi084 ceph-mon[40024]: Deploying daemon osd.7 on smithi154 2024-09-20T22:41:15.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:14 smithi084 ceph-mon[40024]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-09-20T22:41:16.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:15 smithi154 ceph-mon[33264]: purged_snaps scrub starts 2024-09-20T22:41:16.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:15 smithi154 ceph-mon[33264]: purged_snaps scrub ok 2024-09-20T22:41:16.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:15 smithi154 ceph-mon[33264]: osdmap e56: 8 total, 7 up, 8 in 2024-09-20T22:41:16.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:15 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-20T22:41:16.114 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:15 smithi084 ceph-mon[29774]: purged_snaps scrub starts 2024-09-20T22:41:16.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:15 smithi084 ceph-mon[29774]: purged_snaps scrub ok 2024-09-20T22:41:16.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:15 smithi084 ceph-mon[29774]: osdmap e56: 8 total, 7 up, 8 in 2024-09-20T22:41:16.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:15 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-20T22:41:16.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:15 smithi084 ceph-mon[40024]: purged_snaps scrub starts 2024-09-20T22:41:16.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:15 smithi084 ceph-mon[40024]: purged_snaps scrub ok 2024-09-20T22:41:16.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:15 smithi084 ceph-mon[40024]: osdmap e56: 8 total, 7 up, 8 in 2024-09-20T22:41:16.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:15 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-20T22:41:17.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:16 smithi154 ceph-mon[33264]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-09-20T22:41:17.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:16 smithi084 ceph-mon[29774]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-09-20T22:41:17.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:16 smithi084 ceph-mon[40024]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-09-20T22:41:19.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:18 smithi154 ceph-mon[33264]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-09-20T22:41:19.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:18 smithi084 ceph-mon[29774]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-09-20T22:41:19.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:18 smithi084 ceph-mon[40024]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-09-20T22:41:20.095 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:19 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:41:20.095 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:19 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:41:20.095 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:19 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:41:20.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:19 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:41:20.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:19 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:41:20.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:19 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:41:20.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:19 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:41:20.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:19 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:41:20.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:19 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:41:20.990 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:20 smithi154 ceph-mon[33264]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-09-20T22:41:21.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:20 smithi084 ceph-mon[29774]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-09-20T22:41:21.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:20 smithi084 ceph-mon[40024]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-09-20T22:41:22.768 INFO:teuthology.orchestra.run.smithi154.stdout:Created osd(s) 7 on host 'smithi154' 2024-09-20T22:41:23.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:22 smithi154 ceph-mon[33264]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-09-20T22:41:23.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:22 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:41:23.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:22 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:41:23.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:22 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:41:23.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:22 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:41:23.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:22 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:41:23.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:22 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:41:23.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:22 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:41:23.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:22 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:41:23.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:22 smithi084 ceph-mon[29774]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-09-20T22:41:23.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:22 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:41:23.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:22 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:41:23.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:22 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:41:23.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:22 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:41:23.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:22 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:41:23.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:22 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:41:23.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:22 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:41:23.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:22 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:41:23.116 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:22 smithi084 ceph-mon[40024]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-09-20T22:41:23.116 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:22 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:41:23.116 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:22 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:41:23.116 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:22 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:41:23.116 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:22 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:41:23.116 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:22 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:41:23.116 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:22 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:41:23.116 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:22 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:41:23.116 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:22 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:41:23.414 DEBUG:teuthology.orchestra.run.smithi154:osd.7> sudo journalctl -f -n 0 -u ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490@osd.7.service 2024-09-20T22:41:23.416 INFO:tasks.cephadm:Waiting for 8 OSDs to come up... 2024-09-20T22:41:23.416 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd stat -f json 2024-09-20T22:41:23.673 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:41:24.312 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-20T22:41:24.312 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:24 smithi084 ceph-mon[29774]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-09-20T22:41:24.312 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:24 smithi084 ceph-mon[40024]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-09-20T22:41:24.503 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:24 smithi154 ceph-mon[33264]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-09-20T22:41:24.503 INFO:journalctl@ceph.osd.7.smithi154.stdout:Sep 20 22:41:24 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-7[54754]: 2024-09-20T22:41:24.323+0000 7fee66018640 -1 osd.7 0 log_to_monitors true 2024-09-20T22:41:24.966 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":56,"num_osds":8,"num_up_osds":7,"osd_up_since":1726872061,"num_in_osds":8,"osd_in_since":1726872066,"num_remapped_pgs":0} 2024-09-20T22:41:25.299 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:25 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/1177506844' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-20T22:41:25.299 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:25 smithi084 ceph-mon[29774]: from='osd.7 v2:172.21.15.154:6812/3717704491' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-20T22:41:25.299 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:25 smithi084 ceph-mon[29774]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-20T22:41:25.300 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:25 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/1177506844' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-20T22:41:25.300 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:25 smithi084 ceph-mon[40024]: from='osd.7 v2:172.21.15.154:6812/3717704491' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-20T22:41:25.300 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:25 smithi084 ceph-mon[40024]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-20T22:41:25.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:25 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/1177506844' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-20T22:41:25.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:25 smithi154 ceph-mon[33264]: from='osd.7 v2:172.21.15.154:6812/3717704491' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-20T22:41:25.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:25 smithi154 ceph-mon[33264]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-20T22:41:25.967 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd stat -f json 2024-09-20T22:41:26.232 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:41:26.354 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:26 smithi084 ceph-mon[29774]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-20T22:41:26.354 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:26 smithi084 ceph-mon[29774]: osdmap e57: 8 total, 7 up, 8 in 2024-09-20T22:41:26.354 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:26 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-20T22:41:26.355 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:26 smithi084 ceph-mon[29774]: from='osd.7 v2:172.21.15.154:6812/3717704491' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2024-09-20T22:41:26.355 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:26 smithi084 ceph-mon[29774]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2024-09-20T22:41:26.355 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:26 smithi084 ceph-mon[29774]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-09-20T22:41:26.355 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:26 smithi084 ceph-mon[40024]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-20T22:41:26.355 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:26 smithi084 ceph-mon[40024]: osdmap e57: 8 total, 7 up, 8 in 2024-09-20T22:41:26.355 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:26 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-20T22:41:26.355 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:26 smithi084 ceph-mon[40024]: from='osd.7 v2:172.21.15.154:6812/3717704491' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2024-09-20T22:41:26.355 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:26 smithi084 ceph-mon[40024]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2024-09-20T22:41:26.356 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:26 smithi084 ceph-mon[40024]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-09-20T22:41:26.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:26 smithi154 ceph-mon[33264]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-20T22:41:26.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:26 smithi154 ceph-mon[33264]: osdmap e57: 8 total, 7 up, 8 in 2024-09-20T22:41:26.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:26 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-20T22:41:26.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:26 smithi154 ceph-mon[33264]: from='osd.7 v2:172.21.15.154:6812/3717704491' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2024-09-20T22:41:26.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:26 smithi154 ceph-mon[33264]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]: dispatch 2024-09-20T22:41:26.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:26 smithi154 ceph-mon[33264]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-09-20T22:41:26.879 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-20T22:41:27.004 INFO:journalctl@ceph.osd.7.smithi154.stdout:Sep 20 22:41:26 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-7[54754]: 2024-09-20T22:41:26.551+0000 7fee61f9b640 -1 osd.7 0 waiting for initial osdmap 2024-09-20T22:41:27.004 INFO:journalctl@ceph.osd.7.smithi154.stdout:Sep 20 22:41:26 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-7[54754]: 2024-09-20T22:41:26.558+0000 7fee59dbb640 -1 osd.7 58 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-20T22:41:27.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:27 smithi084 ceph-mon[40024]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]': finished 2024-09-20T22:41:27.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:27 smithi084 ceph-mon[40024]: osdmap e58: 8 total, 7 up, 8 in 2024-09-20T22:41:27.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:27 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-20T22:41:27.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:27 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-20T22:41:27.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:27 smithi084 ceph-mon[40024]: Detected new or changed devices on smithi154 2024-09-20T22:41:27.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:27 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:41:27.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:27 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:41:27.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:27 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:41:27.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:27 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:41:27.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:27 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:41:27.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:27 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:41:27.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:27 smithi084 ceph-mon[40024]: Adjusting osd_memory_target on smithi154 to 4272M 2024-09-20T22:41:27.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:27 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:41:27.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:27 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:41:27.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:27 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:41:27.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:27 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:41:27.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:27 smithi084 ceph-mon[40024]: from='osd.7 ' entity='osd.7' 2024-09-20T22:41:27.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:27 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/382194759' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-20T22:41:27.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:27 smithi084 ceph-mon[40024]: osd.7 v2:172.21.15.154:6812/3717704491 boot 2024-09-20T22:41:27.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:27 smithi084 ceph-mon[40024]: osdmap e59: 8 total, 8 up, 8 in 2024-09-20T22:41:27.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:27 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-20T22:41:27.367 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:27 smithi084 ceph-mon[29774]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]': finished 2024-09-20T22:41:27.367 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:27 smithi084 ceph-mon[29774]: osdmap e58: 8 total, 7 up, 8 in 2024-09-20T22:41:27.367 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:27 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-20T22:41:27.367 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:27 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-20T22:41:27.367 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:27 smithi084 ceph-mon[29774]: Detected new or changed devices on smithi154 2024-09-20T22:41:27.367 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:27 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:41:27.367 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:27 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:41:27.367 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:27 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:41:27.367 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:27 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:41:27.367 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:27 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:41:27.367 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:27 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:41:27.367 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:27 smithi084 ceph-mon[29774]: Adjusting osd_memory_target on smithi154 to 4272M 2024-09-20T22:41:27.367 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:27 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:41:27.367 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:27 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:41:27.368 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:27 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:41:27.368 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:27 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:41:27.368 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:27 smithi084 ceph-mon[29774]: from='osd.7 ' entity='osd.7' 2024-09-20T22:41:27.368 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:27 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/382194759' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-20T22:41:27.368 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:27 smithi084 ceph-mon[29774]: osd.7 v2:172.21.15.154:6812/3717704491 boot 2024-09-20T22:41:27.368 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:27 smithi084 ceph-mon[29774]: osdmap e59: 8 total, 8 up, 8 in 2024-09-20T22:41:27.368 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:27 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-20T22:41:27.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:27 smithi154 ceph-mon[33264]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi154", "root=default"]}]': finished 2024-09-20T22:41:27.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:27 smithi154 ceph-mon[33264]: osdmap e58: 8 total, 7 up, 8 in 2024-09-20T22:41:27.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:27 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-20T22:41:27.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:27 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-20T22:41:27.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:27 smithi154 ceph-mon[33264]: Detected new or changed devices on smithi154 2024-09-20T22:41:27.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:27 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:41:27.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:27 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:41:27.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:27 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:41:27.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:27 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:41:27.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:27 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:41:27.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:27 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:41:27.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:27 smithi154 ceph-mon[33264]: Adjusting osd_memory_target on smithi154 to 4272M 2024-09-20T22:41:27.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:27 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:41:27.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:27 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:41:27.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:27 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:41:27.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:27 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:41:27.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:27 smithi154 ceph-mon[33264]: from='osd.7 ' entity='osd.7' 2024-09-20T22:41:27.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:27 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/382194759' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-20T22:41:27.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:27 smithi154 ceph-mon[33264]: osd.7 v2:172.21.15.154:6812/3717704491 boot 2024-09-20T22:41:27.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:27 smithi154 ceph-mon[33264]: osdmap e59: 8 total, 8 up, 8 in 2024-09-20T22:41:27.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:27 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-20T22:41:27.507 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":58,"num_osds":8,"num_up_osds":7,"osd_up_since":1726872061,"num_in_osds":8,"osd_in_since":1726872066,"num_remapped_pgs":0} 2024-09-20T22:41:28.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:28 smithi084 ceph-mon[29774]: purged_snaps scrub starts 2024-09-20T22:41:28.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:28 smithi084 ceph-mon[29774]: purged_snaps scrub ok 2024-09-20T22:41:28.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:28 smithi084 ceph-mon[29774]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 500 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:41:28.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:28 smithi084 ceph-mon[29774]: osdmap e60: 8 total, 8 up, 8 in 2024-09-20T22:41:28.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:28 smithi084 ceph-mon[40024]: purged_snaps scrub starts 2024-09-20T22:41:28.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:28 smithi084 ceph-mon[40024]: purged_snaps scrub ok 2024-09-20T22:41:28.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:28 smithi084 ceph-mon[40024]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 500 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:41:28.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:28 smithi084 ceph-mon[40024]: osdmap e60: 8 total, 8 up, 8 in 2024-09-20T22:41:28.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:28 smithi154 ceph-mon[33264]: purged_snaps scrub starts 2024-09-20T22:41:28.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:28 smithi154 ceph-mon[33264]: purged_snaps scrub ok 2024-09-20T22:41:28.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:28 smithi154 ceph-mon[33264]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 500 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:41:28.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:28 smithi154 ceph-mon[33264]: osdmap e60: 8 total, 8 up, 8 in 2024-09-20T22:41:28.508 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd stat -f json 2024-09-20T22:41:28.767 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:41:29.419 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-20T22:41:30.049 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":61,"num_osds":8,"num_up_osds":8,"osd_up_since":1726872087,"num_in_osds":8,"osd_in_since":1726872066,"num_remapped_pgs":1} 2024-09-20T22:41:30.049 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd dump --format=json 2024-09-20T22:41:30.297 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:41:30.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:30 smithi084 ceph-mon[29774]: osdmap e61: 8 total, 8 up, 8 in 2024-09-20T22:41:30.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:30 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/1450585127' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-20T22:41:30.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:30 smithi084 ceph-mon[29774]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 500 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:41:30.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:30 smithi084 ceph-mon[40024]: osdmap e61: 8 total, 8 up, 8 in 2024-09-20T22:41:30.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:30 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/1450585127' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-20T22:41:30.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:30 smithi084 ceph-mon[40024]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 500 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:41:30.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:30 smithi154 ceph-mon[33264]: osdmap e61: 8 total, 8 up, 8 in 2024-09-20T22:41:30.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:30 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/1450585127' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-20T22:41:30.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:30 smithi154 ceph-mon[33264]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 500 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:41:30.934 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-20T22:41:30.935 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":62,"fsid":"a9c0bc70-77a0-11ef-baf5-efdc52797490","created":"2024-09-20T22:36:30.240626+0000","modified":"2024-09-20T22:41:30.079411+0000","last_up_change":"2024-09-20T22:41:27.098565+0000","last_in_change":"2024-09-20T22:41:06.226708+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-20T22:39:21.984381+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ea36a489-6f91-4b2f-b784-b90506b9805d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6801","nonce":268084626}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6802","nonce":268084626}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6804","nonce":268084626}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6803","nonce":268084626}]},"public_addr":"172.21.15.84:6801/268084626","cluster_addr":"172.21.15.84:6802/268084626","heartbeat_back_addr":"172.21.15.84:6804/268084626","heartbeat_front_addr":"172.21.15.84:6803/268084626","state":["exists","up"]},{"osd":1,"uuid":"c8649d2f-27ef-47f8-bc94-f1cd9e62bdb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6805","nonce":418374136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6806","nonce":418374136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6808","nonce":418374136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6807","nonce":418374136}]},"public_addr":"172.21.15.84:6805/418374136","cluster_addr":"172.21.15.84:6806/418374136","heartbeat_back_addr":"172.21.15.84:6808/418374136","heartbeat_front_addr":"172.21.15.84:6807/418374136","state":["exists","up"]},{"osd":2,"uuid":"c728eb48-a0f0-4de4-a651-7f3bec867280","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6809","nonce":3977168611}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6810","nonce":3977168611}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6812","nonce":3977168611}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6811","nonce":3977168611}]},"public_addr":"172.21.15.84:6809/3977168611","cluster_addr":"172.21.15.84:6810/3977168611","heartbeat_back_addr":"172.21.15.84:6812/3977168611","heartbeat_front_addr":"172.21.15.84:6811/3977168611","state":["exists","up"]},{"osd":3,"uuid":"573f44eb-0853-4630-a7d8-9caaeb256f9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6813","nonce":2192239945}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6814","nonce":2192239945}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6816","nonce":2192239945}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6815","nonce":2192239945}]},"public_addr":"172.21.15.84:6813/2192239945","cluster_addr":"172.21.15.84:6814/2192239945","heartbeat_back_addr":"172.21.15.84:6816/2192239945","heartbeat_front_addr":"172.21.15.84:6815/2192239945","state":["exists","up"]},{"osd":4,"uuid":"a52c3b94-c5f3-490f-8ac8-09061b9b2abb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":1793136002}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6801","nonce":1793136002}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6803","nonce":1793136002}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":1793136002}]},"public_addr":"172.21.15.154:6800/1793136002","cluster_addr":"172.21.15.154:6801/1793136002","heartbeat_back_addr":"172.21.15.154:6803/1793136002","heartbeat_front_addr":"172.21.15.154:6802/1793136002","state":["exists","up"]},{"osd":5,"uuid":"4fa0dc8e-25cc-46cd-96ff-33f20b6faec3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":493899927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6805","nonce":493899927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6807","nonce":493899927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":493899927}]},"public_addr":"172.21.15.154:6804/493899927","cluster_addr":"172.21.15.154:6805/493899927","heartbeat_back_addr":"172.21.15.154:6807/493899927","heartbeat_front_addr":"172.21.15.154:6806/493899927","state":["exists","up"]},{"osd":6,"uuid":"56788d74-810e-4827-a615-958c12f8a368","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":2375091033}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6809","nonce":2375091033}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6811","nonce":2375091033}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":2375091033}]},"public_addr":"172.21.15.154:6808/2375091033","cluster_addr":"172.21.15.154:6809/2375091033","heartbeat_back_addr":"172.21.15.154:6811/2375091033","heartbeat_front_addr":"172.21.15.154:6810/2375091033","state":["exists","up"]},{"osd":7,"uuid":"83503727-afc1-4b95-8b96-ecb63f3338c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":3717704491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6813","nonce":3717704491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6815","nonce":3717704491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":3717704491}]},"public_addr":"172.21.15.154:6812/3717704491","cluster_addr":"172.21.15.154:6813/3717704491","heartbeat_back_addr":"172.21.15.154:6815/3717704491","heartbeat_front_addr":"172.21.15.154:6814/3717704491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T22:41:09.485558+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T22:41:14.497933+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T22:40:31.540262+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T22:41:01.300651+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T22:41:10.316534+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T22:40:33.503712+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T22:40:59.192804+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T22:41:25.354165+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.84:0/1361143214":"2024-09-21T22:37:17.690104+0000","172.21.15.84:0/3879263635":"2024-09-21T22:37:17.690104+0000","172.21.15.84:0/1176849276":"2024-09-21T22:37:17.690104+0000","172.21.15.84:0/567614908":"2024-09-21T22:36:54.917812+0000","172.21.15.84:6800/2932420050":"2024-09-21T22:37:17.690104+0000","172.21.15.84:6800/820908470":"2024-09-21T22:36:54.917812+0000","172.21.15.84:0/25873236":"2024-09-21T22:36:54.917812+0000","172.21.15.84:0/969472279":"2024-09-21T22:36:54.917812+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-20T22:41:31.208 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:31 smithi084 ceph-mon[40024]: osdmap e62: 8 total, 8 up, 8 in 2024-09-20T22:41:31.208 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:31 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/869409116' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-20T22:41:31.209 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:31 smithi084 ceph-mon[29774]: osdmap e62: 8 total, 8 up, 8 in 2024-09-20T22:41:31.209 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:31 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/869409116' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-20T22:41:31.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:31 smithi154 ceph-mon[33264]: osdmap e62: 8 total, 8 up, 8 in 2024-09-20T22:41:31.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:31 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/869409116' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-20T22:41:31.587 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-20T22:39:21.984381+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-20T22:41:31.587 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd pool get .mgr pg_num 2024-09-20T22:41:31.829 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:41:32.287 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:32 smithi084 ceph-mon[29774]: osdmap e63: 8 total, 8 up, 8 in 2024-09-20T22:41:32.288 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:32 smithi084 ceph-mon[29774]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 500 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:41:32.288 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:32 smithi084 ceph-mon[40024]: osdmap e63: 8 total, 8 up, 8 in 2024-09-20T22:41:32.288 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:32 smithi084 ceph-mon[40024]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 500 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:41:32.451 INFO:teuthology.orchestra.run.smithi084.stdout:pg_num: 1 2024-09-20T22:41:32.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:32 smithi154 ceph-mon[33264]: osdmap e63: 8 total, 8 up, 8 in 2024-09-20T22:41:32.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:32 smithi154 ceph-mon[33264]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 500 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:41:33.130 INFO:tasks.cephadm:Adding ceph.rgw.foo.a on smithi084 2024-09-20T22:41:33.130 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph orch apply rgw foo.a --placement '1;smithi084=foo.a' 2024-09-20T22:41:33.364 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:33 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/4123899798' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-20T22:41:33.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:33 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/4123899798' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-20T22:41:33.387 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:41:33.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:33 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/4123899798' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-20T22:41:34.011 INFO:teuthology.orchestra.run.smithi154.stdout:Scheduled rgw.foo.a update... 2024-09-20T22:41:34.455 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:34 smithi084 ceph-mon[29774]: purged_snaps scrub starts 2024-09-20T22:41:34.456 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:34 smithi084 ceph-mon[29774]: purged_snaps scrub ok 2024-09-20T22:41:34.456 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:34 smithi084 ceph-mon[29774]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 102 KiB/s, 0 objects/s recovering 2024-09-20T22:41:34.456 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:34 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:41:34.456 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:34 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:41:34.456 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:34 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:41:34.456 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:34 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:41:34.456 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:34 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:41:34.456 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:34 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-20T22:41:34.456 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:34 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-09-20T22:41:34.456 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:34 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:41:34.456 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:34 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:41:34.457 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:34 smithi084 ceph-mon[40024]: purged_snaps scrub starts 2024-09-20T22:41:34.457 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:34 smithi084 ceph-mon[40024]: purged_snaps scrub ok 2024-09-20T22:41:34.457 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:34 smithi084 ceph-mon[40024]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 102 KiB/s, 0 objects/s recovering 2024-09-20T22:41:34.457 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:34 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:41:34.457 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:34 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:41:34.457 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:34 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:41:34.457 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:34 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:41:34.457 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:34 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:41:34.457 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:34 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-20T22:41:34.457 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:34 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-09-20T22:41:34.457 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:34 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:41:34.457 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:34 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:41:34.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:34 smithi154 ceph-mon[33264]: purged_snaps scrub starts 2024-09-20T22:41:34.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:34 smithi154 ceph-mon[33264]: purged_snaps scrub ok 2024-09-20T22:41:34.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:34 smithi154 ceph-mon[33264]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 102 KiB/s, 0 objects/s recovering 2024-09-20T22:41:34.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:34 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:41:34.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:34 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:41:34.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:34 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:41:34.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:34 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:41:34.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:34 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:41:34.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:34 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-20T22:41:34.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:34 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-09-20T22:41:34.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:34 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:41:34.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:34 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:41:34.712 DEBUG:teuthology.orchestra.run.smithi084:rgw.foo.a> sudo journalctl -f -n 0 -u ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490@rgw.foo.a.service 2024-09-20T22:41:34.714 INFO:tasks.cephadm:Adding ceph.iscsi.iscsi.a on smithi154 2024-09-20T22:41:34.715 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd pool create datapool 3 3 replicated 2024-09-20T22:41:34.958 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:41:35.383 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:35 smithi084 ceph-mon[40024]: from='client.24316 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi084=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:41:35.384 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:35 smithi084 ceph-mon[40024]: Saving service rgw.foo.a spec with placement smithi084=foo.a;count:1 2024-09-20T22:41:35.384 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:35 smithi084 ceph-mon[40024]: Deploying daemon rgw.foo.a on smithi084 2024-09-20T22:41:35.384 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:35 smithi084 ceph-mon[40024]: osdmap e64: 8 total, 8 up, 8 in 2024-09-20T22:41:35.384 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:35 smithi084 ceph-mon[29774]: from='client.24316 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi084=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:41:35.384 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:35 smithi084 ceph-mon[29774]: Saving service rgw.foo.a spec with placement smithi084=foo.a;count:1 2024-09-20T22:41:35.384 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:35 smithi084 ceph-mon[29774]: Deploying daemon rgw.foo.a on smithi084 2024-09-20T22:41:35.384 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:35 smithi084 ceph-mon[29774]: osdmap e64: 8 total, 8 up, 8 in 2024-09-20T22:41:35.406 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:35 smithi154 ceph-mon[33264]: from='client.24316 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi084=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:41:35.406 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:35 smithi154 ceph-mon[33264]: Saving service rgw.foo.a spec with placement smithi084=foo.a;count:1 2024-09-20T22:41:35.406 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:35 smithi154 ceph-mon[33264]: Deploying daemon rgw.foo.a on smithi084 2024-09-20T22:41:35.406 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:35 smithi154 ceph-mon[33264]: osdmap e64: 8 total, 8 up, 8 in 2024-09-20T22:41:36.059 INFO:journalctl@ceph.rgw.foo.a.smithi084.stdout:Sep 20 22:41:35 smithi084 systemd[1]: Starting Ceph rgw.foo.a for a9c0bc70-77a0-11ef-baf5-efdc52797490... 2024-09-20T22:41:36.141 INFO:teuthology.orchestra.run.smithi154.stderr:pool 'datapool' created 2024-09-20T22:41:36.364 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:36 smithi084 ceph-mon[29774]: from='client.? 172.21.15.154:0/192667530' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-09-20T22:41:36.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:36 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-09-20T22:41:36.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:36 smithi084 ceph-mon[29774]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-20T22:41:36.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:36 smithi084 ceph-mon[40024]: from='client.? 172.21.15.154:0/192667530' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-09-20T22:41:36.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:36 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-09-20T22:41:36.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:36 smithi084 ceph-mon[40024]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-20T22:41:36.365 INFO:journalctl@ceph.rgw.foo.a.smithi084.stdout:Sep 20 22:41:36 smithi084 podman[66066]: 2024-09-20 22:41:36.057465915 +0000 UTC m=+0.112269744 container create 965332737d0498104993f06eef5bf117688ec60a7a771574f9a7de8a0aea92b7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-rgw-foo-a, io.buildah.version=1.37.2, RELEASE=squid-9e2ac70, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-09-20T22:41:36.365 INFO:journalctl@ceph.rgw.foo.a.smithi084.stdout:Sep 20 22:41:36 smithi084 podman[66066]: 2024-09-20 22:41:35.960938333 +0000 UTC m=+0.015742122 image pull cc9251b744a3faafca7c3095f79bfff81873afe823f31a03a11e015a34abcdbd quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5 2024-09-20T22:41:36.365 INFO:journalctl@ceph.rgw.foo.a.smithi084.stdout:Sep 20 22:41:36 smithi084 podman[66066]: 2024-09-20 22:41:36.268367178 +0000 UTC m=+0.323170965 container init 965332737d0498104993f06eef5bf117688ec60a7a771574f9a7de8a0aea92b7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-rgw-foo-a, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, RELEASE=squid-9e2ac70, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-20T22:41:36.366 INFO:journalctl@ceph.rgw.foo.a.smithi084.stdout:Sep 20 22:41:36 smithi084 podman[66066]: 2024-09-20 22:41:36.271337968 +0000 UTC m=+0.326141764 container start 965332737d0498104993f06eef5bf117688ec60a7a771574f9a7de8a0aea92b7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-rgw-foo-a, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, RELEASE=squid-9e2ac70, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-09-20T22:41:36.459 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:36 smithi154 ceph-mon[33264]: from='client.? 172.21.15.154:0/192667530' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-09-20T22:41:36.459 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:36 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-09-20T22:41:36.460 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:36 smithi154 ceph-mon[33264]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-20T22:41:36.796 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- rbd pool init datapool 2024-09-20T22:41:36.864 INFO:journalctl@ceph.rgw.foo.a.smithi084.stdout:Sep 20 22:41:36 smithi084 bash[66066]: 965332737d0498104993f06eef5bf117688ec60a7a771574f9a7de8a0aea92b7 2024-09-20T22:41:36.865 INFO:journalctl@ceph.rgw.foo.a.smithi084.stdout:Sep 20 22:41:36 smithi084 systemd[1]: Started Ceph rgw.foo.a for a9c0bc70-77a0-11ef-baf5-efdc52797490. 2024-09-20T22:41:37.044 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:41:37.325 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:37 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-09-20T22:41:37.342 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:37 smithi154 ceph-mon[33264]: osdmap e65: 8 total, 8 up, 8 in 2024-09-20T22:41:37.342 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:37 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:41:37.343 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:37 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:41:37.343 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:37 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:41:37.343 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:37 smithi154 ceph-mon[33264]: Saving service rgw.foo.a spec with placement smithi084=foo.a;count:1 2024-09-20T22:41:37.343 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:37 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:41:37.343 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:37 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:41:37.343 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:37 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:41:37.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:37 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-09-20T22:41:37.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:37 smithi084 ceph-mon[29774]: osdmap e65: 8 total, 8 up, 8 in 2024-09-20T22:41:37.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:37 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:41:37.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:37 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:41:37.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:37 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:41:37.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:37 smithi084 ceph-mon[29774]: Saving service rgw.foo.a spec with placement smithi084=foo.a;count:1 2024-09-20T22:41:37.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:37 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:41:37.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:37 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:41:37.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:37 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:41:37.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:37 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-09-20T22:41:37.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:37 smithi084 ceph-mon[40024]: osdmap e65: 8 total, 8 up, 8 in 2024-09-20T22:41:37.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:37 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:41:37.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:37 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:41:37.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:37 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:41:37.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:37 smithi084 ceph-mon[40024]: Saving service rgw.foo.a spec with placement smithi084=foo.a;count:1 2024-09-20T22:41:37.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:37 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:41:37.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:37 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:41:37.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:37 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:41:38.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:38 smithi154 ceph-mon[33264]: osdmap e66: 8 total, 8 up, 8 in 2024-09-20T22:41:38.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:38 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/2183377608' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-09-20T22:41:38.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:38 smithi154 ceph-mon[33264]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-09-20T22:41:38.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:38 smithi154 ceph-mon[33264]: from='client.? 172.21.15.154:0/343304630' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-09-20T22:41:38.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:38 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-09-20T22:41:38.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:38 smithi154 ceph-mon[33264]: pgmap v186: 36 pgs: 2 creating+peering, 1 active+clean, 33 unknown; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-20T22:41:38.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:38 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:41:38.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:38 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:41:38.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:38 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:41:38.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:38 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:41:38.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:38 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:41:38.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:38 smithi154 ceph-mon[33264]: Checking dashboard <-> RGW credentials 2024-09-20T22:41:38.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:38 smithi084 ceph-mon[40024]: osdmap e66: 8 total, 8 up, 8 in 2024-09-20T22:41:38.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:38 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/2183377608' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-09-20T22:41:38.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:38 smithi084 ceph-mon[40024]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-09-20T22:41:38.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:38 smithi084 ceph-mon[40024]: from='client.? 172.21.15.154:0/343304630' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-09-20T22:41:38.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:38 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-09-20T22:41:38.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:38 smithi084 ceph-mon[40024]: pgmap v186: 36 pgs: 2 creating+peering, 1 active+clean, 33 unknown; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-20T22:41:38.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:38 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:41:38.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:38 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:41:38.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:38 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:41:38.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:38 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:41:38.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:38 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:41:38.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:38 smithi084 ceph-mon[40024]: Checking dashboard <-> RGW credentials 2024-09-20T22:41:38.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:38 smithi084 ceph-mon[29774]: osdmap e66: 8 total, 8 up, 8 in 2024-09-20T22:41:38.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:38 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/2183377608' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-09-20T22:41:38.617 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:38 smithi084 ceph-mon[29774]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-09-20T22:41:38.617 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:38 smithi084 ceph-mon[29774]: from='client.? 172.21.15.154:0/343304630' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-09-20T22:41:38.617 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:38 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-09-20T22:41:38.617 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:38 smithi084 ceph-mon[29774]: pgmap v186: 36 pgs: 2 creating+peering, 1 active+clean, 33 unknown; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-20T22:41:38.617 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:38 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:41:38.617 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:38 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:41:38.617 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:38 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:41:38.617 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:38 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:41:38.617 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:38 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:41:38.617 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:38 smithi084 ceph-mon[29774]: Checking dashboard <-> RGW credentials 2024-09-20T22:41:39.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:39 smithi154 ceph-mon[33264]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-09-20T22:41:39.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:39 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-09-20T22:41:39.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:39 smithi154 ceph-mon[33264]: osdmap e67: 8 total, 8 up, 8 in 2024-09-20T22:41:39.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:39 smithi154 ceph-mon[33264]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-20T22:41:39.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:39 smithi084 ceph-mon[29774]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-09-20T22:41:39.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:39 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-09-20T22:41:39.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:39 smithi084 ceph-mon[29774]: osdmap e67: 8 total, 8 up, 8 in 2024-09-20T22:41:39.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:39 smithi084 ceph-mon[29774]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-20T22:41:39.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:39 smithi084 ceph-mon[40024]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-09-20T22:41:39.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:39 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-09-20T22:41:39.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:39 smithi084 ceph-mon[40024]: osdmap e67: 8 total, 8 up, 8 in 2024-09-20T22:41:39.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:39 smithi084 ceph-mon[40024]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-20T22:41:40.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:40 smithi154 ceph-mon[33264]: osdmap e68: 8 total, 8 up, 8 in 2024-09-20T22:41:40.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:40 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/2994273271' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-09-20T22:41:40.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:40 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/3480761128' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-09-20T22:41:40.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:40 smithi154 ceph-mon[33264]: pgmap v189: 68 pgs: 2 creating+peering, 5 active+clean, 61 unknown; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:41:40.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:40 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/2994273271' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-09-20T22:41:40.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:40 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/3480761128' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-09-20T22:41:40.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:40 smithi154 ceph-mon[33264]: osdmap e69: 8 total, 8 up, 8 in 2024-09-20T22:41:40.516 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:40 smithi084 ceph-mon[29774]: osdmap e68: 8 total, 8 up, 8 in 2024-09-20T22:41:40.516 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:40 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/2994273271' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-09-20T22:41:40.516 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:40 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/3480761128' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-09-20T22:41:40.516 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:40 smithi084 ceph-mon[29774]: pgmap v189: 68 pgs: 2 creating+peering, 5 active+clean, 61 unknown; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:41:40.516 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:40 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/2994273271' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-09-20T22:41:40.516 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:40 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/3480761128' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-09-20T22:41:40.516 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:40 smithi084 ceph-mon[29774]: osdmap e69: 8 total, 8 up, 8 in 2024-09-20T22:41:40.517 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:40 smithi084 ceph-mon[40024]: osdmap e68: 8 total, 8 up, 8 in 2024-09-20T22:41:40.517 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:40 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/2994273271' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-09-20T22:41:40.517 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:40 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/3480761128' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-09-20T22:41:40.517 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:40 smithi084 ceph-mon[40024]: pgmap v189: 68 pgs: 2 creating+peering, 5 active+clean, 61 unknown; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:41:40.517 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:40 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/2994273271' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-09-20T22:41:40.517 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:40 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/3480761128' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-09-20T22:41:40.517 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:40 smithi084 ceph-mon[40024]: osdmap e69: 8 total, 8 up, 8 in 2024-09-20T22:41:40.833 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph orch apply iscsi datapool admin admin --trusted_ip_list 172.21.15.154 --placement '1;smithi154=iscsi.a' 2024-09-20T22:41:41.078 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:41:41.716 INFO:teuthology.orchestra.run.smithi154.stdout:Scheduled iscsi.datapool update... 2024-09-20T22:41:42.391 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:42 smithi154 ceph-mon[33264]: osdmap e70: 8 total, 8 up, 8 in 2024-09-20T22:41:42.391 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:42 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/2994273271' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-09-20T22:41:42.391 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:42 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/3480761128' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-09-20T22:41:42.391 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:42 smithi154 ceph-mon[33264]: from='client.24358 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.154", "placement": "1;smithi154=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:41:42.391 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:42 smithi154 ceph-mon[33264]: Saving service iscsi.datapool spec with placement smithi154=iscsi.a;count:1 2024-09-20T22:41:42.391 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:42 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:41:42.391 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:42 smithi154 ceph-mon[33264]: pgmap v192: 100 pgs: 2 creating+activating, 2 creating+peering, 35 active+clean, 61 unknown; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1.2 KiB/s wr, 2 op/s 2024-09-20T22:41:42.417 INFO:tasks.cephadm:Distributing iscsi-gateway.cfg... 2024-09-20T22:41:42.418 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-09-20T22:41:42.418 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd of=/etc/ceph/iscsi-gateway.cfg 2024-09-20T22:41:42.440 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:42 smithi084 ceph-mon[40024]: osdmap e70: 8 total, 8 up, 8 in 2024-09-20T22:41:42.440 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:42 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/2994273271' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-09-20T22:41:42.440 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:42 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/3480761128' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-09-20T22:41:42.441 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:42 smithi084 ceph-mon[40024]: from='client.24358 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.154", "placement": "1;smithi154=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:41:42.441 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:42 smithi084 ceph-mon[40024]: Saving service iscsi.datapool spec with placement smithi154=iscsi.a;count:1 2024-09-20T22:41:42.441 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:42 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:41:42.441 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:42 smithi084 ceph-mon[40024]: pgmap v192: 100 pgs: 2 creating+activating, 2 creating+peering, 35 active+clean, 61 unknown; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1.2 KiB/s wr, 2 op/s 2024-09-20T22:41:42.441 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:42 smithi084 ceph-mon[29774]: osdmap e70: 8 total, 8 up, 8 in 2024-09-20T22:41:42.441 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:42 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/2994273271' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-09-20T22:41:42.441 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:42 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/3480761128' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-09-20T22:41:42.441 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:42 smithi084 ceph-mon[29774]: from='client.24358 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.154", "placement": "1;smithi154=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:41:42.441 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:42 smithi084 ceph-mon[29774]: Saving service iscsi.datapool spec with placement smithi154=iscsi.a;count:1 2024-09-20T22:41:42.441 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:42 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:41:42.441 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:42 smithi084 ceph-mon[29774]: pgmap v192: 100 pgs: 2 creating+activating, 2 creating+peering, 35 active+clean, 61 unknown; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1.2 KiB/s wr, 2 op/s 2024-09-20T22:41:42.445 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-09-20T22:41:42.445 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd of=/etc/ceph/iscsi-gateway.cfg 2024-09-20T22:41:42.471 DEBUG:teuthology.orchestra.run.smithi154:iscsi.iscsi.a> sudo journalctl -f -n 0 -u ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490@iscsi.iscsi.a.service 2024-09-20T22:41:42.513 INFO:tasks.cephadm:Adding prometheus.a on smithi154 2024-09-20T22:41:42.513 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph orch apply prometheus '1;smithi154=a' 2024-09-20T22:41:42.794 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:41:43.460 INFO:teuthology.orchestra.run.smithi154.stdout:Scheduled prometheus update... 2024-09-20T22:41:43.460 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:43 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/2994273271' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-09-20T22:41:43.460 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:43 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/3480761128' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-09-20T22:41:43.460 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:43 smithi154 ceph-mon[33264]: osdmap e71: 8 total, 8 up, 8 in 2024-09-20T22:41:43.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:43 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/2994273271' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-09-20T22:41:43.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:43 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/3480761128' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-09-20T22:41:43.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:43 smithi084 ceph-mon[40024]: osdmap e71: 8 total, 8 up, 8 in 2024-09-20T22:41:43.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:43 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/2994273271' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-09-20T22:41:43.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:43 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/3480761128' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-09-20T22:41:43.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:43 smithi084 ceph-mon[29774]: osdmap e71: 8 total, 8 up, 8 in 2024-09-20T22:41:44.086 DEBUG:teuthology.orchestra.run.smithi154:prometheus.a> sudo journalctl -f -n 0 -u ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490@prometheus.a.service 2024-09-20T22:41:44.088 INFO:tasks.cephadm:Adding node-exporter.a on smithi084 2024-09-20T22:41:44.088 INFO:tasks.cephadm:Adding node-exporter.b on smithi154 2024-09-20T22:41:44.088 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph orch apply node-exporter '2;smithi084=a;smithi154=b' 2024-09-20T22:41:44.378 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:41:44.499 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:44 smithi154 ceph-mon[33264]: osdmap e72: 8 total, 8 up, 8 in 2024-09-20T22:41:44.499 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:44 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/2994273271' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-09-20T22:41:44.499 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:44 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/3480761128' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-09-20T22:41:44.499 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:44 smithi154 ceph-mon[33264]: from='client.24364 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi154=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:41:44.499 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:44 smithi154 ceph-mon[33264]: Saving service prometheus spec with placement smithi154=a;count:1 2024-09-20T22:41:44.499 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:44 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:41:44.499 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:44 smithi154 ceph-mon[33264]: pgmap v195: 132 pgs: 13 creating+peering, 2 creating+activating, 57 active+clean, 60 unknown; 578 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 2.0 KiB/s wr, 6 op/s 2024-09-20T22:41:44.499 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:44 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/2994273271' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-09-20T22:41:44.499 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:44 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/3480761128' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-09-20T22:41:44.499 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:44 smithi154 ceph-mon[33264]: osdmap e73: 8 total, 8 up, 8 in 2024-09-20T22:41:44.500 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:44 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/3480761128' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-09-20T22:41:44.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:44 smithi084 ceph-mon[40024]: osdmap e72: 8 total, 8 up, 8 in 2024-09-20T22:41:44.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:44 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/2994273271' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-09-20T22:41:44.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:44 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/3480761128' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-09-20T22:41:44.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:44 smithi084 ceph-mon[40024]: from='client.24364 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi154=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:41:44.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:44 smithi084 ceph-mon[40024]: Saving service prometheus spec with placement smithi154=a;count:1 2024-09-20T22:41:44.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:44 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:41:44.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:44 smithi084 ceph-mon[40024]: pgmap v195: 132 pgs: 13 creating+peering, 2 creating+activating, 57 active+clean, 60 unknown; 578 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 2.0 KiB/s wr, 6 op/s 2024-09-20T22:41:44.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:44 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/2994273271' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-09-20T22:41:44.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:44 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/3480761128' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-09-20T22:41:44.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:44 smithi084 ceph-mon[40024]: osdmap e73: 8 total, 8 up, 8 in 2024-09-20T22:41:44.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:44 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/3480761128' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-09-20T22:41:44.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:44 smithi084 ceph-mon[29774]: osdmap e72: 8 total, 8 up, 8 in 2024-09-20T22:41:44.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:44 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/2994273271' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-09-20T22:41:44.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:44 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/3480761128' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-09-20T22:41:44.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:44 smithi084 ceph-mon[29774]: from='client.24364 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi154=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:41:44.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:44 smithi084 ceph-mon[29774]: Saving service prometheus spec with placement smithi154=a;count:1 2024-09-20T22:41:44.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:44 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:41:44.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:44 smithi084 ceph-mon[29774]: pgmap v195: 132 pgs: 13 creating+peering, 2 creating+activating, 57 active+clean, 60 unknown; 578 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 2.0 KiB/s wr, 6 op/s 2024-09-20T22:41:44.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:44 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/2994273271' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-09-20T22:41:44.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:44 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/3480761128' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-09-20T22:41:44.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:44 smithi084 ceph-mon[29774]: osdmap e73: 8 total, 8 up, 8 in 2024-09-20T22:41:44.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:44 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/3480761128' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-09-20T22:41:45.004 INFO:teuthology.orchestra.run.smithi154.stdout:Scheduled node-exporter update... 2024-09-20T22:41:45.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:45 smithi154 ceph-mon[33264]: 1.0 scrub starts 2024-09-20T22:41:45.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:45 smithi154 ceph-mon[33264]: 1.0 scrub ok 2024-09-20T22:41:45.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:45 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/2994273271' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-09-20T22:41:45.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:45 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:41:45.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:45 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/3480761128' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-09-20T22:41:45.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:45 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/2994273271' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-09-20T22:41:45.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:45 smithi154 ceph-mon[33264]: osdmap e74: 8 total, 8 up, 8 in 2024-09-20T22:41:45.614 INFO:journalctl@ceph.rgw.foo.a.smithi084.stdout:Sep 20 22:41:45 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-rgw-foo-a[66081]: 2024-09-20T22:41:45.244+0000 7f95fe673880 -1 LDAP not started since no server URIs were provided in the configuration. 2024-09-20T22:41:45.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:45 smithi084 ceph-mon[40024]: 1.0 scrub starts 2024-09-20T22:41:45.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:45 smithi084 ceph-mon[40024]: 1.0 scrub ok 2024-09-20T22:41:45.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:45 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/2994273271' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-09-20T22:41:45.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:45 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:41:45.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:45 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/3480761128' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-09-20T22:41:45.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:45 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/2994273271' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-09-20T22:41:45.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:45 smithi084 ceph-mon[40024]: osdmap e74: 8 total, 8 up, 8 in 2024-09-20T22:41:45.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:45 smithi084 ceph-mon[29774]: 1.0 scrub starts 2024-09-20T22:41:45.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:45 smithi084 ceph-mon[29774]: 1.0 scrub ok 2024-09-20T22:41:45.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:45 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/2994273271' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-09-20T22:41:45.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:45 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:41:45.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:45 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/3480761128' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-09-20T22:41:45.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:45 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/2994273271' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-09-20T22:41:45.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:45 smithi084 ceph-mon[29774]: osdmap e74: 8 total, 8 up, 8 in 2024-09-20T22:41:45.697 DEBUG:teuthology.orchestra.run.smithi084:node-exporter.a> sudo journalctl -f -n 0 -u ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490@node-exporter.a.service 2024-09-20T22:41:45.699 DEBUG:teuthology.orchestra.run.smithi154:node-exporter.b> sudo journalctl -f -n 0 -u ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490@node-exporter.b.service 2024-09-20T22:41:45.701 INFO:tasks.cephadm:Adding alertmanager.a on smithi084 2024-09-20T22:41:45.702 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph orch apply alertmanager '1;smithi084=a' 2024-09-20T22:41:46.422 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:41:46.647 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:46 smithi154 ceph-mon[33264]: from='client.24370 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi084=a;smithi154=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:41:46.647 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:46 smithi154 ceph-mon[33264]: Saving service node-exporter spec with placement smithi084=a;smithi154=b;count:2 2024-09-20T22:41:46.647 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:46 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:41:46.647 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:46 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:41:46.647 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:46 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:41:46.647 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:46 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-20T22:41:46.647 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:46 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-09-20T22:41:46.648 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:46 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:41:46.648 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:46 smithi154 ceph-mon[33264]: Deploying daemon iscsi.iscsi.a on smithi154 2024-09-20T22:41:46.648 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:46 smithi154 ceph-mon[33264]: pgmap v198: 132 pgs: 18 creating+peering, 81 active+clean, 33 unknown; 578 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2024-09-20T22:41:46.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:46 smithi084 ceph-mon[29774]: from='client.24370 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi084=a;smithi154=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:41:46.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:46 smithi084 ceph-mon[29774]: Saving service node-exporter spec with placement smithi084=a;smithi154=b;count:2 2024-09-20T22:41:46.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:46 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:41:46.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:46 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:41:46.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:46 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:41:46.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:46 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-20T22:41:46.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:46 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-09-20T22:41:46.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:46 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:41:46.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:46 smithi084 ceph-mon[29774]: Deploying daemon iscsi.iscsi.a on smithi154 2024-09-20T22:41:46.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:46 smithi084 ceph-mon[29774]: pgmap v198: 132 pgs: 18 creating+peering, 81 active+clean, 33 unknown; 578 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2024-09-20T22:41:46.866 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:46 smithi084 ceph-mon[40024]: from='client.24370 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi084=a;smithi154=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:41:46.866 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:46 smithi084 ceph-mon[40024]: Saving service node-exporter spec with placement smithi084=a;smithi154=b;count:2 2024-09-20T22:41:46.866 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:46 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:41:46.866 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:46 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:41:46.866 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:46 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:41:46.866 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:46 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-20T22:41:46.866 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:46 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-09-20T22:41:46.866 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:46 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:41:46.866 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:46 smithi084 ceph-mon[40024]: Deploying daemon iscsi.iscsi.a on smithi154 2024-09-20T22:41:46.866 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:46 smithi084 ceph-mon[40024]: pgmap v198: 132 pgs: 18 creating+peering, 81 active+clean, 33 unknown; 578 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2024-09-20T22:41:47.062 INFO:teuthology.orchestra.run.smithi154.stdout:Scheduled alertmanager update... 2024-09-20T22:41:47.734 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:47 smithi154 ceph-mon[33264]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-20T22:41:47.734 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:47 smithi154 ceph-mon[33264]: Cluster is now healthy 2024-09-20T22:41:47.734 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:47 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:41:47.871 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:47 smithi084 ceph-mon[29774]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-20T22:41:47.871 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:47 smithi084 ceph-mon[29774]: Cluster is now healthy 2024-09-20T22:41:47.871 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:47 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:41:47.872 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:47 smithi084 ceph-mon[40024]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-20T22:41:47.872 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:47 smithi084 ceph-mon[40024]: Cluster is now healthy 2024-09-20T22:41:47.872 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:47 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:41:48.004 INFO:journalctl@ceph.iscsi.iscsi.a.smithi154.stdout:Sep 20 22:41:47 smithi154 systemd[1]: Starting Ceph iscsi.iscsi.a for a9c0bc70-77a0-11ef-baf5-efdc52797490... 2024-09-20T22:41:48.144 DEBUG:teuthology.orchestra.run.smithi084:alertmanager.a> sudo journalctl -f -n 0 -u ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490@alertmanager.a.service 2024-09-20T22:41:48.147 INFO:tasks.cephadm:Adding grafana.a on smithi154 2024-09-20T22:41:48.147 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph orch apply grafana '1;smithi154=a' 2024-09-20T22:41:48.403 INFO:journalctl@ceph.iscsi.iscsi.a.smithi154.stdout:Sep 20 22:41:48 smithi154 podman[59690]: 2024-09-20 22:41:48.229074854 +0000 UTC m=+0.016175987 image pull cc9251b744a3faafca7c3095f79bfff81873afe823f31a03a11e015a34abcdbd quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5 2024-09-20T22:41:48.403 INFO:journalctl@ceph.iscsi.iscsi.a.smithi154.stdout:Sep 20 22:41:48 smithi154 podman[59690]: 2024-09-20 22:41:48.341993579 +0000 UTC m=+0.129094696 container create 5a673a67abedc3e3b721b39df9309f1c8484cb4208530eb77544cf1480f20c7b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-iscsi-iscsi-a, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, RELEASE=squid-9e2ac70) 2024-09-20T22:41:48.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:48 smithi084 ceph-mon[29774]: from='client.24391 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi084=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:41:48.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:48 smithi084 ceph-mon[29774]: Saving service alertmanager spec with placement smithi084=a;count:1 2024-09-20T22:41:48.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:48 smithi084 ceph-mon[29774]: pgmap v199: 132 pgs: 5 creating+peering, 127 active+clean; 581 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 5.0 KiB/s wr, 141 op/s 2024-09-20T22:41:48.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:48 smithi084 ceph-mon[29774]: osdmap e75: 8 total, 8 up, 8 in 2024-09-20T22:41:48.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:48 smithi084 ceph-mon[40024]: from='client.24391 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi084=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:41:48.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:48 smithi084 ceph-mon[40024]: Saving service alertmanager spec with placement smithi084=a;count:1 2024-09-20T22:41:48.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:48 smithi084 ceph-mon[40024]: pgmap v199: 132 pgs: 5 creating+peering, 127 active+clean; 581 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 5.0 KiB/s wr, 141 op/s 2024-09-20T22:41:48.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:48 smithi084 ceph-mon[40024]: osdmap e75: 8 total, 8 up, 8 in 2024-09-20T22:41:48.662 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:48 smithi154 ceph-mon[33264]: from='client.24391 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi084=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:41:48.663 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:48 smithi154 ceph-mon[33264]: Saving service alertmanager spec with placement smithi084=a;count:1 2024-09-20T22:41:48.663 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:48 smithi154 ceph-mon[33264]: pgmap v199: 132 pgs: 5 creating+peering, 127 active+clean; 581 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 5.0 KiB/s wr, 141 op/s 2024-09-20T22:41:48.663 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:48 smithi154 ceph-mon[33264]: osdmap e75: 8 total, 8 up, 8 in 2024-09-20T22:41:48.663 INFO:journalctl@ceph.iscsi.iscsi.a.smithi154.stdout:Sep 20 22:41:48 smithi154 podman[59690]: 2024-09-20 22:41:48.56114575 +0000 UTC m=+0.348246877 container init 5a673a67abedc3e3b721b39df9309f1c8484cb4208530eb77544cf1480f20c7b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-iscsi-iscsi-a, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-9e2ac70, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD) 2024-09-20T22:41:48.663 INFO:journalctl@ceph.iscsi.iscsi.a.smithi154.stdout:Sep 20 22:41:48 smithi154 podman[59690]: 2024-09-20 22:41:48.564553473 +0000 UTC m=+0.351654592 container start 5a673a67abedc3e3b721b39df9309f1c8484cb4208530eb77544cf1480f20c7b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-iscsi-iscsi-a, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, RELEASE=squid-9e2ac70, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_BRANCH=HEAD) 2024-09-20T22:41:48.712 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:41:48.968 INFO:journalctl@ceph.iscsi.iscsi.a.smithi154.stdout:Sep 20 22:41:48 smithi154 bash[59690]: 5a673a67abedc3e3b721b39df9309f1c8484cb4208530eb77544cf1480f20c7b 2024-09-20T22:41:48.968 INFO:journalctl@ceph.iscsi.iscsi.a.smithi154.stdout:Sep 20 22:41:48 smithi154 systemd[1]: Started Ceph iscsi.iscsi.a for a9c0bc70-77a0-11ef-baf5-efdc52797490. 2024-09-20T22:41:48.968 INFO:journalctl@ceph.iscsi.iscsi.a.smithi154.stdout:Sep 20 22:41:48 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-iscsi-iscsi-a[59730]: debug Started the configuration object watcher 2024-09-20T22:41:48.968 INFO:journalctl@ceph.iscsi.iscsi.a.smithi154.stdout:Sep 20 22:41:48 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-iscsi-iscsi-a[59730]: debug Checking for config object changes every 1s 2024-09-20T22:41:48.968 INFO:journalctl@ceph.iscsi.iscsi.a.smithi154.stdout:Sep 20 22:41:48 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-iscsi-iscsi-a[59730]: debug Processing osd blocklist entries for this node 2024-09-20T22:41:49.249 INFO:journalctl@ceph.iscsi.iscsi.a.smithi154.stdout:Sep 20 22:41:49 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-iscsi-iscsi-a[59730]: debug Reading the configuration object to update local LIO configuration 2024-09-20T22:41:49.249 INFO:journalctl@ceph.iscsi.iscsi.a.smithi154.stdout:Sep 20 22:41:49 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-iscsi-iscsi-a[59730]: debug Configuration does not have an entry for this host(smithi154.front.sepia.ceph.com) - nothing to define to LIO 2024-09-20T22:41:49.249 INFO:journalctl@ceph.iscsi.iscsi.a.smithi154.stdout:Sep 20 22:41:49 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-iscsi-iscsi-a[59730]: * Serving Flask app 'rbd-target-api' (lazy loading) 2024-09-20T22:41:49.249 INFO:journalctl@ceph.iscsi.iscsi.a.smithi154.stdout:Sep 20 22:41:49 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-iscsi-iscsi-a[59730]: * Environment: production 2024-09-20T22:41:49.249 INFO:journalctl@ceph.iscsi.iscsi.a.smithi154.stdout:Sep 20 22:41:49 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-iscsi-iscsi-a[59730]: WARNING: This is a development server. Do not use it in a production deployment. 2024-09-20T22:41:49.249 INFO:journalctl@ceph.iscsi.iscsi.a.smithi154.stdout:Sep 20 22:41:49 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-iscsi-iscsi-a[59730]: Use a production WSGI server instead. 2024-09-20T22:41:49.249 INFO:journalctl@ceph.iscsi.iscsi.a.smithi154.stdout:Sep 20 22:41:49 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-iscsi-iscsi-a[59730]: * Debug mode: off 2024-09-20T22:41:49.249 INFO:journalctl@ceph.iscsi.iscsi.a.smithi154.stdout:Sep 20 22:41:49 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-iscsi-iscsi-a[59730]: debug * Running on all addresses. 2024-09-20T22:41:49.249 INFO:journalctl@ceph.iscsi.iscsi.a.smithi154.stdout:Sep 20 22:41:49 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-iscsi-iscsi-a[59730]: WARNING: This is a development server. Do not use it in a production deployment. 2024-09-20T22:41:49.250 INFO:journalctl@ceph.iscsi.iscsi.a.smithi154.stdout:Sep 20 22:41:49 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-iscsi-iscsi-a[59730]: * Running on all addresses. 2024-09-20T22:41:49.250 INFO:journalctl@ceph.iscsi.iscsi.a.smithi154.stdout:Sep 20 22:41:49 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-iscsi-iscsi-a[59730]: WARNING: This is a development server. Do not use it in a production deployment. 2024-09-20T22:41:49.250 INFO:journalctl@ceph.iscsi.iscsi.a.smithi154.stdout:Sep 20 22:41:49 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-iscsi-iscsi-a[59730]: debug * Running on http://[fe80::ec4:7aff:fe88:7761]:5000/ (Press CTRL+C to quit) 2024-09-20T22:41:49.250 INFO:journalctl@ceph.iscsi.iscsi.a.smithi154.stdout:Sep 20 22:41:49 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-iscsi-iscsi-a[59730]: * Running on http://[fe80::ec4:7aff:fe88:7761]:5000/ (Press CTRL+C to quit) 2024-09-20T22:41:49.670 INFO:teuthology.orchestra.run.smithi154.stdout:Scheduled grafana update... 2024-09-20T22:41:49.999 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:49 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:41:49.999 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:49 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:41:49.999 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:49 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:41:49.999 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:49 smithi154 ceph-mon[33264]: Checking pool "datapool" exists for service iscsi.datapool 2024-09-20T22:41:49.999 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:49 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:41:49.999 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:49 smithi154 ceph-mon[33264]: from='client.? 172.21.15.154:0/1488661058' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-20T22:41:49.999 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:49 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:41:50.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:49 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:41:50.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:49 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:41:50.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:49 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:41:50.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:49 smithi084 ceph-mon[29774]: Checking pool "datapool" exists for service iscsi.datapool 2024-09-20T22:41:50.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:49 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:41:50.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:49 smithi084 ceph-mon[29774]: from='client.? 172.21.15.154:0/1488661058' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-20T22:41:50.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:49 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:41:50.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:49 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:41:50.116 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:49 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:41:50.116 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:49 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:41:50.116 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:49 smithi084 ceph-mon[40024]: Checking pool "datapool" exists for service iscsi.datapool 2024-09-20T22:41:50.116 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:49 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:41:50.116 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:49 smithi084 ceph-mon[40024]: from='client.? 172.21.15.154:0/1488661058' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-20T22:41:50.116 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:49 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:41:50.322 DEBUG:teuthology.orchestra.run.smithi154:grafana.a> sudo journalctl -f -n 0 -u ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490@grafana.a.service 2024-09-20T22:41:50.324 INFO:tasks.cephadm:Setting up client nodes... 2024-09-20T22:41:50.325 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-09-20T22:41:50.578 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:41:50.950 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:50 smithi084 ceph-mon[40024]: Deploying daemon prometheus.a on smithi154 2024-09-20T22:41:50.950 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:50 smithi084 ceph-mon[40024]: from='client.24427 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi154=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:41:50.950 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:50 smithi084 ceph-mon[40024]: Saving service grafana spec with placement smithi154=a;count:1 2024-09-20T22:41:50.950 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:50 smithi084 ceph-mon[40024]: pgmap v201: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 6.0 KiB/s wr, 177 op/s 2024-09-20T22:41:50.950 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:50 smithi084 ceph-mon[29774]: Deploying daemon prometheus.a on smithi154 2024-09-20T22:41:50.951 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:50 smithi084 ceph-mon[29774]: from='client.24427 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi154=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:41:50.951 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:50 smithi084 ceph-mon[29774]: Saving service grafana spec with placement smithi154=a;count:1 2024-09-20T22:41:50.951 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:50 smithi084 ceph-mon[29774]: pgmap v201: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 6.0 KiB/s wr, 177 op/s 2024-09-20T22:41:51.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:50 smithi154 ceph-mon[33264]: Deploying daemon prometheus.a on smithi154 2024-09-20T22:41:51.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:50 smithi154 ceph-mon[33264]: from='client.24427 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi154=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T22:41:51.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:50 smithi154 ceph-mon[33264]: Saving service grafana spec with placement smithi154=a;count:1 2024-09-20T22:41:51.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:50 smithi154 ceph-mon[33264]: pgmap v201: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 6.0 KiB/s wr, 177 op/s 2024-09-20T22:41:51.284 INFO:teuthology.orchestra.run.smithi084.stdout:[client.0] 2024-09-20T22:41:51.284 INFO:teuthology.orchestra.run.smithi084.stdout: key = AQAv+u1mZAF7EBAAUrQ9v7Oa27r8Boey6VLmDg== 2024-09-20T22:41:51.971 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:51 smithi084 ceph-mon[40024]: mgrmap e16: y(active, since 4m), standbys: x 2024-09-20T22:41:51.971 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:51 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/2544582952' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-20T22:41:51.972 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:51 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-20T22:41:51.972 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:51 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-20T22:41:51.972 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:51 smithi084 ceph-mon[29774]: mgrmap e16: y(active, since 4m), standbys: x 2024-09-20T22:41:51.972 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:51 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/2544582952' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-20T22:41:51.972 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:51 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-20T22:41:51.972 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:51 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-20T22:41:51.973 DEBUG:teuthology.orchestra.run.smithi084:> set -ex 2024-09-20T22:41:51.973 DEBUG:teuthology.orchestra.run.smithi084:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-09-20T22:41:51.973 DEBUG:teuthology.orchestra.run.smithi084:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-09-20T22:41:52.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:51 smithi154 ceph-mon[33264]: mgrmap e16: y(active, since 4m), standbys: x 2024-09-20T22:41:52.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:51 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/2544582952' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-20T22:41:52.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:51 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-20T22:41:52.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:51 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-20T22:41:52.010 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-09-20T22:41:52.373 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:41:52.752 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:52 smithi154 ceph-mon[33264]: pgmap v202: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 61 KiB/s rd, 4.8 KiB/s wr, 149 op/s 2024-09-20T22:41:53.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:52 smithi084 ceph-mon[29774]: pgmap v202: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 61 KiB/s rd, 4.8 KiB/s wr, 149 op/s 2024-09-20T22:41:53.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:52 smithi084 ceph-mon[40024]: pgmap v202: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 61 KiB/s rd, 4.8 KiB/s wr, 149 op/s 2024-09-20T22:41:53.317 INFO:teuthology.orchestra.run.smithi154.stdout:[client.1] 2024-09-20T22:41:53.318 INFO:teuthology.orchestra.run.smithi154.stdout: key = AQAx+u1mvIirEhAApxozJQW9e2gd3Mlv4Z4ugQ== 2024-09-20T22:41:54.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:53 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:41:54.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:53 smithi154 ceph-mon[33264]: from='client.? 172.21.15.154:0/2524189072' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-20T22:41:54.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:53 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-20T22:41:54.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:53 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-20T22:41:54.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:53 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:41:54.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:53 smithi084 ceph-mon[29774]: from='client.? 172.21.15.154:0/2524189072' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-20T22:41:54.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:53 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-20T22:41:54.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:53 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-20T22:41:54.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:53 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:41:54.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:53 smithi084 ceph-mon[40024]: from='client.? 172.21.15.154:0/2524189072' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-20T22:41:54.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:53 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-20T22:41:54.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:53 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-20T22:41:54.409 DEBUG:teuthology.orchestra.run.smithi154:> set -ex 2024-09-20T22:41:54.410 DEBUG:teuthology.orchestra.run.smithi154:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-09-20T22:41:54.410 DEBUG:teuthology.orchestra.run.smithi154:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-09-20T22:41:54.447 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-20T22:41:54.448 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-09-20T22:41:54.448 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph mgr dump --format=json 2024-09-20T22:41:54.704 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:41:55.082 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:54 smithi084 ceph-mon[40024]: pgmap v203: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 4.3 KiB/s wr, 133 op/s 2024-09-20T22:41:55.082 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:54 smithi084 ceph-mon[29774]: pgmap v203: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 4.3 KiB/s wr, 133 op/s 2024-09-20T22:41:55.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:54 smithi154 ceph-mon[33264]: pgmap v203: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 4.3 KiB/s wr, 133 op/s 2024-09-20T22:41:55.386 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-20T22:41:56.052 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:55 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/2134963407' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-09-20T22:41:56.053 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:55 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/2134963407' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-09-20T22:41:56.080 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":16,"flags":0,"active_gid":14150,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6800","nonce":4024278029}]},"active_addr":"172.21.15.84:6800/4024278029","active_change":"2024-09-20T22:37:17.690247+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24106,"name":"x","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.84:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"cephadm","addrvec":[{"type":"v2","addr":"172.21.15.84:0","nonce":3353126955}]},{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.84:0","nonce":542020545}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.84:0","nonce":547902262}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.84:0","nonce":2963622459}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.84:0","nonce":1242929317}]}]} 2024-09-20T22:41:56.084 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-09-20T22:41:56.084 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-09-20T22:41:56.084 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd dump --format=json 2024-09-20T22:41:56.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:55 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/2134963407' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-09-20T22:41:56.339 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:41:57.015 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-20T22:41:57.015 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:56 smithi084 ceph-mon[29774]: pgmap v204: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 47 KiB/s rd, 3.5 KiB/s wr, 113 op/s 2024-09-20T22:41:57.015 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:56 smithi084 ceph-mon[40024]: pgmap v204: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 47 KiB/s rd, 3.5 KiB/s wr, 113 op/s 2024-09-20T22:41:57.016 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":75,"fsid":"a9c0bc70-77a0-11ef-baf5-efdc52797490","created":"2024-09-20T22:36:30.240626+0000","modified":"2024-09-20T22:41:48.081375+0000","last_up_change":"2024-09-20T22:41:27.098565+0000","last_in_change":"2024-09-20T22:41:06.226708+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-20T22:39:21.984381+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"datapool","create_time":"2024-09-20T22:41:35.820081+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":69,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":".rgw.root","create_time":"2024-09-20T22:41:36.546389+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-09-20T22:41:38.454640+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-09-20T22:41:40.407129+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"72","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-09-20T22:41:42.476716+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"74","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ea36a489-6f91-4b2f-b784-b90506b9805d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6801","nonce":268084626}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6802","nonce":268084626}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6804","nonce":268084626}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6803","nonce":268084626}]},"public_addr":"172.21.15.84:6801/268084626","cluster_addr":"172.21.15.84:6802/268084626","heartbeat_back_addr":"172.21.15.84:6804/268084626","heartbeat_front_addr":"172.21.15.84:6803/268084626","state":["exists","up"]},{"osd":1,"uuid":"c8649d2f-27ef-47f8-bc94-f1cd9e62bdb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6805","nonce":418374136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6806","nonce":418374136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6808","nonce":418374136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6807","nonce":418374136}]},"public_addr":"172.21.15.84:6805/418374136","cluster_addr":"172.21.15.84:6806/418374136","heartbeat_back_addr":"172.21.15.84:6808/418374136","heartbeat_front_addr":"172.21.15.84:6807/418374136","state":["exists","up"]},{"osd":2,"uuid":"c728eb48-a0f0-4de4-a651-7f3bec867280","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6809","nonce":3977168611}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6810","nonce":3977168611}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6812","nonce":3977168611}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6811","nonce":3977168611}]},"public_addr":"172.21.15.84:6809/3977168611","cluster_addr":"172.21.15.84:6810/3977168611","heartbeat_back_addr":"172.21.15.84:6812/3977168611","heartbeat_front_addr":"172.21.15.84:6811/3977168611","state":["exists","up"]},{"osd":3,"uuid":"573f44eb-0853-4630-a7d8-9caaeb256f9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6813","nonce":2192239945}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6814","nonce":2192239945}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6816","nonce":2192239945}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6815","nonce":2192239945}]},"public_addr":"172.21.15.84:6813/2192239945","cluster_addr":"172.21.15.84:6814/2192239945","heartbeat_back_addr":"172.21.15.84:6816/2192239945","heartbeat_front_addr":"172.21.15.84:6815/2192239945","state":["exists","up"]},{"osd":4,"uuid":"a52c3b94-c5f3-490f-8ac8-09061b9b2abb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":1793136002}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6801","nonce":1793136002}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6803","nonce":1793136002}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":1793136002}]},"public_addr":"172.21.15.154:6800/1793136002","cluster_addr":"172.21.15.154:6801/1793136002","heartbeat_back_addr":"172.21.15.154:6803/1793136002","heartbeat_front_addr":"172.21.15.154:6802/1793136002","state":["exists","up"]},{"osd":5,"uuid":"4fa0dc8e-25cc-46cd-96ff-33f20b6faec3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":493899927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6805","nonce":493899927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6807","nonce":493899927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":493899927}]},"public_addr":"172.21.15.154:6804/493899927","cluster_addr":"172.21.15.154:6805/493899927","heartbeat_back_addr":"172.21.15.154:6807/493899927","heartbeat_front_addr":"172.21.15.154:6806/493899927","state":["exists","up"]},{"osd":6,"uuid":"56788d74-810e-4827-a615-958c12f8a368","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":2375091033}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6809","nonce":2375091033}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6811","nonce":2375091033}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":2375091033}]},"public_addr":"172.21.15.154:6808/2375091033","cluster_addr":"172.21.15.154:6809/2375091033","heartbeat_back_addr":"172.21.15.154:6811/2375091033","heartbeat_front_addr":"172.21.15.154:6810/2375091033","state":["exists","up"]},{"osd":7,"uuid":"83503727-afc1-4b95-8b96-ecb63f3338c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":3717704491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6813","nonce":3717704491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6815","nonce":3717704491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":3717704491}]},"public_addr":"172.21.15.154:6812/3717704491","cluster_addr":"172.21.15.154:6813/3717704491","heartbeat_back_addr":"172.21.15.154:6815/3717704491","heartbeat_front_addr":"172.21.15.154:6814/3717704491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T22:41:09.485558+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T22:41:14.497933+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T22:41:33.307669+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T22:41:01.300651+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T22:41:10.316534+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T22:40:33.503712+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T22:40:59.192804+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T22:41:25.354165+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.84:0/1361143214":"2024-09-21T22:37:17.690104+0000","172.21.15.84:0/3879263635":"2024-09-21T22:37:17.690104+0000","172.21.15.84:0/1176849276":"2024-09-21T22:37:17.690104+0000","172.21.15.84:0/567614908":"2024-09-21T22:36:54.917812+0000","172.21.15.84:6800/2932420050":"2024-09-21T22:37:17.690104+0000","172.21.15.84:6800/820908470":"2024-09-21T22:36:54.917812+0000","172.21.15.84:0/25873236":"2024-09-21T22:36:54.917812+0000","172.21.15.84:0/969472279":"2024-09-21T22:36:54.917812+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-20T22:41:57.155 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:56 smithi154 ceph-mon[33264]: pgmap v204: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 47 KiB/s rd, 3.5 KiB/s wr, 113 op/s 2024-09-20T22:41:57.689 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-09-20T22:41:57.689 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd dump --format=json 2024-09-20T22:41:57.938 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:41:58.059 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:57 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/2623189326' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-20T22:41:58.059 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:57 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/2623189326' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-20T22:41:58.208 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:57 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/2623189326' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-20T22:41:58.555 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-20T22:41:58.555 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":75,"fsid":"a9c0bc70-77a0-11ef-baf5-efdc52797490","created":"2024-09-20T22:36:30.240626+0000","modified":"2024-09-20T22:41:48.081375+0000","last_up_change":"2024-09-20T22:41:27.098565+0000","last_in_change":"2024-09-20T22:41:06.226708+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-20T22:39:21.984381+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"datapool","create_time":"2024-09-20T22:41:35.820081+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":69,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":".rgw.root","create_time":"2024-09-20T22:41:36.546389+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-09-20T22:41:38.454640+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-09-20T22:41:40.407129+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"72","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-09-20T22:41:42.476716+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"74","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ea36a489-6f91-4b2f-b784-b90506b9805d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6801","nonce":268084626}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6802","nonce":268084626}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6804","nonce":268084626}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6803","nonce":268084626}]},"public_addr":"172.21.15.84:6801/268084626","cluster_addr":"172.21.15.84:6802/268084626","heartbeat_back_addr":"172.21.15.84:6804/268084626","heartbeat_front_addr":"172.21.15.84:6803/268084626","state":["exists","up"]},{"osd":1,"uuid":"c8649d2f-27ef-47f8-bc94-f1cd9e62bdb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6805","nonce":418374136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6806","nonce":418374136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6808","nonce":418374136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6807","nonce":418374136}]},"public_addr":"172.21.15.84:6805/418374136","cluster_addr":"172.21.15.84:6806/418374136","heartbeat_back_addr":"172.21.15.84:6808/418374136","heartbeat_front_addr":"172.21.15.84:6807/418374136","state":["exists","up"]},{"osd":2,"uuid":"c728eb48-a0f0-4de4-a651-7f3bec867280","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6809","nonce":3977168611}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6810","nonce":3977168611}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6812","nonce":3977168611}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6811","nonce":3977168611}]},"public_addr":"172.21.15.84:6809/3977168611","cluster_addr":"172.21.15.84:6810/3977168611","heartbeat_back_addr":"172.21.15.84:6812/3977168611","heartbeat_front_addr":"172.21.15.84:6811/3977168611","state":["exists","up"]},{"osd":3,"uuid":"573f44eb-0853-4630-a7d8-9caaeb256f9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6813","nonce":2192239945}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6814","nonce":2192239945}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6816","nonce":2192239945}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6815","nonce":2192239945}]},"public_addr":"172.21.15.84:6813/2192239945","cluster_addr":"172.21.15.84:6814/2192239945","heartbeat_back_addr":"172.21.15.84:6816/2192239945","heartbeat_front_addr":"172.21.15.84:6815/2192239945","state":["exists","up"]},{"osd":4,"uuid":"a52c3b94-c5f3-490f-8ac8-09061b9b2abb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":1793136002}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6801","nonce":1793136002}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6803","nonce":1793136002}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":1793136002}]},"public_addr":"172.21.15.154:6800/1793136002","cluster_addr":"172.21.15.154:6801/1793136002","heartbeat_back_addr":"172.21.15.154:6803/1793136002","heartbeat_front_addr":"172.21.15.154:6802/1793136002","state":["exists","up"]},{"osd":5,"uuid":"4fa0dc8e-25cc-46cd-96ff-33f20b6faec3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":493899927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6805","nonce":493899927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6807","nonce":493899927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":493899927}]},"public_addr":"172.21.15.154:6804/493899927","cluster_addr":"172.21.15.154:6805/493899927","heartbeat_back_addr":"172.21.15.154:6807/493899927","heartbeat_front_addr":"172.21.15.154:6806/493899927","state":["exists","up"]},{"osd":6,"uuid":"56788d74-810e-4827-a615-958c12f8a368","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":2375091033}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6809","nonce":2375091033}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6811","nonce":2375091033}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":2375091033}]},"public_addr":"172.21.15.154:6808/2375091033","cluster_addr":"172.21.15.154:6809/2375091033","heartbeat_back_addr":"172.21.15.154:6811/2375091033","heartbeat_front_addr":"172.21.15.154:6810/2375091033","state":["exists","up"]},{"osd":7,"uuid":"83503727-afc1-4b95-8b96-ecb63f3338c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":3717704491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6813","nonce":3717704491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6815","nonce":3717704491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":3717704491}]},"public_addr":"172.21.15.154:6812/3717704491","cluster_addr":"172.21.15.154:6813/3717704491","heartbeat_back_addr":"172.21.15.154:6815/3717704491","heartbeat_front_addr":"172.21.15.154:6814/3717704491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T22:41:09.485558+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T22:41:14.497933+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T22:41:33.307669+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T22:41:01.300651+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T22:41:10.316534+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T22:40:33.503712+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T22:40:59.192804+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T22:41:25.354165+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.84:0/1361143214":"2024-09-21T22:37:17.690104+0000","172.21.15.84:0/3879263635":"2024-09-21T22:37:17.690104+0000","172.21.15.84:0/1176849276":"2024-09-21T22:37:17.690104+0000","172.21.15.84:0/567614908":"2024-09-21T22:36:54.917812+0000","172.21.15.84:6800/2932420050":"2024-09-21T22:37:17.690104+0000","172.21.15.84:6800/820908470":"2024-09-21T22:36:54.917812+0000","172.21.15.84:0/25873236":"2024-09-21T22:36:54.917812+0000","172.21.15.84:0/969472279":"2024-09-21T22:36:54.917812+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-20T22:41:59.040 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:58 smithi154 ceph-mon[33264]: pgmap v205: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 921 B/s wr, 35 op/s 2024-09-20T22:41:59.040 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:58 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:41:59.040 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:58 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/3860518151' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-20T22:41:59.041 INFO:journalctl@ceph.iscsi.iscsi.a.smithi154.stdout:Sep 20 22:41:58 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-iscsi-iscsi-a[59730]: debug there is no tcmu-runner data available 2024-09-20T22:41:59.114 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:58 smithi084 ceph-mon[40024]: pgmap v205: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 921 B/s wr, 35 op/s 2024-09-20T22:41:59.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:58 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:41:59.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:58 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/3860518151' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-20T22:41:59.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:58 smithi084 ceph-mon[29774]: pgmap v205: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 921 B/s wr, 35 op/s 2024-09-20T22:41:59.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:58 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:41:59.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:58 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/3860518151' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-20T22:41:59.244 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph tell osd.0 flush_pg_stats 2024-09-20T22:41:59.245 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph tell osd.1 flush_pg_stats 2024-09-20T22:41:59.245 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph tell osd.2 flush_pg_stats 2024-09-20T22:41:59.246 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph tell osd.3 flush_pg_stats 2024-09-20T22:41:59.246 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph tell osd.4 flush_pg_stats 2024-09-20T22:41:59.247 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph tell osd.5 flush_pg_stats 2024-09-20T22:41:59.247 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph tell osd.6 flush_pg_stats 2024-09-20T22:41:59.247 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph tell osd.7 flush_pg_stats 2024-09-20T22:41:59.733 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:41:59.770 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:41:59.771 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:41:59.775 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:41:59.776 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:41:59.779 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:41:59.788 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:42:00.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:41:59 smithi154 ceph-mon[33264]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:42:00.036 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:41:59 smithi084 ceph-mon[40024]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:42:00.036 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:41:59 smithi084 ceph-mon[29774]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:42:00.421 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:42:01.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:00 smithi154 ceph-mon[33264]: pgmap v206: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 790 B/s wr, 30 op/s 2024-09-20T22:42:01.004 INFO:journalctl@ceph.prometheus.a.smithi154.stdout:Sep 20 22:42:00 smithi154 systemd[1]: Starting Ceph prometheus.a for a9c0bc70-77a0-11ef-baf5-efdc52797490... 2024-09-20T22:42:01.062 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:00 smithi084 ceph-mon[40024]: pgmap v206: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 790 B/s wr, 30 op/s 2024-09-20T22:42:01.062 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:00 smithi084 ceph-mon[29774]: pgmap v206: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 790 B/s wr, 30 op/s 2024-09-20T22:42:01.339 INFO:journalctl@ceph.prometheus.a.smithi154.stdout:Sep 20 22:42:01 smithi154 podman[60619]: 2024-09-20 22:42:00.985873935 +0000 UTC m=+0.017270588 image pull a07b618ecd1dce142bce4c52f0e80982eaf1f14265a2415c2d35978ccaa0a464 quay.io/prometheus/prometheus:v2.43.0 2024-09-20T22:42:01.340 INFO:journalctl@ceph.prometheus.a.smithi154.stdout:Sep 20 22:42:01 smithi154 podman[60619]: 2024-09-20 22:42:01.099971507 +0000 UTC m=+0.131368189 container create 19b4ae16e0bb92baebbe2bd7cf4786b7d7635d14bec4196ae1f63b5c04734830 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-20T22:42:01.340 INFO:journalctl@ceph.prometheus.a.smithi154.stdout:Sep 20 22:42:01 smithi154 podman[60619]: 2024-09-20 22:42:01.310322238 +0000 UTC m=+0.341718880 container init 19b4ae16e0bb92baebbe2bd7cf4786b7d7635d14bec4196ae1f63b5c04734830 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-20T22:42:01.340 INFO:journalctl@ceph.prometheus.a.smithi154.stdout:Sep 20 22:42:01 smithi154 podman[60619]: 2024-09-20 22:42:01.313149794 +0000 UTC m=+0.344546434 container start 19b4ae16e0bb92baebbe2bd7cf4786b7d7635d14bec4196ae1f63b5c04734830 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-20T22:42:01.559 INFO:teuthology.orchestra.run.smithi084.stdout:55834574888 2024-09-20T22:42:01.560 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd last-stat-seq osd.1 2024-09-20T22:42:01.755 INFO:journalctl@ceph.prometheus.a.smithi154.stdout:Sep 20 22:42:01 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-prometheus-a[60633]: ts=2024-09-20T22:42:01.338Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-09-20T22:42:01.755 INFO:journalctl@ceph.prometheus.a.smithi154.stdout:Sep 20 22:42:01 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-prometheus-a[60633]: ts=2024-09-20T22:42:01.338Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-09-20T22:42:01.755 INFO:journalctl@ceph.prometheus.a.smithi154.stdout:Sep 20 22:42:01 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-prometheus-a[60633]: ts=2024-09-20T22:42:01.338Z caller=main.go:570 level=info host_details="(Linux 5.14.0-508.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024 x86_64 smithi154 (none))" 2024-09-20T22:42:01.755 INFO:journalctl@ceph.prometheus.a.smithi154.stdout:Sep 20 22:42:01 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-prometheus-a[60633]: ts=2024-09-20T22:42:01.338Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-09-20T22:42:01.755 INFO:journalctl@ceph.prometheus.a.smithi154.stdout:Sep 20 22:42:01 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-prometheus-a[60633]: ts=2024-09-20T22:42:01.338Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-09-20T22:42:01.755 INFO:journalctl@ceph.prometheus.a.smithi154.stdout:Sep 20 22:42:01 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-prometheus-a[60633]: ts=2024-09-20T22:42:01.339Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-09-20T22:42:01.755 INFO:journalctl@ceph.prometheus.a.smithi154.stdout:Sep 20 22:42:01 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-prometheus-a[60633]: ts=2024-09-20T22:42:01.340Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-09-20T22:42:01.755 INFO:journalctl@ceph.prometheus.a.smithi154.stdout:Sep 20 22:42:01 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-prometheus-a[60633]: ts=2024-09-20T22:42:01.340Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-09-20T22:42:01.755 INFO:journalctl@ceph.prometheus.a.smithi154.stdout:Sep 20 22:42:01 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-prometheus-a[60633]: ts=2024-09-20T22:42:01.340Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-09-20T22:42:01.755 INFO:journalctl@ceph.prometheus.a.smithi154.stdout:Sep 20 22:42:01 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-prometheus-a[60633]: ts=2024-09-20T22:42:01.343Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-09-20T22:42:01.755 INFO:journalctl@ceph.prometheus.a.smithi154.stdout:Sep 20 22:42:01 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-prometheus-a[60633]: ts=2024-09-20T22:42:01.343Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=2.59µs 2024-09-20T22:42:01.756 INFO:journalctl@ceph.prometheus.a.smithi154.stdout:Sep 20 22:42:01 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-prometheus-a[60633]: ts=2024-09-20T22:42:01.343Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-09-20T22:42:01.756 INFO:journalctl@ceph.prometheus.a.smithi154.stdout:Sep 20 22:42:01 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-prometheus-a[60633]: ts=2024-09-20T22:42:01.343Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0 2024-09-20T22:42:01.756 INFO:journalctl@ceph.prometheus.a.smithi154.stdout:Sep 20 22:42:01 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-prometheus-a[60633]: ts=2024-09-20T22:42:01.343Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=26.653µs wal_replay_duration=286.327µs wbl_replay_duration=126ns total_replay_duration=338.704µs 2024-09-20T22:42:01.756 INFO:journalctl@ceph.prometheus.a.smithi154.stdout:Sep 20 22:42:01 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-prometheus-a[60633]: ts=2024-09-20T22:42:01.344Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-09-20T22:42:01.756 INFO:journalctl@ceph.prometheus.a.smithi154.stdout:Sep 20 22:42:01 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-prometheus-a[60633]: ts=2024-09-20T22:42:01.344Z caller=main.go:1029 level=info msg="TSDB started" 2024-09-20T22:42:01.756 INFO:journalctl@ceph.prometheus.a.smithi154.stdout:Sep 20 22:42:01 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-prometheus-a[60633]: ts=2024-09-20T22:42:01.344Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-09-20T22:42:01.756 INFO:journalctl@ceph.prometheus.a.smithi154.stdout:Sep 20 22:42:01 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-prometheus-a[60633]: ts=2024-09-20T22:42:01.365Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=20.604232ms db_storage=759ns remote_storage=943ns web_handler=380ns query_engine=507ns scrape=464.856µs scrape_sd=93.988µs notify=800ns notify_sd=1.328µs rules=19.774287ms tracing=5.674µs 2024-09-20T22:42:01.756 INFO:journalctl@ceph.prometheus.a.smithi154.stdout:Sep 20 22:42:01 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-prometheus-a[60633]: ts=2024-09-20T22:42:01.365Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-09-20T22:42:01.756 INFO:journalctl@ceph.prometheus.a.smithi154.stdout:Sep 20 22:42:01 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-prometheus-a[60633]: ts=2024-09-20T22:42:01.365Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-09-20T22:42:01.756 INFO:journalctl@ceph.prometheus.a.smithi154.stdout:Sep 20 22:42:01 smithi154 bash[60619]: 19b4ae16e0bb92baebbe2bd7cf4786b7d7635d14bec4196ae1f63b5c04734830 2024-09-20T22:42:01.756 INFO:journalctl@ceph.prometheus.a.smithi154.stdout:Sep 20 22:42:01 smithi154 systemd[1]: Started Ceph prometheus.a for a9c0bc70-77a0-11ef-baf5-efdc52797490. 2024-09-20T22:42:02.617 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:02 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:42:02.617 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:02 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:42:02.617 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:02 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:42:02.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:02 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-09-20T22:42:02.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:02 smithi084 ceph-mon[29774]: pgmap v207: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 3.1 KiB/s rd, 85 B/s wr, 6 op/s 2024-09-20T22:42:02.618 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:42:02 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: ignoring --setuser ceph since I am not root 2024-09-20T22:42:02.618 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:42:02 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: ignoring --setgroup ceph since I am not root 2024-09-20T22:42:02.618 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:02 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:42:02.618 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:02 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:42:02.619 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:02 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:42:02.619 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:02 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-09-20T22:42:02.619 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:02 smithi084 ceph-mon[40024]: pgmap v207: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 3.1 KiB/s rd, 85 B/s wr, 6 op/s 2024-09-20T22:42:02.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:02 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:42:02.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:02 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:42:02.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:02 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' 2024-09-20T22:42:02.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:02 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-09-20T22:42:02.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:02 smithi154 ceph-mon[33264]: pgmap v207: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 3.1 KiB/s rd, 85 B/s wr, 6 op/s 2024-09-20T22:42:02.755 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Sep 20 22:42:02 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-x[35077]: ignoring --setuser ceph since I am not root 2024-09-20T22:42:02.755 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Sep 20 22:42:02 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-x[35077]: ignoring --setgroup ceph since I am not root 2024-09-20T22:42:02.755 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Sep 20 22:42:02 smithi154 ceph-mgr[35081]: -- 172.21.15.154:0/4293827548 <== mon.1 v2:172.21.15.154:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x557b39e5f0e0 con 0x557b39e3d400 2024-09-20T22:42:02.755 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Sep 20 22:42:02 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-x[35077]: 2024-09-20T22:42:02.656+0000 7fe721041040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-20T22:42:02.755 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Sep 20 22:42:02 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-x[35077]: 2024-09-20T22:42:02.712+0000 7fe721041040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-20T22:42:02.925 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:42:02 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: 2024-09-20T22:42:02.658+0000 7fd61afd7040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-20T22:42:02.925 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:42:02 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: 2024-09-20T22:42:02.716+0000 7fd61afd7040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-20T22:42:02.925 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:42:02 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: 2024-09-20T22:42:02.865+0000 7fd61afd7040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-20T22:42:02.954 INFO:teuthology.orchestra.run.smithi084.stdout:133143986201 2024-09-20T22:42:02.954 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd last-stat-seq osd.4 2024-09-20T22:42:03.220 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Sep 20 22:42:02 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-x[35077]: 2024-09-20T22:42:02.858+0000 7fe721041040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-20T22:42:03.220 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Sep 20 22:42:02 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-x[35077]: 2024-09-20T22:42:02.926+0000 7fe721041040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-20T22:42:03.221 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Sep 20 22:42:03 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-x[35077]: 2024-09-20T22:42:03.073+0000 7fe721041040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-20T22:42:03.247 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:42:02 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: 2024-09-20T22:42:02.933+0000 7fd61afd7040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-20T22:42:03.248 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:42:03 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: 2024-09-20T22:42:03.095+0000 7fd61afd7040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-20T22:42:03.248 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:42:03 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: 2024-09-20T22:42:03.245+0000 7fd61afd7040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-20T22:42:03.504 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Sep 20 22:42:03 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-x[35077]: 2024-09-20T22:42:03.218+0000 7fe721041040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-20T22:42:03.777 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:03 smithi154 ceph-mon[33264]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-09-20T22:42:03.777 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:03 smithi154 ceph-mon[33264]: mgrmap e17: y(active, since 4m), standbys: x 2024-09-20T22:42:03.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Sep 20 22:42:03 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-x[35077]: 2024-09-20T22:42:03.531+0000 7fe721041040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-20T22:42:03.777 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Sep 20 22:42:03 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-x[35077]: 2024-09-20T22:42:03.592+0000 7fe721041040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-20T22:42:03.778 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Sep 20 22:42:03 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-x[35077]: 2024-09-20T22:42:03.775+0000 7fe721041040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-20T22:42:03.808 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:03 smithi084 ceph-mon[29774]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-09-20T22:42:03.808 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:03 smithi084 ceph-mon[29774]: mgrmap e17: y(active, since 4m), standbys: x 2024-09-20T22:42:03.808 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:42:03 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: 2024-09-20T22:42:03.577+0000 7fd61afd7040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-20T22:42:03.808 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:42:03 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: 2024-09-20T22:42:03.643+0000 7fd61afd7040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-20T22:42:03.808 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:03 smithi084 ceph-mon[40024]: from='mgr.14150 172.21.15.84:0/4149909046' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-09-20T22:42:03.808 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:03 smithi084 ceph-mon[40024]: mgrmap e17: y(active, since 4m), standbys: x 2024-09-20T22:42:04.060 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:42:03 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: 2024-09-20T22:42:03.848+0000 7fd61afd7040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-20T22:42:04.732 INFO:teuthology.orchestra.run.smithi084.stdout:34359738414 2024-09-20T22:42:04.733 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd last-stat-seq osd.0 2024-09-20T22:42:04.794 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:04 smithi084 ceph-mon[40024]: purged_snaps scrub starts 2024-09-20T22:42:04.795 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:04 smithi084 ceph-mon[40024]: purged_snaps scrub ok 2024-09-20T22:42:04.795 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:04 smithi084 ceph-mon[40024]: osdmap e76: 8 total, 8 up, 8 in 2024-09-20T22:42:04.795 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:04 smithi084 ceph-mon[29774]: purged_snaps scrub starts 2024-09-20T22:42:04.795 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:04 smithi084 ceph-mon[29774]: purged_snaps scrub ok 2024-09-20T22:42:04.795 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:04 smithi084 ceph-mon[29774]: osdmap e76: 8 total, 8 up, 8 in 2024-09-20T22:42:05.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:04 smithi154 ceph-mon[33264]: purged_snaps scrub starts 2024-09-20T22:42:05.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:04 smithi154 ceph-mon[33264]: purged_snaps scrub ok 2024-09-20T22:42:05.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:04 smithi154 ceph-mon[33264]: osdmap e76: 8 total, 8 up, 8 in 2024-09-20T22:42:05.004 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Sep 20 22:42:04 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-x[35077]: 2024-09-20T22:42:04.681+0000 7fe721041040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-20T22:42:05.004 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Sep 20 22:42:04 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-x[35077]: 2024-09-20T22:42:04.748+0000 7fe721041040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-20T22:42:05.114 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:42:04 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: 2024-09-20T22:42:04.792+0000 7fd61afd7040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-20T22:42:05.115 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:42:04 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: 2024-09-20T22:42:04.863+0000 7fd61afd7040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-20T22:42:05.712 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Sep 20 22:42:05 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-x[35077]: 2024-09-20T22:42:05.451+0000 7fe721041040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-20T22:42:05.712 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Sep 20 22:42:05 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-x[35077]: 2024-09-20T22:42:05.567+0000 7fe721041040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-20T22:42:05.862 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:42:05 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: 2024-09-20T22:42:05.586+0000 7fd61afd7040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-20T22:42:05.862 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:42:05 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: 2024-09-20T22:42:05.709+0000 7fd61afd7040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-20T22:42:05.990 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Sep 20 22:42:05 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-x[35077]: 2024-09-20T22:42:05.710+0000 7fe721041040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-20T22:42:05.990 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Sep 20 22:42:05 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-x[35077]: 2024-09-20T22:42:05.771+0000 7fe721041040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-20T22:42:05.990 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Sep 20 22:42:05 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-x[35077]: 2024-09-20T22:42:05.920+0000 7fe721041040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-20T22:42:05.990 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Sep 20 22:42:05 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-x[35077]: 2024-09-20T22:42:05.988+0000 7fe721041040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-20T22:42:06.040 INFO:teuthology.orchestra.run.smithi084.stdout:206158430223 2024-09-20T22:42:06.041 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd last-stat-seq osd.6 2024-09-20T22:42:06.115 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:42:05 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: 2024-09-20T22:42:05.860+0000 7fd61afd7040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-20T22:42:06.115 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:42:05 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: 2024-09-20T22:42:05.924+0000 7fd61afd7040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-20T22:42:06.115 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:42:06 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: 2024-09-20T22:42:06.093+0000 7fd61afd7040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-20T22:42:06.254 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Sep 20 22:42:06 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-x[35077]: 2024-09-20T22:42:06.047+0000 7fe721041040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-20T22:42:06.254 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Sep 20 22:42:06 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-x[35077]: 2024-09-20T22:42:06.209+0000 7fe721041040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-20T22:42:06.399 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:42:06 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: 2024-09-20T22:42:06.167+0000 7fd61afd7040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-20T22:42:06.399 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:42:06 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: 2024-09-20T22:42:06.234+0000 7fd61afd7040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-20T22:42:06.399 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:42:06 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: 2024-09-20T22:42:06.397+0000 7fd61afd7040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-20T22:42:06.489 INFO:teuthology.orchestra.run.smithi084.stdout:171798691860 2024-09-20T22:42:06.489 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd last-stat-seq osd.5 2024-09-20T22:42:06.540 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Sep 20 22:42:06 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-x[35077]: 2024-09-20T22:42:06.274+0000 7fe721041040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-20T22:42:06.540 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Sep 20 22:42:06 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-x[35077]: 2024-09-20T22:42:06.340+0000 7fe721041040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-20T22:42:06.540 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Sep 20 22:42:06 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-x[35077]: 2024-09-20T22:42:06.421+0000 7fe721041040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-20T22:42:06.540 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Sep 20 22:42:06 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-x[35077]: 2024-09-20T22:42:06.480+0000 7fe721041040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-20T22:42:06.541 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Sep 20 22:42:06 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-x[35077]: 2024-09-20T22:42:06.538+0000 7fe721041040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-20T22:42:06.670 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:42:06 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: 2024-09-20T22:42:06.464+0000 7fd61afd7040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-20T22:42:06.670 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:42:06 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: 2024-09-20T22:42:06.536+0000 7fd61afd7040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-20T22:42:06.670 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:42:06 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: 2024-09-20T22:42:06.624+0000 7fd61afd7040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-20T22:42:07.017 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:42:06 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: 2024-09-20T22:42:06.697+0000 7fd61afd7040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-20T22:42:07.017 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:42:06 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: 2024-09-20T22:42:06.764+0000 7fd61afd7040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-20T22:42:07.045 INFO:teuthology.orchestra.run.smithi084.stdout:77309411364 2024-09-20T22:42:07.045 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd last-stat-seq osd.2 2024-09-20T22:42:07.615 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:42:07 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: 2024-09-20T22:42:07.593+0000 7fd61afd7040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-20T22:42:07.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:07 smithi084 ceph-mon[29774]: Standby manager daemon x restarted 2024-09-20T22:42:07.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:07 smithi084 ceph-mon[29774]: Standby manager daemon x started 2024-09-20T22:42:07.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:07 smithi084 ceph-mon[29774]: from='mgr.? 172.21.15.154:0/3317413327' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-20T22:42:07.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:07 smithi084 ceph-mon[29774]: from='mgr.? 172.21.15.154:0/3317413327' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-20T22:42:07.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:07 smithi084 ceph-mon[29774]: from='mgr.? 172.21.15.154:0/3317413327' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-20T22:42:07.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:07 smithi084 ceph-mon[29774]: from='mgr.? 172.21.15.154:0/3317413327' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-20T22:42:07.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:07 smithi084 ceph-mon[40024]: Standby manager daemon x restarted 2024-09-20T22:42:07.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:07 smithi084 ceph-mon[40024]: Standby manager daemon x started 2024-09-20T22:42:07.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:07 smithi084 ceph-mon[40024]: from='mgr.? 172.21.15.154:0/3317413327' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-20T22:42:07.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:07 smithi084 ceph-mon[40024]: from='mgr.? 172.21.15.154:0/3317413327' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-20T22:42:07.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:07 smithi084 ceph-mon[40024]: from='mgr.? 172.21.15.154:0/3317413327' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-20T22:42:07.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:07 smithi084 ceph-mon[40024]: from='mgr.? 172.21.15.154:0/3317413327' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-20T22:42:07.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:07 smithi154 ceph-mon[33264]: Standby manager daemon x restarted 2024-09-20T22:42:07.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:07 smithi154 ceph-mon[33264]: Standby manager daemon x started 2024-09-20T22:42:07.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:07 smithi154 ceph-mon[33264]: from='mgr.? 172.21.15.154:0/3317413327' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-20T22:42:07.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:07 smithi154 ceph-mon[33264]: from='mgr.? 172.21.15.154:0/3317413327' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-20T22:42:07.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:07 smithi154 ceph-mon[33264]: from='mgr.? 172.21.15.154:0/3317413327' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-20T22:42:07.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:07 smithi154 ceph-mon[33264]: from='mgr.? 172.21.15.154:0/3317413327' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-20T22:42:07.755 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Sep 20 22:42:07 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-x[35077]: 2024-09-20T22:42:07.320+0000 7fe721041040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-20T22:42:07.755 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Sep 20 22:42:07 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-x[35077]: [20/Sep/2024:22:42:07] ENGINE Bus STARTING 2024-09-20T22:42:07.755 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Sep 20 22:42:07 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-x[35077]: CherryPy Checker: 2024-09-20T22:42:07.755 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Sep 20 22:42:07 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-x[35077]: The Application mounted at '' has an empty config. 2024-09-20T22:42:07.755 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Sep 20 22:42:07 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-x[35077]: 2024-09-20T22:42:07.755 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Sep 20 22:42:07 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-x[35077]: [20/Sep/2024:22:42:07] ENGINE Serving on http://:::9283 2024-09-20T22:42:07.755 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Sep 20 22:42:07 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-x[35077]: [20/Sep/2024:22:42:07] ENGINE Bus STARTED 2024-09-20T22:42:07.962 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:42:07 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: [20/Sep/2024:22:42:07] ENGINE Bus STARTING 2024-09-20T22:42:07.962 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:42:07 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: CherryPy Checker: 2024-09-20T22:42:07.963 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:42:07 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: The Application mounted at '' has an empty config. 2024-09-20T22:42:07.963 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:42:07 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: 2024-09-20T22:42:07.963 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:42:07 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: [20/Sep/2024:22:42:07] ENGINE Serving on http://:::9283 2024-09-20T22:42:07.963 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:42:07 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: [20/Sep/2024:22:42:07] ENGINE Bus STARTED 2024-09-20T22:42:08.614 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:42:08 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: ::ffff:172.21.15.154 - - [20/Sep/2024:22:42:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-20T22:42:08.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:08 smithi084 ceph-mon[40024]: mgrmap e18: y(active, since 4m), standbys: x 2024-09-20T22:42:08.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:08 smithi084 ceph-mon[40024]: Active manager daemon y restarted 2024-09-20T22:42:08.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:08 smithi084 ceph-mon[40024]: Activating manager daemon y 2024-09-20T22:42:08.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:08 smithi084 ceph-mon[40024]: osdmap e77: 8 total, 8 up, 8 in 2024-09-20T22:42:08.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:08 smithi084 ceph-mon[40024]: mgrmap e19: y(active, starting, since 0.00833649s), standbys: x 2024-09-20T22:42:08.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:08 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-20T22:42:08.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:08 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T22:42:08.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:08 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T22:42:08.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:08 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-20T22:42:08.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:08 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-20T22:42:08.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:08 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-20T22:42:08.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:08 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-20T22:42:08.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:08 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-20T22:42:08.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:08 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T22:42:08.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:08 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-20T22:42:08.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:08 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-20T22:42:08.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:08 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-20T22:42:08.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:08 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-20T22:42:08.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:08 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-20T22:42:08.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:08 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-20T22:42:08.617 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:08 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-20T22:42:08.617 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:08 smithi084 ceph-mon[40024]: Manager daemon y is now available 2024-09-20T22:42:08.617 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:08 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:42:08.617 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:08 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:42:08.617 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:08 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:42:08.617 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:08 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-20T22:42:08.617 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:08 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-20T22:42:08.617 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:08 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-20T22:42:08.617 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:08 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-20T22:42:08.617 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:08 smithi084 ceph-mon[29774]: mgrmap e18: y(active, since 4m), standbys: x 2024-09-20T22:42:08.617 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:08 smithi084 ceph-mon[29774]: Active manager daemon y restarted 2024-09-20T22:42:08.617 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:08 smithi084 ceph-mon[29774]: Activating manager daemon y 2024-09-20T22:42:08.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:08 smithi084 ceph-mon[29774]: osdmap e77: 8 total, 8 up, 8 in 2024-09-20T22:42:08.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:08 smithi084 ceph-mon[29774]: mgrmap e19: y(active, starting, since 0.00833649s), standbys: x 2024-09-20T22:42:08.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:08 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-20T22:42:08.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:08 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T22:42:08.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:08 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T22:42:08.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:08 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-20T22:42:08.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:08 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-20T22:42:08.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:08 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-20T22:42:08.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:08 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-20T22:42:08.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:08 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-20T22:42:08.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:08 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T22:42:08.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:08 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-20T22:42:08.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:08 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-20T22:42:08.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:08 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-20T22:42:08.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:08 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-20T22:42:08.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:08 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-20T22:42:08.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:08 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-20T22:42:08.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:08 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-20T22:42:08.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:08 smithi084 ceph-mon[29774]: Manager daemon y is now available 2024-09-20T22:42:08.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:08 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:42:08.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:08 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:42:08.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:08 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:42:08.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:08 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-20T22:42:08.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:08 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-20T22:42:08.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:08 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-20T22:42:08.619 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:08 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-20T22:42:08.719 INFO:teuthology.orchestra.run.smithi084.stdout:253403070473 2024-09-20T22:42:08.719 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd last-stat-seq osd.7 2024-09-20T22:42:08.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:08 smithi154 ceph-mon[33264]: mgrmap e18: y(active, since 4m), standbys: x 2024-09-20T22:42:08.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:08 smithi154 ceph-mon[33264]: Active manager daemon y restarted 2024-09-20T22:42:08.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:08 smithi154 ceph-mon[33264]: Activating manager daemon y 2024-09-20T22:42:08.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:08 smithi154 ceph-mon[33264]: osdmap e77: 8 total, 8 up, 8 in 2024-09-20T22:42:08.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:08 smithi154 ceph-mon[33264]: mgrmap e19: y(active, starting, since 0.00833649s), standbys: x 2024-09-20T22:42:08.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:08 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-20T22:42:08.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:08 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-20T22:42:08.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:08 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-20T22:42:08.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:08 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-20T22:42:08.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:08 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-20T22:42:08.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:08 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-20T22:42:08.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:08 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-20T22:42:08.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:08 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-20T22:42:08.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:08 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T22:42:08.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:08 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-20T22:42:08.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:08 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-20T22:42:08.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:08 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-20T22:42:08.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:08 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-20T22:42:08.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:08 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-20T22:42:08.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:08 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-20T22:42:08.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:08 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-20T22:42:08.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:08 smithi154 ceph-mon[33264]: Manager daemon y is now available 2024-09-20T22:42:08.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:08 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:42:08.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:08 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:42:08.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:08 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:42:08.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:08 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-20T22:42:08.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:08 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-20T22:42:08.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:08 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-20T22:42:08.757 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:08 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-20T22:42:09.145 INFO:teuthology.orchestra.run.smithi084.stdout:107374182431 2024-09-20T22:42:09.145 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd last-stat-seq osd.3 2024-09-20T22:42:09.254 INFO:journalctl@ceph.iscsi.iscsi.a.smithi154.stdout:Sep 20 22:42:08 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-iscsi-iscsi-a[59730]: debug there is no tcmu-runner data available 2024-09-20T22:42:09.264 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:42:09.265 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:42:09.272 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:42:09.272 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:42:09.286 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:42:09.319 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:42:09.322 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:42:09.405 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:42:09 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: 2024-09-20T22:42:09.176+0000 7fd571f49640 -1 log_channel(cephadm) log [ERR] : [20/Sep/2024:22:42:09] ENGINE Error in HTTPServer.serve 2024-09-20T22:42:09.405 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:42:09 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: Traceback (most recent call last): 2024-09-20T22:42:09.405 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:42:09 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-20T22:42:09.405 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:42:09 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: self._connections.run(self.expiration_interval) 2024-09-20T22:42:09.405 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:42:09 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-20T22:42:09.405 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:42:09 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: self._run(expiration_interval) 2024-09-20T22:42:09.405 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:42:09 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-20T22:42:09.405 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:42:09 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: new_conn = self._from_server_socket(self.server.socket) 2024-09-20T22:42:09.405 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:42:09 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-20T22:42:09.405 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:42:09 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-20T22:42:09.406 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:42:09 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-20T22:42:09.406 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:42:09 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: s = self.context.wrap_socket( 2024-09-20T22:42:09.406 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:42:09 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-20T22:42:09.406 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:42:09 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: return self.sslsocket_class._create( 2024-09-20T22:42:09.406 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:42:09 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-20T22:42:09.406 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:42:09 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: self.do_handshake() 2024-09-20T22:42:09.406 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:42:09 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-20T22:42:09.406 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:42:09 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: self._sslobj.do_handshake() 2024-09-20T22:42:09.406 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:42:09 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-20T22:42:09.406 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:42:09 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: 2024-09-20T22:42:09.694 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:09 smithi084 ceph-mon[40024]: mgrmap e20: y(active, since 1.01152s), standbys: x 2024-09-20T22:42:09.694 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:09 smithi084 ceph-mon[40024]: [20/Sep/2024:22:42:08] ENGINE Bus STARTING 2024-09-20T22:42:09.694 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:09 smithi084 ceph-mon[40024]: [20/Sep/2024:22:42:09] ENGINE Serving on http://172.21.15.84:8765 2024-09-20T22:42:09.695 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:09 smithi084 ceph-mon[40024]: [20/Sep/2024:22:42:09] ENGINE Serving on https://172.21.15.84:7150 2024-09-20T22:42:09.695 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:09 smithi084 ceph-mon[40024]: [20/Sep/2024:22:42:09] ENGINE Bus STARTED 2024-09-20T22:42:09.695 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:09 smithi084 ceph-mon[40024]: [20/Sep/2024:22:42:09] ENGINE Error in HTTPServer.serve 2024-09-20T22:42:09.695 INFO:journalctl@ceph.mon.c.smithi084.stdout: Traceback (most recent call last): 2024-09-20T22:42:09.695 INFO:journalctl@ceph.mon.c.smithi084.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-20T22:42:09.695 INFO:journalctl@ceph.mon.c.smithi084.stdout: self._connections.run(self.expiration_interval) 2024-09-20T22:42:09.695 INFO:journalctl@ceph.mon.c.smithi084.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-20T22:42:09.695 INFO:journalctl@ceph.mon.c.smithi084.stdout: self._run(expiration_interval) 2024-09-20T22:42:09.695 INFO:journalctl@ceph.mon.c.smithi084.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-20T22:42:09.695 INFO:journalctl@ceph.mon.c.smithi084.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-20T22:42:09.695 INFO:journalctl@ceph.mon.c.smithi084.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-20T22:42:09.695 INFO:journalctl@ceph.mon.c.smithi084.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-20T22:42:09.695 INFO:journalctl@ceph.mon.c.smithi084.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-20T22:42:09.696 INFO:journalctl@ceph.mon.c.smithi084.stdout: s = self.context.wrap_socket( 2024-09-20T22:42:09.696 INFO:journalctl@ceph.mon.c.smithi084.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-20T22:42:09.696 INFO:journalctl@ceph.mon.c.smithi084.stdout: return self.sslsocket_class._create( 2024-09-20T22:42:09.696 INFO:journalctl@ceph.mon.c.smithi084.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-20T22:42:09.696 INFO:journalctl@ceph.mon.c.smithi084.stdout: self.do_handshake() 2024-09-20T22:42:09.696 INFO:journalctl@ceph.mon.c.smithi084.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-20T22:42:09.696 INFO:journalctl@ceph.mon.c.smithi084.stdout: self._sslobj.do_handshake() 2024-09-20T22:42:09.696 INFO:journalctl@ceph.mon.c.smithi084.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-20T22:42:09.696 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:09 smithi084 ceph-mon[29774]: mgrmap e20: y(active, since 1.01152s), standbys: x 2024-09-20T22:42:09.696 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:09 smithi084 ceph-mon[29774]: [20/Sep/2024:22:42:08] ENGINE Bus STARTING 2024-09-20T22:42:09.696 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:09 smithi084 ceph-mon[29774]: [20/Sep/2024:22:42:09] ENGINE Serving on http://172.21.15.84:8765 2024-09-20T22:42:09.697 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:09 smithi084 ceph-mon[29774]: [20/Sep/2024:22:42:09] ENGINE Serving on https://172.21.15.84:7150 2024-09-20T22:42:09.697 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:09 smithi084 ceph-mon[29774]: [20/Sep/2024:22:42:09] ENGINE Bus STARTED 2024-09-20T22:42:09.697 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:09 smithi084 ceph-mon[29774]: [20/Sep/2024:22:42:09] ENGINE Error in HTTPServer.serve 2024-09-20T22:42:09.697 INFO:journalctl@ceph.mon.a.smithi084.stdout: Traceback (most recent call last): 2024-09-20T22:42:09.697 INFO:journalctl@ceph.mon.a.smithi084.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-20T22:42:09.697 INFO:journalctl@ceph.mon.a.smithi084.stdout: self._connections.run(self.expiration_interval) 2024-09-20T22:42:09.697 INFO:journalctl@ceph.mon.a.smithi084.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-20T22:42:09.697 INFO:journalctl@ceph.mon.a.smithi084.stdout: self._run(expiration_interval) 2024-09-20T22:42:09.697 INFO:journalctl@ceph.mon.a.smithi084.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-20T22:42:09.697 INFO:journalctl@ceph.mon.a.smithi084.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-20T22:42:09.697 INFO:journalctl@ceph.mon.a.smithi084.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-20T22:42:09.697 INFO:journalctl@ceph.mon.a.smithi084.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-20T22:42:09.697 INFO:journalctl@ceph.mon.a.smithi084.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-20T22:42:09.698 INFO:journalctl@ceph.mon.a.smithi084.stdout: s = self.context.wrap_socket( 2024-09-20T22:42:09.698 INFO:journalctl@ceph.mon.a.smithi084.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-20T22:42:09.698 INFO:journalctl@ceph.mon.a.smithi084.stdout: return self.sslsocket_class._create( 2024-09-20T22:42:09.698 INFO:journalctl@ceph.mon.a.smithi084.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-20T22:42:09.698 INFO:journalctl@ceph.mon.a.smithi084.stdout: self.do_handshake() 2024-09-20T22:42:09.698 INFO:journalctl@ceph.mon.a.smithi084.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-20T22:42:09.698 INFO:journalctl@ceph.mon.a.smithi084.stdout: self._sslobj.do_handshake() 2024-09-20T22:42:09.698 INFO:journalctl@ceph.mon.a.smithi084.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-20T22:42:09.786 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:09 smithi154 ceph-mon[33264]: mgrmap e20: y(active, since 1.01152s), standbys: x 2024-09-20T22:42:09.786 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:09 smithi154 ceph-mon[33264]: [20/Sep/2024:22:42:08] ENGINE Bus STARTING 2024-09-20T22:42:09.786 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:09 smithi154 ceph-mon[33264]: [20/Sep/2024:22:42:09] ENGINE Serving on http://172.21.15.84:8765 2024-09-20T22:42:09.786 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:09 smithi154 ceph-mon[33264]: [20/Sep/2024:22:42:09] ENGINE Serving on https://172.21.15.84:7150 2024-09-20T22:42:09.786 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:09 smithi154 ceph-mon[33264]: [20/Sep/2024:22:42:09] ENGINE Bus STARTED 2024-09-20T22:42:09.786 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:09 smithi154 ceph-mon[33264]: [20/Sep/2024:22:42:09] ENGINE Error in HTTPServer.serve 2024-09-20T22:42:09.786 INFO:journalctl@ceph.mon.b.smithi154.stdout: Traceback (most recent call last): 2024-09-20T22:42:09.786 INFO:journalctl@ceph.mon.b.smithi154.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-20T22:42:09.787 INFO:journalctl@ceph.mon.b.smithi154.stdout: self._connections.run(self.expiration_interval) 2024-09-20T22:42:09.787 INFO:journalctl@ceph.mon.b.smithi154.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-20T22:42:09.787 INFO:journalctl@ceph.mon.b.smithi154.stdout: self._run(expiration_interval) 2024-09-20T22:42:09.787 INFO:journalctl@ceph.mon.b.smithi154.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-20T22:42:09.787 INFO:journalctl@ceph.mon.b.smithi154.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-20T22:42:09.787 INFO:journalctl@ceph.mon.b.smithi154.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-20T22:42:09.787 INFO:journalctl@ceph.mon.b.smithi154.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-20T22:42:09.787 INFO:journalctl@ceph.mon.b.smithi154.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-20T22:42:09.787 INFO:journalctl@ceph.mon.b.smithi154.stdout: s = self.context.wrap_socket( 2024-09-20T22:42:09.787 INFO:journalctl@ceph.mon.b.smithi154.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-20T22:42:09.787 INFO:journalctl@ceph.mon.b.smithi154.stdout: return self.sslsocket_class._create( 2024-09-20T22:42:09.788 INFO:journalctl@ceph.mon.b.smithi154.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-20T22:42:09.788 INFO:journalctl@ceph.mon.b.smithi154.stdout: self.do_handshake() 2024-09-20T22:42:09.788 INFO:journalctl@ceph.mon.b.smithi154.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-20T22:42:09.788 INFO:journalctl@ceph.mon.b.smithi154.stdout: self._sslobj.do_handshake() 2024-09-20T22:42:09.788 INFO:journalctl@ceph.mon.b.smithi154.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-20T22:42:10.105 INFO:teuthology.orchestra.run.smithi084.stdout:133143986202 2024-09-20T22:42:10.318 INFO:teuthology.orchestra.run.smithi084.stdout:171798691861 2024-09-20T22:42:10.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:10 smithi154 ceph-mon[33264]: pgmap v4: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:42:10.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:10 smithi154 ceph-mon[33264]: mgrmap e21: y(active, since 2s), standbys: x 2024-09-20T22:42:10.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:10 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/2794983629' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-20T22:42:10.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:10 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/788999167' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-20T22:42:10.864 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:10 smithi084 ceph-mon[29774]: pgmap v4: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:42:10.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:10 smithi084 ceph-mon[29774]: mgrmap e21: y(active, since 2s), standbys: x 2024-09-20T22:42:10.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:10 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/2794983629' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-20T22:42:10.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:10 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/788999167' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-20T22:42:10.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:10 smithi084 ceph-mon[40024]: pgmap v4: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:42:10.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:10 smithi084 ceph-mon[40024]: mgrmap e21: y(active, since 2s), standbys: x 2024-09-20T22:42:10.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:10 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/2794983629' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-20T22:42:10.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:10 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/788999167' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-20T22:42:11.539 INFO:teuthology.orchestra.run.smithi084.stdout:34359738415 2024-09-20T22:42:11.790 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:11 smithi084 ceph-mon[40024]: purged_snaps scrub starts 2024-09-20T22:42:11.790 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:11 smithi084 ceph-mon[40024]: purged_snaps scrub ok 2024-09-20T22:42:11.790 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:11 smithi084 ceph-mon[40024]: osdmap e78: 8 total, 8 up, 8 in 2024-09-20T22:42:11.790 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:11 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:42:11.790 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:11 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:42:11.791 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:11 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:42:11.791 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:11 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:42:11.791 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:11 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:42:11.791 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:11 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:42:11.791 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:11 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:42:11.791 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:11 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:42:11.791 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:11 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:42:11.791 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:11 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:42:11.791 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:11 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:42:11.791 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:11 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:42:11.791 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:11 smithi084 ceph-mon[40024]: Adjusting osd_memory_target on smithi154 to 3760M 2024-09-20T22:42:11.791 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:11 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:42:11.791 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:11 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/268526487' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-20T22:42:11.792 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:11 smithi084 ceph-mon[29774]: purged_snaps scrub starts 2024-09-20T22:42:11.792 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:11 smithi084 ceph-mon[29774]: purged_snaps scrub ok 2024-09-20T22:42:11.792 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:11 smithi084 ceph-mon[29774]: osdmap e78: 8 total, 8 up, 8 in 2024-09-20T22:42:11.792 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:11 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:42:11.792 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:11 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:42:11.792 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:11 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:42:11.792 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:11 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:42:11.792 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:11 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:42:11.792 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:11 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:42:11.792 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:11 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:42:11.792 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:11 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:42:11.792 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:11 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:42:11.793 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:11 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:42:11.793 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:11 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:42:11.793 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:11 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:42:11.793 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:11 smithi084 ceph-mon[29774]: Adjusting osd_memory_target on smithi154 to 3760M 2024-09-20T22:42:11.793 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:11 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:42:11.793 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:11 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/268526487' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-20T22:42:12.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:11 smithi154 ceph-mon[33264]: purged_snaps scrub starts 2024-09-20T22:42:12.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:11 smithi154 ceph-mon[33264]: purged_snaps scrub ok 2024-09-20T22:42:12.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:11 smithi154 ceph-mon[33264]: osdmap e78: 8 total, 8 up, 8 in 2024-09-20T22:42:12.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:11 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:42:12.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:11 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:42:12.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:11 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:42:12.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:11 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:42:12.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:11 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:42:12.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:11 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:42:12.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:11 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:42:12.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:11 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:42:12.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:11 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:42:12.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:11 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:42:12.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:11 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:42:12.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:11 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:42:12.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:11 smithi154 ceph-mon[33264]: Adjusting osd_memory_target on smithi154 to 3760M 2024-09-20T22:42:12.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:11 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:42:12.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:11 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/268526487' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-20T22:42:12.053 INFO:teuthology.orchestra.run.smithi084.stdout:253403070474 2024-09-20T22:42:12.763 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:12 smithi084 ceph-mon[29774]: pgmap v6: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:42:12.763 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:12 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/3481347167' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-20T22:42:12.763 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:12 smithi084 ceph-mon[29774]: mgrmap e22: y(active, since 4s), standbys: x 2024-09-20T22:42:12.764 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:12 smithi084 ceph-mon[40024]: pgmap v6: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:42:12.764 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:12 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/3481347167' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-20T22:42:12.764 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:12 smithi084 ceph-mon[40024]: mgrmap e22: y(active, since 4s), standbys: x 2024-09-20T22:42:12.792 INFO:tasks.cephadm.ceph_manager.ceph:need seq 253403070473 got 253403070474 for osd.7 2024-09-20T22:42:12.792 DEBUG:teuthology.parallel:result is None 2024-09-20T22:42:13.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:12 smithi154 ceph-mon[33264]: pgmap v6: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:42:13.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:12 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/3481347167' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-20T22:42:13.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:12 smithi154 ceph-mon[33264]: mgrmap e22: y(active, since 4s), standbys: x 2024-09-20T22:42:13.563 INFO:tasks.cephadm.ceph_manager.ceph:need seq 133143986201 got 133143986202 for osd.4 2024-09-20T22:42:13.563 DEBUG:teuthology.parallel:result is None 2024-09-20T22:42:13.872 INFO:tasks.cephadm.ceph_manager.ceph:need seq 171798691860 got 171798691861 for osd.5 2024-09-20T22:42:13.872 DEBUG:teuthology.parallel:result is None 2024-09-20T22:42:14.155 INFO:teuthology.orchestra.run.smithi084.stdout:55834574890 2024-09-20T22:42:14.243 INFO:teuthology.orchestra.run.smithi084.stdout:77309411366 2024-09-20T22:42:14.243 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:14 smithi084 ceph-mon[29774]: osdmap e79: 8 total, 8 up, 8 in 2024-09-20T22:42:14.243 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:14 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/1728197021' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-20T22:42:14.244 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:14 smithi084 ceph-mon[40024]: osdmap e79: 8 total, 8 up, 8 in 2024-09-20T22:42:14.244 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:14 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/1728197021' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-20T22:42:14.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:14 smithi154 ceph-mon[33264]: osdmap e79: 8 total, 8 up, 8 in 2024-09-20T22:42:14.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:14 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/1728197021' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-20T22:42:15.074 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411364 got 77309411366 for osd.2 2024-09-20T22:42:15.075 DEBUG:teuthology.parallel:result is None 2024-09-20T22:42:15.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:15 smithi084 ceph-mon[40024]: pgmap v8: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:42:15.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:15 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/3815310750' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-20T22:42:15.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:15 smithi084 ceph-mon[29774]: pgmap v8: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:42:15.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:15 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/3815310750' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-20T22:42:15.444 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574888 got 55834574890 for osd.1 2024-09-20T22:42:15.445 DEBUG:teuthology.parallel:result is None 2024-09-20T22:42:15.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:15 smithi154 ceph-mon[33264]: pgmap v8: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:42:15.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:15 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/3815310750' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-20T22:42:16.026 INFO:teuthology.orchestra.run.smithi084.stdout:206158430224 2024-09-20T22:42:16.364 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:16 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/166787143' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-20T22:42:16.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:16 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/166787143' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-20T22:42:16.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:16 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/166787143' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-20T22:42:17.027 INFO:tasks.cephadm.ceph_manager.ceph:need seq 206158430223 got 206158430224 for osd.6 2024-09-20T22:42:17.027 DEBUG:teuthology.parallel:result is None 2024-09-20T22:42:17.068 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:42:17.098 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738414 got 34359738415 for osd.0 2024-09-20T22:42:17.099 DEBUG:teuthology.parallel:result is None 2024-09-20T22:42:17.314 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:17 smithi084 ceph-mon[40024]: pgmap v9: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-09-20T22:42:17.314 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:17 smithi084 ceph-mon[29774]: pgmap v9: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-09-20T22:42:17.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:17 smithi154 ceph-mon[33264]: pgmap v9: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-09-20T22:42:17.706 INFO:teuthology.orchestra.run.smithi084.stdout:107374182433 2024-09-20T22:42:18.364 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:42:18 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: ::ffff:172.21.15.154 - - [20/Sep/2024:22:42:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-20T22:42:18.542 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:18 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/3866547853' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-20T22:42:18.543 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:18 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:42:18.543 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:18 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:42:18.543 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:18 smithi084 ceph-mon[29774]: osdmap e80: 8 total, 8 up, 8 in 2024-09-20T22:42:18.543 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:18 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/3866547853' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-20T22:42:18.543 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:18 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:42:18.543 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:18 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:42:18.543 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:18 smithi084 ceph-mon[40024]: osdmap e80: 8 total, 8 up, 8 in 2024-09-20T22:42:18.544 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182431 got 107374182433 for osd.3 2024-09-20T22:42:18.544 DEBUG:teuthology.parallel:result is None 2024-09-20T22:42:18.544 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-09-20T22:42:18.545 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph pg dump --format=json 2024-09-20T22:42:18.545 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:18 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/3866547853' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-20T22:42:18.545 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:18 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:42:18.545 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:18 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:42:18.545 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:18 smithi154 ceph-mon[33264]: osdmap e80: 8 total, 8 up, 8 in 2024-09-20T22:42:18.798 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:42:19.253 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:19 smithi084 ceph-mon[29774]: pgmap v10: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 0 B/s wr, 15 op/s 2024-09-20T22:42:19.253 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:19 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:42:19.253 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:19 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:42:19.254 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:19 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:42:19.254 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:19 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:42:19.254 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:19 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:42:19.254 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:19 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:42:19.254 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:19 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:42:19.254 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:19 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:42:19.254 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:19 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:42:19.254 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:19 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:42:19.254 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:19 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:42:19.254 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:19 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:42:19.254 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:19 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:42:19.254 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:19 smithi084 ceph-mon[40024]: pgmap v10: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 0 B/s wr, 15 op/s 2024-09-20T22:42:19.255 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:19 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:42:19.255 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:19 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:42:19.255 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:19 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:42:19.255 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:19 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:42:19.255 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:19 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:42:19.255 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:19 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:42:19.255 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:19 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:42:19.255 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:19 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:42:19.255 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:19 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:42:19.255 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:19 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:42:19.255 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:19 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:42:19.255 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:19 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:42:19.255 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:19 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:42:19.256 INFO:journalctl@ceph.iscsi.iscsi.a.smithi154.stdout:Sep 20 22:42:18 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-iscsi-iscsi-a[59730]: debug there is no tcmu-runner data available 2024-09-20T22:42:19.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:19 smithi154 ceph-mon[33264]: pgmap v10: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 0 B/s wr, 15 op/s 2024-09-20T22:42:19.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:19 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:42:19.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:19 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:42:19.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:19 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:42:19.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:19 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:42:19.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:19 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:42:19.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:19 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:42:19.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:19 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:42:19.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:19 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:42:19.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:19 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:42:19.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:19 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:42:19.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:19 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:42:19.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:19 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:42:19.257 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:19 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:42:19.554 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-20T22:42:19.555 INFO:teuthology.orchestra.run.smithi084.stderr:dumped all 2024-09-20T22:42:20.334 INFO:teuthology.orchestra.run.smithi084.stdout:{"pg_ready":true,"pg_map":{"version":12,"stamp":"2024-09-20T22:42:19.119448+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596507,"num_objects":199,"num_object_clones":0,"num_object_copies":597,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":199,"num_whiteouts":0,"num_read":969,"num_read_kb":815,"num_write":615,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":601,"ondisk_log_size":601,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":239980,"kb_used_data":7112,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":749721236,"statfs":{"total":767960285184,"available":767714545664,"internally_reserved":0,"allocated":7282688,"data_stored":4095589,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":106,"num_read_kb":216,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.508335"},"pg_stats":[{"pgid":"3.1f","version":"67'1","reported_seq":43,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.091842+0000","last_change":"2024-09-20T22:41:38.159365+0000","last_active":"2024-09-20T22:42:18.091842+0000","last_peered":"2024-09-20T22:42:18.091842+0000","last_clean":"2024-09-20T22:42:18.091842+0000","last_became_active":"2024-09-20T22:41:38.159085+0000","last_became_peered":"2024-09-20T22:41:38.159085+0000","last_unstale":"2024-09-20T22:42:18.091842+0000","last_undegraded":"2024-09-20T22:42:18.091842+0000","last_fullsized":"2024-09-20T22:42:18.091842+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:37.732756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"74'9","reported_seq":49,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.092121+0000","last_change":"2024-09-20T22:41:40.169673+0000","last_active":"2024-09-20T22:42:18.092121+0000","last_peered":"2024-09-20T22:42:18.092121+0000","last_clean":"2024-09-20T22:42:18.092121+0000","last_became_active":"2024-09-20T22:41:40.168934+0000","last_became_peered":"2024-09-20T22:41:40.168934+0000","last_unstale":"2024-09-20T22:42:18.092121+0000","last_undegraded":"2024-09-20T22:42:18.092121+0000","last_fullsized":"2024-09-20T22:42:18.092121+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:54.489342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":29,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.091084+0000","last_change":"2024-09-20T22:41:42.182304+0000","last_active":"2024-09-20T22:42:18.091084+0000","last_peered":"2024-09-20T22:42:18.091084+0000","last_clean":"2024-09-20T22:42:18.091084+0000","last_became_active":"2024-09-20T22:41:42.182024+0000","last_became_peered":"2024-09-20T22:41:42.182024+0000","last_unstale":"2024-09-20T22:42:18.091084+0000","last_undegraded":"2024-09-20T22:42:18.091084+0000","last_fullsized":"2024-09-20T22:42:18.091084+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:57.397887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.092216+0000","last_change":"2024-09-20T22:41:44.198207+0000","last_active":"2024-09-20T22:42:18.092216+0000","last_peered":"2024-09-20T22:42:18.092216+0000","last_clean":"2024-09-20T22:42:18.092216+0000","last_became_active":"2024-09-20T22:41:44.197897+0000","last_became_peered":"2024-09-20T22:41:44.197897+0000","last_unstale":"2024-09-20T22:42:18.092216+0000","last_undegraded":"2024-09-20T22:42:18.092216+0000","last_fullsized":"2024-09-20T22:42:18.092216+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:53.466451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.254290+0000","last_change":"2024-09-20T22:41:44.832449+0000","last_active":"2024-09-20T22:42:18.254290+0000","last_peered":"2024-09-20T22:42:18.254290+0000","last_clean":"2024-09-20T22:42:18.254290+0000","last_became_active":"2024-09-20T22:41:44.832099+0000","last_became_peered":"2024-09-20T22:41:44.832099+0000","last_unstale":"2024-09-20T22:42:18.254290+0000","last_undegraded":"2024-09-20T22:42:18.254290+0000","last_fullsized":"2024-09-20T22:42:18.254290+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:45.326749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":37,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.254797+0000","last_change":"2024-09-20T22:41:38.157685+0000","last_active":"2024-09-20T22:42:18.254797+0000","last_peered":"2024-09-20T22:42:18.254797+0000","last_clean":"2024-09-20T22:42:18.254797+0000","last_became_active":"2024-09-20T22:41:38.157298+0000","last_became_peered":"2024-09-20T22:41:38.157298+0000","last_unstale":"2024-09-20T22:42:18.254797+0000","last_undegraded":"2024-09-20T22:42:18.254797+0000","last_fullsized":"2024-09-20T22:42:18.254797+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:00.813798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"74'15","reported_seq":58,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.254425+0000","last_change":"2024-09-20T22:41:40.163585+0000","last_active":"2024-09-20T22:42:18.254425+0000","last_peered":"2024-09-20T22:42:18.254425+0000","last_clean":"2024-09-20T22:42:18.254425+0000","last_became_active":"2024-09-20T22:41:40.163242+0000","last_became_peered":"2024-09-20T22:41:40.163242+0000","last_unstale":"2024-09-20T22:42:18.254425+0000","last_undegraded":"2024-09-20T22:42:18.254425+0000","last_fullsized":"2024-09-20T22:42:18.254425+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:58.410323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":29,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.092118+0000","last_change":"2024-09-20T22:41:42.187359+0000","last_active":"2024-09-20T22:42:18.092118+0000","last_peered":"2024-09-20T22:42:18.092118+0000","last_clean":"2024-09-20T22:42:18.092118+0000","last_became_active":"2024-09-20T22:41:42.187076+0000","last_became_peered":"2024-09-20T22:41:42.187076+0000","last_unstale":"2024-09-20T22:42:18.092118+0000","last_undegraded":"2024-09-20T22:42:18.092118+0000","last_fullsized":"2024-09-20T22:42:18.092118+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:42.282475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":37,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.547950+0000","last_change":"2024-09-20T22:41:38.156694+0000","last_active":"2024-09-20T22:42:18.547950+0000","last_peered":"2024-09-20T22:42:18.547950+0000","last_clean":"2024-09-20T22:42:18.547950+0000","last_became_active":"2024-09-20T22:41:38.154827+0000","last_became_peered":"2024-09-20T22:41:38.154827+0000","last_unstale":"2024-09-20T22:42:18.547950+0000","last_undegraded":"2024-09-20T22:42:18.547950+0000","last_fullsized":"2024-09-20T22:42:18.547950+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:48.618357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"74'9","reported_seq":49,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.092795+0000","last_change":"2024-09-20T22:41:40.163006+0000","last_active":"2024-09-20T22:42:18.092795+0000","last_peered":"2024-09-20T22:42:18.092795+0000","last_clean":"2024-09-20T22:42:18.092795+0000","last_became_active":"2024-09-20T22:41:40.162546+0000","last_became_peered":"2024-09-20T22:41:40.162546+0000","last_unstale":"2024-09-20T22:42:18.092795+0000","last_undegraded":"2024-09-20T22:42:18.092795+0000","last_fullsized":"2024-09-20T22:42:18.092795+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:46.501647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":29,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.548539+0000","last_change":"2024-09-20T22:41:42.185835+0000","last_active":"2024-09-20T22:42:18.548539+0000","last_peered":"2024-09-20T22:42:18.548539+0000","last_clean":"2024-09-20T22:42:18.548539+0000","last_became_active":"2024-09-20T22:41:42.185538+0000","last_became_peered":"2024-09-20T22:41:42.185538+0000","last_unstale":"2024-09-20T22:42:18.548539+0000","last_undegraded":"2024-09-20T22:42:18.548539+0000","last_fullsized":"2024-09-20T22:42:18.548539+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:57.305761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.090692+0000","last_change":"2024-09-20T22:41:44.198169+0000","last_active":"2024-09-20T22:42:18.090692+0000","last_peered":"2024-09-20T22:42:18.090692+0000","last_clean":"2024-09-20T22:42:18.090692+0000","last_became_active":"2024-09-20T22:41:44.197113+0000","last_became_peered":"2024-09-20T22:41:44.197113+0000","last_unstale":"2024-09-20T22:42:18.090692+0000","last_undegraded":"2024-09-20T22:42:18.090692+0000","last_fullsized":"2024-09-20T22:42:18.090692+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:02.374640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":37,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.547916+0000","last_change":"2024-09-20T22:41:38.156116+0000","last_active":"2024-09-20T22:42:18.547916+0000","last_peered":"2024-09-20T22:42:18.547916+0000","last_clean":"2024-09-20T22:42:18.547916+0000","last_became_active":"2024-09-20T22:41:38.154650+0000","last_became_peered":"2024-09-20T22:41:38.154650+0000","last_unstale":"2024-09-20T22:42:18.547916+0000","last_undegraded":"2024-09-20T22:42:18.547916+0000","last_fullsized":"2024-09-20T22:42:18.547916+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:44.595033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"74'5","reported_seq":43,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.092397+0000","last_change":"2024-09-20T22:41:40.171883+0000","last_active":"2024-09-20T22:42:18.092397+0000","last_peered":"2024-09-20T22:42:18.092397+0000","last_clean":"2024-09-20T22:42:18.092397+0000","last_became_active":"2024-09-20T22:41:40.171081+0000","last_became_peered":"2024-09-20T22:41:40.171081+0000","last_unstale":"2024-09-20T22:42:18.092397+0000","last_undegraded":"2024-09-20T22:42:18.092397+0000","last_fullsized":"2024-09-20T22:42:18.092397+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:55.048729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":29,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.465214+0000","last_change":"2024-09-20T22:41:42.183965+0000","last_active":"2024-09-20T22:42:18.465214+0000","last_peered":"2024-09-20T22:42:18.465214+0000","last_clean":"2024-09-20T22:42:18.465214+0000","last_became_active":"2024-09-20T22:41:42.182916+0000","last_became_peered":"2024-09-20T22:41:42.182916+0000","last_unstale":"2024-09-20T22:42:18.465214+0000","last_undegraded":"2024-09-20T22:42:18.465214+0000","last_fullsized":"2024-09-20T22:42:18.465214+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:42.258865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.548312+0000","last_change":"2024-09-20T22:41:44.198221+0000","last_active":"2024-09-20T22:42:18.548312+0000","last_peered":"2024-09-20T22:42:18.548312+0000","last_clean":"2024-09-20T22:42:18.548312+0000","last_became_active":"2024-09-20T22:41:44.197927+0000","last_became_peered":"2024-09-20T22:41:44.197927+0000","last_unstale":"2024-09-20T22:42:18.548312+0000","last_undegraded":"2024-09-20T22:42:18.548312+0000","last_fullsized":"2024-09-20T22:42:18.548312+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:47.985353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.092499+0000","last_change":"2024-09-20T22:41:44.832646+0000","last_active":"2024-09-20T22:42:18.092499+0000","last_peered":"2024-09-20T22:42:18.092499+0000","last_clean":"2024-09-20T22:42:18.092499+0000","last_became_active":"2024-09-20T22:41:44.832360+0000","last_became_peered":"2024-09-20T22:41:44.832360+0000","last_unstale":"2024-09-20T22:42:18.092499+0000","last_undegraded":"2024-09-20T22:42:18.092499+0000","last_fullsized":"2024-09-20T22:42:18.092499+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:45.707782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":37,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.091790+0000","last_change":"2024-09-20T22:41:38.152726+0000","last_active":"2024-09-20T22:42:18.091790+0000","last_peered":"2024-09-20T22:42:18.091790+0000","last_clean":"2024-09-20T22:42:18.091790+0000","last_became_active":"2024-09-20T22:41:38.152443+0000","last_became_peered":"2024-09-20T22:41:38.152443+0000","last_unstale":"2024-09-20T22:42:18.091790+0000","last_undegraded":"2024-09-20T22:42:18.091790+0000","last_fullsized":"2024-09-20T22:42:18.091790+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:03.325746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"74'15","reported_seq":58,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.318481+0000","last_change":"2024-09-20T22:41:40.170414+0000","last_active":"2024-09-20T22:42:18.318481+0000","last_peered":"2024-09-20T22:42:18.318481+0000","last_clean":"2024-09-20T22:42:18.318481+0000","last_became_active":"2024-09-20T22:41:40.170131+0000","last_became_peered":"2024-09-20T22:41:40.170131+0000","last_unstale":"2024-09-20T22:42:18.318481+0000","last_undegraded":"2024-09-20T22:42:18.318481+0000","last_fullsized":"2024-09-20T22:42:18.318481+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:57.393598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":29,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.092312+0000","last_change":"2024-09-20T22:41:42.185171+0000","last_active":"2024-09-20T22:42:18.092312+0000","last_peered":"2024-09-20T22:42:18.092312+0000","last_clean":"2024-09-20T22:42:18.092312+0000","last_became_active":"2024-09-20T22:41:42.184759+0000","last_became_peered":"2024-09-20T22:41:42.184759+0000","last_unstale":"2024-09-20T22:42:18.092312+0000","last_undegraded":"2024-09-20T22:42:18.092312+0000","last_fullsized":"2024-09-20T22:42:18.092312+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:01.601642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.254753+0000","last_change":"2024-09-20T22:41:44.832468+0000","last_active":"2024-09-20T22:42:18.254753+0000","last_peered":"2024-09-20T22:42:18.254753+0000","last_clean":"2024-09-20T22:42:18.254753+0000","last_became_active":"2024-09-20T22:41:44.832131+0000","last_became_peered":"2024-09-20T22:41:44.832131+0000","last_unstale":"2024-09-20T22:42:18.254753+0000","last_undegraded":"2024-09-20T22:42:18.254753+0000","last_fullsized":"2024-09-20T22:42:18.254753+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:08.595231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":37,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.093150+0000","last_change":"2024-09-20T22:41:38.150584+0000","last_active":"2024-09-20T22:42:18.093150+0000","last_peered":"2024-09-20T22:42:18.093150+0000","last_clean":"2024-09-20T22:42:18.093150+0000","last_became_active":"2024-09-20T22:41:38.150232+0000","last_became_peered":"2024-09-20T22:41:38.150232+0000","last_unstale":"2024-09-20T22:42:18.093150+0000","last_undegraded":"2024-09-20T22:42:18.093150+0000","last_fullsized":"2024-09-20T22:42:18.093150+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:02.237312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"74'12","reported_seq":56,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.254603+0000","last_change":"2024-09-20T22:41:40.166344+0000","last_active":"2024-09-20T22:42:18.254603+0000","last_peered":"2024-09-20T22:42:18.254603+0000","last_clean":"2024-09-20T22:42:18.254603+0000","last_became_active":"2024-09-20T22:41:40.166087+0000","last_became_peered":"2024-09-20T22:41:40.166087+0000","last_unstale":"2024-09-20T22:42:18.254603+0000","last_undegraded":"2024-09-20T22:42:18.254603+0000","last_fullsized":"2024-09-20T22:42:18.254603+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:01.373771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":29,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.092462+0000","last_change":"2024-09-20T22:41:42.183266+0000","last_active":"2024-09-20T22:42:18.092462+0000","last_peered":"2024-09-20T22:42:18.092462+0000","last_clean":"2024-09-20T22:42:18.092462+0000","last_became_active":"2024-09-20T22:41:42.183005+0000","last_became_peered":"2024-09-20T22:41:42.183005+0000","last_unstale":"2024-09-20T22:42:18.092462+0000","last_undegraded":"2024-09-20T22:42:18.092462+0000","last_fullsized":"2024-09-20T22:42:18.092462+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:01.169906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.465546+0000","last_change":"2024-09-20T22:41:44.200115+0000","last_active":"2024-09-20T22:42:18.465546+0000","last_peered":"2024-09-20T22:42:18.465546+0000","last_clean":"2024-09-20T22:42:18.465546+0000","last_became_active":"2024-09-20T22:41:44.199822+0000","last_became_peered":"2024-09-20T22:41:44.199822+0000","last_unstale":"2024-09-20T22:42:18.465546+0000","last_undegraded":"2024-09-20T22:42:18.465546+0000","last_fullsized":"2024-09-20T22:42:18.465546+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:01.277411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"67'1","reported_seq":38,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.091235+0000","last_change":"2024-09-20T22:41:38.155735+0000","last_active":"2024-09-20T22:42:18.091235+0000","last_peered":"2024-09-20T22:42:18.091235+0000","last_clean":"2024-09-20T22:42:18.091235+0000","last_became_active":"2024-09-20T22:41:38.155424+0000","last_became_peered":"2024-09-20T22:41:38.155424+0000","last_unstale":"2024-09-20T22:42:18.091235+0000","last_undegraded":"2024-09-20T22:42:18.091235+0000","last_fullsized":"2024-09-20T22:42:18.091235+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:52.584133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"74'10","reported_seq":48,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.091915+0000","last_change":"2024-09-20T22:41:40.252498+0000","last_active":"2024-09-20T22:42:18.091915+0000","last_peered":"2024-09-20T22:42:18.091915+0000","last_clean":"2024-09-20T22:42:18.091915+0000","last_became_active":"2024-09-20T22:41:40.252001+0000","last_became_peered":"2024-09-20T22:41:40.252001+0000","last_unstale":"2024-09-20T22:42:18.091915+0000","last_undegraded":"2024-09-20T22:42:18.091915+0000","last_fullsized":"2024-09-20T22:42:18.091915+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:01.417640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"74'8","reported_seq":50,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-20T22:42:13.209735+0000","last_change":"2024-09-20T22:41:42.186385+0000","last_active":"2024-09-20T22:42:13.209735+0000","last_peered":"2024-09-20T22:42:13.209735+0000","last_clean":"2024-09-20T22:42:13.209735+0000","last_became_active":"2024-09-20T22:41:42.186071+0000","last_became_peered":"2024-09-20T22:41:42.186071+0000","last_unstale":"2024-09-20T22:42:13.209735+0000","last_undegraded":"2024-09-20T22:42:13.209735+0000","last_fullsized":"2024-09-20T22:42:13.209735+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:56.929679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.f","version":"74'15","reported_seq":58,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.092219+0000","last_change":"2024-09-20T22:41:40.168188+0000","last_active":"2024-09-20T22:42:18.092219+0000","last_peered":"2024-09-20T22:42:18.092219+0000","last_clean":"2024-09-20T22:42:18.092219+0000","last_became_active":"2024-09-20T22:41:40.165963+0000","last_became_peered":"2024-09-20T22:41:40.165963+0000","last_unstale":"2024-09-20T22:42:18.092219+0000","last_undegraded":"2024-09-20T22:42:18.092219+0000","last_fullsized":"2024-09-20T22:42:18.092219+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:46.519265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":37,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.253822+0000","last_change":"2024-09-20T22:41:38.157043+0000","last_active":"2024-09-20T22:42:18.253822+0000","last_peered":"2024-09-20T22:42:18.253822+0000","last_clean":"2024-09-20T22:42:18.253822+0000","last_became_active":"2024-09-20T22:41:38.156749+0000","last_became_peered":"2024-09-20T22:41:38.156749+0000","last_unstale":"2024-09-20T22:42:18.253822+0000","last_undegraded":"2024-09-20T22:42:18.253822+0000","last_fullsized":"2024-09-20T22:42:18.253822+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:03.406143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"74'8","reported_seq":46,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.092780+0000","last_change":"2024-09-20T22:41:42.182555+0000","last_active":"2024-09-20T22:42:18.092780+0000","last_peered":"2024-09-20T22:42:18.092780+0000","last_clean":"2024-09-20T22:42:18.092780+0000","last_became_active":"2024-09-20T22:41:42.182224+0000","last_became_peered":"2024-09-20T22:41:42.182224+0000","last_unstale":"2024-09-20T22:42:18.092780+0000","last_undegraded":"2024-09-20T22:42:18.092780+0000","last_fullsized":"2024-09-20T22:42:18.092780+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:07.384166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.548714+0000","last_change":"2024-09-20T22:41:44.198655+0000","last_active":"2024-09-20T22:42:18.548714+0000","last_peered":"2024-09-20T22:42:18.548714+0000","last_clean":"2024-09-20T22:42:18.548714+0000","last_became_active":"2024-09-20T22:41:44.198347+0000","last_became_peered":"2024-09-20T22:41:44.198347+0000","last_unstale":"2024-09-20T22:42:18.548714+0000","last_undegraded":"2024-09-20T22:42:18.548714+0000","last_fullsized":"2024-09-20T22:42:18.548714+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:54.119646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.0","version":"74'18","reported_seq":65,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.254174+0000","last_change":"2024-09-20T22:41:40.252203+0000","last_active":"2024-09-20T22:42:18.254174+0000","last_peered":"2024-09-20T22:42:18.254174+0000","last_clean":"2024-09-20T22:42:18.254174+0000","last_became_active":"2024-09-20T22:41:40.251868+0000","last_became_peered":"2024-09-20T22:41:40.251868+0000","last_unstale":"2024-09-20T22:42:18.254174+0000","last_undegraded":"2024-09-20T22:42:18.254174+0000","last_fullsized":"2024-09-20T22:42:18.254174+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:57.966357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":37,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.253939+0000","last_change":"2024-09-20T22:41:38.161991+0000","last_active":"2024-09-20T22:42:18.253939+0000","last_peered":"2024-09-20T22:42:18.253939+0000","last_clean":"2024-09-20T22:42:18.253939+0000","last_became_active":"2024-09-20T22:41:38.161335+0000","last_became_peered":"2024-09-20T22:41:38.161335+0000","last_unstale":"2024-09-20T22:42:18.253939+0000","last_undegraded":"2024-09-20T22:42:18.253939+0000","last_fullsized":"2024-09-20T22:42:18.253939+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:48.134689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":29,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.092219+0000","last_change":"2024-09-20T22:41:42.190157+0000","last_active":"2024-09-20T22:42:18.092219+0000","last_peered":"2024-09-20T22:42:18.092219+0000","last_clean":"2024-09-20T22:42:18.092219+0000","last_became_active":"2024-09-20T22:41:42.189743+0000","last_became_peered":"2024-09-20T22:41:42.189743+0000","last_unstale":"2024-09-20T22:42:18.092219+0000","last_undegraded":"2024-09-20T22:42:18.092219+0000","last_fullsized":"2024-09-20T22:42:18.092219+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:42.602485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.092640+0000","last_change":"2024-09-20T22:41:44.193944+0000","last_active":"2024-09-20T22:42:18.092640+0000","last_peered":"2024-09-20T22:42:18.092640+0000","last_clean":"2024-09-20T22:42:18.092640+0000","last_became_active":"2024-09-20T22:41:44.193646+0000","last_became_peered":"2024-09-20T22:41:44.193646+0000","last_unstale":"2024-09-20T22:42:18.092640+0000","last_undegraded":"2024-09-20T22:42:18.092640+0000","last_fullsized":"2024-09-20T22:42:18.092640+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:50.182915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"74'14","reported_seq":54,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.092192+0000","last_change":"2024-09-20T22:41:40.164011+0000","last_active":"2024-09-20T22:42:18.092192+0000","last_peered":"2024-09-20T22:42:18.092192+0000","last_clean":"2024-09-20T22:42:18.092192+0000","last_became_active":"2024-09-20T22:41:40.163695+0000","last_became_peered":"2024-09-20T22:41:40.163695+0000","last_unstale":"2024-09-20T22:42:18.092192+0000","last_undegraded":"2024-09-20T22:42:18.092192+0000","last_fullsized":"2024-09-20T22:42:18.092192+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:04.888108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"67'1","reported_seq":38,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.092025+0000","last_change":"2024-09-20T22:41:38.150707+0000","last_active":"2024-09-20T22:42:18.092025+0000","last_peered":"2024-09-20T22:42:18.092025+0000","last_clean":"2024-09-20T22:42:18.092025+0000","last_became_active":"2024-09-20T22:41:38.150331+0000","last_became_peered":"2024-09-20T22:41:38.150331+0000","last_unstale":"2024-09-20T22:42:18.092025+0000","last_undegraded":"2024-09-20T22:42:18.092025+0000","last_fullsized":"2024-09-20T22:42:18.092025+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:53.797157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"74'8","reported_seq":46,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.254075+0000","last_change":"2024-09-20T22:41:42.179893+0000","last_active":"2024-09-20T22:42:18.254075+0000","last_peered":"2024-09-20T22:42:18.254075+0000","last_clean":"2024-09-20T22:42:18.254075+0000","last_became_active":"2024-09-20T22:41:42.179613+0000","last_became_peered":"2024-09-20T22:41:42.179613+0000","last_unstale":"2024-09-20T22:42:18.254075+0000","last_undegraded":"2024-09-20T22:42:18.254075+0000","last_fullsized":"2024-09-20T22:42:18.254075+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:03.642295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.465380+0000","last_change":"2024-09-20T22:41:44.832161+0000","last_active":"2024-09-20T22:42:18.465380+0000","last_peered":"2024-09-20T22:42:18.465380+0000","last_clean":"2024-09-20T22:42:18.465380+0000","last_became_active":"2024-09-20T22:41:44.831901+0000","last_became_peered":"2024-09-20T22:41:44.831901+0000","last_unstale":"2024-09-20T22:42:18.465380+0000","last_undegraded":"2024-09-20T22:42:18.465380+0000","last_fullsized":"2024-09-20T22:42:18.465380+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:10.151190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2","version":"74'10","reported_seq":48,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.091491+0000","last_change":"2024-09-20T22:41:40.167041+0000","last_active":"2024-09-20T22:42:18.091491+0000","last_peered":"2024-09-20T22:42:18.091491+0000","last_clean":"2024-09-20T22:42:18.091491+0000","last_became_active":"2024-09-20T22:41:40.165653+0000","last_became_peered":"2024-09-20T22:41:40.165653+0000","last_unstale":"2024-09-20T22:42:18.091491+0000","last_undegraded":"2024-09-20T22:42:18.091491+0000","last_fullsized":"2024-09-20T22:42:18.091491+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:47.255874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":37,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.549041+0000","last_change":"2024-09-20T22:41:38.155924+0000","last_active":"2024-09-20T22:42:18.549041+0000","last_peered":"2024-09-20T22:42:18.549041+0000","last_clean":"2024-09-20T22:42:18.549041+0000","last_became_active":"2024-09-20T22:41:38.155629+0000","last_became_peered":"2024-09-20T22:41:38.155629+0000","last_unstale":"2024-09-20T22:42:18.549041+0000","last_undegraded":"2024-09-20T22:42:18.549041+0000","last_fullsized":"2024-09-20T22:42:18.549041+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:39.438618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"74'8","reported_seq":46,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.091439+0000","last_change":"2024-09-20T22:41:42.185583+0000","last_active":"2024-09-20T22:42:18.091439+0000","last_peered":"2024-09-20T22:42:18.091439+0000","last_clean":"2024-09-20T22:42:18.091439+0000","last_became_active":"2024-09-20T22:41:42.185213+0000","last_became_peered":"2024-09-20T22:41:42.185213+0000","last_unstale":"2024-09-20T22:42:18.091439+0000","last_undegraded":"2024-09-20T22:42:18.091439+0000","last_fullsized":"2024-09-20T22:42:18.091439+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:48.459335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.091670+0000","last_change":"2024-09-20T22:41:44.197894+0000","last_active":"2024-09-20T22:42:18.091670+0000","last_peered":"2024-09-20T22:42:18.091670+0000","last_clean":"2024-09-20T22:42:18.091670+0000","last_became_active":"2024-09-20T22:41:44.197619+0000","last_became_peered":"2024-09-20T22:41:44.197619+0000","last_unstale":"2024-09-20T22:42:18.091670+0000","last_undegraded":"2024-09-20T22:42:18.091670+0000","last_fullsized":"2024-09-20T22:42:18.091670+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:00.802751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"74'19","reported_seq":69,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.091566+0000","last_change":"2024-09-20T22:41:40.252159+0000","last_active":"2024-09-20T22:42:18.091566+0000","last_peered":"2024-09-20T22:42:18.091566+0000","last_clean":"2024-09-20T22:42:18.091566+0000","last_became_active":"2024-09-20T22:41:40.251866+0000","last_became_peered":"2024-09-20T22:41:40.251866+0000","last_unstale":"2024-09-20T22:42:18.091566+0000","last_undegraded":"2024-09-20T22:42:18.091566+0000","last_fullsized":"2024-09-20T22:42:18.091566+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:43.089389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":37,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.092585+0000","last_change":"2024-09-20T22:41:38.161236+0000","last_active":"2024-09-20T22:42:18.092585+0000","last_peered":"2024-09-20T22:42:18.092585+0000","last_clean":"2024-09-20T22:42:18.092585+0000","last_became_active":"2024-09-20T22:41:38.160832+0000","last_became_peered":"2024-09-20T22:41:38.160832+0000","last_unstale":"2024-09-20T22:42:18.092585+0000","last_undegraded":"2024-09-20T22:42:18.092585+0000","last_fullsized":"2024-09-20T22:42:18.092585+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:50.975305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":27,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-20T22:42:13.093143+0000","last_change":"2024-09-20T22:41:42.185638+0000","last_active":"2024-09-20T22:42:13.093143+0000","last_peered":"2024-09-20T22:42:13.093143+0000","last_clean":"2024-09-20T22:42:13.093143+0000","last_became_active":"2024-09-20T22:41:42.184412+0000","last_became_peered":"2024-09-20T22:41:42.184412+0000","last_unstale":"2024-09-20T22:42:13.093143+0000","last_undegraded":"2024-09-20T22:42:13.093143+0000","last_fullsized":"2024-09-20T22:42:13.093143+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:57.435580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.090990+0000","last_change":"2024-09-20T22:41:44.830917+0000","last_active":"2024-09-20T22:42:18.090990+0000","last_peered":"2024-09-20T22:42:18.090990+0000","last_clean":"2024-09-20T22:42:18.090990+0000","last_became_active":"2024-09-20T22:41:44.830538+0000","last_became_peered":"2024-09-20T22:41:44.830538+0000","last_unstale":"2024-09-20T22:42:18.090990+0000","last_undegraded":"2024-09-20T22:42:18.090990+0000","last_fullsized":"2024-09-20T22:42:18.090990+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:54.961702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"74'28","reported_seq":90,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.092554+0000","last_change":"2024-09-20T22:41:40.167313+0000","last_active":"2024-09-20T22:42:18.092554+0000","last_peered":"2024-09-20T22:42:18.092554+0000","last_clean":"2024-09-20T22:42:18.092554+0000","last_became_active":"2024-09-20T22:41:40.166958+0000","last_became_peered":"2024-09-20T22:41:40.166958+0000","last_unstale":"2024-09-20T22:42:18.092554+0000","last_undegraded":"2024-09-20T22:42:18.092554+0000","last_fullsized":"2024-09-20T22:42:18.092554+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:42.825818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":37,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.092991+0000","last_change":"2024-09-20T22:41:38.152921+0000","last_active":"2024-09-20T22:42:18.092991+0000","last_peered":"2024-09-20T22:42:18.092991+0000","last_clean":"2024-09-20T22:42:18.092991+0000","last_became_active":"2024-09-20T22:41:38.152525+0000","last_became_peered":"2024-09-20T22:41:38.152525+0000","last_unstale":"2024-09-20T22:42:18.092991+0000","last_undegraded":"2024-09-20T22:42:18.092991+0000","last_fullsized":"2024-09-20T22:42:18.092991+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:37.641089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"69'2","reported_seq":47,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.548637+0000","last_change":"2024-09-20T22:41:40.154106+0000","last_active":"2024-09-20T22:42:18.548637+0000","last_peered":"2024-09-20T22:42:18.548637+0000","last_clean":"2024-09-20T22:42:18.548637+0000","last_became_active":"2024-09-20T22:41:37.148479+0000","last_became_peered":"2024-09-20T22:41:37.148479+0000","last_unstale":"2024-09-20T22:42:18.548637+0000","last_undegraded":"2024-09-20T22:42:18.548637+0000","last_fullsized":"2024-09-20T22:42:18.548637+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:36.130684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:36.130684+0000","last_clean_scrub_stamp":"2024-09-20T22:41:36.130684+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:48.204536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087280599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":29,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.091602+0000","last_change":"2024-09-20T22:41:42.179979+0000","last_active":"2024-09-20T22:42:18.091602+0000","last_peered":"2024-09-20T22:42:18.091602+0000","last_clean":"2024-09-20T22:42:18.091602+0000","last_became_active":"2024-09-20T22:41:42.179742+0000","last_became_peered":"2024-09-20T22:41:42.179742+0000","last_unstale":"2024-09-20T22:42:18.091602+0000","last_undegraded":"2024-09-20T22:42:18.091602+0000","last_fullsized":"2024-09-20T22:42:18.091602+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:03.273567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.254628+0000","last_change":"2024-09-20T22:41:44.192602+0000","last_active":"2024-09-20T22:42:18.254628+0000","last_peered":"2024-09-20T22:42:18.254628+0000","last_clean":"2024-09-20T22:42:18.254628+0000","last_became_active":"2024-09-20T22:41:44.192128+0000","last_became_peered":"2024-09-20T22:41:44.192128+0000","last_unstale":"2024-09-20T22:42:18.254628+0000","last_undegraded":"2024-09-20T22:42:18.254628+0000","last_fullsized":"2024-09-20T22:42:18.254628+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:46.416059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"74'13","reported_seq":60,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.092319+0000","last_change":"2024-09-20T22:41:40.169777+0000","last_active":"2024-09-20T22:42:18.092319+0000","last_peered":"2024-09-20T22:42:18.092319+0000","last_clean":"2024-09-20T22:42:18.092319+0000","last_became_active":"2024-09-20T22:41:40.169473+0000","last_became_peered":"2024-09-20T22:41:40.169473+0000","last_unstale":"2024-09-20T22:42:18.092319+0000","last_undegraded":"2024-09-20T22:42:18.092319+0000","last_fullsized":"2024-09-20T22:42:18.092319+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:48.626103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":37,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.091334+0000","last_change":"2024-09-20T22:41:38.157113+0000","last_active":"2024-09-20T22:42:18.091334+0000","last_peered":"2024-09-20T22:42:18.091334+0000","last_clean":"2024-09-20T22:42:18.091334+0000","last_became_active":"2024-09-20T22:41:38.156837+0000","last_became_peered":"2024-09-20T22:41:38.156837+0000","last_unstale":"2024-09-20T22:42:18.091334+0000","last_undegraded":"2024-09-20T22:42:18.091334+0000","last_fullsized":"2024-09-20T22:42:18.091334+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:46.990363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"67'1","reported_seq":46,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.318042+0000","last_change":"2024-09-20T22:41:40.162396+0000","last_active":"2024-09-20T22:42:18.318042+0000","last_peered":"2024-09-20T22:42:18.318042+0000","last_clean":"2024-09-20T22:42:18.318042+0000","last_became_active":"2024-09-20T22:41:37.148165+0000","last_became_peered":"2024-09-20T22:41:37.148165+0000","last_unstale":"2024-09-20T22:42:18.318042+0000","last_undegraded":"2024-09-20T22:42:18.318042+0000","last_fullsized":"2024-09-20T22:42:18.318042+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:36.130684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:36.130684+0000","last_clean_scrub_stamp":"2024-09-20T22:41:36.130684+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:54.327893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001301459,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":29,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.318398+0000","last_change":"2024-09-20T22:41:42.185412+0000","last_active":"2024-09-20T22:42:18.318398+0000","last_peered":"2024-09-20T22:42:18.318398+0000","last_clean":"2024-09-20T22:42:18.318398+0000","last_became_active":"2024-09-20T22:41:42.185112+0000","last_became_peered":"2024-09-20T22:41:42.185112+0000","last_unstale":"2024-09-20T22:42:18.318398+0000","last_undegraded":"2024-09-20T22:42:18.318398+0000","last_fullsized":"2024-09-20T22:42:18.318398+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:05.899621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.465592+0000","last_change":"2024-09-20T22:41:44.830804+0000","last_active":"2024-09-20T22:42:18.465592+0000","last_peered":"2024-09-20T22:42:18.465592+0000","last_clean":"2024-09-20T22:42:18.465592+0000","last_became_active":"2024-09-20T22:41:44.830515+0000","last_became_peered":"2024-09-20T22:41:44.830515+0000","last_unstale":"2024-09-20T22:42:18.465592+0000","last_undegraded":"2024-09-20T22:42:18.465592+0000","last_fullsized":"2024-09-20T22:42:18.465592+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:04.958838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"74'12","reported_seq":51,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.091925+0000","last_change":"2024-09-20T22:41:40.167483+0000","last_active":"2024-09-20T22:42:18.091925+0000","last_peered":"2024-09-20T22:42:18.091925+0000","last_clean":"2024-09-20T22:42:18.091925+0000","last_became_active":"2024-09-20T22:41:40.167169+0000","last_became_peered":"2024-09-20T22:41:40.167169+0000","last_unstale":"2024-09-20T22:42:18.091925+0000","last_undegraded":"2024-09-20T22:42:18.091925+0000","last_fullsized":"2024-09-20T22:42:18.091925+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:48.803800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":37,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.091015+0000","last_change":"2024-09-20T22:41:38.153448+0000","last_active":"2024-09-20T22:42:18.091015+0000","last_peered":"2024-09-20T22:42:18.091015+0000","last_clean":"2024-09-20T22:42:18.091015+0000","last_became_active":"2024-09-20T22:41:38.153058+0000","last_became_peered":"2024-09-20T22:41:38.153058+0000","last_unstale":"2024-09-20T22:42:18.091015+0000","last_undegraded":"2024-09-20T22:42:18.091015+0000","last_fullsized":"2024-09-20T22:42:18.091015+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:04.425050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"75'5","reported_seq":83,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.465214+0000","last_change":"2024-09-20T22:41:40.251762+0000","last_active":"2024-09-20T22:42:18.465214+0000","last_peered":"2024-09-20T22:42:18.465214+0000","last_clean":"2024-09-20T22:42:18.465214+0000","last_became_active":"2024-09-20T22:41:37.148383+0000","last_became_peered":"2024-09-20T22:41:37.148383+0000","last_unstale":"2024-09-20T22:42:18.465214+0000","last_undegraded":"2024-09-20T22:42:18.465214+0000","last_fullsized":"2024-09-20T22:42:18.465214+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:36.130684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:36.130684+0000","last_clean_scrub_stamp":"2024-09-20T22:41:36.130684+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:04.208717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00078631700000000003,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":35,"num_read_kb":30,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":29,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.549126+0000","last_change":"2024-09-20T22:41:42.187618+0000","last_active":"2024-09-20T22:42:18.549126+0000","last_peered":"2024-09-20T22:42:18.549126+0000","last_clean":"2024-09-20T22:42:18.549126+0000","last_became_active":"2024-09-20T22:41:42.187346+0000","last_became_peered":"2024-09-20T22:41:42.187346+0000","last_unstale":"2024-09-20T22:42:18.549126+0000","last_undegraded":"2024-09-20T22:42:18.549126+0000","last_fullsized":"2024-09-20T22:42:18.549126+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:58.427580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.092040+0000","last_change":"2024-09-20T22:41:44.199301+0000","last_active":"2024-09-20T22:42:18.092040+0000","last_peered":"2024-09-20T22:42:18.092040+0000","last_clean":"2024-09-20T22:42:18.092040+0000","last_became_active":"2024-09-20T22:41:44.199046+0000","last_became_peered":"2024-09-20T22:41:44.199046+0000","last_unstale":"2024-09-20T22:42:18.092040+0000","last_undegraded":"2024-09-20T22:42:18.092040+0000","last_fullsized":"2024-09-20T22:42:18.092040+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:43.190792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"74'16","reported_seq":62,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-20T22:42:13.209804+0000","last_change":"2024-09-20T22:41:40.253524+0000","last_active":"2024-09-20T22:42:13.209804+0000","last_peered":"2024-09-20T22:42:13.209804+0000","last_clean":"2024-09-20T22:42:13.209804+0000","last_became_active":"2024-09-20T22:41:40.252933+0000","last_became_peered":"2024-09-20T22:41:40.252933+0000","last_unstale":"2024-09-20T22:42:13.209804+0000","last_undegraded":"2024-09-20T22:42:13.209804+0000","last_fullsized":"2024-09-20T22:42:13.209804+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:54.546845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"67'1","reported_seq":43,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.254369+0000","last_change":"2024-09-20T22:41:38.159706+0000","last_active":"2024-09-20T22:42:18.254369+0000","last_peered":"2024-09-20T22:42:18.254369+0000","last_clean":"2024-09-20T22:42:18.254369+0000","last_became_active":"2024-09-20T22:41:38.158917+0000","last_became_peered":"2024-09-20T22:41:38.158917+0000","last_unstale":"2024-09-20T22:42:18.254369+0000","last_undegraded":"2024-09-20T22:42:18.254369+0000","last_fullsized":"2024-09-20T22:42:18.254369+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:03.995423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"77'128","reported_seq":294,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.465364+0000","last_change":"2024-09-20T22:41:43.408723+0000","last_active":"2024-09-20T22:42:18.465364+0000","last_peered":"2024-09-20T22:42:18.465364+0000","last_clean":"2024-09-20T22:42:18.465364+0000","last_became_active":"2024-09-20T22:41:31.091135+0000","last_became_peered":"2024-09-20T22:41:31.091135+0000","last_unstale":"2024-09-20T22:42:18.465364+0000","last_undegraded":"2024-09-20T22:42:18.465364+0000","last_fullsized":"2024-09-20T22:42:18.465364+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'117","last_scrub_stamp":"2024-09-20T22:41:43.408644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.408644+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:48.125134+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":29,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.465620+0000","last_change":"2024-09-20T22:41:42.185135+0000","last_active":"2024-09-20T22:42:18.465620+0000","last_peered":"2024-09-20T22:42:18.465620+0000","last_clean":"2024-09-20T22:42:18.465620+0000","last_became_active":"2024-09-20T22:41:42.184387+0000","last_became_peered":"2024-09-20T22:41:42.184387+0000","last_unstale":"2024-09-20T22:42:18.465620+0000","last_undegraded":"2024-09-20T22:42:18.465620+0000","last_fullsized":"2024-09-20T22:42:18.465620+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:00.112877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.548983+0000","last_change":"2024-09-20T22:41:44.191113+0000","last_active":"2024-09-20T22:42:18.548983+0000","last_peered":"2024-09-20T22:42:18.548983+0000","last_clean":"2024-09-20T22:42:18.548983+0000","last_became_active":"2024-09-20T22:41:44.190658+0000","last_became_peered":"2024-09-20T22:41:44.190658+0000","last_unstale":"2024-09-20T22:42:18.548983+0000","last_undegraded":"2024-09-20T22:42:18.548983+0000","last_fullsized":"2024-09-20T22:42:18.548983+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:44.921190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"74'11","reported_seq":52,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.092699+0000","last_change":"2024-09-20T22:41:40.169542+0000","last_active":"2024-09-20T22:42:18.092699+0000","last_peered":"2024-09-20T22:42:18.092699+0000","last_clean":"2024-09-20T22:42:18.092699+0000","last_became_active":"2024-09-20T22:41:40.168942+0000","last_became_peered":"2024-09-20T22:41:40.168942+0000","last_unstale":"2024-09-20T22:42:18.092699+0000","last_undegraded":"2024-09-20T22:42:18.092699+0000","last_fullsized":"2024-09-20T22:42:18.092699+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:58.224721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":37,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.093019+0000","last_change":"2024-09-20T22:41:38.161801+0000","last_active":"2024-09-20T22:42:18.093019+0000","last_peered":"2024-09-20T22:42:18.093019+0000","last_clean":"2024-09-20T22:42:18.093019+0000","last_became_active":"2024-09-20T22:41:38.161500+0000","last_became_peered":"2024-09-20T22:41:38.161500+0000","last_unstale":"2024-09-20T22:42:18.093019+0000","last_undegraded":"2024-09-20T22:42:18.093019+0000","last_fullsized":"2024-09-20T22:42:18.093019+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:48.636045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":29,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.548988+0000","last_change":"2024-09-20T22:41:42.183601+0000","last_active":"2024-09-20T22:42:18.548988+0000","last_peered":"2024-09-20T22:42:18.548988+0000","last_clean":"2024-09-20T22:42:18.548988+0000","last_became_active":"2024-09-20T22:41:42.183302+0000","last_became_peered":"2024-09-20T22:41:42.183302+0000","last_unstale":"2024-09-20T22:42:18.548988+0000","last_undegraded":"2024-09-20T22:42:18.548988+0000","last_fullsized":"2024-09-20T22:42:18.548988+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:10.335337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.254396+0000","last_change":"2024-09-20T22:41:44.832295+0000","last_active":"2024-09-20T22:42:18.254396+0000","last_peered":"2024-09-20T22:42:18.254396+0000","last_clean":"2024-09-20T22:42:18.254396+0000","last_became_active":"2024-09-20T22:41:44.832009+0000","last_became_peered":"2024-09-20T22:41:44.832009+0000","last_unstale":"2024-09-20T22:42:18.254396+0000","last_undegraded":"2024-09-20T22:42:18.254396+0000","last_fullsized":"2024-09-20T22:42:18.254396+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:56.047489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"74'17","reported_seq":61,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.092915+0000","last_change":"2024-09-20T22:41:40.169498+0000","last_active":"2024-09-20T22:42:18.092915+0000","last_peered":"2024-09-20T22:42:18.092915+0000","last_clean":"2024-09-20T22:42:18.092915+0000","last_became_active":"2024-09-20T22:41:40.169192+0000","last_became_peered":"2024-09-20T22:41:40.169192+0000","last_unstale":"2024-09-20T22:42:18.092915+0000","last_undegraded":"2024-09-20T22:42:18.092915+0000","last_fullsized":"2024-09-20T22:42:18.092915+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:40.654708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":35,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-20T22:42:13.093468+0000","last_change":"2024-09-20T22:41:38.154220+0000","last_active":"2024-09-20T22:42:13.093468+0000","last_peered":"2024-09-20T22:42:13.093468+0000","last_clean":"2024-09-20T22:42:13.093468+0000","last_became_active":"2024-09-20T22:41:38.153926+0000","last_became_peered":"2024-09-20T22:41:38.153926+0000","last_unstale":"2024-09-20T22:42:13.093468+0000","last_undegraded":"2024-09-20T22:42:13.093468+0000","last_fullsized":"2024-09-20T22:42:13.093468+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:54.196177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":29,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.092130+0000","last_change":"2024-09-20T22:41:42.183482+0000","last_active":"2024-09-20T22:42:18.092130+0000","last_peered":"2024-09-20T22:42:18.092130+0000","last_clean":"2024-09-20T22:42:18.092130+0000","last_became_active":"2024-09-20T22:41:42.183225+0000","last_became_peered":"2024-09-20T22:41:42.183225+0000","last_unstale":"2024-09-20T22:42:18.092130+0000","last_undegraded":"2024-09-20T22:42:18.092130+0000","last_fullsized":"2024-09-20T22:42:18.092130+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:10.228360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.318441+0000","last_change":"2024-09-20T22:41:44.197855+0000","last_active":"2024-09-20T22:42:18.318441+0000","last_peered":"2024-09-20T22:42:18.318441+0000","last_clean":"2024-09-20T22:42:18.318441+0000","last_became_active":"2024-09-20T22:41:44.197561+0000","last_became_peered":"2024-09-20T22:41:44.197561+0000","last_unstale":"2024-09-20T22:42:18.318441+0000","last_undegraded":"2024-09-20T22:42:18.318441+0000","last_fullsized":"2024-09-20T22:42:18.318441+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:45.431229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"74'10","reported_seq":48,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.092575+0000","last_change":"2024-09-20T22:41:40.162973+0000","last_active":"2024-09-20T22:42:18.092575+0000","last_peered":"2024-09-20T22:42:18.092575+0000","last_clean":"2024-09-20T22:42:18.092575+0000","last_became_active":"2024-09-20T22:41:40.162494+0000","last_became_peered":"2024-09-20T22:41:40.162494+0000","last_unstale":"2024-09-20T22:42:18.092575+0000","last_undegraded":"2024-09-20T22:42:18.092575+0000","last_fullsized":"2024-09-20T22:42:18.092575+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:49.601652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":37,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.253850+0000","last_change":"2024-09-20T22:41:38.154861+0000","last_active":"2024-09-20T22:42:18.253850+0000","last_peered":"2024-09-20T22:42:18.253850+0000","last_clean":"2024-09-20T22:42:18.253850+0000","last_became_active":"2024-09-20T22:41:38.154636+0000","last_became_peered":"2024-09-20T22:41:38.154636+0000","last_unstale":"2024-09-20T22:42:18.253850+0000","last_undegraded":"2024-09-20T22:42:18.253850+0000","last_fullsized":"2024-09-20T22:42:18.253850+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:45.035202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"74'8","reported_seq":46,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.318307+0000","last_change":"2024-09-20T22:41:42.186528+0000","last_active":"2024-09-20T22:42:18.318307+0000","last_peered":"2024-09-20T22:42:18.318307+0000","last_clean":"2024-09-20T22:42:18.318307+0000","last_became_active":"2024-09-20T22:41:42.185863+0000","last_became_peered":"2024-09-20T22:41:42.185863+0000","last_unstale":"2024-09-20T22:42:18.318307+0000","last_undegraded":"2024-09-20T22:42:18.318307+0000","last_fullsized":"2024-09-20T22:42:18.318307+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:00.762928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.092467+0000","last_change":"2024-09-20T22:41:44.195767+0000","last_active":"2024-09-20T22:42:18.092467+0000","last_peered":"2024-09-20T22:42:18.092467+0000","last_clean":"2024-09-20T22:42:18.092467+0000","last_became_active":"2024-09-20T22:41:44.195160+0000","last_became_peered":"2024-09-20T22:41:44.195160+0000","last_unstale":"2024-09-20T22:42:18.092467+0000","last_undegraded":"2024-09-20T22:42:18.092467+0000","last_fullsized":"2024-09-20T22:42:18.092467+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:07.188986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"74'9","reported_seq":49,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.091485+0000","last_change":"2024-09-20T22:41:40.163774+0000","last_active":"2024-09-20T22:42:18.091485+0000","last_peered":"2024-09-20T22:42:18.091485+0000","last_clean":"2024-09-20T22:42:18.091485+0000","last_became_active":"2024-09-20T22:41:40.163495+0000","last_became_peered":"2024-09-20T22:41:40.163495+0000","last_unstale":"2024-09-20T22:42:18.091485+0000","last_undegraded":"2024-09-20T22:42:18.091485+0000","last_fullsized":"2024-09-20T22:42:18.091485+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:41.614897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":37,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.548693+0000","last_change":"2024-09-20T22:41:38.156470+0000","last_active":"2024-09-20T22:42:18.548693+0000","last_peered":"2024-09-20T22:42:18.548693+0000","last_clean":"2024-09-20T22:42:18.548693+0000","last_became_active":"2024-09-20T22:41:38.155953+0000","last_became_peered":"2024-09-20T22:41:38.155953+0000","last_unstale":"2024-09-20T22:42:18.548693+0000","last_undegraded":"2024-09-20T22:42:18.548693+0000","last_fullsized":"2024-09-20T22:42:18.548693+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:54.057798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":29,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.318125+0000","last_change":"2024-09-20T22:41:42.183269+0000","last_active":"2024-09-20T22:42:18.318125+0000","last_peered":"2024-09-20T22:42:18.318125+0000","last_clean":"2024-09-20T22:42:18.318125+0000","last_became_active":"2024-09-20T22:41:42.182999+0000","last_became_peered":"2024-09-20T22:41:42.182999+0000","last_unstale":"2024-09-20T22:42:18.318125+0000","last_undegraded":"2024-09-20T22:42:18.318125+0000","last_fullsized":"2024-09-20T22:42:18.318125+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:57.955307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.090929+0000","last_change":"2024-09-20T22:41:44.196969+0000","last_active":"2024-09-20T22:42:18.090929+0000","last_peered":"2024-09-20T22:42:18.090929+0000","last_clean":"2024-09-20T22:42:18.090929+0000","last_became_active":"2024-09-20T22:41:44.196633+0000","last_became_peered":"2024-09-20T22:41:44.196633+0000","last_unstale":"2024-09-20T22:42:18.090929+0000","last_undegraded":"2024-09-20T22:42:18.090929+0000","last_fullsized":"2024-09-20T22:42:18.090929+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:03.513323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"74'19","reported_seq":62,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-20T22:42:13.093286+0000","last_change":"2024-09-20T22:41:40.252962+0000","last_active":"2024-09-20T22:42:13.093286+0000","last_peered":"2024-09-20T22:42:13.093286+0000","last_clean":"2024-09-20T22:42:13.093286+0000","last_became_active":"2024-09-20T22:41:40.252011+0000","last_became_peered":"2024-09-20T22:41:40.252011+0000","last_unstale":"2024-09-20T22:42:13.093286+0000","last_undegraded":"2024-09-20T22:42:13.093286+0000","last_fullsized":"2024-09-20T22:42:13.093286+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:51.633432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":37,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.465861+0000","last_change":"2024-09-20T22:41:38.161800+0000","last_active":"2024-09-20T22:42:18.465861+0000","last_peered":"2024-09-20T22:42:18.465861+0000","last_clean":"2024-09-20T22:42:18.465861+0000","last_became_active":"2024-09-20T22:41:38.161524+0000","last_became_peered":"2024-09-20T22:41:38.161524+0000","last_unstale":"2024-09-20T22:42:18.465861+0000","last_undegraded":"2024-09-20T22:42:18.465861+0000","last_fullsized":"2024-09-20T22:42:18.465861+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:07.121209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":29,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.318184+0000","last_change":"2024-09-20T22:41:42.184605+0000","last_active":"2024-09-20T22:42:18.318184+0000","last_peered":"2024-09-20T22:42:18.318184+0000","last_clean":"2024-09-20T22:42:18.318184+0000","last_became_active":"2024-09-20T22:41:42.184342+0000","last_became_peered":"2024-09-20T22:41:42.184342+0000","last_unstale":"2024-09-20T22:42:18.318184+0000","last_undegraded":"2024-09-20T22:42:18.318184+0000","last_fullsized":"2024-09-20T22:42:18.318184+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:03.475959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.465304+0000","last_change":"2024-09-20T22:41:44.199874+0000","last_active":"2024-09-20T22:42:18.465304+0000","last_peered":"2024-09-20T22:42:18.465304+0000","last_clean":"2024-09-20T22:42:18.465304+0000","last_became_active":"2024-09-20T22:41:44.199395+0000","last_became_peered":"2024-09-20T22:41:44.199395+0000","last_unstale":"2024-09-20T22:42:18.465304+0000","last_undegraded":"2024-09-20T22:42:18.465304+0000","last_fullsized":"2024-09-20T22:42:18.465304+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:44.605488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"74'12","reported_seq":56,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.092473+0000","last_change":"2024-09-20T22:41:40.170746+0000","last_active":"2024-09-20T22:42:18.092473+0000","last_peered":"2024-09-20T22:42:18.092473+0000","last_clean":"2024-09-20T22:42:18.092473+0000","last_became_active":"2024-09-20T22:41:40.170133+0000","last_became_peered":"2024-09-20T22:41:40.170133+0000","last_unstale":"2024-09-20T22:42:18.092473+0000","last_undegraded":"2024-09-20T22:42:18.092473+0000","last_fullsized":"2024-09-20T22:42:18.092473+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:39.588868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":37,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.464779+0000","last_change":"2024-09-20T22:41:38.163612+0000","last_active":"2024-09-20T22:42:18.464779+0000","last_peered":"2024-09-20T22:42:18.464779+0000","last_clean":"2024-09-20T22:42:18.464779+0000","last_became_active":"2024-09-20T22:41:38.163292+0000","last_became_peered":"2024-09-20T22:41:38.163292+0000","last_unstale":"2024-09-20T22:42:18.464779+0000","last_undegraded":"2024-09-20T22:42:18.464779+0000","last_fullsized":"2024-09-20T22:42:18.464779+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:45.666357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":29,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.317960+0000","last_change":"2024-09-20T22:41:42.182102+0000","last_active":"2024-09-20T22:42:18.317960+0000","last_peered":"2024-09-20T22:42:18.317960+0000","last_clean":"2024-09-20T22:42:18.317960+0000","last_became_active":"2024-09-20T22:41:42.181811+0000","last_became_peered":"2024-09-20T22:41:42.181811+0000","last_unstale":"2024-09-20T22:42:18.317960+0000","last_undegraded":"2024-09-20T22:42:18.317960+0000","last_fullsized":"2024-09-20T22:42:18.317960+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:52.248246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.254327+0000","last_change":"2024-09-20T22:41:44.193432+0000","last_active":"2024-09-20T22:42:18.254327+0000","last_peered":"2024-09-20T22:42:18.254327+0000","last_clean":"2024-09-20T22:42:18.254327+0000","last_became_active":"2024-09-20T22:41:44.192609+0000","last_became_peered":"2024-09-20T22:41:44.192609+0000","last_unstale":"2024-09-20T22:42:18.254327+0000","last_undegraded":"2024-09-20T22:42:18.254327+0000","last_fullsized":"2024-09-20T22:42:18.254327+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:50.351142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"74'15","reported_seq":58,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.548178+0000","last_change":"2024-09-20T22:41:40.252427+0000","last_active":"2024-09-20T22:42:18.548178+0000","last_peered":"2024-09-20T22:42:18.548178+0000","last_clean":"2024-09-20T22:42:18.548178+0000","last_became_active":"2024-09-20T22:41:40.252113+0000","last_became_peered":"2024-09-20T22:41:40.252113+0000","last_unstale":"2024-09-20T22:42:18.548178+0000","last_undegraded":"2024-09-20T22:42:18.548178+0000","last_fullsized":"2024-09-20T22:42:18.548178+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:06.207870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'2","reported_seq":49,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.464888+0000","last_change":"2024-09-20T22:41:38.163593+0000","last_active":"2024-09-20T22:42:18.464888+0000","last_peered":"2024-09-20T22:42:18.464888+0000","last_clean":"2024-09-20T22:42:18.464888+0000","last_became_active":"2024-09-20T22:41:38.163219+0000","last_became_peered":"2024-09-20T22:41:38.163219+0000","last_unstale":"2024-09-20T22:42:18.464888+0000","last_undegraded":"2024-09-20T22:42:18.464888+0000","last_fullsized":"2024-09-20T22:42:18.464888+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:02.881623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"74'8","reported_seq":46,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.465161+0000","last_change":"2024-09-20T22:41:42.188223+0000","last_active":"2024-09-20T22:42:18.465161+0000","last_peered":"2024-09-20T22:42:18.465161+0000","last_clean":"2024-09-20T22:42:18.465161+0000","last_became_active":"2024-09-20T22:41:42.186635+0000","last_became_peered":"2024-09-20T22:41:42.186635+0000","last_unstale":"2024-09-20T22:42:18.465161+0000","last_undegraded":"2024-09-20T22:42:18.465161+0000","last_fullsized":"2024-09-20T22:42:18.465161+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:52.120303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.548397+0000","last_change":"2024-09-20T22:41:44.831435+0000","last_active":"2024-09-20T22:42:18.548397+0000","last_peered":"2024-09-20T22:42:18.548397+0000","last_clean":"2024-09-20T22:42:18.548397+0000","last_became_active":"2024-09-20T22:41:44.830556+0000","last_became_peered":"2024-09-20T22:41:44.830556+0000","last_unstale":"2024-09-20T22:42:18.548397+0000","last_undegraded":"2024-09-20T22:42:18.548397+0000","last_fullsized":"2024-09-20T22:42:18.548397+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:08.027826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":35,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-20T22:42:13.093186+0000","last_change":"2024-09-20T22:41:38.159437+0000","last_active":"2024-09-20T22:42:13.093186+0000","last_peered":"2024-09-20T22:42:13.093186+0000","last_clean":"2024-09-20T22:42:13.093186+0000","last_became_active":"2024-09-20T22:41:38.159139+0000","last_became_peered":"2024-09-20T22:41:38.159139+0000","last_unstale":"2024-09-20T22:42:13.093186+0000","last_undegraded":"2024-09-20T22:42:13.093186+0000","last_fullsized":"2024-09-20T22:42:13.093186+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:47.658845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"74'6","reported_seq":42,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.254853+0000","last_change":"2024-09-20T22:41:40.252310+0000","last_active":"2024-09-20T22:42:18.254853+0000","last_peered":"2024-09-20T22:42:18.254853+0000","last_clean":"2024-09-20T22:42:18.254853+0000","last_became_active":"2024-09-20T22:41:40.251867+0000","last_became_peered":"2024-09-20T22:41:40.251867+0000","last_unstale":"2024-09-20T22:42:18.254853+0000","last_undegraded":"2024-09-20T22:42:18.254853+0000","last_fullsized":"2024-09-20T22:42:18.254853+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:40.015664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":29,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.548937+0000","last_change":"2024-09-20T22:41:42.185218+0000","last_active":"2024-09-20T22:42:18.548937+0000","last_peered":"2024-09-20T22:42:18.548937+0000","last_clean":"2024-09-20T22:42:18.548937+0000","last_became_active":"2024-09-20T22:41:42.184930+0000","last_became_peered":"2024-09-20T22:41:42.184930+0000","last_unstale":"2024-09-20T22:42:18.548937+0000","last_undegraded":"2024-09-20T22:42:18.548937+0000","last_fullsized":"2024-09-20T22:42:18.548937+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:47.818365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"74'1","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.465557+0000","last_change":"2024-09-20T22:41:44.832085+0000","last_active":"2024-09-20T22:42:18.465557+0000","last_peered":"2024-09-20T22:42:18.465557+0000","last_clean":"2024-09-20T22:42:18.465557+0000","last_became_active":"2024-09-20T22:41:44.831805+0000","last_became_peered":"2024-09-20T22:41:44.831805+0000","last_unstale":"2024-09-20T22:42:18.465557+0000","last_undegraded":"2024-09-20T22:42:18.465557+0000","last_fullsized":"2024-09-20T22:42:18.465557+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:54.445212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"74'9","reported_seq":49,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.092560+0000","last_change":"2024-09-20T22:41:40.253633+0000","last_active":"2024-09-20T22:42:18.092560+0000","last_peered":"2024-09-20T22:42:18.092560+0000","last_clean":"2024-09-20T22:42:18.092560+0000","last_became_active":"2024-09-20T22:41:40.253280+0000","last_became_peered":"2024-09-20T22:41:40.253280+0000","last_unstale":"2024-09-20T22:42:18.092560+0000","last_undegraded":"2024-09-20T22:42:18.092560+0000","last_fullsized":"2024-09-20T22:42:18.092560+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:39.928882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":37,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.464667+0000","last_change":"2024-09-20T22:41:38.156773+0000","last_active":"2024-09-20T22:42:18.464667+0000","last_peered":"2024-09-20T22:42:18.464667+0000","last_clean":"2024-09-20T22:42:18.464667+0000","last_became_active":"2024-09-20T22:41:38.156497+0000","last_became_peered":"2024-09-20T22:41:38.156497+0000","last_unstale":"2024-09-20T22:42:18.464667+0000","last_undegraded":"2024-09-20T22:42:18.464667+0000","last_fullsized":"2024-09-20T22:42:18.464667+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:01.943269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":29,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.254776+0000","last_change":"2024-09-20T22:41:42.188166+0000","last_active":"2024-09-20T22:42:18.254776+0000","last_peered":"2024-09-20T22:42:18.254776+0000","last_clean":"2024-09-20T22:42:18.254776+0000","last_became_active":"2024-09-20T22:41:42.187706+0000","last_became_peered":"2024-09-20T22:41:42.187706+0000","last_unstale":"2024-09-20T22:42:18.254776+0000","last_undegraded":"2024-09-20T22:42:18.254776+0000","last_fullsized":"2024-09-20T22:42:18.254776+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:47.610076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.318219+0000","last_change":"2024-09-20T22:41:44.201642+0000","last_active":"2024-09-20T22:42:18.318219+0000","last_peered":"2024-09-20T22:42:18.318219+0000","last_clean":"2024-09-20T22:42:18.318219+0000","last_became_active":"2024-09-20T22:41:44.201433+0000","last_became_peered":"2024-09-20T22:41:44.201433+0000","last_unstale":"2024-09-20T22:42:18.318219+0000","last_undegraded":"2024-09-20T22:42:18.318219+0000","last_fullsized":"2024-09-20T22:42:18.318219+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:55.257671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"74'9","reported_seq":49,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.548923+0000","last_change":"2024-09-20T22:41:40.252840+0000","last_active":"2024-09-20T22:42:18.548923+0000","last_peered":"2024-09-20T22:42:18.548923+0000","last_clean":"2024-09-20T22:42:18.548923+0000","last_became_active":"2024-09-20T22:41:40.252486+0000","last_became_peered":"2024-09-20T22:41:40.252486+0000","last_unstale":"2024-09-20T22:42:18.548923+0000","last_undegraded":"2024-09-20T22:42:18.548923+0000","last_fullsized":"2024-09-20T22:42:18.548923+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:51.443985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":37,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.091310+0000","last_change":"2024-09-20T22:41:38.158479+0000","last_active":"2024-09-20T22:42:18.091310+0000","last_peered":"2024-09-20T22:42:18.091310+0000","last_clean":"2024-09-20T22:42:18.091310+0000","last_became_active":"2024-09-20T22:41:38.158146+0000","last_became_peered":"2024-09-20T22:41:38.158146+0000","last_unstale":"2024-09-20T22:42:18.091310+0000","last_undegraded":"2024-09-20T22:42:18.091310+0000","last_fullsized":"2024-09-20T22:42:18.091310+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:55.658232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"74'8","reported_seq":46,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.254494+0000","last_change":"2024-09-20T22:41:42.186630+0000","last_active":"2024-09-20T22:42:18.254494+0000","last_peered":"2024-09-20T22:42:18.254494+0000","last_clean":"2024-09-20T22:42:18.254494+0000","last_became_active":"2024-09-20T22:41:42.186352+0000","last_became_peered":"2024-09-20T22:41:42.186352+0000","last_unstale":"2024-09-20T22:42:18.254494+0000","last_undegraded":"2024-09-20T22:42:18.254494+0000","last_fullsized":"2024-09-20T22:42:18.254494+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:48.572027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.093209+0000","last_change":"2024-09-20T22:41:44.194510+0000","last_active":"2024-09-20T22:42:18.093209+0000","last_peered":"2024-09-20T22:42:18.093209+0000","last_clean":"2024-09-20T22:42:18.093209+0000","last_became_active":"2024-09-20T22:41:44.194053+0000","last_became_peered":"2024-09-20T22:41:44.194053+0000","last_unstale":"2024-09-20T22:42:18.093209+0000","last_undegraded":"2024-09-20T22:42:18.093209+0000","last_fullsized":"2024-09-20T22:42:18.093209+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:01.207219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"74'10","reported_seq":48,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.254659+0000","last_change":"2024-09-20T22:41:40.253880+0000","last_active":"2024-09-20T22:42:18.254659+0000","last_peered":"2024-09-20T22:42:18.254659+0000","last_clean":"2024-09-20T22:42:18.254659+0000","last_became_active":"2024-09-20T22:41:40.253614+0000","last_became_peered":"2024-09-20T22:41:40.253614+0000","last_unstale":"2024-09-20T22:42:18.254659+0000","last_undegraded":"2024-09-20T22:42:18.254659+0000","last_fullsized":"2024-09-20T22:42:18.254659+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:47.554109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":37,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.464750+0000","last_change":"2024-09-20T22:41:38.158008+0000","last_active":"2024-09-20T22:42:18.464750+0000","last_peered":"2024-09-20T22:42:18.464750+0000","last_clean":"2024-09-20T22:42:18.464750+0000","last_became_active":"2024-09-20T22:41:38.157647+0000","last_became_peered":"2024-09-20T22:41:38.157647+0000","last_unstale":"2024-09-20T22:42:18.464750+0000","last_undegraded":"2024-09-20T22:42:18.464750+0000","last_fullsized":"2024-09-20T22:42:18.464750+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:57.980474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"74'8","reported_seq":46,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.548788+0000","last_change":"2024-09-20T22:41:42.189757+0000","last_active":"2024-09-20T22:42:18.548788+0000","last_peered":"2024-09-20T22:42:18.548788+0000","last_clean":"2024-09-20T22:42:18.548788+0000","last_became_active":"2024-09-20T22:41:42.189451+0000","last_became_peered":"2024-09-20T22:41:42.189451+0000","last_unstale":"2024-09-20T22:42:18.548788+0000","last_undegraded":"2024-09-20T22:42:18.548788+0000","last_fullsized":"2024-09-20T22:42:18.548788+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:43.846079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.091788+0000","last_change":"2024-09-20T22:41:44.191549+0000","last_active":"2024-09-20T22:42:18.091788+0000","last_peered":"2024-09-20T22:42:18.091788+0000","last_clean":"2024-09-20T22:42:18.091788+0000","last_became_active":"2024-09-20T22:41:44.191206+0000","last_became_peered":"2024-09-20T22:41:44.191206+0000","last_unstale":"2024-09-20T22:42:18.091788+0000","last_undegraded":"2024-09-20T22:42:18.091788+0000","last_fullsized":"2024-09-20T22:42:18.091788+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:58.564765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"74'11","reported_seq":52,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.092470+0000","last_change":"2024-09-20T22:41:40.171449+0000","last_active":"2024-09-20T22:42:18.092470+0000","last_peered":"2024-09-20T22:42:18.092470+0000","last_clean":"2024-09-20T22:42:18.092470+0000","last_became_active":"2024-09-20T22:41:40.170035+0000","last_became_peered":"2024-09-20T22:41:40.170035+0000","last_unstale":"2024-09-20T22:42:18.092470+0000","last_undegraded":"2024-09-20T22:42:18.092470+0000","last_fullsized":"2024-09-20T22:42:18.092470+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:49.140455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":37,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.092961+0000","last_change":"2024-09-20T22:41:38.161385+0000","last_active":"2024-09-20T22:42:18.092961+0000","last_peered":"2024-09-20T22:42:18.092961+0000","last_clean":"2024-09-20T22:42:18.092961+0000","last_became_active":"2024-09-20T22:41:38.158142+0000","last_became_peered":"2024-09-20T22:41:38.158142+0000","last_unstale":"2024-09-20T22:42:18.092961+0000","last_undegraded":"2024-09-20T22:42:18.092961+0000","last_fullsized":"2024-09-20T22:42:18.092961+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:51.728688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":29,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.091421+0000","last_change":"2024-09-20T22:41:42.182119+0000","last_active":"2024-09-20T22:42:18.091421+0000","last_peered":"2024-09-20T22:42:18.091421+0000","last_clean":"2024-09-20T22:42:18.091421+0000","last_became_active":"2024-09-20T22:41:42.181736+0000","last_became_peered":"2024-09-20T22:41:42.181736+0000","last_unstale":"2024-09-20T22:42:18.091421+0000","last_undegraded":"2024-09-20T22:42:18.091421+0000","last_fullsized":"2024-09-20T22:42:18.091421+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:02.572377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.254184+0000","last_change":"2024-09-20T22:41:44.193289+0000","last_active":"2024-09-20T22:42:18.254184+0000","last_peered":"2024-09-20T22:42:18.254184+0000","last_clean":"2024-09-20T22:42:18.254184+0000","last_became_active":"2024-09-20T22:41:44.192960+0000","last_became_peered":"2024-09-20T22:41:44.192960+0000","last_unstale":"2024-09-20T22:42:18.254184+0000","last_undegraded":"2024-09-20T22:42:18.254184+0000","last_fullsized":"2024-09-20T22:42:18.254184+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:49.189046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"74'9","reported_seq":49,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.091501+0000","last_change":"2024-09-20T22:41:40.165116+0000","last_active":"2024-09-20T22:42:18.091501+0000","last_peered":"2024-09-20T22:42:18.091501+0000","last_clean":"2024-09-20T22:42:18.091501+0000","last_became_active":"2024-09-20T22:41:40.164796+0000","last_became_peered":"2024-09-20T22:41:40.164796+0000","last_unstale":"2024-09-20T22:42:18.091501+0000","last_undegraded":"2024-09-20T22:42:18.091501+0000","last_fullsized":"2024-09-20T22:42:18.091501+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:48.483009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":37,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.464733+0000","last_change":"2024-09-20T22:41:38.159941+0000","last_active":"2024-09-20T22:42:18.464733+0000","last_peered":"2024-09-20T22:42:18.464733+0000","last_clean":"2024-09-20T22:42:18.464733+0000","last_became_active":"2024-09-20T22:41:38.159657+0000","last_became_peered":"2024-09-20T22:41:38.159657+0000","last_unstale":"2024-09-20T22:42:18.464733+0000","last_undegraded":"2024-09-20T22:42:18.464733+0000","last_fullsized":"2024-09-20T22:42:18.464733+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:53.225190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":29,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.254383+0000","last_change":"2024-09-20T22:41:42.183254+0000","last_active":"2024-09-20T22:42:18.254383+0000","last_peered":"2024-09-20T22:42:18.254383+0000","last_clean":"2024-09-20T22:42:18.254383+0000","last_became_active":"2024-09-20T22:41:42.182966+0000","last_became_peered":"2024-09-20T22:41:42.182966+0000","last_unstale":"2024-09-20T22:42:18.254383+0000","last_undegraded":"2024-09-20T22:42:18.254383+0000","last_fullsized":"2024-09-20T22:42:18.254383+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:51.689726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.090618+0000","last_change":"2024-09-20T22:41:44.194845+0000","last_active":"2024-09-20T22:42:18.090618+0000","last_peered":"2024-09-20T22:42:18.090618+0000","last_clean":"2024-09-20T22:42:18.090618+0000","last_became_active":"2024-09-20T22:41:44.194551+0000","last_became_peered":"2024-09-20T22:41:44.194551+0000","last_unstale":"2024-09-20T22:42:18.090618+0000","last_undegraded":"2024-09-20T22:42:18.090618+0000","last_fullsized":"2024-09-20T22:42:18.090618+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:45.187444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"74'11","reported_seq":52,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.254292+0000","last_change":"2024-09-20T22:41:40.252149+0000","last_active":"2024-09-20T22:42:18.254292+0000","last_peered":"2024-09-20T22:42:18.254292+0000","last_clean":"2024-09-20T22:42:18.254292+0000","last_became_active":"2024-09-20T22:41:40.251829+0000","last_became_peered":"2024-09-20T22:41:40.251829+0000","last_unstale":"2024-09-20T22:42:18.254292+0000","last_undegraded":"2024-09-20T22:42:18.254292+0000","last_fullsized":"2024-09-20T22:42:18.254292+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:56.020491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":37,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.547951+0000","last_change":"2024-09-20T22:41:38.159064+0000","last_active":"2024-09-20T22:42:18.547951+0000","last_peered":"2024-09-20T22:42:18.547951+0000","last_clean":"2024-09-20T22:42:18.547951+0000","last_became_active":"2024-09-20T22:41:38.158701+0000","last_became_peered":"2024-09-20T22:41:38.158701+0000","last_unstale":"2024-09-20T22:42:18.547951+0000","last_undegraded":"2024-09-20T22:42:18.547951+0000","last_fullsized":"2024-09-20T22:42:18.547951+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:47.727493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":29,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.464964+0000","last_change":"2024-09-20T22:41:42.185991+0000","last_active":"2024-09-20T22:42:18.464964+0000","last_peered":"2024-09-20T22:42:18.464964+0000","last_clean":"2024-09-20T22:42:18.464964+0000","last_became_active":"2024-09-20T22:41:42.185296+0000","last_became_peered":"2024-09-20T22:41:42.185296+0000","last_unstale":"2024-09-20T22:42:18.464964+0000","last_undegraded":"2024-09-20T22:42:18.464964+0000","last_fullsized":"2024-09-20T22:42:18.464964+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:55.259880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.254496+0000","last_change":"2024-09-20T22:41:44.832293+0000","last_active":"2024-09-20T22:42:18.254496+0000","last_peered":"2024-09-20T22:42:18.254496+0000","last_clean":"2024-09-20T22:42:18.254496+0000","last_became_active":"2024-09-20T22:41:44.831967+0000","last_became_peered":"2024-09-20T22:41:44.831967+0000","last_unstale":"2024-09-20T22:42:18.254496+0000","last_undegraded":"2024-09-20T22:42:18.254496+0000","last_fullsized":"2024-09-20T22:42:18.254496+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:03.371689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"74'4","reported_seq":39,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.254126+0000","last_change":"2024-09-20T22:41:40.166409+0000","last_active":"2024-09-20T22:42:18.254126+0000","last_peered":"2024-09-20T22:42:18.254126+0000","last_clean":"2024-09-20T22:42:18.254126+0000","last_became_active":"2024-09-20T22:41:40.166123+0000","last_became_peered":"2024-09-20T22:41:40.166123+0000","last_unstale":"2024-09-20T22:42:18.254126+0000","last_undegraded":"2024-09-20T22:42:18.254126+0000","last_fullsized":"2024-09-20T22:42:18.254126+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:52.943341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":37,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.091882+0000","last_change":"2024-09-20T22:41:38.158861+0000","last_active":"2024-09-20T22:42:18.091882+0000","last_peered":"2024-09-20T22:42:18.091882+0000","last_clean":"2024-09-20T22:42:18.091882+0000","last_became_active":"2024-09-20T22:41:38.158347+0000","last_became_peered":"2024-09-20T22:41:38.158347+0000","last_unstale":"2024-09-20T22:42:18.091882+0000","last_undegraded":"2024-09-20T22:42:18.091882+0000","last_fullsized":"2024-09-20T22:42:18.091882+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:44.061004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":27,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-20T22:42:13.093113+0000","last_change":"2024-09-20T22:41:42.187707+0000","last_active":"2024-09-20T22:42:13.093113+0000","last_peered":"2024-09-20T22:42:13.093113+0000","last_clean":"2024-09-20T22:42:13.093113+0000","last_became_active":"2024-09-20T22:41:42.186068+0000","last_became_peered":"2024-09-20T22:41:42.186068+0000","last_unstale":"2024-09-20T22:42:13.093113+0000","last_undegraded":"2024-09-20T22:42:13.093113+0000","last_fullsized":"2024-09-20T22:42:13.093113+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:50.526395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.465190+0000","last_change":"2024-09-20T22:41:44.194011+0000","last_active":"2024-09-20T22:42:18.465190+0000","last_peered":"2024-09-20T22:42:18.465190+0000","last_clean":"2024-09-20T22:42:18.465190+0000","last_became_active":"2024-09-20T22:41:44.193706+0000","last_became_peered":"2024-09-20T22:41:44.193706+0000","last_unstale":"2024-09-20T22:42:18.465190+0000","last_undegraded":"2024-09-20T22:42:18.465190+0000","last_fullsized":"2024-09-20T22:42:18.465190+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:49.835925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.092142+0000","last_change":"2024-09-20T22:41:44.198154+0000","last_active":"2024-09-20T22:42:18.092142+0000","last_peered":"2024-09-20T22:42:18.092142+0000","last_clean":"2024-09-20T22:42:18.092142+0000","last_became_active":"2024-09-20T22:41:44.197855+0000","last_became_peered":"2024-09-20T22:41:44.197855+0000","last_unstale":"2024-09-20T22:42:18.092142+0000","last_undegraded":"2024-09-20T22:42:18.092142+0000","last_fullsized":"2024-09-20T22:42:18.092142+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:56.991123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":37,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.253830+0000","last_change":"2024-09-20T22:41:38.155001+0000","last_active":"2024-09-20T22:42:18.253830+0000","last_peered":"2024-09-20T22:42:18.253830+0000","last_clean":"2024-09-20T22:42:18.253830+0000","last_became_active":"2024-09-20T22:41:38.154703+0000","last_became_peered":"2024-09-20T22:41:38.154703+0000","last_unstale":"2024-09-20T22:42:18.253830+0000","last_undegraded":"2024-09-20T22:42:18.253830+0000","last_fullsized":"2024-09-20T22:42:18.253830+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:44.782808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"74'11","reported_seq":50,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-20T22:42:13.093880+0000","last_change":"2024-09-20T22:41:40.171840+0000","last_active":"2024-09-20T22:42:13.093880+0000","last_peered":"2024-09-20T22:42:13.093880+0000","last_clean":"2024-09-20T22:42:13.093880+0000","last_became_active":"2024-09-20T22:41:40.171256+0000","last_became_peered":"2024-09-20T22:41:40.171256+0000","last_unstale":"2024-09-20T22:42:13.093880+0000","last_undegraded":"2024-09-20T22:42:13.093880+0000","last_fullsized":"2024-09-20T22:42:13.093880+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:53.884281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":29,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.091807+0000","last_change":"2024-09-20T22:41:42.178413+0000","last_active":"2024-09-20T22:42:18.091807+0000","last_peered":"2024-09-20T22:42:18.091807+0000","last_clean":"2024-09-20T22:42:18.091807+0000","last_became_active":"2024-09-20T22:41:42.178157+0000","last_became_peered":"2024-09-20T22:41:42.178157+0000","last_unstale":"2024-09-20T22:42:18.091807+0000","last_undegraded":"2024-09-20T22:42:18.091807+0000","last_fullsized":"2024-09-20T22:42:18.091807+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:01.837489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":393,"ondisk_log_size":393,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":35,"num_read_kb":30,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070476,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32436,"kb_used_data":1248,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712716,"statfs":{"total":95995035648,"available":95961821184,"internally_reserved":0,"allocated":1277952,"data_stored":879981,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430226,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32424,"kb_used_data":1232,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712728,"statfs":{"total":95995035648,"available":95961833472,"internally_reserved":0,"allocated":1261568,"data_stored":879889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691863,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27684,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717468,"statfs":{"total":95995035648,"available":95966687232,"internally_reserved":0,"allocated":671744,"data_stored":290303,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986204,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27704,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717448,"statfs":{"total":95995035648,"available":95966666752,"internally_reserved":0,"allocated":692224,"data_stored":289133,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182434,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27724,"kb_used_data":696,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717428,"statfs":{"total":95995035648,"available":95966646272,"internally_reserved":0,"allocated":712704,"data_stored":296771,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411367,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31832,"kb_used_data":644,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713320,"statfs":{"total":95995035648,"available":95962439680,"internally_reserved":0,"allocated":659456,"data_stored":289099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574892,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27736,"kb_used_data":708,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717416,"statfs":{"total":95995035648,"available":95966633984,"internally_reserved":0,"allocated":724992,"data_stored":290300,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738417,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32440,"kb_used_data":1252,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712712,"statfs":{"total":95995035648,"available":95961817088,"internally_reserved":0,"allocated":1282048,"data_stored":880113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-20T22:42:20.339 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph pg dump --format=json 2024-09-20T22:42:20.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:20 smithi154 ceph-mon[33264]: purged_snaps scrub starts 2024-09-20T22:42:20.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:20 smithi154 ceph-mon[33264]: purged_snaps scrub ok 2024-09-20T22:42:20.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:20 smithi154 ceph-mon[33264]: Adjusting osd_memory_target on smithi084 to 3760M 2024-09-20T22:42:20.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:20 smithi154 ceph-mon[33264]: Updating smithi084:/etc/ceph/ceph.conf 2024-09-20T22:42:20.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:20 smithi154 ceph-mon[33264]: Updating smithi154:/etc/ceph/ceph.conf 2024-09-20T22:42:20.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:20 smithi154 ceph-mon[33264]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:42:20.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:20 smithi154 ceph-mon[33264]: Updating smithi084:/var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T22:42:20.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:20 smithi154 ceph-mon[33264]: Updating smithi154:/var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T22:42:20.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:20 smithi154 ceph-mon[33264]: osdmap e81: 8 total, 8 up, 8 in 2024-09-20T22:42:20.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:20 smithi154 ceph-mon[33264]: Updating smithi084:/etc/ceph/ceph.client.admin.keyring 2024-09-20T22:42:20.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:20 smithi154 ceph-mon[33264]: Updating smithi154:/etc/ceph/ceph.client.admin.keyring 2024-09-20T22:42:20.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:20 smithi154 ceph-mon[33264]: Updating smithi084:/var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-20T22:42:20.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:20 smithi154 ceph-mon[33264]: Updating smithi154:/var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-20T22:42:20.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:20 smithi154 ceph-mon[33264]: from='client.14733 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T22:42:20.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:20 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:42:20.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:20 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:42:20.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:20 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:42:20.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:20 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:42:20.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:20 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:42:20.593 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:42:20.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:20 smithi084 ceph-mon[29774]: purged_snaps scrub starts 2024-09-20T22:42:20.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:20 smithi084 ceph-mon[29774]: purged_snaps scrub ok 2024-09-20T22:42:20.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:20 smithi084 ceph-mon[29774]: Adjusting osd_memory_target on smithi084 to 3760M 2024-09-20T22:42:20.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:20 smithi084 ceph-mon[29774]: Updating smithi084:/etc/ceph/ceph.conf 2024-09-20T22:42:20.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:20 smithi084 ceph-mon[29774]: Updating smithi154:/etc/ceph/ceph.conf 2024-09-20T22:42:20.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:20 smithi084 ceph-mon[29774]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:42:20.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:20 smithi084 ceph-mon[29774]: Updating smithi084:/var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T22:42:20.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:20 smithi084 ceph-mon[29774]: Updating smithi154:/var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T22:42:20.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:20 smithi084 ceph-mon[29774]: osdmap e81: 8 total, 8 up, 8 in 2024-09-20T22:42:20.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:20 smithi084 ceph-mon[29774]: Updating smithi084:/etc/ceph/ceph.client.admin.keyring 2024-09-20T22:42:20.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:20 smithi084 ceph-mon[29774]: Updating smithi154:/etc/ceph/ceph.client.admin.keyring 2024-09-20T22:42:20.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:20 smithi084 ceph-mon[29774]: Updating smithi084:/var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-20T22:42:20.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:20 smithi084 ceph-mon[29774]: Updating smithi154:/var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-20T22:42:20.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:20 smithi084 ceph-mon[29774]: from='client.14733 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T22:42:20.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:20 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:42:20.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:20 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:42:20.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:20 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:42:20.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:20 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:42:20.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:20 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:42:20.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:20 smithi084 ceph-mon[40024]: purged_snaps scrub starts 2024-09-20T22:42:20.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:20 smithi084 ceph-mon[40024]: purged_snaps scrub ok 2024-09-20T22:42:20.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:20 smithi084 ceph-mon[40024]: Adjusting osd_memory_target on smithi084 to 3760M 2024-09-20T22:42:20.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:20 smithi084 ceph-mon[40024]: Updating smithi084:/etc/ceph/ceph.conf 2024-09-20T22:42:20.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:20 smithi084 ceph-mon[40024]: Updating smithi154:/etc/ceph/ceph.conf 2024-09-20T22:42:20.617 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:20 smithi084 ceph-mon[40024]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:42:20.617 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:20 smithi084 ceph-mon[40024]: Updating smithi084:/var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T22:42:20.617 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:20 smithi084 ceph-mon[40024]: Updating smithi154:/var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T22:42:20.617 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:20 smithi084 ceph-mon[40024]: osdmap e81: 8 total, 8 up, 8 in 2024-09-20T22:42:20.617 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:20 smithi084 ceph-mon[40024]: Updating smithi084:/etc/ceph/ceph.client.admin.keyring 2024-09-20T22:42:20.617 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:20 smithi084 ceph-mon[40024]: Updating smithi154:/etc/ceph/ceph.client.admin.keyring 2024-09-20T22:42:20.617 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:20 smithi084 ceph-mon[40024]: Updating smithi084:/var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-20T22:42:20.617 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:20 smithi084 ceph-mon[40024]: Updating smithi154:/var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-20T22:42:20.617 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:20 smithi084 ceph-mon[40024]: from='client.14733 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T22:42:20.617 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:20 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:42:20.617 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:20 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:42:20.617 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:20 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:42:20.617 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:20 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:42:20.618 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:20 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:42:21.240 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-20T22:42:21.240 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:21 smithi084 ceph-mon[40024]: pgmap v13: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 19 op/s 2024-09-20T22:42:21.240 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:21 smithi084 ceph-mon[40024]: Deploying daemon alertmanager.a on smithi084 2024-09-20T22:42:21.241 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:21 smithi084 ceph-mon[29774]: pgmap v13: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 19 op/s 2024-09-20T22:42:21.241 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:21 smithi084 ceph-mon[29774]: Deploying daemon alertmanager.a on smithi084 2024-09-20T22:42:21.248 INFO:teuthology.orchestra.run.smithi084.stderr:dumped all 2024-09-20T22:42:21.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:21 smithi154 ceph-mon[33264]: pgmap v13: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 19 op/s 2024-09-20T22:42:21.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:21 smithi154 ceph-mon[33264]: Deploying daemon alertmanager.a on smithi084 2024-09-20T22:42:22.262 INFO:teuthology.orchestra.run.smithi084.stdout:{"pg_ready":true,"pg_map":{"version":13,"stamp":"2024-09-20T22:42:19.612732+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596507,"num_objects":199,"num_object_clones":0,"num_object_copies":597,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":199,"num_whiteouts":0,"num_read":969,"num_read_kb":815,"num_write":615,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":601,"ondisk_log_size":601,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":239980,"kb_used_data":7112,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":749721236,"statfs":{"total":767960285184,"available":767714545664,"internally_reserved":0,"allocated":7282688,"data_stored":4095589,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":106,"num_read_kb":216,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.523938"},"pg_stats":[{"pgid":"3.1f","version":"67'1","reported_seq":43,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.091842+0000","last_change":"2024-09-20T22:41:38.159365+0000","last_active":"2024-09-20T22:42:18.091842+0000","last_peered":"2024-09-20T22:42:18.091842+0000","last_clean":"2024-09-20T22:42:18.091842+0000","last_became_active":"2024-09-20T22:41:38.159085+0000","last_became_peered":"2024-09-20T22:41:38.159085+0000","last_unstale":"2024-09-20T22:42:18.091842+0000","last_undegraded":"2024-09-20T22:42:18.091842+0000","last_fullsized":"2024-09-20T22:42:18.091842+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:37.732756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"74'9","reported_seq":49,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.092121+0000","last_change":"2024-09-20T22:41:40.169673+0000","last_active":"2024-09-20T22:42:18.092121+0000","last_peered":"2024-09-20T22:42:18.092121+0000","last_clean":"2024-09-20T22:42:18.092121+0000","last_became_active":"2024-09-20T22:41:40.168934+0000","last_became_peered":"2024-09-20T22:41:40.168934+0000","last_unstale":"2024-09-20T22:42:18.092121+0000","last_undegraded":"2024-09-20T22:42:18.092121+0000","last_fullsized":"2024-09-20T22:42:18.092121+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:54.489342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":29,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.091084+0000","last_change":"2024-09-20T22:41:42.182304+0000","last_active":"2024-09-20T22:42:18.091084+0000","last_peered":"2024-09-20T22:42:18.091084+0000","last_clean":"2024-09-20T22:42:18.091084+0000","last_became_active":"2024-09-20T22:41:42.182024+0000","last_became_peered":"2024-09-20T22:41:42.182024+0000","last_unstale":"2024-09-20T22:42:18.091084+0000","last_undegraded":"2024-09-20T22:42:18.091084+0000","last_fullsized":"2024-09-20T22:42:18.091084+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:57.397887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.092216+0000","last_change":"2024-09-20T22:41:44.198207+0000","last_active":"2024-09-20T22:42:18.092216+0000","last_peered":"2024-09-20T22:42:18.092216+0000","last_clean":"2024-09-20T22:42:18.092216+0000","last_became_active":"2024-09-20T22:41:44.197897+0000","last_became_peered":"2024-09-20T22:41:44.197897+0000","last_unstale":"2024-09-20T22:42:18.092216+0000","last_undegraded":"2024-09-20T22:42:18.092216+0000","last_fullsized":"2024-09-20T22:42:18.092216+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:53.466451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.254290+0000","last_change":"2024-09-20T22:41:44.832449+0000","last_active":"2024-09-20T22:42:18.254290+0000","last_peered":"2024-09-20T22:42:18.254290+0000","last_clean":"2024-09-20T22:42:18.254290+0000","last_became_active":"2024-09-20T22:41:44.832099+0000","last_became_peered":"2024-09-20T22:41:44.832099+0000","last_unstale":"2024-09-20T22:42:18.254290+0000","last_undegraded":"2024-09-20T22:42:18.254290+0000","last_fullsized":"2024-09-20T22:42:18.254290+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:45.326749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":37,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.254797+0000","last_change":"2024-09-20T22:41:38.157685+0000","last_active":"2024-09-20T22:42:18.254797+0000","last_peered":"2024-09-20T22:42:18.254797+0000","last_clean":"2024-09-20T22:42:18.254797+0000","last_became_active":"2024-09-20T22:41:38.157298+0000","last_became_peered":"2024-09-20T22:41:38.157298+0000","last_unstale":"2024-09-20T22:42:18.254797+0000","last_undegraded":"2024-09-20T22:42:18.254797+0000","last_fullsized":"2024-09-20T22:42:18.254797+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:00.813798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"74'15","reported_seq":58,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.254425+0000","last_change":"2024-09-20T22:41:40.163585+0000","last_active":"2024-09-20T22:42:18.254425+0000","last_peered":"2024-09-20T22:42:18.254425+0000","last_clean":"2024-09-20T22:42:18.254425+0000","last_became_active":"2024-09-20T22:41:40.163242+0000","last_became_peered":"2024-09-20T22:41:40.163242+0000","last_unstale":"2024-09-20T22:42:18.254425+0000","last_undegraded":"2024-09-20T22:42:18.254425+0000","last_fullsized":"2024-09-20T22:42:18.254425+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:58.410323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":29,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.092118+0000","last_change":"2024-09-20T22:41:42.187359+0000","last_active":"2024-09-20T22:42:18.092118+0000","last_peered":"2024-09-20T22:42:18.092118+0000","last_clean":"2024-09-20T22:42:18.092118+0000","last_became_active":"2024-09-20T22:41:42.187076+0000","last_became_peered":"2024-09-20T22:41:42.187076+0000","last_unstale":"2024-09-20T22:42:18.092118+0000","last_undegraded":"2024-09-20T22:42:18.092118+0000","last_fullsized":"2024-09-20T22:42:18.092118+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:42.282475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":37,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.547950+0000","last_change":"2024-09-20T22:41:38.156694+0000","last_active":"2024-09-20T22:42:18.547950+0000","last_peered":"2024-09-20T22:42:18.547950+0000","last_clean":"2024-09-20T22:42:18.547950+0000","last_became_active":"2024-09-20T22:41:38.154827+0000","last_became_peered":"2024-09-20T22:41:38.154827+0000","last_unstale":"2024-09-20T22:42:18.547950+0000","last_undegraded":"2024-09-20T22:42:18.547950+0000","last_fullsized":"2024-09-20T22:42:18.547950+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:48.618357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"74'9","reported_seq":49,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.092795+0000","last_change":"2024-09-20T22:41:40.163006+0000","last_active":"2024-09-20T22:42:18.092795+0000","last_peered":"2024-09-20T22:42:18.092795+0000","last_clean":"2024-09-20T22:42:18.092795+0000","last_became_active":"2024-09-20T22:41:40.162546+0000","last_became_peered":"2024-09-20T22:41:40.162546+0000","last_unstale":"2024-09-20T22:42:18.092795+0000","last_undegraded":"2024-09-20T22:42:18.092795+0000","last_fullsized":"2024-09-20T22:42:18.092795+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:46.501647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":29,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.548539+0000","last_change":"2024-09-20T22:41:42.185835+0000","last_active":"2024-09-20T22:42:18.548539+0000","last_peered":"2024-09-20T22:42:18.548539+0000","last_clean":"2024-09-20T22:42:18.548539+0000","last_became_active":"2024-09-20T22:41:42.185538+0000","last_became_peered":"2024-09-20T22:41:42.185538+0000","last_unstale":"2024-09-20T22:42:18.548539+0000","last_undegraded":"2024-09-20T22:42:18.548539+0000","last_fullsized":"2024-09-20T22:42:18.548539+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:57.305761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.090692+0000","last_change":"2024-09-20T22:41:44.198169+0000","last_active":"2024-09-20T22:42:18.090692+0000","last_peered":"2024-09-20T22:42:18.090692+0000","last_clean":"2024-09-20T22:42:18.090692+0000","last_became_active":"2024-09-20T22:41:44.197113+0000","last_became_peered":"2024-09-20T22:41:44.197113+0000","last_unstale":"2024-09-20T22:42:18.090692+0000","last_undegraded":"2024-09-20T22:42:18.090692+0000","last_fullsized":"2024-09-20T22:42:18.090692+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:02.374640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":37,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.547916+0000","last_change":"2024-09-20T22:41:38.156116+0000","last_active":"2024-09-20T22:42:18.547916+0000","last_peered":"2024-09-20T22:42:18.547916+0000","last_clean":"2024-09-20T22:42:18.547916+0000","last_became_active":"2024-09-20T22:41:38.154650+0000","last_became_peered":"2024-09-20T22:41:38.154650+0000","last_unstale":"2024-09-20T22:42:18.547916+0000","last_undegraded":"2024-09-20T22:42:18.547916+0000","last_fullsized":"2024-09-20T22:42:18.547916+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:44.595033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"74'5","reported_seq":43,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.092397+0000","last_change":"2024-09-20T22:41:40.171883+0000","last_active":"2024-09-20T22:42:18.092397+0000","last_peered":"2024-09-20T22:42:18.092397+0000","last_clean":"2024-09-20T22:42:18.092397+0000","last_became_active":"2024-09-20T22:41:40.171081+0000","last_became_peered":"2024-09-20T22:41:40.171081+0000","last_unstale":"2024-09-20T22:42:18.092397+0000","last_undegraded":"2024-09-20T22:42:18.092397+0000","last_fullsized":"2024-09-20T22:42:18.092397+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:55.048729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":29,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.465214+0000","last_change":"2024-09-20T22:41:42.183965+0000","last_active":"2024-09-20T22:42:18.465214+0000","last_peered":"2024-09-20T22:42:18.465214+0000","last_clean":"2024-09-20T22:42:18.465214+0000","last_became_active":"2024-09-20T22:41:42.182916+0000","last_became_peered":"2024-09-20T22:41:42.182916+0000","last_unstale":"2024-09-20T22:42:18.465214+0000","last_undegraded":"2024-09-20T22:42:18.465214+0000","last_fullsized":"2024-09-20T22:42:18.465214+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:42.258865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.548312+0000","last_change":"2024-09-20T22:41:44.198221+0000","last_active":"2024-09-20T22:42:18.548312+0000","last_peered":"2024-09-20T22:42:18.548312+0000","last_clean":"2024-09-20T22:42:18.548312+0000","last_became_active":"2024-09-20T22:41:44.197927+0000","last_became_peered":"2024-09-20T22:41:44.197927+0000","last_unstale":"2024-09-20T22:42:18.548312+0000","last_undegraded":"2024-09-20T22:42:18.548312+0000","last_fullsized":"2024-09-20T22:42:18.548312+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:47.985353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.092499+0000","last_change":"2024-09-20T22:41:44.832646+0000","last_active":"2024-09-20T22:42:18.092499+0000","last_peered":"2024-09-20T22:42:18.092499+0000","last_clean":"2024-09-20T22:42:18.092499+0000","last_became_active":"2024-09-20T22:41:44.832360+0000","last_became_peered":"2024-09-20T22:41:44.832360+0000","last_unstale":"2024-09-20T22:42:18.092499+0000","last_undegraded":"2024-09-20T22:42:18.092499+0000","last_fullsized":"2024-09-20T22:42:18.092499+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:45.707782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":37,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.091790+0000","last_change":"2024-09-20T22:41:38.152726+0000","last_active":"2024-09-20T22:42:18.091790+0000","last_peered":"2024-09-20T22:42:18.091790+0000","last_clean":"2024-09-20T22:42:18.091790+0000","last_became_active":"2024-09-20T22:41:38.152443+0000","last_became_peered":"2024-09-20T22:41:38.152443+0000","last_unstale":"2024-09-20T22:42:18.091790+0000","last_undegraded":"2024-09-20T22:42:18.091790+0000","last_fullsized":"2024-09-20T22:42:18.091790+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:03.325746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"74'15","reported_seq":58,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.318481+0000","last_change":"2024-09-20T22:41:40.170414+0000","last_active":"2024-09-20T22:42:18.318481+0000","last_peered":"2024-09-20T22:42:18.318481+0000","last_clean":"2024-09-20T22:42:18.318481+0000","last_became_active":"2024-09-20T22:41:40.170131+0000","last_became_peered":"2024-09-20T22:41:40.170131+0000","last_unstale":"2024-09-20T22:42:18.318481+0000","last_undegraded":"2024-09-20T22:42:18.318481+0000","last_fullsized":"2024-09-20T22:42:18.318481+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:57.393598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":29,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.092312+0000","last_change":"2024-09-20T22:41:42.185171+0000","last_active":"2024-09-20T22:42:18.092312+0000","last_peered":"2024-09-20T22:42:18.092312+0000","last_clean":"2024-09-20T22:42:18.092312+0000","last_became_active":"2024-09-20T22:41:42.184759+0000","last_became_peered":"2024-09-20T22:41:42.184759+0000","last_unstale":"2024-09-20T22:42:18.092312+0000","last_undegraded":"2024-09-20T22:42:18.092312+0000","last_fullsized":"2024-09-20T22:42:18.092312+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:01.601642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.254753+0000","last_change":"2024-09-20T22:41:44.832468+0000","last_active":"2024-09-20T22:42:18.254753+0000","last_peered":"2024-09-20T22:42:18.254753+0000","last_clean":"2024-09-20T22:42:18.254753+0000","last_became_active":"2024-09-20T22:41:44.832131+0000","last_became_peered":"2024-09-20T22:41:44.832131+0000","last_unstale":"2024-09-20T22:42:18.254753+0000","last_undegraded":"2024-09-20T22:42:18.254753+0000","last_fullsized":"2024-09-20T22:42:18.254753+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:08.595231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":37,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.093150+0000","last_change":"2024-09-20T22:41:38.150584+0000","last_active":"2024-09-20T22:42:18.093150+0000","last_peered":"2024-09-20T22:42:18.093150+0000","last_clean":"2024-09-20T22:42:18.093150+0000","last_became_active":"2024-09-20T22:41:38.150232+0000","last_became_peered":"2024-09-20T22:41:38.150232+0000","last_unstale":"2024-09-20T22:42:18.093150+0000","last_undegraded":"2024-09-20T22:42:18.093150+0000","last_fullsized":"2024-09-20T22:42:18.093150+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:02.237312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"74'12","reported_seq":56,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.254603+0000","last_change":"2024-09-20T22:41:40.166344+0000","last_active":"2024-09-20T22:42:18.254603+0000","last_peered":"2024-09-20T22:42:18.254603+0000","last_clean":"2024-09-20T22:42:18.254603+0000","last_became_active":"2024-09-20T22:41:40.166087+0000","last_became_peered":"2024-09-20T22:41:40.166087+0000","last_unstale":"2024-09-20T22:42:18.254603+0000","last_undegraded":"2024-09-20T22:42:18.254603+0000","last_fullsized":"2024-09-20T22:42:18.254603+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:01.373771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":29,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.092462+0000","last_change":"2024-09-20T22:41:42.183266+0000","last_active":"2024-09-20T22:42:18.092462+0000","last_peered":"2024-09-20T22:42:18.092462+0000","last_clean":"2024-09-20T22:42:18.092462+0000","last_became_active":"2024-09-20T22:41:42.183005+0000","last_became_peered":"2024-09-20T22:41:42.183005+0000","last_unstale":"2024-09-20T22:42:18.092462+0000","last_undegraded":"2024-09-20T22:42:18.092462+0000","last_fullsized":"2024-09-20T22:42:18.092462+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:01.169906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.465546+0000","last_change":"2024-09-20T22:41:44.200115+0000","last_active":"2024-09-20T22:42:18.465546+0000","last_peered":"2024-09-20T22:42:18.465546+0000","last_clean":"2024-09-20T22:42:18.465546+0000","last_became_active":"2024-09-20T22:41:44.199822+0000","last_became_peered":"2024-09-20T22:41:44.199822+0000","last_unstale":"2024-09-20T22:42:18.465546+0000","last_undegraded":"2024-09-20T22:42:18.465546+0000","last_fullsized":"2024-09-20T22:42:18.465546+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:01.277411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"67'1","reported_seq":38,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.091235+0000","last_change":"2024-09-20T22:41:38.155735+0000","last_active":"2024-09-20T22:42:18.091235+0000","last_peered":"2024-09-20T22:42:18.091235+0000","last_clean":"2024-09-20T22:42:18.091235+0000","last_became_active":"2024-09-20T22:41:38.155424+0000","last_became_peered":"2024-09-20T22:41:38.155424+0000","last_unstale":"2024-09-20T22:42:18.091235+0000","last_undegraded":"2024-09-20T22:42:18.091235+0000","last_fullsized":"2024-09-20T22:42:18.091235+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:52.584133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"74'10","reported_seq":48,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.091915+0000","last_change":"2024-09-20T22:41:40.252498+0000","last_active":"2024-09-20T22:42:18.091915+0000","last_peered":"2024-09-20T22:42:18.091915+0000","last_clean":"2024-09-20T22:42:18.091915+0000","last_became_active":"2024-09-20T22:41:40.252001+0000","last_became_peered":"2024-09-20T22:41:40.252001+0000","last_unstale":"2024-09-20T22:42:18.091915+0000","last_undegraded":"2024-09-20T22:42:18.091915+0000","last_fullsized":"2024-09-20T22:42:18.091915+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:01.417640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"74'8","reported_seq":50,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-20T22:42:13.209735+0000","last_change":"2024-09-20T22:41:42.186385+0000","last_active":"2024-09-20T22:42:13.209735+0000","last_peered":"2024-09-20T22:42:13.209735+0000","last_clean":"2024-09-20T22:42:13.209735+0000","last_became_active":"2024-09-20T22:41:42.186071+0000","last_became_peered":"2024-09-20T22:41:42.186071+0000","last_unstale":"2024-09-20T22:42:13.209735+0000","last_undegraded":"2024-09-20T22:42:13.209735+0000","last_fullsized":"2024-09-20T22:42:13.209735+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:56.929679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.f","version":"74'15","reported_seq":58,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.092219+0000","last_change":"2024-09-20T22:41:40.168188+0000","last_active":"2024-09-20T22:42:18.092219+0000","last_peered":"2024-09-20T22:42:18.092219+0000","last_clean":"2024-09-20T22:42:18.092219+0000","last_became_active":"2024-09-20T22:41:40.165963+0000","last_became_peered":"2024-09-20T22:41:40.165963+0000","last_unstale":"2024-09-20T22:42:18.092219+0000","last_undegraded":"2024-09-20T22:42:18.092219+0000","last_fullsized":"2024-09-20T22:42:18.092219+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:46.519265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":37,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.253822+0000","last_change":"2024-09-20T22:41:38.157043+0000","last_active":"2024-09-20T22:42:18.253822+0000","last_peered":"2024-09-20T22:42:18.253822+0000","last_clean":"2024-09-20T22:42:18.253822+0000","last_became_active":"2024-09-20T22:41:38.156749+0000","last_became_peered":"2024-09-20T22:41:38.156749+0000","last_unstale":"2024-09-20T22:42:18.253822+0000","last_undegraded":"2024-09-20T22:42:18.253822+0000","last_fullsized":"2024-09-20T22:42:18.253822+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:03.406143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"74'8","reported_seq":46,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.092780+0000","last_change":"2024-09-20T22:41:42.182555+0000","last_active":"2024-09-20T22:42:18.092780+0000","last_peered":"2024-09-20T22:42:18.092780+0000","last_clean":"2024-09-20T22:42:18.092780+0000","last_became_active":"2024-09-20T22:41:42.182224+0000","last_became_peered":"2024-09-20T22:41:42.182224+0000","last_unstale":"2024-09-20T22:42:18.092780+0000","last_undegraded":"2024-09-20T22:42:18.092780+0000","last_fullsized":"2024-09-20T22:42:18.092780+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:07.384166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.548714+0000","last_change":"2024-09-20T22:41:44.198655+0000","last_active":"2024-09-20T22:42:18.548714+0000","last_peered":"2024-09-20T22:42:18.548714+0000","last_clean":"2024-09-20T22:42:18.548714+0000","last_became_active":"2024-09-20T22:41:44.198347+0000","last_became_peered":"2024-09-20T22:41:44.198347+0000","last_unstale":"2024-09-20T22:42:18.548714+0000","last_undegraded":"2024-09-20T22:42:18.548714+0000","last_fullsized":"2024-09-20T22:42:18.548714+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:54.119646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.0","version":"74'18","reported_seq":65,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.254174+0000","last_change":"2024-09-20T22:41:40.252203+0000","last_active":"2024-09-20T22:42:18.254174+0000","last_peered":"2024-09-20T22:42:18.254174+0000","last_clean":"2024-09-20T22:42:18.254174+0000","last_became_active":"2024-09-20T22:41:40.251868+0000","last_became_peered":"2024-09-20T22:41:40.251868+0000","last_unstale":"2024-09-20T22:42:18.254174+0000","last_undegraded":"2024-09-20T22:42:18.254174+0000","last_fullsized":"2024-09-20T22:42:18.254174+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:57.966357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":37,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.253939+0000","last_change":"2024-09-20T22:41:38.161991+0000","last_active":"2024-09-20T22:42:18.253939+0000","last_peered":"2024-09-20T22:42:18.253939+0000","last_clean":"2024-09-20T22:42:18.253939+0000","last_became_active":"2024-09-20T22:41:38.161335+0000","last_became_peered":"2024-09-20T22:41:38.161335+0000","last_unstale":"2024-09-20T22:42:18.253939+0000","last_undegraded":"2024-09-20T22:42:18.253939+0000","last_fullsized":"2024-09-20T22:42:18.253939+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:48.134689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":29,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.092219+0000","last_change":"2024-09-20T22:41:42.190157+0000","last_active":"2024-09-20T22:42:18.092219+0000","last_peered":"2024-09-20T22:42:18.092219+0000","last_clean":"2024-09-20T22:42:18.092219+0000","last_became_active":"2024-09-20T22:41:42.189743+0000","last_became_peered":"2024-09-20T22:41:42.189743+0000","last_unstale":"2024-09-20T22:42:18.092219+0000","last_undegraded":"2024-09-20T22:42:18.092219+0000","last_fullsized":"2024-09-20T22:42:18.092219+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:42.602485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.092640+0000","last_change":"2024-09-20T22:41:44.193944+0000","last_active":"2024-09-20T22:42:18.092640+0000","last_peered":"2024-09-20T22:42:18.092640+0000","last_clean":"2024-09-20T22:42:18.092640+0000","last_became_active":"2024-09-20T22:41:44.193646+0000","last_became_peered":"2024-09-20T22:41:44.193646+0000","last_unstale":"2024-09-20T22:42:18.092640+0000","last_undegraded":"2024-09-20T22:42:18.092640+0000","last_fullsized":"2024-09-20T22:42:18.092640+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:50.182915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"74'14","reported_seq":54,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.092192+0000","last_change":"2024-09-20T22:41:40.164011+0000","last_active":"2024-09-20T22:42:18.092192+0000","last_peered":"2024-09-20T22:42:18.092192+0000","last_clean":"2024-09-20T22:42:18.092192+0000","last_became_active":"2024-09-20T22:41:40.163695+0000","last_became_peered":"2024-09-20T22:41:40.163695+0000","last_unstale":"2024-09-20T22:42:18.092192+0000","last_undegraded":"2024-09-20T22:42:18.092192+0000","last_fullsized":"2024-09-20T22:42:18.092192+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:04.888108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"67'1","reported_seq":38,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.092025+0000","last_change":"2024-09-20T22:41:38.150707+0000","last_active":"2024-09-20T22:42:18.092025+0000","last_peered":"2024-09-20T22:42:18.092025+0000","last_clean":"2024-09-20T22:42:18.092025+0000","last_became_active":"2024-09-20T22:41:38.150331+0000","last_became_peered":"2024-09-20T22:41:38.150331+0000","last_unstale":"2024-09-20T22:42:18.092025+0000","last_undegraded":"2024-09-20T22:42:18.092025+0000","last_fullsized":"2024-09-20T22:42:18.092025+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:53.797157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"74'8","reported_seq":46,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.254075+0000","last_change":"2024-09-20T22:41:42.179893+0000","last_active":"2024-09-20T22:42:18.254075+0000","last_peered":"2024-09-20T22:42:18.254075+0000","last_clean":"2024-09-20T22:42:18.254075+0000","last_became_active":"2024-09-20T22:41:42.179613+0000","last_became_peered":"2024-09-20T22:41:42.179613+0000","last_unstale":"2024-09-20T22:42:18.254075+0000","last_undegraded":"2024-09-20T22:42:18.254075+0000","last_fullsized":"2024-09-20T22:42:18.254075+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:03.642295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.465380+0000","last_change":"2024-09-20T22:41:44.832161+0000","last_active":"2024-09-20T22:42:18.465380+0000","last_peered":"2024-09-20T22:42:18.465380+0000","last_clean":"2024-09-20T22:42:18.465380+0000","last_became_active":"2024-09-20T22:41:44.831901+0000","last_became_peered":"2024-09-20T22:41:44.831901+0000","last_unstale":"2024-09-20T22:42:18.465380+0000","last_undegraded":"2024-09-20T22:42:18.465380+0000","last_fullsized":"2024-09-20T22:42:18.465380+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:10.151190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2","version":"74'10","reported_seq":48,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.091491+0000","last_change":"2024-09-20T22:41:40.167041+0000","last_active":"2024-09-20T22:42:18.091491+0000","last_peered":"2024-09-20T22:42:18.091491+0000","last_clean":"2024-09-20T22:42:18.091491+0000","last_became_active":"2024-09-20T22:41:40.165653+0000","last_became_peered":"2024-09-20T22:41:40.165653+0000","last_unstale":"2024-09-20T22:42:18.091491+0000","last_undegraded":"2024-09-20T22:42:18.091491+0000","last_fullsized":"2024-09-20T22:42:18.091491+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:47.255874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":37,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.549041+0000","last_change":"2024-09-20T22:41:38.155924+0000","last_active":"2024-09-20T22:42:18.549041+0000","last_peered":"2024-09-20T22:42:18.549041+0000","last_clean":"2024-09-20T22:42:18.549041+0000","last_became_active":"2024-09-20T22:41:38.155629+0000","last_became_peered":"2024-09-20T22:41:38.155629+0000","last_unstale":"2024-09-20T22:42:18.549041+0000","last_undegraded":"2024-09-20T22:42:18.549041+0000","last_fullsized":"2024-09-20T22:42:18.549041+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:39.438618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"74'8","reported_seq":46,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.091439+0000","last_change":"2024-09-20T22:41:42.185583+0000","last_active":"2024-09-20T22:42:18.091439+0000","last_peered":"2024-09-20T22:42:18.091439+0000","last_clean":"2024-09-20T22:42:18.091439+0000","last_became_active":"2024-09-20T22:41:42.185213+0000","last_became_peered":"2024-09-20T22:41:42.185213+0000","last_unstale":"2024-09-20T22:42:18.091439+0000","last_undegraded":"2024-09-20T22:42:18.091439+0000","last_fullsized":"2024-09-20T22:42:18.091439+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:48.459335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.091670+0000","last_change":"2024-09-20T22:41:44.197894+0000","last_active":"2024-09-20T22:42:18.091670+0000","last_peered":"2024-09-20T22:42:18.091670+0000","last_clean":"2024-09-20T22:42:18.091670+0000","last_became_active":"2024-09-20T22:41:44.197619+0000","last_became_peered":"2024-09-20T22:41:44.197619+0000","last_unstale":"2024-09-20T22:42:18.091670+0000","last_undegraded":"2024-09-20T22:42:18.091670+0000","last_fullsized":"2024-09-20T22:42:18.091670+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:00.802751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"74'19","reported_seq":69,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.091566+0000","last_change":"2024-09-20T22:41:40.252159+0000","last_active":"2024-09-20T22:42:18.091566+0000","last_peered":"2024-09-20T22:42:18.091566+0000","last_clean":"2024-09-20T22:42:18.091566+0000","last_became_active":"2024-09-20T22:41:40.251866+0000","last_became_peered":"2024-09-20T22:41:40.251866+0000","last_unstale":"2024-09-20T22:42:18.091566+0000","last_undegraded":"2024-09-20T22:42:18.091566+0000","last_fullsized":"2024-09-20T22:42:18.091566+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:43.089389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":37,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.092585+0000","last_change":"2024-09-20T22:41:38.161236+0000","last_active":"2024-09-20T22:42:18.092585+0000","last_peered":"2024-09-20T22:42:18.092585+0000","last_clean":"2024-09-20T22:42:18.092585+0000","last_became_active":"2024-09-20T22:41:38.160832+0000","last_became_peered":"2024-09-20T22:41:38.160832+0000","last_unstale":"2024-09-20T22:42:18.092585+0000","last_undegraded":"2024-09-20T22:42:18.092585+0000","last_fullsized":"2024-09-20T22:42:18.092585+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:50.975305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":27,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-20T22:42:13.093143+0000","last_change":"2024-09-20T22:41:42.185638+0000","last_active":"2024-09-20T22:42:13.093143+0000","last_peered":"2024-09-20T22:42:13.093143+0000","last_clean":"2024-09-20T22:42:13.093143+0000","last_became_active":"2024-09-20T22:41:42.184412+0000","last_became_peered":"2024-09-20T22:41:42.184412+0000","last_unstale":"2024-09-20T22:42:13.093143+0000","last_undegraded":"2024-09-20T22:42:13.093143+0000","last_fullsized":"2024-09-20T22:42:13.093143+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:57.435580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.090990+0000","last_change":"2024-09-20T22:41:44.830917+0000","last_active":"2024-09-20T22:42:18.090990+0000","last_peered":"2024-09-20T22:42:18.090990+0000","last_clean":"2024-09-20T22:42:18.090990+0000","last_became_active":"2024-09-20T22:41:44.830538+0000","last_became_peered":"2024-09-20T22:41:44.830538+0000","last_unstale":"2024-09-20T22:42:18.090990+0000","last_undegraded":"2024-09-20T22:42:18.090990+0000","last_fullsized":"2024-09-20T22:42:18.090990+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:54.961702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"74'28","reported_seq":90,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.092554+0000","last_change":"2024-09-20T22:41:40.167313+0000","last_active":"2024-09-20T22:42:18.092554+0000","last_peered":"2024-09-20T22:42:18.092554+0000","last_clean":"2024-09-20T22:42:18.092554+0000","last_became_active":"2024-09-20T22:41:40.166958+0000","last_became_peered":"2024-09-20T22:41:40.166958+0000","last_unstale":"2024-09-20T22:42:18.092554+0000","last_undegraded":"2024-09-20T22:42:18.092554+0000","last_fullsized":"2024-09-20T22:42:18.092554+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:42.825818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":37,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.092991+0000","last_change":"2024-09-20T22:41:38.152921+0000","last_active":"2024-09-20T22:42:18.092991+0000","last_peered":"2024-09-20T22:42:18.092991+0000","last_clean":"2024-09-20T22:42:18.092991+0000","last_became_active":"2024-09-20T22:41:38.152525+0000","last_became_peered":"2024-09-20T22:41:38.152525+0000","last_unstale":"2024-09-20T22:42:18.092991+0000","last_undegraded":"2024-09-20T22:42:18.092991+0000","last_fullsized":"2024-09-20T22:42:18.092991+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:37.641089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"69'2","reported_seq":47,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.548637+0000","last_change":"2024-09-20T22:41:40.154106+0000","last_active":"2024-09-20T22:42:18.548637+0000","last_peered":"2024-09-20T22:42:18.548637+0000","last_clean":"2024-09-20T22:42:18.548637+0000","last_became_active":"2024-09-20T22:41:37.148479+0000","last_became_peered":"2024-09-20T22:41:37.148479+0000","last_unstale":"2024-09-20T22:42:18.548637+0000","last_undegraded":"2024-09-20T22:42:18.548637+0000","last_fullsized":"2024-09-20T22:42:18.548637+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:36.130684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:36.130684+0000","last_clean_scrub_stamp":"2024-09-20T22:41:36.130684+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:48.204536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087280599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":29,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.091602+0000","last_change":"2024-09-20T22:41:42.179979+0000","last_active":"2024-09-20T22:42:18.091602+0000","last_peered":"2024-09-20T22:42:18.091602+0000","last_clean":"2024-09-20T22:42:18.091602+0000","last_became_active":"2024-09-20T22:41:42.179742+0000","last_became_peered":"2024-09-20T22:41:42.179742+0000","last_unstale":"2024-09-20T22:42:18.091602+0000","last_undegraded":"2024-09-20T22:42:18.091602+0000","last_fullsized":"2024-09-20T22:42:18.091602+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:03.273567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.254628+0000","last_change":"2024-09-20T22:41:44.192602+0000","last_active":"2024-09-20T22:42:18.254628+0000","last_peered":"2024-09-20T22:42:18.254628+0000","last_clean":"2024-09-20T22:42:18.254628+0000","last_became_active":"2024-09-20T22:41:44.192128+0000","last_became_peered":"2024-09-20T22:41:44.192128+0000","last_unstale":"2024-09-20T22:42:18.254628+0000","last_undegraded":"2024-09-20T22:42:18.254628+0000","last_fullsized":"2024-09-20T22:42:18.254628+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:46.416059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"74'13","reported_seq":60,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.092319+0000","last_change":"2024-09-20T22:41:40.169777+0000","last_active":"2024-09-20T22:42:18.092319+0000","last_peered":"2024-09-20T22:42:18.092319+0000","last_clean":"2024-09-20T22:42:18.092319+0000","last_became_active":"2024-09-20T22:41:40.169473+0000","last_became_peered":"2024-09-20T22:41:40.169473+0000","last_unstale":"2024-09-20T22:42:18.092319+0000","last_undegraded":"2024-09-20T22:42:18.092319+0000","last_fullsized":"2024-09-20T22:42:18.092319+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:48.626103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":37,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.091334+0000","last_change":"2024-09-20T22:41:38.157113+0000","last_active":"2024-09-20T22:42:18.091334+0000","last_peered":"2024-09-20T22:42:18.091334+0000","last_clean":"2024-09-20T22:42:18.091334+0000","last_became_active":"2024-09-20T22:41:38.156837+0000","last_became_peered":"2024-09-20T22:41:38.156837+0000","last_unstale":"2024-09-20T22:42:18.091334+0000","last_undegraded":"2024-09-20T22:42:18.091334+0000","last_fullsized":"2024-09-20T22:42:18.091334+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:46.990363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"67'1","reported_seq":46,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.318042+0000","last_change":"2024-09-20T22:41:40.162396+0000","last_active":"2024-09-20T22:42:18.318042+0000","last_peered":"2024-09-20T22:42:18.318042+0000","last_clean":"2024-09-20T22:42:18.318042+0000","last_became_active":"2024-09-20T22:41:37.148165+0000","last_became_peered":"2024-09-20T22:41:37.148165+0000","last_unstale":"2024-09-20T22:42:18.318042+0000","last_undegraded":"2024-09-20T22:42:18.318042+0000","last_fullsized":"2024-09-20T22:42:18.318042+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:36.130684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:36.130684+0000","last_clean_scrub_stamp":"2024-09-20T22:41:36.130684+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:54.327893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001301459,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":29,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.318398+0000","last_change":"2024-09-20T22:41:42.185412+0000","last_active":"2024-09-20T22:42:18.318398+0000","last_peered":"2024-09-20T22:42:18.318398+0000","last_clean":"2024-09-20T22:42:18.318398+0000","last_became_active":"2024-09-20T22:41:42.185112+0000","last_became_peered":"2024-09-20T22:41:42.185112+0000","last_unstale":"2024-09-20T22:42:18.318398+0000","last_undegraded":"2024-09-20T22:42:18.318398+0000","last_fullsized":"2024-09-20T22:42:18.318398+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:05.899621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.465592+0000","last_change":"2024-09-20T22:41:44.830804+0000","last_active":"2024-09-20T22:42:18.465592+0000","last_peered":"2024-09-20T22:42:18.465592+0000","last_clean":"2024-09-20T22:42:18.465592+0000","last_became_active":"2024-09-20T22:41:44.830515+0000","last_became_peered":"2024-09-20T22:41:44.830515+0000","last_unstale":"2024-09-20T22:42:18.465592+0000","last_undegraded":"2024-09-20T22:42:18.465592+0000","last_fullsized":"2024-09-20T22:42:18.465592+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:04.958838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"74'12","reported_seq":51,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.091925+0000","last_change":"2024-09-20T22:41:40.167483+0000","last_active":"2024-09-20T22:42:18.091925+0000","last_peered":"2024-09-20T22:42:18.091925+0000","last_clean":"2024-09-20T22:42:18.091925+0000","last_became_active":"2024-09-20T22:41:40.167169+0000","last_became_peered":"2024-09-20T22:41:40.167169+0000","last_unstale":"2024-09-20T22:42:18.091925+0000","last_undegraded":"2024-09-20T22:42:18.091925+0000","last_fullsized":"2024-09-20T22:42:18.091925+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:48.803800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":37,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.091015+0000","last_change":"2024-09-20T22:41:38.153448+0000","last_active":"2024-09-20T22:42:18.091015+0000","last_peered":"2024-09-20T22:42:18.091015+0000","last_clean":"2024-09-20T22:42:18.091015+0000","last_became_active":"2024-09-20T22:41:38.153058+0000","last_became_peered":"2024-09-20T22:41:38.153058+0000","last_unstale":"2024-09-20T22:42:18.091015+0000","last_undegraded":"2024-09-20T22:42:18.091015+0000","last_fullsized":"2024-09-20T22:42:18.091015+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:04.425050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"75'5","reported_seq":83,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.465214+0000","last_change":"2024-09-20T22:41:40.251762+0000","last_active":"2024-09-20T22:42:18.465214+0000","last_peered":"2024-09-20T22:42:18.465214+0000","last_clean":"2024-09-20T22:42:18.465214+0000","last_became_active":"2024-09-20T22:41:37.148383+0000","last_became_peered":"2024-09-20T22:41:37.148383+0000","last_unstale":"2024-09-20T22:42:18.465214+0000","last_undegraded":"2024-09-20T22:42:18.465214+0000","last_fullsized":"2024-09-20T22:42:18.465214+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:36.130684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:36.130684+0000","last_clean_scrub_stamp":"2024-09-20T22:41:36.130684+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:04.208717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00078631700000000003,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":35,"num_read_kb":30,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":29,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.549126+0000","last_change":"2024-09-20T22:41:42.187618+0000","last_active":"2024-09-20T22:42:18.549126+0000","last_peered":"2024-09-20T22:42:18.549126+0000","last_clean":"2024-09-20T22:42:18.549126+0000","last_became_active":"2024-09-20T22:41:42.187346+0000","last_became_peered":"2024-09-20T22:41:42.187346+0000","last_unstale":"2024-09-20T22:42:18.549126+0000","last_undegraded":"2024-09-20T22:42:18.549126+0000","last_fullsized":"2024-09-20T22:42:18.549126+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:58.427580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.092040+0000","last_change":"2024-09-20T22:41:44.199301+0000","last_active":"2024-09-20T22:42:18.092040+0000","last_peered":"2024-09-20T22:42:18.092040+0000","last_clean":"2024-09-20T22:42:18.092040+0000","last_became_active":"2024-09-20T22:41:44.199046+0000","last_became_peered":"2024-09-20T22:41:44.199046+0000","last_unstale":"2024-09-20T22:42:18.092040+0000","last_undegraded":"2024-09-20T22:42:18.092040+0000","last_fullsized":"2024-09-20T22:42:18.092040+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:43.190792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"74'16","reported_seq":62,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-20T22:42:13.209804+0000","last_change":"2024-09-20T22:41:40.253524+0000","last_active":"2024-09-20T22:42:13.209804+0000","last_peered":"2024-09-20T22:42:13.209804+0000","last_clean":"2024-09-20T22:42:13.209804+0000","last_became_active":"2024-09-20T22:41:40.252933+0000","last_became_peered":"2024-09-20T22:41:40.252933+0000","last_unstale":"2024-09-20T22:42:13.209804+0000","last_undegraded":"2024-09-20T22:42:13.209804+0000","last_fullsized":"2024-09-20T22:42:13.209804+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:54.546845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"67'1","reported_seq":43,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.254369+0000","last_change":"2024-09-20T22:41:38.159706+0000","last_active":"2024-09-20T22:42:18.254369+0000","last_peered":"2024-09-20T22:42:18.254369+0000","last_clean":"2024-09-20T22:42:18.254369+0000","last_became_active":"2024-09-20T22:41:38.158917+0000","last_became_peered":"2024-09-20T22:41:38.158917+0000","last_unstale":"2024-09-20T22:42:18.254369+0000","last_undegraded":"2024-09-20T22:42:18.254369+0000","last_fullsized":"2024-09-20T22:42:18.254369+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:03.995423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"77'128","reported_seq":294,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.465364+0000","last_change":"2024-09-20T22:41:43.408723+0000","last_active":"2024-09-20T22:42:18.465364+0000","last_peered":"2024-09-20T22:42:18.465364+0000","last_clean":"2024-09-20T22:42:18.465364+0000","last_became_active":"2024-09-20T22:41:31.091135+0000","last_became_peered":"2024-09-20T22:41:31.091135+0000","last_unstale":"2024-09-20T22:42:18.465364+0000","last_undegraded":"2024-09-20T22:42:18.465364+0000","last_fullsized":"2024-09-20T22:42:18.465364+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'117","last_scrub_stamp":"2024-09-20T22:41:43.408644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.408644+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:48.125134+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":29,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.465620+0000","last_change":"2024-09-20T22:41:42.185135+0000","last_active":"2024-09-20T22:42:18.465620+0000","last_peered":"2024-09-20T22:42:18.465620+0000","last_clean":"2024-09-20T22:42:18.465620+0000","last_became_active":"2024-09-20T22:41:42.184387+0000","last_became_peered":"2024-09-20T22:41:42.184387+0000","last_unstale":"2024-09-20T22:42:18.465620+0000","last_undegraded":"2024-09-20T22:42:18.465620+0000","last_fullsized":"2024-09-20T22:42:18.465620+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:00.112877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.548983+0000","last_change":"2024-09-20T22:41:44.191113+0000","last_active":"2024-09-20T22:42:18.548983+0000","last_peered":"2024-09-20T22:42:18.548983+0000","last_clean":"2024-09-20T22:42:18.548983+0000","last_became_active":"2024-09-20T22:41:44.190658+0000","last_became_peered":"2024-09-20T22:41:44.190658+0000","last_unstale":"2024-09-20T22:42:18.548983+0000","last_undegraded":"2024-09-20T22:42:18.548983+0000","last_fullsized":"2024-09-20T22:42:18.548983+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:44.921190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"74'11","reported_seq":52,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.092699+0000","last_change":"2024-09-20T22:41:40.169542+0000","last_active":"2024-09-20T22:42:18.092699+0000","last_peered":"2024-09-20T22:42:18.092699+0000","last_clean":"2024-09-20T22:42:18.092699+0000","last_became_active":"2024-09-20T22:41:40.168942+0000","last_became_peered":"2024-09-20T22:41:40.168942+0000","last_unstale":"2024-09-20T22:42:18.092699+0000","last_undegraded":"2024-09-20T22:42:18.092699+0000","last_fullsized":"2024-09-20T22:42:18.092699+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:58.224721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":37,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.093019+0000","last_change":"2024-09-20T22:41:38.161801+0000","last_active":"2024-09-20T22:42:18.093019+0000","last_peered":"2024-09-20T22:42:18.093019+0000","last_clean":"2024-09-20T22:42:18.093019+0000","last_became_active":"2024-09-20T22:41:38.161500+0000","last_became_peered":"2024-09-20T22:41:38.161500+0000","last_unstale":"2024-09-20T22:42:18.093019+0000","last_undegraded":"2024-09-20T22:42:18.093019+0000","last_fullsized":"2024-09-20T22:42:18.093019+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:48.636045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":29,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.548988+0000","last_change":"2024-09-20T22:41:42.183601+0000","last_active":"2024-09-20T22:42:18.548988+0000","last_peered":"2024-09-20T22:42:18.548988+0000","last_clean":"2024-09-20T22:42:18.548988+0000","last_became_active":"2024-09-20T22:41:42.183302+0000","last_became_peered":"2024-09-20T22:41:42.183302+0000","last_unstale":"2024-09-20T22:42:18.548988+0000","last_undegraded":"2024-09-20T22:42:18.548988+0000","last_fullsized":"2024-09-20T22:42:18.548988+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:10.335337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.254396+0000","last_change":"2024-09-20T22:41:44.832295+0000","last_active":"2024-09-20T22:42:18.254396+0000","last_peered":"2024-09-20T22:42:18.254396+0000","last_clean":"2024-09-20T22:42:18.254396+0000","last_became_active":"2024-09-20T22:41:44.832009+0000","last_became_peered":"2024-09-20T22:41:44.832009+0000","last_unstale":"2024-09-20T22:42:18.254396+0000","last_undegraded":"2024-09-20T22:42:18.254396+0000","last_fullsized":"2024-09-20T22:42:18.254396+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:56.047489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"74'17","reported_seq":61,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.092915+0000","last_change":"2024-09-20T22:41:40.169498+0000","last_active":"2024-09-20T22:42:18.092915+0000","last_peered":"2024-09-20T22:42:18.092915+0000","last_clean":"2024-09-20T22:42:18.092915+0000","last_became_active":"2024-09-20T22:41:40.169192+0000","last_became_peered":"2024-09-20T22:41:40.169192+0000","last_unstale":"2024-09-20T22:42:18.092915+0000","last_undegraded":"2024-09-20T22:42:18.092915+0000","last_fullsized":"2024-09-20T22:42:18.092915+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:40.654708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":35,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-20T22:42:13.093468+0000","last_change":"2024-09-20T22:41:38.154220+0000","last_active":"2024-09-20T22:42:13.093468+0000","last_peered":"2024-09-20T22:42:13.093468+0000","last_clean":"2024-09-20T22:42:13.093468+0000","last_became_active":"2024-09-20T22:41:38.153926+0000","last_became_peered":"2024-09-20T22:41:38.153926+0000","last_unstale":"2024-09-20T22:42:13.093468+0000","last_undegraded":"2024-09-20T22:42:13.093468+0000","last_fullsized":"2024-09-20T22:42:13.093468+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:54.196177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":29,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.092130+0000","last_change":"2024-09-20T22:41:42.183482+0000","last_active":"2024-09-20T22:42:18.092130+0000","last_peered":"2024-09-20T22:42:18.092130+0000","last_clean":"2024-09-20T22:42:18.092130+0000","last_became_active":"2024-09-20T22:41:42.183225+0000","last_became_peered":"2024-09-20T22:41:42.183225+0000","last_unstale":"2024-09-20T22:42:18.092130+0000","last_undegraded":"2024-09-20T22:42:18.092130+0000","last_fullsized":"2024-09-20T22:42:18.092130+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:10.228360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.318441+0000","last_change":"2024-09-20T22:41:44.197855+0000","last_active":"2024-09-20T22:42:18.318441+0000","last_peered":"2024-09-20T22:42:18.318441+0000","last_clean":"2024-09-20T22:42:18.318441+0000","last_became_active":"2024-09-20T22:41:44.197561+0000","last_became_peered":"2024-09-20T22:41:44.197561+0000","last_unstale":"2024-09-20T22:42:18.318441+0000","last_undegraded":"2024-09-20T22:42:18.318441+0000","last_fullsized":"2024-09-20T22:42:18.318441+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:45.431229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"74'10","reported_seq":48,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.092575+0000","last_change":"2024-09-20T22:41:40.162973+0000","last_active":"2024-09-20T22:42:18.092575+0000","last_peered":"2024-09-20T22:42:18.092575+0000","last_clean":"2024-09-20T22:42:18.092575+0000","last_became_active":"2024-09-20T22:41:40.162494+0000","last_became_peered":"2024-09-20T22:41:40.162494+0000","last_unstale":"2024-09-20T22:42:18.092575+0000","last_undegraded":"2024-09-20T22:42:18.092575+0000","last_fullsized":"2024-09-20T22:42:18.092575+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:49.601652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":37,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.253850+0000","last_change":"2024-09-20T22:41:38.154861+0000","last_active":"2024-09-20T22:42:18.253850+0000","last_peered":"2024-09-20T22:42:18.253850+0000","last_clean":"2024-09-20T22:42:18.253850+0000","last_became_active":"2024-09-20T22:41:38.154636+0000","last_became_peered":"2024-09-20T22:41:38.154636+0000","last_unstale":"2024-09-20T22:42:18.253850+0000","last_undegraded":"2024-09-20T22:42:18.253850+0000","last_fullsized":"2024-09-20T22:42:18.253850+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:45.035202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"74'8","reported_seq":46,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.318307+0000","last_change":"2024-09-20T22:41:42.186528+0000","last_active":"2024-09-20T22:42:18.318307+0000","last_peered":"2024-09-20T22:42:18.318307+0000","last_clean":"2024-09-20T22:42:18.318307+0000","last_became_active":"2024-09-20T22:41:42.185863+0000","last_became_peered":"2024-09-20T22:41:42.185863+0000","last_unstale":"2024-09-20T22:42:18.318307+0000","last_undegraded":"2024-09-20T22:42:18.318307+0000","last_fullsized":"2024-09-20T22:42:18.318307+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:00.762928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.092467+0000","last_change":"2024-09-20T22:41:44.195767+0000","last_active":"2024-09-20T22:42:18.092467+0000","last_peered":"2024-09-20T22:42:18.092467+0000","last_clean":"2024-09-20T22:42:18.092467+0000","last_became_active":"2024-09-20T22:41:44.195160+0000","last_became_peered":"2024-09-20T22:41:44.195160+0000","last_unstale":"2024-09-20T22:42:18.092467+0000","last_undegraded":"2024-09-20T22:42:18.092467+0000","last_fullsized":"2024-09-20T22:42:18.092467+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:07.188986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"74'9","reported_seq":49,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.091485+0000","last_change":"2024-09-20T22:41:40.163774+0000","last_active":"2024-09-20T22:42:18.091485+0000","last_peered":"2024-09-20T22:42:18.091485+0000","last_clean":"2024-09-20T22:42:18.091485+0000","last_became_active":"2024-09-20T22:41:40.163495+0000","last_became_peered":"2024-09-20T22:41:40.163495+0000","last_unstale":"2024-09-20T22:42:18.091485+0000","last_undegraded":"2024-09-20T22:42:18.091485+0000","last_fullsized":"2024-09-20T22:42:18.091485+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:41.614897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":37,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.548693+0000","last_change":"2024-09-20T22:41:38.156470+0000","last_active":"2024-09-20T22:42:18.548693+0000","last_peered":"2024-09-20T22:42:18.548693+0000","last_clean":"2024-09-20T22:42:18.548693+0000","last_became_active":"2024-09-20T22:41:38.155953+0000","last_became_peered":"2024-09-20T22:41:38.155953+0000","last_unstale":"2024-09-20T22:42:18.548693+0000","last_undegraded":"2024-09-20T22:42:18.548693+0000","last_fullsized":"2024-09-20T22:42:18.548693+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:54.057798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":29,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.318125+0000","last_change":"2024-09-20T22:41:42.183269+0000","last_active":"2024-09-20T22:42:18.318125+0000","last_peered":"2024-09-20T22:42:18.318125+0000","last_clean":"2024-09-20T22:42:18.318125+0000","last_became_active":"2024-09-20T22:41:42.182999+0000","last_became_peered":"2024-09-20T22:41:42.182999+0000","last_unstale":"2024-09-20T22:42:18.318125+0000","last_undegraded":"2024-09-20T22:42:18.318125+0000","last_fullsized":"2024-09-20T22:42:18.318125+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:57.955307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.090929+0000","last_change":"2024-09-20T22:41:44.196969+0000","last_active":"2024-09-20T22:42:18.090929+0000","last_peered":"2024-09-20T22:42:18.090929+0000","last_clean":"2024-09-20T22:42:18.090929+0000","last_became_active":"2024-09-20T22:41:44.196633+0000","last_became_peered":"2024-09-20T22:41:44.196633+0000","last_unstale":"2024-09-20T22:42:18.090929+0000","last_undegraded":"2024-09-20T22:42:18.090929+0000","last_fullsized":"2024-09-20T22:42:18.090929+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:03.513323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"74'19","reported_seq":62,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-20T22:42:13.093286+0000","last_change":"2024-09-20T22:41:40.252962+0000","last_active":"2024-09-20T22:42:13.093286+0000","last_peered":"2024-09-20T22:42:13.093286+0000","last_clean":"2024-09-20T22:42:13.093286+0000","last_became_active":"2024-09-20T22:41:40.252011+0000","last_became_peered":"2024-09-20T22:41:40.252011+0000","last_unstale":"2024-09-20T22:42:13.093286+0000","last_undegraded":"2024-09-20T22:42:13.093286+0000","last_fullsized":"2024-09-20T22:42:13.093286+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:51.633432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":37,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.465861+0000","last_change":"2024-09-20T22:41:38.161800+0000","last_active":"2024-09-20T22:42:18.465861+0000","last_peered":"2024-09-20T22:42:18.465861+0000","last_clean":"2024-09-20T22:42:18.465861+0000","last_became_active":"2024-09-20T22:41:38.161524+0000","last_became_peered":"2024-09-20T22:41:38.161524+0000","last_unstale":"2024-09-20T22:42:18.465861+0000","last_undegraded":"2024-09-20T22:42:18.465861+0000","last_fullsized":"2024-09-20T22:42:18.465861+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:07.121209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":29,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.318184+0000","last_change":"2024-09-20T22:41:42.184605+0000","last_active":"2024-09-20T22:42:18.318184+0000","last_peered":"2024-09-20T22:42:18.318184+0000","last_clean":"2024-09-20T22:42:18.318184+0000","last_became_active":"2024-09-20T22:41:42.184342+0000","last_became_peered":"2024-09-20T22:41:42.184342+0000","last_unstale":"2024-09-20T22:42:18.318184+0000","last_undegraded":"2024-09-20T22:42:18.318184+0000","last_fullsized":"2024-09-20T22:42:18.318184+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:03.475959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.465304+0000","last_change":"2024-09-20T22:41:44.199874+0000","last_active":"2024-09-20T22:42:18.465304+0000","last_peered":"2024-09-20T22:42:18.465304+0000","last_clean":"2024-09-20T22:42:18.465304+0000","last_became_active":"2024-09-20T22:41:44.199395+0000","last_became_peered":"2024-09-20T22:41:44.199395+0000","last_unstale":"2024-09-20T22:42:18.465304+0000","last_undegraded":"2024-09-20T22:42:18.465304+0000","last_fullsized":"2024-09-20T22:42:18.465304+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:44.605488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"74'12","reported_seq":56,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.092473+0000","last_change":"2024-09-20T22:41:40.170746+0000","last_active":"2024-09-20T22:42:18.092473+0000","last_peered":"2024-09-20T22:42:18.092473+0000","last_clean":"2024-09-20T22:42:18.092473+0000","last_became_active":"2024-09-20T22:41:40.170133+0000","last_became_peered":"2024-09-20T22:41:40.170133+0000","last_unstale":"2024-09-20T22:42:18.092473+0000","last_undegraded":"2024-09-20T22:42:18.092473+0000","last_fullsized":"2024-09-20T22:42:18.092473+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:39.588868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":37,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.464779+0000","last_change":"2024-09-20T22:41:38.163612+0000","last_active":"2024-09-20T22:42:18.464779+0000","last_peered":"2024-09-20T22:42:18.464779+0000","last_clean":"2024-09-20T22:42:18.464779+0000","last_became_active":"2024-09-20T22:41:38.163292+0000","last_became_peered":"2024-09-20T22:41:38.163292+0000","last_unstale":"2024-09-20T22:42:18.464779+0000","last_undegraded":"2024-09-20T22:42:18.464779+0000","last_fullsized":"2024-09-20T22:42:18.464779+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:45.666357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":29,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.317960+0000","last_change":"2024-09-20T22:41:42.182102+0000","last_active":"2024-09-20T22:42:18.317960+0000","last_peered":"2024-09-20T22:42:18.317960+0000","last_clean":"2024-09-20T22:42:18.317960+0000","last_became_active":"2024-09-20T22:41:42.181811+0000","last_became_peered":"2024-09-20T22:41:42.181811+0000","last_unstale":"2024-09-20T22:42:18.317960+0000","last_undegraded":"2024-09-20T22:42:18.317960+0000","last_fullsized":"2024-09-20T22:42:18.317960+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:52.248246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.254327+0000","last_change":"2024-09-20T22:41:44.193432+0000","last_active":"2024-09-20T22:42:18.254327+0000","last_peered":"2024-09-20T22:42:18.254327+0000","last_clean":"2024-09-20T22:42:18.254327+0000","last_became_active":"2024-09-20T22:41:44.192609+0000","last_became_peered":"2024-09-20T22:41:44.192609+0000","last_unstale":"2024-09-20T22:42:18.254327+0000","last_undegraded":"2024-09-20T22:42:18.254327+0000","last_fullsized":"2024-09-20T22:42:18.254327+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:50.351142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"74'15","reported_seq":58,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.548178+0000","last_change":"2024-09-20T22:41:40.252427+0000","last_active":"2024-09-20T22:42:18.548178+0000","last_peered":"2024-09-20T22:42:18.548178+0000","last_clean":"2024-09-20T22:42:18.548178+0000","last_became_active":"2024-09-20T22:41:40.252113+0000","last_became_peered":"2024-09-20T22:41:40.252113+0000","last_unstale":"2024-09-20T22:42:18.548178+0000","last_undegraded":"2024-09-20T22:42:18.548178+0000","last_fullsized":"2024-09-20T22:42:18.548178+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:06.207870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'2","reported_seq":49,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.464888+0000","last_change":"2024-09-20T22:41:38.163593+0000","last_active":"2024-09-20T22:42:18.464888+0000","last_peered":"2024-09-20T22:42:18.464888+0000","last_clean":"2024-09-20T22:42:18.464888+0000","last_became_active":"2024-09-20T22:41:38.163219+0000","last_became_peered":"2024-09-20T22:41:38.163219+0000","last_unstale":"2024-09-20T22:42:18.464888+0000","last_undegraded":"2024-09-20T22:42:18.464888+0000","last_fullsized":"2024-09-20T22:42:18.464888+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:02.881623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"74'8","reported_seq":46,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.465161+0000","last_change":"2024-09-20T22:41:42.188223+0000","last_active":"2024-09-20T22:42:18.465161+0000","last_peered":"2024-09-20T22:42:18.465161+0000","last_clean":"2024-09-20T22:42:18.465161+0000","last_became_active":"2024-09-20T22:41:42.186635+0000","last_became_peered":"2024-09-20T22:41:42.186635+0000","last_unstale":"2024-09-20T22:42:18.465161+0000","last_undegraded":"2024-09-20T22:42:18.465161+0000","last_fullsized":"2024-09-20T22:42:18.465161+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:52.120303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.548397+0000","last_change":"2024-09-20T22:41:44.831435+0000","last_active":"2024-09-20T22:42:18.548397+0000","last_peered":"2024-09-20T22:42:18.548397+0000","last_clean":"2024-09-20T22:42:18.548397+0000","last_became_active":"2024-09-20T22:41:44.830556+0000","last_became_peered":"2024-09-20T22:41:44.830556+0000","last_unstale":"2024-09-20T22:42:18.548397+0000","last_undegraded":"2024-09-20T22:42:18.548397+0000","last_fullsized":"2024-09-20T22:42:18.548397+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:08.027826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":35,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-20T22:42:13.093186+0000","last_change":"2024-09-20T22:41:38.159437+0000","last_active":"2024-09-20T22:42:13.093186+0000","last_peered":"2024-09-20T22:42:13.093186+0000","last_clean":"2024-09-20T22:42:13.093186+0000","last_became_active":"2024-09-20T22:41:38.159139+0000","last_became_peered":"2024-09-20T22:41:38.159139+0000","last_unstale":"2024-09-20T22:42:13.093186+0000","last_undegraded":"2024-09-20T22:42:13.093186+0000","last_fullsized":"2024-09-20T22:42:13.093186+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:47.658845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"74'6","reported_seq":42,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.254853+0000","last_change":"2024-09-20T22:41:40.252310+0000","last_active":"2024-09-20T22:42:18.254853+0000","last_peered":"2024-09-20T22:42:18.254853+0000","last_clean":"2024-09-20T22:42:18.254853+0000","last_became_active":"2024-09-20T22:41:40.251867+0000","last_became_peered":"2024-09-20T22:41:40.251867+0000","last_unstale":"2024-09-20T22:42:18.254853+0000","last_undegraded":"2024-09-20T22:42:18.254853+0000","last_fullsized":"2024-09-20T22:42:18.254853+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:40.015664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":29,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.548937+0000","last_change":"2024-09-20T22:41:42.185218+0000","last_active":"2024-09-20T22:42:18.548937+0000","last_peered":"2024-09-20T22:42:18.548937+0000","last_clean":"2024-09-20T22:42:18.548937+0000","last_became_active":"2024-09-20T22:41:42.184930+0000","last_became_peered":"2024-09-20T22:41:42.184930+0000","last_unstale":"2024-09-20T22:42:18.548937+0000","last_undegraded":"2024-09-20T22:42:18.548937+0000","last_fullsized":"2024-09-20T22:42:18.548937+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:47.818365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"74'1","reported_seq":26,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.465557+0000","last_change":"2024-09-20T22:41:44.832085+0000","last_active":"2024-09-20T22:42:18.465557+0000","last_peered":"2024-09-20T22:42:18.465557+0000","last_clean":"2024-09-20T22:42:18.465557+0000","last_became_active":"2024-09-20T22:41:44.831805+0000","last_became_peered":"2024-09-20T22:41:44.831805+0000","last_unstale":"2024-09-20T22:42:18.465557+0000","last_undegraded":"2024-09-20T22:42:18.465557+0000","last_fullsized":"2024-09-20T22:42:18.465557+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:54.445212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"74'9","reported_seq":49,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.092560+0000","last_change":"2024-09-20T22:41:40.253633+0000","last_active":"2024-09-20T22:42:18.092560+0000","last_peered":"2024-09-20T22:42:18.092560+0000","last_clean":"2024-09-20T22:42:18.092560+0000","last_became_active":"2024-09-20T22:41:40.253280+0000","last_became_peered":"2024-09-20T22:41:40.253280+0000","last_unstale":"2024-09-20T22:42:18.092560+0000","last_undegraded":"2024-09-20T22:42:18.092560+0000","last_fullsized":"2024-09-20T22:42:18.092560+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:39.928882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":37,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.464667+0000","last_change":"2024-09-20T22:41:38.156773+0000","last_active":"2024-09-20T22:42:18.464667+0000","last_peered":"2024-09-20T22:42:18.464667+0000","last_clean":"2024-09-20T22:42:18.464667+0000","last_became_active":"2024-09-20T22:41:38.156497+0000","last_became_peered":"2024-09-20T22:41:38.156497+0000","last_unstale":"2024-09-20T22:42:18.464667+0000","last_undegraded":"2024-09-20T22:42:18.464667+0000","last_fullsized":"2024-09-20T22:42:18.464667+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:01.943269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":29,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.254776+0000","last_change":"2024-09-20T22:41:42.188166+0000","last_active":"2024-09-20T22:42:18.254776+0000","last_peered":"2024-09-20T22:42:18.254776+0000","last_clean":"2024-09-20T22:42:18.254776+0000","last_became_active":"2024-09-20T22:41:42.187706+0000","last_became_peered":"2024-09-20T22:41:42.187706+0000","last_unstale":"2024-09-20T22:42:18.254776+0000","last_undegraded":"2024-09-20T22:42:18.254776+0000","last_fullsized":"2024-09-20T22:42:18.254776+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:47.610076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.318219+0000","last_change":"2024-09-20T22:41:44.201642+0000","last_active":"2024-09-20T22:42:18.318219+0000","last_peered":"2024-09-20T22:42:18.318219+0000","last_clean":"2024-09-20T22:42:18.318219+0000","last_became_active":"2024-09-20T22:41:44.201433+0000","last_became_peered":"2024-09-20T22:41:44.201433+0000","last_unstale":"2024-09-20T22:42:18.318219+0000","last_undegraded":"2024-09-20T22:42:18.318219+0000","last_fullsized":"2024-09-20T22:42:18.318219+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:55.257671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"74'9","reported_seq":49,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.548923+0000","last_change":"2024-09-20T22:41:40.252840+0000","last_active":"2024-09-20T22:42:18.548923+0000","last_peered":"2024-09-20T22:42:18.548923+0000","last_clean":"2024-09-20T22:42:18.548923+0000","last_became_active":"2024-09-20T22:41:40.252486+0000","last_became_peered":"2024-09-20T22:41:40.252486+0000","last_unstale":"2024-09-20T22:42:18.548923+0000","last_undegraded":"2024-09-20T22:42:18.548923+0000","last_fullsized":"2024-09-20T22:42:18.548923+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:51.443985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":37,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.091310+0000","last_change":"2024-09-20T22:41:38.158479+0000","last_active":"2024-09-20T22:42:18.091310+0000","last_peered":"2024-09-20T22:42:18.091310+0000","last_clean":"2024-09-20T22:42:18.091310+0000","last_became_active":"2024-09-20T22:41:38.158146+0000","last_became_peered":"2024-09-20T22:41:38.158146+0000","last_unstale":"2024-09-20T22:42:18.091310+0000","last_undegraded":"2024-09-20T22:42:18.091310+0000","last_fullsized":"2024-09-20T22:42:18.091310+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:55.658232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"74'8","reported_seq":46,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.254494+0000","last_change":"2024-09-20T22:41:42.186630+0000","last_active":"2024-09-20T22:42:18.254494+0000","last_peered":"2024-09-20T22:42:18.254494+0000","last_clean":"2024-09-20T22:42:18.254494+0000","last_became_active":"2024-09-20T22:41:42.186352+0000","last_became_peered":"2024-09-20T22:41:42.186352+0000","last_unstale":"2024-09-20T22:42:18.254494+0000","last_undegraded":"2024-09-20T22:42:18.254494+0000","last_fullsized":"2024-09-20T22:42:18.254494+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:48.572027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.093209+0000","last_change":"2024-09-20T22:41:44.194510+0000","last_active":"2024-09-20T22:42:18.093209+0000","last_peered":"2024-09-20T22:42:18.093209+0000","last_clean":"2024-09-20T22:42:18.093209+0000","last_became_active":"2024-09-20T22:41:44.194053+0000","last_became_peered":"2024-09-20T22:41:44.194053+0000","last_unstale":"2024-09-20T22:42:18.093209+0000","last_undegraded":"2024-09-20T22:42:18.093209+0000","last_fullsized":"2024-09-20T22:42:18.093209+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:01.207219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"74'10","reported_seq":48,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.254659+0000","last_change":"2024-09-20T22:41:40.253880+0000","last_active":"2024-09-20T22:42:18.254659+0000","last_peered":"2024-09-20T22:42:18.254659+0000","last_clean":"2024-09-20T22:42:18.254659+0000","last_became_active":"2024-09-20T22:41:40.253614+0000","last_became_peered":"2024-09-20T22:41:40.253614+0000","last_unstale":"2024-09-20T22:42:18.254659+0000","last_undegraded":"2024-09-20T22:42:18.254659+0000","last_fullsized":"2024-09-20T22:42:18.254659+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:47.554109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":37,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.464750+0000","last_change":"2024-09-20T22:41:38.158008+0000","last_active":"2024-09-20T22:42:18.464750+0000","last_peered":"2024-09-20T22:42:18.464750+0000","last_clean":"2024-09-20T22:42:18.464750+0000","last_became_active":"2024-09-20T22:41:38.157647+0000","last_became_peered":"2024-09-20T22:41:38.157647+0000","last_unstale":"2024-09-20T22:42:18.464750+0000","last_undegraded":"2024-09-20T22:42:18.464750+0000","last_fullsized":"2024-09-20T22:42:18.464750+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:57.980474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"74'8","reported_seq":46,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.548788+0000","last_change":"2024-09-20T22:41:42.189757+0000","last_active":"2024-09-20T22:42:18.548788+0000","last_peered":"2024-09-20T22:42:18.548788+0000","last_clean":"2024-09-20T22:42:18.548788+0000","last_became_active":"2024-09-20T22:41:42.189451+0000","last_became_peered":"2024-09-20T22:41:42.189451+0000","last_unstale":"2024-09-20T22:42:18.548788+0000","last_undegraded":"2024-09-20T22:42:18.548788+0000","last_fullsized":"2024-09-20T22:42:18.548788+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:43.846079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.091788+0000","last_change":"2024-09-20T22:41:44.191549+0000","last_active":"2024-09-20T22:42:18.091788+0000","last_peered":"2024-09-20T22:42:18.091788+0000","last_clean":"2024-09-20T22:42:18.091788+0000","last_became_active":"2024-09-20T22:41:44.191206+0000","last_became_peered":"2024-09-20T22:41:44.191206+0000","last_unstale":"2024-09-20T22:42:18.091788+0000","last_undegraded":"2024-09-20T22:42:18.091788+0000","last_fullsized":"2024-09-20T22:42:18.091788+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:58.564765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"74'11","reported_seq":52,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.092470+0000","last_change":"2024-09-20T22:41:40.171449+0000","last_active":"2024-09-20T22:42:18.092470+0000","last_peered":"2024-09-20T22:42:18.092470+0000","last_clean":"2024-09-20T22:42:18.092470+0000","last_became_active":"2024-09-20T22:41:40.170035+0000","last_became_peered":"2024-09-20T22:41:40.170035+0000","last_unstale":"2024-09-20T22:42:18.092470+0000","last_undegraded":"2024-09-20T22:42:18.092470+0000","last_fullsized":"2024-09-20T22:42:18.092470+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:49.140455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":37,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.092961+0000","last_change":"2024-09-20T22:41:38.161385+0000","last_active":"2024-09-20T22:42:18.092961+0000","last_peered":"2024-09-20T22:42:18.092961+0000","last_clean":"2024-09-20T22:42:18.092961+0000","last_became_active":"2024-09-20T22:41:38.158142+0000","last_became_peered":"2024-09-20T22:41:38.158142+0000","last_unstale":"2024-09-20T22:42:18.092961+0000","last_undegraded":"2024-09-20T22:42:18.092961+0000","last_fullsized":"2024-09-20T22:42:18.092961+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:51.728688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":29,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.091421+0000","last_change":"2024-09-20T22:41:42.182119+0000","last_active":"2024-09-20T22:42:18.091421+0000","last_peered":"2024-09-20T22:42:18.091421+0000","last_clean":"2024-09-20T22:42:18.091421+0000","last_became_active":"2024-09-20T22:41:42.181736+0000","last_became_peered":"2024-09-20T22:41:42.181736+0000","last_unstale":"2024-09-20T22:42:18.091421+0000","last_undegraded":"2024-09-20T22:42:18.091421+0000","last_fullsized":"2024-09-20T22:42:18.091421+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:02.572377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.254184+0000","last_change":"2024-09-20T22:41:44.193289+0000","last_active":"2024-09-20T22:42:18.254184+0000","last_peered":"2024-09-20T22:42:18.254184+0000","last_clean":"2024-09-20T22:42:18.254184+0000","last_became_active":"2024-09-20T22:41:44.192960+0000","last_became_peered":"2024-09-20T22:41:44.192960+0000","last_unstale":"2024-09-20T22:42:18.254184+0000","last_undegraded":"2024-09-20T22:42:18.254184+0000","last_fullsized":"2024-09-20T22:42:18.254184+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:49.189046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"74'9","reported_seq":49,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.091501+0000","last_change":"2024-09-20T22:41:40.165116+0000","last_active":"2024-09-20T22:42:18.091501+0000","last_peered":"2024-09-20T22:42:18.091501+0000","last_clean":"2024-09-20T22:42:18.091501+0000","last_became_active":"2024-09-20T22:41:40.164796+0000","last_became_peered":"2024-09-20T22:41:40.164796+0000","last_unstale":"2024-09-20T22:42:18.091501+0000","last_undegraded":"2024-09-20T22:42:18.091501+0000","last_fullsized":"2024-09-20T22:42:18.091501+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:48.483009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":37,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.464733+0000","last_change":"2024-09-20T22:41:38.159941+0000","last_active":"2024-09-20T22:42:18.464733+0000","last_peered":"2024-09-20T22:42:18.464733+0000","last_clean":"2024-09-20T22:42:18.464733+0000","last_became_active":"2024-09-20T22:41:38.159657+0000","last_became_peered":"2024-09-20T22:41:38.159657+0000","last_unstale":"2024-09-20T22:42:18.464733+0000","last_undegraded":"2024-09-20T22:42:18.464733+0000","last_fullsized":"2024-09-20T22:42:18.464733+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:53.225190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":29,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.254383+0000","last_change":"2024-09-20T22:41:42.183254+0000","last_active":"2024-09-20T22:42:18.254383+0000","last_peered":"2024-09-20T22:42:18.254383+0000","last_clean":"2024-09-20T22:42:18.254383+0000","last_became_active":"2024-09-20T22:41:42.182966+0000","last_became_peered":"2024-09-20T22:41:42.182966+0000","last_unstale":"2024-09-20T22:42:18.254383+0000","last_undegraded":"2024-09-20T22:42:18.254383+0000","last_fullsized":"2024-09-20T22:42:18.254383+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:51.689726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.090618+0000","last_change":"2024-09-20T22:41:44.194845+0000","last_active":"2024-09-20T22:42:18.090618+0000","last_peered":"2024-09-20T22:42:18.090618+0000","last_clean":"2024-09-20T22:42:18.090618+0000","last_became_active":"2024-09-20T22:41:44.194551+0000","last_became_peered":"2024-09-20T22:41:44.194551+0000","last_unstale":"2024-09-20T22:42:18.090618+0000","last_undegraded":"2024-09-20T22:42:18.090618+0000","last_fullsized":"2024-09-20T22:42:18.090618+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:45.187444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"74'11","reported_seq":52,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.254292+0000","last_change":"2024-09-20T22:41:40.252149+0000","last_active":"2024-09-20T22:42:18.254292+0000","last_peered":"2024-09-20T22:42:18.254292+0000","last_clean":"2024-09-20T22:42:18.254292+0000","last_became_active":"2024-09-20T22:41:40.251829+0000","last_became_peered":"2024-09-20T22:41:40.251829+0000","last_unstale":"2024-09-20T22:42:18.254292+0000","last_undegraded":"2024-09-20T22:42:18.254292+0000","last_fullsized":"2024-09-20T22:42:18.254292+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:56.020491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":37,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.547951+0000","last_change":"2024-09-20T22:41:38.159064+0000","last_active":"2024-09-20T22:42:18.547951+0000","last_peered":"2024-09-20T22:42:18.547951+0000","last_clean":"2024-09-20T22:42:18.547951+0000","last_became_active":"2024-09-20T22:41:38.158701+0000","last_became_peered":"2024-09-20T22:41:38.158701+0000","last_unstale":"2024-09-20T22:42:18.547951+0000","last_undegraded":"2024-09-20T22:42:18.547951+0000","last_fullsized":"2024-09-20T22:42:18.547951+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:47.727493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":29,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.464964+0000","last_change":"2024-09-20T22:41:42.185991+0000","last_active":"2024-09-20T22:42:18.464964+0000","last_peered":"2024-09-20T22:42:18.464964+0000","last_clean":"2024-09-20T22:42:18.464964+0000","last_became_active":"2024-09-20T22:41:42.185296+0000","last_became_peered":"2024-09-20T22:41:42.185296+0000","last_unstale":"2024-09-20T22:42:18.464964+0000","last_undegraded":"2024-09-20T22:42:18.464964+0000","last_fullsized":"2024-09-20T22:42:18.464964+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:55.259880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.254496+0000","last_change":"2024-09-20T22:41:44.832293+0000","last_active":"2024-09-20T22:42:18.254496+0000","last_peered":"2024-09-20T22:42:18.254496+0000","last_clean":"2024-09-20T22:42:18.254496+0000","last_became_active":"2024-09-20T22:41:44.831967+0000","last_became_peered":"2024-09-20T22:41:44.831967+0000","last_unstale":"2024-09-20T22:42:18.254496+0000","last_undegraded":"2024-09-20T22:42:18.254496+0000","last_fullsized":"2024-09-20T22:42:18.254496+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:03.371689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"74'4","reported_seq":39,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.254126+0000","last_change":"2024-09-20T22:41:40.166409+0000","last_active":"2024-09-20T22:42:18.254126+0000","last_peered":"2024-09-20T22:42:18.254126+0000","last_clean":"2024-09-20T22:42:18.254126+0000","last_became_active":"2024-09-20T22:41:40.166123+0000","last_became_peered":"2024-09-20T22:41:40.166123+0000","last_unstale":"2024-09-20T22:42:18.254126+0000","last_undegraded":"2024-09-20T22:42:18.254126+0000","last_fullsized":"2024-09-20T22:42:18.254126+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:52.943341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":37,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.091882+0000","last_change":"2024-09-20T22:41:38.158861+0000","last_active":"2024-09-20T22:42:18.091882+0000","last_peered":"2024-09-20T22:42:18.091882+0000","last_clean":"2024-09-20T22:42:18.091882+0000","last_became_active":"2024-09-20T22:41:38.158347+0000","last_became_peered":"2024-09-20T22:41:38.158347+0000","last_unstale":"2024-09-20T22:42:18.091882+0000","last_undegraded":"2024-09-20T22:42:18.091882+0000","last_fullsized":"2024-09-20T22:42:18.091882+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:44.061004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":27,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-20T22:42:13.093113+0000","last_change":"2024-09-20T22:41:42.187707+0000","last_active":"2024-09-20T22:42:13.093113+0000","last_peered":"2024-09-20T22:42:13.093113+0000","last_clean":"2024-09-20T22:42:13.093113+0000","last_became_active":"2024-09-20T22:41:42.186068+0000","last_became_peered":"2024-09-20T22:41:42.186068+0000","last_unstale":"2024-09-20T22:42:13.093113+0000","last_undegraded":"2024-09-20T22:42:13.093113+0000","last_fullsized":"2024-09-20T22:42:13.093113+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:50.526395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.465190+0000","last_change":"2024-09-20T22:41:44.194011+0000","last_active":"2024-09-20T22:42:18.465190+0000","last_peered":"2024-09-20T22:42:18.465190+0000","last_clean":"2024-09-20T22:42:18.465190+0000","last_became_active":"2024-09-20T22:41:44.193706+0000","last_became_peered":"2024-09-20T22:41:44.193706+0000","last_unstale":"2024-09-20T22:42:18.465190+0000","last_undegraded":"2024-09-20T22:42:18.465190+0000","last_fullsized":"2024-09-20T22:42:18.465190+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:49.835925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.092142+0000","last_change":"2024-09-20T22:41:44.198154+0000","last_active":"2024-09-20T22:42:18.092142+0000","last_peered":"2024-09-20T22:42:18.092142+0000","last_clean":"2024-09-20T22:42:18.092142+0000","last_became_active":"2024-09-20T22:41:44.197855+0000","last_became_peered":"2024-09-20T22:41:44.197855+0000","last_unstale":"2024-09-20T22:42:18.092142+0000","last_undegraded":"2024-09-20T22:42:18.092142+0000","last_fullsized":"2024-09-20T22:42:18.092142+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:56.991123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":37,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.253830+0000","last_change":"2024-09-20T22:41:38.155001+0000","last_active":"2024-09-20T22:42:18.253830+0000","last_peered":"2024-09-20T22:42:18.253830+0000","last_clean":"2024-09-20T22:42:18.253830+0000","last_became_active":"2024-09-20T22:41:38.154703+0000","last_became_peered":"2024-09-20T22:41:38.154703+0000","last_unstale":"2024-09-20T22:42:18.253830+0000","last_undegraded":"2024-09-20T22:42:18.253830+0000","last_fullsized":"2024-09-20T22:42:18.253830+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:44.782808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"74'11","reported_seq":50,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-20T22:42:13.093880+0000","last_change":"2024-09-20T22:41:40.171840+0000","last_active":"2024-09-20T22:42:13.093880+0000","last_peered":"2024-09-20T22:42:13.093880+0000","last_clean":"2024-09-20T22:42:13.093880+0000","last_became_active":"2024-09-20T22:41:40.171256+0000","last_became_peered":"2024-09-20T22:41:40.171256+0000","last_unstale":"2024-09-20T22:42:13.093880+0000","last_undegraded":"2024-09-20T22:42:13.093880+0000","last_fullsized":"2024-09-20T22:42:13.093880+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:53.884281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":29,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-20T22:42:18.091807+0000","last_change":"2024-09-20T22:41:42.178413+0000","last_active":"2024-09-20T22:42:18.091807+0000","last_peered":"2024-09-20T22:42:18.091807+0000","last_clean":"2024-09-20T22:42:18.091807+0000","last_became_active":"2024-09-20T22:41:42.178157+0000","last_became_peered":"2024-09-20T22:41:42.178157+0000","last_unstale":"2024-09-20T22:42:18.091807+0000","last_undegraded":"2024-09-20T22:42:18.091807+0000","last_fullsized":"2024-09-20T22:42:18.091807+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:01.837489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":393,"ondisk_log_size":393,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":35,"num_read_kb":30,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070476,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32436,"kb_used_data":1248,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712716,"statfs":{"total":95995035648,"available":95961821184,"internally_reserved":0,"allocated":1277952,"data_stored":879981,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430226,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32424,"kb_used_data":1232,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712728,"statfs":{"total":95995035648,"available":95961833472,"internally_reserved":0,"allocated":1261568,"data_stored":879889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691863,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27684,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717468,"statfs":{"total":95995035648,"available":95966687232,"internally_reserved":0,"allocated":671744,"data_stored":290303,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986204,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27704,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717448,"statfs":{"total":95995035648,"available":95966666752,"internally_reserved":0,"allocated":692224,"data_stored":289133,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182434,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27724,"kb_used_data":696,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717428,"statfs":{"total":95995035648,"available":95966646272,"internally_reserved":0,"allocated":712704,"data_stored":296771,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411367,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31832,"kb_used_data":644,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713320,"statfs":{"total":95995035648,"available":95962439680,"internally_reserved":0,"allocated":659456,"data_stored":289099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574892,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27736,"kb_used_data":708,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717416,"statfs":{"total":95995035648,"available":95966633984,"internally_reserved":0,"allocated":724992,"data_stored":290300,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738417,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32440,"kb_used_data":1252,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712712,"statfs":{"total":95995035648,"available":95961817088,"internally_reserved":0,"allocated":1282048,"data_stored":880113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-20T22:42:22.267 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-09-20T22:42:22.267 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-20T22:42:22.268 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-09-20T22:42:22.268 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph health --format=json 2024-09-20T22:42:22.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:22 smithi154 ceph-mon[33264]: from='client.14739 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T22:42:22.534 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:42:22.614 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:22 smithi084 ceph-mon[40024]: from='client.14739 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T22:42:22.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:22 smithi084 ceph-mon[29774]: from='client.14739 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T22:42:23.350 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-20T22:42:23.350 INFO:teuthology.orchestra.run.smithi084.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-20T22:42:23.351 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:23 smithi084 ceph-mon[29774]: pgmap v14: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 0 B/s wr, 15 op/s 2024-09-20T22:42:23.351 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:23 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:42:23.351 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:23 smithi084 ceph-mon[29774]: osdmap e82: 8 total, 8 up, 8 in 2024-09-20T22:42:23.351 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:23 smithi084 ceph-mon[40024]: pgmap v14: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 0 B/s wr, 15 op/s 2024-09-20T22:42:23.351 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:23 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:42:23.351 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:23 smithi084 ceph-mon[40024]: osdmap e82: 8 total, 8 up, 8 in 2024-09-20T22:42:23.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:23 smithi154 ceph-mon[33264]: pgmap v14: 132 pgs: 132 active+clean; 583 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 0 B/s wr, 15 op/s 2024-09-20T22:42:23.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:23 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:42:23.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:23 smithi154 ceph-mon[33264]: osdmap e82: 8 total, 8 up, 8 in 2024-09-20T22:42:24.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:24 smithi154 ceph-mon[33264]: purged_snaps scrub starts 2024-09-20T22:42:24.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:24 smithi154 ceph-mon[33264]: purged_snaps scrub ok 2024-09-20T22:42:24.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:24 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/1938901951' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T22:42:24.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:24 smithi084 ceph-mon[40024]: purged_snaps scrub starts 2024-09-20T22:42:24.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:24 smithi084 ceph-mon[40024]: purged_snaps scrub ok 2024-09-20T22:42:24.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:24 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/1938901951' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T22:42:24.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:24 smithi084 ceph-mon[29774]: purged_snaps scrub starts 2024-09-20T22:42:24.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:24 smithi084 ceph-mon[29774]: purged_snaps scrub ok 2024-09-20T22:42:24.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:24 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/1938901951' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T22:42:24.995 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-09-20T22:42:24.996 INFO:tasks.cephadm:Setup complete, yielding 2024-09-20T22:42:24.996 INFO:teuthology.run_tasks:Running task thrashosds... 2024-09-20T22:42:25.997 INFO:tasks.thrashosds:config is {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-09-20T22:42:25.997 INFO:tasks.thrashosds:overrides is {'admin_socket': {'branch': 'squid'}, 'ceph': {'conf': {'global': {'ms bind msgr1': False, 'ms bind msgr2': True, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd max backfills': 3, 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'CEPHADM_STRAY_DAEMON', 'CEPHADM_FAILED_DAEMON', 'but it is still running', 'objects unfound and apparently lost', '\\(POOL_APP_NOT_ENABLED\\)'], 'log-only-match': ['CEPHADM_'], 'sha1': '9e2ac70bb7aea67d34f0d50002344abbe7803736'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'cephadm': {'cephadm_mode': 'root'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '9e2ac70bb7aea67d34f0d50002344abbe7803736'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0']}, 'workunit': {'branch': 'squid', 'sha1': '4e45e6b490e6cbe082cdf155fea780438ec53fc7'}} 2024-09-20T22:42:25.997 INFO:tasks.thrashosds:config is {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-09-20T22:42:25.997 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph config set mgr mgr_debug_aggressive_pg_num_changes true 2024-09-20T22:42:25.998 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:25 smithi154 ceph-mon[33264]: pgmap v16: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-20T22:42:25.998 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:25 smithi084 ceph-mon[40024]: pgmap v16: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-20T22:42:25.999 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:25 smithi084 ceph-mon[29774]: pgmap v16: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-20T22:42:27.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:27 smithi154 ceph-mon[33264]: pgmap v17: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T22:42:27.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:27 smithi084 ceph-mon[29774]: pgmap v17: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T22:42:27.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:27 smithi084 ceph-mon[40024]: pgmap v17: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T22:42:28.233 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:42:28.422 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:42:28 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: ::ffff:172.21.15.154 - - [20/Sep/2024:22:42:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-20T22:42:29.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:29 smithi154 ceph-mon[33264]: pgmap v18: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 602 B/s rd, 0 op/s 2024-09-20T22:42:29.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:29 smithi154 ceph-mon[33264]: osdmap e83: 8 total, 8 up, 8 in 2024-09-20T22:42:29.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:29 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/3847455352' entity='client.admin' 2024-09-20T22:42:29.254 INFO:journalctl@ceph.iscsi.iscsi.a.smithi154.stdout:Sep 20 22:42:28 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-iscsi-iscsi-a[59730]: debug there is no tcmu-runner data available 2024-09-20T22:42:29.322 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:29 smithi084 ceph-mon[40024]: pgmap v18: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 602 B/s rd, 0 op/s 2024-09-20T22:42:29.322 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:29 smithi084 ceph-mon[40024]: osdmap e83: 8 total, 8 up, 8 in 2024-09-20T22:42:29.322 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:29 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/3847455352' entity='client.admin' 2024-09-20T22:42:29.322 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:29 smithi084 ceph-mon[29774]: pgmap v18: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 602 B/s rd, 0 op/s 2024-09-20T22:42:29.322 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:29 smithi084 ceph-mon[29774]: osdmap e83: 8 total, 8 up, 8 in 2024-09-20T22:42:29.322 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:29 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/3847455352' entity='client.admin' 2024-09-20T22:42:29.322 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:42:29 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: [20/Sep/2024:22:42:29] ENGINE Bus STOPPING 2024-09-20T22:42:29.323 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:42:29 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: [20/Sep/2024:22:42:29] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-20T22:42:29.323 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:42:29 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: [20/Sep/2024:22:42:29] ENGINE Bus STOPPED 2024-09-20T22:42:29.323 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:42:29 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: [20/Sep/2024:22:42:29] ENGINE Bus STARTING 2024-09-20T22:42:29.614 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:42:29 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: [20/Sep/2024:22:42:29] ENGINE Serving on http://:::9283 2024-09-20T22:42:29.615 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:42:29 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: [20/Sep/2024:22:42:29] ENGINE Bus STARTED 2024-09-20T22:42:29.984 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:42:29 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: [20/Sep/2024:22:42:29] ENGINE Bus STOPPING 2024-09-20T22:42:29.985 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:42:29 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: [20/Sep/2024:22:42:29] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-20T22:42:29.985 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:42:29 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: [20/Sep/2024:22:42:29] ENGINE Bus STOPPED 2024-09-20T22:42:29.985 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:42:29 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: [20/Sep/2024:22:42:29] ENGINE Bus STARTING 2024-09-20T22:42:29.985 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:42:29 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: [20/Sep/2024:22:42:29] ENGINE Serving on http://:::9283 2024-09-20T22:42:30.236 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:42:29 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: [20/Sep/2024:22:42:29] ENGINE Bus STARTED 2024-09-20T22:42:30.236 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:30 smithi084 ceph-mon[40024]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:42:30.236 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:30 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:42:30.236 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:30 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:42:30.236 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:30 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:42:30.237 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:30 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:42:30.237 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:30 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:42:30.237 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:30 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:42:30.237 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:30 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-20T22:42:30.237 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:30 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:42:30.237 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:30 smithi084 ceph-mon[29774]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:42:30.238 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:30 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:42:30.238 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:30 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:42:30.238 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:30 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:42:30.238 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:30 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:42:30.238 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:30 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:42:30.238 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:30 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:42:30.238 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:30 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-20T22:42:30.238 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:30 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:42:30.263 INFO:tasks.thrashosds:Beginning thrashosds... 2024-09-20T22:42:30.263 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-09-20T22:42:30.263 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph pg dump --format=json 2024-09-20T22:42:30.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:30 smithi154 ceph-mon[33264]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:42:30.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:30 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:42:30.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:30 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:42:30.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:30 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:42:30.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:30 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:42:30.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:30 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:42:30.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:30 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:42:30.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:30 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-20T22:42:30.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:30 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:42:30.525 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:42:31.145 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-20T22:42:31.146 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:31 smithi084 ceph-mon[40024]: Regenerating cephadm self-signed grafana TLS certificates 2024-09-20T22:42:31.146 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:31 smithi084 ceph-mon[40024]: pgmap v20: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:42:31.146 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:31 smithi084 ceph-mon[40024]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-20T22:42:31.146 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:31 smithi084 ceph-mon[40024]: Deploying daemon grafana.a on smithi154 2024-09-20T22:42:31.147 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:31 smithi084 ceph-mon[40024]: purged_snaps scrub starts 2024-09-20T22:42:31.147 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:31 smithi084 ceph-mon[40024]: purged_snaps scrub ok 2024-09-20T22:42:31.148 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:31 smithi084 ceph-mon[29774]: Regenerating cephadm self-signed grafana TLS certificates 2024-09-20T22:42:31.148 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:31 smithi084 ceph-mon[29774]: pgmap v20: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:42:31.148 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:31 smithi084 ceph-mon[29774]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-20T22:42:31.148 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:31 smithi084 ceph-mon[29774]: Deploying daemon grafana.a on smithi154 2024-09-20T22:42:31.148 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:31 smithi084 ceph-mon[29774]: purged_snaps scrub starts 2024-09-20T22:42:31.148 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:31 smithi084 ceph-mon[29774]: purged_snaps scrub ok 2024-09-20T22:42:31.153 INFO:teuthology.orchestra.run.smithi084.stderr:dumped all 2024-09-20T22:42:31.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:31 smithi154 ceph-mon[33264]: Regenerating cephadm self-signed grafana TLS certificates 2024-09-20T22:42:31.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:31 smithi154 ceph-mon[33264]: pgmap v20: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:42:31.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:31 smithi154 ceph-mon[33264]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-20T22:42:31.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:31 smithi154 ceph-mon[33264]: Deploying daemon grafana.a on smithi154 2024-09-20T22:42:31.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:31 smithi154 ceph-mon[33264]: purged_snaps scrub starts 2024-09-20T22:42:31.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:31 smithi154 ceph-mon[33264]: purged_snaps scrub ok 2024-09-20T22:42:32.135 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:32 smithi084 ceph-mon[40024]: from='client.24656 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T22:42:32.135 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:32 smithi084 ceph-mon[40024]: osdmap e84: 8 total, 8 up, 8 in 2024-09-20T22:42:32.135 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:32 smithi084 ceph-mon[29774]: from='client.24656 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T22:42:32.135 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:32 smithi084 ceph-mon[29774]: osdmap e84: 8 total, 8 up, 8 in 2024-09-20T22:42:32.164 INFO:teuthology.orchestra.run.smithi084.stdout:{"pg_ready":true,"pg_map":{"version":20,"stamp":"2024-09-20T22:42:29.614600+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596507,"num_objects":199,"num_object_clones":0,"num_object_copies":597,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":199,"num_whiteouts":0,"num_read":979,"num_read_kb":825,"num_write":615,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":601,"ondisk_log_size":601,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":240252,"kb_used_data":7388,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":749720964,"statfs":{"total":767960285184,"available":767714267136,"internally_reserved":0,"allocated":7565312,"data_stored":4242945,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001550"},"pg_stats":[{"pgid":"3.1f","version":"67'1","reported_seq":49,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.101413+0000","last_change":"2024-09-20T22:41:38.159365+0000","last_active":"2024-09-20T22:42:28.101413+0000","last_peered":"2024-09-20T22:42:28.101413+0000","last_clean":"2024-09-20T22:42:28.101413+0000","last_became_active":"2024-09-20T22:41:38.159085+0000","last_became_peered":"2024-09-20T22:41:38.159085+0000","last_unstale":"2024-09-20T22:42:28.101413+0000","last_undegraded":"2024-09-20T22:42:28.101413+0000","last_fullsized":"2024-09-20T22:42:28.101413+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:37.732756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"74'9","reported_seq":55,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.099427+0000","last_change":"2024-09-20T22:41:40.169673+0000","last_active":"2024-09-20T22:42:28.099427+0000","last_peered":"2024-09-20T22:42:28.099427+0000","last_clean":"2024-09-20T22:42:28.099427+0000","last_became_active":"2024-09-20T22:41:40.168934+0000","last_became_peered":"2024-09-20T22:41:40.168934+0000","last_unstale":"2024-09-20T22:42:28.099427+0000","last_undegraded":"2024-09-20T22:42:28.099427+0000","last_fullsized":"2024-09-20T22:42:28.099427+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:54.489342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.099551+0000","last_change":"2024-09-20T22:41:42.182304+0000","last_active":"2024-09-20T22:42:28.099551+0000","last_peered":"2024-09-20T22:42:28.099551+0000","last_clean":"2024-09-20T22:42:28.099551+0000","last_became_active":"2024-09-20T22:41:42.182024+0000","last_became_peered":"2024-09-20T22:41:42.182024+0000","last_unstale":"2024-09-20T22:42:28.099551+0000","last_undegraded":"2024-09-20T22:42:28.099551+0000","last_fullsized":"2024-09-20T22:42:28.099551+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:57.397887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":31,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.099576+0000","last_change":"2024-09-20T22:41:44.198207+0000","last_active":"2024-09-20T22:42:28.099576+0000","last_peered":"2024-09-20T22:42:28.099576+0000","last_clean":"2024-09-20T22:42:28.099576+0000","last_became_active":"2024-09-20T22:41:44.197897+0000","last_became_peered":"2024-09-20T22:41:44.197897+0000","last_unstale":"2024-09-20T22:42:28.099576+0000","last_undegraded":"2024-09-20T22:42:28.099576+0000","last_fullsized":"2024-09-20T22:42:28.099576+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:53.466451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":31,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.477740+0000","last_change":"2024-09-20T22:41:44.832449+0000","last_active":"2024-09-20T22:42:28.477740+0000","last_peered":"2024-09-20T22:42:28.477740+0000","last_clean":"2024-09-20T22:42:28.477740+0000","last_became_active":"2024-09-20T22:41:44.832099+0000","last_became_peered":"2024-09-20T22:41:44.832099+0000","last_unstale":"2024-09-20T22:42:28.477740+0000","last_undegraded":"2024-09-20T22:42:28.477740+0000","last_fullsized":"2024-09-20T22:42:28.477740+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:45.326749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":43,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.478137+0000","last_change":"2024-09-20T22:41:38.157685+0000","last_active":"2024-09-20T22:42:28.478137+0000","last_peered":"2024-09-20T22:42:28.478137+0000","last_clean":"2024-09-20T22:42:28.478137+0000","last_became_active":"2024-09-20T22:41:38.157298+0000","last_became_peered":"2024-09-20T22:41:38.157298+0000","last_unstale":"2024-09-20T22:42:28.478137+0000","last_undegraded":"2024-09-20T22:42:28.478137+0000","last_fullsized":"2024-09-20T22:42:28.478137+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:00.813798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"74'15","reported_seq":64,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.476697+0000","last_change":"2024-09-20T22:41:40.163585+0000","last_active":"2024-09-20T22:42:28.476697+0000","last_peered":"2024-09-20T22:42:28.476697+0000","last_clean":"2024-09-20T22:42:28.476697+0000","last_became_active":"2024-09-20T22:41:40.163242+0000","last_became_peered":"2024-09-20T22:41:40.163242+0000","last_unstale":"2024-09-20T22:42:28.476697+0000","last_undegraded":"2024-09-20T22:42:28.476697+0000","last_fullsized":"2024-09-20T22:42:28.476697+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:58.410323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.099423+0000","last_change":"2024-09-20T22:41:42.187359+0000","last_active":"2024-09-20T22:42:28.099423+0000","last_peered":"2024-09-20T22:42:28.099423+0000","last_clean":"2024-09-20T22:42:28.099423+0000","last_became_active":"2024-09-20T22:41:42.187076+0000","last_became_peered":"2024-09-20T22:41:42.187076+0000","last_unstale":"2024-09-20T22:42:28.099423+0000","last_undegraded":"2024-09-20T22:42:28.099423+0000","last_fullsized":"2024-09-20T22:42:28.099423+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:42.282475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":43,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.519203+0000","last_change":"2024-09-20T22:41:38.156694+0000","last_active":"2024-09-20T22:42:28.519203+0000","last_peered":"2024-09-20T22:42:28.519203+0000","last_clean":"2024-09-20T22:42:28.519203+0000","last_became_active":"2024-09-20T22:41:38.154827+0000","last_became_peered":"2024-09-20T22:41:38.154827+0000","last_unstale":"2024-09-20T22:42:28.519203+0000","last_undegraded":"2024-09-20T22:42:28.519203+0000","last_fullsized":"2024-09-20T22:42:28.519203+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:48.618357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"74'9","reported_seq":55,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.099718+0000","last_change":"2024-09-20T22:41:40.163006+0000","last_active":"2024-09-20T22:42:28.099718+0000","last_peered":"2024-09-20T22:42:28.099718+0000","last_clean":"2024-09-20T22:42:28.099718+0000","last_became_active":"2024-09-20T22:41:40.162546+0000","last_became_peered":"2024-09-20T22:41:40.162546+0000","last_unstale":"2024-09-20T22:42:28.099718+0000","last_undegraded":"2024-09-20T22:42:28.099718+0000","last_fullsized":"2024-09-20T22:42:28.099718+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:46.501647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.518950+0000","last_change":"2024-09-20T22:41:42.185835+0000","last_active":"2024-09-20T22:42:28.518950+0000","last_peered":"2024-09-20T22:42:28.518950+0000","last_clean":"2024-09-20T22:42:28.518950+0000","last_became_active":"2024-09-20T22:41:42.185538+0000","last_became_peered":"2024-09-20T22:41:42.185538+0000","last_unstale":"2024-09-20T22:42:28.518950+0000","last_undegraded":"2024-09-20T22:42:28.518950+0000","last_fullsized":"2024-09-20T22:42:28.518950+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:57.305761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":31,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.099354+0000","last_change":"2024-09-20T22:41:44.198169+0000","last_active":"2024-09-20T22:42:28.099354+0000","last_peered":"2024-09-20T22:42:28.099354+0000","last_clean":"2024-09-20T22:42:28.099354+0000","last_became_active":"2024-09-20T22:41:44.197113+0000","last_became_peered":"2024-09-20T22:41:44.197113+0000","last_unstale":"2024-09-20T22:42:28.099354+0000","last_undegraded":"2024-09-20T22:42:28.099354+0000","last_fullsized":"2024-09-20T22:42:28.099354+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:02.374640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":43,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.519140+0000","last_change":"2024-09-20T22:41:38.156116+0000","last_active":"2024-09-20T22:42:28.519140+0000","last_peered":"2024-09-20T22:42:28.519140+0000","last_clean":"2024-09-20T22:42:28.519140+0000","last_became_active":"2024-09-20T22:41:38.154650+0000","last_became_peered":"2024-09-20T22:41:38.154650+0000","last_unstale":"2024-09-20T22:42:28.519140+0000","last_undegraded":"2024-09-20T22:42:28.519140+0000","last_fullsized":"2024-09-20T22:42:28.519140+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:44.595033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"74'5","reported_seq":49,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.099778+0000","last_change":"2024-09-20T22:41:40.171883+0000","last_active":"2024-09-20T22:42:28.099778+0000","last_peered":"2024-09-20T22:42:28.099778+0000","last_clean":"2024-09-20T22:42:28.099778+0000","last_became_active":"2024-09-20T22:41:40.171081+0000","last_became_peered":"2024-09-20T22:41:40.171081+0000","last_unstale":"2024-09-20T22:42:28.099778+0000","last_undegraded":"2024-09-20T22:42:28.099778+0000","last_fullsized":"2024-09-20T22:42:28.099778+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:55.048729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.386085+0000","last_change":"2024-09-20T22:41:42.183965+0000","last_active":"2024-09-20T22:42:28.386085+0000","last_peered":"2024-09-20T22:42:28.386085+0000","last_clean":"2024-09-20T22:42:28.386085+0000","last_became_active":"2024-09-20T22:41:42.182916+0000","last_became_peered":"2024-09-20T22:41:42.182916+0000","last_unstale":"2024-09-20T22:42:28.386085+0000","last_undegraded":"2024-09-20T22:42:28.386085+0000","last_fullsized":"2024-09-20T22:42:28.386085+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:42.258865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":31,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.518829+0000","last_change":"2024-09-20T22:41:44.198221+0000","last_active":"2024-09-20T22:42:28.518829+0000","last_peered":"2024-09-20T22:42:28.518829+0000","last_clean":"2024-09-20T22:42:28.518829+0000","last_became_active":"2024-09-20T22:41:44.197927+0000","last_became_peered":"2024-09-20T22:41:44.197927+0000","last_unstale":"2024-09-20T22:42:28.518829+0000","last_undegraded":"2024-09-20T22:42:28.518829+0000","last_fullsized":"2024-09-20T22:42:28.518829+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:47.985353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":31,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.099979+0000","last_change":"2024-09-20T22:41:44.832646+0000","last_active":"2024-09-20T22:42:28.099979+0000","last_peered":"2024-09-20T22:42:28.099979+0000","last_clean":"2024-09-20T22:42:28.099979+0000","last_became_active":"2024-09-20T22:41:44.832360+0000","last_became_peered":"2024-09-20T22:41:44.832360+0000","last_unstale":"2024-09-20T22:42:28.099979+0000","last_undegraded":"2024-09-20T22:42:28.099979+0000","last_fullsized":"2024-09-20T22:42:28.099979+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:45.707782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":43,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.100583+0000","last_change":"2024-09-20T22:41:38.152726+0000","last_active":"2024-09-20T22:42:28.100583+0000","last_peered":"2024-09-20T22:42:28.100583+0000","last_clean":"2024-09-20T22:42:28.100583+0000","last_became_active":"2024-09-20T22:41:38.152443+0000","last_became_peered":"2024-09-20T22:41:38.152443+0000","last_unstale":"2024-09-20T22:42:28.100583+0000","last_undegraded":"2024-09-20T22:42:28.100583+0000","last_fullsized":"2024-09-20T22:42:28.100583+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:03.325746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"74'15","reported_seq":64,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.266322+0000","last_change":"2024-09-20T22:41:40.170414+0000","last_active":"2024-09-20T22:42:28.266322+0000","last_peered":"2024-09-20T22:42:28.266322+0000","last_clean":"2024-09-20T22:42:28.266322+0000","last_became_active":"2024-09-20T22:41:40.170131+0000","last_became_peered":"2024-09-20T22:41:40.170131+0000","last_unstale":"2024-09-20T22:42:28.266322+0000","last_undegraded":"2024-09-20T22:42:28.266322+0000","last_fullsized":"2024-09-20T22:42:28.266322+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:57.393598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.100537+0000","last_change":"2024-09-20T22:41:42.185171+0000","last_active":"2024-09-20T22:42:28.100537+0000","last_peered":"2024-09-20T22:42:28.100537+0000","last_clean":"2024-09-20T22:42:28.100537+0000","last_became_active":"2024-09-20T22:41:42.184759+0000","last_became_peered":"2024-09-20T22:41:42.184759+0000","last_unstale":"2024-09-20T22:42:28.100537+0000","last_undegraded":"2024-09-20T22:42:28.100537+0000","last_fullsized":"2024-09-20T22:42:28.100537+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:01.601642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":31,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.478139+0000","last_change":"2024-09-20T22:41:44.832468+0000","last_active":"2024-09-20T22:42:28.478139+0000","last_peered":"2024-09-20T22:42:28.478139+0000","last_clean":"2024-09-20T22:42:28.478139+0000","last_became_active":"2024-09-20T22:41:44.832131+0000","last_became_peered":"2024-09-20T22:41:44.832131+0000","last_unstale":"2024-09-20T22:42:28.478139+0000","last_undegraded":"2024-09-20T22:42:28.478139+0000","last_fullsized":"2024-09-20T22:42:28.478139+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:08.595231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":43,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.099825+0000","last_change":"2024-09-20T22:41:38.150584+0000","last_active":"2024-09-20T22:42:28.099825+0000","last_peered":"2024-09-20T22:42:28.099825+0000","last_clean":"2024-09-20T22:42:28.099825+0000","last_became_active":"2024-09-20T22:41:38.150232+0000","last_became_peered":"2024-09-20T22:41:38.150232+0000","last_unstale":"2024-09-20T22:42:28.099825+0000","last_undegraded":"2024-09-20T22:42:28.099825+0000","last_fullsized":"2024-09-20T22:42:28.099825+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:02.237312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"74'12","reported_seq":62,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.476952+0000","last_change":"2024-09-20T22:41:40.166344+0000","last_active":"2024-09-20T22:42:28.476952+0000","last_peered":"2024-09-20T22:42:28.476952+0000","last_clean":"2024-09-20T22:42:28.476952+0000","last_became_active":"2024-09-20T22:41:40.166087+0000","last_became_peered":"2024-09-20T22:41:40.166087+0000","last_unstale":"2024-09-20T22:42:28.476952+0000","last_undegraded":"2024-09-20T22:42:28.476952+0000","last_fullsized":"2024-09-20T22:42:28.476952+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:01.373771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.099869+0000","last_change":"2024-09-20T22:41:42.183266+0000","last_active":"2024-09-20T22:42:28.099869+0000","last_peered":"2024-09-20T22:42:28.099869+0000","last_clean":"2024-09-20T22:42:28.099869+0000","last_became_active":"2024-09-20T22:41:42.183005+0000","last_became_peered":"2024-09-20T22:41:42.183005+0000","last_unstale":"2024-09-20T22:42:28.099869+0000","last_undegraded":"2024-09-20T22:42:28.099869+0000","last_fullsized":"2024-09-20T22:42:28.099869+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:01.169906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":32,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.386277+0000","last_change":"2024-09-20T22:41:44.200115+0000","last_active":"2024-09-20T22:42:28.386277+0000","last_peered":"2024-09-20T22:42:28.386277+0000","last_clean":"2024-09-20T22:42:28.386277+0000","last_became_active":"2024-09-20T22:41:44.199822+0000","last_became_peered":"2024-09-20T22:41:44.199822+0000","last_unstale":"2024-09-20T22:42:28.386277+0000","last_undegraded":"2024-09-20T22:42:28.386277+0000","last_fullsized":"2024-09-20T22:42:28.386277+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:01.277411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"67'1","reported_seq":44,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.099727+0000","last_change":"2024-09-20T22:41:38.155735+0000","last_active":"2024-09-20T22:42:28.099727+0000","last_peered":"2024-09-20T22:42:28.099727+0000","last_clean":"2024-09-20T22:42:28.099727+0000","last_became_active":"2024-09-20T22:41:38.155424+0000","last_became_peered":"2024-09-20T22:41:38.155424+0000","last_unstale":"2024-09-20T22:42:28.099727+0000","last_undegraded":"2024-09-20T22:42:28.099727+0000","last_fullsized":"2024-09-20T22:42:28.099727+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:52.584133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"74'10","reported_seq":54,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.101040+0000","last_change":"2024-09-20T22:41:40.252498+0000","last_active":"2024-09-20T22:42:28.101040+0000","last_peered":"2024-09-20T22:42:28.101040+0000","last_clean":"2024-09-20T22:42:28.101040+0000","last_became_active":"2024-09-20T22:41:40.252001+0000","last_became_peered":"2024-09-20T22:41:40.252001+0000","last_unstale":"2024-09-20T22:42:28.101040+0000","last_undegraded":"2024-09-20T22:42:28.101040+0000","last_fullsized":"2024-09-20T22:42:28.101040+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:01.417640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"74'8","reported_seq":60,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.477716+0000","last_change":"2024-09-20T22:41:42.186385+0000","last_active":"2024-09-20T22:42:28.477716+0000","last_peered":"2024-09-20T22:42:28.477716+0000","last_clean":"2024-09-20T22:42:28.477716+0000","last_became_active":"2024-09-20T22:41:42.186071+0000","last_became_peered":"2024-09-20T22:41:42.186071+0000","last_unstale":"2024-09-20T22:42:28.477716+0000","last_undegraded":"2024-09-20T22:42:28.477716+0000","last_fullsized":"2024-09-20T22:42:28.477716+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:56.929679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.f","version":"74'15","reported_seq":64,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.101184+0000","last_change":"2024-09-20T22:41:40.168188+0000","last_active":"2024-09-20T22:42:28.101184+0000","last_peered":"2024-09-20T22:42:28.101184+0000","last_clean":"2024-09-20T22:42:28.101184+0000","last_became_active":"2024-09-20T22:41:40.165963+0000","last_became_peered":"2024-09-20T22:41:40.165963+0000","last_unstale":"2024-09-20T22:42:28.101184+0000","last_undegraded":"2024-09-20T22:42:28.101184+0000","last_fullsized":"2024-09-20T22:42:28.101184+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:46.519265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":43,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.477315+0000","last_change":"2024-09-20T22:41:38.157043+0000","last_active":"2024-09-20T22:42:28.477315+0000","last_peered":"2024-09-20T22:42:28.477315+0000","last_clean":"2024-09-20T22:42:28.477315+0000","last_became_active":"2024-09-20T22:41:38.156749+0000","last_became_peered":"2024-09-20T22:41:38.156749+0000","last_unstale":"2024-09-20T22:42:28.477315+0000","last_undegraded":"2024-09-20T22:42:28.477315+0000","last_fullsized":"2024-09-20T22:42:28.477315+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:03.406143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"74'8","reported_seq":54,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.100151+0000","last_change":"2024-09-20T22:41:42.182555+0000","last_active":"2024-09-20T22:42:28.100151+0000","last_peered":"2024-09-20T22:42:28.100151+0000","last_clean":"2024-09-20T22:42:28.100151+0000","last_became_active":"2024-09-20T22:41:42.182224+0000","last_became_peered":"2024-09-20T22:41:42.182224+0000","last_unstale":"2024-09-20T22:42:28.100151+0000","last_undegraded":"2024-09-20T22:42:28.100151+0000","last_fullsized":"2024-09-20T22:42:28.100151+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:07.384166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":31,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.518949+0000","last_change":"2024-09-20T22:41:44.198655+0000","last_active":"2024-09-20T22:42:28.518949+0000","last_peered":"2024-09-20T22:42:28.518949+0000","last_clean":"2024-09-20T22:42:28.518949+0000","last_became_active":"2024-09-20T22:41:44.198347+0000","last_became_peered":"2024-09-20T22:41:44.198347+0000","last_unstale":"2024-09-20T22:42:28.518949+0000","last_undegraded":"2024-09-20T22:42:28.518949+0000","last_fullsized":"2024-09-20T22:42:28.518949+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:54.119646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.0","version":"74'18","reported_seq":71,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.476692+0000","last_change":"2024-09-20T22:41:40.252203+0000","last_active":"2024-09-20T22:42:28.476692+0000","last_peered":"2024-09-20T22:42:28.476692+0000","last_clean":"2024-09-20T22:42:28.476692+0000","last_became_active":"2024-09-20T22:41:40.251868+0000","last_became_peered":"2024-09-20T22:41:40.251868+0000","last_unstale":"2024-09-20T22:42:28.476692+0000","last_undegraded":"2024-09-20T22:42:28.476692+0000","last_fullsized":"2024-09-20T22:42:28.476692+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:57.966357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":43,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.478253+0000","last_change":"2024-09-20T22:41:38.161991+0000","last_active":"2024-09-20T22:42:28.478253+0000","last_peered":"2024-09-20T22:42:28.478253+0000","last_clean":"2024-09-20T22:42:28.478253+0000","last_became_active":"2024-09-20T22:41:38.161335+0000","last_became_peered":"2024-09-20T22:41:38.161335+0000","last_unstale":"2024-09-20T22:42:28.478253+0000","last_undegraded":"2024-09-20T22:42:28.478253+0000","last_fullsized":"2024-09-20T22:42:28.478253+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:48.134689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.099465+0000","last_change":"2024-09-20T22:41:42.190157+0000","last_active":"2024-09-20T22:42:28.099465+0000","last_peered":"2024-09-20T22:42:28.099465+0000","last_clean":"2024-09-20T22:42:28.099465+0000","last_became_active":"2024-09-20T22:41:42.189743+0000","last_became_peered":"2024-09-20T22:41:42.189743+0000","last_unstale":"2024-09-20T22:42:28.099465+0000","last_undegraded":"2024-09-20T22:42:28.099465+0000","last_fullsized":"2024-09-20T22:42:28.099465+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:42.602485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":31,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.099689+0000","last_change":"2024-09-20T22:41:44.193944+0000","last_active":"2024-09-20T22:42:28.099689+0000","last_peered":"2024-09-20T22:42:28.099689+0000","last_clean":"2024-09-20T22:42:28.099689+0000","last_became_active":"2024-09-20T22:41:44.193646+0000","last_became_peered":"2024-09-20T22:41:44.193646+0000","last_unstale":"2024-09-20T22:42:28.099689+0000","last_undegraded":"2024-09-20T22:42:28.099689+0000","last_fullsized":"2024-09-20T22:42:28.099689+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:50.182915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"74'14","reported_seq":60,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.099501+0000","last_change":"2024-09-20T22:41:40.164011+0000","last_active":"2024-09-20T22:42:28.099501+0000","last_peered":"2024-09-20T22:42:28.099501+0000","last_clean":"2024-09-20T22:42:28.099501+0000","last_became_active":"2024-09-20T22:41:40.163695+0000","last_became_peered":"2024-09-20T22:41:40.163695+0000","last_unstale":"2024-09-20T22:42:28.099501+0000","last_undegraded":"2024-09-20T22:42:28.099501+0000","last_fullsized":"2024-09-20T22:42:28.099501+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:04.888108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"67'1","reported_seq":44,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.101402+0000","last_change":"2024-09-20T22:41:38.150707+0000","last_active":"2024-09-20T22:42:28.101402+0000","last_peered":"2024-09-20T22:42:28.101402+0000","last_clean":"2024-09-20T22:42:28.101402+0000","last_became_active":"2024-09-20T22:41:38.150331+0000","last_became_peered":"2024-09-20T22:41:38.150331+0000","last_unstale":"2024-09-20T22:42:28.101402+0000","last_undegraded":"2024-09-20T22:42:28.101402+0000","last_fullsized":"2024-09-20T22:42:28.101402+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:53.797157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"74'8","reported_seq":54,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.477173+0000","last_change":"2024-09-20T22:41:42.179893+0000","last_active":"2024-09-20T22:42:28.477173+0000","last_peered":"2024-09-20T22:42:28.477173+0000","last_clean":"2024-09-20T22:42:28.477173+0000","last_became_active":"2024-09-20T22:41:42.179613+0000","last_became_peered":"2024-09-20T22:41:42.179613+0000","last_unstale":"2024-09-20T22:42:28.477173+0000","last_undegraded":"2024-09-20T22:42:28.477173+0000","last_fullsized":"2024-09-20T22:42:28.477173+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:03.642295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":31,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.386167+0000","last_change":"2024-09-20T22:41:44.832161+0000","last_active":"2024-09-20T22:42:28.386167+0000","last_peered":"2024-09-20T22:42:28.386167+0000","last_clean":"2024-09-20T22:42:28.386167+0000","last_became_active":"2024-09-20T22:41:44.831901+0000","last_became_peered":"2024-09-20T22:41:44.831901+0000","last_unstale":"2024-09-20T22:42:28.386167+0000","last_undegraded":"2024-09-20T22:42:28.386167+0000","last_fullsized":"2024-09-20T22:42:28.386167+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:10.151190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2","version":"74'10","reported_seq":54,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.100294+0000","last_change":"2024-09-20T22:41:40.167041+0000","last_active":"2024-09-20T22:42:28.100294+0000","last_peered":"2024-09-20T22:42:28.100294+0000","last_clean":"2024-09-20T22:42:28.100294+0000","last_became_active":"2024-09-20T22:41:40.165653+0000","last_became_peered":"2024-09-20T22:41:40.165653+0000","last_unstale":"2024-09-20T22:42:28.100294+0000","last_undegraded":"2024-09-20T22:42:28.100294+0000","last_fullsized":"2024-09-20T22:42:28.100294+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:47.255874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":43,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.519276+0000","last_change":"2024-09-20T22:41:38.155924+0000","last_active":"2024-09-20T22:42:28.519276+0000","last_peered":"2024-09-20T22:42:28.519276+0000","last_clean":"2024-09-20T22:42:28.519276+0000","last_became_active":"2024-09-20T22:41:38.155629+0000","last_became_peered":"2024-09-20T22:41:38.155629+0000","last_unstale":"2024-09-20T22:42:28.519276+0000","last_undegraded":"2024-09-20T22:42:28.519276+0000","last_fullsized":"2024-09-20T22:42:28.519276+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:39.438618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"74'8","reported_seq":54,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.100192+0000","last_change":"2024-09-20T22:41:42.185583+0000","last_active":"2024-09-20T22:42:28.100192+0000","last_peered":"2024-09-20T22:42:28.100192+0000","last_clean":"2024-09-20T22:42:28.100192+0000","last_became_active":"2024-09-20T22:41:42.185213+0000","last_became_peered":"2024-09-20T22:41:42.185213+0000","last_unstale":"2024-09-20T22:42:28.100192+0000","last_undegraded":"2024-09-20T22:42:28.100192+0000","last_fullsized":"2024-09-20T22:42:28.100192+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:48.459335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":31,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.099546+0000","last_change":"2024-09-20T22:41:44.197894+0000","last_active":"2024-09-20T22:42:28.099546+0000","last_peered":"2024-09-20T22:42:28.099546+0000","last_clean":"2024-09-20T22:42:28.099546+0000","last_became_active":"2024-09-20T22:41:44.197619+0000","last_became_peered":"2024-09-20T22:41:44.197619+0000","last_unstale":"2024-09-20T22:42:28.099546+0000","last_undegraded":"2024-09-20T22:42:28.099546+0000","last_fullsized":"2024-09-20T22:42:28.099546+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:00.802751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"74'19","reported_seq":75,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.100352+0000","last_change":"2024-09-20T22:41:40.252159+0000","last_active":"2024-09-20T22:42:28.100352+0000","last_peered":"2024-09-20T22:42:28.100352+0000","last_clean":"2024-09-20T22:42:28.100352+0000","last_became_active":"2024-09-20T22:41:40.251866+0000","last_became_peered":"2024-09-20T22:41:40.251866+0000","last_unstale":"2024-09-20T22:42:28.100352+0000","last_undegraded":"2024-09-20T22:42:28.100352+0000","last_fullsized":"2024-09-20T22:42:28.100352+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:43.089389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":43,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.101213+0000","last_change":"2024-09-20T22:41:38.161236+0000","last_active":"2024-09-20T22:42:28.101213+0000","last_peered":"2024-09-20T22:42:28.101213+0000","last_clean":"2024-09-20T22:42:28.101213+0000","last_became_active":"2024-09-20T22:41:38.160832+0000","last_became_peered":"2024-09-20T22:41:38.160832+0000","last_unstale":"2024-09-20T22:42:28.101213+0000","last_undegraded":"2024-09-20T22:42:28.101213+0000","last_fullsized":"2024-09-20T22:42:28.101213+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:50.975305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.477283+0000","last_change":"2024-09-20T22:41:42.185638+0000","last_active":"2024-09-20T22:42:28.477283+0000","last_peered":"2024-09-20T22:42:28.477283+0000","last_clean":"2024-09-20T22:42:28.477283+0000","last_became_active":"2024-09-20T22:41:42.184412+0000","last_became_peered":"2024-09-20T22:41:42.184412+0000","last_unstale":"2024-09-20T22:42:28.477283+0000","last_undegraded":"2024-09-20T22:42:28.477283+0000","last_fullsized":"2024-09-20T22:42:28.477283+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:57.435580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":31,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.099499+0000","last_change":"2024-09-20T22:41:44.830917+0000","last_active":"2024-09-20T22:42:28.099499+0000","last_peered":"2024-09-20T22:42:28.099499+0000","last_clean":"2024-09-20T22:42:28.099499+0000","last_became_active":"2024-09-20T22:41:44.830538+0000","last_became_peered":"2024-09-20T22:41:44.830538+0000","last_unstale":"2024-09-20T22:42:28.099499+0000","last_undegraded":"2024-09-20T22:42:28.099499+0000","last_fullsized":"2024-09-20T22:42:28.099499+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:54.961702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"74'28","reported_seq":98,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.101115+0000","last_change":"2024-09-20T22:41:40.167313+0000","last_active":"2024-09-20T22:42:28.101115+0000","last_peered":"2024-09-20T22:42:28.101115+0000","last_clean":"2024-09-20T22:42:28.101115+0000","last_became_active":"2024-09-20T22:41:40.166958+0000","last_became_peered":"2024-09-20T22:41:40.166958+0000","last_unstale":"2024-09-20T22:42:28.101115+0000","last_undegraded":"2024-09-20T22:42:28.101115+0000","last_fullsized":"2024-09-20T22:42:28.101115+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:42.825818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":43,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.100061+0000","last_change":"2024-09-20T22:41:38.152921+0000","last_active":"2024-09-20T22:42:28.100061+0000","last_peered":"2024-09-20T22:42:28.100061+0000","last_clean":"2024-09-20T22:42:28.100061+0000","last_became_active":"2024-09-20T22:41:38.152525+0000","last_became_peered":"2024-09-20T22:41:38.152525+0000","last_unstale":"2024-09-20T22:42:28.100061+0000","last_undegraded":"2024-09-20T22:42:28.100061+0000","last_fullsized":"2024-09-20T22:42:28.100061+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:37.641089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"69'2","reported_seq":53,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.519001+0000","last_change":"2024-09-20T22:41:40.154106+0000","last_active":"2024-09-20T22:42:28.519001+0000","last_peered":"2024-09-20T22:42:28.519001+0000","last_clean":"2024-09-20T22:42:28.519001+0000","last_became_active":"2024-09-20T22:41:37.148479+0000","last_became_peered":"2024-09-20T22:41:37.148479+0000","last_unstale":"2024-09-20T22:42:28.519001+0000","last_undegraded":"2024-09-20T22:42:28.519001+0000","last_fullsized":"2024-09-20T22:42:28.519001+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:36.130684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:36.130684+0000","last_clean_scrub_stamp":"2024-09-20T22:41:36.130684+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:48.204536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087280599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.100077+0000","last_change":"2024-09-20T22:41:42.179979+0000","last_active":"2024-09-20T22:42:28.100077+0000","last_peered":"2024-09-20T22:42:28.100077+0000","last_clean":"2024-09-20T22:42:28.100077+0000","last_became_active":"2024-09-20T22:41:42.179742+0000","last_became_peered":"2024-09-20T22:41:42.179742+0000","last_unstale":"2024-09-20T22:42:28.100077+0000","last_undegraded":"2024-09-20T22:42:28.100077+0000","last_fullsized":"2024-09-20T22:42:28.100077+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:03.273567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":31,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.478033+0000","last_change":"2024-09-20T22:41:44.192602+0000","last_active":"2024-09-20T22:42:28.478033+0000","last_peered":"2024-09-20T22:42:28.478033+0000","last_clean":"2024-09-20T22:42:28.478033+0000","last_became_active":"2024-09-20T22:41:44.192128+0000","last_became_peered":"2024-09-20T22:41:44.192128+0000","last_unstale":"2024-09-20T22:42:28.478033+0000","last_undegraded":"2024-09-20T22:42:28.478033+0000","last_fullsized":"2024-09-20T22:42:28.478033+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:46.416059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"74'13","reported_seq":66,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.101279+0000","last_change":"2024-09-20T22:41:40.169777+0000","last_active":"2024-09-20T22:42:28.101279+0000","last_peered":"2024-09-20T22:42:28.101279+0000","last_clean":"2024-09-20T22:42:28.101279+0000","last_became_active":"2024-09-20T22:41:40.169473+0000","last_became_peered":"2024-09-20T22:41:40.169473+0000","last_unstale":"2024-09-20T22:42:28.101279+0000","last_undegraded":"2024-09-20T22:42:28.101279+0000","last_fullsized":"2024-09-20T22:42:28.101279+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:48.626103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":43,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.101859+0000","last_change":"2024-09-20T22:41:38.157113+0000","last_active":"2024-09-20T22:42:28.101859+0000","last_peered":"2024-09-20T22:42:28.101859+0000","last_clean":"2024-09-20T22:42:28.101859+0000","last_became_active":"2024-09-20T22:41:38.156837+0000","last_became_peered":"2024-09-20T22:41:38.156837+0000","last_unstale":"2024-09-20T22:42:28.101859+0000","last_undegraded":"2024-09-20T22:42:28.101859+0000","last_fullsized":"2024-09-20T22:42:28.101859+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:46.990363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"67'1","reported_seq":52,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.266395+0000","last_change":"2024-09-20T22:41:40.162396+0000","last_active":"2024-09-20T22:42:28.266395+0000","last_peered":"2024-09-20T22:42:28.266395+0000","last_clean":"2024-09-20T22:42:28.266395+0000","last_became_active":"2024-09-20T22:41:37.148165+0000","last_became_peered":"2024-09-20T22:41:37.148165+0000","last_unstale":"2024-09-20T22:42:28.266395+0000","last_undegraded":"2024-09-20T22:42:28.266395+0000","last_fullsized":"2024-09-20T22:42:28.266395+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:36.130684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:36.130684+0000","last_clean_scrub_stamp":"2024-09-20T22:41:36.130684+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:54.327893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001301459,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.267442+0000","last_change":"2024-09-20T22:41:42.185412+0000","last_active":"2024-09-20T22:42:28.267442+0000","last_peered":"2024-09-20T22:42:28.267442+0000","last_clean":"2024-09-20T22:42:28.267442+0000","last_became_active":"2024-09-20T22:41:42.185112+0000","last_became_peered":"2024-09-20T22:41:42.185112+0000","last_unstale":"2024-09-20T22:42:28.267442+0000","last_undegraded":"2024-09-20T22:42:28.267442+0000","last_fullsized":"2024-09-20T22:42:28.267442+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:05.899621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":31,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.386386+0000","last_change":"2024-09-20T22:41:44.830804+0000","last_active":"2024-09-20T22:42:28.386386+0000","last_peered":"2024-09-20T22:42:28.386386+0000","last_clean":"2024-09-20T22:42:28.386386+0000","last_became_active":"2024-09-20T22:41:44.830515+0000","last_became_peered":"2024-09-20T22:41:44.830515+0000","last_unstale":"2024-09-20T22:42:28.386386+0000","last_undegraded":"2024-09-20T22:42:28.386386+0000","last_fullsized":"2024-09-20T22:42:28.386386+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:04.958838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"74'12","reported_seq":57,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.101150+0000","last_change":"2024-09-20T22:41:40.167483+0000","last_active":"2024-09-20T22:42:28.101150+0000","last_peered":"2024-09-20T22:42:28.101150+0000","last_clean":"2024-09-20T22:42:28.101150+0000","last_became_active":"2024-09-20T22:41:40.167169+0000","last_became_peered":"2024-09-20T22:41:40.167169+0000","last_unstale":"2024-09-20T22:42:28.101150+0000","last_undegraded":"2024-09-20T22:42:28.101150+0000","last_fullsized":"2024-09-20T22:42:28.101150+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:48.803800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":43,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.099306+0000","last_change":"2024-09-20T22:41:38.153448+0000","last_active":"2024-09-20T22:42:28.099306+0000","last_peered":"2024-09-20T22:42:28.099306+0000","last_clean":"2024-09-20T22:42:28.099306+0000","last_became_active":"2024-09-20T22:41:38.153058+0000","last_became_peered":"2024-09-20T22:41:38.153058+0000","last_unstale":"2024-09-20T22:42:28.099306+0000","last_undegraded":"2024-09-20T22:42:28.099306+0000","last_fullsized":"2024-09-20T22:42:28.099306+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:04.425050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"75'5","reported_seq":99,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.386040+0000","last_change":"2024-09-20T22:41:40.251762+0000","last_active":"2024-09-20T22:42:28.386040+0000","last_peered":"2024-09-20T22:42:28.386040+0000","last_clean":"2024-09-20T22:42:28.386040+0000","last_became_active":"2024-09-20T22:41:37.148383+0000","last_became_peered":"2024-09-20T22:41:37.148383+0000","last_unstale":"2024-09-20T22:42:28.386040+0000","last_undegraded":"2024-09-20T22:42:28.386040+0000","last_fullsized":"2024-09-20T22:42:28.386040+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:36.130684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:36.130684+0000","last_clean_scrub_stamp":"2024-09-20T22:41:36.130684+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:04.208717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00078631700000000003,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.519377+0000","last_change":"2024-09-20T22:41:42.187618+0000","last_active":"2024-09-20T22:42:28.519377+0000","last_peered":"2024-09-20T22:42:28.519377+0000","last_clean":"2024-09-20T22:42:28.519377+0000","last_became_active":"2024-09-20T22:41:42.187346+0000","last_became_peered":"2024-09-20T22:41:42.187346+0000","last_unstale":"2024-09-20T22:42:28.519377+0000","last_undegraded":"2024-09-20T22:42:28.519377+0000","last_fullsized":"2024-09-20T22:42:28.519377+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:58.427580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":31,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.100779+0000","last_change":"2024-09-20T22:41:44.199301+0000","last_active":"2024-09-20T22:42:28.100779+0000","last_peered":"2024-09-20T22:42:28.100779+0000","last_clean":"2024-09-20T22:42:28.100779+0000","last_became_active":"2024-09-20T22:41:44.199046+0000","last_became_peered":"2024-09-20T22:41:44.199046+0000","last_unstale":"2024-09-20T22:42:28.100779+0000","last_undegraded":"2024-09-20T22:42:28.100779+0000","last_fullsized":"2024-09-20T22:42:28.100779+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:43.190792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"74'16","reported_seq":72,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.476721+0000","last_change":"2024-09-20T22:41:40.253524+0000","last_active":"2024-09-20T22:42:28.476721+0000","last_peered":"2024-09-20T22:42:28.476721+0000","last_clean":"2024-09-20T22:42:28.476721+0000","last_became_active":"2024-09-20T22:41:40.252933+0000","last_became_peered":"2024-09-20T22:41:40.252933+0000","last_unstale":"2024-09-20T22:42:28.476721+0000","last_undegraded":"2024-09-20T22:42:28.476721+0000","last_fullsized":"2024-09-20T22:42:28.476721+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:54.546845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"67'1","reported_seq":49,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.477599+0000","last_change":"2024-09-20T22:41:38.159706+0000","last_active":"2024-09-20T22:42:28.477599+0000","last_peered":"2024-09-20T22:42:28.477599+0000","last_clean":"2024-09-20T22:42:28.477599+0000","last_became_active":"2024-09-20T22:41:38.158917+0000","last_became_peered":"2024-09-20T22:41:38.158917+0000","last_unstale":"2024-09-20T22:42:28.477599+0000","last_undegraded":"2024-09-20T22:42:28.477599+0000","last_fullsized":"2024-09-20T22:42:28.477599+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:03.995423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"77'128","reported_seq":300,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.386165+0000","last_change":"2024-09-20T22:41:43.408723+0000","last_active":"2024-09-20T22:42:28.386165+0000","last_peered":"2024-09-20T22:42:28.386165+0000","last_clean":"2024-09-20T22:42:28.386165+0000","last_became_active":"2024-09-20T22:41:31.091135+0000","last_became_peered":"2024-09-20T22:41:31.091135+0000","last_unstale":"2024-09-20T22:42:28.386165+0000","last_undegraded":"2024-09-20T22:42:28.386165+0000","last_fullsized":"2024-09-20T22:42:28.386165+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'117","last_scrub_stamp":"2024-09-20T22:41:43.408644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.408644+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:48.125134+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.386384+0000","last_change":"2024-09-20T22:41:42.185135+0000","last_active":"2024-09-20T22:42:28.386384+0000","last_peered":"2024-09-20T22:42:28.386384+0000","last_clean":"2024-09-20T22:42:28.386384+0000","last_became_active":"2024-09-20T22:41:42.184387+0000","last_became_peered":"2024-09-20T22:41:42.184387+0000","last_unstale":"2024-09-20T22:42:28.386384+0000","last_undegraded":"2024-09-20T22:42:28.386384+0000","last_fullsized":"2024-09-20T22:42:28.386384+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:00.112877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":31,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.519320+0000","last_change":"2024-09-20T22:41:44.191113+0000","last_active":"2024-09-20T22:42:28.519320+0000","last_peered":"2024-09-20T22:42:28.519320+0000","last_clean":"2024-09-20T22:42:28.519320+0000","last_became_active":"2024-09-20T22:41:44.190658+0000","last_became_peered":"2024-09-20T22:41:44.190658+0000","last_unstale":"2024-09-20T22:42:28.519320+0000","last_undegraded":"2024-09-20T22:42:28.519320+0000","last_fullsized":"2024-09-20T22:42:28.519320+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:44.921190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"74'11","reported_seq":58,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.100206+0000","last_change":"2024-09-20T22:41:40.169542+0000","last_active":"2024-09-20T22:42:28.100206+0000","last_peered":"2024-09-20T22:42:28.100206+0000","last_clean":"2024-09-20T22:42:28.100206+0000","last_became_active":"2024-09-20T22:41:40.168942+0000","last_became_peered":"2024-09-20T22:41:40.168942+0000","last_unstale":"2024-09-20T22:42:28.100206+0000","last_undegraded":"2024-09-20T22:42:28.100206+0000","last_fullsized":"2024-09-20T22:42:28.100206+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:58.224721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":43,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.099711+0000","last_change":"2024-09-20T22:41:38.161801+0000","last_active":"2024-09-20T22:42:28.099711+0000","last_peered":"2024-09-20T22:42:28.099711+0000","last_clean":"2024-09-20T22:42:28.099711+0000","last_became_active":"2024-09-20T22:41:38.161500+0000","last_became_peered":"2024-09-20T22:41:38.161500+0000","last_unstale":"2024-09-20T22:42:28.099711+0000","last_undegraded":"2024-09-20T22:42:28.099711+0000","last_fullsized":"2024-09-20T22:42:28.099711+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:48.636045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.519276+0000","last_change":"2024-09-20T22:41:42.183601+0000","last_active":"2024-09-20T22:42:28.519276+0000","last_peered":"2024-09-20T22:42:28.519276+0000","last_clean":"2024-09-20T22:42:28.519276+0000","last_became_active":"2024-09-20T22:41:42.183302+0000","last_became_peered":"2024-09-20T22:41:42.183302+0000","last_unstale":"2024-09-20T22:42:28.519276+0000","last_undegraded":"2024-09-20T22:42:28.519276+0000","last_fullsized":"2024-09-20T22:42:28.519276+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:10.335337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":31,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.477770+0000","last_change":"2024-09-20T22:41:44.832295+0000","last_active":"2024-09-20T22:42:28.477770+0000","last_peered":"2024-09-20T22:42:28.477770+0000","last_clean":"2024-09-20T22:42:28.477770+0000","last_became_active":"2024-09-20T22:41:44.832009+0000","last_became_peered":"2024-09-20T22:41:44.832009+0000","last_unstale":"2024-09-20T22:42:28.477770+0000","last_undegraded":"2024-09-20T22:42:28.477770+0000","last_fullsized":"2024-09-20T22:42:28.477770+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:56.047489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"74'17","reported_seq":67,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.100327+0000","last_change":"2024-09-20T22:41:40.169498+0000","last_active":"2024-09-20T22:42:28.100327+0000","last_peered":"2024-09-20T22:42:28.100327+0000","last_clean":"2024-09-20T22:42:28.100327+0000","last_became_active":"2024-09-20T22:41:40.169192+0000","last_became_peered":"2024-09-20T22:41:40.169192+0000","last_unstale":"2024-09-20T22:42:28.100327+0000","last_undegraded":"2024-09-20T22:42:28.100327+0000","last_fullsized":"2024-09-20T22:42:28.100327+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:40.654708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":43,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.477418+0000","last_change":"2024-09-20T22:41:38.154220+0000","last_active":"2024-09-20T22:42:28.477418+0000","last_peered":"2024-09-20T22:42:28.477418+0000","last_clean":"2024-09-20T22:42:28.477418+0000","last_became_active":"2024-09-20T22:41:38.153926+0000","last_became_peered":"2024-09-20T22:41:38.153926+0000","last_unstale":"2024-09-20T22:42:28.477418+0000","last_undegraded":"2024-09-20T22:42:28.477418+0000","last_fullsized":"2024-09-20T22:42:28.477418+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:54.196177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.100876+0000","last_change":"2024-09-20T22:41:42.183482+0000","last_active":"2024-09-20T22:42:28.100876+0000","last_peered":"2024-09-20T22:42:28.100876+0000","last_clean":"2024-09-20T22:42:28.100876+0000","last_became_active":"2024-09-20T22:41:42.183225+0000","last_became_peered":"2024-09-20T22:41:42.183225+0000","last_unstale":"2024-09-20T22:42:28.100876+0000","last_undegraded":"2024-09-20T22:42:28.100876+0000","last_fullsized":"2024-09-20T22:42:28.100876+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:10.228360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":31,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.267176+0000","last_change":"2024-09-20T22:41:44.197855+0000","last_active":"2024-09-20T22:42:28.267176+0000","last_peered":"2024-09-20T22:42:28.267176+0000","last_clean":"2024-09-20T22:42:28.267176+0000","last_became_active":"2024-09-20T22:41:44.197561+0000","last_became_peered":"2024-09-20T22:41:44.197561+0000","last_unstale":"2024-09-20T22:42:28.267176+0000","last_undegraded":"2024-09-20T22:42:28.267176+0000","last_fullsized":"2024-09-20T22:42:28.267176+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:45.431229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"74'10","reported_seq":54,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.100016+0000","last_change":"2024-09-20T22:41:40.162973+0000","last_active":"2024-09-20T22:42:28.100016+0000","last_peered":"2024-09-20T22:42:28.100016+0000","last_clean":"2024-09-20T22:42:28.100016+0000","last_became_active":"2024-09-20T22:41:40.162494+0000","last_became_peered":"2024-09-20T22:41:40.162494+0000","last_unstale":"2024-09-20T22:42:28.100016+0000","last_undegraded":"2024-09-20T22:42:28.100016+0000","last_fullsized":"2024-09-20T22:42:28.100016+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:49.601652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":43,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.477895+0000","last_change":"2024-09-20T22:41:38.154861+0000","last_active":"2024-09-20T22:42:28.477895+0000","last_peered":"2024-09-20T22:42:28.477895+0000","last_clean":"2024-09-20T22:42:28.477895+0000","last_became_active":"2024-09-20T22:41:38.154636+0000","last_became_peered":"2024-09-20T22:41:38.154636+0000","last_unstale":"2024-09-20T22:42:28.477895+0000","last_undegraded":"2024-09-20T22:42:28.477895+0000","last_fullsized":"2024-09-20T22:42:28.477895+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:45.035202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"74'8","reported_seq":54,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.266877+0000","last_change":"2024-09-20T22:41:42.186528+0000","last_active":"2024-09-20T22:42:28.266877+0000","last_peered":"2024-09-20T22:42:28.266877+0000","last_clean":"2024-09-20T22:42:28.266877+0000","last_became_active":"2024-09-20T22:41:42.185863+0000","last_became_peered":"2024-09-20T22:41:42.185863+0000","last_unstale":"2024-09-20T22:42:28.266877+0000","last_undegraded":"2024-09-20T22:42:28.266877+0000","last_fullsized":"2024-09-20T22:42:28.266877+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:00.762928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":31,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.099871+0000","last_change":"2024-09-20T22:41:44.195767+0000","last_active":"2024-09-20T22:42:28.099871+0000","last_peered":"2024-09-20T22:42:28.099871+0000","last_clean":"2024-09-20T22:42:28.099871+0000","last_became_active":"2024-09-20T22:41:44.195160+0000","last_became_peered":"2024-09-20T22:41:44.195160+0000","last_unstale":"2024-09-20T22:42:28.099871+0000","last_undegraded":"2024-09-20T22:42:28.099871+0000","last_fullsized":"2024-09-20T22:42:28.099871+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:07.188986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"74'9","reported_seq":55,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.100241+0000","last_change":"2024-09-20T22:41:40.163774+0000","last_active":"2024-09-20T22:42:28.100241+0000","last_peered":"2024-09-20T22:42:28.100241+0000","last_clean":"2024-09-20T22:42:28.100241+0000","last_became_active":"2024-09-20T22:41:40.163495+0000","last_became_peered":"2024-09-20T22:41:40.163495+0000","last_unstale":"2024-09-20T22:42:28.100241+0000","last_undegraded":"2024-09-20T22:42:28.100241+0000","last_fullsized":"2024-09-20T22:42:28.100241+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:41.614897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":43,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.519191+0000","last_change":"2024-09-20T22:41:38.156470+0000","last_active":"2024-09-20T22:42:28.519191+0000","last_peered":"2024-09-20T22:42:28.519191+0000","last_clean":"2024-09-20T22:42:28.519191+0000","last_became_active":"2024-09-20T22:41:38.155953+0000","last_became_peered":"2024-09-20T22:41:38.155953+0000","last_unstale":"2024-09-20T22:42:28.519191+0000","last_undegraded":"2024-09-20T22:42:28.519191+0000","last_fullsized":"2024-09-20T22:42:28.519191+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:54.057798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.267149+0000","last_change":"2024-09-20T22:41:42.183269+0000","last_active":"2024-09-20T22:42:28.267149+0000","last_peered":"2024-09-20T22:42:28.267149+0000","last_clean":"2024-09-20T22:42:28.267149+0000","last_became_active":"2024-09-20T22:41:42.182999+0000","last_became_peered":"2024-09-20T22:41:42.182999+0000","last_unstale":"2024-09-20T22:42:28.267149+0000","last_undegraded":"2024-09-20T22:42:28.267149+0000","last_fullsized":"2024-09-20T22:42:28.267149+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:57.955307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":31,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.099253+0000","last_change":"2024-09-20T22:41:44.196969+0000","last_active":"2024-09-20T22:42:28.099253+0000","last_peered":"2024-09-20T22:42:28.099253+0000","last_clean":"2024-09-20T22:42:28.099253+0000","last_became_active":"2024-09-20T22:41:44.196633+0000","last_became_peered":"2024-09-20T22:41:44.196633+0000","last_unstale":"2024-09-20T22:42:28.099253+0000","last_undegraded":"2024-09-20T22:42:28.099253+0000","last_fullsized":"2024-09-20T22:42:28.099253+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:03.513323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"74'19","reported_seq":70,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.476645+0000","last_change":"2024-09-20T22:41:40.252962+0000","last_active":"2024-09-20T22:42:28.476645+0000","last_peered":"2024-09-20T22:42:28.476645+0000","last_clean":"2024-09-20T22:42:28.476645+0000","last_became_active":"2024-09-20T22:41:40.252011+0000","last_became_peered":"2024-09-20T22:41:40.252011+0000","last_unstale":"2024-09-20T22:42:28.476645+0000","last_undegraded":"2024-09-20T22:42:28.476645+0000","last_fullsized":"2024-09-20T22:42:28.476645+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:51.633432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":43,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.386630+0000","last_change":"2024-09-20T22:41:38.161800+0000","last_active":"2024-09-20T22:42:28.386630+0000","last_peered":"2024-09-20T22:42:28.386630+0000","last_clean":"2024-09-20T22:42:28.386630+0000","last_became_active":"2024-09-20T22:41:38.161524+0000","last_became_peered":"2024-09-20T22:41:38.161524+0000","last_unstale":"2024-09-20T22:42:28.386630+0000","last_undegraded":"2024-09-20T22:42:28.386630+0000","last_fullsized":"2024-09-20T22:42:28.386630+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:07.121209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.266881+0000","last_change":"2024-09-20T22:41:42.184605+0000","last_active":"2024-09-20T22:42:28.266881+0000","last_peered":"2024-09-20T22:42:28.266881+0000","last_clean":"2024-09-20T22:42:28.266881+0000","last_became_active":"2024-09-20T22:41:42.184342+0000","last_became_peered":"2024-09-20T22:41:42.184342+0000","last_unstale":"2024-09-20T22:42:28.266881+0000","last_undegraded":"2024-09-20T22:42:28.266881+0000","last_fullsized":"2024-09-20T22:42:28.266881+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:03.475959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":31,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.386113+0000","last_change":"2024-09-20T22:41:44.199874+0000","last_active":"2024-09-20T22:42:28.386113+0000","last_peered":"2024-09-20T22:42:28.386113+0000","last_clean":"2024-09-20T22:42:28.386113+0000","last_became_active":"2024-09-20T22:41:44.199395+0000","last_became_peered":"2024-09-20T22:41:44.199395+0000","last_unstale":"2024-09-20T22:42:28.386113+0000","last_undegraded":"2024-09-20T22:42:28.386113+0000","last_fullsized":"2024-09-20T22:42:28.386113+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:44.605488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"74'12","reported_seq":62,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.099597+0000","last_change":"2024-09-20T22:41:40.170746+0000","last_active":"2024-09-20T22:42:28.099597+0000","last_peered":"2024-09-20T22:42:28.099597+0000","last_clean":"2024-09-20T22:42:28.099597+0000","last_became_active":"2024-09-20T22:41:40.170133+0000","last_became_peered":"2024-09-20T22:41:40.170133+0000","last_unstale":"2024-09-20T22:42:28.099597+0000","last_undegraded":"2024-09-20T22:42:28.099597+0000","last_fullsized":"2024-09-20T22:42:28.099597+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:39.588868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":43,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.386606+0000","last_change":"2024-09-20T22:41:38.163612+0000","last_active":"2024-09-20T22:42:28.386606+0000","last_peered":"2024-09-20T22:42:28.386606+0000","last_clean":"2024-09-20T22:42:28.386606+0000","last_became_active":"2024-09-20T22:41:38.163292+0000","last_became_peered":"2024-09-20T22:41:38.163292+0000","last_unstale":"2024-09-20T22:42:28.386606+0000","last_undegraded":"2024-09-20T22:42:28.386606+0000","last_fullsized":"2024-09-20T22:42:28.386606+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:45.666357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.266571+0000","last_change":"2024-09-20T22:41:42.182102+0000","last_active":"2024-09-20T22:42:28.266571+0000","last_peered":"2024-09-20T22:42:28.266571+0000","last_clean":"2024-09-20T22:42:28.266571+0000","last_became_active":"2024-09-20T22:41:42.181811+0000","last_became_peered":"2024-09-20T22:41:42.181811+0000","last_unstale":"2024-09-20T22:42:28.266571+0000","last_undegraded":"2024-09-20T22:42:28.266571+0000","last_fullsized":"2024-09-20T22:42:28.266571+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:52.248246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":31,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.477729+0000","last_change":"2024-09-20T22:41:44.193432+0000","last_active":"2024-09-20T22:42:28.477729+0000","last_peered":"2024-09-20T22:42:28.477729+0000","last_clean":"2024-09-20T22:42:28.477729+0000","last_became_active":"2024-09-20T22:41:44.192609+0000","last_became_peered":"2024-09-20T22:41:44.192609+0000","last_unstale":"2024-09-20T22:42:28.477729+0000","last_undegraded":"2024-09-20T22:42:28.477729+0000","last_fullsized":"2024-09-20T22:42:28.477729+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:50.351142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"74'15","reported_seq":64,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.518696+0000","last_change":"2024-09-20T22:41:40.252427+0000","last_active":"2024-09-20T22:42:28.518696+0000","last_peered":"2024-09-20T22:42:28.518696+0000","last_clean":"2024-09-20T22:42:28.518696+0000","last_became_active":"2024-09-20T22:41:40.252113+0000","last_became_peered":"2024-09-20T22:41:40.252113+0000","last_unstale":"2024-09-20T22:42:28.518696+0000","last_undegraded":"2024-09-20T22:42:28.518696+0000","last_fullsized":"2024-09-20T22:42:28.518696+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:06.207870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'2","reported_seq":55,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.386646+0000","last_change":"2024-09-20T22:41:38.163593+0000","last_active":"2024-09-20T22:42:28.386646+0000","last_peered":"2024-09-20T22:42:28.386646+0000","last_clean":"2024-09-20T22:42:28.386646+0000","last_became_active":"2024-09-20T22:41:38.163219+0000","last_became_peered":"2024-09-20T22:41:38.163219+0000","last_unstale":"2024-09-20T22:42:28.386646+0000","last_undegraded":"2024-09-20T22:42:28.386646+0000","last_fullsized":"2024-09-20T22:42:28.386646+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:02.881623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"74'8","reported_seq":54,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.385953+0000","last_change":"2024-09-20T22:41:42.188223+0000","last_active":"2024-09-20T22:42:28.385953+0000","last_peered":"2024-09-20T22:42:28.385953+0000","last_clean":"2024-09-20T22:42:28.385953+0000","last_became_active":"2024-09-20T22:41:42.186635+0000","last_became_peered":"2024-09-20T22:41:42.186635+0000","last_unstale":"2024-09-20T22:42:28.385953+0000","last_undegraded":"2024-09-20T22:42:28.385953+0000","last_fullsized":"2024-09-20T22:42:28.385953+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:52.120303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":31,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.518852+0000","last_change":"2024-09-20T22:41:44.831435+0000","last_active":"2024-09-20T22:42:28.518852+0000","last_peered":"2024-09-20T22:42:28.518852+0000","last_clean":"2024-09-20T22:42:28.518852+0000","last_became_active":"2024-09-20T22:41:44.830556+0000","last_became_peered":"2024-09-20T22:41:44.830556+0000","last_unstale":"2024-09-20T22:42:28.518852+0000","last_undegraded":"2024-09-20T22:42:28.518852+0000","last_fullsized":"2024-09-20T22:42:28.518852+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:08.027826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":43,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.476997+0000","last_change":"2024-09-20T22:41:38.159437+0000","last_active":"2024-09-20T22:42:28.476997+0000","last_peered":"2024-09-20T22:42:28.476997+0000","last_clean":"2024-09-20T22:42:28.476997+0000","last_became_active":"2024-09-20T22:41:38.159139+0000","last_became_peered":"2024-09-20T22:41:38.159139+0000","last_unstale":"2024-09-20T22:42:28.476997+0000","last_undegraded":"2024-09-20T22:42:28.476997+0000","last_fullsized":"2024-09-20T22:42:28.476997+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:47.658845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"74'6","reported_seq":48,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.476969+0000","last_change":"2024-09-20T22:41:40.252310+0000","last_active":"2024-09-20T22:42:28.476969+0000","last_peered":"2024-09-20T22:42:28.476969+0000","last_clean":"2024-09-20T22:42:28.476969+0000","last_became_active":"2024-09-20T22:41:40.251867+0000","last_became_peered":"2024-09-20T22:41:40.251867+0000","last_unstale":"2024-09-20T22:42:28.476969+0000","last_undegraded":"2024-09-20T22:42:28.476969+0000","last_fullsized":"2024-09-20T22:42:28.476969+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:40.015664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.519471+0000","last_change":"2024-09-20T22:41:42.185218+0000","last_active":"2024-09-20T22:42:28.519471+0000","last_peered":"2024-09-20T22:42:28.519471+0000","last_clean":"2024-09-20T22:42:28.519471+0000","last_became_active":"2024-09-20T22:41:42.184930+0000","last_became_peered":"2024-09-20T22:41:42.184930+0000","last_unstale":"2024-09-20T22:42:28.519471+0000","last_undegraded":"2024-09-20T22:42:28.519471+0000","last_fullsized":"2024-09-20T22:42:28.519471+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:47.818365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"74'1","reported_seq":32,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.386326+0000","last_change":"2024-09-20T22:41:44.832085+0000","last_active":"2024-09-20T22:42:28.386326+0000","last_peered":"2024-09-20T22:42:28.386326+0000","last_clean":"2024-09-20T22:42:28.386326+0000","last_became_active":"2024-09-20T22:41:44.831805+0000","last_became_peered":"2024-09-20T22:41:44.831805+0000","last_unstale":"2024-09-20T22:42:28.386326+0000","last_undegraded":"2024-09-20T22:42:28.386326+0000","last_fullsized":"2024-09-20T22:42:28.386326+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:54.445212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"74'9","reported_seq":55,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.101282+0000","last_change":"2024-09-20T22:41:40.253633+0000","last_active":"2024-09-20T22:42:28.101282+0000","last_peered":"2024-09-20T22:42:28.101282+0000","last_clean":"2024-09-20T22:42:28.101282+0000","last_became_active":"2024-09-20T22:41:40.253280+0000","last_became_peered":"2024-09-20T22:41:40.253280+0000","last_unstale":"2024-09-20T22:42:28.101282+0000","last_undegraded":"2024-09-20T22:42:28.101282+0000","last_fullsized":"2024-09-20T22:42:28.101282+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:39.928882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":43,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.386175+0000","last_change":"2024-09-20T22:41:38.156773+0000","last_active":"2024-09-20T22:42:28.386175+0000","last_peered":"2024-09-20T22:42:28.386175+0000","last_clean":"2024-09-20T22:42:28.386175+0000","last_became_active":"2024-09-20T22:41:38.156497+0000","last_became_peered":"2024-09-20T22:41:38.156497+0000","last_unstale":"2024-09-20T22:42:28.386175+0000","last_undegraded":"2024-09-20T22:42:28.386175+0000","last_fullsized":"2024-09-20T22:42:28.386175+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:01.943269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.478147+0000","last_change":"2024-09-20T22:41:42.188166+0000","last_active":"2024-09-20T22:42:28.478147+0000","last_peered":"2024-09-20T22:42:28.478147+0000","last_clean":"2024-09-20T22:42:28.478147+0000","last_became_active":"2024-09-20T22:41:42.187706+0000","last_became_peered":"2024-09-20T22:41:42.187706+0000","last_unstale":"2024-09-20T22:42:28.478147+0000","last_undegraded":"2024-09-20T22:42:28.478147+0000","last_fullsized":"2024-09-20T22:42:28.478147+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:47.610076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":31,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.267029+0000","last_change":"2024-09-20T22:41:44.201642+0000","last_active":"2024-09-20T22:42:28.267029+0000","last_peered":"2024-09-20T22:42:28.267029+0000","last_clean":"2024-09-20T22:42:28.267029+0000","last_became_active":"2024-09-20T22:41:44.201433+0000","last_became_peered":"2024-09-20T22:41:44.201433+0000","last_unstale":"2024-09-20T22:42:28.267029+0000","last_undegraded":"2024-09-20T22:42:28.267029+0000","last_fullsized":"2024-09-20T22:42:28.267029+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:55.257671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"74'9","reported_seq":55,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.519182+0000","last_change":"2024-09-20T22:41:40.252840+0000","last_active":"2024-09-20T22:42:28.519182+0000","last_peered":"2024-09-20T22:42:28.519182+0000","last_clean":"2024-09-20T22:42:28.519182+0000","last_became_active":"2024-09-20T22:41:40.252486+0000","last_became_peered":"2024-09-20T22:41:40.252486+0000","last_unstale":"2024-09-20T22:42:28.519182+0000","last_undegraded":"2024-09-20T22:42:28.519182+0000","last_fullsized":"2024-09-20T22:42:28.519182+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:51.443985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":43,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.099861+0000","last_change":"2024-09-20T22:41:38.158479+0000","last_active":"2024-09-20T22:42:28.099861+0000","last_peered":"2024-09-20T22:42:28.099861+0000","last_clean":"2024-09-20T22:42:28.099861+0000","last_became_active":"2024-09-20T22:41:38.158146+0000","last_became_peered":"2024-09-20T22:41:38.158146+0000","last_unstale":"2024-09-20T22:42:28.099861+0000","last_undegraded":"2024-09-20T22:42:28.099861+0000","last_fullsized":"2024-09-20T22:42:28.099861+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:55.658232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"74'8","reported_seq":54,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.477829+0000","last_change":"2024-09-20T22:41:42.186630+0000","last_active":"2024-09-20T22:42:28.477829+0000","last_peered":"2024-09-20T22:42:28.477829+0000","last_clean":"2024-09-20T22:42:28.477829+0000","last_became_active":"2024-09-20T22:41:42.186352+0000","last_became_peered":"2024-09-20T22:41:42.186352+0000","last_unstale":"2024-09-20T22:42:28.477829+0000","last_undegraded":"2024-09-20T22:42:28.477829+0000","last_fullsized":"2024-09-20T22:42:28.477829+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:48.572027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":31,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.100029+0000","last_change":"2024-09-20T22:41:44.194510+0000","last_active":"2024-09-20T22:42:28.100029+0000","last_peered":"2024-09-20T22:42:28.100029+0000","last_clean":"2024-09-20T22:42:28.100029+0000","last_became_active":"2024-09-20T22:41:44.194053+0000","last_became_peered":"2024-09-20T22:41:44.194053+0000","last_unstale":"2024-09-20T22:42:28.100029+0000","last_undegraded":"2024-09-20T22:42:28.100029+0000","last_fullsized":"2024-09-20T22:42:28.100029+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:01.207219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"74'10","reported_seq":54,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.476803+0000","last_change":"2024-09-20T22:41:40.253880+0000","last_active":"2024-09-20T22:42:28.476803+0000","last_peered":"2024-09-20T22:42:28.476803+0000","last_clean":"2024-09-20T22:42:28.476803+0000","last_became_active":"2024-09-20T22:41:40.253614+0000","last_became_peered":"2024-09-20T22:41:40.253614+0000","last_unstale":"2024-09-20T22:42:28.476803+0000","last_undegraded":"2024-09-20T22:42:28.476803+0000","last_fullsized":"2024-09-20T22:42:28.476803+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:47.554109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":43,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.386327+0000","last_change":"2024-09-20T22:41:38.158008+0000","last_active":"2024-09-20T22:42:28.386327+0000","last_peered":"2024-09-20T22:42:28.386327+0000","last_clean":"2024-09-20T22:42:28.386327+0000","last_became_active":"2024-09-20T22:41:38.157647+0000","last_became_peered":"2024-09-20T22:41:38.157647+0000","last_unstale":"2024-09-20T22:42:28.386327+0000","last_undegraded":"2024-09-20T22:42:28.386327+0000","last_fullsized":"2024-09-20T22:42:28.386327+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:57.980474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"74'8","reported_seq":54,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.519065+0000","last_change":"2024-09-20T22:41:42.189757+0000","last_active":"2024-09-20T22:42:28.519065+0000","last_peered":"2024-09-20T22:42:28.519065+0000","last_clean":"2024-09-20T22:42:28.519065+0000","last_became_active":"2024-09-20T22:41:42.189451+0000","last_became_peered":"2024-09-20T22:41:42.189451+0000","last_unstale":"2024-09-20T22:42:28.519065+0000","last_undegraded":"2024-09-20T22:42:28.519065+0000","last_fullsized":"2024-09-20T22:42:28.519065+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:43.846079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":31,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.100733+0000","last_change":"2024-09-20T22:41:44.191549+0000","last_active":"2024-09-20T22:42:28.100733+0000","last_peered":"2024-09-20T22:42:28.100733+0000","last_clean":"2024-09-20T22:42:28.100733+0000","last_became_active":"2024-09-20T22:41:44.191206+0000","last_became_peered":"2024-09-20T22:41:44.191206+0000","last_unstale":"2024-09-20T22:42:28.100733+0000","last_undegraded":"2024-09-20T22:42:28.100733+0000","last_fullsized":"2024-09-20T22:42:28.100733+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:58.564765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"74'11","reported_seq":58,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.099734+0000","last_change":"2024-09-20T22:41:40.171449+0000","last_active":"2024-09-20T22:42:28.099734+0000","last_peered":"2024-09-20T22:42:28.099734+0000","last_clean":"2024-09-20T22:42:28.099734+0000","last_became_active":"2024-09-20T22:41:40.170035+0000","last_became_peered":"2024-09-20T22:41:40.170035+0000","last_unstale":"2024-09-20T22:42:28.099734+0000","last_undegraded":"2024-09-20T22:42:28.099734+0000","last_fullsized":"2024-09-20T22:42:28.099734+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:49.140455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":43,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.100170+0000","last_change":"2024-09-20T22:41:38.161385+0000","last_active":"2024-09-20T22:42:28.100170+0000","last_peered":"2024-09-20T22:42:28.100170+0000","last_clean":"2024-09-20T22:42:28.100170+0000","last_became_active":"2024-09-20T22:41:38.158142+0000","last_became_peered":"2024-09-20T22:41:38.158142+0000","last_unstale":"2024-09-20T22:42:28.100170+0000","last_undegraded":"2024-09-20T22:42:28.100170+0000","last_fullsized":"2024-09-20T22:42:28.100170+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:51.728688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.100124+0000","last_change":"2024-09-20T22:41:42.182119+0000","last_active":"2024-09-20T22:42:28.100124+0000","last_peered":"2024-09-20T22:42:28.100124+0000","last_clean":"2024-09-20T22:42:28.100124+0000","last_became_active":"2024-09-20T22:41:42.181736+0000","last_became_peered":"2024-09-20T22:41:42.181736+0000","last_unstale":"2024-09-20T22:42:28.100124+0000","last_undegraded":"2024-09-20T22:42:28.100124+0000","last_fullsized":"2024-09-20T22:42:28.100124+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:02.572377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":31,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.477331+0000","last_change":"2024-09-20T22:41:44.193289+0000","last_active":"2024-09-20T22:42:28.477331+0000","last_peered":"2024-09-20T22:42:28.477331+0000","last_clean":"2024-09-20T22:42:28.477331+0000","last_became_active":"2024-09-20T22:41:44.192960+0000","last_became_peered":"2024-09-20T22:41:44.192960+0000","last_unstale":"2024-09-20T22:42:28.477331+0000","last_undegraded":"2024-09-20T22:42:28.477331+0000","last_fullsized":"2024-09-20T22:42:28.477331+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:49.189046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"74'9","reported_seq":55,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.100394+0000","last_change":"2024-09-20T22:41:40.165116+0000","last_active":"2024-09-20T22:42:28.100394+0000","last_peered":"2024-09-20T22:42:28.100394+0000","last_clean":"2024-09-20T22:42:28.100394+0000","last_became_active":"2024-09-20T22:41:40.164796+0000","last_became_peered":"2024-09-20T22:41:40.164796+0000","last_unstale":"2024-09-20T22:42:28.100394+0000","last_undegraded":"2024-09-20T22:42:28.100394+0000","last_fullsized":"2024-09-20T22:42:28.100394+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:48.483009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":43,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.386534+0000","last_change":"2024-09-20T22:41:38.159941+0000","last_active":"2024-09-20T22:42:28.386534+0000","last_peered":"2024-09-20T22:42:28.386534+0000","last_clean":"2024-09-20T22:42:28.386534+0000","last_became_active":"2024-09-20T22:41:38.159657+0000","last_became_peered":"2024-09-20T22:41:38.159657+0000","last_unstale":"2024-09-20T22:42:28.386534+0000","last_undegraded":"2024-09-20T22:42:28.386534+0000","last_fullsized":"2024-09-20T22:42:28.386534+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:53.225190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.477809+0000","last_change":"2024-09-20T22:41:42.183254+0000","last_active":"2024-09-20T22:42:28.477809+0000","last_peered":"2024-09-20T22:42:28.477809+0000","last_clean":"2024-09-20T22:42:28.477809+0000","last_became_active":"2024-09-20T22:41:42.182966+0000","last_became_peered":"2024-09-20T22:41:42.182966+0000","last_unstale":"2024-09-20T22:42:28.477809+0000","last_undegraded":"2024-09-20T22:42:28.477809+0000","last_fullsized":"2024-09-20T22:42:28.477809+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:51.689726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":31,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.099148+0000","last_change":"2024-09-20T22:41:44.194845+0000","last_active":"2024-09-20T22:42:28.099148+0000","last_peered":"2024-09-20T22:42:28.099148+0000","last_clean":"2024-09-20T22:42:28.099148+0000","last_became_active":"2024-09-20T22:41:44.194551+0000","last_became_peered":"2024-09-20T22:41:44.194551+0000","last_unstale":"2024-09-20T22:42:28.099148+0000","last_undegraded":"2024-09-20T22:42:28.099148+0000","last_fullsized":"2024-09-20T22:42:28.099148+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:45.187444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"74'11","reported_seq":58,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.477453+0000","last_change":"2024-09-20T22:41:40.252149+0000","last_active":"2024-09-20T22:42:28.477453+0000","last_peered":"2024-09-20T22:42:28.477453+0000","last_clean":"2024-09-20T22:42:28.477453+0000","last_became_active":"2024-09-20T22:41:40.251829+0000","last_became_peered":"2024-09-20T22:41:40.251829+0000","last_unstale":"2024-09-20T22:42:28.477453+0000","last_undegraded":"2024-09-20T22:42:28.477453+0000","last_fullsized":"2024-09-20T22:42:28.477453+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:56.020491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":43,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.519509+0000","last_change":"2024-09-20T22:41:38.159064+0000","last_active":"2024-09-20T22:42:28.519509+0000","last_peered":"2024-09-20T22:42:28.519509+0000","last_clean":"2024-09-20T22:42:28.519509+0000","last_became_active":"2024-09-20T22:41:38.158701+0000","last_became_peered":"2024-09-20T22:41:38.158701+0000","last_unstale":"2024-09-20T22:42:28.519509+0000","last_undegraded":"2024-09-20T22:42:28.519509+0000","last_fullsized":"2024-09-20T22:42:28.519509+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:47.727493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.385845+0000","last_change":"2024-09-20T22:41:42.185991+0000","last_active":"2024-09-20T22:42:28.385845+0000","last_peered":"2024-09-20T22:42:28.385845+0000","last_clean":"2024-09-20T22:42:28.385845+0000","last_became_active":"2024-09-20T22:41:42.185296+0000","last_became_peered":"2024-09-20T22:41:42.185296+0000","last_unstale":"2024-09-20T22:42:28.385845+0000","last_undegraded":"2024-09-20T22:42:28.385845+0000","last_fullsized":"2024-09-20T22:42:28.385845+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:55.259880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":31,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.477860+0000","last_change":"2024-09-20T22:41:44.832293+0000","last_active":"2024-09-20T22:42:28.477860+0000","last_peered":"2024-09-20T22:42:28.477860+0000","last_clean":"2024-09-20T22:42:28.477860+0000","last_became_active":"2024-09-20T22:41:44.831967+0000","last_became_peered":"2024-09-20T22:41:44.831967+0000","last_unstale":"2024-09-20T22:42:28.477860+0000","last_undegraded":"2024-09-20T22:42:28.477860+0000","last_fullsized":"2024-09-20T22:42:28.477860+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:03.371689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"74'4","reported_seq":45,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.476697+0000","last_change":"2024-09-20T22:41:40.166409+0000","last_active":"2024-09-20T22:42:28.476697+0000","last_peered":"2024-09-20T22:42:28.476697+0000","last_clean":"2024-09-20T22:42:28.476697+0000","last_became_active":"2024-09-20T22:41:40.166123+0000","last_became_peered":"2024-09-20T22:41:40.166123+0000","last_unstale":"2024-09-20T22:42:28.476697+0000","last_undegraded":"2024-09-20T22:42:28.476697+0000","last_fullsized":"2024-09-20T22:42:28.476697+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:52.943341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":43,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.101609+0000","last_change":"2024-09-20T22:41:38.158861+0000","last_active":"2024-09-20T22:42:28.101609+0000","last_peered":"2024-09-20T22:42:28.101609+0000","last_clean":"2024-09-20T22:42:28.101609+0000","last_became_active":"2024-09-20T22:41:38.158347+0000","last_became_peered":"2024-09-20T22:41:38.158347+0000","last_unstale":"2024-09-20T22:42:28.101609+0000","last_undegraded":"2024-09-20T22:42:28.101609+0000","last_fullsized":"2024-09-20T22:42:28.101609+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:44.061004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.477085+0000","last_change":"2024-09-20T22:41:42.187707+0000","last_active":"2024-09-20T22:42:28.477085+0000","last_peered":"2024-09-20T22:42:28.477085+0000","last_clean":"2024-09-20T22:42:28.477085+0000","last_became_active":"2024-09-20T22:41:42.186068+0000","last_became_peered":"2024-09-20T22:41:42.186068+0000","last_unstale":"2024-09-20T22:42:28.477085+0000","last_undegraded":"2024-09-20T22:42:28.477085+0000","last_fullsized":"2024-09-20T22:42:28.477085+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:50.526395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":31,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.385995+0000","last_change":"2024-09-20T22:41:44.194011+0000","last_active":"2024-09-20T22:42:28.385995+0000","last_peered":"2024-09-20T22:42:28.385995+0000","last_clean":"2024-09-20T22:42:28.385995+0000","last_became_active":"2024-09-20T22:41:44.193706+0000","last_became_peered":"2024-09-20T22:41:44.193706+0000","last_unstale":"2024-09-20T22:42:28.385995+0000","last_undegraded":"2024-09-20T22:42:28.385995+0000","last_fullsized":"2024-09-20T22:42:28.385995+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:49.835925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":31,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.100377+0000","last_change":"2024-09-20T22:41:44.198154+0000","last_active":"2024-09-20T22:42:28.100377+0000","last_peered":"2024-09-20T22:42:28.100377+0000","last_clean":"2024-09-20T22:42:28.100377+0000","last_became_active":"2024-09-20T22:41:44.197855+0000","last_became_peered":"2024-09-20T22:41:44.197855+0000","last_unstale":"2024-09-20T22:42:28.100377+0000","last_undegraded":"2024-09-20T22:42:28.100377+0000","last_fullsized":"2024-09-20T22:42:28.100377+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:56.991123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":43,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.477270+0000","last_change":"2024-09-20T22:41:38.155001+0000","last_active":"2024-09-20T22:42:28.477270+0000","last_peered":"2024-09-20T22:42:28.477270+0000","last_clean":"2024-09-20T22:42:28.477270+0000","last_became_active":"2024-09-20T22:41:38.154703+0000","last_became_peered":"2024-09-20T22:41:38.154703+0000","last_unstale":"2024-09-20T22:42:28.477270+0000","last_undegraded":"2024-09-20T22:42:28.477270+0000","last_fullsized":"2024-09-20T22:42:28.477270+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:44.782808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"74'11","reported_seq":58,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.476930+0000","last_change":"2024-09-20T22:41:40.171840+0000","last_active":"2024-09-20T22:42:28.476930+0000","last_peered":"2024-09-20T22:42:28.476930+0000","last_clean":"2024-09-20T22:42:28.476930+0000","last_became_active":"2024-09-20T22:41:40.171256+0000","last_became_peered":"2024-09-20T22:41:40.171256+0000","last_unstale":"2024-09-20T22:42:28.476930+0000","last_undegraded":"2024-09-20T22:42:28.476930+0000","last_fullsized":"2024-09-20T22:42:28.476930+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:53.884281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.100928+0000","last_change":"2024-09-20T22:41:42.178413+0000","last_active":"2024-09-20T22:42:28.100928+0000","last_peered":"2024-09-20T22:42:28.100928+0000","last_clean":"2024-09-20T22:42:28.100928+0000","last_became_active":"2024-09-20T22:41:42.178157+0000","last_became_peered":"2024-09-20T22:41:42.178157+0000","last_unstale":"2024-09-20T22:42:28.100928+0000","last_undegraded":"2024-09-20T22:42:28.100928+0000","last_fullsized":"2024-09-20T22:42:28.100928+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:01.837489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":393,"ondisk_log_size":393,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070478,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32472,"kb_used_data":1284,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712680,"statfs":{"total":95995035648,"available":95961784320,"internally_reserved":0,"allocated":1314816,"data_stored":899197,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430228,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32456,"kb_used_data":1268,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712696,"statfs":{"total":95995035648,"available":95961800704,"internally_reserved":0,"allocated":1298432,"data_stored":899105,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691865,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27720,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717432,"statfs":{"total":95995035648,"available":95966650368,"internally_reserved":0,"allocated":708608,"data_stored":309519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986206,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27740,"kb_used_data":712,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717412,"statfs":{"total":95995035648,"available":95966629888,"internally_reserved":0,"allocated":729088,"data_stored":308349,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182436,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27748,"kb_used_data":720,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717404,"statfs":{"total":95995035648,"available":95966621696,"internally_reserved":0,"allocated":737280,"data_stored":309615,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411369,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31868,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713284,"statfs":{"total":95995035648,"available":95962402816,"internally_reserved":0,"allocated":696320,"data_stored":308315,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574894,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27772,"kb_used_data":744,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717380,"statfs":{"total":95995035648,"available":95966597120,"internally_reserved":0,"allocated":761856,"data_stored":309516,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738419,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32476,"kb_used_data":1288,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712676,"statfs":{"total":95995035648,"available":95961780224,"internally_reserved":0,"allocated":1318912,"data_stored":899329,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-20T22:42:32.170 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph pg dump --format=json 2024-09-20T22:42:32.416 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:42:32.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:32 smithi154 ceph-mon[33264]: from='client.24656 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T22:42:32.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:32 smithi154 ceph-mon[33264]: osdmap e84: 8 total, 8 up, 8 in 2024-09-20T22:42:33.034 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-20T22:42:33.036 INFO:teuthology.orchestra.run.smithi084.stderr:dumped all 2024-09-20T22:42:33.364 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:33 smithi084 ceph-mon[40024]: pgmap v22: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T22:42:33.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:33 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:42:33.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:33 smithi084 ceph-mon[29774]: pgmap v22: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T22:42:33.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:33 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:42:33.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:33 smithi154 ceph-mon[33264]: pgmap v22: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T22:42:33.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:33 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:42:33.863 INFO:teuthology.orchestra.run.smithi084.stdout:{"pg_ready":true,"pg_map":{"version":22,"stamp":"2024-09-20T22:42:31.615153+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596507,"num_objects":199,"num_object_clones":0,"num_object_copies":597,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":199,"num_whiteouts":0,"num_read":979,"num_read_kb":825,"num_write":615,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":601,"ondisk_log_size":601,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":240252,"kb_used_data":7388,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":749720964,"statfs":{"total":767960285184,"available":767714267136,"internally_reserved":0,"allocated":7565312,"data_stored":4242945,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001817"},"pg_stats":[{"pgid":"3.1f","version":"67'1","reported_seq":49,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.101413+0000","last_change":"2024-09-20T22:41:38.159365+0000","last_active":"2024-09-20T22:42:28.101413+0000","last_peered":"2024-09-20T22:42:28.101413+0000","last_clean":"2024-09-20T22:42:28.101413+0000","last_became_active":"2024-09-20T22:41:38.159085+0000","last_became_peered":"2024-09-20T22:41:38.159085+0000","last_unstale":"2024-09-20T22:42:28.101413+0000","last_undegraded":"2024-09-20T22:42:28.101413+0000","last_fullsized":"2024-09-20T22:42:28.101413+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:37.732756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"74'9","reported_seq":55,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.099427+0000","last_change":"2024-09-20T22:41:40.169673+0000","last_active":"2024-09-20T22:42:28.099427+0000","last_peered":"2024-09-20T22:42:28.099427+0000","last_clean":"2024-09-20T22:42:28.099427+0000","last_became_active":"2024-09-20T22:41:40.168934+0000","last_became_peered":"2024-09-20T22:41:40.168934+0000","last_unstale":"2024-09-20T22:42:28.099427+0000","last_undegraded":"2024-09-20T22:42:28.099427+0000","last_fullsized":"2024-09-20T22:42:28.099427+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:54.489342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.099551+0000","last_change":"2024-09-20T22:41:42.182304+0000","last_active":"2024-09-20T22:42:28.099551+0000","last_peered":"2024-09-20T22:42:28.099551+0000","last_clean":"2024-09-20T22:42:28.099551+0000","last_became_active":"2024-09-20T22:41:42.182024+0000","last_became_peered":"2024-09-20T22:41:42.182024+0000","last_unstale":"2024-09-20T22:42:28.099551+0000","last_undegraded":"2024-09-20T22:42:28.099551+0000","last_fullsized":"2024-09-20T22:42:28.099551+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:57.397887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":31,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.099576+0000","last_change":"2024-09-20T22:41:44.198207+0000","last_active":"2024-09-20T22:42:28.099576+0000","last_peered":"2024-09-20T22:42:28.099576+0000","last_clean":"2024-09-20T22:42:28.099576+0000","last_became_active":"2024-09-20T22:41:44.197897+0000","last_became_peered":"2024-09-20T22:41:44.197897+0000","last_unstale":"2024-09-20T22:42:28.099576+0000","last_undegraded":"2024-09-20T22:42:28.099576+0000","last_fullsized":"2024-09-20T22:42:28.099576+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:53.466451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":31,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.477740+0000","last_change":"2024-09-20T22:41:44.832449+0000","last_active":"2024-09-20T22:42:28.477740+0000","last_peered":"2024-09-20T22:42:28.477740+0000","last_clean":"2024-09-20T22:42:28.477740+0000","last_became_active":"2024-09-20T22:41:44.832099+0000","last_became_peered":"2024-09-20T22:41:44.832099+0000","last_unstale":"2024-09-20T22:42:28.477740+0000","last_undegraded":"2024-09-20T22:42:28.477740+0000","last_fullsized":"2024-09-20T22:42:28.477740+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:45.326749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":43,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.478137+0000","last_change":"2024-09-20T22:41:38.157685+0000","last_active":"2024-09-20T22:42:28.478137+0000","last_peered":"2024-09-20T22:42:28.478137+0000","last_clean":"2024-09-20T22:42:28.478137+0000","last_became_active":"2024-09-20T22:41:38.157298+0000","last_became_peered":"2024-09-20T22:41:38.157298+0000","last_unstale":"2024-09-20T22:42:28.478137+0000","last_undegraded":"2024-09-20T22:42:28.478137+0000","last_fullsized":"2024-09-20T22:42:28.478137+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:00.813798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"74'15","reported_seq":64,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.476697+0000","last_change":"2024-09-20T22:41:40.163585+0000","last_active":"2024-09-20T22:42:28.476697+0000","last_peered":"2024-09-20T22:42:28.476697+0000","last_clean":"2024-09-20T22:42:28.476697+0000","last_became_active":"2024-09-20T22:41:40.163242+0000","last_became_peered":"2024-09-20T22:41:40.163242+0000","last_unstale":"2024-09-20T22:42:28.476697+0000","last_undegraded":"2024-09-20T22:42:28.476697+0000","last_fullsized":"2024-09-20T22:42:28.476697+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:58.410323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.099423+0000","last_change":"2024-09-20T22:41:42.187359+0000","last_active":"2024-09-20T22:42:28.099423+0000","last_peered":"2024-09-20T22:42:28.099423+0000","last_clean":"2024-09-20T22:42:28.099423+0000","last_became_active":"2024-09-20T22:41:42.187076+0000","last_became_peered":"2024-09-20T22:41:42.187076+0000","last_unstale":"2024-09-20T22:42:28.099423+0000","last_undegraded":"2024-09-20T22:42:28.099423+0000","last_fullsized":"2024-09-20T22:42:28.099423+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:42.282475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":43,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.519203+0000","last_change":"2024-09-20T22:41:38.156694+0000","last_active":"2024-09-20T22:42:28.519203+0000","last_peered":"2024-09-20T22:42:28.519203+0000","last_clean":"2024-09-20T22:42:28.519203+0000","last_became_active":"2024-09-20T22:41:38.154827+0000","last_became_peered":"2024-09-20T22:41:38.154827+0000","last_unstale":"2024-09-20T22:42:28.519203+0000","last_undegraded":"2024-09-20T22:42:28.519203+0000","last_fullsized":"2024-09-20T22:42:28.519203+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:48.618357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"74'9","reported_seq":55,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.099718+0000","last_change":"2024-09-20T22:41:40.163006+0000","last_active":"2024-09-20T22:42:28.099718+0000","last_peered":"2024-09-20T22:42:28.099718+0000","last_clean":"2024-09-20T22:42:28.099718+0000","last_became_active":"2024-09-20T22:41:40.162546+0000","last_became_peered":"2024-09-20T22:41:40.162546+0000","last_unstale":"2024-09-20T22:42:28.099718+0000","last_undegraded":"2024-09-20T22:42:28.099718+0000","last_fullsized":"2024-09-20T22:42:28.099718+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:46.501647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.518950+0000","last_change":"2024-09-20T22:41:42.185835+0000","last_active":"2024-09-20T22:42:28.518950+0000","last_peered":"2024-09-20T22:42:28.518950+0000","last_clean":"2024-09-20T22:42:28.518950+0000","last_became_active":"2024-09-20T22:41:42.185538+0000","last_became_peered":"2024-09-20T22:41:42.185538+0000","last_unstale":"2024-09-20T22:42:28.518950+0000","last_undegraded":"2024-09-20T22:42:28.518950+0000","last_fullsized":"2024-09-20T22:42:28.518950+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:57.305761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":31,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.099354+0000","last_change":"2024-09-20T22:41:44.198169+0000","last_active":"2024-09-20T22:42:28.099354+0000","last_peered":"2024-09-20T22:42:28.099354+0000","last_clean":"2024-09-20T22:42:28.099354+0000","last_became_active":"2024-09-20T22:41:44.197113+0000","last_became_peered":"2024-09-20T22:41:44.197113+0000","last_unstale":"2024-09-20T22:42:28.099354+0000","last_undegraded":"2024-09-20T22:42:28.099354+0000","last_fullsized":"2024-09-20T22:42:28.099354+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:02.374640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":43,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.519140+0000","last_change":"2024-09-20T22:41:38.156116+0000","last_active":"2024-09-20T22:42:28.519140+0000","last_peered":"2024-09-20T22:42:28.519140+0000","last_clean":"2024-09-20T22:42:28.519140+0000","last_became_active":"2024-09-20T22:41:38.154650+0000","last_became_peered":"2024-09-20T22:41:38.154650+0000","last_unstale":"2024-09-20T22:42:28.519140+0000","last_undegraded":"2024-09-20T22:42:28.519140+0000","last_fullsized":"2024-09-20T22:42:28.519140+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:44.595033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"74'5","reported_seq":49,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.099778+0000","last_change":"2024-09-20T22:41:40.171883+0000","last_active":"2024-09-20T22:42:28.099778+0000","last_peered":"2024-09-20T22:42:28.099778+0000","last_clean":"2024-09-20T22:42:28.099778+0000","last_became_active":"2024-09-20T22:41:40.171081+0000","last_became_peered":"2024-09-20T22:41:40.171081+0000","last_unstale":"2024-09-20T22:42:28.099778+0000","last_undegraded":"2024-09-20T22:42:28.099778+0000","last_fullsized":"2024-09-20T22:42:28.099778+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:55.048729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.386085+0000","last_change":"2024-09-20T22:41:42.183965+0000","last_active":"2024-09-20T22:42:28.386085+0000","last_peered":"2024-09-20T22:42:28.386085+0000","last_clean":"2024-09-20T22:42:28.386085+0000","last_became_active":"2024-09-20T22:41:42.182916+0000","last_became_peered":"2024-09-20T22:41:42.182916+0000","last_unstale":"2024-09-20T22:42:28.386085+0000","last_undegraded":"2024-09-20T22:42:28.386085+0000","last_fullsized":"2024-09-20T22:42:28.386085+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:42.258865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":31,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.518829+0000","last_change":"2024-09-20T22:41:44.198221+0000","last_active":"2024-09-20T22:42:28.518829+0000","last_peered":"2024-09-20T22:42:28.518829+0000","last_clean":"2024-09-20T22:42:28.518829+0000","last_became_active":"2024-09-20T22:41:44.197927+0000","last_became_peered":"2024-09-20T22:41:44.197927+0000","last_unstale":"2024-09-20T22:42:28.518829+0000","last_undegraded":"2024-09-20T22:42:28.518829+0000","last_fullsized":"2024-09-20T22:42:28.518829+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:47.985353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":31,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.099979+0000","last_change":"2024-09-20T22:41:44.832646+0000","last_active":"2024-09-20T22:42:28.099979+0000","last_peered":"2024-09-20T22:42:28.099979+0000","last_clean":"2024-09-20T22:42:28.099979+0000","last_became_active":"2024-09-20T22:41:44.832360+0000","last_became_peered":"2024-09-20T22:41:44.832360+0000","last_unstale":"2024-09-20T22:42:28.099979+0000","last_undegraded":"2024-09-20T22:42:28.099979+0000","last_fullsized":"2024-09-20T22:42:28.099979+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:45.707782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":43,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.100583+0000","last_change":"2024-09-20T22:41:38.152726+0000","last_active":"2024-09-20T22:42:28.100583+0000","last_peered":"2024-09-20T22:42:28.100583+0000","last_clean":"2024-09-20T22:42:28.100583+0000","last_became_active":"2024-09-20T22:41:38.152443+0000","last_became_peered":"2024-09-20T22:41:38.152443+0000","last_unstale":"2024-09-20T22:42:28.100583+0000","last_undegraded":"2024-09-20T22:42:28.100583+0000","last_fullsized":"2024-09-20T22:42:28.100583+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:03.325746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"74'15","reported_seq":64,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.266322+0000","last_change":"2024-09-20T22:41:40.170414+0000","last_active":"2024-09-20T22:42:28.266322+0000","last_peered":"2024-09-20T22:42:28.266322+0000","last_clean":"2024-09-20T22:42:28.266322+0000","last_became_active":"2024-09-20T22:41:40.170131+0000","last_became_peered":"2024-09-20T22:41:40.170131+0000","last_unstale":"2024-09-20T22:42:28.266322+0000","last_undegraded":"2024-09-20T22:42:28.266322+0000","last_fullsized":"2024-09-20T22:42:28.266322+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:57.393598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.100537+0000","last_change":"2024-09-20T22:41:42.185171+0000","last_active":"2024-09-20T22:42:28.100537+0000","last_peered":"2024-09-20T22:42:28.100537+0000","last_clean":"2024-09-20T22:42:28.100537+0000","last_became_active":"2024-09-20T22:41:42.184759+0000","last_became_peered":"2024-09-20T22:41:42.184759+0000","last_unstale":"2024-09-20T22:42:28.100537+0000","last_undegraded":"2024-09-20T22:42:28.100537+0000","last_fullsized":"2024-09-20T22:42:28.100537+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:01.601642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":31,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.478139+0000","last_change":"2024-09-20T22:41:44.832468+0000","last_active":"2024-09-20T22:42:28.478139+0000","last_peered":"2024-09-20T22:42:28.478139+0000","last_clean":"2024-09-20T22:42:28.478139+0000","last_became_active":"2024-09-20T22:41:44.832131+0000","last_became_peered":"2024-09-20T22:41:44.832131+0000","last_unstale":"2024-09-20T22:42:28.478139+0000","last_undegraded":"2024-09-20T22:42:28.478139+0000","last_fullsized":"2024-09-20T22:42:28.478139+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:08.595231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":43,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.099825+0000","last_change":"2024-09-20T22:41:38.150584+0000","last_active":"2024-09-20T22:42:28.099825+0000","last_peered":"2024-09-20T22:42:28.099825+0000","last_clean":"2024-09-20T22:42:28.099825+0000","last_became_active":"2024-09-20T22:41:38.150232+0000","last_became_peered":"2024-09-20T22:41:38.150232+0000","last_unstale":"2024-09-20T22:42:28.099825+0000","last_undegraded":"2024-09-20T22:42:28.099825+0000","last_fullsized":"2024-09-20T22:42:28.099825+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:02.237312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"74'12","reported_seq":62,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.476952+0000","last_change":"2024-09-20T22:41:40.166344+0000","last_active":"2024-09-20T22:42:28.476952+0000","last_peered":"2024-09-20T22:42:28.476952+0000","last_clean":"2024-09-20T22:42:28.476952+0000","last_became_active":"2024-09-20T22:41:40.166087+0000","last_became_peered":"2024-09-20T22:41:40.166087+0000","last_unstale":"2024-09-20T22:42:28.476952+0000","last_undegraded":"2024-09-20T22:42:28.476952+0000","last_fullsized":"2024-09-20T22:42:28.476952+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:01.373771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.099869+0000","last_change":"2024-09-20T22:41:42.183266+0000","last_active":"2024-09-20T22:42:28.099869+0000","last_peered":"2024-09-20T22:42:28.099869+0000","last_clean":"2024-09-20T22:42:28.099869+0000","last_became_active":"2024-09-20T22:41:42.183005+0000","last_became_peered":"2024-09-20T22:41:42.183005+0000","last_unstale":"2024-09-20T22:42:28.099869+0000","last_undegraded":"2024-09-20T22:42:28.099869+0000","last_fullsized":"2024-09-20T22:42:28.099869+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:01.169906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":32,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.386277+0000","last_change":"2024-09-20T22:41:44.200115+0000","last_active":"2024-09-20T22:42:28.386277+0000","last_peered":"2024-09-20T22:42:28.386277+0000","last_clean":"2024-09-20T22:42:28.386277+0000","last_became_active":"2024-09-20T22:41:44.199822+0000","last_became_peered":"2024-09-20T22:41:44.199822+0000","last_unstale":"2024-09-20T22:42:28.386277+0000","last_undegraded":"2024-09-20T22:42:28.386277+0000","last_fullsized":"2024-09-20T22:42:28.386277+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:01.277411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"67'1","reported_seq":44,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.099727+0000","last_change":"2024-09-20T22:41:38.155735+0000","last_active":"2024-09-20T22:42:28.099727+0000","last_peered":"2024-09-20T22:42:28.099727+0000","last_clean":"2024-09-20T22:42:28.099727+0000","last_became_active":"2024-09-20T22:41:38.155424+0000","last_became_peered":"2024-09-20T22:41:38.155424+0000","last_unstale":"2024-09-20T22:42:28.099727+0000","last_undegraded":"2024-09-20T22:42:28.099727+0000","last_fullsized":"2024-09-20T22:42:28.099727+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:52.584133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"74'10","reported_seq":54,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.101040+0000","last_change":"2024-09-20T22:41:40.252498+0000","last_active":"2024-09-20T22:42:28.101040+0000","last_peered":"2024-09-20T22:42:28.101040+0000","last_clean":"2024-09-20T22:42:28.101040+0000","last_became_active":"2024-09-20T22:41:40.252001+0000","last_became_peered":"2024-09-20T22:41:40.252001+0000","last_unstale":"2024-09-20T22:42:28.101040+0000","last_undegraded":"2024-09-20T22:42:28.101040+0000","last_fullsized":"2024-09-20T22:42:28.101040+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:01.417640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"74'8","reported_seq":60,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.477716+0000","last_change":"2024-09-20T22:41:42.186385+0000","last_active":"2024-09-20T22:42:28.477716+0000","last_peered":"2024-09-20T22:42:28.477716+0000","last_clean":"2024-09-20T22:42:28.477716+0000","last_became_active":"2024-09-20T22:41:42.186071+0000","last_became_peered":"2024-09-20T22:41:42.186071+0000","last_unstale":"2024-09-20T22:42:28.477716+0000","last_undegraded":"2024-09-20T22:42:28.477716+0000","last_fullsized":"2024-09-20T22:42:28.477716+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:56.929679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.f","version":"74'15","reported_seq":64,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.101184+0000","last_change":"2024-09-20T22:41:40.168188+0000","last_active":"2024-09-20T22:42:28.101184+0000","last_peered":"2024-09-20T22:42:28.101184+0000","last_clean":"2024-09-20T22:42:28.101184+0000","last_became_active":"2024-09-20T22:41:40.165963+0000","last_became_peered":"2024-09-20T22:41:40.165963+0000","last_unstale":"2024-09-20T22:42:28.101184+0000","last_undegraded":"2024-09-20T22:42:28.101184+0000","last_fullsized":"2024-09-20T22:42:28.101184+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:46.519265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":43,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.477315+0000","last_change":"2024-09-20T22:41:38.157043+0000","last_active":"2024-09-20T22:42:28.477315+0000","last_peered":"2024-09-20T22:42:28.477315+0000","last_clean":"2024-09-20T22:42:28.477315+0000","last_became_active":"2024-09-20T22:41:38.156749+0000","last_became_peered":"2024-09-20T22:41:38.156749+0000","last_unstale":"2024-09-20T22:42:28.477315+0000","last_undegraded":"2024-09-20T22:42:28.477315+0000","last_fullsized":"2024-09-20T22:42:28.477315+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:03.406143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"74'8","reported_seq":54,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.100151+0000","last_change":"2024-09-20T22:41:42.182555+0000","last_active":"2024-09-20T22:42:28.100151+0000","last_peered":"2024-09-20T22:42:28.100151+0000","last_clean":"2024-09-20T22:42:28.100151+0000","last_became_active":"2024-09-20T22:41:42.182224+0000","last_became_peered":"2024-09-20T22:41:42.182224+0000","last_unstale":"2024-09-20T22:42:28.100151+0000","last_undegraded":"2024-09-20T22:42:28.100151+0000","last_fullsized":"2024-09-20T22:42:28.100151+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:07.384166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":31,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.518949+0000","last_change":"2024-09-20T22:41:44.198655+0000","last_active":"2024-09-20T22:42:28.518949+0000","last_peered":"2024-09-20T22:42:28.518949+0000","last_clean":"2024-09-20T22:42:28.518949+0000","last_became_active":"2024-09-20T22:41:44.198347+0000","last_became_peered":"2024-09-20T22:41:44.198347+0000","last_unstale":"2024-09-20T22:42:28.518949+0000","last_undegraded":"2024-09-20T22:42:28.518949+0000","last_fullsized":"2024-09-20T22:42:28.518949+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:54.119646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.0","version":"74'18","reported_seq":71,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.476692+0000","last_change":"2024-09-20T22:41:40.252203+0000","last_active":"2024-09-20T22:42:28.476692+0000","last_peered":"2024-09-20T22:42:28.476692+0000","last_clean":"2024-09-20T22:42:28.476692+0000","last_became_active":"2024-09-20T22:41:40.251868+0000","last_became_peered":"2024-09-20T22:41:40.251868+0000","last_unstale":"2024-09-20T22:42:28.476692+0000","last_undegraded":"2024-09-20T22:42:28.476692+0000","last_fullsized":"2024-09-20T22:42:28.476692+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:57.966357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":43,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.478253+0000","last_change":"2024-09-20T22:41:38.161991+0000","last_active":"2024-09-20T22:42:28.478253+0000","last_peered":"2024-09-20T22:42:28.478253+0000","last_clean":"2024-09-20T22:42:28.478253+0000","last_became_active":"2024-09-20T22:41:38.161335+0000","last_became_peered":"2024-09-20T22:41:38.161335+0000","last_unstale":"2024-09-20T22:42:28.478253+0000","last_undegraded":"2024-09-20T22:42:28.478253+0000","last_fullsized":"2024-09-20T22:42:28.478253+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:48.134689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.099465+0000","last_change":"2024-09-20T22:41:42.190157+0000","last_active":"2024-09-20T22:42:28.099465+0000","last_peered":"2024-09-20T22:42:28.099465+0000","last_clean":"2024-09-20T22:42:28.099465+0000","last_became_active":"2024-09-20T22:41:42.189743+0000","last_became_peered":"2024-09-20T22:41:42.189743+0000","last_unstale":"2024-09-20T22:42:28.099465+0000","last_undegraded":"2024-09-20T22:42:28.099465+0000","last_fullsized":"2024-09-20T22:42:28.099465+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:42.602485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":31,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.099689+0000","last_change":"2024-09-20T22:41:44.193944+0000","last_active":"2024-09-20T22:42:28.099689+0000","last_peered":"2024-09-20T22:42:28.099689+0000","last_clean":"2024-09-20T22:42:28.099689+0000","last_became_active":"2024-09-20T22:41:44.193646+0000","last_became_peered":"2024-09-20T22:41:44.193646+0000","last_unstale":"2024-09-20T22:42:28.099689+0000","last_undegraded":"2024-09-20T22:42:28.099689+0000","last_fullsized":"2024-09-20T22:42:28.099689+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:50.182915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"74'14","reported_seq":60,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.099501+0000","last_change":"2024-09-20T22:41:40.164011+0000","last_active":"2024-09-20T22:42:28.099501+0000","last_peered":"2024-09-20T22:42:28.099501+0000","last_clean":"2024-09-20T22:42:28.099501+0000","last_became_active":"2024-09-20T22:41:40.163695+0000","last_became_peered":"2024-09-20T22:41:40.163695+0000","last_unstale":"2024-09-20T22:42:28.099501+0000","last_undegraded":"2024-09-20T22:42:28.099501+0000","last_fullsized":"2024-09-20T22:42:28.099501+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:04.888108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"67'1","reported_seq":44,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.101402+0000","last_change":"2024-09-20T22:41:38.150707+0000","last_active":"2024-09-20T22:42:28.101402+0000","last_peered":"2024-09-20T22:42:28.101402+0000","last_clean":"2024-09-20T22:42:28.101402+0000","last_became_active":"2024-09-20T22:41:38.150331+0000","last_became_peered":"2024-09-20T22:41:38.150331+0000","last_unstale":"2024-09-20T22:42:28.101402+0000","last_undegraded":"2024-09-20T22:42:28.101402+0000","last_fullsized":"2024-09-20T22:42:28.101402+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:53.797157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"74'8","reported_seq":54,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.477173+0000","last_change":"2024-09-20T22:41:42.179893+0000","last_active":"2024-09-20T22:42:28.477173+0000","last_peered":"2024-09-20T22:42:28.477173+0000","last_clean":"2024-09-20T22:42:28.477173+0000","last_became_active":"2024-09-20T22:41:42.179613+0000","last_became_peered":"2024-09-20T22:41:42.179613+0000","last_unstale":"2024-09-20T22:42:28.477173+0000","last_undegraded":"2024-09-20T22:42:28.477173+0000","last_fullsized":"2024-09-20T22:42:28.477173+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:03.642295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":31,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.386167+0000","last_change":"2024-09-20T22:41:44.832161+0000","last_active":"2024-09-20T22:42:28.386167+0000","last_peered":"2024-09-20T22:42:28.386167+0000","last_clean":"2024-09-20T22:42:28.386167+0000","last_became_active":"2024-09-20T22:41:44.831901+0000","last_became_peered":"2024-09-20T22:41:44.831901+0000","last_unstale":"2024-09-20T22:42:28.386167+0000","last_undegraded":"2024-09-20T22:42:28.386167+0000","last_fullsized":"2024-09-20T22:42:28.386167+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:10.151190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2","version":"74'10","reported_seq":54,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.100294+0000","last_change":"2024-09-20T22:41:40.167041+0000","last_active":"2024-09-20T22:42:28.100294+0000","last_peered":"2024-09-20T22:42:28.100294+0000","last_clean":"2024-09-20T22:42:28.100294+0000","last_became_active":"2024-09-20T22:41:40.165653+0000","last_became_peered":"2024-09-20T22:41:40.165653+0000","last_unstale":"2024-09-20T22:42:28.100294+0000","last_undegraded":"2024-09-20T22:42:28.100294+0000","last_fullsized":"2024-09-20T22:42:28.100294+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:47.255874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":43,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.519276+0000","last_change":"2024-09-20T22:41:38.155924+0000","last_active":"2024-09-20T22:42:28.519276+0000","last_peered":"2024-09-20T22:42:28.519276+0000","last_clean":"2024-09-20T22:42:28.519276+0000","last_became_active":"2024-09-20T22:41:38.155629+0000","last_became_peered":"2024-09-20T22:41:38.155629+0000","last_unstale":"2024-09-20T22:42:28.519276+0000","last_undegraded":"2024-09-20T22:42:28.519276+0000","last_fullsized":"2024-09-20T22:42:28.519276+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:39.438618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"74'8","reported_seq":54,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.100192+0000","last_change":"2024-09-20T22:41:42.185583+0000","last_active":"2024-09-20T22:42:28.100192+0000","last_peered":"2024-09-20T22:42:28.100192+0000","last_clean":"2024-09-20T22:42:28.100192+0000","last_became_active":"2024-09-20T22:41:42.185213+0000","last_became_peered":"2024-09-20T22:41:42.185213+0000","last_unstale":"2024-09-20T22:42:28.100192+0000","last_undegraded":"2024-09-20T22:42:28.100192+0000","last_fullsized":"2024-09-20T22:42:28.100192+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:48.459335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":31,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.099546+0000","last_change":"2024-09-20T22:41:44.197894+0000","last_active":"2024-09-20T22:42:28.099546+0000","last_peered":"2024-09-20T22:42:28.099546+0000","last_clean":"2024-09-20T22:42:28.099546+0000","last_became_active":"2024-09-20T22:41:44.197619+0000","last_became_peered":"2024-09-20T22:41:44.197619+0000","last_unstale":"2024-09-20T22:42:28.099546+0000","last_undegraded":"2024-09-20T22:42:28.099546+0000","last_fullsized":"2024-09-20T22:42:28.099546+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:00.802751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"74'19","reported_seq":75,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.100352+0000","last_change":"2024-09-20T22:41:40.252159+0000","last_active":"2024-09-20T22:42:28.100352+0000","last_peered":"2024-09-20T22:42:28.100352+0000","last_clean":"2024-09-20T22:42:28.100352+0000","last_became_active":"2024-09-20T22:41:40.251866+0000","last_became_peered":"2024-09-20T22:41:40.251866+0000","last_unstale":"2024-09-20T22:42:28.100352+0000","last_undegraded":"2024-09-20T22:42:28.100352+0000","last_fullsized":"2024-09-20T22:42:28.100352+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:43.089389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":43,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.101213+0000","last_change":"2024-09-20T22:41:38.161236+0000","last_active":"2024-09-20T22:42:28.101213+0000","last_peered":"2024-09-20T22:42:28.101213+0000","last_clean":"2024-09-20T22:42:28.101213+0000","last_became_active":"2024-09-20T22:41:38.160832+0000","last_became_peered":"2024-09-20T22:41:38.160832+0000","last_unstale":"2024-09-20T22:42:28.101213+0000","last_undegraded":"2024-09-20T22:42:28.101213+0000","last_fullsized":"2024-09-20T22:42:28.101213+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:50.975305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.477283+0000","last_change":"2024-09-20T22:41:42.185638+0000","last_active":"2024-09-20T22:42:28.477283+0000","last_peered":"2024-09-20T22:42:28.477283+0000","last_clean":"2024-09-20T22:42:28.477283+0000","last_became_active":"2024-09-20T22:41:42.184412+0000","last_became_peered":"2024-09-20T22:41:42.184412+0000","last_unstale":"2024-09-20T22:42:28.477283+0000","last_undegraded":"2024-09-20T22:42:28.477283+0000","last_fullsized":"2024-09-20T22:42:28.477283+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:57.435580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":31,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.099499+0000","last_change":"2024-09-20T22:41:44.830917+0000","last_active":"2024-09-20T22:42:28.099499+0000","last_peered":"2024-09-20T22:42:28.099499+0000","last_clean":"2024-09-20T22:42:28.099499+0000","last_became_active":"2024-09-20T22:41:44.830538+0000","last_became_peered":"2024-09-20T22:41:44.830538+0000","last_unstale":"2024-09-20T22:42:28.099499+0000","last_undegraded":"2024-09-20T22:42:28.099499+0000","last_fullsized":"2024-09-20T22:42:28.099499+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:54.961702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"74'28","reported_seq":98,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.101115+0000","last_change":"2024-09-20T22:41:40.167313+0000","last_active":"2024-09-20T22:42:28.101115+0000","last_peered":"2024-09-20T22:42:28.101115+0000","last_clean":"2024-09-20T22:42:28.101115+0000","last_became_active":"2024-09-20T22:41:40.166958+0000","last_became_peered":"2024-09-20T22:41:40.166958+0000","last_unstale":"2024-09-20T22:42:28.101115+0000","last_undegraded":"2024-09-20T22:42:28.101115+0000","last_fullsized":"2024-09-20T22:42:28.101115+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:42.825818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":43,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.100061+0000","last_change":"2024-09-20T22:41:38.152921+0000","last_active":"2024-09-20T22:42:28.100061+0000","last_peered":"2024-09-20T22:42:28.100061+0000","last_clean":"2024-09-20T22:42:28.100061+0000","last_became_active":"2024-09-20T22:41:38.152525+0000","last_became_peered":"2024-09-20T22:41:38.152525+0000","last_unstale":"2024-09-20T22:42:28.100061+0000","last_undegraded":"2024-09-20T22:42:28.100061+0000","last_fullsized":"2024-09-20T22:42:28.100061+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:37.641089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"69'2","reported_seq":53,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.519001+0000","last_change":"2024-09-20T22:41:40.154106+0000","last_active":"2024-09-20T22:42:28.519001+0000","last_peered":"2024-09-20T22:42:28.519001+0000","last_clean":"2024-09-20T22:42:28.519001+0000","last_became_active":"2024-09-20T22:41:37.148479+0000","last_became_peered":"2024-09-20T22:41:37.148479+0000","last_unstale":"2024-09-20T22:42:28.519001+0000","last_undegraded":"2024-09-20T22:42:28.519001+0000","last_fullsized":"2024-09-20T22:42:28.519001+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:36.130684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:36.130684+0000","last_clean_scrub_stamp":"2024-09-20T22:41:36.130684+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:48.204536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087280599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.100077+0000","last_change":"2024-09-20T22:41:42.179979+0000","last_active":"2024-09-20T22:42:28.100077+0000","last_peered":"2024-09-20T22:42:28.100077+0000","last_clean":"2024-09-20T22:42:28.100077+0000","last_became_active":"2024-09-20T22:41:42.179742+0000","last_became_peered":"2024-09-20T22:41:42.179742+0000","last_unstale":"2024-09-20T22:42:28.100077+0000","last_undegraded":"2024-09-20T22:42:28.100077+0000","last_fullsized":"2024-09-20T22:42:28.100077+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:03.273567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":31,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.478033+0000","last_change":"2024-09-20T22:41:44.192602+0000","last_active":"2024-09-20T22:42:28.478033+0000","last_peered":"2024-09-20T22:42:28.478033+0000","last_clean":"2024-09-20T22:42:28.478033+0000","last_became_active":"2024-09-20T22:41:44.192128+0000","last_became_peered":"2024-09-20T22:41:44.192128+0000","last_unstale":"2024-09-20T22:42:28.478033+0000","last_undegraded":"2024-09-20T22:42:28.478033+0000","last_fullsized":"2024-09-20T22:42:28.478033+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:46.416059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"74'13","reported_seq":66,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.101279+0000","last_change":"2024-09-20T22:41:40.169777+0000","last_active":"2024-09-20T22:42:28.101279+0000","last_peered":"2024-09-20T22:42:28.101279+0000","last_clean":"2024-09-20T22:42:28.101279+0000","last_became_active":"2024-09-20T22:41:40.169473+0000","last_became_peered":"2024-09-20T22:41:40.169473+0000","last_unstale":"2024-09-20T22:42:28.101279+0000","last_undegraded":"2024-09-20T22:42:28.101279+0000","last_fullsized":"2024-09-20T22:42:28.101279+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:48.626103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":43,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.101859+0000","last_change":"2024-09-20T22:41:38.157113+0000","last_active":"2024-09-20T22:42:28.101859+0000","last_peered":"2024-09-20T22:42:28.101859+0000","last_clean":"2024-09-20T22:42:28.101859+0000","last_became_active":"2024-09-20T22:41:38.156837+0000","last_became_peered":"2024-09-20T22:41:38.156837+0000","last_unstale":"2024-09-20T22:42:28.101859+0000","last_undegraded":"2024-09-20T22:42:28.101859+0000","last_fullsized":"2024-09-20T22:42:28.101859+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:46.990363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"67'1","reported_seq":52,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.266395+0000","last_change":"2024-09-20T22:41:40.162396+0000","last_active":"2024-09-20T22:42:28.266395+0000","last_peered":"2024-09-20T22:42:28.266395+0000","last_clean":"2024-09-20T22:42:28.266395+0000","last_became_active":"2024-09-20T22:41:37.148165+0000","last_became_peered":"2024-09-20T22:41:37.148165+0000","last_unstale":"2024-09-20T22:42:28.266395+0000","last_undegraded":"2024-09-20T22:42:28.266395+0000","last_fullsized":"2024-09-20T22:42:28.266395+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:36.130684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:36.130684+0000","last_clean_scrub_stamp":"2024-09-20T22:41:36.130684+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:54.327893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001301459,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.267442+0000","last_change":"2024-09-20T22:41:42.185412+0000","last_active":"2024-09-20T22:42:28.267442+0000","last_peered":"2024-09-20T22:42:28.267442+0000","last_clean":"2024-09-20T22:42:28.267442+0000","last_became_active":"2024-09-20T22:41:42.185112+0000","last_became_peered":"2024-09-20T22:41:42.185112+0000","last_unstale":"2024-09-20T22:42:28.267442+0000","last_undegraded":"2024-09-20T22:42:28.267442+0000","last_fullsized":"2024-09-20T22:42:28.267442+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:05.899621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":31,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.386386+0000","last_change":"2024-09-20T22:41:44.830804+0000","last_active":"2024-09-20T22:42:28.386386+0000","last_peered":"2024-09-20T22:42:28.386386+0000","last_clean":"2024-09-20T22:42:28.386386+0000","last_became_active":"2024-09-20T22:41:44.830515+0000","last_became_peered":"2024-09-20T22:41:44.830515+0000","last_unstale":"2024-09-20T22:42:28.386386+0000","last_undegraded":"2024-09-20T22:42:28.386386+0000","last_fullsized":"2024-09-20T22:42:28.386386+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:04.958838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"74'12","reported_seq":57,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.101150+0000","last_change":"2024-09-20T22:41:40.167483+0000","last_active":"2024-09-20T22:42:28.101150+0000","last_peered":"2024-09-20T22:42:28.101150+0000","last_clean":"2024-09-20T22:42:28.101150+0000","last_became_active":"2024-09-20T22:41:40.167169+0000","last_became_peered":"2024-09-20T22:41:40.167169+0000","last_unstale":"2024-09-20T22:42:28.101150+0000","last_undegraded":"2024-09-20T22:42:28.101150+0000","last_fullsized":"2024-09-20T22:42:28.101150+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:48.803800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":43,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.099306+0000","last_change":"2024-09-20T22:41:38.153448+0000","last_active":"2024-09-20T22:42:28.099306+0000","last_peered":"2024-09-20T22:42:28.099306+0000","last_clean":"2024-09-20T22:42:28.099306+0000","last_became_active":"2024-09-20T22:41:38.153058+0000","last_became_peered":"2024-09-20T22:41:38.153058+0000","last_unstale":"2024-09-20T22:42:28.099306+0000","last_undegraded":"2024-09-20T22:42:28.099306+0000","last_fullsized":"2024-09-20T22:42:28.099306+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:04.425050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"75'5","reported_seq":99,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.386040+0000","last_change":"2024-09-20T22:41:40.251762+0000","last_active":"2024-09-20T22:42:28.386040+0000","last_peered":"2024-09-20T22:42:28.386040+0000","last_clean":"2024-09-20T22:42:28.386040+0000","last_became_active":"2024-09-20T22:41:37.148383+0000","last_became_peered":"2024-09-20T22:41:37.148383+0000","last_unstale":"2024-09-20T22:42:28.386040+0000","last_undegraded":"2024-09-20T22:42:28.386040+0000","last_fullsized":"2024-09-20T22:42:28.386040+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:36.130684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:36.130684+0000","last_clean_scrub_stamp":"2024-09-20T22:41:36.130684+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:04.208717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00078631700000000003,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.519377+0000","last_change":"2024-09-20T22:41:42.187618+0000","last_active":"2024-09-20T22:42:28.519377+0000","last_peered":"2024-09-20T22:42:28.519377+0000","last_clean":"2024-09-20T22:42:28.519377+0000","last_became_active":"2024-09-20T22:41:42.187346+0000","last_became_peered":"2024-09-20T22:41:42.187346+0000","last_unstale":"2024-09-20T22:42:28.519377+0000","last_undegraded":"2024-09-20T22:42:28.519377+0000","last_fullsized":"2024-09-20T22:42:28.519377+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:58.427580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":31,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.100779+0000","last_change":"2024-09-20T22:41:44.199301+0000","last_active":"2024-09-20T22:42:28.100779+0000","last_peered":"2024-09-20T22:42:28.100779+0000","last_clean":"2024-09-20T22:42:28.100779+0000","last_became_active":"2024-09-20T22:41:44.199046+0000","last_became_peered":"2024-09-20T22:41:44.199046+0000","last_unstale":"2024-09-20T22:42:28.100779+0000","last_undegraded":"2024-09-20T22:42:28.100779+0000","last_fullsized":"2024-09-20T22:42:28.100779+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:43.190792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"74'16","reported_seq":72,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.476721+0000","last_change":"2024-09-20T22:41:40.253524+0000","last_active":"2024-09-20T22:42:28.476721+0000","last_peered":"2024-09-20T22:42:28.476721+0000","last_clean":"2024-09-20T22:42:28.476721+0000","last_became_active":"2024-09-20T22:41:40.252933+0000","last_became_peered":"2024-09-20T22:41:40.252933+0000","last_unstale":"2024-09-20T22:42:28.476721+0000","last_undegraded":"2024-09-20T22:42:28.476721+0000","last_fullsized":"2024-09-20T22:42:28.476721+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:54.546845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"67'1","reported_seq":49,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.477599+0000","last_change":"2024-09-20T22:41:38.159706+0000","last_active":"2024-09-20T22:42:28.477599+0000","last_peered":"2024-09-20T22:42:28.477599+0000","last_clean":"2024-09-20T22:42:28.477599+0000","last_became_active":"2024-09-20T22:41:38.158917+0000","last_became_peered":"2024-09-20T22:41:38.158917+0000","last_unstale":"2024-09-20T22:42:28.477599+0000","last_undegraded":"2024-09-20T22:42:28.477599+0000","last_fullsized":"2024-09-20T22:42:28.477599+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:03.995423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"77'128","reported_seq":300,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.386165+0000","last_change":"2024-09-20T22:41:43.408723+0000","last_active":"2024-09-20T22:42:28.386165+0000","last_peered":"2024-09-20T22:42:28.386165+0000","last_clean":"2024-09-20T22:42:28.386165+0000","last_became_active":"2024-09-20T22:41:31.091135+0000","last_became_peered":"2024-09-20T22:41:31.091135+0000","last_unstale":"2024-09-20T22:42:28.386165+0000","last_undegraded":"2024-09-20T22:42:28.386165+0000","last_fullsized":"2024-09-20T22:42:28.386165+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'117","last_scrub_stamp":"2024-09-20T22:41:43.408644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.408644+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:48.125134+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.386384+0000","last_change":"2024-09-20T22:41:42.185135+0000","last_active":"2024-09-20T22:42:28.386384+0000","last_peered":"2024-09-20T22:42:28.386384+0000","last_clean":"2024-09-20T22:42:28.386384+0000","last_became_active":"2024-09-20T22:41:42.184387+0000","last_became_peered":"2024-09-20T22:41:42.184387+0000","last_unstale":"2024-09-20T22:42:28.386384+0000","last_undegraded":"2024-09-20T22:42:28.386384+0000","last_fullsized":"2024-09-20T22:42:28.386384+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:00.112877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":31,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.519320+0000","last_change":"2024-09-20T22:41:44.191113+0000","last_active":"2024-09-20T22:42:28.519320+0000","last_peered":"2024-09-20T22:42:28.519320+0000","last_clean":"2024-09-20T22:42:28.519320+0000","last_became_active":"2024-09-20T22:41:44.190658+0000","last_became_peered":"2024-09-20T22:41:44.190658+0000","last_unstale":"2024-09-20T22:42:28.519320+0000","last_undegraded":"2024-09-20T22:42:28.519320+0000","last_fullsized":"2024-09-20T22:42:28.519320+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:44.921190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"74'11","reported_seq":58,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.100206+0000","last_change":"2024-09-20T22:41:40.169542+0000","last_active":"2024-09-20T22:42:28.100206+0000","last_peered":"2024-09-20T22:42:28.100206+0000","last_clean":"2024-09-20T22:42:28.100206+0000","last_became_active":"2024-09-20T22:41:40.168942+0000","last_became_peered":"2024-09-20T22:41:40.168942+0000","last_unstale":"2024-09-20T22:42:28.100206+0000","last_undegraded":"2024-09-20T22:42:28.100206+0000","last_fullsized":"2024-09-20T22:42:28.100206+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:58.224721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":43,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.099711+0000","last_change":"2024-09-20T22:41:38.161801+0000","last_active":"2024-09-20T22:42:28.099711+0000","last_peered":"2024-09-20T22:42:28.099711+0000","last_clean":"2024-09-20T22:42:28.099711+0000","last_became_active":"2024-09-20T22:41:38.161500+0000","last_became_peered":"2024-09-20T22:41:38.161500+0000","last_unstale":"2024-09-20T22:42:28.099711+0000","last_undegraded":"2024-09-20T22:42:28.099711+0000","last_fullsized":"2024-09-20T22:42:28.099711+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:48.636045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.519276+0000","last_change":"2024-09-20T22:41:42.183601+0000","last_active":"2024-09-20T22:42:28.519276+0000","last_peered":"2024-09-20T22:42:28.519276+0000","last_clean":"2024-09-20T22:42:28.519276+0000","last_became_active":"2024-09-20T22:41:42.183302+0000","last_became_peered":"2024-09-20T22:41:42.183302+0000","last_unstale":"2024-09-20T22:42:28.519276+0000","last_undegraded":"2024-09-20T22:42:28.519276+0000","last_fullsized":"2024-09-20T22:42:28.519276+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:10.335337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":31,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.477770+0000","last_change":"2024-09-20T22:41:44.832295+0000","last_active":"2024-09-20T22:42:28.477770+0000","last_peered":"2024-09-20T22:42:28.477770+0000","last_clean":"2024-09-20T22:42:28.477770+0000","last_became_active":"2024-09-20T22:41:44.832009+0000","last_became_peered":"2024-09-20T22:41:44.832009+0000","last_unstale":"2024-09-20T22:42:28.477770+0000","last_undegraded":"2024-09-20T22:42:28.477770+0000","last_fullsized":"2024-09-20T22:42:28.477770+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:56.047489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"74'17","reported_seq":67,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.100327+0000","last_change":"2024-09-20T22:41:40.169498+0000","last_active":"2024-09-20T22:42:28.100327+0000","last_peered":"2024-09-20T22:42:28.100327+0000","last_clean":"2024-09-20T22:42:28.100327+0000","last_became_active":"2024-09-20T22:41:40.169192+0000","last_became_peered":"2024-09-20T22:41:40.169192+0000","last_unstale":"2024-09-20T22:42:28.100327+0000","last_undegraded":"2024-09-20T22:42:28.100327+0000","last_fullsized":"2024-09-20T22:42:28.100327+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:40.654708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":43,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.477418+0000","last_change":"2024-09-20T22:41:38.154220+0000","last_active":"2024-09-20T22:42:28.477418+0000","last_peered":"2024-09-20T22:42:28.477418+0000","last_clean":"2024-09-20T22:42:28.477418+0000","last_became_active":"2024-09-20T22:41:38.153926+0000","last_became_peered":"2024-09-20T22:41:38.153926+0000","last_unstale":"2024-09-20T22:42:28.477418+0000","last_undegraded":"2024-09-20T22:42:28.477418+0000","last_fullsized":"2024-09-20T22:42:28.477418+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:54.196177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.100876+0000","last_change":"2024-09-20T22:41:42.183482+0000","last_active":"2024-09-20T22:42:28.100876+0000","last_peered":"2024-09-20T22:42:28.100876+0000","last_clean":"2024-09-20T22:42:28.100876+0000","last_became_active":"2024-09-20T22:41:42.183225+0000","last_became_peered":"2024-09-20T22:41:42.183225+0000","last_unstale":"2024-09-20T22:42:28.100876+0000","last_undegraded":"2024-09-20T22:42:28.100876+0000","last_fullsized":"2024-09-20T22:42:28.100876+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:10.228360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":31,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.267176+0000","last_change":"2024-09-20T22:41:44.197855+0000","last_active":"2024-09-20T22:42:28.267176+0000","last_peered":"2024-09-20T22:42:28.267176+0000","last_clean":"2024-09-20T22:42:28.267176+0000","last_became_active":"2024-09-20T22:41:44.197561+0000","last_became_peered":"2024-09-20T22:41:44.197561+0000","last_unstale":"2024-09-20T22:42:28.267176+0000","last_undegraded":"2024-09-20T22:42:28.267176+0000","last_fullsized":"2024-09-20T22:42:28.267176+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:45.431229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"74'10","reported_seq":54,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.100016+0000","last_change":"2024-09-20T22:41:40.162973+0000","last_active":"2024-09-20T22:42:28.100016+0000","last_peered":"2024-09-20T22:42:28.100016+0000","last_clean":"2024-09-20T22:42:28.100016+0000","last_became_active":"2024-09-20T22:41:40.162494+0000","last_became_peered":"2024-09-20T22:41:40.162494+0000","last_unstale":"2024-09-20T22:42:28.100016+0000","last_undegraded":"2024-09-20T22:42:28.100016+0000","last_fullsized":"2024-09-20T22:42:28.100016+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:49.601652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":43,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.477895+0000","last_change":"2024-09-20T22:41:38.154861+0000","last_active":"2024-09-20T22:42:28.477895+0000","last_peered":"2024-09-20T22:42:28.477895+0000","last_clean":"2024-09-20T22:42:28.477895+0000","last_became_active":"2024-09-20T22:41:38.154636+0000","last_became_peered":"2024-09-20T22:41:38.154636+0000","last_unstale":"2024-09-20T22:42:28.477895+0000","last_undegraded":"2024-09-20T22:42:28.477895+0000","last_fullsized":"2024-09-20T22:42:28.477895+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:45.035202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"74'8","reported_seq":54,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.266877+0000","last_change":"2024-09-20T22:41:42.186528+0000","last_active":"2024-09-20T22:42:28.266877+0000","last_peered":"2024-09-20T22:42:28.266877+0000","last_clean":"2024-09-20T22:42:28.266877+0000","last_became_active":"2024-09-20T22:41:42.185863+0000","last_became_peered":"2024-09-20T22:41:42.185863+0000","last_unstale":"2024-09-20T22:42:28.266877+0000","last_undegraded":"2024-09-20T22:42:28.266877+0000","last_fullsized":"2024-09-20T22:42:28.266877+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:00.762928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":31,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.099871+0000","last_change":"2024-09-20T22:41:44.195767+0000","last_active":"2024-09-20T22:42:28.099871+0000","last_peered":"2024-09-20T22:42:28.099871+0000","last_clean":"2024-09-20T22:42:28.099871+0000","last_became_active":"2024-09-20T22:41:44.195160+0000","last_became_peered":"2024-09-20T22:41:44.195160+0000","last_unstale":"2024-09-20T22:42:28.099871+0000","last_undegraded":"2024-09-20T22:42:28.099871+0000","last_fullsized":"2024-09-20T22:42:28.099871+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:07.188986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"74'9","reported_seq":55,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.100241+0000","last_change":"2024-09-20T22:41:40.163774+0000","last_active":"2024-09-20T22:42:28.100241+0000","last_peered":"2024-09-20T22:42:28.100241+0000","last_clean":"2024-09-20T22:42:28.100241+0000","last_became_active":"2024-09-20T22:41:40.163495+0000","last_became_peered":"2024-09-20T22:41:40.163495+0000","last_unstale":"2024-09-20T22:42:28.100241+0000","last_undegraded":"2024-09-20T22:42:28.100241+0000","last_fullsized":"2024-09-20T22:42:28.100241+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:41.614897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":43,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.519191+0000","last_change":"2024-09-20T22:41:38.156470+0000","last_active":"2024-09-20T22:42:28.519191+0000","last_peered":"2024-09-20T22:42:28.519191+0000","last_clean":"2024-09-20T22:42:28.519191+0000","last_became_active":"2024-09-20T22:41:38.155953+0000","last_became_peered":"2024-09-20T22:41:38.155953+0000","last_unstale":"2024-09-20T22:42:28.519191+0000","last_undegraded":"2024-09-20T22:42:28.519191+0000","last_fullsized":"2024-09-20T22:42:28.519191+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:54.057798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.267149+0000","last_change":"2024-09-20T22:41:42.183269+0000","last_active":"2024-09-20T22:42:28.267149+0000","last_peered":"2024-09-20T22:42:28.267149+0000","last_clean":"2024-09-20T22:42:28.267149+0000","last_became_active":"2024-09-20T22:41:42.182999+0000","last_became_peered":"2024-09-20T22:41:42.182999+0000","last_unstale":"2024-09-20T22:42:28.267149+0000","last_undegraded":"2024-09-20T22:42:28.267149+0000","last_fullsized":"2024-09-20T22:42:28.267149+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:57.955307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":31,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.099253+0000","last_change":"2024-09-20T22:41:44.196969+0000","last_active":"2024-09-20T22:42:28.099253+0000","last_peered":"2024-09-20T22:42:28.099253+0000","last_clean":"2024-09-20T22:42:28.099253+0000","last_became_active":"2024-09-20T22:41:44.196633+0000","last_became_peered":"2024-09-20T22:41:44.196633+0000","last_unstale":"2024-09-20T22:42:28.099253+0000","last_undegraded":"2024-09-20T22:42:28.099253+0000","last_fullsized":"2024-09-20T22:42:28.099253+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:03.513323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"74'19","reported_seq":70,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.476645+0000","last_change":"2024-09-20T22:41:40.252962+0000","last_active":"2024-09-20T22:42:28.476645+0000","last_peered":"2024-09-20T22:42:28.476645+0000","last_clean":"2024-09-20T22:42:28.476645+0000","last_became_active":"2024-09-20T22:41:40.252011+0000","last_became_peered":"2024-09-20T22:41:40.252011+0000","last_unstale":"2024-09-20T22:42:28.476645+0000","last_undegraded":"2024-09-20T22:42:28.476645+0000","last_fullsized":"2024-09-20T22:42:28.476645+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:51.633432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":43,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.386630+0000","last_change":"2024-09-20T22:41:38.161800+0000","last_active":"2024-09-20T22:42:28.386630+0000","last_peered":"2024-09-20T22:42:28.386630+0000","last_clean":"2024-09-20T22:42:28.386630+0000","last_became_active":"2024-09-20T22:41:38.161524+0000","last_became_peered":"2024-09-20T22:41:38.161524+0000","last_unstale":"2024-09-20T22:42:28.386630+0000","last_undegraded":"2024-09-20T22:42:28.386630+0000","last_fullsized":"2024-09-20T22:42:28.386630+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:07.121209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.266881+0000","last_change":"2024-09-20T22:41:42.184605+0000","last_active":"2024-09-20T22:42:28.266881+0000","last_peered":"2024-09-20T22:42:28.266881+0000","last_clean":"2024-09-20T22:42:28.266881+0000","last_became_active":"2024-09-20T22:41:42.184342+0000","last_became_peered":"2024-09-20T22:41:42.184342+0000","last_unstale":"2024-09-20T22:42:28.266881+0000","last_undegraded":"2024-09-20T22:42:28.266881+0000","last_fullsized":"2024-09-20T22:42:28.266881+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:03.475959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":31,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.386113+0000","last_change":"2024-09-20T22:41:44.199874+0000","last_active":"2024-09-20T22:42:28.386113+0000","last_peered":"2024-09-20T22:42:28.386113+0000","last_clean":"2024-09-20T22:42:28.386113+0000","last_became_active":"2024-09-20T22:41:44.199395+0000","last_became_peered":"2024-09-20T22:41:44.199395+0000","last_unstale":"2024-09-20T22:42:28.386113+0000","last_undegraded":"2024-09-20T22:42:28.386113+0000","last_fullsized":"2024-09-20T22:42:28.386113+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:44.605488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"74'12","reported_seq":62,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.099597+0000","last_change":"2024-09-20T22:41:40.170746+0000","last_active":"2024-09-20T22:42:28.099597+0000","last_peered":"2024-09-20T22:42:28.099597+0000","last_clean":"2024-09-20T22:42:28.099597+0000","last_became_active":"2024-09-20T22:41:40.170133+0000","last_became_peered":"2024-09-20T22:41:40.170133+0000","last_unstale":"2024-09-20T22:42:28.099597+0000","last_undegraded":"2024-09-20T22:42:28.099597+0000","last_fullsized":"2024-09-20T22:42:28.099597+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:39.588868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":43,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.386606+0000","last_change":"2024-09-20T22:41:38.163612+0000","last_active":"2024-09-20T22:42:28.386606+0000","last_peered":"2024-09-20T22:42:28.386606+0000","last_clean":"2024-09-20T22:42:28.386606+0000","last_became_active":"2024-09-20T22:41:38.163292+0000","last_became_peered":"2024-09-20T22:41:38.163292+0000","last_unstale":"2024-09-20T22:42:28.386606+0000","last_undegraded":"2024-09-20T22:42:28.386606+0000","last_fullsized":"2024-09-20T22:42:28.386606+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:45.666357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.266571+0000","last_change":"2024-09-20T22:41:42.182102+0000","last_active":"2024-09-20T22:42:28.266571+0000","last_peered":"2024-09-20T22:42:28.266571+0000","last_clean":"2024-09-20T22:42:28.266571+0000","last_became_active":"2024-09-20T22:41:42.181811+0000","last_became_peered":"2024-09-20T22:41:42.181811+0000","last_unstale":"2024-09-20T22:42:28.266571+0000","last_undegraded":"2024-09-20T22:42:28.266571+0000","last_fullsized":"2024-09-20T22:42:28.266571+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:52.248246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":31,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.477729+0000","last_change":"2024-09-20T22:41:44.193432+0000","last_active":"2024-09-20T22:42:28.477729+0000","last_peered":"2024-09-20T22:42:28.477729+0000","last_clean":"2024-09-20T22:42:28.477729+0000","last_became_active":"2024-09-20T22:41:44.192609+0000","last_became_peered":"2024-09-20T22:41:44.192609+0000","last_unstale":"2024-09-20T22:42:28.477729+0000","last_undegraded":"2024-09-20T22:42:28.477729+0000","last_fullsized":"2024-09-20T22:42:28.477729+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:50.351142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"74'15","reported_seq":64,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.518696+0000","last_change":"2024-09-20T22:41:40.252427+0000","last_active":"2024-09-20T22:42:28.518696+0000","last_peered":"2024-09-20T22:42:28.518696+0000","last_clean":"2024-09-20T22:42:28.518696+0000","last_became_active":"2024-09-20T22:41:40.252113+0000","last_became_peered":"2024-09-20T22:41:40.252113+0000","last_unstale":"2024-09-20T22:42:28.518696+0000","last_undegraded":"2024-09-20T22:42:28.518696+0000","last_fullsized":"2024-09-20T22:42:28.518696+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:06.207870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'2","reported_seq":55,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.386646+0000","last_change":"2024-09-20T22:41:38.163593+0000","last_active":"2024-09-20T22:42:28.386646+0000","last_peered":"2024-09-20T22:42:28.386646+0000","last_clean":"2024-09-20T22:42:28.386646+0000","last_became_active":"2024-09-20T22:41:38.163219+0000","last_became_peered":"2024-09-20T22:41:38.163219+0000","last_unstale":"2024-09-20T22:42:28.386646+0000","last_undegraded":"2024-09-20T22:42:28.386646+0000","last_fullsized":"2024-09-20T22:42:28.386646+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:02.881623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"74'8","reported_seq":54,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.385953+0000","last_change":"2024-09-20T22:41:42.188223+0000","last_active":"2024-09-20T22:42:28.385953+0000","last_peered":"2024-09-20T22:42:28.385953+0000","last_clean":"2024-09-20T22:42:28.385953+0000","last_became_active":"2024-09-20T22:41:42.186635+0000","last_became_peered":"2024-09-20T22:41:42.186635+0000","last_unstale":"2024-09-20T22:42:28.385953+0000","last_undegraded":"2024-09-20T22:42:28.385953+0000","last_fullsized":"2024-09-20T22:42:28.385953+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:52.120303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":31,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.518852+0000","last_change":"2024-09-20T22:41:44.831435+0000","last_active":"2024-09-20T22:42:28.518852+0000","last_peered":"2024-09-20T22:42:28.518852+0000","last_clean":"2024-09-20T22:42:28.518852+0000","last_became_active":"2024-09-20T22:41:44.830556+0000","last_became_peered":"2024-09-20T22:41:44.830556+0000","last_unstale":"2024-09-20T22:42:28.518852+0000","last_undegraded":"2024-09-20T22:42:28.518852+0000","last_fullsized":"2024-09-20T22:42:28.518852+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:08.027826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":43,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.476997+0000","last_change":"2024-09-20T22:41:38.159437+0000","last_active":"2024-09-20T22:42:28.476997+0000","last_peered":"2024-09-20T22:42:28.476997+0000","last_clean":"2024-09-20T22:42:28.476997+0000","last_became_active":"2024-09-20T22:41:38.159139+0000","last_became_peered":"2024-09-20T22:41:38.159139+0000","last_unstale":"2024-09-20T22:42:28.476997+0000","last_undegraded":"2024-09-20T22:42:28.476997+0000","last_fullsized":"2024-09-20T22:42:28.476997+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:47.658845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"74'6","reported_seq":48,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.476969+0000","last_change":"2024-09-20T22:41:40.252310+0000","last_active":"2024-09-20T22:42:28.476969+0000","last_peered":"2024-09-20T22:42:28.476969+0000","last_clean":"2024-09-20T22:42:28.476969+0000","last_became_active":"2024-09-20T22:41:40.251867+0000","last_became_peered":"2024-09-20T22:41:40.251867+0000","last_unstale":"2024-09-20T22:42:28.476969+0000","last_undegraded":"2024-09-20T22:42:28.476969+0000","last_fullsized":"2024-09-20T22:42:28.476969+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:40.015664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.519471+0000","last_change":"2024-09-20T22:41:42.185218+0000","last_active":"2024-09-20T22:42:28.519471+0000","last_peered":"2024-09-20T22:42:28.519471+0000","last_clean":"2024-09-20T22:42:28.519471+0000","last_became_active":"2024-09-20T22:41:42.184930+0000","last_became_peered":"2024-09-20T22:41:42.184930+0000","last_unstale":"2024-09-20T22:42:28.519471+0000","last_undegraded":"2024-09-20T22:42:28.519471+0000","last_fullsized":"2024-09-20T22:42:28.519471+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:47.818365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"74'1","reported_seq":32,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.386326+0000","last_change":"2024-09-20T22:41:44.832085+0000","last_active":"2024-09-20T22:42:28.386326+0000","last_peered":"2024-09-20T22:42:28.386326+0000","last_clean":"2024-09-20T22:42:28.386326+0000","last_became_active":"2024-09-20T22:41:44.831805+0000","last_became_peered":"2024-09-20T22:41:44.831805+0000","last_unstale":"2024-09-20T22:42:28.386326+0000","last_undegraded":"2024-09-20T22:42:28.386326+0000","last_fullsized":"2024-09-20T22:42:28.386326+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:54.445212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"74'9","reported_seq":55,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.101282+0000","last_change":"2024-09-20T22:41:40.253633+0000","last_active":"2024-09-20T22:42:28.101282+0000","last_peered":"2024-09-20T22:42:28.101282+0000","last_clean":"2024-09-20T22:42:28.101282+0000","last_became_active":"2024-09-20T22:41:40.253280+0000","last_became_peered":"2024-09-20T22:41:40.253280+0000","last_unstale":"2024-09-20T22:42:28.101282+0000","last_undegraded":"2024-09-20T22:42:28.101282+0000","last_fullsized":"2024-09-20T22:42:28.101282+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:39.928882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":43,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.386175+0000","last_change":"2024-09-20T22:41:38.156773+0000","last_active":"2024-09-20T22:42:28.386175+0000","last_peered":"2024-09-20T22:42:28.386175+0000","last_clean":"2024-09-20T22:42:28.386175+0000","last_became_active":"2024-09-20T22:41:38.156497+0000","last_became_peered":"2024-09-20T22:41:38.156497+0000","last_unstale":"2024-09-20T22:42:28.386175+0000","last_undegraded":"2024-09-20T22:42:28.386175+0000","last_fullsized":"2024-09-20T22:42:28.386175+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:01.943269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.478147+0000","last_change":"2024-09-20T22:41:42.188166+0000","last_active":"2024-09-20T22:42:28.478147+0000","last_peered":"2024-09-20T22:42:28.478147+0000","last_clean":"2024-09-20T22:42:28.478147+0000","last_became_active":"2024-09-20T22:41:42.187706+0000","last_became_peered":"2024-09-20T22:41:42.187706+0000","last_unstale":"2024-09-20T22:42:28.478147+0000","last_undegraded":"2024-09-20T22:42:28.478147+0000","last_fullsized":"2024-09-20T22:42:28.478147+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:47.610076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":31,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.267029+0000","last_change":"2024-09-20T22:41:44.201642+0000","last_active":"2024-09-20T22:42:28.267029+0000","last_peered":"2024-09-20T22:42:28.267029+0000","last_clean":"2024-09-20T22:42:28.267029+0000","last_became_active":"2024-09-20T22:41:44.201433+0000","last_became_peered":"2024-09-20T22:41:44.201433+0000","last_unstale":"2024-09-20T22:42:28.267029+0000","last_undegraded":"2024-09-20T22:42:28.267029+0000","last_fullsized":"2024-09-20T22:42:28.267029+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:55.257671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"74'9","reported_seq":55,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.519182+0000","last_change":"2024-09-20T22:41:40.252840+0000","last_active":"2024-09-20T22:42:28.519182+0000","last_peered":"2024-09-20T22:42:28.519182+0000","last_clean":"2024-09-20T22:42:28.519182+0000","last_became_active":"2024-09-20T22:41:40.252486+0000","last_became_peered":"2024-09-20T22:41:40.252486+0000","last_unstale":"2024-09-20T22:42:28.519182+0000","last_undegraded":"2024-09-20T22:42:28.519182+0000","last_fullsized":"2024-09-20T22:42:28.519182+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:51.443985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":43,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.099861+0000","last_change":"2024-09-20T22:41:38.158479+0000","last_active":"2024-09-20T22:42:28.099861+0000","last_peered":"2024-09-20T22:42:28.099861+0000","last_clean":"2024-09-20T22:42:28.099861+0000","last_became_active":"2024-09-20T22:41:38.158146+0000","last_became_peered":"2024-09-20T22:41:38.158146+0000","last_unstale":"2024-09-20T22:42:28.099861+0000","last_undegraded":"2024-09-20T22:42:28.099861+0000","last_fullsized":"2024-09-20T22:42:28.099861+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:55.658232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"74'8","reported_seq":54,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.477829+0000","last_change":"2024-09-20T22:41:42.186630+0000","last_active":"2024-09-20T22:42:28.477829+0000","last_peered":"2024-09-20T22:42:28.477829+0000","last_clean":"2024-09-20T22:42:28.477829+0000","last_became_active":"2024-09-20T22:41:42.186352+0000","last_became_peered":"2024-09-20T22:41:42.186352+0000","last_unstale":"2024-09-20T22:42:28.477829+0000","last_undegraded":"2024-09-20T22:42:28.477829+0000","last_fullsized":"2024-09-20T22:42:28.477829+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:48.572027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":31,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.100029+0000","last_change":"2024-09-20T22:41:44.194510+0000","last_active":"2024-09-20T22:42:28.100029+0000","last_peered":"2024-09-20T22:42:28.100029+0000","last_clean":"2024-09-20T22:42:28.100029+0000","last_became_active":"2024-09-20T22:41:44.194053+0000","last_became_peered":"2024-09-20T22:41:44.194053+0000","last_unstale":"2024-09-20T22:42:28.100029+0000","last_undegraded":"2024-09-20T22:42:28.100029+0000","last_fullsized":"2024-09-20T22:42:28.100029+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:01.207219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"74'10","reported_seq":54,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.476803+0000","last_change":"2024-09-20T22:41:40.253880+0000","last_active":"2024-09-20T22:42:28.476803+0000","last_peered":"2024-09-20T22:42:28.476803+0000","last_clean":"2024-09-20T22:42:28.476803+0000","last_became_active":"2024-09-20T22:41:40.253614+0000","last_became_peered":"2024-09-20T22:41:40.253614+0000","last_unstale":"2024-09-20T22:42:28.476803+0000","last_undegraded":"2024-09-20T22:42:28.476803+0000","last_fullsized":"2024-09-20T22:42:28.476803+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:47.554109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":43,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.386327+0000","last_change":"2024-09-20T22:41:38.158008+0000","last_active":"2024-09-20T22:42:28.386327+0000","last_peered":"2024-09-20T22:42:28.386327+0000","last_clean":"2024-09-20T22:42:28.386327+0000","last_became_active":"2024-09-20T22:41:38.157647+0000","last_became_peered":"2024-09-20T22:41:38.157647+0000","last_unstale":"2024-09-20T22:42:28.386327+0000","last_undegraded":"2024-09-20T22:42:28.386327+0000","last_fullsized":"2024-09-20T22:42:28.386327+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:57.980474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"74'8","reported_seq":54,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.519065+0000","last_change":"2024-09-20T22:41:42.189757+0000","last_active":"2024-09-20T22:42:28.519065+0000","last_peered":"2024-09-20T22:42:28.519065+0000","last_clean":"2024-09-20T22:42:28.519065+0000","last_became_active":"2024-09-20T22:41:42.189451+0000","last_became_peered":"2024-09-20T22:41:42.189451+0000","last_unstale":"2024-09-20T22:42:28.519065+0000","last_undegraded":"2024-09-20T22:42:28.519065+0000","last_fullsized":"2024-09-20T22:42:28.519065+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:43.846079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":31,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.100733+0000","last_change":"2024-09-20T22:41:44.191549+0000","last_active":"2024-09-20T22:42:28.100733+0000","last_peered":"2024-09-20T22:42:28.100733+0000","last_clean":"2024-09-20T22:42:28.100733+0000","last_became_active":"2024-09-20T22:41:44.191206+0000","last_became_peered":"2024-09-20T22:41:44.191206+0000","last_unstale":"2024-09-20T22:42:28.100733+0000","last_undegraded":"2024-09-20T22:42:28.100733+0000","last_fullsized":"2024-09-20T22:42:28.100733+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:58.564765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"74'11","reported_seq":58,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.099734+0000","last_change":"2024-09-20T22:41:40.171449+0000","last_active":"2024-09-20T22:42:28.099734+0000","last_peered":"2024-09-20T22:42:28.099734+0000","last_clean":"2024-09-20T22:42:28.099734+0000","last_became_active":"2024-09-20T22:41:40.170035+0000","last_became_peered":"2024-09-20T22:41:40.170035+0000","last_unstale":"2024-09-20T22:42:28.099734+0000","last_undegraded":"2024-09-20T22:42:28.099734+0000","last_fullsized":"2024-09-20T22:42:28.099734+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:49.140455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":43,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.100170+0000","last_change":"2024-09-20T22:41:38.161385+0000","last_active":"2024-09-20T22:42:28.100170+0000","last_peered":"2024-09-20T22:42:28.100170+0000","last_clean":"2024-09-20T22:42:28.100170+0000","last_became_active":"2024-09-20T22:41:38.158142+0000","last_became_peered":"2024-09-20T22:41:38.158142+0000","last_unstale":"2024-09-20T22:42:28.100170+0000","last_undegraded":"2024-09-20T22:42:28.100170+0000","last_fullsized":"2024-09-20T22:42:28.100170+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:51.728688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.100124+0000","last_change":"2024-09-20T22:41:42.182119+0000","last_active":"2024-09-20T22:42:28.100124+0000","last_peered":"2024-09-20T22:42:28.100124+0000","last_clean":"2024-09-20T22:42:28.100124+0000","last_became_active":"2024-09-20T22:41:42.181736+0000","last_became_peered":"2024-09-20T22:41:42.181736+0000","last_unstale":"2024-09-20T22:42:28.100124+0000","last_undegraded":"2024-09-20T22:42:28.100124+0000","last_fullsized":"2024-09-20T22:42:28.100124+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:02.572377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":31,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.477331+0000","last_change":"2024-09-20T22:41:44.193289+0000","last_active":"2024-09-20T22:42:28.477331+0000","last_peered":"2024-09-20T22:42:28.477331+0000","last_clean":"2024-09-20T22:42:28.477331+0000","last_became_active":"2024-09-20T22:41:44.192960+0000","last_became_peered":"2024-09-20T22:41:44.192960+0000","last_unstale":"2024-09-20T22:42:28.477331+0000","last_undegraded":"2024-09-20T22:42:28.477331+0000","last_fullsized":"2024-09-20T22:42:28.477331+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:49.189046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"74'9","reported_seq":55,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.100394+0000","last_change":"2024-09-20T22:41:40.165116+0000","last_active":"2024-09-20T22:42:28.100394+0000","last_peered":"2024-09-20T22:42:28.100394+0000","last_clean":"2024-09-20T22:42:28.100394+0000","last_became_active":"2024-09-20T22:41:40.164796+0000","last_became_peered":"2024-09-20T22:41:40.164796+0000","last_unstale":"2024-09-20T22:42:28.100394+0000","last_undegraded":"2024-09-20T22:42:28.100394+0000","last_fullsized":"2024-09-20T22:42:28.100394+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:48.483009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":43,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.386534+0000","last_change":"2024-09-20T22:41:38.159941+0000","last_active":"2024-09-20T22:42:28.386534+0000","last_peered":"2024-09-20T22:42:28.386534+0000","last_clean":"2024-09-20T22:42:28.386534+0000","last_became_active":"2024-09-20T22:41:38.159657+0000","last_became_peered":"2024-09-20T22:41:38.159657+0000","last_unstale":"2024-09-20T22:42:28.386534+0000","last_undegraded":"2024-09-20T22:42:28.386534+0000","last_fullsized":"2024-09-20T22:42:28.386534+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:53.225190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.477809+0000","last_change":"2024-09-20T22:41:42.183254+0000","last_active":"2024-09-20T22:42:28.477809+0000","last_peered":"2024-09-20T22:42:28.477809+0000","last_clean":"2024-09-20T22:42:28.477809+0000","last_became_active":"2024-09-20T22:41:42.182966+0000","last_became_peered":"2024-09-20T22:41:42.182966+0000","last_unstale":"2024-09-20T22:42:28.477809+0000","last_undegraded":"2024-09-20T22:42:28.477809+0000","last_fullsized":"2024-09-20T22:42:28.477809+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:51.689726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":31,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.099148+0000","last_change":"2024-09-20T22:41:44.194845+0000","last_active":"2024-09-20T22:42:28.099148+0000","last_peered":"2024-09-20T22:42:28.099148+0000","last_clean":"2024-09-20T22:42:28.099148+0000","last_became_active":"2024-09-20T22:41:44.194551+0000","last_became_peered":"2024-09-20T22:41:44.194551+0000","last_unstale":"2024-09-20T22:42:28.099148+0000","last_undegraded":"2024-09-20T22:42:28.099148+0000","last_fullsized":"2024-09-20T22:42:28.099148+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:45.187444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"74'11","reported_seq":58,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.477453+0000","last_change":"2024-09-20T22:41:40.252149+0000","last_active":"2024-09-20T22:42:28.477453+0000","last_peered":"2024-09-20T22:42:28.477453+0000","last_clean":"2024-09-20T22:42:28.477453+0000","last_became_active":"2024-09-20T22:41:40.251829+0000","last_became_peered":"2024-09-20T22:41:40.251829+0000","last_unstale":"2024-09-20T22:42:28.477453+0000","last_undegraded":"2024-09-20T22:42:28.477453+0000","last_fullsized":"2024-09-20T22:42:28.477453+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:56.020491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":43,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.519509+0000","last_change":"2024-09-20T22:41:38.159064+0000","last_active":"2024-09-20T22:42:28.519509+0000","last_peered":"2024-09-20T22:42:28.519509+0000","last_clean":"2024-09-20T22:42:28.519509+0000","last_became_active":"2024-09-20T22:41:38.158701+0000","last_became_peered":"2024-09-20T22:41:38.158701+0000","last_unstale":"2024-09-20T22:42:28.519509+0000","last_undegraded":"2024-09-20T22:42:28.519509+0000","last_fullsized":"2024-09-20T22:42:28.519509+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:47.727493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.385845+0000","last_change":"2024-09-20T22:41:42.185991+0000","last_active":"2024-09-20T22:42:28.385845+0000","last_peered":"2024-09-20T22:42:28.385845+0000","last_clean":"2024-09-20T22:42:28.385845+0000","last_became_active":"2024-09-20T22:41:42.185296+0000","last_became_peered":"2024-09-20T22:41:42.185296+0000","last_unstale":"2024-09-20T22:42:28.385845+0000","last_undegraded":"2024-09-20T22:42:28.385845+0000","last_fullsized":"2024-09-20T22:42:28.385845+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:55.259880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":31,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.477860+0000","last_change":"2024-09-20T22:41:44.832293+0000","last_active":"2024-09-20T22:42:28.477860+0000","last_peered":"2024-09-20T22:42:28.477860+0000","last_clean":"2024-09-20T22:42:28.477860+0000","last_became_active":"2024-09-20T22:41:44.831967+0000","last_became_peered":"2024-09-20T22:41:44.831967+0000","last_unstale":"2024-09-20T22:42:28.477860+0000","last_undegraded":"2024-09-20T22:42:28.477860+0000","last_fullsized":"2024-09-20T22:42:28.477860+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:03.371689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"74'4","reported_seq":45,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.476697+0000","last_change":"2024-09-20T22:41:40.166409+0000","last_active":"2024-09-20T22:42:28.476697+0000","last_peered":"2024-09-20T22:42:28.476697+0000","last_clean":"2024-09-20T22:42:28.476697+0000","last_became_active":"2024-09-20T22:41:40.166123+0000","last_became_peered":"2024-09-20T22:41:40.166123+0000","last_unstale":"2024-09-20T22:42:28.476697+0000","last_undegraded":"2024-09-20T22:42:28.476697+0000","last_fullsized":"2024-09-20T22:42:28.476697+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:52.943341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":43,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.101609+0000","last_change":"2024-09-20T22:41:38.158861+0000","last_active":"2024-09-20T22:42:28.101609+0000","last_peered":"2024-09-20T22:42:28.101609+0000","last_clean":"2024-09-20T22:42:28.101609+0000","last_became_active":"2024-09-20T22:41:38.158347+0000","last_became_peered":"2024-09-20T22:41:38.158347+0000","last_unstale":"2024-09-20T22:42:28.101609+0000","last_undegraded":"2024-09-20T22:42:28.101609+0000","last_fullsized":"2024-09-20T22:42:28.101609+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:44.061004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.477085+0000","last_change":"2024-09-20T22:41:42.187707+0000","last_active":"2024-09-20T22:42:28.477085+0000","last_peered":"2024-09-20T22:42:28.477085+0000","last_clean":"2024-09-20T22:42:28.477085+0000","last_became_active":"2024-09-20T22:41:42.186068+0000","last_became_peered":"2024-09-20T22:41:42.186068+0000","last_unstale":"2024-09-20T22:42:28.477085+0000","last_undegraded":"2024-09-20T22:42:28.477085+0000","last_fullsized":"2024-09-20T22:42:28.477085+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:50.526395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":31,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.385995+0000","last_change":"2024-09-20T22:41:44.194011+0000","last_active":"2024-09-20T22:42:28.385995+0000","last_peered":"2024-09-20T22:42:28.385995+0000","last_clean":"2024-09-20T22:42:28.385995+0000","last_became_active":"2024-09-20T22:41:44.193706+0000","last_became_peered":"2024-09-20T22:41:44.193706+0000","last_unstale":"2024-09-20T22:42:28.385995+0000","last_undegraded":"2024-09-20T22:42:28.385995+0000","last_fullsized":"2024-09-20T22:42:28.385995+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:49.835925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":31,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.100377+0000","last_change":"2024-09-20T22:41:44.198154+0000","last_active":"2024-09-20T22:42:28.100377+0000","last_peered":"2024-09-20T22:42:28.100377+0000","last_clean":"2024-09-20T22:42:28.100377+0000","last_became_active":"2024-09-20T22:41:44.197855+0000","last_became_peered":"2024-09-20T22:41:44.197855+0000","last_unstale":"2024-09-20T22:42:28.100377+0000","last_undegraded":"2024-09-20T22:42:28.100377+0000","last_fullsized":"2024-09-20T22:42:28.100377+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:56.991123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":43,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.477270+0000","last_change":"2024-09-20T22:41:38.155001+0000","last_active":"2024-09-20T22:42:28.477270+0000","last_peered":"2024-09-20T22:42:28.477270+0000","last_clean":"2024-09-20T22:42:28.477270+0000","last_became_active":"2024-09-20T22:41:38.154703+0000","last_became_peered":"2024-09-20T22:41:38.154703+0000","last_unstale":"2024-09-20T22:42:28.477270+0000","last_undegraded":"2024-09-20T22:42:28.477270+0000","last_fullsized":"2024-09-20T22:42:28.477270+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:44.782808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"74'11","reported_seq":58,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.476930+0000","last_change":"2024-09-20T22:41:40.171840+0000","last_active":"2024-09-20T22:42:28.476930+0000","last_peered":"2024-09-20T22:42:28.476930+0000","last_clean":"2024-09-20T22:42:28.476930+0000","last_became_active":"2024-09-20T22:41:40.171256+0000","last_became_peered":"2024-09-20T22:41:40.171256+0000","last_unstale":"2024-09-20T22:42:28.476930+0000","last_undegraded":"2024-09-20T22:42:28.476930+0000","last_fullsized":"2024-09-20T22:42:28.476930+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:53.884281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-20T22:42:28.100928+0000","last_change":"2024-09-20T22:41:42.178413+0000","last_active":"2024-09-20T22:42:28.100928+0000","last_peered":"2024-09-20T22:42:28.100928+0000","last_clean":"2024-09-20T22:42:28.100928+0000","last_became_active":"2024-09-20T22:41:42.178157+0000","last_became_peered":"2024-09-20T22:41:42.178157+0000","last_unstale":"2024-09-20T22:42:28.100928+0000","last_undegraded":"2024-09-20T22:42:28.100928+0000","last_fullsized":"2024-09-20T22:42:28.100928+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:01.837489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":393,"ondisk_log_size":393,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070478,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32472,"kb_used_data":1284,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712680,"statfs":{"total":95995035648,"available":95961784320,"internally_reserved":0,"allocated":1314816,"data_stored":899197,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430228,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32456,"kb_used_data":1268,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712696,"statfs":{"total":95995035648,"available":95961800704,"internally_reserved":0,"allocated":1298432,"data_stored":899105,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691865,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27720,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717432,"statfs":{"total":95995035648,"available":95966650368,"internally_reserved":0,"allocated":708608,"data_stored":309519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986206,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27740,"kb_used_data":712,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717412,"statfs":{"total":95995035648,"available":95966629888,"internally_reserved":0,"allocated":729088,"data_stored":308349,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182436,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27748,"kb_used_data":720,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717404,"statfs":{"total":95995035648,"available":95966621696,"internally_reserved":0,"allocated":737280,"data_stored":309615,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411369,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31868,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713284,"statfs":{"total":95995035648,"available":95962402816,"internally_reserved":0,"allocated":696320,"data_stored":308315,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574894,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27772,"kb_used_data":744,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717380,"statfs":{"total":95995035648,"available":95966597120,"internally_reserved":0,"allocated":761856,"data_stored":309516,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738419,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32476,"kb_used_data":1288,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712676,"statfs":{"total":95995035648,"available":95961780224,"internally_reserved":0,"allocated":1318912,"data_stored":899329,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-20T22:42:33.869 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-09-20T22:42:33.869 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd dump 2024-09-20T22:42:34.114 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:42:34.364 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:34 smithi084 ceph-mon[40024]: from='client.14763 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T22:42:34.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:34 smithi084 ceph-mon[29774]: from='client.14763 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T22:42:34.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:34 smithi154 ceph-mon[33264]: from='client.14763 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T22:42:34.726 INFO:teuthology.orchestra.run.smithi084.stdout:epoch 84 2024-09-20T22:42:34.726 INFO:teuthology.orchestra.run.smithi084.stdout:fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 2024-09-20T22:42:34.726 INFO:teuthology.orchestra.run.smithi084.stdout:created 2024-09-20T22:36:30.240626+0000 2024-09-20T22:42:34.727 INFO:teuthology.orchestra.run.smithi084.stdout:modified 2024-09-20T22:42:31.603160+0000 2024-09-20T22:42:34.727 INFO:teuthology.orchestra.run.smithi084.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-09-20T22:42:34.727 INFO:teuthology.orchestra.run.smithi084.stdout:crush_version 18 2024-09-20T22:42:34.727 INFO:teuthology.orchestra.run.smithi084.stdout:full_ratio 0.95 2024-09-20T22:42:34.727 INFO:teuthology.orchestra.run.smithi084.stdout:backfillfull_ratio 0.9 2024-09-20T22:42:34.727 INFO:teuthology.orchestra.run.smithi084.stdout:nearfull_ratio 0.85 2024-09-20T22:42:34.727 INFO:teuthology.orchestra.run.smithi084.stdout:require_min_compat_client luminous 2024-09-20T22:42:34.727 INFO:teuthology.orchestra.run.smithi084.stdout:min_compat_client jewel 2024-09-20T22:42:34.727 INFO:teuthology.orchestra.run.smithi084.stdout:require_osd_release squid 2024-09-20T22:42:34.727 INFO:teuthology.orchestra.run.smithi084.stdout:stretch_mode_enabled false 2024-09-20T22:42:34.727 INFO:teuthology.orchestra.run.smithi084.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 21 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 7.89 2024-09-20T22:42:34.727 INFO:teuthology.orchestra.run.smithi084.stdout:pool 2 'datapool' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 3 pgp_num 3 autoscale_mode off last_change 69 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd read_balance_score 2.65 2024-09-20T22:42:34.727 INFO:teuthology.orchestra.run.smithi084.stdout:pool 3 '.rgw.root' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 68 flags hashpspool stripe_width 0 application rgw read_balance_score 1.50 2024-09-20T22:42:34.728 INFO:teuthology.orchestra.run.smithi084.stdout:pool 4 'default.rgw.log' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 70 flags hashpspool stripe_width 0 application rgw read_balance_score 2.25 2024-09-20T22:42:34.728 INFO:teuthology.orchestra.run.smithi084.stdout:pool 5 'default.rgw.control' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 72 flags hashpspool stripe_width 0 application rgw read_balance_score 1.25 2024-09-20T22:42:34.728 INFO:teuthology.orchestra.run.smithi084.stdout:pool 6 'default.rgw.meta' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 74 flags hashpspool stripe_width 0 pg_autoscale_bias 4 application rgw read_balance_score 1.75 2024-09-20T22:42:34.728 INFO:teuthology.orchestra.run.smithi084.stdout:max_osd 8 2024-09-20T22:42:34.728 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0 up in weight 1 up_from 8 up_thru 72 down_at 0 last_clean_interval [0,0) v2:172.21.15.84:6801/268084626 v2:172.21.15.84:6802/268084626 exists,up ea36a489-6f91-4b2f-b784-b90506b9805d 2024-09-20T22:42:34.728 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1 up in weight 1 up_from 13 up_thru 72 down_at 0 last_clean_interval [0,0) v2:172.21.15.84:6805/418374136 v2:172.21.15.84:6806/418374136 exists,up c8649d2f-27ef-47f8-bc94-f1cd9e62bdb8 2024-09-20T22:42:34.728 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2 up in weight 1 up_from 18 up_thru 72 down_at 0 last_clean_interval [0,0) v2:172.21.15.84:6809/3977168611 v2:172.21.15.84:6810/3977168611 exists,up c728eb48-a0f0-4de4-a651-7f3bec867280 2024-09-20T22:42:34.728 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3 up in weight 1 up_from 25 up_thru 72 down_at 0 last_clean_interval [0,0) v2:172.21.15.84:6813/2192239945 v2:172.21.15.84:6814/2192239945 exists,up 573f44eb-0853-4630-a7d8-9caaeb256f9b 2024-09-20T22:42:34.728 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4 up in weight 1 up_from 31 up_thru 72 down_at 0 last_clean_interval [0,0) v2:172.21.15.154:6800/1793136002 v2:172.21.15.154:6801/1793136002 exists,up a52c3b94-c5f3-490f-8ac8-09061b9b2abb 2024-09-20T22:42:34.728 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5 up in weight 1 up_from 40 up_thru 72 down_at 0 last_clean_interval [0,0) v2:172.21.15.154:6804/493899927 v2:172.21.15.154:6805/493899927 exists,up 4fa0dc8e-25cc-46cd-96ff-33f20b6faec3 2024-09-20T22:42:34.728 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6 up in weight 1 up_from 48 up_thru 70 down_at 0 last_clean_interval [0,0) v2:172.21.15.154:6808/2375091033 v2:172.21.15.154:6809/2375091033 exists,up 56788d74-810e-4827-a615-958c12f8a368 2024-09-20T22:42:34.728 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7 up in weight 1 up_from 59 up_thru 72 down_at 0 last_clean_interval [0,0) v2:172.21.15.154:6812/3717704491 v2:172.21.15.154:6813/3717704491 exists,up 83503727-afc1-4b95-8b96-ecb63f3338c2 2024-09-20T22:42:34.728 INFO:teuthology.orchestra.run.smithi084.stdout:blocklist 172.21.15.84:0/542020545 expires 2024-09-21T22:42:07.596045+0000 2024-09-20T22:42:34.728 INFO:teuthology.orchestra.run.smithi084.stdout:blocklist 172.21.15.84:0/25873236 expires 2024-09-21T22:36:54.917812+0000 2024-09-20T22:42:34.729 INFO:teuthology.orchestra.run.smithi084.stdout:blocklist 172.21.15.84:6800/820908470 expires 2024-09-21T22:36:54.917812+0000 2024-09-20T22:42:34.729 INFO:teuthology.orchestra.run.smithi084.stdout:blocklist 172.21.15.84:0/567614908 expires 2024-09-21T22:36:54.917812+0000 2024-09-20T22:42:34.729 INFO:teuthology.orchestra.run.smithi084.stdout:blocklist 172.21.15.84:0/2963622459 expires 2024-09-21T22:42:07.596045+0000 2024-09-20T22:42:34.729 INFO:teuthology.orchestra.run.smithi084.stdout:blocklist 172.21.15.84:0/1176849276 expires 2024-09-21T22:37:17.690104+0000 2024-09-20T22:42:34.729 INFO:teuthology.orchestra.run.smithi084.stdout:blocklist 172.21.15.84:6800/2932420050 expires 2024-09-21T22:37:17.690104+0000 2024-09-20T22:42:34.729 INFO:teuthology.orchestra.run.smithi084.stdout:blocklist 172.21.15.84:0/3879263635 expires 2024-09-21T22:37:17.690104+0000 2024-09-20T22:42:34.729 INFO:teuthology.orchestra.run.smithi084.stdout:blocklist 172.21.15.84:0/969472279 expires 2024-09-21T22:36:54.917812+0000 2024-09-20T22:42:34.729 INFO:teuthology.orchestra.run.smithi084.stdout:blocklist 172.21.15.84:0/1361143214 expires 2024-09-21T22:37:17.690104+0000 2024-09-20T22:42:34.729 INFO:teuthology.orchestra.run.smithi084.stdout:blocklist 172.21.15.84:0/3353126955 expires 2024-09-21T22:42:07.596045+0000 2024-09-20T22:42:34.729 INFO:teuthology.orchestra.run.smithi084.stdout:blocklist 172.21.15.84:6800/4024278029 expires 2024-09-21T22:42:07.596045+0000 2024-09-20T22:42:34.729 INFO:teuthology.orchestra.run.smithi084.stdout:blocklist 172.21.15.84:0/547902262 expires 2024-09-21T22:42:07.596045+0000 2024-09-20T22:42:34.729 INFO:teuthology.orchestra.run.smithi084.stdout:blocklist 172.21.15.84:0/1242929317 expires 2024-09-21T22:42:07.596045+0000 2024-09-20T22:42:35.476 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:35 smithi084 ceph-mon[40024]: pgmap v23: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:42:35.476 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:35 smithi084 ceph-mon[40024]: purged_snaps scrub starts 2024-09-20T22:42:35.477 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:35 smithi084 ceph-mon[40024]: purged_snaps scrub ok 2024-09-20T22:42:35.477 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:35 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/4110958433' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-09-20T22:42:35.477 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:35 smithi084 ceph-mon[29774]: pgmap v23: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:42:35.477 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:35 smithi084 ceph-mon[29774]: purged_snaps scrub starts 2024-09-20T22:42:35.477 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:35 smithi084 ceph-mon[29774]: purged_snaps scrub ok 2024-09-20T22:42:35.477 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:35 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/4110958433' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-09-20T22:42:35.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:35 smithi154 ceph-mon[33264]: pgmap v23: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:42:35.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:35 smithi154 ceph-mon[33264]: purged_snaps scrub starts 2024-09-20T22:42:35.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:35 smithi154 ceph-mon[33264]: purged_snaps scrub ok 2024-09-20T22:42:35.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:35 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/4110958433' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-09-20T22:42:35.507 INFO:tasks.cephadm.ceph_manager.ceph:['osd.0 up in weight 1 up_from 8 up_thru 72 down_at 0 last_clean_interval [0,0) v2:172.21.15.84:6801/268084626 v2:172.21.15.84:6802/268084626 exists,up ea36a489-6f91-4b2f-b784-b90506b9805d', 'osd.1 up in weight 1 up_from 13 up_thru 72 down_at 0 last_clean_interval [0,0) v2:172.21.15.84:6805/418374136 v2:172.21.15.84:6806/418374136 exists,up c8649d2f-27ef-47f8-bc94-f1cd9e62bdb8', 'osd.2 up in weight 1 up_from 18 up_thru 72 down_at 0 last_clean_interval [0,0) v2:172.21.15.84:6809/3977168611 v2:172.21.15.84:6810/3977168611 exists,up c728eb48-a0f0-4de4-a651-7f3bec867280', 'osd.3 up in weight 1 up_from 25 up_thru 72 down_at 0 last_clean_interval [0,0) v2:172.21.15.84:6813/2192239945 v2:172.21.15.84:6814/2192239945 exists,up 573f44eb-0853-4630-a7d8-9caaeb256f9b', 'osd.4 up in weight 1 up_from 31 up_thru 72 down_at 0 last_clean_interval [0,0) v2:172.21.15.154:6800/1793136002 v2:172.21.15.154:6801/1793136002 exists,up a52c3b94-c5f3-490f-8ac8-09061b9b2abb', 'osd.5 up in weight 1 up_from 40 up_thru 72 down_at 0 last_clean_interval [0,0) v2:172.21.15.154:6804/493899927 v2:172.21.15.154:6805/493899927 exists,up 4fa0dc8e-25cc-46cd-96ff-33f20b6faec3', 'osd.6 up in weight 1 up_from 48 up_thru 70 down_at 0 last_clean_interval [0,0) v2:172.21.15.154:6808/2375091033 v2:172.21.15.154:6809/2375091033 exists,up 56788d74-810e-4827-a615-958c12f8a368', 'osd.7 up in weight 1 up_from 59 up_thru 72 down_at 0 last_clean_interval [0,0) v2:172.21.15.154:6812/3717704491 v2:172.21.15.154:6813/3717704491 exists,up 83503727-afc1-4b95-8b96-ecb63f3338c2'] 2024-09-20T22:42:35.507 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon mon.a config show 2024-09-20T22:42:35.754 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:42:36.224 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-20T22:42:36.224 INFO:teuthology.orchestra.run.smithi084.stdout: "name": "mon.a", 2024-09-20T22:42:36.224 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster": "ceph", 2024-09-20T22:42:36.225 INFO:teuthology.orchestra.run.smithi084.stdout: "admin_socket": "/var/run/ceph/ceph-mon.a.asok", 2024-09-20T22:42:36.225 INFO:teuthology.orchestra.run.smithi084.stdout: "admin_socket_mode": "", 2024-09-20T22:42:36.225 INFO:teuthology.orchestra.run.smithi084.stdout: "auth_allow_insecure_global_id_reclaim": "false", 2024-09-20T22:42:36.225 INFO:teuthology.orchestra.run.smithi084.stdout: "auth_client_required": "cephx, none", 2024-09-20T22:42:36.225 INFO:teuthology.orchestra.run.smithi084.stdout: "auth_cluster_required": "cephx", 2024-09-20T22:42:36.225 INFO:teuthology.orchestra.run.smithi084.stdout: "auth_debug": "true", 2024-09-20T22:42:36.225 INFO:teuthology.orchestra.run.smithi084.stdout: "auth_expose_insecure_global_id_reclaim": "true", 2024-09-20T22:42:36.225 INFO:teuthology.orchestra.run.smithi084.stdout: "auth_mon_ticket_ttl": "660.000000", 2024-09-20T22:42:36.225 INFO:teuthology.orchestra.run.smithi084.stdout: "auth_service_required": "cephx", 2024-09-20T22:42:36.225 INFO:teuthology.orchestra.run.smithi084.stdout: "auth_service_ticket_ttl": "240.000000", 2024-09-20T22:42:36.225 INFO:teuthology.orchestra.run.smithi084.stdout: "auth_supported": "", 2024-09-20T22:42:36.225 INFO:teuthology.orchestra.run.smithi084.stdout: "bdev_aio": "true", 2024-09-20T22:42:36.225 INFO:teuthology.orchestra.run.smithi084.stdout: "bdev_aio_max_queue_depth": "1024", 2024-09-20T22:42:36.225 INFO:teuthology.orchestra.run.smithi084.stdout: "bdev_aio_poll_ms": "250", 2024-09-20T22:42:36.226 INFO:teuthology.orchestra.run.smithi084.stdout: "bdev_aio_reap_max": "16", 2024-09-20T22:42:36.226 INFO:teuthology.orchestra.run.smithi084.stdout: "bdev_async_discard": "false", 2024-09-20T22:42:36.226 INFO:teuthology.orchestra.run.smithi084.stdout: "bdev_block_size": "4096", 2024-09-20T22:42:36.226 INFO:teuthology.orchestra.run.smithi084.stdout: "bdev_debug_aio": "false", 2024-09-20T22:42:36.226 INFO:teuthology.orchestra.run.smithi084.stdout: "bdev_debug_aio_log_age": "5.000000", 2024-09-20T22:42:36.226 INFO:teuthology.orchestra.run.smithi084.stdout: "bdev_debug_aio_suicide_timeout": "60.000000", 2024-09-20T22:42:36.226 INFO:teuthology.orchestra.run.smithi084.stdout: "bdev_debug_inflight_ios": "false", 2024-09-20T22:42:36.226 INFO:teuthology.orchestra.run.smithi084.stdout: "bdev_enable_discard": "false", 2024-09-20T22:42:36.226 INFO:teuthology.orchestra.run.smithi084.stdout: "bdev_flock_retry": "3", 2024-09-20T22:42:36.226 INFO:teuthology.orchestra.run.smithi084.stdout: "bdev_flock_retry_interval": "0.100000", 2024-09-20T22:42:36.226 INFO:teuthology.orchestra.run.smithi084.stdout: "bdev_inject_crash": "0", 2024-09-20T22:42:36.226 INFO:teuthology.orchestra.run.smithi084.stdout: "bdev_inject_crash_flush_delay": "2", 2024-09-20T22:42:36.227 INFO:teuthology.orchestra.run.smithi084.stdout: "bdev_ioring": "false", 2024-09-20T22:42:36.227 INFO:teuthology.orchestra.run.smithi084.stdout: "bdev_ioring_hipri": "false", 2024-09-20T22:42:36.227 INFO:teuthology.orchestra.run.smithi084.stdout: "bdev_ioring_sqthread_poll": "false", 2024-09-20T22:42:36.227 INFO:teuthology.orchestra.run.smithi084.stdout: "bdev_nvme_unbind_from_kernel": "false", 2024-09-20T22:42:36.227 INFO:teuthology.orchestra.run.smithi084.stdout: "bdev_read_buffer_alignment": "4096", 2024-09-20T22:42:36.227 INFO:teuthology.orchestra.run.smithi084.stdout: "bdev_read_preallocated_huge_buffers": "", 2024-09-20T22:42:36.227 INFO:teuthology.orchestra.run.smithi084.stdout: "bdev_stalled_read_warn_lifetime": "86400", 2024-09-20T22:42:36.227 INFO:teuthology.orchestra.run.smithi084.stdout: "bdev_stalled_read_warn_threshold": "1", 2024-09-20T22:42:36.227 INFO:teuthology.orchestra.run.smithi084.stdout: "bdev_type": "", 2024-09-20T22:42:36.227 INFO:teuthology.orchestra.run.smithi084.stdout: "bluefs_alloc_size": "1048576", 2024-09-20T22:42:36.227 INFO:teuthology.orchestra.run.smithi084.stdout: "bluefs_allocator": "hybrid", 2024-09-20T22:42:36.227 INFO:teuthology.orchestra.run.smithi084.stdout: "bluefs_buffered_io": "true", 2024-09-20T22:42:36.227 INFO:teuthology.orchestra.run.smithi084.stdout: "bluefs_check_for_zeros": "false", 2024-09-20T22:42:36.227 INFO:teuthology.orchestra.run.smithi084.stdout: "bluefs_check_volume_selector_often": "false", 2024-09-20T22:42:36.228 INFO:teuthology.orchestra.run.smithi084.stdout: "bluefs_check_volume_selector_on_umount": "false", 2024-09-20T22:42:36.228 INFO:teuthology.orchestra.run.smithi084.stdout: "bluefs_compact_log_sync": "false", 2024-09-20T22:42:36.228 INFO:teuthology.orchestra.run.smithi084.stdout: "bluefs_failed_shared_alloc_cooldown": "600.000000", 2024-09-20T22:42:36.228 INFO:teuthology.orchestra.run.smithi084.stdout: "bluefs_log_compact_min_ratio": "5.000000", 2024-09-20T22:42:36.228 INFO:teuthology.orchestra.run.smithi084.stdout: "bluefs_log_compact_min_size": "16777216", 2024-09-20T22:42:36.228 INFO:teuthology.orchestra.run.smithi084.stdout: "bluefs_log_replay_check_allocations": "true", 2024-09-20T22:42:36.228 INFO:teuthology.orchestra.run.smithi084.stdout: "bluefs_max_log_runway": "4194304", 2024-09-20T22:42:36.228 INFO:teuthology.orchestra.run.smithi084.stdout: "bluefs_max_prefetch": "1048576", 2024-09-20T22:42:36.228 INFO:teuthology.orchestra.run.smithi084.stdout: "bluefs_min_flush_size": "524288", 2024-09-20T22:42:36.228 INFO:teuthology.orchestra.run.smithi084.stdout: "bluefs_min_log_runway": "1048576", 2024-09-20T22:42:36.228 INFO:teuthology.orchestra.run.smithi084.stdout: "bluefs_replay_recovery": "false", 2024-09-20T22:42:36.228 INFO:teuthology.orchestra.run.smithi084.stdout: "bluefs_replay_recovery_disable_compact": "false", 2024-09-20T22:42:36.229 INFO:teuthology.orchestra.run.smithi084.stdout: "bluefs_shared_alloc_size": "65536", 2024-09-20T22:42:36.229 INFO:teuthology.orchestra.run.smithi084.stdout: "bluefs_sync_write": "false", 2024-09-20T22:42:36.229 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_2q_cache_kin_ratio": "0.500000", 2024-09-20T22:42:36.229 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_2q_cache_kout_ratio": "0.500000", 2024-09-20T22:42:36.229 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_alloc_stats_dump_interval": "86400.000000", 2024-09-20T22:42:36.229 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_allocation_from_file": "true", 2024-09-20T22:42:36.229 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_allocator": "hybrid", 2024-09-20T22:42:36.229 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_avl_alloc_bf_free_pct": "4", 2024-09-20T22:42:36.229 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_avl_alloc_bf_threshold": "131072", 2024-09-20T22:42:36.229 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_avl_alloc_ff_max_search_bytes": "16777216", 2024-09-20T22:42:36.229 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_avl_alloc_ff_max_search_count": "100", 2024-09-20T22:42:36.229 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_bdev_label_multi": "true", 2024-09-20T22:42:36.229 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_bdev_label_multi_upgrade": "false", 2024-09-20T22:42:36.230 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_bdev_label_require_all": "true", 2024-09-20T22:42:36.230 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_bitmapallocator_blocks_per_zone": "1024", 2024-09-20T22:42:36.230 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_bitmapallocator_span_size": "1024", 2024-09-20T22:42:36.230 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_blobid_prealloc": "10240", 2024-09-20T22:42:36.230 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_block_create": "true", 2024-09-20T22:42:36.230 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_block_db_create": "false", 2024-09-20T22:42:36.230 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_block_db_path": "", 2024-09-20T22:42:36.230 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_block_db_size": "0", 2024-09-20T22:42:36.230 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_block_path": "", 2024-09-20T22:42:36.230 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_block_preallocate_file": "false", 2024-09-20T22:42:36.230 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_block_size": "107374182400", 2024-09-20T22:42:36.230 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_block_wal_create": "false", 2024-09-20T22:42:36.230 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_block_wal_path": "", 2024-09-20T22:42:36.231 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_block_wal_size": "100663296", 2024-09-20T22:42:36.231 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_bluefs": "true", 2024-09-20T22:42:36.231 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_bluefs_alloc_failure_dump_interval": "0.000000", 2024-09-20T22:42:36.231 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_bluefs_env_mirror": "false", 2024-09-20T22:42:36.231 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_bluefs_max_free": "10737418240", 2024-09-20T22:42:36.231 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_cache_age_bin_interval": "1.000000", 2024-09-20T22:42:36.231 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_cache_age_bins_data": "1 2 6 24 120 720 0 0 0 0", 2024-09-20T22:42:36.231 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_cache_age_bins_kv": "1 2 6 24 120 720 0 0 0 0", 2024-09-20T22:42:36.231 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_cache_age_bins_kv_onode": "0 0 0 0 0 0 0 0 0 720", 2024-09-20T22:42:36.231 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_cache_age_bins_meta": "1 2 6 24 120 720 0 0 0 0", 2024-09-20T22:42:36.231 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_cache_autotune": "true", 2024-09-20T22:42:36.231 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_cache_autotune_interval": "5.000000", 2024-09-20T22:42:36.231 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_cache_kv_onode_ratio": "0.040000", 2024-09-20T22:42:36.231 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_cache_kv_ratio": "0.450000", 2024-09-20T22:42:36.232 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_cache_meta_ratio": "0.450000", 2024-09-20T22:42:36.232 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_cache_size": "0", 2024-09-20T22:42:36.232 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_cache_size_hdd": "1073741824", 2024-09-20T22:42:36.232 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_cache_size_ssd": "3221225472", 2024-09-20T22:42:36.232 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_cache_trim_interval": "0.050000", 2024-09-20T22:42:36.232 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_cache_trim_max_skip_pinned": "1000", 2024-09-20T22:42:36.232 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_cache_type": "2q", 2024-09-20T22:42:36.232 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_cleaner_sleep_interval": "5.000000", 2024-09-20T22:42:36.232 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_clone_cow": "true", 2024-09-20T22:42:36.232 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_compression_algorithm": "snappy", 2024-09-20T22:42:36.232 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_compression_max_blob_size": "0", 2024-09-20T22:42:36.232 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_compression_max_blob_size_hdd": "65536", 2024-09-20T22:42:36.233 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_compression_max_blob_size_ssd": "65536", 2024-09-20T22:42:36.233 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_compression_min_blob_size": "0", 2024-09-20T22:42:36.233 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_compression_min_blob_size_hdd": "8192", 2024-09-20T22:42:36.233 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_compression_min_blob_size_ssd": "65536", 2024-09-20T22:42:36.233 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_compression_mode": "none", 2024-09-20T22:42:36.233 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_compression_required_ratio": "0.875000", 2024-09-20T22:42:36.233 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_csum_type": "crc32c", 2024-09-20T22:42:36.233 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_debug_enforce_settings": "default", 2024-09-20T22:42:36.233 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_debug_freelist": "false", 2024-09-20T22:42:36.233 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_debug_fsck_abort": "false", 2024-09-20T22:42:36.233 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_debug_inject_allocation_from_file_failure": "0.000000", 2024-09-20T22:42:36.233 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_debug_inject_csum_err_probability": "0.000000", 2024-09-20T22:42:36.233 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_debug_inject_read_err": "false", 2024-09-20T22:42:36.233 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_debug_legacy_omap": "false", 2024-09-20T22:42:36.233 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_debug_no_reuse_blocks": "false", 2024-09-20T22:42:36.234 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_debug_omit_block_device_write": "false", 2024-09-20T22:42:36.234 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_debug_omit_kv_commit": "false", 2024-09-20T22:42:36.234 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_debug_permit_any_bdev_label": "false", 2024-09-20T22:42:36.234 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_debug_prefragment_max": "1048576", 2024-09-20T22:42:36.234 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_debug_random_read_err": "0.000000", 2024-09-20T22:42:36.234 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_debug_randomize_serial_transaction": "0", 2024-09-20T22:42:36.234 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_debug_small_allocations": "0", 2024-09-20T22:42:36.234 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_debug_too_many_blobs_threshold": "24576", 2024-09-20T22:42:36.234 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_default_buffered_read": "true", 2024-09-20T22:42:36.234 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_default_buffered_write": "false", 2024-09-20T22:42:36.234 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_deferred_batch_ops": "0", 2024-09-20T22:42:36.234 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_deferred_batch_ops_hdd": "64", 2024-09-20T22:42:36.234 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_deferred_batch_ops_ssd": "16", 2024-09-20T22:42:36.234 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_elastic_shared_blobs": "true", 2024-09-20T22:42:36.235 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_extent_map_inline_shard_prealloc_size": "256", 2024-09-20T22:42:36.235 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_extent_map_shard_max_size": "1200", 2024-09-20T22:42:36.235 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_extent_map_shard_min_size": "150", 2024-09-20T22:42:36.235 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_extent_map_shard_target_size": "500", 2024-09-20T22:42:36.235 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_extent_map_shard_target_size_slop": "0.200000", 2024-09-20T22:42:36.235 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_fail_eio": "false", 2024-09-20T22:42:36.235 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_freelist_blocks_per_key": "128", 2024-09-20T22:42:36.235 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_fsck_error_on_no_per_pg_omap": "false", 2024-09-20T22:42:36.235 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_fsck_error_on_no_per_pool_omap": "false", 2024-09-20T22:42:36.235 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_fsck_error_on_no_per_pool_stats": "false", 2024-09-20T22:42:36.235 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_fsck_on_mkfs": "true", 2024-09-20T22:42:36.235 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_fsck_on_mkfs_deep": "false", 2024-09-20T22:42:36.235 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_fsck_on_mount": "false", 2024-09-20T22:42:36.235 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_fsck_on_mount_deep": "false", 2024-09-20T22:42:36.235 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_fsck_on_umount": "false", 2024-09-20T22:42:36.236 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_fsck_on_umount_deep": "false", 2024-09-20T22:42:36.236 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_fsck_quick_fix_on_mount": "false", 2024-09-20T22:42:36.236 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_fsck_quick_fix_threads": "2", 2024-09-20T22:42:36.236 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_fsck_read_bytes_cap": "67108864", 2024-09-20T22:42:36.236 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_fsck_shared_blob_tracker_size": "0.031250", 2024-09-20T22:42:36.236 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_gc_enable_blob_threshold": "0", 2024-09-20T22:42:36.236 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_gc_enable_total_threshold": "0", 2024-09-20T22:42:36.236 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_hybrid_alloc_mem_cap": "67108864", 2024-09-20T22:42:36.236 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_ignore_data_csum": "false", 2024-09-20T22:42:36.236 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_kv_sync_util_logging_s": "10.000000", 2024-09-20T22:42:36.236 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_kvbackend": "rocksdb", 2024-09-20T22:42:36.236 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_log_collection_list_age": "60.000000", 2024-09-20T22:42:36.236 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_log_omap_iterator_age": "5.000000", 2024-09-20T22:42:36.236 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_log_op_age": "5.000000", 2024-09-20T22:42:36.236 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_max_alloc_size": "0", 2024-09-20T22:42:36.237 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_max_blob_size": "0", 2024-09-20T22:42:36.237 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_max_blob_size_hdd": "65536", 2024-09-20T22:42:36.237 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_max_blob_size_ssd": "65536", 2024-09-20T22:42:36.237 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_max_defer_interval": "3.000000", 2024-09-20T22:42:36.237 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_max_deferred_txc": "32", 2024-09-20T22:42:36.237 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_min_alloc_size": "0", 2024-09-20T22:42:36.237 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_min_alloc_size_hdd": "4096", 2024-09-20T22:42:36.237 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_min_alloc_size_ssd": "4096", 2024-09-20T22:42:36.237 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_nid_prealloc": "1024", 2024-09-20T22:42:36.237 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_prefer_deferred_size": "0", 2024-09-20T22:42:36.237 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_prefer_deferred_size_hdd": "65536", 2024-09-20T22:42:36.237 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_prefer_deferred_size_ssd": "0", 2024-09-20T22:42:36.237 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_qfsck_on_mount": "true", 2024-09-20T22:42:36.238 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_retry_disk_reads": "3", 2024-09-20T22:42:36.238 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_rocksdb_cf": "true", 2024-09-20T22:42:36.238 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_rocksdb_cfs": "m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L=min_write_buffer_number_to_merge=32 P=min_write_buffer_number_to_merge=32", 2024-09-20T22:42:36.238 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_rocksdb_options": "compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0", 2024-09-20T22:42:36.238 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_rocksdb_options_annex": "", 2024-09-20T22:42:36.238 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_slow_ops_warn_lifetime": "86400", 2024-09-20T22:42:36.238 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_slow_ops_warn_threshold": "1", 2024-09-20T22:42:36.238 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_spdk_coremask": "0x1", 2024-09-20T22:42:36.238 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_spdk_io_sleep": "5", 2024-09-20T22:42:36.238 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_spdk_max_io_completion": "0", 2024-09-20T22:42:36.238 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_spdk_mem": "512", 2024-09-20T22:42:36.238 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_sync_submit_transaction": "false", 2024-09-20T22:42:36.239 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_throttle_bytes": "67108864", 2024-09-20T22:42:36.239 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_throttle_cost_per_io": "0", 2024-09-20T22:42:36.239 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_throttle_cost_per_io_hdd": "670000", 2024-09-20T22:42:36.239 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_throttle_cost_per_io_ssd": "4000", 2024-09-20T22:42:36.239 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_throttle_deferred_bytes": "134217728", 2024-09-20T22:42:36.239 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_throttle_trace_rate": "0.000000", 2024-09-20T22:42:36.239 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_tracing": "false", 2024-09-20T22:42:36.239 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_use_optimal_io_size_for_min_alloc_size": "false", 2024-09-20T22:42:36.239 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_volume_selection_policy": "use_some_extra", 2024-09-20T22:42:36.239 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_volume_selection_reserved": "0", 2024-09-20T22:42:36.239 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_volume_selection_reserved_factor": "2.000000", 2024-09-20T22:42:36.239 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_warn_on_bluefs_spillover": "true", 2024-09-20T22:42:36.239 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_warn_on_legacy_statfs": "true", 2024-09-20T22:42:36.240 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_warn_on_no_per_pg_omap": "false", 2024-09-20T22:42:36.240 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_warn_on_no_per_pool_omap": "true", 2024-09-20T22:42:36.240 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_warn_on_spurious_read_errors": "true", 2024-09-20T22:42:36.240 INFO:teuthology.orchestra.run.smithi084.stdout: "bluestore_zero_block_detection": "false", 2024-09-20T22:42:36.240 INFO:teuthology.orchestra.run.smithi084.stdout: "cephadm_path": "/usr/sbin/cephadm", 2024-09-20T22:42:36.240 INFO:teuthology.orchestra.run.smithi084.stdout: "cephfs_mirror_action_update_interval": "2", 2024-09-20T22:42:36.240 INFO:teuthology.orchestra.run.smithi084.stdout: "cephfs_mirror_directory_scan_interval": "10", 2024-09-20T22:42:36.240 INFO:teuthology.orchestra.run.smithi084.stdout: "cephfs_mirror_max_concurrent_directory_syncs": "3", 2024-09-20T22:42:36.240 INFO:teuthology.orchestra.run.smithi084.stdout: "cephfs_mirror_max_consecutive_failures_per_directory": "10", 2024-09-20T22:42:36.240 INFO:teuthology.orchestra.run.smithi084.stdout: "cephfs_mirror_max_snapshot_sync_per_cycle": "3", 2024-09-20T22:42:36.240 INFO:teuthology.orchestra.run.smithi084.stdout: "cephfs_mirror_mount_timeout": "10", 2024-09-20T22:42:36.240 INFO:teuthology.orchestra.run.smithi084.stdout: "cephfs_mirror_perf_stats_prio": "5", 2024-09-20T22:42:36.240 INFO:teuthology.orchestra.run.smithi084.stdout: "cephfs_mirror_restart_mirror_on_blocklist_interval": "30", 2024-09-20T22:42:36.240 INFO:teuthology.orchestra.run.smithi084.stdout: "cephfs_mirror_restart_mirror_on_failure_interval": "20", 2024-09-20T22:42:36.241 INFO:teuthology.orchestra.run.smithi084.stdout: "cephfs_mirror_retry_failed_directories_interval": "60", 2024-09-20T22:42:36.241 INFO:teuthology.orchestra.run.smithi084.stdout: "cephsqlite_blocklist_dead_locker": "true", 2024-09-20T22:42:36.241 INFO:teuthology.orchestra.run.smithi084.stdout: "cephsqlite_lock_renewal_interval": "2000", 2024-09-20T22:42:36.241 INFO:teuthology.orchestra.run.smithi084.stdout: "cephsqlite_lock_renewal_timeout": "30000", 2024-09-20T22:42:36.241 INFO:teuthology.orchestra.run.smithi084.stdout: "cephx_cluster_require_signatures": "false", 2024-09-20T22:42:36.241 INFO:teuthology.orchestra.run.smithi084.stdout: "cephx_cluster_require_version": "2", 2024-09-20T22:42:36.241 INFO:teuthology.orchestra.run.smithi084.stdout: "cephx_require_signatures": "false", 2024-09-20T22:42:36.241 INFO:teuthology.orchestra.run.smithi084.stdout: "cephx_require_version": "2", 2024-09-20T22:42:36.241 INFO:teuthology.orchestra.run.smithi084.stdout: "cephx_service_require_signatures": "false", 2024-09-20T22:42:36.241 INFO:teuthology.orchestra.run.smithi084.stdout: "cephx_service_require_version": "2", 2024-09-20T22:42:36.241 INFO:teuthology.orchestra.run.smithi084.stdout: "cephx_sign_messages": "true", 2024-09-20T22:42:36.241 INFO:teuthology.orchestra.run.smithi084.stdout: "chdir": "", 2024-09-20T22:42:36.241 INFO:teuthology.orchestra.run.smithi084.stdout: "client_acl_type": "", 2024-09-20T22:42:36.241 INFO:teuthology.orchestra.run.smithi084.stdout: "client_asio_thread_count": "2", 2024-09-20T22:42:36.241 INFO:teuthology.orchestra.run.smithi084.stdout: "client_cache_mid": "0.750000", 2024-09-20T22:42:36.241 INFO:teuthology.orchestra.run.smithi084.stdout: "client_cache_size": "16384", 2024-09-20T22:42:36.242 INFO:teuthology.orchestra.run.smithi084.stdout: "client_caps_release_delay": "5", 2024-09-20T22:42:36.242 INFO:teuthology.orchestra.run.smithi084.stdout: "client_check_pool_perm": "true", 2024-09-20T22:42:36.242 INFO:teuthology.orchestra.run.smithi084.stdout: "client_collect_and_send_global_metrics": "false", 2024-09-20T22:42:36.242 INFO:teuthology.orchestra.run.smithi084.stdout: "client_debug_force_sync_read": "false", 2024-09-20T22:42:36.242 INFO:teuthology.orchestra.run.smithi084.stdout: "client_debug_getattr_caps": "false", 2024-09-20T22:42:36.242 INFO:teuthology.orchestra.run.smithi084.stdout: "client_debug_inject_features": "", 2024-09-20T22:42:36.242 INFO:teuthology.orchestra.run.smithi084.stdout: "client_debug_inject_tick_delay": "0", 2024-09-20T22:42:36.242 INFO:teuthology.orchestra.run.smithi084.stdout: "client_die_on_failed_dentry_invalidate": "true", 2024-09-20T22:42:36.242 INFO:teuthology.orchestra.run.smithi084.stdout: "client_die_on_failed_remount": "false", 2024-09-20T22:42:36.242 INFO:teuthology.orchestra.run.smithi084.stdout: "client_dirsize_rbytes": "true", 2024-09-20T22:42:36.242 INFO:teuthology.orchestra.run.smithi084.stdout: "client_force_lazyio": "false", 2024-09-20T22:42:36.242 INFO:teuthology.orchestra.run.smithi084.stdout: "client_fs": "", 2024-09-20T22:42:36.242 INFO:teuthology.orchestra.run.smithi084.stdout: "client_inject_fixed_oldest_tid": "false", 2024-09-20T22:42:36.243 INFO:teuthology.orchestra.run.smithi084.stdout: "client_inject_release_failure": "false", 2024-09-20T22:42:36.243 INFO:teuthology.orchestra.run.smithi084.stdout: "client_max_inline_size": "4096", 2024-09-20T22:42:36.243 INFO:teuthology.orchestra.run.smithi084.stdout: "client_max_retries_on_remount_failure": "5", 2024-09-20T22:42:36.243 INFO:teuthology.orchestra.run.smithi084.stdout: "client_mds_namespace": "", 2024-09-20T22:42:36.243 INFO:teuthology.orchestra.run.smithi084.stdout: "client_metadata": "", 2024-09-20T22:42:36.243 INFO:teuthology.orchestra.run.smithi084.stdout: "client_mount_gid": "-1", 2024-09-20T22:42:36.243 INFO:teuthology.orchestra.run.smithi084.stdout: "client_mount_timeout": "300", 2024-09-20T22:42:36.243 INFO:teuthology.orchestra.run.smithi084.stdout: "client_mount_uid": "-1", 2024-09-20T22:42:36.243 INFO:teuthology.orchestra.run.smithi084.stdout: "client_mountpoint": "/", 2024-09-20T22:42:36.243 INFO:teuthology.orchestra.run.smithi084.stdout: "client_notify_timeout": "10", 2024-09-20T22:42:36.243 INFO:teuthology.orchestra.run.smithi084.stdout: "client_oc": "true", 2024-09-20T22:42:36.243 INFO:teuthology.orchestra.run.smithi084.stdout: "client_oc_max_dirty": "104857600", 2024-09-20T22:42:36.243 INFO:teuthology.orchestra.run.smithi084.stdout: "client_oc_max_dirty_age": "5.000000", 2024-09-20T22:42:36.243 INFO:teuthology.orchestra.run.smithi084.stdout: "client_oc_max_objects": "1000", 2024-09-20T22:42:36.243 INFO:teuthology.orchestra.run.smithi084.stdout: "client_oc_size": "209715200", 2024-09-20T22:42:36.243 INFO:teuthology.orchestra.run.smithi084.stdout: "client_oc_target_dirty": "8388608", 2024-09-20T22:42:36.244 INFO:teuthology.orchestra.run.smithi084.stdout: "client_permissions": "true", 2024-09-20T22:42:36.244 INFO:teuthology.orchestra.run.smithi084.stdout: "client_quota": "true", 2024-09-20T22:42:36.244 INFO:teuthology.orchestra.run.smithi084.stdout: "client_quota_df": "true", 2024-09-20T22:42:36.244 INFO:teuthology.orchestra.run.smithi084.stdout: "client_readahead_max_bytes": "0", 2024-09-20T22:42:36.244 INFO:teuthology.orchestra.run.smithi084.stdout: "client_readahead_max_periods": "4", 2024-09-20T22:42:36.244 INFO:teuthology.orchestra.run.smithi084.stdout: "client_readahead_min": "131072", 2024-09-20T22:42:36.244 INFO:teuthology.orchestra.run.smithi084.stdout: "client_reconnect_stale": "false", 2024-09-20T22:42:36.244 INFO:teuthology.orchestra.run.smithi084.stdout: "client_shutdown_timeout": "30", 2024-09-20T22:42:36.244 INFO:teuthology.orchestra.run.smithi084.stdout: "client_snapdir": ".snap", 2024-09-20T22:42:36.244 INFO:teuthology.orchestra.run.smithi084.stdout: "client_tick_interval": "1", 2024-09-20T22:42:36.244 INFO:teuthology.orchestra.run.smithi084.stdout: "client_trace": "", 2024-09-20T22:42:36.244 INFO:teuthology.orchestra.run.smithi084.stdout: "client_try_dentry_invalidate": "false", 2024-09-20T22:42:36.244 INFO:teuthology.orchestra.run.smithi084.stdout: "client_use_faked_inos": "false", 2024-09-20T22:42:36.244 INFO:teuthology.orchestra.run.smithi084.stdout: "client_use_random_mds": "false", 2024-09-20T22:42:36.244 INFO:teuthology.orchestra.run.smithi084.stdout: "clog_to_graylog": "false", 2024-09-20T22:42:36.245 INFO:teuthology.orchestra.run.smithi084.stdout: "clog_to_graylog_host": "127.0.0.1", 2024-09-20T22:42:36.245 INFO:teuthology.orchestra.run.smithi084.stdout: "clog_to_graylog_port": "12201", 2024-09-20T22:42:36.245 INFO:teuthology.orchestra.run.smithi084.stdout: "clog_to_monitors": "default=true", 2024-09-20T22:42:36.245 INFO:teuthology.orchestra.run.smithi084.stdout: "clog_to_syslog": "false", 2024-09-20T22:42:36.245 INFO:teuthology.orchestra.run.smithi084.stdout: "clog_to_syslog_facility": "default=daemon audit=local0", 2024-09-20T22:42:36.245 INFO:teuthology.orchestra.run.smithi084.stdout: "clog_to_syslog_level": "info", 2024-09-20T22:42:36.245 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "-", 2024-09-20T22:42:36.245 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_network": "", 2024-09-20T22:42:36.245 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_network_interface": "", 2024-09-20T22:42:36.245 INFO:teuthology.orchestra.run.smithi084.stdout: "compressor_zlib_isal": "false", 2024-09-20T22:42:36.245 INFO:teuthology.orchestra.run.smithi084.stdout: "compressor_zlib_level": "5", 2024-09-20T22:42:36.245 INFO:teuthology.orchestra.run.smithi084.stdout: "compressor_zlib_winsize": "-15", 2024-09-20T22:42:36.245 INFO:teuthology.orchestra.run.smithi084.stdout: "compressor_zstd_level": "1", 2024-09-20T22:42:36.245 INFO:teuthology.orchestra.run.smithi084.stdout: "container_image": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5", 2024-09-20T22:42:36.245 INFO:teuthology.orchestra.run.smithi084.stdout: "crash_dir": "/var/lib/ceph/crash", 2024-09-20T22:42:36.246 INFO:teuthology.orchestra.run.smithi084.stdout: "crimson_alien_op_num_threads": "6", 2024-09-20T22:42:36.246 INFO:teuthology.orchestra.run.smithi084.stdout: "crimson_alien_thread_cpu_cores": "", 2024-09-20T22:42:36.246 INFO:teuthology.orchestra.run.smithi084.stdout: "crimson_osd_obc_lru_size": "512", 2024-09-20T22:42:36.246 INFO:teuthology.orchestra.run.smithi084.stdout: "crimson_osd_scheduler_concurrency": "0", 2024-09-20T22:42:36.246 INFO:teuthology.orchestra.run.smithi084.stdout: "crimson_osd_stat_interval": "0", 2024-09-20T22:42:36.246 INFO:teuthology.orchestra.run.smithi084.stdout: "crimson_seastar_cpu_cores": "", 2024-09-20T22:42:36.246 INFO:teuthology.orchestra.run.smithi084.stdout: "crimson_seastar_num_threads": "0", 2024-09-20T22:42:36.246 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_location": "", 2024-09-20T22:42:36.246 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_location_hook": "", 2024-09-20T22:42:36.246 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_location_hook_timeout": "10", 2024-09-20T22:42:36.246 INFO:teuthology.orchestra.run.smithi084.stdout: "daemonize": "false", 2024-09-20T22:42:36.246 INFO:teuthology.orchestra.run.smithi084.stdout: "daos_pool": "tank", 2024-09-20T22:42:36.246 INFO:teuthology.orchestra.run.smithi084.stdout: "dbstore_config_uri": "file:/var/lib/ceph/radosgw/dbstore-config.db", 2024-09-20T22:42:36.246 INFO:teuthology.orchestra.run.smithi084.stdout: "dbstore_db_dir": "/var/lib/ceph/radosgw", 2024-09-20T22:42:36.247 INFO:teuthology.orchestra.run.smithi084.stdout: "dbstore_db_name_prefix": "dbstore", 2024-09-20T22:42:36.247 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_alienstore": "0/5", 2024-09-20T22:42:36.247 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_allow_any_pool_priority": "false", 2024-09-20T22:42:36.247 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_asok": "1/5", 2024-09-20T22:42:36.247 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_asok_assert_abort": "false", 2024-09-20T22:42:36.247 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_asserts_on_shutdown": "true", 2024-09-20T22:42:36.247 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_auth": "1/5", 2024-09-20T22:42:36.247 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_bdev": "1/3", 2024-09-20T22:42:36.247 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_bluefs": "1/5", 2024-09-20T22:42:36.247 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_bluestore": "1/5", 2024-09-20T22:42:36.247 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_buffer": "0/1", 2024-09-20T22:42:36.247 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_ceph_exporter": "1/5", 2024-09-20T22:42:36.247 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_cephfs_mirror": "0/5", 2024-09-20T22:42:36.247 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_cephsqlite": "0/5", 2024-09-20T22:42:36.247 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_client": "0/5", 2024-09-20T22:42:36.247 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_compressor": "1/5", 2024-09-20T22:42:36.248 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_context": "0/1", 2024-09-20T22:42:36.248 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_crimson_interrupt": "0/5", 2024-09-20T22:42:36.248 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_crush": "1/1", 2024-09-20T22:42:36.248 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_crypto": "1/5", 2024-09-20T22:42:36.248 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_cyanstore": "0/5", 2024-09-20T22:42:36.248 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_deliberately_leak_memory": "false", 2024-09-20T22:42:36.248 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_disable_randomized_ping": "false", 2024-09-20T22:42:36.248 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_dpdk": "1/5", 2024-09-20T22:42:36.248 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_eventtrace": "1/5", 2024-09-20T22:42:36.248 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_filer": "0/1", 2024-09-20T22:42:36.248 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_filestore": "1/3", 2024-09-20T22:42:36.248 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_finisher": "1/1", 2024-09-20T22:42:36.248 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_fuse": "1/5", 2024-09-20T22:42:36.248 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_heartbeat_testing_span": "0", 2024-09-20T22:42:36.248 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_heartbeatmap": "1/5", 2024-09-20T22:42:36.248 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_immutable_obj_cache": "0/5", 2024-09-20T22:42:36.249 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_javaclient": "1/5", 2024-09-20T22:42:36.249 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_journal": "1/3", 2024-09-20T22:42:36.249 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_journaler": "0/5", 2024-09-20T22:42:36.249 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_kstore": "1/5", 2024-09-20T22:42:36.249 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_lockdep": "0/1", 2024-09-20T22:42:36.249 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_mclock": "1/5", 2024-09-20T22:42:36.249 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_mds": "1/5", 2024-09-20T22:42:36.249 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_mds_balancer": "1/5", 2024-09-20T22:42:36.249 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_mds_locker": "1/5", 2024-09-20T22:42:36.249 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_mds_log": "1/5", 2024-09-20T22:42:36.249 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_mds_log_expire": "1/5", 2024-09-20T22:42:36.249 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_mds_migrator": "1/5", 2024-09-20T22:42:36.249 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_mds_quiesce": "3/5", 2024-09-20T22:42:36.250 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_memstore": "1/5", 2024-09-20T22:42:36.250 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_mgr": "2/5", 2024-09-20T22:42:36.250 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_mgrc": "1/5", 2024-09-20T22:42:36.250 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_mon": "20/20", 2024-09-20T22:42:36.250 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_monc": "0/10", 2024-09-20T22:42:36.250 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_ms": "1/1", 2024-09-20T22:42:36.250 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_none": "0/5", 2024-09-20T22:42:36.250 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_objclass": "0/5", 2024-09-20T22:42:36.250 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_objectcacher": "0/5", 2024-09-20T22:42:36.250 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_objecter": "0/1", 2024-09-20T22:42:36.250 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_optracker": "0/5", 2024-09-20T22:42:36.250 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_osd": "1/5", 2024-09-20T22:42:36.250 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_paxos": "20/20", 2024-09-20T22:42:36.250 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_perfcounter": "1/5", 2024-09-20T22:42:36.251 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_prioritycache": "1/5", 2024-09-20T22:42:36.251 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_rados": "0/5", 2024-09-20T22:42:36.251 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_rbd": "0/5", 2024-09-20T22:42:36.251 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_rbd_mirror": "0/5", 2024-09-20T22:42:36.251 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_rbd_pwl": "0/5", 2024-09-20T22:42:36.251 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_rbd_replay": "0/5", 2024-09-20T22:42:36.251 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_refs": "0/0", 2024-09-20T22:42:36.251 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_reserver": "1/1", 2024-09-20T22:42:36.251 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_rgw": "1/5", 2024-09-20T22:42:36.251 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_rgw_access": "1/5", 2024-09-20T22:42:36.251 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_rgw_datacache": "1/5", 2024-09-20T22:42:36.251 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_rgw_dbstore": "1/5", 2024-09-20T22:42:36.251 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_rgw_flight": "1/5", 2024-09-20T22:42:36.251 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_rgw_lifecycle": "1/5", 2024-09-20T22:42:36.251 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_rgw_sync": "1/5", 2024-09-20T22:42:36.252 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_rocksdb": "4/5", 2024-09-20T22:42:36.252 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_seastore": "0/5", 2024-09-20T22:42:36.252 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_seastore_backref": "0/5", 2024-09-20T22:42:36.252 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_seastore_cache": "0/5", 2024-09-20T22:42:36.252 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_seastore_cleaner": "0/5", 2024-09-20T22:42:36.252 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_seastore_device": "0/5", 2024-09-20T22:42:36.252 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_seastore_epm": "0/5", 2024-09-20T22:42:36.252 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_seastore_fixedkv_tree": "0/5", 2024-09-20T22:42:36.252 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_seastore_journal": "0/5", 2024-09-20T22:42:36.252 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_seastore_lba": "0/5", 2024-09-20T22:42:36.252 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_seastore_odata": "0/5", 2024-09-20T22:42:36.252 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_seastore_omap": "0/5", 2024-09-20T22:42:36.253 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_seastore_onode": "0/5", 2024-09-20T22:42:36.253 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_seastore_t": "0/5", 2024-09-20T22:42:36.253 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_seastore_tm": "0/5", 2024-09-20T22:42:36.253 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_striper": "0/1", 2024-09-20T22:42:36.253 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_test": "0/5", 2024-09-20T22:42:36.253 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_throttle": "1/1", 2024-09-20T22:42:36.253 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_timer": "0/1", 2024-09-20T22:42:36.253 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_tp": "0/5", 2024-09-20T22:42:36.253 INFO:teuthology.orchestra.run.smithi084.stdout: "debug_trace": "1/5", 2024-09-20T22:42:36.253 INFO:teuthology.orchestra.run.smithi084.stdout: "defer_client_eviction_on_laggy_osds": "false", 2024-09-20T22:42:36.253 INFO:teuthology.orchestra.run.smithi084.stdout: "device_failure_prediction_mode": "none", 2024-09-20T22:42:36.253 INFO:teuthology.orchestra.run.smithi084.stdout: "enable_experimental_unrecoverable_data_corrupting_features": "", 2024-09-20T22:42:36.253 INFO:teuthology.orchestra.run.smithi084.stdout: "erasure_code_dir": "/usr/lib64/ceph/erasure-code", 2024-09-20T22:42:36.253 INFO:teuthology.orchestra.run.smithi084.stdout: "err_to_graylog": "false", 2024-09-20T22:42:36.254 INFO:teuthology.orchestra.run.smithi084.stdout: "err_to_journald": "false", 2024-09-20T22:42:36.254 INFO:teuthology.orchestra.run.smithi084.stdout: "err_to_stderr": "true", 2024-09-20T22:42:36.254 INFO:teuthology.orchestra.run.smithi084.stdout: "err_to_syslog": "false", 2024-09-20T22:42:36.254 INFO:teuthology.orchestra.run.smithi084.stdout: "event_tracing": "false", 2024-09-20T22:42:36.254 INFO:teuthology.orchestra.run.smithi084.stdout: "exporter_addr": "0.0.0.0", 2024-09-20T22:42:36.254 INFO:teuthology.orchestra.run.smithi084.stdout: "exporter_http_port": "9926", 2024-09-20T22:42:36.254 INFO:teuthology.orchestra.run.smithi084.stdout: "exporter_prio_limit": "5", 2024-09-20T22:42:36.254 INFO:teuthology.orchestra.run.smithi084.stdout: "exporter_sock_dir": "/var/run/ceph/", 2024-09-20T22:42:36.254 INFO:teuthology.orchestra.run.smithi084.stdout: "exporter_sort_metrics": "true", 2024-09-20T22:42:36.254 INFO:teuthology.orchestra.run.smithi084.stdout: "exporter_stats_period": "5", 2024-09-20T22:42:36.254 INFO:teuthology.orchestra.run.smithi084.stdout: "fake_statfs_for_testing": "0", 2024-09-20T22:42:36.254 INFO:teuthology.orchestra.run.smithi084.stdout: "fatal_signal_handlers": "true", 2024-09-20T22:42:36.254 INFO:teuthology.orchestra.run.smithi084.stdout: "filer_max_purge_ops": "10", 2024-09-20T22:42:36.255 INFO:teuthology.orchestra.run.smithi084.stdout: "filer_max_truncate_ops": "128", 2024-09-20T22:42:36.255 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_apply_finisher_threads": "1", 2024-09-20T22:42:36.255 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_blackhole": "false", 2024-09-20T22:42:36.255 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_btrfs_clone_range": "true", 2024-09-20T22:42:36.255 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_btrfs_snap": "true", 2024-09-20T22:42:36.255 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_caller_concurrency": "10", 2024-09-20T22:42:36.255 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_collect_device_partition_information": "true", 2024-09-20T22:42:36.255 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_commit_timeout": "600.000000", 2024-09-20T22:42:36.255 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_debug_inject_read_err": "false", 2024-09-20T22:42:36.255 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_debug_omap_check": "false", 2024-09-20T22:42:36.255 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_debug_random_read_err": "0.000000", 2024-09-20T22:42:36.255 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_debug_verify_split": "false", 2024-09-20T22:42:36.255 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_dump_file": "", 2024-09-20T22:42:36.255 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_expected_throughput_bytes": "209715200.000000", 2024-09-20T22:42:36.256 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_expected_throughput_ops": "200.000000", 2024-09-20T22:42:36.256 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_fadvise": "true", 2024-09-20T22:42:36.256 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_fail_eio": "true", 2024-09-20T22:42:36.256 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_fd_cache_shards": "16", 2024-09-20T22:42:36.256 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_fd_cache_size": "128", 2024-09-20T22:42:36.256 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_fiemap": "false", 2024-09-20T22:42:36.256 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_fiemap_threshold": "4096", 2024-09-20T22:42:36.256 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_fsync_flushes_journal_data": "false", 2024-09-20T22:42:36.256 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_index_retry_probability": "0.000000", 2024-09-20T22:42:36.256 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_inject_stall": "0", 2024-09-20T22:42:36.256 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_journal_parallel": "false", 2024-09-20T22:42:36.256 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_journal_trailing": "false", 2024-09-20T22:42:36.256 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_journal_writeahead": "false", 2024-09-20T22:42:36.256 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_kill_at": "0", 2024-09-20T22:42:36.257 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_max_alloc_hint_size": "1048576", 2024-09-20T22:42:36.257 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_max_inline_xattr_size": "0", 2024-09-20T22:42:36.257 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_max_inline_xattr_size_btrfs": "2048", 2024-09-20T22:42:36.257 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_max_inline_xattr_size_other": "512", 2024-09-20T22:42:36.257 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_max_inline_xattr_size_xfs": "65536", 2024-09-20T22:42:36.257 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_max_inline_xattrs": "0", 2024-09-20T22:42:36.257 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_max_inline_xattrs_btrfs": "10", 2024-09-20T22:42:36.257 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_max_inline_xattrs_other": "2", 2024-09-20T22:42:36.257 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_max_inline_xattrs_xfs": "10", 2024-09-20T22:42:36.257 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_max_sync_interval": "5.000000", 2024-09-20T22:42:36.257 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_max_xattr_value_size": "0", 2024-09-20T22:42:36.257 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_max_xattr_value_size_btrfs": "65536", 2024-09-20T22:42:36.257 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_max_xattr_value_size_other": "1024", 2024-09-20T22:42:36.257 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_max_xattr_value_size_xfs": "65536", 2024-09-20T22:42:36.257 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_merge_threshold": "-10", 2024-09-20T22:42:36.258 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_min_sync_interval": "0.010000", 2024-09-20T22:42:36.258 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_odsync_write": "false", 2024-09-20T22:42:36.258 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_omap_backend": "rocksdb", 2024-09-20T22:42:36.258 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_omap_backend_path": "", 2024-09-20T22:42:36.258 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_omap_header_cache_size": "1024", 2024-09-20T22:42:36.258 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_ondisk_finisher_threads": "1", 2024-09-20T22:42:36.258 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_op_thread_suicide_timeout": "180", 2024-09-20T22:42:36.258 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_op_thread_timeout": "60", 2024-09-20T22:42:36.258 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_op_threads": "2", 2024-09-20T22:42:36.258 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_punch_hole": "false", 2024-09-20T22:42:36.258 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_queue_high_delay_multiple": "0.000000", 2024-09-20T22:42:36.258 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_queue_high_delay_multiple_bytes": "0.000000", 2024-09-20T22:42:36.258 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_queue_high_delay_multiple_ops": "0.000000", 2024-09-20T22:42:36.258 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_queue_high_threshhold": "0.900000", 2024-09-20T22:42:36.259 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_queue_low_threshhold": "0.300000", 2024-09-20T22:42:36.259 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_queue_max_bytes": "104857600", 2024-09-20T22:42:36.259 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_queue_max_delay_multiple": "0.000000", 2024-09-20T22:42:36.259 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_queue_max_delay_multiple_bytes": "0.000000", 2024-09-20T22:42:36.259 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_queue_max_delay_multiple_ops": "0.000000", 2024-09-20T22:42:36.259 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_queue_max_ops": "50", 2024-09-20T22:42:36.259 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_rocksdb_options": "max_background_jobs=10,compaction_readahead_size=2097152,compression=kNoCompression", 2024-09-20T22:42:36.259 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_seek_data_hole": "false", 2024-09-20T22:42:36.259 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_sloppy_crc": "false", 2024-09-20T22:42:36.259 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_sloppy_crc_block_size": "65536", 2024-09-20T22:42:36.259 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_splice": "false", 2024-09-20T22:42:36.259 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_split_multiple": "2", 2024-09-20T22:42:36.260 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_split_rand_factor": "20", 2024-09-20T22:42:36.260 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_update_to": "1000", 2024-09-20T22:42:36.260 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_wbthrottle_btrfs_bytes_hard_limit": "419430400", 2024-09-20T22:42:36.260 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_wbthrottle_btrfs_bytes_start_flusher": "41943040", 2024-09-20T22:42:36.260 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_wbthrottle_btrfs_inodes_hard_limit": "5000", 2024-09-20T22:42:36.260 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_wbthrottle_btrfs_inodes_start_flusher": "500", 2024-09-20T22:42:36.260 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_wbthrottle_btrfs_ios_hard_limit": "5000", 2024-09-20T22:42:36.260 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_wbthrottle_btrfs_ios_start_flusher": "500", 2024-09-20T22:42:36.260 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_wbthrottle_enable": "true", 2024-09-20T22:42:36.260 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_wbthrottle_xfs_bytes_hard_limit": "419430400", 2024-09-20T22:42:36.260 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_wbthrottle_xfs_bytes_start_flusher": "41943040", 2024-09-20T22:42:36.260 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_wbthrottle_xfs_inodes_hard_limit": "5000", 2024-09-20T22:42:36.260 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_wbthrottle_xfs_inodes_start_flusher": "500", 2024-09-20T22:42:36.260 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_wbthrottle_xfs_ios_hard_limit": "5000", 2024-09-20T22:42:36.260 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_wbthrottle_xfs_ios_start_flusher": "500", 2024-09-20T22:42:36.261 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_xfs_extsize": "false", 2024-09-20T22:42:36.261 INFO:teuthology.orchestra.run.smithi084.stdout: "filestore_zfs_snap": "false", 2024-09-20T22:42:36.261 INFO:teuthology.orchestra.run.smithi084.stdout: "fio_dir": "/tmp/fio", 2024-09-20T22:42:36.261 INFO:teuthology.orchestra.run.smithi084.stdout: "fsid": "a9c0bc70-77a0-11ef-baf5-efdc52797490", 2024-09-20T22:42:36.261 INFO:teuthology.orchestra.run.smithi084.stdout: "fuse_allow_other": "true", 2024-09-20T22:42:36.261 INFO:teuthology.orchestra.run.smithi084.stdout: "fuse_atomic_o_trunc": "true", 2024-09-20T22:42:36.261 INFO:teuthology.orchestra.run.smithi084.stdout: "fuse_big_writes": "true", 2024-09-20T22:42:36.261 INFO:teuthology.orchestra.run.smithi084.stdout: "fuse_debug": "false", 2024-09-20T22:42:36.261 INFO:teuthology.orchestra.run.smithi084.stdout: "fuse_default_permissions": "false", 2024-09-20T22:42:36.261 INFO:teuthology.orchestra.run.smithi084.stdout: "fuse_disable_pagecache": "false", 2024-09-20T22:42:36.261 INFO:teuthology.orchestra.run.smithi084.stdout: "fuse_max_write": "0", 2024-09-20T22:42:36.261 INFO:teuthology.orchestra.run.smithi084.stdout: "fuse_multithreaded": "true", 2024-09-20T22:42:36.261 INFO:teuthology.orchestra.run.smithi084.stdout: "fuse_require_active_mds": "true", 2024-09-20T22:42:36.261 INFO:teuthology.orchestra.run.smithi084.stdout: "fuse_set_user_groups": "true", 2024-09-20T22:42:36.261 INFO:teuthology.orchestra.run.smithi084.stdout: "fuse_splice_move": "true", 2024-09-20T22:42:36.262 INFO:teuthology.orchestra.run.smithi084.stdout: "fuse_splice_read": "true", 2024-09-20T22:42:36.262 INFO:teuthology.orchestra.run.smithi084.stdout: "fuse_splice_write": "true", 2024-09-20T22:42:36.262 INFO:teuthology.orchestra.run.smithi084.stdout: "fuse_syncfs_on_mksnap": "true", 2024-09-20T22:42:36.262 INFO:teuthology.orchestra.run.smithi084.stdout: "fuse_use_invalidate_cb": "true", 2024-09-20T22:42:36.262 INFO:teuthology.orchestra.run.smithi084.stdout: "gss_ktab_client_file": "/var/lib/ceph/mon.a/gss_client_mon.a.ktab", 2024-09-20T22:42:36.262 INFO:teuthology.orchestra.run.smithi084.stdout: "gss_target_name": "ceph", 2024-09-20T22:42:36.262 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_file": "", 2024-09-20T22:42:36.262 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_inject_failure": "0", 2024-09-20T22:42:36.262 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_interval": "5", 2024-09-20T22:42:36.262 INFO:teuthology.orchestra.run.smithi084.stdout: "host": "smithi084", 2024-09-20T22:42:36.262 INFO:teuthology.orchestra.run.smithi084.stdout: "immutable_object_cache_client_dedicated_thread_num": "2", 2024-09-20T22:42:36.262 INFO:teuthology.orchestra.run.smithi084.stdout: "immutable_object_cache_max_inflight_ops": "128", 2024-09-20T22:42:36.262 INFO:teuthology.orchestra.run.smithi084.stdout: "immutable_object_cache_max_size": "1073741824", 2024-09-20T22:42:36.262 INFO:teuthology.orchestra.run.smithi084.stdout: "immutable_object_cache_path": "/tmp/ceph_immutable_object_cache", 2024-09-20T22:42:36.263 INFO:teuthology.orchestra.run.smithi084.stdout: "immutable_object_cache_qos_bps_burst": "0", 2024-09-20T22:42:36.263 INFO:teuthology.orchestra.run.smithi084.stdout: "immutable_object_cache_qos_bps_burst_seconds": "1", 2024-09-20T22:42:36.263 INFO:teuthology.orchestra.run.smithi084.stdout: "immutable_object_cache_qos_bps_limit": "0", 2024-09-20T22:42:36.263 INFO:teuthology.orchestra.run.smithi084.stdout: "immutable_object_cache_qos_iops_burst": "0", 2024-09-20T22:42:36.263 INFO:teuthology.orchestra.run.smithi084.stdout: "immutable_object_cache_qos_iops_burst_seconds": "1", 2024-09-20T22:42:36.263 INFO:teuthology.orchestra.run.smithi084.stdout: "immutable_object_cache_qos_iops_limit": "0", 2024-09-20T22:42:36.263 INFO:teuthology.orchestra.run.smithi084.stdout: "immutable_object_cache_qos_schedule_tick_min": "50", 2024-09-20T22:42:36.263 INFO:teuthology.orchestra.run.smithi084.stdout: "immutable_object_cache_sock": "/var/run/ceph/immutable_object_cache_sock", 2024-09-20T22:42:36.263 INFO:teuthology.orchestra.run.smithi084.stdout: "immutable_object_cache_watermark": "0.900000", 2024-09-20T22:42:36.263 INFO:teuthology.orchestra.run.smithi084.stdout: "inject_early_sigterm": "false", 2024-09-20T22:42:36.263 INFO:teuthology.orchestra.run.smithi084.stdout: "jaeger_agent_port": "6799", 2024-09-20T22:42:36.263 INFO:teuthology.orchestra.run.smithi084.stdout: "jaeger_tracing_enable": "false", 2024-09-20T22:42:36.263 INFO:teuthology.orchestra.run.smithi084.stdout: "journal_aio": "true", 2024-09-20T22:42:36.263 INFO:teuthology.orchestra.run.smithi084.stdout: "journal_align_min_size": "65536", 2024-09-20T22:42:36.264 INFO:teuthology.orchestra.run.smithi084.stdout: "journal_block_align": "true", 2024-09-20T22:42:36.264 INFO:teuthology.orchestra.run.smithi084.stdout: "journal_block_size": "4096", 2024-09-20T22:42:36.264 INFO:teuthology.orchestra.run.smithi084.stdout: "journal_dio": "true", 2024-09-20T22:42:36.264 INFO:teuthology.orchestra.run.smithi084.stdout: "journal_discard": "false", 2024-09-20T22:42:36.264 INFO:teuthology.orchestra.run.smithi084.stdout: "journal_force_aio": "false", 2024-09-20T22:42:36.264 INFO:teuthology.orchestra.run.smithi084.stdout: "journal_ignore_corruption": "false", 2024-09-20T22:42:36.264 INFO:teuthology.orchestra.run.smithi084.stdout: "journal_max_write_bytes": "10485760", 2024-09-20T22:42:36.264 INFO:teuthology.orchestra.run.smithi084.stdout: "journal_max_write_entries": "100", 2024-09-20T22:42:36.264 INFO:teuthology.orchestra.run.smithi084.stdout: "journal_replay_from": "0", 2024-09-20T22:42:36.264 INFO:teuthology.orchestra.run.smithi084.stdout: "journal_throttle_high_multiple": "0.000000", 2024-09-20T22:42:36.264 INFO:teuthology.orchestra.run.smithi084.stdout: "journal_throttle_high_threshhold": "0.900000", 2024-09-20T22:42:36.264 INFO:teuthology.orchestra.run.smithi084.stdout: "journal_throttle_low_threshhold": "0.600000", 2024-09-20T22:42:36.264 INFO:teuthology.orchestra.run.smithi084.stdout: "journal_throttle_max_multiple": "0.000000", 2024-09-20T22:42:36.264 INFO:teuthology.orchestra.run.smithi084.stdout: "journal_write_header_frequency": "0", 2024-09-20T22:42:36.264 INFO:teuthology.orchestra.run.smithi084.stdout: "journal_zero_on_create": "false", 2024-09-20T22:42:36.265 INFO:teuthology.orchestra.run.smithi084.stdout: "journaler_prefetch_periods": "10", 2024-09-20T22:42:36.265 INFO:teuthology.orchestra.run.smithi084.stdout: "journaler_prezero_periods": "5", 2024-09-20T22:42:36.265 INFO:teuthology.orchestra.run.smithi084.stdout: "journaler_write_head_interval": "15", 2024-09-20T22:42:36.265 INFO:teuthology.orchestra.run.smithi084.stdout: "key": "", 2024-09-20T22:42:36.265 INFO:teuthology.orchestra.run.smithi084.stdout: "keyfile": "", 2024-09-20T22:42:36.265 INFO:teuthology.orchestra.run.smithi084.stdout: "keyring": "/var/lib/ceph/mon/ceph-a/keyring", 2024-09-20T22:42:36.265 INFO:teuthology.orchestra.run.smithi084.stdout: "kstore_backend": "rocksdb", 2024-09-20T22:42:36.265 INFO:teuthology.orchestra.run.smithi084.stdout: "kstore_default_stripe_size": "65536", 2024-09-20T22:42:36.265 INFO:teuthology.orchestra.run.smithi084.stdout: "kstore_fsck_on_mount": "false", 2024-09-20T22:42:36.265 INFO:teuthology.orchestra.run.smithi084.stdout: "kstore_fsck_on_mount_deep": "true", 2024-09-20T22:42:36.265 INFO:teuthology.orchestra.run.smithi084.stdout: "kstore_max_bytes": "67108864", 2024-09-20T22:42:36.265 INFO:teuthology.orchestra.run.smithi084.stdout: "kstore_max_ops": "512", 2024-09-20T22:42:36.265 INFO:teuthology.orchestra.run.smithi084.stdout: "kstore_nid_prealloc": "1024", 2024-09-20T22:42:36.265 INFO:teuthology.orchestra.run.smithi084.stdout: "kstore_onode_map_size": "1024", 2024-09-20T22:42:36.265 INFO:teuthology.orchestra.run.smithi084.stdout: "kstore_rocksdb_options": "compression=kNoCompression", 2024-09-20T22:42:36.265 INFO:teuthology.orchestra.run.smithi084.stdout: "kstore_sync_submit_transaction": "false", 2024-09-20T22:42:36.266 INFO:teuthology.orchestra.run.smithi084.stdout: "kstore_sync_transaction": "false", 2024-09-20T22:42:36.266 INFO:teuthology.orchestra.run.smithi084.stdout: "librados_thread_count": "2", 2024-09-20T22:42:36.266 INFO:teuthology.orchestra.run.smithi084.stdout: "lockdep": "false", 2024-09-20T22:42:36.266 INFO:teuthology.orchestra.run.smithi084.stdout: "lockdep_force_backtrace": "false", 2024-09-20T22:42:36.266 INFO:teuthology.orchestra.run.smithi084.stdout: "log_coarse_timestamps": "true", 2024-09-20T22:42:36.266 INFO:teuthology.orchestra.run.smithi084.stdout: "log_file": "/var/log/ceph/ceph-mon.a.log", 2024-09-20T22:42:36.266 INFO:teuthology.orchestra.run.smithi084.stdout: "log_flush_on_exit": "false", 2024-09-20T22:42:36.266 INFO:teuthology.orchestra.run.smithi084.stdout: "log_graylog_host": "127.0.0.1", 2024-09-20T22:42:36.266 INFO:teuthology.orchestra.run.smithi084.stdout: "log_graylog_port": "12201", 2024-09-20T22:42:36.266 INFO:teuthology.orchestra.run.smithi084.stdout: "log_max_new": "1000", 2024-09-20T22:42:36.266 INFO:teuthology.orchestra.run.smithi084.stdout: "log_max_recent": "10000", 2024-09-20T22:42:36.266 INFO:teuthology.orchestra.run.smithi084.stdout: "log_stderr_prefix": "", 2024-09-20T22:42:36.266 INFO:teuthology.orchestra.run.smithi084.stdout: "log_stop_at_utilization": "0.970000", 2024-09-20T22:42:36.266 INFO:teuthology.orchestra.run.smithi084.stdout: "log_to_file": "true", 2024-09-20T22:42:36.267 INFO:teuthology.orchestra.run.smithi084.stdout: "log_to_graylog": "false", 2024-09-20T22:42:36.267 INFO:teuthology.orchestra.run.smithi084.stdout: "log_to_journald": "false", 2024-09-20T22:42:36.267 INFO:teuthology.orchestra.run.smithi084.stdout: "log_to_stderr": "false", 2024-09-20T22:42:36.267 INFO:teuthology.orchestra.run.smithi084.stdout: "log_to_syslog": "false", 2024-09-20T22:42:36.267 INFO:teuthology.orchestra.run.smithi084.stdout: "max_rotating_auth_attempts": "10", 2024-09-20T22:42:36.267 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_abort_on_newly_corrupt_dentry": "true", 2024-09-20T22:42:36.267 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_action_on_write_error": "1", 2024-09-20T22:42:36.267 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_alternate_name_max": "8192", 2024-09-20T22:42:36.267 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_asio_thread_count": "2", 2024-09-20T22:42:36.267 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_bal_export_pin": "true", 2024-09-20T22:42:36.267 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_bal_fragment_dirs": "true", 2024-09-20T22:42:36.267 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_bal_fragment_fast_factor": "1.500000", 2024-09-20T22:42:36.267 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_bal_fragment_interval": "5", 2024-09-20T22:42:36.267 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_bal_fragment_size_max": "100000", 2024-09-20T22:42:36.268 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_bal_idle_threshold": "0.000000", 2024-09-20T22:42:36.268 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_bal_interval": "10", 2024-09-20T22:42:36.268 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_bal_max": "-1", 2024-09-20T22:42:36.268 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_bal_max_until": "-1", 2024-09-20T22:42:36.268 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_bal_merge_size": "50", 2024-09-20T22:42:36.268 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_bal_midchunk": "0.300000", 2024-09-20T22:42:36.268 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_bal_min_rebalance": "0.100000", 2024-09-20T22:42:36.268 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_bal_min_start": "0.200000", 2024-09-20T22:42:36.268 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_bal_minchunk": "0.001000", 2024-09-20T22:42:36.268 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_bal_mode": "0", 2024-09-20T22:42:36.268 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_bal_need_max": "1.200000", 2024-09-20T22:42:36.268 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_bal_need_min": "0.800000", 2024-09-20T22:42:36.268 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_bal_overload_epochs": "2", 2024-09-20T22:42:36.268 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_bal_replicate_threshold": "8000.000000", 2024-09-20T22:42:36.268 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_bal_sample_interval": "3.000000", 2024-09-20T22:42:36.268 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_bal_split_bits": "3", 2024-09-20T22:42:36.269 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_bal_split_rd": "25000.000000", 2024-09-20T22:42:36.269 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_bal_split_size": "10000", 2024-09-20T22:42:36.269 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_bal_split_wr": "10000.000000", 2024-09-20T22:42:36.269 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_bal_target_decay": "10.000000", 2024-09-20T22:42:36.269 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_bal_unreplicate_threshold": "0.000000", 2024-09-20T22:42:36.269 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_beacon_grace": "15.000000", 2024-09-20T22:42:36.269 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_beacon_interval": "4.000000", 2024-09-20T22:42:36.269 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_beacon_mon_down_grace": "60", 2024-09-20T22:42:36.269 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_cache_memory_limit": "4294967296", 2024-09-20T22:42:36.269 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_cache_mid": "0.700000", 2024-09-20T22:42:36.269 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_cache_quiesce_decay_rate": "1.000000", 2024-09-20T22:42:36.269 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_cache_quiesce_delay": "0", 2024-09-20T22:42:36.269 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_cache_quiesce_sleep": "200", 2024-09-20T22:42:36.270 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_cache_quiesce_splitauth": "true", 2024-09-20T22:42:36.270 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_cache_quiesce_threshold": "524288", 2024-09-20T22:42:36.270 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_cache_release_free_interval": "10", 2024-09-20T22:42:36.270 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_cache_reservation": "0.050000", 2024-09-20T22:42:36.270 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_cache_trim_decay_rate": "1.000000", 2024-09-20T22:42:36.270 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_cache_trim_interval": "1", 2024-09-20T22:42:36.270 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_cache_trim_threshold": "262144", 2024-09-20T22:42:36.270 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_cap_acquisition_throttle_retry_request_timeout": "0.500000", 2024-09-20T22:42:36.270 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_cap_revoke_eviction_timeout": "0.000000", 2024-09-20T22:42:36.270 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_client_delegate_inos_pct": "50", 2024-09-20T22:42:36.270 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_client_prealloc_inos": "1000", 2024-09-20T22:42:36.270 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_client_writeable_range_max_inc_objs": "1024", 2024-09-20T22:42:36.271 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_connect_bootstrapping": "false", 2024-09-20T22:42:36.271 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_damage_table_max_entries": "10000", 2024-09-20T22:42:36.271 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_data": "/var/lib/ceph/mds/ceph-a", 2024-09-20T22:42:36.271 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_debug_auth_pins": "false", 2024-09-20T22:42:36.271 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_debug_frag": "false", 2024-09-20T22:42:36.271 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_debug_scatterstat": "false", 2024-09-20T22:42:36.271 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_debug_subtrees": "false", 2024-09-20T22:42:36.271 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_decay_halflife": "5.000000", 2024-09-20T22:42:36.271 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_default_dir_hash": "2", 2024-09-20T22:42:36.271 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_defer_session_stale": "true", 2024-09-20T22:42:36.271 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_deny_all_reconnect": "false", 2024-09-20T22:42:36.271 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_dir_keys_per_op": "16384", 2024-09-20T22:42:36.271 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_dir_max_commit_size": "10", 2024-09-20T22:42:36.272 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_dir_max_entries": "0", 2024-09-20T22:42:36.272 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_dir_prefetch": "true", 2024-09-20T22:42:36.272 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_dirstat_min_interval": "1.000000", 2024-09-20T22:42:36.272 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_dump_cache_after_rejoin": "false", 2024-09-20T22:42:36.272 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_dump_cache_on_map": "false", 2024-09-20T22:42:36.272 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_dump_cache_threshold_file": "0", 2024-09-20T22:42:36.272 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_dump_cache_threshold_formatter": "1073741824", 2024-09-20T22:42:36.272 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_early_reply": "true", 2024-09-20T22:42:36.272 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_enable_op_tracker": "true", 2024-09-20T22:42:36.272 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_enforce_unique_name": "true", 2024-09-20T22:42:36.272 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_export_ephemeral_distributed": "true", 2024-09-20T22:42:36.272 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_export_ephemeral_distributed_factor": "2.000000", 2024-09-20T22:42:36.272 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_export_ephemeral_random": "true", 2024-09-20T22:42:36.272 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_export_ephemeral_random_max": "0.010000", 2024-09-20T22:42:36.272 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_extraordinary_events_dump_interval": "0", 2024-09-20T22:42:36.273 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_forward_all_requests_to_auth": "false", 2024-09-20T22:42:36.273 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_freeze_tree_timeout": "30.000000", 2024-09-20T22:42:36.273 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_fscrypt_last_block_max_size": "4096", 2024-09-20T22:42:36.273 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_go_bad_corrupt_dentry": "true", 2024-09-20T22:42:36.273 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_hack_allow_loading_invalid_metadata": "false", 2024-09-20T22:42:36.273 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_health_cache_threshold": "1.500000", 2024-09-20T22:42:36.273 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_health_summarize_threshold": "10", 2024-09-20T22:42:36.273 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_heartbeat_grace": "15.000000", 2024-09-20T22:42:36.273 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_heartbeat_reset_grace": "1000", 2024-09-20T22:42:36.273 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_inject_health_dummy": "false", 2024-09-20T22:42:36.273 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_inject_journal_corrupt_dentry_first": "0.000000", 2024-09-20T22:42:36.273 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_inject_migrator_session_race": "false", 2024-09-20T22:42:36.273 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_inject_rename_corrupt_dentry_first": "0.000000", 2024-09-20T22:42:36.273 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_inject_skip_replaying_inotable": "false", 2024-09-20T22:42:36.274 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_inject_traceless_reply_probability": "0.000000", 2024-09-20T22:42:36.274 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_join_fs": "", 2024-09-20T22:42:36.274 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_journal_format": "1", 2024-09-20T22:42:36.274 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_kill_after_journal_logs_flushed": "false", 2024-09-20T22:42:36.274 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_kill_create_at": "0", 2024-09-20T22:42:36.274 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_kill_export_at": "0", 2024-09-20T22:42:36.274 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_kill_import_at": "0", 2024-09-20T22:42:36.274 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_kill_journal_at": "0", 2024-09-20T22:42:36.274 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_kill_journal_expire_at": "0", 2024-09-20T22:42:36.274 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_kill_journal_replay_at": "0", 2024-09-20T22:42:36.274 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_kill_link_at": "0", 2024-09-20T22:42:36.274 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_kill_mdstable_at": "0", 2024-09-20T22:42:36.274 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_kill_openc_at": "0", 2024-09-20T22:42:36.275 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_kill_rename_at": "0", 2024-09-20T22:42:36.275 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_kill_shutdown_at": "0", 2024-09-20T22:42:36.275 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_log_event_large_threshold": "524288", 2024-09-20T22:42:36.275 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_log_events_per_segment": "1024", 2024-09-20T22:42:36.275 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_log_major_segment_event_ratio": "12", 2024-09-20T22:42:36.275 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_log_max_events": "-1", 2024-09-20T22:42:36.275 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_log_max_segments": "128", 2024-09-20T22:42:36.275 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_log_pause": "false", 2024-09-20T22:42:36.275 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_log_segment_size": "0", 2024-09-20T22:42:36.275 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_log_skip_corrupt_events": "false", 2024-09-20T22:42:36.275 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_log_skip_unbounded_events": "false", 2024-09-20T22:42:36.275 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_log_trim_decay_rate": "1.000000", 2024-09-20T22:42:36.275 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_log_trim_threshold": "128", 2024-09-20T22:42:36.275 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_log_trim_upkeep_interval": "1000", 2024-09-20T22:42:36.275 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_log_warn_factor": "2.000000", 2024-09-20T22:42:36.276 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_max_caps_per_client": "1048576", 2024-09-20T22:42:36.276 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_max_completed_flushes": "100000", 2024-09-20T22:42:36.276 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_max_completed_requests": "100000", 2024-09-20T22:42:36.276 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_max_export_size": "20971520", 2024-09-20T22:42:36.276 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_max_file_recover": "32", 2024-09-20T22:42:36.276 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_max_purge_files": "64", 2024-09-20T22:42:36.276 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_max_purge_ops": "8192", 2024-09-20T22:42:36.276 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_max_purge_ops_per_pg": "0.500000", 2024-09-20T22:42:36.276 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_max_scrub_ops_in_progress": "5", 2024-09-20T22:42:36.276 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_max_snaps_per_dir": "100", 2024-09-20T22:42:36.276 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_metrics_update_interval": "2", 2024-09-20T22:42:36.276 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_min_caps_per_client": "100", 2024-09-20T22:42:36.276 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_min_caps_working_set": "10000", 2024-09-20T22:42:36.277 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_mon_shutdown_timeout": "5.000000", 2024-09-20T22:42:36.277 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_numa_node": "-1", 2024-09-20T22:42:36.277 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_oft_prefetch_dirfrags": "false", 2024-09-20T22:42:36.277 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_op_complaint_time": "30.000000", 2024-09-20T22:42:36.277 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_op_history_duration": "600", 2024-09-20T22:42:36.277 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_op_history_size": "20", 2024-09-20T22:42:36.277 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_op_history_slow_op_size": "20", 2024-09-20T22:42:36.277 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_op_history_slow_op_threshold": "10", 2024-09-20T22:42:36.277 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_op_log_threshold": "5", 2024-09-20T22:42:36.277 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_ping_grace": "15", 2024-09-20T22:42:36.277 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_ping_interval": "5", 2024-09-20T22:42:36.277 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_purge_queue_busy_flush_period": "1.000000", 2024-09-20T22:42:36.277 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_recall_global_max_decay_threshold": "131072", 2024-09-20T22:42:36.277 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_recall_max_caps": "30000", 2024-09-20T22:42:36.277 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_recall_max_decay_rate": "1.500000", 2024-09-20T22:42:36.278 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_recall_max_decay_threshold": "131072", 2024-09-20T22:42:36.278 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_recall_warning_decay_rate": "60.000000", 2024-09-20T22:42:36.278 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_recall_warning_threshold": "262144", 2024-09-20T22:42:36.278 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_reconnect_timeout": "45.000000", 2024-09-20T22:42:36.278 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_replay_interval": "1.000000", 2024-09-20T22:42:36.278 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_replay_unsafe_with_closed_session": "false", 2024-09-20T22:42:36.278 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_request_load_average_decay_rate": "60.000000", 2024-09-20T22:42:36.278 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_root_ino_gid": "0", 2024-09-20T22:42:36.278 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_root_ino_uid": "0", 2024-09-20T22:42:36.278 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_scatter_nudge_interval": "5.000000", 2024-09-20T22:42:36.278 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_session_blocklist_on_evict": "true", 2024-09-20T22:42:36.278 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_session_blocklist_on_timeout": "true", 2024-09-20T22:42:36.278 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_session_cache_liveness_decay_rate": "300.000000", 2024-09-20T22:42:36.278 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_session_cache_liveness_magnitude": "10", 2024-09-20T22:42:36.278 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_session_cap_acquisition_decay_rate": "30.000000", 2024-09-20T22:42:36.278 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_session_cap_acquisition_throttle": "100000", 2024-09-20T22:42:36.279 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_session_max_caps_throttle_ratio": "1.100000", 2024-09-20T22:42:36.279 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_session_metadata_threshold": "16777216", 2024-09-20T22:42:36.279 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_sessionmap_keys_per_op": "1024", 2024-09-20T22:42:36.279 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_shutdown_check": "0", 2024-09-20T22:42:36.279 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_skip_ino": "0", 2024-09-20T22:42:36.279 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_sleep_rank_change": "0.000000", 2024-09-20T22:42:36.279 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_snap_max_uid": "4294967294", 2024-09-20T22:42:36.279 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_snap_min_uid": "0", 2024-09-20T22:42:36.279 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_snap_rstat": "false", 2024-09-20T22:42:36.279 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_standby_replay_damaged": "false", 2024-09-20T22:42:36.279 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_symlink_recovery": "true", 2024-09-20T22:42:36.279 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_task_status_update_interval": "2.000000", 2024-09-20T22:42:36.279 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_thrash_exports": "0", 2024-09-20T22:42:36.279 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_thrash_fragments": "0", 2024-09-20T22:42:36.279 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_tick_interval": "5.000000", 2024-09-20T22:42:36.279 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_valgrind_exit": "false", 2024-09-20T22:42:36.280 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_verify_backtrace": "1", 2024-09-20T22:42:36.280 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_verify_scatter": "false", 2024-09-20T22:42:36.280 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_wipe_ino_prealloc": "false", 2024-09-20T22:42:36.280 INFO:teuthology.orchestra.run.smithi084.stdout: "mds_wipe_sessions": "false", 2024-09-20T22:42:36.280 INFO:teuthology.orchestra.run.smithi084.stdout: "mempool_debug": "false", 2024-09-20T22:42:36.280 INFO:teuthology.orchestra.run.smithi084.stdout: "memstore_debug_omit_block_device_write": "false", 2024-09-20T22:42:36.280 INFO:teuthology.orchestra.run.smithi084.stdout: "memstore_device_bytes": "1073741824", 2024-09-20T22:42:36.280 INFO:teuthology.orchestra.run.smithi084.stdout: "memstore_page_set": "false", 2024-09-20T22:42:36.280 INFO:teuthology.orchestra.run.smithi084.stdout: "memstore_page_size": "65536", 2024-09-20T22:42:36.280 INFO:teuthology.orchestra.run.smithi084.stdout: "mgr_client_bytes": "134217728", 2024-09-20T22:42:36.280 INFO:teuthology.orchestra.run.smithi084.stdout: "mgr_client_messages": "512", 2024-09-20T22:42:36.280 INFO:teuthology.orchestra.run.smithi084.stdout: "mgr_client_service_daemon_unregister_timeout": "1.000000", 2024-09-20T22:42:36.280 INFO:teuthology.orchestra.run.smithi084.stdout: "mgr_connect_retry_interval": "1.000000", 2024-09-20T22:42:36.280 INFO:teuthology.orchestra.run.smithi084.stdout: "mgr_data": "/var/lib/ceph/mgr/ceph-a", 2024-09-20T22:42:36.280 INFO:teuthology.orchestra.run.smithi084.stdout: "mgr_debug_aggressive_pg_num_changes": "false", 2024-09-20T22:42:36.281 INFO:teuthology.orchestra.run.smithi084.stdout: "mgr_disabled_modules": "", 2024-09-20T22:42:36.281 INFO:teuthology.orchestra.run.smithi084.stdout: "mgr_enable_op_tracker": "true", 2024-09-20T22:42:36.281 INFO:teuthology.orchestra.run.smithi084.stdout: "mgr_initial_modules": "restful iostat nfs", 2024-09-20T22:42:36.281 INFO:teuthology.orchestra.run.smithi084.stdout: "mgr_max_pg_num_change": "128", 2024-09-20T22:42:36.281 INFO:teuthology.orchestra.run.smithi084.stdout: "mgr_mds_bytes": "134217728", 2024-09-20T22:42:36.281 INFO:teuthology.orchestra.run.smithi084.stdout: "mgr_mds_messages": "128", 2024-09-20T22:42:36.281 INFO:teuthology.orchestra.run.smithi084.stdout: "mgr_module_path": "/usr/share/ceph/mgr", 2024-09-20T22:42:36.281 INFO:teuthology.orchestra.run.smithi084.stdout: "mgr_mon_bytes": "134217728", 2024-09-20T22:42:36.281 INFO:teuthology.orchestra.run.smithi084.stdout: "mgr_mon_messages": "128", 2024-09-20T22:42:36.281 INFO:teuthology.orchestra.run.smithi084.stdout: "mgr_num_op_tracker_shard": "32", 2024-09-20T22:42:36.281 INFO:teuthology.orchestra.run.smithi084.stdout: "mgr_op_complaint_time": "30.000000", 2024-09-20T22:42:36.281 INFO:teuthology.orchestra.run.smithi084.stdout: "mgr_op_history_duration": "600", 2024-09-20T22:42:36.281 INFO:teuthology.orchestra.run.smithi084.stdout: "mgr_op_history_size": "20", 2024-09-20T22:42:36.281 INFO:teuthology.orchestra.run.smithi084.stdout: "mgr_op_history_slow_op_size": "20", 2024-09-20T22:42:36.281 INFO:teuthology.orchestra.run.smithi084.stdout: "mgr_op_history_slow_op_threshold": "10.000000", 2024-09-20T22:42:36.282 INFO:teuthology.orchestra.run.smithi084.stdout: "mgr_op_log_threshold": "5", 2024-09-20T22:42:36.282 INFO:teuthology.orchestra.run.smithi084.stdout: "mgr_osd_bytes": "536870912", 2024-09-20T22:42:36.282 INFO:teuthology.orchestra.run.smithi084.stdout: "mgr_osd_messages": "8192", 2024-09-20T22:42:36.282 INFO:teuthology.orchestra.run.smithi084.stdout: "mgr_pool": "true", 2024-09-20T22:42:36.282 INFO:teuthology.orchestra.run.smithi084.stdout: "mgr_service_beacon_grace": "60.000000", 2024-09-20T22:42:36.282 INFO:teuthology.orchestra.run.smithi084.stdout: "mgr_standby_modules": "true", 2024-09-20T22:42:36.282 INFO:teuthology.orchestra.run.smithi084.stdout: "mgr_stats_period": "5", 2024-09-20T22:42:36.282 INFO:teuthology.orchestra.run.smithi084.stdout: "mgr_stats_threshold": "5", 2024-09-20T22:42:36.282 INFO:teuthology.orchestra.run.smithi084.stdout: "mgr_tick_period": "2", 2024-09-20T22:42:36.282 INFO:teuthology.orchestra.run.smithi084.stdout: "mgr_ttl_cache_expire_seconds": "0", 2024-09-20T22:42:36.282 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_accept_timeout_factor": "2.000000", 2024-09-20T22:42:36.282 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_allow_pool_delete": "true", 2024-09-20T22:42:36.283 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_allow_pool_size_one": "false", 2024-09-20T22:42:36.283 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_auth_validate_all_caps": "true", 2024-09-20T22:42:36.283 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_cache_target_full_warn_ratio": "0.660000", 2024-09-20T22:42:36.283 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_clean_pg_upmaps_per_chunk": "256", 2024-09-20T22:42:36.283 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_client_bytes": "104857600", 2024-09-20T22:42:36.283 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_client_directed_command_retry": "2", 2024-09-20T22:42:36.283 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_client_hunt_interval": "3.000000", 2024-09-20T22:42:36.283 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_client_hunt_interval_backoff": "1.500000", 2024-09-20T22:42:36.283 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_client_hunt_interval_max_multiple": "10.000000", 2024-09-20T22:42:36.283 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_client_hunt_interval_min_multiple": "1.000000", 2024-09-20T22:42:36.283 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_client_hunt_parallel": "3", 2024-09-20T22:42:36.283 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_client_log_interval": "1.000000", 2024-09-20T22:42:36.283 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_client_max_log_entries_per_message": "1000", 2024-09-20T22:42:36.283 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_client_ping_interval": "10.000000", 2024-09-20T22:42:36.283 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_client_ping_timeout": "30.000000", 2024-09-20T22:42:36.283 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_client_target_rank": "-1", 2024-09-20T22:42:36.284 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_clock_drift_allowed": "1.000000", 2024-09-20T22:42:36.284 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_clock_drift_warn_backoff": "5.000000", 2024-09-20T22:42:36.284 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", 2024-09-20T22:42:36.284 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_cluster_log_file_level": "debug", 2024-09-20T22:42:36.284 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_cluster_log_to_file": "true", 2024-09-20T22:42:36.284 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_cluster_log_to_graylog": "false", 2024-09-20T22:42:36.284 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_cluster_log_to_graylog_host": "127.0.0.1", 2024-09-20T22:42:36.284 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_cluster_log_to_graylog_port": "12201", 2024-09-20T22:42:36.284 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_cluster_log_to_journald": "true", 2024-09-20T22:42:36.284 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_cluster_log_to_stderr": "false", 2024-09-20T22:42:36.284 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_cluster_log_to_syslog": "default=false", 2024-09-20T22:42:36.284 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_cluster_log_to_syslog_facility": "daemon", 2024-09-20T22:42:36.284 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_cluster_log_to_syslog_level": "info", 2024-09-20T22:42:36.284 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_compact_on_bootstrap": "false", 2024-09-20T22:42:36.284 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_compact_on_start": "false", 2024-09-20T22:42:36.285 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_compact_on_trim": "true", 2024-09-20T22:42:36.285 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_con_tracker_persist_interval": "10", 2024-09-20T22:42:36.285 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_con_tracker_score_halflife": "43200", 2024-09-20T22:42:36.285 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_config_key_max_entry_size": "65536", 2024-09-20T22:42:36.285 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_cpu_threads": "4", 2024-09-20T22:42:36.285 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_crush_min_required_version": "hammer", 2024-09-20T22:42:36.285 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_daemon_bytes": "419430400", 2024-09-20T22:42:36.285 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_data": "/var/lib/ceph/mon/ceph-a", 2024-09-20T22:42:36.285 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_data_avail_crit": "5", 2024-09-20T22:42:36.285 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_data_avail_warn": "5", 2024-09-20T22:42:36.285 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_data_size_warn": "16106127360", 2024-09-20T22:42:36.285 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_debug_block_osdmap_trim": "false", 2024-09-20T22:42:36.285 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_debug_deprecated_as_obsolete": "false", 2024-09-20T22:42:36.285 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_debug_dump_json": "false", 2024-09-20T22:42:36.285 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_debug_dump_location": "/var/log/ceph/ceph-mon.a.tdump", 2024-09-20T22:42:36.285 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_debug_dump_transactions": "false", 2024-09-20T22:42:36.286 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_debug_extra_checks": "false", 2024-09-20T22:42:36.286 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_debug_no_initial_persistent_features": "false", 2024-09-20T22:42:36.286 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_debug_no_require_bluestore_for_ec_overwrites": "false", 2024-09-20T22:42:36.286 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_debug_no_require_reef": "false", 2024-09-20T22:42:36.286 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_debug_no_require_squid": "false", 2024-09-20T22:42:36.286 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_debug_unsafe_allow_tier_with_nonempty_snaps": "false", 2024-09-20T22:42:36.286 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_delta_reset_interval": "10.000000", 2024-09-20T22:42:36.286 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_dns_srv_name": "ceph-mon", 2024-09-20T22:42:36.286 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_down_mkfs_grace": "60", 2024-09-20T22:42:36.286 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_down_uptime_grace": "60", 2024-09-20T22:42:36.286 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_election_default_strategy": "1", 2024-09-20T22:42:36.286 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_election_timeout": "5.000000", 2024-09-20T22:42:36.286 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_elector_ignore_propose_margin": "0.000500", 2024-09-20T22:42:36.286 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_elector_ping_divisor": "2", 2024-09-20T22:42:36.286 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_elector_ping_timeout": "2.000000", 2024-09-20T22:42:36.287 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_enable_op_tracker": "true", 2024-09-20T22:42:36.287 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_fake_pool_delete": "false", 2024-09-20T22:42:36.287 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_force_quorum_join": "false", 2024-09-20T22:42:36.287 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_fsmap_prune_threshold": "300", 2024-09-20T22:42:36.287 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_globalid_prealloc": "10000", 2024-09-20T22:42:36.287 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_health_detail_to_clog": "true", 2024-09-20T22:42:36.287 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_health_log_update_period": "5", 2024-09-20T22:42:36.287 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_health_max_detail": "50", 2024-09-20T22:42:36.287 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_health_to_clog": "true", 2024-09-20T22:42:36.287 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_health_to_clog_interval": "600", 2024-09-20T22:42:36.287 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_health_to_clog_tick_interval": "60.000000", 2024-09-20T22:42:36.287 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_host": "[v2:172.21.15.84:3300/0,v1:172.21.15.84:6789/0]", 2024-09-20T22:42:36.287 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_host_override": "", 2024-09-20T22:42:36.287 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_initial_members": "", 2024-09-20T22:42:36.288 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_inject_pg_merge_bounce_probability": "0.000000", 2024-09-20T22:42:36.288 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_inject_sync_get_chunk_delay": "0.000000", 2024-09-20T22:42:36.288 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_inject_transaction_delay_max": "10.000000", 2024-09-20T22:42:36.288 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_inject_transaction_delay_probability": "0.000000", 2024-09-20T22:42:36.288 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_keyvaluedb": "rocksdb", 2024-09-20T22:42:36.288 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_lease": "5.000000", 2024-09-20T22:42:36.288 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_lease_ack_timeout_factor": "2.000000", 2024-09-20T22:42:36.288 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_lease_renew_interval_factor": "0.600000", 2024-09-20T22:42:36.288 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_log_full_interval": "50", 2024-09-20T22:42:36.288 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_log_max": "10000", 2024-09-20T22:42:36.288 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_log_max_summary": "50", 2024-09-20T22:42:36.288 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_max_log_entries_per_event": "4096", 2024-09-20T22:42:36.288 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_max_log_epochs": "500", 2024-09-20T22:42:36.289 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_max_mdsmap_epochs": "500", 2024-09-20T22:42:36.289 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_max_mgrmap_epochs": "500", 2024-09-20T22:42:36.289 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_max_osd": "10000", 2024-09-20T22:42:36.289 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_max_pg_per_osd": "10000", 2024-09-20T22:42:36.289 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_max_pool_pg_num": "65536", 2024-09-20T22:42:36.289 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_max_snap_prune_per_epoch": "100", 2024-09-20T22:42:36.289 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_mds_blocklist_interval": "86400.000000", 2024-09-20T22:42:36.289 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_mds_force_trim_to": "0", 2024-09-20T22:42:36.289 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_mds_skip_sanity": "false", 2024-09-20T22:42:36.289 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_memory_autotune": "true", 2024-09-20T22:42:36.289 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_memory_target": "2147483648", 2024-09-20T22:42:36.289 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_mgr_beacon_grace": "30", 2024-09-20T22:42:36.289 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_mgr_blocklist_interval": "86400.000000", 2024-09-20T22:42:36.289 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_mgr_digest_period": "5", 2024-09-20T22:42:36.290 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_mgr_inactive_grace": "60", 2024-09-20T22:42:36.290 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_mgr_mkfs_grace": "240", 2024-09-20T22:42:36.290 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_mgr_proxy_client_bytes_ratio": "0.300000", 2024-09-20T22:42:36.290 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_min_osdmap_epochs": "50", 2024-09-20T22:42:36.290 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_op_complaint_time": "30", 2024-09-20T22:42:36.290 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_op_history_duration": "600", 2024-09-20T22:42:36.290 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_op_history_size": "20", 2024-09-20T22:42:36.290 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_op_history_slow_op_size": "20", 2024-09-20T22:42:36.290 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_op_history_slow_op_threshold": "10", 2024-09-20T22:42:36.290 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_op_log_threshold": "5", 2024-09-20T22:42:36.290 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_osd_adjust_down_out_interval": "true", 2024-09-20T22:42:36.290 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_osd_adjust_heartbeat_grace": "true", 2024-09-20T22:42:36.290 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_osd_auto_mark_auto_out_in": "true", 2024-09-20T22:42:36.290 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_osd_auto_mark_in": "false", 2024-09-20T22:42:36.291 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_osd_auto_mark_new_in": "true", 2024-09-20T22:42:36.291 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_osd_backfillfull_ratio": "0.900000", 2024-09-20T22:42:36.291 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_osd_blocklist_default_expire": "3600.000000", 2024-09-20T22:42:36.291 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_osd_cache_size": "500", 2024-09-20T22:42:36.291 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_osd_cache_size_min": "134217728", 2024-09-20T22:42:36.291 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_osd_crush_smoke_test": "true", 2024-09-20T22:42:36.291 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_osd_destroyed_out_interval": "600", 2024-09-20T22:42:36.291 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_osd_down_out_interval": "600", 2024-09-20T22:42:36.291 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_osd_down_out_subtree_limit": "rack", 2024-09-20T22:42:36.291 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_osd_err_op_age_ratio": "128.000000", 2024-09-20T22:42:36.291 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_osd_force_trim_to": "0", 2024-09-20T22:42:36.291 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_osd_full_ratio": "0.950000", 2024-09-20T22:42:36.291 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_osd_initial_require_min_compat_client": "luminous", 2024-09-20T22:42:36.291 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_osd_laggy_halflife": "3600", 2024-09-20T22:42:36.291 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_osd_laggy_max_interval": "300", 2024-09-20T22:42:36.292 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_osd_laggy_weight": "0.300000", 2024-09-20T22:42:36.292 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_osd_mapping_pgs_per_chunk": "4096", 2024-09-20T22:42:36.292 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_osd_max_creating_pgs": "1024", 2024-09-20T22:42:36.292 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_osd_max_initial_pgs": "1024", 2024-09-20T22:42:36.292 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_osd_min_down_reporters": "2", 2024-09-20T22:42:36.292 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_osd_min_in_ratio": "0.750000", 2024-09-20T22:42:36.292 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_osd_min_up_ratio": "0.300000", 2024-09-20T22:42:36.292 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_osd_nearfull_ratio": "0.850000", 2024-09-20T22:42:36.292 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_osd_prime_pg_temp": "true", 2024-09-20T22:42:36.292 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_osd_prime_pg_temp_max_estimate": "0.250000", 2024-09-20T22:42:36.292 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_osd_prime_pg_temp_max_time": "0.500000", 2024-09-20T22:42:36.292 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_osd_report_timeout": "900", 2024-09-20T22:42:36.292 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_osd_reporter_subtree_level": "osd", 2024-09-20T22:42:36.292 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_osd_snap_trim_queue_warn_on": "32768", 2024-09-20T22:42:36.292 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_osd_warn_num_repaired": "10", 2024-09-20T22:42:36.293 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_osd_warn_op_age": "32.000000", 2024-09-20T22:42:36.293 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_osdmap_full_prune_enabled": "true", 2024-09-20T22:42:36.293 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_osdmap_full_prune_interval": "2", 2024-09-20T22:42:36.293 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_osdmap_full_prune_min": "15", 2024-09-20T22:42:36.293 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_osdmap_full_prune_txsize": "2", 2024-09-20T22:42:36.293 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_pg_check_down_all_threshold": "0.500000", 2024-09-20T22:42:36.293 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_pg_stuck_threshold": "60", 2024-09-20T22:42:36.293 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_pg_warn_max_object_skew": "0.000000", 2024-09-20T22:42:36.293 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_pg_warn_min_objects": "10000", 2024-09-20T22:42:36.293 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_pg_warn_min_per_osd": "0", 2024-09-20T22:42:36.293 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_pg_warn_min_pool_objects": "1000", 2024-09-20T22:42:36.293 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_pool_quota_crit_threshold": "0", 2024-09-20T22:42:36.294 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_pool_quota_warn_threshold": "0", 2024-09-20T22:42:36.295 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_probe_timeout": "2.000000", 2024-09-20T22:42:36.295 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_reweight_max_change": "0.050000", 2024-09-20T22:42:36.295 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_reweight_max_osds": "4", 2024-09-20T22:42:36.295 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_reweight_min_bytes_per_osd": "10", 2024-09-20T22:42:36.295 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_reweight_min_pgs_per_osd": "4", 2024-09-20T22:42:36.295 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", 2024-09-20T22:42:36.295 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_scrub_inject_crc_mismatch": "0.000000", 2024-09-20T22:42:36.295 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_scrub_inject_missing_keys": "0.000000", 2024-09-20T22:42:36.295 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_scrub_interval": "86400", 2024-09-20T22:42:36.295 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_scrub_max_keys": "100", 2024-09-20T22:42:36.295 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_scrub_timeout": "300", 2024-09-20T22:42:36.295 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_session_timeout": "300", 2024-09-20T22:42:36.295 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_smart_report_timeout": "5", 2024-09-20T22:42:36.296 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_stat_smooth_intervals": "6", 2024-09-20T22:42:36.296 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_stretch_cluster_recovery_ratio": "0.600000", 2024-09-20T22:42:36.296 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_stretch_pool_min_size": "2", 2024-09-20T22:42:36.296 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_stretch_pool_size": "4", 2024-09-20T22:42:36.296 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_stretch_recovery_min_wait": "15.000000", 2024-09-20T22:42:36.296 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_subscribe_interval": "86400.000000", 2024-09-20T22:42:36.296 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_sync_debug": "false", 2024-09-20T22:42:36.296 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_sync_max_payload_keys": "2000", 2024-09-20T22:42:36.296 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_sync_max_payload_size": "1048576", 2024-09-20T22:42:36.296 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_sync_provider_kill_at": "0", 2024-09-20T22:42:36.296 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_sync_requester_kill_at": "0", 2024-09-20T22:42:36.296 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_sync_timeout": "60.000000", 2024-09-20T22:42:36.296 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_target_pg_per_osd": "100", 2024-09-20T22:42:36.296 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_tick_interval": "5", 2024-09-20T22:42:36.297 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_timecheck_interval": "300.000000", 2024-09-20T22:42:36.297 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_timecheck_skew_interval": "30.000000", 2024-09-20T22:42:36.297 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_use_min_delay_socket": "false", 2024-09-20T22:42:36.297 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_warn_older_version_delay": "604800", 2024-09-20T22:42:36.297 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_warn_on_cache_pools_without_hit_sets": "true", 2024-09-20T22:42:36.297 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_warn_on_crush_straw_calc_version_zero": "false", 2024-09-20T22:42:36.297 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_warn_on_degraded_stretch_mode": "true", 2024-09-20T22:42:36.297 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_warn_on_filestore_osds": "true", 2024-09-20T22:42:36.297 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_warn_on_insecure_global_id_reclaim": "false", 2024-09-20T22:42:36.297 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_warn_on_insecure_global_id_reclaim_allowed": "false", 2024-09-20T22:42:36.297 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_warn_on_legacy_crush_tunables": "false", 2024-09-20T22:42:36.297 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_warn_on_misplaced": "false", 2024-09-20T22:42:36.297 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_warn_on_msgr2_not_enabled": "true", 2024-09-20T22:42:36.297 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_warn_on_older_version": "true", 2024-09-20T22:42:36.298 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_warn_on_osd_down_out_interval_zero": "false", 2024-09-20T22:42:36.298 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_warn_on_pool_no_app": "true", 2024-09-20T22:42:36.298 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_warn_on_pool_no_redundancy": "true", 2024-09-20T22:42:36.298 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_warn_on_pool_pg_num_not_power_of_two": "false", 2024-09-20T22:42:36.298 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_warn_on_slow_ping_ratio": "0.050000", 2024-09-20T22:42:36.298 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_warn_on_slow_ping_time": "0.000000", 2024-09-20T22:42:36.298 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_warn_on_too_few_osds": "false", 2024-09-20T22:42:36.298 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", 2024-09-20T22:42:36.298 INFO:teuthology.orchestra.run.smithi084.stdout: "mon_warn_pg_not_scrubbed_ratio": "0.500000", 2024-09-20T22:42:36.298 INFO:teuthology.orchestra.run.smithi084.stdout: "monmap": "", 2024-09-20T22:42:36.298 INFO:teuthology.orchestra.run.smithi084.stdout: "motr_admin_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-09-20T22:42:36.298 INFO:teuthology.orchestra.run.smithi084.stdout: "motr_admin_fid": "0x7200000000000001:0x0", 2024-09-20T22:42:36.298 INFO:teuthology.orchestra.run.smithi084.stdout: "motr_ha_endpoint": "192.168.180.182@tcp:12345:1:1", 2024-09-20T22:42:36.298 INFO:teuthology.orchestra.run.smithi084.stdout: "motr_my_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-09-20T22:42:36.299 INFO:teuthology.orchestra.run.smithi084.stdout: "motr_my_fid": "0x7200000000000001:0x0", 2024-09-20T22:42:36.299 INFO:teuthology.orchestra.run.smithi084.stdout: "motr_profile_fid": "0x7000000000000001:0x0", 2024-09-20T22:42:36.299 INFO:teuthology.orchestra.run.smithi084.stdout: "motr_tracing_enabled": "false", 2024-09-20T22:42:36.299 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_async_op_threads": "3", 2024-09-20T22:42:36.299 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_async_rdma_buffer_size": "131072", 2024-09-20T22:42:36.299 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_async_rdma_cm": "false", 2024-09-20T22:42:36.299 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_async_rdma_device_name": "", 2024-09-20T22:42:36.299 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_async_rdma_dscp": "96", 2024-09-20T22:42:36.299 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_async_rdma_enable_hugepage": "false", 2024-09-20T22:42:36.299 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_async_rdma_gid_idx": "0", 2024-09-20T22:42:36.299 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_async_rdma_local_gid": "", 2024-09-20T22:42:36.299 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_async_rdma_polling_us": "1000", 2024-09-20T22:42:36.300 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_async_rdma_port_num": "1", 2024-09-20T22:42:36.300 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_async_rdma_receive_buffers": "32768", 2024-09-20T22:42:36.300 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_async_rdma_receive_queue_len": "4096", 2024-09-20T22:42:36.300 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_async_rdma_roce_ver": "1", 2024-09-20T22:42:36.300 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_async_rdma_send_buffers": "1024", 2024-09-20T22:42:36.300 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_async_rdma_sl": "3", 2024-09-20T22:42:36.300 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_async_rdma_support_srq": "true", 2024-09-20T22:42:36.300 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_async_rdma_type": "ib", 2024-09-20T22:42:36.300 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_async_reap_threshold": "5", 2024-09-20T22:42:36.300 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_bind_before_connect": "false", 2024-09-20T22:42:36.300 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_bind_ipv4": "true", 2024-09-20T22:42:36.300 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_bind_ipv6": "false", 2024-09-20T22:42:36.300 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_bind_msgr1": "false", 2024-09-20T22:42:36.300 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_bind_msgr2": "true", 2024-09-20T22:42:36.301 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_bind_port_max": "7568", 2024-09-20T22:42:36.301 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_bind_port_min": "6800", 2024-09-20T22:42:36.301 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_bind_prefer_ipv4": "false", 2024-09-20T22:42:36.301 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_bind_retry_count": "3", 2024-09-20T22:42:36.301 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_bind_retry_delay": "5", 2024-09-20T22:42:36.301 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_blackhole_client": "false", 2024-09-20T22:42:36.301 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_blackhole_mds": "false", 2024-09-20T22:42:36.301 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_blackhole_mgr": "false", 2024-09-20T22:42:36.301 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_blackhole_mon": "false", 2024-09-20T22:42:36.301 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_blackhole_osd": "false", 2024-09-20T22:42:36.301 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_client_mode": "crc secure", 2024-09-20T22:42:36.301 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_cluster_mode": "crc secure", 2024-09-20T22:42:36.301 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_cluster_type": "", 2024-09-20T22:42:36.302 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_compress_secure": "false", 2024-09-20T22:42:36.302 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_connection_idle_timeout": "900", 2024-09-20T22:42:36.302 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_connection_ready_timeout": "10", 2024-09-20T22:42:36.302 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_crc_data": "true", 2024-09-20T22:42:36.302 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_crc_header": "true", 2024-09-20T22:42:36.302 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_die_on_bad_msg": "false", 2024-09-20T22:42:36.302 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_die_on_bug": "true", 2024-09-20T22:42:36.302 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_die_on_old_message": "true", 2024-09-20T22:42:36.302 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_die_on_skipped_message": "false", 2024-09-20T22:42:36.302 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_die_on_unhandled_msg": "false", 2024-09-20T22:42:36.302 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_dispatch_throttle_bytes": "104857600", 2024-09-20T22:42:36.302 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_dpdk_coremask": "0xF", 2024-09-20T22:42:36.302 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_dpdk_debug_allow_loopback": "false", 2024-09-20T22:42:36.302 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_dpdk_devs_allowlist": "", 2024-09-20T22:42:36.302 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_dpdk_enable_tso": "true", 2024-09-20T22:42:36.303 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_dpdk_gateway_ipv4_addr": "", 2024-09-20T22:42:36.303 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_dpdk_host_ipv4_addr": "", 2024-09-20T22:42:36.303 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_dpdk_hugepages": "", 2024-09-20T22:42:36.303 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_dpdk_hw_flow_control": "true", 2024-09-20T22:42:36.303 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_dpdk_hw_queue_weight": "1.000000", 2024-09-20T22:42:36.303 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_dpdk_lro": "true", 2024-09-20T22:42:36.303 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_dpdk_memory_channel": "4", 2024-09-20T22:42:36.303 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_dpdk_netmask_ipv4_addr": "", 2024-09-20T22:42:36.303 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_dpdk_pmd": "", 2024-09-20T22:42:36.303 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_dpdk_port_id": "0", 2024-09-20T22:42:36.303 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_dpdk_rx_buffer_count_per_core": "8192", 2024-09-20T22:42:36.303 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_dump_corrupt_message_level": "1", 2024-09-20T22:42:36.303 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_dump_on_send": "false", 2024-09-20T22:42:36.303 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_initial_backoff": "0.200000", 2024-09-20T22:42:36.304 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_inject_delay_max": "1.000000", 2024-09-20T22:42:36.304 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_inject_delay_probability": "0.000000", 2024-09-20T22:42:36.304 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_inject_delay_type": "", 2024-09-20T22:42:36.304 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_inject_internal_delays": "0.000000", 2024-09-20T22:42:36.304 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_inject_network_congestion": "0", 2024-09-20T22:42:36.304 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_inject_socket_failures": "0", 2024-09-20T22:42:36.304 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_learn_addr_from_peer": "true", 2024-09-20T22:42:36.304 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_max_accept_failures": "4", 2024-09-20T22:42:36.304 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_max_backoff": "15.000000", 2024-09-20T22:42:36.304 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_mon_client_mode": "secure crc", 2024-09-20T22:42:36.304 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_mon_cluster_mode": "secure crc", 2024-09-20T22:42:36.304 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_mon_service_mode": "secure crc", 2024-09-20T22:42:36.304 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_osd_compress_min_size": "1024", 2024-09-20T22:42:36.304 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_osd_compress_mode": "none", 2024-09-20T22:42:36.304 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_osd_compression_algorithm": "snappy", 2024-09-20T22:42:36.305 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_pq_max_tokens_per_priority": "16777216", 2024-09-20T22:42:36.305 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_pq_min_cost": "65536", 2024-09-20T22:42:36.305 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_public_type": "", 2024-09-20T22:42:36.305 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_service_mode": "crc secure", 2024-09-20T22:42:36.305 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_tcp_listen_backlog": "512", 2024-09-20T22:42:36.305 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_tcp_nodelay": "true", 2024-09-20T22:42:36.305 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_tcp_prefetch_max_size": "4096", 2024-09-20T22:42:36.305 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_tcp_rcvbuf": "0", 2024-09-20T22:42:36.305 INFO:teuthology.orchestra.run.smithi084.stdout: "ms_type": "async+posix", 2024-09-20T22:42:36.305 INFO:teuthology.orchestra.run.smithi084.stdout: "no_config_file": "false", 2024-09-20T22:42:36.305 INFO:teuthology.orchestra.run.smithi084.stdout: "objecter_completion_locks_per_session": "32", 2024-09-20T22:42:36.305 INFO:teuthology.orchestra.run.smithi084.stdout: "objecter_debug_inject_relock_delay": "false", 2024-09-20T22:42:36.305 INFO:teuthology.orchestra.run.smithi084.stdout: "objecter_inflight_op_bytes": "104857600", 2024-09-20T22:42:36.305 INFO:teuthology.orchestra.run.smithi084.stdout: "objecter_inflight_ops": "1024", 2024-09-20T22:42:36.305 INFO:teuthology.orchestra.run.smithi084.stdout: "objecter_inject_no_watch_ping": "false", 2024-09-20T22:42:36.306 INFO:teuthology.orchestra.run.smithi084.stdout: "objecter_retry_writes_after_first_reply": "false", 2024-09-20T22:42:36.306 INFO:teuthology.orchestra.run.smithi084.stdout: "objecter_tick_interval": "5.000000", 2024-09-20T22:42:36.306 INFO:teuthology.orchestra.run.smithi084.stdout: "objecter_timeout": "10.000000", 2024-09-20T22:42:36.306 INFO:teuthology.orchestra.run.smithi084.stdout: "objectstore_blackhole": "false", 2024-09-20T22:42:36.306 INFO:teuthology.orchestra.run.smithi084.stdout: "objectstore_debug_throw_on_failed_txc": "false", 2024-09-20T22:42:36.306 INFO:teuthology.orchestra.run.smithi084.stdout: "openssl_engine_opts": "", 2024-09-20T22:42:36.306 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_agent_delay_time": "5.000000", 2024-09-20T22:42:36.306 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_agent_hist_halflife": "1000", 2024-09-20T22:42:36.306 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_agent_max_low_ops": "2", 2024-09-20T22:42:36.306 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_agent_max_ops": "4", 2024-09-20T22:42:36.306 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_agent_min_evict_effort": "0.100000", 2024-09-20T22:42:36.306 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_agent_quantize_effort": "0.100000", 2024-09-20T22:42:36.306 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_agent_slop": "0.020000", 2024-09-20T22:42:36.306 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_aggregated_slow_ops_logging": "true", 2024-09-20T22:42:36.307 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_allow_recovery_below_min_size": "true", 2024-09-20T22:42:36.307 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_asio_thread_count": "2", 2024-09-20T22:42:36.307 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_async_recovery_min_cost": "100", 2024-09-20T22:42:36.307 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_auto_mark_unfound_lost": "false", 2024-09-20T22:42:36.307 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_backfill_retry_interval": "30.000000", 2024-09-20T22:42:36.307 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_backfill_scan_max": "512", 2024-09-20T22:42:36.307 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_backfill_scan_min": "64", 2024-09-20T22:42:36.307 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_backoff_on_degraded": "false", 2024-09-20T22:42:36.307 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_backoff_on_peering": "false", 2024-09-20T22:42:36.307 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_backoff_on_unfound": "true", 2024-09-20T22:42:36.307 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_beacon_report_interval": "300", 2024-09-20T22:42:36.307 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_bench_duration": "30", 2024-09-20T22:42:36.307 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_bench_large_size_max_throughput": "104857600", 2024-09-20T22:42:36.307 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_bench_max_block_size": "67108864", 2024-09-20T22:42:36.308 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_bench_small_size_max_iops": "100", 2024-09-20T22:42:36.308 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_blkin_trace_all": "false", 2024-09-20T22:42:36.308 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_blocked_scrub_grace_period": "120", 2024-09-20T22:42:36.308 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_calc_pg_upmaps_aggressively": "true", 2024-09-20T22:42:36.308 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_calc_pg_upmaps_aggressively_fast": "true", 2024-09-20T22:42:36.308 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_calc_pg_upmaps_local_fallback_retries": "100", 2024-09-20T22:42:36.308 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_check_for_log_corruption": "false", 2024-09-20T22:42:36.308 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_check_max_object_name_len_on_startup": "true", 2024-09-20T22:42:36.308 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_class_default_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-09-20T22:42:36.308 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_class_dir": "/usr/lib64/rados-classes", 2024-09-20T22:42:36.308 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_class_load_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-09-20T22:42:36.308 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_class_update_on_start": "true", 2024-09-20T22:42:36.308 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_client_message_cap": "256", 2024-09-20T22:42:36.308 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_client_message_size_cap": "524288000", 2024-09-20T22:42:36.309 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_client_op_priority": "63", 2024-09-20T22:42:36.309 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_client_watch_timeout": "30", 2024-09-20T22:42:36.309 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_command_max_records": "256", 2024-09-20T22:42:36.309 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_compact_on_start": "false", 2024-09-20T22:42:36.309 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_copyfrom_max_chunk": "8388608", 2024-09-20T22:42:36.309 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_crush_chooseleaf_type": "1", 2024-09-20T22:42:36.309 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_crush_initial_weight": "-1.000000", 2024-09-20T22:42:36.309 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_crush_update_on_start": "true", 2024-09-20T22:42:36.309 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_crush_update_weight_set": "true", 2024-09-20T22:42:36.309 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_data": "/var/lib/ceph/osd/ceph-a", 2024-09-20T22:42:36.309 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_debug_crash_on_ignored_backoff": "false", 2024-09-20T22:42:36.309 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_debug_deep_scrub_sleep": "0.000000", 2024-09-20T22:42:36.309 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_debug_drop_ping_duration": "0", 2024-09-20T22:42:36.310 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_debug_drop_ping_probability": "0.000000", 2024-09-20T22:42:36.310 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_debug_feed_pullee": "-1", 2024-09-20T22:42:36.310 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_debug_inject_copyfrom_error": "false", 2024-09-20T22:42:36.310 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_debug_inject_dispatch_delay_duration": "0.100000", 2024-09-20T22:42:36.310 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_debug_inject_dispatch_delay_probability": "0.000000", 2024-09-20T22:42:36.310 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_debug_misdirected_ops": "false", 2024-09-20T22:42:36.310 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_debug_no_acting_change": "false", 2024-09-20T22:42:36.310 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_debug_no_purge_strays": "false", 2024-09-20T22:42:36.310 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_debug_op_order": "false", 2024-09-20T22:42:36.310 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_debug_pg_log_writeout": "false", 2024-09-20T22:42:36.310 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_debug_pretend_recovery_active": "false", 2024-09-20T22:42:36.310 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_debug_random_push_read_error": "0.000000", 2024-09-20T22:42:36.310 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_debug_reject_backfill_probability": "0.000000", 2024-09-20T22:42:36.310 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_debug_shutdown": "false", 2024-09-20T22:42:36.311 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_debug_skip_full_check_in_backfill_reservation": "false", 2024-09-20T22:42:36.311 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_debug_skip_full_check_in_recovery": "false", 2024-09-20T22:42:36.311 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_debug_trim_objects": "false", 2024-09-20T22:42:36.311 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_debug_verify_cached_snaps": "false", 2024-09-20T22:42:36.311 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_debug_verify_missing_on_start": "false", 2024-09-20T22:42:36.311 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_debug_verify_snaps": "false", 2024-09-20T22:42:36.311 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_debug_verify_stray_on_activate": "false", 2024-09-20T22:42:36.311 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_deep_scrub_interval": "604800.000000", 2024-09-20T22:42:36.311 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_deep_scrub_keys": "1024", 2024-09-20T22:42:36.311 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_deep_scrub_large_omap_object_key_threshold": "200000", 2024-09-20T22:42:36.311 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", 2024-09-20T22:42:36.311 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_deep_scrub_randomize_ratio": "0.150000", 2024-09-20T22:42:36.311 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_deep_scrub_stride": "524288", 2024-09-20T22:42:36.311 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_deep_scrub_update_digest_min_age": "7200", 2024-09-20T22:42:36.311 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_default_data_pool_replay_window": "45", 2024-09-20T22:42:36.312 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_default_notify_timeout": "30", 2024-09-20T22:42:36.312 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_delete_sleep": "0.000000", 2024-09-20T22:42:36.312 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_delete_sleep_hdd": "5.000000", 2024-09-20T22:42:36.312 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_delete_sleep_hybrid": "1.000000", 2024-09-20T22:42:36.312 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_delete_sleep_ssd": "1.000000", 2024-09-20T22:42:36.312 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_discard_disconnected_ops": "true", 2024-09-20T22:42:36.312 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_enable_op_tracker": "true", 2024-09-20T22:42:36.312 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_erasure_code_plugins": "jerasure lrc", 2024-09-20T22:42:36.312 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_extblkdev_plugins": "vdo", 2024-09-20T22:42:36.312 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_failsafe_full_ratio": "0.970000", 2024-09-20T22:42:36.312 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_fast_fail_on_connection_refused": "true", 2024-09-20T22:42:36.312 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_fast_info": "true", 2024-09-20T22:42:36.312 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_fast_shutdown": "true", 2024-09-20T22:42:36.312 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_fast_shutdown_notify_mon": "true", 2024-09-20T22:42:36.312 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_fast_shutdown_timeout": "15", 2024-09-20T22:42:36.313 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_find_best_info_ignore_history_les": "false", 2024-09-20T22:42:36.313 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_force_auth_primary_missing_objects": "100", 2024-09-20T22:42:36.313 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_force_recovery_pg_log_entries_factor": "1.300000", 2024-09-20T22:42:36.313 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_function_tracing": "false", 2024-09-20T22:42:36.313 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_heartbeat_grace": "20", 2024-09-20T22:42:36.313 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_heartbeat_interval": "6", 2024-09-20T22:42:36.313 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_heartbeat_min_healthy_ratio": "0.330000", 2024-09-20T22:42:36.313 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_heartbeat_min_peers": "10", 2024-09-20T22:42:36.313 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_heartbeat_min_size": "2000", 2024-09-20T22:42:36.313 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_heartbeat_stale": "600", 2024-09-20T22:42:36.313 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_heartbeat_use_min_delay_socket": "false", 2024-09-20T22:42:36.313 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_hit_set_max_size": "100000", 2024-09-20T22:42:36.313 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_hit_set_min_size": "1000", 2024-09-20T22:42:36.313 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_hit_set_namespace": ".ceph-internal", 2024-09-20T22:42:36.313 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_ignore_stale_divergent_priors": "false", 2024-09-20T22:42:36.313 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_inject_bad_map_crc_probability": "0.000000", 2024-09-20T22:42:36.314 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_inject_failure_on_pg_removal": "false", 2024-09-20T22:42:36.314 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_journal": "/var/lib/ceph/osd/ceph-a/journal", 2024-09-20T22:42:36.314 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_journal_flush_on_shutdown": "true", 2024-09-20T22:42:36.314 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_journal_size": "5120", 2024-09-20T22:42:36.314 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_kill_backfill_at": "0", 2024-09-20T22:42:36.314 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_loop_before_reset_tphandle": "64", 2024-09-20T22:42:36.314 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_map_cache_size": "50", 2024-09-20T22:42:36.314 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_map_dedup": "true", 2024-09-20T22:42:36.314 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_map_message_max": "40", 2024-09-20T22:42:36.314 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_map_message_max_bytes": "10485760", 2024-09-20T22:42:36.314 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_map_share_max_epochs": "40", 2024-09-20T22:42:36.314 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_max_attr_name_len": "100", 2024-09-20T22:42:36.314 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_max_attr_size": "0", 2024-09-20T22:42:36.315 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_max_backfills": "1", 2024-09-20T22:42:36.315 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_max_markdown_count": "5", 2024-09-20T22:42:36.315 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_max_markdown_period": "600", 2024-09-20T22:42:36.315 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_max_object_name_len": "2048", 2024-09-20T22:42:36.315 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_max_object_namespace_len": "256", 2024-09-20T22:42:36.315 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_max_object_size": "134217728", 2024-09-20T22:42:36.315 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_max_omap_bytes_per_request": "1073741824", 2024-09-20T22:42:36.315 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_max_omap_entries_per_request": "1024", 2024-09-20T22:42:36.315 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_max_pg_blocked_by": "16", 2024-09-20T22:42:36.315 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_max_pg_log_entries": "10000", 2024-09-20T22:42:36.315 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_max_pg_per_osd_hard_ratio": "3.000000", 2024-09-20T22:42:36.315 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_max_pgls": "1024", 2024-09-20T22:42:36.315 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_max_push_cost": "8388608", 2024-09-20T22:42:36.315 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_max_push_objects": "10", 2024-09-20T22:42:36.315 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_max_scrubs": "3", 2024-09-20T22:42:36.316 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_max_snap_prune_intervals_per_epoch": "512", 2024-09-20T22:42:36.316 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_max_trimming_pgs": "2", 2024-09-20T22:42:36.316 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_max_write_op_reply_len": "64", 2024-09-20T22:42:36.316 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_max_write_size": "90", 2024-09-20T22:42:36.316 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_mclock_force_run_benchmark_on_init": "false", 2024-09-20T22:42:36.316 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_mclock_iops_capacity_threshold_hdd": "500.000000", 2024-09-20T22:42:36.316 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_mclock_iops_capacity_threshold_ssd": "80000.000000", 2024-09-20T22:42:36.316 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_mclock_max_capacity_iops_hdd": "315.000000", 2024-09-20T22:42:36.316 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_mclock_max_capacity_iops_ssd": "21500.000000", 2024-09-20T22:42:36.316 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_mclock_max_sequential_bandwidth_hdd": "157286400", 2024-09-20T22:42:36.316 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_mclock_max_sequential_bandwidth_ssd": "1258291200", 2024-09-20T22:42:36.316 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_mclock_override_recovery_settings": "false", 2024-09-20T22:42:36.316 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_mclock_profile": "balanced", 2024-09-20T22:42:36.316 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_mclock_scheduler_anticipation_timeout": "0.000000", 2024-09-20T22:42:36.316 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_mclock_scheduler_background_best_effort_lim": "0.000000", 2024-09-20T22:42:36.317 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_mclock_scheduler_background_best_effort_res": "0.000000", 2024-09-20T22:42:36.317 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_mclock_scheduler_background_best_effort_wgt": "1", 2024-09-20T22:42:36.317 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_mclock_scheduler_background_recovery_lim": "0.000000", 2024-09-20T22:42:36.317 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_mclock_scheduler_background_recovery_res": "0.000000", 2024-09-20T22:42:36.317 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_mclock_scheduler_background_recovery_wgt": "1", 2024-09-20T22:42:36.317 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_mclock_scheduler_client_lim": "0.000000", 2024-09-20T22:42:36.317 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_mclock_scheduler_client_res": "0.000000", 2024-09-20T22:42:36.317 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_mclock_scheduler_client_wgt": "1", 2024-09-20T22:42:36.317 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_mclock_skip_benchmark": "false", 2024-09-20T22:42:36.317 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_memory_base": "805306368", 2024-09-20T22:42:36.317 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_memory_cache_min": "134217728", 2024-09-20T22:42:36.317 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_memory_cache_resize_interval": "1.000000", 2024-09-20T22:42:36.317 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_memory_expected_fragmentation": "0.150000", 2024-09-20T22:42:36.317 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_memory_target": "4294967296", 2024-09-20T22:42:36.317 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_memory_target_autotune": "false", 2024-09-20T22:42:36.317 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_memory_target_cgroup_limit_ratio": "0.800000", 2024-09-20T22:42:36.318 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_min_pg_log_entries": "250", 2024-09-20T22:42:36.318 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_min_recovery_priority": "0", 2024-09-20T22:42:36.318 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_mon_heartbeat_interval": "30", 2024-09-20T22:42:36.318 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_mon_heartbeat_stat_stale": "3600", 2024-09-20T22:42:36.318 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_mon_report_interval": "5", 2024-09-20T22:42:36.318 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_mon_report_max_in_flight": "2", 2024-09-20T22:42:36.318 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_mon_shutdown_timeout": "5.000000", 2024-09-20T22:42:36.318 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_num_cache_shards": "32", 2024-09-20T22:42:36.318 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_num_op_tracker_shard": "32", 2024-09-20T22:42:36.318 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_numa_auto_affinity": "true", 2024-09-20T22:42:36.318 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_numa_node": "-1", 2024-09-20T22:42:36.318 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_numa_prefer_iface": "true", 2024-09-20T22:42:36.318 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_object_clean_region_max_num_intervals": "10", 2024-09-20T22:42:36.319 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_objecter_finishers": "1", 2024-09-20T22:42:36.319 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_objectstore": "bluestore", 2024-09-20T22:42:36.319 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_objectstore_fuse": "false", 2024-09-20T22:42:36.319 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_objectstore_tracing": "false", 2024-09-20T22:42:36.319 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_op_complaint_time": "30.000000", 2024-09-20T22:42:36.319 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_op_history_duration": "600", 2024-09-20T22:42:36.319 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_op_history_size": "20", 2024-09-20T22:42:36.319 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_op_history_slow_op_size": "20", 2024-09-20T22:42:36.319 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_op_history_slow_op_threshold": "10.000000", 2024-09-20T22:42:36.319 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_op_log_threshold": "5", 2024-09-20T22:42:36.319 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_op_num_shards": "0", 2024-09-20T22:42:36.319 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_op_num_shards_hdd": "5", 2024-09-20T22:42:36.319 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_op_num_shards_ssd": "8", 2024-09-20T22:42:36.320 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_op_num_threads_per_shard": "0", 2024-09-20T22:42:36.320 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_op_num_threads_per_shard_hdd": "1", 2024-09-20T22:42:36.320 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_op_num_threads_per_shard_ssd": "2", 2024-09-20T22:42:36.320 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_op_pq_max_tokens_per_priority": "4194304", 2024-09-20T22:42:36.320 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_op_pq_min_cost": "65536", 2024-09-20T22:42:36.320 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_op_queue": "mclock_scheduler", 2024-09-20T22:42:36.320 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_op_queue_cut_off": "high", 2024-09-20T22:42:36.320 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_op_thread_suicide_timeout": "150", 2024-09-20T22:42:36.320 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_op_thread_timeout": "15", 2024-09-20T22:42:36.320 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_open_classes_on_start": "true", 2024-09-20T22:42:36.320 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_os_flags": "0", 2024-09-20T22:42:36.320 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_peering_op_priority": "255", 2024-09-20T22:42:36.320 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_pg_delete_cost": "1048576", 2024-09-20T22:42:36.321 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_pg_delete_priority": "5", 2024-09-20T22:42:36.321 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_pg_epoch_max_lag_factor": "2.000000", 2024-09-20T22:42:36.321 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_pg_epoch_persisted_max_stale": "40", 2024-09-20T22:42:36.321 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_pg_log_dups_tracked": "3000", 2024-09-20T22:42:36.321 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_pg_log_trim_max": "10000", 2024-09-20T22:42:36.321 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_pg_log_trim_min": "100", 2024-09-20T22:42:36.321 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_pg_max_concurrent_snap_trims": "2", 2024-09-20T22:42:36.321 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_pg_object_context_cache_count": "64", 2024-09-20T22:42:36.321 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_pg_stat_report_interval_max_epochs": "500", 2024-09-20T22:42:36.321 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_pg_stat_report_interval_max_seconds": "5", 2024-09-20T22:42:36.321 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_pool_default_cache_max_evict_check_size": "10", 2024-09-20T22:42:36.321 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_pool_default_cache_min_evict_age": "0", 2024-09-20T22:42:36.321 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_pool_default_cache_min_flush_age": "0", 2024-09-20T22:42:36.322 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_pool_default_cache_target_dirty_high_ratio": "0.600000", 2024-09-20T22:42:36.322 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_pool_default_cache_target_dirty_ratio": "0.400000", 2024-09-20T22:42:36.322 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_pool_default_cache_target_full_ratio": "0.800000", 2024-09-20T22:42:36.322 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_pool_default_crimson": "false", 2024-09-20T22:42:36.322 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_pool_default_crush_rule": "-1", 2024-09-20T22:42:36.322 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_pool_default_ec_fast_read": "false", 2024-09-20T22:42:36.322 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_pool_default_erasure_code_profile": "plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd", 2024-09-20T22:42:36.322 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_pool_default_flag_bulk": "false", 2024-09-20T22:42:36.322 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_pool_default_flag_hashpspool": "true", 2024-09-20T22:42:36.322 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_pool_default_flag_nodelete": "false", 2024-09-20T22:42:36.322 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_pool_default_flag_nopgchange": "false", 2024-09-20T22:42:36.323 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_pool_default_flag_nosizechange": "false", 2024-09-20T22:42:36.323 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_pool_default_flags": "0", 2024-09-20T22:42:36.323 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_pool_default_hit_set_bloom_fpp": "0.050000", 2024-09-20T22:42:36.323 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_pool_default_min_size": "0", 2024-09-20T22:42:36.323 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_pool_default_pg_autoscale_mode": "off", 2024-09-20T22:42:36.323 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_pool_default_pg_num": "32", 2024-09-20T22:42:36.323 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_pool_default_pgp_num": "0", 2024-09-20T22:42:36.323 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_pool_default_read_lease_ratio": "0.800000", 2024-09-20T22:42:36.323 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_pool_default_read_ratio": "70", 2024-09-20T22:42:36.323 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_pool_default_size": "3", 2024-09-20T22:42:36.323 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_pool_default_type": "replicated", 2024-09-20T22:42:36.323 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_pool_erasure_code_stripe_unit": "4096", 2024-09-20T22:42:36.323 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_pool_use_gmt_hitset": "true", 2024-09-20T22:42:36.324 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_push_per_object_cost": "1000", 2024-09-20T22:42:36.324 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_read_ec_check_for_errors": "false", 2024-09-20T22:42:36.324 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_recover_clone_overlap": "true", 2024-09-20T22:42:36.324 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_recover_clone_overlap_limit": "10", 2024-09-20T22:42:36.324 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_recovery_cost": "20971520", 2024-09-20T22:42:36.324 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_recovery_delay_start": "0.000000", 2024-09-20T22:42:36.324 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_recovery_max_active": "0", 2024-09-20T22:42:36.324 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_recovery_max_active_hdd": "3", 2024-09-20T22:42:36.324 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_recovery_max_active_ssd": "10", 2024-09-20T22:42:36.324 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_recovery_max_chunk": "8388608", 2024-09-20T22:42:36.324 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_recovery_max_omap_entries_per_chunk": "8096", 2024-09-20T22:42:36.324 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_recovery_max_single_start": "1", 2024-09-20T22:42:36.324 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_recovery_op_priority": "3", 2024-09-20T22:42:36.324 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_recovery_op_warn_multiple": "16", 2024-09-20T22:42:36.325 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_recovery_priority": "5", 2024-09-20T22:42:36.325 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_recovery_retry_interval": "30.000000", 2024-09-20T22:42:36.325 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_recovery_sleep": "0.000000", 2024-09-20T22:42:36.325 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_recovery_sleep_hdd": "0.100000", 2024-09-20T22:42:36.325 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_recovery_sleep_hybrid": "0.025000", 2024-09-20T22:42:36.325 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_recovery_sleep_ssd": "0.000000", 2024-09-20T22:42:36.325 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_repair_during_recovery": "false", 2024-09-20T22:42:36.325 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_requested_scrub_priority": "120", 2024-09-20T22:42:36.325 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_rocksdb_iterator_bounds_enabled": "true", 2024-09-20T22:42:36.325 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_rollback_to_cluster_snap": "", 2024-09-20T22:42:36.325 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_scrub_auto_repair": "false", 2024-09-20T22:42:36.325 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_scrub_auto_repair_num_errors": "5", 2024-09-20T22:42:36.325 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_scrub_backoff_ratio": "0.660000", 2024-09-20T22:42:36.326 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_scrub_begin_hour": "0", 2024-09-20T22:42:36.326 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_scrub_begin_week_day": "0", 2024-09-20T22:42:36.326 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_scrub_chunk_max": "25", 2024-09-20T22:42:36.326 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_scrub_chunk_min": "5", 2024-09-20T22:42:36.326 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_scrub_cost": "52428800", 2024-09-20T22:42:36.326 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_scrub_disable_reservation_queuing": "false", 2024-09-20T22:42:36.326 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_scrub_during_recovery": "false", 2024-09-20T22:42:36.326 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_scrub_end_hour": "0", 2024-09-20T22:42:36.326 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_scrub_end_week_day": "0", 2024-09-20T22:42:36.326 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_scrub_event_cost": "4096", 2024-09-20T22:42:36.326 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_scrub_extended_sleep": "0.000000", 2024-09-20T22:42:36.326 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_scrub_interval_randomize_ratio": "0.500000", 2024-09-20T22:42:36.326 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_scrub_invalid_stats": "true", 2024-09-20T22:42:36.327 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_scrub_load_threshold": "0.500000", 2024-09-20T22:42:36.327 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_scrub_max_interval": "604800.000000", 2024-09-20T22:42:36.327 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_scrub_max_preemptions": "5", 2024-09-20T22:42:36.327 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_scrub_min_interval": "86400.000000", 2024-09-20T22:42:36.327 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_scrub_priority": "5", 2024-09-20T22:42:36.327 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_scrub_sleep": "0.000000", 2024-09-20T22:42:36.327 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_shallow_scrub_chunk_max": "100", 2024-09-20T22:42:36.327 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_shallow_scrub_chunk_min": "50", 2024-09-20T22:42:36.327 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_shutdown_pgref_assert": "false", 2024-09-20T22:42:36.327 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_skip_data_digest": "false", 2024-09-20T22:42:36.327 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_smart_report_timeout": "5", 2024-09-20T22:42:36.327 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_snap_trim_cost": "1048576", 2024-09-20T22:42:36.328 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_snap_trim_priority": "5", 2024-09-20T22:42:36.328 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_snap_trim_sleep": "0.000000", 2024-09-20T22:42:36.328 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_snap_trim_sleep_hdd": "5.000000", 2024-09-20T22:42:36.328 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_snap_trim_sleep_hybrid": "2.000000", 2024-09-20T22:42:36.328 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_snap_trim_sleep_ssd": "0.000000", 2024-09-20T22:42:36.328 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_stats_update_period_not_scrubbing": "120", 2024-09-20T22:42:36.328 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_stats_update_period_scrubbing": "15", 2024-09-20T22:42:36.328 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_target_pg_log_entries_per_osd": "300000", 2024-09-20T22:42:36.328 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_target_transaction_size": "30", 2024-09-20T22:42:36.328 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_tier_default_cache_hit_set_count": "4", 2024-09-20T22:42:36.328 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_tier_default_cache_hit_set_grade_decay_rate": "20", 2024-09-20T22:42:36.328 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_tier_default_cache_hit_set_period": "1200", 2024-09-20T22:42:36.328 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_tier_default_cache_hit_set_search_last_n": "1", 2024-09-20T22:42:36.329 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_tier_default_cache_hit_set_type": "bloom", 2024-09-20T22:42:36.329 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_tier_default_cache_min_read_recency_for_promote": "1", 2024-09-20T22:42:36.329 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_tier_default_cache_min_write_recency_for_promote": "1", 2024-09-20T22:42:36.329 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_tier_default_cache_mode": "writeback", 2024-09-20T22:42:36.329 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_tier_promote_max_bytes_sec": "5242880", 2024-09-20T22:42:36.329 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_tier_promote_max_objects_sec": "25", 2024-09-20T22:42:36.329 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_tracing": "false", 2024-09-20T22:42:36.329 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_use_stale_snap": "false", 2024-09-20T22:42:36.329 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_uuid": "00000000-0000-0000-0000-000000000000", 2024-09-20T22:42:36.329 INFO:teuthology.orchestra.run.smithi084.stdout: "osdc_blkin_trace_all": "false", 2024-09-20T22:42:36.329 INFO:teuthology.orchestra.run.smithi084.stdout: "paxos_kill_at": "0", 2024-09-20T22:42:36.329 INFO:teuthology.orchestra.run.smithi084.stdout: "paxos_max_join_drift": "10", 2024-09-20T22:42:36.329 INFO:teuthology.orchestra.run.smithi084.stdout: "paxos_min": "500", 2024-09-20T22:42:36.330 INFO:teuthology.orchestra.run.smithi084.stdout: "paxos_min_wait": "0.050000", 2024-09-20T22:42:36.330 INFO:teuthology.orchestra.run.smithi084.stdout: "paxos_propose_interval": "1.000000", 2024-09-20T22:42:36.330 INFO:teuthology.orchestra.run.smithi084.stdout: "paxos_service_trim_max": "500", 2024-09-20T22:42:36.330 INFO:teuthology.orchestra.run.smithi084.stdout: "paxos_service_trim_max_multiplier": "20", 2024-09-20T22:42:36.330 INFO:teuthology.orchestra.run.smithi084.stdout: "paxos_service_trim_min": "10", 2024-09-20T22:42:36.330 INFO:teuthology.orchestra.run.smithi084.stdout: "paxos_stash_full_interval": "25", 2024-09-20T22:42:36.330 INFO:teuthology.orchestra.run.smithi084.stdout: "paxos_trim_max": "500", 2024-09-20T22:42:36.330 INFO:teuthology.orchestra.run.smithi084.stdout: "paxos_trim_min": "250", 2024-09-20T22:42:36.330 INFO:teuthology.orchestra.run.smithi084.stdout: "perf": "true", 2024-09-20T22:42:36.330 INFO:teuthology.orchestra.run.smithi084.stdout: "pid_file": "", 2024-09-20T22:42:36.330 INFO:teuthology.orchestra.run.smithi084.stdout: "plugin_crypto_accelerator": "crypto_isal", 2024-09-20T22:42:36.330 INFO:teuthology.orchestra.run.smithi084.stdout: "plugin_dir": "/usr/lib64/ceph", 2024-09-20T22:42:36.331 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "-", 2024-09-20T22:42:36.331 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrv": "", 2024-09-20T22:42:36.331 INFO:teuthology.orchestra.run.smithi084.stdout: "public_bind_addr": "-", 2024-09-20T22:42:36.331 INFO:teuthology.orchestra.run.smithi084.stdout: "public_network": "", 2024-09-20T22:42:36.331 INFO:teuthology.orchestra.run.smithi084.stdout: "public_network_interface": "", 2024-09-20T22:42:36.331 INFO:teuthology.orchestra.run.smithi084.stdout: "qat_compressor_busy_polling": "false", 2024-09-20T22:42:36.331 INFO:teuthology.orchestra.run.smithi084.stdout: "qat_compressor_enabled": "false", 2024-09-20T22:42:36.331 INFO:teuthology.orchestra.run.smithi084.stdout: "qat_compressor_session_max_number": "256", 2024-09-20T22:42:36.331 INFO:teuthology.orchestra.run.smithi084.stdout: "rados_mon_op_timeout": "0", 2024-09-20T22:42:36.331 INFO:teuthology.orchestra.run.smithi084.stdout: "rados_osd_op_timeout": "0", 2024-09-20T22:42:36.331 INFO:teuthology.orchestra.run.smithi084.stdout: "rados_tracing": "false", 2024-09-20T22:42:36.331 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_atime_update_interval": "60", 2024-09-20T22:42:36.332 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_auto_exclusive_lock_until_manual_request": "true", 2024-09-20T22:42:36.332 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_balance_parent_reads": "false", 2024-09-20T22:42:36.332 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_balance_snap_reads": "false", 2024-09-20T22:42:36.332 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_blkin_trace_all": "false", 2024-09-20T22:42:36.332 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_blocklist_expire_seconds": "0", 2024-09-20T22:42:36.332 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_blocklist_on_break_lock": "true", 2024-09-20T22:42:36.332 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_cache": "true", 2024-09-20T22:42:36.332 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_cache_block_writes_upfront": "false", 2024-09-20T22:42:36.332 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_cache_max_dirty": "25165824", 2024-09-20T22:42:36.332 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_cache_max_dirty_age": "1.000000", 2024-09-20T22:42:36.332 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_cache_max_dirty_object": "0", 2024-09-20T22:42:36.332 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_cache_policy": "writearound", 2024-09-20T22:42:36.332 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_cache_size": "33554432", 2024-09-20T22:42:36.333 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_cache_target_dirty": "16777216", 2024-09-20T22:42:36.333 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_cache_writethrough_until_flush": "true", 2024-09-20T22:42:36.333 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_clone_copy_on_read": "false", 2024-09-20T22:42:36.333 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_compression_hint": "none", 2024-09-20T22:42:36.333 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_concurrent_management_ops": "10", 2024-09-20T22:42:36.333 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_config_pool_override_update_timestamp": "0", 2024-09-20T22:42:36.333 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_default_clone_format": "auto", 2024-09-20T22:42:36.333 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_default_data_pool": "", 2024-09-20T22:42:36.333 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_default_features": "61", 2024-09-20T22:42:36.333 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_default_format": "2", 2024-09-20T22:42:36.333 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_default_map_options": "", 2024-09-20T22:42:36.333 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_default_order": "22", 2024-09-20T22:42:36.334 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_default_pool": "rbd", 2024-09-20T22:42:36.334 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_default_snapshot_quiesce_mode": "required", 2024-09-20T22:42:36.334 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_default_stripe_count": "0", 2024-09-20T22:42:36.334 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_default_stripe_unit": "0", 2024-09-20T22:42:36.334 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_disable_zero_copy_writes": "true", 2024-09-20T22:42:36.334 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_discard_granularity_bytes": "65536", 2024-09-20T22:42:36.334 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_discard_on_zeroed_write_same": "true", 2024-09-20T22:42:36.334 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_enable_alloc_hint": "true", 2024-09-20T22:42:36.334 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_invalidate_object_map_on_timeout": "true", 2024-09-20T22:42:36.334 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_io_scheduler": "simple", 2024-09-20T22:42:36.334 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_io_scheduler_simple_max_delay": "0", 2024-09-20T22:42:36.334 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_journal_commit_age": "5.000000", 2024-09-20T22:42:36.335 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_journal_max_concurrent_object_sets": "0", 2024-09-20T22:42:36.335 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_journal_max_payload_bytes": "16384", 2024-09-20T22:42:36.335 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_journal_object_flush_age": "0.000000", 2024-09-20T22:42:36.335 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_journal_object_flush_bytes": "1048576", 2024-09-20T22:42:36.335 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_journal_object_flush_interval": "0", 2024-09-20T22:42:36.335 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_journal_object_max_in_flight_appends": "0", 2024-09-20T22:42:36.335 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_journal_object_writethrough_until_flush": "true", 2024-09-20T22:42:36.335 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_journal_order": "24", 2024-09-20T22:42:36.335 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_journal_pool": "", 2024-09-20T22:42:36.335 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_journal_splay_width": "4", 2024-09-20T22:42:36.335 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_localize_parent_reads": "false", 2024-09-20T22:42:36.335 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_localize_snap_reads": "false", 2024-09-20T22:42:36.335 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_mirror_concurrent_image_deletions": "1", 2024-09-20T22:42:36.335 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_mirror_concurrent_image_syncs": "5", 2024-09-20T22:42:36.336 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_mirror_delete_retry_interval": "30.000000", 2024-09-20T22:42:36.336 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_mirror_image_perf_stats_prio": "5", 2024-09-20T22:42:36.336 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_mirror_image_policy_migration_throttle": "300", 2024-09-20T22:42:36.336 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_mirror_image_policy_rebalance_timeout": "0.000000", 2024-09-20T22:42:36.336 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_mirror_image_policy_type": "simple", 2024-09-20T22:42:36.336 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_mirror_image_policy_update_throttle_interval": "1.000000", 2024-09-20T22:42:36.336 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_mirror_image_state_check_interval": "30", 2024-09-20T22:42:36.336 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_mirror_journal_commit_age": "5.000000", 2024-09-20T22:42:36.336 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_mirror_journal_poll_age": "5.000000", 2024-09-20T22:42:36.336 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_mirror_leader_heartbeat_interval": "5", 2024-09-20T22:42:36.336 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_mirror_leader_max_acquire_attempts_before_break": "3", 2024-09-20T22:42:36.336 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_mirror_leader_max_missed_heartbeats": "2", 2024-09-20T22:42:36.336 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_mirror_memory_autotune": "true", 2024-09-20T22:42:36.336 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_mirror_memory_base": "805306368", 2024-09-20T22:42:36.337 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_mirror_memory_cache_autotune_interval": "30.000000", 2024-09-20T22:42:36.337 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_mirror_memory_cache_min": "134217728", 2024-09-20T22:42:36.337 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_mirror_memory_cache_resize_interval": "5.000000", 2024-09-20T22:42:36.337 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_mirror_memory_expected_fragmentation": "0.150000", 2024-09-20T22:42:36.337 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_mirror_memory_target": "4294967296", 2024-09-20T22:42:36.337 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_mirror_perf_stats_prio": "5", 2024-09-20T22:42:36.337 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_mirror_pool_replayers_refresh_interval": "30", 2024-09-20T22:42:36.337 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_mirror_sync_point_update_age": "30.000000", 2024-09-20T22:42:36.337 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_mirroring_delete_delay": "0", 2024-09-20T22:42:36.337 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_mirroring_max_mirroring_snapshots": "5", 2024-09-20T22:42:36.337 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_mirroring_replay_delay": "0", 2024-09-20T22:42:36.337 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_mirroring_resync_after_disconnect": "false", 2024-09-20T22:42:36.337 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_move_parent_to_trash_on_remove": "false", 2024-09-20T22:42:36.337 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_move_to_trash_on_remove": "false", 2024-09-20T22:42:36.338 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_move_to_trash_on_remove_expire_seconds": "0", 2024-09-20T22:42:36.338 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_mtime_update_interval": "60", 2024-09-20T22:42:36.338 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_non_blocking_aio": "true", 2024-09-20T22:42:36.338 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_op_thread_timeout": "60", 2024-09-20T22:42:36.338 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_op_threads": "1", 2024-09-20T22:42:36.338 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_parent_cache_enabled": "false", 2024-09-20T22:42:36.338 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_persistent_cache_mode": "disabled", 2024-09-20T22:42:36.338 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_persistent_cache_path": "/tmp", 2024-09-20T22:42:36.338 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_persistent_cache_size": "1073741824", 2024-09-20T22:42:36.338 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_plugins": "", 2024-09-20T22:42:36.338 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_qos_bps_burst": "0", 2024-09-20T22:42:36.338 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_qos_bps_burst_seconds": "1", 2024-09-20T22:42:36.338 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_qos_bps_limit": "0", 2024-09-20T22:42:36.338 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_qos_exclude_ops": "0", 2024-09-20T22:42:36.339 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_qos_iops_burst": "0", 2024-09-20T22:42:36.339 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_qos_iops_burst_seconds": "1", 2024-09-20T22:42:36.339 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_qos_iops_limit": "0", 2024-09-20T22:42:36.339 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_qos_read_bps_burst": "0", 2024-09-20T22:42:36.339 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_qos_read_bps_burst_seconds": "1", 2024-09-20T22:42:36.339 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_qos_read_bps_limit": "0", 2024-09-20T22:42:36.339 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_qos_read_iops_burst": "0", 2024-09-20T22:42:36.339 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_qos_read_iops_burst_seconds": "1", 2024-09-20T22:42:36.339 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_qos_read_iops_limit": "0", 2024-09-20T22:42:36.339 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_qos_schedule_tick_min": "50", 2024-09-20T22:42:36.339 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_qos_write_bps_burst": "0", 2024-09-20T22:42:36.339 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_qos_write_bps_burst_seconds": "1", 2024-09-20T22:42:36.339 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_qos_write_bps_limit": "0", 2024-09-20T22:42:36.340 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_qos_write_iops_burst": "0", 2024-09-20T22:42:36.340 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_qos_write_iops_burst_seconds": "1", 2024-09-20T22:42:36.340 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_qos_write_iops_limit": "0", 2024-09-20T22:42:36.340 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_quiesce_notification_attempts": "10", 2024-09-20T22:42:36.340 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_read_from_replica_policy": "default", 2024-09-20T22:42:36.340 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_readahead_disable_after_bytes": "52428800", 2024-09-20T22:42:36.340 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_readahead_max_bytes": "524288", 2024-09-20T22:42:36.340 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_readahead_trigger_requests": "10", 2024-09-20T22:42:36.340 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_request_timed_out_seconds": "30", 2024-09-20T22:42:36.340 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_skip_partial_discard": "true", 2024-09-20T22:42:36.340 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_sparse_read_threshold_bytes": "65536", 2024-09-20T22:42:36.340 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_tracing": "false", 2024-09-20T22:42:36.340 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_validate_names": "true", 2024-09-20T22:42:36.341 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd_validate_pool": "true", 2024-09-20T22:42:36.341 INFO:teuthology.orchestra.run.smithi084.stdout: "restapi_base_url": "", 2024-09-20T22:42:36.341 INFO:teuthology.orchestra.run.smithi084.stdout: "restapi_log_level": "", 2024-09-20T22:42:36.341 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_account_default_quota_max_objects": "-1", 2024-09-20T22:42:36.341 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_account_default_quota_max_size": "-1", 2024-09-20T22:42:36.341 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_acl_grants_max_num": "100", 2024-09-20T22:42:36.341 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_admin_entry": "admin", 2024-09-20T22:42:36.341 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_allow_notification_secrets_in_cleartext": "false", 2024-09-20T22:42:36.341 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_backend_store": "rados", 2024-09-20T22:42:36.341 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_barbican_url": "", 2024-09-20T22:42:36.341 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_beast_enable_async": "true", 2024-09-20T22:42:36.341 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_bucket_counters_cache": "false", 2024-09-20T22:42:36.341 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_bucket_counters_cache_size": "10000", 2024-09-20T22:42:36.341 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_bucket_default_quota_max_objects": "-1", 2024-09-20T22:42:36.342 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_bucket_default_quota_max_size": "-1", 2024-09-20T22:42:36.342 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_bucket_index_max_aio": "128", 2024-09-20T22:42:36.342 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_bucket_index_transaction_instrumentation": "false", 2024-09-20T22:42:36.342 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_bucket_quota_cache_size": "10000", 2024-09-20T22:42:36.342 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_bucket_quota_ttl": "600", 2024-09-20T22:42:36.342 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_bucket_sync_spawn_window": "20", 2024-09-20T22:42:36.342 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_cache_enabled": "true", 2024-09-20T22:42:36.342 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_cache_expiry_interval": "900", 2024-09-20T22:42:36.342 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_cache_lru_size": "25000", 2024-09-20T22:42:36.342 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_config_store": "rados", 2024-09-20T22:42:36.342 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_content_length_compat": "false", 2024-09-20T22:42:36.342 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_copy_obj_progress": "true", 2024-09-20T22:42:36.342 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_copy_obj_progress_every_bytes": "1048576", 2024-09-20T22:42:36.342 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_cors_rules_max_num": "100", 2024-09-20T22:42:36.343 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_cross_domain_policy": "", 2024-09-20T22:42:36.343 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_crypt_default_encryption_key": "", 2024-09-20T22:42:36.343 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_crypt_kmip_addr": "", 2024-09-20T22:42:36.343 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_crypt_kmip_ca_path": "", 2024-09-20T22:42:36.343 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_crypt_kmip_client_cert": "", 2024-09-20T22:42:36.343 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_crypt_kmip_client_key": "", 2024-09-20T22:42:36.343 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_crypt_kmip_kms_key_template": "", 2024-09-20T22:42:36.343 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_crypt_kmip_password": "", 2024-09-20T22:42:36.343 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_crypt_kmip_s3_key_template": "$keyid", 2024-09-20T22:42:36.343 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_crypt_kmip_username": "", 2024-09-20T22:42:36.343 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_crypt_require_ssl": "true", 2024-09-20T22:42:36.343 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_crypt_s3_kms_backend": "barbican", 2024-09-20T22:42:36.343 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_crypt_s3_kms_encryption_keys": "", 2024-09-20T22:42:36.343 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_crypt_sse_s3_backend": "vault", 2024-09-20T22:42:36.344 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_crypt_sse_s3_key_template": "%bucket_id", 2024-09-20T22:42:36.344 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_crypt_sse_s3_vault_addr": "", 2024-09-20T22:42:36.344 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_crypt_sse_s3_vault_auth": "token", 2024-09-20T22:42:36.344 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_crypt_sse_s3_vault_namespace": "", 2024-09-20T22:42:36.344 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_crypt_sse_s3_vault_prefix": "", 2024-09-20T22:42:36.344 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_crypt_sse_s3_vault_secret_engine": "transit", 2024-09-20T22:42:36.344 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_crypt_sse_s3_vault_ssl_cacert": "", 2024-09-20T22:42:36.344 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_crypt_sse_s3_vault_ssl_clientcert": "", 2024-09-20T22:42:36.344 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_crypt_sse_s3_vault_ssl_clientkey": "", 2024-09-20T22:42:36.344 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_crypt_sse_s3_vault_token_file": "", 2024-09-20T22:42:36.344 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_crypt_sse_s3_vault_verify_ssl": "true", 2024-09-20T22:42:36.344 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_crypt_suppress_logs": "true", 2024-09-20T22:42:36.344 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_crypt_vault_addr": "", 2024-09-20T22:42:36.344 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_crypt_vault_auth": "token", 2024-09-20T22:42:36.345 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_crypt_vault_namespace": "", 2024-09-20T22:42:36.345 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_crypt_vault_prefix": "", 2024-09-20T22:42:36.345 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_crypt_vault_secret_engine": "transit", 2024-09-20T22:42:36.345 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_crypt_vault_ssl_cacert": "", 2024-09-20T22:42:36.345 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_crypt_vault_ssl_clientcert": "", 2024-09-20T22:42:36.345 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_crypt_vault_ssl_clientkey": "", 2024-09-20T22:42:36.345 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_crypt_vault_token_file": "", 2024-09-20T22:42:36.345 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_crypt_vault_verify_ssl": "true", 2024-09-20T22:42:36.345 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_curl_buffersize": "524288", 2024-09-20T22:42:36.345 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_curl_low_speed_limit": "1024", 2024-09-20T22:42:36.345 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_curl_low_speed_time": "300", 2024-09-20T22:42:36.345 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_curl_tcp_keepalive": "0", 2024-09-20T22:42:36.345 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_curl_wait_timeout_ms": "1000", 2024-09-20T22:42:36.345 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_d3n_l1_datacache_persistent_path": "/tmp/rgw_datacache/", 2024-09-20T22:42:36.345 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_d3n_l1_datacache_size": "1073741824", 2024-09-20T22:42:36.346 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_d3n_l1_evict_cache_on_start": "true", 2024-09-20T22:42:36.346 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_d3n_l1_eviction_policy": "lru", 2024-09-20T22:42:36.346 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_d3n_l1_fadvise": "4", 2024-09-20T22:42:36.346 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_d3n_l1_local_datacache_enabled": "false", 2024-09-20T22:42:36.346 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_d3n_libaio_aio_num": "64", 2024-09-20T22:42:36.346 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_d3n_libaio_aio_threads": "20", 2024-09-20T22:42:36.346 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_d4n_host": "127.0.0.1", 2024-09-20T22:42:36.346 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_d4n_port": "6379", 2024-09-20T22:42:36.346 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_data": "/var/lib/ceph/radosgw/ceph-a", 2024-09-20T22:42:36.346 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_data_log_changes_size": "1000", 2024-09-20T22:42:36.346 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_data_log_num_shards": "128", 2024-09-20T22:42:36.346 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_data_log_obj_prefix": "data_log", 2024-09-20T22:42:36.346 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_data_log_window": "30", 2024-09-20T22:42:36.346 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_data_notify_interval_msec": "0", 2024-09-20T22:42:36.346 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_data_sync_poll_interval": "20", 2024-09-20T22:42:36.347 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_data_sync_spawn_window": "20", 2024-09-20T22:42:36.347 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_debug_inject_latency_bi_unlink": "0", 2024-09-20T22:42:36.347 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_debug_inject_olh_cancel_modification_err": "false", 2024-09-20T22:42:36.347 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_debug_inject_set_olh_err": "0", 2024-09-20T22:42:36.347 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_default_data_log_backing": "fifo", 2024-09-20T22:42:36.347 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_default_realm_info_oid": "default.realm", 2024-09-20T22:42:36.347 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_default_region_info_oid": "default.region", 2024-09-20T22:42:36.347 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_default_zone_info_oid": "default.zone", 2024-09-20T22:42:36.347 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_default_zonegroup_info_oid": "default.zonegroup", 2024-09-20T22:42:36.347 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_defer_to_bucket_acls": "", 2024-09-20T22:42:36.347 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_delete_multi_obj_max_num": "1000", 2024-09-20T22:42:36.347 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_dmclock_admin_lim": "0.000000", 2024-09-20T22:42:36.347 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_dmclock_admin_res": "100.000000", 2024-09-20T22:42:36.347 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_dmclock_admin_wgt": "100.000000", 2024-09-20T22:42:36.347 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_dmclock_auth_lim": "0.000000", 2024-09-20T22:42:36.347 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_dmclock_auth_res": "200.000000", 2024-09-20T22:42:36.348 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_dmclock_auth_wgt": "100.000000", 2024-09-20T22:42:36.348 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_dmclock_data_lim": "0.000000", 2024-09-20T22:42:36.348 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_dmclock_data_res": "500.000000", 2024-09-20T22:42:36.348 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_dmclock_data_wgt": "500.000000", 2024-09-20T22:42:36.348 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_dmclock_metadata_lim": "0.000000", 2024-09-20T22:42:36.348 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_dmclock_metadata_res": "500.000000", 2024-09-20T22:42:36.348 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_dmclock_metadata_wgt": "500.000000", 2024-09-20T22:42:36.348 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_dns_name": "", 2024-09-20T22:42:36.348 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_dns_s3website_name": "", 2024-09-20T22:42:36.348 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_dynamic_resharding": "true", 2024-09-20T22:42:36.348 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin, sts, iam, notifications", 2024-09-20T22:42:36.348 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_enable_gc_threads": "true", 2024-09-20T22:42:36.348 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_enable_lc_threads": "true", 2024-09-20T22:42:36.348 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_enable_ops_log": "false", 2024-09-20T22:42:36.349 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_enable_quota_threads": "true", 2024-09-20T22:42:36.349 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_enable_static_website": "false", 2024-09-20T22:42:36.349 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_enable_usage_log": "false", 2024-09-20T22:42:36.349 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_enforce_swift_acls": "true", 2024-09-20T22:42:36.349 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_exit_timeout_secs": "120", 2024-09-20T22:42:36.349 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_expose_bucket": "false", 2024-09-20T22:42:36.349 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_extended_http_attrs": "", 2024-09-20T22:42:36.349 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_filter": "none", 2024-09-20T22:42:36.349 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_frontend_defaults": "beast ssl_certificate=config://rgw/cert/$realm/$zone.crt ssl_private_key=config://rgw/cert/$realm/$zone.key", 2024-09-20T22:42:36.349 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_frontends": "beast port=7480", 2024-09-20T22:42:36.349 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_gc_max_concurrent_io": "10", 2024-09-20T22:42:36.349 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_gc_max_deferred": "50", 2024-09-20T22:42:36.349 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_gc_max_deferred_entries_size": "3072", 2024-09-20T22:42:36.349 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_gc_max_objs": "32", 2024-09-20T22:42:36.350 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_gc_max_queue_size": "134213632", 2024-09-20T22:42:36.350 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_gc_max_trim_chunk": "16", 2024-09-20T22:42:36.350 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_gc_obj_min_wait": "7200", 2024-09-20T22:42:36.350 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_gc_processor_max_time": "3600", 2024-09-20T22:42:36.350 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_gc_processor_period": "3600", 2024-09-20T22:42:36.350 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_get_obj_max_req_size": "4194304", 2024-09-20T22:42:36.350 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_get_obj_window_size": "16777216", 2024-09-20T22:42:36.350 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_healthcheck_disabling_path": "", 2024-09-20T22:42:36.350 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_ignore_get_invalid_range": "false", 2024-09-20T22:42:36.350 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_init_timeout": "300", 2024-09-20T22:42:36.350 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_inject_notify_timeout_probability": "0.000000", 2024-09-20T22:42:36.350 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_json_config": "/var/lib/ceph/radosgw/config.json", 2024-09-20T22:42:36.350 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_kafka_connection_idle": "300", 2024-09-20T22:42:36.351 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_kafka_message_timeout": "5000", 2024-09-20T22:42:36.351 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_kafka_sleep_timeout": "10", 2024-09-20T22:42:36.351 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_keystone_accepted_admin_roles": "", 2024-09-20T22:42:36.351 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_keystone_accepted_reader_roles": "", 2024-09-20T22:42:36.351 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_keystone_accepted_roles": "Member, admin", 2024-09-20T22:42:36.351 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_keystone_admin_domain": "", 2024-09-20T22:42:36.351 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_keystone_admin_password": "", 2024-09-20T22:42:36.351 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_keystone_admin_password_path": "", 2024-09-20T22:42:36.351 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_keystone_admin_project": "", 2024-09-20T22:42:36.351 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_keystone_admin_tenant": "", 2024-09-20T22:42:36.351 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_keystone_admin_token": "", 2024-09-20T22:42:36.351 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_keystone_admin_token_path": "", 2024-09-20T22:42:36.352 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_keystone_admin_user": "", 2024-09-20T22:42:36.352 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_keystone_api_version": "2", 2024-09-20T22:42:36.352 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_keystone_barbican_domain": "", 2024-09-20T22:42:36.352 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_keystone_barbican_password": "", 2024-09-20T22:42:36.352 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_keystone_barbican_project": "", 2024-09-20T22:42:36.352 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_keystone_barbican_tenant": "", 2024-09-20T22:42:36.352 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_keystone_barbican_user": "", 2024-09-20T22:42:36.352 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_keystone_expired_token_cache_expiration": "3600", 2024-09-20T22:42:36.352 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_keystone_implicit_tenants": "false", 2024-09-20T22:42:36.352 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_keystone_service_token_accepted_roles": "admin", 2024-09-20T22:42:36.352 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_keystone_service_token_enabled": "false", 2024-09-20T22:42:36.352 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_keystone_token_cache_size": "10000", 2024-09-20T22:42:36.353 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_keystone_url": "", 2024-09-20T22:42:36.353 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_keystone_verify_ssl": "true", 2024-09-20T22:42:36.353 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_lc_debug_interval": "-1", 2024-09-20T22:42:36.353 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_lc_lock_max_time": "90", 2024-09-20T22:42:36.353 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_lc_max_objs": "32", 2024-09-20T22:42:36.353 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_lc_max_rules": "1000", 2024-09-20T22:42:36.353 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_lc_max_worker": "3", 2024-09-20T22:42:36.353 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_lc_max_wp_worker": "3", 2024-09-20T22:42:36.353 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_lc_thread_delay": "0", 2024-09-20T22:42:36.353 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com", 2024-09-20T22:42:36.353 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_ldap_dnattr": "uid", 2024-09-20T22:42:36.353 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com", 2024-09-20T22:42:36.353 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_ldap_searchfilter": "", 2024-09-20T22:42:36.354 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_ldap_secret": "/etc/openldap/secret", 2024-09-20T22:42:36.354 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_ldap_uri": "", 2024-09-20T22:42:36.354 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_lifecycle_work_time": "00:00-06:00", 2024-09-20T22:42:36.354 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_list_bucket_min_readahead": "1000", 2024-09-20T22:42:36.354 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_list_buckets_max_chunk": "1000", 2024-09-20T22:42:36.354 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_log_http_headers": "", 2024-09-20T22:42:36.354 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_log_nonexistent_bucket": "false", 2024-09-20T22:42:36.354 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n", 2024-09-20T22:42:36.354 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_log_object_name_utc": "false", 2024-09-20T22:42:36.354 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_lua_max_memory_per_state": "128000", 2024-09-20T22:42:36.354 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_luarocks_location": "/tmp/rgw_luarocks/mon.a", 2024-09-20T22:42:36.354 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_max_attr_name_len": "0", 2024-09-20T22:42:36.354 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_max_attr_size": "0", 2024-09-20T22:42:36.354 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_max_attrs_num_in_req": "0", 2024-09-20T22:42:36.355 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_max_chunk_size": "4194304", 2024-09-20T22:42:36.355 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_max_concurrent_requests": "1024", 2024-09-20T22:42:36.355 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_max_copy_obj_concurrent_io": "10", 2024-09-20T22:42:36.355 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_max_dynamic_shards": "1999", 2024-09-20T22:42:36.355 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_max_listing_results": "1000", 2024-09-20T22:42:36.355 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_max_notify_retries": "10", 2024-09-20T22:42:36.355 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_max_objs_per_shard": "100000", 2024-09-20T22:42:36.355 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_max_put_param_size": "1048576", 2024-09-20T22:42:36.355 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_max_put_size": "5368709120", 2024-09-20T22:42:36.355 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_max_slo_entries": "1000", 2024-09-20T22:42:36.355 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_md_log_max_shards": "64", 2024-09-20T22:42:36.355 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_md_notify_interval_msec": "200", 2024-09-20T22:42:36.355 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_meta_sync_poll_interval": "20", 2024-09-20T22:42:36.355 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_meta_sync_spawn_window": "20", 2024-09-20T22:42:36.355 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_mime_types_file": "/etc/mime.types", 2024-09-20T22:42:36.356 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_mp_lock_max_time": "600", 2024-09-20T22:42:36.356 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_multi_obj_del_max_aio": "16", 2024-09-20T22:42:36.356 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_multipart_min_part_size": "5242880", 2024-09-20T22:42:36.356 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_multipart_part_upload_limit": "10000", 2024-09-20T22:42:36.356 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_nfs_fhcache_partitions": "3", 2024-09-20T22:42:36.356 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_nfs_fhcache_size": "2017", 2024-09-20T22:42:36.356 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_nfs_frontends": "rgw-nfs", 2024-09-20T22:42:36.356 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_nfs_lru_lane_hiwat": "911", 2024-09-20T22:42:36.356 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_nfs_lru_lanes": "5", 2024-09-20T22:42:36.356 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_nfs_max_gc": "300", 2024-09-20T22:42:36.356 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_nfs_namespace_expire_secs": "300", 2024-09-20T22:42:36.356 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_nfs_run_gc_threads": "false", 2024-09-20T22:42:36.356 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_nfs_run_lc_threads": "false", 2024-09-20T22:42:36.356 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_nfs_run_quota_threads": "false", 2024-09-20T22:42:36.357 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_nfs_run_sync_thread": "false", 2024-09-20T22:42:36.357 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_nfs_s3_fast_attrs": "false", 2024-09-20T22:42:36.357 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_nfs_write_completion_interval_s": "10", 2024-09-20T22:42:36.357 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_num_async_rados_threads": "32", 2024-09-20T22:42:36.357 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_num_control_oids": "8", 2024-09-20T22:42:36.357 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_numa_node": "-1", 2024-09-20T22:42:36.357 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_obj_stripe_size": "4194304", 2024-09-20T22:42:36.357 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_obj_tombstone_cache_size": "1000", 2024-09-20T22:42:36.357 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_objexp_chunk_size": "100", 2024-09-20T22:42:36.357 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_objexp_gc_interval": "600", 2024-09-20T22:42:36.357 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_objexp_hints_num_shards": "127", 2024-09-20T22:42:36.357 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_olh_pending_timeout_sec": "3600", 2024-09-20T22:42:36.357 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_op_thread_suicide_timeout": "0", 2024-09-20T22:42:36.357 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_op_thread_timeout": "600", 2024-09-20T22:42:36.358 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_op_tracing": "false", 2024-09-20T22:42:36.358 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_opa_token": "", 2024-09-20T22:42:36.358 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_opa_url": "", 2024-09-20T22:42:36.358 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_opa_verify_ssl": "true", 2024-09-20T22:42:36.358 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_ops_log_data_backlog": "5242880", 2024-09-20T22:42:36.358 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_ops_log_file_path": "/var/log/ceph/ops-log-ceph-mon.a.log", 2024-09-20T22:42:36.358 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_ops_log_rados": "false", 2024-09-20T22:42:36.358 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_ops_log_socket_path": "", 2024-09-20T22:42:36.358 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_override_bucket_index_max_shards": "0", 2024-09-20T22:42:36.358 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_pending_bucket_index_op_expiration": "120", 2024-09-20T22:42:36.358 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_period_latest_epoch_info_oid": ".latest_epoch", 2024-09-20T22:42:36.358 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_period_push_interval": "2.000000", 2024-09-20T22:42:36.358 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_period_push_interval_max": "30.000000", 2024-09-20T22:42:36.358 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_period_root_pool": ".rgw.root", 2024-09-20T22:42:36.359 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_policy_reject_invalid_principals": "true", 2024-09-20T22:42:36.359 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_posix_base_path": "/tmp/rgw_posix_driver", 2024-09-20T22:42:36.359 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_posix_cache_lanes": "3", 2024-09-20T22:42:36.359 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_posix_cache_lmdb_count": "3", 2024-09-20T22:42:36.359 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_posix_cache_max_buckets": "100", 2024-09-20T22:42:36.359 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_posix_cache_partitions": "3", 2024-09-20T22:42:36.359 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_posix_database_root": "/var/lib/ceph/radosgw", 2024-09-20T22:42:36.359 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_print_continue": "true", 2024-09-20T22:42:36.359 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_print_prohibited_content_length": "false", 2024-09-20T22:42:36.359 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_put_obj_max_window_size": "67108864", 2024-09-20T22:42:36.359 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_put_obj_min_window_size": "16777216", 2024-09-20T22:42:36.359 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_rados_pool_autoscale_bias": "4.000000", 2024-09-20T22:42:36.359 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_rados_pool_recovery_priority": "5", 2024-09-20T22:42:36.360 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_rados_tracing": "false", 2024-09-20T22:42:36.360 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_realm": "", 2024-09-20T22:42:36.360 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_realm_id": "", 2024-09-20T22:42:36.360 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_realm_root_pool": ".rgw.root", 2024-09-20T22:42:36.360 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_region": "", 2024-09-20T22:42:36.360 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_region_root_pool": ".rgw.root", 2024-09-20T22:42:36.360 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_relaxed_region_enforcement": "false", 2024-09-20T22:42:36.360 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_relaxed_s3_bucket_names": "false", 2024-09-20T22:42:36.360 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_remote_addr_param": "REMOTE_ADDR", 2024-09-20T22:42:36.360 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_request_uri": "", 2024-09-20T22:42:36.360 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_reshard_batch_size": "64", 2024-09-20T22:42:36.360 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_reshard_bucket_lock_duration": "360", 2024-09-20T22:42:36.360 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_reshard_max_aio": "128", 2024-09-20T22:42:36.361 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_reshard_num_logs": "16", 2024-09-20T22:42:36.361 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_reshard_thread_interval": "600", 2024-09-20T22:42:36.361 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_resolve_cname": "false", 2024-09-20T22:42:36.361 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_rest_getusage_op_compat": "false", 2024-09-20T22:42:36.361 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_run_sync_thread": "true", 2024-09-20T22:42:36.361 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_s3_auth_disable_signature_url": "false", 2024-09-20T22:42:36.361 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_s3_auth_order": "sts, external, local", 2024-09-20T22:42:36.361 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_s3_auth_use_keystone": "false", 2024-09-20T22:42:36.361 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_s3_auth_use_ldap": "false", 2024-09-20T22:42:36.361 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_s3_auth_use_rados": "true", 2024-09-20T22:42:36.361 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_s3_auth_use_sts": "false", 2024-09-20T22:42:36.361 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_s3_client_max_sig_ver": "-1", 2024-09-20T22:42:36.361 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_s3_success_create_obj_status": "0", 2024-09-20T22:42:36.361 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_safe_max_objects_per_shard": "102400", 2024-09-20T22:42:36.362 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_scheduler_type": "throttler", 2024-09-20T22:42:36.362 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_script_uri": "", 2024-09-20T22:42:36.362 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_service_provider_name": "", 2024-09-20T22:42:36.362 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_shard_warning_threshold": "90.000000", 2024-09-20T22:42:36.362 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_sts_client_id": "", 2024-09-20T22:42:36.362 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_sts_client_secret": "", 2024-09-20T22:42:36.362 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_sts_entry": "sts", 2024-09-20T22:42:36.362 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_sts_key": "sts", 2024-09-20T22:42:36.362 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_sts_max_session_duration": "43200", 2024-09-20T22:42:36.362 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_sts_min_session_duration": "900", 2024-09-20T22:42:36.362 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_sts_token_introspection_url": "", 2024-09-20T22:42:36.362 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_swift_account_in_url": "false", 2024-09-20T22:42:36.362 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_swift_auth_entry": "auth", 2024-09-20T22:42:36.363 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_swift_auth_url": "", 2024-09-20T22:42:36.363 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_swift_custom_header": "", 2024-09-20T22:42:36.363 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_swift_enforce_content_length": "false", 2024-09-20T22:42:36.363 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_swift_need_stats": "true", 2024-09-20T22:42:36.363 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_swift_tenant_name": "", 2024-09-20T22:42:36.363 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_swift_token_expiration": "86400", 2024-09-20T22:42:36.363 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_swift_url": "", 2024-09-20T22:42:36.363 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_swift_url_prefix": "swift", 2024-09-20T22:42:36.363 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_swift_versioning_enabled": "false", 2024-09-20T22:42:36.363 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_sync_data_full_inject_err_probability": "0.000000", 2024-09-20T22:42:36.363 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_sync_data_inject_err_probability": "0.000000", 2024-09-20T22:42:36.363 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_sync_lease_period": "120", 2024-09-20T22:42:36.363 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_sync_log_trim_concurrent_buckets": "4", 2024-09-20T22:42:36.363 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_sync_log_trim_interval": "1200", 2024-09-20T22:42:36.363 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_sync_log_trim_max_buckets": "16", 2024-09-20T22:42:36.364 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_sync_log_trim_min_cold_buckets": "4", 2024-09-20T22:42:36.364 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_sync_meta_inject_err_probability": "0.000000", 2024-09-20T22:42:36.364 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_sync_obj_etag_verify": "false", 2024-09-20T22:42:36.364 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_sync_trace_history_size": "4096", 2024-09-20T22:42:36.364 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_sync_trace_per_node_log_size": "32", 2024-09-20T22:42:36.364 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_sync_trace_servicemap_update_interval": "10", 2024-09-20T22:42:36.364 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_thread_pool_size": "512", 2024-09-20T22:42:36.364 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_topic_persistency_max_retries": "0", 2024-09-20T22:42:36.364 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_topic_persistency_sleep_duration": "0", 2024-09-20T22:42:36.364 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_topic_persistency_time_to_live": "0", 2024-09-20T22:42:36.364 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_topic_require_publish_policy": "false", 2024-09-20T22:42:36.364 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_torrent_comment": "", 2024-09-20T22:42:36.364 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_torrent_createby": "", 2024-09-20T22:42:36.364 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_torrent_encoding": "", 2024-09-20T22:42:36.364 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_torrent_flag": "false", 2024-09-20T22:42:36.364 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_torrent_max_size": "5368709120", 2024-09-20T22:42:36.365 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_torrent_origin": "", 2024-09-20T22:42:36.365 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_torrent_sha_unit": "524288", 2024-09-20T22:42:36.365 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_torrent_tracker": "", 2024-09-20T22:42:36.365 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_trust_forwarded_https": "false", 2024-09-20T22:42:36.365 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_usage_log_flush_threshold": "1024", 2024-09-20T22:42:36.365 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_usage_log_tick_interval": "30", 2024-09-20T22:42:36.365 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_usage_max_shards": "32", 2024-09-20T22:42:36.365 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_usage_max_user_shards": "1", 2024-09-20T22:42:36.365 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_use_opa_authz": "false", 2024-09-20T22:42:36.365 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_user_counters_cache": "false", 2024-09-20T22:42:36.365 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_user_counters_cache_size": "10000", 2024-09-20T22:42:36.365 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_user_default_quota_max_objects": "-1", 2024-09-20T22:42:36.365 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_user_default_quota_max_size": "-1", 2024-09-20T22:42:36.365 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_user_max_buckets": "1000", 2024-09-20T22:42:36.365 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_user_policies_max_num": "100", 2024-09-20T22:42:36.365 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_user_quota_bucket_sync_interval": "180", 2024-09-20T22:42:36.366 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_user_quota_sync_idle_users": "false", 2024-09-20T22:42:36.366 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_user_quota_sync_interval": "86400", 2024-09-20T22:42:36.366 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_user_quota_sync_wait_time": "86400", 2024-09-20T22:42:36.366 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_user_unique_email": "true", 2024-09-20T22:42:36.366 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_verify_ssl": "true", 2024-09-20T22:42:36.366 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_website_routing_rules_max_num": "50", 2024-09-20T22:42:36.366 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_zone": "", 2024-09-20T22:42:36.366 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_zone_id": "", 2024-09-20T22:42:36.366 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_zone_root_pool": ".rgw.root", 2024-09-20T22:42:36.366 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_zonegroup": "", 2024-09-20T22:42:36.366 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_zonegroup_id": "", 2024-09-20T22:42:36.366 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw_zonegroup_root_pool": ".rgw.root", 2024-09-20T22:42:36.366 INFO:teuthology.orchestra.run.smithi084.stdout: "rgwlc_auto_session_clear": "true", 2024-09-20T22:42:36.366 INFO:teuthology.orchestra.run.smithi084.stdout: "rgwlc_skip_bucket_step": "false", 2024-09-20T22:42:36.366 INFO:teuthology.orchestra.run.smithi084.stdout: "rocksdb_block_size": "4096", 2024-09-20T22:42:36.367 INFO:teuthology.orchestra.run.smithi084.stdout: "rocksdb_bloom_bits_per_key": "20", 2024-09-20T22:42:36.367 INFO:teuthology.orchestra.run.smithi084.stdout: "rocksdb_cache_index_and_filter_blocks": "true", 2024-09-20T22:42:36.367 INFO:teuthology.orchestra.run.smithi084.stdout: "rocksdb_cache_index_and_filter_blocks_with_high_priority": "false", 2024-09-20T22:42:36.367 INFO:teuthology.orchestra.run.smithi084.stdout: "rocksdb_cache_row_ratio": "0.000000", 2024-09-20T22:42:36.367 INFO:teuthology.orchestra.run.smithi084.stdout: "rocksdb_cache_shard_bits": "4", 2024-09-20T22:42:36.367 INFO:teuthology.orchestra.run.smithi084.stdout: "rocksdb_cache_size": "536870912", 2024-09-20T22:42:36.367 INFO:teuthology.orchestra.run.smithi084.stdout: "rocksdb_cache_type": "binned_lru", 2024-09-20T22:42:36.367 INFO:teuthology.orchestra.run.smithi084.stdout: "rocksdb_cf_compact_on_deletion": "true", 2024-09-20T22:42:36.367 INFO:teuthology.orchestra.run.smithi084.stdout: "rocksdb_cf_compact_on_deletion_sliding_window": "32768", 2024-09-20T22:42:36.367 INFO:teuthology.orchestra.run.smithi084.stdout: "rocksdb_cf_compact_on_deletion_trigger": "16384", 2024-09-20T22:42:36.367 INFO:teuthology.orchestra.run.smithi084.stdout: "rocksdb_collect_compaction_stats": "false", 2024-09-20T22:42:36.367 INFO:teuthology.orchestra.run.smithi084.stdout: "rocksdb_collect_extended_stats": "false", 2024-09-20T22:42:36.367 INFO:teuthology.orchestra.run.smithi084.stdout: "rocksdb_collect_memory_stats": "false", 2024-09-20T22:42:36.368 INFO:teuthology.orchestra.run.smithi084.stdout: "rocksdb_delete_range_threshold": "1048576", 2024-09-20T22:42:36.368 INFO:teuthology.orchestra.run.smithi084.stdout: "rocksdb_index_type": "binary_search", 2024-09-20T22:42:36.368 INFO:teuthology.orchestra.run.smithi084.stdout: "rocksdb_log_to_ceph_log": "true", 2024-09-20T22:42:36.368 INFO:teuthology.orchestra.run.smithi084.stdout: "rocksdb_metadata_block_size": "4096", 2024-09-20T22:42:36.368 INFO:teuthology.orchestra.run.smithi084.stdout: "rocksdb_partition_filters": "false", 2024-09-20T22:42:36.368 INFO:teuthology.orchestra.run.smithi084.stdout: "rocksdb_perf": "false", 2024-09-20T22:42:36.368 INFO:teuthology.orchestra.run.smithi084.stdout: "rocksdb_pin_l0_filter_and_index_blocks_in_cache": "false", 2024-09-20T22:42:36.368 INFO:teuthology.orchestra.run.smithi084.stdout: "rotating_keys_bootstrap_timeout": "30", 2024-09-20T22:42:36.368 INFO:teuthology.orchestra.run.smithi084.stdout: "rotating_keys_renewal_timeout": "10", 2024-09-20T22:42:36.368 INFO:teuthology.orchestra.run.smithi084.stdout: "run_dir": "/var/run/ceph", 2024-09-20T22:42:36.368 INFO:teuthology.orchestra.run.smithi084.stdout: "seastore_block_create": "true", 2024-09-20T22:42:36.368 INFO:teuthology.orchestra.run.smithi084.stdout: "seastore_cache_lru_size": "67108864", 2024-09-20T22:42:36.368 INFO:teuthology.orchestra.run.smithi084.stdout: "seastore_cbjournal_size": "5368709120", 2024-09-20T22:42:36.368 INFO:teuthology.orchestra.run.smithi084.stdout: "seastore_data_delta_based_overwrite": "0", 2024-09-20T22:42:36.369 INFO:teuthology.orchestra.run.smithi084.stdout: "seastore_default_max_object_size": "16777216", 2024-09-20T22:42:36.369 INFO:teuthology.orchestra.run.smithi084.stdout: "seastore_default_object_metadata_reservation": "16777216", 2024-09-20T22:42:36.369 INFO:teuthology.orchestra.run.smithi084.stdout: "seastore_device_size": "53687091200", 2024-09-20T22:42:36.369 INFO:teuthology.orchestra.run.smithi084.stdout: "seastore_disable_end_to_end_data_protection": "true", 2024-09-20T22:42:36.369 INFO:teuthology.orchestra.run.smithi084.stdout: "seastore_full_integrity_check": "false", 2024-09-20T22:42:36.369 INFO:teuthology.orchestra.run.smithi084.stdout: "seastore_journal_batch_capacity": "16", 2024-09-20T22:42:36.369 INFO:teuthology.orchestra.run.smithi084.stdout: "seastore_journal_batch_flush_size": "16777216", 2024-09-20T22:42:36.369 INFO:teuthology.orchestra.run.smithi084.stdout: "seastore_journal_batch_preferred_fullness": "0.950000", 2024-09-20T22:42:36.369 INFO:teuthology.orchestra.run.smithi084.stdout: "seastore_journal_iodepth_limit": "5", 2024-09-20T22:42:36.369 INFO:teuthology.orchestra.run.smithi084.stdout: "seastore_main_device_type": "SSD", 2024-09-20T22:42:36.369 INFO:teuthology.orchestra.run.smithi084.stdout: "seastore_max_concurrent_transactions": "8", 2024-09-20T22:42:36.369 INFO:teuthology.orchestra.run.smithi084.stdout: "seastore_max_data_allocation_size": "32768", 2024-09-20T22:42:36.369 INFO:teuthology.orchestra.run.smithi084.stdout: "seastore_multiple_tiers_default_evict_ratio": "0.600000", 2024-09-20T22:42:36.369 INFO:teuthology.orchestra.run.smithi084.stdout: "seastore_multiple_tiers_fast_evict_ratio": "0.700000", 2024-09-20T22:42:36.369 INFO:teuthology.orchestra.run.smithi084.stdout: "seastore_multiple_tiers_stop_evict_ratio": "0.500000", 2024-09-20T22:42:36.369 INFO:teuthology.orchestra.run.smithi084.stdout: "seastore_obj_data_write_amplification": "1.250000", 2024-09-20T22:42:36.369 INFO:teuthology.orchestra.run.smithi084.stdout: "seastore_segment_size": "67108864", 2024-09-20T22:42:36.370 INFO:teuthology.orchestra.run.smithi084.stdout: "set_keepcaps": "false", 2024-09-20T22:42:36.370 INFO:teuthology.orchestra.run.smithi084.stdout: "setgroup": "ceph", 2024-09-20T22:42:36.370 INFO:teuthology.orchestra.run.smithi084.stdout: "setuser": "ceph", 2024-09-20T22:42:36.370 INFO:teuthology.orchestra.run.smithi084.stdout: "setuser_match_path": "", 2024-09-20T22:42:36.370 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_misplaced_ratio": "0.050000", 2024-09-20T22:42:36.370 INFO:teuthology.orchestra.run.smithi084.stdout: "thp": "false", 2024-09-20T22:42:36.370 INFO:teuthology.orchestra.run.smithi084.stdout: "threadpool_default_timeout": "60", 2024-09-20T22:42:36.370 INFO:teuthology.orchestra.run.smithi084.stdout: "threadpool_empty_queue_max_wait": "2", 2024-09-20T22:42:36.370 INFO:teuthology.orchestra.run.smithi084.stdout: "throttler_perf_counter": "true" 2024-09-20T22:42:36.370 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-20T22:42:36.381 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:36 smithi154 ceph-mon[33264]: purged_snaps scrub starts 2024-09-20T22:42:36.381 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:36 smithi154 ceph-mon[33264]: purged_snaps scrub ok 2024-09-20T22:42:36.381 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:36 smithi154 ceph-mon[33264]: osdmap e85: 8 total, 8 up, 8 in 2024-09-20T22:42:36.544 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:36 smithi084 ceph-mon[40024]: purged_snaps scrub starts 2024-09-20T22:42:36.544 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:36 smithi084 ceph-mon[40024]: purged_snaps scrub ok 2024-09-20T22:42:36.544 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:36 smithi084 ceph-mon[40024]: osdmap e85: 8 total, 8 up, 8 in 2024-09-20T22:42:36.544 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:36 smithi084 ceph-mon[29774]: purged_snaps scrub starts 2024-09-20T22:42:36.544 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:36 smithi084 ceph-mon[29774]: purged_snaps scrub ok 2024-09-20T22:42:36.544 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:36 smithi084 ceph-mon[29774]: osdmap e85: 8 total, 8 up, 8 in 2024-09-20T22:42:37.027 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=0 2024-09-20T22:42:37.271 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:42:37.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:37 smithi084 ceph-mon[29774]: pgmap v25: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T22:42:37.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:37 smithi084 ceph-mon[40024]: pgmap v25: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T22:42:37.384 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:37 smithi154 ceph-mon[33264]: pgmap v25: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T22:42:37.837 INFO:teuthology.orchestra.run.smithi084.stdout:mon.a: {} 2024-09-20T22:42:37.837 INFO:teuthology.orchestra.run.smithi084.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-20T22:42:37.847 INFO:teuthology.orchestra.run.smithi084.stdout:mon.b: {} 2024-09-20T22:42:37.847 INFO:teuthology.orchestra.run.smithi084.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-20T22:42:37.855 INFO:teuthology.orchestra.run.smithi084.stdout:mon.c: {} 2024-09-20T22:42:37.855 INFO:teuthology.orchestra.run.smithi084.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-20T22:42:38.384 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:38 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:42:38.385 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:38 smithi154 ceph-mon[33264]: 3.3 scrub starts 2024-09-20T22:42:38.385 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:38 smithi154 ceph-mon[33264]: 3.3 scrub ok 2024-09-20T22:42:38.385 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:38 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/3270682282' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-20T22:42:38.385 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:38 smithi154 ceph-mon[33264]: osdmap e86: 8 total, 8 up, 8 in 2024-09-20T22:42:38.527 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:42:38 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: ::ffff:172.21.15.154 - - [20/Sep/2024:22:42:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-20T22:42:38.528 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:38 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:42:38.528 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:38 smithi084 ceph-mon[29774]: 3.3 scrub starts 2024-09-20T22:42:38.528 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:38 smithi084 ceph-mon[29774]: 3.3 scrub ok 2024-09-20T22:42:38.528 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:38 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/3270682282' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-20T22:42:38.528 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:38 smithi084 ceph-mon[29774]: osdmap e86: 8 total, 8 up, 8 in 2024-09-20T22:42:38.528 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:38 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:42:38.528 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:38 smithi084 ceph-mon[40024]: 3.3 scrub starts 2024-09-20T22:42:38.528 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:38 smithi084 ceph-mon[40024]: 3.3 scrub ok 2024-09-20T22:42:38.528 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:38 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/3270682282' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-20T22:42:38.528 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:38 smithi084 ceph-mon[40024]: osdmap e86: 8 total, 8 up, 8 in 2024-09-20T22:42:38.554 INFO:teuthology.run_tasks:Running task rados... 2024-09-20T22:42:39.655 INFO:tasks.rados:Beginning rados... 2024-09-20T22:42:39.656 DEBUG:teuthology.run_tasks:Unwinding manager rados 2024-09-20T22:42:39.670 INFO:tasks.rados:joining rados 2024-09-20T22:42:39.671 INFO:tasks.thrashosds.thrasher:starting do_thrash 2024-09-20T22:42:39.671 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-20T22:42:39.671 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-20T22:42:39.671 INFO:tasks.thrashosds.thrasher:Growing pool .mgr 2024-09-20T22:42:39.671 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph pg dump --format=json 2024-09-20T22:42:39.672 INFO:tasks.thrashosds.thrasher:starting do_sighup with a delay of 0.1 2024-09-20T22:42:39.672 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:42:39.672 INFO:tasks.thrashosds.thrasher:starting do_optrack_toggle with a delay of 2.0 2024-09-20T22:42:39.672 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-20T22:42:39.673 INFO:tasks.thrashosds.thrasher:starting do_dump_ops 2024-09-20T22:42:39.673 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-20T22:42:39.673 INFO:tasks.thrashosds.thrasher:starting do_noscrub_toggle with a delay of 2.0 2024-09-20T22:42:39.673 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd set noscrub 2024-09-20T22:42:39.674 INFO:tasks.rados:clients are ['client.0', 'client.1'] 2024-09-20T22:42:39.674 INFO:tasks.rados:starting run 0 out of 1 2024-09-20T22:42:39.675 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-mon[33264]: pgmap v26: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T22:42:39.675 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:39 smithi084 ceph-mon[40024]: pgmap v26: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T22:42:39.676 INFO:journalctl@ceph.iscsi.iscsi.a.smithi154.stdout:Sep 20 22:42:38 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-iscsi-iscsi-a[59730]: debug there is no tcmu-runner data available 2024-09-20T22:42:39.676 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:39 smithi084 ceph-mon[29774]: pgmap v26: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T22:42:39.677 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 systemd[1]: Starting Ceph grafana.a for a9c0bc70-77a0-11ef-baf5-efdc52797490... 2024-09-20T22:42:39.683 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 podman[62603]: 2024-09-20 22:42:39.257731288 +0000 UTC m=+0.016697803 image pull 2bacad6d85d802cd176c146528adcc0e47a8a83e94bc4a8b7ba1a9b9ef2c65e7 quay.io/ceph/grafana:9.4.12 2024-09-20T22:42:39.684 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 podman[62603]: 2024-09-20 22:42:39.389140532 +0000 UTC m=+0.148107040 container create 919357740c28646a2ab509029b568ec9ad0633a06f1a561fc4007b3f8dba9eed (image=quay.io/ceph/grafana:9.4.12, name=ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a, maintainer=Grafana Labs ) 2024-09-20T22:42:39.684 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 podman[62603]: 2024-09-20 22:42:39.59954223 +0000 UTC m=+0.358508744 container init 919357740c28646a2ab509029b568ec9ad0633a06f1a561fc4007b3f8dba9eed (image=quay.io/ceph/grafana:9.4.12, name=ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a, maintainer=Grafana Labs ) 2024-09-20T22:42:39.684 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 podman[62603]: 2024-09-20 22:42:39.6023802 +0000 UTC m=+0.361346697 container start 919357740c28646a2ab509029b568ec9ad0633a06f1a561fc4007b3f8dba9eed (image=quay.io/ceph/grafana:9.4.12, name=ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a, maintainer=Grafana Labs ) 2024-09-20T22:42:39.699 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:42:39.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:42:39.700 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:42:39.801 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:42:39.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:42:39.823 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:42:39.824 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:42:39.925 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:42:39.935 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 bash[62603]: 919357740c28646a2ab509029b568ec9ad0633a06f1a561fc4007b3f8dba9eed 2024-09-20T22:42:39.935 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 systemd[1]: Started Ceph grafana.a for a9c0bc70-77a0-11ef-baf5-efdc52797490. 2024-09-20T22:42:39.936 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=settings t=2024-09-20T22:42:39.755125619Z level=info msg="Starting Grafana" version=9.4.12 commit=37ee25ba46 branch=HEAD compiled=2023-05-22T15:58:05Z 2024-09-20T22:42:39.936 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=settings t=2024-09-20T22:42:39.755298543Z level=warn msg="\"sentry\" frontend logging provider is deprecated and will be removed in the next major version. Use \"grafana\" provider instead." 2024-09-20T22:42:39.936 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=settings t=2024-09-20T22:42:39.755309134Z level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini 2024-09-20T22:42:39.936 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=settings t=2024-09-20T22:42:39.755312561Z level=info msg="Config loaded from" file=/etc/grafana/grafana.ini 2024-09-20T22:42:39.936 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=settings t=2024-09-20T22:42:39.755315314Z level=info msg="Config overridden from command line" arg="default.paths.data=/var/lib/grafana" 2024-09-20T22:42:39.936 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=settings t=2024-09-20T22:42:39.755318323Z level=info msg="Config overridden from command line" arg="default.paths.logs=/var/log/grafana" 2024-09-20T22:42:39.936 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=settings t=2024-09-20T22:42:39.755320844Z level=info msg="Config overridden from command line" arg="default.paths.plugins=/var/lib/grafana/plugins" 2024-09-20T22:42:39.936 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=settings t=2024-09-20T22:42:39.755323352Z level=info msg="Config overridden from command line" arg="default.paths.provisioning=/etc/grafana/provisioning" 2024-09-20T22:42:39.936 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=settings t=2024-09-20T22:42:39.755325872Z level=info msg="Config overridden from command line" arg="default.log.mode=console" 2024-09-20T22:42:39.936 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=settings t=2024-09-20T22:42:39.755328915Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_DATA=/var/lib/grafana" 2024-09-20T22:42:39.936 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=settings t=2024-09-20T22:42:39.755331436Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_LOGS=/var/log/grafana" 2024-09-20T22:42:39.936 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=settings t=2024-09-20T22:42:39.755334438Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-09-20T22:42:39.936 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=settings t=2024-09-20T22:42:39.755336964Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-09-20T22:42:39.937 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=settings t=2024-09-20T22:42:39.755339493Z level=info msg="Path Home" path=/usr/share/grafana 2024-09-20T22:42:39.937 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=settings t=2024-09-20T22:42:39.75534208Z level=info msg="Path Data" path=/var/lib/grafana 2024-09-20T22:42:39.937 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=settings t=2024-09-20T22:42:39.755344635Z level=info msg="Path Logs" path=/var/log/grafana 2024-09-20T22:42:39.937 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=settings t=2024-09-20T22:42:39.755347006Z level=info msg="Path Plugins" path=/var/lib/grafana/plugins 2024-09-20T22:42:39.938 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=settings t=2024-09-20T22:42:39.755349557Z level=info msg="Path Provisioning" path=/etc/grafana/provisioning 2024-09-20T22:42:39.938 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=settings t=2024-09-20T22:42:39.755352371Z level=info msg="App mode production" 2024-09-20T22:42:39.938 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=sqlstore t=2024-09-20T22:42:39.755395576Z level=info msg="Connecting to DB" dbtype=sqlite3 2024-09-20T22:42:39.938 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=sqlstore t=2024-09-20T22:42:39.75541543Z level=warn msg="SQLite database file has broader permissions than it should" path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-09-20T22:42:39.938 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.756739302Z level=info msg="Starting DB migrations" 2024-09-20T22:42:39.938 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.756839618Z level=info msg="Executing migration" id="create migration_log table" 2024-09-20T22:42:39.938 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.757407744Z level=info msg="Executing migration" id="create user table" 2024-09-20T22:42:39.938 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.757861861Z level=info msg="Executing migration" id="add unique index user.login" 2024-09-20T22:42:39.939 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.758304138Z level=info msg="Executing migration" id="add unique index user.email" 2024-09-20T22:42:39.939 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.758709632Z level=info msg="Executing migration" id="drop index UQE_user_login - v1" 2024-09-20T22:42:39.939 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.759355624Z level=info msg="Executing migration" id="drop index UQE_user_email - v1" 2024-09-20T22:42:39.939 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.759759051Z level=info msg="Executing migration" id="Rename table user to user_v1 - v1" 2024-09-20T22:42:39.939 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.760572475Z level=info msg="Executing migration" id="create user table v2" 2024-09-20T22:42:39.939 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.760966271Z level=info msg="Executing migration" id="create index UQE_user_login - v2" 2024-09-20T22:42:39.939 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.761360237Z level=info msg="Executing migration" id="create index UQE_user_email - v2" 2024-09-20T22:42:39.939 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.76178377Z level=info msg="Executing migration" id="copy data_source v1 to v2" 2024-09-20T22:42:39.939 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.762149994Z level=info msg="Executing migration" id="Drop old table user_v1" 2024-09-20T22:42:39.939 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.762520688Z level=info msg="Executing migration" id="Add column help_flags1 to user table" 2024-09-20T22:42:39.939 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.762972067Z level=info msg="Executing migration" id="Update user table charset" 2024-09-20T22:42:39.939 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.763266853Z level=info msg="Executing migration" id="Add last_seen_at column to user" 2024-09-20T22:42:39.939 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.763664206Z level=info msg="Executing migration" id="Add missing user data" 2024-09-20T22:42:39.940 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.76422506Z level=info msg="Executing migration" id="Add is_disabled column to user" 2024-09-20T22:42:39.940 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.764615784Z level=info msg="Executing migration" id="Add index user.login/user.email" 2024-09-20T22:42:39.940 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.765025316Z level=info msg="Executing migration" id="Add is_service_account column to user" 2024-09-20T22:42:39.940 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.765448433Z level=info msg="Executing migration" id="Update is_service_account column to nullable" 2024-09-20T22:42:39.940 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.76683708Z level=info msg="Executing migration" id="create temp user table v1-7" 2024-09-20T22:42:39.940 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.767315568Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v1-7" 2024-09-20T22:42:39.940 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.767750867Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v1-7" 2024-09-20T22:42:39.940 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.768179435Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v1-7" 2024-09-20T22:42:39.940 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.768569066Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v1-7" 2024-09-20T22:42:39.940 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.768970887Z level=info msg="Executing migration" id="Update temp_user table charset" 2024-09-20T22:42:39.940 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.769263894Z level=info msg="Executing migration" id="drop index IDX_temp_user_email - v1" 2024-09-20T22:42:39.940 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.769630271Z level=info msg="Executing migration" id="drop index IDX_temp_user_org_id - v1" 2024-09-20T22:42:39.940 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.770056566Z level=info msg="Executing migration" id="drop index IDX_temp_user_code - v1" 2024-09-20T22:42:39.940 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.770438955Z level=info msg="Executing migration" id="drop index IDX_temp_user_status - v1" 2024-09-20T22:42:39.940 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.770786411Z level=info msg="Executing migration" id="Rename table temp_user to temp_user_tmp_qwerty - v1" 2024-09-20T22:42:39.941 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.771579276Z level=info msg="Executing migration" id="create temp_user v2" 2024-09-20T22:42:39.941 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.771985419Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v2" 2024-09-20T22:42:39.941 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.772398627Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v2" 2024-09-20T22:42:39.941 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.77276376Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v2" 2024-09-20T22:42:39.941 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.77319141Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v2" 2024-09-20T22:42:39.941 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.773589009Z level=info msg="Executing migration" id="copy temp_user v1 to v2" 2024-09-20T22:42:39.941 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.773902223Z level=info msg="Executing migration" id="drop temp_user_tmp_qwerty" 2024-09-20T22:42:39.941 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.774254185Z level=info msg="Executing migration" id="Set created for temp users that will otherwise prematurely expire" 2024-09-20T22:42:39.941 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.774565026Z level=info msg="Executing migration" id="create star table" 2024-09-20T22:42:39.941 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.77490901Z level=info msg="Executing migration" id="add unique index star.user_id_dashboard_id" 2024-09-20T22:42:39.941 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.775356607Z level=info msg="Executing migration" id="create org table v1" 2024-09-20T22:42:39.941 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.775732958Z level=info msg="Executing migration" id="create index UQE_org_name - v1" 2024-09-20T22:42:39.942 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.776138108Z level=info msg="Executing migration" id="create org_user table v1" 2024-09-20T22:42:39.942 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.776502961Z level=info msg="Executing migration" id="create index IDX_org_user_org_id - v1" 2024-09-20T22:42:39.942 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.776918689Z level=info msg="Executing migration" id="create index UQE_org_user_org_id_user_id - v1" 2024-09-20T22:42:39.942 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.777719706Z level=info msg="Executing migration" id="create index IDX_org_user_user_id - v1" 2024-09-20T22:42:39.942 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.77814507Z level=info msg="Executing migration" id="Update org table charset" 2024-09-20T22:42:39.942 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.778982252Z level=info msg="Executing migration" id="Update org_user table charset" 2024-09-20T22:42:39.942 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.779855495Z level=info msg="Executing migration" id="Migrate all Read Only Viewers to Viewers" 2024-09-20T22:42:39.942 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.780864949Z level=info msg="Executing migration" id="create dashboard table" 2024-09-20T22:42:39.942 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.781555135Z level=info msg="Executing migration" id="add index dashboard.account_id" 2024-09-20T22:42:39.942 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.7825668Z level=info msg="Executing migration" id="add unique index dashboard_account_id_slug" 2024-09-20T22:42:39.942 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.783321652Z level=info msg="Executing migration" id="create dashboard_tag table" 2024-09-20T22:42:39.942 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.784303727Z level=info msg="Executing migration" id="add unique index dashboard_tag.dasboard_id_term" 2024-09-20T22:42:39.942 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.78497889Z level=info msg="Executing migration" id="drop index UQE_dashboard_tag_dashboard_id_term - v1" 2024-09-20T22:42:39.943 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.785873212Z level=info msg="Executing migration" id="Rename table dashboard to dashboard_v1 - v1" 2024-09-20T22:42:39.943 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.786937575Z level=info msg="Executing migration" id="create dashboard v2" 2024-09-20T22:42:39.943 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.787790765Z level=info msg="Executing migration" id="create index IDX_dashboard_org_id - v2" 2024-09-20T22:42:39.943 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.788752531Z level=info msg="Executing migration" id="create index UQE_dashboard_org_id_slug - v2" 2024-09-20T22:42:39.943 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.789633751Z level=info msg="Executing migration" id="copy dashboard v1 to v2" 2024-09-20T22:42:39.943 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.790332278Z level=info msg="Executing migration" id="drop table dashboard_v1" 2024-09-20T22:42:39.943 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.791319062Z level=info msg="Executing migration" id="alter dashboard.data to mediumtext v1" 2024-09-20T22:42:39.943 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.792061579Z level=info msg="Executing migration" id="Add column updated_by in dashboard - v2" 2024-09-20T22:42:39.943 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.792907448Z level=info msg="Executing migration" id="Add column created_by in dashboard - v2" 2024-09-20T22:42:39.943 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.793762505Z level=info msg="Executing migration" id="Add column gnetId in dashboard" 2024-09-20T22:42:39.943 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.794615094Z level=info msg="Executing migration" id="Add index for gnetId in dashboard" 2024-09-20T22:42:39.944 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.795504313Z level=info msg="Executing migration" id="Add column plugin_id in dashboard" 2024-09-20T22:42:39.944 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.798402273Z level=info msg="Executing migration" id="Add index for plugin_id in dashboard" 2024-09-20T22:42:39.944 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.801187953Z level=info msg="Executing migration" id="Add index for dashboard_id in dashboard_tag" 2024-09-20T22:42:39.944 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.80379143Z level=info msg="Executing migration" id="Update dashboard table charset" 2024-09-20T22:42:39.944 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.806560091Z level=info msg="Executing migration" id="Update dashboard_tag table charset" 2024-09-20T22:42:39.944 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.809589201Z level=info msg="Executing migration" id="Add column folder_id in dashboard" 2024-09-20T22:42:39.944 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.812289146Z level=info msg="Executing migration" id="Add column isFolder in dashboard" 2024-09-20T22:42:39.944 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.812846451Z level=info msg="Executing migration" id="Add column has_acl in dashboard" 2024-09-20T22:42:39.944 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.8156373Z level=info msg="Executing migration" id="Add column uid in dashboard" 2024-09-20T22:42:39.944 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.818519395Z level=info msg="Executing migration" id="Update uid column values in dashboard" 2024-09-20T22:42:39.944 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.821323364Z level=info msg="Executing migration" id="Add unique index dashboard_org_id_uid" 2024-09-20T22:42:39.944 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.824147378Z level=info msg="Executing migration" id="Remove unique index org_id_slug" 2024-09-20T22:42:39.944 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.826872695Z level=info msg="Executing migration" id="Update dashboard title length" 2024-09-20T22:42:39.945 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.829630393Z level=info msg="Executing migration" id="Add unique index for dashboard_org_id_title_folder_id" 2024-09-20T22:42:39.945 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.832603764Z level=info msg="Executing migration" id="create dashboard_provisioning" 2024-09-20T22:42:39.945 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.835452475Z level=info msg="Executing migration" id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" 2024-09-20T22:42:39.945 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.838217551Z level=info msg="Executing migration" id="create dashboard_provisioning v2" 2024-09-20T22:42:39.945 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.840958383Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id - v2" 2024-09-20T22:42:39.945 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.843801113Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" 2024-09-20T22:42:39.945 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.846642813Z level=info msg="Executing migration" id="copy dashboard_provisioning v1 to v2" 2024-09-20T22:42:39.945 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.849284395Z level=info msg="Executing migration" id="drop dashboard_provisioning_tmp_qwerty" 2024-09-20T22:42:39.946 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.852052151Z level=info msg="Executing migration" id="Add check_sum column" 2024-09-20T22:42:39.946 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.85493223Z level=info msg="Executing migration" id="Add index for dashboard_title" 2024-09-20T22:42:39.946 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.857374329Z level=info msg="Executing migration" id="delete tags for deleted dashboards" 2024-09-20T22:42:39.946 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.86003358Z level=info msg="Executing migration" id="delete stars for deleted dashboards" 2024-09-20T22:42:39.946 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.862707538Z level=info msg="Executing migration" id="Add index for dashboard_is_folder" 2024-09-20T22:42:39.946 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.865578269Z level=info msg="Executing migration" id="Add isPublic for dashboard" 2024-09-20T22:42:39.946 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.868106669Z level=info msg="Executing migration" id="create data_source table" 2024-09-20T22:42:39.946 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.870903279Z level=info msg="Executing migration" id="add index data_source.account_id" 2024-09-20T22:42:39.946 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.871329821Z level=info msg="Executing migration" id="add unique index data_source.account_id_name" 2024-09-20T22:42:39.946 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.87356097Z level=info msg="Executing migration" id="drop index IDX_data_source_account_id - v1" 2024-09-20T22:42:39.946 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.876345021Z level=info msg="Executing migration" id="drop index UQE_data_source_account_id_name - v1" 2024-09-20T22:42:39.946 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.879065963Z level=info msg="Executing migration" id="Rename table data_source to data_source_v1 - v1" 2024-09-20T22:42:39.947 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.882018473Z level=info msg="Executing migration" id="create data_source table v2" 2024-09-20T22:42:39.947 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.88468569Z level=info msg="Executing migration" id="create index IDX_data_source_org_id - v2" 2024-09-20T22:42:39.947 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.887362783Z level=info msg="Executing migration" id="create index UQE_data_source_org_id_name - v2" 2024-09-20T22:42:39.947 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.889961946Z level=info msg="Executing migration" id="Drop old table data_source_v1 #2" 2024-09-20T22:42:39.947 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.892782988Z level=info msg="Executing migration" id="Add column with_credentials" 2024-09-20T22:42:39.947 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.893539963Z level=info msg="Executing migration" id="Add secure json data column" 2024-09-20T22:42:39.947 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.894338387Z level=info msg="Executing migration" id="Update data_source table charset" 2024-09-20T22:42:39.947 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.895344898Z level=info msg="Executing migration" id="Update initial version to 1" 2024-09-20T22:42:39.947 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.895937348Z level=info msg="Executing migration" id="Add read_only data column" 2024-09-20T22:42:39.947 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.896776105Z level=info msg="Executing migration" id="Migrate logging ds to loki ds" 2024-09-20T22:42:39.947 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.897731533Z level=info msg="Executing migration" id="Update json_data with nulls" 2024-09-20T22:42:39.948 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.898542953Z level=info msg="Executing migration" id="Add uid column" 2024-09-20T22:42:39.948 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.899408312Z level=info msg="Executing migration" id="Update uid value" 2024-09-20T22:42:39.948 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.900246457Z level=info msg="Executing migration" id="Add unique index datasource_org_id_uid" 2024-09-20T22:42:39.948 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.901085449Z level=info msg="Executing migration" id="add unique index datasource_org_id_is_default" 2024-09-20T22:42:39.948 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.90194059Z level=info msg="Executing migration" id="create api_key table" 2024-09-20T22:42:39.949 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.902869067Z level=info msg="Executing migration" id="add index api_key.account_id" 2024-09-20T22:42:39.949 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.903867946Z level=info msg="Executing migration" id="add index api_key.key" 2024-09-20T22:42:39.949 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.904578432Z level=info msg="Executing migration" id="add index api_key.account_id_name" 2024-09-20T22:42:39.949 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.905527068Z level=info msg="Executing migration" id="drop index IDX_api_key_account_id - v1" 2024-09-20T22:42:39.949 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.906302695Z level=info msg="Executing migration" id="drop index UQE_api_key_key - v1" 2024-09-20T22:42:39.949 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.907249028Z level=info msg="Executing migration" id="drop index UQE_api_key_account_id_name - v1" 2024-09-20T22:42:39.949 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.907961708Z level=info msg="Executing migration" id="Rename table api_key to api_key_v1 - v1" 2024-09-20T22:42:39.949 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.909279694Z level=info msg="Executing migration" id="create api_key table v2" 2024-09-20T22:42:39.949 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.909717097Z level=info msg="Executing migration" id="create index IDX_api_key_org_id - v2" 2024-09-20T22:42:39.949 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.91068915Z level=info msg="Executing migration" id="create index UQE_api_key_key - v2" 2024-09-20T22:42:39.950 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.913460707Z level=info msg="Executing migration" id="create index UQE_api_key_org_id_name - v2" 2024-09-20T22:42:39.950 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.916221343Z level=info msg="Executing migration" id="copy api_key v1 to v2" 2024-09-20T22:42:39.950 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.916582004Z level=info msg="Executing migration" id="Drop old table api_key_v1" 2024-09-20T22:42:39.950 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.919388252Z level=info msg="Executing migration" id="Update api_key table charset" 2024-09-20T22:42:39.950 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.922247055Z level=info msg="Executing migration" id="Add expires to api_key table" 2024-09-20T22:42:39.950 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.925099608Z level=info msg="Executing migration" id="Add service account foreign key" 2024-09-20T22:42:39.950 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.925764575Z level=info msg="Executing migration" id="set service account foreign key to nil if 0" 2024-09-20T22:42:39.950 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.928661003Z level=info msg="Executing migration" id="Add last_used_at to api_key table" 2024-09-20T22:42:39.951 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.931432028Z level=info msg="Executing migration" id="Add is_revoked column to api_key table" 2024-09-20T22:42:39.951 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.934424496Z level=info msg="Executing migration" id="create dashboard_snapshot table v4" 2024-09-20T22:42:39.951 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.934979292Z level=info msg="Executing migration" id="drop table dashboard_snapshot_v4 #1" 2024-09-20T22:42:39.951 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.937732783Z level=info msg="Executing migration" id="create dashboard_snapshot table v5 #2" 2024-09-20T22:42:39.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:42:39.952 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:42:39.952 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:42:39.979 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:42:39.986 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:42:40.053 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:42:40.074 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:42:40.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:42:40.075 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:42:40.175 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:42:40.188 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-mon[33264]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:42:40.188 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:42:40.188 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:42:40.188 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:42:40.188 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:42:40.189 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.940459241Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_key - v5" 2024-09-20T22:42:40.189 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.943139389Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_delete_key - v5" 2024-09-20T22:42:40.189 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.943595918Z level=info msg="Executing migration" id="create index IDX_dashboard_snapshot_user_id - v5" 2024-09-20T22:42:40.189 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.946384384Z level=info msg="Executing migration" id="alter dashboard_snapshot to mediumtext v2" 2024-09-20T22:42:40.189 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.949272833Z level=info msg="Executing migration" id="Update dashboard_snapshot table charset" 2024-09-20T22:42:40.189 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.951977055Z level=info msg="Executing migration" id="Add column external_delete_url to dashboard_snapshots table" 2024-09-20T22:42:40.189 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.952630138Z level=info msg="Executing migration" id="Add encrypted dashboard json column" 2024-09-20T22:42:40.189 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.955450999Z level=info msg="Executing migration" id="Change dashboard_encrypted column to MEDIUMBLOB" 2024-09-20T22:42:40.189 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.958179251Z level=info msg="Executing migration" id="create quota table v1" 2024-09-20T22:42:40.189 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.960932663Z level=info msg="Executing migration" id="create index UQE_quota_org_id_user_id_target - v1" 2024-09-20T22:42:40.189 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.9634403Z level=info msg="Executing migration" id="Update quota table charset" 2024-09-20T22:42:40.189 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.966282597Z level=info msg="Executing migration" id="create plugin_setting table" 2024-09-20T22:42:40.190 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.968640094Z level=info msg="Executing migration" id="create index UQE_plugin_setting_org_id_plugin_id - v1" 2024-09-20T22:42:40.190 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.971469709Z level=info msg="Executing migration" id="Add column plugin_version to plugin_settings" 2024-09-20T22:42:40.190 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.972126114Z level=info msg="Executing migration" id="Update plugin_setting table charset" 2024-09-20T22:42:40.190 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.974983428Z level=info msg="Executing migration" id="create session table" 2024-09-20T22:42:40.190 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.977764144Z level=info msg="Executing migration" id="Drop old table playlist table" 2024-09-20T22:42:40.190 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.980356511Z level=info msg="Executing migration" id="Drop old table playlist_item table" 2024-09-20T22:42:40.190 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.980663498Z level=info msg="Executing migration" id="create playlist table v2" 2024-09-20T22:42:40.190 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.983514352Z level=info msg="Executing migration" id="create playlist item table v2" 2024-09-20T22:42:40.190 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.986192148Z level=info msg="Executing migration" id="Update playlist table charset" 2024-09-20T22:42:40.190 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.98899802Z level=info msg="Executing migration" id="Update playlist_item table charset" 2024-09-20T22:42:40.190 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.989279355Z level=info msg="Executing migration" id="drop preferences table v2" 2024-09-20T22:42:40.190 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.990064887Z level=info msg="Executing migration" id="drop preferences table v3" 2024-09-20T22:42:40.190 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.990390988Z level=info msg="Executing migration" id="create preferences table v3" 2024-09-20T22:42:40.191 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.990975911Z level=info msg="Executing migration" id="Update preferences table charset" 2024-09-20T22:42:40.191 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.991914726Z level=info msg="Executing migration" id="Add column team_id in preferences" 2024-09-20T22:42:40.191 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.992570622Z level=info msg="Executing migration" id="Update team_id column values in preferences" 2024-09-20T22:42:40.191 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.993432374Z level=info msg="Executing migration" id="Add column week_start in preferences" 2024-09-20T22:42:40.191 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.994344791Z level=info msg="Executing migration" id="Add column preferences.json_data" 2024-09-20T22:42:40.191 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.994972962Z level=info msg="Executing migration" id="alter preferences.json_data to mediumtext v1" 2024-09-20T22:42:40.191 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.995946306Z level=info msg="Executing migration" id="Add preferences index org_id" 2024-09-20T22:42:40.191 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.996509231Z level=info msg="Executing migration" id="Add preferences index user_id" 2024-09-20T22:42:40.191 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.997377956Z level=info msg="Executing migration" id="create alert table v1" 2024-09-20T22:42:40.191 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.998218583Z level=info msg="Executing migration" id="add index alert org_id & id " 2024-09-20T22:42:40.191 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.998983349Z level=info msg="Executing migration" id="add index alert state" 2024-09-20T22:42:40.191 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:39.999406333Z level=info msg="Executing migration" id="add index alert dashboard_id" 2024-09-20T22:42:40.191 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.000223232Z level=info msg="Executing migration" id="Create alert_rule_tag table v1" 2024-09-20T22:42:40.192 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.00117117Z level=info msg="Executing migration" id="Add unique index alert_rule_tag.alert_id_tag_id" 2024-09-20T22:42:40.192 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.002003855Z level=info msg="Executing migration" id="drop index UQE_alert_rule_tag_alert_id_tag_id - v1" 2024-09-20T22:42:40.193 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.002578741Z level=info msg="Executing migration" id="Rename table alert_rule_tag to alert_rule_tag_v1 - v1" 2024-09-20T22:42:40.193 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.004291162Z level=info msg="Executing migration" id="Create alert_rule_tag table v2" 2024-09-20T22:42:40.193 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.005152288Z level=info msg="Executing migration" id="create index UQE_alert_rule_tag_alert_id_tag_id - Add unique index alert_rule_tag.alert_id_tag_id V2" 2024-09-20T22:42:40.193 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.006145133Z level=info msg="Executing migration" id="copy alert_rule_tag v1 to v2" 2024-09-20T22:42:40.193 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.006485218Z level=info msg="Executing migration" id="drop table alert_rule_tag_v1" 2024-09-20T22:42:40.193 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.00741266Z level=info msg="Executing migration" id="create alert_notification table v1" 2024-09-20T22:42:40.193 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.008041275Z level=info msg="Executing migration" id="Add column is_default" 2024-09-20T22:42:40.193 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.008765159Z level=info msg="Executing migration" id="Add column frequency" 2024-09-20T22:42:40.193 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.009663509Z level=info msg="Executing migration" id="Add column send_reminder" 2024-09-20T22:42:40.193 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.010418966Z level=info msg="Executing migration" id="Add column disable_resolve_message" 2024-09-20T22:42:40.193 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.011282139Z level=info msg="Executing migration" id="add index alert_notification org_id & name" 2024-09-20T22:42:40.193 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.012133454Z level=info msg="Executing migration" id="Update alert table charset" 2024-09-20T22:42:40.194 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.012956354Z level=info msg="Executing migration" id="Update alert_notification table charset" 2024-09-20T22:42:40.194 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.013270215Z level=info msg="Executing migration" id="create notification_journal table v1" 2024-09-20T22:42:40.194 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.014128999Z level=info msg="Executing migration" id="add index notification_journal org_id & alert_id & notifier_id" 2024-09-20T22:42:40.194 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.014954036Z level=info msg="Executing migration" id="drop alert_notification_journal" 2024-09-20T22:42:40.194 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.015964521Z level=info msg="Executing migration" id="create alert_notification_state table v1" 2024-09-20T22:42:40.194 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.016488331Z level=info msg="Executing migration" id="add index alert_notification_state org_id & alert_id & notifier_id" 2024-09-20T22:42:40.194 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.019234247Z level=info msg="Executing migration" id="Add for to alert table" 2024-09-20T22:42:40.194 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.020177147Z level=info msg="Executing migration" id="Add column uid in alert_notification" 2024-09-20T22:42:40.194 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.022147125Z level=info msg="Executing migration" id="Update uid column values in alert_notification" 2024-09-20T22:42:40.194 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.024910278Z level=info msg="Executing migration" id="Add unique index alert_notification_org_id_uid" 2024-09-20T22:42:40.194 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.025342676Z level=info msg="Executing migration" id="Remove unique index org_id_name" 2024-09-20T22:42:40.194 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.028170627Z level=info msg="Executing migration" id="Add column secure_settings in alert_notification" 2024-09-20T22:42:40.194 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.029150241Z level=info msg="Executing migration" id="alter alert.settings to mediumtext" 2024-09-20T22:42:40.195 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.031060542Z level=info msg="Executing migration" id="Add non-unique index alert_notification_state_alert_id" 2024-09-20T22:42:40.195 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.033585768Z level=info msg="Executing migration" id="Add non-unique index alert_rule_tag_alert_id" 2024-09-20T22:42:40.195 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.034034489Z level=info msg="Executing migration" id="Drop old annotation table v4" 2024-09-20T22:42:40.195 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.036912506Z level=info msg="Executing migration" id="create annotation table v5" 2024-09-20T22:42:40.195 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.039780508Z level=info msg="Executing migration" id="add index annotation 0 v3" 2024-09-20T22:42:40.195 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.042474585Z level=info msg="Executing migration" id="add index annotation 1 v3" 2024-09-20T22:42:40.195 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.042936738Z level=info msg="Executing migration" id="add index annotation 2 v3" 2024-09-20T22:42:40.195 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.045680292Z level=info msg="Executing migration" id="add index annotation 3 v3" 2024-09-20T22:42:40.195 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.048517183Z level=info msg="Executing migration" id="add index annotation 4 v3" 2024-09-20T22:42:40.195 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.051359213Z level=info msg="Executing migration" id="Update annotation table charset" 2024-09-20T22:42:40.195 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.051762657Z level=info msg="Executing migration" id="Add column region_id to annotation table" 2024-09-20T22:42:40.195 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.054564757Z level=info msg="Executing migration" id="Drop category_id index" 2024-09-20T22:42:40.195 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.057222541Z level=info msg="Executing migration" id="Add column tags to annotation table" 2024-09-20T22:42:40.195 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.059959569Z level=info msg="Executing migration" id="Create annotation_tag table v2" 2024-09-20T22:42:40.196 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.060525269Z level=info msg="Executing migration" id="Add unique index annotation_tag.annotation_id_tag_id" 2024-09-20T22:42:40.196 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.06331365Z level=info msg="Executing migration" id="drop index UQE_annotation_tag_annotation_id_tag_id - v2" 2024-09-20T22:42:40.196 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.066009306Z level=info msg="Executing migration" id="Rename table annotation_tag to annotation_tag_v2 - v2" 2024-09-20T22:42:40.196 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.068676963Z level=info msg="Executing migration" id="Create annotation_tag table v3" 2024-09-20T22:42:40.196 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.069264494Z level=info msg="Executing migration" id="create index UQE_annotation_tag_annotation_id_tag_id - Add unique index annotation_tag.annotation_id_tag_id V3" 2024-09-20T22:42:40.196 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.07193148Z level=info msg="Executing migration" id="copy annotation_tag v2 to v3" 2024-09-20T22:42:40.196 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.074754048Z level=info msg="Executing migration" id="drop table annotation_tag_v2" 2024-09-20T22:42:40.196 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.077404201Z level=info msg="Executing migration" id="Update alert annotations and set TEXT to empty" 2024-09-20T22:42:40.196 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.077749563Z level=info msg="Executing migration" id="Add created time to annotation table" 2024-09-20T22:42:40.196 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.080616962Z level=info msg="Executing migration" id="Add updated time to annotation table" 2024-09-20T22:42:40.196 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.083137258Z level=info msg="Executing migration" id="Add index for created in annotation table" 2024-09-20T22:42:40.196 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.085973352Z level=info msg="Executing migration" id="Add index for updated in annotation table" 2024-09-20T22:42:40.196 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.086507673Z level=info msg="Executing migration" id="Convert existing annotations from seconds to milliseconds" 2024-09-20T22:42:40.196 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.089231228Z level=info msg="Executing migration" id="Add epoch_end column" 2024-09-20T22:42:40.197 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.090354557Z level=info msg="Executing migration" id="Add index for epoch_end" 2024-09-20T22:42:40.197 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.09080991Z level=info msg="Executing migration" id="Make epoch_end the same as epoch" 2024-09-20T22:42:40.197 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.091622503Z level=info msg="Executing migration" id="Move region to single row" 2024-09-20T22:42:40.197 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.092557622Z level=info msg="Executing migration" id="Remove index org_id_epoch from annotation table" 2024-09-20T22:42:40.197 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.09324931Z level=info msg="Executing migration" id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" 2024-09-20T22:42:40.197 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.09409553Z level=info msg="Executing migration" id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" 2024-09-20T22:42:40.197 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.094923267Z level=info msg="Executing migration" id="Add index for org_id_epoch_end_epoch on annotation table" 2024-09-20T22:42:40.197 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.095412073Z level=info msg="Executing migration" id="Remove index org_id_epoch_epoch_end from annotation table" 2024-09-20T22:42:40.197 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.096233816Z level=info msg="Executing migration" id="Add index for alert_id on annotation table" 2024-09-20T22:42:40.197 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.096683652Z level=info msg="Executing migration" id="Increase tags column to length 4096" 2024-09-20T22:42:40.197 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.097116028Z level=info msg="Executing migration" id="create test_data table" 2024-09-20T22:42:40.197 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.097525401Z level=info msg="Executing migration" id="create dashboard_version table v1" 2024-09-20T22:42:40.197 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.097975169Z level=info msg="Executing migration" id="add index dashboard_version.dashboard_id" 2024-09-20T22:42:40.197 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.098851902Z level=info msg="Executing migration" id="add unique index dashboard_version.dashboard_id and dashboard_version.version" 2024-09-20T22:42:40.197 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.099749196Z level=info msg="Executing migration" id="Set dashboard version to 1 where 0" 2024-09-20T22:42:40.198 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.100535736Z level=info msg="Executing migration" id="save existing dashboard data in dashboard_version table v1" 2024-09-20T22:42:40.198 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.101492373Z level=info msg="Executing migration" id="alter dashboard_version.data to mediumtext v1" 2024-09-20T22:42:40.198 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.102358705Z level=info msg="Executing migration" id="create team table" 2024-09-20T22:42:40.198 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.10315141Z level=info msg="Executing migration" id="add index team.org_id" 2024-09-20T22:42:40.198 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.10399392Z level=info msg="Executing migration" id="add unique index team_org_id_name" 2024-09-20T22:42:40.198 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.104861245Z level=info msg="Executing migration" id="create team member table" 2024-09-20T22:42:40.198 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.105757831Z level=info msg="Executing migration" id="add index team_member.org_id" 2024-09-20T22:42:40.198 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.106556587Z level=info msg="Executing migration" id="add unique index team_member_org_id_team_id_user_id" 2024-09-20T22:42:40.198 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.107424798Z level=info msg="Executing migration" id="add index team_member.team_id" 2024-09-20T22:42:40.198 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.108391904Z level=info msg="Executing migration" id="Add column email to team table" 2024-09-20T22:42:40.198 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.109405653Z level=info msg="Executing migration" id="Add column external to team_member table" 2024-09-20T22:42:40.198 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.110295511Z level=info msg="Executing migration" id="Add column permission to team_member table" 2024-09-20T22:42:40.198 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.111157932Z level=info msg="Executing migration" id="create dashboard acl table" 2024-09-20T22:42:40.198 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.112012269Z level=info msg="Executing migration" id="add index dashboard_acl_dashboard_id" 2024-09-20T22:42:40.199 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.114756581Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_user_id" 2024-09-20T22:42:40.199 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.117622838Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_team_id" 2024-09-20T22:42:40.199 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.118104242Z level=info msg="Executing migration" id="add index dashboard_acl_user_id" 2024-09-20T22:42:40.199 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.120878293Z level=info msg="Executing migration" id="add index dashboard_acl_team_id" 2024-09-20T22:42:40.199 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.123739018Z level=info msg="Executing migration" id="add index dashboard_acl_org_id_role" 2024-09-20T22:42:40.199 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.126454985Z level=info msg="Executing migration" id="add index dashboard_permission" 2024-09-20T22:42:40.199 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.126948189Z level=info msg="Executing migration" id="save default acl rules in dashboard_acl table" 2024-09-20T22:42:40.199 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.129902984Z level=info msg="Executing migration" id="delete acl rules for deleted dashboards and folders" 2024-09-20T22:42:40.199 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.132606822Z level=info msg="Executing migration" id="create tag table" 2024-09-20T22:42:40.199 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.135456771Z level=info msg="Executing migration" id="add index tag.key_value" 2024-09-20T22:42:40.199 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.135980266Z level=info msg="Executing migration" id="create login attempt table" 2024-09-20T22:42:40.199 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.138634537Z level=info msg="Executing migration" id="add index login_attempt.username" 2024-09-20T22:42:40.199 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.141578689Z level=info msg="Executing migration" id="drop index IDX_login_attempt_username - v1" 2024-09-20T22:42:40.199 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.144336658Z level=info msg="Executing migration" id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" 2024-09-20T22:42:40.200 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.147293445Z level=info msg="Executing migration" id="create login_attempt v2" 2024-09-20T22:42:40.200 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.150007119Z level=info msg="Executing migration" id="create index IDX_login_attempt_username - v2" 2024-09-20T22:42:40.200 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.152779969Z level=info msg="Executing migration" id="copy login_attempt v1 to v2" 2024-09-20T22:42:40.200 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.155479186Z level=info msg="Executing migration" id="drop login_attempt_tmp_qwerty" 2024-09-20T22:42:40.200 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.158286865Z level=info msg="Executing migration" id="create user auth table" 2024-09-20T22:42:40.200 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.161025769Z level=info msg="Executing migration" id="create index IDX_user_auth_auth_module_auth_id - v1" 2024-09-20T22:42:40.200 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.163861958Z level=info msg="Executing migration" id="alter user_auth.auth_id to length 190" 2024-09-20T22:42:40.200 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.16415952Z level=info msg="Executing migration" id="Add OAuth access token to user_auth" 2024-09-20T22:42:40.200 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.166889892Z level=info msg="Executing migration" id="Add OAuth refresh token to user_auth" 2024-09-20T22:42:40.200 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.169604452Z level=info msg="Executing migration" id="Add OAuth token type to user_auth" 2024-09-20T22:42:40.200 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.172207955Z level=info msg="Executing migration" id="Add OAuth expiry to user_auth" 2024-09-20T22:42:40.200 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.173155867Z level=info msg="Executing migration" id="Add index to user_id column in user_auth" 2024-09-20T22:42:40.200 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.175927328Z level=info msg="Executing migration" id="Add OAuth ID token to user_auth" 2024-09-20T22:42:40.200 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.178582817Z level=info msg="Executing migration" id="create server_lock table" 2024-09-20T22:42:40.200 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.181255521Z level=info msg="Executing migration" id="add index server_lock.operation_uid" 2024-09-20T22:42:40.201 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.181750527Z level=info msg="Executing migration" id="create user auth token table" 2024-09-20T22:42:40.201 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.184496215Z level=info msg="Executing migration" id="add unique index user_auth_token.auth_token" 2024-09-20T22:42:40.201 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.187235367Z level=info msg="Executing migration" id="add unique index user_auth_token.prev_auth_token" 2024-09-20T22:42:40.201 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.189992222Z level=info msg="Executing migration" id="add index user_auth_token.user_id" 2024-09-20T22:42:40.201 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.190484131Z level=info msg="Executing migration" id="Add revoked_at to the user auth token" 2024-09-20T22:42:40.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:42:40.202 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:42:40.202 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:42:40.303 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:42:40.324 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:42:40.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:42:40.325 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:42:40.364 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:40 smithi084 ceph-mon[29774]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:42:40.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:40 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:42:40.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:40 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:42:40.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:40 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:42:40.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:40 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:42:40.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:40 smithi084 ceph-mon[40024]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:42:40.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:40 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:42:40.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:40 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:42:40.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:40 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:42:40.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:40 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:42:40.426 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:42:40.446 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.193182621Z level=info msg="Executing migration" id="create cache_data table" 2024-09-20T22:42:40.446 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.195893438Z level=info msg="Executing migration" id="add unique index cache_data.cache_key" 2024-09-20T22:42:40.446 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.196688825Z level=info msg="Executing migration" id="create short_url table v1" 2024-09-20T22:42:40.446 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.197265573Z level=info msg="Executing migration" id="add index short_url.org_id-uid" 2024-09-20T22:42:40.446 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.198051098Z level=info msg="Executing migration" id="alter table short_url alter column created_by type to bigint" 2024-09-20T22:42:40.446 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.198843768Z level=info msg="Executing migration" id="delete alert_definition table" 2024-09-20T22:42:40.446 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.199249477Z level=info msg="Executing migration" id="recreate alert_definition table" 2024-09-20T22:42:40.446 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.199755605Z level=info msg="Executing migration" id="add index in alert_definition on org_id and title columns" 2024-09-20T22:42:40.446 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.200540046Z level=info msg="Executing migration" id="add index in alert_definition on org_id and uid columns" 2024-09-20T22:42:40.446 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.201095783Z level=info msg="Executing migration" id="alter alert_definition table data column to mediumtext in mysql" 2024-09-20T22:42:40.446 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.201391707Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and title columns" 2024-09-20T22:42:40.446 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.201800186Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and uid columns" 2024-09-20T22:42:40.447 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.202313454Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and title columns" 2024-09-20T22:42:40.447 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.202743773Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and uid columns" 2024-09-20T22:42:40.447 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.203204066Z level=info msg="Executing migration" id="Add column paused in alert_definition" 2024-09-20T22:42:40.447 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.204110507Z level=info msg="Executing migration" id="drop alert_definition table" 2024-09-20T22:42:40.447 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.20455205Z level=info msg="Executing migration" id="delete alert_definition_version table" 2024-09-20T22:42:40.447 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.204832569Z level=info msg="Executing migration" id="recreate alert_definition_version table" 2024-09-20T22:42:40.447 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.205242661Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_id and version columns" 2024-09-20T22:42:40.447 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.205656861Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_uid and version columns" 2024-09-20T22:42:40.447 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.206102459Z level=info msg="Executing migration" id="alter alert_definition_version table data column to mediumtext in mysql" 2024-09-20T22:42:40.448 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.206393185Z level=info msg="Executing migration" id="drop alert_definition_version table" 2024-09-20T22:42:40.448 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.206855119Z level=info msg="Executing migration" id="create alert_instance table" 2024-09-20T22:42:40.448 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.207277173Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, def_uid and current_state columns" 2024-09-20T22:42:40.448 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.207696728Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, current_state columns" 2024-09-20T22:42:40.448 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.208153973Z level=info msg="Executing migration" id="add column current_state_end to alert_instance" 2024-09-20T22:42:40.448 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.209049625Z level=info msg="Executing migration" id="remove index def_org_id, def_uid, current_state on alert_instance" 2024-09-20T22:42:40.448 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.209466195Z level=info msg="Executing migration" id="remove index def_org_id, current_state on alert_instance" 2024-09-20T22:42:40.448 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.20987949Z level=info msg="Executing migration" id="rename def_org_id to rule_org_id in alert_instance" 2024-09-20T22:42:40.448 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.214274494Z level=info msg="Executing migration" id="rename def_uid to rule_uid in alert_instance" 2024-09-20T22:42:40.448 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.218636721Z level=info msg="Executing migration" id="add index rule_org_id, rule_uid, current_state on alert_instance" 2024-09-20T22:42:40.448 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.219125506Z level=info msg="Executing migration" id="add index rule_org_id, current_state on alert_instance" 2024-09-20T22:42:40.448 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.219597301Z level=info msg="Executing migration" id="add current_reason column related to current_state" 2024-09-20T22:42:40.449 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.220581684Z level=info msg="Executing migration" id="create alert_rule table" 2024-09-20T22:42:40.449 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.221061339Z level=info msg="Executing migration" id="add index in alert_rule on org_id and title columns" 2024-09-20T22:42:40.449 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:42:40.449 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.221498966Z level=info msg="Executing migration" id="add index in alert_rule on org_id and uid columns" 2024-09-20T22:42:40.449 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.221931537Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespace_uid, group_uid columns" 2024-09-20T22:42:40.449 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.222368174Z level=info msg="Executing migration" id="alter alert_rule table data column to mediumtext in mysql" 2024-09-20T22:42:40.449 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.222632737Z level=info msg="Executing migration" id="add column for to alert_rule" 2024-09-20T22:42:40.449 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.223609414Z level=info msg="Executing migration" id="add column annotations to alert_rule" 2024-09-20T22:42:40.450 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.22455387Z level=info msg="Executing migration" id="add column labels to alert_rule" 2024-09-20T22:42:40.450 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.225488787Z level=info msg="Executing migration" id="remove unique index from alert_rule on org_id, title columns" 2024-09-20T22:42:40.450 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.225908176Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespase_uid and title columns" 2024-09-20T22:42:40.450 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.22632904Z level=info msg="Executing migration" id="add dashboard_uid column to alert_rule" 2024-09-20T22:42:40.450 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.22730939Z level=info msg="Executing migration" id="add panel_id column to alert_rule" 2024-09-20T22:42:40.450 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.228222532Z level=info msg="Executing migration" id="add index in alert_rule on org_id, dashboard_uid and panel_id columns" 2024-09-20T22:42:40.450 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.22864518Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule" 2024-09-20T22:42:40.450 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.229569024Z level=info msg="Executing migration" id="add is_paused column to alert_rule table" 2024-09-20T22:42:40.450 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.230515093Z level=info msg="Executing migration" id="fix is_paused column for alert_rule table" 2024-09-20T22:42:40.450 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.230827936Z level=info msg="Executing migration" id="create alert_rule_version table" 2024-09-20T22:42:40.450 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.231308506Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_uid and version columns" 2024-09-20T22:42:40.450 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.23178173Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_namespace_uid and rule_group columns" 2024-09-20T22:42:40.450 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.232240131Z level=info msg="Executing migration" id="alter alert_rule_version table data column to mediumtext in mysql" 2024-09-20T22:42:40.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:42:40.451 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:42:40.452 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.232529146Z level=info msg="Executing migration" id="add column for to alert_rule_version" 2024-09-20T22:42:40.453 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.233530906Z level=info msg="Executing migration" id="add column annotations to alert_rule_version" 2024-09-20T22:42:40.453 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.234509507Z level=info msg="Executing migration" id="add column labels to alert_rule_version" 2024-09-20T22:42:40.453 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.235471247Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule_version" 2024-09-20T22:42:40.453 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.236443594Z level=info msg="Executing migration" id="add is_paused column to alert_rule_versions table" 2024-09-20T22:42:40.453 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.237468027Z level=info msg="Executing migration" id="fix is_paused column for alert_rule_version table" 2024-09-20T22:42:40.453 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.238333899Z level=info msg="Executing migration" id=create_alert_configuration_table 2024-09-20T22:42:40.453 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.238778806Z level=info msg="Executing migration" id="Add column default in alert_configuration" 2024-09-20T22:42:40.453 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.239723537Z level=info msg="Executing migration" id="alert alert_configuration alertmanager_configuration column from TEXT to MEDIUMTEXT if mysql" 2024-09-20T22:42:40.453 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.240599753Z level=info msg="Executing migration" id="add column org_id in alert_configuration" 2024-09-20T22:42:40.453 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.241631807Z level=info msg="Executing migration" id="add index in alert_configuration table on org_id column" 2024-09-20T22:42:40.453 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.242111792Z level=info msg="Executing migration" id="add configuration_hash column to alert_configuration" 2024-09-20T22:42:40.453 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.243106356Z level=info msg="Executing migration" id=create_ngalert_configuration_table 2024-09-20T22:42:40.453 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.243947525Z level=info msg="Executing migration" id="add index in ngalert_configuration on org_id column" 2024-09-20T22:42:40.454 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.244792678Z level=info msg="Executing migration" id="add column send_alerts_to in ngalert_configuration" 2024-09-20T22:42:40.454 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.245807114Z level=info msg="Executing migration" id="create provenance_type table" 2024-09-20T22:42:40.454 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.24676942Z level=info msg="Executing migration" id="add index to uniquify (record_key, record_type, org_id) columns" 2024-09-20T22:42:40.454 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.247520327Z level=info msg="Executing migration" id="create alert_image table" 2024-09-20T22:42:40.454 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.248519532Z level=info msg="Executing migration" id="add unique index on token to alert_image table" 2024-09-20T22:42:40.454 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.249086993Z level=info msg="Executing migration" id="support longer URLs in alert_image table" 2024-09-20T22:42:40.454 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.251890462Z level=info msg="Executing migration" id=create_alert_configuration_history_table 2024-09-20T22:42:40.454 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.254576515Z level=info msg="Executing migration" id="drop non-unique orgID index on alert_configuration" 2024-09-20T22:42:40.454 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.257226646Z level=info msg="Executing migration" id="drop unique orgID index on alert_configuration if exists" 2024-09-20T22:42:40.454 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.257334567Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop unique orgID index on alert_configuration if exists" 2024-09-20T22:42:40.454 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.25768147Z level=info msg="Executing migration" id="extract alertmanager configuration history to separate table" 2024-09-20T22:42:40.454 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.258489135Z level=info msg="Executing migration" id="add unique index on orgID to alert_configuration" 2024-09-20T22:42:40.454 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.259284274Z level=info msg="Executing migration" id="clear migration entry \"remove unified alerting data\"" 2024-09-20T22:42:40.454 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.259418893Z level=info msg="Executing migration" id="move dashboard alerts to unified alerting" 2024-09-20T22:42:40.455 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.259519062Z level=info msg="alerts found to migrate" alerts=0 2024-09-20T22:42:40.455 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.260092968Z level=info msg="Executing migration" id="create library_element table v1" 2024-09-20T22:42:40.455 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.260922896Z level=info msg="Executing migration" id="add index library_element org_id-folder_id-name-kind" 2024-09-20T22:42:40.455 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.261852961Z level=info msg="Executing migration" id="create library_element_connection table v1" 2024-09-20T22:42:40.455 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.262558193Z level=info msg="Executing migration" id="add index library_element_connection element_id-kind-connection_id" 2024-09-20T22:42:40.455 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.263563511Z level=info msg="Executing migration" id="add unique index library_element org_id_uid" 2024-09-20T22:42:40.455 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.264246676Z level=info msg="Executing migration" id="increase max description length to 2048" 2024-09-20T22:42:40.455 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.265214483Z level=info msg="Executing migration" id="clone move dashboard alerts to unified alerting" 2024-09-20T22:42:40.455 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.265948871Z level=info msg="Executing migration" id="create data_keys table" 2024-09-20T22:42:40.455 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.266899628Z level=info msg="Executing migration" id="create secrets table" 2024-09-20T22:42:40.455 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.26763606Z level=info msg="Executing migration" id="rename data_keys name column to id" 2024-09-20T22:42:40.455 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.273139445Z level=info msg="Executing migration" id="add name column into data_keys" 2024-09-20T22:42:40.455 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.274184343Z level=info msg="Executing migration" id="copy data_keys id column values into name" 2024-09-20T22:42:40.455 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.275048377Z level=info msg="Executing migration" id="rename data_keys name column to label" 2024-09-20T22:42:40.456 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.28061504Z level=info msg="Executing migration" id="rename data_keys id column back to name" 2024-09-20T22:42:40.456 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.286198356Z level=info msg="Executing migration" id="create kv_store table v1" 2024-09-20T22:42:40.456 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.286979936Z level=info msg="Executing migration" id="add index kv_store.org_id-namespace-key" 2024-09-20T22:42:40.456 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.287980981Z level=info msg="Executing migration" id="update dashboard_uid and panel_id from existing annotations" 2024-09-20T22:42:40.456 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.288706259Z level=info msg="Executing migration" id="create permission table" 2024-09-20T22:42:40.456 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.289671815Z level=info msg="Executing migration" id="add unique index permission.role_id" 2024-09-20T22:42:40.456 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.290406605Z level=info msg="Executing migration" id="add unique index role_id_action_scope" 2024-09-20T22:42:40.456 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.291396606Z level=info msg="Executing migration" id="create role table" 2024-09-20T22:42:40.456 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.292165233Z level=info msg="Executing migration" id="add column display_name" 2024-09-20T22:42:40.456 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.293318232Z level=info msg="Executing migration" id="add column group_name" 2024-09-20T22:42:40.456 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.294400744Z level=info msg="Executing migration" id="add index role.org_id" 2024-09-20T22:42:40.456 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.2953812Z level=info msg="Executing migration" id="add unique index role_org_id_name" 2024-09-20T22:42:40.456 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.296113852Z level=info msg="Executing migration" id="add index role_org_id_uid" 2024-09-20T22:42:40.456 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.296955191Z level=info msg="Executing migration" id="create team role table" 2024-09-20T22:42:40.456 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.297914507Z level=info msg="Executing migration" id="add index team_role.org_id" 2024-09-20T22:42:40.456 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.29864758Z level=info msg="Executing migration" id="add unique index team_role_org_id_team_id_role_id" 2024-09-20T22:42:40.457 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.29950152Z level=info msg="Executing migration" id="add index team_role.team_id" 2024-09-20T22:42:40.457 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.300351575Z level=info msg="Executing migration" id="create user role table" 2024-09-20T22:42:40.457 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.301355611Z level=info msg="Executing migration" id="add index user_role.org_id" 2024-09-20T22:42:40.457 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.302053156Z level=info msg="Executing migration" id="add unique index user_role_org_id_user_id_role_id" 2024-09-20T22:42:40.457 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.305006947Z level=info msg="Executing migration" id="add index user_role.user_id" 2024-09-20T22:42:40.457 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.307705273Z level=info msg="Executing migration" id="create builtin role table" 2024-09-20T22:42:40.457 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.310615204Z level=info msg="Executing migration" id="add index builtin_role.role_id" 2024-09-20T22:42:40.457 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.313415388Z level=info msg="Executing migration" id="add index builtin_role.name" 2024-09-20T22:42:40.457 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.316049003Z level=info msg="Executing migration" id="Add column org_id to builtin_role table" 2024-09-20T22:42:40.457 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.318866797Z level=info msg="Executing migration" id="add index builtin_role.org_id" 2024-09-20T22:42:40.457 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.321742627Z level=info msg="Executing migration" id="add unique index builtin_role_org_id_role_id_role" 2024-09-20T22:42:40.457 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.324647118Z level=info msg="Executing migration" id="Remove unique index role_org_id_uid" 2024-09-20T22:42:40.457 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.327500858Z level=info msg="Executing migration" id="add unique index role.uid" 2024-09-20T22:42:40.457 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.330232812Z level=info msg="Executing migration" id="create seed assignment table" 2024-09-20T22:42:40.458 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.333050421Z level=info msg="Executing migration" id="add unique index builtin_role_role_name" 2024-09-20T22:42:40.458 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.335702325Z level=info msg="Executing migration" id="add column hidden to role table" 2024-09-20T22:42:40.458 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.338559082Z level=info msg="Executing migration" id="create query_history table v1" 2024-09-20T22:42:40.458 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.341252622Z level=info msg="Executing migration" id="add index query_history.org_id-created_by-datasource_uid" 2024-09-20T22:42:40.458 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.344195513Z level=info msg="Executing migration" id="alter table query_history alter column created_by type to bigint" 2024-09-20T22:42:40.458 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.346966554Z level=info msg="Executing migration" id="teams permissions migration" 2024-09-20T22:42:40.458 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.349765474Z level=info msg="Executing migration" id="dashboard permissions" 2024-09-20T22:42:40.458 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.352468893Z level=info msg="Executing migration" id="dashboard permissions uid scopes" 2024-09-20T22:42:40.458 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.355120245Z level=info msg="Executing migration" id="drop managed folder create actions" 2024-09-20T22:42:40.458 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.357888811Z level=info msg="Executing migration" id="alerting notification permissions" 2024-09-20T22:42:40.458 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.360609239Z level=info msg="Executing migration" id="create query_history_star table v1" 2024-09-20T22:42:40.458 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.363434417Z level=info msg="Executing migration" id="add index query_history.user_id-query_uid" 2024-09-20T22:42:40.458 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.366090074Z level=info msg="Executing migration" id="add column org_id in query_history_star" 2024-09-20T22:42:40.458 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.368677846Z level=info msg="Executing migration" id="alter table query_history_star_mig column user_id type to bigint" 2024-09-20T22:42:40.459 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.371386066Z level=info msg="Executing migration" id="create correlation table v1" 2024-09-20T22:42:40.459 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.374092934Z level=info msg="Executing migration" id="add index correlations.uid" 2024-09-20T22:42:40.459 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.376925563Z level=info msg="Executing migration" id="add index correlations.source_uid" 2024-09-20T22:42:40.459 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.379555275Z level=info msg="Executing migration" id="add correlation config column" 2024-09-20T22:42:40.459 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.382244048Z level=info msg="Executing migration" id="create entity_events table" 2024-09-20T22:42:40.459 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.384906933Z level=info msg="Executing migration" id="create dashboard public config v1" 2024-09-20T22:42:40.459 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.387657232Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v1" 2024-09-20T22:42:40.459 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.387752021Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index UQE_dashboard_public_config_uid - v1" 2024-09-20T22:42:40.459 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.390459564Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-09-20T22:42:40.459 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.390627691Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-09-20T22:42:40.459 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.393078722Z level=info msg="Executing migration" id="Drop old dashboard public config table" 2024-09-20T22:42:40.459 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.395736016Z level=info msg="Executing migration" id="recreate dashboard public config v1" 2024-09-20T22:42:40.459 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.398425033Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v1" 2024-09-20T22:42:40.459 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.399233205Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-09-20T22:42:40.459 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.400011458Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v2" 2024-09-20T22:42:40.460 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.400917797Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-09-20T22:42:40.460 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.401600831Z level=info msg="Executing migration" id="Drop public config table" 2024-09-20T22:42:40.460 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.403023225Z level=info msg="Executing migration" id="Recreate dashboard public config v2" 2024-09-20T22:42:40.460 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.403689338Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v2" 2024-09-20T22:42:40.460 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.404527464Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-09-20T22:42:40.460 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.405323731Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_access_token - v2" 2024-09-20T22:42:40.460 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.406153425Z level=info msg="Executing migration" id="Rename table dashboard_public_config to dashboard_public - v2" 2024-09-20T22:42:40.460 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.410033631Z level=info msg="Executing migration" id="add annotations_enabled column" 2024-09-20T22:42:40.460 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.411223164Z level=info msg="Executing migration" id="add time_selection_enabled column" 2024-09-20T22:42:40.460 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.412408132Z level=info msg="Executing migration" id="delete orphaned public dashboards" 2024-09-20T22:42:40.460 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.413343292Z level=info msg="Executing migration" id="add share column" 2024-09-20T22:42:40.460 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.414613014Z level=info msg="Executing migration" id="create default alerting folders" 2024-09-20T22:42:40.460 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.4155418Z level=info msg="Executing migration" id="create file table" 2024-09-20T22:42:40.460 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.41625667Z level=info msg="Executing migration" id="file table idx: path natural pk" 2024-09-20T22:42:40.461 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.417034444Z level=info msg="Executing migration" id="file table idx: parent_folder_path_hash fast folder retrieval" 2024-09-20T22:42:40.461 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.417947009Z level=info msg="Executing migration" id="create file_meta table" 2024-09-20T22:42:40.461 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.418767166Z level=info msg="Executing migration" id="file table idx: path key" 2024-09-20T22:42:40.461 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.419730629Z level=info msg="Executing migration" id="set path collation in file table" 2024-09-20T22:42:40.461 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.420610461Z level=info msg="Executing migration" id="managed permissions migration" 2024-09-20T22:42:40.461 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.42135992Z level=info msg="Executing migration" id="managed folder permissions alert actions migration" 2024-09-20T22:42:40.461 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.422234074Z level=info msg="Executing migration" id="RBAC action name migrator" 2024-09-20T22:42:40.461 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.422933965Z level=info msg="Executing migration" id="Add UID column to playlist" 2024-09-20T22:42:40.461 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.424112645Z level=info msg="Executing migration" id="Update uid column values in playlist" 2024-09-20T22:42:40.461 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.425147152Z level=info msg="Executing migration" id="Add index for uid in playlist" 2024-09-20T22:42:40.461 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.425922794Z level=info msg="Executing migration" id="update group index for alert rules" 2024-09-20T22:42:40.461 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.42672892Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated migration" 2024-09-20T22:42:40.461 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.427589419Z level=info msg="Executing migration" id="admin only folder/dashboard permission" 2024-09-20T22:42:40.461 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.428463455Z level=info msg="Executing migration" id="add action column to seed_assignment" 2024-09-20T22:42:40.461 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.429643437Z level=info msg="Executing migration" id="add scope column to seed_assignment" 2024-09-20T22:42:40.462 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.43084764Z level=info msg="Executing migration" id="remove unique index builtin_role_role_name before nullable update" 2024-09-20T22:42:40.462 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.431656038Z level=info msg="Executing migration" id="update seed_assignment role_name column to nullable" 2024-09-20T22:42:40.462 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.444615709Z level=info msg="Executing migration" id="add unique index builtin_role_name back" 2024-09-20T22:42:40.462 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.445444647Z level=info msg="Executing migration" id="add unique index builtin_role_action_scope" 2024-09-20T22:42:40.462 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.448294102Z level=info msg="Executing migration" id="add primary key to seed_assigment" 2024-09-20T22:42:40.552 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:42:40.573 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:42:40.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:42:40.574 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:42:40.675 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:42:40.696 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:42:40.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:42:40.696 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:42:40.754 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.452890243Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated fixed migration" 2024-09-20T22:42:40.755 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.455504508Z level=info msg="Executing migration" id="migrate external alertmanagers to datsourcse" 2024-09-20T22:42:40.755 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.458244379Z level=info msg="Executing migration" id="create folder table" 2024-09-20T22:42:40.755 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.46093833Z level=info msg="Executing migration" id="Add index for parent_uid" 2024-09-20T22:42:40.755 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.463717915Z level=info msg="Executing migration" id="Add unique index for folder.uid and folder.org_id" 2024-09-20T22:42:40.755 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.466514869Z level=info msg="Executing migration" id="Update folder title length" 2024-09-20T22:42:40.755 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.469458056Z level=info msg="Executing migration" id="Add unique index for folder.title and folder.parent_uid" 2024-09-20T22:42:40.755 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=migrator t=2024-09-20T22:42:40.472247533Z level=info msg="migrations completed" performed=485 skipped=0 duration=715.443653ms 2024-09-20T22:42:40.755 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=sqlstore t=2024-09-20T22:42:40.472658583Z level=info msg="Created default organization" 2024-09-20T22:42:40.755 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=plugin.finder t=2024-09-20T22:42:40.506517265Z level=warn msg="Skipping finding plugins as directory does not exist" path=/usr/share/grafana/plugins-bundled 2024-09-20T22:42:40.755 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=secrets t=2024-09-20T22:42:40.506620622Z level=info msg="Envelope encryption state" enabled=true currentprovider=secretKey.v1 2024-09-20T22:42:40.755 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=query_data t=2024-09-20T22:42:40.508770081Z level=info msg="Query Service initialization" 2024-09-20T22:42:40.755 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=live.push_http t=2024-09-20T22:42:40.511246357Z level=info msg="Live Push Gateway initialization" 2024-09-20T22:42:40.797 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:42:40.820 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:42:40.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:42:40.821 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:42:40.897 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-20T22:42:40.899 INFO:teuthology.orchestra.run.smithi084.stderr:dumped all 2024-09-20T22:42:40.921 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:42:40.942 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:42:40.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:42:40.943 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:42:40.973 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-20T22:42:40.973 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2024-09-20T22:42:40.973 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2024-09-20T22:42:40.973 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-20T22:42:41.044 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:42:41.065 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:42:41.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:42:41.065 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:42:41.166 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:42:41.188 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:42:41.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:42:41.189 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:42:41.290 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:42:41.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:42:41.313 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:42:41.313 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:42:41.414 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:42:41.429 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:41 smithi154 ceph-mon[33264]: 3.5 scrub starts 2024-09-20T22:42:41.430 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:41 smithi154 ceph-mon[33264]: 3.5 scrub ok 2024-09-20T22:42:41.430 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:41 smithi154 ceph-mon[33264]: pgmap v28: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:42:41.430 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:41 smithi154 ceph-mon[33264]: 3.1f deep-scrub starts 2024-09-20T22:42:41.430 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:41 smithi154 ceph-mon[33264]: 3.1f deep-scrub ok 2024-09-20T22:42:41.430 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:41 smithi154 ceph-mon[33264]: Deploying daemon node-exporter.a on smithi084 2024-09-20T22:42:41.430 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:41 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=infra.usagestats.collector t=2024-09-20T22:42:41.125414301Z level=info msg="registering usage stat providers" usageStatsProvidersLen=2 2024-09-20T22:42:41.430 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:41 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=provisioning.datasources t=2024-09-20T22:42:41.126411053Z level=info msg="inserting datasource from configuration " name=Dashboard1 uid=P43CA22E17D0F9596 2024-09-20T22:42:41.430 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:41 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=provisioning.datasources t=2024-09-20T22:42:41.137750717Z level=info msg="inserting datasource from configuration " name=Loki uid=P8E80F9AEF21F6940 2024-09-20T22:42:41.431 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:41 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=provisioning.alerting t=2024-09-20T22:42:41.148566618Z level=info msg="starting to provision alerting" 2024-09-20T22:42:41.431 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:41 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=provisioning.alerting t=2024-09-20T22:42:41.148577525Z level=info msg="finished to provision alerting" 2024-09-20T22:42:41.431 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:41 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=ngalert.state.manager t=2024-09-20T22:42:41.148734454Z level=info msg="Warming state cache for startup" 2024-09-20T22:42:41.431 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:41 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=ngalert.state.manager t=2024-09-20T22:42:41.148878448Z level=info msg="State cache has been initialized" states=0 duration=142.995µs 2024-09-20T22:42:41.431 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:41 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=ticker t=2024-09-20T22:42:41.148977268Z level=info msg=starting first_tick=2024-09-20T22:42:50Z 2024-09-20T22:42:41.431 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:41 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=ngalert.multiorg.alertmanager t=2024-09-20T22:42:41.149132889Z level=info msg="starting MultiOrg Alertmanager" 2024-09-20T22:42:41.431 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:41 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=grafanaStorageLogger t=2024-09-20T22:42:41.149446726Z level=info msg="storage starting" 2024-09-20T22:42:41.431 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:41 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=http.server t=2024-09-20T22:42:41.150777512Z level=info msg="HTTP Server Listen" address=[::]:3000 protocol=https subUrl= socket= 2024-09-20T22:42:41.431 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:41 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=sqlstore.transactions t=2024-09-20T22:42:41.164398894Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-09-20T22:42:41.431 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:41 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=sqlstore.transactions t=2024-09-20T22:42:41.175634193Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=1 code="database is locked" 2024-09-20T22:42:41.432 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:42:41 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=sqlstore.transactions t=2024-09-20T22:42:41.1861415Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=2 code="database is locked" 2024-09-20T22:42:41.439 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:42:41.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:42:41.440 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:42:41.541 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:42:41.543 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:41 smithi084 ceph-mon[40024]: 3.5 scrub starts 2024-09-20T22:42:41.544 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:41 smithi084 ceph-mon[40024]: 3.5 scrub ok 2024-09-20T22:42:41.544 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:41 smithi084 ceph-mon[40024]: pgmap v28: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:42:41.544 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:41 smithi084 ceph-mon[40024]: 3.1f deep-scrub starts 2024-09-20T22:42:41.544 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:41 smithi084 ceph-mon[40024]: 3.1f deep-scrub ok 2024-09-20T22:42:41.544 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:41 smithi084 ceph-mon[40024]: Deploying daemon node-exporter.a on smithi084 2024-09-20T22:42:41.544 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:41 smithi084 ceph-mon[29774]: 3.5 scrub starts 2024-09-20T22:42:41.544 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:41 smithi084 ceph-mon[29774]: 3.5 scrub ok 2024-09-20T22:42:41.544 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:41 smithi084 ceph-mon[29774]: pgmap v28: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:42:41.544 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:41 smithi084 ceph-mon[29774]: 3.1f deep-scrub starts 2024-09-20T22:42:41.544 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:41 smithi084 ceph-mon[29774]: 3.1f deep-scrub ok 2024-09-20T22:42:41.544 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:41 smithi084 ceph-mon[29774]: Deploying daemon node-exporter.a on smithi084 2024-09-20T22:42:41.564 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:42:41.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:42:41.565 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:42:41.584 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.0 dump_blocked_ops 2024-09-20T22:42:41.666 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:42:41.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:42:41.698 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:42:41.698 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:42:41.799 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:42:41.821 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:42:41.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:42:41.822 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:42:41.853 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:42:41.854 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:42:41.923 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:42:41.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:42:41.946 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:42:41.946 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:42:42.047 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:42:42.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:42:42.069 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:42:42.069 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:42:42.115 INFO:journalctl@ceph.node-exporter.a.smithi084.stdout:Sep 20 22:42:42 smithi084 systemd[1]: Starting Ceph node-exporter.a for a9c0bc70-77a0-11ef-baf5-efdc52797490... 2024-09-20T22:42:42.170 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:42:42.192 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:42:42.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:42:42.193 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:42:42.293 INFO:teuthology.orchestra.run.smithi084.stdout:{"pg_ready":true,"pg_map":{"version":28,"stamp":"2024-09-20T22:42:39.616567+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596507,"num_objects":199,"num_object_clones":0,"num_object_copies":597,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":199,"num_whiteouts":0,"num_read":989,"num_read_kb":835,"num_write":615,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":601,"ondisk_log_size":601,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":240552,"kb_used_data":7688,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":749720664,"statfs":{"total":767960285184,"available":767713959936,"internally_reserved":0,"allocated":7872512,"data_stored":4403445,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001413"},"pg_stats":[{"pgid":"3.1f","version":"67'1","reported_seq":55,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-20T22:42:38.100275+0000","last_change":"2024-09-20T22:41:38.159365+0000","last_active":"2024-09-20T22:42:38.100275+0000","last_peered":"2024-09-20T22:42:38.100275+0000","last_clean":"2024-09-20T22:42:38.100275+0000","last_became_active":"2024-09-20T22:41:38.159085+0000","last_became_peered":"2024-09-20T22:41:38.159085+0000","last_unstale":"2024-09-20T22:42:38.100275+0000","last_undegraded":"2024-09-20T22:42:38.100275+0000","last_fullsized":"2024-09-20T22:42:38.100275+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"74'9","reported_seq":61,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-20T22:42:38.096993+0000","last_change":"2024-09-20T22:41:40.169673+0000","last_active":"2024-09-20T22:42:38.096993+0000","last_peered":"2024-09-20T22:42:38.096993+0000","last_clean":"2024-09-20T22:42:38.096993+0000","last_became_active":"2024-09-20T22:41:40.168934+0000","last_became_peered":"2024-09-20T22:41:40.168934+0000","last_unstale":"2024-09-20T22:42:38.096993+0000","last_undegraded":"2024-09-20T22:42:38.096993+0000","last_fullsized":"2024-09-20T22:42:38.096993+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:54.489342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-20T22:42:36.094322+0000","last_change":"2024-09-20T22:41:42.182304+0000","last_active":"2024-09-20T22:42:36.094322+0000","last_peered":"2024-09-20T22:42:36.094322+0000","last_clean":"2024-09-20T22:42:36.094322+0000","last_became_active":"2024-09-20T22:41:42.182024+0000","last_became_peered":"2024-09-20T22:41:42.182024+0000","last_unstale":"2024-09-20T22:42:36.094322+0000","last_undegraded":"2024-09-20T22:42:36.094322+0000","last_fullsized":"2024-09-20T22:42:36.094322+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:57.397887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":37,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-20T22:42:38.097028+0000","last_change":"2024-09-20T22:41:44.198207+0000","last_active":"2024-09-20T22:42:38.097028+0000","last_peered":"2024-09-20T22:42:38.097028+0000","last_clean":"2024-09-20T22:42:38.097028+0000","last_became_active":"2024-09-20T22:41:44.197897+0000","last_became_peered":"2024-09-20T22:41:44.197897+0000","last_unstale":"2024-09-20T22:42:38.097028+0000","last_undegraded":"2024-09-20T22:42:38.097028+0000","last_fullsized":"2024-09-20T22:42:38.097028+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:53.466451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":37,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-20T22:42:38.429112+0000","last_change":"2024-09-20T22:41:44.832449+0000","last_active":"2024-09-20T22:42:38.429112+0000","last_peered":"2024-09-20T22:42:38.429112+0000","last_clean":"2024-09-20T22:42:38.429112+0000","last_became_active":"2024-09-20T22:41:44.832099+0000","last_became_peered":"2024-09-20T22:41:44.832099+0000","last_unstale":"2024-09-20T22:42:38.429112+0000","last_undegraded":"2024-09-20T22:42:38.429112+0000","last_fullsized":"2024-09-20T22:42:38.429112+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:45.326749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":49,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-20T22:42:38.429637+0000","last_change":"2024-09-20T22:41:38.157685+0000","last_active":"2024-09-20T22:42:38.429637+0000","last_peered":"2024-09-20T22:42:38.429637+0000","last_clean":"2024-09-20T22:42:38.429637+0000","last_became_active":"2024-09-20T22:41:38.157298+0000","last_became_peered":"2024-09-20T22:41:38.157298+0000","last_unstale":"2024-09-20T22:42:38.429637+0000","last_undegraded":"2024-09-20T22:42:38.429637+0000","last_fullsized":"2024-09-20T22:42:38.429637+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:00.813798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"74'15","reported_seq":70,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-20T22:42:38.429326+0000","last_change":"2024-09-20T22:41:40.163585+0000","last_active":"2024-09-20T22:42:38.429326+0000","last_peered":"2024-09-20T22:42:38.429326+0000","last_clean":"2024-09-20T22:42:38.429326+0000","last_became_active":"2024-09-20T22:41:40.163242+0000","last_became_peered":"2024-09-20T22:41:40.163242+0000","last_unstale":"2024-09-20T22:42:38.429326+0000","last_undegraded":"2024-09-20T22:42:38.429326+0000","last_fullsized":"2024-09-20T22:42:38.429326+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:58.410323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":41,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-20T22:42:38.096968+0000","last_change":"2024-09-20T22:41:42.187359+0000","last_active":"2024-09-20T22:42:38.096968+0000","last_peered":"2024-09-20T22:42:38.096968+0000","last_clean":"2024-09-20T22:42:38.096968+0000","last_became_active":"2024-09-20T22:41:42.187076+0000","last_became_peered":"2024-09-20T22:41:42.187076+0000","last_unstale":"2024-09-20T22:42:38.096968+0000","last_undegraded":"2024-09-20T22:42:38.096968+0000","last_fullsized":"2024-09-20T22:42:38.096968+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:42.282475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":49,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-20T22:42:38.564380+0000","last_change":"2024-09-20T22:41:38.156694+0000","last_active":"2024-09-20T22:42:38.564380+0000","last_peered":"2024-09-20T22:42:38.564380+0000","last_clean":"2024-09-20T22:42:38.564380+0000","last_became_active":"2024-09-20T22:41:38.154827+0000","last_became_peered":"2024-09-20T22:41:38.154827+0000","last_unstale":"2024-09-20T22:42:38.564380+0000","last_undegraded":"2024-09-20T22:42:38.564380+0000","last_fullsized":"2024-09-20T22:42:38.564380+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:48.618357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"74'9","reported_seq":61,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-20T22:42:38.097245+0000","last_change":"2024-09-20T22:41:40.163006+0000","last_active":"2024-09-20T22:42:38.097245+0000","last_peered":"2024-09-20T22:42:38.097245+0000","last_clean":"2024-09-20T22:42:38.097245+0000","last_became_active":"2024-09-20T22:41:40.162546+0000","last_became_peered":"2024-09-20T22:41:40.162546+0000","last_unstale":"2024-09-20T22:42:38.097245+0000","last_undegraded":"2024-09-20T22:42:38.097245+0000","last_fullsized":"2024-09-20T22:42:38.097245+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:46.501647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":41,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-20T22:42:38.563344+0000","last_change":"2024-09-20T22:41:42.185835+0000","last_active":"2024-09-20T22:42:38.563344+0000","last_peered":"2024-09-20T22:42:38.563344+0000","last_clean":"2024-09-20T22:42:38.563344+0000","last_became_active":"2024-09-20T22:41:42.185538+0000","last_became_peered":"2024-09-20T22:41:42.185538+0000","last_unstale":"2024-09-20T22:42:38.563344+0000","last_undegraded":"2024-09-20T22:42:38.563344+0000","last_fullsized":"2024-09-20T22:42:38.563344+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:57.305761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":37,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-20T22:42:38.098980+0000","last_change":"2024-09-20T22:41:44.198169+0000","last_active":"2024-09-20T22:42:38.098980+0000","last_peered":"2024-09-20T22:42:38.098980+0000","last_clean":"2024-09-20T22:42:38.098980+0000","last_became_active":"2024-09-20T22:41:44.197113+0000","last_became_peered":"2024-09-20T22:41:44.197113+0000","last_unstale":"2024-09-20T22:42:38.098980+0000","last_undegraded":"2024-09-20T22:42:38.098980+0000","last_fullsized":"2024-09-20T22:42:38.098980+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:02.374640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":49,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-20T22:42:38.564245+0000","last_change":"2024-09-20T22:41:38.156116+0000","last_active":"2024-09-20T22:42:38.564245+0000","last_peered":"2024-09-20T22:42:38.564245+0000","last_clean":"2024-09-20T22:42:38.564245+0000","last_became_active":"2024-09-20T22:41:38.154650+0000","last_became_peered":"2024-09-20T22:41:38.154650+0000","last_unstale":"2024-09-20T22:42:38.564245+0000","last_undegraded":"2024-09-20T22:42:38.564245+0000","last_fullsized":"2024-09-20T22:42:38.564245+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:44.595033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"74'5","reported_seq":55,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-20T22:42:38.097221+0000","last_change":"2024-09-20T22:41:40.171883+0000","last_active":"2024-09-20T22:42:38.097221+0000","last_peered":"2024-09-20T22:42:38.097221+0000","last_clean":"2024-09-20T22:42:38.097221+0000","last_became_active":"2024-09-20T22:41:40.171081+0000","last_became_peered":"2024-09-20T22:41:40.171081+0000","last_unstale":"2024-09-20T22:42:38.097221+0000","last_undegraded":"2024-09-20T22:42:38.097221+0000","last_fullsized":"2024-09-20T22:42:38.097221+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:55.048729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":41,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-20T22:42:38.335839+0000","last_change":"2024-09-20T22:41:42.183965+0000","last_active":"2024-09-20T22:42:38.335839+0000","last_peered":"2024-09-20T22:42:38.335839+0000","last_clean":"2024-09-20T22:42:38.335839+0000","last_became_active":"2024-09-20T22:41:42.182916+0000","last_became_peered":"2024-09-20T22:41:42.182916+0000","last_unstale":"2024-09-20T22:42:38.335839+0000","last_undegraded":"2024-09-20T22:42:38.335839+0000","last_fullsized":"2024-09-20T22:42:38.335839+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:42.258865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":37,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-20T22:42:38.563639+0000","last_change":"2024-09-20T22:41:44.198221+0000","last_active":"2024-09-20T22:42:38.563639+0000","last_peered":"2024-09-20T22:42:38.563639+0000","last_clean":"2024-09-20T22:42:38.563639+0000","last_became_active":"2024-09-20T22:41:44.197927+0000","last_became_peered":"2024-09-20T22:41:44.197927+0000","last_unstale":"2024-09-20T22:42:38.563639+0000","last_undegraded":"2024-09-20T22:42:38.563639+0000","last_fullsized":"2024-09-20T22:42:38.563639+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:47.985353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":37,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-20T22:42:38.097563+0000","last_change":"2024-09-20T22:41:44.832646+0000","last_active":"2024-09-20T22:42:38.097563+0000","last_peered":"2024-09-20T22:42:38.097563+0000","last_clean":"2024-09-20T22:42:38.097563+0000","last_became_active":"2024-09-20T22:41:44.832360+0000","last_became_peered":"2024-09-20T22:41:44.832360+0000","last_unstale":"2024-09-20T22:42:38.097563+0000","last_undegraded":"2024-09-20T22:42:38.097563+0000","last_fullsized":"2024-09-20T22:42:38.097563+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:45.707782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":49,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-20T22:42:38.100750+0000","last_change":"2024-09-20T22:41:38.152726+0000","last_active":"2024-09-20T22:42:38.100750+0000","last_peered":"2024-09-20T22:42:38.100750+0000","last_clean":"2024-09-20T22:42:38.100750+0000","last_became_active":"2024-09-20T22:41:38.152443+0000","last_became_peered":"2024-09-20T22:41:38.152443+0000","last_unstale":"2024-09-20T22:42:38.100750+0000","last_undegraded":"2024-09-20T22:42:38.100750+0000","last_fullsized":"2024-09-20T22:42:38.100750+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:03.325746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"74'15","reported_seq":70,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-20T22:42:38.099838+0000","last_change":"2024-09-20T22:41:40.170414+0000","last_active":"2024-09-20T22:42:38.099838+0000","last_peered":"2024-09-20T22:42:38.099838+0000","last_clean":"2024-09-20T22:42:38.099838+0000","last_became_active":"2024-09-20T22:41:40.170131+0000","last_became_peered":"2024-09-20T22:41:40.170131+0000","last_unstale":"2024-09-20T22:42:38.099838+0000","last_undegraded":"2024-09-20T22:42:38.099838+0000","last_fullsized":"2024-09-20T22:42:38.099838+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:57.393598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-20T22:42:36.094989+0000","last_change":"2024-09-20T22:41:42.185171+0000","last_active":"2024-09-20T22:42:36.094989+0000","last_peered":"2024-09-20T22:42:36.094989+0000","last_clean":"2024-09-20T22:42:36.094989+0000","last_became_active":"2024-09-20T22:41:42.184759+0000","last_became_peered":"2024-09-20T22:41:42.184759+0000","last_unstale":"2024-09-20T22:42:36.094989+0000","last_undegraded":"2024-09-20T22:42:36.094989+0000","last_fullsized":"2024-09-20T22:42:36.094989+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:01.601642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":37,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-20T22:42:38.429485+0000","last_change":"2024-09-20T22:41:44.832468+0000","last_active":"2024-09-20T22:42:38.429485+0000","last_peered":"2024-09-20T22:42:38.429485+0000","last_clean":"2024-09-20T22:42:38.429485+0000","last_became_active":"2024-09-20T22:41:44.832131+0000","last_became_peered":"2024-09-20T22:41:44.832131+0000","last_unstale":"2024-09-20T22:42:38.429485+0000","last_undegraded":"2024-09-20T22:42:38.429485+0000","last_fullsized":"2024-09-20T22:42:38.429485+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:08.595231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":49,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-20T22:42:38.097547+0000","last_change":"2024-09-20T22:41:38.150584+0000","last_active":"2024-09-20T22:42:38.097547+0000","last_peered":"2024-09-20T22:42:38.097547+0000","last_clean":"2024-09-20T22:42:38.097547+0000","last_became_active":"2024-09-20T22:41:38.150232+0000","last_became_peered":"2024-09-20T22:41:38.150232+0000","last_unstale":"2024-09-20T22:42:38.097547+0000","last_undegraded":"2024-09-20T22:42:38.097547+0000","last_fullsized":"2024-09-20T22:42:38.097547+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:02.237312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"74'12","reported_seq":68,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-20T22:42:38.429411+0000","last_change":"2024-09-20T22:41:40.166344+0000","last_active":"2024-09-20T22:42:38.429411+0000","last_peered":"2024-09-20T22:42:38.429411+0000","last_clean":"2024-09-20T22:42:38.429411+0000","last_became_active":"2024-09-20T22:41:40.166087+0000","last_became_peered":"2024-09-20T22:41:40.166087+0000","last_unstale":"2024-09-20T22:42:38.429411+0000","last_undegraded":"2024-09-20T22:42:38.429411+0000","last_fullsized":"2024-09-20T22:42:38.429411+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:01.373771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":41,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-20T22:42:38.097214+0000","last_change":"2024-09-20T22:41:42.183266+0000","last_active":"2024-09-20T22:42:38.097214+0000","last_peered":"2024-09-20T22:42:38.097214+0000","last_clean":"2024-09-20T22:42:38.097214+0000","last_became_active":"2024-09-20T22:41:42.183005+0000","last_became_peered":"2024-09-20T22:41:42.183005+0000","last_unstale":"2024-09-20T22:42:38.097214+0000","last_undegraded":"2024-09-20T22:42:38.097214+0000","last_fullsized":"2024-09-20T22:42:38.097214+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:01.169906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":38,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-20T22:42:38.336447+0000","last_change":"2024-09-20T22:41:44.200115+0000","last_active":"2024-09-20T22:42:38.336447+0000","last_peered":"2024-09-20T22:42:38.336447+0000","last_clean":"2024-09-20T22:42:38.336447+0000","last_became_active":"2024-09-20T22:41:44.199822+0000","last_became_peered":"2024-09-20T22:41:44.199822+0000","last_unstale":"2024-09-20T22:42:38.336447+0000","last_undegraded":"2024-09-20T22:42:38.336447+0000","last_fullsized":"2024-09-20T22:42:38.336447+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:01.277411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"67'1","reported_seq":48,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-20T22:42:36.094491+0000","last_change":"2024-09-20T22:41:38.155735+0000","last_active":"2024-09-20T22:42:36.094491+0000","last_peered":"2024-09-20T22:42:36.094491+0000","last_clean":"2024-09-20T22:42:36.094491+0000","last_became_active":"2024-09-20T22:41:38.155424+0000","last_became_peered":"2024-09-20T22:41:38.155424+0000","last_unstale":"2024-09-20T22:42:36.094491+0000","last_undegraded":"2024-09-20T22:42:36.094491+0000","last_fullsized":"2024-09-20T22:42:36.094491+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:52.584133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"74'10","reported_seq":60,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-20T22:42:38.100034+0000","last_change":"2024-09-20T22:41:40.252498+0000","last_active":"2024-09-20T22:42:38.100034+0000","last_peered":"2024-09-20T22:42:38.100034+0000","last_clean":"2024-09-20T22:42:38.100034+0000","last_became_active":"2024-09-20T22:41:40.252001+0000","last_became_peered":"2024-09-20T22:41:40.252001+0000","last_unstale":"2024-09-20T22:42:38.100034+0000","last_undegraded":"2024-09-20T22:42:38.100034+0000","last_fullsized":"2024-09-20T22:42:38.100034+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:01.417640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"74'8","reported_seq":66,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-20T22:42:36.096630+0000","last_change":"2024-09-20T22:41:42.186385+0000","last_active":"2024-09-20T22:42:36.096630+0000","last_peered":"2024-09-20T22:42:36.096630+0000","last_clean":"2024-09-20T22:42:36.096630+0000","last_became_active":"2024-09-20T22:41:42.186071+0000","last_became_peered":"2024-09-20T22:41:42.186071+0000","last_unstale":"2024-09-20T22:42:36.096630+0000","last_undegraded":"2024-09-20T22:42:36.096630+0000","last_fullsized":"2024-09-20T22:42:36.096630+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:56.929679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.f","version":"74'15","reported_seq":68,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-20T22:42:36.095138+0000","last_change":"2024-09-20T22:41:40.168188+0000","last_active":"2024-09-20T22:42:36.095138+0000","last_peered":"2024-09-20T22:42:36.095138+0000","last_clean":"2024-09-20T22:42:36.095138+0000","last_became_active":"2024-09-20T22:41:40.165963+0000","last_became_peered":"2024-09-20T22:41:40.165963+0000","last_unstale":"2024-09-20T22:42:36.095138+0000","last_undegraded":"2024-09-20T22:42:36.095138+0000","last_fullsized":"2024-09-20T22:42:36.095138+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:46.519265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":49,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-20T22:42:38.429065+0000","last_change":"2024-09-20T22:41:38.157043+0000","last_active":"2024-09-20T22:42:38.429065+0000","last_peered":"2024-09-20T22:42:38.429065+0000","last_clean":"2024-09-20T22:42:38.429065+0000","last_became_active":"2024-09-20T22:41:38.156749+0000","last_became_peered":"2024-09-20T22:41:38.156749+0000","last_unstale":"2024-09-20T22:42:38.429065+0000","last_undegraded":"2024-09-20T22:42:38.429065+0000","last_fullsized":"2024-09-20T22:42:38.429065+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:03.406143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"74'8","reported_seq":62,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-20T22:42:38.097667+0000","last_change":"2024-09-20T22:41:42.182555+0000","last_active":"2024-09-20T22:42:38.097667+0000","last_peered":"2024-09-20T22:42:38.097667+0000","last_clean":"2024-09-20T22:42:38.097667+0000","last_became_active":"2024-09-20T22:41:42.182224+0000","last_became_peered":"2024-09-20T22:41:42.182224+0000","last_unstale":"2024-09-20T22:42:38.097667+0000","last_undegraded":"2024-09-20T22:42:38.097667+0000","last_fullsized":"2024-09-20T22:42:38.097667+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:07.384166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":37,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-20T22:42:38.564162+0000","last_change":"2024-09-20T22:41:44.198655+0000","last_active":"2024-09-20T22:42:38.564162+0000","last_peered":"2024-09-20T22:42:38.564162+0000","last_clean":"2024-09-20T22:42:38.564162+0000","last_became_active":"2024-09-20T22:41:44.198347+0000","last_became_peered":"2024-09-20T22:41:44.198347+0000","last_unstale":"2024-09-20T22:42:38.564162+0000","last_undegraded":"2024-09-20T22:42:38.564162+0000","last_fullsized":"2024-09-20T22:42:38.564162+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:54.119646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.0","version":"74'18","reported_seq":77,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-20T22:42:38.428934+0000","last_change":"2024-09-20T22:41:40.252203+0000","last_active":"2024-09-20T22:42:38.428934+0000","last_peered":"2024-09-20T22:42:38.428934+0000","last_clean":"2024-09-20T22:42:38.428934+0000","last_became_active":"2024-09-20T22:41:40.251868+0000","last_became_peered":"2024-09-20T22:41:40.251868+0000","last_unstale":"2024-09-20T22:42:38.428934+0000","last_undegraded":"2024-09-20T22:42:38.428934+0000","last_fullsized":"2024-09-20T22:42:38.428934+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:57.966357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":49,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-20T22:42:38.429734+0000","last_change":"2024-09-20T22:41:38.161991+0000","last_active":"2024-09-20T22:42:38.429734+0000","last_peered":"2024-09-20T22:42:38.429734+0000","last_clean":"2024-09-20T22:42:38.429734+0000","last_became_active":"2024-09-20T22:41:38.161335+0000","last_became_peered":"2024-09-20T22:41:38.161335+0000","last_unstale":"2024-09-20T22:42:38.429734+0000","last_undegraded":"2024-09-20T22:42:38.429734+0000","last_fullsized":"2024-09-20T22:42:38.429734+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:48.134689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":41,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-20T22:42:38.096984+0000","last_change":"2024-09-20T22:41:42.190157+0000","last_active":"2024-09-20T22:42:38.096984+0000","last_peered":"2024-09-20T22:42:38.096984+0000","last_clean":"2024-09-20T22:42:38.096984+0000","last_became_active":"2024-09-20T22:41:42.189743+0000","last_became_peered":"2024-09-20T22:41:42.189743+0000","last_unstale":"2024-09-20T22:42:38.096984+0000","last_undegraded":"2024-09-20T22:42:38.096984+0000","last_fullsized":"2024-09-20T22:42:38.096984+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:42.602485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":37,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-20T22:42:38.097238+0000","last_change":"2024-09-20T22:41:44.193944+0000","last_active":"2024-09-20T22:42:38.097238+0000","last_peered":"2024-09-20T22:42:38.097238+0000","last_clean":"2024-09-20T22:42:38.097238+0000","last_became_active":"2024-09-20T22:41:44.193646+0000","last_became_peered":"2024-09-20T22:41:44.193646+0000","last_unstale":"2024-09-20T22:42:38.097238+0000","last_undegraded":"2024-09-20T22:42:38.097238+0000","last_fullsized":"2024-09-20T22:42:38.097238+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:50.182915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"74'14","reported_seq":66,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-20T22:42:38.097017+0000","last_change":"2024-09-20T22:41:40.164011+0000","last_active":"2024-09-20T22:42:38.097017+0000","last_peered":"2024-09-20T22:42:38.097017+0000","last_clean":"2024-09-20T22:42:38.097017+0000","last_became_active":"2024-09-20T22:41:40.163695+0000","last_became_peered":"2024-09-20T22:41:40.163695+0000","last_unstale":"2024-09-20T22:42:38.097017+0000","last_undegraded":"2024-09-20T22:42:38.097017+0000","last_fullsized":"2024-09-20T22:42:38.097017+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:04.888108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"67'1","reported_seq":50,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-20T22:42:38.100296+0000","last_change":"2024-09-20T22:41:38.150707+0000","last_active":"2024-09-20T22:42:38.100296+0000","last_peered":"2024-09-20T22:42:38.100296+0000","last_clean":"2024-09-20T22:42:38.100296+0000","last_became_active":"2024-09-20T22:41:38.150331+0000","last_became_peered":"2024-09-20T22:41:38.150331+0000","last_unstale":"2024-09-20T22:42:38.100296+0000","last_undegraded":"2024-09-20T22:42:38.100296+0000","last_fullsized":"2024-09-20T22:42:38.100296+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:53.797157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"74'8","reported_seq":62,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-20T22:42:38.428748+0000","last_change":"2024-09-20T22:41:42.179893+0000","last_active":"2024-09-20T22:42:38.428748+0000","last_peered":"2024-09-20T22:42:38.428748+0000","last_clean":"2024-09-20T22:42:38.428748+0000","last_became_active":"2024-09-20T22:41:42.179613+0000","last_became_peered":"2024-09-20T22:41:42.179613+0000","last_unstale":"2024-09-20T22:42:38.428748+0000","last_undegraded":"2024-09-20T22:42:38.428748+0000","last_fullsized":"2024-09-20T22:42:38.428748+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:03.642295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":37,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-20T22:42:38.336198+0000","last_change":"2024-09-20T22:41:44.832161+0000","last_active":"2024-09-20T22:42:38.336198+0000","last_peered":"2024-09-20T22:42:38.336198+0000","last_clean":"2024-09-20T22:42:38.336198+0000","last_became_active":"2024-09-20T22:41:44.831901+0000","last_became_peered":"2024-09-20T22:41:44.831901+0000","last_unstale":"2024-09-20T22:42:38.336198+0000","last_undegraded":"2024-09-20T22:42:38.336198+0000","last_fullsized":"2024-09-20T22:42:38.336198+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:10.151190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2","version":"74'10","reported_seq":58,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-20T22:42:36.094512+0000","last_change":"2024-09-20T22:41:40.167041+0000","last_active":"2024-09-20T22:42:36.094512+0000","last_peered":"2024-09-20T22:42:36.094512+0000","last_clean":"2024-09-20T22:42:36.094512+0000","last_became_active":"2024-09-20T22:41:40.165653+0000","last_became_peered":"2024-09-20T22:41:40.165653+0000","last_unstale":"2024-09-20T22:42:36.094512+0000","last_undegraded":"2024-09-20T22:42:36.094512+0000","last_fullsized":"2024-09-20T22:42:36.094512+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:47.255874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":49,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-20T22:42:38.564487+0000","last_change":"2024-09-20T22:41:38.155924+0000","last_active":"2024-09-20T22:42:38.564487+0000","last_peered":"2024-09-20T22:42:38.564487+0000","last_clean":"2024-09-20T22:42:38.564487+0000","last_became_active":"2024-09-20T22:41:38.155629+0000","last_became_peered":"2024-09-20T22:41:38.155629+0000","last_unstale":"2024-09-20T22:42:38.564487+0000","last_undegraded":"2024-09-20T22:42:38.564487+0000","last_fullsized":"2024-09-20T22:42:38.564487+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:39.438618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"74'8","reported_seq":62,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-20T22:42:38.100588+0000","last_change":"2024-09-20T22:41:42.185583+0000","last_active":"2024-09-20T22:42:38.100588+0000","last_peered":"2024-09-20T22:42:38.100588+0000","last_clean":"2024-09-20T22:42:38.100588+0000","last_became_active":"2024-09-20T22:41:42.185213+0000","last_became_peered":"2024-09-20T22:41:42.185213+0000","last_unstale":"2024-09-20T22:42:38.100588+0000","last_undegraded":"2024-09-20T22:42:38.100588+0000","last_fullsized":"2024-09-20T22:42:38.100588+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:48.459335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":37,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-20T22:42:38.099075+0000","last_change":"2024-09-20T22:41:44.197894+0000","last_active":"2024-09-20T22:42:38.099075+0000","last_peered":"2024-09-20T22:42:38.099075+0000","last_clean":"2024-09-20T22:42:38.099075+0000","last_became_active":"2024-09-20T22:41:44.197619+0000","last_became_peered":"2024-09-20T22:41:44.197619+0000","last_unstale":"2024-09-20T22:42:38.099075+0000","last_undegraded":"2024-09-20T22:42:38.099075+0000","last_fullsized":"2024-09-20T22:42:38.099075+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:00.802751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"74'19","reported_seq":81,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-20T22:42:38.100646+0000","last_change":"2024-09-20T22:41:40.252159+0000","last_active":"2024-09-20T22:42:38.100646+0000","last_peered":"2024-09-20T22:42:38.100646+0000","last_clean":"2024-09-20T22:42:38.100646+0000","last_became_active":"2024-09-20T22:41:40.251866+0000","last_became_peered":"2024-09-20T22:41:40.251866+0000","last_unstale":"2024-09-20T22:42:38.100646+0000","last_undegraded":"2024-09-20T22:42:38.100646+0000","last_fullsized":"2024-09-20T22:42:38.100646+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:43.089389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":47,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-20T22:42:36.095346+0000","last_change":"2024-09-20T22:41:38.161236+0000","last_active":"2024-09-20T22:42:36.095346+0000","last_peered":"2024-09-20T22:42:36.095346+0000","last_clean":"2024-09-20T22:42:36.095346+0000","last_became_active":"2024-09-20T22:41:38.160832+0000","last_became_peered":"2024-09-20T22:41:38.160832+0000","last_unstale":"2024-09-20T22:42:36.095346+0000","last_undegraded":"2024-09-20T22:42:36.095346+0000","last_fullsized":"2024-09-20T22:42:36.095346+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:50.975305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-20T22:42:36.094899+0000","last_change":"2024-09-20T22:41:42.185638+0000","last_active":"2024-09-20T22:42:36.094899+0000","last_peered":"2024-09-20T22:42:36.094899+0000","last_clean":"2024-09-20T22:42:36.094899+0000","last_became_active":"2024-09-20T22:41:42.184412+0000","last_became_peered":"2024-09-20T22:41:42.184412+0000","last_unstale":"2024-09-20T22:42:36.094899+0000","last_undegraded":"2024-09-20T22:42:36.094899+0000","last_fullsized":"2024-09-20T22:42:36.094899+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:57.435580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":35,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-20T22:42:36.094225+0000","last_change":"2024-09-20T22:41:44.830917+0000","last_active":"2024-09-20T22:42:36.094225+0000","last_peered":"2024-09-20T22:42:36.094225+0000","last_clean":"2024-09-20T22:42:36.094225+0000","last_became_active":"2024-09-20T22:41:44.830538+0000","last_became_peered":"2024-09-20T22:41:44.830538+0000","last_unstale":"2024-09-20T22:42:36.094225+0000","last_undegraded":"2024-09-20T22:42:36.094225+0000","last_fullsized":"2024-09-20T22:42:36.094225+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:54.961702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"74'28","reported_seq":104,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-20T22:42:36.095362+0000","last_change":"2024-09-20T22:41:40.167313+0000","last_active":"2024-09-20T22:42:36.095362+0000","last_peered":"2024-09-20T22:42:36.095362+0000","last_clean":"2024-09-20T22:42:36.095362+0000","last_became_active":"2024-09-20T22:41:40.166958+0000","last_became_peered":"2024-09-20T22:41:40.166958+0000","last_unstale":"2024-09-20T22:42:36.095362+0000","last_undegraded":"2024-09-20T22:42:36.095362+0000","last_fullsized":"2024-09-20T22:42:36.095362+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:42.825818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":57,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-20T22:42:38.097611+0000","last_change":"2024-09-20T22:42:37.810752+0000","last_active":"2024-09-20T22:42:38.097611+0000","last_peered":"2024-09-20T22:42:38.097611+0000","last_clean":"2024-09-20T22:42:38.097611+0000","last_became_active":"2024-09-20T22:41:38.152525+0000","last_became_peered":"2024-09-20T22:41:38.152525+0000","last_unstale":"2024-09-20T22:42:38.097611+0000","last_undegraded":"2024-09-20T22:42:38.097611+0000","last_fullsized":"2024-09-20T22:42:38.097611+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:42:37.810656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:42:37.810656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:57.321042+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"69'2","reported_seq":59,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-20T22:42:38.564016+0000","last_change":"2024-09-20T22:41:40.154106+0000","last_active":"2024-09-20T22:42:38.564016+0000","last_peered":"2024-09-20T22:42:38.564016+0000","last_clean":"2024-09-20T22:42:38.564016+0000","last_became_active":"2024-09-20T22:41:37.148479+0000","last_became_peered":"2024-09-20T22:41:37.148479+0000","last_unstale":"2024-09-20T22:42:38.564016+0000","last_undegraded":"2024-09-20T22:42:38.564016+0000","last_fullsized":"2024-09-20T22:42:38.564016+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:36.130684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:36.130684+0000","last_clean_scrub_stamp":"2024-09-20T22:41:36.130684+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:48.204536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087280599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":41,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-20T22:42:38.099981+0000","last_change":"2024-09-20T22:41:42.179979+0000","last_active":"2024-09-20T22:42:38.099981+0000","last_peered":"2024-09-20T22:42:38.099981+0000","last_clean":"2024-09-20T22:42:38.099981+0000","last_became_active":"2024-09-20T22:41:42.179742+0000","last_became_peered":"2024-09-20T22:41:42.179742+0000","last_unstale":"2024-09-20T22:42:38.099981+0000","last_undegraded":"2024-09-20T22:42:38.099981+0000","last_fullsized":"2024-09-20T22:42:38.099981+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:03.273567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":37,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-20T22:42:38.429520+0000","last_change":"2024-09-20T22:41:44.192602+0000","last_active":"2024-09-20T22:42:38.429520+0000","last_peered":"2024-09-20T22:42:38.429520+0000","last_clean":"2024-09-20T22:42:38.429520+0000","last_became_active":"2024-09-20T22:41:44.192128+0000","last_became_peered":"2024-09-20T22:41:44.192128+0000","last_unstale":"2024-09-20T22:42:38.429520+0000","last_undegraded":"2024-09-20T22:42:38.429520+0000","last_fullsized":"2024-09-20T22:42:38.429520+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:46.416059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"74'13","reported_seq":70,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-20T22:42:36.095240+0000","last_change":"2024-09-20T22:41:40.169777+0000","last_active":"2024-09-20T22:42:36.095240+0000","last_peered":"2024-09-20T22:42:36.095240+0000","last_clean":"2024-09-20T22:42:36.095240+0000","last_became_active":"2024-09-20T22:41:40.169473+0000","last_became_peered":"2024-09-20T22:41:40.169473+0000","last_unstale":"2024-09-20T22:42:36.095240+0000","last_undegraded":"2024-09-20T22:42:36.095240+0000","last_fullsized":"2024-09-20T22:42:36.095240+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:48.626103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":47,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-20T22:42:36.094564+0000","last_change":"2024-09-20T22:41:38.157113+0000","last_active":"2024-09-20T22:42:36.094564+0000","last_peered":"2024-09-20T22:42:36.094564+0000","last_clean":"2024-09-20T22:42:36.094564+0000","last_became_active":"2024-09-20T22:41:38.156837+0000","last_became_peered":"2024-09-20T22:41:38.156837+0000","last_unstale":"2024-09-20T22:42:36.094564+0000","last_undegraded":"2024-09-20T22:42:36.094564+0000","last_fullsized":"2024-09-20T22:42:36.094564+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:46.990363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"67'1","reported_seq":58,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-20T22:42:38.099751+0000","last_change":"2024-09-20T22:41:40.162396+0000","last_active":"2024-09-20T22:42:38.099751+0000","last_peered":"2024-09-20T22:42:38.099751+0000","last_clean":"2024-09-20T22:42:38.099751+0000","last_became_active":"2024-09-20T22:41:37.148165+0000","last_became_peered":"2024-09-20T22:41:37.148165+0000","last_unstale":"2024-09-20T22:42:38.099751+0000","last_undegraded":"2024-09-20T22:42:38.099751+0000","last_fullsized":"2024-09-20T22:42:38.099751+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:36.130684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:36.130684+0000","last_clean_scrub_stamp":"2024-09-20T22:41:36.130684+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:54.327893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001301459,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":41,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-20T22:42:38.099821+0000","last_change":"2024-09-20T22:41:42.185412+0000","last_active":"2024-09-20T22:42:38.099821+0000","last_peered":"2024-09-20T22:42:38.099821+0000","last_clean":"2024-09-20T22:42:38.099821+0000","last_became_active":"2024-09-20T22:41:42.185112+0000","last_became_peered":"2024-09-20T22:41:42.185112+0000","last_unstale":"2024-09-20T22:42:38.099821+0000","last_undegraded":"2024-09-20T22:42:38.099821+0000","last_fullsized":"2024-09-20T22:42:38.099821+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:05.899621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":37,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-20T22:42:38.336491+0000","last_change":"2024-09-20T22:41:44.830804+0000","last_active":"2024-09-20T22:42:38.336491+0000","last_peered":"2024-09-20T22:42:38.336491+0000","last_clean":"2024-09-20T22:42:38.336491+0000","last_became_active":"2024-09-20T22:41:44.830515+0000","last_became_peered":"2024-09-20T22:41:44.830515+0000","last_unstale":"2024-09-20T22:42:38.336491+0000","last_undegraded":"2024-09-20T22:42:38.336491+0000","last_fullsized":"2024-09-20T22:42:38.336491+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:04.958838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"74'12","reported_seq":63,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-20T22:42:38.100124+0000","last_change":"2024-09-20T22:41:40.167483+0000","last_active":"2024-09-20T22:42:38.100124+0000","last_peered":"2024-09-20T22:42:38.100124+0000","last_clean":"2024-09-20T22:42:38.100124+0000","last_became_active":"2024-09-20T22:41:40.167169+0000","last_became_peered":"2024-09-20T22:41:40.167169+0000","last_unstale":"2024-09-20T22:42:38.100124+0000","last_undegraded":"2024-09-20T22:42:38.100124+0000","last_fullsized":"2024-09-20T22:42:38.100124+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:48.803800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":49,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-20T22:42:38.099318+0000","last_change":"2024-09-20T22:41:38.153448+0000","last_active":"2024-09-20T22:42:38.099318+0000","last_peered":"2024-09-20T22:42:38.099318+0000","last_clean":"2024-09-20T22:42:38.099318+0000","last_became_active":"2024-09-20T22:41:38.153058+0000","last_became_peered":"2024-09-20T22:41:38.153058+0000","last_unstale":"2024-09-20T22:42:38.099318+0000","last_undegraded":"2024-09-20T22:42:38.099318+0000","last_fullsized":"2024-09-20T22:42:38.099318+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:04.425050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"75'5","reported_seq":115,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-20T22:42:38.336096+0000","last_change":"2024-09-20T22:41:40.251762+0000","last_active":"2024-09-20T22:42:38.336096+0000","last_peered":"2024-09-20T22:42:38.336096+0000","last_clean":"2024-09-20T22:42:38.336096+0000","last_became_active":"2024-09-20T22:41:37.148383+0000","last_became_peered":"2024-09-20T22:41:37.148383+0000","last_unstale":"2024-09-20T22:42:38.336096+0000","last_undegraded":"2024-09-20T22:42:38.336096+0000","last_fullsized":"2024-09-20T22:42:38.336096+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:36.130684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:36.130684+0000","last_clean_scrub_stamp":"2024-09-20T22:41:36.130684+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:04.208717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00078631700000000003,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":55,"num_read_kb":50,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":41,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-20T22:42:38.563499+0000","last_change":"2024-09-20T22:41:42.187618+0000","last_active":"2024-09-20T22:42:38.563499+0000","last_peered":"2024-09-20T22:42:38.563499+0000","last_clean":"2024-09-20T22:42:38.563499+0000","last_became_active":"2024-09-20T22:41:42.187346+0000","last_became_peered":"2024-09-20T22:41:42.187346+0000","last_unstale":"2024-09-20T22:42:38.563499+0000","last_undegraded":"2024-09-20T22:42:38.563499+0000","last_fullsized":"2024-09-20T22:42:38.563499+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:58.427580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":35,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-20T22:42:36.095145+0000","last_change":"2024-09-20T22:41:44.199301+0000","last_active":"2024-09-20T22:42:36.095145+0000","last_peered":"2024-09-20T22:42:36.095145+0000","last_clean":"2024-09-20T22:42:36.095145+0000","last_became_active":"2024-09-20T22:41:44.199046+0000","last_became_peered":"2024-09-20T22:41:44.199046+0000","last_unstale":"2024-09-20T22:42:36.095145+0000","last_undegraded":"2024-09-20T22:42:36.095145+0000","last_fullsized":"2024-09-20T22:42:36.095145+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:43.190792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"74'16","reported_seq":78,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-20T22:42:36.096139+0000","last_change":"2024-09-20T22:41:40.253524+0000","last_active":"2024-09-20T22:42:36.096139+0000","last_peered":"2024-09-20T22:42:36.096139+0000","last_clean":"2024-09-20T22:42:36.096139+0000","last_became_active":"2024-09-20T22:41:40.252933+0000","last_became_peered":"2024-09-20T22:41:40.252933+0000","last_unstale":"2024-09-20T22:42:36.096139+0000","last_undegraded":"2024-09-20T22:42:36.096139+0000","last_fullsized":"2024-09-20T22:42:36.096139+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:54.546845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"67'1","reported_seq":55,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-20T22:42:38.429382+0000","last_change":"2024-09-20T22:41:38.159706+0000","last_active":"2024-09-20T22:42:38.429382+0000","last_peered":"2024-09-20T22:42:38.429382+0000","last_clean":"2024-09-20T22:42:38.429382+0000","last_became_active":"2024-09-20T22:41:38.158917+0000","last_became_peered":"2024-09-20T22:41:38.158917+0000","last_unstale":"2024-09-20T22:42:38.429382+0000","last_undegraded":"2024-09-20T22:42:38.429382+0000","last_fullsized":"2024-09-20T22:42:38.429382+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:03.995423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"77'128","reported_seq":306,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-20T22:42:38.336206+0000","last_change":"2024-09-20T22:41:43.408723+0000","last_active":"2024-09-20T22:42:38.336206+0000","last_peered":"2024-09-20T22:42:38.336206+0000","last_clean":"2024-09-20T22:42:38.336206+0000","last_became_active":"2024-09-20T22:41:31.091135+0000","last_became_peered":"2024-09-20T22:41:31.091135+0000","last_unstale":"2024-09-20T22:42:38.336206+0000","last_undegraded":"2024-09-20T22:42:38.336206+0000","last_fullsized":"2024-09-20T22:42:38.336206+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'117","last_scrub_stamp":"2024-09-20T22:41:43.408644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.408644+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:48.125134+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":41,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-20T22:42:38.335801+0000","last_change":"2024-09-20T22:41:42.185135+0000","last_active":"2024-09-20T22:42:38.335801+0000","last_peered":"2024-09-20T22:42:38.335801+0000","last_clean":"2024-09-20T22:42:38.335801+0000","last_became_active":"2024-09-20T22:41:42.184387+0000","last_became_peered":"2024-09-20T22:41:42.184387+0000","last_unstale":"2024-09-20T22:42:38.335801+0000","last_undegraded":"2024-09-20T22:42:38.335801+0000","last_fullsized":"2024-09-20T22:42:38.335801+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:00.112877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":37,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-20T22:42:38.564473+0000","last_change":"2024-09-20T22:41:44.191113+0000","last_active":"2024-09-20T22:42:38.564473+0000","last_peered":"2024-09-20T22:42:38.564473+0000","last_clean":"2024-09-20T22:42:38.564473+0000","last_became_active":"2024-09-20T22:41:44.190658+0000","last_became_peered":"2024-09-20T22:41:44.190658+0000","last_unstale":"2024-09-20T22:42:38.564473+0000","last_undegraded":"2024-09-20T22:42:38.564473+0000","last_fullsized":"2024-09-20T22:42:38.564473+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:44.921190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"74'11","reported_seq":64,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-20T22:42:38.097731+0000","last_change":"2024-09-20T22:41:40.169542+0000","last_active":"2024-09-20T22:42:38.097731+0000","last_peered":"2024-09-20T22:42:38.097731+0000","last_clean":"2024-09-20T22:42:38.097731+0000","last_became_active":"2024-09-20T22:41:40.168942+0000","last_became_peered":"2024-09-20T22:41:40.168942+0000","last_unstale":"2024-09-20T22:42:38.097731+0000","last_undegraded":"2024-09-20T22:42:38.097731+0000","last_fullsized":"2024-09-20T22:42:38.097731+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:58.224721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":49,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-20T22:42:38.097386+0000","last_change":"2024-09-20T22:41:38.161801+0000","last_active":"2024-09-20T22:42:38.097386+0000","last_peered":"2024-09-20T22:42:38.097386+0000","last_clean":"2024-09-20T22:42:38.097386+0000","last_became_active":"2024-09-20T22:41:38.161500+0000","last_became_peered":"2024-09-20T22:41:38.161500+0000","last_unstale":"2024-09-20T22:42:38.097386+0000","last_undegraded":"2024-09-20T22:42:38.097386+0000","last_fullsized":"2024-09-20T22:42:38.097386+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:48.636045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":41,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-20T22:42:38.563630+0000","last_change":"2024-09-20T22:41:42.183601+0000","last_active":"2024-09-20T22:42:38.563630+0000","last_peered":"2024-09-20T22:42:38.563630+0000","last_clean":"2024-09-20T22:42:38.563630+0000","last_became_active":"2024-09-20T22:41:42.183302+0000","last_became_peered":"2024-09-20T22:41:42.183302+0000","last_unstale":"2024-09-20T22:42:38.563630+0000","last_undegraded":"2024-09-20T22:42:38.563630+0000","last_fullsized":"2024-09-20T22:42:38.563630+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:10.335337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":37,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-20T22:42:38.429224+0000","last_change":"2024-09-20T22:41:44.832295+0000","last_active":"2024-09-20T22:42:38.429224+0000","last_peered":"2024-09-20T22:42:38.429224+0000","last_clean":"2024-09-20T22:42:38.429224+0000","last_became_active":"2024-09-20T22:41:44.832009+0000","last_became_peered":"2024-09-20T22:41:44.832009+0000","last_unstale":"2024-09-20T22:42:38.429224+0000","last_undegraded":"2024-09-20T22:42:38.429224+0000","last_fullsized":"2024-09-20T22:42:38.429224+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:56.047489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"74'17","reported_seq":73,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-20T22:42:38.097562+0000","last_change":"2024-09-20T22:41:40.169498+0000","last_active":"2024-09-20T22:42:38.097562+0000","last_peered":"2024-09-20T22:42:38.097562+0000","last_clean":"2024-09-20T22:42:38.097562+0000","last_became_active":"2024-09-20T22:41:40.169192+0000","last_became_peered":"2024-09-20T22:41:40.169192+0000","last_unstale":"2024-09-20T22:42:38.097562+0000","last_undegraded":"2024-09-20T22:42:38.097562+0000","last_fullsized":"2024-09-20T22:42:38.097562+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:40.654708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":47,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-20T22:42:36.095344+0000","last_change":"2024-09-20T22:41:38.154220+0000","last_active":"2024-09-20T22:42:36.095344+0000","last_peered":"2024-09-20T22:42:36.095344+0000","last_clean":"2024-09-20T22:42:36.095344+0000","last_became_active":"2024-09-20T22:41:38.153926+0000","last_became_peered":"2024-09-20T22:41:38.153926+0000","last_unstale":"2024-09-20T22:42:36.095344+0000","last_undegraded":"2024-09-20T22:42:36.095344+0000","last_fullsized":"2024-09-20T22:42:36.095344+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:54.196177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-20T22:42:36.095142+0000","last_change":"2024-09-20T22:41:42.183482+0000","last_active":"2024-09-20T22:42:36.095142+0000","last_peered":"2024-09-20T22:42:36.095142+0000","last_clean":"2024-09-20T22:42:36.095142+0000","last_became_active":"2024-09-20T22:41:42.183225+0000","last_became_peered":"2024-09-20T22:41:42.183225+0000","last_unstale":"2024-09-20T22:42:36.095142+0000","last_undegraded":"2024-09-20T22:42:36.095142+0000","last_fullsized":"2024-09-20T22:42:36.095142+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:10.228360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":37,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-20T22:42:38.100023+0000","last_change":"2024-09-20T22:41:44.197855+0000","last_active":"2024-09-20T22:42:38.100023+0000","last_peered":"2024-09-20T22:42:38.100023+0000","last_clean":"2024-09-20T22:42:38.100023+0000","last_became_active":"2024-09-20T22:41:44.197561+0000","last_became_peered":"2024-09-20T22:41:44.197561+0000","last_unstale":"2024-09-20T22:42:38.100023+0000","last_undegraded":"2024-09-20T22:42:38.100023+0000","last_fullsized":"2024-09-20T22:42:38.100023+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:45.431229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"74'10","reported_seq":60,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-20T22:42:38.097354+0000","last_change":"2024-09-20T22:41:40.162973+0000","last_active":"2024-09-20T22:42:38.097354+0000","last_peered":"2024-09-20T22:42:38.097354+0000","last_clean":"2024-09-20T22:42:38.097354+0000","last_became_active":"2024-09-20T22:41:40.162494+0000","last_became_peered":"2024-09-20T22:41:40.162494+0000","last_unstale":"2024-09-20T22:42:38.097354+0000","last_undegraded":"2024-09-20T22:42:38.097354+0000","last_fullsized":"2024-09-20T22:42:38.097354+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:49.601652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":49,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-20T22:42:38.429289+0000","last_change":"2024-09-20T22:41:38.154861+0000","last_active":"2024-09-20T22:42:38.429289+0000","last_peered":"2024-09-20T22:42:38.429289+0000","last_clean":"2024-09-20T22:42:38.429289+0000","last_became_active":"2024-09-20T22:41:38.154636+0000","last_became_peered":"2024-09-20T22:41:38.154636+0000","last_unstale":"2024-09-20T22:42:38.429289+0000","last_undegraded":"2024-09-20T22:42:38.429289+0000","last_fullsized":"2024-09-20T22:42:38.429289+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:45.035202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"74'8","reported_seq":62,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-20T22:42:38.099327+0000","last_change":"2024-09-20T22:41:42.186528+0000","last_active":"2024-09-20T22:42:38.099327+0000","last_peered":"2024-09-20T22:42:38.099327+0000","last_clean":"2024-09-20T22:42:38.099327+0000","last_became_active":"2024-09-20T22:41:42.185863+0000","last_became_peered":"2024-09-20T22:41:42.185863+0000","last_unstale":"2024-09-20T22:42:38.099327+0000","last_undegraded":"2024-09-20T22:42:38.099327+0000","last_fullsized":"2024-09-20T22:42:38.099327+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:00.762928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":37,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-20T22:42:38.097472+0000","last_change":"2024-09-20T22:41:44.195767+0000","last_active":"2024-09-20T22:42:38.097472+0000","last_peered":"2024-09-20T22:42:38.097472+0000","last_clean":"2024-09-20T22:42:38.097472+0000","last_became_active":"2024-09-20T22:41:44.195160+0000","last_became_peered":"2024-09-20T22:41:44.195160+0000","last_unstale":"2024-09-20T22:42:38.097472+0000","last_undegraded":"2024-09-20T22:42:38.097472+0000","last_fullsized":"2024-09-20T22:42:38.097472+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:07.188986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"74'9","reported_seq":61,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-20T22:42:38.100547+0000","last_change":"2024-09-20T22:41:40.163774+0000","last_active":"2024-09-20T22:42:38.100547+0000","last_peered":"2024-09-20T22:42:38.100547+0000","last_clean":"2024-09-20T22:42:38.100547+0000","last_became_active":"2024-09-20T22:41:40.163495+0000","last_became_peered":"2024-09-20T22:41:40.163495+0000","last_unstale":"2024-09-20T22:42:38.100547+0000","last_undegraded":"2024-09-20T22:42:38.100547+0000","last_fullsized":"2024-09-20T22:42:38.100547+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:41.614897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":49,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-20T22:42:38.564320+0000","last_change":"2024-09-20T22:41:38.156470+0000","last_active":"2024-09-20T22:42:38.564320+0000","last_peered":"2024-09-20T22:42:38.564320+0000","last_clean":"2024-09-20T22:42:38.564320+0000","last_became_active":"2024-09-20T22:41:38.155953+0000","last_became_peered":"2024-09-20T22:41:38.155953+0000","last_unstale":"2024-09-20T22:42:38.564320+0000","last_undegraded":"2024-09-20T22:42:38.564320+0000","last_fullsized":"2024-09-20T22:42:38.564320+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:54.057798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":41,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-20T22:42:38.099797+0000","last_change":"2024-09-20T22:41:42.183269+0000","last_active":"2024-09-20T22:42:38.099797+0000","last_peered":"2024-09-20T22:42:38.099797+0000","last_clean":"2024-09-20T22:42:38.099797+0000","last_became_active":"2024-09-20T22:41:42.182999+0000","last_became_peered":"2024-09-20T22:41:42.182999+0000","last_unstale":"2024-09-20T22:42:38.099797+0000","last_undegraded":"2024-09-20T22:42:38.099797+0000","last_fullsized":"2024-09-20T22:42:38.099797+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:57.955307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":37,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-20T22:42:38.099240+0000","last_change":"2024-09-20T22:41:44.196969+0000","last_active":"2024-09-20T22:42:38.099240+0000","last_peered":"2024-09-20T22:42:38.099240+0000","last_clean":"2024-09-20T22:42:38.099240+0000","last_became_active":"2024-09-20T22:41:44.196633+0000","last_became_peered":"2024-09-20T22:41:44.196633+0000","last_unstale":"2024-09-20T22:42:38.099240+0000","last_undegraded":"2024-09-20T22:42:38.099240+0000","last_fullsized":"2024-09-20T22:42:38.099240+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:03.513323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"74'19","reported_seq":74,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-20T22:42:36.095050+0000","last_change":"2024-09-20T22:41:40.252962+0000","last_active":"2024-09-20T22:42:36.095050+0000","last_peered":"2024-09-20T22:42:36.095050+0000","last_clean":"2024-09-20T22:42:36.095050+0000","last_became_active":"2024-09-20T22:41:40.252011+0000","last_became_peered":"2024-09-20T22:41:40.252011+0000","last_unstale":"2024-09-20T22:42:36.095050+0000","last_undegraded":"2024-09-20T22:42:36.095050+0000","last_fullsized":"2024-09-20T22:42:36.095050+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:51.633432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":49,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-20T22:42:38.336681+0000","last_change":"2024-09-20T22:41:38.161800+0000","last_active":"2024-09-20T22:42:38.336681+0000","last_peered":"2024-09-20T22:42:38.336681+0000","last_clean":"2024-09-20T22:42:38.336681+0000","last_became_active":"2024-09-20T22:41:38.161524+0000","last_became_peered":"2024-09-20T22:41:38.161524+0000","last_unstale":"2024-09-20T22:42:38.336681+0000","last_undegraded":"2024-09-20T22:42:38.336681+0000","last_fullsized":"2024-09-20T22:42:38.336681+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:07.121209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":41,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-20T22:42:38.099475+0000","last_change":"2024-09-20T22:41:42.184605+0000","last_active":"2024-09-20T22:42:38.099475+0000","last_peered":"2024-09-20T22:42:38.099475+0000","last_clean":"2024-09-20T22:42:38.099475+0000","last_became_active":"2024-09-20T22:41:42.184342+0000","last_became_peered":"2024-09-20T22:41:42.184342+0000","last_unstale":"2024-09-20T22:42:38.099475+0000","last_undegraded":"2024-09-20T22:42:38.099475+0000","last_fullsized":"2024-09-20T22:42:38.099475+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:03.475959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":37,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-20T22:42:38.336318+0000","last_change":"2024-09-20T22:41:44.199874+0000","last_active":"2024-09-20T22:42:38.336318+0000","last_peered":"2024-09-20T22:42:38.336318+0000","last_clean":"2024-09-20T22:42:38.336318+0000","last_became_active":"2024-09-20T22:41:44.199395+0000","last_became_peered":"2024-09-20T22:41:44.199395+0000","last_unstale":"2024-09-20T22:42:38.336318+0000","last_undegraded":"2024-09-20T22:42:38.336318+0000","last_fullsized":"2024-09-20T22:42:38.336318+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:44.605488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"74'12","reported_seq":68,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-20T22:42:38.097159+0000","last_change":"2024-09-20T22:41:40.170746+0000","last_active":"2024-09-20T22:42:38.097159+0000","last_peered":"2024-09-20T22:42:38.097159+0000","last_clean":"2024-09-20T22:42:38.097159+0000","last_became_active":"2024-09-20T22:41:40.170133+0000","last_became_peered":"2024-09-20T22:41:40.170133+0000","last_unstale":"2024-09-20T22:42:38.097159+0000","last_undegraded":"2024-09-20T22:42:38.097159+0000","last_fullsized":"2024-09-20T22:42:38.097159+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:39.588868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":49,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-20T22:42:38.336744+0000","last_change":"2024-09-20T22:41:38.163612+0000","last_active":"2024-09-20T22:42:38.336744+0000","last_peered":"2024-09-20T22:42:38.336744+0000","last_clean":"2024-09-20T22:42:38.336744+0000","last_became_active":"2024-09-20T22:41:38.163292+0000","last_became_peered":"2024-09-20T22:41:38.163292+0000","last_unstale":"2024-09-20T22:42:38.336744+0000","last_undegraded":"2024-09-20T22:42:38.336744+0000","last_fullsized":"2024-09-20T22:42:38.336744+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:45.666357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":41,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-20T22:42:38.099405+0000","last_change":"2024-09-20T22:41:42.182102+0000","last_active":"2024-09-20T22:42:38.099405+0000","last_peered":"2024-09-20T22:42:38.099405+0000","last_clean":"2024-09-20T22:42:38.099405+0000","last_became_active":"2024-09-20T22:41:42.181811+0000","last_became_peered":"2024-09-20T22:41:42.181811+0000","last_unstale":"2024-09-20T22:42:38.099405+0000","last_undegraded":"2024-09-20T22:42:38.099405+0000","last_fullsized":"2024-09-20T22:42:38.099405+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:52.248246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":37,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-20T22:42:38.429164+0000","last_change":"2024-09-20T22:41:44.193432+0000","last_active":"2024-09-20T22:42:38.429164+0000","last_peered":"2024-09-20T22:42:38.429164+0000","last_clean":"2024-09-20T22:42:38.429164+0000","last_became_active":"2024-09-20T22:41:44.192609+0000","last_became_peered":"2024-09-20T22:41:44.192609+0000","last_unstale":"2024-09-20T22:42:38.429164+0000","last_undegraded":"2024-09-20T22:42:38.429164+0000","last_fullsized":"2024-09-20T22:42:38.429164+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:50.351142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"74'15","reported_seq":70,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-20T22:42:38.563683+0000","last_change":"2024-09-20T22:41:40.252427+0000","last_active":"2024-09-20T22:42:38.563683+0000","last_peered":"2024-09-20T22:42:38.563683+0000","last_clean":"2024-09-20T22:42:38.563683+0000","last_became_active":"2024-09-20T22:41:40.252113+0000","last_became_peered":"2024-09-20T22:41:40.252113+0000","last_unstale":"2024-09-20T22:42:38.563683+0000","last_undegraded":"2024-09-20T22:42:38.563683+0000","last_fullsized":"2024-09-20T22:42:38.563683+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:06.207870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'2","reported_seq":61,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-20T22:42:38.336762+0000","last_change":"2024-09-20T22:41:38.163593+0000","last_active":"2024-09-20T22:42:38.336762+0000","last_peered":"2024-09-20T22:42:38.336762+0000","last_clean":"2024-09-20T22:42:38.336762+0000","last_became_active":"2024-09-20T22:41:38.163219+0000","last_became_peered":"2024-09-20T22:41:38.163219+0000","last_unstale":"2024-09-20T22:42:38.336762+0000","last_undegraded":"2024-09-20T22:42:38.336762+0000","last_fullsized":"2024-09-20T22:42:38.336762+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:02.881623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"74'8","reported_seq":62,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-20T22:42:38.335796+0000","last_change":"2024-09-20T22:41:42.188223+0000","last_active":"2024-09-20T22:42:38.335796+0000","last_peered":"2024-09-20T22:42:38.335796+0000","last_clean":"2024-09-20T22:42:38.335796+0000","last_became_active":"2024-09-20T22:41:42.186635+0000","last_became_peered":"2024-09-20T22:41:42.186635+0000","last_unstale":"2024-09-20T22:42:38.335796+0000","last_undegraded":"2024-09-20T22:42:38.335796+0000","last_fullsized":"2024-09-20T22:42:38.335796+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:52.120303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":37,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-20T22:42:38.563658+0000","last_change":"2024-09-20T22:41:44.831435+0000","last_active":"2024-09-20T22:42:38.563658+0000","last_peered":"2024-09-20T22:42:38.563658+0000","last_clean":"2024-09-20T22:42:38.563658+0000","last_became_active":"2024-09-20T22:41:44.830556+0000","last_became_peered":"2024-09-20T22:41:44.830556+0000","last_unstale":"2024-09-20T22:42:38.563658+0000","last_undegraded":"2024-09-20T22:42:38.563658+0000","last_fullsized":"2024-09-20T22:42:38.563658+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:08.027826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":47,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-20T22:42:36.094760+0000","last_change":"2024-09-20T22:41:38.159437+0000","last_active":"2024-09-20T22:42:36.094760+0000","last_peered":"2024-09-20T22:42:36.094760+0000","last_clean":"2024-09-20T22:42:36.094760+0000","last_became_active":"2024-09-20T22:41:38.159139+0000","last_became_peered":"2024-09-20T22:41:38.159139+0000","last_unstale":"2024-09-20T22:42:36.094760+0000","last_undegraded":"2024-09-20T22:42:36.094760+0000","last_fullsized":"2024-09-20T22:42:36.094760+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:47.658845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"74'6","reported_seq":54,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-20T22:42:38.429691+0000","last_change":"2024-09-20T22:41:40.252310+0000","last_active":"2024-09-20T22:42:38.429691+0000","last_peered":"2024-09-20T22:42:38.429691+0000","last_clean":"2024-09-20T22:42:38.429691+0000","last_became_active":"2024-09-20T22:41:40.251867+0000","last_became_peered":"2024-09-20T22:41:40.251867+0000","last_unstale":"2024-09-20T22:42:38.429691+0000","last_undegraded":"2024-09-20T22:42:38.429691+0000","last_fullsized":"2024-09-20T22:42:38.429691+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:40.015664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":41,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-20T22:42:38.564391+0000","last_change":"2024-09-20T22:41:42.185218+0000","last_active":"2024-09-20T22:42:38.564391+0000","last_peered":"2024-09-20T22:42:38.564391+0000","last_clean":"2024-09-20T22:42:38.564391+0000","last_became_active":"2024-09-20T22:41:42.184930+0000","last_became_peered":"2024-09-20T22:41:42.184930+0000","last_unstale":"2024-09-20T22:42:38.564391+0000","last_undegraded":"2024-09-20T22:42:38.564391+0000","last_fullsized":"2024-09-20T22:42:38.564391+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:47.818365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"74'1","reported_seq":38,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-20T22:42:38.336460+0000","last_change":"2024-09-20T22:41:44.832085+0000","last_active":"2024-09-20T22:42:38.336460+0000","last_peered":"2024-09-20T22:42:38.336460+0000","last_clean":"2024-09-20T22:42:38.336460+0000","last_became_active":"2024-09-20T22:41:44.831805+0000","last_became_peered":"2024-09-20T22:41:44.831805+0000","last_unstale":"2024-09-20T22:42:38.336460+0000","last_undegraded":"2024-09-20T22:42:38.336460+0000","last_fullsized":"2024-09-20T22:42:38.336460+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:54.445212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"74'9","reported_seq":61,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-20T22:42:38.100209+0000","last_change":"2024-09-20T22:41:40.253633+0000","last_active":"2024-09-20T22:42:38.100209+0000","last_peered":"2024-09-20T22:42:38.100209+0000","last_clean":"2024-09-20T22:42:38.100209+0000","last_became_active":"2024-09-20T22:41:40.253280+0000","last_became_peered":"2024-09-20T22:41:40.253280+0000","last_unstale":"2024-09-20T22:42:38.100209+0000","last_undegraded":"2024-09-20T22:42:38.100209+0000","last_fullsized":"2024-09-20T22:42:38.100209+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:39.928882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":49,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-20T22:42:38.336321+0000","last_change":"2024-09-20T22:41:38.156773+0000","last_active":"2024-09-20T22:42:38.336321+0000","last_peered":"2024-09-20T22:42:38.336321+0000","last_clean":"2024-09-20T22:42:38.336321+0000","last_became_active":"2024-09-20T22:41:38.156497+0000","last_became_peered":"2024-09-20T22:41:38.156497+0000","last_unstale":"2024-09-20T22:42:38.336321+0000","last_undegraded":"2024-09-20T22:42:38.336321+0000","last_fullsized":"2024-09-20T22:42:38.336321+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:01.943269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":41,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-20T22:42:38.429579+0000","last_change":"2024-09-20T22:41:42.188166+0000","last_active":"2024-09-20T22:42:38.429579+0000","last_peered":"2024-09-20T22:42:38.429579+0000","last_clean":"2024-09-20T22:42:38.429579+0000","last_became_active":"2024-09-20T22:41:42.187706+0000","last_became_peered":"2024-09-20T22:41:42.187706+0000","last_unstale":"2024-09-20T22:42:38.429579+0000","last_undegraded":"2024-09-20T22:42:38.429579+0000","last_fullsized":"2024-09-20T22:42:38.429579+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:47.610076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":37,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-20T22:42:38.099476+0000","last_change":"2024-09-20T22:41:44.201642+0000","last_active":"2024-09-20T22:42:38.099476+0000","last_peered":"2024-09-20T22:42:38.099476+0000","last_clean":"2024-09-20T22:42:38.099476+0000","last_became_active":"2024-09-20T22:41:44.201433+0000","last_became_peered":"2024-09-20T22:41:44.201433+0000","last_unstale":"2024-09-20T22:42:38.099476+0000","last_undegraded":"2024-09-20T22:42:38.099476+0000","last_fullsized":"2024-09-20T22:42:38.099476+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:55.257671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"74'9","reported_seq":61,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-20T22:42:38.564346+0000","last_change":"2024-09-20T22:41:40.252840+0000","last_active":"2024-09-20T22:42:38.564346+0000","last_peered":"2024-09-20T22:42:38.564346+0000","last_clean":"2024-09-20T22:42:38.564346+0000","last_became_active":"2024-09-20T22:41:40.252486+0000","last_became_peered":"2024-09-20T22:41:40.252486+0000","last_unstale":"2024-09-20T22:42:38.564346+0000","last_undegraded":"2024-09-20T22:42:38.564346+0000","last_fullsized":"2024-09-20T22:42:38.564346+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:51.443985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":49,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-20T22:42:38.099659+0000","last_change":"2024-09-20T22:41:38.158479+0000","last_active":"2024-09-20T22:42:38.099659+0000","last_peered":"2024-09-20T22:42:38.099659+0000","last_clean":"2024-09-20T22:42:38.099659+0000","last_became_active":"2024-09-20T22:41:38.158146+0000","last_became_peered":"2024-09-20T22:41:38.158146+0000","last_unstale":"2024-09-20T22:42:38.099659+0000","last_undegraded":"2024-09-20T22:42:38.099659+0000","last_fullsized":"2024-09-20T22:42:38.099659+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:55.658232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"74'8","reported_seq":62,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-20T22:42:38.428793+0000","last_change":"2024-09-20T22:41:42.186630+0000","last_active":"2024-09-20T22:42:38.428793+0000","last_peered":"2024-09-20T22:42:38.428793+0000","last_clean":"2024-09-20T22:42:38.428793+0000","last_became_active":"2024-09-20T22:41:42.186352+0000","last_became_peered":"2024-09-20T22:41:42.186352+0000","last_unstale":"2024-09-20T22:42:38.428793+0000","last_undegraded":"2024-09-20T22:42:38.428793+0000","last_fullsized":"2024-09-20T22:42:38.428793+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:48.572027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":37,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-20T22:42:38.097775+0000","last_change":"2024-09-20T22:41:44.194510+0000","last_active":"2024-09-20T22:42:38.097775+0000","last_peered":"2024-09-20T22:42:38.097775+0000","last_clean":"2024-09-20T22:42:38.097775+0000","last_became_active":"2024-09-20T22:41:44.194053+0000","last_became_peered":"2024-09-20T22:41:44.194053+0000","last_unstale":"2024-09-20T22:42:38.097775+0000","last_undegraded":"2024-09-20T22:42:38.097775+0000","last_fullsized":"2024-09-20T22:42:38.097775+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:01.207219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"74'10","reported_seq":60,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-20T22:42:38.429601+0000","last_change":"2024-09-20T22:41:40.253880+0000","last_active":"2024-09-20T22:42:38.429601+0000","last_peered":"2024-09-20T22:42:38.429601+0000","last_clean":"2024-09-20T22:42:38.429601+0000","last_became_active":"2024-09-20T22:41:40.253614+0000","last_became_peered":"2024-09-20T22:41:40.253614+0000","last_unstale":"2024-09-20T22:42:38.429601+0000","last_undegraded":"2024-09-20T22:42:38.429601+0000","last_fullsized":"2024-09-20T22:42:38.429601+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:47.554109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":49,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-20T22:42:38.336413+0000","last_change":"2024-09-20T22:41:38.158008+0000","last_active":"2024-09-20T22:42:38.336413+0000","last_peered":"2024-09-20T22:42:38.336413+0000","last_clean":"2024-09-20T22:42:38.336413+0000","last_became_active":"2024-09-20T22:41:38.157647+0000","last_became_peered":"2024-09-20T22:41:38.157647+0000","last_unstale":"2024-09-20T22:42:38.336413+0000","last_undegraded":"2024-09-20T22:42:38.336413+0000","last_fullsized":"2024-09-20T22:42:38.336413+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:57.980474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"74'8","reported_seq":62,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-20T22:42:38.563344+0000","last_change":"2024-09-20T22:41:42.189757+0000","last_active":"2024-09-20T22:42:38.563344+0000","last_peered":"2024-09-20T22:42:38.563344+0000","last_clean":"2024-09-20T22:42:38.563344+0000","last_became_active":"2024-09-20T22:41:42.189451+0000","last_became_peered":"2024-09-20T22:41:42.189451+0000","last_unstale":"2024-09-20T22:42:38.563344+0000","last_undegraded":"2024-09-20T22:42:38.563344+0000","last_fullsized":"2024-09-20T22:42:38.563344+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:43.846079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":37,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-20T22:42:38.099219+0000","last_change":"2024-09-20T22:41:44.191549+0000","last_active":"2024-09-20T22:42:38.099219+0000","last_peered":"2024-09-20T22:42:38.099219+0000","last_clean":"2024-09-20T22:42:38.099219+0000","last_became_active":"2024-09-20T22:41:44.191206+0000","last_became_peered":"2024-09-20T22:41:44.191206+0000","last_unstale":"2024-09-20T22:42:38.099219+0000","last_undegraded":"2024-09-20T22:42:38.099219+0000","last_fullsized":"2024-09-20T22:42:38.099219+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:58.564765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"74'11","reported_seq":64,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-20T22:42:38.097323+0000","last_change":"2024-09-20T22:41:40.171449+0000","last_active":"2024-09-20T22:42:38.097323+0000","last_peered":"2024-09-20T22:42:38.097323+0000","last_clean":"2024-09-20T22:42:38.097323+0000","last_became_active":"2024-09-20T22:41:40.170035+0000","last_became_peered":"2024-09-20T22:41:40.170035+0000","last_unstale":"2024-09-20T22:42:38.097323+0000","last_undegraded":"2024-09-20T22:42:38.097323+0000","last_fullsized":"2024-09-20T22:42:38.097323+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:49.140455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":49,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-20T22:42:38.097496+0000","last_change":"2024-09-20T22:41:38.161385+0000","last_active":"2024-09-20T22:42:38.097496+0000","last_peered":"2024-09-20T22:42:38.097496+0000","last_clean":"2024-09-20T22:42:38.097496+0000","last_became_active":"2024-09-20T22:41:38.158142+0000","last_became_peered":"2024-09-20T22:41:38.158142+0000","last_unstale":"2024-09-20T22:42:38.097496+0000","last_undegraded":"2024-09-20T22:42:38.097496+0000","last_fullsized":"2024-09-20T22:42:38.097496+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:51.728688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-20T22:42:36.094368+0000","last_change":"2024-09-20T22:41:42.182119+0000","last_active":"2024-09-20T22:42:36.094368+0000","last_peered":"2024-09-20T22:42:36.094368+0000","last_clean":"2024-09-20T22:42:36.094368+0000","last_became_active":"2024-09-20T22:41:42.181736+0000","last_became_peered":"2024-09-20T22:41:42.181736+0000","last_unstale":"2024-09-20T22:42:36.094368+0000","last_undegraded":"2024-09-20T22:42:36.094368+0000","last_fullsized":"2024-09-20T22:42:36.094368+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:02.572377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":37,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-20T22:42:38.429038+0000","last_change":"2024-09-20T22:41:44.193289+0000","last_active":"2024-09-20T22:42:38.429038+0000","last_peered":"2024-09-20T22:42:38.429038+0000","last_clean":"2024-09-20T22:42:38.429038+0000","last_became_active":"2024-09-20T22:41:44.192960+0000","last_became_peered":"2024-09-20T22:41:44.192960+0000","last_unstale":"2024-09-20T22:42:38.429038+0000","last_undegraded":"2024-09-20T22:42:38.429038+0000","last_fullsized":"2024-09-20T22:42:38.429038+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:49.189046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"74'9","reported_seq":59,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-20T22:42:36.094606+0000","last_change":"2024-09-20T22:41:40.165116+0000","last_active":"2024-09-20T22:42:36.094606+0000","last_peered":"2024-09-20T22:42:36.094606+0000","last_clean":"2024-09-20T22:42:36.094606+0000","last_became_active":"2024-09-20T22:41:40.164796+0000","last_became_peered":"2024-09-20T22:41:40.164796+0000","last_unstale":"2024-09-20T22:42:36.094606+0000","last_undegraded":"2024-09-20T22:42:36.094606+0000","last_fullsized":"2024-09-20T22:42:36.094606+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:48.483009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":49,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-20T22:42:38.336590+0000","last_change":"2024-09-20T22:41:38.159941+0000","last_active":"2024-09-20T22:42:38.336590+0000","last_peered":"2024-09-20T22:42:38.336590+0000","last_clean":"2024-09-20T22:42:38.336590+0000","last_became_active":"2024-09-20T22:41:38.159657+0000","last_became_peered":"2024-09-20T22:41:38.159657+0000","last_unstale":"2024-09-20T22:42:38.336590+0000","last_undegraded":"2024-09-20T22:42:38.336590+0000","last_fullsized":"2024-09-20T22:42:38.336590+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:53.225190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":41,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-20T22:42:38.428795+0000","last_change":"2024-09-20T22:41:42.183254+0000","last_active":"2024-09-20T22:42:38.428795+0000","last_peered":"2024-09-20T22:42:38.428795+0000","last_clean":"2024-09-20T22:42:38.428795+0000","last_became_active":"2024-09-20T22:41:42.182966+0000","last_became_peered":"2024-09-20T22:41:42.182966+0000","last_unstale":"2024-09-20T22:42:38.428795+0000","last_undegraded":"2024-09-20T22:42:38.428795+0000","last_fullsized":"2024-09-20T22:42:38.428795+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:51.689726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":37,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-20T22:42:38.098906+0000","last_change":"2024-09-20T22:41:44.194845+0000","last_active":"2024-09-20T22:42:38.098906+0000","last_peered":"2024-09-20T22:42:38.098906+0000","last_clean":"2024-09-20T22:42:38.098906+0000","last_became_active":"2024-09-20T22:41:44.194551+0000","last_became_peered":"2024-09-20T22:41:44.194551+0000","last_unstale":"2024-09-20T22:42:38.098906+0000","last_undegraded":"2024-09-20T22:42:38.098906+0000","last_fullsized":"2024-09-20T22:42:38.098906+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:45.187444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"74'11","reported_seq":64,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-20T22:42:38.429139+0000","last_change":"2024-09-20T22:41:40.252149+0000","last_active":"2024-09-20T22:42:38.429139+0000","last_peered":"2024-09-20T22:42:38.429139+0000","last_clean":"2024-09-20T22:42:38.429139+0000","last_became_active":"2024-09-20T22:41:40.251829+0000","last_became_peered":"2024-09-20T22:41:40.251829+0000","last_unstale":"2024-09-20T22:42:38.429139+0000","last_undegraded":"2024-09-20T22:42:38.429139+0000","last_fullsized":"2024-09-20T22:42:38.429139+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:56.020491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":49,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-20T22:42:38.564495+0000","last_change":"2024-09-20T22:41:38.159064+0000","last_active":"2024-09-20T22:42:38.564495+0000","last_peered":"2024-09-20T22:42:38.564495+0000","last_clean":"2024-09-20T22:42:38.564495+0000","last_became_active":"2024-09-20T22:41:38.158701+0000","last_became_peered":"2024-09-20T22:41:38.158701+0000","last_unstale":"2024-09-20T22:42:38.564495+0000","last_undegraded":"2024-09-20T22:42:38.564495+0000","last_fullsized":"2024-09-20T22:42:38.564495+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:47.727493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":41,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-20T22:42:38.335792+0000","last_change":"2024-09-20T22:41:42.185991+0000","last_active":"2024-09-20T22:42:38.335792+0000","last_peered":"2024-09-20T22:42:38.335792+0000","last_clean":"2024-09-20T22:42:38.335792+0000","last_became_active":"2024-09-20T22:41:42.185296+0000","last_became_peered":"2024-09-20T22:41:42.185296+0000","last_unstale":"2024-09-20T22:42:38.335792+0000","last_undegraded":"2024-09-20T22:42:38.335792+0000","last_fullsized":"2024-09-20T22:42:38.335792+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:55.259880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":37,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-20T22:42:38.429283+0000","last_change":"2024-09-20T22:41:44.832293+0000","last_active":"2024-09-20T22:42:38.429283+0000","last_peered":"2024-09-20T22:42:38.429283+0000","last_clean":"2024-09-20T22:42:38.429283+0000","last_became_active":"2024-09-20T22:41:44.831967+0000","last_became_peered":"2024-09-20T22:41:44.831967+0000","last_unstale":"2024-09-20T22:42:38.429283+0000","last_undegraded":"2024-09-20T22:42:38.429283+0000","last_fullsized":"2024-09-20T22:42:38.429283+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:03.371689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"74'4","reported_seq":51,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-20T22:42:38.428920+0000","last_change":"2024-09-20T22:41:40.166409+0000","last_active":"2024-09-20T22:42:38.428920+0000","last_peered":"2024-09-20T22:42:38.428920+0000","last_clean":"2024-09-20T22:42:38.428920+0000","last_became_active":"2024-09-20T22:41:40.166123+0000","last_became_peered":"2024-09-20T22:41:40.166123+0000","last_unstale":"2024-09-20T22:42:38.428920+0000","last_undegraded":"2024-09-20T22:42:38.428920+0000","last_fullsized":"2024-09-20T22:42:38.428920+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:52.943341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":49,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-20T22:42:38.100360+0000","last_change":"2024-09-20T22:41:38.158861+0000","last_active":"2024-09-20T22:42:38.100360+0000","last_peered":"2024-09-20T22:42:38.100360+0000","last_clean":"2024-09-20T22:42:38.100360+0000","last_became_active":"2024-09-20T22:41:38.158347+0000","last_became_peered":"2024-09-20T22:41:38.158347+0000","last_unstale":"2024-09-20T22:42:38.100360+0000","last_undegraded":"2024-09-20T22:42:38.100360+0000","last_fullsized":"2024-09-20T22:42:38.100360+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:44.061004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-20T22:42:36.094783+0000","last_change":"2024-09-20T22:41:42.187707+0000","last_active":"2024-09-20T22:42:36.094783+0000","last_peered":"2024-09-20T22:42:36.094783+0000","last_clean":"2024-09-20T22:42:36.094783+0000","last_became_active":"2024-09-20T22:41:42.186068+0000","last_became_peered":"2024-09-20T22:41:42.186068+0000","last_unstale":"2024-09-20T22:42:36.094783+0000","last_undegraded":"2024-09-20T22:42:36.094783+0000","last_fullsized":"2024-09-20T22:42:36.094783+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:50.526395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":37,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-20T22:42:38.336198+0000","last_change":"2024-09-20T22:41:44.194011+0000","last_active":"2024-09-20T22:42:38.336198+0000","last_peered":"2024-09-20T22:42:38.336198+0000","last_clean":"2024-09-20T22:42:38.336198+0000","last_became_active":"2024-09-20T22:41:44.193706+0000","last_became_peered":"2024-09-20T22:41:44.193706+0000","last_unstale":"2024-09-20T22:42:38.336198+0000","last_undegraded":"2024-09-20T22:42:38.336198+0000","last_fullsized":"2024-09-20T22:42:38.336198+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:49.835925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":35,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-20T22:42:36.094322+0000","last_change":"2024-09-20T22:41:44.198154+0000","last_active":"2024-09-20T22:42:36.094322+0000","last_peered":"2024-09-20T22:42:36.094322+0000","last_clean":"2024-09-20T22:42:36.094322+0000","last_became_active":"2024-09-20T22:41:44.197855+0000","last_became_peered":"2024-09-20T22:41:44.197855+0000","last_unstale":"2024-09-20T22:42:36.094322+0000","last_undegraded":"2024-09-20T22:42:36.094322+0000","last_fullsized":"2024-09-20T22:42:36.094322+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:56.991123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":49,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-20T22:42:38.429032+0000","last_change":"2024-09-20T22:41:38.155001+0000","last_active":"2024-09-20T22:42:38.429032+0000","last_peered":"2024-09-20T22:42:38.429032+0000","last_clean":"2024-09-20T22:42:38.429032+0000","last_became_active":"2024-09-20T22:41:38.154703+0000","last_became_peered":"2024-09-20T22:41:38.154703+0000","last_unstale":"2024-09-20T22:42:38.429032+0000","last_undegraded":"2024-09-20T22:42:38.429032+0000","last_fullsized":"2024-09-20T22:42:38.429032+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:44.782808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"74'11","reported_seq":62,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-09-20T22:42:36.096587+0000","last_change":"2024-09-20T22:41:40.171840+0000","last_active":"2024-09-20T22:42:36.096587+0000","last_peered":"2024-09-20T22:42:36.096587+0000","last_clean":"2024-09-20T22:42:36.096587+0000","last_became_active":"2024-09-20T22:41:40.171256+0000","last_became_peered":"2024-09-20T22:41:40.171256+0000","last_unstale":"2024-09-20T22:42:36.096587+0000","last_undegraded":"2024-09-20T22:42:36.096587+0000","last_fullsized":"2024-09-20T22:42:36.096587+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:42:53.884281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":41,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-09-20T22:42:38.099351+0000","last_change":"2024-09-20T22:41:42.178413+0000","last_active":"2024-09-20T22:42:38.099351+0000","last_peered":"2024-09-20T22:42:38.099351+0000","last_clean":"2024-09-20T22:42:38.099351+0000","last_became_active":"2024-09-20T22:41:42.178157+0000","last_became_peered":"2024-09-20T22:41:42.178157+0000","last_unstale":"2024-09-20T22:42:38.099351+0000","last_undegraded":"2024-09-20T22:42:38.099351+0000","last_fullsized":"2024-09-20T22:42:38.099351+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:01.837489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":393,"ondisk_log_size":393,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":55,"num_read_kb":50,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070480,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32508,"kb_used_data":1320,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712644,"statfs":{"total":95995035648,"available":95961747456,"internally_reserved":0,"allocated":1351680,"data_stored":918463,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430230,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32492,"kb_used_data":1304,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712660,"statfs":{"total":95995035648,"available":95961763840,"internally_reserved":0,"allocated":1335296,"data_stored":918371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691867,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27756,"kb_used_data":728,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717396,"statfs":{"total":95995035648,"available":95966613504,"internally_reserved":0,"allocated":745472,"data_stored":328785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986208,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27776,"kb_used_data":748,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717376,"statfs":{"total":95995035648,"available":95966593024,"internally_reserved":0,"allocated":765952,"data_stored":327615,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182438,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27784,"kb_used_data":756,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717368,"statfs":{"total":95995035648,"available":95966584832,"internally_reserved":0,"allocated":774144,"data_stored":328881,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411371,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31916,"kb_used_data":728,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713236,"statfs":{"total":95995035648,"available":95962353664,"internally_reserved":0,"allocated":745472,"data_stored":333953,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574896,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27808,"kb_used_data":780,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717344,"statfs":{"total":95995035648,"available":95966560256,"internally_reserved":0,"allocated":798720,"data_stored":328782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738421,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32512,"kb_used_data":1324,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712640,"statfs":{"total":95995035648,"available":95961743360,"internally_reserved":0,"allocated":1355776,"data_stored":918595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-20T22:42:42.299 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-09-20T22:42:42.299 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd pool set .mgr pg_num 11 2024-09-20T22:42:42.300 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:42:42.321 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:42:42.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:42:42.322 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:42:42.410 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:42 smithi084 ceph-mon[29774]: from='client.24686 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T22:42:42.411 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:42 smithi084 ceph-mon[40024]: from='client.24686 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T22:42:42.423 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:42:42.444 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:42:42.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:42:42.445 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:42:42.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:42 smithi154 ceph-mon[33264]: from='client.24686 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T22:42:42.546 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:42:42.573 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:42:42.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:42:42.574 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:42:42.675 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:42:42.696 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:42:42.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:42:42.697 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:42:42.763 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:42:42.763 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:42:42.783 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:42:42.783 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:42:42.798 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:42:42.804 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:42:42.805 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:42:42.817 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:42:42.819 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:42:42.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:42:42.820 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:42:42.828 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:42:42.829 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-20T22:42:42.830 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T22:42:42.848 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:42:42.848 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:42:42.865 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:42:42.866 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:42:42.882 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:42:42.882 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:42:42.898 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:42:42.899 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:42:42.921 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:42:42.942 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:42:42.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:42:42.943 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:42:43.044 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:42:43.067 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:42:43.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:42:43.068 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:42:43.168 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:42:43.191 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:42:43.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:42:43.192 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:42:43.292 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:42:43.314 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:42:43.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:42:43.315 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:42:43.364 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:43 smithi084 ceph-mon[40024]: 4.17 scrub starts 2024-09-20T22:42:43.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:43 smithi084 ceph-mon[40024]: 4.17 scrub ok 2024-09-20T22:42:43.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:43 smithi084 ceph-mon[40024]: pgmap v29: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T22:42:43.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:43 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:42:43.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:43 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/1005560092' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:42:43.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:43 smithi084 ceph-mon[29774]: 4.17 scrub starts 2024-09-20T22:42:43.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:43 smithi084 ceph-mon[29774]: 4.17 scrub ok 2024-09-20T22:42:43.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:43 smithi084 ceph-mon[29774]: pgmap v29: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T22:42:43.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:43 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:42:43.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:43 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/1005560092' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:42:43.416 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:42:43.442 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:42:43.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:42:43.443 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:42:43.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:43 smithi154 ceph-mon[33264]: 4.17 scrub starts 2024-09-20T22:42:43.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:43 smithi154 ceph-mon[33264]: 4.17 scrub ok 2024-09-20T22:42:43.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:43 smithi154 ceph-mon[33264]: pgmap v29: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T22:42:43.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:43 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:42:43.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:43 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/1005560092' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:42:43.544 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:42:43.569 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:42:43.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:42:43.570 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:42:43.670 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:42:43.695 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:42:43.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:42:43.696 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:42:43.797 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:42:43.822 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:42:43.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:42:43.823 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:42:43.923 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:42:43.930 INFO:journalctl@ceph.node-exporter.a.smithi084.stdout:Sep 20 22:42:43 smithi084 bash[75055]: Trying to pull quay.io/prometheus/node-exporter:v1.5.0... 2024-09-20T22:42:43.948 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:42:43.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:42:43.949 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:42:44.049 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:42:44.072 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:42:44.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:42:44.073 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:42:44.160 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is set 2024-09-20T22:42:44.174 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:42:44.197 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:42:44.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:42:44.198 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:42:44.268 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:44 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/1450093913' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-20T22:42:44.268 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:44 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/1450093913' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-20T22:42:44.299 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:42:44.321 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:42:44.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:42:44.322 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:42:44.423 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:42:44.445 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:42:44.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:42:44.446 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:42:44.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:44 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/1450093913' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-20T22:42:44.545 INFO:journalctl@ceph.node-exporter.a.smithi084.stdout:Sep 20 22:42:44 smithi084 bash[75055]: Getting image source signatures 2024-09-20T22:42:44.545 INFO:journalctl@ceph.node-exporter.a.smithi084.stdout:Sep 20 22:42:44 smithi084 bash[75055]: Copying blob sha256:c0e87333d3805a7b7e1c25783dd25f27da0fe70a8e262eee7ed190c8dc843fa7 2024-09-20T22:42:44.545 INFO:journalctl@ceph.node-exporter.a.smithi084.stdout:Sep 20 22:42:44 smithi084 bash[75055]: Copying blob sha256:22b70bddd3acadc892fca4c2af4260629bfda5dfd11ebc106a93ce24e752b5ed 2024-09-20T22:42:44.545 INFO:journalctl@ceph.node-exporter.a.smithi084.stdout:Sep 20 22:42:44 smithi084 bash[75055]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-09-20T22:42:44.547 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:42:44.570 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:42:44.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:42:44.571 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:42:44.671 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:42:44.695 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:42:44.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:42:44.696 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:42:44.796 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:42:44.819 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:42:44.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:42:44.820 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:42:44.921 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:42:44.944 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:42:44.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:42:44.945 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:42:45.046 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:42:45.070 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:42:45.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:42:45.071 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:42:45.167 INFO:teuthology.orchestra.run.smithi084.stderr:set pool 1 pg_num to 11 2024-09-20T22:42:45.172 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:42:45.180 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:45 smithi084 ceph-mon[40024]: pgmap v30: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:42:45.180 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:45 smithi084 ceph-mon[40024]: 4.16 scrub starts 2024-09-20T22:42:45.180 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:45 smithi084 ceph-mon[40024]: 4.16 scrub ok 2024-09-20T22:42:45.180 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:45 smithi084 ceph-mon[40024]: 4.4 deep-scrub starts 2024-09-20T22:42:45.180 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:45 smithi084 ceph-mon[40024]: 4.4 deep-scrub ok 2024-09-20T22:42:45.181 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:45 smithi084 ceph-mon[40024]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:42:45.181 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:45 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/1450093913' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T22:42:45.181 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:45 smithi084 ceph-mon[40024]: osdmap e87: 8 total, 8 up, 8 in 2024-09-20T22:42:45.181 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:45 smithi084 ceph-mon[40024]: 5.1a scrub starts 2024-09-20T22:42:45.181 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:45 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/3270273510' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-09-20T22:42:45.181 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:45 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-09-20T22:42:45.181 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:45 smithi084 ceph-mon[29774]: pgmap v30: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:42:45.181 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:45 smithi084 ceph-mon[29774]: 4.16 scrub starts 2024-09-20T22:42:45.182 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:45 smithi084 ceph-mon[29774]: 4.16 scrub ok 2024-09-20T22:42:45.182 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:45 smithi084 ceph-mon[29774]: 4.4 deep-scrub starts 2024-09-20T22:42:45.182 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:45 smithi084 ceph-mon[29774]: 4.4 deep-scrub ok 2024-09-20T22:42:45.182 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:45 smithi084 ceph-mon[29774]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:42:45.182 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:45 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/1450093913' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T22:42:45.182 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:45 smithi084 ceph-mon[29774]: osdmap e87: 8 total, 8 up, 8 in 2024-09-20T22:42:45.182 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:45 smithi084 ceph-mon[29774]: 5.1a scrub starts 2024-09-20T22:42:45.182 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:45 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/3270273510' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-09-20T22:42:45.182 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:45 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-09-20T22:42:45.194 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:42:45.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:42:45.195 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:42:45.296 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:42:45.318 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:42:45.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:42:45.319 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:42:45.420 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:42:45.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:42:45.443 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:42:45.444 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:42:45.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:45 smithi154 ceph-mon[33264]: pgmap v30: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:42:45.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:45 smithi154 ceph-mon[33264]: 4.16 scrub starts 2024-09-20T22:42:45.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:45 smithi154 ceph-mon[33264]: 4.16 scrub ok 2024-09-20T22:42:45.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:45 smithi154 ceph-mon[33264]: 4.4 deep-scrub starts 2024-09-20T22:42:45.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:45 smithi154 ceph-mon[33264]: 4.4 deep-scrub ok 2024-09-20T22:42:45.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:45 smithi154 ceph-mon[33264]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:42:45.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:45 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/1450093913' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T22:42:45.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:45 smithi154 ceph-mon[33264]: osdmap e87: 8 total, 8 up, 8 in 2024-09-20T22:42:45.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:45 smithi154 ceph-mon[33264]: 5.1a scrub starts 2024-09-20T22:42:45.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:45 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/3270273510' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-09-20T22:42:45.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:45 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-09-20T22:42:45.544 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:42:45.567 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:42:45.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:42:45.568 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:42:45.669 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:42:45.690 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:42:45.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:42:45.691 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:42:45.792 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:42:45.813 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:42:45.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:42:45.814 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:42:45.915 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:42:45.941 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:42:45.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:42:45.942 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:42:46.043 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:42:46.064 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:42:46.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:42:46.065 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:42:46.166 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:42:46.195 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:42:46.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:42:46.196 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:42:46.275 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-20T22:42:46.275 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:42:46.297 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:42:46.317 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:42:46.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:42:46.318 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:42:46.410 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:46 smithi084 ceph-mon[40024]: 5.15 scrub starts 2024-09-20T22:42:46.410 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:46 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-09-20T22:42:46.410 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:46 smithi084 ceph-mon[40024]: osdmap e88: 8 total, 8 up, 8 in 2024-09-20T22:42:46.410 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:46 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-09-20T22:42:46.411 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:46 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-09-20T22:42:46.411 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:46 smithi084 ceph-mon[29774]: 5.15 scrub starts 2024-09-20T22:42:46.411 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:46 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-09-20T22:42:46.411 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:46 smithi084 ceph-mon[29774]: osdmap e88: 8 total, 8 up, 8 in 2024-09-20T22:42:46.411 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:46 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-09-20T22:42:46.411 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:46 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-09-20T22:42:46.420 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:42:46.441 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:42:46.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:42:46.442 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:42:46.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:46 smithi154 ceph-mon[33264]: 5.15 scrub starts 2024-09-20T22:42:46.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:46 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-09-20T22:42:46.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:46 smithi154 ceph-mon[33264]: osdmap e88: 8 total, 8 up, 8 in 2024-09-20T22:42:46.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:46 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-09-20T22:42:46.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:46 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-09-20T22:42:46.543 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:42:46.569 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:42:46.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:42:46.570 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:42:46.573 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-20T22:42:46.671 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:42:46.697 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:42:46.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:42:46.698 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:42:46.797 INFO:journalctl@ceph.node-exporter.a.smithi084.stdout:Sep 20 22:42:46 smithi084 bash[75055]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-09-20T22:42:46.797 INFO:journalctl@ceph.node-exporter.a.smithi084.stdout:Sep 20 22:42:46 smithi084 bash[75055]: Writing manifest to image destination 2024-09-20T22:42:46.798 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:42:46.819 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:42:46.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:42:46.820 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:42:46.921 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:42:46.947 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:42:46.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:42:46.948 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:42:47.049 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:42:47.070 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:42:47.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:42:47.071 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:42:47.172 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:42:47.202 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:42:47.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:42:47.203 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:42:47.304 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:42:47.328 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:42:47.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:42:47.329 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:42:47.361 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:47 smithi084 ceph-mon[40024]: pgmap v33: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T22:42:47.361 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:47 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-09-20T22:42:47.362 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:47 smithi084 ceph-mon[40024]: osdmap e89: 8 total, 8 up, 8 in 2024-09-20T22:42:47.362 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:47 smithi084 ceph-mon[29774]: pgmap v33: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T22:42:47.362 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:47 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-09-20T22:42:47.362 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:47 smithi084 ceph-mon[29774]: osdmap e89: 8 total, 8 up, 8 in 2024-09-20T22:42:47.362 INFO:journalctl@ceph.node-exporter.a.smithi084.stdout:Sep 20 22:42:47 smithi084 podman[75055]: 2024-09-20 22:42:47.13604218 +0000 UTC m=+4.324790109 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-09-20T22:42:47.388 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.0 dump_historic_ops 2024-09-20T22:42:47.429 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:42:47.453 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:42:47.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:42:47.454 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:42:47.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:47 smithi154 ceph-mon[33264]: pgmap v33: 132 pgs: 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-20T22:42:47.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:47 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-09-20T22:42:47.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:47 smithi154 ceph-mon[33264]: osdmap e89: 8 total, 8 up, 8 in 2024-09-20T22:42:47.555 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:42:47.576 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:42:47.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:42:47.577 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:42:47.614 INFO:journalctl@ceph.node-exporter.a.smithi084.stdout:Sep 20 22:42:47 smithi084 podman[75055]: 2024-09-20 22:42:47.511111377 +0000 UTC m=+4.699859293 container create ea6c1ea853e358f65915c309c5219781960766415d008c985531d46bd82c7b3b (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-20T22:42:47.678 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:42:47.701 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:42:47.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:42:47.702 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:42:47.803 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:42:47.825 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:42:47.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:42:47.826 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:42:47.927 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:42:47.941 INFO:journalctl@ceph.node-exporter.a.smithi084.stdout:Sep 20 22:42:47 smithi084 podman[75055]: 2024-09-20 22:42:47.83931482 +0000 UTC m=+5.028062737 container init ea6c1ea853e358f65915c309c5219781960766415d008c985531d46bd82c7b3b (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-20T22:42:47.941 INFO:journalctl@ceph.node-exporter.a.smithi084.stdout:Sep 20 22:42:47 smithi084 podman[75055]: 2024-09-20 22:42:47.842114174 +0000 UTC m=+5.030862092 container start ea6c1ea853e358f65915c309c5219781960766415d008c985531d46bd82c7b3b (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-20T22:42:47.941 INFO:journalctl@ceph.node-exporter.a.smithi084.stdout:Sep 20 22:42:47 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-node-exporter-a[75619]: ts=2024-09-20T22:42:47.846Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-09-20T22:42:47.941 INFO:journalctl@ceph.node-exporter.a.smithi084.stdout:Sep 20 22:42:47 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-node-exporter-a[75619]: ts=2024-09-20T22:42:47.846Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-09-20T22:42:47.941 INFO:journalctl@ceph.node-exporter.a.smithi084.stdout:Sep 20 22:42:47 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-node-exporter-a[75619]: ts=2024-09-20T22:42:47.846Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-09-20T22:42:47.941 INFO:journalctl@ceph.node-exporter.a.smithi084.stdout:Sep 20 22:42:47 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-node-exporter-a[75619]: ts=2024-09-20T22:42:47.846Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-09-20T22:42:47.941 INFO:journalctl@ceph.node-exporter.a.smithi084.stdout:Sep 20 22:42:47 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-node-exporter-a[75619]: ts=2024-09-20T22:42:47.847Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-09-20T22:42:47.941 INFO:journalctl@ceph.node-exporter.a.smithi084.stdout:Sep 20 22:42:47 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-node-exporter-a[75619]: ts=2024-09-20T22:42:47.847Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-09-20T22:42:47.941 INFO:journalctl@ceph.node-exporter.a.smithi084.stdout:Sep 20 22:42:47 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-node-exporter-a[75619]: ts=2024-09-20T22:42:47.847Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-09-20T22:42:47.941 INFO:journalctl@ceph.node-exporter.a.smithi084.stdout:Sep 20 22:42:47 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-node-exporter-a[75619]: ts=2024-09-20T22:42:47.847Z caller=node_exporter.go:117 level=info collector=arp 2024-09-20T22:42:47.942 INFO:journalctl@ceph.node-exporter.a.smithi084.stdout:Sep 20 22:42:47 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-node-exporter-a[75619]: ts=2024-09-20T22:42:47.847Z caller=node_exporter.go:117 level=info collector=bcache 2024-09-20T22:42:47.942 INFO:journalctl@ceph.node-exporter.a.smithi084.stdout:Sep 20 22:42:47 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-node-exporter-a[75619]: ts=2024-09-20T22:42:47.847Z caller=node_exporter.go:117 level=info collector=bonding 2024-09-20T22:42:47.942 INFO:journalctl@ceph.node-exporter.a.smithi084.stdout:Sep 20 22:42:47 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-node-exporter-a[75619]: ts=2024-09-20T22:42:47.847Z caller=node_exporter.go:117 level=info collector=btrfs 2024-09-20T22:42:47.942 INFO:journalctl@ceph.node-exporter.a.smithi084.stdout:Sep 20 22:42:47 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-node-exporter-a[75619]: ts=2024-09-20T22:42:47.847Z caller=node_exporter.go:117 level=info collector=conntrack 2024-09-20T22:42:47.942 INFO:journalctl@ceph.node-exporter.a.smithi084.stdout:Sep 20 22:42:47 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-node-exporter-a[75619]: ts=2024-09-20T22:42:47.847Z caller=node_exporter.go:117 level=info collector=cpu 2024-09-20T22:42:47.942 INFO:journalctl@ceph.node-exporter.a.smithi084.stdout:Sep 20 22:42:47 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-node-exporter-a[75619]: ts=2024-09-20T22:42:47.847Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-09-20T22:42:47.943 INFO:journalctl@ceph.node-exporter.a.smithi084.stdout:Sep 20 22:42:47 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-node-exporter-a[75619]: ts=2024-09-20T22:42:47.847Z caller=node_exporter.go:117 level=info collector=diskstats 2024-09-20T22:42:47.943 INFO:journalctl@ceph.node-exporter.a.smithi084.stdout:Sep 20 22:42:47 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-node-exporter-a[75619]: ts=2024-09-20T22:42:47.847Z caller=node_exporter.go:117 level=info collector=dmi 2024-09-20T22:42:47.943 INFO:journalctl@ceph.node-exporter.a.smithi084.stdout:Sep 20 22:42:47 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-node-exporter-a[75619]: ts=2024-09-20T22:42:47.847Z caller=node_exporter.go:117 level=info collector=edac 2024-09-20T22:42:47.943 INFO:journalctl@ceph.node-exporter.a.smithi084.stdout:Sep 20 22:42:47 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-node-exporter-a[75619]: ts=2024-09-20T22:42:47.847Z caller=node_exporter.go:117 level=info collector=entropy 2024-09-20T22:42:47.943 INFO:journalctl@ceph.node-exporter.a.smithi084.stdout:Sep 20 22:42:47 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-node-exporter-a[75619]: ts=2024-09-20T22:42:47.847Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-09-20T22:42:47.943 INFO:journalctl@ceph.node-exporter.a.smithi084.stdout:Sep 20 22:42:47 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-node-exporter-a[75619]: ts=2024-09-20T22:42:47.847Z caller=node_exporter.go:117 level=info collector=filefd 2024-09-20T22:42:47.943 INFO:journalctl@ceph.node-exporter.a.smithi084.stdout:Sep 20 22:42:47 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-node-exporter-a[75619]: ts=2024-09-20T22:42:47.847Z caller=node_exporter.go:117 level=info collector=filesystem 2024-09-20T22:42:47.943 INFO:journalctl@ceph.node-exporter.a.smithi084.stdout:Sep 20 22:42:47 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-node-exporter-a[75619]: ts=2024-09-20T22:42:47.847Z caller=node_exporter.go:117 level=info collector=hwmon 2024-09-20T22:42:47.943 INFO:journalctl@ceph.node-exporter.a.smithi084.stdout:Sep 20 22:42:47 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-node-exporter-a[75619]: ts=2024-09-20T22:42:47.847Z caller=node_exporter.go:117 level=info collector=infiniband 2024-09-20T22:42:47.943 INFO:journalctl@ceph.node-exporter.a.smithi084.stdout:Sep 20 22:42:47 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-node-exporter-a[75619]: ts=2024-09-20T22:42:47.847Z caller=node_exporter.go:117 level=info collector=ipvs 2024-09-20T22:42:47.943 INFO:journalctl@ceph.node-exporter.a.smithi084.stdout:Sep 20 22:42:47 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-node-exporter-a[75619]: ts=2024-09-20T22:42:47.847Z caller=node_exporter.go:117 level=info collector=loadavg 2024-09-20T22:42:47.943 INFO:journalctl@ceph.node-exporter.a.smithi084.stdout:Sep 20 22:42:47 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-node-exporter-a[75619]: ts=2024-09-20T22:42:47.847Z caller=node_exporter.go:117 level=info collector=mdadm 2024-09-20T22:42:47.944 INFO:journalctl@ceph.node-exporter.a.smithi084.stdout:Sep 20 22:42:47 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-node-exporter-a[75619]: ts=2024-09-20T22:42:47.847Z caller=node_exporter.go:117 level=info collector=meminfo 2024-09-20T22:42:47.944 INFO:journalctl@ceph.node-exporter.a.smithi084.stdout:Sep 20 22:42:47 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-node-exporter-a[75619]: ts=2024-09-20T22:42:47.847Z caller=node_exporter.go:117 level=info collector=netclass 2024-09-20T22:42:47.944 INFO:journalctl@ceph.node-exporter.a.smithi084.stdout:Sep 20 22:42:47 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-node-exporter-a[75619]: ts=2024-09-20T22:42:47.847Z caller=node_exporter.go:117 level=info collector=netdev 2024-09-20T22:42:47.944 INFO:journalctl@ceph.node-exporter.a.smithi084.stdout:Sep 20 22:42:47 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-node-exporter-a[75619]: ts=2024-09-20T22:42:47.847Z caller=node_exporter.go:117 level=info collector=netstat 2024-09-20T22:42:47.944 INFO:journalctl@ceph.node-exporter.a.smithi084.stdout:Sep 20 22:42:47 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-node-exporter-a[75619]: ts=2024-09-20T22:42:47.847Z caller=node_exporter.go:117 level=info collector=nfs 2024-09-20T22:42:47.944 INFO:journalctl@ceph.node-exporter.a.smithi084.stdout:Sep 20 22:42:47 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-node-exporter-a[75619]: ts=2024-09-20T22:42:47.847Z caller=node_exporter.go:117 level=info collector=nfsd 2024-09-20T22:42:47.944 INFO:journalctl@ceph.node-exporter.a.smithi084.stdout:Sep 20 22:42:47 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-node-exporter-a[75619]: ts=2024-09-20T22:42:47.847Z caller=node_exporter.go:117 level=info collector=nvme 2024-09-20T22:42:47.944 INFO:journalctl@ceph.node-exporter.a.smithi084.stdout:Sep 20 22:42:47 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-node-exporter-a[75619]: ts=2024-09-20T22:42:47.847Z caller=node_exporter.go:117 level=info collector=os 2024-09-20T22:42:47.944 INFO:journalctl@ceph.node-exporter.a.smithi084.stdout:Sep 20 22:42:47 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-node-exporter-a[75619]: ts=2024-09-20T22:42:47.847Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-09-20T22:42:47.944 INFO:journalctl@ceph.node-exporter.a.smithi084.stdout:Sep 20 22:42:47 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-node-exporter-a[75619]: ts=2024-09-20T22:42:47.847Z caller=node_exporter.go:117 level=info collector=pressure 2024-09-20T22:42:47.944 INFO:journalctl@ceph.node-exporter.a.smithi084.stdout:Sep 20 22:42:47 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-node-exporter-a[75619]: ts=2024-09-20T22:42:47.847Z caller=node_exporter.go:117 level=info collector=rapl 2024-09-20T22:42:47.944 INFO:journalctl@ceph.node-exporter.a.smithi084.stdout:Sep 20 22:42:47 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-node-exporter-a[75619]: ts=2024-09-20T22:42:47.847Z caller=node_exporter.go:117 level=info collector=schedstat 2024-09-20T22:42:47.944 INFO:journalctl@ceph.node-exporter.a.smithi084.stdout:Sep 20 22:42:47 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-node-exporter-a[75619]: ts=2024-09-20T22:42:47.847Z caller=node_exporter.go:117 level=info collector=selinux 2024-09-20T22:42:47.944 INFO:journalctl@ceph.node-exporter.a.smithi084.stdout:Sep 20 22:42:47 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-node-exporter-a[75619]: ts=2024-09-20T22:42:47.847Z caller=node_exporter.go:117 level=info collector=sockstat 2024-09-20T22:42:47.945 INFO:journalctl@ceph.node-exporter.a.smithi084.stdout:Sep 20 22:42:47 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-node-exporter-a[75619]: ts=2024-09-20T22:42:47.847Z caller=node_exporter.go:117 level=info collector=softnet 2024-09-20T22:42:47.945 INFO:journalctl@ceph.node-exporter.a.smithi084.stdout:Sep 20 22:42:47 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-node-exporter-a[75619]: ts=2024-09-20T22:42:47.847Z caller=node_exporter.go:117 level=info collector=stat 2024-09-20T22:42:47.945 INFO:journalctl@ceph.node-exporter.a.smithi084.stdout:Sep 20 22:42:47 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-node-exporter-a[75619]: ts=2024-09-20T22:42:47.847Z caller=node_exporter.go:117 level=info collector=tapestats 2024-09-20T22:42:47.945 INFO:journalctl@ceph.node-exporter.a.smithi084.stdout:Sep 20 22:42:47 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-node-exporter-a[75619]: ts=2024-09-20T22:42:47.847Z caller=node_exporter.go:117 level=info collector=textfile 2024-09-20T22:42:47.945 INFO:journalctl@ceph.node-exporter.a.smithi084.stdout:Sep 20 22:42:47 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-node-exporter-a[75619]: ts=2024-09-20T22:42:47.847Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-09-20T22:42:47.945 INFO:journalctl@ceph.node-exporter.a.smithi084.stdout:Sep 20 22:42:47 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-node-exporter-a[75619]: ts=2024-09-20T22:42:47.847Z caller=node_exporter.go:117 level=info collector=time 2024-09-20T22:42:47.945 INFO:journalctl@ceph.node-exporter.a.smithi084.stdout:Sep 20 22:42:47 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-node-exporter-a[75619]: ts=2024-09-20T22:42:47.847Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-09-20T22:42:47.945 INFO:journalctl@ceph.node-exporter.a.smithi084.stdout:Sep 20 22:42:47 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-node-exporter-a[75619]: ts=2024-09-20T22:42:47.847Z caller=node_exporter.go:117 level=info collector=uname 2024-09-20T22:42:47.945 INFO:journalctl@ceph.node-exporter.a.smithi084.stdout:Sep 20 22:42:47 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-node-exporter-a[75619]: ts=2024-09-20T22:42:47.847Z caller=node_exporter.go:117 level=info collector=vmstat 2024-09-20T22:42:47.945 INFO:journalctl@ceph.node-exporter.a.smithi084.stdout:Sep 20 22:42:47 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-node-exporter-a[75619]: ts=2024-09-20T22:42:47.847Z caller=node_exporter.go:117 level=info collector=xfs 2024-09-20T22:42:47.945 INFO:journalctl@ceph.node-exporter.a.smithi084.stdout:Sep 20 22:42:47 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-node-exporter-a[75619]: ts=2024-09-20T22:42:47.847Z caller=node_exporter.go:117 level=info collector=zfs 2024-09-20T22:42:47.945 INFO:journalctl@ceph.node-exporter.a.smithi084.stdout:Sep 20 22:42:47 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-node-exporter-a[75619]: ts=2024-09-20T22:42:47.847Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-09-20T22:42:47.945 INFO:journalctl@ceph.node-exporter.a.smithi084.stdout:Sep 20 22:42:47 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-node-exporter-a[75619]: ts=2024-09-20T22:42:47.847Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-09-20T22:42:47.946 INFO:journalctl@ceph.node-exporter.a.smithi084.stdout:Sep 20 22:42:47 smithi084 bash[75055]: ea6c1ea853e358f65915c309c5219781960766415d008c985531d46bd82c7b3b 2024-09-20T22:42:47.949 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:42:47.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:42:47.950 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:42:48.051 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:42:48.073 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:42:48.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:42:48.074 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:42:48.175 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:42:48.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:42:48.202 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:42:48.203 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:42:48.251 INFO:journalctl@ceph.node-exporter.a.smithi084.stdout:Sep 20 22:42:47 smithi084 systemd[1]: Started Ceph node-exporter.a for a9c0bc70-77a0-11ef-baf5-efdc52797490. 2024-09-20T22:42:48.251 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:48 smithi084 ceph-mon[40024]: osdmap e90: 8 total, 8 up, 8 in 2024-09-20T22:42:48.251 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:48 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-09-20T22:42:48.251 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:48 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-09-20T22:42:48.251 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:48 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:42:48.252 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:48 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:42:48.252 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:48 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:42:48.252 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:48 smithi084 ceph-mon[29774]: osdmap e90: 8 total, 8 up, 8 in 2024-09-20T22:42:48.252 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:48 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-09-20T22:42:48.252 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:48 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-09-20T22:42:48.252 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:48 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:42:48.252 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:48 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:42:48.252 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:48 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:42:48.280 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_0 2024-09-20T22:42:48.281 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd pool create unique_pool_0 16 2024-09-20T22:42:48.303 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:42:48.350 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:42:48.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:42:48.351 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:42:48.373 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:42:48.384 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:42:48.439 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-20T22:42:48.452 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:42:48.474 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:42:48.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:42:48.475 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:42:48.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:48 smithi154 ceph-mon[33264]: osdmap e90: 8 total, 8 up, 8 in 2024-09-20T22:42:48.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:48 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-09-20T22:42:48.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:48 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-09-20T22:42:48.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:48 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:42:48.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:48 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:42:48.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:48 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:42:48.522 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:42:48 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: ::ffff:172.21.15.154 - - [20/Sep/2024:22:42:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-20T22:42:48.576 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:42:48.599 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:42:48.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:42:48.600 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:42:48.701 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:42:48.722 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:42:48.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:42:48.723 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:42:48.823 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:42:48.847 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:42:48.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:42:48.848 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:42:48.938 INFO:journalctl@ceph.node-exporter.b.smithi154.stdout:Sep 20 22:42:48 smithi154 systemd[1]: Starting Ceph node-exporter.b for a9c0bc70-77a0-11ef-baf5-efdc52797490... 2024-09-20T22:42:48.948 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:42:48.970 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:42:48.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:42:48.971 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:42:49.072 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:42:49.093 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:42:49.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:42:49.094 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:42:49.103 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-20T22:42:49.103 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:42:49.195 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:42:49.200 INFO:journalctl@ceph.iscsi.iscsi.a.smithi154.stdout:Sep 20 22:42:48 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-iscsi-iscsi-a[59730]: debug there is no tcmu-runner data available 2024-09-20T22:42:49.299 INFO:journalctl@ceph.node-exporter.b.smithi154.stdout:Sep 20 22:42:48 smithi154 bash[63559]: Trying to pull quay.io/prometheus/node-exporter:v1.5.0... 2024-09-20T22:42:49.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:42:49.300 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:42:49.300 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:42:49.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:49 smithi084 ceph-mon[29774]: pgmap v36: 142 pgs: 10 unknown, 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:42:49.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:49 smithi084 ceph-mon[29774]: Deploying daemon node-exporter.b on smithi154 2024-09-20T22:42:49.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:49 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-09-20T22:42:49.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:49 smithi084 ceph-mon[29774]: osdmap e91: 8 total, 8 up, 8 in 2024-09-20T22:42:49.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:49 smithi084 ceph-mon[40024]: pgmap v36: 142 pgs: 10 unknown, 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:42:49.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:49 smithi084 ceph-mon[40024]: Deploying daemon node-exporter.b on smithi154 2024-09-20T22:42:49.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:49 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-09-20T22:42:49.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:49 smithi084 ceph-mon[40024]: osdmap e91: 8 total, 8 up, 8 in 2024-09-20T22:42:49.401 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:42:49.423 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:42:49.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:42:49.424 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:42:49.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:49 smithi154 ceph-mon[33264]: pgmap v36: 142 pgs: 10 unknown, 132 active+clean; 583 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:42:49.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:49 smithi154 ceph-mon[33264]: Deploying daemon node-exporter.b on smithi154 2024-09-20T22:42:49.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:49 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-09-20T22:42:49.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:49 smithi154 ceph-mon[33264]: osdmap e91: 8 total, 8 up, 8 in 2024-09-20T22:42:49.505 INFO:journalctl@ceph.node-exporter.b.smithi154.stdout:Sep 20 22:42:49 smithi154 bash[63559]: Getting image source signatures 2024-09-20T22:42:49.505 INFO:journalctl@ceph.node-exporter.b.smithi154.stdout:Sep 20 22:42:49 smithi154 bash[63559]: Copying blob sha256:c0e87333d3805a7b7e1c25783dd25f27da0fe70a8e262eee7ed190c8dc843fa7 2024-09-20T22:42:49.505 INFO:journalctl@ceph.node-exporter.b.smithi154.stdout:Sep 20 22:42:49 smithi154 bash[63559]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-09-20T22:42:49.505 INFO:journalctl@ceph.node-exporter.b.smithi154.stdout:Sep 20 22:42:49 smithi154 bash[63559]: Copying blob sha256:22b70bddd3acadc892fca4c2af4260629bfda5dfd11ebc106a93ce24e752b5ed 2024-09-20T22:42:49.525 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:42:49.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:42:49.547 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:42:49.547 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:42:49.648 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:42:49.670 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:42:49.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:42:49.671 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:42:49.772 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:42:49.794 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:42:49.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:42:49.795 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:42:49.896 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:42:49.917 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:42:49.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:42:49.918 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:42:50.019 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:42:50.040 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:42:50.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:42:50.041 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:42:50.074 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-20T22:42:50.142 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:42:50.165 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:42:50.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:42:50.166 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:42:50.267 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:42:50.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:42:50.290 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:42:50.290 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:42:50.326 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:50 smithi084 ceph-mon[29774]: purged_snaps scrub starts 2024-09-20T22:42:50.326 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:50 smithi084 ceph-mon[29774]: purged_snaps scrub ok 2024-09-20T22:42:50.326 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:50 smithi084 ceph-mon[29774]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:42:50.326 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:50 smithi084 ceph-mon[29774]: osdmap e92: 8 total, 8 up, 8 in 2024-09-20T22:42:50.326 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:50 smithi084 ceph-mon[29774]: osdmap e93: 8 total, 8 up, 8 in 2024-09-20T22:42:50.326 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:50 smithi084 ceph-mon[40024]: purged_snaps scrub starts 2024-09-20T22:42:50.326 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:50 smithi084 ceph-mon[40024]: purged_snaps scrub ok 2024-09-20T22:42:50.327 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:50 smithi084 ceph-mon[40024]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:42:50.327 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:50 smithi084 ceph-mon[40024]: osdmap e92: 8 total, 8 up, 8 in 2024-09-20T22:42:50.327 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:50 smithi084 ceph-mon[40024]: osdmap e93: 8 total, 8 up, 8 in 2024-09-20T22:42:50.329 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:42:50.330 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:42:50.337 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:42:50.344 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:42:50.344 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:42:50.365 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:42:50.365 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:42:50.372 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:42:50.391 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:42:50.392 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-20T22:42:50.392 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T22:42:50.412 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:42:50.413 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:42:50.416 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:42:50.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:42:50.417 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:42:50.430 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:42:50.431 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:42:50.448 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:42:50.448 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:42:50.465 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:42:50.466 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:42:50.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:50 smithi154 ceph-mon[33264]: purged_snaps scrub starts 2024-09-20T22:42:50.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:50 smithi154 ceph-mon[33264]: purged_snaps scrub ok 2024-09-20T22:42:50.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:50 smithi154 ceph-mon[33264]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:42:50.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:50 smithi154 ceph-mon[33264]: osdmap e92: 8 total, 8 up, 8 in 2024-09-20T22:42:50.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:50 smithi154 ceph-mon[33264]: osdmap e93: 8 total, 8 up, 8 in 2024-09-20T22:42:50.519 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:42:50.541 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:42:50.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:42:50.542 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:42:50.643 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:42:50.700 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:42:50.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:42:50.701 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:42:50.801 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:42:50.823 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:42:50.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:42:50.824 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:42:50.924 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:42:50.948 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:42:50.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:42:50.949 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:42:51.049 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:42:51.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:42:51.071 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:42:51.071 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:42:51.172 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:42:51.193 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:42:51.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:42:51.194 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:42:51.254 INFO:journalctl@ceph.node-exporter.b.smithi154.stdout:Sep 20 22:42:50 smithi154 bash[63559]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-09-20T22:42:51.254 INFO:journalctl@ceph.node-exporter.b.smithi154.stdout:Sep 20 22:42:50 smithi154 bash[63559]: Writing manifest to image destination 2024-09-20T22:42:51.295 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:42:51.315 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:42:51.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:42:51.316 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:42:51.357 INFO:teuthology.orchestra.run.smithi084.stderr:pool 'unique_pool_0' created 2024-09-20T22:42:51.417 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:42:51.439 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:42:51.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:42:51.440 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:42:51.530 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:51 smithi084 ceph-mon[29774]: pgmap v39: 142 pgs: 10 unknown, 132 active+clean; 268 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:42:51.530 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:51 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/353657183' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:42:51.530 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:51 smithi084 ceph-mon[40024]: pgmap v39: 142 pgs: 10 unknown, 132 active+clean; 268 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:42:51.530 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:51 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/353657183' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:42:51.540 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:42:51.561 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:42:51.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:42:51.562 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:42:51.589 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:51 smithi154 ceph-mon[33264]: pgmap v39: 142 pgs: 10 unknown, 132 active+clean; 268 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:42:51.589 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:51 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/353657183' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:42:51.590 INFO:journalctl@ceph.node-exporter.b.smithi154.stdout:Sep 20 22:42:51 smithi154 podman[63559]: 2024-09-20 22:42:51.237901161 +0000 UTC m=+2.352744483 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-09-20T22:42:51.590 INFO:journalctl@ceph.node-exporter.b.smithi154.stdout:Sep 20 22:42:51 smithi154 podman[63559]: 2024-09-20 22:42:51.370021375 +0000 UTC m=+2.484864691 container create 750facd7728a4051f1d7a46be367007cdedb623e0e04a2fd7a44da494645fe3e (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-20T22:42:51.590 INFO:journalctl@ceph.node-exporter.b.smithi154.stdout:Sep 20 22:42:51 smithi154 podman[63559]: 2024-09-20 22:42:51.588789086 +0000 UTC m=+2.703632407 container init 750facd7728a4051f1d7a46be367007cdedb623e0e04a2fd7a44da494645fe3e (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-20T22:42:51.663 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:42:51.684 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:42:51.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:42:51.685 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:42:51.786 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:42:51.807 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:42:51.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:42:51.808 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:42:51.908 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:42:51.929 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:42:51.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:42:51.930 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:42:52.005 INFO:journalctl@ceph.node-exporter.b.smithi154.stdout:Sep 20 22:42:51 smithi154 podman[63559]: 2024-09-20 22:42:51.591837883 +0000 UTC m=+2.706681211 container start 750facd7728a4051f1d7a46be367007cdedb623e0e04a2fd7a44da494645fe3e (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-20T22:42:52.005 INFO:journalctl@ceph.node-exporter.b.smithi154.stdout:Sep 20 22:42:51 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-node-exporter-b[63821]: ts=2024-09-20T22:42:51.595Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-09-20T22:42:52.005 INFO:journalctl@ceph.node-exporter.b.smithi154.stdout:Sep 20 22:42:51 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-node-exporter-b[63821]: ts=2024-09-20T22:42:51.595Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-09-20T22:42:52.005 INFO:journalctl@ceph.node-exporter.b.smithi154.stdout:Sep 20 22:42:51 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-node-exporter-b[63821]: ts=2024-09-20T22:42:51.595Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-09-20T22:42:52.005 INFO:journalctl@ceph.node-exporter.b.smithi154.stdout:Sep 20 22:42:51 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-node-exporter-b[63821]: ts=2024-09-20T22:42:51.595Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-09-20T22:42:52.005 INFO:journalctl@ceph.node-exporter.b.smithi154.stdout:Sep 20 22:42:51 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-node-exporter-b[63821]: ts=2024-09-20T22:42:51.595Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-09-20T22:42:52.005 INFO:journalctl@ceph.node-exporter.b.smithi154.stdout:Sep 20 22:42:51 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-node-exporter-b[63821]: ts=2024-09-20T22:42:51.595Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-09-20T22:42:52.005 INFO:journalctl@ceph.node-exporter.b.smithi154.stdout:Sep 20 22:42:51 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-node-exporter-b[63821]: ts=2024-09-20T22:42:51.596Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-09-20T22:42:52.005 INFO:journalctl@ceph.node-exporter.b.smithi154.stdout:Sep 20 22:42:51 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-node-exporter-b[63821]: ts=2024-09-20T22:42:51.596Z caller=node_exporter.go:117 level=info collector=arp 2024-09-20T22:42:52.005 INFO:journalctl@ceph.node-exporter.b.smithi154.stdout:Sep 20 22:42:51 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-node-exporter-b[63821]: ts=2024-09-20T22:42:51.596Z caller=node_exporter.go:117 level=info collector=bcache 2024-09-20T22:42:52.005 INFO:journalctl@ceph.node-exporter.b.smithi154.stdout:Sep 20 22:42:51 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-node-exporter-b[63821]: ts=2024-09-20T22:42:51.596Z caller=node_exporter.go:117 level=info collector=bonding 2024-09-20T22:42:52.006 INFO:journalctl@ceph.node-exporter.b.smithi154.stdout:Sep 20 22:42:51 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-node-exporter-b[63821]: ts=2024-09-20T22:42:51.596Z caller=node_exporter.go:117 level=info collector=btrfs 2024-09-20T22:42:52.006 INFO:journalctl@ceph.node-exporter.b.smithi154.stdout:Sep 20 22:42:51 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-node-exporter-b[63821]: ts=2024-09-20T22:42:51.596Z caller=node_exporter.go:117 level=info collector=conntrack 2024-09-20T22:42:52.006 INFO:journalctl@ceph.node-exporter.b.smithi154.stdout:Sep 20 22:42:51 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-node-exporter-b[63821]: ts=2024-09-20T22:42:51.596Z caller=node_exporter.go:117 level=info collector=cpu 2024-09-20T22:42:52.006 INFO:journalctl@ceph.node-exporter.b.smithi154.stdout:Sep 20 22:42:51 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-node-exporter-b[63821]: ts=2024-09-20T22:42:51.596Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-09-20T22:42:52.006 INFO:journalctl@ceph.node-exporter.b.smithi154.stdout:Sep 20 22:42:51 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-node-exporter-b[63821]: ts=2024-09-20T22:42:51.596Z caller=node_exporter.go:117 level=info collector=diskstats 2024-09-20T22:42:52.006 INFO:journalctl@ceph.node-exporter.b.smithi154.stdout:Sep 20 22:42:51 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-node-exporter-b[63821]: ts=2024-09-20T22:42:51.596Z caller=node_exporter.go:117 level=info collector=dmi 2024-09-20T22:42:52.006 INFO:journalctl@ceph.node-exporter.b.smithi154.stdout:Sep 20 22:42:51 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-node-exporter-b[63821]: ts=2024-09-20T22:42:51.596Z caller=node_exporter.go:117 level=info collector=edac 2024-09-20T22:42:52.007 INFO:journalctl@ceph.node-exporter.b.smithi154.stdout:Sep 20 22:42:51 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-node-exporter-b[63821]: ts=2024-09-20T22:42:51.596Z caller=node_exporter.go:117 level=info collector=entropy 2024-09-20T22:42:52.007 INFO:journalctl@ceph.node-exporter.b.smithi154.stdout:Sep 20 22:42:51 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-node-exporter-b[63821]: ts=2024-09-20T22:42:51.596Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-09-20T22:42:52.007 INFO:journalctl@ceph.node-exporter.b.smithi154.stdout:Sep 20 22:42:51 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-node-exporter-b[63821]: ts=2024-09-20T22:42:51.596Z caller=node_exporter.go:117 level=info collector=filefd 2024-09-20T22:42:52.007 INFO:journalctl@ceph.node-exporter.b.smithi154.stdout:Sep 20 22:42:51 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-node-exporter-b[63821]: ts=2024-09-20T22:42:51.596Z caller=node_exporter.go:117 level=info collector=filesystem 2024-09-20T22:42:52.007 INFO:journalctl@ceph.node-exporter.b.smithi154.stdout:Sep 20 22:42:51 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-node-exporter-b[63821]: ts=2024-09-20T22:42:51.596Z caller=node_exporter.go:117 level=info collector=hwmon 2024-09-20T22:42:52.007 INFO:journalctl@ceph.node-exporter.b.smithi154.stdout:Sep 20 22:42:51 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-node-exporter-b[63821]: ts=2024-09-20T22:42:51.596Z caller=node_exporter.go:117 level=info collector=infiniband 2024-09-20T22:42:52.007 INFO:journalctl@ceph.node-exporter.b.smithi154.stdout:Sep 20 22:42:51 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-node-exporter-b[63821]: ts=2024-09-20T22:42:51.596Z caller=node_exporter.go:117 level=info collector=ipvs 2024-09-20T22:42:52.007 INFO:journalctl@ceph.node-exporter.b.smithi154.stdout:Sep 20 22:42:51 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-node-exporter-b[63821]: ts=2024-09-20T22:42:51.596Z caller=node_exporter.go:117 level=info collector=loadavg 2024-09-20T22:42:52.007 INFO:journalctl@ceph.node-exporter.b.smithi154.stdout:Sep 20 22:42:51 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-node-exporter-b[63821]: ts=2024-09-20T22:42:51.596Z caller=node_exporter.go:117 level=info collector=mdadm 2024-09-20T22:42:52.007 INFO:journalctl@ceph.node-exporter.b.smithi154.stdout:Sep 20 22:42:51 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-node-exporter-b[63821]: ts=2024-09-20T22:42:51.596Z caller=node_exporter.go:117 level=info collector=meminfo 2024-09-20T22:42:52.007 INFO:journalctl@ceph.node-exporter.b.smithi154.stdout:Sep 20 22:42:51 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-node-exporter-b[63821]: ts=2024-09-20T22:42:51.596Z caller=node_exporter.go:117 level=info collector=netclass 2024-09-20T22:42:52.007 INFO:journalctl@ceph.node-exporter.b.smithi154.stdout:Sep 20 22:42:51 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-node-exporter-b[63821]: ts=2024-09-20T22:42:51.596Z caller=node_exporter.go:117 level=info collector=netdev 2024-09-20T22:42:52.008 INFO:journalctl@ceph.node-exporter.b.smithi154.stdout:Sep 20 22:42:51 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-node-exporter-b[63821]: ts=2024-09-20T22:42:51.596Z caller=node_exporter.go:117 level=info collector=netstat 2024-09-20T22:42:52.008 INFO:journalctl@ceph.node-exporter.b.smithi154.stdout:Sep 20 22:42:51 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-node-exporter-b[63821]: ts=2024-09-20T22:42:51.596Z caller=node_exporter.go:117 level=info collector=nfs 2024-09-20T22:42:52.008 INFO:journalctl@ceph.node-exporter.b.smithi154.stdout:Sep 20 22:42:51 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-node-exporter-b[63821]: ts=2024-09-20T22:42:51.596Z caller=node_exporter.go:117 level=info collector=nfsd 2024-09-20T22:42:52.008 INFO:journalctl@ceph.node-exporter.b.smithi154.stdout:Sep 20 22:42:51 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-node-exporter-b[63821]: ts=2024-09-20T22:42:51.596Z caller=node_exporter.go:117 level=info collector=nvme 2024-09-20T22:42:52.008 INFO:journalctl@ceph.node-exporter.b.smithi154.stdout:Sep 20 22:42:51 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-node-exporter-b[63821]: ts=2024-09-20T22:42:51.596Z caller=node_exporter.go:117 level=info collector=os 2024-09-20T22:42:52.008 INFO:journalctl@ceph.node-exporter.b.smithi154.stdout:Sep 20 22:42:51 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-node-exporter-b[63821]: ts=2024-09-20T22:42:51.596Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-09-20T22:42:52.008 INFO:journalctl@ceph.node-exporter.b.smithi154.stdout:Sep 20 22:42:51 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-node-exporter-b[63821]: ts=2024-09-20T22:42:51.596Z caller=node_exporter.go:117 level=info collector=pressure 2024-09-20T22:42:52.008 INFO:journalctl@ceph.node-exporter.b.smithi154.stdout:Sep 20 22:42:51 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-node-exporter-b[63821]: ts=2024-09-20T22:42:51.596Z caller=node_exporter.go:117 level=info collector=rapl 2024-09-20T22:42:52.008 INFO:journalctl@ceph.node-exporter.b.smithi154.stdout:Sep 20 22:42:51 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-node-exporter-b[63821]: ts=2024-09-20T22:42:51.596Z caller=node_exporter.go:117 level=info collector=schedstat 2024-09-20T22:42:52.008 INFO:journalctl@ceph.node-exporter.b.smithi154.stdout:Sep 20 22:42:51 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-node-exporter-b[63821]: ts=2024-09-20T22:42:51.596Z caller=node_exporter.go:117 level=info collector=selinux 2024-09-20T22:42:52.008 INFO:journalctl@ceph.node-exporter.b.smithi154.stdout:Sep 20 22:42:51 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-node-exporter-b[63821]: ts=2024-09-20T22:42:51.596Z caller=node_exporter.go:117 level=info collector=sockstat 2024-09-20T22:42:52.008 INFO:journalctl@ceph.node-exporter.b.smithi154.stdout:Sep 20 22:42:51 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-node-exporter-b[63821]: ts=2024-09-20T22:42:51.596Z caller=node_exporter.go:117 level=info collector=softnet 2024-09-20T22:42:52.008 INFO:journalctl@ceph.node-exporter.b.smithi154.stdout:Sep 20 22:42:51 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-node-exporter-b[63821]: ts=2024-09-20T22:42:51.596Z caller=node_exporter.go:117 level=info collector=stat 2024-09-20T22:42:52.008 INFO:journalctl@ceph.node-exporter.b.smithi154.stdout:Sep 20 22:42:51 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-node-exporter-b[63821]: ts=2024-09-20T22:42:51.596Z caller=node_exporter.go:117 level=info collector=tapestats 2024-09-20T22:42:52.009 INFO:journalctl@ceph.node-exporter.b.smithi154.stdout:Sep 20 22:42:51 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-node-exporter-b[63821]: ts=2024-09-20T22:42:51.596Z caller=node_exporter.go:117 level=info collector=textfile 2024-09-20T22:42:52.009 INFO:journalctl@ceph.node-exporter.b.smithi154.stdout:Sep 20 22:42:51 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-node-exporter-b[63821]: ts=2024-09-20T22:42:51.596Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-09-20T22:42:52.009 INFO:journalctl@ceph.node-exporter.b.smithi154.stdout:Sep 20 22:42:51 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-node-exporter-b[63821]: ts=2024-09-20T22:42:51.596Z caller=node_exporter.go:117 level=info collector=time 2024-09-20T22:42:52.009 INFO:journalctl@ceph.node-exporter.b.smithi154.stdout:Sep 20 22:42:51 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-node-exporter-b[63821]: ts=2024-09-20T22:42:51.596Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-09-20T22:42:52.009 INFO:journalctl@ceph.node-exporter.b.smithi154.stdout:Sep 20 22:42:51 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-node-exporter-b[63821]: ts=2024-09-20T22:42:51.596Z caller=node_exporter.go:117 level=info collector=uname 2024-09-20T22:42:52.009 INFO:journalctl@ceph.node-exporter.b.smithi154.stdout:Sep 20 22:42:51 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-node-exporter-b[63821]: ts=2024-09-20T22:42:51.596Z caller=node_exporter.go:117 level=info collector=vmstat 2024-09-20T22:42:52.009 INFO:journalctl@ceph.node-exporter.b.smithi154.stdout:Sep 20 22:42:51 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-node-exporter-b[63821]: ts=2024-09-20T22:42:51.596Z caller=node_exporter.go:117 level=info collector=xfs 2024-09-20T22:42:52.009 INFO:journalctl@ceph.node-exporter.b.smithi154.stdout:Sep 20 22:42:51 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-node-exporter-b[63821]: ts=2024-09-20T22:42:51.596Z caller=node_exporter.go:117 level=info collector=zfs 2024-09-20T22:42:52.009 INFO:journalctl@ceph.node-exporter.b.smithi154.stdout:Sep 20 22:42:51 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-node-exporter-b[63821]: ts=2024-09-20T22:42:51.596Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-09-20T22:42:52.009 INFO:journalctl@ceph.node-exporter.b.smithi154.stdout:Sep 20 22:42:51 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-node-exporter-b[63821]: ts=2024-09-20T22:42:51.596Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-09-20T22:42:52.009 INFO:journalctl@ceph.node-exporter.b.smithi154.stdout:Sep 20 22:42:51 smithi154 bash[63559]: 750facd7728a4051f1d7a46be367007cdedb623e0e04a2fd7a44da494645fe3e 2024-09-20T22:42:52.009 INFO:journalctl@ceph.node-exporter.b.smithi154.stdout:Sep 20 22:42:51 smithi154 systemd[1]: Started Ceph node-exporter.b for a9c0bc70-77a0-11ef-baf5-efdc52797490. 2024-09-20T22:42:52.031 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:42:52.052 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:42:52.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:42:52.053 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:42:52.094 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd pool application enable unique_pool_0 rados --yes-i-really-mean-it || true 2024-09-20T22:42:52.154 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:42:52.176 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:42:52.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:42:52.177 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:42:52.278 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:42:52.299 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:42:52.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:42:52.300 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:42:52.400 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:42:52.427 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:42:52.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:42:52.428 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:42:52.529 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:42:52.550 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:42:52.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:42:52.551 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:42:52.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:52 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/4049064831' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-09-20T22:42:52.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:52 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/4049064831' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-09-20T22:42:52.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:52 smithi084 ceph-mon[40024]: osdmap e94: 8 total, 8 up, 8 in 2024-09-20T22:42:52.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:52 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:42:52.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:52 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:42:52.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:52 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:42:52.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:52 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:42:52.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:52 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:42:52.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:52 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:42:52.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:52 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/4049064831' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-09-20T22:42:52.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:52 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/4049064831' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-09-20T22:42:52.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:52 smithi084 ceph-mon[29774]: osdmap e94: 8 total, 8 up, 8 in 2024-09-20T22:42:52.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:52 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:42:52.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:52 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:42:52.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:52 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:42:52.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:52 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:42:52.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:52 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:42:52.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:52 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:42:52.624 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:42:52.624 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:42:52.651 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:42:52.673 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:42:52.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:42:52.674 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:42:52.711 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:52 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/4049064831' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-09-20T22:42:52.711 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:52 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/4049064831' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-09-20T22:42:52.711 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:52 smithi154 ceph-mon[33264]: osdmap e94: 8 total, 8 up, 8 in 2024-09-20T22:42:52.711 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:52 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:42:52.711 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:52 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:42:52.712 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:52 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:42:52.712 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:52 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:42:52.712 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:52 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:42:52.712 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:52 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:42:52.775 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:42:52.797 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:42:52.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:42:52.798 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:42:52.899 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:42:52.920 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:42:52.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:42:52.921 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:42:53.022 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:42:53.043 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:42:53.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:42:53.044 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:42:53.104 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is set 2024-09-20T22:42:53.145 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:42:53.168 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:42:53.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:42:53.169 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:42:53.270 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:42:53.281 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-20T22:42:53.282 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-20T22:42:53.282 INFO:tasks.thrashosds.thrasher:inject_pause on osd.5 2024-09-20T22:42:53.282 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-20T22:42:53.282 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-20T22:42:53.282 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.5 config set filestore_inject_stall 3 2024-09-20T22:42:53.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:42:53.292 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:42:53.292 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:42:53.393 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:42:53.416 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:42:53.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:42:53.417 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:42:53.443 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-20T22:42:53.517 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:42:53.539 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:42:53.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:42:53.540 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:42:53.641 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:42:53.662 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:42:53.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:42:53.663 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:42:53.676 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:53 smithi084 ceph-mon[40024]: pgmap v42: 158 pgs: 26 unknown, 132 active+clean; 268 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:42:53.676 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:53 smithi084 ceph-mon[40024]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-20T22:42:53.676 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:53 smithi084 ceph-mon[40024]: osdmap e95: 8 total, 8 up, 8 in 2024-09-20T22:42:53.676 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:53 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:42:53.676 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:53 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:42:53.677 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:53 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:42:53.677 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:53 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/943352959' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:42:53.677 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:53 smithi084 ceph-mon[40024]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:42:53.677 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:53 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/943352959' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T22:42:53.677 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:53 smithi084 ceph-mon[40024]: osdmap e96: 8 total, 8 up, 8 in 2024-09-20T22:42:53.677 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:53 smithi084 ceph-mon[29774]: pgmap v42: 158 pgs: 26 unknown, 132 active+clean; 268 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:42:53.677 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:53 smithi084 ceph-mon[29774]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-20T22:42:53.677 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:53 smithi084 ceph-mon[29774]: osdmap e95: 8 total, 8 up, 8 in 2024-09-20T22:42:53.677 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:53 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:42:53.677 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:53 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:42:53.677 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:53 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:42:53.677 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:53 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/943352959' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:42:53.677 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:53 smithi084 ceph-mon[29774]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:42:53.678 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:53 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/943352959' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T22:42:53.678 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:53 smithi084 ceph-mon[29774]: osdmap e96: 8 total, 8 up, 8 in 2024-09-20T22:42:53.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:53 smithi154 ceph-mon[33264]: pgmap v42: 158 pgs: 26 unknown, 132 active+clean; 268 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:42:53.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:53 smithi154 ceph-mon[33264]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-20T22:42:53.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:53 smithi154 ceph-mon[33264]: osdmap e95: 8 total, 8 up, 8 in 2024-09-20T22:42:53.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:53 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:42:53.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:53 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:42:53.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:53 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:42:53.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:53 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/943352959' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:42:53.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:53 smithi154 ceph-mon[33264]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:42:53.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:53 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/943352959' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T22:42:53.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:53 smithi154 ceph-mon[33264]: osdmap e96: 8 total, 8 up, 8 in 2024-09-20T22:42:53.764 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:42:53.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:42:53.787 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:42:53.787 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:42:53.888 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:42:53.909 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:42:53.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:42:53.910 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:42:54.011 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:42:54.032 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:42:54.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:42:54.033 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:42:54.134 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:42:54.156 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:42:54.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:42:54.157 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:42:54.230 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-20T22:42:54.230 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2024-09-20T22:42:54.230 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2024-09-20T22:42:54.230 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-20T22:42:54.230 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:42:54.257 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:42:54.279 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:42:54.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:42:54.279 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:42:54.380 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:42:54.402 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:42:54.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:42:54.403 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:42:54.420 INFO:teuthology.orchestra.run.smithi084.stderr:enabled application 'rados' on pool 'unique_pool_0' 2024-09-20T22:42:54.504 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:42:54.524 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:42:54.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:42:54.525 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:42:54.614 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:54 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/3323026301' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-09-20T22:42:54.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:54 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/3323026301' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-09-20T22:42:54.626 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:42:54.648 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:42:54.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:42:54.649 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:42:54.749 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:42:54.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:54 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/3323026301' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-09-20T22:42:54.771 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:42:54.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:42:54.772 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:42:54.873 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:42:54.894 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:42:54.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:42:54.895 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:42:54.970 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:42:54.996 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:42:55.017 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:42:55.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:42:55.018 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:42:55.118 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:42:55.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:42:55.140 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:42:55.141 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:42:55.241 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:42:55.263 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:42:55.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:42:55.264 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:42:55.364 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:42:55.385 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:42:55.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:42:55.386 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:42:55.487 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:42:55.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:42:55.509 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:42:55.510 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:42:55.610 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:42:55.620 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:55 smithi154 ceph-mon[33264]: pgmap v45: 158 pgs: 158 active+clean; 583 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 329 KiB/s wr, 52 op/s; 380 KiB/s, 1 objects/s recovering 2024-09-20T22:42:55.620 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:55 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/3323026301' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-20T22:42:55.620 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:55 smithi154 ceph-mon[33264]: osdmap e97: 8 total, 8 up, 8 in 2024-09-20T22:42:55.631 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:42:55.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:42:55.632 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:42:55.712 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:55 smithi084 ceph-mon[40024]: pgmap v45: 158 pgs: 158 active+clean; 583 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 329 KiB/s wr, 52 op/s; 380 KiB/s, 1 objects/s recovering 2024-09-20T22:42:55.712 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:55 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/3323026301' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-20T22:42:55.712 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:55 smithi084 ceph-mon[40024]: osdmap e97: 8 total, 8 up, 8 in 2024-09-20T22:42:55.713 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:55 smithi084 ceph-mon[29774]: pgmap v45: 158 pgs: 158 active+clean; 583 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 329 KiB/s wr, 52 op/s; 380 KiB/s, 1 objects/s recovering 2024-09-20T22:42:55.713 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:55 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/3323026301' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-20T22:42:55.713 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:55 smithi084 ceph-mon[29774]: osdmap e97: 8 total, 8 up, 8 in 2024-09-20T22:42:55.733 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:42:55.756 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:42:55.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:42:55.757 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:42:55.761 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-20T22:42:55.761 INFO:teuthology.orchestra.run.smithi154.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-20T22:42:55.761 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-20T22:42:55.858 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:42:55.879 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:42:55.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:42:55.880 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:42:55.981 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:42:55.987 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd unset noscrub 2024-09-20T22:42:56.002 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:42:56.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:42:56.003 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:42:56.104 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:42:56.125 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:42:56.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:42:56.126 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:42:56.226 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:42:56.248 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:42:56.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:42:56.248 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:42:56.306 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.1 dump_blocked_ops 2024-09-20T22:42:56.341 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:42:56.349 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:42:56.372 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:42:56.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:42:56.373 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:42:56.474 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:42:56.495 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:42:56.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:42:56.496 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:42:56.597 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:42:56.602 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:56 smithi084 ceph-mon[40024]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-20T22:42:56.602 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:56 smithi084 ceph-mon[29774]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-20T22:42:56.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:42:56.620 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:42:56.620 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:42:56.721 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:42:56.741 DEBUG:teuthology.orchestra.run.smithi084:> CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 4000 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op copy_from 50 --op write_excl 50 --pool unique_pool_0 2024-09-20T22:42:56.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:42:56.743 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:42:56.743 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:42:56.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:56 smithi154 ceph-mon[33264]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-20T22:42:56.832 INFO:tasks.rados.rados.0.smithi084.stdout:adding op weight read -> 100 2024-09-20T22:42:56.832 INFO:tasks.rados.rados.0.smithi084.stdout:adding op weight write -> 50 2024-09-20T22:42:56.832 INFO:tasks.rados.rados.0.smithi084.stdout:adding op weight delete -> 50 2024-09-20T22:42:56.832 INFO:tasks.rados.rados.0.smithi084.stdout:adding op weight snap_create -> 50 2024-09-20T22:42:56.832 INFO:tasks.rados.rados.0.smithi084.stdout:adding op weight snap_remove -> 50 2024-09-20T22:42:56.832 INFO:tasks.rados.rados.0.smithi084.stdout:adding op weight rollback -> 50 2024-09-20T22:42:56.832 INFO:tasks.rados.rados.0.smithi084.stdout:adding op weight copy_from -> 50 2024-09-20T22:42:56.832 INFO:tasks.rados.rados.0.smithi084.stdout:adding op weight write_excl -> 50 2024-09-20T22:42:56.832 INFO:tasks.rados.rados.0.smithi084.stdout:ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable) 2024-09-20T22:42:56.833 INFO:tasks.rados.rados.0.smithi084.stdout:Configuration: 2024-09-20T22:42:56.833 INFO:tasks.rados.rados.0.smithi084.stdout: Number of operations: 4000 2024-09-20T22:42:56.833 INFO:tasks.rados.rados.0.smithi084.stdout: Number of objects: 50 2024-09-20T22:42:56.833 INFO:tasks.rados.rados.0.smithi084.stdout: Max in flight operations: 16 2024-09-20T22:42:56.833 INFO:tasks.rados.rados.0.smithi084.stdout: Object size (in bytes): 4000000 2024-09-20T22:42:56.833 INFO:tasks.rados.rados.0.smithi084.stdout: Write stride min: 400000 2024-09-20T22:42:56.833 INFO:tasks.rados.rados.0.smithi084.stdout: Write stride max: 800000 2024-09-20T22:42:56.843 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:42:56.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:42:56.865 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:42:56.866 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:42:56.967 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:42:56.988 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:42:56.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:42:56.989 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:42:57.090 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:42:57.112 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:42:57.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:42:57.113 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:42:57.196 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:42:57.196 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:42:57.211 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:42:57.211 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:42:57.214 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:42:57.227 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:42:57.227 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:42:57.235 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:42:57.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:42:57.236 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:42:57.243 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-20T22:42:57.243 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T22:42:57.260 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:42:57.260 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:42:57.277 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:42:57.277 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:42:57.293 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:42:57.293 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:42:57.310 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:42:57.311 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:42:57.337 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:42:57.358 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:42:57.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:42:57.359 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:42:57.460 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:42:57.478 INFO:tasks.rados.rados.0.smithi084.stdout:1: write initial oid 1 2024-09-20T22:42:57.478 INFO:tasks.rados.rados.0.smithi084.stdout:1: seq_num 0 ranges {416807=475249,1542129=474678} 2024-09-20T22:42:57.483 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:42:57.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:42:57.485 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:42:57.485 INFO:tasks.rados.rados.0.smithi084.stdout:1: writing smithi08477203-1 from 416807 to 892056 tid 1 2024-09-20T22:42:57.491 INFO:tasks.rados.rados.0.smithi084.stdout:1: writing smithi08477203-1 from 1542129 to 2016807 tid 2 2024-09-20T22:42:57.491 INFO:tasks.rados.rados.0.smithi084.stdout:2: write initial oid 2 2024-09-20T22:42:57.491 INFO:tasks.rados.rados.0.smithi084.stdout:2: seq_num 1 ranges {416807=475249,1542129=474678} 2024-09-20T22:42:57.496 INFO:tasks.rados.rados.0.smithi084.stdout:2: writing smithi08477203-2 from 416807 to 892056 tid 1 2024-09-20T22:42:57.503 INFO:tasks.rados.rados.0.smithi084.stdout:2: writing smithi08477203-2 from 1542129 to 2016807 tid 2 2024-09-20T22:42:57.503 INFO:tasks.rados.rados.0.smithi084.stdout:3: write initial oid 3 2024-09-20T22:42:57.503 INFO:tasks.rados.rados.0.smithi084.stdout:3: seq_num 2 ranges {433614=550498,1600611=433003} 2024-09-20T22:42:57.508 INFO:tasks.rados.rados.0.smithi084.stdout:3: writing smithi08477203-3 from 433614 to 984112 tid 1 2024-09-20T22:42:57.515 INFO:tasks.rados.rados.0.smithi084.stdout:3: writing smithi08477203-3 from 1600611 to 2033614 tid 2 2024-09-20T22:42:57.515 INFO:tasks.rados.rados.0.smithi084.stdout:4: write initial oid 4 2024-09-20T22:42:57.515 INFO:tasks.rados.rados.0.smithi084.stdout:4: seq_num 3 ranges {450421=625747,1659093=391328} 2024-09-20T22:42:57.522 INFO:tasks.rados.rados.0.smithi084.stdout:4: writing smithi08477203-4 from 450421 to 1076168 tid 1 2024-09-20T22:42:57.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:57 smithi154 ceph-mon[33264]: pgmap v47: 158 pgs: 158 active+clean; 583 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 39 KiB/s rd, 309 KiB/s wr, 49 op/s; 357 KiB/s, 1 objects/s recovering 2024-09-20T22:42:57.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:57 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/3101266294' entity='client.0' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "write_fadvise_dontneed", "val": "false"}]: dispatch 2024-09-20T22:42:57.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:57 smithi154 ceph-mon[33264]: from='client.? ' entity='client.0' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "write_fadvise_dontneed", "val": "false"}]: dispatch 2024-09-20T22:42:57.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:57 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/1113134392' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:42:57.524 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:57 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:42:57.525 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:57 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:42:57.526 INFO:tasks.rados.rados.0.smithi084.stdout:4: writing smithi08477203-4 from 1659093 to 2050421 tid 2 2024-09-20T22:42:57.526 INFO:tasks.rados.rados.0.smithi084.stdout:5: write initial oid 5 2024-09-20T22:42:57.527 INFO:tasks.rados.rados.0.smithi084.stdout:5: seq_num 4 ranges {467228=700996,1717575=349653} 2024-09-20T22:42:57.533 INFO:tasks.rados.rados.0.smithi084.stdout:5: writing smithi08477203-5 from 467228 to 1168224 tid 1 2024-09-20T22:42:57.538 INFO:tasks.rados.rados.0.smithi084.stdout:5: writing smithi08477203-5 from 1717575 to 2067228 tid 2 2024-09-20T22:42:57.538 INFO:tasks.rados.rados.0.smithi084.stdout:6: write initial oid 6 2024-09-20T22:42:57.538 INFO:tasks.rados.rados.0.smithi084.stdout:6: seq_num 5 ranges {484035=776245,2059704=24331} 2024-09-20T22:42:57.545 INFO:tasks.rados.rados.0.smithi084.stdout:6: writing smithi08477203-6 from 484035 to 1260280 tid 1 2024-09-20T22:42:57.549 INFO:tasks.rados.rados.0.smithi084.stdout:6: writing smithi08477203-6 from 2059704 to 2084035 tid 2 2024-09-20T22:42:57.550 INFO:tasks.rados.rados.0.smithi084.stdout:7: write initial oid 7 2024-09-20T22:42:57.550 INFO:tasks.rados.rados.0.smithi084.stdout:7: seq_num 6 ranges {500842=451494,1718186=382656} 2024-09-20T22:42:57.554 INFO:tasks.rados.rados.0.smithi084.stdout:7: writing smithi08477203-7 from 500842 to 952336 tid 1 2024-09-20T22:42:57.561 INFO:tasks.rados.rados.0.smithi084.stdout:7: writing smithi08477203-7 from 1718186 to 2100842 tid 2 2024-09-20T22:42:57.561 INFO:tasks.rados.rados.0.smithi084.stdout:8: write initial oid 8 2024-09-20T22:42:57.561 INFO:tasks.rados.rados.0.smithi084.stdout:8: seq_num 7 ranges {517649=526743,1776668=340981} 2024-09-20T22:42:57.567 INFO:tasks.rados.rados.0.smithi084.stdout:8: writing smithi08477203-8 from 517649 to 1044392 tid 1 2024-09-20T22:42:57.573 INFO:tasks.rados.rados.0.smithi084.stdout:8: writing smithi08477203-8 from 1776668 to 2117649 tid 2 2024-09-20T22:42:57.573 INFO:tasks.rados.rados.0.smithi084.stdout:9: write initial oid 9 2024-09-20T22:42:57.573 INFO:tasks.rados.rados.0.smithi084.stdout:1: finishing write tid 1 to smithi08477203-1 2024-09-20T22:42:57.573 INFO:tasks.rados.rados.0.smithi084.stdout:1: finishing write tid 2 to smithi08477203-1 2024-09-20T22:42:57.573 INFO:tasks.rados.rados.0.smithi084.stdout:1: finishing write tid 3 to smithi08477203-1 2024-09-20T22:42:57.573 INFO:tasks.rados.rados.0.smithi084.stdout:9: seq_num 8 ranges {534456=718345,1951503=182953} 2024-09-20T22:42:57.580 INFO:tasks.rados.rados.0.smithi084.stdout:9: writing smithi08477203-9 from 534456 to 1252801 tid 1 2024-09-20T22:42:57.585 INFO:tasks.rados.rados.0.smithi084.stdout:9: writing smithi08477203-9 from 1951503 to 2134456 tid 2 2024-09-20T22:42:57.585 INFO:tasks.rados.rados.0.smithi084.stdout:10: write initial oid 10 2024-09-20T22:42:57.585 INFO:tasks.rados.rados.0.smithi084.stdout:10: seq_num 9 ranges {551263=793594,1893632=257631} 2024-09-20T22:42:57.585 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:42:57.592 INFO:tasks.rados.rados.0.smithi084.stdout:10: writing smithi08477203-10 from 551263 to 1344857 tid 1 2024-09-20T22:42:57.597 INFO:tasks.rados.rados.0.smithi084.stdout:10: writing smithi08477203-10 from 1893632 to 2151263 tid 2 2024-09-20T22:42:57.597 INFO:tasks.rados.rados.0.smithi084.stdout:11: write initial oid 11 2024-09-20T22:42:57.597 INFO:tasks.rados.rados.0.smithi084.stdout:11: seq_num 10 ranges {568070=468843,1552114=615956} 2024-09-20T22:42:57.602 INFO:tasks.rados.rados.0.smithi084.stdout:11: writing smithi08477203-11 from 568070 to 1036913 tid 1 2024-09-20T22:42:57.607 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:42:57.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:42:57.607 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:42:57.608 INFO:tasks.rados.rados.0.smithi084.stdout:11: writing smithi08477203-11 from 1552114 to 2168070 tid 2 2024-09-20T22:42:57.608 INFO:tasks.rados.rados.0.smithi084.stdout:12: write initial oid 12 2024-09-20T22:42:57.608 INFO:tasks.rados.rados.0.smithi084.stdout:12: seq_num 11 ranges {584877=544092,1610596=562003} 2024-09-20T22:42:57.615 INFO:tasks.rados.rados.0.smithi084.stdout:12: writing smithi08477203-12 from 584877 to 1128969 tid 1 2024-09-20T22:42:57.621 INFO:tasks.rados.rados.0.smithi084.stdout:12: writing smithi08477203-12 from 1610596 to 2172599 tid 2 2024-09-20T22:42:57.621 INFO:tasks.rados.rados.0.smithi084.stdout:13: write initial oid 13 2024-09-20T22:42:57.621 INFO:tasks.rados.rados.0.smithi084.stdout:13: seq_num 12 ranges {601684=619341,1669078=532606} 2024-09-20T22:42:57.628 INFO:tasks.rados.rados.0.smithi084.stdout:13: writing smithi08477203-13 from 601684 to 1221025 tid 1 2024-09-20T22:42:57.634 INFO:tasks.rados.rados.0.smithi084.stdout:13: writing smithi08477203-13 from 1669078 to 2201684 tid 2 2024-09-20T22:42:57.634 INFO:tasks.rados.rados.0.smithi084.stdout:14: write initial oid 14 2024-09-20T22:42:57.634 INFO:tasks.rados.rados.0.smithi084.stdout:14: seq_num 13 ranges {618491=694590,2011207=207284} 2024-09-20T22:42:57.641 INFO:tasks.rados.rados.0.smithi084.stdout:14: writing smithi08477203-14 from 618491 to 1313081 tid 1 2024-09-20T22:42:57.646 INFO:tasks.rados.rados.0.smithi084.stdout:14: writing smithi08477203-14 from 2011207 to 2218491 tid 2 2024-09-20T22:42:57.646 INFO:tasks.rados.rados.0.smithi084.stdout:15: write initial oid 15 2024-09-20T22:42:57.646 INFO:tasks.rados.rados.0.smithi084.stdout:15: seq_num 14 ranges {635298=769839,2069689=165609} 2024-09-20T22:42:57.655 INFO:tasks.rados.rados.0.smithi084.stdout:15: writing smithi08477203-15 from 635298 to 1405137 tid 1 2024-09-20T22:42:57.660 INFO:tasks.rados.rados.0.smithi084.stdout:15: writing smithi08477203-15 from 2069689 to 2235298 tid 2 2024-09-20T22:42:57.660 INFO:tasks.rados.rados.0.smithi084.stdout:16: write initial oid 16 2024-09-20T22:42:57.660 INFO:tasks.rados.rados.0.smithi084.stdout:16: seq_num 15 ranges {652105=445088,1728171=452988} 2024-09-20T22:42:57.667 INFO:tasks.rados.rados.0.smithi084.stdout:16: writing smithi08477203-16 from 652105 to 1097193 tid 1 2024-09-20T22:42:57.674 INFO:tasks.rados.rados.0.smithi084.stdout:16: writing smithi08477203-16 from 1728171 to 2181159 tid 2 2024-09-20T22:42:57.674 INFO:tasks.rados.rados.0.smithi084.stdout: waiting on 16 2024-09-20T22:42:57.674 INFO:tasks.rados.rados.0.smithi084.stdout:1: finishing write tid 4 to smithi08477203-1 2024-09-20T22:42:57.674 INFO:tasks.rados.rados.0.smithi084.stdout:1: oid 1 updating version 0 to 1 2024-09-20T22:42:57.675 INFO:tasks.rados.rados.0.smithi084.stdout:1: oid 1 updating version 1 to 2 2024-09-20T22:42:57.675 INFO:tasks.rados.rados.0.smithi084.stdout:1: oid 1 updating version 2 to 3 2024-09-20T22:42:57.675 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 1 v 3 (ObjNum 0 snap 0 seq_num 0) dirty exists 2024-09-20T22:42:57.675 INFO:tasks.rados.rados.0.smithi084.stdout:1: left oid 1 (ObjNum 0 snap 0 seq_num 0) 2024-09-20T22:42:57.675 INFO:tasks.rados.rados.0.smithi084.stdout:2: finishing write tid 1 to smithi08477203-2 2024-09-20T22:42:57.675 INFO:tasks.rados.rados.0.smithi084.stdout:2: finishing write tid 2 to smithi08477203-2 2024-09-20T22:42:57.675 INFO:tasks.rados.rados.0.smithi084.stdout:2: finishing write tid 3 to smithi08477203-2 2024-09-20T22:42:57.675 INFO:tasks.rados.rados.0.smithi084.stdout:2: finishing write tid 4 to smithi08477203-2 2024-09-20T22:42:57.675 INFO:tasks.rados.rados.0.smithi084.stdout:2: oid 2 updating version 0 to 2 2024-09-20T22:42:57.675 INFO:tasks.rados.rados.0.smithi084.stdout:2: oid 2 updating version 2 to 3 2024-09-20T22:42:57.675 INFO:tasks.rados.rados.0.smithi084.stdout:2: oid 2 version 3 is already newer than 1 2024-09-20T22:42:57.675 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 2 v 3 (ObjNum 1 snap 0 seq_num 1) dirty exists 2024-09-20T22:42:57.675 INFO:tasks.rados.rados.0.smithi084.stdout:2: left oid 2 (ObjNum 1 snap 0 seq_num 1) 2024-09-20T22:42:57.675 INFO:tasks.rados.rados.0.smithi084.stdout:3: finishing write tid 1 to smithi08477203-3 2024-09-20T22:42:57.675 INFO:tasks.rados.rados.0.smithi084.stdout:3: finishing write tid 2 to smithi08477203-3 2024-09-20T22:42:57.676 INFO:tasks.rados.rados.0.smithi084.stdout:1: done (15 left) 2024-09-20T22:42:57.676 INFO:tasks.rados.rados.0.smithi084.stdout:2: done (14 left) 2024-09-20T22:42:57.676 INFO:tasks.rados.rados.0.smithi084.stdout:17: write initial oid 17 2024-09-20T22:42:57.676 INFO:tasks.rados.rados.0.smithi084.stdout:17: seq_num 16 ranges {668912=636690,1903006=365906} 2024-09-20T22:42:57.676 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:42:57.681 INFO:tasks.rados.rados.0.smithi084.stdout:17: writing smithi08477203-17 from 668912 to 1305602 tid 1 2024-09-20T22:42:57.687 INFO:tasks.rados.rados.0.smithi084.stdout:17: writing smithi08477203-17 from 1903006 to 2268912 tid 2 2024-09-20T22:42:57.687 INFO:tasks.rados.rados.0.smithi084.stdout:18: write initial oid 18 2024-09-20T22:42:57.687 INFO:tasks.rados.rados.0.smithi084.stdout:18: seq_num 17 ranges {685719=711939,1845135=440584} 2024-09-20T22:42:57.696 INFO:tasks.rados.rados.0.smithi084.stdout:18: writing smithi08477203-18 from 685719 to 1397658 tid 1 2024-09-20T22:42:57.702 INFO:tasks.rados.rados.0.smithi084.stdout:18: writing smithi08477203-18 from 1845135 to 2285719 tid 2 2024-09-20T22:42:57.702 INFO:tasks.rados.rados.0.smithi084.stdout: waiting on 16 2024-09-20T22:42:57.702 INFO:tasks.rados.rados.0.smithi084.stdout:3: finishing write tid 3 to smithi08477203-3 2024-09-20T22:42:57.703 INFO:tasks.rados.rados.0.smithi084.stdout:3: finishing write tid 4 to smithi08477203-3 2024-09-20T22:42:57.703 INFO:tasks.rados.rados.0.smithi084.stdout:3: oid 3 updating version 0 to 1 2024-09-20T22:42:57.703 INFO:tasks.rados.rados.0.smithi084.stdout:3: oid 3 updating version 1 to 3 2024-09-20T22:42:57.703 INFO:tasks.rados.rados.0.smithi084.stdout:3: oid 3 version 3 is already newer than 2 2024-09-20T22:42:57.703 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 3 v 3 (ObjNum 2 snap 0 seq_num 2) dirty exists 2024-09-20T22:42:57.703 INFO:tasks.rados.rados.0.smithi084.stdout:3: left oid 3 (ObjNum 2 snap 0 seq_num 2) 2024-09-20T22:42:57.703 INFO:tasks.rados.rados.0.smithi084.stdout:4: finishing write tid 1 to smithi08477203-4 2024-09-20T22:42:57.703 INFO:tasks.rados.rados.0.smithi084.stdout:3: done (15 left) 2024-09-20T22:42:57.704 INFO:tasks.rados.rados.0.smithi084.stdout:19: write initial oid 19 2024-09-20T22:42:57.704 INFO:tasks.rados.rados.0.smithi084.stdout:19: seq_num 18 ranges {702526=787188,1903617=398909} 2024-09-20T22:42:57.708 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:42:57.710 INFO:tasks.rados.rados.0.smithi084.stdout:19: writing smithi08477203-19 from 702526 to 1489714 tid 1 2024-09-20T22:42:57.715 INFO:tasks.rados.rados.0.smithi084.stdout:19: writing smithi08477203-19 from 1903617 to 2302526 tid 2 2024-09-20T22:42:57.715 INFO:tasks.rados.rados.0.smithi084.stdout: waiting on 16 2024-09-20T22:42:57.715 INFO:tasks.rados.rados.0.smithi084.stdout:4: finishing write tid 2 to smithi08477203-4 2024-09-20T22:42:57.715 INFO:tasks.rados.rados.0.smithi084.stdout:4: finishing write tid 3 to smithi08477203-4 2024-09-20T22:42:57.715 INFO:tasks.rados.rados.0.smithi084.stdout:4: finishing write tid 4 to smithi08477203-4 2024-09-20T22:42:57.715 INFO:tasks.rados.rados.0.smithi084.stdout:4: oid 4 updating version 0 to 1 2024-09-20T22:42:57.715 INFO:tasks.rados.rados.0.smithi084.stdout:4: oid 4 updating version 1 to 2 2024-09-20T22:42:57.716 INFO:tasks.rados.rados.0.smithi084.stdout:4: oid 4 updating version 2 to 3 2024-09-20T22:42:57.716 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 4 v 3 (ObjNum 3 snap 0 seq_num 3) dirty exists 2024-09-20T22:42:57.716 INFO:tasks.rados.rados.0.smithi084.stdout:4: left oid 4 (ObjNum 3 snap 0 seq_num 3) 2024-09-20T22:42:57.716 INFO:tasks.rados.rados.0.smithi084.stdout:5: finishing write tid 1 to smithi08477203-5 2024-09-20T22:42:57.716 INFO:tasks.rados.rados.0.smithi084.stdout:4: done (15 left) 2024-09-20T22:42:57.716 INFO:tasks.rados.rados.0.smithi084.stdout:20: write initial oid 20 2024-09-20T22:42:57.716 INFO:tasks.rados.rados.0.smithi084.stdout:20: seq_num 19 ranges {719333=462437,1962099=357234} 2024-09-20T22:42:57.721 INFO:tasks.rados.rados.0.smithi084.stdout:20: writing smithi08477203-20 from 719333 to 1181770 tid 1 2024-09-20T22:42:57.728 INFO:tasks.rados.rados.0.smithi084.stdout:20: writing smithi08477203-20 from 1962099 to 2319333 tid 2 2024-09-20T22:42:57.728 INFO:tasks.rados.rados.0.smithi084.stdout: waiting on 16 2024-09-20T22:42:57.728 INFO:tasks.rados.rados.0.smithi084.stdout:5: finishing write tid 2 to smithi08477203-5 2024-09-20T22:42:57.728 INFO:tasks.rados.rados.0.smithi084.stdout:5: finishing write tid 3 to smithi08477203-5 2024-09-20T22:42:57.728 INFO:tasks.rados.rados.0.smithi084.stdout:5: finishing write tid 4 to smithi08477203-5 2024-09-20T22:42:57.728 INFO:tasks.rados.rados.0.smithi084.stdout:5: oid 5 updating version 0 to 2 2024-09-20T22:42:57.728 INFO:tasks.rados.rados.0.smithi084.stdout:5: oid 5 version 2 is already newer than 1 2024-09-20T22:42:57.728 INFO:tasks.rados.rados.0.smithi084.stdout:5: oid 5 updating version 2 to 3 2024-09-20T22:42:57.728 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 5 v 3 (ObjNum 4 snap 0 seq_num 4) dirty exists 2024-09-20T22:42:57.728 INFO:tasks.rados.rados.0.smithi084.stdout:5: left oid 5 (ObjNum 4 snap 0 seq_num 4) 2024-09-20T22:42:57.729 INFO:tasks.rados.rados.0.smithi084.stdout:6: finishing write tid 1 to smithi08477203-6 2024-09-20T22:42:57.729 INFO:tasks.rados.rados.0.smithi084.stdout:5: done (15 left) 2024-09-20T22:42:57.729 INFO:tasks.rados.rados.0.smithi084.stdout:21: write initial oid 21 2024-09-20T22:42:57.729 INFO:tasks.rados.rados.0.smithi084.stdout:21: seq_num 20 ranges {736140=537686,2020581=315559} 2024-09-20T22:42:57.731 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:42:57.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:42:57.731 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:42:57.735 INFO:tasks.rados.rados.0.smithi084.stdout:21: writing smithi08477203-21 from 736140 to 1273826 tid 1 2024-09-20T22:42:57.741 INFO:tasks.rados.rados.0.smithi084.stdout:21: writing smithi08477203-21 from 2020581 to 2336140 tid 2 2024-09-20T22:42:57.741 INFO:tasks.rados.rados.0.smithi084.stdout: waiting on 16 2024-09-20T22:42:57.741 INFO:tasks.rados.rados.0.smithi084.stdout:6: finishing write tid 2 to smithi08477203-6 2024-09-20T22:42:57.741 INFO:tasks.rados.rados.0.smithi084.stdout:6: finishing write tid 3 to smithi08477203-6 2024-09-20T22:42:57.741 INFO:tasks.rados.rados.0.smithi084.stdout:6: finishing write tid 4 to smithi08477203-6 2024-09-20T22:42:57.741 INFO:tasks.rados.rados.0.smithi084.stdout:6: oid 6 updating version 0 to 1 2024-09-20T22:42:57.741 INFO:tasks.rados.rados.0.smithi084.stdout:6: oid 6 updating version 1 to 2 2024-09-20T22:42:57.741 INFO:tasks.rados.rados.0.smithi084.stdout:6: oid 6 updating version 2 to 3 2024-09-20T22:42:57.741 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 6 v 3 (ObjNum 5 snap 0 seq_num 5) dirty exists 2024-09-20T22:42:57.741 INFO:tasks.rados.rados.0.smithi084.stdout:6: left oid 6 (ObjNum 5 snap 0 seq_num 5) 2024-09-20T22:42:57.742 INFO:tasks.rados.rados.0.smithi084.stdout:7: finishing write tid 1 to smithi08477203-7 2024-09-20T22:42:57.742 INFO:tasks.rados.rados.0.smithi084.stdout:6: done (15 left) 2024-09-20T22:42:57.742 INFO:tasks.rados.rados.0.smithi084.stdout:22: write initial oid 22 2024-09-20T22:42:57.742 INFO:tasks.rados.rados.0.smithi084.stdout:22: seq_num 21 ranges {752947=612935,1962710=390237} 2024-09-20T22:42:57.748 INFO:tasks.rados.rados.0.smithi084.stdout:22: writing smithi08477203-22 from 752947 to 1365882 tid 1 2024-09-20T22:42:57.754 INFO:tasks.rados.rados.0.smithi084.stdout:22: writing smithi08477203-22 from 1962710 to 2352947 tid 2 2024-09-20T22:42:57.754 INFO:tasks.rados.rados.0.smithi084.stdout: waiting on 16 2024-09-20T22:42:57.754 INFO:tasks.rados.rados.0.smithi084.stdout:7: finishing write tid 2 to smithi08477203-7 2024-09-20T22:42:57.754 INFO:tasks.rados.rados.0.smithi084.stdout:7: finishing write tid 3 to smithi08477203-7 2024-09-20T22:42:57.754 INFO:tasks.rados.rados.0.smithi084.stdout:7: finishing write tid 4 to smithi08477203-7 2024-09-20T22:42:57.754 INFO:tasks.rados.rados.0.smithi084.stdout:7: oid 7 updating version 0 to 2 2024-09-20T22:42:57.754 INFO:tasks.rados.rados.0.smithi084.stdout:7: oid 7 version 2 is already newer than 1 2024-09-20T22:42:57.754 INFO:tasks.rados.rados.0.smithi084.stdout:7: oid 7 updating version 2 to 3 2024-09-20T22:42:57.754 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 7 v 3 (ObjNum 6 snap 0 seq_num 6) dirty exists 2024-09-20T22:42:57.755 INFO:tasks.rados.rados.0.smithi084.stdout:7: left oid 7 (ObjNum 6 snap 0 seq_num 6) 2024-09-20T22:42:57.755 INFO:tasks.rados.rados.0.smithi084.stdout:8: finishing write tid 1 to smithi08477203-8 2024-09-20T22:42:57.755 INFO:tasks.rados.rados.0.smithi084.stdout:7: done (15 left) 2024-09-20T22:42:57.755 INFO:tasks.rados.rados.0.smithi084.stdout:23: write initial oid 23 2024-09-20T22:42:57.755 INFO:tasks.rados.rados.0.smithi084.stdout:23: seq_num 22 ranges {769754=688184,2021192=348562} 2024-09-20T22:42:57.761 INFO:tasks.rados.rados.0.smithi084.stdout:23: writing smithi08477203-23 from 769754 to 1457938 tid 1 2024-09-20T22:42:57.767 INFO:tasks.rados.rados.0.smithi084.stdout:23: writing smithi08477203-23 from 2021192 to 2369754 tid 2 2024-09-20T22:42:57.767 INFO:tasks.rados.rados.0.smithi084.stdout: waiting on 16 2024-09-20T22:42:57.767 INFO:tasks.rados.rados.0.smithi084.stdout:8: finishing write tid 2 to smithi08477203-8 2024-09-20T22:42:57.767 INFO:tasks.rados.rados.0.smithi084.stdout:8: finishing write tid 3 to smithi08477203-8 2024-09-20T22:42:57.767 INFO:tasks.rados.rados.0.smithi084.stdout:8: finishing write tid 4 to smithi08477203-8 2024-09-20T22:42:57.767 INFO:tasks.rados.rados.0.smithi084.stdout:8: oid 8 updating version 0 to 4 2024-09-20T22:42:57.767 INFO:tasks.rados.rados.0.smithi084.stdout:8: oid 8 updating version 4 to 6 2024-09-20T22:42:57.767 INFO:tasks.rados.rados.0.smithi084.stdout:8: oid 8 version 6 is already newer than 5 2024-09-20T22:42:57.767 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 8 v 6 (ObjNum 7 snap 0 seq_num 7) dirty exists 2024-09-20T22:42:57.767 INFO:tasks.rados.rados.0.smithi084.stdout:8: left oid 8 (ObjNum 7 snap 0 seq_num 7) 2024-09-20T22:42:57.768 INFO:tasks.rados.rados.0.smithi084.stdout:9: finishing write tid 1 to smithi08477203-9 2024-09-20T22:42:57.768 INFO:tasks.rados.rados.0.smithi084.stdout:8: done (15 left) 2024-09-20T22:42:57.768 INFO:tasks.rados.rados.0.smithi084.stdout:24: write initial oid 24 2024-09-20T22:42:57.768 INFO:tasks.rados.rados.0.smithi084.stdout:24: seq_num 23 ranges {786561=479786,1796027=467664} 2024-09-20T22:42:57.773 INFO:tasks.rados.rados.0.smithi084.stdout:24: writing smithi08477203-24 from 786561 to 1266347 tid 1 2024-09-20T22:42:57.780 INFO:tasks.rados.rados.0.smithi084.stdout:24: writing smithi08477203-24 from 1796027 to 2263691 tid 2 2024-09-20T22:42:57.780 INFO:tasks.rados.rados.0.smithi084.stdout: waiting on 16 2024-09-20T22:42:57.780 INFO:tasks.rados.rados.0.smithi084.stdout:9: finishing write tid 2 to smithi08477203-9 2024-09-20T22:42:57.780 INFO:tasks.rados.rados.0.smithi084.stdout:9: finishing write tid 3 to smithi08477203-9 2024-09-20T22:42:57.780 INFO:tasks.rados.rados.0.smithi084.stdout:9: finishing write tid 4 to smithi08477203-9 2024-09-20T22:42:57.780 INFO:tasks.rados.rados.0.smithi084.stdout:9: oid 9 updating version 0 to 4 2024-09-20T22:42:57.780 INFO:tasks.rados.rados.0.smithi084.stdout:9: oid 9 updating version 4 to 5 2024-09-20T22:42:57.780 INFO:tasks.rados.rados.0.smithi084.stdout:9: oid 9 updating version 5 to 6 2024-09-20T22:42:57.780 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 9 v 6 (ObjNum 8 snap 0 seq_num 8) dirty exists 2024-09-20T22:42:57.780 INFO:tasks.rados.rados.0.smithi084.stdout:9: left oid 9 (ObjNum 8 snap 0 seq_num 8) 2024-09-20T22:42:57.780 INFO:tasks.rados.rados.0.smithi084.stdout:10: finishing write tid 1 to smithi08477203-10 2024-09-20T22:42:57.781 INFO:tasks.rados.rados.0.smithi084.stdout:9: done (15 left) 2024-09-20T22:42:57.781 INFO:tasks.rados.rados.0.smithi084.stdout:25: write initial oid 25 2024-09-20T22:42:57.781 INFO:tasks.rados.rados.0.smithi084.stdout:25: seq_num 24 ranges {403368=555035,1454509=727675} 2024-09-20T22:42:57.785 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:57 smithi084 ceph-mon[40024]: pgmap v47: 158 pgs: 158 active+clean; 583 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 39 KiB/s rd, 309 KiB/s wr, 49 op/s; 357 KiB/s, 1 objects/s recovering 2024-09-20T22:42:57.786 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:57 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/3101266294' entity='client.0' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "write_fadvise_dontneed", "val": "false"}]: dispatch 2024-09-20T22:42:57.786 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:57 smithi084 ceph-mon[40024]: from='client.? ' entity='client.0' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "write_fadvise_dontneed", "val": "false"}]: dispatch 2024-09-20T22:42:57.786 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:57 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/1113134392' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:42:57.786 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:57 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:42:57.786 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:57 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:42:57.786 INFO:tasks.rados.rados.0.smithi084.stdout:25: writing smithi08477203-25 from 403368 to 958403 tid 1 2024-09-20T22:42:57.786 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:57 smithi084 ceph-mon[29774]: pgmap v47: 158 pgs: 158 active+clean; 583 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 39 KiB/s rd, 309 KiB/s wr, 49 op/s; 357 KiB/s, 1 objects/s recovering 2024-09-20T22:42:57.786 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:57 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/3101266294' entity='client.0' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "write_fadvise_dontneed", "val": "false"}]: dispatch 2024-09-20T22:42:57.786 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:57 smithi084 ceph-mon[29774]: from='client.? ' entity='client.0' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "write_fadvise_dontneed", "val": "false"}]: dispatch 2024-09-20T22:42:57.787 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:57 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/1113134392' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:42:57.787 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:57 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:42:57.787 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:57 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:42:57.793 INFO:tasks.rados.rados.0.smithi084.stdout:25: writing smithi08477203-25 from 1454509 to 2182184 tid 2 2024-09-20T22:42:57.793 INFO:tasks.rados.rados.0.smithi084.stdout: waiting on 16 2024-09-20T22:42:57.793 INFO:tasks.rados.rados.0.smithi084.stdout:10: finishing write tid 2 to smithi08477203-10 2024-09-20T22:42:57.793 INFO:tasks.rados.rados.0.smithi084.stdout:10: finishing write tid 3 to smithi08477203-10 2024-09-20T22:42:57.793 INFO:tasks.rados.rados.0.smithi084.stdout:10: finishing write tid 4 to smithi08477203-10 2024-09-20T22:42:57.793 INFO:tasks.rados.rados.0.smithi084.stdout:10: oid 10 updating version 0 to 7 2024-09-20T22:42:57.794 INFO:tasks.rados.rados.0.smithi084.stdout:10: oid 10 updating version 7 to 8 2024-09-20T22:42:57.794 INFO:tasks.rados.rados.0.smithi084.stdout:10: oid 10 updating version 8 to 9 2024-09-20T22:42:57.794 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 10 v 9 (ObjNum 9 snap 0 seq_num 9) dirty exists 2024-09-20T22:42:57.794 INFO:tasks.rados.rados.0.smithi084.stdout:10: left oid 10 (ObjNum 9 snap 0 seq_num 9) 2024-09-20T22:42:57.794 INFO:tasks.rados.rados.0.smithi084.stdout:11: finishing write tid 1 to smithi08477203-11 2024-09-20T22:42:57.794 INFO:tasks.rados.rados.0.smithi084.stdout:11: finishing write tid 2 to smithi08477203-11 2024-09-20T22:42:57.794 INFO:tasks.rados.rados.0.smithi084.stdout:11: finishing write tid 3 to smithi08477203-11 2024-09-20T22:42:57.794 INFO:tasks.rados.rados.0.smithi084.stdout:11: finishing write tid 4 to smithi08477203-11 2024-09-20T22:42:57.794 INFO:tasks.rados.rados.0.smithi084.stdout:11: oid 11 updating version 0 to 5 2024-09-20T22:42:57.794 INFO:tasks.rados.rados.0.smithi084.stdout:11: oid 11 updating version 5 to 6 2024-09-20T22:42:57.794 INFO:tasks.rados.rados.0.smithi084.stdout:11: oid 11 version 6 is already newer than 4 2024-09-20T22:42:57.794 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 11 v 6 (ObjNum 10 snap 0 seq_num 10) dirty exists 2024-09-20T22:42:57.794 INFO:tasks.rados.rados.0.smithi084.stdout:11: left oid 11 (ObjNum 10 snap 0 seq_num 10) 2024-09-20T22:42:57.794 INFO:tasks.rados.rados.0.smithi084.stdout:12: finishing write tid 1 to smithi08477203-12 2024-09-20T22:42:57.795 INFO:tasks.rados.rados.0.smithi084.stdout:10: done (15 left) 2024-09-20T22:42:57.795 INFO:tasks.rados.rados.0.smithi084.stdout:11: done (14 left) 2024-09-20T22:42:57.795 INFO:tasks.rados.rados.0.smithi084.stdout:26: write initial oid 26 2024-09-20T22:42:57.795 INFO:tasks.rados.rados.0.smithi084.stdout:26: seq_num 25 ranges {420175=630284,1796638=471333} 2024-09-20T22:42:57.798 INFO:tasks.rados.rados.0.smithi084.stdout:26: writing smithi08477203-26 from 420175 to 1050459 tid 1 2024-09-20T22:42:57.805 INFO:tasks.rados.rados.0.smithi084.stdout:26: writing smithi08477203-26 from 1796638 to 2267971 tid 2 2024-09-20T22:42:57.805 INFO:tasks.rados.rados.0.smithi084.stdout:27: write initial oid 27 2024-09-20T22:42:57.805 INFO:tasks.rados.rados.0.smithi084.stdout:27: seq_num 26 ranges {436982=705533,1855120=581862} 2024-09-20T22:42:57.811 INFO:tasks.rados.rados.0.smithi084.stdout:27: writing smithi08477203-27 from 436982 to 1142515 tid 1 2024-09-20T22:42:57.819 INFO:tasks.rados.rados.0.smithi084.stdout:27: writing smithi08477203-27 from 1855120 to 2436982 tid 2 2024-09-20T22:42:57.819 INFO:tasks.rados.rados.0.smithi084.stdout: waiting on 16 2024-09-20T22:42:57.819 INFO:tasks.rados.rados.0.smithi084.stdout:12: finishing write tid 2 to smithi08477203-12 2024-09-20T22:42:57.819 INFO:tasks.rados.rados.0.smithi084.stdout:12: finishing write tid 3 to smithi08477203-12 2024-09-20T22:42:57.819 INFO:tasks.rados.rados.0.smithi084.stdout:12: finishing write tid 4 to smithi08477203-12 2024-09-20T22:42:57.819 INFO:tasks.rados.rados.0.smithi084.stdout:12: oid 12 updating version 0 to 3 2024-09-20T22:42:57.819 INFO:tasks.rados.rados.0.smithi084.stdout:12: oid 12 version 3 is already newer than 1 2024-09-20T22:42:57.819 INFO:tasks.rados.rados.0.smithi084.stdout:12: oid 12 version 3 is already newer than 2 2024-09-20T22:42:57.819 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 12 v 3 (ObjNum 11 snap 0 seq_num 11) dirty exists 2024-09-20T22:42:57.819 INFO:tasks.rados.rados.0.smithi084.stdout:12: left oid 12 (ObjNum 11 snap 0 seq_num 11) 2024-09-20T22:42:57.819 INFO:tasks.rados.rados.0.smithi084.stdout:13: finishing write tid 1 to smithi08477203-13 2024-09-20T22:42:57.819 INFO:tasks.rados.rados.0.smithi084.stdout:13: finishing write tid 2 to smithi08477203-13 2024-09-20T22:42:57.820 INFO:tasks.rados.rados.0.smithi084.stdout:13: finishing write tid 3 to smithi08477203-13 2024-09-20T22:42:57.820 INFO:tasks.rados.rados.0.smithi084.stdout:12: done (15 left) 2024-09-20T22:42:57.820 INFO:tasks.rados.rados.0.smithi084.stdout:28: write initial oid 28 2024-09-20T22:42:57.820 INFO:tasks.rados.rados.0.smithi084.stdout:28: seq_num 27 ranges {453789=780782,1913602=475002} 2024-09-20T22:42:57.825 INFO:tasks.rados.rados.0.smithi084.stdout:28: writing smithi08477203-28 from 453789 to 1234571 tid 1 2024-09-20T22:42:57.832 INFO:tasks.rados.rados.0.smithi084.stdout:28: writing smithi08477203-28 from 1913602 to 2388604 tid 2 2024-09-20T22:42:57.832 INFO:tasks.rados.rados.0.smithi084.stdout: waiting on 16 2024-09-20T22:42:57.832 INFO:tasks.rados.rados.0.smithi084.stdout:13: finishing write tid 4 to smithi08477203-13 2024-09-20T22:42:57.832 INFO:tasks.rados.rados.0.smithi084.stdout:13: oid 13 updating version 0 to 5 2024-09-20T22:42:57.832 INFO:tasks.rados.rados.0.smithi084.stdout:13: oid 13 version 5 is already newer than 4 2024-09-20T22:42:57.832 INFO:tasks.rados.rados.0.smithi084.stdout:13: oid 13 updating version 5 to 6 2024-09-20T22:42:57.832 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 13 v 6 (ObjNum 12 snap 0 seq_num 12) dirty exists 2024-09-20T22:42:57.832 INFO:tasks.rados.rados.0.smithi084.stdout:13: left oid 13 (ObjNum 12 snap 0 seq_num 12) 2024-09-20T22:42:57.832 INFO:tasks.rados.rados.0.smithi084.stdout:14: finishing write tid 1 to smithi08477203-14 2024-09-20T22:42:57.833 INFO:tasks.rados.rados.0.smithi084.stdout:14: finishing write tid 2 to smithi08477203-14 2024-09-20T22:42:57.833 INFO:tasks.rados.rados.0.smithi084.stdout:14: finishing write tid 3 to smithi08477203-14 2024-09-20T22:42:57.833 INFO:tasks.rados.rados.0.smithi084.stdout:14: finishing write tid 4 to smithi08477203-14 2024-09-20T22:42:57.833 INFO:tasks.rados.rados.0.smithi084.stdout:14: oid 14 updating version 0 to 1 2024-09-20T22:42:57.833 INFO:tasks.rados.rados.0.smithi084.stdout:14: oid 14 updating version 1 to 3 2024-09-20T22:42:57.833 INFO:tasks.rados.rados.0.smithi084.stdout:14: oid 14 version 3 is already newer than 2 2024-09-20T22:42:57.833 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 14 v 3 (ObjNum 13 snap 0 seq_num 13) dirty exists 2024-09-20T22:42:57.833 INFO:tasks.rados.rados.0.smithi084.stdout:14: left oid 14 (ObjNum 13 snap 0 seq_num 13) 2024-09-20T22:42:57.833 INFO:tasks.rados.rados.0.smithi084.stdout:15: finishing write tid 1 to smithi08477203-15 2024-09-20T22:42:57.833 INFO:tasks.rados.rados.0.smithi084.stdout:15: finishing write tid 2 to smithi08477203-15 2024-09-20T22:42:57.833 INFO:tasks.rados.rados.0.smithi084.stdout:15: finishing write tid 3 to smithi08477203-15 2024-09-20T22:42:57.833 INFO:tasks.rados.rados.0.smithi084.stdout:13: done (15 left) 2024-09-20T22:42:57.833 INFO:tasks.rados.rados.0.smithi084.stdout:14: done (14 left) 2024-09-20T22:42:57.833 INFO:tasks.rados.rados.0.smithi084.stdout:29: write initial oid 29 2024-09-20T22:42:57.833 INFO:tasks.rados.rados.0.smithi084.stdout:29: seq_num 28 ranges {470596=456031,1572084=618660} 2024-09-20T22:42:57.834 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:42:57.836 INFO:tasks.rados.rados.0.smithi084.stdout:29: writing smithi08477203-29 from 470596 to 926627 tid 1 2024-09-20T22:42:57.844 INFO:tasks.rados.rados.0.smithi084.stdout:29: writing smithi08477203-29 from 1572084 to 2190744 tid 2 2024-09-20T22:42:57.844 INFO:tasks.rados.rados.0.smithi084.stdout:30: write initial oid 30 2024-09-20T22:42:57.844 INFO:tasks.rados.rados.0.smithi084.stdout:30: seq_num 29 ranges {487403=531280,1514213=478671} 2024-09-20T22:42:57.849 INFO:tasks.rados.rados.0.smithi084.stdout:30: writing smithi08477203-30 from 487403 to 1018683 tid 1 2024-09-20T22:42:57.855 INFO:tasks.rados.rados.0.smithi084.stdout:30: writing smithi08477203-30 from 1514213 to 1992884 tid 2 2024-09-20T22:42:57.855 INFO:tasks.rados.rados.0.smithi084.stdout: waiting on 16 2024-09-20T22:42:57.855 INFO:tasks.rados.rados.0.smithi084.stdout:15: finishing write tid 4 to smithi08477203-15 2024-09-20T22:42:57.855 INFO:tasks.rados.rados.0.smithi084.stdout:15: oid 15 updating version 0 to 8 2024-09-20T22:42:57.855 INFO:tasks.rados.rados.0.smithi084.stdout:15: oid 15 updating version 8 to 9 2024-09-20T22:42:57.855 INFO:tasks.rados.rados.0.smithi084.stdout:15: oid 15 version 9 is already newer than 7 2024-09-20T22:42:57.855 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 15 v 9 (ObjNum 14 snap 0 seq_num 14) dirty exists 2024-09-20T22:42:57.855 INFO:tasks.rados.rados.0.smithi084.stdout:15: left oid 15 (ObjNum 14 snap 0 seq_num 14) 2024-09-20T22:42:57.855 INFO:tasks.rados.rados.0.smithi084.stdout:16: finishing write tid 1 to smithi08477203-16 2024-09-20T22:42:57.855 INFO:tasks.rados.rados.0.smithi084.stdout:16: finishing write tid 2 to smithi08477203-16 2024-09-20T22:42:57.856 INFO:tasks.rados.rados.0.smithi084.stdout:16: finishing write tid 3 to smithi08477203-16 2024-09-20T22:42:57.856 INFO:tasks.rados.rados.0.smithi084.stdout:15: done (15 left) 2024-09-20T22:42:57.856 INFO:tasks.rados.rados.0.smithi084.stdout:31: write initial oid 31 2024-09-20T22:42:57.856 INFO:tasks.rados.rados.0.smithi084.stdout:31: seq_num 30 ranges {504210=606529,1572695=622329} 2024-09-20T22:42:57.856 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:42:57.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:42:57.857 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:42:57.860 INFO:tasks.rados.rados.0.smithi084.stdout:31: writing smithi08477203-31 from 504210 to 1110739 tid 1 2024-09-20T22:42:57.867 INFO:tasks.rados.rados.0.smithi084.stdout:31: writing smithi08477203-31 from 1572695 to 2195024 tid 2 2024-09-20T22:42:57.867 INFO:tasks.rados.rados.0.smithi084.stdout: waiting on 16 2024-09-20T22:42:57.868 INFO:tasks.rados.rados.0.smithi084.stdout:16: finishing write tid 4 to smithi08477203-16 2024-09-20T22:42:57.868 INFO:tasks.rados.rados.0.smithi084.stdout:16: oid 16 updating version 0 to 3 2024-09-20T22:42:57.868 INFO:tasks.rados.rados.0.smithi084.stdout:16: oid 16 version 3 is already newer than 1 2024-09-20T22:42:57.868 INFO:tasks.rados.rados.0.smithi084.stdout:16: oid 16 version 3 is already newer than 2 2024-09-20T22:42:57.868 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 16 v 3 (ObjNum 15 snap 0 seq_num 15) dirty exists 2024-09-20T22:42:57.868 INFO:tasks.rados.rados.0.smithi084.stdout:16: left oid 16 (ObjNum 15 snap 0 seq_num 15) 2024-09-20T22:42:57.868 INFO:tasks.rados.rados.0.smithi084.stdout:17: finishing write tid 1 to smithi08477203-17 2024-09-20T22:42:57.868 INFO:tasks.rados.rados.0.smithi084.stdout:17: finishing write tid 2 to smithi08477203-17 2024-09-20T22:42:57.868 INFO:tasks.rados.rados.0.smithi084.stdout:16: done (15 left) 2024-09-20T22:42:57.868 INFO:tasks.rados.rados.0.smithi084.stdout:32: write initial oid 32 2024-09-20T22:42:57.868 INFO:tasks.rados.rados.0.smithi084.stdout:17: finishing write tid 3 to smithi08477203-17 2024-09-20T22:42:57.868 INFO:tasks.rados.rados.0.smithi084.stdout:17: finishing write tid 4 to smithi08477203-17 2024-09-20T22:42:57.868 INFO:tasks.rados.rados.0.smithi084.stdout:17: oid 17 updating version 0 to 5 2024-09-20T22:42:57.868 INFO:tasks.rados.rados.0.smithi084.stdout:17: oid 17 updating version 5 to 6 2024-09-20T22:42:57.869 INFO:tasks.rados.rados.0.smithi084.stdout:17: oid 17 version 6 is already newer than 4 2024-09-20T22:42:57.869 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 17 v 6 (ObjNum 16 snap 0 seq_num 16) dirty exists 2024-09-20T22:42:57.869 INFO:tasks.rados.rados.0.smithi084.stdout:17: left oid 17 (ObjNum 16 snap 0 seq_num 16) 2024-09-20T22:42:57.869 INFO:tasks.rados.rados.0.smithi084.stdout:18: finishing write tid 1 to smithi08477203-18 2024-09-20T22:42:57.869 INFO:tasks.rados.rados.0.smithi084.stdout:18: finishing write tid 2 to smithi08477203-18 2024-09-20T22:42:57.869 INFO:tasks.rados.rados.0.smithi084.stdout:18: finishing write tid 3 to smithi08477203-18 2024-09-20T22:42:57.869 INFO:tasks.rados.rados.0.smithi084.stdout:32: seq_num 31 ranges {521017=798131,1747530=482340} 2024-09-20T22:42:57.874 INFO:tasks.rados.rados.0.smithi084.stdout:32: writing smithi08477203-32 from 521017 to 1319148 tid 1 2024-09-20T22:42:57.880 INFO:tasks.rados.rados.0.smithi084.stdout:32: writing smithi08477203-32 from 1747530 to 2229870 tid 2 2024-09-20T22:42:57.880 INFO:tasks.rados.rados.0.smithi084.stdout:17: done (15 left) 2024-09-20T22:42:57.880 INFO:tasks.rados.rados.0.smithi084.stdout:33: write initial oid 33 2024-09-20T22:42:57.880 INFO:tasks.rados.rados.0.smithi084.stdout:33: seq_num 32 ranges {537824=473380,1806012=625998} 2024-09-20T22:42:57.885 INFO:tasks.rados.rados.0.smithi084.stdout:33: writing smithi08477203-33 from 537824 to 1011204 tid 1 2024-09-20T22:42:57.894 INFO:tasks.rados.rados.0.smithi084.stdout:33: writing smithi08477203-33 from 1806012 to 2432010 tid 2 2024-09-20T22:42:57.894 INFO:tasks.rados.rados.0.smithi084.stdout: waiting on 16 2024-09-20T22:42:57.894 INFO:tasks.rados.rados.0.smithi084.stdout:18: finishing write tid 4 to smithi08477203-18 2024-09-20T22:42:57.894 INFO:tasks.rados.rados.0.smithi084.stdout:18: oid 18 updating version 0 to 11 2024-09-20T22:42:57.894 INFO:tasks.rados.rados.0.smithi084.stdout:18: oid 18 updating version 11 to 12 2024-09-20T22:42:57.894 INFO:tasks.rados.rados.0.smithi084.stdout:18: oid 18 version 12 is already newer than 10 2024-09-20T22:42:57.894 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 18 v 12 (ObjNum 17 snap 0 seq_num 17) dirty exists 2024-09-20T22:42:57.894 INFO:tasks.rados.rados.0.smithi084.stdout:18: left oid 18 (ObjNum 17 snap 0 seq_num 17) 2024-09-20T22:42:57.894 INFO:tasks.rados.rados.0.smithi084.stdout:19: finishing write tid 1 to smithi08477203-19 2024-09-20T22:42:57.894 INFO:tasks.rados.rados.0.smithi084.stdout:19: finishing write tid 2 to smithi08477203-19 2024-09-20T22:42:57.894 INFO:tasks.rados.rados.0.smithi084.stdout:19: finishing write tid 3 to smithi08477203-19 2024-09-20T22:42:57.895 INFO:tasks.rados.rados.0.smithi084.stdout:19: finishing write tid 4 to smithi08477203-19 2024-09-20T22:42:57.895 INFO:tasks.rados.rados.0.smithi084.stdout:19: oid 19 updating version 0 to 8 2024-09-20T22:42:57.895 INFO:tasks.rados.rados.0.smithi084.stdout:19: oid 19 updating version 8 to 9 2024-09-20T22:42:57.895 INFO:tasks.rados.rados.0.smithi084.stdout:19: oid 19 version 9 is already newer than 7 2024-09-20T22:42:57.895 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 19 v 9 (ObjNum 18 snap 0 seq_num 18) dirty exists 2024-09-20T22:42:57.895 INFO:tasks.rados.rados.0.smithi084.stdout:19: left oid 19 (ObjNum 18 snap 0 seq_num 18) 2024-09-20T22:42:57.895 INFO:tasks.rados.rados.0.smithi084.stdout:20: finishing write tid 1 to smithi08477203-20 2024-09-20T22:42:57.895 INFO:tasks.rados.rados.0.smithi084.stdout:20: finishing write tid 2 to smithi08477203-20 2024-09-20T22:42:57.895 INFO:tasks.rados.rados.0.smithi084.stdout:20: finishing write tid 3 to smithi08477203-20 2024-09-20T22:42:57.895 INFO:tasks.rados.rados.0.smithi084.stdout:20: finishing write tid 4 to smithi08477203-20 2024-09-20T22:42:57.895 INFO:tasks.rados.rados.0.smithi084.stdout:20: oid 20 updating version 0 to 4 2024-09-20T22:42:57.895 INFO:tasks.rados.rados.0.smithi084.stdout:20: oid 20 updating version 4 to 6 2024-09-20T22:42:57.895 INFO:tasks.rados.rados.0.smithi084.stdout:20: oid 20 version 6 is already newer than 5 2024-09-20T22:42:57.895 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 20 v 6 (ObjNum 19 snap 0 seq_num 19) dirty exists 2024-09-20T22:42:57.896 INFO:tasks.rados.rados.0.smithi084.stdout:20: left oid 20 (ObjNum 19 snap 0 seq_num 19) 2024-09-20T22:42:57.896 INFO:tasks.rados.rados.0.smithi084.stdout:21: finishing write tid 1 to smithi08477203-21 2024-09-20T22:42:57.896 INFO:tasks.rados.rados.0.smithi084.stdout:18: done (15 left) 2024-09-20T22:42:57.896 INFO:tasks.rados.rados.0.smithi084.stdout:19: done (14 left) 2024-09-20T22:42:57.896 INFO:tasks.rados.rados.0.smithi084.stdout:20: done (13 left) 2024-09-20T22:42:57.896 INFO:tasks.rados.rados.0.smithi084.stdout:34: write initial oid 34 2024-09-20T22:42:57.896 INFO:tasks.rados.rados.0.smithi084.stdout:34: seq_num 33 ranges {554631=548629,1748141=486009} 2024-09-20T22:42:57.899 INFO:tasks.rados.rados.0.smithi084.stdout:34: writing smithi08477203-34 from 554631 to 1103260 tid 1 2024-09-20T22:42:57.905 INFO:tasks.rados.rados.0.smithi084.stdout:34: writing smithi08477203-34 from 1748141 to 2234150 tid 2 2024-09-20T22:42:57.905 INFO:tasks.rados.rados.0.smithi084.stdout:35: write initial oid 35 2024-09-20T22:42:57.905 INFO:tasks.rados.rados.0.smithi084.stdout:35: seq_num 34 ranges {571438=623878,1806623=629667} 2024-09-20T22:42:57.912 INFO:tasks.rados.rados.0.smithi084.stdout:35: writing smithi08477203-35 from 571438 to 1195316 tid 1 2024-09-20T22:42:57.919 INFO:tasks.rados.rados.0.smithi084.stdout:35: writing smithi08477203-35 from 1806623 to 2436290 tid 2 2024-09-20T22:42:57.919 INFO:tasks.rados.rados.0.smithi084.stdout:36: write initial oid 36 2024-09-20T22:42:57.919 INFO:tasks.rados.rados.0.smithi084.stdout:36: seq_num 35 ranges {588245=699127,1865105=489678} 2024-09-20T22:42:57.926 INFO:tasks.rados.rados.0.smithi084.stdout:36: writing smithi08477203-36 from 588245 to 1287372 tid 1 2024-09-20T22:42:57.932 INFO:tasks.rados.rados.0.smithi084.stdout:36: writing smithi08477203-36 from 1865105 to 2354783 tid 2 2024-09-20T22:42:57.932 INFO:tasks.rados.rados.0.smithi084.stdout: waiting on 16 2024-09-20T22:42:57.932 INFO:tasks.rados.rados.0.smithi084.stdout:21: finishing write tid 2 to smithi08477203-21 2024-09-20T22:42:57.933 INFO:tasks.rados.rados.0.smithi084.stdout:21: finishing write tid 3 to smithi08477203-21 2024-09-20T22:42:57.933 INFO:tasks.rados.rados.0.smithi084.stdout:21: finishing write tid 4 to smithi08477203-21 2024-09-20T22:42:57.933 INFO:tasks.rados.rados.0.smithi084.stdout:21: oid 21 updating version 0 to 6 2024-09-20T22:42:57.933 INFO:tasks.rados.rados.0.smithi084.stdout:21: oid 21 version 6 is already newer than 5 2024-09-20T22:42:57.933 INFO:tasks.rados.rados.0.smithi084.stdout:21: oid 21 version 6 is already newer than 4 2024-09-20T22:42:57.933 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 21 v 6 (ObjNum 20 snap 0 seq_num 20) dirty exists 2024-09-20T22:42:57.933 INFO:tasks.rados.rados.0.smithi084.stdout:21: left oid 21 (ObjNum 20 snap 0 seq_num 20) 2024-09-20T22:42:57.933 INFO:tasks.rados.rados.0.smithi084.stdout:22: finishing write tid 1 to smithi08477203-22 2024-09-20T22:42:57.933 INFO:tasks.rados.rados.0.smithi084.stdout:22: finishing write tid 2 to smithi08477203-22 2024-09-20T22:42:57.933 INFO:tasks.rados.rados.0.smithi084.stdout:22: finishing write tid 3 to smithi08477203-22 2024-09-20T22:42:57.933 INFO:tasks.rados.rados.0.smithi084.stdout:21: done (15 left) 2024-09-20T22:42:57.933 INFO:tasks.rados.rados.0.smithi084.stdout:37: write initial oid 37 2024-09-20T22:42:57.933 INFO:tasks.rados.rados.0.smithi084.stdout:37: seq_num 36 ranges {605052=774376,1923587=633336} 2024-09-20T22:42:57.939 INFO:tasks.rados.rados.0.smithi084.stdout:37: writing smithi08477203-37 from 605052 to 1379428 tid 1 2024-09-20T22:42:57.947 INFO:tasks.rados.rados.0.smithi084.stdout:37: writing smithi08477203-37 from 1923587 to 2556923 tid 2 2024-09-20T22:42:57.947 INFO:tasks.rados.rados.0.smithi084.stdout: waiting on 16 2024-09-20T22:42:57.947 INFO:tasks.rados.rados.0.smithi084.stdout:22: finishing write tid 4 to smithi08477203-22 2024-09-20T22:42:57.947 INFO:tasks.rados.rados.0.smithi084.stdout:22: oid 22 updating version 0 to 1 2024-09-20T22:42:57.947 INFO:tasks.rados.rados.0.smithi084.stdout:22: oid 22 updating version 1 to 2 2024-09-20T22:42:57.947 INFO:tasks.rados.rados.0.smithi084.stdout:22: oid 22 updating version 2 to 3 2024-09-20T22:42:57.947 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 22 v 3 (ObjNum 21 snap 0 seq_num 21) dirty exists 2024-09-20T22:42:57.947 INFO:tasks.rados.rados.0.smithi084.stdout:22: left oid 22 (ObjNum 21 snap 0 seq_num 21) 2024-09-20T22:42:57.947 INFO:tasks.rados.rados.0.smithi084.stdout:23: finishing write tid 1 to smithi08477203-23 2024-09-20T22:42:57.947 INFO:tasks.rados.rados.0.smithi084.stdout:23: finishing write tid 2 to smithi08477203-23 2024-09-20T22:42:57.947 INFO:tasks.rados.rados.0.smithi084.stdout:23: finishing write tid 3 to smithi08477203-23 2024-09-20T22:42:57.947 INFO:tasks.rados.rados.0.smithi084.stdout:23: finishing write tid 4 to smithi08477203-23 2024-09-20T22:42:57.948 INFO:tasks.rados.rados.0.smithi084.stdout:23: oid 23 updating version 0 to 2 2024-09-20T22:42:57.948 INFO:tasks.rados.rados.0.smithi084.stdout:23: oid 23 version 2 is already newer than 1 2024-09-20T22:42:57.948 INFO:tasks.rados.rados.0.smithi084.stdout:23: oid 23 updating version 2 to 3 2024-09-20T22:42:57.948 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 23 v 3 (ObjNum 22 snap 0 seq_num 22) dirty exists 2024-09-20T22:42:57.948 INFO:tasks.rados.rados.0.smithi084.stdout:23: left oid 23 (ObjNum 22 snap 0 seq_num 22) 2024-09-20T22:42:57.948 INFO:tasks.rados.rados.0.smithi084.stdout:24: finishing write tid 1 to smithi08477203-24 2024-09-20T22:42:57.948 INFO:tasks.rados.rados.0.smithi084.stdout:22: done (15 left) 2024-09-20T22:42:57.948 INFO:tasks.rados.rados.0.smithi084.stdout:23: done (14 left) 2024-09-20T22:42:57.948 INFO:tasks.rados.rados.0.smithi084.stdout:38: write initial oid 38 2024-09-20T22:42:57.948 INFO:tasks.rados.rados.0.smithi084.stdout:38: seq_num 37 ranges {621859=449625,1865716=756143} 2024-09-20T22:42:57.952 INFO:tasks.rados.rados.0.smithi084.stdout:38: writing smithi08477203-38 from 621859 to 1071484 tid 1 2024-09-20T22:42:57.958 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:42:57.961 INFO:tasks.rados.rados.0.smithi084.stdout:38: writing smithi08477203-38 from 1865716 to 2621859 tid 2 2024-09-20T22:42:57.961 INFO:tasks.rados.rados.0.smithi084.stdout:39: write initial oid 39 2024-09-20T22:42:57.961 INFO:tasks.rados.rados.0.smithi084.stdout:39: seq_num 38 ranges {638666=524874,1924198=637005} 2024-09-20T22:42:57.967 INFO:tasks.rados.rados.0.smithi084.stdout:39: writing smithi08477203-39 from 638666 to 1163540 tid 1 2024-09-20T22:42:57.975 INFO:tasks.rados.rados.0.smithi084.stdout:39: writing smithi08477203-39 from 1924198 to 2561203 tid 2 2024-09-20T22:42:57.975 INFO:tasks.rados.rados.0.smithi084.stdout: waiting on 16 2024-09-20T22:42:57.975 INFO:tasks.rados.rados.0.smithi084.stdout:24: finishing write tid 2 to smithi08477203-24 2024-09-20T22:42:57.976 INFO:tasks.rados.rados.0.smithi084.stdout:24: finishing write tid 3 to smithi08477203-24 2024-09-20T22:42:57.976 INFO:tasks.rados.rados.0.smithi084.stdout:24: finishing write tid 4 to smithi08477203-24 2024-09-20T22:42:57.976 INFO:tasks.rados.rados.0.smithi084.stdout:24: oid 24 updating version 0 to 3 2024-09-20T22:42:57.976 INFO:tasks.rados.rados.0.smithi084.stdout:24: oid 24 version 3 is already newer than 2 2024-09-20T22:42:57.976 INFO:tasks.rados.rados.0.smithi084.stdout:24: oid 24 version 3 is already newer than 1 2024-09-20T22:42:57.976 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 24 v 3 (ObjNum 23 snap 0 seq_num 23) dirty exists 2024-09-20T22:42:57.976 INFO:tasks.rados.rados.0.smithi084.stdout:24: left oid 24 (ObjNum 23 snap 0 seq_num 23) 2024-09-20T22:42:57.976 INFO:tasks.rados.rados.0.smithi084.stdout:25: finishing write tid 1 to smithi08477203-25 2024-09-20T22:42:57.976 INFO:tasks.rados.rados.0.smithi084.stdout:25: finishing write tid 2 to smithi08477203-25 2024-09-20T22:42:57.976 INFO:tasks.rados.rados.0.smithi084.stdout:25: finishing write tid 3 to smithi08477203-25 2024-09-20T22:42:57.976 INFO:tasks.rados.rados.0.smithi084.stdout:25: finishing write tid 4 to smithi08477203-25 2024-09-20T22:42:57.976 INFO:tasks.rados.rados.0.smithi084.stdout:25: oid 25 updating version 0 to 12 2024-09-20T22:42:57.976 INFO:tasks.rados.rados.0.smithi084.stdout:25: oid 25 version 12 is already newer than 10 2024-09-20T22:42:57.976 INFO:tasks.rados.rados.0.smithi084.stdout:25: oid 25 version 12 is already newer than 11 2024-09-20T22:42:57.976 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 25 v 12 (ObjNum 24 snap 0 seq_num 24) dirty exists 2024-09-20T22:42:57.977 INFO:tasks.rados.rados.0.smithi084.stdout:25: left oid 25 (ObjNum 24 snap 0 seq_num 24) 2024-09-20T22:42:57.977 INFO:tasks.rados.rados.0.smithi084.stdout:26: finishing write tid 1 to smithi08477203-26 2024-09-20T22:42:57.977 INFO:tasks.rados.rados.0.smithi084.stdout:24: done (15 left) 2024-09-20T22:42:57.977 INFO:tasks.rados.rados.0.smithi084.stdout:25: done (14 left) 2024-09-20T22:42:57.977 INFO:tasks.rados.rados.0.smithi084.stdout:40: write initial oid 40 2024-09-20T22:42:57.977 INFO:tasks.rados.rados.0.smithi084.stdout:26: finishing write tid 2 to smithi08477203-26 2024-09-20T22:42:57.977 INFO:tasks.rados.rados.0.smithi084.stdout:26: finishing write tid 3 to smithi08477203-26 2024-09-20T22:42:57.977 INFO:tasks.rados.rados.0.smithi084.stdout:40: seq_num 39 ranges {655473=716476,2099033=556440} 2024-09-20T22:42:57.980 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:42:57.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:42:57.981 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:42:57.982 INFO:tasks.rados.rados.0.smithi084.stdout:40: writing smithi08477203-40 from 655473 to 1371949 tid 1 2024-09-20T22:42:57.989 INFO:tasks.rados.rados.0.smithi084.stdout:40: writing smithi08477203-40 from 2099033 to 2655473 tid 2 2024-09-20T22:42:57.990 INFO:tasks.rados.rados.0.smithi084.stdout:41: write initial oid 41 2024-09-20T22:42:57.990 INFO:tasks.rados.rados.0.smithi084.stdout:41: seq_num 40 ranges {672280=791725,2157515=514765} 2024-09-20T22:42:57.997 INFO:tasks.rados.rados.0.smithi084.stdout:41: writing smithi08477203-41 from 672280 to 1464005 tid 1 2024-09-20T22:42:58.004 INFO:tasks.rados.rados.0.smithi084.stdout:41: writing smithi08477203-41 from 2157515 to 2672280 tid 2 2024-09-20T22:42:58.005 INFO:tasks.rados.rados.0.smithi084.stdout: waiting on 16 2024-09-20T22:42:58.005 INFO:tasks.rados.rados.0.smithi084.stdout:26: finishing write tid 4 to smithi08477203-26 2024-09-20T22:42:58.005 INFO:tasks.rados.rados.0.smithi084.stdout:26: oid 26 updating version 0 to 5 2024-09-20T22:42:58.005 INFO:tasks.rados.rados.0.smithi084.stdout:26: oid 26 version 5 is already newer than 4 2024-09-20T22:42:58.005 INFO:tasks.rados.rados.0.smithi084.stdout:26: oid 26 updating version 5 to 6 2024-09-20T22:42:58.005 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 26 v 6 (ObjNum 25 snap 0 seq_num 25) dirty exists 2024-09-20T22:42:58.005 INFO:tasks.rados.rados.0.smithi084.stdout:26: left oid 26 (ObjNum 25 snap 0 seq_num 25) 2024-09-20T22:42:58.005 INFO:tasks.rados.rados.0.smithi084.stdout:27: finishing write tid 1 to smithi08477203-27 2024-09-20T22:42:58.005 INFO:tasks.rados.rados.0.smithi084.stdout:26: done (15 left) 2024-09-20T22:42:58.005 INFO:tasks.rados.rados.0.smithi084.stdout:42: write initial oid 42 2024-09-20T22:42:58.005 INFO:tasks.rados.rados.0.smithi084.stdout:42: seq_num 41 ranges {689087=466974,1699644=784332} 2024-09-20T22:42:58.011 INFO:tasks.rados.rados.0.smithi084.stdout:42: writing smithi08477203-42 from 689087 to 1156061 tid 1 2024-09-20T22:42:58.018 INFO:tasks.rados.rados.0.smithi084.stdout:42: writing smithi08477203-42 from 1699644 to 2483976 tid 2 2024-09-20T22:42:58.018 INFO:tasks.rados.rados.0.smithi084.stdout: waiting on 16 2024-09-20T22:42:58.018 INFO:tasks.rados.rados.0.smithi084.stdout:27: finishing write tid 2 to smithi08477203-27 2024-09-20T22:42:58.019 INFO:tasks.rados.rados.0.smithi084.stdout:27: finishing write tid 3 to smithi08477203-27 2024-09-20T22:42:58.019 INFO:tasks.rados.rados.0.smithi084.stdout:27: finishing write tid 4 to smithi08477203-27 2024-09-20T22:42:58.019 INFO:tasks.rados.rados.0.smithi084.stdout:27: oid 27 updating version 0 to 5 2024-09-20T22:42:58.019 INFO:tasks.rados.rados.0.smithi084.stdout:27: oid 27 version 5 is already newer than 4 2024-09-20T22:42:58.019 INFO:tasks.rados.rados.0.smithi084.stdout:27: oid 27 updating version 5 to 6 2024-09-20T22:42:58.019 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 27 v 6 (ObjNum 26 snap 0 seq_num 26) dirty exists 2024-09-20T22:42:58.019 INFO:tasks.rados.rados.0.smithi084.stdout:27: left oid 27 (ObjNum 26 snap 0 seq_num 26) 2024-09-20T22:42:58.019 INFO:tasks.rados.rados.0.smithi084.stdout:28: finishing write tid 1 to smithi08477203-28 2024-09-20T22:42:58.019 INFO:tasks.rados.rados.0.smithi084.stdout:28: finishing write tid 2 to smithi08477203-28 2024-09-20T22:42:58.019 INFO:tasks.rados.rados.0.smithi084.stdout:28: finishing write tid 3 to smithi08477203-28 2024-09-20T22:42:58.019 INFO:tasks.rados.rados.0.smithi084.stdout:27: done (15 left) 2024-09-20T22:42:58.019 INFO:tasks.rados.rados.0.smithi084.stdout:43: write initial oid 43 2024-09-20T22:42:58.019 INFO:tasks.rados.rados.0.smithi084.stdout:43: seq_num 42 ranges {705894=542223,1758126=644343} 2024-09-20T22:42:58.025 INFO:tasks.rados.rados.0.smithi084.stdout:43: writing smithi08477203-43 from 705894 to 1248117 tid 1 2024-09-20T22:42:58.032 INFO:tasks.rados.rados.0.smithi084.stdout:43: writing smithi08477203-43 from 1758126 to 2402469 tid 2 2024-09-20T22:42:58.032 INFO:tasks.rados.rados.0.smithi084.stdout: waiting on 16 2024-09-20T22:42:58.032 INFO:tasks.rados.rados.0.smithi084.stdout:28: finishing write tid 4 to smithi08477203-28 2024-09-20T22:42:58.032 INFO:tasks.rados.rados.0.smithi084.stdout:28: oid 28 updating version 0 to 15 2024-09-20T22:42:58.032 INFO:tasks.rados.rados.0.smithi084.stdout:28: oid 28 version 15 is already newer than 14 2024-09-20T22:42:58.032 INFO:tasks.rados.rados.0.smithi084.stdout:28: oid 28 version 15 is already newer than 13 2024-09-20T22:42:58.032 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 28 v 15 (ObjNum 27 snap 0 seq_num 27) dirty exists 2024-09-20T22:42:58.032 INFO:tasks.rados.rados.0.smithi084.stdout:28: left oid 28 (ObjNum 27 snap 0 seq_num 27) 2024-09-20T22:42:58.032 INFO:tasks.rados.rados.0.smithi084.stdout:29: finishing write tid 1 to smithi08477203-29 2024-09-20T22:42:58.032 INFO:tasks.rados.rados.0.smithi084.stdout:29: finishing write tid 2 to smithi08477203-29 2024-09-20T22:42:58.032 INFO:tasks.rados.rados.0.smithi084.stdout:29: finishing write tid 3 to smithi08477203-29 2024-09-20T22:42:58.033 INFO:tasks.rados.rados.0.smithi084.stdout:28: done (15 left) 2024-09-20T22:42:58.033 INFO:tasks.rados.rados.0.smithi084.stdout:44: write initial oid 44 2024-09-20T22:42:58.033 INFO:tasks.rados.rados.0.smithi084.stdout:44: seq_num 43 ranges {722701=617472,1816608=788001} 2024-09-20T22:42:58.038 INFO:tasks.rados.rados.0.smithi084.stdout:44: writing smithi08477203-44 from 722701 to 1340173 tid 1 2024-09-20T22:42:58.046 INFO:tasks.rados.rados.0.smithi084.stdout:44: writing smithi08477203-44 from 1816608 to 2604609 tid 2 2024-09-20T22:42:58.046 INFO:tasks.rados.rados.0.smithi084.stdout: waiting on 16 2024-09-20T22:42:58.046 INFO:tasks.rados.rados.0.smithi084.stdout:29: finishing write tid 4 to smithi08477203-29 2024-09-20T22:42:58.046 INFO:tasks.rados.rados.0.smithi084.stdout:29: oid 29 updating version 0 to 17 2024-09-20T22:42:58.046 INFO:tasks.rados.rados.0.smithi084.stdout:29: oid 29 updating version 17 to 18 2024-09-20T22:42:58.046 INFO:tasks.rados.rados.0.smithi084.stdout:29: oid 29 version 18 is already newer than 16 2024-09-20T22:42:58.046 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 29 v 18 (ObjNum 28 snap 0 seq_num 28) dirty exists 2024-09-20T22:42:58.046 INFO:tasks.rados.rados.0.smithi084.stdout:29: left oid 29 (ObjNum 28 snap 0 seq_num 28) 2024-09-20T22:42:58.046 INFO:tasks.rados.rados.0.smithi084.stdout:30: finishing write tid 1 to smithi08477203-30 2024-09-20T22:42:58.047 INFO:tasks.rados.rados.0.smithi084.stdout:30: finishing write tid 2 to smithi08477203-30 2024-09-20T22:42:58.047 INFO:tasks.rados.rados.0.smithi084.stdout:29: done (15 left) 2024-09-20T22:42:58.047 INFO:tasks.rados.rados.0.smithi084.stdout:45: write initial oid 45 2024-09-20T22:42:58.047 INFO:tasks.rados.rados.0.smithi084.stdout:45: seq_num 44 ranges {739508=692721,1875090=648012} 2024-09-20T22:42:58.054 INFO:tasks.rados.rados.0.smithi084.stdout:45: writing smithi08477203-45 from 739508 to 1432229 tid 1 2024-09-20T22:42:58.060 INFO:tasks.rados.rados.0.smithi084.stdout:45: writing smithi08477203-45 from 1875090 to 2523102 tid 2 2024-09-20T22:42:58.060 INFO:tasks.rados.rados.0.smithi084.stdout: waiting on 16 2024-09-20T22:42:58.060 INFO:tasks.rados.rados.0.smithi084.stdout:30: finishing write tid 3 to smithi08477203-30 2024-09-20T22:42:58.060 INFO:tasks.rados.rados.0.smithi084.stdout:30: finishing write tid 4 to smithi08477203-30 2024-09-20T22:42:58.060 INFO:tasks.rados.rados.0.smithi084.stdout:30: oid 30 updating version 0 to 10 2024-09-20T22:42:58.060 INFO:tasks.rados.rados.0.smithi084.stdout:30: oid 30 updating version 10 to 11 2024-09-20T22:42:58.061 INFO:tasks.rados.rados.0.smithi084.stdout:30: oid 30 updating version 11 to 12 2024-09-20T22:42:58.061 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 30 v 12 (ObjNum 29 snap 0 seq_num 29) dirty exists 2024-09-20T22:42:58.061 INFO:tasks.rados.rados.0.smithi084.stdout:30: left oid 30 (ObjNum 29 snap 0 seq_num 29) 2024-09-20T22:42:58.061 INFO:tasks.rados.rados.0.smithi084.stdout:31: finishing write tid 1 to smithi08477203-31 2024-09-20T22:42:58.061 INFO:tasks.rados.rados.0.smithi084.stdout:31: finishing write tid 2 to smithi08477203-31 2024-09-20T22:42:58.061 INFO:tasks.rados.rados.0.smithi084.stdout:30: done (15 left) 2024-09-20T22:42:58.061 INFO:tasks.rados.rados.0.smithi084.stdout:46: write initial oid 46 2024-09-20T22:42:58.061 INFO:tasks.rados.rados.0.smithi084.stdout:46: seq_num 45 ranges {756315=767970,1933572=791670} 2024-09-20T22:42:58.068 INFO:tasks.rados.rados.0.smithi084.stdout:46: writing smithi08477203-46 from 756315 to 1524285 tid 1 2024-09-20T22:42:58.075 INFO:tasks.rados.rados.0.smithi084.stdout:46: writing smithi08477203-46 from 1933572 to 2725242 tid 2 2024-09-20T22:42:58.075 INFO:tasks.rados.rados.0.smithi084.stdout: waiting on 16 2024-09-20T22:42:58.075 INFO:tasks.rados.rados.0.smithi084.stdout:31: finishing write tid 3 to smithi08477203-31 2024-09-20T22:42:58.075 INFO:tasks.rados.rados.0.smithi084.stdout:31: finishing write tid 4 to smithi08477203-31 2024-09-20T22:42:58.075 INFO:tasks.rados.rados.0.smithi084.stdout:31: oid 31 updating version 0 to 8 2024-09-20T22:42:58.075 INFO:tasks.rados.rados.0.smithi084.stdout:31: oid 31 version 8 is already newer than 7 2024-09-20T22:42:58.076 INFO:tasks.rados.rados.0.smithi084.stdout:31: oid 31 updating version 8 to 9 2024-09-20T22:42:58.076 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 31 v 9 (ObjNum 30 snap 0 seq_num 30) dirty exists 2024-09-20T22:42:58.076 INFO:tasks.rados.rados.0.smithi084.stdout:31: left oid 31 (ObjNum 30 snap 0 seq_num 30) 2024-09-20T22:42:58.076 INFO:tasks.rados.rados.0.smithi084.stdout:32: finishing write tid 1 to smithi08477203-32 2024-09-20T22:42:58.076 INFO:tasks.rados.rados.0.smithi084.stdout:32: finishing write tid 2 to smithi08477203-32 2024-09-20T22:42:58.076 INFO:tasks.rados.rados.0.smithi084.stdout:31: done (15 left) 2024-09-20T22:42:58.076 INFO:tasks.rados.rados.0.smithi084.stdout:47: write initial oid 47 2024-09-20T22:42:58.076 INFO:tasks.rados.rados.0.smithi084.stdout:47: seq_num 46 ranges {773122=559572,1992054=651681} 2024-09-20T22:42:58.082 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:42:58.083 INFO:tasks.rados.rados.0.smithi084.stdout:47: writing smithi08477203-47 from 773122 to 1332694 tid 1 2024-09-20T22:42:58.091 INFO:tasks.rados.rados.0.smithi084.stdout:47: writing smithi08477203-47 from 1992054 to 2643735 tid 2 2024-09-20T22:42:58.091 INFO:tasks.rados.rados.0.smithi084.stdout: waiting on 16 2024-09-20T22:42:58.091 INFO:tasks.rados.rados.0.smithi084.stdout:32: finishing write tid 3 to smithi08477203-32 2024-09-20T22:42:58.091 INFO:tasks.rados.rados.0.smithi084.stdout:32: finishing write tid 4 to smithi08477203-32 2024-09-20T22:42:58.091 INFO:tasks.rados.rados.0.smithi084.stdout:32: oid 32 updating version 0 to 3 2024-09-20T22:42:58.091 INFO:tasks.rados.rados.0.smithi084.stdout:32: oid 32 version 3 is already newer than 2 2024-09-20T22:42:58.092 INFO:tasks.rados.rados.0.smithi084.stdout:32: oid 32 version 3 is already newer than 1 2024-09-20T22:42:58.092 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 32 v 3 (ObjNum 31 snap 0 seq_num 31) dirty exists 2024-09-20T22:42:58.092 INFO:tasks.rados.rados.0.smithi084.stdout:32: left oid 32 (ObjNum 31 snap 0 seq_num 31) 2024-09-20T22:42:58.092 INFO:tasks.rados.rados.0.smithi084.stdout:33: finishing write tid 1 to smithi08477203-33 2024-09-20T22:42:58.092 INFO:tasks.rados.rados.0.smithi084.stdout:33: finishing write tid 2 to smithi08477203-33 2024-09-20T22:42:58.092 INFO:tasks.rados.rados.0.smithi084.stdout:33: finishing write tid 3 to smithi08477203-33 2024-09-20T22:42:58.092 INFO:tasks.rados.rados.0.smithi084.stdout:33: finishing write tid 4 to smithi08477203-33 2024-09-20T22:42:58.092 INFO:tasks.rados.rados.0.smithi084.stdout:33: oid 33 updating version 0 to 4 2024-09-20T22:42:58.092 INFO:tasks.rados.rados.0.smithi084.stdout:33: oid 33 updating version 4 to 6 2024-09-20T22:42:58.092 INFO:tasks.rados.rados.0.smithi084.stdout:33: oid 33 version 6 is already newer than 5 2024-09-20T22:42:58.092 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 33 v 6 (ObjNum 32 snap 0 seq_num 32) dirty exists 2024-09-20T22:42:58.092 INFO:tasks.rados.rados.0.smithi084.stdout:33: left oid 33 (ObjNum 32 snap 0 seq_num 32) 2024-09-20T22:42:58.092 INFO:tasks.rados.rados.0.smithi084.stdout:34: finishing write tid 1 to smithi08477203-34 2024-09-20T22:42:58.092 INFO:tasks.rados.rados.0.smithi084.stdout:34: finishing write tid 2 to smithi08477203-34 2024-09-20T22:42:58.092 INFO:tasks.rados.rados.0.smithi084.stdout:34: finishing write tid 3 to smithi08477203-34 2024-09-20T22:42:58.093 INFO:tasks.rados.rados.0.smithi084.stdout:32: done (15 left) 2024-09-20T22:42:58.093 INFO:tasks.rados.rados.0.smithi084.stdout:33: done (14 left) 2024-09-20T22:42:58.093 INFO:tasks.rados.rados.0.smithi084.stdout:48: write initial oid 48 2024-09-20T22:42:58.093 INFO:tasks.rados.rados.0.smithi084.stdout:34: finishing write tid 4 to smithi08477203-34 2024-09-20T22:42:58.093 INFO:tasks.rados.rados.0.smithi084.stdout:34: oid 34 updating version 0 to 9 2024-09-20T22:42:58.093 INFO:tasks.rados.rados.0.smithi084.stdout:34: oid 34 version 9 is already newer than 8 2024-09-20T22:42:58.093 INFO:tasks.rados.rados.0.smithi084.stdout:34: oid 34 version 9 is already newer than 7 2024-09-20T22:42:58.093 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 34 v 9 (ObjNum 33 snap 0 seq_num 33) dirty exists 2024-09-20T22:42:58.093 INFO:tasks.rados.rados.0.smithi084.stdout:34: left oid 34 (ObjNum 33 snap 0 seq_num 33) 2024-09-20T22:42:58.093 INFO:tasks.rados.rados.0.smithi084.stdout:35: finishing write tid 1 to smithi08477203-35 2024-09-20T22:42:58.093 INFO:tasks.rados.rados.0.smithi084.stdout:35: finishing write tid 2 to smithi08477203-35 2024-09-20T22:42:58.093 INFO:tasks.rados.rados.0.smithi084.stdout:35: finishing write tid 3 to smithi08477203-35 2024-09-20T22:42:58.093 INFO:tasks.rados.rados.0.smithi084.stdout:35: finishing write tid 4 to smithi08477203-35 2024-09-20T22:42:58.093 INFO:tasks.rados.rados.0.smithi084.stdout:35: oid 35 updating version 0 to 7 2024-09-20T22:42:58.094 INFO:tasks.rados.rados.0.smithi084.stdout:35: oid 35 updating version 7 to 8 2024-09-20T22:42:58.094 INFO:tasks.rados.rados.0.smithi084.stdout:35: oid 35 updating version 8 to 9 2024-09-20T22:42:58.094 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 35 v 9 (ObjNum 34 snap 0 seq_num 34) dirty exists 2024-09-20T22:42:58.094 INFO:tasks.rados.rados.0.smithi084.stdout:35: left oid 35 (ObjNum 34 snap 0 seq_num 34) 2024-09-20T22:42:58.094 INFO:tasks.rados.rados.0.smithi084.stdout:36: finishing write tid 1 to smithi08477203-36 2024-09-20T22:42:58.094 INFO:tasks.rados.rados.0.smithi084.stdout:48: seq_num 47 ranges {789929=634821,2050536=739393} 2024-09-20T22:42:58.098 INFO:tasks.rados.rados.0.smithi084.stdout:48: writing smithi08477203-48 from 789929 to 1424750 tid 1 2024-09-20T22:42:58.105 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:42:58.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:42:58.106 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:42:58.106 INFO:tasks.rados.rados.0.smithi084.stdout:48: writing smithi08477203-48 from 2050536 to 2789929 tid 2 2024-09-20T22:42:58.106 INFO:tasks.rados.rados.0.smithi084.stdout:34: done (14 left) 2024-09-20T22:42:58.106 INFO:tasks.rados.rados.0.smithi084.stdout:35: done (13 left) 2024-09-20T22:42:58.106 INFO:tasks.rados.rados.0.smithi084.stdout:49: write initial oid 49 2024-09-20T22:42:58.107 INFO:tasks.rados.rados.0.smithi084.stdout:49: seq_num 48 ranges {406736=710070,1709018=655350} 2024-09-20T22:42:58.112 INFO:tasks.rados.rados.0.smithi084.stdout:49: writing smithi08477203-49 from 406736 to 1116806 tid 1 2024-09-20T22:42:58.119 INFO:tasks.rados.rados.0.smithi084.stdout:49: writing smithi08477203-49 from 1709018 to 2364368 tid 2 2024-09-20T22:42:58.119 INFO:tasks.rados.rados.0.smithi084.stdout:50: write initial oid 50 2024-09-20T22:42:58.119 INFO:tasks.rados.rados.0.smithi084.stdout:50: seq_num 49 ranges {423543=785319,1767500=799008} 2024-09-20T22:42:58.125 INFO:tasks.rados.rados.0.smithi084.stdout:50: writing smithi08477203-50 from 423543 to 1208862 tid 1 2024-09-20T22:42:58.134 INFO:tasks.rados.rados.0.smithi084.stdout:50: writing smithi08477203-50 from 1767500 to 2566508 tid 2 2024-09-20T22:42:58.134 INFO:tasks.rados.rados.0.smithi084.stdout:51: copy_from oid 2 from oid 11 current snap is 0 2024-09-20T22:42:58.134 INFO:tasks.rados.rados.0.smithi084.stdout:36: finishing write tid 2 to smithi08477203-36 2024-09-20T22:42:58.134 INFO:tasks.rados.rados.0.smithi084.stdout:36: finishing write tid 3 to smithi08477203-36 2024-09-20T22:42:58.134 INFO:tasks.rados.rados.0.smithi084.stdout:36: finishing write tid 4 to smithi08477203-36 2024-09-20T22:42:58.134 INFO:tasks.rados.rados.0.smithi084.stdout:36: oid 36 updating version 0 to 3 2024-09-20T22:42:58.134 INFO:tasks.rados.rados.0.smithi084.stdout:36: oid 36 version 3 is already newer than 2 2024-09-20T22:42:58.134 INFO:tasks.rados.rados.0.smithi084.stdout:36: oid 36 version 3 is already newer than 1 2024-09-20T22:42:58.134 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 36 v 3 (ObjNum 35 snap 0 seq_num 35) dirty exists 2024-09-20T22:42:58.134 INFO:tasks.rados.rados.0.smithi084.stdout:36: left oid 36 (ObjNum 35 snap 0 seq_num 35) 2024-09-20T22:42:58.135 INFO:tasks.rados.rados.0.smithi084.stdout:37: finishing write tid 1 to smithi08477203-37 2024-09-20T22:42:58.135 INFO:tasks.rados.rados.0.smithi084.stdout:36: done (15 left) 2024-09-20T22:42:58.135 INFO:tasks.rados.rados.0.smithi084.stdout:52: write oid 25 current snap is 0 2024-09-20T22:42:58.135 INFO:tasks.rados.rados.0.smithi084.stdout:37: finishing write tid 2 to smithi08477203-37 2024-09-20T22:42:58.135 INFO:tasks.rados.rados.0.smithi084.stdout:37: finishing write tid 3 to smithi08477203-37 2024-09-20T22:42:58.135 INFO:tasks.rados.rados.0.smithi084.stdout:52: seq_num 50 ranges {440350=460568,1309629=542666,2323973=516377} 2024-09-20T22:42:58.139 INFO:tasks.rados.rados.0.smithi084.stdout:52: writing smithi08477203-25 from 440350 to 900918 tid 1 2024-09-20T22:42:58.144 INFO:tasks.rados.rados.0.smithi084.stdout:52: writing smithi08477203-25 from 1309629 to 1852295 tid 2 2024-09-20T22:42:58.151 INFO:tasks.rados.rados.0.smithi084.stdout:52: writing smithi08477203-25 from 2323973 to 2840350 tid 3 2024-09-20T22:42:58.152 INFO:tasks.rados.rados.0.smithi084.stdout: waiting on 16 2024-09-20T22:42:58.152 INFO:tasks.rados.rados.0.smithi084.stdout:37: finishing write tid 4 to smithi08477203-37 2024-09-20T22:42:58.152 INFO:tasks.rados.rados.0.smithi084.stdout:37: oid 37 updating version 0 to 6 2024-09-20T22:42:58.152 INFO:tasks.rados.rados.0.smithi084.stdout:37: oid 37 version 6 is already newer than 5 2024-09-20T22:42:58.152 INFO:tasks.rados.rados.0.smithi084.stdout:37: oid 37 version 6 is already newer than 4 2024-09-20T22:42:58.152 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 37 v 6 (ObjNum 36 snap 0 seq_num 36) dirty exists 2024-09-20T22:42:58.153 INFO:tasks.rados.rados.0.smithi084.stdout:37: left oid 37 (ObjNum 36 snap 0 seq_num 36) 2024-09-20T22:42:58.153 INFO:tasks.rados.rados.0.smithi084.stdout:38: finishing write tid 1 to smithi08477203-38 2024-09-20T22:42:58.153 INFO:tasks.rados.rados.0.smithi084.stdout:38: finishing write tid 2 to smithi08477203-38 2024-09-20T22:42:58.153 INFO:tasks.rados.rados.0.smithi084.stdout:37: done (15 left) 2024-09-20T22:42:58.153 INFO:tasks.rados.rados.0.smithi084.stdout:53: rollback oid 30 current snap is 0 2024-09-20T22:42:58.153 INFO:tasks.rados.rados.0.smithi084.stdout:53: done (15 left) 2024-09-20T22:42:58.153 INFO:tasks.rados.rados.0.smithi084.stdout:54: write (excl) oid 21 current snap is 0 2024-09-20T22:42:58.153 INFO:tasks.rados.rados.0.smithi084.stdout:38: finishing write tid 3 to smithi08477203-38 2024-09-20T22:42:58.153 INFO:tasks.rados.rados.0.smithi084.stdout:38: finishing write tid 4 to smithi08477203-38 2024-09-20T22:42:58.153 INFO:tasks.rados.rados.0.smithi084.stdout:38: oid 38 updating version 0 to 12 2024-09-20T22:42:58.153 INFO:tasks.rados.rados.0.smithi084.stdout:38: oid 38 version 12 is already newer than 11 2024-09-20T22:42:58.153 INFO:tasks.rados.rados.0.smithi084.stdout:38: oid 38 version 12 is already newer than 10 2024-09-20T22:42:58.153 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 38 v 12 (ObjNum 37 snap 0 seq_num 37) dirty exists 2024-09-20T22:42:58.154 INFO:tasks.rados.rados.0.smithi084.stdout:38: left oid 38 (ObjNum 37 snap 0 seq_num 37) 2024-09-20T22:42:58.154 INFO:tasks.rados.rados.0.smithi084.stdout:39: finishing write tid 1 to smithi08477203-39 2024-09-20T22:42:58.154 INFO:tasks.rados.rados.0.smithi084.stdout:39: finishing write tid 2 to smithi08477203-39 2024-09-20T22:42:58.154 INFO:tasks.rados.rados.0.smithi084.stdout:54: seq_num 51 ranges {457157=535817,1768111=402677} 2024-09-20T22:42:58.156 INFO:tasks.rados.rados.0.smithi084.stdout:54: writing smithi08477203-21 from 457157 to 992974 tid 1 2024-09-20T22:42:58.164 INFO:tasks.rados.rados.0.smithi084.stdout:54: writing smithi08477203-21 from 1768111 to 2170788 tid 2 2024-09-20T22:42:58.164 INFO:tasks.rados.rados.0.smithi084.stdout:38: done (15 left) 2024-09-20T22:42:58.164 INFO:tasks.rados.rados.0.smithi084.stdout:55: snap_create 2024-09-20T22:42:58.164 INFO:tasks.rados.rados.0.smithi084.stdout:39: finishing write tid 3 to smithi08477203-39 2024-09-20T22:42:58.164 INFO:tasks.rados.rados.0.smithi084.stdout:39: finishing write tid 4 to smithi08477203-39 2024-09-20T22:42:58.164 INFO:tasks.rados.rados.0.smithi084.stdout:39: oid 39 updating version 0 to 15 2024-09-20T22:42:58.164 INFO:tasks.rados.rados.0.smithi084.stdout:39: oid 39 version 15 is already newer than 13 2024-09-20T22:42:58.164 INFO:tasks.rados.rados.0.smithi084.stdout:39: oid 39 version 15 is already newer than 14 2024-09-20T22:42:58.164 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 39 v 15 (ObjNum 38 snap 0 seq_num 38) dirty exists 2024-09-20T22:42:58.164 INFO:tasks.rados.rados.0.smithi084.stdout:39: left oid 39 (ObjNum 38 snap 0 seq_num 38) 2024-09-20T22:42:58.164 INFO:tasks.rados.rados.0.smithi084.stdout:40: finishing write tid 1 to smithi08477203-40 2024-09-20T22:42:58.164 INFO:tasks.rados.rados.0.smithi084.stdout:40: finishing write tid 2 to smithi08477203-40 2024-09-20T22:42:58.165 INFO:tasks.rados.rados.0.smithi084.stdout:40: finishing write tid 3 to smithi08477203-40 2024-09-20T22:42:58.165 INFO:tasks.rados.rados.0.smithi084.stdout:40: finishing write tid 4 to smithi08477203-40 2024-09-20T22:42:58.165 INFO:tasks.rados.rados.0.smithi084.stdout:40: oid 40 updating version 0 to 7 2024-09-20T22:42:58.165 INFO:tasks.rados.rados.0.smithi084.stdout:40: oid 40 updating version 7 to 9 2024-09-20T22:42:58.165 INFO:tasks.rados.rados.0.smithi084.stdout:40: oid 40 version 9 is already newer than 8 2024-09-20T22:42:58.165 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 40 v 9 (ObjNum 39 snap 0 seq_num 39) dirty exists 2024-09-20T22:42:58.165 INFO:tasks.rados.rados.0.smithi084.stdout:40: left oid 40 (ObjNum 39 snap 0 seq_num 39) 2024-09-20T22:42:58.165 INFO:tasks.rados.rados.0.smithi084.stdout:41: finishing write tid 1 to smithi08477203-41 2024-09-20T22:42:58.165 INFO:tasks.rados.rados.0.smithi084.stdout:41: finishing write tid 2 to smithi08477203-41 2024-09-20T22:42:58.165 INFO:tasks.rados.rados.0.smithi084.stdout:41: finishing write tid 3 to smithi08477203-41 2024-09-20T22:42:58.165 INFO:tasks.rados.rados.0.smithi084.stdout:41: finishing write tid 4 to smithi08477203-41 2024-09-20T22:42:58.165 INFO:tasks.rados.rados.0.smithi084.stdout:41: oid 41 updating version 0 to 1 2024-09-20T22:42:58.165 INFO:tasks.rados.rados.0.smithi084.stdout:41: oid 41 updating version 1 to 2 2024-09-20T22:42:58.165 INFO:tasks.rados.rados.0.smithi084.stdout:41: oid 41 updating version 2 to 3 2024-09-20T22:42:58.166 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 41 v 3 (ObjNum 40 snap 0 seq_num 40) dirty exists 2024-09-20T22:42:58.166 INFO:tasks.rados.rados.0.smithi084.stdout:41: left oid 41 (ObjNum 40 snap 0 seq_num 40) 2024-09-20T22:42:58.166 INFO:tasks.rados.rados.0.smithi084.stdout:42: finishing write tid 1 to smithi08477203-42 2024-09-20T22:42:58.166 INFO:tasks.rados.rados.0.smithi084.stdout:42: finishing write tid 2 to smithi08477203-42 2024-09-20T22:42:58.166 INFO:tasks.rados.rados.0.smithi084.stdout:42: finishing write tid 3 to smithi08477203-42 2024-09-20T22:42:58.166 INFO:tasks.rados.rados.0.smithi084.stdout:42: finishing write tid 4 to smithi08477203-42 2024-09-20T22:42:58.166 INFO:tasks.rados.rados.0.smithi084.stdout:42: oid 42 updating version 0 to 5 2024-09-20T22:42:58.166 INFO:tasks.rados.rados.0.smithi084.stdout:42: oid 42 version 5 is already newer than 4 2024-09-20T22:42:58.166 INFO:tasks.rados.rados.0.smithi084.stdout:42: oid 42 updating version 5 to 6 2024-09-20T22:42:58.166 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 42 v 6 (ObjNum 41 snap 0 seq_num 41) dirty exists 2024-09-20T22:42:58.166 INFO:tasks.rados.rados.0.smithi084.stdout:42: left oid 42 (ObjNum 41 snap 0 seq_num 41) 2024-09-20T22:42:58.166 INFO:tasks.rados.rados.0.smithi084.stdout:43: finishing write tid 1 to smithi08477203-43 2024-09-20T22:42:58.166 INFO:tasks.rados.rados.0.smithi084.stdout:43: finishing write tid 2 to smithi08477203-43 2024-09-20T22:42:58.166 INFO:tasks.rados.rados.0.smithi084.stdout:43: finishing write tid 3 to smithi08477203-43 2024-09-20T22:42:58.167 INFO:tasks.rados.rados.0.smithi084.stdout:43: finishing write tid 4 to smithi08477203-43 2024-09-20T22:42:58.167 INFO:tasks.rados.rados.0.smithi084.stdout:43: oid 43 updating version 0 to 4 2024-09-20T22:42:58.167 INFO:tasks.rados.rados.0.smithi084.stdout:43: oid 43 updating version 4 to 6 2024-09-20T22:42:58.167 INFO:tasks.rados.rados.0.smithi084.stdout:43: oid 43 version 6 is already newer than 5 2024-09-20T22:42:58.167 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 43 v 6 (ObjNum 42 snap 0 seq_num 42) dirty exists 2024-09-20T22:42:58.167 INFO:tasks.rados.rados.0.smithi084.stdout:43: left oid 43 (ObjNum 42 snap 0 seq_num 42) 2024-09-20T22:42:58.167 INFO:tasks.rados.rados.0.smithi084.stdout:44: finishing write tid 1 to smithi08477203-44 2024-09-20T22:42:58.167 INFO:tasks.rados.rados.0.smithi084.stdout:44: finishing write tid 2 to smithi08477203-44 2024-09-20T22:42:58.167 INFO:tasks.rados.rados.0.smithi084.stdout:44: finishing write tid 3 to smithi08477203-44 2024-09-20T22:42:58.167 INFO:tasks.rados.rados.0.smithi084.stdout:44: finishing write tid 4 to smithi08477203-44 2024-09-20T22:42:58.167 INFO:tasks.rados.rados.0.smithi084.stdout:44: oid 44 updating version 0 to 19 2024-09-20T22:42:58.167 INFO:tasks.rados.rados.0.smithi084.stdout:44: oid 44 updating version 19 to 21 2024-09-20T22:42:58.167 INFO:tasks.rados.rados.0.smithi084.stdout:44: oid 44 version 21 is already newer than 20 2024-09-20T22:42:58.167 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 44 v 21 (ObjNum 43 snap 0 seq_num 43) dirty exists 2024-09-20T22:42:58.168 INFO:tasks.rados.rados.0.smithi084.stdout:44: left oid 44 (ObjNum 43 snap 0 seq_num 43) 2024-09-20T22:42:58.168 INFO:tasks.rados.rados.0.smithi084.stdout:45: finishing write tid 1 to smithi08477203-45 2024-09-20T22:42:58.168 INFO:tasks.rados.rados.0.smithi084.stdout:45: finishing write tid 2 to smithi08477203-45 2024-09-20T22:42:58.168 INFO:tasks.rados.rados.0.smithi084.stdout:45: finishing write tid 3 to smithi08477203-45 2024-09-20T22:42:58.168 INFO:tasks.rados.rados.0.smithi084.stdout:45: finishing write tid 4 to smithi08477203-45 2024-09-20T22:42:58.168 INFO:tasks.rados.rados.0.smithi084.stdout:45: oid 45 updating version 0 to 4 2024-09-20T22:42:58.168 INFO:tasks.rados.rados.0.smithi084.stdout:45: oid 45 updating version 4 to 5 2024-09-20T22:42:58.168 INFO:tasks.rados.rados.0.smithi084.stdout:45: oid 45 updating version 5 to 6 2024-09-20T22:42:58.168 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 45 v 6 (ObjNum 44 snap 0 seq_num 44) dirty exists 2024-09-20T22:42:58.168 INFO:tasks.rados.rados.0.smithi084.stdout:45: left oid 45 (ObjNum 44 snap 0 seq_num 44) 2024-09-20T22:42:58.168 INFO:tasks.rados.rados.0.smithi084.stdout:46: finishing write tid 1 to smithi08477203-46 2024-09-20T22:42:58.168 INFO:tasks.rados.rados.0.smithi084.stdout:46: finishing write tid 2 to smithi08477203-46 2024-09-20T22:42:58.168 INFO:tasks.rados.rados.0.smithi084.stdout:46: finishing write tid 3 to smithi08477203-46 2024-09-20T22:42:58.168 INFO:tasks.rados.rados.0.smithi084.stdout:46: finishing write tid 4 to smithi08477203-46 2024-09-20T22:42:58.169 INFO:tasks.rados.rados.0.smithi084.stdout:46: oid 46 updating version 0 to 15 2024-09-20T22:42:58.169 INFO:tasks.rados.rados.0.smithi084.stdout:46: oid 46 version 15 is already newer than 13 2024-09-20T22:42:58.169 INFO:tasks.rados.rados.0.smithi084.stdout:46: oid 46 version 15 is already newer than 14 2024-09-20T22:42:58.169 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 46 v 15 (ObjNum 45 snap 0 seq_num 45) dirty exists 2024-09-20T22:42:58.169 INFO:tasks.rados.rados.0.smithi084.stdout:46: left oid 46 (ObjNum 45 snap 0 seq_num 45) 2024-09-20T22:42:58.169 INFO:tasks.rados.rados.0.smithi084.stdout:47: finishing write tid 1 to smithi08477203-47 2024-09-20T22:42:58.169 INFO:tasks.rados.rados.0.smithi084.stdout:47: finishing write tid 2 to smithi08477203-47 2024-09-20T22:42:58.169 INFO:tasks.rados.rados.0.smithi084.stdout:47: finishing write tid 3 to smithi08477203-47 2024-09-20T22:42:58.169 INFO:tasks.rados.rados.0.smithi084.stdout:47: finishing write tid 4 to smithi08477203-47 2024-09-20T22:42:58.169 INFO:tasks.rados.rados.0.smithi084.stdout:47: oid 47 updating version 0 to 14 2024-09-20T22:42:58.169 INFO:tasks.rados.rados.0.smithi084.stdout:47: oid 47 updating version 14 to 15 2024-09-20T22:42:58.169 INFO:tasks.rados.rados.0.smithi084.stdout:47: oid 47 version 15 is already newer than 13 2024-09-20T22:42:58.169 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 47 v 15 (ObjNum 46 snap 0 seq_num 46) dirty exists 2024-09-20T22:42:58.169 INFO:tasks.rados.rados.0.smithi084.stdout:47: left oid 47 (ObjNum 46 snap 0 seq_num 46) 2024-09-20T22:42:58.170 INFO:tasks.rados.rados.0.smithi084.stdout:48: finishing write tid 1 to smithi08477203-48 2024-09-20T22:42:58.170 INFO:tasks.rados.rados.0.smithi084.stdout:48: finishing write tid 2 to smithi08477203-48 2024-09-20T22:42:58.170 INFO:tasks.rados.rados.0.smithi084.stdout:48: finishing write tid 3 to smithi08477203-48 2024-09-20T22:42:58.170 INFO:tasks.rados.rados.0.smithi084.stdout:48: finishing write tid 4 to smithi08477203-48 2024-09-20T22:42:58.170 INFO:tasks.rados.rados.0.smithi084.stdout:48: oid 48 updating version 0 to 8 2024-09-20T22:42:58.170 INFO:tasks.rados.rados.0.smithi084.stdout:48: oid 48 updating version 8 to 9 2024-09-20T22:42:58.170 INFO:tasks.rados.rados.0.smithi084.stdout:48: oid 48 version 9 is already newer than 7 2024-09-20T22:42:58.170 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 48 v 9 (ObjNum 47 snap 0 seq_num 47) dirty exists 2024-09-20T22:42:58.170 INFO:tasks.rados.rados.0.smithi084.stdout:48: left oid 48 (ObjNum 47 snap 0 seq_num 47) 2024-09-20T22:42:58.170 INFO:tasks.rados.rados.0.smithi084.stdout:49: finishing write tid 1 to smithi08477203-49 2024-09-20T22:42:58.170 INFO:tasks.rados.rados.0.smithi084.stdout:49: finishing write tid 2 to smithi08477203-49 2024-09-20T22:42:58.170 INFO:tasks.rados.rados.0.smithi084.stdout:49: finishing write tid 3 to smithi08477203-49 2024-09-20T22:42:58.170 INFO:tasks.rados.rados.0.smithi084.stdout:49: finishing write tid 4 to smithi08477203-49 2024-09-20T22:42:58.170 INFO:tasks.rados.rados.0.smithi084.stdout:49: oid 49 updating version 0 to 10 2024-09-20T22:42:58.170 INFO:tasks.rados.rados.0.smithi084.stdout:49: oid 49 updating version 10 to 11 2024-09-20T22:42:58.171 INFO:tasks.rados.rados.0.smithi084.stdout:49: oid 49 updating version 11 to 12 2024-09-20T22:42:58.171 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 49 v 12 (ObjNum 48 snap 0 seq_num 48) dirty exists 2024-09-20T22:42:58.171 INFO:tasks.rados.rados.0.smithi084.stdout:49: left oid 49 (ObjNum 48 snap 0 seq_num 48) 2024-09-20T22:42:58.171 INFO:tasks.rados.rados.0.smithi084.stdout:50: finishing write tid 1 to smithi08477203-50 2024-09-20T22:42:58.171 INFO:tasks.rados.rados.0.smithi084.stdout:50: finishing write tid 2 to smithi08477203-50 2024-09-20T22:42:58.171 INFO:tasks.rados.rados.0.smithi084.stdout:50: finishing write tid 3 to smithi08477203-50 2024-09-20T22:42:58.171 INFO:tasks.rados.rados.0.smithi084.stdout:50: finishing write tid 4 to smithi08477203-50 2024-09-20T22:42:58.171 INFO:tasks.rados.rados.0.smithi084.stdout:50: oid 50 updating version 0 to 9 2024-09-20T22:42:58.171 INFO:tasks.rados.rados.0.smithi084.stdout:50: oid 50 version 9 is already newer than 7 2024-09-20T22:42:58.171 INFO:tasks.rados.rados.0.smithi084.stdout:50: oid 50 version 9 is already newer than 8 2024-09-20T22:42:58.171 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 50 v 9 (ObjNum 49 snap 0 seq_num 49) dirty exists 2024-09-20T22:42:58.171 INFO:tasks.rados.rados.0.smithi084.stdout:50: left oid 50 (ObjNum 49 snap 0 seq_num 49) 2024-09-20T22:42:58.171 INFO:tasks.rados.rados.0.smithi084.stdout:51: finishing copy_from racing read to smithi08477203-2 2024-09-20T22:42:58.171 INFO:tasks.rados.rados.0.smithi084.stdout:51: finishing copy_from to smithi08477203-2 2024-09-20T22:42:58.171 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 2 v 6 (ObjNum 10 snap 0 seq_num 10) dirty exists 2024-09-20T22:42:58.172 INFO:tasks.rados.rados.0.smithi084.stdout:52: finishing write tid 1 to smithi08477203-25 2024-09-20T22:42:58.172 INFO:tasks.rados.rados.0.smithi084.stdout:52: finishing write tid 2 to smithi08477203-25 2024-09-20T22:42:58.172 INFO:tasks.rados.rados.0.smithi084.stdout:52: finishing write tid 3 to smithi08477203-25 2024-09-20T22:42:58.172 INFO:tasks.rados.rados.0.smithi084.stdout:52: finishing write tid 4 to smithi08477203-25 2024-09-20T22:42:58.172 INFO:tasks.rados.rados.0.smithi084.stdout:52: finishing write tid 5 to smithi08477203-25 2024-09-20T22:42:58.172 INFO:tasks.rados.rados.0.smithi084.stdout:52: oid 25 updating version 0 to 25 2024-09-20T22:42:58.172 INFO:tasks.rados.rados.0.smithi084.stdout:52: oid 25 version 25 is already newer than 22 2024-09-20T22:42:58.172 INFO:tasks.rados.rados.0.smithi084.stdout:52: oid 25 version 25 is already newer than 23 2024-09-20T22:42:58.172 INFO:tasks.rados.rados.0.smithi084.stdout:52: oid 25 version 25 is already newer than 24 2024-09-20T22:42:58.172 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 25 v 25 (ObjNum 50 snap 0 seq_num 50) dirty exists 2024-09-20T22:42:58.172 INFO:tasks.rados.rados.0.smithi084.stdout:52: left oid 25 (ObjNum 50 snap 0 seq_num 50) 2024-09-20T22:42:58.172 INFO:tasks.rados.rados.0.smithi084.stdout:54: finishing write tid 1 to smithi08477203-21 2024-09-20T22:42:58.173 INFO:tasks.rados.rados.0.smithi084.stdout:54: finishing write tid 2 to smithi08477203-21 2024-09-20T22:42:58.173 INFO:tasks.rados.rados.0.smithi084.stdout:54: finishing write tid 3 to smithi08477203-21 2024-09-20T22:42:58.173 INFO:tasks.rados.rados.0.smithi084.stdout:54: finishing write tid 4 to smithi08477203-21 2024-09-20T22:42:58.173 INFO:tasks.rados.rados.0.smithi084.stdout:54: oid 21 updating version 0 to 14 2024-09-20T22:42:58.173 INFO:tasks.rados.rados.0.smithi084.stdout:54: oid 21 updating version 14 to 15 2024-09-20T22:42:58.174 INFO:tasks.rados.rados.0.smithi084.stdout:54: oid 21 updating version 15 to 16 2024-09-20T22:42:58.174 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 21 v 16 (ObjNum 51 snap 0 seq_num 51) dirty exists 2024-09-20T22:42:58.174 INFO:tasks.rados.rados.0.smithi084.stdout:54: left oid 21 (ObjNum 51 snap 0 seq_num 51) 2024-09-20T22:42:58.206 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:42:58.228 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:42:58.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:42:58.229 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:42:58.329 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:42:58.352 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:42:58.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:42:58.353 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:42:58.354 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:42:58 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: ::ffff:172.21.15.154 - - [20/Sep/2024:22:42:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-20T22:42:58.454 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:42:58.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:42:58.476 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:42:58.476 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:42:58.501 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is unset 2024-09-20T22:42:58.501 INFO:tasks.rados.rados.0.smithi084.stdout:39: done (15 left) 2024-09-20T22:42:58.501 INFO:tasks.rados.rados.0.smithi084.stdout:40: done (14 left) 2024-09-20T22:42:58.501 INFO:tasks.rados.rados.0.smithi084.stdout:41: done (13 left) 2024-09-20T22:42:58.502 INFO:tasks.rados.rados.0.smithi084.stdout:42: done (12 left) 2024-09-20T22:42:58.502 INFO:tasks.rados.rados.0.smithi084.stdout:43: done (11 left) 2024-09-20T22:42:58.502 INFO:tasks.rados.rados.0.smithi084.stdout:44: done (10 left) 2024-09-20T22:42:58.502 INFO:tasks.rados.rados.0.smithi084.stdout:45: done (9 left) 2024-09-20T22:42:58.502 INFO:tasks.rados.rados.0.smithi084.stdout:46: done (8 left) 2024-09-20T22:42:58.502 INFO:tasks.rados.rados.0.smithi084.stdout:47: done (7 left) 2024-09-20T22:42:58.502 INFO:tasks.rados.rados.0.smithi084.stdout:48: done (6 left) 2024-09-20T22:42:58.502 INFO:tasks.rados.rados.0.smithi084.stdout:49: done (5 left) 2024-09-20T22:42:58.502 INFO:tasks.rados.rados.0.smithi084.stdout:50: done (4 left) 2024-09-20T22:42:58.502 INFO:tasks.rados.rados.0.smithi084.stdout:51: done (3 left) 2024-09-20T22:42:58.502 INFO:tasks.rados.rados.0.smithi084.stdout:52: done (2 left) 2024-09-20T22:42:58.502 INFO:tasks.rados.rados.0.smithi084.stdout:54: done (1 left) 2024-09-20T22:42:58.502 INFO:tasks.rados.rados.0.smithi084.stdout:55: done (0 left) 2024-09-20T22:42:58.502 INFO:tasks.rados.rados.0.smithi084.stdout:56: delete oid 34 current snap is 1 2024-09-20T22:42:58.503 INFO:tasks.rados.rados.0.smithi084.stdout:56: done (0 left) 2024-09-20T22:42:58.504 INFO:tasks.rados.rados.0.smithi084.stdout:57: snap_create 2024-09-20T22:42:58.577 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:42:58.587 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-20T22:42:58.587 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:42:58.601 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:42:58.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:42:58.602 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:42:58.614 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:58 smithi084 ceph-mon[40024]: from='client.? ' entity='client.0' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "write_fadvise_dontneed", "val": "false"}]': finished 2024-09-20T22:42:58.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:58 smithi084 ceph-mon[40024]: osdmap e98: 8 total, 8 up, 8 in 2024-09-20T22:42:58.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:58 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/1097899925' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:42:58.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:58 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:42:58.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:58 smithi084 ceph-mon[29774]: from='client.? ' entity='client.0' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "write_fadvise_dontneed", "val": "false"}]': finished 2024-09-20T22:42:58.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:58 smithi084 ceph-mon[29774]: osdmap e98: 8 total, 8 up, 8 in 2024-09-20T22:42:58.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:58 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/1097899925' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:42:58.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:58 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:42:58.703 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:42:58.725 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:42:58.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:42:58.726 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:42:58.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:58 smithi154 ceph-mon[33264]: from='client.? ' entity='client.0' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "write_fadvise_dontneed", "val": "false"}]': finished 2024-09-20T22:42:58.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:58 smithi154 ceph-mon[33264]: osdmap e98: 8 total, 8 up, 8 in 2024-09-20T22:42:58.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:58 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/1097899925' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:42:58.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:58 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:42:58.827 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:42:58.850 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:42:58.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:42:58.851 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:42:58.951 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:42:58.973 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:42:58.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:42:58.974 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:42:59.074 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:42:59.096 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:42:59.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:42:59.097 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:42:59.197 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:42:59.219 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:42:59.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:42:59.220 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:42:59.254 INFO:journalctl@ceph.iscsi.iscsi.a.smithi154.stdout:Sep 20 22:42:58 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-iscsi-iscsi-a[59730]: debug there is no tcmu-runner data available 2024-09-20T22:42:59.321 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:42:59.342 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:42:59.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:42:59.343 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:42:59.444 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:42:59.466 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:42:59.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:42:59.467 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:42:59.497 INFO:tasks.rados.rados.0.smithi084.stdout:57: done (0 left) 2024-09-20T22:42:59.497 INFO:tasks.rados.rados.0.smithi084.stdout:58: write oid 33 current snap is 2 2024-09-20T22:42:59.498 INFO:tasks.rados.rados.0.smithi084.stdout:58: seq_num 53 ranges {490771=686315,1885075=406346} 2024-09-20T22:42:59.504 INFO:tasks.rados.rados.0.smithi084.stdout:58: writing smithi08477203-33 from 490771 to 1177086 tid 1 2024-09-20T22:42:59.511 INFO:tasks.rados.rados.0.smithi084.stdout:58: writing smithi08477203-33 from 1885075 to 2291421 tid 2 2024-09-20T22:42:59.511 INFO:tasks.rados.rados.0.smithi084.stdout:59: delete oid 3 current snap is 2 2024-09-20T22:42:59.511 INFO:tasks.rados.rados.0.smithi084.stdout:58: finishing write tid 1 to smithi08477203-33 2024-09-20T22:42:59.513 INFO:tasks.rados.rados.0.smithi084.stdout:59: done (1 left) 2024-09-20T22:42:59.513 INFO:tasks.rados.rados.0.smithi084.stdout:60: write oid 12 current snap is 2 2024-09-20T22:42:59.513 INFO:tasks.rados.rados.0.smithi084.stdout:60: seq_num 55 ranges {524385=553166,1602039=410015,2577441=346944} 2024-09-20T22:42:59.519 INFO:tasks.rados.rados.0.smithi084.stdout:60: writing smithi08477203-12 from 524385 to 1077551 tid 1 2024-09-20T22:42:59.523 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:59 smithi084 ceph-mon[40024]: pgmap v49: 158 pgs: 158 active+clean; 583 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 32 KiB/s rd, 252 KiB/s wr, 40 op/s; 291 KiB/s, 1 objects/s recovering 2024-09-20T22:42:59.523 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:59 smithi084 ceph-mon[40024]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:42:59.523 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:59 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T22:42:59.523 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:59 smithi084 ceph-mon[40024]: osdmap e99: 8 total, 8 up, 8 in 2024-09-20T22:42:59.523 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:59 smithi084 ceph-mon[40024]: 6.16 scrub starts 2024-09-20T22:42:59.523 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:59 smithi084 ceph-mon[40024]: 6.16 scrub ok 2024-09-20T22:42:59.523 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:59 smithi084 ceph-mon[40024]: 1.5 scrub starts 2024-09-20T22:42:59.523 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:59 smithi084 ceph-mon[40024]: 1.5 scrub ok 2024-09-20T22:42:59.524 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:59 smithi084 ceph-mon[40024]: 1.7 scrub starts 2024-09-20T22:42:59.524 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:59 smithi084 ceph-mon[40024]: 1.7 scrub ok 2024-09-20T22:42:59.524 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:59 smithi084 ceph-mon[40024]: 1.0 scrub starts 2024-09-20T22:42:59.524 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:42:59 smithi084 ceph-mon[40024]: 1.0 scrub ok 2024-09-20T22:42:59.524 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:59 smithi084 ceph-mon[29774]: pgmap v49: 158 pgs: 158 active+clean; 583 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 32 KiB/s rd, 252 KiB/s wr, 40 op/s; 291 KiB/s, 1 objects/s recovering 2024-09-20T22:42:59.524 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:59 smithi084 ceph-mon[29774]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:42:59.524 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:59 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T22:42:59.524 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:59 smithi084 ceph-mon[29774]: osdmap e99: 8 total, 8 up, 8 in 2024-09-20T22:42:59.524 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:59 smithi084 ceph-mon[29774]: 6.16 scrub starts 2024-09-20T22:42:59.524 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:59 smithi084 ceph-mon[29774]: 6.16 scrub ok 2024-09-20T22:42:59.525 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:59 smithi084 ceph-mon[29774]: 1.5 scrub starts 2024-09-20T22:42:59.525 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:59 smithi084 ceph-mon[29774]: 1.5 scrub ok 2024-09-20T22:42:59.525 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:59 smithi084 ceph-mon[29774]: 1.7 scrub starts 2024-09-20T22:42:59.525 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:59 smithi084 ceph-mon[29774]: 1.7 scrub ok 2024-09-20T22:42:59.525 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:59 smithi084 ceph-mon[29774]: 1.0 scrub starts 2024-09-20T22:42:59.525 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:42:59 smithi084 ceph-mon[29774]: 1.0 scrub ok 2024-09-20T22:42:59.526 INFO:tasks.rados.rados.0.smithi084.stdout:60: writing smithi08477203-12 from 1602039 to 2012054 tid 2 2024-09-20T22:42:59.531 INFO:tasks.rados.rados.0.smithi084.stdout:60: writing smithi08477203-12 from 2577441 to 2924385 tid 3 2024-09-20T22:42:59.531 INFO:tasks.rados.rados.0.smithi084.stdout:61: rollback oid 47 current snap is 2 2024-09-20T22:42:59.531 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 47 to 1 2024-09-20T22:42:59.531 INFO:tasks.rados.rados.0.smithi084.stdout:58: finishing write tid 2 to smithi08477203-33 2024-09-20T22:42:59.531 INFO:tasks.rados.rados.0.smithi084.stdout:58: finishing write tid 3 to smithi08477203-33 2024-09-20T22:42:59.531 INFO:tasks.rados.rados.0.smithi084.stdout:58: finishing write tid 4 to smithi08477203-33 2024-09-20T22:42:59.531 INFO:tasks.rados.rados.0.smithi084.stdout:58: oid 33 updating version 0 to 11 2024-09-20T22:42:59.531 INFO:tasks.rados.rados.0.smithi084.stdout:58: oid 33 updating version 11 to 13 2024-09-20T22:42:59.531 INFO:tasks.rados.rados.0.smithi084.stdout:58: oid 33 version 13 is already newer than 12 2024-09-20T22:42:59.531 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 33 v 13 (ObjNum 53 snap 2 seq_num 53) dirty exists 2024-09-20T22:42:59.531 INFO:tasks.rados.rados.0.smithi084.stdout:58: left oid 33 (ObjNum 53 snap 2 seq_num 53) 2024-09-20T22:42:59.531 INFO:tasks.rados.rados.0.smithi084.stdout:60: finishing write tid 1 to smithi08477203-12 2024-09-20T22:42:59.531 INFO:tasks.rados.rados.0.smithi084.stdout:60: finishing write tid 2 to smithi08477203-12 2024-09-20T22:42:59.531 INFO:tasks.rados.rados.0.smithi084.stdout:58: done (2 left) 2024-09-20T22:42:59.532 INFO:tasks.rados.rados.0.smithi084.stdout:62: snap_remove snap 0 2024-09-20T22:42:59.567 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:42:59.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:42:59.589 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:42:59.589 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:42:59.690 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:42:59.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:42:59.713 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:42:59.713 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:42:59.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:59 smithi154 ceph-mon[33264]: pgmap v49: 158 pgs: 158 active+clean; 583 KiB data, 236 MiB used, 715 GiB / 715 GiB avail; 32 KiB/s rd, 252 KiB/s wr, 40 op/s; 291 KiB/s, 1 objects/s recovering 2024-09-20T22:42:59.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:59 smithi154 ceph-mon[33264]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:42:59.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:59 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T22:42:59.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:59 smithi154 ceph-mon[33264]: osdmap e99: 8 total, 8 up, 8 in 2024-09-20T22:42:59.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:59 smithi154 ceph-mon[33264]: 6.16 scrub starts 2024-09-20T22:42:59.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:59 smithi154 ceph-mon[33264]: 6.16 scrub ok 2024-09-20T22:42:59.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:59 smithi154 ceph-mon[33264]: 1.5 scrub starts 2024-09-20T22:42:59.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:59 smithi154 ceph-mon[33264]: 1.5 scrub ok 2024-09-20T22:42:59.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:59 smithi154 ceph-mon[33264]: 1.7 scrub starts 2024-09-20T22:42:59.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:59 smithi154 ceph-mon[33264]: 1.7 scrub ok 2024-09-20T22:42:59.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:59 smithi154 ceph-mon[33264]: 1.0 scrub starts 2024-09-20T22:42:59.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:42:59 smithi154 ceph-mon[33264]: 1.0 scrub ok 2024-09-20T22:42:59.814 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:42:59.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:42:59.836 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:42:59.836 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:42:59.937 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:42:59.958 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:42:59.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:42:59.960 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:42:59.993 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.1 dump_historic_ops 2024-09-20T22:43:00.060 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:43:00.098 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:00.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:00.100 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:43:00.118 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-20T22:43:00.201 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:43:00.223 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:00.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:00.224 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:43:00.325 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:43:00.347 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:00.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:00.348 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:43:00.449 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:43:00.469 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:00.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:00.471 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:43:00.510 INFO:tasks.rados.rados.0.smithi084.stdout:62: done (2 left) 2024-09-20T22:43:00.510 INFO:tasks.rados.rados.0.smithi084.stdout:63: read oid 28 snap -1 2024-09-20T22:43:00.510 INFO:tasks.rados.rados.0.smithi084.stdout:63: expect (ObjNum 27 snap 0 seq_num 27) 2024-09-20T22:43:00.510 INFO:tasks.rados.rados.0.smithi084.stdout:64: read oid 33 snap -1 2024-09-20T22:43:00.510 INFO:tasks.rados.rados.0.smithi084.stdout:64: expect (ObjNum 53 snap 2 seq_num 53) 2024-09-20T22:43:00.510 INFO:tasks.rados.rados.0.smithi084.stdout:65: delete oid 2 current snap is 2 2024-09-20T22:43:00.510 INFO:tasks.rados.rados.0.smithi084.stdout:60: finishing write tid 3 to smithi08477203-12 2024-09-20T22:43:00.510 INFO:tasks.rados.rados.0.smithi084.stdout:60: finishing write tid 4 to smithi08477203-12 2024-09-20T22:43:00.510 INFO:tasks.rados.rados.0.smithi084.stdout:60: finishing write tid 5 to smithi08477203-12 2024-09-20T22:43:00.511 INFO:tasks.rados.rados.0.smithi084.stdout:60: oid 12 updating version 0 to 18 2024-09-20T22:43:00.511 INFO:tasks.rados.rados.0.smithi084.stdout:60: oid 12 updating version 18 to 20 2024-09-20T22:43:00.511 INFO:tasks.rados.rados.0.smithi084.stdout:60: oid 12 version 20 is already newer than 19 2024-09-20T22:43:00.511 INFO:tasks.rados.rados.0.smithi084.stdout:60: oid 12 version 20 is already newer than 17 2024-09-20T22:43:00.511 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 12 v 20 (ObjNum 55 snap 2 seq_num 55) dirty exists 2024-09-20T22:43:00.511 INFO:tasks.rados.rados.0.smithi084.stdout:60: left oid 12 (ObjNum 55 snap 2 seq_num 55) 2024-09-20T22:43:00.511 INFO:tasks.rados.rados.0.smithi084.stdout:61: finishing rollback tid 0 to smithi08477203-47 2024-09-20T22:43:00.511 INFO:tasks.rados.rados.0.smithi084.stdout:61: finishing rollback tid 1 to smithi08477203-47 2024-09-20T22:43:00.511 INFO:tasks.rados.rados.0.smithi084.stdout:61: finishing rollback tid 2 to smithi08477203-47 2024-09-20T22:43:00.511 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 47 v 24 (ObjNum 46 snap 0 seq_num 46) dirty exists 2024-09-20T22:43:00.521 INFO:tasks.rados.rados.0.smithi084.stdout:63: expect (ObjNum 27 snap 0 seq_num 27) 2024-09-20T22:43:00.553 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:00 smithi084 ceph-mon[29774]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:43:00.553 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:00 smithi084 ceph-mon[29774]: 6.f scrub starts 2024-09-20T22:43:00.553 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:00 smithi084 ceph-mon[29774]: 6.f scrub ok 2024-09-20T22:43:00.553 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:00 smithi084 ceph-mon[29774]: osdmap e100: 8 total, 8 up, 8 in 2024-09-20T22:43:00.553 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:00 smithi084 ceph-mon[29774]: 1.6 scrub starts 2024-09-20T22:43:00.553 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:00 smithi084 ceph-mon[29774]: 1.6 scrub ok 2024-09-20T22:43:00.553 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:00 smithi084 ceph-mon[29774]: 4.9 scrub starts 2024-09-20T22:43:00.553 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:00 smithi084 ceph-mon[29774]: 4.9 scrub ok 2024-09-20T22:43:00.553 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:00 smithi084 ceph-mon[29774]: 6.4 scrub starts 2024-09-20T22:43:00.554 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:00 smithi084 ceph-mon[29774]: 6.4 scrub ok 2024-09-20T22:43:00.554 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:00 smithi084 ceph-mon[40024]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:43:00.554 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:00 smithi084 ceph-mon[40024]: 6.f scrub starts 2024-09-20T22:43:00.554 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:00 smithi084 ceph-mon[40024]: 6.f scrub ok 2024-09-20T22:43:00.554 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:00 smithi084 ceph-mon[40024]: osdmap e100: 8 total, 8 up, 8 in 2024-09-20T22:43:00.554 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:00 smithi084 ceph-mon[40024]: 1.6 scrub starts 2024-09-20T22:43:00.554 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:00 smithi084 ceph-mon[40024]: 1.6 scrub ok 2024-09-20T22:43:00.554 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:00 smithi084 ceph-mon[40024]: 4.9 scrub starts 2024-09-20T22:43:00.554 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:00 smithi084 ceph-mon[40024]: 4.9 scrub ok 2024-09-20T22:43:00.554 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:00 smithi084 ceph-mon[40024]: 6.4 scrub starts 2024-09-20T22:43:00.554 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:00 smithi084 ceph-mon[40024]: 6.4 scrub ok 2024-09-20T22:43:00.572 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:43:00.593 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:00.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:00.594 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:43:00.602 INFO:tasks.rados.rados.0.smithi084.stdout:60: done (4 left) 2024-09-20T22:43:00.603 INFO:tasks.rados.rados.0.smithi084.stdout:61: done (3 left) 2024-09-20T22:43:00.603 INFO:tasks.rados.rados.0.smithi084.stdout:63: done (2 left) 2024-09-20T22:43:00.603 INFO:tasks.rados.rados.0.smithi084.stdout:65: done (1 left) 2024-09-20T22:43:00.603 INFO:tasks.rados.rados.0.smithi084.stdout:66: copy_from oid 27 from oid 22 current snap is 2 2024-09-20T22:43:00.603 INFO:tasks.rados.rados.0.smithi084.stdout:64: expect (ObjNum 53 snap 2 seq_num 53) 2024-09-20T22:43:00.695 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:43:00.716 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:00.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:00.717 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:43:00.739 INFO:tasks.rados.rados.0.smithi084.stdout:66: finishing copy_from racing read to smithi08477203-27 2024-09-20T22:43:00.739 INFO:tasks.rados.rados.0.smithi084.stdout:66: finishing copy_from to smithi08477203-27 2024-09-20T22:43:00.739 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 27 v 3 (ObjNum 21 snap 0 seq_num 21) dirty exists 2024-09-20T22:43:00.739 INFO:tasks.rados.rados.0.smithi084.stdout:64: done (1 left) 2024-09-20T22:43:00.739 INFO:tasks.rados.rados.0.smithi084.stdout:66: done (0 left) 2024-09-20T22:43:00.739 INFO:tasks.rados.rados.0.smithi084.stdout:67: snap_create 2024-09-20T22:43:00.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:00 smithi154 ceph-mon[33264]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:43:00.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:00 smithi154 ceph-mon[33264]: 6.f scrub starts 2024-09-20T22:43:00.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:00 smithi154 ceph-mon[33264]: 6.f scrub ok 2024-09-20T22:43:00.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:00 smithi154 ceph-mon[33264]: osdmap e100: 8 total, 8 up, 8 in 2024-09-20T22:43:00.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:00 smithi154 ceph-mon[33264]: 1.6 scrub starts 2024-09-20T22:43:00.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:00 smithi154 ceph-mon[33264]: 1.6 scrub ok 2024-09-20T22:43:00.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:00 smithi154 ceph-mon[33264]: 4.9 scrub starts 2024-09-20T22:43:00.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:00 smithi154 ceph-mon[33264]: 4.9 scrub ok 2024-09-20T22:43:00.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:00 smithi154 ceph-mon[33264]: 6.4 scrub starts 2024-09-20T22:43:00.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:00 smithi154 ceph-mon[33264]: 6.4 scrub ok 2024-09-20T22:43:00.818 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:43:00.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:00.841 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:00.841 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:43:00.942 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:43:00.963 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:00.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:00.964 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:43:01.065 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:43:01.086 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:01.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:01.087 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:43:01.188 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:43:01.209 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:01.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:01.210 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:43:01.311 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:43:01.333 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:01.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:01.334 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:43:01.435 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:43:01.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:01.457 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:01.458 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:43:01.504 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-20T22:43:01.504 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-20T22:43:01.504 INFO:tasks.thrashosds.thrasher:Growing pool .mgr 2024-09-20T22:43:01.505 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph pg dump --format=json 2024-09-20T22:43:01.518 INFO:tasks.rados.rados.0.smithi084.stdout:67: done (0 left) 2024-09-20T22:43:01.518 INFO:tasks.rados.rados.0.smithi084.stdout:68: snap_remove snap 1 2024-09-20T22:43:01.527 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-20T22:43:01.559 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:43:01.591 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:01.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:01.592 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:43:01.657 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:43:01.658 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:43:01.693 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:43:01.719 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:01.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:01.720 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:43:01.786 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:01 smithi084 ceph-mon[29774]: pgmap v52: 158 pgs: 158 active+clean; 117 MiB data, 363 MiB used, 715 GiB / 715 GiB avail; 11 KiB/s rd, 11 MiB/s wr, 61 op/s 2024-09-20T22:43:01.786 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:01 smithi084 ceph-mon[29774]: 5.15 scrub starts 2024-09-20T22:43:01.786 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:01 smithi084 ceph-mon[29774]: 5.15 scrub ok 2024-09-20T22:43:01.786 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:01 smithi084 ceph-mon[29774]: 3.18 scrub starts 2024-09-20T22:43:01.786 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:01 smithi084 ceph-mon[29774]: 3.18 scrub ok 2024-09-20T22:43:01.786 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:01 smithi084 ceph-mon[29774]: osdmap e101: 8 total, 8 up, 8 in 2024-09-20T22:43:01.787 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:01 smithi084 ceph-mon[29774]: 4.b scrub starts 2024-09-20T22:43:01.787 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:01 smithi084 ceph-mon[29774]: 4.b scrub ok 2024-09-20T22:43:01.787 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:01 smithi084 ceph-mon[29774]: 4.d scrub starts 2024-09-20T22:43:01.787 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:01 smithi084 ceph-mon[29774]: 4.d scrub ok 2024-09-20T22:43:01.787 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:01 smithi084 ceph-mon[29774]: 5.1a scrub starts 2024-09-20T22:43:01.787 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:01 smithi084 ceph-mon[29774]: 5.1a scrub ok 2024-09-20T22:43:01.787 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:01 smithi084 ceph-mon[40024]: pgmap v52: 158 pgs: 158 active+clean; 117 MiB data, 363 MiB used, 715 GiB / 715 GiB avail; 11 KiB/s rd, 11 MiB/s wr, 61 op/s 2024-09-20T22:43:01.787 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:01 smithi084 ceph-mon[40024]: 5.15 scrub starts 2024-09-20T22:43:01.789 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:01 smithi084 ceph-mon[40024]: 5.15 scrub ok 2024-09-20T22:43:01.789 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:01 smithi084 ceph-mon[40024]: 3.18 scrub starts 2024-09-20T22:43:01.789 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:01 smithi084 ceph-mon[40024]: 3.18 scrub ok 2024-09-20T22:43:01.789 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:01 smithi084 ceph-mon[40024]: osdmap e101: 8 total, 8 up, 8 in 2024-09-20T22:43:01.789 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:01 smithi084 ceph-mon[40024]: 4.b scrub starts 2024-09-20T22:43:01.789 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:01 smithi084 ceph-mon[40024]: 4.b scrub ok 2024-09-20T22:43:01.789 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:01 smithi084 ceph-mon[40024]: 4.d scrub starts 2024-09-20T22:43:01.789 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:01 smithi084 ceph-mon[40024]: 4.d scrub ok 2024-09-20T22:43:01.789 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:01 smithi084 ceph-mon[40024]: 5.1a scrub starts 2024-09-20T22:43:01.789 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:01 smithi084 ceph-mon[40024]: 5.1a scrub ok 2024-09-20T22:43:01.821 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:43:01.842 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:01.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:01.843 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:43:01.944 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:43:01.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:01.969 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:01.969 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:43:02.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:01 smithi154 ceph-mon[33264]: pgmap v52: 158 pgs: 158 active+clean; 117 MiB data, 363 MiB used, 715 GiB / 715 GiB avail; 11 KiB/s rd, 11 MiB/s wr, 61 op/s 2024-09-20T22:43:02.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:01 smithi154 ceph-mon[33264]: 5.15 scrub starts 2024-09-20T22:43:02.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:01 smithi154 ceph-mon[33264]: 5.15 scrub ok 2024-09-20T22:43:02.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:01 smithi154 ceph-mon[33264]: 3.18 scrub starts 2024-09-20T22:43:02.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:01 smithi154 ceph-mon[33264]: 3.18 scrub ok 2024-09-20T22:43:02.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:01 smithi154 ceph-mon[33264]: osdmap e101: 8 total, 8 up, 8 in 2024-09-20T22:43:02.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:01 smithi154 ceph-mon[33264]: 4.b scrub starts 2024-09-20T22:43:02.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:01 smithi154 ceph-mon[33264]: 4.b scrub ok 2024-09-20T22:43:02.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:01 smithi154 ceph-mon[33264]: 4.d scrub starts 2024-09-20T22:43:02.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:01 smithi154 ceph-mon[33264]: 4.d scrub ok 2024-09-20T22:43:02.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:01 smithi154 ceph-mon[33264]: 5.1a scrub starts 2024-09-20T22:43:02.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:01 smithi154 ceph-mon[33264]: 5.1a scrub ok 2024-09-20T22:43:02.070 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:43:02.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:02.092 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:02.092 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:43:02.193 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:43:02.216 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:02.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:02.217 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:43:02.286 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-20T22:43:02.287 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:43:02.318 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:43:02.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:02.340 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:02.340 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:43:02.441 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:43:02.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:02.464 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:02.465 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:43:02.495 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:43:02.514 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:02 smithi084 ceph-mon[29774]: 3.1c scrub starts 2024-09-20T22:43:02.514 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:02 smithi084 ceph-mon[29774]: 3.1c scrub ok 2024-09-20T22:43:02.514 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:02 smithi084 ceph-mon[29774]: osdmap e102: 8 total, 8 up, 8 in 2024-09-20T22:43:02.514 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:02 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:43:02.514 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:02 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:43:02.515 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:02 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:43:02.515 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:02 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:43:02.515 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:02 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:43:02.515 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:02 smithi084 ceph-mon[29774]: 4.3 scrub starts 2024-09-20T22:43:02.515 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:02 smithi084 ceph-mon[29774]: 4.3 scrub ok 2024-09-20T22:43:02.530 INFO:tasks.rados.rados.0.smithi084.stdout:68: done (0 left) 2024-09-20T22:43:02.530 INFO:tasks.rados.rados.0.smithi084.stdout:69: read oid 23 snap -1 2024-09-20T22:43:02.530 INFO:tasks.rados.rados.0.smithi084.stdout:69: expect (ObjNum 22 snap 0 seq_num 22) 2024-09-20T22:43:02.531 INFO:tasks.rados.rados.0.smithi084.stdout:70: read oid 24 snap -1 2024-09-20T22:43:02.531 INFO:tasks.rados.rados.0.smithi084.stdout:70: expect (ObjNum 23 snap 0 seq_num 23) 2024-09-20T22:43:02.531 INFO:tasks.rados.rados.0.smithi084.stdout:71: read oid 41 snap -1 2024-09-20T22:43:02.531 INFO:tasks.rados.rados.0.smithi084.stdout:71: expect (ObjNum 40 snap 0 seq_num 40) 2024-09-20T22:43:02.531 INFO:tasks.rados.rados.0.smithi084.stdout:72: snap_remove snap 2 2024-09-20T22:43:02.565 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:43:02.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:02.587 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:02.587 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:43:02.688 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:43:02.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:02.710 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:02.711 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:43:02.811 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:43:02.833 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:02.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:02.834 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:43:02.864 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:02 smithi084 ceph-mon[29774]: 3.10 scrub starts 2024-09-20T22:43:02.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:02 smithi084 ceph-mon[29774]: 3.10 scrub ok 2024-09-20T22:43:02.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:02 smithi084 ceph-mon[29774]: 6.8 scrub starts 2024-09-20T22:43:02.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:02 smithi084 ceph-mon[29774]: 6.8 scrub ok 2024-09-20T22:43:02.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:02 smithi084 ceph-mon[40024]: 3.1c scrub starts 2024-09-20T22:43:02.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:02 smithi084 ceph-mon[40024]: 3.1c scrub ok 2024-09-20T22:43:02.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:02 smithi084 ceph-mon[40024]: osdmap e102: 8 total, 8 up, 8 in 2024-09-20T22:43:02.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:02 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:43:02.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:02 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:43:02.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:02 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:43:02.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:02 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:43:02.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:02 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:43:02.866 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:02 smithi084 ceph-mon[40024]: 4.3 scrub starts 2024-09-20T22:43:02.866 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:02 smithi084 ceph-mon[40024]: 4.3 scrub ok 2024-09-20T22:43:02.866 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:02 smithi084 ceph-mon[40024]: 3.10 scrub starts 2024-09-20T22:43:02.866 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:02 smithi084 ceph-mon[40024]: 3.10 scrub ok 2024-09-20T22:43:02.866 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:02 smithi084 ceph-mon[40024]: 6.8 scrub starts 2024-09-20T22:43:02.866 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:02 smithi084 ceph-mon[40024]: 6.8 scrub ok 2024-09-20T22:43:02.934 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:43:02.956 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:02.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:02.957 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:43:02.990 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-20T22:43:03.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:02 smithi154 ceph-mon[33264]: 3.1c scrub starts 2024-09-20T22:43:03.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:02 smithi154 ceph-mon[33264]: 3.1c scrub ok 2024-09-20T22:43:03.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:02 smithi154 ceph-mon[33264]: osdmap e102: 8 total, 8 up, 8 in 2024-09-20T22:43:03.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:02 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:43:03.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:02 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:43:03.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:02 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:43:03.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:02 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:43:03.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:02 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:43:03.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:02 smithi154 ceph-mon[33264]: 4.3 scrub starts 2024-09-20T22:43:03.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:02 smithi154 ceph-mon[33264]: 4.3 scrub ok 2024-09-20T22:43:03.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:02 smithi154 ceph-mon[33264]: 3.10 scrub starts 2024-09-20T22:43:03.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:02 smithi154 ceph-mon[33264]: 3.10 scrub ok 2024-09-20T22:43:03.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:02 smithi154 ceph-mon[33264]: 6.8 scrub starts 2024-09-20T22:43:03.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:02 smithi154 ceph-mon[33264]: 6.8 scrub ok 2024-09-20T22:43:03.058 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:43:03.079 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:03.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:03.080 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:43:03.110 INFO:tasks.rados.rados.0.smithi084.stdout:72: done (3 left) 2024-09-20T22:43:03.110 INFO:tasks.rados.rados.0.smithi084.stdout:73: write oid 47 current snap is 3 2024-09-20T22:43:03.110 INFO:tasks.rados.rados.0.smithi084.stdout:73: seq_num 57 ranges {557999=703664,1719003=413684,2632287=325712} 2024-09-20T22:43:03.118 INFO:tasks.rados.rados.0.smithi084.stdout:73: writing smithi08477203-47 from 557999 to 1261663 tid 1 2024-09-20T22:43:03.123 INFO:tasks.rados.rados.0.smithi084.stdout:73: writing smithi08477203-47 from 1719003 to 2132687 tid 2 2024-09-20T22:43:03.129 INFO:tasks.rados.rados.0.smithi084.stdout:73: writing smithi08477203-47 from 2632287 to 2957999 tid 3 2024-09-20T22:43:03.129 INFO:tasks.rados.rados.0.smithi084.stdout:74: copy_from oid 48 from oid 25 current snap is 3 2024-09-20T22:43:03.129 INFO:tasks.rados.rados.0.smithi084.stdout:71: expect (ObjNum 40 snap 0 seq_num 40) 2024-09-20T22:43:03.181 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:43:03.202 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:03.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:03.203 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:43:03.224 INFO:tasks.rados.rados.0.smithi084.stdout:70: expect (ObjNum 23 snap 0 seq_num 23) 2024-09-20T22:43:03.304 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:43:03.305 INFO:tasks.rados.rados.0.smithi084.stdout:69: expect (ObjNum 22 snap 0 seq_num 22) 2024-09-20T22:43:03.325 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:03.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:03.326 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:43:03.383 INFO:tasks.rados.rados.0.smithi084.stdout:73: finishing write tid 1 to smithi08477203-47 2024-09-20T22:43:03.383 INFO:tasks.rados.rados.0.smithi084.stdout:73: finishing write tid 2 to smithi08477203-47 2024-09-20T22:43:03.383 INFO:tasks.rados.rados.0.smithi084.stdout:73: finishing write tid 3 to smithi08477203-47 2024-09-20T22:43:03.383 INFO:tasks.rados.rados.0.smithi084.stdout:73: finishing write tid 4 to smithi08477203-47 2024-09-20T22:43:03.384 INFO:tasks.rados.rados.0.smithi084.stdout:73: finishing write tid 5 to smithi08477203-47 2024-09-20T22:43:03.384 INFO:tasks.rados.rados.0.smithi084.stdout:73: oid 47 updating version 0 to 33 2024-09-20T22:43:03.384 INFO:tasks.rados.rados.0.smithi084.stdout:73: oid 47 updating version 33 to 35 2024-09-20T22:43:03.384 INFO:tasks.rados.rados.0.smithi084.stdout:73: oid 47 updating version 35 to 36 2024-09-20T22:43:03.384 INFO:tasks.rados.rados.0.smithi084.stdout:73: oid 47 version 36 is already newer than 34 2024-09-20T22:43:03.384 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 47 v 36 (ObjNum 57 snap 3 seq_num 57) dirty exists 2024-09-20T22:43:03.384 INFO:tasks.rados.rados.0.smithi084.stdout:73: left oid 47 (ObjNum 57 snap 3 seq_num 57) 2024-09-20T22:43:03.384 INFO:tasks.rados.rados.0.smithi084.stdout:74: finishing copy_from racing read to smithi08477203-48 2024-09-20T22:43:03.384 INFO:tasks.rados.rados.0.smithi084.stdout:74: finishing copy_from to smithi08477203-48 2024-09-20T22:43:03.384 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 48 v 25 (ObjNum 50 snap 0 seq_num 50) dirty exists 2024-09-20T22:43:03.384 INFO:tasks.rados.rados.0.smithi084.stdout:69: done (4 left) 2024-09-20T22:43:03.384 INFO:tasks.rados.rados.0.smithi084.stdout:70: done (3 left) 2024-09-20T22:43:03.384 INFO:tasks.rados.rados.0.smithi084.stdout:71: done (2 left) 2024-09-20T22:43:03.384 INFO:tasks.rados.rados.0.smithi084.stdout:73: done (1 left) 2024-09-20T22:43:03.384 INFO:tasks.rados.rados.0.smithi084.stdout:74: done (0 left) 2024-09-20T22:43:03.385 INFO:tasks.rados.rados.0.smithi084.stdout:75: copy_from oid 50 from oid 22 current snap is 3 2024-09-20T22:43:03.385 INFO:tasks.rados.rados.0.smithi084.stdout:76: snap_create 2024-09-20T22:43:03.391 INFO:tasks.rados.rados.0.smithi084.stdout:75: finishing copy_from racing read to smithi08477203-50 2024-09-20T22:43:03.397 INFO:tasks.rados.rados.0.smithi084.stdout:75: finishing copy_from to smithi08477203-50 2024-09-20T22:43:03.397 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 50 v 3 (ObjNum 21 snap 0 seq_num 21) dirty exists 2024-09-20T22:43:03.426 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:43:03.448 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:03.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:03.449 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:43:03.550 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:43:03.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:03.573 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:03.573 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:43:03.674 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:43:03.695 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:03.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:03.696 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:43:03.797 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:43:03.818 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:03.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:03.819 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:43:03.864 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:03 smithi084 ceph-mon[40024]: pgmap v55: 158 pgs: 158 active+clean; 117 MiB data, 363 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 15 MiB/s wr, 79 op/s 2024-09-20T22:43:03.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:03 smithi084 ceph-mon[40024]: Reconfiguring alertmanager.a (dependencies changed)... 2024-09-20T22:43:03.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:03 smithi084 ceph-mon[40024]: Reconfiguring daemon alertmanager.a on smithi084 2024-09-20T22:43:03.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:03 smithi084 ceph-mon[40024]: 5.8 scrub starts 2024-09-20T22:43:03.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:03 smithi084 ceph-mon[40024]: 5.8 scrub ok 2024-09-20T22:43:03.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:03 smithi084 ceph-mon[40024]: 1.4 scrub starts 2024-09-20T22:43:03.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:03 smithi084 ceph-mon[40024]: 1.4 scrub ok 2024-09-20T22:43:03.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:03 smithi084 ceph-mon[40024]: osdmap e103: 8 total, 8 up, 8 in 2024-09-20T22:43:03.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:03 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:43:03.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:03 smithi084 ceph-mon[40024]: 5.18 scrub starts 2024-09-20T22:43:03.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:03 smithi084 ceph-mon[40024]: 5.18 scrub ok 2024-09-20T22:43:03.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:03 smithi084 ceph-mon[40024]: osdmap e104: 8 total, 8 up, 8 in 2024-09-20T22:43:03.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:03 smithi084 ceph-mon[40024]: 1.9 scrub starts 2024-09-20T22:43:03.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:03 smithi084 ceph-mon[40024]: 1.9 scrub ok 2024-09-20T22:43:03.866 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:03 smithi084 ceph-mon[29774]: pgmap v55: 158 pgs: 158 active+clean; 117 MiB data, 363 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 15 MiB/s wr, 79 op/s 2024-09-20T22:43:03.866 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:03 smithi084 ceph-mon[29774]: Reconfiguring alertmanager.a (dependencies changed)... 2024-09-20T22:43:03.866 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:03 smithi084 ceph-mon[29774]: Reconfiguring daemon alertmanager.a on smithi084 2024-09-20T22:43:03.866 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:03 smithi084 ceph-mon[29774]: 5.8 scrub starts 2024-09-20T22:43:03.866 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:03 smithi084 ceph-mon[29774]: 5.8 scrub ok 2024-09-20T22:43:03.866 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:03 smithi084 ceph-mon[29774]: 1.4 scrub starts 2024-09-20T22:43:03.866 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:03 smithi084 ceph-mon[29774]: 1.4 scrub ok 2024-09-20T22:43:03.866 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:03 smithi084 ceph-mon[29774]: osdmap e103: 8 total, 8 up, 8 in 2024-09-20T22:43:03.866 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:03 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:43:03.866 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:03 smithi084 ceph-mon[29774]: 5.18 scrub starts 2024-09-20T22:43:03.866 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:03 smithi084 ceph-mon[29774]: 5.18 scrub ok 2024-09-20T22:43:03.867 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:03 smithi084 ceph-mon[29774]: osdmap e104: 8 total, 8 up, 8 in 2024-09-20T22:43:03.867 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:03 smithi084 ceph-mon[29774]: 1.9 scrub starts 2024-09-20T22:43:03.867 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:03 smithi084 ceph-mon[29774]: 1.9 scrub ok 2024-09-20T22:43:03.920 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:43:03.942 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:03.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:03.942 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:43:04.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:03 smithi154 ceph-mon[33264]: pgmap v55: 158 pgs: 158 active+clean; 117 MiB data, 363 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 15 MiB/s wr, 79 op/s 2024-09-20T22:43:04.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:03 smithi154 ceph-mon[33264]: Reconfiguring alertmanager.a (dependencies changed)... 2024-09-20T22:43:04.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:03 smithi154 ceph-mon[33264]: Reconfiguring daemon alertmanager.a on smithi084 2024-09-20T22:43:04.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:03 smithi154 ceph-mon[33264]: 5.8 scrub starts 2024-09-20T22:43:04.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:03 smithi154 ceph-mon[33264]: 5.8 scrub ok 2024-09-20T22:43:04.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:03 smithi154 ceph-mon[33264]: 1.4 scrub starts 2024-09-20T22:43:04.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:03 smithi154 ceph-mon[33264]: 1.4 scrub ok 2024-09-20T22:43:04.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:03 smithi154 ceph-mon[33264]: osdmap e103: 8 total, 8 up, 8 in 2024-09-20T22:43:04.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:03 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:43:04.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:03 smithi154 ceph-mon[33264]: 5.18 scrub starts 2024-09-20T22:43:04.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:03 smithi154 ceph-mon[33264]: 5.18 scrub ok 2024-09-20T22:43:04.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:03 smithi154 ceph-mon[33264]: osdmap e104: 8 total, 8 up, 8 in 2024-09-20T22:43:04.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:03 smithi154 ceph-mon[33264]: 1.9 scrub starts 2024-09-20T22:43:04.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:03 smithi154 ceph-mon[33264]: 1.9 scrub ok 2024-09-20T22:43:04.043 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:43:04.065 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:04.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:04.066 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:43:04.111 INFO:tasks.rados.rados.0.smithi084.stdout:75: done (1 left) 2024-09-20T22:43:04.111 INFO:tasks.rados.rados.0.smithi084.stdout:76: done (0 left) 2024-09-20T22:43:04.112 INFO:tasks.rados.rados.0.smithi084.stdout:77: copy_from oid 22 from oid 49 current snap is 4 2024-09-20T22:43:04.112 INFO:tasks.rados.rados.0.smithi084.stdout:78: read oid 39 snap 3 2024-09-20T22:43:04.112 INFO:tasks.rados.rados.0.smithi084.stdout:78: expect (ObjNum 38 snap 0 seq_num 38) 2024-09-20T22:43:04.112 INFO:tasks.rados.rados.0.smithi084.stdout:79: write oid 30 current snap is 4 2024-09-20T22:43:04.112 INFO:tasks.rados.rados.0.smithi084.stdout:79: seq_num 58 ranges {574806=778913,2061132=557342} 2024-09-20T22:43:04.119 INFO:tasks.rados.rados.0.smithi084.stdout:79: writing smithi08477203-30 from 574806 to 1353719 tid 1 2024-09-20T22:43:04.126 INFO:tasks.rados.rados.0.smithi084.stdout:79: writing smithi08477203-30 from 2061132 to 2618474 tid 2 2024-09-20T22:43:04.126 INFO:tasks.rados.rados.0.smithi084.stdout:80: write (excl) oid 29 current snap is 4 2024-09-20T22:43:04.127 INFO:tasks.rados.rados.0.smithi084.stdout:80: seq_num 59 ranges {591613=454162,1719614=417353,2570780=420833} 2024-09-20T22:43:04.132 INFO:tasks.rados.rados.0.smithi084.stdout:80: writing smithi08477203-29 from 591613 to 1045775 tid 1 2024-09-20T22:43:04.138 INFO:tasks.rados.rados.0.smithi084.stdout:80: writing smithi08477203-29 from 1719614 to 2136967 tid 2 2024-09-20T22:43:04.143 INFO:tasks.rados.rados.0.smithi084.stdout:80: writing smithi08477203-29 from 2570780 to 2991613 tid 3 2024-09-20T22:43:04.144 INFO:tasks.rados.rados.0.smithi084.stdout:81: read oid 36 snap 3 2024-09-20T22:43:04.144 INFO:tasks.rados.rados.0.smithi084.stdout:81: expect (ObjNum 35 snap 0 seq_num 35) 2024-09-20T22:43:04.144 INFO:tasks.rados.rados.0.smithi084.stdout:82: copy_from oid 42 from oid 28 current snap is 4 2024-09-20T22:43:04.144 INFO:tasks.rados.rados.0.smithi084.stdout:78: expect (ObjNum 38 snap 0 seq_num 38) 2024-09-20T22:43:04.166 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:43:04.188 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:04.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:04.189 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:43:04.234 INFO:tasks.rados.rados.0.smithi084.stdout:77: finishing copy_from racing read to smithi08477203-22 2024-09-20T22:43:04.235 INFO:tasks.rados.rados.0.smithi084.stdout:79: finishing write tid 1 to smithi08477203-30 2024-09-20T22:43:04.235 INFO:tasks.rados.rados.0.smithi084.stdout:77: finishing copy_from to smithi08477203-22 2024-09-20T22:43:04.235 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 22 v 12 (ObjNum 48 snap 0 seq_num 48) dirty exists 2024-09-20T22:43:04.235 INFO:tasks.rados.rados.0.smithi084.stdout:79: finishing write tid 2 to smithi08477203-30 2024-09-20T22:43:04.235 INFO:tasks.rados.rados.0.smithi084.stdout:79: finishing write tid 3 to smithi08477203-30 2024-09-20T22:43:04.235 INFO:tasks.rados.rados.0.smithi084.stdout:77: done (5 left) 2024-09-20T22:43:04.235 INFO:tasks.rados.rados.0.smithi084.stdout:78: done (4 left) 2024-09-20T22:43:04.235 INFO:tasks.rados.rados.0.smithi084.stdout:83: rollback oid 4 current snap is 4 2024-09-20T22:43:04.235 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 4 to 3 2024-09-20T22:43:04.235 INFO:tasks.rados.rados.0.smithi084.stdout:79: finishing write tid 4 to smithi08477203-30 2024-09-20T22:43:04.235 INFO:tasks.rados.rados.0.smithi084.stdout:79: oid 30 updating version 0 to 24 2024-09-20T22:43:04.236 INFO:tasks.rados.rados.0.smithi084.stdout:79: oid 30 version 24 is already newer than 23 2024-09-20T22:43:04.236 INFO:tasks.rados.rados.0.smithi084.stdout:79: oid 30 updating version 24 to 25 2024-09-20T22:43:04.236 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 30 v 25 (ObjNum 58 snap 4 seq_num 58) dirty exists 2024-09-20T22:43:04.236 INFO:tasks.rados.rados.0.smithi084.stdout:79: left oid 30 (ObjNum 58 snap 4 seq_num 58) 2024-09-20T22:43:04.236 INFO:tasks.rados.rados.0.smithi084.stdout:80: finishing write tid 1 to smithi08477203-29 2024-09-20T22:43:04.236 INFO:tasks.rados.rados.0.smithi084.stdout:80: finishing write tid 2 to smithi08477203-29 2024-09-20T22:43:04.236 INFO:tasks.rados.rados.0.smithi084.stdout:80: finishing write tid 3 to smithi08477203-29 2024-09-20T22:43:04.236 INFO:tasks.rados.rados.0.smithi084.stdout:81: expect (ObjNum 35 snap 0 seq_num 35) 2024-09-20T22:43:04.289 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:43:04.311 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:04.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:04.312 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:43:04.322 INFO:tasks.rados.rados.0.smithi084.stdout:80: finishing write tid 4 to smithi08477203-29 2024-09-20T22:43:04.322 INFO:tasks.rados.rados.0.smithi084.stdout:80: finishing write tid 5 to smithi08477203-29 2024-09-20T22:43:04.322 INFO:tasks.rados.rados.0.smithi084.stdout:80: oid 29 updating version 0 to 27 2024-09-20T22:43:04.322 INFO:tasks.rados.rados.0.smithi084.stdout:80: oid 29 updating version 27 to 29 2024-09-20T22:43:04.322 INFO:tasks.rados.rados.0.smithi084.stdout:80: oid 29 updating version 29 to 30 2024-09-20T22:43:04.322 INFO:tasks.rados.rados.0.smithi084.stdout:80: oid 29 version 30 is already newer than 28 2024-09-20T22:43:04.323 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 29 v 30 (ObjNum 59 snap 4 seq_num 59) dirty exists 2024-09-20T22:43:04.323 INFO:tasks.rados.rados.0.smithi084.stdout:80: left oid 29 (ObjNum 59 snap 4 seq_num 59) 2024-09-20T22:43:04.323 INFO:tasks.rados.rados.0.smithi084.stdout:82: finishing copy_from racing read to smithi08477203-42 2024-09-20T22:43:04.323 INFO:tasks.rados.rados.0.smithi084.stdout:82: finishing copy_from to smithi08477203-42 2024-09-20T22:43:04.323 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 42 v 15 (ObjNum 27 snap 0 seq_num 27) dirty exists 2024-09-20T22:43:04.323 INFO:tasks.rados.rados.0.smithi084.stdout:79: done (4 left) 2024-09-20T22:43:04.323 INFO:tasks.rados.rados.0.smithi084.stdout:80: done (3 left) 2024-09-20T22:43:04.323 INFO:tasks.rados.rados.0.smithi084.stdout:81: done (2 left) 2024-09-20T22:43:04.323 INFO:tasks.rados.rados.0.smithi084.stdout:82: done (1 left) 2024-09-20T22:43:04.323 INFO:tasks.rados.rados.0.smithi084.stdout:84: read oid 14 snap -1 2024-09-20T22:43:04.323 INFO:tasks.rados.rados.0.smithi084.stdout:84: expect (ObjNum 13 snap 0 seq_num 13) 2024-09-20T22:43:04.323 INFO:tasks.rados.rados.0.smithi084.stdout:85: read oid 29 snap -1 2024-09-20T22:43:04.323 INFO:tasks.rados.rados.0.smithi084.stdout:85: expect (ObjNum 59 snap 4 seq_num 59) 2024-09-20T22:43:04.323 INFO:tasks.rados.rados.0.smithi084.stdout:86: read oid 6 snap 3 2024-09-20T22:43:04.323 INFO:tasks.rados.rados.0.smithi084.stdout:86: expect (ObjNum 5 snap 0 seq_num 5) 2024-09-20T22:43:04.324 INFO:tasks.rados.rados.0.smithi084.stdout:87: read oid 24 snap -1 2024-09-20T22:43:04.324 INFO:tasks.rados.rados.0.smithi084.stdout:87: expect (ObjNum 23 snap 0 seq_num 23) 2024-09-20T22:43:04.324 INFO:tasks.rados.rados.0.smithi084.stdout:88: copy_from oid 21 from oid 26 current snap is 4 2024-09-20T22:43:04.324 INFO:tasks.rados.rados.0.smithi084.stdout:83: finishing rollback tid 0 to smithi08477203-4 2024-09-20T22:43:04.324 INFO:tasks.rados.rados.0.smithi084.stdout:83: finishing rollback tid 1 to smithi08477203-4 2024-09-20T22:43:04.324 INFO:tasks.rados.rados.0.smithi084.stdout:83: finishing rollback tid 2 to smithi08477203-4 2024-09-20T22:43:04.324 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 4 v 34 (ObjNum 3 snap 0 seq_num 3) dirty exists 2024-09-20T22:43:04.324 INFO:tasks.rados.rados.0.smithi084.stdout:83: done (5 left) 2024-09-20T22:43:04.324 INFO:tasks.rados.rados.0.smithi084.stdout:89: read oid 2 snap -1 2024-09-20T22:43:04.324 INFO:tasks.rados.rados.0.smithi084.stdout:89: expect deleted 2024-09-20T22:43:04.324 INFO:tasks.rados.rados.0.smithi084.stdout:90: read oid 37 snap -1 2024-09-20T22:43:04.324 INFO:tasks.rados.rados.0.smithi084.stdout:90: expect (ObjNum 36 snap 0 seq_num 36) 2024-09-20T22:43:04.324 INFO:tasks.rados.rados.0.smithi084.stdout:91: write oid 38 current snap is 4 2024-09-20T22:43:04.324 INFO:tasks.rados.rados.0.smithi084.stdout:91: seq_num 60 ranges {608420=529411,1778096=561011,2881850=126570} 2024-09-20T22:43:04.329 INFO:tasks.rados.rados.0.smithi084.stdout:91: writing smithi08477203-38 from 608420 to 1137831 tid 1 2024-09-20T22:43:04.337 INFO:tasks.rados.rados.0.smithi084.stdout:91: writing smithi08477203-38 from 1778096 to 2339107 tid 2 2024-09-20T22:43:04.341 INFO:tasks.rados.rados.0.smithi084.stdout:91: writing smithi08477203-38 from 2881850 to 3008420 tid 3 2024-09-20T22:43:04.341 INFO:tasks.rados.rados.0.smithi084.stdout:84: expect (ObjNum 13 snap 0 seq_num 13) 2024-09-20T22:43:04.413 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:43:04.416 INFO:tasks.rados.rados.0.smithi084.stdout:90: expect (ObjNum 36 snap 0 seq_num 36) 2024-09-20T22:43:04.436 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:04.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:04.437 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:43:04.466 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:43:04.466 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:43:04.496 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:43:04.496 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:43:04.523 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:43:04.523 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:43:04.532 INFO:tasks.rados.rados.0.smithi084.stdout:86: expect (ObjNum 5 snap 0 seq_num 5) 2024-09-20T22:43:04.538 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:43:04.543 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-20T22:43:04.543 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T22:43:04.559 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:04.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:04.561 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:43:04.561 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:43:04.561 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:43:04.577 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:43:04.577 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:43:04.596 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:43:04.596 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:43:04.605 INFO:tasks.rados.rados.0.smithi084.stdout:85: expect (ObjNum 59 snap 4 seq_num 59) 2024-09-20T22:43:04.613 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:43:04.613 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:43:04.661 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:43:04.684 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:04.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:04.685 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:43:04.745 INFO:tasks.rados.rados.0.smithi084.stdout:91: finishing write tid 1 to smithi08477203-38 2024-09-20T22:43:04.745 INFO:tasks.rados.rados.0.smithi084.stdout:92: read oid 17 snap -1 2024-09-20T22:43:04.745 INFO:tasks.rados.rados.0.smithi084.stdout:92: expect (ObjNum 16 snap 0 seq_num 16) 2024-09-20T22:43:04.745 INFO:tasks.rados.rados.0.smithi084.stdout:84: done (8 left) 2024-09-20T22:43:04.745 INFO:tasks.rados.rados.0.smithi084.stdout:85: done (7 left) 2024-09-20T22:43:04.745 INFO:tasks.rados.rados.0.smithi084.stdout:86: done (6 left) 2024-09-20T22:43:04.745 INFO:tasks.rados.rados.0.smithi084.stdout:89: done (5 left) 2024-09-20T22:43:04.745 INFO:tasks.rados.rados.0.smithi084.stdout:90: done (4 left) 2024-09-20T22:43:04.746 INFO:tasks.rados.rados.0.smithi084.stdout:93: write (excl) oid 34 current snap is 4 2024-09-20T22:43:04.746 INFO:tasks.rados.rados.0.smithi084.stdout:87: expect (ObjNum 23 snap 0 seq_num 23) 2024-09-20T22:43:04.786 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:43:04.807 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:04.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:04.808 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:43:04.827 INFO:tasks.rados.rados.0.smithi084.stdout:91: finishing write tid 2 to smithi08477203-38 2024-09-20T22:43:04.827 INFO:tasks.rados.rados.0.smithi084.stdout:91: finishing write tid 3 to smithi08477203-38 2024-09-20T22:43:04.827 INFO:tasks.rados.rados.0.smithi084.stdout:93: seq_num 61 ranges {625227=721013,1952931=704669} 2024-09-20T22:43:04.834 INFO:tasks.rados.rados.0.smithi084.stdout:93: writing smithi08477203-34 from 625227 to 1346240 tid 1 2024-09-20T22:43:04.842 INFO:tasks.rados.rados.0.smithi084.stdout:93: writing smithi08477203-34 from 1952931 to 2657600 tid 2 2024-09-20T22:43:04.842 INFO:tasks.rados.rados.0.smithi084.stdout:87: done (4 left) 2024-09-20T22:43:04.842 INFO:tasks.rados.rados.0.smithi084.stdout:94: write oid 20 current snap is 4 2024-09-20T22:43:04.842 INFO:tasks.rados.rados.0.smithi084.stdout:91: finishing write tid 4 to smithi08477203-38 2024-09-20T22:43:04.842 INFO:tasks.rados.rados.0.smithi084.stdout:91: finishing write tid 5 to smithi08477203-38 2024-09-20T22:43:04.842 INFO:tasks.rados.rados.0.smithi084.stdout:91: oid 38 updating version 0 to 39 2024-09-20T22:43:04.843 INFO:tasks.rados.rados.0.smithi084.stdout:91: oid 38 version 39 is already newer than 38 2024-09-20T22:43:04.843 INFO:tasks.rados.rados.0.smithi084.stdout:91: oid 38 updating version 39 to 41 2024-09-20T22:43:04.843 INFO:tasks.rados.rados.0.smithi084.stdout:91: oid 38 version 41 is already newer than 40 2024-09-20T22:43:04.843 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 38 v 41 (ObjNum 60 snap 4 seq_num 60) dirty exists 2024-09-20T22:43:04.843 INFO:tasks.rados.rados.0.smithi084.stdout:91: left oid 38 (ObjNum 60 snap 4 seq_num 60) 2024-09-20T22:43:04.843 INFO:tasks.rados.rados.0.smithi084.stdout:88: finishing copy_from racing read to smithi08477203-21 2024-09-20T22:43:04.843 INFO:tasks.rados.rados.0.smithi084.stdout:88: finishing copy_from to smithi08477203-21 2024-09-20T22:43:04.843 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 21 v 6 (ObjNum 25 snap 0 seq_num 25) dirty exists 2024-09-20T22:43:04.843 INFO:tasks.rados.rados.0.smithi084.stdout:94: seq_num 62 ranges {642034=796262,1895060=564680} 2024-09-20T22:43:04.849 INFO:tasks.rados.rados.0.smithi084.stdout:94: writing smithi08477203-20 from 642034 to 1438296 tid 1 2024-09-20T22:43:04.855 INFO:tasks.rados.rados.0.smithi084.stdout:94: writing smithi08477203-20 from 1895060 to 2459740 tid 2 2024-09-20T22:43:04.856 INFO:tasks.rados.rados.0.smithi084.stdout:88: done (4 left) 2024-09-20T22:43:04.856 INFO:tasks.rados.rados.0.smithi084.stdout:91: done (3 left) 2024-09-20T22:43:04.856 INFO:tasks.rados.rados.0.smithi084.stdout:95: read oid 14 snap -1 2024-09-20T22:43:04.856 INFO:tasks.rados.rados.0.smithi084.stdout:95: expect (ObjNum 13 snap 0 seq_num 13) 2024-09-20T22:43:04.856 INFO:tasks.rados.rados.0.smithi084.stdout:96: copy_from oid 9 from oid 16 current snap is 4 2024-09-20T22:43:04.856 INFO:tasks.rados.rados.0.smithi084.stdout:92: expect (ObjNum 16 snap 0 seq_num 16) 2024-09-20T22:43:04.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:04 smithi084 ceph-mon[40024]: 6.7 scrub starts 2024-09-20T22:43:04.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:04 smithi084 ceph-mon[40024]: 6.7 scrub ok 2024-09-20T22:43:04.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:04 smithi084 ceph-mon[40024]: 3.17 scrub starts 2024-09-20T22:43:04.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:04 smithi084 ceph-mon[40024]: 3.17 scrub ok 2024-09-20T22:43:04.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:04 smithi084 ceph-mon[40024]: 1.3 scrub starts 2024-09-20T22:43:04.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:04 smithi084 ceph-mon[40024]: 1.3 scrub ok 2024-09-20T22:43:04.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:04 smithi084 ceph-mon[40024]: osdmap e105: 8 total, 8 up, 8 in 2024-09-20T22:43:04.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:04 smithi084 ceph-mon[40024]: 1.2 scrub starts 2024-09-20T22:43:04.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:04 smithi084 ceph-mon[40024]: 1.2 scrub ok 2024-09-20T22:43:04.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:04 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/498047395' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:43:04.866 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:04 smithi084 ceph-mon[29774]: 6.7 scrub starts 2024-09-20T22:43:04.866 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:04 smithi084 ceph-mon[29774]: 6.7 scrub ok 2024-09-20T22:43:04.866 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:04 smithi084 ceph-mon[29774]: 3.17 scrub starts 2024-09-20T22:43:04.866 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:04 smithi084 ceph-mon[29774]: 3.17 scrub ok 2024-09-20T22:43:04.866 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:04 smithi084 ceph-mon[29774]: 1.3 scrub starts 2024-09-20T22:43:04.866 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:04 smithi084 ceph-mon[29774]: 1.3 scrub ok 2024-09-20T22:43:04.866 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:04 smithi084 ceph-mon[29774]: osdmap e105: 8 total, 8 up, 8 in 2024-09-20T22:43:04.866 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:04 smithi084 ceph-mon[29774]: 1.2 scrub starts 2024-09-20T22:43:04.866 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:04 smithi084 ceph-mon[29774]: 1.2 scrub ok 2024-09-20T22:43:04.866 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:04 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/498047395' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:43:04.909 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:43:04.930 INFO:tasks.rados.rados.0.smithi084.stdout:93: finishing write tid 1 to smithi08477203-34 2024-09-20T22:43:04.931 INFO:tasks.rados.rados.0.smithi084.stdout:93: finishing write tid 2 to smithi08477203-34 2024-09-20T22:43:04.931 INFO:tasks.rados.rados.0.smithi084.stdout:93: finishing write tid 3 to smithi08477203-34 2024-09-20T22:43:04.931 INFO:tasks.rados.rados.0.smithi084.stdout:93: finishing write tid 4 to smithi08477203-34 2024-09-20T22:43:04.931 INFO:tasks.rados.rados.0.smithi084.stdout:93: oid 34 updating version 0 to 27 2024-09-20T22:43:04.931 INFO:tasks.rados.rados.0.smithi084.stdout:93: oid 34 updating version 27 to 28 2024-09-20T22:43:04.931 INFO:tasks.rados.rados.0.smithi084.stdout:93: oid 34 updating version 28 to 29 2024-09-20T22:43:04.931 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 34 v 29 (ObjNum 61 snap 4 seq_num 61) dirty exists 2024-09-20T22:43:04.931 INFO:tasks.rados.rados.0.smithi084.stdout:93: left oid 34 (ObjNum 61 snap 4 seq_num 61) 2024-09-20T22:43:04.931 INFO:tasks.rados.rados.0.smithi084.stdout:92: done (4 left) 2024-09-20T22:43:04.931 INFO:tasks.rados.rados.0.smithi084.stdout:93: done (3 left) 2024-09-20T22:43:04.931 INFO:tasks.rados.rados.0.smithi084.stdout:97: snap_remove snap 3 2024-09-20T22:43:04.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:04.932 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:04.932 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:43:05.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:04 smithi154 ceph-mon[33264]: 6.7 scrub starts 2024-09-20T22:43:05.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:04 smithi154 ceph-mon[33264]: 6.7 scrub ok 2024-09-20T22:43:05.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:04 smithi154 ceph-mon[33264]: 3.17 scrub starts 2024-09-20T22:43:05.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:04 smithi154 ceph-mon[33264]: 3.17 scrub ok 2024-09-20T22:43:05.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:04 smithi154 ceph-mon[33264]: 1.3 scrub starts 2024-09-20T22:43:05.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:04 smithi154 ceph-mon[33264]: 1.3 scrub ok 2024-09-20T22:43:05.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:04 smithi154 ceph-mon[33264]: osdmap e105: 8 total, 8 up, 8 in 2024-09-20T22:43:05.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:04 smithi154 ceph-mon[33264]: 1.2 scrub starts 2024-09-20T22:43:05.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:04 smithi154 ceph-mon[33264]: 1.2 scrub ok 2024-09-20T22:43:05.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:04 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/498047395' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:43:05.033 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:43:05.055 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:05.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:05.056 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:43:05.157 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:43:05.178 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:05.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:05.179 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:43:05.280 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:43:05.302 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:05.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:05.303 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:43:05.404 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:43:05.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:05.426 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:05.426 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:43:05.527 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:43:05.528 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is unset 2024-09-20T22:43:05.542 INFO:tasks.rados.rados.0.smithi084.stdout:97: done (3 left) 2024-09-20T22:43:05.542 INFO:tasks.rados.rados.0.smithi084.stdout:98: write oid 11 current snap is 4 2024-09-20T22:43:05.542 INFO:tasks.rados.rados.0.smithi084.stdout:94: finishing write tid 1 to smithi08477203-20 2024-09-20T22:43:05.542 INFO:tasks.rados.rados.0.smithi084.stdout:98: seq_num 63 ranges {658841=471511,1553542=708338,2964119=94722} 2024-09-20T22:43:05.548 INFO:tasks.rados.rados.0.smithi084.stdout:98: writing smithi08477203-11 from 658841 to 1130352 tid 1 2024-09-20T22:43:05.554 INFO:tasks.rados.rados.0.smithi084.stdout:98: writing smithi08477203-11 from 1553542 to 2261880 tid 2 2024-09-20T22:43:05.557 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:05.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:05.558 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:43:05.559 INFO:tasks.rados.rados.0.smithi084.stdout:98: writing smithi08477203-11 from 2964119 to 3058841 tid 3 2024-09-20T22:43:05.559 INFO:tasks.rados.rados.0.smithi084.stdout:99: read oid 44 snap -1 2024-09-20T22:43:05.559 INFO:tasks.rados.rados.0.smithi084.stdout:99: expect (ObjNum 43 snap 0 seq_num 43) 2024-09-20T22:43:05.559 INFO:tasks.rados.rados.0.smithi084.stdout:100: rollback oid 25 current snap is 4 2024-09-20T22:43:05.559 INFO:tasks.rados.rados.0.smithi084.stdout:100: done (5 left) 2024-09-20T22:43:05.559 INFO:tasks.rados.rados.0.smithi084.stdout:101: read oid 39 snap -1 2024-09-20T22:43:05.559 INFO:tasks.rados.rados.0.smithi084.stdout:101: expect (ObjNum 38 snap 0 seq_num 38) 2024-09-20T22:43:05.559 INFO:tasks.rados.rados.0.smithi084.stdout:102: snap_create 2024-09-20T22:43:05.559 INFO:tasks.rados.rados.0.smithi084.stdout:94: finishing write tid 2 to smithi08477203-20 2024-09-20T22:43:05.560 INFO:tasks.rados.rados.0.smithi084.stdout:94: finishing write tid 3 to smithi08477203-20 2024-09-20T22:43:05.560 INFO:tasks.rados.rados.0.smithi084.stdout:94: finishing write tid 4 to smithi08477203-20 2024-09-20T22:43:05.560 INFO:tasks.rados.rados.0.smithi084.stdout:94: oid 20 updating version 0 to 9 2024-09-20T22:43:05.560 INFO:tasks.rados.rados.0.smithi084.stdout:94: oid 20 updating version 9 to 10 2024-09-20T22:43:05.560 INFO:tasks.rados.rados.0.smithi084.stdout:94: oid 20 version 10 is already newer than 8 2024-09-20T22:43:05.560 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 20 v 10 (ObjNum 62 snap 4 seq_num 62) dirty exists 2024-09-20T22:43:05.560 INFO:tasks.rados.rados.0.smithi084.stdout:94: left oid 20 (ObjNum 62 snap 4 seq_num 62) 2024-09-20T22:43:05.560 INFO:tasks.rados.rados.0.smithi084.stdout:95: expect (ObjNum 13 snap 0 seq_num 13) 2024-09-20T22:43:05.635 INFO:tasks.rados.rados.0.smithi084.stdout:96: finishing copy_from racing read to smithi08477203-9 2024-09-20T22:43:05.635 INFO:tasks.rados.rados.0.smithi084.stdout:96: finishing copy_from to smithi08477203-9 2024-09-20T22:43:05.635 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 9 v 3 (ObjNum 15 snap 0 seq_num 15) dirty exists 2024-09-20T22:43:05.635 INFO:tasks.rados.rados.0.smithi084.stdout:98: finishing write tid 1 to smithi08477203-11 2024-09-20T22:43:05.635 INFO:tasks.rados.rados.0.smithi084.stdout:98: finishing write tid 2 to smithi08477203-11 2024-09-20T22:43:05.635 INFO:tasks.rados.rados.0.smithi084.stdout:98: finishing write tid 3 to smithi08477203-11 2024-09-20T22:43:05.635 INFO:tasks.rados.rados.0.smithi084.stdout:98: finishing write tid 4 to smithi08477203-11 2024-09-20T22:43:05.635 INFO:tasks.rados.rados.0.smithi084.stdout:98: finishing write tid 5 to smithi08477203-11 2024-09-20T22:43:05.635 INFO:tasks.rados.rados.0.smithi084.stdout:98: oid 11 updating version 0 to 12 2024-09-20T22:43:05.635 INFO:tasks.rados.rados.0.smithi084.stdout:98: oid 11 version 12 is already newer than 11 2024-09-20T22:43:05.635 INFO:tasks.rados.rados.0.smithi084.stdout:98: oid 11 version 12 is already newer than 10 2024-09-20T22:43:05.635 INFO:tasks.rados.rados.0.smithi084.stdout:98: oid 11 updating version 12 to 13 2024-09-20T22:43:05.636 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 11 v 13 (ObjNum 63 snap 4 seq_num 63) dirty exists 2024-09-20T22:43:05.636 INFO:tasks.rados.rados.0.smithi084.stdout:98: left oid 11 (ObjNum 63 snap 4 seq_num 63) 2024-09-20T22:43:05.636 INFO:tasks.rados.rados.0.smithi084.stdout:101: expect (ObjNum 38 snap 0 seq_num 38) 2024-09-20T22:43:05.659 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:43:05.681 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:05.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:05.682 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:43:05.719 INFO:tasks.rados.rados.0.smithi084.stdout:99: expect (ObjNum 43 snap 0 seq_num 43) 2024-09-20T22:43:05.783 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:43:05.804 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:05.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:05.805 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:43:05.864 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:05 smithi084 ceph-mon[29774]: pgmap v58: 158 pgs: 1 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 154 active+clean; 111 MiB data, 407 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:43:05.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:05 smithi084 ceph-mon[29774]: 1.1 scrub starts 2024-09-20T22:43:05.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:05 smithi084 ceph-mon[29774]: 1.1 scrub ok 2024-09-20T22:43:05.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:05 smithi084 ceph-mon[29774]: 2.1 scrub starts 2024-09-20T22:43:05.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:05 smithi084 ceph-mon[29774]: 2.1 scrub ok 2024-09-20T22:43:05.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:05 smithi084 ceph-mon[29774]: 6.10 scrub starts 2024-09-20T22:43:05.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:05 smithi084 ceph-mon[29774]: 6.10 scrub ok 2024-09-20T22:43:05.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:05 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/2364608650' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:43:05.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:05 smithi084 ceph-mon[29774]: 3.e scrub starts 2024-09-20T22:43:05.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:05 smithi084 ceph-mon[29774]: 3.e scrub ok 2024-09-20T22:43:05.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:05 smithi084 ceph-mon[40024]: pgmap v58: 158 pgs: 1 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 154 active+clean; 111 MiB data, 407 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:43:05.866 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:05 smithi084 ceph-mon[40024]: 1.1 scrub starts 2024-09-20T22:43:05.866 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:05 smithi084 ceph-mon[40024]: 1.1 scrub ok 2024-09-20T22:43:05.866 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:05 smithi084 ceph-mon[40024]: 2.1 scrub starts 2024-09-20T22:43:05.866 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:05 smithi084 ceph-mon[40024]: 2.1 scrub ok 2024-09-20T22:43:05.866 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:05 smithi084 ceph-mon[40024]: 6.10 scrub starts 2024-09-20T22:43:05.866 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:05 smithi084 ceph-mon[40024]: 6.10 scrub ok 2024-09-20T22:43:05.866 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:05 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/2364608650' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:43:05.866 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:05 smithi084 ceph-mon[40024]: 3.e scrub starts 2024-09-20T22:43:05.866 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:05 smithi084 ceph-mon[40024]: 3.e scrub ok 2024-09-20T22:43:05.906 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:43:05.927 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:05.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:05.928 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:43:06.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:05 smithi154 ceph-mon[33264]: pgmap v58: 158 pgs: 1 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 154 active+clean; 111 MiB data, 407 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:43:06.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:05 smithi154 ceph-mon[33264]: 1.1 scrub starts 2024-09-20T22:43:06.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:05 smithi154 ceph-mon[33264]: 1.1 scrub ok 2024-09-20T22:43:06.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:05 smithi154 ceph-mon[33264]: 2.1 scrub starts 2024-09-20T22:43:06.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:05 smithi154 ceph-mon[33264]: 2.1 scrub ok 2024-09-20T22:43:06.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:05 smithi154 ceph-mon[33264]: 6.10 scrub starts 2024-09-20T22:43:06.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:05 smithi154 ceph-mon[33264]: 6.10 scrub ok 2024-09-20T22:43:06.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:05 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/2364608650' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:43:06.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:05 smithi154 ceph-mon[33264]: 3.e scrub starts 2024-09-20T22:43:06.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:05 smithi154 ceph-mon[33264]: 3.e scrub ok 2024-09-20T22:43:06.029 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:43:06.050 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:06.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:06.051 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:43:06.152 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:43:06.174 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:06.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:06.175 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:43:06.276 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:43:06.297 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:06.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:06.298 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:43:06.399 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:43:06.420 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:06.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:06.421 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:43:06.521 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:43:06.543 INFO:tasks.rados.rados.0.smithi084.stdout:94: done (6 left) 2024-09-20T22:43:06.544 INFO:tasks.rados.rados.0.smithi084.stdout:95: done (5 left) 2024-09-20T22:43:06.544 INFO:tasks.rados.rados.0.smithi084.stdout:96: done (4 left) 2024-09-20T22:43:06.544 INFO:tasks.rados.rados.0.smithi084.stdout:98: done (3 left) 2024-09-20T22:43:06.544 INFO:tasks.rados.rados.0.smithi084.stdout:99: done (2 left) 2024-09-20T22:43:06.544 INFO:tasks.rados.rados.0.smithi084.stdout:101: done (1 left) 2024-09-20T22:43:06.544 INFO:tasks.rados.rados.0.smithi084.stdout:102: done (0 left) 2024-09-20T22:43:06.544 INFO:tasks.rados.rados.0.smithi084.stdout:103: copy_from oid 31 from oid 29 current snap is 5 2024-09-20T22:43:06.544 INFO:tasks.rados.rados.0.smithi084.stdout:104: read oid 50 snap -1 2024-09-20T22:43:06.544 INFO:tasks.rados.rados.0.smithi084.stdout:104: expect (ObjNum 21 snap 0 seq_num 21) 2024-09-20T22:43:06.544 INFO:tasks.rados.rados.0.smithi084.stdout:105: read oid 35 snap -1 2024-09-20T22:43:06.544 INFO:tasks.rados.rados.0.smithi084.stdout:105: expect (ObjNum 34 snap 0 seq_num 34) 2024-09-20T22:43:06.544 INFO:tasks.rados.rados.0.smithi084.stdout:106: snap_remove snap 4 2024-09-20T22:43:06.546 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:06.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:06.547 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:43:06.648 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:43:06.671 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:06.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:06.672 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:43:06.766 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:06 smithi084 ceph-mon[40024]: 1.8 scrub starts 2024-09-20T22:43:06.766 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:06 smithi084 ceph-mon[40024]: 1.8 scrub ok 2024-09-20T22:43:06.766 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:06 smithi084 ceph-mon[40024]: 3.b scrub starts 2024-09-20T22:43:06.766 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:06 smithi084 ceph-mon[40024]: 3.b scrub ok 2024-09-20T22:43:06.766 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:06 smithi084 ceph-mon[40024]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T22:43:06.766 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:06 smithi084 ceph-mon[40024]: Cluster is now healthy 2024-09-20T22:43:06.766 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:06 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/2364608650' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T22:43:06.766 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:06 smithi084 ceph-mon[40024]: osdmap e106: 8 total, 8 up, 8 in 2024-09-20T22:43:06.766 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:06 smithi084 ceph-mon[29774]: 1.8 scrub starts 2024-09-20T22:43:06.767 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:06 smithi084 ceph-mon[29774]: 1.8 scrub ok 2024-09-20T22:43:06.767 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:06 smithi084 ceph-mon[29774]: 3.b scrub starts 2024-09-20T22:43:06.767 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:06 smithi084 ceph-mon[29774]: 3.b scrub ok 2024-09-20T22:43:06.767 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:06 smithi084 ceph-mon[29774]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T22:43:06.767 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:06 smithi084 ceph-mon[29774]: Cluster is now healthy 2024-09-20T22:43:06.767 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:06 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/2364608650' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T22:43:06.767 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:06 smithi084 ceph-mon[29774]: osdmap e106: 8 total, 8 up, 8 in 2024-09-20T22:43:06.772 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:43:06.794 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:06.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:06.795 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:43:06.896 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:43:06.916 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:06.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:06.917 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:43:07.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:06 smithi154 ceph-mon[33264]: 1.8 scrub starts 2024-09-20T22:43:07.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:06 smithi154 ceph-mon[33264]: 1.8 scrub ok 2024-09-20T22:43:07.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:06 smithi154 ceph-mon[33264]: 3.b scrub starts 2024-09-20T22:43:07.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:06 smithi154 ceph-mon[33264]: 3.b scrub ok 2024-09-20T22:43:07.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:06 smithi154 ceph-mon[33264]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T22:43:07.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:06 smithi154 ceph-mon[33264]: Cluster is now healthy 2024-09-20T22:43:07.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:06 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/2364608650' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T22:43:07.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:06 smithi154 ceph-mon[33264]: osdmap e106: 8 total, 8 up, 8 in 2024-09-20T22:43:07.018 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:43:07.040 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:07.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:07.041 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:43:07.142 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:43:07.165 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:07.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:07.167 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:43:07.267 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:43:07.289 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:07.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:07.290 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:43:07.391 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:43:07.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:07.414 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:07.414 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:43:07.499 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:43:07.499 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:43:07.515 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:43:07.537 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:07.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:07.538 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:43:07.546 INFO:tasks.rados.rados.0.smithi084.stdout:106: done (3 left) 2024-09-20T22:43:07.546 INFO:tasks.rados.rados.0.smithi084.stdout:107: read oid 34 snap -1 2024-09-20T22:43:07.546 INFO:tasks.rados.rados.0.smithi084.stdout:107: expect (ObjNum 61 snap 4 seq_num 61) 2024-09-20T22:43:07.546 INFO:tasks.rados.rados.0.smithi084.stdout:108: copy_from oid 12 from oid 26 current snap is 5 2024-09-20T22:43:07.546 INFO:tasks.rados.rados.0.smithi084.stdout:105: expect (ObjNum 34 snap 0 seq_num 34) 2024-09-20T22:43:07.632 INFO:tasks.rados.rados.0.smithi084.stdout:103: finishing copy_from racing read to smithi08477203-31 2024-09-20T22:43:07.632 INFO:tasks.rados.rados.0.smithi084.stdout:104: expect (ObjNum 21 snap 0 seq_num 21) 2024-09-20T22:43:07.639 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:43:07.659 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:07.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:07.660 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:43:07.715 INFO:tasks.rados.rados.0.smithi084.stdout:103: finishing copy_from to smithi08477203-31 2024-09-20T22:43:07.715 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 31 v 30 (ObjNum 59 snap 4 seq_num 59) dirty exists 2024-09-20T22:43:07.715 INFO:tasks.rados.rados.0.smithi084.stdout:107: expect (ObjNum 61 snap 4 seq_num 61) 2024-09-20T22:43:07.761 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:43:07.782 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:07.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:07.783 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:43:07.814 INFO:tasks.rados.rados.0.smithi084.stdout:108: finishing copy_from racing read to smithi08477203-12 2024-09-20T22:43:07.814 INFO:tasks.rados.rados.0.smithi084.stdout:108: finishing copy_from to smithi08477203-12 2024-09-20T22:43:07.814 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 12 v 6 (ObjNum 25 snap 0 seq_num 25) dirty exists 2024-09-20T22:43:07.814 INFO:tasks.rados.rados.0.smithi084.stdout:103: done (4 left) 2024-09-20T22:43:07.815 INFO:tasks.rados.rados.0.smithi084.stdout:104: done (3 left) 2024-09-20T22:43:07.815 INFO:tasks.rados.rados.0.smithi084.stdout:105: done (2 left) 2024-09-20T22:43:07.815 INFO:tasks.rados.rados.0.smithi084.stdout:107: done (1 left) 2024-09-20T22:43:07.815 INFO:tasks.rados.rados.0.smithi084.stdout:108: done (0 left) 2024-09-20T22:43:07.815 INFO:tasks.rados.rados.0.smithi084.stdout:109: read oid 25 snap -1 2024-09-20T22:43:07.815 INFO:tasks.rados.rados.0.smithi084.stdout:109: expect (ObjNum 50 snap 0 seq_num 50) 2024-09-20T22:43:07.815 INFO:tasks.rados.rados.0.smithi084.stdout:110: copy_from oid 37 from oid 1 current snap is 5 2024-09-20T22:43:07.815 INFO:tasks.rados.rados.0.smithi084.stdout:111: write oid 7 current snap is 5 2024-09-20T22:43:07.815 INFO:tasks.rados.rados.0.smithi084.stdout:111: seq_num 64 ranges {675648=546760,2012024=568349} 2024-09-20T22:43:07.821 INFO:tasks.rados.rados.0.smithi084.stdout:111: writing smithi08477203-7 from 675648 to 1222408 tid 1 2024-09-20T22:43:07.829 INFO:tasks.rados.rados.0.smithi084.stdout:111: writing smithi08477203-7 from 2012024 to 2580373 tid 2 2024-09-20T22:43:07.829 INFO:tasks.rados.rados.0.smithi084.stdout:112: copy_from oid 8 from oid 43 current snap is 5 2024-09-20T22:43:07.829 INFO:tasks.rados.rados.0.smithi084.stdout:110: finishing copy_from racing read to smithi08477203-37 2024-09-20T22:43:07.830 INFO:tasks.rados.rados.0.smithi084.stdout:109: expect (ObjNum 50 snap 0 seq_num 50) 2024-09-20T22:43:07.864 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:07 smithi084 ceph-mon[29774]: pgmap v61: 158 pgs: 1 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 154 active+clean; 111 MiB data, 407 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:43:07.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:07 smithi084 ceph-mon[29774]: 1.a scrub starts 2024-09-20T22:43:07.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:07 smithi084 ceph-mon[29774]: 1.a scrub ok 2024-09-20T22:43:07.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:07 smithi084 ceph-mon[29774]: osdmap e107: 8 total, 8 up, 8 in 2024-09-20T22:43:07.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:07 smithi084 ceph-mon[29774]: 5.11 scrub starts 2024-09-20T22:43:07.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:07 smithi084 ceph-mon[29774]: 5.11 scrub ok 2024-09-20T22:43:07.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:07 smithi084 ceph-mon[40024]: pgmap v61: 158 pgs: 1 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 154 active+clean; 111 MiB data, 407 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:43:07.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:07 smithi084 ceph-mon[40024]: 1.a scrub starts 2024-09-20T22:43:07.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:07 smithi084 ceph-mon[40024]: 1.a scrub ok 2024-09-20T22:43:07.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:07 smithi084 ceph-mon[40024]: osdmap e107: 8 total, 8 up, 8 in 2024-09-20T22:43:07.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:07 smithi084 ceph-mon[40024]: 5.11 scrub starts 2024-09-20T22:43:07.866 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:07 smithi084 ceph-mon[40024]: 5.11 scrub ok 2024-09-20T22:43:07.884 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:43:07.905 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:07.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:07.906 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:43:07.950 INFO:tasks.rados.rados.0.smithi084.stdout:110: finishing copy_from to smithi08477203-37 2024-09-20T22:43:07.950 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 37 v 3 (ObjNum 0 snap 0 seq_num 0) dirty exists 2024-09-20T22:43:07.950 INFO:tasks.rados.rados.0.smithi084.stdout:109: done (3 left) 2024-09-20T22:43:07.950 INFO:tasks.rados.rados.0.smithi084.stdout:110: done (2 left) 2024-09-20T22:43:07.950 INFO:tasks.rados.rados.0.smithi084.stdout:113: write (excl) oid 47 current snap is 5 2024-09-20T22:43:07.950 INFO:tasks.rados.rados.0.smithi084.stdout:111: finishing write tid 1 to smithi08477203-7 2024-09-20T22:43:07.951 INFO:tasks.rados.rados.0.smithi084.stdout:111: finishing write tid 2 to smithi08477203-7 2024-09-20T22:43:07.951 INFO:tasks.rados.rados.0.smithi084.stdout:113: seq_num 65 ranges {692455=622009,2070506=712007} 2024-09-20T22:43:07.957 INFO:tasks.rados.rados.0.smithi084.stdout:113: writing smithi08477203-47 from 692455 to 1314464 tid 1 2024-09-20T22:43:07.965 INFO:tasks.rados.rados.0.smithi084.stdout:113: writing smithi08477203-47 from 2070506 to 2782513 tid 2 2024-09-20T22:43:07.965 INFO:tasks.rados.rados.0.smithi084.stdout:114: snap_create 2024-09-20T22:43:07.965 INFO:tasks.rados.rados.0.smithi084.stdout:111: finishing write tid 3 to smithi08477203-7 2024-09-20T22:43:07.965 INFO:tasks.rados.rados.0.smithi084.stdout:111: finishing write tid 4 to smithi08477203-7 2024-09-20T22:43:07.965 INFO:tasks.rados.rados.0.smithi084.stdout:111: oid 7 updating version 0 to 26 2024-09-20T22:43:07.965 INFO:tasks.rados.rados.0.smithi084.stdout:111: oid 7 updating version 26 to 28 2024-09-20T22:43:07.965 INFO:tasks.rados.rados.0.smithi084.stdout:111: oid 7 version 28 is already newer than 27 2024-09-20T22:43:07.965 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 7 v 28 (ObjNum 64 snap 5 seq_num 64) dirty exists 2024-09-20T22:43:07.965 INFO:tasks.rados.rados.0.smithi084.stdout:111: left oid 7 (ObjNum 64 snap 5 seq_num 64) 2024-09-20T22:43:07.966 INFO:tasks.rados.rados.0.smithi084.stdout:112: finishing copy_from racing read to smithi08477203-8 2024-09-20T22:43:07.966 INFO:tasks.rados.rados.0.smithi084.stdout:112: finishing copy_from to smithi08477203-8 2024-09-20T22:43:07.966 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 8 v 6 (ObjNum 42 snap 0 seq_num 42) dirty exists 2024-09-20T22:43:07.966 INFO:tasks.rados.rados.0.smithi084.stdout:113: finishing write tid 1 to smithi08477203-47 2024-09-20T22:43:07.969 INFO:tasks.rados.rados.0.smithi084.stdout:113: finishing write tid 2 to smithi08477203-47 2024-09-20T22:43:07.969 INFO:tasks.rados.rados.0.smithi084.stdout:113: finishing write tid 3 to smithi08477203-47 2024-09-20T22:43:07.969 INFO:tasks.rados.rados.0.smithi084.stdout:113: finishing write tid 4 to smithi08477203-47 2024-09-20T22:43:07.969 INFO:tasks.rados.rados.0.smithi084.stdout:113: oid 47 updating version 0 to 49 2024-09-20T22:43:07.969 INFO:tasks.rados.rados.0.smithi084.stdout:113: oid 47 updating version 49 to 50 2024-09-20T22:43:07.969 INFO:tasks.rados.rados.0.smithi084.stdout:113: oid 47 updating version 50 to 51 2024-09-20T22:43:07.969 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 47 v 51 (ObjNum 65 snap 5 seq_num 65) dirty exists 2024-09-20T22:43:07.969 INFO:tasks.rados.rados.0.smithi084.stdout:113: left oid 47 (ObjNum 65 snap 5 seq_num 65) 2024-09-20T22:43:08.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:07 smithi154 ceph-mon[33264]: pgmap v61: 158 pgs: 1 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 154 active+clean; 111 MiB data, 407 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:43:08.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:07 smithi154 ceph-mon[33264]: 1.a scrub starts 2024-09-20T22:43:08.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:07 smithi154 ceph-mon[33264]: 1.a scrub ok 2024-09-20T22:43:08.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:07 smithi154 ceph-mon[33264]: osdmap e107: 8 total, 8 up, 8 in 2024-09-20T22:43:08.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:07 smithi154 ceph-mon[33264]: 5.11 scrub starts 2024-09-20T22:43:08.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:07 smithi154 ceph-mon[33264]: 5.11 scrub ok 2024-09-20T22:43:08.007 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:43:08.028 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:08.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:08.029 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:43:08.117 INFO:tasks.rados.rados.0.smithi084.stdout:111: done (3 left) 2024-09-20T22:43:08.117 INFO:tasks.rados.rados.0.smithi084.stdout:112: done (2 left) 2024-09-20T22:43:08.117 INFO:tasks.rados.rados.0.smithi084.stdout:113: done (1 left) 2024-09-20T22:43:08.117 INFO:tasks.rados.rados.0.smithi084.stdout:114: done (0 left) 2024-09-20T22:43:08.117 INFO:tasks.rados.rados.0.smithi084.stdout:115: snap_create 2024-09-20T22:43:08.130 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:43:08.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:08.151 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:08.151 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:43:08.252 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:43:08.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:08.275 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:08.275 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:43:08.376 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:43:08.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:08.398 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:08.399 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:43:08.449 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd set noscrub 2024-09-20T22:43:08.499 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:43:08.522 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:08.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:08.523 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:43:08.551 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:43:08 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: ::ffff:172.21.15.154 - - [20/Sep/2024:22:43:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-20T22:43:08.624 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:43:08.647 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:08.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:08.648 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:43:08.749 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:43:08.762 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-20T22:43:08.763 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2024-09-20T22:43:08.763 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2024-09-20T22:43:08.763 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-20T22:43:08.770 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:08.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:08.771 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:43:08.865 INFO:journalctl@ceph.alertmanager.a.smithi084.stdout:Sep 20 22:43:08 smithi084 podman[79036]: 2024-09-20 22:43:08.785163157 +0000 UTC m=+0.670601206 container died be57313a721ef3d24bc3541604acb8c8478f4e40572b4628050c528e8fc5775d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=recursing_ramanujan, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-9e2ac70, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-20T22:43:08.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:08 smithi084 ceph-mon[29774]: 3.16 deep-scrub starts 2024-09-20T22:43:08.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:08 smithi084 ceph-mon[29774]: 3.16 deep-scrub ok 2024-09-20T22:43:08.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:08 smithi084 ceph-mon[29774]: 6.1b scrub starts 2024-09-20T22:43:08.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:08 smithi084 ceph-mon[29774]: 6.1b scrub ok 2024-09-20T22:43:08.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:08 smithi084 ceph-mon[29774]: osdmap e108: 8 total, 8 up, 8 in 2024-09-20T22:43:08.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:08 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]: dispatch 2024-09-20T22:43:08.866 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:08 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]}]: dispatch 2024-09-20T22:43:08.866 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:08 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]: dispatch 2024-09-20T22:43:08.866 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:08 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]: dispatch 2024-09-20T22:43:08.866 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:08 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]}]: dispatch 2024-09-20T22:43:08.866 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:08 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]: dispatch 2024-09-20T22:43:08.866 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:08 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:43:08.866 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:08 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:43:08.866 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:08 smithi084 ceph-mon[29774]: 5.1 deep-scrub starts 2024-09-20T22:43:08.866 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:08 smithi084 ceph-mon[29774]: 5.1 deep-scrub ok 2024-09-20T22:43:08.866 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:08 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]': finished 2024-09-20T22:43:08.866 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:08 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]}]': finished 2024-09-20T22:43:08.866 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:08 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]': finished 2024-09-20T22:43:08.867 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:08 smithi084 ceph-mon[29774]: osdmap e109: 8 total, 8 up, 8 in 2024-09-20T22:43:08.867 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:08 smithi084 ceph-mon[40024]: 3.16 deep-scrub starts 2024-09-20T22:43:08.867 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:08 smithi084 ceph-mon[40024]: 3.16 deep-scrub ok 2024-09-20T22:43:08.867 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:08 smithi084 ceph-mon[40024]: 6.1b scrub starts 2024-09-20T22:43:08.867 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:08 smithi084 ceph-mon[40024]: 6.1b scrub ok 2024-09-20T22:43:08.867 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:08 smithi084 ceph-mon[40024]: osdmap e108: 8 total, 8 up, 8 in 2024-09-20T22:43:08.867 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:08 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]: dispatch 2024-09-20T22:43:08.867 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:08 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]}]: dispatch 2024-09-20T22:43:08.867 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:08 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]: dispatch 2024-09-20T22:43:08.867 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:08 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]: dispatch 2024-09-20T22:43:08.867 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:08 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]}]: dispatch 2024-09-20T22:43:08.867 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:08 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]: dispatch 2024-09-20T22:43:08.868 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:08 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:43:08.868 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:08 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:43:08.868 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:08 smithi084 ceph-mon[40024]: 5.1 deep-scrub starts 2024-09-20T22:43:08.868 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:08 smithi084 ceph-mon[40024]: 5.1 deep-scrub ok 2024-09-20T22:43:08.868 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:08 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]': finished 2024-09-20T22:43:08.868 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:08 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]}]': finished 2024-09-20T22:43:08.868 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:08 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]': finished 2024-09-20T22:43:08.868 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:08 smithi084 ceph-mon[40024]: osdmap e109: 8 total, 8 up, 8 in 2024-09-20T22:43:08.872 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:43:08.893 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:08.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:08.894 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:43:08.953 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:08 smithi154 ceph-mon[33264]: 3.16 deep-scrub starts 2024-09-20T22:43:08.953 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:08 smithi154 ceph-mon[33264]: 3.16 deep-scrub ok 2024-09-20T22:43:08.953 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:08 smithi154 ceph-mon[33264]: 6.1b scrub starts 2024-09-20T22:43:08.953 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:08 smithi154 ceph-mon[33264]: 6.1b scrub ok 2024-09-20T22:43:08.953 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:08 smithi154 ceph-mon[33264]: osdmap e108: 8 total, 8 up, 8 in 2024-09-20T22:43:08.954 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:08 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]: dispatch 2024-09-20T22:43:08.954 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:08 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]}]: dispatch 2024-09-20T22:43:08.954 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:08 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]: dispatch 2024-09-20T22:43:08.954 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:08 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]: dispatch 2024-09-20T22:43:08.954 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:08 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]}]: dispatch 2024-09-20T22:43:08.954 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:08 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]: dispatch 2024-09-20T22:43:08.954 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:08 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:43:08.954 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:08 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:43:08.954 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:08 smithi154 ceph-mon[33264]: 5.1 deep-scrub starts 2024-09-20T22:43:08.954 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:08 smithi154 ceph-mon[33264]: 5.1 deep-scrub ok 2024-09-20T22:43:08.954 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:08 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]': finished 2024-09-20T22:43:08.954 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:08 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]}]': finished 2024-09-20T22:43:08.954 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:08 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]': finished 2024-09-20T22:43:08.955 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:08 smithi154 ceph-mon[33264]: osdmap e109: 8 total, 8 up, 8 in 2024-09-20T22:43:08.995 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:43:09.016 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:09.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:09.017 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:43:09.078 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-20T22:43:09.113 INFO:tasks.rados.rados.0.smithi084.stdout:115: done (0 left) 2024-09-20T22:43:09.113 INFO:tasks.rados.rados.0.smithi084.stdout:116: delete oid 39 current snap is 7 2024-09-20T22:43:09.118 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:43:09.119 INFO:tasks.rados.rados.0.smithi084.stdout:116: done (0 left) 2024-09-20T22:43:09.119 INFO:tasks.rados.rados.0.smithi084.stdout:117: snap_remove snap 6 2024-09-20T22:43:09.140 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:09.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:09.141 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:43:09.242 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:43:09.254 INFO:journalctl@ceph.iscsi.iscsi.a.smithi154.stdout:Sep 20 22:43:08 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-iscsi-iscsi-a[59730]: debug there is no tcmu-runner data available 2024-09-20T22:43:09.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:09.264 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:09.265 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:43:09.365 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:43:09.387 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:09.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:09.388 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:43:09.476 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.2 dump_blocked_ops 2024-09-20T22:43:09.488 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:43:09.539 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:09.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:09.540 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:43:09.641 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:43:09.663 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:09.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:09.664 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:43:09.765 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:43:09.787 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:09.788 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:09 smithi084 ceph-mon[29774]: pgmap v64: 158 pgs: 1 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 154 active+clean; 111 MiB data, 407 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:43:09.788 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:09 smithi084 ceph-mon[29774]: 6.14 scrub starts 2024-09-20T22:43:09.788 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:09 smithi084 ceph-mon[29774]: 6.14 scrub ok 2024-09-20T22:43:09.788 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:09 smithi084 ceph-mon[29774]: 5.3 scrub starts 2024-09-20T22:43:09.788 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:09 smithi084 ceph-mon[29774]: 5.3 scrub ok 2024-09-20T22:43:09.788 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:09 smithi084 ceph-mon[29774]: osdmap e110: 8 total, 8 up, 8 in 2024-09-20T22:43:09.789 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:09 smithi084 ceph-mon[40024]: pgmap v64: 158 pgs: 1 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 154 active+clean; 111 MiB data, 407 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:43:09.789 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:09 smithi084 ceph-mon[40024]: 6.14 scrub starts 2024-09-20T22:43:09.789 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:09 smithi084 ceph-mon[40024]: 6.14 scrub ok 2024-09-20T22:43:09.789 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:09 smithi084 ceph-mon[40024]: 5.3 scrub starts 2024-09-20T22:43:09.789 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:09 smithi084 ceph-mon[40024]: 5.3 scrub ok 2024-09-20T22:43:09.789 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:09 smithi084 ceph-mon[40024]: osdmap e110: 8 total, 8 up, 8 in 2024-09-20T22:43:09.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:09.789 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:43:09.890 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:43:09.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:09.914 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:09.914 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:43:10.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:09 smithi154 ceph-mon[33264]: pgmap v64: 158 pgs: 1 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 154 active+clean; 111 MiB data, 407 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:43:10.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:09 smithi154 ceph-mon[33264]: 6.14 scrub starts 2024-09-20T22:43:10.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:09 smithi154 ceph-mon[33264]: 6.14 scrub ok 2024-09-20T22:43:10.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:09 smithi154 ceph-mon[33264]: 5.3 scrub starts 2024-09-20T22:43:10.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:09 smithi154 ceph-mon[33264]: 5.3 scrub ok 2024-09-20T22:43:10.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:09 smithi154 ceph-mon[33264]: osdmap e110: 8 total, 8 up, 8 in 2024-09-20T22:43:10.015 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:43:10.038 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:10.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:10.039 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:43:10.045 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:43:10.049 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:43:10.053 INFO:journalctl@ceph.alertmanager.a.smithi084.stdout:Sep 20 22:43:10 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-alertmanager-a[73531]: ts=2024-09-20T22:43:10.032Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-09-20T22:43:10.083 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-20T22:43:10.085 INFO:teuthology.orchestra.run.smithi084.stderr:dumped all 2024-09-20T22:43:10.113 INFO:tasks.rados.rados.0.smithi084.stdout:117: done (0 left) 2024-09-20T22:43:10.114 INFO:tasks.rados.rados.0.smithi084.stdout:118: read oid 14 snap 5 2024-09-20T22:43:10.114 INFO:tasks.rados.rados.0.smithi084.stdout:118: expect (ObjNum 13 snap 0 seq_num 13) 2024-09-20T22:43:10.114 INFO:tasks.rados.rados.0.smithi084.stdout:119: write oid 47 current snap is 7 2024-09-20T22:43:10.114 INFO:tasks.rados.rados.0.smithi084.stdout:119: seq_num 67 ranges {726069=772507,2071117=715676} 2024-09-20T22:43:10.122 INFO:tasks.rados.rados.0.smithi084.stdout:119: writing smithi08477203-47 from 726069 to 1498576 tid 1 2024-09-20T22:43:10.129 INFO:tasks.rados.rados.0.smithi084.stdout:119: writing smithi08477203-47 from 2071117 to 2786793 tid 2 2024-09-20T22:43:10.130 INFO:tasks.rados.rados.0.smithi084.stdout:120: read oid 13 snap -1 2024-09-20T22:43:10.130 INFO:tasks.rados.rados.0.smithi084.stdout:120: expect (ObjNum 12 snap 0 seq_num 12) 2024-09-20T22:43:10.130 INFO:tasks.rados.rados.0.smithi084.stdout:118: expect (ObjNum 13 snap 0 seq_num 13) 2024-09-20T22:43:10.139 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:43:10.161 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:10.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:10.162 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:43:10.205 INFO:tasks.rados.rados.0.smithi084.stdout:119: finishing write tid 1 to smithi08477203-47 2024-09-20T22:43:10.205 INFO:tasks.rados.rados.0.smithi084.stdout:118: done (2 left) 2024-09-20T22:43:10.205 INFO:tasks.rados.rados.0.smithi084.stdout:121: write (excl) oid 22 current snap is 7 2024-09-20T22:43:10.205 INFO:tasks.rados.rados.0.smithi084.stdout:119: finishing write tid 2 to smithi08477203-47 2024-09-20T22:43:10.205 INFO:tasks.rados.rados.0.smithi084.stdout:119: finishing write tid 3 to smithi08477203-47 2024-09-20T22:43:10.205 INFO:tasks.rados.rados.0.smithi084.stdout:119: finishing write tid 4 to smithi08477203-47 2024-09-20T22:43:10.205 INFO:tasks.rados.rados.0.smithi084.stdout:119: oid 47 updating version 0 to 55 2024-09-20T22:43:10.205 INFO:tasks.rados.rados.0.smithi084.stdout:119: oid 47 version 55 is already newer than 53 2024-09-20T22:43:10.205 INFO:tasks.rados.rados.0.smithi084.stdout:119: oid 47 version 55 is already newer than 54 2024-09-20T22:43:10.206 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 47 v 55 (ObjNum 67 snap 7 seq_num 67) dirty exists 2024-09-20T22:43:10.206 INFO:tasks.rados.rados.0.smithi084.stdout:119: left oid 47 (ObjNum 67 snap 7 seq_num 67) 2024-09-20T22:43:10.206 INFO:tasks.rados.rados.0.smithi084.stdout:121: seq_num 68 ranges {742876=447756,1729599=575687,3101234=41642} 2024-09-20T22:43:10.211 INFO:tasks.rados.rados.0.smithi084.stdout:121: writing smithi08477203-22 from 742876 to 1190632 tid 1 2024-09-20T22:43:10.217 INFO:tasks.rados.rados.0.smithi084.stdout:121: writing smithi08477203-22 from 1729599 to 2305286 tid 2 2024-09-20T22:43:10.222 INFO:tasks.rados.rados.0.smithi084.stdout:121: writing smithi08477203-22 from 3101234 to 3142876 tid 3 2024-09-20T22:43:10.222 INFO:tasks.rados.rados.0.smithi084.stdout:119: done (2 left) 2024-09-20T22:43:10.222 INFO:tasks.rados.rados.0.smithi084.stdout:122: snap_remove snap 5 2024-09-20T22:43:10.263 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:43:10.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:10.287 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:10.288 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:43:10.364 INFO:journalctl@ceph.alertmanager.a.smithi084.stdout:Sep 20 22:43:10 smithi084 podman[79377]: 2024-09-20 22:43:10.052586841 +0000 UTC m=+0.133556904 container died f8a0254a947f2e06943c648762037a40697bea24670a496865059ffd123b36a4 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-20T22:43:10.389 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:43:10.411 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:10.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:10.412 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:43:10.513 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:43:10.534 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:10.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:10.535 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:43:10.598 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:43:10.636 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:43:10.658 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:10.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:10.659 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:43:10.698 INFO:journalctl@ceph.alertmanager.a.smithi084.stdout:Sep 20 22:43:10 smithi084 podman[79377]: 2024-09-20 22:43:10.403278721 +0000 UTC m=+0.484248777 container cleanup f8a0254a947f2e06943c648762037a40697bea24670a496865059ffd123b36a4 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-20T22:43:10.699 INFO:journalctl@ceph.alertmanager.a.smithi084.stdout:Sep 20 22:43:10 smithi084 bash[79377]: ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-alertmanager-a 2024-09-20T22:43:10.699 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:10 smithi084 ceph-mon[40024]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:43:10.699 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:10 smithi084 ceph-mon[40024]: 6.6 scrub starts 2024-09-20T22:43:10.699 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:10 smithi084 ceph-mon[40024]: 6.6 scrub ok 2024-09-20T22:43:10.699 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:10 smithi084 ceph-mon[40024]: 6.1e scrub starts 2024-09-20T22:43:10.699 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:10 smithi084 ceph-mon[40024]: 6.1e scrub ok 2024-09-20T22:43:10.699 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:10 smithi084 ceph-mon[40024]: 4.f scrub starts 2024-09-20T22:43:10.699 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:10 smithi084 ceph-mon[40024]: 4.f scrub ok 2024-09-20T22:43:10.699 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:10 smithi084 ceph-mon[40024]: osdmap e111: 8 total, 8 up, 8 in 2024-09-20T22:43:10.699 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:10 smithi084 ceph-mon[40024]: 6.12 deep-scrub starts 2024-09-20T22:43:10.699 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:10 smithi084 ceph-mon[40024]: 6.12 deep-scrub ok 2024-09-20T22:43:10.700 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:10 smithi084 ceph-mon[29774]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:43:10.700 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:10 smithi084 ceph-mon[29774]: 6.6 scrub starts 2024-09-20T22:43:10.700 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:10 smithi084 ceph-mon[29774]: 6.6 scrub ok 2024-09-20T22:43:10.700 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:10 smithi084 ceph-mon[29774]: 6.1e scrub starts 2024-09-20T22:43:10.700 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:10 smithi084 ceph-mon[29774]: 6.1e scrub ok 2024-09-20T22:43:10.700 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:10 smithi084 ceph-mon[29774]: 4.f scrub starts 2024-09-20T22:43:10.700 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:10 smithi084 ceph-mon[29774]: 4.f scrub ok 2024-09-20T22:43:10.700 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:10 smithi084 ceph-mon[29774]: osdmap e111: 8 total, 8 up, 8 in 2024-09-20T22:43:10.700 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:10 smithi084 ceph-mon[29774]: 6.12 deep-scrub starts 2024-09-20T22:43:10.700 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:10 smithi084 ceph-mon[29774]: 6.12 deep-scrub ok 2024-09-20T22:43:10.760 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:43:10.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:10.783 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:10.784 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:43:10.884 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:43:10.906 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:10.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:10.907 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:43:11.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:10 smithi154 ceph-mon[33264]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:43:11.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:10 smithi154 ceph-mon[33264]: 6.6 scrub starts 2024-09-20T22:43:11.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:10 smithi154 ceph-mon[33264]: 6.6 scrub ok 2024-09-20T22:43:11.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:10 smithi154 ceph-mon[33264]: 6.1e scrub starts 2024-09-20T22:43:11.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:10 smithi154 ceph-mon[33264]: 6.1e scrub ok 2024-09-20T22:43:11.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:10 smithi154 ceph-mon[33264]: 4.f scrub starts 2024-09-20T22:43:11.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:10 smithi154 ceph-mon[33264]: 4.f scrub ok 2024-09-20T22:43:11.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:10 smithi154 ceph-mon[33264]: osdmap e111: 8 total, 8 up, 8 in 2024-09-20T22:43:11.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:10 smithi154 ceph-mon[33264]: 6.12 deep-scrub starts 2024-09-20T22:43:11.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:10 smithi154 ceph-mon[33264]: 6.12 deep-scrub ok 2024-09-20T22:43:11.008 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:43:11.030 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:11.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:11.031 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:43:11.117 INFO:tasks.rados.rados.0.smithi084.stdout:122: done (2 left) 2024-09-20T22:43:11.117 INFO:tasks.rados.rados.0.smithi084.stdout:123: read oid 33 snap -1 2024-09-20T22:43:11.117 INFO:tasks.rados.rados.0.smithi084.stdout:123: expect (ObjNum 53 snap 2 seq_num 53) 2024-09-20T22:43:11.117 INFO:tasks.rados.rados.0.smithi084.stdout:124: delete oid 28 current snap is 7 2024-09-20T22:43:11.117 INFO:tasks.rados.rados.0.smithi084.stdout:120: expect (ObjNum 12 snap 0 seq_num 12) 2024-09-20T22:43:11.131 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:43:11.154 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:11.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:11.155 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:43:11.192 INFO:tasks.rados.rados.0.smithi084.stdout:121: finishing write tid 1 to smithi08477203-22 2024-09-20T22:43:11.192 INFO:tasks.rados.rados.0.smithi084.stdout:121: finishing write tid 2 to smithi08477203-22 2024-09-20T22:43:11.193 INFO:tasks.rados.rados.0.smithi084.stdout:121: finishing write tid 3 to smithi08477203-22 2024-09-20T22:43:11.193 INFO:tasks.rados.rados.0.smithi084.stdout:121: finishing write tid 4 to smithi08477203-22 2024-09-20T22:43:11.193 INFO:tasks.rados.rados.0.smithi084.stdout:121: finishing write tid 5 to smithi08477203-22 2024-09-20T22:43:11.193 INFO:tasks.rados.rados.0.smithi084.stdout:121: oid 22 updating version 0 to 19 2024-09-20T22:43:11.193 INFO:tasks.rados.rados.0.smithi084.stdout:121: oid 22 version 19 is already newer than 18 2024-09-20T22:43:11.193 INFO:tasks.rados.rados.0.smithi084.stdout:121: oid 22 version 19 is already newer than 16 2024-09-20T22:43:11.193 INFO:tasks.rados.rados.0.smithi084.stdout:121: oid 22 version 19 is already newer than 17 2024-09-20T22:43:11.193 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 22 v 19 (ObjNum 68 snap 7 seq_num 68) dirty exists 2024-09-20T22:43:11.207 INFO:tasks.rados.rados.0.smithi084.stdout:121: left oid 22 (ObjNum 68 snap 7 seq_num 68) 2024-09-20T22:43:11.207 INFO:tasks.rados.rados.0.smithi084.stdout:120: done (3 left) 2024-09-20T22:43:11.207 INFO:tasks.rados.rados.0.smithi084.stdout:121: done (2 left) 2024-09-20T22:43:11.207 INFO:tasks.rados.rados.0.smithi084.stdout:124: done (1 left) 2024-09-20T22:43:11.207 INFO:tasks.rados.rados.0.smithi084.stdout:125: write oid 50 current snap is 7 2024-09-20T22:43:11.207 INFO:tasks.rados.rados.0.smithi084.stdout:123: expect (ObjNum 53 snap 2 seq_num 53) 2024-09-20T22:43:11.255 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:43:11.278 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:11.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:11.279 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:43:11.329 INFO:tasks.rados.rados.0.smithi084.stdout:125: seq_num 70 ranges {776490=714607,2246563=579356} 2024-09-20T22:43:11.337 INFO:tasks.rados.rados.0.smithi084.stdout:125: writing smithi08477203-50 from 776490 to 1491097 tid 1 2024-09-20T22:43:11.345 INFO:tasks.rados.rados.0.smithi084.stdout:125: writing smithi08477203-50 from 2246563 to 2825919 tid 2 2024-09-20T22:43:11.345 INFO:tasks.rados.rados.0.smithi084.stdout:123: done (1 left) 2024-09-20T22:43:11.345 INFO:tasks.rados.rados.0.smithi084.stdout:126: rollback oid 43 current snap is 7 2024-09-20T22:43:11.345 INFO:tasks.rados.rados.0.smithi084.stdout:126: done (1 left) 2024-09-20T22:43:11.345 INFO:tasks.rados.rados.0.smithi084.stdout:127: rollback oid 43 current snap is 7 2024-09-20T22:43:11.346 INFO:tasks.rados.rados.0.smithi084.stdout:127: done (1 left) 2024-09-20T22:43:11.346 INFO:tasks.rados.rados.0.smithi084.stdout:128: write (excl) oid 10 current snap is 7 2024-09-20T22:43:11.346 INFO:tasks.rados.rados.0.smithi084.stdout:125: finishing write tid 1 to smithi08477203-50 2024-09-20T22:43:11.346 INFO:tasks.rados.rados.0.smithi084.stdout:128: seq_num 71 ranges {793297=789856,2305045=723014} 2024-09-20T22:43:11.354 INFO:tasks.rados.rados.0.smithi084.stdout:128: writing smithi08477203-10 from 793297 to 1583153 tid 1 2024-09-20T22:43:11.362 INFO:tasks.rados.rados.0.smithi084.stdout:128: writing smithi08477203-10 from 2305045 to 3028059 tid 2 2024-09-20T22:43:11.362 INFO:tasks.rados.rados.0.smithi084.stdout:129: copy_from oid 8 from oid 6 current snap is 7 2024-09-20T22:43:11.362 INFO:tasks.rados.rados.0.smithi084.stdout:125: finishing write tid 2 to smithi08477203-50 2024-09-20T22:43:11.362 INFO:tasks.rados.rados.0.smithi084.stdout:125: finishing write tid 3 to smithi08477203-50 2024-09-20T22:43:11.362 INFO:tasks.rados.rados.0.smithi084.stdout:125: finishing write tid 4 to smithi08477203-50 2024-09-20T22:43:11.363 INFO:tasks.rados.rados.0.smithi084.stdout:125: oid 50 updating version 0 to 12 2024-09-20T22:43:11.363 INFO:tasks.rados.rados.0.smithi084.stdout:125: oid 50 version 12 is already newer than 11 2024-09-20T22:43:11.363 INFO:tasks.rados.rados.0.smithi084.stdout:125: oid 50 updating version 12 to 13 2024-09-20T22:43:11.363 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 50 v 13 (ObjNum 70 snap 7 seq_num 70) dirty exists 2024-09-20T22:43:11.363 INFO:tasks.rados.rados.0.smithi084.stdout:125: left oid 50 (ObjNum 70 snap 7 seq_num 70) 2024-09-20T22:43:11.363 INFO:tasks.rados.rados.0.smithi084.stdout:128: finishing write tid 1 to smithi08477203-10 2024-09-20T22:43:11.363 INFO:tasks.rados.rados.0.smithi084.stdout:125: done (2 left) 2024-09-20T22:43:11.363 INFO:tasks.rados.rados.0.smithi084.stdout:130: read oid 46 snap -1 2024-09-20T22:43:11.363 INFO:tasks.rados.rados.0.smithi084.stdout:130: expect (ObjNum 45 snap 0 seq_num 45) 2024-09-20T22:43:11.363 INFO:tasks.rados.rados.0.smithi084.stdout:131: write (excl) oid 20 current snap is 7 2024-09-20T22:43:11.363 INFO:tasks.rados.rados.0.smithi084.stdout:131: seq_num 72 ranges {410104=465105,1563527=583025,2810926=399178} 2024-09-20T22:43:11.367 INFO:tasks.rados.rados.0.smithi084.stdout:131: writing smithi08477203-20 from 410104 to 875209 tid 1 2024-09-20T22:43:11.374 INFO:tasks.rados.rados.0.smithi084.stdout:131: writing smithi08477203-20 from 1563527 to 2146552 tid 2 2024-09-20T22:43:11.380 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:43:11.381 INFO:tasks.rados.rados.0.smithi084.stdout:131: writing smithi08477203-20 from 2810926 to 3210104 tid 3 2024-09-20T22:43:11.381 INFO:tasks.rados.rados.0.smithi084.stdout:132: snap_create 2024-09-20T22:43:11.381 INFO:tasks.rados.rados.0.smithi084.stdout:128: finishing write tid 2 to smithi08477203-10 2024-09-20T22:43:11.381 INFO:tasks.rados.rados.0.smithi084.stdout:128: finishing write tid 3 to smithi08477203-10 2024-09-20T22:43:11.381 INFO:tasks.rados.rados.0.smithi084.stdout:128: finishing write tid 4 to smithi08477203-10 2024-09-20T22:43:11.381 INFO:tasks.rados.rados.0.smithi084.stdout:128: oid 10 updating version 0 to 26 2024-09-20T22:43:11.381 INFO:tasks.rados.rados.0.smithi084.stdout:128: oid 10 version 26 is already newer than 25 2024-09-20T22:43:11.381 INFO:tasks.rados.rados.0.smithi084.stdout:128: oid 10 version 26 is already newer than 24 2024-09-20T22:43:11.381 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 10 v 26 (ObjNum 71 snap 7 seq_num 71) dirty exists 2024-09-20T22:43:11.381 INFO:tasks.rados.rados.0.smithi084.stdout:128: left oid 10 (ObjNum 71 snap 7 seq_num 71) 2024-09-20T22:43:11.381 INFO:tasks.rados.rados.0.smithi084.stdout:129: finishing copy_from racing read to smithi08477203-8 2024-09-20T22:43:11.382 INFO:tasks.rados.rados.0.smithi084.stdout:130: expect (ObjNum 45 snap 0 seq_num 45) 2024-09-20T22:43:11.402 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:11.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:11.404 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:43:11.465 INFO:tasks.rados.rados.0.smithi084.stdout:131: finishing write tid 1 to smithi08477203-20 2024-09-20T22:43:11.465 INFO:tasks.rados.rados.0.smithi084.stdout:129: finishing copy_from to smithi08477203-8 2024-09-20T22:43:11.465 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 8 v 3 (ObjNum 5 snap 0 seq_num 5) dirty exists 2024-09-20T22:43:11.465 INFO:tasks.rados.rados.0.smithi084.stdout:131: finishing write tid 2 to smithi08477203-20 2024-09-20T22:43:11.465 INFO:tasks.rados.rados.0.smithi084.stdout:131: finishing write tid 3 to smithi08477203-20 2024-09-20T22:43:11.465 INFO:tasks.rados.rados.0.smithi084.stdout:131: finishing write tid 4 to smithi08477203-20 2024-09-20T22:43:11.465 INFO:tasks.rados.rados.0.smithi084.stdout:131: finishing write tid 5 to smithi08477203-20 2024-09-20T22:43:11.465 INFO:tasks.rados.rados.0.smithi084.stdout:131: oid 20 updating version 0 to 14 2024-09-20T22:43:11.465 INFO:tasks.rados.rados.0.smithi084.stdout:131: oid 20 updating version 14 to 16 2024-09-20T22:43:11.465 INFO:tasks.rados.rados.0.smithi084.stdout:131: oid 20 version 16 is already newer than 15 2024-09-20T22:43:11.465 INFO:tasks.rados.rados.0.smithi084.stdout:131: oid 20 version 16 is already newer than 13 2024-09-20T22:43:11.465 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 20 v 16 (ObjNum 72 snap 7 seq_num 72) dirty exists 2024-09-20T22:43:11.465 INFO:tasks.rados.rados.0.smithi084.stdout:131: left oid 20 (ObjNum 72 snap 7 seq_num 72) 2024-09-20T22:43:11.504 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:43:11.525 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:11.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:11.526 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:43:11.558 INFO:journalctl@ceph.alertmanager.a.smithi084.stdout:Sep 20 22:43:11 smithi084 podman[79428]: 2024-09-20 22:43:11.2758379 +0000 UTC m=+1.222291418 container remove f8a0254a947f2e06943c648762037a40697bea24670a496865059ffd123b36a4 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-20T22:43:11.558 INFO:journalctl@ceph.alertmanager.a.smithi084.stdout:Sep 20 22:43:11 smithi084 podman[79428]: 2024-09-20 22:43:11.358534519 +0000 UTC m=+1.304988035 volume remove 47d8b74ca24a0f7ef6b50ed0562e14bce979c708ab157d4b7b3625f8d4e6b806 2024-09-20T22:43:11.626 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:43:11.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:11.649 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:11.649 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:43:11.749 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:43:11.771 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:11.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:11.772 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:43:11.849 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:11 smithi084 ceph-mon[40024]: pgmap v67: 158 pgs: 3 peering, 155 active+clean; 118 MiB data, 454 MiB used, 715 GiB / 715 GiB avail; 32 MiB/s rd, 14 op/s 2024-09-20T22:43:11.849 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:11 smithi084 ceph-mon[40024]: 5.16 deep-scrub starts 2024-09-20T22:43:11.850 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:11 smithi084 ceph-mon[40024]: 5.16 deep-scrub ok 2024-09-20T22:43:11.850 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:11 smithi084 ceph-mon[40024]: from='client.14865 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T22:43:11.850 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:11 smithi084 ceph-mon[40024]: osdmap e112: 8 total, 8 up, 8 in 2024-09-20T22:43:11.850 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:11 smithi084 ceph-mon[40024]: 5.9 scrub starts 2024-09-20T22:43:11.850 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:11 smithi084 ceph-mon[40024]: 5.9 scrub ok 2024-09-20T22:43:11.850 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:11 smithi084 ceph-mon[40024]: 4.14 scrub starts 2024-09-20T22:43:11.850 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:11 smithi084 ceph-mon[40024]: 4.14 scrub ok 2024-09-20T22:43:11.850 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:11 smithi084 ceph-mon[29774]: pgmap v67: 158 pgs: 3 peering, 155 active+clean; 118 MiB data, 454 MiB used, 715 GiB / 715 GiB avail; 32 MiB/s rd, 14 op/s 2024-09-20T22:43:11.850 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:11 smithi084 ceph-mon[29774]: 5.16 deep-scrub starts 2024-09-20T22:43:11.850 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:11 smithi084 ceph-mon[29774]: 5.16 deep-scrub ok 2024-09-20T22:43:11.850 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:11 smithi084 ceph-mon[29774]: from='client.14865 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T22:43:11.850 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:11 smithi084 ceph-mon[29774]: osdmap e112: 8 total, 8 up, 8 in 2024-09-20T22:43:11.851 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:11 smithi084 ceph-mon[29774]: 5.9 scrub starts 2024-09-20T22:43:11.851 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:11 smithi084 ceph-mon[29774]: 5.9 scrub ok 2024-09-20T22:43:11.851 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:11 smithi084 ceph-mon[29774]: 4.14 scrub starts 2024-09-20T22:43:11.851 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:11 smithi084 ceph-mon[29774]: 4.14 scrub ok 2024-09-20T22:43:11.873 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:43:11.878 INFO:teuthology.orchestra.run.smithi084.stdout:{"pg_ready":true,"pg_map":{"version":67,"stamp":"2024-09-20T22:43:09.622906+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":123353251,"num_objects":247,"num_object_clones":0,"num_object_copies":741,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":247,"num_whiteouts":0,"num_read":1105,"num_read_kb":175889,"num_write":713,"num_write_kb":100370,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2167,"ondisk_log_size":2167,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":474,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":464608,"kb_used_data":225900,"kb_used_omap":12,"kb_used_meta":236851,"kb_avail":749496608,"statfs":{"total":767960285184,"available":767484526592,"internally_reserved":0,"allocated":231321600,"data_stored":225696585,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":242535984},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":7019632,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":58,"num_read_kb":130909,"num_write":-38,"num_write_kb":28626,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-4,"num_bytes_recovered":-1067736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000596"},"pg_stats":[{"pgid":"3.1f","version":"86'2","reported_seq":110,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-20T22:43:08.110718+0000","last_change":"2024-09-20T22:42:39.664558+0000","last_active":"2024-09-20T22:43:08.110718+0000","last_peered":"2024-09-20T22:43:08.110718+0000","last_clean":"2024-09-20T22:43:08.110718+0000","last_became_active":"2024-09-20T22:41:38.159085+0000","last_became_peered":"2024-09-20T22:41:38.159085+0000","last_unstale":"2024-09-20T22:43:08.110718+0000","last_undegraded":"2024-09-20T22:43:08.110718+0000","last_fullsized":"2024-09-20T22:43:08.110718+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'2","last_scrub_stamp":"2024-09-20T22:42:39.664421+0000","last_deep_scrub":"86'2","last_deep_scrub_stamp":"2024-09-20T22:42:39.664421+0000","last_clean_scrub_stamp":"2024-09-20T22:42:39.664421+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:40.189792+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"74'9","reported_seq":107,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-20T22:43:08.107879+0000","last_change":"2024-09-20T22:41:40.169673+0000","last_active":"2024-09-20T22:43:08.107879+0000","last_peered":"2024-09-20T22:43:08.107879+0000","last_clean":"2024-09-20T22:43:08.107879+0000","last_became_active":"2024-09-20T22:41:40.168934+0000","last_became_peered":"2024-09-20T22:41:40.168934+0000","last_unstale":"2024-09-20T22:43:08.107879+0000","last_undegraded":"2024-09-20T22:43:08.107879+0000","last_fullsized":"2024-09-20T22:43:08.107879+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":87,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-20T22:43:08.119797+0000","last_change":"2024-09-20T22:41:42.182304+0000","last_active":"2024-09-20T22:43:08.119797+0000","last_peered":"2024-09-20T22:43:08.119797+0000","last_clean":"2024-09-20T22:43:08.119797+0000","last_became_active":"2024-09-20T22:41:42.182024+0000","last_became_peered":"2024-09-20T22:41:42.182024+0000","last_unstale":"2024-09-20T22:43:08.119797+0000","last_undegraded":"2024-09-20T22:43:08.119797+0000","last_fullsized":"2024-09-20T22:43:08.119797+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":83,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-20T22:43:08.107960+0000","last_change":"2024-09-20T22:41:44.198207+0000","last_active":"2024-09-20T22:43:08.107960+0000","last_peered":"2024-09-20T22:43:08.107960+0000","last_clean":"2024-09-20T22:43:08.107960+0000","last_became_active":"2024-09-20T22:41:44.197897+0000","last_became_peered":"2024-09-20T22:41:44.197897+0000","last_unstale":"2024-09-20T22:43:08.107960+0000","last_undegraded":"2024-09-20T22:43:08.107960+0000","last_fullsized":"2024-09-20T22:43:08.107960+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":91,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-20T22:43:08.111747+0000","last_change":"2024-09-20T22:43:07.476040+0000","last_active":"2024-09-20T22:43:08.111747+0000","last_peered":"2024-09-20T22:43:08.111747+0000","last_clean":"2024-09-20T22:43:08.111747+0000","last_became_active":"2024-09-20T22:41:44.832099+0000","last_became_peered":"2024-09-20T22:41:44.832099+0000","last_unstale":"2024-09-20T22:43:08.111747+0000","last_undegraded":"2024-09-20T22:43:08.111747+0000","last_fullsized":"2024-09-20T22:43:08.111747+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:07.475964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:43:07.475964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:15.747336+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":95,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-20T22:43:08.113727+0000","last_change":"2024-09-20T22:41:38.157685+0000","last_active":"2024-09-20T22:43:08.113727+0000","last_peered":"2024-09-20T22:43:08.113727+0000","last_clean":"2024-09-20T22:43:08.113727+0000","last_became_active":"2024-09-20T22:41:38.157298+0000","last_became_peered":"2024-09-20T22:41:38.157298+0000","last_unstale":"2024-09-20T22:43:08.113727+0000","last_undegraded":"2024-09-20T22:43:08.113727+0000","last_fullsized":"2024-09-20T22:43:08.113727+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"74'15","reported_seq":116,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-20T22:43:08.111115+0000","last_change":"2024-09-20T22:41:40.163585+0000","last_active":"2024-09-20T22:43:08.111115+0000","last_peered":"2024-09-20T22:43:08.111115+0000","last_clean":"2024-09-20T22:43:08.111115+0000","last_became_active":"2024-09-20T22:41:40.163242+0000","last_became_peered":"2024-09-20T22:41:40.163242+0000","last_unstale":"2024-09-20T22:43:08.111115+0000","last_undegraded":"2024-09-20T22:43:08.111115+0000","last_fullsized":"2024-09-20T22:43:08.111115+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":95,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-20T22:43:08.107782+0000","last_change":"2024-09-20T22:43:02.971265+0000","last_active":"2024-09-20T22:43:08.107782+0000","last_peered":"2024-09-20T22:43:08.107782+0000","last_clean":"2024-09-20T22:43:08.107782+0000","last_became_active":"2024-09-20T22:41:42.187076+0000","last_became_peered":"2024-09-20T22:41:42.187076+0000","last_unstale":"2024-09-20T22:43:08.107782+0000","last_undegraded":"2024-09-20T22:43:08.107782+0000","last_fullsized":"2024-09-20T22:43:08.107782+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:02.971194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:43:02.971194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:08.332638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":95,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-20T22:43:08.112425+0000","last_change":"2024-09-20T22:41:38.156694+0000","last_active":"2024-09-20T22:43:08.112425+0000","last_peered":"2024-09-20T22:43:08.112425+0000","last_clean":"2024-09-20T22:43:08.112425+0000","last_became_active":"2024-09-20T22:41:38.154827+0000","last_became_peered":"2024-09-20T22:41:38.154827+0000","last_unstale":"2024-09-20T22:43:08.112425+0000","last_undegraded":"2024-09-20T22:43:08.112425+0000","last_fullsized":"2024-09-20T22:43:08.112425+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"74'9","reported_seq":107,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-20T22:43:08.108133+0000","last_change":"2024-09-20T22:41:40.163006+0000","last_active":"2024-09-20T22:43:08.108133+0000","last_peered":"2024-09-20T22:43:08.108133+0000","last_clean":"2024-09-20T22:43:08.108133+0000","last_became_active":"2024-09-20T22:41:40.162546+0000","last_became_peered":"2024-09-20T22:41:40.162546+0000","last_unstale":"2024-09-20T22:43:08.108133+0000","last_undegraded":"2024-09-20T22:43:08.108133+0000","last_fullsized":"2024-09-20T22:43:08.108133+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":87,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-20T22:43:08.112241+0000","last_change":"2024-09-20T22:41:42.185835+0000","last_active":"2024-09-20T22:43:08.112241+0000","last_peered":"2024-09-20T22:43:08.112241+0000","last_clean":"2024-09-20T22:43:08.112241+0000","last_became_active":"2024-09-20T22:41:42.185538+0000","last_became_peered":"2024-09-20T22:41:42.185538+0000","last_unstale":"2024-09-20T22:43:08.112241+0000","last_undegraded":"2024-09-20T22:43:08.112241+0000","last_fullsized":"2024-09-20T22:43:08.112241+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":83,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-20T22:43:08.107824+0000","last_change":"2024-09-20T22:41:44.198169+0000","last_active":"2024-09-20T22:43:08.107824+0000","last_peered":"2024-09-20T22:43:08.107824+0000","last_clean":"2024-09-20T22:43:08.107824+0000","last_became_active":"2024-09-20T22:41:44.197113+0000","last_became_peered":"2024-09-20T22:41:44.197113+0000","last_unstale":"2024-09-20T22:43:08.107824+0000","last_undegraded":"2024-09-20T22:43:08.107824+0000","last_fullsized":"2024-09-20T22:43:08.107824+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":103,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-20T22:43:08.112310+0000","last_change":"2024-09-20T22:43:00.713953+0000","last_active":"2024-09-20T22:43:08.112310+0000","last_peered":"2024-09-20T22:43:08.112310+0000","last_clean":"2024-09-20T22:43:08.112310+0000","last_became_active":"2024-09-20T22:41:38.154650+0000","last_became_peered":"2024-09-20T22:41:38.154650+0000","last_unstale":"2024-09-20T22:43:08.112310+0000","last_undegraded":"2024-09-20T22:43:08.112310+0000","last_fullsized":"2024-09-20T22:43:08.112310+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:00.713878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:43:00.713878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:21.268170+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"74'5","reported_seq":101,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-20T22:43:08.108510+0000","last_change":"2024-09-20T22:41:40.171883+0000","last_active":"2024-09-20T22:43:08.108510+0000","last_peered":"2024-09-20T22:43:08.108510+0000","last_clean":"2024-09-20T22:43:08.108510+0000","last_became_active":"2024-09-20T22:41:40.171081+0000","last_became_peered":"2024-09-20T22:41:40.171081+0000","last_unstale":"2024-09-20T22:43:08.108510+0000","last_undegraded":"2024-09-20T22:43:08.108510+0000","last_fullsized":"2024-09-20T22:43:08.108510+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":98,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-20T22:43:08.116912+0000","last_change":"2024-09-20T22:43:01.248846+0000","last_active":"2024-09-20T22:43:08.116912+0000","last_peered":"2024-09-20T22:43:08.116912+0000","last_clean":"2024-09-20T22:43:08.116912+0000","last_became_active":"2024-09-20T22:41:42.182916+0000","last_became_peered":"2024-09-20T22:41:42.182916+0000","last_unstale":"2024-09-20T22:43:08.116912+0000","last_undegraded":"2024-09-20T22:43:08.116912+0000","last_fullsized":"2024-09-20T22:43:08.116912+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:01.248740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:43:01.248740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:09.032402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":83,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-20T22:43:08.111892+0000","last_change":"2024-09-20T22:41:44.198221+0000","last_active":"2024-09-20T22:43:08.111892+0000","last_peered":"2024-09-20T22:43:08.111892+0000","last_clean":"2024-09-20T22:43:08.111892+0000","last_became_active":"2024-09-20T22:41:44.197927+0000","last_became_peered":"2024-09-20T22:41:44.197927+0000","last_unstale":"2024-09-20T22:43:08.111892+0000","last_undegraded":"2024-09-20T22:43:08.111892+0000","last_fullsized":"2024-09-20T22:43:08.111892+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":83,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-20T22:43:08.108624+0000","last_change":"2024-09-20T22:41:44.832646+0000","last_active":"2024-09-20T22:43:08.108624+0000","last_peered":"2024-09-20T22:43:08.108624+0000","last_clean":"2024-09-20T22:43:08.108624+0000","last_became_active":"2024-09-20T22:41:44.832360+0000","last_became_peered":"2024-09-20T22:41:44.832360+0000","last_unstale":"2024-09-20T22:43:08.108624+0000","last_undegraded":"2024-09-20T22:43:08.108624+0000","last_fullsized":"2024-09-20T22:43:08.108624+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":95,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-20T22:43:08.110095+0000","last_change":"2024-09-20T22:41:38.152726+0000","last_active":"2024-09-20T22:43:08.110095+0000","last_peered":"2024-09-20T22:43:08.110095+0000","last_clean":"2024-09-20T22:43:08.110095+0000","last_became_active":"2024-09-20T22:41:38.152443+0000","last_became_peered":"2024-09-20T22:41:38.152443+0000","last_unstale":"2024-09-20T22:43:08.110095+0000","last_undegraded":"2024-09-20T22:43:08.110095+0000","last_fullsized":"2024-09-20T22:43:08.110095+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"74'15","reported_seq":116,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-20T22:43:08.113687+0000","last_change":"2024-09-20T22:41:40.170414+0000","last_active":"2024-09-20T22:43:08.113687+0000","last_peered":"2024-09-20T22:43:08.113687+0000","last_clean":"2024-09-20T22:43:08.113687+0000","last_became_active":"2024-09-20T22:41:40.170131+0000","last_became_peered":"2024-09-20T22:41:40.170131+0000","last_unstale":"2024-09-20T22:43:08.113687+0000","last_undegraded":"2024-09-20T22:43:08.113687+0000","last_fullsized":"2024-09-20T22:43:08.113687+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":87,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-20T22:43:08.118789+0000","last_change":"2024-09-20T22:41:42.185171+0000","last_active":"2024-09-20T22:43:08.118789+0000","last_peered":"2024-09-20T22:43:08.118789+0000","last_clean":"2024-09-20T22:43:08.118789+0000","last_became_active":"2024-09-20T22:41:42.184759+0000","last_became_peered":"2024-09-20T22:41:42.184759+0000","last_unstale":"2024-09-20T22:43:08.118789+0000","last_undegraded":"2024-09-20T22:43:08.118789+0000","last_fullsized":"2024-09-20T22:43:08.118789+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":83,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-20T22:43:08.114624+0000","last_change":"2024-09-20T22:41:44.832468+0000","last_active":"2024-09-20T22:43:08.114624+0000","last_peered":"2024-09-20T22:43:08.114624+0000","last_clean":"2024-09-20T22:43:08.114624+0000","last_became_active":"2024-09-20T22:41:44.832131+0000","last_became_peered":"2024-09-20T22:41:44.832131+0000","last_unstale":"2024-09-20T22:43:08.114624+0000","last_undegraded":"2024-09-20T22:43:08.114624+0000","last_fullsized":"2024-09-20T22:43:08.114624+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:08.595231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":95,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-20T22:43:08.108237+0000","last_change":"2024-09-20T22:41:38.150584+0000","last_active":"2024-09-20T22:43:08.108237+0000","last_peered":"2024-09-20T22:43:08.108237+0000","last_clean":"2024-09-20T22:43:08.108237+0000","last_became_active":"2024-09-20T22:41:38.150232+0000","last_became_peered":"2024-09-20T22:41:38.150232+0000","last_unstale":"2024-09-20T22:43:08.108237+0000","last_undegraded":"2024-09-20T22:43:08.108237+0000","last_fullsized":"2024-09-20T22:43:08.108237+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"74'12","reported_seq":113,"reported_epoch":108,"state":"peering","last_fresh":"2024-09-20T22:43:07.549845+0000","last_change":"2024-09-20T22:43:08.111784+0000","last_active":"2024-09-20T22:43:08.111784+0000","last_peered":"2024-09-20T22:43:07.549845+0000","last_clean":"2024-09-20T22:43:07.549845+0000","last_became_active":"2024-09-20T22:41:40.166087+0000","last_became_peered":"2024-09-20T22:41:40.166087+0000","last_unstale":"2024-09-20T22:43:07.549845+0000","last_undegraded":"2024-09-20T22:43:07.549845+0000","last_fullsized":"2024-09-20T22:43:07.549845+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,2,4],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":87,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-20T22:43:08.108281+0000","last_change":"2024-09-20T22:41:42.183266+0000","last_active":"2024-09-20T22:43:08.108281+0000","last_peered":"2024-09-20T22:43:08.108281+0000","last_clean":"2024-09-20T22:43:08.108281+0000","last_became_active":"2024-09-20T22:41:42.183005+0000","last_became_peered":"2024-09-20T22:41:42.183005+0000","last_unstale":"2024-09-20T22:43:08.108281+0000","last_undegraded":"2024-09-20T22:43:08.108281+0000","last_fullsized":"2024-09-20T22:43:08.108281+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":84,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-20T22:43:08.117300+0000","last_change":"2024-09-20T22:41:44.200115+0000","last_active":"2024-09-20T22:43:08.117300+0000","last_peered":"2024-09-20T22:43:08.117300+0000","last_clean":"2024-09-20T22:43:08.117300+0000","last_became_active":"2024-09-20T22:41:44.199822+0000","last_became_peered":"2024-09-20T22:41:44.199822+0000","last_unstale":"2024-09-20T22:43:08.117300+0000","last_undegraded":"2024-09-20T22:43:08.117300+0000","last_fullsized":"2024-09-20T22:43:08.117300+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"67'1","reported_seq":96,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-20T22:43:08.119922+0000","last_change":"2024-09-20T22:41:38.155735+0000","last_active":"2024-09-20T22:43:08.119922+0000","last_peered":"2024-09-20T22:43:08.119922+0000","last_clean":"2024-09-20T22:43:08.119922+0000","last_became_active":"2024-09-20T22:41:38.155424+0000","last_became_peered":"2024-09-20T22:41:38.155424+0000","last_unstale":"2024-09-20T22:43:08.119922+0000","last_undegraded":"2024-09-20T22:43:08.119922+0000","last_fullsized":"2024-09-20T22:43:08.119922+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"74'10","reported_seq":106,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-20T22:43:08.109417+0000","last_change":"2024-09-20T22:41:40.252498+0000","last_active":"2024-09-20T22:43:08.109417+0000","last_peered":"2024-09-20T22:43:08.109417+0000","last_clean":"2024-09-20T22:43:08.109417+0000","last_became_active":"2024-09-20T22:41:40.252001+0000","last_became_peered":"2024-09-20T22:41:40.252001+0000","last_unstale":"2024-09-20T22:43:08.109417+0000","last_undegraded":"2024-09-20T22:43:08.109417+0000","last_fullsized":"2024-09-20T22:43:08.109417+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"74'8","reported_seq":118,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-20T22:43:07.549468+0000","last_change":"2024-09-20T22:41:42.186385+0000","last_active":"2024-09-20T22:43:07.549468+0000","last_peered":"2024-09-20T22:43:07.549468+0000","last_clean":"2024-09-20T22:43:07.549468+0000","last_became_active":"2024-09-20T22:41:42.186071+0000","last_became_peered":"2024-09-20T22:41:42.186071+0000","last_unstale":"2024-09-20T22:43:07.549468+0000","last_undegraded":"2024-09-20T22:43:07.549468+0000","last_fullsized":"2024-09-20T22:43:07.549468+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.c","version":"106'38","reported_seq":128,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-20T22:43:07.814292+0000","last_change":"2024-09-20T22:43:07.549682+0000","last_active":"2024-09-20T22:43:07.814292+0000","last_peered":"2024-09-20T22:43:07.814292+0000","last_clean":"2024-09-20T22:43:07.814292+0000","last_became_active":"2024-09-20T22:42:52.404027+0000","last_became_peered":"2024-09-20T22:42:52.404027+0000","last_unstale":"2024-09-20T22:43:07.814292+0000","last_undegraded":"2024-09-20T22:43:07.814292+0000","last_fullsized":"2024-09-20T22:43:07.814292+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:42:51.354519+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:01.332745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00057854800000000002,"stat_sum":{"num_bytes":17563084,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":51,"num_read_kb":41193,"num_write":40,"num_write_kb":10933,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"5"}]},{"pgid":"1.a","version":"77'128","reported_seq":336,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-20T22:43:08.111857+0000","last_change":"2024-09-20T22:43:05.763879+0000","last_active":"2024-09-20T22:43:08.111857+0000","last_peered":"2024-09-20T22:43:08.111857+0000","last_clean":"2024-09-20T22:43:08.111857+0000","last_became_active":"2024-09-20T22:42:52.407791+0000","last_became_peered":"2024-09-20T22:42:52.407791+0000","last_unstale":"2024-09-20T22:43:08.111857+0000","last_undegraded":"2024-09-20T22:43:08.111857+0000","last_fullsized":"2024-09-20T22:43:08.111857+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:43:05.763796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:43:05.763796+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:34.531815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"74'15","reported_seq":116,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-20T22:43:08.120008+0000","last_change":"2024-09-20T22:41:40.168188+0000","last_active":"2024-09-20T22:43:08.120008+0000","last_peered":"2024-09-20T22:43:08.120008+0000","last_clean":"2024-09-20T22:43:08.120008+0000","last_became_active":"2024-09-20T22:41:40.165963+0000","last_became_peered":"2024-09-20T22:41:40.165963+0000","last_unstale":"2024-09-20T22:43:08.120008+0000","last_undegraded":"2024-09-20T22:43:08.120008+0000","last_fullsized":"2024-09-20T22:43:08.120008+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":95,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-20T22:43:08.108621+0000","last_change":"2024-09-20T22:41:38.157043+0000","last_active":"2024-09-20T22:43:08.108621+0000","last_peered":"2024-09-20T22:43:08.108621+0000","last_clean":"2024-09-20T22:43:08.108621+0000","last_became_active":"2024-09-20T22:41:38.156749+0000","last_became_peered":"2024-09-20T22:41:38.156749+0000","last_unstale":"2024-09-20T22:43:08.108621+0000","last_undegraded":"2024-09-20T22:43:08.108621+0000","last_fullsized":"2024-09-20T22:43:08.108621+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"74'8","reported_seq":114,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-20T22:43:08.109005+0000","last_change":"2024-09-20T22:41:42.182555+0000","last_active":"2024-09-20T22:43:08.109005+0000","last_peered":"2024-09-20T22:43:08.109005+0000","last_clean":"2024-09-20T22:43:08.109005+0000","last_became_active":"2024-09-20T22:41:42.182224+0000","last_became_peered":"2024-09-20T22:41:42.182224+0000","last_unstale":"2024-09-20T22:43:08.109005+0000","last_undegraded":"2024-09-20T22:43:08.109005+0000","last_fullsized":"2024-09-20T22:43:08.109005+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":83,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-20T22:43:08.112022+0000","last_change":"2024-09-20T22:41:44.198655+0000","last_active":"2024-09-20T22:43:08.112022+0000","last_peered":"2024-09-20T22:43:08.112022+0000","last_clean":"2024-09-20T22:43:08.112022+0000","last_became_active":"2024-09-20T22:41:44.198347+0000","last_became_peered":"2024-09-20T22:41:44.198347+0000","last_unstale":"2024-09-20T22:43:08.112022+0000","last_undegraded":"2024-09-20T22:43:08.112022+0000","last_fullsized":"2024-09-20T22:43:08.112022+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3","version":"98'3","reported_seq":76,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-20T22:43:08.111622+0000","last_change":"2024-09-20T22:43:07.551059+0000","last_active":"2024-09-20T22:43:08.111622+0000","last_peered":"2024-09-20T22:43:08.111622+0000","last_clean":"2024-09-20T22:43:08.111622+0000","last_became_active":"2024-09-20T22:42:52.406126+0000","last_became_peered":"2024-09-20T22:42:52.406126+0000","last_unstale":"2024-09-20T22:43:08.111622+0000","last_undegraded":"2024-09-20T22:43:08.111622+0000","last_fullsized":"2024-09-20T22:43:08.111622+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:42:51.354519+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:07.571370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00124,"stat_sum":{"num_bytes":2521017,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":4,"num_write_kb":1252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"3","length":"4"}]},{"pgid":"1.5","version":"77'128","reported_seq":336,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-20T22:43:08.108849+0000","last_change":"2024-09-20T22:42:58.843492+0000","last_active":"2024-09-20T22:43:08.108849+0000","last_peered":"2024-09-20T22:43:08.108849+0000","last_clean":"2024-09-20T22:43:08.108849+0000","last_became_active":"2024-09-20T22:42:52.410288+0000","last_became_peered":"2024-09-20T22:42:52.410288+0000","last_unstale":"2024-09-20T22:43:08.108849+0000","last_undegraded":"2024-09-20T22:43:08.108849+0000","last_fullsized":"2024-09-20T22:43:08.108849+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:42:58.843419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:42:58.843419+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:23.009410+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"74'18","reported_seq":123,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-20T22:43:08.108422+0000","last_change":"2024-09-20T22:41:40.252203+0000","last_active":"2024-09-20T22:43:08.108422+0000","last_peered":"2024-09-20T22:43:08.108422+0000","last_clean":"2024-09-20T22:43:08.108422+0000","last_became_active":"2024-09-20T22:41:40.251868+0000","last_became_peered":"2024-09-20T22:41:40.251868+0000","last_unstale":"2024-09-20T22:43:08.108422+0000","last_undegraded":"2024-09-20T22:43:08.108422+0000","last_fullsized":"2024-09-20T22:43:08.108422+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":95,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-20T22:43:08.115756+0000","last_change":"2024-09-20T22:41:38.161991+0000","last_active":"2024-09-20T22:43:08.115756+0000","last_peered":"2024-09-20T22:43:08.115756+0000","last_clean":"2024-09-20T22:43:08.115756+0000","last_became_active":"2024-09-20T22:41:38.161335+0000","last_became_peered":"2024-09-20T22:41:38.161335+0000","last_unstale":"2024-09-20T22:43:08.115756+0000","last_undegraded":"2024-09-20T22:43:08.115756+0000","last_fullsized":"2024-09-20T22:43:08.115756+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":95,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-20T22:43:08.108083+0000","last_change":"2024-09-20T22:43:07.948019+0000","last_active":"2024-09-20T22:43:08.108083+0000","last_peered":"2024-09-20T22:43:08.108083+0000","last_clean":"2024-09-20T22:43:08.108083+0000","last_became_active":"2024-09-20T22:41:42.189743+0000","last_became_peered":"2024-09-20T22:41:42.189743+0000","last_unstale":"2024-09-20T22:43:08.108083+0000","last_undegraded":"2024-09-20T22:43:08.108083+0000","last_fullsized":"2024-09-20T22:43:08.108083+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:07.947945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:07.947945+0000","last_clean_scrub_stamp":"2024-09-20T22:43:07.947945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:23.340969+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":83,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-20T22:43:08.108086+0000","last_change":"2024-09-20T22:41:44.193944+0000","last_active":"2024-09-20T22:43:08.108086+0000","last_peered":"2024-09-20T22:43:08.108086+0000","last_clean":"2024-09-20T22:43:08.108086+0000","last_became_active":"2024-09-20T22:41:44.193646+0000","last_became_peered":"2024-09-20T22:41:44.193646+0000","last_unstale":"2024-09-20T22:43:08.108086+0000","last_undegraded":"2024-09-20T22:43:08.108086+0000","last_fullsized":"2024-09-20T22:43:08.108086+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2","version":"106'31","reported_seq":112,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-20T22:43:08.111402+0000","last_change":"2024-09-20T22:43:07.552853+0000","last_active":"2024-09-20T22:43:08.111402+0000","last_peered":"2024-09-20T22:43:08.111402+0000","last_clean":"2024-09-20T22:43:08.111402+0000","last_became_active":"2024-09-20T22:42:52.402252+0000","last_became_peered":"2024-09-20T22:42:52.402252+0000","last_unstale":"2024-09-20T22:43:08.111402+0000","last_undegraded":"2024-09-20T22:43:08.111402+0000","last_fullsized":"2024-09-20T22:43:08.111402+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:42:51.354519+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:18.866888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003131029,"stat_sum":{"num_bytes":8252138,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":22,"num_read_kb":4189,"num_write":28,"num_write_kb":10903,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"3","length":"4"}]},{"pgid":"1.4","version":"77'128","reported_seq":336,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-20T22:43:08.109932+0000","last_change":"2024-09-20T22:43:02.448293+0000","last_active":"2024-09-20T22:43:08.109932+0000","last_peered":"2024-09-20T22:43:08.109932+0000","last_clean":"2024-09-20T22:43:08.109932+0000","last_became_active":"2024-09-20T22:42:52.405176+0000","last_became_peered":"2024-09-20T22:42:52.405176+0000","last_unstale":"2024-09-20T22:43:08.109932+0000","last_undegraded":"2024-09-20T22:43:08.109932+0000","last_fullsized":"2024-09-20T22:43:08.109932+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:43:02.448201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:43:02.448201+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:24.115959+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"74'14","reported_seq":112,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-20T22:43:08.108312+0000","last_change":"2024-09-20T22:41:40.164011+0000","last_active":"2024-09-20T22:43:08.108312+0000","last_peered":"2024-09-20T22:43:08.108312+0000","last_clean":"2024-09-20T22:43:08.108312+0000","last_became_active":"2024-09-20T22:41:40.163695+0000","last_became_peered":"2024-09-20T22:41:40.163695+0000","last_unstale":"2024-09-20T22:43:08.108312+0000","last_undegraded":"2024-09-20T22:43:08.108312+0000","last_fullsized":"2024-09-20T22:43:08.108312+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"67'1","reported_seq":96,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-20T22:43:08.110405+0000","last_change":"2024-09-20T22:41:38.150707+0000","last_active":"2024-09-20T22:43:08.110405+0000","last_peered":"2024-09-20T22:43:08.110405+0000","last_clean":"2024-09-20T22:43:08.110405+0000","last_became_active":"2024-09-20T22:41:38.150331+0000","last_became_peered":"2024-09-20T22:41:38.150331+0000","last_unstale":"2024-09-20T22:43:08.110405+0000","last_undegraded":"2024-09-20T22:43:08.110405+0000","last_fullsized":"2024-09-20T22:43:08.110405+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"74'8","reported_seq":114,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-20T22:43:08.108212+0000","last_change":"2024-09-20T22:41:42.179893+0000","last_active":"2024-09-20T22:43:08.108212+0000","last_peered":"2024-09-20T22:43:08.108212+0000","last_clean":"2024-09-20T22:43:08.108212+0000","last_became_active":"2024-09-20T22:41:42.179613+0000","last_became_peered":"2024-09-20T22:41:42.179613+0000","last_unstale":"2024-09-20T22:43:08.108212+0000","last_undegraded":"2024-09-20T22:43:08.108212+0000","last_fullsized":"2024-09-20T22:43:08.108212+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":83,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-20T22:43:08.116927+0000","last_change":"2024-09-20T22:41:44.832161+0000","last_active":"2024-09-20T22:43:08.116927+0000","last_peered":"2024-09-20T22:43:08.116927+0000","last_clean":"2024-09-20T22:43:08.116927+0000","last_became_active":"2024-09-20T22:41:44.831901+0000","last_became_peered":"2024-09-20T22:41:44.831901+0000","last_unstale":"2024-09-20T22:43:08.116927+0000","last_undegraded":"2024-09-20T22:43:08.116927+0000","last_fullsized":"2024-09-20T22:43:08.116927+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:10.151190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1","version":"106'14","reported_seq":90,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-20T22:43:08.108009+0000","last_change":"2024-09-20T22:43:07.783462+0000","last_active":"2024-09-20T22:43:08.108009+0000","last_peered":"2024-09-20T22:43:08.108009+0000","last_clean":"2024-09-20T22:43:08.108009+0000","last_became_active":"2024-09-20T22:42:52.405353+0000","last_became_peered":"2024-09-20T22:42:52.405353+0000","last_unstale":"2024-09-20T22:43:08.108009+0000","last_undegraded":"2024-09-20T22:43:08.108009+0000","last_fullsized":"2024-09-20T22:43:08.108009+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:42:51.354519+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:08.130445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001031056,"stat_sum":{"num_bytes":5159683,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":10,"num_write_kb":7278,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"3","length":"4"}]},{"pgid":"1.7","version":"77'128","reported_seq":341,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-20T22:43:07.549253+0000","last_change":"2024-09-20T22:42:58.847432+0000","last_active":"2024-09-20T22:43:07.549253+0000","last_peered":"2024-09-20T22:43:07.549253+0000","last_clean":"2024-09-20T22:43:07.549253+0000","last_became_active":"2024-09-20T22:42:52.403126+0000","last_became_peered":"2024-09-20T22:42:52.403126+0000","last_unstale":"2024-09-20T22:43:07.549253+0000","last_undegraded":"2024-09-20T22:43:07.549253+0000","last_fullsized":"2024-09-20T22:43:07.549253+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:42:58.847355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:42:58.847355+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:04.937053+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"74'10","reported_seq":106,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-20T22:43:08.118782+0000","last_change":"2024-09-20T22:41:40.167041+0000","last_active":"2024-09-20T22:43:08.118782+0000","last_peered":"2024-09-20T22:43:08.118782+0000","last_clean":"2024-09-20T22:43:08.118782+0000","last_became_active":"2024-09-20T22:41:40.165653+0000","last_became_peered":"2024-09-20T22:41:40.165653+0000","last_unstale":"2024-09-20T22:43:08.118782+0000","last_undegraded":"2024-09-20T22:43:08.118782+0000","last_fullsized":"2024-09-20T22:43:08.118782+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":103,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-20T22:43:08.112443+0000","last_change":"2024-09-20T22:42:39.595884+0000","last_active":"2024-09-20T22:43:08.112443+0000","last_peered":"2024-09-20T22:43:08.112443+0000","last_clean":"2024-09-20T22:43:08.112443+0000","last_became_active":"2024-09-20T22:41:38.155629+0000","last_became_peered":"2024-09-20T22:41:38.155629+0000","last_unstale":"2024-09-20T22:43:08.112443+0000","last_undegraded":"2024-09-20T22:43:08.112443+0000","last_fullsized":"2024-09-20T22:43:08.112443+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:42:39.595799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:42:39.595799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:39.804106+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"74'8","reported_seq":114,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-20T22:43:08.109168+0000","last_change":"2024-09-20T22:41:42.185583+0000","last_active":"2024-09-20T22:43:08.109168+0000","last_peered":"2024-09-20T22:43:08.109168+0000","last_clean":"2024-09-20T22:43:08.109168+0000","last_became_active":"2024-09-20T22:41:42.185213+0000","last_became_peered":"2024-09-20T22:41:42.185213+0000","last_unstale":"2024-09-20T22:43:08.109168+0000","last_undegraded":"2024-09-20T22:43:08.109168+0000","last_fullsized":"2024-09-20T22:43:08.109168+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":83,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-20T22:43:08.107972+0000","last_change":"2024-09-20T22:41:44.197894+0000","last_active":"2024-09-20T22:43:08.107972+0000","last_peered":"2024-09-20T22:43:08.107972+0000","last_clean":"2024-09-20T22:43:08.107972+0000","last_became_active":"2024-09-20T22:41:44.197619+0000","last_became_peered":"2024-09-20T22:41:44.197619+0000","last_unstale":"2024-09-20T22:43:08.107972+0000","last_undegraded":"2024-09-20T22:43:08.107972+0000","last_fullsized":"2024-09-20T22:43:08.107972+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.0","version":"104'10","reported_seq":95,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-20T22:43:08.111838+0000","last_change":"2024-09-20T22:43:07.655948+0000","last_active":"2024-09-20T22:43:08.111838+0000","last_peered":"2024-09-20T22:43:08.111838+0000","last_clean":"2024-09-20T22:43:08.111838+0000","last_became_active":"2024-09-20T22:42:52.406385+0000","last_became_peered":"2024-09-20T22:42:52.406385+0000","last_unstale":"2024-09-20T22:43:08.111838+0000","last_undegraded":"2024-09-20T22:43:08.111838+0000","last_fullsized":"2024-09-20T22:43:08.111838+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:42:51.354519+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:11.914395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00070411400000000004,"stat_sum":{"num_bytes":7479016,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":34,"num_read_kb":30567,"num_write":13,"num_write_kb":6082,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"3","length":"4"}]},{"pgid":"1.6","version":"77'128","reported_seq":341,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-20T22:43:07.549172+0000","last_change":"2024-09-20T22:42:59.831663+0000","last_active":"2024-09-20T22:43:07.549172+0000","last_peered":"2024-09-20T22:43:07.549172+0000","last_clean":"2024-09-20T22:43:07.549172+0000","last_became_active":"2024-09-20T22:42:52.404024+0000","last_became_peered":"2024-09-20T22:42:52.404024+0000","last_unstale":"2024-09-20T22:43:07.549172+0000","last_undegraded":"2024-09-20T22:43:07.549172+0000","last_fullsized":"2024-09-20T22:43:07.549172+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:42:59.831571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:42:59.831571+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:13.419358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"74'19","reported_seq":135,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-20T22:43:08.109306+0000","last_change":"2024-09-20T22:43:01.750427+0000","last_active":"2024-09-20T22:43:08.109306+0000","last_peered":"2024-09-20T22:43:08.109306+0000","last_clean":"2024-09-20T22:43:08.109306+0000","last_became_active":"2024-09-20T22:41:40.251866+0000","last_became_peered":"2024-09-20T22:41:40.251866+0000","last_unstale":"2024-09-20T22:43:08.109306+0000","last_undegraded":"2024-09-20T22:43:08.109306+0000","last_fullsized":"2024-09-20T22:43:08.109306+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'19","last_scrub_stamp":"2024-09-20T22:43:01.750344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:43:01.750344+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:06.070428+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":95,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-20T22:43:08.119103+0000","last_change":"2024-09-20T22:41:38.161236+0000","last_active":"2024-09-20T22:43:08.119103+0000","last_peered":"2024-09-20T22:43:08.119103+0000","last_clean":"2024-09-20T22:43:08.119103+0000","last_became_active":"2024-09-20T22:41:38.160832+0000","last_became_peered":"2024-09-20T22:41:38.160832+0000","last_unstale":"2024-09-20T22:43:08.119103+0000","last_undegraded":"2024-09-20T22:43:08.119103+0000","last_fullsized":"2024-09-20T22:43:08.119103+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":85,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-20T22:43:07.548893+0000","last_change":"2024-09-20T22:41:42.185638+0000","last_active":"2024-09-20T22:43:07.548893+0000","last_peered":"2024-09-20T22:43:07.548893+0000","last_clean":"2024-09-20T22:43:07.548893+0000","last_became_active":"2024-09-20T22:41:42.184412+0000","last_became_peered":"2024-09-20T22:41:42.184412+0000","last_unstale":"2024-09-20T22:43:07.548893+0000","last_undegraded":"2024-09-20T22:43:07.548893+0000","last_fullsized":"2024-09-20T22:43:07.548893+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":83,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-20T22:43:08.119796+0000","last_change":"2024-09-20T22:41:44.830917+0000","last_active":"2024-09-20T22:43:08.119796+0000","last_peered":"2024-09-20T22:43:08.119796+0000","last_clean":"2024-09-20T22:43:08.119796+0000","last_became_active":"2024-09-20T22:41:44.830538+0000","last_became_peered":"2024-09-20T22:41:44.830538+0000","last_unstale":"2024-09-20T22:43:08.119796+0000","last_undegraded":"2024-09-20T22:43:08.119796+0000","last_fullsized":"2024-09-20T22:43:08.119796+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.7","version":"108'51","reported_seq":127,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-20T22:43:07.968448+0000","last_change":"2024-09-20T22:43:07.555262+0000","last_active":"2024-09-20T22:43:07.968448+0000","last_peered":"2024-09-20T22:43:07.968448+0000","last_clean":"2024-09-20T22:43:07.968448+0000","last_became_active":"2024-09-20T22:42:52.404632+0000","last_became_peered":"2024-09-20T22:42:52.404632+0000","last_unstale":"2024-09-20T22:43:07.968448+0000","last_undegraded":"2024-09-20T22:43:07.968448+0000","last_fullsized":"2024-09-20T22:43:07.968448+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:42:51.354519+0000","objects_scrubbed":0,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:13.638540+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.0057430550000000004,"stat_sum":{"num_bytes":13932838,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":17,"num_read_kb":9,"num_write":43,"num_write_kb":16194,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"5"}]},{"pgid":"1.1","version":"77'128","reported_seq":336,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-20T22:43:08.111773+0000","last_change":"2024-09-20T22:43:03.743535+0000","last_active":"2024-09-20T22:43:08.111773+0000","last_peered":"2024-09-20T22:43:08.111773+0000","last_clean":"2024-09-20T22:43:08.111773+0000","last_became_active":"2024-09-20T22:42:52.405670+0000","last_became_peered":"2024-09-20T22:42:52.405670+0000","last_unstale":"2024-09-20T22:43:08.111773+0000","last_undegraded":"2024-09-20T22:43:08.111773+0000","last_fullsized":"2024-09-20T22:43:08.111773+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:43:03.743457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:43:03.743457+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:26.644825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"86'38","reported_seq":176,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-20T22:43:08.119001+0000","last_change":"2024-09-20T22:42:43.935768+0000","last_active":"2024-09-20T22:43:08.119001+0000","last_peered":"2024-09-20T22:43:08.119001+0000","last_clean":"2024-09-20T22:43:08.119001+0000","last_became_active":"2024-09-20T22:41:40.166958+0000","last_became_peered":"2024-09-20T22:41:40.166958+0000","last_unstale":"2024-09-20T22:43:08.119001+0000","last_undegraded":"2024-09-20T22:43:08.119001+0000","last_fullsized":"2024-09-20T22:43:08.119001+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'38","last_scrub_stamp":"2024-09-20T22:42:43.935677+0000","last_deep_scrub":"86'38","last_deep_scrub_stamp":"2024-09-20T22:42:43.935677+0000","last_clean_scrub_stamp":"2024-09-20T22:42:43.935677+0000","objects_scrubbed":10,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:53.919117+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":103,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-20T22:43:08.108683+0000","last_change":"2024-09-20T22:42:37.810752+0000","last_active":"2024-09-20T22:43:08.108683+0000","last_peered":"2024-09-20T22:43:08.108683+0000","last_clean":"2024-09-20T22:43:08.108683+0000","last_became_active":"2024-09-20T22:41:38.152525+0000","last_became_peered":"2024-09-20T22:41:38.152525+0000","last_unstale":"2024-09-20T22:43:08.108683+0000","last_undegraded":"2024-09-20T22:43:08.108683+0000","last_fullsized":"2024-09-20T22:43:08.108683+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:42:37.810656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:42:37.810656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:57.321042+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"69'2","reported_seq":105,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-20T22:43:08.112509+0000","last_change":"2024-09-20T22:41:40.154106+0000","last_active":"2024-09-20T22:43:08.112509+0000","last_peered":"2024-09-20T22:43:08.112509+0000","last_clean":"2024-09-20T22:43:08.112509+0000","last_became_active":"2024-09-20T22:41:37.148479+0000","last_became_peered":"2024-09-20T22:41:37.148479+0000","last_unstale":"2024-09-20T22:43:08.112509+0000","last_undegraded":"2024-09-20T22:43:08.112509+0000","last_fullsized":"2024-09-20T22:43:08.112509+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:36.130684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:36.130684+0000","last_clean_scrub_stamp":"2024-09-20T22:41:36.130684+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087280599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":87,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-20T22:43:08.108257+0000","last_change":"2024-09-20T22:41:42.179979+0000","last_active":"2024-09-20T22:43:08.108257+0000","last_peered":"2024-09-20T22:43:08.108257+0000","last_clean":"2024-09-20T22:43:08.108257+0000","last_became_active":"2024-09-20T22:41:42.179742+0000","last_became_peered":"2024-09-20T22:41:42.179742+0000","last_unstale":"2024-09-20T22:43:08.108257+0000","last_undegraded":"2024-09-20T22:43:08.108257+0000","last_fullsized":"2024-09-20T22:43:08.108257+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":83,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-20T22:43:08.111404+0000","last_change":"2024-09-20T22:41:44.192602+0000","last_active":"2024-09-20T22:43:08.111404+0000","last_peered":"2024-09-20T22:43:08.111404+0000","last_clean":"2024-09-20T22:43:08.111404+0000","last_became_active":"2024-09-20T22:41:44.192128+0000","last_became_peered":"2024-09-20T22:41:44.192128+0000","last_unstale":"2024-09-20T22:43:08.111404+0000","last_undegraded":"2024-09-20T22:43:08.111404+0000","last_fullsized":"2024-09-20T22:43:08.111404+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"98'6","reported_seq":84,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-20T22:43:08.107974+0000","last_change":"2024-09-20T22:43:07.542487+0000","last_active":"2024-09-20T22:43:08.107974+0000","last_peered":"2024-09-20T22:43:08.107974+0000","last_clean":"2024-09-20T22:43:08.107974+0000","last_became_active":"2024-09-20T22:42:52.400310+0000","last_became_peered":"2024-09-20T22:42:52.400310+0000","last_unstale":"2024-09-20T22:43:08.107974+0000","last_undegraded":"2024-09-20T22:43:08.107974+0000","last_fullsized":"2024-09-20T22:43:08.107974+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:42:51.354519+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:19.880763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031965500000000002,"stat_sum":{"num_bytes":5378174,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":7778,"num_write":8,"num_write_kb":2437,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"3","length":"4"}]},{"pgid":"1.2","version":"77'128","reported_seq":377,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-20T22:43:08.116766+0000","last_change":"2024-09-20T22:43:04.267790+0000","last_active":"2024-09-20T22:43:08.116766+0000","last_peered":"2024-09-20T22:43:08.116766+0000","last_clean":"2024-09-20T22:43:08.116766+0000","last_became_active":"2024-09-20T22:42:52.403378+0000","last_became_peered":"2024-09-20T22:42:52.403378+0000","last_unstale":"2024-09-20T22:43:08.116766+0000","last_undegraded":"2024-09-20T22:43:08.116766+0000","last_fullsized":"2024-09-20T22:43:08.116766+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:43:04.267724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:43:04.267724+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:32.655523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"74'13","reported_seq":118,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-20T22:43:08.120111+0000","last_change":"2024-09-20T22:41:40.169777+0000","last_active":"2024-09-20T22:43:08.120111+0000","last_peered":"2024-09-20T22:43:08.120111+0000","last_clean":"2024-09-20T22:43:08.120111+0000","last_became_active":"2024-09-20T22:41:40.169473+0000","last_became_peered":"2024-09-20T22:41:40.169473+0000","last_unstale":"2024-09-20T22:43:08.120111+0000","last_undegraded":"2024-09-20T22:43:08.120111+0000","last_fullsized":"2024-09-20T22:43:08.120111+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":95,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-20T22:43:08.118888+0000","last_change":"2024-09-20T22:41:38.157113+0000","last_active":"2024-09-20T22:43:08.118888+0000","last_peered":"2024-09-20T22:43:08.118888+0000","last_clean":"2024-09-20T22:43:08.118888+0000","last_became_active":"2024-09-20T22:41:38.156837+0000","last_became_peered":"2024-09-20T22:41:38.156837+0000","last_unstale":"2024-09-20T22:43:08.118888+0000","last_undegraded":"2024-09-20T22:43:08.118888+0000","last_fullsized":"2024-09-20T22:43:08.118888+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"67'1","reported_seq":112,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-20T22:43:08.113250+0000","last_change":"2024-09-20T22:43:04.330733+0000","last_active":"2024-09-20T22:43:08.113250+0000","last_peered":"2024-09-20T22:43:08.113250+0000","last_clean":"2024-09-20T22:43:08.113250+0000","last_became_active":"2024-09-20T22:41:37.148165+0000","last_became_peered":"2024-09-20T22:41:37.148165+0000","last_unstale":"2024-09-20T22:43:08.113250+0000","last_undegraded":"2024-09-20T22:43:08.113250+0000","last_fullsized":"2024-09-20T22:43:08.113250+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-09-20T22:43:04.330598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:36.130684+0000","last_clean_scrub_stamp":"2024-09-20T22:43:04.330598+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:24.245637+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001301459,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":87,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-20T22:43:08.115245+0000","last_change":"2024-09-20T22:41:42.185412+0000","last_active":"2024-09-20T22:43:08.115245+0000","last_peered":"2024-09-20T22:43:08.115245+0000","last_clean":"2024-09-20T22:43:08.115245+0000","last_became_active":"2024-09-20T22:41:42.185112+0000","last_became_peered":"2024-09-20T22:41:42.185112+0000","last_unstale":"2024-09-20T22:43:08.115245+0000","last_undegraded":"2024-09-20T22:43:08.115245+0000","last_fullsized":"2024-09-20T22:43:08.115245+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":83,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-20T22:43:08.117398+0000","last_change":"2024-09-20T22:41:44.830804+0000","last_active":"2024-09-20T22:43:08.117398+0000","last_peered":"2024-09-20T22:43:08.117398+0000","last_clean":"2024-09-20T22:43:08.117398+0000","last_became_active":"2024-09-20T22:41:44.830515+0000","last_became_peered":"2024-09-20T22:41:44.830515+0000","last_unstale":"2024-09-20T22:43:08.117398+0000","last_undegraded":"2024-09-20T22:43:08.117398+0000","last_fullsized":"2024-09-20T22:43:08.117398+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"98'3","reported_seq":77,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-20T22:43:08.117891+0000","last_change":"2024-09-20T22:43:07.544734+0000","last_active":"2024-09-20T22:43:08.117891+0000","last_peered":"2024-09-20T22:43:08.117891+0000","last_clean":"2024-09-20T22:43:08.117891+0000","last_became_active":"2024-09-20T22:42:52.399644+0000","last_became_peered":"2024-09-20T22:42:52.399644+0000","last_unstale":"2024-09-20T22:43:08.117891+0000","last_undegraded":"2024-09-20T22:43:08.117891+0000","last_fullsized":"2024-09-20T22:43:08.117891+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:42:51.354519+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:12.919100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00080946800000000004,"stat_sum":{"num_bytes":2252105,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":4,"num_write_kb":878,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"3","length":"4"}]},{"pgid":"1.3","version":"77'128","reported_seq":336,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-20T22:43:08.120309+0000","last_change":"2024-09-20T22:43:03.999531+0000","last_active":"2024-09-20T22:43:08.120309+0000","last_peered":"2024-09-20T22:43:08.120309+0000","last_clean":"2024-09-20T22:43:08.120309+0000","last_became_active":"2024-09-20T22:42:52.409569+0000","last_became_peered":"2024-09-20T22:42:52.409569+0000","last_unstale":"2024-09-20T22:43:08.120309+0000","last_undegraded":"2024-09-20T22:43:08.120309+0000","last_fullsized":"2024-09-20T22:43:08.120309+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:43:03.999456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:43:03.999456+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:20.299971+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"74'12","reported_seq":108,"reported_epoch":108,"state":"peering","last_fresh":"2024-09-20T22:43:07.544223+0000","last_change":"2024-09-20T22:43:08.110027+0000","last_active":"2024-09-20T22:43:08.110027+0000","last_peered":"2024-09-20T22:43:07.544223+0000","last_clean":"2024-09-20T22:43:07.544223+0000","last_became_active":"2024-09-20T22:41:40.167169+0000","last_became_peered":"2024-09-20T22:41:40.167169+0000","last_unstale":"2024-09-20T22:43:07.544223+0000","last_undegraded":"2024-09-20T22:43:07.544223+0000","last_fullsized":"2024-09-20T22:43:07.544223+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,2,5],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":95,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-20T22:43:08.110434+0000","last_change":"2024-09-20T22:41:38.153448+0000","last_active":"2024-09-20T22:43:08.110434+0000","last_peered":"2024-09-20T22:43:08.110434+0000","last_clean":"2024-09-20T22:43:08.110434+0000","last_became_active":"2024-09-20T22:41:38.153058+0000","last_became_peered":"2024-09-20T22:41:38.153058+0000","last_unstale":"2024-09-20T22:43:08.110434+0000","last_undegraded":"2024-09-20T22:43:08.110434+0000","last_fullsized":"2024-09-20T22:43:08.110434+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"75'5","reported_seq":191,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-20T22:43:08.117288+0000","last_change":"2024-09-20T22:41:40.251762+0000","last_active":"2024-09-20T22:43:08.117288+0000","last_peered":"2024-09-20T22:43:08.117288+0000","last_clean":"2024-09-20T22:43:08.117288+0000","last_became_active":"2024-09-20T22:41:37.148383+0000","last_became_peered":"2024-09-20T22:41:37.148383+0000","last_unstale":"2024-09-20T22:43:08.117288+0000","last_undegraded":"2024-09-20T22:43:08.117288+0000","last_fullsized":"2024-09-20T22:43:08.117288+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:36.130684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:36.130684+0000","last_clean_scrub_stamp":"2024-09-20T22:41:36.130684+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00078631700000000003,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":85,"num_read_kb":80,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":87,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-20T22:43:08.112610+0000","last_change":"2024-09-20T22:41:42.187618+0000","last_active":"2024-09-20T22:43:08.112610+0000","last_peered":"2024-09-20T22:43:08.112610+0000","last_clean":"2024-09-20T22:43:08.112610+0000","last_became_active":"2024-09-20T22:41:42.187346+0000","last_became_peered":"2024-09-20T22:41:42.187346+0000","last_unstale":"2024-09-20T22:43:08.112610+0000","last_undegraded":"2024-09-20T22:43:08.112610+0000","last_fullsized":"2024-09-20T22:43:08.112610+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":91,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-20T22:43:08.118629+0000","last_change":"2024-09-20T22:43:00.128772+0000","last_active":"2024-09-20T22:43:08.118629+0000","last_peered":"2024-09-20T22:43:08.118629+0000","last_clean":"2024-09-20T22:43:08.118629+0000","last_became_active":"2024-09-20T22:41:44.199046+0000","last_became_peered":"2024-09-20T22:41:44.199046+0000","last_unstale":"2024-09-20T22:43:08.118629+0000","last_undegraded":"2024-09-20T22:43:08.118629+0000","last_fullsized":"2024-09-20T22:43:08.118629+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:00.128705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:43:00.128705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:10.682100+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.6","version":"106'12","reported_seq":92,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-20T22:43:08.108603+0000","last_change":"2024-09-20T22:43:07.544518+0000","last_active":"2024-09-20T22:43:08.108603+0000","last_peered":"2024-09-20T22:43:08.108603+0000","last_clean":"2024-09-20T22:43:08.108603+0000","last_became_active":"2024-09-20T22:42:52.401182+0000","last_became_peered":"2024-09-20T22:42:52.401182+0000","last_unstale":"2024-09-20T22:43:08.108603+0000","last_undegraded":"2024-09-20T22:43:08.108603+0000","last_fullsized":"2024-09-20T22:43:08.108603+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:42:51.354519+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:58.655300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00068414400000000003,"stat_sum":{"num_bytes":5260525,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":21,"num_read_kb":12203,"num_write":12,"num_write_kb":3013,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"3","length":"4"}]},{"pgid":"4.5","version":"74'16","reported_seq":130,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-20T22:43:07.549287+0000","last_change":"2024-09-20T22:41:40.253524+0000","last_active":"2024-09-20T22:43:07.549287+0000","last_peered":"2024-09-20T22:43:07.549287+0000","last_clean":"2024-09-20T22:43:07.549287+0000","last_became_active":"2024-09-20T22:41:40.252933+0000","last_became_peered":"2024-09-20T22:41:40.252933+0000","last_unstale":"2024-09-20T22:43:07.549287+0000","last_undegraded":"2024-09-20T22:43:07.549287+0000","last_fullsized":"2024-09-20T22:43:07.549287+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"67'1","reported_seq":101,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-20T22:43:08.115964+0000","last_change":"2024-09-20T22:41:38.159706+0000","last_active":"2024-09-20T22:43:08.115964+0000","last_peered":"2024-09-20T22:43:08.115964+0000","last_clean":"2024-09-20T22:43:08.115964+0000","last_became_active":"2024-09-20T22:41:38.158917+0000","last_became_peered":"2024-09-20T22:41:38.158917+0000","last_unstale":"2024-09-20T22:43:08.115964+0000","last_undegraded":"2024-09-20T22:43:08.115964+0000","last_fullsized":"2024-09-20T22:43:08.115964+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"77'128","reported_seq":369,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-20T22:43:08.117490+0000","last_change":"2024-09-20T22:42:59.238462+0000","last_active":"2024-09-20T22:43:08.117490+0000","last_peered":"2024-09-20T22:43:08.117490+0000","last_clean":"2024-09-20T22:43:08.117490+0000","last_became_active":"2024-09-20T22:42:47.192979+0000","last_became_peered":"2024-09-20T22:42:47.192979+0000","last_unstale":"2024-09-20T22:43:08.117490+0000","last_undegraded":"2024-09-20T22:43:08.117490+0000","last_fullsized":"2024-09-20T22:43:08.117490+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:42:59.238379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:42:59.238379+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:02.686288+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":87,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-20T22:43:08.117416+0000","last_change":"2024-09-20T22:41:42.185135+0000","last_active":"2024-09-20T22:43:08.117416+0000","last_peered":"2024-09-20T22:43:08.117416+0000","last_clean":"2024-09-20T22:43:08.117416+0000","last_became_active":"2024-09-20T22:41:42.184387+0000","last_became_peered":"2024-09-20T22:41:42.184387+0000","last_unstale":"2024-09-20T22:43:08.117416+0000","last_undegraded":"2024-09-20T22:43:08.117416+0000","last_fullsized":"2024-09-20T22:43:08.117416+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":91,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-20T22:43:08.112327+0000","last_change":"2024-09-20T22:43:02.696182+0000","last_active":"2024-09-20T22:43:08.112327+0000","last_peered":"2024-09-20T22:43:08.112327+0000","last_clean":"2024-09-20T22:43:08.112327+0000","last_became_active":"2024-09-20T22:41:44.190658+0000","last_became_peered":"2024-09-20T22:41:44.190658+0000","last_unstale":"2024-09-20T22:43:08.112327+0000","last_undegraded":"2024-09-20T22:43:08.112327+0000","last_fullsized":"2024-09-20T22:43:08.112327+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:02.696105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:43:02.696105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:13.579400+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"106'10","reported_seq":84,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-20T22:43:07.549301+0000","last_change":"2024-09-20T22:43:07.549548+0000","last_active":"2024-09-20T22:43:07.549301+0000","last_peered":"2024-09-20T22:43:07.549301+0000","last_clean":"2024-09-20T22:43:07.549301+0000","last_became_active":"2024-09-20T22:42:52.410381+0000","last_became_peered":"2024-09-20T22:42:52.410381+0000","last_unstale":"2024-09-20T22:43:07.549301+0000","last_undegraded":"2024-09-20T22:43:07.549301+0000","last_fullsized":"2024-09-20T22:43:07.549301+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:42:51.354519+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:16.114806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045886599999999999,"stat_sum":{"num_bytes":5042034,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":3489,"num_write":9,"num_write_kb":4782,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"5"}]},{"pgid":"4.e","version":"74'11","reported_seq":110,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-20T22:43:08.108808+0000","last_change":"2024-09-20T22:41:40.169542+0000","last_active":"2024-09-20T22:43:08.108808+0000","last_peered":"2024-09-20T22:43:08.108808+0000","last_clean":"2024-09-20T22:43:08.108808+0000","last_became_active":"2024-09-20T22:41:40.168942+0000","last_became_peered":"2024-09-20T22:41:40.168942+0000","last_unstale":"2024-09-20T22:43:08.108808+0000","last_undegraded":"2024-09-20T22:43:08.108808+0000","last_fullsized":"2024-09-20T22:43:08.108808+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":95,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-20T22:43:08.108485+0000","last_change":"2024-09-20T22:41:38.161801+0000","last_active":"2024-09-20T22:43:08.108485+0000","last_peered":"2024-09-20T22:43:08.108485+0000","last_clean":"2024-09-20T22:43:08.108485+0000","last_became_active":"2024-09-20T22:41:38.161500+0000","last_became_peered":"2024-09-20T22:41:38.161500+0000","last_unstale":"2024-09-20T22:43:08.108485+0000","last_undegraded":"2024-09-20T22:43:08.108485+0000","last_fullsized":"2024-09-20T22:43:08.108485+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":87,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-20T22:43:08.112296+0000","last_change":"2024-09-20T22:41:42.183601+0000","last_active":"2024-09-20T22:43:08.112296+0000","last_peered":"2024-09-20T22:43:08.112296+0000","last_clean":"2024-09-20T22:43:08.112296+0000","last_became_active":"2024-09-20T22:41:42.183302+0000","last_became_peered":"2024-09-20T22:41:42.183302+0000","last_unstale":"2024-09-20T22:43:08.112296+0000","last_undegraded":"2024-09-20T22:43:08.112296+0000","last_fullsized":"2024-09-20T22:43:08.112296+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:10.335337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":83,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-20T22:43:08.110039+0000","last_change":"2024-09-20T22:41:44.832295+0000","last_active":"2024-09-20T22:43:08.110039+0000","last_peered":"2024-09-20T22:43:08.110039+0000","last_clean":"2024-09-20T22:43:08.110039+0000","last_became_active":"2024-09-20T22:41:44.832009+0000","last_became_peered":"2024-09-20T22:41:44.832009+0000","last_unstale":"2024-09-20T22:43:08.110039+0000","last_undegraded":"2024-09-20T22:43:08.110039+0000","last_fullsized":"2024-09-20T22:43:08.110039+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.e","version":"98'6","reported_seq":86,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-20T22:43:08.114983+0000","last_change":"2024-09-20T22:43:07.543664+0000","last_active":"2024-09-20T22:43:08.114983+0000","last_peered":"2024-09-20T22:43:08.114983+0000","last_clean":"2024-09-20T22:43:08.114983+0000","last_became_active":"2024-09-20T22:42:52.405039+0000","last_became_peered":"2024-09-20T22:42:52.405039+0000","last_unstale":"2024-09-20T22:43:08.114983+0000","last_undegraded":"2024-09-20T22:43:08.114983+0000","last_fullsized":"2024-09-20T22:43:08.114983+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:42:51.354519+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:20.585552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059160599999999995,"stat_sum":{"num_bytes":4352947,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":21,"num_read_kb":15328,"num_write":8,"num_write_kb":1763,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"3","length":"4"}]},{"pgid":"1.8","version":"77'128","reported_seq":336,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-20T22:43:08.111716+0000","last_change":"2024-09-20T22:43:04.748279+0000","last_active":"2024-09-20T22:43:08.111716+0000","last_peered":"2024-09-20T22:43:08.111716+0000","last_clean":"2024-09-20T22:43:08.111716+0000","last_became_active":"2024-09-20T22:42:52.405594+0000","last_became_peered":"2024-09-20T22:42:52.405594+0000","last_unstale":"2024-09-20T22:43:08.111716+0000","last_undegraded":"2024-09-20T22:43:08.111716+0000","last_fullsized":"2024-09-20T22:43:08.111716+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:43:04.748176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:43:04.748176+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:15.996730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"74'17","reported_seq":127,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-20T22:43:08.108699+0000","last_change":"2024-09-20T22:43:00.910562+0000","last_active":"2024-09-20T22:43:08.108699+0000","last_peered":"2024-09-20T22:43:08.108699+0000","last_clean":"2024-09-20T22:43:08.108699+0000","last_became_active":"2024-09-20T22:41:40.169192+0000","last_became_peered":"2024-09-20T22:41:40.169192+0000","last_unstale":"2024-09-20T22:43:08.108699+0000","last_undegraded":"2024-09-20T22:43:08.108699+0000","last_fullsized":"2024-09-20T22:43:08.108699+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'17","last_scrub_stamp":"2024-09-20T22:43:00.910480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:43:00.910480+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:30.015789+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":93,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-20T22:43:07.549142+0000","last_change":"2024-09-20T22:41:38.154220+0000","last_active":"2024-09-20T22:43:07.549142+0000","last_peered":"2024-09-20T22:43:07.549142+0000","last_clean":"2024-09-20T22:43:07.549142+0000","last_became_active":"2024-09-20T22:41:38.153926+0000","last_became_peered":"2024-09-20T22:41:38.153926+0000","last_unstale":"2024-09-20T22:43:07.549142+0000","last_undegraded":"2024-09-20T22:43:07.549142+0000","last_fullsized":"2024-09-20T22:43:07.549142+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":87,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-20T22:43:08.118734+0000","last_change":"2024-09-20T22:41:42.183482+0000","last_active":"2024-09-20T22:43:08.118734+0000","last_peered":"2024-09-20T22:43:08.118734+0000","last_clean":"2024-09-20T22:43:08.118734+0000","last_became_active":"2024-09-20T22:41:42.183225+0000","last_became_peered":"2024-09-20T22:41:42.183225+0000","last_unstale":"2024-09-20T22:43:08.118734+0000","last_undegraded":"2024-09-20T22:43:08.118734+0000","last_fullsized":"2024-09-20T22:43:08.118734+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:10.228360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":91,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-20T22:43:08.112764+0000","last_change":"2024-09-20T22:42:59.250714+0000","last_active":"2024-09-20T22:43:08.112764+0000","last_peered":"2024-09-20T22:43:08.112764+0000","last_clean":"2024-09-20T22:43:08.112764+0000","last_became_active":"2024-09-20T22:41:44.197561+0000","last_became_peered":"2024-09-20T22:41:44.197561+0000","last_unstale":"2024-09-20T22:43:08.112764+0000","last_undegraded":"2024-09-20T22:43:08.112764+0000","last_fullsized":"2024-09-20T22:43:08.112764+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:42:59.250616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:42:59.250616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:10.878091+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"108'14","reported_seq":95,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-20T22:43:08.108620+0000","last_change":"2024-09-20T22:43:07.542761+0000","last_active":"2024-09-20T22:43:08.108620+0000","last_peered":"2024-09-20T22:43:08.108620+0000","last_clean":"2024-09-20T22:43:08.108620+0000","last_became_active":"2024-09-20T22:42:52.401210+0000","last_became_peered":"2024-09-20T22:42:52.401210+0000","last_unstale":"2024-09-20T22:43:08.108620+0000","last_undegraded":"2024-09-20T22:43:08.108620+0000","last_fullsized":"2024-09-20T22:43:08.108620+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:42:51.354519+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:03.041070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010310759999999999,"stat_sum":{"num_bytes":7932805,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":11559,"num_write":18,"num_write_kb":9368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"3","length":"4"}]},{"pgid":"1.9","version":"77'128","reported_seq":377,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-20T22:43:08.116921+0000","last_change":"2024-09-20T22:43:03.239347+0000","last_active":"2024-09-20T22:43:08.116921+0000","last_peered":"2024-09-20T22:43:08.116921+0000","last_clean":"2024-09-20T22:43:08.116921+0000","last_became_active":"2024-09-20T22:42:52.400353+0000","last_became_peered":"2024-09-20T22:42:52.400353+0000","last_unstale":"2024-09-20T22:43:08.116921+0000","last_undegraded":"2024-09-20T22:43:08.116921+0000","last_fullsized":"2024-09-20T22:43:08.116921+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:43:03.239276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:43:03.239276+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:16.195446+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"74'10","reported_seq":106,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-20T22:43:08.108396+0000","last_change":"2024-09-20T22:41:40.162973+0000","last_active":"2024-09-20T22:43:08.108396+0000","last_peered":"2024-09-20T22:43:08.108396+0000","last_clean":"2024-09-20T22:43:08.108396+0000","last_became_active":"2024-09-20T22:41:40.162494+0000","last_became_peered":"2024-09-20T22:41:40.162494+0000","last_unstale":"2024-09-20T22:43:08.108396+0000","last_undegraded":"2024-09-20T22:43:08.108396+0000","last_fullsized":"2024-09-20T22:43:08.108396+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":103,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-20T22:43:08.116035+0000","last_change":"2024-09-20T22:43:05.431444+0000","last_active":"2024-09-20T22:43:08.116035+0000","last_peered":"2024-09-20T22:43:08.116035+0000","last_clean":"2024-09-20T22:43:08.116035+0000","last_became_active":"2024-09-20T22:41:38.154636+0000","last_became_peered":"2024-09-20T22:41:38.154636+0000","last_unstale":"2024-09-20T22:43:08.116035+0000","last_undegraded":"2024-09-20T22:43:08.116035+0000","last_fullsized":"2024-09-20T22:43:08.116035+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:05.431346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:43:05.431346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:30.313458+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"74'8","reported_seq":114,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-20T22:43:08.113243+0000","last_change":"2024-09-20T22:41:42.186528+0000","last_active":"2024-09-20T22:43:08.113243+0000","last_peered":"2024-09-20T22:43:08.113243+0000","last_clean":"2024-09-20T22:43:08.113243+0000","last_became_active":"2024-09-20T22:41:42.185863+0000","last_became_peered":"2024-09-20T22:41:42.185863+0000","last_unstale":"2024-09-20T22:43:08.113243+0000","last_undegraded":"2024-09-20T22:43:08.113243+0000","last_fullsized":"2024-09-20T22:43:08.113243+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":83,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-20T22:43:08.108629+0000","last_change":"2024-09-20T22:41:44.195767+0000","last_active":"2024-09-20T22:43:08.108629+0000","last_peered":"2024-09-20T22:43:08.108629+0000","last_clean":"2024-09-20T22:43:08.108629+0000","last_became_active":"2024-09-20T22:41:44.195160+0000","last_became_peered":"2024-09-20T22:41:44.195160+0000","last_unstale":"2024-09-20T22:43:08.108629+0000","last_undegraded":"2024-09-20T22:43:08.108629+0000","last_fullsized":"2024-09-20T22:43:08.108629+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.8","version":"106'13","reported_seq":93,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-20T22:43:08.113403+0000","last_change":"2024-09-20T22:43:07.542068+0000","last_active":"2024-09-20T22:43:08.113403+0000","last_peered":"2024-09-20T22:43:08.113403+0000","last_clean":"2024-09-20T22:43:08.113403+0000","last_became_active":"2024-09-20T22:42:52.409070+0000","last_became_peered":"2024-09-20T22:42:52.409070+0000","last_unstale":"2024-09-20T22:43:08.113403+0000","last_undegraded":"2024-09-20T22:43:08.113403+0000","last_fullsized":"2024-09-20T22:43:08.113403+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:42:51.354519+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:04.291620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00068422100000000005,"stat_sum":{"num_bytes":7697507,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":16,"num_read_kb":11281,"num_write":17,"num_write_kb":4558,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"3","length":"4"}]},{"pgid":"4.b","version":"74'9","reported_seq":115,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-20T22:43:08.109218+0000","last_change":"2024-09-20T22:43:00.749532+0000","last_active":"2024-09-20T22:43:08.109218+0000","last_peered":"2024-09-20T22:43:08.109218+0000","last_clean":"2024-09-20T22:43:08.109218+0000","last_became_active":"2024-09-20T22:41:40.163495+0000","last_became_peered":"2024-09-20T22:41:40.163495+0000","last_unstale":"2024-09-20T22:43:08.109218+0000","last_undegraded":"2024-09-20T22:43:08.109218+0000","last_fullsized":"2024-09-20T22:43:08.109218+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-09-20T22:43:00.749456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:43:00.749456+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:15.372038+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":95,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-20T22:43:08.112437+0000","last_change":"2024-09-20T22:41:38.156470+0000","last_active":"2024-09-20T22:43:08.112437+0000","last_peered":"2024-09-20T22:43:08.112437+0000","last_clean":"2024-09-20T22:43:08.112437+0000","last_became_active":"2024-09-20T22:41:38.155953+0000","last_became_peered":"2024-09-20T22:41:38.155953+0000","last_unstale":"2024-09-20T22:43:08.112437+0000","last_undegraded":"2024-09-20T22:43:08.112437+0000","last_fullsized":"2024-09-20T22:43:08.112437+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":87,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-20T22:43:08.115767+0000","last_change":"2024-09-20T22:41:42.183269+0000","last_active":"2024-09-20T22:43:08.115767+0000","last_peered":"2024-09-20T22:43:08.115767+0000","last_clean":"2024-09-20T22:43:08.115767+0000","last_became_active":"2024-09-20T22:41:42.182999+0000","last_became_peered":"2024-09-20T22:41:42.182999+0000","last_unstale":"2024-09-20T22:43:08.115767+0000","last_undegraded":"2024-09-20T22:43:08.115767+0000","last_fullsized":"2024-09-20T22:43:08.115767+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":83,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-20T22:43:08.110352+0000","last_change":"2024-09-20T22:41:44.196969+0000","last_active":"2024-09-20T22:43:08.110352+0000","last_peered":"2024-09-20T22:43:08.110352+0000","last_clean":"2024-09-20T22:43:08.110352+0000","last_became_active":"2024-09-20T22:41:44.196633+0000","last_became_peered":"2024-09-20T22:41:44.196633+0000","last_unstale":"2024-09-20T22:43:08.110352+0000","last_undegraded":"2024-09-20T22:43:08.110352+0000","last_fullsized":"2024-09-20T22:43:08.110352+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"108'28","reported_seq":105,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-20T22:43:08.110127+0000","last_change":"2024-09-20T22:43:07.543967+0000","last_active":"2024-09-20T22:43:08.110127+0000","last_peered":"2024-09-20T22:43:08.110127+0000","last_clean":"2024-09-20T22:43:08.110127+0000","last_became_active":"2024-09-20T22:42:52.407089+0000","last_became_peered":"2024-09-20T22:42:52.407089+0000","last_unstale":"2024-09-20T22:43:08.110127+0000","last_undegraded":"2024-09-20T22:43:08.110127+0000","last_fullsized":"2024-09-20T22:43:08.110127+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:42:51.354519+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:13.166761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059916699999999995,"stat_sum":{"num_bytes":10672313,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":12,"num_read_kb":6,"num_write":26,"num_write_kb":9183,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"3","length":"4"}]},{"pgid":"4.a","version":"74'19","reported_seq":120,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-20T22:43:07.548903+0000","last_change":"2024-09-20T22:41:40.252962+0000","last_active":"2024-09-20T22:43:07.548903+0000","last_peered":"2024-09-20T22:43:07.548903+0000","last_clean":"2024-09-20T22:43:07.548903+0000","last_became_active":"2024-09-20T22:41:40.252011+0000","last_became_peered":"2024-09-20T22:41:40.252011+0000","last_unstale":"2024-09-20T22:43:07.548903+0000","last_undegraded":"2024-09-20T22:43:07.548903+0000","last_fullsized":"2024-09-20T22:43:07.548903+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":95,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-20T22:43:08.117766+0000","last_change":"2024-09-20T22:41:38.161800+0000","last_active":"2024-09-20T22:43:08.117766+0000","last_peered":"2024-09-20T22:43:08.117766+0000","last_clean":"2024-09-20T22:43:08.117766+0000","last_became_active":"2024-09-20T22:41:38.161524+0000","last_became_peered":"2024-09-20T22:41:38.161524+0000","last_unstale":"2024-09-20T22:43:08.117766+0000","last_undegraded":"2024-09-20T22:43:08.117766+0000","last_fullsized":"2024-09-20T22:43:08.117766+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":87,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-20T22:43:08.113984+0000","last_change":"2024-09-20T22:41:42.184605+0000","last_active":"2024-09-20T22:43:08.113984+0000","last_peered":"2024-09-20T22:43:08.113984+0000","last_clean":"2024-09-20T22:43:08.113984+0000","last_became_active":"2024-09-20T22:41:42.184342+0000","last_became_peered":"2024-09-20T22:41:42.184342+0000","last_unstale":"2024-09-20T22:43:08.113984+0000","last_undegraded":"2024-09-20T22:43:08.113984+0000","last_fullsized":"2024-09-20T22:43:08.113984+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":91,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-20T22:43:08.117380+0000","last_change":"2024-09-20T22:43:02.234978+0000","last_active":"2024-09-20T22:43:08.117380+0000","last_peered":"2024-09-20T22:43:08.117380+0000","last_clean":"2024-09-20T22:43:08.117380+0000","last_became_active":"2024-09-20T22:41:44.199395+0000","last_became_peered":"2024-09-20T22:41:44.199395+0000","last_unstale":"2024-09-20T22:43:08.117380+0000","last_undegraded":"2024-09-20T22:43:08.117380+0000","last_fullsized":"2024-09-20T22:43:08.117380+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:02.234872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:43:02.234872+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:25.214149+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.a","version":"103'17","reported_seq":98,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-20T22:43:08.111129+0000","last_change":"2024-09-20T22:43:07.550530+0000","last_active":"2024-09-20T22:43:08.111129+0000","last_peered":"2024-09-20T22:43:08.111129+0000","last_clean":"2024-09-20T22:43:08.111129+0000","last_became_active":"2024-09-20T22:42:52.405085+0000","last_became_peered":"2024-09-20T22:42:52.405085+0000","last_unstale":"2024-09-20T22:43:08.111129+0000","last_undegraded":"2024-09-20T22:43:08.111129+0000","last_fullsized":"2024-09-20T22:43:08.111129+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:42:51.354519+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:52.411911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00090711299999999997,"stat_sum":{"num_bytes":7915998,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":25,"num_read_kb":22863,"num_write":16,"num_write_kb":4401,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"3","length":"4"}]},{"pgid":"4.9","version":"74'12","reported_seq":122,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-20T22:43:08.108359+0000","last_change":"2024-09-20T22:42:59.862739+0000","last_active":"2024-09-20T22:43:08.108359+0000","last_peered":"2024-09-20T22:43:08.108359+0000","last_clean":"2024-09-20T22:43:08.108359+0000","last_became_active":"2024-09-20T22:41:40.170133+0000","last_became_peered":"2024-09-20T22:41:40.170133+0000","last_unstale":"2024-09-20T22:43:08.108359+0000","last_undegraded":"2024-09-20T22:43:08.108359+0000","last_fullsized":"2024-09-20T22:43:08.108359+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-09-20T22:42:59.862632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:42:59.862632+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:23.948333+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":103,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-20T22:43:08.117355+0000","last_change":"2024-09-20T22:43:05.229655+0000","last_active":"2024-09-20T22:43:08.117355+0000","last_peered":"2024-09-20T22:43:08.117355+0000","last_clean":"2024-09-20T22:43:08.117355+0000","last_became_active":"2024-09-20T22:41:38.163292+0000","last_became_peered":"2024-09-20T22:41:38.163292+0000","last_unstale":"2024-09-20T22:43:08.117355+0000","last_undegraded":"2024-09-20T22:43:08.117355+0000","last_fullsized":"2024-09-20T22:43:08.117355+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:05.229558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:43:05.229558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:22.822578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":95,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-20T22:43:08.113727+0000","last_change":"2024-09-20T22:43:02.326233+0000","last_active":"2024-09-20T22:43:08.113727+0000","last_peered":"2024-09-20T22:43:08.113727+0000","last_clean":"2024-09-20T22:43:08.113727+0000","last_became_active":"2024-09-20T22:41:42.181811+0000","last_became_peered":"2024-09-20T22:41:42.181811+0000","last_unstale":"2024-09-20T22:43:08.113727+0000","last_undegraded":"2024-09-20T22:43:08.113727+0000","last_fullsized":"2024-09-20T22:43:08.113727+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:02.326135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:43:02.326135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:18.212222+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":83,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-20T22:43:08.111842+0000","last_change":"2024-09-20T22:41:44.193432+0000","last_active":"2024-09-20T22:43:08.111842+0000","last_peered":"2024-09-20T22:43:08.111842+0000","last_clean":"2024-09-20T22:43:08.111842+0000","last_became_active":"2024-09-20T22:41:44.192609+0000","last_became_peered":"2024-09-20T22:41:44.192609+0000","last_unstale":"2024-09-20T22:43:08.111842+0000","last_undegraded":"2024-09-20T22:43:08.111842+0000","last_fullsized":"2024-09-20T22:43:08.111842+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"106'19","reported_seq":103,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-20T22:43:08.120185+0000","last_change":"2024-09-20T22:43:07.783793+0000","last_active":"2024-09-20T22:43:08.120185+0000","last_peered":"2024-09-20T22:43:08.120185+0000","last_clean":"2024-09-20T22:43:08.120185+0000","last_became_active":"2024-09-20T22:42:52.409642+0000","last_became_peered":"2024-09-20T22:42:52.409642+0000","last_unstale":"2024-09-20T22:43:08.120185+0000","last_undegraded":"2024-09-20T22:43:08.120185+0000","last_fullsized":"2024-09-20T22:43:08.120185+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:42:51.354519+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:15.378718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010075139999999999,"stat_sum":{"num_bytes":11344560,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":27,"num_read_kb":14861,"num_write":21,"num_write_kb":7300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"3","length":"4"}]},{"pgid":"4.8","version":"74'15","reported_seq":116,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-20T22:43:08.111899+0000","last_change":"2024-09-20T22:41:40.252427+0000","last_active":"2024-09-20T22:43:08.111899+0000","last_peered":"2024-09-20T22:43:08.111899+0000","last_clean":"2024-09-20T22:43:08.111899+0000","last_became_active":"2024-09-20T22:41:40.252113+0000","last_became_peered":"2024-09-20T22:41:40.252113+0000","last_unstale":"2024-09-20T22:43:08.111899+0000","last_undegraded":"2024-09-20T22:43:08.111899+0000","last_fullsized":"2024-09-20T22:43:08.111899+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'2","reported_seq":107,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-20T22:43:08.117992+0000","last_change":"2024-09-20T22:41:38.163593+0000","last_active":"2024-09-20T22:43:08.117992+0000","last_peered":"2024-09-20T22:43:08.117992+0000","last_clean":"2024-09-20T22:43:08.117992+0000","last_became_active":"2024-09-20T22:41:38.163219+0000","last_became_peered":"2024-09-20T22:41:38.163219+0000","last_unstale":"2024-09-20T22:43:08.117992+0000","last_undegraded":"2024-09-20T22:43:08.117992+0000","last_fullsized":"2024-09-20T22:43:08.117992+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"74'8","reported_seq":114,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-20T22:43:08.116814+0000","last_change":"2024-09-20T22:41:42.188223+0000","last_active":"2024-09-20T22:43:08.116814+0000","last_peered":"2024-09-20T22:43:08.116814+0000","last_clean":"2024-09-20T22:43:08.116814+0000","last_became_active":"2024-09-20T22:41:42.186635+0000","last_became_peered":"2024-09-20T22:41:42.186635+0000","last_unstale":"2024-09-20T22:43:08.116814+0000","last_undegraded":"2024-09-20T22:43:08.116814+0000","last_fullsized":"2024-09-20T22:43:08.116814+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":83,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-20T22:43:08.111937+0000","last_change":"2024-09-20T22:41:44.831435+0000","last_active":"2024-09-20T22:43:08.111937+0000","last_peered":"2024-09-20T22:43:08.111937+0000","last_clean":"2024-09-20T22:43:08.111937+0000","last_became_active":"2024-09-20T22:41:44.830556+0000","last_became_peered":"2024-09-20T22:41:44.830556+0000","last_unstale":"2024-09-20T22:43:08.111937+0000","last_undegraded":"2024-09-20T22:43:08.111937+0000","last_fullsized":"2024-09-20T22:43:08.111937+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":101,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-20T22:43:07.548767+0000","last_change":"2024-09-20T22:43:01.806383+0000","last_active":"2024-09-20T22:43:07.548767+0000","last_peered":"2024-09-20T22:43:07.548767+0000","last_clean":"2024-09-20T22:43:07.548767+0000","last_became_active":"2024-09-20T22:41:38.159139+0000","last_became_peered":"2024-09-20T22:41:38.159139+0000","last_unstale":"2024-09-20T22:43:07.548767+0000","last_undegraded":"2024-09-20T22:43:07.548767+0000","last_fullsized":"2024-09-20T22:43:07.548767+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:01.806314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:43:01.806314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:16.773475+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"74'6","reported_seq":107,"reported_epoch":108,"state":"peering","last_fresh":"2024-09-20T22:43:07.550763+0000","last_change":"2024-09-20T22:43:08.115373+0000","last_active":"2024-09-20T22:43:08.115373+0000","last_peered":"2024-09-20T22:43:07.550763+0000","last_clean":"2024-09-20T22:43:07.550763+0000","last_became_active":"2024-09-20T22:41:40.251867+0000","last_became_peered":"2024-09-20T22:41:40.251867+0000","last_unstale":"2024-09-20T22:43:07.550763+0000","last_undegraded":"2024-09-20T22:43:07.550763+0000","last_fullsized":"2024-09-20T22:43:07.550763+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'6","last_scrub_stamp":"2024-09-20T22:42:41.469214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:42:41.469214+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,2,7],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":87,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-20T22:43:08.112273+0000","last_change":"2024-09-20T22:41:42.185218+0000","last_active":"2024-09-20T22:43:08.112273+0000","last_peered":"2024-09-20T22:43:08.112273+0000","last_clean":"2024-09-20T22:43:08.112273+0000","last_became_active":"2024-09-20T22:41:42.184930+0000","last_became_peered":"2024-09-20T22:41:42.184930+0000","last_unstale":"2024-09-20T22:43:08.112273+0000","last_undegraded":"2024-09-20T22:43:08.112273+0000","last_fullsized":"2024-09-20T22:43:08.112273+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"74'1","reported_seq":84,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-20T22:43:08.117139+0000","last_change":"2024-09-20T22:41:44.832085+0000","last_active":"2024-09-20T22:43:08.117139+0000","last_peered":"2024-09-20T22:43:08.117139+0000","last_clean":"2024-09-20T22:43:08.117139+0000","last_became_active":"2024-09-20T22:41:44.831805+0000","last_became_peered":"2024-09-20T22:41:44.831805+0000","last_unstale":"2024-09-20T22:43:08.117139+0000","last_undegraded":"2024-09-20T22:43:08.117139+0000","last_fullsized":"2024-09-20T22:43:08.117139+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"74'9","reported_seq":115,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-20T22:43:08.110252+0000","last_change":"2024-09-20T22:42:43.666853+0000","last_active":"2024-09-20T22:43:08.110252+0000","last_peered":"2024-09-20T22:43:08.110252+0000","last_clean":"2024-09-20T22:43:08.110252+0000","last_became_active":"2024-09-20T22:41:40.253280+0000","last_became_peered":"2024-09-20T22:41:40.253280+0000","last_unstale":"2024-09-20T22:43:08.110252+0000","last_undegraded":"2024-09-20T22:43:08.110252+0000","last_fullsized":"2024-09-20T22:43:08.110252+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-09-20T22:42:43.666774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:42:43.666774+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:12.132272+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":95,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-20T22:43:08.117214+0000","last_change":"2024-09-20T22:41:38.156773+0000","last_active":"2024-09-20T22:43:08.117214+0000","last_peered":"2024-09-20T22:43:08.117214+0000","last_clean":"2024-09-20T22:43:08.117214+0000","last_became_active":"2024-09-20T22:41:38.156497+0000","last_became_peered":"2024-09-20T22:41:38.156497+0000","last_unstale":"2024-09-20T22:43:08.117214+0000","last_undegraded":"2024-09-20T22:43:08.117214+0000","last_fullsized":"2024-09-20T22:43:08.117214+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":87,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-20T22:43:08.114711+0000","last_change":"2024-09-20T22:41:42.188166+0000","last_active":"2024-09-20T22:43:08.114711+0000","last_peered":"2024-09-20T22:43:08.114711+0000","last_clean":"2024-09-20T22:43:08.114711+0000","last_became_active":"2024-09-20T22:41:42.187706+0000","last_became_peered":"2024-09-20T22:41:42.187706+0000","last_unstale":"2024-09-20T22:43:08.114711+0000","last_undegraded":"2024-09-20T22:43:08.114711+0000","last_fullsized":"2024-09-20T22:43:08.114711+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":91,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-20T22:43:08.232340+0000","last_change":"2024-09-20T22:43:08.232359+0000","last_active":"2024-09-20T22:43:08.232340+0000","last_peered":"2024-09-20T22:43:08.232340+0000","last_clean":"2024-09-20T22:43:08.232340+0000","last_became_active":"2024-09-20T22:41:44.201433+0000","last_became_peered":"2024-09-20T22:41:44.201433+0000","last_unstale":"2024-09-20T22:43:08.232340+0000","last_undegraded":"2024-09-20T22:43:08.232340+0000","last_fullsized":"2024-09-20T22:43:08.232340+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:08.232288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:43:08.232288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:08.925242+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"74'9","reported_seq":107,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-20T22:43:08.112350+0000","last_change":"2024-09-20T22:41:40.252840+0000","last_active":"2024-09-20T22:43:08.112350+0000","last_peered":"2024-09-20T22:43:08.112350+0000","last_clean":"2024-09-20T22:43:08.112350+0000","last_became_active":"2024-09-20T22:41:40.252486+0000","last_became_peered":"2024-09-20T22:41:40.252486+0000","last_unstale":"2024-09-20T22:43:08.112350+0000","last_undegraded":"2024-09-20T22:43:08.112350+0000","last_fullsized":"2024-09-20T22:43:08.112350+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":95,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-20T22:43:08.109874+0000","last_change":"2024-09-20T22:41:38.158479+0000","last_active":"2024-09-20T22:43:08.109874+0000","last_peered":"2024-09-20T22:43:08.109874+0000","last_clean":"2024-09-20T22:43:08.109874+0000","last_became_active":"2024-09-20T22:41:38.158146+0000","last_became_peered":"2024-09-20T22:41:38.158146+0000","last_unstale":"2024-09-20T22:43:08.109874+0000","last_undegraded":"2024-09-20T22:43:08.109874+0000","last_fullsized":"2024-09-20T22:43:08.109874+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"74'8","reported_seq":114,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-20T22:43:08.110220+0000","last_change":"2024-09-20T22:41:42.186630+0000","last_active":"2024-09-20T22:43:08.110220+0000","last_peered":"2024-09-20T22:43:08.110220+0000","last_clean":"2024-09-20T22:43:08.110220+0000","last_became_active":"2024-09-20T22:41:42.186352+0000","last_became_peered":"2024-09-20T22:41:42.186352+0000","last_unstale":"2024-09-20T22:43:08.110220+0000","last_undegraded":"2024-09-20T22:43:08.110220+0000","last_fullsized":"2024-09-20T22:43:08.110220+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":83,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-20T22:43:08.108732+0000","last_change":"2024-09-20T22:41:44.194510+0000","last_active":"2024-09-20T22:43:08.108732+0000","last_peered":"2024-09-20T22:43:08.108732+0000","last_clean":"2024-09-20T22:43:08.108732+0000","last_became_active":"2024-09-20T22:41:44.194053+0000","last_became_peered":"2024-09-20T22:41:44.194053+0000","last_unstale":"2024-09-20T22:43:08.108732+0000","last_undegraded":"2024-09-20T22:43:08.108732+0000","last_fullsized":"2024-09-20T22:43:08.108732+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"74'10","reported_seq":106,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-20T22:43:08.110535+0000","last_change":"2024-09-20T22:41:40.253880+0000","last_active":"2024-09-20T22:43:08.110535+0000","last_peered":"2024-09-20T22:43:08.110535+0000","last_clean":"2024-09-20T22:43:08.110535+0000","last_became_active":"2024-09-20T22:41:40.253614+0000","last_became_peered":"2024-09-20T22:41:40.253614+0000","last_unstale":"2024-09-20T22:43:08.110535+0000","last_undegraded":"2024-09-20T22:43:08.110535+0000","last_fullsized":"2024-09-20T22:43:08.110535+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":95,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-20T22:43:08.117244+0000","last_change":"2024-09-20T22:41:38.158008+0000","last_active":"2024-09-20T22:43:08.117244+0000","last_peered":"2024-09-20T22:43:08.117244+0000","last_clean":"2024-09-20T22:43:08.117244+0000","last_became_active":"2024-09-20T22:41:38.157647+0000","last_became_peered":"2024-09-20T22:41:38.157647+0000","last_unstale":"2024-09-20T22:43:08.117244+0000","last_undegraded":"2024-09-20T22:43:08.117244+0000","last_fullsized":"2024-09-20T22:43:08.117244+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"74'8","reported_seq":125,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-20T22:43:08.112214+0000","last_change":"2024-09-20T22:42:59.718431+0000","last_active":"2024-09-20T22:43:08.112214+0000","last_peered":"2024-09-20T22:43:08.112214+0000","last_clean":"2024-09-20T22:43:08.112214+0000","last_became_active":"2024-09-20T22:41:42.189451+0000","last_became_peered":"2024-09-20T22:41:42.189451+0000","last_unstale":"2024-09-20T22:43:08.112214+0000","last_undegraded":"2024-09-20T22:43:08.112214+0000","last_fullsized":"2024-09-20T22:43:08.112214+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-09-20T22:42:59.718354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:42:59.718354+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:07.105181+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":91,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-20T22:43:08.110784+0000","last_change":"2024-09-20T22:42:58.680721+0000","last_active":"2024-09-20T22:43:08.110784+0000","last_peered":"2024-09-20T22:43:08.110784+0000","last_clean":"2024-09-20T22:43:08.110784+0000","last_became_active":"2024-09-20T22:41:44.191206+0000","last_became_peered":"2024-09-20T22:41:44.191206+0000","last_unstale":"2024-09-20T22:43:08.110784+0000","last_undegraded":"2024-09-20T22:43:08.110784+0000","last_fullsized":"2024-09-20T22:43:08.110784+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:42:58.680648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:42:58.680648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:22.824715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"74'11","reported_seq":110,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-20T22:43:08.108449+0000","last_change":"2024-09-20T22:41:40.171449+0000","last_active":"2024-09-20T22:43:08.108449+0000","last_peered":"2024-09-20T22:43:08.108449+0000","last_clean":"2024-09-20T22:43:08.108449+0000","last_became_active":"2024-09-20T22:41:40.170035+0000","last_became_peered":"2024-09-20T22:41:40.170035+0000","last_unstale":"2024-09-20T22:43:08.108449+0000","last_undegraded":"2024-09-20T22:43:08.108449+0000","last_fullsized":"2024-09-20T22:43:08.108449+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":95,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-20T22:43:08.108519+0000","last_change":"2024-09-20T22:41:38.161385+0000","last_active":"2024-09-20T22:43:08.108519+0000","last_peered":"2024-09-20T22:43:08.108519+0000","last_clean":"2024-09-20T22:43:08.108519+0000","last_became_active":"2024-09-20T22:41:38.158142+0000","last_became_peered":"2024-09-20T22:41:38.158142+0000","last_unstale":"2024-09-20T22:43:08.108519+0000","last_undegraded":"2024-09-20T22:43:08.108519+0000","last_fullsized":"2024-09-20T22:43:08.108519+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":87,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-20T22:43:08.118614+0000","last_change":"2024-09-20T22:41:42.182119+0000","last_active":"2024-09-20T22:43:08.118614+0000","last_peered":"2024-09-20T22:43:08.118614+0000","last_clean":"2024-09-20T22:43:08.118614+0000","last_became_active":"2024-09-20T22:41:42.181736+0000","last_became_peered":"2024-09-20T22:41:42.181736+0000","last_unstale":"2024-09-20T22:43:08.118614+0000","last_undegraded":"2024-09-20T22:43:08.118614+0000","last_fullsized":"2024-09-20T22:43:08.118614+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":83,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-20T22:43:08.108777+0000","last_change":"2024-09-20T22:41:44.193289+0000","last_active":"2024-09-20T22:43:08.108777+0000","last_peered":"2024-09-20T22:43:08.108777+0000","last_clean":"2024-09-20T22:43:08.108777+0000","last_became_active":"2024-09-20T22:41:44.192960+0000","last_became_peered":"2024-09-20T22:41:44.192960+0000","last_unstale":"2024-09-20T22:43:08.108777+0000","last_undegraded":"2024-09-20T22:43:08.108777+0000","last_fullsized":"2024-09-20T22:43:08.108777+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"74'9","reported_seq":107,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-20T22:43:08.118885+0000","last_change":"2024-09-20T22:41:40.165116+0000","last_active":"2024-09-20T22:43:08.118885+0000","last_peered":"2024-09-20T22:43:08.118885+0000","last_clean":"2024-09-20T22:43:08.118885+0000","last_became_active":"2024-09-20T22:41:40.164796+0000","last_became_peered":"2024-09-20T22:41:40.164796+0000","last_unstale":"2024-09-20T22:43:08.118885+0000","last_undegraded":"2024-09-20T22:43:08.118885+0000","last_fullsized":"2024-09-20T22:43:08.118885+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":95,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-20T22:43:08.117625+0000","last_change":"2024-09-20T22:41:38.159941+0000","last_active":"2024-09-20T22:43:08.117625+0000","last_peered":"2024-09-20T22:43:08.117625+0000","last_clean":"2024-09-20T22:43:08.117625+0000","last_became_active":"2024-09-20T22:41:38.159657+0000","last_became_peered":"2024-09-20T22:41:38.159657+0000","last_unstale":"2024-09-20T22:43:08.117625+0000","last_undegraded":"2024-09-20T22:43:08.117625+0000","last_fullsized":"2024-09-20T22:43:08.117625+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":87,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-20T22:43:08.115862+0000","last_change":"2024-09-20T22:41:42.183254+0000","last_active":"2024-09-20T22:43:08.115862+0000","last_peered":"2024-09-20T22:43:08.115862+0000","last_clean":"2024-09-20T22:43:08.115862+0000","last_became_active":"2024-09-20T22:41:42.182966+0000","last_became_peered":"2024-09-20T22:41:42.182966+0000","last_unstale":"2024-09-20T22:43:08.115862+0000","last_undegraded":"2024-09-20T22:43:08.115862+0000","last_fullsized":"2024-09-20T22:43:08.115862+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":91,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-20T22:43:08.107657+0000","last_change":"2024-09-20T22:43:04.649683+0000","last_active":"2024-09-20T22:43:08.107657+0000","last_peered":"2024-09-20T22:43:08.107657+0000","last_clean":"2024-09-20T22:43:08.107657+0000","last_became_active":"2024-09-20T22:41:44.194551+0000","last_became_peered":"2024-09-20T22:41:44.194551+0000","last_unstale":"2024-09-20T22:43:08.107657+0000","last_undegraded":"2024-09-20T22:43:08.107657+0000","last_fullsized":"2024-09-20T22:43:08.107657+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:04.649566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:43:04.649566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:09.553594+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"74'11","reported_seq":110,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-20T22:43:08.111053+0000","last_change":"2024-09-20T22:41:40.252149+0000","last_active":"2024-09-20T22:43:08.111053+0000","last_peered":"2024-09-20T22:43:08.111053+0000","last_clean":"2024-09-20T22:43:08.111053+0000","last_became_active":"2024-09-20T22:41:40.251829+0000","last_became_peered":"2024-09-20T22:41:40.251829+0000","last_unstale":"2024-09-20T22:43:08.111053+0000","last_undegraded":"2024-09-20T22:43:08.111053+0000","last_fullsized":"2024-09-20T22:43:08.111053+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":103,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-20T22:43:08.112370+0000","last_change":"2024-09-20T22:43:06.775869+0000","last_active":"2024-09-20T22:43:08.112370+0000","last_peered":"2024-09-20T22:43:08.112370+0000","last_clean":"2024-09-20T22:43:08.112370+0000","last_became_active":"2024-09-20T22:41:38.158701+0000","last_became_peered":"2024-09-20T22:41:38.158701+0000","last_unstale":"2024-09-20T22:43:08.112370+0000","last_undegraded":"2024-09-20T22:43:08.112370+0000","last_fullsized":"2024-09-20T22:43:08.112370+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:06.775781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:06.775781+0000","last_clean_scrub_stamp":"2024-09-20T22:43:06.775781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:09.259667+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":87,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-20T22:43:08.116630+0000","last_change":"2024-09-20T22:41:42.185991+0000","last_active":"2024-09-20T22:43:08.116630+0000","last_peered":"2024-09-20T22:43:08.116630+0000","last_clean":"2024-09-20T22:43:08.116630+0000","last_became_active":"2024-09-20T22:41:42.185296+0000","last_became_peered":"2024-09-20T22:41:42.185296+0000","last_unstale":"2024-09-20T22:43:08.116630+0000","last_undegraded":"2024-09-20T22:43:08.116630+0000","last_fullsized":"2024-09-20T22:43:08.116630+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":83,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-20T22:43:08.115947+0000","last_change":"2024-09-20T22:41:44.832293+0000","last_active":"2024-09-20T22:43:08.115947+0000","last_peered":"2024-09-20T22:43:08.115947+0000","last_clean":"2024-09-20T22:43:08.115947+0000","last_became_active":"2024-09-20T22:41:44.831967+0000","last_became_peered":"2024-09-20T22:41:44.831967+0000","last_unstale":"2024-09-20T22:43:08.115947+0000","last_undegraded":"2024-09-20T22:43:08.115947+0000","last_fullsized":"2024-09-20T22:43:08.115947+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"74'4","reported_seq":97,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-20T22:43:08.108335+0000","last_change":"2024-09-20T22:41:40.166409+0000","last_active":"2024-09-20T22:43:08.108335+0000","last_peered":"2024-09-20T22:43:08.108335+0000","last_clean":"2024-09-20T22:43:08.108335+0000","last_became_active":"2024-09-20T22:41:40.166123+0000","last_became_peered":"2024-09-20T22:41:40.166123+0000","last_unstale":"2024-09-20T22:43:08.108335+0000","last_undegraded":"2024-09-20T22:43:08.108335+0000","last_fullsized":"2024-09-20T22:43:08.108335+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":103,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-20T22:43:08.110815+0000","last_change":"2024-09-20T22:43:03.680628+0000","last_active":"2024-09-20T22:43:08.110815+0000","last_peered":"2024-09-20T22:43:08.110815+0000","last_clean":"2024-09-20T22:43:08.110815+0000","last_became_active":"2024-09-20T22:41:38.158347+0000","last_became_peered":"2024-09-20T22:41:38.158347+0000","last_unstale":"2024-09-20T22:43:08.110815+0000","last_undegraded":"2024-09-20T22:43:08.110815+0000","last_fullsized":"2024-09-20T22:43:08.110815+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:03.680528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:43:03.680528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:18.111001+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":93,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-20T22:43:07.548813+0000","last_change":"2024-09-20T22:43:06.821377+0000","last_active":"2024-09-20T22:43:07.548813+0000","last_peered":"2024-09-20T22:43:07.548813+0000","last_clean":"2024-09-20T22:43:07.548813+0000","last_became_active":"2024-09-20T22:41:42.186068+0000","last_became_peered":"2024-09-20T22:41:42.186068+0000","last_unstale":"2024-09-20T22:43:07.548813+0000","last_undegraded":"2024-09-20T22:43:07.548813+0000","last_fullsized":"2024-09-20T22:43:07.548813+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:06.821310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:43:06.821310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:29.282062+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":83,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-20T22:43:08.116788+0000","last_change":"2024-09-20T22:41:44.194011+0000","last_active":"2024-09-20T22:43:08.116788+0000","last_peered":"2024-09-20T22:43:08.116788+0000","last_clean":"2024-09-20T22:43:08.116788+0000","last_became_active":"2024-09-20T22:41:44.193706+0000","last_became_peered":"2024-09-20T22:41:44.193706+0000","last_unstale":"2024-09-20T22:43:08.116788+0000","last_undegraded":"2024-09-20T22:43:08.116788+0000","last_fullsized":"2024-09-20T22:43:08.116788+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":83,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-20T22:43:08.118493+0000","last_change":"2024-09-20T22:41:44.198154+0000","last_active":"2024-09-20T22:43:08.118493+0000","last_peered":"2024-09-20T22:43:08.118493+0000","last_clean":"2024-09-20T22:43:08.118493+0000","last_became_active":"2024-09-20T22:41:44.197855+0000","last_became_peered":"2024-09-20T22:41:44.197855+0000","last_unstale":"2024-09-20T22:43:08.118493+0000","last_undegraded":"2024-09-20T22:43:08.118493+0000","last_fullsized":"2024-09-20T22:43:08.118493+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":103,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-20T22:43:08.108548+0000","last_change":"2024-09-20T22:43:00.430541+0000","last_active":"2024-09-20T22:43:08.108548+0000","last_peered":"2024-09-20T22:43:08.108548+0000","last_clean":"2024-09-20T22:43:08.108548+0000","last_became_active":"2024-09-20T22:41:38.154703+0000","last_became_peered":"2024-09-20T22:41:38.154703+0000","last_unstale":"2024-09-20T22:43:08.108548+0000","last_undegraded":"2024-09-20T22:43:08.108548+0000","last_fullsized":"2024-09-20T22:43:08.108548+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:00.430469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:43:00.430469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:08.693135+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"74'11","reported_seq":108,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-09-20T22:43:07.549516+0000","last_change":"2024-09-20T22:41:40.171840+0000","last_active":"2024-09-20T22:43:07.549516+0000","last_peered":"2024-09-20T22:43:07.549516+0000","last_clean":"2024-09-20T22:43:07.549516+0000","last_became_active":"2024-09-20T22:41:40.171256+0000","last_became_peered":"2024-09-20T22:41:40.171256+0000","last_unstale":"2024-09-20T22:43:07.549516+0000","last_undegraded":"2024-09-20T22:43:07.549516+0000","last_fullsized":"2024-09-20T22:43:07.549516+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":87,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-09-20T22:43:08.109328+0000","last_change":"2024-09-20T22:41:42.178413+0000","last_active":"2024-09-20T22:43:08.109328+0000","last_peered":"2024-09-20T22:43:08.109328+0000","last_clean":"2024-09-20T22:43:08.109328+0000","last_became_active":"2024-09-20T22:41:42.178157+0000","last_became_peered":"2024-09-20T22:41:42.178157+0000","last_unstale":"2024-09-20T22:43:08.109328+0000","last_undegraded":"2024-09-20T22:43:08.109328+0000","last_fullsized":"2024-09-20T22:43:08.109328+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":122756744,"num_objects":48,"num_object_clones":0,"num_object_copies":144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":298,"num_read_kb":175330,"num_write":277,"num_write_kb":100325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":224329728,"data_stored":223196847,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":275,"ondisk_log_size":275,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":403,"ondisk_log_size":403,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":85,"num_read_kb":80,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1408,"ondisk_log_size":1408,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070486,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":55352,"kb_used_data":23800,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93689800,"statfs":{"total":95995035648,"available":95938355200,"internally_reserved":0,"allocated":24371200,"data_stored":23670582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430236,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":66528,"kb_used_data":35340,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93678624,"statfs":{"total":95995035648,"available":95926910976,"internally_reserved":0,"allocated":36188160,"data_stored":35459485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691873,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":44960,"kb_used_data":17928,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93700192,"statfs":{"total":95995035648,"available":95948996608,"internally_reserved":0,"allocated":18358272,"data_stored":17712780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986214,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":47388,"kb_used_data":20356,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93697764,"statfs":{"total":95995035648,"available":95946510336,"internally_reserved":0,"allocated":20844544,"data_stored":20194791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182444,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":77272,"kb_used_data":48940,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93667880,"statfs":{"total":95995035648,"available":95915909120,"internally_reserved":0,"allocated":50114560,"data_stored":49306392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411377,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":48656,"kb_used_data":17460,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93696496,"statfs":{"total":95995035648,"available":95945211904,"internally_reserved":0,"allocated":17879040,"data_stored":17243230,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574902,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":61340,"kb_used_data":30152,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93683812,"statfs":{"total":95995035648,"available":95932223488,"internally_reserved":0,"allocated":30875648,"data_stored":30147500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738427,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":63112,"kb_used_data":31924,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93682040,"statfs":{"total":95995035648,"available":95930408960,"internally_reserved":0,"allocated":32690176,"data_stored":31961825,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":34820096,"data_stored":34670775,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":29794304,"data_stored":29659027,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16584704,"data_stored":16485935,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":48451584,"data_stored":48220117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":19795968,"data_stored":19707485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16736256,"data_stored":16633936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":35155968,"data_stored":34971791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":22990848,"data_stored":22847781,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-20T22:43:11.884 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-09-20T22:43:11.884 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd pool set .mgr pg_num 21 2024-09-20T22:43:11.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:11.897 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:11.897 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:43:11.998 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:43:12.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:11 smithi154 ceph-mon[33264]: pgmap v67: 158 pgs: 3 peering, 155 active+clean; 118 MiB data, 454 MiB used, 715 GiB / 715 GiB avail; 32 MiB/s rd, 14 op/s 2024-09-20T22:43:12.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:11 smithi154 ceph-mon[33264]: 5.16 deep-scrub starts 2024-09-20T22:43:12.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:11 smithi154 ceph-mon[33264]: 5.16 deep-scrub ok 2024-09-20T22:43:12.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:11 smithi154 ceph-mon[33264]: from='client.14865 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T22:43:12.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:11 smithi154 ceph-mon[33264]: osdmap e112: 8 total, 8 up, 8 in 2024-09-20T22:43:12.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:11 smithi154 ceph-mon[33264]: 5.9 scrub starts 2024-09-20T22:43:12.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:11 smithi154 ceph-mon[33264]: 5.9 scrub ok 2024-09-20T22:43:12.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:11 smithi154 ceph-mon[33264]: 4.14 scrub starts 2024-09-20T22:43:12.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:11 smithi154 ceph-mon[33264]: 4.14 scrub ok 2024-09-20T22:43:12.019 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:12.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:12.020 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:43:12.118 INFO:tasks.rados.rados.0.smithi084.stdout:128: done (4 left) 2024-09-20T22:43:12.118 INFO:tasks.rados.rados.0.smithi084.stdout:129: done (3 left) 2024-09-20T22:43:12.118 INFO:tasks.rados.rados.0.smithi084.stdout:130: done (2 left) 2024-09-20T22:43:12.118 INFO:tasks.rados.rados.0.smithi084.stdout:131: done (1 left) 2024-09-20T22:43:12.118 INFO:tasks.rados.rados.0.smithi084.stdout:132: done (0 left) 2024-09-20T22:43:12.118 INFO:tasks.rados.rados.0.smithi084.stdout:133: snap_create 2024-09-20T22:43:12.121 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:43:12.142 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:12.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:12.143 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:43:12.244 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:43:12.265 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:12.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:12.266 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:43:12.367 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:43:12.373 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-20T22:43:12.374 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2024-09-20T22:43:12.374 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2024-09-20T22:43:12.374 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2024-09-20T22:43:12.374 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-20T22:43:12.388 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:12.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:12.389 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:43:12.490 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:43:12.512 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:12.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:12.513 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:43:12.614 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:43:12.635 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:12.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:12.636 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:43:12.737 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:43:12.759 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:12.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:12.760 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:43:12.860 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:43:12.881 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:12.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:12.882 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:43:12.983 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:43:13.005 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:13.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:13.007 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:43:13.088 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.2 dump_historic_ops 2024-09-20T22:43:13.108 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:43:13.109 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is set 2024-09-20T22:43:13.116 INFO:tasks.rados.rados.0.smithi084.stdout:133: done (0 left) 2024-09-20T22:43:13.116 INFO:tasks.rados.rados.0.smithi084.stdout:134: snap_remove snap 8 2024-09-20T22:43:13.133 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:13.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:13.134 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:43:13.235 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:43:13.256 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:13.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:13.257 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:43:13.358 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:43:13.364 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:13 smithi084 ceph-mon[40024]: pgmap v70: 158 pgs: 3 peering, 155 active+clean; 118 MiB data, 454 MiB used, 715 GiB / 715 GiB avail; 32 MiB/s rd, 14 op/s 2024-09-20T22:43:13.364 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:13 smithi084 ceph-mon[40024]: 3.0 scrub starts 2024-09-20T22:43:13.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:13 smithi084 ceph-mon[40024]: 3.0 scrub ok 2024-09-20T22:43:13.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:13 smithi084 ceph-mon[40024]: osdmap e113: 8 total, 8 up, 8 in 2024-09-20T22:43:13.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:13 smithi084 ceph-mon[40024]: 4.1c deep-scrub starts 2024-09-20T22:43:13.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:13 smithi084 ceph-mon[40024]: 4.1c deep-scrub ok 2024-09-20T22:43:13.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:13 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/2027969196' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-20T22:43:13.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:13 smithi084 ceph-mon[40024]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:43:13.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:13 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/2027969196' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T22:43:13.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:13 smithi084 ceph-mon[29774]: pgmap v70: 158 pgs: 3 peering, 155 active+clean; 118 MiB data, 454 MiB used, 715 GiB / 715 GiB avail; 32 MiB/s rd, 14 op/s 2024-09-20T22:43:13.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:13 smithi084 ceph-mon[29774]: 3.0 scrub starts 2024-09-20T22:43:13.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:13 smithi084 ceph-mon[29774]: 3.0 scrub ok 2024-09-20T22:43:13.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:13 smithi084 ceph-mon[29774]: osdmap e113: 8 total, 8 up, 8 in 2024-09-20T22:43:13.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:13 smithi084 ceph-mon[29774]: 4.1c deep-scrub starts 2024-09-20T22:43:13.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:13 smithi084 ceph-mon[29774]: 4.1c deep-scrub ok 2024-09-20T22:43:13.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:13 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/2027969196' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-20T22:43:13.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:13 smithi084 ceph-mon[29774]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:43:13.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:13 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/2027969196' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T22:43:13.381 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:13.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:13.382 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:43:13.482 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:43:13.504 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:13.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:13 smithi154 ceph-mon[33264]: pgmap v70: 158 pgs: 3 peering, 155 active+clean; 118 MiB data, 454 MiB used, 715 GiB / 715 GiB avail; 32 MiB/s rd, 14 op/s 2024-09-20T22:43:13.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:13 smithi154 ceph-mon[33264]: 3.0 scrub starts 2024-09-20T22:43:13.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:13 smithi154 ceph-mon[33264]: 3.0 scrub ok 2024-09-20T22:43:13.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:13 smithi154 ceph-mon[33264]: osdmap e113: 8 total, 8 up, 8 in 2024-09-20T22:43:13.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:13 smithi154 ceph-mon[33264]: 4.1c deep-scrub starts 2024-09-20T22:43:13.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:13 smithi154 ceph-mon[33264]: 4.1c deep-scrub ok 2024-09-20T22:43:13.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:13 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/2027969196' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-20T22:43:13.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:13 smithi154 ceph-mon[33264]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:43:13.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:13 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/2027969196' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T22:43:13.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:13.506 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:43:13.606 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:43:13.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:13.628 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:13.629 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:43:13.729 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:43:13.751 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:13.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:13.752 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:43:13.853 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:43:13.876 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:13.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:13.877 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:43:13.978 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:43:13.999 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:13.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:14.000 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:43:14.100 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:43:14.122 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:14.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:14.123 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:43:14.135 INFO:tasks.rados.rados.0.smithi084.stdout:134: done (0 left) 2024-09-20T22:43:14.135 INFO:tasks.rados.rados.0.smithi084.stdout:135: write oid 21 current snap is 9 2024-09-20T22:43:14.135 INFO:tasks.rados.rados.0.smithi084.stdout:135: seq_num 73 ranges {426911=540354,1622009=726683,3121996=104915} 2024-09-20T22:43:14.141 INFO:tasks.rados.rados.0.smithi084.stdout:135: writing smithi08477203-21 from 426911 to 967265 tid 1 2024-09-20T22:43:14.149 INFO:tasks.rados.rados.0.smithi084.stdout:135: writing smithi08477203-21 from 1622009 to 2348692 tid 2 2024-09-20T22:43:14.154 INFO:tasks.rados.rados.0.smithi084.stdout:135: writing smithi08477203-21 from 3121996 to 3226911 tid 3 2024-09-20T22:43:14.154 INFO:tasks.rados.rados.0.smithi084.stdout:136: read oid 47 snap -1 2024-09-20T22:43:14.154 INFO:tasks.rados.rados.0.smithi084.stdout:136: expect (ObjNum 67 snap 7 seq_num 67) 2024-09-20T22:43:14.154 INFO:tasks.rados.rados.0.smithi084.stdout:137: delete oid 19 current snap is 9 2024-09-20T22:43:14.155 INFO:tasks.rados.rados.0.smithi084.stdout:135: finishing write tid 1 to smithi08477203-21 2024-09-20T22:43:14.155 INFO:tasks.rados.rados.0.smithi084.stdout:135: finishing write tid 2 to smithi08477203-21 2024-09-20T22:43:14.155 INFO:tasks.rados.rados.0.smithi084.stdout:135: finishing write tid 3 to smithi08477203-21 2024-09-20T22:43:14.156 INFO:tasks.rados.rados.0.smithi084.stdout:135: finishing write tid 4 to smithi08477203-21 2024-09-20T22:43:14.156 INFO:tasks.rados.rados.0.smithi084.stdout:135: finishing write tid 5 to smithi08477203-21 2024-09-20T22:43:14.156 INFO:tasks.rados.rados.0.smithi084.stdout:135: oid 21 updating version 0 to 35 2024-09-20T22:43:14.156 INFO:tasks.rados.rados.0.smithi084.stdout:135: oid 21 version 35 is already newer than 33 2024-09-20T22:43:14.156 INFO:tasks.rados.rados.0.smithi084.stdout:135: oid 21 version 35 is already newer than 34 2024-09-20T22:43:14.156 INFO:tasks.rados.rados.0.smithi084.stdout:135: oid 21 updating version 35 to 36 2024-09-20T22:43:14.156 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 21 v 36 (ObjNum 73 snap 9 seq_num 73) dirty exists 2024-09-20T22:43:14.156 INFO:tasks.rados.rados.0.smithi084.stdout:135: left oid 21 (ObjNum 73 snap 9 seq_num 73) 2024-09-20T22:43:14.159 INFO:tasks.rados.rados.0.smithi084.stdout:135: done (2 left) 2024-09-20T22:43:14.159 INFO:tasks.rados.rados.0.smithi084.stdout:137: done (1 left) 2024-09-20T22:43:14.159 INFO:tasks.rados.rados.0.smithi084.stdout:138: write oid 24 current snap is 9 2024-09-20T22:43:14.159 INFO:tasks.rados.rados.0.smithi084.stdout:138: seq_num 75 ranges {460525=690852,1622620=730352,3060489=200036} 2024-09-20T22:43:14.166 INFO:tasks.rados.rados.0.smithi084.stdout:138: writing smithi08477203-24 from 460525 to 1151377 tid 1 2024-09-20T22:43:14.173 INFO:tasks.rados.rados.0.smithi084.stdout:138: writing smithi08477203-24 from 1622620 to 2352972 tid 2 2024-09-20T22:43:14.179 INFO:tasks.rados.rados.0.smithi084.stdout:138: writing smithi08477203-24 from 3060489 to 3260525 tid 3 2024-09-20T22:43:14.179 INFO:tasks.rados.rados.0.smithi084.stdout:139: write (excl) oid 28 current snap is 9 2024-09-20T22:43:14.179 INFO:tasks.rados.rados.0.smithi084.stdout:139: seq_num 76 ranges {477332=766101,1681102=474010,2687912=589420} 2024-09-20T22:43:14.186 INFO:tasks.rados.rados.0.smithi084.stdout:139: writing smithi08477203-28 from 477332 to 1243433 tid 1 2024-09-20T22:43:14.191 INFO:tasks.rados.rados.0.smithi084.stdout:139: writing smithi08477203-28 from 1681102 to 2155112 tid 2 2024-09-20T22:43:14.192 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:14 smithi084 ceph-mon[40024]: 4.a deep-scrub starts 2024-09-20T22:43:14.192 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:14 smithi084 ceph-mon[40024]: 4.a deep-scrub ok 2024-09-20T22:43:14.192 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:14 smithi084 ceph-mon[40024]: 4.2 scrub starts 2024-09-20T22:43:14.192 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:14 smithi084 ceph-mon[40024]: 4.2 scrub ok 2024-09-20T22:43:14.192 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:14 smithi084 ceph-mon[40024]: osdmap e114: 8 total, 8 up, 8 in 2024-09-20T22:43:14.192 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:14 smithi084 ceph-mon[40024]: 3.15 scrub starts 2024-09-20T22:43:14.192 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:14 smithi084 ceph-mon[29774]: 4.a deep-scrub starts 2024-09-20T22:43:14.192 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:14 smithi084 ceph-mon[29774]: 4.a deep-scrub ok 2024-09-20T22:43:14.192 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:14 smithi084 ceph-mon[29774]: 4.2 scrub starts 2024-09-20T22:43:14.192 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:14 smithi084 ceph-mon[29774]: 4.2 scrub ok 2024-09-20T22:43:14.193 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:14 smithi084 ceph-mon[29774]: osdmap e114: 8 total, 8 up, 8 in 2024-09-20T22:43:14.193 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:14 smithi084 ceph-mon[29774]: 3.15 scrub starts 2024-09-20T22:43:14.198 INFO:tasks.rados.rados.0.smithi084.stdout:139: writing smithi08477203-28 from 2687912 to 3277332 tid 3 2024-09-20T22:43:14.198 INFO:tasks.rados.rados.0.smithi084.stdout:140: write (excl) oid 8 current snap is 9 2024-09-20T22:43:14.198 INFO:tasks.rados.rados.0.smithi084.stdout:140: seq_num 77 ranges {494139=557703,1455937=734021,2948041=346098} 2024-09-20T22:43:14.203 INFO:tasks.rados.rados.0.smithi084.stdout:140: writing smithi08477203-8 from 494139 to 1051842 tid 1 2024-09-20T22:43:14.215 INFO:tasks.rados.rados.0.smithi084.stdout:140: writing smithi08477203-8 from 1455937 to 2189958 tid 2 2024-09-20T22:43:14.216 INFO:tasks.rados.rados.0.smithi084.stdout:140: writing smithi08477203-8 from 2948041 to 3294139 tid 3 2024-09-20T22:43:14.217 INFO:tasks.rados.rados.0.smithi084.stdout:141: delete oid 50 current snap is 9 2024-09-20T22:43:14.217 INFO:tasks.rados.rados.0.smithi084.stdout:136: expect (ObjNum 67 snap 7 seq_num 67) 2024-09-20T22:43:14.224 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:43:14.246 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:14.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:14.247 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:43:14.348 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:43:14.370 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:14.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:14.371 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:43:14.406 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:43:14.406 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:43:14.413 INFO:tasks.rados.rados.0.smithi084.stdout:138: finishing write tid 1 to smithi08477203-24 2024-09-20T22:43:14.413 INFO:tasks.rados.rados.0.smithi084.stdout:138: finishing write tid 2 to smithi08477203-24 2024-09-20T22:43:14.413 INFO:tasks.rados.rados.0.smithi084.stdout:136: done (4 left) 2024-09-20T22:43:14.413 INFO:tasks.rados.rados.0.smithi084.stdout:141: done (3 left) 2024-09-20T22:43:14.413 INFO:tasks.rados.rados.0.smithi084.stdout:142: rollback oid 26 current snap is 9 2024-09-20T22:43:14.413 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 26 to 7 2024-09-20T22:43:14.414 INFO:tasks.rados.rados.0.smithi084.stdout:138: finishing write tid 3 to smithi08477203-24 2024-09-20T22:43:14.414 INFO:tasks.rados.rados.0.smithi084.stdout:138: finishing write tid 4 to smithi08477203-24 2024-09-20T22:43:14.414 INFO:tasks.rados.rados.0.smithi084.stdout:138: finishing write tid 5 to smithi08477203-24 2024-09-20T22:43:14.414 INFO:tasks.rados.rados.0.smithi084.stdout:138: oid 24 updating version 0 to 15 2024-09-20T22:43:14.414 INFO:tasks.rados.rados.0.smithi084.stdout:138: oid 24 updating version 15 to 18 2024-09-20T22:43:14.414 INFO:tasks.rados.rados.0.smithi084.stdout:138: oid 24 version 18 is already newer than 17 2024-09-20T22:43:14.414 INFO:tasks.rados.rados.0.smithi084.stdout:138: oid 24 version 18 is already newer than 16 2024-09-20T22:43:14.414 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 24 v 18 (ObjNum 75 snap 9 seq_num 75) dirty exists 2024-09-20T22:43:14.414 INFO:tasks.rados.rados.0.smithi084.stdout:138: left oid 24 (ObjNum 75 snap 9 seq_num 75) 2024-09-20T22:43:14.414 INFO:tasks.rados.rados.0.smithi084.stdout:139: finishing write tid 1 to smithi08477203-28 2024-09-20T22:43:14.414 INFO:tasks.rados.rados.0.smithi084.stdout:139: finishing write tid 2 to smithi08477203-28 2024-09-20T22:43:14.414 INFO:tasks.rados.rados.0.smithi084.stdout:139: finishing write tid 3 to smithi08477203-28 2024-09-20T22:43:14.414 INFO:tasks.rados.rados.0.smithi084.stdout:139: finishing write tid 4 to smithi08477203-28 2024-09-20T22:43:14.414 INFO:tasks.rados.rados.0.smithi084.stdout:139: finishing write tid 5 to smithi08477203-28 2024-09-20T22:43:14.415 INFO:tasks.rados.rados.0.smithi084.stdout:139: oid 28 updating version 0 to 42 2024-09-20T22:43:14.415 INFO:tasks.rados.rados.0.smithi084.stdout:139: oid 28 updating version 42 to 43 2024-09-20T22:43:14.415 INFO:tasks.rados.rados.0.smithi084.stdout:139: oid 28 updating version 43 to 45 2024-09-20T22:43:14.415 INFO:tasks.rados.rados.0.smithi084.stdout:139: oid 28 version 45 is already newer than 44 2024-09-20T22:43:14.415 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 28 v 45 (ObjNum 76 snap 9 seq_num 76) dirty exists 2024-09-20T22:43:14.415 INFO:tasks.rados.rados.0.smithi084.stdout:139: left oid 28 (ObjNum 76 snap 9 seq_num 76) 2024-09-20T22:43:14.415 INFO:tasks.rados.rados.0.smithi084.stdout:140: finishing write tid 1 to smithi08477203-8 2024-09-20T22:43:14.415 INFO:tasks.rados.rados.0.smithi084.stdout:140: finishing write tid 2 to smithi08477203-8 2024-09-20T22:43:14.415 INFO:tasks.rados.rados.0.smithi084.stdout:140: finishing write tid 3 to smithi08477203-8 2024-09-20T22:43:14.415 INFO:tasks.rados.rados.0.smithi084.stdout:140: finishing write tid 4 to smithi08477203-8 2024-09-20T22:43:14.415 INFO:tasks.rados.rados.0.smithi084.stdout:140: finishing write tid 5 to smithi08477203-8 2024-09-20T22:43:14.415 INFO:tasks.rados.rados.0.smithi084.stdout:140: oid 8 updating version 0 to 31 2024-09-20T22:43:14.415 INFO:tasks.rados.rados.0.smithi084.stdout:140: oid 8 updating version 31 to 32 2024-09-20T22:43:14.416 INFO:tasks.rados.rados.0.smithi084.stdout:140: oid 8 updating version 32 to 33 2024-09-20T22:43:14.416 INFO:tasks.rados.rados.0.smithi084.stdout:140: oid 8 updating version 33 to 34 2024-09-20T22:43:14.416 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 8 v 34 (ObjNum 77 snap 9 seq_num 77) dirty exists 2024-09-20T22:43:14.416 INFO:tasks.rados.rados.0.smithi084.stdout:140: left oid 8 (ObjNum 77 snap 9 seq_num 77) 2024-09-20T22:43:14.416 INFO:tasks.rados.rados.0.smithi084.stdout:138: done (3 left) 2024-09-20T22:43:14.416 INFO:tasks.rados.rados.0.smithi084.stdout:139: done (2 left) 2024-09-20T22:43:14.416 INFO:tasks.rados.rados.0.smithi084.stdout:140: done (1 left) 2024-09-20T22:43:14.416 INFO:tasks.rados.rados.0.smithi084.stdout:143: read oid 11 snap -1 2024-09-20T22:43:14.416 INFO:tasks.rados.rados.0.smithi084.stdout:143: expect (ObjNum 63 snap 4 seq_num 63) 2024-09-20T22:43:14.416 INFO:tasks.rados.rados.0.smithi084.stdout:144: write (excl) oid 21 current snap is 9 2024-09-20T22:43:14.416 INFO:tasks.rados.rados.0.smithi084.stdout:144: seq_num 79 ranges {527753=708201,1856548=737690,3286534=41219} 2024-09-20T22:43:14.420 INFO:tasks.rados.rados.0.smithi084.stdout:144: writing smithi08477203-21 from 527753 to 1235954 tid 1 2024-09-20T22:43:14.428 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:43:14.429 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:43:14.429 INFO:tasks.rados.rados.0.smithi084.stdout:144: writing smithi08477203-21 from 1856548 to 2594238 tid 2 2024-09-20T22:43:14.433 INFO:tasks.rados.rados.0.smithi084.stdout:144: writing smithi08477203-21 from 3286534 to 3327753 tid 3 2024-09-20T22:43:14.433 INFO:tasks.rados.rados.0.smithi084.stdout:145: write oid 20 current snap is 9 2024-09-20T22:43:14.433 INFO:tasks.rados.rados.0.smithi084.stdout:142: finishing rollback tid 0 to smithi08477203-26 2024-09-20T22:43:14.433 INFO:tasks.rados.rados.0.smithi084.stdout:145: seq_num 80 ranges {544560=783450,1915030=481348,2797604=479761} 2024-09-20T22:43:14.441 INFO:tasks.rados.rados.0.smithi084.stdout:145: writing smithi08477203-20 from 544560 to 1328010 tid 1 2024-09-20T22:43:14.447 INFO:tasks.rados.rados.0.smithi084.stdout:145: writing smithi08477203-20 from 1915030 to 2396378 tid 2 2024-09-20T22:43:14.452 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:43:14.452 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:43:14.453 INFO:tasks.rados.rados.0.smithi084.stdout:145: writing smithi08477203-20 from 2797604 to 3277365 tid 3 2024-09-20T22:43:14.454 INFO:tasks.rados.rados.0.smithi084.stdout:146: snap_create 2024-09-20T22:43:14.454 INFO:tasks.rados.rados.0.smithi084.stdout:142: finishing rollback tid 1 to smithi08477203-26 2024-09-20T22:43:14.454 INFO:tasks.rados.rados.0.smithi084.stdout:142: finishing rollback tid 2 to smithi08477203-26 2024-09-20T22:43:14.454 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 26 v 61 (ObjNum 25 snap 0 seq_num 25) dirty exists 2024-09-20T22:43:14.454 INFO:tasks.rados.rados.0.smithi084.stdout:143: expect (ObjNum 63 snap 4 seq_num 63) 2024-09-20T22:43:14.457 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:43:14.459 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:43:14.471 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:43:14.475 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-20T22:43:14.475 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T22:43:14.496 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:43:14.497 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:43:14.500 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:14.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:14.501 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:43:14.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:14 smithi154 ceph-mon[33264]: 4.a deep-scrub starts 2024-09-20T22:43:14.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:14 smithi154 ceph-mon[33264]: 4.a deep-scrub ok 2024-09-20T22:43:14.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:14 smithi154 ceph-mon[33264]: 4.2 scrub starts 2024-09-20T22:43:14.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:14 smithi154 ceph-mon[33264]: 4.2 scrub ok 2024-09-20T22:43:14.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:14 smithi154 ceph-mon[33264]: osdmap e114: 8 total, 8 up, 8 in 2024-09-20T22:43:14.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:14 smithi154 ceph-mon[33264]: 3.15 scrub starts 2024-09-20T22:43:14.518 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:43:14.518 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:43:14.534 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:43:14.535 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:43:14.551 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:43:14.551 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:43:14.603 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:43:14.609 INFO:tasks.rados.rados.0.smithi084.stdout:144: finishing write tid 1 to smithi08477203-21 2024-09-20T22:43:14.609 INFO:tasks.rados.rados.0.smithi084.stdout:144: finishing write tid 2 to smithi08477203-21 2024-09-20T22:43:14.609 INFO:tasks.rados.rados.0.smithi084.stdout:144: finishing write tid 3 to smithi08477203-21 2024-09-20T22:43:14.609 INFO:tasks.rados.rados.0.smithi084.stdout:144: finishing write tid 4 to smithi08477203-21 2024-09-20T22:43:14.609 INFO:tasks.rados.rados.0.smithi084.stdout:144: finishing write tid 5 to smithi08477203-21 2024-09-20T22:43:14.609 INFO:tasks.rados.rados.0.smithi084.stdout:144: oid 21 updating version 0 to 37 2024-09-20T22:43:14.609 INFO:tasks.rados.rados.0.smithi084.stdout:144: oid 21 updating version 37 to 39 2024-09-20T22:43:14.609 INFO:tasks.rados.rados.0.smithi084.stdout:144: oid 21 version 39 is already newer than 38 2024-09-20T22:43:14.609 INFO:tasks.rados.rados.0.smithi084.stdout:144: oid 21 updating version 39 to 40 2024-09-20T22:43:14.609 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 21 v 40 (ObjNum 79 snap 9 seq_num 79) dirty exists 2024-09-20T22:43:14.609 INFO:tasks.rados.rados.0.smithi084.stdout:144: left oid 21 (ObjNum 79 snap 9 seq_num 79) 2024-09-20T22:43:14.609 INFO:tasks.rados.rados.0.smithi084.stdout:145: finishing write tid 1 to smithi08477203-20 2024-09-20T22:43:14.610 INFO:tasks.rados.rados.0.smithi084.stdout:145: finishing write tid 2 to smithi08477203-20 2024-09-20T22:43:14.610 INFO:tasks.rados.rados.0.smithi084.stdout:145: finishing write tid 3 to smithi08477203-20 2024-09-20T22:43:14.610 INFO:tasks.rados.rados.0.smithi084.stdout:145: finishing write tid 4 to smithi08477203-20 2024-09-20T22:43:14.610 INFO:tasks.rados.rados.0.smithi084.stdout:145: finishing write tid 5 to smithi08477203-20 2024-09-20T22:43:14.610 INFO:tasks.rados.rados.0.smithi084.stdout:145: oid 20 updating version 0 to 20 2024-09-20T22:43:14.610 INFO:tasks.rados.rados.0.smithi084.stdout:145: oid 20 version 20 is already newer than 19 2024-09-20T22:43:14.610 INFO:tasks.rados.rados.0.smithi084.stdout:145: oid 20 version 20 is already newer than 18 2024-09-20T22:43:14.610 INFO:tasks.rados.rados.0.smithi084.stdout:145: oid 20 updating version 20 to 21 2024-09-20T22:43:14.610 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 20 v 21 (ObjNum 80 snap 9 seq_num 80) dirty exists 2024-09-20T22:43:14.610 INFO:tasks.rados.rados.0.smithi084.stdout:145: left oid 20 (ObjNum 80 snap 9 seq_num 80) 2024-09-20T22:43:14.624 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:14.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:14.625 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:43:14.718 INFO:journalctl@ceph.alertmanager.a.smithi084.stdout:Sep 20 22:43:14 smithi084 systemd[1]: ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490@alertmanager.a.service: Deactivated successfully. 2024-09-20T22:43:14.718 INFO:journalctl@ceph.alertmanager.a.smithi084.stdout:Sep 20 22:43:14 smithi084 systemd[1]: Stopped Ceph alertmanager.a for a9c0bc70-77a0-11ef-baf5-efdc52797490. 2024-09-20T22:43:14.718 INFO:journalctl@ceph.alertmanager.a.smithi084.stdout:Sep 20 22:43:14 smithi084 systemd[1]: Starting Ceph alertmanager.a for a9c0bc70-77a0-11ef-baf5-efdc52797490... 2024-09-20T22:43:14.725 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:43:14.748 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:14.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:14.749 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:43:14.850 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:43:14.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:14.872 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:14.872 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:43:14.973 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:43:14.995 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:14.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:14.996 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:43:15.097 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:43:15.122 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:15.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:15.123 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:43:15.137 INFO:tasks.rados.rados.0.smithi084.stdout:142: done (4 left) 2024-09-20T22:43:15.137 INFO:tasks.rados.rados.0.smithi084.stdout:143: done (3 left) 2024-09-20T22:43:15.137 INFO:tasks.rados.rados.0.smithi084.stdout:144: done (2 left) 2024-09-20T22:43:15.137 INFO:tasks.rados.rados.0.smithi084.stdout:145: done (1 left) 2024-09-20T22:43:15.137 INFO:tasks.rados.rados.0.smithi084.stdout:146: done (0 left) 2024-09-20T22:43:15.137 INFO:tasks.rados.rados.0.smithi084.stdout:147: read oid 29 snap 9 2024-09-20T22:43:15.137 INFO:tasks.rados.rados.0.smithi084.stdout:147: expect (ObjNum 59 snap 4 seq_num 59) 2024-09-20T22:43:15.137 INFO:tasks.rados.rados.0.smithi084.stdout:148: write (excl) oid 21 current snap is 10 2024-09-20T22:43:15.137 INFO:tasks.rados.rados.0.smithi084.stdout:148: seq_num 81 ranges {561367=458699,1573512=741359,2941380=419987} 2024-09-20T22:43:15.142 INFO:tasks.rados.rados.0.smithi084.stdout:148: writing smithi08477203-21 from 561367 to 1020066 tid 1 2024-09-20T22:43:15.150 INFO:tasks.rados.rados.0.smithi084.stdout:148: writing smithi08477203-21 from 1573512 to 2314871 tid 2 2024-09-20T22:43:15.156 INFO:tasks.rados.rados.0.smithi084.stdout:148: writing smithi08477203-21 from 2941380 to 3361367 tid 3 2024-09-20T22:43:15.156 INFO:tasks.rados.rados.0.smithi084.stdout:149: delete oid 23 current snap is 10 2024-09-20T22:43:15.156 INFO:tasks.rados.rados.0.smithi084.stdout:147: expect (ObjNum 59 snap 4 seq_num 59) 2024-09-20T22:43:15.224 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:43:15.245 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:15.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:15.246 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:43:15.268 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:15 smithi084 ceph-mon[29774]: pgmap v73: 158 pgs: 1 active+clean+snaptrim, 1 active+clean+scrubbing, 156 active+clean; 114 MiB data, 464 MiB used, 715 GiB / 715 GiB avail; 7.8 MiB/s rd, 2.1 MiB/s wr, 17 op/s; 54 B/s, 3 objects/s recovering 2024-09-20T22:43:15.268 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:15 smithi084 ceph-mon[29774]: osdmap e115: 8 total, 8 up, 8 in 2024-09-20T22:43:15.268 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:15 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/1933731854' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:43:15.268 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:15 smithi084 ceph-mon[40024]: pgmap v73: 158 pgs: 1 active+clean+snaptrim, 1 active+clean+scrubbing, 156 active+clean; 114 MiB data, 464 MiB used, 715 GiB / 715 GiB avail; 7.8 MiB/s rd, 2.1 MiB/s wr, 17 op/s; 54 B/s, 3 objects/s recovering 2024-09-20T22:43:15.268 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:15 smithi084 ceph-mon[40024]: osdmap e115: 8 total, 8 up, 8 in 2024-09-20T22:43:15.268 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:15 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/1933731854' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:43:15.295 INFO:tasks.rados.rados.0.smithi084.stdout:148: finishing write tid 1 to smithi08477203-21 2024-09-20T22:43:15.295 INFO:tasks.rados.rados.0.smithi084.stdout:148: finishing write tid 2 to smithi08477203-21 2024-09-20T22:43:15.295 INFO:tasks.rados.rados.0.smithi084.stdout:148: finishing write tid 3 to smithi08477203-21 2024-09-20T22:43:15.295 INFO:tasks.rados.rados.0.smithi084.stdout:148: finishing write tid 4 to smithi08477203-21 2024-09-20T22:43:15.295 INFO:tasks.rados.rados.0.smithi084.stdout:148: finishing write tid 5 to smithi08477203-21 2024-09-20T22:43:15.295 INFO:tasks.rados.rados.0.smithi084.stdout:148: oid 21 updating version 0 to 45 2024-09-20T22:43:15.296 INFO:tasks.rados.rados.0.smithi084.stdout:148: oid 21 version 45 is already newer than 43 2024-09-20T22:43:15.296 INFO:tasks.rados.rados.0.smithi084.stdout:148: oid 21 version 45 is already newer than 44 2024-09-20T22:43:15.297 INFO:tasks.rados.rados.0.smithi084.stdout:148: oid 21 version 45 is already newer than 42 2024-09-20T22:43:15.297 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 21 v 45 (ObjNum 81 snap 10 seq_num 81) dirty exists 2024-09-20T22:43:15.297 INFO:tasks.rados.rados.0.smithi084.stdout:148: left oid 21 (ObjNum 81 snap 10 seq_num 81) 2024-09-20T22:43:15.297 INFO:tasks.rados.rados.0.smithi084.stdout:147: done (2 left) 2024-09-20T22:43:15.297 INFO:tasks.rados.rados.0.smithi084.stdout:148: done (1 left) 2024-09-20T22:43:15.297 INFO:tasks.rados.rados.0.smithi084.stdout:149: done (0 left) 2024-09-20T22:43:15.297 INFO:tasks.rados.rados.0.smithi084.stdout:150: read oid 28 snap 7 2024-09-20T22:43:15.298 INFO:tasks.rados.rados.0.smithi084.stdout:150: expect deleted 2024-09-20T22:43:15.298 INFO:tasks.rados.rados.0.smithi084.stdout:151: snap_remove snap 9 2024-09-20T22:43:15.347 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:43:15.369 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:15.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:15.370 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:43:15.376 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-20T22:43:15.376 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:43:15.472 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:43:15.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:15.498 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:15.499 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:43:15.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:15 smithi154 ceph-mon[33264]: pgmap v73: 158 pgs: 1 active+clean+snaptrim, 1 active+clean+scrubbing, 156 active+clean; 114 MiB data, 464 MiB used, 715 GiB / 715 GiB avail; 7.8 MiB/s rd, 2.1 MiB/s wr, 17 op/s; 54 B/s, 3 objects/s recovering 2024-09-20T22:43:15.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:15 smithi154 ceph-mon[33264]: osdmap e115: 8 total, 8 up, 8 in 2024-09-20T22:43:15.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:15 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/1933731854' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:43:15.600 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:43:15.623 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:15.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:15.624 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:43:15.724 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:43:15.750 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:15.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:15.751 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:43:15.852 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:43:15.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:15.875 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:15.875 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:43:15.961 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-20T22:43:15.975 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:43:16.024 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:16.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:16.026 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:43:16.070 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-20T22:43:16.127 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:43:16.137 INFO:teuthology.orchestra.run.smithi084.stderr:set pool 1 pg_num to 21 2024-09-20T22:43:16.145 INFO:tasks.rados.rados.0.smithi084.stdout:151: done (1 left) 2024-09-20T22:43:16.145 INFO:tasks.rados.rados.0.smithi084.stdout:152: write oid 44 current snap is 10 2024-09-20T22:43:16.145 INFO:tasks.rados.rados.0.smithi084.stdout:152: seq_num 83 ranges {594981=609197,1974123=745028,3279873=115108} 2024-09-20T22:43:16.152 INFO:tasks.rados.rados.0.smithi084.stdout:152: writing smithi08477203-44 from 594981 to 1204178 tid 1 2024-09-20T22:43:16.154 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:16.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:16.155 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:43:16.164 INFO:tasks.rados.rados.0.smithi084.stdout:152: writing smithi08477203-44 from 1974123 to 2719151 tid 2 2024-09-20T22:43:16.169 INFO:tasks.rados.rados.0.smithi084.stdout:152: writing smithi08477203-44 from 3279873 to 3394981 tid 3 2024-09-20T22:43:16.169 INFO:tasks.rados.rados.0.smithi084.stdout:152: finishing write tid 1 to smithi08477203-44 2024-09-20T22:43:16.170 INFO:tasks.rados.rados.0.smithi084.stdout:152: finishing write tid 2 to smithi08477203-44 2024-09-20T22:43:16.170 INFO:tasks.rados.rados.0.smithi084.stdout:150: done (1 left) 2024-09-20T22:43:16.170 INFO:tasks.rados.rados.0.smithi084.stdout:153: snap_remove snap 7 2024-09-20T22:43:16.256 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:43:16.260 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:16 smithi084 ceph-mon[40024]: osdmap e116: 8 total, 8 up, 8 in 2024-09-20T22:43:16.260 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:16 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/1811330335' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]: dispatch 2024-09-20T22:43:16.260 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:16 smithi084 ceph-mon[29774]: osdmap e116: 8 total, 8 up, 8 in 2024-09-20T22:43:16.260 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:16 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/1811330335' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]: dispatch 2024-09-20T22:43:16.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:16.280 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:16.280 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:43:16.381 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:43:16.402 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:16.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:16.403 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:43:16.504 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:43:16.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:16 smithi154 ceph-mon[33264]: osdmap e116: 8 total, 8 up, 8 in 2024-09-20T22:43:16.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:16 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/1811330335' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]: dispatch 2024-09-20T22:43:16.529 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:16.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:16.530 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:43:16.631 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:43:16.652 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:16.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:16.653 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:43:16.754 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:43:16.780 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:16.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:16.781 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:43:16.882 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:43:16.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:16.905 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:16.905 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:43:17.006 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:43:17.027 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:17.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:17.029 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:43:17.086 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:43:17.130 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:43:17.149 INFO:tasks.rados.rados.0.smithi084.stdout:153: done (1 left) 2024-09-20T22:43:17.149 INFO:tasks.rados.rados.0.smithi084.stdout:154: read oid 23 snap -1 2024-09-20T22:43:17.149 INFO:tasks.rados.rados.0.smithi084.stdout:154: expect deleted 2024-09-20T22:43:17.149 INFO:tasks.rados.rados.0.smithi084.stdout:155: snap_create 2024-09-20T22:43:17.150 INFO:tasks.rados.rados.0.smithi084.stdout:152: finishing write tid 3 to smithi08477203-44 2024-09-20T22:43:17.150 INFO:tasks.rados.rados.0.smithi084.stdout:152: finishing write tid 4 to smithi08477203-44 2024-09-20T22:43:17.150 INFO:tasks.rados.rados.0.smithi084.stdout:152: finishing write tid 5 to smithi08477203-44 2024-09-20T22:43:17.150 INFO:tasks.rados.rados.0.smithi084.stdout:152: oid 44 updating version 0 to 50 2024-09-20T22:43:17.150 INFO:tasks.rados.rados.0.smithi084.stdout:152: oid 44 version 50 is already newer than 48 2024-09-20T22:43:17.150 INFO:tasks.rados.rados.0.smithi084.stdout:152: oid 44 version 50 is already newer than 47 2024-09-20T22:43:17.150 INFO:tasks.rados.rados.0.smithi084.stdout:152: oid 44 version 50 is already newer than 49 2024-09-20T22:43:17.150 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 44 v 50 (ObjNum 83 snap 10 seq_num 83) dirty exists 2024-09-20T22:43:17.150 INFO:tasks.rados.rados.0.smithi084.stdout:152: left oid 44 (ObjNum 83 snap 10 seq_num 83) 2024-09-20T22:43:17.155 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:17.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:17.156 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:43:17.257 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:43:17.284 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:17.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:17.285 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:43:17.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:17 smithi084 ceph-mon[29774]: pgmap v76: 158 pgs: 1 active+clean+snaptrim, 1 active+clean+scrubbing, 156 active+clean; 114 MiB data, 464 MiB used, 715 GiB / 715 GiB avail; 7.8 MiB/s rd, 2.1 MiB/s wr, 17 op/s; 54 B/s, 3 objects/s recovering 2024-09-20T22:43:17.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:17 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/1811330335' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]': finished 2024-09-20T22:43:17.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:17 smithi084 ceph-mon[29774]: osdmap e117: 8 total, 8 up, 8 in 2024-09-20T22:43:17.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:17 smithi084 ceph-mon[40024]: pgmap v76: 158 pgs: 1 active+clean+snaptrim, 1 active+clean+scrubbing, 156 active+clean; 114 MiB data, 464 MiB used, 715 GiB / 715 GiB avail; 7.8 MiB/s rd, 2.1 MiB/s wr, 17 op/s; 54 B/s, 3 objects/s recovering 2024-09-20T22:43:17.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:17 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/1811330335' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]': finished 2024-09-20T22:43:17.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:17 smithi084 ceph-mon[40024]: osdmap e117: 8 total, 8 up, 8 in 2024-09-20T22:43:17.386 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:43:17.407 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:17.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:17.408 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:43:17.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:17 smithi154 ceph-mon[33264]: pgmap v76: 158 pgs: 1 active+clean+snaptrim, 1 active+clean+scrubbing, 156 active+clean; 114 MiB data, 464 MiB used, 715 GiB / 715 GiB avail; 7.8 MiB/s rd, 2.1 MiB/s wr, 17 op/s; 54 B/s, 3 objects/s recovering 2024-09-20T22:43:17.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:17 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/1811330335' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "21"}]': finished 2024-09-20T22:43:17.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:17 smithi154 ceph-mon[33264]: osdmap e117: 8 total, 8 up, 8 in 2024-09-20T22:43:17.509 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:43:17.534 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:17.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:17.535 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:43:17.636 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:43:17.658 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:17.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:17.659 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:43:17.759 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:43:17.785 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:17.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:17.786 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:43:17.887 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:43:17.913 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:17.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:17.913 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:43:18.014 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:43:18.040 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:18.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:18.041 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:43:18.117 INFO:journalctl@ceph.alertmanager.a.smithi084.stdout:Sep 20 22:43:17 smithi084 podman[80540]: 2024-09-20 22:43:17.887722017 +0000 UTC m=+0.247611351 image pull c8568f914cd25b2062c44e9f79f9c18da6e3b85fe0c47a12a2191c61426c2b19 quay.io/prometheus/alertmanager:v0.25.0 2024-09-20T22:43:18.118 INFO:tasks.rados.rados.0.smithi084.stdout:152: done (2 left) 2024-09-20T22:43:18.118 INFO:tasks.rados.rados.0.smithi084.stdout:154: done (1 left) 2024-09-20T22:43:18.118 INFO:tasks.rados.rados.0.smithi084.stdout:155: done (0 left) 2024-09-20T22:43:18.118 INFO:tasks.rados.rados.0.smithi084.stdout:156: rollback oid 38 current snap is 11 2024-09-20T22:43:18.118 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 38 to 10 2024-09-20T22:43:18.118 INFO:tasks.rados.rados.0.smithi084.stdout:157: write oid 33 current snap is 11 2024-09-20T22:43:18.118 INFO:tasks.rados.rados.0.smithi084.stdout:157: seq_num 84 ranges {611788=400799,1748958=488686,2907296=504492} 2024-09-20T22:43:18.124 INFO:tasks.rados.rados.0.smithi084.stdout:157: writing smithi08477203-33 from 611788 to 1012587 tid 1 2024-09-20T22:43:18.131 INFO:tasks.rados.rados.0.smithi084.stdout:157: writing smithi08477203-33 from 1748958 to 2237644 tid 2 2024-09-20T22:43:18.135 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:43:18.138 INFO:tasks.rados.rados.0.smithi084.stdout:157: writing smithi08477203-33 from 2907296 to 3411788 tid 3 2024-09-20T22:43:18.138 INFO:tasks.rados.rados.0.smithi084.stdout:158: read oid 34 snap 10 2024-09-20T22:43:18.138 INFO:tasks.rados.rados.0.smithi084.stdout:158: expect (ObjNum 61 snap 4 seq_num 61) 2024-09-20T22:43:18.138 INFO:tasks.rados.rados.0.smithi084.stdout:159: write (excl) oid 22 current snap is 11 2024-09-20T22:43:18.138 INFO:tasks.rados.rados.0.smithi084.stdout:156: finishing rollback tid 0 to smithi08477203-38 2024-09-20T22:43:18.138 INFO:tasks.rados.rados.0.smithi084.stdout:159: seq_num 85 ranges {628595=476048,1807440=632344,2934719=452474} 2024-09-20T22:43:18.142 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:43:18.145 INFO:tasks.rados.rados.0.smithi084.stdout:159: writing smithi08477203-22 from 628595 to 1104643 tid 1 2024-09-20T22:43:18.152 INFO:tasks.rados.rados.0.smithi084.stdout:159: writing smithi08477203-22 from 1807440 to 2439784 tid 2 2024-09-20T22:43:18.157 INFO:tasks.rados.rados.0.smithi084.stdout:159: writing smithi08477203-22 from 2934719 to 3387193 tid 3 2024-09-20T22:43:18.157 INFO:tasks.rados.rados.0.smithi084.stdout:160: read oid 19 snap -1 2024-09-20T22:43:18.158 INFO:tasks.rados.rados.0.smithi084.stdout:160: expect deleted 2024-09-20T22:43:18.158 INFO:tasks.rados.rados.0.smithi084.stdout:161: delete oid 31 current snap is 11 2024-09-20T22:43:18.158 INFO:tasks.rados.rados.0.smithi084.stdout:156: finishing rollback tid 1 to smithi08477203-38 2024-09-20T22:43:18.158 INFO:tasks.rados.rados.0.smithi084.stdout:156: finishing rollback tid 2 to smithi08477203-38 2024-09-20T22:43:18.158 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 38 v 67 (ObjNum 60 snap 4 seq_num 60) dirty exists 2024-09-20T22:43:18.158 INFO:tasks.rados.rados.0.smithi084.stdout:157: finishing write tid 1 to smithi08477203-33 2024-09-20T22:43:18.158 INFO:tasks.rados.rados.0.smithi084.stdout:157: finishing write tid 2 to smithi08477203-33 2024-09-20T22:43:18.158 INFO:tasks.rados.rados.0.smithi084.stdout:157: finishing write tid 3 to smithi08477203-33 2024-09-20T22:43:18.158 INFO:tasks.rados.rados.0.smithi084.stdout:157: finishing write tid 4 to smithi08477203-33 2024-09-20T22:43:18.158 INFO:tasks.rados.rados.0.smithi084.stdout:158: expect (ObjNum 61 snap 4 seq_num 61) 2024-09-20T22:43:18.168 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:18.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:18.169 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:43:18.261 INFO:tasks.rados.rados.0.smithi084.stdout:157: finishing write tid 5 to smithi08477203-33 2024-09-20T22:43:18.261 INFO:tasks.rados.rados.0.smithi084.stdout:157: oid 33 updating version 0 to 24 2024-09-20T22:43:18.261 INFO:tasks.rados.rados.0.smithi084.stdout:157: oid 33 version 24 is already newer than 23 2024-09-20T22:43:18.261 INFO:tasks.rados.rados.0.smithi084.stdout:157: oid 33 updating version 24 to 25 2024-09-20T22:43:18.261 INFO:tasks.rados.rados.0.smithi084.stdout:157: oid 33 updating version 25 to 26 2024-09-20T22:43:18.261 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 33 v 26 (ObjNum 84 snap 11 seq_num 84) dirty exists 2024-09-20T22:43:18.261 INFO:tasks.rados.rados.0.smithi084.stdout:157: left oid 33 (ObjNum 84 snap 11 seq_num 84) 2024-09-20T22:43:18.261 INFO:tasks.rados.rados.0.smithi084.stdout:159: finishing write tid 1 to smithi08477203-22 2024-09-20T22:43:18.261 INFO:tasks.rados.rados.0.smithi084.stdout:156: done (5 left) 2024-09-20T22:43:18.262 INFO:tasks.rados.rados.0.smithi084.stdout:157: done (4 left) 2024-09-20T22:43:18.262 INFO:tasks.rados.rados.0.smithi084.stdout:158: done (3 left) 2024-09-20T22:43:18.262 INFO:tasks.rados.rados.0.smithi084.stdout:161: done (2 left) 2024-09-20T22:43:18.262 INFO:tasks.rados.rados.0.smithi084.stdout:162: write (excl) oid 34 current snap is 11 2024-09-20T22:43:18.262 INFO:tasks.rados.rados.0.smithi084.stdout:159: finishing write tid 2 to smithi08477203-22 2024-09-20T22:43:18.262 INFO:tasks.rados.rados.0.smithi084.stdout:162: seq_num 87 ranges {662209=626546,1808051=636013,2873212=588997} 2024-09-20T22:43:18.267 INFO:tasks.rados.rados.0.smithi084.stdout:162: writing smithi08477203-34 from 662209 to 1288755 tid 1 2024-09-20T22:43:18.269 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:43:18.274 INFO:tasks.rados.rados.0.smithi084.stdout:162: writing smithi08477203-34 from 1808051 to 2444064 tid 2 2024-09-20T22:43:18.280 INFO:tasks.rados.rados.0.smithi084.stdout:162: writing smithi08477203-34 from 2873212 to 3462209 tid 3 2024-09-20T22:43:18.280 INFO:tasks.rados.rados.0.smithi084.stdout:163: snap_create 2024-09-20T22:43:18.280 INFO:tasks.rados.rados.0.smithi084.stdout:159: finishing write tid 3 to smithi08477203-22 2024-09-20T22:43:18.280 INFO:tasks.rados.rados.0.smithi084.stdout:159: finishing write tid 4 to smithi08477203-22 2024-09-20T22:43:18.280 INFO:tasks.rados.rados.0.smithi084.stdout:159: finishing write tid 5 to smithi08477203-22 2024-09-20T22:43:18.280 INFO:tasks.rados.rados.0.smithi084.stdout:159: oid 22 updating version 0 to 26 2024-09-20T22:43:18.280 INFO:tasks.rados.rados.0.smithi084.stdout:159: oid 22 version 26 is already newer than 23 2024-09-20T22:43:18.280 INFO:tasks.rados.rados.0.smithi084.stdout:159: oid 22 version 26 is already newer than 24 2024-09-20T22:43:18.280 INFO:tasks.rados.rados.0.smithi084.stdout:159: oid 22 version 26 is already newer than 25 2024-09-20T22:43:18.281 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 22 v 26 (ObjNum 85 snap 11 seq_num 85) dirty exists 2024-09-20T22:43:18.281 INFO:tasks.rados.rados.0.smithi084.stdout:159: left oid 22 (ObjNum 85 snap 11 seq_num 85) 2024-09-20T22:43:18.281 INFO:tasks.rados.rados.0.smithi084.stdout:162: finishing write tid 1 to smithi08477203-34 2024-09-20T22:43:18.281 INFO:tasks.rados.rados.0.smithi084.stdout:162: finishing write tid 2 to smithi08477203-34 2024-09-20T22:43:18.283 INFO:tasks.rados.rados.0.smithi084.stdout:162: finishing write tid 3 to smithi08477203-34 2024-09-20T22:43:18.283 INFO:tasks.rados.rados.0.smithi084.stdout:162: finishing write tid 4 to smithi08477203-34 2024-09-20T22:43:18.283 INFO:tasks.rados.rados.0.smithi084.stdout:162: finishing write tid 5 to smithi08477203-34 2024-09-20T22:43:18.283 INFO:tasks.rados.rados.0.smithi084.stdout:162: oid 34 updating version 0 to 52 2024-09-20T22:43:18.283 INFO:tasks.rados.rados.0.smithi084.stdout:162: oid 34 updating version 52 to 53 2024-09-20T22:43:18.283 INFO:tasks.rados.rados.0.smithi084.stdout:162: oid 34 updating version 53 to 54 2024-09-20T22:43:18.283 INFO:tasks.rados.rados.0.smithi084.stdout:162: oid 34 version 54 is already newer than 51 2024-09-20T22:43:18.283 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 34 v 54 (ObjNum 87 snap 11 seq_num 87) dirty exists 2024-09-20T22:43:18.283 INFO:tasks.rados.rados.0.smithi084.stdout:162: left oid 34 (ObjNum 87 snap 11 seq_num 87) 2024-09-20T22:43:18.292 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:18.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:18.293 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:43:18.394 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:43:18.415 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:18.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:18.416 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:43:18.493 INFO:journalctl@ceph.alertmanager.a.smithi084.stdout:Sep 20 22:43:18 smithi084 podman[80540]: 2024-09-20 22:43:18.234404654 +0000 UTC m=+0.594293988 volume create 0ed528b373d9ea1f04e053aaada923b3cb6fd1bd18b53b4878c12a3f3e749132 2024-09-20T22:43:18.493 INFO:journalctl@ceph.alertmanager.a.smithi084.stdout:Sep 20 22:43:18 smithi084 podman[80540]: 2024-09-20 22:43:18.335086732 +0000 UTC m=+0.694976065 container create 355d6bdc525d4efb9427e6bd14cfbfc93cf60304576dd4d8b6088a6fb50b0281 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-20T22:43:18.493 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:18 smithi084 ceph-mon[40024]: osdmap e118: 8 total, 8 up, 8 in 2024-09-20T22:43:18.493 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:18 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-09-20T22:43:18.493 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:18 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-09-20T22:43:18.493 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:18 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]': finished 2024-09-20T22:43:18.493 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:18 smithi084 ceph-mon[40024]: osdmap e119: 8 total, 8 up, 8 in 2024-09-20T22:43:18.494 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:18 smithi084 ceph-mon[29774]: osdmap e118: 8 total, 8 up, 8 in 2024-09-20T22:43:18.494 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:18 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-09-20T22:43:18.494 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:18 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-09-20T22:43:18.494 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:18 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]': finished 2024-09-20T22:43:18.494 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:18 smithi084 ceph-mon[29774]: osdmap e119: 8 total, 8 up, 8 in 2024-09-20T22:43:18.494 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:43:18 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: ::ffff:172.21.15.154 - - [20/Sep/2024:22:43:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-20T22:43:18.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:18 smithi154 ceph-mon[33264]: osdmap e118: 8 total, 8 up, 8 in 2024-09-20T22:43:18.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:18 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-09-20T22:43:18.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:18 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-09-20T22:43:18.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:18 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "21"}]': finished 2024-09-20T22:43:18.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:18 smithi154 ceph-mon[33264]: osdmap e119: 8 total, 8 up, 8 in 2024-09-20T22:43:18.516 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:43:18.542 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:18.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:18.543 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:43:18.644 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:43:18.666 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:18.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:18.667 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:43:18.767 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:43:18.789 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:18.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:18.790 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:43:18.891 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:43:18.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:18.919 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:18.919 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:43:19.019 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:43:19.042 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:19.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:19.044 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:43:19.063 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-20T22:43:19.107 INFO:journalctl@ceph.alertmanager.a.smithi084.stdout:Sep 20 22:43:18 smithi084 podman[80540]: 2024-09-20 22:43:18.849199447 +0000 UTC m=+1.209088781 container init 355d6bdc525d4efb9427e6bd14cfbfc93cf60304576dd4d8b6088a6fb50b0281 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-20T22:43:19.107 INFO:journalctl@ceph.alertmanager.a.smithi084.stdout:Sep 20 22:43:18 smithi084 podman[80540]: 2024-09-20 22:43:18.852208557 +0000 UTC m=+1.212097891 container start 355d6bdc525d4efb9427e6bd14cfbfc93cf60304576dd4d8b6088a6fb50b0281 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-20T22:43:19.107 INFO:journalctl@ceph.alertmanager.a.smithi084.stdout:Sep 20 22:43:18 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-alertmanager-a[80657]: ts=2024-09-20T22:43:18.864Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2024-09-20T22:43:19.107 INFO:journalctl@ceph.alertmanager.a.smithi084.stdout:Sep 20 22:43:18 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-alertmanager-a[80657]: ts=2024-09-20T22:43:18.864Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2024-09-20T22:43:19.107 INFO:journalctl@ceph.alertmanager.a.smithi084.stdout:Sep 20 22:43:18 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-alertmanager-a[80657]: ts=2024-09-20T22:43:18.865Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.84 port=9094 2024-09-20T22:43:19.107 INFO:journalctl@ceph.alertmanager.a.smithi084.stdout:Sep 20 22:43:18 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-alertmanager-a[80657]: ts=2024-09-20T22:43:18.866Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-09-20T22:43:19.108 INFO:journalctl@ceph.alertmanager.a.smithi084.stdout:Sep 20 22:43:18 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-alertmanager-a[80657]: ts=2024-09-20T22:43:18.888Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-09-20T22:43:19.108 INFO:journalctl@ceph.alertmanager.a.smithi084.stdout:Sep 20 22:43:18 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-alertmanager-a[80657]: ts=2024-09-20T22:43:18.889Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-09-20T22:43:19.108 INFO:journalctl@ceph.alertmanager.a.smithi084.stdout:Sep 20 22:43:18 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-alertmanager-a[80657]: ts=2024-09-20T22:43:18.891Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2024-09-20T22:43:19.108 INFO:journalctl@ceph.alertmanager.a.smithi084.stdout:Sep 20 22:43:18 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-alertmanager-a[80657]: ts=2024-09-20T22:43:18.891Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-09-20T22:43:19.108 INFO:journalctl@ceph.alertmanager.a.smithi084.stdout:Sep 20 22:43:19 smithi084 bash[80540]: 355d6bdc525d4efb9427e6bd14cfbfc93cf60304576dd4d8b6088a6fb50b0281 2024-09-20T22:43:19.108 INFO:journalctl@ceph.alertmanager.a.smithi084.stdout:Sep 20 22:43:19 smithi084 systemd[1]: Started Ceph alertmanager.a for a9c0bc70-77a0-11ef-baf5-efdc52797490. 2024-09-20T22:43:19.111 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is set 2024-09-20T22:43:19.123 INFO:tasks.rados.rados.0.smithi084.stdout:159: done (3 left) 2024-09-20T22:43:19.123 INFO:tasks.rados.rados.0.smithi084.stdout:160: done (2 left) 2024-09-20T22:43:19.123 INFO:tasks.rados.rados.0.smithi084.stdout:162: done (1 left) 2024-09-20T22:43:19.123 INFO:tasks.rados.rados.0.smithi084.stdout:163: done (0 left) 2024-09-20T22:43:19.123 INFO:tasks.rados.rados.0.smithi084.stdout:164: snap_create 2024-09-20T22:43:19.145 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:43:19.170 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:19.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:19.171 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:43:19.254 INFO:journalctl@ceph.iscsi.iscsi.a.smithi154.stdout:Sep 20 22:43:18 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-iscsi-iscsi-a[59730]: debug there is no tcmu-runner data available 2024-09-20T22:43:19.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:19 smithi154 ceph-mon[33264]: pgmap v79: 158 pgs: 1 active+clean+snaptrim, 1 active+clean+scrubbing, 156 active+clean; 114 MiB data, 464 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:43:19.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:19 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/3999643760' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:43:19.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:19 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:43:19.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:19 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:43:19.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:19 smithi154 ceph-mon[33264]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:43:19.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:19 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/3999643760' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T22:43:19.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:19 smithi154 ceph-mon[33264]: osdmap e120: 8 total, 8 up, 8 in 2024-09-20T22:43:19.271 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:43:19.295 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:19.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:19.296 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:43:19.300 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-20T22:43:19.300 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:43:19.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:19 smithi084 ceph-mon[29774]: pgmap v79: 158 pgs: 1 active+clean+snaptrim, 1 active+clean+scrubbing, 156 active+clean; 114 MiB data, 464 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:43:19.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:19 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/3999643760' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:43:19.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:19 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:43:19.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:19 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:43:19.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:19 smithi084 ceph-mon[29774]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:43:19.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:19 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/3999643760' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T22:43:19.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:19 smithi084 ceph-mon[29774]: osdmap e120: 8 total, 8 up, 8 in 2024-09-20T22:43:19.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:19 smithi084 ceph-mon[40024]: pgmap v79: 158 pgs: 1 active+clean+snaptrim, 1 active+clean+scrubbing, 156 active+clean; 114 MiB data, 464 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:43:19.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:19 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/3999643760' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:43:19.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:19 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:43:19.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:19 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:43:19.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:19 smithi084 ceph-mon[40024]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:43:19.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:19 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/3999643760' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T22:43:19.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:19 smithi084 ceph-mon[40024]: osdmap e120: 8 total, 8 up, 8 in 2024-09-20T22:43:19.396 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:43:19.418 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:19.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:19.419 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:43:19.520 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:43:19.541 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:19.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:19.542 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:43:19.643 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:43:19.664 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:19.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:19.665 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:43:19.766 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:43:19.789 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:19.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:19.790 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:43:19.890 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:43:19.912 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:19.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:19.913 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:43:20.014 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:43:20.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:20.036 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:20.036 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:43:20.121 INFO:tasks.rados.rados.0.smithi084.stdout:164: done (0 left) 2024-09-20T22:43:20.121 INFO:tasks.rados.rados.0.smithi084.stdout:165: write (excl) oid 5 current snap is 13 2024-09-20T22:43:20.121 INFO:tasks.rados.rados.0.smithi084.stdout:165: seq_num 88 ranges {679016=701795,1866533=496024,2900635=402643} 2024-09-20T22:43:20.128 INFO:tasks.rados.rados.0.smithi084.stdout:165: writing smithi08477203-5 from 679016 to 1380811 tid 1 2024-09-20T22:43:20.135 INFO:tasks.rados.rados.0.smithi084.stdout:165: writing smithi08477203-5 from 1866533 to 2362557 tid 2 2024-09-20T22:43:20.137 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:43:20.141 INFO:tasks.rados.rados.0.smithi084.stdout:165: writing smithi08477203-5 from 2900635 to 3303278 tid 3 2024-09-20T22:43:20.141 INFO:tasks.rados.rados.0.smithi084.stdout:166: read oid 22 snap -1 2024-09-20T22:43:20.141 INFO:tasks.rados.rados.0.smithi084.stdout:166: expect (ObjNum 85 snap 11 seq_num 85) 2024-09-20T22:43:20.141 INFO:tasks.rados.rados.0.smithi084.stdout:167: snap_remove snap 10 2024-09-20T22:43:20.161 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:20.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:20.162 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:43:20.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:20 smithi154 ceph-mon[33264]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:43:20.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:20 smithi154 ceph-mon[33264]: purged_snaps scrub starts 2024-09-20T22:43:20.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:20 smithi154 ceph-mon[33264]: purged_snaps scrub ok 2024-09-20T22:43:20.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:20 smithi154 ceph-mon[33264]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-20T22:43:20.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:20 smithi154 ceph-mon[33264]: Reconfiguring daemon prometheus.a on smithi154 2024-09-20T22:43:20.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:20 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]: dispatch 2024-09-20T22:43:20.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:20 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]: dispatch 2024-09-20T22:43:20.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:20 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]': finished 2024-09-20T22:43:20.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:20 smithi154 ceph-mon[33264]: osdmap e121: 8 total, 8 up, 8 in 2024-09-20T22:43:20.263 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:43:20.284 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:20.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:20.285 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:43:20.386 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:43:20.408 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:20.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:20.409 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:43:20.510 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:43:20.532 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:20.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:20.533 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:43:20.614 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:20 smithi084 ceph-mon[40024]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:43:20.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:20 smithi084 ceph-mon[40024]: purged_snaps scrub starts 2024-09-20T22:43:20.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:20 smithi084 ceph-mon[40024]: purged_snaps scrub ok 2024-09-20T22:43:20.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:20 smithi084 ceph-mon[40024]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-20T22:43:20.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:20 smithi084 ceph-mon[40024]: Reconfiguring daemon prometheus.a on smithi154 2024-09-20T22:43:20.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:20 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]: dispatch 2024-09-20T22:43:20.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:20 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]: dispatch 2024-09-20T22:43:20.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:20 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]': finished 2024-09-20T22:43:20.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:20 smithi084 ceph-mon[40024]: osdmap e121: 8 total, 8 up, 8 in 2024-09-20T22:43:20.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:20 smithi084 ceph-mon[29774]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:43:20.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:20 smithi084 ceph-mon[29774]: purged_snaps scrub starts 2024-09-20T22:43:20.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:20 smithi084 ceph-mon[29774]: purged_snaps scrub ok 2024-09-20T22:43:20.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:20 smithi084 ceph-mon[29774]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-20T22:43:20.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:20 smithi084 ceph-mon[29774]: Reconfiguring daemon prometheus.a on smithi154 2024-09-20T22:43:20.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:20 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]: dispatch 2024-09-20T22:43:20.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:20 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]: dispatch 2024-09-20T22:43:20.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:20 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "21"}]': finished 2024-09-20T22:43:20.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:20 smithi084 ceph-mon[29774]: osdmap e121: 8 total, 8 up, 8 in 2024-09-20T22:43:20.633 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:43:20.655 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:20.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:20.656 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:43:20.704 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:43:20.731 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.3 dump_blocked_ops 2024-09-20T22:43:20.757 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:43:20.781 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:20.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:20.782 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:43:20.882 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:43:20.905 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:20.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:20.906 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:43:20.954 INFO:journalctl@ceph.alertmanager.a.smithi084.stdout:Sep 20 22:43:20 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-alertmanager-a[80657]: ts=2024-09-20T22:43:20.866Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000255398s 2024-09-20T22:43:21.006 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:43:21.027 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:21.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:21.028 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:43:21.129 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:43:21.130 INFO:tasks.rados.rados.0.smithi084.stdout:167: done (2 left) 2024-09-20T22:43:21.130 INFO:tasks.rados.rados.0.smithi084.stdout:168: copy_from oid 47 from oid 43 current snap is 13 2024-09-20T22:43:21.130 INFO:tasks.rados.rados.0.smithi084.stdout:169: snap_create 2024-09-20T22:43:21.130 INFO:tasks.rados.rados.0.smithi084.stdout:165: finishing write tid 1 to smithi08477203-5 2024-09-20T22:43:21.130 INFO:tasks.rados.rados.0.smithi084.stdout:165: finishing write tid 2 to smithi08477203-5 2024-09-20T22:43:21.130 INFO:tasks.rados.rados.0.smithi084.stdout:165: finishing write tid 3 to smithi08477203-5 2024-09-20T22:43:21.130 INFO:tasks.rados.rados.0.smithi084.stdout:165: finishing write tid 4 to smithi08477203-5 2024-09-20T22:43:21.130 INFO:tasks.rados.rados.0.smithi084.stdout:165: finishing write tid 5 to smithi08477203-5 2024-09-20T22:43:21.131 INFO:tasks.rados.rados.0.smithi084.stdout:165: oid 5 updating version 0 to 17 2024-09-20T22:43:21.131 INFO:tasks.rados.rados.0.smithi084.stdout:165: oid 5 version 17 is already newer than 16 2024-09-20T22:43:21.131 INFO:tasks.rados.rados.0.smithi084.stdout:165: oid 5 updating version 17 to 18 2024-09-20T22:43:21.131 INFO:tasks.rados.rados.0.smithi084.stdout:165: oid 5 version 18 is already newer than 15 2024-09-20T22:43:21.131 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 5 v 18 (ObjNum 88 snap 13 seq_num 88) dirty exists 2024-09-20T22:43:21.131 INFO:tasks.rados.rados.0.smithi084.stdout:165: left oid 5 (ObjNum 88 snap 13 seq_num 88) 2024-09-20T22:43:21.131 INFO:tasks.rados.rados.0.smithi084.stdout:166: expect (ObjNum 85 snap 11 seq_num 85) 2024-09-20T22:43:21.153 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:21.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:21.154 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:43:21.219 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:21 smithi084 ceph-mon[29774]: pgmap v82: 168 pgs: 8 peering, 10 unknown, 3 active+clean+snaptrim, 147 active+clean; 118 MiB data, 464 MiB used, 715 GiB / 715 GiB avail; 11 MiB/s rd, 4.2 MiB/s wr, 29 op/s 2024-09-20T22:43:21.219 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:21 smithi084 ceph-mon[29774]: osdmap e122: 8 total, 8 up, 8 in 2024-09-20T22:43:21.219 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:21 smithi084 ceph-mon[40024]: pgmap v82: 168 pgs: 8 peering, 10 unknown, 3 active+clean+snaptrim, 147 active+clean; 118 MiB data, 464 MiB used, 715 GiB / 715 GiB avail; 11 MiB/s rd, 4.2 MiB/s wr, 29 op/s 2024-09-20T22:43:21.219 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:21 smithi084 ceph-mon[40024]: osdmap e122: 8 total, 8 up, 8 in 2024-09-20T22:43:21.222 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:43:21.222 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:43:21.249 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:43:21.250 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:43:21.255 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:43:21.264 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:43:21.268 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:43:21.269 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:43:21.276 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:21.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:21.277 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:43:21.286 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-20T22:43:21.286 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T22:43:21.304 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:43:21.304 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:43:21.320 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:43:21.320 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:43:21.336 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:43:21.337 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:43:21.339 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:21 smithi154 ceph-mon[33264]: pgmap v82: 168 pgs: 8 peering, 10 unknown, 3 active+clean+snaptrim, 147 active+clean; 118 MiB data, 464 MiB used, 715 GiB / 715 GiB avail; 11 MiB/s rd, 4.2 MiB/s wr, 29 op/s 2024-09-20T22:43:21.339 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:21 smithi154 ceph-mon[33264]: osdmap e122: 8 total, 8 up, 8 in 2024-09-20T22:43:21.353 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:43:21.353 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:43:21.357 INFO:tasks.rados.rados.0.smithi084.stdout:168: finishing copy_from racing read to smithi08477203-47 2024-09-20T22:43:21.357 INFO:tasks.rados.rados.0.smithi084.stdout:168: finishing copy_from to smithi08477203-47 2024-09-20T22:43:21.357 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 47 v 6 (ObjNum 42 snap 0 seq_num 42) dirty exists 2024-09-20T22:43:21.378 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:43:21.400 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:21.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:21.401 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:43:21.501 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:43:21.523 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:21.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:21.524 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:43:21.625 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:43:21.646 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:21.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:21.647 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:43:21.748 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:43:21.769 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:21.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:21.770 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:43:21.870 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:43:21.891 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:21.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:21.892 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:43:21.993 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:43:22.015 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:22.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:22.016 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:43:22.117 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:43:22.123 INFO:tasks.rados.rados.0.smithi084.stdout:165: done (3 left) 2024-09-20T22:43:22.124 INFO:tasks.rados.rados.0.smithi084.stdout:166: done (2 left) 2024-09-20T22:43:22.124 INFO:tasks.rados.rados.0.smithi084.stdout:168: done (1 left) 2024-09-20T22:43:22.124 INFO:tasks.rados.rados.0.smithi084.stdout:169: done (0 left) 2024-09-20T22:43:22.124 INFO:tasks.rados.rados.0.smithi084.stdout:170: write (excl) oid 42 current snap is 14 2024-09-20T22:43:22.124 INFO:tasks.rados.rados.0.smithi084.stdout:170: seq_num 89 ranges {695823=777044,1925015=639682,3328058=167765} 2024-09-20T22:43:22.133 INFO:tasks.rados.rados.0.smithi084.stdout:170: writing smithi08477203-42 from 695823 to 1472867 tid 1 2024-09-20T22:43:22.139 INFO:tasks.rados.rados.0.smithi084.stdout:170: writing smithi08477203-42 from 1925015 to 2564697 tid 2 2024-09-20T22:43:22.145 INFO:tasks.rados.rados.0.smithi084.stdout:170: writing smithi08477203-42 from 3328058 to 3495823 tid 3 2024-09-20T22:43:22.145 INFO:tasks.rados.rados.0.smithi084.stdout:171: copy_from oid 6 from oid 31 current snap is 14 2024-09-20T22:43:22.145 INFO:tasks.rados.rados.0.smithi084.stdout:170: finishing write tid 1 to smithi08477203-42 2024-09-20T22:43:22.145 INFO:tasks.rados.rados.0.smithi084.stdout:170: finishing write tid 2 to smithi08477203-42 2024-09-20T22:43:22.145 INFO:tasks.rados.rados.0.smithi084.stdout:172: snap_remove snap 12 2024-09-20T22:43:22.145 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:22.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:22.146 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:43:22.177 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd unset noscrub 2024-09-20T22:43:22.247 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:43:22.255 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-20T22:43:22.255 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2024-09-20T22:43:22.255 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2024-09-20T22:43:22.255 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2024-09-20T22:43:22.256 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-20T22:43:22.269 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:22.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:22.270 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:43:22.298 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:22 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/3211322932' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:43:22.299 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:22 smithi084 ceph-mon[40024]: osdmap e123: 8 total, 8 up, 8 in 2024-09-20T22:43:22.299 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:22 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/3211322932' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:43:22.299 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:22 smithi084 ceph-mon[29774]: osdmap e123: 8 total, 8 up, 8 in 2024-09-20T22:43:22.370 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:43:22.391 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:22.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:22.392 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:43:22.404 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:22 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/3211322932' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:43:22.404 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:22 smithi154 ceph-mon[33264]: osdmap e123: 8 total, 8 up, 8 in 2024-09-20T22:43:22.493 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:43:22.515 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:22.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:22.516 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:43:22.616 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-20T22:43:22.616 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-20T22:43:22.617 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-20T22:43:22.617 INFO:tasks.thrashosds.thrasher:Setting osd 1 primary_affinity to 1.000000 2024-09-20T22:43:22.617 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd primary-affinity 1 1 2024-09-20T22:43:22.617 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:43:22.640 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:22.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:22.641 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:43:22.741 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:43:22.754 INFO:journalctl@ceph.prometheus.a.smithi154.stdout:Sep 20 22:43:22 smithi154 systemd[1]: Stopping Ceph prometheus.a for a9c0bc70-77a0-11ef-baf5-efdc52797490... 2024-09-20T22:43:22.764 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:22.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:22.765 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:43:22.865 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:43:22.886 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:22.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:22.887 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:43:22.988 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:43:22.996 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.3 dump_historic_ops 2024-09-20T22:43:23.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:23.010 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:23.011 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:43:23.051 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:43:23.076 INFO:journalctl@ceph.prometheus.a.smithi154.stdout:Sep 20 22:43:22 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-prometheus-a[60633]: ts=2024-09-20T22:43:22.775Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-09-20T22:43:23.076 INFO:journalctl@ceph.prometheus.a.smithi154.stdout:Sep 20 22:43:22 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-prometheus-a[60633]: ts=2024-09-20T22:43:22.776Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-09-20T22:43:23.076 INFO:journalctl@ceph.prometheus.a.smithi154.stdout:Sep 20 22:43:22 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-prometheus-a[60633]: ts=2024-09-20T22:43:22.776Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-09-20T22:43:23.076 INFO:journalctl@ceph.prometheus.a.smithi154.stdout:Sep 20 22:43:22 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-prometheus-a[60633]: ts=2024-09-20T22:43:22.776Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-09-20T22:43:23.076 INFO:journalctl@ceph.prometheus.a.smithi154.stdout:Sep 20 22:43:22 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-prometheus-a[60633]: ts=2024-09-20T22:43:22.776Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-09-20T22:43:23.076 INFO:journalctl@ceph.prometheus.a.smithi154.stdout:Sep 20 22:43:22 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-prometheus-a[60633]: ts=2024-09-20T22:43:22.776Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-09-20T22:43:23.076 INFO:journalctl@ceph.prometheus.a.smithi154.stdout:Sep 20 22:43:22 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-prometheus-a[60633]: ts=2024-09-20T22:43:22.776Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-09-20T22:43:23.076 INFO:journalctl@ceph.prometheus.a.smithi154.stdout:Sep 20 22:43:22 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-prometheus-a[60633]: ts=2024-09-20T22:43:22.776Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-09-20T22:43:23.077 INFO:journalctl@ceph.prometheus.a.smithi154.stdout:Sep 20 22:43:22 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-prometheus-a[60633]: ts=2024-09-20T22:43:22.776Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-09-20T22:43:23.077 INFO:journalctl@ceph.prometheus.a.smithi154.stdout:Sep 20 22:43:22 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-prometheus-a[60633]: ts=2024-09-20T22:43:22.776Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-09-20T22:43:23.077 INFO:journalctl@ceph.prometheus.a.smithi154.stdout:Sep 20 22:43:22 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-prometheus-a[60633]: ts=2024-09-20T22:43:22.776Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-09-20T22:43:23.077 INFO:journalctl@ceph.prometheus.a.smithi154.stdout:Sep 20 22:43:22 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-prometheus-a[60633]: ts=2024-09-20T22:43:22.776Z caller=main.go:1147 level=info msg="See you next time!" 2024-09-20T22:43:23.077 INFO:journalctl@ceph.prometheus.a.smithi154.stdout:Sep 20 22:43:22 smithi154 podman[66934]: 2024-09-20 22:43:22.785974372 +0000 UTC m=+0.117916206 container died 19b4ae16e0bb92baebbe2bd7cf4786b7d7635d14bec4196ae1f63b5c04734830 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-20T22:43:23.111 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:43:23.128 INFO:tasks.rados.rados.0.smithi084.stdout:172: done (2 left) 2024-09-20T22:43:23.128 INFO:tasks.rados.rados.0.smithi084.stdout:173: copy_from oid 15 from oid 7 current snap is 14 2024-09-20T22:43:23.129 INFO:tasks.rados.rados.0.smithi084.stdout:174: delete oid 46 current snap is 14 2024-09-20T22:43:23.129 INFO:tasks.rados.rados.0.smithi084.stdout:171: finishing copy_from to smithi08477203-6 2024-09-20T22:43:23.129 INFO:tasks.rados.rados.0.smithi084.stdout:171: got expected ENOENT (src dne) 2024-09-20T22:43:23.129 INFO:tasks.rados.rados.0.smithi084.stdout:170: finishing write tid 3 to smithi08477203-42 2024-09-20T22:43:23.129 INFO:tasks.rados.rados.0.smithi084.stdout:171: finishing copy_from racing read to smithi08477203-6 2024-09-20T22:43:23.129 INFO:tasks.rados.rados.0.smithi084.stdout:170: finishing write tid 4 to smithi08477203-42 2024-09-20T22:43:23.129 INFO:tasks.rados.rados.0.smithi084.stdout:170: finishing write tid 5 to smithi08477203-42 2024-09-20T22:43:23.129 INFO:tasks.rados.rados.0.smithi084.stdout:170: oid 42 updating version 0 to 16 2024-09-20T22:43:23.129 INFO:tasks.rados.rados.0.smithi084.stdout:170: oid 42 updating version 16 to 17 2024-09-20T22:43:23.129 INFO:tasks.rados.rados.0.smithi084.stdout:170: oid 42 updating version 17 to 19 2024-09-20T22:43:23.129 INFO:tasks.rados.rados.0.smithi084.stdout:170: oid 42 version 19 is already newer than 18 2024-09-20T22:43:23.129 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 42 v 19 (ObjNum 89 snap 14 seq_num 89) dirty exists 2024-09-20T22:43:23.129 INFO:tasks.rados.rados.0.smithi084.stdout:170: left oid 42 (ObjNum 89 snap 14 seq_num 89) 2024-09-20T22:43:23.133 INFO:tasks.rados.rados.0.smithi084.stdout:170: done (3 left) 2024-09-20T22:43:23.133 INFO:tasks.rados.rados.0.smithi084.stdout:171: done (2 left) 2024-09-20T22:43:23.133 INFO:tasks.rados.rados.0.smithi084.stdout:174: done (1 left) 2024-09-20T22:43:23.134 INFO:tasks.rados.rados.0.smithi084.stdout:175: rollback oid 19 current snap is 14 2024-09-20T22:43:23.134 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 19 to 11 2024-09-20T22:43:23.134 INFO:tasks.rados.rados.0.smithi084.stdout:176: delete oid 11 current snap is 14 2024-09-20T22:43:23.134 INFO:tasks.rados.rados.0.smithi084.stdout:175: finishing rollback tid 1 to smithi08477203-19 2024-09-20T22:43:23.134 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 19 v 0 (ObjNum 129 snap 0 seq_num 2088622320) dirty dne 2024-09-20T22:43:23.136 INFO:tasks.rados.rados.0.smithi084.stdout:175: done (2 left) 2024-09-20T22:43:23.136 INFO:tasks.rados.rados.0.smithi084.stdout:176: done (1 left) 2024-09-20T22:43:23.136 INFO:tasks.rados.rados.0.smithi084.stdout:177: snap_create 2024-09-20T22:43:23.143 INFO:tasks.rados.rados.0.smithi084.stdout:173: finishing copy_from racing read to smithi08477203-15 2024-09-20T22:43:23.149 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:23.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:23.150 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:43:23.154 INFO:tasks.rados.rados.0.smithi084.stdout:173: finishing copy_from to smithi08477203-15 2024-09-20T22:43:23.154 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 15 v 28 (ObjNum 64 snap 5 seq_num 64) dirty exists 2024-09-20T22:43:23.219 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:23 smithi084 ceph-mon[40024]: pgmap v85: 168 pgs: 8 peering, 10 unknown, 3 active+clean+snaptrim, 147 active+clean; 118 MiB data, 464 MiB used, 715 GiB / 715 GiB avail; 11 MiB/s rd, 4.2 MiB/s wr, 29 op/s 2024-09-20T22:43:23.220 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:23 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:43:23.220 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:23 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:43:23.220 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:23 smithi084 ceph-mon[40024]: osdmap e124: 8 total, 8 up, 8 in 2024-09-20T22:43:23.220 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:23 smithi084 ceph-mon[29774]: pgmap v85: 168 pgs: 8 peering, 10 unknown, 3 active+clean+snaptrim, 147 active+clean; 118 MiB data, 464 MiB used, 715 GiB / 715 GiB avail; 11 MiB/s rd, 4.2 MiB/s wr, 29 op/s 2024-09-20T22:43:23.220 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:23 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:43:23.220 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:23 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:43:23.220 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:23 smithi084 ceph-mon[29774]: osdmap e124: 8 total, 8 up, 8 in 2024-09-20T22:43:23.251 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:43:23.272 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:23.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:23.273 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:43:23.374 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:43:23.395 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:23.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:23.396 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:43:23.443 INFO:journalctl@ceph.prometheus.a.smithi154.stdout:Sep 20 22:43:23 smithi154 podman[66934]: 2024-09-20 22:43:23.182622835 +0000 UTC m=+0.514564673 container cleanup 19b4ae16e0bb92baebbe2bd7cf4786b7d7635d14bec4196ae1f63b5c04734830 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-20T22:43:23.443 INFO:journalctl@ceph.prometheus.a.smithi154.stdout:Sep 20 22:43:23 smithi154 bash[66934]: ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-prometheus-a 2024-09-20T22:43:23.444 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:23 smithi154 ceph-mon[33264]: pgmap v85: 168 pgs: 8 peering, 10 unknown, 3 active+clean+snaptrim, 147 active+clean; 118 MiB data, 464 MiB used, 715 GiB / 715 GiB avail; 11 MiB/s rd, 4.2 MiB/s wr, 29 op/s 2024-09-20T22:43:23.444 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:23 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:43:23.444 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:23 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:43:23.444 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:23 smithi154 ceph-mon[33264]: osdmap e124: 8 total, 8 up, 8 in 2024-09-20T22:43:23.497 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:43:23.520 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:23.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:23.521 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:43:23.537 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:43:23.622 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:43:23.643 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:23.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:23.644 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:43:23.745 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:43:23.754 INFO:journalctl@ceph.prometheus.a.smithi154.stdout:Sep 20 22:43:23 smithi154 podman[66964]: 2024-09-20 22:43:23.442408949 +0000 UTC m=+0.657323235 container remove 19b4ae16e0bb92baebbe2bd7cf4786b7d7635d14bec4196ae1f63b5c04734830 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-20T22:43:23.754 INFO:journalctl@ceph.prometheus.a.smithi154.stdout:Sep 20 22:43:23 smithi154 systemd[1]: ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490@prometheus.a.service: Deactivated successfully. 2024-09-20T22:43:23.754 INFO:journalctl@ceph.prometheus.a.smithi154.stdout:Sep 20 22:43:23 smithi154 systemd[1]: Stopped Ceph prometheus.a for a9c0bc70-77a0-11ef-baf5-efdc52797490. 2024-09-20T22:43:23.754 INFO:journalctl@ceph.prometheus.a.smithi154.stdout:Sep 20 22:43:23 smithi154 systemd[1]: ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490@prometheus.a.service: Consumed 1.060s CPU time. 2024-09-20T22:43:23.754 INFO:journalctl@ceph.prometheus.a.smithi154.stdout:Sep 20 22:43:23 smithi154 systemd[1]: Starting Ceph prometheus.a for a9c0bc70-77a0-11ef-baf5-efdc52797490... 2024-09-20T22:43:23.756 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:43:23.767 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:23.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:23.768 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:43:23.868 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:43:23.889 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:23.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:23.891 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:43:23.992 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:43:24.013 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:24.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:24.014 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:43:24.072 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-20T22:43:24.113 INFO:journalctl@ceph.prometheus.a.smithi154.stdout:Sep 20 22:43:23 smithi154 podman[67092]: 2024-09-20 22:43:23.722454825 +0000 UTC m=+0.016953706 image pull a07b618ecd1dce142bce4c52f0e80982eaf1f14265a2415c2d35978ccaa0a464 quay.io/prometheus/prometheus:v2.43.0 2024-09-20T22:43:24.113 INFO:journalctl@ceph.prometheus.a.smithi154.stdout:Sep 20 22:43:23 smithi154 podman[67092]: 2024-09-20 22:43:23.868465186 +0000 UTC m=+0.162964097 container create 5c4f4836aceb1176504ae6451e7c68b4f7ea65a6eb77cc448425fb930ada26fc (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-20T22:43:24.113 INFO:journalctl@ceph.prometheus.a.smithi154.stdout:Sep 20 22:43:24 smithi154 podman[67092]: 2024-09-20 22:43:24.112458842 +0000 UTC m=+0.406957710 container init 5c4f4836aceb1176504ae6451e7c68b4f7ea65a6eb77cc448425fb930ada26fc (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-20T22:43:24.114 INFO:teuthology.orchestra.run.smithi084.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-09-20T22:43:24.114 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:43:24.120 INFO:tasks.rados.rados.0.smithi084.stdout:173: done (1 left) 2024-09-20T22:43:24.120 INFO:tasks.rados.rados.0.smithi084.stdout:177: done (0 left) 2024-09-20T22:43:24.120 INFO:tasks.rados.rados.0.smithi084.stdout:178: write (excl) oid 12 current snap is 15 2024-09-20T22:43:24.120 INFO:tasks.rados.rados.0.smithi084.stdout:178: seq_num 92 ranges {746244=719144,2100461=503362,3126680=419564} 2024-09-20T22:43:24.131 INFO:tasks.rados.rados.0.smithi084.stdout:178: writing smithi08477203-12 from 746244 to 1465388 tid 1 2024-09-20T22:43:24.138 INFO:tasks.rados.rados.0.smithi084.stdout:178: writing smithi08477203-12 from 2100461 to 2603823 tid 2 2024-09-20T22:43:24.144 INFO:tasks.rados.rados.0.smithi084.stdout:178: writing smithi08477203-12 from 3126680 to 3546244 tid 3 2024-09-20T22:43:24.144 INFO:tasks.rados.rados.0.smithi084.stdout:179: snap_create 2024-09-20T22:43:24.144 INFO:tasks.rados.rados.0.smithi084.stdout:178: finishing write tid 1 to smithi08477203-12 2024-09-20T22:43:24.145 INFO:tasks.rados.rados.0.smithi084.stdout:178: finishing write tid 2 to smithi08477203-12 2024-09-20T22:43:24.147 INFO:tasks.rados.rados.0.smithi084.stdout:178: finishing write tid 3 to smithi08477203-12 2024-09-20T22:43:24.147 INFO:tasks.rados.rados.0.smithi084.stdout:178: finishing write tid 4 to smithi08477203-12 2024-09-20T22:43:24.147 INFO:tasks.rados.rados.0.smithi084.stdout:178: finishing write tid 5 to smithi08477203-12 2024-09-20T22:43:24.147 INFO:tasks.rados.rados.0.smithi084.stdout:178: oid 12 updating version 0 to 81 2024-09-20T22:43:24.147 INFO:tasks.rados.rados.0.smithi084.stdout:178: oid 12 version 81 is already newer than 80 2024-09-20T22:43:24.147 INFO:tasks.rados.rados.0.smithi084.stdout:178: oid 12 version 81 is already newer than 79 2024-09-20T22:43:24.148 INFO:tasks.rados.rados.0.smithi084.stdout:178: oid 12 updating version 81 to 82 2024-09-20T22:43:24.148 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 12 v 82 (ObjNum 92 snap 15 seq_num 92) dirty exists 2024-09-20T22:43:24.148 INFO:tasks.rados.rados.0.smithi084.stdout:178: left oid 12 (ObjNum 92 snap 15 seq_num 92) 2024-09-20T22:43:24.150 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:24.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:24.151 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:43:24.252 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:43:24.277 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:24.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:24.279 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:43:24.379 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:43:24.386 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:24 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/4131921447' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-09-20T22:43:24.386 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:24 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/4131921447' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-09-20T22:43:24.387 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:24 smithi084 ceph-mon[29774]: osdmap e125: 8 total, 8 up, 8 in 2024-09-20T22:43:24.387 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:43:24 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: [20/Sep/2024:22:43:24] ENGINE Bus STOPPING 2024-09-20T22:43:24.387 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:43:24 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: [20/Sep/2024:22:43:24] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-20T22:43:24.387 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:43:24 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: [20/Sep/2024:22:43:24] ENGINE Bus STOPPED 2024-09-20T22:43:24.387 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:43:24 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: [20/Sep/2024:22:43:24] ENGINE Bus STARTING 2024-09-20T22:43:24.387 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:43:24 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: [20/Sep/2024:22:43:24] ENGINE Serving on http://:::9283 2024-09-20T22:43:24.387 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:43:24 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: [20/Sep/2024:22:43:24] ENGINE Bus STARTED 2024-09-20T22:43:24.387 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:24 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/4131921447' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-09-20T22:43:24.387 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:24 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/4131921447' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-09-20T22:43:24.388 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:24 smithi084 ceph-mon[40024]: osdmap e125: 8 total, 8 up, 8 in 2024-09-20T22:43:24.400 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:24.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:24.401 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:43:24.484 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:24 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/4131921447' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-09-20T22:43:24.485 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:24 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/4131921447' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-09-20T22:43:24.485 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:24 smithi154 ceph-mon[33264]: osdmap e125: 8 total, 8 up, 8 in 2024-09-20T22:43:24.485 INFO:journalctl@ceph.prometheus.a.smithi154.stdout:Sep 20 22:43:24 smithi154 podman[67092]: 2024-09-20 22:43:24.116244452 +0000 UTC m=+0.410743323 container start 5c4f4836aceb1176504ae6451e7c68b4f7ea65a6eb77cc448425fb930ada26fc (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-20T22:43:24.485 INFO:journalctl@ceph.prometheus.a.smithi154.stdout:Sep 20 22:43:24 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-prometheus-a[67141]: ts=2024-09-20T22:43:24.147Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-09-20T22:43:24.486 INFO:journalctl@ceph.prometheus.a.smithi154.stdout:Sep 20 22:43:24 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-prometheus-a[67141]: ts=2024-09-20T22:43:24.147Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-09-20T22:43:24.486 INFO:journalctl@ceph.prometheus.a.smithi154.stdout:Sep 20 22:43:24 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-prometheus-a[67141]: ts=2024-09-20T22:43:24.147Z caller=main.go:570 level=info host_details="(Linux 5.14.0-508.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024 x86_64 smithi154 (none))" 2024-09-20T22:43:24.486 INFO:journalctl@ceph.prometheus.a.smithi154.stdout:Sep 20 22:43:24 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-prometheus-a[67141]: ts=2024-09-20T22:43:24.147Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-09-20T22:43:24.486 INFO:journalctl@ceph.prometheus.a.smithi154.stdout:Sep 20 22:43:24 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-prometheus-a[67141]: ts=2024-09-20T22:43:24.147Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-09-20T22:43:24.486 INFO:journalctl@ceph.prometheus.a.smithi154.stdout:Sep 20 22:43:24 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-prometheus-a[67141]: ts=2024-09-20T22:43:24.149Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-09-20T22:43:24.486 INFO:journalctl@ceph.prometheus.a.smithi154.stdout:Sep 20 22:43:24 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-prometheus-a[67141]: ts=2024-09-20T22:43:24.149Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-09-20T22:43:24.486 INFO:journalctl@ceph.prometheus.a.smithi154.stdout:Sep 20 22:43:24 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-prometheus-a[67141]: ts=2024-09-20T22:43:24.151Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-09-20T22:43:24.486 INFO:journalctl@ceph.prometheus.a.smithi154.stdout:Sep 20 22:43:24 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-prometheus-a[67141]: ts=2024-09-20T22:43:24.151Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-09-20T22:43:24.486 INFO:journalctl@ceph.prometheus.a.smithi154.stdout:Sep 20 22:43:24 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-prometheus-a[67141]: ts=2024-09-20T22:43:24.152Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-09-20T22:43:24.486 INFO:journalctl@ceph.prometheus.a.smithi154.stdout:Sep 20 22:43:24 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-prometheus-a[67141]: ts=2024-09-20T22:43:24.152Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.456µs 2024-09-20T22:43:24.486 INFO:journalctl@ceph.prometheus.a.smithi154.stdout:Sep 20 22:43:24 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-prometheus-a[67141]: ts=2024-09-20T22:43:24.152Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-09-20T22:43:24.486 INFO:journalctl@ceph.prometheus.a.smithi154.stdout:Sep 20 22:43:24 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-prometheus-a[67141]: ts=2024-09-20T22:43:24.153Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=1 2024-09-20T22:43:24.487 INFO:journalctl@ceph.prometheus.a.smithi154.stdout:Sep 20 22:43:24 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-prometheus-a[67141]: ts=2024-09-20T22:43:24.153Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=1 2024-09-20T22:43:24.487 INFO:journalctl@ceph.prometheus.a.smithi154.stdout:Sep 20 22:43:24 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-prometheus-a[67141]: ts=2024-09-20T22:43:24.153Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=19.634µs wal_replay_duration=625.97µs wbl_replay_duration=112ns total_replay_duration=663.182µs 2024-09-20T22:43:24.487 INFO:journalctl@ceph.prometheus.a.smithi154.stdout:Sep 20 22:43:24 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-prometheus-a[67141]: ts=2024-09-20T22:43:24.154Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-09-20T22:43:24.487 INFO:journalctl@ceph.prometheus.a.smithi154.stdout:Sep 20 22:43:24 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-prometheus-a[67141]: ts=2024-09-20T22:43:24.154Z caller=main.go:1029 level=info msg="TSDB started" 2024-09-20T22:43:24.487 INFO:journalctl@ceph.prometheus.a.smithi154.stdout:Sep 20 22:43:24 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-prometheus-a[67141]: ts=2024-09-20T22:43:24.154Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-09-20T22:43:24.487 INFO:journalctl@ceph.prometheus.a.smithi154.stdout:Sep 20 22:43:24 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-prometheus-a[67141]: ts=2024-09-20T22:43:24.176Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=22.26845ms db_storage=783ns remote_storage=1.054µs web_handler=568ns query_engine=692ns scrape=465.337µs scrape_sd=114.137µs notify=31.146µs notify_sd=46.746µs rules=21.284162ms tracing=6.055µs 2024-09-20T22:43:24.487 INFO:journalctl@ceph.prometheus.a.smithi154.stdout:Sep 20 22:43:24 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-prometheus-a[67141]: ts=2024-09-20T22:43:24.176Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-09-20T22:43:24.487 INFO:journalctl@ceph.prometheus.a.smithi154.stdout:Sep 20 22:43:24 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-prometheus-a[67141]: ts=2024-09-20T22:43:24.176Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-09-20T22:43:24.487 INFO:journalctl@ceph.prometheus.a.smithi154.stdout:Sep 20 22:43:24 smithi154 bash[67092]: 5c4f4836aceb1176504ae6451e7c68b4f7ea65a6eb77cc448425fb930ada26fc 2024-09-20T22:43:24.487 INFO:journalctl@ceph.prometheus.a.smithi154.stdout:Sep 20 22:43:24 smithi154 systemd[1]: Started Ceph prometheus.a for a9c0bc70-77a0-11ef-baf5-efdc52797490. 2024-09-20T22:43:24.502 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:43:24.523 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:24.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:24.524 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:43:24.625 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:43:24.648 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:24.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:24.649 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:43:24.683 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:43:24 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: [20/Sep/2024:22:43:24] ENGINE Bus STOPPING 2024-09-20T22:43:24.683 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:43:24 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: [20/Sep/2024:22:43:24] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-20T22:43:24.683 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:43:24 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: [20/Sep/2024:22:43:24] ENGINE Bus STOPPED 2024-09-20T22:43:24.683 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:43:24 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: [20/Sep/2024:22:43:24] ENGINE Bus STARTING 2024-09-20T22:43:24.683 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:43:24 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: [20/Sep/2024:22:43:24] ENGINE Serving on http://:::9283 2024-09-20T22:43:24.683 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:43:24 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: [20/Sep/2024:22:43:24] ENGINE Bus STARTED 2024-09-20T22:43:24.683 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:43:24 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: [20/Sep/2024:22:43:24] ENGINE Bus STOPPING 2024-09-20T22:43:24.750 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:43:24.773 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:24.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:24.774 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:43:24.783 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-20T22:43:24.783 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2024-09-20T22:43:24.783 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2024-09-20T22:43:24.783 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [ 2024-09-20T22:43:24.784 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.784 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24334.0:1 2.1 e67/65 2:c4c92e5a:::rbd_trash:head v 67'1, mlcod=0'0)", 2024-09-20T22:43:24.784 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:41:38.150976+0000", 2024-09-20T22:43:24.784 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 106.629700032, 2024-09-20T22:43:24.784 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026331269999999999, 2024-09-20T22:43:24.784 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:43:24.784 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:43:24.784 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:43:24.784 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:43:24.784 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.784 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:43:24.784 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:41:38.150976+0000", 2024-09-20T22:43:24.784 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:43:24.785 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.785 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.785 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:43:24.785 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:41:38.150976+0000", 2024-09-20T22:43:24.785 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:43:24.785 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.785 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.785 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:43:24.785 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:41:38.150976+0000", 2024-09-20T22:43:24.785 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.0999999999999999e-07 2024-09-20T22:43:24.785 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.785 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.785 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:43:24.785 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:41:38.150982+0000", 2024-09-20T22:43:24.785 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.9070000000000004e-06 2024-09-20T22:43:24.786 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.786 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.786 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:43:24.786 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:41:38.150988+0000", 2024-09-20T22:43:24.786 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.19e-06 2024-09-20T22:43:24.786 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.786 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.786 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:43:24.786 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:41:38.150996+0000", 2024-09-20T22:43:24.786 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.7549999999999994e-06 2024-09-20T22:43:24.786 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.786 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.786 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:43:24.787 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:41:38.152298+0000", 2024-09-20T22:43:24.787 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001301553 2024-09-20T22:43:24.787 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.787 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.787 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:43:24.787 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:41:38.152320+0000", 2024-09-20T22:43:24.787 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.2101999999999999e-05 2024-09-20T22:43:24.787 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.787 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.787 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:43:24.787 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:41:38.153579+0000", 2024-09-20T22:43:24.787 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0012597000000000001 2024-09-20T22:43:24.787 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.787 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.787 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:43:24.788 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:41:38.153609+0000", 2024-09-20T22:43:24.788 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.951e-05 2024-09-20T22:43:24.788 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:43:24.788 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:43:24.788 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:43:24.788 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.788 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.788 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.14478.0:205 4.19 4:9a6dd8c0:gc::gc.0:head [create,call version.check_conds in=18b,call rgw_gc.rgw_gc_queue_init in=23b,call version.set in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-09-20T22:43:24.788 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:41:45.187341+0000", 2024-09-20T22:43:24.788 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 99.593334412000004, 2024-09-20T22:43:24.788 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0027462810000000002, 2024-09-20T22:43:24.788 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:43:24.789 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:43:24.789 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:43:24.789 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:43:24.789 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.14478", 2024-09-20T22:43:24.789 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/2994273271", 2024-09-20T22:43:24.789 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 205 2024-09-20T22:43:24.789 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.789 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:43:24.789 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.789 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:43:24.789 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:41:45.187341+0000", 2024-09-20T22:43:24.789 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:43:24.789 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.789 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.790 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:43:24.790 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:41:45.187341+0000", 2024-09-20T22:43:24.790 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:43:24.790 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.790 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.790 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:43:24.790 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:41:45.187344+0000", 2024-09-20T22:43:24.790 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.4449999999999999e-06 2024-09-20T22:43:24.790 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.790 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.790 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:43:24.790 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:41:45.187354+0000", 2024-09-20T22:43:24.791 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0084999999999999e-05 2024-09-20T22:43:24.791 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.791 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.791 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:43:24.791 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:41:45.187379+0000", 2024-09-20T22:43:24.791 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.5230999999999999e-05 2024-09-20T22:43:24.791 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.791 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.791 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:43:24.791 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:41:45.187394+0000", 2024-09-20T22:43:24.791 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.4548e-05 2024-09-20T22:43:24.791 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.791 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.791 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:43:24.792 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:41:45.187486+0000", 2024-09-20T22:43:24.792 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.2142e-05 2024-09-20T22:43:24.792 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.792 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.792 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:43:24.792 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:41:45.187668+0000", 2024-09-20T22:43:24.792 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00018178700000000001 2024-09-20T22:43:24.792 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.792 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.792 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 0,2", 2024-09-20T22:43:24.792 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:41:45.187966+0000", 2024-09-20T22:43:24.792 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00029887199999999999 2024-09-20T22:43:24.792 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.792 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.793 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:43:24.793 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:41:45.189247+0000", 2024-09-20T22:43:24.793 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001280193 2024-09-20T22:43:24.793 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.793 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.793 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:43:24.793 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:41:45.189888+0000", 2024-09-20T22:43:24.793 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00064099899999999995 2024-09-20T22:43:24.793 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.794 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.794 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:43:24.794 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:41:45.189976+0000", 2024-09-20T22:43:24.794 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.8270999999999996e-05 2024-09-20T22:43:24.794 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.794 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.794 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:43:24.794 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:41:45.190038+0000", 2024-09-20T22:43:24.794 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.1839000000000002e-05 2024-09-20T22:43:24.795 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.795 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.795 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:43:24.795 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:41:45.190088+0000", 2024-09-20T22:43:24.795 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.9869e-05 2024-09-20T22:43:24.795 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:43:24.795 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:43:24.795 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:43:24.795 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.795 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.795 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.14478.0:346 4.19 4:9ed7f5b7:::obj_delete_at_hint.0000000004:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-20T22:43:24.795 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:41:45.247757+0000", 2024-09-20T22:43:24.795 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 99.532918572, 2024-09-20T22:43:24.796 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0027190220000000002, 2024-09-20T22:43:24.796 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:43:24.796 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:43:24.796 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:43:24.796 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:43:24.796 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.14478", 2024-09-20T22:43:24.796 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/2994273271", 2024-09-20T22:43:24.796 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 346 2024-09-20T22:43:24.796 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.796 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:43:24.796 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.796 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:43:24.796 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:41:45.247757+0000", 2024-09-20T22:43:24.796 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:43:24.797 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.797 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.797 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:43:24.797 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:41:45.247757+0000", 2024-09-20T22:43:24.797 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:43:24.797 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.797 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.797 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:43:24.797 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:41:45.247758+0000", 2024-09-20T22:43:24.797 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.088e-06 2024-09-20T22:43:24.797 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.798 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.798 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:43:24.798 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:41:45.247765+0000", 2024-09-20T22:43:24.798 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.4590000000000003e-06 2024-09-20T22:43:24.798 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.798 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.798 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:43:24.798 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:41:45.247777+0000", 2024-09-20T22:43:24.798 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2628e-05 2024-09-20T22:43:24.798 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.799 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.799 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:43:24.800 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:41:45.247792+0000", 2024-09-20T22:43:24.800 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.5026e-05 2024-09-20T22:43:24.800 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.800 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.800 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:43:24.800 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:41:45.247880+0000", 2024-09-20T22:43:24.800 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.7293000000000006e-05 2024-09-20T22:43:24.800 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.800 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.800 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:43:24.800 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:41:45.248074+0000", 2024-09-20T22:43:24.800 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00019385999999999999 2024-09-20T22:43:24.801 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.801 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.801 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 0,2", 2024-09-20T22:43:24.801 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:41:45.248241+0000", 2024-09-20T22:43:24.801 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00016786000000000001 2024-09-20T22:43:24.801 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.801 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.801 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:43:24.801 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:41:45.249119+0000", 2024-09-20T22:43:24.801 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00087750000000000002 2024-09-20T22:43:24.801 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.801 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.801 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:43:24.801 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:41:45.249775+0000", 2024-09-20T22:43:24.801 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00065609200000000005 2024-09-20T22:43:24.802 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.802 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.802 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:43:24.802 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:41:45.250386+0000", 2024-09-20T22:43:24.802 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00061113599999999997 2024-09-20T22:43:24.802 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.802 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.802 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:43:24.802 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:41:45.250437+0000", 2024-09-20T22:43:24.802 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.0494000000000001e-05 2024-09-20T22:43:24.802 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.802 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.803 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:43:24.803 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:41:45.250476+0000", 2024-09-20T22:43:24.803 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.9585999999999997e-05 2024-09-20T22:43:24.803 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:43:24.803 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:43:24.803 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:43:24.803 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.803 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.803 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(osd.4.0:115 7.1 e106/94 7:8422f82d:::smithi08477203-22:5 v 106'14, mlcod=103'10)", 2024-09-20T22:43:24.803 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:43:05.532480+0000", 2024-09-20T22:43:24.803 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 19.248195691999999, 2024-09-20T22:43:24.804 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0040933799999999998, 2024-09-20T22:43:24.804 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:43:24.804 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:43:24.804 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:43:24.804 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:43:24.804 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.804 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:43:24.804 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:05.532480+0000", 2024-09-20T22:43:24.804 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:43:24.804 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.805 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.805 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:43:24.805 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:05.532480+0000", 2024-09-20T22:43:24.805 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:43:24.805 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.805 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.805 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:43:24.805 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:05.532481+0000", 2024-09-20T22:43:24.805 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.3799999999999997e-07 2024-09-20T22:43:24.805 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.805 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.805 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:43:24.805 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:05.532489+0000", 2024-09-20T22:43:24.805 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.1799999999999996e-06 2024-09-20T22:43:24.806 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.806 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.806 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:43:24.806 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:05.532499+0000", 2024-09-20T22:43:24.806 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0132000000000001e-05 2024-09-20T22:43:24.806 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.806 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.806 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:43:24.806 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:05.532509+0000", 2024-09-20T22:43:24.806 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0266e-05 2024-09-20T22:43:24.806 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.806 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.806 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:43:24.806 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:05.532617+0000", 2024-09-20T22:43:24.807 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00010778200000000001 2024-09-20T22:43:24.807 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.807 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.807 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:43:24.807 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:05.532639+0000", 2024-09-20T22:43:24.807 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.2096999999999999e-05 2024-09-20T22:43:24.807 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.807 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.807 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:43:24.807 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:05.536530+0000", 2024-09-20T22:43:24.807 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.003890555 2024-09-20T22:43:24.807 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.808 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.808 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:43:24.808 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:05.536573+0000", 2024-09-20T22:43:24.808 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.3829999999999999e-05 2024-09-20T22:43:24.808 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:43:24.808 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:43:24.808 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:43:24.808 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.808 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.808 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(osd.0.0:117 7.6 e106/94 7:64bcd2fc:::smithi08477203-20:5 v 106'12, mlcod=105'9)", 2024-09-20T22:43:24.808 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:43:05.533153+0000", 2024-09-20T22:43:24.808 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 19.247522443000001, 2024-09-20T22:43:24.808 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0071229780000000003, 2024-09-20T22:43:24.808 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:43:24.809 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:43:24.809 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:43:24.809 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:43:24.809 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.809 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:43:24.809 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:05.533153+0000", 2024-09-20T22:43:24.809 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:43:24.809 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.809 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.809 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:43:24.810 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:05.533153+0000", 2024-09-20T22:43:24.810 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:43:24.810 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.810 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.810 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:43:24.810 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:05.533154+0000", 2024-09-20T22:43:24.810 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.9499999999999998e-07 2024-09-20T22:43:24.811 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.811 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.811 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:43:24.811 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:05.533171+0000", 2024-09-20T22:43:24.811 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.7343999999999999e-05 2024-09-20T22:43:24.811 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.812 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.812 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:43:24.812 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:05.533178+0000", 2024-09-20T22:43:24.812 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.6549999999999998e-06 2024-09-20T22:43:24.812 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.812 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.812 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:43:24.812 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:05.533184+0000", 2024-09-20T22:43:24.813 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.3620000000000002e-06 2024-09-20T22:43:24.813 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.813 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.813 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:43:24.813 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:05.533262+0000", 2024-09-20T22:43:24.813 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.7972999999999994e-05 2024-09-20T22:43:24.813 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.813 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.814 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:43:24.814 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:05.533283+0000", 2024-09-20T22:43:24.814 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.0548e-05 2024-09-20T22:43:24.814 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.814 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.814 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:43:24.814 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:05.540242+0000", 2024-09-20T22:43:24.815 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0069598230000000004 2024-09-20T22:43:24.815 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.815 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.815 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:43:24.815 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:05.540276+0000", 2024-09-20T22:43:24.815 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.3977999999999999e-05 2024-09-20T22:43:24.815 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:43:24.815 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:43:24.815 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:43:24.816 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.816 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.816 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(osd.6.0:128 7.c e106/94 7:35bf785a:::smithi08477203-29:5 v 106'36, mlcod=105'33)", 2024-09-20T22:43:24.816 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:43:05.535654+0000", 2024-09-20T22:43:24.816 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 19.245021547, 2024-09-20T22:43:24.816 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0064874859999999998, 2024-09-20T22:43:24.816 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:43:24.816 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:43:24.816 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:43:24.816 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:43:24.816 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.816 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:43:24.816 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:05.535654+0000", 2024-09-20T22:43:24.816 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:43:24.817 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.817 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.817 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:43:24.817 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:05.535654+0000", 2024-09-20T22:43:24.817 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:43:24.817 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.817 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.817 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:43:24.817 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:05.535655+0000", 2024-09-20T22:43:24.817 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.7199999999999998e-07 2024-09-20T22:43:24.817 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.817 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.817 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:43:24.817 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:05.535663+0000", 2024-09-20T22:43:24.818 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.5059999999999995e-06 2024-09-20T22:43:24.818 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.818 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.818 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:43:24.818 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:05.535677+0000", 2024-09-20T22:43:24.818 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3505e-05 2024-09-20T22:43:24.818 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.818 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.818 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:43:24.818 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:05.535685+0000", 2024-09-20T22:43:24.818 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.2600000000000005e-06 2024-09-20T22:43:24.818 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.818 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.819 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:43:24.819 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:05.541062+0000", 2024-09-20T22:43:24.819 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0053772280000000004 2024-09-20T22:43:24.819 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.819 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.819 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:43:24.819 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:05.541083+0000", 2024-09-20T22:43:24.819 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.0129e-05 2024-09-20T22:43:24.819 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.819 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.819 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:43:24.819 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:05.542095+0000", 2024-09-20T22:43:24.819 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010126149999999999 2024-09-20T22:43:24.819 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.820 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.820 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:43:24.820 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:05.542142+0000", 2024-09-20T22:43:24.820 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.6471000000000003e-05 2024-09-20T22:43:24.820 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:43:24.820 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:43:24.820 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:43:24.820 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.820 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.820 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(osd.6.0:129 7.c e106/94 7:3f0fe232:::smithi08477203-4:5 v 106'38, mlcod=105'33)", 2024-09-20T22:43:24.820 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:43:05.536773+0000", 2024-09-20T22:43:24.820 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 19.243902404, 2024-09-20T22:43:24.820 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0055027619999999996, 2024-09-20T22:43:24.821 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:43:24.821 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:43:24.821 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:43:24.821 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:43:24.821 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.821 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:43:24.821 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:05.536773+0000", 2024-09-20T22:43:24.821 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:43:24.821 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.821 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.821 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:43:24.821 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:05.536773+0000", 2024-09-20T22:43:24.821 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:43:24.821 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.822 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.822 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:43:24.822 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:05.536774+0000", 2024-09-20T22:43:24.822 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.879999999999999e-07 2024-09-20T22:43:24.822 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.822 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.822 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:43:24.822 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:05.536788+0000", 2024-09-20T22:43:24.822 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3688e-05 2024-09-20T22:43:24.822 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.822 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.822 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:43:24.822 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:05.536800+0000", 2024-09-20T22:43:24.823 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.163e-05 2024-09-20T22:43:24.823 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.823 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.823 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:43:24.823 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:05.536808+0000", 2024-09-20T22:43:24.823 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.4360000000000002e-06 2024-09-20T22:43:24.823 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.823 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.823 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:43:24.823 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:05.541384+0000", 2024-09-20T22:43:24.823 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0045759640000000001 2024-09-20T22:43:24.823 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.823 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.823 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:43:24.824 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:05.541402+0000", 2024-09-20T22:43:24.824 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.7725999999999999e-05 2024-09-20T22:43:24.824 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.824 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.824 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:43:24.824 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:05.542235+0000", 2024-09-20T22:43:24.824 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00083332199999999995 2024-09-20T22:43:24.824 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.824 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.824 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:43:24.824 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:05.542276+0000", 2024-09-20T22:43:24.824 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.1007999999999998e-05 2024-09-20T22:43:24.825 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:43:24.825 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:43:24.825 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:43:24.825 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.825 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.825 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24652.0:336 7.7 e107/94 7:e67788e2:::smithi08477203-31:head v 107'45, mlcod=105'41)", 2024-09-20T22:43:24.825 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:43:06.546796+0000", 2024-09-20T22:43:24.825 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 18.233880190000001, 2024-09-20T22:43:24.825 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.015311521999999999, 2024-09-20T22:43:24.825 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:43:24.825 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:43:24.825 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:43:24.825 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:43:24.825 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.826 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:43:24.826 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:06.546796+0000", 2024-09-20T22:43:24.826 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:43:24.826 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.826 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.826 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:43:24.826 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:06.546796+0000", 2024-09-20T22:43:24.826 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:43:24.826 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.826 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.826 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:43:24.827 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:06.546796+0000", 2024-09-20T22:43:24.827 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.84e-07 2024-09-20T22:43:24.827 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.827 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.827 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:43:24.827 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:06.554340+0000", 2024-09-20T22:43:24.827 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0075444680000000004 2024-09-20T22:43:24.827 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.827 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.827 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:43:24.827 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:06.554366+0000", 2024-09-20T22:43:24.827 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.5531e-05 2024-09-20T22:43:24.827 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.827 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.828 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:43:24.828 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:06.554388+0000", 2024-09-20T22:43:24.828 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.2263e-05 2024-09-20T22:43:24.828 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.828 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.828 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:43:24.828 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:06.554465+0000", 2024-09-20T22:43:24.828 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.7296e-05 2024-09-20T22:43:24.828 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.828 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.829 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:43:24.829 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:06.554494+0000", 2024-09-20T22:43:24.829 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.8092000000000001e-05 2024-09-20T22:43:24.829 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.829 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.829 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:43:24.829 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:06.562070+0000", 2024-09-20T22:43:24.829 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0075764730000000002 2024-09-20T22:43:24.829 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.829 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.829 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:43:24.829 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:06.562107+0000", 2024-09-20T22:43:24.830 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.7115e-05 2024-09-20T22:43:24.830 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:43:24.830 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:43:24.830 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:43:24.830 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.830 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.830 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:345 7.2 7:4d20232a:::smithi08477203-34:head [sparse-read 0~1425682,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e108)", 2024-09-20T22:43:24.830 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:43:07.545505+0000", 2024-09-20T22:43:24.830 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 17.235170766, 2024-09-20T22:43:24.830 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0067132909999999997, 2024-09-20T22:43:24.830 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:43:24.830 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:43:24.830 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-20T22:43:24.831 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:43:24.831 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:43:24.831 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:43:24.831 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 345 2024-09-20T22:43:24.831 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.831 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:43:24.831 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.831 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:43:24.831 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.545505+0000", 2024-09-20T22:43:24.831 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:43:24.831 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.831 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.831 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:43:24.831 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.545505+0000", 2024-09-20T22:43:24.831 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:43:24.832 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.832 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.832 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:43:24.832 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.545507+0000", 2024-09-20T22:43:24.832 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.764e-06 2024-09-20T22:43:24.832 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.832 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.832 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:43:24.832 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.545516+0000", 2024-09-20T22:43:24.832 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.0899999999999994e-06 2024-09-20T22:43:24.832 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.832 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.833 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:43:24.833 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.545531+0000", 2024-09-20T22:43:24.833 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.4885000000000001e-05 2024-09-20T22:43:24.833 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.833 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.833 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:43:24.833 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.545542+0000", 2024-09-20T22:43:24.833 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1368e-05 2024-09-20T22:43:24.833 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.833 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.833 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:43:24.833 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.545601+0000", 2024-09-20T22:43:24.833 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.9039000000000002e-05 2024-09-20T22:43:24.834 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.834 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.834 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op must wait for map", 2024-09-20T22:43:24.834 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.545608+0000", 2024-09-20T22:43:24.834 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.8809999999999998e-06 2024-09-20T22:43:24.834 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.834 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.834 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:43:24.834 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.550587+0000", 2024-09-20T22:43:24.834 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0049785089999999999 2024-09-20T22:43:24.834 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.834 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.834 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:43:24.835 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.550679+0000", 2024-09-20T22:43:24.835 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.2839999999999999e-05 2024-09-20T22:43:24.835 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.835 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.835 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:43:24.835 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.552218+0000", 2024-09-20T22:43:24.835 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015389150000000001 2024-09-20T22:43:24.835 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:43:24.835 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:43:24.835 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:43:24.835 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.835 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.835 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:346 7.2 7:4d20232a:::smithi08477203-34:head [sparse-read 0~1425682] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e108)", 2024-09-20T22:43:24.835 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:43:07.545568+0000", 2024-09-20T22:43:24.836 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 17.235108257, 2024-09-20T22:43:24.836 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0069445380000000001, 2024-09-20T22:43:24.836 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:43:24.836 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:43:24.836 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-20T22:43:24.836 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:43:24.836 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:43:24.836 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:43:24.836 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 346 2024-09-20T22:43:24.836 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.836 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:43:24.836 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.836 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:43:24.836 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.545568+0000", 2024-09-20T22:43:24.837 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:43:24.837 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.837 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.837 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:43:24.837 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.545568+0000", 2024-09-20T22:43:24.837 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:43:24.837 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.837 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.837 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:43:24.837 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.545568+0000", 2024-09-20T22:43:24.837 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.0999999999999998e-07 2024-09-20T22:43:24.838 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.838 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.838 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:43:24.838 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.545571+0000", 2024-09-20T22:43:24.838 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.9160000000000001e-06 2024-09-20T22:43:24.838 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.838 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.838 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:43:24.838 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.545578+0000", 2024-09-20T22:43:24.839 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.2450000000000001e-06 2024-09-20T22:43:24.839 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.839 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.839 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:43:24.839 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.545584+0000", 2024-09-20T22:43:24.839 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.8150000000000002e-06 2024-09-20T22:43:24.839 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.839 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.839 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:43:24.839 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.545657+0000", 2024-09-20T22:43:24.839 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.2768999999999994e-05 2024-09-20T22:43:24.839 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.839 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.839 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting_for_map not empty", 2024-09-20T22:43:24.840 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.545660+0000", 2024-09-20T22:43:24.840 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.9409999999999999e-06 2024-09-20T22:43:24.840 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.840 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.840 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:43:24.840 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.552254+0000", 2024-09-20T22:43:24.840 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.006594056 2024-09-20T22:43:24.840 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.840 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.840 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:43:24.840 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.552350+0000", 2024-09-20T22:43:24.840 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.6236999999999994e-05 2024-09-20T22:43:24.840 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.840 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.840 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:43:24.841 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.552512+0000", 2024-09-20T22:43:24.841 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00016194899999999999 2024-09-20T22:43:24.841 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:43:24.841 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:43:24.841 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:43:24.841 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.841 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.841 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:347 7.2 7:4d20232a:::smithi08477203-34:head [sparse-read 0~1425682] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e108)", 2024-09-20T22:43:24.841 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:43:07.545592+0000", 2024-09-20T22:43:24.841 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 17.235083327000002, 2024-09-20T22:43:24.841 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.007075975, 2024-09-20T22:43:24.841 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:43:24.841 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:43:24.841 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-20T22:43:24.842 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:43:24.842 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:43:24.842 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:43:24.842 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 347 2024-09-20T22:43:24.842 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.842 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:43:24.842 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.842 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:43:24.842 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.545592+0000", 2024-09-20T22:43:24.842 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:43:24.842 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.842 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.842 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:43:24.842 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.545592+0000", 2024-09-20T22:43:24.843 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:43:24.843 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.843 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.843 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:43:24.843 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.545593+0000", 2024-09-20T22:43:24.843 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.2899999999999999e-07 2024-09-20T22:43:24.843 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.843 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.843 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:43:24.843 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.545595+0000", 2024-09-20T22:43:24.843 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.0320000000000002e-06 2024-09-20T22:43:24.843 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.843 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.843 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:43:24.844 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.545600+0000", 2024-09-20T22:43:24.844 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.8579999999999999e-06 2024-09-20T22:43:24.844 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.844 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.844 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:43:24.844 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.545604+0000", 2024-09-20T22:43:24.844 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.5079999999999999e-06 2024-09-20T22:43:24.844 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.844 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.844 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:43:24.844 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.545683+0000", 2024-09-20T22:43:24.844 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.8298000000000001e-05 2024-09-20T22:43:24.844 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.844 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.844 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting_for_map not empty", 2024-09-20T22:43:24.845 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.545687+0000", 2024-09-20T22:43:24.845 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.1110000000000001e-06 2024-09-20T22:43:24.845 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.845 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.845 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:43:24.845 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.552543+0000", 2024-09-20T22:43:24.845 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0068558660000000004 2024-09-20T22:43:24.845 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.845 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.845 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:43:24.845 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.552594+0000", 2024-09-20T22:43:24.845 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.1671e-05 2024-09-20T22:43:24.845 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.845 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.846 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:43:24.846 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.552668+0000", 2024-09-20T22:43:24.846 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.4202000000000005e-05 2024-09-20T22:43:24.846 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:43:24.846 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:43:24.846 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:43:24.846 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.846 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.846 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24652.0:348 7.7 e108/94 7:ed54065f:::smithi08477203-12:head v 108'48, mlcod=106'43)", 2024-09-20T22:43:24.846 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:43:07.552756+0000", 2024-09-20T22:43:24.847 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 17.227920206, 2024-09-20T22:43:24.847 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0095795429999999994, 2024-09-20T22:43:24.847 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:43:24.847 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:43:24.847 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:43:24.847 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:43:24.847 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.847 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:43:24.847 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.552756+0000", 2024-09-20T22:43:24.848 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:43:24.848 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.848 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.848 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:43:24.848 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.552756+0000", 2024-09-20T22:43:24.848 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:43:24.848 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.848 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.848 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:43:24.849 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.552756+0000", 2024-09-20T22:43:24.849 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.5000000000000002e-07 2024-09-20T22:43:24.849 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.849 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.849 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:43:24.849 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.556504+0000", 2024-09-20T22:43:24.849 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0037481580000000001 2024-09-20T22:43:24.849 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.849 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.849 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:43:24.849 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.556534+0000", 2024-09-20T22:43:24.849 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.9147999999999999e-05 2024-09-20T22:43:24.849 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.849 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.850 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:43:24.850 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.556548+0000", 2024-09-20T22:43:24.850 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.4773000000000001e-05 2024-09-20T22:43:24.850 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.850 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.850 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:43:24.850 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.556640+0000", 2024-09-20T22:43:24.850 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.1545999999999994e-05 2024-09-20T22:43:24.850 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.850 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.850 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:43:24.850 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.556664+0000", 2024-09-20T22:43:24.850 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.4465999999999999e-05 2024-09-20T22:43:24.851 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.851 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.851 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:43:24.851 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.562301+0000", 2024-09-20T22:43:24.851 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.005636393 2024-09-20T22:43:24.851 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.851 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.851 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:43:24.851 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.562335+0000", 2024-09-20T22:43:24.851 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.4309e-05 2024-09-20T22:43:24.851 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:43:24.851 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:43:24.851 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:43:24.851 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.851 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.852 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24652.0:353 7.f e108/94 7:f93102e8:::smithi08477203-37:head v 108'11, mlcod=98'9)", 2024-09-20T22:43:24.852 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:43:07.817310+0000", 2024-09-20T22:43:24.852 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 16.963366117, 2024-09-20T22:43:24.852 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002901023, 2024-09-20T22:43:24.852 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:43:24.852 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:43:24.852 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:43:24.852 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:43:24.852 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.852 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:43:24.852 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.817310+0000", 2024-09-20T22:43:24.852 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:43:24.852 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.852 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.853 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:43:24.853 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.817310+0000", 2024-09-20T22:43:24.853 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:43:24.853 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.853 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.853 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:43:24.853 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.817310+0000", 2024-09-20T22:43:24.853 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.3299999999999998e-07 2024-09-20T22:43:24.853 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.853 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.853 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:43:24.853 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.817797+0000", 2024-09-20T22:43:24.853 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000487274 2024-09-20T22:43:24.853 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.853 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.854 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:43:24.854 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.817814+0000", 2024-09-20T22:43:24.854 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.6884999999999998e-05 2024-09-20T22:43:24.854 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.854 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.854 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:43:24.854 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.817825+0000", 2024-09-20T22:43:24.854 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0910999999999999e-05 2024-09-20T22:43:24.854 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.854 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.854 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:43:24.854 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.817895+0000", 2024-09-20T22:43:24.854 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.0419000000000005e-05 2024-09-20T22:43:24.855 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.855 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.855 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:43:24.855 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.817920+0000", 2024-09-20T22:43:24.855 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.4199e-05 2024-09-20T22:43:24.855 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.855 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.855 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:43:24.855 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.820168+0000", 2024-09-20T22:43:24.855 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002248196 2024-09-20T22:43:24.855 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.855 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.855 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:43:24.855 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.820211+0000", 2024-09-20T22:43:24.855 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.2806000000000003e-05 2024-09-20T22:43:24.856 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:43:24.856 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:43:24.856 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:43:24.856 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.856 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.856 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24652.0:372 7.7 e111/94 7:e1d5c412:::smithi08477203-47:head v 111'53, mlcod=108'50)", 2024-09-20T22:43:24.856 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:43:10.124780+0000", 2024-09-20T22:43:24.856 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 14.655895594, 2024-09-20T22:43:24.856 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.003145756, 2024-09-20T22:43:24.856 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:43:24.857 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:43:24.857 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:43:24.857 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:43:24.857 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.857 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:43:24.857 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:10.124780+0000", 2024-09-20T22:43:24.857 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:43:24.857 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.857 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.857 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:43:24.857 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:10.124780+0000", 2024-09-20T22:43:24.857 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:43:24.858 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.858 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.858 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:43:24.858 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:10.124780+0000", 2024-09-20T22:43:24.858 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.4999999999999999e-07 2024-09-20T22:43:24.858 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.858 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.858 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:43:24.858 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:10.126153+0000", 2024-09-20T22:43:24.858 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001372774 2024-09-20T22:43:24.858 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.858 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.858 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:43:24.859 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:10.126162+0000", 2024-09-20T22:43:24.859 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.2299999999999997e-06 2024-09-20T22:43:24.859 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.859 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.859 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:43:24.859 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:10.126183+0000", 2024-09-20T22:43:24.859 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.0335000000000001e-05 2024-09-20T22:43:24.859 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.859 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.859 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:43:24.859 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:10.126246+0000", 2024-09-20T22:43:24.859 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.3414e-05 2024-09-20T22:43:24.859 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.859 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.859 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:43:24.860 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:10.126268+0000", 2024-09-20T22:43:24.860 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.1404999999999999e-05 2024-09-20T22:43:24.860 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.860 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.860 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:43:24.860 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:10.127877+0000", 2024-09-20T22:43:24.860 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016090430000000001 2024-09-20T22:43:24.860 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.860 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.860 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:43:24.860 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:10.127926+0000", 2024-09-20T22:43:24.860 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.9305000000000003e-05 2024-09-20T22:43:24.860 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:43:24.860 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:43:24.861 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:43:24.861 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.861 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.861 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24652.0:389 7.0 e112/94 7:0f524219:::smithi08477203-50:head v 112'11, mlcod=98'9)", 2024-09-20T22:43:24.861 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:43:11.338624+0000", 2024-09-20T22:43:24.861 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 13.442051778, 2024-09-20T22:43:24.861 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0027897360000000001, 2024-09-20T22:43:24.861 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:43:24.861 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:43:24.861 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:43:24.861 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:43:24.861 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.861 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:43:24.861 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:11.338624+0000", 2024-09-20T22:43:24.862 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:43:24.862 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.862 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.862 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:43:24.862 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:11.338624+0000", 2024-09-20T22:43:24.862 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:43:24.862 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.862 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.862 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:43:24.862 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:11.338625+0000", 2024-09-20T22:43:24.862 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.145e-06 2024-09-20T22:43:24.862 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.862 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.863 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:43:24.863 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:11.340095+0000", 2024-09-20T22:43:24.863 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001469663 2024-09-20T22:43:24.863 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.863 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.863 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:43:24.863 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:11.340114+0000", 2024-09-20T22:43:24.863 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.9148999999999999e-05 2024-09-20T22:43:24.863 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.863 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.863 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:43:24.863 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:11.340123+0000", 2024-09-20T22:43:24.863 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.9469999999999992e-06 2024-09-20T22:43:24.863 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.864 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.864 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:43:24.864 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:11.340190+0000", 2024-09-20T22:43:24.864 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.6885999999999996e-05 2024-09-20T22:43:24.864 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.864 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.864 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:43:24.864 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:11.340219+0000", 2024-09-20T22:43:24.864 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.8867999999999999e-05 2024-09-20T22:43:24.864 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.864 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.864 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:43:24.864 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:11.341370+0000", 2024-09-20T22:43:24.864 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00115138 2024-09-20T22:43:24.865 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.865 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.865 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:43:24.865 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:11.341414+0000", 2024-09-20T22:43:24.865 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.3698000000000001e-05 2024-09-20T22:43:24.865 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:43:24.865 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:43:24.865 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:43:24.865 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.865 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.865 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:410 7.2 7:4fa4b2b3:::smithi08477203-21:head [write 426911~540354] snapc a=[9] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-09-20T22:43:24.865 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:43:14.140544+0000", 2024-09-20T22:43:24.865 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 10.640131911999999, 2024-09-20T22:43:24.865 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.005781975, 2024-09-20T22:43:24.866 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:43:24.866 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:43:24.866 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:43:24.866 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:43:24.866 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:43:24.866 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:43:24.866 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 410 2024-09-20T22:43:24.866 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.866 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:43:24.866 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.866 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:43:24.866 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:14.140544+0000", 2024-09-20T22:43:24.866 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:43:24.867 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.867 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.867 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:43:24.867 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:14.140544+0000", 2024-09-20T22:43:24.867 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:43:24.867 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.867 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.867 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:43:24.867 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:14.140545+0000", 2024-09-20T22:43:24.867 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.401e-06 2024-09-20T22:43:24.867 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.868 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.868 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:43:24.868 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:14.140989+0000", 2024-09-20T22:43:24.868 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00044377000000000002 2024-09-20T22:43:24.868 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.868 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.868 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:43:24.868 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:14.141004+0000", 2024-09-20T22:43:24.868 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.5209e-05 2024-09-20T22:43:24.868 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.868 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.868 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:43:24.868 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:14.141015+0000", 2024-09-20T22:43:24.868 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0603e-05 2024-09-20T22:43:24.869 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.869 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.869 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:43:24.869 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:14.141076+0000", 2024-09-20T22:43:24.869 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.0810999999999998e-05 2024-09-20T22:43:24.869 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.869 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.869 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op must wait for map", 2024-09-20T22:43:24.869 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:14.141081+0000", 2024-09-20T22:43:24.869 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.4190000000000001e-06 2024-09-20T22:43:24.869 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.869 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.869 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:43:24.870 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:14.143116+0000", 2024-09-20T22:43:24.870 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0020353469999999999 2024-09-20T22:43:24.870 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.870 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.870 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:43:24.870 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:14.143215+0000", 2024-09-20T22:43:24.870 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.8175000000000001e-05 2024-09-20T22:43:24.870 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.870 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.870 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 1,2", 2024-09-20T22:43:24.870 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:14.143391+0000", 2024-09-20T22:43:24.870 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00017648300000000001 2024-09-20T22:43:24.870 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.870 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.871 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:43:24.871 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:14.146002+0000", 2024-09-20T22:43:24.871 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026110460000000001 2024-09-20T22:43:24.871 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.871 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.871 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:43:24.871 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:14.146025+0000", 2024-09-20T22:43:24.871 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.2629e-05 2024-09-20T22:43:24.871 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.871 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.871 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:43:24.871 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:14.146200+0000", 2024-09-20T22:43:24.871 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00017526 2024-09-20T22:43:24.871 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.872 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.872 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:43:24.872 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:14.146280+0000", 2024-09-20T22:43:24.872 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.9548000000000005e-05 2024-09-20T22:43:24.872 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.872 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.872 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:43:24.872 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:14.146326+0000", 2024-09-20T22:43:24.872 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.6273999999999998e-05 2024-09-20T22:43:24.872 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:43:24.872 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:43:24.872 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:43:24.872 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.872 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.873 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24652.0:419 7.0 e115/94 7:05bb442d:::smithi08477203-24:head v 115'15, mlcod=112'12)", 2024-09-20T22:43:24.873 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:43:14.169561+0000", 2024-09-20T22:43:24.873 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 10.611115011999999, 2024-09-20T22:43:24.873 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026789560000000001, 2024-09-20T22:43:24.873 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:43:24.873 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:43:24.873 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:43:24.873 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:43:24.873 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.873 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:43:24.873 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:14.169561+0000", 2024-09-20T22:43:24.873 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:43:24.873 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.873 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.874 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:43:24.874 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:14.169561+0000", 2024-09-20T22:43:24.874 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:43:24.874 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.874 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.874 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:43:24.874 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:14.169561+0000", 2024-09-20T22:43:24.874 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.8799999999999998e-07 2024-09-20T22:43:24.874 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.874 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.874 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:43:24.874 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:14.170711+0000", 2024-09-20T22:43:24.874 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001149553 2024-09-20T22:43:24.875 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.875 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.875 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:43:24.875 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:14.170722+0000", 2024-09-20T22:43:24.875 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1004e-05 2024-09-20T22:43:24.875 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.875 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.875 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:43:24.875 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:14.170733+0000", 2024-09-20T22:43:24.875 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1426e-05 2024-09-20T22:43:24.875 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.875 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.875 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:43:24.876 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:14.170793+0000", 2024-09-20T22:43:24.876 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.9976999999999999e-05 2024-09-20T22:43:24.876 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.876 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.876 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:43:24.876 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:14.170811+0000", 2024-09-20T22:43:24.876 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.8156999999999999e-05 2024-09-20T22:43:24.876 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.876 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.876 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:43:24.876 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:14.172185+0000", 2024-09-20T22:43:24.876 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001373925 2024-09-20T22:43:24.876 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.876 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.877 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:43:24.877 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:14.172240+0000", 2024-09-20T22:43:24.877 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.4626000000000002e-05 2024-09-20T22:43:24.877 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:43:24.877 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:43:24.877 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:43:24.877 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.877 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.877 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24652.0:446 7.6 e115/94 7:64bcd2fc:::smithi08477203-20:head v 115'18, mlcod=112'15)", 2024-09-20T22:43:24.877 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:43:14.444164+0000", 2024-09-20T22:43:24.878 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 10.336512218999999, 2024-09-20T22:43:24.878 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0028949330000000001, 2024-09-20T22:43:24.878 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:43:24.878 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:43:24.878 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:43:24.878 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:43:24.878 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.878 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:43:24.878 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:14.444164+0000", 2024-09-20T22:43:24.878 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:43:24.878 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.878 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.879 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:43:24.879 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:14.444164+0000", 2024-09-20T22:43:24.879 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:43:24.879 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.879 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.879 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:43:24.879 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:14.444164+0000", 2024-09-20T22:43:24.879 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.08e-07 2024-09-20T22:43:24.879 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.879 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.879 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:43:24.879 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:14.444584+0000", 2024-09-20T22:43:24.879 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00041996900000000002 2024-09-20T22:43:24.880 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.880 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.880 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:43:24.880 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:14.444601+0000", 2024-09-20T22:43:24.880 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.6872999999999999e-05 2024-09-20T22:43:24.880 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.880 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.880 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:43:24.880 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:14.444612+0000", 2024-09-20T22:43:24.880 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.102e-05 2024-09-20T22:43:24.880 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.880 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.880 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:43:24.880 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:14.444848+0000", 2024-09-20T22:43:24.880 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000236133 2024-09-20T22:43:24.881 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.881 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.881 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:43:24.881 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:14.444875+0000", 2024-09-20T22:43:24.881 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.6760000000000001e-05 2024-09-20T22:43:24.881 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.881 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.881 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:43:24.881 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:14.446997+0000", 2024-09-20T22:43:24.881 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0021226109999999999 2024-09-20T22:43:24.881 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.881 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.881 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:43:24.881 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:14.447058+0000", 2024-09-20T22:43:24.882 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.1159000000000002e-05 2024-09-20T22:43:24.882 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:43:24.882 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:43:24.882 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:43:24.882 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.882 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.882 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(osd.6.0:178 7.7 e124/94 7:e1d5c412:::smithi08477203-47:e v 124'77, mlcod=122'73)", 2024-09-20T22:43:24.882 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:43:23.125481+0000", 2024-09-20T22:43:24.882 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 1.6551943250000001, 2024-09-20T22:43:24.882 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0070359730000000001, 2024-09-20T22:43:24.882 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:43:24.882 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:43:24.883 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:43:24.883 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:43:24.883 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.883 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:43:24.883 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:23.125481+0000", 2024-09-20T22:43:24.883 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:43:24.883 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.883 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.883 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:43:24.883 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:23.125481+0000", 2024-09-20T22:43:24.883 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:43:24.883 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.884 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.884 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:43:24.884 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:23.125482+0000", 2024-09-20T22:43:24.884 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.9799999999999999e-07 2024-09-20T22:43:24.884 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.884 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.884 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:43:24.884 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:23.125487+0000", 2024-09-20T22:43:24.884 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.4829999999999999e-06 2024-09-20T22:43:24.884 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.884 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.885 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:43:24.885 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:23.125494+0000", 2024-09-20T22:43:24.885 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.6220000000000003e-06 2024-09-20T22:43:24.885 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.885 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.885 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:43:24.885 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:23.125502+0000", 2024-09-20T22:43:24.885 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.7330000000000003e-06 2024-09-20T22:43:24.885 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.885 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.885 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:43:24.885 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:23.128526+0000", 2024-09-20T22:43:24.886 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0030240359999999999 2024-09-20T22:43:24.886 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.886 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.886 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:43:24.886 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:23.128545+0000", 2024-09-20T22:43:24.886 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.9176000000000001e-05 2024-09-20T22:43:24.886 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.886 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.886 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:43:24.886 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:23.132458+0000", 2024-09-20T22:43:24.886 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0039127720000000001 2024-09-20T22:43:24.886 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.886 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.887 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:43:24.887 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:23.132517+0000", 2024-09-20T22:43:24.887 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.9753000000000002e-05 2024-09-20T22:43:24.887 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:43:24.887 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:43:24.887 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:43:24.887 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.887 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.887 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24652.0:528 7.7 e125/94 7:ed54065f:::smithi08477203-12:head v 125'79, mlcod=122'75)", 2024-09-20T22:43:24.887 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:43:24.132038+0000", 2024-09-20T22:43:24.887 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 0.64863768099999997, 2024-09-20T22:43:24.887 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0033382630000000002, 2024-09-20T22:43:24.887 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:43:24.888 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:43:24.888 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:43:24.888 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:43:24.888 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.888 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:43:24.888 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:24.132038+0000", 2024-09-20T22:43:24.888 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:43:24.888 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.888 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.888 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:43:24.888 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:24.132038+0000", 2024-09-20T22:43:24.888 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:43:24.889 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.889 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.889 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:43:24.889 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:24.132038+0000", 2024-09-20T22:43:24.889 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.0699999999999998e-07 2024-09-20T22:43:24.889 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.889 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.889 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:43:24.889 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:24.133592+0000", 2024-09-20T22:43:24.889 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015532219999999999 2024-09-20T22:43:24.889 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.889 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.889 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:43:24.889 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:24.133610+0000", 2024-09-20T22:43:24.890 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.7978e-05 2024-09-20T22:43:24.890 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.890 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.890 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:43:24.890 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:24.133619+0000", 2024-09-20T22:43:24.890 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.6430000000000007e-06 2024-09-20T22:43:24.890 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.890 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.890 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:43:24.890 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:24.133707+0000", 2024-09-20T22:43:24.890 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.7869999999999997e-05 2024-09-20T22:43:24.890 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.891 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.891 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:43:24.891 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:24.133729+0000", 2024-09-20T22:43:24.891 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.1676999999999999e-05 2024-09-20T22:43:24.891 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.891 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.891 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:43:24.891 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:24.135293+0000", 2024-09-20T22:43:24.891 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001564406 2024-09-20T22:43:24.891 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:24.891 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:24.891 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:43:24.891 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:24.135376+0000", 2024-09-20T22:43:24.891 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.3059999999999994e-05 2024-09-20T22:43:24.892 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:43:24.892 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:43:24.892 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:43:24.892 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:43:24.892 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:43:24.892 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-20T22:43:24.892 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:43:24.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:24.915 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:24.915 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:43:25.016 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:43:25.037 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:25.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:25.038 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:43:25.116 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is unset 2024-09-20T22:43:25.122 INFO:tasks.rados.rados.0.smithi084.stdout:178: done (1 left) 2024-09-20T22:43:25.122 INFO:tasks.rados.rados.0.smithi084.stdout:179: done (0 left) 2024-09-20T22:43:25.122 INFO:tasks.rados.rados.0.smithi084.stdout:180: delete oid 7 current snap is 16 2024-09-20T22:43:25.123 INFO:tasks.rados.rados.0.smithi084.stdout:180: done (0 left) 2024-09-20T22:43:25.124 INFO:tasks.rados.rados.0.smithi084.stdout:181: snap_create 2024-09-20T22:43:25.139 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:43:25.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:25.161 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:25.161 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:43:25.262 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:43:25.283 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:25.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:25.284 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:43:25.365 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:43:25 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: [20/Sep/2024:22:43:25] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-20T22:43:25.365 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:43:25 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: [20/Sep/2024:22:43:25] ENGINE Bus STOPPED 2024-09-20T22:43:25.365 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:43:25 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: [20/Sep/2024:22:43:25] ENGINE Bus STARTING 2024-09-20T22:43:25.365 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:43:25 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: [20/Sep/2024:22:43:25] ENGINE Serving on http://:::9283 2024-09-20T22:43:25.365 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:43:25 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: [20/Sep/2024:22:43:25] ENGINE Bus STARTED 2024-09-20T22:43:25.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:25 smithi084 ceph-mon[29774]: pgmap v88: 168 pgs: 10 peering, 158 active+clean; 122 MiB data, 474 MiB used, 715 GiB / 715 GiB avail; 3.2 MiB/s rd, 2.3 MiB/s wr, 30 op/s; 178 KiB/s, 1 objects/s recovering 2024-09-20T22:43:25.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:25 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:43:25.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:25 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:43:25.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:25 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-20T22:43:25.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:25 smithi084 ceph-mon[29774]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-20T22:43:25.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:25 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi084.front.sepia.ceph.com:9093"}]: dispatch 2024-09-20T22:43:25.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:25 smithi084 ceph-mon[29774]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi084.front.sepia.ceph.com:9093"}]: dispatch 2024-09-20T22:43:25.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:25 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:43:25.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:25 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-20T22:43:25.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:25 smithi084 ceph-mon[29774]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-20T22:43:25.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:25 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi154.front.sepia.ceph.com:9095"}]: dispatch 2024-09-20T22:43:25.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:25 smithi084 ceph-mon[29774]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi154.front.sepia.ceph.com:9095"}]: dispatch 2024-09-20T22:43:25.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:25 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:43:25.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:25 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-20T22:43:25.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:25 smithi084 ceph-mon[29774]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-20T22:43:25.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:25 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi154.front.sepia.ceph.com:3000"}]: dispatch 2024-09-20T22:43:25.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:25 smithi084 ceph-mon[29774]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi154.front.sepia.ceph.com:3000"}]: dispatch 2024-09-20T22:43:25.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:25 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:43:25.367 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:25 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:43:25.367 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:25 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/732718030' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:43:25.367 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:25 smithi084 ceph-mon[29774]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:43:25.367 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:25 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/732718030' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T22:43:25.367 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:25 smithi084 ceph-mon[29774]: osdmap e126: 8 total, 8 up, 8 in 2024-09-20T22:43:25.367 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:25 smithi084 ceph-mon[40024]: pgmap v88: 168 pgs: 10 peering, 158 active+clean; 122 MiB data, 474 MiB used, 715 GiB / 715 GiB avail; 3.2 MiB/s rd, 2.3 MiB/s wr, 30 op/s; 178 KiB/s, 1 objects/s recovering 2024-09-20T22:43:25.367 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:25 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:43:25.367 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:25 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:43:25.367 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:25 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-20T22:43:25.367 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:25 smithi084 ceph-mon[40024]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-20T22:43:25.367 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:25 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi084.front.sepia.ceph.com:9093"}]: dispatch 2024-09-20T22:43:25.368 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:25 smithi084 ceph-mon[40024]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi084.front.sepia.ceph.com:9093"}]: dispatch 2024-09-20T22:43:25.368 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:25 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:43:25.368 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:25 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-20T22:43:25.368 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:25 smithi084 ceph-mon[40024]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-20T22:43:25.368 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:25 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi154.front.sepia.ceph.com:9095"}]: dispatch 2024-09-20T22:43:25.368 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:25 smithi084 ceph-mon[40024]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi154.front.sepia.ceph.com:9095"}]: dispatch 2024-09-20T22:43:25.368 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:25 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:43:25.368 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:25 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-20T22:43:25.368 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:25 smithi084 ceph-mon[40024]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-20T22:43:25.368 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:25 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi154.front.sepia.ceph.com:3000"}]: dispatch 2024-09-20T22:43:25.368 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:25 smithi084 ceph-mon[40024]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi154.front.sepia.ceph.com:3000"}]: dispatch 2024-09-20T22:43:25.368 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:25 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:43:25.368 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:25 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:43:25.369 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:25 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/732718030' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:43:25.369 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:25 smithi084 ceph-mon[40024]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:43:25.369 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:25 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/732718030' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T22:43:25.369 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:25 smithi084 ceph-mon[40024]: osdmap e126: 8 total, 8 up, 8 in 2024-09-20T22:43:25.385 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:43:25.406 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:25.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:25.406 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:43:25.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:25 smithi154 ceph-mon[33264]: pgmap v88: 168 pgs: 10 peering, 158 active+clean; 122 MiB data, 474 MiB used, 715 GiB / 715 GiB avail; 3.2 MiB/s rd, 2.3 MiB/s wr, 30 op/s; 178 KiB/s, 1 objects/s recovering 2024-09-20T22:43:25.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:25 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:43:25.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:25 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:43:25.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:25 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-20T22:43:25.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:25 smithi154 ceph-mon[33264]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-20T22:43:25.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:25 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi084.front.sepia.ceph.com:9093"}]: dispatch 2024-09-20T22:43:25.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:25 smithi154 ceph-mon[33264]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi084.front.sepia.ceph.com:9093"}]: dispatch 2024-09-20T22:43:25.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:25 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:43:25.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:25 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-20T22:43:25.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:25 smithi154 ceph-mon[33264]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-20T22:43:25.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:25 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi154.front.sepia.ceph.com:9095"}]: dispatch 2024-09-20T22:43:25.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:25 smithi154 ceph-mon[33264]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi154.front.sepia.ceph.com:9095"}]: dispatch 2024-09-20T22:43:25.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:25 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:43:25.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:25 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-20T22:43:25.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:25 smithi154 ceph-mon[33264]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-20T22:43:25.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:25 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi154.front.sepia.ceph.com:3000"}]: dispatch 2024-09-20T22:43:25.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:25 smithi154 ceph-mon[33264]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi154.front.sepia.ceph.com:3000"}]: dispatch 2024-09-20T22:43:25.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:25 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:43:25.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:25 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:43:25.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:25 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/732718030' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:43:25.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:25 smithi154 ceph-mon[33264]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:43:25.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:25 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/732718030' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T22:43:25.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:25 smithi154 ceph-mon[33264]: osdmap e126: 8 total, 8 up, 8 in 2024-09-20T22:43:25.507 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:43:25.529 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:25.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:25.530 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:43:25.631 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:43:25.652 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:25.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:25.653 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:43:25.754 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:43:25.775 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:25.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:25.776 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:43:25.877 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:43:25.899 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:25.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:25.900 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:43:26.001 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:43:26.022 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:26.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:26.023 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:43:26.123 INFO:tasks.rados.rados.0.smithi084.stdout:181: done (0 left) 2024-09-20T22:43:26.123 INFO:tasks.rados.rados.0.smithi084.stdout:182: read oid 32 snap 13 2024-09-20T22:43:26.123 INFO:tasks.rados.rados.0.smithi084.stdout:182: expect (ObjNum 31 snap 0 seq_num 31) 2024-09-20T22:43:26.123 INFO:tasks.rados.rados.0.smithi084.stdout:183: snap_create 2024-09-20T22:43:26.124 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:43:26.131 INFO:tasks.rados.rados.0.smithi084.stdout:182: expect (ObjNum 31 snap 0 seq_num 31) 2024-09-20T22:43:26.145 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:26.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:26.146 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:43:26.247 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:43:26.268 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:26.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:26.269 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:43:26.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:26 smithi084 ceph-mon[40024]: 1.2 scrub starts 2024-09-20T22:43:26.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:26 smithi084 ceph-mon[40024]: 1.2 scrub ok 2024-09-20T22:43:26.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:26 smithi084 ceph-mon[40024]: 1.3 scrub starts 2024-09-20T22:43:26.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:26 smithi084 ceph-mon[40024]: 1.3 scrub ok 2024-09-20T22:43:26.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:26 smithi084 ceph-mon[40024]: purged_snaps scrub starts 2024-09-20T22:43:26.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:26 smithi084 ceph-mon[40024]: purged_snaps scrub ok 2024-09-20T22:43:26.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:26 smithi084 ceph-mon[40024]: 1.5 scrub starts 2024-09-20T22:43:26.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:26 smithi084 ceph-mon[40024]: 1.5 scrub ok 2024-09-20T22:43:26.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:26 smithi084 ceph-mon[40024]: osdmap e127: 8 total, 8 up, 8 in 2024-09-20T22:43:26.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:26 smithi084 ceph-mon[29774]: 1.2 scrub starts 2024-09-20T22:43:26.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:26 smithi084 ceph-mon[29774]: 1.2 scrub ok 2024-09-20T22:43:26.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:26 smithi084 ceph-mon[29774]: 1.3 scrub starts 2024-09-20T22:43:26.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:26 smithi084 ceph-mon[29774]: 1.3 scrub ok 2024-09-20T22:43:26.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:26 smithi084 ceph-mon[29774]: purged_snaps scrub starts 2024-09-20T22:43:26.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:26 smithi084 ceph-mon[29774]: purged_snaps scrub ok 2024-09-20T22:43:26.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:26 smithi084 ceph-mon[29774]: 1.5 scrub starts 2024-09-20T22:43:26.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:26 smithi084 ceph-mon[29774]: 1.5 scrub ok 2024-09-20T22:43:26.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:26 smithi084 ceph-mon[29774]: osdmap e127: 8 total, 8 up, 8 in 2024-09-20T22:43:26.370 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:43:26.391 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:26.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:26.392 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:43:26.493 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:43:26.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:26 smithi154 ceph-mon[33264]: 1.2 scrub starts 2024-09-20T22:43:26.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:26 smithi154 ceph-mon[33264]: 1.2 scrub ok 2024-09-20T22:43:26.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:26 smithi154 ceph-mon[33264]: 1.3 scrub starts 2024-09-20T22:43:26.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:26 smithi154 ceph-mon[33264]: 1.3 scrub ok 2024-09-20T22:43:26.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:26 smithi154 ceph-mon[33264]: purged_snaps scrub starts 2024-09-20T22:43:26.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:26 smithi154 ceph-mon[33264]: purged_snaps scrub ok 2024-09-20T22:43:26.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:26 smithi154 ceph-mon[33264]: 1.5 scrub starts 2024-09-20T22:43:26.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:26 smithi154 ceph-mon[33264]: 1.5 scrub ok 2024-09-20T22:43:26.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:26 smithi154 ceph-mon[33264]: osdmap e127: 8 total, 8 up, 8 in 2024-09-20T22:43:26.514 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:26.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:26.515 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:43:26.602 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:43:26.616 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:43:26.638 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:26.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:26.639 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:43:26.740 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:43:26.761 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:26.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:26.762 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:43:26.863 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:43:26.884 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:26.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:26.885 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:43:26.968 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-20T22:43:26.986 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:43:27.007 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:27.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:27.008 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:43:27.109 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:43:27.124 INFO:tasks.rados.rados.0.smithi084.stdout:182: done (1 left) 2024-09-20T22:43:27.124 INFO:tasks.rados.rados.0.smithi084.stdout:183: done (0 left) 2024-09-20T22:43:27.124 INFO:tasks.rados.rados.0.smithi084.stdout:184: copy_from oid 19 from oid 22 current snap is 18 2024-09-20T22:43:27.124 INFO:tasks.rados.rados.0.smithi084.stdout:185: read oid 21 snap 14 2024-09-20T22:43:27.124 INFO:tasks.rados.rados.0.smithi084.stdout:185: expect (ObjNum 81 snap 10 seq_num 81) 2024-09-20T22:43:27.124 INFO:tasks.rados.rados.0.smithi084.stdout:186: write oid 49 current snap is 18 2024-09-20T22:43:27.124 INFO:tasks.rados.rados.0.smithi084.stdout:186: seq_num 94 ranges {779858=469642,1817425=507031,2781526=586628} 2024-09-20T22:43:27.131 INFO:tasks.rados.rados.0.smithi084.stdout:186: writing smithi08477203-49 from 779858 to 1249500 tid 1 2024-09-20T22:43:27.133 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:27.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:27.135 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:43:27.137 INFO:tasks.rados.rados.0.smithi084.stdout:186: writing smithi08477203-49 from 1817425 to 2324456 tid 2 2024-09-20T22:43:27.144 INFO:tasks.rados.rados.0.smithi084.stdout:186: writing smithi08477203-49 from 2781526 to 3368154 tid 3 2024-09-20T22:43:27.144 INFO:tasks.rados.rados.0.smithi084.stdout:187: read oid 24 snap -1 2024-09-20T22:43:27.144 INFO:tasks.rados.rados.0.smithi084.stdout:187: expect (ObjNum 75 snap 9 seq_num 75) 2024-09-20T22:43:27.144 INFO:tasks.rados.rados.0.smithi084.stdout:188: delete oid 43 current snap is 18 2024-09-20T22:43:27.144 INFO:tasks.rados.rados.0.smithi084.stdout:185: expect (ObjNum 81 snap 10 seq_num 81) 2024-09-20T22:43:27.235 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:43:27.245 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:43:27.256 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:27.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:27.257 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:43:27.337 INFO:tasks.rados.rados.0.smithi084.stdout:184: finishing copy_from racing read to smithi08477203-19 2024-09-20T22:43:27.337 INFO:tasks.rados.rados.0.smithi084.stdout:186: finishing write tid 1 to smithi08477203-49 2024-09-20T22:43:27.337 INFO:tasks.rados.rados.0.smithi084.stdout:185: done (4 left) 2024-09-20T22:43:27.338 INFO:tasks.rados.rados.0.smithi084.stdout:188: done (3 left) 2024-09-20T22:43:27.338 INFO:tasks.rados.rados.0.smithi084.stdout:189: copy_from oid 33 from oid 13 current snap is 18 2024-09-20T22:43:27.338 INFO:tasks.rados.rados.0.smithi084.stdout:184: finishing copy_from to smithi08477203-19 2024-09-20T22:43:27.338 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 19 v 26 (ObjNum 85 snap 11 seq_num 85) dirty exists 2024-09-20T22:43:27.338 INFO:tasks.rados.rados.0.smithi084.stdout:186: finishing write tid 2 to smithi08477203-49 2024-09-20T22:43:27.338 INFO:tasks.rados.rados.0.smithi084.stdout:186: finishing write tid 3 to smithi08477203-49 2024-09-20T22:43:27.338 INFO:tasks.rados.rados.0.smithi084.stdout:186: finishing write tid 4 to smithi08477203-49 2024-09-20T22:43:27.338 INFO:tasks.rados.rados.0.smithi084.stdout:186: finishing write tid 5 to smithi08477203-49 2024-09-20T22:43:27.338 INFO:tasks.rados.rados.0.smithi084.stdout:186: oid 49 updating version 0 to 60 2024-09-20T22:43:27.338 INFO:tasks.rados.rados.0.smithi084.stdout:186: oid 49 version 60 is already newer than 58 2024-09-20T22:43:27.338 INFO:tasks.rados.rados.0.smithi084.stdout:186: oid 49 version 60 is already newer than 59 2024-09-20T22:43:27.339 INFO:tasks.rados.rados.0.smithi084.stdout:186: oid 49 updating version 60 to 61 2024-09-20T22:43:27.339 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 49 v 61 (ObjNum 94 snap 18 seq_num 94) dirty exists 2024-09-20T22:43:27.339 INFO:tasks.rados.rados.0.smithi084.stdout:186: left oid 49 (ObjNum 94 snap 18 seq_num 94) 2024-09-20T22:43:27.339 INFO:tasks.rados.rados.0.smithi084.stdout:187: expect (ObjNum 75 snap 9 seq_num 75) 2024-09-20T22:43:27.358 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:43:27.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:27.380 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:27.380 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:43:27.478 INFO:tasks.rados.rados.0.smithi084.stdout:189: finishing copy_from racing read to smithi08477203-33 2024-09-20T22:43:27.478 INFO:tasks.rados.rados.0.smithi084.stdout:189: finishing copy_from to smithi08477203-33 2024-09-20T22:43:27.478 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 33 v 6 (ObjNum 12 snap 0 seq_num 12) dirty exists 2024-09-20T22:43:27.478 INFO:tasks.rados.rados.0.smithi084.stdout:184: done (3 left) 2024-09-20T22:43:27.479 INFO:tasks.rados.rados.0.smithi084.stdout:186: done (2 left) 2024-09-20T22:43:27.479 INFO:tasks.rados.rados.0.smithi084.stdout:187: done (1 left) 2024-09-20T22:43:27.479 INFO:tasks.rados.rados.0.smithi084.stdout:189: done (0 left) 2024-09-20T22:43:27.479 INFO:tasks.rados.rados.0.smithi084.stdout:190: snap_create 2024-09-20T22:43:27.480 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:43:27.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:27.504 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:27.504 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:43:27.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:27 smithi154 ceph-mon[33264]: pgmap v91: 168 pgs: 10 peering, 158 active+clean; 122 MiB data, 474 MiB used, 715 GiB / 715 GiB avail; 3.2 MiB/s rd, 2.3 MiB/s wr, 30 op/s; 178 KiB/s, 1 objects/s recovering 2024-09-20T22:43:27.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:27 smithi154 ceph-mon[33264]: 1.4 scrub starts 2024-09-20T22:43:27.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:27 smithi154 ceph-mon[33264]: 1.4 scrub ok 2024-09-20T22:43:27.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:27 smithi154 ceph-mon[33264]: 1.0 scrub starts 2024-09-20T22:43:27.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:27 smithi154 ceph-mon[33264]: 1.0 scrub ok 2024-09-20T22:43:27.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:27 smithi154 ceph-mon[33264]: 1.7 scrub starts 2024-09-20T22:43:27.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:27 smithi154 ceph-mon[33264]: 1.7 scrub ok 2024-09-20T22:43:27.509 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:27 smithi154 ceph-mon[33264]: osdmap e128: 8 total, 8 up, 8 in 2024-09-20T22:43:27.569 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:27 smithi084 ceph-mon[40024]: pgmap v91: 168 pgs: 10 peering, 158 active+clean; 122 MiB data, 474 MiB used, 715 GiB / 715 GiB avail; 3.2 MiB/s rd, 2.3 MiB/s wr, 30 op/s; 178 KiB/s, 1 objects/s recovering 2024-09-20T22:43:27.569 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:27 smithi084 ceph-mon[40024]: 1.4 scrub starts 2024-09-20T22:43:27.569 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:27 smithi084 ceph-mon[40024]: 1.4 scrub ok 2024-09-20T22:43:27.569 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:27 smithi084 ceph-mon[40024]: 1.0 scrub starts 2024-09-20T22:43:27.570 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:27 smithi084 ceph-mon[40024]: 1.0 scrub ok 2024-09-20T22:43:27.570 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:27 smithi084 ceph-mon[40024]: 1.7 scrub starts 2024-09-20T22:43:27.570 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:27 smithi084 ceph-mon[40024]: 1.7 scrub ok 2024-09-20T22:43:27.570 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:27 smithi084 ceph-mon[40024]: osdmap e128: 8 total, 8 up, 8 in 2024-09-20T22:43:27.570 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:27 smithi084 ceph-mon[29774]: pgmap v91: 168 pgs: 10 peering, 158 active+clean; 122 MiB data, 474 MiB used, 715 GiB / 715 GiB avail; 3.2 MiB/s rd, 2.3 MiB/s wr, 30 op/s; 178 KiB/s, 1 objects/s recovering 2024-09-20T22:43:27.570 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:27 smithi084 ceph-mon[29774]: 1.4 scrub starts 2024-09-20T22:43:27.570 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:27 smithi084 ceph-mon[29774]: 1.4 scrub ok 2024-09-20T22:43:27.570 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:27 smithi084 ceph-mon[29774]: 1.0 scrub starts 2024-09-20T22:43:27.570 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:27 smithi084 ceph-mon[29774]: 1.0 scrub ok 2024-09-20T22:43:27.570 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:27 smithi084 ceph-mon[29774]: 1.7 scrub starts 2024-09-20T22:43:27.571 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:27 smithi084 ceph-mon[29774]: 1.7 scrub ok 2024-09-20T22:43:27.571 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:27 smithi084 ceph-mon[29774]: osdmap e128: 8 total, 8 up, 8 in 2024-09-20T22:43:27.583 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:43:27.584 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:43:27.600 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:43:27.600 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:43:27.605 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:43:27.616 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:43:27.616 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:43:27.627 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:27.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:27.628 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:43:27.633 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-20T22:43:27.634 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T22:43:27.650 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:43:27.650 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:43:27.667 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:43:27.667 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:43:27.685 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:43:27.685 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:43:27.703 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:43:27.703 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:43:27.728 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:43:27.750 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:27.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:27.751 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:43:27.852 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:43:27.875 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:27.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:27.876 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:43:27.977 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:43:27.999 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:27.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:28.000 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:43:28.100 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:43:28.121 INFO:tasks.rados.rados.0.smithi084.stdout:190: done (0 left) 2024-09-20T22:43:28.121 INFO:tasks.rados.rados.0.smithi084.stdout:191: copy_from oid 41 from oid 40 current snap is 19 2024-09-20T22:43:28.121 INFO:tasks.rados.rados.0.smithi084.stdout:192: copy_from oid 6 from oid 47 current snap is 19 2024-09-20T22:43:28.121 INFO:tasks.rados.rados.0.smithi084.stdout:193: read oid 3 snap -1 2024-09-20T22:43:28.121 INFO:tasks.rados.rados.0.smithi084.stdout:193: expect deleted 2024-09-20T22:43:28.121 INFO:tasks.rados.rados.0.smithi084.stdout:194: rollback oid 48 current snap is 19 2024-09-20T22:43:28.121 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 48 to 18 2024-09-20T22:43:28.121 INFO:tasks.rados.rados.0.smithi084.stdout:195: write (excl) oid 45 current snap is 19 2024-09-20T22:43:28.121 INFO:tasks.rados.rados.0.smithi084.stdout:195: seq_num 96 ranges {413472=620140,1818036=510700,3120019=493453} 2024-09-20T22:43:28.125 INFO:tasks.rados.rados.0.smithi084.stdout:195: writing smithi08477203-45 from 413472 to 1033612 tid 1 2024-09-20T22:43:28.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:28.132 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:28.132 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:43:28.134 INFO:tasks.rados.rados.0.smithi084.stdout:195: writing smithi08477203-45 from 1818036 to 2328736 tid 2 2024-09-20T22:43:28.142 INFO:tasks.rados.rados.0.smithi084.stdout:195: writing smithi08477203-45 from 3120019 to 3613472 tid 3 2024-09-20T22:43:28.142 INFO:tasks.rados.rados.0.smithi084.stdout:196: read oid 50 snap -1 2024-09-20T22:43:28.142 INFO:tasks.rados.rados.0.smithi084.stdout:196: expect deleted 2024-09-20T22:43:28.142 INFO:tasks.rados.rados.0.smithi084.stdout:197: write (excl) oid 44 current snap is 19 2024-09-20T22:43:28.142 INFO:tasks.rados.rados.0.smithi084.stdout:197: seq_num 97 ranges {430279=695389,1876518=654358,3031089=536797} 2024-09-20T22:43:28.148 INFO:tasks.rados.rados.0.smithi084.stdout:197: writing smithi08477203-44 from 430279 to 1125668 tid 1 2024-09-20T22:43:28.156 INFO:tasks.rados.rados.0.smithi084.stdout:197: writing smithi08477203-44 from 1876518 to 2530876 tid 2 2024-09-20T22:43:28.163 INFO:tasks.rados.rados.0.smithi084.stdout:197: writing smithi08477203-44 from 3031089 to 3567886 tid 3 2024-09-20T22:43:28.164 INFO:tasks.rados.rados.0.smithi084.stdout:198: copy_from oid 9 from oid 30 current snap is 19 2024-09-20T22:43:28.164 INFO:tasks.rados.rados.0.smithi084.stdout:194: finishing rollback tid 0 to smithi08477203-48 2024-09-20T22:43:28.164 INFO:tasks.rados.rados.0.smithi084.stdout:194: finishing rollback tid 1 to smithi08477203-48 2024-09-20T22:43:28.164 INFO:tasks.rados.rados.0.smithi084.stdout:194: finishing rollback tid 2 to smithi08477203-48 2024-09-20T22:43:28.164 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 48 v 32 (ObjNum 50 snap 0 seq_num 50) dirty exists 2024-09-20T22:43:28.164 INFO:tasks.rados.rados.0.smithi084.stdout:191: finishing copy_from racing read to smithi08477203-41 2024-09-20T22:43:28.164 INFO:tasks.rados.rados.0.smithi084.stdout:191: finishing copy_from to smithi08477203-41 2024-09-20T22:43:28.164 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 41 v 9 (ObjNum 39 snap 0 seq_num 39) dirty exists 2024-09-20T22:43:28.164 INFO:tasks.rados.rados.0.smithi084.stdout:195: finishing write tid 1 to smithi08477203-45 2024-09-20T22:43:28.164 INFO:tasks.rados.rados.0.smithi084.stdout:197: finishing write tid 1 to smithi08477203-44 2024-09-20T22:43:28.164 INFO:tasks.rados.rados.0.smithi084.stdout:195: finishing write tid 2 to smithi08477203-45 2024-09-20T22:43:28.165 INFO:tasks.rados.rados.0.smithi084.stdout:195: finishing write tid 3 to smithi08477203-45 2024-09-20T22:43:28.165 INFO:tasks.rados.rados.0.smithi084.stdout:195: finishing write tid 4 to smithi08477203-45 2024-09-20T22:43:28.165 INFO:tasks.rados.rados.0.smithi084.stdout:192: finishing copy_from racing read to smithi08477203-6 2024-09-20T22:43:28.165 INFO:tasks.rados.rados.0.smithi084.stdout:191: done (7 left) 2024-09-20T22:43:28.165 INFO:tasks.rados.rados.0.smithi084.stdout:193: done (6 left) 2024-09-20T22:43:28.165 INFO:tasks.rados.rados.0.smithi084.stdout:194: done (5 left) 2024-09-20T22:43:28.165 INFO:tasks.rados.rados.0.smithi084.stdout:196: done (4 left) 2024-09-20T22:43:28.166 INFO:tasks.rados.rados.0.smithi084.stdout:199: write (excl) oid 41 current snap is 19 2024-09-20T22:43:28.166 INFO:tasks.rados.rados.0.smithi084.stdout:199: seq_num 98 ranges {447086=770638,1935000=798016,3458512=188574} 2024-09-20T22:43:28.169 INFO:tasks.rados.rados.0.smithi084.stdout:199: writing smithi08477203-41 from 447086 to 1217724 tid 1 2024-09-20T22:43:28.178 INFO:tasks.rados.rados.0.smithi084.stdout:199: writing smithi08477203-41 from 1935000 to 2733016 tid 2 2024-09-20T22:43:28.183 INFO:tasks.rados.rados.0.smithi084.stdout:199: writing smithi08477203-41 from 3458512 to 3647086 tid 3 2024-09-20T22:43:28.183 INFO:tasks.rados.rados.0.smithi084.stdout:200: snap_create 2024-09-20T22:43:28.183 INFO:tasks.rados.rados.0.smithi084.stdout:195: finishing write tid 5 to smithi08477203-45 2024-09-20T22:43:28.183 INFO:tasks.rados.rados.0.smithi084.stdout:195: oid 45 updating version 0 to 28 2024-09-20T22:43:28.184 INFO:tasks.rados.rados.0.smithi084.stdout:195: oid 45 updating version 28 to 29 2024-09-20T22:43:28.184 INFO:tasks.rados.rados.0.smithi084.stdout:195: oid 45 version 29 is already newer than 27 2024-09-20T22:43:28.184 INFO:tasks.rados.rados.0.smithi084.stdout:195: oid 45 version 29 is already newer than 26 2024-09-20T22:43:28.184 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 45 v 29 (ObjNum 96 snap 19 seq_num 96) dirty exists 2024-09-20T22:43:28.184 INFO:tasks.rados.rados.0.smithi084.stdout:195: left oid 45 (ObjNum 96 snap 19 seq_num 96) 2024-09-20T22:43:28.184 INFO:tasks.rados.rados.0.smithi084.stdout:192: finishing copy_from to smithi08477203-6 2024-09-20T22:43:28.184 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 6 v 6 (ObjNum 42 snap 0 seq_num 42) dirty exists 2024-09-20T22:43:28.184 INFO:tasks.rados.rados.0.smithi084.stdout:197: finishing write tid 2 to smithi08477203-44 2024-09-20T22:43:28.184 INFO:tasks.rados.rados.0.smithi084.stdout:197: finishing write tid 3 to smithi08477203-44 2024-09-20T22:43:28.184 INFO:tasks.rados.rados.0.smithi084.stdout:197: finishing write tid 4 to smithi08477203-44 2024-09-20T22:43:28.184 INFO:tasks.rados.rados.0.smithi084.stdout:197: finishing write tid 5 to smithi08477203-44 2024-09-20T22:43:28.184 INFO:tasks.rados.rados.0.smithi084.stdout:197: oid 44 updating version 0 to 59 2024-09-20T22:43:28.184 INFO:tasks.rados.rados.0.smithi084.stdout:197: oid 44 version 59 is already newer than 58 2024-09-20T22:43:28.184 INFO:tasks.rados.rados.0.smithi084.stdout:197: oid 44 updating version 59 to 60 2024-09-20T22:43:28.184 INFO:tasks.rados.rados.0.smithi084.stdout:197: oid 44 version 60 is already newer than 57 2024-09-20T22:43:28.185 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 44 v 60 (ObjNum 97 snap 19 seq_num 97) dirty exists 2024-09-20T22:43:28.185 INFO:tasks.rados.rados.0.smithi084.stdout:197: left oid 44 (ObjNum 97 snap 19 seq_num 97) 2024-09-20T22:43:28.185 INFO:tasks.rados.rados.0.smithi084.stdout:198: finishing copy_from racing read to smithi08477203-9 2024-09-20T22:43:28.185 INFO:tasks.rados.rados.0.smithi084.stdout:199: finishing write tid 1 to smithi08477203-41 2024-09-20T22:43:28.185 INFO:tasks.rados.rados.0.smithi084.stdout:198: finishing copy_from to smithi08477203-9 2024-09-20T22:43:28.185 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 9 v 25 (ObjNum 58 snap 4 seq_num 58) dirty exists 2024-09-20T22:43:28.185 INFO:tasks.rados.rados.0.smithi084.stdout:199: finishing write tid 2 to smithi08477203-41 2024-09-20T22:43:28.185 INFO:tasks.rados.rados.0.smithi084.stdout:199: finishing write tid 3 to smithi08477203-41 2024-09-20T22:43:28.185 INFO:tasks.rados.rados.0.smithi084.stdout:199: finishing write tid 4 to smithi08477203-41 2024-09-20T22:43:28.186 INFO:tasks.rados.rados.0.smithi084.stdout:199: finishing write tid 5 to smithi08477203-41 2024-09-20T22:43:28.186 INFO:tasks.rados.rados.0.smithi084.stdout:199: oid 41 updating version 0 to 14 2024-09-20T22:43:28.187 INFO:tasks.rados.rados.0.smithi084.stdout:199: oid 41 version 14 is already newer than 12 2024-09-20T22:43:28.187 INFO:tasks.rados.rados.0.smithi084.stdout:199: oid 41 version 14 is already newer than 13 2024-09-20T22:43:28.187 INFO:tasks.rados.rados.0.smithi084.stdout:199: oid 41 version 14 is already newer than 11 2024-09-20T22:43:28.187 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 41 v 14 (ObjNum 98 snap 19 seq_num 98) dirty exists 2024-09-20T22:43:28.187 INFO:tasks.rados.rados.0.smithi084.stdout:199: left oid 41 (ObjNum 98 snap 19 seq_num 98) 2024-09-20T22:43:28.232 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:43:28.254 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:28.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:28.255 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:43:28.355 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:43:28.364 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:28 smithi084 ceph-mon[29774]: 4.6 scrub starts 2024-09-20T22:43:28.364 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:28 smithi084 ceph-mon[29774]: 4.6 scrub ok 2024-09-20T22:43:28.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:28 smithi084 ceph-mon[29774]: 1.1 scrub starts 2024-09-20T22:43:28.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:28 smithi084 ceph-mon[29774]: 1.1 scrub ok 2024-09-20T22:43:28.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:28 smithi084 ceph-mon[29774]: 1.6 scrub starts 2024-09-20T22:43:28.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:28 smithi084 ceph-mon[29774]: 1.6 scrub ok 2024-09-20T22:43:28.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:28 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/3028649890' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:43:28.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:28 smithi084 ceph-mon[29774]: osdmap e129: 8 total, 8 up, 8 in 2024-09-20T22:43:28.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:28 smithi084 ceph-mon[40024]: 4.6 scrub starts 2024-09-20T22:43:28.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:28 smithi084 ceph-mon[40024]: 4.6 scrub ok 2024-09-20T22:43:28.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:28 smithi084 ceph-mon[40024]: 1.1 scrub starts 2024-09-20T22:43:28.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:28 smithi084 ceph-mon[40024]: 1.1 scrub ok 2024-09-20T22:43:28.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:28 smithi084 ceph-mon[40024]: 1.6 scrub starts 2024-09-20T22:43:28.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:28 smithi084 ceph-mon[40024]: 1.6 scrub ok 2024-09-20T22:43:28.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:28 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/3028649890' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:43:28.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:28 smithi084 ceph-mon[40024]: osdmap e129: 8 total, 8 up, 8 in 2024-09-20T22:43:28.368 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-20T22:43:28.368 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:43:28.376 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:28.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:28.377 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:43:28.422 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-20T22:43:28.479 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:43:28.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:28.502 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:28.502 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:43:28.602 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:43:28.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:28.625 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:28.626 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:43:28.726 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:43:28.748 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:28.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:28.749 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:43:28.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:28 smithi154 ceph-mon[33264]: 4.6 scrub starts 2024-09-20T22:43:28.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:28 smithi154 ceph-mon[33264]: 4.6 scrub ok 2024-09-20T22:43:28.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:28 smithi154 ceph-mon[33264]: 1.1 scrub starts 2024-09-20T22:43:28.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:28 smithi154 ceph-mon[33264]: 1.1 scrub ok 2024-09-20T22:43:28.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:28 smithi154 ceph-mon[33264]: 1.6 scrub starts 2024-09-20T22:43:28.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:28 smithi154 ceph-mon[33264]: 1.6 scrub ok 2024-09-20T22:43:28.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:28 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/3028649890' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:43:28.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:28 smithi154 ceph-mon[33264]: osdmap e129: 8 total, 8 up, 8 in 2024-09-20T22:43:28.850 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:43:28.871 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:28.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:28.872 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:43:28.973 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:43:28.995 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:28.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:28.996 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:43:29.097 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:43:29.118 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:29.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:29.119 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:43:29.126 INFO:tasks.rados.rados.0.smithi084.stdout:192: done (5 left) 2024-09-20T22:43:29.126 INFO:tasks.rados.rados.0.smithi084.stdout:195: done (4 left) 2024-09-20T22:43:29.126 INFO:tasks.rados.rados.0.smithi084.stdout:197: done (3 left) 2024-09-20T22:43:29.126 INFO:tasks.rados.rados.0.smithi084.stdout:198: done (2 left) 2024-09-20T22:43:29.126 INFO:tasks.rados.rados.0.smithi084.stdout:199: done (1 left) 2024-09-20T22:43:29.127 INFO:tasks.rados.rados.0.smithi084.stdout:200: done (0 left) 2024-09-20T22:43:29.127 INFO:tasks.rados.rados.0.smithi084.stdout:201: write oid 27 current snap is 20 2024-09-20T22:43:29.127 INFO:tasks.rados.rados.0.smithi084.stdout:201: seq_num 99 ranges {463893=562240,1593482=658027,2685935=559341} 2024-09-20T22:43:29.131 INFO:tasks.rados.rados.0.smithi084.stdout:201: writing smithi08477203-27 from 463893 to 1026133 tid 1 2024-09-20T22:43:29.139 INFO:tasks.rados.rados.0.smithi084.stdout:201: writing smithi08477203-27 from 1593482 to 2251509 tid 2 2024-09-20T22:43:29.145 INFO:tasks.rados.rados.0.smithi084.stdout:201: writing smithi08477203-27 from 2685935 to 3245276 tid 3 2024-09-20T22:43:29.145 INFO:tasks.rados.rados.0.smithi084.stdout:202: write (excl) oid 20 current snap is 20 2024-09-20T22:43:29.145 INFO:tasks.rados.rados.0.smithi084.stdout:201: finishing write tid 1 to smithi08477203-27 2024-09-20T22:43:29.145 INFO:tasks.rados.rados.0.smithi084.stdout:202: seq_num 100 ranges {480700=637489,1651964=401685,2713358=770613} 2024-09-20T22:43:29.151 INFO:tasks.rados.rados.0.smithi084.stdout:202: writing smithi08477203-20 from 480700 to 1118189 tid 1 2024-09-20T22:43:29.156 INFO:tasks.rados.rados.0.smithi084.stdout:202: writing smithi08477203-20 from 1651964 to 2053649 tid 2 2024-09-20T22:43:29.164 INFO:tasks.rados.rados.0.smithi084.stdout:202: writing smithi08477203-20 from 2713358 to 3483971 tid 3 2024-09-20T22:43:29.164 INFO:tasks.rados.rados.0.smithi084.stdout:203: write (excl) oid 5 current snap is 20 2024-09-20T22:43:29.165 INFO:tasks.rados.rados.0.smithi084.stdout:201: finishing write tid 2 to smithi08477203-27 2024-09-20T22:43:29.165 INFO:tasks.rados.rados.0.smithi084.stdout:203: seq_num 101 ranges {497507=712738,1710446=661696,3140781=556726} 2024-09-20T22:43:29.171 INFO:tasks.rados.rados.0.smithi084.stdout:203: writing smithi08477203-5 from 497507 to 1210245 tid 1 2024-09-20T22:43:29.178 INFO:tasks.rados.rados.0.smithi084.stdout:203: writing smithi08477203-5 from 1710446 to 2372142 tid 2 2024-09-20T22:43:29.187 INFO:tasks.rados.rados.0.smithi084.stdout:203: writing smithi08477203-5 from 3140781 to 3697507 tid 3 2024-09-20T22:43:29.188 INFO:tasks.rados.rados.0.smithi084.stdout:201: finishing write tid 3 to smithi08477203-27 2024-09-20T22:43:29.188 INFO:tasks.rados.rados.0.smithi084.stdout:201: finishing write tid 4 to smithi08477203-27 2024-09-20T22:43:29.188 INFO:tasks.rados.rados.0.smithi084.stdout:201: finishing write tid 5 to smithi08477203-27 2024-09-20T22:43:29.188 INFO:tasks.rados.rados.0.smithi084.stdout:201: oid 27 updating version 0 to 33 2024-09-20T22:43:29.188 INFO:tasks.rados.rados.0.smithi084.stdout:201: oid 27 version 33 is already newer than 31 2024-09-20T22:43:29.188 INFO:tasks.rados.rados.0.smithi084.stdout:201: oid 27 version 33 is already newer than 32 2024-09-20T22:43:29.188 INFO:tasks.rados.rados.0.smithi084.stdout:201: oid 27 version 33 is already newer than 30 2024-09-20T22:43:29.188 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 27 v 33 (ObjNum 99 snap 20 seq_num 99) dirty exists 2024-09-20T22:43:29.188 INFO:tasks.rados.rados.0.smithi084.stdout:201: left oid 27 (ObjNum 99 snap 20 seq_num 99) 2024-09-20T22:43:29.189 INFO:tasks.rados.rados.0.smithi084.stdout:202: finishing write tid 1 to smithi08477203-20 2024-09-20T22:43:29.189 INFO:tasks.rados.rados.0.smithi084.stdout:202: finishing write tid 2 to smithi08477203-20 2024-09-20T22:43:29.189 INFO:tasks.rados.rados.0.smithi084.stdout:202: finishing write tid 3 to smithi08477203-20 2024-09-20T22:43:29.189 INFO:tasks.rados.rados.0.smithi084.stdout:202: finishing write tid 4 to smithi08477203-20 2024-09-20T22:43:29.189 INFO:tasks.rados.rados.0.smithi084.stdout:202: finishing write tid 5 to smithi08477203-20 2024-09-20T22:43:29.189 INFO:tasks.rados.rados.0.smithi084.stdout:202: oid 20 updating version 0 to 27 2024-09-20T22:43:29.189 INFO:tasks.rados.rados.0.smithi084.stdout:202: oid 20 updating version 27 to 28 2024-09-20T22:43:29.189 INFO:tasks.rados.rados.0.smithi084.stdout:202: oid 20 version 28 is already newer than 25 2024-09-20T22:43:29.189 INFO:tasks.rados.rados.0.smithi084.stdout:202: oid 20 version 28 is already newer than 26 2024-09-20T22:43:29.189 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 20 v 28 (ObjNum 100 snap 20 seq_num 100) dirty exists 2024-09-20T22:43:29.189 INFO:tasks.rados.rados.0.smithi084.stdout:202: left oid 20 (ObjNum 100 snap 20 seq_num 100) 2024-09-20T22:43:29.189 INFO:tasks.rados.rados.0.smithi084.stdout:203: finishing write tid 1 to smithi08477203-5 2024-09-20T22:43:29.189 INFO:tasks.rados.rados.0.smithi084.stdout:203: finishing write tid 2 to smithi08477203-5 2024-09-20T22:43:29.189 INFO:tasks.rados.rados.0.smithi084.stdout:201: done (2 left) 2024-09-20T22:43:29.190 INFO:tasks.rados.rados.0.smithi084.stdout:202: done (1 left) 2024-09-20T22:43:29.190 INFO:tasks.rados.rados.0.smithi084.stdout:204: delete oid 21 current snap is 20 2024-09-20T22:43:29.190 INFO:tasks.rados.rados.0.smithi084.stdout:204: done (1 left) 2024-09-20T22:43:29.190 INFO:tasks.rados.rados.0.smithi084.stdout:205: snap_remove snap 13 2024-09-20T22:43:29.220 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:43:29.223 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:43:29.241 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:29.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:29.243 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:43:29.254 INFO:journalctl@ceph.iscsi.iscsi.a.smithi154.stdout:Sep 20 22:43:28 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-iscsi-iscsi-a[59730]: debug there is no tcmu-runner data available 2024-09-20T22:43:29.268 INFO:journalctl@ceph.alertmanager.a.smithi084.stdout:Sep 20 22:43:28 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-alertmanager-a[80657]: ts=2024-09-20T22:43:28.868Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.00248563s 2024-09-20T22:43:29.344 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:43:29.365 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:29.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:29.367 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:43:29.467 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:43:29.490 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:29.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:29.491 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:43:29.520 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:29 smithi084 ceph-mon[40024]: pgmap v94: 168 pgs: 10 peering, 158 active+clean; 122 MiB data, 474 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:43:29.520 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:29 smithi084 ceph-mon[40024]: 4.12 scrub starts 2024-09-20T22:43:29.520 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:29 smithi084 ceph-mon[40024]: 4.12 scrub ok 2024-09-20T22:43:29.520 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:29 smithi084 ceph-mon[40024]: 4.1f scrub starts 2024-09-20T22:43:29.520 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:29 smithi084 ceph-mon[40024]: 4.1f scrub ok 2024-09-20T22:43:29.520 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:29 smithi084 ceph-mon[40024]: osdmap e130: 8 total, 8 up, 8 in 2024-09-20T22:43:29.521 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:29 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:43:29.521 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:29 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:43:29.521 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:29 smithi084 ceph-mon[29774]: pgmap v94: 168 pgs: 10 peering, 158 active+clean; 122 MiB data, 474 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:43:29.521 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:29 smithi084 ceph-mon[29774]: 4.12 scrub starts 2024-09-20T22:43:29.521 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:29 smithi084 ceph-mon[29774]: 4.12 scrub ok 2024-09-20T22:43:29.521 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:29 smithi084 ceph-mon[29774]: 4.1f scrub starts 2024-09-20T22:43:29.521 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:29 smithi084 ceph-mon[29774]: 4.1f scrub ok 2024-09-20T22:43:29.521 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:29 smithi084 ceph-mon[29774]: osdmap e130: 8 total, 8 up, 8 in 2024-09-20T22:43:29.521 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:29 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:43:29.521 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:29 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:43:29.592 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:43:29.601 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:29 smithi154 ceph-mon[33264]: pgmap v94: 168 pgs: 10 peering, 158 active+clean; 122 MiB data, 474 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:43:29.601 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:29 smithi154 ceph-mon[33264]: 4.12 scrub starts 2024-09-20T22:43:29.601 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:29 smithi154 ceph-mon[33264]: 4.12 scrub ok 2024-09-20T22:43:29.601 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:29 smithi154 ceph-mon[33264]: 4.1f scrub starts 2024-09-20T22:43:29.601 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:29 smithi154 ceph-mon[33264]: 4.1f scrub ok 2024-09-20T22:43:29.601 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:29 smithi154 ceph-mon[33264]: osdmap e130: 8 total, 8 up, 8 in 2024-09-20T22:43:29.602 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:29 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:43:29.602 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:29 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:43:29.616 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:29.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:29.617 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:43:29.631 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.4 dump_blocked_ops 2024-09-20T22:43:29.717 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:43:29.739 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:29.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:29.740 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:43:29.840 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:43:29.862 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:29.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:29.863 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:43:29.893 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:43:29.964 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:43:29.985 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:29.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:29.986 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:43:30.086 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:43:30.109 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:30.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:30.110 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:43:30.123 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is unset 2024-09-20T22:43:30.136 INFO:tasks.rados.rados.0.smithi084.stdout:205: done (1 left) 2024-09-20T22:43:30.136 INFO:tasks.rados.rados.0.smithi084.stdout:206: snap_remove snap 18 2024-09-20T22:43:30.211 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:43:30.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:30.234 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:30.234 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:43:30.335 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:43:30.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:30.357 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:30.358 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:43:30.364 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:30 smithi084 ceph-mon[40024]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:43:30.364 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:30 smithi084 ceph-mon[40024]: 1.b scrub starts 2024-09-20T22:43:30.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:30 smithi084 ceph-mon[40024]: 1.b scrub ok 2024-09-20T22:43:30.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:30 smithi084 ceph-mon[40024]: 1.14 scrub starts 2024-09-20T22:43:30.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:30 smithi084 ceph-mon[40024]: 1.14 scrub ok 2024-09-20T22:43:30.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:30 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/557358210' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:43:30.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:30 smithi084 ceph-mon[40024]: 1.e scrub starts 2024-09-20T22:43:30.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:30 smithi084 ceph-mon[40024]: 1.e scrub ok 2024-09-20T22:43:30.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:30 smithi084 ceph-mon[40024]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T22:43:30.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:30 smithi084 ceph-mon[40024]: Cluster is now healthy 2024-09-20T22:43:30.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:30 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/557358210' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T22:43:30.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:30 smithi084 ceph-mon[40024]: osdmap e131: 8 total, 8 up, 8 in 2024-09-20T22:43:30.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:30 smithi084 ceph-mon[29774]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:43:30.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:30 smithi084 ceph-mon[29774]: 1.b scrub starts 2024-09-20T22:43:30.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:30 smithi084 ceph-mon[29774]: 1.b scrub ok 2024-09-20T22:43:30.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:30 smithi084 ceph-mon[29774]: 1.14 scrub starts 2024-09-20T22:43:30.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:30 smithi084 ceph-mon[29774]: 1.14 scrub ok 2024-09-20T22:43:30.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:30 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/557358210' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:43:30.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:30 smithi084 ceph-mon[29774]: 1.e scrub starts 2024-09-20T22:43:30.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:30 smithi084 ceph-mon[29774]: 1.e scrub ok 2024-09-20T22:43:30.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:30 smithi084 ceph-mon[29774]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T22:43:30.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:30 smithi084 ceph-mon[29774]: Cluster is now healthy 2024-09-20T22:43:30.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:30 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/557358210' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T22:43:30.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:30 smithi084 ceph-mon[29774]: osdmap e131: 8 total, 8 up, 8 in 2024-09-20T22:43:30.411 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-20T22:43:30.411 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:43:30.459 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:43:30.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:30.481 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:30.481 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:43:30.503 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-20T22:43:30.503 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-20T22:43:30.503 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_0 2024-09-20T22:43:30.504 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph pg dump --format=json 2024-09-20T22:43:30.581 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:43:30.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:30.603 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:30.603 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:43:30.704 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:43:30.726 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:30.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:30.727 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:43:30.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:30 smithi154 ceph-mon[33264]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:43:30.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:30 smithi154 ceph-mon[33264]: 1.b scrub starts 2024-09-20T22:43:30.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:30 smithi154 ceph-mon[33264]: 1.b scrub ok 2024-09-20T22:43:30.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:30 smithi154 ceph-mon[33264]: 1.14 scrub starts 2024-09-20T22:43:30.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:30 smithi154 ceph-mon[33264]: 1.14 scrub ok 2024-09-20T22:43:30.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:30 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/557358210' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:43:30.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:30 smithi154 ceph-mon[33264]: 1.e scrub starts 2024-09-20T22:43:30.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:30 smithi154 ceph-mon[33264]: 1.e scrub ok 2024-09-20T22:43:30.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:30 smithi154 ceph-mon[33264]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T22:43:30.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:30 smithi154 ceph-mon[33264]: Cluster is now healthy 2024-09-20T22:43:30.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:30 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/557358210' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T22:43:30.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:30 smithi154 ceph-mon[33264]: osdmap e131: 8 total, 8 up, 8 in 2024-09-20T22:43:30.828 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:43:30.849 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:30.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:30.850 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:43:30.951 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:43:30.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:30.974 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:30.974 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:43:31.075 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:43:31.083 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-20T22:43:31.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:31.097 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:31.097 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:43:31.133 INFO:tasks.rados.rados.0.smithi084.stdout:206: done (1 left) 2024-09-20T22:43:31.134 INFO:tasks.rados.rados.0.smithi084.stdout:207: rollback oid 19 current snap is 20 2024-09-20T22:43:31.134 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 19 to 16 2024-09-20T22:43:31.134 INFO:tasks.rados.rados.0.smithi084.stdout:203: finishing write tid 3 to smithi08477203-5 2024-09-20T22:43:31.134 INFO:tasks.rados.rados.0.smithi084.stdout:203: finishing write tid 4 to smithi08477203-5 2024-09-20T22:43:31.134 INFO:tasks.rados.rados.0.smithi084.stdout:203: finishing write tid 5 to smithi08477203-5 2024-09-20T22:43:31.134 INFO:tasks.rados.rados.0.smithi084.stdout:203: oid 5 updating version 0 to 27 2024-09-20T22:43:31.134 INFO:tasks.rados.rados.0.smithi084.stdout:203: oid 5 updating version 27 to 28 2024-09-20T22:43:31.134 INFO:tasks.rados.rados.0.smithi084.stdout:203: oid 5 updating version 28 to 29 2024-09-20T22:43:31.134 INFO:tasks.rados.rados.0.smithi084.stdout:203: oid 5 version 29 is already newer than 26 2024-09-20T22:43:31.134 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 5 v 29 (ObjNum 101 snap 20 seq_num 101) dirty exists 2024-09-20T22:43:31.134 INFO:tasks.rados.rados.0.smithi084.stdout:203: left oid 5 (ObjNum 101 snap 20 seq_num 101) 2024-09-20T22:43:31.134 INFO:tasks.rados.rados.0.smithi084.stdout:203: done (1 left) 2024-09-20T22:43:31.134 INFO:tasks.rados.rados.0.smithi084.stdout:208: read oid 40 snap -1 2024-09-20T22:43:31.135 INFO:tasks.rados.rados.0.smithi084.stdout:208: expect (ObjNum 39 snap 0 seq_num 39) 2024-09-20T22:43:31.135 INFO:tasks.rados.rados.0.smithi084.stdout:209: write (excl) oid 4 current snap is 20 2024-09-20T22:43:31.135 INFO:tasks.rados.rados.0.smithi084.stdout:209: seq_num 103 ranges {531121=463236,1711057=665365,3079274=651847} 2024-09-20T22:43:31.140 INFO:tasks.rados.rados.0.smithi084.stdout:209: writing smithi08477203-4 from 531121 to 994357 tid 1 2024-09-20T22:43:31.147 INFO:tasks.rados.rados.0.smithi084.stdout:209: writing smithi08477203-4 from 1711057 to 2376422 tid 2 2024-09-20T22:43:31.155 INFO:tasks.rados.rados.0.smithi084.stdout:209: writing smithi08477203-4 from 3079274 to 3731121 tid 3 2024-09-20T22:43:31.155 INFO:tasks.rados.rados.0.smithi084.stdout:210: copy_from oid 36 from oid 23 current snap is 20 2024-09-20T22:43:31.155 INFO:tasks.rados.rados.0.smithi084.stdout:208: expect (ObjNum 39 snap 0 seq_num 39) 2024-09-20T22:43:31.186 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.4 dump_historic_ops 2024-09-20T22:43:31.198 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:43:31.220 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:31.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:31.221 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:43:31.244 INFO:tasks.rados.rados.0.smithi084.stdout:207: finishing rollback tid 0 to smithi08477203-19 2024-09-20T22:43:31.244 INFO:tasks.rados.rados.0.smithi084.stdout:207: finishing rollback tid 1 to smithi08477203-19 2024-09-20T22:43:31.244 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 19 v 65 (ObjNum 33 snap 0 seq_num 2091096112) dirty dne 2024-09-20T22:43:31.244 INFO:tasks.rados.rados.0.smithi084.stdout:209: finishing write tid 1 to smithi08477203-4 2024-09-20T22:43:31.244 INFO:tasks.rados.rados.0.smithi084.stdout:209: finishing write tid 2 to smithi08477203-4 2024-09-20T22:43:31.244 INFO:tasks.rados.rados.0.smithi084.stdout:207: done (3 left) 2024-09-20T22:43:31.244 INFO:tasks.rados.rados.0.smithi084.stdout:208: done (2 left) 2024-09-20T22:43:31.244 INFO:tasks.rados.rados.0.smithi084.stdout:211: copy_from oid 38 from oid 28 current snap is 20 2024-09-20T22:43:31.244 INFO:tasks.rados.rados.0.smithi084.stdout:210: finishing copy_from to smithi08477203-36 2024-09-20T22:43:31.244 INFO:tasks.rados.rados.0.smithi084.stdout:210: got expected ENOENT (src dne) 2024-09-20T22:43:31.245 INFO:tasks.rados.rados.0.smithi084.stdout:210: finishing copy_from racing read to smithi08477203-36 2024-09-20T22:43:31.245 INFO:tasks.rados.rados.0.smithi084.stdout:209: finishing write tid 3 to smithi08477203-4 2024-09-20T22:43:31.245 INFO:tasks.rados.rados.0.smithi084.stdout:209: finishing write tid 4 to smithi08477203-4 2024-09-20T22:43:31.245 INFO:tasks.rados.rados.0.smithi084.stdout:209: finishing write tid 5 to smithi08477203-4 2024-09-20T22:43:31.245 INFO:tasks.rados.rados.0.smithi084.stdout:209: oid 4 updating version 0 to 69 2024-09-20T22:43:31.245 INFO:tasks.rados.rados.0.smithi084.stdout:209: oid 4 updating version 69 to 70 2024-09-20T22:43:31.245 INFO:tasks.rados.rados.0.smithi084.stdout:209: oid 4 updating version 70 to 72 2024-09-20T22:43:31.245 INFO:tasks.rados.rados.0.smithi084.stdout:209: oid 4 version 72 is already newer than 71 2024-09-20T22:43:31.245 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 4 v 72 (ObjNum 103 snap 20 seq_num 103) dirty exists 2024-09-20T22:43:31.245 INFO:tasks.rados.rados.0.smithi084.stdout:209: left oid 4 (ObjNum 103 snap 20 seq_num 103) 2024-09-20T22:43:31.245 INFO:tasks.rados.rados.0.smithi084.stdout:209: done (2 left) 2024-09-20T22:43:31.245 INFO:tasks.rados.rados.0.smithi084.stdout:210: done (1 left) 2024-09-20T22:43:31.245 INFO:tasks.rados.rados.0.smithi084.stdout:212: snap_create 2024-09-20T22:43:31.252 INFO:tasks.rados.rados.0.smithi084.stdout:211: finishing copy_from racing read to smithi08477203-38 2024-09-20T22:43:31.259 INFO:tasks.rados.rados.0.smithi084.stdout:211: finishing copy_from to smithi08477203-38 2024-09-20T22:43:31.259 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 38 v 45 (ObjNum 76 snap 9 seq_num 76) dirty exists 2024-09-20T22:43:31.322 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:43:31.332 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:43:31.346 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:31.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:31.347 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:43:31.374 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:43:31.436 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:43:31.449 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:43:31.458 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:31 smithi084 ceph-mon[29774]: pgmap v97: 168 pgs: 168 active+clean; 144 MiB data, 526 MiB used, 715 GiB / 715 GiB avail; 9.3 MiB/s rd, 5.4 MiB/s wr, 19 op/s 2024-09-20T22:43:31.458 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:31 smithi084 ceph-mon[29774]: 3.6 scrub starts 2024-09-20T22:43:31.459 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:31 smithi084 ceph-mon[29774]: 3.6 scrub ok 2024-09-20T22:43:31.459 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:31 smithi084 ceph-mon[29774]: 1.12 scrub starts 2024-09-20T22:43:31.459 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:31 smithi084 ceph-mon[29774]: 1.12 scrub ok 2024-09-20T22:43:31.459 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:31 smithi084 ceph-mon[29774]: 1.c scrub starts 2024-09-20T22:43:31.459 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:31 smithi084 ceph-mon[29774]: 1.c scrub ok 2024-09-20T22:43:31.459 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:31 smithi084 ceph-mon[29774]: 1.d scrub starts 2024-09-20T22:43:31.459 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:31 smithi084 ceph-mon[29774]: 1.d scrub ok 2024-09-20T22:43:31.459 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:31 smithi084 ceph-mon[29774]: 1.13 scrub starts 2024-09-20T22:43:31.459 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:31 smithi084 ceph-mon[29774]: 1.13 scrub ok 2024-09-20T22:43:31.459 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:31 smithi084 ceph-mon[29774]: 1.f scrub starts 2024-09-20T22:43:31.459 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:31 smithi084 ceph-mon[29774]: 1.f scrub ok 2024-09-20T22:43:31.459 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:31 smithi084 ceph-mon[29774]: 4.1a scrub starts 2024-09-20T22:43:31.459 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:31 smithi084 ceph-mon[29774]: 4.1a scrub ok 2024-09-20T22:43:31.460 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:31 smithi084 ceph-mon[29774]: osdmap e132: 8 total, 8 up, 8 in 2024-09-20T22:43:31.460 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:31 smithi084 ceph-mon[40024]: pgmap v97: 168 pgs: 168 active+clean; 144 MiB data, 526 MiB used, 715 GiB / 715 GiB avail; 9.3 MiB/s rd, 5.4 MiB/s wr, 19 op/s 2024-09-20T22:43:31.460 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:31 smithi084 ceph-mon[40024]: 3.6 scrub starts 2024-09-20T22:43:31.460 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:31 smithi084 ceph-mon[40024]: 3.6 scrub ok 2024-09-20T22:43:31.460 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:31 smithi084 ceph-mon[40024]: 1.12 scrub starts 2024-09-20T22:43:31.460 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:31 smithi084 ceph-mon[40024]: 1.12 scrub ok 2024-09-20T22:43:31.460 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:31 smithi084 ceph-mon[40024]: 1.c scrub starts 2024-09-20T22:43:31.460 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:31 smithi084 ceph-mon[40024]: 1.c scrub ok 2024-09-20T22:43:31.460 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:31 smithi084 ceph-mon[40024]: 1.d scrub starts 2024-09-20T22:43:31.460 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:31 smithi084 ceph-mon[40024]: 1.d scrub ok 2024-09-20T22:43:31.460 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:31 smithi084 ceph-mon[40024]: 1.13 scrub starts 2024-09-20T22:43:31.460 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:31 smithi084 ceph-mon[40024]: 1.13 scrub ok 2024-09-20T22:43:31.460 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:31 smithi084 ceph-mon[40024]: 1.f scrub starts 2024-09-20T22:43:31.460 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:31 smithi084 ceph-mon[40024]: 1.f scrub ok 2024-09-20T22:43:31.461 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:31 smithi084 ceph-mon[40024]: 4.1a scrub starts 2024-09-20T22:43:31.461 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:31 smithi084 ceph-mon[40024]: 4.1a scrub ok 2024-09-20T22:43:31.461 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:31 smithi084 ceph-mon[40024]: osdmap e132: 8 total, 8 up, 8 in 2024-09-20T22:43:31.471 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:31.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:31.472 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:43:31.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:31 smithi154 ceph-mon[33264]: pgmap v97: 168 pgs: 168 active+clean; 144 MiB data, 526 MiB used, 715 GiB / 715 GiB avail; 9.3 MiB/s rd, 5.4 MiB/s wr, 19 op/s 2024-09-20T22:43:31.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:31 smithi154 ceph-mon[33264]: 3.6 scrub starts 2024-09-20T22:43:31.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:31 smithi154 ceph-mon[33264]: 3.6 scrub ok 2024-09-20T22:43:31.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:31 smithi154 ceph-mon[33264]: 1.12 scrub starts 2024-09-20T22:43:31.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:31 smithi154 ceph-mon[33264]: 1.12 scrub ok 2024-09-20T22:43:31.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:31 smithi154 ceph-mon[33264]: 1.c scrub starts 2024-09-20T22:43:31.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:31 smithi154 ceph-mon[33264]: 1.c scrub ok 2024-09-20T22:43:31.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:31 smithi154 ceph-mon[33264]: 1.d scrub starts 2024-09-20T22:43:31.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:31 smithi154 ceph-mon[33264]: 1.d scrub ok 2024-09-20T22:43:31.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:31 smithi154 ceph-mon[33264]: 1.13 scrub starts 2024-09-20T22:43:31.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:31 smithi154 ceph-mon[33264]: 1.13 scrub ok 2024-09-20T22:43:31.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:31 smithi154 ceph-mon[33264]: 1.f scrub starts 2024-09-20T22:43:31.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:31 smithi154 ceph-mon[33264]: 1.f scrub ok 2024-09-20T22:43:31.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:31 smithi154 ceph-mon[33264]: 4.1a scrub starts 2024-09-20T22:43:31.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:31 smithi154 ceph-mon[33264]: 4.1a scrub ok 2024-09-20T22:43:31.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:31 smithi154 ceph-mon[33264]: osdmap e132: 8 total, 8 up, 8 in 2024-09-20T22:43:31.573 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:43:31.595 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:31.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:31.595 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:43:31.696 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:43:31.717 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:31.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:31.718 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:43:31.818 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:43:31.839 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:31.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:31.840 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:43:31.930 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-20T22:43:31.930 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:43:31.940 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:43:31.961 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:31.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:31.962 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:43:32.063 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:43:32.085 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:32.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:32.085 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:43:32.131 INFO:tasks.rados.rados.0.smithi084.stdout:211: done (1 left) 2024-09-20T22:43:32.131 INFO:tasks.rados.rados.0.smithi084.stdout:212: done (0 left) 2024-09-20T22:43:32.131 INFO:tasks.rados.rados.0.smithi084.stdout:213: write oid 24 current snap is 21 2024-09-20T22:43:32.131 INFO:tasks.rados.rados.0.smithi084.stdout:213: seq_num 104 ranges {547928=538485,1769539=409023,2706697=532054} 2024-09-20T22:43:32.138 INFO:tasks.rados.rados.0.smithi084.stdout:213: writing smithi08477203-24 from 547928 to 1086413 tid 1 2024-09-20T22:43:32.145 INFO:tasks.rados.rados.0.smithi084.stdout:213: writing smithi08477203-24 from 1769539 to 2178562 tid 2 2024-09-20T22:43:32.152 INFO:tasks.rados.rados.0.smithi084.stdout:213: writing smithi08477203-24 from 2706697 to 3238751 tid 3 2024-09-20T22:43:32.153 INFO:tasks.rados.rados.0.smithi084.stdout:214: read oid 34 snap -1 2024-09-20T22:43:32.153 INFO:tasks.rados.rados.0.smithi084.stdout:214: expect (ObjNum 87 snap 11 seq_num 87) 2024-09-20T22:43:32.153 INFO:tasks.rados.rados.0.smithi084.stdout:215: rollback oid 36 current snap is 21 2024-09-20T22:43:32.153 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 36 to 14 2024-09-20T22:43:32.153 INFO:tasks.rados.rados.0.smithi084.stdout:213: finishing write tid 1 to smithi08477203-24 2024-09-20T22:43:32.153 INFO:tasks.rados.rados.0.smithi084.stdout:213: finishing write tid 2 to smithi08477203-24 2024-09-20T22:43:32.153 INFO:tasks.rados.rados.0.smithi084.stdout:216: snap_create 2024-09-20T22:43:32.154 INFO:tasks.rados.rados.0.smithi084.stdout:215: finishing rollback tid 0 to smithi08477203-36 2024-09-20T22:43:32.155 INFO:tasks.rados.rados.0.smithi084.stdout:215: finishing rollback tid 1 to smithi08477203-36 2024-09-20T22:43:32.155 INFO:tasks.rados.rados.0.smithi084.stdout:215: finishing rollback tid 2 to smithi08477203-36 2024-09-20T22:43:32.155 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 36 v 23 (ObjNum 35 snap 0 seq_num 35) dirty exists 2024-09-20T22:43:32.156 INFO:tasks.rados.rados.0.smithi084.stdout:213: finishing write tid 3 to smithi08477203-24 2024-09-20T22:43:32.156 INFO:tasks.rados.rados.0.smithi084.stdout:213: finishing write tid 4 to smithi08477203-24 2024-09-20T22:43:32.157 INFO:tasks.rados.rados.0.smithi084.stdout:213: finishing write tid 5 to smithi08477203-24 2024-09-20T22:43:32.157 INFO:tasks.rados.rados.0.smithi084.stdout:213: oid 24 updating version 0 to 36 2024-09-20T22:43:32.157 INFO:tasks.rados.rados.0.smithi084.stdout:213: oid 24 updating version 36 to 38 2024-09-20T22:43:32.157 INFO:tasks.rados.rados.0.smithi084.stdout:213: oid 24 version 38 is already newer than 37 2024-09-20T22:43:32.157 INFO:tasks.rados.rados.0.smithi084.stdout:213: oid 24 version 38 is already newer than 35 2024-09-20T22:43:32.157 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 24 v 38 (ObjNum 104 snap 21 seq_num 104) dirty exists 2024-09-20T22:43:32.157 INFO:tasks.rados.rados.0.smithi084.stdout:213: left oid 24 (ObjNum 104 snap 21 seq_num 104) 2024-09-20T22:43:32.157 INFO:tasks.rados.rados.0.smithi084.stdout:214: expect (ObjNum 87 snap 11 seq_num 87) 2024-09-20T22:43:32.186 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:43:32.204 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-20T22:43:32.206 INFO:teuthology.orchestra.run.smithi084.stderr:dumped all 2024-09-20T22:43:32.212 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:32.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:32.213 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:43:32.218 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:43:32.218 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:43:32.241 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:43:32.242 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:43:32.258 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:43:32.258 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:43:32.274 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-20T22:43:32.274 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T22:43:32.294 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:43:32.294 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:43:32.311 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:43:32.311 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:43:32.314 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:43:32.327 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:43:32.327 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:43:32.336 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:32.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:32.337 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:43:32.343 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:43:32.343 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:43:32.353 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:32 smithi084 ceph-mon[40024]: 1.11 scrub starts 2024-09-20T22:43:32.353 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:32 smithi084 ceph-mon[40024]: 1.11 scrub ok 2024-09-20T22:43:32.353 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:32 smithi084 ceph-mon[40024]: 1.10 scrub starts 2024-09-20T22:43:32.353 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:32 smithi084 ceph-mon[40024]: 1.10 scrub ok 2024-09-20T22:43:32.353 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:32 smithi084 ceph-mon[40024]: 3.15 deep-scrub starts 2024-09-20T22:43:32.353 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:32 smithi084 ceph-mon[40024]: 3.15 deep-scrub ok 2024-09-20T22:43:32.353 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:32 smithi084 ceph-mon[40024]: 4.7 scrub starts 2024-09-20T22:43:32.353 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:32 smithi084 ceph-mon[40024]: 4.7 scrub ok 2024-09-20T22:43:32.353 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:32 smithi084 ceph-mon[40024]: 3.9 scrub starts 2024-09-20T22:43:32.353 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:32 smithi084 ceph-mon[40024]: 3.9 scrub ok 2024-09-20T22:43:32.353 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:32 smithi084 ceph-mon[40024]: osdmap e133: 8 total, 8 up, 8 in 2024-09-20T22:43:32.353 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:32 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/2388377076' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:43:32.354 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:32 smithi084 ceph-mon[29774]: 1.11 scrub starts 2024-09-20T22:43:32.354 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:32 smithi084 ceph-mon[29774]: 1.11 scrub ok 2024-09-20T22:43:32.354 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:32 smithi084 ceph-mon[29774]: 1.10 scrub starts 2024-09-20T22:43:32.354 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:32 smithi084 ceph-mon[29774]: 1.10 scrub ok 2024-09-20T22:43:32.354 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:32 smithi084 ceph-mon[29774]: 3.15 deep-scrub starts 2024-09-20T22:43:32.354 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:32 smithi084 ceph-mon[29774]: 3.15 deep-scrub ok 2024-09-20T22:43:32.354 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:32 smithi084 ceph-mon[29774]: 4.7 scrub starts 2024-09-20T22:43:32.354 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:32 smithi084 ceph-mon[29774]: 4.7 scrub ok 2024-09-20T22:43:32.354 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:32 smithi084 ceph-mon[29774]: 3.9 scrub starts 2024-09-20T22:43:32.354 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:32 smithi084 ceph-mon[29774]: 3.9 scrub ok 2024-09-20T22:43:32.354 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:32 smithi084 ceph-mon[29774]: osdmap e133: 8 total, 8 up, 8 in 2024-09-20T22:43:32.355 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:32 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/2388377076' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:43:32.438 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:43:32.460 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:32.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:32.461 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:43:32.562 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:43:32.583 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:32.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:32.584 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:43:32.585 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:32 smithi154 ceph-mon[33264]: 1.11 scrub starts 2024-09-20T22:43:32.585 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:32 smithi154 ceph-mon[33264]: 1.11 scrub ok 2024-09-20T22:43:32.585 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:32 smithi154 ceph-mon[33264]: 1.10 scrub starts 2024-09-20T22:43:32.585 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:32 smithi154 ceph-mon[33264]: 1.10 scrub ok 2024-09-20T22:43:32.585 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:32 smithi154 ceph-mon[33264]: 3.15 deep-scrub starts 2024-09-20T22:43:32.585 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:32 smithi154 ceph-mon[33264]: 3.15 deep-scrub ok 2024-09-20T22:43:32.585 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:32 smithi154 ceph-mon[33264]: 4.7 scrub starts 2024-09-20T22:43:32.585 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:32 smithi154 ceph-mon[33264]: 4.7 scrub ok 2024-09-20T22:43:32.585 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:32 smithi154 ceph-mon[33264]: 3.9 scrub starts 2024-09-20T22:43:32.585 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:32 smithi154 ceph-mon[33264]: 3.9 scrub ok 2024-09-20T22:43:32.585 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:32 smithi154 ceph-mon[33264]: osdmap e133: 8 total, 8 up, 8 in 2024-09-20T22:43:32.585 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:32 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/2388377076' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:43:32.611 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-20T22:43:32.685 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:43:32.706 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:32.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:32.708 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:43:32.808 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:43:32.829 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:32.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:32.831 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:43:32.897 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:43:32.931 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:43:32.935 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd set noscrub 2024-09-20T22:43:32.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:32.954 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:32.954 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:43:33.055 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:43:33.077 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:33.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:33.078 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:43:33.122 INFO:tasks.rados.rados.0.smithi084.stdout:213: done (3 left) 2024-09-20T22:43:33.123 INFO:tasks.rados.rados.0.smithi084.stdout:214: done (2 left) 2024-09-20T22:43:33.123 INFO:tasks.rados.rados.0.smithi084.stdout:215: done (1 left) 2024-09-20T22:43:33.123 INFO:tasks.rados.rados.0.smithi084.stdout:216: done (0 left) 2024-09-20T22:43:33.123 INFO:tasks.rados.rados.0.smithi084.stdout:217: snap_remove snap 14 2024-09-20T22:43:33.178 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:43:33.200 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:33.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:33.201 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:43:33.302 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:43:33.323 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:33.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:33.324 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:43:33.381 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-20T22:43:33.381 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [], 2024-09-20T22:43:33.381 INFO:teuthology.orchestra.run.smithi154.stdout: "num_ops": 0 2024-09-20T22:43:33.381 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-20T22:43:33.424 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:43:33.446 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:33.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:33.447 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:43:33.547 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:43:33.571 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:33.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:33.572 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:43:33.593 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:33 smithi084 ceph-mon[40024]: pgmap v100: 168 pgs: 168 active+clean; 144 MiB data, 526 MiB used, 715 GiB / 715 GiB avail; 9.2 MiB/s rd, 5.3 MiB/s wr, 19 op/s 2024-09-20T22:43:33.593 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:33 smithi084 ceph-mon[40024]: 5.17 scrub starts 2024-09-20T22:43:33.593 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:33 smithi084 ceph-mon[40024]: 5.17 scrub ok 2024-09-20T22:43:33.593 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:33 smithi084 ceph-mon[40024]: from='client.14925 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T22:43:33.593 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:33 smithi084 ceph-mon[40024]: 3.4 scrub starts 2024-09-20T22:43:33.593 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:33 smithi084 ceph-mon[40024]: 3.4 scrub ok 2024-09-20T22:43:33.593 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:33 smithi084 ceph-mon[40024]: osdmap e134: 8 total, 8 up, 8 in 2024-09-20T22:43:33.593 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:33 smithi084 ceph-mon[29774]: pgmap v100: 168 pgs: 168 active+clean; 144 MiB data, 526 MiB used, 715 GiB / 715 GiB avail; 9.2 MiB/s rd, 5.3 MiB/s wr, 19 op/s 2024-09-20T22:43:33.594 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:33 smithi084 ceph-mon[29774]: 5.17 scrub starts 2024-09-20T22:43:33.594 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:33 smithi084 ceph-mon[29774]: 5.17 scrub ok 2024-09-20T22:43:33.594 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:33 smithi084 ceph-mon[29774]: from='client.14925 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T22:43:33.594 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:33 smithi084 ceph-mon[29774]: 3.4 scrub starts 2024-09-20T22:43:33.594 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:33 smithi084 ceph-mon[29774]: 3.4 scrub ok 2024-09-20T22:43:33.594 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:33 smithi084 ceph-mon[29774]: osdmap e134: 8 total, 8 up, 8 in 2024-09-20T22:43:33.623 INFO:teuthology.orchestra.run.smithi084.stdout:{"pg_ready":true,"pg_map":{"version":101,"stamp":"2024-09-20T22:43:32.125563+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":150875360,"num_objects":262,"num_object_clones":17,"num_object_copies":786,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":262,"num_whiteouts":4,"num_read":1321,"num_read_kb":305217,"num_write":891,"num_write_kb":157487,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":730425,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3662,"ondisk_log_size":3662,"up":504,"acting":504,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":504,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":538728,"kb_used_data":291896,"kb_used_omap":12,"kb_used_meta":245171,"kb_avail":749422488,"statfs":{"total":767960285184,"available":767408627712,"internally_reserved":0,"allocated":298901504,"data_stored":309480875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":251055664},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":22742512,"num_objects":12,"num_object_clones":11,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":2,"num_read":46,"num_read_kb":38011,"num_write":34,"num_write_kb":21926,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.012231"},"pg_stats":[{"pgid":"3.1f","version":"86'2","reported_seq":150,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.115525+0000","last_change":"2024-09-20T22:42:39.664558+0000","last_active":"2024-09-20T22:43:28.115525+0000","last_peered":"2024-09-20T22:43:28.115525+0000","last_clean":"2024-09-20T22:43:28.115525+0000","last_became_active":"2024-09-20T22:41:38.159085+0000","last_became_peered":"2024-09-20T22:41:38.159085+0000","last_unstale":"2024-09-20T22:43:28.115525+0000","last_undegraded":"2024-09-20T22:43:28.115525+0000","last_fullsized":"2024-09-20T22:43:28.115525+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'2","last_scrub_stamp":"2024-09-20T22:42:39.664421+0000","last_deep_scrub":"86'2","last_deep_scrub_stamp":"2024-09-20T22:42:39.664421+0000","last_clean_scrub_stamp":"2024-09-20T22:42:39.664421+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:40.189792+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"74'9","reported_seq":147,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.115285+0000","last_change":"2024-09-20T22:41:40.169673+0000","last_active":"2024-09-20T22:43:28.115285+0000","last_peered":"2024-09-20T22:43:28.115285+0000","last_clean":"2024-09-20T22:43:28.115285+0000","last_became_active":"2024-09-20T22:41:40.168934+0000","last_became_peered":"2024-09-20T22:41:40.168934+0000","last_unstale":"2024-09-20T22:43:28.115285+0000","last_undegraded":"2024-09-20T22:43:28.115285+0000","last_fullsized":"2024-09-20T22:43:28.115285+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":127,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.138003+0000","last_change":"2024-09-20T22:41:42.182304+0000","last_active":"2024-09-20T22:43:28.138003+0000","last_peered":"2024-09-20T22:43:28.138003+0000","last_clean":"2024-09-20T22:43:28.138003+0000","last_became_active":"2024-09-20T22:41:42.182024+0000","last_became_peered":"2024-09-20T22:41:42.182024+0000","last_unstale":"2024-09-20T22:43:28.138003+0000","last_undegraded":"2024-09-20T22:43:28.138003+0000","last_fullsized":"2024-09-20T22:43:28.138003+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":123,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.115304+0000","last_change":"2024-09-20T22:41:44.198207+0000","last_active":"2024-09-20T22:43:28.115304+0000","last_peered":"2024-09-20T22:43:28.115304+0000","last_clean":"2024-09-20T22:43:28.115304+0000","last_became_active":"2024-09-20T22:41:44.197897+0000","last_became_peered":"2024-09-20T22:41:44.197897+0000","last_unstale":"2024-09-20T22:43:28.115304+0000","last_undegraded":"2024-09-20T22:43:28.115304+0000","last_fullsized":"2024-09-20T22:43:28.115304+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":131,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.115738+0000","last_change":"2024-09-20T22:43:07.476040+0000","last_active":"2024-09-20T22:43:28.115738+0000","last_peered":"2024-09-20T22:43:28.115738+0000","last_clean":"2024-09-20T22:43:28.115738+0000","last_became_active":"2024-09-20T22:41:44.832099+0000","last_became_peered":"2024-09-20T22:41:44.832099+0000","last_unstale":"2024-09-20T22:43:28.115738+0000","last_undegraded":"2024-09-20T22:43:28.115738+0000","last_fullsized":"2024-09-20T22:43:28.115738+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:07.475964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:43:07.475964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:15.747336+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":135,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.115621+0000","last_change":"2024-09-20T22:41:38.157685+0000","last_active":"2024-09-20T22:43:28.115621+0000","last_peered":"2024-09-20T22:43:28.115621+0000","last_clean":"2024-09-20T22:43:28.115621+0000","last_became_active":"2024-09-20T22:41:38.157298+0000","last_became_peered":"2024-09-20T22:41:38.157298+0000","last_unstale":"2024-09-20T22:43:28.115621+0000","last_undegraded":"2024-09-20T22:43:28.115621+0000","last_fullsized":"2024-09-20T22:43:28.115621+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:28.191437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"74'15","reported_seq":156,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.115494+0000","last_change":"2024-09-20T22:41:40.163585+0000","last_active":"2024-09-20T22:43:28.115494+0000","last_peered":"2024-09-20T22:43:28.115494+0000","last_clean":"2024-09-20T22:43:28.115494+0000","last_became_active":"2024-09-20T22:41:40.163242+0000","last_became_peered":"2024-09-20T22:41:40.163242+0000","last_unstale":"2024-09-20T22:43:28.115494+0000","last_undegraded":"2024-09-20T22:43:28.115494+0000","last_fullsized":"2024-09-20T22:43:28.115494+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:28.191424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":135,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.115175+0000","last_change":"2024-09-20T22:43:02.971265+0000","last_active":"2024-09-20T22:43:28.115175+0000","last_peered":"2024-09-20T22:43:28.115175+0000","last_clean":"2024-09-20T22:43:28.115175+0000","last_became_active":"2024-09-20T22:41:42.187076+0000","last_became_peered":"2024-09-20T22:41:42.187076+0000","last_unstale":"2024-09-20T22:43:28.115175+0000","last_undegraded":"2024-09-20T22:43:28.115175+0000","last_fullsized":"2024-09-20T22:43:28.115175+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:02.971194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:43:02.971194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:08.332638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":135,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.127987+0000","last_change":"2024-09-20T22:41:38.156694+0000","last_active":"2024-09-20T22:43:28.127987+0000","last_peered":"2024-09-20T22:43:28.127987+0000","last_clean":"2024-09-20T22:43:28.127987+0000","last_became_active":"2024-09-20T22:41:38.154827+0000","last_became_peered":"2024-09-20T22:41:38.154827+0000","last_unstale":"2024-09-20T22:43:28.127987+0000","last_undegraded":"2024-09-20T22:43:28.127987+0000","last_fullsized":"2024-09-20T22:43:28.127987+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"74'9","reported_seq":147,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.115471+0000","last_change":"2024-09-20T22:41:40.163006+0000","last_active":"2024-09-20T22:43:28.115471+0000","last_peered":"2024-09-20T22:43:28.115471+0000","last_clean":"2024-09-20T22:43:28.115471+0000","last_became_active":"2024-09-20T22:41:40.162546+0000","last_became_peered":"2024-09-20T22:41:40.162546+0000","last_unstale":"2024-09-20T22:43:28.115471+0000","last_undegraded":"2024-09-20T22:43:28.115471+0000","last_fullsized":"2024-09-20T22:43:28.115471+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":127,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.127780+0000","last_change":"2024-09-20T22:41:42.185835+0000","last_active":"2024-09-20T22:43:28.127780+0000","last_peered":"2024-09-20T22:43:28.127780+0000","last_clean":"2024-09-20T22:43:28.127780+0000","last_became_active":"2024-09-20T22:41:42.185538+0000","last_became_peered":"2024-09-20T22:41:42.185538+0000","last_unstale":"2024-09-20T22:43:28.127780+0000","last_undegraded":"2024-09-20T22:43:28.127780+0000","last_fullsized":"2024-09-20T22:43:28.127780+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":123,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.116957+0000","last_change":"2024-09-20T22:41:44.198169+0000","last_active":"2024-09-20T22:43:28.116957+0000","last_peered":"2024-09-20T22:43:28.116957+0000","last_clean":"2024-09-20T22:43:28.116957+0000","last_became_active":"2024-09-20T22:41:44.197113+0000","last_became_peered":"2024-09-20T22:41:44.197113+0000","last_unstale":"2024-09-20T22:43:28.116957+0000","last_undegraded":"2024-09-20T22:43:28.116957+0000","last_fullsized":"2024-09-20T22:43:28.116957+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":143,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.127781+0000","last_change":"2024-09-20T22:43:00.713953+0000","last_active":"2024-09-20T22:43:28.127781+0000","last_peered":"2024-09-20T22:43:28.127781+0000","last_clean":"2024-09-20T22:43:28.127781+0000","last_became_active":"2024-09-20T22:41:38.154650+0000","last_became_peered":"2024-09-20T22:41:38.154650+0000","last_unstale":"2024-09-20T22:43:28.127781+0000","last_undegraded":"2024-09-20T22:43:28.127781+0000","last_fullsized":"2024-09-20T22:43:28.127781+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:00.713878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:43:00.713878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:21.268170+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"74'5","reported_seq":141,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.115577+0000","last_change":"2024-09-20T22:41:40.171883+0000","last_active":"2024-09-20T22:43:28.115577+0000","last_peered":"2024-09-20T22:43:28.115577+0000","last_clean":"2024-09-20T22:43:28.115577+0000","last_became_active":"2024-09-20T22:41:40.171081+0000","last_became_peered":"2024-09-20T22:41:40.171081+0000","last_unstale":"2024-09-20T22:43:28.115577+0000","last_undegraded":"2024-09-20T22:43:28.115577+0000","last_fullsized":"2024-09-20T22:43:28.115577+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":138,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.122006+0000","last_change":"2024-09-20T22:43:01.248846+0000","last_active":"2024-09-20T22:43:28.122006+0000","last_peered":"2024-09-20T22:43:28.122006+0000","last_clean":"2024-09-20T22:43:28.122006+0000","last_became_active":"2024-09-20T22:41:42.182916+0000","last_became_peered":"2024-09-20T22:41:42.182916+0000","last_unstale":"2024-09-20T22:43:28.122006+0000","last_undegraded":"2024-09-20T22:43:28.122006+0000","last_fullsized":"2024-09-20T22:43:28.122006+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:01.248740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:43:01.248740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:09.032402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":123,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.127343+0000","last_change":"2024-09-20T22:41:44.198221+0000","last_active":"2024-09-20T22:43:28.127343+0000","last_peered":"2024-09-20T22:43:28.127343+0000","last_clean":"2024-09-20T22:43:28.127343+0000","last_became_active":"2024-09-20T22:41:44.197927+0000","last_became_peered":"2024-09-20T22:41:44.197927+0000","last_unstale":"2024-09-20T22:43:28.127343+0000","last_undegraded":"2024-09-20T22:43:28.127343+0000","last_fullsized":"2024-09-20T22:43:28.127343+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":131,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.116117+0000","last_change":"2024-09-20T22:43:10.000692+0000","last_active":"2024-09-20T22:43:28.116117+0000","last_peered":"2024-09-20T22:43:28.116117+0000","last_clean":"2024-09-20T22:43:28.116117+0000","last_became_active":"2024-09-20T22:41:44.832360+0000","last_became_peered":"2024-09-20T22:41:44.832360+0000","last_unstale":"2024-09-20T22:43:28.116117+0000","last_undegraded":"2024-09-20T22:43:28.116117+0000","last_fullsized":"2024-09-20T22:43:28.116117+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:10.000610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:43:10.000610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:33.885490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":135,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.116351+0000","last_change":"2024-09-20T22:41:38.152726+0000","last_active":"2024-09-20T22:43:28.116351+0000","last_peered":"2024-09-20T22:43:28.116351+0000","last_clean":"2024-09-20T22:43:28.116351+0000","last_became_active":"2024-09-20T22:41:38.152443+0000","last_became_peered":"2024-09-20T22:41:38.152443+0000","last_unstale":"2024-09-20T22:43:28.116351+0000","last_undegraded":"2024-09-20T22:43:28.116351+0000","last_fullsized":"2024-09-20T22:43:28.116351+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"113'22","reported_seq":171,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.130858+0000","last_change":"2024-09-20T22:43:12.191980+0000","last_active":"2024-09-20T22:43:28.130858+0000","last_peered":"2024-09-20T22:43:28.130858+0000","last_clean":"2024-09-20T22:43:28.130858+0000","last_became_active":"2024-09-20T22:41:40.170131+0000","last_became_peered":"2024-09-20T22:41:40.170131+0000","last_unstale":"2024-09-20T22:43:28.130858+0000","last_undegraded":"2024-09-20T22:43:28.130858+0000","last_fullsized":"2024-09-20T22:43:28.130858+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'22","last_scrub_stamp":"2024-09-20T22:43:12.191878+0000","last_deep_scrub":"113'22","last_deep_scrub_stamp":"2024-09-20T22:43:12.191878+0000","last_clean_scrub_stamp":"2024-09-20T22:43:12.191878+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:28.270781+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":127,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.136742+0000","last_change":"2024-09-20T22:41:42.185171+0000","last_active":"2024-09-20T22:43:28.136742+0000","last_peered":"2024-09-20T22:43:28.136742+0000","last_clean":"2024-09-20T22:43:28.136742+0000","last_became_active":"2024-09-20T22:41:42.184759+0000","last_became_peered":"2024-09-20T22:41:42.184759+0000","last_unstale":"2024-09-20T22:43:28.136742+0000","last_undegraded":"2024-09-20T22:43:28.136742+0000","last_fullsized":"2024-09-20T22:43:28.136742+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":123,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.116505+0000","last_change":"2024-09-20T22:41:44.832468+0000","last_active":"2024-09-20T22:43:28.116505+0000","last_peered":"2024-09-20T22:43:28.116505+0000","last_clean":"2024-09-20T22:43:28.116505+0000","last_became_active":"2024-09-20T22:41:44.832131+0000","last_became_peered":"2024-09-20T22:41:44.832131+0000","last_unstale":"2024-09-20T22:43:28.116505+0000","last_undegraded":"2024-09-20T22:43:28.116505+0000","last_fullsized":"2024-09-20T22:43:28.116505+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:28.191525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":135,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.115627+0000","last_change":"2024-09-20T22:41:38.150584+0000","last_active":"2024-09-20T22:43:28.115627+0000","last_peered":"2024-09-20T22:43:28.115627+0000","last_clean":"2024-09-20T22:43:28.115627+0000","last_became_active":"2024-09-20T22:41:38.150232+0000","last_became_peered":"2024-09-20T22:41:38.150232+0000","last_unstale":"2024-09-20T22:43:28.115627+0000","last_undegraded":"2024-09-20T22:43:28.115627+0000","last_fullsized":"2024-09-20T22:43:28.115627+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"74'12","reported_seq":172,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.114971+0000","last_change":"2024-09-20T22:43:09.123893+0000","last_active":"2024-09-20T22:43:28.114971+0000","last_peered":"2024-09-20T22:43:28.114971+0000","last_clean":"2024-09-20T22:43:28.114971+0000","last_became_active":"2024-09-20T22:43:09.116985+0000","last_became_peered":"2024-09-20T22:43:09.116985+0000","last_unstale":"2024-09-20T22:43:28.114971+0000","last_undegraded":"2024-09-20T22:43:28.114971+0000","last_fullsized":"2024-09-20T22:43:28.114971+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:28.191511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":127,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.115834+0000","last_change":"2024-09-20T22:41:42.183266+0000","last_active":"2024-09-20T22:43:28.115834+0000","last_peered":"2024-09-20T22:43:28.115834+0000","last_clean":"2024-09-20T22:43:28.115834+0000","last_became_active":"2024-09-20T22:41:42.183005+0000","last_became_peered":"2024-09-20T22:41:42.183005+0000","last_unstale":"2024-09-20T22:43:28.115834+0000","last_undegraded":"2024-09-20T22:43:28.115834+0000","last_fullsized":"2024-09-20T22:43:28.115834+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":124,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.122758+0000","last_change":"2024-09-20T22:41:44.200115+0000","last_active":"2024-09-20T22:43:28.122758+0000","last_peered":"2024-09-20T22:43:28.122758+0000","last_clean":"2024-09-20T22:43:28.122758+0000","last_became_active":"2024-09-20T22:41:44.199822+0000","last_became_peered":"2024-09-20T22:41:44.199822+0000","last_unstale":"2024-09-20T22:43:28.122758+0000","last_undegraded":"2024-09-20T22:43:28.122758+0000","last_fullsized":"2024-09-20T22:43:28.122758+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:28.181398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"67'1","reported_seq":136,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.138146+0000","last_change":"2024-09-20T22:41:38.155735+0000","last_active":"2024-09-20T22:43:28.138146+0000","last_peered":"2024-09-20T22:43:28.138146+0000","last_clean":"2024-09-20T22:43:28.138146+0000","last_became_active":"2024-09-20T22:41:38.155424+0000","last_became_peered":"2024-09-20T22:41:38.155424+0000","last_unstale":"2024-09-20T22:43:28.138146+0000","last_undegraded":"2024-09-20T22:43:28.138146+0000","last_fullsized":"2024-09-20T22:43:28.138146+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"74'10","reported_seq":146,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.119124+0000","last_change":"2024-09-20T22:41:40.252498+0000","last_active":"2024-09-20T22:43:28.119124+0000","last_peered":"2024-09-20T22:43:28.119124+0000","last_clean":"2024-09-20T22:43:28.119124+0000","last_became_active":"2024-09-20T22:41:40.252001+0000","last_became_peered":"2024-09-20T22:41:40.252001+0000","last_unstale":"2024-09-20T22:43:28.119124+0000","last_undegraded":"2024-09-20T22:43:28.119124+0000","last_fullsized":"2024-09-20T22:43:28.119124+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"74'8","reported_seq":164,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.139390+0000","last_change":"2024-09-20T22:41:42.186385+0000","last_active":"2024-09-20T22:43:28.139390+0000","last_peered":"2024-09-20T22:43:28.139390+0000","last_clean":"2024-09-20T22:43:28.139390+0000","last_became_active":"2024-09-20T22:41:42.186071+0000","last_became_peered":"2024-09-20T22:41:42.186071+0000","last_unstale":"2024-09-20T22:43:28.139390+0000","last_undegraded":"2024-09-20T22:43:28.139390+0000","last_fullsized":"2024-09-20T22:43:28.139390+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.c","version":"129'60","reported_seq":249,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.164560+0000","last_change":"2024-09-20T22:43:23.121754+0000","last_active":"2024-09-20T22:43:28.164560+0000","last_peered":"2024-09-20T22:43:28.164560+0000","last_clean":"2024-09-20T22:43:28.164560+0000","last_became_active":"2024-09-20T22:42:52.404027+0000","last_became_peered":"2024-09-20T22:42:52.404027+0000","last_unstale":"2024-09-20T22:43:28.164560+0000","last_undegraded":"2024-09-20T22:43:28.164560+0000","last_fullsized":"2024-09-20T22:43:28.164560+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:42:51.354519+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:08.958164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003035888,"stat_sum":{"num_bytes":22133913,"num_objects":8,"num_object_clones":1,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":75,"num_read_kb":64198,"num_write":58,"num_write_kb":19351,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.a","version":"77'128","reported_seq":379,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.127343+0000","last_change":"2024-09-20T22:43:05.763879+0000","last_active":"2024-09-20T22:43:28.127343+0000","last_peered":"2024-09-20T22:43:28.127343+0000","last_clean":"2024-09-20T22:43:28.127343+0000","last_became_active":"2024-09-20T22:42:52.407791+0000","last_became_peered":"2024-09-20T22:42:52.407791+0000","last_unstale":"2024-09-20T22:43:28.127343+0000","last_undegraded":"2024-09-20T22:43:28.127343+0000","last_fullsized":"2024-09-20T22:43:28.127343+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:43:05.763796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:43:05.763796+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:11.866914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"74'15","reported_seq":164,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.136068+0000","last_change":"2024-09-20T22:43:10.076804+0000","last_active":"2024-09-20T22:43:28.136068+0000","last_peered":"2024-09-20T22:43:28.136068+0000","last_clean":"2024-09-20T22:43:28.136068+0000","last_became_active":"2024-09-20T22:41:40.165963+0000","last_became_peered":"2024-09-20T22:41:40.165963+0000","last_unstale":"2024-09-20T22:43:28.136068+0000","last_undegraded":"2024-09-20T22:43:28.136068+0000","last_fullsized":"2024-09-20T22:43:28.136068+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-09-20T22:43:10.076699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:43:10.076699+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:37.109196+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":135,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.114749+0000","last_change":"2024-09-20T22:41:38.157043+0000","last_active":"2024-09-20T22:43:28.114749+0000","last_peered":"2024-09-20T22:43:28.114749+0000","last_clean":"2024-09-20T22:43:28.114749+0000","last_became_active":"2024-09-20T22:41:38.156749+0000","last_became_peered":"2024-09-20T22:41:38.156749+0000","last_unstale":"2024-09-20T22:43:28.114749+0000","last_undegraded":"2024-09-20T22:43:28.114749+0000","last_fullsized":"2024-09-20T22:43:28.114749+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:28.191465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"74'8","reported_seq":158,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.116215+0000","last_change":"2024-09-20T22:41:42.182555+0000","last_active":"2024-09-20T22:43:28.116215+0000","last_peered":"2024-09-20T22:43:28.116215+0000","last_clean":"2024-09-20T22:43:28.116215+0000","last_became_active":"2024-09-20T22:41:42.182224+0000","last_became_peered":"2024-09-20T22:41:42.182224+0000","last_unstale":"2024-09-20T22:43:28.116215+0000","last_undegraded":"2024-09-20T22:43:28.116215+0000","last_fullsized":"2024-09-20T22:43:28.116215+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":123,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.127741+0000","last_change":"2024-09-20T22:41:44.198655+0000","last_active":"2024-09-20T22:43:28.127741+0000","last_peered":"2024-09-20T22:43:28.127741+0000","last_clean":"2024-09-20T22:43:28.127741+0000","last_became_active":"2024-09-20T22:41:44.198347+0000","last_became_peered":"2024-09-20T22:41:44.198347+0000","last_unstale":"2024-09-20T22:43:28.127741+0000","last_undegraded":"2024-09-20T22:43:28.127741+0000","last_fullsized":"2024-09-20T22:43:28.127741+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3","version":"98'3","reported_seq":167,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.115786+0000","last_change":"2024-09-20T22:43:23.131977+0000","last_active":"2024-09-20T22:43:28.115786+0000","last_peered":"2024-09-20T22:43:28.115786+0000","last_clean":"2024-09-20T22:43:28.115786+0000","last_became_active":"2024-09-20T22:42:52.406126+0000","last_became_peered":"2024-09-20T22:42:52.406126+0000","last_unstale":"2024-09-20T22:43:28.115786+0000","last_undegraded":"2024-09-20T22:43:28.115786+0000","last_fullsized":"2024-09-20T22:43:28.115786+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:42:51.354519+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:57.108447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0027806689999999999,"stat_sum":{"num_bytes":2521017,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":11,"num_read_kb":11104,"num_write":4,"num_write_kb":1252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"77'128","reported_seq":392,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.116245+0000","last_change":"2024-09-20T22:43:25.851297+0000","last_active":"2024-09-20T22:43:28.116245+0000","last_peered":"2024-09-20T22:43:28.116245+0000","last_clean":"2024-09-20T22:43:28.116245+0000","last_became_active":"2024-09-20T22:43:19.128074+0000","last_became_peered":"2024-09-20T22:43:19.128074+0000","last_unstale":"2024-09-20T22:43:28.116245+0000","last_undegraded":"2024-09-20T22:43:28.116245+0000","last_fullsized":"2024-09-20T22:43:28.116245+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:43:25.851220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:43:25.851220+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:36.958582+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"74'18","reported_seq":163,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.114641+0000","last_change":"2024-09-20T22:41:40.252203+0000","last_active":"2024-09-20T22:43:28.114641+0000","last_peered":"2024-09-20T22:43:28.114641+0000","last_clean":"2024-09-20T22:43:28.114641+0000","last_became_active":"2024-09-20T22:41:40.251868+0000","last_became_peered":"2024-09-20T22:41:40.251868+0000","last_unstale":"2024-09-20T22:43:28.114641+0000","last_undegraded":"2024-09-20T22:43:28.114641+0000","last_fullsized":"2024-09-20T22:43:28.114641+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:28.191408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":135,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.116942+0000","last_change":"2024-09-20T22:41:38.161991+0000","last_active":"2024-09-20T22:43:28.116942+0000","last_peered":"2024-09-20T22:43:28.116942+0000","last_clean":"2024-09-20T22:43:28.116942+0000","last_became_active":"2024-09-20T22:41:38.161335+0000","last_became_peered":"2024-09-20T22:41:38.161335+0000","last_unstale":"2024-09-20T22:43:28.116942+0000","last_undegraded":"2024-09-20T22:43:28.116942+0000","last_fullsized":"2024-09-20T22:43:28.116942+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:28.191287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":135,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.115338+0000","last_change":"2024-09-20T22:43:07.948019+0000","last_active":"2024-09-20T22:43:28.115338+0000","last_peered":"2024-09-20T22:43:28.115338+0000","last_clean":"2024-09-20T22:43:28.115338+0000","last_became_active":"2024-09-20T22:41:42.189743+0000","last_became_peered":"2024-09-20T22:41:42.189743+0000","last_unstale":"2024-09-20T22:43:28.115338+0000","last_undegraded":"2024-09-20T22:43:28.115338+0000","last_fullsized":"2024-09-20T22:43:28.115338+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:07.947945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:07.947945+0000","last_clean_scrub_stamp":"2024-09-20T22:43:07.947945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:23.340969+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":123,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.115463+0000","last_change":"2024-09-20T22:41:44.193944+0000","last_active":"2024-09-20T22:43:28.115463+0000","last_peered":"2024-09-20T22:43:28.115463+0000","last_clean":"2024-09-20T22:43:28.115463+0000","last_became_active":"2024-09-20T22:41:44.193646+0000","last_became_peered":"2024-09-20T22:41:44.193646+0000","last_unstale":"2024-09-20T22:43:28.115463+0000","last_undegraded":"2024-09-20T22:43:28.115463+0000","last_fullsized":"2024-09-20T22:43:28.115463+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2","version":"128'61","reported_seq":237,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.117626+0000","last_change":"2024-09-20T22:43:23.132196+0000","last_active":"2024-09-20T22:43:28.117626+0000","last_peered":"2024-09-20T22:43:28.117626+0000","last_clean":"2024-09-20T22:43:28.117626+0000","last_became_active":"2024-09-20T22:42:52.402252+0000","last_became_peered":"2024-09-20T22:42:52.402252+0000","last_unstale":"2024-09-20T22:43:28.117626+0000","last_undegraded":"2024-09-20T22:43:28.117626+0000","last_fullsized":"2024-09-20T22:43:28.117626+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:42:51.354519+0000","objects_scrubbed":0,"log_size":61,"log_dups_size":0,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:52.720470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024147159999999999,"stat_sum":{"num_bytes":11405317,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":47,"num_read_kb":29869,"num_write":53,"num_write_kb":18619,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"77'128","reported_seq":392,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.117076+0000","last_change":"2024-09-20T22:43:26.151507+0000","last_active":"2024-09-20T22:43:28.117076+0000","last_peered":"2024-09-20T22:43:28.117076+0000","last_clean":"2024-09-20T22:43:28.117076+0000","last_became_active":"2024-09-20T22:43:19.125641+0000","last_became_peered":"2024-09-20T22:43:19.125641+0000","last_unstale":"2024-09-20T22:43:28.117076+0000","last_undegraded":"2024-09-20T22:43:28.117076+0000","last_fullsized":"2024-09-20T22:43:28.117076+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:43:26.151406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:43:26.151406+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:28.804449+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"74'14","reported_seq":152,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.115426+0000","last_change":"2024-09-20T22:41:40.164011+0000","last_active":"2024-09-20T22:43:28.115426+0000","last_peered":"2024-09-20T22:43:28.115426+0000","last_clean":"2024-09-20T22:43:28.115426+0000","last_became_active":"2024-09-20T22:41:40.163695+0000","last_became_peered":"2024-09-20T22:41:40.163695+0000","last_unstale":"2024-09-20T22:43:28.115426+0000","last_undegraded":"2024-09-20T22:43:28.115426+0000","last_fullsized":"2024-09-20T22:43:28.115426+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"67'1","reported_seq":136,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.119406+0000","last_change":"2024-09-20T22:41:38.150707+0000","last_active":"2024-09-20T22:43:28.119406+0000","last_peered":"2024-09-20T22:43:28.119406+0000","last_clean":"2024-09-20T22:43:28.119406+0000","last_became_active":"2024-09-20T22:41:38.150331+0000","last_became_peered":"2024-09-20T22:41:38.150331+0000","last_unstale":"2024-09-20T22:43:28.119406+0000","last_undegraded":"2024-09-20T22:43:28.119406+0000","last_fullsized":"2024-09-20T22:43:28.119406+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"74'8","reported_seq":158,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.114509+0000","last_change":"2024-09-20T22:41:42.179893+0000","last_active":"2024-09-20T22:43:28.114509+0000","last_peered":"2024-09-20T22:43:28.114509+0000","last_clean":"2024-09-20T22:43:28.114509+0000","last_became_active":"2024-09-20T22:41:42.179613+0000","last_became_peered":"2024-09-20T22:41:42.179613+0000","last_unstale":"2024-09-20T22:43:28.114509+0000","last_undegraded":"2024-09-20T22:43:28.114509+0000","last_fullsized":"2024-09-20T22:43:28.114509+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":123,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.122068+0000","last_change":"2024-09-20T22:41:44.832161+0000","last_active":"2024-09-20T22:43:28.122068+0000","last_peered":"2024-09-20T22:43:28.122068+0000","last_clean":"2024-09-20T22:43:28.122068+0000","last_became_active":"2024-09-20T22:41:44.831901+0000","last_became_peered":"2024-09-20T22:41:44.831901+0000","last_unstale":"2024-09-20T22:43:28.122068+0000","last_undegraded":"2024-09-20T22:43:28.122068+0000","last_fullsized":"2024-09-20T22:43:28.122068+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:28.181567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1","version":"122'28","reported_seq":196,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.115319+0000","last_change":"2024-09-20T22:43:23.120117+0000","last_active":"2024-09-20T22:43:28.115319+0000","last_peered":"2024-09-20T22:43:28.115319+0000","last_clean":"2024-09-20T22:43:28.115319+0000","last_became_active":"2024-09-20T22:42:52.405353+0000","last_became_peered":"2024-09-20T22:42:52.405353+0000","last_unstale":"2024-09-20T22:43:28.115319+0000","last_undegraded":"2024-09-20T22:43:28.115319+0000","last_fullsized":"2024-09-20T22:43:28.115319+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:42:51.354519+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:52.949598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0058022220000000001,"stat_sum":{"num_bytes":5781542,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":18,"num_read_kb":13072,"num_write":20,"num_write_kb":9845,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.7","version":"77'128","reported_seq":399,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.138969+0000","last_change":"2024-09-20T22:43:26.566566+0000","last_active":"2024-09-20T22:43:28.138969+0000","last_peered":"2024-09-20T22:43:28.138969+0000","last_clean":"2024-09-20T22:43:28.138969+0000","last_became_active":"2024-09-20T22:43:19.129015+0000","last_became_peered":"2024-09-20T22:43:19.129015+0000","last_unstale":"2024-09-20T22:43:28.138969+0000","last_undegraded":"2024-09-20T22:43:28.138969+0000","last_fullsized":"2024-09-20T22:43:28.138969+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:43:26.566436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:43:26.566436+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:54.250940+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"74'10","reported_seq":154,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.137558+0000","last_change":"2024-09-20T22:43:13.094497+0000","last_active":"2024-09-20T22:43:28.137558+0000","last_peered":"2024-09-20T22:43:28.137558+0000","last_clean":"2024-09-20T22:43:28.137558+0000","last_became_active":"2024-09-20T22:41:40.165653+0000","last_became_peered":"2024-09-20T22:41:40.165653+0000","last_unstale":"2024-09-20T22:43:28.137558+0000","last_undegraded":"2024-09-20T22:43:28.137558+0000","last_fullsized":"2024-09-20T22:43:28.137558+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-09-20T22:43:13.094419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:43:13.094419+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:25.645533+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":143,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.128067+0000","last_change":"2024-09-20T22:42:39.595884+0000","last_active":"2024-09-20T22:43:28.128067+0000","last_peered":"2024-09-20T22:43:28.128067+0000","last_clean":"2024-09-20T22:43:28.128067+0000","last_became_active":"2024-09-20T22:41:38.155629+0000","last_became_peered":"2024-09-20T22:41:38.155629+0000","last_unstale":"2024-09-20T22:43:28.128067+0000","last_undegraded":"2024-09-20T22:43:28.128067+0000","last_fullsized":"2024-09-20T22:43:28.128067+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:42:39.595799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:42:39.595799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:39.804106+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"74'8","reported_seq":166,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.115962+0000","last_change":"2024-09-20T22:43:08.643664+0000","last_active":"2024-09-20T22:43:28.115962+0000","last_peered":"2024-09-20T22:43:28.115962+0000","last_clean":"2024-09-20T22:43:28.115962+0000","last_became_active":"2024-09-20T22:41:42.185213+0000","last_became_peered":"2024-09-20T22:41:42.185213+0000","last_unstale":"2024-09-20T22:43:28.115962+0000","last_undegraded":"2024-09-20T22:43:28.115962+0000","last_fullsized":"2024-09-20T22:43:28.115962+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-09-20T22:43:08.643587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:43:08.643587+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:29.139515+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":123,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.116304+0000","last_change":"2024-09-20T22:41:44.197894+0000","last_active":"2024-09-20T22:43:28.116304+0000","last_peered":"2024-09-20T22:43:28.116304+0000","last_clean":"2024-09-20T22:43:28.116304+0000","last_became_active":"2024-09-20T22:41:44.197619+0000","last_became_peered":"2024-09-20T22:41:44.197619+0000","last_unstale":"2024-09-20T22:43:28.116304+0000","last_undegraded":"2024-09-20T22:43:28.116304+0000","last_fullsized":"2024-09-20T22:43:28.116304+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.0","version":"129'29","reported_seq":204,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.153720+0000","last_change":"2024-09-20T22:43:23.117845+0000","last_active":"2024-09-20T22:43:28.153720+0000","last_peered":"2024-09-20T22:43:28.153720+0000","last_clean":"2024-09-20T22:43:28.153720+0000","last_became_active":"2024-09-20T22:42:52.406385+0000","last_became_peered":"2024-09-20T22:42:52.406385+0000","last_unstale":"2024-09-20T22:43:28.153720+0000","last_undegraded":"2024-09-20T22:43:28.153720+0000","last_fullsized":"2024-09-20T22:43:28.153720+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:42:51.354519+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:52.897077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00069119400000000001,"stat_sum":{"num_bytes":8004837,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":48,"num_read_kb":45010,"num_write":28,"num_write_kb":10518,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"77'128","reported_seq":399,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.138923+0000","last_change":"2024-09-20T22:43:27.532172+0000","last_active":"2024-09-20T22:43:28.138923+0000","last_peered":"2024-09-20T22:43:28.138923+0000","last_clean":"2024-09-20T22:43:28.138923+0000","last_became_active":"2024-09-20T22:43:19.130331+0000","last_became_peered":"2024-09-20T22:43:19.130331+0000","last_unstale":"2024-09-20T22:43:28.138923+0000","last_undegraded":"2024-09-20T22:43:28.138923+0000","last_fullsized":"2024-09-20T22:43:28.138923+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:43:27.532085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:43:27.532085+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:28.211854+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"74'19","reported_seq":175,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.116112+0000","last_change":"2024-09-20T22:43:01.750427+0000","last_active":"2024-09-20T22:43:28.116112+0000","last_peered":"2024-09-20T22:43:28.116112+0000","last_clean":"2024-09-20T22:43:28.116112+0000","last_became_active":"2024-09-20T22:41:40.251866+0000","last_became_peered":"2024-09-20T22:41:40.251866+0000","last_unstale":"2024-09-20T22:43:28.116112+0000","last_undegraded":"2024-09-20T22:43:28.116112+0000","last_fullsized":"2024-09-20T22:43:28.116112+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'19","last_scrub_stamp":"2024-09-20T22:43:01.750344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:43:01.750344+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:06.070428+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":135,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.137128+0000","last_change":"2024-09-20T22:41:38.161236+0000","last_active":"2024-09-20T22:43:28.137128+0000","last_peered":"2024-09-20T22:43:28.137128+0000","last_clean":"2024-09-20T22:43:28.137128+0000","last_became_active":"2024-09-20T22:41:38.160832+0000","last_became_peered":"2024-09-20T22:41:38.160832+0000","last_unstale":"2024-09-20T22:43:28.137128+0000","last_undegraded":"2024-09-20T22:43:28.137128+0000","last_fullsized":"2024-09-20T22:43:28.137128+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":127,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.138581+0000","last_change":"2024-09-20T22:41:42.185638+0000","last_active":"2024-09-20T22:43:28.138581+0000","last_peered":"2024-09-20T22:43:28.138581+0000","last_clean":"2024-09-20T22:43:28.138581+0000","last_became_active":"2024-09-20T22:41:42.184412+0000","last_became_peered":"2024-09-20T22:41:42.184412+0000","last_unstale":"2024-09-20T22:43:28.138581+0000","last_undegraded":"2024-09-20T22:43:28.138581+0000","last_fullsized":"2024-09-20T22:43:28.138581+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":123,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.137326+0000","last_change":"2024-09-20T22:41:44.830917+0000","last_active":"2024-09-20T22:43:28.137326+0000","last_peered":"2024-09-20T22:43:28.137326+0000","last_clean":"2024-09-20T22:43:28.137326+0000","last_became_active":"2024-09-20T22:41:44.830538+0000","last_became_peered":"2024-09-20T22:41:44.830538+0000","last_unstale":"2024-09-20T22:43:28.137326+0000","last_undegraded":"2024-09-20T22:43:28.137326+0000","last_fullsized":"2024-09-20T22:43:28.137326+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.7","version":"125'82","reported_seq":249,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.139274+0000","last_change":"2024-09-20T22:43:23.133945+0000","last_active":"2024-09-20T22:43:28.139274+0000","last_peered":"2024-09-20T22:43:28.139274+0000","last_clean":"2024-09-20T22:43:28.139274+0000","last_became_active":"2024-09-20T22:42:52.404632+0000","last_became_peered":"2024-09-20T22:42:52.404632+0000","last_unstale":"2024-09-20T22:43:28.139274+0000","last_undegraded":"2024-09-20T22:43:28.139274+0000","last_fullsized":"2024-09-20T22:43:28.139274+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:42:51.354519+0000","objects_scrubbed":0,"log_size":82,"log_dups_size":0,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:17.643406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011352269,"stat_sum":{"num_bytes":15845660,"num_objects":6,"num_object_clones":2,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":28,"num_read_kb":6402,"num_write":58,"num_write_kb":21896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"77'128","reported_seq":392,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.127221+0000","last_change":"2024-09-20T22:43:26.761137+0000","last_active":"2024-09-20T22:43:28.127221+0000","last_peered":"2024-09-20T22:43:28.127221+0000","last_clean":"2024-09-20T22:43:28.127221+0000","last_became_active":"2024-09-20T22:43:19.125806+0000","last_became_peered":"2024-09-20T22:43:19.125806+0000","last_unstale":"2024-09-20T22:43:28.127221+0000","last_undegraded":"2024-09-20T22:43:28.127221+0000","last_fullsized":"2024-09-20T22:43:28.127221+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:43:26.761032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:43:26.761032+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:49.343774+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"86'38","reported_seq":220,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.137011+0000","last_change":"2024-09-20T22:42:43.935768+0000","last_active":"2024-09-20T22:43:28.137011+0000","last_peered":"2024-09-20T22:43:28.137011+0000","last_clean":"2024-09-20T22:43:28.137011+0000","last_became_active":"2024-09-20T22:41:40.166958+0000","last_became_peered":"2024-09-20T22:41:40.166958+0000","last_unstale":"2024-09-20T22:43:28.137011+0000","last_undegraded":"2024-09-20T22:43:28.137011+0000","last_fullsized":"2024-09-20T22:43:28.137011+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'38","last_scrub_stamp":"2024-09-20T22:42:43.935677+0000","last_deep_scrub":"86'38","last_deep_scrub_stamp":"2024-09-20T22:42:43.935677+0000","last_clean_scrub_stamp":"2024-09-20T22:42:43.935677+0000","objects_scrubbed":10,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:53.919117+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":143,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.115763+0000","last_change":"2024-09-20T22:42:37.810752+0000","last_active":"2024-09-20T22:43:28.115763+0000","last_peered":"2024-09-20T22:43:28.115763+0000","last_clean":"2024-09-20T22:43:28.115763+0000","last_became_active":"2024-09-20T22:41:38.152525+0000","last_became_peered":"2024-09-20T22:41:38.152525+0000","last_unstale":"2024-09-20T22:43:28.115763+0000","last_undegraded":"2024-09-20T22:43:28.115763+0000","last_fullsized":"2024-09-20T22:43:28.115763+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:42:37.810656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:42:37.810656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:57.321042+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"69'2","reported_seq":145,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.127710+0000","last_change":"2024-09-20T22:41:40.154106+0000","last_active":"2024-09-20T22:43:28.127710+0000","last_peered":"2024-09-20T22:43:28.127710+0000","last_clean":"2024-09-20T22:43:28.127710+0000","last_became_active":"2024-09-20T22:41:37.148479+0000","last_became_peered":"2024-09-20T22:41:37.148479+0000","last_unstale":"2024-09-20T22:43:28.127710+0000","last_undegraded":"2024-09-20T22:43:28.127710+0000","last_fullsized":"2024-09-20T22:43:28.127710+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:36.130684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:36.130684+0000","last_clean_scrub_stamp":"2024-09-20T22:41:36.130684+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087280599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":127,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.117501+0000","last_change":"2024-09-20T22:41:42.179979+0000","last_active":"2024-09-20T22:43:28.117501+0000","last_peered":"2024-09-20T22:43:28.117501+0000","last_clean":"2024-09-20T22:43:28.117501+0000","last_became_active":"2024-09-20T22:41:42.179742+0000","last_became_peered":"2024-09-20T22:41:42.179742+0000","last_unstale":"2024-09-20T22:43:28.117501+0000","last_undegraded":"2024-09-20T22:43:28.117501+0000","last_fullsized":"2024-09-20T22:43:28.117501+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":131,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.115405+0000","last_change":"2024-09-20T22:43:09.406207+0000","last_active":"2024-09-20T22:43:28.115405+0000","last_peered":"2024-09-20T22:43:28.115405+0000","last_clean":"2024-09-20T22:43:28.115405+0000","last_became_active":"2024-09-20T22:41:44.192128+0000","last_became_peered":"2024-09-20T22:41:44.192128+0000","last_unstale":"2024-09-20T22:43:28.115405+0000","last_undegraded":"2024-09-20T22:43:28.115405+0000","last_fullsized":"2024-09-20T22:43:28.115405+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:09.406141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:43:09.406141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:21.764472+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"129'14","reported_seq":181,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.184252+0000","last_change":"2024-09-20T22:43:23.124733+0000","last_active":"2024-09-20T22:43:28.184252+0000","last_peered":"2024-09-20T22:43:28.184252+0000","last_clean":"2024-09-20T22:43:28.184252+0000","last_became_active":"2024-09-20T22:42:52.400310+0000","last_became_peered":"2024-09-20T22:42:52.400310+0000","last_unstale":"2024-09-20T22:43:28.184252+0000","last_undegraded":"2024-09-20T22:43:28.184252+0000","last_fullsized":"2024-09-20T22:43:28.184252+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:42:51.354519+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:19.524690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0058968639999999999,"stat_sum":{"num_bytes":9025260,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":15,"num_read_kb":7779,"num_write":15,"num_write_kb":6749,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"77'128","reported_seq":433,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.121771+0000","last_change":"2024-09-20T22:43:25.207458+0000","last_active":"2024-09-20T22:43:28.121771+0000","last_peered":"2024-09-20T22:43:28.121771+0000","last_clean":"2024-09-20T22:43:28.121771+0000","last_became_active":"2024-09-20T22:43:19.122179+0000","last_became_peered":"2024-09-20T22:43:19.122179+0000","last_unstale":"2024-09-20T22:43:28.121771+0000","last_undegraded":"2024-09-20T22:43:28.121771+0000","last_fullsized":"2024-09-20T22:43:28.121771+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:43:25.207385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:43:25.207385+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:50.578381+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"74'13","reported_seq":158,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.136177+0000","last_change":"2024-09-20T22:41:40.169777+0000","last_active":"2024-09-20T22:43:28.136177+0000","last_peered":"2024-09-20T22:43:28.136177+0000","last_clean":"2024-09-20T22:43:28.136177+0000","last_became_active":"2024-09-20T22:41:40.169473+0000","last_became_peered":"2024-09-20T22:41:40.169473+0000","last_unstale":"2024-09-20T22:43:28.136177+0000","last_undegraded":"2024-09-20T22:43:28.136177+0000","last_fullsized":"2024-09-20T22:43:28.136177+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":143,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.135293+0000","last_change":"2024-09-20T22:43:12.048243+0000","last_active":"2024-09-20T22:43:28.135293+0000","last_peered":"2024-09-20T22:43:28.135293+0000","last_clean":"2024-09-20T22:43:28.135293+0000","last_became_active":"2024-09-20T22:41:38.156837+0000","last_became_peered":"2024-09-20T22:41:38.156837+0000","last_unstale":"2024-09-20T22:43:28.135293+0000","last_undegraded":"2024-09-20T22:43:28.135293+0000","last_fullsized":"2024-09-20T22:43:28.135293+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:12.048122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:43:12.048122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:29.457781+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"67'1","reported_seq":152,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.131308+0000","last_change":"2024-09-20T22:43:04.330733+0000","last_active":"2024-09-20T22:43:28.131308+0000","last_peered":"2024-09-20T22:43:28.131308+0000","last_clean":"2024-09-20T22:43:28.131308+0000","last_became_active":"2024-09-20T22:41:37.148165+0000","last_became_peered":"2024-09-20T22:41:37.148165+0000","last_unstale":"2024-09-20T22:43:28.131308+0000","last_undegraded":"2024-09-20T22:43:28.131308+0000","last_fullsized":"2024-09-20T22:43:28.131308+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-09-20T22:43:04.330598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:36.130684+0000","last_clean_scrub_stamp":"2024-09-20T22:43:04.330598+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:24.245637+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001301459,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":127,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.129819+0000","last_change":"2024-09-20T22:41:42.185412+0000","last_active":"2024-09-20T22:43:28.129819+0000","last_peered":"2024-09-20T22:43:28.129819+0000","last_clean":"2024-09-20T22:43:28.129819+0000","last_became_active":"2024-09-20T22:41:42.185112+0000","last_became_peered":"2024-09-20T22:41:42.185112+0000","last_unstale":"2024-09-20T22:43:28.129819+0000","last_undegraded":"2024-09-20T22:43:28.129819+0000","last_fullsized":"2024-09-20T22:43:28.129819+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:30.250551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":123,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.122533+0000","last_change":"2024-09-20T22:41:44.830804+0000","last_active":"2024-09-20T22:43:28.122533+0000","last_peered":"2024-09-20T22:43:28.122533+0000","last_clean":"2024-09-20T22:43:28.122533+0000","last_became_active":"2024-09-20T22:41:44.830515+0000","last_became_peered":"2024-09-20T22:41:44.830515+0000","last_unstale":"2024-09-20T22:43:28.122533+0000","last_undegraded":"2024-09-20T22:43:28.122533+0000","last_fullsized":"2024-09-20T22:43:28.122533+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:28.181516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"98'3","reported_seq":165,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.122508+0000","last_change":"2024-09-20T22:43:23.120147+0000","last_active":"2024-09-20T22:43:28.122508+0000","last_peered":"2024-09-20T22:43:28.122508+0000","last_clean":"2024-09-20T22:43:28.122508+0000","last_became_active":"2024-09-20T22:42:52.399644+0000","last_became_peered":"2024-09-20T22:42:52.399644+0000","last_unstale":"2024-09-20T22:43:28.122508+0000","last_undegraded":"2024-09-20T22:43:28.122508+0000","last_fullsized":"2024-09-20T22:43:28.122508+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:42:51.354519+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:55.382304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0016425140000000001,"stat_sum":{"num_bytes":2252105,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":4,"num_write_kb":878,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"77'128","reported_seq":392,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.137466+0000","last_change":"2024-09-20T22:43:25.244748+0000","last_active":"2024-09-20T22:43:28.137466+0000","last_peered":"2024-09-20T22:43:28.137466+0000","last_clean":"2024-09-20T22:43:28.137466+0000","last_became_active":"2024-09-20T22:43:19.126250+0000","last_became_peered":"2024-09-20T22:43:19.126250+0000","last_unstale":"2024-09-20T22:43:28.137466+0000","last_undegraded":"2024-09-20T22:43:28.137466+0000","last_fullsized":"2024-09-20T22:43:28.137466+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:43:25.244624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:43:25.244624+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:41.317253+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"74'12","reported_seq":177,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.119215+0000","last_change":"2024-09-20T22:43:26.720587+0000","last_active":"2024-09-20T22:43:28.119215+0000","last_peered":"2024-09-20T22:43:28.119215+0000","last_clean":"2024-09-20T22:43:28.119215+0000","last_became_active":"2024-09-20T22:43:09.120425+0000","last_became_peered":"2024-09-20T22:43:09.120425+0000","last_unstale":"2024-09-20T22:43:28.119215+0000","last_undegraded":"2024-09-20T22:43:28.119215+0000","last_fullsized":"2024-09-20T22:43:28.119215+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-09-20T22:43:26.720499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:43:26.720499+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:30.458476+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":135,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.117495+0000","last_change":"2024-09-20T22:41:38.153448+0000","last_active":"2024-09-20T22:43:28.117495+0000","last_peered":"2024-09-20T22:43:28.117495+0000","last_clean":"2024-09-20T22:43:28.117495+0000","last_became_active":"2024-09-20T22:41:38.153058+0000","last_became_peered":"2024-09-20T22:41:38.153058+0000","last_unstale":"2024-09-20T22:43:28.117495+0000","last_undegraded":"2024-09-20T22:43:28.117495+0000","last_fullsized":"2024-09-20T22:43:28.117495+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"75'5","reported_seq":251,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.121842+0000","last_change":"2024-09-20T22:41:40.251762+0000","last_active":"2024-09-20T22:43:28.121842+0000","last_peered":"2024-09-20T22:43:28.121842+0000","last_clean":"2024-09-20T22:43:28.121842+0000","last_became_active":"2024-09-20T22:41:37.148383+0000","last_became_peered":"2024-09-20T22:41:37.148383+0000","last_unstale":"2024-09-20T22:43:28.121842+0000","last_undegraded":"2024-09-20T22:43:28.121842+0000","last_fullsized":"2024-09-20T22:43:28.121842+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:36.130684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:36.130684+0000","last_clean_scrub_stamp":"2024-09-20T22:41:36.130684+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:28.181488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00078631700000000003,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":105,"num_read_kb":100,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":127,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.128331+0000","last_change":"2024-09-20T22:41:42.187618+0000","last_active":"2024-09-20T22:43:28.128331+0000","last_peered":"2024-09-20T22:43:28.128331+0000","last_clean":"2024-09-20T22:43:28.128331+0000","last_became_active":"2024-09-20T22:41:42.187346+0000","last_became_peered":"2024-09-20T22:41:42.187346+0000","last_unstale":"2024-09-20T22:43:28.128331+0000","last_undegraded":"2024-09-20T22:43:28.128331+0000","last_fullsized":"2024-09-20T22:43:28.128331+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":131,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.136553+0000","last_change":"2024-09-20T22:43:00.128772+0000","last_active":"2024-09-20T22:43:28.136553+0000","last_peered":"2024-09-20T22:43:28.136553+0000","last_clean":"2024-09-20T22:43:28.136553+0000","last_became_active":"2024-09-20T22:41:44.199046+0000","last_became_peered":"2024-09-20T22:41:44.199046+0000","last_unstale":"2024-09-20T22:43:28.136553+0000","last_undegraded":"2024-09-20T22:43:28.136553+0000","last_fullsized":"2024-09-20T22:43:28.136553+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:00.128705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:43:00.128705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:10.682100+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.6","version":"118'23","reported_seq":195,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.119436+0000","last_change":"2024-09-20T22:43:23.122356+0000","last_active":"2024-09-20T22:43:28.119436+0000","last_peered":"2024-09-20T22:43:28.119436+0000","last_clean":"2024-09-20T22:43:28.119436+0000","last_became_active":"2024-09-20T22:42:52.401182+0000","last_became_peered":"2024-09-20T22:42:52.401182+0000","last_unstale":"2024-09-20T22:43:28.119436+0000","last_undegraded":"2024-09-20T22:43:28.119436+0000","last_fullsized":"2024-09-20T22:43:28.119436+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:42:51.354519+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:20.109695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048656050000000003,"stat_sum":{"num_bytes":5563051,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":32,"num_read_kb":18305,"num_write":22,"num_write_kb":6134,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.5","version":"74'16","reported_seq":176,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.139115+0000","last_change":"2024-09-20T22:41:40.253524+0000","last_active":"2024-09-20T22:43:28.139115+0000","last_peered":"2024-09-20T22:43:28.139115+0000","last_clean":"2024-09-20T22:43:28.139115+0000","last_became_active":"2024-09-20T22:41:40.252933+0000","last_became_peered":"2024-09-20T22:41:40.252933+0000","last_unstale":"2024-09-20T22:43:28.139115+0000","last_undegraded":"2024-09-20T22:43:28.139115+0000","last_fullsized":"2024-09-20T22:43:28.139115+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"67'1","reported_seq":141,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.118696+0000","last_change":"2024-09-20T22:41:38.159706+0000","last_active":"2024-09-20T22:43:28.118696+0000","last_peered":"2024-09-20T22:43:28.118696+0000","last_clean":"2024-09-20T22:43:28.118696+0000","last_became_active":"2024-09-20T22:41:38.158917+0000","last_became_peered":"2024-09-20T22:41:38.158917+0000","last_unstale":"2024-09-20T22:43:28.118696+0000","last_undegraded":"2024-09-20T22:43:28.118696+0000","last_fullsized":"2024-09-20T22:43:28.118696+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:28.191495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"77'128","reported_seq":425,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.121953+0000","last_change":"2024-09-20T22:43:26.158086+0000","last_active":"2024-09-20T22:43:28.121953+0000","last_peered":"2024-09-20T22:43:28.121953+0000","last_clean":"2024-09-20T22:43:28.121953+0000","last_became_active":"2024-09-20T22:43:19.126925+0000","last_became_peered":"2024-09-20T22:43:19.126925+0000","last_unstale":"2024-09-20T22:43:28.121953+0000","last_undegraded":"2024-09-20T22:43:28.121953+0000","last_fullsized":"2024-09-20T22:43:28.121953+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:43:26.158017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:43:26.158017+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:49.938296+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":127,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.122628+0000","last_change":"2024-09-20T22:41:42.185135+0000","last_active":"2024-09-20T22:43:28.122628+0000","last_peered":"2024-09-20T22:43:28.122628+0000","last_clean":"2024-09-20T22:43:28.122628+0000","last_became_active":"2024-09-20T22:41:42.184387+0000","last_became_peered":"2024-09-20T22:41:42.184387+0000","last_unstale":"2024-09-20T22:43:28.122628+0000","last_undegraded":"2024-09-20T22:43:28.122628+0000","last_fullsized":"2024-09-20T22:43:28.122628+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:28.181368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":131,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.128239+0000","last_change":"2024-09-20T22:43:02.696182+0000","last_active":"2024-09-20T22:43:28.128239+0000","last_peered":"2024-09-20T22:43:28.128239+0000","last_clean":"2024-09-20T22:43:28.128239+0000","last_became_active":"2024-09-20T22:41:44.190658+0000","last_became_peered":"2024-09-20T22:41:44.190658+0000","last_unstale":"2024-09-20T22:43:28.128239+0000","last_undegraded":"2024-09-20T22:43:28.128239+0000","last_fullsized":"2024-09-20T22:43:28.128239+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:02.696105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:43:02.696105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:13.579400+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"77'128","reported_seq":392,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.137307+0000","last_change":"2024-09-20T22:43:24.124638+0000","last_active":"2024-09-20T22:43:28.137307+0000","last_peered":"2024-09-20T22:43:28.137307+0000","last_clean":"2024-09-20T22:43:28.137307+0000","last_became_active":"2024-09-20T22:43:24.124252+0000","last_became_peered":"2024-09-20T22:43:24.124252+0000","last_unstale":"2024-09-20T22:43:28.137307+0000","last_undegraded":"2024-09-20T22:43:28.137307+0000","last_fullsized":"2024-09-20T22:43:28.137307+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:43:03.999456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:43:03.999456+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:29.239969+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.d","version":"124'17","reported_seq":182,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.139012+0000","last_change":"2024-09-20T22:43:23.128959+0000","last_active":"2024-09-20T22:43:28.139012+0000","last_peered":"2024-09-20T22:43:28.139012+0000","last_clean":"2024-09-20T22:43:28.139012+0000","last_became_active":"2024-09-20T22:42:52.410381+0000","last_became_peered":"2024-09-20T22:42:52.410381+0000","last_unstale":"2024-09-20T22:43:28.139012+0000","last_undegraded":"2024-09-20T22:43:28.139012+0000","last_fullsized":"2024-09-20T22:43:28.139012+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:42:51.354519+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:53.579673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0069819360000000002,"stat_sum":{"num_bytes":7389886,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":14,"num_read_kb":3490,"num_write":14,"num_write_kb":6330,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"74'11","reported_seq":150,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.116218+0000","last_change":"2024-09-20T22:41:40.169542+0000","last_active":"2024-09-20T22:43:28.116218+0000","last_peered":"2024-09-20T22:43:28.116218+0000","last_clean":"2024-09-20T22:43:28.116218+0000","last_became_active":"2024-09-20T22:41:40.168942+0000","last_became_peered":"2024-09-20T22:41:40.168942+0000","last_unstale":"2024-09-20T22:43:28.116218+0000","last_undegraded":"2024-09-20T22:43:28.116218+0000","last_fullsized":"2024-09-20T22:43:28.116218+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":135,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.115728+0000","last_change":"2024-09-20T22:41:38.161801+0000","last_active":"2024-09-20T22:43:28.115728+0000","last_peered":"2024-09-20T22:43:28.115728+0000","last_clean":"2024-09-20T22:43:28.115728+0000","last_became_active":"2024-09-20T22:41:38.161500+0000","last_became_peered":"2024-09-20T22:41:38.161500+0000","last_unstale":"2024-09-20T22:43:28.115728+0000","last_undegraded":"2024-09-20T22:43:28.115728+0000","last_fullsized":"2024-09-20T22:43:28.115728+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":127,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.128198+0000","last_change":"2024-09-20T22:41:42.183601+0000","last_active":"2024-09-20T22:43:28.128198+0000","last_peered":"2024-09-20T22:43:28.128198+0000","last_clean":"2024-09-20T22:43:28.128198+0000","last_became_active":"2024-09-20T22:41:42.183302+0000","last_became_peered":"2024-09-20T22:41:42.183302+0000","last_unstale":"2024-09-20T22:43:28.128198+0000","last_undegraded":"2024-09-20T22:43:28.128198+0000","last_fullsized":"2024-09-20T22:43:28.128198+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":123,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.117162+0000","last_change":"2024-09-20T22:41:44.832295+0000","last_active":"2024-09-20T22:43:28.117162+0000","last_peered":"2024-09-20T22:43:28.117162+0000","last_clean":"2024-09-20T22:43:28.117162+0000","last_became_active":"2024-09-20T22:41:44.832009+0000","last_became_peered":"2024-09-20T22:41:44.832009+0000","last_unstale":"2024-09-20T22:43:28.117162+0000","last_undegraded":"2024-09-20T22:43:28.117162+0000","last_fullsized":"2024-09-20T22:43:28.117162+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:28.191392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.e","version":"129'8","reported_seq":178,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.153766+0000","last_change":"2024-09-20T22:43:23.123656+0000","last_active":"2024-09-20T22:43:28.153766+0000","last_peered":"2024-09-20T22:43:28.153766+0000","last_clean":"2024-09-20T22:43:28.153766+0000","last_became_active":"2024-09-20T22:42:52.405039+0000","last_became_peered":"2024-09-20T22:42:52.405039+0000","last_unstale":"2024-09-20T22:43:28.153766+0000","last_undegraded":"2024-09-20T22:43:28.153766+0000","last_fullsized":"2024-09-20T22:43:28.153766+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:42:51.354519+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:04.770203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001749614,"stat_sum":{"num_bytes":7058841,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":23,"num_read_kb":15328,"num_write":9,"num_write_kb":4406,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"77'128","reported_seq":379,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.127266+0000","last_change":"2024-09-20T22:43:04.748279+0000","last_active":"2024-09-20T22:43:28.127266+0000","last_peered":"2024-09-20T22:43:28.127266+0000","last_clean":"2024-09-20T22:43:28.127266+0000","last_became_active":"2024-09-20T22:42:52.405594+0000","last_became_peered":"2024-09-20T22:42:52.405594+0000","last_unstale":"2024-09-20T22:43:28.127266+0000","last_undegraded":"2024-09-20T22:43:28.127266+0000","last_fullsized":"2024-09-20T22:43:28.127266+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:43:04.748176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:43:04.748176+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:13.534518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"74'17","reported_seq":167,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.116139+0000","last_change":"2024-09-20T22:43:00.910562+0000","last_active":"2024-09-20T22:43:28.116139+0000","last_peered":"2024-09-20T22:43:28.116139+0000","last_clean":"2024-09-20T22:43:28.116139+0000","last_became_active":"2024-09-20T22:41:40.169192+0000","last_became_peered":"2024-09-20T22:41:40.169192+0000","last_unstale":"2024-09-20T22:43:28.116139+0000","last_undegraded":"2024-09-20T22:43:28.116139+0000","last_fullsized":"2024-09-20T22:43:28.116139+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'17","last_scrub_stamp":"2024-09-20T22:43:00.910480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:43:00.910480+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:30.015789+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":135,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.138759+0000","last_change":"2024-09-20T22:41:38.154220+0000","last_active":"2024-09-20T22:43:28.138759+0000","last_peered":"2024-09-20T22:43:28.138759+0000","last_clean":"2024-09-20T22:43:28.138759+0000","last_became_active":"2024-09-20T22:41:38.153926+0000","last_became_peered":"2024-09-20T22:41:38.153926+0000","last_unstale":"2024-09-20T22:43:28.138759+0000","last_undegraded":"2024-09-20T22:43:28.138759+0000","last_fullsized":"2024-09-20T22:43:28.138759+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":127,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.136661+0000","last_change":"2024-09-20T22:41:42.183482+0000","last_active":"2024-09-20T22:43:28.136661+0000","last_peered":"2024-09-20T22:43:28.136661+0000","last_clean":"2024-09-20T22:43:28.136661+0000","last_became_active":"2024-09-20T22:41:42.183225+0000","last_became_peered":"2024-09-20T22:41:42.183225+0000","last_unstale":"2024-09-20T22:43:28.136661+0000","last_undegraded":"2024-09-20T22:43:28.136661+0000","last_fullsized":"2024-09-20T22:43:28.136661+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":131,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.129106+0000","last_change":"2024-09-20T22:42:59.250714+0000","last_active":"2024-09-20T22:43:28.129106+0000","last_peered":"2024-09-20T22:43:28.129106+0000","last_clean":"2024-09-20T22:43:28.129106+0000","last_became_active":"2024-09-20T22:41:44.197561+0000","last_became_peered":"2024-09-20T22:41:44.197561+0000","last_unstale":"2024-09-20T22:43:28.129106+0000","last_undegraded":"2024-09-20T22:43:28.129106+0000","last_fullsized":"2024-09-20T22:43:28.129106+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:42:59.250616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:42:59.250616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:10.878091+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"129'20","reported_seq":188,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.127634+0000","last_change":"2024-09-20T22:43:23.125030+0000","last_active":"2024-09-20T22:43:28.127634+0000","last_peered":"2024-09-20T22:43:28.127634+0000","last_clean":"2024-09-20T22:43:28.127634+0000","last_became_active":"2024-09-20T22:42:52.401210+0000","last_became_peered":"2024-09-20T22:42:52.401210+0000","last_unstale":"2024-09-20T22:43:28.127634+0000","last_undegraded":"2024-09-20T22:43:28.127634+0000","last_fullsized":"2024-09-20T22:43:28.127634+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:42:51.354519+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:12.341937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0061391650000000002,"stat_sum":{"num_bytes":7932805,"num_objects":5,"num_object_clones":2,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":19,"num_read_kb":11559,"num_write":21,"num_write_kb":9368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.9","version":"77'128","reported_seq":420,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.121895+0000","last_change":"2024-09-20T22:43:03.239347+0000","last_active":"2024-09-20T22:43:28.121895+0000","last_peered":"2024-09-20T22:43:28.121895+0000","last_clean":"2024-09-20T22:43:28.121895+0000","last_became_active":"2024-09-20T22:42:52.400353+0000","last_became_peered":"2024-09-20T22:42:52.400353+0000","last_unstale":"2024-09-20T22:43:28.121895+0000","last_undegraded":"2024-09-20T22:43:28.121895+0000","last_fullsized":"2024-09-20T22:43:28.121895+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:43:03.239276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:43:03.239276+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:26.152766+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"74'10","reported_seq":146,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.115924+0000","last_change":"2024-09-20T22:41:40.162973+0000","last_active":"2024-09-20T22:43:28.115924+0000","last_peered":"2024-09-20T22:43:28.115924+0000","last_clean":"2024-09-20T22:43:28.115924+0000","last_became_active":"2024-09-20T22:41:40.162494+0000","last_became_peered":"2024-09-20T22:41:40.162494+0000","last_unstale":"2024-09-20T22:43:28.115924+0000","last_undegraded":"2024-09-20T22:43:28.115924+0000","last_fullsized":"2024-09-20T22:43:28.115924+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":143,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.116120+0000","last_change":"2024-09-20T22:43:05.431444+0000","last_active":"2024-09-20T22:43:28.116120+0000","last_peered":"2024-09-20T22:43:28.116120+0000","last_clean":"2024-09-20T22:43:28.116120+0000","last_became_active":"2024-09-20T22:41:38.154636+0000","last_became_peered":"2024-09-20T22:41:38.154636+0000","last_unstale":"2024-09-20T22:43:28.116120+0000","last_undegraded":"2024-09-20T22:43:28.116120+0000","last_fullsized":"2024-09-20T22:43:28.116120+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:05.431346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:43:05.431346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:30.313458+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"74'8","reported_seq":158,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.132705+0000","last_change":"2024-09-20T22:41:42.186528+0000","last_active":"2024-09-20T22:43:28.132705+0000","last_peered":"2024-09-20T22:43:28.132705+0000","last_clean":"2024-09-20T22:43:28.132705+0000","last_became_active":"2024-09-20T22:41:42.185863+0000","last_became_peered":"2024-09-20T22:41:42.185863+0000","last_unstale":"2024-09-20T22:43:28.132705+0000","last_undegraded":"2024-09-20T22:43:28.132705+0000","last_fullsized":"2024-09-20T22:43:28.132705+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:30.250519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":123,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.115996+0000","last_change":"2024-09-20T22:41:44.195767+0000","last_active":"2024-09-20T22:43:28.115996+0000","last_peered":"2024-09-20T22:43:28.115996+0000","last_clean":"2024-09-20T22:43:28.115996+0000","last_became_active":"2024-09-20T22:41:44.195160+0000","last_became_peered":"2024-09-20T22:41:44.195160+0000","last_unstale":"2024-09-20T22:43:28.115996+0000","last_undegraded":"2024-09-20T22:43:28.115996+0000","last_fullsized":"2024-09-20T22:43:28.115996+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.e","version":"77'128","reported_seq":309,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.115969+0000","last_change":"2024-09-20T22:43:24.126078+0000","last_active":"2024-09-20T22:43:28.115969+0000","last_peered":"2024-09-20T22:43:28.115969+0000","last_clean":"2024-09-20T22:43:28.115969+0000","last_became_active":"2024-09-20T22:43:24.125722+0000","last_became_peered":"2024-09-20T22:43:24.125722+0000","last_unstale":"2024-09-20T22:43:28.115969+0000","last_undegraded":"2024-09-20T22:43:28.115969+0000","last_fullsized":"2024-09-20T22:43:28.115969+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:42:59.831571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:42:59.831571+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:29.819811+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9,"num_read_kb":14,"num_write":8,"num_write_kb":110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":80505,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.8","version":"124'24","reported_seq":194,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.133180+0000","last_change":"2024-09-20T22:43:23.129906+0000","last_active":"2024-09-20T22:43:28.133180+0000","last_peered":"2024-09-20T22:43:28.133180+0000","last_clean":"2024-09-20T22:43:28.133180+0000","last_became_active":"2024-09-20T22:42:52.409070+0000","last_became_peered":"2024-09-20T22:42:52.409070+0000","last_unstale":"2024-09-20T22:43:28.133180+0000","last_undegraded":"2024-09-20T22:43:28.133180+0000","last_fullsized":"2024-09-20T22:43:28.133180+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:42:51.354519+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:16.822880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0062757969999999996,"stat_sum":{"num_bytes":10011785,"num_objects":5,"num_object_clones":2,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":28,"num_read_kb":24573,"num_write":23,"num_write_kb":6123,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"74'9","reported_seq":155,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.116011+0000","last_change":"2024-09-20T22:43:00.749532+0000","last_active":"2024-09-20T22:43:28.116011+0000","last_peered":"2024-09-20T22:43:28.116011+0000","last_clean":"2024-09-20T22:43:28.116011+0000","last_became_active":"2024-09-20T22:41:40.163495+0000","last_became_peered":"2024-09-20T22:41:40.163495+0000","last_unstale":"2024-09-20T22:43:28.116011+0000","last_undegraded":"2024-09-20T22:43:28.116011+0000","last_fullsized":"2024-09-20T22:43:28.116011+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-09-20T22:43:00.749456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:43:00.749456+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:15.372038+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":135,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.127914+0000","last_change":"2024-09-20T22:41:38.156470+0000","last_active":"2024-09-20T22:43:28.127914+0000","last_peered":"2024-09-20T22:43:28.127914+0000","last_clean":"2024-09-20T22:43:28.127914+0000","last_became_active":"2024-09-20T22:41:38.155953+0000","last_became_peered":"2024-09-20T22:41:38.155953+0000","last_unstale":"2024-09-20T22:43:28.127914+0000","last_undegraded":"2024-09-20T22:43:28.127914+0000","last_fullsized":"2024-09-20T22:43:28.127914+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":127,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.134277+0000","last_change":"2024-09-20T22:41:42.183269+0000","last_active":"2024-09-20T22:43:28.134277+0000","last_peered":"2024-09-20T22:43:28.134277+0000","last_clean":"2024-09-20T22:43:28.134277+0000","last_became_active":"2024-09-20T22:41:42.182999+0000","last_became_peered":"2024-09-20T22:41:42.182999+0000","last_unstale":"2024-09-20T22:43:28.134277+0000","last_undegraded":"2024-09-20T22:43:28.134277+0000","last_fullsized":"2024-09-20T22:43:28.134277+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:30.250499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":123,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.117427+0000","last_change":"2024-09-20T22:41:44.196969+0000","last_active":"2024-09-20T22:43:28.117427+0000","last_peered":"2024-09-20T22:43:28.117427+0000","last_clean":"2024-09-20T22:43:28.117427+0000","last_became_active":"2024-09-20T22:41:44.196633+0000","last_became_peered":"2024-09-20T22:41:44.196633+0000","last_unstale":"2024-09-20T22:43:28.117427+0000","last_undegraded":"2024-09-20T22:43:28.117427+0000","last_fullsized":"2024-09-20T22:43:28.117427+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.f","version":"77'128","reported_seq":399,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.139344+0000","last_change":"2024-09-20T22:43:24.125957+0000","last_active":"2024-09-20T22:43:28.139344+0000","last_peered":"2024-09-20T22:43:28.139344+0000","last_clean":"2024-09-20T22:43:28.139344+0000","last_became_active":"2024-09-20T22:43:24.125196+0000","last_became_peered":"2024-09-20T22:43:24.125196+0000","last_unstale":"2024-09-20T22:43:28.139344+0000","last_undegraded":"2024-09-20T22:43:28.139344+0000","last_fullsized":"2024-09-20T22:43:28.139344+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:42:58.847355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:42:58.847355+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:29.549759+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.9","version":"124'40","reported_seq":209,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.117013+0000","last_change":"2024-09-20T22:43:23.124908+0000","last_active":"2024-09-20T22:43:28.117013+0000","last_peered":"2024-09-20T22:43:28.117013+0000","last_clean":"2024-09-20T22:43:28.117013+0000","last_became_active":"2024-09-20T22:42:52.407089+0000","last_became_peered":"2024-09-20T22:42:52.407089+0000","last_unstale":"2024-09-20T22:43:28.117013+0000","last_undegraded":"2024-09-20T22:43:28.117013+0000","last_fullsized":"2024-09-20T22:43:28.117013+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:42:51.354519+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:00.035782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0060842350000000003,"stat_sum":{"num_bytes":14336206,"num_objects":6,"num_object_clones":2,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":25,"num_read_kb":12303,"num_write":34,"num_write_kb":15823,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"113'28","reported_seq":179,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.138610+0000","last_change":"2024-09-20T22:43:12.726018+0000","last_active":"2024-09-20T22:43:28.138610+0000","last_peered":"2024-09-20T22:43:28.138610+0000","last_clean":"2024-09-20T22:43:28.138610+0000","last_became_active":"2024-09-20T22:41:40.252011+0000","last_became_peered":"2024-09-20T22:41:40.252011+0000","last_unstale":"2024-09-20T22:43:28.138610+0000","last_undegraded":"2024-09-20T22:43:28.138610+0000","last_fullsized":"2024-09-20T22:43:28.138610+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'28","last_scrub_stamp":"2024-09-20T22:43:12.725900+0000","last_deep_scrub":"113'28","last_deep_scrub_stamp":"2024-09-20T22:43:12.725900+0000","last_clean_scrub_stamp":"2024-09-20T22:43:12.725900+0000","objects_scrubbed":9,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:21.371299+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":135,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.122753+0000","last_change":"2024-09-20T22:41:38.161800+0000","last_active":"2024-09-20T22:43:28.122753+0000","last_peered":"2024-09-20T22:43:28.122753+0000","last_clean":"2024-09-20T22:43:28.122753+0000","last_became_active":"2024-09-20T22:41:38.161524+0000","last_became_peered":"2024-09-20T22:41:38.161524+0000","last_unstale":"2024-09-20T22:43:28.122753+0000","last_undegraded":"2024-09-20T22:43:28.122753+0000","last_fullsized":"2024-09-20T22:43:28.122753+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:28.181543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":127,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.128798+0000","last_change":"2024-09-20T22:41:42.184605+0000","last_active":"2024-09-20T22:43:28.128798+0000","last_peered":"2024-09-20T22:43:28.128798+0000","last_clean":"2024-09-20T22:43:28.128798+0000","last_became_active":"2024-09-20T22:41:42.184342+0000","last_became_peered":"2024-09-20T22:41:42.184342+0000","last_unstale":"2024-09-20T22:43:28.128798+0000","last_undegraded":"2024-09-20T22:43:28.128798+0000","last_fullsized":"2024-09-20T22:43:28.128798+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:30.250536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":131,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.121921+0000","last_change":"2024-09-20T22:43:02.234978+0000","last_active":"2024-09-20T22:43:28.121921+0000","last_peered":"2024-09-20T22:43:28.121921+0000","last_clean":"2024-09-20T22:43:28.121921+0000","last_became_active":"2024-09-20T22:41:44.199395+0000","last_became_peered":"2024-09-20T22:41:44.199395+0000","last_unstale":"2024-09-20T22:43:28.121921+0000","last_undegraded":"2024-09-20T22:43:28.121921+0000","last_fullsized":"2024-09-20T22:43:28.121921+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:02.234872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:43:02.234872+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:25.214149+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.c","version":"77'128","reported_seq":398,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.117568+0000","last_change":"2024-09-20T22:43:24.131423+0000","last_active":"2024-09-20T22:43:28.117568+0000","last_peered":"2024-09-20T22:43:28.117568+0000","last_clean":"2024-09-20T22:43:28.117568+0000","last_became_active":"2024-09-20T22:43:24.130949+0000","last_became_peered":"2024-09-20T22:43:24.130949+0000","last_unstale":"2024-09-20T22:43:28.117568+0000","last_undegraded":"2024-09-20T22:43:28.117568+0000","last_fullsized":"2024-09-20T22:43:28.117568+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:43:02.448201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:43:02.448201+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:29.159789+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109227,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"128'32","reported_seq":204,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.118496+0000","last_change":"2024-09-20T22:43:23.132157+0000","last_active":"2024-09-20T22:43:28.118496+0000","last_peered":"2024-09-20T22:43:28.118496+0000","last_clean":"2024-09-20T22:43:28.118496+0000","last_became_active":"2024-09-20T22:42:52.405085+0000","last_became_peered":"2024-09-20T22:42:52.405085+0000","last_unstale":"2024-09-20T22:43:28.118496+0000","last_undegraded":"2024-09-20T22:43:28.118496+0000","last_fullsized":"2024-09-20T22:43:28.118496+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:42:51.354519+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:01.654438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0026246350000000002,"stat_sum":{"num_bytes":8268945,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":34,"num_read_kb":26714,"num_write":23,"num_write_kb":7915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"74'12","reported_seq":162,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.115598+0000","last_change":"2024-09-20T22:42:59.862739+0000","last_active":"2024-09-20T22:43:28.115598+0000","last_peered":"2024-09-20T22:43:28.115598+0000","last_clean":"2024-09-20T22:43:28.115598+0000","last_became_active":"2024-09-20T22:41:40.170133+0000","last_became_peered":"2024-09-20T22:41:40.170133+0000","last_unstale":"2024-09-20T22:43:28.115598+0000","last_undegraded":"2024-09-20T22:43:28.115598+0000","last_fullsized":"2024-09-20T22:43:28.115598+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-09-20T22:42:59.862632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:42:59.862632+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:23.948333+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":143,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.122458+0000","last_change":"2024-09-20T22:43:05.229655+0000","last_active":"2024-09-20T22:43:28.122458+0000","last_peered":"2024-09-20T22:43:28.122458+0000","last_clean":"2024-09-20T22:43:28.122458+0000","last_became_active":"2024-09-20T22:41:38.163292+0000","last_became_peered":"2024-09-20T22:41:38.163292+0000","last_unstale":"2024-09-20T22:43:28.122458+0000","last_undegraded":"2024-09-20T22:43:28.122458+0000","last_fullsized":"2024-09-20T22:43:28.122458+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:05.229558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:43:05.229558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:22.822578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":135,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.133339+0000","last_change":"2024-09-20T22:43:02.326233+0000","last_active":"2024-09-20T22:43:28.133339+0000","last_peered":"2024-09-20T22:43:28.133339+0000","last_clean":"2024-09-20T22:43:28.133339+0000","last_became_active":"2024-09-20T22:41:42.181811+0000","last_became_peered":"2024-09-20T22:41:42.181811+0000","last_unstale":"2024-09-20T22:43:28.133339+0000","last_undegraded":"2024-09-20T22:43:28.133339+0000","last_fullsized":"2024-09-20T22:43:28.133339+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:02.326135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:43:02.326135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:18.212222+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":123,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.115821+0000","last_change":"2024-09-20T22:41:44.193432+0000","last_active":"2024-09-20T22:43:28.115821+0000","last_peered":"2024-09-20T22:43:28.115821+0000","last_clean":"2024-09-20T22:43:28.115821+0000","last_became_active":"2024-09-20T22:41:44.192609+0000","last_became_peered":"2024-09-20T22:41:44.192609+0000","last_unstale":"2024-09-20T22:43:28.115821+0000","last_undegraded":"2024-09-20T22:43:28.115821+0000","last_fullsized":"2024-09-20T22:43:28.115821+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:28.191333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.d","version":"77'128","reported_seq":311,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.122066+0000","last_change":"2024-09-20T22:43:24.127538+0000","last_active":"2024-09-20T22:43:28.122066+0000","last_peered":"2024-09-20T22:43:28.122066+0000","last_clean":"2024-09-20T22:43:28.122066+0000","last_became_active":"2024-09-20T22:43:24.127144+0000","last_became_peered":"2024-09-20T22:43:24.127144+0000","last_unstale":"2024-09-20T22:43:28.122066+0000","last_undegraded":"2024-09-20T22:43:28.122066+0000","last_fullsized":"2024-09-20T22:43:28.122066+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:42:58.843419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:42:58.843419+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:29.209060+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9,"num_read_kb":14,"num_write":8,"num_write_kb":110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":80505,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.b","version":"129'30","reported_seq":202,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.169776+0000","last_change":"2024-09-20T22:43:23.123639+0000","last_active":"2024-09-20T22:43:28.169776+0000","last_peered":"2024-09-20T22:43:28.169776+0000","last_clean":"2024-09-20T22:43:28.169776+0000","last_became_active":"2024-09-20T22:42:52.409642+0000","last_became_peered":"2024-09-20T22:42:52.409642+0000","last_unstale":"2024-09-20T22:43:28.169776+0000","last_undegraded":"2024-09-20T22:43:28.169776+0000","last_fullsized":"2024-09-20T22:43:28.169776+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:42:51.354519+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:08.531599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047938499999999998,"stat_sum":{"num_bytes":12722734,"num_objects":5,"num_object_clones":1,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":14862,"num_write":27,"num_write_kb":11685,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"74'15","reported_seq":156,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.127487+0000","last_change":"2024-09-20T22:41:40.252427+0000","last_active":"2024-09-20T22:43:28.127487+0000","last_peered":"2024-09-20T22:43:28.127487+0000","last_clean":"2024-09-20T22:43:28.127487+0000","last_became_active":"2024-09-20T22:41:40.252113+0000","last_became_peered":"2024-09-20T22:41:40.252113+0000","last_unstale":"2024-09-20T22:43:28.127487+0000","last_undegraded":"2024-09-20T22:43:28.127487+0000","last_fullsized":"2024-09-20T22:43:28.127487+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'2","reported_seq":147,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.122856+0000","last_change":"2024-09-20T22:41:38.163593+0000","last_active":"2024-09-20T22:43:28.122856+0000","last_peered":"2024-09-20T22:43:28.122856+0000","last_clean":"2024-09-20T22:43:28.122856+0000","last_became_active":"2024-09-20T22:41:38.163219+0000","last_became_peered":"2024-09-20T22:41:38.163219+0000","last_unstale":"2024-09-20T22:43:28.122856+0000","last_undegraded":"2024-09-20T22:43:28.122856+0000","last_fullsized":"2024-09-20T22:43:28.122856+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:28.181457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"74'8","reported_seq":166,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.121832+0000","last_change":"2024-09-20T22:43:11.266918+0000","last_active":"2024-09-20T22:43:28.121832+0000","last_peered":"2024-09-20T22:43:28.121832+0000","last_clean":"2024-09-20T22:43:28.121832+0000","last_became_active":"2024-09-20T22:41:42.186635+0000","last_became_peered":"2024-09-20T22:41:42.186635+0000","last_unstale":"2024-09-20T22:43:28.121832+0000","last_undegraded":"2024-09-20T22:43:28.121832+0000","last_fullsized":"2024-09-20T22:43:28.121832+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-09-20T22:43:11.266835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:43:11.266835+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:21.514341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":123,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.127458+0000","last_change":"2024-09-20T22:41:44.831435+0000","last_active":"2024-09-20T22:43:28.127458+0000","last_peered":"2024-09-20T22:43:28.127458+0000","last_clean":"2024-09-20T22:43:28.127458+0000","last_became_active":"2024-09-20T22:41:44.830556+0000","last_became_peered":"2024-09-20T22:41:44.830556+0000","last_unstale":"2024-09-20T22:43:28.127458+0000","last_undegraded":"2024-09-20T22:43:28.127458+0000","last_fullsized":"2024-09-20T22:43:28.127458+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.12","version":"77'128","reported_seq":309,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.127563+0000","last_change":"2024-09-20T22:43:24.124712+0000","last_active":"2024-09-20T22:43:28.127563+0000","last_peered":"2024-09-20T22:43:28.127563+0000","last_clean":"2024-09-20T22:43:28.127563+0000","last_became_active":"2024-09-20T22:43:24.124301+0000","last_became_peered":"2024-09-20T22:43:24.124301+0000","last_unstale":"2024-09-20T22:43:28.127563+0000","last_undegraded":"2024-09-20T22:43:28.127563+0000","last_fullsized":"2024-09-20T22:43:28.127563+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:43:04.267724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:43:04.267724+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:29.719569+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":9,"num_read_kb":14,"num_write":9,"num_write_kb":110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":80505,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":143,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.138505+0000","last_change":"2024-09-20T22:43:01.806383+0000","last_active":"2024-09-20T22:43:28.138505+0000","last_peered":"2024-09-20T22:43:28.138505+0000","last_clean":"2024-09-20T22:43:28.138505+0000","last_became_active":"2024-09-20T22:41:38.159139+0000","last_became_peered":"2024-09-20T22:41:38.159139+0000","last_unstale":"2024-09-20T22:43:28.138505+0000","last_undegraded":"2024-09-20T22:43:28.138505+0000","last_fullsized":"2024-09-20T22:43:28.138505+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:01.806314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:43:01.806314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:16.773475+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"74'6","reported_seq":162,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.116793+0000","last_change":"2024-09-20T22:43:09.121694+0000","last_active":"2024-09-20T22:43:28.116793+0000","last_peered":"2024-09-20T22:43:28.116793+0000","last_clean":"2024-09-20T22:43:28.116793+0000","last_became_active":"2024-09-20T22:43:09.116887+0000","last_became_peered":"2024-09-20T22:43:09.116887+0000","last_unstale":"2024-09-20T22:43:28.116793+0000","last_undegraded":"2024-09-20T22:43:28.116793+0000","last_fullsized":"2024-09-20T22:43:28.116793+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'6","last_scrub_stamp":"2024-09-20T22:42:41.469214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:42:41.469214+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:54.350192+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":135,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.128086+0000","last_change":"2024-09-20T22:43:09.705715+0000","last_active":"2024-09-20T22:43:28.128086+0000","last_peered":"2024-09-20T22:43:28.128086+0000","last_clean":"2024-09-20T22:43:28.128086+0000","last_became_active":"2024-09-20T22:41:42.184930+0000","last_became_peered":"2024-09-20T22:41:42.184930+0000","last_unstale":"2024-09-20T22:43:28.128086+0000","last_undegraded":"2024-09-20T22:43:28.128086+0000","last_fullsized":"2024-09-20T22:43:28.128086+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:09.705632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:09.705632+0000","last_clean_scrub_stamp":"2024-09-20T22:43:09.705632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:29.284713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"74'1","reported_seq":124,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.122390+0000","last_change":"2024-09-20T22:41:44.832085+0000","last_active":"2024-09-20T22:43:28.122390+0000","last_peered":"2024-09-20T22:43:28.122390+0000","last_clean":"2024-09-20T22:43:28.122390+0000","last_became_active":"2024-09-20T22:41:44.831805+0000","last_became_peered":"2024-09-20T22:41:44.831805+0000","last_unstale":"2024-09-20T22:43:28.122390+0000","last_undegraded":"2024-09-20T22:43:28.122390+0000","last_fullsized":"2024-09-20T22:43:28.122390+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:28.181270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.13","version":"77'128","reported_seq":392,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.137131+0000","last_change":"2024-09-20T22:43:24.126825+0000","last_active":"2024-09-20T22:43:28.137131+0000","last_peered":"2024-09-20T22:43:28.137131+0000","last_clean":"2024-09-20T22:43:28.137131+0000","last_became_active":"2024-09-20T22:43:24.126469+0000","last_became_peered":"2024-09-20T22:43:24.126469+0000","last_unstale":"2024-09-20T22:43:28.137131+0000","last_undegraded":"2024-09-20T22:43:28.137131+0000","last_fullsized":"2024-09-20T22:43:28.137131+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:43:03.999456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:43:03.999456+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:29.240009+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"74'9","reported_seq":155,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.119314+0000","last_change":"2024-09-20T22:42:43.666853+0000","last_active":"2024-09-20T22:43:28.119314+0000","last_peered":"2024-09-20T22:43:28.119314+0000","last_clean":"2024-09-20T22:43:28.119314+0000","last_became_active":"2024-09-20T22:41:40.253280+0000","last_became_peered":"2024-09-20T22:41:40.253280+0000","last_unstale":"2024-09-20T22:43:28.119314+0000","last_undegraded":"2024-09-20T22:43:28.119314+0000","last_fullsized":"2024-09-20T22:43:28.119314+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-09-20T22:42:43.666774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:42:43.666774+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:12.132272+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":135,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.122004+0000","last_change":"2024-09-20T22:41:38.156773+0000","last_active":"2024-09-20T22:43:28.122004+0000","last_peered":"2024-09-20T22:43:28.122004+0000","last_clean":"2024-09-20T22:43:28.122004+0000","last_became_active":"2024-09-20T22:41:38.156497+0000","last_became_peered":"2024-09-20T22:41:38.156497+0000","last_unstale":"2024-09-20T22:43:28.122004+0000","last_undegraded":"2024-09-20T22:43:28.122004+0000","last_fullsized":"2024-09-20T22:43:28.122004+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:28.181426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":127,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.116593+0000","last_change":"2024-09-20T22:41:42.188166+0000","last_active":"2024-09-20T22:43:28.116593+0000","last_peered":"2024-09-20T22:43:28.116593+0000","last_clean":"2024-09-20T22:43:28.116593+0000","last_became_active":"2024-09-20T22:41:42.187706+0000","last_became_peered":"2024-09-20T22:41:42.187706+0000","last_unstale":"2024-09-20T22:43:28.116593+0000","last_undegraded":"2024-09-20T22:43:28.116593+0000","last_fullsized":"2024-09-20T22:43:28.116593+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:28.191270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":131,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.130462+0000","last_change":"2024-09-20T22:43:08.232359+0000","last_active":"2024-09-20T22:43:28.130462+0000","last_peered":"2024-09-20T22:43:28.130462+0000","last_clean":"2024-09-20T22:43:28.130462+0000","last_became_active":"2024-09-20T22:41:44.201433+0000","last_became_peered":"2024-09-20T22:41:44.201433+0000","last_unstale":"2024-09-20T22:43:28.130462+0000","last_undegraded":"2024-09-20T22:43:28.130462+0000","last_fullsized":"2024-09-20T22:43:28.130462+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:08.232288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:43:08.232288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:08.925242+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.10","version":"77'128","reported_seq":309,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.127151+0000","last_change":"2024-09-20T22:43:24.125968+0000","last_active":"2024-09-20T22:43:28.127151+0000","last_peered":"2024-09-20T22:43:28.127151+0000","last_clean":"2024-09-20T22:43:28.127151+0000","last_became_active":"2024-09-20T22:43:24.125651+0000","last_became_peered":"2024-09-20T22:43:24.125651+0000","last_unstale":"2024-09-20T22:43:28.127151+0000","last_undegraded":"2024-09-20T22:43:28.127151+0000","last_fullsized":"2024-09-20T22:43:28.127151+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:42:59.238379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:42:59.238379+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:29.719611+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":9,"num_read_kb":13,"num_write":8,"num_write_kb":110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":80505,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.15","version":"74'9","reported_seq":147,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.127940+0000","last_change":"2024-09-20T22:41:40.252840+0000","last_active":"2024-09-20T22:43:28.127940+0000","last_peered":"2024-09-20T22:43:28.127940+0000","last_clean":"2024-09-20T22:43:28.127940+0000","last_became_active":"2024-09-20T22:41:40.252486+0000","last_became_peered":"2024-09-20T22:41:40.252486+0000","last_unstale":"2024-09-20T22:43:28.127940+0000","last_undegraded":"2024-09-20T22:43:28.127940+0000","last_fullsized":"2024-09-20T22:43:28.127940+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":135,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.114917+0000","last_change":"2024-09-20T22:41:38.158479+0000","last_active":"2024-09-20T22:43:28.114917+0000","last_peered":"2024-09-20T22:43:28.114917+0000","last_clean":"2024-09-20T22:43:28.114917+0000","last_became_active":"2024-09-20T22:41:38.158146+0000","last_became_peered":"2024-09-20T22:41:38.158146+0000","last_unstale":"2024-09-20T22:43:28.114917+0000","last_undegraded":"2024-09-20T22:43:28.114917+0000","last_fullsized":"2024-09-20T22:43:28.114917+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"74'8","reported_seq":158,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.117332+0000","last_change":"2024-09-20T22:41:42.186630+0000","last_active":"2024-09-20T22:43:28.117332+0000","last_peered":"2024-09-20T22:43:28.117332+0000","last_clean":"2024-09-20T22:43:28.117332+0000","last_became_active":"2024-09-20T22:41:42.186352+0000","last_became_peered":"2024-09-20T22:41:42.186352+0000","last_unstale":"2024-09-20T22:43:28.117332+0000","last_undegraded":"2024-09-20T22:43:28.117332+0000","last_fullsized":"2024-09-20T22:43:28.117332+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":123,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.116341+0000","last_change":"2024-09-20T22:41:44.194510+0000","last_active":"2024-09-20T22:43:28.116341+0000","last_peered":"2024-09-20T22:43:28.116341+0000","last_clean":"2024-09-20T22:43:28.116341+0000","last_became_active":"2024-09-20T22:41:44.194053+0000","last_became_peered":"2024-09-20T22:41:44.194053+0000","last_unstale":"2024-09-20T22:43:28.116341+0000","last_undegraded":"2024-09-20T22:43:28.116341+0000","last_fullsized":"2024-09-20T22:43:28.116341+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.11","version":"77'128","reported_seq":323,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.117582+0000","last_change":"2024-09-20T22:43:24.129214+0000","last_active":"2024-09-20T22:43:28.117582+0000","last_peered":"2024-09-20T22:43:28.117582+0000","last_clean":"2024-09-20T22:43:28.117582+0000","last_became_active":"2024-09-20T22:43:24.128884+0000","last_became_peered":"2024-09-20T22:43:24.128884+0000","last_unstale":"2024-09-20T22:43:28.117582+0000","last_undegraded":"2024-09-20T22:43:28.117582+0000","last_fullsized":"2024-09-20T22:43:28.117582+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:43:03.743457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:43:03.743457+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:29.726038+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26835,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":9,"num_read_kb":14,"num_write":9,"num_write_kb":110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":80505,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"74'10","reported_seq":154,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.117687+0000","last_change":"2024-09-20T22:43:11.355914+0000","last_active":"2024-09-20T22:43:28.117687+0000","last_peered":"2024-09-20T22:43:28.117687+0000","last_clean":"2024-09-20T22:43:28.117687+0000","last_became_active":"2024-09-20T22:41:40.253614+0000","last_became_peered":"2024-09-20T22:41:40.253614+0000","last_unstale":"2024-09-20T22:43:28.117687+0000","last_undegraded":"2024-09-20T22:43:28.117687+0000","last_fullsized":"2024-09-20T22:43:28.117687+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-09-20T22:43:11.355829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:43:11.355829+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:38.251196+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":135,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.122326+0000","last_change":"2024-09-20T22:41:38.158008+0000","last_active":"2024-09-20T22:43:28.122326+0000","last_peered":"2024-09-20T22:43:28.122326+0000","last_clean":"2024-09-20T22:43:28.122326+0000","last_became_active":"2024-09-20T22:41:38.157647+0000","last_became_peered":"2024-09-20T22:41:38.157647+0000","last_unstale":"2024-09-20T22:43:28.122326+0000","last_undegraded":"2024-09-20T22:43:28.122326+0000","last_fullsized":"2024-09-20T22:43:28.122326+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:28.181327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"74'8","reported_seq":169,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.127888+0000","last_change":"2024-09-20T22:42:59.718431+0000","last_active":"2024-09-20T22:43:28.127888+0000","last_peered":"2024-09-20T22:43:28.127888+0000","last_clean":"2024-09-20T22:43:28.127888+0000","last_became_active":"2024-09-20T22:41:42.189451+0000","last_became_peered":"2024-09-20T22:41:42.189451+0000","last_unstale":"2024-09-20T22:43:28.127888+0000","last_undegraded":"2024-09-20T22:43:28.127888+0000","last_fullsized":"2024-09-20T22:43:28.127888+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-09-20T22:42:59.718354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:42:59.718354+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:07.105181+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":131,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.118077+0000","last_change":"2024-09-20T22:42:58.680721+0000","last_active":"2024-09-20T22:43:28.118077+0000","last_peered":"2024-09-20T22:43:28.118077+0000","last_clean":"2024-09-20T22:43:28.118077+0000","last_became_active":"2024-09-20T22:41:44.191206+0000","last_became_peered":"2024-09-20T22:41:44.191206+0000","last_unstale":"2024-09-20T22:43:28.118077+0000","last_undegraded":"2024-09-20T22:43:28.118077+0000","last_fullsized":"2024-09-20T22:43:28.118077+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:42:58.680648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:42:58.680648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:22.824715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"74'11","reported_seq":150,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.115622+0000","last_change":"2024-09-20T22:41:40.171449+0000","last_active":"2024-09-20T22:43:28.115622+0000","last_peered":"2024-09-20T22:43:28.115622+0000","last_clean":"2024-09-20T22:43:28.115622+0000","last_became_active":"2024-09-20T22:41:40.170035+0000","last_became_peered":"2024-09-20T22:41:40.170035+0000","last_unstale":"2024-09-20T22:43:28.115622+0000","last_undegraded":"2024-09-20T22:43:28.115622+0000","last_fullsized":"2024-09-20T22:43:28.115622+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":135,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.116050+0000","last_change":"2024-09-20T22:41:38.161385+0000","last_active":"2024-09-20T22:43:28.116050+0000","last_peered":"2024-09-20T22:43:28.116050+0000","last_clean":"2024-09-20T22:43:28.116050+0000","last_became_active":"2024-09-20T22:41:38.158142+0000","last_became_peered":"2024-09-20T22:41:38.158142+0000","last_unstale":"2024-09-20T22:43:28.116050+0000","last_undegraded":"2024-09-20T22:43:28.116050+0000","last_fullsized":"2024-09-20T22:43:28.116050+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":127,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.136566+0000","last_change":"2024-09-20T22:41:42.182119+0000","last_active":"2024-09-20T22:43:28.136566+0000","last_peered":"2024-09-20T22:43:28.136566+0000","last_clean":"2024-09-20T22:43:28.136566+0000","last_became_active":"2024-09-20T22:41:42.181736+0000","last_became_peered":"2024-09-20T22:41:42.181736+0000","last_unstale":"2024-09-20T22:43:28.136566+0000","last_undegraded":"2024-09-20T22:43:28.136566+0000","last_fullsized":"2024-09-20T22:43:28.136566+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":123,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.114986+0000","last_change":"2024-09-20T22:41:44.193289+0000","last_active":"2024-09-20T22:43:28.114986+0000","last_peered":"2024-09-20T22:43:28.114986+0000","last_clean":"2024-09-20T22:43:28.114986+0000","last_became_active":"2024-09-20T22:41:44.192960+0000","last_became_peered":"2024-09-20T22:41:44.192960+0000","last_unstale":"2024-09-20T22:43:28.114986+0000","last_undegraded":"2024-09-20T22:43:28.114986+0000","last_fullsized":"2024-09-20T22:43:28.114986+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:28.191319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"74'9","reported_seq":155,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.210941+0000","last_change":"2024-09-20T22:43:28.210970+0000","last_active":"2024-09-20T22:43:28.210941+0000","last_peered":"2024-09-20T22:43:28.210941+0000","last_clean":"2024-09-20T22:43:28.210941+0000","last_became_active":"2024-09-20T22:41:40.164796+0000","last_became_peered":"2024-09-20T22:41:40.164796+0000","last_unstale":"2024-09-20T22:43:28.210941+0000","last_undegraded":"2024-09-20T22:43:28.210941+0000","last_fullsized":"2024-09-20T22:43:28.210941+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-09-20T22:43:28.210878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:43:28.210878+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:36.368330+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":138,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.122683+0000","last_change":"2024-09-20T22:43:14.159358+0000","last_active":"2024-09-20T22:43:28.122683+0000","last_peered":"2024-09-20T22:43:28.122683+0000","last_clean":"2024-09-20T22:43:28.122683+0000","last_became_active":"2024-09-20T22:41:38.159657+0000","last_became_peered":"2024-09-20T22:41:38.159657+0000","last_unstale":"2024-09-20T22:43:28.122683+0000","last_undegraded":"2024-09-20T22:43:28.122683+0000","last_fullsized":"2024-09-20T22:43:28.122683+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:28.181233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":127,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.115913+0000","last_change":"2024-09-20T22:41:42.183254+0000","last_active":"2024-09-20T22:43:28.115913+0000","last_peered":"2024-09-20T22:43:28.115913+0000","last_clean":"2024-09-20T22:43:28.115913+0000","last_became_active":"2024-09-20T22:41:42.182966+0000","last_became_peered":"2024-09-20T22:41:42.182966+0000","last_unstale":"2024-09-20T22:43:28.115913+0000","last_undegraded":"2024-09-20T22:43:28.115913+0000","last_fullsized":"2024-09-20T22:43:28.115913+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:28.191347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":131,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.114306+0000","last_change":"2024-09-20T22:43:04.649683+0000","last_active":"2024-09-20T22:43:28.114306+0000","last_peered":"2024-09-20T22:43:28.114306+0000","last_clean":"2024-09-20T22:43:28.114306+0000","last_became_active":"2024-09-20T22:41:44.194551+0000","last_became_peered":"2024-09-20T22:41:44.194551+0000","last_unstale":"2024-09-20T22:43:28.114306+0000","last_undegraded":"2024-09-20T22:43:28.114306+0000","last_fullsized":"2024-09-20T22:43:28.114306+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:04.649566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:43:04.649566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:09.553594+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.14","version":"77'128","reported_seq":387,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.130353+0000","last_change":"2024-09-20T22:43:24.126021+0000","last_active":"2024-09-20T22:43:28.130353+0000","last_peered":"2024-09-20T22:43:28.130353+0000","last_clean":"2024-09-20T22:43:28.130353+0000","last_became_active":"2024-09-20T22:43:24.125656+0000","last_became_peered":"2024-09-20T22:43:24.125656+0000","last_unstale":"2024-09-20T22:43:28.130353+0000","last_undegraded":"2024-09-20T22:43:28.130353+0000","last_fullsized":"2024-09-20T22:43:28.130353+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:43:02.448201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:43:02.448201+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:29.205672+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109227,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"74'11","reported_seq":150,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.115179+0000","last_change":"2024-09-20T22:41:40.252149+0000","last_active":"2024-09-20T22:43:28.115179+0000","last_peered":"2024-09-20T22:43:28.115179+0000","last_clean":"2024-09-20T22:43:28.115179+0000","last_became_active":"2024-09-20T22:41:40.251829+0000","last_became_peered":"2024-09-20T22:41:40.251829+0000","last_unstale":"2024-09-20T22:43:28.115179+0000","last_undegraded":"2024-09-20T22:43:28.115179+0000","last_fullsized":"2024-09-20T22:43:28.115179+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:28.191378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":143,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.128145+0000","last_change":"2024-09-20T22:43:06.775869+0000","last_active":"2024-09-20T22:43:28.128145+0000","last_peered":"2024-09-20T22:43:28.128145+0000","last_clean":"2024-09-20T22:43:28.128145+0000","last_became_active":"2024-09-20T22:41:38.158701+0000","last_became_peered":"2024-09-20T22:41:38.158701+0000","last_unstale":"2024-09-20T22:43:28.128145+0000","last_undegraded":"2024-09-20T22:43:28.128145+0000","last_fullsized":"2024-09-20T22:43:28.128145+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:06.775781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:06.775781+0000","last_clean_scrub_stamp":"2024-09-20T22:43:06.775781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:09.259667+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":127,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.121614+0000","last_change":"2024-09-20T22:41:42.185991+0000","last_active":"2024-09-20T22:43:28.121614+0000","last_peered":"2024-09-20T22:43:28.121614+0000","last_clean":"2024-09-20T22:43:28.121614+0000","last_became_active":"2024-09-20T22:41:42.185296+0000","last_became_peered":"2024-09-20T22:41:42.185296+0000","last_unstale":"2024-09-20T22:43:28.121614+0000","last_undegraded":"2024-09-20T22:43:28.121614+0000","last_fullsized":"2024-09-20T22:43:28.121614+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:28.181300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":123,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.116005+0000","last_change":"2024-09-20T22:41:44.832293+0000","last_active":"2024-09-20T22:43:28.116005+0000","last_peered":"2024-09-20T22:43:28.116005+0000","last_clean":"2024-09-20T22:43:28.116005+0000","last_became_active":"2024-09-20T22:41:44.831967+0000","last_became_peered":"2024-09-20T22:41:44.831967+0000","last_unstale":"2024-09-20T22:43:28.116005+0000","last_undegraded":"2024-09-20T22:43:28.116005+0000","last_fullsized":"2024-09-20T22:43:28.116005+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:28.191451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"74'4","reported_seq":137,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.114594+0000","last_change":"2024-09-20T22:41:40.166409+0000","last_active":"2024-09-20T22:43:28.114594+0000","last_peered":"2024-09-20T22:43:28.114594+0000","last_clean":"2024-09-20T22:43:28.114594+0000","last_became_active":"2024-09-20T22:41:40.166123+0000","last_became_peered":"2024-09-20T22:41:40.166123+0000","last_unstale":"2024-09-20T22:43:28.114594+0000","last_undegraded":"2024-09-20T22:43:28.114594+0000","last_fullsized":"2024-09-20T22:43:28.114594+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:28.191363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":143,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.115612+0000","last_change":"2024-09-20T22:43:03.680628+0000","last_active":"2024-09-20T22:43:28.115612+0000","last_peered":"2024-09-20T22:43:28.115612+0000","last_clean":"2024-09-20T22:43:28.115612+0000","last_became_active":"2024-09-20T22:41:38.158347+0000","last_became_peered":"2024-09-20T22:41:38.158347+0000","last_unstale":"2024-09-20T22:43:28.115612+0000","last_undegraded":"2024-09-20T22:43:28.115612+0000","last_fullsized":"2024-09-20T22:43:28.115612+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:03.680528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:43:03.680528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:18.111001+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":135,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.138456+0000","last_change":"2024-09-20T22:43:06.821377+0000","last_active":"2024-09-20T22:43:28.138456+0000","last_peered":"2024-09-20T22:43:28.138456+0000","last_clean":"2024-09-20T22:43:28.138456+0000","last_became_active":"2024-09-20T22:41:42.186068+0000","last_became_peered":"2024-09-20T22:41:42.186068+0000","last_unstale":"2024-09-20T22:43:28.138456+0000","last_undegraded":"2024-09-20T22:43:28.138456+0000","last_fullsized":"2024-09-20T22:43:28.138456+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:06.821310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:43:06.821310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:29.282062+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":131,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.121908+0000","last_change":"2024-09-20T22:43:10.233593+0000","last_active":"2024-09-20T22:43:28.121908+0000","last_peered":"2024-09-20T22:43:28.121908+0000","last_clean":"2024-09-20T22:43:28.121908+0000","last_became_active":"2024-09-20T22:41:44.193706+0000","last_became_peered":"2024-09-20T22:41:44.193706+0000","last_unstale":"2024-09-20T22:43:28.121908+0000","last_undegraded":"2024-09-20T22:43:28.121908+0000","last_fullsized":"2024-09-20T22:43:28.121908+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:10.233487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:10.233487+0000","last_clean_scrub_stamp":"2024-09-20T22:43:10.233487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:25.761110+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":123,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.136459+0000","last_change":"2024-09-20T22:41:44.198154+0000","last_active":"2024-09-20T22:43:28.136459+0000","last_peered":"2024-09-20T22:43:28.136459+0000","last_clean":"2024-09-20T22:43:28.136459+0000","last_became_active":"2024-09-20T22:41:44.197855+0000","last_became_peered":"2024-09-20T22:41:44.197855+0000","last_unstale":"2024-09-20T22:43:28.136459+0000","last_undegraded":"2024-09-20T22:43:28.136459+0000","last_fullsized":"2024-09-20T22:43:28.136459+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":143,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.114727+0000","last_change":"2024-09-20T22:43:00.430541+0000","last_active":"2024-09-20T22:43:28.114727+0000","last_peered":"2024-09-20T22:43:28.114727+0000","last_clean":"2024-09-20T22:43:28.114727+0000","last_became_active":"2024-09-20T22:41:38.154703+0000","last_became_peered":"2024-09-20T22:41:38.154703+0000","last_unstale":"2024-09-20T22:43:28.114727+0000","last_undegraded":"2024-09-20T22:43:28.114727+0000","last_fullsized":"2024-09-20T22:43:28.114727+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:00.430469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:43:00.430469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:08.693135+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"74'11","reported_seq":158,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.500016+0000","last_change":"2024-09-20T22:43:28.500040+0000","last_active":"2024-09-20T22:43:28.500016+0000","last_peered":"2024-09-20T22:43:28.500016+0000","last_clean":"2024-09-20T22:43:28.500016+0000","last_became_active":"2024-09-20T22:41:40.171256+0000","last_became_peered":"2024-09-20T22:41:40.171256+0000","last_unstale":"2024-09-20T22:43:28.500016+0000","last_undegraded":"2024-09-20T22:43:28.500016+0000","last_fullsized":"2024-09-20T22:43:28.500016+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-09-20T22:43:28.499961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:43:28.499961+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:31.456215+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":127,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-20T22:43:28.119031+0000","last_change":"2024-09-20T22:41:42.178413+0000","last_active":"2024-09-20T22:43:28.119031+0000","last_peered":"2024-09-20T22:43:28.119031+0000","last_clean":"2024-09-20T22:43:28.119031+0000","last_became_active":"2024-09-20T22:41:42.178157+0000","last_became_peered":"2024-09-20T22:41:42.178157+0000","last_unstale":"2024-09-20T22:43:28.119031+0000","last_undegraded":"2024-09-20T22:43:28.119031+0000","last_fullsized":"2024-09-20T22:43:28.119031+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":150253904,"num_objects":61,"num_object_clones":17,"num_object_copies":183,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61,"num_whiteouts":4,"num_read":449,"num_read_kb":304569,"num_write":413,"num_write_kb":156892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":306720768,"data_stored":326864517,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":474,"ondisk_log_size":474,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":419,"ondisk_log_size":419,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":105,"num_read_kb":100,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":615317,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":45,"num_read_kb":69,"num_write":42,"num_write_kb":550,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":730205,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2688,"ondisk_log_size":2688,"up":63,"acting":63,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070490,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":61952,"kb_used_data":30760,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93683200,"statfs":{"total":95995035648,"available":95931596800,"internally_reserved":0,"allocated":31498240,"data_stored":34436422,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430240,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":78292,"kb_used_data":47096,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93666860,"statfs":{"total":95995035648,"available":95914864640,"internally_reserved":0,"allocated":48226304,"data_stored":50327650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691877,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":44844,"kb_used_data":17816,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93700308,"statfs":{"total":95995035648,"available":95949115392,"internally_reserved":0,"allocated":18243584,"data_stored":17494425,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986218,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":56692,"kb_used_data":25496,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93688460,"statfs":{"total":95995035648,"available":95936983040,"internally_reserved":0,"allocated":26107904,"data_stored":25661880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182448,"num_pgs":74,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":89540,"kb_used_data":58344,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93655612,"statfs":{"total":95995035648,"available":95903346688,"internally_reserved":0,"allocated":59744256,"data_stored":63442149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411381,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":57688,"kb_used_data":26500,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93687464,"statfs":{"total":95995035648,"available":95935963136,"internally_reserved":0,"allocated":27136000,"data_stored":27197283,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574906,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":75984,"kb_used_data":43344,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93669168,"statfs":{"total":95995035648,"available":95917228032,"internally_reserved":0,"allocated":44384256,"data_stored":45702050,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738431,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":73736,"kb_used_data":42540,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93671416,"statfs":{"total":95995035648,"available":95919529984,"internally_reserved":0,"allocated":43560960,"data_stored":45219016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":47579136,"data_stored":52723655,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":43053056,"data_stored":45058884,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":25587712,"data_stored":26292190,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61370368,"data_stored":66774806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":27742208,"data_stored":27948567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":24031232,"data_stored":24493640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":46616576,"data_stored":49357363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":30740480,"data_stored":34215412,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-20T22:43:33.631 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-09-20T22:43:33.631 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd pool set unique_pool_0 pg_num 26 2024-09-20T22:43:33.672 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:43:33.694 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:33.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:33.695 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:43:33.743 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:33 smithi154 ceph-mon[33264]: pgmap v100: 168 pgs: 168 active+clean; 144 MiB data, 526 MiB used, 715 GiB / 715 GiB avail; 9.2 MiB/s rd, 5.3 MiB/s wr, 19 op/s 2024-09-20T22:43:33.743 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:33 smithi154 ceph-mon[33264]: 5.17 scrub starts 2024-09-20T22:43:33.743 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:33 smithi154 ceph-mon[33264]: 5.17 scrub ok 2024-09-20T22:43:33.743 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:33 smithi154 ceph-mon[33264]: from='client.14925 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T22:43:33.743 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:33 smithi154 ceph-mon[33264]: 3.4 scrub starts 2024-09-20T22:43:33.743 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:33 smithi154 ceph-mon[33264]: 3.4 scrub ok 2024-09-20T22:43:33.743 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:33 smithi154 ceph-mon[33264]: osdmap e134: 8 total, 8 up, 8 in 2024-09-20T22:43:33.796 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:43:33.817 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:33.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:33.818 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:43:33.919 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:43:33.941 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:33.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:33.942 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:43:34.043 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:43:34.064 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.5 dump_blocked_ops 2024-09-20T22:43:34.066 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:34.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:34.067 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:43:34.129 INFO:tasks.rados.rados.0.smithi084.stdout:217: done (0 left) 2024-09-20T22:43:34.129 INFO:tasks.rados.rados.0.smithi084.stdout:218: write oid 20 current snap is 22 2024-09-20T22:43:34.129 INFO:tasks.rados.rados.0.smithi084.stdout:218: seq_num 105 ranges {564735=613734,1828021=669034,3134120=630615} 2024-09-20T22:43:34.135 INFO:tasks.rados.rados.0.smithi084.stdout:218: writing smithi08477203-20 from 564735 to 1178469 tid 1 2024-09-20T22:43:34.143 INFO:tasks.rados.rados.0.smithi084.stdout:218: writing smithi08477203-20 from 1828021 to 2497055 tid 2 2024-09-20T22:43:34.150 INFO:tasks.rados.rados.0.smithi084.stdout:218: writing smithi08477203-20 from 3134120 to 3764735 tid 3 2024-09-20T22:43:34.150 INFO:tasks.rados.rados.0.smithi084.stdout:219: write oid 32 current snap is 22 2024-09-20T22:43:34.151 INFO:tasks.rados.rados.0.smithi084.stdout:218: finishing write tid 1 to smithi08477203-20 2024-09-20T22:43:34.151 INFO:tasks.rados.rados.0.smithi084.stdout:219: seq_num 106 ranges {581542=688983,1886503=412692,2761543=670951} 2024-09-20T22:43:34.157 INFO:tasks.rados.rados.0.smithi084.stdout:219: writing smithi08477203-32 from 581542 to 1270525 tid 1 2024-09-20T22:43:34.163 INFO:tasks.rados.rados.0.smithi084.stdout:219: writing smithi08477203-32 from 1886503 to 2299195 tid 2 2024-09-20T22:43:34.167 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:43:34.170 INFO:tasks.rados.rados.0.smithi084.stdout:219: writing smithi08477203-32 from 2761543 to 3432494 tid 3 2024-09-20T22:43:34.170 INFO:tasks.rados.rados.0.smithi084.stdout:220: delete oid 17 current snap is 22 2024-09-20T22:43:34.170 INFO:tasks.rados.rados.0.smithi084.stdout:218: finishing write tid 2 to smithi08477203-20 2024-09-20T22:43:34.170 INFO:tasks.rados.rados.0.smithi084.stdout:218: finishing write tid 3 to smithi08477203-20 2024-09-20T22:43:34.170 INFO:tasks.rados.rados.0.smithi084.stdout:218: finishing write tid 4 to smithi08477203-20 2024-09-20T22:43:34.170 INFO:tasks.rados.rados.0.smithi084.stdout:218: finishing write tid 5 to smithi08477203-20 2024-09-20T22:43:34.170 INFO:tasks.rados.rados.0.smithi084.stdout:218: oid 20 updating version 0 to 39 2024-09-20T22:43:34.170 INFO:tasks.rados.rados.0.smithi084.stdout:218: oid 20 version 39 is already newer than 38 2024-09-20T22:43:34.171 INFO:tasks.rados.rados.0.smithi084.stdout:218: oid 20 version 39 is already newer than 37 2024-09-20T22:43:34.171 INFO:tasks.rados.rados.0.smithi084.stdout:218: oid 20 version 39 is already newer than 36 2024-09-20T22:43:34.171 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 20 v 39 (ObjNum 105 snap 22 seq_num 105) dirty exists 2024-09-20T22:43:34.171 INFO:tasks.rados.rados.0.smithi084.stdout:218: left oid 20 (ObjNum 105 snap 22 seq_num 105) 2024-09-20T22:43:34.171 INFO:tasks.rados.rados.0.smithi084.stdout:219: finishing write tid 1 to smithi08477203-32 2024-09-20T22:43:34.171 INFO:tasks.rados.rados.0.smithi084.stdout:219: finishing write tid 2 to smithi08477203-32 2024-09-20T22:43:34.172 INFO:tasks.rados.rados.0.smithi084.stdout:218: done (2 left) 2024-09-20T22:43:34.172 INFO:tasks.rados.rados.0.smithi084.stdout:220: done (1 left) 2024-09-20T22:43:34.172 INFO:tasks.rados.rados.0.smithi084.stdout:221: write (excl) oid 29 current snap is 22 2024-09-20T22:43:34.172 INFO:tasks.rados.rados.0.smithi084.stdout:221: seq_num 108 ranges {615156=555834,1603467=416361,2816389=693495} 2024-09-20T22:43:34.178 INFO:tasks.rados.rados.0.smithi084.stdout:221: writing smithi08477203-29 from 615156 to 1170990 tid 1 2024-09-20T22:43:34.183 INFO:tasks.rados.rados.0.smithi084.stdout:221: writing smithi08477203-29 from 1603467 to 2019828 tid 2 2024-09-20T22:43:34.193 INFO:tasks.rados.rados.0.smithi084.stdout:221: writing smithi08477203-29 from 2816389 to 3509884 tid 3 2024-09-20T22:43:34.193 INFO:tasks.rados.rados.0.smithi084.stdout:222: copy_from oid 34 from oid 7 current snap is 22 2024-09-20T22:43:34.193 INFO:tasks.rados.rados.0.smithi084.stdout:219: finishing write tid 3 to smithi08477203-32 2024-09-20T22:43:34.193 INFO:tasks.rados.rados.0.smithi084.stdout:219: finishing write tid 4 to smithi08477203-32 2024-09-20T22:43:34.193 INFO:tasks.rados.rados.0.smithi084.stdout:219: finishing write tid 5 to smithi08477203-32 2024-09-20T22:43:34.193 INFO:tasks.rados.rados.0.smithi084.stdout:219: oid 32 updating version 0 to 7 2024-09-20T22:43:34.193 INFO:tasks.rados.rados.0.smithi084.stdout:219: oid 32 version 7 is already newer than 5 2024-09-20T22:43:34.193 INFO:tasks.rados.rados.0.smithi084.stdout:219: oid 32 version 7 is already newer than 6 2024-09-20T22:43:34.193 INFO:tasks.rados.rados.0.smithi084.stdout:219: oid 32 updating version 7 to 8 2024-09-20T22:43:34.193 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 32 v 8 (ObjNum 106 snap 22 seq_num 106) dirty exists 2024-09-20T22:43:34.194 INFO:tasks.rados.rados.0.smithi084.stdout:219: left oid 32 (ObjNum 106 snap 22 seq_num 106) 2024-09-20T22:43:34.194 INFO:tasks.rados.rados.0.smithi084.stdout:221: finishing write tid 1 to smithi08477203-29 2024-09-20T22:43:34.194 INFO:tasks.rados.rados.0.smithi084.stdout:221: finishing write tid 2 to smithi08477203-29 2024-09-20T22:43:34.194 INFO:tasks.rados.rados.0.smithi084.stdout:219: done (2 left) 2024-09-20T22:43:34.194 INFO:tasks.rados.rados.0.smithi084.stdout:223: delete oid 48 current snap is 22 2024-09-20T22:43:34.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:34.195 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:34.195 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:43:34.195 INFO:tasks.rados.rados.0.smithi084.stdout:222: finishing copy_from to smithi08477203-34 2024-09-20T22:43:34.196 INFO:tasks.rados.rados.0.smithi084.stdout:222: got expected ENOENT (src dne) 2024-09-20T22:43:34.196 INFO:tasks.rados.rados.0.smithi084.stdout:222: finishing copy_from racing read to smithi08477203-34 2024-09-20T22:43:34.196 INFO:tasks.rados.rados.0.smithi084.stdout:222: done (2 left) 2024-09-20T22:43:34.196 INFO:tasks.rados.rados.0.smithi084.stdout:223: done (1 left) 2024-09-20T22:43:34.196 INFO:tasks.rados.rados.0.smithi084.stdout:224: snap_remove snap 16 2024-09-20T22:43:34.296 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:43:34.317 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:34.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:34.318 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:43:34.322 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:43:34.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:34 smithi084 ceph-mon[40024]: 3.12 scrub starts 2024-09-20T22:43:34.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:34 smithi084 ceph-mon[40024]: 3.12 scrub ok 2024-09-20T22:43:34.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:34 smithi084 ceph-mon[40024]: 6.19 scrub starts 2024-09-20T22:43:34.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:34 smithi084 ceph-mon[40024]: 6.19 scrub ok 2024-09-20T22:43:34.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:34 smithi084 ceph-mon[40024]: 6.15 scrub starts 2024-09-20T22:43:34.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:34 smithi084 ceph-mon[40024]: 6.15 scrub ok 2024-09-20T22:43:34.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:34 smithi084 ceph-mon[40024]: 4.13 scrub starts 2024-09-20T22:43:34.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:34 smithi084 ceph-mon[40024]: 4.13 scrub ok 2024-09-20T22:43:34.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:34 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:43:34.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:34 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:43:34.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:34 smithi084 ceph-mon[40024]: osdmap e135: 8 total, 8 up, 8 in 2024-09-20T22:43:34.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:34 smithi084 ceph-mon[29774]: 3.12 scrub starts 2024-09-20T22:43:34.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:34 smithi084 ceph-mon[29774]: 3.12 scrub ok 2024-09-20T22:43:34.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:34 smithi084 ceph-mon[29774]: 6.19 scrub starts 2024-09-20T22:43:34.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:34 smithi084 ceph-mon[29774]: 6.19 scrub ok 2024-09-20T22:43:34.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:34 smithi084 ceph-mon[29774]: 6.15 scrub starts 2024-09-20T22:43:34.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:34 smithi084 ceph-mon[29774]: 6.15 scrub ok 2024-09-20T22:43:34.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:34 smithi084 ceph-mon[29774]: 4.13 scrub starts 2024-09-20T22:43:34.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:34 smithi084 ceph-mon[29774]: 4.13 scrub ok 2024-09-20T22:43:34.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:34 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:43:34.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:34 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:43:34.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:34 smithi084 ceph-mon[29774]: osdmap e135: 8 total, 8 up, 8 in 2024-09-20T22:43:34.420 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:43:34.429 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:43:34.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:34.444 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:34.445 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:43:34.455 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:43:34.545 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:43:34.560 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:34 smithi154 ceph-mon[33264]: 3.12 scrub starts 2024-09-20T22:43:34.560 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:34 smithi154 ceph-mon[33264]: 3.12 scrub ok 2024-09-20T22:43:34.560 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:34 smithi154 ceph-mon[33264]: 6.19 scrub starts 2024-09-20T22:43:34.560 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:34 smithi154 ceph-mon[33264]: 6.19 scrub ok 2024-09-20T22:43:34.560 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:34 smithi154 ceph-mon[33264]: 6.15 scrub starts 2024-09-20T22:43:34.560 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:34 smithi154 ceph-mon[33264]: 6.15 scrub ok 2024-09-20T22:43:34.560 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:34 smithi154 ceph-mon[33264]: 4.13 scrub starts 2024-09-20T22:43:34.560 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:34 smithi154 ceph-mon[33264]: 4.13 scrub ok 2024-09-20T22:43:34.560 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:34 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:43:34.560 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:34 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:43:34.560 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:34 smithi154 ceph-mon[33264]: osdmap e135: 8 total, 8 up, 8 in 2024-09-20T22:43:34.568 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:34.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:34.569 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:43:34.669 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:43:34.691 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:34.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:34.692 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:43:34.792 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:43:34.813 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:34.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:34.814 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:43:34.855 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-20T22:43:34.856 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [], 2024-09-20T22:43:34.856 INFO:teuthology.orchestra.run.smithi154.stdout: "complaint_time": 30, 2024-09-20T22:43:34.856 INFO:teuthology.orchestra.run.smithi154.stdout: "num_blocked_ops": 0 2024-09-20T22:43:34.856 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-20T22:43:34.915 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:43:34.936 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:34.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:34.937 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:43:35.038 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:43:35.060 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:35.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:35.062 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:43:35.126 INFO:tasks.rados.rados.0.smithi084.stdout:224: done (1 left) 2024-09-20T22:43:35.127 INFO:tasks.rados.rados.0.smithi084.stdout:225: rollback oid 26 current snap is 22 2024-09-20T22:43:35.127 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 26 to 20 2024-09-20T22:43:35.127 INFO:tasks.rados.rados.0.smithi084.stdout:226: snap_remove snap 17 2024-09-20T22:43:35.127 INFO:teuthology.orchestra.run.smithi084.stderr:set pool 7 pg_num to 26 2024-09-20T22:43:35.163 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:43:35.183 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:35.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:35.184 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:43:35.285 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:43:35.306 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:35.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:35.307 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:43:35.364 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:35 smithi084 ceph-mon[29774]: pgmap v103: 168 pgs: 4 active+clean+snaptrim, 164 active+clean; 157 MiB data, 579 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:43:35.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:35 smithi084 ceph-mon[29774]: 6.0 scrub starts 2024-09-20T22:43:35.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:35 smithi084 ceph-mon[29774]: 6.0 scrub ok 2024-09-20T22:43:35.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:35 smithi084 ceph-mon[29774]: 2.2 scrub starts 2024-09-20T22:43:35.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:35 smithi084 ceph-mon[29774]: 2.2 scrub ok 2024-09-20T22:43:35.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:35 smithi084 ceph-mon[29774]: purged_snaps scrub starts 2024-09-20T22:43:35.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:35 smithi084 ceph-mon[29774]: purged_snaps scrub ok 2024-09-20T22:43:35.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:35 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:43:35.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:35 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:43:35.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:35 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:43:35.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:35 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/34012742' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"}]: dispatch 2024-09-20T22:43:35.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:35 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"}]: dispatch 2024-09-20T22:43:35.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:35 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"}]': finished 2024-09-20T22:43:35.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:35 smithi084 ceph-mon[29774]: osdmap e136: 8 total, 8 up, 8 in 2024-09-20T22:43:35.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:35 smithi084 ceph-mon[40024]: pgmap v103: 168 pgs: 4 active+clean+snaptrim, 164 active+clean; 157 MiB data, 579 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:43:35.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:35 smithi084 ceph-mon[40024]: 6.0 scrub starts 2024-09-20T22:43:35.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:35 smithi084 ceph-mon[40024]: 6.0 scrub ok 2024-09-20T22:43:35.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:35 smithi084 ceph-mon[40024]: 2.2 scrub starts 2024-09-20T22:43:35.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:35 smithi084 ceph-mon[40024]: 2.2 scrub ok 2024-09-20T22:43:35.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:35 smithi084 ceph-mon[40024]: purged_snaps scrub starts 2024-09-20T22:43:35.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:35 smithi084 ceph-mon[40024]: purged_snaps scrub ok 2024-09-20T22:43:35.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:35 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:43:35.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:35 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:43:35.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:35 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:43:35.367 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:35 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/34012742' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"}]: dispatch 2024-09-20T22:43:35.367 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:35 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"}]: dispatch 2024-09-20T22:43:35.367 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:35 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"}]': finished 2024-09-20T22:43:35.367 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:35 smithi084 ceph-mon[40024]: osdmap e136: 8 total, 8 up, 8 in 2024-09-20T22:43:35.408 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:43:35.430 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:35.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:35.431 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:43:35.532 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:43:35.555 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:35.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:35.556 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:43:35.656 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:43:35.678 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:35 smithi154 ceph-mon[33264]: pgmap v103: 168 pgs: 4 active+clean+snaptrim, 164 active+clean; 157 MiB data, 579 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:43:35.678 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:35 smithi154 ceph-mon[33264]: 6.0 scrub starts 2024-09-20T22:43:35.678 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:35 smithi154 ceph-mon[33264]: 6.0 scrub ok 2024-09-20T22:43:35.678 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:35 smithi154 ceph-mon[33264]: 2.2 scrub starts 2024-09-20T22:43:35.678 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:35 smithi154 ceph-mon[33264]: 2.2 scrub ok 2024-09-20T22:43:35.678 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:35 smithi154 ceph-mon[33264]: purged_snaps scrub starts 2024-09-20T22:43:35.678 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:35 smithi154 ceph-mon[33264]: purged_snaps scrub ok 2024-09-20T22:43:35.678 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:35 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:43:35.678 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:35 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:43:35.678 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:35 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:43:35.679 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:35 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/34012742' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"}]: dispatch 2024-09-20T22:43:35.679 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:35 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"}]: dispatch 2024-09-20T22:43:35.679 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:35 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"}]': finished 2024-09-20T22:43:35.679 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:35 smithi154 ceph-mon[33264]: osdmap e136: 8 total, 8 up, 8 in 2024-09-20T22:43:35.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:35.679 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:35.680 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:43:35.704 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.5 dump_historic_ops 2024-09-20T22:43:35.780 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:43:35.802 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:35.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:35.803 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:43:35.903 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:43:35.925 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:35.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:35.926 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:43:35.950 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:43:36.026 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:43:36.047 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:36.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:36.049 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:43:36.121 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is set 2024-09-20T22:43:36.136 INFO:tasks.rados.rados.0.smithi084.stdout:226: done (2 left) 2024-09-20T22:43:36.137 INFO:tasks.rados.rados.0.smithi084.stdout:227: write oid 39 current snap is 22 2024-09-20T22:43:36.137 INFO:tasks.rados.rados.0.smithi084.stdout:227: seq_num 110 ranges {648770=706332,2120431=420030,3271235=432392} 2024-09-20T22:43:36.140 INFO:tasks.rados.rados.0.smithi084.stdout:227: writing smithi08477203-39 from 648770 to 1355102 tid 1 2024-09-20T22:43:36.147 INFO:tasks.rados.rados.0.smithi084.stdout:227: writing smithi08477203-39 from 2120431 to 2540461 tid 2 2024-09-20T22:43:36.149 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:43:36.153 INFO:tasks.rados.rados.0.smithi084.stdout:227: writing smithi08477203-39 from 3271235 to 3703627 tid 3 2024-09-20T22:43:36.154 INFO:tasks.rados.rados.0.smithi084.stdout:228: write (excl) oid 21 current snap is 22 2024-09-20T22:43:36.154 INFO:tasks.rados.rados.0.smithi084.stdout:221: finishing write tid 3 to smithi08477203-29 2024-09-20T22:43:36.154 INFO:tasks.rados.rados.0.smithi084.stdout:228: seq_num 111 ranges {665577=781581,2062560=563688,3182305=643664} 2024-09-20T22:43:36.162 INFO:tasks.rados.rados.0.smithi084.stdout:228: writing smithi08477203-21 from 665577 to 1447158 tid 1 2024-09-20T22:43:36.168 INFO:tasks.rados.rados.0.smithi084.stdout:228: writing smithi08477203-21 from 2062560 to 2626248 tid 2 2024-09-20T22:43:36.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:36.172 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:36.172 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:43:36.176 INFO:tasks.rados.rados.0.smithi084.stdout:228: writing smithi08477203-21 from 3182305 to 3825969 tid 3 2024-09-20T22:43:36.176 INFO:tasks.rados.rados.0.smithi084.stdout:229: snap_create 2024-09-20T22:43:36.176 INFO:tasks.rados.rados.0.smithi084.stdout:221: finishing write tid 4 to smithi08477203-29 2024-09-20T22:43:36.176 INFO:tasks.rados.rados.0.smithi084.stdout:221: finishing write tid 5 to smithi08477203-29 2024-09-20T22:43:36.177 INFO:tasks.rados.rados.0.smithi084.stdout:221: oid 29 updating version 0 to 78 2024-09-20T22:43:36.177 INFO:tasks.rados.rados.0.smithi084.stdout:221: oid 29 updating version 78 to 80 2024-09-20T22:43:36.177 INFO:tasks.rados.rados.0.smithi084.stdout:221: oid 29 updating version 80 to 81 2024-09-20T22:43:36.177 INFO:tasks.rados.rados.0.smithi084.stdout:221: oid 29 version 81 is already newer than 79 2024-09-20T22:43:36.177 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 29 v 81 (ObjNum 108 snap 22 seq_num 108) dirty exists 2024-09-20T22:43:36.177 INFO:tasks.rados.rados.0.smithi084.stdout:221: left oid 29 (ObjNum 108 snap 22 seq_num 108) 2024-09-20T22:43:36.178 INFO:tasks.rados.rados.0.smithi084.stdout:225: finishing rollback tid 0 to smithi08477203-26 2024-09-20T22:43:36.178 INFO:tasks.rados.rados.0.smithi084.stdout:225: finishing rollback tid 1 to smithi08477203-26 2024-09-20T22:43:36.178 INFO:tasks.rados.rados.0.smithi084.stdout:225: finishing rollback tid 2 to smithi08477203-26 2024-09-20T22:43:36.178 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 26 v 94 (ObjNum 25 snap 0 seq_num 25) dirty exists 2024-09-20T22:43:36.178 INFO:tasks.rados.rados.0.smithi084.stdout:227: finishing write tid 1 to smithi08477203-39 2024-09-20T22:43:36.178 INFO:tasks.rados.rados.0.smithi084.stdout:227: finishing write tid 2 to smithi08477203-39 2024-09-20T22:43:36.179 INFO:tasks.rados.rados.0.smithi084.stdout:227: finishing write tid 3 to smithi08477203-39 2024-09-20T22:43:36.179 INFO:tasks.rados.rados.0.smithi084.stdout:227: finishing write tid 4 to smithi08477203-39 2024-09-20T22:43:36.179 INFO:tasks.rados.rados.0.smithi084.stdout:227: finishing write tid 5 to smithi08477203-39 2024-09-20T22:43:36.179 INFO:tasks.rados.rados.0.smithi084.stdout:227: oid 39 updating version 0 to 37 2024-09-20T22:43:36.179 INFO:tasks.rados.rados.0.smithi084.stdout:227: oid 39 updating version 37 to 40 2024-09-20T22:43:36.179 INFO:tasks.rados.rados.0.smithi084.stdout:227: oid 39 updating version 40 to 41 2024-09-20T22:43:36.179 INFO:tasks.rados.rados.0.smithi084.stdout:227: oid 39 updating version 41 to 42 2024-09-20T22:43:36.179 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 39 v 42 (ObjNum 110 snap 22 seq_num 110) dirty exists 2024-09-20T22:43:36.179 INFO:tasks.rados.rados.0.smithi084.stdout:227: left oid 39 (ObjNum 110 snap 22 seq_num 110) 2024-09-20T22:43:36.274 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:43:36.296 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:36.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:36.297 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:43:36.322 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:36 smithi154 ceph-mon[33264]: 3.1d scrub starts 2024-09-20T22:43:36.322 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:36 smithi154 ceph-mon[33264]: 3.1d scrub ok 2024-09-20T22:43:36.323 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:36 smithi154 ceph-mon[33264]: 3.19 scrub starts 2024-09-20T22:43:36.323 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:36 smithi154 ceph-mon[33264]: 3.19 scrub ok 2024-09-20T22:43:36.323 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:36 smithi154 ceph-mon[33264]: 3.a deep-scrub starts 2024-09-20T22:43:36.323 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:36 smithi154 ceph-mon[33264]: 3.a deep-scrub ok 2024-09-20T22:43:36.323 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:36 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-09-20T22:43:36.323 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:36 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-09-20T22:43:36.323 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:36 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/2337557203' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-20T22:43:36.323 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:36 smithi154 ceph-mon[33264]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:43:36.323 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:36 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"}]': finished 2024-09-20T22:43:36.323 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:36 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/2337557203' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T22:43:36.323 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:36 smithi154 ceph-mon[33264]: osdmap e137: 8 total, 8 up, 8 in 2024-09-20T22:43:36.398 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:43:36.402 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-20T22:43:36.419 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:36.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:36.420 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:43:36.423 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-20T22:43:36.423 INFO:teuthology.orchestra.run.smithi154.stdout: "size": 20, 2024-09-20T22:43:36.424 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 600, 2024-09-20T22:43:36.424 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [ 2024-09-20T22:43:36.424 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.424 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24334.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-09-20T22:43:36.424 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:41:39.154468+0000", 2024-09-20T22:43:36.424 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 117.266522184, 2024-09-20T22:43:36.424 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0047727469999999999, 2024-09-20T22:43:36.424 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:43:36.424 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:43:36.425 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:43:36.425 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:43:36.425 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24334", 2024-09-20T22:43:36.425 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/343304630", 2024-09-20T22:43:36.425 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 5 2024-09-20T22:43:36.425 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.425 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:43:36.425 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.425 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:43:36.426 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:39.154468+0000", 2024-09-20T22:43:36.426 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:43:36.426 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.426 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.426 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:43:36.426 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:39.154468+0000", 2024-09-20T22:43:36.426 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:43:36.426 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.426 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.426 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:43:36.426 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:39.154471+0000", 2024-09-20T22:43:36.426 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.773e-06 2024-09-20T22:43:36.427 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.427 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.427 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:43:36.427 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:39.154481+0000", 2024-09-20T22:43:36.427 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.6649999999999998e-06 2024-09-20T22:43:36.427 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.427 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.427 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:43:36.427 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:39.154517+0000", 2024-09-20T22:43:36.427 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.6146000000000003e-05 2024-09-20T22:43:36.427 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.427 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.427 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:43:36.427 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:39.154541+0000", 2024-09-20T22:43:36.428 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.3677999999999999e-05 2024-09-20T22:43:36.428 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.428 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.428 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:43:36.428 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:39.155215+0000", 2024-09-20T22:43:36.428 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00067437699999999998 2024-09-20T22:43:36.428 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.428 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.428 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:43:36.428 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:39.155352+0000", 2024-09-20T22:43:36.428 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000137365 2024-09-20T22:43:36.428 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.428 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.428 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 1,6", 2024-09-20T22:43:36.429 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:39.155511+0000", 2024-09-20T22:43:36.429 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000158608 2024-09-20T22:43:36.429 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.429 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.429 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:43:36.429 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:39.157366+0000", 2024-09-20T22:43:36.429 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001854589 2024-09-20T22:43:36.429 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.429 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.429 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:43:36.429 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:39.158647+0000", 2024-09-20T22:43:36.429 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001281344 2024-09-20T22:43:36.429 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.429 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.430 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:43:36.430 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:39.159164+0000", 2024-09-20T22:43:36.430 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00051662699999999997 2024-09-20T22:43:36.430 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.430 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.430 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:43:36.430 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:39.159203+0000", 2024-09-20T22:43:36.430 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.9209999999999999e-05 2024-09-20T22:43:36.430 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.430 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.430 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:43:36.430 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:39.159241+0000", 2024-09-20T22:43:36.430 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.8365000000000003e-05 2024-09-20T22:43:36.430 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:43:36.430 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:43:36.431 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:43:36.431 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.431 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.431 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24334.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-09-20T22:43:36.431 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:41:40.156321+0000", 2024-09-20T22:43:36.431 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 116.264669165, 2024-09-20T22:43:36.431 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.012506929, 2024-09-20T22:43:36.431 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:43:36.431 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:43:36.431 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:43:36.431 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:43:36.431 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24334", 2024-09-20T22:43:36.431 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/343304630", 2024-09-20T22:43:36.431 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 7 2024-09-20T22:43:36.431 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.432 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:43:36.432 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.432 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:43:36.432 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:40.156321+0000", 2024-09-20T22:43:36.432 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:43:36.432 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.432 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.432 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:43:36.432 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:40.156321+0000", 2024-09-20T22:43:36.432 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:43:36.432 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.432 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.432 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:43:36.432 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:40.156323+0000", 2024-09-20T22:43:36.433 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9700000000000002e-06 2024-09-20T22:43:36.433 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.433 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.433 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:43:36.433 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:40.156330+0000", 2024-09-20T22:43:36.433 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.9809999999999997e-06 2024-09-20T22:43:36.433 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.433 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.433 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:43:36.433 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:40.156348+0000", 2024-09-20T22:43:36.433 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.7938e-05 2024-09-20T22:43:36.433 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.433 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.433 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:43:36.434 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:40.156362+0000", 2024-09-20T22:43:36.434 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3641e-05 2024-09-20T22:43:36.434 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.434 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.434 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:43:36.434 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:40.156430+0000", 2024-09-20T22:43:36.434 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.7543e-05 2024-09-20T22:43:36.434 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.434 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.434 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:43:36.434 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:40.156543+0000", 2024-09-20T22:43:36.434 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00011390100000000001 2024-09-20T22:43:36.434 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.434 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.435 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 1,6", 2024-09-20T22:43:36.435 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:40.156712+0000", 2024-09-20T22:43:36.435 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00016825499999999999 2024-09-20T22:43:36.435 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.435 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.435 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:43:36.435 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:40.157640+0000", 2024-09-20T22:43:36.435 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00092863699999999995 2024-09-20T22:43:36.435 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.435 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.435 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:43:36.435 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:40.159647+0000", 2024-09-20T22:43:36.435 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0020068320000000001 2024-09-20T22:43:36.435 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.435 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.436 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:43:36.436 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:40.168687+0000", 2024-09-20T22:43:36.436 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0090397749999999999 2024-09-20T22:43:36.436 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.436 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.436 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:43:36.436 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:40.168768+0000", 2024-09-20T22:43:36.436 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.0819e-05 2024-09-20T22:43:36.436 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.436 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.436 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:43:36.436 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:40.168828+0000", 2024-09-20T22:43:36.436 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.0637000000000002e-05 2024-09-20T22:43:36.437 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:43:36.437 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:43:36.437 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:43:36.437 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.437 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.437 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.14496.0:210 5.15 5:a93a5511:::notify.2:head [watch unwatch cookie 94608265078256] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-20T22:43:36.437 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:41:45.198238+0000", 2024-09-20T22:43:36.437 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 111.222752697, 2024-09-20T22:43:36.437 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.002654741, 2024-09-20T22:43:36.437 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:43:36.437 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:43:36.437 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:43:36.437 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:43:36.437 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.14496", 2024-09-20T22:43:36.437 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3480761128", 2024-09-20T22:43:36.438 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 210 2024-09-20T22:43:36.438 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.438 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:43:36.438 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.438 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:43:36.439 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:45.198238+0000", 2024-09-20T22:43:36.439 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:43:36.439 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.439 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.439 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:43:36.439 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:45.198238+0000", 2024-09-20T22:43:36.439 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:43:36.439 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.439 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.439 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:43:36.439 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:45.198240+0000", 2024-09-20T22:43:36.439 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.6270000000000001e-06 2024-09-20T22:43:36.440 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.440 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.440 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:43:36.440 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:45.198246+0000", 2024-09-20T22:43:36.440 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.3929999999999996e-06 2024-09-20T22:43:36.440 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.440 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.440 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:43:36.440 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:45.198260+0000", 2024-09-20T22:43:36.440 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4251e-05 2024-09-20T22:43:36.440 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.440 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.440 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:43:36.440 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:45.198269+0000", 2024-09-20T22:43:36.440 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.1339999999999993e-06 2024-09-20T22:43:36.441 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.441 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.441 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:43:36.441 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:45.198322+0000", 2024-09-20T22:43:36.441 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.2271999999999997e-05 2024-09-20T22:43:36.441 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.441 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.441 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op must wait for map", 2024-09-20T22:43:36.441 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:45.198326+0000", 2024-09-20T22:43:36.441 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.2910000000000001e-06 2024-09-20T22:43:36.441 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.441 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.441 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:43:36.442 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:45.199804+0000", 2024-09-20T22:43:36.442 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001478348 2024-09-20T22:43:36.442 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.442 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.442 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:43:36.442 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:45.199918+0000", 2024-09-20T22:43:36.442 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000114003 2024-09-20T22:43:36.442 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.443 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.443 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 0,1", 2024-09-20T22:43:36.443 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:45.200053+0000", 2024-09-20T22:43:36.443 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00013520200000000001 2024-09-20T22:43:36.443 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.443 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.443 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:43:36.443 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:45.200387+0000", 2024-09-20T22:43:36.443 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00033323100000000002 2024-09-20T22:43:36.443 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.443 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.444 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:43:36.444 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:45.200689+0000", 2024-09-20T22:43:36.444 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00030267199999999998 2024-09-20T22:43:36.444 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.444 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.444 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:43:36.444 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:45.200822+0000", 2024-09-20T22:43:36.444 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000132232 2024-09-20T22:43:36.444 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.444 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.444 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:43:36.444 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:45.200853+0000", 2024-09-20T22:43:36.444 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.0919e-05 2024-09-20T22:43:36.445 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.445 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.445 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:43:36.445 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:45.200893+0000", 2024-09-20T22:43:36.445 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.0166000000000001e-05 2024-09-20T22:43:36.445 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:43:36.445 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:43:36.445 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:43:36.445 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.445 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.445 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "MOSDPGPush(1.4 93/92 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 77'128, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@77'128, size: 327680, copy_subset: [(0, 327680)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 327680, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-20T22:43:36.445 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:42:50.646393+0000", 2024-09-20T22:43:36.446 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 45.774597726000003, 2024-09-20T22:43:36.446 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.033640915, 2024-09-20T22:43:36.446 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:43:36.446 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:43:36.446 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:43:36.446 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:43:36.446 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.446 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:43:36.446 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:42:50.646393+0000", 2024-09-20T22:43:36.446 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:43:36.446 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.446 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.446 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:43:36.446 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:42:50.646393+0000", 2024-09-20T22:43:36.447 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:43:36.447 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.447 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.447 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:43:36.447 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:42:50.646393+0000", 2024-09-20T22:43:36.447 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.23e-07 2024-09-20T22:43:36.447 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.447 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.447 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:43:36.447 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:42:50.679431+0000", 2024-09-20T22:43:36.447 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.033038349000000002 2024-09-20T22:43:36.447 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.447 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.447 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:43:36.447 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:42:50.679478+0000", 2024-09-20T22:43:36.447 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.6828e-05 2024-09-20T22:43:36.448 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.448 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.448 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:43:36.448 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:42:50.679492+0000", 2024-09-20T22:43:36.448 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4062e-05 2024-09-20T22:43:36.448 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.448 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.448 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:43:36.448 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:42:50.679593+0000", 2024-09-20T22:43:36.448 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000100865 2024-09-20T22:43:36.448 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.449 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.449 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:43:36.449 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:42:50.679604+0000", 2024-09-20T22:43:36.449 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0348e-05 2024-09-20T22:43:36.449 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.449 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.449 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:43:36.449 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:42:50.680034+0000", 2024-09-20T22:43:36.449 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00043023999999999997 2024-09-20T22:43:36.449 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:43:36.449 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:43:36.449 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:43:36.449 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.449 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.450 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:321 7.b e105/94 7:df02c955:::smithi08477203-9:head v 105'17, mlcod=98'14)", 2024-09-20T22:43:36.450 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:04.861367+0000", 2024-09-20T22:43:36.450 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 31.559623835, 2024-09-20T22:43:36.450 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0067649240000000003, 2024-09-20T22:43:36.450 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:43:36.450 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:43:36.450 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:43:36.450 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:43:36.450 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.450 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:43:36.450 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:04.861367+0000", 2024-09-20T22:43:36.450 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:43:36.451 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.451 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.451 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:43:36.451 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:04.861367+0000", 2024-09-20T22:43:36.451 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:43:36.451 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.451 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.451 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:43:36.451 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:04.861369+0000", 2024-09-20T22:43:36.451 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.8470000000000001e-06 2024-09-20T22:43:36.451 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.452 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.452 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:43:36.452 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:04.865306+0000", 2024-09-20T22:43:36.452 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0039377969999999998 2024-09-20T22:43:36.452 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.452 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.452 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:43:36.452 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:04.865326+0000", 2024-09-20T22:43:36.452 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9865e-05 2024-09-20T22:43:36.452 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.453 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.453 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:43:36.453 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:04.865337+0000", 2024-09-20T22:43:36.453 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0961000000000001e-05 2024-09-20T22:43:36.453 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.453 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.453 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:43:36.453 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:04.865404+0000", 2024-09-20T22:43:36.453 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.6869e-05 2024-09-20T22:43:36.453 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.454 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.454 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:43:36.454 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:04.865422+0000", 2024-09-20T22:43:36.454 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.7824000000000001e-05 2024-09-20T22:43:36.454 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.454 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.454 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:43:36.454 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:04.868076+0000", 2024-09-20T22:43:36.454 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0026539789999999999 2024-09-20T22:43:36.454 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.454 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.454 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:43:36.454 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:04.868132+0000", 2024-09-20T22:43:36.454 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.5782000000000002e-05 2024-09-20T22:43:36.455 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:43:36.455 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:43:36.455 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:43:36.455 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.455 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.455 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:338 7.0 7:0f524219:::smithi08477203-50:head [read 0~2352947,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e107)", 2024-09-20T22:43:36.455 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:06.542991+0000", 2024-09-20T22:43:36.455 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 29.877999854999999, 2024-09-20T22:43:36.455 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0037864420000000001, 2024-09-20T22:43:36.455 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:43:36.456 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:43:36.456 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:43:36.456 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:43:36.456 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:43:36.456 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:43:36.456 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 338 2024-09-20T22:43:36.456 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.456 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:43:36.456 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.456 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:43:36.456 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:06.542991+0000", 2024-09-20T22:43:36.457 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:43:36.457 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.457 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.457 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:43:36.457 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:06.542991+0000", 2024-09-20T22:43:36.457 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:43:36.457 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.457 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.457 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:43:36.457 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:06.542992+0000", 2024-09-20T22:43:36.457 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1349999999999999e-06 2024-09-20T22:43:36.457 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.457 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.458 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:43:36.458 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:06.542997+0000", 2024-09-20T22:43:36.458 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.3789999999999997e-06 2024-09-20T22:43:36.458 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.458 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.458 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:43:36.458 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:06.543009+0000", 2024-09-20T22:43:36.458 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1309e-05 2024-09-20T22:43:36.458 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.458 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.458 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:43:36.459 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:06.543015+0000", 2024-09-20T22:43:36.459 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.4049999999999997e-06 2024-09-20T22:43:36.459 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.459 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.459 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:43:36.459 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:06.543074+0000", 2024-09-20T22:43:36.459 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.8684999999999998e-05 2024-09-20T22:43:36.459 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.459 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.459 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op must wait for map", 2024-09-20T22:43:36.460 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:06.543081+0000", 2024-09-20T22:43:36.460 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.9789999999999996e-06 2024-09-20T22:43:36.460 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.460 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.460 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:43:36.460 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:06.544383+0000", 2024-09-20T22:43:36.460 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001302503 2024-09-20T22:43:36.460 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.460 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.460 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:43:36.460 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:06.544503+0000", 2024-09-20T22:43:36.460 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000119735 2024-09-20T22:43:36.460 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.460 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.461 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:43:36.461 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:06.546777+0000", 2024-09-20T22:43:36.461 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0022743120000000001 2024-09-20T22:43:36.461 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:43:36.461 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:43:36.461 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:43:36.461 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.461 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.461 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:339 7.0 7:0f524219:::smithi08477203-50:head [read 0~2352947,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e107)", 2024-09-20T22:43:36.461 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:06.543037+0000", 2024-09-20T22:43:36.461 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 29.877953730000002, 2024-09-20T22:43:36.461 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0041257179999999996, 2024-09-20T22:43:36.461 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:43:36.462 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:43:36.462 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:43:36.462 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:43:36.462 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:43:36.462 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:43:36.462 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 339 2024-09-20T22:43:36.462 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.462 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:43:36.462 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.462 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:43:36.462 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:06.543037+0000", 2024-09-20T22:43:36.462 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:43:36.463 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.463 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.463 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:43:36.463 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:06.543037+0000", 2024-09-20T22:43:36.463 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:43:36.463 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.463 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.464 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:43:36.464 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:06.543037+0000", 2024-09-20T22:43:36.464 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.6400000000000003e-07 2024-09-20T22:43:36.464 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.464 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.464 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:43:36.464 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:06.543041+0000", 2024-09-20T22:43:36.464 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.8249999999999998e-06 2024-09-20T22:43:36.464 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.464 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.464 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:43:36.464 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:06.543047+0000", 2024-09-20T22:43:36.465 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.229e-06 2024-09-20T22:43:36.465 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.465 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.465 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:43:36.465 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:06.543053+0000", 2024-09-20T22:43:36.465 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.4149999999999998e-06 2024-09-20T22:43:36.465 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.465 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.465 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:43:36.465 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:06.543143+0000", 2024-09-20T22:43:36.465 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.9888000000000004e-05 2024-09-20T22:43:36.465 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.465 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.465 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting_for_map not empty", 2024-09-20T22:43:36.466 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:06.543146+0000", 2024-09-20T22:43:36.466 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.5999999999999998e-06 2024-09-20T22:43:36.466 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.466 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.466 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:43:36.466 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:06.546827+0000", 2024-09-20T22:43:36.466 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.003680259 2024-09-20T22:43:36.466 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.466 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.466 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:43:36.466 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:06.546917+0000", 2024-09-20T22:43:36.466 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.0836999999999998e-05 2024-09-20T22:43:36.467 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.467 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.467 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:43:36.467 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:06.547163+0000", 2024-09-20T22:43:36.467 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00024520100000000001 2024-09-20T22:43:36.467 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:43:36.467 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:43:36.467 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:43:36.467 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.467 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.467 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:340 7.0 7:0f524219:::smithi08477203-50:head [read 0~2352947,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e107)", 2024-09-20T22:43:36.467 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:06.543062+0000", 2024-09-20T22:43:36.467 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 29.877928747999999, 2024-09-20T22:43:36.467 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0048265390000000003, 2024-09-20T22:43:36.468 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:43:36.468 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:43:36.468 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:43:36.468 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:43:36.468 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:43:36.468 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:43:36.468 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 340 2024-09-20T22:43:36.468 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.468 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:43:36.468 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.468 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:43:36.468 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:06.543062+0000", 2024-09-20T22:43:36.469 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:43:36.469 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.469 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.469 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:43:36.469 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:06.543062+0000", 2024-09-20T22:43:36.469 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:43:36.469 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.469 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.469 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:43:36.469 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:06.543063+0000", 2024-09-20T22:43:36.469 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.2699999999999999e-07 2024-09-20T22:43:36.469 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.469 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.469 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:43:36.470 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:06.543065+0000", 2024-09-20T22:43:36.470 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.8789999999999999e-06 2024-09-20T22:43:36.470 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.470 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.470 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:43:36.470 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:06.543071+0000", 2024-09-20T22:43:36.470 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.3249999999999998e-06 2024-09-20T22:43:36.470 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.470 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.470 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:43:36.470 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:06.543074+0000", 2024-09-20T22:43:36.470 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.4340000000000001e-06 2024-09-20T22:43:36.470 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.470 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.471 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:43:36.471 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:06.543174+0000", 2024-09-20T22:43:36.471 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.9910999999999999e-05 2024-09-20T22:43:36.471 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.471 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.471 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting_for_map not empty", 2024-09-20T22:43:36.471 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:06.543177+0000", 2024-09-20T22:43:36.471 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.4089999999999999e-06 2024-09-20T22:43:36.471 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.471 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.471 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:43:36.471 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:06.547207+0000", 2024-09-20T22:43:36.471 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00402908 2024-09-20T22:43:36.471 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.471 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.472 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:43:36.472 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:06.547319+0000", 2024-09-20T22:43:36.472 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000112877 2024-09-20T22:43:36.472 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.472 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.472 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:43:36.472 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:06.547888+0000", 2024-09-20T22:43:36.472 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00056899700000000001 2024-09-20T22:43:36.472 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:43:36.472 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:43:36.472 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:43:36.472 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.472 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.472 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:389 7.0 7:0f524219:::smithi08477203-50:head [write 776490~714607] snapc 8=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-09-20T22:43:36.473 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:11.337342+0000", 2024-09-20T22:43:36.473 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 25.083648832000002, 2024-09-20T22:43:36.473 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0050302089999999999, 2024-09-20T22:43:36.473 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:43:36.473 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:43:36.473 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:43:36.473 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:43:36.473 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:43:36.473 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:43:36.473 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 389 2024-09-20T22:43:36.473 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.473 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:43:36.473 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.473 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:43:36.473 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:11.337342+0000", 2024-09-20T22:43:36.474 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:43:36.474 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.474 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.474 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:43:36.474 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:11.337342+0000", 2024-09-20T22:43:36.474 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:43:36.474 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.474 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.474 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:43:36.474 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:11.337344+0000", 2024-09-20T22:43:36.474 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.5289999999999998e-06 2024-09-20T22:43:36.474 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.474 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.474 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:43:36.475 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:11.338047+0000", 2024-09-20T22:43:36.475 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00070280099999999997 2024-09-20T22:43:36.475 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.475 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.475 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:43:36.475 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:11.338069+0000", 2024-09-20T22:43:36.475 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.2283e-05 2024-09-20T22:43:36.475 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.475 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.475 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:43:36.475 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:11.338096+0000", 2024-09-20T22:43:36.475 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.6254999999999998e-05 2024-09-20T22:43:36.475 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.475 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.476 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:43:36.476 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:11.338172+0000", 2024-09-20T22:43:36.476 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.6590999999999999e-05 2024-09-20T22:43:36.476 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.476 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.476 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:43:36.476 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:11.338282+0000", 2024-09-20T22:43:36.476 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000109443 2024-09-20T22:43:36.476 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.476 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.476 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 1,3", 2024-09-20T22:43:36.476 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:11.338448+0000", 2024-09-20T22:43:36.476 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000166193 2024-09-20T22:43:36.476 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.477 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.477 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:43:36.477 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:11.339797+0000", 2024-09-20T22:43:36.477 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0013489400000000001 2024-09-20T22:43:36.477 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.477 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.477 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:43:36.477 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:11.341600+0000", 2024-09-20T22:43:36.477 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0018033140000000001 2024-09-20T22:43:36.477 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.477 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.477 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:43:36.477 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:11.342297+0000", 2024-09-20T22:43:36.478 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00069641199999999999 2024-09-20T22:43:36.478 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.478 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.478 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:43:36.478 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:11.342334+0000", 2024-09-20T22:43:36.478 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.7299999999999999e-05 2024-09-20T22:43:36.478 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.478 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.478 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:43:36.478 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:11.342372+0000", 2024-09-20T22:43:36.478 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.8148000000000001e-05 2024-09-20T22:43:36.478 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:43:36.478 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:43:36.478 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:43:36.479 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.479 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.479 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:390 7.0 7:0f524219:::smithi08477203-50:head [write 2246563~579356] snapc 8=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-09-20T22:43:36.479 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:11.344787+0000", 2024-09-20T22:43:36.479 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 25.076203612, 2024-09-20T22:43:36.479 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.003104374, 2024-09-20T22:43:36.479 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:43:36.479 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:43:36.479 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:43:36.479 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:43:36.479 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:43:36.479 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:43:36.479 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 390 2024-09-20T22:43:36.479 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.480 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:43:36.480 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.480 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:43:36.480 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:11.344787+0000", 2024-09-20T22:43:36.480 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:43:36.480 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.480 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.480 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:43:36.480 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:11.344787+0000", 2024-09-20T22:43:36.480 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:43:36.480 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.480 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.480 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:43:36.481 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:11.344788+0000", 2024-09-20T22:43:36.481 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.4300000000000001e-07 2024-09-20T22:43:36.481 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.481 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.481 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:43:36.481 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:11.345307+0000", 2024-09-20T22:43:36.481 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00051905799999999998 2024-09-20T22:43:36.481 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.481 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.481 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:43:36.481 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:11.345318+0000", 2024-09-20T22:43:36.481 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1308000000000001e-05 2024-09-20T22:43:36.481 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.481 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.482 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:43:36.482 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:11.345326+0000", 2024-09-20T22:43:36.482 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.4900000000000003e-06 2024-09-20T22:43:36.482 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.482 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.482 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:43:36.482 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:11.345395+0000", 2024-09-20T22:43:36.482 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.9510000000000004e-05 2024-09-20T22:43:36.482 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.482 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.482 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:43:36.482 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:11.345454+0000", 2024-09-20T22:43:36.482 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.8412000000000002e-05 2024-09-20T22:43:36.483 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.483 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.483 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 1,3", 2024-09-20T22:43:36.483 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:11.345540+0000", 2024-09-20T22:43:36.483 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.6100000000000006e-05 2024-09-20T22:43:36.483 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.483 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.483 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:43:36.483 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:11.346643+0000", 2024-09-20T22:43:36.483 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001102853 2024-09-20T22:43:36.483 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.483 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.483 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:43:36.483 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:11.347671+0000", 2024-09-20T22:43:36.483 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001027986 2024-09-20T22:43:36.484 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.484 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.484 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:43:36.484 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:11.347796+0000", 2024-09-20T22:43:36.484 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00012548299999999999 2024-09-20T22:43:36.484 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.484 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.484 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:43:36.484 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:11.347857+0000", 2024-09-20T22:43:36.484 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.0517999999999999e-05 2024-09-20T22:43:36.484 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.484 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.484 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:43:36.484 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:11.347891+0000", 2024-09-20T22:43:36.484 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.4712999999999998e-05 2024-09-20T22:43:36.485 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:43:36.485 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:43:36.485 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:43:36.485 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.485 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.485 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:391 7.0 7:0f524219:::smithi08477203-50:head [setxattr _header (56) in=7b,truncate 3176490] snapc 8=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-09-20T22:43:36.485 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:11.345341+0000", 2024-09-20T22:43:36.485 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 25.075649155000001, 2024-09-20T22:43:36.485 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0026961810000000002, 2024-09-20T22:43:36.485 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:43:36.485 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:43:36.485 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:43:36.485 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:43:36.485 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:43:36.486 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:43:36.486 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 391 2024-09-20T22:43:36.486 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.486 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:43:36.486 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.486 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:43:36.486 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:11.345341+0000", 2024-09-20T22:43:36.486 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:43:36.486 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.486 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.486 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:43:36.486 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:11.345341+0000", 2024-09-20T22:43:36.486 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:43:36.486 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.487 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.487 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:43:36.487 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:11.345342+0000", 2024-09-20T22:43:36.487 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.6300000000000004e-07 2024-09-20T22:43:36.487 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.487 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.487 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:43:36.487 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:11.345362+0000", 2024-09-20T22:43:36.487 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9655000000000001e-05 2024-09-20T22:43:36.487 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.487 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.487 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:43:36.487 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:11.345373+0000", 2024-09-20T22:43:36.487 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0618999999999999e-05 2024-09-20T22:43:36.487 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.488 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.488 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:43:36.488 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:11.345378+0000", 2024-09-20T22:43:36.488 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.8799999999999996e-06 2024-09-20T22:43:36.488 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.488 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.488 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:43:36.488 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:11.346247+0000", 2024-09-20T22:43:36.488 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00086858400000000004 2024-09-20T22:43:36.488 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.488 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.489 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:43:36.489 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:11.346301+0000", 2024-09-20T22:43:36.489 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.3757e-05 2024-09-20T22:43:36.489 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.489 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.489 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 1,3", 2024-09-20T22:43:36.489 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:11.346384+0000", 2024-09-20T22:43:36.489 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.2916000000000004e-05 2024-09-20T22:43:36.490 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.490 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.490 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:43:36.490 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:11.346689+0000", 2024-09-20T22:43:36.490 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00030563200000000002 2024-09-20T22:43:36.490 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.490 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.490 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:43:36.490 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:11.347750+0000", 2024-09-20T22:43:36.490 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001060564 2024-09-20T22:43:36.490 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.490 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.490 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:43:36.490 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:11.347929+0000", 2024-09-20T22:43:36.491 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00017956299999999999 2024-09-20T22:43:36.491 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.491 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.491 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:43:36.491 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:11.347982+0000", 2024-09-20T22:43:36.491 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.2744000000000002e-05 2024-09-20T22:43:36.491 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.491 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.491 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:43:36.491 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:11.348038+0000", 2024-09-20T22:43:36.491 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.5504000000000003e-05 2024-09-20T22:43:36.492 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:43:36.492 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:43:36.492 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:43:36.492 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.492 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.492 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:392 7.0 7:0f524219:::smithi08477203-50:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e112)", 2024-09-20T22:43:36.492 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:11.345387+0000", 2024-09-20T22:43:36.492 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 25.075603730000001, 2024-09-20T22:43:36.492 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0027712230000000002, 2024-09-20T22:43:36.492 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:43:36.492 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:43:36.492 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:43:36.493 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:43:36.493 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:43:36.493 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:43:36.493 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 392 2024-09-20T22:43:36.493 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.493 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:43:36.493 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.493 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:43:36.493 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:11.345387+0000", 2024-09-20T22:43:36.493 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:43:36.493 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.493 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.494 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:43:36.494 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:11.345387+0000", 2024-09-20T22:43:36.494 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:43:36.494 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.494 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.494 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:43:36.494 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:11.345387+0000", 2024-09-20T22:43:36.494 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.9999999999999998e-07 2024-09-20T22:43:36.494 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.494 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.494 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:43:36.494 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:11.345390+0000", 2024-09-20T22:43:36.495 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.413e-06 2024-09-20T22:43:36.495 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.495 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.495 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:43:36.495 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:11.345396+0000", 2024-09-20T22:43:36.495 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.5139999999999998e-06 2024-09-20T22:43:36.495 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.495 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.495 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:43:36.495 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:11.345402+0000", 2024-09-20T22:43:36.495 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.4829999999999999e-06 2024-09-20T22:43:36.495 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.496 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.496 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:43:36.496 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:11.346537+0000", 2024-09-20T22:43:36.496 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0011353520000000001 2024-09-20T22:43:36.496 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.496 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.496 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for rw locks", 2024-09-20T22:43:36.496 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:11.346614+0000", 2024-09-20T22:43:36.496 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.6441999999999998e-05 2024-09-20T22:43:36.496 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.497 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.497 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:43:36.497 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:11.348062+0000", 2024-09-20T22:43:36.497 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0014479530000000001 2024-09-20T22:43:36.497 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.497 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.497 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:43:36.497 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:11.348104+0000", 2024-09-20T22:43:36.497 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.2141000000000002e-05 2024-09-20T22:43:36.497 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.497 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.498 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:43:36.498 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:11.348158+0000", 2024-09-20T22:43:36.498 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.4425000000000002e-05 2024-09-20T22:43:36.498 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:43:36.498 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:43:36.498 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:43:36.498 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.498 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.498 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:393 7.b e112/94 7:df7b2b73:::smithi08477203-10:head v 112'24, mlcod=110'21)", 2024-09-20T22:43:36.498 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:11.354336+0000", 2024-09-20T22:43:36.499 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 25.066655084000001, 2024-09-20T22:43:36.499 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0025565499999999999, 2024-09-20T22:43:36.499 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:43:36.499 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:43:36.499 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:43:36.499 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:43:36.500 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.500 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:43:36.500 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:11.354336+0000", 2024-09-20T22:43:36.500 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:43:36.500 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.501 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.501 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:43:36.501 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:11.354336+0000", 2024-09-20T22:43:36.501 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:43:36.501 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.501 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.501 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:43:36.501 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:11.354336+0000", 2024-09-20T22:43:36.501 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.2300000000000001e-07 2024-09-20T22:43:36.502 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.502 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.502 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:43:36.502 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:11.355592+0000", 2024-09-20T22:43:36.502 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0012562700000000001 2024-09-20T22:43:36.502 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.502 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.502 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:43:36.502 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:11.355602+0000", 2024-09-20T22:43:36.502 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0020000000000001e-05 2024-09-20T22:43:36.502 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.502 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.503 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:43:36.503 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:11.355607+0000", 2024-09-20T22:43:36.503 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.8280000000000002e-06 2024-09-20T22:43:36.503 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.503 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.503 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:43:36.503 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:11.355720+0000", 2024-09-20T22:43:36.503 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00011309100000000001 2024-09-20T22:43:36.503 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.503 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.503 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:43:36.503 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:11.355765+0000", 2024-09-20T22:43:36.503 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.4202000000000001e-05 2024-09-20T22:43:36.503 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.503 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.503 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:43:36.504 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:11.356862+0000", 2024-09-20T22:43:36.504 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001097687 2024-09-20T22:43:36.504 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.504 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.504 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:43:36.504 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:11.356892+0000", 2024-09-20T22:43:36.504 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.9828999999999999e-05 2024-09-20T22:43:36.504 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:43:36.504 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:43:36.504 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:43:36.504 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.504 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.504 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:394 7.b e112/94 7:df7b2b73:::smithi08477203-10:head v 112'25, mlcod=111'23)", 2024-09-20T22:43:36.504 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:11.362114+0000", 2024-09-20T22:43:36.504 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 25.058876736999999, 2024-09-20T22:43:36.504 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0025260349999999998, 2024-09-20T22:43:36.504 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:43:36.505 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:43:36.505 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:43:36.505 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:43:36.505 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.505 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:43:36.505 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:11.362114+0000", 2024-09-20T22:43:36.505 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:43:36.505 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.505 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.505 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:43:36.505 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:11.362114+0000", 2024-09-20T22:43:36.505 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:43:36.505 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.505 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.505 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:43:36.505 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:11.362115+0000", 2024-09-20T22:43:36.506 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.1999999999999998e-07 2024-09-20T22:43:36.506 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.506 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.506 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:43:36.506 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:11.363767+0000", 2024-09-20T22:43:36.506 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001651862 2024-09-20T22:43:36.506 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.506 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.506 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:43:36.506 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:11.363780+0000", 2024-09-20T22:43:36.506 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3661e-05 2024-09-20T22:43:36.506 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.506 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.506 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:43:36.506 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:11.363787+0000", 2024-09-20T22:43:36.507 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.9739999999999996e-06 2024-09-20T22:43:36.507 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.507 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.507 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:43:36.507 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:11.363864+0000", 2024-09-20T22:43:36.507 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.6499000000000001e-05 2024-09-20T22:43:36.507 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.507 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.507 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:43:36.507 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:11.363885+0000", 2024-09-20T22:43:36.507 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.1438999999999999e-05 2024-09-20T22:43:36.507 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.507 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.507 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:43:36.507 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:11.364597+0000", 2024-09-20T22:43:36.508 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00071161599999999998 2024-09-20T22:43:36.508 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.508 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.508 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:43:36.508 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:11.364640+0000", 2024-09-20T22:43:36.508 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.3164000000000002e-05 2024-09-20T22:43:36.508 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:43:36.508 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:43:36.508 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:43:36.508 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.508 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.508 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:419 7.0 7:05bb442d:::smithi08477203-24:head [write 460525~690852] snapc a=[9] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-09-20T22:43:36.508 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:14.165736+0000", 2024-09-20T22:43:36.508 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 22.255254831999999, 2024-09-20T22:43:36.508 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0072523500000000003, 2024-09-20T22:43:36.508 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:43:36.509 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:43:36.509 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:43:36.509 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:43:36.509 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:43:36.509 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:43:36.509 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 419 2024-09-20T22:43:36.509 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.509 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:43:36.509 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.509 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:43:36.509 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:14.165736+0000", 2024-09-20T22:43:36.509 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:43:36.509 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.509 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.509 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:43:36.510 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:14.165736+0000", 2024-09-20T22:43:36.510 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:43:36.510 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.510 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.510 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:43:36.510 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:14.165737+0000", 2024-09-20T22:43:36.510 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.615e-06 2024-09-20T22:43:36.510 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.510 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.510 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:43:36.510 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:14.166920+0000", 2024-09-20T22:43:36.510 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001182535 2024-09-20T22:43:36.510 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.511 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.511 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:43:36.511 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:14.166933+0000", 2024-09-20T22:43:36.511 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3263e-05 2024-09-20T22:43:36.511 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.511 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.511 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:43:36.511 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:14.166942+0000", 2024-09-20T22:43:36.511 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.3329999999999997e-06 2024-09-20T22:43:36.511 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.511 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.511 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:43:36.511 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:14.167001+0000", 2024-09-20T22:43:36.511 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.9305000000000002e-05 2024-09-20T22:43:36.511 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.512 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.512 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op must wait for map", 2024-09-20T22:43:36.512 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:14.167008+0000", 2024-09-20T22:43:36.512 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.5580000000000003e-06 2024-09-20T22:43:36.512 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.512 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.512 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:43:36.512 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:14.169081+0000", 2024-09-20T22:43:36.512 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0020726270000000001 2024-09-20T22:43:36.512 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.512 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.512 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:43:36.512 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:14.169179+0000", 2024-09-20T22:43:36.512 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.7613000000000002e-05 2024-09-20T22:43:36.512 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.513 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.513 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 1,3", 2024-09-20T22:43:36.513 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:14.169364+0000", 2024-09-20T22:43:36.513 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000185642 2024-09-20T22:43:36.513 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.513 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.513 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:43:36.513 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:14.171014+0000", 2024-09-20T22:43:36.513 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0016497129999999999 2024-09-20T22:43:36.513 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.513 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.513 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:43:36.513 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:14.172374+0000", 2024-09-20T22:43:36.513 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0013601240000000001 2024-09-20T22:43:36.513 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.514 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.514 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:43:36.514 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:14.172925+0000", 2024-09-20T22:43:36.514 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00055099499999999998 2024-09-20T22:43:36.514 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.514 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.514 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:43:36.514 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:14.172959+0000", 2024-09-20T22:43:36.514 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.4069000000000002e-05 2024-09-20T22:43:36.514 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.514 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.514 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:43:36.514 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:14.172988+0000", 2024-09-20T22:43:36.514 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.8958e-05 2024-09-20T22:43:36.514 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:43:36.515 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:43:36.515 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:43:36.515 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.515 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.515 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:420 7.0 7:05bb442d:::smithi08477203-24:head [write 1622620~730352] snapc a=[9] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-09-20T22:43:36.515 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:14.173353+0000", 2024-09-20T22:43:36.515 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 22.247637477000001, 2024-09-20T22:43:36.515 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.003497212, 2024-09-20T22:43:36.515 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:43:36.515 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:43:36.515 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:43:36.515 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:43:36.515 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:43:36.515 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:43:36.515 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 420 2024-09-20T22:43:36.516 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.516 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:43:36.516 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.516 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:43:36.516 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:14.173353+0000", 2024-09-20T22:43:36.516 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:43:36.516 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.516 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.516 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:43:36.516 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:14.173353+0000", 2024-09-20T22:43:36.516 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:43:36.516 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.516 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.516 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:43:36.517 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:14.173354+0000", 2024-09-20T22:43:36.517 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.139e-06 2024-09-20T22:43:36.517 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.517 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.517 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:43:36.517 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:14.174045+0000", 2024-09-20T22:43:36.517 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00069063900000000005 2024-09-20T22:43:36.517 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.517 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.517 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:43:36.517 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:14.174056+0000", 2024-09-20T22:43:36.517 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0954e-05 2024-09-20T22:43:36.517 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.517 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.518 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:43:36.518 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:14.174063+0000", 2024-09-20T22:43:36.518 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.0149999999999998e-06 2024-09-20T22:43:36.518 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.518 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.518 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:43:36.518 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:14.174154+0000", 2024-09-20T22:43:36.518 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.0871000000000005e-05 2024-09-20T22:43:36.518 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.518 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.518 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:43:36.518 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:14.174218+0000", 2024-09-20T22:43:36.518 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.4565000000000003e-05 2024-09-20T22:43:36.518 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.518 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.519 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 1,3", 2024-09-20T22:43:36.519 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:14.174313+0000", 2024-09-20T22:43:36.519 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.4198999999999997e-05 2024-09-20T22:43:36.519 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.519 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.519 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:43:36.519 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:14.175425+0000", 2024-09-20T22:43:36.519 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0011125950000000001 2024-09-20T22:43:36.519 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.519 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.519 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:43:36.519 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:14.176714+0000", 2024-09-20T22:43:36.519 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001288955 2024-09-20T22:43:36.519 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.519 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.520 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:43:36.520 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:14.176784+0000", 2024-09-20T22:43:36.520 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.9436000000000004e-05 2024-09-20T22:43:36.520 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.520 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.520 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:43:36.520 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:14.176822+0000", 2024-09-20T22:43:36.520 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.8373e-05 2024-09-20T22:43:36.520 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.520 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.520 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:43:36.520 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:14.176850+0000", 2024-09-20T22:43:36.520 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.8470999999999999e-05 2024-09-20T22:43:36.520 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:43:36.520 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:43:36.520 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:43:36.520 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.521 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.521 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:550 7.0 7:05bb442d:::smithi08477203-24:head [read 0~3260525,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e128)", 2024-09-20T22:43:36.521 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:27.143379+0000", 2024-09-20T22:43:36.521 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 9.2776113169999999, 2024-09-20T22:43:36.521 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0059942010000000002, 2024-09-20T22:43:36.521 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:43:36.521 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:43:36.521 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:43:36.521 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:43:36.521 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:43:36.521 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:43:36.521 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 550 2024-09-20T22:43:36.521 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.522 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:43:36.522 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.522 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:43:36.522 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.143379+0000", 2024-09-20T22:43:36.522 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:43:36.522 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.522 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.522 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:43:36.522 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.143379+0000", 2024-09-20T22:43:36.522 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:43:36.523 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.523 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.523 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:43:36.523 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.143382+0000", 2024-09-20T22:43:36.523 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.774e-06 2024-09-20T22:43:36.523 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.523 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.523 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:43:36.523 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.143393+0000", 2024-09-20T22:43:36.523 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1246e-05 2024-09-20T22:43:36.523 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.523 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.523 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:43:36.523 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.143416+0000", 2024-09-20T22:43:36.524 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.2340999999999998e-05 2024-09-20T22:43:36.524 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.524 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.524 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:43:36.524 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.143426+0000", 2024-09-20T22:43:36.524 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0010000000000001e-05 2024-09-20T22:43:36.524 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.525 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.525 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:43:36.525 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.143500+0000", 2024-09-20T22:43:36.525 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.4266999999999999e-05 2024-09-20T22:43:36.525 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.525 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.525 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op must wait for map", 2024-09-20T22:43:36.525 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.143508+0000", 2024-09-20T22:43:36.525 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.2830000000000001e-06 2024-09-20T22:43:36.525 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.525 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.525 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:43:36.525 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.145018+0000", 2024-09-20T22:43:36.525 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0015100280000000001 2024-09-20T22:43:36.525 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.525 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.526 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:43:36.526 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.145097+0000", 2024-09-20T22:43:36.526 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.8893000000000005e-05 2024-09-20T22:43:36.526 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.526 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.526 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:43:36.526 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.149374+0000", 2024-09-20T22:43:36.526 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0042763590000000004 2024-09-20T22:43:36.526 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:43:36.527 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:43:36.527 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:43:36.527 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.527 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.527 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:551 7.0 7:05bb442d:::smithi08477203-24:head [read 0~3260525,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e128)", 2024-09-20T22:43:36.527 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:27.143449+0000", 2024-09-20T22:43:36.527 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 9.2775417390000001, 2024-09-20T22:43:36.527 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0071533109999999999, 2024-09-20T22:43:36.527 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:43:36.527 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:43:36.527 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:43:36.527 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:43:36.527 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:43:36.527 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:43:36.528 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 551 2024-09-20T22:43:36.528 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.528 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:43:36.528 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.528 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:43:36.528 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.143449+0000", 2024-09-20T22:43:36.528 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:43:36.528 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.529 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.529 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:43:36.529 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.143449+0000", 2024-09-20T22:43:36.529 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:43:36.529 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.529 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.529 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:43:36.529 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.143449+0000", 2024-09-20T22:43:36.529 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.2000000000000001e-07 2024-09-20T22:43:36.529 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.529 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.529 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:43:36.529 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.143456+0000", 2024-09-20T22:43:36.530 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.8859999999999998e-06 2024-09-20T22:43:36.530 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.530 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.530 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:43:36.530 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.143462+0000", 2024-09-20T22:43:36.530 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.5940000000000001e-06 2024-09-20T22:43:36.530 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.530 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.530 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:43:36.531 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.143465+0000", 2024-09-20T22:43:36.531 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.631e-06 2024-09-20T22:43:36.531 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.531 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.531 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:43:36.531 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.143568+0000", 2024-09-20T22:43:36.531 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000103 2024-09-20T22:43:36.531 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.531 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.531 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting_for_map not empty", 2024-09-20T22:43:36.531 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.143576+0000", 2024-09-20T22:43:36.532 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.2699999999999999e-06 2024-09-20T22:43:36.532 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.532 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.532 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:43:36.532 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.149401+0000", 2024-09-20T22:43:36.532 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0058254489999999999 2024-09-20T22:43:36.532 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.532 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.532 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:43:36.532 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.149466+0000", 2024-09-20T22:43:36.532 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.4772999999999998e-05 2024-09-20T22:43:36.533 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.533 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.533 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:43:36.533 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.150602+0000", 2024-09-20T22:43:36.533 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0011363879999999999 2024-09-20T22:43:36.533 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:43:36.533 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:43:36.533 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:43:36.533 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.533 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.534 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:552 7.0 7:05bb442d:::smithi08477203-24:head [sparse-read 0~1736210] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e128)", 2024-09-20T22:43:36.534 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:27.143470+0000", 2024-09-20T22:43:36.534 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 9.2775204630000001, 2024-09-20T22:43:36.534 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0075036570000000004, 2024-09-20T22:43:36.534 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:43:36.534 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:43:36.534 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:43:36.534 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:43:36.534 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:43:36.534 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:43:36.534 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 552 2024-09-20T22:43:36.535 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.535 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:43:36.535 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.535 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:43:36.535 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.143470+0000", 2024-09-20T22:43:36.535 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:43:36.535 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.535 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.535 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:43:36.535 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.143470+0000", 2024-09-20T22:43:36.535 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:43:36.535 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.535 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.536 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:43:36.536 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.143470+0000", 2024-09-20T22:43:36.536 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.84e-07 2024-09-20T22:43:36.536 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.536 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.536 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:43:36.536 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.143472+0000", 2024-09-20T22:43:36.536 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4139999999999999e-06 2024-09-20T22:43:36.536 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.536 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.536 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:43:36.536 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.143475+0000", 2024-09-20T22:43:36.536 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.1360000000000001e-06 2024-09-20T22:43:36.536 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.536 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.536 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:43:36.536 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.143478+0000", 2024-09-20T22:43:36.537 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.9000000000000002e-06 2024-09-20T22:43:36.537 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.537 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.537 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:43:36.537 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.143601+0000", 2024-09-20T22:43:36.537 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00012325199999999999 2024-09-20T22:43:36.537 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.537 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.537 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting_for_map not empty", 2024-09-20T22:43:36.537 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.143605+0000", 2024-09-20T22:43:36.537 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.5540000000000002e-06 2024-09-20T22:43:36.537 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.537 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.537 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:43:36.538 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.150656+0000", 2024-09-20T22:43:36.538 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0070513290000000003 2024-09-20T22:43:36.538 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.538 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.538 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:43:36.538 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.150787+0000", 2024-09-20T22:43:36.538 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00013061099999999999 2024-09-20T22:43:36.538 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.538 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.538 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:43:36.538 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.150974+0000", 2024-09-20T22:43:36.538 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00018717699999999999 2024-09-20T22:43:36.539 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:43:36.539 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:43:36.539 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:43:36.539 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.539 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.539 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:554 7.a e128/94 7:5122292a:::smithi08477203-33:head v 128'32, mlcod=119'28)", 2024-09-20T22:43:36.539 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:27.341250+0000", 2024-09-20T22:43:36.539 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 9.0797401420000003, 2024-09-20T22:43:36.539 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.051661319999999997, 2024-09-20T22:43:36.539 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:43:36.539 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:43:36.539 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:43:36.540 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:43:36.540 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.540 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:43:36.540 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.341250+0000", 2024-09-20T22:43:36.540 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:43:36.540 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.540 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.540 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:43:36.540 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.341250+0000", 2024-09-20T22:43:36.540 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:43:36.540 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.540 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.540 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:43:36.540 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.341251+0000", 2024-09-20T22:43:36.540 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.7299999999999996e-07 2024-09-20T22:43:36.540 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.541 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.541 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:43:36.541 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.390395+0000", 2024-09-20T22:43:36.541 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.049143539999999999 2024-09-20T22:43:36.541 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.541 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.541 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:43:36.541 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.390413+0000", 2024-09-20T22:43:36.541 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.8709999999999999e-05 2024-09-20T22:43:36.541 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.541 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.541 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:43:36.541 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.390423+0000", 2024-09-20T22:43:36.541 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.4660000000000005e-06 2024-09-20T22:43:36.541 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.542 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.542 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:43:36.542 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.390484+0000", 2024-09-20T22:43:36.542 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.0851999999999999e-05 2024-09-20T22:43:36.542 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.542 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.542 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:43:36.542 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.390504+0000", 2024-09-20T22:43:36.542 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9976999999999999e-05 2024-09-20T22:43:36.542 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.542 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.542 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:43:36.542 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.392875+0000", 2024-09-20T22:43:36.542 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0023713710000000002 2024-09-20T22:43:36.543 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:36.543 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:36.543 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:43:36.543 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.392912+0000", 2024-09-20T22:43:36.543 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.6831e-05 2024-09-20T22:43:36.543 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:43:36.543 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:43:36.543 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:43:36.543 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:43:36.543 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:43:36.543 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-20T22:43:36.543 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:43:36.565 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:36.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:36.566 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:43:36.614 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:36 smithi084 ceph-mon[29774]: 3.1d scrub starts 2024-09-20T22:43:36.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:36 smithi084 ceph-mon[29774]: 3.1d scrub ok 2024-09-20T22:43:36.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:36 smithi084 ceph-mon[29774]: 3.19 scrub starts 2024-09-20T22:43:36.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:36 smithi084 ceph-mon[29774]: 3.19 scrub ok 2024-09-20T22:43:36.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:36 smithi084 ceph-mon[29774]: 3.a deep-scrub starts 2024-09-20T22:43:36.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:36 smithi084 ceph-mon[29774]: 3.a deep-scrub ok 2024-09-20T22:43:36.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:36 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-09-20T22:43:36.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:36 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-09-20T22:43:36.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:36 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/2337557203' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-20T22:43:36.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:36 smithi084 ceph-mon[29774]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:43:36.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:36 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"}]': finished 2024-09-20T22:43:36.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:36 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/2337557203' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T22:43:36.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:36 smithi084 ceph-mon[29774]: osdmap e137: 8 total, 8 up, 8 in 2024-09-20T22:43:36.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:36 smithi084 ceph-mon[40024]: 3.1d scrub starts 2024-09-20T22:43:36.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:36 smithi084 ceph-mon[40024]: 3.1d scrub ok 2024-09-20T22:43:36.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:36 smithi084 ceph-mon[40024]: 3.19 scrub starts 2024-09-20T22:43:36.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:36 smithi084 ceph-mon[40024]: 3.19 scrub ok 2024-09-20T22:43:36.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:36 smithi084 ceph-mon[40024]: 3.a deep-scrub starts 2024-09-20T22:43:36.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:36 smithi084 ceph-mon[40024]: 3.a deep-scrub ok 2024-09-20T22:43:36.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:36 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-09-20T22:43:36.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:36 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-09-20T22:43:36.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:36 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/2337557203' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-20T22:43:36.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:36 smithi084 ceph-mon[40024]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:43:36.617 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:36 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"}]': finished 2024-09-20T22:43:36.617 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:36 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/2337557203' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T22:43:36.617 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:36 smithi084 ceph-mon[40024]: osdmap e137: 8 total, 8 up, 8 in 2024-09-20T22:43:36.667 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:43:36.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:36.691 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:36.691 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:43:36.792 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:43:36.814 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:36.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:36.815 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:43:36.877 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:43:36.915 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:43:36.936 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:36.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:36.938 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:43:37.038 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:43:37.059 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:37.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:37.060 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:43:37.136 INFO:tasks.rados.rados.0.smithi084.stdout:221: done (4 left) 2024-09-20T22:43:37.136 INFO:tasks.rados.rados.0.smithi084.stdout:225: done (3 left) 2024-09-20T22:43:37.136 INFO:tasks.rados.rados.0.smithi084.stdout:227: done (2 left) 2024-09-20T22:43:37.136 INFO:tasks.rados.rados.0.smithi084.stdout:229: done (1 left) 2024-09-20T22:43:37.136 INFO:tasks.rados.rados.0.smithi084.stdout:230: write oid 2 current snap is 23 2024-09-20T22:43:37.136 INFO:tasks.rados.rados.0.smithi084.stdout:230: seq_num 112 ranges {682384=456830,1721042=423699,2809728=454936,3731357=151027} 2024-09-20T22:43:37.143 INFO:tasks.rados.rados.0.smithi084.stdout:230: writing smithi08477203-2 from 682384 to 1139214 tid 1 2024-09-20T22:43:37.156 INFO:tasks.rados.rados.0.smithi084.stdout:230: writing smithi08477203-2 from 1721042 to 2144741 tid 2 2024-09-20T22:43:37.156 INFO:tasks.rados.rados.0.smithi084.stdout:230: writing smithi08477203-2 from 2809728 to 3264664 tid 3 2024-09-20T22:43:37.160 INFO:tasks.rados.rados.0.smithi084.stdout:230: writing smithi08477203-2 from 3731357 to 3882384 tid 4 2024-09-20T22:43:37.160 INFO:tasks.rados.rados.0.smithi084.stdout:231: read oid 15 snap -1 2024-09-20T22:43:37.160 INFO:tasks.rados.rados.0.smithi084.stdout:231: expect (ObjNum 64 snap 5 seq_num 64) 2024-09-20T22:43:37.160 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:43:37.176 INFO:tasks.rados.rados.0.smithi084.stdout:232: snap_remove snap 22 2024-09-20T22:43:37.177 INFO:tasks.rados.rados.0.smithi084.stdout:228: finishing write tid 1 to smithi08477203-21 2024-09-20T22:43:37.182 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:37.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:37.183 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:43:37.283 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:43:37.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:37.307 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:37.308 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:43:37.370 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:37 smithi084 ceph-mon[40024]: pgmap v106: 168 pgs: 4 active+clean+snaptrim, 164 active+clean; 157 MiB data, 579 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:43:37.370 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:37 smithi084 ceph-mon[40024]: osdmap e138: 8 total, 8 up, 8 in 2024-09-20T22:43:37.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:37 smithi084 ceph-mon[29774]: pgmap v106: 168 pgs: 4 active+clean+snaptrim, 164 active+clean; 157 MiB data, 579 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:43:37.370 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:37 smithi084 ceph-mon[29774]: osdmap e138: 8 total, 8 up, 8 in 2024-09-20T22:43:37.409 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:43:37.415 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:43:37.415 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:43:37.415 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:43:37.415 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:43:37.415 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:43:37.415 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:43:37.418 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-20T22:43:37.418 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T22:43:37.430 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:37.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:37.431 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:43:37.435 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:43:37.435 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:43:37.452 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:43:37.452 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:43:37.468 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:43:37.468 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:43:37.477 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-20T22:43:37.484 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:43:37.484 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:43:37.532 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:43:37.555 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:37.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:37.556 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:43:37.566 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:37 smithi154 ceph-mon[33264]: pgmap v106: 168 pgs: 4 active+clean+snaptrim, 164 active+clean; 157 MiB data, 579 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:43:37.566 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:37 smithi154 ceph-mon[33264]: osdmap e138: 8 total, 8 up, 8 in 2024-09-20T22:43:37.657 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:43:37.679 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:37.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:37.680 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:43:37.735 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:43:37.781 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:43:37.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:37.804 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:37.804 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:43:37.905 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:43:37.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:37.927 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:37.927 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:43:38.028 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:43:38.050 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:38.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:38.051 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:43:38.121 INFO:tasks.rados.rados.0.smithi084.stdout:232: done (3 left) 2024-09-20T22:43:38.121 INFO:tasks.rados.rados.0.smithi084.stdout:233: write (excl) oid 38 current snap is 23 2024-09-20T22:43:38.121 INFO:tasks.rados.rados.0.smithi084.stdout:233: seq_num 113 ranges {699191=532079,1779524=567357,2837151=666208} 2024-09-20T22:43:38.128 INFO:tasks.rados.rados.0.smithi084.stdout:233: writing smithi08477203-38 from 699191 to 1231270 tid 1 2024-09-20T22:43:38.136 INFO:tasks.rados.rados.0.smithi084.stdout:233: writing smithi08477203-38 from 1779524 to 2346881 tid 2 2024-09-20T22:43:38.144 INFO:tasks.rados.rados.0.smithi084.stdout:233: writing smithi08477203-38 from 2837151 to 3503359 tid 3 2024-09-20T22:43:38.144 INFO:tasks.rados.rados.0.smithi084.stdout:234: write (excl) oid 20 current snap is 23 2024-09-20T22:43:38.145 INFO:tasks.rados.rados.0.smithi084.stdout:228: finishing write tid 2 to smithi08477203-21 2024-09-20T22:43:38.145 INFO:tasks.rados.rados.0.smithi084.stdout:234: seq_num 114 ranges {715998=607328,1838006=427368,2864574=477480,3863835=52163} 2024-09-20T22:43:38.152 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:43:38.152 INFO:tasks.rados.rados.0.smithi084.stdout:234: writing smithi08477203-20 from 715998 to 1323326 tid 1 2024-09-20T22:43:38.157 INFO:tasks.rados.rados.0.smithi084.stdout:234: writing smithi08477203-20 from 1838006 to 2265374 tid 2 2024-09-20T22:43:38.164 INFO:tasks.rados.rados.0.smithi084.stdout:234: writing smithi08477203-20 from 2864574 to 3342054 tid 3 2024-09-20T22:43:38.168 INFO:tasks.rados.rados.0.smithi084.stdout:234: writing smithi08477203-20 from 3863835 to 3915998 tid 4 2024-09-20T22:43:38.168 INFO:tasks.rados.rados.0.smithi084.stdout:235: read oid 13 snap -1 2024-09-20T22:43:38.168 INFO:tasks.rados.rados.0.smithi084.stdout:235: expect (ObjNum 12 snap 0 seq_num 12) 2024-09-20T22:43:38.168 INFO:tasks.rados.rados.0.smithi084.stdout:236: read oid 42 snap -1 2024-09-20T22:43:38.168 INFO:tasks.rados.rados.0.smithi084.stdout:236: expect (ObjNum 89 snap 14 seq_num 89) 2024-09-20T22:43:38.168 INFO:tasks.rados.rados.0.smithi084.stdout:228: finishing write tid 3 to smithi08477203-21 2024-09-20T22:43:38.168 INFO:tasks.rados.rados.0.smithi084.stdout:228: finishing write tid 4 to smithi08477203-21 2024-09-20T22:43:38.168 INFO:tasks.rados.rados.0.smithi084.stdout:228: finishing write tid 5 to smithi08477203-21 2024-09-20T22:43:38.168 INFO:tasks.rados.rados.0.smithi084.stdout:228: oid 21 updating version 0 to 74 2024-09-20T22:43:38.169 INFO:tasks.rados.rados.0.smithi084.stdout:228: oid 21 updating version 74 to 77 2024-09-20T22:43:38.169 INFO:tasks.rados.rados.0.smithi084.stdout:228: oid 21 version 77 is already newer than 76 2024-09-20T22:43:38.169 INFO:tasks.rados.rados.0.smithi084.stdout:228: oid 21 version 77 is already newer than 75 2024-09-20T22:43:38.169 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 21 v 77 (ObjNum 111 snap 22 seq_num 111) dirty exists 2024-09-20T22:43:38.169 INFO:tasks.rados.rados.0.smithi084.stdout:228: left oid 21 (ObjNum 111 snap 22 seq_num 111) 2024-09-20T22:43:38.169 INFO:tasks.rados.rados.0.smithi084.stdout:230: finishing write tid 1 to smithi08477203-2 2024-09-20T22:43:38.169 INFO:tasks.rados.rados.0.smithi084.stdout:230: finishing write tid 2 to smithi08477203-2 2024-09-20T22:43:38.169 INFO:tasks.rados.rados.0.smithi084.stdout:230: finishing write tid 3 to smithi08477203-2 2024-09-20T22:43:38.169 INFO:tasks.rados.rados.0.smithi084.stdout:230: finishing write tid 4 to smithi08477203-2 2024-09-20T22:43:38.169 INFO:tasks.rados.rados.0.smithi084.stdout:230: finishing write tid 5 to smithi08477203-2 2024-09-20T22:43:38.169 INFO:tasks.rados.rados.0.smithi084.stdout:230: finishing write tid 6 to smithi08477203-2 2024-09-20T22:43:38.169 INFO:tasks.rados.rados.0.smithi084.stdout:230: oid 2 updating version 0 to 75 2024-09-20T22:43:38.169 INFO:tasks.rados.rados.0.smithi084.stdout:230: oid 2 updating version 75 to 77 2024-09-20T22:43:38.169 INFO:tasks.rados.rados.0.smithi084.stdout:230: oid 2 version 77 is already newer than 76 2024-09-20T22:43:38.169 INFO:tasks.rados.rados.0.smithi084.stdout:230: oid 2 updating version 77 to 78 2024-09-20T22:43:38.170 INFO:tasks.rados.rados.0.smithi084.stdout:230: oid 2 version 78 is already newer than 74 2024-09-20T22:43:38.170 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 2 v 78 (ObjNum 112 snap 23 seq_num 112) dirty exists 2024-09-20T22:43:38.170 INFO:tasks.rados.rados.0.smithi084.stdout:230: left oid 2 (ObjNum 112 snap 23 seq_num 112) 2024-09-20T22:43:38.170 INFO:tasks.rados.rados.0.smithi084.stdout:231: expect (ObjNum 64 snap 5 seq_num 64) 2024-09-20T22:43:38.176 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-20T22:43:38.177 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:43:38.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:38.180 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:38.180 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:43:38.280 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:43:38.298 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:43:38 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: ::ffff:172.21.15.154 - - [20/Sep/2024:22:43:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-20T22:43:38.302 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:38.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:38.303 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:43:38.309 INFO:tasks.rados.rados.0.smithi084.stdout:233: finishing write tid 1 to smithi08477203-38 2024-09-20T22:43:38.309 INFO:tasks.rados.rados.0.smithi084.stdout:233: finishing write tid 2 to smithi08477203-38 2024-09-20T22:43:38.309 INFO:tasks.rados.rados.0.smithi084.stdout:233: finishing write tid 3 to smithi08477203-38 2024-09-20T22:43:38.309 INFO:tasks.rados.rados.0.smithi084.stdout:233: finishing write tid 4 to smithi08477203-38 2024-09-20T22:43:38.309 INFO:tasks.rados.rados.0.smithi084.stdout:228: done (6 left) 2024-09-20T22:43:38.309 INFO:tasks.rados.rados.0.smithi084.stdout:230: done (5 left) 2024-09-20T22:43:38.309 INFO:tasks.rados.rados.0.smithi084.stdout:231: done (4 left) 2024-09-20T22:43:38.309 INFO:tasks.rados.rados.0.smithi084.stdout:237: snap_remove snap 15 2024-09-20T22:43:38.404 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:43:38.425 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:38.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:38.426 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:43:38.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:38 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/3760783558' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:43:38.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:38 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-09-20T22:43:38.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:38 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-09-20T22:43:38.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:38 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:43:38.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:38 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"}]': finished 2024-09-20T22:43:38.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:38 smithi154 ceph-mon[33264]: osdmap e139: 8 total, 8 up, 8 in 2024-09-20T22:43:38.527 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:43:38.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:38.550 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:38.551 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:43:38.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:38 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/3760783558' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:43:38.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:38 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-09-20T22:43:38.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:38 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-09-20T22:43:38.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:38 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:43:38.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:38 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"}]': finished 2024-09-20T22:43:38.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:38 smithi084 ceph-mon[40024]: osdmap e139: 8 total, 8 up, 8 in 2024-09-20T22:43:38.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:38 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/3760783558' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:43:38.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:38 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-09-20T22:43:38.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:38 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-09-20T22:43:38.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:38 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:43:38.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:38 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"}]': finished 2024-09-20T22:43:38.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:38 smithi084 ceph-mon[29774]: osdmap e139: 8 total, 8 up, 8 in 2024-09-20T22:43:38.651 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:43:38.673 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:38.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:38.674 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:43:38.763 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.6 dump_blocked_ops 2024-09-20T22:43:38.775 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:43:38.794 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-20T22:43:38.826 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:38.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:38.827 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:43:38.928 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:43:38.949 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:38.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:38.950 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:43:39.012 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:43:39.040 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:43:39.052 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:43:39.075 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:39.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:39.076 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:43:39.134 INFO:tasks.rados.rados.0.smithi084.stdout:237: done (4 left) 2024-09-20T22:43:39.135 INFO:tasks.rados.rados.0.smithi084.stdout:238: delete oid 18 current snap is 23 2024-09-20T22:43:39.135 INFO:tasks.rados.rados.0.smithi084.stdout:233: finishing write tid 5 to smithi08477203-38 2024-09-20T22:43:39.135 INFO:tasks.rados.rados.0.smithi084.stdout:233: oid 38 updating version 0 to 103 2024-09-20T22:43:39.135 INFO:tasks.rados.rados.0.smithi084.stdout:233: oid 38 version 103 is already newer than 102 2024-09-20T22:43:39.135 INFO:tasks.rados.rados.0.smithi084.stdout:233: oid 38 version 103 is already newer than 101 2024-09-20T22:43:39.135 INFO:tasks.rados.rados.0.smithi084.stdout:233: oid 38 version 103 is already newer than 100 2024-09-20T22:43:39.135 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 38 v 103 (ObjNum 113 snap 23 seq_num 113) dirty exists 2024-09-20T22:43:39.135 INFO:tasks.rados.rados.0.smithi084.stdout:233: left oid 38 (ObjNum 113 snap 23 seq_num 113) 2024-09-20T22:43:39.135 INFO:tasks.rados.rados.0.smithi084.stdout:234: finishing write tid 1 to smithi08477203-20 2024-09-20T22:43:39.135 INFO:tasks.rados.rados.0.smithi084.stdout:234: finishing write tid 2 to smithi08477203-20 2024-09-20T22:43:39.135 INFO:tasks.rados.rados.0.smithi084.stdout:234: finishing write tid 3 to smithi08477203-20 2024-09-20T22:43:39.135 INFO:tasks.rados.rados.0.smithi084.stdout:234: finishing write tid 4 to smithi08477203-20 2024-09-20T22:43:39.135 INFO:tasks.rados.rados.0.smithi084.stdout:234: finishing write tid 5 to smithi08477203-20 2024-09-20T22:43:39.135 INFO:tasks.rados.rados.0.smithi084.stdout:234: finishing write tid 6 to smithi08477203-20 2024-09-20T22:43:39.136 INFO:tasks.rados.rados.0.smithi084.stdout:234: oid 20 updating version 0 to 44 2024-09-20T22:43:39.136 INFO:tasks.rados.rados.0.smithi084.stdout:234: oid 20 updating version 44 to 46 2024-09-20T22:43:39.136 INFO:tasks.rados.rados.0.smithi084.stdout:234: oid 20 version 46 is already newer than 45 2024-09-20T22:43:39.136 INFO:tasks.rados.rados.0.smithi084.stdout:234: oid 20 updating version 46 to 48 2024-09-20T22:43:39.136 INFO:tasks.rados.rados.0.smithi084.stdout:234: oid 20 version 48 is already newer than 47 2024-09-20T22:43:39.136 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 20 v 48 (ObjNum 114 snap 23 seq_num 114) dirty exists 2024-09-20T22:43:39.136 INFO:tasks.rados.rados.0.smithi084.stdout:234: left oid 20 (ObjNum 114 snap 23 seq_num 114) 2024-09-20T22:43:39.136 INFO:tasks.rados.rados.0.smithi084.stdout:235: expect (ObjNum 12 snap 0 seq_num 12) 2024-09-20T22:43:39.177 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:43:39.203 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:39.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:39.204 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:43:39.218 INFO:tasks.rados.rados.0.smithi084.stdout:236: expect (ObjNum 89 snap 14 seq_num 89) 2024-09-20T22:43:39.254 INFO:journalctl@ceph.iscsi.iscsi.a.smithi154.stdout:Sep 20 22:43:38 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-iscsi-iscsi-a[59730]: debug there is no tcmu-runner data available 2024-09-20T22:43:39.305 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:43:39.329 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:39.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:39.330 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:43:39.386 INFO:tasks.rados.rados.0.smithi084.stdout:233: done (4 left) 2024-09-20T22:43:39.386 INFO:tasks.rados.rados.0.smithi084.stdout:234: done (3 left) 2024-09-20T22:43:39.386 INFO:tasks.rados.rados.0.smithi084.stdout:235: done (2 left) 2024-09-20T22:43:39.386 INFO:tasks.rados.rados.0.smithi084.stdout:236: done (1 left) 2024-09-20T22:43:39.387 INFO:tasks.rados.rados.0.smithi084.stdout:238: done (0 left) 2024-09-20T22:43:39.387 INFO:tasks.rados.rados.0.smithi084.stdout:239: snap_remove snap 19 2024-09-20T22:43:39.431 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:43:39.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:39.454 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:39.455 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:43:39.455 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:39 smithi084 ceph-mon[40024]: pgmap v109: 178 pgs: 10 unknown, 4 active+clean+snaptrim, 164 active+clean; 157 MiB data, 579 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:43:39.455 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:39 smithi084 ceph-mon[40024]: osdmap e140: 8 total, 8 up, 8 in 2024-09-20T22:43:39.456 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:39 smithi084 ceph-mon[29774]: pgmap v109: 178 pgs: 10 unknown, 4 active+clean+snaptrim, 164 active+clean; 157 MiB data, 579 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:43:39.456 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:39 smithi084 ceph-mon[29774]: osdmap e140: 8 total, 8 up, 8 in 2024-09-20T22:43:39.532 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-20T22:43:39.532 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:43:39.540 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:39 smithi154 ceph-mon[33264]: pgmap v109: 178 pgs: 10 unknown, 4 active+clean+snaptrim, 164 active+clean; 157 MiB data, 579 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:43:39.541 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:39 smithi154 ceph-mon[33264]: osdmap e140: 8 total, 8 up, 8 in 2024-09-20T22:43:39.556 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:43:39.577 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:39.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:39.578 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:43:39.679 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:43:39.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:39.701 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:39.701 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:43:39.802 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:43:39.823 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:39.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:39.824 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:43:39.925 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:43:39.945 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:39.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:39.946 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:43:40.047 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:43:40.068 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:40.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:40.070 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:43:40.155 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.6 dump_historic_ops 2024-09-20T22:43:40.171 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:43:40.180 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-20T22:43:40.220 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:40.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:40.221 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:43:40.316 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is set 2024-09-20T22:43:40.316 INFO:tasks.rados.rados.0.smithi084.stdout:239: done (0 left) 2024-09-20T22:43:40.316 INFO:tasks.rados.rados.0.smithi084.stdout:240: write oid 27 current snap is 23 2024-09-20T22:43:40.316 INFO:tasks.rados.rados.0.smithi084.stdout:240: seq_num 116 ranges {749612=474179,1954970=431037,2919420=616377} 2024-09-20T22:43:40.322 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:43:40.325 INFO:tasks.rados.rados.0.smithi084.stdout:240: writing smithi08477203-27 from 749612 to 1223791 tid 1 2024-09-20T22:43:40.332 INFO:tasks.rados.rados.0.smithi084.stdout:240: writing smithi08477203-27 from 1954970 to 2386007 tid 2 2024-09-20T22:43:40.340 INFO:tasks.rados.rados.0.smithi084.stdout:240: writing smithi08477203-27 from 2919420 to 3535797 tid 3 2024-09-20T22:43:40.340 INFO:tasks.rados.rados.0.smithi084.stdout:241: snap_remove snap 20 2024-09-20T22:43:40.347 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:40.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:40.349 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:43:40.413 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:43:40.449 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:43:40.470 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:40.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:40.472 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:43:40.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:40 smithi154 ceph-mon[33264]: purged_snaps scrub starts 2024-09-20T22:43:40.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:40 smithi154 ceph-mon[33264]: purged_snaps scrub ok 2024-09-20T22:43:40.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:40 smithi154 ceph-mon[33264]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:43:40.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:40 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/1169168622' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:43:40.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:40 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:43:40.573 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:43:40.595 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:40.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:40.596 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:43:40.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:40 smithi084 ceph-mon[40024]: purged_snaps scrub starts 2024-09-20T22:43:40.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:40 smithi084 ceph-mon[40024]: purged_snaps scrub ok 2024-09-20T22:43:40.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:40 smithi084 ceph-mon[40024]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:43:40.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:40 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/1169168622' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:43:40.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:40 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:43:40.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:40 smithi084 ceph-mon[29774]: purged_snaps scrub starts 2024-09-20T22:43:40.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:40 smithi084 ceph-mon[29774]: purged_snaps scrub ok 2024-09-20T22:43:40.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:40 smithi084 ceph-mon[29774]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:43:40.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:40 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/1169168622' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:43:40.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:40 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:43:40.696 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:43:40.717 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:40.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:40.718 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:43:40.819 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:43:40.840 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:40.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:40.841 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:43:40.906 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-20T22:43:40.906 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:43:40.942 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:43:40.964 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:40.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:40.965 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:43:41.040 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-20T22:43:41.040 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-20T22:43:41.040 INFO:tasks.thrashosds.thrasher:fixing pg num pool .mgr 2024-09-20T22:43:41.041 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph pg dump --format=json 2024-09-20T22:43:41.065 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:43:41.105 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:41.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:41.106 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:43:41.184 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:43:41.206 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:43:41.229 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:41.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:41.231 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:43:41.304 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:41 smithi084 ceph-mon[29774]: pgmap v112: 178 pgs: 10 peering, 2 active+clean+snaptrim, 166 active+clean; 177 MiB data, 611 MiB used, 715 GiB / 715 GiB avail; 5.3 MiB/s rd, 3.3 MiB/s wr, 20 op/s 2024-09-20T22:43:41.304 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:41 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T22:43:41.321 INFO:tasks.rados.rados.0.smithi084.stdout:241: done (1 left) 2024-09-20T22:43:41.321 INFO:tasks.rados.rados.0.smithi084.stdout:242: delete oid 38 current snap is 23 2024-09-20T22:43:41.321 INFO:tasks.rados.rados.0.smithi084.stdout:240: finishing write tid 1 to smithi08477203-27 2024-09-20T22:43:41.322 INFO:tasks.rados.rados.0.smithi084.stdout:240: finishing write tid 2 to smithi08477203-27 2024-09-20T22:43:41.322 INFO:tasks.rados.rados.0.smithi084.stdout:240: finishing write tid 3 to smithi08477203-27 2024-09-20T22:43:41.322 INFO:tasks.rados.rados.0.smithi084.stdout:240: finishing write tid 4 to smithi08477203-27 2024-09-20T22:43:41.322 INFO:tasks.rados.rados.0.smithi084.stdout:240: finishing write tid 5 to smithi08477203-27 2024-09-20T22:43:41.322 INFO:tasks.rados.rados.0.smithi084.stdout:240: oid 27 updating version 0 to 48 2024-09-20T22:43:41.322 INFO:tasks.rados.rados.0.smithi084.stdout:240: oid 27 version 48 is already newer than 47 2024-09-20T22:43:41.322 INFO:tasks.rados.rados.0.smithi084.stdout:240: oid 27 version 48 is already newer than 46 2024-09-20T22:43:41.322 INFO:tasks.rados.rados.0.smithi084.stdout:240: oid 27 version 48 is already newer than 45 2024-09-20T22:43:41.322 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 27 v 48 (ObjNum 116 snap 23 seq_num 116) dirty exists 2024-09-20T22:43:41.322 INFO:tasks.rados.rados.0.smithi084.stdout:240: left oid 27 (ObjNum 116 snap 23 seq_num 116) 2024-09-20T22:43:41.325 INFO:tasks.rados.rados.0.smithi084.stdout:240: done (1 left) 2024-09-20T22:43:41.325 INFO:tasks.rados.rados.0.smithi084.stdout:242: done (0 left) 2024-09-20T22:43:41.325 INFO:tasks.rados.rados.0.smithi084.stdout:243: delete oid 20 current snap is 23 2024-09-20T22:43:41.327 INFO:tasks.rados.rados.0.smithi084.stdout:243: done (0 left) 2024-09-20T22:43:41.327 INFO:tasks.rados.rados.0.smithi084.stdout:244: copy_from oid 48 from oid 47 current snap is 23 2024-09-20T22:43:41.327 INFO:tasks.rados.rados.0.smithi084.stdout:245: copy_from oid 34 from oid 25 current snap is 23 2024-09-20T22:43:41.327 INFO:tasks.rados.rados.0.smithi084.stdout:246: copy_from oid 19 from oid 10 current snap is 23 2024-09-20T22:43:41.327 INFO:tasks.rados.rados.0.smithi084.stdout:247: read oid 9 snap -1 2024-09-20T22:43:41.327 INFO:tasks.rados.rados.0.smithi084.stdout:247: expect (ObjNum 58 snap 4 seq_num 58) 2024-09-20T22:43:41.327 INFO:tasks.rados.rados.0.smithi084.stdout:248: read oid 46 snap -1 2024-09-20T22:43:41.327 INFO:tasks.rados.rados.0.smithi084.stdout:248: expect deleted 2024-09-20T22:43:41.327 INFO:tasks.rados.rados.0.smithi084.stdout:249: snap_remove snap 11 2024-09-20T22:43:41.332 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:43:41.355 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:41.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:41.357 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:43:41.457 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:43:41.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:41.480 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:41.480 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:43:41.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:41 smithi154 ceph-mon[33264]: pgmap v112: 178 pgs: 10 peering, 2 active+clean+snaptrim, 166 active+clean; 177 MiB data, 611 MiB used, 715 GiB / 715 GiB avail; 5.3 MiB/s rd, 3.3 MiB/s wr, 20 op/s 2024-09-20T22:43:41.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:41 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T22:43:41.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:41 smithi154 ceph-mon[33264]: osdmap e141: 8 total, 8 up, 8 in 2024-09-20T22:43:41.572 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-20T22:43:41.581 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:43:41.595 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:41 smithi084 ceph-mon[40024]: pgmap v112: 178 pgs: 10 peering, 2 active+clean+snaptrim, 166 active+clean; 177 MiB data, 611 MiB used, 715 GiB / 715 GiB avail; 5.3 MiB/s rd, 3.3 MiB/s wr, 20 op/s 2024-09-20T22:43:41.596 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:41 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T22:43:41.596 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:41 smithi084 ceph-mon[40024]: osdmap e141: 8 total, 8 up, 8 in 2024-09-20T22:43:41.596 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:41 smithi084 ceph-mon[29774]: osdmap e141: 8 total, 8 up, 8 in 2024-09-20T22:43:41.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:41.606 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:41.606 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:43:41.705 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:43:41.705 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:43:41.707 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:43:41.712 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:43:41.713 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:43:41.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:41.729 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:41.730 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:43:41.730 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:43:41.730 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:43:41.741 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:43:41.747 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-20T22:43:41.747 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T22:43:41.765 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:43:41.765 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:43:41.784 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:43:41.784 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:43:41.801 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:43:41.801 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:43:41.818 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:43:41.818 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:43:41.831 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:43:41.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:41.855 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:41.855 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:43:41.861 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:43:41.956 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:43:41.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:41.978 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:41.978 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:43:42.079 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:43:42.103 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:42.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:42.104 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:43:42.205 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:43:42.226 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:42.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:42.227 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:43:42.325 INFO:tasks.rados.rados.0.smithi084.stdout:249: done (5 left) 2024-09-20T22:43:42.326 INFO:tasks.rados.rados.0.smithi084.stdout:250: rollback oid 26 current snap is 23 2024-09-20T22:43:42.326 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 26 to 21 2024-09-20T22:43:42.326 INFO:tasks.rados.rados.0.smithi084.stdout:251: read oid 36 snap 21 2024-09-20T22:43:42.326 INFO:tasks.rados.rados.0.smithi084.stdout:251: expect (ObjNum 35 snap 0 seq_num 35) 2024-09-20T22:43:42.326 INFO:tasks.rados.rados.0.smithi084.stdout:252: copy_from oid 30 from oid 29 current snap is 23 2024-09-20T22:43:42.326 INFO:tasks.rados.rados.0.smithi084.stdout:253: delete oid 33 current snap is 23 2024-09-20T22:43:42.326 INFO:tasks.rados.rados.0.smithi084.stdout:245: finishing copy_from racing read to smithi08477203-34 2024-09-20T22:43:42.326 INFO:tasks.rados.rados.0.smithi084.stdout:247: expect (ObjNum 58 snap 4 seq_num 58) 2024-09-20T22:43:42.327 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:43:42.349 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-20T22:43:42.349 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [], 2024-09-20T22:43:42.349 INFO:teuthology.orchestra.run.smithi154.stdout: "num_ops": 0 2024-09-20T22:43:42.349 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-20T22:43:42.359 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:42.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:42.360 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:43:42.460 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:43:42.481 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:42.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:42.482 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:43:42.494 INFO:tasks.rados.rados.0.smithi084.stdout:244: finishing copy_from racing read to smithi08477203-48 2024-09-20T22:43:42.494 INFO:tasks.rados.rados.0.smithi084.stdout:246: finishing copy_from racing read to smithi08477203-19 2024-09-20T22:43:42.494 INFO:tasks.rados.rados.0.smithi084.stdout:244: finishing copy_from to smithi08477203-48 2024-09-20T22:43:42.494 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 48 v 6 (ObjNum 42 snap 0 seq_num 42) dirty exists 2024-09-20T22:43:42.494 INFO:tasks.rados.rados.0.smithi084.stdout:245: finishing copy_from to smithi08477203-34 2024-09-20T22:43:42.494 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 34 v 25 (ObjNum 50 snap 0 seq_num 50) dirty exists 2024-09-20T22:43:42.495 INFO:tasks.rados.rados.0.smithi084.stdout:246: finishing copy_from to smithi08477203-19 2024-09-20T22:43:42.495 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 19 v 26 (ObjNum 71 snap 7 seq_num 71) dirty exists 2024-09-20T22:43:42.495 INFO:tasks.rados.rados.0.smithi084.stdout:244: done (8 left) 2024-09-20T22:43:42.495 INFO:tasks.rados.rados.0.smithi084.stdout:245: done (7 left) 2024-09-20T22:43:42.495 INFO:tasks.rados.rados.0.smithi084.stdout:246: done (6 left) 2024-09-20T22:43:42.495 INFO:tasks.rados.rados.0.smithi084.stdout:247: done (5 left) 2024-09-20T22:43:42.495 INFO:tasks.rados.rados.0.smithi084.stdout:248: done (4 left) 2024-09-20T22:43:42.495 INFO:tasks.rados.rados.0.smithi084.stdout:253: done (3 left) 2024-09-20T22:43:42.495 INFO:tasks.rados.rados.0.smithi084.stdout:254: write oid 31 current snap is 23 2024-09-20T22:43:42.495 INFO:tasks.rados.rados.0.smithi084.stdout:250: finishing rollback tid 0 to smithi08477203-26 2024-09-20T22:43:42.495 INFO:tasks.rados.rados.0.smithi084.stdout:250: finishing rollback tid 1 to smithi08477203-26 2024-09-20T22:43:42.495 INFO:tasks.rados.rados.0.smithi084.stdout:250: finishing rollback tid 2 to smithi08477203-26 2024-09-20T22:43:42.495 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 26 v 107 (ObjNum 25 snap 0 seq_num 25) dirty exists 2024-09-20T22:43:42.495 INFO:tasks.rados.rados.0.smithi084.stdout:251: expect (ObjNum 35 snap 0 seq_num 35) 2024-09-20T22:43:42.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:42 smithi154 ceph-mon[33264]: osdmap e142: 8 total, 8 up, 8 in 2024-09-20T22:43:42.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:42 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/1954793160' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:43:42.579 INFO:tasks.rados.rados.0.smithi084.stdout:252: finishing copy_from racing read to smithi08477203-30 2024-09-20T22:43:42.579 INFO:tasks.rados.rados.0.smithi084.stdout:252: finishing copy_from to smithi08477203-30 2024-09-20T22:43:42.579 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 30 v 81 (ObjNum 108 snap 22 seq_num 108) dirty exists 2024-09-20T22:43:42.579 INFO:tasks.rados.rados.0.smithi084.stdout:254: seq_num 120 ranges {416840=775175,1672545=344295} 2024-09-20T22:43:42.583 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:43:42.586 INFO:tasks.rados.rados.0.smithi084.stdout:254: writing smithi08477203-31 from 416840 to 1192015 tid 1 2024-09-20T22:43:42.591 INFO:tasks.rados.rados.0.smithi084.stdout:254: writing smithi08477203-31 from 1672545 to 2016840 tid 2 2024-09-20T22:43:42.591 INFO:tasks.rados.rados.0.smithi084.stdout:250: done (3 left) 2024-09-20T22:43:42.591 INFO:tasks.rados.rados.0.smithi084.stdout:251: done (2 left) 2024-09-20T22:43:42.591 INFO:tasks.rados.rados.0.smithi084.stdout:252: done (1 left) 2024-09-20T22:43:42.591 INFO:tasks.rados.rados.0.smithi084.stdout:255: write oid 4 current snap is 23 2024-09-20T22:43:42.591 INFO:tasks.rados.rados.0.smithi084.stdout:254: finishing write tid 1 to smithi08477203-31 2024-09-20T22:43:42.591 INFO:tasks.rados.rados.0.smithi084.stdout:255: seq_num 121 ranges {433647=450424,1331027=582033} 2024-09-20T22:43:42.595 INFO:tasks.rados.rados.0.smithi084.stdout:255: writing smithi08477203-4 from 433647 to 884071 tid 1 2024-09-20T22:43:42.599 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:42 smithi084 ceph-mon[29774]: osdmap e142: 8 total, 8 up, 8 in 2024-09-20T22:43:42.599 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:42 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/1954793160' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:43:42.599 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:42 smithi084 ceph-mon[40024]: osdmap e142: 8 total, 8 up, 8 in 2024-09-20T22:43:42.599 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:42 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/1954793160' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:43:42.603 INFO:tasks.rados.rados.0.smithi084.stdout:255: writing smithi08477203-4 from 1331027 to 1913060 tid 2 2024-09-20T22:43:42.603 INFO:tasks.rados.rados.0.smithi084.stdout:256: read oid 24 snap -1 2024-09-20T22:43:42.603 INFO:tasks.rados.rados.0.smithi084.stdout:256: expect (ObjNum 104 snap 21 seq_num 104) 2024-09-20T22:43:42.603 INFO:tasks.rados.rados.0.smithi084.stdout:254: finishing write tid 2 to smithi08477203-31 2024-09-20T22:43:42.603 INFO:tasks.rados.rados.0.smithi084.stdout:254: finishing write tid 3 to smithi08477203-31 2024-09-20T22:43:42.603 INFO:tasks.rados.rados.0.smithi084.stdout:254: finishing write tid 4 to smithi08477203-31 2024-09-20T22:43:42.603 INFO:tasks.rados.rados.0.smithi084.stdout:254: oid 31 updating version 0 to 115 2024-09-20T22:43:42.603 INFO:tasks.rados.rados.0.smithi084.stdout:254: oid 31 updating version 115 to 116 2024-09-20T22:43:42.603 INFO:tasks.rados.rados.0.smithi084.stdout:254: oid 31 updating version 116 to 117 2024-09-20T22:43:42.603 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 31 v 117 (ObjNum 120 snap 23 seq_num 120) dirty exists 2024-09-20T22:43:42.603 INFO:tasks.rados.rados.0.smithi084.stdout:254: left oid 31 (ObjNum 120 snap 23 seq_num 120) 2024-09-20T22:43:42.603 INFO:tasks.rados.rados.0.smithi084.stdout:255: finishing write tid 1 to smithi08477203-4 2024-09-20T22:43:42.603 INFO:tasks.rados.rados.0.smithi084.stdout:254: done (2 left) 2024-09-20T22:43:42.604 INFO:tasks.rados.rados.0.smithi084.stdout:257: delete oid 48 current snap is 23 2024-09-20T22:43:42.604 INFO:tasks.rados.rados.0.smithi084.stdout:257: done (2 left) 2024-09-20T22:43:42.604 INFO:tasks.rados.rados.0.smithi084.stdout:258: write (excl) oid 12 current snap is 23 2024-09-20T22:43:42.604 INFO:tasks.rados.rados.0.smithi084.stdout:258: seq_num 123 ranges {467261=717275,1847991=219270} 2024-09-20T22:43:42.605 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:42.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:42.606 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:43:42.611 INFO:tasks.rados.rados.0.smithi084.stdout:258: writing smithi08477203-12 from 467261 to 1184536 tid 1 2024-09-20T22:43:42.618 INFO:tasks.rados.rados.0.smithi084.stdout:258: writing smithi08477203-12 from 1847991 to 2067261 tid 2 2024-09-20T22:43:42.618 INFO:tasks.rados.rados.0.smithi084.stdout:259: rollback oid 22 current snap is 23 2024-09-20T22:43:42.618 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 22 to 21 2024-09-20T22:43:42.618 INFO:tasks.rados.rados.0.smithi084.stdout:255: finishing write tid 2 to smithi08477203-4 2024-09-20T22:43:42.618 INFO:tasks.rados.rados.0.smithi084.stdout:255: finishing write tid 3 to smithi08477203-4 2024-09-20T22:43:42.618 INFO:tasks.rados.rados.0.smithi084.stdout:255: finishing write tid 4 to smithi08477203-4 2024-09-20T22:43:42.618 INFO:tasks.rados.rados.0.smithi084.stdout:255: oid 4 updating version 0 to 117 2024-09-20T22:43:42.618 INFO:tasks.rados.rados.0.smithi084.stdout:255: oid 4 version 117 is already newer than 116 2024-09-20T22:43:42.618 INFO:tasks.rados.rados.0.smithi084.stdout:255: oid 4 updating version 117 to 118 2024-09-20T22:43:42.618 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 4 v 118 (ObjNum 121 snap 23 seq_num 121) dirty exists 2024-09-20T22:43:42.619 INFO:tasks.rados.rados.0.smithi084.stdout:255: left oid 4 (ObjNum 121 snap 23 seq_num 121) 2024-09-20T22:43:42.619 INFO:tasks.rados.rados.0.smithi084.stdout:256: expect (ObjNum 104 snap 21 seq_num 104) 2024-09-20T22:43:42.707 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:43:42.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:42.729 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:42.729 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:43:42.830 INFO:tasks.rados.rados.0.smithi084.stdout:255: done (3 left) 2024-09-20T22:43:42.830 INFO:tasks.rados.rados.0.smithi084.stdout:256: done (2 left) 2024-09-20T22:43:42.830 INFO:tasks.rados.rados.0.smithi084.stdout:260: read oid 24 snap -1 2024-09-20T22:43:42.830 INFO:tasks.rados.rados.0.smithi084.stdout:260: expect (ObjNum 104 snap 21 seq_num 104) 2024-09-20T22:43:42.831 INFO:tasks.rados.rados.0.smithi084.stdout:261: write oid 23 current snap is 23 2024-09-20T22:43:42.831 INFO:tasks.rados.rados.0.smithi084.stdout:258: finishing write tid 1 to smithi08477203-12 2024-09-20T22:43:42.831 INFO:tasks.rados.rados.0.smithi084.stdout:258: finishing write tid 2 to smithi08477203-12 2024-09-20T22:43:42.831 INFO:tasks.rados.rados.0.smithi084.stdout:258: finishing write tid 3 to smithi08477203-12 2024-09-20T22:43:42.831 INFO:tasks.rados.rados.0.smithi084.stdout:258: finishing write tid 4 to smithi08477203-12 2024-09-20T22:43:42.831 INFO:tasks.rados.rados.0.smithi084.stdout:258: oid 12 updating version 0 to 114 2024-09-20T22:43:42.831 INFO:tasks.rados.rados.0.smithi084.stdout:258: oid 12 updating version 114 to 115 2024-09-20T22:43:42.831 INFO:tasks.rados.rados.0.smithi084.stdout:258: oid 12 version 115 is already newer than 113 2024-09-20T22:43:42.831 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 12 v 115 (ObjNum 123 snap 23 seq_num 123) dirty exists 2024-09-20T22:43:42.831 INFO:tasks.rados.rados.0.smithi084.stdout:258: left oid 12 (ObjNum 123 snap 23 seq_num 123) 2024-09-20T22:43:42.831 INFO:tasks.rados.rados.0.smithi084.stdout:259: finishing rollback tid 0 to smithi08477203-22 2024-09-20T22:43:42.831 INFO:tasks.rados.rados.0.smithi084.stdout:259: finishing rollback tid 1 to smithi08477203-22 2024-09-20T22:43:42.831 INFO:tasks.rados.rados.0.smithi084.stdout:259: finishing rollback tid 2 to smithi08477203-22 2024-09-20T22:43:42.831 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 22 v 54 (ObjNum 85 snap 11 seq_num 85) dirty exists 2024-09-20T22:43:42.832 INFO:tasks.rados.rados.0.smithi084.stdout:261: seq_num 124 ranges {484068=792524,1906473=177595} 2024-09-20T22:43:42.832 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:43:42.835 INFO:tasks.rados.rados.0.smithi084.stdout:261: writing smithi08477203-23 from 484068 to 1276592 tid 1 2024-09-20T22:43:42.840 INFO:tasks.rados.rados.0.smithi084.stdout:261: writing smithi08477203-23 from 1906473 to 2084068 tid 2 2024-09-20T22:43:42.840 INFO:tasks.rados.rados.0.smithi084.stdout:258: done (3 left) 2024-09-20T22:43:42.841 INFO:tasks.rados.rados.0.smithi084.stdout:259: done (2 left) 2024-09-20T22:43:42.841 INFO:tasks.rados.rados.0.smithi084.stdout:262: read oid 46 snap -1 2024-09-20T22:43:42.841 INFO:tasks.rados.rados.0.smithi084.stdout:262: expect deleted 2024-09-20T22:43:42.841 INFO:tasks.rados.rados.0.smithi084.stdout:263: read oid 38 snap -1 2024-09-20T22:43:42.841 INFO:tasks.rados.rados.0.smithi084.stdout:263: expect deleted 2024-09-20T22:43:42.841 INFO:tasks.rados.rados.0.smithi084.stdout:264: write (excl) oid 42 current snap is 23 2024-09-20T22:43:42.841 INFO:tasks.rados.rados.0.smithi084.stdout:264: seq_num 125 ranges {500875=467773,1564955=535920} 2024-09-20T22:43:42.845 INFO:tasks.rados.rados.0.smithi084.stdout:264: writing smithi08477203-42 from 500875 to 968648 tid 1 2024-09-20T22:43:42.852 INFO:tasks.rados.rados.0.smithi084.stdout:264: writing smithi08477203-42 from 1564955 to 2100875 tid 2 2024-09-20T22:43:42.852 INFO:tasks.rados.rados.0.smithi084.stdout:265: write oid 33 current snap is 23 2024-09-20T22:43:42.853 INFO:tasks.rados.rados.0.smithi084.stdout:260: expect (ObjNum 104 snap 21 seq_num 104) 2024-09-20T22:43:42.854 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:42.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:42.855 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:43:42.956 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:43:42.963 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.7 dump_blocked_ops 2024-09-20T22:43:42.977 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:42.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:42.978 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:43:43.063 INFO:tasks.rados.rados.0.smithi084.stdout:265: seq_num 126 ranges {517682=543022,1623437=494245} 2024-09-20T22:43:43.069 INFO:tasks.rados.rados.0.smithi084.stdout:265: writing smithi08477203-33 from 517682 to 1060704 tid 1 2024-09-20T22:43:43.076 INFO:tasks.rados.rados.0.smithi084.stdout:265: writing smithi08477203-33 from 1623437 to 2117682 tid 2 2024-09-20T22:43:43.076 INFO:tasks.rados.rados.0.smithi084.stdout:260: done (5 left) 2024-09-20T22:43:43.076 INFO:tasks.rados.rados.0.smithi084.stdout:266: write (excl) oid 43 current snap is 23 2024-09-20T22:43:43.076 INFO:tasks.rados.rados.0.smithi084.stdout:261: finishing write tid 1 to smithi08477203-23 2024-09-20T22:43:43.076 INFO:tasks.rados.rados.0.smithi084.stdout:266: seq_num 127 ranges {534489=618271,1565566=568923} 2024-09-20T22:43:43.079 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:43:43.082 INFO:tasks.rados.rados.0.smithi084.stdout:266: writing smithi08477203-43 from 534489 to 1152760 tid 1 2024-09-20T22:43:43.088 INFO:tasks.rados.rados.0.smithi084.stdout:266: writing smithi08477203-43 from 1565566 to 2134489 tid 2 2024-09-20T22:43:43.088 INFO:tasks.rados.rados.0.smithi084.stdout:267: write (excl) oid 26 current snap is 23 2024-09-20T22:43:43.088 INFO:tasks.rados.rados.0.smithi084.stdout:267: seq_num 128 ranges {551296=693520,2024048=127248} 2024-09-20T22:43:43.088 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd unset noscrub 2024-09-20T22:43:43.095 INFO:tasks.rados.rados.0.smithi084.stdout:267: writing smithi08477203-26 from 551296 to 1244816 tid 1 2024-09-20T22:43:43.100 INFO:tasks.rados.rados.0.smithi084.stdout:267: writing smithi08477203-26 from 2024048 to 2151296 tid 2 2024-09-20T22:43:43.100 INFO:tasks.rados.rados.0.smithi084.stdout:268: write (excl) oid 48 current snap is 23 2024-09-20T22:43:43.100 INFO:tasks.rados.rados.0.smithi084.stdout:268: seq_num 129 ranges {568103=768769,2082530=85573} 2024-09-20T22:43:43.101 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:43.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:43.102 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:43:43.107 INFO:tasks.rados.rados.0.smithi084.stdout:268: writing smithi08477203-48 from 568103 to 1336872 tid 1 2024-09-20T22:43:43.113 INFO:tasks.rados.rados.0.smithi084.stdout:268: writing smithi08477203-48 from 2082530 to 2168103 tid 2 2024-09-20T22:43:43.113 INFO:tasks.rados.rados.0.smithi084.stdout:262: done (7 left) 2024-09-20T22:43:43.113 INFO:tasks.rados.rados.0.smithi084.stdout:263: done (6 left) 2024-09-20T22:43:43.113 INFO:tasks.rados.rados.0.smithi084.stdout:269: read oid 29 snap -1 2024-09-20T22:43:43.113 INFO:tasks.rados.rados.0.smithi084.stdout:269: expect (ObjNum 108 snap 22 seq_num 108) 2024-09-20T22:43:43.113 INFO:tasks.rados.rados.0.smithi084.stdout:270: snap_create 2024-09-20T22:43:43.113 INFO:tasks.rados.rados.0.smithi084.stdout:261: finishing write tid 2 to smithi08477203-23 2024-09-20T22:43:43.113 INFO:tasks.rados.rados.0.smithi084.stdout:261: finishing write tid 3 to smithi08477203-23 2024-09-20T22:43:43.114 INFO:tasks.rados.rados.0.smithi084.stdout:261: finishing write tid 4 to smithi08477203-23 2024-09-20T22:43:43.114 INFO:tasks.rados.rados.0.smithi084.stdout:261: oid 23 updating version 0 to 47 2024-09-20T22:43:43.114 INFO:tasks.rados.rados.0.smithi084.stdout:261: oid 23 version 47 is already newer than 45 2024-09-20T22:43:43.114 INFO:tasks.rados.rados.0.smithi084.stdout:261: oid 23 version 47 is already newer than 46 2024-09-20T22:43:43.114 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 23 v 47 (ObjNum 124 snap 23 seq_num 124) dirty exists 2024-09-20T22:43:43.114 INFO:tasks.rados.rados.0.smithi084.stdout:261: left oid 23 (ObjNum 124 snap 23 seq_num 124) 2024-09-20T22:43:43.114 INFO:tasks.rados.rados.0.smithi084.stdout:264: finishing write tid 1 to smithi08477203-42 2024-09-20T22:43:43.114 INFO:tasks.rados.rados.0.smithi084.stdout:264: finishing write tid 2 to smithi08477203-42 2024-09-20T22:43:43.114 INFO:tasks.rados.rados.0.smithi084.stdout:264: finishing write tid 3 to smithi08477203-42 2024-09-20T22:43:43.114 INFO:tasks.rados.rados.0.smithi084.stdout:264: finishing write tid 4 to smithi08477203-42 2024-09-20T22:43:43.114 INFO:tasks.rados.rados.0.smithi084.stdout:264: oid 42 updating version 0 to 42 2024-09-20T22:43:43.114 INFO:tasks.rados.rados.0.smithi084.stdout:264: oid 42 version 42 is already newer than 41 2024-09-20T22:43:43.115 INFO:tasks.rados.rados.0.smithi084.stdout:264: oid 42 updating version 42 to 43 2024-09-20T22:43:43.115 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 42 v 43 (ObjNum 125 snap 23 seq_num 125) dirty exists 2024-09-20T22:43:43.115 INFO:tasks.rados.rados.0.smithi084.stdout:264: left oid 42 (ObjNum 125 snap 23 seq_num 125) 2024-09-20T22:43:43.115 INFO:tasks.rados.rados.0.smithi084.stdout:265: finishing write tid 1 to smithi08477203-33 2024-09-20T22:43:43.115 INFO:tasks.rados.rados.0.smithi084.stdout:265: finishing write tid 2 to smithi08477203-33 2024-09-20T22:43:43.115 INFO:tasks.rados.rados.0.smithi084.stdout:265: finishing write tid 3 to smithi08477203-33 2024-09-20T22:43:43.115 INFO:tasks.rados.rados.0.smithi084.stdout:265: finishing write tid 4 to smithi08477203-33 2024-09-20T22:43:43.115 INFO:tasks.rados.rados.0.smithi084.stdout:265: oid 33 updating version 0 to 48 2024-09-20T22:43:43.115 INFO:tasks.rados.rados.0.smithi084.stdout:265: oid 33 updating version 48 to 49 2024-09-20T22:43:43.115 INFO:tasks.rados.rados.0.smithi084.stdout:265: oid 33 updating version 49 to 50 2024-09-20T22:43:43.115 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 33 v 50 (ObjNum 126 snap 23 seq_num 126) dirty exists 2024-09-20T22:43:43.115 INFO:tasks.rados.rados.0.smithi084.stdout:265: left oid 33 (ObjNum 126 snap 23 seq_num 126) 2024-09-20T22:43:43.116 INFO:tasks.rados.rados.0.smithi084.stdout:266: finishing write tid 1 to smithi08477203-43 2024-09-20T22:43:43.116 INFO:tasks.rados.rados.0.smithi084.stdout:266: finishing write tid 2 to smithi08477203-43 2024-09-20T22:43:43.116 INFO:tasks.rados.rados.0.smithi084.stdout:266: finishing write tid 3 to smithi08477203-43 2024-09-20T22:43:43.116 INFO:tasks.rados.rados.0.smithi084.stdout:266: finishing write tid 4 to smithi08477203-43 2024-09-20T22:43:43.116 INFO:tasks.rados.rados.0.smithi084.stdout:266: oid 43 updating version 0 to 37 2024-09-20T22:43:43.116 INFO:tasks.rados.rados.0.smithi084.stdout:266: oid 43 version 37 is already newer than 36 2024-09-20T22:43:43.116 INFO:tasks.rados.rados.0.smithi084.stdout:266: oid 43 version 37 is already newer than 35 2024-09-20T22:43:43.116 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 43 v 37 (ObjNum 127 snap 23 seq_num 127) dirty exists 2024-09-20T22:43:43.116 INFO:tasks.rados.rados.0.smithi084.stdout:266: left oid 43 (ObjNum 127 snap 23 seq_num 127) 2024-09-20T22:43:43.116 INFO:tasks.rados.rados.0.smithi084.stdout:267: finishing write tid 1 to smithi08477203-26 2024-09-20T22:43:43.116 INFO:tasks.rados.rados.0.smithi084.stdout:267: finishing write tid 2 to smithi08477203-26 2024-09-20T22:43:43.116 INFO:tasks.rados.rados.0.smithi084.stdout:267: finishing write tid 3 to smithi08477203-26 2024-09-20T22:43:43.117 INFO:tasks.rados.rados.0.smithi084.stdout:267: finishing write tid 4 to smithi08477203-26 2024-09-20T22:43:43.117 INFO:tasks.rados.rados.0.smithi084.stdout:267: oid 26 updating version 0 to 117 2024-09-20T22:43:43.117 INFO:tasks.rados.rados.0.smithi084.stdout:267: oid 26 version 117 is already newer than 116 2024-09-20T22:43:43.117 INFO:tasks.rados.rados.0.smithi084.stdout:267: oid 26 updating version 117 to 118 2024-09-20T22:43:43.117 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 26 v 118 (ObjNum 128 snap 23 seq_num 128) dirty exists 2024-09-20T22:43:43.117 INFO:tasks.rados.rados.0.smithi084.stdout:267: left oid 26 (ObjNum 128 snap 23 seq_num 128) 2024-09-20T22:43:43.117 INFO:tasks.rados.rados.0.smithi084.stdout:268: finishing write tid 1 to smithi08477203-48 2024-09-20T22:43:43.118 INFO:tasks.rados.rados.0.smithi084.stdout:268: finishing write tid 2 to smithi08477203-48 2024-09-20T22:43:43.118 INFO:tasks.rados.rados.0.smithi084.stdout:268: finishing write tid 3 to smithi08477203-48 2024-09-20T22:43:43.118 INFO:tasks.rados.rados.0.smithi084.stdout:268: finishing write tid 4 to smithi08477203-48 2024-09-20T22:43:43.118 INFO:tasks.rados.rados.0.smithi084.stdout:268: oid 48 updating version 0 to 53 2024-09-20T22:43:43.118 INFO:tasks.rados.rados.0.smithi084.stdout:268: oid 48 version 53 is already newer than 52 2024-09-20T22:43:43.118 INFO:tasks.rados.rados.0.smithi084.stdout:268: oid 48 version 53 is already newer than 51 2024-09-20T22:43:43.118 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 48 v 53 (ObjNum 129 snap 23 seq_num 129) dirty exists 2024-09-20T22:43:43.118 INFO:tasks.rados.rados.0.smithi084.stdout:268: left oid 48 (ObjNum 129 snap 23 seq_num 129) 2024-09-20T22:43:43.124 INFO:tasks.rados.rados.0.smithi084.stdout:269: expect (ObjNum 108 snap 22 seq_num 108) 2024-09-20T22:43:43.203 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:43:43.220 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:43:43.224 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:43.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:43.225 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:43:43.317 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:43 smithi084 ceph-mon[29774]: pgmap v115: 178 pgs: 10 peering, 2 active+clean+snaptrim, 166 active+clean; 177 MiB data, 611 MiB used, 715 GiB / 715 GiB avail; 5.3 MiB/s rd, 3.3 MiB/s wr, 20 op/s 2024-09-20T22:43:43.317 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:43 smithi084 ceph-mon[29774]: osdmap e143: 8 total, 8 up, 8 in 2024-09-20T22:43:43.318 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:43 smithi084 ceph-mon[29774]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:43:43.318 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:43 smithi084 ceph-mon[29774]: osdmap e144: 8 total, 8 up, 8 in 2024-09-20T22:43:43.326 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:43:43.341 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-20T22:43:43.344 INFO:teuthology.orchestra.run.smithi084.stderr:dumped all 2024-09-20T22:43:43.348 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:43.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:43.349 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:43:43.384 INFO:tasks.rados.rados.0.smithi084.stdout:261: done (7 left) 2024-09-20T22:43:43.384 INFO:tasks.rados.rados.0.smithi084.stdout:264: done (6 left) 2024-09-20T22:43:43.384 INFO:tasks.rados.rados.0.smithi084.stdout:265: done (5 left) 2024-09-20T22:43:43.384 INFO:tasks.rados.rados.0.smithi084.stdout:266: done (4 left) 2024-09-20T22:43:43.384 INFO:tasks.rados.rados.0.smithi084.stdout:267: done (3 left) 2024-09-20T22:43:43.384 INFO:tasks.rados.rados.0.smithi084.stdout:268: done (2 left) 2024-09-20T22:43:43.384 INFO:tasks.rados.rados.0.smithi084.stdout:269: done (1 left) 2024-09-20T22:43:43.384 INFO:tasks.rados.rados.0.smithi084.stdout:270: done (0 left) 2024-09-20T22:43:43.384 INFO:tasks.rados.rados.0.smithi084.stdout:271: snap_remove snap 23 2024-09-20T22:43:43.450 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:43:43.471 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:43.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:43.472 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:43:43.573 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:43:43.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:43.596 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:43.596 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:43:43.606 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:43 smithi154 ceph-mon[33264]: pgmap v115: 178 pgs: 10 peering, 2 active+clean+snaptrim, 166 active+clean; 177 MiB data, 611 MiB used, 715 GiB / 715 GiB avail; 5.3 MiB/s rd, 3.3 MiB/s wr, 20 op/s 2024-09-20T22:43:43.606 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:43 smithi154 ceph-mon[33264]: osdmap e143: 8 total, 8 up, 8 in 2024-09-20T22:43:43.606 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:43 smithi154 ceph-mon[33264]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:43:43.606 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:43 smithi154 ceph-mon[33264]: osdmap e144: 8 total, 8 up, 8 in 2024-09-20T22:43:43.614 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:43 smithi084 ceph-mon[40024]: pgmap v115: 178 pgs: 10 peering, 2 active+clean+snaptrim, 166 active+clean; 177 MiB data, 611 MiB used, 715 GiB / 715 GiB avail; 5.3 MiB/s rd, 3.3 MiB/s wr, 20 op/s 2024-09-20T22:43:43.614 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:43 smithi084 ceph-mon[40024]: osdmap e143: 8 total, 8 up, 8 in 2024-09-20T22:43:43.614 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:43 smithi084 ceph-mon[40024]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:43:43.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:43 smithi084 ceph-mon[40024]: osdmap e144: 8 total, 8 up, 8 in 2024-09-20T22:43:43.697 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:43:43.706 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-20T22:43:43.706 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [], 2024-09-20T22:43:43.706 INFO:teuthology.orchestra.run.smithi154.stdout: "complaint_time": 30, 2024-09-20T22:43:43.706 INFO:teuthology.orchestra.run.smithi154.stdout: "num_blocked_ops": 0 2024-09-20T22:43:43.706 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-20T22:43:43.719 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:43.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:43.720 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:43:43.806 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:43:43.821 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:43:43.842 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:43.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:43.843 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:43:43.944 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:43:43.966 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:43.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:43.967 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:43:44.068 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:43:44.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:44.090 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:44.090 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:43:44.191 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:43:44.212 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:44.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:44.213 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:43:44.314 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:43:44.320 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.7 dump_historic_ops 2024-09-20T22:43:44.333 INFO:tasks.rados.rados.0.smithi084.stdout:271: done (0 left) 2024-09-20T22:43:44.333 INFO:tasks.rados.rados.0.smithi084.stdout:272: copy_from oid 12 from oid 49 current snap is 24 2024-09-20T22:43:44.334 INFO:tasks.rados.rados.0.smithi084.stdout:273: delete oid 18 current snap is 24 2024-09-20T22:43:44.335 INFO:tasks.rados.rados.0.smithi084.stdout:273: done (1 left) 2024-09-20T22:43:44.335 INFO:tasks.rados.rados.0.smithi084.stdout:274: delete oid 45 current snap is 24 2024-09-20T22:43:44.338 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:44.339 INFO:tasks.rados.rados.0.smithi084.stdout:274: done (1 left) 2024-09-20T22:43:44.339 INFO:tasks.rados.rados.0.smithi084.stdout:275: rollback oid 43 current snap is 24 2024-09-20T22:43:44.339 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 43 to 21 2024-09-20T22:43:44.339 INFO:tasks.rados.rados.0.smithi084.stdout:276: write (excl) oid 37 current snap is 24 2024-09-20T22:43:44.339 INFO:tasks.rados.rados.0.smithi084.stdout:276: seq_num 132 ranges {618524=710869,1857976=360548} 2024-09-20T22:43:44.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:44.340 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:43:44.347 INFO:teuthology.orchestra.run.smithi084.stdout:{"pg_ready":true,"pg_map":{"version":117,"stamp":"2024-09-20T22:43:43.120327+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":185617315,"num_objects":278,"num_object_clones":33,"num_object_copies":834,"num_objects_missing_on_primary":0,"num_objects_missing":31,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":278,"num_whiteouts":7,"num_read":1343,"num_read_kb":345738,"num_write":926,"num_write_kb":182130,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4315,"ondisk_log_size":4315,"up":534,"acting":534,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":531,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":625656,"kb_used_data":380284,"kb_used_omap":12,"kb_used_meta":245171,"kb_avail":749335560,"statfs":{"total":767960285184,"available":767319613440,"internally_reserved":0,"allocated":389410816,"data_stored":427710284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":251055664},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000345"},"pg_stats":[{"pgid":"3.1f","version":"86'2","reported_seq":170,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-20T22:43:38.119387+0000","last_change":"2024-09-20T22:42:39.664558+0000","last_active":"2024-09-20T22:43:38.119387+0000","last_peered":"2024-09-20T22:43:38.119387+0000","last_clean":"2024-09-20T22:43:38.119387+0000","last_became_active":"2024-09-20T22:41:38.159085+0000","last_became_peered":"2024-09-20T22:41:38.159085+0000","last_unstale":"2024-09-20T22:43:38.119387+0000","last_undegraded":"2024-09-20T22:43:38.119387+0000","last_fullsized":"2024-09-20T22:43:38.119387+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'2","last_scrub_stamp":"2024-09-20T22:42:39.664421+0000","last_deep_scrub":"86'2","last_deep_scrub_stamp":"2024-09-20T22:42:39.664421+0000","last_clean_scrub_stamp":"2024-09-20T22:42:39.664421+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:40.189792+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"74'9","reported_seq":167,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-20T22:43:38.117954+0000","last_change":"2024-09-20T22:41:40.169673+0000","last_active":"2024-09-20T22:43:38.117954+0000","last_peered":"2024-09-20T22:43:38.117954+0000","last_clean":"2024-09-20T22:43:38.117954+0000","last_became_active":"2024-09-20T22:41:40.168934+0000","last_became_peered":"2024-09-20T22:41:40.168934+0000","last_unstale":"2024-09-20T22:43:38.117954+0000","last_undegraded":"2024-09-20T22:43:38.117954+0000","last_fullsized":"2024-09-20T22:43:38.117954+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":147,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-20T22:43:38.126849+0000","last_change":"2024-09-20T22:41:42.182304+0000","last_active":"2024-09-20T22:43:38.126849+0000","last_peered":"2024-09-20T22:43:38.126849+0000","last_clean":"2024-09-20T22:43:38.126849+0000","last_became_active":"2024-09-20T22:41:42.182024+0000","last_became_peered":"2024-09-20T22:41:42.182024+0000","last_unstale":"2024-09-20T22:43:38.126849+0000","last_undegraded":"2024-09-20T22:43:38.126849+0000","last_fullsized":"2024-09-20T22:43:38.126849+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:42.253432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":143,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-20T22:43:38.117909+0000","last_change":"2024-09-20T22:41:44.198207+0000","last_active":"2024-09-20T22:43:38.117909+0000","last_peered":"2024-09-20T22:43:38.117909+0000","last_clean":"2024-09-20T22:43:38.117909+0000","last_became_active":"2024-09-20T22:41:44.197897+0000","last_became_peered":"2024-09-20T22:41:44.197897+0000","last_unstale":"2024-09-20T22:43:38.117909+0000","last_undegraded":"2024-09-20T22:43:38.117909+0000","last_fullsized":"2024-09-20T22:43:38.117909+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":151,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-20T22:43:38.122893+0000","last_change":"2024-09-20T22:43:07.476040+0000","last_active":"2024-09-20T22:43:38.122893+0000","last_peered":"2024-09-20T22:43:38.122893+0000","last_clean":"2024-09-20T22:43:38.122893+0000","last_became_active":"2024-09-20T22:41:44.832099+0000","last_became_peered":"2024-09-20T22:41:44.832099+0000","last_unstale":"2024-09-20T22:43:38.122893+0000","last_undegraded":"2024-09-20T22:43:38.122893+0000","last_fullsized":"2024-09-20T22:43:38.122893+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:07.475964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:43:07.475964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:15.747336+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":155,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-20T22:43:38.122536+0000","last_change":"2024-09-20T22:41:38.157685+0000","last_active":"2024-09-20T22:43:38.122536+0000","last_peered":"2024-09-20T22:43:38.122536+0000","last_clean":"2024-09-20T22:43:38.122536+0000","last_became_active":"2024-09-20T22:41:38.157298+0000","last_became_peered":"2024-09-20T22:41:38.157298+0000","last_unstale":"2024-09-20T22:43:38.122536+0000","last_undegraded":"2024-09-20T22:43:38.122536+0000","last_fullsized":"2024-09-20T22:43:38.122536+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"74'15","reported_seq":176,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-20T22:43:38.122194+0000","last_change":"2024-09-20T22:41:40.163585+0000","last_active":"2024-09-20T22:43:38.122194+0000","last_peered":"2024-09-20T22:43:38.122194+0000","last_clean":"2024-09-20T22:43:38.122194+0000","last_became_active":"2024-09-20T22:41:40.163242+0000","last_became_peered":"2024-09-20T22:41:40.163242+0000","last_unstale":"2024-09-20T22:43:38.122194+0000","last_undegraded":"2024-09-20T22:43:38.122194+0000","last_fullsized":"2024-09-20T22:43:38.122194+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":155,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-20T22:43:38.117766+0000","last_change":"2024-09-20T22:43:02.971265+0000","last_active":"2024-09-20T22:43:38.117766+0000","last_peered":"2024-09-20T22:43:38.117766+0000","last_clean":"2024-09-20T22:43:38.117766+0000","last_became_active":"2024-09-20T22:41:42.187076+0000","last_became_peered":"2024-09-20T22:41:42.187076+0000","last_unstale":"2024-09-20T22:43:38.117766+0000","last_undegraded":"2024-09-20T22:43:38.117766+0000","last_fullsized":"2024-09-20T22:43:38.117766+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:02.971194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:43:02.971194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:08.332638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.19","version":"131'44","reported_seq":2,"reported_epoch":139,"state":"peering","last_fresh":"2024-09-20T22:43:37.164346+0000","last_change":"2024-09-20T22:43:38.621097+0000","last_active":"2024-09-20T22:43:38.621097+0000","last_peered":"2024-09-20T22:43:37.164346+0000","last_clean":"2024-09-20T22:43:37.164346+0000","last_became_active":"2024-09-20T22:43:37.143059+0000","last_became_peered":"2024-09-20T22:43:37.143059+0000","last_unstale":"2024-09-20T22:43:37.164346+0000","last_undegraded":"2024-09-20T22:43:37.164346+0000","last_fullsized":"2024-09-20T22:43:37.164346+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:42:51.354519+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043941,"stat_sum":{"num_bytes":7168103,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":6,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":13,"num_read_kb":6152,"num_write":17,"num_write_kb":7912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"f","length":"2"},{"start":"12","length":"3"}]},{"pgid":"3.1d","version":"0'0","reported_seq":163,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-20T22:43:38.124178+0000","last_change":"2024-09-20T22:43:34.682062+0000","last_active":"2024-09-20T22:43:38.124178+0000","last_peered":"2024-09-20T22:43:38.124178+0000","last_clean":"2024-09-20T22:43:38.124178+0000","last_became_active":"2024-09-20T22:41:38.154827+0000","last_became_peered":"2024-09-20T22:41:38.154827+0000","last_unstale":"2024-09-20T22:43:38.124178+0000","last_undegraded":"2024-09-20T22:43:38.124178+0000","last_fullsized":"2024-09-20T22:43:38.124178+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:34.681993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:43:34.681993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:35.009253+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"74'9","reported_seq":175,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-20T22:43:38.118255+0000","last_change":"2024-09-20T22:43:30.938429+0000","last_active":"2024-09-20T22:43:38.118255+0000","last_peered":"2024-09-20T22:43:38.118255+0000","last_clean":"2024-09-20T22:43:38.118255+0000","last_became_active":"2024-09-20T22:41:40.162546+0000","last_became_peered":"2024-09-20T22:41:40.162546+0000","last_unstale":"2024-09-20T22:43:38.118255+0000","last_undegraded":"2024-09-20T22:43:38.118255+0000","last_fullsized":"2024-09-20T22:43:38.118255+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-09-20T22:43:30.938345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:43:30.938345+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:39.379969+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":147,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-20T22:43:38.122129+0000","last_change":"2024-09-20T22:41:42.185835+0000","last_active":"2024-09-20T22:43:38.122129+0000","last_peered":"2024-09-20T22:43:38.122129+0000","last_clean":"2024-09-20T22:43:38.122129+0000","last_became_active":"2024-09-20T22:41:42.185538+0000","last_became_peered":"2024-09-20T22:41:42.185538+0000","last_unstale":"2024-09-20T22:43:38.122129+0000","last_undegraded":"2024-09-20T22:43:38.122129+0000","last_fullsized":"2024-09-20T22:43:38.122129+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":143,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-20T22:43:38.117138+0000","last_change":"2024-09-20T22:41:44.198169+0000","last_active":"2024-09-20T22:43:38.117138+0000","last_peered":"2024-09-20T22:43:38.117138+0000","last_clean":"2024-09-20T22:43:38.117138+0000","last_became_active":"2024-09-20T22:41:44.197113+0000","last_became_peered":"2024-09-20T22:41:44.197113+0000","last_unstale":"2024-09-20T22:43:38.117138+0000","last_undegraded":"2024-09-20T22:43:38.117138+0000","last_fullsized":"2024-09-20T22:43:38.117138+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.18","version":"138'41","reported_seq":2,"reported_epoch":139,"state":"peering","last_fresh":"2024-09-20T22:43:37.178674+0000","last_change":"2024-09-20T22:43:38.615807+0000","last_active":"2024-09-20T22:43:38.615807+0000","last_peered":"2024-09-20T22:43:37.178674+0000","last_clean":"2024-09-20T22:43:37.178674+0000","last_became_active":"2024-09-20T22:43:37.158939+0000","last_became_peered":"2024-09-20T22:43:37.158939+0000","last_unstale":"2024-09-20T22:43:37.178674+0000","last_undegraded":"2024-09-20T22:43:37.178674+0000","last_fullsized":"2024-09-20T22:43:37.178674+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:42:51.354519+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.019421418999999999,"stat_sum":{"num_bytes":5971473,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":4,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":15,"num_read_kb":12287,"num_write":14,"num_write_kb":4006,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f","length":"2"},{"start":"12","length":"3"}]},{"pgid":"3.1c","version":"0'0","reported_seq":163,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-20T22:43:38.123333+0000","last_change":"2024-09-20T22:43:00.713953+0000","last_active":"2024-09-20T22:43:38.123333+0000","last_peered":"2024-09-20T22:43:38.123333+0000","last_clean":"2024-09-20T22:43:38.123333+0000","last_became_active":"2024-09-20T22:41:38.154650+0000","last_became_peered":"2024-09-20T22:41:38.154650+0000","last_unstale":"2024-09-20T22:43:38.123333+0000","last_undegraded":"2024-09-20T22:43:38.123333+0000","last_fullsized":"2024-09-20T22:43:38.123333+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:00.713878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:43:00.713878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:21.268170+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"74'5","reported_seq":161,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-20T22:43:38.118652+0000","last_change":"2024-09-20T22:41:40.171883+0000","last_active":"2024-09-20T22:43:38.118652+0000","last_peered":"2024-09-20T22:43:38.118652+0000","last_clean":"2024-09-20T22:43:38.118652+0000","last_became_active":"2024-09-20T22:41:40.171081+0000","last_became_peered":"2024-09-20T22:41:40.171081+0000","last_unstale":"2024-09-20T22:43:38.118652+0000","last_undegraded":"2024-09-20T22:43:38.118652+0000","last_fullsized":"2024-09-20T22:43:38.118652+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":158,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-20T22:43:38.121913+0000","last_change":"2024-09-20T22:43:01.248846+0000","last_active":"2024-09-20T22:43:38.121913+0000","last_peered":"2024-09-20T22:43:38.121913+0000","last_clean":"2024-09-20T22:43:38.121913+0000","last_became_active":"2024-09-20T22:41:42.182916+0000","last_became_peered":"2024-09-20T22:41:42.182916+0000","last_unstale":"2024-09-20T22:43:38.121913+0000","last_undegraded":"2024-09-20T22:43:38.121913+0000","last_fullsized":"2024-09-20T22:43:38.121913+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:01.248740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:43:01.248740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:09.032402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":151,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-20T22:43:38.121951+0000","last_change":"2024-09-20T22:43:32.712751+0000","last_active":"2024-09-20T22:43:38.121951+0000","last_peered":"2024-09-20T22:43:38.121951+0000","last_clean":"2024-09-20T22:43:38.121951+0000","last_became_active":"2024-09-20T22:41:44.197927+0000","last_became_peered":"2024-09-20T22:41:44.197927+0000","last_unstale":"2024-09-20T22:43:38.121951+0000","last_undegraded":"2024-09-20T22:43:38.121951+0000","last_fullsized":"2024-09-20T22:43:38.121951+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:32.712676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:43:32.712676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:02.620815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":151,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-20T22:43:38.124567+0000","last_change":"2024-09-20T22:43:10.000692+0000","last_active":"2024-09-20T22:43:38.124567+0000","last_peered":"2024-09-20T22:43:38.124567+0000","last_clean":"2024-09-20T22:43:38.124567+0000","last_became_active":"2024-09-20T22:41:44.832360+0000","last_became_peered":"2024-09-20T22:41:44.832360+0000","last_unstale":"2024-09-20T22:43:38.124567+0000","last_undegraded":"2024-09-20T22:43:38.124567+0000","last_fullsized":"2024-09-20T22:43:38.124567+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:10.000610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:43:10.000610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:33.885490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":155,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-20T22:43:38.120088+0000","last_change":"2024-09-20T22:41:38.152726+0000","last_active":"2024-09-20T22:43:38.120088+0000","last_peered":"2024-09-20T22:43:38.120088+0000","last_clean":"2024-09-20T22:43:38.120088+0000","last_became_active":"2024-09-20T22:41:38.152443+0000","last_became_peered":"2024-09-20T22:41:38.152443+0000","last_unstale":"2024-09-20T22:43:38.120088+0000","last_undegraded":"2024-09-20T22:43:38.120088+0000","last_fullsized":"2024-09-20T22:43:38.120088+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"113'22","reported_seq":191,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-20T22:43:38.130975+0000","last_change":"2024-09-20T22:43:12.191980+0000","last_active":"2024-09-20T22:43:38.130975+0000","last_peered":"2024-09-20T22:43:38.130975+0000","last_clean":"2024-09-20T22:43:38.130975+0000","last_became_active":"2024-09-20T22:41:40.170131+0000","last_became_peered":"2024-09-20T22:41:40.170131+0000","last_unstale":"2024-09-20T22:43:38.130975+0000","last_undegraded":"2024-09-20T22:43:38.130975+0000","last_fullsized":"2024-09-20T22:43:38.130975+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'22","last_scrub_stamp":"2024-09-20T22:43:12.191878+0000","last_deep_scrub":"113'22","last_deep_scrub_stamp":"2024-09-20T22:43:12.191878+0000","last_clean_scrub_stamp":"2024-09-20T22:43:12.191878+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:28.270781+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":147,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-20T22:43:38.131369+0000","last_change":"2024-09-20T22:41:42.185171+0000","last_active":"2024-09-20T22:43:38.131369+0000","last_peered":"2024-09-20T22:43:38.131369+0000","last_clean":"2024-09-20T22:43:38.131369+0000","last_became_active":"2024-09-20T22:41:42.184759+0000","last_became_peered":"2024-09-20T22:41:42.184759+0000","last_unstale":"2024-09-20T22:43:38.131369+0000","last_undegraded":"2024-09-20T22:43:38.131369+0000","last_fullsized":"2024-09-20T22:43:38.131369+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:42.253447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":143,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-20T22:43:38.121454+0000","last_change":"2024-09-20T22:41:44.832468+0000","last_active":"2024-09-20T22:43:38.121454+0000","last_peered":"2024-09-20T22:43:38.121454+0000","last_clean":"2024-09-20T22:43:38.121454+0000","last_became_active":"2024-09-20T22:41:44.832131+0000","last_became_peered":"2024-09-20T22:41:44.832131+0000","last_unstale":"2024-09-20T22:43:38.121454+0000","last_undegraded":"2024-09-20T22:43:38.121454+0000","last_fullsized":"2024-09-20T22:43:38.121454+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":155,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-20T22:43:38.118309+0000","last_change":"2024-09-20T22:41:38.150584+0000","last_active":"2024-09-20T22:43:38.118309+0000","last_peered":"2024-09-20T22:43:38.118309+0000","last_clean":"2024-09-20T22:43:38.118309+0000","last_became_active":"2024-09-20T22:41:38.150232+0000","last_became_peered":"2024-09-20T22:41:38.150232+0000","last_unstale":"2024-09-20T22:43:38.118309+0000","last_undegraded":"2024-09-20T22:43:38.118309+0000","last_fullsized":"2024-09-20T22:43:38.118309+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"74'12","reported_seq":192,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-20T22:43:38.119428+0000","last_change":"2024-09-20T22:43:09.123893+0000","last_active":"2024-09-20T22:43:38.119428+0000","last_peered":"2024-09-20T22:43:38.119428+0000","last_clean":"2024-09-20T22:43:38.119428+0000","last_became_active":"2024-09-20T22:43:09.116985+0000","last_became_peered":"2024-09-20T22:43:09.116985+0000","last_unstale":"2024-09-20T22:43:38.119428+0000","last_undegraded":"2024-09-20T22:43:38.119428+0000","last_fullsized":"2024-09-20T22:43:38.119428+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":147,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-20T22:43:38.119612+0000","last_change":"2024-09-20T22:41:42.183266+0000","last_active":"2024-09-20T22:43:38.119612+0000","last_peered":"2024-09-20T22:43:38.119612+0000","last_clean":"2024-09-20T22:43:38.119612+0000","last_became_active":"2024-09-20T22:41:42.183005+0000","last_became_peered":"2024-09-20T22:41:42.183005+0000","last_unstale":"2024-09-20T22:43:38.119612+0000","last_undegraded":"2024-09-20T22:43:38.119612+0000","last_fullsized":"2024-09-20T22:43:38.119612+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":144,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-20T22:43:38.122998+0000","last_change":"2024-09-20T22:41:44.200115+0000","last_active":"2024-09-20T22:43:38.122998+0000","last_peered":"2024-09-20T22:43:38.122998+0000","last_clean":"2024-09-20T22:43:38.122998+0000","last_became_active":"2024-09-20T22:41:44.199822+0000","last_became_peered":"2024-09-20T22:41:44.199822+0000","last_unstale":"2024-09-20T22:43:38.122998+0000","last_undegraded":"2024-09-20T22:43:38.122998+0000","last_fullsized":"2024-09-20T22:43:38.122998+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:41.182601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"67'1","reported_seq":164,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-20T22:43:38.127004+0000","last_change":"2024-09-20T22:43:35.266315+0000","last_active":"2024-09-20T22:43:38.127004+0000","last_peered":"2024-09-20T22:43:38.127004+0000","last_clean":"2024-09-20T22:43:38.127004+0000","last_became_active":"2024-09-20T22:41:38.155424+0000","last_became_peered":"2024-09-20T22:41:38.155424+0000","last_unstale":"2024-09-20T22:43:38.127004+0000","last_undegraded":"2024-09-20T22:43:38.127004+0000","last_fullsized":"2024-09-20T22:43:38.127004+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-09-20T22:43:35.266242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:43:35.266242+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:49.485624+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"74'10","reported_seq":166,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-20T22:43:38.120279+0000","last_change":"2024-09-20T22:41:40.252498+0000","last_active":"2024-09-20T22:43:38.120279+0000","last_peered":"2024-09-20T22:43:38.120279+0000","last_clean":"2024-09-20T22:43:38.120279+0000","last_became_active":"2024-09-20T22:41:40.252001+0000","last_became_peered":"2024-09-20T22:41:40.252001+0000","last_unstale":"2024-09-20T22:43:38.120279+0000","last_undegraded":"2024-09-20T22:43:38.120279+0000","last_fullsized":"2024-09-20T22:43:38.120279+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"74'8","reported_seq":186,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-20T22:43:38.128523+0000","last_change":"2024-09-20T22:41:42.186385+0000","last_active":"2024-09-20T22:43:38.128523+0000","last_peered":"2024-09-20T22:43:38.128523+0000","last_clean":"2024-09-20T22:43:38.128523+0000","last_became_active":"2024-09-20T22:41:42.186071+0000","last_became_peered":"2024-09-20T22:41:42.186071+0000","last_unstale":"2024-09-20T22:43:38.128523+0000","last_undegraded":"2024-09-20T22:43:38.128523+0000","last_fullsized":"2024-09-20T22:43:38.128523+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:42.428271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.c","version":"137'93","reported_seq":336,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-20T22:43:38.168352+0000","last_change":"2024-09-20T22:43:38.127237+0000","last_active":"2024-09-20T22:43:38.168352+0000","last_peered":"2024-09-20T22:43:38.168352+0000","last_clean":"2024-09-20T22:43:38.168352+0000","last_became_active":"2024-09-20T22:42:52.404027+0000","last_became_peered":"2024-09-20T22:42:52.404027+0000","last_unstale":"2024-09-20T22:43:38.168352+0000","last_undegraded":"2024-09-20T22:43:38.168352+0000","last_fullsized":"2024-09-20T22:43:38.168352+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:42:51.354519+0000","objects_scrubbed":0,"log_size":93,"log_dups_size":0,"ondisk_log_size":93,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:20.348008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048673400000000001,"stat_sum":{"num_bytes":26588175,"num_objects":11,"num_object_clones":4,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":1,"num_read":88,"num_read_kb":73933,"num_write":70,"num_write_kb":22719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"10","length":"1"},{"start":"12","length":"3"},{"start":"18","length":"1"}]},{"pgid":"1.a","version":"77'128","reported_seq":399,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-20T22:43:38.122500+0000","last_change":"2024-09-20T22:43:05.763879+0000","last_active":"2024-09-20T22:43:38.122500+0000","last_peered":"2024-09-20T22:43:38.122500+0000","last_clean":"2024-09-20T22:43:38.122500+0000","last_became_active":"2024-09-20T22:42:52.407791+0000","last_became_peered":"2024-09-20T22:42:52.407791+0000","last_unstale":"2024-09-20T22:43:38.122500+0000","last_undegraded":"2024-09-20T22:43:38.122500+0000","last_fullsized":"2024-09-20T22:43:38.122500+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:43:05.763796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:43:05.763796+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:11.866914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"74'15","reported_seq":184,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-20T22:43:38.130515+0000","last_change":"2024-09-20T22:43:10.076804+0000","last_active":"2024-09-20T22:43:38.130515+0000","last_peered":"2024-09-20T22:43:38.130515+0000","last_clean":"2024-09-20T22:43:38.130515+0000","last_became_active":"2024-09-20T22:41:40.165963+0000","last_became_peered":"2024-09-20T22:41:40.165963+0000","last_unstale":"2024-09-20T22:43:38.130515+0000","last_undegraded":"2024-09-20T22:43:38.130515+0000","last_fullsized":"2024-09-20T22:43:38.130515+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-09-20T22:43:10.076699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:43:10.076699+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:37.109196+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":155,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-20T22:43:38.120481+0000","last_change":"2024-09-20T22:41:38.157043+0000","last_active":"2024-09-20T22:43:38.120481+0000","last_peered":"2024-09-20T22:43:38.120481+0000","last_clean":"2024-09-20T22:43:38.120481+0000","last_became_active":"2024-09-20T22:41:38.156749+0000","last_became_peered":"2024-09-20T22:41:38.156749+0000","last_unstale":"2024-09-20T22:43:38.120481+0000","last_undegraded":"2024-09-20T22:43:38.120481+0000","last_fullsized":"2024-09-20T22:43:38.120481+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"74'8","reported_seq":180,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-20T22:43:38.124689+0000","last_change":"2024-09-20T22:41:42.182555+0000","last_active":"2024-09-20T22:43:38.124689+0000","last_peered":"2024-09-20T22:43:38.124689+0000","last_clean":"2024-09-20T22:43:38.124689+0000","last_became_active":"2024-09-20T22:41:42.182224+0000","last_became_peered":"2024-09-20T22:41:42.182224+0000","last_unstale":"2024-09-20T22:43:38.124689+0000","last_undegraded":"2024-09-20T22:43:38.124689+0000","last_fullsized":"2024-09-20T22:43:38.124689+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":143,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-20T22:43:38.123855+0000","last_change":"2024-09-20T22:41:44.198655+0000","last_active":"2024-09-20T22:43:38.123855+0000","last_peered":"2024-09-20T22:43:38.123855+0000","last_clean":"2024-09-20T22:43:38.123855+0000","last_became_active":"2024-09-20T22:41:44.198347+0000","last_became_peered":"2024-09-20T22:41:44.198347+0000","last_unstale":"2024-09-20T22:43:38.123855+0000","last_undegraded":"2024-09-20T22:43:38.123855+0000","last_fullsized":"2024-09-20T22:43:38.123855+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3","version":"135'8","reported_seq":233,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-20T22:43:38.122972+0000","last_change":"2024-09-20T22:43:38.125425+0000","last_active":"2024-09-20T22:43:38.122972+0000","last_peered":"2024-09-20T22:43:38.122972+0000","last_clean":"2024-09-20T22:43:38.122972+0000","last_became_active":"2024-09-20T22:43:37.140067+0000","last_became_peered":"2024-09-20T22:43:37.140067+0000","last_unstale":"2024-09-20T22:43:38.122972+0000","last_undegraded":"2024-09-20T22:43:38.122972+0000","last_fullsized":"2024-09-20T22:43:38.122972+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:42:51.354519+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0026960869999999998,"stat_sum":{"num_bytes":2441608,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":5552,"num_write":4,"num_write_kb":1492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"10","length":"1"},{"start":"12","length":"3"},{"start":"18","length":"1"}]},{"pgid":"1.5","version":"77'128","reported_seq":412,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-20T22:43:38.123873+0000","last_change":"2024-09-20T22:43:25.851297+0000","last_active":"2024-09-20T22:43:38.123873+0000","last_peered":"2024-09-20T22:43:38.123873+0000","last_clean":"2024-09-20T22:43:38.123873+0000","last_became_active":"2024-09-20T22:43:19.128074+0000","last_became_peered":"2024-09-20T22:43:19.128074+0000","last_unstale":"2024-09-20T22:43:38.123873+0000","last_undegraded":"2024-09-20T22:43:38.123873+0000","last_fullsized":"2024-09-20T22:43:38.123873+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:43:25.851220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:43:25.851220+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:36.958582+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"74'18","reported_seq":183,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-20T22:43:38.120264+0000","last_change":"2024-09-20T22:41:40.252203+0000","last_active":"2024-09-20T22:43:38.120264+0000","last_peered":"2024-09-20T22:43:38.120264+0000","last_clean":"2024-09-20T22:43:38.120264+0000","last_became_active":"2024-09-20T22:41:40.251868+0000","last_became_peered":"2024-09-20T22:41:40.251868+0000","last_unstale":"2024-09-20T22:43:38.120264+0000","last_undegraded":"2024-09-20T22:43:38.120264+0000","last_fullsized":"2024-09-20T22:43:38.120264+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":155,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-20T22:43:38.121684+0000","last_change":"2024-09-20T22:41:38.161991+0000","last_active":"2024-09-20T22:43:38.121684+0000","last_peered":"2024-09-20T22:43:38.121684+0000","last_clean":"2024-09-20T22:43:38.121684+0000","last_became_active":"2024-09-20T22:41:38.161335+0000","last_became_peered":"2024-09-20T22:41:38.161335+0000","last_unstale":"2024-09-20T22:43:38.121684+0000","last_undegraded":"2024-09-20T22:43:38.121684+0000","last_fullsized":"2024-09-20T22:43:38.121684+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":155,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-20T22:43:38.119098+0000","last_change":"2024-09-20T22:43:07.948019+0000","last_active":"2024-09-20T22:43:38.119098+0000","last_peered":"2024-09-20T22:43:38.119098+0000","last_clean":"2024-09-20T22:43:38.119098+0000","last_became_active":"2024-09-20T22:41:42.189743+0000","last_became_peered":"2024-09-20T22:41:42.189743+0000","last_unstale":"2024-09-20T22:43:38.119098+0000","last_undegraded":"2024-09-20T22:43:38.119098+0000","last_fullsized":"2024-09-20T22:43:38.119098+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:07.947945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:07.947945+0000","last_clean_scrub_stamp":"2024-09-20T22:43:07.947945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:23.340969+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":143,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-20T22:43:38.118162+0000","last_change":"2024-09-20T22:41:44.193944+0000","last_active":"2024-09-20T22:43:38.118162+0000","last_peered":"2024-09-20T22:43:38.118162+0000","last_clean":"2024-09-20T22:43:38.118162+0000","last_became_active":"2024-09-20T22:41:44.193646+0000","last_became_peered":"2024-09-20T22:41:44.193646+0000","last_unstale":"2024-09-20T22:43:38.118162+0000","last_undegraded":"2024-09-20T22:43:38.118162+0000","last_fullsized":"2024-09-20T22:43:38.118162+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2","version":"138'78","reported_seq":318,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-20T22:43:38.125486+0000","last_change":"2024-09-20T22:43:38.125916+0000","last_active":"2024-09-20T22:43:38.125486+0000","last_peered":"2024-09-20T22:43:38.125486+0000","last_clean":"2024-09-20T22:43:38.125486+0000","last_became_active":"2024-09-20T22:43:37.134031+0000","last_became_peered":"2024-09-20T22:43:37.134031+0000","last_unstale":"2024-09-20T22:43:38.125486+0000","last_undegraded":"2024-09-20T22:43:38.125486+0000","last_fullsized":"2024-09-20T22:43:38.125486+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:42:51.354519+0000","objects_scrubbed":0,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012631750000000001,"stat_sum":{"num_bytes":9585042,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":29,"num_read_kb":20428,"num_write":33,"num_write_kb":10763,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"10","length":"1"},{"start":"12","length":"3"},{"start":"18","length":"1"}]},{"pgid":"1.4","version":"77'128","reported_seq":412,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-20T22:43:38.120584+0000","last_change":"2024-09-20T22:43:26.151507+0000","last_active":"2024-09-20T22:43:38.120584+0000","last_peered":"2024-09-20T22:43:38.120584+0000","last_clean":"2024-09-20T22:43:38.120584+0000","last_became_active":"2024-09-20T22:43:19.125641+0000","last_became_peered":"2024-09-20T22:43:19.125641+0000","last_unstale":"2024-09-20T22:43:38.120584+0000","last_undegraded":"2024-09-20T22:43:38.120584+0000","last_fullsized":"2024-09-20T22:43:38.120584+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:43:26.151406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:43:26.151406+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:28.804449+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"74'14","reported_seq":172,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-20T22:43:38.119157+0000","last_change":"2024-09-20T22:41:40.164011+0000","last_active":"2024-09-20T22:43:38.119157+0000","last_peered":"2024-09-20T22:43:38.119157+0000","last_clean":"2024-09-20T22:43:38.119157+0000","last_became_active":"2024-09-20T22:41:40.163695+0000","last_became_peered":"2024-09-20T22:41:40.163695+0000","last_unstale":"2024-09-20T22:43:38.119157+0000","last_undegraded":"2024-09-20T22:43:38.119157+0000","last_fullsized":"2024-09-20T22:43:38.119157+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"67'1","reported_seq":164,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-20T22:43:38.120483+0000","last_change":"2024-09-20T22:43:29.695069+0000","last_active":"2024-09-20T22:43:38.120483+0000","last_peered":"2024-09-20T22:43:38.120483+0000","last_clean":"2024-09-20T22:43:38.120483+0000","last_became_active":"2024-09-20T22:41:38.150331+0000","last_became_peered":"2024-09-20T22:41:38.150331+0000","last_unstale":"2024-09-20T22:43:38.120483+0000","last_undegraded":"2024-09-20T22:43:38.120483+0000","last_fullsized":"2024-09-20T22:43:38.120483+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-09-20T22:43:29.694945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:43:29.694945+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:33.150906+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"74'8","reported_seq":180,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-20T22:43:38.120037+0000","last_change":"2024-09-20T22:41:42.179893+0000","last_active":"2024-09-20T22:43:38.120037+0000","last_peered":"2024-09-20T22:43:38.120037+0000","last_clean":"2024-09-20T22:43:38.120037+0000","last_became_active":"2024-09-20T22:41:42.179613+0000","last_became_peered":"2024-09-20T22:41:42.179613+0000","last_unstale":"2024-09-20T22:43:38.120037+0000","last_undegraded":"2024-09-20T22:43:38.120037+0000","last_fullsized":"2024-09-20T22:43:38.120037+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":143,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-20T22:43:38.121507+0000","last_change":"2024-09-20T22:41:44.832161+0000","last_active":"2024-09-20T22:43:38.121507+0000","last_peered":"2024-09-20T22:43:38.121507+0000","last_clean":"2024-09-20T22:43:38.121507+0000","last_became_active":"2024-09-20T22:41:44.831901+0000","last_became_peered":"2024-09-20T22:41:44.831901+0000","last_unstale":"2024-09-20T22:43:38.121507+0000","last_undegraded":"2024-09-20T22:43:38.121507+0000","last_fullsized":"2024-09-20T22:43:38.121507+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:41.182693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1","version":"138'41","reported_seq":261,"reported_epoch":139,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T22:43:38.118042+0000","last_change":"2024-09-20T22:43:37.140488+0000","last_active":"2024-09-20T22:43:38.118042+0000","last_peered":"2024-09-20T22:43:38.118042+0000","last_clean":"2024-09-20T22:43:38.118042+0000","last_became_active":"2024-09-20T22:43:37.139569+0000","last_became_peered":"2024-09-20T22:43:37.139569+0000","last_unstale":"2024-09-20T22:43:38.118042+0000","last_undegraded":"2024-09-20T22:43:38.118042+0000","last_fullsized":"2024-09-20T22:43:38.118042+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:42:51.354519+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040462499999999998,"stat_sum":{"num_bytes":4156377,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":9,"num_read_kb":6536,"num_write":12,"num_write_kb":5792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"10","length":"1"},{"start":"14","length":"1"}]},{"pgid":"1.7","version":"77'128","reported_seq":419,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-20T22:43:38.127890+0000","last_change":"2024-09-20T22:43:26.566566+0000","last_active":"2024-09-20T22:43:38.127890+0000","last_peered":"2024-09-20T22:43:38.127890+0000","last_clean":"2024-09-20T22:43:38.127890+0000","last_became_active":"2024-09-20T22:43:19.129015+0000","last_became_peered":"2024-09-20T22:43:19.129015+0000","last_unstale":"2024-09-20T22:43:38.127890+0000","last_undegraded":"2024-09-20T22:43:38.127890+0000","last_fullsized":"2024-09-20T22:43:38.127890+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:43:26.566436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:43:26.566436+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:54.250940+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"74'10","reported_seq":174,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-20T22:43:38.131928+0000","last_change":"2024-09-20T22:43:13.094497+0000","last_active":"2024-09-20T22:43:38.131928+0000","last_peered":"2024-09-20T22:43:38.131928+0000","last_clean":"2024-09-20T22:43:38.131928+0000","last_became_active":"2024-09-20T22:41:40.165653+0000","last_became_peered":"2024-09-20T22:41:40.165653+0000","last_unstale":"2024-09-20T22:43:38.131928+0000","last_undegraded":"2024-09-20T22:43:38.131928+0000","last_fullsized":"2024-09-20T22:43:38.131928+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-09-20T22:43:13.094419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:43:13.094419+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:25.645533+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":163,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-20T22:43:38.124266+0000","last_change":"2024-09-20T22:42:39.595884+0000","last_active":"2024-09-20T22:43:38.124266+0000","last_peered":"2024-09-20T22:43:38.124266+0000","last_clean":"2024-09-20T22:43:38.124266+0000","last_became_active":"2024-09-20T22:41:38.155629+0000","last_became_peered":"2024-09-20T22:41:38.155629+0000","last_unstale":"2024-09-20T22:43:38.124266+0000","last_undegraded":"2024-09-20T22:43:38.124266+0000","last_fullsized":"2024-09-20T22:43:38.124266+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:42:39.595799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:42:39.595799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:39.804106+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"74'8","reported_seq":188,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-20T22:43:38.117882+0000","last_change":"2024-09-20T22:43:08.643664+0000","last_active":"2024-09-20T22:43:38.117882+0000","last_peered":"2024-09-20T22:43:38.117882+0000","last_clean":"2024-09-20T22:43:38.117882+0000","last_became_active":"2024-09-20T22:41:42.185213+0000","last_became_peered":"2024-09-20T22:41:42.185213+0000","last_unstale":"2024-09-20T22:43:38.117882+0000","last_undegraded":"2024-09-20T22:43:38.117882+0000","last_fullsized":"2024-09-20T22:43:38.117882+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-09-20T22:43:08.643587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:43:08.643587+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:29.139515+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":151,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-20T22:43:38.117241+0000","last_change":"2024-09-20T22:43:33.657671+0000","last_active":"2024-09-20T22:43:38.117241+0000","last_peered":"2024-09-20T22:43:38.117241+0000","last_clean":"2024-09-20T22:43:38.117241+0000","last_became_active":"2024-09-20T22:41:44.197619+0000","last_became_peered":"2024-09-20T22:41:44.197619+0000","last_unstale":"2024-09-20T22:43:38.117241+0000","last_undegraded":"2024-09-20T22:43:38.117241+0000","last_fullsized":"2024-09-20T22:43:38.117241+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:33.657587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:43:33.657587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:50.951002+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.0","version":"138'46","reported_seq":271,"reported_epoch":139,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T22:43:38.129421+0000","last_change":"2024-09-20T22:43:37.137765+0000","last_active":"2024-09-20T22:43:38.129421+0000","last_peered":"2024-09-20T22:43:38.129421+0000","last_clean":"2024-09-20T22:43:38.129421+0000","last_became_active":"2024-09-20T22:43:37.136845+0000","last_became_peered":"2024-09-20T22:43:37.136845+0000","last_unstale":"2024-09-20T22:43:38.129421+0000","last_undegraded":"2024-09-20T22:43:38.129421+0000","last_fullsized":"2024-09-20T22:43:38.129421+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:42:51.354519+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0003748,"stat_sum":{"num_bytes":4985901,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":24,"num_read_kb":22505,"num_write":16,"num_write_kb":5982,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"10","length":"1"},{"start":"14","length":"1"}]},{"pgid":"1.6","version":"77'128","reported_seq":419,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-20T22:43:38.127201+0000","last_change":"2024-09-20T22:43:27.532172+0000","last_active":"2024-09-20T22:43:38.127201+0000","last_peered":"2024-09-20T22:43:38.127201+0000","last_clean":"2024-09-20T22:43:38.127201+0000","last_became_active":"2024-09-20T22:43:19.130331+0000","last_became_peered":"2024-09-20T22:43:19.130331+0000","last_unstale":"2024-09-20T22:43:38.127201+0000","last_undegraded":"2024-09-20T22:43:38.127201+0000","last_fullsized":"2024-09-20T22:43:38.127201+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:43:27.532085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:43:27.532085+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:28.211854+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"74'19","reported_seq":195,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-20T22:43:38.119829+0000","last_change":"2024-09-20T22:43:01.750427+0000","last_active":"2024-09-20T22:43:38.119829+0000","last_peered":"2024-09-20T22:43:38.119829+0000","last_clean":"2024-09-20T22:43:38.119829+0000","last_became_active":"2024-09-20T22:41:40.251866+0000","last_became_peered":"2024-09-20T22:41:40.251866+0000","last_unstale":"2024-09-20T22:43:38.119829+0000","last_undegraded":"2024-09-20T22:43:38.119829+0000","last_fullsized":"2024-09-20T22:43:38.119829+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'19","last_scrub_stamp":"2024-09-20T22:43:01.750344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:43:01.750344+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:06.070428+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":163,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-20T22:43:38.129036+0000","last_change":"2024-09-20T22:43:32.231716+0000","last_active":"2024-09-20T22:43:38.129036+0000","last_peered":"2024-09-20T22:43:38.129036+0000","last_clean":"2024-09-20T22:43:38.129036+0000","last_became_active":"2024-09-20T22:41:38.160832+0000","last_became_peered":"2024-09-20T22:41:38.160832+0000","last_unstale":"2024-09-20T22:43:38.129036+0000","last_undegraded":"2024-09-20T22:43:38.129036+0000","last_fullsized":"2024-09-20T22:43:38.129036+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:32.231644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:43:32.231644+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:44.061026+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":147,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-20T22:43:38.121218+0000","last_change":"2024-09-20T22:41:42.185638+0000","last_active":"2024-09-20T22:43:38.121218+0000","last_peered":"2024-09-20T22:43:38.121218+0000","last_clean":"2024-09-20T22:43:38.121218+0000","last_became_active":"2024-09-20T22:41:42.184412+0000","last_became_peered":"2024-09-20T22:41:42.184412+0000","last_unstale":"2024-09-20T22:43:38.121218+0000","last_undegraded":"2024-09-20T22:43:38.121218+0000","last_fullsized":"2024-09-20T22:43:38.121218+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:42.428286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":143,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-20T22:43:38.126745+0000","last_change":"2024-09-20T22:41:44.830917+0000","last_active":"2024-09-20T22:43:38.126745+0000","last_peered":"2024-09-20T22:43:38.126745+0000","last_clean":"2024-09-20T22:43:38.126745+0000","last_became_active":"2024-09-20T22:41:44.830538+0000","last_became_peered":"2024-09-20T22:41:44.830538+0000","last_unstale":"2024-09-20T22:43:38.126745+0000","last_undegraded":"2024-09-20T22:43:38.126745+0000","last_fullsized":"2024-09-20T22:43:38.126745+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:42.253401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.7","version":"139'103","reported_seq":327,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-20T22:43:38.155561+0000","last_change":"2024-09-20T22:43:38.128943+0000","last_active":"2024-09-20T22:43:38.155561+0000","last_peered":"2024-09-20T22:43:38.155561+0000","last_clean":"2024-09-20T22:43:38.155561+0000","last_became_active":"2024-09-20T22:43:37.132943+0000","last_became_peered":"2024-09-20T22:43:37.132943+0000","last_unstale":"2024-09-20T22:43:38.155561+0000","last_undegraded":"2024-09-20T22:43:38.155561+0000","last_fullsized":"2024-09-20T22:43:38.155561+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:42:51.354519+0000","objects_scrubbed":0,"log_size":103,"log_dups_size":0,"ondisk_log_size":103,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00068798600000000002,"stat_sum":{"num_bytes":11722972,"num_objects":5,"num_object_clones":3,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":16,"num_read_kb":3202,"num_write":35,"num_write_kb":14274,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"10","length":"1"},{"start":"12","length":"3"},{"start":"18","length":"1"}]},{"pgid":"1.1","version":"77'128","reported_seq":412,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-20T22:43:38.121767+0000","last_change":"2024-09-20T22:43:26.761137+0000","last_active":"2024-09-20T22:43:38.121767+0000","last_peered":"2024-09-20T22:43:38.121767+0000","last_clean":"2024-09-20T22:43:38.121767+0000","last_became_active":"2024-09-20T22:43:19.125806+0000","last_became_peered":"2024-09-20T22:43:19.125806+0000","last_unstale":"2024-09-20T22:43:38.121767+0000","last_undegraded":"2024-09-20T22:43:38.121767+0000","last_fullsized":"2024-09-20T22:43:38.121767+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:43:26.761032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:43:26.761032+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:49.343774+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"86'38","reported_seq":242,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-20T22:43:38.128935+0000","last_change":"2024-09-20T22:42:43.935768+0000","last_active":"2024-09-20T22:43:38.128935+0000","last_peered":"2024-09-20T22:43:38.128935+0000","last_clean":"2024-09-20T22:43:38.128935+0000","last_became_active":"2024-09-20T22:41:40.166958+0000","last_became_peered":"2024-09-20T22:41:40.166958+0000","last_unstale":"2024-09-20T22:43:38.128935+0000","last_undegraded":"2024-09-20T22:43:38.128935+0000","last_fullsized":"2024-09-20T22:43:38.128935+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'38","last_scrub_stamp":"2024-09-20T22:42:43.935677+0000","last_deep_scrub":"86'38","last_deep_scrub_stamp":"2024-09-20T22:42:43.935677+0000","last_clean_scrub_stamp":"2024-09-20T22:42:43.935677+0000","objects_scrubbed":10,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:53.919117+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":163,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-20T22:43:38.118988+0000","last_change":"2024-09-20T22:42:37.810752+0000","last_active":"2024-09-20T22:43:38.118988+0000","last_peered":"2024-09-20T22:43:38.118988+0000","last_clean":"2024-09-20T22:43:38.118988+0000","last_became_active":"2024-09-20T22:41:38.152525+0000","last_became_peered":"2024-09-20T22:41:38.152525+0000","last_unstale":"2024-09-20T22:43:38.118988+0000","last_undegraded":"2024-09-20T22:43:38.118988+0000","last_fullsized":"2024-09-20T22:43:38.118988+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:42:37.810656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:42:37.810656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:57.321042+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"69'2","reported_seq":173,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-20T22:43:38.123132+0000","last_change":"2024-09-20T22:43:33.719404+0000","last_active":"2024-09-20T22:43:38.123132+0000","last_peered":"2024-09-20T22:43:38.123132+0000","last_clean":"2024-09-20T22:43:38.123132+0000","last_became_active":"2024-09-20T22:41:37.148479+0000","last_became_peered":"2024-09-20T22:41:37.148479+0000","last_unstale":"2024-09-20T22:43:38.123132+0000","last_undegraded":"2024-09-20T22:43:38.123132+0000","last_fullsized":"2024-09-20T22:43:38.123132+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2024-09-20T22:43:33.719336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:36.130684+0000","last_clean_scrub_stamp":"2024-09-20T22:43:33.719336+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:35.387200+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00087280599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":147,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-20T22:43:38.118563+0000","last_change":"2024-09-20T22:41:42.179979+0000","last_active":"2024-09-20T22:43:38.118563+0000","last_peered":"2024-09-20T22:43:38.118563+0000","last_clean":"2024-09-20T22:43:38.118563+0000","last_became_active":"2024-09-20T22:41:42.179742+0000","last_became_peered":"2024-09-20T22:41:42.179742+0000","last_unstale":"2024-09-20T22:43:38.118563+0000","last_undegraded":"2024-09-20T22:43:38.118563+0000","last_fullsized":"2024-09-20T22:43:38.118563+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":151,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-20T22:43:38.122138+0000","last_change":"2024-09-20T22:43:09.406207+0000","last_active":"2024-09-20T22:43:38.122138+0000","last_peered":"2024-09-20T22:43:38.122138+0000","last_clean":"2024-09-20T22:43:38.122138+0000","last_became_active":"2024-09-20T22:41:44.192128+0000","last_became_peered":"2024-09-20T22:41:44.192128+0000","last_unstale":"2024-09-20T22:43:38.122138+0000","last_undegraded":"2024-09-20T22:43:38.122138+0000","last_fullsized":"2024-09-20T22:43:38.122138+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:09.406141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:43:09.406141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:21.764472+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"138'30","reported_seq":253,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-20T22:43:38.120505+0000","last_change":"2024-09-20T22:43:38.120676+0000","last_active":"2024-09-20T22:43:38.120505+0000","last_peered":"2024-09-20T22:43:38.120505+0000","last_clean":"2024-09-20T22:43:38.120505+0000","last_became_active":"2024-09-20T22:43:37.130799+0000","last_became_peered":"2024-09-20T22:43:37.130799+0000","last_unstale":"2024-09-20T22:43:38.120505+0000","last_undegraded":"2024-09-20T22:43:38.120505+0000","last_fullsized":"2024-09-20T22:43:38.120505+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:42:51.354519+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022498409999999998,"stat_sum":{"num_bytes":4512630,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":3889,"num_write":7,"num_write_kb":3374,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"10","length":"1"},{"start":"12","length":"3"},{"start":"18","length":"1"}]},{"pgid":"1.2","version":"77'128","reported_seq":453,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-20T22:43:38.121352+0000","last_change":"2024-09-20T22:43:25.207458+0000","last_active":"2024-09-20T22:43:38.121352+0000","last_peered":"2024-09-20T22:43:38.121352+0000","last_clean":"2024-09-20T22:43:38.121352+0000","last_became_active":"2024-09-20T22:43:19.122179+0000","last_became_peered":"2024-09-20T22:43:19.122179+0000","last_unstale":"2024-09-20T22:43:38.121352+0000","last_undegraded":"2024-09-20T22:43:38.121352+0000","last_fullsized":"2024-09-20T22:43:38.121352+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:43:25.207385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:43:25.207385+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:50.578381+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"74'13","reported_seq":186,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-20T22:43:38.130678+0000","last_change":"2024-09-20T22:43:31.240331+0000","last_active":"2024-09-20T22:43:38.130678+0000","last_peered":"2024-09-20T22:43:38.130678+0000","last_clean":"2024-09-20T22:43:38.130678+0000","last_became_active":"2024-09-20T22:41:40.169473+0000","last_became_peered":"2024-09-20T22:41:40.169473+0000","last_unstale":"2024-09-20T22:43:38.130678+0000","last_undegraded":"2024-09-20T22:43:38.130678+0000","last_fullsized":"2024-09-20T22:43:38.130678+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'13","last_scrub_stamp":"2024-09-20T22:43:31.240196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:43:31.240196+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:57.654965+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":163,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-20T22:43:38.125914+0000","last_change":"2024-09-20T22:43:12.048243+0000","last_active":"2024-09-20T22:43:38.125914+0000","last_peered":"2024-09-20T22:43:38.125914+0000","last_clean":"2024-09-20T22:43:38.125914+0000","last_became_active":"2024-09-20T22:41:38.156837+0000","last_became_peered":"2024-09-20T22:41:38.156837+0000","last_unstale":"2024-09-20T22:43:38.125914+0000","last_undegraded":"2024-09-20T22:43:38.125914+0000","last_fullsized":"2024-09-20T22:43:38.125914+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:12.048122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:43:12.048122+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:29.457781+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"67'1","reported_seq":173,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-20T22:43:38.128147+0000","last_change":"2024-09-20T22:43:04.330733+0000","last_active":"2024-09-20T22:43:38.128147+0000","last_peered":"2024-09-20T22:43:38.128147+0000","last_clean":"2024-09-20T22:43:38.128147+0000","last_became_active":"2024-09-20T22:41:37.148165+0000","last_became_peered":"2024-09-20T22:41:37.148165+0000","last_unstale":"2024-09-20T22:43:38.128147+0000","last_undegraded":"2024-09-20T22:43:38.128147+0000","last_fullsized":"2024-09-20T22:43:38.128147+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-09-20T22:43:04.330598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:36.130684+0000","last_clean_scrub_stamp":"2024-09-20T22:43:04.330598+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:24.245637+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001301459,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":147,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-20T22:43:38.128872+0000","last_change":"2024-09-20T22:41:42.185412+0000","last_active":"2024-09-20T22:43:38.128872+0000","last_peered":"2024-09-20T22:43:38.128872+0000","last_clean":"2024-09-20T22:43:38.128872+0000","last_became_active":"2024-09-20T22:41:42.185112+0000","last_became_peered":"2024-09-20T22:41:42.185112+0000","last_unstale":"2024-09-20T22:43:38.128872+0000","last_undegraded":"2024-09-20T22:43:38.128872+0000","last_fullsized":"2024-09-20T22:43:38.128872+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:41.391529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":143,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-20T22:43:38.123319+0000","last_change":"2024-09-20T22:41:44.830804+0000","last_active":"2024-09-20T22:43:38.123319+0000","last_peered":"2024-09-20T22:43:38.123319+0000","last_clean":"2024-09-20T22:43:38.123319+0000","last_became_active":"2024-09-20T22:41:44.830515+0000","last_became_peered":"2024-09-20T22:41:44.830515+0000","last_unstale":"2024-09-20T22:43:38.123319+0000","last_undegraded":"2024-09-20T22:43:38.123319+0000","last_fullsized":"2024-09-20T22:43:38.123319+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:41.182665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"98'3","reported_seq":224,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-20T22:43:38.123425+0000","last_change":"2024-09-20T22:43:38.123884+0000","last_active":"2024-09-20T22:43:38.123425+0000","last_peered":"2024-09-20T22:43:38.123425+0000","last_clean":"2024-09-20T22:43:38.123425+0000","last_became_active":"2024-09-20T22:43:37.133478+0000","last_became_peered":"2024-09-20T22:43:37.133478+0000","last_unstale":"2024-09-20T22:43:38.123425+0000","last_undegraded":"2024-09-20T22:43:38.123425+0000","last_fullsized":"2024-09-20T22:43:38.123425+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:42:51.354519+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00077409400000000002,"stat_sum":{"num_bytes":1126052,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":2,"num_write_kb":439,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"10","length":"1"},{"start":"12","length":"3"},{"start":"18","length":"1"}]},{"pgid":"1.3","version":"77'128","reported_seq":412,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-20T22:43:38.133773+0000","last_change":"2024-09-20T22:43:25.244748+0000","last_active":"2024-09-20T22:43:38.133773+0000","last_peered":"2024-09-20T22:43:38.133773+0000","last_clean":"2024-09-20T22:43:38.133773+0000","last_became_active":"2024-09-20T22:43:19.126250+0000","last_became_peered":"2024-09-20T22:43:19.126250+0000","last_unstale":"2024-09-20T22:43:38.133773+0000","last_undegraded":"2024-09-20T22:43:38.133773+0000","last_fullsized":"2024-09-20T22:43:38.133773+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:43:25.244624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:43:25.244624+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:41.317253+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"74'12","reported_seq":197,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-20T22:43:38.120578+0000","last_change":"2024-09-20T22:43:26.720587+0000","last_active":"2024-09-20T22:43:38.120578+0000","last_peered":"2024-09-20T22:43:38.120578+0000","last_clean":"2024-09-20T22:43:38.120578+0000","last_became_active":"2024-09-20T22:43:09.120425+0000","last_became_peered":"2024-09-20T22:43:09.120425+0000","last_unstale":"2024-09-20T22:43:38.120578+0000","last_undegraded":"2024-09-20T22:43:38.120578+0000","last_fullsized":"2024-09-20T22:43:38.120578+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-09-20T22:43:26.720499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:43:26.720499+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:30.458476+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":155,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-20T22:43:38.117716+0000","last_change":"2024-09-20T22:41:38.153448+0000","last_active":"2024-09-20T22:43:38.117716+0000","last_peered":"2024-09-20T22:43:38.117716+0000","last_clean":"2024-09-20T22:43:38.117716+0000","last_became_active":"2024-09-20T22:41:38.153058+0000","last_became_peered":"2024-09-20T22:41:38.153058+0000","last_unstale":"2024-09-20T22:43:38.117716+0000","last_undegraded":"2024-09-20T22:43:38.117716+0000","last_fullsized":"2024-09-20T22:43:38.117716+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"75'5","reported_seq":282,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-20T22:43:38.121931+0000","last_change":"2024-09-20T22:41:40.251762+0000","last_active":"2024-09-20T22:43:38.121931+0000","last_peered":"2024-09-20T22:43:38.121931+0000","last_clean":"2024-09-20T22:43:38.121931+0000","last_became_active":"2024-09-20T22:41:37.148383+0000","last_became_peered":"2024-09-20T22:41:37.148383+0000","last_unstale":"2024-09-20T22:43:38.121931+0000","last_undegraded":"2024-09-20T22:43:38.121931+0000","last_fullsized":"2024-09-20T22:43:38.121931+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:36.130684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:36.130684+0000","last_clean_scrub_stamp":"2024-09-20T22:41:36.130684+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:41.182650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00078631700000000003,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":115,"num_read_kb":110,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":147,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-20T22:43:38.127328+0000","last_change":"2024-09-20T22:41:42.187618+0000","last_active":"2024-09-20T22:43:38.127328+0000","last_peered":"2024-09-20T22:43:38.127328+0000","last_clean":"2024-09-20T22:43:38.127328+0000","last_became_active":"2024-09-20T22:41:42.187346+0000","last_became_peered":"2024-09-20T22:41:42.187346+0000","last_unstale":"2024-09-20T22:43:38.127328+0000","last_undegraded":"2024-09-20T22:43:38.127328+0000","last_fullsized":"2024-09-20T22:43:38.127328+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":151,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-20T22:43:38.128591+0000","last_change":"2024-09-20T22:43:00.128772+0000","last_active":"2024-09-20T22:43:38.128591+0000","last_peered":"2024-09-20T22:43:38.128591+0000","last_clean":"2024-09-20T22:43:38.128591+0000","last_became_active":"2024-09-20T22:41:44.199046+0000","last_became_peered":"2024-09-20T22:41:44.199046+0000","last_unstale":"2024-09-20T22:43:38.128591+0000","last_undegraded":"2024-09-20T22:43:38.128591+0000","last_fullsized":"2024-09-20T22:43:38.128591+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:00.128705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:43:00.128705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:10.682100+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.6","version":"139'48","reported_seq":280,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-20T22:43:38.168999+0000","last_change":"2024-09-20T22:43:38.121170+0000","last_active":"2024-09-20T22:43:38.168999+0000","last_peered":"2024-09-20T22:43:38.168999+0000","last_clean":"2024-09-20T22:43:38.168999+0000","last_became_active":"2024-09-20T22:43:37.136591+0000","last_became_peered":"2024-09-20T22:43:37.136591+0000","last_unstale":"2024-09-20T22:43:38.168999+0000","last_undegraded":"2024-09-20T22:43:38.168999+0000","last_fullsized":"2024-09-20T22:43:38.168999+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:42:51.354519+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00096083099999999999,"stat_sum":{"num_bytes":5307141,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":19,"num_read_kb":9154,"num_write":22,"num_write_kb":6416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"10","length":"1"},{"start":"12","length":"3"},{"start":"18","length":"1"}]},{"pgid":"4.5","version":"74'16","reported_seq":198,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-20T22:43:38.127213+0000","last_change":"2024-09-20T22:41:40.253524+0000","last_active":"2024-09-20T22:43:38.127213+0000","last_peered":"2024-09-20T22:43:38.127213+0000","last_clean":"2024-09-20T22:43:38.127213+0000","last_became_active":"2024-09-20T22:41:40.252933+0000","last_became_peered":"2024-09-20T22:41:40.252933+0000","last_unstale":"2024-09-20T22:43:38.127213+0000","last_undegraded":"2024-09-20T22:43:38.127213+0000","last_fullsized":"2024-09-20T22:43:38.127213+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:42.428256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"67'1","reported_seq":161,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-20T22:43:38.125059+0000","last_change":"2024-09-20T22:41:38.159706+0000","last_active":"2024-09-20T22:43:38.125059+0000","last_peered":"2024-09-20T22:43:38.125059+0000","last_clean":"2024-09-20T22:43:38.125059+0000","last_became_active":"2024-09-20T22:41:38.158917+0000","last_became_peered":"2024-09-20T22:41:38.158917+0000","last_unstale":"2024-09-20T22:43:38.125059+0000","last_undegraded":"2024-09-20T22:43:38.125059+0000","last_fullsized":"2024-09-20T22:43:38.125059+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"77'128","reported_seq":446,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-20T22:43:38.122748+0000","last_change":"2024-09-20T22:43:26.158086+0000","last_active":"2024-09-20T22:43:38.122748+0000","last_peered":"2024-09-20T22:43:38.122748+0000","last_clean":"2024-09-20T22:43:38.122748+0000","last_became_active":"2024-09-20T22:43:19.126925+0000","last_became_peered":"2024-09-20T22:43:19.126925+0000","last_unstale":"2024-09-20T22:43:38.122748+0000","last_undegraded":"2024-09-20T22:43:38.122748+0000","last_fullsized":"2024-09-20T22:43:38.122748+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:43:26.158017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:43:26.158017+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:49.938296+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":147,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-20T22:43:38.123541+0000","last_change":"2024-09-20T22:41:42.185135+0000","last_active":"2024-09-20T22:43:38.123541+0000","last_peered":"2024-09-20T22:43:38.123541+0000","last_clean":"2024-09-20T22:43:38.123541+0000","last_became_active":"2024-09-20T22:41:42.184387+0000","last_became_peered":"2024-09-20T22:41:42.184387+0000","last_unstale":"2024-09-20T22:43:38.123541+0000","last_undegraded":"2024-09-20T22:43:38.123541+0000","last_fullsized":"2024-09-20T22:43:38.123541+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:41.182582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":151,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-20T22:43:38.127160+0000","last_change":"2024-09-20T22:43:02.696182+0000","last_active":"2024-09-20T22:43:38.127160+0000","last_peered":"2024-09-20T22:43:38.127160+0000","last_clean":"2024-09-20T22:43:38.127160+0000","last_became_active":"2024-09-20T22:41:44.190658+0000","last_became_peered":"2024-09-20T22:41:44.190658+0000","last_unstale":"2024-09-20T22:43:38.127160+0000","last_undegraded":"2024-09-20T22:43:38.127160+0000","last_fullsized":"2024-09-20T22:43:38.127160+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:02.696105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:43:02.696105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:13.579400+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"77'128","reported_seq":420,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-20T22:43:38.133046+0000","last_change":"2024-09-20T22:43:29.253689+0000","last_active":"2024-09-20T22:43:38.133046+0000","last_peered":"2024-09-20T22:43:38.133046+0000","last_clean":"2024-09-20T22:43:38.133046+0000","last_became_active":"2024-09-20T22:43:24.124252+0000","last_became_peered":"2024-09-20T22:43:24.124252+0000","last_unstale":"2024-09-20T22:43:38.133046+0000","last_undegraded":"2024-09-20T22:43:38.133046+0000","last_fullsized":"2024-09-20T22:43:38.133046+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:43:29.253585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:43:29.253585+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:44.186807+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.d","version":"137'29","reported_seq":251,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-20T22:43:38.311108+0000","last_change":"2024-09-20T22:43:38.127687+0000","last_active":"2024-09-20T22:43:38.311108+0000","last_peered":"2024-09-20T22:43:38.311108+0000","last_clean":"2024-09-20T22:43:38.311108+0000","last_became_active":"2024-09-20T22:42:52.410381+0000","last_became_peered":"2024-09-20T22:42:52.410381+0000","last_unstale":"2024-09-20T22:43:38.311108+0000","last_undegraded":"2024-09-20T22:43:38.311108+0000","last_fullsized":"2024-09-20T22:43:38.311108+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:42:51.354519+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:57.056540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062103699999999996,"stat_sum":{"num_bytes":7389886,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":23,"num_read_kb":15660,"num_write":16,"num_write_kb":6330,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"10","length":"1"},{"start":"12","length":"3"},{"start":"18","length":"1"}]},{"pgid":"4.e","version":"74'11","reported_seq":170,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-20T22:43:38.124668+0000","last_change":"2024-09-20T22:41:40.169542+0000","last_active":"2024-09-20T22:43:38.124668+0000","last_peered":"2024-09-20T22:43:38.124668+0000","last_clean":"2024-09-20T22:43:38.124668+0000","last_became_active":"2024-09-20T22:41:40.168942+0000","last_became_peered":"2024-09-20T22:41:40.168942+0000","last_unstale":"2024-09-20T22:43:38.124668+0000","last_undegraded":"2024-09-20T22:43:38.124668+0000","last_fullsized":"2024-09-20T22:43:38.124668+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":163,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-20T22:43:38.119387+0000","last_change":"2024-09-20T22:43:31.979253+0000","last_active":"2024-09-20T22:43:38.119387+0000","last_peered":"2024-09-20T22:43:38.119387+0000","last_clean":"2024-09-20T22:43:38.119387+0000","last_became_active":"2024-09-20T22:41:38.161500+0000","last_became_peered":"2024-09-20T22:41:38.161500+0000","last_unstale":"2024-09-20T22:43:38.119387+0000","last_undegraded":"2024-09-20T22:43:38.119387+0000","last_fullsized":"2024-09-20T22:43:38.119387+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:31.979152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:43:31.979152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:45.890889+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":147,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-20T22:43:38.127011+0000","last_change":"2024-09-20T22:41:42.183601+0000","last_active":"2024-09-20T22:43:38.127011+0000","last_peered":"2024-09-20T22:43:38.127011+0000","last_clean":"2024-09-20T22:43:38.127011+0000","last_became_active":"2024-09-20T22:41:42.183302+0000","last_became_peered":"2024-09-20T22:41:42.183302+0000","last_unstale":"2024-09-20T22:43:38.127011+0000","last_undegraded":"2024-09-20T22:43:38.127011+0000","last_fullsized":"2024-09-20T22:43:38.127011+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":143,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-20T22:43:38.120674+0000","last_change":"2024-09-20T22:41:44.832295+0000","last_active":"2024-09-20T22:43:38.120674+0000","last_peered":"2024-09-20T22:43:38.120674+0000","last_clean":"2024-09-20T22:43:38.120674+0000","last_became_active":"2024-09-20T22:41:44.832009+0000","last_became_peered":"2024-09-20T22:41:44.832009+0000","last_unstale":"2024-09-20T22:43:38.120674+0000","last_undegraded":"2024-09-20T22:43:38.120674+0000","last_fullsized":"2024-09-20T22:43:38.120674+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.e","version":"137'24","reported_seq":245,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-20T22:43:38.129894+0000","last_change":"2024-09-20T22:43:38.130216+0000","last_active":"2024-09-20T22:43:38.129894+0000","last_peered":"2024-09-20T22:43:38.129894+0000","last_clean":"2024-09-20T22:43:38.129894+0000","last_became_active":"2024-09-20T22:42:52.405039+0000","last_became_peered":"2024-09-20T22:42:52.405039+0000","last_unstale":"2024-09-20T22:43:38.129894+0000","last_undegraded":"2024-09-20T22:43:38.129894+0000","last_fullsized":"2024-09-20T22:43:38.129894+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:42:51.354519+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:20.421754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001903117,"stat_sum":{"num_bytes":7058841,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":23,"num_read_kb":15328,"num_write":10,"num_write_kb":4406,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"10","length":"1"},{"start":"12","length":"3"},{"start":"18","length":"1"}]},{"pgid":"1.8","version":"77'128","reported_seq":399,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-20T22:43:38.129416+0000","last_change":"2024-09-20T22:43:04.748279+0000","last_active":"2024-09-20T22:43:38.129416+0000","last_peered":"2024-09-20T22:43:38.129416+0000","last_clean":"2024-09-20T22:43:38.129416+0000","last_became_active":"2024-09-20T22:42:52.405594+0000","last_became_peered":"2024-09-20T22:42:52.405594+0000","last_unstale":"2024-09-20T22:43:38.129416+0000","last_undegraded":"2024-09-20T22:43:38.129416+0000","last_fullsized":"2024-09-20T22:43:38.129416+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:43:04.748176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:43:04.748176+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:13.534518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"74'17","reported_seq":187,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-20T22:43:38.123650+0000","last_change":"2024-09-20T22:43:00.910562+0000","last_active":"2024-09-20T22:43:38.123650+0000","last_peered":"2024-09-20T22:43:38.123650+0000","last_clean":"2024-09-20T22:43:38.123650+0000","last_became_active":"2024-09-20T22:41:40.169192+0000","last_became_peered":"2024-09-20T22:41:40.169192+0000","last_unstale":"2024-09-20T22:43:38.123650+0000","last_undegraded":"2024-09-20T22:43:38.123650+0000","last_fullsized":"2024-09-20T22:43:38.123650+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'17","last_scrub_stamp":"2024-09-20T22:43:00.910480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:43:00.910480+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:30.015789+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":163,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-20T22:43:38.126347+0000","last_change":"2024-09-20T22:43:35.492653+0000","last_active":"2024-09-20T22:43:38.126347+0000","last_peered":"2024-09-20T22:43:38.126347+0000","last_clean":"2024-09-20T22:43:38.126347+0000","last_became_active":"2024-09-20T22:41:38.153926+0000","last_became_peered":"2024-09-20T22:41:38.153926+0000","last_unstale":"2024-09-20T22:43:38.126347+0000","last_undegraded":"2024-09-20T22:43:38.126347+0000","last_fullsized":"2024-09-20T22:43:38.126347+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:35.492562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:35.492562+0000","last_clean_scrub_stamp":"2024-09-20T22:43:35.492562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:51.632015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":147,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-20T22:43:38.128681+0000","last_change":"2024-09-20T22:41:42.183482+0000","last_active":"2024-09-20T22:43:38.128681+0000","last_peered":"2024-09-20T22:43:38.128681+0000","last_clean":"2024-09-20T22:43:38.128681+0000","last_became_active":"2024-09-20T22:41:42.183225+0000","last_became_peered":"2024-09-20T22:41:42.183225+0000","last_unstale":"2024-09-20T22:43:38.128681+0000","last_undegraded":"2024-09-20T22:43:38.128681+0000","last_fullsized":"2024-09-20T22:43:38.128681+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:42.253477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":151,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-20T22:43:38.129133+0000","last_change":"2024-09-20T22:42:59.250714+0000","last_active":"2024-09-20T22:43:38.129133+0000","last_peered":"2024-09-20T22:43:38.129133+0000","last_clean":"2024-09-20T22:43:38.129133+0000","last_became_active":"2024-09-20T22:41:44.197561+0000","last_became_peered":"2024-09-20T22:41:44.197561+0000","last_unstale":"2024-09-20T22:43:38.129133+0000","last_undegraded":"2024-09-20T22:43:38.129133+0000","last_fullsized":"2024-09-20T22:43:38.129133+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:42:59.250616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:42:59.250616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:10.878091+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"137'36","reported_seq":255,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-20T22:43:38.120435+0000","last_change":"2024-09-20T22:43:38.120815+0000","last_active":"2024-09-20T22:43:38.120435+0000","last_peered":"2024-09-20T22:43:38.120435+0000","last_clean":"2024-09-20T22:43:38.120435+0000","last_became_active":"2024-09-20T22:42:52.401210+0000","last_became_peered":"2024-09-20T22:42:52.401210+0000","last_unstale":"2024-09-20T22:43:38.120435+0000","last_undegraded":"2024-09-20T22:43:38.120435+0000","last_fullsized":"2024-09-20T22:43:38.120435+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:42:51.354519+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:16.907959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0023399950000000001,"stat_sum":{"num_bytes":7932805,"num_objects":6,"num_object_clones":3,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":2,"num_read":20,"num_read_kb":11559,"num_write":22,"num_write_kb":9368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"10","length":"1"},{"start":"12","length":"3"},{"start":"18","length":"1"}]},{"pgid":"1.9","version":"77'128","reported_seq":440,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-20T22:43:38.121663+0000","last_change":"2024-09-20T22:43:03.239347+0000","last_active":"2024-09-20T22:43:38.121663+0000","last_peered":"2024-09-20T22:43:38.121663+0000","last_clean":"2024-09-20T22:43:38.121663+0000","last_became_active":"2024-09-20T22:42:52.400353+0000","last_became_peered":"2024-09-20T22:42:52.400353+0000","last_unstale":"2024-09-20T22:43:38.121663+0000","last_undegraded":"2024-09-20T22:43:38.121663+0000","last_fullsized":"2024-09-20T22:43:38.121663+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:43:03.239276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:43:03.239276+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:26.152766+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"74'10","reported_seq":166,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-20T22:43:38.119794+0000","last_change":"2024-09-20T22:41:40.162973+0000","last_active":"2024-09-20T22:43:38.119794+0000","last_peered":"2024-09-20T22:43:38.119794+0000","last_clean":"2024-09-20T22:43:38.119794+0000","last_became_active":"2024-09-20T22:41:40.162494+0000","last_became_peered":"2024-09-20T22:41:40.162494+0000","last_unstale":"2024-09-20T22:43:38.119794+0000","last_undegraded":"2024-09-20T22:43:38.119794+0000","last_fullsized":"2024-09-20T22:43:38.119794+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":163,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-20T22:43:38.123257+0000","last_change":"2024-09-20T22:43:05.431444+0000","last_active":"2024-09-20T22:43:38.123257+0000","last_peered":"2024-09-20T22:43:38.123257+0000","last_clean":"2024-09-20T22:43:38.123257+0000","last_became_active":"2024-09-20T22:41:38.154636+0000","last_became_peered":"2024-09-20T22:41:38.154636+0000","last_unstale":"2024-09-20T22:43:38.123257+0000","last_undegraded":"2024-09-20T22:43:38.123257+0000","last_fullsized":"2024-09-20T22:43:38.123257+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:05.431346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:43:05.431346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:30.313458+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"74'8","reported_seq":180,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-20T22:43:38.129791+0000","last_change":"2024-09-20T22:41:42.186528+0000","last_active":"2024-09-20T22:43:38.129791+0000","last_peered":"2024-09-20T22:43:38.129791+0000","last_clean":"2024-09-20T22:43:38.129791+0000","last_became_active":"2024-09-20T22:41:42.185863+0000","last_became_peered":"2024-09-20T22:41:42.185863+0000","last_unstale":"2024-09-20T22:43:38.129791+0000","last_undegraded":"2024-09-20T22:43:38.129791+0000","last_fullsized":"2024-09-20T22:43:38.129791+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:41.391498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":143,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-20T22:43:38.124539+0000","last_change":"2024-09-20T22:41:44.195767+0000","last_active":"2024-09-20T22:43:38.124539+0000","last_peered":"2024-09-20T22:43:38.124539+0000","last_clean":"2024-09-20T22:43:38.124539+0000","last_became_active":"2024-09-20T22:41:44.195160+0000","last_became_peered":"2024-09-20T22:41:44.195160+0000","last_unstale":"2024-09-20T22:43:38.124539+0000","last_undegraded":"2024-09-20T22:43:38.124539+0000","last_fullsized":"2024-09-20T22:43:38.124539+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.e","version":"77'128","reported_seq":337,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-20T22:43:38.124408+0000","last_change":"2024-09-20T22:43:29.902041+0000","last_active":"2024-09-20T22:43:38.124408+0000","last_peered":"2024-09-20T22:43:38.124408+0000","last_clean":"2024-09-20T22:43:38.124408+0000","last_became_active":"2024-09-20T22:43:24.125722+0000","last_became_peered":"2024-09-20T22:43:24.125722+0000","last_unstale":"2024-09-20T22:43:38.124408+0000","last_undegraded":"2024-09-20T22:43:38.124408+0000","last_fullsized":"2024-09-20T22:43:38.124408+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:43:29.901965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:43:29.901965+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:51.235964+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,3],"acting":[4,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.8","version":"136'39","reported_seq":267,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-20T22:43:38.133417+0000","last_change":"2024-09-20T22:43:38.133792+0000","last_active":"2024-09-20T22:43:38.133417+0000","last_peered":"2024-09-20T22:43:38.133417+0000","last_clean":"2024-09-20T22:43:38.133417+0000","last_became_active":"2024-09-20T22:43:37.133672+0000","last_became_peered":"2024-09-20T22:43:37.133672+0000","last_unstale":"2024-09-20T22:43:38.133417+0000","last_undegraded":"2024-09-20T22:43:38.133417+0000","last_fullsized":"2024-09-20T22:43:38.133417+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:42:51.354519+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003109868,"stat_sum":{"num_bytes":5971472,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":15,"num_read_kb":12287,"num_write":14,"num_write_kb":4005,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"10","length":"1"},{"start":"12","length":"3"},{"start":"18","length":"1"}]},{"pgid":"4.b","version":"74'9","reported_seq":175,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-20T22:43:38.117945+0000","last_change":"2024-09-20T22:43:00.749532+0000","last_active":"2024-09-20T22:43:38.117945+0000","last_peered":"2024-09-20T22:43:38.117945+0000","last_clean":"2024-09-20T22:43:38.117945+0000","last_became_active":"2024-09-20T22:41:40.163495+0000","last_became_peered":"2024-09-20T22:41:40.163495+0000","last_unstale":"2024-09-20T22:43:38.117945+0000","last_undegraded":"2024-09-20T22:43:38.117945+0000","last_fullsized":"2024-09-20T22:43:38.117945+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-09-20T22:43:00.749456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:43:00.749456+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:15.372038+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":155,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-20T22:43:38.123536+0000","last_change":"2024-09-20T22:41:38.156470+0000","last_active":"2024-09-20T22:43:38.123536+0000","last_peered":"2024-09-20T22:43:38.123536+0000","last_clean":"2024-09-20T22:43:38.123536+0000","last_became_active":"2024-09-20T22:41:38.155953+0000","last_became_peered":"2024-09-20T22:41:38.155953+0000","last_unstale":"2024-09-20T22:43:38.123536+0000","last_undegraded":"2024-09-20T22:43:38.123536+0000","last_fullsized":"2024-09-20T22:43:38.123536+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":147,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-20T22:43:38.128657+0000","last_change":"2024-09-20T22:41:42.183269+0000","last_active":"2024-09-20T22:43:38.128657+0000","last_peered":"2024-09-20T22:43:38.128657+0000","last_clean":"2024-09-20T22:43:38.128657+0000","last_became_active":"2024-09-20T22:41:42.182999+0000","last_became_peered":"2024-09-20T22:41:42.182999+0000","last_unstale":"2024-09-20T22:43:38.128657+0000","last_undegraded":"2024-09-20T22:43:38.128657+0000","last_fullsized":"2024-09-20T22:43:38.128657+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:41.391479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":143,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-20T22:43:38.117626+0000","last_change":"2024-09-20T22:41:44.196969+0000","last_active":"2024-09-20T22:43:38.117626+0000","last_peered":"2024-09-20T22:43:38.117626+0000","last_clean":"2024-09-20T22:43:38.117626+0000","last_became_active":"2024-09-20T22:41:44.196633+0000","last_became_peered":"2024-09-20T22:41:44.196633+0000","last_unstale":"2024-09-20T22:43:38.117626+0000","last_undegraded":"2024-09-20T22:43:38.117626+0000","last_fullsized":"2024-09-20T22:43:38.117626+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.f","version":"77'128","reported_seq":427,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-20T22:43:38.128477+0000","last_change":"2024-09-20T22:43:30.421734+0000","last_active":"2024-09-20T22:43:38.128477+0000","last_peered":"2024-09-20T22:43:38.128477+0000","last_clean":"2024-09-20T22:43:38.128477+0000","last_became_active":"2024-09-20T22:43:24.125196+0000","last_became_peered":"2024-09-20T22:43:24.125196+0000","last_unstale":"2024-09-20T22:43:38.128477+0000","last_undegraded":"2024-09-20T22:43:38.128477+0000","last_fullsized":"2024-09-20T22:43:38.128477+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:43:30.421648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:43:30.421648+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:35.706758+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,4],"acting":[6,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.9","version":"131'44","reported_seq":271,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-20T22:43:38.117537+0000","last_change":"2024-09-20T22:43:38.118050+0000","last_active":"2024-09-20T22:43:38.117537+0000","last_peered":"2024-09-20T22:43:38.117537+0000","last_clean":"2024-09-20T22:43:38.117537+0000","last_became_active":"2024-09-20T22:43:37.134013+0000","last_became_peered":"2024-09-20T22:43:37.134013+0000","last_unstale":"2024-09-20T22:43:38.117537+0000","last_undegraded":"2024-09-20T22:43:38.117537+0000","last_fullsized":"2024-09-20T22:43:38.117537+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:42:51.354519+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000711176,"stat_sum":{"num_bytes":7168103,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":12,"num_read_kb":6151,"num_write":17,"num_write_kb":7911,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"10","length":"1"},{"start":"12","length":"3"},{"start":"18","length":"1"}]},{"pgid":"4.a","version":"113'28","reported_seq":199,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-20T22:43:38.126071+0000","last_change":"2024-09-20T22:43:12.726018+0000","last_active":"2024-09-20T22:43:38.126071+0000","last_peered":"2024-09-20T22:43:38.126071+0000","last_clean":"2024-09-20T22:43:38.126071+0000","last_became_active":"2024-09-20T22:41:40.252011+0000","last_became_peered":"2024-09-20T22:41:40.252011+0000","last_unstale":"2024-09-20T22:43:38.126071+0000","last_undegraded":"2024-09-20T22:43:38.126071+0000","last_fullsized":"2024-09-20T22:43:38.126071+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'28","last_scrub_stamp":"2024-09-20T22:43:12.725900+0000","last_deep_scrub":"113'28","last_deep_scrub_stamp":"2024-09-20T22:43:12.725900+0000","last_clean_scrub_stamp":"2024-09-20T22:43:12.725900+0000","objects_scrubbed":9,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:21.371299+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":155,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-20T22:43:38.123628+0000","last_change":"2024-09-20T22:41:38.161800+0000","last_active":"2024-09-20T22:43:38.123628+0000","last_peered":"2024-09-20T22:43:38.123628+0000","last_clean":"2024-09-20T22:43:38.123628+0000","last_became_active":"2024-09-20T22:41:38.161524+0000","last_became_peered":"2024-09-20T22:41:38.161524+0000","last_unstale":"2024-09-20T22:43:38.123628+0000","last_undegraded":"2024-09-20T22:43:38.123628+0000","last_fullsized":"2024-09-20T22:43:38.123628+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:41.182679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":147,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-20T22:43:38.129043+0000","last_change":"2024-09-20T22:41:42.184605+0000","last_active":"2024-09-20T22:43:38.129043+0000","last_peered":"2024-09-20T22:43:38.129043+0000","last_clean":"2024-09-20T22:43:38.129043+0000","last_became_active":"2024-09-20T22:41:42.184342+0000","last_became_peered":"2024-09-20T22:41:42.184342+0000","last_unstale":"2024-09-20T22:43:38.129043+0000","last_undegraded":"2024-09-20T22:43:38.129043+0000","last_fullsized":"2024-09-20T22:43:38.129043+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:41.391514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":151,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-20T22:43:38.122643+0000","last_change":"2024-09-20T22:43:02.234978+0000","last_active":"2024-09-20T22:43:38.122643+0000","last_peered":"2024-09-20T22:43:38.122643+0000","last_clean":"2024-09-20T22:43:38.122643+0000","last_became_active":"2024-09-20T22:41:44.199395+0000","last_became_peered":"2024-09-20T22:41:44.199395+0000","last_unstale":"2024-09-20T22:43:38.122643+0000","last_undegraded":"2024-09-20T22:43:38.122643+0000","last_fullsized":"2024-09-20T22:43:38.122643+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:02.234872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:43:02.234872+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:25.214149+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.c","version":"77'128","reported_seq":426,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-20T22:43:38.122691+0000","last_change":"2024-09-20T22:43:30.045082+0000","last_active":"2024-09-20T22:43:38.122691+0000","last_peered":"2024-09-20T22:43:38.122691+0000","last_clean":"2024-09-20T22:43:38.122691+0000","last_became_active":"2024-09-20T22:43:24.130949+0000","last_became_peered":"2024-09-20T22:43:24.130949+0000","last_unstale":"2024-09-20T22:43:38.122691+0000","last_undegraded":"2024-09-20T22:43:38.122691+0000","last_fullsized":"2024-09-20T22:43:38.122691+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:43:30.044975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:43:30.044975+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:53.205672+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"137'38","reported_seq":267,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-20T22:43:38.139986+0000","last_change":"2024-09-20T22:43:38.140160+0000","last_active":"2024-09-20T22:43:38.139986+0000","last_peered":"2024-09-20T22:43:38.139986+0000","last_clean":"2024-09-20T22:43:38.139986+0000","last_became_active":"2024-09-20T22:42:52.405085+0000","last_became_peered":"2024-09-20T22:42:52.405085+0000","last_unstale":"2024-09-20T22:43:38.139986+0000","last_undegraded":"2024-09-20T22:43:38.139986+0000","last_fullsized":"2024-09-20T22:43:38.139986+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:42:51.354519+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:59.494483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015634335999999999,"stat_sum":{"num_bytes":8268945,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":42,"num_read_kb":34394,"num_write":23,"num_write_kb":7915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"10","length":"1"},{"start":"12","length":"3"},{"start":"18","length":"1"}]},{"pgid":"4.9","version":"74'12","reported_seq":182,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-20T22:43:38.119284+0000","last_change":"2024-09-20T22:42:59.862739+0000","last_active":"2024-09-20T22:43:38.119284+0000","last_peered":"2024-09-20T22:43:38.119284+0000","last_clean":"2024-09-20T22:43:38.119284+0000","last_became_active":"2024-09-20T22:41:40.170133+0000","last_became_peered":"2024-09-20T22:41:40.170133+0000","last_unstale":"2024-09-20T22:43:38.119284+0000","last_undegraded":"2024-09-20T22:43:38.119284+0000","last_fullsized":"2024-09-20T22:43:38.119284+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-09-20T22:42:59.862632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:42:59.862632+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:23.948333+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":163,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-20T22:43:38.123152+0000","last_change":"2024-09-20T22:43:05.229655+0000","last_active":"2024-09-20T22:43:38.123152+0000","last_peered":"2024-09-20T22:43:38.123152+0000","last_clean":"2024-09-20T22:43:38.123152+0000","last_became_active":"2024-09-20T22:41:38.163292+0000","last_became_peered":"2024-09-20T22:41:38.163292+0000","last_unstale":"2024-09-20T22:43:38.123152+0000","last_undegraded":"2024-09-20T22:43:38.123152+0000","last_fullsized":"2024-09-20T22:43:38.123152+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:05.229558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:43:05.229558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:22.822578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":155,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-20T22:43:38.128703+0000","last_change":"2024-09-20T22:43:02.326233+0000","last_active":"2024-09-20T22:43:38.128703+0000","last_peered":"2024-09-20T22:43:38.128703+0000","last_clean":"2024-09-20T22:43:38.128703+0000","last_became_active":"2024-09-20T22:41:42.181811+0000","last_became_peered":"2024-09-20T22:41:42.181811+0000","last_unstale":"2024-09-20T22:43:38.128703+0000","last_undegraded":"2024-09-20T22:43:38.128703+0000","last_fullsized":"2024-09-20T22:43:38.128703+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:02.326135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:43:02.326135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:18.212222+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":143,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-20T22:43:38.123019+0000","last_change":"2024-09-20T22:41:44.193432+0000","last_active":"2024-09-20T22:43:38.123019+0000","last_peered":"2024-09-20T22:43:38.123019+0000","last_clean":"2024-09-20T22:43:38.123019+0000","last_became_active":"2024-09-20T22:41:44.192609+0000","last_became_peered":"2024-09-20T22:41:44.192609+0000","last_unstale":"2024-09-20T22:43:38.123019+0000","last_undegraded":"2024-09-20T22:43:38.123019+0000","last_fullsized":"2024-09-20T22:43:38.123019+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.d","version":"77'128","reported_seq":339,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-20T22:43:38.122851+0000","last_change":"2024-09-20T22:43:30.175346+0000","last_active":"2024-09-20T22:43:38.122851+0000","last_peered":"2024-09-20T22:43:38.122851+0000","last_clean":"2024-09-20T22:43:38.122851+0000","last_became_active":"2024-09-20T22:43:24.127144+0000","last_became_peered":"2024-09-20T22:43:24.127144+0000","last_unstale":"2024-09-20T22:43:38.122851+0000","last_undegraded":"2024-09-20T22:43:38.122851+0000","last_fullsized":"2024-09-20T22:43:38.122851+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:43:30.175229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:43:30.175229+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:54.094878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.b","version":"137'42","reported_seq":269,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-20T22:43:38.145991+0000","last_change":"2024-09-20T22:43:38.146150+0000","last_active":"2024-09-20T22:43:38.145991+0000","last_peered":"2024-09-20T22:43:38.145991+0000","last_clean":"2024-09-20T22:43:38.145991+0000","last_became_active":"2024-09-20T22:42:52.409642+0000","last_became_peered":"2024-09-20T22:42:52.409642+0000","last_unstale":"2024-09-20T22:43:38.145991+0000","last_undegraded":"2024-09-20T22:43:38.145991+0000","last_fullsized":"2024-09-20T22:43:38.145991+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:42:51.354519+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:59.484996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.013555152000000001,"stat_sum":{"num_bytes":16571504,"num_objects":6,"num_object_clones":1,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":31,"num_read_kb":14863,"num_write":32,"num_write_kb":13209,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"10","length":"1"},{"start":"12","length":"3"},{"start":"18","length":"1"}]},{"pgid":"4.8","version":"74'15","reported_seq":176,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-20T22:43:38.129549+0000","last_change":"2024-09-20T22:41:40.252427+0000","last_active":"2024-09-20T22:43:38.129549+0000","last_peered":"2024-09-20T22:43:38.129549+0000","last_clean":"2024-09-20T22:43:38.129549+0000","last_became_active":"2024-09-20T22:41:40.252113+0000","last_became_peered":"2024-09-20T22:41:40.252113+0000","last_unstale":"2024-09-20T22:43:38.129549+0000","last_undegraded":"2024-09-20T22:43:38.129549+0000","last_fullsized":"2024-09-20T22:43:38.129549+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'2","reported_seq":167,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-20T22:43:38.122474+0000","last_change":"2024-09-20T22:41:38.163593+0000","last_active":"2024-09-20T22:43:38.122474+0000","last_peered":"2024-09-20T22:43:38.122474+0000","last_clean":"2024-09-20T22:43:38.122474+0000","last_became_active":"2024-09-20T22:41:38.163219+0000","last_became_peered":"2024-09-20T22:41:38.163219+0000","last_unstale":"2024-09-20T22:43:38.122474+0000","last_undegraded":"2024-09-20T22:43:38.122474+0000","last_fullsized":"2024-09-20T22:43:38.122474+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:41.182634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"74'8","reported_seq":188,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-20T22:43:38.121456+0000","last_change":"2024-09-20T22:43:11.266918+0000","last_active":"2024-09-20T22:43:38.121456+0000","last_peered":"2024-09-20T22:43:38.121456+0000","last_clean":"2024-09-20T22:43:38.121456+0000","last_became_active":"2024-09-20T22:41:42.186635+0000","last_became_peered":"2024-09-20T22:41:42.186635+0000","last_unstale":"2024-09-20T22:43:38.121456+0000","last_undegraded":"2024-09-20T22:43:38.121456+0000","last_fullsized":"2024-09-20T22:43:38.121456+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-09-20T22:43:11.266835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:43:11.266835+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:21.514341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":143,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-20T22:43:38.122652+0000","last_change":"2024-09-20T22:41:44.831435+0000","last_active":"2024-09-20T22:43:38.122652+0000","last_peered":"2024-09-20T22:43:38.122652+0000","last_clean":"2024-09-20T22:43:38.122652+0000","last_became_active":"2024-09-20T22:41:44.830556+0000","last_became_peered":"2024-09-20T22:41:44.830556+0000","last_unstale":"2024-09-20T22:43:38.122652+0000","last_undegraded":"2024-09-20T22:43:38.122652+0000","last_fullsized":"2024-09-20T22:43:38.122652+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.14","version":"138'30","reported_seq":2,"reported_epoch":139,"state":"peering","last_fresh":"2024-09-20T22:43:37.178585+0000","last_change":"2024-09-20T22:43:38.620260+0000","last_active":"2024-09-20T22:43:38.620260+0000","last_peered":"2024-09-20T22:43:37.178585+0000","last_clean":"2024-09-20T22:43:37.178585+0000","last_became_active":"2024-09-20T22:43:37.144629+0000","last_became_peered":"2024-09-20T22:43:37.144629+0000","last_unstale":"2024-09-20T22:43:37.178585+0000","last_undegraded":"2024-09-20T22:43:37.178585+0000","last_fullsized":"2024-09-20T22:43:37.178585+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:42:51.354519+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.033531703000000003,"stat_sum":{"num_bytes":4512630,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":2,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":8,"num_read_kb":3890,"num_write":8,"num_write_kb":3375,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"f","length":"2"},{"start":"12","length":"3"}]},{"pgid":"1.12","version":"77'128","reported_seq":337,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-20T22:43:38.122842+0000","last_change":"2024-09-20T22:43:29.735118+0000","last_active":"2024-09-20T22:43:38.122842+0000","last_peered":"2024-09-20T22:43:38.122842+0000","last_clean":"2024-09-20T22:43:38.122842+0000","last_became_active":"2024-09-20T22:43:24.124301+0000","last_became_peered":"2024-09-20T22:43:24.124301+0000","last_unstale":"2024-09-20T22:43:38.122842+0000","last_undegraded":"2024-09-20T22:43:38.122842+0000","last_fullsized":"2024-09-20T22:43:38.122842+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:43:29.735047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:43:29.735047+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:34.983091+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":163,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-20T22:43:38.121119+0000","last_change":"2024-09-20T22:43:01.806383+0000","last_active":"2024-09-20T22:43:38.121119+0000","last_peered":"2024-09-20T22:43:38.121119+0000","last_clean":"2024-09-20T22:43:38.121119+0000","last_became_active":"2024-09-20T22:41:38.159139+0000","last_became_peered":"2024-09-20T22:41:38.159139+0000","last_unstale":"2024-09-20T22:43:38.121119+0000","last_undegraded":"2024-09-20T22:43:38.121119+0000","last_fullsized":"2024-09-20T22:43:38.121119+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:01.806314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:43:01.806314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:16.773475+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"74'6","reported_seq":182,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-20T22:43:38.121608+0000","last_change":"2024-09-20T22:43:09.121694+0000","last_active":"2024-09-20T22:43:38.121608+0000","last_peered":"2024-09-20T22:43:38.121608+0000","last_clean":"2024-09-20T22:43:38.121608+0000","last_became_active":"2024-09-20T22:43:09.116887+0000","last_became_peered":"2024-09-20T22:43:09.116887+0000","last_unstale":"2024-09-20T22:43:38.121608+0000","last_undegraded":"2024-09-20T22:43:38.121608+0000","last_fullsized":"2024-09-20T22:43:38.121608+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'6","last_scrub_stamp":"2024-09-20T22:42:41.469214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:42:41.469214+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:54.350192+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":155,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-20T22:43:38.124399+0000","last_change":"2024-09-20T22:43:09.705715+0000","last_active":"2024-09-20T22:43:38.124399+0000","last_peered":"2024-09-20T22:43:38.124399+0000","last_clean":"2024-09-20T22:43:38.124399+0000","last_became_active":"2024-09-20T22:41:42.184930+0000","last_became_peered":"2024-09-20T22:41:42.184930+0000","last_unstale":"2024-09-20T22:43:38.124399+0000","last_undegraded":"2024-09-20T22:43:38.124399+0000","last_fullsized":"2024-09-20T22:43:38.124399+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:09.705632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:09.705632+0000","last_clean_scrub_stamp":"2024-09-20T22:43:09.705632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:29.284713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"74'1","reported_seq":152,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-20T22:43:38.123192+0000","last_change":"2024-09-20T22:43:33.181024+0000","last_active":"2024-09-20T22:43:38.123192+0000","last_peered":"2024-09-20T22:43:38.123192+0000","last_clean":"2024-09-20T22:43:38.123192+0000","last_became_active":"2024-09-20T22:41:44.831805+0000","last_became_peered":"2024-09-20T22:41:44.831805+0000","last_unstale":"2024-09-20T22:43:38.123192+0000","last_undegraded":"2024-09-20T22:43:38.123192+0000","last_fullsized":"2024-09-20T22:43:38.123192+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-09-20T22:43:33.180910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:43:33.180910+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:57.579073+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.15","version":"98'3","reported_seq":217,"reported_epoch":139,"state":"peering","last_fresh":"2024-09-20T22:43:38.122615+0000","last_change":"2024-09-20T22:43:38.122615+0000","last_active":"2024-09-20T22:43:38.122615+0000","last_peered":"2024-09-20T22:43:37.134855+0000","last_clean":"2024-09-20T22:43:37.134855+0000","last_became_active":"2024-09-20T22:43:37.133556+0000","last_became_peered":"2024-09-20T22:43:37.133556+0000","last_unstale":"2024-09-20T22:43:38.122615+0000","last_undegraded":"2024-09-20T22:43:38.122615+0000","last_fullsized":"2024-09-20T22:43:38.122615+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:42:51.354519+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082552999999999999,"stat_sum":{"num_bytes":1126053,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":439,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"f","length":"2"},{"start":"12","length":"3"}]},{"pgid":"1.13","version":"77'128","reported_seq":420,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-20T22:43:38.132870+0000","last_change":"2024-09-20T22:43:30.247397+0000","last_active":"2024-09-20T22:43:38.132870+0000","last_peered":"2024-09-20T22:43:38.132870+0000","last_clean":"2024-09-20T22:43:38.132870+0000","last_became_active":"2024-09-20T22:43:24.126469+0000","last_became_peered":"2024-09-20T22:43:24.126469+0000","last_unstale":"2024-09-20T22:43:38.132870+0000","last_undegraded":"2024-09-20T22:43:38.132870+0000","last_fullsized":"2024-09-20T22:43:38.132870+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:43:30.247317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:43:30.247317+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:36.498672+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"74'9","reported_seq":175,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-20T22:43:38.120383+0000","last_change":"2024-09-20T22:42:43.666853+0000","last_active":"2024-09-20T22:43:38.120383+0000","last_peered":"2024-09-20T22:43:38.120383+0000","last_clean":"2024-09-20T22:43:38.120383+0000","last_became_active":"2024-09-20T22:41:40.253280+0000","last_became_peered":"2024-09-20T22:41:40.253280+0000","last_unstale":"2024-09-20T22:43:38.120383+0000","last_undegraded":"2024-09-20T22:43:38.120383+0000","last_fullsized":"2024-09-20T22:43:38.120383+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-09-20T22:42:43.666774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:42:43.666774+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:12.132272+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":155,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-20T22:43:38.122704+0000","last_change":"2024-09-20T22:41:38.156773+0000","last_active":"2024-09-20T22:43:38.122704+0000","last_peered":"2024-09-20T22:43:38.122704+0000","last_clean":"2024-09-20T22:43:38.122704+0000","last_became_active":"2024-09-20T22:41:38.156497+0000","last_became_peered":"2024-09-20T22:41:38.156497+0000","last_unstale":"2024-09-20T22:43:38.122704+0000","last_undegraded":"2024-09-20T22:43:38.122704+0000","last_fullsized":"2024-09-20T22:43:38.122704+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:41.182617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":155,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-20T22:43:38.121476+0000","last_change":"2024-09-20T22:43:32.057321+0000","last_active":"2024-09-20T22:43:38.121476+0000","last_peered":"2024-09-20T22:43:38.121476+0000","last_clean":"2024-09-20T22:43:38.121476+0000","last_became_active":"2024-09-20T22:41:42.187706+0000","last_became_peered":"2024-09-20T22:41:42.187706+0000","last_unstale":"2024-09-20T22:43:38.121476+0000","last_undegraded":"2024-09-20T22:43:38.121476+0000","last_fullsized":"2024-09-20T22:43:38.121476+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:32.057240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:43:32.057240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:52.443902+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":151,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-20T22:43:38.128968+0000","last_change":"2024-09-20T22:43:08.232359+0000","last_active":"2024-09-20T22:43:38.128968+0000","last_peered":"2024-09-20T22:43:38.128968+0000","last_clean":"2024-09-20T22:43:38.128968+0000","last_became_active":"2024-09-20T22:41:44.201433+0000","last_became_peered":"2024-09-20T22:41:44.201433+0000","last_unstale":"2024-09-20T22:43:38.128968+0000","last_undegraded":"2024-09-20T22:43:38.128968+0000","last_fullsized":"2024-09-20T22:43:38.128968+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:08.232288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:43:08.232288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:08.925242+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.16","version":"136'41","reported_seq":2,"reported_epoch":139,"state":"peering","last_fresh":"2024-09-20T22:43:37.153320+0000","last_change":"2024-09-20T22:43:38.619667+0000","last_active":"2024-09-20T22:43:38.619667+0000","last_peered":"2024-09-20T22:43:37.153320+0000","last_clean":"2024-09-20T22:43:37.153320+0000","last_became_active":"2024-09-20T22:43:37.146318+0000","last_became_peered":"2024-09-20T22:43:37.146318+0000","last_unstale":"2024-09-20T22:43:37.153320+0000","last_undegraded":"2024-09-20T22:43:37.153320+0000","last_fullsized":"2024-09-20T22:43:37.153320+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:42:51.354519+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0065909540000000004,"stat_sum":{"num_bytes":4741475,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":18,"num_read_kb":9154,"num_write":16,"num_write_kb":4887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"f","length":"2"},{"start":"12","length":"3"}]},{"pgid":"1.10","version":"77'128","reported_seq":337,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-20T22:43:38.129001+0000","last_change":"2024-09-20T22:43:30.720601+0000","last_active":"2024-09-20T22:43:38.129001+0000","last_peered":"2024-09-20T22:43:38.129001+0000","last_clean":"2024-09-20T22:43:38.129001+0000","last_became_active":"2024-09-20T22:43:24.125651+0000","last_became_peered":"2024-09-20T22:43:24.125651+0000","last_unstale":"2024-09-20T22:43:38.129001+0000","last_undegraded":"2024-09-20T22:43:38.129001+0000","last_fullsized":"2024-09-20T22:43:38.129001+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:43:30.720476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:43:30.720476+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:48.337797+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.15","version":"74'9","reported_seq":167,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-20T22:43:38.124081+0000","last_change":"2024-09-20T22:41:40.252840+0000","last_active":"2024-09-20T22:43:38.124081+0000","last_peered":"2024-09-20T22:43:38.124081+0000","last_clean":"2024-09-20T22:43:38.124081+0000","last_became_active":"2024-09-20T22:41:40.252486+0000","last_became_peered":"2024-09-20T22:41:40.252486+0000","last_unstale":"2024-09-20T22:43:38.124081+0000","last_undegraded":"2024-09-20T22:43:38.124081+0000","last_fullsized":"2024-09-20T22:43:38.124081+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":163,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-20T22:43:38.117688+0000","last_change":"2024-09-20T22:43:32.652702+0000","last_active":"2024-09-20T22:43:38.117688+0000","last_peered":"2024-09-20T22:43:38.117688+0000","last_clean":"2024-09-20T22:43:38.117688+0000","last_became_active":"2024-09-20T22:41:38.158146+0000","last_became_peered":"2024-09-20T22:41:38.158146+0000","last_unstale":"2024-09-20T22:43:38.117688+0000","last_undegraded":"2024-09-20T22:43:38.117688+0000","last_fullsized":"2024-09-20T22:43:38.117688+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:32.652587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:43:32.652587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:41.565410+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"74'8","reported_seq":180,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-20T22:43:38.122410+0000","last_change":"2024-09-20T22:41:42.186630+0000","last_active":"2024-09-20T22:43:38.122410+0000","last_peered":"2024-09-20T22:43:38.122410+0000","last_clean":"2024-09-20T22:43:38.122410+0000","last_became_active":"2024-09-20T22:41:42.186352+0000","last_became_peered":"2024-09-20T22:41:42.186352+0000","last_unstale":"2024-09-20T22:43:38.122410+0000","last_undegraded":"2024-09-20T22:43:38.122410+0000","last_fullsized":"2024-09-20T22:43:38.122410+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":143,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-20T22:43:38.125487+0000","last_change":"2024-09-20T22:41:44.194510+0000","last_active":"2024-09-20T22:43:38.125487+0000","last_peered":"2024-09-20T22:43:38.125487+0000","last_clean":"2024-09-20T22:43:38.125487+0000","last_became_active":"2024-09-20T22:41:44.194053+0000","last_became_peered":"2024-09-20T22:41:44.194053+0000","last_unstale":"2024-09-20T22:43:38.125487+0000","last_undegraded":"2024-09-20T22:43:38.125487+0000","last_fullsized":"2024-09-20T22:43:38.125487+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"138'98","reported_seq":2,"reported_epoch":139,"state":"peering","last_fresh":"2024-09-20T22:43:37.136100+0000","last_change":"2024-09-20T22:43:38.620077+0000","last_active":"2024-09-20T22:43:38.620077+0000","last_peered":"2024-09-20T22:43:37.136100+0000","last_clean":"2024-09-20T22:43:37.136100+0000","last_became_active":"2024-09-20T22:43:37.135430+0000","last_became_peered":"2024-09-20T22:43:37.135430+0000","last_unstale":"2024-09-20T22:43:37.136100+0000","last_undegraded":"2024-09-20T22:43:37.136100+0000","last_fullsized":"2024-09-20T22:43:37.136100+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:42:51.354519+0000","objects_scrubbed":0,"log_size":98,"log_dups_size":0,"ondisk_log_size":98,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9561496,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":6,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":15,"num_read_kb":3201,"num_write":31,"num_write_kb":12549,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"f","length":"2"},{"start":"12","length":"3"}]},{"pgid":"1.11","version":"77'128","reported_seq":351,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-20T22:43:38.117799+0000","last_change":"2024-09-20T22:43:30.651129+0000","last_active":"2024-09-20T22:43:38.117799+0000","last_peered":"2024-09-20T22:43:38.117799+0000","last_clean":"2024-09-20T22:43:38.117799+0000","last_became_active":"2024-09-20T22:43:24.128884+0000","last_became_peered":"2024-09-20T22:43:24.128884+0000","last_unstale":"2024-09-20T22:43:38.117799+0000","last_undegraded":"2024-09-20T22:43:38.117799+0000","last_fullsized":"2024-09-20T22:43:38.117799+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:43:30.651054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:43:30.651054+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:37.365013+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"74'10","reported_seq":174,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-20T22:43:38.122849+0000","last_change":"2024-09-20T22:43:11.355914+0000","last_active":"2024-09-20T22:43:38.122849+0000","last_peered":"2024-09-20T22:43:38.122849+0000","last_clean":"2024-09-20T22:43:38.122849+0000","last_became_active":"2024-09-20T22:41:40.253614+0000","last_became_peered":"2024-09-20T22:41:40.253614+0000","last_unstale":"2024-09-20T22:43:38.122849+0000","last_undegraded":"2024-09-20T22:43:38.122849+0000","last_fullsized":"2024-09-20T22:43:38.122849+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-09-20T22:43:11.355829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:43:11.355829+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:38.251196+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":155,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-20T22:43:38.122044+0000","last_change":"2024-09-20T22:41:38.158008+0000","last_active":"2024-09-20T22:43:38.122044+0000","last_peered":"2024-09-20T22:43:38.122044+0000","last_clean":"2024-09-20T22:43:38.122044+0000","last_became_active":"2024-09-20T22:41:38.157647+0000","last_became_peered":"2024-09-20T22:41:38.157647+0000","last_unstale":"2024-09-20T22:43:38.122044+0000","last_undegraded":"2024-09-20T22:43:38.122044+0000","last_fullsized":"2024-09-20T22:43:38.122044+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:41.182567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"74'8","reported_seq":191,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-20T22:43:38.123796+0000","last_change":"2024-09-20T22:42:59.718431+0000","last_active":"2024-09-20T22:43:38.123796+0000","last_peered":"2024-09-20T22:43:38.123796+0000","last_clean":"2024-09-20T22:43:38.123796+0000","last_became_active":"2024-09-20T22:41:42.189451+0000","last_became_peered":"2024-09-20T22:41:42.189451+0000","last_unstale":"2024-09-20T22:43:38.123796+0000","last_undegraded":"2024-09-20T22:43:38.123796+0000","last_fullsized":"2024-09-20T22:43:38.123796+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-09-20T22:42:59.718354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:42:59.718354+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:07.105181+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":151,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-20T22:43:38.119056+0000","last_change":"2024-09-20T22:42:58.680721+0000","last_active":"2024-09-20T22:43:38.119056+0000","last_peered":"2024-09-20T22:43:38.119056+0000","last_clean":"2024-09-20T22:43:38.119056+0000","last_became_active":"2024-09-20T22:41:44.191206+0000","last_became_peered":"2024-09-20T22:41:44.191206+0000","last_unstale":"2024-09-20T22:43:38.119056+0000","last_undegraded":"2024-09-20T22:43:38.119056+0000","last_fullsized":"2024-09-20T22:43:38.119056+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:42:58.680648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:42:58.680648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:22.824715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.10","version":"135'42","reported_seq":244,"reported_epoch":139,"state":"peering","last_fresh":"2024-09-20T22:43:38.119612+0000","last_change":"2024-09-20T22:43:38.119612+0000","last_active":"2024-09-20T22:43:38.119612+0000","last_peered":"2024-09-20T22:43:34.143368+0000","last_clean":"2024-09-20T22:43:34.143368+0000","last_became_active":"2024-09-20T22:42:52.406385+0000","last_became_peered":"2024-09-20T22:42:52.406385+0000","last_unstale":"2024-09-20T22:43:38.119612+0000","last_undegraded":"2024-09-20T22:43:38.119612+0000","last_fullsized":"2024-09-20T22:43:38.119612+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:42:51.354519+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4985901,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":25,"num_read_kb":22506,"num_write":17,"num_write_kb":5982,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"f","length":"2"},{"start":"12","length":"3"}]},{"pgid":"4.13","version":"74'11","reported_seq":178,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-20T22:43:38.118744+0000","last_change":"2024-09-20T22:43:33.923366+0000","last_active":"2024-09-20T22:43:38.118744+0000","last_peered":"2024-09-20T22:43:38.118744+0000","last_clean":"2024-09-20T22:43:38.118744+0000","last_became_active":"2024-09-20T22:41:40.170035+0000","last_became_peered":"2024-09-20T22:41:40.170035+0000","last_unstale":"2024-09-20T22:43:38.118744+0000","last_undegraded":"2024-09-20T22:43:38.118744+0000","last_fullsized":"2024-09-20T22:43:38.118744+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-09-20T22:43:33.923283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:43:33.923283+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:49.315189+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":155,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-20T22:43:38.119986+0000","last_change":"2024-09-20T22:41:38.161385+0000","last_active":"2024-09-20T22:43:38.119986+0000","last_peered":"2024-09-20T22:43:38.119986+0000","last_clean":"2024-09-20T22:43:38.119986+0000","last_became_active":"2024-09-20T22:41:38.158142+0000","last_became_peered":"2024-09-20T22:41:38.158142+0000","last_unstale":"2024-09-20T22:43:38.119986+0000","last_undegraded":"2024-09-20T22:43:38.119986+0000","last_fullsized":"2024-09-20T22:43:38.119986+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":147,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-20T22:43:38.131708+0000","last_change":"2024-09-20T22:41:42.182119+0000","last_active":"2024-09-20T22:43:38.131708+0000","last_peered":"2024-09-20T22:43:38.131708+0000","last_clean":"2024-09-20T22:43:38.131708+0000","last_became_active":"2024-09-20T22:41:42.181736+0000","last_became_peered":"2024-09-20T22:41:42.181736+0000","last_unstale":"2024-09-20T22:43:38.131708+0000","last_undegraded":"2024-09-20T22:43:38.131708+0000","last_fullsized":"2024-09-20T22:43:38.131708+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:42.253461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":143,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-20T22:43:38.121697+0000","last_change":"2024-09-20T22:41:44.193289+0000","last_active":"2024-09-20T22:43:38.121697+0000","last_peered":"2024-09-20T22:43:38.121697+0000","last_clean":"2024-09-20T22:43:38.121697+0000","last_became_active":"2024-09-20T22:41:44.192960+0000","last_became_peered":"2024-09-20T22:41:44.192960+0000","last_unstale":"2024-09-20T22:43:38.121697+0000","last_undegraded":"2024-09-20T22:43:38.121697+0000","last_fullsized":"2024-09-20T22:43:38.121697+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"135'39","reported_seq":2,"reported_epoch":139,"state":"peering","last_fresh":"2024-09-20T22:43:34.139946+0000","last_change":"2024-09-20T22:43:38.620018+0000","last_active":"2024-09-20T22:43:38.620018+0000","last_peered":"2024-09-20T22:43:34.139946+0000","last_clean":"2024-09-20T22:43:34.139946+0000","last_became_active":"2024-09-20T22:42:52.405353+0000","last_became_peered":"2024-09-20T22:42:52.405353+0000","last_unstale":"2024-09-20T22:43:34.139946+0000","last_undegraded":"2024-09-20T22:43:34.139946+0000","last_fullsized":"2024-09-20T22:43:34.139946+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:42:51.354519+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4156378,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":6537,"num_write":13,"num_write_kb":5793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"f","length":"2"},{"start":"12","length":"3"}]},{"pgid":"4.12","version":"74'9","reported_seq":175,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-20T22:43:38.132025+0000","last_change":"2024-09-20T22:43:28.210970+0000","last_active":"2024-09-20T22:43:38.132025+0000","last_peered":"2024-09-20T22:43:38.132025+0000","last_clean":"2024-09-20T22:43:38.132025+0000","last_became_active":"2024-09-20T22:41:40.164796+0000","last_became_peered":"2024-09-20T22:41:40.164796+0000","last_unstale":"2024-09-20T22:43:38.132025+0000","last_undegraded":"2024-09-20T22:43:38.132025+0000","last_fullsized":"2024-09-20T22:43:38.132025+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-09-20T22:43:28.210878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:43:28.210878+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:36.368330+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":166,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-20T22:43:38.123534+0000","last_change":"2024-09-20T22:43:31.138136+0000","last_active":"2024-09-20T22:43:38.123534+0000","last_peered":"2024-09-20T22:43:38.123534+0000","last_clean":"2024-09-20T22:43:38.123534+0000","last_became_active":"2024-09-20T22:41:38.159657+0000","last_became_peered":"2024-09-20T22:41:38.159657+0000","last_unstale":"2024-09-20T22:43:38.123534+0000","last_undegraded":"2024-09-20T22:43:38.123534+0000","last_fullsized":"2024-09-20T22:43:38.123534+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:31.138067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:31.138067+0000","last_clean_scrub_stamp":"2024-09-20T22:43:31.138067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:50.695756+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":147,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-20T22:43:38.123110+0000","last_change":"2024-09-20T22:41:42.183254+0000","last_active":"2024-09-20T22:43:38.123110+0000","last_peered":"2024-09-20T22:43:38.123110+0000","last_clean":"2024-09-20T22:43:38.123110+0000","last_became_active":"2024-09-20T22:41:42.182966+0000","last_became_peered":"2024-09-20T22:41:42.182966+0000","last_unstale":"2024-09-20T22:43:38.123110+0000","last_undegraded":"2024-09-20T22:43:38.123110+0000","last_fullsized":"2024-09-20T22:43:38.123110+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":151,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-20T22:43:38.117074+0000","last_change":"2024-09-20T22:43:04.649683+0000","last_active":"2024-09-20T22:43:38.117074+0000","last_peered":"2024-09-20T22:43:38.117074+0000","last_clean":"2024-09-20T22:43:38.117074+0000","last_became_active":"2024-09-20T22:41:44.194551+0000","last_became_peered":"2024-09-20T22:41:44.194551+0000","last_unstale":"2024-09-20T22:43:38.117074+0000","last_undegraded":"2024-09-20T22:43:38.117074+0000","last_fullsized":"2024-09-20T22:43:38.117074+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:04.649566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:43:04.649566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:09.553594+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.12","version":"138'79","reported_seq":2,"reported_epoch":139,"state":"peering","last_fresh":"2024-09-20T22:43:37.149586+0000","last_change":"2024-09-20T22:43:38.621118+0000","last_active":"2024-09-20T22:43:38.621118+0000","last_peered":"2024-09-20T22:43:37.149586+0000","last_clean":"2024-09-20T22:43:37.149586+0000","last_became_active":"2024-09-20T22:43:37.142808+0000","last_became_peered":"2024-09-20T22:43:37.142808+0000","last_unstale":"2024-09-20T22:43:37.149586+0000","last_undegraded":"2024-09-20T22:43:37.149586+0000","last_fullsized":"2024-09-20T22:43:37.149586+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:42:51.354519+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0054566989999999998,"stat_sum":{"num_bytes":9568236,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":10,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":30,"num_read_kb":20428,"num_write":32,"num_write_kb":11254,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"f","length":"2"},{"start":"14","length":"1"}]},{"pgid":"1.14","version":"77'128","reported_seq":415,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-20T22:43:38.128210+0000","last_change":"2024-09-20T22:43:29.332021+0000","last_active":"2024-09-20T22:43:38.128210+0000","last_peered":"2024-09-20T22:43:38.128210+0000","last_clean":"2024-09-20T22:43:38.128210+0000","last_became_active":"2024-09-20T22:43:24.125656+0000","last_became_peered":"2024-09-20T22:43:24.125656+0000","last_unstale":"2024-09-20T22:43:38.128210+0000","last_undegraded":"2024-09-20T22:43:38.128210+0000","last_fullsized":"2024-09-20T22:43:38.128210+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:43:29.331919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:43:29.331919+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:58.884397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"74'11","reported_seq":170,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-20T22:43:38.121893+0000","last_change":"2024-09-20T22:41:40.252149+0000","last_active":"2024-09-20T22:43:38.121893+0000","last_peered":"2024-09-20T22:43:38.121893+0000","last_clean":"2024-09-20T22:43:38.121893+0000","last_became_active":"2024-09-20T22:41:40.251829+0000","last_became_peered":"2024-09-20T22:41:40.251829+0000","last_unstale":"2024-09-20T22:43:38.121893+0000","last_undegraded":"2024-09-20T22:43:38.121893+0000","last_fullsized":"2024-09-20T22:43:38.121893+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":163,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-20T22:43:38.124484+0000","last_change":"2024-09-20T22:43:06.775869+0000","last_active":"2024-09-20T22:43:38.124484+0000","last_peered":"2024-09-20T22:43:38.124484+0000","last_clean":"2024-09-20T22:43:38.124484+0000","last_became_active":"2024-09-20T22:41:38.158701+0000","last_became_peered":"2024-09-20T22:41:38.158701+0000","last_unstale":"2024-09-20T22:43:38.124484+0000","last_undegraded":"2024-09-20T22:43:38.124484+0000","last_fullsized":"2024-09-20T22:43:38.124484+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:06.775781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:06.775781+0000","last_clean_scrub_stamp":"2024-09-20T22:43:06.775781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:09.259667+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":147,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-20T22:43:38.121420+0000","last_change":"2024-09-20T22:41:42.185991+0000","last_active":"2024-09-20T22:43:38.121420+0000","last_peered":"2024-09-20T22:43:38.121420+0000","last_clean":"2024-09-20T22:43:38.121420+0000","last_became_active":"2024-09-20T22:41:42.185296+0000","last_became_peered":"2024-09-20T22:41:42.185296+0000","last_unstale":"2024-09-20T22:43:38.121420+0000","last_undegraded":"2024-09-20T22:43:38.121420+0000","last_fullsized":"2024-09-20T22:43:38.121420+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:41.182550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":143,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-20T22:43:38.123198+0000","last_change":"2024-09-20T22:41:44.832293+0000","last_active":"2024-09-20T22:43:38.123198+0000","last_peered":"2024-09-20T22:43:38.123198+0000","last_clean":"2024-09-20T22:43:38.123198+0000","last_became_active":"2024-09-20T22:41:44.831967+0000","last_became_peered":"2024-09-20T22:41:44.831967+0000","last_unstale":"2024-09-20T22:43:38.123198+0000","last_undegraded":"2024-09-20T22:43:38.123198+0000","last_fullsized":"2024-09-20T22:43:38.123198+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"135'8","reported_seq":2,"reported_epoch":139,"state":"peering","last_fresh":"2024-09-20T22:43:34.164281+0000","last_change":"2024-09-20T22:43:38.615852+0000","last_active":"2024-09-20T22:43:38.615852+0000","last_peered":"2024-09-20T22:43:34.164281+0000","last_clean":"2024-09-20T22:43:34.164281+0000","last_became_active":"2024-09-20T22:42:52.406126+0000","last_became_peered":"2024-09-20T22:42:52.406126+0000","last_unstale":"2024-09-20T22:43:34.164281+0000","last_undegraded":"2024-09-20T22:43:34.164281+0000","last_fullsized":"2024-09-20T22:43:34.164281+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:42:51.354519+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2441609,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":2,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":5552,"num_write":5,"num_write_kb":1493,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f","length":"2"},{"start":"14","length":"1"}]},{"pgid":"4.10","version":"74'4","reported_seq":157,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-20T22:43:38.120168+0000","last_change":"2024-09-20T22:41:40.166409+0000","last_active":"2024-09-20T22:43:38.120168+0000","last_peered":"2024-09-20T22:43:38.120168+0000","last_clean":"2024-09-20T22:43:38.120168+0000","last_became_active":"2024-09-20T22:41:40.166123+0000","last_became_peered":"2024-09-20T22:41:40.166123+0000","last_unstale":"2024-09-20T22:43:38.120168+0000","last_undegraded":"2024-09-20T22:43:38.120168+0000","last_fullsized":"2024-09-20T22:43:38.120168+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":163,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-20T22:43:38.119483+0000","last_change":"2024-09-20T22:43:03.680628+0000","last_active":"2024-09-20T22:43:38.119483+0000","last_peered":"2024-09-20T22:43:38.119483+0000","last_clean":"2024-09-20T22:43:38.119483+0000","last_became_active":"2024-09-20T22:41:38.158347+0000","last_became_peered":"2024-09-20T22:41:38.158347+0000","last_unstale":"2024-09-20T22:43:38.119483+0000","last_undegraded":"2024-09-20T22:43:38.119483+0000","last_fullsized":"2024-09-20T22:43:38.119483+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:03.680528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:43:03.680528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:18.111001+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":155,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-20T22:43:38.126014+0000","last_change":"2024-09-20T22:43:06.821377+0000","last_active":"2024-09-20T22:43:38.126014+0000","last_peered":"2024-09-20T22:43:38.126014+0000","last_clean":"2024-09-20T22:43:38.126014+0000","last_became_active":"2024-09-20T22:41:42.186068+0000","last_became_peered":"2024-09-20T22:41:42.186068+0000","last_unstale":"2024-09-20T22:43:38.126014+0000","last_undegraded":"2024-09-20T22:43:38.126014+0000","last_fullsized":"2024-09-20T22:43:38.126014+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:06.821310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:43:06.821310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:29.282062+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":151,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-20T22:43:38.121858+0000","last_change":"2024-09-20T22:43:10.233593+0000","last_active":"2024-09-20T22:43:38.121858+0000","last_peered":"2024-09-20T22:43:38.121858+0000","last_clean":"2024-09-20T22:43:38.121858+0000","last_became_active":"2024-09-20T22:41:44.193706+0000","last_became_peered":"2024-09-20T22:41:44.193706+0000","last_unstale":"2024-09-20T22:43:38.121858+0000","last_undegraded":"2024-09-20T22:43:38.121858+0000","last_fullsized":"2024-09-20T22:43:38.121858+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:10.233487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:10.233487+0000","last_clean_scrub_stamp":"2024-09-20T22:43:10.233487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:25.761110+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":143,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-20T22:43:38.131097+0000","last_change":"2024-09-20T22:41:44.198154+0000","last_active":"2024-09-20T22:43:38.131097+0000","last_peered":"2024-09-20T22:43:38.131097+0000","last_clean":"2024-09-20T22:43:38.131097+0000","last_became_active":"2024-09-20T22:41:44.197855+0000","last_became_peered":"2024-09-20T22:41:44.197855+0000","last_unstale":"2024-09-20T22:43:38.131097+0000","last_undegraded":"2024-09-20T22:43:38.131097+0000","last_fullsized":"2024-09-20T22:43:38.131097+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:43:42.253417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":163,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-20T22:43:38.120400+0000","last_change":"2024-09-20T22:43:00.430541+0000","last_active":"2024-09-20T22:43:38.120400+0000","last_peered":"2024-09-20T22:43:38.120400+0000","last_clean":"2024-09-20T22:43:38.120400+0000","last_became_active":"2024-09-20T22:41:38.154703+0000","last_became_peered":"2024-09-20T22:41:38.154703+0000","last_unstale":"2024-09-20T22:43:38.120400+0000","last_undegraded":"2024-09-20T22:43:38.120400+0000","last_fullsized":"2024-09-20T22:43:38.120400+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:00.430469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:43:00.430469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:08.693135+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"74'11","reported_seq":178,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-20T22:43:38.128801+0000","last_change":"2024-09-20T22:43:28.500040+0000","last_active":"2024-09-20T22:43:38.128801+0000","last_peered":"2024-09-20T22:43:38.128801+0000","last_clean":"2024-09-20T22:43:38.128801+0000","last_became_active":"2024-09-20T22:41:40.171256+0000","last_became_peered":"2024-09-20T22:41:40.171256+0000","last_unstale":"2024-09-20T22:43:38.128801+0000","last_undegraded":"2024-09-20T22:43:38.128801+0000","last_fullsized":"2024-09-20T22:43:38.128801+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-09-20T22:43:28.499961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:43:28.499961+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:31.456215+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":147,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-09-20T22:43:38.120189+0000","last_change":"2024-09-20T22:41:42.178413+0000","last_active":"2024-09-20T22:43:38.120189+0000","last_peered":"2024-09-20T22:43:38.120189+0000","last_clean":"2024-09-20T22:43:38.120189+0000","last_became_active":"2024-09-20T22:41:42.178157+0000","last_became_peered":"2024-09-20T22:41:42.178157+0000","last_unstale":"2024-09-20T22:43:38.120189+0000","last_undegraded":"2024-09-20T22:43:38.120189+0000","last_fullsized":"2024-09-20T22:43:38.120189+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":26,"stat_sum":{"num_bytes":185020808,"num_objects":79,"num_object_clones":33,"num_object_copies":237,"num_objects_missing_on_primary":0,"num_objects_missing":31,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":79,"num_whiteouts":7,"num_read":506,"num_read_kb":345149,"num_write":490,"num_write_kb":182085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":381272064,"data_stored":427406034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1127,"ondisk_log_size":1127,"up":78,"acting":78,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":419,"ondisk_log_size":419,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":115,"num_read_kb":110,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2688,"ondisk_log_size":2688,"up":63,"acting":63,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070492,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":71548,"kb_used_data":40352,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93673604,"statfs":{"total":95995035648,"available":95921770496,"internally_reserved":0,"allocated":41320448,"data_stored":47264336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430242,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":90016,"kb_used_data":58828,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93655136,"statfs":{"total":95995035648,"available":95902859264,"internally_reserved":0,"allocated":60239872,"data_stored":68195764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691879,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":55116,"kb_used_data":28084,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93690036,"statfs":{"total":95995035648,"available":95938596864,"internally_reserved":0,"allocated":28758016,"data_stored":29113983,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986220,"num_pgs":76,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":65060,"kb_used_data":33872,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93680092,"statfs":{"total":95995035648,"available":95928414208,"internally_reserved":0,"allocated":34684928,"data_stored":35961482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182450,"num_pgs":80,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":111732,"kb_used_data":80536,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93633420,"statfs":{"total":95995035648,"available":95880622080,"internally_reserved":0,"allocated":82468864,"data_stored":92912776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411383,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":64892,"kb_used_data":33700,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93680260,"statfs":{"total":95995035648,"available":95928586240,"internally_reserved":0,"allocated":34508800,"data_stored":35138484,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574908,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":81080,"kb_used_data":49888,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93664072,"statfs":{"total":95995035648,"available":95912009728,"internally_reserved":0,"allocated":51085312,"data_stored":54087914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738433,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":86212,"kb_used_data":55024,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93658940,"statfs":{"total":95995035648,"available":95906754560,"internally_reserved":0,"allocated":56344576,"data_stored":65035545,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":56856576,"data_stored":67555046,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49631232,"data_stored":53371870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32837632,"data_stored":34160513,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":82702336,"data_stored":95168935,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":33267712,"data_stored":35253753,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":27095040,"data_stored":28142396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":58507264,"data_stored":67152599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":40374272,"data_stored":46600922,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-20T22:43:44.353 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd pool set .mgr pgp_num 21 2024-09-20T22:43:44.354 INFO:tasks.rados.rados.0.smithi084.stdout:276: writing smithi08477203-37 from 618524 to 1329393 tid 1 2024-09-20T22:43:44.355 INFO:tasks.rados.rados.0.smithi084.stdout:276: writing smithi08477203-37 from 1857976 to 2218524 tid 2 2024-09-20T22:43:44.355 INFO:tasks.rados.rados.0.smithi084.stdout:277: read oid 36 snap 21 2024-09-20T22:43:44.355 INFO:tasks.rados.rados.0.smithi084.stdout:277: expect (ObjNum 35 snap 0 seq_num 35) 2024-09-20T22:43:44.355 INFO:tasks.rados.rados.0.smithi084.stdout:278: write oid 13 current snap is 24 2024-09-20T22:43:44.355 INFO:tasks.rados.rados.0.smithi084.stdout:275: finishing rollback tid 0 to smithi08477203-43 2024-09-20T22:43:44.355 INFO:tasks.rados.rados.0.smithi084.stdout:278: seq_num 133 ranges {635331=786118,1916458=318873} 2024-09-20T22:43:44.360 INFO:tasks.rados.rados.0.smithi084.stdout:278: writing smithi08477203-13 from 635331 to 1421449 tid 1 2024-09-20T22:43:44.365 INFO:tasks.rados.rados.0.smithi084.stdout:278: writing smithi08477203-13 from 1916458 to 2235331 tid 2 2024-09-20T22:43:44.365 INFO:tasks.rados.rados.0.smithi084.stdout:279: delete oid 41 current snap is 24 2024-09-20T22:43:44.365 INFO:tasks.rados.rados.0.smithi084.stdout:275: finishing rollback tid 1 to smithi08477203-43 2024-09-20T22:43:44.365 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 43 v 39 (ObjNum 49 snap 0 seq_num 1953066355) dirty dne 2024-09-20T22:43:44.365 INFO:tasks.rados.rados.0.smithi084.stdout:272: finishing copy_from racing read to smithi08477203-12 2024-09-20T22:43:44.365 INFO:tasks.rados.rados.0.smithi084.stdout:276: finishing write tid 1 to smithi08477203-37 2024-09-20T22:43:44.366 INFO:tasks.rados.rados.0.smithi084.stdout:272: finishing copy_from to smithi08477203-12 2024-09-20T22:43:44.366 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 12 v 61 (ObjNum 94 snap 18 seq_num 94) dirty exists 2024-09-20T22:43:44.366 INFO:tasks.rados.rados.0.smithi084.stdout:276: finishing write tid 2 to smithi08477203-37 2024-09-20T22:43:44.366 INFO:tasks.rados.rados.0.smithi084.stdout:276: finishing write tid 3 to smithi08477203-37 2024-09-20T22:43:44.366 INFO:tasks.rados.rados.0.smithi084.stdout:276: finishing write tid 4 to smithi08477203-37 2024-09-20T22:43:44.366 INFO:tasks.rados.rados.0.smithi084.stdout:276: oid 37 updating version 0 to 55 2024-09-20T22:43:44.366 INFO:tasks.rados.rados.0.smithi084.stdout:276: oid 37 updating version 55 to 56 2024-09-20T22:43:44.366 INFO:tasks.rados.rados.0.smithi084.stdout:276: oid 37 updating version 56 to 57 2024-09-20T22:43:44.366 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 37 v 57 (ObjNum 132 snap 24 seq_num 132) dirty exists 2024-09-20T22:43:44.366 INFO:tasks.rados.rados.0.smithi084.stdout:276: left oid 37 (ObjNum 132 snap 24 seq_num 132) 2024-09-20T22:43:44.366 INFO:tasks.rados.rados.0.smithi084.stdout:277: expect (ObjNum 35 snap 0 seq_num 35) 2024-09-20T22:43:44.441 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:43:44.452 INFO:tasks.rados.rados.0.smithi084.stdout:278: finishing write tid 1 to smithi08477203-13 2024-09-20T22:43:44.452 INFO:tasks.rados.rados.0.smithi084.stdout:278: finishing write tid 2 to smithi08477203-13 2024-09-20T22:43:44.452 INFO:tasks.rados.rados.0.smithi084.stdout:278: finishing write tid 3 to smithi08477203-13 2024-09-20T22:43:44.452 INFO:tasks.rados.rados.0.smithi084.stdout:278: finishing write tid 4 to smithi08477203-13 2024-09-20T22:43:44.452 INFO:tasks.rados.rados.0.smithi084.stdout:278: oid 13 updating version 0 to 120 2024-09-20T22:43:44.452 INFO:tasks.rados.rados.0.smithi084.stdout:278: oid 13 updating version 120 to 122 2024-09-20T22:43:44.452 INFO:tasks.rados.rados.0.smithi084.stdout:278: oid 13 version 122 is already newer than 121 2024-09-20T22:43:44.452 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 13 v 122 (ObjNum 133 snap 24 seq_num 133) dirty exists 2024-09-20T22:43:44.452 INFO:tasks.rados.rados.0.smithi084.stdout:278: left oid 13 (ObjNum 133 snap 24 seq_num 133) 2024-09-20T22:43:44.453 INFO:tasks.rados.rados.0.smithi084.stdout:272: done (5 left) 2024-09-20T22:43:44.453 INFO:tasks.rados.rados.0.smithi084.stdout:275: done (4 left) 2024-09-20T22:43:44.453 INFO:tasks.rados.rados.0.smithi084.stdout:276: done (3 left) 2024-09-20T22:43:44.453 INFO:tasks.rados.rados.0.smithi084.stdout:277: done (2 left) 2024-09-20T22:43:44.453 INFO:tasks.rados.rados.0.smithi084.stdout:278: done (1 left) 2024-09-20T22:43:44.453 INFO:tasks.rados.rados.0.smithi084.stdout:279: done (0 left) 2024-09-20T22:43:44.453 INFO:tasks.rados.rados.0.smithi084.stdout:280: copy_from oid 26 from oid 12 current snap is 24 2024-09-20T22:43:44.453 INFO:tasks.rados.rados.0.smithi084.stdout:281: snap_create 2024-09-20T22:43:44.460 INFO:tasks.rados.rados.0.smithi084.stdout:280: finishing copy_from racing read to smithi08477203-26 2024-09-20T22:43:44.462 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:44.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:44.463 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:43:44.464 INFO:tasks.rados.rados.0.smithi084.stdout:280: finishing copy_from to smithi08477203-26 2024-09-20T22:43:44.464 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 26 v 61 (ObjNum 94 snap 18 seq_num 94) dirty exists 2024-09-20T22:43:44.563 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:43:44.586 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:44.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:44.587 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:43:44.592 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:43:44.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:44 smithi084 ceph-mon[29774]: from='client.14967 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T22:43:44.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:44 smithi084 ceph-mon[40024]: from='client.14967 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T22:43:44.687 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:43:44.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:44.709 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:44.709 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:43:44.713 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:44 smithi154 ceph-mon[33264]: from='client.14967 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T22:43:44.811 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:43:44.827 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-20T22:43:44.833 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:44.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:44.834 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:43:44.935 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:43:44.956 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:44.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:44.957 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:43:45.032 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:43:45.058 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:43:45.064 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-20T22:43:45.064 INFO:teuthology.orchestra.run.smithi154.stdout: "size": 20, 2024-09-20T22:43:45.064 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 600, 2024-09-20T22:43:45.064 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [ 2024-09-20T22:43:45.064 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.064 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(osd.0.0:117 7.6 e106/94 7:64bcd2fc:::smithi08477203-20:5 v 106'12, mlcod=105'9)", 2024-09-20T22:43:45.064 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:05.533736+0000", 2024-09-20T22:43:45.064 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 39.527566856999997, 2024-09-20T22:43:45.065 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0038210459999999998, 2024-09-20T22:43:45.065 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:43:45.065 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:43:45.065 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:43:45.065 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:43:45.065 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.065 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:43:45.065 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:05.533736+0000", 2024-09-20T22:43:45.065 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:43:45.065 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.065 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.065 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:43:45.065 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:05.533736+0000", 2024-09-20T22:43:45.065 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:43:45.066 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.066 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.066 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:43:45.066 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:05.533737+0000", 2024-09-20T22:43:45.066 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.5899999999999996e-07 2024-09-20T22:43:45.066 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.066 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.066 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:43:45.066 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:05.533743+0000", 2024-09-20T22:43:45.066 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.8089999999999998e-06 2024-09-20T22:43:45.066 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.067 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.067 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:43:45.067 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:05.533751+0000", 2024-09-20T22:43:45.067 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.1270000000000003e-06 2024-09-20T22:43:45.067 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.067 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.067 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:43:45.067 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:05.533758+0000", 2024-09-20T22:43:45.067 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.7619999999999995e-06 2024-09-20T22:43:45.067 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.067 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.067 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:43:45.067 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:05.533828+0000", 2024-09-20T22:43:45.067 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.9839000000000007e-05 2024-09-20T22:43:45.067 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.068 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.068 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:43:45.068 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:05.533850+0000", 2024-09-20T22:43:45.068 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.1926999999999999e-05 2024-09-20T22:43:45.068 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.068 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.068 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:43:45.068 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:05.537516+0000", 2024-09-20T22:43:45.068 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0036659790000000002 2024-09-20T22:43:45.068 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.068 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.068 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:43:45.068 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:05.537557+0000", 2024-09-20T22:43:45.068 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.1044000000000002e-05 2024-09-20T22:43:45.068 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:43:45.069 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:43:45.069 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:43:45.069 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.069 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.069 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(osd.6.0:128 7.c e106/94 7:35bf785a:::smithi08477203-29:5 v 106'36, mlcod=105'33)", 2024-09-20T22:43:45.069 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:05.534696+0000", 2024-09-20T22:43:45.069 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 39.526606690999998, 2024-09-20T22:43:45.069 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0053454280000000002, 2024-09-20T22:43:45.069 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:43:45.069 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:43:45.069 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:43:45.069 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:43:45.069 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.070 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:43:45.070 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:05.534696+0000", 2024-09-20T22:43:45.070 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:43:45.070 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.070 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.070 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:43:45.070 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:05.534696+0000", 2024-09-20T22:43:45.070 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:43:45.070 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.070 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.070 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:43:45.070 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:05.534697+0000", 2024-09-20T22:43:45.070 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.7000000000000001e-07 2024-09-20T22:43:45.070 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.071 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.071 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:43:45.071 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:05.534701+0000", 2024-09-20T22:43:45.071 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.8489999999999997e-06 2024-09-20T22:43:45.071 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.071 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.071 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:43:45.071 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:05.534712+0000", 2024-09-20T22:43:45.071 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0219e-05 2024-09-20T22:43:45.071 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.071 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.071 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:43:45.071 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:05.534719+0000", 2024-09-20T22:43:45.072 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.1620000000000002e-06 2024-09-20T22:43:45.072 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.072 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.072 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:43:45.072 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:05.534770+0000", 2024-09-20T22:43:45.072 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.1112999999999997e-05 2024-09-20T22:43:45.072 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.072 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.072 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:43:45.072 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:05.534786+0000", 2024-09-20T22:43:45.072 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.5733999999999999e-05 2024-09-20T22:43:45.072 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.072 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.072 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:43:45.073 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:05.540001+0000", 2024-09-20T22:43:45.073 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0052150809999999999 2024-09-20T22:43:45.073 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.073 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.073 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:43:45.073 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:05.540042+0000", 2024-09-20T22:43:45.073 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.1e-05 2024-09-20T22:43:45.073 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:43:45.073 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:43:45.073 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:43:45.073 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.073 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.073 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(osd.6.0:129 7.c e106/94 7:3f0fe232:::smithi08477203-4:5 v 106'38, mlcod=105'33)", 2024-09-20T22:43:45.074 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:05.535085+0000", 2024-09-20T22:43:45.074 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 39.526218364000002, 2024-09-20T22:43:45.074 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0051208909999999998, 2024-09-20T22:43:45.074 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:43:45.074 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:43:45.074 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:43:45.074 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:43:45.074 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.074 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:43:45.074 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:05.535085+0000", 2024-09-20T22:43:45.074 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:43:45.074 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.074 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.075 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:43:45.075 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:05.535085+0000", 2024-09-20T22:43:45.075 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:43:45.075 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.075 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.075 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:43:45.075 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:05.535085+0000", 2024-09-20T22:43:45.075 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.5499999999999998e-07 2024-09-20T22:43:45.075 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.075 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.075 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:43:45.075 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:05.535091+0000", 2024-09-20T22:43:45.075 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.6790000000000002e-06 2024-09-20T22:43:45.076 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.076 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.076 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:43:45.076 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:05.535100+0000", 2024-09-20T22:43:45.076 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.0580000000000004e-06 2024-09-20T22:43:45.076 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.076 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.076 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:43:45.076 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:05.535106+0000", 2024-09-20T22:43:45.077 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.02e-06 2024-09-20T22:43:45.077 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.077 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.077 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:43:45.077 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:05.535342+0000", 2024-09-20T22:43:45.077 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000236085 2024-09-20T22:43:45.077 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.077 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.077 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:43:45.077 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:05.535361+0000", 2024-09-20T22:43:45.077 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.8546000000000002e-05 2024-09-20T22:43:45.077 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.077 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.078 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:43:45.078 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:05.540175+0000", 2024-09-20T22:43:45.078 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0048142239999999998 2024-09-20T22:43:45.078 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.078 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.078 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:43:45.078 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:05.540206+0000", 2024-09-20T22:43:45.078 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.0623999999999997e-05 2024-09-20T22:43:45.078 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:43:45.078 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:43:45.078 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:43:45.078 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.078 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.079 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:353 7.f e108/94 7:f93102e8:::smithi08477203-37:head v 108'11, mlcod=98'9)", 2024-09-20T22:43:45.079 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:07.817412+0000", 2024-09-20T22:43:45.079 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 37.243890524999998, 2024-09-20T22:43:45.079 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.01012206, 2024-09-20T22:43:45.079 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:43:45.079 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:43:45.080 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:43:45.080 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:43:45.081 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.081 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:43:45.081 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:07.817412+0000", 2024-09-20T22:43:45.081 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:43:45.081 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.081 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.081 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:43:45.081 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:07.817412+0000", 2024-09-20T22:43:45.082 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:43:45.082 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.082 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.082 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:43:45.082 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:07.817413+0000", 2024-09-20T22:43:45.082 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.92e-07 2024-09-20T22:43:45.082 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.082 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.082 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:43:45.082 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:07.820706+0000", 2024-09-20T22:43:45.082 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0032935149999999999 2024-09-20T22:43:45.082 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.082 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.083 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:43:45.083 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:07.820720+0000", 2024-09-20T22:43:45.083 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3515e-05 2024-09-20T22:43:45.083 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.083 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.083 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:43:45.083 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:07.820728+0000", 2024-09-20T22:43:45.083 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.1440000000000003e-06 2024-09-20T22:43:45.083 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.083 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.083 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:43:45.083 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:07.820836+0000", 2024-09-20T22:43:45.084 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000108615 2024-09-20T22:43:45.084 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.084 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.085 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:43:45.085 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:07.820866+0000", 2024-09-20T22:43:45.085 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.9285000000000002e-05 2024-09-20T22:43:45.085 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.086 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.086 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:43:45.086 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:07.827497+0000", 2024-09-20T22:43:45.086 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0066308230000000001 2024-09-20T22:43:45.086 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.086 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.086 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:43:45.086 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:07.827535+0000", 2024-09-20T22:43:45.086 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.7970999999999999e-05 2024-09-20T22:43:45.086 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:43:45.086 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:43:45.086 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:43:45.087 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.087 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.087 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:355 7.f e108/94 7:fa55e5ae:::smithi08477203-7:head v 108'12, mlcod=98'9)", 2024-09-20T22:43:45.087 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:07.822311+0000", 2024-09-20T22:43:45.087 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 37.238991611000003, 2024-09-20T22:43:45.087 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0065935890000000004, 2024-09-20T22:43:45.087 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:43:45.087 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:43:45.087 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:43:45.088 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:43:45.088 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.088 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:43:45.088 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:07.822311+0000", 2024-09-20T22:43:45.088 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:43:45.088 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.088 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.088 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:43:45.088 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:07.822311+0000", 2024-09-20T22:43:45.088 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:43:45.089 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.089 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.089 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:43:45.089 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:07.822312+0000", 2024-09-20T22:43:45.089 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.2699999999999998e-07 2024-09-20T22:43:45.089 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.089 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.089 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:43:45.089 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:07.822827+0000", 2024-09-20T22:43:45.089 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00051488700000000001 2024-09-20T22:43:45.089 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.090 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.090 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:43:45.090 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:07.822835+0000", 2024-09-20T22:43:45.090 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.1510000000000004e-06 2024-09-20T22:43:45.090 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.090 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.090 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:43:45.090 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:07.822842+0000", 2024-09-20T22:43:45.090 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.939e-06 2024-09-20T22:43:45.090 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.090 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.090 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:43:45.090 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:07.822890+0000", 2024-09-20T22:43:45.091 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.8155999999999998e-05 2024-09-20T22:43:45.091 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.091 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.091 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:43:45.091 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:07.822912+0000", 2024-09-20T22:43:45.091 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.1945000000000001e-05 2024-09-20T22:43:45.091 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.091 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.091 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:43:45.091 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:07.828813+0000", 2024-09-20T22:43:45.091 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0059010549999999997 2024-09-20T22:43:45.091 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.091 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.091 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:43:45.091 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:07.828905+0000", 2024-09-20T22:43:45.092 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.1629e-05 2024-09-20T22:43:45.092 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:43:45.092 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:43:45.092 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:43:45.092 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.092 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.092 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:540 7.c e128/94 7:39b84aab:::smithi08477203-19:head v 128'55, mlcod=118'52)", 2024-09-20T22:43:45.092 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:27.129500+0000", 2024-09-20T22:43:45.092 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 17.931802993000002, 2024-09-20T22:43:45.092 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0083102890000000002, 2024-09-20T22:43:45.092 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:43:45.092 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:43:45.092 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:43:45.092 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:43:45.093 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.093 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:43:45.093 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.129500+0000", 2024-09-20T22:43:45.093 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:43:45.093 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.093 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.093 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:43:45.093 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.129500+0000", 2024-09-20T22:43:45.093 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:43:45.093 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.093 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.094 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:43:45.094 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.129501+0000", 2024-09-20T22:43:45.094 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.0999999999999997e-07 2024-09-20T22:43:45.094 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.094 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.094 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:43:45.094 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.132397+0000", 2024-09-20T22:43:45.094 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0028959279999999999 2024-09-20T22:43:45.094 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.094 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.094 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:43:45.094 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.132410+0000", 2024-09-20T22:43:45.095 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3395e-05 2024-09-20T22:43:45.095 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.095 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.095 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:43:45.095 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.132423+0000", 2024-09-20T22:43:45.095 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2412000000000001e-05 2024-09-20T22:43:45.095 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.095 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.095 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:43:45.095 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.132484+0000", 2024-09-20T22:43:45.095 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.1002000000000003e-05 2024-09-20T22:43:45.095 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.095 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.096 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:43:45.096 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.132508+0000", 2024-09-20T22:43:45.096 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.4467000000000001e-05 2024-09-20T22:43:45.096 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.096 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.096 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:43:45.096 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.137762+0000", 2024-09-20T22:43:45.096 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.005253789 2024-09-20T22:43:45.096 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.096 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.096 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:43:45.096 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.137810+0000", 2024-09-20T22:43:45.096 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.8486000000000003e-05 2024-09-20T22:43:45.097 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:43:45.097 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:43:45.097 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:43:45.097 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.097 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.097 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:720 7.17 7:ea720b85:::smithi08477203-26:head [append 0~0] snapc 18=[17] ondisk+write+known_if_redirected+supports_pool_eio e143)", 2024-09-20T22:43:45.097 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:42.325164+0000", 2024-09-20T22:43:45.097 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 2.7361394690000003, 2024-09-20T22:43:45.097 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0042406229999999998, 2024-09-20T22:43:45.097 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:43:45.097 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:43:45.097 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:43:45.097 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:43:45.097 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:43:45.098 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:43:45.098 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 720 2024-09-20T22:43:45.098 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.098 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:43:45.098 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.098 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:43:45.098 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.325164+0000", 2024-09-20T22:43:45.098 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:43:45.098 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.098 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.098 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:43:45.098 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.325164+0000", 2024-09-20T22:43:45.098 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:43:45.099 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.099 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.099 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:43:45.099 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.325165+0000", 2024-09-20T22:43:45.099 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4610000000000001e-06 2024-09-20T22:43:45.099 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.099 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.099 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:43:45.099 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.325169+0000", 2024-09-20T22:43:45.099 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.3100000000000002e-06 2024-09-20T22:43:45.099 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.099 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.100 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:43:45.100 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.325179+0000", 2024-09-20T22:43:45.100 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.4129999999999995e-06 2024-09-20T22:43:45.100 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.100 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.100 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:43:45.100 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.325185+0000", 2024-09-20T22:43:45.100 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.5069999999999997e-06 2024-09-20T22:43:45.100 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.100 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.100 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:43:45.100 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.325731+0000", 2024-09-20T22:43:45.100 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00054544999999999995 2024-09-20T22:43:45.100 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.101 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.101 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:43:45.101 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.325826+0000", 2024-09-20T22:43:45.101 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.5024000000000004e-05 2024-09-20T22:43:45.101 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.101 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.101 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 2,5", 2024-09-20T22:43:45.101 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.325934+0000", 2024-09-20T22:43:45.101 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000108031 2024-09-20T22:43:45.101 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.101 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.101 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:43:45.101 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.328968+0000", 2024-09-20T22:43:45.101 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0030346510000000002 2024-09-20T22:43:45.101 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.102 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.102 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:43:45.102 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.329011+0000", 2024-09-20T22:43:45.102 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.2345999999999997e-05 2024-09-20T22:43:45.102 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.102 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.102 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:43:45.102 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.329284+0000", 2024-09-20T22:43:45.102 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00027291300000000001 2024-09-20T22:43:45.102 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.102 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.102 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:43:45.102 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.329354+0000", 2024-09-20T22:43:45.103 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.0370999999999994e-05 2024-09-20T22:43:45.103 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.103 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.103 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:43:45.103 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.329404+0000", 2024-09-20T22:43:45.103 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.0145999999999997e-05 2024-09-20T22:43:45.103 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:43:45.103 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:43:45.103 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:43:45.103 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.103 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.103 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:721 7.17 7:ea720b85:::smithi08477203-26:head [rollback 17] snapc 18=[17] ondisk+write+known_if_redirected+supports_pool_eio e143)", 2024-09-20T22:43:45.104 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:42.325650+0000", 2024-09-20T22:43:45.104 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 2.7356534520000002, 2024-09-20T22:43:45.104 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0043792989999999997, 2024-09-20T22:43:45.104 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:43:45.104 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:43:45.104 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:43:45.104 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:43:45.104 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:43:45.104 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:43:45.104 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 721 2024-09-20T22:43:45.104 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.104 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:43:45.104 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.104 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:43:45.105 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.325650+0000", 2024-09-20T22:43:45.105 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:43:45.105 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.105 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.105 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:43:45.105 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.325650+0000", 2024-09-20T22:43:45.105 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:43:45.105 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.105 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.105 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:43:45.105 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.325650+0000", 2024-09-20T22:43:45.105 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.8899999999999999e-07 2024-09-20T22:43:45.105 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.106 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.106 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:43:45.106 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.325654+0000", 2024-09-20T22:43:45.106 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.9770000000000002e-06 2024-09-20T22:43:45.106 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.106 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.106 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:43:45.106 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.325662+0000", 2024-09-20T22:43:45.107 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.1170000000000004e-06 2024-09-20T22:43:45.107 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.107 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.107 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:43:45.107 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.325669+0000", 2024-09-20T22:43:45.107 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.4520000000000002e-06 2024-09-20T22:43:45.107 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.107 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.107 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:43:45.107 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.326182+0000", 2024-09-20T22:43:45.107 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00051302699999999999 2024-09-20T22:43:45.107 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.107 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.107 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:43:45.108 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.326258+0000", 2024-09-20T22:43:45.108 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.6049000000000003e-05 2024-09-20T22:43:45.108 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.108 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.108 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 2,5", 2024-09-20T22:43:45.108 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.326430+0000", 2024-09-20T22:43:45.108 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00017220399999999999 2024-09-20T22:43:45.108 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.108 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.108 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:43:45.108 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.329051+0000", 2024-09-20T22:43:45.108 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0026208360000000001 2024-09-20T22:43:45.108 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.109 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.109 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:43:45.109 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.329412+0000", 2024-09-20T22:43:45.109 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00036120999999999999 2024-09-20T22:43:45.109 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.109 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.109 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:43:45.109 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.329922+0000", 2024-09-20T22:43:45.109 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00050959200000000001 2024-09-20T22:43:45.109 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.109 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.109 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:43:45.109 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.329982+0000", 2024-09-20T22:43:45.109 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.0324e-05 2024-09-20T22:43:45.110 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.110 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.110 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:43:45.110 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.330029+0000", 2024-09-20T22:43:45.110 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.6822e-05 2024-09-20T22:43:45.110 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:43:45.110 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:43:45.110 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:43:45.110 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.110 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.110 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:722 7.17 7:ea720b85:::smithi08477203-26:head [append 0~0] snapc 18=[17] ondisk+write+known_if_redirected+supports_pool_eio e143)", 2024-09-20T22:43:45.110 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:42.325679+0000", 2024-09-20T22:43:45.110 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 2.7356240089999999, 2024-09-20T22:43:45.110 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0044988160000000001, 2024-09-20T22:43:45.111 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:43:45.111 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:43:45.111 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:43:45.111 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:43:45.111 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:43:45.111 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:43:45.111 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 722 2024-09-20T22:43:45.111 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.111 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:43:45.111 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.111 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:43:45.112 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.325679+0000", 2024-09-20T22:43:45.112 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:43:45.112 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.112 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.112 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:43:45.112 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.325679+0000", 2024-09-20T22:43:45.112 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:43:45.112 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.112 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.112 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:43:45.112 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.325679+0000", 2024-09-20T22:43:45.113 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.4700000000000002e-07 2024-09-20T22:43:45.113 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.113 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.113 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:43:45.113 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.325684+0000", 2024-09-20T22:43:45.113 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.2420000000000002e-06 2024-09-20T22:43:45.113 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.113 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.113 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:43:45.113 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.325689+0000", 2024-09-20T22:43:45.113 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.2769999999999996e-06 2024-09-20T22:43:45.113 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.113 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.114 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:43:45.114 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.325694+0000", 2024-09-20T22:43:45.114 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.0309999999999998e-06 2024-09-20T22:43:45.114 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.114 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.114 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:43:45.114 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.326923+0000", 2024-09-20T22:43:45.114 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0012287890000000001 2024-09-20T22:43:45.114 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.114 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.115 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:43:45.115 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.327002+0000", 2024-09-20T22:43:45.115 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.8745999999999994e-05 2024-09-20T22:43:45.115 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.115 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.115 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 2,5", 2024-09-20T22:43:45.115 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.327140+0000", 2024-09-20T22:43:45.115 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000138252 2024-09-20T22:43:45.115 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.115 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.115 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:43:45.115 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.329089+0000", 2024-09-20T22:43:45.116 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001949501 2024-09-20T22:43:45.116 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.116 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.116 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:43:45.116 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.329418+0000", 2024-09-20T22:43:45.116 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00032850700000000002 2024-09-20T22:43:45.116 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.116 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.116 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:43:45.116 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.330079+0000", 2024-09-20T22:43:45.116 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00066079400000000003 2024-09-20T22:43:45.117 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.117 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.117 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:43:45.117 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.330135+0000", 2024-09-20T22:43:45.117 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.5943999999999998e-05 2024-09-20T22:43:45.117 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.117 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.117 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:43:45.117 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.330178+0000", 2024-09-20T22:43:45.117 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.3285999999999999e-05 2024-09-20T22:43:45.118 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:43:45.118 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:43:45.118 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:43:45.118 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.118 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.118 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(osd.0.0:204 7.13 e143/139 7:cae5792a:::smithi08477203-32:17 v 143'16, mlcod=141'12)", 2024-09-20T22:43:45.118 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:42.330852+0000", 2024-09-20T22:43:45.119 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 2.7304508780000001, 2024-09-20T22:43:45.119 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0062453989999999996, 2024-09-20T22:43:45.119 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:43:45.119 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:43:45.119 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:43:45.119 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:43:45.119 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.119 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:43:45.119 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.330852+0000", 2024-09-20T22:43:45.120 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:43:45.120 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.120 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.120 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:43:45.120 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.330852+0000", 2024-09-20T22:43:45.120 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:43:45.120 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.120 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.120 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:43:45.120 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.330853+0000", 2024-09-20T22:43:45.120 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.9999999999999998e-07 2024-09-20T22:43:45.121 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.121 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.121 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:43:45.121 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.335011+0000", 2024-09-20T22:43:45.121 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0041588730000000004 2024-09-20T22:43:45.121 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.121 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.121 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:43:45.121 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.335022+0000", 2024-09-20T22:43:45.121 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1080000000000001e-05 2024-09-20T22:43:45.121 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.121 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.122 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:43:45.122 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.335030+0000", 2024-09-20T22:43:45.122 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.6399999999999997e-06 2024-09-20T22:43:45.122 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.122 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.122 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:43:45.122 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.335133+0000", 2024-09-20T22:43:45.122 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00010285699999999999 2024-09-20T22:43:45.122 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.122 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.122 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:43:45.122 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.335171+0000", 2024-09-20T22:43:45.123 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.7540999999999999e-05 2024-09-20T22:43:45.123 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.123 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.123 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:43:45.123 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.337037+0000", 2024-09-20T22:43:45.123 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001866726 2024-09-20T22:43:45.123 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.123 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.123 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:43:45.123 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.337098+0000", 2024-09-20T22:43:45.123 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.0282000000000003e-05 2024-09-20T22:43:45.124 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:43:45.124 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:43:45.124 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:43:45.124 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.124 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.124 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:726 7.19 e143/139 7:9eac4560:::smithi08477203-30:head v 143'50, mlcod=143'46)", 2024-09-20T22:43:45.124 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:42.332370+0000", 2024-09-20T22:43:45.124 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 2.7289326549999999, 2024-09-20T22:43:45.124 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.010080573000000001, 2024-09-20T22:43:45.125 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:43:45.125 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:43:45.125 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:43:45.125 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:43:45.125 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.125 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:43:45.125 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.332370+0000", 2024-09-20T22:43:45.125 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:43:45.125 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.125 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.125 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:43:45.125 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.332370+0000", 2024-09-20T22:43:45.125 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:43:45.125 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.126 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.126 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:43:45.126 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.332371+0000", 2024-09-20T22:43:45.126 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.4700000000000001e-07 2024-09-20T22:43:45.126 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.126 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.126 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:43:45.126 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.335262+0000", 2024-09-20T22:43:45.126 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0028912909999999998 2024-09-20T22:43:45.126 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.126 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.126 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:43:45.126 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.335274+0000", 2024-09-20T22:43:45.126 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1809e-05 2024-09-20T22:43:45.127 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.127 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.127 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:43:45.127 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.335282+0000", 2024-09-20T22:43:45.127 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.5859999999999999e-06 2024-09-20T22:43:45.127 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.127 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.127 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:43:45.127 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.335380+0000", 2024-09-20T22:43:45.127 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.8362000000000005e-05 2024-09-20T22:43:45.127 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.127 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.127 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:43:45.127 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.335403+0000", 2024-09-20T22:43:45.128 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.3167999999999999e-05 2024-09-20T22:43:45.128 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.128 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.128 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:43:45.128 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.342398+0000", 2024-09-20T22:43:45.128 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0069946460000000002 2024-09-20T22:43:45.128 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.128 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.128 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:43:45.128 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.342451+0000", 2024-09-20T22:43:45.128 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.2964000000000003e-05 2024-09-20T22:43:45.128 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:43:45.128 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:43:45.129 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:43:45.129 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.129 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.129 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(osd.0.0:205 7.6 e143/137 7:64bcd2fc:::smithi08477203-20:15 v 143'57, mlcod=142'54)", 2024-09-20T22:43:45.129 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:42.339345+0000", 2024-09-20T22:43:45.129 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 2.7219580720000001, 2024-09-20T22:43:45.129 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0034853710000000001, 2024-09-20T22:43:45.129 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:43:45.129 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:43:45.129 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:43:45.129 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:43:45.130 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.130 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:43:45.130 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.339345+0000", 2024-09-20T22:43:45.130 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:43:45.130 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.130 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.130 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:43:45.130 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.339345+0000", 2024-09-20T22:43:45.130 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:43:45.130 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.131 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.131 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:43:45.131 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.339346+0000", 2024-09-20T22:43:45.131 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.1900000000000001e-07 2024-09-20T22:43:45.131 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.131 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.131 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:43:45.131 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.339353+0000", 2024-09-20T22:43:45.131 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.1799999999999999e-06 2024-09-20T22:43:45.131 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.131 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.131 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:43:45.131 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.339363+0000", 2024-09-20T22:43:45.132 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.9180000000000006e-06 2024-09-20T22:43:45.132 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.132 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.132 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:43:45.132 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.339371+0000", 2024-09-20T22:43:45.132 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.8350000000000004e-06 2024-09-20T22:43:45.132 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.132 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.132 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:43:45.132 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.339445+0000", 2024-09-20T22:43:45.132 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.4331000000000004e-05 2024-09-20T22:43:45.132 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.133 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.133 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:43:45.133 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.339472+0000", 2024-09-20T22:43:45.133 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.7027000000000001e-05 2024-09-20T22:43:45.133 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.133 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.133 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:43:45.133 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.342779+0000", 2024-09-20T22:43:45.133 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0033070669999999999 2024-09-20T22:43:45.133 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.133 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.133 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:43:45.133 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.342830+0000", 2024-09-20T22:43:45.134 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.1094000000000002e-05 2024-09-20T22:43:45.134 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:43:45.134 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:43:45.134 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:43:45.134 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.134 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.134 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:741 7.17 7:ed54065f:::smithi08477203-12:head [write 467261~717275,stat] snapc 18=[17] ondisk+write+known_if_redirected+supports_pool_eio e143)", 2024-09-20T22:43:45.134 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:42.611381+0000", 2024-09-20T22:43:45.134 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 2.4499215670000001, 2024-09-20T22:43:45.134 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0069469290000000001, 2024-09-20T22:43:45.134 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:43:45.134 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:43:45.134 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:43:45.135 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:43:45.135 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:43:45.135 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:43:45.135 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 741 2024-09-20T22:43:45.135 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.135 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:43:45.135 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.135 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:43:45.135 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.611381+0000", 2024-09-20T22:43:45.135 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:43:45.135 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.136 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.136 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:43:45.136 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.611381+0000", 2024-09-20T22:43:45.136 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:43:45.136 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.136 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.136 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:43:45.136 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.611384+0000", 2024-09-20T22:43:45.136 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.9299999999999999e-06 2024-09-20T22:43:45.136 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.137 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.137 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:43:45.137 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.614328+0000", 2024-09-20T22:43:45.137 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0029432519999999999 2024-09-20T22:43:45.137 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.137 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.137 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:43:45.137 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.614363+0000", 2024-09-20T22:43:45.137 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.5846000000000002e-05 2024-09-20T22:43:45.137 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.137 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.137 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:43:45.137 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.614386+0000", 2024-09-20T22:43:45.138 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.232e-05 2024-09-20T22:43:45.138 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.138 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.138 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:43:45.138 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.614513+0000", 2024-09-20T22:43:45.138 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00012690600000000001 2024-09-20T22:43:45.138 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.138 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.138 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:43:45.138 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.614677+0000", 2024-09-20T22:43:45.138 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00016395599999999999 2024-09-20T22:43:45.138 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.138 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.138 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 2,5", 2024-09-20T22:43:45.139 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.614842+0000", 2024-09-20T22:43:45.139 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000165444 2024-09-20T22:43:45.139 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.139 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.139 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:43:45.139 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.616278+0000", 2024-09-20T22:43:45.139 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001435525 2024-09-20T22:43:45.139 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.139 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.139 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:43:45.139 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.616548+0000", 2024-09-20T22:43:45.139 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00027032599999999999 2024-09-20T22:43:45.139 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.140 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.140 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:43:45.140 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.618234+0000", 2024-09-20T22:43:45.140 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0016859360000000001 2024-09-20T22:43:45.140 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.140 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.140 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:43:45.140 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.618272+0000", 2024-09-20T22:43:45.140 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.8411999999999997e-05 2024-09-20T22:43:45.140 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.140 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.140 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:43:45.140 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.618328+0000", 2024-09-20T22:43:45.140 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.6076000000000003e-05 2024-09-20T22:43:45.141 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:43:45.141 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:43:45.141 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:43:45.141 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.141 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.141 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:786 7.17 7:ed54065f:::smithi08477203-12:head [copy-from ver 61] snapc 19=[17] ondisk+write+known_if_redirected+supports_pool_eio e145)", 2024-09-20T22:43:45.141 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:44.333196+0000", 2024-09-20T22:43:45.141 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 0.72810745600000004, 2024-09-20T22:43:45.141 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.023328503, 2024-09-20T22:43:45.141 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:43:45.141 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:43:45.141 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:43:45.142 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:43:45.142 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:43:45.142 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:43:45.142 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 786 2024-09-20T22:43:45.142 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.142 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:43:45.142 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.142 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:43:45.142 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.333196+0000", 2024-09-20T22:43:45.142 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:43:45.143 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.143 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.143 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:43:45.143 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.333196+0000", 2024-09-20T22:43:45.143 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:43:45.143 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.143 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.143 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:43:45.143 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.333197+0000", 2024-09-20T22:43:45.143 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.486e-06 2024-09-20T22:43:45.143 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.143 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.143 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:43:45.143 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.333207+0000", 2024-09-20T22:43:45.144 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0307e-05 2024-09-20T22:43:45.144 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.144 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.144 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:43:45.144 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.333220+0000", 2024-09-20T22:43:45.144 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.306e-05 2024-09-20T22:43:45.144 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.144 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.144 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:43:45.144 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.333229+0000", 2024-09-20T22:43:45.144 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.3450000000000006e-06 2024-09-20T22:43:45.144 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.144 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.144 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:43:45.145 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.333292+0000", 2024-09-20T22:43:45.145 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.3137000000000002e-05 2024-09-20T22:43:45.145 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.145 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.145 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op must wait for map", 2024-09-20T22:43:45.145 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.333298+0000", 2024-09-20T22:43:45.145 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.7780000000000004e-06 2024-09-20T22:43:45.145 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.145 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.145 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:43:45.145 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.336639+0000", 2024-09-20T22:43:45.145 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0033414479999999999 2024-09-20T22:43:45.145 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.145 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.146 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:43:45.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:45.147 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:45.147 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:43:45.148 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.336729+0000", 2024-09-20T22:43:45.148 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.9888000000000004e-05 2024-09-20T22:43:45.148 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.148 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.148 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 2,5", 2024-09-20T22:43:45.148 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.344876+0000", 2024-09-20T22:43:45.148 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0081473989999999996 2024-09-20T22:43:45.149 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.149 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.149 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:43:45.149 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.355138+0000", 2024-09-20T22:43:45.149 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.010261462000000001 2024-09-20T22:43:45.149 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.149 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.149 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:43:45.149 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.355972+0000", 2024-09-20T22:43:45.149 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00083377899999999999 2024-09-20T22:43:45.149 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.149 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.149 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:43:45.149 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.356451+0000", 2024-09-20T22:43:45.150 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00047985200000000001 2024-09-20T22:43:45.150 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.150 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.150 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:43:45.150 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.356486+0000", 2024-09-20T22:43:45.150 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.4653000000000002e-05 2024-09-20T22:43:45.150 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.150 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.150 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:43:45.150 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.356524+0000", 2024-09-20T22:43:45.150 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.7908999999999999e-05 2024-09-20T22:43:45.150 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:43:45.150 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:43:45.150 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:43:45.151 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.151 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.151 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:787 7.17 7:ed54065f:::smithi08477203-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e145)", 2024-09-20T22:43:45.151 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:44.333248+0000", 2024-09-20T22:43:45.151 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 0.72805536100000001, 2024-09-20T22:43:45.151 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.016881184, 2024-09-20T22:43:45.151 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:43:45.151 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:43:45.151 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:43:45.151 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:43:45.151 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:43:45.151 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:43:45.151 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 787 2024-09-20T22:43:45.152 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.152 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:43:45.152 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.152 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:43:45.152 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.333248+0000", 2024-09-20T22:43:45.152 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:43:45.152 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.152 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.152 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:43:45.152 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.333248+0000", 2024-09-20T22:43:45.152 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:43:45.152 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.152 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.152 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:43:45.153 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.333248+0000", 2024-09-20T22:43:45.153 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.1399999999999997e-07 2024-09-20T22:43:45.153 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.153 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.153 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:43:45.153 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.333250+0000", 2024-09-20T22:43:45.153 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.8199999999999999e-06 2024-09-20T22:43:45.153 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.153 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.153 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:43:45.153 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.333257+0000", 2024-09-20T22:43:45.153 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.43e-06 2024-09-20T22:43:45.153 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.154 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.154 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:43:45.154 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.333261+0000", 2024-09-20T22:43:45.154 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.3670000000000001e-06 2024-09-20T22:43:45.154 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.154 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.154 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:43:45.154 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.333351+0000", 2024-09-20T22:43:45.154 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.0486000000000001e-05 2024-09-20T22:43:45.154 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.154 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.154 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting_for_map not empty", 2024-09-20T22:43:45.154 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.333357+0000", 2024-09-20T22:43:45.155 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.3520000000000003e-06 2024-09-20T22:43:45.155 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.155 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.155 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:43:45.155 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.336912+0000", 2024-09-20T22:43:45.155 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.003554612 2024-09-20T22:43:45.155 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.155 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.155 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for blocked object", 2024-09-20T22:43:45.155 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.336957+0000", 2024-09-20T22:43:45.155 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.4919000000000001e-05 2024-09-20T22:43:45.155 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.155 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.155 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:43:45.156 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.349989+0000", 2024-09-20T22:43:45.156 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.01303238 2024-09-20T22:43:45.156 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.156 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.156 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:43:45.156 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.350058+0000", 2024-09-20T22:43:45.156 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.8418999999999997e-05 2024-09-20T22:43:45.156 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.156 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.156 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:43:45.156 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.350129+0000", 2024-09-20T22:43:45.156 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.0984999999999999e-05 2024-09-20T22:43:45.156 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:43:45.156 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:43:45.157 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:43:45.157 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.157 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.157 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:792 7.f e145/94 7:f93102e8:::smithi08477203-37:head v 145'55, mlcod=143'52)", 2024-09-20T22:43:45.157 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:44.347269+0000", 2024-09-20T22:43:45.157 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 0.71403373400000003, 2024-09-20T22:43:45.157 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0089841160000000003, 2024-09-20T22:43:45.157 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:43:45.157 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:43:45.157 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:43:45.157 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:43:45.157 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.158 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:43:45.158 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.347269+0000", 2024-09-20T22:43:45.158 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:43:45.158 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.158 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.158 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:43:45.158 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.347269+0000", 2024-09-20T22:43:45.158 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:43:45.158 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.158 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.158 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:43:45.158 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.347270+0000", 2024-09-20T22:43:45.158 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.99e-07 2024-09-20T22:43:45.158 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.159 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.159 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:43:45.159 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.348569+0000", 2024-09-20T22:43:45.159 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0012989989999999999 2024-09-20T22:43:45.159 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.159 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.159 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:43:45.159 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.348599+0000", 2024-09-20T22:43:45.159 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.0213e-05 2024-09-20T22:43:45.159 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.159 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.160 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:43:45.160 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.348613+0000", 2024-09-20T22:43:45.160 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3475999999999999e-05 2024-09-20T22:43:45.160 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.160 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.160 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:43:45.160 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.348701+0000", 2024-09-20T22:43:45.160 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.8243000000000002e-05 2024-09-20T22:43:45.160 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.161 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.161 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:43:45.161 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.348728+0000", 2024-09-20T22:43:45.161 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.7226e-05 2024-09-20T22:43:45.161 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.161 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.161 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:43:45.161 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.356216+0000", 2024-09-20T22:43:45.161 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0074874210000000002 2024-09-20T22:43:45.161 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.161 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.161 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:43:45.162 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.356253+0000", 2024-09-20T22:43:45.162 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.7839000000000001e-05 2024-09-20T22:43:45.162 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:43:45.162 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:43:45.162 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:43:45.162 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.162 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.162 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:793 7.f e145/94 7:f93102e8:::smithi08477203-37:head v 145'56, mlcod=143'52)", 2024-09-20T22:43:45.162 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:44.352409+0000", 2024-09-20T22:43:45.162 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 0.70889385900000002, 2024-09-20T22:43:45.162 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0041361260000000004, 2024-09-20T22:43:45.162 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:43:45.162 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:43:45.163 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:43:45.163 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:43:45.163 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.163 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:43:45.163 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.352409+0000", 2024-09-20T22:43:45.163 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:43:45.163 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.163 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.163 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:43:45.163 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.352409+0000", 2024-09-20T22:43:45.163 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:43:45.163 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.163 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.163 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:43:45.164 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.352410+0000", 2024-09-20T22:43:45.164 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2100000000000001e-06 2024-09-20T22:43:45.164 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.164 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.164 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:43:45.164 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.352724+0000", 2024-09-20T22:43:45.164 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000313701 2024-09-20T22:43:45.164 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.164 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.164 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:43:45.164 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.352736+0000", 2024-09-20T22:43:45.164 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2332e-05 2024-09-20T22:43:45.164 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.164 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.165 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:43:45.165 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.352744+0000", 2024-09-20T22:43:45.165 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.6699999999999994e-06 2024-09-20T22:43:45.165 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.165 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.165 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:43:45.165 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.352799+0000", 2024-09-20T22:43:45.165 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.4574000000000003e-05 2024-09-20T22:43:45.165 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.165 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.165 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:43:45.165 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.352821+0000", 2024-09-20T22:43:45.165 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.2478000000000001e-05 2024-09-20T22:43:45.166 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.166 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.166 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:43:45.166 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.356527+0000", 2024-09-20T22:43:45.166 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.003705982 2024-09-20T22:43:45.166 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.166 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.166 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:43:45.166 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.356545+0000", 2024-09-20T22:43:45.166 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.8179e-05 2024-09-20T22:43:45.166 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:43:45.166 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:43:45.166 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:43:45.166 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.167 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.167 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:794 7.f e145/94 7:f93102e8:::smithi08477203-37:head v 145'57, mlcod=143'52)", 2024-09-20T22:43:45.167 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:44.353178+0000", 2024-09-20T22:43:45.167 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 0.70812483000000004, 2024-09-20T22:43:45.167 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0033818889999999999, 2024-09-20T22:43:45.167 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:43:45.167 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:43:45.167 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:43:45.167 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:43:45.167 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.167 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:43:45.167 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.353178+0000", 2024-09-20T22:43:45.167 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:43:45.168 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.168 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.168 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:43:45.168 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.353178+0000", 2024-09-20T22:43:45.168 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:43:45.168 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.168 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.168 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:43:45.168 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.353179+0000", 2024-09-20T22:43:45.168 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.54e-07 2024-09-20T22:43:45.168 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.168 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.169 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:43:45.169 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.353185+0000", 2024-09-20T22:43:45.169 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.2230000000000004e-06 2024-09-20T22:43:45.169 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.169 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.169 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:43:45.169 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.353194+0000", 2024-09-20T22:43:45.169 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.183e-06 2024-09-20T22:43:45.169 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.169 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.169 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:43:45.169 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.353202+0000", 2024-09-20T22:43:45.169 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.3509999999999996e-06 2024-09-20T22:43:45.170 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.170 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.170 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:43:45.170 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.353257+0000", 2024-09-20T22:43:45.170 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.4780000000000001e-05 2024-09-20T22:43:45.170 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.170 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.170 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:43:45.170 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.353280+0000", 2024-09-20T22:43:45.170 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.3604e-05 2024-09-20T22:43:45.170 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.170 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.170 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:43:45.170 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.356547+0000", 2024-09-20T22:43:45.170 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0032669779999999998 2024-09-20T22:43:45.171 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.171 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.171 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:43:45.171 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.356560+0000", 2024-09-20T22:43:45.171 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2916e-05 2024-09-20T22:43:45.171 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:43:45.171 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:43:45.171 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:43:45.171 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.171 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.171 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:804 7.17 7:ea720b85:::smithi08477203-26:head [copy-from ver 61] snapc 19=[17] ondisk+write+known_if_redirected+supports_pool_eio e145)", 2024-09-20T22:43:45.171 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:44.451241+0000", 2024-09-20T22:43:45.171 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 0.61006152599999997, 2024-09-20T22:43:45.172 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.012865619999999999, 2024-09-20T22:43:45.172 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:43:45.172 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:43:45.172 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:43:45.172 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:43:45.172 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:43:45.172 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:43:45.172 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 804 2024-09-20T22:43:45.172 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.172 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:43:45.172 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.172 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:43:45.172 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.451241+0000", 2024-09-20T22:43:45.172 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:43:45.173 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.173 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.173 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:43:45.173 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.451241+0000", 2024-09-20T22:43:45.173 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:43:45.173 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.173 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.173 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:43:45.173 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.451244+0000", 2024-09-20T22:43:45.173 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.2689999999999998e-06 2024-09-20T22:43:45.173 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.173 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.173 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:43:45.173 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.451255+0000", 2024-09-20T22:43:45.174 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1151e-05 2024-09-20T22:43:45.174 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.174 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.174 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:43:45.174 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.451274+0000", 2024-09-20T22:43:45.174 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9389999999999999e-05 2024-09-20T22:43:45.174 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.174 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.174 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:43:45.174 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.451286+0000", 2024-09-20T22:43:45.174 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1703e-05 2024-09-20T22:43:45.174 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.174 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.175 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:43:45.175 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.451361+0000", 2024-09-20T22:43:45.175 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.5310999999999998e-05 2024-09-20T22:43:45.175 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.175 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.175 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:43:45.175 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.451462+0000", 2024-09-20T22:43:45.175 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000100809 2024-09-20T22:43:45.175 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.175 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.175 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 2,5", 2024-09-20T22:43:45.175 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.456603+0000", 2024-09-20T22:43:45.175 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0051411169999999997 2024-09-20T22:43:45.176 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.176 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.176 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:43:45.176 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.463017+0000", 2024-09-20T22:43:45.176 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.006413307 2024-09-20T22:43:45.176 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.176 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.176 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:43:45.176 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.463754+0000", 2024-09-20T22:43:45.176 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00073758000000000005 2024-09-20T22:43:45.176 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.176 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.177 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:43:45.177 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.464036+0000", 2024-09-20T22:43:45.177 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00028171000000000001 2024-09-20T22:43:45.177 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.177 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.177 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:43:45.177 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.464076+0000", 2024-09-20T22:43:45.177 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.9736e-05 2024-09-20T22:43:45.177 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.177 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.177 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:43:45.177 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.464107+0000", 2024-09-20T22:43:45.178 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.1537e-05 2024-09-20T22:43:45.178 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:43:45.178 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:43:45.178 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:43:45.178 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.178 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.178 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:805 7.17 7:ea720b85:::smithi08477203-26:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e145)", 2024-09-20T22:43:45.178 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:44.451315+0000", 2024-09-20T22:43:45.178 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 0.60998772099999998, 2024-09-20T22:43:45.179 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0083409929999999997, 2024-09-20T22:43:45.179 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:43:45.179 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:43:45.179 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:43:45.179 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:43:45.179 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:43:45.179 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:43:45.179 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 805 2024-09-20T22:43:45.179 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.179 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:43:45.179 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.179 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:43:45.179 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.451315+0000", 2024-09-20T22:43:45.179 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:43:45.180 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.180 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.180 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:43:45.180 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.451315+0000", 2024-09-20T22:43:45.180 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:43:45.180 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.180 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.180 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:43:45.180 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.451316+0000", 2024-09-20T22:43:45.180 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.7999999999999995e-07 2024-09-20T22:43:45.181 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.181 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.181 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:43:45.181 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.451318+0000", 2024-09-20T22:43:45.181 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.5610000000000001e-06 2024-09-20T22:43:45.181 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.181 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.181 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:43:45.181 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.451324+0000", 2024-09-20T22:43:45.181 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.9429999999999997e-06 2024-09-20T22:43:45.181 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.181 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.182 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:43:45.182 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.451330+0000", 2024-09-20T22:43:45.182 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.6110000000000001e-06 2024-09-20T22:43:45.182 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.182 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.182 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:43:45.182 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.451581+0000", 2024-09-20T22:43:45.182 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00025096599999999998 2024-09-20T22:43:45.182 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.182 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.182 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for blocked object", 2024-09-20T22:43:45.182 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.451609+0000", 2024-09-20T22:43:45.183 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.7875e-05 2024-09-20T22:43:45.183 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.183 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.183 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:43:45.183 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.459506+0000", 2024-09-20T22:43:45.183 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0078970740000000005 2024-09-20T22:43:45.183 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.183 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.183 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:43:45.183 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.459576+0000", 2024-09-20T22:43:45.183 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.0266999999999997e-05 2024-09-20T22:43:45.184 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:43:45.184 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:43:45.184 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:43:45.184 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.459656+0000", 2024-09-20T22:43:45.184 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.0116000000000003e-05 2024-09-20T22:43:45.184 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:43:45.184 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:43:45.184 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:43:45.184 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:43:45.184 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:43:45.185 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-20T22:43:45.248 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:43:45.270 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:45.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:45.271 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:43:45.330 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is unset 2024-09-20T22:43:45.337 INFO:tasks.rados.rados.0.smithi084.stdout:280: done (1 left) 2024-09-20T22:43:45.338 INFO:tasks.rados.rados.0.smithi084.stdout:281: done (0 left) 2024-09-20T22:43:45.338 INFO:tasks.rados.rados.0.smithi084.stdout:282: delete oid 50 current snap is 25 2024-09-20T22:43:45.344 INFO:tasks.rados.rados.0.smithi084.stdout:282: done (0 left) 2024-09-20T22:43:45.345 INFO:tasks.rados.rados.0.smithi084.stdout:283: write (excl) oid 28 current snap is 25 2024-09-20T22:43:45.345 INFO:tasks.rados.rados.0.smithi084.stdout:283: seq_num 136 ranges {685752=611865,1975551=310201} 2024-09-20T22:43:45.351 INFO:tasks.rados.rados.0.smithi084.stdout:283: writing smithi08477203-28 from 685752 to 1297617 tid 1 2024-09-20T22:43:45.357 INFO:tasks.rados.rados.0.smithi084.stdout:283: writing smithi08477203-28 from 1975551 to 2285752 tid 2 2024-09-20T22:43:45.357 INFO:tasks.rados.rados.0.smithi084.stdout:284: delete oid 26 current snap is 25 2024-09-20T22:43:45.357 INFO:tasks.rados.rados.0.smithi084.stdout:283: finishing write tid 1 to smithi08477203-28 2024-09-20T22:43:45.358 INFO:tasks.rados.rados.0.smithi084.stdout:284: done (1 left) 2024-09-20T22:43:45.359 INFO:tasks.rados.rados.0.smithi084.stdout:285: snap_remove snap 21 2024-09-20T22:43:45.372 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:43:45.393 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:45.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:45.394 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:43:45.418 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:45 smithi084 ceph-mon[40024]: pgmap v118: 178 pgs: 5 active+clean+snaptrim, 173 active+clean; 149 MiB data, 576 MiB used, 715 GiB / 715 GiB avail; 16 MiB/s rd, 5.5 MiB/s wr, 30 op/s; 9.6 MiB/s, 8 objects/s recovering 2024-09-20T22:43:45.418 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:45 smithi084 ceph-mon[40024]: osdmap e145: 8 total, 8 up, 8 in 2024-09-20T22:43:45.418 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:45 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/1338773827' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:43:45.419 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:45 smithi084 ceph-mon[29774]: pgmap v118: 178 pgs: 5 active+clean+snaptrim, 173 active+clean; 149 MiB data, 576 MiB used, 715 GiB / 715 GiB avail; 16 MiB/s rd, 5.5 MiB/s wr, 30 op/s; 9.6 MiB/s, 8 objects/s recovering 2024-09-20T22:43:45.419 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:45 smithi084 ceph-mon[29774]: osdmap e145: 8 total, 8 up, 8 in 2024-09-20T22:43:45.419 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:45 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/1338773827' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:43:45.494 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:43:45.516 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:45.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:45.517 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:43:45.618 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:43:45.639 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:45.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:45.640 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:43:45.707 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:43:45.741 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:43:45.747 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:45 smithi154 ceph-mon[33264]: pgmap v118: 178 pgs: 5 active+clean+snaptrim, 173 active+clean; 149 MiB data, 576 MiB used, 715 GiB / 715 GiB avail; 16 MiB/s rd, 5.5 MiB/s wr, 30 op/s; 9.6 MiB/s, 8 objects/s recovering 2024-09-20T22:43:45.748 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:45 smithi154 ceph-mon[33264]: osdmap e145: 8 total, 8 up, 8 in 2024-09-20T22:43:45.748 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:45 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/1338773827' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:43:45.762 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:45.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:45.763 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:43:45.864 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:43:45.885 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:45.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:45.886 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:43:45.987 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:43:46.008 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:46.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:46.009 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:43:46.082 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-20T22:43:46.110 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:43:46.154 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:46.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:46.156 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:43:46.256 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:43:46.278 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:46.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:46.279 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:43:46.363 INFO:tasks.rados.rados.0.smithi084.stdout:285: done (1 left) 2024-09-20T22:43:46.364 INFO:tasks.rados.rados.0.smithi084.stdout:286: read oid 20 snap 24 2024-09-20T22:43:46.364 INFO:tasks.rados.rados.0.smithi084.stdout:286: expect deleted 2024-09-20T22:43:46.364 INFO:tasks.rados.rados.0.smithi084.stdout:287: snap_create 2024-09-20T22:43:46.364 INFO:tasks.rados.rados.0.smithi084.stdout:283: finishing write tid 2 to smithi08477203-28 2024-09-20T22:43:46.364 INFO:tasks.rados.rados.0.smithi084.stdout:283: finishing write tid 3 to smithi08477203-28 2024-09-20T22:43:46.364 INFO:tasks.rados.rados.0.smithi084.stdout:283: finishing write tid 4 to smithi08477203-28 2024-09-20T22:43:46.364 INFO:tasks.rados.rados.0.smithi084.stdout:283: oid 28 updating version 0 to 126 2024-09-20T22:43:46.364 INFO:tasks.rados.rados.0.smithi084.stdout:283: oid 28 version 126 is already newer than 124 2024-09-20T22:43:46.364 INFO:tasks.rados.rados.0.smithi084.stdout:283: oid 28 version 126 is already newer than 125 2024-09-20T22:43:46.364 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 28 v 126 (ObjNum 136 snap 25 seq_num 136) dirty exists 2024-09-20T22:43:46.364 INFO:tasks.rados.rados.0.smithi084.stdout:283: left oid 28 (ObjNum 136 snap 25 seq_num 136) 2024-09-20T22:43:46.368 INFO:teuthology.orchestra.run.smithi084.stderr:set pool 1 pgp_num to 21 2024-09-20T22:43:46.380 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:43:46.401 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:46.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:46.402 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:43:46.469 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:46 smithi084 ceph-mon[29774]: purged_snaps scrub starts 2024-09-20T22:43:46.469 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:46 smithi084 ceph-mon[29774]: purged_snaps scrub ok 2024-09-20T22:43:46.469 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:46 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/1338773827' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T22:43:46.469 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:46 smithi084 ceph-mon[29774]: osdmap e146: 8 total, 8 up, 8 in 2024-09-20T22:43:46.469 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:46 smithi084 ceph-mon[29774]: 7.19 scrub starts 2024-09-20T22:43:46.469 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:46 smithi084 ceph-mon[29774]: 7.19 scrub ok 2024-09-20T22:43:46.469 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:46 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/3345164373' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"}]: dispatch 2024-09-20T22:43:46.469 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:46 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"}]: dispatch 2024-09-20T22:43:46.469 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:46 smithi084 ceph-mon[29774]: 7.18 scrub starts 2024-09-20T22:43:46.469 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:46 smithi084 ceph-mon[29774]: 7.18 scrub ok 2024-09-20T22:43:46.469 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:46 smithi084 ceph-mon[29774]: 7.16 scrub starts 2024-09-20T22:43:46.470 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:46 smithi084 ceph-mon[29774]: 7.16 scrub ok 2024-09-20T22:43:46.470 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:46 smithi084 ceph-mon[40024]: purged_snaps scrub starts 2024-09-20T22:43:46.470 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:46 smithi084 ceph-mon[40024]: purged_snaps scrub ok 2024-09-20T22:43:46.470 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:46 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/1338773827' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T22:43:46.470 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:46 smithi084 ceph-mon[40024]: osdmap e146: 8 total, 8 up, 8 in 2024-09-20T22:43:46.470 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:46 smithi084 ceph-mon[40024]: 7.19 scrub starts 2024-09-20T22:43:46.470 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:46 smithi084 ceph-mon[40024]: 7.19 scrub ok 2024-09-20T22:43:46.470 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:46 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/3345164373' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"}]: dispatch 2024-09-20T22:43:46.470 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:46 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"}]: dispatch 2024-09-20T22:43:46.470 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:46 smithi084 ceph-mon[40024]: 7.18 scrub starts 2024-09-20T22:43:46.471 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:46 smithi084 ceph-mon[40024]: 7.18 scrub ok 2024-09-20T22:43:46.471 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:46 smithi084 ceph-mon[40024]: 7.16 scrub starts 2024-09-20T22:43:46.471 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:46 smithi084 ceph-mon[40024]: 7.16 scrub ok 2024-09-20T22:43:46.503 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:43:46.524 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:46.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:46.525 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:43:46.567 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:43:46.567 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:43:46.584 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:43:46.584 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:43:46.600 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:43:46.600 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:43:46.616 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-20T22:43:46.616 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T22:43:46.625 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:43:46.632 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:43:46.632 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:43:46.646 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:46.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:46.647 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:43:46.648 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:43:46.649 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:43:46.667 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:43:46.667 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:43:46.684 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:43:46.684 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:43:46.749 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:43:46.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:46 smithi154 ceph-mon[33264]: purged_snaps scrub starts 2024-09-20T22:43:46.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:46 smithi154 ceph-mon[33264]: purged_snaps scrub ok 2024-09-20T22:43:46.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:46 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/1338773827' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T22:43:46.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:46 smithi154 ceph-mon[33264]: osdmap e146: 8 total, 8 up, 8 in 2024-09-20T22:43:46.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:46 smithi154 ceph-mon[33264]: 7.19 scrub starts 2024-09-20T22:43:46.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:46 smithi154 ceph-mon[33264]: 7.19 scrub ok 2024-09-20T22:43:46.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:46 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/3345164373' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"}]: dispatch 2024-09-20T22:43:46.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:46 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"}]: dispatch 2024-09-20T22:43:46.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:46 smithi154 ceph-mon[33264]: 7.18 scrub starts 2024-09-20T22:43:46.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:46 smithi154 ceph-mon[33264]: 7.18 scrub ok 2024-09-20T22:43:46.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:46 smithi154 ceph-mon[33264]: 7.16 scrub starts 2024-09-20T22:43:46.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:46 smithi154 ceph-mon[33264]: 7.16 scrub ok 2024-09-20T22:43:46.770 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:46.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:46.771 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:43:46.872 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:43:46.894 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:46.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:46.895 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:43:46.996 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:43:47.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:47.018 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:47.018 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:43:47.118 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:43:47.140 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:47.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:47.141 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:43:47.242 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:43:47.265 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:47.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:47.266 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:43:47.356 INFO:tasks.rados.rados.0.smithi084.stdout:283: done (2 left) 2024-09-20T22:43:47.356 INFO:tasks.rados.rados.0.smithi084.stdout:286: done (1 left) 2024-09-20T22:43:47.356 INFO:tasks.rados.rados.0.smithi084.stdout:287: done (0 left) 2024-09-20T22:43:47.356 INFO:tasks.rados.rados.0.smithi084.stdout:288: read oid 39 snap 25 2024-09-20T22:43:47.356 INFO:tasks.rados.rados.0.smithi084.stdout:288: expect (ObjNum 110 snap 22 seq_num 110) 2024-09-20T22:43:47.356 INFO:tasks.rados.rados.0.smithi084.stdout:289: read oid 38 snap -1 2024-09-20T22:43:47.356 INFO:tasks.rados.rados.0.smithi084.stdout:289: expect deleted 2024-09-20T22:43:47.356 INFO:tasks.rados.rados.0.smithi084.stdout:290: copy_from oid 3 from oid 30 current snap is 26 2024-09-20T22:43:47.356 INFO:tasks.rados.rados.0.smithi084.stdout:291: write (excl) oid 35 current snap is 26 2024-09-20T22:43:47.356 INFO:tasks.rados.rados.0.smithi084.stdout:291: seq_num 138 ranges {719366=478716,1808868=510498} 2024-09-20T22:43:47.362 INFO:tasks.rados.rados.0.smithi084.stdout:291: writing smithi08477203-35 from 719366 to 1198082 tid 1 2024-09-20T22:43:47.367 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:43:47.370 INFO:tasks.rados.rados.0.smithi084.stdout:291: writing smithi08477203-35 from 1808868 to 2319366 tid 2 2024-09-20T22:43:47.370 INFO:tasks.rados.rados.0.smithi084.stdout:292: read oid 4 snap -1 2024-09-20T22:43:47.370 INFO:tasks.rados.rados.0.smithi084.stdout:292: expect (ObjNum 121 snap 23 seq_num 121) 2024-09-20T22:43:47.370 INFO:tasks.rados.rados.0.smithi084.stdout:293: rollback oid 42 current snap is 26 2024-09-20T22:43:47.370 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 42 to 25 2024-09-20T22:43:47.370 INFO:tasks.rados.rados.0.smithi084.stdout:288: expect (ObjNum 110 snap 22 seq_num 110) 2024-09-20T22:43:47.390 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:47.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:47.391 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:43:47.492 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:43:47.499 INFO:tasks.rados.rados.0.smithi084.stdout:291: finishing write tid 1 to smithi08477203-35 2024-09-20T22:43:47.499 INFO:tasks.rados.rados.0.smithi084.stdout:288: done (5 left) 2024-09-20T22:43:47.500 INFO:tasks.rados.rados.0.smithi084.stdout:289: done (4 left) 2024-09-20T22:43:47.500 INFO:tasks.rados.rados.0.smithi084.stdout:294: copy_from oid 27 from oid 45 current snap is 26 2024-09-20T22:43:47.500 INFO:tasks.rados.rados.0.smithi084.stdout:290: finishing copy_from racing read to smithi08477203-3 2024-09-20T22:43:47.500 INFO:tasks.rados.rados.0.smithi084.stdout:293: finishing rollback tid 0 to smithi08477203-42 2024-09-20T22:43:47.500 INFO:tasks.rados.rados.0.smithi084.stdout:291: finishing write tid 2 to smithi08477203-35 2024-09-20T22:43:47.500 INFO:tasks.rados.rados.0.smithi084.stdout:291: finishing write tid 3 to smithi08477203-35 2024-09-20T22:43:47.500 INFO:tasks.rados.rados.0.smithi084.stdout:291: finishing write tid 4 to smithi08477203-35 2024-09-20T22:43:47.500 INFO:tasks.rados.rados.0.smithi084.stdout:291: oid 35 updating version 0 to 51 2024-09-20T22:43:47.500 INFO:tasks.rados.rados.0.smithi084.stdout:291: oid 35 version 51 is already newer than 49 2024-09-20T22:43:47.500 INFO:tasks.rados.rados.0.smithi084.stdout:291: oid 35 version 51 is already newer than 50 2024-09-20T22:43:47.500 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 35 v 51 (ObjNum 138 snap 26 seq_num 138) dirty exists 2024-09-20T22:43:47.500 INFO:tasks.rados.rados.0.smithi084.stdout:291: left oid 35 (ObjNum 138 snap 26 seq_num 138) 2024-09-20T22:43:47.500 INFO:tasks.rados.rados.0.smithi084.stdout:292: expect (ObjNum 121 snap 23 seq_num 121) 2024-09-20T22:43:47.514 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:47.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:47.515 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:43:47.531 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:43:47.613 INFO:tasks.rados.rados.0.smithi084.stdout:293: finishing rollback tid 1 to smithi08477203-42 2024-09-20T22:43:47.613 INFO:tasks.rados.rados.0.smithi084.stdout:293: finishing rollback tid 2 to smithi08477203-42 2024-09-20T22:43:47.613 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 42 v 49 (ObjNum 125 snap 23 seq_num 125) dirty exists 2024-09-20T22:43:47.613 INFO:tasks.rados.rados.0.smithi084.stdout:290: finishing copy_from to smithi08477203-3 2024-09-20T22:43:47.614 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 3 v 81 (ObjNum 108 snap 22 seq_num 108) dirty exists 2024-09-20T22:43:47.614 INFO:tasks.rados.rados.0.smithi084.stdout:294: finishing copy_from to smithi08477203-27 2024-09-20T22:43:47.614 INFO:tasks.rados.rados.0.smithi084.stdout:294: got expected ENOENT (src dne) 2024-09-20T22:43:47.614 INFO:tasks.rados.rados.0.smithi084.stdout:294: finishing copy_from racing read to smithi08477203-27 2024-09-20T22:43:47.614 INFO:tasks.rados.rados.0.smithi084.stdout:290: done (4 left) 2024-09-20T22:43:47.614 INFO:tasks.rados.rados.0.smithi084.stdout:291: done (3 left) 2024-09-20T22:43:47.614 INFO:tasks.rados.rados.0.smithi084.stdout:292: done (2 left) 2024-09-20T22:43:47.614 INFO:tasks.rados.rados.0.smithi084.stdout:293: done (1 left) 2024-09-20T22:43:47.614 INFO:tasks.rados.rados.0.smithi084.stdout:294: done (0 left) 2024-09-20T22:43:47.614 INFO:tasks.rados.rados.0.smithi084.stdout:295: delete oid 8 current snap is 26 2024-09-20T22:43:47.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:47 smithi084 ceph-mon[40024]: pgmap v121: 178 pgs: 5 active+clean+snaptrim, 173 active+clean; 149 MiB data, 576 MiB used, 715 GiB / 715 GiB avail; 16 MiB/s rd, 5.5 MiB/s wr, 30 op/s; 9.6 MiB/s, 8 objects/s recovering 2024-09-20T22:43:47.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:47 smithi084 ceph-mon[40024]: 7.0 scrub starts 2024-09-20T22:43:47.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:47 smithi084 ceph-mon[40024]: 7.0 scrub ok 2024-09-20T22:43:47.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:47 smithi084 ceph-mon[40024]: 7.10 scrub starts 2024-09-20T22:43:47.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:47 smithi084 ceph-mon[40024]: 7.10 scrub ok 2024-09-20T22:43:47.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:47 smithi084 ceph-mon[40024]: 7.11 scrub starts 2024-09-20T22:43:47.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:47 smithi084 ceph-mon[40024]: 7.11 scrub ok 2024-09-20T22:43:47.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:47 smithi084 ceph-mon[40024]: 7.8 scrub starts 2024-09-20T22:43:47.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:47 smithi084 ceph-mon[40024]: 7.8 scrub ok 2024-09-20T22:43:47.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:47 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"}]': finished 2024-09-20T22:43:47.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:47 smithi084 ceph-mon[40024]: osdmap e147: 8 total, 8 up, 8 in 2024-09-20T22:43:47.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:47 smithi084 ceph-mon[40024]: 7.12 scrub starts 2024-09-20T22:43:47.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:47 smithi084 ceph-mon[40024]: 7.12 scrub ok 2024-09-20T22:43:47.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:47 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/1346260315' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:43:47.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:47 smithi084 ceph-mon[40024]: purged_snaps scrub starts 2024-09-20T22:43:47.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:47 smithi084 ceph-mon[40024]: purged_snaps scrub ok 2024-09-20T22:43:47.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:47 smithi084 ceph-mon[40024]: 7.9 scrub starts 2024-09-20T22:43:47.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:47 smithi084 ceph-mon[40024]: 7.9 scrub ok 2024-09-20T22:43:47.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:47 smithi084 ceph-mon[29774]: pgmap v121: 178 pgs: 5 active+clean+snaptrim, 173 active+clean; 149 MiB data, 576 MiB used, 715 GiB / 715 GiB avail; 16 MiB/s rd, 5.5 MiB/s wr, 30 op/s; 9.6 MiB/s, 8 objects/s recovering 2024-09-20T22:43:47.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:47 smithi084 ceph-mon[29774]: 7.0 scrub starts 2024-09-20T22:43:47.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:47 smithi084 ceph-mon[29774]: 7.0 scrub ok 2024-09-20T22:43:47.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:47 smithi084 ceph-mon[29774]: 7.10 scrub starts 2024-09-20T22:43:47.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:47 smithi084 ceph-mon[29774]: 7.10 scrub ok 2024-09-20T22:43:47.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:47 smithi084 ceph-mon[29774]: 7.11 scrub starts 2024-09-20T22:43:47.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:47 smithi084 ceph-mon[29774]: 7.11 scrub ok 2024-09-20T22:43:47.617 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:47 smithi084 ceph-mon[29774]: 7.8 scrub starts 2024-09-20T22:43:47.617 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:47 smithi084 ceph-mon[29774]: 7.8 scrub ok 2024-09-20T22:43:47.617 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:47 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"}]': finished 2024-09-20T22:43:47.617 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:47 smithi084 ceph-mon[29774]: osdmap e147: 8 total, 8 up, 8 in 2024-09-20T22:43:47.617 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:47 smithi084 ceph-mon[29774]: 7.12 scrub starts 2024-09-20T22:43:47.617 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:47 smithi084 ceph-mon[29774]: 7.12 scrub ok 2024-09-20T22:43:47.617 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:47 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/1346260315' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:43:47.617 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:47 smithi084 ceph-mon[29774]: purged_snaps scrub starts 2024-09-20T22:43:47.617 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:47 smithi084 ceph-mon[29774]: purged_snaps scrub ok 2024-09-20T22:43:47.617 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:47 smithi084 ceph-mon[29774]: 7.9 scrub starts 2024-09-20T22:43:47.617 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:47 smithi084 ceph-mon[29774]: 7.9 scrub ok 2024-09-20T22:43:47.617 INFO:tasks.rados.rados.0.smithi084.stdout:295: done (0 left) 2024-09-20T22:43:47.617 INFO:tasks.rados.rados.0.smithi084.stdout:296: snap_create 2024-09-20T22:43:47.618 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:43:47.639 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:47.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:47.640 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:43:47.741 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:43:47.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:47 smithi154 ceph-mon[33264]: pgmap v121: 178 pgs: 5 active+clean+snaptrim, 173 active+clean; 149 MiB data, 576 MiB used, 715 GiB / 715 GiB avail; 16 MiB/s rd, 5.5 MiB/s wr, 30 op/s; 9.6 MiB/s, 8 objects/s recovering 2024-09-20T22:43:47.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:47 smithi154 ceph-mon[33264]: 7.0 scrub starts 2024-09-20T22:43:47.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:47 smithi154 ceph-mon[33264]: 7.0 scrub ok 2024-09-20T22:43:47.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:47 smithi154 ceph-mon[33264]: 7.10 scrub starts 2024-09-20T22:43:47.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:47 smithi154 ceph-mon[33264]: 7.10 scrub ok 2024-09-20T22:43:47.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:47 smithi154 ceph-mon[33264]: 7.11 scrub starts 2024-09-20T22:43:47.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:47 smithi154 ceph-mon[33264]: 7.11 scrub ok 2024-09-20T22:43:47.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:47 smithi154 ceph-mon[33264]: 7.8 scrub starts 2024-09-20T22:43:47.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:47 smithi154 ceph-mon[33264]: 7.8 scrub ok 2024-09-20T22:43:47.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:47 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "21"}]': finished 2024-09-20T22:43:47.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:47 smithi154 ceph-mon[33264]: osdmap e147: 8 total, 8 up, 8 in 2024-09-20T22:43:47.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:47 smithi154 ceph-mon[33264]: 7.12 scrub starts 2024-09-20T22:43:47.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:47 smithi154 ceph-mon[33264]: 7.12 scrub ok 2024-09-20T22:43:47.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:47 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/1346260315' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:43:47.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:47 smithi154 ceph-mon[33264]: purged_snaps scrub starts 2024-09-20T22:43:47.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:47 smithi154 ceph-mon[33264]: purged_snaps scrub ok 2024-09-20T22:43:47.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:47 smithi154 ceph-mon[33264]: 7.9 scrub starts 2024-09-20T22:43:47.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:47 smithi154 ceph-mon[33264]: 7.9 scrub ok 2024-09-20T22:43:47.762 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:47.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:47.763 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:43:47.864 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:43:47.886 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:47.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:47.887 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:43:47.988 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:43:48.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:48.010 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:48.010 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:43:48.094 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-20T22:43:48.111 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:43:48.130 INFO:tasks.rados.rados.0.smithi084.stdout:296: done (0 left) 2024-09-20T22:43:48.130 INFO:tasks.rados.rados.0.smithi084.stdout:297: snap_remove snap 25 2024-09-20T22:43:48.154 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:48.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:48.155 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:43:48.256 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:43:48.278 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:48.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:48.279 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:43:48.380 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:43:48.401 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:48.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:48.402 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:43:48.474 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-20T22:43:48.474 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:43:48.502 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:43:48.524 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:48.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:48.525 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:43:48.615 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:43:48 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: ::ffff:172.21.15.154 - - [20/Sep/2024:22:43:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-20T22:43:48.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:48 smithi084 ceph-mon[29774]: 7.3 scrub starts 2024-09-20T22:43:48.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:48 smithi084 ceph-mon[29774]: 7.3 scrub ok 2024-09-20T22:43:48.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:48 smithi084 ceph-mon[29774]: 7.15 scrub starts 2024-09-20T22:43:48.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:48 smithi084 ceph-mon[29774]: 7.15 scrub ok 2024-09-20T22:43:48.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:48 smithi084 ceph-mon[29774]: osdmap e148: 8 total, 8 up, 8 in 2024-09-20T22:43:48.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:48 smithi084 ceph-mon[29774]: 7.7 scrub starts 2024-09-20T22:43:48.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:48 smithi084 ceph-mon[29774]: 7.7 scrub ok 2024-09-20T22:43:48.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:48 smithi084 ceph-mon[29774]: 7.4 scrub starts 2024-09-20T22:43:48.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:48 smithi084 ceph-mon[29774]: 7.4 scrub ok 2024-09-20T22:43:48.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:48 smithi084 ceph-mon[29774]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:43:48.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:48 smithi084 ceph-mon[29774]: osdmap e149: 8 total, 8 up, 8 in 2024-09-20T22:43:48.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:48 smithi084 ceph-mon[40024]: 7.3 scrub starts 2024-09-20T22:43:48.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:48 smithi084 ceph-mon[40024]: 7.3 scrub ok 2024-09-20T22:43:48.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:48 smithi084 ceph-mon[40024]: 7.15 scrub starts 2024-09-20T22:43:48.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:48 smithi084 ceph-mon[40024]: 7.15 scrub ok 2024-09-20T22:43:48.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:48 smithi084 ceph-mon[40024]: osdmap e148: 8 total, 8 up, 8 in 2024-09-20T22:43:48.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:48 smithi084 ceph-mon[40024]: 7.7 scrub starts 2024-09-20T22:43:48.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:48 smithi084 ceph-mon[40024]: 7.7 scrub ok 2024-09-20T22:43:48.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:48 smithi084 ceph-mon[40024]: 7.4 scrub starts 2024-09-20T22:43:48.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:48 smithi084 ceph-mon[40024]: 7.4 scrub ok 2024-09-20T22:43:48.617 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:48 smithi084 ceph-mon[40024]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:43:48.617 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:48 smithi084 ceph-mon[40024]: osdmap e149: 8 total, 8 up, 8 in 2024-09-20T22:43:48.626 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:43:48.648 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:48.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:48.649 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:43:48.750 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:43:48.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:48 smithi154 ceph-mon[33264]: 7.3 scrub starts 2024-09-20T22:43:48.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:48 smithi154 ceph-mon[33264]: 7.3 scrub ok 2024-09-20T22:43:48.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:48 smithi154 ceph-mon[33264]: 7.15 scrub starts 2024-09-20T22:43:48.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:48 smithi154 ceph-mon[33264]: 7.15 scrub ok 2024-09-20T22:43:48.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:48 smithi154 ceph-mon[33264]: osdmap e148: 8 total, 8 up, 8 in 2024-09-20T22:43:48.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:48 smithi154 ceph-mon[33264]: 7.7 scrub starts 2024-09-20T22:43:48.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:48 smithi154 ceph-mon[33264]: 7.7 scrub ok 2024-09-20T22:43:48.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:48 smithi154 ceph-mon[33264]: 7.4 scrub starts 2024-09-20T22:43:48.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:48 smithi154 ceph-mon[33264]: 7.4 scrub ok 2024-09-20T22:43:48.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:48 smithi154 ceph-mon[33264]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:43:48.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:48 smithi154 ceph-mon[33264]: osdmap e149: 8 total, 8 up, 8 in 2024-09-20T22:43:48.772 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:48.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:48.773 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:43:48.873 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:43:48.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:48.895 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:48.895 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:43:48.996 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:43:49.019 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:49.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:49.020 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:43:49.121 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:43:49.139 INFO:tasks.rados.rados.0.smithi084.stdout:297: done (0 left) 2024-09-20T22:43:49.139 INFO:tasks.rados.rados.0.smithi084.stdout:298: delete oid 20 current snap is 27 2024-09-20T22:43:49.142 INFO:tasks.rados.rados.0.smithi084.stdout:298: done (0 left) 2024-09-20T22:43:49.142 INFO:tasks.rados.rados.0.smithi084.stdout:299: snap_remove snap 24 2024-09-20T22:43:49.147 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:49.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:49.148 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:43:49.222 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.0 dump_blocked_ops 2024-09-20T22:43:49.249 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:43:49.254 INFO:journalctl@ceph.iscsi.iscsi.a.smithi154.stdout:Sep 20 22:43:48 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-iscsi-iscsi-a[59730]: debug there is no tcmu-runner data available 2024-09-20T22:43:49.288 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:49.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:49.289 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:43:49.317 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:43:49.364 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:49 smithi084 ceph-mon[40024]: pgmap v124: 178 pgs: 5 active+clean+snaptrim, 173 active+clean; 149 MiB data, 576 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:43:49.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:49 smithi084 ceph-mon[40024]: 7.2 scrub starts 2024-09-20T22:43:49.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:49 smithi084 ceph-mon[40024]: 7.2 scrub ok 2024-09-20T22:43:49.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:49 smithi084 ceph-mon[40024]: 7.17 scrub starts 2024-09-20T22:43:49.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:49 smithi084 ceph-mon[40024]: 7.17 scrub ok 2024-09-20T22:43:49.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:49 smithi084 ceph-mon[40024]: 7.13 scrub starts 2024-09-20T22:43:49.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:49 smithi084 ceph-mon[40024]: 7.13 scrub ok 2024-09-20T22:43:49.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:49 smithi084 ceph-mon[40024]: osdmap e150: 8 total, 8 up, 8 in 2024-09-20T22:43:49.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:49 smithi084 ceph-mon[29774]: pgmap v124: 178 pgs: 5 active+clean+snaptrim, 173 active+clean; 149 MiB data, 576 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:43:49.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:49 smithi084 ceph-mon[29774]: 7.2 scrub starts 2024-09-20T22:43:49.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:49 smithi084 ceph-mon[29774]: 7.2 scrub ok 2024-09-20T22:43:49.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:49 smithi084 ceph-mon[29774]: 7.17 scrub starts 2024-09-20T22:43:49.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:49 smithi084 ceph-mon[29774]: 7.17 scrub ok 2024-09-20T22:43:49.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:49 smithi084 ceph-mon[29774]: 7.13 scrub starts 2024-09-20T22:43:49.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:49 smithi084 ceph-mon[29774]: 7.13 scrub ok 2024-09-20T22:43:49.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:49 smithi084 ceph-mon[29774]: osdmap e150: 8 total, 8 up, 8 in 2024-09-20T22:43:49.390 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:43:49.414 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:49.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:49.415 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:43:49.516 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:43:49.538 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:49.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:49.539 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:43:49.640 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:43:49.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:49.688 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:49.688 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:43:49.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:49 smithi154 ceph-mon[33264]: pgmap v124: 178 pgs: 5 active+clean+snaptrim, 173 active+clean; 149 MiB data, 576 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:43:49.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:49 smithi154 ceph-mon[33264]: 7.2 scrub starts 2024-09-20T22:43:49.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:49 smithi154 ceph-mon[33264]: 7.2 scrub ok 2024-09-20T22:43:49.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:49 smithi154 ceph-mon[33264]: 7.17 scrub starts 2024-09-20T22:43:49.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:49 smithi154 ceph-mon[33264]: 7.17 scrub ok 2024-09-20T22:43:49.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:49 smithi154 ceph-mon[33264]: 7.13 scrub starts 2024-09-20T22:43:49.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:49 smithi154 ceph-mon[33264]: 7.13 scrub ok 2024-09-20T22:43:49.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:49 smithi154 ceph-mon[33264]: osdmap e150: 8 total, 8 up, 8 in 2024-09-20T22:43:49.789 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:43:49.812 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:49.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:49.813 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:43:49.885 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:43:49.914 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:43:49.935 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:49.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:49.936 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:43:49.970 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-20T22:43:50.038 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:43:50.061 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:50.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:50.062 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:43:50.136 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is unset 2024-09-20T22:43:50.147 INFO:tasks.rados.rados.0.smithi084.stdout:299: done (0 left) 2024-09-20T22:43:50.147 INFO:tasks.rados.rados.0.smithi084.stdout:300: rollback oid 31 current snap is 27 2024-09-20T22:43:50.147 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 31 to 26 2024-09-20T22:43:50.148 INFO:tasks.rados.rados.0.smithi084.stdout:301: rollback oid 19 current snap is 27 2024-09-20T22:43:50.148 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 19 to 26 2024-09-20T22:43:50.148 INFO:tasks.rados.rados.0.smithi084.stdout:302: write (excl) oid 26 current snap is 27 2024-09-20T22:43:50.148 INFO:tasks.rados.rados.0.smithi084.stdout:302: seq_num 141 ranges {769787=704463,2267961=101826} 2024-09-20T22:43:50.154 INFO:tasks.rados.rados.0.smithi084.stdout:302: writing smithi08477203-26 from 769787 to 1474250 tid 1 2024-09-20T22:43:50.160 INFO:tasks.rados.rados.0.smithi084.stdout:302: writing smithi08477203-26 from 2267961 to 2369787 tid 2 2024-09-20T22:43:50.160 INFO:tasks.rados.rados.0.smithi084.stdout:300: finishing rollback tid 0 to smithi08477203-31 2024-09-20T22:43:50.160 INFO:tasks.rados.rados.0.smithi084.stdout:301: finishing rollback tid 0 to smithi08477203-19 2024-09-20T22:43:50.160 INFO:tasks.rados.rados.0.smithi084.stdout:300: finishing rollback tid 1 to smithi08477203-31 2024-09-20T22:43:50.160 INFO:tasks.rados.rados.0.smithi084.stdout:300: finishing rollback tid 2 to smithi08477203-31 2024-09-20T22:43:50.160 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 31 v 123 (ObjNum 120 snap 23 seq_num 120) dirty exists 2024-09-20T22:43:50.160 INFO:tasks.rados.rados.0.smithi084.stdout:301: finishing rollback tid 1 to smithi08477203-19 2024-09-20T22:43:50.160 INFO:tasks.rados.rados.0.smithi084.stdout:301: finishing rollback tid 2 to smithi08477203-19 2024-09-20T22:43:50.160 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 19 v 140 (ObjNum 71 snap 7 seq_num 71) dirty exists 2024-09-20T22:43:50.161 INFO:tasks.rados.rados.0.smithi084.stdout:302: finishing write tid 1 to smithi08477203-26 2024-09-20T22:43:50.161 INFO:tasks.rados.rados.0.smithi084.stdout:300: done (2 left) 2024-09-20T22:43:50.161 INFO:tasks.rados.rados.0.smithi084.stdout:301: done (1 left) 2024-09-20T22:43:50.161 INFO:tasks.rados.rados.0.smithi084.stdout:303: read oid 18 snap -1 2024-09-20T22:43:50.161 INFO:tasks.rados.rados.0.smithi084.stdout:303: expect deleted 2024-09-20T22:43:50.161 INFO:tasks.rados.rados.0.smithi084.stdout:304: copy_from oid 16 from oid 22 current snap is 27 2024-09-20T22:43:50.161 INFO:tasks.rados.rados.0.smithi084.stdout:305: rollback oid 7 current snap is 27 2024-09-20T22:43:50.161 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 7 to 26 2024-09-20T22:43:50.161 INFO:tasks.rados.rados.0.smithi084.stdout:306: delete oid 37 current snap is 27 2024-09-20T22:43:50.161 INFO:tasks.rados.rados.0.smithi084.stdout:305: finishing rollback tid 1 to smithi08477203-7 2024-09-20T22:43:50.161 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 7 v 0 (ObjNum 129 snap 0 seq_num 2090973240) dirty dne 2024-09-20T22:43:50.162 INFO:tasks.rados.rados.0.smithi084.stdout:302: finishing write tid 2 to smithi08477203-26 2024-09-20T22:43:50.162 INFO:tasks.rados.rados.0.smithi084.stdout:302: finishing write tid 3 to smithi08477203-26 2024-09-20T22:43:50.162 INFO:tasks.rados.rados.0.smithi084.stdout:302: finishing write tid 4 to smithi08477203-26 2024-09-20T22:43:50.162 INFO:tasks.rados.rados.0.smithi084.stdout:302: oid 26 updating version 0 to 129 2024-09-20T22:43:50.162 INFO:tasks.rados.rados.0.smithi084.stdout:302: oid 26 updating version 129 to 131 2024-09-20T22:43:50.163 INFO:tasks.rados.rados.0.smithi084.stdout:302: oid 26 version 131 is already newer than 130 2024-09-20T22:43:50.163 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 26 v 131 (ObjNum 141 snap 27 seq_num 141) dirty exists 2024-09-20T22:43:50.163 INFO:tasks.rados.rados.0.smithi084.stdout:302: left oid 26 (ObjNum 141 snap 27 seq_num 141) 2024-09-20T22:43:50.163 INFO:tasks.rados.rados.0.smithi084.stdout:302: done (4 left) 2024-09-20T22:43:50.163 INFO:tasks.rados.rados.0.smithi084.stdout:303: done (3 left) 2024-09-20T22:43:50.163 INFO:tasks.rados.rados.0.smithi084.stdout:305: done (2 left) 2024-09-20T22:43:50.163 INFO:tasks.rados.rados.0.smithi084.stdout:306: done (1 left) 2024-09-20T22:43:50.163 INFO:tasks.rados.rados.0.smithi084.stdout:307: write oid 21 current snap is 27 2024-09-20T22:43:50.163 INFO:tasks.rados.rados.0.smithi084.stdout:307: seq_num 143 ranges {403401=454961,1584925=506039} 2024-09-20T22:43:50.163 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:43:50.167 INFO:tasks.rados.rados.0.smithi084.stdout:307: writing smithi08477203-21 from 403401 to 858362 tid 1 2024-09-20T22:43:50.175 INFO:tasks.rados.rados.0.smithi084.stdout:307: writing smithi08477203-21 from 1584925 to 2090964 tid 2 2024-09-20T22:43:50.176 INFO:tasks.rados.rados.0.smithi084.stdout:308: snap_create 2024-09-20T22:43:50.176 INFO:tasks.rados.rados.0.smithi084.stdout:304: finishing copy_from racing read to smithi08477203-16 2024-09-20T22:43:50.177 INFO:tasks.rados.rados.0.smithi084.stdout:307: finishing write tid 1 to smithi08477203-21 2024-09-20T22:43:50.183 INFO:tasks.rados.rados.0.smithi084.stdout:304: finishing copy_from to smithi08477203-16 2024-09-20T22:43:50.183 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 16 v 54 (ObjNum 85 snap 11 seq_num 85) dirty exists 2024-09-20T22:43:50.183 INFO:tasks.rados.rados.0.smithi084.stdout:307: finishing write tid 2 to smithi08477203-21 2024-09-20T22:43:50.183 INFO:tasks.rados.rados.0.smithi084.stdout:307: finishing write tid 3 to smithi08477203-21 2024-09-20T22:43:50.183 INFO:tasks.rados.rados.0.smithi084.stdout:307: finishing write tid 4 to smithi08477203-21 2024-09-20T22:43:50.183 INFO:tasks.rados.rados.0.smithi084.stdout:307: oid 21 updating version 0 to 98 2024-09-20T22:43:50.184 INFO:tasks.rados.rados.0.smithi084.stdout:307: oid 21 updating version 98 to 99 2024-09-20T22:43:50.184 INFO:tasks.rados.rados.0.smithi084.stdout:307: oid 21 version 99 is already newer than 97 2024-09-20T22:43:50.184 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 21 v 99 (ObjNum 143 snap 27 seq_num 143) dirty exists 2024-09-20T22:43:50.184 INFO:tasks.rados.rados.0.smithi084.stdout:307: left oid 21 (ObjNum 143 snap 27 seq_num 143) 2024-09-20T22:43:50.188 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:50.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:50.189 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:43:50.291 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:43:50.313 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:50.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:50.314 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:43:50.415 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:43:50.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:50.439 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:50.439 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:43:50.540 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:43:50.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:50.564 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:50.564 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:43:50.614 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-20T22:43:50.614 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:43:50.614 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:50 smithi084 ceph-mon[40024]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:43:50.614 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:50 smithi084 ceph-mon[40024]: 7.14 scrub starts 2024-09-20T22:43:50.614 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:50 smithi084 ceph-mon[40024]: 7.14 scrub ok 2024-09-20T22:43:50.614 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:50 smithi084 ceph-mon[40024]: 7.5 scrub starts 2024-09-20T22:43:50.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:50 smithi084 ceph-mon[40024]: 7.5 scrub ok 2024-09-20T22:43:50.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:50 smithi084 ceph-mon[40024]: 7.6 scrub starts 2024-09-20T22:43:50.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:50 smithi084 ceph-mon[40024]: 7.6 scrub ok 2024-09-20T22:43:50.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:50 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/2795975900' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:43:50.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:50 smithi084 ceph-mon[40024]: 7.1 scrub starts 2024-09-20T22:43:50.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:50 smithi084 ceph-mon[40024]: 7.1 scrub ok 2024-09-20T22:43:50.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:50 smithi084 ceph-mon[40024]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T22:43:50.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:50 smithi084 ceph-mon[40024]: Cluster is now healthy 2024-09-20T22:43:50.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:50 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/2795975900' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T22:43:50.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:50 smithi084 ceph-mon[40024]: osdmap e151: 8 total, 8 up, 8 in 2024-09-20T22:43:50.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:50 smithi084 ceph-mon[29774]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:43:50.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:50 smithi084 ceph-mon[29774]: 7.14 scrub starts 2024-09-20T22:43:50.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:50 smithi084 ceph-mon[29774]: 7.14 scrub ok 2024-09-20T22:43:50.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:50 smithi084 ceph-mon[29774]: 7.5 scrub starts 2024-09-20T22:43:50.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:50 smithi084 ceph-mon[29774]: 7.5 scrub ok 2024-09-20T22:43:50.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:50 smithi084 ceph-mon[29774]: 7.6 scrub starts 2024-09-20T22:43:50.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:50 smithi084 ceph-mon[29774]: 7.6 scrub ok 2024-09-20T22:43:50.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:50 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/2795975900' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:43:50.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:50 smithi084 ceph-mon[29774]: 7.1 scrub starts 2024-09-20T22:43:50.617 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:50 smithi084 ceph-mon[29774]: 7.1 scrub ok 2024-09-20T22:43:50.617 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:50 smithi084 ceph-mon[29774]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T22:43:50.617 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:50 smithi084 ceph-mon[29774]: Cluster is now healthy 2024-09-20T22:43:50.617 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:50 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/2795975900' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T22:43:50.617 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:50 smithi084 ceph-mon[29774]: osdmap e151: 8 total, 8 up, 8 in 2024-09-20T22:43:50.664 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:43:50.711 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:50.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:50.713 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:43:50.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:50 smithi154 ceph-mon[33264]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:43:50.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:50 smithi154 ceph-mon[33264]: 7.14 scrub starts 2024-09-20T22:43:50.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:50 smithi154 ceph-mon[33264]: 7.14 scrub ok 2024-09-20T22:43:50.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:50 smithi154 ceph-mon[33264]: 7.5 scrub starts 2024-09-20T22:43:50.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:50 smithi154 ceph-mon[33264]: 7.5 scrub ok 2024-09-20T22:43:50.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:50 smithi154 ceph-mon[33264]: 7.6 scrub starts 2024-09-20T22:43:50.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:50 smithi154 ceph-mon[33264]: 7.6 scrub ok 2024-09-20T22:43:50.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:50 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/2795975900' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:43:50.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:50 smithi154 ceph-mon[33264]: 7.1 scrub starts 2024-09-20T22:43:50.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:50 smithi154 ceph-mon[33264]: 7.1 scrub ok 2024-09-20T22:43:50.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:50 smithi154 ceph-mon[33264]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T22:43:50.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:50 smithi154 ceph-mon[33264]: Cluster is now healthy 2024-09-20T22:43:50.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:50 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/2795975900' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T22:43:50.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:50 smithi154 ceph-mon[33264]: osdmap e151: 8 total, 8 up, 8 in 2024-09-20T22:43:50.813 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:43:50.836 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:50.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:50.837 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:43:50.938 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:43:50.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:50.962 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:50.962 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:43:51.063 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:43:51.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:51.086 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:51.086 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:43:51.145 INFO:tasks.rados.rados.0.smithi084.stdout:304: done (2 left) 2024-09-20T22:43:51.145 INFO:tasks.rados.rados.0.smithi084.stdout:307: done (1 left) 2024-09-20T22:43:51.145 INFO:tasks.rados.rados.0.smithi084.stdout:308: done (0 left) 2024-09-20T22:43:51.145 INFO:tasks.rados.rados.0.smithi084.stdout:309: rollback oid 8 current snap is 28 2024-09-20T22:43:51.145 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 8 to 27 2024-09-20T22:43:51.145 INFO:tasks.rados.rados.0.smithi084.stdout:310: snap_create 2024-09-20T22:43:51.145 INFO:tasks.rados.rados.0.smithi084.stdout:309: finishing rollback tid 1 to smithi08477203-8 2024-09-20T22:43:51.146 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 8 v 0 (ObjNum 289 snap 0 seq_num 0) dirty dne 2024-09-20T22:43:51.186 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:43:51.208 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:51.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:51.209 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:43:51.310 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:43:51.331 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:51.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:51.332 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:43:51.433 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:43:51.454 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:51.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:51.455 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:43:51.556 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:43:51.576 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:51.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:51.577 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:43:51.614 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:51 smithi084 ceph-mon[40024]: pgmap v127: 178 pgs: 1 active+clean+snaptrim, 177 active+clean; 107 MiB data, 505 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:43:51.614 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:51 smithi084 ceph-mon[40024]: 4.15 scrub starts 2024-09-20T22:43:51.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:51 smithi084 ceph-mon[40024]: 4.15 scrub ok 2024-09-20T22:43:51.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:51 smithi084 ceph-mon[40024]: 5.10 deep-scrub starts 2024-09-20T22:43:51.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:51 smithi084 ceph-mon[40024]: 5.10 deep-scrub ok 2024-09-20T22:43:51.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:51 smithi084 ceph-mon[40024]: 4.1e deep-scrub starts 2024-09-20T22:43:51.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:51 smithi084 ceph-mon[40024]: 4.1e deep-scrub ok 2024-09-20T22:43:51.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:51 smithi084 ceph-mon[40024]: 4.c scrub starts 2024-09-20T22:43:51.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:51 smithi084 ceph-mon[40024]: 4.c scrub ok 2024-09-20T22:43:51.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:51 smithi084 ceph-mon[40024]: 3.13 scrub starts 2024-09-20T22:43:51.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:51 smithi084 ceph-mon[40024]: 3.13 scrub ok 2024-09-20T22:43:51.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:51 smithi084 ceph-mon[40024]: osdmap e152: 8 total, 8 up, 8 in 2024-09-20T22:43:51.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:51 smithi084 ceph-mon[29774]: pgmap v127: 178 pgs: 1 active+clean+snaptrim, 177 active+clean; 107 MiB data, 505 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:43:51.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:51 smithi084 ceph-mon[29774]: 4.15 scrub starts 2024-09-20T22:43:51.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:51 smithi084 ceph-mon[29774]: 4.15 scrub ok 2024-09-20T22:43:51.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:51 smithi084 ceph-mon[29774]: 5.10 deep-scrub starts 2024-09-20T22:43:51.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:51 smithi084 ceph-mon[29774]: 5.10 deep-scrub ok 2024-09-20T22:43:51.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:51 smithi084 ceph-mon[29774]: 4.1e deep-scrub starts 2024-09-20T22:43:51.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:51 smithi084 ceph-mon[29774]: 4.1e deep-scrub ok 2024-09-20T22:43:51.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:51 smithi084 ceph-mon[29774]: 4.c scrub starts 2024-09-20T22:43:51.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:51 smithi084 ceph-mon[29774]: 4.c scrub ok 2024-09-20T22:43:51.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:51 smithi084 ceph-mon[29774]: 3.13 scrub starts 2024-09-20T22:43:51.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:51 smithi084 ceph-mon[29774]: 3.13 scrub ok 2024-09-20T22:43:51.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:51 smithi084 ceph-mon[29774]: osdmap e152: 8 total, 8 up, 8 in 2024-09-20T22:43:51.678 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:43:51.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:51.700 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:51.700 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:43:51.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:51 smithi154 ceph-mon[33264]: pgmap v127: 178 pgs: 1 active+clean+snaptrim, 177 active+clean; 107 MiB data, 505 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:43:51.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:51 smithi154 ceph-mon[33264]: 4.15 scrub starts 2024-09-20T22:43:51.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:51 smithi154 ceph-mon[33264]: 4.15 scrub ok 2024-09-20T22:43:51.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:51 smithi154 ceph-mon[33264]: 5.10 deep-scrub starts 2024-09-20T22:43:51.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:51 smithi154 ceph-mon[33264]: 5.10 deep-scrub ok 2024-09-20T22:43:51.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:51 smithi154 ceph-mon[33264]: 4.1e deep-scrub starts 2024-09-20T22:43:51.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:51 smithi154 ceph-mon[33264]: 4.1e deep-scrub ok 2024-09-20T22:43:51.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:51 smithi154 ceph-mon[33264]: 4.c scrub starts 2024-09-20T22:43:51.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:51 smithi154 ceph-mon[33264]: 4.c scrub ok 2024-09-20T22:43:51.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:51 smithi154 ceph-mon[33264]: 3.13 scrub starts 2024-09-20T22:43:51.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:51 smithi154 ceph-mon[33264]: 3.13 scrub ok 2024-09-20T22:43:51.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:51 smithi154 ceph-mon[33264]: osdmap e152: 8 total, 8 up, 8 in 2024-09-20T22:43:51.802 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:43:51.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:51.824 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:51.824 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:43:51.856 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.0 dump_historic_ops 2024-09-20T22:43:51.924 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:43:51.961 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:51.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:51.962 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:43:51.985 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:43:52.063 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:43:52.084 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:52.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:52.085 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:43:52.144 INFO:tasks.rados.rados.0.smithi084.stdout:309: done (1 left) 2024-09-20T22:43:52.145 INFO:tasks.rados.rados.0.smithi084.stdout:310: done (0 left) 2024-09-20T22:43:52.145 INFO:tasks.rados.rados.0.smithi084.stdout:311: snap_remove snap 28 2024-09-20T22:43:52.185 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:43:52.208 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:52.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:52.209 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:43:52.310 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:43:52.330 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:52.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:52.332 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:43:52.433 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:43:52.459 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:52.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:52.460 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:43:52.468 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-20T22:43:52.468 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-20T22:43:52.468 INFO:tasks.thrashosds.thrasher:Removing osd 4, in_osds are: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-20T22:43:52.469 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd out 4 2024-09-20T22:43:52.490 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:43:52.491 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:43:52.502 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:43:52.512 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:43:52.512 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:43:52.535 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:43:52.535 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:43:52.553 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-20T22:43:52.553 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T22:43:52.561 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:43:52.572 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:43:52.572 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:43:52.583 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:52.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:52.584 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:43:52.590 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:43:52.590 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:43:52.607 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:43:52.608 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:43:52.624 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:43:52.625 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:43:52.625 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:52 smithi084 ceph-mon[29774]: 3.7 scrub starts 2024-09-20T22:43:52.625 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:52 smithi084 ceph-mon[29774]: 3.7 scrub ok 2024-09-20T22:43:52.625 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:52 smithi084 ceph-mon[29774]: 5.1e deep-scrub starts 2024-09-20T22:43:52.625 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:52 smithi084 ceph-mon[29774]: 5.1e deep-scrub ok 2024-09-20T22:43:52.625 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:52 smithi084 ceph-mon[29774]: osdmap e153: 8 total, 8 up, 8 in 2024-09-20T22:43:52.625 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:52 smithi084 ceph-mon[29774]: 5.4 scrub starts 2024-09-20T22:43:52.625 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:52 smithi084 ceph-mon[29774]: 5.4 scrub ok 2024-09-20T22:43:52.626 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:52 smithi084 ceph-mon[40024]: 3.7 scrub starts 2024-09-20T22:43:52.626 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:52 smithi084 ceph-mon[40024]: 3.7 scrub ok 2024-09-20T22:43:52.626 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:52 smithi084 ceph-mon[40024]: 5.1e deep-scrub starts 2024-09-20T22:43:52.626 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:52 smithi084 ceph-mon[40024]: 5.1e deep-scrub ok 2024-09-20T22:43:52.626 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:52 smithi084 ceph-mon[40024]: osdmap e153: 8 total, 8 up, 8 in 2024-09-20T22:43:52.626 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:52 smithi084 ceph-mon[40024]: 5.4 scrub starts 2024-09-20T22:43:52.626 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:52 smithi084 ceph-mon[40024]: 5.4 scrub ok 2024-09-20T22:43:52.685 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:43:52.711 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:52.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:52.712 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:43:52.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:52 smithi154 ceph-mon[33264]: 3.7 scrub starts 2024-09-20T22:43:52.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:52 smithi154 ceph-mon[33264]: 3.7 scrub ok 2024-09-20T22:43:52.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:52 smithi154 ceph-mon[33264]: 5.1e deep-scrub starts 2024-09-20T22:43:52.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:52 smithi154 ceph-mon[33264]: 5.1e deep-scrub ok 2024-09-20T22:43:52.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:52 smithi154 ceph-mon[33264]: osdmap e153: 8 total, 8 up, 8 in 2024-09-20T22:43:52.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:52 smithi154 ceph-mon[33264]: 5.4 scrub starts 2024-09-20T22:43:52.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:52 smithi154 ceph-mon[33264]: 5.4 scrub ok 2024-09-20T22:43:52.813 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:43:52.837 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:52.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:52.838 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:43:52.939 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:43:52.961 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:52.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:52.962 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:43:53.063 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:43:53.084 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:53.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:53.085 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:43:53.127 INFO:tasks.rados.rados.0.smithi084.stdout:311: done (0 left) 2024-09-20T22:43:53.127 INFO:tasks.rados.rados.0.smithi084.stdout:312: copy_from oid 29 from oid 45 current snap is 29 2024-09-20T22:43:53.127 INFO:tasks.rados.rados.0.smithi084.stdout:313: rollback oid 34 current snap is 29 2024-09-20T22:43:53.127 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 34 to 26 2024-09-20T22:43:53.127 INFO:tasks.rados.rados.0.smithi084.stdout:314: read oid 13 snap -1 2024-09-20T22:43:53.127 INFO:tasks.rados.rados.0.smithi084.stdout:314: expect (ObjNum 133 snap 24 seq_num 133) 2024-09-20T22:43:53.127 INFO:tasks.rados.rados.0.smithi084.stdout:315: read oid 26 snap -1 2024-09-20T22:43:53.127 INFO:tasks.rados.rados.0.smithi084.stdout:315: expect (ObjNum 141 snap 27 seq_num 141) 2024-09-20T22:43:53.127 INFO:tasks.rados.rados.0.smithi084.stdout:316: write oid 17 current snap is 29 2024-09-20T22:43:53.128 INFO:tasks.rados.rados.0.smithi084.stdout:316: seq_num 144 ranges {420208=530210,1527054=766050} 2024-09-20T22:43:53.131 INFO:tasks.rados.rados.0.smithi084.stdout:316: writing smithi08477203-17 from 420208 to 950418 tid 1 2024-09-20T22:43:53.140 INFO:tasks.rados.rados.0.smithi084.stdout:316: writing smithi08477203-17 from 1527054 to 2293104 tid 2 2024-09-20T22:43:53.140 INFO:tasks.rados.rados.0.smithi084.stdout:317: snap_remove snap 27 2024-09-20T22:43:53.187 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:43:53.208 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:53.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:53.209 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:43:53.310 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:43:53.333 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:53.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:53.334 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:43:53.435 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:43:53.437 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd set noscrub 2024-09-20T22:43:53.456 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:53.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:53.457 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:43:53.558 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:43:53.582 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:53.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:53.583 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:43:53.684 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:43:53.700 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:53 smithi084 ceph-mon[40024]: pgmap v130: 178 pgs: 1 active+clean+snaptrim, 177 active+clean; 107 MiB data, 505 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:43:53.700 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:53 smithi084 ceph-mon[40024]: 5.14 scrub starts 2024-09-20T22:43:53.700 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:53 smithi084 ceph-mon[40024]: 5.14 scrub ok 2024-09-20T22:43:53.700 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:53 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/3022409775' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:43:53.701 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:53 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:43:53.701 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:53 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:43:53.701 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:53 smithi084 ceph-mon[40024]: purged_snaps scrub starts 2024-09-20T22:43:53.701 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:53 smithi084 ceph-mon[40024]: purged_snaps scrub ok 2024-09-20T22:43:53.701 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:53 smithi084 ceph-mon[40024]: 6.2 deep-scrub starts 2024-09-20T22:43:53.701 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:53 smithi084 ceph-mon[40024]: 6.2 deep-scrub ok 2024-09-20T22:43:53.701 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:53 smithi084 ceph-mon[40024]: osdmap e154: 8 total, 8 up, 8 in 2024-09-20T22:43:53.701 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:53 smithi084 ceph-mon[40024]: 6.11 scrub starts 2024-09-20T22:43:53.701 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:53 smithi084 ceph-mon[40024]: 6.11 scrub ok 2024-09-20T22:43:53.701 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:53 smithi084 ceph-mon[29774]: pgmap v130: 178 pgs: 1 active+clean+snaptrim, 177 active+clean; 107 MiB data, 505 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:43:53.701 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:53 smithi084 ceph-mon[29774]: 5.14 scrub starts 2024-09-20T22:43:53.701 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:53 smithi084 ceph-mon[29774]: 5.14 scrub ok 2024-09-20T22:43:53.702 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:53 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/3022409775' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:43:53.702 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:53 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:43:53.702 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:53 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:43:53.702 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:53 smithi084 ceph-mon[29774]: purged_snaps scrub starts 2024-09-20T22:43:53.702 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:53 smithi084 ceph-mon[29774]: purged_snaps scrub ok 2024-09-20T22:43:53.702 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:53 smithi084 ceph-mon[29774]: 6.2 deep-scrub starts 2024-09-20T22:43:53.702 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:53 smithi084 ceph-mon[29774]: 6.2 deep-scrub ok 2024-09-20T22:43:53.702 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:53 smithi084 ceph-mon[29774]: osdmap e154: 8 total, 8 up, 8 in 2024-09-20T22:43:53.702 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:53 smithi084 ceph-mon[29774]: 6.11 scrub starts 2024-09-20T22:43:53.702 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:53 smithi084 ceph-mon[29774]: 6.11 scrub ok 2024-09-20T22:43:53.707 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:53.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:53.708 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:43:53.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:53 smithi154 ceph-mon[33264]: pgmap v130: 178 pgs: 1 active+clean+snaptrim, 177 active+clean; 107 MiB data, 505 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:43:53.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:53 smithi154 ceph-mon[33264]: 5.14 scrub starts 2024-09-20T22:43:53.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:53 smithi154 ceph-mon[33264]: 5.14 scrub ok 2024-09-20T22:43:53.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:53 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/3022409775' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:43:53.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:53 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:43:53.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:53 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:43:53.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:53 smithi154 ceph-mon[33264]: purged_snaps scrub starts 2024-09-20T22:43:53.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:53 smithi154 ceph-mon[33264]: purged_snaps scrub ok 2024-09-20T22:43:53.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:53 smithi154 ceph-mon[33264]: 6.2 deep-scrub starts 2024-09-20T22:43:53.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:53 smithi154 ceph-mon[33264]: 6.2 deep-scrub ok 2024-09-20T22:43:53.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:53 smithi154 ceph-mon[33264]: osdmap e154: 8 total, 8 up, 8 in 2024-09-20T22:43:53.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:53 smithi154 ceph-mon[33264]: 6.11 scrub starts 2024-09-20T22:43:53.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:53 smithi154 ceph-mon[33264]: 6.11 scrub ok 2024-09-20T22:43:53.809 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:43:53.832 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:53.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:53.833 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:43:53.845 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-20T22:43:53.845 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2024-09-20T22:43:53.845 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2024-09-20T22:43:53.845 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [ 2024-09-20T22:43:53.846 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.846 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.14460.0:1 3.12 3:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-20T22:43:53.846 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:41:37.966675+0000", 2024-09-20T22:43:53.846 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 135.875648138, 2024-09-20T22:43:53.846 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.19206436900000001, 2024-09-20T22:43:53.846 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:43:53.846 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:43:53.846 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "reached pg", 2024-09-20T22:43:53.846 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:43:53.846 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.14460", 2024-09-20T22:43:53.846 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/1273482728", 2024-09-20T22:43:53.846 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1 2024-09-20T22:43:53.846 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.847 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:43:53.847 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.847 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:43:53.847 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:41:37.966675+0000", 2024-09-20T22:43:53.847 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:43:53.847 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.847 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.847 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:43:53.847 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:41:37.966675+0000", 2024-09-20T22:43:53.847 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:43:53.847 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.847 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.847 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:43:53.847 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:41:37.966682+0000", 2024-09-20T22:43:53.848 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.3100000000000003e-06 2024-09-20T22:43:53.848 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.848 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.848 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:43:53.848 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:41:37.966688+0000", 2024-09-20T22:43:53.848 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.1209999999999998e-06 2024-09-20T22:43:53.848 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.848 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.848 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:43:53.848 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:41:37.966698+0000", 2024-09-20T22:43:53.848 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0220999999999999e-05 2024-09-20T22:43:53.848 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.848 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.848 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:43:53.849 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:41:37.966706+0000", 2024-09-20T22:43:53.849 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.4460000000000001e-06 2024-09-20T22:43:53.849 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.849 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.849 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:43:53.849 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:41:37.966763+0000", 2024-09-20T22:43:53.849 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.7025999999999999e-05 2024-09-20T22:43:53.849 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.849 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.849 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-09-20T22:43:53.849 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:41:37.966771+0000", 2024-09-20T22:43:53.849 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.9540000000000004e-06 2024-09-20T22:43:53.849 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.849 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.850 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:43:53.850 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:41:38.158654+0000", 2024-09-20T22:43:53.850 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.19188287000000001 2024-09-20T22:43:53.850 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.850 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.850 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:43:53.850 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:41:38.158740+0000", 2024-09-20T22:43:53.850 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.5420999999999994e-05 2024-09-20T22:43:53.850 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:43:53.850 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:43:53.850 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:43:53.850 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.850 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.850 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24652.0:336 7.7 e107/94 7:e67788e2:::smithi08477203-31:head v 107'45, mlcod=105'41)", 2024-09-20T22:43:53.850 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:43:06.546789+0000", 2024-09-20T22:43:53.851 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 47.295534363999998, 2024-09-20T22:43:53.851 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013900732000000001, 2024-09-20T22:43:53.851 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:43:53.851 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:43:53.851 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:43:53.851 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:43:53.851 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.851 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:43:53.851 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:06.546789+0000", 2024-09-20T22:43:53.851 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:43:53.851 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.851 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.851 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:43:53.851 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:06.546789+0000", 2024-09-20T22:43:53.852 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:43:53.852 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.852 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.852 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:43:53.852 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:06.546789+0000", 2024-09-20T22:43:53.852 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.1500000000000001e-07 2024-09-20T22:43:53.852 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.852 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.852 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:43:53.852 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:06.554560+0000", 2024-09-20T22:43:53.852 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0077711159999999998 2024-09-20T22:43:53.852 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.852 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.852 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:43:53.852 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:06.554577+0000", 2024-09-20T22:43:53.853 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.6509000000000001e-05 2024-09-20T22:43:53.853 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.853 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.853 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:43:53.853 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:06.554586+0000", 2024-09-20T22:43:53.853 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.7089999999999997e-06 2024-09-20T22:43:53.853 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.853 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.853 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:43:53.853 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:06.554645+0000", 2024-09-20T22:43:53.853 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.8202999999999998e-05 2024-09-20T22:43:53.853 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.853 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.853 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:43:53.854 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:06.554661+0000", 2024-09-20T22:43:53.854 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.6099e-05 2024-09-20T22:43:53.854 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.854 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.854 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:43:53.854 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:06.560643+0000", 2024-09-20T22:43:53.854 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0059824750000000001 2024-09-20T22:43:53.854 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.854 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.854 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:43:53.854 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:06.560690+0000", 2024-09-20T22:43:53.854 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.6406000000000003e-05 2024-09-20T22:43:53.854 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:43:53.855 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:43:53.855 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:43:53.855 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.855 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.855 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24652.0:348 7.7 e108/94 7:ed54065f:::smithi08477203-12:head v 108'48, mlcod=106'43)", 2024-09-20T22:43:53.855 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:43:07.552563+0000", 2024-09-20T22:43:53.855 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 46.289760573000002, 2024-09-20T22:43:53.855 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0077376759999999998, 2024-09-20T22:43:53.855 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:43:53.855 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:43:53.855 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:43:53.855 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:43:53.855 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.855 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:43:53.856 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.552563+0000", 2024-09-20T22:43:53.856 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:43:53.856 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.856 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.856 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:43:53.856 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.552563+0000", 2024-09-20T22:43:53.856 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:43:53.856 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.856 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.856 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:43:53.856 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.552564+0000", 2024-09-20T22:43:53.856 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.7399999999999991e-07 2024-09-20T22:43:53.856 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.856 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.856 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:43:53.857 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.557319+0000", 2024-09-20T22:43:53.857 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0047555669999999996 2024-09-20T22:43:53.857 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.857 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.857 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:43:53.857 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.557335+0000", 2024-09-20T22:43:53.857 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.5488000000000001e-05 2024-09-20T22:43:53.857 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.857 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.857 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:43:53.857 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.557342+0000", 2024-09-20T22:43:53.857 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.1790000000000002e-06 2024-09-20T22:43:53.857 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.857 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.857 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:43:53.858 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.557434+0000", 2024-09-20T22:43:53.858 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.2041999999999997e-05 2024-09-20T22:43:53.858 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.858 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.858 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:43:53.858 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.557452+0000", 2024-09-20T22:43:53.858 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.8317000000000001e-05 2024-09-20T22:43:53.858 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.858 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.858 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:43:53.858 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.560257+0000", 2024-09-20T22:43:53.858 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002804216 2024-09-20T22:43:53.858 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.858 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.858 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:43:53.859 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.560300+0000", 2024-09-20T22:43:53.859 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.3893000000000001e-05 2024-09-20T22:43:53.859 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:43:53.859 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:43:53.859 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:43:53.859 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.859 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.860 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:353 7.f 7:f93102e8:::smithi08477203-37:head [copy-from ver 3] snapc 6=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-09-20T22:43:53.860 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:43:07.813602+0000", 2024-09-20T22:43:53.860 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 46.028721730000001, 2024-09-20T22:43:53.860 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014089683, 2024-09-20T22:43:53.860 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:43:53.860 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:43:53.860 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:43:53.860 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:43:53.861 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:43:53.861 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:43:53.861 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 353 2024-09-20T22:43:53.861 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.861 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:43:53.861 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.861 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:43:53.861 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.813602+0000", 2024-09-20T22:43:53.861 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:43:53.861 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.861 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.861 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:43:53.861 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.813602+0000", 2024-09-20T22:43:53.861 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:43:53.862 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.862 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.862 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:43:53.862 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.813604+0000", 2024-09-20T22:43:53.862 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.9939999999999999e-06 2024-09-20T22:43:53.862 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.862 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.862 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:43:53.862 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.813612+0000", 2024-09-20T22:43:53.862 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.3690000000000007e-06 2024-09-20T22:43:53.862 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.862 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.862 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:43:53.862 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.813631+0000", 2024-09-20T22:43:53.863 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.8589000000000001e-05 2024-09-20T22:43:53.863 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.863 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.863 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:43:53.863 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.813642+0000", 2024-09-20T22:43:53.863 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1766999999999999e-05 2024-09-20T22:43:53.863 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.863 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.863 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:43:53.863 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.813711+0000", 2024-09-20T22:43:53.863 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.9175000000000001e-05 2024-09-20T22:43:53.863 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.863 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.863 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:43:53.863 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.813811+0000", 2024-09-20T22:43:53.864 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.9672000000000004e-05 2024-09-20T22:43:53.864 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.864 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.864 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 3,7", 2024-09-20T22:43:53.864 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.817022+0000", 2024-09-20T22:43:53.864 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00321135 2024-09-20T22:43:53.864 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.864 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.864 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:43:53.864 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.820378+0000", 2024-09-20T22:43:53.864 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.003355224 2024-09-20T22:43:53.864 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.864 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.864 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:43:53.864 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.820397+0000", 2024-09-20T22:43:53.865 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.8899999999999999e-05 2024-09-20T22:43:53.865 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.865 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.865 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:43:53.865 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.827619+0000", 2024-09-20T22:43:53.865 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.007222816 2024-09-20T22:43:53.865 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.865 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.865 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:43:53.865 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.827653+0000", 2024-09-20T22:43:53.865 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.392e-05 2024-09-20T22:43:53.865 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.865 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.865 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:43:53.866 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.827691+0000", 2024-09-20T22:43:53.866 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.7907000000000001e-05 2024-09-20T22:43:53.866 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:43:53.866 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:43:53.866 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:43:53.866 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.866 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.866 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:354 7.f 7:f93102e8:::smithi08477203-37:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e108)", 2024-09-20T22:43:53.866 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:43:07.813667+0000", 2024-09-20T22:43:53.866 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 46.028656761000001, 2024-09-20T22:43:53.866 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0059987549999999997, 2024-09-20T22:43:53.866 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:43:53.866 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:43:53.866 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-20T22:43:53.866 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:43:53.867 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:43:53.867 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:43:53.867 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 354 2024-09-20T22:43:53.867 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.867 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:43:53.867 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.867 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:43:53.867 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.813667+0000", 2024-09-20T22:43:53.867 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:43:53.867 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.867 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.867 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:43:53.867 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.813667+0000", 2024-09-20T22:43:53.867 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:43:53.867 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.868 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.868 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:43:53.868 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.813667+0000", 2024-09-20T22:43:53.868 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.1399999999999997e-07 2024-09-20T22:43:53.868 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.868 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.868 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:43:53.868 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.813669+0000", 2024-09-20T22:43:53.868 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.2400000000000002e-06 2024-09-20T22:43:53.868 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.868 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.868 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:43:53.868 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.813676+0000", 2024-09-20T22:43:53.868 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.849e-06 2024-09-20T22:43:53.869 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.869 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.869 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:43:53.869 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.813681+0000", 2024-09-20T22:43:53.869 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.9760000000000003e-06 2024-09-20T22:43:53.869 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.869 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.869 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:43:53.869 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.814025+0000", 2024-09-20T22:43:53.869 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000343579 2024-09-20T22:43:53.869 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.869 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.869 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for blocked object", 2024-09-20T22:43:53.869 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.814056+0000", 2024-09-20T22:43:53.869 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.1197e-05 2024-09-20T22:43:53.870 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.870 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.870 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:43:53.870 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.819479+0000", 2024-09-20T22:43:53.870 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0054231440000000004 2024-09-20T22:43:53.870 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.870 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.870 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:43:53.870 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.819571+0000", 2024-09-20T22:43:53.870 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.1885999999999994e-05 2024-09-20T22:43:53.870 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.870 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.870 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:43:53.871 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.819665+0000", 2024-09-20T22:43:53.871 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.4469999999999995e-05 2024-09-20T22:43:53.871 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:43:53.871 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:43:53.871 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:43:53.871 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.871 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.871 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:355 7.f 7:fa55e5ae:::smithi08477203-7:head [write 675648~546760] snapc 6=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-09-20T22:43:53.871 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:43:07.821345+0000", 2024-09-20T22:43:53.871 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 46.020978659999997, 2024-09-20T22:43:53.871 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.007713574, 2024-09-20T22:43:53.871 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:43:53.871 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:43:53.871 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:43:53.871 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:43:53.872 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:43:53.872 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:43:53.872 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 355 2024-09-20T22:43:53.872 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.872 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:43:53.872 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.872 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:43:53.872 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.821345+0000", 2024-09-20T22:43:53.872 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:43:53.872 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.872 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.872 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:43:53.872 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.821345+0000", 2024-09-20T22:43:53.872 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:43:53.872 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.873 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.873 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:43:53.873 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.821346+0000", 2024-09-20T22:43:53.873 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.637e-06 2024-09-20T22:43:53.873 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.873 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.873 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:43:53.873 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.821822+0000", 2024-09-20T22:43:53.873 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00047602099999999997 2024-09-20T22:43:53.873 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.873 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.873 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:43:53.873 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.821835+0000", 2024-09-20T22:43:53.873 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2293000000000001e-05 2024-09-20T22:43:53.873 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.874 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.874 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:43:53.874 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.821842+0000", 2024-09-20T22:43:53.874 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.5639999999999999e-06 2024-09-20T22:43:53.874 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.874 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.874 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:43:53.874 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.821910+0000", 2024-09-20T22:43:53.874 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.7781999999999996e-05 2024-09-20T22:43:53.874 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.874 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.874 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:43:53.874 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.822016+0000", 2024-09-20T22:43:53.874 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000105677 2024-09-20T22:43:53.875 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.875 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.875 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 3,7", 2024-09-20T22:43:53.875 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.822168+0000", 2024-09-20T22:43:53.875 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00015269599999999999 2024-09-20T22:43:53.875 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.875 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.875 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:43:53.875 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.823075+0000", 2024-09-20T22:43:53.875 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00090635599999999998 2024-09-20T22:43:53.875 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.875 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.875 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:43:53.875 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.823706+0000", 2024-09-20T22:43:53.876 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00063161999999999995 2024-09-20T22:43:53.876 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.876 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.876 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:43:53.876 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.829000+0000", 2024-09-20T22:43:53.876 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.005293847 2024-09-20T22:43:53.877 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.877 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.877 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:43:53.877 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.829033+0000", 2024-09-20T22:43:53.878 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.2537999999999999e-05 2024-09-20T22:43:53.878 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.878 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.878 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:43:53.878 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.829058+0000", 2024-09-20T22:43:53.878 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.5542999999999999e-05 2024-09-20T22:43:53.878 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:43:53.878 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:43:53.878 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:43:53.878 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.878 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.878 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:359 7.9 7:91f2d47c:::smithi08477203-8:head [copy-from ver 6] snapc 6=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-09-20T22:43:53.878 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:43:07.829163+0000", 2024-09-20T22:43:53.878 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 46.013160710999998, 2024-09-20T22:43:53.879 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011899852000000001, 2024-09-20T22:43:53.879 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:43:53.879 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:43:53.879 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:43:53.879 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:43:53.879 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:43:53.879 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:43:53.879 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 359 2024-09-20T22:43:53.879 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.879 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:43:53.879 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.879 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:43:53.879 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.829163+0000", 2024-09-20T22:43:53.879 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:43:53.880 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.880 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.880 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:43:53.880 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.829163+0000", 2024-09-20T22:43:53.880 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:43:53.880 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.880 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.880 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:43:53.880 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.829163+0000", 2024-09-20T22:43:53.880 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.6199999999999999e-07 2024-09-20T22:43:53.880 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.880 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.880 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:43:53.880 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.829165+0000", 2024-09-20T22:43:53.880 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.7209999999999999e-06 2024-09-20T22:43:53.881 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.881 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.881 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:43:53.881 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.829169+0000", 2024-09-20T22:43:53.881 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.121e-06 2024-09-20T22:43:53.881 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.881 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.881 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:43:53.881 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.829172+0000", 2024-09-20T22:43:53.881 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.8839999999999998e-06 2024-09-20T22:43:53.882 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.882 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.882 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:43:53.882 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.829211+0000", 2024-09-20T22:43:53.882 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.9551000000000001e-05 2024-09-20T22:43:53.882 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.882 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.882 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:43:53.882 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.829294+0000", 2024-09-20T22:43:53.882 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.2548999999999999e-05 2024-09-20T22:43:53.882 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.882 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.882 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 1,4", 2024-09-20T22:43:53.883 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.834578+0000", 2024-09-20T22:43:53.883 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.005284179 2024-09-20T22:43:53.883 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.883 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.883 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:43:53.883 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.840537+0000", 2024-09-20T22:43:53.883 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0059587670000000002 2024-09-20T22:43:53.883 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.883 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.883 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:43:53.884 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.840753+0000", 2024-09-20T22:43:53.884 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00021661799999999999 2024-09-20T22:43:53.884 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.884 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.884 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:43:53.884 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.840999+0000", 2024-09-20T22:43:53.884 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000246009 2024-09-20T22:43:53.884 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.884 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.884 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:43:53.884 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.841032+0000", 2024-09-20T22:43:53.884 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.2567999999999997e-05 2024-09-20T22:43:53.884 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.884 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.884 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:43:53.885 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.841062+0000", 2024-09-20T22:43:53.885 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.0623000000000001e-05 2024-09-20T22:43:53.885 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:43:53.885 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:43:53.885 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:43:53.885 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.885 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.885 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:360 7.9 7:91f2d47c:::smithi08477203-8:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e108)", 2024-09-20T22:43:53.885 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:43:07.829179+0000", 2024-09-20T22:43:53.885 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 46.013143988000003, 2024-09-20T22:43:53.885 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0083981460000000004, 2024-09-20T22:43:53.885 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:43:53.885 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:43:53.886 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-20T22:43:53.886 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:43:53.886 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:43:53.886 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:43:53.886 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 360 2024-09-20T22:43:53.886 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.886 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:43:53.886 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.886 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:43:53.886 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.829179+0000", 2024-09-20T22:43:53.886 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:43:53.886 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.886 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.886 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:43:53.887 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.829179+0000", 2024-09-20T22:43:53.887 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:43:53.887 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.887 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.887 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:43:53.887 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.829180+0000", 2024-09-20T22:43:53.887 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.7899999999999999e-07 2024-09-20T22:43:53.887 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.887 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.887 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:43:53.887 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.829181+0000", 2024-09-20T22:43:53.887 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.607e-06 2024-09-20T22:43:53.887 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.887 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.887 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:43:53.888 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.829186+0000", 2024-09-20T22:43:53.888 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.7190000000000001e-06 2024-09-20T22:43:53.888 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.888 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.888 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:43:53.888 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.829191+0000", 2024-09-20T22:43:53.888 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.5000000000000001e-06 2024-09-20T22:43:53.888 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.888 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.888 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:43:53.888 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.829473+0000", 2024-09-20T22:43:53.888 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00028275099999999998 2024-09-20T22:43:53.888 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.888 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.889 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for blocked object", 2024-09-20T22:43:53.889 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.829520+0000", 2024-09-20T22:43:53.889 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.6777999999999998e-05 2024-09-20T22:43:53.889 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.889 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.889 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:43:53.889 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.837366+0000", 2024-09-20T22:43:53.889 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0078455790000000001 2024-09-20T22:43:53.889 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.889 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.889 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:43:53.889 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.837461+0000", 2024-09-20T22:43:53.889 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.5087999999999995e-05 2024-09-20T22:43:53.890 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.890 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.890 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:43:53.890 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.837577+0000", 2024-09-20T22:43:53.890 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00011674499999999999 2024-09-20T22:43:53.890 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:43:53.890 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:43:53.890 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:43:53.890 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.890 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.890 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:397 7.9 7:91f2d47c:::smithi08477203-8:head [copy-from ver 3] snapc 8=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-09-20T22:43:53.890 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:43:11.361736+0000", 2024-09-20T22:43:53.890 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 42.480587792000001, 2024-09-20T22:43:53.890 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011019584000000001, 2024-09-20T22:43:53.891 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:43:53.891 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:43:53.891 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:43:53.891 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:43:53.891 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:43:53.891 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:43:53.891 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 397 2024-09-20T22:43:53.891 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.891 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:43:53.891 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.891 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:43:53.891 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:11.361736+0000", 2024-09-20T22:43:53.891 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:43:53.891 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.892 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.892 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:43:53.892 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:11.361736+0000", 2024-09-20T22:43:53.892 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:43:53.892 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.892 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.892 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:43:53.892 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:11.361738+0000", 2024-09-20T22:43:53.892 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.2120000000000002e-06 2024-09-20T22:43:53.892 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.892 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.892 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:43:53.892 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:11.361746+0000", 2024-09-20T22:43:53.892 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.4749999999999993e-06 2024-09-20T22:43:53.892 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.893 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.893 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:43:53.893 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:11.361766+0000", 2024-09-20T22:43:53.893 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.9519999999999999e-05 2024-09-20T22:43:53.893 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.893 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.893 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:43:53.893 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:11.361776+0000", 2024-09-20T22:43:53.893 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0719e-05 2024-09-20T22:43:53.893 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.893 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.893 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:43:53.893 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:11.361847+0000", 2024-09-20T22:43:53.893 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.0829999999999998e-05 2024-09-20T22:43:53.894 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.894 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.894 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:43:53.894 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:11.361951+0000", 2024-09-20T22:43:53.894 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000103538 2024-09-20T22:43:53.894 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.894 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.894 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 1,4", 2024-09-20T22:43:53.894 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:11.366189+0000", 2024-09-20T22:43:53.894 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0042376990000000002 2024-09-20T22:43:53.894 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.894 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.894 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:43:53.894 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:11.372340+0000", 2024-09-20T22:43:53.895 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0061511650000000001 2024-09-20T22:43:53.895 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.895 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.895 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:43:53.895 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:11.372390+0000", 2024-09-20T22:43:53.895 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.0210999999999997e-05 2024-09-20T22:43:53.895 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.895 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.895 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:43:53.895 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:11.372645+0000", 2024-09-20T22:43:53.895 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000255233 2024-09-20T22:43:53.895 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.896 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.896 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:43:53.896 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:11.372707+0000", 2024-09-20T22:43:53.896 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.1427999999999995e-05 2024-09-20T22:43:53.896 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.896 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.896 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:43:53.896 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:11.372755+0000", 2024-09-20T22:43:53.896 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.8554000000000003e-05 2024-09-20T22:43:53.896 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:43:53.896 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:43:53.896 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:43:53.896 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.896 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.896 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:398 7.9 7:91f2d47c:::smithi08477203-8:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e112)", 2024-09-20T22:43:53.897 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:43:11.361803+0000", 2024-09-20T22:43:53.897 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 42.480520235999997, 2024-09-20T22:43:53.897 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0074706269999999996, 2024-09-20T22:43:53.897 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:43:53.897 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:43:53.897 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-20T22:43:53.897 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:43:53.897 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:43:53.897 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:43:53.897 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 398 2024-09-20T22:43:53.897 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.897 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:43:53.897 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.897 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:43:53.898 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:11.361803+0000", 2024-09-20T22:43:53.898 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:43:53.898 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.898 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.898 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:43:53.898 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:11.361803+0000", 2024-09-20T22:43:53.898 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:43:53.898 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.898 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.898 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:43:53.898 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:11.361804+0000", 2024-09-20T22:43:53.898 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.13e-07 2024-09-20T22:43:53.898 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.898 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.899 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:43:53.899 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:11.361806+0000", 2024-09-20T22:43:53.899 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.7020000000000001e-06 2024-09-20T22:43:53.899 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.899 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.899 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:43:53.899 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:11.361814+0000", 2024-09-20T22:43:53.899 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.4390000000000003e-06 2024-09-20T22:43:53.899 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.899 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.899 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:43:53.899 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:11.361820+0000", 2024-09-20T22:43:53.899 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.127e-06 2024-09-20T22:43:53.899 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.900 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.900 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:43:53.900 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:11.362183+0000", 2024-09-20T22:43:53.900 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00036265400000000001 2024-09-20T22:43:53.900 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.900 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.900 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for blocked object", 2024-09-20T22:43:53.900 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:11.362233+0000", 2024-09-20T22:43:53.900 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.0921000000000003e-05 2024-09-20T22:43:53.900 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.900 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.900 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:43:53.900 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:11.369157+0000", 2024-09-20T22:43:53.900 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0069239159999999996 2024-09-20T22:43:53.900 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.901 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.901 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:43:53.901 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:11.369211+0000", 2024-09-20T22:43:53.901 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.3430000000000002e-05 2024-09-20T22:43:53.901 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.901 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.901 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:43:53.901 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:11.369274+0000", 2024-09-20T22:43:53.901 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.2924999999999998e-05 2024-09-20T22:43:53.901 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:43:53.901 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:43:53.901 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:43:53.901 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.901 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.902 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:401 7.9 7:9cc0b7e2:::smithi08477203-46:head [read 0~2756315,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e112)", 2024-09-20T22:43:53.902 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:43:11.361914+0000", 2024-09-20T22:43:53.902 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 42.480409657000003, 2024-09-20T22:43:53.902 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0087258990000000005, 2024-09-20T22:43:53.902 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:43:53.902 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:43:53.902 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-20T22:43:53.902 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:43:53.902 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:43:53.902 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:43:53.902 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 401 2024-09-20T22:43:53.902 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.902 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:43:53.902 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.903 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:43:53.903 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:11.361914+0000", 2024-09-20T22:43:53.904 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:43:53.904 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.904 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.904 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:43:53.904 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:11.361914+0000", 2024-09-20T22:43:53.904 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:43:53.904 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.904 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.904 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:43:53.904 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:11.361914+0000", 2024-09-20T22:43:53.904 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.58e-07 2024-09-20T22:43:53.904 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.904 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.904 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:43:53.904 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:11.361918+0000", 2024-09-20T22:43:53.905 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.2830000000000003e-06 2024-09-20T22:43:53.905 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.905 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.905 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:43:53.905 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:11.361925+0000", 2024-09-20T22:43:53.905 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.178e-06 2024-09-20T22:43:53.905 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.905 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.905 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:43:53.905 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:11.361933+0000", 2024-09-20T22:43:53.905 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.3299999999999999e-06 2024-09-20T22:43:53.905 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.905 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.905 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:43:53.905 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:11.369315+0000", 2024-09-20T22:43:53.906 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0073816400000000001 2024-09-20T22:43:53.906 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.906 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.906 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:43:53.906 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:11.369406+0000", 2024-09-20T22:43:53.906 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.1897000000000005e-05 2024-09-20T22:43:53.906 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.906 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.906 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:43:53.906 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:11.370640+0000", 2024-09-20T22:43:53.906 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0012331130000000001 2024-09-20T22:43:53.906 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:43:53.906 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:43:53.906 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:43:53.907 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.907 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.907 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(osd.6.0:178 7.7 e124/94 7:e1d5c412:::smithi08477203-47:e v 124'77, mlcod=122'73)", 2024-09-20T22:43:53.907 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:43:23.126018+0000", 2024-09-20T22:43:53.907 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 30.716305733999999, 2024-09-20T22:43:53.907 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.007539675, 2024-09-20T22:43:53.907 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:43:53.907 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:43:53.907 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:43:53.907 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:43:53.907 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.907 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:43:53.907 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:23.126018+0000", 2024-09-20T22:43:53.907 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:43:53.907 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.908 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.908 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:43:53.908 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:23.126018+0000", 2024-09-20T22:43:53.908 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:43:53.908 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.908 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.908 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:43:53.908 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:23.126018+0000", 2024-09-20T22:43:53.908 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.6600000000000003e-07 2024-09-20T22:43:53.908 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.908 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.908 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:43:53.908 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:23.126022+0000", 2024-09-20T22:43:53.908 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.8500000000000004e-06 2024-09-20T22:43:53.909 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.909 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.909 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:43:53.909 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:23.126027+0000", 2024-09-20T22:43:53.909 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.0529999999999997e-06 2024-09-20T22:43:53.909 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.909 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.909 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:43:53.909 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:23.126031+0000", 2024-09-20T22:43:53.909 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.5369999999999999e-06 2024-09-20T22:43:53.909 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.909 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.909 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:43:53.909 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:23.126066+0000", 2024-09-20T22:43:53.910 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.4431999999999998e-05 2024-09-20T22:43:53.910 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.910 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.910 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:43:53.910 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:23.126080+0000", 2024-09-20T22:43:53.910 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.473e-05 2024-09-20T22:43:53.910 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.910 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.910 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:43:53.910 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:23.133518+0000", 2024-09-20T22:43:53.910 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0074375489999999999 2024-09-20T22:43:53.910 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.910 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.910 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:43:53.910 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:23.133557+0000", 2024-09-20T22:43:53.911 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.9258000000000003e-05 2024-09-20T22:43:53.911 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:43:53.911 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:43:53.911 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:43:53.911 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.911 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.911 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:522 7.9 7:9970b578:::smithi08477203-15:head [copy-from ver 28] snapc f=[f,d] ondisk+write+known_if_redirected+supports_pool_eio e124)", 2024-09-20T22:43:53.911 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:43:23.126433+0000", 2024-09-20T22:43:53.911 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 30.715890582, 2024-09-20T22:43:53.911 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.027726022999999999, 2024-09-20T22:43:53.911 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:43:53.911 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:43:53.911 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:43:53.911 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:43:53.912 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:43:53.912 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:43:53.912 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 522 2024-09-20T22:43:53.912 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.912 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:43:53.912 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.912 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:43:53.912 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:23.126433+0000", 2024-09-20T22:43:53.912 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:43:53.912 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.912 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.912 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:43:53.912 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:23.126433+0000", 2024-09-20T22:43:53.912 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:43:53.912 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.912 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.913 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:43:53.913 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:23.126434+0000", 2024-09-20T22:43:53.913 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3200000000000001e-06 2024-09-20T22:43:53.913 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.913 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.913 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:43:53.913 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:23.126438+0000", 2024-09-20T22:43:53.913 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.6959999999999998e-06 2024-09-20T22:43:53.913 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.913 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.913 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:43:53.913 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:23.126448+0000", 2024-09-20T22:43:53.913 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0436e-05 2024-09-20T22:43:53.913 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.914 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.914 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:43:53.914 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:23.126455+0000", 2024-09-20T22:43:53.914 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.7399999999999998e-06 2024-09-20T22:43:53.914 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.914 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.914 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:43:53.914 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:23.126505+0000", 2024-09-20T22:43:53.914 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.9622e-05 2024-09-20T22:43:53.914 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.914 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.914 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:43:53.914 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:23.126598+0000", 2024-09-20T22:43:53.914 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.3905000000000003e-05 2024-09-20T22:43:53.914 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.915 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.915 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 1,4", 2024-09-20T22:43:53.915 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:23.136852+0000", 2024-09-20T22:43:53.915 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010253372 2024-09-20T22:43:53.915 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.915 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.915 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:43:53.915 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:23.146781+0000", 2024-09-20T22:43:53.915 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0099292019999999998 2024-09-20T22:43:53.915 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.915 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.915 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:43:53.915 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:23.146838+0000", 2024-09-20T22:43:53.915 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.7046999999999998e-05 2024-09-20T22:43:53.915 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.916 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.916 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:43:53.916 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:23.154054+0000", 2024-09-20T22:43:53.916 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0072155910000000004 2024-09-20T22:43:53.916 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.916 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.916 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:43:53.916 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:23.154100+0000", 2024-09-20T22:43:53.916 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.6613999999999998e-05 2024-09-20T22:43:53.916 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.916 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.916 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:43:53.916 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:23.154159+0000", 2024-09-20T22:43:53.916 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.8477999999999998e-05 2024-09-20T22:43:53.916 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:43:53.917 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:43:53.917 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:43:53.917 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.917 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.917 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:523 7.9 7:9970b578:::smithi08477203-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e124)", 2024-09-20T22:43:53.917 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:43:23.126473+0000", 2024-09-20T22:43:53.917 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 30.715850466999999, 2024-09-20T22:43:53.917 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.015897446999999999, 2024-09-20T22:43:53.917 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:43:53.917 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:43:53.917 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-20T22:43:53.917 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:43:53.917 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:43:53.917 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:43:53.917 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 523 2024-09-20T22:43:53.918 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.918 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:43:53.918 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.918 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:43:53.918 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:23.126473+0000", 2024-09-20T22:43:53.918 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:43:53.918 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.918 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.918 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:43:53.918 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:23.126473+0000", 2024-09-20T22:43:53.918 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:43:53.918 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.918 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.918 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:43:53.919 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:23.126473+0000", 2024-09-20T22:43:53.919 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.5599999999999995e-07 2024-09-20T22:43:53.919 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.919 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.919 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:43:53.919 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:23.126476+0000", 2024-09-20T22:43:53.919 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.357e-06 2024-09-20T22:43:53.919 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.919 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.919 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:43:53.919 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:23.126482+0000", 2024-09-20T22:43:53.919 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.5899999999999996e-06 2024-09-20T22:43:53.919 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.919 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.919 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:43:53.920 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:23.126488+0000", 2024-09-20T22:43:53.920 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.6219999999999997e-06 2024-09-20T22:43:53.920 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.920 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.920 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:43:53.920 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:23.126755+0000", 2024-09-20T22:43:53.920 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026716100000000003 2024-09-20T22:43:53.920 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.920 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.920 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for blocked object", 2024-09-20T22:43:53.920 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:23.126808+0000", 2024-09-20T22:43:53.920 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.2859999999999999e-05 2024-09-20T22:43:53.920 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.920 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.920 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:43:53.921 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:23.142223+0000", 2024-09-20T22:43:53.921 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.015414773999999999 2024-09-20T22:43:53.921 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.921 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.921 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:43:53.921 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:23.142296+0000", 2024-09-20T22:43:53.921 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.3394999999999998e-05 2024-09-20T22:43:53.921 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.921 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.921 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:43:53.921 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:23.142370+0000", 2024-09-20T22:43:53.921 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.4132000000000001e-05 2024-09-20T22:43:53.921 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:43:53.921 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:43:53.921 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:43:53.922 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.922 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.922 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(osd.0.8:5 7.f 7:fa55e5ae:::smithi08477203-7:head [assert-version v28,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e124)", 2024-09-20T22:43:53.922 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:43:23.127044+0000", 2024-09-20T22:43:53.922 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 30.715279453000001, 2024-09-20T22:43:53.922 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0064647940000000003, 2024-09-20T22:43:53.922 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:43:53.922 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:43:53.922 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-20T22:43:53.922 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:43:53.922 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.922 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:43:53.922 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:23.127044+0000", 2024-09-20T22:43:53.922 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:43:53.923 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.923 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.923 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:43:53.923 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:23.127044+0000", 2024-09-20T22:43:53.923 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:43:53.923 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.923 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.923 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:43:53.923 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:23.127045+0000", 2024-09-20T22:43:53.923 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1620000000000001e-06 2024-09-20T22:43:53.923 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.923 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.923 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:43:53.923 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:23.127052+0000", 2024-09-20T22:43:53.923 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.8379999999999996e-06 2024-09-20T22:43:53.924 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.924 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.924 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:43:53.924 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:23.127063+0000", 2024-09-20T22:43:53.924 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1022000000000001e-05 2024-09-20T22:43:53.924 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.924 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.924 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:43:53.924 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:23.127071+0000", 2024-09-20T22:43:53.924 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.8550000000000002e-06 2024-09-20T22:43:53.924 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.924 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.924 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:43:53.924 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:23.127126+0000", 2024-09-20T22:43:53.924 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.5170000000000002e-05 2024-09-20T22:43:53.925 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.925 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.925 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:43:53.925 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:23.127193+0000", 2024-09-20T22:43:53.925 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.7466999999999996e-05 2024-09-20T22:43:53.925 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.925 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.925 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:43:53.925 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:23.133509+0000", 2024-09-20T22:43:53.925 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0063152800000000004 2024-09-20T22:43:53.925 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:43:53.925 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:43:53.925 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:43:53.925 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.925 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.926 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:792 7.f 7:f93102e8:::smithi08477203-37:head [write 618524~710869,stat] snapc 19=[17] ondisk+write+known_if_redirected+supports_pool_eio e145)", 2024-09-20T22:43:53.926 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:43:44.346227+0000", 2024-09-20T22:43:53.926 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 9.4960960500000002, 2024-09-20T22:43:53.926 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010284069, 2024-09-20T22:43:53.926 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:43:53.926 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:43:53.926 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:43:53.926 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:43:53.926 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:43:53.926 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:43:53.926 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 792 2024-09-20T22:43:53.926 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.926 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:43:53.926 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.926 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:43:53.927 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:44.346227+0000", 2024-09-20T22:43:53.927 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:43:53.927 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.927 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.927 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:43:53.927 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:44.346227+0000", 2024-09-20T22:43:53.927 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:43:53.927 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.927 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.927 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:43:53.927 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:44.346229+0000", 2024-09-20T22:43:53.927 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.9290000000000001e-06 2024-09-20T22:43:53.927 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.927 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.927 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:43:53.927 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:44.346616+0000", 2024-09-20T22:43:53.928 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00038728800000000001 2024-09-20T22:43:53.928 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.928 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.928 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:43:53.928 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:44.346635+0000", 2024-09-20T22:43:53.928 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.8380999999999999e-05 2024-09-20T22:43:53.928 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.928 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.928 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:43:53.928 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:44.346646+0000", 2024-09-20T22:43:53.928 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1106e-05 2024-09-20T22:43:53.928 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.928 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.928 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:43:53.928 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:44.346716+0000", 2024-09-20T22:43:53.929 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.0067000000000005e-05 2024-09-20T22:43:53.929 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.929 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.929 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:43:53.929 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:44.346832+0000", 2024-09-20T22:43:53.929 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000116221 2024-09-20T22:43:53.929 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.929 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.929 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 3,7", 2024-09-20T22:43:53.929 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:44.347050+0000", 2024-09-20T22:43:53.929 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000218011 2024-09-20T22:43:53.929 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.929 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.929 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:43:53.930 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:44.348787+0000", 2024-09-20T22:43:53.930 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00173689 2024-09-20T22:43:53.930 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.930 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.930 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:43:53.930 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:44.349085+0000", 2024-09-20T22:43:53.930 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00029818299999999999 2024-09-20T22:43:53.930 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.930 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.930 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:43:53.930 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:44.356381+0000", 2024-09-20T22:43:53.930 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0072956920000000003 2024-09-20T22:43:53.930 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.930 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.931 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:43:53.931 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:44.356449+0000", 2024-09-20T22:43:53.931 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.8219000000000005e-05 2024-09-20T22:43:53.931 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.931 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.931 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:43:53.931 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:44.356511+0000", 2024-09-20T22:43:53.931 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.2082000000000006e-05 2024-09-20T22:43:53.931 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:43:53.931 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:43:53.931 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:43:53.931 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.931 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.931 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:793 7.f 7:f93102e8:::smithi08477203-37:head [write 1857976~360548] snapc 19=[17] ondisk+write+known_if_redirected+supports_pool_eio e145)", 2024-09-20T22:43:53.931 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:43:44.351663+0000", 2024-09-20T22:43:53.932 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 9.4906603539999992, 2024-09-20T22:43:53.932 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.005264492, 2024-09-20T22:43:53.932 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:43:53.932 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:43:53.932 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:43:53.932 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:43:53.932 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:43:53.932 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:43:53.932 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 793 2024-09-20T22:43:53.932 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.932 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:43:53.932 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.932 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:43:53.932 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:44.351663+0000", 2024-09-20T22:43:53.932 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:43:53.933 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.933 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.933 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:43:53.933 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:44.351663+0000", 2024-09-20T22:43:53.933 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:43:53.933 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.933 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.933 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:43:53.933 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:44.351665+0000", 2024-09-20T22:43:53.933 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.368e-06 2024-09-20T22:43:53.933 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.933 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.933 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:43:53.933 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:44.351861+0000", 2024-09-20T22:43:53.934 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000196138 2024-09-20T22:43:53.934 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.934 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.934 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:43:53.934 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:44.351879+0000", 2024-09-20T22:43:53.934 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.7278e-05 2024-09-20T22:43:53.934 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.934 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.934 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:43:53.934 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:44.351887+0000", 2024-09-20T22:43:53.934 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.0320000000000003e-06 2024-09-20T22:43:53.934 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.934 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.934 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:43:53.935 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:44.351964+0000", 2024-09-20T22:43:53.935 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.6779999999999993e-05 2024-09-20T22:43:53.935 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.935 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.935 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:43:53.935 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:44.352075+0000", 2024-09-20T22:43:53.935 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00011124399999999999 2024-09-20T22:43:53.935 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.935 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.935 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 3,7", 2024-09-20T22:43:53.935 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:44.352243+0000", 2024-09-20T22:43:53.935 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00016773600000000001 2024-09-20T22:43:53.935 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.936 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.936 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:43:53.936 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:44.355710+0000", 2024-09-20T22:43:53.936 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.003467684 2024-09-20T22:43:53.936 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.936 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.936 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:43:53.936 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:44.356230+0000", 2024-09-20T22:43:53.936 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00051967199999999999 2024-09-20T22:43:53.936 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.936 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.936 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:43:53.936 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:44.356793+0000", 2024-09-20T22:43:53.936 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00056284599999999998 2024-09-20T22:43:53.936 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.937 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.937 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:43:53.937 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:44.356861+0000", 2024-09-20T22:43:53.937 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.8578000000000006e-05 2024-09-20T22:43:53.937 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.937 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.937 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:43:53.937 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:44.356927+0000", 2024-09-20T22:43:53.937 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.6136000000000005e-05 2024-09-20T22:43:53.937 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:43:53.937 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:43:53.937 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:43:53.937 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.937 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.937 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:794 7.f 7:f93102e8:::smithi08477203-37:head [setxattr _header (57) in=7b,truncate 2218524] snapc 19=[17] ondisk+write+known_if_redirected+supports_pool_eio e145)", 2024-09-20T22:43:53.938 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:43:44.351919+0000", 2024-09-20T22:43:53.938 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 9.4904038790000005, 2024-09-20T22:43:53.938 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.005197447, 2024-09-20T22:43:53.938 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:43:53.938 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:43:53.938 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:43:53.938 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:43:53.938 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:43:53.938 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:43:53.938 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 794 2024-09-20T22:43:53.938 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.938 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:43:53.938 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.938 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:43:53.939 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:44.351919+0000", 2024-09-20T22:43:53.939 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:43:53.939 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.939 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.939 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:43:53.939 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:44.351919+0000", 2024-09-20T22:43:53.939 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:43:53.939 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.939 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.939 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:43:53.939 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:44.351920+0000", 2024-09-20T22:43:53.939 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.37e-07 2024-09-20T22:43:53.939 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.939 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.939 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:43:53.940 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:44.351924+0000", 2024-09-20T22:43:53.940 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.9620000000000004e-06 2024-09-20T22:43:53.940 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.940 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.940 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:43:53.940 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:44.351930+0000", 2024-09-20T22:43:53.940 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.1619999999999996e-06 2024-09-20T22:43:53.940 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.940 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.940 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:43:53.940 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:44.351934+0000", 2024-09-20T22:43:53.940 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.495e-06 2024-09-20T22:43:53.940 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.940 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.940 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:43:53.941 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:44.352869+0000", 2024-09-20T22:43:53.941 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00093539199999999995 2024-09-20T22:43:53.941 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.941 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.941 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:43:53.941 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:44.352947+0000", 2024-09-20T22:43:53.941 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.7626000000000005e-05 2024-09-20T22:43:53.941 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.941 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.941 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 3,7", 2024-09-20T22:43:53.941 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:44.353041+0000", 2024-09-20T22:43:53.941 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.3857000000000006e-05 2024-09-20T22:43:53.941 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.941 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.941 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:43:53.942 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:44.355722+0000", 2024-09-20T22:43:53.942 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026817410000000001 2024-09-20T22:43:53.942 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.942 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.942 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:43:53.942 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:44.356287+0000", 2024-09-20T22:43:53.942 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00056446999999999997 2024-09-20T22:43:53.942 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.942 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.942 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:43:53.942 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:44.356987+0000", 2024-09-20T22:43:53.942 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00070023600000000002 2024-09-20T22:43:53.942 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.942 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.942 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:43:53.943 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:44.357050+0000", 2024-09-20T22:43:53.943 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.3189999999999996e-05 2024-09-20T22:43:53.943 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.943 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.943 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:43:53.943 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:44.357117+0000", 2024-09-20T22:43:53.943 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.6679000000000003e-05 2024-09-20T22:43:53.943 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:43:53.943 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:43:53.943 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:43:53.943 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.943 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.943 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:795 7.f 7:f93102e8:::smithi08477203-37:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e145)", 2024-09-20T22:43:53.943 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:43:44.351951+0000", 2024-09-20T22:43:53.944 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 9.4903724230000002, 2024-09-20T22:43:53.944 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.005547672, 2024-09-20T22:43:53.944 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:43:53.944 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:43:53.944 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-20T22:43:53.944 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:43:53.944 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:43:53.944 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:43:53.944 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 795 2024-09-20T22:43:53.944 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.944 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:43:53.944 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.944 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:43:53.944 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:44.351951+0000", 2024-09-20T22:43:53.944 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:43:53.945 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.945 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.945 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:43:53.945 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:44.351951+0000", 2024-09-20T22:43:53.945 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:43:53.945 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.945 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.945 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:43:53.945 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:44.351951+0000", 2024-09-20T22:43:53.945 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.4499999999999997e-07 2024-09-20T22:43:53.945 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.945 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.945 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:43:53.945 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:44.351954+0000", 2024-09-20T22:43:53.945 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.3609999999999999e-06 2024-09-20T22:43:53.946 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.946 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.946 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:43:53.946 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:44.351960+0000", 2024-09-20T22:43:53.946 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.3559999999999998e-06 2024-09-20T22:43:53.946 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.946 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.946 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:43:53.946 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:44.351965+0000", 2024-09-20T22:43:53.946 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.0030000000000002e-06 2024-09-20T22:43:53.946 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.946 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.946 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:43:53.946 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:44.353260+0000", 2024-09-20T22:43:53.946 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00129484 2024-09-20T22:43:53.947 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.947 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.947 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for rw locks", 2024-09-20T22:43:53.947 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:44.353342+0000", 2024-09-20T22:43:53.947 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.2126999999999995e-05 2024-09-20T22:43:53.947 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.947 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.947 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:43:53.947 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:44.357160+0000", 2024-09-20T22:43:53.947 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.003818166 2024-09-20T22:43:53.947 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.947 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.947 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:43:53.947 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:44.357265+0000", 2024-09-20T22:43:53.947 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00010433900000000001 2024-09-20T22:43:53.948 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.948 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.948 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:43:53.948 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:44.357499+0000", 2024-09-20T22:43:53.948 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000234035 2024-09-20T22:43:53.948 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:43:53.948 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:43:53.948 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:43:53.948 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.948 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.948 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:816 7.6 7:64bcd2fc:::smithi08477203-20:1a [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e147)", 2024-09-20T22:43:53.948 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:43:46.363799+0000", 2024-09-20T22:43:53.948 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 7.4785245939999996, 2024-09-20T22:43:53.948 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0051147170000000004, 2024-09-20T22:43:53.948 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:43:53.949 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:43:53.949 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "reached pg", 2024-09-20T22:43:53.949 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:43:53.949 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:43:53.949 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:43:53.949 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 816 2024-09-20T22:43:53.949 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.949 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:43:53.949 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.949 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:43:53.949 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:46.363799+0000", 2024-09-20T22:43:53.949 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:43:53.949 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.949 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.949 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:43:53.950 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:46.363799+0000", 2024-09-20T22:43:53.950 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:43:53.950 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.950 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.950 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:43:53.950 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:46.363799+0000", 2024-09-20T22:43:53.950 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.6199999999999998e-07 2024-09-20T22:43:53.950 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.950 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.950 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:43:53.950 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:46.363804+0000", 2024-09-20T22:43:53.950 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.7609999999999998e-06 2024-09-20T22:43:53.950 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.950 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.951 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:43:53.951 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:46.363810+0000", 2024-09-20T22:43:53.951 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.1789999999999996e-06 2024-09-20T22:43:53.951 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.951 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.951 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:43:53.951 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:46.363815+0000", 2024-09-20T22:43:53.951 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.2560000000000002e-06 2024-09-20T22:43:53.951 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.951 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.951 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:43:53.951 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:46.363983+0000", 2024-09-20T22:43:53.951 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00016761599999999999 2024-09-20T22:43:53.951 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.952 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.952 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for rw locks", 2024-09-20T22:43:53.952 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:46.364048+0000", 2024-09-20T22:43:53.952 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.4851999999999995e-05 2024-09-20T22:43:53.952 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.952 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.952 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:43:53.952 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:46.368799+0000", 2024-09-20T22:43:53.952 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0047509600000000003 2024-09-20T22:43:53.952 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:43:53.952 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:43:53.952 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:43:53.952 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:46.368913+0000", 2024-09-20T22:43:53.952 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00011453100000000001 2024-09-20T22:43:53.952 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:43:53.953 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:43:53.953 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:43:53.953 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:43:53.953 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:43:53.953 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-20T22:43:53.953 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:43:53.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:53.977 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:53.977 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:43:54.078 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:43:54.084 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:43:54.100 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:54.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:54.101 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:43:54.131 INFO:tasks.rados.rados.0.smithi084.stdout:317: done (5 left) 2024-09-20T22:43:54.131 INFO:tasks.rados.rados.0.smithi084.stdout:318: read oid 39 snap 26 2024-09-20T22:43:54.131 INFO:tasks.rados.rados.0.smithi084.stdout:318: expect (ObjNum 110 snap 22 seq_num 110) 2024-09-20T22:43:54.131 INFO:tasks.rados.rados.0.smithi084.stdout:319: delete oid 31 current snap is 29 2024-09-20T22:43:54.132 INFO:tasks.rados.rados.0.smithi084.stdout:316: finishing write tid 1 to smithi08477203-17 2024-09-20T22:43:54.132 INFO:tasks.rados.rados.0.smithi084.stdout:313: finishing rollback tid 0 to smithi08477203-34 2024-09-20T22:43:54.132 INFO:tasks.rados.rados.0.smithi084.stdout:316: finishing write tid 2 to smithi08477203-17 2024-09-20T22:43:54.132 INFO:tasks.rados.rados.0.smithi084.stdout:316: finishing write tid 3 to smithi08477203-17 2024-09-20T22:43:54.132 INFO:tasks.rados.rados.0.smithi084.stdout:315: expect (ObjNum 141 snap 27 seq_num 141) 2024-09-20T22:43:54.201 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:43:54.216 INFO:tasks.rados.rados.0.smithi084.stdout:316: finishing write tid 4 to smithi08477203-17 2024-09-20T22:43:54.217 INFO:tasks.rados.rados.0.smithi084.stdout:316: oid 17 updating version 0 to 40 2024-09-20T22:43:54.217 INFO:tasks.rados.rados.0.smithi084.stdout:316: oid 17 updating version 40 to 41 2024-09-20T22:43:54.217 INFO:tasks.rados.rados.0.smithi084.stdout:316: oid 17 version 41 is already newer than 39 2024-09-20T22:43:54.217 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 17 v 41 (ObjNum 144 snap 29 seq_num 144) dirty exists 2024-09-20T22:43:54.217 INFO:tasks.rados.rados.0.smithi084.stdout:316: left oid 17 (ObjNum 144 snap 29 seq_num 144) 2024-09-20T22:43:54.218 INFO:tasks.rados.rados.0.smithi084.stdout:314: expect (ObjNum 133 snap 24 seq_num 133) 2024-09-20T22:43:54.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:54.224 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:54.224 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:43:54.316 INFO:tasks.rados.rados.0.smithi084.stdout:313: finishing rollback tid 1 to smithi08477203-34 2024-09-20T22:43:54.316 INFO:tasks.rados.rados.0.smithi084.stdout:312: finishing copy_from to smithi08477203-29 2024-09-20T22:43:54.316 INFO:tasks.rados.rados.0.smithi084.stdout:312: got expected ENOENT (src dne) 2024-09-20T22:43:54.316 INFO:tasks.rados.rados.0.smithi084.stdout:312: finishing copy_from racing read to smithi08477203-29 2024-09-20T22:43:54.316 INFO:tasks.rados.rados.0.smithi084.stdout:313: finishing rollback tid 2 to smithi08477203-34 2024-09-20T22:43:54.316 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 34 v 103 (ObjNum 50 snap 0 seq_num 50) dirty exists 2024-09-20T22:43:54.316 INFO:tasks.rados.rados.0.smithi084.stdout:312: done (6 left) 2024-09-20T22:43:54.316 INFO:tasks.rados.rados.0.smithi084.stdout:313: done (5 left) 2024-09-20T22:43:54.316 INFO:tasks.rados.rados.0.smithi084.stdout:314: done (4 left) 2024-09-20T22:43:54.316 INFO:tasks.rados.rados.0.smithi084.stdout:315: done (3 left) 2024-09-20T22:43:54.317 INFO:tasks.rados.rados.0.smithi084.stdout:316: done (2 left) 2024-09-20T22:43:54.317 INFO:tasks.rados.rados.0.smithi084.stdout:319: done (1 left) 2024-09-20T22:43:54.317 INFO:tasks.rados.rados.0.smithi084.stdout:320: rollback oid 28 current snap is 29 2024-09-20T22:43:54.317 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 28 to 26 2024-09-20T22:43:54.317 INFO:tasks.rados.rados.0.smithi084.stdout:318: expect (ObjNum 110 snap 22 seq_num 110) 2024-09-20T22:43:54.325 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:43:54.347 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:54.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:54.348 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:43:54.446 INFO:tasks.rados.rados.0.smithi084.stdout:320: finishing rollback tid 0 to smithi08477203-28 2024-09-20T22:43:54.446 INFO:tasks.rados.rados.0.smithi084.stdout:320: finishing rollback tid 1 to smithi08477203-28 2024-09-20T22:43:54.446 INFO:tasks.rados.rados.0.smithi084.stdout:318: done (1 left) 2024-09-20T22:43:54.446 INFO:tasks.rados.rados.0.smithi084.stdout:321: read oid 30 snap -1 2024-09-20T22:43:54.446 INFO:tasks.rados.rados.0.smithi084.stdout:321: expect (ObjNum 108 snap 22 seq_num 108) 2024-09-20T22:43:54.446 INFO:tasks.rados.rados.0.smithi084.stdout:322: copy_from oid 23 from oid 32 current snap is 29 2024-09-20T22:43:54.447 INFO:tasks.rados.rados.0.smithi084.stdout:320: finishing rollback tid 2 to smithi08477203-28 2024-09-20T22:43:54.447 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 28 v 144 (ObjNum 136 snap 25 seq_num 136) dirty exists 2024-09-20T22:43:54.447 INFO:tasks.rados.rados.0.smithi084.stdout:320: done (2 left) 2024-09-20T22:43:54.447 INFO:tasks.rados.rados.0.smithi084.stdout:323: snap_remove snap 26 2024-09-20T22:43:54.448 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:43:54.473 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:54.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:54.474 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:43:54.575 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:43:54.596 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:54.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:54.597 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:43:54.614 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:54 smithi084 ceph-mon[29774]: 3.c scrub starts 2024-09-20T22:43:54.614 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:54 smithi084 ceph-mon[29774]: 3.c scrub ok 2024-09-20T22:43:54.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:54 smithi084 ceph-mon[29774]: 6.1 scrub starts 2024-09-20T22:43:54.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:54 smithi084 ceph-mon[29774]: 6.1 scrub ok 2024-09-20T22:43:54.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:54 smithi084 ceph-mon[29774]: 6.1c deep-scrub starts 2024-09-20T22:43:54.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:54 smithi084 ceph-mon[29774]: osdmap e155: 8 total, 8 up, 8 in 2024-09-20T22:43:54.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:54 smithi084 ceph-mon[29774]: 6.1c deep-scrub ok 2024-09-20T22:43:54.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:54 smithi084 ceph-mon[40024]: 3.c scrub starts 2024-09-20T22:43:54.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:54 smithi084 ceph-mon[40024]: 3.c scrub ok 2024-09-20T22:43:54.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:54 smithi084 ceph-mon[40024]: 6.1 scrub starts 2024-09-20T22:43:54.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:54 smithi084 ceph-mon[40024]: 6.1 scrub ok 2024-09-20T22:43:54.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:54 smithi084 ceph-mon[40024]: 6.1c deep-scrub starts 2024-09-20T22:43:54.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:54 smithi084 ceph-mon[40024]: osdmap e155: 8 total, 8 up, 8 in 2024-09-20T22:43:54.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:54 smithi084 ceph-mon[40024]: 6.1c deep-scrub ok 2024-09-20T22:43:54.698 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:43:54.719 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:54.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:54.720 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:43:54.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:54 smithi154 ceph-mon[33264]: 3.c scrub starts 2024-09-20T22:43:54.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:54 smithi154 ceph-mon[33264]: 3.c scrub ok 2024-09-20T22:43:54.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:54 smithi154 ceph-mon[33264]: 6.1 scrub starts 2024-09-20T22:43:54.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:54 smithi154 ceph-mon[33264]: 6.1 scrub ok 2024-09-20T22:43:54.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:54 smithi154 ceph-mon[33264]: 6.1c deep-scrub starts 2024-09-20T22:43:54.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:54 smithi154 ceph-mon[33264]: osdmap e155: 8 total, 8 up, 8 in 2024-09-20T22:43:54.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:54 smithi154 ceph-mon[33264]: 6.1c deep-scrub ok 2024-09-20T22:43:54.821 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:43:54.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:54.842 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:54.843 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:43:54.893 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-20T22:43:54.943 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:43:54.966 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:54.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:54.967 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:43:55.068 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:43:55.089 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:55.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:55.090 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:43:55.191 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:43:55.194 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:43:55.215 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:55.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:55.216 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:43:55.317 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:43:55.338 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:55.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:55.339 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:43:55.387 INFO:tasks.rados.rados.0.smithi084.stdout:323: done (2 left) 2024-09-20T22:43:55.387 INFO:tasks.rados.rados.0.smithi084.stdout:324: write (excl) oid 38 current snap is 29 2024-09-20T22:43:55.387 INFO:tasks.rados.rados.0.smithi084.stdout:324: seq_num 146 ranges {453822=797061,1760371=653366} 2024-09-20T22:43:55.394 INFO:tasks.rados.rados.0.smithi084.stdout:324: writing smithi08477203-38 from 453822 to 1250883 tid 1 2024-09-20T22:43:55.402 INFO:tasks.rados.rados.0.smithi084.stdout:324: writing smithi08477203-38 from 1760371 to 2413737 tid 2 2024-09-20T22:43:55.403 INFO:tasks.rados.rados.0.smithi084.stdout:322: finishing copy_from racing read to smithi08477203-23 2024-09-20T22:43:55.403 INFO:tasks.rados.rados.0.smithi084.stdout:321: expect (ObjNum 108 snap 22 seq_num 108) 2024-09-20T22:43:55.414 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:43:55.440 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:43:55.465 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:55.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:55.466 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:43:55.495 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:55 smithi084 ceph-mon[40024]: pgmap v133: 178 pgs: 178 active+clean; 114 MiB data, 501 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:43:55.495 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:55 smithi084 ceph-mon[40024]: 6.d scrub starts 2024-09-20T22:43:55.495 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:55 smithi084 ceph-mon[40024]: 6.d scrub ok 2024-09-20T22:43:55.495 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:55 smithi084 ceph-mon[40024]: 3.14 scrub starts 2024-09-20T22:43:55.495 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:55 smithi084 ceph-mon[40024]: 3.14 scrub ok 2024-09-20T22:43:55.495 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:55 smithi084 ceph-mon[40024]: 6.b deep-scrub starts 2024-09-20T22:43:55.496 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:55 smithi084 ceph-mon[40024]: 6.b deep-scrub ok 2024-09-20T22:43:55.496 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:55 smithi084 ceph-mon[29774]: pgmap v133: 178 pgs: 178 active+clean; 114 MiB data, 501 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:43:55.496 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:55 smithi084 ceph-mon[29774]: 6.d scrub starts 2024-09-20T22:43:55.496 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:55 smithi084 ceph-mon[29774]: 6.d scrub ok 2024-09-20T22:43:55.496 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:55 smithi084 ceph-mon[29774]: 3.14 scrub starts 2024-09-20T22:43:55.496 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:55 smithi084 ceph-mon[29774]: 3.14 scrub ok 2024-09-20T22:43:55.496 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:55 smithi084 ceph-mon[29774]: 6.b deep-scrub starts 2024-09-20T22:43:55.496 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:55 smithi084 ceph-mon[29774]: 6.b deep-scrub ok 2024-09-20T22:43:55.551 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-20T22:43:55.566 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:43:55.591 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:55.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:55.592 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:43:55.625 INFO:tasks.rados.rados.0.smithi084.stdout:322: finishing copy_from to smithi08477203-23 2024-09-20T22:43:55.626 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 23 v 8 (ObjNum 106 snap 22 seq_num 106) dirty exists 2024-09-20T22:43:55.626 INFO:tasks.rados.rados.0.smithi084.stdout:324: finishing write tid 1 to smithi08477203-38 2024-09-20T22:43:55.626 INFO:tasks.rados.rados.0.smithi084.stdout:321: done (2 left) 2024-09-20T22:43:55.626 INFO:tasks.rados.rados.0.smithi084.stdout:322: done (1 left) 2024-09-20T22:43:55.626 INFO:tasks.rados.rados.0.smithi084.stdout:325: copy_from oid 17 from oid 27 current snap is 29 2024-09-20T22:43:55.626 INFO:tasks.rados.rados.0.smithi084.stdout:324: finishing write tid 2 to smithi08477203-38 2024-09-20T22:43:55.626 INFO:tasks.rados.rados.0.smithi084.stdout:324: finishing write tid 3 to smithi08477203-38 2024-09-20T22:43:55.626 INFO:tasks.rados.rados.0.smithi084.stdout:324: finishing write tid 4 to smithi08477203-38 2024-09-20T22:43:55.626 INFO:tasks.rados.rados.0.smithi084.stdout:324: oid 38 updating version 0 to 129 2024-09-20T22:43:55.626 INFO:tasks.rados.rados.0.smithi084.stdout:324: oid 38 version 129 is already newer than 128 2024-09-20T22:43:55.626 INFO:tasks.rados.rados.0.smithi084.stdout:324: oid 38 version 129 is already newer than 125 2024-09-20T22:43:55.626 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 38 v 129 (ObjNum 146 snap 29 seq_num 146) dirty exists 2024-09-20T22:43:55.627 INFO:tasks.rados.rados.0.smithi084.stdout:324: left oid 38 (ObjNum 146 snap 29 seq_num 146) 2024-09-20T22:43:55.627 INFO:tasks.rados.rados.0.smithi084.stdout:324: done (1 left) 2024-09-20T22:43:55.627 INFO:tasks.rados.rados.0.smithi084.stdout:326: read oid 39 snap -1 2024-09-20T22:43:55.627 INFO:tasks.rados.rados.0.smithi084.stdout:326: expect (ObjNum 110 snap 22 seq_num 110) 2024-09-20T22:43:55.627 INFO:tasks.rados.rados.0.smithi084.stdout:327: read oid 40 snap -1 2024-09-20T22:43:55.627 INFO:tasks.rados.rados.0.smithi084.stdout:327: expect (ObjNum 39 snap 0 seq_num 39) 2024-09-20T22:43:55.627 INFO:tasks.rados.rados.0.smithi084.stdout:328: read oid 8 snap -1 2024-09-20T22:43:55.627 INFO:tasks.rados.rados.0.smithi084.stdout:328: expect deleted 2024-09-20T22:43:55.627 INFO:tasks.rados.rados.0.smithi084.stdout:329: read oid 7 snap -1 2024-09-20T22:43:55.627 INFO:tasks.rados.rados.0.smithi084.stdout:329: expect deleted 2024-09-20T22:43:55.627 INFO:tasks.rados.rados.0.smithi084.stdout:330: snap_create 2024-09-20T22:43:55.630 INFO:tasks.rados.rados.0.smithi084.stdout:327: expect (ObjNum 39 snap 0 seq_num 39) 2024-09-20T22:43:55.692 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:43:55.717 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:55.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:55.718 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:43:55.724 INFO:tasks.rados.rados.0.smithi084.stdout:326: expect (ObjNum 110 snap 22 seq_num 110) 2024-09-20T22:43:55.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:55 smithi154 ceph-mon[33264]: pgmap v133: 178 pgs: 178 active+clean; 114 MiB data, 501 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:43:55.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:55 smithi154 ceph-mon[33264]: 6.d scrub starts 2024-09-20T22:43:55.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:55 smithi154 ceph-mon[33264]: 6.d scrub ok 2024-09-20T22:43:55.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:55 smithi154 ceph-mon[33264]: 3.14 scrub starts 2024-09-20T22:43:55.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:55 smithi154 ceph-mon[33264]: 3.14 scrub ok 2024-09-20T22:43:55.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:55 smithi154 ceph-mon[33264]: 6.b deep-scrub starts 2024-09-20T22:43:55.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:55 smithi154 ceph-mon[33264]: 6.b deep-scrub ok 2024-09-20T22:43:55.769 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:55 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mon-a[29770]: 2024-09-20T22:43:55.570+0000 7f26ad5e8640 -1 mon.a@0(leader).osd e156 definitely_dead 0 2024-09-20T22:43:55.819 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:43:55.843 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:55.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:55.844 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:43:55.852 INFO:tasks.rados.rados.0.smithi084.stdout:325: finishing copy_from racing read to smithi08477203-17 2024-09-20T22:43:55.853 INFO:tasks.rados.rados.0.smithi084.stdout:325: finishing copy_from to smithi08477203-17 2024-09-20T22:43:55.853 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 17 v 48 (ObjNum 116 snap 23 seq_num 116) dirty exists 2024-09-20T22:43:55.945 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:43:55.969 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:55.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:55.970 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:43:56.071 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:43:56.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:56.096 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:56.096 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:43:56.197 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:43:56.221 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:56.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:56.222 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:43:56.314 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-20T22:43:56.314 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2024-09-20T22:43:56.314 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2024-09-20T22:43:56.314 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-20T22:43:56.323 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:43:56.345 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:56.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:56.346 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:43:56.396 INFO:teuthology.orchestra.run.smithi084.stderr:marked out osd.4. 2024-09-20T22:43:56.396 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is set 2024-09-20T22:43:56.404 INFO:tasks.rados.rados.0.smithi084.stdout:325: done (5 left) 2024-09-20T22:43:56.404 INFO:tasks.rados.rados.0.smithi084.stdout:326: done (4 left) 2024-09-20T22:43:56.404 INFO:tasks.rados.rados.0.smithi084.stdout:327: done (3 left) 2024-09-20T22:43:56.404 INFO:tasks.rados.rados.0.smithi084.stdout:328: done (2 left) 2024-09-20T22:43:56.404 INFO:tasks.rados.rados.0.smithi084.stdout:329: done (1 left) 2024-09-20T22:43:56.404 INFO:tasks.rados.rados.0.smithi084.stdout:330: done (0 left) 2024-09-20T22:43:56.404 INFO:tasks.rados.rados.0.smithi084.stdout:331: snap_remove snap 29 2024-09-20T22:43:56.447 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:43:56.468 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:56.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:56.469 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:43:56.570 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:43:56.593 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:56.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:56.594 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:43:56.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:56 smithi084 ceph-mon[40024]: 5.a scrub starts 2024-09-20T22:43:56.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:56 smithi084 ceph-mon[40024]: 5.a scrub ok 2024-09-20T22:43:56.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:56 smithi084 ceph-mon[40024]: osdmap e156: 8 total, 8 up, 8 in 2024-09-20T22:43:56.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:56 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/3332086785' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["4"]}]: dispatch 2024-09-20T22:43:56.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:56 smithi084 ceph-mon[40024]: Client client.admin marked osd.4 out, while it was still marked up 2024-09-20T22:43:56.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:56 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/1020814782' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-20T22:43:56.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:56 smithi084 ceph-mon[29774]: 5.a scrub starts 2024-09-20T22:43:56.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:56 smithi084 ceph-mon[29774]: 5.a scrub ok 2024-09-20T22:43:56.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:56 smithi084 ceph-mon[29774]: osdmap e156: 8 total, 8 up, 8 in 2024-09-20T22:43:56.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:56 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/3332086785' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["4"]}]: dispatch 2024-09-20T22:43:56.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:56 smithi084 ceph-mon[29774]: Client client.admin marked osd.4 out, while it was still marked up 2024-09-20T22:43:56.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:56 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/1020814782' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-20T22:43:56.695 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:43:56.717 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:56.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:56.719 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:43:56.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:56 smithi154 ceph-mon[33264]: 5.a scrub starts 2024-09-20T22:43:56.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:56 smithi154 ceph-mon[33264]: 5.a scrub ok 2024-09-20T22:43:56.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:56 smithi154 ceph-mon[33264]: osdmap e156: 8 total, 8 up, 8 in 2024-09-20T22:43:56.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:56 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/3332086785' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["4"]}]: dispatch 2024-09-20T22:43:56.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:56 smithi154 ceph-mon[33264]: Client client.admin marked osd.4 out, while it was still marked up 2024-09-20T22:43:56.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:56 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/1020814782' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-20T22:43:56.819 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:43:56.841 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:56.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:56.842 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:43:56.942 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:43:56.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:56.965 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:56.965 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:43:57.066 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:43:57.089 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:57.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:57.090 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:43:57.190 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:43:57.214 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:57.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:57.215 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:43:57.315 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:43:57.337 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:57.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:57.338 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:43:57.406 INFO:tasks.rados.rados.0.smithi084.stdout:331: done (0 left) 2024-09-20T22:43:57.406 INFO:tasks.rados.rados.0.smithi084.stdout:332: rollback oid 4 current snap is 30 2024-09-20T22:43:57.406 INFO:tasks.rados.rados.0.smithi084.stdout:332: done (0 left) 2024-09-20T22:43:57.407 INFO:tasks.rados.rados.0.smithi084.stdout:333: write oid 33 current snap is 30 2024-09-20T22:43:57.407 INFO:tasks.rados.rados.0.smithi084.stdout:333: seq_num 147 ranges {470629=472310,1418853=513377} 2024-09-20T22:43:57.410 INFO:tasks.rados.rados.0.smithi084.stdout:333: writing smithi08477203-33 from 470629 to 942939 tid 1 2024-09-20T22:43:57.417 INFO:tasks.rados.rados.0.smithi084.stdout:333: writing smithi08477203-33 from 1418853 to 1932230 tid 2 2024-09-20T22:43:57.417 INFO:tasks.rados.rados.0.smithi084.stdout:334: write (excl) oid 30 current snap is 30 2024-09-20T22:43:57.417 INFO:tasks.rados.rados.0.smithi084.stdout:334: seq_num 148 ranges {487436=547559,1760982=657035} 2024-09-20T22:43:57.423 INFO:tasks.rados.rados.0.smithi084.stdout:334: writing smithi08477203-30 from 487436 to 1034995 tid 1 2024-09-20T22:43:57.433 INFO:tasks.rados.rados.0.smithi084.stdout:334: writing smithi08477203-30 from 1760982 to 2418017 tid 2 2024-09-20T22:43:57.433 INFO:tasks.rados.rados.0.smithi084.stdout:335: read oid 29 snap -1 2024-09-20T22:43:57.433 INFO:tasks.rados.rados.0.smithi084.stdout:335: expect (ObjNum 108 snap 22 seq_num 108) 2024-09-20T22:43:57.434 INFO:tasks.rados.rados.0.smithi084.stdout:336: rollback oid 7 current snap is 30 2024-09-20T22:43:57.434 INFO:tasks.rados.rados.0.smithi084.stdout:336: done (3 left) 2024-09-20T22:43:57.434 INFO:tasks.rados.rados.0.smithi084.stdout:337: delete oid 8 current snap is 30 2024-09-20T22:43:57.439 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:43:57.445 INFO:tasks.rados.rados.0.smithi084.stdout:333: finishing write tid 1 to smithi08477203-33 2024-09-20T22:43:57.445 INFO:tasks.rados.rados.0.smithi084.stdout:333: finishing write tid 2 to smithi08477203-33 2024-09-20T22:43:57.447 INFO:tasks.rados.rados.0.smithi084.stdout:333: finishing write tid 3 to smithi08477203-33 2024-09-20T22:43:57.447 INFO:tasks.rados.rados.0.smithi084.stdout:333: finishing write tid 4 to smithi08477203-33 2024-09-20T22:43:57.447 INFO:tasks.rados.rados.0.smithi084.stdout:333: oid 33 updating version 0 to 58 2024-09-20T22:43:57.447 INFO:tasks.rados.rados.0.smithi084.stdout:333: oid 33 version 58 is already newer than 57 2024-09-20T22:43:57.447 INFO:tasks.rados.rados.0.smithi084.stdout:333: oid 33 updating version 58 to 59 2024-09-20T22:43:57.447 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 33 v 59 (ObjNum 147 snap 30 seq_num 147) dirty exists 2024-09-20T22:43:57.447 INFO:tasks.rados.rados.0.smithi084.stdout:333: left oid 33 (ObjNum 147 snap 30 seq_num 147) 2024-09-20T22:43:57.447 INFO:tasks.rados.rados.0.smithi084.stdout:333: done (3 left) 2024-09-20T22:43:57.447 INFO:tasks.rados.rados.0.smithi084.stdout:337: done (2 left) 2024-09-20T22:43:57.448 INFO:tasks.rados.rados.0.smithi084.stdout:338: write (excl) oid 15 current snap is 30 2024-09-20T22:43:57.448 INFO:tasks.rados.rados.0.smithi084.stdout:338: seq_num 150 ranges {521050=698057,1877946=643104} 2024-09-20T22:43:57.454 INFO:tasks.rados.rados.0.smithi084.stdout:338: writing smithi08477203-15 from 521050 to 1219107 tid 1 2024-09-20T22:43:57.462 INFO:tasks.rados.rados.0.smithi084.stdout:338: writing smithi08477203-15 from 1877946 to 2521050 tid 2 2024-09-20T22:43:57.463 INFO:tasks.rados.rados.0.smithi084.stdout:334: finishing write tid 1 to smithi08477203-30 2024-09-20T22:43:57.463 INFO:tasks.rados.rados.0.smithi084.stdout:335: expect (ObjNum 108 snap 22 seq_num 108) 2024-09-20T22:43:57.474 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:57.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:57.475 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:43:57.576 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:43:57.598 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:57.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:57.599 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:43:57.671 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:57 smithi154 ceph-mon[33264]: pgmap v136: 178 pgs: 178 active+clean; 114 MiB data, 501 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:43:57.671 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:57 smithi154 ceph-mon[33264]: 5.d scrub starts 2024-09-20T22:43:57.671 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:57 smithi154 ceph-mon[33264]: 5.d scrub ok 2024-09-20T22:43:57.671 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:57 smithi154 ceph-mon[33264]: 4.5 scrub starts 2024-09-20T22:43:57.671 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:57 smithi154 ceph-mon[33264]: 4.5 scrub ok 2024-09-20T22:43:57.671 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:57 smithi154 ceph-mon[33264]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:43:57.671 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:57 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/3332086785' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["4"]}]': finished 2024-09-20T22:43:57.671 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:57 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/1020814782' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T22:43:57.671 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:57 smithi154 ceph-mon[33264]: osdmap e157: 8 total, 8 up, 7 in 2024-09-20T22:43:57.689 INFO:tasks.rados.rados.0.smithi084.stdout:334: finishing write tid 2 to smithi08477203-30 2024-09-20T22:43:57.689 INFO:tasks.rados.rados.0.smithi084.stdout:334: finishing write tid 3 to smithi08477203-30 2024-09-20T22:43:57.689 INFO:tasks.rados.rados.0.smithi084.stdout:334: finishing write tid 4 to smithi08477203-30 2024-09-20T22:43:57.689 INFO:tasks.rados.rados.0.smithi084.stdout:334: oid 30 updating version 0 to 82 2024-09-20T22:43:57.689 INFO:tasks.rados.rados.0.smithi084.stdout:334: oid 30 updating version 82 to 83 2024-09-20T22:43:57.689 INFO:tasks.rados.rados.0.smithi084.stdout:334: oid 30 updating version 83 to 84 2024-09-20T22:43:57.689 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 30 v 84 (ObjNum 148 snap 30 seq_num 148) dirty exists 2024-09-20T22:43:57.689 INFO:tasks.rados.rados.0.smithi084.stdout:334: left oid 30 (ObjNum 148 snap 30 seq_num 148) 2024-09-20T22:43:57.690 INFO:tasks.rados.rados.0.smithi084.stdout:338: finishing write tid 1 to smithi08477203-15 2024-09-20T22:43:57.690 INFO:tasks.rados.rados.0.smithi084.stdout:334: done (2 left) 2024-09-20T22:43:57.690 INFO:tasks.rados.rados.0.smithi084.stdout:335: done (1 left) 2024-09-20T22:43:57.690 INFO:tasks.rados.rados.0.smithi084.stdout:339: write (excl) oid 50 current snap is 30 2024-09-20T22:43:57.690 INFO:tasks.rados.rados.0.smithi084.stdout:338: finishing write tid 2 to smithi08477203-15 2024-09-20T22:43:57.690 INFO:tasks.rados.rados.0.smithi084.stdout:339: seq_num 151 ranges {537857=773306,1936428=520715} 2024-09-20T22:43:57.696 INFO:tasks.rados.rados.0.smithi084.stdout:339: writing smithi08477203-50 from 537857 to 1311163 tid 1 2024-09-20T22:43:57.700 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:43:57.703 INFO:tasks.rados.rados.0.smithi084.stdout:339: writing smithi08477203-50 from 1936428 to 2457143 tid 2 2024-09-20T22:43:57.703 INFO:tasks.rados.rados.0.smithi084.stdout:340: read oid 47 snap -1 2024-09-20T22:43:57.703 INFO:tasks.rados.rados.0.smithi084.stdout:340: expect (ObjNum 42 snap 0 seq_num 42) 2024-09-20T22:43:57.703 INFO:tasks.rados.rados.0.smithi084.stdout:341: rollback oid 46 current snap is 30 2024-09-20T22:43:57.703 INFO:tasks.rados.rados.0.smithi084.stdout:341: done (3 left) 2024-09-20T22:43:57.703 INFO:tasks.rados.rados.0.smithi084.stdout:342: write oid 46 current snap is 30 2024-09-20T22:43:57.703 INFO:tasks.rados.rados.0.smithi084.stdout:338: finishing write tid 3 to smithi08477203-15 2024-09-20T22:43:57.704 INFO:tasks.rados.rados.0.smithi084.stdout:338: finishing write tid 4 to smithi08477203-15 2024-09-20T22:43:57.704 INFO:tasks.rados.rados.0.smithi084.stdout:338: oid 15 updating version 0 to 87 2024-09-20T22:43:57.704 INFO:tasks.rados.rados.0.smithi084.stdout:338: oid 15 version 87 is already newer than 85 2024-09-20T22:43:57.704 INFO:tasks.rados.rados.0.smithi084.stdout:338: oid 15 version 87 is already newer than 86 2024-09-20T22:43:57.704 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 15 v 87 (ObjNum 150 snap 30 seq_num 150) dirty exists 2024-09-20T22:43:57.704 INFO:tasks.rados.rados.0.smithi084.stdout:338: left oid 15 (ObjNum 150 snap 30 seq_num 150) 2024-09-20T22:43:57.704 INFO:tasks.rados.rados.0.smithi084.stdout:342: seq_num 152 ranges {554664=448555,1478557=664373} 2024-09-20T22:43:57.708 INFO:tasks.rados.rados.0.smithi084.stdout:342: writing smithi08477203-46 from 554664 to 1003219 tid 1 2024-09-20T22:43:57.715 INFO:tasks.rados.rados.0.smithi084.stdout:342: writing smithi08477203-46 from 1478557 to 2142930 tid 2 2024-09-20T22:43:57.715 INFO:tasks.rados.rados.0.smithi084.stdout:338: done (3 left) 2024-09-20T22:43:57.715 INFO:tasks.rados.rados.0.smithi084.stdout:343: snap_create 2024-09-20T22:43:57.715 INFO:tasks.rados.rados.0.smithi084.stdout:339: finishing write tid 1 to smithi08477203-50 2024-09-20T22:43:57.715 INFO:tasks.rados.rados.0.smithi084.stdout:339: finishing write tid 2 to smithi08477203-50 2024-09-20T22:43:57.715 INFO:tasks.rados.rados.0.smithi084.stdout:339: finishing write tid 3 to smithi08477203-50 2024-09-20T22:43:57.715 INFO:tasks.rados.rados.0.smithi084.stdout:339: finishing write tid 4 to smithi08477203-50 2024-09-20T22:43:57.715 INFO:tasks.rados.rados.0.smithi084.stdout:339: oid 50 updating version 0 to 46 2024-09-20T22:43:57.715 INFO:tasks.rados.rados.0.smithi084.stdout:339: oid 50 version 46 is already newer than 45 2024-09-20T22:43:57.716 INFO:tasks.rados.rados.0.smithi084.stdout:339: oid 50 version 46 is already newer than 44 2024-09-20T22:43:57.716 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 50 v 46 (ObjNum 151 snap 30 seq_num 151) dirty exists 2024-09-20T22:43:57.716 INFO:tasks.rados.rados.0.smithi084.stdout:339: left oid 50 (ObjNum 151 snap 30 seq_num 151) 2024-09-20T22:43:57.716 INFO:tasks.rados.rados.0.smithi084.stdout:340: expect (ObjNum 42 snap 0 seq_num 42) 2024-09-20T22:43:57.724 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:57.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:57.725 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:43:57.803 INFO:tasks.rados.rados.0.smithi084.stdout:342: finishing write tid 1 to smithi08477203-46 2024-09-20T22:43:57.803 INFO:tasks.rados.rados.0.smithi084.stdout:342: finishing write tid 2 to smithi08477203-46 2024-09-20T22:43:57.803 INFO:tasks.rados.rados.0.smithi084.stdout:342: finishing write tid 3 to smithi08477203-46 2024-09-20T22:43:57.803 INFO:tasks.rados.rados.0.smithi084.stdout:342: finishing write tid 4 to smithi08477203-46 2024-09-20T22:43:57.803 INFO:tasks.rados.rados.0.smithi084.stdout:342: oid 46 updating version 0 to 89 2024-09-20T22:43:57.803 INFO:tasks.rados.rados.0.smithi084.stdout:342: oid 46 updating version 89 to 90 2024-09-20T22:43:57.803 INFO:tasks.rados.rados.0.smithi084.stdout:342: oid 46 version 90 is already newer than 88 2024-09-20T22:43:57.804 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 46 v 90 (ObjNum 152 snap 30 seq_num 152) dirty exists 2024-09-20T22:43:57.804 INFO:tasks.rados.rados.0.smithi084.stdout:342: left oid 46 (ObjNum 152 snap 30 seq_num 152) 2024-09-20T22:43:57.825 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:43:57.846 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:57.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:57.847 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:43:57.864 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:57 smithi084 ceph-mon[40024]: pgmap v136: 178 pgs: 178 active+clean; 114 MiB data, 501 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:43:57.864 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:57 smithi084 ceph-mon[40024]: 5.d scrub starts 2024-09-20T22:43:57.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:57 smithi084 ceph-mon[40024]: 5.d scrub ok 2024-09-20T22:43:57.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:57 smithi084 ceph-mon[40024]: 4.5 scrub starts 2024-09-20T22:43:57.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:57 smithi084 ceph-mon[40024]: 4.5 scrub ok 2024-09-20T22:43:57.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:57 smithi084 ceph-mon[40024]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:43:57.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:57 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/3332086785' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["4"]}]': finished 2024-09-20T22:43:57.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:57 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/1020814782' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T22:43:57.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:57 smithi084 ceph-mon[40024]: osdmap e157: 8 total, 8 up, 7 in 2024-09-20T22:43:57.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:57 smithi084 ceph-mon[29774]: pgmap v136: 178 pgs: 178 active+clean; 114 MiB data, 501 MiB used, 715 GiB / 715 GiB avail 2024-09-20T22:43:57.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:57 smithi084 ceph-mon[29774]: 5.d scrub starts 2024-09-20T22:43:57.866 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:57 smithi084 ceph-mon[29774]: 5.d scrub ok 2024-09-20T22:43:57.866 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:57 smithi084 ceph-mon[29774]: 4.5 scrub starts 2024-09-20T22:43:57.866 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:57 smithi084 ceph-mon[29774]: 4.5 scrub ok 2024-09-20T22:43:57.866 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:57 smithi084 ceph-mon[29774]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:43:57.866 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:57 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/3332086785' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["4"]}]': finished 2024-09-20T22:43:57.866 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:57 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/1020814782' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T22:43:57.866 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:57 smithi084 ceph-mon[29774]: osdmap e157: 8 total, 8 up, 7 in 2024-09-20T22:43:57.947 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:43:57.969 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:57.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:57.970 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:43:57.975 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.1 dump_blocked_ops 2024-09-20T22:43:58.070 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:43:58.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:58.092 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:58.092 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:43:58.141 INFO:tasks.rados.rados.0.smithi084.stdout:339: done (3 left) 2024-09-20T22:43:58.141 INFO:tasks.rados.rados.0.smithi084.stdout:340: done (2 left) 2024-09-20T22:43:58.141 INFO:tasks.rados.rados.0.smithi084.stdout:342: done (1 left) 2024-09-20T22:43:58.141 INFO:tasks.rados.rados.0.smithi084.stdout:343: done (0 left) 2024-09-20T22:43:58.141 INFO:tasks.rados.rados.0.smithi084.stdout:344: rollback oid 7 current snap is 31 2024-09-20T22:43:58.141 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 7 to 30 2024-09-20T22:43:58.141 INFO:tasks.rados.rados.0.smithi084.stdout:345: write oid 13 current snap is 31 2024-09-20T22:43:58.141 INFO:tasks.rados.rados.0.smithi084.stdout:345: seq_num 153 ranges {571471=640157,1653392=524384} 2024-09-20T22:43:58.147 INFO:tasks.rados.rados.0.smithi084.stdout:345: writing smithi08477203-13 from 571471 to 1211628 tid 1 2024-09-20T22:43:58.152 INFO:tasks.rados.rados.0.smithi084.stdout:345: writing smithi08477203-13 from 1653392 to 2177776 tid 2 2024-09-20T22:43:58.152 INFO:tasks.rados.rados.0.smithi084.stdout:346: copy_from oid 35 from oid 6 current snap is 31 2024-09-20T22:43:58.153 INFO:tasks.rados.rados.0.smithi084.stdout:344: finishing rollback tid 1 to smithi08477203-7 2024-09-20T22:43:58.153 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 7 v 0 (ObjNum 177 snap 0 seq_num 2091025152) dirty dne 2024-09-20T22:43:58.153 INFO:tasks.rados.rados.0.smithi084.stdout:345: finishing write tid 1 to smithi08477203-13 2024-09-20T22:43:58.153 INFO:tasks.rados.rados.0.smithi084.stdout:344: done (2 left) 2024-09-20T22:43:58.153 INFO:tasks.rados.rados.0.smithi084.stdout:347: snap_create 2024-09-20T22:43:58.162 INFO:tasks.rados.rados.0.smithi084.stdout:346: finishing copy_from racing read to smithi08477203-35 2024-09-20T22:43:58.167 INFO:tasks.rados.rados.0.smithi084.stdout:345: finishing write tid 2 to smithi08477203-13 2024-09-20T22:43:58.167 INFO:tasks.rados.rados.0.smithi084.stdout:345: finishing write tid 3 to smithi08477203-13 2024-09-20T22:43:58.167 INFO:tasks.rados.rados.0.smithi084.stdout:345: finishing write tid 4 to smithi08477203-13 2024-09-20T22:43:58.167 INFO:tasks.rados.rados.0.smithi084.stdout:345: oid 13 updating version 0 to 152 2024-09-20T22:43:58.167 INFO:tasks.rados.rados.0.smithi084.stdout:345: oid 13 version 152 is already newer than 151 2024-09-20T22:43:58.168 INFO:tasks.rados.rados.0.smithi084.stdout:345: oid 13 version 152 is already newer than 150 2024-09-20T22:43:58.168 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 13 v 152 (ObjNum 153 snap 31 seq_num 153) dirty exists 2024-09-20T22:43:58.168 INFO:tasks.rados.rados.0.smithi084.stdout:345: left oid 13 (ObjNum 153 snap 31 seq_num 153) 2024-09-20T22:43:58.169 INFO:tasks.rados.rados.0.smithi084.stdout:346: finishing copy_from to smithi08477203-35 2024-09-20T22:43:58.170 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 35 v 6 (ObjNum 42 snap 0 seq_num 42) dirty exists 2024-09-20T22:43:58.193 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:43:58.215 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:58.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:58.216 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:43:58.316 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:43:58.338 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:58.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:58.339 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:43:58.440 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:43:58.461 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:58.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:58.462 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:43:58.562 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:43:58.584 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:58.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:58.585 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:43:58.614 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:58 smithi084 ceph-mon[40024]: osdmap e158: 8 total, 8 up, 7 in 2024-09-20T22:43:58.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:58 smithi084 ceph-mon[40024]: osdmap e159: 8 total, 8 up, 7 in 2024-09-20T22:43:58.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:58 smithi084 ceph-mon[29774]: osdmap e158: 8 total, 8 up, 7 in 2024-09-20T22:43:58.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:58 smithi084 ceph-mon[29774]: osdmap e159: 8 total, 8 up, 7 in 2024-09-20T22:43:58.615 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:43:58 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: ::ffff:172.21.15.154 - - [20/Sep/2024:22:43:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-20T22:43:58.686 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:43:58.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:58.708 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:58.708 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:43:58.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:58 smithi154 ceph-mon[33264]: osdmap e158: 8 total, 8 up, 7 in 2024-09-20T22:43:58.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:58 smithi154 ceph-mon[33264]: osdmap e159: 8 total, 8 up, 7 in 2024-09-20T22:43:58.809 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:43:58.815 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:43:58.831 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:58.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:58.832 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:43:58.933 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:43:58.955 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:58.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:58.956 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:43:59.056 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:43:59.079 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:59.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:59.080 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:43:59.139 INFO:tasks.rados.rados.0.smithi084.stdout:345: done (2 left) 2024-09-20T22:43:59.139 INFO:tasks.rados.rados.0.smithi084.stdout:346: done (1 left) 2024-09-20T22:43:59.139 INFO:tasks.rados.rados.0.smithi084.stdout:347: done (0 left) 2024-09-20T22:43:59.139 INFO:tasks.rados.rados.0.smithi084.stdout:348: delete oid 45 current snap is 32 2024-09-20T22:43:59.143 INFO:tasks.rados.rados.0.smithi084.stdout:348: done (0 left) 2024-09-20T22:43:59.143 INFO:tasks.rados.rados.0.smithi084.stdout:349: write oid 19 current snap is 32 2024-09-20T22:43:59.143 INFO:tasks.rados.rados.0.smithi084.stdout:349: seq_num 155 ranges {605085=790655,2170356=434729} 2024-09-20T22:43:59.150 INFO:tasks.rados.rados.0.smithi084.stdout:349: writing smithi08477203-19 from 605085 to 1395740 tid 1 2024-09-20T22:43:59.157 INFO:tasks.rados.rados.0.smithi084.stdout:349: writing smithi08477203-19 from 2170356 to 2605085 tid 2 2024-09-20T22:43:59.157 INFO:tasks.rados.rados.0.smithi084.stdout:350: snap_create 2024-09-20T22:43:59.157 INFO:tasks.rados.rados.0.smithi084.stdout:349: finishing write tid 1 to smithi08477203-19 2024-09-20T22:43:59.159 INFO:tasks.rados.rados.0.smithi084.stdout:349: finishing write tid 2 to smithi08477203-19 2024-09-20T22:43:59.160 INFO:tasks.rados.rados.0.smithi084.stdout:349: finishing write tid 3 to smithi08477203-19 2024-09-20T22:43:59.160 INFO:tasks.rados.rados.0.smithi084.stdout:349: finishing write tid 4 to smithi08477203-19 2024-09-20T22:43:59.160 INFO:tasks.rados.rados.0.smithi084.stdout:349: oid 19 updating version 0 to 156 2024-09-20T22:43:59.160 INFO:tasks.rados.rados.0.smithi084.stdout:349: oid 19 version 156 is already newer than 155 2024-09-20T22:43:59.160 INFO:tasks.rados.rados.0.smithi084.stdout:349: oid 19 version 156 is already newer than 154 2024-09-20T22:43:59.160 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 19 v 156 (ObjNum 155 snap 32 seq_num 155) dirty exists 2024-09-20T22:43:59.160 INFO:tasks.rados.rados.0.smithi084.stdout:349: left oid 19 (ObjNum 155 snap 32 seq_num 155) 2024-09-20T22:43:59.180 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:43:59.202 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:59.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:59.203 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:43:59.254 INFO:journalctl@ceph.iscsi.iscsi.a.smithi154.stdout:Sep 20 22:43:59 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-iscsi-iscsi-a[59730]: debug there is no tcmu-runner data available 2024-09-20T22:43:59.304 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:43:59.326 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:59.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:59.327 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:43:59.428 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:43:59.440 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:43:59.440 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:43:59.451 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:43:59.451 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:43:59.451 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:59.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:59.452 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:43:59.469 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:43:59.470 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:43:59.487 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-20T22:43:59.487 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T22:43:59.505 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:43:59.505 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:43:59.514 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-20T22:43:59.527 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:43:59.527 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:43:59.528 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:43:59.546 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:43:59.546 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:43:59.553 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:43:59.568 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:43:59.568 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:43:59.578 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:59.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:59.579 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:43:59.614 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:59 smithi084 ceph-mon[40024]: pgmap v139: 178 pgs: 178 active+clean; 114 MiB data, 458 MiB used, 625 GiB / 626 GiB avail 2024-09-20T22:43:59.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:59 smithi084 ceph-mon[40024]: osdmap e160: 8 total, 8 up, 7 in 2024-09-20T22:43:59.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:43:59 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/1914100934' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:43:59.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:59 smithi084 ceph-mon[29774]: pgmap v139: 178 pgs: 178 active+clean; 114 MiB data, 458 MiB used, 625 GiB / 626 GiB avail 2024-09-20T22:43:59.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:59 smithi084 ceph-mon[29774]: osdmap e160: 8 total, 8 up, 7 in 2024-09-20T22:43:59.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:43:59 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/1914100934' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:43:59.680 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:43:59.702 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:59.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:59.703 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:43:59.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:59 smithi154 ceph-mon[33264]: pgmap v139: 178 pgs: 178 active+clean; 114 MiB data, 458 MiB used, 625 GiB / 626 GiB avail 2024-09-20T22:43:59.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:59 smithi154 ceph-mon[33264]: osdmap e160: 8 total, 8 up, 7 in 2024-09-20T22:43:59.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:43:59 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/1914100934' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:43:59.804 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:43:59.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:59.829 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:43:59.830 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:43:59.931 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:43:59.953 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:43:59.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:43:59.954 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:44:00.055 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:44:00.078 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:00.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:00.079 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:44:00.142 INFO:tasks.rados.rados.0.smithi084.stdout:349: done (1 left) 2024-09-20T22:44:00.142 INFO:tasks.rados.rados.0.smithi084.stdout:350: done (0 left) 2024-09-20T22:44:00.142 INFO:tasks.rados.rados.0.smithi084.stdout:351: write (excl) oid 46 current snap is 33 2024-09-20T22:44:00.142 INFO:tasks.rados.rados.0.smithi084.stdout:351: seq_num 156 ranges {621892=465904,1712485=671711} 2024-09-20T22:44:00.145 INFO:tasks.rados.rados.0.smithi084.stdout:351: writing smithi08477203-46 from 621892 to 1087796 tid 1 2024-09-20T22:44:00.153 INFO:tasks.rados.rados.0.smithi084.stdout:351: writing smithi08477203-46 from 1712485 to 2384196 tid 2 2024-09-20T22:44:00.153 INFO:tasks.rados.rados.0.smithi084.stdout:352: copy_from oid 40 from oid 14 current snap is 33 2024-09-20T22:44:00.153 INFO:tasks.rados.rados.0.smithi084.stdout:351: finishing write tid 1 to smithi08477203-46 2024-09-20T22:44:00.154 INFO:tasks.rados.rados.0.smithi084.stdout:353: read oid 6 snap -1 2024-09-20T22:44:00.154 INFO:tasks.rados.rados.0.smithi084.stdout:353: expect (ObjNum 42 snap 0 seq_num 42) 2024-09-20T22:44:00.154 INFO:tasks.rados.rados.0.smithi084.stdout:354: rollback oid 39 current snap is 33 2024-09-20T22:44:00.154 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 39 to 32 2024-09-20T22:44:00.154 INFO:tasks.rados.rados.0.smithi084.stdout:355: delete oid 42 current snap is 33 2024-09-20T22:44:00.157 INFO:tasks.rados.rados.0.smithi084.stdout:351: finishing write tid 2 to smithi08477203-46 2024-09-20T22:44:00.157 INFO:tasks.rados.rados.0.smithi084.stdout:354: finishing rollback tid 0 to smithi08477203-39 2024-09-20T22:44:00.157 INFO:tasks.rados.rados.0.smithi084.stdout:351: finishing write tid 3 to smithi08477203-46 2024-09-20T22:44:00.157 INFO:tasks.rados.rados.0.smithi084.stdout:353: expect (ObjNum 42 snap 0 seq_num 42) 2024-09-20T22:44:00.180 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:44:00.202 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:00.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:00.203 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:44:00.249 INFO:tasks.rados.rados.0.smithi084.stdout:354: finishing rollback tid 1 to smithi08477203-39 2024-09-20T22:44:00.249 INFO:tasks.rados.rados.0.smithi084.stdout:351: finishing write tid 4 to smithi08477203-46 2024-09-20T22:44:00.249 INFO:tasks.rados.rados.0.smithi084.stdout:351: oid 46 updating version 0 to 93 2024-09-20T22:44:00.249 INFO:tasks.rados.rados.0.smithi084.stdout:351: oid 46 version 93 is already newer than 91 2024-09-20T22:44:00.249 INFO:tasks.rados.rados.0.smithi084.stdout:351: oid 46 version 93 is already newer than 92 2024-09-20T22:44:00.249 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 46 v 93 (ObjNum 156 snap 33 seq_num 156) dirty exists 2024-09-20T22:44:00.250 INFO:tasks.rados.rados.0.smithi084.stdout:351: left oid 46 (ObjNum 156 snap 33 seq_num 156) 2024-09-20T22:44:00.250 INFO:tasks.rados.rados.0.smithi084.stdout:354: finishing rollback tid 2 to smithi08477203-39 2024-09-20T22:44:00.250 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 39 v 59 (ObjNum 110 snap 22 seq_num 110) dirty exists 2024-09-20T22:44:00.250 INFO:tasks.rados.rados.0.smithi084.stdout:352: finishing copy_from racing read to smithi08477203-40 2024-09-20T22:44:00.250 INFO:tasks.rados.rados.0.smithi084.stdout:352: finishing copy_from to smithi08477203-40 2024-09-20T22:44:00.250 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 40 v 3 (ObjNum 13 snap 0 seq_num 13) dirty exists 2024-09-20T22:44:00.250 INFO:tasks.rados.rados.0.smithi084.stdout:351: done (4 left) 2024-09-20T22:44:00.250 INFO:tasks.rados.rados.0.smithi084.stdout:352: done (3 left) 2024-09-20T22:44:00.250 INFO:tasks.rados.rados.0.smithi084.stdout:353: done (2 left) 2024-09-20T22:44:00.250 INFO:tasks.rados.rados.0.smithi084.stdout:354: done (1 left) 2024-09-20T22:44:00.250 INFO:tasks.rados.rados.0.smithi084.stdout:355: done (0 left) 2024-09-20T22:44:00.250 INFO:tasks.rados.rados.0.smithi084.stdout:356: snap_create 2024-09-20T22:44:00.304 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:44:00.325 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:00.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:00.326 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:44:00.427 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:44:00.452 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:00.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:00.453 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:44:00.554 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:44:00.575 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:00.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:00.576 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:44:00.677 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:44:00.702 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:00.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:00.703 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:44:00.734 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:00 smithi084 ceph-mon[29774]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:44:00.734 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:00 smithi084 ceph-mon[29774]: osdmap e161: 8 total, 8 up, 7 in 2024-09-20T22:44:00.734 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:00 smithi084 ceph-mon[40024]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:44:00.734 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:00 smithi084 ceph-mon[40024]: osdmap e161: 8 total, 8 up, 7 in 2024-09-20T22:44:00.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:00 smithi154 ceph-mon[33264]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:44:00.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:00 smithi154 ceph-mon[33264]: osdmap e161: 8 total, 8 up, 7 in 2024-09-20T22:44:00.758 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-20T22:44:00.758 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:44:00.804 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:44:00.830 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:00.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:00.831 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:44:00.931 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:44:00.956 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:00.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:00.957 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:44:01.058 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:44:01.080 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:01.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:01.081 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:44:01.140 INFO:tasks.rados.rados.0.smithi084.stdout:356: done (0 left) 2024-09-20T22:44:01.140 INFO:tasks.rados.rados.0.smithi084.stdout:357: read oid 33 snap -1 2024-09-20T22:44:01.140 INFO:tasks.rados.rados.0.smithi084.stdout:357: expect (ObjNum 147 snap 30 seq_num 147) 2024-09-20T22:44:01.140 INFO:tasks.rados.rados.0.smithi084.stdout:358: snap_remove snap 30 2024-09-20T22:44:01.183 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:44:01.209 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:01.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:01.210 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:44:01.311 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:44:01.332 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:01.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:01.333 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:44:01.378 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.1 dump_historic_ops 2024-09-20T22:44:01.427 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:44:01.434 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:44:01.459 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:01.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:01.460 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:44:01.561 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:44:01.584 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:01.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:01.585 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:44:01.614 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:01 smithi084 ceph-mon[40024]: pgmap v142: 178 pgs: 8 active+undersized+remapped, 170 active+clean; 116 MiB data, 498 MiB used, 625 GiB / 626 GiB avail; 27 MiB/s rd, 7.6 MiB/s wr, 47 op/s; 2.7 MiB/s, 25 objects/s recovering 2024-09-20T22:44:01.614 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:01 smithi084 ceph-mon[40024]: osdmap e162: 8 total, 8 up, 7 in 2024-09-20T22:44:01.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:01 smithi084 ceph-mon[29774]: pgmap v142: 178 pgs: 8 active+undersized+remapped, 170 active+clean; 116 MiB data, 498 MiB used, 625 GiB / 626 GiB avail; 27 MiB/s rd, 7.6 MiB/s wr, 47 op/s; 2.7 MiB/s, 25 objects/s recovering 2024-09-20T22:44:01.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:01 smithi084 ceph-mon[29774]: osdmap e162: 8 total, 8 up, 7 in 2024-09-20T22:44:01.685 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:44:01.708 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:01.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:01.709 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:44:01.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:01 smithi154 ceph-mon[33264]: pgmap v142: 178 pgs: 8 active+undersized+remapped, 170 active+clean; 116 MiB data, 498 MiB used, 625 GiB / 626 GiB avail; 27 MiB/s rd, 7.6 MiB/s wr, 47 op/s; 2.7 MiB/s, 25 objects/s recovering 2024-09-20T22:44:01.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:01 smithi154 ceph-mon[33264]: osdmap e162: 8 total, 8 up, 7 in 2024-09-20T22:44:01.810 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:44:01.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:01.836 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:01.836 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:44:01.937 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:44:01.963 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:01.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:01.964 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:44:01.966 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:44:02.066 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:44:02.087 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:02.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:02.088 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:44:02.145 INFO:tasks.rados.rados.0.smithi084.stdout:358: done (1 left) 2024-09-20T22:44:02.145 INFO:tasks.rados.rados.0.smithi084.stdout:359: copy_from oid 47 from oid 36 current snap is 34 2024-09-20T22:44:02.145 INFO:tasks.rados.rados.0.smithi084.stdout:357: expect (ObjNum 147 snap 30 seq_num 147) 2024-09-20T22:44:02.145 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is set 2024-09-20T22:44:02.189 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:44:02.214 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:02.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:02.215 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:44:02.272 INFO:tasks.rados.rados.0.smithi084.stdout:357: done (1 left) 2024-09-20T22:44:02.273 INFO:tasks.rados.rados.0.smithi084.stdout:360: read oid 48 snap -1 2024-09-20T22:44:02.273 INFO:tasks.rados.rados.0.smithi084.stdout:360: expect (ObjNum 129 snap 23 seq_num 129) 2024-09-20T22:44:02.273 INFO:tasks.rados.rados.0.smithi084.stdout:361: write (excl) oid 38 current snap is 34 2024-09-20T22:44:02.273 INFO:tasks.rados.rados.0.smithi084.stdout:361: seq_num 158 ranges {655506=616402,1829449=675380} 2024-09-20T22:44:02.279 INFO:tasks.rados.rados.0.smithi084.stdout:361: writing smithi08477203-38 from 655506 to 1271908 tid 1 2024-09-20T22:44:02.286 INFO:tasks.rados.rados.0.smithi084.stdout:361: writing smithi08477203-38 from 1829449 to 2504829 tid 2 2024-09-20T22:44:02.286 INFO:tasks.rados.rados.0.smithi084.stdout:362: write oid 49 current snap is 34 2024-09-20T22:44:02.286 INFO:tasks.rados.rados.0.smithi084.stdout:362: seq_num 159 ranges {672313=691651,1887931=419038} 2024-09-20T22:44:02.293 INFO:tasks.rados.rados.0.smithi084.stdout:362: writing smithi08477203-49 from 672313 to 1363964 tid 1 2024-09-20T22:44:02.298 INFO:tasks.rados.rados.0.smithi084.stdout:362: writing smithi08477203-49 from 1887931 to 2306969 tid 2 2024-09-20T22:44:02.299 INFO:tasks.rados.rados.0.smithi084.stdout:363: write oid 6 current snap is 34 2024-09-20T22:44:02.299 INFO:tasks.rados.rados.0.smithi084.stdout:363: seq_num 160 ranges {689120=483253,1946413=679049} 2024-09-20T22:44:02.306 INFO:tasks.rados.rados.0.smithi084.stdout:363: writing smithi08477203-6 from 689120 to 1172373 tid 1 2024-09-20T22:44:02.314 INFO:tasks.rados.rados.0.smithi084.stdout:363: writing smithi08477203-6 from 1946413 to 2625462 tid 2 2024-09-20T22:44:02.314 INFO:tasks.rados.rados.0.smithi084.stdout:364: snap_create 2024-09-20T22:44:02.314 INFO:tasks.rados.rados.0.smithi084.stdout:359: finishing copy_from racing read to smithi08477203-47 2024-09-20T22:44:02.314 INFO:tasks.rados.rados.0.smithi084.stdout:359: finishing copy_from to smithi08477203-47 2024-09-20T22:44:02.314 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 47 v 23 (ObjNum 35 snap 0 seq_num 35) dirty exists 2024-09-20T22:44:02.314 INFO:tasks.rados.rados.0.smithi084.stdout:360: expect (ObjNum 129 snap 23 seq_num 129) 2024-09-20T22:44:02.316 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:44:02.339 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:02.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:02.340 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:44:02.386 INFO:tasks.rados.rados.0.smithi084.stdout:361: finishing write tid 1 to smithi08477203-38 2024-09-20T22:44:02.386 INFO:tasks.rados.rados.0.smithi084.stdout:362: finishing write tid 1 to smithi08477203-49 2024-09-20T22:44:02.386 INFO:tasks.rados.rados.0.smithi084.stdout:361: finishing write tid 2 to smithi08477203-38 2024-09-20T22:44:02.386 INFO:tasks.rados.rados.0.smithi084.stdout:362: finishing write tid 2 to smithi08477203-49 2024-09-20T22:44:02.387 INFO:tasks.rados.rados.0.smithi084.stdout:361: finishing write tid 3 to smithi08477203-38 2024-09-20T22:44:02.387 INFO:tasks.rados.rados.0.smithi084.stdout:362: finishing write tid 3 to smithi08477203-49 2024-09-20T22:44:02.387 INFO:tasks.rados.rados.0.smithi084.stdout:361: finishing write tid 4 to smithi08477203-38 2024-09-20T22:44:02.387 INFO:tasks.rados.rados.0.smithi084.stdout:361: oid 38 updating version 0 to 133 2024-09-20T22:44:02.387 INFO:tasks.rados.rados.0.smithi084.stdout:361: oid 38 updating version 133 to 135 2024-09-20T22:44:02.387 INFO:tasks.rados.rados.0.smithi084.stdout:361: oid 38 version 135 is already newer than 134 2024-09-20T22:44:02.387 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 38 v 135 (ObjNum 158 snap 34 seq_num 158) dirty exists 2024-09-20T22:44:02.387 INFO:tasks.rados.rados.0.smithi084.stdout:361: left oid 38 (ObjNum 158 snap 34 seq_num 158) 2024-09-20T22:44:02.387 INFO:tasks.rados.rados.0.smithi084.stdout:362: finishing write tid 4 to smithi08477203-49 2024-09-20T22:44:02.387 INFO:tasks.rados.rados.0.smithi084.stdout:362: oid 49 updating version 0 to 112 2024-09-20T22:44:02.387 INFO:tasks.rados.rados.0.smithi084.stdout:362: oid 49 updating version 112 to 113 2024-09-20T22:44:02.387 INFO:tasks.rados.rados.0.smithi084.stdout:362: oid 49 version 113 is already newer than 111 2024-09-20T22:44:02.387 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 49 v 113 (ObjNum 159 snap 34 seq_num 159) dirty exists 2024-09-20T22:44:02.387 INFO:tasks.rados.rados.0.smithi084.stdout:362: left oid 49 (ObjNum 159 snap 34 seq_num 159) 2024-09-20T22:44:02.388 INFO:tasks.rados.rados.0.smithi084.stdout:363: finishing write tid 1 to smithi08477203-6 2024-09-20T22:44:02.388 INFO:tasks.rados.rados.0.smithi084.stdout:363: finishing write tid 2 to smithi08477203-6 2024-09-20T22:44:02.388 INFO:tasks.rados.rados.0.smithi084.stdout:363: finishing write tid 3 to smithi08477203-6 2024-09-20T22:44:02.388 INFO:tasks.rados.rados.0.smithi084.stdout:363: finishing write tid 4 to smithi08477203-6 2024-09-20T22:44:02.388 INFO:tasks.rados.rados.0.smithi084.stdout:363: oid 6 updating version 0 to 50 2024-09-20T22:44:02.388 INFO:tasks.rados.rados.0.smithi084.stdout:363: oid 6 updating version 50 to 51 2024-09-20T22:44:02.388 INFO:tasks.rados.rados.0.smithi084.stdout:363: oid 6 version 51 is already newer than 49 2024-09-20T22:44:02.388 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 6 v 51 (ObjNum 160 snap 34 seq_num 160) dirty exists 2024-09-20T22:44:02.388 INFO:tasks.rados.rados.0.smithi084.stdout:363: left oid 6 (ObjNum 160 snap 34 seq_num 160) 2024-09-20T22:44:02.440 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:44:02.453 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-20T22:44:02.463 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:02.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:02.464 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:44:02.565 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:44:02.587 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:02.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:02.588 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:44:02.689 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:44:02.708 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:02 smithi084 ceph-mon[40024]: purged_snaps scrub starts 2024-09-20T22:44:02.709 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:02 smithi084 ceph-mon[40024]: purged_snaps scrub ok 2024-09-20T22:44:02.709 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:02 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/840305451' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:44:02.709 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:02 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:44:02.709 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:02 smithi084 ceph-mon[40024]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:44:02.709 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:02 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T22:44:02.709 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:02 smithi084 ceph-mon[40024]: osdmap e163: 8 total, 8 up, 7 in 2024-09-20T22:44:02.709 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:02 smithi084 ceph-mon[29774]: purged_snaps scrub starts 2024-09-20T22:44:02.709 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:02 smithi084 ceph-mon[29774]: purged_snaps scrub ok 2024-09-20T22:44:02.709 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:02 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/840305451' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:44:02.710 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:02 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:44:02.710 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:02 smithi084 ceph-mon[29774]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:44:02.710 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:02 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T22:44:02.710 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:02 smithi084 ceph-mon[29774]: osdmap e163: 8 total, 8 up, 7 in 2024-09-20T22:44:02.710 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:02.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:02.711 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:44:02.712 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-20T22:44:02.712 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:44:02.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:02 smithi154 ceph-mon[33264]: purged_snaps scrub starts 2024-09-20T22:44:02.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:02 smithi154 ceph-mon[33264]: purged_snaps scrub ok 2024-09-20T22:44:02.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:02 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/840305451' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:44:02.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:02 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:44:02.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:02 smithi154 ceph-mon[33264]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:44:02.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:02 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T22:44:02.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:02 smithi154 ceph-mon[33264]: osdmap e163: 8 total, 8 up, 7 in 2024-09-20T22:44:02.813 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:44:02.834 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:02.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:02.835 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:44:02.935 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:44:02.957 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:02.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:02.958 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:44:03.059 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:44:03.081 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:03.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:03.082 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:44:03.129 INFO:tasks.rados.rados.0.smithi084.stdout:359: done (5 left) 2024-09-20T22:44:03.129 INFO:tasks.rados.rados.0.smithi084.stdout:360: done (4 left) 2024-09-20T22:44:03.129 INFO:tasks.rados.rados.0.smithi084.stdout:361: done (3 left) 2024-09-20T22:44:03.129 INFO:tasks.rados.rados.0.smithi084.stdout:362: done (2 left) 2024-09-20T22:44:03.129 INFO:tasks.rados.rados.0.smithi084.stdout:363: done (1 left) 2024-09-20T22:44:03.129 INFO:tasks.rados.rados.0.smithi084.stdout:364: done (0 left) 2024-09-20T22:44:03.129 INFO:tasks.rados.rados.0.smithi084.stdout:365: read oid 45 snap -1 2024-09-20T22:44:03.129 INFO:tasks.rados.rados.0.smithi084.stdout:365: expect deleted 2024-09-20T22:44:03.129 INFO:tasks.rados.rados.0.smithi084.stdout:366: delete oid 42 current snap is 35 2024-09-20T22:44:03.133 INFO:tasks.rados.rados.0.smithi084.stdout:365: done (1 left) 2024-09-20T22:44:03.134 INFO:tasks.rados.rados.0.smithi084.stdout:366: done (0 left) 2024-09-20T22:44:03.134 INFO:tasks.rados.rados.0.smithi084.stdout:367: rollback oid 3 current snap is 35 2024-09-20T22:44:03.134 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 3 to 32 2024-09-20T22:44:03.134 INFO:tasks.rados.rados.0.smithi084.stdout:368: write (excl) oid 38 current snap is 35 2024-09-20T22:44:03.134 INFO:tasks.rados.rados.0.smithi084.stdout:368: seq_num 162 ranges {722734=633751,2063377=659357} 2024-09-20T22:44:03.141 INFO:tasks.rados.rados.0.smithi084.stdout:368: writing smithi08477203-38 from 722734 to 1356485 tid 1 2024-09-20T22:44:03.149 INFO:tasks.rados.rados.0.smithi084.stdout:368: writing smithi08477203-38 from 2063377 to 2722734 tid 2 2024-09-20T22:44:03.149 INFO:tasks.rados.rados.0.smithi084.stdout:369: write oid 22 current snap is 35 2024-09-20T22:44:03.149 INFO:tasks.rados.rados.0.smithi084.stdout:369: seq_num 163 ranges {739541=709000,2121859=426376} 2024-09-20T22:44:03.157 INFO:tasks.rados.rados.0.smithi084.stdout:369: writing smithi08477203-22 from 739541 to 1448541 tid 1 2024-09-20T22:44:03.163 INFO:tasks.rados.rados.0.smithi084.stdout:369: writing smithi08477203-22 from 2121859 to 2548235 tid 2 2024-09-20T22:44:03.163 INFO:tasks.rados.rados.0.smithi084.stdout:370: write (excl) oid 8 current snap is 35 2024-09-20T22:44:03.163 INFO:tasks.rados.rados.0.smithi084.stdout:367: finishing rollback tid 0 to smithi08477203-3 2024-09-20T22:44:03.163 INFO:tasks.rados.rados.0.smithi084.stdout:370: seq_num 164 ranges {756348=784249,2063988=686387} 2024-09-20T22:44:03.172 INFO:tasks.rados.rados.0.smithi084.stdout:370: writing smithi08477203-8 from 756348 to 1540597 tid 1 2024-09-20T22:44:03.178 INFO:tasks.rados.rados.0.smithi084.stdout:370: writing smithi08477203-8 from 2063988 to 2750375 tid 2 2024-09-20T22:44:03.178 INFO:tasks.rados.rados.0.smithi084.stdout:371: snap_create 2024-09-20T22:44:03.179 INFO:tasks.rados.rados.0.smithi084.stdout:367: finishing rollback tid 1 to smithi08477203-3 2024-09-20T22:44:03.179 INFO:tasks.rados.rados.0.smithi084.stdout:367: finishing rollback tid 2 to smithi08477203-3 2024-09-20T22:44:03.179 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 3 v 85 (ObjNum 108 snap 22 seq_num 108) dirty exists 2024-09-20T22:44:03.179 INFO:tasks.rados.rados.0.smithi084.stdout:368: finishing write tid 1 to smithi08477203-38 2024-09-20T22:44:03.179 INFO:tasks.rados.rados.0.smithi084.stdout:368: finishing write tid 2 to smithi08477203-38 2024-09-20T22:44:03.179 INFO:tasks.rados.rados.0.smithi084.stdout:368: finishing write tid 3 to smithi08477203-38 2024-09-20T22:44:03.179 INFO:tasks.rados.rados.0.smithi084.stdout:368: finishing write tid 4 to smithi08477203-38 2024-09-20T22:44:03.179 INFO:tasks.rados.rados.0.smithi084.stdout:368: oid 38 updating version 0 to 137 2024-09-20T22:44:03.179 INFO:tasks.rados.rados.0.smithi084.stdout:368: oid 38 updating version 137 to 138 2024-09-20T22:44:03.179 INFO:tasks.rados.rados.0.smithi084.stdout:368: oid 38 updating version 138 to 139 2024-09-20T22:44:03.179 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 38 v 139 (ObjNum 162 snap 35 seq_num 162) dirty exists 2024-09-20T22:44:03.179 INFO:tasks.rados.rados.0.smithi084.stdout:368: left oid 38 (ObjNum 162 snap 35 seq_num 162) 2024-09-20T22:44:03.179 INFO:tasks.rados.rados.0.smithi084.stdout:369: finishing write tid 1 to smithi08477203-22 2024-09-20T22:44:03.180 INFO:tasks.rados.rados.0.smithi084.stdout:369: finishing write tid 2 to smithi08477203-22 2024-09-20T22:44:03.180 INFO:tasks.rados.rados.0.smithi084.stdout:369: finishing write tid 3 to smithi08477203-22 2024-09-20T22:44:03.180 INFO:tasks.rados.rados.0.smithi084.stdout:369: finishing write tid 4 to smithi08477203-22 2024-09-20T22:44:03.180 INFO:tasks.rados.rados.0.smithi084.stdout:369: oid 22 updating version 0 to 65 2024-09-20T22:44:03.180 INFO:tasks.rados.rados.0.smithi084.stdout:369: oid 22 version 65 is already newer than 64 2024-09-20T22:44:03.180 INFO:tasks.rados.rados.0.smithi084.stdout:369: oid 22 updating version 65 to 66 2024-09-20T22:44:03.180 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 22 v 66 (ObjNum 163 snap 35 seq_num 163) dirty exists 2024-09-20T22:44:03.180 INFO:tasks.rados.rados.0.smithi084.stdout:369: left oid 22 (ObjNum 163 snap 35 seq_num 163) 2024-09-20T22:44:03.180 INFO:tasks.rados.rados.0.smithi084.stdout:370: finishing write tid 1 to smithi08477203-8 2024-09-20T22:44:03.181 INFO:tasks.rados.rados.0.smithi084.stdout:370: finishing write tid 2 to smithi08477203-8 2024-09-20T22:44:03.181 INFO:tasks.rados.rados.0.smithi084.stdout:370: finishing write tid 3 to smithi08477203-8 2024-09-20T22:44:03.181 INFO:tasks.rados.rados.0.smithi084.stdout:370: finishing write tid 4 to smithi08477203-8 2024-09-20T22:44:03.181 INFO:tasks.rados.rados.0.smithi084.stdout:370: oid 8 updating version 0 to 87 2024-09-20T22:44:03.181 INFO:tasks.rados.rados.0.smithi084.stdout:370: oid 8 version 87 is already newer than 86 2024-09-20T22:44:03.181 INFO:tasks.rados.rados.0.smithi084.stdout:370: oid 8 updating version 87 to 88 2024-09-20T22:44:03.181 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 8 v 88 (ObjNum 164 snap 35 seq_num 164) dirty exists 2024-09-20T22:44:03.181 INFO:tasks.rados.rados.0.smithi084.stdout:370: left oid 8 (ObjNum 164 snap 35 seq_num 164) 2024-09-20T22:44:03.183 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:44:03.204 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:03.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:03.205 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:44:03.306 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:44:03.327 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:03.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:03.328 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:44:03.362 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-20T22:44:03.428 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:44:03.465 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:03.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:03.466 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:44:03.566 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:44:03.591 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:03.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:03.592 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:44:03.614 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:03 smithi084 ceph-mon[40024]: pgmap v145: 178 pgs: 8 active+undersized+remapped, 170 active+clean; 116 MiB data, 498 MiB used, 625 GiB / 626 GiB avail; 27 MiB/s rd, 7.6 MiB/s wr, 47 op/s; 2.7 MiB/s, 25 objects/s recovering 2024-09-20T22:44:03.614 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:03 smithi084 ceph-mon[40024]: osdmap e164: 8 total, 8 up, 7 in 2024-09-20T22:44:03.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:03 smithi084 ceph-mon[29774]: pgmap v145: 178 pgs: 8 active+undersized+remapped, 170 active+clean; 116 MiB data, 498 MiB used, 625 GiB / 626 GiB avail; 27 MiB/s rd, 7.6 MiB/s wr, 47 op/s; 2.7 MiB/s, 25 objects/s recovering 2024-09-20T22:44:03.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:03 smithi084 ceph-mon[29774]: osdmap e164: 8 total, 8 up, 7 in 2024-09-20T22:44:03.693 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:44:03.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:03.716 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:03.716 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:44:03.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:03 smithi154 ceph-mon[33264]: pgmap v145: 178 pgs: 8 active+undersized+remapped, 170 active+clean; 116 MiB data, 498 MiB used, 625 GiB / 626 GiB avail; 27 MiB/s rd, 7.6 MiB/s wr, 47 op/s; 2.7 MiB/s, 25 objects/s recovering 2024-09-20T22:44:03.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:03 smithi154 ceph-mon[33264]: osdmap e164: 8 total, 8 up, 7 in 2024-09-20T22:44:03.785 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 7] out_osds: [4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-20T22:44:03.785 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-20T22:44:03.786 INFO:tasks.thrashosds.thrasher:reset_purged_snaps_last 2024-09-20T22:44:03.786 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph tell osd.0 reset_purged_snaps_last 2024-09-20T22:44:03.817 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:44:03.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:03.853 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:03.853 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:44:03.870 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:44:03.889 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:44:03.955 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:44:03.976 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:03.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:03.977 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:44:04.078 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:44:04.104 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:04.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:04.105 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:44:04.133 INFO:tasks.rados.rados.0.smithi084.stdout:367: done (4 left) 2024-09-20T22:44:04.133 INFO:tasks.rados.rados.0.smithi084.stdout:368: done (3 left) 2024-09-20T22:44:04.133 INFO:tasks.rados.rados.0.smithi084.stdout:369: done (2 left) 2024-09-20T22:44:04.133 INFO:tasks.rados.rados.0.smithi084.stdout:370: done (1 left) 2024-09-20T22:44:04.133 INFO:tasks.rados.rados.0.smithi084.stdout:371: done (0 left) 2024-09-20T22:44:04.133 INFO:tasks.rados.rados.0.smithi084.stdout:372: delete oid 30 current snap is 36 2024-09-20T22:44:04.137 INFO:tasks.rados.rados.0.smithi084.stdout:372: done (0 left) 2024-09-20T22:44:04.137 INFO:tasks.rados.rados.0.smithi084.stdout:373: snap_remove snap 32 2024-09-20T22:44:04.205 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:44:04.231 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:04.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:04.232 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:44:04.333 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:44:04.358 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:04.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:04.359 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:44:04.460 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:44:04.482 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:04.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:04.483 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:44:04.533 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:44:04.533 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:44:04.542 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:44:04.542 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:44:04.560 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:44:04.560 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:44:04.576 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-20T22:44:04.576 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T22:44:04.584 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:44:04.593 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:44:04.593 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:44:04.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:04.606 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:04.607 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:44:04.610 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:44:04.610 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:44:04.627 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:44:04.628 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:44:04.644 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:44:04.644 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:44:04.707 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:44:04.729 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:04.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:04.730 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:44:04.831 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:44:04.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:04.852 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:04.852 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:44:04.953 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:44:04.979 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:04.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:04.980 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:44:05.081 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:44:05.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:05.107 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:05.107 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:44:05.131 INFO:tasks.rados.rados.0.smithi084.stdout:373: done (0 left) 2024-09-20T22:44:05.131 INFO:tasks.rados.rados.0.smithi084.stdout:374: copy_from oid 46 from oid 21 current snap is 36 2024-09-20T22:44:05.131 INFO:tasks.rados.rados.0.smithi084.stdout:375: read oid 29 snap -1 2024-09-20T22:44:05.131 INFO:tasks.rados.rados.0.smithi084.stdout:375: expect (ObjNum 108 snap 22 seq_num 108) 2024-09-20T22:44:05.131 INFO:tasks.rados.rados.0.smithi084.stdout:376: write (excl) oid 33 current snap is 36 2024-09-20T22:44:05.131 INFO:tasks.rados.rados.0.smithi084.stdout:376: seq_num 166 ranges {789962=534747,1780952=690056} 2024-09-20T22:44:05.139 INFO:tasks.rados.rados.0.smithi084.stdout:376: writing smithi08477203-33 from 789962 to 1324709 tid 1 2024-09-20T22:44:05.147 INFO:tasks.rados.rados.0.smithi084.stdout:376: writing smithi08477203-33 from 1780952 to 2471008 tid 2 2024-09-20T22:44:05.147 INFO:tasks.rados.rados.0.smithi084.stdout:377: read oid 27 snap -1 2024-09-20T22:44:05.147 INFO:tasks.rados.rados.0.smithi084.stdout:377: expect (ObjNum 116 snap 23 seq_num 116) 2024-09-20T22:44:05.147 INFO:tasks.rados.rados.0.smithi084.stdout:378: write oid 41 current snap is 36 2024-09-20T22:44:05.147 INFO:tasks.rados.rados.0.smithi084.stdout:378: seq_num 167 ranges {406769=609996,1439434=433714,2303522=503247} 2024-09-20T22:44:05.153 INFO:tasks.rados.rados.0.smithi084.stdout:378: writing smithi08477203-41 from 406769 to 1016765 tid 1 2024-09-20T22:44:05.158 INFO:tasks.rados.rados.0.smithi084.stdout:378: writing smithi08477203-41 from 1439434 to 1873148 tid 2 2024-09-20T22:44:05.163 INFO:tasks.rados.rados.0.smithi084.stdout:378: writing smithi08477203-41 from 2303522 to 2806769 tid 3 2024-09-20T22:44:05.164 INFO:tasks.rados.rados.0.smithi084.stdout:379: copy_from oid 18 from oid 14 current snap is 36 2024-09-20T22:44:05.164 INFO:tasks.rados.rados.0.smithi084.stdout:375: expect (ObjNum 108 snap 22 seq_num 108) 2024-09-20T22:44:05.209 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:44:05.212 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-20T22:44:05.212 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2024-09-20T22:44:05.212 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2024-09-20T22:44:05.212 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-20T22:44:05.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:05.233 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:05.233 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:44:05.334 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:44:05.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:05.357 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:05.358 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:44:05.364 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:05 smithi084 ceph-mon[40024]: pgmap v148: 178 pgs: 2 active+clean+snaptrim, 176 active+clean; 128 MiB data, 527 MiB used, 625 GiB / 626 GiB avail; 6.4 MiB/s rd, 3.5 MiB/s wr, 20 op/s 2024-09-20T22:44:05.364 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:05 smithi084 ceph-mon[40024]: osdmap e165: 8 total, 8 up, 7 in 2024-09-20T22:44:05.364 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:05 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/792163503' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:44:05.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:05 smithi084 ceph-mon[29774]: pgmap v148: 178 pgs: 2 active+clean+snaptrim, 176 active+clean; 128 MiB data, 527 MiB used, 625 GiB / 626 GiB avail; 6.4 MiB/s rd, 3.5 MiB/s wr, 20 op/s 2024-09-20T22:44:05.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:05 smithi084 ceph-mon[29774]: osdmap e165: 8 total, 8 up, 7 in 2024-09-20T22:44:05.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:05 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/792163503' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:44:05.381 INFO:tasks.rados.rados.0.smithi084.stdout:374: finishing copy_from racing read to smithi08477203-46 2024-09-20T22:44:05.381 INFO:tasks.rados.rados.0.smithi084.stdout:374: finishing copy_from to smithi08477203-46 2024-09-20T22:44:05.381 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 46 v 99 (ObjNum 143 snap 27 seq_num 143) dirty exists 2024-09-20T22:44:05.381 INFO:tasks.rados.rados.0.smithi084.stdout:376: finishing write tid 1 to smithi08477203-33 2024-09-20T22:44:05.381 INFO:tasks.rados.rados.0.smithi084.stdout:374: done (5 left) 2024-09-20T22:44:05.381 INFO:tasks.rados.rados.0.smithi084.stdout:375: done (4 left) 2024-09-20T22:44:05.381 INFO:tasks.rados.rados.0.smithi084.stdout:380: delete oid 42 current snap is 36 2024-09-20T22:44:05.381 INFO:tasks.rados.rados.0.smithi084.stdout:376: finishing write tid 2 to smithi08477203-33 2024-09-20T22:44:05.381 INFO:tasks.rados.rados.0.smithi084.stdout:377: expect (ObjNum 116 snap 23 seq_num 116) 2024-09-20T22:44:05.458 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:44:05.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:05.480 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:05.481 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:44:05.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:05 smithi154 ceph-mon[33264]: pgmap v148: 178 pgs: 2 active+clean+snaptrim, 176 active+clean; 128 MiB data, 527 MiB used, 625 GiB / 626 GiB avail; 6.4 MiB/s rd, 3.5 MiB/s wr, 20 op/s 2024-09-20T22:44:05.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:05 smithi154 ceph-mon[33264]: osdmap e165: 8 total, 8 up, 7 in 2024-09-20T22:44:05.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:05 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/792163503' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:44:05.582 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:44:05.603 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:05.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:05.604 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:44:05.616 INFO:tasks.rados.rados.0.smithi084.stdout:376: finishing write tid 3 to smithi08477203-33 2024-09-20T22:44:05.616 INFO:tasks.rados.rados.0.smithi084.stdout:376: finishing write tid 4 to smithi08477203-33 2024-09-20T22:44:05.616 INFO:tasks.rados.rados.0.smithi084.stdout:376: oid 33 updating version 0 to 65 2024-09-20T22:44:05.617 INFO:tasks.rados.rados.0.smithi084.stdout:376: oid 33 updating version 65 to 66 2024-09-20T22:44:05.617 INFO:tasks.rados.rados.0.smithi084.stdout:376: oid 33 updating version 66 to 67 2024-09-20T22:44:05.617 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 33 v 67 (ObjNum 166 snap 36 seq_num 166) dirty exists 2024-09-20T22:44:05.617 INFO:tasks.rados.rados.0.smithi084.stdout:376: left oid 33 (ObjNum 166 snap 36 seq_num 166) 2024-09-20T22:44:05.617 INFO:tasks.rados.rados.0.smithi084.stdout:378: finishing write tid 1 to smithi08477203-41 2024-09-20T22:44:05.617 INFO:tasks.rados.rados.0.smithi084.stdout:378: finishing write tid 2 to smithi08477203-41 2024-09-20T22:44:05.617 INFO:tasks.rados.rados.0.smithi084.stdout:376: done (4 left) 2024-09-20T22:44:05.618 INFO:tasks.rados.rados.0.smithi084.stdout:377: done (3 left) 2024-09-20T22:44:05.618 INFO:tasks.rados.rados.0.smithi084.stdout:380: done (2 left) 2024-09-20T22:44:05.618 INFO:tasks.rados.rados.0.smithi084.stdout:381: read oid 3 snap -1 2024-09-20T22:44:05.618 INFO:tasks.rados.rados.0.smithi084.stdout:381: expect (ObjNum 108 snap 22 seq_num 108) 2024-09-20T22:44:05.618 INFO:tasks.rados.rados.0.smithi084.stdout:382: delete oid 34 current snap is 36 2024-09-20T22:44:05.618 INFO:tasks.rados.rados.0.smithi084.stdout:378: finishing write tid 3 to smithi08477203-41 2024-09-20T22:44:05.619 INFO:tasks.rados.rados.0.smithi084.stdout:378: finishing write tid 4 to smithi08477203-41 2024-09-20T22:44:05.619 INFO:tasks.rados.rados.0.smithi084.stdout:378: finishing write tid 5 to smithi08477203-41 2024-09-20T22:44:05.619 INFO:tasks.rados.rados.0.smithi084.stdout:378: oid 41 updating version 0 to 40 2024-09-20T22:44:05.619 INFO:tasks.rados.rados.0.smithi084.stdout:378: oid 41 updating version 40 to 42 2024-09-20T22:44:05.619 INFO:tasks.rados.rados.0.smithi084.stdout:378: oid 41 version 42 is already newer than 39 2024-09-20T22:44:05.619 INFO:tasks.rados.rados.0.smithi084.stdout:378: oid 41 version 42 is already newer than 41 2024-09-20T22:44:05.620 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 41 v 42 (ObjNum 167 snap 36 seq_num 167) dirty exists 2024-09-20T22:44:05.620 INFO:tasks.rados.rados.0.smithi084.stdout:378: left oid 41 (ObjNum 167 snap 36 seq_num 167) 2024-09-20T22:44:05.620 INFO:tasks.rados.rados.0.smithi084.stdout:379: finishing copy_from racing read to smithi08477203-18 2024-09-20T22:44:05.620 INFO:tasks.rados.rados.0.smithi084.stdout:379: finishing copy_from to smithi08477203-18 2024-09-20T22:44:05.620 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 18 v 3 (ObjNum 13 snap 0 seq_num 13) dirty exists 2024-09-20T22:44:05.621 INFO:tasks.rados.rados.0.smithi084.stdout:378: done (3 left) 2024-09-20T22:44:05.621 INFO:tasks.rados.rados.0.smithi084.stdout:379: done (2 left) 2024-09-20T22:44:05.622 INFO:tasks.rados.rados.0.smithi084.stdout:382: done (1 left) 2024-09-20T22:44:05.622 INFO:tasks.rados.rados.0.smithi084.stdout:383: rollback oid 39 current snap is 36 2024-09-20T22:44:05.622 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 39 to 33 2024-09-20T22:44:05.622 INFO:tasks.rados.rados.0.smithi084.stdout:384: rollback oid 34 current snap is 36 2024-09-20T22:44:05.622 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 34 to 33 2024-09-20T22:44:05.622 INFO:tasks.rados.rados.0.smithi084.stdout:385: write oid 11 current snap is 36 2024-09-20T22:44:05.622 INFO:tasks.rados.rados.0.smithi084.stdout:385: seq_num 170 ranges {457190=552096,1614880=581041,2785791=71399} 2024-09-20T22:44:05.624 INFO:tasks.rados.rados.0.smithi084.stdout:385: writing smithi08477203-11 from 457190 to 1009286 tid 1 2024-09-20T22:44:05.630 INFO:tasks.rados.rados.0.smithi084.stdout:385: writing smithi08477203-11 from 1614880 to 2195921 tid 2 2024-09-20T22:44:05.635 INFO:tasks.rados.rados.0.smithi084.stdout:385: writing smithi08477203-11 from 2785791 to 2857190 tid 3 2024-09-20T22:44:05.635 INFO:tasks.rados.rados.0.smithi084.stdout:386: read oid 12 snap 33 2024-09-20T22:44:05.635 INFO:tasks.rados.rados.0.smithi084.stdout:386: expect (ObjNum 94 snap 18 seq_num 94) 2024-09-20T22:44:05.635 INFO:tasks.rados.rados.0.smithi084.stdout:387: copy_from oid 21 from oid 33 current snap is 36 2024-09-20T22:44:05.635 INFO:tasks.rados.rados.0.smithi084.stdout:384: finishing rollback tid 1 to smithi08477203-34 2024-09-20T22:44:05.636 INFO:tasks.rados.rados.0.smithi084.stdout:384: finishing rollback tid 2 to smithi08477203-34 2024-09-20T22:44:05.636 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 34 v 119 (ObjNum 50 snap 0 seq_num 50) dirty exists 2024-09-20T22:44:05.636 INFO:tasks.rados.rados.0.smithi084.stdout:384: done (5 left) 2024-09-20T22:44:05.636 INFO:tasks.rados.rados.0.smithi084.stdout:388: snap_remove snap 35 2024-09-20T22:44:05.705 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:44:05.727 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:05.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:05.728 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:44:05.829 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:44:05.834 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd unset noscrub 2024-09-20T22:44:05.850 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:05.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:05.851 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:44:05.952 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:44:05.972 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:05.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:05.973 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:44:06.074 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:44:06.097 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:06.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:06.110 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:44:06.142 INFO:tasks.rados.rados.0.smithi084.stdout:388: done (5 left) 2024-09-20T22:44:06.142 INFO:tasks.rados.rados.0.smithi084.stdout:389: snap_create 2024-09-20T22:44:06.142 INFO:tasks.rados.rados.0.smithi084.stdout:381: expect (ObjNum 108 snap 22 seq_num 108) 2024-09-20T22:44:06.211 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:44:06.234 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:06.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:06.235 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:44:06.336 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:44:06.354 INFO:tasks.rados.rados.0.smithi084.stdout:383: finishing rollback tid 0 to smithi08477203-39 2024-09-20T22:44:06.354 INFO:tasks.rados.rados.0.smithi084.stdout:383: finishing rollback tid 1 to smithi08477203-39 2024-09-20T22:44:06.354 INFO:tasks.rados.rados.0.smithi084.stdout:383: finishing rollback tid 2 to smithi08477203-39 2024-09-20T22:44:06.354 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 39 v 68 (ObjNum 110 snap 22 seq_num 110) dirty exists 2024-09-20T22:44:06.355 INFO:tasks.rados.rados.0.smithi084.stdout:385: finishing write tid 1 to smithi08477203-11 2024-09-20T22:44:06.355 INFO:tasks.rados.rados.0.smithi084.stdout:385: finishing write tid 2 to smithi08477203-11 2024-09-20T22:44:06.355 INFO:tasks.rados.rados.0.smithi084.stdout:385: finishing write tid 3 to smithi08477203-11 2024-09-20T22:44:06.355 INFO:tasks.rados.rados.0.smithi084.stdout:385: finishing write tid 4 to smithi08477203-11 2024-09-20T22:44:06.355 INFO:tasks.rados.rados.0.smithi084.stdout:385: finishing write tid 5 to smithi08477203-11 2024-09-20T22:44:06.355 INFO:tasks.rados.rados.0.smithi084.stdout:385: oid 11 updating version 0 to 43 2024-09-20T22:44:06.355 INFO:tasks.rados.rados.0.smithi084.stdout:385: oid 11 version 43 is already newer than 42 2024-09-20T22:44:06.355 INFO:tasks.rados.rados.0.smithi084.stdout:385: oid 11 updating version 43 to 45 2024-09-20T22:44:06.355 INFO:tasks.rados.rados.0.smithi084.stdout:385: oid 11 version 45 is already newer than 44 2024-09-20T22:44:06.355 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 11 v 45 (ObjNum 170 snap 36 seq_num 170) dirty exists 2024-09-20T22:44:06.355 INFO:tasks.rados.rados.0.smithi084.stdout:385: left oid 11 (ObjNum 170 snap 36 seq_num 170) 2024-09-20T22:44:06.355 INFO:tasks.rados.rados.0.smithi084.stdout:386: expect (ObjNum 94 snap 18 seq_num 94) 2024-09-20T22:44:06.357 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:06.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:06.358 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:44:06.364 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:06 smithi084 ceph-mon[40024]: osdmap e166: 8 total, 8 up, 7 in 2024-09-20T22:44:06.364 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:06 smithi084 ceph-mon[29774]: osdmap e166: 8 total, 8 up, 7 in 2024-09-20T22:44:06.459 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:44:06.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:06.483 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:06.483 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:44:06.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:06 smithi154 ceph-mon[33264]: osdmap e166: 8 total, 8 up, 7 in 2024-09-20T22:44:06.505 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:44:06.529 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.2 dump_blocked_ops 2024-09-20T22:44:06.534 INFO:tasks.rados.rados.0.smithi084.stdout:387: finishing copy_from racing read to smithi08477203-21 2024-09-20T22:44:06.534 INFO:tasks.rados.rados.0.smithi084.stdout:387: finishing copy_from to smithi08477203-21 2024-09-20T22:44:06.534 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 21 v 67 (ObjNum 166 snap 36 seq_num 166) dirty exists 2024-09-20T22:44:06.584 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:44:06.607 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:06.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:06.608 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:44:06.709 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:44:06.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:06.732 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:06.733 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:44:06.833 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:44:06.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:06.856 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:06.857 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:44:06.958 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:44:06.980 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:06.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:06.981 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:44:07.081 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:44:07.103 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:07.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:07.104 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:44:07.146 INFO:tasks.rados.rados.0.smithi084.stdout:381: done (5 left) 2024-09-20T22:44:07.146 INFO:tasks.rados.rados.0.smithi084.stdout:383: done (4 left) 2024-09-20T22:44:07.146 INFO:tasks.rados.rados.0.smithi084.stdout:385: done (3 left) 2024-09-20T22:44:07.146 INFO:tasks.rados.rados.0.smithi084.stdout:386: done (2 left) 2024-09-20T22:44:07.146 INFO:tasks.rados.rados.0.smithi084.stdout:387: done (1 left) 2024-09-20T22:44:07.146 INFO:tasks.rados.rados.0.smithi084.stdout:389: done (0 left) 2024-09-20T22:44:07.146 INFO:tasks.rados.rados.0.smithi084.stdout:390: write oid 49 current snap is 37 2024-09-20T22:44:07.146 INFO:tasks.rados.rados.0.smithi084.stdout:390: seq_num 171 ranges {473997=627345,1673362=441052,2529567=344430} 2024-09-20T22:44:07.151 INFO:tasks.rados.rados.0.smithi084.stdout:390: writing smithi08477203-49 from 473997 to 1101342 tid 1 2024-09-20T22:44:07.157 INFO:tasks.rados.rados.0.smithi084.stdout:390: writing smithi08477203-49 from 1673362 to 2114414 tid 2 2024-09-20T22:44:07.162 INFO:tasks.rados.rados.0.smithi084.stdout:390: writing smithi08477203-49 from 2529567 to 2873997 tid 3 2024-09-20T22:44:07.162 INFO:tasks.rados.rados.0.smithi084.stdout:391: delete oid 18 current snap is 37 2024-09-20T22:44:07.162 INFO:tasks.rados.rados.0.smithi084.stdout:390: finishing write tid 1 to smithi08477203-49 2024-09-20T22:44:07.162 INFO:tasks.rados.rados.0.smithi084.stdout:390: finishing write tid 2 to smithi08477203-49 2024-09-20T22:44:07.163 INFO:tasks.rados.rados.0.smithi084.stdout:391: done (1 left) 2024-09-20T22:44:07.164 INFO:tasks.rados.rados.0.smithi084.stdout:392: snap_remove snap 33 2024-09-20T22:44:07.205 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:44:07.225 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:44:07.226 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:44:07.227 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:07.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:07.228 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:44:07.329 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:44:07.350 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:07.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:07.351 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:44:07.452 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:44:07.475 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:07.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:07.475 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:44:07.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:07 smithi154 ceph-mon[33264]: pgmap v151: 178 pgs: 2 active+clean+snaptrim, 176 active+clean; 128 MiB data, 527 MiB used, 625 GiB / 626 GiB avail; 6.4 MiB/s rd, 3.5 MiB/s wr, 20 op/s 2024-09-20T22:44:07.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:07 smithi154 ceph-mon[33264]: osdmap e167: 8 total, 8 up, 7 in 2024-09-20T22:44:07.577 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:44:07.600 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:07.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:07.602 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:44:07.614 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:07 smithi084 ceph-mon[40024]: pgmap v151: 178 pgs: 2 active+clean+snaptrim, 176 active+clean; 128 MiB data, 527 MiB used, 625 GiB / 626 GiB avail; 6.4 MiB/s rd, 3.5 MiB/s wr, 20 op/s 2024-09-20T22:44:07.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:07 smithi084 ceph-mon[40024]: osdmap e167: 8 total, 8 up, 7 in 2024-09-20T22:44:07.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:07 smithi084 ceph-mon[29774]: pgmap v151: 178 pgs: 2 active+clean+snaptrim, 176 active+clean; 128 MiB data, 527 MiB used, 625 GiB / 626 GiB avail; 6.4 MiB/s rd, 3.5 MiB/s wr, 20 op/s 2024-09-20T22:44:07.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:07 smithi084 ceph-mon[29774]: osdmap e167: 8 total, 8 up, 7 in 2024-09-20T22:44:07.702 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:44:07.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:07.726 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:07.727 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:44:07.743 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph tell osd.1 reset_purged_snaps_last 2024-09-20T22:44:07.771 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-20T22:44:07.827 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:44:07.851 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:07.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:07.853 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:44:07.954 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:44:07.977 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:07.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:07.978 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:44:08.079 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:44:08.102 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:08.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:08.104 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:44:08.160 INFO:tasks.rados.rados.0.smithi084.stdout:392: done (1 left) 2024-09-20T22:44:08.160 INFO:tasks.rados.rados.0.smithi084.stdout:393: snap_remove snap 31 2024-09-20T22:44:08.205 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:44:08.225 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:08.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:08.226 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:44:08.236 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:08 smithi084 ceph-mon[40024]: osdmap e168: 8 total, 8 up, 7 in 2024-09-20T22:44:08.236 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:08 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:44:08.236 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:08 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:44:08.237 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:08 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [3, 2]}]: dispatch 2024-09-20T22:44:08.237 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:08 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.11", "id": [3, 5]}]: dispatch 2024-09-20T22:44:08.237 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:08 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [3, 2]}]: dispatch 2024-09-20T22:44:08.237 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:08 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.3", "id": [0, 3]}]: dispatch 2024-09-20T22:44:08.237 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:08 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "7.2", "id": [3, 5]}]: dispatch 2024-09-20T22:44:08.237 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:08 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "7.13", "id": [7, 1]}]: dispatch 2024-09-20T22:44:08.237 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:08 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.11", "id": [3, 5]}]: dispatch 2024-09-20T22:44:08.237 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:08 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "7.15", "id": [2, 6]}]: dispatch 2024-09-20T22:44:08.237 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:08 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.3", "id": [0, 3]}]: dispatch 2024-09-20T22:44:08.237 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:08 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "7.2", "id": [3, 5]}]: dispatch 2024-09-20T22:44:08.237 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:08 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "7.13", "id": [7, 1]}]: dispatch 2024-09-20T22:44:08.237 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:08 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "7.15", "id": [2, 6]}]: dispatch 2024-09-20T22:44:08.238 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:08 smithi084 ceph-mon[29774]: osdmap e168: 8 total, 8 up, 7 in 2024-09-20T22:44:08.238 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:08 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:44:08.238 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:08 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:44:08.238 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:08 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [3, 2]}]: dispatch 2024-09-20T22:44:08.238 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:08 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.11", "id": [3, 5]}]: dispatch 2024-09-20T22:44:08.238 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:08 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [3, 2]}]: dispatch 2024-09-20T22:44:08.238 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:08 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.3", "id": [0, 3]}]: dispatch 2024-09-20T22:44:08.238 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:08 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "7.2", "id": [3, 5]}]: dispatch 2024-09-20T22:44:08.238 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:08 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "7.13", "id": [7, 1]}]: dispatch 2024-09-20T22:44:08.238 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:08 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.11", "id": [3, 5]}]: dispatch 2024-09-20T22:44:08.238 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:08 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "7.15", "id": [2, 6]}]: dispatch 2024-09-20T22:44:08.238 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:08 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.3", "id": [0, 3]}]: dispatch 2024-09-20T22:44:08.238 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:08 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "7.2", "id": [3, 5]}]: dispatch 2024-09-20T22:44:08.238 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:08 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "7.13", "id": [7, 1]}]: dispatch 2024-09-20T22:44:08.239 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:08 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "7.15", "id": [2, 6]}]: dispatch 2024-09-20T22:44:08.277 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:44:08.278 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:44:08.327 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:44:08.351 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:08.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:08.352 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:44:08.453 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:44:08.476 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:08.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:08.477 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:44:08.488 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:44:08 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: ::ffff:172.21.15.154 - - [20/Sep/2024:22:44:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-20T22:44:08.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:08 smithi154 ceph-mon[33264]: osdmap e168: 8 total, 8 up, 7 in 2024-09-20T22:44:08.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:08 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:44:08.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:08 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:44:08.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:08 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [3, 2]}]: dispatch 2024-09-20T22:44:08.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:08 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.11", "id": [3, 5]}]: dispatch 2024-09-20T22:44:08.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:08 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [3, 2]}]: dispatch 2024-09-20T22:44:08.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:08 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.3", "id": [0, 3]}]: dispatch 2024-09-20T22:44:08.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:08 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "7.2", "id": [3, 5]}]: dispatch 2024-09-20T22:44:08.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:08 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "7.13", "id": [7, 1]}]: dispatch 2024-09-20T22:44:08.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:08 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.11", "id": [3, 5]}]: dispatch 2024-09-20T22:44:08.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:08 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "7.15", "id": [2, 6]}]: dispatch 2024-09-20T22:44:08.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:08 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.3", "id": [0, 3]}]: dispatch 2024-09-20T22:44:08.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:08 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "7.2", "id": [3, 5]}]: dispatch 2024-09-20T22:44:08.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:08 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "7.13", "id": [7, 1]}]: dispatch 2024-09-20T22:44:08.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:08 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "7.15", "id": [2, 6]}]: dispatch 2024-09-20T22:44:08.578 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:44:08.600 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:08.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:08.601 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:44:08.702 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:44:08.724 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:08.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:08.725 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:44:08.826 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:44:08.847 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:08.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:08.848 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:44:08.949 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:44:08.989 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:08.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:08.990 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:44:09.091 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:44:09.112 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:09.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:09.113 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:44:09.160 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is unset 2024-09-20T22:44:09.169 INFO:tasks.rados.rados.0.smithi084.stdout:393: done (1 left) 2024-09-20T22:44:09.169 INFO:tasks.rados.rados.0.smithi084.stdout:394: write oid 28 current snap is 37 2024-09-20T22:44:09.169 INFO:tasks.rados.rados.0.smithi084.stdout:394: seq_num 173 ranges {507611=777843,2073973=444721} 2024-09-20T22:44:09.175 INFO:tasks.rados.rados.0.smithi084.stdout:394: writing smithi08477203-28 from 507611 to 1285454 tid 1 2024-09-20T22:44:09.184 INFO:tasks.rados.rados.0.smithi084.stdout:394: writing smithi08477203-28 from 2073973 to 2518694 tid 2 2024-09-20T22:44:09.184 INFO:tasks.rados.rados.0.smithi084.stdout:395: rollback oid 48 current snap is 37 2024-09-20T22:44:09.185 INFO:tasks.rados.rados.0.smithi084.stdout:390: finishing write tid 3 to smithi08477203-49 2024-09-20T22:44:09.185 INFO:tasks.rados.rados.0.smithi084.stdout:390: finishing write tid 4 to smithi08477203-49 2024-09-20T22:44:09.185 INFO:tasks.rados.rados.0.smithi084.stdout:390: finishing write tid 5 to smithi08477203-49 2024-09-20T22:44:09.185 INFO:tasks.rados.rados.0.smithi084.stdout:390: oid 49 updating version 0 to 127 2024-09-20T22:44:09.185 INFO:tasks.rados.rados.0.smithi084.stdout:390: oid 49 updating version 127 to 129 2024-09-20T22:44:09.185 INFO:tasks.rados.rados.0.smithi084.stdout:390: oid 49 version 129 is already newer than 128 2024-09-20T22:44:09.185 INFO:tasks.rados.rados.0.smithi084.stdout:390: oid 49 updating version 129 to 130 2024-09-20T22:44:09.185 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 49 v 130 (ObjNum 171 snap 37 seq_num 171) dirty exists 2024-09-20T22:44:09.186 INFO:tasks.rados.rados.0.smithi084.stdout:390: left oid 49 (ObjNum 171 snap 37 seq_num 171) 2024-09-20T22:44:09.186 INFO:tasks.rados.rados.0.smithi084.stdout:394: finishing write tid 1 to smithi08477203-28 2024-09-20T22:44:09.186 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 48 to 36 2024-09-20T22:44:09.186 INFO:tasks.rados.rados.0.smithi084.stdout:390: done (2 left) 2024-09-20T22:44:09.186 INFO:tasks.rados.rados.0.smithi084.stdout:396: rollback oid 3 current snap is 37 2024-09-20T22:44:09.186 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 3 to 36 2024-09-20T22:44:09.186 INFO:tasks.rados.rados.0.smithi084.stdout:397: read oid 37 snap 34 2024-09-20T22:44:09.186 INFO:tasks.rados.rados.0.smithi084.stdout:397: expect deleted 2024-09-20T22:44:09.186 INFO:tasks.rados.rados.0.smithi084.stdout:398: copy_from oid 7 from oid 15 current snap is 37 2024-09-20T22:44:09.186 INFO:tasks.rados.rados.0.smithi084.stdout:399: rollback oid 42 current snap is 37 2024-09-20T22:44:09.186 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 42 to 36 2024-09-20T22:44:09.186 INFO:tasks.rados.rados.0.smithi084.stdout:400: delete oid 10 current snap is 37 2024-09-20T22:44:09.187 INFO:tasks.rados.rados.0.smithi084.stdout:399: finishing rollback tid 1 to smithi08477203-42 2024-09-20T22:44:09.187 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 42 v 0 (ObjNum 289 snap 0 seq_num 1) dirty dne 2024-09-20T22:44:09.189 INFO:tasks.rados.rados.0.smithi084.stdout:394: finishing write tid 2 to smithi08477203-28 2024-09-20T22:44:09.190 INFO:tasks.rados.rados.0.smithi084.stdout:394: finishing write tid 3 to smithi08477203-28 2024-09-20T22:44:09.190 INFO:tasks.rados.rados.0.smithi084.stdout:394: finishing write tid 4 to smithi08477203-28 2024-09-20T22:44:09.190 INFO:tasks.rados.rados.0.smithi084.stdout:394: oid 28 updating version 0 to 166 2024-09-20T22:44:09.190 INFO:tasks.rados.rados.0.smithi084.stdout:394: oid 28 version 166 is already newer than 165 2024-09-20T22:44:09.190 INFO:tasks.rados.rados.0.smithi084.stdout:394: oid 28 version 166 is already newer than 164 2024-09-20T22:44:09.190 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 28 v 166 (ObjNum 173 snap 37 seq_num 173) dirty exists 2024-09-20T22:44:09.190 INFO:tasks.rados.rados.0.smithi084.stdout:394: left oid 28 (ObjNum 173 snap 37 seq_num 173) 2024-09-20T22:44:09.190 INFO:tasks.rados.rados.0.smithi084.stdout:395: finishing rollback tid 0 to smithi08477203-48 2024-09-20T22:44:09.191 INFO:tasks.rados.rados.0.smithi084.stdout:395: finishing rollback tid 1 to smithi08477203-48 2024-09-20T22:44:09.193 INFO:tasks.rados.rados.0.smithi084.stdout:394: done (6 left) 2024-09-20T22:44:09.193 INFO:tasks.rados.rados.0.smithi084.stdout:397: done (5 left) 2024-09-20T22:44:09.193 INFO:tasks.rados.rados.0.smithi084.stdout:399: done (4 left) 2024-09-20T22:44:09.193 INFO:tasks.rados.rados.0.smithi084.stdout:400: done (3 left) 2024-09-20T22:44:09.193 INFO:tasks.rados.rados.0.smithi084.stdout:401: read oid 29 snap 34 2024-09-20T22:44:09.193 INFO:tasks.rados.rados.0.smithi084.stdout:401: expect (ObjNum 108 snap 22 seq_num 108) 2024-09-20T22:44:09.193 INFO:tasks.rados.rados.0.smithi084.stdout:402: read oid 28 snap -1 2024-09-20T22:44:09.193 INFO:tasks.rados.rados.0.smithi084.stdout:402: expect (ObjNum 173 snap 37 seq_num 173) 2024-09-20T22:44:09.193 INFO:tasks.rados.rados.0.smithi084.stdout:403: read oid 34 snap 36 2024-09-20T22:44:09.194 INFO:tasks.rados.rados.0.smithi084.stdout:403: expect (ObjNum 50 snap 0 seq_num 50) 2024-09-20T22:44:09.194 INFO:tasks.rados.rados.0.smithi084.stdout:404: rollback oid 12 current snap is 37 2024-09-20T22:44:09.194 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 12 to 36 2024-09-20T22:44:09.194 INFO:tasks.rados.rados.0.smithi084.stdout:405: rollback oid 42 current snap is 37 2024-09-20T22:44:09.194 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 42 to 36 2024-09-20T22:44:09.194 INFO:tasks.rados.rados.0.smithi084.stdout:406: read oid 47 snap 36 2024-09-20T22:44:09.194 INFO:tasks.rados.rados.0.smithi084.stdout:406: expect (ObjNum 35 snap 0 seq_num 35) 2024-09-20T22:44:09.194 INFO:tasks.rados.rados.0.smithi084.stdout:407: read oid 24 snap -1 2024-09-20T22:44:09.194 INFO:tasks.rados.rados.0.smithi084.stdout:407: expect (ObjNum 104 snap 21 seq_num 104) 2024-09-20T22:44:09.194 INFO:tasks.rados.rados.0.smithi084.stdout:408: delete oid 5 current snap is 37 2024-09-20T22:44:09.195 INFO:tasks.rados.rados.0.smithi084.stdout:396: finishing rollback tid 0 to smithi08477203-3 2024-09-20T22:44:09.195 INFO:tasks.rados.rados.0.smithi084.stdout:395: finishing rollback tid 2 to smithi08477203-48 2024-09-20T22:44:09.195 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 48 v 70 (ObjNum 129 snap 23 seq_num 129) dirty exists 2024-09-20T22:44:09.195 INFO:tasks.rados.rados.0.smithi084.stdout:396: finishing rollback tid 1 to smithi08477203-3 2024-09-20T22:44:09.196 INFO:tasks.rados.rados.0.smithi084.stdout:396: finishing rollback tid 2 to smithi08477203-3 2024-09-20T22:44:09.196 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 3 v 91 (ObjNum 108 snap 22 seq_num 108) dirty exists 2024-09-20T22:44:09.198 INFO:tasks.rados.rados.0.smithi084.stdout:404: finishing rollback tid 0 to smithi08477203-12 2024-09-20T22:44:09.199 INFO:tasks.rados.rados.0.smithi084.stdout:404: finishing rollback tid 1 to smithi08477203-12 2024-09-20T22:44:09.199 INFO:tasks.rados.rados.0.smithi084.stdout:404: finishing rollback tid 2 to smithi08477203-12 2024-09-20T22:44:09.199 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 12 v 135 (ObjNum 94 snap 18 seq_num 94) dirty exists 2024-09-20T22:44:09.199 INFO:tasks.rados.rados.0.smithi084.stdout:395: done (10 left) 2024-09-20T22:44:09.199 INFO:tasks.rados.rados.0.smithi084.stdout:396: done (9 left) 2024-09-20T22:44:09.199 INFO:tasks.rados.rados.0.smithi084.stdout:404: done (8 left) 2024-09-20T22:44:09.199 INFO:tasks.rados.rados.0.smithi084.stdout:408: done (7 left) 2024-09-20T22:44:09.199 INFO:tasks.rados.rados.0.smithi084.stdout:409: rollback oid 9 current snap is 37 2024-09-20T22:44:09.199 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 9 to 34 2024-09-20T22:44:09.199 INFO:tasks.rados.rados.0.smithi084.stdout:410: read oid 4 snap -1 2024-09-20T22:44:09.200 INFO:tasks.rados.rados.0.smithi084.stdout:410: expect (ObjNum 121 snap 23 seq_num 121) 2024-09-20T22:44:09.200 INFO:tasks.rados.rados.0.smithi084.stdout:411: delete oid 16 current snap is 37 2024-09-20T22:44:09.202 INFO:tasks.rados.rados.0.smithi084.stdout:409: finishing rollback tid 0 to smithi08477203-9 2024-09-20T22:44:09.202 INFO:tasks.rados.rados.0.smithi084.stdout:411: done (9 left) 2024-09-20T22:44:09.202 INFO:tasks.rados.rados.0.smithi084.stdout:412: write (excl) oid 46 current snap is 37 2024-09-20T22:44:09.202 INFO:tasks.rados.rados.0.smithi084.stdout:412: seq_num 177 ranges {574839=795192,1907901=452059} 2024-09-20T22:44:09.213 INFO:tasks.rados.rados.0.smithi084.stdout:412: writing smithi08477203-46 from 574839 to 1370031 tid 1 2024-09-20T22:44:09.214 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:44:09.219 INFO:tasks.rados.rados.0.smithi084.stdout:412: writing smithi08477203-46 from 1907901 to 2359960 tid 2 2024-09-20T22:44:09.219 INFO:tasks.rados.rados.0.smithi084.stdout:409: finishing rollback tid 1 to smithi08477203-9 2024-09-20T22:44:09.219 INFO:tasks.rados.rados.0.smithi084.stdout:409: finishing rollback tid 2 to smithi08477203-9 2024-09-20T22:44:09.219 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 9 v 80 (ObjNum 58 snap 4 seq_num 58) dirty exists 2024-09-20T22:44:09.219 INFO:tasks.rados.rados.0.smithi084.stdout:398: finishing copy_from racing read to smithi08477203-7 2024-09-20T22:44:09.219 INFO:tasks.rados.rados.0.smithi084.stdout:407: expect (ObjNum 104 snap 21 seq_num 104) 2024-09-20T22:44:09.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:09.238 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:09.238 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:44:09.254 INFO:journalctl@ceph.iscsi.iscsi.a.smithi154.stdout:Sep 20 22:44:09 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-iscsi-iscsi-a[59730]: debug there is no tcmu-runner data available 2024-09-20T22:44:09.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:09 smithi154 ceph-mon[33264]: pgmap v154: 178 pgs: 2 active+clean+snaptrim, 176 active+clean; 128 MiB data, 527 MiB used, 625 GiB / 626 GiB avail 2024-09-20T22:44:09.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:09 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [3, 2]}]': finished 2024-09-20T22:44:09.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:09 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.11", "id": [3, 5]}]': finished 2024-09-20T22:44:09.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:09 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.3", "id": [0, 3]}]': finished 2024-09-20T22:44:09.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:09 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "7.2", "id": [3, 5]}]': finished 2024-09-20T22:44:09.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:09 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "7.13", "id": [7, 1]}]': finished 2024-09-20T22:44:09.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:09 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "7.15", "id": [2, 6]}]': finished 2024-09-20T22:44:09.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:09 smithi154 ceph-mon[33264]: osdmap e169: 8 total, 8 up, 7 in 2024-09-20T22:44:09.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:09 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/765502152' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:44:09.292 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-20T22:44:09.292 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2024-09-20T22:44:09.292 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2024-09-20T22:44:09.292 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2024-09-20T22:44:09.292 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-20T22:44:09.339 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:44:09.360 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:09.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:09.361 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:44:09.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:09 smithi084 ceph-mon[29774]: pgmap v154: 178 pgs: 2 active+clean+snaptrim, 176 active+clean; 128 MiB data, 527 MiB used, 625 GiB / 626 GiB avail 2024-09-20T22:44:09.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:09 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [3, 2]}]': finished 2024-09-20T22:44:09.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:09 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.11", "id": [3, 5]}]': finished 2024-09-20T22:44:09.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:09 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.3", "id": [0, 3]}]': finished 2024-09-20T22:44:09.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:09 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "7.2", "id": [3, 5]}]': finished 2024-09-20T22:44:09.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:09 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "7.13", "id": [7, 1]}]': finished 2024-09-20T22:44:09.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:09 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "7.15", "id": [2, 6]}]': finished 2024-09-20T22:44:09.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:09 smithi084 ceph-mon[29774]: osdmap e169: 8 total, 8 up, 7 in 2024-09-20T22:44:09.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:09 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/765502152' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:44:09.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:09 smithi084 ceph-mon[40024]: pgmap v154: 178 pgs: 2 active+clean+snaptrim, 176 active+clean; 128 MiB data, 527 MiB used, 625 GiB / 626 GiB avail 2024-09-20T22:44:09.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:09 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [3, 2]}]': finished 2024-09-20T22:44:09.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:09 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.11", "id": [3, 5]}]': finished 2024-09-20T22:44:09.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:09 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.3", "id": [0, 3]}]': finished 2024-09-20T22:44:09.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:09 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "7.2", "id": [3, 5]}]': finished 2024-09-20T22:44:09.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:09 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "7.13", "id": [7, 1]}]': finished 2024-09-20T22:44:09.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:09 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "7.15", "id": [2, 6]}]': finished 2024-09-20T22:44:09.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:09 smithi084 ceph-mon[40024]: osdmap e169: 8 total, 8 up, 7 in 2024-09-20T22:44:09.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:09 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/765502152' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:44:09.432 INFO:tasks.rados.rados.0.smithi084.stdout:401: expect (ObjNum 108 snap 22 seq_num 108) 2024-09-20T22:44:09.461 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:44:09.483 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:09.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:09.484 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:44:09.584 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:44:09.606 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:09.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:09.607 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:44:09.643 INFO:tasks.rados.rados.0.smithi084.stdout:405: finishing rollback tid 1 to smithi08477203-42 2024-09-20T22:44:09.643 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 42 v 0 (ObjNum 289 snap 0 seq_num 1) dirty dne 2024-09-20T22:44:09.643 INFO:tasks.rados.rados.0.smithi084.stdout:398: finishing copy_from to smithi08477203-7 2024-09-20T22:44:09.643 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 7 v 87 (ObjNum 150 snap 30 seq_num 150) dirty exists 2024-09-20T22:44:09.643 INFO:tasks.rados.rados.0.smithi084.stdout:398: done (9 left) 2024-09-20T22:44:09.643 INFO:tasks.rados.rados.0.smithi084.stdout:401: done (8 left) 2024-09-20T22:44:09.643 INFO:tasks.rados.rados.0.smithi084.stdout:405: done (7 left) 2024-09-20T22:44:09.644 INFO:tasks.rados.rados.0.smithi084.stdout:407: done (6 left) 2024-09-20T22:44:09.644 INFO:tasks.rados.rados.0.smithi084.stdout:409: done (5 left) 2024-09-20T22:44:09.644 INFO:tasks.rados.rados.0.smithi084.stdout:413: delete oid 26 current snap is 37 2024-09-20T22:44:09.644 INFO:tasks.rados.rados.0.smithi084.stdout:406: expect (ObjNum 35 snap 0 seq_num 35) 2024-09-20T22:44:09.707 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:44:09.727 INFO:tasks.rados.rados.0.smithi084.stdout:403: expect (ObjNum 50 snap 0 seq_num 50) 2024-09-20T22:44:09.728 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:09.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:09.729 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:44:09.830 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:44:09.841 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:44:09.841 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:44:09.849 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:44:09.849 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:44:09.850 INFO:tasks.rados.rados.0.smithi084.stdout:402: expect (ObjNum 173 snap 37 seq_num 173) 2024-09-20T22:44:09.852 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:09.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:09.853 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:44:09.867 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:44:09.868 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:44:09.883 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-20T22:44:09.884 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T22:44:09.900 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:44:09.900 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:44:09.917 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:44:09.917 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:44:09.933 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:44:09.934 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:44:09.950 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:44:09.950 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:44:09.953 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:44:09.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:09.976 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:09.976 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:44:10.017 INFO:tasks.rados.rados.0.smithi084.stdout:410: expect (ObjNum 121 snap 23 seq_num 121) 2024-09-20T22:44:10.077 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:44:10.100 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:10.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:10.100 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:44:10.137 INFO:tasks.rados.rados.0.smithi084.stdout:412: finishing write tid 1 to smithi08477203-46 2024-09-20T22:44:10.137 INFO:tasks.rados.rados.0.smithi084.stdout:412: finishing write tid 2 to smithi08477203-46 2024-09-20T22:44:10.138 INFO:tasks.rados.rados.0.smithi084.stdout:402: done (5 left) 2024-09-20T22:44:10.138 INFO:tasks.rados.rados.0.smithi084.stdout:403: done (4 left) 2024-09-20T22:44:10.138 INFO:tasks.rados.rados.0.smithi084.stdout:406: done (3 left) 2024-09-20T22:44:10.138 INFO:tasks.rados.rados.0.smithi084.stdout:410: done (2 left) 2024-09-20T22:44:10.138 INFO:tasks.rados.rados.0.smithi084.stdout:413: done (1 left) 2024-09-20T22:44:10.138 INFO:tasks.rados.rados.0.smithi084.stdout:414: rollback oid 6 current snap is 37 2024-09-20T22:44:10.138 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 6 to 36 2024-09-20T22:44:10.138 INFO:tasks.rados.rados.0.smithi084.stdout:412: finishing write tid 3 to smithi08477203-46 2024-09-20T22:44:10.138 INFO:tasks.rados.rados.0.smithi084.stdout:412: finishing write tid 4 to smithi08477203-46 2024-09-20T22:44:10.138 INFO:tasks.rados.rados.0.smithi084.stdout:412: oid 46 updating version 0 to 102 2024-09-20T22:44:10.138 INFO:tasks.rados.rados.0.smithi084.stdout:412: oid 46 version 102 is already newer than 100 2024-09-20T22:44:10.138 INFO:tasks.rados.rados.0.smithi084.stdout:412: oid 46 version 102 is already newer than 101 2024-09-20T22:44:10.138 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 46 v 102 (ObjNum 177 snap 37 seq_num 177) dirty exists 2024-09-20T22:44:10.138 INFO:tasks.rados.rados.0.smithi084.stdout:412: left oid 46 (ObjNum 177 snap 37 seq_num 177) 2024-09-20T22:44:10.139 INFO:tasks.rados.rados.0.smithi084.stdout:412: done (1 left) 2024-09-20T22:44:10.139 INFO:tasks.rados.rados.0.smithi084.stdout:415: write oid 5 current snap is 37 2024-09-20T22:44:10.139 INFO:tasks.rados.rados.0.smithi084.stdout:415: seq_num 179 ranges {608453=545690,1624865=455728,2632598=375855} 2024-09-20T22:44:10.142 INFO:tasks.rados.rados.0.smithi084.stdout:415: writing smithi08477203-5 from 608453 to 1154143 tid 1 2024-09-20T22:44:10.147 INFO:tasks.rados.rados.0.smithi084.stdout:415: writing smithi08477203-5 from 1624865 to 2080593 tid 2 2024-09-20T22:44:10.153 INFO:tasks.rados.rados.0.smithi084.stdout:415: writing smithi08477203-5 from 2632598 to 3008453 tid 3 2024-09-20T22:44:10.153 INFO:tasks.rados.rados.0.smithi084.stdout:416: read oid 46 snap -1 2024-09-20T22:44:10.153 INFO:tasks.rados.rados.0.smithi084.stdout:416: expect (ObjNum 177 snap 37 seq_num 177) 2024-09-20T22:44:10.154 INFO:tasks.rados.rados.0.smithi084.stdout:417: read oid 48 snap -1 2024-09-20T22:44:10.154 INFO:tasks.rados.rados.0.smithi084.stdout:417: expect (ObjNum 129 snap 23 seq_num 129) 2024-09-20T22:44:10.154 INFO:tasks.rados.rados.0.smithi084.stdout:418: read oid 16 snap -1 2024-09-20T22:44:10.154 INFO:tasks.rados.rados.0.smithi084.stdout:418: expect deleted 2024-09-20T22:44:10.154 INFO:tasks.rados.rados.0.smithi084.stdout:419: read oid 14 snap -1 2024-09-20T22:44:10.154 INFO:tasks.rados.rados.0.smithi084.stdout:419: expect (ObjNum 13 snap 0 seq_num 13) 2024-09-20T22:44:10.154 INFO:tasks.rados.rados.0.smithi084.stdout:420: copy_from oid 8 from oid 30 current snap is 37 2024-09-20T22:44:10.154 INFO:tasks.rados.rados.0.smithi084.stdout:414: finishing rollback tid 0 to smithi08477203-6 2024-09-20T22:44:10.154 INFO:tasks.rados.rados.0.smithi084.stdout:414: finishing rollback tid 1 to smithi08477203-6 2024-09-20T22:44:10.154 INFO:tasks.rados.rados.0.smithi084.stdout:421: copy_from oid 37 from oid 15 current snap is 37 2024-09-20T22:44:10.154 INFO:tasks.rados.rados.0.smithi084.stdout:414: finishing rollback tid 2 to smithi08477203-6 2024-09-20T22:44:10.154 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 6 v 56 (ObjNum 160 snap 34 seq_num 160) dirty exists 2024-09-20T22:44:10.154 INFO:tasks.rados.rados.0.smithi084.stdout:415: finishing write tid 1 to smithi08477203-5 2024-09-20T22:44:10.154 INFO:tasks.rados.rados.0.smithi084.stdout:415: finishing write tid 2 to smithi08477203-5 2024-09-20T22:44:10.155 INFO:tasks.rados.rados.0.smithi084.stdout:414: done (7 left) 2024-09-20T22:44:10.155 INFO:tasks.rados.rados.0.smithi084.stdout:422: snap_remove snap 36 2024-09-20T22:44:10.174 INFO:tasks.rados.rados.0.smithi084.stdout:422: done (7 left) 2024-09-20T22:44:10.175 INFO:tasks.rados.rados.0.smithi084.stdout:423: write (excl) oid 4 current snap is 37 2024-09-20T22:44:10.175 INFO:tasks.rados.rados.0.smithi084.stdout:423: seq_num 180 ranges {625260=620939,1683347=599386,2943668=81592} 2024-09-20T22:44:10.182 INFO:tasks.rados.rados.0.smithi084.stdout:423: writing smithi08477203-4 from 625260 to 1246199 tid 1 2024-09-20T22:44:10.188 INFO:tasks.rados.rados.0.smithi084.stdout:423: writing smithi08477203-4 from 1683347 to 2282733 tid 2 2024-09-20T22:44:10.193 INFO:tasks.rados.rados.0.smithi084.stdout:423: writing smithi08477203-4 from 2943668 to 3025260 tid 3 2024-09-20T22:44:10.193 INFO:tasks.rados.rados.0.smithi084.stdout:424: snap_create 2024-09-20T22:44:10.193 INFO:tasks.rados.rados.0.smithi084.stdout:415: finishing write tid 3 to smithi08477203-5 2024-09-20T22:44:10.193 INFO:tasks.rados.rados.0.smithi084.stdout:415: finishing write tid 4 to smithi08477203-5 2024-09-20T22:44:10.193 INFO:tasks.rados.rados.0.smithi084.stdout:415: finishing write tid 5 to smithi08477203-5 2024-09-20T22:44:10.193 INFO:tasks.rados.rados.0.smithi084.stdout:415: oid 5 updating version 0 to 65 2024-09-20T22:44:10.193 INFO:tasks.rados.rados.0.smithi084.stdout:415: oid 5 version 65 is already newer than 64 2024-09-20T22:44:10.193 INFO:tasks.rados.rados.0.smithi084.stdout:415: oid 5 version 65 is already newer than 62 2024-09-20T22:44:10.193 INFO:tasks.rados.rados.0.smithi084.stdout:415: oid 5 version 65 is already newer than 63 2024-09-20T22:44:10.193 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 5 v 65 (ObjNum 179 snap 37 seq_num 179) dirty exists 2024-09-20T22:44:10.193 INFO:tasks.rados.rados.0.smithi084.stdout:415: left oid 5 (ObjNum 179 snap 37 seq_num 179) 2024-09-20T22:44:10.193 INFO:tasks.rados.rados.0.smithi084.stdout:419: expect (ObjNum 13 snap 0 seq_num 13) 2024-09-20T22:44:10.201 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:44:10.222 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:10.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:10.223 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:44:10.267 INFO:tasks.rados.rados.0.smithi084.stdout:420: finishing copy_from to smithi08477203-8 2024-09-20T22:44:10.267 INFO:tasks.rados.rados.0.smithi084.stdout:420: got expected ENOENT (src dne) 2024-09-20T22:44:10.267 INFO:tasks.rados.rados.0.smithi084.stdout:420: finishing copy_from racing read to smithi08477203-8 2024-09-20T22:44:10.267 INFO:tasks.rados.rados.0.smithi084.stdout:417: expect (ObjNum 129 snap 23 seq_num 129) 2024-09-20T22:44:10.323 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:44:10.339 INFO:tasks.rados.rados.0.smithi084.stdout:416: expect (ObjNum 177 snap 37 seq_num 177) 2024-09-20T22:44:10.345 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:10.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:10.346 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:44:10.364 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:10 smithi084 ceph-mon[40024]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:44:10.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:10 smithi084 ceph-mon[40024]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:44:10.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:10 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/765502152' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T22:44:10.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:10 smithi084 ceph-mon[40024]: osdmap e170: 8 total, 8 up, 7 in 2024-09-20T22:44:10.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:10 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/3133589702' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:44:10.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:10 smithi084 ceph-mon[29774]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:44:10.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:10 smithi084 ceph-mon[29774]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:44:10.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:10 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/765502152' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T22:44:10.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:10 smithi084 ceph-mon[29774]: osdmap e170: 8 total, 8 up, 7 in 2024-09-20T22:44:10.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:10 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/3133589702' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:44:10.447 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:44:10.469 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:10.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:10.470 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:44:10.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:10 smithi154 ceph-mon[33264]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:44:10.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:10 smithi154 ceph-mon[33264]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:44:10.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:10 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/765502152' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T22:44:10.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:10 smithi154 ceph-mon[33264]: osdmap e170: 8 total, 8 up, 7 in 2024-09-20T22:44:10.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:10 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/3133589702' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:44:10.504 INFO:tasks.rados.rados.0.smithi084.stdout:421: finishing copy_from racing read to smithi08477203-37 2024-09-20T22:44:10.505 INFO:tasks.rados.rados.0.smithi084.stdout:421: finishing copy_from to smithi08477203-37 2024-09-20T22:44:10.505 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 37 v 87 (ObjNum 150 snap 30 seq_num 150) dirty exists 2024-09-20T22:44:10.505 INFO:tasks.rados.rados.0.smithi084.stdout:423: finishing write tid 1 to smithi08477203-4 2024-09-20T22:44:10.505 INFO:tasks.rados.rados.0.smithi084.stdout:423: finishing write tid 2 to smithi08477203-4 2024-09-20T22:44:10.505 INFO:tasks.rados.rados.0.smithi084.stdout:423: finishing write tid 3 to smithi08477203-4 2024-09-20T22:44:10.505 INFO:tasks.rados.rados.0.smithi084.stdout:423: finishing write tid 4 to smithi08477203-4 2024-09-20T22:44:10.505 INFO:tasks.rados.rados.0.smithi084.stdout:423: finishing write tid 5 to smithi08477203-4 2024-09-20T22:44:10.505 INFO:tasks.rados.rados.0.smithi084.stdout:423: oid 4 updating version 0 to 173 2024-09-20T22:44:10.505 INFO:tasks.rados.rados.0.smithi084.stdout:423: oid 4 version 173 is already newer than 170 2024-09-20T22:44:10.505 INFO:tasks.rados.rados.0.smithi084.stdout:423: oid 4 version 173 is already newer than 172 2024-09-20T22:44:10.505 INFO:tasks.rados.rados.0.smithi084.stdout:423: oid 4 version 173 is already newer than 171 2024-09-20T22:44:10.505 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 4 v 173 (ObjNum 180 snap 37 seq_num 180) dirty exists 2024-09-20T22:44:10.505 INFO:tasks.rados.rados.0.smithi084.stdout:423: left oid 4 (ObjNum 180 snap 37 seq_num 180) 2024-09-20T22:44:10.571 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:44:10.590 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.2 dump_historic_ops 2024-09-20T22:44:10.592 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:10.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:10.593 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:44:10.694 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:44:10.715 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:10.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:10.716 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:44:10.817 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:44:10.838 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:10.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:10.839 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:44:10.940 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:44:10.961 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:10.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:10.962 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:44:11.062 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:44:11.084 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:11.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:11.085 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:44:11.182 INFO:tasks.rados.rados.0.smithi084.stdout:415: done (8 left) 2024-09-20T22:44:11.182 INFO:tasks.rados.rados.0.smithi084.stdout:416: done (7 left) 2024-09-20T22:44:11.182 INFO:tasks.rados.rados.0.smithi084.stdout:417: done (6 left) 2024-09-20T22:44:11.182 INFO:tasks.rados.rados.0.smithi084.stdout:418: done (5 left) 2024-09-20T22:44:11.182 INFO:tasks.rados.rados.0.smithi084.stdout:419: done (4 left) 2024-09-20T22:44:11.182 INFO:tasks.rados.rados.0.smithi084.stdout:420: done (3 left) 2024-09-20T22:44:11.182 INFO:tasks.rados.rados.0.smithi084.stdout:421: done (2 left) 2024-09-20T22:44:11.182 INFO:tasks.rados.rados.0.smithi084.stdout:423: done (1 left) 2024-09-20T22:44:11.182 INFO:tasks.rados.rados.0.smithi084.stdout:424: done (0 left) 2024-09-20T22:44:11.182 INFO:tasks.rados.rados.0.smithi084.stdout:425: write oid 13 current snap is 38 2024-09-20T22:44:11.183 INFO:tasks.rados.rados.0.smithi084.stdout:425: seq_num 181 ranges {642067=696188,2025476=459397,2971091=70976} 2024-09-20T22:44:11.190 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:44:11.191 INFO:tasks.rados.rados.0.smithi084.stdout:425: writing smithi08477203-13 from 642067 to 1338255 tid 1 2024-09-20T22:44:11.195 INFO:tasks.rados.rados.0.smithi084.stdout:425: writing smithi08477203-13 from 2025476 to 2484873 tid 2 2024-09-20T22:44:11.203 INFO:tasks.rados.rados.0.smithi084.stdout:425: writing smithi08477203-13 from 2971091 to 3042067 tid 3 2024-09-20T22:44:11.203 INFO:tasks.rados.rados.0.smithi084.stdout:426: snap_remove snap 37 2024-09-20T22:44:11.203 INFO:tasks.rados.rados.0.smithi084.stdout:425: finishing write tid 1 to smithi08477203-13 2024-09-20T22:44:11.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:11.216 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:11.216 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:44:11.317 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:44:11.336 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:11 smithi084 ceph-mon[29774]: pgmap v157: 178 pgs: 6 peering, 3 active+clean+snaptrim, 169 active+clean; 152 MiB data, 571 MiB used, 625 GiB / 626 GiB avail; 21 MiB/s rd, 5.7 MiB/s wr, 34 op/s 2024-09-20T22:44:11.336 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:11 smithi084 ceph-mon[29774]: 7.2 scrub starts 2024-09-20T22:44:11.336 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:11 smithi084 ceph-mon[29774]: 7.2 scrub ok 2024-09-20T22:44:11.336 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:11 smithi084 ceph-mon[29774]: 7.16 scrub starts 2024-09-20T22:44:11.336 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:11 smithi084 ceph-mon[29774]: 7.16 scrub ok 2024-09-20T22:44:11.336 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:11 smithi084 ceph-mon[29774]: osdmap e171: 8 total, 8 up, 7 in 2024-09-20T22:44:11.336 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:11 smithi084 ceph-mon[29774]: 1.5 scrub starts 2024-09-20T22:44:11.336 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:11 smithi084 ceph-mon[29774]: 1.5 scrub ok 2024-09-20T22:44:11.336 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:11 smithi084 ceph-mon[29774]: 3.18 scrub starts 2024-09-20T22:44:11.336 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:11 smithi084 ceph-mon[29774]: 3.18 scrub ok 2024-09-20T22:44:11.337 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:11 smithi084 ceph-mon[40024]: pgmap v157: 178 pgs: 6 peering, 3 active+clean+snaptrim, 169 active+clean; 152 MiB data, 571 MiB used, 625 GiB / 626 GiB avail; 21 MiB/s rd, 5.7 MiB/s wr, 34 op/s 2024-09-20T22:44:11.337 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:11 smithi084 ceph-mon[40024]: 7.2 scrub starts 2024-09-20T22:44:11.337 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:11 smithi084 ceph-mon[40024]: 7.2 scrub ok 2024-09-20T22:44:11.337 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:11 smithi084 ceph-mon[40024]: 7.16 scrub starts 2024-09-20T22:44:11.337 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:11 smithi084 ceph-mon[40024]: 7.16 scrub ok 2024-09-20T22:44:11.337 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:11 smithi084 ceph-mon[40024]: osdmap e171: 8 total, 8 up, 7 in 2024-09-20T22:44:11.337 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:11 smithi084 ceph-mon[40024]: 1.5 scrub starts 2024-09-20T22:44:11.337 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:11 smithi084 ceph-mon[40024]: 1.5 scrub ok 2024-09-20T22:44:11.337 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:11 smithi084 ceph-mon[40024]: 3.18 scrub starts 2024-09-20T22:44:11.337 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:11 smithi084 ceph-mon[40024]: 3.18 scrub ok 2024-09-20T22:44:11.339 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:11.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:11.340 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:44:11.365 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph tell osd.2 reset_purged_snaps_last 2024-09-20T22:44:11.441 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:44:11.463 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:11.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:11.464 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:44:11.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:11 smithi154 ceph-mon[33264]: pgmap v157: 178 pgs: 6 peering, 3 active+clean+snaptrim, 169 active+clean; 152 MiB data, 571 MiB used, 625 GiB / 626 GiB avail; 21 MiB/s rd, 5.7 MiB/s wr, 34 op/s 2024-09-20T22:44:11.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:11 smithi154 ceph-mon[33264]: 7.2 scrub starts 2024-09-20T22:44:11.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:11 smithi154 ceph-mon[33264]: 7.2 scrub ok 2024-09-20T22:44:11.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:11 smithi154 ceph-mon[33264]: 7.16 scrub starts 2024-09-20T22:44:11.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:11 smithi154 ceph-mon[33264]: 7.16 scrub ok 2024-09-20T22:44:11.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:11 smithi154 ceph-mon[33264]: osdmap e171: 8 total, 8 up, 7 in 2024-09-20T22:44:11.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:11 smithi154 ceph-mon[33264]: 1.5 scrub starts 2024-09-20T22:44:11.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:11 smithi154 ceph-mon[33264]: 1.5 scrub ok 2024-09-20T22:44:11.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:11 smithi154 ceph-mon[33264]: 3.18 scrub starts 2024-09-20T22:44:11.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:11 smithi154 ceph-mon[33264]: 3.18 scrub ok 2024-09-20T22:44:11.564 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:44:11.588 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:11.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:11.589 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:44:11.690 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:44:11.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:11.712 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:11.713 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:44:11.813 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:44:11.837 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:11.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:11.839 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:44:11.939 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:44:11.961 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:11.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:11.962 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:44:12.062 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:44:12.083 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:12.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:12.084 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:44:12.185 INFO:tasks.rados.rados.0.smithi084.stdout:426: done (1 left) 2024-09-20T22:44:12.185 INFO:tasks.rados.rados.0.smithi084.stdout:427: write oid 25 current snap is 38 2024-09-20T22:44:12.185 INFO:tasks.rados.rados.0.smithi084.stdout:425: finishing write tid 2 to smithi08477203-13 2024-09-20T22:44:12.185 INFO:tasks.rados.rados.0.smithi084.stdout:425: finishing write tid 3 to smithi08477203-13 2024-09-20T22:44:12.185 INFO:tasks.rados.rados.0.smithi084.stdout:427: seq_num 182 ranges {658874=771437,2083958=603055} 2024-09-20T22:44:12.185 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:44:12.193 INFO:tasks.rados.rados.0.smithi084.stdout:427: writing smithi08477203-25 from 658874 to 1430311 tid 1 2024-09-20T22:44:12.202 INFO:tasks.rados.rados.0.smithi084.stdout:427: writing smithi08477203-25 from 2083958 to 2687013 tid 2 2024-09-20T22:44:12.202 INFO:tasks.rados.rados.0.smithi084.stdout:428: snap_remove snap 34 2024-09-20T22:44:12.208 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:12.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:12.209 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:44:12.233 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:44:12.310 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:44:12.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:12.336 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:12.336 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:44:12.437 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:44:12.458 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:12.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:12.459 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:44:12.489 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:12 smithi084 ceph-mon[29774]: 5.c scrub starts 2024-09-20T22:44:12.489 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:12 smithi084 ceph-mon[29774]: 5.c scrub ok 2024-09-20T22:44:12.489 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:12 smithi084 ceph-mon[29774]: osdmap e172: 8 total, 8 up, 7 in 2024-09-20T22:44:12.489 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:12 smithi084 ceph-mon[40024]: 5.c scrub starts 2024-09-20T22:44:12.489 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:12 smithi084 ceph-mon[40024]: 5.c scrub ok 2024-09-20T22:44:12.489 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:12 smithi084 ceph-mon[40024]: osdmap e172: 8 total, 8 up, 7 in 2024-09-20T22:44:12.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:12 smithi154 ceph-mon[33264]: 5.c scrub starts 2024-09-20T22:44:12.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:12 smithi154 ceph-mon[33264]: 5.c scrub ok 2024-09-20T22:44:12.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:12 smithi154 ceph-mon[33264]: osdmap e172: 8 total, 8 up, 7 in 2024-09-20T22:44:12.508 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:44:12.561 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:44:12.582 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:12.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:12.583 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:44:12.684 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:44:12.705 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:12.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:12.706 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:44:12.756 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-20T22:44:12.756 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:44:12.807 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:44:12.829 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:12.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:12.829 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:44:12.931 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:44:12.953 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:12.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:12.954 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:44:13.055 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:44:13.077 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:13.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:13.078 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:44:13.136 INFO:tasks.rados.rados.0.smithi084.stdout:428: done (2 left) 2024-09-20T22:44:13.136 INFO:tasks.rados.rados.0.smithi084.stdout:429: delete oid 9 current snap is 38 2024-09-20T22:44:13.136 INFO:tasks.rados.rados.0.smithi084.stdout:425: finishing write tid 4 to smithi08477203-13 2024-09-20T22:44:13.136 INFO:tasks.rados.rados.0.smithi084.stdout:425: finishing write tid 5 to smithi08477203-13 2024-09-20T22:44:13.136 INFO:tasks.rados.rados.0.smithi084.stdout:425: oid 13 updating version 0 to 177 2024-09-20T22:44:13.137 INFO:tasks.rados.rados.0.smithi084.stdout:425: oid 13 version 177 is already newer than 175 2024-09-20T22:44:13.137 INFO:tasks.rados.rados.0.smithi084.stdout:425: oid 13 version 177 is already newer than 176 2024-09-20T22:44:13.137 INFO:tasks.rados.rados.0.smithi084.stdout:425: oid 13 updating version 177 to 178 2024-09-20T22:44:13.137 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 13 v 178 (ObjNum 181 snap 38 seq_num 181) dirty exists 2024-09-20T22:44:13.137 INFO:tasks.rados.rados.0.smithi084.stdout:425: left oid 13 (ObjNum 181 snap 38 seq_num 181) 2024-09-20T22:44:13.137 INFO:tasks.rados.rados.0.smithi084.stdout:427: finishing write tid 1 to smithi08477203-25 2024-09-20T22:44:13.137 INFO:tasks.rados.rados.0.smithi084.stdout:427: finishing write tid 2 to smithi08477203-25 2024-09-20T22:44:13.137 INFO:tasks.rados.rados.0.smithi084.stdout:427: finishing write tid 3 to smithi08477203-25 2024-09-20T22:44:13.137 INFO:tasks.rados.rados.0.smithi084.stdout:427: finishing write tid 4 to smithi08477203-25 2024-09-20T22:44:13.137 INFO:tasks.rados.rados.0.smithi084.stdout:427: oid 25 updating version 0 to 183 2024-09-20T22:44:13.137 INFO:tasks.rados.rados.0.smithi084.stdout:427: oid 25 updating version 183 to 184 2024-09-20T22:44:13.137 INFO:tasks.rados.rados.0.smithi084.stdout:427: oid 25 version 184 is already newer than 180 2024-09-20T22:44:13.137 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 25 v 184 (ObjNum 182 snap 38 seq_num 182) dirty exists 2024-09-20T22:44:13.138 INFO:tasks.rados.rados.0.smithi084.stdout:427: left oid 25 (ObjNum 182 snap 38 seq_num 182) 2024-09-20T22:44:13.140 INFO:tasks.rados.rados.0.smithi084.stdout:425: done (2 left) 2024-09-20T22:44:13.140 INFO:tasks.rados.rados.0.smithi084.stdout:427: done (1 left) 2024-09-20T22:44:13.140 INFO:tasks.rados.rados.0.smithi084.stdout:429: done (0 left) 2024-09-20T22:44:13.140 INFO:tasks.rados.rados.0.smithi084.stdout:430: copy_from oid 5 from oid 9 current snap is 38 2024-09-20T22:44:13.140 INFO:tasks.rados.rados.0.smithi084.stdout:431: snap_create 2024-09-20T22:44:13.143 INFO:tasks.rados.rados.0.smithi084.stdout:430: finishing copy_from to smithi08477203-5 2024-09-20T22:44:13.144 INFO:tasks.rados.rados.0.smithi084.stdout:430: got expected ENOENT (src dne) 2024-09-20T22:44:13.144 INFO:tasks.rados.rados.0.smithi084.stdout:430: finishing copy_from racing read to smithi08477203-5 2024-09-20T22:44:13.179 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:44:13.201 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:13.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:13.202 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:44:13.303 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:44:13.326 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:13.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:13.327 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:44:13.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:13 smithi084 ceph-mon[29774]: pgmap v160: 178 pgs: 6 peering, 3 active+clean+snaptrim, 169 active+clean; 152 MiB data, 571 MiB used, 625 GiB / 626 GiB avail; 21 MiB/s rd, 5.7 MiB/s wr, 34 op/s 2024-09-20T22:44:13.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:13 smithi084 ceph-mon[29774]: 5.5 scrub starts 2024-09-20T22:44:13.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:13 smithi084 ceph-mon[29774]: 5.5 scrub ok 2024-09-20T22:44:13.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:13 smithi084 ceph-mon[29774]: 3.11 scrub starts 2024-09-20T22:44:13.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:13 smithi084 ceph-mon[29774]: 3.11 scrub ok 2024-09-20T22:44:13.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:13 smithi084 ceph-mon[29774]: 3.1a scrub starts 2024-09-20T22:44:13.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:13 smithi084 ceph-mon[29774]: 3.1a scrub ok 2024-09-20T22:44:13.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:13 smithi084 ceph-mon[29774]: osdmap e173: 8 total, 8 up, 7 in 2024-09-20T22:44:13.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:13 smithi084 ceph-mon[29774]: 7.1 scrub starts 2024-09-20T22:44:13.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:13 smithi084 ceph-mon[29774]: 7.1 scrub ok 2024-09-20T22:44:13.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:13 smithi084 ceph-mon[29774]: 5.b scrub starts 2024-09-20T22:44:13.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:13 smithi084 ceph-mon[29774]: 5.b scrub ok 2024-09-20T22:44:13.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:13 smithi084 ceph-mon[29774]: osdmap e174: 8 total, 8 up, 7 in 2024-09-20T22:44:13.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:13 smithi084 ceph-mon[40024]: pgmap v160: 178 pgs: 6 peering, 3 active+clean+snaptrim, 169 active+clean; 152 MiB data, 571 MiB used, 625 GiB / 626 GiB avail; 21 MiB/s rd, 5.7 MiB/s wr, 34 op/s 2024-09-20T22:44:13.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:13 smithi084 ceph-mon[40024]: 5.5 scrub starts 2024-09-20T22:44:13.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:13 smithi084 ceph-mon[40024]: 5.5 scrub ok 2024-09-20T22:44:13.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:13 smithi084 ceph-mon[40024]: 3.11 scrub starts 2024-09-20T22:44:13.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:13 smithi084 ceph-mon[40024]: 3.11 scrub ok 2024-09-20T22:44:13.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:13 smithi084 ceph-mon[40024]: 3.1a scrub starts 2024-09-20T22:44:13.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:13 smithi084 ceph-mon[40024]: 3.1a scrub ok 2024-09-20T22:44:13.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:13 smithi084 ceph-mon[40024]: osdmap e173: 8 total, 8 up, 7 in 2024-09-20T22:44:13.367 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:13 smithi084 ceph-mon[40024]: 7.1 scrub starts 2024-09-20T22:44:13.367 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:13 smithi084 ceph-mon[40024]: 7.1 scrub ok 2024-09-20T22:44:13.367 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:13 smithi084 ceph-mon[40024]: 5.b scrub starts 2024-09-20T22:44:13.367 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:13 smithi084 ceph-mon[40024]: 5.b scrub ok 2024-09-20T22:44:13.367 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:13 smithi084 ceph-mon[40024]: osdmap e174: 8 total, 8 up, 7 in 2024-09-20T22:44:13.427 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:44:13.450 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:13.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:13.451 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:44:13.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:13 smithi154 ceph-mon[33264]: pgmap v160: 178 pgs: 6 peering, 3 active+clean+snaptrim, 169 active+clean; 152 MiB data, 571 MiB used, 625 GiB / 626 GiB avail; 21 MiB/s rd, 5.7 MiB/s wr, 34 op/s 2024-09-20T22:44:13.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:13 smithi154 ceph-mon[33264]: 5.5 scrub starts 2024-09-20T22:44:13.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:13 smithi154 ceph-mon[33264]: 5.5 scrub ok 2024-09-20T22:44:13.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:13 smithi154 ceph-mon[33264]: 3.11 scrub starts 2024-09-20T22:44:13.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:13 smithi154 ceph-mon[33264]: 3.11 scrub ok 2024-09-20T22:44:13.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:13 smithi154 ceph-mon[33264]: 3.1a scrub starts 2024-09-20T22:44:13.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:13 smithi154 ceph-mon[33264]: 3.1a scrub ok 2024-09-20T22:44:13.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:13 smithi154 ceph-mon[33264]: osdmap e173: 8 total, 8 up, 7 in 2024-09-20T22:44:13.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:13 smithi154 ceph-mon[33264]: 7.1 scrub starts 2024-09-20T22:44:13.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:13 smithi154 ceph-mon[33264]: 7.1 scrub ok 2024-09-20T22:44:13.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:13 smithi154 ceph-mon[33264]: 5.b scrub starts 2024-09-20T22:44:13.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:13 smithi154 ceph-mon[33264]: 5.b scrub ok 2024-09-20T22:44:13.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:13 smithi154 ceph-mon[33264]: osdmap e174: 8 total, 8 up, 7 in 2024-09-20T22:44:13.551 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:44:13.573 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:13.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:13.574 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:44:13.657 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-20T22:44:13.676 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:44:13.697 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:13.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:13.698 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:44:13.787 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-20T22:44:13.799 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:44:13.852 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:13.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:13.853 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:44:13.954 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:44:13.976 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:13.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:13.977 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:44:14.078 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:44:14.100 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:14.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:14.101 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:44:14.134 INFO:tasks.rados.rados.0.smithi084.stdout:430: done (1 left) 2024-09-20T22:44:14.134 INFO:tasks.rados.rados.0.smithi084.stdout:431: done (0 left) 2024-09-20T22:44:14.134 INFO:tasks.rados.rados.0.smithi084.stdout:432: write oid 45 current snap is 39 2024-09-20T22:44:14.134 INFO:tasks.rados.rados.0.smithi084.stdout:432: seq_num 184 ranges {692488=638288,1917275=606724} 2024-09-20T22:44:14.141 INFO:tasks.rados.rados.0.smithi084.stdout:432: writing smithi08477203-45 from 692488 to 1330776 tid 1 2024-09-20T22:44:14.149 INFO:tasks.rados.rados.0.smithi084.stdout:432: writing smithi08477203-45 from 1917275 to 2523999 tid 2 2024-09-20T22:44:14.149 INFO:tasks.rados.rados.0.smithi084.stdout:433: read oid 2 snap -1 2024-09-20T22:44:14.149 INFO:tasks.rados.rados.0.smithi084.stdout:433: expect (ObjNum 112 snap 23 seq_num 112) 2024-09-20T22:44:14.150 INFO:tasks.rados.rados.0.smithi084.stdout:434: copy_from oid 5 from oid 18 current snap is 39 2024-09-20T22:44:14.150 INFO:tasks.rados.rados.0.smithi084.stdout:435: write oid 47 current snap is 39 2024-09-20T22:44:14.150 INFO:tasks.rados.rados.0.smithi084.stdout:435: seq_num 185 ranges {709295=713537,1859404=750382} 2024-09-20T22:44:14.157 INFO:tasks.rados.rados.0.smithi084.stdout:435: writing smithi08477203-47 from 709295 to 1422832 tid 1 2024-09-20T22:44:14.164 INFO:tasks.rados.rados.0.smithi084.stdout:435: writing smithi08477203-47 from 1859404 to 2609786 tid 2 2024-09-20T22:44:14.164 INFO:tasks.rados.rados.0.smithi084.stdout:436: write (excl) oid 40 current snap is 39 2024-09-20T22:44:14.165 INFO:tasks.rados.rados.0.smithi084.stdout:432: finishing write tid 1 to smithi08477203-45 2024-09-20T22:44:14.165 INFO:tasks.rados.rados.0.smithi084.stdout:436: seq_num 186 ranges {726102=788786,1917886=610393,3108206=17896} 2024-09-20T22:44:14.173 INFO:tasks.rados.rados.0.smithi084.stdout:436: writing smithi08477203-40 from 726102 to 1514888 tid 1 2024-09-20T22:44:14.178 INFO:tasks.rados.rados.0.smithi084.stdout:436: writing smithi08477203-40 from 1917886 to 2528279 tid 2 2024-09-20T22:44:14.182 INFO:tasks.rados.rados.0.smithi084.stdout:436: writing smithi08477203-40 from 3108206 to 3126102 tid 3 2024-09-20T22:44:14.182 INFO:tasks.rados.rados.0.smithi084.stdout:437: copy_from oid 16 from oid 34 current snap is 39 2024-09-20T22:44:14.182 INFO:tasks.rados.rados.0.smithi084.stdout:434: finishing copy_from to smithi08477203-5 2024-09-20T22:44:14.182 INFO:tasks.rados.rados.0.smithi084.stdout:434: got expected ENOENT (src dne) 2024-09-20T22:44:14.182 INFO:tasks.rados.rados.0.smithi084.stdout:434: finishing copy_from racing read to smithi08477203-5 2024-09-20T22:44:14.183 INFO:tasks.rados.rados.0.smithi084.stdout:432: finishing write tid 2 to smithi08477203-45 2024-09-20T22:44:14.183 INFO:tasks.rados.rados.0.smithi084.stdout:432: finishing write tid 3 to smithi08477203-45 2024-09-20T22:44:14.183 INFO:tasks.rados.rados.0.smithi084.stdout:432: finishing write tid 4 to smithi08477203-45 2024-09-20T22:44:14.183 INFO:tasks.rados.rados.0.smithi084.stdout:432: oid 45 updating version 0 to 63 2024-09-20T22:44:14.183 INFO:tasks.rados.rados.0.smithi084.stdout:432: oid 45 updating version 63 to 64 2024-09-20T22:44:14.183 INFO:tasks.rados.rados.0.smithi084.stdout:432: oid 45 version 64 is already newer than 62 2024-09-20T22:44:14.183 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 45 v 64 (ObjNum 184 snap 39 seq_num 184) dirty exists 2024-09-20T22:44:14.183 INFO:tasks.rados.rados.0.smithi084.stdout:432: left oid 45 (ObjNum 184 snap 39 seq_num 184) 2024-09-20T22:44:14.183 INFO:tasks.rados.rados.0.smithi084.stdout:433: expect (ObjNum 112 snap 23 seq_num 112) 2024-09-20T22:44:14.201 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:44:14.223 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:14.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:14.224 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:44:14.267 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-20T22:44:14.315 INFO:tasks.rados.rados.0.smithi084.stdout:435: finishing write tid 1 to smithi08477203-47 2024-09-20T22:44:14.315 INFO:tasks.rados.rados.0.smithi084.stdout:435: finishing write tid 2 to smithi08477203-47 2024-09-20T22:44:14.315 INFO:tasks.rados.rados.0.smithi084.stdout:435: finishing write tid 3 to smithi08477203-47 2024-09-20T22:44:14.315 INFO:tasks.rados.rados.0.smithi084.stdout:435: finishing write tid 4 to smithi08477203-47 2024-09-20T22:44:14.316 INFO:tasks.rados.rados.0.smithi084.stdout:435: oid 47 updating version 0 to 155 2024-09-20T22:44:14.316 INFO:tasks.rados.rados.0.smithi084.stdout:435: oid 47 updating version 155 to 156 2024-09-20T22:44:14.316 INFO:tasks.rados.rados.0.smithi084.stdout:435: oid 47 updating version 156 to 157 2024-09-20T22:44:14.316 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 47 v 157 (ObjNum 185 snap 39 seq_num 185) dirty exists 2024-09-20T22:44:14.316 INFO:tasks.rados.rados.0.smithi084.stdout:435: left oid 47 (ObjNum 185 snap 39 seq_num 185) 2024-09-20T22:44:14.316 INFO:tasks.rados.rados.0.smithi084.stdout:436: finishing write tid 1 to smithi08477203-40 2024-09-20T22:44:14.316 INFO:tasks.rados.rados.0.smithi084.stdout:432: done (5 left) 2024-09-20T22:44:14.316 INFO:tasks.rados.rados.0.smithi084.stdout:433: done (4 left) 2024-09-20T22:44:14.316 INFO:tasks.rados.rados.0.smithi084.stdout:434: done (3 left) 2024-09-20T22:44:14.316 INFO:tasks.rados.rados.0.smithi084.stdout:435: done (2 left) 2024-09-20T22:44:14.316 INFO:tasks.rados.rados.0.smithi084.stdout:438: write oid 25 current snap is 39 2024-09-20T22:44:14.316 INFO:tasks.rados.rados.0.smithi084.stdout:436: finishing write tid 2 to smithi08477203-40 2024-09-20T22:44:14.316 INFO:tasks.rados.rados.0.smithi084.stdout:438: seq_num 187 ranges {742909=464035,1976368=754051} 2024-09-20T22:44:14.321 INFO:tasks.rados.rados.0.smithi084.stdout:438: writing smithi08477203-25 from 742909 to 1206944 tid 1 2024-09-20T22:44:14.325 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:44:14.330 INFO:tasks.rados.rados.0.smithi084.stdout:438: writing smithi08477203-25 from 1976368 to 2730419 tid 2 2024-09-20T22:44:14.330 INFO:tasks.rados.rados.0.smithi084.stdout:439: write oid 38 current snap is 39 2024-09-20T22:44:14.330 INFO:tasks.rados.rados.0.smithi084.stdout:439: seq_num 188 ranges {759716=539284,2034850=614062} 2024-09-20T22:44:14.337 INFO:tasks.rados.rados.0.smithi084.stdout:439: writing smithi08477203-38 from 759716 to 1299000 tid 1 2024-09-20T22:44:14.345 INFO:tasks.rados.rados.0.smithi084.stdout:439: writing smithi08477203-38 from 2034850 to 2648912 tid 2 2024-09-20T22:44:14.345 INFO:tasks.rados.rados.0.smithi084.stdout:440: copy_from oid 4 from oid 29 current snap is 39 2024-09-20T22:44:14.345 INFO:tasks.rados.rados.0.smithi084.stdout:436: finishing write tid 3 to smithi08477203-40 2024-09-20T22:44:14.345 INFO:tasks.rados.rados.0.smithi084.stdout:436: finishing write tid 4 to smithi08477203-40 2024-09-20T22:44:14.345 INFO:tasks.rados.rados.0.smithi084.stdout:436: finishing write tid 5 to smithi08477203-40 2024-09-20T22:44:14.346 INFO:tasks.rados.rados.0.smithi084.stdout:436: oid 40 updating version 0 to 84 2024-09-20T22:44:14.346 INFO:tasks.rados.rados.0.smithi084.stdout:436: oid 40 version 84 is already newer than 82 2024-09-20T22:44:14.346 INFO:tasks.rados.rados.0.smithi084.stdout:436: oid 40 version 84 is already newer than 81 2024-09-20T22:44:14.346 INFO:tasks.rados.rados.0.smithi084.stdout:436: oid 40 version 84 is already newer than 83 2024-09-20T22:44:14.346 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 40 v 84 (ObjNum 186 snap 39 seq_num 186) dirty exists 2024-09-20T22:44:14.346 INFO:tasks.rados.rados.0.smithi084.stdout:436: left oid 40 (ObjNum 186 snap 39 seq_num 186) 2024-09-20T22:44:14.346 INFO:tasks.rados.rados.0.smithi084.stdout:437: finishing copy_from racing read to smithi08477203-16 2024-09-20T22:44:14.346 INFO:tasks.rados.rados.0.smithi084.stdout:437: finishing copy_from to smithi08477203-16 2024-09-20T22:44:14.346 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 16 v 119 (ObjNum 50 snap 0 seq_num 50) dirty exists 2024-09-20T22:44:14.346 INFO:tasks.rados.rados.0.smithi084.stdout:436: done (4 left) 2024-09-20T22:44:14.346 INFO:tasks.rados.rados.0.smithi084.stdout:437: done (3 left) 2024-09-20T22:44:14.346 INFO:tasks.rados.rados.0.smithi084.stdout:441: snap_remove snap 38 2024-09-20T22:44:14.347 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:14.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:14.348 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:44:14.449 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:44:14.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:14.471 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:14.471 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:44:14.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:14 smithi154 ceph-mon[33264]: 1.e scrub starts 2024-09-20T22:44:14.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:14 smithi154 ceph-mon[33264]: 1.e scrub ok 2024-09-20T22:44:14.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:14 smithi154 ceph-mon[33264]: 6.17 scrub starts 2024-09-20T22:44:14.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:14 smithi154 ceph-mon[33264]: 6.17 scrub ok 2024-09-20T22:44:14.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:14 smithi154 ceph-mon[33264]: osdmap e175: 8 total, 8 up, 7 in 2024-09-20T22:44:14.527 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:14 smithi084 ceph-mon[40024]: 1.e scrub starts 2024-09-20T22:44:14.527 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:14 smithi084 ceph-mon[40024]: 1.e scrub ok 2024-09-20T22:44:14.527 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:14 smithi084 ceph-mon[40024]: 6.17 scrub starts 2024-09-20T22:44:14.527 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:14 smithi084 ceph-mon[40024]: 6.17 scrub ok 2024-09-20T22:44:14.527 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:14 smithi084 ceph-mon[40024]: osdmap e175: 8 total, 8 up, 7 in 2024-09-20T22:44:14.527 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:14 smithi084 ceph-mon[29774]: 1.e scrub starts 2024-09-20T22:44:14.527 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:14 smithi084 ceph-mon[29774]: 1.e scrub ok 2024-09-20T22:44:14.528 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:14 smithi084 ceph-mon[29774]: 6.17 scrub starts 2024-09-20T22:44:14.528 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:14 smithi084 ceph-mon[29774]: 6.17 scrub ok 2024-09-20T22:44:14.528 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:14 smithi084 ceph-mon[29774]: osdmap e175: 8 total, 8 up, 7 in 2024-09-20T22:44:14.559 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph tell osd.3 reset_purged_snaps_last 2024-09-20T22:44:14.572 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:44:14.606 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:44:14.616 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:44:14.627 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:44:14.633 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:14.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:14.634 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:44:14.735 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:44:14.756 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:14.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:14.757 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:44:14.857 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:44:14.880 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:14.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:14.881 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:44:14.982 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:44:15.003 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:15.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:15.004 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:44:15.105 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:44:15.127 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:15.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:15.128 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:44:15.205 INFO:tasks.rados.rados.0.smithi084.stdout:441: done (3 left) 2024-09-20T22:44:15.205 INFO:tasks.rados.rados.0.smithi084.stdout:442: snap_create 2024-09-20T22:44:15.205 INFO:tasks.rados.rados.0.smithi084.stdout:438: finishing write tid 1 to smithi08477203-25 2024-09-20T22:44:15.205 INFO:tasks.rados.rados.0.smithi084.stdout:438: finishing write tid 2 to smithi08477203-25 2024-09-20T22:44:15.205 INFO:tasks.rados.rados.0.smithi084.stdout:438: finishing write tid 3 to smithi08477203-25 2024-09-20T22:44:15.205 INFO:tasks.rados.rados.0.smithi084.stdout:438: finishing write tid 4 to smithi08477203-25 2024-09-20T22:44:15.205 INFO:tasks.rados.rados.0.smithi084.stdout:438: oid 25 updating version 0 to 194 2024-09-20T22:44:15.205 INFO:tasks.rados.rados.0.smithi084.stdout:438: oid 25 updating version 194 to 196 2024-09-20T22:44:15.205 INFO:tasks.rados.rados.0.smithi084.stdout:438: oid 25 version 196 is already newer than 195 2024-09-20T22:44:15.206 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 25 v 196 (ObjNum 187 snap 39 seq_num 187) dirty exists 2024-09-20T22:44:15.206 INFO:tasks.rados.rados.0.smithi084.stdout:438: left oid 25 (ObjNum 187 snap 39 seq_num 187) 2024-09-20T22:44:15.206 INFO:tasks.rados.rados.0.smithi084.stdout:439: finishing write tid 1 to smithi08477203-38 2024-09-20T22:44:15.206 INFO:tasks.rados.rados.0.smithi084.stdout:439: finishing write tid 2 to smithi08477203-38 2024-09-20T22:44:15.206 INFO:tasks.rados.rados.0.smithi084.stdout:439: finishing write tid 3 to smithi08477203-38 2024-09-20T22:44:15.206 INFO:tasks.rados.rados.0.smithi084.stdout:439: finishing write tid 4 to smithi08477203-38 2024-09-20T22:44:15.206 INFO:tasks.rados.rados.0.smithi084.stdout:439: oid 38 updating version 0 to 160 2024-09-20T22:44:15.206 INFO:tasks.rados.rados.0.smithi084.stdout:439: oid 38 updating version 160 to 161 2024-09-20T22:44:15.206 INFO:tasks.rados.rados.0.smithi084.stdout:439: oid 38 version 161 is already newer than 159 2024-09-20T22:44:15.206 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 38 v 161 (ObjNum 188 snap 39 seq_num 188) dirty exists 2024-09-20T22:44:15.206 INFO:tasks.rados.rados.0.smithi084.stdout:439: left oid 38 (ObjNum 188 snap 39 seq_num 188) 2024-09-20T22:44:15.206 INFO:tasks.rados.rados.0.smithi084.stdout:440: finishing copy_from racing read to smithi08477203-4 2024-09-20T22:44:15.206 INFO:tasks.rados.rados.0.smithi084.stdout:440: finishing copy_from to smithi08477203-4 2024-09-20T22:44:15.206 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 4 v 81 (ObjNum 108 snap 22 seq_num 108) dirty exists 2024-09-20T22:44:15.229 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:44:15.252 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:15.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:15.253 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:44:15.354 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:44:15.375 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:15.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:15.376 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:44:15.450 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:15 smithi084 ceph-mon[40024]: pgmap v163: 178 pgs: 3 active+clean+snaptrim, 175 active+clean; 117 MiB data, 533 MiB used, 625 GiB / 626 GiB avail; 12 MiB/s rd, 3 op/s; 0 B/s, 0 objects/s recovering 2024-09-20T22:44:15.450 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:15 smithi084 ceph-mon[40024]: 2.0 scrub starts 2024-09-20T22:44:15.450 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:15 smithi084 ceph-mon[40024]: 2.0 scrub ok 2024-09-20T22:44:15.450 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:15 smithi084 ceph-mon[40024]: 6.e scrub starts 2024-09-20T22:44:15.450 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:15 smithi084 ceph-mon[40024]: 6.e scrub ok 2024-09-20T22:44:15.450 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:15 smithi084 ceph-mon[40024]: 5.1c scrub starts 2024-09-20T22:44:15.450 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:15 smithi084 ceph-mon[40024]: 5.1c scrub ok 2024-09-20T22:44:15.450 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:15 smithi084 ceph-mon[40024]: 5.6 scrub starts 2024-09-20T22:44:15.451 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:15 smithi084 ceph-mon[40024]: 5.6 scrub ok 2024-09-20T22:44:15.451 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:15 smithi084 ceph-mon[29774]: pgmap v163: 178 pgs: 3 active+clean+snaptrim, 175 active+clean; 117 MiB data, 533 MiB used, 625 GiB / 626 GiB avail; 12 MiB/s rd, 3 op/s; 0 B/s, 0 objects/s recovering 2024-09-20T22:44:15.451 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:15 smithi084 ceph-mon[29774]: 2.0 scrub starts 2024-09-20T22:44:15.451 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:15 smithi084 ceph-mon[29774]: 2.0 scrub ok 2024-09-20T22:44:15.451 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:15 smithi084 ceph-mon[29774]: 6.e scrub starts 2024-09-20T22:44:15.451 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:15 smithi084 ceph-mon[29774]: 6.e scrub ok 2024-09-20T22:44:15.451 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:15 smithi084 ceph-mon[29774]: 5.1c scrub starts 2024-09-20T22:44:15.451 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:15 smithi084 ceph-mon[29774]: 5.1c scrub ok 2024-09-20T22:44:15.451 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:15 smithi084 ceph-mon[29774]: 5.6 scrub starts 2024-09-20T22:44:15.451 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:15 smithi084 ceph-mon[29774]: 5.6 scrub ok 2024-09-20T22:44:15.477 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:44:15.486 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:44:15.487 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:44:15.487 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:44:15.487 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-20T22:44:15.487 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:44:15.487 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:44:15.487 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:44:15.487 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:44:15.487 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:44:15.487 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:44:15.487 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:44:15.487 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T22:44:15.488 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:44:15.488 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:44:15.488 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:44:15.488 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:44:15.498 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:15.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:15.499 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:44:15.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:15 smithi154 ceph-mon[33264]: pgmap v163: 178 pgs: 3 active+clean+snaptrim, 175 active+clean; 117 MiB data, 533 MiB used, 625 GiB / 626 GiB avail; 12 MiB/s rd, 3 op/s; 0 B/s, 0 objects/s recovering 2024-09-20T22:44:15.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:15 smithi154 ceph-mon[33264]: 2.0 scrub starts 2024-09-20T22:44:15.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:15 smithi154 ceph-mon[33264]: 2.0 scrub ok 2024-09-20T22:44:15.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:15 smithi154 ceph-mon[33264]: 6.e scrub starts 2024-09-20T22:44:15.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:15 smithi154 ceph-mon[33264]: 6.e scrub ok 2024-09-20T22:44:15.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:15 smithi154 ceph-mon[33264]: 5.1c scrub starts 2024-09-20T22:44:15.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:15 smithi154 ceph-mon[33264]: 5.1c scrub ok 2024-09-20T22:44:15.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:15 smithi154 ceph-mon[33264]: 5.6 scrub starts 2024-09-20T22:44:15.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:15 smithi154 ceph-mon[33264]: 5.6 scrub ok 2024-09-20T22:44:15.600 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:44:15.621 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:15.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:15.622 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:44:15.723 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:44:15.744 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:15.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:15.744 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:44:15.845 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:44:15.867 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:15.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:15.868 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:44:15.969 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:44:15.990 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:15.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:15.991 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:44:16.092 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:44:16.114 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:16.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:16.115 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:44:16.209 INFO:tasks.rados.rados.0.smithi084.stdout:438: done (3 left) 2024-09-20T22:44:16.209 INFO:tasks.rados.rados.0.smithi084.stdout:439: done (2 left) 2024-09-20T22:44:16.209 INFO:tasks.rados.rados.0.smithi084.stdout:440: done (1 left) 2024-09-20T22:44:16.209 INFO:tasks.rados.rados.0.smithi084.stdout:442: done (0 left) 2024-09-20T22:44:16.209 INFO:tasks.rados.rados.0.smithi084.stdout:443: read oid 20 snap -1 2024-09-20T22:44:16.210 INFO:tasks.rados.rados.0.smithi084.stdout:443: expect deleted 2024-09-20T22:44:16.210 INFO:tasks.rados.rados.0.smithi084.stdout:444: rollback oid 43 current snap is 40 2024-09-20T22:44:16.210 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 43 to 39 2024-09-20T22:44:16.210 INFO:tasks.rados.rados.0.smithi084.stdout:445: delete oid 42 current snap is 40 2024-09-20T22:44:16.210 INFO:tasks.rados.rados.0.smithi084.stdout:444: finishing rollback tid 1 to smithi08477203-43 2024-09-20T22:44:16.210 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 43 v 0 (ObjNum 177 snap 0 seq_num 2091085424) dirty dne 2024-09-20T22:44:16.212 INFO:tasks.rados.rados.0.smithi084.stdout:443: done (2 left) 2024-09-20T22:44:16.212 INFO:tasks.rados.rados.0.smithi084.stdout:444: done (1 left) 2024-09-20T22:44:16.212 INFO:tasks.rados.rados.0.smithi084.stdout:445: done (0 left) 2024-09-20T22:44:16.212 INFO:tasks.rados.rados.0.smithi084.stdout:446: snap_remove snap 39 2024-09-20T22:44:16.215 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:44:16.236 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:16.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:16.237 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:44:16.338 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:44:16.363 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:16.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:16.364 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:44:16.464 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:44:16.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:16.488 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:16.488 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:44:16.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:16 smithi154 ceph-mon[33264]: 4.18 scrub starts 2024-09-20T22:44:16.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:16 smithi154 ceph-mon[33264]: 4.18 scrub ok 2024-09-20T22:44:16.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:16 smithi154 ceph-mon[33264]: 4.11 scrub starts 2024-09-20T22:44:16.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:16 smithi154 ceph-mon[33264]: 4.11 scrub ok 2024-09-20T22:44:16.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:16 smithi154 ceph-mon[33264]: 6.5 scrub starts 2024-09-20T22:44:16.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:16 smithi154 ceph-mon[33264]: 6.5 scrub ok 2024-09-20T22:44:16.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:16 smithi154 ceph-mon[33264]: osdmap e176: 8 total, 8 up, 7 in 2024-09-20T22:44:16.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:16 smithi154 ceph-mon[33264]: 5.13 scrub starts 2024-09-20T22:44:16.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:16 smithi154 ceph-mon[33264]: 5.13 scrub ok 2024-09-20T22:44:16.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:16 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/831163770' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:44:16.584 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:16 smithi084 ceph-mon[29774]: 4.18 scrub starts 2024-09-20T22:44:16.584 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:16 smithi084 ceph-mon[29774]: 4.18 scrub ok 2024-09-20T22:44:16.584 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:16 smithi084 ceph-mon[29774]: 4.11 scrub starts 2024-09-20T22:44:16.584 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:16 smithi084 ceph-mon[29774]: 4.11 scrub ok 2024-09-20T22:44:16.585 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:16 smithi084 ceph-mon[29774]: 6.5 scrub starts 2024-09-20T22:44:16.585 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:16 smithi084 ceph-mon[29774]: 6.5 scrub ok 2024-09-20T22:44:16.585 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:16 smithi084 ceph-mon[29774]: osdmap e176: 8 total, 8 up, 7 in 2024-09-20T22:44:16.585 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:16 smithi084 ceph-mon[29774]: 5.13 scrub starts 2024-09-20T22:44:16.585 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:16 smithi084 ceph-mon[29774]: 5.13 scrub ok 2024-09-20T22:44:16.585 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:16 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/831163770' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:44:16.585 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:16 smithi084 ceph-mon[40024]: 4.18 scrub starts 2024-09-20T22:44:16.585 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:16 smithi084 ceph-mon[40024]: 4.18 scrub ok 2024-09-20T22:44:16.585 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:16 smithi084 ceph-mon[40024]: 4.11 scrub starts 2024-09-20T22:44:16.585 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:16 smithi084 ceph-mon[40024]: 4.11 scrub ok 2024-09-20T22:44:16.585 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:16 smithi084 ceph-mon[40024]: 6.5 scrub starts 2024-09-20T22:44:16.586 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:16 smithi084 ceph-mon[40024]: 6.5 scrub ok 2024-09-20T22:44:16.586 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:16 smithi084 ceph-mon[40024]: osdmap e176: 8 total, 8 up, 7 in 2024-09-20T22:44:16.586 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:16 smithi084 ceph-mon[40024]: 5.13 scrub starts 2024-09-20T22:44:16.586 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:16 smithi084 ceph-mon[40024]: 5.13 scrub ok 2024-09-20T22:44:16.586 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:16 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/831163770' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:44:16.589 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:44:16.611 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:16.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:16.612 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:44:16.713 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:44:16.734 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:16.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:16.735 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:44:16.775 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-20T22:44:16.776 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2024-09-20T22:44:16.776 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2024-09-20T22:44:16.776 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-20T22:44:16.835 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:44:16.857 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:16.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:16.858 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:44:16.959 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:44:16.980 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:16.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:16.981 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:44:17.082 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:44:17.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:17.104 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:17.104 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:44:17.204 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:44:17.210 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is unset 2024-09-20T22:44:17.211 INFO:tasks.rados.rados.0.smithi084.stdout:446: done (0 left) 2024-09-20T22:44:17.211 INFO:tasks.rados.rados.0.smithi084.stdout:447: rollback oid 43 current snap is 40 2024-09-20T22:44:17.211 INFO:tasks.rados.rados.0.smithi084.stdout:447: done (0 left) 2024-09-20T22:44:17.211 INFO:tasks.rados.rados.0.smithi084.stdout:448: read oid 43 snap -1 2024-09-20T22:44:17.211 INFO:tasks.rados.rados.0.smithi084.stdout:448: expect deleted 2024-09-20T22:44:17.211 INFO:tasks.rados.rados.0.smithi084.stdout:449: write oid 44 current snap is 40 2024-09-20T22:44:17.211 INFO:tasks.rados.rados.0.smithi084.stdout:449: seq_num 190 ranges {793330=689782,2035461=617731,3101545=91785} 2024-09-20T22:44:17.218 INFO:tasks.rados.rados.0.smithi084.stdout:449: writing smithi08477203-44 from 793330 to 1483112 tid 1 2024-09-20T22:44:17.225 INFO:tasks.rados.rados.0.smithi084.stdout:449: writing smithi08477203-44 from 2035461 to 2653192 tid 2 2024-09-20T22:44:17.229 INFO:tasks.rados.rados.0.smithi084.stdout:449: writing smithi08477203-44 from 3101545 to 3193330 tid 3 2024-09-20T22:44:17.229 INFO:tasks.rados.rados.0.smithi084.stdout:450: rollback oid 21 current snap is 40 2024-09-20T22:44:17.229 INFO:tasks.rados.rados.0.smithi084.stdout:450: done (2 left) 2024-09-20T22:44:17.229 INFO:tasks.rados.rados.0.smithi084.stdout:451: read oid 50 snap -1 2024-09-20T22:44:17.229 INFO:tasks.rados.rados.0.smithi084.stdout:451: expect (ObjNum 151 snap 30 seq_num 151) 2024-09-20T22:44:17.230 INFO:tasks.rados.rados.0.smithi084.stdout:452: read oid 14 snap -1 2024-09-20T22:44:17.230 INFO:tasks.rados.rados.0.smithi084.stdout:452: expect (ObjNum 13 snap 0 seq_num 13) 2024-09-20T22:44:17.230 INFO:tasks.rados.rados.0.smithi084.stdout:453: copy_from oid 4 from oid 37 current snap is 40 2024-09-20T22:44:17.230 INFO:tasks.rados.rados.0.smithi084.stdout:449: finishing write tid 1 to smithi08477203-44 2024-09-20T22:44:17.230 INFO:tasks.rados.rados.0.smithi084.stdout:449: finishing write tid 2 to smithi08477203-44 2024-09-20T22:44:17.230 INFO:tasks.rados.rados.0.smithi084.stdout:448: done (4 left) 2024-09-20T22:44:17.230 INFO:tasks.rados.rados.0.smithi084.stdout:454: write (excl) oid 47 current snap is 40 2024-09-20T22:44:17.230 INFO:tasks.rados.rados.0.smithi084.stdout:454: seq_num 191 ranges {410137=481384,1410296=761389,2728968=481169} 2024-09-20T22:44:17.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:17.231 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:17.232 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:44:17.234 INFO:tasks.rados.rados.0.smithi084.stdout:454: writing smithi08477203-47 from 410137 to 891521 tid 1 2024-09-20T22:44:17.241 INFO:tasks.rados.rados.0.smithi084.stdout:454: writing smithi08477203-47 from 1410296 to 2171685 tid 2 2024-09-20T22:44:17.248 INFO:tasks.rados.rados.0.smithi084.stdout:454: writing smithi08477203-47 from 2728968 to 3210137 tid 3 2024-09-20T22:44:17.248 INFO:tasks.rados.rados.0.smithi084.stdout:455: rollback oid 40 current snap is 40 2024-09-20T22:44:17.248 INFO:tasks.rados.rados.0.smithi084.stdout:455: done (5 left) 2024-09-20T22:44:17.248 INFO:tasks.rados.rados.0.smithi084.stdout:456: copy_from oid 27 from oid 33 current snap is 40 2024-09-20T22:44:17.248 INFO:tasks.rados.rados.0.smithi084.stdout:449: finishing write tid 3 to smithi08477203-44 2024-09-20T22:44:17.248 INFO:tasks.rados.rados.0.smithi084.stdout:452: expect (ObjNum 13 snap 0 seq_num 13) 2024-09-20T22:44:17.323 INFO:tasks.rados.rados.0.smithi084.stdout:449: finishing write tid 4 to smithi08477203-44 2024-09-20T22:44:17.323 INFO:tasks.rados.rados.0.smithi084.stdout:449: finishing write tid 5 to smithi08477203-44 2024-09-20T22:44:17.323 INFO:tasks.rados.rados.0.smithi084.stdout:449: oid 44 updating version 0 to 203 2024-09-20T22:44:17.323 INFO:tasks.rados.rados.0.smithi084.stdout:449: oid 44 updating version 203 to 206 2024-09-20T22:44:17.323 INFO:tasks.rados.rados.0.smithi084.stdout:449: oid 44 version 206 is already newer than 205 2024-09-20T22:44:17.323 INFO:tasks.rados.rados.0.smithi084.stdout:449: oid 44 version 206 is already newer than 204 2024-09-20T22:44:17.323 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 44 v 206 (ObjNum 190 snap 40 seq_num 190) dirty exists 2024-09-20T22:44:17.323 INFO:tasks.rados.rados.0.smithi084.stdout:449: left oid 44 (ObjNum 190 snap 40 seq_num 190) 2024-09-20T22:44:17.323 INFO:tasks.rados.rados.0.smithi084.stdout:451: expect (ObjNum 151 snap 30 seq_num 151) 2024-09-20T22:44:17.333 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:44:17.354 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:17.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:17.355 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:44:17.404 INFO:tasks.rados.rados.0.smithi084.stdout:454: finishing write tid 1 to smithi08477203-47 2024-09-20T22:44:17.404 INFO:tasks.rados.rados.0.smithi084.stdout:453: finishing copy_from racing read to smithi08477203-4 2024-09-20T22:44:17.404 INFO:tasks.rados.rados.0.smithi084.stdout:453: finishing copy_from to smithi08477203-4 2024-09-20T22:44:17.404 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 4 v 87 (ObjNum 150 snap 30 seq_num 150) dirty exists 2024-09-20T22:44:17.404 INFO:tasks.rados.rados.0.smithi084.stdout:454: finishing write tid 2 to smithi08477203-47 2024-09-20T22:44:17.404 INFO:tasks.rados.rados.0.smithi084.stdout:454: finishing write tid 3 to smithi08477203-47 2024-09-20T22:44:17.405 INFO:tasks.rados.rados.0.smithi084.stdout:454: finishing write tid 4 to smithi08477203-47 2024-09-20T22:44:17.405 INFO:tasks.rados.rados.0.smithi084.stdout:454: finishing write tid 5 to smithi08477203-47 2024-09-20T22:44:17.405 INFO:tasks.rados.rados.0.smithi084.stdout:454: oid 47 updating version 0 to 166 2024-09-20T22:44:17.405 INFO:tasks.rados.rados.0.smithi084.stdout:454: oid 47 updating version 166 to 168 2024-09-20T22:44:17.405 INFO:tasks.rados.rados.0.smithi084.stdout:454: oid 47 updating version 168 to 169 2024-09-20T22:44:17.405 INFO:tasks.rados.rados.0.smithi084.stdout:454: oid 47 version 169 is already newer than 167 2024-09-20T22:44:17.405 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 47 v 169 (ObjNum 191 snap 40 seq_num 191) dirty exists 2024-09-20T22:44:17.405 INFO:tasks.rados.rados.0.smithi084.stdout:454: left oid 47 (ObjNum 191 snap 40 seq_num 191) 2024-09-20T22:44:17.405 INFO:tasks.rados.rados.0.smithi084.stdout:456: finishing copy_from racing read to smithi08477203-27 2024-09-20T22:44:17.405 INFO:tasks.rados.rados.0.smithi084.stdout:456: finishing copy_from to smithi08477203-27 2024-09-20T22:44:17.405 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 27 v 67 (ObjNum 166 snap 36 seq_num 166) dirty exists 2024-09-20T22:44:17.406 INFO:tasks.rados.rados.0.smithi084.stdout:449: done (5 left) 2024-09-20T22:44:17.406 INFO:tasks.rados.rados.0.smithi084.stdout:451: done (4 left) 2024-09-20T22:44:17.406 INFO:tasks.rados.rados.0.smithi084.stdout:452: done (3 left) 2024-09-20T22:44:17.406 INFO:tasks.rados.rados.0.smithi084.stdout:453: done (2 left) 2024-09-20T22:44:17.406 INFO:tasks.rados.rados.0.smithi084.stdout:454: done (1 left) 2024-09-20T22:44:17.406 INFO:tasks.rados.rados.0.smithi084.stdout:456: done (0 left) 2024-09-20T22:44:17.406 INFO:tasks.rados.rados.0.smithi084.stdout:457: delete oid 5 current snap is 40 2024-09-20T22:44:17.407 INFO:tasks.rados.rados.0.smithi084.stdout:457: done (0 left) 2024-09-20T22:44:17.407 INFO:tasks.rados.rados.0.smithi084.stdout:458: write (excl) oid 16 current snap is 40 2024-09-20T22:44:17.407 INFO:tasks.rados.rados.0.smithi084.stdout:458: seq_num 193 ranges {443751=631882,1810907=765058,3067461=176290} 2024-09-20T22:44:17.411 INFO:tasks.rados.rados.0.smithi084.stdout:458: writing smithi08477203-16 from 443751 to 1075633 tid 1 2024-09-20T22:44:17.419 INFO:tasks.rados.rados.0.smithi084.stdout:458: writing smithi08477203-16 from 1810907 to 2575965 tid 2 2024-09-20T22:44:17.423 INFO:tasks.rados.rados.0.smithi084.stdout:458: writing smithi08477203-16 from 3067461 to 3243751 tid 3 2024-09-20T22:44:17.423 INFO:tasks.rados.rados.0.smithi084.stdout:459: read oid 47 snap -1 2024-09-20T22:44:17.424 INFO:tasks.rados.rados.0.smithi084.stdout:459: expect (ObjNum 191 snap 40 seq_num 191) 2024-09-20T22:44:17.424 INFO:tasks.rados.rados.0.smithi084.stdout:460: read oid 5 snap -1 2024-09-20T22:44:17.424 INFO:tasks.rados.rados.0.smithi084.stdout:460: expect deleted 2024-09-20T22:44:17.424 INFO:tasks.rados.rados.0.smithi084.stdout:461: write oid 43 current snap is 40 2024-09-20T22:44:17.424 INFO:tasks.rados.rados.0.smithi084.stdout:458: finishing write tid 1 to smithi08477203-16 2024-09-20T22:44:17.424 INFO:tasks.rados.rados.0.smithi084.stdout:461: seq_num 194 ranges {460558=707131,1869389=508716,3094884=165674} 2024-09-20T22:44:17.430 INFO:tasks.rados.rados.0.smithi084.stdout:461: writing smithi08477203-43 from 460558 to 1167689 tid 1 2024-09-20T22:44:17.438 INFO:tasks.rados.rados.0.smithi084.stdout:461: writing smithi08477203-43 from 1869389 to 2378105 tid 2 2024-09-20T22:44:17.443 INFO:tasks.rados.rados.0.smithi084.stdout:461: writing smithi08477203-43 from 3094884 to 3260558 tid 3 2024-09-20T22:44:17.443 INFO:tasks.rados.rados.0.smithi084.stdout:462: rollback oid 48 current snap is 40 2024-09-20T22:44:17.443 INFO:tasks.rados.rados.0.smithi084.stdout:462: done (4 left) 2024-09-20T22:44:17.443 INFO:tasks.rados.rados.0.smithi084.stdout:463: snap_create 2024-09-20T22:44:17.443 INFO:tasks.rados.rados.0.smithi084.stdout:458: finishing write tid 2 to smithi08477203-16 2024-09-20T22:44:17.444 INFO:tasks.rados.rados.0.smithi084.stdout:458: finishing write tid 3 to smithi08477203-16 2024-09-20T22:44:17.444 INFO:tasks.rados.rados.0.smithi084.stdout:458: finishing write tid 4 to smithi08477203-16 2024-09-20T22:44:17.444 INFO:tasks.rados.rados.0.smithi084.stdout:458: finishing write tid 5 to smithi08477203-16 2024-09-20T22:44:17.444 INFO:tasks.rados.rados.0.smithi084.stdout:458: oid 16 updating version 0 to 123 2024-09-20T22:44:17.444 INFO:tasks.rados.rados.0.smithi084.stdout:458: oid 16 version 123 is already newer than 120 2024-09-20T22:44:17.444 INFO:tasks.rados.rados.0.smithi084.stdout:458: oid 16 version 123 is already newer than 121 2024-09-20T22:44:17.444 INFO:tasks.rados.rados.0.smithi084.stdout:458: oid 16 version 123 is already newer than 122 2024-09-20T22:44:17.444 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 16 v 123 (ObjNum 193 snap 40 seq_num 193) dirty exists 2024-09-20T22:44:17.444 INFO:tasks.rados.rados.0.smithi084.stdout:458: left oid 16 (ObjNum 193 snap 40 seq_num 193) 2024-09-20T22:44:17.444 INFO:tasks.rados.rados.0.smithi084.stdout:459: expect (ObjNum 191 snap 40 seq_num 191) 2024-09-20T22:44:17.456 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:44:17.456 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:17 smithi084 ceph-mon[29774]: 5.1f scrub starts 2024-09-20T22:44:17.457 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:17 smithi084 ceph-mon[29774]: 5.1f scrub ok 2024-09-20T22:44:17.457 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:17 smithi084 ceph-mon[29774]: pgmap v166: 178 pgs: 3 active+clean+snaptrim, 175 active+clean; 117 MiB data, 533 MiB used, 625 GiB / 626 GiB avail; 12 MiB/s rd, 3 op/s; 0 B/s, 0 objects/s recovering 2024-09-20T22:44:17.457 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:17 smithi084 ceph-mon[29774]: 5.1b scrub starts 2024-09-20T22:44:17.457 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:17 smithi084 ceph-mon[29774]: 5.1b scrub ok 2024-09-20T22:44:17.457 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:17 smithi084 ceph-mon[29774]: osdmap e177: 8 total, 8 up, 7 in 2024-09-20T22:44:17.457 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:17 smithi084 ceph-mon[29774]: 4.e scrub starts 2024-09-20T22:44:17.457 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:17 smithi084 ceph-mon[29774]: 4.e scrub ok 2024-09-20T22:44:17.457 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:17 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/2872376294' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:44:17.457 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:17 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:44:17.457 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:17 smithi084 ceph-mon[40024]: 5.1f scrub starts 2024-09-20T22:44:17.458 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:17 smithi084 ceph-mon[40024]: 5.1f scrub ok 2024-09-20T22:44:17.458 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:17 smithi084 ceph-mon[40024]: pgmap v166: 178 pgs: 3 active+clean+snaptrim, 175 active+clean; 117 MiB data, 533 MiB used, 625 GiB / 626 GiB avail; 12 MiB/s rd, 3 op/s; 0 B/s, 0 objects/s recovering 2024-09-20T22:44:17.458 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:17 smithi084 ceph-mon[40024]: 5.1b scrub starts 2024-09-20T22:44:17.458 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:17 smithi084 ceph-mon[40024]: 5.1b scrub ok 2024-09-20T22:44:17.458 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:17 smithi084 ceph-mon[40024]: osdmap e177: 8 total, 8 up, 7 in 2024-09-20T22:44:17.458 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:17 smithi084 ceph-mon[40024]: 4.e scrub starts 2024-09-20T22:44:17.458 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:17 smithi084 ceph-mon[40024]: 4.e scrub ok 2024-09-20T22:44:17.458 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:17 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/2872376294' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:44:17.458 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:17 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:44:17.479 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:17.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:17.480 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:44:17.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:17 smithi154 ceph-mon[33264]: 5.1f scrub starts 2024-09-20T22:44:17.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:17 smithi154 ceph-mon[33264]: 5.1f scrub ok 2024-09-20T22:44:17.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:17 smithi154 ceph-mon[33264]: pgmap v166: 178 pgs: 3 active+clean+snaptrim, 175 active+clean; 117 MiB data, 533 MiB used, 625 GiB / 626 GiB avail; 12 MiB/s rd, 3 op/s; 0 B/s, 0 objects/s recovering 2024-09-20T22:44:17.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:17 smithi154 ceph-mon[33264]: 5.1b scrub starts 2024-09-20T22:44:17.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:17 smithi154 ceph-mon[33264]: 5.1b scrub ok 2024-09-20T22:44:17.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:17 smithi154 ceph-mon[33264]: osdmap e177: 8 total, 8 up, 7 in 2024-09-20T22:44:17.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:17 smithi154 ceph-mon[33264]: 4.e scrub starts 2024-09-20T22:44:17.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:17 smithi154 ceph-mon[33264]: 4.e scrub ok 2024-09-20T22:44:17.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:17 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/2872376294' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:44:17.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:17 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:44:17.581 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:44:17.602 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:17.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:17.603 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:44:17.613 INFO:tasks.rados.rados.0.smithi084.stdout:461: finishing write tid 1 to smithi08477203-43 2024-09-20T22:44:17.614 INFO:tasks.rados.rados.0.smithi084.stdout:461: finishing write tid 2 to smithi08477203-43 2024-09-20T22:44:17.614 INFO:tasks.rados.rados.0.smithi084.stdout:461: finishing write tid 3 to smithi08477203-43 2024-09-20T22:44:17.614 INFO:tasks.rados.rados.0.smithi084.stdout:461: finishing write tid 4 to smithi08477203-43 2024-09-20T22:44:17.614 INFO:tasks.rados.rados.0.smithi084.stdout:461: finishing write tid 5 to smithi08477203-43 2024-09-20T22:44:17.614 INFO:tasks.rados.rados.0.smithi084.stdout:461: oid 43 updating version 0 to 43 2024-09-20T22:44:17.614 INFO:tasks.rados.rados.0.smithi084.stdout:461: oid 43 version 43 is already newer than 41 2024-09-20T22:44:17.614 INFO:tasks.rados.rados.0.smithi084.stdout:461: oid 43 version 43 is already newer than 42 2024-09-20T22:44:17.614 INFO:tasks.rados.rados.0.smithi084.stdout:461: oid 43 updating version 43 to 44 2024-09-20T22:44:17.614 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 43 v 44 (ObjNum 194 snap 40 seq_num 194) dirty exists 2024-09-20T22:44:17.614 INFO:tasks.rados.rados.0.smithi084.stdout:461: left oid 43 (ObjNum 194 snap 40 seq_num 194) 2024-09-20T22:44:17.704 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:44:17.726 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:17.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:17.727 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:44:17.827 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:44:17.829 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.3 dump_blocked_ops 2024-09-20T22:44:17.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:17.849 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:17.850 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:44:17.950 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:44:17.971 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:17.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:17.972 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:44:18.073 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:44:18.094 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:18.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:18.095 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:44:18.134 INFO:tasks.rados.rados.0.smithi084.stdout:458: done (4 left) 2024-09-20T22:44:18.134 INFO:tasks.rados.rados.0.smithi084.stdout:459: done (3 left) 2024-09-20T22:44:18.134 INFO:tasks.rados.rados.0.smithi084.stdout:460: done (2 left) 2024-09-20T22:44:18.134 INFO:tasks.rados.rados.0.smithi084.stdout:461: done (1 left) 2024-09-20T22:44:18.134 INFO:tasks.rados.rados.0.smithi084.stdout:463: done (0 left) 2024-09-20T22:44:18.134 INFO:tasks.rados.rados.0.smithi084.stdout:464: read oid 23 snap 40 2024-09-20T22:44:18.134 INFO:tasks.rados.rados.0.smithi084.stdout:464: expect (ObjNum 106 snap 22 seq_num 106) 2024-09-20T22:44:18.134 INFO:tasks.rados.rados.0.smithi084.stdout:465: write oid 18 current snap is 41 2024-09-20T22:44:18.135 INFO:tasks.rados.rados.0.smithi084.stdout:465: seq_num 195 ranges {477365=782380,1927871=768727,3122307=155058} 2024-09-20T22:44:18.141 INFO:tasks.rados.rados.0.smithi084.stdout:465: writing smithi08477203-18 from 477365 to 1259745 tid 1 2024-09-20T22:44:18.149 INFO:tasks.rados.rados.0.smithi084.stdout:465: writing smithi08477203-18 from 1927871 to 2696598 tid 2 2024-09-20T22:44:18.153 INFO:tasks.rados.rados.0.smithi084.stdout:465: writing smithi08477203-18 from 3122307 to 3277365 tid 3 2024-09-20T22:44:18.153 INFO:tasks.rados.rados.0.smithi084.stdout:466: rollback oid 34 current snap is 41 2024-09-20T22:44:18.153 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 34 to 40 2024-09-20T22:44:18.153 INFO:tasks.rados.rados.0.smithi084.stdout:464: expect (ObjNum 106 snap 22 seq_num 106) 2024-09-20T22:44:18.195 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:44:18.217 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:18.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:18.218 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:44:18.318 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:44:18.331 INFO:tasks.rados.rados.0.smithi084.stdout:465: finishing write tid 1 to smithi08477203-18 2024-09-20T22:44:18.331 INFO:tasks.rados.rados.0.smithi084.stdout:465: finishing write tid 2 to smithi08477203-18 2024-09-20T22:44:18.331 INFO:tasks.rados.rados.0.smithi084.stdout:464: done (2 left) 2024-09-20T22:44:18.332 INFO:tasks.rados.rados.0.smithi084.stdout:467: write oid 6 current snap is 41 2024-09-20T22:44:18.332 INFO:tasks.rados.rados.0.smithi084.stdout:465: finishing write tid 3 to smithi08477203-18 2024-09-20T22:44:18.332 INFO:tasks.rados.rados.0.smithi084.stdout:465: finishing write tid 4 to smithi08477203-18 2024-09-20T22:44:18.332 INFO:tasks.rados.rados.0.smithi084.stdout:465: finishing write tid 5 to smithi08477203-18 2024-09-20T22:44:18.332 INFO:tasks.rados.rados.0.smithi084.stdout:465: oid 18 updating version 0 to 95 2024-09-20T22:44:18.332 INFO:tasks.rados.rados.0.smithi084.stdout:465: oid 18 version 95 is already newer than 92 2024-09-20T22:44:18.332 INFO:tasks.rados.rados.0.smithi084.stdout:465: oid 18 version 95 is already newer than 93 2024-09-20T22:44:18.332 INFO:tasks.rados.rados.0.smithi084.stdout:465: oid 18 version 95 is already newer than 94 2024-09-20T22:44:18.332 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 18 v 95 (ObjNum 195 snap 41 seq_num 195) dirty exists 2024-09-20T22:44:18.332 INFO:tasks.rados.rados.0.smithi084.stdout:465: left oid 18 (ObjNum 195 snap 41 seq_num 195) 2024-09-20T22:44:18.332 INFO:tasks.rados.rados.0.smithi084.stdout:466: finishing rollback tid 0 to smithi08477203-34 2024-09-20T22:44:18.332 INFO:tasks.rados.rados.0.smithi084.stdout:466: finishing rollback tid 1 to smithi08477203-34 2024-09-20T22:44:18.332 INFO:tasks.rados.rados.0.smithi084.stdout:466: finishing rollback tid 2 to smithi08477203-34 2024-09-20T22:44:18.333 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 34 v 154 (ObjNum 50 snap 0 seq_num 50) dirty exists 2024-09-20T22:44:18.333 INFO:tasks.rados.rados.0.smithi084.stdout:467: seq_num 196 ranges {494172=457629,1586353=512385,2749730=544442} 2024-09-20T22:44:18.336 INFO:tasks.rados.rados.0.smithi084.stdout:467: writing smithi08477203-6 from 494172 to 951801 tid 1 2024-09-20T22:44:18.339 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:44:18.343 INFO:tasks.rados.rados.0.smithi084.stdout:467: writing smithi08477203-6 from 1586353 to 2098738 tid 2 2024-09-20T22:44:18.343 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:18.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:18.344 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:44:18.350 INFO:tasks.rados.rados.0.smithi084.stdout:467: writing smithi08477203-6 from 2749730 to 3294172 tid 3 2024-09-20T22:44:18.350 INFO:tasks.rados.rados.0.smithi084.stdout:465: done (2 left) 2024-09-20T22:44:18.350 INFO:tasks.rados.rados.0.smithi084.stdout:466: done (1 left) 2024-09-20T22:44:18.351 INFO:tasks.rados.rados.0.smithi084.stdout:468: read oid 33 snap 40 2024-09-20T22:44:18.351 INFO:tasks.rados.rados.0.smithi084.stdout:468: expect (ObjNum 166 snap 36 seq_num 166) 2024-09-20T22:44:18.351 INFO:tasks.rados.rados.0.smithi084.stdout:469: write (excl) oid 23 current snap is 41 2024-09-20T22:44:18.351 INFO:tasks.rados.rados.0.smithi084.stdout:467: finishing write tid 1 to smithi08477203-6 2024-09-20T22:44:18.351 INFO:tasks.rados.rados.0.smithi084.stdout:469: seq_num 197 ranges {510979=532878,1528482=772396,3060800=250179} 2024-09-20T22:44:18.356 INFO:tasks.rados.rados.0.smithi084.stdout:469: writing smithi08477203-23 from 510979 to 1043857 tid 1 2024-09-20T22:44:18.364 INFO:tasks.rados.rados.0.smithi084.stdout:469: writing smithi08477203-23 from 1528482 to 2300878 tid 2 2024-09-20T22:44:18.370 INFO:tasks.rados.rados.0.smithi084.stdout:469: writing smithi08477203-23 from 3060800 to 3310979 tid 3 2024-09-20T22:44:18.370 INFO:tasks.rados.rados.0.smithi084.stdout:470: read oid 16 snap 40 2024-09-20T22:44:18.370 INFO:tasks.rados.rados.0.smithi084.stdout:470: expect (ObjNum 193 snap 40 seq_num 193) 2024-09-20T22:44:18.371 INFO:tasks.rados.rados.0.smithi084.stdout:471: write (excl) oid 35 current snap is 41 2024-09-20T22:44:18.371 INFO:tasks.rados.rados.0.smithi084.stdout:467: finishing write tid 2 to smithi08477203-6 2024-09-20T22:44:18.371 INFO:tasks.rados.rados.0.smithi084.stdout:471: seq_num 198 ranges {527786=724480,1703317=516054,2804576=435035} 2024-09-20T22:44:18.378 INFO:tasks.rados.rados.0.smithi084.stdout:471: writing smithi08477203-35 from 527786 to 1252266 tid 1 2024-09-20T22:44:18.384 INFO:tasks.rados.rados.0.smithi084.stdout:471: writing smithi08477203-35 from 1703317 to 2219371 tid 2 2024-09-20T22:44:18.390 INFO:tasks.rados.rados.0.smithi084.stdout:471: writing smithi08477203-35 from 2804576 to 3239611 tid 3 2024-09-20T22:44:18.390 INFO:tasks.rados.rados.0.smithi084.stdout:472: delete oid 24 current snap is 41 2024-09-20T22:44:18.390 INFO:tasks.rados.rados.0.smithi084.stdout:467: finishing write tid 3 to smithi08477203-6 2024-09-20T22:44:18.390 INFO:tasks.rados.rados.0.smithi084.stdout:467: finishing write tid 4 to smithi08477203-6 2024-09-20T22:44:18.390 INFO:tasks.rados.rados.0.smithi084.stdout:467: finishing write tid 5 to smithi08477203-6 2024-09-20T22:44:18.391 INFO:tasks.rados.rados.0.smithi084.stdout:467: oid 6 updating version 0 to 65 2024-09-20T22:44:18.391 INFO:tasks.rados.rados.0.smithi084.stdout:467: oid 6 version 65 is already newer than 64 2024-09-20T22:44:18.391 INFO:tasks.rados.rados.0.smithi084.stdout:467: oid 6 version 65 is already newer than 62 2024-09-20T22:44:18.391 INFO:tasks.rados.rados.0.smithi084.stdout:467: oid 6 version 65 is already newer than 63 2024-09-20T22:44:18.391 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 6 v 65 (ObjNum 196 snap 41 seq_num 196) dirty exists 2024-09-20T22:44:18.391 INFO:tasks.rados.rados.0.smithi084.stdout:467: left oid 6 (ObjNum 196 snap 41 seq_num 196) 2024-09-20T22:44:18.391 INFO:tasks.rados.rados.0.smithi084.stdout:468: expect (ObjNum 166 snap 36 seq_num 166) 2024-09-20T22:44:18.444 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-20T22:44:18.445 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:44:18.466 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:18.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:18.467 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:44:18.498 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:44:18 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: ::ffff:172.21.15.154 - - [20/Sep/2024:22:44:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-20T22:44:18.498 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:18 smithi084 ceph-mon[40024]: 3.f scrub starts 2024-09-20T22:44:18.498 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:18 smithi084 ceph-mon[40024]: 3.f scrub ok 2024-09-20T22:44:18.498 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:18 smithi084 ceph-mon[40024]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T22:44:18.498 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:18 smithi084 ceph-mon[40024]: Cluster is now healthy 2024-09-20T22:44:18.498 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:18 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T22:44:18.498 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:18 smithi084 ceph-mon[40024]: osdmap e178: 8 total, 8 up, 7 in 2024-09-20T22:44:18.498 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:18 smithi084 ceph-mon[40024]: 4.10 scrub starts 2024-09-20T22:44:18.498 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:18 smithi084 ceph-mon[40024]: 4.10 scrub ok 2024-09-20T22:44:18.499 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:18 smithi084 ceph-mon[40024]: 5.19 scrub starts 2024-09-20T22:44:18.499 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:18 smithi084 ceph-mon[40024]: 5.19 scrub ok 2024-09-20T22:44:18.499 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:18 smithi084 ceph-mon[40024]: osdmap e179: 8 total, 8 up, 7 in 2024-09-20T22:44:18.499 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:18 smithi084 ceph-mon[29774]: 3.f scrub starts 2024-09-20T22:44:18.499 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:18 smithi084 ceph-mon[29774]: 3.f scrub ok 2024-09-20T22:44:18.499 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:18 smithi084 ceph-mon[29774]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T22:44:18.499 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:18 smithi084 ceph-mon[29774]: Cluster is now healthy 2024-09-20T22:44:18.499 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:18 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T22:44:18.499 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:18 smithi084 ceph-mon[29774]: osdmap e178: 8 total, 8 up, 7 in 2024-09-20T22:44:18.499 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:18 smithi084 ceph-mon[29774]: 4.10 scrub starts 2024-09-20T22:44:18.499 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:18 smithi084 ceph-mon[29774]: 4.10 scrub ok 2024-09-20T22:44:18.499 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:18 smithi084 ceph-mon[29774]: 5.19 scrub starts 2024-09-20T22:44:18.500 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:18 smithi084 ceph-mon[29774]: 5.19 scrub ok 2024-09-20T22:44:18.500 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:18 smithi084 ceph-mon[29774]: osdmap e179: 8 total, 8 up, 7 in 2024-09-20T22:44:18.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:18 smithi154 ceph-mon[33264]: 3.f scrub starts 2024-09-20T22:44:18.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:18 smithi154 ceph-mon[33264]: 3.f scrub ok 2024-09-20T22:44:18.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:18 smithi154 ceph-mon[33264]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T22:44:18.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:18 smithi154 ceph-mon[33264]: Cluster is now healthy 2024-09-20T22:44:18.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:18 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T22:44:18.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:18 smithi154 ceph-mon[33264]: osdmap e178: 8 total, 8 up, 7 in 2024-09-20T22:44:18.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:18 smithi154 ceph-mon[33264]: 4.10 scrub starts 2024-09-20T22:44:18.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:18 smithi154 ceph-mon[33264]: 4.10 scrub ok 2024-09-20T22:44:18.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:18 smithi154 ceph-mon[33264]: 5.19 scrub starts 2024-09-20T22:44:18.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:18 smithi154 ceph-mon[33264]: 5.19 scrub ok 2024-09-20T22:44:18.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:18 smithi154 ceph-mon[33264]: osdmap e179: 8 total, 8 up, 7 in 2024-09-20T22:44:18.515 INFO:tasks.rados.rados.0.smithi084.stdout:469: finishing write tid 1 to smithi08477203-23 2024-09-20T22:44:18.515 INFO:tasks.rados.rados.0.smithi084.stdout:469: finishing write tid 2 to smithi08477203-23 2024-09-20T22:44:18.515 INFO:tasks.rados.rados.0.smithi084.stdout:467: done (5 left) 2024-09-20T22:44:18.515 INFO:tasks.rados.rados.0.smithi084.stdout:468: done (4 left) 2024-09-20T22:44:18.515 INFO:tasks.rados.rados.0.smithi084.stdout:472: done (3 left) 2024-09-20T22:44:18.515 INFO:tasks.rados.rados.0.smithi084.stdout:473: write oid 48 current snap is 41 2024-09-20T22:44:18.516 INFO:tasks.rados.rados.0.smithi084.stdout:469: finishing write tid 3 to smithi08477203-23 2024-09-20T22:44:18.516 INFO:tasks.rados.rados.0.smithi084.stdout:469: finishing write tid 4 to smithi08477203-23 2024-09-20T22:44:18.516 INFO:tasks.rados.rados.0.smithi084.stdout:473: seq_num 200 ranges {561400=474978,1820281=519723,2859422=457579} 2024-09-20T22:44:18.521 INFO:tasks.rados.rados.0.smithi084.stdout:473: writing smithi08477203-48 from 561400 to 1036378 tid 1 2024-09-20T22:44:18.530 INFO:tasks.rados.rados.0.smithi084.stdout:473: writing smithi08477203-48 from 1820281 to 2340004 tid 2 2024-09-20T22:44:18.536 INFO:tasks.rados.rados.0.smithi084.stdout:473: writing smithi08477203-48 from 2859422 to 3317001 tid 3 2024-09-20T22:44:18.536 INFO:tasks.rados.rados.0.smithi084.stdout:474: copy_from oid 27 from oid 14 current snap is 41 2024-09-20T22:44:18.536 INFO:tasks.rados.rados.0.smithi084.stdout:469: finishing write tid 5 to smithi08477203-23 2024-09-20T22:44:18.536 INFO:tasks.rados.rados.0.smithi084.stdout:469: oid 23 updating version 0 to 91 2024-09-20T22:44:18.537 INFO:tasks.rados.rados.0.smithi084.stdout:469: oid 23 version 91 is already newer than 90 2024-09-20T22:44:18.537 INFO:tasks.rados.rados.0.smithi084.stdout:469: oid 23 version 91 is already newer than 89 2024-09-20T22:44:18.537 INFO:tasks.rados.rados.0.smithi084.stdout:469: oid 23 version 91 is already newer than 88 2024-09-20T22:44:18.537 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 23 v 91 (ObjNum 197 snap 41 seq_num 197) dirty exists 2024-09-20T22:44:18.537 INFO:tasks.rados.rados.0.smithi084.stdout:469: left oid 23 (ObjNum 197 snap 41 seq_num 197) 2024-09-20T22:44:18.537 INFO:tasks.rados.rados.0.smithi084.stdout:470: expect (ObjNum 193 snap 40 seq_num 193) 2024-09-20T22:44:18.567 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:44:18.587 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:18.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:18.588 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:44:18.689 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:44:18.691 INFO:tasks.rados.rados.0.smithi084.stdout:471: finishing write tid 1 to smithi08477203-35 2024-09-20T22:44:18.692 INFO:tasks.rados.rados.0.smithi084.stdout:469: done (4 left) 2024-09-20T22:44:18.692 INFO:tasks.rados.rados.0.smithi084.stdout:470: done (3 left) 2024-09-20T22:44:18.692 INFO:tasks.rados.rados.0.smithi084.stdout:475: rollback oid 33 current snap is 41 2024-09-20T22:44:18.692 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 33 to 40 2024-09-20T22:44:18.692 INFO:tasks.rados.rados.0.smithi084.stdout:471: finishing write tid 2 to smithi08477203-35 2024-09-20T22:44:18.692 INFO:tasks.rados.rados.0.smithi084.stdout:471: finishing write tid 3 to smithi08477203-35 2024-09-20T22:44:18.692 INFO:tasks.rados.rados.0.smithi084.stdout:471: finishing write tid 4 to smithi08477203-35 2024-09-20T22:44:18.692 INFO:tasks.rados.rados.0.smithi084.stdout:471: finishing write tid 5 to smithi08477203-35 2024-09-20T22:44:18.692 INFO:tasks.rados.rados.0.smithi084.stdout:471: oid 35 updating version 0 to 75 2024-09-20T22:44:18.692 INFO:tasks.rados.rados.0.smithi084.stdout:471: oid 35 version 75 is already newer than 73 2024-09-20T22:44:18.692 INFO:tasks.rados.rados.0.smithi084.stdout:471: oid 35 version 75 is already newer than 74 2024-09-20T22:44:18.692 INFO:tasks.rados.rados.0.smithi084.stdout:471: oid 35 version 75 is already newer than 72 2024-09-20T22:44:18.692 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 35 v 75 (ObjNum 198 snap 41 seq_num 198) dirty exists 2024-09-20T22:44:18.693 INFO:tasks.rados.rados.0.smithi084.stdout:471: left oid 35 (ObjNum 198 snap 41 seq_num 198) 2024-09-20T22:44:18.693 INFO:tasks.rados.rados.0.smithi084.stdout:473: finishing write tid 1 to smithi08477203-48 2024-09-20T22:44:18.693 INFO:tasks.rados.rados.0.smithi084.stdout:473: finishing write tid 2 to smithi08477203-48 2024-09-20T22:44:18.693 INFO:tasks.rados.rados.0.smithi084.stdout:473: finishing write tid 3 to smithi08477203-48 2024-09-20T22:44:18.693 INFO:tasks.rados.rados.0.smithi084.stdout:473: finishing write tid 4 to smithi08477203-48 2024-09-20T22:44:18.693 INFO:tasks.rados.rados.0.smithi084.stdout:473: finishing write tid 5 to smithi08477203-48 2024-09-20T22:44:18.693 INFO:tasks.rados.rados.0.smithi084.stdout:473: oid 48 updating version 0 to 90 2024-09-20T22:44:18.693 INFO:tasks.rados.rados.0.smithi084.stdout:473: oid 48 version 90 is already newer than 88 2024-09-20T22:44:18.693 INFO:tasks.rados.rados.0.smithi084.stdout:473: oid 48 version 90 is already newer than 89 2024-09-20T22:44:18.693 INFO:tasks.rados.rados.0.smithi084.stdout:473: oid 48 updating version 90 to 91 2024-09-20T22:44:18.693 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 48 v 91 (ObjNum 200 snap 41 seq_num 200) dirty exists 2024-09-20T22:44:18.693 INFO:tasks.rados.rados.0.smithi084.stdout:473: left oid 48 (ObjNum 200 snap 41 seq_num 200) 2024-09-20T22:44:18.693 INFO:tasks.rados.rados.0.smithi084.stdout:474: finishing copy_from racing read to smithi08477203-27 2024-09-20T22:44:18.693 INFO:tasks.rados.rados.0.smithi084.stdout:474: finishing copy_from to smithi08477203-27 2024-09-20T22:44:18.694 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 27 v 3 (ObjNum 13 snap 0 seq_num 13) dirty exists 2024-09-20T22:44:18.694 INFO:tasks.rados.rados.0.smithi084.stdout:471: done (3 left) 2024-09-20T22:44:18.694 INFO:tasks.rados.rados.0.smithi084.stdout:473: done (2 left) 2024-09-20T22:44:18.694 INFO:tasks.rados.rados.0.smithi084.stdout:474: done (1 left) 2024-09-20T22:44:18.694 INFO:tasks.rados.rados.0.smithi084.stdout:476: read oid 49 snap -1 2024-09-20T22:44:18.694 INFO:tasks.rados.rados.0.smithi084.stdout:476: expect (ObjNum 171 snap 37 seq_num 171) 2024-09-20T22:44:18.694 INFO:tasks.rados.rados.0.smithi084.stdout:477: write (excl) oid 2 current snap is 41 2024-09-20T22:44:18.694 INFO:tasks.rados.rados.0.smithi084.stdout:477: seq_num 201 ranges {578207=550227,1762410=779734,3286845=91362} 2024-09-20T22:44:18.697 INFO:tasks.rados.rados.0.smithi084.stdout:477: writing smithi08477203-2 from 578207 to 1128434 tid 1 2024-09-20T22:44:18.706 INFO:tasks.rados.rados.0.smithi084.stdout:477: writing smithi08477203-2 from 1762410 to 2542144 tid 2 2024-09-20T22:44:18.711 INFO:tasks.rados.rados.0.smithi084.stdout:477: writing smithi08477203-2 from 3286845 to 3378207 tid 3 2024-09-20T22:44:18.711 INFO:tasks.rados.rados.0.smithi084.stdout:478: write oid 12 current snap is 41 2024-09-20T22:44:18.711 INFO:tasks.rados.rados.0.smithi084.stdout:475: finishing rollback tid 0 to smithi08477203-33 2024-09-20T22:44:18.711 INFO:tasks.rados.rados.0.smithi084.stdout:475: finishing rollback tid 1 to smithi08477203-33 2024-09-20T22:44:18.711 INFO:tasks.rados.rados.0.smithi084.stdout:478: seq_num 202 ranges {595014=625476,1820892=523392,2797915=596476} 2024-09-20T22:44:18.712 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:18.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:18.713 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:44:18.717 INFO:tasks.rados.rados.0.smithi084.stdout:478: writing smithi08477203-12 from 595014 to 1220490 tid 1 2024-09-20T22:44:18.723 INFO:tasks.rados.rados.0.smithi084.stdout:478: writing smithi08477203-12 from 1820892 to 2344284 tid 2 2024-09-20T22:44:18.730 INFO:tasks.rados.rados.0.smithi084.stdout:478: writing smithi08477203-12 from 2797915 to 3394391 tid 3 2024-09-20T22:44:18.730 INFO:tasks.rados.rados.0.smithi084.stdout:479: rollback oid 48 current snap is 41 2024-09-20T22:44:18.730 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 48 to 40 2024-09-20T22:44:18.730 INFO:tasks.rados.rados.0.smithi084.stdout:475: finishing rollback tid 2 to smithi08477203-33 2024-09-20T22:44:18.730 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 33 v 95 (ObjNum 166 snap 36 seq_num 166) dirty exists 2024-09-20T22:44:18.730 INFO:tasks.rados.rados.0.smithi084.stdout:475: done (4 left) 2024-09-20T22:44:18.730 INFO:tasks.rados.rados.0.smithi084.stdout:480: write oid 17 current snap is 41 2024-09-20T22:44:18.730 INFO:tasks.rados.rados.0.smithi084.stdout:477: finishing write tid 1 to smithi08477203-2 2024-09-20T22:44:18.730 INFO:tasks.rados.rados.0.smithi084.stdout:480: seq_num 203 ranges {611821=700725,1879374=783403,3341691=70130} 2024-09-20T22:44:18.736 INFO:tasks.rados.rados.0.smithi084.stdout:480: writing smithi08477203-17 from 611821 to 1312546 tid 1 2024-09-20T22:44:18.744 INFO:tasks.rados.rados.0.smithi084.stdout:480: writing smithi08477203-17 from 1879374 to 2662777 tid 2 2024-09-20T22:44:18.748 INFO:tasks.rados.rados.0.smithi084.stdout:480: writing smithi08477203-17 from 3341691 to 3411821 tid 3 2024-09-20T22:44:18.748 INFO:tasks.rados.rados.0.smithi084.stdout:481: read oid 40 snap -1 2024-09-20T22:44:18.748 INFO:tasks.rados.rados.0.smithi084.stdout:481: expect (ObjNum 186 snap 39 seq_num 186) 2024-09-20T22:44:18.748 INFO:tasks.rados.rados.0.smithi084.stdout:482: read oid 35 snap 40 2024-09-20T22:44:18.749 INFO:tasks.rados.rados.0.smithi084.stdout:482: expect (ObjNum 42 snap 0 seq_num 42) 2024-09-20T22:44:18.749 INFO:tasks.rados.rados.0.smithi084.stdout:483: write (excl) oid 10 current snap is 41 2024-09-20T22:44:18.749 INFO:tasks.rados.rados.0.smithi084.stdout:483: seq_num 204 ranges {628628=775974,1937856=527061,3252761=175867} 2024-09-20T22:44:18.757 INFO:tasks.rados.rados.0.smithi084.stdout:483: writing smithi08477203-10 from 628628 to 1404602 tid 1 2024-09-20T22:44:18.763 INFO:tasks.rados.rados.0.smithi084.stdout:483: writing smithi08477203-10 from 1937856 to 2464917 tid 2 2024-09-20T22:44:18.768 INFO:tasks.rados.rados.0.smithi084.stdout:483: writing smithi08477203-10 from 3252761 to 3428628 tid 3 2024-09-20T22:44:18.769 INFO:tasks.rados.rados.0.smithi084.stdout:484: snap_create 2024-09-20T22:44:18.769 INFO:tasks.rados.rados.0.smithi084.stdout:476: expect (ObjNum 171 snap 37 seq_num 171) 2024-09-20T22:44:18.814 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:44:18.837 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:18.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:18.838 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:44:18.939 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:44:18.961 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:18.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:18.961 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:44:18.969 INFO:tasks.rados.rados.0.smithi084.stdout:477: finishing write tid 2 to smithi08477203-2 2024-09-20T22:44:18.969 INFO:tasks.rados.rados.0.smithi084.stdout:477: finishing write tid 3 to smithi08477203-2 2024-09-20T22:44:18.969 INFO:tasks.rados.rados.0.smithi084.stdout:477: finishing write tid 4 to smithi08477203-2 2024-09-20T22:44:18.970 INFO:tasks.rados.rados.0.smithi084.stdout:477: finishing write tid 5 to smithi08477203-2 2024-09-20T22:44:18.970 INFO:tasks.rados.rados.0.smithi084.stdout:477: oid 2 updating version 0 to 83 2024-09-20T22:44:18.970 INFO:tasks.rados.rados.0.smithi084.stdout:477: oid 2 version 83 is already newer than 82 2024-09-20T22:44:18.970 INFO:tasks.rados.rados.0.smithi084.stdout:477: oid 2 version 83 is already newer than 80 2024-09-20T22:44:18.970 INFO:tasks.rados.rados.0.smithi084.stdout:477: oid 2 version 83 is already newer than 81 2024-09-20T22:44:18.970 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 2 v 83 (ObjNum 201 snap 41 seq_num 201) dirty exists 2024-09-20T22:44:18.970 INFO:tasks.rados.rados.0.smithi084.stdout:477: left oid 2 (ObjNum 201 snap 41 seq_num 201) 2024-09-20T22:44:18.970 INFO:tasks.rados.rados.0.smithi084.stdout:478: finishing write tid 1 to smithi08477203-12 2024-09-20T22:44:18.970 INFO:tasks.rados.rados.0.smithi084.stdout:478: finishing write tid 2 to smithi08477203-12 2024-09-20T22:44:18.970 INFO:tasks.rados.rados.0.smithi084.stdout:479: finishing rollback tid 0 to smithi08477203-48 2024-09-20T22:44:18.970 INFO:tasks.rados.rados.0.smithi084.stdout:479: finishing rollback tid 1 to smithi08477203-48 2024-09-20T22:44:18.970 INFO:tasks.rados.rados.0.smithi084.stdout:479: finishing rollback tid 2 to smithi08477203-48 2024-09-20T22:44:18.970 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 48 v 94 (ObjNum 129 snap 23 seq_num 129) dirty exists 2024-09-20T22:44:18.971 INFO:tasks.rados.rados.0.smithi084.stdout:478: finishing write tid 3 to smithi08477203-12 2024-09-20T22:44:18.971 INFO:tasks.rados.rados.0.smithi084.stdout:478: finishing write tid 4 to smithi08477203-12 2024-09-20T22:44:18.971 INFO:tasks.rados.rados.0.smithi084.stdout:478: finishing write tid 5 to smithi08477203-12 2024-09-20T22:44:18.971 INFO:tasks.rados.rados.0.smithi084.stdout:478: oid 12 updating version 0 to 147 2024-09-20T22:44:18.971 INFO:tasks.rados.rados.0.smithi084.stdout:478: oid 12 updating version 147 to 148 2024-09-20T22:44:18.971 INFO:tasks.rados.rados.0.smithi084.stdout:478: oid 12 updating version 148 to 150 2024-09-20T22:44:18.971 INFO:tasks.rados.rados.0.smithi084.stdout:478: oid 12 version 150 is already newer than 149 2024-09-20T22:44:18.971 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 12 v 150 (ObjNum 202 snap 41 seq_num 202) dirty exists 2024-09-20T22:44:18.971 INFO:tasks.rados.rados.0.smithi084.stdout:478: left oid 12 (ObjNum 202 snap 41 seq_num 202) 2024-09-20T22:44:18.971 INFO:tasks.rados.rados.0.smithi084.stdout:480: finishing write tid 1 to smithi08477203-17 2024-09-20T22:44:18.971 INFO:tasks.rados.rados.0.smithi084.stdout:480: finishing write tid 2 to smithi08477203-17 2024-09-20T22:44:18.971 INFO:tasks.rados.rados.0.smithi084.stdout:480: finishing write tid 3 to smithi08477203-17 2024-09-20T22:44:18.971 INFO:tasks.rados.rados.0.smithi084.stdout:480: finishing write tid 4 to smithi08477203-17 2024-09-20T22:44:18.971 INFO:tasks.rados.rados.0.smithi084.stdout:481: expect (ObjNum 186 snap 39 seq_num 186) 2024-09-20T22:44:19.062 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:44:19.085 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:19.085 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:44:19.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:19.086 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:44:19.112 INFO:tasks.rados.rados.0.smithi084.stdout:480: finishing write tid 5 to smithi08477203-17 2024-09-20T22:44:19.112 INFO:tasks.rados.rados.0.smithi084.stdout:480: oid 17 updating version 0 to 70 2024-09-20T22:44:19.113 INFO:tasks.rados.rados.0.smithi084.stdout:480: oid 17 version 70 is already newer than 67 2024-09-20T22:44:19.113 INFO:tasks.rados.rados.0.smithi084.stdout:480: oid 17 version 70 is already newer than 69 2024-09-20T22:44:19.113 INFO:tasks.rados.rados.0.smithi084.stdout:480: oid 17 version 70 is already newer than 68 2024-09-20T22:44:19.113 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 17 v 70 (ObjNum 203 snap 41 seq_num 203) dirty exists 2024-09-20T22:44:19.113 INFO:tasks.rados.rados.0.smithi084.stdout:480: left oid 17 (ObjNum 203 snap 41 seq_num 203) 2024-09-20T22:44:19.113 INFO:tasks.rados.rados.0.smithi084.stdout:482: expect (ObjNum 42 snap 0 seq_num 42) 2024-09-20T22:44:19.187 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:44:19.202 INFO:tasks.rados.rados.0.smithi084.stdout:483: finishing write tid 1 to smithi08477203-10 2024-09-20T22:44:19.202 INFO:tasks.rados.rados.0.smithi084.stdout:483: finishing write tid 2 to smithi08477203-10 2024-09-20T22:44:19.202 INFO:tasks.rados.rados.0.smithi084.stdout:483: finishing write tid 3 to smithi08477203-10 2024-09-20T22:44:19.202 INFO:tasks.rados.rados.0.smithi084.stdout:483: finishing write tid 4 to smithi08477203-10 2024-09-20T22:44:19.202 INFO:tasks.rados.rados.0.smithi084.stdout:483: finishing write tid 5 to smithi08477203-10 2024-09-20T22:44:19.203 INFO:tasks.rados.rados.0.smithi084.stdout:483: oid 10 updating version 0 to 98 2024-09-20T22:44:19.203 INFO:tasks.rados.rados.0.smithi084.stdout:483: oid 10 version 98 is already newer than 96 2024-09-20T22:44:19.203 INFO:tasks.rados.rados.0.smithi084.stdout:483: oid 10 updating version 98 to 99 2024-09-20T22:44:19.203 INFO:tasks.rados.rados.0.smithi084.stdout:483: oid 10 version 99 is already newer than 97 2024-09-20T22:44:19.203 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 10 v 99 (ObjNum 204 snap 41 seq_num 204) dirty exists 2024-09-20T22:44:19.203 INFO:tasks.rados.rados.0.smithi084.stdout:483: left oid 10 (ObjNum 204 snap 41 seq_num 204) 2024-09-20T22:44:19.208 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:19.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:19.209 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:44:19.226 INFO:tasks.rados.rados.0.smithi084.stdout:476: done (8 left) 2024-09-20T22:44:19.226 INFO:tasks.rados.rados.0.smithi084.stdout:477: done (7 left) 2024-09-20T22:44:19.226 INFO:tasks.rados.rados.0.smithi084.stdout:478: done (6 left) 2024-09-20T22:44:19.226 INFO:tasks.rados.rados.0.smithi084.stdout:479: done (5 left) 2024-09-20T22:44:19.226 INFO:tasks.rados.rados.0.smithi084.stdout:480: done (4 left) 2024-09-20T22:44:19.226 INFO:tasks.rados.rados.0.smithi084.stdout:481: done (3 left) 2024-09-20T22:44:19.226 INFO:tasks.rados.rados.0.smithi084.stdout:482: done (2 left) 2024-09-20T22:44:19.226 INFO:tasks.rados.rados.0.smithi084.stdout:483: done (1 left) 2024-09-20T22:44:19.226 INFO:tasks.rados.rados.0.smithi084.stdout:484: done (0 left) 2024-09-20T22:44:19.226 INFO:tasks.rados.rados.0.smithi084.stdout:485: delete oid 24 current snap is 42 2024-09-20T22:44:19.234 INFO:tasks.rados.rados.0.smithi084.stdout:485: done (0 left) 2024-09-20T22:44:19.234 INFO:tasks.rados.rados.0.smithi084.stdout:486: write oid 5 current snap is 42 2024-09-20T22:44:19.235 INFO:tasks.rados.rados.0.smithi084.stdout:486: seq_num 206 ranges {662242=642825,2054820=530730,3307607=154635} 2024-09-20T22:44:19.241 INFO:tasks.rados.rados.0.smithi084.stdout:486: writing smithi08477203-5 from 662242 to 1305067 tid 1 2024-09-20T22:44:19.248 INFO:tasks.rados.rados.0.smithi084.stdout:486: writing smithi08477203-5 from 2054820 to 2585550 tid 2 2024-09-20T22:44:19.255 INFO:tasks.rados.rados.0.smithi084.stdout:486: writing smithi08477203-5 from 3307607 to 3462242 tid 3 2024-09-20T22:44:19.255 INFO:tasks.rados.rados.0.smithi084.stdout:487: read oid 17 snap -1 2024-09-20T22:44:19.255 INFO:tasks.rados.rados.0.smithi084.stdout:487: expect (ObjNum 203 snap 41 seq_num 203) 2024-09-20T22:44:19.255 INFO:tasks.rados.rados.0.smithi084.stdout:488: read oid 6 snap -1 2024-09-20T22:44:19.255 INFO:tasks.rados.rados.0.smithi084.stdout:488: expect (ObjNum 196 snap 41 seq_num 196) 2024-09-20T22:44:19.255 INFO:tasks.rados.rados.0.smithi084.stdout:489: read oid 48 snap -1 2024-09-20T22:44:19.255 INFO:tasks.rados.rados.0.smithi084.stdout:489: expect (ObjNum 129 snap 23 seq_num 129) 2024-09-20T22:44:19.255 INFO:tasks.rados.rados.0.smithi084.stdout:490: write oid 10 current snap is 42 2024-09-20T22:44:19.255 INFO:tasks.rados.rados.0.smithi084.stdout:486: finishing write tid 1 to smithi08477203-5 2024-09-20T22:44:19.256 INFO:tasks.rados.rados.0.smithi084.stdout:486: finishing write tid 2 to smithi08477203-5 2024-09-20T22:44:19.256 INFO:tasks.rados.rados.0.smithi084.stdout:490: seq_num 207 ranges {679049=718074,2113302=674388,3335030=144019} 2024-09-20T22:44:19.262 INFO:tasks.rados.rados.0.smithi084.stdout:490: writing smithi08477203-10 from 679049 to 1397123 tid 1 2024-09-20T22:44:19.269 INFO:tasks.rados.rados.0.smithi084.stdout:490: writing smithi08477203-10 from 2113302 to 2787690 tid 2 2024-09-20T22:44:19.275 INFO:tasks.rados.rados.0.smithi084.stdout:490: writing smithi08477203-10 from 3335030 to 3479049 tid 3 2024-09-20T22:44:19.275 INFO:tasks.rados.rados.0.smithi084.stdout:491: read oid 43 snap 40 2024-09-20T22:44:19.275 INFO:tasks.rados.rados.0.smithi084.stdout:491: expect (ObjNum 194 snap 40 seq_num 194) 2024-09-20T22:44:19.275 INFO:tasks.rados.rados.0.smithi084.stdout:492: read oid 1 snap -1 2024-09-20T22:44:19.276 INFO:tasks.rados.rados.0.smithi084.stdout:492: expect (ObjNum 0 snap 0 seq_num 0) 2024-09-20T22:44:19.276 INFO:tasks.rados.rados.0.smithi084.stdout:493: rollback oid 3 current snap is 42 2024-09-20T22:44:19.276 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 3 to 40 2024-09-20T22:44:19.276 INFO:tasks.rados.rados.0.smithi084.stdout:486: finishing write tid 3 to smithi08477203-5 2024-09-20T22:44:19.276 INFO:tasks.rados.rados.0.smithi084.stdout:486: finishing write tid 4 to smithi08477203-5 2024-09-20T22:44:19.276 INFO:tasks.rados.rados.0.smithi084.stdout:486: finishing write tid 5 to smithi08477203-5 2024-09-20T22:44:19.276 INFO:tasks.rados.rados.0.smithi084.stdout:486: oid 5 updating version 0 to 79 2024-09-20T22:44:19.277 INFO:tasks.rados.rados.0.smithi084.stdout:486: oid 5 version 79 is already newer than 78 2024-09-20T22:44:19.277 INFO:tasks.rados.rados.0.smithi084.stdout:486: oid 5 version 79 is already newer than 77 2024-09-20T22:44:19.277 INFO:tasks.rados.rados.0.smithi084.stdout:486: oid 5 version 79 is already newer than 76 2024-09-20T22:44:19.277 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 5 v 79 (ObjNum 206 snap 42 seq_num 206) dirty exists 2024-09-20T22:44:19.277 INFO:tasks.rados.rados.0.smithi084.stdout:486: left oid 5 (ObjNum 206 snap 42 seq_num 206) 2024-09-20T22:44:19.277 INFO:tasks.rados.rados.0.smithi084.stdout:489: expect (ObjNum 129 snap 23 seq_num 129) 2024-09-20T22:44:19.310 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:44:19.332 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:19.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:19.333 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:44:19.347 INFO:tasks.rados.rados.0.smithi084.stdout:487: expect (ObjNum 203 snap 41 seq_num 203) 2024-09-20T22:44:19.433 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:44:19.454 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:19.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:19.455 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:44:19.491 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:19 smithi084 ceph-mon[40024]: pgmap v169: 178 pgs: 3 active+clean+snaptrim, 175 active+clean; 117 MiB data, 533 MiB used, 625 GiB / 626 GiB avail 2024-09-20T22:44:19.491 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:19 smithi084 ceph-mon[40024]: 4.1 scrub starts 2024-09-20T22:44:19.491 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:19 smithi084 ceph-mon[40024]: 4.1 scrub ok 2024-09-20T22:44:19.491 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:19 smithi084 ceph-mon[40024]: 7.e deep-scrub starts 2024-09-20T22:44:19.491 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:19 smithi084 ceph-mon[40024]: 7.e deep-scrub ok 2024-09-20T22:44:19.491 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:19 smithi084 ceph-mon[29774]: pgmap v169: 178 pgs: 3 active+clean+snaptrim, 175 active+clean; 117 MiB data, 533 MiB used, 625 GiB / 626 GiB avail 2024-09-20T22:44:19.491 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:19 smithi084 ceph-mon[29774]: 4.1 scrub starts 2024-09-20T22:44:19.491 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:19 smithi084 ceph-mon[29774]: 4.1 scrub ok 2024-09-20T22:44:19.492 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:19 smithi084 ceph-mon[29774]: 7.e deep-scrub starts 2024-09-20T22:44:19.492 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:19 smithi084 ceph-mon[29774]: 7.e deep-scrub ok 2024-09-20T22:44:19.492 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph tell osd.5 reset_purged_snaps_last 2024-09-20T22:44:19.504 INFO:journalctl@ceph.iscsi.iscsi.a.smithi154.stdout:Sep 20 22:44:19 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-iscsi-iscsi-a[59730]: debug there is no tcmu-runner data available 2024-09-20T22:44:19.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:19 smithi154 ceph-mon[33264]: pgmap v169: 178 pgs: 3 active+clean+snaptrim, 175 active+clean; 117 MiB data, 533 MiB used, 625 GiB / 626 GiB avail 2024-09-20T22:44:19.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:19 smithi154 ceph-mon[33264]: 4.1 scrub starts 2024-09-20T22:44:19.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:19 smithi154 ceph-mon[33264]: 4.1 scrub ok 2024-09-20T22:44:19.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:19 smithi154 ceph-mon[33264]: 7.e deep-scrub starts 2024-09-20T22:44:19.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:19 smithi154 ceph-mon[33264]: 7.e deep-scrub ok 2024-09-20T22:44:19.526 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:44:19.556 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:44:19.577 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:19.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:19.578 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:44:19.592 INFO:tasks.rados.rados.0.smithi084.stdout:490: finishing write tid 1 to smithi08477203-10 2024-09-20T22:44:19.592 INFO:tasks.rados.rados.0.smithi084.stdout:488: expect (ObjNum 196 snap 41 seq_num 196) 2024-09-20T22:44:19.679 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:44:19.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:19.701 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:19.701 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:44:19.757 INFO:tasks.rados.rados.0.smithi084.stdout:490: finishing write tid 2 to smithi08477203-10 2024-09-20T22:44:19.757 INFO:tasks.rados.rados.0.smithi084.stdout:486: done (7 left) 2024-09-20T22:44:19.757 INFO:tasks.rados.rados.0.smithi084.stdout:487: done (6 left) 2024-09-20T22:44:19.757 INFO:tasks.rados.rados.0.smithi084.stdout:488: done (5 left) 2024-09-20T22:44:19.757 INFO:tasks.rados.rados.0.smithi084.stdout:489: done (4 left) 2024-09-20T22:44:19.757 INFO:tasks.rados.rados.0.smithi084.stdout:494: snap_remove snap 41 2024-09-20T22:44:19.802 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:44:19.823 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:19.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:19.824 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:44:19.925 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:44:19.946 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:19.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:19.947 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:44:20.048 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:44:20.069 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:20.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:20.070 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:44:20.171 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:44:20.192 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:20.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:20.193 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:44:20.196 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-20T22:44:20.196 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2024-09-20T22:44:20.196 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2024-09-20T22:44:20.196 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2024-09-20T22:44:20.196 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-20T22:44:20.240 INFO:tasks.rados.rados.0.smithi084.stdout:494: done (4 left) 2024-09-20T22:44:20.240 INFO:tasks.rados.rados.0.smithi084.stdout:495: write (excl) oid 47 current snap is 42 2024-09-20T22:44:20.241 INFO:tasks.rados.rados.0.smithi084.stdout:495: seq_num 208 ranges {695856=793323,2171784=534399,3362453=133403} 2024-09-20T22:44:20.249 INFO:tasks.rados.rados.0.smithi084.stdout:495: writing smithi08477203-47 from 695856 to 1489179 tid 1 2024-09-20T22:44:20.257 INFO:tasks.rados.rados.0.smithi084.stdout:495: writing smithi08477203-47 from 2171784 to 2706183 tid 2 2024-09-20T22:44:20.262 INFO:tasks.rados.rados.0.smithi084.stdout:495: writing smithi08477203-47 from 3362453 to 3495856 tid 3 2024-09-20T22:44:20.262 INFO:tasks.rados.rados.0.smithi084.stdout:496: delete oid 16 current snap is 42 2024-09-20T22:44:20.262 INFO:tasks.rados.rados.0.smithi084.stdout:493: finishing rollback tid 0 to smithi08477203-3 2024-09-20T22:44:20.262 INFO:tasks.rados.rados.0.smithi084.stdout:490: finishing write tid 3 to smithi08477203-10 2024-09-20T22:44:20.262 INFO:tasks.rados.rados.0.smithi084.stdout:490: finishing write tid 4 to smithi08477203-10 2024-09-20T22:44:20.262 INFO:tasks.rados.rados.0.smithi084.stdout:490: finishing write tid 5 to smithi08477203-10 2024-09-20T22:44:20.262 INFO:tasks.rados.rados.0.smithi084.stdout:490: oid 10 updating version 0 to 102 2024-09-20T22:44:20.262 INFO:tasks.rados.rados.0.smithi084.stdout:490: oid 10 version 102 is already newer than 101 2024-09-20T22:44:20.262 INFO:tasks.rados.rados.0.smithi084.stdout:490: oid 10 updating version 102 to 103 2024-09-20T22:44:20.263 INFO:tasks.rados.rados.0.smithi084.stdout:490: oid 10 updating version 103 to 104 2024-09-20T22:44:20.263 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 10 v 104 (ObjNum 207 snap 42 seq_num 207) dirty exists 2024-09-20T22:44:20.263 INFO:tasks.rados.rados.0.smithi084.stdout:490: left oid 10 (ObjNum 207 snap 42 seq_num 207) 2024-09-20T22:44:20.263 INFO:tasks.rados.rados.0.smithi084.stdout:493: finishing rollback tid 1 to smithi08477203-3 2024-09-20T22:44:20.263 INFO:tasks.rados.rados.0.smithi084.stdout:493: finishing rollback tid 2 to smithi08477203-3 2024-09-20T22:44:20.263 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 3 v 94 (ObjNum 108 snap 22 seq_num 108) dirty exists 2024-09-20T22:44:20.263 INFO:tasks.rados.rados.0.smithi084.stdout:492: expect (ObjNum 0 snap 0 seq_num 0) 2024-09-20T22:44:20.295 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:44:20.316 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:20.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:20.317 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:44:20.328 INFO:tasks.rados.rados.0.smithi084.stdout:491: expect (ObjNum 194 snap 40 seq_num 194) 2024-09-20T22:44:20.348 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd set noscrub 2024-09-20T22:44:20.364 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:20 smithi084 ceph-mon[40024]: 3.1b scrub starts 2024-09-20T22:44:20.364 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:20 smithi084 ceph-mon[40024]: 3.1b scrub ok 2024-09-20T22:44:20.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:20 smithi084 ceph-mon[40024]: 3.d scrub starts 2024-09-20T22:44:20.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:20 smithi084 ceph-mon[40024]: 3.d scrub ok 2024-09-20T22:44:20.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:20 smithi084 ceph-mon[40024]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:44:20.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:20 smithi084 ceph-mon[40024]: osdmap e180: 8 total, 8 up, 7 in 2024-09-20T22:44:20.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:20 smithi084 ceph-mon[40024]: 5.2 scrub starts 2024-09-20T22:44:20.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:20 smithi084 ceph-mon[40024]: 5.2 scrub ok 2024-09-20T22:44:20.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:20 smithi084 ceph-mon[40024]: 5.12 scrub starts 2024-09-20T22:44:20.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:20 smithi084 ceph-mon[40024]: 5.12 scrub ok 2024-09-20T22:44:20.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:20 smithi084 ceph-mon[29774]: 3.1b scrub starts 2024-09-20T22:44:20.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:20 smithi084 ceph-mon[29774]: 3.1b scrub ok 2024-09-20T22:44:20.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:20 smithi084 ceph-mon[29774]: 3.d scrub starts 2024-09-20T22:44:20.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:20 smithi084 ceph-mon[29774]: 3.d scrub ok 2024-09-20T22:44:20.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:20 smithi084 ceph-mon[29774]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:44:20.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:20 smithi084 ceph-mon[29774]: osdmap e180: 8 total, 8 up, 7 in 2024-09-20T22:44:20.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:20 smithi084 ceph-mon[29774]: 5.2 scrub starts 2024-09-20T22:44:20.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:20 smithi084 ceph-mon[29774]: 5.2 scrub ok 2024-09-20T22:44:20.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:20 smithi084 ceph-mon[29774]: 5.12 scrub starts 2024-09-20T22:44:20.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:20 smithi084 ceph-mon[29774]: 5.12 scrub ok 2024-09-20T22:44:20.415 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:44:20.415 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:44:20.417 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:44:20.439 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:20.440 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:44:20.440 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:44:20.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:20.441 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:44:20.451 INFO:tasks.rados.rados.0.smithi084.stdout:495: finishing write tid 1 to smithi08477203-47 2024-09-20T22:44:20.451 INFO:tasks.rados.rados.0.smithi084.stdout:495: finishing write tid 2 to smithi08477203-47 2024-09-20T22:44:20.451 INFO:tasks.rados.rados.0.smithi084.stdout:490: done (5 left) 2024-09-20T22:44:20.451 INFO:tasks.rados.rados.0.smithi084.stdout:491: done (4 left) 2024-09-20T22:44:20.451 INFO:tasks.rados.rados.0.smithi084.stdout:492: done (3 left) 2024-09-20T22:44:20.451 INFO:tasks.rados.rados.0.smithi084.stdout:493: done (2 left) 2024-09-20T22:44:20.451 INFO:tasks.rados.rados.0.smithi084.stdout:496: done (1 left) 2024-09-20T22:44:20.451 INFO:tasks.rados.rados.0.smithi084.stdout:497: snap_remove snap 40 2024-09-20T22:44:20.456 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:44:20.456 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:44:20.472 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-20T22:44:20.472 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T22:44:20.489 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:44:20.489 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:44:20.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:20 smithi154 ceph-mon[33264]: 3.1b scrub starts 2024-09-20T22:44:20.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:20 smithi154 ceph-mon[33264]: 3.1b scrub ok 2024-09-20T22:44:20.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:20 smithi154 ceph-mon[33264]: 3.d scrub starts 2024-09-20T22:44:20.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:20 smithi154 ceph-mon[33264]: 3.d scrub ok 2024-09-20T22:44:20.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:20 smithi154 ceph-mon[33264]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:44:20.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:20 smithi154 ceph-mon[33264]: osdmap e180: 8 total, 8 up, 7 in 2024-09-20T22:44:20.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:20 smithi154 ceph-mon[33264]: 5.2 scrub starts 2024-09-20T22:44:20.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:20 smithi154 ceph-mon[33264]: 5.2 scrub ok 2024-09-20T22:44:20.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:20 smithi154 ceph-mon[33264]: 5.12 scrub starts 2024-09-20T22:44:20.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:20 smithi154 ceph-mon[33264]: 5.12 scrub ok 2024-09-20T22:44:20.506 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:44:20.507 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:44:20.523 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:44:20.523 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:44:20.540 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:44:20.540 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:44:20.541 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:44:20.563 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:20.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:20.564 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:44:20.665 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:44:20.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:20.686 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:20.687 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:44:20.787 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:44:20.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:20.809 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:20.810 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:44:20.853 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:44:20.910 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:44:20.931 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:20.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:20.932 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:44:21.033 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:44:21.055 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:21.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:21.056 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:44:21.157 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:44:21.178 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:21.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:21.179 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:44:21.244 INFO:tasks.rados.rados.0.smithi084.stdout:497: done (1 left) 2024-09-20T22:44:21.244 INFO:tasks.rados.rados.0.smithi084.stdout:498: delete oid 12 current snap is 42 2024-09-20T22:44:21.244 INFO:tasks.rados.rados.0.smithi084.stdout:495: finishing write tid 3 to smithi08477203-47 2024-09-20T22:44:21.244 INFO:tasks.rados.rados.0.smithi084.stdout:495: finishing write tid 4 to smithi08477203-47 2024-09-20T22:44:21.244 INFO:tasks.rados.rados.0.smithi084.stdout:495: finishing write tid 5 to smithi08477203-47 2024-09-20T22:44:21.244 INFO:tasks.rados.rados.0.smithi084.stdout:495: oid 47 updating version 0 to 173 2024-09-20T22:44:21.244 INFO:tasks.rados.rados.0.smithi084.stdout:495: oid 47 version 173 is already newer than 172 2024-09-20T22:44:21.244 INFO:tasks.rados.rados.0.smithi084.stdout:495: oid 47 updating version 173 to 174 2024-09-20T22:44:21.244 INFO:tasks.rados.rados.0.smithi084.stdout:495: oid 47 version 174 is already newer than 171 2024-09-20T22:44:21.244 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 47 v 174 (ObjNum 208 snap 42 seq_num 208) dirty exists 2024-09-20T22:44:21.244 INFO:tasks.rados.rados.0.smithi084.stdout:495: left oid 47 (ObjNum 208 snap 42 seq_num 208) 2024-09-20T22:44:21.251 INFO:tasks.rados.rados.0.smithi084.stdout:495: done (1 left) 2024-09-20T22:44:21.251 INFO:tasks.rados.rados.0.smithi084.stdout:498: done (0 left) 2024-09-20T22:44:21.251 INFO:tasks.rados.rados.0.smithi084.stdout:499: snap_create 2024-09-20T22:44:21.280 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:44:21.302 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:21.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:21.303 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:44:21.374 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.3 dump_historic_ops 2024-09-20T22:44:21.403 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:44:21.404 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:44:21.426 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:21.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:21.427 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:44:21.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:21 smithi154 ceph-mon[33264]: pgmap v172: 178 pgs: 1 active+clean+snaptrim, 177 active+clean; 130 MiB data, 535 MiB used, 625 GiB / 626 GiB avail; 22 MiB/s rd, 8.3 MiB/s wr, 41 op/s 2024-09-20T22:44:21.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:21 smithi154 ceph-mon[33264]: 3.1 deep-scrub starts 2024-09-20T22:44:21.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:21 smithi154 ceph-mon[33264]: 3.1 deep-scrub ok 2024-09-20T22:44:21.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:21 smithi154 ceph-mon[33264]: 6.3 scrub starts 2024-09-20T22:44:21.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:21 smithi154 ceph-mon[33264]: 6.3 scrub ok 2024-09-20T22:44:21.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:21 smithi154 ceph-mon[33264]: osdmap e181: 8 total, 8 up, 7 in 2024-09-20T22:44:21.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:21 smithi154 ceph-mon[33264]: 4.1b deep-scrub starts 2024-09-20T22:44:21.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:21 smithi154 ceph-mon[33264]: 4.1b deep-scrub ok 2024-09-20T22:44:21.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:21 smithi154 ceph-mon[33264]: 6.f scrub starts 2024-09-20T22:44:21.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:21 smithi154 ceph-mon[33264]: 6.f scrub ok 2024-09-20T22:44:21.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:21 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/1176058794' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:44:21.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:21 smithi154 ceph-mon[33264]: 6.1d scrub starts 2024-09-20T22:44:21.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:21 smithi154 ceph-mon[33264]: 6.1d scrub ok 2024-09-20T22:44:21.506 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:21 smithi084 ceph-mon[40024]: pgmap v172: 178 pgs: 1 active+clean+snaptrim, 177 active+clean; 130 MiB data, 535 MiB used, 625 GiB / 626 GiB avail; 22 MiB/s rd, 8.3 MiB/s wr, 41 op/s 2024-09-20T22:44:21.506 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:21 smithi084 ceph-mon[40024]: 3.1 deep-scrub starts 2024-09-20T22:44:21.507 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:21 smithi084 ceph-mon[40024]: 3.1 deep-scrub ok 2024-09-20T22:44:21.507 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:21 smithi084 ceph-mon[40024]: 6.3 scrub starts 2024-09-20T22:44:21.507 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:21 smithi084 ceph-mon[40024]: 6.3 scrub ok 2024-09-20T22:44:21.507 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:21 smithi084 ceph-mon[40024]: osdmap e181: 8 total, 8 up, 7 in 2024-09-20T22:44:21.507 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:21 smithi084 ceph-mon[40024]: 4.1b deep-scrub starts 2024-09-20T22:44:21.507 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:21 smithi084 ceph-mon[40024]: 4.1b deep-scrub ok 2024-09-20T22:44:21.507 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:21 smithi084 ceph-mon[40024]: 6.f scrub starts 2024-09-20T22:44:21.507 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:21 smithi084 ceph-mon[40024]: 6.f scrub ok 2024-09-20T22:44:21.507 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:21 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/1176058794' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:44:21.507 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:21 smithi084 ceph-mon[40024]: 6.1d scrub starts 2024-09-20T22:44:21.507 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:21 smithi084 ceph-mon[40024]: 6.1d scrub ok 2024-09-20T22:44:21.508 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:21 smithi084 ceph-mon[29774]: pgmap v172: 178 pgs: 1 active+clean+snaptrim, 177 active+clean; 130 MiB data, 535 MiB used, 625 GiB / 626 GiB avail; 22 MiB/s rd, 8.3 MiB/s wr, 41 op/s 2024-09-20T22:44:21.508 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:21 smithi084 ceph-mon[29774]: 3.1 deep-scrub starts 2024-09-20T22:44:21.508 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:21 smithi084 ceph-mon[29774]: 3.1 deep-scrub ok 2024-09-20T22:44:21.508 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:21 smithi084 ceph-mon[29774]: 6.3 scrub starts 2024-09-20T22:44:21.508 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:21 smithi084 ceph-mon[29774]: 6.3 scrub ok 2024-09-20T22:44:21.508 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:21 smithi084 ceph-mon[29774]: osdmap e181: 8 total, 8 up, 7 in 2024-09-20T22:44:21.508 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:21 smithi084 ceph-mon[29774]: 4.1b deep-scrub starts 2024-09-20T22:44:21.508 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:21 smithi084 ceph-mon[29774]: 4.1b deep-scrub ok 2024-09-20T22:44:21.508 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:21 smithi084 ceph-mon[29774]: 6.f scrub starts 2024-09-20T22:44:21.508 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:21 smithi084 ceph-mon[29774]: 6.f scrub ok 2024-09-20T22:44:21.508 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:21 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/1176058794' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:44:21.508 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:21 smithi084 ceph-mon[29774]: 6.1d scrub starts 2024-09-20T22:44:21.508 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:21 smithi084 ceph-mon[29774]: 6.1d scrub ok 2024-09-20T22:44:21.528 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:44:21.549 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:21.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:21.550 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:44:21.651 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:44:21.694 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:21.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:21.695 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:44:21.796 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:44:21.817 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:21.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:21.817 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:44:21.918 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:44:21.940 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:21.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:21.941 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:44:22.042 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:44:22.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:22.065 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:22.066 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:44:22.166 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:44:22.188 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:22.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:22.189 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:44:22.249 INFO:tasks.rados.rados.0.smithi084.stdout:499: done (0 left) 2024-09-20T22:44:22.249 INFO:tasks.rados.rados.0.smithi084.stdout:500: snap_create 2024-09-20T22:44:22.290 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:44:22.310 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:44:22.312 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:22.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:22.313 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:44:22.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:22 smithi084 ceph-mon[40024]: 6.18 scrub starts 2024-09-20T22:44:22.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:22 smithi084 ceph-mon[40024]: 6.18 scrub ok 2024-09-20T22:44:22.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:22 smithi084 ceph-mon[40024]: 3.e scrub starts 2024-09-20T22:44:22.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:22 smithi084 ceph-mon[40024]: 3.e scrub ok 2024-09-20T22:44:22.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:22 smithi084 ceph-mon[40024]: 6.c scrub starts 2024-09-20T22:44:22.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:22 smithi084 ceph-mon[40024]: 6.c scrub ok 2024-09-20T22:44:22.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:22 smithi084 ceph-mon[40024]: osdmap e182: 8 total, 8 up, 7 in 2024-09-20T22:44:22.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:22 smithi084 ceph-mon[40024]: 4.d scrub starts 2024-09-20T22:44:22.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:22 smithi084 ceph-mon[40024]: 4.d scrub ok 2024-09-20T22:44:22.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:22 smithi084 ceph-mon[40024]: 5.1d scrub starts 2024-09-20T22:44:22.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:22 smithi084 ceph-mon[40024]: 5.1d scrub ok 2024-09-20T22:44:22.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:22 smithi084 ceph-mon[29774]: 6.18 scrub starts 2024-09-20T22:44:22.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:22 smithi084 ceph-mon[29774]: 6.18 scrub ok 2024-09-20T22:44:22.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:22 smithi084 ceph-mon[29774]: 3.e scrub starts 2024-09-20T22:44:22.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:22 smithi084 ceph-mon[29774]: 3.e scrub ok 2024-09-20T22:44:22.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:22 smithi084 ceph-mon[29774]: 6.c scrub starts 2024-09-20T22:44:22.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:22 smithi084 ceph-mon[29774]: 6.c scrub ok 2024-09-20T22:44:22.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:22 smithi084 ceph-mon[29774]: osdmap e182: 8 total, 8 up, 7 in 2024-09-20T22:44:22.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:22 smithi084 ceph-mon[29774]: 4.d scrub starts 2024-09-20T22:44:22.367 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:22 smithi084 ceph-mon[29774]: 4.d scrub ok 2024-09-20T22:44:22.367 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:22 smithi084 ceph-mon[29774]: 5.1d scrub starts 2024-09-20T22:44:22.367 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:22 smithi084 ceph-mon[29774]: 5.1d scrub ok 2024-09-20T22:44:22.413 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:44:22.435 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:22.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:22.436 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:44:22.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:22 smithi154 ceph-mon[33264]: 6.18 scrub starts 2024-09-20T22:44:22.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:22 smithi154 ceph-mon[33264]: 6.18 scrub ok 2024-09-20T22:44:22.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:22 smithi154 ceph-mon[33264]: 3.e scrub starts 2024-09-20T22:44:22.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:22 smithi154 ceph-mon[33264]: 3.e scrub ok 2024-09-20T22:44:22.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:22 smithi154 ceph-mon[33264]: 6.c scrub starts 2024-09-20T22:44:22.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:22 smithi154 ceph-mon[33264]: 6.c scrub ok 2024-09-20T22:44:22.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:22 smithi154 ceph-mon[33264]: osdmap e182: 8 total, 8 up, 7 in 2024-09-20T22:44:22.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:22 smithi154 ceph-mon[33264]: 4.d scrub starts 2024-09-20T22:44:22.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:22 smithi154 ceph-mon[33264]: 4.d scrub ok 2024-09-20T22:44:22.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:22 smithi154 ceph-mon[33264]: 5.1d scrub starts 2024-09-20T22:44:22.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:22 smithi154 ceph-mon[33264]: 5.1d scrub ok 2024-09-20T22:44:22.537 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:44:22.558 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:22.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:22.559 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:44:22.660 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:44:22.681 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:22.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:22.682 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:44:22.783 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:44:22.805 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:22.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:22.806 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:44:22.907 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:44:22.929 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:22.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:22.930 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:44:23.030 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:44:23.053 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:23.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:23.053 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:44:23.055 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph tell osd.6 reset_purged_snaps_last 2024-09-20T22:44:23.136 INFO:tasks.rados.rados.0.smithi084.stdout:500: done (0 left) 2024-09-20T22:44:23.136 INFO:tasks.rados.rados.0.smithi084.stdout:501: write (excl) oid 16 current snap is 44 2024-09-20T22:44:23.136 INFO:tasks.rados.rados.0.smithi084.stdout:501: seq_num 211 ranges {746277=619070,1830877=681726,2928369=617908} 2024-09-20T22:44:23.144 INFO:tasks.rados.rados.0.smithi084.stdout:501: writing smithi08477203-16 from 746277 to 1365347 tid 1 2024-09-20T22:44:23.151 INFO:tasks.rados.rados.0.smithi084.stdout:501: writing smithi08477203-16 from 1830877 to 2512603 tid 2 2024-09-20T22:44:23.154 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:44:23.157 INFO:tasks.rados.rados.0.smithi084.stdout:501: writing smithi08477203-16 from 2928369 to 3546277 tid 3 2024-09-20T22:44:23.157 INFO:tasks.rados.rados.0.smithi084.stdout:502: write oid 40 current snap is 44 2024-09-20T22:44:23.157 INFO:tasks.rados.rados.0.smithi084.stdout:502: seq_num 212 ranges {763084=694319,1889359=541737,2955792=541902} 2024-09-20T22:44:23.165 INFO:tasks.rados.rados.0.smithi084.stdout:502: writing smithi08477203-40 from 763084 to 1457403 tid 1 2024-09-20T22:44:23.171 INFO:tasks.rados.rados.0.smithi084.stdout:502: writing smithi08477203-40 from 1889359 to 2431096 tid 2 2024-09-20T22:44:23.177 INFO:tasks.rados.rados.0.smithi084.stdout:502: writing smithi08477203-40 from 2955792 to 3497694 tid 3 2024-09-20T22:44:23.177 INFO:tasks.rados.rados.0.smithi084.stdout:503: snap_remove snap 43 2024-09-20T22:44:23.179 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:23.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:23.180 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:44:23.281 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:44:23.289 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:23 smithi084 ceph-mon[40024]: pgmap v175: 178 pgs: 1 active+clean+snaptrim, 177 active+clean; 130 MiB data, 535 MiB used, 625 GiB / 626 GiB avail; 22 MiB/s rd, 8.3 MiB/s wr, 41 op/s 2024-09-20T22:44:23.289 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:23 smithi084 ceph-mon[40024]: osdmap e183: 8 total, 8 up, 7 in 2024-09-20T22:44:23.289 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:23 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:44:23.289 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:23 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:44:23.289 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:23 smithi084 ceph-mon[40024]: osdmap e184: 8 total, 8 up, 7 in 2024-09-20T22:44:23.289 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:23 smithi084 ceph-mon[29774]: pgmap v175: 178 pgs: 1 active+clean+snaptrim, 177 active+clean; 130 MiB data, 535 MiB used, 625 GiB / 626 GiB avail; 22 MiB/s rd, 8.3 MiB/s wr, 41 op/s 2024-09-20T22:44:23.289 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:23 smithi084 ceph-mon[29774]: osdmap e183: 8 total, 8 up, 7 in 2024-09-20T22:44:23.290 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:23 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:44:23.290 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:23 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:44:23.290 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:23 smithi084 ceph-mon[29774]: osdmap e184: 8 total, 8 up, 7 in 2024-09-20T22:44:23.302 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:23.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:23.303 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:44:23.404 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:44:23.426 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:23.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:23.427 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:44:23.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:23 smithi154 ceph-mon[33264]: pgmap v175: 178 pgs: 1 active+clean+snaptrim, 177 active+clean; 130 MiB data, 535 MiB used, 625 GiB / 626 GiB avail; 22 MiB/s rd, 8.3 MiB/s wr, 41 op/s 2024-09-20T22:44:23.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:23 smithi154 ceph-mon[33264]: osdmap e183: 8 total, 8 up, 7 in 2024-09-20T22:44:23.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:23 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:44:23.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:23 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:44:23.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:23 smithi154 ceph-mon[33264]: osdmap e184: 8 total, 8 up, 7 in 2024-09-20T22:44:23.528 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:44:23.549 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:23.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:23.550 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:44:23.652 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:44:23.673 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:23.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:23.674 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:44:23.775 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:44:23.796 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:23.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:23.797 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:44:23.828 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-20T22:44:23.898 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:44:23.919 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:23.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:23.920 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:44:23.993 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-20T22:44:23.993 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:44:23.995 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:44:24.021 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:44:24.043 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:24.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:24.044 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:44:24.131 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is set 2024-09-20T22:44:24.139 INFO:tasks.rados.rados.0.smithi084.stdout:503: done (2 left) 2024-09-20T22:44:24.139 INFO:tasks.rados.rados.0.smithi084.stdout:504: snap_remove snap 42 2024-09-20T22:44:24.145 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:44:24.168 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:24.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:24.169 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:44:24.270 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:44:24.291 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:24.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:24.292 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:44:24.393 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:44:24.414 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:24.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:24.415 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:44:24.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:24 smithi154 ceph-mon[33264]: 6.9 scrub starts 2024-09-20T22:44:24.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:24 smithi154 ceph-mon[33264]: 6.9 scrub ok 2024-09-20T22:44:24.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:24 smithi154 ceph-mon[33264]: 4.0 scrub starts 2024-09-20T22:44:24.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:24 smithi154 ceph-mon[33264]: 4.0 scrub ok 2024-09-20T22:44:24.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:24 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/3535544267' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-20T22:44:24.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:24 smithi154 ceph-mon[33264]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:44:24.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:24 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/3535544267' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T22:44:24.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:24 smithi154 ceph-mon[33264]: osdmap e185: 8 total, 8 up, 7 in 2024-09-20T22:44:24.515 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:44:24.537 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:24 smithi084 ceph-mon[40024]: 6.9 scrub starts 2024-09-20T22:44:24.537 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:24 smithi084 ceph-mon[40024]: 6.9 scrub ok 2024-09-20T22:44:24.537 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:24 smithi084 ceph-mon[40024]: 4.0 scrub starts 2024-09-20T22:44:24.537 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:24 smithi084 ceph-mon[40024]: 4.0 scrub ok 2024-09-20T22:44:24.537 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:24 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/3535544267' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-20T22:44:24.537 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:24 smithi084 ceph-mon[40024]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:44:24.537 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:24 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/3535544267' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T22:44:24.537 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:24 smithi084 ceph-mon[40024]: osdmap e185: 8 total, 8 up, 7 in 2024-09-20T22:44:24.538 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:24 smithi084 ceph-mon[29774]: 6.9 scrub starts 2024-09-20T22:44:24.538 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:24 smithi084 ceph-mon[29774]: 6.9 scrub ok 2024-09-20T22:44:24.538 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:24 smithi084 ceph-mon[29774]: 4.0 scrub starts 2024-09-20T22:44:24.538 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:24 smithi084 ceph-mon[29774]: 4.0 scrub ok 2024-09-20T22:44:24.538 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:24 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/3535544267' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-20T22:44:24.538 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:24 smithi084 ceph-mon[29774]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:44:24.538 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:24 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/3535544267' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T22:44:24.538 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:24 smithi084 ceph-mon[29774]: osdmap e185: 8 total, 8 up, 7 in 2024-09-20T22:44:24.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:24.539 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:24.539 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:44:24.640 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:44:24.661 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:24.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:24.662 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:44:24.763 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:44:24.783 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:24.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:24.784 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:44:24.885 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:44:24.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:24.907 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:24.908 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:44:25.008 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:44:25.031 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:25.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:25.032 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:44:25.133 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:44:25.139 INFO:tasks.rados.rados.0.smithi084.stdout:504: done (2 left) 2024-09-20T22:44:25.139 INFO:tasks.rados.rados.0.smithi084.stdout:505: copy_from oid 35 from oid 15 current snap is 44 2024-09-20T22:44:25.139 INFO:tasks.rados.rados.0.smithi084.stdout:506: write oid 49 current snap is 44 2024-09-20T22:44:25.139 INFO:tasks.rados.rados.0.smithi084.stdout:506: seq_num 213 ranges {779891=485921,1947841=685395,3383215=196676} 2024-09-20T22:44:25.145 INFO:tasks.rados.rados.0.smithi084.stdout:506: writing smithi08477203-49 from 779891 to 1265812 tid 1 2024-09-20T22:44:25.153 INFO:tasks.rados.rados.0.smithi084.stdout:506: writing smithi08477203-49 from 1947841 to 2633236 tid 2 2024-09-20T22:44:25.158 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:25.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:25.159 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:44:25.160 INFO:tasks.rados.rados.0.smithi084.stdout:506: writing smithi08477203-49 from 3383215 to 3579891 tid 3 2024-09-20T22:44:25.160 INFO:tasks.rados.rados.0.smithi084.stdout:507: write (excl) oid 23 current snap is 44 2024-09-20T22:44:25.160 INFO:tasks.rados.rados.0.smithi084.stdout:507: seq_num 214 ranges {796698=561170,2006323=545406,3126991=469707} 2024-09-20T22:44:25.166 INFO:tasks.rados.rados.0.smithi084.stdout:507: writing smithi08477203-23 from 796698 to 1357868 tid 1 2024-09-20T22:44:25.173 INFO:tasks.rados.rados.0.smithi084.stdout:507: writing smithi08477203-23 from 2006323 to 2551729 tid 2 2024-09-20T22:44:25.180 INFO:tasks.rados.rados.0.smithi084.stdout:507: writing smithi08477203-23 from 3126991 to 3596698 tid 3 2024-09-20T22:44:25.180 INFO:tasks.rados.rados.0.smithi084.stdout:508: delete oid 38 current snap is 44 2024-09-20T22:44:25.180 INFO:tasks.rados.rados.0.smithi084.stdout:501: finishing write tid 1 to smithi08477203-16 2024-09-20T22:44:25.180 INFO:tasks.rados.rados.0.smithi084.stdout:501: finishing write tid 2 to smithi08477203-16 2024-09-20T22:44:25.180 INFO:tasks.rados.rados.0.smithi084.stdout:501: finishing write tid 3 to smithi08477203-16 2024-09-20T22:44:25.180 INFO:tasks.rados.rados.0.smithi084.stdout:501: finishing write tid 4 to smithi08477203-16 2024-09-20T22:44:25.180 INFO:tasks.rados.rados.0.smithi084.stdout:501: finishing write tid 5 to smithi08477203-16 2024-09-20T22:44:25.180 INFO:tasks.rados.rados.0.smithi084.stdout:501: oid 16 updating version 0 to 125 2024-09-20T22:44:25.181 INFO:tasks.rados.rados.0.smithi084.stdout:501: oid 16 updating version 125 to 127 2024-09-20T22:44:25.181 INFO:tasks.rados.rados.0.smithi084.stdout:501: oid 16 version 127 is already newer than 126 2024-09-20T22:44:25.181 INFO:tasks.rados.rados.0.smithi084.stdout:501: oid 16 updating version 127 to 128 2024-09-20T22:44:25.181 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 16 v 128 (ObjNum 211 snap 44 seq_num 211) dirty exists 2024-09-20T22:44:25.181 INFO:tasks.rados.rados.0.smithi084.stdout:501: left oid 16 (ObjNum 211 snap 44 seq_num 211) 2024-09-20T22:44:25.181 INFO:tasks.rados.rados.0.smithi084.stdout:502: finishing write tid 1 to smithi08477203-40 2024-09-20T22:44:25.181 INFO:tasks.rados.rados.0.smithi084.stdout:502: finishing write tid 2 to smithi08477203-40 2024-09-20T22:44:25.181 INFO:tasks.rados.rados.0.smithi084.stdout:502: finishing write tid 3 to smithi08477203-40 2024-09-20T22:44:25.181 INFO:tasks.rados.rados.0.smithi084.stdout:502: finishing write tid 4 to smithi08477203-40 2024-09-20T22:44:25.181 INFO:tasks.rados.rados.0.smithi084.stdout:502: finishing write tid 5 to smithi08477203-40 2024-09-20T22:44:25.181 INFO:tasks.rados.rados.0.smithi084.stdout:502: oid 40 updating version 0 to 101 2024-09-20T22:44:25.181 INFO:tasks.rados.rados.0.smithi084.stdout:502: oid 40 updating version 101 to 103 2024-09-20T22:44:25.181 INFO:tasks.rados.rados.0.smithi084.stdout:502: oid 40 updating version 103 to 104 2024-09-20T22:44:25.181 INFO:tasks.rados.rados.0.smithi084.stdout:502: oid 40 version 104 is already newer than 102 2024-09-20T22:44:25.181 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 40 v 104 (ObjNum 212 snap 44 seq_num 212) dirty exists 2024-09-20T22:44:25.182 INFO:tasks.rados.rados.0.smithi084.stdout:502: left oid 40 (ObjNum 212 snap 44 seq_num 212) 2024-09-20T22:44:25.182 INFO:tasks.rados.rados.0.smithi084.stdout:505: finishing copy_from racing read to smithi08477203-35 2024-09-20T22:44:25.182 INFO:tasks.rados.rados.0.smithi084.stdout:506: finishing write tid 1 to smithi08477203-49 2024-09-20T22:44:25.182 INFO:tasks.rados.rados.0.smithi084.stdout:505: finishing copy_from to smithi08477203-35 2024-09-20T22:44:25.182 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 35 v 87 (ObjNum 150 snap 30 seq_num 150) dirty exists 2024-09-20T22:44:25.182 INFO:tasks.rados.rados.0.smithi084.stdout:506: finishing write tid 2 to smithi08477203-49 2024-09-20T22:44:25.182 INFO:tasks.rados.rados.0.smithi084.stdout:506: finishing write tid 3 to smithi08477203-49 2024-09-20T22:44:25.182 INFO:tasks.rados.rados.0.smithi084.stdout:506: finishing write tid 4 to smithi08477203-49 2024-09-20T22:44:25.182 INFO:tasks.rados.rados.0.smithi084.stdout:506: finishing write tid 5 to smithi08477203-49 2024-09-20T22:44:25.182 INFO:tasks.rados.rados.0.smithi084.stdout:506: oid 49 updating version 0 to 160 2024-09-20T22:44:25.182 INFO:tasks.rados.rados.0.smithi084.stdout:506: oid 49 version 160 is already newer than 157 2024-09-20T22:44:25.182 INFO:tasks.rados.rados.0.smithi084.stdout:506: oid 49 version 160 is already newer than 158 2024-09-20T22:44:25.182 INFO:tasks.rados.rados.0.smithi084.stdout:506: oid 49 version 160 is already newer than 159 2024-09-20T22:44:25.182 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 49 v 160 (ObjNum 213 snap 44 seq_num 213) dirty exists 2024-09-20T22:44:25.183 INFO:tasks.rados.rados.0.smithi084.stdout:506: left oid 49 (ObjNum 213 snap 44 seq_num 213) 2024-09-20T22:44:25.183 INFO:tasks.rados.rados.0.smithi084.stdout:507: finishing write tid 1 to smithi08477203-23 2024-09-20T22:44:25.183 INFO:tasks.rados.rados.0.smithi084.stdout:507: finishing write tid 2 to smithi08477203-23 2024-09-20T22:44:25.183 INFO:tasks.rados.rados.0.smithi084.stdout:501: done (5 left) 2024-09-20T22:44:25.183 INFO:tasks.rados.rados.0.smithi084.stdout:502: done (4 left) 2024-09-20T22:44:25.184 INFO:tasks.rados.rados.0.smithi084.stdout:505: done (3 left) 2024-09-20T22:44:25.184 INFO:tasks.rados.rados.0.smithi084.stdout:506: done (2 left) 2024-09-20T22:44:25.184 INFO:tasks.rados.rados.0.smithi084.stdout:508: done (1 left) 2024-09-20T22:44:25.184 INFO:tasks.rados.rados.0.smithi084.stdout:509: delete oid 10 current snap is 44 2024-09-20T22:44:25.184 INFO:tasks.rados.rados.0.smithi084.stdout:507: finishing write tid 3 to smithi08477203-23 2024-09-20T22:44:25.184 INFO:tasks.rados.rados.0.smithi084.stdout:507: finishing write tid 4 to smithi08477203-23 2024-09-20T22:44:25.184 INFO:tasks.rados.rados.0.smithi084.stdout:507: finishing write tid 5 to smithi08477203-23 2024-09-20T22:44:25.184 INFO:tasks.rados.rados.0.smithi084.stdout:507: oid 23 updating version 0 to 109 2024-09-20T22:44:25.184 INFO:tasks.rados.rados.0.smithi084.stdout:507: oid 23 version 109 is already newer than 108 2024-09-20T22:44:25.184 INFO:tasks.rados.rados.0.smithi084.stdout:507: oid 23 version 109 is already newer than 107 2024-09-20T22:44:25.184 INFO:tasks.rados.rados.0.smithi084.stdout:507: oid 23 updating version 109 to 110 2024-09-20T22:44:25.184 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 23 v 110 (ObjNum 214 snap 44 seq_num 214) dirty exists 2024-09-20T22:44:25.184 INFO:tasks.rados.rados.0.smithi084.stdout:507: left oid 23 (ObjNum 214 snap 44 seq_num 214) 2024-09-20T22:44:25.185 INFO:tasks.rados.rados.0.smithi084.stdout:507: done (1 left) 2024-09-20T22:44:25.185 INFO:tasks.rados.rados.0.smithi084.stdout:509: done (0 left) 2024-09-20T22:44:25.185 INFO:tasks.rados.rados.0.smithi084.stdout:510: copy_from oid 28 from oid 37 current snap is 44 2024-09-20T22:44:25.185 INFO:tasks.rados.rados.0.smithi084.stdout:511: snap_create 2024-09-20T22:44:25.191 INFO:tasks.rados.rados.0.smithi084.stdout:510: finishing copy_from racing read to smithi08477203-28 2024-09-20T22:44:25.194 INFO:tasks.rados.rados.0.smithi084.stdout:510: finishing copy_from to smithi08477203-28 2024-09-20T22:44:25.194 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 28 v 87 (ObjNum 150 snap 30 seq_num 150) dirty exists 2024-09-20T22:44:25.260 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:44:25.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:25.282 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:25.282 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:44:25.383 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:44:25.403 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:25.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:25.404 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:44:25.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:25 smithi154 ceph-mon[33264]: pgmap v178: 178 pgs: 178 active+clean; 123 MiB data, 506 MiB used, 625 GiB / 626 GiB avail; 23 MiB/s rd, 3.4 MiB/s wr, 35 op/s 2024-09-20T22:44:25.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:25 smithi154 ceph-mon[33264]: 3.1f scrub starts 2024-09-20T22:44:25.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:25 smithi154 ceph-mon[33264]: 3.1f scrub ok 2024-09-20T22:44:25.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:25 smithi154 ceph-mon[33264]: osdmap e186: 8 total, 8 up, 7 in 2024-09-20T22:44:25.506 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:44:25.527 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:25.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:25.529 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:44:25.539 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:25 smithi084 ceph-mon[40024]: pgmap v178: 178 pgs: 178 active+clean; 123 MiB data, 506 MiB used, 625 GiB / 626 GiB avail; 23 MiB/s rd, 3.4 MiB/s wr, 35 op/s 2024-09-20T22:44:25.539 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:25 smithi084 ceph-mon[40024]: 3.1f scrub starts 2024-09-20T22:44:25.540 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:25 smithi084 ceph-mon[40024]: 3.1f scrub ok 2024-09-20T22:44:25.540 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:25 smithi084 ceph-mon[40024]: osdmap e186: 8 total, 8 up, 7 in 2024-09-20T22:44:25.540 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:25 smithi084 ceph-mon[29774]: pgmap v178: 178 pgs: 178 active+clean; 123 MiB data, 506 MiB used, 625 GiB / 626 GiB avail; 23 MiB/s rd, 3.4 MiB/s wr, 35 op/s 2024-09-20T22:44:25.540 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:25 smithi084 ceph-mon[29774]: 3.1f scrub starts 2024-09-20T22:44:25.540 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:25 smithi084 ceph-mon[29774]: 3.1f scrub ok 2024-09-20T22:44:25.540 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:25 smithi084 ceph-mon[29774]: osdmap e186: 8 total, 8 up, 7 in 2024-09-20T22:44:25.629 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph tell osd.7 reset_purged_snaps_last 2024-09-20T22:44:25.630 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:44:25.653 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:25.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:25.654 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:44:25.755 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:44:25.778 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:25.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:25.778 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:44:25.879 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:44:25.902 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:25.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:25.903 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:44:26.003 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:44:26.025 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:26.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:26.026 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:44:26.127 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:44:26.139 INFO:tasks.rados.rados.0.smithi084.stdout:510: done (1 left) 2024-09-20T22:44:26.139 INFO:tasks.rados.rados.0.smithi084.stdout:511: done (0 left) 2024-09-20T22:44:26.139 INFO:tasks.rados.rados.0.smithi084.stdout:512: write (excl) oid 26 current snap is 45 2024-09-20T22:44:26.139 INFO:tasks.rados.rados.0.smithi084.stdout:512: seq_num 217 ranges {447119=786917,1665416=692733,2976554=514615} 2024-09-20T22:44:26.145 INFO:tasks.rados.rados.0.smithi084.stdout:512: writing smithi08477203-26 from 447119 to 1234036 tid 1 2024-09-20T22:44:26.152 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:26.153 INFO:tasks.rados.rados.0.smithi084.stdout:512: writing smithi08477203-26 from 1665416 to 2358149 tid 2 2024-09-20T22:44:26.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:26.153 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:44:26.159 INFO:tasks.rados.rados.0.smithi084.stdout:512: writing smithi08477203-26 from 2976554 to 3491169 tid 3 2024-09-20T22:44:26.159 INFO:tasks.rados.rados.0.smithi084.stdout:513: read oid 36 snap -1 2024-09-20T22:44:26.159 INFO:tasks.rados.rados.0.smithi084.stdout:513: expect (ObjNum 35 snap 0 seq_num 35) 2024-09-20T22:44:26.159 INFO:tasks.rados.rados.0.smithi084.stdout:514: read oid 8 snap -1 2024-09-20T22:44:26.159 INFO:tasks.rados.rados.0.smithi084.stdout:514: expect (ObjNum 164 snap 35 seq_num 164) 2024-09-20T22:44:26.160 INFO:tasks.rados.rados.0.smithi084.stdout:515: snap_remove snap 44 2024-09-20T22:44:26.254 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:44:26.277 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:26.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:26.278 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:44:26.379 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:44:26.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:26.404 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:26.404 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:44:26.457 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:44:26.485 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-20T22:44:26.506 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:44:26.551 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:26.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:26.552 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:44:26.652 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:44:26.674 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:26.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:26.675 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:44:26.743 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:44:26.776 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:44:26.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:26.800 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:26.801 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:44:26.902 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:44:26.925 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:26.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:26.926 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:44:26.980 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:44:26.980 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:44:27.004 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:44:27.004 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:44:27.004 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:44:27.027 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:44:27.028 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:44:27.029 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:44:27.050 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:27.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:27.051 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:44:27.051 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-20T22:44:27.051 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T22:44:27.074 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:44:27.074 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:44:27.091 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:44:27.091 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:44:27.107 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:44:27.107 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:44:27.125 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:44:27.125 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:44:27.149 INFO:tasks.rados.rados.0.smithi084.stdout:515: done (3 left) 2024-09-20T22:44:27.149 INFO:tasks.rados.rados.0.smithi084.stdout:516: read oid 9 snap -1 2024-09-20T22:44:27.149 INFO:tasks.rados.rados.0.smithi084.stdout:516: expect deleted 2024-09-20T22:44:27.149 INFO:tasks.rados.rados.0.smithi084.stdout:517: delete oid 32 current snap is 45 2024-09-20T22:44:27.150 INFO:tasks.rados.rados.0.smithi084.stdout:512: finishing write tid 1 to smithi08477203-26 2024-09-20T22:44:27.150 INFO:tasks.rados.rados.0.smithi084.stdout:512: finishing write tid 2 to smithi08477203-26 2024-09-20T22:44:27.151 INFO:tasks.rados.rados.0.smithi084.stdout:512: finishing write tid 3 to smithi08477203-26 2024-09-20T22:44:27.151 INFO:tasks.rados.rados.0.smithi084.stdout:512: finishing write tid 4 to smithi08477203-26 2024-09-20T22:44:27.151 INFO:tasks.rados.rados.0.smithi084.stdout:512: finishing write tid 5 to smithi08477203-26 2024-09-20T22:44:27.151 INFO:tasks.rados.rados.0.smithi084.stdout:512: oid 26 updating version 0 to 156 2024-09-20T22:44:27.151 INFO:tasks.rados.rados.0.smithi084.stdout:512: oid 26 updating version 156 to 157 2024-09-20T22:44:27.151 INFO:tasks.rados.rados.0.smithi084.stdout:512: oid 26 version 157 is already newer than 154 2024-09-20T22:44:27.151 INFO:tasks.rados.rados.0.smithi084.stdout:512: oid 26 version 157 is already newer than 155 2024-09-20T22:44:27.151 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 26 v 157 (ObjNum 217 snap 45 seq_num 217) dirty exists 2024-09-20T22:44:27.151 INFO:tasks.rados.rados.0.smithi084.stdout:512: left oid 26 (ObjNum 217 snap 45 seq_num 217) 2024-09-20T22:44:27.151 INFO:tasks.rados.rados.0.smithi084.stdout:514: expect (ObjNum 164 snap 35 seq_num 164) 2024-09-20T22:44:27.152 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:44:27.176 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:27.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:27.177 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:44:27.209 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-20T22:44:27.209 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [], 2024-09-20T22:44:27.209 INFO:teuthology.orchestra.run.smithi154.stdout: "num_ops": 0 2024-09-20T22:44:27.209 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-20T22:44:27.237 INFO:tasks.rados.rados.0.smithi084.stdout:513: expect (ObjNum 35 snap 0 seq_num 35) 2024-09-20T22:44:27.274 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:27 smithi084 ceph-mon[40024]: pgmap v181: 178 pgs: 178 active+clean; 123 MiB data, 506 MiB used, 625 GiB / 626 GiB avail; 23 MiB/s rd, 3.4 MiB/s wr, 35 op/s 2024-09-20T22:44:27.274 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:27 smithi084 ceph-mon[40024]: osdmap e187: 8 total, 8 up, 7 in 2024-09-20T22:44:27.274 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:27 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/2760056615' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:44:27.274 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:27 smithi084 ceph-mon[29774]: pgmap v181: 178 pgs: 178 active+clean; 123 MiB data, 506 MiB used, 625 GiB / 626 GiB avail; 23 MiB/s rd, 3.4 MiB/s wr, 35 op/s 2024-09-20T22:44:27.274 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:27 smithi084 ceph-mon[29774]: osdmap e187: 8 total, 8 up, 7 in 2024-09-20T22:44:27.274 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:27 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/2760056615' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:44:27.278 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:44:27.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:27.299 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:27.299 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:44:27.324 INFO:tasks.rados.rados.0.smithi084.stdout:512: done (4 left) 2024-09-20T22:44:27.324 INFO:tasks.rados.rados.0.smithi084.stdout:513: done (3 left) 2024-09-20T22:44:27.324 INFO:tasks.rados.rados.0.smithi084.stdout:514: done (2 left) 2024-09-20T22:44:27.324 INFO:tasks.rados.rados.0.smithi084.stdout:517: done (1 left) 2024-09-20T22:44:27.324 INFO:tasks.rados.rados.0.smithi084.stdout:518: rollback oid 46 current snap is 45 2024-09-20T22:44:27.325 INFO:tasks.rados.rados.0.smithi084.stdout:518: done (1 left) 2024-09-20T22:44:27.325 INFO:tasks.rados.rados.0.smithi084.stdout:519: read oid 20 snap -1 2024-09-20T22:44:27.325 INFO:tasks.rados.rados.0.smithi084.stdout:519: expect deleted 2024-09-20T22:44:27.325 INFO:tasks.rados.rados.0.smithi084.stdout:520: read oid 41 snap -1 2024-09-20T22:44:27.325 INFO:tasks.rados.rados.0.smithi084.stdout:520: expect (ObjNum 167 snap 36 seq_num 167) 2024-09-20T22:44:27.325 INFO:tasks.rados.rados.0.smithi084.stdout:521: read oid 31 snap -1 2024-09-20T22:44:27.325 INFO:tasks.rados.rados.0.smithi084.stdout:521: expect deleted 2024-09-20T22:44:27.325 INFO:tasks.rados.rados.0.smithi084.stdout:522: read oid 26 snap -1 2024-09-20T22:44:27.325 INFO:tasks.rados.rados.0.smithi084.stdout:522: expect (ObjNum 217 snap 45 seq_num 217) 2024-09-20T22:44:27.325 INFO:tasks.rados.rados.0.smithi084.stdout:523: snap_create 2024-09-20T22:44:27.328 INFO:tasks.rados.rados.0.smithi084.stdout:520: expect (ObjNum 167 snap 36 seq_num 167) 2024-09-20T22:44:27.400 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:44:27.417 INFO:tasks.rados.rados.0.smithi084.stdout:522: expect (ObjNum 217 snap 45 seq_num 217) 2024-09-20T22:44:27.422 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:27.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:27.423 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:44:27.430 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:27 smithi154 ceph-mon[33264]: pgmap v181: 178 pgs: 178 active+clean; 123 MiB data, 506 MiB used, 625 GiB / 626 GiB avail; 23 MiB/s rd, 3.4 MiB/s wr, 35 op/s 2024-09-20T22:44:27.430 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:27 smithi154 ceph-mon[33264]: osdmap e187: 8 total, 8 up, 7 in 2024-09-20T22:44:27.430 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:27 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/2760056615' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:44:27.524 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:44:27.545 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:27.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:27.546 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:44:27.647 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:44:27.669 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:27.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:27.670 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:44:27.771 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:44:27.793 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:27.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:27.794 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:44:27.895 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:44:27.917 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:27.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:27.918 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:44:28.018 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:44:28.041 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:28.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:28.042 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:44:28.058 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.4 dump_blocked_ops 2024-09-20T22:44:28.084 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-20T22:44:28.136 INFO:tasks.rados.rados.0.smithi084.stdout:516: done (5 left) 2024-09-20T22:44:28.136 INFO:tasks.rados.rados.0.smithi084.stdout:519: done (4 left) 2024-09-20T22:44:28.136 INFO:tasks.rados.rados.0.smithi084.stdout:520: done (3 left) 2024-09-20T22:44:28.136 INFO:tasks.rados.rados.0.smithi084.stdout:521: done (2 left) 2024-09-20T22:44:28.136 INFO:tasks.rados.rados.0.smithi084.stdout:522: done (1 left) 2024-09-20T22:44:28.137 INFO:tasks.rados.rados.0.smithi084.stdout:523: done (0 left) 2024-09-20T22:44:28.137 INFO:tasks.rados.rados.0.smithi084.stdout:524: read oid 7 snap 45 2024-09-20T22:44:28.137 INFO:tasks.rados.rados.0.smithi084.stdout:524: expect (ObjNum 150 snap 30 seq_num 150) 2024-09-20T22:44:28.137 INFO:tasks.rados.rados.0.smithi084.stdout:525: rollback oid 27 current snap is 46 2024-09-20T22:44:28.137 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 27 to 45 2024-09-20T22:44:28.137 INFO:tasks.rados.rados.0.smithi084.stdout:526: write (excl) oid 46 current snap is 46 2024-09-20T22:44:28.137 INFO:tasks.rados.rados.0.smithi084.stdout:526: seq_num 219 ranges {480733=537415,1782380=696402,3031400=537159} 2024-09-20T22:44:28.142 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:44:28.143 INFO:tasks.rados.rados.0.smithi084.stdout:526: writing smithi08477203-46 from 480733 to 1018148 tid 1 2024-09-20T22:44:28.156 INFO:tasks.rados.rados.0.smithi084.stdout:526: writing smithi08477203-46 from 1782380 to 2478782 tid 2 2024-09-20T22:44:28.161 INFO:tasks.rados.rados.0.smithi084.stdout:526: writing smithi08477203-46 from 3031400 to 3568559 tid 3 2024-09-20T22:44:28.161 INFO:tasks.rados.rados.0.smithi084.stdout:527: rollback oid 36 current snap is 46 2024-09-20T22:44:28.161 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 36 to 45 2024-09-20T22:44:28.162 INFO:tasks.rados.rados.0.smithi084.stdout:528: delete oid 30 current snap is 46 2024-09-20T22:44:28.162 INFO:tasks.rados.rados.0.smithi084.stdout:524: expect (ObjNum 150 snap 30 seq_num 150) 2024-09-20T22:44:28.173 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:28.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:28.175 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:44:28.275 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:44:28.294 INFO:tasks.rados.rados.0.smithi084.stdout:525: finishing rollback tid 0 to smithi08477203-27 2024-09-20T22:44:28.294 INFO:tasks.rados.rados.0.smithi084.stdout:525: finishing rollback tid 1 to smithi08477203-27 2024-09-20T22:44:28.294 INFO:tasks.rados.rados.0.smithi084.stdout:524: done (4 left) 2024-09-20T22:44:28.294 INFO:tasks.rados.rados.0.smithi084.stdout:528: done (3 left) 2024-09-20T22:44:28.294 INFO:tasks.rados.rados.0.smithi084.stdout:529: rollback oid 9 current snap is 46 2024-09-20T22:44:28.294 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 9 to 45 2024-09-20T22:44:28.294 INFO:tasks.rados.rados.0.smithi084.stdout:526: finishing write tid 1 to smithi08477203-46 2024-09-20T22:44:28.295 INFO:tasks.rados.rados.0.smithi084.stdout:527: finishing rollback tid 0 to smithi08477203-36 2024-09-20T22:44:28.295 INFO:tasks.rados.rados.0.smithi084.stdout:527: finishing rollback tid 1 to smithi08477203-36 2024-09-20T22:44:28.295 INFO:tasks.rados.rados.0.smithi084.stdout:525: finishing rollback tid 2 to smithi08477203-27 2024-09-20T22:44:28.295 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 27 v 83 (ObjNum 13 snap 0 seq_num 13) dirty exists 2024-09-20T22:44:28.295 INFO:tasks.rados.rados.0.smithi084.stdout:527: finishing rollback tid 2 to smithi08477203-36 2024-09-20T22:44:28.295 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 36 v 69 (ObjNum 35 snap 0 seq_num 35) dirty exists 2024-09-20T22:44:28.295 INFO:tasks.rados.rados.0.smithi084.stdout:526: finishing write tid 2 to smithi08477203-46 2024-09-20T22:44:28.295 INFO:tasks.rados.rados.0.smithi084.stdout:526: finishing write tid 3 to smithi08477203-46 2024-09-20T22:44:28.295 INFO:tasks.rados.rados.0.smithi084.stdout:526: finishing write tid 4 to smithi08477203-46 2024-09-20T22:44:28.295 INFO:tasks.rados.rados.0.smithi084.stdout:526: finishing write tid 5 to smithi08477203-46 2024-09-20T22:44:28.295 INFO:tasks.rados.rados.0.smithi084.stdout:526: oid 46 updating version 0 to 106 2024-09-20T22:44:28.295 INFO:tasks.rados.rados.0.smithi084.stdout:526: oid 46 version 106 is already newer than 105 2024-09-20T22:44:28.295 INFO:tasks.rados.rados.0.smithi084.stdout:526: oid 46 version 106 is already newer than 103 2024-09-20T22:44:28.295 INFO:tasks.rados.rados.0.smithi084.stdout:526: oid 46 version 106 is already newer than 104 2024-09-20T22:44:28.296 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 46 v 106 (ObjNum 219 snap 46 seq_num 219) dirty exists 2024-09-20T22:44:28.296 INFO:tasks.rados.rados.0.smithi084.stdout:526: left oid 46 (ObjNum 219 snap 46 seq_num 219) 2024-09-20T22:44:28.296 INFO:tasks.rados.rados.0.smithi084.stdout:525: done (3 left) 2024-09-20T22:44:28.296 INFO:tasks.rados.rados.0.smithi084.stdout:526: done (2 left) 2024-09-20T22:44:28.296 INFO:tasks.rados.rados.0.smithi084.stdout:527: done (1 left) 2024-09-20T22:44:28.296 INFO:tasks.rados.rados.0.smithi084.stdout:530: rollback oid 10 current snap is 46 2024-09-20T22:44:28.296 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 10 to 45 2024-09-20T22:44:28.296 INFO:tasks.rados.rados.0.smithi084.stdout:531: read oid 40 snap 45 2024-09-20T22:44:28.296 INFO:tasks.rados.rados.0.smithi084.stdout:531: expect (ObjNum 212 snap 44 seq_num 212) 2024-09-20T22:44:28.296 INFO:tasks.rados.rados.0.smithi084.stdout:532: snap_create 2024-09-20T22:44:28.296 INFO:tasks.rados.rados.0.smithi084.stdout:529: finishing rollback tid 1 to smithi08477203-9 2024-09-20T22:44:28.296 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 9 v 0 (ObjNum 129 snap 0 seq_num 2091016448) dirty dne 2024-09-20T22:44:28.297 INFO:tasks.rados.rados.0.smithi084.stdout:530: finishing rollback tid 1 to smithi08477203-10 2024-09-20T22:44:28.297 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 10 v 0 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-09-20T22:44:28.299 INFO:tasks.rados.rados.0.smithi084.stdout:531: expect (ObjNum 212 snap 44 seq_num 212) 2024-09-20T22:44:28.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:28.300 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:28.300 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:44:28.324 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:44:28.364 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:28 smithi084 ceph-mon[40024]: osdmap e188: 8 total, 8 up, 7 in 2024-09-20T22:44:28.364 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:28 smithi084 ceph-mon[40024]: osdmap e189: 8 total, 8 up, 7 in 2024-09-20T22:44:28.365 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:44:28 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: ::ffff:172.21.15.154 - - [20/Sep/2024:22:44:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-20T22:44:28.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:28 smithi084 ceph-mon[29774]: osdmap e188: 8 total, 8 up, 7 in 2024-09-20T22:44:28.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:28 smithi084 ceph-mon[29774]: osdmap e189: 8 total, 8 up, 7 in 2024-09-20T22:44:28.401 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:44:28.423 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:28.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:28.424 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:44:28.435 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:28 smithi154 ceph-mon[33264]: osdmap e188: 8 total, 8 up, 7 in 2024-09-20T22:44:28.436 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:28 smithi154 ceph-mon[33264]: osdmap e189: 8 total, 8 up, 7 in 2024-09-20T22:44:28.525 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:44:28.546 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:28.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:28.547 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:44:28.648 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:44:28.669 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:28.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:28.670 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:44:28.771 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:44:28.792 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:28.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:28.793 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:44:28.810 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-20T22:44:28.810 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [], 2024-09-20T22:44:28.810 INFO:teuthology.orchestra.run.smithi154.stdout: "complaint_time": 30, 2024-09-20T22:44:28.811 INFO:teuthology.orchestra.run.smithi154.stdout: "num_blocked_ops": 0 2024-09-20T22:44:28.811 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-20T22:44:28.894 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:44:28.916 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:28.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:28.917 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:44:28.918 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:44:29.018 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:44:29.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:29.041 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:29.041 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:44:29.142 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:44:29.166 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:29.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:29.167 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:44:29.169 INFO:tasks.rados.rados.0.smithi084.stdout:529: done (3 left) 2024-09-20T22:44:29.170 INFO:tasks.rados.rados.0.smithi084.stdout:530: done (2 left) 2024-09-20T22:44:29.170 INFO:tasks.rados.rados.0.smithi084.stdout:531: done (1 left) 2024-09-20T22:44:29.170 INFO:tasks.rados.rados.0.smithi084.stdout:532: done (0 left) 2024-09-20T22:44:29.170 INFO:tasks.rados.rados.0.smithi084.stdout:533: read oid 42 snap -1 2024-09-20T22:44:29.170 INFO:tasks.rados.rados.0.smithi084.stdout:533: expect deleted 2024-09-20T22:44:29.170 INFO:tasks.rados.rados.0.smithi084.stdout:534: write oid 3 current snap is 47 2024-09-20T22:44:29.170 INFO:tasks.rados.rados.0.smithi084.stdout:534: seq_num 221 ranges {514347=404266,1499344=700071,2686246=676056} 2024-09-20T22:44:29.174 INFO:tasks.rados.rados.0.smithi084.stdout:534: writing smithi08477203-3 from 514347 to 918613 tid 1 2024-09-20T22:44:29.181 INFO:tasks.rados.rados.0.smithi084.stdout:534: writing smithi08477203-3 from 1499344 to 2199415 tid 2 2024-09-20T22:44:29.188 INFO:tasks.rados.rados.0.smithi084.stdout:534: writing smithi08477203-3 from 2686246 to 3362302 tid 3 2024-09-20T22:44:29.188 INFO:tasks.rados.rados.0.smithi084.stdout:535: snap_create 2024-09-20T22:44:29.188 INFO:tasks.rados.rados.0.smithi084.stdout:534: finishing write tid 1 to smithi08477203-3 2024-09-20T22:44:29.188 INFO:tasks.rados.rados.0.smithi084.stdout:534: finishing write tid 2 to smithi08477203-3 2024-09-20T22:44:29.190 INFO:tasks.rados.rados.0.smithi084.stdout:534: finishing write tid 3 to smithi08477203-3 2024-09-20T22:44:29.190 INFO:tasks.rados.rados.0.smithi084.stdout:534: finishing write tid 4 to smithi08477203-3 2024-09-20T22:44:29.192 INFO:tasks.rados.rados.0.smithi084.stdout:534: finishing write tid 5 to smithi08477203-3 2024-09-20T22:44:29.192 INFO:tasks.rados.rados.0.smithi084.stdout:534: oid 3 updating version 0 to 98 2024-09-20T22:44:29.192 INFO:tasks.rados.rados.0.smithi084.stdout:534: oid 3 version 98 is already newer than 95 2024-09-20T22:44:29.192 INFO:tasks.rados.rados.0.smithi084.stdout:534: oid 3 version 98 is already newer than 97 2024-09-20T22:44:29.192 INFO:tasks.rados.rados.0.smithi084.stdout:534: oid 3 version 98 is already newer than 96 2024-09-20T22:44:29.192 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 3 v 98 (ObjNum 221 snap 47 seq_num 221) dirty exists 2024-09-20T22:44:29.193 INFO:tasks.rados.rados.0.smithi084.stdout:534: left oid 3 (ObjNum 221 snap 47 seq_num 221) 2024-09-20T22:44:29.267 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:44:29.289 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:29.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:29.290 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:44:29.391 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:44:29.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:29.412 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:29.413 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:44:29.421 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:29 smithi154 ceph-mon[33264]: pgmap v184: 178 pgs: 178 active+clean; 123 MiB data, 506 MiB used, 625 GiB / 626 GiB avail 2024-09-20T22:44:29.422 INFO:journalctl@ceph.iscsi.iscsi.a.smithi154.stdout:Sep 20 22:44:29 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-iscsi-iscsi-a[59730]: debug there is no tcmu-runner data available 2024-09-20T22:44:29.450 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.4 dump_historic_ops 2024-09-20T22:44:29.513 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:44:29.534 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:29.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:29.535 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:44:29.614 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:29 smithi084 ceph-mon[29774]: pgmap v184: 178 pgs: 178 active+clean; 123 MiB data, 506 MiB used, 625 GiB / 626 GiB avail 2024-09-20T22:44:29.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:29 smithi084 ceph-mon[40024]: pgmap v184: 178 pgs: 178 active+clean; 123 MiB data, 506 MiB used, 625 GiB / 626 GiB avail 2024-09-20T22:44:29.636 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:44:29.658 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:29.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:29.659 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:44:29.695 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:44:29.760 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:44:29.782 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:29.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:29.783 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:44:29.883 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:44:29.905 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:29.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:29.906 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:44:30.007 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:44:30.029 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:30.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:30.030 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:44:30.131 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:44:30.153 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:30.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:30.154 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:44:30.173 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is set 2024-09-20T22:44:30.175 INFO:tasks.rados.rados.0.smithi084.stdout:533: done (2 left) 2024-09-20T22:44:30.176 INFO:tasks.rados.rados.0.smithi084.stdout:534: done (1 left) 2024-09-20T22:44:30.176 INFO:tasks.rados.rados.0.smithi084.stdout:535: done (0 left) 2024-09-20T22:44:30.176 INFO:tasks.rados.rados.0.smithi084.stdout:536: snap_remove snap 46 2024-09-20T22:44:30.205 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-20T22:44:30.205 INFO:teuthology.orchestra.run.smithi154.stdout: "size": 20, 2024-09-20T22:44:30.205 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 600, 2024-09-20T22:44:30.205 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [ 2024-09-20T22:44:30.205 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.205 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.14478.0:53 5.e 5:7759931f:::notify.3:head [watch watch cookie 94843898686720] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2024-09-20T22:44:30.205 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:41:42.194704+0000", 2024-09-20T22:44:30.205 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 168.00697778700001, 2024-09-20T22:44:30.205 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0018746430000000001, 2024-09-20T22:44:30.205 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:44:30.206 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:44:30.206 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:44:30.206 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:44:30.206 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.14478", 2024-09-20T22:44:30.206 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/2994273271", 2024-09-20T22:44:30.206 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 53 2024-09-20T22:44:30.206 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.206 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:44:30.206 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.206 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:44:30.206 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:42.194704+0000", 2024-09-20T22:44:30.206 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:44:30.207 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.207 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.207 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:44:30.207 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:42.194704+0000", 2024-09-20T22:44:30.207 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:44:30.207 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.207 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.207 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:44:30.207 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:42.194705+0000", 2024-09-20T22:44:30.207 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.9800000000000003e-07 2024-09-20T22:44:30.207 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.207 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.207 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:44:30.208 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:42.194708+0000", 2024-09-20T22:44:30.208 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.2279999999999998e-06 2024-09-20T22:44:30.208 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.208 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.208 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:44:30.208 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:42.194715+0000", 2024-09-20T22:44:30.208 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.9299999999999997e-06 2024-09-20T22:44:30.208 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.208 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.208 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:44:30.208 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:42.194720+0000", 2024-09-20T22:44:30.209 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.8230000000000002e-06 2024-09-20T22:44:30.209 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.209 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.209 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:44:30.209 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:42.195053+0000", 2024-09-20T22:44:30.209 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000333312 2024-09-20T22:44:30.209 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.209 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.209 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for rw locks", 2024-09-20T22:44:30.209 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:42.195134+0000", 2024-09-20T22:44:30.209 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.1036000000000001e-05 2024-09-20T22:44:30.209 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.209 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.210 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:44:30.210 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:42.195679+0000", 2024-09-20T22:44:30.210 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00054494600000000004 2024-09-20T22:44:30.210 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.210 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.210 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:44:30.210 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:42.195732+0000", 2024-09-20T22:44:30.210 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.2605999999999997e-05 2024-09-20T22:44:30.210 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.210 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.210 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 0,5", 2024-09-20T22:44:30.210 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:42.195869+0000", 2024-09-20T22:44:30.210 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00013703899999999999 2024-09-20T22:44:30.211 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.211 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.211 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:44:30.211 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:42.196212+0000", 2024-09-20T22:44:30.211 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00034263300000000001 2024-09-20T22:44:30.211 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.211 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.211 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:44:30.211 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:42.196356+0000", 2024-09-20T22:44:30.211 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00014463799999999999 2024-09-20T22:44:30.211 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.211 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.211 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:44:30.212 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:42.196509+0000", 2024-09-20T22:44:30.212 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000152893 2024-09-20T22:44:30.212 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.212 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.212 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:44:30.212 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:42.196539+0000", 2024-09-20T22:44:30.212 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.9876e-05 2024-09-20T22:44:30.212 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.212 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.212 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:44:30.212 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:42.196579+0000", 2024-09-20T22:44:30.212 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.9984999999999997e-05 2024-09-20T22:44:30.212 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:44:30.212 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:44:30.213 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:44:30.213 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.213 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.213 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.14478.0:206 4.18 4:1e85421f:gc::gc.1:head [create,call version.check_conds in=18b,call rgw_gc.rgw_gc_queue_init in=23b,call version.set in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-09-20T22:44:30.213 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:41:45.190343+0000", 2024-09-20T22:44:30.213 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 165.011339308, 2024-09-20T22:44:30.213 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0022015450000000001, 2024-09-20T22:44:30.213 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:44:30.213 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:44:30.213 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:44:30.213 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:44:30.213 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.14478", 2024-09-20T22:44:30.213 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/2994273271", 2024-09-20T22:44:30.213 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 206 2024-09-20T22:44:30.214 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.214 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:44:30.214 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.214 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:44:30.214 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:45.190343+0000", 2024-09-20T22:44:30.214 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:44:30.214 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.214 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.214 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:44:30.214 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:45.190343+0000", 2024-09-20T22:44:30.214 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:44:30.214 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.214 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.214 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:44:30.215 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:45.190345+0000", 2024-09-20T22:44:30.215 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.843e-06 2024-09-20T22:44:30.215 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.215 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.215 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:44:30.215 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:45.190351+0000", 2024-09-20T22:44:30.215 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.7750000000000004e-06 2024-09-20T22:44:30.215 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.215 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.215 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:44:30.215 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:45.190365+0000", 2024-09-20T22:44:30.215 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4123e-05 2024-09-20T22:44:30.215 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.215 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.216 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:44:30.216 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:45.190373+0000", 2024-09-20T22:44:30.216 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.0199999999999997e-06 2024-09-20T22:44:30.216 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.216 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.216 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:44:30.216 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:45.190438+0000", 2024-09-20T22:44:30.216 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.5446999999999999e-05 2024-09-20T22:44:30.216 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.216 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.216 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:44:30.216 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:45.190562+0000", 2024-09-20T22:44:30.216 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00012417999999999999 2024-09-20T22:44:30.216 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.217 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.217 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 1,6", 2024-09-20T22:44:30.217 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:45.190826+0000", 2024-09-20T22:44:30.217 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00026341199999999998 2024-09-20T22:44:30.217 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.217 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.217 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:44:30.217 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:45.191210+0000", 2024-09-20T22:44:30.217 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00038417799999999999 2024-09-20T22:44:30.217 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.217 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.217 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:44:30.217 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:45.191549+0000", 2024-09-20T22:44:30.217 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00033966500000000001 2024-09-20T22:44:30.218 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.218 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.218 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:44:30.218 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:45.192471+0000", 2024-09-20T22:44:30.218 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00092185200000000002 2024-09-20T22:44:30.218 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.218 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.218 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:44:30.218 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:45.192520+0000", 2024-09-20T22:44:30.218 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.8600999999999998e-05 2024-09-20T22:44:30.218 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.218 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.218 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:44:30.218 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:45.192544+0000", 2024-09-20T22:44:30.219 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.4448999999999999e-05 2024-09-20T22:44:30.219 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:44:30.219 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:44:30.219 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:44:30.219 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.219 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.219 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(osd.0.0:116 7.9 e106/94 7:9eac4560:::smithi08477203-30:5 v 106'27, mlcod=105'24)", 2024-09-20T22:44:30.220 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:05.532754+0000", 2024-09-20T22:44:30.220 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 84.668928321999999, 2024-09-20T22:44:30.220 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0023975530000000002, 2024-09-20T22:44:30.220 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:44:30.220 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:44:30.220 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:44:30.220 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:44:30.220 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.221 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:44:30.221 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:05.532754+0000", 2024-09-20T22:44:30.221 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:44:30.221 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.221 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.221 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:44:30.221 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:05.532754+0000", 2024-09-20T22:44:30.221 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:44:30.221 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.221 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.221 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:44:30.221 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:05.532754+0000", 2024-09-20T22:44:30.221 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.4999999999999999e-07 2024-09-20T22:44:30.221 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.222 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.222 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:44:30.222 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:05.532758+0000", 2024-09-20T22:44:30.222 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.0389999999999998e-06 2024-09-20T22:44:30.222 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.222 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.222 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:44:30.222 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:05.532776+0000", 2024-09-20T22:44:30.222 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.8045000000000001e-05 2024-09-20T22:44:30.222 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.222 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.222 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:44:30.222 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:05.532783+0000", 2024-09-20T22:44:30.222 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.3300000000000001e-06 2024-09-20T22:44:30.223 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.223 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.223 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:44:30.223 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:05.532851+0000", 2024-09-20T22:44:30.223 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.7939000000000001e-05 2024-09-20T22:44:30.223 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.223 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.223 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:44:30.223 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:05.532865+0000", 2024-09-20T22:44:30.223 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3416e-05 2024-09-20T22:44:30.223 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.223 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.223 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:44:30.223 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:05.535106+0000", 2024-09-20T22:44:30.224 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0022412159999999999 2024-09-20T22:44:30.224 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.224 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.224 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:44:30.224 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:05.535151+0000", 2024-09-20T22:44:30.224 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.5318000000000002e-05 2024-09-20T22:44:30.224 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:44:30.224 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:44:30.224 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:44:30.224 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.224 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.224 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:324 7.8 e106/94 7:12e9c8db:::smithi08477203-11:head v 106'10, mlcod=98'8)", 2024-09-20T22:44:30.224 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:05.548487+0000", 2024-09-20T22:44:30.224 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 84.653195311000005, 2024-09-20T22:44:30.225 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.002619522, 2024-09-20T22:44:30.225 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:44:30.225 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:44:30.225 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:44:30.225 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:44:30.225 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.225 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:44:30.225 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:05.548487+0000", 2024-09-20T22:44:30.225 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:44:30.225 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.225 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.225 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:44:30.225 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:05.548487+0000", 2024-09-20T22:44:30.225 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:44:30.226 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.226 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.226 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:44:30.226 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:05.548487+0000", 2024-09-20T22:44:30.226 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.7300000000000001e-07 2024-09-20T22:44:30.226 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.226 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.226 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:44:30.226 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:05.550209+0000", 2024-09-20T22:44:30.226 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0017213370000000001 2024-09-20T22:44:30.226 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.226 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.226 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:44:30.227 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:05.550218+0000", 2024-09-20T22:44:30.227 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.9189999999999994e-06 2024-09-20T22:44:30.227 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.227 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.227 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:44:30.227 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:05.550224+0000", 2024-09-20T22:44:30.227 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.5260000000000001e-06 2024-09-20T22:44:30.227 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.227 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.227 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:44:30.227 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:05.550267+0000", 2024-09-20T22:44:30.227 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.3231e-05 2024-09-20T22:44:30.227 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.228 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.228 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:44:30.228 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:05.550283+0000", 2024-09-20T22:44:30.228 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.5421999999999999e-05 2024-09-20T22:44:30.228 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.228 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.228 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:44:30.228 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:05.551070+0000", 2024-09-20T22:44:30.228 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00078768700000000002 2024-09-20T22:44:30.228 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.228 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.228 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:44:30.228 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:05.551106+0000", 2024-09-20T22:44:30.228 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.5926999999999997e-05 2024-09-20T22:44:30.229 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:44:30.229 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:44:30.229 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:44:30.229 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.229 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.229 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:325 7.8 e106/94 7:12e9c8db:::smithi08477203-11:head v 106'11, mlcod=98'9)", 2024-09-20T22:44:30.229 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:05.554854+0000", 2024-09-20T22:44:30.229 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 84.646827569999999, 2024-09-20T22:44:30.229 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001914212, 2024-09-20T22:44:30.229 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:44:30.229 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:44:30.229 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:44:30.229 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:44:30.229 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.230 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:44:30.230 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:05.554854+0000", 2024-09-20T22:44:30.230 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:44:30.230 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.230 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.230 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:44:30.230 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:05.554854+0000", 2024-09-20T22:44:30.230 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:44:30.230 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.230 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.230 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:44:30.230 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:05.554855+0000", 2024-09-20T22:44:30.230 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.3300000000000001e-07 2024-09-20T22:44:30.231 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.231 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.231 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:44:30.231 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:05.555828+0000", 2024-09-20T22:44:30.231 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00097240100000000002 2024-09-20T22:44:30.231 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.231 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.231 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:44:30.231 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:05.555840+0000", 2024-09-20T22:44:30.231 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2711000000000001e-05 2024-09-20T22:44:30.231 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.231 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.231 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:44:30.231 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:05.555847+0000", 2024-09-20T22:44:30.232 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.4119999999999998e-06 2024-09-20T22:44:30.232 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.232 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.232 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:44:30.232 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:05.555898+0000", 2024-09-20T22:44:30.232 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.1107999999999999e-05 2024-09-20T22:44:30.232 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.232 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.232 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:44:30.232 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:05.555914+0000", 2024-09-20T22:44:30.232 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.5977999999999998e-05 2024-09-20T22:44:30.232 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.232 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.232 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:44:30.233 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:05.556720+0000", 2024-09-20T22:44:30.233 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00080616200000000002 2024-09-20T22:44:30.233 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.233 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.233 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:44:30.233 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:05.556769+0000", 2024-09-20T22:44:30.233 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.8606999999999997e-05 2024-09-20T22:44:30.233 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:44:30.233 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:44:30.233 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:44:30.233 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.233 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.233 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "MOSDScrubReserve(5.11,QREQUEST,e:107,from:6,reservation_nonce:2)", 2024-09-20T22:44:30.234 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:06.817932+0000", 2024-09-20T22:44:30.234 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 83.383750360999997, 2024-09-20T22:44:30.234 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0020011370000000001, 2024-09-20T22:44:30.234 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:44:30.234 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:44:30.234 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:44:30.234 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:44:30.234 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.234 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:44:30.234 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:06.817932+0000", 2024-09-20T22:44:30.234 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:44:30.234 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.234 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.235 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:44:30.235 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:06.817932+0000", 2024-09-20T22:44:30.235 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:44:30.235 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.235 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.235 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:44:30.235 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:06.817933+0000", 2024-09-20T22:44:30.235 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.251e-06 2024-09-20T22:44:30.235 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.235 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.235 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:44:30.235 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:06.817939+0000", 2024-09-20T22:44:30.235 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.2879999999999998e-06 2024-09-20T22:44:30.236 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.236 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.236 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:44:30.236 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:06.817953+0000", 2024-09-20T22:44:30.236 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3757999999999999e-05 2024-09-20T22:44:30.236 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.236 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.236 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:44:30.236 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:06.817968+0000", 2024-09-20T22:44:30.236 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.5184000000000001e-05 2024-09-20T22:44:30.236 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.236 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.236 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:44:30.236 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:06.818023+0000", 2024-09-20T22:44:30.237 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.4833999999999997e-05 2024-09-20T22:44:30.237 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.237 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.237 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op must wait for map", 2024-09-20T22:44:30.237 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:06.818029+0000", 2024-09-20T22:44:30.237 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.7899999999999996e-06 2024-09-20T22:44:30.237 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.237 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.237 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:44:30.237 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:06.819889+0000", 2024-09-20T22:44:30.237 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001860163 2024-09-20T22:44:30.237 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.237 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.237 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:44:30.237 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:06.819899+0000", 2024-09-20T22:44:30.238 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0451999999999999e-05 2024-09-20T22:44:30.238 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.238 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.238 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:44:30.238 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:06.819933+0000", 2024-09-20T22:44:30.238 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.3417000000000002e-05 2024-09-20T22:44:30.238 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:44:30.238 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:44:30.238 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:44:30.238 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.238 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.238 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:359 7.9 e108/94 7:91f2d47c:::smithi08477203-8:head v 108'28, mlcod=105'25)", 2024-09-20T22:44:30.238 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:07.834938+0000", 2024-09-20T22:44:30.239 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 82.366743663999998, 2024-09-20T22:44:30.239 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0054540810000000004, 2024-09-20T22:44:30.239 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:44:30.239 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:44:30.239 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:44:30.239 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:44:30.239 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.239 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:44:30.239 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:07.834938+0000", 2024-09-20T22:44:30.239 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:44:30.239 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.239 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.239 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:44:30.239 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:07.834938+0000", 2024-09-20T22:44:30.240 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:44:30.240 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.240 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.240 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:44:30.240 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:07.834939+0000", 2024-09-20T22:44:30.240 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.2399999999999998e-07 2024-09-20T22:44:30.240 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.240 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.240 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:44:30.240 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:07.837461+0000", 2024-09-20T22:44:30.240 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0025222769999999999 2024-09-20T22:44:30.240 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.240 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.240 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:44:30.241 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:07.837473+0000", 2024-09-20T22:44:30.241 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1422e-05 2024-09-20T22:44:30.241 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.241 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.241 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:44:30.241 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:07.837479+0000", 2024-09-20T22:44:30.241 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.083e-06 2024-09-20T22:44:30.241 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.241 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.241 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:44:30.241 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:07.837561+0000", 2024-09-20T22:44:30.241 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.2243999999999994e-05 2024-09-20T22:44:30.241 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.241 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.241 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:44:30.242 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:07.837586+0000", 2024-09-20T22:44:30.242 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.4638999999999999e-05 2024-09-20T22:44:30.242 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.242 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.242 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:44:30.242 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:07.840355+0000", 2024-09-20T22:44:30.242 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.002769681 2024-09-20T22:44:30.242 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.242 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.242 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:44:30.242 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:07.840392+0000", 2024-09-20T22:44:30.242 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.7211e-05 2024-09-20T22:44:30.242 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:44:30.243 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:44:30.243 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:44:30.243 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.243 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.243 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:379 7.1 7:8422f82d:::smithi08477203-22:head [write 742876~447756,stat] snapc 8=[7] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-09-20T22:44:30.243 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:10.210918+0000", 2024-09-20T22:44:30.243 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 79.990763743000002, 2024-09-20T22:44:30.243 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0030494419999999999, 2024-09-20T22:44:30.243 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:44:30.243 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:44:30.243 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:44:30.243 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:44:30.243 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:44:30.243 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:44:30.244 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 379 2024-09-20T22:44:30.244 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.244 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:44:30.244 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.244 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:44:30.244 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:10.210918+0000", 2024-09-20T22:44:30.244 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:44:30.244 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.244 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.244 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:44:30.244 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:10.210918+0000", 2024-09-20T22:44:30.244 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:44:30.244 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.244 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.244 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:44:30.245 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:10.210920+0000", 2024-09-20T22:44:30.245 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.8419999999999999e-06 2024-09-20T22:44:30.245 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.245 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.245 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:44:30.245 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:10.211774+0000", 2024-09-20T22:44:30.245 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00085377000000000001 2024-09-20T22:44:30.245 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.245 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.245 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:44:30.245 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:10.211791+0000", 2024-09-20T22:44:30.245 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.7186000000000001e-05 2024-09-20T22:44:30.245 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.245 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.246 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:44:30.246 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:10.211799+0000", 2024-09-20T22:44:30.246 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.9179999999999994e-06 2024-09-20T22:44:30.246 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.246 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.246 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:44:30.246 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:10.211870+0000", 2024-09-20T22:44:30.246 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.1271000000000003e-05 2024-09-20T22:44:30.246 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.246 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.246 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:44:30.246 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:10.211939+0000", 2024-09-20T22:44:30.246 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.8245999999999997e-05 2024-09-20T22:44:30.246 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.247 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.247 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 3,6", 2024-09-20T22:44:30.247 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:10.212067+0000", 2024-09-20T22:44:30.247 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000127995 2024-09-20T22:44:30.247 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.247 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.247 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:44:30.247 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:10.213450+0000", 2024-09-20T22:44:30.247 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001383205 2024-09-20T22:44:30.247 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.247 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.247 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:44:30.247 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:10.213457+0000", 2024-09-20T22:44:30.247 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.3159999999999999e-06 2024-09-20T22:44:30.248 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.248 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.248 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:44:30.248 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:10.213884+0000", 2024-09-20T22:44:30.248 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000426444 2024-09-20T22:44:30.248 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.248 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.248 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:44:30.248 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:10.213920+0000", 2024-09-20T22:44:30.248 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.6665000000000002e-05 2024-09-20T22:44:30.248 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.248 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.248 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:44:30.248 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:10.213968+0000", 2024-09-20T22:44:30.248 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.7583999999999997e-05 2024-09-20T22:44:30.249 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:44:30.249 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:44:30.249 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:44:30.249 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.249 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.249 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:380 7.1 7:8422f82d:::smithi08477203-22:head [write 1729599~575687] snapc 8=[7] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-09-20T22:44:30.249 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:10.216967+0000", 2024-09-20T22:44:30.249 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 79.984714612999994, 2024-09-20T22:44:30.249 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0023329090000000002, 2024-09-20T22:44:30.249 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:44:30.249 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:44:30.249 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:44:30.249 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:44:30.250 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:44:30.250 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:44:30.250 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 380 2024-09-20T22:44:30.250 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.250 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:44:30.250 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.250 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:44:30.250 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:10.216967+0000", 2024-09-20T22:44:30.250 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:44:30.250 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.250 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.250 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:44:30.250 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:10.216967+0000", 2024-09-20T22:44:30.250 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:44:30.251 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.251 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.251 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:44:30.251 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:10.216969+0000", 2024-09-20T22:44:30.251 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.111e-06 2024-09-20T22:44:30.251 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.251 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.251 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:44:30.251 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:10.217597+0000", 2024-09-20T22:44:30.251 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00062867999999999995 2024-09-20T22:44:30.251 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.251 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.251 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:44:30.252 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:10.217610+0000", 2024-09-20T22:44:30.252 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2842e-05 2024-09-20T22:44:30.252 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.252 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.252 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:44:30.252 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:10.217616+0000", 2024-09-20T22:44:30.252 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.4360000000000001e-06 2024-09-20T22:44:30.252 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.252 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.252 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:44:30.252 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:10.217655+0000", 2024-09-20T22:44:30.252 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.9032000000000002e-05 2024-09-20T22:44:30.252 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.253 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.253 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:44:30.253 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:10.217708+0000", 2024-09-20T22:44:30.253 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.3022999999999997e-05 2024-09-20T22:44:30.253 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.253 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.253 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 3,6", 2024-09-20T22:44:30.253 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:10.217792+0000", 2024-09-20T22:44:30.253 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.4425000000000006e-05 2024-09-20T22:44:30.253 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.253 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.253 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:44:30.253 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:10.218924+0000", 2024-09-20T22:44:30.253 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00113226 2024-09-20T22:44:30.254 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.254 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.254 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:44:30.254 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:10.219144+0000", 2024-09-20T22:44:30.254 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000219583 2024-09-20T22:44:30.254 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.254 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.254 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:44:30.254 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:10.219224+0000", 2024-09-20T22:44:30.254 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.0269e-05 2024-09-20T22:44:30.254 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.254 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.254 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:44:30.255 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:10.219258+0000", 2024-09-20T22:44:30.255 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.3843000000000001e-05 2024-09-20T22:44:30.255 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.255 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.255 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:44:30.255 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:10.219300+0000", 2024-09-20T22:44:30.255 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.2404999999999998e-05 2024-09-20T22:44:30.255 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:44:30.255 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:44:30.255 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:44:30.255 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.255 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.255 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:383 7.1 7:8422f82d:::smithi08477203-22:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e111)", 2024-09-20T22:44:30.256 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:10.221599+0000", 2024-09-20T22:44:30.256 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 79.980083308000005, 2024-09-20T22:44:30.256 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0016595360000000001, 2024-09-20T22:44:30.256 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:44:30.256 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:44:30.256 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:44:30.256 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:44:30.256 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:44:30.256 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:44:30.256 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 383 2024-09-20T22:44:30.256 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.256 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:44:30.256 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.256 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:44:30.257 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:10.221599+0000", 2024-09-20T22:44:30.257 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:44:30.257 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.257 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.257 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:44:30.257 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:10.221599+0000", 2024-09-20T22:44:30.257 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:44:30.257 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.257 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.257 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:44:30.257 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:10.221599+0000", 2024-09-20T22:44:30.257 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.7599999999999997e-07 2024-09-20T22:44:30.257 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.257 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.258 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:44:30.258 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:10.221602+0000", 2024-09-20T22:44:30.258 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.3879999999999998e-06 2024-09-20T22:44:30.258 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.258 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.258 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:44:30.258 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:10.221608+0000", 2024-09-20T22:44:30.258 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.9900000000000002e-06 2024-09-20T22:44:30.258 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.258 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.258 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:44:30.258 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:10.221614+0000", 2024-09-20T22:44:30.258 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.032e-06 2024-09-20T22:44:30.258 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.259 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.259 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:44:30.259 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:10.222474+0000", 2024-09-20T22:44:30.259 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00086004699999999996 2024-09-20T22:44:30.259 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.259 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.259 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for rw locks", 2024-09-20T22:44:30.259 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:10.222519+0000", 2024-09-20T22:44:30.259 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.5050999999999999e-05 2024-09-20T22:44:30.259 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.259 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.259 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:44:30.259 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:10.223040+0000", 2024-09-20T22:44:30.259 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00052144099999999998 2024-09-20T22:44:30.260 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.260 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.260 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:44:30.260 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:10.223082+0000", 2024-09-20T22:44:30.260 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.1777999999999999e-05 2024-09-20T22:44:30.260 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.260 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.260 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:44:30.260 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:10.223258+0000", 2024-09-20T22:44:30.260 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00017633300000000001 2024-09-20T22:44:30.260 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:44:30.260 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:44:30.260 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:44:30.260 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.261 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.261 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:397 7.9 e112/94 7:91f2d47c:::smithi08477203-8:head v 112'29, mlcod=106'27)", 2024-09-20T22:44:30.261 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:11.366618+0000", 2024-09-20T22:44:30.261 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 78.835063719000004, 2024-09-20T22:44:30.261 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.005929686, 2024-09-20T22:44:30.261 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:44:30.261 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:44:30.261 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:44:30.261 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:44:30.261 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.261 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:44:30.261 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:11.366618+0000", 2024-09-20T22:44:30.261 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:44:30.262 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.262 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.262 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:44:30.262 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:11.366618+0000", 2024-09-20T22:44:30.262 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:44:30.262 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.262 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.262 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:44:30.262 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:11.366619+0000", 2024-09-20T22:44:30.262 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.67e-07 2024-09-20T22:44:30.262 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.262 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.262 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:44:30.263 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:11.370298+0000", 2024-09-20T22:44:30.263 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0036796559999999999 2024-09-20T22:44:30.263 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.263 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.263 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:44:30.263 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:11.370311+0000", 2024-09-20T22:44:30.263 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3230999999999999e-05 2024-09-20T22:44:30.263 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.263 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.263 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:44:30.263 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:11.370318+0000", 2024-09-20T22:44:30.263 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.3990000000000001e-06 2024-09-20T22:44:30.263 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.263 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.264 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:44:30.264 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:11.370380+0000", 2024-09-20T22:44:30.264 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.2385999999999995e-05 2024-09-20T22:44:30.264 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.264 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.264 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:44:30.264 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:11.370403+0000", 2024-09-20T22:44:30.264 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.2707000000000001e-05 2024-09-20T22:44:30.264 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.264 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.264 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:44:30.264 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:11.372510+0000", 2024-09-20T22:44:30.264 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0021069370000000001 2024-09-20T22:44:30.264 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.264 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.265 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:44:30.265 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:11.372548+0000", 2024-09-20T22:44:30.265 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.8203e-05 2024-09-20T22:44:30.265 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:44:30.265 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:44:30.265 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:44:30.265 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.265 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.265 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:429 7.9 e115/94 7:91f2d47c:::smithi08477203-8:head v 115'31, mlcod=108'28)", 2024-09-20T22:44:30.265 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:14.204349+0000", 2024-09-20T22:44:30.265 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 75.997333514999994, 2024-09-20T22:44:30.265 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001754469, 2024-09-20T22:44:30.265 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:44:30.266 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:44:30.266 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:44:30.266 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:44:30.266 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.266 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:44:30.266 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:14.204349+0000", 2024-09-20T22:44:30.266 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:44:30.266 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.266 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.266 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:44:30.266 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:14.204349+0000", 2024-09-20T22:44:30.266 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:44:30.266 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.266 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.267 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:44:30.267 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:14.204349+0000", 2024-09-20T22:44:30.267 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.73e-07 2024-09-20T22:44:30.267 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.267 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.267 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:44:30.267 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:14.204888+0000", 2024-09-20T22:44:30.267 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00053951200000000002 2024-09-20T22:44:30.267 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.267 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.267 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:44:30.267 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:14.204902+0000", 2024-09-20T22:44:30.267 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3295000000000001e-05 2024-09-20T22:44:30.267 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.268 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.268 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:44:30.268 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:14.204909+0000", 2024-09-20T22:44:30.268 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.3930000000000002e-06 2024-09-20T22:44:30.268 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.268 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.268 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:44:30.268 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:14.204972+0000", 2024-09-20T22:44:30.268 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.2954999999999996e-05 2024-09-20T22:44:30.268 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.268 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.268 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:44:30.268 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:14.204996+0000", 2024-09-20T22:44:30.268 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.3879999999999998e-05 2024-09-20T22:44:30.269 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.269 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.269 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:44:30.269 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:14.206071+0000", 2024-09-20T22:44:30.269 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001075073 2024-09-20T22:44:30.269 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.269 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.269 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:44:30.269 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:14.206103+0000", 2024-09-20T22:44:30.269 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.2187999999999997e-05 2024-09-20T22:44:30.269 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:44:30.269 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:44:30.269 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:44:30.269 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.270 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.270 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:522 7.9 e124/94 7:9970b578:::smithi08477203-15:head v 124'40, mlcod=118'36)", 2024-09-20T22:44:30.270 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:23.137032+0000", 2024-09-20T22:44:30.270 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 67.064650334000007, 2024-09-20T22:44:30.270 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0095727380000000008, 2024-09-20T22:44:30.270 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:44:30.270 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:44:30.270 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:44:30.270 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:44:30.270 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.270 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:44:30.270 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:23.137032+0000", 2024-09-20T22:44:30.270 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:44:30.271 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.271 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.271 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:44:30.271 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:23.137032+0000", 2024-09-20T22:44:30.271 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:44:30.271 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.271 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.271 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:44:30.271 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:23.137033+0000", 2024-09-20T22:44:30.271 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.034e-06 2024-09-20T22:44:30.271 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.271 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.271 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:44:30.271 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:23.140240+0000", 2024-09-20T22:44:30.272 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0032075419999999999 2024-09-20T22:44:30.272 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.272 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.272 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:44:30.272 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:23.140261+0000", 2024-09-20T22:44:30.272 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.1155999999999999e-05 2024-09-20T22:44:30.272 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.272 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.272 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:44:30.272 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:23.140271+0000", 2024-09-20T22:44:30.272 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.6970000000000005e-06 2024-09-20T22:44:30.272 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.272 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.272 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:44:30.273 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:23.140339+0000", 2024-09-20T22:44:30.273 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.7964000000000002e-05 2024-09-20T22:44:30.273 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.273 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.273 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:44:30.273 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:23.140361+0000", 2024-09-20T22:44:30.273 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.196e-05 2024-09-20T22:44:30.273 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.273 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.273 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:44:30.273 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:23.146563+0000", 2024-09-20T22:44:30.273 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0062014879999999998 2024-09-20T22:44:30.273 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.273 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.274 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:44:30.274 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:23.146604+0000", 2024-09-20T22:44:30.274 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.1897000000000002e-05 2024-09-20T22:44:30.274 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:44:30.274 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:44:30.274 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:44:30.274 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.274 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.274 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "MOSDScrubReserve(1.2,QREQUEST,e:126,from:7,reservation_nonce:3)", 2024-09-20T22:44:30.274 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:25.203729+0000", 2024-09-20T22:44:30.274 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 64.997952601999998, 2024-09-20T22:44:30.274 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0020268320000000001, 2024-09-20T22:44:30.275 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:44:30.275 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:44:30.275 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:44:30.275 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:44:30.275 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.275 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:44:30.275 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:25.203729+0000", 2024-09-20T22:44:30.275 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:44:30.275 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.275 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.275 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:44:30.275 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:25.203729+0000", 2024-09-20T22:44:30.275 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:44:30.275 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.276 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.276 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:44:30.276 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:25.203731+0000", 2024-09-20T22:44:30.276 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1710000000000001e-06 2024-09-20T22:44:30.276 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.276 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.276 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:44:30.276 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:25.203736+0000", 2024-09-20T22:44:30.276 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.6210000000000001e-06 2024-09-20T22:44:30.276 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.276 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.276 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:44:30.276 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:25.203751+0000", 2024-09-20T22:44:30.276 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4423000000000001e-05 2024-09-20T22:44:30.276 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.277 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.277 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:44:30.277 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:25.203758+0000", 2024-09-20T22:44:30.277 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:44:30.278 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.1110000000000002e-06 2024-09-20T22:44:30.278 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.278 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.278 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:44:30.278 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:25.203819+0000", 2024-09-20T22:44:30.279 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.1476999999999994e-05 2024-09-20T22:44:30.279 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.279 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.279 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op must wait for map", 2024-09-20T22:44:30.279 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:25.203828+0000", 2024-09-20T22:44:30.279 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.5490000000000006e-06 2024-09-20T22:44:30.279 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.279 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.279 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:44:30.279 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:25.205701+0000", 2024-09-20T22:44:30.279 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0018732 2024-09-20T22:44:30.279 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.279 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.280 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:44:30.280 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:25.205718+0000", 2024-09-20T22:44:30.280 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.6790999999999999e-05 2024-09-20T22:44:30.280 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.280 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.280 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:44:30.280 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:25.205756+0000", 2024-09-20T22:44:30.280 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.8488999999999997e-05 2024-09-20T22:44:30.280 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:44:30.280 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:44:30.280 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:44:30.280 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.280 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.280 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:639 7.3 e135/94 7:cae5792a:::smithi08477203-32:head v 135'5, mlcod=98'2)", 2024-09-20T22:44:30.281 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:34.157877+0000", 2024-09-20T22:44:30.281 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 56.043804561000002, 2024-09-20T22:44:30.281 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0019217240000000001, 2024-09-20T22:44:30.281 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:44:30.281 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:44:30.281 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:44:30.281 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:44:30.281 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.281 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:44:30.281 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:34.157877+0000", 2024-09-20T22:44:30.281 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:44:30.281 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.281 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.281 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:44:30.281 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:34.157877+0000", 2024-09-20T22:44:30.282 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:44:30.282 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.282 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.282 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:44:30.282 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:34.157878+0000", 2024-09-20T22:44:30.282 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.9999999999999995e-07 2024-09-20T22:44:30.282 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.282 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.282 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:44:30.282 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:34.158752+0000", 2024-09-20T22:44:30.282 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00087380499999999996 2024-09-20T22:44:30.282 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.282 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.283 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:44:30.283 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:34.158765+0000", 2024-09-20T22:44:30.283 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2320999999999999e-05 2024-09-20T22:44:30.283 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.283 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.283 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:44:30.283 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:34.158771+0000", 2024-09-20T22:44:30.283 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.438e-06 2024-09-20T22:44:30.283 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.283 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.283 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:44:30.283 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:34.158826+0000", 2024-09-20T22:44:30.283 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.4663000000000002e-05 2024-09-20T22:44:30.283 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.283 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.284 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:44:30.284 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:34.158844+0000", 2024-09-20T22:44:30.284 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.8158000000000001e-05 2024-09-20T22:44:30.284 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.284 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.284 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:44:30.284 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:34.159770+0000", 2024-09-20T22:44:30.284 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00092567699999999997 2024-09-20T22:44:30.284 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.284 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.284 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:44:30.284 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:34.159799+0000", 2024-09-20T22:44:30.284 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.9662000000000002e-05 2024-09-20T22:44:30.284 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:44:30.285 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:44:30.285 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:44:30.285 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.285 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.285 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:745 7.1 7:8422f82d:::smithi08477203-22:head [append 0~0] snapc 18=[17] ondisk+write+known_if_redirected+supports_pool_eio e143)", 2024-09-20T22:44:30.285 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:42.616952+0000", 2024-09-20T22:44:30.285 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 47.584729942000003, 2024-09-20T22:44:30.285 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0025256609999999998, 2024-09-20T22:44:30.285 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:44:30.285 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:44:30.285 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:44:30.285 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:44:30.285 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:44:30.285 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:44:30.286 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 745 2024-09-20T22:44:30.286 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.286 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:44:30.286 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.286 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:44:30.286 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.616952+0000", 2024-09-20T22:44:30.286 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:44:30.286 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.286 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.286 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:44:30.286 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.616952+0000", 2024-09-20T22:44:30.286 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:44:30.286 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.286 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.287 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:44:30.287 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.616954+0000", 2024-09-20T22:44:30.287 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.449e-06 2024-09-20T22:44:30.287 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.287 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.287 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:44:30.287 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.616961+0000", 2024-09-20T22:44:30.287 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.1250000000000004e-06 2024-09-20T22:44:30.287 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.287 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.287 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:44:30.287 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.616975+0000", 2024-09-20T22:44:30.287 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4669e-05 2024-09-20T22:44:30.287 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.288 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.288 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:44:30.288 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.616983+0000", 2024-09-20T22:44:30.288 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.3769999999999998e-06 2024-09-20T22:44:30.288 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.288 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.288 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:44:30.288 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.617040+0000", 2024-09-20T22:44:30.288 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.7147e-05 2024-09-20T22:44:30.288 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.288 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.288 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:44:30.288 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.617152+0000", 2024-09-20T22:44:30.288 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000111809 2024-09-20T22:44:30.289 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.289 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.289 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 3,6", 2024-09-20T22:44:30.289 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.617272+0000", 2024-09-20T22:44:30.289 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000120677 2024-09-20T22:44:30.289 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.289 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.289 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:44:30.289 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.619251+0000", 2024-09-20T22:44:30.289 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001978914 2024-09-20T22:44:30.289 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.289 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.289 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:44:30.289 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.619298+0000", 2024-09-20T22:44:30.290 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.7067000000000002e-05 2024-09-20T22:44:30.290 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.290 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.290 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:44:30.290 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.619391+0000", 2024-09-20T22:44:30.290 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.2793000000000004e-05 2024-09-20T22:44:30.290 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.290 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.290 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:44:30.290 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.619425+0000", 2024-09-20T22:44:30.290 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.3626e-05 2024-09-20T22:44:30.290 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.290 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.290 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:44:30.291 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.619478+0000", 2024-09-20T22:44:30.291 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.3007999999999998e-05 2024-09-20T22:44:30.291 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:44:30.291 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:44:30.291 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:44:30.291 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.291 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.291 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:746 7.1 7:8422f82d:::smithi08477203-22:head [rollback 17] snapc 18=[17] ondisk+write+known_if_redirected+supports_pool_eio e143)", 2024-09-20T22:44:30.291 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:42.617006+0000", 2024-09-20T22:44:30.291 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 47.584676414999997, 2024-09-20T22:44:30.291 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0025901660000000001, 2024-09-20T22:44:30.291 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:44:30.291 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:44:30.291 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:44:30.291 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:44:30.292 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:44:30.292 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:44:30.292 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 746 2024-09-20T22:44:30.292 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.292 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:44:30.292 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.292 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:44:30.292 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.617006+0000", 2024-09-20T22:44:30.292 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:44:30.292 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.292 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.292 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:44:30.292 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.617006+0000", 2024-09-20T22:44:30.293 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:44:30.293 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.293 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.293 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:44:30.293 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.617006+0000", 2024-09-20T22:44:30.293 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.4499999999999997e-07 2024-09-20T22:44:30.293 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.293 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.293 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:44:30.293 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.617009+0000", 2024-09-20T22:44:30.293 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.0120000000000001e-06 2024-09-20T22:44:30.293 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.293 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.293 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:44:30.294 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.617022+0000", 2024-09-20T22:44:30.294 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2468e-05 2024-09-20T22:44:30.294 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.294 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.294 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:44:30.294 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.617028+0000", 2024-09-20T22:44:30.294 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.9190000000000004e-06 2024-09-20T22:44:30.294 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.294 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.294 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:44:30.294 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.617952+0000", 2024-09-20T22:44:30.294 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000924488 2024-09-20T22:44:30.294 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.294 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.295 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:44:30.295 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.618027+0000", 2024-09-20T22:44:30.295 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.5164000000000001e-05 2024-09-20T22:44:30.295 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.295 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.295 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 3,6", 2024-09-20T22:44:30.295 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.618216+0000", 2024-09-20T22:44:30.295 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00018830799999999999 2024-09-20T22:44:30.295 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.295 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.295 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:44:30.295 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.619377+0000", 2024-09-20T22:44:30.295 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0011610909999999999 2024-09-20T22:44:30.295 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.296 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.296 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:44:30.296 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.619527+0000", 2024-09-20T22:44:30.296 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00015048500000000001 2024-09-20T22:44:30.296 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.296 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.296 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:44:30.296 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.619544+0000", 2024-09-20T22:44:30.296 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.6691e-05 2024-09-20T22:44:30.296 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.296 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.296 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:44:30.296 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.619573+0000", 2024-09-20T22:44:30.296 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.8895e-05 2024-09-20T22:44:30.297 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.297 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.297 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:44:30.297 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.619596+0000", 2024-09-20T22:44:30.297 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.3099999999999999e-05 2024-09-20T22:44:30.297 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:44:30.297 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:44:30.297 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:44:30.297 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.297 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.297 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:747 7.1 7:8422f82d:::smithi08477203-22:head [append 0~0] snapc 18=[17] ondisk+write+known_if_redirected+supports_pool_eio e143)", 2024-09-20T22:44:30.297 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:42.617038+0000", 2024-09-20T22:44:30.297 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 47.584644205000004, 2024-09-20T22:44:30.297 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0026729190000000002, 2024-09-20T22:44:30.298 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:44:30.298 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:44:30.298 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:44:30.298 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:44:30.298 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:44:30.298 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:44:30.298 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 747 2024-09-20T22:44:30.298 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.298 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:44:30.298 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.298 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:44:30.298 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.617038+0000", 2024-09-20T22:44:30.298 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:44:30.298 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.299 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.299 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:44:30.299 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.617038+0000", 2024-09-20T22:44:30.299 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:44:30.299 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.299 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.299 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:44:30.299 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.617038+0000", 2024-09-20T22:44:30.299 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.8500000000000002e-07 2024-09-20T22:44:30.299 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.299 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.299 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:44:30.299 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.617041+0000", 2024-09-20T22:44:30.299 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.6800000000000002e-06 2024-09-20T22:44:30.299 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.300 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.300 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:44:30.300 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.617047+0000", 2024-09-20T22:44:30.300 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.7699999999999998e-06 2024-09-20T22:44:30.300 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.300 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.300 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:44:30.300 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.617052+0000", 2024-09-20T22:44:30.300 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.9069999999999998e-06 2024-09-20T22:44:30.300 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.300 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.300 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:44:30.300 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.618852+0000", 2024-09-20T22:44:30.300 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001800458 2024-09-20T22:44:30.301 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.301 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.301 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:44:30.301 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.618940+0000", 2024-09-20T22:44:30.301 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.8208000000000007e-05 2024-09-20T22:44:30.301 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.301 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.301 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 3,6", 2024-09-20T22:44:30.301 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.619042+0000", 2024-09-20T22:44:30.301 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000101364 2024-09-20T22:44:30.301 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.301 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.301 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:44:30.301 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.619601+0000", 2024-09-20T22:44:30.302 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00055948900000000004 2024-09-20T22:44:30.302 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.302 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.302 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:44:30.302 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.619634+0000", 2024-09-20T22:44:30.302 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.2325e-05 2024-09-20T22:44:30.302 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.302 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.302 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:44:30.302 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.619661+0000", 2024-09-20T22:44:30.302 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.7169999999999999e-05 2024-09-20T22:44:30.302 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.302 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.302 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:44:30.302 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.619690+0000", 2024-09-20T22:44:30.303 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.8903000000000001e-05 2024-09-20T22:44:30.303 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.303 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.303 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:44:30.303 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.619711+0000", 2024-09-20T22:44:30.303 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.1160000000000001e-05 2024-09-20T22:44:30.303 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:44:30.303 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:44:30.303 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:44:30.303 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.303 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.303 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:769 7.4 e143/137 7:24b1a612:::smithi08477203-43:head v 143'35, mlcod=140'32)", 2024-09-20T22:44:30.303 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:43.082896+0000", 2024-09-20T22:44:30.303 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 47.118786360000001, 2024-09-20T22:44:30.304 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0016841919999999999, 2024-09-20T22:44:30.304 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:44:30.304 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:44:30.304 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:44:30.304 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:44:30.304 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.304 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:44:30.304 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:43.082896+0000", 2024-09-20T22:44:30.304 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:44:30.304 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.304 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.304 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:44:30.304 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:43.082896+0000", 2024-09-20T22:44:30.304 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:44:30.304 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.305 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.305 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:44:30.305 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:43.082898+0000", 2024-09-20T22:44:30.305 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.12e-06 2024-09-20T22:44:30.305 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.305 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.305 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:44:30.305 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:43.083692+0000", 2024-09-20T22:44:30.305 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00079414999999999998 2024-09-20T22:44:30.305 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.305 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.305 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:44:30.305 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:43.083715+0000", 2024-09-20T22:44:30.305 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.3456999999999999e-05 2024-09-20T22:44:30.305 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.306 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.306 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:44:30.306 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:43.083727+0000", 2024-09-20T22:44:30.306 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1488000000000001e-05 2024-09-20T22:44:30.306 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.306 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.306 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:44:30.306 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:43.083799+0000", 2024-09-20T22:44:30.306 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.2434000000000005e-05 2024-09-20T22:44:30.306 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.306 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.306 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:44:30.306 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:43.083832+0000", 2024-09-20T22:44:30.307 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.3093000000000003e-05 2024-09-20T22:44:30.307 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.307 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.307 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:44:30.307 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:43.084544+0000", 2024-09-20T22:44:30.307 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00071157800000000004 2024-09-20T22:44:30.307 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.307 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.307 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:44:30.307 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:43.084580+0000", 2024-09-20T22:44:30.307 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.5871999999999998e-05 2024-09-20T22:44:30.307 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:44:30.307 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:44:30.307 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:44:30.307 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.308 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.308 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(osd.2.18:3 7.1 7:84a866f5:::smithi08477203-27:head [assert-version v48,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e156)", 2024-09-20T22:44:30.308 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:55.627602+0000", 2024-09-20T22:44:30.308 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 34.574079683000001, 2024-09-20T22:44:30.308 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0032967339999999999, 2024-09-20T22:44:30.308 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:44:30.308 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:44:30.308 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:44:30.308 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:44:30.308 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.308 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:44:30.308 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:55.627602+0000", 2024-09-20T22:44:30.308 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:44:30.308 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.308 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.309 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:44:30.309 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:55.627602+0000", 2024-09-20T22:44:30.309 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:44:30.309 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.309 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.309 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:44:30.309 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:55.627604+0000", 2024-09-20T22:44:30.309 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.703e-06 2024-09-20T22:44:30.309 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.309 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.309 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:44:30.309 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:55.627622+0000", 2024-09-20T22:44:30.309 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.7683999999999999e-05 2024-09-20T22:44:30.309 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.310 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.310 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:44:30.310 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:55.627632+0000", 2024-09-20T22:44:30.310 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0719e-05 2024-09-20T22:44:30.310 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.310 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.310 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:44:30.310 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:55.627644+0000", 2024-09-20T22:44:30.310 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1097000000000001e-05 2024-09-20T22:44:30.310 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.310 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.310 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:44:30.310 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:55.627739+0000", 2024-09-20T22:44:30.310 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.5031000000000005e-05 2024-09-20T22:44:30.311 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.311 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.311 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:44:30.311 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:55.627869+0000", 2024-09-20T22:44:30.311 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00012995500000000001 2024-09-20T22:44:30.311 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:30.311 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:30.311 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:44:30.311 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:55.630899+0000", 2024-09-20T22:44:30.311 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0030305449999999999 2024-09-20T22:44:30.311 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:44:30.311 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:44:30.311 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:44:30.311 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:44:30.311 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:44:30.312 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-20T22:44:30.332 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:30.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:30.333 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:44:30.434 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:44:30.446 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-20T22:44:30.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:30.458 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:30.458 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:44:30.477 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:30 smithi084 ceph-mon[40024]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:44:30.477 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:30 smithi084 ceph-mon[40024]: osdmap e190: 8 total, 8 up, 7 in 2024-09-20T22:44:30.477 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:30 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/1331920843' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:44:30.477 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:30 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:44:30.477 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:30 smithi084 ceph-mon[29774]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:44:30.477 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:30 smithi084 ceph-mon[29774]: osdmap e190: 8 total, 8 up, 7 in 2024-09-20T22:44:30.478 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:30 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/1331920843' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:44:30.478 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:30 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:44:30.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:30 smithi154 ceph-mon[33264]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:44:30.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:30 smithi154 ceph-mon[33264]: osdmap e190: 8 total, 8 up, 7 in 2024-09-20T22:44:30.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:30 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/1331920843' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:44:30.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:30 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:44:30.559 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:44:30.580 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:30.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:30.582 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:44:30.682 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:44:30.703 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:30.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:30.704 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:44:30.805 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:44:30.825 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:30.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:30.827 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:44:30.927 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:44:30.948 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:30.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:30.949 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:44:31.018 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-20T22:44:31.050 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:44:31.082 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:31.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:31.083 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:44:31.138 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:44:31.180 INFO:tasks.rados.rados.0.smithi084.stdout:536: done (0 left) 2024-09-20T22:44:31.180 INFO:tasks.rados.rados.0.smithi084.stdout:537: write (excl) oid 28 current snap is 48 2024-09-20T22:44:31.180 INFO:tasks.rados.rados.0.smithi084.stdout:537: seq_num 222 ranges {531154=479515,1557826=443729,2713669=487328,3679295=51859} 2024-09-20T22:44:31.184 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:44:31.187 INFO:tasks.rados.rados.0.smithi084.stdout:537: writing smithi08477203-28 from 531154 to 1010669 tid 1 2024-09-20T22:44:31.195 INFO:tasks.rados.rados.0.smithi084.stdout:537: writing smithi08477203-28 from 1557826 to 2001555 tid 2 2024-09-20T22:44:31.199 INFO:tasks.rados.rados.0.smithi084.stdout:537: writing smithi08477203-28 from 2713669 to 3200997 tid 3 2024-09-20T22:44:31.202 INFO:tasks.rados.rados.0.smithi084.stdout:537: writing smithi08477203-28 from 3679295 to 3731154 tid 4 2024-09-20T22:44:31.202 INFO:tasks.rados.rados.0.smithi084.stdout:538: delete oid 25 current snap is 48 2024-09-20T22:44:31.202 INFO:tasks.rados.rados.0.smithi084.stdout:537: finishing write tid 1 to smithi08477203-28 2024-09-20T22:44:31.203 INFO:tasks.rados.rados.0.smithi084.stdout:537: finishing write tid 2 to smithi08477203-28 2024-09-20T22:44:31.203 INFO:tasks.rados.rados.0.smithi084.stdout:537: finishing write tid 3 to smithi08477203-28 2024-09-20T22:44:31.208 INFO:tasks.rados.rados.0.smithi084.stdout:537: finishing write tid 4 to smithi08477203-28 2024-09-20T22:44:31.208 INFO:tasks.rados.rados.0.smithi084.stdout:537: finishing write tid 5 to smithi08477203-28 2024-09-20T22:44:31.208 INFO:tasks.rados.rados.0.smithi084.stdout:537: finishing write tid 6 to smithi08477203-28 2024-09-20T22:44:31.208 INFO:tasks.rados.rados.0.smithi084.stdout:537: oid 28 updating version 0 to 212 2024-09-20T22:44:31.208 INFO:tasks.rados.rados.0.smithi084.stdout:537: oid 28 version 212 is already newer than 210 2024-09-20T22:44:31.208 INFO:tasks.rados.rados.0.smithi084.stdout:537: oid 28 updating version 212 to 214 2024-09-20T22:44:31.208 INFO:tasks.rados.rados.0.smithi084.stdout:537: oid 28 version 214 is already newer than 211 2024-09-20T22:44:31.209 INFO:tasks.rados.rados.0.smithi084.stdout:537: oid 28 version 214 is already newer than 213 2024-09-20T22:44:31.209 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 28 v 214 (ObjNum 222 snap 48 seq_num 222) dirty exists 2024-09-20T22:44:31.209 INFO:tasks.rados.rados.0.smithi084.stdout:537: left oid 28 (ObjNum 222 snap 48 seq_num 222) 2024-09-20T22:44:31.209 INFO:tasks.rados.rados.0.smithi084.stdout:537: done (1 left) 2024-09-20T22:44:31.209 INFO:tasks.rados.rados.0.smithi084.stdout:538: done (0 left) 2024-09-20T22:44:31.209 INFO:tasks.rados.rados.0.smithi084.stdout:539: snap_remove snap 47 2024-09-20T22:44:31.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:31.211 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:31.212 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:44:31.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:31 smithi154 ceph-mon[33264]: pgmap v187: 178 pgs: 178 active+clean; 116 MiB data, 501 MiB used, 625 GiB / 626 GiB avail; 19 MiB/s rd, 2.8 MiB/s wr, 24 op/s 2024-09-20T22:44:31.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:31 smithi154 ceph-mon[33264]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:44:31.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:31 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T22:44:31.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:31 smithi154 ceph-mon[33264]: osdmap e191: 8 total, 8 up, 7 in 2024-09-20T22:44:31.283 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:44:31.312 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:44:31.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:31.335 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:31.335 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:44:31.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:31 smithi084 ceph-mon[29774]: pgmap v187: 178 pgs: 178 active+clean; 116 MiB data, 501 MiB used, 625 GiB / 626 GiB avail; 19 MiB/s rd, 2.8 MiB/s wr, 24 op/s 2024-09-20T22:44:31.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:31 smithi084 ceph-mon[29774]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:44:31.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:31 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T22:44:31.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:31 smithi084 ceph-mon[29774]: osdmap e191: 8 total, 8 up, 7 in 2024-09-20T22:44:31.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:31 smithi084 ceph-mon[40024]: pgmap v187: 178 pgs: 178 active+clean; 116 MiB data, 501 MiB used, 625 GiB / 626 GiB avail; 19 MiB/s rd, 2.8 MiB/s wr, 24 op/s 2024-09-20T22:44:31.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:31 smithi084 ceph-mon[40024]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:44:31.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:31 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T22:44:31.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:31 smithi084 ceph-mon[40024]: osdmap e191: 8 total, 8 up, 7 in 2024-09-20T22:44:31.436 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:44:31.457 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:31.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:31.458 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:44:31.558 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:44:31.580 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:31.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:31.581 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:44:31.587 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:44:31.588 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:44:31.604 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:44:31.604 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:44:31.619 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:44:31.619 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:44:31.635 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-20T22:44:31.635 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T22:44:31.656 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:44:31.657 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:44:31.673 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:44:31.673 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:44:31.682 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:44:31.691 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:44:31.691 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:44:31.703 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:31.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:31.704 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:44:31.707 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:44:31.707 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:44:31.788 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-20T22:44:31.788 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:44:31.805 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:44:31.826 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:31.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:31.827 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:44:31.928 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:44:31.949 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:31.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:31.950 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:44:32.051 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:44:32.073 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:32.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:32.073 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:44:32.174 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:44:32.183 INFO:tasks.rados.rados.0.smithi084.stdout:539: done (0 left) 2024-09-20T22:44:32.183 INFO:tasks.rados.rados.0.smithi084.stdout:540: write oid 6 current snap is 48 2024-09-20T22:44:32.183 INFO:tasks.rados.rados.0.smithi084.stdout:540: seq_num 224 ranges {564768=630013,1674790=447398,2768515=626225} 2024-09-20T22:44:32.189 INFO:tasks.rados.rados.0.smithi084.stdout:540: writing smithi08477203-6 from 564768 to 1194781 tid 1 2024-09-20T22:44:32.194 INFO:tasks.rados.rados.0.smithi084.stdout:540: writing smithi08477203-6 from 1674790 to 2122188 tid 2 2024-09-20T22:44:32.199 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:32.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:32.200 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:44:32.202 INFO:tasks.rados.rados.0.smithi084.stdout:540: writing smithi08477203-6 from 2768515 to 3394740 tid 3 2024-09-20T22:44:32.202 INFO:tasks.rados.rados.0.smithi084.stdout:541: rollback oid 35 current snap is 48 2024-09-20T22:44:32.203 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 35 to 45 2024-09-20T22:44:32.203 INFO:tasks.rados.rados.0.smithi084.stdout:540: finishing write tid 1 to smithi08477203-6 2024-09-20T22:44:32.203 INFO:tasks.rados.rados.0.smithi084.stdout:540: finishing write tid 2 to smithi08477203-6 2024-09-20T22:44:32.203 INFO:tasks.rados.rados.0.smithi084.stdout:542: copy_from oid 22 from oid 26 current snap is 48 2024-09-20T22:44:32.203 INFO:tasks.rados.rados.0.smithi084.stdout:543: write oid 1 current snap is 48 2024-09-20T22:44:32.203 INFO:tasks.rados.rados.0.smithi084.stdout:543: seq_num 225 ranges {581575=705262,1733272=707409,3195938=437497} 2024-09-20T22:44:32.210 INFO:tasks.rados.rados.0.smithi084.stdout:543: writing smithi08477203-1 from 581575 to 1286837 tid 1 2024-09-20T22:44:32.217 INFO:tasks.rados.rados.0.smithi084.stdout:543: writing smithi08477203-1 from 1733272 to 2440681 tid 2 2024-09-20T22:44:32.224 INFO:tasks.rados.rados.0.smithi084.stdout:543: writing smithi08477203-1 from 3195938 to 3633435 tid 3 2024-09-20T22:44:32.224 INFO:tasks.rados.rados.0.smithi084.stdout:544: write (excl) oid 40 current snap is 48 2024-09-20T22:44:32.224 INFO:tasks.rados.rados.0.smithi084.stdout:540: finishing write tid 3 to smithi08477203-6 2024-09-20T22:44:32.224 INFO:tasks.rados.rados.0.smithi084.stdout:544: seq_num 226 ranges {598382=780511,2075401=451067,3107008=648769} 2024-09-20T22:44:32.231 INFO:tasks.rados.rados.0.smithi084.stdout:544: writing smithi08477203-40 from 598382 to 1378893 tid 1 2024-09-20T22:44:32.237 INFO:tasks.rados.rados.0.smithi084.stdout:544: writing smithi08477203-40 from 2075401 to 2526468 tid 2 2024-09-20T22:44:32.244 INFO:tasks.rados.rados.0.smithi084.stdout:544: writing smithi08477203-40 from 3107008 to 3755777 tid 3 2024-09-20T22:44:32.244 INFO:tasks.rados.rados.0.smithi084.stdout:545: write (excl) oid 21 current snap is 48 2024-09-20T22:44:32.244 INFO:tasks.rados.rados.0.smithi084.stdout:545: seq_num 227 ranges {615189=455760,1733883=711078,3134431=460041} 2024-09-20T22:44:32.250 INFO:tasks.rados.rados.0.smithi084.stdout:545: writing smithi08477203-21 from 615189 to 1070949 tid 1 2024-09-20T22:44:32.258 INFO:tasks.rados.rados.0.smithi084.stdout:545: writing smithi08477203-21 from 1733883 to 2444961 tid 2 2024-09-20T22:44:32.265 INFO:tasks.rados.rados.0.smithi084.stdout:545: writing smithi08477203-21 from 3134431 to 3594472 tid 3 2024-09-20T22:44:32.265 INFO:tasks.rados.rados.0.smithi084.stdout:546: delete oid 13 current snap is 48 2024-09-20T22:44:32.265 INFO:tasks.rados.rados.0.smithi084.stdout:540: finishing write tid 4 to smithi08477203-6 2024-09-20T22:44:32.266 INFO:tasks.rados.rados.0.smithi084.stdout:541: finishing rollback tid 0 to smithi08477203-35 2024-09-20T22:44:32.266 INFO:tasks.rados.rados.0.smithi084.stdout:541: finishing rollback tid 1 to smithi08477203-35 2024-09-20T22:44:32.266 INFO:tasks.rados.rados.0.smithi084.stdout:540: finishing write tid 5 to smithi08477203-6 2024-09-20T22:44:32.266 INFO:tasks.rados.rados.0.smithi084.stdout:540: oid 6 updating version 0 to 76 2024-09-20T22:44:32.266 INFO:tasks.rados.rados.0.smithi084.stdout:540: oid 6 updating version 76 to 77 2024-09-20T22:44:32.266 INFO:tasks.rados.rados.0.smithi084.stdout:540: oid 6 updating version 77 to 78 2024-09-20T22:44:32.266 INFO:tasks.rados.rados.0.smithi084.stdout:540: oid 6 updating version 78 to 79 2024-09-20T22:44:32.266 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 6 v 79 (ObjNum 224 snap 48 seq_num 224) dirty exists 2024-09-20T22:44:32.266 INFO:tasks.rados.rados.0.smithi084.stdout:540: left oid 6 (ObjNum 224 snap 48 seq_num 224) 2024-09-20T22:44:32.266 INFO:tasks.rados.rados.0.smithi084.stdout:541: finishing rollback tid 2 to smithi08477203-35 2024-09-20T22:44:32.266 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 35 v 90 (ObjNum 150 snap 30 seq_num 150) dirty exists 2024-09-20T22:44:32.267 INFO:tasks.rados.rados.0.smithi084.stdout:542: finishing copy_from racing read to smithi08477203-22 2024-09-20T22:44:32.267 INFO:tasks.rados.rados.0.smithi084.stdout:543: finishing write tid 1 to smithi08477203-1 2024-09-20T22:44:32.267 INFO:tasks.rados.rados.0.smithi084.stdout:542: finishing copy_from to smithi08477203-22 2024-09-20T22:44:32.267 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 22 v 157 (ObjNum 217 snap 45 seq_num 217) dirty exists 2024-09-20T22:44:32.267 INFO:tasks.rados.rados.0.smithi084.stdout:543: finishing write tid 2 to smithi08477203-1 2024-09-20T22:44:32.267 INFO:tasks.rados.rados.0.smithi084.stdout:543: finishing write tid 3 to smithi08477203-1 2024-09-20T22:44:32.267 INFO:tasks.rados.rados.0.smithi084.stdout:543: finishing write tid 4 to smithi08477203-1 2024-09-20T22:44:32.267 INFO:tasks.rados.rados.0.smithi084.stdout:543: finishing write tid 5 to smithi08477203-1 2024-09-20T22:44:32.267 INFO:tasks.rados.rados.0.smithi084.stdout:543: oid 1 updating version 0 to 110 2024-09-20T22:44:32.267 INFO:tasks.rados.rados.0.smithi084.stdout:543: oid 1 updating version 110 to 112 2024-09-20T22:44:32.267 INFO:tasks.rados.rados.0.smithi084.stdout:543: oid 1 version 112 is already newer than 111 2024-09-20T22:44:32.267 INFO:tasks.rados.rados.0.smithi084.stdout:543: oid 1 updating version 112 to 113 2024-09-20T22:44:32.268 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 1 v 113 (ObjNum 225 snap 48 seq_num 225) dirty exists 2024-09-20T22:44:32.268 INFO:tasks.rados.rados.0.smithi084.stdout:543: left oid 1 (ObjNum 225 snap 48 seq_num 225) 2024-09-20T22:44:32.268 INFO:tasks.rados.rados.0.smithi084.stdout:544: finishing write tid 1 to smithi08477203-40 2024-09-20T22:44:32.268 INFO:tasks.rados.rados.0.smithi084.stdout:544: finishing write tid 2 to smithi08477203-40 2024-09-20T22:44:32.268 INFO:tasks.rados.rados.0.smithi084.stdout:544: finishing write tid 3 to smithi08477203-40 2024-09-20T22:44:32.268 INFO:tasks.rados.rados.0.smithi084.stdout:544: finishing write tid 4 to smithi08477203-40 2024-09-20T22:44:32.268 INFO:tasks.rados.rados.0.smithi084.stdout:544: finishing write tid 5 to smithi08477203-40 2024-09-20T22:44:32.268 INFO:tasks.rados.rados.0.smithi084.stdout:544: oid 40 updating version 0 to 116 2024-09-20T22:44:32.268 INFO:tasks.rados.rados.0.smithi084.stdout:544: oid 40 version 116 is already newer than 114 2024-09-20T22:44:32.268 INFO:tasks.rados.rados.0.smithi084.stdout:544: oid 40 updating version 116 to 117 2024-09-20T22:44:32.268 INFO:tasks.rados.rados.0.smithi084.stdout:544: oid 40 version 117 is already newer than 115 2024-09-20T22:44:32.269 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 40 v 117 (ObjNum 226 snap 48 seq_num 226) dirty exists 2024-09-20T22:44:32.269 INFO:tasks.rados.rados.0.smithi084.stdout:544: left oid 40 (ObjNum 226 snap 48 seq_num 226) 2024-09-20T22:44:32.269 INFO:tasks.rados.rados.0.smithi084.stdout:545: finishing write tid 1 to smithi08477203-21 2024-09-20T22:44:32.269 INFO:tasks.rados.rados.0.smithi084.stdout:545: finishing write tid 2 to smithi08477203-21 2024-09-20T22:44:32.269 INFO:tasks.rados.rados.0.smithi084.stdout:540: done (6 left) 2024-09-20T22:44:32.269 INFO:tasks.rados.rados.0.smithi084.stdout:541: done (5 left) 2024-09-20T22:44:32.270 INFO:tasks.rados.rados.0.smithi084.stdout:542: done (4 left) 2024-09-20T22:44:32.270 INFO:tasks.rados.rados.0.smithi084.stdout:543: done (3 left) 2024-09-20T22:44:32.270 INFO:tasks.rados.rados.0.smithi084.stdout:544: done (2 left) 2024-09-20T22:44:32.270 INFO:tasks.rados.rados.0.smithi084.stdout:546: done (1 left) 2024-09-20T22:44:32.270 INFO:tasks.rados.rados.0.smithi084.stdout:547: copy_from oid 27 from oid 46 current snap is 48 2024-09-20T22:44:32.270 INFO:tasks.rados.rados.0.smithi084.stdout:548: snap_remove snap 45 2024-09-20T22:44:32.300 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:44:32.321 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:32.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:32.322 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:44:32.364 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:32 smithi084 ceph-mon[29774]: purged_snaps scrub starts 2024-09-20T22:44:32.364 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:32 smithi084 ceph-mon[29774]: purged_snaps scrub ok 2024-09-20T22:44:32.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:32 smithi084 ceph-mon[29774]: osdmap e192: 8 total, 8 up, 7 in 2024-09-20T22:44:32.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:32 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/2731085240' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:44:32.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:32 smithi084 ceph-mon[40024]: purged_snaps scrub starts 2024-09-20T22:44:32.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:32 smithi084 ceph-mon[40024]: purged_snaps scrub ok 2024-09-20T22:44:32.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:32 smithi084 ceph-mon[40024]: osdmap e192: 8 total, 8 up, 7 in 2024-09-20T22:44:32.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:32 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/2731085240' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:44:32.424 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:44:32.437 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:32 smithi154 ceph-mon[33264]: purged_snaps scrub starts 2024-09-20T22:44:32.437 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:32 smithi154 ceph-mon[33264]: purged_snaps scrub ok 2024-09-20T22:44:32.437 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:32 smithi154 ceph-mon[33264]: osdmap e192: 8 total, 8 up, 7 in 2024-09-20T22:44:32.437 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:32 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/2731085240' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:44:32.446 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:32.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:32.447 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:44:32.547 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:44:32.568 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:32.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:32.569 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:44:32.594 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.5 dump_blocked_ops 2024-09-20T22:44:32.670 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:44:32.691 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:32.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:32.692 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:44:32.793 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:44:32.815 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:32.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:32.816 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:44:32.842 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:44:32.917 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:44:32.939 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:32.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:32.940 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:44:33.040 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:44:33.063 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:33.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:33.064 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:44:33.104 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd unset noscrub 2024-09-20T22:44:33.144 INFO:tasks.rados.rados.0.smithi084.stdout:548: done (2 left) 2024-09-20T22:44:33.144 INFO:tasks.rados.rados.0.smithi084.stdout:549: read oid 43 snap -1 2024-09-20T22:44:33.145 INFO:tasks.rados.rados.0.smithi084.stdout:549: expect (ObjNum 194 snap 40 seq_num 194) 2024-09-20T22:44:33.145 INFO:tasks.rados.rados.0.smithi084.stdout:550: write oid 33 current snap is 48 2024-09-20T22:44:33.145 INFO:tasks.rados.rados.0.smithi084.stdout:550: seq_num 229 ranges {648803=722611,1967200=714747,3421983=426820} 2024-09-20T22:44:33.148 INFO:tasks.rados.rados.0.smithi084.stdout:550: writing smithi08477203-33 from 648803 to 1371414 tid 1 2024-09-20T22:44:33.157 INFO:tasks.rados.rados.0.smithi084.stdout:550: writing smithi08477203-33 from 1967200 to 2681947 tid 2 2024-09-20T22:44:33.164 INFO:tasks.rados.rados.0.smithi084.stdout:550: writing smithi08477203-33 from 3421983 to 3848803 tid 3 2024-09-20T22:44:33.164 INFO:tasks.rados.rados.0.smithi084.stdout:551: read oid 35 snap -1 2024-09-20T22:44:33.164 INFO:tasks.rados.rados.0.smithi084.stdout:551: expect (ObjNum 150 snap 30 seq_num 150) 2024-09-20T22:44:33.164 INFO:tasks.rados.rados.0.smithi084.stdout:552: rollback oid 39 current snap is 48 2024-09-20T22:44:33.165 INFO:tasks.rados.rados.0.smithi084.stdout:552: done (5 left) 2024-09-20T22:44:33.165 INFO:tasks.rados.rados.0.smithi084.stdout:553: read oid 40 snap -1 2024-09-20T22:44:33.165 INFO:tasks.rados.rados.0.smithi084.stdout:553: expect (ObjNum 226 snap 48 seq_num 226) 2024-09-20T22:44:33.165 INFO:tasks.rados.rados.0.smithi084.stdout:554: rollback oid 49 current snap is 48 2024-09-20T22:44:33.165 INFO:tasks.rados.rados.0.smithi084.stdout:554: done (6 left) 2024-09-20T22:44:33.165 INFO:tasks.rados.rados.0.smithi084.stdout:555: copy_from oid 2 from oid 32 current snap is 48 2024-09-20T22:44:33.165 INFO:tasks.rados.rados.0.smithi084.stdout:545: finishing write tid 3 to smithi08477203-21 2024-09-20T22:44:33.165 INFO:tasks.rados.rados.0.smithi084.stdout:545: finishing write tid 4 to smithi08477203-21 2024-09-20T22:44:33.165 INFO:tasks.rados.rados.0.smithi084.stdout:545: finishing write tid 5 to smithi08477203-21 2024-09-20T22:44:33.165 INFO:tasks.rados.rados.0.smithi084.stdout:545: oid 21 updating version 0 to 163 2024-09-20T22:44:33.165 INFO:tasks.rados.rados.0.smithi084.stdout:545: oid 21 updating version 163 to 164 2024-09-20T22:44:33.165 INFO:tasks.rados.rados.0.smithi084.stdout:545: oid 21 version 164 is already newer than 162 2024-09-20T22:44:33.165 INFO:tasks.rados.rados.0.smithi084.stdout:545: oid 21 updating version 164 to 165 2024-09-20T22:44:33.165 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 21 v 165 (ObjNum 227 snap 48 seq_num 227) dirty exists 2024-09-20T22:44:33.165 INFO:tasks.rados.rados.0.smithi084.stdout:545: left oid 21 (ObjNum 227 snap 48 seq_num 227) 2024-09-20T22:44:33.166 INFO:tasks.rados.rados.0.smithi084.stdout:547: finishing copy_from racing read to smithi08477203-27 2024-09-20T22:44:33.166 INFO:tasks.rados.rados.0.smithi084.stdout:545: done (6 left) 2024-09-20T22:44:33.166 INFO:tasks.rados.rados.0.smithi084.stdout:556: write (excl) oid 37 current snap is 48 2024-09-20T22:44:33.166 INFO:tasks.rados.rados.0.smithi084.stdout:547: finishing copy_from to smithi08477203-27 2024-09-20T22:44:33.166 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 27 v 106 (ObjNum 219 snap 46 seq_num 219) dirty exists 2024-09-20T22:44:33.166 INFO:tasks.rados.rados.0.smithi084.stdout:556: seq_num 230 ranges {665610=797860,1909329=458405,2816700=410210} 2024-09-20T22:44:33.166 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:44:33.172 INFO:tasks.rados.rados.0.smithi084.stdout:556: writing smithi08477203-37 from 665610 to 1463470 tid 1 2024-09-20T22:44:33.177 INFO:tasks.rados.rados.0.smithi084.stdout:556: writing smithi08477203-37 from 1909329 to 2367734 tid 2 2024-09-20T22:44:33.183 INFO:tasks.rados.rados.0.smithi084.stdout:556: writing smithi08477203-37 from 2816700 to 3226910 tid 3 2024-09-20T22:44:33.183 INFO:tasks.rados.rados.0.smithi084.stdout:547: done (6 left) 2024-09-20T22:44:33.183 INFO:tasks.rados.rados.0.smithi084.stdout:557: copy_from oid 18 from oid 41 current snap is 48 2024-09-20T22:44:33.183 INFO:tasks.rados.rados.0.smithi084.stdout:549: expect (ObjNum 194 snap 40 seq_num 194) 2024-09-20T22:44:33.195 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:33.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:33.196 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:44:33.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:33 smithi154 ceph-mon[33264]: pgmap v190: 178 pgs: 178 active+clean; 116 MiB data, 501 MiB used, 625 GiB / 626 GiB avail; 19 MiB/s rd, 2.8 MiB/s wr, 24 op/s 2024-09-20T22:44:33.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:33 smithi154 ceph-mon[33264]: osdmap e193: 8 total, 8 up, 7 in 2024-09-20T22:44:33.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:33 smithi154 ceph-mon[33264]: osdmap e194: 8 total, 8 up, 7 in 2024-09-20T22:44:33.293 INFO:tasks.rados.rados.0.smithi084.stdout:550: finishing write tid 1 to smithi08477203-33 2024-09-20T22:44:33.294 INFO:tasks.rados.rados.0.smithi084.stdout:550: finishing write tid 2 to smithi08477203-33 2024-09-20T22:44:33.294 INFO:tasks.rados.rados.0.smithi084.stdout:555: finishing copy_from to smithi08477203-2 2024-09-20T22:44:33.294 INFO:tasks.rados.rados.0.smithi084.stdout:555: got expected ENOENT (src dne) 2024-09-20T22:44:33.294 INFO:tasks.rados.rados.0.smithi084.stdout:555: finishing copy_from racing read to smithi08477203-2 2024-09-20T22:44:33.294 INFO:tasks.rados.rados.0.smithi084.stdout:551: expect (ObjNum 150 snap 30 seq_num 150) 2024-09-20T22:44:33.297 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:44:33.301 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-20T22:44:33.301 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:44:33.318 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:33.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:33.319 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:44:33.382 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:44:33.420 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:44:33.426 INFO:tasks.rados.rados.0.smithi084.stdout:550: finishing write tid 3 to smithi08477203-33 2024-09-20T22:44:33.427 INFO:tasks.rados.rados.0.smithi084.stdout:550: finishing write tid 4 to smithi08477203-33 2024-09-20T22:44:33.427 INFO:tasks.rados.rados.0.smithi084.stdout:550: finishing write tid 5 to smithi08477203-33 2024-09-20T22:44:33.427 INFO:tasks.rados.rados.0.smithi084.stdout:550: oid 33 updating version 0 to 121 2024-09-20T22:44:33.427 INFO:tasks.rados.rados.0.smithi084.stdout:550: oid 33 version 121 is already newer than 120 2024-09-20T22:44:33.427 INFO:tasks.rados.rados.0.smithi084.stdout:550: oid 33 updating version 121 to 123 2024-09-20T22:44:33.427 INFO:tasks.rados.rados.0.smithi084.stdout:550: oid 33 version 123 is already newer than 122 2024-09-20T22:44:33.427 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 33 v 123 (ObjNum 229 snap 48 seq_num 229) dirty exists 2024-09-20T22:44:33.427 INFO:tasks.rados.rados.0.smithi084.stdout:550: left oid 33 (ObjNum 229 snap 48 seq_num 229) 2024-09-20T22:44:33.427 INFO:tasks.rados.rados.0.smithi084.stdout:553: expect (ObjNum 226 snap 48 seq_num 226) 2024-09-20T22:44:33.442 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:33.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:33.443 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:44:33.504 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:33 smithi084 ceph-mon[40024]: pgmap v190: 178 pgs: 178 active+clean; 116 MiB data, 501 MiB used, 625 GiB / 626 GiB avail; 19 MiB/s rd, 2.8 MiB/s wr, 24 op/s 2024-09-20T22:44:33.504 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:33 smithi084 ceph-mon[40024]: osdmap e193: 8 total, 8 up, 7 in 2024-09-20T22:44:33.504 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:33 smithi084 ceph-mon[40024]: osdmap e194: 8 total, 8 up, 7 in 2024-09-20T22:44:33.504 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:33 smithi084 ceph-mon[29774]: pgmap v190: 178 pgs: 178 active+clean; 116 MiB data, 501 MiB used, 625 GiB / 626 GiB avail; 19 MiB/s rd, 2.8 MiB/s wr, 24 op/s 2024-09-20T22:44:33.504 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:33 smithi084 ceph-mon[29774]: osdmap e193: 8 total, 8 up, 7 in 2024-09-20T22:44:33.504 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:33 smithi084 ceph-mon[29774]: osdmap e194: 8 total, 8 up, 7 in 2024-09-20T22:44:33.544 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:44:33.564 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:33.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:33.565 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:44:33.666 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:44:33.685 INFO:tasks.rados.rados.0.smithi084.stdout:556: finishing write tid 1 to smithi08477203-37 2024-09-20T22:44:33.685 INFO:tasks.rados.rados.0.smithi084.stdout:556: finishing write tid 2 to smithi08477203-37 2024-09-20T22:44:33.685 INFO:tasks.rados.rados.0.smithi084.stdout:556: finishing write tid 3 to smithi08477203-37 2024-09-20T22:44:33.685 INFO:tasks.rados.rados.0.smithi084.stdout:556: finishing write tid 4 to smithi08477203-37 2024-09-20T22:44:33.686 INFO:tasks.rados.rados.0.smithi084.stdout:556: finishing write tid 5 to smithi08477203-37 2024-09-20T22:44:33.686 INFO:tasks.rados.rados.0.smithi084.stdout:556: oid 37 updating version 0 to 95 2024-09-20T22:44:33.686 INFO:tasks.rados.rados.0.smithi084.stdout:556: oid 37 updating version 95 to 96 2024-09-20T22:44:33.686 INFO:tasks.rados.rados.0.smithi084.stdout:556: oid 37 updating version 96 to 97 2024-09-20T22:44:33.686 INFO:tasks.rados.rados.0.smithi084.stdout:556: oid 37 updating version 97 to 98 2024-09-20T22:44:33.686 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 37 v 98 (ObjNum 230 snap 48 seq_num 230) dirty exists 2024-09-20T22:44:33.686 INFO:tasks.rados.rados.0.smithi084.stdout:556: left oid 37 (ObjNum 230 snap 48 seq_num 230) 2024-09-20T22:44:33.686 INFO:tasks.rados.rados.0.smithi084.stdout:557: finishing copy_from racing read to smithi08477203-18 2024-09-20T22:44:33.686 INFO:tasks.rados.rados.0.smithi084.stdout:557: finishing copy_from to smithi08477203-18 2024-09-20T22:44:33.686 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 18 v 42 (ObjNum 167 snap 36 seq_num 167) dirty exists 2024-09-20T22:44:33.686 INFO:tasks.rados.rados.0.smithi084.stdout:549: done (6 left) 2024-09-20T22:44:33.686 INFO:tasks.rados.rados.0.smithi084.stdout:550: done (5 left) 2024-09-20T22:44:33.686 INFO:tasks.rados.rados.0.smithi084.stdout:551: done (4 left) 2024-09-20T22:44:33.686 INFO:tasks.rados.rados.0.smithi084.stdout:553: done (3 left) 2024-09-20T22:44:33.686 INFO:tasks.rados.rados.0.smithi084.stdout:555: done (2 left) 2024-09-20T22:44:33.687 INFO:tasks.rados.rados.0.smithi084.stdout:556: done (1 left) 2024-09-20T22:44:33.687 INFO:tasks.rados.rados.0.smithi084.stdout:557: done (0 left) 2024-09-20T22:44:33.687 INFO:tasks.rados.rados.0.smithi084.stdout:558: read oid 49 snap -1 2024-09-20T22:44:33.687 INFO:tasks.rados.rados.0.smithi084.stdout:558: expect (ObjNum 213 snap 44 seq_num 213) 2024-09-20T22:44:33.687 INFO:tasks.rados.rados.0.smithi084.stdout:559: snap_create 2024-09-20T22:44:33.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:33.688 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:33.688 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:44:33.693 INFO:tasks.rados.rados.0.smithi084.stdout:558: expect (ObjNum 213 snap 44 seq_num 213) 2024-09-20T22:44:33.789 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:44:33.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:33.811 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:33.811 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:44:33.908 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 7] out_osds: [4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-20T22:44:33.908 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-20T22:44:33.908 INFO:tasks.thrashosds.thrasher:inject_pause on osd.5 2024-09-20T22:44:33.908 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-20T22:44:33.908 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-20T22:44:33.909 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.5 config set filestore_inject_stall 3 2024-09-20T22:44:33.912 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:44:33.933 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:33.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:33.934 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:44:33.987 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.5 dump_historic_ops 2024-09-20T22:44:34.035 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:44:34.057 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:34.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:34.058 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:44:34.159 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:44:34.165 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:44:34.184 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:34.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:34.185 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:44:34.189 INFO:tasks.rados.rados.0.smithi084.stdout:558: done (1 left) 2024-09-20T22:44:34.189 INFO:tasks.rados.rados.0.smithi084.stdout:559: done (0 left) 2024-09-20T22:44:34.189 INFO:tasks.rados.rados.0.smithi084.stdout:560: snap_remove snap 48 2024-09-20T22:44:34.189 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is unset 2024-09-20T22:44:34.285 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:44:34.307 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:34.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:34.308 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:44:34.408 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:44:34.431 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:34.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:34.432 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:44:34.449 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:34 smithi154 ceph-mon[33264]: purged_snaps scrub starts 2024-09-20T22:44:34.449 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:34 smithi154 ceph-mon[33264]: purged_snaps scrub ok 2024-09-20T22:44:34.449 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:34 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/980624584' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:44:34.449 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:34 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:44:34.479 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-20T22:44:34.500 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:34 smithi084 ceph-mon[29774]: purged_snaps scrub starts 2024-09-20T22:44:34.501 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:34 smithi084 ceph-mon[29774]: purged_snaps scrub ok 2024-09-20T22:44:34.501 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:34 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/980624584' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:44:34.501 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:34 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:44:34.501 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:34 smithi084 ceph-mon[40024]: purged_snaps scrub starts 2024-09-20T22:44:34.501 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:34 smithi084 ceph-mon[40024]: purged_snaps scrub ok 2024-09-20T22:44:34.501 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:34 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/980624584' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:44:34.501 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:34 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:44:34.534 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:44:34.556 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:34.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:34.557 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:44:34.658 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:44:34.679 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:34.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:34.680 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:44:34.686 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-20T22:44:34.686 INFO:teuthology.orchestra.run.smithi154.stdout: "success": "" 2024-09-20T22:44:34.686 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-20T22:44:34.781 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:44:34.803 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:34.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:34.804 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:44:34.905 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:44:34.927 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:34.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:34.927 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:44:35.028 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:44:35.049 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:35.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:35.050 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:44:35.060 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:44:35.151 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:44:35.172 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:35.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:35.173 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:44:35.195 INFO:tasks.rados.rados.0.smithi084.stdout:560: done (0 left) 2024-09-20T22:44:35.195 INFO:tasks.rados.rados.0.smithi084.stdout:561: rollback oid 10 current snap is 49 2024-09-20T22:44:35.195 INFO:tasks.rados.rados.0.smithi084.stdout:561: done (0 left) 2024-09-20T22:44:35.195 INFO:tasks.rados.rados.0.smithi084.stdout:562: read oid 2 snap -1 2024-09-20T22:44:35.195 INFO:tasks.rados.rados.0.smithi084.stdout:562: expect (ObjNum 201 snap 41 seq_num 201) 2024-09-20T22:44:35.195 INFO:tasks.rados.rados.0.smithi084.stdout:563: write oid 40 current snap is 49 2024-09-20T22:44:35.195 INFO:tasks.rados.rados.0.smithi084.stdout:563: seq_num 231 ranges {682417=473109,1567811=602063,2844123=621482} 2024-09-20T22:44:35.200 INFO:tasks.rados.rados.0.smithi084.stdout:563: writing smithi08477203-40 from 682417 to 1155526 tid 1 2024-09-20T22:44:35.207 INFO:tasks.rados.rados.0.smithi084.stdout:563: writing smithi08477203-40 from 1567811 to 2169874 tid 2 2024-09-20T22:44:35.216 INFO:tasks.rados.rados.0.smithi084.stdout:563: writing smithi08477203-40 from 2844123 to 3465605 tid 3 2024-09-20T22:44:35.216 INFO:tasks.rados.rados.0.smithi084.stdout:564: snap_create 2024-09-20T22:44:35.216 INFO:tasks.rados.rados.0.smithi084.stdout:562: expect (ObjNum 201 snap 41 seq_num 201) 2024-09-20T22:44:35.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:35 smithi154 ceph-mon[33264]: pgmap v193: 178 pgs: 1 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 174 active+clean; 118 MiB data, 534 MiB used, 625 GiB / 626 GiB avail; 10 MiB/s rd, 5.7 MiB/s wr, 23 op/s 2024-09-20T22:44:35.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:35 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T22:44:35.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:35 smithi154 ceph-mon[33264]: osdmap e195: 8 total, 8 up, 7 in 2024-09-20T22:44:35.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:35 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:44:35.273 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:44:35.294 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:35.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:35.295 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:44:35.326 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:35 smithi084 ceph-mon[40024]: pgmap v193: 178 pgs: 1 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 174 active+clean; 118 MiB data, 534 MiB used, 625 GiB / 626 GiB avail; 10 MiB/s rd, 5.7 MiB/s wr, 23 op/s 2024-09-20T22:44:35.326 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:35 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T22:44:35.326 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:35 smithi084 ceph-mon[40024]: osdmap e195: 8 total, 8 up, 7 in 2024-09-20T22:44:35.326 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:35 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:44:35.326 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:35 smithi084 ceph-mon[29774]: pgmap v193: 178 pgs: 1 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 174 active+clean; 118 MiB data, 534 MiB used, 625 GiB / 626 GiB avail; 10 MiB/s rd, 5.7 MiB/s wr, 23 op/s 2024-09-20T22:44:35.326 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:35 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T22:44:35.327 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:35 smithi084 ceph-mon[29774]: osdmap e195: 8 total, 8 up, 7 in 2024-09-20T22:44:35.327 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:35 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:44:35.378 INFO:tasks.rados.rados.0.smithi084.stdout:563: finishing write tid 1 to smithi08477203-40 2024-09-20T22:44:35.378 INFO:tasks.rados.rados.0.smithi084.stdout:563: finishing write tid 2 to smithi08477203-40 2024-09-20T22:44:35.378 INFO:tasks.rados.rados.0.smithi084.stdout:563: finishing write tid 3 to smithi08477203-40 2024-09-20T22:44:35.378 INFO:tasks.rados.rados.0.smithi084.stdout:563: finishing write tid 4 to smithi08477203-40 2024-09-20T22:44:35.378 INFO:tasks.rados.rados.0.smithi084.stdout:563: finishing write tid 5 to smithi08477203-40 2024-09-20T22:44:35.378 INFO:tasks.rados.rados.0.smithi084.stdout:563: oid 40 updating version 0 to 127 2024-09-20T22:44:35.378 INFO:tasks.rados.rados.0.smithi084.stdout:563: oid 40 version 127 is already newer than 125 2024-09-20T22:44:35.378 INFO:tasks.rados.rados.0.smithi084.stdout:563: oid 40 version 127 is already newer than 124 2024-09-20T22:44:35.378 INFO:tasks.rados.rados.0.smithi084.stdout:563: oid 40 version 127 is already newer than 126 2024-09-20T22:44:35.378 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 40 v 127 (ObjNum 231 snap 49 seq_num 231) dirty exists 2024-09-20T22:44:35.378 INFO:tasks.rados.rados.0.smithi084.stdout:563: left oid 40 (ObjNum 231 snap 49 seq_num 231) 2024-09-20T22:44:35.396 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:44:35.417 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:35.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:35.418 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:44:35.454 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:44:35.519 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:44:35.541 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:35.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:35.542 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:44:35.643 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:44:35.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:35.666 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:35.666 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:44:35.733 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:44:35.733 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:44:35.751 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:44:35.751 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:44:35.767 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:44:35.767 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:44:35.767 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:44:35.783 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-20T22:44:35.783 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T22:44:35.790 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:35.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:35.790 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:44:35.827 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:44:35.827 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:44:35.842 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:44:35.842 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:44:35.859 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:44:35.859 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:44:35.875 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:44:35.875 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:44:35.892 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:44:35.912 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:35.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:35.913 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:44:35.964 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-20T22:44:35.964 INFO:teuthology.orchestra.run.smithi154.stdout: "size": 20, 2024-09-20T22:44:35.964 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 600, 2024-09-20T22:44:35.965 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [ 2024-09-20T22:44:35.965 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:35.965 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24334.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-09-20T22:44:35.965 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:41:39.154468+0000", 2024-09-20T22:44:35.965 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 176.806690081, 2024-09-20T22:44:35.965 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0047727469999999999, 2024-09-20T22:44:35.965 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:44:35.966 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:44:35.966 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:44:35.966 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:44:35.966 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24334", 2024-09-20T22:44:35.966 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/343304630", 2024-09-20T22:44:35.966 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 5 2024-09-20T22:44:35.967 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:35.967 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:44:35.967 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:35.967 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:44:35.967 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:39.154468+0000", 2024-09-20T22:44:35.967 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:44:35.968 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:35.968 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:35.968 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:44:35.968 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:39.154468+0000", 2024-09-20T22:44:35.968 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:44:35.968 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:35.969 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:35.969 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:44:35.969 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:39.154471+0000", 2024-09-20T22:44:35.969 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.773e-06 2024-09-20T22:44:35.969 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:35.969 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:35.970 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:44:35.970 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:39.154481+0000", 2024-09-20T22:44:35.970 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.6649999999999998e-06 2024-09-20T22:44:35.970 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:35.970 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:35.970 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:44:35.971 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:39.154517+0000", 2024-09-20T22:44:35.971 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.6146000000000003e-05 2024-09-20T22:44:35.971 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:35.971 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:35.971 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:44:35.971 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:39.154541+0000", 2024-09-20T22:44:35.971 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.3677999999999999e-05 2024-09-20T22:44:35.971 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:35.971 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:35.971 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:44:35.971 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:39.155215+0000", 2024-09-20T22:44:35.971 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00067437699999999998 2024-09-20T22:44:35.972 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:35.972 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:35.972 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:44:35.972 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:39.155352+0000", 2024-09-20T22:44:35.972 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000137365 2024-09-20T22:44:35.972 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:35.972 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:35.972 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 1,6", 2024-09-20T22:44:35.972 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:39.155511+0000", 2024-09-20T22:44:35.972 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000158608 2024-09-20T22:44:35.972 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:35.972 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:35.973 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:44:35.973 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:39.157366+0000", 2024-09-20T22:44:35.973 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001854589 2024-09-20T22:44:35.973 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:35.973 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:35.973 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:44:35.973 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:39.158647+0000", 2024-09-20T22:44:35.973 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001281344 2024-09-20T22:44:35.973 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:35.973 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:35.973 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:44:35.973 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:39.159164+0000", 2024-09-20T22:44:35.973 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00051662699999999997 2024-09-20T22:44:35.973 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:35.973 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:35.974 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:44:35.974 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:39.159203+0000", 2024-09-20T22:44:35.974 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.9209999999999999e-05 2024-09-20T22:44:35.974 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:35.974 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:35.974 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:44:35.974 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:39.159241+0000", 2024-09-20T22:44:35.974 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.8365000000000003e-05 2024-09-20T22:44:35.974 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:44:35.974 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:44:35.974 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:44:35.974 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:35.974 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:35.975 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24334.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-09-20T22:44:35.975 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:41:40.156321+0000", 2024-09-20T22:44:35.975 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 175.80483706199999, 2024-09-20T22:44:35.975 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.012506929, 2024-09-20T22:44:35.975 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:44:35.975 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:44:35.975 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:44:35.975 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:44:35.975 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24334", 2024-09-20T22:44:35.975 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/343304630", 2024-09-20T22:44:35.975 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 7 2024-09-20T22:44:35.975 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:35.975 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:44:35.976 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:35.976 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:44:35.976 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:40.156321+0000", 2024-09-20T22:44:35.976 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:44:35.976 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:35.976 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:35.976 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:44:35.976 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:40.156321+0000", 2024-09-20T22:44:35.976 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:44:35.976 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:35.977 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:35.977 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:44:35.977 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:40.156323+0000", 2024-09-20T22:44:35.977 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9700000000000002e-06 2024-09-20T22:44:35.977 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:35.977 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:35.977 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:44:35.977 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:40.156330+0000", 2024-09-20T22:44:35.977 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.9809999999999997e-06 2024-09-20T22:44:35.977 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:35.977 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:35.977 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:44:35.977 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:40.156348+0000", 2024-09-20T22:44:35.978 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.7938e-05 2024-09-20T22:44:35.978 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:35.978 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:35.978 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:44:35.978 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:40.156362+0000", 2024-09-20T22:44:35.978 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3641e-05 2024-09-20T22:44:35.978 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:35.978 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:35.978 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:44:35.978 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:40.156430+0000", 2024-09-20T22:44:35.978 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.7543e-05 2024-09-20T22:44:35.978 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:35.978 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:35.979 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:44:35.979 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:40.156543+0000", 2024-09-20T22:44:35.979 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00011390100000000001 2024-09-20T22:44:35.979 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:35.979 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:35.979 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 1,6", 2024-09-20T22:44:35.979 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:40.156712+0000", 2024-09-20T22:44:35.979 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00016825499999999999 2024-09-20T22:44:35.979 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:35.979 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:35.979 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:44:35.979 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:40.157640+0000", 2024-09-20T22:44:35.979 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00092863699999999995 2024-09-20T22:44:35.980 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:35.980 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:35.980 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:44:35.980 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:40.159647+0000", 2024-09-20T22:44:35.980 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0020068320000000001 2024-09-20T22:44:35.980 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:35.980 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:35.980 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:44:35.980 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:40.168687+0000", 2024-09-20T22:44:35.980 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0090397749999999999 2024-09-20T22:44:35.980 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:35.980 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:35.980 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:44:35.981 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:40.168768+0000", 2024-09-20T22:44:35.981 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.0819e-05 2024-09-20T22:44:35.981 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:35.981 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:35.981 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:44:35.981 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:40.168828+0000", 2024-09-20T22:44:35.981 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.0637000000000002e-05 2024-09-20T22:44:35.981 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:44:35.981 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:44:35.981 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:44:35.981 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:35.981 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:35.981 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "MOSDPGPush(1.4 93/92 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 77'128, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@77'128, size: 327680, copy_subset: [(0, 327680)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 327680, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-20T22:44:35.982 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:42:50.646393+0000", 2024-09-20T22:44:35.982 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 105.314765623, 2024-09-20T22:44:35.982 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.033640915, 2024-09-20T22:44:35.982 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:44:35.982 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:44:35.982 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:44:35.982 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:44:35.982 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:35.982 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:44:35.982 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:42:50.646393+0000", 2024-09-20T22:44:35.982 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:44:35.982 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:35.983 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:35.983 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:44:35.983 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:42:50.646393+0000", 2024-09-20T22:44:35.983 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:44:35.983 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:35.983 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:35.983 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:44:35.983 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:42:50.646393+0000", 2024-09-20T22:44:35.983 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.23e-07 2024-09-20T22:44:35.983 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:35.983 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:35.983 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:44:35.983 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:42:50.679431+0000", 2024-09-20T22:44:35.983 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.033038349000000002 2024-09-20T22:44:35.984 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:35.984 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:35.984 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:44:35.984 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:42:50.679478+0000", 2024-09-20T22:44:35.984 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.6828e-05 2024-09-20T22:44:35.984 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:35.984 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:35.984 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:44:35.984 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:42:50.679492+0000", 2024-09-20T22:44:35.984 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4062e-05 2024-09-20T22:44:35.984 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:35.984 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:35.984 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:44:35.985 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:42:50.679593+0000", 2024-09-20T22:44:35.985 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000100865 2024-09-20T22:44:35.985 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:35.985 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:35.985 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:44:35.985 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:42:50.679604+0000", 2024-09-20T22:44:35.985 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0348e-05 2024-09-20T22:44:35.985 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:35.985 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:35.985 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:44:35.985 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:42:50.680034+0000", 2024-09-20T22:44:35.985 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00043023999999999997 2024-09-20T22:44:35.985 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:44:35.985 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:44:35.986 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:44:35.986 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:35.986 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:35.986 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:321 7.b e105/94 7:df02c955:::smithi08477203-9:head v 105'17, mlcod=98'14)", 2024-09-20T22:44:35.986 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:04.861367+0000", 2024-09-20T22:44:35.986 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 91.099791732, 2024-09-20T22:44:35.986 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0067649240000000003, 2024-09-20T22:44:35.986 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:44:35.986 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:44:35.986 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:44:35.986 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:44:35.986 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:35.986 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:44:35.986 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:04.861367+0000", 2024-09-20T22:44:35.987 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:44:35.987 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:35.987 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:35.987 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:44:35.987 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:04.861367+0000", 2024-09-20T22:44:35.987 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:44:35.987 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:35.987 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:35.987 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:44:35.987 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:04.861369+0000", 2024-09-20T22:44:35.987 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.8470000000000001e-06 2024-09-20T22:44:35.987 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:35.987 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:35.987 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:44:35.988 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:04.865306+0000", 2024-09-20T22:44:35.988 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0039377969999999998 2024-09-20T22:44:35.988 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:35.988 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:35.988 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:44:35.988 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:04.865326+0000", 2024-09-20T22:44:35.988 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9865e-05 2024-09-20T22:44:35.988 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:35.988 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:35.988 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:44:35.988 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:04.865337+0000", 2024-09-20T22:44:35.988 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0961000000000001e-05 2024-09-20T22:44:35.988 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:35.988 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:35.988 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:44:35.989 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:04.865404+0000", 2024-09-20T22:44:35.989 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.6869e-05 2024-09-20T22:44:35.989 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:35.989 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:35.989 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:44:35.989 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:04.865422+0000", 2024-09-20T22:44:35.989 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.7824000000000001e-05 2024-09-20T22:44:35.989 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:35.989 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:35.989 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:44:35.989 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:04.868076+0000", 2024-09-20T22:44:35.989 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0026539789999999999 2024-09-20T22:44:35.989 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:35.990 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:35.990 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:44:35.990 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:04.868132+0000", 2024-09-20T22:44:35.990 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.5782000000000002e-05 2024-09-20T22:44:35.990 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:44:35.990 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:44:35.990 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:44:35.990 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:35.990 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:35.990 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:340 7.0 7:0f524219:::smithi08477203-50:head [read 0~2352947,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e107)", 2024-09-20T22:44:35.990 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:06.543062+0000", 2024-09-20T22:44:35.990 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 89.418096645000006, 2024-09-20T22:44:35.990 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0048265390000000003, 2024-09-20T22:44:35.990 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:44:35.990 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:44:35.991 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:44:35.991 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:44:35.991 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:44:35.991 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:44:35.991 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 340 2024-09-20T22:44:35.991 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:35.991 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:44:35.991 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:35.991 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:44:35.991 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:06.543062+0000", 2024-09-20T22:44:35.991 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:44:35.991 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:35.991 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:35.992 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:44:35.992 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:06.543062+0000", 2024-09-20T22:44:35.992 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:44:35.992 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:35.992 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:35.992 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:44:35.992 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:06.543063+0000", 2024-09-20T22:44:35.992 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.2699999999999999e-07 2024-09-20T22:44:35.992 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:35.992 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:35.992 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:44:35.992 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:06.543065+0000", 2024-09-20T22:44:35.992 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.8789999999999999e-06 2024-09-20T22:44:35.992 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:35.992 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:35.993 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:44:35.993 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:06.543071+0000", 2024-09-20T22:44:35.993 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.3249999999999998e-06 2024-09-20T22:44:35.993 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:35.993 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:35.993 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:44:35.993 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:06.543074+0000", 2024-09-20T22:44:35.993 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.4340000000000001e-06 2024-09-20T22:44:35.993 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:35.993 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:35.993 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:44:35.993 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:06.543174+0000", 2024-09-20T22:44:35.993 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.9910999999999999e-05 2024-09-20T22:44:35.994 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:35.994 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:35.994 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting_for_map not empty", 2024-09-20T22:44:35.994 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:06.543177+0000", 2024-09-20T22:44:35.994 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.4089999999999999e-06 2024-09-20T22:44:35.994 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:35.994 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:35.994 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:44:35.994 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:06.547207+0000", 2024-09-20T22:44:35.994 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00402908 2024-09-20T22:44:35.994 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:35.994 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:35.994 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:44:35.994 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:06.547319+0000", 2024-09-20T22:44:35.995 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000112877 2024-09-20T22:44:35.995 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:35.995 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:35.995 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:44:35.995 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:06.547888+0000", 2024-09-20T22:44:35.995 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00056899700000000001 2024-09-20T22:44:35.995 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:44:35.995 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:44:35.995 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:44:35.995 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:35.995 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:35.995 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:389 7.0 7:0f524219:::smithi08477203-50:head [write 776490~714607] snapc 8=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-09-20T22:44:35.995 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:11.337342+0000", 2024-09-20T22:44:35.996 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 84.623816728999998, 2024-09-20T22:44:35.996 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0050302089999999999, 2024-09-20T22:44:35.996 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:44:35.996 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:44:35.997 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:44:35.997 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:44:35.997 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:44:35.997 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:44:35.997 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 389 2024-09-20T22:44:35.997 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:35.997 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:44:35.998 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:35.998 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:44:35.998 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:11.337342+0000", 2024-09-20T22:44:35.998 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:44:35.998 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:35.998 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:35.998 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:44:35.998 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:11.337342+0000", 2024-09-20T22:44:35.998 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:44:35.998 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:35.998 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:35.998 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:44:35.999 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:11.337344+0000", 2024-09-20T22:44:35.999 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.5289999999999998e-06 2024-09-20T22:44:35.999 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:35.999 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:35.999 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:44:35.999 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:11.338047+0000", 2024-09-20T22:44:36.000 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00070280099999999997 2024-09-20T22:44:36.000 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.000 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.000 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:44:36.000 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:11.338069+0000", 2024-09-20T22:44:36.000 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.2283e-05 2024-09-20T22:44:36.000 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.000 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.000 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:44:36.000 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:11.338096+0000", 2024-09-20T22:44:36.000 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.6254999999999998e-05 2024-09-20T22:44:36.000 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.000 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.000 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:44:36.001 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:11.338172+0000", 2024-09-20T22:44:36.001 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.6590999999999999e-05 2024-09-20T22:44:36.001 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.001 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.001 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:44:36.001 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:11.338282+0000", 2024-09-20T22:44:36.001 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000109443 2024-09-20T22:44:36.001 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.001 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.001 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 1,3", 2024-09-20T22:44:36.001 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:11.338448+0000", 2024-09-20T22:44:36.001 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000166193 2024-09-20T22:44:36.001 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.002 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.002 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:44:36.002 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:11.339797+0000", 2024-09-20T22:44:36.002 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0013489400000000001 2024-09-20T22:44:36.002 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.002 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.002 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:44:36.002 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:11.341600+0000", 2024-09-20T22:44:36.002 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0018033140000000001 2024-09-20T22:44:36.002 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.002 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.002 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:44:36.002 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:11.342297+0000", 2024-09-20T22:44:36.002 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00069641199999999999 2024-09-20T22:44:36.002 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.003 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.003 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:44:36.003 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:11.342334+0000", 2024-09-20T22:44:36.003 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.7299999999999999e-05 2024-09-20T22:44:36.003 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.003 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.003 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:44:36.003 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:11.342372+0000", 2024-09-20T22:44:36.003 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.8148000000000001e-05 2024-09-20T22:44:36.003 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:44:36.003 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:44:36.003 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:44:36.003 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.003 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.004 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:419 7.0 7:05bb442d:::smithi08477203-24:head [write 460525~690852] snapc a=[9] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-09-20T22:44:36.004 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:14.165736+0000", 2024-09-20T22:44:36.004 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 81.795422728999995, 2024-09-20T22:44:36.004 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0072523500000000003, 2024-09-20T22:44:36.004 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:44:36.004 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:44:36.004 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:44:36.004 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:44:36.004 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:44:36.004 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:44:36.004 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 419 2024-09-20T22:44:36.004 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.004 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:44:36.004 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.004 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:44:36.005 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:14.165736+0000", 2024-09-20T22:44:36.005 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:44:36.005 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.005 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.005 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:44:36.005 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:14.165736+0000", 2024-09-20T22:44:36.005 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:44:36.005 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.005 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.005 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:44:36.005 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:14.165737+0000", 2024-09-20T22:44:36.005 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.615e-06 2024-09-20T22:44:36.005 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.005 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.006 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:44:36.006 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:14.166920+0000", 2024-09-20T22:44:36.006 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001182535 2024-09-20T22:44:36.006 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.006 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.006 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:44:36.006 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:14.166933+0000", 2024-09-20T22:44:36.006 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3263e-05 2024-09-20T22:44:36.006 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.006 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.006 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:44:36.006 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:14.166942+0000", 2024-09-20T22:44:36.006 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.3329999999999997e-06 2024-09-20T22:44:36.006 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.007 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.007 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:44:36.007 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:14.167001+0000", 2024-09-20T22:44:36.007 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.9305000000000002e-05 2024-09-20T22:44:36.007 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.007 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.007 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op must wait for map", 2024-09-20T22:44:36.007 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:14.167008+0000", 2024-09-20T22:44:36.007 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.5580000000000003e-06 2024-09-20T22:44:36.007 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.007 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.007 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:44:36.007 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:14.169081+0000", 2024-09-20T22:44:36.007 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0020726270000000001 2024-09-20T22:44:36.007 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.008 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.008 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:44:36.008 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:14.169179+0000", 2024-09-20T22:44:36.008 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.7613000000000002e-05 2024-09-20T22:44:36.008 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.008 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.008 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 1,3", 2024-09-20T22:44:36.008 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:14.169364+0000", 2024-09-20T22:44:36.008 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000185642 2024-09-20T22:44:36.008 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.008 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.008 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:44:36.008 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:14.171014+0000", 2024-09-20T22:44:36.008 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0016497129999999999 2024-09-20T22:44:36.009 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.009 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.009 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:44:36.009 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:14.172374+0000", 2024-09-20T22:44:36.009 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0013601240000000001 2024-09-20T22:44:36.009 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.009 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.009 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:44:36.009 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:14.172925+0000", 2024-09-20T22:44:36.009 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00055099499999999998 2024-09-20T22:44:36.009 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.009 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.009 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:44:36.009 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:14.172959+0000", 2024-09-20T22:44:36.010 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.4069000000000002e-05 2024-09-20T22:44:36.010 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.010 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.010 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:44:36.010 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:14.172988+0000", 2024-09-20T22:44:36.010 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.8958e-05 2024-09-20T22:44:36.010 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:44:36.010 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:44:36.010 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:44:36.010 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.010 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.010 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:550 7.0 7:05bb442d:::smithi08477203-24:head [read 0~3260525,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e128)", 2024-09-20T22:44:36.010 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:27.143379+0000", 2024-09-20T22:44:36.010 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 68.817779213999998, 2024-09-20T22:44:36.010 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0059942010000000002, 2024-09-20T22:44:36.010 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:44:36.011 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:44:36.011 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:44:36.011 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:44:36.011 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:44:36.011 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:44:36.011 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 550 2024-09-20T22:44:36.011 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.011 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:44:36.011 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.011 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:44:36.011 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.143379+0000", 2024-09-20T22:44:36.011 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:44:36.011 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.011 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.011 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:44:36.012 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.143379+0000", 2024-09-20T22:44:36.012 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:44:36.012 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.012 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.012 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:44:36.012 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.143382+0000", 2024-09-20T22:44:36.012 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.774e-06 2024-09-20T22:44:36.012 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.012 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.012 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:44:36.012 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.143393+0000", 2024-09-20T22:44:36.012 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1246e-05 2024-09-20T22:44:36.012 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.012 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.013 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:44:36.013 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.143416+0000", 2024-09-20T22:44:36.013 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.2340999999999998e-05 2024-09-20T22:44:36.013 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.013 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.013 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:44:36.013 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.143426+0000", 2024-09-20T22:44:36.013 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0010000000000001e-05 2024-09-20T22:44:36.013 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.013 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.013 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:44:36.013 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.143500+0000", 2024-09-20T22:44:36.013 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.4266999999999999e-05 2024-09-20T22:44:36.013 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.013 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.013 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op must wait for map", 2024-09-20T22:44:36.014 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.143508+0000", 2024-09-20T22:44:36.014 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.2830000000000001e-06 2024-09-20T22:44:36.014 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.014 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.014 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:44:36.014 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.145018+0000", 2024-09-20T22:44:36.014 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0015100280000000001 2024-09-20T22:44:36.014 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.014 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.014 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:44:36.014 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.145097+0000", 2024-09-20T22:44:36.014 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.8893000000000005e-05 2024-09-20T22:44:36.014 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.014 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.015 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:44:36.015 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.149374+0000", 2024-09-20T22:44:36.015 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0042763590000000004 2024-09-20T22:44:36.015 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:44:36.015 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:44:36.015 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:44:36.015 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.015 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.015 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:551 7.0 7:05bb442d:::smithi08477203-24:head [read 0~3260525,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e128)", 2024-09-20T22:44:36.015 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:27.143449+0000", 2024-09-20T22:44:36.015 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 68.817709636000004, 2024-09-20T22:44:36.015 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0071533109999999999, 2024-09-20T22:44:36.015 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:44:36.015 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:44:36.015 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:44:36.016 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:44:36.016 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:44:36.016 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:44:36.016 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 551 2024-09-20T22:44:36.016 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.016 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:44:36.016 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.016 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:44:36.016 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.143449+0000", 2024-09-20T22:44:36.016 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:44:36.016 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.016 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.016 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:44:36.016 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.143449+0000", 2024-09-20T22:44:36.017 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:44:36.017 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.017 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.017 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:44:36.017 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.143449+0000", 2024-09-20T22:44:36.017 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.2000000000000001e-07 2024-09-20T22:44:36.017 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.017 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.017 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:44:36.017 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.143456+0000", 2024-09-20T22:44:36.017 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.8859999999999998e-06 2024-09-20T22:44:36.017 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.017 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.017 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:44:36.017 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.143462+0000", 2024-09-20T22:44:36.018 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.5940000000000001e-06 2024-09-20T22:44:36.018 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.018 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.018 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:44:36.018 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.143465+0000", 2024-09-20T22:44:36.018 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.631e-06 2024-09-20T22:44:36.018 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.018 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.018 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:44:36.018 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.143568+0000", 2024-09-20T22:44:36.018 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000103 2024-09-20T22:44:36.018 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.018 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.018 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting_for_map not empty", 2024-09-20T22:44:36.019 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.143576+0000", 2024-09-20T22:44:36.019 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.2699999999999999e-06 2024-09-20T22:44:36.019 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.019 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.019 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:44:36.019 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.149401+0000", 2024-09-20T22:44:36.019 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0058254489999999999 2024-09-20T22:44:36.019 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.019 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.019 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:44:36.019 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.149466+0000", 2024-09-20T22:44:36.019 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.4772999999999998e-05 2024-09-20T22:44:36.019 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.019 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.019 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:44:36.019 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.150602+0000", 2024-09-20T22:44:36.020 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0011363879999999999 2024-09-20T22:44:36.020 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:44:36.020 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:44:36.020 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:44:36.020 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.020 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.020 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:552 7.0 7:05bb442d:::smithi08477203-24:head [sparse-read 0~1736210] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e128)", 2024-09-20T22:44:36.020 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:27.143470+0000", 2024-09-20T22:44:36.020 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 68.817688360000005, 2024-09-20T22:44:36.020 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0075036570000000004, 2024-09-20T22:44:36.020 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:44:36.020 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:44:36.020 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:44:36.020 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:44:36.021 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:44:36.021 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:44:36.021 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 552 2024-09-20T22:44:36.021 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.021 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:44:36.021 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.021 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:44:36.021 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.143470+0000", 2024-09-20T22:44:36.021 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:44:36.021 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.021 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.021 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:44:36.021 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.143470+0000", 2024-09-20T22:44:36.021 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:44:36.022 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.022 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.022 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:44:36.022 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.143470+0000", 2024-09-20T22:44:36.022 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.84e-07 2024-09-20T22:44:36.022 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.022 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.022 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:44:36.022 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.143472+0000", 2024-09-20T22:44:36.022 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4139999999999999e-06 2024-09-20T22:44:36.022 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.022 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.022 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:44:36.022 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.143475+0000", 2024-09-20T22:44:36.023 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.1360000000000001e-06 2024-09-20T22:44:36.023 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.023 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.023 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:44:36.023 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.143478+0000", 2024-09-20T22:44:36.023 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.9000000000000002e-06 2024-09-20T22:44:36.023 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.023 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.023 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:44:36.023 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.143601+0000", 2024-09-20T22:44:36.023 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00012325199999999999 2024-09-20T22:44:36.023 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.023 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.023 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting_for_map not empty", 2024-09-20T22:44:36.024 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.143605+0000", 2024-09-20T22:44:36.024 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.5540000000000002e-06 2024-09-20T22:44:36.024 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.024 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.024 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:44:36.024 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.150656+0000", 2024-09-20T22:44:36.024 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0070513290000000003 2024-09-20T22:44:36.024 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.024 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.024 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:44:36.024 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.150787+0000", 2024-09-20T22:44:36.024 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00013061099999999999 2024-09-20T22:44:36.024 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.025 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.025 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:44:36.025 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.150974+0000", 2024-09-20T22:44:36.025 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00018717699999999999 2024-09-20T22:44:36.025 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:44:36.025 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:44:36.025 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:44:36.025 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.025 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.025 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:554 7.a e128/94 7:5122292a:::smithi08477203-33:head v 128'32, mlcod=119'28)", 2024-09-20T22:44:36.025 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:27.341250+0000", 2024-09-20T22:44:36.025 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 68.619908038999995, 2024-09-20T22:44:36.025 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.051661319999999997, 2024-09-20T22:44:36.025 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:44:36.025 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:44:36.025 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:44:36.026 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:44:36.026 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.026 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:44:36.026 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.341250+0000", 2024-09-20T22:44:36.026 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:44:36.026 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.026 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.026 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:44:36.026 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.341250+0000", 2024-09-20T22:44:36.026 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:44:36.026 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.026 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.026 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:44:36.026 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.341251+0000", 2024-09-20T22:44:36.026 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.7299999999999996e-07 2024-09-20T22:44:36.027 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.027 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.027 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:44:36.027 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.390395+0000", 2024-09-20T22:44:36.027 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.049143539999999999 2024-09-20T22:44:36.027 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.027 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.027 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:44:36.027 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.390413+0000", 2024-09-20T22:44:36.027 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.8709999999999999e-05 2024-09-20T22:44:36.027 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.027 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.027 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:44:36.028 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.390423+0000", 2024-09-20T22:44:36.028 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.4660000000000005e-06 2024-09-20T22:44:36.028 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.028 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.028 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:44:36.028 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.390484+0000", 2024-09-20T22:44:36.028 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.0851999999999999e-05 2024-09-20T22:44:36.028 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.028 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.028 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:44:36.028 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.390504+0000", 2024-09-20T22:44:36.028 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9976999999999999e-05 2024-09-20T22:44:36.028 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.028 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.028 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:44:36.029 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.392875+0000", 2024-09-20T22:44:36.029 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0023713710000000002 2024-09-20T22:44:36.029 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.029 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.029 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:44:36.029 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.392912+0000", 2024-09-20T22:44:36.029 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.6831e-05 2024-09-20T22:44:36.029 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:44:36.029 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:44:36.029 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:44:36.029 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.029 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.029 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:739 7.0 7:05bb442d:::smithi08477203-24:head [sparse-read 0~2090002] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e143)", 2024-09-20T22:44:36.029 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:42.601960+0000", 2024-09-20T22:44:36.029 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 53.359198886999998, 2024-09-20T22:44:36.029 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0049875850000000001, 2024-09-20T22:44:36.030 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:44:36.030 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:44:36.030 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:44:36.030 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:44:36.030 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:44:36.030 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:44:36.030 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 739 2024-09-20T22:44:36.030 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.030 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:44:36.030 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.030 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:44:36.030 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.601960+0000", 2024-09-20T22:44:36.030 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:44:36.031 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.031 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.031 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:44:36.031 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.601960+0000", 2024-09-20T22:44:36.031 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:44:36.031 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.031 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.031 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:44:36.031 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.601960+0000", 2024-09-20T22:44:36.031 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.1899999999999998e-07 2024-09-20T22:44:36.031 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.031 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.031 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:44:36.031 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.601962+0000", 2024-09-20T22:44:36.031 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.663e-06 2024-09-20T22:44:36.032 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.032 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.032 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:44:36.032 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.601965+0000", 2024-09-20T22:44:36.032 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.6040000000000001e-06 2024-09-20T22:44:36.032 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.032 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.032 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:44:36.032 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.601968+0000", 2024-09-20T22:44:36.032 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.0130000000000002e-06 2024-09-20T22:44:36.032 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.032 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.032 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:44:36.032 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.606598+0000", 2024-09-20T22:44:36.032 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0046293860000000001 2024-09-20T22:44:36.033 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.033 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.033 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:44:36.033 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.606723+0000", 2024-09-20T22:44:36.033 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00012586800000000001 2024-09-20T22:44:36.033 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.033 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.033 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:44:36.033 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.606947+0000", 2024-09-20T22:44:36.033 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00022373200000000001 2024-09-20T22:44:36.033 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:44:36.033 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:44:36.033 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:44:36.034 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.034 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.034 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:786 7.17 e145/139 7:ed54065f:::smithi08477203-12:head v 145'119, mlcod=143'117)", 2024-09-20T22:44:36.034 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:44.345115+0000", 2024-09-20T22:44:36.034 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 51.616043556999998, 2024-09-20T22:44:36.034 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0098654899999999993, 2024-09-20T22:44:36.034 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:44:36.034 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:44:36.034 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:44:36.034 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:44:36.034 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.034 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:44:36.034 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.345115+0000", 2024-09-20T22:44:36.034 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:44:36.035 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.035 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.035 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:44:36.035 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.345115+0000", 2024-09-20T22:44:36.035 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:44:36.035 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.035 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.035 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:44:36.035 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.345116+0000", 2024-09-20T22:44:36.035 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.8299999999999995e-07 2024-09-20T22:44:36.035 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.035 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.035 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:44:36.035 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.347955+0000", 2024-09-20T22:44:36.035 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.002839028 2024-09-20T22:44:36.035 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.036 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.036 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:44:36.036 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.347977+0000", 2024-09-20T22:44:36.036 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.1897999999999999e-05 2024-09-20T22:44:36.036 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.036 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.036 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:44:36.036 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.347992+0000", 2024-09-20T22:44:36.036 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4761e-05 2024-09-20T22:44:36.036 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.036 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.036 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:44:36.036 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.348072+0000", 2024-09-20T22:44:36.036 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.0586000000000004e-05 2024-09-20T22:44:36.036 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.037 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.037 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:44:36.037 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.348097+0000", 2024-09-20T22:44:36.037 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.5026000000000001e-05 2024-09-20T22:44:36.037 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.037 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.037 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:44:36.037 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.354924+0000", 2024-09-20T22:44:36.037 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0068271240000000004 2024-09-20T22:44:36.037 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.037 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.037 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:44:36.037 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.354980+0000", 2024-09-20T22:44:36.037 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.6184000000000003e-05 2024-09-20T22:44:36.037 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:44:36.037 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:44:36.038 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:44:36.038 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.038 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.038 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:804 7.17 e145/139 7:ea720b85:::smithi08477203-26:head v 145'120, mlcod=143'118)", 2024-09-20T22:44:36.038 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:44.456907+0000", 2024-09-20T22:44:36.038 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 51.504251803000002, 2024-09-20T22:44:36.038 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0070184640000000003, 2024-09-20T22:44:36.038 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:44:36.038 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:44:36.038 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:44:36.038 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:44:36.038 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.038 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:44:36.038 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.456907+0000", 2024-09-20T22:44:36.039 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:44:36.039 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.039 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.039 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:44:36.039 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.456907+0000", 2024-09-20T22:44:36.039 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:44:36.039 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.039 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.039 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:44:36.039 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.456909+0000", 2024-09-20T22:44:36.039 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.384e-06 2024-09-20T22:44:36.039 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.039 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.040 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:44:36.040 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.458084+0000", 2024-09-20T22:44:36.040 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00117508 2024-09-20T22:44:36.040 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.040 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.040 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:44:36.040 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.458121+0000", 2024-09-20T22:44:36.040 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.6711999999999997e-05 2024-09-20T22:44:36.040 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.040 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.041 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:44:36.041 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.458135+0000", 2024-09-20T22:44:36.041 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4508999999999999e-05 2024-09-20T22:44:36.041 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.041 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.041 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:44:36.041 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.458226+0000", 2024-09-20T22:44:36.041 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.0107999999999999e-05 2024-09-20T22:44:36.041 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.041 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.041 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:44:36.041 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.458246+0000", 2024-09-20T22:44:36.041 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.0891999999999999e-05 2024-09-20T22:44:36.042 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.042 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.042 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:44:36.042 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.463881+0000", 2024-09-20T22:44:36.042 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0056345859999999996 2024-09-20T22:44:36.042 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.042 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.042 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:44:36.042 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.463925+0000", 2024-09-20T22:44:36.042 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.4193000000000002e-05 2024-09-20T22:44:36.042 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:44:36.042 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:44:36.042 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:44:36.043 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.043 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.043 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(osd.6.48:11 7.0 7:064f66ab:::smithi08477203-45:head [copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e154)", 2024-09-20T22:44:36.043 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:53.132246+0000", 2024-09-20T22:44:36.043 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 42.828912520000003, 2024-09-20T22:44:36.043 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0051662619999999996, 2024-09-20T22:44:36.043 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:44:36.043 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:44:36.043 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "reached pg", 2024-09-20T22:44:36.043 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:44:36.043 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.043 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:44:36.043 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:53.132246+0000", 2024-09-20T22:44:36.044 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:44:36.044 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.044 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.044 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:44:36.044 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:53.132246+0000", 2024-09-20T22:44:36.044 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:44:36.044 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.044 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.044 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:44:36.044 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:53.132249+0000", 2024-09-20T22:44:36.044 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.5170000000000002e-06 2024-09-20T22:44:36.044 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.044 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.044 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:44:36.044 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:53.132256+0000", 2024-09-20T22:44:36.045 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.6179999999999997e-06 2024-09-20T22:44:36.045 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.045 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.045 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:44:36.045 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:53.132271+0000", 2024-09-20T22:44:36.045 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.5231999999999999e-05 2024-09-20T22:44:36.045 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.045 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.045 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:44:36.045 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:53.132283+0000", 2024-09-20T22:44:36.045 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1518000000000001e-05 2024-09-20T22:44:36.045 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.045 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.045 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:44:36.046 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:53.132359+0000", 2024-09-20T22:44:36.046 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.6520999999999995e-05 2024-09-20T22:44:36.046 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.046 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.046 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op must wait for map", 2024-09-20T22:44:36.046 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:53.132369+0000", 2024-09-20T22:44:36.046 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.8749999999999995e-06 2024-09-20T22:44:36.046 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.046 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.046 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:44:36.046 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:53.137300+0000", 2024-09-20T22:44:36.046 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0049310229999999997 2024-09-20T22:44:36.046 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.046 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.047 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:44:36.047 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:53.137412+0000", 2024-09-20T22:44:36.047 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000111958 2024-09-20T22:44:36.047 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:44:36.047 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:44:36.047 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:44:36.047 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.047 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.047 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:894 7.a e155/94 7:50097c2d:::smithi08477203-23:head v 155'54, mlcod=143'50)", 2024-09-20T22:44:36.047 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:54.450745+0000", 2024-09-20T22:44:36.047 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 41.510413469, 2024-09-20T22:44:36.047 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0091901029999999998, 2024-09-20T22:44:36.047 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:44:36.047 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:44:36.047 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:44:36.048 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:44:36.048 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.048 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:44:36.048 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:54.450745+0000", 2024-09-20T22:44:36.048 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:44:36.048 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.048 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.048 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:44:36.048 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:54.450745+0000", 2024-09-20T22:44:36.048 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:44:36.048 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.048 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.048 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:44:36.048 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:54.450745+0000", 2024-09-20T22:44:36.048 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.4200000000000002e-07 2024-09-20T22:44:36.049 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.049 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.049 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:44:36.049 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:54.456336+0000", 2024-09-20T22:44:36.049 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.005590589 2024-09-20T22:44:36.049 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.049 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.049 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:44:36.049 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:54.456353+0000", 2024-09-20T22:44:36.049 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.7187e-05 2024-09-20T22:44:36.049 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.049 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.049 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:44:36.049 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:54.456362+0000", 2024-09-20T22:44:36.050 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.7409999999999998e-06 2024-09-20T22:44:36.050 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.050 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.050 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:44:36.050 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:54.456418+0000", 2024-09-20T22:44:36.050 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.6431000000000002e-05 2024-09-20T22:44:36.050 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.050 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.050 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:44:36.050 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:54.456438+0000", 2024-09-20T22:44:36.050 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9718999999999999e-05 2024-09-20T22:44:36.050 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.050 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.051 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:44:36.051 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:54.459899+0000", 2024-09-20T22:44:36.051 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.003461254 2024-09-20T22:44:36.051 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.051 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.051 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:44:36.051 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:54.459935+0000", 2024-09-20T22:44:36.051 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.5939999999999998e-05 2024-09-20T22:44:36.051 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:44:36.051 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:44:36.051 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:44:36.051 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.052 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.052 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:901 7.e e156/94 7:7c62ced4:::smithi08477203-17:head v 156'42, mlcod=154'40)", 2024-09-20T22:44:36.052 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:55.635897+0000", 2024-09-20T22:44:36.052 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 40.325261075999997, 2024-09-20T22:44:36.052 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0082336760000000005, 2024-09-20T22:44:36.052 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:44:36.052 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:44:36.052 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:44:36.052 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:44:36.052 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.052 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:44:36.052 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:55.635897+0000", 2024-09-20T22:44:36.052 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:44:36.052 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.052 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.053 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:44:36.053 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:55.635897+0000", 2024-09-20T22:44:36.053 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:44:36.053 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.053 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.053 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:44:36.053 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:55.635898+0000", 2024-09-20T22:44:36.053 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.23e-07 2024-09-20T22:44:36.053 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.053 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.053 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:44:36.053 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:55.640084+0000", 2024-09-20T22:44:36.053 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0041865499999999998 2024-09-20T22:44:36.053 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.053 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.054 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:44:36.054 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:55.640118+0000", 2024-09-20T22:44:36.054 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.4093e-05 2024-09-20T22:44:36.054 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.054 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.054 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:44:36.054 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:55.640129+0000", 2024-09-20T22:44:36.054 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0281e-05 2024-09-20T22:44:36.054 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.054 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.054 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:44:36.054 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:55.640230+0000", 2024-09-20T22:44:36.054 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000101538 2024-09-20T22:44:36.055 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.055 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.055 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:44:36.055 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:55.640251+0000", 2024-09-20T22:44:36.055 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.1293000000000001e-05 2024-09-20T22:44:36.055 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.055 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.055 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:44:36.055 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:55.644088+0000", 2024-09-20T22:44:36.055 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.003836178 2024-09-20T22:44:36.055 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.055 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.055 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:44:36.055 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:55.644131+0000", 2024-09-20T22:44:36.055 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.3519999999999997e-05 2024-09-20T22:44:36.056 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:44:36.056 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:44:36.056 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:44:36.056 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.056 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.056 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:1034 7.b e166/94 7:d960cad4:::smithi08477203-18:head v 166'64, mlcod=163'61)", 2024-09-20T22:44:36.056 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:44:05.166370+0000", 2024-09-20T22:44:36.056 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 30.794788184000002, 2024-09-20T22:44:36.056 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0065985330000000002, 2024-09-20T22:44:36.056 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:44:36.056 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:44:36.056 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:44:36.056 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:44:36.056 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.057 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:44:36.057 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:05.166370+0000", 2024-09-20T22:44:36.057 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:44:36.057 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.057 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.057 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:44:36.057 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:05.166370+0000", 2024-09-20T22:44:36.057 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:44:36.057 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.057 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.057 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:44:36.057 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:05.166371+0000", 2024-09-20T22:44:36.057 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.2900000000000003e-07 2024-09-20T22:44:36.057 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.058 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.058 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:44:36.058 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:05.170473+0000", 2024-09-20T22:44:36.058 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0041020919999999999 2024-09-20T22:44:36.058 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.058 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.058 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:44:36.058 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:05.170486+0000", 2024-09-20T22:44:36.058 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2847000000000001e-05 2024-09-20T22:44:36.058 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.058 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.058 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:44:36.058 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:05.170493+0000", 2024-09-20T22:44:36.059 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.6630000000000004e-06 2024-09-20T22:44:36.059 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.059 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.059 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:44:36.059 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:05.170546+0000", 2024-09-20T22:44:36.059 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.3547000000000001e-05 2024-09-20T22:44:36.059 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.059 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.059 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:44:36.059 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:05.170562+0000", 2024-09-20T22:44:36.059 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.5515e-05 2024-09-20T22:44:36.059 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.059 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.059 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:44:36.060 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:05.172925+0000", 2024-09-20T22:44:36.060 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.002363633 2024-09-20T22:44:36.060 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.060 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.060 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:44:36.060 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:05.172969+0000", 2024-09-20T22:44:36.060 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.3606999999999998e-05 2024-09-20T22:44:36.060 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:44:36.060 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:44:36.060 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:44:36.060 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.060 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.060 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:1286 7.2 7:45b54b6b:::smithi08477203-2:head [write 578207~550227,stat] snapc 2a=[2a] ondisk+write+known_if_redirected+supports_pool_eio e179)", 2024-09-20T22:44:36.060 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:44:18.697522+0000", 2024-09-20T22:44:36.060 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 17.263636160000001, 2024-09-20T22:44:36.061 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0059766469999999999, 2024-09-20T22:44:36.061 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:44:36.061 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:44:36.061 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:44:36.061 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:44:36.061 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:44:36.061 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:44:36.061 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 1286 2024-09-20T22:44:36.061 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.061 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:44:36.061 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.061 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:44:36.061 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:18.697522+0000", 2024-09-20T22:44:36.061 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:44:36.061 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.061 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.062 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:44:36.062 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:18.697522+0000", 2024-09-20T22:44:36.062 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:44:36.062 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.062 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.062 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:44:36.062 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:18.697525+0000", 2024-09-20T22:44:36.062 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.2500000000000001e-06 2024-09-20T22:44:36.062 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.062 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.062 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:44:36.063 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:18.699052+0000", 2024-09-20T22:44:36.063 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0015269109999999999 2024-09-20T22:44:36.063 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.063 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.063 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:44:36.063 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:18.699074+0000", 2024-09-20T22:44:36.063 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.2478000000000001e-05 2024-09-20T22:44:36.063 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.063 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.063 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:44:36.063 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:18.699088+0000", 2024-09-20T22:44:36.063 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4314000000000001e-05 2024-09-20T22:44:36.063 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.064 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.064 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:44:36.064 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:18.699162+0000", 2024-09-20T22:44:36.064 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.4085000000000006e-05 2024-09-20T22:44:36.064 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.064 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.064 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:44:36.064 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:18.699237+0000", 2024-09-20T22:44:36.064 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.4561999999999995e-05 2024-09-20T22:44:36.064 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.064 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.064 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 1,2", 2024-09-20T22:44:36.064 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:18.699404+0000", 2024-09-20T22:44:36.064 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000166885 2024-09-20T22:44:36.065 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.065 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.065 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:44:36.065 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:18.700707+0000", 2024-09-20T22:44:36.065 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0013030240000000001 2024-09-20T22:44:36.065 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.065 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.065 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:44:36.065 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:18.703183+0000", 2024-09-20T22:44:36.065 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0024764269999999998 2024-09-20T22:44:36.065 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.065 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.065 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:44:36.065 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:18.703418+0000", 2024-09-20T22:44:36.066 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00023483900000000001 2024-09-20T22:44:36.066 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.066 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.066 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:44:36.066 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:18.703457+0000", 2024-09-20T22:44:36.066 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.8554999999999999e-05 2024-09-20T22:44:36.066 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.066 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.066 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:44:36.066 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:18.703499+0000", 2024-09-20T22:44:36.066 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.2317000000000002e-05 2024-09-20T22:44:36.066 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:44:36.066 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:44:36.066 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:44:36.067 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.067 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.067 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:1316 7.0 7:05bb442d:::smithi08477203-24:head [stat,delete] snapc 2b=[2b,2a] ondisk+write+known_if_redirected+supports_pool_eio e180)", 2024-09-20T22:44:36.067 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:44:19.225492+0000", 2024-09-20T22:44:36.067 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 16.735666505000001, 2024-09-20T22:44:36.067 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0083403250000000009, 2024-09-20T22:44:36.067 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:44:36.067 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:44:36.067 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:44:36.067 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:44:36.067 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:44:36.067 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:44:36.067 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 1316 2024-09-20T22:44:36.067 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.068 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:44:36.068 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.068 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:44:36.068 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:19.225492+0000", 2024-09-20T22:44:36.068 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:44:36.068 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.068 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.068 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:44:36.068 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:19.225492+0000", 2024-09-20T22:44:36.068 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:44:36.068 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.068 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.068 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:44:36.069 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:19.225495+0000", 2024-09-20T22:44:36.069 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.96e-06 2024-09-20T22:44:36.069 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.069 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.069 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:44:36.069 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:19.225508+0000", 2024-09-20T22:44:36.069 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3343000000000001e-05 2024-09-20T22:44:36.069 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.069 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.069 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:44:36.069 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:19.225542+0000", 2024-09-20T22:44:36.069 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.3973000000000002e-05 2024-09-20T22:44:36.069 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.069 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.070 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:44:36.070 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:19.225570+0000", 2024-09-20T22:44:36.070 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.7713999999999999e-05 2024-09-20T22:44:36.070 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.070 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.070 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:44:36.070 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:19.225655+0000", 2024-09-20T22:44:36.070 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.4944999999999994e-05 2024-09-20T22:44:36.070 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.070 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.070 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op must wait for map", 2024-09-20T22:44:36.070 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:19.225660+0000", 2024-09-20T22:44:36.070 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.5470000000000004e-06 2024-09-20T22:44:36.070 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.071 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.071 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:44:36.071 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:19.232618+0000", 2024-09-20T22:44:36.071 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0069571069999999997 2024-09-20T22:44:36.071 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.071 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.071 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:44:36.071 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:19.232717+0000", 2024-09-20T22:44:36.071 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.9306000000000001e-05 2024-09-20T22:44:36.071 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:36.071 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:36.071 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:44:36.071 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:19.233832+0000", 2024-09-20T22:44:36.072 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0011154299999999999 2024-09-20T22:44:36.072 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:44:36.072 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:44:36.072 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:44:36.072 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:44:36.072 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:44:36.072 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-20T22:44:36.072 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:44:36.093 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:36.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:36.094 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:44:36.195 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:44:36.200 INFO:tasks.rados.rados.0.smithi084.stdout:562: done (2 left) 2024-09-20T22:44:36.200 INFO:tasks.rados.rados.0.smithi084.stdout:563: done (1 left) 2024-09-20T22:44:36.200 INFO:tasks.rados.rados.0.smithi084.stdout:564: done (0 left) 2024-09-20T22:44:36.200 INFO:tasks.rados.rados.0.smithi084.stdout:565: read oid 22 snap 49 2024-09-20T22:44:36.200 INFO:tasks.rados.rados.0.smithi084.stdout:565: expect (ObjNum 217 snap 45 seq_num 217) 2024-09-20T22:44:36.200 INFO:tasks.rados.rados.0.smithi084.stdout:566: rollback oid 7 current snap is 50 2024-09-20T22:44:36.200 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 7 to 49 2024-09-20T22:44:36.200 INFO:tasks.rados.rados.0.smithi084.stdout:567: copy_from oid 28 from oid 40 current snap is 50 2024-09-20T22:44:36.201 INFO:tasks.rados.rados.0.smithi084.stdout:568: copy_from oid 26 from oid 2 current snap is 50 2024-09-20T22:44:36.201 INFO:tasks.rados.rados.0.smithi084.stdout:569: read oid 17 snap -1 2024-09-20T22:44:36.201 INFO:tasks.rados.rados.0.smithi084.stdout:569: expect (ObjNum 203 snap 41 seq_num 203) 2024-09-20T22:44:36.201 INFO:tasks.rados.rados.0.smithi084.stdout:570: rollback oid 50 current snap is 50 2024-09-20T22:44:36.201 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 50 to 49 2024-09-20T22:44:36.201 INFO:tasks.rados.rados.0.smithi084.stdout:571: write (excl) oid 49 current snap is 50 2024-09-20T22:44:36.201 INFO:tasks.rados.rados.0.smithi084.stdout:571: seq_num 232 ranges {699224=548358,2026293=462074,3271546=432754} 2024-09-20T22:44:36.207 INFO:tasks.rados.rados.0.smithi084.stdout:571: writing smithi08477203-49 from 699224 to 1247582 tid 1 2024-09-20T22:44:36.214 INFO:tasks.rados.rados.0.smithi084.stdout:571: writing smithi08477203-49 from 2026293 to 2488367 tid 2 2024-09-20T22:44:36.222 INFO:tasks.rados.rados.0.smithi084.stdout:571: writing smithi08477203-49 from 3271546 to 3704300 tid 3 2024-09-20T22:44:36.222 INFO:tasks.rados.rados.0.smithi084.stdout:572: snap_create 2024-09-20T22:44:36.222 INFO:tasks.rados.rados.0.smithi084.stdout:566: finishing rollback tid 0 to smithi08477203-7 2024-09-20T22:44:36.222 INFO:tasks.rados.rados.0.smithi084.stdout:566: finishing rollback tid 1 to smithi08477203-7 2024-09-20T22:44:36.222 INFO:tasks.rados.rados.0.smithi084.stdout:566: finishing rollback tid 2 to smithi08477203-7 2024-09-20T22:44:36.222 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 7 v 101 (ObjNum 150 snap 30 seq_num 150) dirty exists 2024-09-20T22:44:36.222 INFO:tasks.rados.rados.0.smithi084.stdout:569: expect (ObjNum 203 snap 41 seq_num 203) 2024-09-20T22:44:36.230 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:36.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:36.230 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:44:36.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:36 smithi154 ceph-mon[33264]: 7.c scrub starts 2024-09-20T22:44:36.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:36 smithi154 ceph-mon[33264]: 7.c scrub ok 2024-09-20T22:44:36.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:36 smithi154 ceph-mon[33264]: 1.10 scrub starts 2024-09-20T22:44:36.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:36 smithi154 ceph-mon[33264]: 1.10 scrub ok 2024-09-20T22:44:36.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:36 smithi154 ceph-mon[33264]: 7.f scrub starts 2024-09-20T22:44:36.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:36 smithi154 ceph-mon[33264]: 7.f scrub ok 2024-09-20T22:44:36.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:36 smithi154 ceph-mon[33264]: 3.0 scrub starts 2024-09-20T22:44:36.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:36 smithi154 ceph-mon[33264]: 3.0 scrub ok 2024-09-20T22:44:36.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:36 smithi154 ceph-mon[33264]: osdmap e196: 8 total, 8 up, 7 in 2024-09-20T22:44:36.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:36 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/1536775458' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:44:36.331 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:44:36.352 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:36.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:36.353 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:44:36.453 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:44:36.460 INFO:tasks.rados.rados.0.smithi084.stdout:571: finishing write tid 1 to smithi08477203-49 2024-09-20T22:44:36.460 INFO:tasks.rados.rados.0.smithi084.stdout:568: finishing copy_from racing read to smithi08477203-26 2024-09-20T22:44:36.460 INFO:tasks.rados.rados.0.smithi084.stdout:570: finishing rollback tid 0 to smithi08477203-50 2024-09-20T22:44:36.460 INFO:tasks.rados.rados.0.smithi084.stdout:565: expect (ObjNum 217 snap 45 seq_num 217) 2024-09-20T22:44:36.474 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:36.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:36.475 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:44:36.573 INFO:tasks.rados.rados.0.smithi084.stdout:570: finishing rollback tid 1 to smithi08477203-50 2024-09-20T22:44:36.573 INFO:tasks.rados.rados.0.smithi084.stdout:571: finishing write tid 2 to smithi08477203-49 2024-09-20T22:44:36.573 INFO:tasks.rados.rados.0.smithi084.stdout:570: finishing rollback tid 2 to smithi08477203-50 2024-09-20T22:44:36.573 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 50 v 50 (ObjNum 151 snap 30 seq_num 151) dirty exists 2024-09-20T22:44:36.573 INFO:tasks.rados.rados.0.smithi084.stdout:568: finishing copy_from to smithi08477203-26 2024-09-20T22:44:36.573 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 26 v 83 (ObjNum 201 snap 41 seq_num 201) dirty exists 2024-09-20T22:44:36.573 INFO:tasks.rados.rados.0.smithi084.stdout:567: finishing copy_from racing read to smithi08477203-28 2024-09-20T22:44:36.573 INFO:tasks.rados.rados.0.smithi084.stdout:567: finishing copy_from to smithi08477203-28 2024-09-20T22:44:36.573 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 28 v 127 (ObjNum 231 snap 49 seq_num 231) dirty exists 2024-09-20T22:44:36.574 INFO:tasks.rados.rados.0.smithi084.stdout:571: finishing write tid 3 to smithi08477203-49 2024-09-20T22:44:36.574 INFO:tasks.rados.rados.0.smithi084.stdout:571: finishing write tid 4 to smithi08477203-49 2024-09-20T22:44:36.574 INFO:tasks.rados.rados.0.smithi084.stdout:571: finishing write tid 5 to smithi08477203-49 2024-09-20T22:44:36.574 INFO:tasks.rados.rados.0.smithi084.stdout:571: oid 49 updating version 0 to 169 2024-09-20T22:44:36.574 INFO:tasks.rados.rados.0.smithi084.stdout:571: oid 49 updating version 169 to 171 2024-09-20T22:44:36.574 INFO:tasks.rados.rados.0.smithi084.stdout:571: oid 49 updating version 171 to 172 2024-09-20T22:44:36.574 INFO:tasks.rados.rados.0.smithi084.stdout:571: oid 49 version 172 is already newer than 170 2024-09-20T22:44:36.574 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 49 v 172 (ObjNum 232 snap 50 seq_num 232) dirty exists 2024-09-20T22:44:36.574 INFO:tasks.rados.rados.0.smithi084.stdout:571: left oid 49 (ObjNum 232 snap 50 seq_num 232) 2024-09-20T22:44:36.576 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:44:36.597 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:36.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:36.597 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:44:36.614 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:36 smithi084 ceph-mon[40024]: 7.c scrub starts 2024-09-20T22:44:36.614 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:36 smithi084 ceph-mon[40024]: 7.c scrub ok 2024-09-20T22:44:36.614 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:36 smithi084 ceph-mon[40024]: 1.10 scrub starts 2024-09-20T22:44:36.614 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:36 smithi084 ceph-mon[40024]: 1.10 scrub ok 2024-09-20T22:44:36.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:36 smithi084 ceph-mon[40024]: 7.f scrub starts 2024-09-20T22:44:36.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:36 smithi084 ceph-mon[40024]: 7.f scrub ok 2024-09-20T22:44:36.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:36 smithi084 ceph-mon[40024]: 3.0 scrub starts 2024-09-20T22:44:36.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:36 smithi084 ceph-mon[40024]: 3.0 scrub ok 2024-09-20T22:44:36.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:36 smithi084 ceph-mon[40024]: osdmap e196: 8 total, 8 up, 7 in 2024-09-20T22:44:36.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:36 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/1536775458' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:44:36.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:36 smithi084 ceph-mon[29774]: 7.c scrub starts 2024-09-20T22:44:36.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:36 smithi084 ceph-mon[29774]: 7.c scrub ok 2024-09-20T22:44:36.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:36 smithi084 ceph-mon[29774]: 1.10 scrub starts 2024-09-20T22:44:36.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:36 smithi084 ceph-mon[29774]: 1.10 scrub ok 2024-09-20T22:44:36.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:36 smithi084 ceph-mon[29774]: 7.f scrub starts 2024-09-20T22:44:36.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:36 smithi084 ceph-mon[29774]: 7.f scrub ok 2024-09-20T22:44:36.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:36 smithi084 ceph-mon[29774]: 3.0 scrub starts 2024-09-20T22:44:36.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:36 smithi084 ceph-mon[29774]: 3.0 scrub ok 2024-09-20T22:44:36.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:36 smithi084 ceph-mon[29774]: osdmap e196: 8 total, 8 up, 7 in 2024-09-20T22:44:36.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:36 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/1536775458' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:44:36.698 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:44:36.719 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:36.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:36.720 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:44:36.821 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:44:36.842 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:36.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:36.843 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:44:36.944 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:44:36.966 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:36.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:36.967 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:44:36.998 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-20T22:44:37.068 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:44:37.091 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:37.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:37.092 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:44:37.192 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:44:37.210 INFO:tasks.rados.rados.0.smithi084.stdout:565: done (7 left) 2024-09-20T22:44:37.210 INFO:tasks.rados.rados.0.smithi084.stdout:566: done (6 left) 2024-09-20T22:44:37.210 INFO:tasks.rados.rados.0.smithi084.stdout:567: done (5 left) 2024-09-20T22:44:37.210 INFO:tasks.rados.rados.0.smithi084.stdout:568: done (4 left) 2024-09-20T22:44:37.210 INFO:tasks.rados.rados.0.smithi084.stdout:569: done (3 left) 2024-09-20T22:44:37.211 INFO:tasks.rados.rados.0.smithi084.stdout:570: done (2 left) 2024-09-20T22:44:37.211 INFO:tasks.rados.rados.0.smithi084.stdout:571: done (1 left) 2024-09-20T22:44:37.211 INFO:tasks.rados.rados.0.smithi084.stdout:572: done (0 left) 2024-09-20T22:44:37.211 INFO:tasks.rados.rados.0.smithi084.stdout:573: rollback oid 25 current snap is 51 2024-09-20T22:44:37.211 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 25 to 49 2024-09-20T22:44:37.211 INFO:tasks.rados.rados.0.smithi084.stdout:574: copy_from oid 49 from oid 36 current snap is 51 2024-09-20T22:44:37.211 INFO:tasks.rados.rados.0.smithi084.stdout:575: delete oid 9 current snap is 51 2024-09-20T22:44:37.212 INFO:tasks.rados.rados.0.smithi084.stdout:573: finishing rollback tid 1 to smithi08477203-25 2024-09-20T22:44:37.212 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 25 v 0 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-09-20T22:44:37.215 INFO:tasks.rados.rados.0.smithi084.stdout:573: done (2 left) 2024-09-20T22:44:37.215 INFO:tasks.rados.rados.0.smithi084.stdout:575: done (1 left) 2024-09-20T22:44:37.215 INFO:tasks.rados.rados.0.smithi084.stdout:576: write oid 27 current snap is 51 2024-09-20T22:44:37.215 INFO:tasks.rados.rados.0.smithi084.stdout:576: seq_num 234 ranges {732838=698856,2026904=465743,3210039=571651} 2024-09-20T22:44:37.225 INFO:tasks.rados.rados.0.smithi084.stdout:576: writing smithi08477203-27 from 732838 to 1431694 tid 1 2024-09-20T22:44:37.229 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:37.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:37.230 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:44:37.231 INFO:tasks.rados.rados.0.smithi084.stdout:576: writing smithi08477203-27 from 2026904 to 2492647 tid 2 2024-09-20T22:44:37.240 INFO:tasks.rados.rados.0.smithi084.stdout:576: writing smithi08477203-27 from 3210039 to 3781690 tid 3 2024-09-20T22:44:37.240 INFO:tasks.rados.rados.0.smithi084.stdout:577: copy_from oid 12 from oid 8 current snap is 51 2024-09-20T22:44:37.240 INFO:tasks.rados.rados.0.smithi084.stdout:574: finishing copy_from racing read to smithi08477203-49 2024-09-20T22:44:37.240 INFO:tasks.rados.rados.0.smithi084.stdout:574: finishing copy_from to smithi08477203-49 2024-09-20T22:44:37.240 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 49 v 69 (ObjNum 35 snap 0 seq_num 35) dirty exists 2024-09-20T22:44:37.240 INFO:tasks.rados.rados.0.smithi084.stdout:576: finishing write tid 1 to smithi08477203-27 2024-09-20T22:44:37.241 INFO:tasks.rados.rados.0.smithi084.stdout:576: finishing write tid 2 to smithi08477203-27 2024-09-20T22:44:37.241 INFO:tasks.rados.rados.0.smithi084.stdout:574: done (2 left) 2024-09-20T22:44:37.241 INFO:tasks.rados.rados.0.smithi084.stdout:578: rollback oid 24 current snap is 51 2024-09-20T22:44:37.241 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 24 to 50 2024-09-20T22:44:37.241 INFO:tasks.rados.rados.0.smithi084.stdout:579: read oid 21 snap 50 2024-09-20T22:44:37.241 INFO:tasks.rados.rados.0.smithi084.stdout:579: expect (ObjNum 227 snap 48 seq_num 227) 2024-09-20T22:44:37.241 INFO:tasks.rados.rados.0.smithi084.stdout:580: read oid 28 snap -1 2024-09-20T22:44:37.242 INFO:tasks.rados.rados.0.smithi084.stdout:580: expect (ObjNum 231 snap 49 seq_num 231) 2024-09-20T22:44:37.242 INFO:tasks.rados.rados.0.smithi084.stdout:581: read oid 47 snap -1 2024-09-20T22:44:37.242 INFO:tasks.rados.rados.0.smithi084.stdout:581: expect (ObjNum 208 snap 42 seq_num 208) 2024-09-20T22:44:37.242 INFO:tasks.rados.rados.0.smithi084.stdout:582: delete oid 38 current snap is 51 2024-09-20T22:44:37.242 INFO:tasks.rados.rados.0.smithi084.stdout:578: finishing rollback tid 1 to smithi08477203-24 2024-09-20T22:44:37.242 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 24 v 0 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-09-20T22:44:37.245 INFO:tasks.rados.rados.0.smithi084.stdout:576: finishing write tid 3 to smithi08477203-27 2024-09-20T22:44:37.245 INFO:tasks.rados.rados.0.smithi084.stdout:576: finishing write tid 4 to smithi08477203-27 2024-09-20T22:44:37.245 INFO:tasks.rados.rados.0.smithi084.stdout:576: finishing write tid 5 to smithi08477203-27 2024-09-20T22:44:37.246 INFO:tasks.rados.rados.0.smithi084.stdout:576: oid 27 updating version 0 to 161 2024-09-20T22:44:37.246 INFO:tasks.rados.rados.0.smithi084.stdout:576: oid 27 version 161 is already newer than 158 2024-09-20T22:44:37.246 INFO:tasks.rados.rados.0.smithi084.stdout:576: oid 27 version 161 is already newer than 159 2024-09-20T22:44:37.246 INFO:tasks.rados.rados.0.smithi084.stdout:576: oid 27 version 161 is already newer than 160 2024-09-20T22:44:37.246 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 27 v 161 (ObjNum 234 snap 51 seq_num 234) dirty exists 2024-09-20T22:44:37.246 INFO:tasks.rados.rados.0.smithi084.stdout:576: left oid 27 (ObjNum 234 snap 51 seq_num 234) 2024-09-20T22:44:37.249 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-20T22:44:37.251 INFO:tasks.rados.rados.0.smithi084.stdout:577: finishing copy_from racing read to smithi08477203-12 2024-09-20T22:44:37.255 INFO:tasks.rados.rados.0.smithi084.stdout:577: finishing copy_from to smithi08477203-12 2024-09-20T22:44:37.255 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 12 v 88 (ObjNum 164 snap 35 seq_num 164) dirty exists 2024-09-20T22:44:37.259 INFO:tasks.rados.rados.0.smithi084.stdout:579: expect (ObjNum 227 snap 48 seq_num 227) 2024-09-20T22:44:37.307 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:44:37.331 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:44:37.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:37.358 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:37.358 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:44:37.364 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:37 smithi084 ceph-mon[40024]: pgmap v196: 178 pgs: 1 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 174 active+clean; 118 MiB data, 534 MiB used, 625 GiB / 626 GiB avail; 10 MiB/s rd, 5.7 MiB/s wr, 23 op/s 2024-09-20T22:44:37.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:37 smithi084 ceph-mon[40024]: 2.2 scrub starts 2024-09-20T22:44:37.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:37 smithi084 ceph-mon[40024]: 2.2 scrub ok 2024-09-20T22:44:37.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:37 smithi084 ceph-mon[40024]: 1.0 scrub starts 2024-09-20T22:44:37.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:37 smithi084 ceph-mon[40024]: 1.0 scrub ok 2024-09-20T22:44:37.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:37 smithi084 ceph-mon[40024]: osdmap e197: 8 total, 8 up, 7 in 2024-09-20T22:44:37.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:37 smithi084 ceph-mon[40024]: 7.b scrub starts 2024-09-20T22:44:37.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:37 smithi084 ceph-mon[40024]: 7.b scrub ok 2024-09-20T22:44:37.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:37 smithi084 ceph-mon[29774]: pgmap v196: 178 pgs: 1 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 174 active+clean; 118 MiB data, 534 MiB used, 625 GiB / 626 GiB avail; 10 MiB/s rd, 5.7 MiB/s wr, 23 op/s 2024-09-20T22:44:37.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:37 smithi084 ceph-mon[29774]: 2.2 scrub starts 2024-09-20T22:44:37.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:37 smithi084 ceph-mon[29774]: 2.2 scrub ok 2024-09-20T22:44:37.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:37 smithi084 ceph-mon[29774]: 1.0 scrub starts 2024-09-20T22:44:37.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:37 smithi084 ceph-mon[29774]: 1.0 scrub ok 2024-09-20T22:44:37.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:37 smithi084 ceph-mon[29774]: osdmap e197: 8 total, 8 up, 7 in 2024-09-20T22:44:37.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:37 smithi084 ceph-mon[29774]: 7.b scrub starts 2024-09-20T22:44:37.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:37 smithi084 ceph-mon[29774]: 7.b scrub ok 2024-09-20T22:44:37.458 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:44:37.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:37.481 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:37.482 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:44:37.488 INFO:tasks.rados.rados.0.smithi084.stdout:581: expect (ObjNum 208 snap 42 seq_num 208) 2024-09-20T22:44:37.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:37 smithi154 ceph-mon[33264]: pgmap v196: 178 pgs: 1 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 174 active+clean; 118 MiB data, 534 MiB used, 625 GiB / 626 GiB avail; 10 MiB/s rd, 5.7 MiB/s wr, 23 op/s 2024-09-20T22:44:37.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:37 smithi154 ceph-mon[33264]: 2.2 scrub starts 2024-09-20T22:44:37.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:37 smithi154 ceph-mon[33264]: 2.2 scrub ok 2024-09-20T22:44:37.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:37 smithi154 ceph-mon[33264]: 1.0 scrub starts 2024-09-20T22:44:37.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:37 smithi154 ceph-mon[33264]: 1.0 scrub ok 2024-09-20T22:44:37.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:37 smithi154 ceph-mon[33264]: osdmap e197: 8 total, 8 up, 7 in 2024-09-20T22:44:37.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:37 smithi154 ceph-mon[33264]: 7.b scrub starts 2024-09-20T22:44:37.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:37 smithi154 ceph-mon[33264]: 7.b scrub ok 2024-09-20T22:44:37.582 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:44:37.603 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:37.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:37.604 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:44:37.614 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:44:37.705 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:44:37.718 INFO:tasks.rados.rados.0.smithi084.stdout:580: expect (ObjNum 231 snap 49 seq_num 231) 2024-09-20T22:44:37.727 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:37.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:37.728 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:44:37.829 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:44:37.850 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:37.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:37.851 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:44:37.951 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:44:37.972 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:37.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:37.973 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:44:38.039 INFO:tasks.rados.rados.0.smithi084.stdout:576: done (6 left) 2024-09-20T22:44:38.039 INFO:tasks.rados.rados.0.smithi084.stdout:577: done (5 left) 2024-09-20T22:44:38.039 INFO:tasks.rados.rados.0.smithi084.stdout:578: done (4 left) 2024-09-20T22:44:38.039 INFO:tasks.rados.rados.0.smithi084.stdout:579: done (3 left) 2024-09-20T22:44:38.039 INFO:tasks.rados.rados.0.smithi084.stdout:580: done (2 left) 2024-09-20T22:44:38.039 INFO:tasks.rados.rados.0.smithi084.stdout:581: done (1 left) 2024-09-20T22:44:38.039 INFO:tasks.rados.rados.0.smithi084.stdout:582: done (0 left) 2024-09-20T22:44:38.039 INFO:tasks.rados.rados.0.smithi084.stdout:583: read oid 35 snap 50 2024-09-20T22:44:38.039 INFO:tasks.rados.rados.0.smithi084.stdout:583: expect (ObjNum 150 snap 30 seq_num 150) 2024-09-20T22:44:38.040 INFO:tasks.rados.rados.0.smithi084.stdout:584: snap_remove snap 49 2024-09-20T22:44:38.074 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:44:38.097 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:38.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:38.098 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:44:38.139 INFO:tasks.rados.rados.0.smithi084.stdout:584: done (1 left) 2024-09-20T22:44:38.139 INFO:tasks.rados.rados.0.smithi084.stdout:585: copy_from oid 29 from oid 48 current snap is 51 2024-09-20T22:44:38.139 INFO:tasks.rados.rados.0.smithi084.stdout:583: expect (ObjNum 150 snap 30 seq_num 150) 2024-09-20T22:44:38.153 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-20T22:44:38.153 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [ 2024-09-20T22:44:38.153 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:38.153 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:1567 7.c 7:35bf785a:::smithi08477203-29:head [copy-from ver 94 in=55b] snapc 34=[34] ondisk+write+known_if_redirected+supports_pool_eio e199)", 2024-09-20T22:44:38.154 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:44:38.138541+0000", 2024-09-20T22:44:38.154 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 0.011812507, 2024-09-20T22:44:38.154 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.011843374, 2024-09-20T22:44:38.154 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:44:38.154 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:44:38.154 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "waiting for sub ops", 2024-09-20T22:44:38.154 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:44:38.154 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:44:38.154 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:44:38.154 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 1567 2024-09-20T22:44:38.154 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:38.155 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:44:38.155 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:38.155 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:44:38.155 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:38.138541+0000", 2024-09-20T22:44:38.155 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:44:38.155 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:38.155 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:38.155 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:44:38.155 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:38.138541+0000", 2024-09-20T22:44:38.155 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:44:38.156 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:38.156 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:38.156 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:44:38.156 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:38.138542+0000", 2024-09-20T22:44:38.156 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1000000000000001e-06 2024-09-20T22:44:38.156 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:38.156 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:38.156 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:44:38.156 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:38.138547+0000", 2024-09-20T22:44:38.156 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.0629999999999996e-06 2024-09-20T22:44:38.156 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:38.156 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:38.157 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:44:38.157 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:38.138556+0000", 2024-09-20T22:44:38.157 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.9279999999999999e-06 2024-09-20T22:44:38.157 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:38.157 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:38.157 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:44:38.157 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:38.138563+0000", 2024-09-20T22:44:38.157 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.323e-06 2024-09-20T22:44:38.157 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:38.157 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:38.157 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:44:38.157 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:38.138618+0000", 2024-09-20T22:44:38.157 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.4867999999999997e-05 2024-09-20T22:44:38.158 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:38.158 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:38.158 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op must wait for map", 2024-09-20T22:44:38.158 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:38.138626+0000", 2024-09-20T22:44:38.158 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.9980000000000003e-06 2024-09-20T22:44:38.158 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:38.158 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:38.158 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:44:38.158 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:38.140846+0000", 2024-09-20T22:44:38.158 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0022198209999999999 2024-09-20T22:44:38.158 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:38.158 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:38.158 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:44:38.158 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:38.140946+0000", 2024-09-20T22:44:38.159 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00010048900000000001 2024-09-20T22:44:38.159 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:38.159 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:38.159 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 3,7", 2024-09-20T22:44:38.159 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:38.147673+0000", 2024-09-20T22:44:38.159 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0067263039999999998 2024-09-20T22:44:38.159 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:44:38.159 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:44:38.159 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:44:38.159 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:38.159 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:38.159 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:1568 7.c 7:35bf785a:::smithi08477203-29:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e199)", 2024-09-20T22:44:38.160 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:44:38.138576+0000", 2024-09-20T22:44:38.160 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 0.011777078999999999, 2024-09-20T22:44:38.160 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.011891046000000001, 2024-09-20T22:44:38.160 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:44:38.160 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:44:38.160 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "waiting for blocked object", 2024-09-20T22:44:38.160 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:44:38.160 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:44:38.160 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:44:38.160 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 1568 2024-09-20T22:44:38.160 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:38.160 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:44:38.160 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:38.160 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:44:38.161 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:38.138576+0000", 2024-09-20T22:44:38.161 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:44:38.161 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:38.161 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:38.161 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:44:38.161 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:38.138576+0000", 2024-09-20T22:44:38.161 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:44:38.161 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:38.161 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:38.161 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:44:38.162 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:38.138577+0000", 2024-09-20T22:44:38.162 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.4999999999999998e-07 2024-09-20T22:44:38.162 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:38.162 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:38.162 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:44:38.162 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:38.138579+0000", 2024-09-20T22:44:38.162 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.165e-06 2024-09-20T22:44:38.162 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:38.162 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:38.163 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:44:38.163 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:38.138585+0000", 2024-09-20T22:44:38.163 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.9989999999999997e-06 2024-09-20T22:44:38.163 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:38.163 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:38.163 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:44:38.163 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:38.138588+0000", 2024-09-20T22:44:38.163 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.2559999999999998e-06 2024-09-20T22:44:38.163 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:38.164 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:38.164 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:44:38.164 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:38.138678+0000", 2024-09-20T22:44:38.164 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.9682e-05 2024-09-20T22:44:38.164 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:38.164 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:38.164 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting_for_map not empty", 2024-09-20T22:44:38.164 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:38.138685+0000", 2024-09-20T22:44:38.165 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.7410000000000003e-06 2024-09-20T22:44:38.165 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:38.165 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:38.165 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:44:38.165 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:38.141124+0000", 2024-09-20T22:44:38.165 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.002439578 2024-09-20T22:44:38.165 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:38.165 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:38.165 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for blocked object", 2024-09-20T22:44:38.166 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:38.141166+0000", 2024-09-20T22:44:38.166 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.1956999999999998e-05 2024-09-20T22:44:38.166 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:44:38.166 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:44:38.166 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:44:38.166 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:38.166 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:38.166 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop_reply(osd.6.0:494 7.c e199/94)", 2024-09-20T22:44:38.166 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:44:38.149457+0000", 2024-09-20T22:44:38.167 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 0.00089635699999999997, 2024-09-20T22:44:38.167 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0010833119999999999, 2024-09-20T22:44:38.167 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:44:38.167 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:44:38.167 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "queued for pg", 2024-09-20T22:44:38.167 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:44:38.167 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:38.167 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:44:38.167 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:38.149457+0000", 2024-09-20T22:44:38.168 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:44:38.168 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:38.168 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:38.168 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:44:38.168 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:38.149457+0000", 2024-09-20T22:44:38.168 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:44:38.168 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:38.168 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:38.169 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:44:38.169 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:38.149458+0000", 2024-09-20T22:44:38.169 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.2399999999999997e-07 2024-09-20T22:44:38.169 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:38.169 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:38.169 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:44:38.169 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:38.149463+0000", 2024-09-20T22:44:38.169 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.463e-06 2024-09-20T22:44:38.169 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:38.170 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:38.170 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:44:38.170 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:38.149473+0000", 2024-09-20T22:44:38.170 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0259e-05 2024-09-20T22:44:38.170 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:38.170 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:38.170 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:44:38.170 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:38.149480+0000", 2024-09-20T22:44:38.171 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.6839999999999999e-06 2024-09-20T22:44:38.171 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:44:38.171 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:44:38.171 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:44:38.171 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:44:38.171 INFO:teuthology.orchestra.run.smithi154.stdout: ], 2024-09-20T22:44:38.171 INFO:teuthology.orchestra.run.smithi154.stdout: "num_ops": 3 2024-09-20T22:44:38.171 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-20T22:44:38.199 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:44:38.220 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:38.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:38.221 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:44:38.273 INFO:tasks.rados.rados.0.smithi084.stdout:585: finishing copy_from racing read to smithi08477203-29 2024-09-20T22:44:38.273 INFO:tasks.rados.rados.0.smithi084.stdout:583: done (1 left) 2024-09-20T22:44:38.273 INFO:tasks.rados.rados.0.smithi084.stdout:586: snap_remove snap 50 2024-09-20T22:44:38.322 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:44:38.343 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:38.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:38.344 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:44:38.364 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:44:38 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: ::ffff:172.21.15.154 - - [20/Sep/2024:22:44:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-20T22:44:38.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:38 smithi084 ceph-mon[40024]: 5.9 scrub starts 2024-09-20T22:44:38.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:38 smithi084 ceph-mon[40024]: 5.9 scrub ok 2024-09-20T22:44:38.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:38 smithi084 ceph-mon[40024]: osdmap e198: 8 total, 8 up, 7 in 2024-09-20T22:44:38.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:38 smithi084 ceph-mon[40024]: 5.8 scrub starts 2024-09-20T22:44:38.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:38 smithi084 ceph-mon[40024]: 5.8 scrub ok 2024-09-20T22:44:38.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:38 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:44:38.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:38 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:44:38.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:38 smithi084 ceph-mon[40024]: 1.13 scrub starts 2024-09-20T22:44:38.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:38 smithi084 ceph-mon[40024]: 1.13 scrub ok 2024-09-20T22:44:38.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:38 smithi084 ceph-mon[40024]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:44:38.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:38 smithi084 ceph-mon[40024]: osdmap e199: 8 total, 8 up, 7 in 2024-09-20T22:44:38.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:38 smithi084 ceph-mon[29774]: 5.9 scrub starts 2024-09-20T22:44:38.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:38 smithi084 ceph-mon[29774]: 5.9 scrub ok 2024-09-20T22:44:38.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:38 smithi084 ceph-mon[29774]: osdmap e198: 8 total, 8 up, 7 in 2024-09-20T22:44:38.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:38 smithi084 ceph-mon[29774]: 5.8 scrub starts 2024-09-20T22:44:38.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:38 smithi084 ceph-mon[29774]: 5.8 scrub ok 2024-09-20T22:44:38.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:38 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:44:38.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:38 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:44:38.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:38 smithi084 ceph-mon[29774]: 1.13 scrub starts 2024-09-20T22:44:38.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:38 smithi084 ceph-mon[29774]: 1.13 scrub ok 2024-09-20T22:44:38.367 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:38 smithi084 ceph-mon[29774]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:44:38.367 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:38 smithi084 ceph-mon[29774]: osdmap e199: 8 total, 8 up, 7 in 2024-09-20T22:44:38.445 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:44:38.467 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:38.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:38.468 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:44:38.501 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:38 smithi154 ceph-mon[33264]: 5.9 scrub starts 2024-09-20T22:44:38.501 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:38 smithi154 ceph-mon[33264]: 5.9 scrub ok 2024-09-20T22:44:38.501 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:38 smithi154 ceph-mon[33264]: osdmap e198: 8 total, 8 up, 7 in 2024-09-20T22:44:38.501 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:38 smithi154 ceph-mon[33264]: 5.8 scrub starts 2024-09-20T22:44:38.501 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:38 smithi154 ceph-mon[33264]: 5.8 scrub ok 2024-09-20T22:44:38.501 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:38 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:44:38.501 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:38 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:44:38.501 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:38 smithi154 ceph-mon[33264]: 1.13 scrub starts 2024-09-20T22:44:38.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:38 smithi154 ceph-mon[33264]: 1.13 scrub ok 2024-09-20T22:44:38.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:38 smithi154 ceph-mon[33264]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:44:38.502 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:38 smithi154 ceph-mon[33264]: osdmap e199: 8 total, 8 up, 7 in 2024-09-20T22:44:38.569 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:44:38.590 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:38.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:38.591 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:44:38.640 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-20T22:44:38.691 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:44:38.713 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:38.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:38.714 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:44:38.814 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:44:38.835 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:38.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:38.837 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:44:38.875 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.6 dump_blocked_ops 2024-09-20T22:44:38.937 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:44:38.958 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:38.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:38.959 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:44:39.060 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:44:39.081 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:39.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:39.082 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:44:39.166 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:44:39.183 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:44:39.204 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:39.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:39.205 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:44:39.223 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:39 smithi084 ceph-mon[29774]: 4.3 scrub starts 2024-09-20T22:44:39.223 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:39 smithi084 ceph-mon[29774]: 4.3 scrub ok 2024-09-20T22:44:39.223 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:39 smithi084 ceph-mon[29774]: pgmap v199: 178 pgs: 1 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 174 active+clean; 118 MiB data, 534 MiB used, 625 GiB / 626 GiB avail 2024-09-20T22:44:39.223 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:39 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/2311950700' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:44:39.223 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:39 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:44:39.223 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:39 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:44:39.223 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:39 smithi084 ceph-mon[40024]: 4.3 scrub starts 2024-09-20T22:44:39.223 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:39 smithi084 ceph-mon[40024]: 4.3 scrub ok 2024-09-20T22:44:39.224 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:39 smithi084 ceph-mon[40024]: pgmap v199: 178 pgs: 1 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 174 active+clean; 118 MiB data, 534 MiB used, 625 GiB / 626 GiB avail 2024-09-20T22:44:39.224 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:39 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/2311950700' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:44:39.224 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:39 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:44:39.224 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:39 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:44:39.224 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is unset 2024-09-20T22:44:39.225 INFO:tasks.rados.rados.0.smithi084.stdout:586: done (1 left) 2024-09-20T22:44:39.225 INFO:tasks.rados.rados.0.smithi084.stdout:587: write oid 16 current snap is 51 2024-09-20T22:44:39.225 INFO:tasks.rados.rados.0.smithi084.stdout:587: seq_num 236 ranges {766452=565707,1860221=469412,2981238=594195} 2024-09-20T22:44:39.233 INFO:tasks.rados.rados.0.smithi084.stdout:587: writing smithi08477203-16 from 766452 to 1332159 tid 1 2024-09-20T22:44:39.239 INFO:tasks.rados.rados.0.smithi084.stdout:587: writing smithi08477203-16 from 1860221 to 2329633 tid 2 2024-09-20T22:44:39.247 INFO:tasks.rados.rados.0.smithi084.stdout:587: writing smithi08477203-16 from 2981238 to 3575433 tid 3 2024-09-20T22:44:39.247 INFO:tasks.rados.rados.0.smithi084.stdout:588: write oid 1 current snap is 51 2024-09-20T22:44:39.247 INFO:tasks.rados.rados.0.smithi084.stdout:585: finishing copy_from to smithi08477203-29 2024-09-20T22:44:39.247 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 29 v 94 (ObjNum 129 snap 23 seq_num 129) dirty exists 2024-09-20T22:44:39.247 INFO:tasks.rados.rados.0.smithi084.stdout:587: finishing write tid 1 to smithi08477203-16 2024-09-20T22:44:39.248 INFO:tasks.rados.rados.0.smithi084.stdout:587: finishing write tid 2 to smithi08477203-16 2024-09-20T22:44:39.248 INFO:tasks.rados.rados.0.smithi084.stdout:588: seq_num 237 ranges {783259=640956,1918703=613070,3008661=405467,3821939=161320} 2024-09-20T22:44:39.256 INFO:tasks.rados.rados.0.smithi084.stdout:588: writing smithi08477203-1 from 783259 to 1424215 tid 1 2024-09-20T22:44:39.263 INFO:tasks.rados.rados.0.smithi084.stdout:588: writing smithi08477203-1 from 1918703 to 2531773 tid 2 2024-09-20T22:44:39.268 INFO:tasks.rados.rados.0.smithi084.stdout:588: writing smithi08477203-1 from 3008661 to 3414128 tid 3 2024-09-20T22:44:39.272 INFO:tasks.rados.rados.0.smithi084.stdout:588: writing smithi08477203-1 from 3821939 to 3983259 tid 4 2024-09-20T22:44:39.272 INFO:tasks.rados.rados.0.smithi084.stdout:585: done (2 left) 2024-09-20T22:44:39.272 INFO:tasks.rados.rados.0.smithi084.stdout:589: snap_create 2024-09-20T22:44:39.272 INFO:tasks.rados.rados.0.smithi084.stdout:587: finishing write tid 3 to smithi08477203-16 2024-09-20T22:44:39.272 INFO:tasks.rados.rados.0.smithi084.stdout:587: finishing write tid 4 to smithi08477203-16 2024-09-20T22:44:39.272 INFO:tasks.rados.rados.0.smithi084.stdout:587: finishing write tid 5 to smithi08477203-16 2024-09-20T22:44:39.272 INFO:tasks.rados.rados.0.smithi084.stdout:587: oid 16 updating version 0 to 130 2024-09-20T22:44:39.272 INFO:tasks.rados.rados.0.smithi084.stdout:587: oid 16 version 130 is already newer than 129 2024-09-20T22:44:39.272 INFO:tasks.rados.rados.0.smithi084.stdout:587: oid 16 updating version 130 to 131 2024-09-20T22:44:39.272 INFO:tasks.rados.rados.0.smithi084.stdout:587: oid 16 updating version 131 to 132 2024-09-20T22:44:39.273 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 16 v 132 (ObjNum 236 snap 51 seq_num 236) dirty exists 2024-09-20T22:44:39.273 INFO:tasks.rados.rados.0.smithi084.stdout:587: left oid 16 (ObjNum 236 snap 51 seq_num 236) 2024-09-20T22:44:39.273 INFO:tasks.rados.rados.0.smithi084.stdout:588: finishing write tid 1 to smithi08477203-1 2024-09-20T22:44:39.273 INFO:tasks.rados.rados.0.smithi084.stdout:588: finishing write tid 2 to smithi08477203-1 2024-09-20T22:44:39.273 INFO:tasks.rados.rados.0.smithi084.stdout:588: finishing write tid 3 to smithi08477203-1 2024-09-20T22:44:39.274 INFO:tasks.rados.rados.0.smithi084.stdout:588: finishing write tid 4 to smithi08477203-1 2024-09-20T22:44:39.274 INFO:tasks.rados.rados.0.smithi084.stdout:588: finishing write tid 5 to smithi08477203-1 2024-09-20T22:44:39.274 INFO:tasks.rados.rados.0.smithi084.stdout:588: finishing write tid 6 to smithi08477203-1 2024-09-20T22:44:39.274 INFO:tasks.rados.rados.0.smithi084.stdout:588: oid 1 updating version 0 to 121 2024-09-20T22:44:39.274 INFO:tasks.rados.rados.0.smithi084.stdout:588: oid 1 version 121 is already newer than 118 2024-09-20T22:44:39.274 INFO:tasks.rados.rados.0.smithi084.stdout:588: oid 1 version 121 is already newer than 120 2024-09-20T22:44:39.274 INFO:tasks.rados.rados.0.smithi084.stdout:588: oid 1 version 121 is already newer than 119 2024-09-20T22:44:39.274 INFO:tasks.rados.rados.0.smithi084.stdout:588: oid 1 updating version 121 to 122 2024-09-20T22:44:39.274 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 1 v 122 (ObjNum 237 snap 51 seq_num 237) dirty exists 2024-09-20T22:44:39.274 INFO:tasks.rados.rados.0.smithi084.stdout:588: left oid 1 (ObjNum 237 snap 51 seq_num 237) 2024-09-20T22:44:39.306 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:44:39.326 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:39.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:39.327 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:44:39.401 INFO:journalctl@ceph.iscsi.iscsi.a.smithi154.stdout:Sep 20 22:44:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-iscsi-iscsi-a[59730]: debug there is no tcmu-runner data available 2024-09-20T22:44:39.401 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:39 smithi154 ceph-mon[33264]: 4.3 scrub starts 2024-09-20T22:44:39.401 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:39 smithi154 ceph-mon[33264]: 4.3 scrub ok 2024-09-20T22:44:39.401 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:39 smithi154 ceph-mon[33264]: pgmap v199: 178 pgs: 1 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 174 active+clean; 118 MiB data, 534 MiB used, 625 GiB / 626 GiB avail 2024-09-20T22:44:39.401 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:39 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/2311950700' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:44:39.401 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:39 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:44:39.401 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:39 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:44:39.429 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:44:39.450 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:39.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:39.451 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:44:39.552 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:44:39.573 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:39.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:39.574 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:44:39.675 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:44:39.696 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:39.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:39.697 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:44:39.796 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:44:39.798 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:44:39.820 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:39.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:39.821 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:44:39.922 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:44:39.945 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:39.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:39.946 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:44:40.046 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:44:40.070 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:40.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:40.071 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:44:40.172 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:44:40.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:40.195 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:40.195 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:44:40.230 INFO:tasks.rados.rados.0.smithi084.stdout:587: done (2 left) 2024-09-20T22:44:40.230 INFO:tasks.rados.rados.0.smithi084.stdout:588: done (1 left) 2024-09-20T22:44:40.230 INFO:tasks.rados.rados.0.smithi084.stdout:589: done (0 left) 2024-09-20T22:44:40.230 INFO:tasks.rados.rados.0.smithi084.stdout:590: rollback oid 20 current snap is 52 2024-09-20T22:44:40.230 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 20 to 51 2024-09-20T22:44:40.230 INFO:tasks.rados.rados.0.smithi084.stdout:591: copy_from oid 5 from oid 34 current snap is 52 2024-09-20T22:44:40.230 INFO:tasks.rados.rados.0.smithi084.stdout:592: read oid 26 snap -1 2024-09-20T22:44:40.230 INFO:tasks.rados.rados.0.smithi084.stdout:592: expect (ObjNum 201 snap 41 seq_num 201) 2024-09-20T22:44:40.230 INFO:tasks.rados.rados.0.smithi084.stdout:593: write (excl) oid 40 current snap is 52 2024-09-20T22:44:40.230 INFO:tasks.rados.rados.0.smithi084.stdout:593: seq_num 238 ranges {400066=716205,1860832=139234} 2024-09-20T22:44:40.235 INFO:tasks.rados.rados.0.smithi084.stdout:593: writing smithi08477203-40 from 400066 to 1116271 tid 1 2024-09-20T22:44:40.241 INFO:tasks.rados.rados.0.smithi084.stdout:593: writing smithi08477203-40 from 1860832 to 2000066 tid 2 2024-09-20T22:44:40.242 INFO:tasks.rados.rados.0.smithi084.stdout:594: delete oid 46 current snap is 52 2024-09-20T22:44:40.242 INFO:tasks.rados.rados.0.smithi084.stdout:590: finishing rollback tid 1 to smithi08477203-20 2024-09-20T22:44:40.242 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 20 v 0 (ObjNum 5313 snap 0 seq_num 803185440) dirty dne 2024-09-20T22:44:40.245 INFO:tasks.rados.rados.0.smithi084.stdout:590: done (4 left) 2024-09-20T22:44:40.245 INFO:tasks.rados.rados.0.smithi084.stdout:594: done (3 left) 2024-09-20T22:44:40.245 INFO:tasks.rados.rados.0.smithi084.stdout:595: write oid 19 current snap is 52 2024-09-20T22:44:40.245 INFO:tasks.rados.rados.0.smithi084.stdout:595: seq_num 240 ranges {433680=466703,1577796=455884} 2024-09-20T22:44:40.251 INFO:tasks.rados.rados.0.smithi084.stdout:595: writing smithi08477203-19 from 433680 to 900383 tid 1 2024-09-20T22:44:40.258 INFO:tasks.rados.rados.0.smithi084.stdout:595: writing smithi08477203-19 from 1577796 to 2033680 tid 2 2024-09-20T22:44:40.259 INFO:tasks.rados.rados.0.smithi084.stdout:596: rollback oid 21 current snap is 52 2024-09-20T22:44:40.259 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 21 to 51 2024-09-20T22:44:40.259 INFO:tasks.rados.rados.0.smithi084.stdout:593: finishing write tid 1 to smithi08477203-40 2024-09-20T22:44:40.259 INFO:tasks.rados.rados.0.smithi084.stdout:593: finishing write tid 2 to smithi08477203-40 2024-09-20T22:44:40.259 INFO:tasks.rados.rados.0.smithi084.stdout:593: finishing write tid 3 to smithi08477203-40 2024-09-20T22:44:40.259 INFO:tasks.rados.rados.0.smithi084.stdout:593: finishing write tid 4 to smithi08477203-40 2024-09-20T22:44:40.259 INFO:tasks.rados.rados.0.smithi084.stdout:593: oid 40 updating version 0 to 129 2024-09-20T22:44:40.259 INFO:tasks.rados.rados.0.smithi084.stdout:593: oid 40 updating version 129 to 131 2024-09-20T22:44:40.259 INFO:tasks.rados.rados.0.smithi084.stdout:593: oid 40 version 131 is already newer than 130 2024-09-20T22:44:40.259 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 40 v 131 (ObjNum 238 snap 52 seq_num 238) dirty exists 2024-09-20T22:44:40.259 INFO:tasks.rados.rados.0.smithi084.stdout:593: left oid 40 (ObjNum 238 snap 52 seq_num 238) 2024-09-20T22:44:40.259 INFO:tasks.rados.rados.0.smithi084.stdout:593: done (4 left) 2024-09-20T22:44:40.259 INFO:tasks.rados.rados.0.smithi084.stdout:597: snap_create 2024-09-20T22:44:40.259 INFO:tasks.rados.rados.0.smithi084.stdout:592: expect (ObjNum 201 snap 41 seq_num 201) 2024-09-20T22:44:40.296 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:44:40.310 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:44:40.311 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:44:40.317 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:40.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:40.318 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:44:40.328 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:44:40.328 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:44:40.328 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:40 smithi084 ceph-mon[40024]: 3.3 scrub starts 2024-09-20T22:44:40.328 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:40 smithi084 ceph-mon[40024]: 3.3 scrub ok 2024-09-20T22:44:40.328 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:40 smithi084 ceph-mon[40024]: 5.7 scrub starts 2024-09-20T22:44:40.328 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:40 smithi084 ceph-mon[40024]: 5.7 scrub ok 2024-09-20T22:44:40.329 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:40 smithi084 ceph-mon[40024]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:44:40.329 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:40 smithi084 ceph-mon[40024]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T22:44:40.329 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:40 smithi084 ceph-mon[40024]: Cluster is now healthy 2024-09-20T22:44:40.329 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:40 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/2311950700' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T22:44:40.329 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:40 smithi084 ceph-mon[40024]: osdmap e200: 8 total, 8 up, 7 in 2024-09-20T22:44:40.329 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:40 smithi084 ceph-mon[40024]: 2.1 scrub starts 2024-09-20T22:44:40.329 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:40 smithi084 ceph-mon[40024]: 2.1 scrub ok 2024-09-20T22:44:40.329 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:40 smithi084 ceph-mon[29774]: 3.3 scrub starts 2024-09-20T22:44:40.329 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:40 smithi084 ceph-mon[29774]: 3.3 scrub ok 2024-09-20T22:44:40.329 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:40 smithi084 ceph-mon[29774]: 5.7 scrub starts 2024-09-20T22:44:40.329 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:40 smithi084 ceph-mon[29774]: 5.7 scrub ok 2024-09-20T22:44:40.329 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:40 smithi084 ceph-mon[29774]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:44:40.330 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:40 smithi084 ceph-mon[29774]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T22:44:40.330 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:40 smithi084 ceph-mon[29774]: Cluster is now healthy 2024-09-20T22:44:40.330 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:40 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/2311950700' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T22:44:40.330 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:40 smithi084 ceph-mon[29774]: osdmap e200: 8 total, 8 up, 7 in 2024-09-20T22:44:40.330 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:40 smithi084 ceph-mon[29774]: 2.1 scrub starts 2024-09-20T22:44:40.330 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:40 smithi084 ceph-mon[29774]: 2.1 scrub ok 2024-09-20T22:44:40.348 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:44:40.348 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:44:40.368 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-20T22:44:40.368 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T22:44:40.372 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 7] out_osds: [4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-20T22:44:40.372 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-20T22:44:40.372 INFO:tasks.thrashosds.thrasher:inject_pause on osd.6 2024-09-20T22:44:40.372 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-20T22:44:40.372 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-20T22:44:40.372 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.6 config set filestore_inject_stall 3 2024-09-20T22:44:40.393 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:44:40.393 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:44:40.409 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:44:40.410 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:44:40.418 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:44:40.421 INFO:tasks.rados.rados.0.smithi084.stdout:591: finishing copy_from racing read to smithi08477203-5 2024-09-20T22:44:40.421 INFO:tasks.rados.rados.0.smithi084.stdout:595: finishing write tid 1 to smithi08477203-19 2024-09-20T22:44:40.421 INFO:tasks.rados.rados.0.smithi084.stdout:591: finishing copy_from to smithi08477203-5 2024-09-20T22:44:40.421 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 5 v 154 (ObjNum 50 snap 0 seq_num 50) dirty exists 2024-09-20T22:44:40.421 INFO:tasks.rados.rados.0.smithi084.stdout:595: finishing write tid 2 to smithi08477203-19 2024-09-20T22:44:40.421 INFO:tasks.rados.rados.0.smithi084.stdout:595: finishing write tid 3 to smithi08477203-19 2024-09-20T22:44:40.421 INFO:tasks.rados.rados.0.smithi084.stdout:596: finishing rollback tid 0 to smithi08477203-21 2024-09-20T22:44:40.422 INFO:tasks.rados.rados.0.smithi084.stdout:595: finishing write tid 4 to smithi08477203-19 2024-09-20T22:44:40.422 INFO:tasks.rados.rados.0.smithi084.stdout:595: oid 19 updating version 0 to 240 2024-09-20T22:44:40.422 INFO:tasks.rados.rados.0.smithi084.stdout:595: oid 19 version 240 is already newer than 239 2024-09-20T22:44:40.422 INFO:tasks.rados.rados.0.smithi084.stdout:595: oid 19 version 240 is already newer than 238 2024-09-20T22:44:40.422 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 19 v 240 (ObjNum 240 snap 52 seq_num 240) dirty exists 2024-09-20T22:44:40.422 INFO:tasks.rados.rados.0.smithi084.stdout:595: left oid 19 (ObjNum 240 snap 52 seq_num 240) 2024-09-20T22:44:40.422 INFO:tasks.rados.rados.0.smithi084.stdout:596: finishing rollback tid 1 to smithi08477203-21 2024-09-20T22:44:40.422 INFO:tasks.rados.rados.0.smithi084.stdout:596: finishing rollback tid 2 to smithi08477203-21 2024-09-20T22:44:40.422 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 21 v 182 (ObjNum 227 snap 48 seq_num 227) dirty exists 2024-09-20T22:44:40.430 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:44:40.430 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:44:40.441 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:40.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:40.442 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:44:40.454 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:44:40.454 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:44:40.483 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:40 smithi154 ceph-mon[33264]: 3.3 scrub starts 2024-09-20T22:44:40.484 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:40 smithi154 ceph-mon[33264]: 3.3 scrub ok 2024-09-20T22:44:40.484 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:40 smithi154 ceph-mon[33264]: 5.7 scrub starts 2024-09-20T22:44:40.484 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:40 smithi154 ceph-mon[33264]: 5.7 scrub ok 2024-09-20T22:44:40.484 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:40 smithi154 ceph-mon[33264]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:44:40.484 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:40 smithi154 ceph-mon[33264]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T22:44:40.484 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:40 smithi154 ceph-mon[33264]: Cluster is now healthy 2024-09-20T22:44:40.484 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:40 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/2311950700' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T22:44:40.484 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:40 smithi154 ceph-mon[33264]: osdmap e200: 8 total, 8 up, 7 in 2024-09-20T22:44:40.484 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:40 smithi154 ceph-mon[33264]: 2.1 scrub starts 2024-09-20T22:44:40.484 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:40 smithi154 ceph-mon[33264]: 2.1 scrub ok 2024-09-20T22:44:40.543 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:44:40.564 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:40.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:40.565 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:44:40.666 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:44:40.688 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:40.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:40.689 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:44:40.789 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:44:40.806 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-20T22:44:40.806 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:44:40.811 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:40.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:40.812 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:44:40.913 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:44:40.934 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:40.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:40.935 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:44:41.037 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:44:41.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:41.059 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:41.060 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:44:41.160 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:44:41.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:41.183 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:41.183 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:44:41.233 INFO:tasks.rados.rados.0.smithi084.stdout:591: done (4 left) 2024-09-20T22:44:41.233 INFO:tasks.rados.rados.0.smithi084.stdout:592: done (3 left) 2024-09-20T22:44:41.233 INFO:tasks.rados.rados.0.smithi084.stdout:595: done (2 left) 2024-09-20T22:44:41.233 INFO:tasks.rados.rados.0.smithi084.stdout:596: done (1 left) 2024-09-20T22:44:41.233 INFO:tasks.rados.rados.0.smithi084.stdout:597: done (0 left) 2024-09-20T22:44:41.233 INFO:tasks.rados.rados.0.smithi084.stdout:598: snap_create 2024-09-20T22:44:41.284 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:44:41.305 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:41.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:41.306 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:44:41.320 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:41 smithi084 ceph-mon[40024]: pgmap v202: 178 pgs: 1 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 175 active+clean; 127 MiB data, 554 MiB used, 625 GiB / 626 GiB avail; 27 MiB/s rd, 4.7 MiB/s wr, 25 op/s 2024-09-20T22:44:41.320 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:41 smithi084 ceph-mon[40024]: 6.10 scrub starts 2024-09-20T22:44:41.320 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:41 smithi084 ceph-mon[40024]: 6.10 scrub ok 2024-09-20T22:44:41.320 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:41 smithi084 ceph-mon[40024]: osdmap e201: 8 total, 8 up, 7 in 2024-09-20T22:44:41.320 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:41 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/3928580850' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:44:41.320 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:41 smithi084 ceph-mon[40024]: 7.d scrub starts 2024-09-20T22:44:41.320 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:41 smithi084 ceph-mon[40024]: 7.d scrub ok 2024-09-20T22:44:41.320 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:41 smithi084 ceph-mon[40024]: 4.4 scrub starts 2024-09-20T22:44:41.320 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:41 smithi084 ceph-mon[40024]: 4.4 scrub ok 2024-09-20T22:44:41.320 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:41 smithi084 ceph-mon[29774]: pgmap v202: 178 pgs: 1 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 175 active+clean; 127 MiB data, 554 MiB used, 625 GiB / 626 GiB avail; 27 MiB/s rd, 4.7 MiB/s wr, 25 op/s 2024-09-20T22:44:41.321 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:41 smithi084 ceph-mon[29774]: 6.10 scrub starts 2024-09-20T22:44:41.321 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:41 smithi084 ceph-mon[29774]: 6.10 scrub ok 2024-09-20T22:44:41.321 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:41 smithi084 ceph-mon[29774]: osdmap e201: 8 total, 8 up, 7 in 2024-09-20T22:44:41.321 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:41 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/3928580850' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:44:41.321 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:41 smithi084 ceph-mon[29774]: 7.d scrub starts 2024-09-20T22:44:41.321 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:41 smithi084 ceph-mon[29774]: 7.d scrub ok 2024-09-20T22:44:41.321 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:41 smithi084 ceph-mon[29774]: 4.4 scrub starts 2024-09-20T22:44:41.321 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:41 smithi084 ceph-mon[29774]: 4.4 scrub ok 2024-09-20T22:44:41.354 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:44:41.407 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:44:41.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:41.429 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:41.429 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:44:41.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:41 smithi154 ceph-mon[33264]: pgmap v202: 178 pgs: 1 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 175 active+clean; 127 MiB data, 554 MiB used, 625 GiB / 626 GiB avail; 27 MiB/s rd, 4.7 MiB/s wr, 25 op/s 2024-09-20T22:44:41.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:41 smithi154 ceph-mon[33264]: 6.10 scrub starts 2024-09-20T22:44:41.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:41 smithi154 ceph-mon[33264]: 6.10 scrub ok 2024-09-20T22:44:41.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:41 smithi154 ceph-mon[33264]: osdmap e201: 8 total, 8 up, 7 in 2024-09-20T22:44:41.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:41 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/3928580850' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:44:41.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:41 smithi154 ceph-mon[33264]: 7.d scrub starts 2024-09-20T22:44:41.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:41 smithi154 ceph-mon[33264]: 7.d scrub ok 2024-09-20T22:44:41.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:41 smithi154 ceph-mon[33264]: 4.4 scrub starts 2024-09-20T22:44:41.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:41 smithi154 ceph-mon[33264]: 4.4 scrub ok 2024-09-20T22:44:41.530 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:44:41.551 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:41.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:41.552 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:44:41.653 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:44:41.675 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:41.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:41.676 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:44:41.776 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:44:41.798 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:41.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:41.799 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:44:41.900 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:44:41.922 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:41.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:41.923 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:44:42.023 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:44:42.045 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:42.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:42.046 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:44:42.096 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.6 dump_historic_ops 2024-09-20T22:44:42.100 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd set noscrub 2024-09-20T22:44:42.147 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:44:42.170 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:42.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:42.171 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:44:42.237 INFO:tasks.rados.rados.0.smithi084.stdout:598: done (0 left) 2024-09-20T22:44:42.237 INFO:tasks.rados.rados.0.smithi084.stdout:599: rollback oid 7 current snap is 54 2024-09-20T22:44:42.238 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 7 to 52 2024-09-20T22:44:42.238 INFO:tasks.rados.rados.0.smithi084.stdout:600: write oid 4 current snap is 54 2024-09-20T22:44:42.238 INFO:tasks.rados.rados.0.smithi084.stdout:600: seq_num 241 ranges {450487=541952,1636278=414209} 2024-09-20T22:44:42.242 INFO:tasks.rados.rados.0.smithi084.stdout:600: writing smithi08477203-4 from 450487 to 992439 tid 1 2024-09-20T22:44:42.248 INFO:tasks.rados.rados.0.smithi084.stdout:600: writing smithi08477203-4 from 1636278 to 2050487 tid 2 2024-09-20T22:44:42.248 INFO:tasks.rados.rados.0.smithi084.stdout:601: rollback oid 25 current snap is 54 2024-09-20T22:44:42.248 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 25 to 52 2024-09-20T22:44:42.248 INFO:tasks.rados.rados.0.smithi084.stdout:599: finishing rollback tid 0 to smithi08477203-7 2024-09-20T22:44:42.248 INFO:tasks.rados.rados.0.smithi084.stdout:602: snap_remove snap 51 2024-09-20T22:44:42.272 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:44:42.293 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:42.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:42.294 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:44:42.366 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:44:42.389 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:42 smithi154 ceph-mon[33264]: 6.1a scrub starts 2024-09-20T22:44:42.389 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:42 smithi154 ceph-mon[33264]: 6.1a scrub ok 2024-09-20T22:44:42.389 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:42 smithi154 ceph-mon[33264]: 4.19 scrub starts 2024-09-20T22:44:42.389 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:42 smithi154 ceph-mon[33264]: 4.19 scrub ok 2024-09-20T22:44:42.389 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:42 smithi154 ceph-mon[33264]: osdmap e202: 8 total, 8 up, 7 in 2024-09-20T22:44:42.389 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:42 smithi154 ceph-mon[33264]: 6.1e scrub starts 2024-09-20T22:44:42.389 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:42 smithi154 ceph-mon[33264]: 6.1e scrub ok 2024-09-20T22:44:42.396 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:44:42.416 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:42.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:42.417 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:44:42.487 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:42 smithi084 ceph-mon[29774]: 6.1a scrub starts 2024-09-20T22:44:42.487 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:42 smithi084 ceph-mon[29774]: 6.1a scrub ok 2024-09-20T22:44:42.487 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:42 smithi084 ceph-mon[29774]: 4.19 scrub starts 2024-09-20T22:44:42.488 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:42 smithi084 ceph-mon[29774]: 4.19 scrub ok 2024-09-20T22:44:42.488 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:42 smithi084 ceph-mon[29774]: osdmap e202: 8 total, 8 up, 7 in 2024-09-20T22:44:42.488 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:42 smithi084 ceph-mon[29774]: 6.1e scrub starts 2024-09-20T22:44:42.488 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:42 smithi084 ceph-mon[29774]: 6.1e scrub ok 2024-09-20T22:44:42.488 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:42 smithi084 ceph-mon[40024]: 6.1a scrub starts 2024-09-20T22:44:42.488 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:42 smithi084 ceph-mon[40024]: 6.1a scrub ok 2024-09-20T22:44:42.488 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:42 smithi084 ceph-mon[40024]: 4.19 scrub starts 2024-09-20T22:44:42.488 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:42 smithi084 ceph-mon[40024]: 4.19 scrub ok 2024-09-20T22:44:42.488 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:42 smithi084 ceph-mon[40024]: osdmap e202: 8 total, 8 up, 7 in 2024-09-20T22:44:42.488 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:42 smithi084 ceph-mon[40024]: 6.1e scrub starts 2024-09-20T22:44:42.488 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:42 smithi084 ceph-mon[40024]: 6.1e scrub ok 2024-09-20T22:44:42.518 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:44:42.539 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:42.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:42.540 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:44:42.641 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:44:42.662 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:42.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:42.663 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:44:42.764 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:44:42.787 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:42.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:42.788 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:44:42.888 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:44:42.910 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:42.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:42.911 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:44:43.012 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:44:43.035 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:43.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:43.036 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:44:43.135 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is set 2024-09-20T22:44:43.137 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:44:43.140 INFO:tasks.rados.rados.0.smithi084.stdout:602: done (3 left) 2024-09-20T22:44:43.140 INFO:tasks.rados.rados.0.smithi084.stdout:603: read oid 9 snap -1 2024-09-20T22:44:43.140 INFO:tasks.rados.rados.0.smithi084.stdout:603: expect deleted 2024-09-20T22:44:43.140 INFO:tasks.rados.rados.0.smithi084.stdout:604: write (excl) oid 49 current snap is 54 2024-09-20T22:44:43.140 INFO:tasks.rados.rados.0.smithi084.stdout:604: seq_num 242 ranges {467294=617201,1578407=488887} 2024-09-20T22:44:43.146 INFO:tasks.rados.rados.0.smithi084.stdout:604: writing smithi08477203-49 from 467294 to 1084495 tid 1 2024-09-20T22:44:43.153 INFO:tasks.rados.rados.0.smithi084.stdout:604: writing smithi08477203-49 from 1578407 to 2067294 tid 2 2024-09-20T22:44:43.154 INFO:tasks.rados.rados.0.smithi084.stdout:599: finishing rollback tid 1 to smithi08477203-7 2024-09-20T22:44:43.154 INFO:tasks.rados.rados.0.smithi084.stdout:599: finishing rollback tid 2 to smithi08477203-7 2024-09-20T22:44:43.154 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 7 v 104 (ObjNum 150 snap 30 seq_num 150) dirty exists 2024-09-20T22:44:43.154 INFO:tasks.rados.rados.0.smithi084.stdout:600: finishing write tid 1 to smithi08477203-4 2024-09-20T22:44:43.154 INFO:tasks.rados.rados.0.smithi084.stdout:600: finishing write tid 2 to smithi08477203-4 2024-09-20T22:44:43.154 INFO:tasks.rados.rados.0.smithi084.stdout:600: finishing write tid 3 to smithi08477203-4 2024-09-20T22:44:43.154 INFO:tasks.rados.rados.0.smithi084.stdout:600: finishing write tid 4 to smithi08477203-4 2024-09-20T22:44:43.154 INFO:tasks.rados.rados.0.smithi084.stdout:600: oid 4 updating version 0 to 243 2024-09-20T22:44:43.154 INFO:tasks.rados.rados.0.smithi084.stdout:600: oid 4 version 243 is already newer than 242 2024-09-20T22:44:43.155 INFO:tasks.rados.rados.0.smithi084.stdout:600: oid 4 updating version 243 to 244 2024-09-20T22:44:43.155 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 4 v 244 (ObjNum 241 snap 54 seq_num 241) dirty exists 2024-09-20T22:44:43.155 INFO:tasks.rados.rados.0.smithi084.stdout:600: left oid 4 (ObjNum 241 snap 54 seq_num 241) 2024-09-20T22:44:43.155 INFO:tasks.rados.rados.0.smithi084.stdout:601: finishing rollback tid 1 to smithi08477203-25 2024-09-20T22:44:43.155 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 25 v 0 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-09-20T22:44:43.155 INFO:tasks.rados.rados.0.smithi084.stdout:599: done (4 left) 2024-09-20T22:44:43.155 INFO:tasks.rados.rados.0.smithi084.stdout:600: done (3 left) 2024-09-20T22:44:43.155 INFO:tasks.rados.rados.0.smithi084.stdout:601: done (2 left) 2024-09-20T22:44:43.155 INFO:tasks.rados.rados.0.smithi084.stdout:603: done (1 left) 2024-09-20T22:44:43.155 INFO:tasks.rados.rados.0.smithi084.stdout:605: snap_remove snap 52 2024-09-20T22:44:43.166 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:43.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:43.167 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:44:43.208 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-20T22:44:43.264 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-20T22:44:43.264 INFO:teuthology.orchestra.run.smithi154.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-20T22:44:43.264 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-20T22:44:43.268 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:44:43.274 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:44:43.291 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:43.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:43.293 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:44:43.393 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:44:43.414 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:43.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:43.415 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:44:43.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:43 smithi154 ceph-mon[33264]: pgmap v205: 178 pgs: 1 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 175 active+clean; 127 MiB data, 554 MiB used, 625 GiB / 626 GiB avail; 27 MiB/s rd, 4.7 MiB/s wr, 25 op/s 2024-09-20T22:44:43.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:43 smithi154 ceph-mon[33264]: 4.1d scrub starts 2024-09-20T22:44:43.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:43 smithi154 ceph-mon[33264]: 4.1d scrub ok 2024-09-20T22:44:43.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:43 smithi154 ceph-mon[33264]: osdmap e203: 8 total, 8 up, 7 in 2024-09-20T22:44:43.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:43 smithi154 ceph-mon[33264]: 3.10 scrub starts 2024-09-20T22:44:43.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:43 smithi154 ceph-mon[33264]: 3.10 scrub ok 2024-09-20T22:44:43.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:43 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:44:43.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:43 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:44:43.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:43 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/1674969555' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-20T22:44:43.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:43 smithi154 ceph-mon[33264]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:44:43.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:43 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/1674969555' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T22:44:43.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:43 smithi154 ceph-mon[33264]: osdmap e204: 8 total, 8 up, 7 in 2024-09-20T22:44:43.516 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:44:43.524 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:43 smithi084 ceph-mon[29774]: pgmap v205: 178 pgs: 1 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 175 active+clean; 127 MiB data, 554 MiB used, 625 GiB / 626 GiB avail; 27 MiB/s rd, 4.7 MiB/s wr, 25 op/s 2024-09-20T22:44:43.524 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:43 smithi084 ceph-mon[29774]: 4.1d scrub starts 2024-09-20T22:44:43.524 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:43 smithi084 ceph-mon[29774]: 4.1d scrub ok 2024-09-20T22:44:43.524 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:43 smithi084 ceph-mon[29774]: osdmap e203: 8 total, 8 up, 7 in 2024-09-20T22:44:43.524 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:43 smithi084 ceph-mon[29774]: 3.10 scrub starts 2024-09-20T22:44:43.524 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:43 smithi084 ceph-mon[29774]: 3.10 scrub ok 2024-09-20T22:44:43.524 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:43 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:44:43.524 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:43 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:44:43.524 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:43 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/1674969555' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-20T22:44:43.524 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:43 smithi084 ceph-mon[29774]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:44:43.524 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:43 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/1674969555' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T22:44:43.525 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:43 smithi084 ceph-mon[29774]: osdmap e204: 8 total, 8 up, 7 in 2024-09-20T22:44:43.525 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:43 smithi084 ceph-mon[40024]: pgmap v205: 178 pgs: 1 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 175 active+clean; 127 MiB data, 554 MiB used, 625 GiB / 626 GiB avail; 27 MiB/s rd, 4.7 MiB/s wr, 25 op/s 2024-09-20T22:44:43.525 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:43 smithi084 ceph-mon[40024]: 4.1d scrub starts 2024-09-20T22:44:43.525 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:43 smithi084 ceph-mon[40024]: 4.1d scrub ok 2024-09-20T22:44:43.525 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:43 smithi084 ceph-mon[40024]: osdmap e203: 8 total, 8 up, 7 in 2024-09-20T22:44:43.525 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:43 smithi084 ceph-mon[40024]: 3.10 scrub starts 2024-09-20T22:44:43.525 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:43 smithi084 ceph-mon[40024]: 3.10 scrub ok 2024-09-20T22:44:43.525 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:43 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:44:43.525 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:43 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:44:43.525 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:43 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/1674969555' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-20T22:44:43.525 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:43 smithi084 ceph-mon[40024]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:44:43.526 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:43 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/1674969555' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T22:44:43.526 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:43 smithi084 ceph-mon[40024]: osdmap e204: 8 total, 8 up, 7 in 2024-09-20T22:44:43.537 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:43.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:43.538 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:44:43.639 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:44:43.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:43.661 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:43.661 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:44:43.762 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:44:43.783 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:43.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:43.784 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:44:43.885 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:44:43.906 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:43.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:43.907 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:44:43.987 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:44:44.008 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:44:44.029 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:44.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:44.030 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:44:44.131 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:44:44.139 INFO:tasks.rados.rados.0.smithi084.stdout:605: done (1 left) 2024-09-20T22:44:44.139 INFO:tasks.rados.rados.0.smithi084.stdout:606: delete oid 11 current snap is 54 2024-09-20T22:44:44.140 INFO:tasks.rados.rados.0.smithi084.stdout:604: finishing write tid 1 to smithi08477203-49 2024-09-20T22:44:44.140 INFO:tasks.rados.rados.0.smithi084.stdout:604: finishing write tid 2 to smithi08477203-49 2024-09-20T22:44:44.140 INFO:tasks.rados.rados.0.smithi084.stdout:604: finishing write tid 3 to smithi08477203-49 2024-09-20T22:44:44.140 INFO:tasks.rados.rados.0.smithi084.stdout:604: finishing write tid 4 to smithi08477203-49 2024-09-20T22:44:44.140 INFO:tasks.rados.rados.0.smithi084.stdout:604: oid 49 updating version 0 to 186 2024-09-20T22:44:44.140 INFO:tasks.rados.rados.0.smithi084.stdout:604: oid 49 updating version 186 to 188 2024-09-20T22:44:44.140 INFO:tasks.rados.rados.0.smithi084.stdout:604: oid 49 version 188 is already newer than 187 2024-09-20T22:44:44.140 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 49 v 188 (ObjNum 242 snap 54 seq_num 242) dirty exists 2024-09-20T22:44:44.140 INFO:tasks.rados.rados.0.smithi084.stdout:604: left oid 49 (ObjNum 242 snap 54 seq_num 242) 2024-09-20T22:44:44.145 INFO:tasks.rados.rados.0.smithi084.stdout:604: done (1 left) 2024-09-20T22:44:44.145 INFO:tasks.rados.rados.0.smithi084.stdout:606: done (0 left) 2024-09-20T22:44:44.146 INFO:tasks.rados.rados.0.smithi084.stdout:607: rollback oid 28 current snap is 54 2024-09-20T22:44:44.146 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 28 to 53 2024-09-20T22:44:44.146 INFO:tasks.rados.rados.0.smithi084.stdout:608: read oid 21 snap 53 2024-09-20T22:44:44.146 INFO:tasks.rados.rados.0.smithi084.stdout:608: expect (ObjNum 227 snap 48 seq_num 227) 2024-09-20T22:44:44.146 INFO:tasks.rados.rados.0.smithi084.stdout:609: write oid 39 current snap is 54 2024-09-20T22:44:44.146 INFO:tasks.rados.rados.0.smithi084.stdout:609: seq_num 244 ranges {500908=484052,1411724=689184} 2024-09-20T22:44:44.150 INFO:tasks.rados.rados.0.smithi084.stdout:609: writing smithi08477203-39 from 500908 to 984960 tid 1 2024-09-20T22:44:44.157 INFO:tasks.rados.rados.0.smithi084.stdout:609: writing smithi08477203-39 from 1411724 to 2100908 tid 2 2024-09-20T22:44:44.157 INFO:tasks.rados.rados.0.smithi084.stdout:610: write (excl) oid 3 current snap is 54 2024-09-20T22:44:44.158 INFO:tasks.rados.rados.0.smithi084.stdout:607: finishing rollback tid 0 to smithi08477203-28 2024-09-20T22:44:44.158 INFO:tasks.rados.rados.0.smithi084.stdout:610: seq_num 245 ranges {517715=559301,1870206=247509} 2024-09-20T22:44:44.158 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:44.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:44.159 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:44:44.165 INFO:tasks.rados.rados.0.smithi084.stdout:610: writing smithi08477203-3 from 517715 to 1077016 tid 1 2024-09-20T22:44:44.171 INFO:tasks.rados.rados.0.smithi084.stdout:610: writing smithi08477203-3 from 1870206 to 2117715 tid 2 2024-09-20T22:44:44.171 INFO:tasks.rados.rados.0.smithi084.stdout:611: write (excl) oid 46 current snap is 54 2024-09-20T22:44:44.171 INFO:tasks.rados.rados.0.smithi084.stdout:607: finishing rollback tid 1 to smithi08477203-28 2024-09-20T22:44:44.171 INFO:tasks.rados.rados.0.smithi084.stdout:607: finishing rollback tid 2 to smithi08477203-28 2024-09-20T22:44:44.171 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 28 v 253 (ObjNum 231 snap 49 seq_num 231) dirty exists 2024-09-20T22:44:44.171 INFO:tasks.rados.rados.0.smithi084.stdout:609: finishing write tid 1 to smithi08477203-39 2024-09-20T22:44:44.171 INFO:tasks.rados.rados.0.smithi084.stdout:611: seq_num 246 ranges {534522=634550,1812335=322187} 2024-09-20T22:44:44.177 INFO:tasks.rados.rados.0.smithi084.stdout:611: writing smithi08477203-46 from 534522 to 1169072 tid 1 2024-09-20T22:44:44.183 INFO:tasks.rados.rados.0.smithi084.stdout:611: writing smithi08477203-46 from 1812335 to 2134522 tid 2 2024-09-20T22:44:44.183 INFO:tasks.rados.rados.0.smithi084.stdout:607: done (4 left) 2024-09-20T22:44:44.183 INFO:tasks.rados.rados.0.smithi084.stdout:612: snap_create 2024-09-20T22:44:44.183 INFO:tasks.rados.rados.0.smithi084.stdout:609: finishing write tid 2 to smithi08477203-39 2024-09-20T22:44:44.183 INFO:tasks.rados.rados.0.smithi084.stdout:609: finishing write tid 3 to smithi08477203-39 2024-09-20T22:44:44.183 INFO:tasks.rados.rados.0.smithi084.stdout:609: finishing write tid 4 to smithi08477203-39 2024-09-20T22:44:44.183 INFO:tasks.rados.rados.0.smithi084.stdout:609: oid 39 updating version 0 to 124 2024-09-20T22:44:44.183 INFO:tasks.rados.rados.0.smithi084.stdout:609: oid 39 updating version 124 to 126 2024-09-20T22:44:44.183 INFO:tasks.rados.rados.0.smithi084.stdout:609: oid 39 version 126 is already newer than 125 2024-09-20T22:44:44.183 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 39 v 126 (ObjNum 244 snap 54 seq_num 244) dirty exists 2024-09-20T22:44:44.183 INFO:tasks.rados.rados.0.smithi084.stdout:609: left oid 39 (ObjNum 244 snap 54 seq_num 244) 2024-09-20T22:44:44.184 INFO:tasks.rados.rados.0.smithi084.stdout:608: expect (ObjNum 227 snap 48 seq_num 227) 2024-09-20T22:44:44.260 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:44:44.281 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:44.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:44.282 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:44:44.364 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:44 smithi084 ceph-mon[40024]: 6.8 scrub starts 2024-09-20T22:44:44.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:44 smithi084 ceph-mon[40024]: 6.8 scrub ok 2024-09-20T22:44:44.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:44 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:44:44.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:44 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:44:44.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:44 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:44:44.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:44 smithi084 ceph-mon[40024]: osdmap e205: 8 total, 8 up, 7 in 2024-09-20T22:44:44.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:44 smithi084 ceph-mon[29774]: 6.8 scrub starts 2024-09-20T22:44:44.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:44 smithi084 ceph-mon[29774]: 6.8 scrub ok 2024-09-20T22:44:44.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:44 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:44:44.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:44 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:44:44.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:44 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:44:44.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:44 smithi084 ceph-mon[29774]: osdmap e205: 8 total, 8 up, 7 in 2024-09-20T22:44:44.383 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:44:44.388 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:44 smithi154 ceph-mon[33264]: 6.8 scrub starts 2024-09-20T22:44:44.388 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:44 smithi154 ceph-mon[33264]: 6.8 scrub ok 2024-09-20T22:44:44.388 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:44 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:44:44.388 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:44 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:44:44.388 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:44 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:44:44.388 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:44 smithi154 ceph-mon[33264]: osdmap e205: 8 total, 8 up, 7 in 2024-09-20T22:44:44.400 INFO:tasks.rados.rados.0.smithi084.stdout:610: finishing write tid 1 to smithi08477203-3 2024-09-20T22:44:44.400 INFO:tasks.rados.rados.0.smithi084.stdout:610: finishing write tid 2 to smithi08477203-3 2024-09-20T22:44:44.400 INFO:tasks.rados.rados.0.smithi084.stdout:610: finishing write tid 3 to smithi08477203-3 2024-09-20T22:44:44.400 INFO:tasks.rados.rados.0.smithi084.stdout:610: finishing write tid 4 to smithi08477203-3 2024-09-20T22:44:44.400 INFO:tasks.rados.rados.0.smithi084.stdout:610: oid 3 updating version 0 to 101 2024-09-20T22:44:44.400 INFO:tasks.rados.rados.0.smithi084.stdout:610: oid 3 version 101 is already newer than 100 2024-09-20T22:44:44.400 INFO:tasks.rados.rados.0.smithi084.stdout:610: oid 3 version 101 is already newer than 99 2024-09-20T22:44:44.400 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 3 v 101 (ObjNum 245 snap 54 seq_num 245) dirty exists 2024-09-20T22:44:44.400 INFO:tasks.rados.rados.0.smithi084.stdout:610: left oid 3 (ObjNum 245 snap 54 seq_num 245) 2024-09-20T22:44:44.401 INFO:tasks.rados.rados.0.smithi084.stdout:611: finishing write tid 1 to smithi08477203-46 2024-09-20T22:44:44.401 INFO:tasks.rados.rados.0.smithi084.stdout:611: finishing write tid 2 to smithi08477203-46 2024-09-20T22:44:44.401 INFO:tasks.rados.rados.0.smithi084.stdout:611: finishing write tid 3 to smithi08477203-46 2024-09-20T22:44:44.401 INFO:tasks.rados.rados.0.smithi084.stdout:611: finishing write tid 4 to smithi08477203-46 2024-09-20T22:44:44.401 INFO:tasks.rados.rados.0.smithi084.stdout:611: oid 46 updating version 0 to 111 2024-09-20T22:44:44.401 INFO:tasks.rados.rados.0.smithi084.stdout:611: oid 46 updating version 111 to 112 2024-09-20T22:44:44.401 INFO:tasks.rados.rados.0.smithi084.stdout:611: oid 46 version 112 is already newer than 110 2024-09-20T22:44:44.401 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 46 v 112 (ObjNum 246 snap 54 seq_num 246) dirty exists 2024-09-20T22:44:44.401 INFO:tasks.rados.rados.0.smithi084.stdout:611: left oid 46 (ObjNum 246 snap 54 seq_num 246) 2024-09-20T22:44:44.405 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:44.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:44.406 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:44:44.498 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:44:44.499 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:44:44.507 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:44:44.510 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:44:44.511 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:44:44.527 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:44:44.527 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:44:44.529 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:44.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:44.530 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:44:44.543 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-20T22:44:44.543 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T22:44:44.559 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:44:44.559 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:44:44.575 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:44:44.575 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:44:44.591 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:44:44.591 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:44:44.607 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:44:44.607 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:44:44.616 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-20T22:44:44.616 INFO:teuthology.orchestra.run.smithi154.stdout: "size": 20, 2024-09-20T22:44:44.616 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 600, 2024-09-20T22:44:44.616 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [ 2024-09-20T22:44:44.616 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.616 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.14496.0:46 5.1f 5:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2024-09-20T22:44:44.616 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:41:42.173689+0000", 2024-09-20T22:44:44.616 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 182.4388371, 2024-09-20T22:44:44.616 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.016618790000000001, 2024-09-20T22:44:44.616 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:44:44.616 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:44:44.616 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:44:44.616 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:44:44.617 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.14496", 2024-09-20T22:44:44.617 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3480761128", 2024-09-20T22:44:44.617 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 46 2024-09-20T22:44:44.617 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.617 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:44:44.617 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.617 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:44:44.617 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:42.173689+0000", 2024-09-20T22:44:44.617 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:44:44.617 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.617 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.617 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:44:44.617 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:42.173689+0000", 2024-09-20T22:44:44.617 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:44:44.618 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.618 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.618 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:44:44.618 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:42.173691+0000", 2024-09-20T22:44:44.618 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4419999999999999e-06 2024-09-20T22:44:44.618 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.618 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.618 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:44:44.618 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:42.173696+0000", 2024-09-20T22:44:44.618 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.3900000000000001e-06 2024-09-20T22:44:44.618 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.618 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.618 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:44:44.618 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:42.173709+0000", 2024-09-20T22:44:44.619 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2605999999999999e-05 2024-09-20T22:44:44.619 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.619 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.619 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:44:44.619 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:42.173719+0000", 2024-09-20T22:44:44.619 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0661e-05 2024-09-20T22:44:44.619 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.619 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.619 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:44:44.619 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:42.173805+0000", 2024-09-20T22:44:44.619 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.5520999999999997e-05 2024-09-20T22:44:44.619 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.620 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.620 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op must wait for map", 2024-09-20T22:44:44.620 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:42.173827+0000", 2024-09-20T22:44:44.620 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.2657999999999999e-05 2024-09-20T22:44:44.620 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.620 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.620 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:44:44.620 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:42.177154+0000", 2024-09-20T22:44:44.620 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0033269939999999998 2024-09-20T22:44:44.620 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.620 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.620 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-20T22:44:44.620 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:42.177163+0000", 2024-09-20T22:44:44.620 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.2900000000000002e-06 2024-09-20T22:44:44.621 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.621 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.621 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:44:44.621 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:42.186500+0000", 2024-09-20T22:44:44.621 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0093368710000000001 2024-09-20T22:44:44.621 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.621 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.621 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:44:44.621 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:42.186610+0000", 2024-09-20T22:44:44.621 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000110237 2024-09-20T22:44:44.621 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.621 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.621 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 4,7", 2024-09-20T22:44:44.621 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:42.186740+0000", 2024-09-20T22:44:44.622 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00012984800000000001 2024-09-20T22:44:44.622 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.622 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.622 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:44:44.622 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:42.188026+0000", 2024-09-20T22:44:44.622 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001286265 2024-09-20T22:44:44.622 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.622 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.622 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:44:44.622 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:42.188504+0000", 2024-09-20T22:44:44.622 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00047854899999999999 2024-09-20T22:44:44.622 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.622 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.622 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:44:44.623 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:42.190209+0000", 2024-09-20T22:44:44.623 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001704491 2024-09-20T22:44:44.623 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.623 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.623 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:44:44.623 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:42.190258+0000", 2024-09-20T22:44:44.623 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.8566999999999998e-05 2024-09-20T22:44:44.623 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.623 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.623 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:44:44.623 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:42.190308+0000", 2024-09-20T22:44:44.623 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.0399999999999999e-05 2024-09-20T22:44:44.623 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:44:44.623 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:44:44.624 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:44:44.624 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.624 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.624 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.14478.0:46 5.1f 5:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2024-09-20T22:44:44.624 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:41:42.174990+0000", 2024-09-20T22:44:44.624 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 182.43753617900001, 2024-09-20T22:44:44.624 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.015960838000000001, 2024-09-20T22:44:44.624 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:44:44.624 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:44:44.624 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:44:44.624 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:44:44.624 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.14478", 2024-09-20T22:44:44.624 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/2994273271", 2024-09-20T22:44:44.624 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 46 2024-09-20T22:44:44.625 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.625 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:44:44.625 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.625 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:44:44.625 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:42.174990+0000", 2024-09-20T22:44:44.625 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:44:44.625 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.625 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.625 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:44:44.625 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:42.174990+0000", 2024-09-20T22:44:44.625 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:44:44.625 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.625 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.625 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:44:44.626 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:42.174991+0000", 2024-09-20T22:44:44.626 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.018e-06 2024-09-20T22:44:44.626 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.626 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.626 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:44:44.626 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:42.174997+0000", 2024-09-20T22:44:44.626 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.2759999999999997e-06 2024-09-20T22:44:44.626 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.626 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.626 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:44:44.626 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:42.175007+0000", 2024-09-20T22:44:44.626 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.516e-06 2024-09-20T22:44:44.626 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.626 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.627 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:44:44.627 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:42.175026+0000", 2024-09-20T22:44:44.627 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.8674000000000001e-05 2024-09-20T22:44:44.627 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.627 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.627 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:44:44.627 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:42.175115+0000", 2024-09-20T22:44:44.627 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.9888000000000004e-05 2024-09-20T22:44:44.627 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.627 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.627 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op must wait for map", 2024-09-20T22:44:44.627 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:42.175123+0000", 2024-09-20T22:44:44.627 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.5639999999999999e-06 2024-09-20T22:44:44.628 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.628 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.628 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:44:44.628 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:42.177191+0000", 2024-09-20T22:44:44.628 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0020681639999999999 2024-09-20T22:44:44.628 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.628 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.628 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-20T22:44:44.628 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:42.177197+0000", 2024-09-20T22:44:44.628 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.5149999999999997e-06 2024-09-20T22:44:44.628 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.628 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.628 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:44:44.628 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:42.186950+0000", 2024-09-20T22:44:44.629 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0097529149999999992 2024-09-20T22:44:44.629 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.629 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.629 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:44:44.629 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:42.187018+0000", 2024-09-20T22:44:44.629 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.8318999999999994e-05 2024-09-20T22:44:44.629 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.629 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.629 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 4,7", 2024-09-20T22:44:44.629 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:42.187129+0000", 2024-09-20T22:44:44.629 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00011135 2024-09-20T22:44:44.629 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.629 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.629 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:44:44.629 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:42.188031+0000", 2024-09-20T22:44:44.630 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000902205 2024-09-20T22:44:44.630 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.630 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.630 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:44:44.630 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:42.188667+0000", 2024-09-20T22:44:44.630 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00063509099999999995 2024-09-20T22:44:44.630 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.630 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.630 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:44:44.631 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:42.190864+0000", 2024-09-20T22:44:44.631 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0021970789999999998 2024-09-20T22:44:44.631 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.631 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.631 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:44:44.631 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:42.190912+0000", 2024-09-20T22:44:44.631 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.8677000000000002e-05 2024-09-20T22:44:44.631 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.631 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.631 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:44:44.632 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:42.190951+0000", 2024-09-20T22:44:44.632 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.8587000000000002e-05 2024-09-20T22:44:44.632 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:44:44.632 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:44:44.632 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:44:44.632 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.632 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.632 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:336 7.7 7:e67788e2:::smithi08477203-31:head [copy-from ver 30] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-09-20T22:44:44.632 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:06.542956+0000", 2024-09-20T22:44:44.632 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 98.069570722999998, 2024-09-20T22:44:44.632 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.019630893999999999, 2024-09-20T22:44:44.632 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:44:44.633 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:44:44.633 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:44:44.633 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:44:44.633 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:44:44.633 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:44:44.633 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 336 2024-09-20T22:44:44.633 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.633 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:44:44.633 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.633 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:44:44.633 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:06.542956+0000", 2024-09-20T22:44:44.633 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:44:44.633 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.634 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.634 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:44:44.634 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:06.542956+0000", 2024-09-20T22:44:44.634 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:44:44.634 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.634 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.634 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:44:44.634 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:06.542957+0000", 2024-09-20T22:44:44.634 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2580000000000001e-06 2024-09-20T22:44:44.634 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.634 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.634 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:44:44.634 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:06.542967+0000", 2024-09-20T22:44:44.634 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0701e-05 2024-09-20T22:44:44.635 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.635 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.635 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:44:44.635 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:06.542987+0000", 2024-09-20T22:44:44.635 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9760000000000001e-05 2024-09-20T22:44:44.635 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.635 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.635 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:44:44.635 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:06.543002+0000", 2024-09-20T22:44:44.635 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4578e-05 2024-09-20T22:44:44.635 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.635 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.635 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:44:44.635 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:06.543070+0000", 2024-09-20T22:44:44.636 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.8263999999999996e-05 2024-09-20T22:44:44.636 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.636 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.636 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:44:44.636 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:06.543159+0000", 2024-09-20T22:44:44.636 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.9241999999999997e-05 2024-09-20T22:44:44.636 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.636 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.636 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 0,3", 2024-09-20T22:44:44.636 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:06.546618+0000", 2024-09-20T22:44:44.636 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0034581949999999998 2024-09-20T22:44:44.636 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.636 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.636 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:44:44.637 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:06.551199+0000", 2024-09-20T22:44:44.637 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0045811309999999996 2024-09-20T22:44:44.637 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.637 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.637 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:44:44.637 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:06.560992+0000", 2024-09-20T22:44:44.637 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0097937270000000003 2024-09-20T22:44:44.637 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.637 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.637 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:44:44.637 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:06.562419+0000", 2024-09-20T22:44:44.637 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0014269040000000001 2024-09-20T22:44:44.637 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.638 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.638 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:44:44.638 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:06.562515+0000", 2024-09-20T22:44:44.638 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.5544999999999994e-05 2024-09-20T22:44:44.638 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.638 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.638 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:44:44.638 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:06.562586+0000", 2024-09-20T22:44:44.638 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.1588999999999995e-05 2024-09-20T22:44:44.638 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:44:44.638 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:44:44.638 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:44:44.639 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.639 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.639 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:348 7.7 7:ed54065f:::smithi08477203-12:head [copy-from ver 6] snapc 6=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-09-20T22:44:44.639 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:07.545584+0000", 2024-09-20T22:44:44.639 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 97.066942505, 2024-09-20T22:44:44.639 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.017264578999999999, 2024-09-20T22:44:44.639 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:44:44.639 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:44:44.639 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:44:44.639 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:44:44.639 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:44:44.639 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:44:44.639 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 348 2024-09-20T22:44:44.640 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.640 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:44:44.640 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.640 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:44:44.640 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:07.545584+0000", 2024-09-20T22:44:44.640 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:44:44.640 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.640 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.640 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:44:44.640 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:07.545584+0000", 2024-09-20T22:44:44.640 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:44:44.640 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.640 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.641 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:44:44.641 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:07.545585+0000", 2024-09-20T22:44:44.641 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4109999999999999e-06 2024-09-20T22:44:44.641 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.641 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.641 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:44:44.641 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:07.545592+0000", 2024-09-20T22:44:44.641 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.6889999999999998e-06 2024-09-20T22:44:44.641 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.641 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.641 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:44:44.641 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:07.545606+0000", 2024-09-20T22:44:44.641 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4634000000000001e-05 2024-09-20T22:44:44.641 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.642 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.642 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:44:44.642 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:07.545615+0000", 2024-09-20T22:44:44.642 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.2509999999999994e-06 2024-09-20T22:44:44.642 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.642 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.642 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:44:44.642 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:07.545677+0000", 2024-09-20T22:44:44.642 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.2316000000000004e-05 2024-09-20T22:44:44.642 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.642 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.642 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op must wait for map", 2024-09-20T22:44:44.642 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:07.545683+0000", 2024-09-20T22:44:44.642 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.1859999999999997e-06 2024-09-20T22:44:44.643 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.643 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.643 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:44:44.643 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:07.549696+0000", 2024-09-20T22:44:44.643 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0040129129999999999 2024-09-20T22:44:44.643 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.643 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.643 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:44:44.643 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:07.549785+0000", 2024-09-20T22:44:44.643 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.8697999999999997e-05 2024-09-20T22:44:44.643 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.643 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.643 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 0,3", 2024-09-20T22:44:44.644 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:07.552497+0000", 2024-09-20T22:44:44.644 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0027121200000000002 2024-09-20T22:44:44.644 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.644 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.644 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:44:44.644 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:07.555493+0000", 2024-09-20T22:44:44.644 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0029957260000000002 2024-09-20T22:44:44.644 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.644 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.644 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:44:44.644 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:07.560584+0000", 2024-09-20T22:44:44.644 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:44:44.646 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0050909220000000003 2024-09-20T22:44:44.646 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.646 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.646 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:44:44.646 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:07.562727+0000", 2024-09-20T22:44:44.646 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0021436570000000002 2024-09-20T22:44:44.646 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.646 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.646 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:44:44.647 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:07.562802+0000", 2024-09-20T22:44:44.647 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.4738000000000002e-05 2024-09-20T22:44:44.647 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.647 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.647 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:44:44.647 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:07.562848+0000", 2024-09-20T22:44:44.647 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.6317999999999999e-05 2024-09-20T22:44:44.647 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:44:44.647 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:44:44.647 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:44:44.647 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.647 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.647 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:540 7.c 7:39b84aab:::smithi08477203-19:head [copy-from ver 26] snapc 13=[13,12,11,10,f,d] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-09-20T22:44:44.648 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:27.123339+0000", 2024-09-20T22:44:44.648 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 77.489187720999993, 2024-09-20T22:44:44.648 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.017326617999999998, 2024-09-20T22:44:44.648 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:44:44.648 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:44:44.648 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:44:44.648 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:44:44.648 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:44:44.648 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:44:44.648 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 540 2024-09-20T22:44:44.648 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.648 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:44:44.648 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.648 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:44:44.649 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.123339+0000", 2024-09-20T22:44:44.649 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:44:44.649 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.649 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.649 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:44:44.649 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.123339+0000", 2024-09-20T22:44:44.649 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:44:44.649 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.649 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.649 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:44:44.649 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.123341+0000", 2024-09-20T22:44:44.649 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.3099999999999999e-06 2024-09-20T22:44:44.649 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.649 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.650 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:44:44.650 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.123351+0000", 2024-09-20T22:44:44.650 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0185e-05 2024-09-20T22:44:44.650 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.650 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.650 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:44:44.650 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.123372+0000", 2024-09-20T22:44:44.650 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.143e-05 2024-09-20T22:44:44.650 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.650 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.650 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:44:44.650 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.123385+0000", 2024-09-20T22:44:44.650 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2279000000000001e-05 2024-09-20T22:44:44.651 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.651 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.651 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:44:44.651 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.123467+0000", 2024-09-20T22:44:44.651 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.2218999999999993e-05 2024-09-20T22:44:44.651 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.651 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.651 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op must wait for map", 2024-09-20T22:44:44.651 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.123474+0000", 2024-09-20T22:44:44.651 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.323e-06 2024-09-20T22:44:44.651 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.651 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.651 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:44:44.651 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.125408+0000", 2024-09-20T22:44:44.652 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001933883 2024-09-20T22:44:44.652 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.652 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.652 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:44:44.652 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.125464+0000", 2024-09-20T22:44:44.652 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.5649000000000002e-05 2024-09-20T22:44:44.652 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.652 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.652 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 3,7", 2024-09-20T22:44:44.652 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.128846+0000", 2024-09-20T22:44:44.652 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0033820999999999999 2024-09-20T22:44:44.652 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.652 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.652 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:44:44.653 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.134732+0000", 2024-09-20T22:44:44.653 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0058861310000000002 2024-09-20T22:44:44.653 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.653 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.653 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:44:44.653 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.137919+0000", 2024-09-20T22:44:44.653 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0031871080000000001 2024-09-20T22:44:44.653 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.653 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.653 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:44:44.653 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.140579+0000", 2024-09-20T22:44:44.653 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0026593979999999999 2024-09-20T22:44:44.653 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.654 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.654 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:44:44.654 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.140618+0000", 2024-09-20T22:44:44.654 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.9286000000000003e-05 2024-09-20T22:44:44.654 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.654 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.654 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:44:44.654 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.140665+0000", 2024-09-20T22:44:44.654 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.7317000000000001e-05 2024-09-20T22:44:44.654 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:44:44.654 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:44:44.654 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:44:44.654 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.655 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.655 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:654 7.7 7:ea720b85:::smithi08477203-26:head [append 0~0] snapc 17=[17,16,15,13,11,d] ondisk+write+known_if_redirected+supports_pool_eio e136)", 2024-09-20T22:44:44.655 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:35.126065+0000", 2024-09-20T22:44:44.655 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 69.486460793000006, 2024-09-20T22:44:44.655 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.069757472000000001, 2024-09-20T22:44:44.655 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:44:44.655 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:44:44.655 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:44:44.655 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:44:44.655 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:44:44.655 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:44:44.655 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 654 2024-09-20T22:44:44.655 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.655 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:44:44.656 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.656 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:44:44.656 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.126065+0000", 2024-09-20T22:44:44.656 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:44:44.656 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.656 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.656 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:44:44.656 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.126065+0000", 2024-09-20T22:44:44.656 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:44:44.656 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.656 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.656 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:44:44.656 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.126067+0000", 2024-09-20T22:44:44.657 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.6139999999999999e-06 2024-09-20T22:44:44.657 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.657 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.657 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:44:44.657 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.126074+0000", 2024-09-20T22:44:44.657 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.1509999999999998e-06 2024-09-20T22:44:44.657 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.657 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.657 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:44:44.657 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.126090+0000", 2024-09-20T22:44:44.657 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.6001000000000001e-05 2024-09-20T22:44:44.657 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.657 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.657 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:44:44.658 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.126098+0000", 2024-09-20T22:44:44.658 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.2379999999999997e-06 2024-09-20T22:44:44.658 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.658 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.658 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:44:44.658 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.126155+0000", 2024-09-20T22:44:44.658 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.6279000000000001e-05 2024-09-20T22:44:44.658 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.658 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.658 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op must wait for map", 2024-09-20T22:44:44.658 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.126160+0000", 2024-09-20T22:44:44.658 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.8110000000000002e-06 2024-09-20T22:44:44.658 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.659 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.659 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:44:44.659 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.193660+0000", 2024-09-20T22:44:44.659 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.067499997000000006 2024-09-20T22:44:44.659 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.659 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.659 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:44:44.659 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.193751+0000", 2024-09-20T22:44:44.659 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.1571999999999997e-05 2024-09-20T22:44:44.659 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.659 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.659 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 0,3", 2024-09-20T22:44:44.659 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.193941+0000", 2024-09-20T22:44:44.660 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00018952699999999999 2024-09-20T22:44:44.660 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.660 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.660 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:44:44.660 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.195649+0000", 2024-09-20T22:44:44.660 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001707966 2024-09-20T22:44:44.660 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.660 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.660 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:44:44.660 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.195707+0000", 2024-09-20T22:44:44.660 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.8649e-05 2024-09-20T22:44:44.660 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.660 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.661 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:44:44.661 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.195722+0000", 2024-09-20T22:44:44.661 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4309e-05 2024-09-20T22:44:44.661 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.661 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.661 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:44:44.661 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.195773+0000", 2024-09-20T22:44:44.661 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.1368e-05 2024-09-20T22:44:44.661 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.661 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.661 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:44:44.661 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.195823+0000", 2024-09-20T22:44:44.661 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.9990000000000001e-05 2024-09-20T22:44:44.661 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:44:44.661 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:44:44.662 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:44:44.662 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.662 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.662 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:655 7.7 7:ea720b85:::smithi08477203-26:head [rollback 16] snapc 17=[17,16,15,13,11,d] ondisk+write+known_if_redirected+supports_pool_eio e136)", 2024-09-20T22:44:44.662 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:35.126118+0000", 2024-09-20T22:44:44.662 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 69.486408539999999, 2024-09-20T22:44:44.662 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.069926453, 2024-09-20T22:44:44.662 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:44:44.662 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:44:44.662 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:44:44.662 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:44:44.662 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:44:44.662 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:44:44.663 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 655 2024-09-20T22:44:44.663 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.663 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:44:44.663 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.663 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:44:44.663 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.126118+0000", 2024-09-20T22:44:44.663 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:44:44.663 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.663 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.663 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:44:44.663 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.126118+0000", 2024-09-20T22:44:44.663 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:44:44.663 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.663 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.664 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:44:44.664 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.126118+0000", 2024-09-20T22:44:44.664 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.3000000000000001e-07 2024-09-20T22:44:44.664 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.664 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.664 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:44:44.664 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.126123+0000", 2024-09-20T22:44:44.664 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.7650000000000001e-06 2024-09-20T22:44:44.664 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.664 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.664 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:44:44.664 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.126141+0000", 2024-09-20T22:44:44.664 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.7538e-05 2024-09-20T22:44:44.665 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.665 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.665 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:44:44.665 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.126150+0000", 2024-09-20T22:44:44.665 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.3549999999999994e-06 2024-09-20T22:44:44.665 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.665 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.665 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:44:44.665 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.126204+0000", 2024-09-20T22:44:44.665 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.4289999999999997e-05 2024-09-20T22:44:44.665 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.665 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.665 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting_for_map not empty", 2024-09-20T22:44:44.665 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.126210+0000", 2024-09-20T22:44:44.665 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.361e-06 2024-09-20T22:44:44.666 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.666 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.666 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:44:44.666 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.194528+0000", 2024-09-20T22:44:44.666 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.068318913999999994 2024-09-20T22:44:44.666 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.666 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.666 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:44:44.666 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.194614+0000", 2024-09-20T22:44:44.666 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.5280999999999999e-05 2024-09-20T22:44:44.666 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.666 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.666 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 0,3", 2024-09-20T22:44:44.667 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.194759+0000", 2024-09-20T22:44:44.667 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000145678 2024-09-20T22:44:44.667 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.667 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.667 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:44:44.667 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.195836+0000", 2024-09-20T22:44:44.667 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001076474 2024-09-20T22:44:44.667 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.667 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.667 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:44:44.667 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.195884+0000", 2024-09-20T22:44:44.667 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.8421000000000003e-05 2024-09-20T22:44:44.667 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.667 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.668 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:44:44.668 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.195947+0000", 2024-09-20T22:44:44.668 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.2522999999999998e-05 2024-09-20T22:44:44.668 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.668 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.668 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:44:44.668 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.196003+0000", 2024-09-20T22:44:44.668 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.5911000000000001e-05 2024-09-20T22:44:44.668 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.668 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.668 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:44:44.668 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.196044+0000", 2024-09-20T22:44:44.668 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.1412000000000003e-05 2024-09-20T22:44:44.669 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:44:44.669 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:44:44.669 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:44:44.669 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.669 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.669 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:656 7.7 7:ea720b85:::smithi08477203-26:head [append 0~0] snapc 17=[17,16,15,13,11,d] ondisk+write+known_if_redirected+supports_pool_eio e136)", 2024-09-20T22:44:44.669 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:35.126159+0000", 2024-09-20T22:44:44.669 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 69.486367705999996, 2024-09-20T22:44:44.669 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.070101607999999996, 2024-09-20T22:44:44.669 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:44:44.669 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:44:44.669 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:44:44.669 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:44:44.669 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:44:44.669 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:44:44.670 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 656 2024-09-20T22:44:44.670 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.670 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:44:44.670 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.670 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:44:44.670 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.126159+0000", 2024-09-20T22:44:44.670 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:44:44.670 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.670 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.670 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:44:44.670 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.126159+0000", 2024-09-20T22:44:44.670 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:44:44.670 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.671 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.671 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:44:44.671 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.126159+0000", 2024-09-20T22:44:44.671 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.1500000000000005e-07 2024-09-20T22:44:44.671 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.671 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.671 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:44:44.671 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.126168+0000", 2024-09-20T22:44:44.671 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.4600000000000003e-06 2024-09-20T22:44:44.671 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.671 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.671 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:44:44.671 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.126173+0000", 2024-09-20T22:44:44.671 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.7089999999999999e-06 2024-09-20T22:44:44.672 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.672 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.672 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:44:44.672 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.126177+0000", 2024-09-20T22:44:44.672 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.9040000000000002e-06 2024-09-20T22:44:44.672 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.672 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.672 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:44:44.672 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.126232+0000", 2024-09-20T22:44:44.672 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.5337999999999998e-05 2024-09-20T22:44:44.672 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.672 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.672 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting_for_map not empty", 2024-09-20T22:44:44.672 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.126238+0000", 2024-09-20T22:44:44.672 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.2390000000000002e-06 2024-09-20T22:44:44.673 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.673 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.673 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:44:44.673 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.195116+0000", 2024-09-20T22:44:44.673 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.068878012000000002 2024-09-20T22:44:44.673 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.673 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.673 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:44:44.673 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.195197+0000", 2024-09-20T22:44:44.673 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.0964999999999995e-05 2024-09-20T22:44:44.673 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.673 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.673 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 0,3", 2024-09-20T22:44:44.674 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.195338+0000", 2024-09-20T22:44:44.674 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000141006 2024-09-20T22:44:44.674 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.674 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.674 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:44:44.674 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.195897+0000", 2024-09-20T22:44:44.674 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00055979999999999995 2024-09-20T22:44:44.674 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.674 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.674 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:44:44.674 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.196102+0000", 2024-09-20T22:44:44.674 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00020405099999999999 2024-09-20T22:44:44.674 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.674 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.675 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:44:44.675 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.196160+0000", 2024-09-20T22:44:44.675 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.8677000000000001e-05 2024-09-20T22:44:44.675 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.675 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.675 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:44:44.675 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.196214+0000", 2024-09-20T22:44:44.675 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.414e-05 2024-09-20T22:44:44.675 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.675 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.675 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:44:44.675 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.196260+0000", 2024-09-20T22:44:44.675 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.5791999999999998e-05 2024-09-20T22:44:44.675 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:44:44.676 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:44:44.676 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:44:44.676 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.676 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.676 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:726 7.19 7:9eac4560:::smithi08477203-30:head [copy-from ver 81] snapc 18=[17] ondisk+write+known_if_redirected+supports_pool_eio e143)", 2024-09-20T22:44:44.676 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:42.325306+0000", 2024-09-20T22:44:44.676 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 62.287219835999998, 2024-09-20T22:44:44.676 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.018367953999999999, 2024-09-20T22:44:44.676 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:44:44.676 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:44:44.676 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:44:44.676 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:44:44.676 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:44:44.677 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:44:44.677 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 726 2024-09-20T22:44:44.677 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.677 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:44:44.677 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.677 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:44:44.677 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.325306+0000", 2024-09-20T22:44:44.677 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:44:44.677 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.677 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.677 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:44:44.677 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.325306+0000", 2024-09-20T22:44:44.677 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:44:44.677 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.677 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.678 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:44:44.678 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.325307+0000", 2024-09-20T22:44:44.678 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.4400000000000001e-07 2024-09-20T22:44:44.678 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.678 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.678 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:44:44.678 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.325309+0000", 2024-09-20T22:44:44.678 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.5500000000000001e-06 2024-09-20T22:44:44.678 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.678 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.678 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:44:44.678 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.325316+0000", 2024-09-20T22:44:44.678 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.4640000000000003e-06 2024-09-20T22:44:44.679 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.679 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.679 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:44:44.679 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.325321+0000", 2024-09-20T22:44:44.679 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.977e-06 2024-09-20T22:44:44.679 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.679 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.679 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:44:44.679 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.325460+0000", 2024-09-20T22:44:44.679 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00013921199999999999 2024-09-20T22:44:44.679 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.679 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.679 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:44:44.680 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.325532+0000", 2024-09-20T22:44:44.680 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.2355999999999996e-05 2024-09-20T22:44:44.680 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.680 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.680 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 2,7", 2024-09-20T22:44:44.680 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.332251+0000", 2024-09-20T22:44:44.680 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0067184000000000002 2024-09-20T22:44:44.680 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.680 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.680 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:44:44.680 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.342590+0000", 2024-09-20T22:44:44.680 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.010339355 2024-09-20T22:44:44.680 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.680 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.681 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:44:44.681 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.342631+0000", 2024-09-20T22:44:44.681 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.1053999999999997e-05 2024-09-20T22:44:44.681 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.681 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.681 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:44:44.681 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.343558+0000", 2024-09-20T22:44:44.681 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00092629099999999999 2024-09-20T22:44:44.681 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.681 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.681 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:44:44.681 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.343620+0000", 2024-09-20T22:44:44.681 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.2836999999999995e-05 2024-09-20T22:44:44.681 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.682 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.682 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:44:44.682 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.343674+0000", 2024-09-20T22:44:44.682 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.4014000000000002e-05 2024-09-20T22:44:44.682 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:44:44.682 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:44:44.682 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:44:44.682 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.682 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.682 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:921 7.19 7:9eac4560:::smithi08477203-30:head [write 487436~547559,stat] snapc 1f=[] ondisk+write+known_if_redirected+supports_pool_eio e158)", 2024-09-20T22:44:44.682 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:57.424404+0000", 2024-09-20T22:44:44.682 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 47.188121995000003, 2024-09-20T22:44:44.682 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.018385357000000001, 2024-09-20T22:44:44.682 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:44:44.683 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:44:44.683 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:44:44.683 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:44:44.683 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:44:44.683 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:44:44.683 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 921 2024-09-20T22:44:44.683 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.683 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:44:44.683 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.683 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:44:44.683 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:57.424404+0000", 2024-09-20T22:44:44.683 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:44:44.683 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.683 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.684 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:44:44.684 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:57.424404+0000", 2024-09-20T22:44:44.684 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:44:44.684 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.684 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.684 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:44:44.684 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:57.424406+0000", 2024-09-20T22:44:44.684 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.6169999999999999e-06 2024-09-20T22:44:44.684 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.684 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.684 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:44:44.684 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:57.428411+0000", 2024-09-20T22:44:44.684 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0040051109999999996 2024-09-20T22:44:44.684 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.685 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.685 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:44:44.685 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:57.428423+0000", 2024-09-20T22:44:44.685 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1590000000000001e-05 2024-09-20T22:44:44.685 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.685 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.685 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:44:44.685 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:57.428432+0000", 2024-09-20T22:44:44.685 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.781e-06 2024-09-20T22:44:44.685 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.685 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.685 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:44:44.685 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:57.431029+0000", 2024-09-20T22:44:44.685 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0025962139999999999 2024-09-20T22:44:44.685 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.686 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.686 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:44:44.686 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:57.431170+0000", 2024-09-20T22:44:44.686 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00014111199999999999 2024-09-20T22:44:44.686 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.686 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.686 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 2,7", 2024-09-20T22:44:44.686 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:57.431332+0000", 2024-09-20T22:44:44.686 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00016258599999999999 2024-09-20T22:44:44.686 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.686 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.686 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:44:44.686 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:57.433472+0000", 2024-09-20T22:44:44.687 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0021397120000000002 2024-09-20T22:44:44.687 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.687 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.687 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:44:44.687 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:57.437960+0000", 2024-09-20T22:44:44.687 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0044878410000000002 2024-09-20T22:44:44.687 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.687 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.687 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:44:44.687 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:57.442690+0000", 2024-09-20T22:44:44.687 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0047299250000000003 2024-09-20T22:44:44.687 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.687 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.687 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:44:44.688 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:57.442743+0000", 2024-09-20T22:44:44.688 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.3368999999999997e-05 2024-09-20T22:44:44.688 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.688 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.688 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:44:44.688 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:57.442790+0000", 2024-09-20T22:44:44.688 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.6499000000000003e-05 2024-09-20T22:44:44.688 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:44:44.688 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:44:44.688 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:44:44.688 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.688 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.688 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:1017 7.19 7:9cc0b7e2:::smithi08477203-46:head [copy-from ver 99] snapc 25=[25,24,23,21] ondisk+write+known_if_redirected+supports_pool_eio e166)", 2024-09-20T22:44:44.688 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:44:05.130713+0000", 2024-09-20T22:44:44.688 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 39.481813158999998, 2024-09-20T22:44:44.689 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.021441722999999999, 2024-09-20T22:44:44.689 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:44:44.689 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:44:44.689 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:44:44.689 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:44:44.689 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:44:44.689 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:44:44.689 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 1017 2024-09-20T22:44:44.689 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.689 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:44:44.689 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.689 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:44:44.689 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:05.130713+0000", 2024-09-20T22:44:44.689 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:44:44.690 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.690 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.690 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:44:44.690 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:05.130713+0000", 2024-09-20T22:44:44.690 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:44:44.690 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.690 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.690 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:44:44.690 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:05.130715+0000", 2024-09-20T22:44:44.690 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.66e-06 2024-09-20T22:44:44.690 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.690 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.690 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:44:44.691 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:05.130721+0000", 2024-09-20T22:44:44.691 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.0430000000000004e-06 2024-09-20T22:44:44.691 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.691 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.691 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:44:44.691 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:05.130732+0000", 2024-09-20T22:44:44.691 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1494999999999999e-05 2024-09-20T22:44:44.691 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.691 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.691 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:44:44.691 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:05.130748+0000", 2024-09-20T22:44:44.691 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.5520999999999999e-05 2024-09-20T22:44:44.691 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.691 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.692 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:44:44.692 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:05.130827+0000", 2024-09-20T22:44:44.692 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.8990000000000001e-05 2024-09-20T22:44:44.692 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.692 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.692 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op must wait for map", 2024-09-20T22:44:44.692 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:05.130834+0000", 2024-09-20T22:44:44.692 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.097e-06 2024-09-20T22:44:44.692 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.692 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.692 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:44:44.692 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:05.133176+0000", 2024-09-20T22:44:44.692 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.002341704 2024-09-20T22:44:44.692 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.693 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.693 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:44:44.693 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:05.133285+0000", 2024-09-20T22:44:44.693 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000109473 2024-09-20T22:44:44.693 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.693 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.693 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 2,7", 2024-09-20T22:44:44.693 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:05.142516+0000", 2024-09-20T22:44:44.693 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0092311130000000009 2024-09-20T22:44:44.693 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.693 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.693 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:44:44.693 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:05.150724+0000", 2024-09-20T22:44:44.693 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0082083139999999995 2024-09-20T22:44:44.694 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.694 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.694 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:44:44.694 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:05.150825+0000", 2024-09-20T22:44:44.694 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000100431 2024-09-20T22:44:44.694 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.694 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.694 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:44:44.694 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:05.152061+0000", 2024-09-20T22:44:44.694 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001235912 2024-09-20T22:44:44.694 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.694 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.694 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:44:44.694 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:05.152108+0000", 2024-09-20T22:44:44.695 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.7068999999999999e-05 2024-09-20T22:44:44.695 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.695 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.695 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:44:44.695 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:05.152155+0000", 2024-09-20T22:44:44.695 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.6900999999999997e-05 2024-09-20T22:44:44.695 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:44:44.695 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:44:44.695 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:44:44.695 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.695 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.695 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:1018 7.19 7:9cc0b7e2:::smithi08477203-46:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e166)", 2024-09-20T22:44:44.695 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:44:05.130767+0000", 2024-09-20T22:44:44.695 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 39.481759267000001, 2024-09-20T22:44:44.696 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.016100247000000002, 2024-09-20T22:44:44.696 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:44:44.696 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:44:44.696 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:44:44.696 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:44:44.696 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:44:44.696 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:44:44.696 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 1018 2024-09-20T22:44:44.696 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.696 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:44:44.696 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.696 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:44:44.696 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:05.130767+0000", 2024-09-20T22:44:44.696 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:44:44.697 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.697 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.697 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:44:44.697 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:05.130767+0000", 2024-09-20T22:44:44.697 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:44:44.697 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.697 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.697 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:44:44.697 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:05.130768+0000", 2024-09-20T22:44:44.697 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.3799999999999997e-07 2024-09-20T22:44:44.697 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.697 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.697 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:44:44.697 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:05.130771+0000", 2024-09-20T22:44:44.697 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.0369999999999999e-06 2024-09-20T22:44:44.698 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.698 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.698 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:44:44.698 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:05.130781+0000", 2024-09-20T22:44:44.698 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0251000000000001e-05 2024-09-20T22:44:44.698 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.698 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.698 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:44:44.698 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:05.130786+0000", 2024-09-20T22:44:44.698 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.2349999999999999e-06 2024-09-20T22:44:44.698 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.698 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.698 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:44:44.699 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:05.130886+0000", 2024-09-20T22:44:44.699 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.9605000000000006e-05 2024-09-20T22:44:44.699 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.699 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.699 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting_for_map not empty", 2024-09-20T22:44:44.699 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:05.130892+0000", 2024-09-20T22:44:44.699 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.8880000000000002e-06 2024-09-20T22:44:44.699 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.699 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.699 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:44:44.699 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:05.133779+0000", 2024-09-20T22:44:44.699 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.002887429 2024-09-20T22:44:44.699 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.699 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.700 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for blocked object", 2024-09-20T22:44:44.700 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:05.133841+0000", 2024-09-20T22:44:44.700 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.2088999999999995e-05 2024-09-20T22:44:44.700 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.700 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.700 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:44:44.700 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:05.146714+0000", 2024-09-20T22:44:44.700 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.01287256 2024-09-20T22:44:44.700 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.700 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.700 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:44:44.700 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:05.146786+0000", 2024-09-20T22:44:44.700 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.2751999999999997e-05 2024-09-20T22:44:44.701 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.701 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.701 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:44:44.701 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:05.146867+0000", 2024-09-20T22:44:44.701 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.0762999999999999e-05 2024-09-20T22:44:44.701 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:44:44.701 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:44:44.701 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:44:44.701 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.701 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.701 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:1054 7.12 7:4fa4b2b3:::smithi08477203-21:head [copy-from ver 67] snapc 25=[25,24,23,21] ondisk+write+known_if_redirected+supports_pool_eio e166)", 2024-09-20T22:44:44.701 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:44:05.634022+0000", 2024-09-20T22:44:44.701 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 38.978503881000002, 2024-09-20T22:44:44.702 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.019656218, 2024-09-20T22:44:44.702 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:44:44.702 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:44:44.702 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:44:44.702 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:44:44.702 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:44:44.702 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:44:44.702 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 1054 2024-09-20T22:44:44.702 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.702 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:44:44.702 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.702 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:44:44.702 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:05.634022+0000", 2024-09-20T22:44:44.702 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:44:44.703 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.703 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.703 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:44:44.703 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:05.634022+0000", 2024-09-20T22:44:44.703 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:44:44.703 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.703 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.703 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:44:44.703 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:05.634024+0000", 2024-09-20T22:44:44.703 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.013e-06 2024-09-20T22:44:44.703 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.703 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.703 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:44:44.704 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:05.634035+0000", 2024-09-20T22:44:44.704 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0644e-05 2024-09-20T22:44:44.704 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.704 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.704 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:44:44.704 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:05.634061+0000", 2024-09-20T22:44:44.704 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.6352000000000001e-05 2024-09-20T22:44:44.704 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.704 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.704 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:44:44.704 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:05.634081+0000", 2024-09-20T22:44:44.704 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9347e-05 2024-09-20T22:44:44.704 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.705 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.705 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:44:44.705 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:05.634257+0000", 2024-09-20T22:44:44.705 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000176365 2024-09-20T22:44:44.705 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.705 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.705 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:44:44.705 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:05.634379+0000", 2024-09-20T22:44:44.705 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000122173 2024-09-20T22:44:44.705 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.705 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.705 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 0,2", 2024-09-20T22:44:44.705 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:05.643905+0000", 2024-09-20T22:44:44.705 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0095258679999999998 2024-09-20T22:44:44.706 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.706 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.706 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:44:44.706 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:05.647750+0000", 2024-09-20T22:44:44.706 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.003844564 2024-09-20T22:44:44.706 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.706 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.706 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:44:44.706 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:05.650803+0000", 2024-09-20T22:44:44.706 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.003053114 2024-09-20T22:44:44.706 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.706 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.706 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:44:44.706 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:05.653610+0000", 2024-09-20T22:44:44.707 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0028071459999999999 2024-09-20T22:44:44.707 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.707 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.707 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:44:44.707 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:05.653645+0000", 2024-09-20T22:44:44.707 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.5231000000000002e-05 2024-09-20T22:44:44.707 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.707 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.707 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:44:44.707 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:05.653679+0000", 2024-09-20T22:44:44.707 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.3401000000000001e-05 2024-09-20T22:44:44.707 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:44:44.707 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:44:44.707 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:44:44.708 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.708 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.708 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:1213 7.c 7:3f0fe232:::smithi08477203-4:head [copy-from ver 87] snapc 29=[] ondisk+write+known_if_redirected+supports_pool_eio e178)", 2024-09-20T22:44:44.708 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:44:17.228357+0000", 2024-09-20T22:44:44.708 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 27.384168990999999, 2024-09-20T22:44:44.708 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.016261893999999999, 2024-09-20T22:44:44.708 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:44:44.708 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:44:44.708 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:44:44.708 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:44:44.708 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:44:44.708 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:44:44.708 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 1213 2024-09-20T22:44:44.708 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.709 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:44:44.709 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.709 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:44:44.709 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:17.228357+0000", 2024-09-20T22:44:44.709 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:44:44.709 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.709 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.709 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:44:44.709 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:17.228357+0000", 2024-09-20T22:44:44.709 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:44:44.709 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.709 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.709 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:44:44.709 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:17.228358+0000", 2024-09-20T22:44:44.710 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.9299999999999999e-07 2024-09-20T22:44:44.710 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.710 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.710 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:44:44.710 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:17.228360+0000", 2024-09-20T22:44:44.710 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.0490000000000002e-06 2024-09-20T22:44:44.710 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.710 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.710 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:44:44.710 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:17.228366+0000", 2024-09-20T22:44:44.710 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.173e-06 2024-09-20T22:44:44.710 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.710 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.710 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:44:44.711 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:17.228369+0000", 2024-09-20T22:44:44.711 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.1480000000000002e-06 2024-09-20T22:44:44.711 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.711 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.711 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:44:44.711 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:17.230954+0000", 2024-09-20T22:44:44.711 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0025853859999999998 2024-09-20T22:44:44.711 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.711 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.711 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:44:44.711 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:17.231004+0000", 2024-09-20T22:44:44.711 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.9554e-05 2024-09-20T22:44:44.711 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.712 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.712 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 3,7", 2024-09-20T22:44:44.712 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:17.237905+0000", 2024-09-20T22:44:44.712 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0069007110000000003 2024-09-20T22:44:44.712 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.712 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.712 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:44:44.712 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:17.242877+0000", 2024-09-20T22:44:44.712 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0049725999999999998 2024-09-20T22:44:44.712 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.712 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.712 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:44:44.712 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:17.244255+0000", 2024-09-20T22:44:44.712 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00137739 2024-09-20T22:44:44.712 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.713 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.713 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:44:44.713 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:17.244552+0000", 2024-09-20T22:44:44.713 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000297819 2024-09-20T22:44:44.713 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.713 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.713 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:44:44.713 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:17.244587+0000", 2024-09-20T22:44:44.713 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.4860000000000002e-05 2024-09-20T22:44:44.713 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.713 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.713 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:44:44.713 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:17.244619+0000", 2024-09-20T22:44:44.713 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.1810999999999997e-05 2024-09-20T22:44:44.714 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:44:44.714 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:44:44.714 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:44:44.714 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.714 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.714 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:1420 7.19 7:9cc0b7e2:::smithi08477203-46:head [write 480733~537415,stat] snapc 2f=[2f] ondisk+write+known_if_redirected+supports_pool_eio e189)", 2024-09-20T22:44:44.714 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:44:28.143096+0000", 2024-09-20T22:44:44.714 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 16.469430362000001, 2024-09-20T22:44:44.714 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.021594787000000001, 2024-09-20T22:44:44.714 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:44:44.714 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:44:44.714 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:44:44.714 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:44:44.714 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:44:44.715 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:44:44.715 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 1420 2024-09-20T22:44:44.715 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.715 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:44:44.715 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.715 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:44:44.715 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:28.143096+0000", 2024-09-20T22:44:44.715 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:44:44.715 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.715 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.715 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:44:44.715 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:28.143096+0000", 2024-09-20T22:44:44.715 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:44:44.716 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.716 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.716 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:44:44.716 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:28.143098+0000", 2024-09-20T22:44:44.716 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.1840000000000002e-06 2024-09-20T22:44:44.716 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.716 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.716 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:44:44.716 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:28.143663+0000", 2024-09-20T22:44:44.716 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00056486000000000004 2024-09-20T22:44:44.716 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.716 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.716 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:44:44.717 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:28.143684+0000", 2024-09-20T22:44:44.717 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.0792e-05 2024-09-20T22:44:44.717 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.717 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.717 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:44:44.717 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:28.143697+0000", 2024-09-20T22:44:44.717 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.292e-05 2024-09-20T22:44:44.717 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.717 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.717 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:44:44.717 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:28.143788+0000", 2024-09-20T22:44:44.717 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.1218999999999995e-05 2024-09-20T22:44:44.717 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.717 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.717 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op must wait for map", 2024-09-20T22:44:44.718 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:28.143798+0000", 2024-09-20T22:44:44.718 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.7999999999999993e-06 2024-09-20T22:44:44.718 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.718 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.718 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:44:44.718 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:28.148428+0000", 2024-09-20T22:44:44.718 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0046303780000000001 2024-09-20T22:44:44.718 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.718 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.718 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:44:44.718 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:28.148611+0000", 2024-09-20T22:44:44.718 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00018319699999999999 2024-09-20T22:44:44.719 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.719 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.719 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 2,7", 2024-09-20T22:44:44.719 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:28.148807+0000", 2024-09-20T22:44:44.719 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00019560399999999999 2024-09-20T22:44:44.719 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.719 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.719 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:44:44.719 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:28.157814+0000", 2024-09-20T22:44:44.719 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0090068449999999994 2024-09-20T22:44:44.719 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.719 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.719 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:44:44.719 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:28.158746+0000", 2024-09-20T22:44:44.719 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000932076 2024-09-20T22:44:44.720 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.720 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.720 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:44:44.720 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:28.164580+0000", 2024-09-20T22:44:44.720 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0058337850000000002 2024-09-20T22:44:44.720 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.720 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.720 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:44:44.720 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:28.164638+0000", 2024-09-20T22:44:44.720 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.8100999999999998e-05 2024-09-20T22:44:44.720 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.720 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.720 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:44:44.720 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:28.164691+0000", 2024-09-20T22:44:44.721 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.3025999999999997e-05 2024-09-20T22:44:44.721 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:44:44.721 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:44:44.721 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:44:44.721 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.721 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.721 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:1421 7.19 7:9cc0b7e2:::smithi08477203-46:head [write 1782380~696402] snapc 2f=[2f] ondisk+write+known_if_redirected+supports_pool_eio e189)", 2024-09-20T22:44:44.721 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:44:28.153749+0000", 2024-09-20T22:44:44.721 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 16.458777481999999, 2024-09-20T22:44:44.721 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.016504498999999999, 2024-09-20T22:44:44.721 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:44:44.721 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:44:44.721 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:44:44.721 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:44:44.722 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:44:44.722 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:44:44.722 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 1421 2024-09-20T22:44:44.722 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.722 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:44:44.722 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.722 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:44:44.722 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:28.153749+0000", 2024-09-20T22:44:44.722 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:44:44.722 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.722 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.722 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:44:44.722 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:28.153749+0000", 2024-09-20T22:44:44.722 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:44:44.723 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.723 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.723 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:44:44.723 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:28.153751+0000", 2024-09-20T22:44:44.723 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.826e-06 2024-09-20T22:44:44.723 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.723 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.723 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:44:44.723 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:28.154385+0000", 2024-09-20T22:44:44.723 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000634171 2024-09-20T22:44:44.723 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.723 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.723 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:44:44.723 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:28.154401+0000", 2024-09-20T22:44:44.724 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.6132000000000001e-05 2024-09-20T22:44:44.724 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.724 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.724 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:44:44.724 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:28.154408+0000", 2024-09-20T22:44:44.724 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.99e-06 2024-09-20T22:44:44.724 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.724 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.724 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:44:44.724 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:28.154480+0000", 2024-09-20T22:44:44.724 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.1873999999999997e-05 2024-09-20T22:44:44.724 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.724 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.724 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:44:44.724 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:28.154592+0000", 2024-09-20T22:44:44.725 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000112131 2024-09-20T22:44:44.725 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.725 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.725 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 2,7", 2024-09-20T22:44:44.725 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:28.154752+0000", 2024-09-20T22:44:44.725 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00016003999999999999 2024-09-20T22:44:44.725 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.725 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.725 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:44:44.725 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:28.157824+0000", 2024-09-20T22:44:44.725 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0030724900000000002 2024-09-20T22:44:44.725 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.725 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.725 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:44:44.725 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:28.160047+0000", 2024-09-20T22:44:44.725 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0022228439999999999 2024-09-20T22:44:44.726 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.726 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.726 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:44:44.726 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:28.170178+0000", 2024-09-20T22:44:44.726 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.010130521999999999 2024-09-20T22:44:44.726 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.726 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.726 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:44:44.726 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:28.170225+0000", 2024-09-20T22:44:44.726 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.7348999999999997e-05 2024-09-20T22:44:44.726 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.726 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.726 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:44:44.726 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:28.170253+0000", 2024-09-20T22:44:44.727 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.813e-05 2024-09-20T22:44:44.727 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:44:44.727 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:44:44.727 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:44:44.727 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.727 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.727 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:1525 7.c 7:3f865ef5:::smithi08477203-28:head [copy-from ver 127] snapc 33=[33] ondisk+write+known_if_redirected+supports_pool_eio e197)", 2024-09-20T22:44:44.727 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:44:36.198921+0000", 2024-09-20T22:44:44.727 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 8.4136054579999993, 2024-09-20T22:44:44.727 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.028621220999999999, 2024-09-20T22:44:44.727 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:44:44.727 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:44:44.727 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:44:44.728 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:44:44.728 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:44:44.728 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:44:44.728 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 1525 2024-09-20T22:44:44.728 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.728 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:44:44.728 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.728 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:44:44.728 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.198921+0000", 2024-09-20T22:44:44.728 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:44:44.728 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.728 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.728 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:44:44.728 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.198921+0000", 2024-09-20T22:44:44.729 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:44:44.729 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.729 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.729 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:44:44.729 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.198923+0000", 2024-09-20T22:44:44.729 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.1090000000000001e-06 2024-09-20T22:44:44.729 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.730 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.730 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:44:44.730 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.198934+0000", 2024-09-20T22:44:44.730 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0837e-05 2024-09-20T22:44:44.730 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.730 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.730 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:44:44.731 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.198955+0000", 2024-09-20T22:44:44.731 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.0789e-05 2024-09-20T22:44:44.731 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.731 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.731 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:44:44.731 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.198966+0000", 2024-09-20T22:44:44.731 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1778e-05 2024-09-20T22:44:44.731 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.731 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.731 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:44:44.731 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.199046+0000", 2024-09-20T22:44:44.731 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.9567999999999994e-05 2024-09-20T22:44:44.731 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.732 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.732 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op must wait for map", 2024-09-20T22:44:44.732 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.199054+0000", 2024-09-20T22:44:44.732 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.9349999999999994e-06 2024-09-20T22:44:44.732 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.732 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.732 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:44:44.732 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.201428+0000", 2024-09-20T22:44:44.732 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0023744389999999999 2024-09-20T22:44:44.732 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.732 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.732 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:44:44.732 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.201527+0000", 2024-09-20T22:44:44.732 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.8554000000000005e-05 2024-09-20T22:44:44.732 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.733 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.733 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 3,7", 2024-09-20T22:44:44.733 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.215277+0000", 2024-09-20T22:44:44.733 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.013750326 2024-09-20T22:44:44.733 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.733 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.733 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:44:44.733 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.226629+0000", 2024-09-20T22:44:44.733 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.011351781999999999 2024-09-20T22:44:44.733 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.733 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.733 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:44:44.733 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.227366+0000", 2024-09-20T22:44:44.733 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00073709199999999996 2024-09-20T22:44:44.734 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.734 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.734 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:44:44.734 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.227450+0000", 2024-09-20T22:44:44.734 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.3902999999999999e-05 2024-09-20T22:44:44.734 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.734 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.734 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:44:44.734 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.227489+0000", 2024-09-20T22:44:44.734 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.9103000000000002e-05 2024-09-20T22:44:44.734 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.734 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.734 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:44:44.734 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.227542+0000", 2024-09-20T22:44:44.735 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.3006e-05 2024-09-20T22:44:44.735 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:44:44.735 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:44:44.735 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:44:44.735 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.735 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.735 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:1526 7.c 7:3f865ef5:::smithi08477203-28:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e197)", 2024-09-20T22:44:44.735 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:44:36.198993+0000", 2024-09-20T22:44:44.735 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 8.4135330919999998, 2024-09-20T22:44:44.735 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.022582641000000001, 2024-09-20T22:44:44.735 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:44:44.735 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:44:44.735 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:44:44.735 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:44:44.736 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:44:44.736 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:44:44.736 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 1526 2024-09-20T22:44:44.736 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.736 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:44:44.736 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.736 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:44:44.736 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.198993+0000", 2024-09-20T22:44:44.736 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:44:44.736 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.736 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.736 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:44:44.736 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.198993+0000", 2024-09-20T22:44:44.737 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:44:44.737 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.737 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.737 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:44:44.737 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.198994+0000", 2024-09-20T22:44:44.737 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.2e-07 2024-09-20T22:44:44.737 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.737 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.737 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:44:44.737 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.198996+0000", 2024-09-20T22:44:44.737 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.6410000000000002e-06 2024-09-20T22:44:44.737 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.737 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.737 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:44:44.738 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.199005+0000", 2024-09-20T22:44:44.738 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.8079999999999994e-06 2024-09-20T22:44:44.738 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.738 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.738 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:44:44.738 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.199011+0000", 2024-09-20T22:44:44.738 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.7010000000000001e-06 2024-09-20T22:44:44.738 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.738 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.738 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:44:44.738 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.199112+0000", 2024-09-20T22:44:44.738 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000101335 2024-09-20T22:44:44.738 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.738 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.739 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting_for_map not empty", 2024-09-20T22:44:44.739 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.199117+0000", 2024-09-20T22:44:44.739 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.2979999999999999e-06 2024-09-20T22:44:44.739 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.739 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.739 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:44:44.739 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.201667+0000", 2024-09-20T22:44:44.739 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.002549922 2024-09-20T22:44:44.739 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.739 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.739 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for blocked object", 2024-09-20T22:44:44.739 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.201708+0000", 2024-09-20T22:44:44.739 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.0772999999999997e-05 2024-09-20T22:44:44.740 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.740 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.740 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:44:44.740 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.221443+0000", 2024-09-20T22:44:44.740 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.019735229 2024-09-20T22:44:44.740 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.740 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.740 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:44:44.740 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.221509+0000", 2024-09-20T22:44:44.740 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.5618999999999996e-05 2024-09-20T22:44:44.740 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.740 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.740 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:44:44.740 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.221576+0000", 2024-09-20T22:44:44.741 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.6795e-05 2024-09-20T22:44:44.741 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:44:44.741 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:44:44.741 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:44:44.741 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.741 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.741 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:1542 7.12 7:4b06cbed:::smithi08477203-49:head [copy-from ver 69] snapc 34=[34,33] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2024-09-20T22:44:44.741 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:44:37.209101+0000", 2024-09-20T22:44:44.741 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 7.403425049, 2024-09-20T22:44:44.741 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.019142659999999999, 2024-09-20T22:44:44.741 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:44:44.741 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:44:44.741 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:44:44.741 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:44:44.742 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:44:44.742 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:44:44.742 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 1542 2024-09-20T22:44:44.742 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.742 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:44:44.742 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.742 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:44:44.742 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:37.209101+0000", 2024-09-20T22:44:44.742 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:44:44.742 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.742 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.742 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:44:44.742 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:37.209101+0000", 2024-09-20T22:44:44.742 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:44:44.743 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.743 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.743 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:44:44.743 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:37.209103+0000", 2024-09-20T22:44:44.743 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.454e-06 2024-09-20T22:44:44.743 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.743 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.743 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:44:44.743 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:37.209111+0000", 2024-09-20T22:44:44.743 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.1089999999999998e-06 2024-09-20T22:44:44.743 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.743 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.743 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:44:44.743 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:37.209130+0000", 2024-09-20T22:44:44.744 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.8933e-05 2024-09-20T22:44:44.744 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.744 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.744 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:44:44.744 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:37.209142+0000", 2024-09-20T22:44:44.744 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1843e-05 2024-09-20T22:44:44.744 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.744 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.744 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:44:44.744 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:37.209255+0000", 2024-09-20T22:44:44.744 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000113204 2024-09-20T22:44:44.744 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.744 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.744 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op must wait for map", 2024-09-20T22:44:44.744 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:37.209266+0000", 2024-09-20T22:44:44.745 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1070000000000001e-05 2024-09-20T22:44:44.745 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.745 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.745 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:44:44.745 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:37.210962+0000", 2024-09-20T22:44:44.745 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001696442 2024-09-20T22:44:44.745 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.745 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.745 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:44:44.745 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:37.211044+0000", 2024-09-20T22:44:44.745 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.2007000000000003e-05 2024-09-20T22:44:44.745 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.745 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.746 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 0,2", 2024-09-20T22:44:44.746 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:37.216569+0000", 2024-09-20T22:44:44.746 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.005525126 2024-09-20T22:44:44.746 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.746 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.746 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:44:44.746 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:37.221473+0000", 2024-09-20T22:44:44.746 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0049038739999999999 2024-09-20T22:44:44.746 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.746 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.746 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:44:44.746 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:37.227857+0000", 2024-09-20T22:44:44.746 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0063841109999999996 2024-09-20T22:44:44.747 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.747 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.747 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:44:44.747 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:37.228154+0000", 2024-09-20T22:44:44.747 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00029620999999999998 2024-09-20T22:44:44.747 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.747 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.747 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:44:44.747 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:37.228193+0000", 2024-09-20T22:44:44.747 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.9097000000000002e-05 2024-09-20T22:44:44.747 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.747 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.747 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:44:44.748 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:37.228244+0000", 2024-09-20T22:44:44.748 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.1180000000000001e-05 2024-09-20T22:44:44.748 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:44:44.748 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:44:44.748 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:44:44.748 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.748 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.748 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:1567 7.c 7:35bf785a:::smithi08477203-29:head [copy-from ver 94 in=55b] snapc 34=[34] ondisk+write+known_if_redirected+supports_pool_eio e199)", 2024-09-20T22:44:44.748 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:44:38.138541+0000", 2024-09-20T22:44:44.748 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 6.4739854530000001, 2024-09-20T22:44:44.748 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.016420443, 2024-09-20T22:44:44.748 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:44:44.748 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:44:44.748 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:44:44.749 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:44:44.749 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:44:44.749 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:44:44.749 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 1567 2024-09-20T22:44:44.749 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.749 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:44:44.749 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.749 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:44:44.749 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:38.138541+0000", 2024-09-20T22:44:44.749 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:44:44.749 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.749 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.749 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:44:44.749 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:38.138541+0000", 2024-09-20T22:44:44.750 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:44:44.750 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.750 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.750 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:44:44.750 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:38.138542+0000", 2024-09-20T22:44:44.750 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1000000000000001e-06 2024-09-20T22:44:44.750 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.750 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.750 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:44:44.750 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:38.138547+0000", 2024-09-20T22:44:44.750 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.0629999999999996e-06 2024-09-20T22:44:44.750 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.750 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.750 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:44:44.751 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:38.138556+0000", 2024-09-20T22:44:44.751 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.9279999999999999e-06 2024-09-20T22:44:44.751 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.751 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.751 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:44:44.751 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:38.138563+0000", 2024-09-20T22:44:44.751 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.323e-06 2024-09-20T22:44:44.751 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.751 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.751 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:44:44.751 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:38.138618+0000", 2024-09-20T22:44:44.751 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.4867999999999997e-05 2024-09-20T22:44:44.751 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.751 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.751 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op must wait for map", 2024-09-20T22:44:44.752 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:38.138626+0000", 2024-09-20T22:44:44.752 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.9980000000000003e-06 2024-09-20T22:44:44.752 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.752 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.752 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:44:44.752 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:38.140846+0000", 2024-09-20T22:44:44.752 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0022198209999999999 2024-09-20T22:44:44.752 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.752 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.752 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:44:44.752 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:38.140946+0000", 2024-09-20T22:44:44.752 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00010048900000000001 2024-09-20T22:44:44.752 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.752 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.753 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 3,7", 2024-09-20T22:44:44.753 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:38.147673+0000", 2024-09-20T22:44:44.753 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0067263039999999998 2024-09-20T22:44:44.753 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.753 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.753 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:44:44.753 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:38.154100+0000", 2024-09-20T22:44:44.753 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0064271550000000004 2024-09-20T22:44:44.753 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.753 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.753 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:44:44.753 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:38.154552+0000", 2024-09-20T22:44:44.753 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000452411 2024-09-20T22:44:44.753 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.753 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.754 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:44:44.754 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:38.154896+0000", 2024-09-20T22:44:44.754 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00034381899999999998 2024-09-20T22:44:44.754 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.754 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.754 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:44:44.754 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:38.154932+0000", 2024-09-20T22:44:44.754 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.5478999999999997e-05 2024-09-20T22:44:44.754 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:44:44.754 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:44:44.754 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:44:44.754 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:38.154961+0000", 2024-09-20T22:44:44.754 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.9685000000000001e-05 2024-09-20T22:44:44.754 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:44:44.755 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:44:44.755 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:44:44.755 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:44:44.755 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:44:44.755 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-20T22:44:44.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:44.756 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:44.756 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:44:44.857 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:44:44.878 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:44.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:44.879 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:44:44.980 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:44:45.001 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:45.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:45.002 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:44:45.103 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:44:45.124 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:45.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:45.125 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:44:45.142 INFO:tasks.rados.rados.0.smithi084.stdout:608: done (4 left) 2024-09-20T22:44:45.142 INFO:tasks.rados.rados.0.smithi084.stdout:609: done (3 left) 2024-09-20T22:44:45.142 INFO:tasks.rados.rados.0.smithi084.stdout:610: done (2 left) 2024-09-20T22:44:45.142 INFO:tasks.rados.rados.0.smithi084.stdout:611: done (1 left) 2024-09-20T22:44:45.142 INFO:tasks.rados.rados.0.smithi084.stdout:612: done (0 left) 2024-09-20T22:44:45.142 INFO:tasks.rados.rados.0.smithi084.stdout:613: delete oid 43 current snap is 55 2024-09-20T22:44:45.144 INFO:tasks.rados.rados.0.smithi084.stdout:613: done (0 left) 2024-09-20T22:44:45.144 INFO:tasks.rados.rados.0.smithi084.stdout:614: write oid 29 current snap is 55 2024-09-20T22:44:45.144 INFO:tasks.rados.rados.0.smithi084.stdout:614: seq_num 248 ranges {568136=785048,1929299=238837} 2024-09-20T22:44:45.151 INFO:tasks.rados.rados.0.smithi084.stdout:614: writing smithi08477203-29 from 568136 to 1353184 tid 1 2024-09-20T22:44:45.156 INFO:tasks.rados.rados.0.smithi084.stdout:614: writing smithi08477203-29 from 1929299 to 2168136 tid 2 2024-09-20T22:44:45.156 INFO:tasks.rados.rados.0.smithi084.stdout:615: read oid 28 snap -1 2024-09-20T22:44:45.156 INFO:tasks.rados.rados.0.smithi084.stdout:615: expect (ObjNum 231 snap 49 seq_num 231) 2024-09-20T22:44:45.156 INFO:tasks.rados.rados.0.smithi084.stdout:616: delete oid 34 current snap is 55 2024-09-20T22:44:45.158 INFO:tasks.rados.rados.0.smithi084.stdout:614: finishing write tid 1 to smithi08477203-29 2024-09-20T22:44:45.158 INFO:tasks.rados.rados.0.smithi084.stdout:614: finishing write tid 2 to smithi08477203-29 2024-09-20T22:44:45.159 INFO:tasks.rados.rados.0.smithi084.stdout:616: done (2 left) 2024-09-20T22:44:45.159 INFO:tasks.rados.rados.0.smithi084.stdout:617: read oid 4 snap -1 2024-09-20T22:44:45.159 INFO:tasks.rados.rados.0.smithi084.stdout:617: expect (ObjNum 241 snap 54 seq_num 241) 2024-09-20T22:44:45.159 INFO:tasks.rados.rados.0.smithi084.stdout:618: delete oid 11 current snap is 55 2024-09-20T22:44:45.160 INFO:tasks.rados.rados.0.smithi084.stdout:618: done (3 left) 2024-09-20T22:44:45.160 INFO:tasks.rados.rados.0.smithi084.stdout:619: copy_from oid 25 from oid 27 current snap is 55 2024-09-20T22:44:45.160 INFO:tasks.rados.rados.0.smithi084.stdout:620: copy_from oid 15 from oid 23 current snap is 55 2024-09-20T22:44:45.160 INFO:tasks.rados.rados.0.smithi084.stdout:621: snap_remove snap 54 2024-09-20T22:44:45.226 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:44:45.247 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:45.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:45.248 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:44:45.349 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:44:45.370 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:45.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:45.371 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:44:45.472 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:44:45.493 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:45.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:45.494 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:44:45.594 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:44:45.610 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-20T22:44:45.614 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:45 smithi084 ceph-mon[29774]: pgmap v208: 178 pgs: 1 active+clean+snaptrim, 177 active+clean; 114 MiB data, 537 MiB used, 625 GiB / 626 GiB avail; 4.8 MiB/s rd, 2.4 MiB/s wr, 14 op/s 2024-09-20T22:44:45.614 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:45 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/790334458' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:44:45.614 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:45 smithi084 ceph-mon[29774]: osdmap e206: 8 total, 8 up, 7 in 2024-09-20T22:44:45.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:45 smithi084 ceph-mon[40024]: pgmap v208: 178 pgs: 1 active+clean+snaptrim, 177 active+clean; 114 MiB data, 537 MiB used, 625 GiB / 626 GiB avail; 4.8 MiB/s rd, 2.4 MiB/s wr, 14 op/s 2024-09-20T22:44:45.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:45 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/790334458' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:44:45.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:45 smithi084 ceph-mon[40024]: osdmap e206: 8 total, 8 up, 7 in 2024-09-20T22:44:45.616 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:45.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:45.617 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:44:45.718 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:44:45.739 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:45.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:45.740 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:44:45.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:45 smithi154 ceph-mon[33264]: pgmap v208: 178 pgs: 1 active+clean+snaptrim, 177 active+clean; 114 MiB data, 537 MiB used, 625 GiB / 626 GiB avail; 4.8 MiB/s rd, 2.4 MiB/s wr, 14 op/s 2024-09-20T22:44:45.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:45 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/790334458' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:44:45.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:45 smithi154 ceph-mon[33264]: osdmap e206: 8 total, 8 up, 7 in 2024-09-20T22:44:45.841 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:44:45.864 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:44:45.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:45.865 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:45.865 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:44:45.880 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-20T22:44:45.966 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:44:45.987 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:45.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:45.989 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:44:46.089 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:44:46.111 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:46.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:46.112 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:44:46.145 INFO:tasks.rados.rados.0.smithi084.stdout:621: done (5 left) 2024-09-20T22:44:46.145 INFO:tasks.rados.rados.0.smithi084.stdout:622: snap_create 2024-09-20T22:44:46.148 INFO:tasks.rados.rados.0.smithi084.stdout:614: finishing write tid 3 to smithi08477203-29 2024-09-20T22:44:46.148 INFO:tasks.rados.rados.0.smithi084.stdout:614: finishing write tid 4 to smithi08477203-29 2024-09-20T22:44:46.148 INFO:tasks.rados.rados.0.smithi084.stdout:614: oid 29 updating version 0 to 259 2024-09-20T22:44:46.148 INFO:tasks.rados.rados.0.smithi084.stdout:614: oid 29 version 259 is already newer than 258 2024-09-20T22:44:46.148 INFO:tasks.rados.rados.0.smithi084.stdout:614: oid 29 version 259 is already newer than 257 2024-09-20T22:44:46.148 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 29 v 259 (ObjNum 248 snap 55 seq_num 248) dirty exists 2024-09-20T22:44:46.149 INFO:tasks.rados.rados.0.smithi084.stdout:614: left oid 29 (ObjNum 248 snap 55 seq_num 248) 2024-09-20T22:44:46.149 INFO:tasks.rados.rados.0.smithi084.stdout:615: expect (ObjNum 231 snap 49 seq_num 231) 2024-09-20T22:44:46.159 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:44:46.213 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:44:46.233 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:46.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:46.234 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:44:46.335 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:44:46.338 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-20T22:44:46.338 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [], 2024-09-20T22:44:46.338 INFO:teuthology.orchestra.run.smithi154.stdout: "num_ops": 0 2024-09-20T22:44:46.338 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-20T22:44:46.357 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:46.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:46.358 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:44:46.459 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:44:46.468 INFO:tasks.rados.rados.0.smithi084.stdout:617: expect (ObjNum 241 snap 54 seq_num 241) 2024-09-20T22:44:46.480 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:46.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:46.481 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:44:46.572 INFO:tasks.rados.rados.0.smithi084.stdout:620: finishing copy_from racing read to smithi08477203-15 2024-09-20T22:44:46.572 INFO:tasks.rados.rados.0.smithi084.stdout:619: finishing copy_from racing read to smithi08477203-25 2024-09-20T22:44:46.572 INFO:tasks.rados.rados.0.smithi084.stdout:620: finishing copy_from to smithi08477203-15 2024-09-20T22:44:46.572 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 15 v 110 (ObjNum 214 snap 44 seq_num 214) dirty exists 2024-09-20T22:44:46.572 INFO:tasks.rados.rados.0.smithi084.stdout:619: finishing copy_from to smithi08477203-25 2024-09-20T22:44:46.572 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 25 v 161 (ObjNum 234 snap 51 seq_num 234) dirty exists 2024-09-20T22:44:46.582 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:44:46.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:46.604 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:46.604 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:44:46.705 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:44:46.726 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:46.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:46.727 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:44:46.828 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:44:46.849 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:46.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:46.850 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:44:46.951 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:44:46.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:46.974 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:46.975 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:44:47.075 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.7 dump_blocked_ops 2024-09-20T22:44:47.075 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:44:47.096 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:47.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:47.097 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:44:47.149 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is set 2024-09-20T22:44:47.149 INFO:tasks.rados.rados.0.smithi084.stdout:614: done (5 left) 2024-09-20T22:44:47.149 INFO:tasks.rados.rados.0.smithi084.stdout:615: done (4 left) 2024-09-20T22:44:47.149 INFO:tasks.rados.rados.0.smithi084.stdout:617: done (3 left) 2024-09-20T22:44:47.150 INFO:tasks.rados.rados.0.smithi084.stdout:619: done (2 left) 2024-09-20T22:44:47.150 INFO:tasks.rados.rados.0.smithi084.stdout:620: done (1 left) 2024-09-20T22:44:47.150 INFO:tasks.rados.rados.0.smithi084.stdout:622: done (0 left) 2024-09-20T22:44:47.150 INFO:tasks.rados.rados.0.smithi084.stdout:623: delete oid 39 current snap is 56 2024-09-20T22:44:47.151 INFO:tasks.rados.rados.0.smithi084.stdout:623: done (0 left) 2024-09-20T22:44:47.151 INFO:tasks.rados.rados.0.smithi084.stdout:624: snap_create 2024-09-20T22:44:47.198 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:44:47.219 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:47.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:47.220 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:44:47.321 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:44:47.327 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:44:47.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:47.344 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:47.344 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:44:47.367 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-20T22:44:47.390 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:47 smithi084 ceph-mon[40024]: pgmap v211: 178 pgs: 1 active+clean+snaptrim, 177 active+clean; 114 MiB data, 537 MiB used, 625 GiB / 626 GiB avail; 4.8 MiB/s rd, 2.4 MiB/s wr, 14 op/s 2024-09-20T22:44:47.390 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:47 smithi084 ceph-mon[40024]: osdmap e207: 8 total, 8 up, 7 in 2024-09-20T22:44:47.391 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:47 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/2519987038' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:44:47.391 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:47 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:44:47.391 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:47 smithi084 ceph-mon[29774]: pgmap v211: 178 pgs: 1 active+clean+snaptrim, 177 active+clean; 114 MiB data, 537 MiB used, 625 GiB / 626 GiB avail; 4.8 MiB/s rd, 2.4 MiB/s wr, 14 op/s 2024-09-20T22:44:47.391 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:47 smithi084 ceph-mon[29774]: osdmap e207: 8 total, 8 up, 7 in 2024-09-20T22:44:47.391 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:47 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/2519987038' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:44:47.391 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:47 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:44:47.446 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:44:47.449 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:47 smithi154 ceph-mon[33264]: pgmap v211: 178 pgs: 1 active+clean+snaptrim, 177 active+clean; 114 MiB data, 537 MiB used, 625 GiB / 626 GiB avail; 4.8 MiB/s rd, 2.4 MiB/s wr, 14 op/s 2024-09-20T22:44:47.449 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:47 smithi154 ceph-mon[33264]: osdmap e207: 8 total, 8 up, 7 in 2024-09-20T22:44:47.449 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:47 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/2519987038' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:44:47.449 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:47 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:44:47.469 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:47.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:47.470 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:44:47.571 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:44:47.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:47.596 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:47.597 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:44:47.698 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:44:47.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:47.721 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:47.721 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:44:47.822 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:44:47.843 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:47.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:47.844 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:44:47.877 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-20T22:44:47.878 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [], 2024-09-20T22:44:47.878 INFO:teuthology.orchestra.run.smithi154.stdout: "complaint_time": 30, 2024-09-20T22:44:47.878 INFO:teuthology.orchestra.run.smithi154.stdout: "num_blocked_ops": 0 2024-09-20T22:44:47.878 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-20T22:44:47.944 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:44:47.946 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:44:47.967 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:47.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:47.968 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:44:48.069 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:44:48.090 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:48.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:48.091 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:44:48.141 INFO:tasks.rados.rados.0.smithi084.stdout:624: done (0 left) 2024-09-20T22:44:48.141 INFO:tasks.rados.rados.0.smithi084.stdout:625: copy_from oid 1 from oid 20 current snap is 57 2024-09-20T22:44:48.141 INFO:tasks.rados.rados.0.smithi084.stdout:626: read oid 48 snap 55 2024-09-20T22:44:48.142 INFO:tasks.rados.rados.0.smithi084.stdout:626: expect (ObjNum 129 snap 23 seq_num 129) 2024-09-20T22:44:48.142 INFO:tasks.rados.rados.0.smithi084.stdout:627: copy_from oid 45 from oid 30 current snap is 57 2024-09-20T22:44:48.142 INFO:tasks.rados.rados.0.smithi084.stdout:628: rollback oid 14 current snap is 57 2024-09-20T22:44:48.142 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 14 to 55 2024-09-20T22:44:48.142 INFO:tasks.rados.rados.0.smithi084.stdout:629: snap_remove snap 53 2024-09-20T22:44:48.192 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:44:48.214 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:48.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:48.215 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:44:48.316 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:44:48.340 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:48.341 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:44:48 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: ::ffff:172.21.15.154 - - [20/Sep/2024:22:44:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-20T22:44:48.341 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:48 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T22:44:48.341 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:48 smithi084 ceph-mon[40024]: osdmap e208: 8 total, 8 up, 7 in 2024-09-20T22:44:48.341 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:48 smithi084 ceph-mon[40024]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:44:48.341 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:48 smithi084 ceph-mon[40024]: osdmap e209: 8 total, 8 up, 7 in 2024-09-20T22:44:48.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:48.342 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:48 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T22:44:48.342 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:48 smithi084 ceph-mon[29774]: osdmap e208: 8 total, 8 up, 7 in 2024-09-20T22:44:48.342 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:48 smithi084 ceph-mon[29774]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:44:48.342 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:48 smithi084 ceph-mon[29774]: osdmap e209: 8 total, 8 up, 7 in 2024-09-20T22:44:48.342 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:44:48.424 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:48 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T22:44:48.424 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:48 smithi154 ceph-mon[33264]: osdmap e208: 8 total, 8 up, 7 in 2024-09-20T22:44:48.424 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:48 smithi154 ceph-mon[33264]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:44:48.424 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:48 smithi154 ceph-mon[33264]: osdmap e209: 8 total, 8 up, 7 in 2024-09-20T22:44:48.438 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:44:48.438 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:44:48.443 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:44:48.455 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:44:48.455 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:44:48.465 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:48.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:48.466 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:44:48.471 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:44:48.472 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:44:48.487 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-20T22:44:48.487 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T22:44:48.503 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:44:48.504 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:44:48.520 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:44:48.520 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:44:48.537 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:44:48.537 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:44:48.553 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:44:48.553 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:44:48.567 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:44:48.589 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:48.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:48.589 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:44:48.690 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:44:48.711 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:48.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:48.712 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:44:48.751 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.7 dump_historic_ops 2024-09-20T22:44:48.812 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:44:48.834 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:48.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:48.835 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:44:48.935 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:44:48.956 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:48.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:48.957 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:44:48.998 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:44:49.058 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:44:49.079 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:49.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:49.080 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:44:49.112 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 7] out_osds: [4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-20T22:44:49.112 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-20T22:44:49.112 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-09-20T22:44:49.113 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd dump -f json-pretty 2024-09-20T22:44:49.147 INFO:tasks.rados.rados.0.smithi084.stdout:629: done (4 left) 2024-09-20T22:44:49.147 INFO:tasks.rados.rados.0.smithi084.stdout:630: read oid 19 snap -1 2024-09-20T22:44:49.147 INFO:tasks.rados.rados.0.smithi084.stdout:630: expect (ObjNum 240 snap 52 seq_num 240) 2024-09-20T22:44:49.147 INFO:tasks.rados.rados.0.smithi084.stdout:631: copy_from oid 21 from oid 39 current snap is 57 2024-09-20T22:44:49.147 INFO:tasks.rados.rados.0.smithi084.stdout:632: snap_create 2024-09-20T22:44:49.147 INFO:tasks.rados.rados.0.smithi084.stdout:625: finishing copy_from to smithi08477203-1 2024-09-20T22:44:49.147 INFO:tasks.rados.rados.0.smithi084.stdout:625: got expected ENOENT (src dne) 2024-09-20T22:44:49.147 INFO:tasks.rados.rados.0.smithi084.stdout:625: finishing copy_from racing read to smithi08477203-1 2024-09-20T22:44:49.147 INFO:tasks.rados.rados.0.smithi084.stdout:626: expect (ObjNum 129 snap 23 seq_num 129) 2024-09-20T22:44:49.181 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:44:49.202 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:49.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:49.203 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:44:49.228 INFO:tasks.rados.rados.0.smithi084.stdout:628: finishing rollback tid 0 to smithi08477203-14 2024-09-20T22:44:49.228 INFO:tasks.rados.rados.0.smithi084.stdout:628: finishing rollback tid 1 to smithi08477203-14 2024-09-20T22:44:49.228 INFO:tasks.rados.rados.0.smithi084.stdout:628: finishing rollback tid 2 to smithi08477203-14 2024-09-20T22:44:49.228 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 14 v 45 (ObjNum 13 snap 0 seq_num 13) dirty exists 2024-09-20T22:44:49.228 INFO:tasks.rados.rados.0.smithi084.stdout:627: finishing copy_from to smithi08477203-45 2024-09-20T22:44:49.228 INFO:tasks.rados.rados.0.smithi084.stdout:627: got expected ENOENT (src dne) 2024-09-20T22:44:49.228 INFO:tasks.rados.rados.0.smithi084.stdout:627: finishing copy_from racing read to smithi08477203-45 2024-09-20T22:44:49.228 INFO:tasks.rados.rados.0.smithi084.stdout:631: finishing copy_from to smithi08477203-21 2024-09-20T22:44:49.228 INFO:tasks.rados.rados.0.smithi084.stdout:631: got expected ENOENT (src dne) 2024-09-20T22:44:49.228 INFO:tasks.rados.rados.0.smithi084.stdout:631: finishing copy_from racing read to smithi08477203-21 2024-09-20T22:44:49.228 INFO:tasks.rados.rados.0.smithi084.stdout:630: expect (ObjNum 240 snap 52 seq_num 240) 2024-09-20T22:44:49.246 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:49 smithi084 ceph-mon[29774]: pgmap v214: 178 pgs: 1 active+clean+snaptrim, 177 active+clean; 114 MiB data, 537 MiB used, 625 GiB / 626 GiB avail 2024-09-20T22:44:49.246 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:49 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/3577967309' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:44:49.246 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:49 smithi084 ceph-mon[29774]: osdmap e210: 8 total, 8 up, 7 in 2024-09-20T22:44:49.246 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:49 smithi084 ceph-mon[40024]: pgmap v214: 178 pgs: 1 active+clean+snaptrim, 177 active+clean; 114 MiB data, 537 MiB used, 625 GiB / 626 GiB avail 2024-09-20T22:44:49.247 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:49 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/3577967309' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:44:49.247 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:49 smithi084 ceph-mon[40024]: osdmap e210: 8 total, 8 up, 7 in 2024-09-20T22:44:49.304 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:44:49.326 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:49.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:49.327 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:44:49.381 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:44:49.386 INFO:journalctl@ceph.iscsi.iscsi.a.smithi154.stdout:Sep 20 22:44:49 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-iscsi-iscsi-a[59730]: debug there is no tcmu-runner data available 2024-09-20T22:44:49.387 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:49 smithi154 ceph-mon[33264]: pgmap v214: 178 pgs: 1 active+clean+snaptrim, 177 active+clean; 114 MiB data, 537 MiB used, 625 GiB / 626 GiB avail 2024-09-20T22:44:49.387 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:49 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/3577967309' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:44:49.387 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:49 smithi154 ceph-mon[33264]: osdmap e210: 8 total, 8 up, 7 in 2024-09-20T22:44:49.428 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:44:49.448 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:49.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:49.449 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:44:49.488 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-20T22:44:49.488 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:44:49.550 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:44:49.572 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:49.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:49.573 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:44:49.674 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:44:49.704 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:49.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:49.705 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:44:49.806 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:44:49.829 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:49.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:49.831 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:44:49.831 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd unset noscrub 2024-09-20T22:44:49.932 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:44:49.953 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:49.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:49.954 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:44:50.025 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-20T22:44:50.025 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-20T22:44:50.025 INFO:teuthology.orchestra.run.smithi084.stdout: "epoch": 210, 2024-09-20T22:44:50.025 INFO:teuthology.orchestra.run.smithi084.stdout: "fsid": "a9c0bc70-77a0-11ef-baf5-efdc52797490", 2024-09-20T22:44:50.025 INFO:teuthology.orchestra.run.smithi084.stdout: "created": "2024-09-20T22:36:30.240626+0000", 2024-09-20T22:44:50.026 INFO:teuthology.orchestra.run.smithi084.stdout: "modified": "2024-09-20T22:44:49.139612+0000", 2024-09-20T22:44:50.026 INFO:teuthology.orchestra.run.smithi084.stdout: "last_up_change": "2024-09-20T22:41:27.098565+0000", 2024-09-20T22:44:50.026 INFO:teuthology.orchestra.run.smithi084.stdout: "last_in_change": "2024-09-20T22:43:56.388784+0000", 2024-09-20T22:44:50.026 INFO:teuthology.orchestra.run.smithi084.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-20T22:44:50.026 INFO:teuthology.orchestra.run.smithi084.stdout: "flags_num": 5806080, 2024-09-20T22:44:50.026 INFO:teuthology.orchestra.run.smithi084.stdout: "flags_set": [ 2024-09-20T22:44:50.027 INFO:teuthology.orchestra.run.smithi084.stdout: "nodeep-scrub", 2024-09-20T22:44:50.027 INFO:teuthology.orchestra.run.smithi084.stdout: "noscrub", 2024-09-20T22:44:50.027 INFO:teuthology.orchestra.run.smithi084.stdout: "pglog_hardlimit", 2024-09-20T22:44:50.027 INFO:teuthology.orchestra.run.smithi084.stdout: "purged_snapdirs", 2024-09-20T22:44:50.027 INFO:teuthology.orchestra.run.smithi084.stdout: "recovery_deletes", 2024-09-20T22:44:50.027 INFO:teuthology.orchestra.run.smithi084.stdout: "sortbitwise" 2024-09-20T22:44:50.028 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2024-09-20T22:44:50.028 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_version": 18, 2024-09-20T22:44:50.028 INFO:teuthology.orchestra.run.smithi084.stdout: "full_ratio": 0.94999998807907104, 2024-09-20T22:44:50.028 INFO:teuthology.orchestra.run.smithi084.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-20T22:44:50.028 INFO:teuthology.orchestra.run.smithi084.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-20T22:44:50.028 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_snapshot": "", 2024-09-20T22:44:50.029 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_max": 7, 2024-09-20T22:44:50.029 INFO:teuthology.orchestra.run.smithi084.stdout: "max_osd": 8, 2024-09-20T22:44:50.029 INFO:teuthology.orchestra.run.smithi084.stdout: "require_min_compat_client": "luminous", 2024-09-20T22:44:50.029 INFO:teuthology.orchestra.run.smithi084.stdout: "min_compat_client": "luminous", 2024-09-20T22:44:50.029 INFO:teuthology.orchestra.run.smithi084.stdout: "require_osd_release": "squid", 2024-09-20T22:44:50.029 INFO:teuthology.orchestra.run.smithi084.stdout: "allow_crimson": false, 2024-09-20T22:44:50.030 INFO:teuthology.orchestra.run.smithi084.stdout: "pools": [ 2024-09-20T22:44:50.030 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:44:50.030 INFO:teuthology.orchestra.run.smithi084.stdout: "pool": 1, 2024-09-20T22:44:50.030 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_name": ".mgr", 2024-09-20T22:44:50.030 INFO:teuthology.orchestra.run.smithi084.stdout: "create_time": "2024-09-20T22:39:21.984381+0000", 2024-09-20T22:44:50.030 INFO:teuthology.orchestra.run.smithi084.stdout: "flags": 1, 2024-09-20T22:44:50.031 INFO:teuthology.orchestra.run.smithi084.stdout: "flags_names": "hashpspool", 2024-09-20T22:44:50.031 INFO:teuthology.orchestra.run.smithi084.stdout: "type": 1, 2024-09-20T22:44:50.031 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 3, 2024-09-20T22:44:50.031 INFO:teuthology.orchestra.run.smithi084.stdout: "min_size": 2, 2024-09-20T22:44:50.031 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_rule": 0, 2024-09-20T22:44:50.031 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_count": 0, 2024-09-20T22:44:50.031 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_target": 0, 2024-09-20T22:44:50.032 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_barrier": 0, 2024-09-20T22:44:50.032 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-20T22:44:50.032 INFO:teuthology.orchestra.run.smithi084.stdout: "is_stretch_pool": false, 2024-09-20T22:44:50.032 INFO:teuthology.orchestra.run.smithi084.stdout: "object_hash": 2, 2024-09-20T22:44:50.032 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_autoscale_mode": "off", 2024-09-20T22:44:50.032 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num": 21, 2024-09-20T22:44:50.033 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num": 21, 2024-09-20T22:44:50.033 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num_target": 21, 2024-09-20T22:44:50.033 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_target": 21, 2024-09-20T22:44:50.033 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_pending": 21, 2024-09-20T22:44:50.033 INFO:teuthology.orchestra.run.smithi084.stdout: "last_pg_merge_meta": { 2024-09-20T22:44:50.033 INFO:teuthology.orchestra.run.smithi084.stdout: "source_pgid": "0.0", 2024-09-20T22:44:50.034 INFO:teuthology.orchestra.run.smithi084.stdout: "ready_epoch": 0, 2024-09-20T22:44:50.034 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_started": 0, 2024-09-20T22:44:50.034 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_clean": 0, 2024-09-20T22:44:50.034 INFO:teuthology.orchestra.run.smithi084.stdout: "source_version": "0'0", 2024-09-20T22:44:50.034 INFO:teuthology.orchestra.run.smithi084.stdout: "target_version": "0'0" 2024-09-20T22:44:50.034 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:44:50.035 INFO:teuthology.orchestra.run.smithi084.stdout: "last_change": "147", 2024-09-20T22:44:50.035 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend": "0", 2024-09-20T22:44:50.035 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-20T22:44:50.035 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_preluminous": "119", 2024-09-20T22:44:50.035 INFO:teuthology.orchestra.run.smithi084.stdout: "auid": 0, 2024-09-20T22:44:50.035 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_mode": "selfmanaged", 2024-09-20T22:44:50.036 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_seq": 0, 2024-09-20T22:44:50.036 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_epoch": 0, 2024-09-20T22:44:50.036 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_snaps": [], 2024-09-20T22:44:50.036 INFO:teuthology.orchestra.run.smithi084.stdout: "removed_snaps": "[]", 2024-09-20T22:44:50.036 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_bytes": 0, 2024-09-20T22:44:50.036 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_objects": 0, 2024-09-20T22:44:50.037 INFO:teuthology.orchestra.run.smithi084.stdout: "tiers": [], 2024-09-20T22:44:50.037 INFO:teuthology.orchestra.run.smithi084.stdout: "tier_of": -1, 2024-09-20T22:44:50.037 INFO:teuthology.orchestra.run.smithi084.stdout: "read_tier": -1, 2024-09-20T22:44:50.037 INFO:teuthology.orchestra.run.smithi084.stdout: "write_tier": -1, 2024-09-20T22:44:50.037 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_mode": "none", 2024-09-20T22:44:50.037 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_bytes": 0, 2024-09-20T22:44:50.037 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_objects": 0, 2024-09-20T22:44:50.038 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-20T22:44:50.038 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-20T22:44:50.038 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-20T22:44:50.038 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_flush_age": 0, 2024-09-20T22:44:50.038 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_evict_age": 0, 2024-09-20T22:44:50.038 INFO:teuthology.orchestra.run.smithi084.stdout: "erasure_code_profile": "", 2024-09-20T22:44:50.039 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_params": { 2024-09-20T22:44:50.039 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "none" 2024-09-20T22:44:50.039 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:44:50.039 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_period": 0, 2024-09-20T22:44:50.039 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_count": 0, 2024-09-20T22:44:50.039 INFO:teuthology.orchestra.run.smithi084.stdout: "use_gmt_hitset": true, 2024-09-20T22:44:50.040 INFO:teuthology.orchestra.run.smithi084.stdout: "min_read_recency_for_promote": 0, 2024-09-20T22:44:50.040 INFO:teuthology.orchestra.run.smithi084.stdout: "min_write_recency_for_promote": 0, 2024-09-20T22:44:50.040 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_grade_decay_rate": 0, 2024-09-20T22:44:50.040 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_search_last_n": 0, 2024-09-20T22:44:50.040 INFO:teuthology.orchestra.run.smithi084.stdout: "grade_table": [], 2024-09-20T22:44:50.040 INFO:teuthology.orchestra.run.smithi084.stdout: "stripe_width": 0, 2024-09-20T22:44:50.041 INFO:teuthology.orchestra.run.smithi084.stdout: "expected_num_objects": 0, 2024-09-20T22:44:50.041 INFO:teuthology.orchestra.run.smithi084.stdout: "fast_read": false, 2024-09-20T22:44:50.041 INFO:teuthology.orchestra.run.smithi084.stdout: "options": { 2024-09-20T22:44:50.041 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_max": 32, 2024-09-20T22:44:50.041 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_min": 1 2024-09-20T22:44:50.041 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:44:50.042 INFO:teuthology.orchestra.run.smithi084.stdout: "application_metadata": { 2024-09-20T22:44:50.042 INFO:teuthology.orchestra.run.smithi084.stdout: "mgr": {} 2024-09-20T22:44:50.042 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:44:50.042 INFO:teuthology.orchestra.run.smithi084.stdout: "read_balance": { 2024-09-20T22:44:50.042 INFO:teuthology.orchestra.run.smithi084.stdout: "score_type": "Fair distribution", 2024-09-20T22:44:50.042 INFO:teuthology.orchestra.run.smithi084.stdout: "score_acting": 1.8999999761581421, 2024-09-20T22:44:50.042 INFO:teuthology.orchestra.run.smithi084.stdout: "score_stable": 1.8999999761581421, 2024-09-20T22:44:50.043 INFO:teuthology.orchestra.run.smithi084.stdout: "optimal_score": 0.87999999523162842, 2024-09-20T22:44:50.043 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_acting": 1.6699999570846558, 2024-09-20T22:44:50.043 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_stable": 1.6699999570846558, 2024-09-20T22:44:50.043 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity_weighted": 1, 2024-09-20T22:44:50.043 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity": 1, 2024-09-20T22:44:50.043 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity_weighted": 1 2024-09-20T22:44:50.044 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:44:50.044 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:44:50.044 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:44:50.044 INFO:teuthology.orchestra.run.smithi084.stdout: "pool": 2, 2024-09-20T22:44:50.044 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_name": "datapool", 2024-09-20T22:44:50.044 INFO:teuthology.orchestra.run.smithi084.stdout: "create_time": "2024-09-20T22:41:35.820081+0000", 2024-09-20T22:44:50.044 INFO:teuthology.orchestra.run.smithi084.stdout: "flags": 8193, 2024-09-20T22:44:50.044 INFO:teuthology.orchestra.run.smithi084.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-20T22:44:50.044 INFO:teuthology.orchestra.run.smithi084.stdout: "type": 1, 2024-09-20T22:44:50.044 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 3, 2024-09-20T22:44:50.044 INFO:teuthology.orchestra.run.smithi084.stdout: "min_size": 2, 2024-09-20T22:44:50.045 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_rule": 0, 2024-09-20T22:44:50.045 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_count": 0, 2024-09-20T22:44:50.045 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_target": 0, 2024-09-20T22:44:50.045 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_barrier": 0, 2024-09-20T22:44:50.045 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-20T22:44:50.045 INFO:teuthology.orchestra.run.smithi084.stdout: "is_stretch_pool": false, 2024-09-20T22:44:50.045 INFO:teuthology.orchestra.run.smithi084.stdout: "object_hash": 2, 2024-09-20T22:44:50.045 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_autoscale_mode": "off", 2024-09-20T22:44:50.045 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num": 3, 2024-09-20T22:44:50.045 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num": 3, 2024-09-20T22:44:50.045 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num_target": 3, 2024-09-20T22:44:50.045 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_target": 3, 2024-09-20T22:44:50.045 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_pending": 3, 2024-09-20T22:44:50.045 INFO:teuthology.orchestra.run.smithi084.stdout: "last_pg_merge_meta": { 2024-09-20T22:44:50.046 INFO:teuthology.orchestra.run.smithi084.stdout: "source_pgid": "0.0", 2024-09-20T22:44:50.046 INFO:teuthology.orchestra.run.smithi084.stdout: "ready_epoch": 0, 2024-09-20T22:44:50.046 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_started": 0, 2024-09-20T22:44:50.046 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_clean": 0, 2024-09-20T22:44:50.046 INFO:teuthology.orchestra.run.smithi084.stdout: "source_version": "0'0", 2024-09-20T22:44:50.046 INFO:teuthology.orchestra.run.smithi084.stdout: "target_version": "0'0" 2024-09-20T22:44:50.046 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:44:50.046 INFO:teuthology.orchestra.run.smithi084.stdout: "last_change": "69", 2024-09-20T22:44:50.046 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend": "0", 2024-09-20T22:44:50.046 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-20T22:44:50.046 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_preluminous": "0", 2024-09-20T22:44:50.046 INFO:teuthology.orchestra.run.smithi084.stdout: "auid": 0, 2024-09-20T22:44:50.046 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_mode": "selfmanaged", 2024-09-20T22:44:50.047 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_seq": 2, 2024-09-20T22:44:50.047 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_epoch": 69, 2024-09-20T22:44:50.047 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_snaps": [], 2024-09-20T22:44:50.047 INFO:teuthology.orchestra.run.smithi084.stdout: "removed_snaps": "[]", 2024-09-20T22:44:50.047 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_bytes": 0, 2024-09-20T22:44:50.047 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_objects": 0, 2024-09-20T22:44:50.047 INFO:teuthology.orchestra.run.smithi084.stdout: "tiers": [], 2024-09-20T22:44:50.047 INFO:teuthology.orchestra.run.smithi084.stdout: "tier_of": -1, 2024-09-20T22:44:50.047 INFO:teuthology.orchestra.run.smithi084.stdout: "read_tier": -1, 2024-09-20T22:44:50.047 INFO:teuthology.orchestra.run.smithi084.stdout: "write_tier": -1, 2024-09-20T22:44:50.047 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_mode": "none", 2024-09-20T22:44:50.047 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_bytes": 0, 2024-09-20T22:44:50.047 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_objects": 0, 2024-09-20T22:44:50.047 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-20T22:44:50.048 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-20T22:44:50.048 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-20T22:44:50.048 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_flush_age": 0, 2024-09-20T22:44:50.048 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_evict_age": 0, 2024-09-20T22:44:50.048 INFO:teuthology.orchestra.run.smithi084.stdout: "erasure_code_profile": "", 2024-09-20T22:44:50.048 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_params": { 2024-09-20T22:44:50.048 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "none" 2024-09-20T22:44:50.048 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:44:50.048 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_period": 0, 2024-09-20T22:44:50.048 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_count": 0, 2024-09-20T22:44:50.048 INFO:teuthology.orchestra.run.smithi084.stdout: "use_gmt_hitset": true, 2024-09-20T22:44:50.048 INFO:teuthology.orchestra.run.smithi084.stdout: "min_read_recency_for_promote": 0, 2024-09-20T22:44:50.048 INFO:teuthology.orchestra.run.smithi084.stdout: "min_write_recency_for_promote": 0, 2024-09-20T22:44:50.048 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_grade_decay_rate": 0, 2024-09-20T22:44:50.048 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_search_last_n": 0, 2024-09-20T22:44:50.049 INFO:teuthology.orchestra.run.smithi084.stdout: "grade_table": [], 2024-09-20T22:44:50.049 INFO:teuthology.orchestra.run.smithi084.stdout: "stripe_width": 0, 2024-09-20T22:44:50.049 INFO:teuthology.orchestra.run.smithi084.stdout: "expected_num_objects": 0, 2024-09-20T22:44:50.049 INFO:teuthology.orchestra.run.smithi084.stdout: "fast_read": false, 2024-09-20T22:44:50.049 INFO:teuthology.orchestra.run.smithi084.stdout: "options": {}, 2024-09-20T22:44:50.049 INFO:teuthology.orchestra.run.smithi084.stdout: "application_metadata": { 2024-09-20T22:44:50.049 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd": {} 2024-09-20T22:44:50.049 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:44:50.049 INFO:teuthology.orchestra.run.smithi084.stdout: "read_balance": { 2024-09-20T22:44:50.049 INFO:teuthology.orchestra.run.smithi084.stdout: "score_type": "Fair distribution", 2024-09-20T22:44:50.049 INFO:teuthology.orchestra.run.smithi084.stdout: "score_acting": 2.6500000953674316, 2024-09-20T22:44:50.049 INFO:teuthology.orchestra.run.smithi084.stdout: "score_stable": 2.6500000953674316, 2024-09-20T22:44:50.049 INFO:teuthology.orchestra.run.smithi084.stdout: "optimal_score": 0.87999999523162842, 2024-09-20T22:44:50.049 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_acting": 2.3299999237060547, 2024-09-20T22:44:50.050 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_stable": 2.3299999237060547, 2024-09-20T22:44:50.050 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity_weighted": 1, 2024-09-20T22:44:50.050 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity": 1, 2024-09-20T22:44:50.050 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity_weighted": 1 2024-09-20T22:44:50.050 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:44:50.050 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:44:50.050 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:44:50.050 INFO:teuthology.orchestra.run.smithi084.stdout: "pool": 3, 2024-09-20T22:44:50.050 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_name": ".rgw.root", 2024-09-20T22:44:50.050 INFO:teuthology.orchestra.run.smithi084.stdout: "create_time": "2024-09-20T22:41:36.546389+0000", 2024-09-20T22:44:50.050 INFO:teuthology.orchestra.run.smithi084.stdout: "flags": 1, 2024-09-20T22:44:50.050 INFO:teuthology.orchestra.run.smithi084.stdout: "flags_names": "hashpspool", 2024-09-20T22:44:50.050 INFO:teuthology.orchestra.run.smithi084.stdout: "type": 1, 2024-09-20T22:44:50.051 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 3, 2024-09-20T22:44:50.051 INFO:teuthology.orchestra.run.smithi084.stdout: "min_size": 2, 2024-09-20T22:44:50.051 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_rule": 0, 2024-09-20T22:44:50.051 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_count": 0, 2024-09-20T22:44:50.051 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_target": 0, 2024-09-20T22:44:50.051 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_barrier": 0, 2024-09-20T22:44:50.051 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-20T22:44:50.051 INFO:teuthology.orchestra.run.smithi084.stdout: "is_stretch_pool": false, 2024-09-20T22:44:50.051 INFO:teuthology.orchestra.run.smithi084.stdout: "object_hash": 2, 2024-09-20T22:44:50.051 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_autoscale_mode": "off", 2024-09-20T22:44:50.051 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num": 32, 2024-09-20T22:44:50.051 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num": 32, 2024-09-20T22:44:50.051 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num_target": 32, 2024-09-20T22:44:50.051 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_target": 32, 2024-09-20T22:44:50.052 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_pending": 32, 2024-09-20T22:44:50.052 INFO:teuthology.orchestra.run.smithi084.stdout: "last_pg_merge_meta": { 2024-09-20T22:44:50.052 INFO:teuthology.orchestra.run.smithi084.stdout: "source_pgid": "0.0", 2024-09-20T22:44:50.052 INFO:teuthology.orchestra.run.smithi084.stdout: "ready_epoch": 0, 2024-09-20T22:44:50.052 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_started": 0, 2024-09-20T22:44:50.052 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_clean": 0, 2024-09-20T22:44:50.052 INFO:teuthology.orchestra.run.smithi084.stdout: "source_version": "0'0", 2024-09-20T22:44:50.052 INFO:teuthology.orchestra.run.smithi084.stdout: "target_version": "0'0" 2024-09-20T22:44:50.052 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:44:50.052 INFO:teuthology.orchestra.run.smithi084.stdout: "last_change": "68", 2024-09-20T22:44:50.052 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend": "0", 2024-09-20T22:44:50.052 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-20T22:44:50.052 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_preluminous": "0", 2024-09-20T22:44:50.052 INFO:teuthology.orchestra.run.smithi084.stdout: "auid": 0, 2024-09-20T22:44:50.053 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_mode": "selfmanaged", 2024-09-20T22:44:50.053 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_seq": 0, 2024-09-20T22:44:50.053 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_epoch": 0, 2024-09-20T22:44:50.053 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_snaps": [], 2024-09-20T22:44:50.053 INFO:teuthology.orchestra.run.smithi084.stdout: "removed_snaps": "[]", 2024-09-20T22:44:50.053 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_bytes": 0, 2024-09-20T22:44:50.053 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_objects": 0, 2024-09-20T22:44:50.053 INFO:teuthology.orchestra.run.smithi084.stdout: "tiers": [], 2024-09-20T22:44:50.053 INFO:teuthology.orchestra.run.smithi084.stdout: "tier_of": -1, 2024-09-20T22:44:50.053 INFO:teuthology.orchestra.run.smithi084.stdout: "read_tier": -1, 2024-09-20T22:44:50.053 INFO:teuthology.orchestra.run.smithi084.stdout: "write_tier": -1, 2024-09-20T22:44:50.053 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_mode": "none", 2024-09-20T22:44:50.053 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_bytes": 0, 2024-09-20T22:44:50.053 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_objects": 0, 2024-09-20T22:44:50.054 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-20T22:44:50.054 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-20T22:44:50.054 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-20T22:44:50.054 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_flush_age": 0, 2024-09-20T22:44:50.054 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_evict_age": 0, 2024-09-20T22:44:50.054 INFO:teuthology.orchestra.run.smithi084.stdout: "erasure_code_profile": "", 2024-09-20T22:44:50.054 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_params": { 2024-09-20T22:44:50.054 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "none" 2024-09-20T22:44:50.054 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:44:50.054 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_period": 0, 2024-09-20T22:44:50.054 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_count": 0, 2024-09-20T22:44:50.054 INFO:teuthology.orchestra.run.smithi084.stdout: "use_gmt_hitset": true, 2024-09-20T22:44:50.054 INFO:teuthology.orchestra.run.smithi084.stdout: "min_read_recency_for_promote": 0, 2024-09-20T22:44:50.054 INFO:teuthology.orchestra.run.smithi084.stdout: "min_write_recency_for_promote": 0, 2024-09-20T22:44:50.055 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_grade_decay_rate": 0, 2024-09-20T22:44:50.055 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_search_last_n": 0, 2024-09-20T22:44:50.055 INFO:teuthology.orchestra.run.smithi084.stdout: "grade_table": [], 2024-09-20T22:44:50.055 INFO:teuthology.orchestra.run.smithi084.stdout: "stripe_width": 0, 2024-09-20T22:44:50.055 INFO:teuthology.orchestra.run.smithi084.stdout: "expected_num_objects": 0, 2024-09-20T22:44:50.055 INFO:teuthology.orchestra.run.smithi084.stdout: "fast_read": false, 2024-09-20T22:44:50.055 INFO:teuthology.orchestra.run.smithi084.stdout: "options": {}, 2024-09-20T22:44:50.055 INFO:teuthology.orchestra.run.smithi084.stdout: "application_metadata": { 2024-09-20T22:44:50.055 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw": {} 2024-09-20T22:44:50.055 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:44:50.055 INFO:teuthology.orchestra.run.smithi084.stdout: "read_balance": { 2024-09-20T22:44:50.055 INFO:teuthology.orchestra.run.smithi084.stdout: "score_type": "Fair distribution", 2024-09-20T22:44:50.056 INFO:teuthology.orchestra.run.smithi084.stdout: "score_acting": 1.7400000095367432, 2024-09-20T22:44:50.056 INFO:teuthology.orchestra.run.smithi084.stdout: "score_stable": 1.7400000095367432, 2024-09-20T22:44:50.056 INFO:teuthology.orchestra.run.smithi084.stdout: "optimal_score": 0.87999999523162842, 2024-09-20T22:44:50.056 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_acting": 1.5299999713897705, 2024-09-20T22:44:50.056 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_stable": 1.5299999713897705, 2024-09-20T22:44:50.056 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity_weighted": 1, 2024-09-20T22:44:50.056 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity": 1, 2024-09-20T22:44:50.056 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity_weighted": 1 2024-09-20T22:44:50.056 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:44:50.056 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:44:50.056 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:44:50.056 INFO:teuthology.orchestra.run.smithi084.stdout: "pool": 4, 2024-09-20T22:44:50.056 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_name": "default.rgw.log", 2024-09-20T22:44:50.056 INFO:teuthology.orchestra.run.smithi084.stdout: "create_time": "2024-09-20T22:41:38.454640+0000", 2024-09-20T22:44:50.057 INFO:teuthology.orchestra.run.smithi084.stdout: "flags": 1, 2024-09-20T22:44:50.057 INFO:teuthology.orchestra.run.smithi084.stdout: "flags_names": "hashpspool", 2024-09-20T22:44:50.057 INFO:teuthology.orchestra.run.smithi084.stdout: "type": 1, 2024-09-20T22:44:50.057 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 3, 2024-09-20T22:44:50.057 INFO:teuthology.orchestra.run.smithi084.stdout: "min_size": 2, 2024-09-20T22:44:50.057 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_rule": 0, 2024-09-20T22:44:50.057 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_count": 0, 2024-09-20T22:44:50.057 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_target": 0, 2024-09-20T22:44:50.057 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_barrier": 0, 2024-09-20T22:44:50.057 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-20T22:44:50.057 INFO:teuthology.orchestra.run.smithi084.stdout: "is_stretch_pool": false, 2024-09-20T22:44:50.057 INFO:teuthology.orchestra.run.smithi084.stdout: "object_hash": 2, 2024-09-20T22:44:50.057 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_autoscale_mode": "off", 2024-09-20T22:44:50.058 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num": 32, 2024-09-20T22:44:50.058 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num": 32, 2024-09-20T22:44:50.058 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num_target": 32, 2024-09-20T22:44:50.058 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_target": 32, 2024-09-20T22:44:50.058 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_pending": 32, 2024-09-20T22:44:50.058 INFO:teuthology.orchestra.run.smithi084.stdout: "last_pg_merge_meta": { 2024-09-20T22:44:50.058 INFO:teuthology.orchestra.run.smithi084.stdout: "source_pgid": "0.0", 2024-09-20T22:44:50.058 INFO:teuthology.orchestra.run.smithi084.stdout: "ready_epoch": 0, 2024-09-20T22:44:50.058 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_started": 0, 2024-09-20T22:44:50.058 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_clean": 0, 2024-09-20T22:44:50.058 INFO:teuthology.orchestra.run.smithi084.stdout: "source_version": "0'0", 2024-09-20T22:44:50.058 INFO:teuthology.orchestra.run.smithi084.stdout: "target_version": "0'0" 2024-09-20T22:44:50.058 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:44:50.059 INFO:teuthology.orchestra.run.smithi084.stdout: "last_change": "70", 2024-09-20T22:44:50.059 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend": "0", 2024-09-20T22:44:50.059 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-20T22:44:50.059 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_preluminous": "0", 2024-09-20T22:44:50.059 INFO:teuthology.orchestra.run.smithi084.stdout: "auid": 0, 2024-09-20T22:44:50.059 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_mode": "selfmanaged", 2024-09-20T22:44:50.059 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_seq": 0, 2024-09-20T22:44:50.059 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_epoch": 0, 2024-09-20T22:44:50.059 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_snaps": [], 2024-09-20T22:44:50.059 INFO:teuthology.orchestra.run.smithi084.stdout: "removed_snaps": "[]", 2024-09-20T22:44:50.060 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_bytes": 0, 2024-09-20T22:44:50.060 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_objects": 0, 2024-09-20T22:44:50.060 INFO:teuthology.orchestra.run.smithi084.stdout: "tiers": [], 2024-09-20T22:44:50.060 INFO:teuthology.orchestra.run.smithi084.stdout: "tier_of": -1, 2024-09-20T22:44:50.060 INFO:teuthology.orchestra.run.smithi084.stdout: "read_tier": -1, 2024-09-20T22:44:50.060 INFO:teuthology.orchestra.run.smithi084.stdout: "write_tier": -1, 2024-09-20T22:44:50.060 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_mode": "none", 2024-09-20T22:44:50.060 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_bytes": 0, 2024-09-20T22:44:50.060 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_objects": 0, 2024-09-20T22:44:50.060 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-20T22:44:50.060 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-20T22:44:50.060 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-20T22:44:50.060 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_flush_age": 0, 2024-09-20T22:44:50.061 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_evict_age": 0, 2024-09-20T22:44:50.061 INFO:teuthology.orchestra.run.smithi084.stdout: "erasure_code_profile": "", 2024-09-20T22:44:50.061 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_params": { 2024-09-20T22:44:50.061 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "none" 2024-09-20T22:44:50.061 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:44:50.061 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_period": 0, 2024-09-20T22:44:50.061 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_count": 0, 2024-09-20T22:44:50.061 INFO:teuthology.orchestra.run.smithi084.stdout: "use_gmt_hitset": true, 2024-09-20T22:44:50.061 INFO:teuthology.orchestra.run.smithi084.stdout: "min_read_recency_for_promote": 0, 2024-09-20T22:44:50.061 INFO:teuthology.orchestra.run.smithi084.stdout: "min_write_recency_for_promote": 0, 2024-09-20T22:44:50.061 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_grade_decay_rate": 0, 2024-09-20T22:44:50.061 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_search_last_n": 0, 2024-09-20T22:44:50.061 INFO:teuthology.orchestra.run.smithi084.stdout: "grade_table": [], 2024-09-20T22:44:50.061 INFO:teuthology.orchestra.run.smithi084.stdout: "stripe_width": 0, 2024-09-20T22:44:50.062 INFO:teuthology.orchestra.run.smithi084.stdout: "expected_num_objects": 0, 2024-09-20T22:44:50.062 INFO:teuthology.orchestra.run.smithi084.stdout: "fast_read": false, 2024-09-20T22:44:50.062 INFO:teuthology.orchestra.run.smithi084.stdout: "options": {}, 2024-09-20T22:44:50.062 INFO:teuthology.orchestra.run.smithi084.stdout: "application_metadata": { 2024-09-20T22:44:50.062 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw": {} 2024-09-20T22:44:50.062 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:44:50.062 INFO:teuthology.orchestra.run.smithi084.stdout: "read_balance": { 2024-09-20T22:44:50.062 INFO:teuthology.orchestra.run.smithi084.stdout: "score_type": "Fair distribution", 2024-09-20T22:44:50.062 INFO:teuthology.orchestra.run.smithi084.stdout: "score_acting": 2.2400000095367432, 2024-09-20T22:44:50.062 INFO:teuthology.orchestra.run.smithi084.stdout: "score_stable": 2.2400000095367432, 2024-09-20T22:44:50.063 INFO:teuthology.orchestra.run.smithi084.stdout: "optimal_score": 0.87999999523162842, 2024-09-20T22:44:50.063 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_acting": 1.9700000286102295, 2024-09-20T22:44:50.063 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_stable": 1.9700000286102295, 2024-09-20T22:44:50.063 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity_weighted": 1, 2024-09-20T22:44:50.063 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity": 1, 2024-09-20T22:44:50.063 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity_weighted": 1 2024-09-20T22:44:50.063 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:44:50.063 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:44:50.063 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:44:50.063 INFO:teuthology.orchestra.run.smithi084.stdout: "pool": 5, 2024-09-20T22:44:50.063 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_name": "default.rgw.control", 2024-09-20T22:44:50.063 INFO:teuthology.orchestra.run.smithi084.stdout: "create_time": "2024-09-20T22:41:40.407129+0000", 2024-09-20T22:44:50.063 INFO:teuthology.orchestra.run.smithi084.stdout: "flags": 1, 2024-09-20T22:44:50.063 INFO:teuthology.orchestra.run.smithi084.stdout: "flags_names": "hashpspool", 2024-09-20T22:44:50.064 INFO:teuthology.orchestra.run.smithi084.stdout: "type": 1, 2024-09-20T22:44:50.064 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 3, 2024-09-20T22:44:50.064 INFO:teuthology.orchestra.run.smithi084.stdout: "min_size": 2, 2024-09-20T22:44:50.064 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_rule": 0, 2024-09-20T22:44:50.064 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_count": 0, 2024-09-20T22:44:50.064 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_target": 0, 2024-09-20T22:44:50.064 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_barrier": 0, 2024-09-20T22:44:50.064 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-20T22:44:50.064 INFO:teuthology.orchestra.run.smithi084.stdout: "is_stretch_pool": false, 2024-09-20T22:44:50.064 INFO:teuthology.orchestra.run.smithi084.stdout: "object_hash": 2, 2024-09-20T22:44:50.064 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_autoscale_mode": "off", 2024-09-20T22:44:50.064 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num": 32, 2024-09-20T22:44:50.064 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num": 32, 2024-09-20T22:44:50.065 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num_target": 32, 2024-09-20T22:44:50.065 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_target": 32, 2024-09-20T22:44:50.065 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_pending": 32, 2024-09-20T22:44:50.065 INFO:teuthology.orchestra.run.smithi084.stdout: "last_pg_merge_meta": { 2024-09-20T22:44:50.065 INFO:teuthology.orchestra.run.smithi084.stdout: "source_pgid": "0.0", 2024-09-20T22:44:50.065 INFO:teuthology.orchestra.run.smithi084.stdout: "ready_epoch": 0, 2024-09-20T22:44:50.065 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_started": 0, 2024-09-20T22:44:50.066 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_clean": 0, 2024-09-20T22:44:50.066 INFO:teuthology.orchestra.run.smithi084.stdout: "source_version": "0'0", 2024-09-20T22:44:50.066 INFO:teuthology.orchestra.run.smithi084.stdout: "target_version": "0'0" 2024-09-20T22:44:50.066 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:44:50.066 INFO:teuthology.orchestra.run.smithi084.stdout: "last_change": "72", 2024-09-20T22:44:50.066 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend": "0", 2024-09-20T22:44:50.066 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-20T22:44:50.066 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_preluminous": "0", 2024-09-20T22:44:50.066 INFO:teuthology.orchestra.run.smithi084.stdout: "auid": 0, 2024-09-20T22:44:50.067 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_mode": "selfmanaged", 2024-09-20T22:44:50.067 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_seq": 0, 2024-09-20T22:44:50.067 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_epoch": 0, 2024-09-20T22:44:50.067 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_snaps": [], 2024-09-20T22:44:50.067 INFO:teuthology.orchestra.run.smithi084.stdout: "removed_snaps": "[]", 2024-09-20T22:44:50.067 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_bytes": 0, 2024-09-20T22:44:50.067 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_objects": 0, 2024-09-20T22:44:50.067 INFO:teuthology.orchestra.run.smithi084.stdout: "tiers": [], 2024-09-20T22:44:50.068 INFO:teuthology.orchestra.run.smithi084.stdout: "tier_of": -1, 2024-09-20T22:44:50.068 INFO:teuthology.orchestra.run.smithi084.stdout: "read_tier": -1, 2024-09-20T22:44:50.068 INFO:teuthology.orchestra.run.smithi084.stdout: "write_tier": -1, 2024-09-20T22:44:50.068 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_mode": "none", 2024-09-20T22:44:50.068 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_bytes": 0, 2024-09-20T22:44:50.068 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_objects": 0, 2024-09-20T22:44:50.068 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-20T22:44:50.068 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-20T22:44:50.069 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-20T22:44:50.069 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_flush_age": 0, 2024-09-20T22:44:50.069 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_evict_age": 0, 2024-09-20T22:44:50.069 INFO:teuthology.orchestra.run.smithi084.stdout: "erasure_code_profile": "", 2024-09-20T22:44:50.069 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_params": { 2024-09-20T22:44:50.069 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "none" 2024-09-20T22:44:50.069 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:44:50.069 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_period": 0, 2024-09-20T22:44:50.069 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_count": 0, 2024-09-20T22:44:50.070 INFO:teuthology.orchestra.run.smithi084.stdout: "use_gmt_hitset": true, 2024-09-20T22:44:50.070 INFO:teuthology.orchestra.run.smithi084.stdout: "min_read_recency_for_promote": 0, 2024-09-20T22:44:50.070 INFO:teuthology.orchestra.run.smithi084.stdout: "min_write_recency_for_promote": 0, 2024-09-20T22:44:50.070 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_grade_decay_rate": 0, 2024-09-20T22:44:50.070 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_search_last_n": 0, 2024-09-20T22:44:50.070 INFO:teuthology.orchestra.run.smithi084.stdout: "grade_table": [], 2024-09-20T22:44:50.070 INFO:teuthology.orchestra.run.smithi084.stdout: "stripe_width": 0, 2024-09-20T22:44:50.070 INFO:teuthology.orchestra.run.smithi084.stdout: "expected_num_objects": 0, 2024-09-20T22:44:50.071 INFO:teuthology.orchestra.run.smithi084.stdout: "fast_read": false, 2024-09-20T22:44:50.071 INFO:teuthology.orchestra.run.smithi084.stdout: "options": {}, 2024-09-20T22:44:50.071 INFO:teuthology.orchestra.run.smithi084.stdout: "application_metadata": { 2024-09-20T22:44:50.071 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw": {} 2024-09-20T22:44:50.071 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:44:50.071 INFO:teuthology.orchestra.run.smithi084.stdout: "read_balance": { 2024-09-20T22:44:50.071 INFO:teuthology.orchestra.run.smithi084.stdout: "score_type": "Fair distribution", 2024-09-20T22:44:50.071 INFO:teuthology.orchestra.run.smithi084.stdout: "score_acting": 1.7400000095367432, 2024-09-20T22:44:50.072 INFO:teuthology.orchestra.run.smithi084.stdout: "score_stable": 1.7400000095367432, 2024-09-20T22:44:50.072 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:44:50.074 INFO:teuthology.orchestra.run.smithi084.stdout: "optimal_score": 0.87999999523162842, 2024-09-20T22:44:50.074 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_acting": 1.5299999713897705, 2024-09-20T22:44:50.074 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_stable": 1.5299999713897705, 2024-09-20T22:44:50.074 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity_weighted": 1, 2024-09-20T22:44:50.075 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity": 1, 2024-09-20T22:44:50.075 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity_weighted": 1 2024-09-20T22:44:50.075 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:44:50.075 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:44:50.075 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:44:50.075 INFO:teuthology.orchestra.run.smithi084.stdout: "pool": 6, 2024-09-20T22:44:50.075 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_name": "default.rgw.meta", 2024-09-20T22:44:50.075 INFO:teuthology.orchestra.run.smithi084.stdout: "create_time": "2024-09-20T22:41:42.476716+0000", 2024-09-20T22:44:50.076 INFO:teuthology.orchestra.run.smithi084.stdout: "flags": 1, 2024-09-20T22:44:50.076 INFO:teuthology.orchestra.run.smithi084.stdout: "flags_names": "hashpspool", 2024-09-20T22:44:50.076 INFO:teuthology.orchestra.run.smithi084.stdout: "type": 1, 2024-09-20T22:44:50.076 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 3, 2024-09-20T22:44:50.076 INFO:teuthology.orchestra.run.smithi084.stdout: "min_size": 2, 2024-09-20T22:44:50.076 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_rule": 0, 2024-09-20T22:44:50.076 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_count": 0, 2024-09-20T22:44:50.076 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_target": 0, 2024-09-20T22:44:50.076 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_barrier": 0, 2024-09-20T22:44:50.076 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-20T22:44:50.077 INFO:teuthology.orchestra.run.smithi084.stdout: "is_stretch_pool": false, 2024-09-20T22:44:50.077 INFO:teuthology.orchestra.run.smithi084.stdout: "object_hash": 2, 2024-09-20T22:44:50.077 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_autoscale_mode": "off", 2024-09-20T22:44:50.077 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num": 32, 2024-09-20T22:44:50.077 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num": 32, 2024-09-20T22:44:50.077 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num_target": 32, 2024-09-20T22:44:50.077 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_target": 32, 2024-09-20T22:44:50.077 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_pending": 32, 2024-09-20T22:44:50.078 INFO:teuthology.orchestra.run.smithi084.stdout: "last_pg_merge_meta": { 2024-09-20T22:44:50.078 INFO:teuthology.orchestra.run.smithi084.stdout: "source_pgid": "0.0", 2024-09-20T22:44:50.078 INFO:teuthology.orchestra.run.smithi084.stdout: "ready_epoch": 0, 2024-09-20T22:44:50.078 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_started": 0, 2024-09-20T22:44:50.078 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_clean": 0, 2024-09-20T22:44:50.078 INFO:teuthology.orchestra.run.smithi084.stdout: "source_version": "0'0", 2024-09-20T22:44:50.078 INFO:teuthology.orchestra.run.smithi084.stdout: "target_version": "0'0" 2024-09-20T22:44:50.078 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:44:50.078 INFO:teuthology.orchestra.run.smithi084.stdout: "last_change": "74", 2024-09-20T22:44:50.078 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend": "0", 2024-09-20T22:44:50.078 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-20T22:44:50.079 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_preluminous": "0", 2024-09-20T22:44:50.079 INFO:teuthology.orchestra.run.smithi084.stdout: "auid": 0, 2024-09-20T22:44:50.079 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_mode": "selfmanaged", 2024-09-20T22:44:50.079 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_seq": 0, 2024-09-20T22:44:50.079 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_epoch": 0, 2024-09-20T22:44:50.079 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_snaps": [], 2024-09-20T22:44:50.079 INFO:teuthology.orchestra.run.smithi084.stdout: "removed_snaps": "[]", 2024-09-20T22:44:50.079 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_bytes": 0, 2024-09-20T22:44:50.079 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_objects": 0, 2024-09-20T22:44:50.079 INFO:teuthology.orchestra.run.smithi084.stdout: "tiers": [], 2024-09-20T22:44:50.079 INFO:teuthology.orchestra.run.smithi084.stdout: "tier_of": -1, 2024-09-20T22:44:50.079 INFO:teuthology.orchestra.run.smithi084.stdout: "read_tier": -1, 2024-09-20T22:44:50.079 INFO:teuthology.orchestra.run.smithi084.stdout: "write_tier": -1, 2024-09-20T22:44:50.080 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_mode": "none", 2024-09-20T22:44:50.080 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_bytes": 0, 2024-09-20T22:44:50.080 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_objects": 0, 2024-09-20T22:44:50.080 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-20T22:44:50.080 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-20T22:44:50.080 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-20T22:44:50.080 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_flush_age": 0, 2024-09-20T22:44:50.080 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_evict_age": 0, 2024-09-20T22:44:50.080 INFO:teuthology.orchestra.run.smithi084.stdout: "erasure_code_profile": "", 2024-09-20T22:44:50.080 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_params": { 2024-09-20T22:44:50.080 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "none" 2024-09-20T22:44:50.080 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:44:50.080 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_period": 0, 2024-09-20T22:44:50.080 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_count": 0, 2024-09-20T22:44:50.080 INFO:teuthology.orchestra.run.smithi084.stdout: "use_gmt_hitset": true, 2024-09-20T22:44:50.081 INFO:teuthology.orchestra.run.smithi084.stdout: "min_read_recency_for_promote": 0, 2024-09-20T22:44:50.081 INFO:teuthology.orchestra.run.smithi084.stdout: "min_write_recency_for_promote": 0, 2024-09-20T22:44:50.081 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_grade_decay_rate": 0, 2024-09-20T22:44:50.081 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_search_last_n": 0, 2024-09-20T22:44:50.081 INFO:teuthology.orchestra.run.smithi084.stdout: "grade_table": [], 2024-09-20T22:44:50.081 INFO:teuthology.orchestra.run.smithi084.stdout: "stripe_width": 0, 2024-09-20T22:44:50.081 INFO:teuthology.orchestra.run.smithi084.stdout: "expected_num_objects": 0, 2024-09-20T22:44:50.081 INFO:teuthology.orchestra.run.smithi084.stdout: "fast_read": false, 2024-09-20T22:44:50.081 INFO:teuthology.orchestra.run.smithi084.stdout: "options": { 2024-09-20T22:44:50.081 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_autoscale_bias": 4 2024-09-20T22:44:50.081 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:44:50.081 INFO:teuthology.orchestra.run.smithi084.stdout: "application_metadata": { 2024-09-20T22:44:50.081 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw": {} 2024-09-20T22:44:50.082 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:44:50.082 INFO:teuthology.orchestra.run.smithi084.stdout: "read_balance": { 2024-09-20T22:44:50.082 INFO:teuthology.orchestra.run.smithi084.stdout: "score_type": "Fair distribution", 2024-09-20T22:44:50.082 INFO:teuthology.orchestra.run.smithi084.stdout: "score_acting": 1.9900000095367432, 2024-09-20T22:44:50.082 INFO:teuthology.orchestra.run.smithi084.stdout: "score_stable": 1.9900000095367432, 2024-09-20T22:44:50.082 INFO:teuthology.orchestra.run.smithi084.stdout: "optimal_score": 0.87999999523162842, 2024-09-20T22:44:50.082 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_acting": 1.75, 2024-09-20T22:44:50.082 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_stable": 1.75, 2024-09-20T22:44:50.082 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity_weighted": 1, 2024-09-20T22:44:50.082 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity": 1, 2024-09-20T22:44:50.082 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity_weighted": 1 2024-09-20T22:44:50.083 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:44:50.083 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:44:50.083 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:44:50.083 INFO:teuthology.orchestra.run.smithi084.stdout: "pool": 7, 2024-09-20T22:44:50.083 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_name": "unique_pool_0", 2024-09-20T22:44:50.083 INFO:teuthology.orchestra.run.smithi084.stdout: "create_time": "2024-09-20T22:42:51.354358+0000", 2024-09-20T22:44:50.083 INFO:teuthology.orchestra.run.smithi084.stdout: "flags": 8193, 2024-09-20T22:44:50.083 INFO:teuthology.orchestra.run.smithi084.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-20T22:44:50.083 INFO:teuthology.orchestra.run.smithi084.stdout: "type": 1, 2024-09-20T22:44:50.083 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 3, 2024-09-20T22:44:50.083 INFO:teuthology.orchestra.run.smithi084.stdout: "min_size": 2, 2024-09-20T22:44:50.083 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_rule": 0, 2024-09-20T22:44:50.083 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_count": 0, 2024-09-20T22:44:50.083 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_target": 0, 2024-09-20T22:44:50.084 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_barrier": 0, 2024-09-20T22:44:50.084 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-20T22:44:50.084 INFO:teuthology.orchestra.run.smithi084.stdout: "is_stretch_pool": false, 2024-09-20T22:44:50.084 INFO:teuthology.orchestra.run.smithi084.stdout: "object_hash": 2, 2024-09-20T22:44:50.084 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_autoscale_mode": "off", 2024-09-20T22:44:50.084 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num": 26, 2024-09-20T22:44:50.084 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num": 26, 2024-09-20T22:44:50.084 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num_target": 26, 2024-09-20T22:44:50.084 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_target": 26, 2024-09-20T22:44:50.084 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_pending": 26, 2024-09-20T22:44:50.084 INFO:teuthology.orchestra.run.smithi084.stdout: "last_pg_merge_meta": { 2024-09-20T22:44:50.084 INFO:teuthology.orchestra.run.smithi084.stdout: "source_pgid": "0.0", 2024-09-20T22:44:50.084 INFO:teuthology.orchestra.run.smithi084.stdout: "ready_epoch": 0, 2024-09-20T22:44:50.084 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_started": 0, 2024-09-20T22:44:50.085 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_clean": 0, 2024-09-20T22:44:50.085 INFO:teuthology.orchestra.run.smithi084.stdout: "source_version": "0'0", 2024-09-20T22:44:50.085 INFO:teuthology.orchestra.run.smithi084.stdout: "target_version": "0'0" 2024-09-20T22:44:50.085 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:44:50.085 INFO:teuthology.orchestra.run.smithi084.stdout: "last_change": "210", 2024-09-20T22:44:50.085 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend": "0", 2024-09-20T22:44:50.085 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-20T22:44:50.085 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_preluminous": "137", 2024-09-20T22:44:50.085 INFO:teuthology.orchestra.run.smithi084.stdout: "auid": 0, 2024-09-20T22:44:50.085 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_mode": "selfmanaged", 2024-09-20T22:44:50.085 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_seq": 58, 2024-09-20T22:44:50.085 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_epoch": 210, 2024-09-20T22:44:50.085 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_snaps": [], 2024-09-20T22:44:50.085 INFO:teuthology.orchestra.run.smithi084.stdout: "removed_snaps": "[]", 2024-09-20T22:44:50.086 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_bytes": 0, 2024-09-20T22:44:50.086 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_objects": 0, 2024-09-20T22:44:50.086 INFO:teuthology.orchestra.run.smithi084.stdout: "tiers": [], 2024-09-20T22:44:50.086 INFO:teuthology.orchestra.run.smithi084.stdout: "tier_of": -1, 2024-09-20T22:44:50.086 INFO:teuthology.orchestra.run.smithi084.stdout: "read_tier": -1, 2024-09-20T22:44:50.086 INFO:teuthology.orchestra.run.smithi084.stdout: "write_tier": -1, 2024-09-20T22:44:50.086 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_mode": "none", 2024-09-20T22:44:50.086 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_bytes": 0, 2024-09-20T22:44:50.086 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_objects": 0, 2024-09-20T22:44:50.086 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-20T22:44:50.086 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-20T22:44:50.086 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-20T22:44:50.086 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_flush_age": 0, 2024-09-20T22:44:50.087 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_evict_age": 0, 2024-09-20T22:44:50.087 INFO:teuthology.orchestra.run.smithi084.stdout: "erasure_code_profile": "", 2024-09-20T22:44:50.087 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_params": { 2024-09-20T22:44:50.087 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "none" 2024-09-20T22:44:50.087 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:44:50.087 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_period": 0, 2024-09-20T22:44:50.087 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_count": 0, 2024-09-20T22:44:50.087 INFO:teuthology.orchestra.run.smithi084.stdout: "use_gmt_hitset": true, 2024-09-20T22:44:50.087 INFO:teuthology.orchestra.run.smithi084.stdout: "min_read_recency_for_promote": 0, 2024-09-20T22:44:50.087 INFO:teuthology.orchestra.run.smithi084.stdout: "min_write_recency_for_promote": 0, 2024-09-20T22:44:50.087 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_grade_decay_rate": 0, 2024-09-20T22:44:50.087 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_search_last_n": 0, 2024-09-20T22:44:50.088 INFO:teuthology.orchestra.run.smithi084.stdout: "grade_table": [], 2024-09-20T22:44:50.088 INFO:teuthology.orchestra.run.smithi084.stdout: "stripe_width": 0, 2024-09-20T22:44:50.088 INFO:teuthology.orchestra.run.smithi084.stdout: "expected_num_objects": 0, 2024-09-20T22:44:50.088 INFO:teuthology.orchestra.run.smithi084.stdout: "fast_read": false, 2024-09-20T22:44:50.088 INFO:teuthology.orchestra.run.smithi084.stdout: "options": {}, 2024-09-20T22:44:50.088 INFO:teuthology.orchestra.run.smithi084.stdout: "application_metadata": { 2024-09-20T22:44:50.088 INFO:teuthology.orchestra.run.smithi084.stdout: "rados": {} 2024-09-20T22:44:50.088 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:44:50.088 INFO:teuthology.orchestra.run.smithi084.stdout: "read_balance": { 2024-09-20T22:44:50.088 INFO:teuthology.orchestra.run.smithi084.stdout: "score_type": "Fair distribution", 2024-09-20T22:44:50.088 INFO:teuthology.orchestra.run.smithi084.stdout: "score_acting": 2.1400001049041748, 2024-09-20T22:44:50.088 INFO:teuthology.orchestra.run.smithi084.stdout: "score_stable": 2.1400001049041748, 2024-09-20T22:44:50.088 INFO:teuthology.orchestra.run.smithi084.stdout: "optimal_score": 0.87999999523162842, 2024-09-20T22:44:50.089 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_acting": 1.8799999952316284, 2024-09-20T22:44:50.089 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_stable": 1.8799999952316284, 2024-09-20T22:44:50.089 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity_weighted": 1, 2024-09-20T22:44:50.089 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity": 1, 2024-09-20T22:44:50.089 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity_weighted": 1 2024-09-20T22:44:50.089 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:44:50.089 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:44:50.089 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2024-09-20T22:44:50.089 INFO:teuthology.orchestra.run.smithi084.stdout: "osds": [ 2024-09-20T22:44:50.089 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:44:50.089 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 0, 2024-09-20T22:44:50.089 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "ea36a489-6f91-4b2f-b784-b90506b9805d", 2024-09-20T22:44:50.089 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-09-20T22:44:50.089 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-09-20T22:44:50.090 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-09-20T22:44:50.090 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-09-20T22:44:50.090 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-09-20T22:44:50.090 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-09-20T22:44:50.090 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 8, 2024-09-20T22:44:50.090 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 169, 2024-09-20T22:44:50.090 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-09-20T22:44:50.090 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-09-20T22:44:50.090 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-09-20T22:44:50.090 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:44:50.090 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:44:50.090 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:44:50.090 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6801", 2024-09-20T22:44:50.090 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 268084626 2024-09-20T22:44:50.090 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:44:50.091 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:44:50.091 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:44:50.091 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-09-20T22:44:50.091 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:44:50.091 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:44:50.091 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:44:50.091 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6802", 2024-09-20T22:44:50.091 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 268084626 2024-09-20T22:44:50.091 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:44:50.091 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:44:50.091 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:44:50.091 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-09-20T22:44:50.091 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:44:50.091 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:44:50.092 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:44:50.092 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6804", 2024-09-20T22:44:50.092 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 268084626 2024-09-20T22:44:50.092 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:44:50.092 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:44:50.092 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:44:50.092 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-09-20T22:44:50.092 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:44:50.092 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:44:50.092 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:44:50.092 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6803", 2024-09-20T22:44:50.092 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 268084626 2024-09-20T22:44:50.092 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:44:50.092 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:44:50.092 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:44:50.093 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.84:6801/268084626", 2024-09-20T22:44:50.093 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.84:6802/268084626", 2024-09-20T22:44:50.093 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.84:6804/268084626", 2024-09-20T22:44:50.093 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.84:6803/268084626", 2024-09-20T22:44:50.093 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-09-20T22:44:50.093 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-09-20T22:44:50.093 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-09-20T22:44:50.093 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:44:50.093 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:44:50.093 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:44:50.093 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 1, 2024-09-20T22:44:50.093 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "c8649d2f-27ef-47f8-bc94-f1cd9e62bdb8", 2024-09-20T22:44:50.094 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-09-20T22:44:50.094 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-09-20T22:44:50.094 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-09-20T22:44:50.094 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-09-20T22:44:50.094 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-09-20T22:44:50.094 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-09-20T22:44:50.094 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 13, 2024-09-20T22:44:50.094 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 160, 2024-09-20T22:44:50.094 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-09-20T22:44:50.094 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-09-20T22:44:50.094 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-09-20T22:44:50.094 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:44:50.094 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:44:50.095 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:44:50.095 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6805", 2024-09-20T22:44:50.095 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 418374136 2024-09-20T22:44:50.095 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:44:50.095 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:44:50.095 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:44:50.095 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-09-20T22:44:50.095 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:44:50.095 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:44:50.095 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:44:50.095 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6806", 2024-09-20T22:44:50.095 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 418374136 2024-09-20T22:44:50.096 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:44:50.096 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:44:50.096 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:44:50.096 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-09-20T22:44:50.096 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:44:50.096 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:44:50.096 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:44:50.096 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6808", 2024-09-20T22:44:50.096 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 418374136 2024-09-20T22:44:50.096 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:44:50.096 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:44:50.096 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:44:50.096 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-09-20T22:44:50.097 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:44:50.097 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:44:50.097 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:44:50.097 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6807", 2024-09-20T22:44:50.097 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 418374136 2024-09-20T22:44:50.097 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:44:50.097 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:44:50.097 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:44:50.097 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.84:6805/418374136", 2024-09-20T22:44:50.097 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.84:6806/418374136", 2024-09-20T22:44:50.097 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.84:6808/418374136", 2024-09-20T22:44:50.097 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.84:6807/418374136", 2024-09-20T22:44:50.098 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-09-20T22:44:50.098 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-09-20T22:44:50.098 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-09-20T22:44:50.098 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:44:50.098 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:44:50.098 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:44:50.098 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 2, 2024-09-20T22:44:50.098 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "c728eb48-a0f0-4de4-a651-7f3bec867280", 2024-09-20T22:44:50.098 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-09-20T22:44:50.098 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-09-20T22:44:50.098 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-09-20T22:44:50.098 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-09-20T22:44:50.098 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-09-20T22:44:50.099 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-09-20T22:44:50.099 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 18, 2024-09-20T22:44:50.099 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 160, 2024-09-20T22:44:50.099 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-09-20T22:44:50.099 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-09-20T22:44:50.099 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-09-20T22:44:50.099 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:44:50.099 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:44:50.099 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:44:50.099 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6809", 2024-09-20T22:44:50.100 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3977168611 2024-09-20T22:44:50.100 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:44:50.100 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:44:50.100 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:44:50.100 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-09-20T22:44:50.100 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:44:50.100 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:44:50.100 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:44:50.100 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6810", 2024-09-20T22:44:50.100 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3977168611 2024-09-20T22:44:50.100 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:44:50.100 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:44:50.100 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:44:50.100 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-09-20T22:44:50.100 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:44:50.101 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:44:50.101 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:44:50.101 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6812", 2024-09-20T22:44:50.101 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3977168611 2024-09-20T22:44:50.101 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:44:50.101 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:44:50.101 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:44:50.101 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-09-20T22:44:50.101 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:44:50.101 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:44:50.101 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:44:50.101 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6811", 2024-09-20T22:44:50.101 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3977168611 2024-09-20T22:44:50.101 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:44:50.101 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:44:50.102 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:44:50.102 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.84:6809/3977168611", 2024-09-20T22:44:50.102 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.84:6810/3977168611", 2024-09-20T22:44:50.102 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.84:6812/3977168611", 2024-09-20T22:44:50.102 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.84:6811/3977168611", 2024-09-20T22:44:50.102 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-09-20T22:44:50.102 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-09-20T22:44:50.102 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-09-20T22:44:50.102 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:44:50.102 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:44:50.102 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:44:50.102 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 3, 2024-09-20T22:44:50.102 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "573f44eb-0853-4630-a7d8-9caaeb256f9b", 2024-09-20T22:44:50.102 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-09-20T22:44:50.103 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-09-20T22:44:50.103 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-09-20T22:44:50.103 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-09-20T22:44:50.103 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-09-20T22:44:50.103 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-09-20T22:44:50.103 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 25, 2024-09-20T22:44:50.103 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 169, 2024-09-20T22:44:50.103 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-09-20T22:44:50.103 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-09-20T22:44:50.103 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-09-20T22:44:50.103 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:44:50.103 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:44:50.103 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:44:50.103 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6813", 2024-09-20T22:44:50.103 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2192239945 2024-09-20T22:44:50.104 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:44:50.104 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:44:50.104 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:44:50.104 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-09-20T22:44:50.104 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:44:50.104 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:44:50.104 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:44:50.104 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6814", 2024-09-20T22:44:50.104 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2192239945 2024-09-20T22:44:50.104 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:44:50.104 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:44:50.104 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:44:50.104 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-09-20T22:44:50.104 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:44:50.104 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:44:50.105 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:44:50.105 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6816", 2024-09-20T22:44:50.105 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2192239945 2024-09-20T22:44:50.105 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:44:50.105 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:44:50.105 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:44:50.105 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-09-20T22:44:50.105 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:44:50.105 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:44:50.105 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:44:50.105 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6815", 2024-09-20T22:44:50.105 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2192239945 2024-09-20T22:44:50.105 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:44:50.105 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:44:50.105 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:44:50.106 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.84:6813/2192239945", 2024-09-20T22:44:50.106 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.84:6814/2192239945", 2024-09-20T22:44:50.106 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.84:6816/2192239945", 2024-09-20T22:44:50.106 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.84:6815/2192239945", 2024-09-20T22:44:50.106 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-09-20T22:44:50.106 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-09-20T22:44:50.106 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-09-20T22:44:50.106 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:44:50.106 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:44:50.106 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:44:50.106 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 4, 2024-09-20T22:44:50.106 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "a52c3b94-c5f3-490f-8ac8-09061b9b2abb", 2024-09-20T22:44:50.107 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-09-20T22:44:50.107 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 0, 2024-09-20T22:44:50.107 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 0, 2024-09-20T22:44:50.107 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-09-20T22:44:50.107 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-09-20T22:44:50.107 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-09-20T22:44:50.107 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 31, 2024-09-20T22:44:50.107 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 139, 2024-09-20T22:44:50.107 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-09-20T22:44:50.107 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-09-20T22:44:50.107 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-09-20T22:44:50.107 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:44:50.107 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:44:50.107 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:44:50.108 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.154:6800", 2024-09-20T22:44:50.108 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1793136002 2024-09-20T22:44:50.108 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:44:50.108 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:44:50.108 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:44:50.108 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-09-20T22:44:50.108 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:44:50.108 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:44:50.108 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:44:50.108 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.154:6801", 2024-09-20T22:44:50.108 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1793136002 2024-09-20T22:44:50.108 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:44:50.108 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:44:50.108 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:44:50.108 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-09-20T22:44:50.109 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:44:50.109 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:44:50.109 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:44:50.109 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.154:6803", 2024-09-20T22:44:50.109 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1793136002 2024-09-20T22:44:50.109 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:44:50.109 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:44:50.109 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:44:50.109 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-09-20T22:44:50.109 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:44:50.109 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:44:50.109 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:44:50.109 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.154:6802", 2024-09-20T22:44:50.109 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1793136002 2024-09-20T22:44:50.109 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:44:50.109 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:44:50.110 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:44:50.110 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.154:6800/1793136002", 2024-09-20T22:44:50.110 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.154:6801/1793136002", 2024-09-20T22:44:50.110 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.154:6803/1793136002", 2024-09-20T22:44:50.110 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.154:6802/1793136002", 2024-09-20T22:44:50.110 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-09-20T22:44:50.110 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-09-20T22:44:50.110 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-09-20T22:44:50.110 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:44:50.110 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:44:50.110 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:44:50.110 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 5, 2024-09-20T22:44:50.110 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "4fa0dc8e-25cc-46cd-96ff-33f20b6faec3", 2024-09-20T22:44:50.110 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-09-20T22:44:50.110 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-09-20T22:44:50.111 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-09-20T22:44:50.111 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-09-20T22:44:50.111 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-09-20T22:44:50.111 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-09-20T22:44:50.111 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 40, 2024-09-20T22:44:50.111 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 169, 2024-09-20T22:44:50.111 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-09-20T22:44:50.111 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-09-20T22:44:50.111 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-09-20T22:44:50.111 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:44:50.111 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:44:50.111 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:44:50.111 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.154:6804", 2024-09-20T22:44:50.111 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 493899927 2024-09-20T22:44:50.111 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:44:50.111 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:44:50.112 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:44:50.112 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-09-20T22:44:50.112 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:44:50.112 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:44:50.112 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:44:50.112 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.154:6805", 2024-09-20T22:44:50.112 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 493899927 2024-09-20T22:44:50.112 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:44:50.112 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:44:50.112 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:44:50.112 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-09-20T22:44:50.112 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:44:50.112 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:44:50.112 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:44:50.113 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.154:6807", 2024-09-20T22:44:50.113 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 493899927 2024-09-20T22:44:50.113 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:44:50.113 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:44:50.113 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:44:50.113 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-09-20T22:44:50.113 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:44:50.113 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:44:50.113 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:44:50.113 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.154:6806", 2024-09-20T22:44:50.113 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 493899927 2024-09-20T22:44:50.113 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:44:50.113 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:44:50.113 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:44:50.114 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.154:6804/493899927", 2024-09-20T22:44:50.114 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.154:6805/493899927", 2024-09-20T22:44:50.114 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.154:6807/493899927", 2024-09-20T22:44:50.114 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.154:6806/493899927", 2024-09-20T22:44:50.114 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-09-20T22:44:50.114 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-09-20T22:44:50.114 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-09-20T22:44:50.114 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:44:50.114 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:44:50.114 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:44:50.114 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 6, 2024-09-20T22:44:50.114 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "56788d74-810e-4827-a615-958c12f8a368", 2024-09-20T22:44:50.114 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-09-20T22:44:50.114 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-09-20T22:44:50.115 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-09-20T22:44:50.115 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-09-20T22:44:50.115 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-09-20T22:44:50.115 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-09-20T22:44:50.115 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 48, 2024-09-20T22:44:50.115 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 169, 2024-09-20T22:44:50.115 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-09-20T22:44:50.115 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-09-20T22:44:50.115 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-09-20T22:44:50.115 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:44:50.115 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:44:50.115 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:44:50.115 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.154:6808", 2024-09-20T22:44:50.116 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2375091033 2024-09-20T22:44:50.116 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:44:50.116 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:44:50.116 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:44:50.116 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-09-20T22:44:50.116 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:44:50.116 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:44:50.116 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:44:50.116 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.154:6809", 2024-09-20T22:44:50.116 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2375091033 2024-09-20T22:44:50.116 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:44:50.116 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:44:50.117 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:44:50.117 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-09-20T22:44:50.117 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:44:50.117 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:44:50.117 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:44:50.117 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.154:6811", 2024-09-20T22:44:50.117 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2375091033 2024-09-20T22:44:50.117 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:44:50.117 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:44:50.117 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:44:50.117 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-09-20T22:44:50.117 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:44:50.117 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:44:50.117 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:44:50.118 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.154:6810", 2024-09-20T22:44:50.118 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2375091033 2024-09-20T22:44:50.118 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:44:50.118 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:44:50.118 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:44:50.118 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.154:6808/2375091033", 2024-09-20T22:44:50.118 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.154:6809/2375091033", 2024-09-20T22:44:50.118 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.154:6811/2375091033", 2024-09-20T22:44:50.118 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.154:6810/2375091033", 2024-09-20T22:44:50.118 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-09-20T22:44:50.118 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-09-20T22:44:50.118 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-09-20T22:44:50.119 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:44:50.119 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:44:50.119 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:44:50.119 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 7, 2024-09-20T22:44:50.119 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "83503727-afc1-4b95-8b96-ecb63f3338c2", 2024-09-20T22:44:50.119 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-09-20T22:44:50.119 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-09-20T22:44:50.119 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-09-20T22:44:50.119 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-09-20T22:44:50.119 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-09-20T22:44:50.119 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-09-20T22:44:50.119 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 59, 2024-09-20T22:44:50.119 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 169, 2024-09-20T22:44:50.120 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-09-20T22:44:50.120 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-09-20T22:44:50.120 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-09-20T22:44:50.120 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:44:50.120 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:44:50.120 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:44:50.120 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.154:6812", 2024-09-20T22:44:50.120 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3717704491 2024-09-20T22:44:50.120 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:44:50.120 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:44:50.120 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:44:50.120 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-09-20T22:44:50.121 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:44:50.121 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:44:50.121 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:44:50.121 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.154:6813", 2024-09-20T22:44:50.121 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3717704491 2024-09-20T22:44:50.121 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:44:50.121 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:44:50.121 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:44:50.121 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-09-20T22:44:50.121 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:44:50.121 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:44:50.121 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:44:50.121 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.154:6815", 2024-09-20T22:44:50.122 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3717704491 2024-09-20T22:44:50.122 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:44:50.122 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:44:50.122 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:44:50.122 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-09-20T22:44:50.122 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:44:50.122 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:44:50.122 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:44:50.122 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.154:6814", 2024-09-20T22:44:50.122 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3717704491 2024-09-20T22:44:50.122 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:44:50.122 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:44:50.122 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:44:50.122 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.154:6812/3717704491", 2024-09-20T22:44:50.122 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.154:6813/3717704491", 2024-09-20T22:44:50.123 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.154:6815/3717704491", 2024-09-20T22:44:50.123 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.154:6814/3717704491", 2024-09-20T22:44:50.123 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-09-20T22:44:50.123 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-09-20T22:44:50.123 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-09-20T22:44:50.123 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:44:50.123 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:44:50.123 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2024-09-20T22:44:50.123 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_xinfo": [ 2024-09-20T22:44:50.123 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:44:50.123 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 0, 2024-09-20T22:44:50.123 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-09-20T22:44:50.123 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-09-20T22:44:50.123 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-09-20T22:44:50.124 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540701547738038271, 2024-09-20T22:44:50.124 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-09-20T22:44:50.124 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-09-20T22:43:38.373571+0000", 2024-09-20T22:44:50.124 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-09-20T22:44:50.124 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:44:50.124 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:44:50.124 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 1, 2024-09-20T22:44:50.124 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-09-20T22:44:50.124 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-09-20T22:44:50.124 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-09-20T22:44:50.124 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540701547738038271, 2024-09-20T22:44:50.124 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-09-20T22:44:50.124 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-09-20T22:44:01.955517+0000", 2024-09-20T22:44:50.124 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-09-20T22:44:50.124 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:44:50.125 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:44:50.125 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 2, 2024-09-20T22:44:50.125 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-09-20T22:44:50.125 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-09-20T22:44:50.125 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-09-20T22:44:50.125 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540701547738038271, 2024-09-20T22:44:50.125 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-09-20T22:44:50.125 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-09-20T22:43:46.670431+0000", 2024-09-20T22:44:50.125 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-09-20T22:44:50.125 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:44:50.125 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:44:50.125 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 3, 2024-09-20T22:44:50.125 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-09-20T22:44:50.125 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-09-20T22:44:50.126 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-09-20T22:44:50.126 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540701547738038271, 2024-09-20T22:44:50.126 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-09-20T22:44:50.126 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-09-20T22:43:44.778504+0000", 2024-09-20T22:44:50.126 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-09-20T22:44:50.126 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:44:50.126 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:44:50.126 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 4, 2024-09-20T22:44:50.126 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-09-20T22:44:50.126 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-09-20T22:44:50.126 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-09-20T22:44:50.126 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540701547738038271, 2024-09-20T22:44:50.126 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 65536, 2024-09-20T22:44:50.127 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-09-20T22:43:34.154869+0000", 2024-09-20T22:44:50.127 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-09-20T22:44:50.127 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:44:50.127 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:44:50.127 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 5, 2024-09-20T22:44:50.127 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-09-20T22:44:50.127 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-09-20T22:44:50.127 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-09-20T22:44:50.127 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540701547738038271, 2024-09-20T22:44:50.127 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-09-20T22:44:50.127 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-09-20T22:44:33.074031+0000", 2024-09-20T22:44:50.127 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-09-20T22:44:50.127 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:44:50.128 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:44:50.128 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 6, 2024-09-20T22:44:50.128 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-09-20T22:44:50.128 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-09-20T22:44:50.128 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-09-20T22:44:50.128 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540701547738038271, 2024-09-20T22:44:50.128 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-09-20T22:44:50.128 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-09-20T22:44:30.460653+0000", 2024-09-20T22:44:50.128 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-09-20T22:44:50.128 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:44:50.128 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:44:50.128 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 7, 2024-09-20T22:44:50.128 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-09-20T22:44:50.129 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-09-20T22:44:50.129 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-09-20T22:44:50.129 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540701547738038271, 2024-09-20T22:44:50.129 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-09-20T22:44:50.129 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-09-20T22:43:53.020113+0000", 2024-09-20T22:44:50.129 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-09-20T22:44:50.129 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:44:50.129 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2024-09-20T22:44:50.129 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_upmap": [], 2024-09-20T22:44:50.129 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_upmap_items": [ 2024-09-20T22:44:50.129 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:44:50.129 INFO:teuthology.orchestra.run.smithi084.stdout: "pgid": "4.6", 2024-09-20T22:44:50.129 INFO:teuthology.orchestra.run.smithi084.stdout: "mappings": [ 2024-09-20T22:44:50.130 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:44:50.130 INFO:teuthology.orchestra.run.smithi084.stdout: "from": 1, 2024-09-20T22:44:50.130 INFO:teuthology.orchestra.run.smithi084.stdout: "to": 5 2024-09-20T22:44:50.130 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:44:50.130 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:44:50.130 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:44:50.130 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:44:50.130 INFO:teuthology.orchestra.run.smithi084.stdout: "pgid": "4.e", 2024-09-20T22:44:50.130 INFO:teuthology.orchestra.run.smithi084.stdout: "mappings": [ 2024-09-20T22:44:50.130 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:44:50.130 INFO:teuthology.orchestra.run.smithi084.stdout: "from": 3, 2024-09-20T22:44:50.130 INFO:teuthology.orchestra.run.smithi084.stdout: "to": 2 2024-09-20T22:44:50.130 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:44:50.131 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:44:50.131 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:44:50.131 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:44:50.131 INFO:teuthology.orchestra.run.smithi084.stdout: "pgid": "4.11", 2024-09-20T22:44:50.131 INFO:teuthology.orchestra.run.smithi084.stdout: "mappings": [ 2024-09-20T22:44:50.131 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:44:50.131 INFO:teuthology.orchestra.run.smithi084.stdout: "from": 3, 2024-09-20T22:44:50.131 INFO:teuthology.orchestra.run.smithi084.stdout: "to": 5 2024-09-20T22:44:50.131 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:44:50.131 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:44:50.131 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:44:50.131 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:44:50.131 INFO:teuthology.orchestra.run.smithi084.stdout: "pgid": "4.17", 2024-09-20T22:44:50.131 INFO:teuthology.orchestra.run.smithi084.stdout: "mappings": [ 2024-09-20T22:44:50.132 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:44:50.132 INFO:teuthology.orchestra.run.smithi084.stdout: "from": 1, 2024-09-20T22:44:50.132 INFO:teuthology.orchestra.run.smithi084.stdout: "to": 2 2024-09-20T22:44:50.132 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:44:50.132 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:44:50.132 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:44:50.132 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:44:50.132 INFO:teuthology.orchestra.run.smithi084.stdout: "pgid": "4.1d", 2024-09-20T22:44:50.132 INFO:teuthology.orchestra.run.smithi084.stdout: "mappings": [ 2024-09-20T22:44:50.132 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:44:50.132 INFO:teuthology.orchestra.run.smithi084.stdout: "from": 1, 2024-09-20T22:44:50.132 INFO:teuthology.orchestra.run.smithi084.stdout: "to": 2 2024-09-20T22:44:50.132 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:44:50.133 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:44:50.133 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:44:50.133 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:44:50.133 INFO:teuthology.orchestra.run.smithi084.stdout: "pgid": "5.3", 2024-09-20T22:44:50.133 INFO:teuthology.orchestra.run.smithi084.stdout: "mappings": [ 2024-09-20T22:44:50.133 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:44:50.133 INFO:teuthology.orchestra.run.smithi084.stdout: "from": 0, 2024-09-20T22:44:50.133 INFO:teuthology.orchestra.run.smithi084.stdout: "to": 3 2024-09-20T22:44:50.133 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:44:50.133 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:44:50.133 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:44:50.133 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:44:50.133 INFO:teuthology.orchestra.run.smithi084.stdout: "pgid": "7.2", 2024-09-20T22:44:50.133 INFO:teuthology.orchestra.run.smithi084.stdout: "mappings": [ 2024-09-20T22:44:50.134 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:44:50.134 INFO:teuthology.orchestra.run.smithi084.stdout: "from": 3, 2024-09-20T22:44:50.134 INFO:teuthology.orchestra.run.smithi084.stdout: "to": 5 2024-09-20T22:44:50.134 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:44:50.134 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:44:50.134 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:44:50.134 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:44:50.134 INFO:teuthology.orchestra.run.smithi084.stdout: "pgid": "7.13", 2024-09-20T22:44:50.134 INFO:teuthology.orchestra.run.smithi084.stdout: "mappings": [ 2024-09-20T22:44:50.134 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:44:50.134 INFO:teuthology.orchestra.run.smithi084.stdout: "from": 7, 2024-09-20T22:44:50.134 INFO:teuthology.orchestra.run.smithi084.stdout: "to": 1 2024-09-20T22:44:50.134 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:44:50.135 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:44:50.135 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:44:50.135 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:44:50.135 INFO:teuthology.orchestra.run.smithi084.stdout: "pgid": "7.15", 2024-09-20T22:44:50.135 INFO:teuthology.orchestra.run.smithi084.stdout: "mappings": [ 2024-09-20T22:44:50.135 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:44:50.135 INFO:teuthology.orchestra.run.smithi084.stdout: "from": 2, 2024-09-20T22:44:50.135 INFO:teuthology.orchestra.run.smithi084.stdout: "to": 6 2024-09-20T22:44:50.135 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:44:50.135 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:44:50.135 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:44:50.135 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2024-09-20T22:44:50.135 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_upmap_primaries": [], 2024-09-20T22:44:50.136 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_temp": [], 2024-09-20T22:44:50.136 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_temp": [], 2024-09-20T22:44:50.136 INFO:teuthology.orchestra.run.smithi084.stdout: "blocklist": { 2024-09-20T22:44:50.136 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/542020545": "2024-09-21T22:42:07.596045+0000", 2024-09-20T22:44:50.136 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/25873236": "2024-09-21T22:36:54.917812+0000", 2024-09-20T22:44:50.136 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:6800/820908470": "2024-09-21T22:36:54.917812+0000", 2024-09-20T22:44:50.136 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/567614908": "2024-09-21T22:36:54.917812+0000", 2024-09-20T22:44:50.136 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/2963622459": "2024-09-21T22:42:07.596045+0000", 2024-09-20T22:44:50.136 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/1176849276": "2024-09-21T22:37:17.690104+0000", 2024-09-20T22:44:50.136 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:6800/2932420050": "2024-09-21T22:37:17.690104+0000", 2024-09-20T22:44:50.136 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/3879263635": "2024-09-21T22:37:17.690104+0000", 2024-09-20T22:44:50.136 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/969472279": "2024-09-21T22:36:54.917812+0000", 2024-09-20T22:44:50.136 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/1361143214": "2024-09-21T22:37:17.690104+0000", 2024-09-20T22:44:50.136 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/3353126955": "2024-09-21T22:42:07.596045+0000", 2024-09-20T22:44:50.137 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:6800/4024278029": "2024-09-21T22:42:07.596045+0000", 2024-09-20T22:44:50.137 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/547902262": "2024-09-21T22:42:07.596045+0000", 2024-09-20T22:44:50.137 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/1242929317": "2024-09-21T22:42:07.596045+0000" 2024-09-20T22:44:50.137 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:44:50.137 INFO:teuthology.orchestra.run.smithi084.stdout: "range_blocklist": {}, 2024-09-20T22:44:50.137 INFO:teuthology.orchestra.run.smithi084.stdout: "erasure_code_profiles": { 2024-09-20T22:44:50.137 INFO:teuthology.orchestra.run.smithi084.stdout: "default": { 2024-09-20T22:44:50.137 INFO:teuthology.orchestra.run.smithi084.stdout: "crush-failure-domain": "osd", 2024-09-20T22:44:50.137 INFO:teuthology.orchestra.run.smithi084.stdout: "k": "2", 2024-09-20T22:44:50.137 INFO:teuthology.orchestra.run.smithi084.stdout: "m": "1", 2024-09-20T22:44:50.137 INFO:teuthology.orchestra.run.smithi084.stdout: "plugin": "jerasure", 2024-09-20T22:44:50.137 INFO:teuthology.orchestra.run.smithi084.stdout: "technique": "reed_sol_van" 2024-09-20T22:44:50.137 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:44:50.137 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:44:50.138 INFO:teuthology.orchestra.run.smithi084.stdout: "removed_snaps_queue": [ 2024-09-20T22:44:50.138 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:44:50.138 INFO:teuthology.orchestra.run.smithi084.stdout: "pool": 7, 2024-09-20T22:44:50.138 INFO:teuthology.orchestra.run.smithi084.stdout: "snaps": [ 2024-09-20T22:44:50.138 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:44:50.138 INFO:teuthology.orchestra.run.smithi084.stdout: "begin": 53, 2024-09-20T22:44:50.138 INFO:teuthology.orchestra.run.smithi084.stdout: "length": 4 2024-09-20T22:44:50.138 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:44:50.138 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:44:50.138 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:44:50.138 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2024-09-20T22:44:50.138 INFO:teuthology.orchestra.run.smithi084.stdout: "new_removed_snaps": [ 2024-09-20T22:44:50.139 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:44:50.139 INFO:teuthology.orchestra.run.smithi084.stdout: "pool": 7, 2024-09-20T22:44:50.139 INFO:teuthology.orchestra.run.smithi084.stdout: "snaps": [ 2024-09-20T22:44:50.139 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:44:50.139 INFO:teuthology.orchestra.run.smithi084.stdout: "begin": 55, 2024-09-20T22:44:50.139 INFO:teuthology.orchestra.run.smithi084.stdout: "length": 1 2024-09-20T22:44:50.139 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:44:50.139 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:44:50.139 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:44:50.139 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2024-09-20T22:44:50.139 INFO:teuthology.orchestra.run.smithi084.stdout: "new_purged_snaps": [], 2024-09-20T22:44:50.139 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_node_flags": {}, 2024-09-20T22:44:50.140 INFO:teuthology.orchestra.run.smithi084.stdout: "device_class_flags": {}, 2024-09-20T22:44:50.140 INFO:teuthology.orchestra.run.smithi084.stdout: "stretch_mode": { 2024-09-20T22:44:50.140 INFO:teuthology.orchestra.run.smithi084.stdout: "stretch_mode_enabled": false, 2024-09-20T22:44:50.140 INFO:teuthology.orchestra.run.smithi084.stdout: "stretch_bucket_count": 0, 2024-09-20T22:44:50.140 INFO:teuthology.orchestra.run.smithi084.stdout: "degraded_stretch_mode": 0, 2024-09-20T22:44:50.140 INFO:teuthology.orchestra.run.smithi084.stdout: "recovering_stretch_mode": 0, 2024-09-20T22:44:50.140 INFO:teuthology.orchestra.run.smithi084.stdout: "stretch_mode_bucket": 0 2024-09-20T22:44:50.140 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:44:50.140 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-20T22:44:50.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:50.142 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:50.142 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:44:50.142 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-20T22:44:50.149 INFO:tasks.rados.rados.0.smithi084.stdout:625: done (6 left) 2024-09-20T22:44:50.149 INFO:tasks.rados.rados.0.smithi084.stdout:626: done (5 left) 2024-09-20T22:44:50.150 INFO:tasks.rados.rados.0.smithi084.stdout:627: done (4 left) 2024-09-20T22:44:50.150 INFO:tasks.rados.rados.0.smithi084.stdout:628: done (3 left) 2024-09-20T22:44:50.150 INFO:tasks.rados.rados.0.smithi084.stdout:630: done (2 left) 2024-09-20T22:44:50.150 INFO:tasks.rados.rados.0.smithi084.stdout:631: done (1 left) 2024-09-20T22:44:50.150 INFO:tasks.rados.rados.0.smithi084.stdout:632: done (0 left) 2024-09-20T22:44:50.150 INFO:tasks.rados.rados.0.smithi084.stdout:633: read oid 44 snap 56 2024-09-20T22:44:50.150 INFO:tasks.rados.rados.0.smithi084.stdout:633: expect (ObjNum 190 snap 40 seq_num 190) 2024-09-20T22:44:50.150 INFO:tasks.rados.rados.0.smithi084.stdout:634: snap_create 2024-09-20T22:44:50.163 INFO:tasks.rados.rados.0.smithi084.stdout:633: expect (ObjNum 190 snap 40 seq_num 190) 2024-09-20T22:44:50.242 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:44:50.264 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:50.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:50.265 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:44:50.364 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:50 smithi084 ceph-mon[29774]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:44:50.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:50 smithi084 ceph-mon[29774]: purged_snaps scrub starts 2024-09-20T22:44:50.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:50 smithi084 ceph-mon[29774]: purged_snaps scrub ok 2024-09-20T22:44:50.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:50 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/2544760385' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-20T22:44:50.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:50 smithi084 ceph-mon[29774]: osdmap e211: 8 total, 8 up, 7 in 2024-09-20T22:44:50.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:50 smithi084 ceph-mon[40024]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:44:50.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:50 smithi084 ceph-mon[40024]: purged_snaps scrub starts 2024-09-20T22:44:50.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:50 smithi084 ceph-mon[40024]: purged_snaps scrub ok 2024-09-20T22:44:50.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:50 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/2544760385' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-20T22:44:50.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:50 smithi084 ceph-mon[40024]: osdmap e211: 8 total, 8 up, 7 in 2024-09-20T22:44:50.366 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:44:50.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:50.386 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:50.387 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:44:50.428 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:44:50.465 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:44:50.488 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:44:50.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:50 smithi154 ceph-mon[33264]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:44:50.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:50 smithi154 ceph-mon[33264]: purged_snaps scrub starts 2024-09-20T22:44:50.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:50 smithi154 ceph-mon[33264]: purged_snaps scrub ok 2024-09-20T22:44:50.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:50 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/2544760385' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-20T22:44:50.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:50 smithi154 ceph-mon[33264]: osdmap e211: 8 total, 8 up, 7 in 2024-09-20T22:44:50.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:50.510 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:50.510 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:44:50.611 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:44:50.632 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:50.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:50.633 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:44:50.734 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:44:50.757 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:50.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:50.758 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:44:50.859 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:44:50.881 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:50.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:50.882 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:44:50.982 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:44:51.003 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:51.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:51.004 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:44:51.102 INFO:tasks.thrashosds.thrasher:j is {'epoch': 210, 'fsid': 'a9c0bc70-77a0-11ef-baf5-efdc52797490', 'created': '2024-09-20T22:36:30.240626+0000', 'modified': '2024-09-20T22:44:49.139612+0000', 'last_up_change': '2024-09-20T22:41:27.098565+0000', 'last_in_change': '2024-09-20T22:43:56.388784+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 18, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 7, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-20T22:39:21.984381+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 21, 'pg_placement_num': 21, 'pg_placement_num_target': 21, 'pg_num_target': 21, 'pg_num_pending': 21, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '147', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '119', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.899999976158142, 'score_stable': 1.899999976158142, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.6699999570846558, 'raw_score_stable': 1.6699999570846558, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-09-20T22:41:35.820081+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '69', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 69, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.6500000953674316, 'score_stable': 2.6500000953674316, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 2.3299999237060547, 'raw_score_stable': 2.3299999237060547, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-09-20T22:41:36.546389+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.7400000095367432, 'score_stable': 1.7400000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.5299999713897705, 'raw_score_stable': 1.5299999713897705, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-09-20T22:41:38.454640+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '70', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.240000009536743, 'score_stable': 2.240000009536743, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.9700000286102295, 'raw_score_stable': 1.9700000286102295, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-09-20T22:41:40.407129+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '72', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.7400000095367432, 'score_stable': 1.7400000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.5299999713897705, 'raw_score_stable': 1.5299999713897705, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-09-20T22:41:42.476716+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '74', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'unique_pool_0', 'create_time': '2024-09-20T22:42:51.354358+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 26, 'pg_placement_num': 26, 'pg_placement_num_target': 26, 'pg_num_target': 26, 'pg_num_pending': 26, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '210', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '137', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 58, 'snap_epoch': 210, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.140000104904175, 'score_stable': 2.140000104904175, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': 'ea36a489-6f91-4b2f-b784-b90506b9805d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 169, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6801', 'nonce': 268084626}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6802', 'nonce': 268084626}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6804', 'nonce': 268084626}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6803', 'nonce': 268084626}]}, 'public_addr': '172.21.15.84:6801/268084626', 'cluster_addr': '172.21.15.84:6802/268084626', 'heartbeat_back_addr': '172.21.15.84:6804/268084626', 'heartbeat_front_addr': '172.21.15.84:6803/268084626', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'c8649d2f-27ef-47f8-bc94-f1cd9e62bdb8', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 160, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6805', 'nonce': 418374136}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6806', 'nonce': 418374136}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6808', 'nonce': 418374136}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6807', 'nonce': 418374136}]}, 'public_addr': '172.21.15.84:6805/418374136', 'cluster_addr': '172.21.15.84:6806/418374136', 'heartbeat_back_addr': '172.21.15.84:6808/418374136', 'heartbeat_front_addr': '172.21.15.84:6807/418374136', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'c728eb48-a0f0-4de4-a651-7f3bec867280', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 160, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6809', 'nonce': 3977168611}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6810', 'nonce': 3977168611}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6812', 'nonce': 3977168611}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6811', 'nonce': 3977168611}]}, 'public_addr': '172.21.15.84:6809/3977168611', 'cluster_addr': '172.21.15.84:6810/3977168611', 'heartbeat_back_addr': '172.21.15.84:6812/3977168611', 'heartbeat_front_addr': '172.21.15.84:6811/3977168611', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '573f44eb-0853-4630-a7d8-9caaeb256f9b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 25, 'up_thru': 169, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6813', 'nonce': 2192239945}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6814', 'nonce': 2192239945}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6816', 'nonce': 2192239945}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6815', 'nonce': 2192239945}]}, 'public_addr': '172.21.15.84:6813/2192239945', 'cluster_addr': '172.21.15.84:6814/2192239945', 'heartbeat_back_addr': '172.21.15.84:6816/2192239945', 'heartbeat_front_addr': '172.21.15.84:6815/2192239945', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'a52c3b94-c5f3-490f-8ac8-09061b9b2abb', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 139, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6800', 'nonce': 1793136002}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6801', 'nonce': 1793136002}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6803', 'nonce': 1793136002}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6802', 'nonce': 1793136002}]}, 'public_addr': '172.21.15.154:6800/1793136002', 'cluster_addr': '172.21.15.154:6801/1793136002', 'heartbeat_back_addr': '172.21.15.154:6803/1793136002', 'heartbeat_front_addr': '172.21.15.154:6802/1793136002', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '4fa0dc8e-25cc-46cd-96ff-33f20b6faec3', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 40, 'up_thru': 169, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6804', 'nonce': 493899927}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6805', 'nonce': 493899927}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6807', 'nonce': 493899927}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6806', 'nonce': 493899927}]}, 'public_addr': '172.21.15.154:6804/493899927', 'cluster_addr': '172.21.15.154:6805/493899927', 'heartbeat_back_addr': '172.21.15.154:6807/493899927', 'heartbeat_front_addr': '172.21.15.154:6806/493899927', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '56788d74-810e-4827-a615-958c12f8a368', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 48, 'up_thru': 169, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6808', 'nonce': 2375091033}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6809', 'nonce': 2375091033}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6811', 'nonce': 2375091033}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6810', 'nonce': 2375091033}]}, 'public_addr': '172.21.15.154:6808/2375091033', 'cluster_addr': '172.21.15.154:6809/2375091033', 'heartbeat_back_addr': '172.21.15.154:6811/2375091033', 'heartbeat_front_addr': '172.21.15.154:6810/2375091033', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '83503727-afc1-4b95-8b96-ecb63f3338c2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 59, 'up_thru': 169, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6812', 'nonce': 3717704491}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6813', 'nonce': 3717704491}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6815', 'nonce': 3717704491}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6814', 'nonce': 3717704491}]}, 'public_addr': '172.21.15.154:6812/3717704491', 'cluster_addr': '172.21.15.154:6813/3717704491', 'heartbeat_back_addr': '172.21.15.154:6815/3717704491', 'heartbeat_front_addr': '172.21.15.154:6814/3717704491', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-20T22:43:38.373571+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-20T22:44:01.955517+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-20T22:43:46.670431+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-20T22:43:44.778504+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-09-20T22:43:34.154869+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-20T22:44:33.074031+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-20T22:44:30.460653+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-20T22:43:53.020113+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '4.6', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.e', 'mappings': [{'from': 3, 'to': 2}]}, {'pgid': '4.11', 'mappings': [{'from': 3, 'to': 5}]}, {'pgid': '4.17', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.1d', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '5.3', 'mappings': [{'from': 0, 'to': 3}]}, {'pgid': '7.2', 'mappings': [{'from': 3, 'to': 5}]}, {'pgid': '7.13', 'mappings': [{'from': 7, 'to': 1}]}, {'pgid': '7.15', 'mappings': [{'from': 2, 'to': 6}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.84:0/542020545': '2024-09-21T22:42:07.596045+0000', '172.21.15.84:0/25873236': '2024-09-21T22:36:54.917812+0000', '172.21.15.84:6800/820908470': '2024-09-21T22:36:54.917812+0000', '172.21.15.84:0/567614908': '2024-09-21T22:36:54.917812+0000', '172.21.15.84:0/2963622459': '2024-09-21T22:42:07.596045+0000', '172.21.15.84:0/1176849276': '2024-09-21T22:37:17.690104+0000', '172.21.15.84:6800/2932420050': '2024-09-21T22:37:17.690104+0000', '172.21.15.84:0/3879263635': '2024-09-21T22:37:17.690104+0000', '172.21.15.84:0/969472279': '2024-09-21T22:36:54.917812+0000', '172.21.15.84:0/1361143214': '2024-09-21T22:37:17.690104+0000', '172.21.15.84:0/3353126955': '2024-09-21T22:42:07.596045+0000', '172.21.15.84:6800/4024278029': '2024-09-21T22:42:07.596045+0000', '172.21.15.84:0/547902262': '2024-09-21T22:42:07.596045+0000', '172.21.15.84:0/1242929317': '2024-09-21T22:42:07.596045+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [{'pool': 7, 'snaps': [{'begin': 53, 'length': 4}]}], 'new_removed_snaps': [{'pool': 7, 'snaps': [{'begin': 55, 'length': 1}]}], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-20T22:44:51.102 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph pg dump --format=json 2024-09-20T22:44:51.104 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:44:51.150 INFO:tasks.rados.rados.0.smithi084.stdout:633: done (1 left) 2024-09-20T22:44:51.150 INFO:tasks.rados.rados.0.smithi084.stdout:634: done (0 left) 2024-09-20T22:44:51.150 INFO:tasks.rados.rados.0.smithi084.stdout:635: delete oid 2 current snap is 59 2024-09-20T22:44:51.157 INFO:tasks.rados.rados.0.smithi084.stdout:635: done (0 left) 2024-09-20T22:44:51.157 INFO:tasks.rados.rados.0.smithi084.stdout:636: write oid 37 current snap is 59 2024-09-20T22:44:51.157 INFO:tasks.rados.rados.0.smithi084.stdout:636: seq_num 253 ranges {652171=477646,1821709=430462} 2024-09-20T22:44:51.163 INFO:tasks.rados.rados.0.smithi084.stdout:636: writing smithi08477203-37 from 652171 to 1129817 tid 1 2024-09-20T22:44:51.170 INFO:tasks.rados.rados.0.smithi084.stdout:636: writing smithi08477203-37 from 1821709 to 2252171 tid 2 2024-09-20T22:44:51.170 INFO:tasks.rados.rados.0.smithi084.stdout:637: snap_create 2024-09-20T22:44:51.170 INFO:tasks.rados.rados.0.smithi084.stdout:636: finishing write tid 1 to smithi08477203-37 2024-09-20T22:44:51.172 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:51.173 INFO:tasks.rados.rados.0.smithi084.stdout:636: finishing write tid 2 to smithi08477203-37 2024-09-20T22:44:51.173 INFO:tasks.rados.rados.0.smithi084.stdout:636: finishing write tid 3 to smithi08477203-37 2024-09-20T22:44:51.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:51.174 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:44:51.174 INFO:tasks.rados.rados.0.smithi084.stdout:636: finishing write tid 4 to smithi08477203-37 2024-09-20T22:44:51.175 INFO:tasks.rados.rados.0.smithi084.stdout:636: oid 37 updating version 0 to 110 2024-09-20T22:44:51.175 INFO:tasks.rados.rados.0.smithi084.stdout:636: oid 37 version 110 is already newer than 108 2024-09-20T22:44:51.175 INFO:tasks.rados.rados.0.smithi084.stdout:636: oid 37 version 110 is already newer than 109 2024-09-20T22:44:51.175 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 37 v 110 (ObjNum 253 snap 59 seq_num 253) dirty exists 2024-09-20T22:44:51.175 INFO:tasks.rados.rados.0.smithi084.stdout:636: left oid 37 (ObjNum 253 snap 59 seq_num 253) 2024-09-20T22:44:51.275 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-20T22:44:51.275 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:44:51.298 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:51.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:51.300 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:44:51.364 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:51 smithi084 ceph-mon[29774]: pgmap v217: 178 pgs: 178 active+clean; 126 MiB data, 559 MiB used, 625 GiB / 626 GiB avail; 13 MiB/s rd, 2.7 MiB/s wr, 19 op/s 2024-09-20T22:44:51.364 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:51 smithi084 ceph-mon[29774]: osdmap e212: 8 total, 8 up, 7 in 2024-09-20T22:44:51.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:51 smithi084 ceph-mon[40024]: pgmap v217: 178 pgs: 178 active+clean; 126 MiB data, 559 MiB used, 625 GiB / 626 GiB avail; 13 MiB/s rd, 2.7 MiB/s wr, 19 op/s 2024-09-20T22:44:51.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:51 smithi084 ceph-mon[40024]: osdmap e212: 8 total, 8 up, 7 in 2024-09-20T22:44:51.401 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:44:51.421 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:51.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:51.422 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:44:51.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:51 smithi154 ceph-mon[33264]: pgmap v217: 178 pgs: 178 active+clean; 126 MiB data, 559 MiB used, 625 GiB / 626 GiB avail; 13 MiB/s rd, 2.7 MiB/s wr, 19 op/s 2024-09-20T22:44:51.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:51 smithi154 ceph-mon[33264]: osdmap e212: 8 total, 8 up, 7 in 2024-09-20T22:44:51.522 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:44:51.548 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:51.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:51.549 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:44:51.650 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:44:51.671 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:51.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:51.672 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:44:51.773 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:44:51.795 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:51.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:51.796 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:44:51.831 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-20T22:44:51.832 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:44:51.897 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:44:51.923 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:51.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:51.924 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:44:52.024 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:44:52.047 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:52.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:52.048 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:44:52.149 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:44:52.170 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:52.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:52.171 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:44:52.172 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is unset 2024-09-20T22:44:52.176 INFO:tasks.rados.rados.0.smithi084.stdout:636: done (1 left) 2024-09-20T22:44:52.176 INFO:tasks.rados.rados.0.smithi084.stdout:637: done (0 left) 2024-09-20T22:44:52.176 INFO:tasks.rados.rados.0.smithi084.stdout:638: delete oid 28 current snap is 60 2024-09-20T22:44:52.181 INFO:tasks.rados.rados.0.smithi084.stdout:638: done (0 left) 2024-09-20T22:44:52.181 INFO:tasks.rados.rados.0.smithi084.stdout:639: copy_from oid 44 from oid 48 current snap is 60 2024-09-20T22:44:52.181 INFO:tasks.rados.rados.0.smithi084.stdout:640: copy_from oid 11 from oid 3 current snap is 60 2024-09-20T22:44:52.181 INFO:tasks.rados.rados.0.smithi084.stdout:641: read oid 9 snap 59 2024-09-20T22:44:52.181 INFO:tasks.rados.rados.0.smithi084.stdout:641: expect deleted 2024-09-20T22:44:52.182 INFO:tasks.rados.rados.0.smithi084.stdout:642: delete oid 15 current snap is 60 2024-09-20T22:44:52.184 INFO:tasks.rados.rados.0.smithi084.stdout:642: done (3 left) 2024-09-20T22:44:52.184 INFO:tasks.rados.rados.0.smithi084.stdout:643: snap_remove snap 57 2024-09-20T22:44:52.272 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:44:52.297 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:52.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:52.298 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:44:52.399 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:44:52.423 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:52.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:52.424 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:44:52.426 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:52 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/3000368025' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:44:52.427 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:52 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/3000368025' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:44:52.452 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.0 dump_blocked_ops 2024-09-20T22:44:52.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:52 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/3000368025' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:44:52.525 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:44:52.556 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:52.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:52.557 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:44:52.657 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:44:52.692 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:52.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:52.693 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:44:52.724 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:44:52.794 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:44:52.820 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:52.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:52.821 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:44:52.922 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:44:52.943 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:52.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:52.944 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:44:53.044 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:44:53.066 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:53.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:53.067 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:44:53.089 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:44:53.142 INFO:tasks.rados.rados.0.smithi084.stdout:643: done (3 left) 2024-09-20T22:44:53.142 INFO:tasks.rados.rados.0.smithi084.stdout:644: delete oid 46 current snap is 60 2024-09-20T22:44:53.142 INFO:tasks.rados.rados.0.smithi084.stdout:639: finishing copy_from racing read to smithi08477203-44 2024-09-20T22:44:53.143 INFO:tasks.rados.rados.0.smithi084.stdout:640: finishing copy_from racing read to smithi08477203-11 2024-09-20T22:44:53.143 INFO:tasks.rados.rados.0.smithi084.stdout:640: finishing copy_from to smithi08477203-11 2024-09-20T22:44:53.143 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 11 v 101 (ObjNum 245 snap 54 seq_num 245) dirty exists 2024-09-20T22:44:53.143 INFO:tasks.rados.rados.0.smithi084.stdout:639: finishing copy_from to smithi08477203-44 2024-09-20T22:44:53.143 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 44 v 94 (ObjNum 129 snap 23 seq_num 129) dirty exists 2024-09-20T22:44:53.150 INFO:tasks.rados.rados.0.smithi084.stdout:639: done (3 left) 2024-09-20T22:44:53.151 INFO:tasks.rados.rados.0.smithi084.stdout:640: done (2 left) 2024-09-20T22:44:53.151 INFO:tasks.rados.rados.0.smithi084.stdout:641: done (1 left) 2024-09-20T22:44:53.151 INFO:tasks.rados.rados.0.smithi084.stdout:644: done (0 left) 2024-09-20T22:44:53.151 INFO:tasks.rados.rados.0.smithi084.stdout:645: write oid 48 current snap is 60 2024-09-20T22:44:53.151 INFO:tasks.rados.rados.0.smithi084.stdout:645: seq_num 257 ranges {719399=778642,1939284=380115} 2024-09-20T22:44:53.159 INFO:tasks.rados.rados.0.smithi084.stdout:645: writing smithi08477203-48 from 719399 to 1498041 tid 1 2024-09-20T22:44:53.163 INFO:tasks.rados.rados.0.smithi084.stdout:645: writing smithi08477203-48 from 1939284 to 2319399 tid 2 2024-09-20T22:44:53.163 INFO:tasks.rados.rados.0.smithi084.stdout:646: write oid 19 current snap is 60 2024-09-20T22:44:53.164 INFO:tasks.rados.rados.0.smithi084.stdout:645: finishing write tid 1 to smithi08477203-48 2024-09-20T22:44:53.164 INFO:tasks.rados.rados.0.smithi084.stdout:646: seq_num 258 ranges {736206=453891,1881413=454793} 2024-09-20T22:44:53.168 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:44:53.170 INFO:tasks.rados.rados.0.smithi084.stdout:646: writing smithi08477203-19 from 736206 to 1190097 tid 1 2024-09-20T22:44:53.177 INFO:tasks.rados.rados.0.smithi084.stdout:646: writing smithi08477203-19 from 1881413 to 2336206 tid 2 2024-09-20T22:44:53.177 INFO:tasks.rados.rados.0.smithi084.stdout:647: snap_remove snap 58 2024-09-20T22:44:53.193 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:53.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:53.194 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:44:53.295 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:44:53.319 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:53.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:53.320 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:44:53.345 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:53 smithi084 ceph-mon[40024]: pgmap v220: 178 pgs: 178 active+clean; 126 MiB data, 559 MiB used, 625 GiB / 626 GiB avail; 13 MiB/s rd, 2.7 MiB/s wr, 19 op/s 2024-09-20T22:44:53.345 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:53 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/3000368025' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T22:44:53.345 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:53 smithi084 ceph-mon[40024]: osdmap e213: 8 total, 8 up, 7 in 2024-09-20T22:44:53.345 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:53 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:44:53.345 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:53 smithi084 ceph-mon[40024]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:44:53.345 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:53 smithi084 ceph-mon[40024]: osdmap e214: 8 total, 8 up, 7 in 2024-09-20T22:44:53.345 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:53 smithi084 ceph-mon[29774]: pgmap v220: 178 pgs: 178 active+clean; 126 MiB data, 559 MiB used, 625 GiB / 626 GiB avail; 13 MiB/s rd, 2.7 MiB/s wr, 19 op/s 2024-09-20T22:44:53.345 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:53 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/3000368025' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T22:44:53.345 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:53 smithi084 ceph-mon[29774]: osdmap e213: 8 total, 8 up, 7 in 2024-09-20T22:44:53.346 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:53 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:44:53.346 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:53 smithi084 ceph-mon[29774]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:44:53.346 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:53 smithi084 ceph-mon[29774]: osdmap e214: 8 total, 8 up, 7 in 2024-09-20T22:44:53.420 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:44:53.443 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:53.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:53.444 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:44:53.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:53 smithi154 ceph-mon[33264]: pgmap v220: 178 pgs: 178 active+clean; 126 MiB data, 559 MiB used, 625 GiB / 626 GiB avail; 13 MiB/s rd, 2.7 MiB/s wr, 19 op/s 2024-09-20T22:44:53.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:53 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/3000368025' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T22:44:53.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:53 smithi154 ceph-mon[33264]: osdmap e213: 8 total, 8 up, 7 in 2024-09-20T22:44:53.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:53 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:44:53.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:53 smithi154 ceph-mon[33264]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:44:53.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:53 smithi154 ceph-mon[33264]: osdmap e214: 8 total, 8 up, 7 in 2024-09-20T22:44:53.546 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:44:53.568 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:53.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:53.569 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:44:53.670 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:44:53.692 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:53.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:53.693 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:44:53.794 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:44:53.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:53.816 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:53.816 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:44:53.917 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:44:53.937 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:53.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:53.938 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:44:53.970 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:44:53.971 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:44:53.980 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-20T22:44:53.983 INFO:teuthology.orchestra.run.smithi084.stderr:dumped all 2024-09-20T22:44:53.989 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:44:53.989 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:44:54.007 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:44:54.007 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:44:54.024 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-20T22:44:54.024 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T22:44:54.039 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:44:54.042 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:44:54.042 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:44:54.058 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:44:54.058 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:44:54.061 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:54.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:54.062 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:44:54.075 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:44:54.075 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:44:54.092 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:44:54.093 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:44:54.163 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:44:54.183 INFO:tasks.rados.rados.0.smithi084.stdout:647: done (2 left) 2024-09-20T22:44:54.183 INFO:tasks.rados.rados.0.smithi084.stdout:648: snap_create 2024-09-20T22:44:54.183 INFO:tasks.rados.rados.0.smithi084.stdout:645: finishing write tid 2 to smithi08477203-48 2024-09-20T22:44:54.183 INFO:tasks.rados.rados.0.smithi084.stdout:645: finishing write tid 3 to smithi08477203-48 2024-09-20T22:44:54.183 INFO:tasks.rados.rados.0.smithi084.stdout:645: finishing write tid 4 to smithi08477203-48 2024-09-20T22:44:54.184 INFO:tasks.rados.rados.0.smithi084.stdout:645: oid 48 updating version 0 to 114 2024-09-20T22:44:54.184 INFO:tasks.rados.rados.0.smithi084.stdout:645: oid 48 updating version 114 to 115 2024-09-20T22:44:54.184 INFO:tasks.rados.rados.0.smithi084.stdout:645: oid 48 updating version 115 to 116 2024-09-20T22:44:54.184 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 48 v 116 (ObjNum 257 snap 60 seq_num 257) dirty exists 2024-09-20T22:44:54.184 INFO:tasks.rados.rados.0.smithi084.stdout:645: left oid 48 (ObjNum 257 snap 60 seq_num 257) 2024-09-20T22:44:54.184 INFO:tasks.rados.rados.0.smithi084.stdout:646: finishing write tid 1 to smithi08477203-19 2024-09-20T22:44:54.184 INFO:tasks.rados.rados.0.smithi084.stdout:646: finishing write tid 2 to smithi08477203-19 2024-09-20T22:44:54.184 INFO:tasks.rados.rados.0.smithi084.stdout:646: finishing write tid 3 to smithi08477203-19 2024-09-20T22:44:54.184 INFO:tasks.rados.rados.0.smithi084.stdout:646: finishing write tid 4 to smithi08477203-19 2024-09-20T22:44:54.184 INFO:tasks.rados.rados.0.smithi084.stdout:646: oid 19 updating version 0 to 278 2024-09-20T22:44:54.184 INFO:tasks.rados.rados.0.smithi084.stdout:646: oid 19 updating version 278 to 280 2024-09-20T22:44:54.184 INFO:tasks.rados.rados.0.smithi084.stdout:646: oid 19 version 280 is already newer than 279 2024-09-20T22:44:54.184 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 19 v 280 (ObjNum 258 snap 60 seq_num 258) dirty exists 2024-09-20T22:44:54.184 INFO:tasks.rados.rados.0.smithi084.stdout:646: left oid 19 (ObjNum 258 snap 60 seq_num 258) 2024-09-20T22:44:54.189 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:54.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:54.190 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:44:54.291 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:44:54.314 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:54.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:54.315 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:44:54.364 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:54 smithi084 ceph-mon[29774]: 4.16 scrub starts 2024-09-20T22:44:54.364 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:54 smithi084 ceph-mon[29774]: 4.16 scrub ok 2024-09-20T22:44:54.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:54 smithi084 ceph-mon[29774]: 4.9 scrub starts 2024-09-20T22:44:54.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:54 smithi084 ceph-mon[29774]: 4.9 scrub ok 2024-09-20T22:44:54.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:54 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/3733520956' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:44:54.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:54 smithi084 ceph-mon[40024]: 4.16 scrub starts 2024-09-20T22:44:54.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:54 smithi084 ceph-mon[40024]: 4.16 scrub ok 2024-09-20T22:44:54.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:54 smithi084 ceph-mon[40024]: 4.9 scrub starts 2024-09-20T22:44:54.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:54 smithi084 ceph-mon[40024]: 4.9 scrub ok 2024-09-20T22:44:54.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:54 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/3733520956' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:44:54.416 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:44:54.438 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:54.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:54.439 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:44:54.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:54 smithi154 ceph-mon[33264]: 4.16 scrub starts 2024-09-20T22:44:54.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:54 smithi154 ceph-mon[33264]: 4.16 scrub ok 2024-09-20T22:44:54.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:54 smithi154 ceph-mon[33264]: 4.9 scrub starts 2024-09-20T22:44:54.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:54 smithi154 ceph-mon[33264]: 4.9 scrub ok 2024-09-20T22:44:54.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:54 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/3733520956' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:44:54.541 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:44:54.562 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:54.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:54.563 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:44:54.664 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:44:54.685 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:54.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:54.686 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:44:54.787 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:44:54.808 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:54.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:54.809 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:44:54.837 INFO:teuthology.orchestra.run.smithi084.stdout:{"pg_ready":true,"pg_map":{"version":223,"stamp":"2024-09-20T22:44:53.659450+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":118834660,"num_objects":247,"num_object_clones":10,"num_object_copies":741,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":247,"num_whiteouts":5,"num_read":1894,"num_read_kb":1021903,"num_write":1121,"num_write_kb":275457,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":5838014,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5521,"ondisk_log_size":5521,"up":534,"acting":534,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":534,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":547404,"kb_used_data":299940,"kb_used_omap":10,"kb_used_meta":247285,"kb_avail":655668660,"statfs":{"total":671965249536,"available":671404707840,"internally_reserved":0,"allocated":307138560,"data_stored":306253526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11157,"internal_metadata":253219947},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-13619936,"num_objects":-5,"num_object_clones":-3,"num_object_copies":-15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-5,"num_whiteouts":1,"num_read":29,"num_read_kb":20325,"num_write":18,"num_write_kb":7097,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000689"},"pg_stats":[{"pgid":"3.1f","version":"86'2","reported_seq":329,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.140822+0000","last_change":"2024-09-20T22:44:23.787416+0000","last_active":"2024-09-20T22:44:53.140822+0000","last_peered":"2024-09-20T22:44:53.140822+0000","last_clean":"2024-09-20T22:44:53.140822+0000","last_became_active":"2024-09-20T22:41:38.159085+0000","last_became_peered":"2024-09-20T22:41:38.159085+0000","last_unstale":"2024-09-20T22:44:53.140822+0000","last_undegraded":"2024-09-20T22:44:53.140822+0000","last_fullsized":"2024-09-20T22:44:53.140822+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'2","last_scrub_stamp":"2024-09-20T22:44:23.787342+0000","last_deep_scrub":"86'2","last_deep_scrub_stamp":"2024-09-20T22:42:39.664421+0000","last_clean_scrub_stamp":"2024-09-20T22:44:23.787342+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:45.057201+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"74'9","reported_seq":177,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.143631+0000","last_change":"2024-09-20T22:44:14.456654+0000","last_active":"2024-09-20T22:44:53.143631+0000","last_peered":"2024-09-20T22:44:53.143631+0000","last_clean":"2024-09-20T22:44:53.143631+0000","last_became_active":"2024-09-20T22:43:57.428786+0000","last_became_peered":"2024-09-20T22:43:57.428786+0000","last_unstale":"2024-09-20T22:44:53.143631+0000","last_undegraded":"2024-09-20T22:44:53.143631+0000","last_fullsized":"2024-09-20T22:44:53.143631+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-09-20T22:44:14.456573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:44:14.456573+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:30.414294+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":306,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.139912+0000","last_change":"2024-09-20T22:44:18.040704+0000","last_active":"2024-09-20T22:44:53.139912+0000","last_peered":"2024-09-20T22:44:53.139912+0000","last_clean":"2024-09-20T22:44:53.139912+0000","last_became_active":"2024-09-20T22:41:42.182024+0000","last_became_peered":"2024-09-20T22:41:42.182024+0000","last_unstale":"2024-09-20T22:44:53.139912+0000","last_undegraded":"2024-09-20T22:44:53.139912+0000","last_fullsized":"2024-09-20T22:44:53.139912+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:44:18.040631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:44:18.040631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:23.293547+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":133,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.616932+0000","last_change":"2024-09-20T22:44:40.746186+0000","last_active":"2024-09-20T22:44:53.616932+0000","last_peered":"2024-09-20T22:44:53.616932+0000","last_clean":"2024-09-20T22:44:53.616932+0000","last_became_active":"2024-09-20T22:43:57.429983+0000","last_became_peered":"2024-09-20T22:43:57.429983+0000","last_unstale":"2024-09-20T22:44:53.616932+0000","last_undegraded":"2024-09-20T22:44:53.616932+0000","last_fullsized":"2024-09-20T22:44:53.616932+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:44:40.746114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:44:40.746114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:40.841196+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":302,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.144026+0000","last_change":"2024-09-20T22:43:07.476040+0000","last_active":"2024-09-20T22:44:53.144026+0000","last_peered":"2024-09-20T22:44:53.144026+0000","last_clean":"2024-09-20T22:44:53.144026+0000","last_became_active":"2024-09-20T22:41:44.832099+0000","last_became_peered":"2024-09-20T22:41:44.832099+0000","last_unstale":"2024-09-20T22:44:53.144026+0000","last_undegraded":"2024-09-20T22:44:53.144026+0000","last_fullsized":"2024-09-20T22:44:53.144026+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:07.475964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:43:07.475964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:55.019803+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":306,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.144954+0000","last_change":"2024-09-20T22:41:38.157685+0000","last_active":"2024-09-20T22:44:53.144954+0000","last_peered":"2024-09-20T22:44:53.144954+0000","last_clean":"2024-09-20T22:44:53.144954+0000","last_became_active":"2024-09-20T22:41:38.157298+0000","last_became_peered":"2024-09-20T22:41:38.157298+0000","last_unstale":"2024-09-20T22:44:53.144954+0000","last_undegraded":"2024-09-20T22:44:53.144954+0000","last_fullsized":"2024-09-20T22:44:53.144954+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:55.019683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"74'15","reported_seq":335,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.145048+0000","last_change":"2024-09-20T22:44:41.108291+0000","last_active":"2024-09-20T22:44:53.145048+0000","last_peered":"2024-09-20T22:44:53.145048+0000","last_clean":"2024-09-20T22:44:53.145048+0000","last_became_active":"2024-09-20T22:41:40.163242+0000","last_became_peered":"2024-09-20T22:41:40.163242+0000","last_unstale":"2024-09-20T22:44:53.145048+0000","last_undegraded":"2024-09-20T22:44:53.145048+0000","last_fullsized":"2024-09-20T22:44:53.145048+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-09-20T22:44:41.108205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:44:41.108205+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:50.242514+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":124,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.143795+0000","last_change":"2024-09-20T22:43:57.424398+0000","last_active":"2024-09-20T22:44:53.143795+0000","last_peered":"2024-09-20T22:44:53.143795+0000","last_clean":"2024-09-20T22:44:53.143795+0000","last_became_active":"2024-09-20T22:43:57.423685+0000","last_became_peered":"2024-09-20T22:43:57.423685+0000","last_unstale":"2024-09-20T22:44:53.143795+0000","last_undegraded":"2024-09-20T22:44:53.143795+0000","last_fullsized":"2024-09-20T22:44:53.143795+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:02.971194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:43:02.971194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:54.611728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.19","version":"214'124","reported_seq":509,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.150453+0000","last_change":"2024-09-20T22:44:53.150929+0000","last_active":"2024-09-20T22:44:53.150453+0000","last_peered":"2024-09-20T22:44:53.150453+0000","last_clean":"2024-09-20T22:44:53.150453+0000","last_became_active":"2024-09-20T22:43:39.130950+0000","last_became_peered":"2024-09-20T22:43:39.130950+0000","last_unstale":"2024-09-20T22:44:53.150453+0000","last_undegraded":"2024-09-20T22:44:53.150453+0000","last_fullsized":"2024-09-20T22:44:53.150453+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":5,"last_scrub":"143'50","last_scrub_stamp":"2024-09-20T22:43:45.392735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:43:45.392735+0000","objects_scrubbed":3,"log_size":124,"log_dups_size":0,"ondisk_log_size":124,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:09.928145+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0072588380000000001,"stat_sum":{"num_bytes":5731220,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":2,"num_read":35,"num_read_kb":29153,"num_write":35,"num_write_kb":14433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"37","length":"1"},{"start":"3b","length":"1"}]},{"pgid":"3.1d","version":"0'0","reported_seq":320,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.616590+0000","last_change":"2024-09-20T22:43:57.430206+0000","last_active":"2024-09-20T22:44:53.616590+0000","last_peered":"2024-09-20T22:44:53.616590+0000","last_clean":"2024-09-20T22:44:53.616590+0000","last_became_active":"2024-09-20T22:43:57.429854+0000","last_became_peered":"2024-09-20T22:43:57.429854+0000","last_unstale":"2024-09-20T22:44:53.616590+0000","last_undegraded":"2024-09-20T22:44:53.616590+0000","last_fullsized":"2024-09-20T22:44:53.616590+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:34.681993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:43:34.681993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"74'9","reported_seq":169,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.143962+0000","last_change":"2024-09-20T22:43:57.488635+0000","last_active":"2024-09-20T22:44:53.143962+0000","last_peered":"2024-09-20T22:44:53.143962+0000","last_clean":"2024-09-20T22:44:53.143962+0000","last_became_active":"2024-09-20T22:43:57.445313+0000","last_became_peered":"2024-09-20T22:43:57.445313+0000","last_unstale":"2024-09-20T22:44:53.143962+0000","last_undegraded":"2024-09-20T22:44:53.143962+0000","last_fullsized":"2024-09-20T22:44:53.143962+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-09-20T22:43:30.938345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:43:30.938345+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:57.033474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":306,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.618056+0000","last_change":"2024-09-20T22:44:15.667340+0000","last_active":"2024-09-20T22:44:53.618056+0000","last_peered":"2024-09-20T22:44:53.618056+0000","last_clean":"2024-09-20T22:44:53.618056+0000","last_became_active":"2024-09-20T22:41:42.185538+0000","last_became_peered":"2024-09-20T22:41:42.185538+0000","last_unstale":"2024-09-20T22:44:53.618056+0000","last_undegraded":"2024-09-20T22:44:53.618056+0000","last_fullsized":"2024-09-20T22:44:53.618056+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:44:15.667266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:44:15.667266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:20.581167+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":302,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.138342+0000","last_change":"2024-09-20T22:44:20.811801+0000","last_active":"2024-09-20T22:44:53.138342+0000","last_peered":"2024-09-20T22:44:53.138342+0000","last_clean":"2024-09-20T22:44:53.138342+0000","last_became_active":"2024-09-20T22:41:44.197113+0000","last_became_peered":"2024-09-20T22:41:44.197113+0000","last_unstale":"2024-09-20T22:44:53.138342+0000","last_undegraded":"2024-09-20T22:44:53.138342+0000","last_fullsized":"2024-09-20T22:44:53.138342+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:44:20.811723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:44:20.811723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:38.989363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.18","version":"204'92","reported_seq":474,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.138307+0000","last_change":"2024-09-20T22:44:53.139541+0000","last_active":"2024-09-20T22:44:53.138307+0000","last_peered":"2024-09-20T22:44:53.138307+0000","last_clean":"2024-09-20T22:44:53.138307+0000","last_became_active":"2024-09-20T22:43:39.132159+0000","last_became_peered":"2024-09-20T22:43:39.132159+0000","last_unstale":"2024-09-20T22:44:53.138307+0000","last_undegraded":"2024-09-20T22:44:53.138307+0000","last_fullsized":"2024-09-20T22:44:53.138307+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":5,"last_scrub":"143'47","last_scrub_stamp":"2024-09-20T22:43:45.760674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:43:45.760674+0000","objects_scrubbed":2,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:54.344421+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001448066,"stat_sum":{"num_bytes":5361400,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":37,"num_read_kb":35570,"num_write":26,"num_write_kb":13129,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"37","length":"1"},{"start":"3b","length":"1"}]},{"pgid":"3.1c","version":"0'0","reported_seq":320,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.616548+0000","last_change":"2024-09-20T22:43:57.430381+0000","last_active":"2024-09-20T22:44:53.616548+0000","last_peered":"2024-09-20T22:44:53.616548+0000","last_clean":"2024-09-20T22:44:53.616548+0000","last_became_active":"2024-09-20T22:43:57.430028+0000","last_became_peered":"2024-09-20T22:43:57.430028+0000","last_unstale":"2024-09-20T22:44:53.616548+0000","last_undegraded":"2024-09-20T22:44:53.616548+0000","last_fullsized":"2024-09-20T22:44:53.616548+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:00.713878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:43:00.713878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:54.631605+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"181'7","reported_seq":169,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.144386+0000","last_change":"2024-09-20T22:44:20.259677+0000","last_active":"2024-09-20T22:44:53.144386+0000","last_peered":"2024-09-20T22:44:53.144386+0000","last_clean":"2024-09-20T22:44:53.144386+0000","last_became_active":"2024-09-20T22:43:57.441058+0000","last_became_peered":"2024-09-20T22:43:57.441058+0000","last_unstale":"2024-09-20T22:44:53.144386+0000","last_undegraded":"2024-09-20T22:44:53.144386+0000","last_fullsized":"2024-09-20T22:44:53.144386+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"181'7","last_scrub_stamp":"2024-09-20T22:44:20.259598+0000","last_deep_scrub":"181'7","last_deep_scrub_stamp":"2024-09-20T22:44:20.259598+0000","last_clean_scrub_stamp":"2024-09-20T22:44:20.259598+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:28.101261+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":6,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":315,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.144025+0000","last_change":"2024-09-20T22:43:57.419813+0000","last_active":"2024-09-20T22:44:53.144025+0000","last_peered":"2024-09-20T22:44:53.144025+0000","last_clean":"2024-09-20T22:44:53.144025+0000","last_became_active":"2024-09-20T22:43:57.419419+0000","last_became_peered":"2024-09-20T22:43:57.419419+0000","last_unstale":"2024-09-20T22:44:53.144025+0000","last_undegraded":"2024-09-20T22:44:53.144025+0000","last_fullsized":"2024-09-20T22:44:53.144025+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:01.248740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:43:01.248740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":302,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.617633+0000","last_change":"2024-09-20T22:43:32.712751+0000","last_active":"2024-09-20T22:44:53.617633+0000","last_peered":"2024-09-20T22:44:53.617633+0000","last_clean":"2024-09-20T22:44:53.617633+0000","last_became_active":"2024-09-20T22:41:44.197927+0000","last_became_peered":"2024-09-20T22:41:44.197927+0000","last_unstale":"2024-09-20T22:44:53.617633+0000","last_undegraded":"2024-09-20T22:44:53.617633+0000","last_fullsized":"2024-09-20T22:44:53.617633+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:32.712676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:43:32.712676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:02.620815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":133,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.145798+0000","last_change":"2024-09-20T22:44:41.574836+0000","last_active":"2024-09-20T22:44:53.145798+0000","last_peered":"2024-09-20T22:44:53.145798+0000","last_clean":"2024-09-20T22:44:53.145798+0000","last_became_active":"2024-09-20T22:43:57.427971+0000","last_became_peered":"2024-09-20T22:43:57.427971+0000","last_unstale":"2024-09-20T22:44:53.145798+0000","last_undegraded":"2024-09-20T22:44:53.145798+0000","last_fullsized":"2024-09-20T22:44:53.145798+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:44:41.574713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:44:41.574713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:46:10.688311+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":320,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.139948+0000","last_change":"2024-09-20T22:44:18.785085+0000","last_active":"2024-09-20T22:44:53.139948+0000","last_peered":"2024-09-20T22:44:53.139948+0000","last_clean":"2024-09-20T22:44:53.139948+0000","last_became_active":"2024-09-20T22:43:57.432650+0000","last_became_peered":"2024-09-20T22:43:57.432650+0000","last_unstale":"2024-09-20T22:44:53.139948+0000","last_undegraded":"2024-09-20T22:44:53.139948+0000","last_fullsized":"2024-09-20T22:44:53.139948+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:44:18.784990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:44:18.784990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:47.957020+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"113'22","reported_seq":341,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.151219+0000","last_change":"2024-09-20T22:43:12.191980+0000","last_active":"2024-09-20T22:44:53.151219+0000","last_peered":"2024-09-20T22:44:53.151219+0000","last_clean":"2024-09-20T22:44:53.151219+0000","last_became_active":"2024-09-20T22:41:40.170131+0000","last_became_peered":"2024-09-20T22:41:40.170131+0000","last_unstale":"2024-09-20T22:44:53.151219+0000","last_undegraded":"2024-09-20T22:44:53.151219+0000","last_fullsized":"2024-09-20T22:44:53.151219+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'22","last_scrub_stamp":"2024-09-20T22:43:12.191878+0000","last_deep_scrub":"113'22","last_deep_scrub_stamp":"2024-09-20T22:43:12.191878+0000","last_clean_scrub_stamp":"2024-09-20T22:43:12.191878+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:54.488276+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":312,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.141000+0000","last_change":"2024-09-20T22:44:22.085304+0000","last_active":"2024-09-20T22:44:53.141000+0000","last_peered":"2024-09-20T22:44:53.141000+0000","last_clean":"2024-09-20T22:44:53.141000+0000","last_became_active":"2024-09-20T22:43:57.435288+0000","last_became_peered":"2024-09-20T22:43:57.435288+0000","last_unstale":"2024-09-20T22:44:53.141000+0000","last_undegraded":"2024-09-20T22:44:53.141000+0000","last_fullsized":"2024-09-20T22:44:53.141000+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:44:22.085224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:44:22.085224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:28.981956+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":294,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.145183+0000","last_change":"2024-09-20T22:41:44.832468+0000","last_active":"2024-09-20T22:44:53.145183+0000","last_peered":"2024-09-20T22:44:53.145183+0000","last_clean":"2024-09-20T22:44:53.145183+0000","last_became_active":"2024-09-20T22:41:44.832131+0000","last_became_peered":"2024-09-20T22:41:44.832131+0000","last_unstale":"2024-09-20T22:44:53.145183+0000","last_undegraded":"2024-09-20T22:44:53.145183+0000","last_fullsized":"2024-09-20T22:44:53.145183+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:55.019757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":132,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.139261+0000","last_change":"2024-09-20T22:44:12.001213+0000","last_active":"2024-09-20T22:44:53.139261+0000","last_peered":"2024-09-20T22:44:53.139261+0000","last_clean":"2024-09-20T22:44:53.139261+0000","last_became_active":"2024-09-20T22:43:57.435636+0000","last_became_peered":"2024-09-20T22:43:57.435636+0000","last_unstale":"2024-09-20T22:44:53.139261+0000","last_undegraded":"2024-09-20T22:44:53.139261+0000","last_fullsized":"2024-09-20T22:44:53.139261+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:44:12.001145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:44:12.001145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:14.872769+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"74'12","reported_seq":369,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.144450+0000","last_change":"2024-09-20T22:44:42.141018+0000","last_active":"2024-09-20T22:44:53.144450+0000","last_peered":"2024-09-20T22:44:53.144450+0000","last_clean":"2024-09-20T22:44:53.144450+0000","last_became_active":"2024-09-20T22:43:57.438162+0000","last_became_peered":"2024-09-20T22:43:57.438162+0000","last_unstale":"2024-09-20T22:44:53.144450+0000","last_undegraded":"2024-09-20T22:44:53.144450+0000","last_fullsized":"2024-09-20T22:44:53.144450+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-09-20T22:44:42.140915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:44:42.140915+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:57.947345+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":132,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.144215+0000","last_change":"2024-09-20T22:44:14.235908+0000","last_active":"2024-09-20T22:44:53.144215+0000","last_peered":"2024-09-20T22:44:53.144215+0000","last_clean":"2024-09-20T22:44:53.144215+0000","last_became_active":"2024-09-20T22:43:57.429179+0000","last_became_peered":"2024-09-20T22:43:57.429179+0000","last_unstale":"2024-09-20T22:44:53.144215+0000","last_undegraded":"2024-09-20T22:44:53.144215+0000","last_fullsized":"2024-09-20T22:44:53.144215+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:44:14.235804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:44:14.235804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:26.950979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"154'2","reported_seq":304,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.144621+0000","last_change":"2024-09-20T22:43:54.134185+0000","last_active":"2024-09-20T22:44:53.144621+0000","last_peered":"2024-09-20T22:44:53.144621+0000","last_clean":"2024-09-20T22:44:53.144621+0000","last_became_active":"2024-09-20T22:41:44.199822+0000","last_became_peered":"2024-09-20T22:41:44.199822+0000","last_unstale":"2024-09-20T22:44:53.144621+0000","last_undegraded":"2024-09-20T22:44:53.144621+0000","last_fullsized":"2024-09-20T22:44:53.144621+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"154'2","last_scrub_stamp":"2024-09-20T22:43:54.134066+0000","last_deep_scrub":"154'2","last_deep_scrub_stamp":"2024-09-20T22:43:54.134066+0000","last_clean_scrub_stamp":"2024-09-20T22:43:54.134066+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:16.124782+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"67'1","reported_seq":325,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.139995+0000","last_change":"2024-09-20T22:43:57.462130+0000","last_active":"2024-09-20T22:44:53.139995+0000","last_peered":"2024-09-20T22:44:53.139995+0000","last_clean":"2024-09-20T22:44:53.139995+0000","last_became_active":"2024-09-20T22:43:57.443155+0000","last_became_peered":"2024-09-20T22:43:57.443155+0000","last_unstale":"2024-09-20T22:44:53.139995+0000","last_undegraded":"2024-09-20T22:44:53.139995+0000","last_fullsized":"2024-09-20T22:44:53.139995+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-09-20T22:43:35.266242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:43:35.266242+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:56.918430+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"151'15","reported_seq":330,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.141685+0000","last_change":"2024-09-20T22:43:50.715845+0000","last_active":"2024-09-20T22:44:53.141685+0000","last_peered":"2024-09-20T22:44:53.141685+0000","last_clean":"2024-09-20T22:44:53.141685+0000","last_became_active":"2024-09-20T22:41:40.252001+0000","last_became_peered":"2024-09-20T22:41:40.252001+0000","last_unstale":"2024-09-20T22:44:53.141685+0000","last_undegraded":"2024-09-20T22:44:53.141685+0000","last_fullsized":"2024-09-20T22:44:53.141685+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"151'15","last_scrub_stamp":"2024-09-20T22:43:50.715761+0000","last_deep_scrub":"151'15","last_deep_scrub_stamp":"2024-09-20T22:43:50.715761+0000","last_clean_scrub_stamp":"2024-09-20T22:43:50.715761+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:57.459180+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"159'9","reported_seq":370,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.148084+0000","last_change":"2024-09-20T22:44:15.499231+0000","last_active":"2024-09-20T22:44:53.148084+0000","last_peered":"2024-09-20T22:44:53.148084+0000","last_clean":"2024-09-20T22:44:53.148084+0000","last_became_active":"2024-09-20T22:43:57.424879+0000","last_became_peered":"2024-09-20T22:43:57.424879+0000","last_unstale":"2024-09-20T22:44:53.148084+0000","last_undegraded":"2024-09-20T22:44:53.148084+0000","last_fullsized":"2024-09-20T22:44:53.148084+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"159'9","last_scrub_stamp":"2024-09-20T22:44:15.499150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:44:15.499150+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:39.142015+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.c","version":"214'280","reported_seq":936,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.179909+0000","last_change":"2024-09-20T22:44:53.152473+0000","last_active":"2024-09-20T22:44:53.179909+0000","last_peered":"2024-09-20T22:44:53.179909+0000","last_clean":"2024-09-20T22:44:53.179909+0000","last_became_active":"2024-09-20T22:42:52.404027+0000","last_became_peered":"2024-09-20T22:42:52.404027+0000","last_unstale":"2024-09-20T22:44:53.179909+0000","last_undegraded":"2024-09-20T22:44:53.179909+0000","last_fullsized":"2024-09-20T22:44:53.179909+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"194'228","last_scrub_stamp":"2024-09-20T22:44:34.610972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:44:34.610972+0000","objects_scrubbed":5,"log_size":280,"log_dups_size":0,"ondisk_log_size":280,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:44.442807+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0038095350000000002,"stat_sum":{"num_bytes":20337675,"num_objects":9,"num_object_clones":3,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":1,"num_read":225,"num_read_kb":220937,"num_write":156,"num_write_kb":64410,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"37","length":"1"},{"start":"3b","length":"1"}]},{"pgid":"1.a","version":"77'128","reported_seq":550,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.617385+0000","last_change":"2024-09-20T22:43:05.763879+0000","last_active":"2024-09-20T22:44:53.617385+0000","last_peered":"2024-09-20T22:44:53.617385+0000","last_clean":"2024-09-20T22:44:53.617385+0000","last_became_active":"2024-09-20T22:42:52.407791+0000","last_became_peered":"2024-09-20T22:42:52.407791+0000","last_unstale":"2024-09-20T22:44:53.617385+0000","last_undegraded":"2024-09-20T22:44:53.617385+0000","last_fullsized":"2024-09-20T22:44:53.617385+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:43:05.763796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:43:05.763796+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:54.631686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"74'15","reported_seq":357,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.142215+0000","last_change":"2024-09-20T22:43:57.459254+0000","last_active":"2024-09-20T22:44:53.142215+0000","last_peered":"2024-09-20T22:44:53.142215+0000","last_clean":"2024-09-20T22:44:53.142215+0000","last_became_active":"2024-09-20T22:43:57.441355+0000","last_became_peered":"2024-09-20T22:43:57.441355+0000","last_unstale":"2024-09-20T22:44:53.142215+0000","last_undegraded":"2024-09-20T22:44:53.142215+0000","last_fullsized":"2024-09-20T22:44:53.142215+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-09-20T22:43:10.076699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:43:10.076699+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":306,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.144163+0000","last_change":"2024-09-20T22:41:38.157043+0000","last_active":"2024-09-20T22:44:53.144163+0000","last_peered":"2024-09-20T22:44:53.144163+0000","last_clean":"2024-09-20T22:44:53.144163+0000","last_became_active":"2024-09-20T22:41:38.156749+0000","last_became_peered":"2024-09-20T22:41:38.156749+0000","last_unstale":"2024-09-20T22:44:53.144163+0000","last_undegraded":"2024-09-20T22:44:53.144163+0000","last_fullsized":"2024-09-20T22:44:53.144163+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:55.019713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"159'9","reported_seq":165,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.616633+0000","last_change":"2024-09-20T22:43:57.438971+0000","last_active":"2024-09-20T22:44:53.616633+0000","last_peered":"2024-09-20T22:44:53.616633+0000","last_clean":"2024-09-20T22:44:53.616633+0000","last_became_active":"2024-09-20T22:43:57.425367+0000","last_became_peered":"2024-09-20T22:43:57.425367+0000","last_unstale":"2024-09-20T22:44:53.616633+0000","last_undegraded":"2024-09-20T22:44:53.616633+0000","last_fullsized":"2024-09-20T22:44:53.616633+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:54.631359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":302,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.618365+0000","last_change":"2024-09-20T22:43:53.790857+0000","last_active":"2024-09-20T22:44:53.618365+0000","last_peered":"2024-09-20T22:44:53.618365+0000","last_clean":"2024-09-20T22:44:53.618365+0000","last_became_active":"2024-09-20T22:41:44.198347+0000","last_became_peered":"2024-09-20T22:41:44.198347+0000","last_unstale":"2024-09-20T22:44:53.618365+0000","last_undegraded":"2024-09-20T22:44:53.618365+0000","last_fullsized":"2024-09-20T22:44:53.618365+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:53.790756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:43:53.790756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:14.003231+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3","version":"135'8","reported_seq":622,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.145177+0000","last_change":"2024-09-20T22:44:53.145946+0000","last_active":"2024-09-20T22:44:53.145177+0000","last_peered":"2024-09-20T22:44:53.145177+0000","last_clean":"2024-09-20T22:44:53.145177+0000","last_became_active":"2024-09-20T22:43:57.439135+0000","last_became_peered":"2024-09-20T22:43:57.439135+0000","last_unstale":"2024-09-20T22:44:53.145177+0000","last_undegraded":"2024-09-20T22:44:53.145177+0000","last_fullsized":"2024-09-20T22:44:53.145177+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'8","last_scrub_stamp":"2024-09-20T22:43:47.153362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:43:47.153362+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:59.656070+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0021087879999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"37","length":"1"},{"start":"3b","length":"1"}]},{"pgid":"1.5","version":"77'128","reported_seq":423,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.151798+0000","last_change":"2024-09-20T22:44:10.229558+0000","last_active":"2024-09-20T22:44:53.151798+0000","last_peered":"2024-09-20T22:44:53.151798+0000","last_clean":"2024-09-20T22:44:53.151798+0000","last_became_active":"2024-09-20T22:44:00.145455+0000","last_became_peered":"2024-09-20T22:44:00.145455+0000","last_unstale":"2024-09-20T22:44:53.151798+0000","last_undegraded":"2024-09-20T22:44:53.151798+0000","last_fullsized":"2024-09-20T22:44:53.151798+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:44:10.229483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:44:10.229483+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:20.129048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,5],"acting":[2,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"74'18","reported_seq":342,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.143888+0000","last_change":"2024-09-20T22:44:23.220792+0000","last_active":"2024-09-20T22:44:53.143888+0000","last_peered":"2024-09-20T22:44:53.143888+0000","last_clean":"2024-09-20T22:44:53.143888+0000","last_became_active":"2024-09-20T22:41:40.251868+0000","last_became_peered":"2024-09-20T22:41:40.251868+0000","last_unstale":"2024-09-20T22:44:53.143888+0000","last_undegraded":"2024-09-20T22:44:53.143888+0000","last_fullsized":"2024-09-20T22:44:53.143888+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'18","last_scrub_stamp":"2024-09-20T22:44:23.220709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:44:23.220709+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:43.393391+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":314,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.145459+0000","last_change":"2024-09-20T22:43:51.170223+0000","last_active":"2024-09-20T22:44:53.145459+0000","last_peered":"2024-09-20T22:44:53.145459+0000","last_clean":"2024-09-20T22:44:53.145459+0000","last_became_active":"2024-09-20T22:41:38.161335+0000","last_became_peered":"2024-09-20T22:41:38.161335+0000","last_unstale":"2024-09-20T22:44:53.145459+0000","last_undegraded":"2024-09-20T22:44:53.145459+0000","last_fullsized":"2024-09-20T22:44:53.145459+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:51.170155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:43:51.170155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:12.305573+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":124,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.144402+0000","last_change":"2024-09-20T22:43:57.427579+0000","last_active":"2024-09-20T22:44:53.144402+0000","last_peered":"2024-09-20T22:44:53.144402+0000","last_clean":"2024-09-20T22:44:53.144402+0000","last_became_active":"2024-09-20T22:43:57.427287+0000","last_became_peered":"2024-09-20T22:43:57.427287+0000","last_unstale":"2024-09-20T22:44:53.144402+0000","last_undegraded":"2024-09-20T22:44:53.144402+0000","last_fullsized":"2024-09-20T22:44:53.144402+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:07.947945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:07.947945+0000","last_clean_scrub_stamp":"2024-09-20T22:43:07.947945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:55.019874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":125,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.143860+0000","last_change":"2024-09-20T22:43:57.435367+0000","last_active":"2024-09-20T22:44:53.143860+0000","last_peered":"2024-09-20T22:44:53.143860+0000","last_clean":"2024-09-20T22:44:53.143860+0000","last_became_active":"2024-09-20T22:43:57.435057+0000","last_became_peered":"2024-09-20T22:43:57.435057+0000","last_unstale":"2024-09-20T22:44:53.143860+0000","last_undegraded":"2024-09-20T22:44:53.143860+0000","last_fullsized":"2024-09-20T22:44:53.143860+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:53.036601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:53.036601+0000","last_clean_scrub_stamp":"2024-09-20T22:43:53.036601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:53.878759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2","version":"214'89","reported_seq":265,"reported_epoch":214,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T22:44:53.617675+0000","last_change":"2024-09-20T22:44:53.617675+0000","last_active":"2024-09-20T22:44:53.617675+0000","last_peered":"2024-09-20T22:44:53.617675+0000","last_clean":"2024-09-20T22:44:53.617675+0000","last_became_active":"2024-09-20T22:44:09.177625+0000","last_became_peered":"2024-09-20T22:44:09.177625+0000","last_unstale":"2024-09-20T22:44:53.617675+0000","last_undegraded":"2024-09-20T22:44:53.617675+0000","last_fullsized":"2024-09-20T22:44:53.617675+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"138'78","last_scrub_stamp":"2024-09-20T22:44:09.667741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:44:09.667741+0000","objects_scrubbed":1,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:35.723043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017229120000000001,"stat_sum":{"num_bytes":3378207,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":21,"num_read_kb":31713,"num_write":6,"num_write_kb":1390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"37","length":"1"}]},{"pgid":"1.4","version":"77'128","reported_seq":564,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.144072+0000","last_change":"2024-09-20T22:43:26.151507+0000","last_active":"2024-09-20T22:44:53.144072+0000","last_peered":"2024-09-20T22:44:53.144072+0000","last_clean":"2024-09-20T22:44:53.144072+0000","last_became_active":"2024-09-20T22:43:19.125641+0000","last_became_peered":"2024-09-20T22:43:19.125641+0000","last_unstale":"2024-09-20T22:44:53.144072+0000","last_undegraded":"2024-09-20T22:44:53.144072+0000","last_fullsized":"2024-09-20T22:44:53.144072+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:43:26.151406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:43:26.151406+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:55.019861+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"74'14","reported_seq":188,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.616497+0000","last_change":"2024-09-20T22:44:17.668446+0000","last_active":"2024-09-20T22:44:53.616497+0000","last_peered":"2024-09-20T22:44:53.616497+0000","last_clean":"2024-09-20T22:44:53.616497+0000","last_became_active":"2024-09-20T22:43:57.422847+0000","last_became_peered":"2024-09-20T22:43:57.422847+0000","last_unstale":"2024-09-20T22:44:53.616497+0000","last_undegraded":"2024-09-20T22:44:53.616497+0000","last_fullsized":"2024-09-20T22:44:53.616497+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'14","last_scrub_stamp":"2024-09-20T22:44:17.668364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:44:17.668364+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:34.632610+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.6","version":"67'1","reported_seq":325,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.141966+0000","last_change":"2024-09-20T22:43:57.457010+0000","last_active":"2024-09-20T22:44:53.141966+0000","last_peered":"2024-09-20T22:44:53.141966+0000","last_clean":"2024-09-20T22:44:53.141966+0000","last_became_active":"2024-09-20T22:43:57.431232+0000","last_became_peered":"2024-09-20T22:43:57.431232+0000","last_unstale":"2024-09-20T22:44:53.141966+0000","last_undegraded":"2024-09-20T22:44:53.141966+0000","last_fullsized":"2024-09-20T22:44:53.141966+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-09-20T22:43:29.694945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:43:29.694945+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"159'9","reported_seq":356,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.143557+0000","last_change":"2024-09-20T22:43:57.481621+0000","last_active":"2024-09-20T22:44:53.143557+0000","last_peered":"2024-09-20T22:44:53.143557+0000","last_clean":"2024-09-20T22:44:53.143557+0000","last_became_active":"2024-09-20T22:43:57.442440+0000","last_became_peered":"2024-09-20T22:43:57.442440+0000","last_unstale":"2024-09-20T22:44:53.143557+0000","last_undegraded":"2024-09-20T22:44:53.143557+0000","last_fullsized":"2024-09-20T22:44:53.143557+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:55.019744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":302,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.143987+0000","last_change":"2024-09-20T22:44:19.972826+0000","last_active":"2024-09-20T22:44:53.143987+0000","last_peered":"2024-09-20T22:44:53.143987+0000","last_clean":"2024-09-20T22:44:53.143987+0000","last_became_active":"2024-09-20T22:41:44.831901+0000","last_became_peered":"2024-09-20T22:41:44.831901+0000","last_unstale":"2024-09-20T22:44:53.143987+0000","last_undegraded":"2024-09-20T22:44:53.143987+0000","last_fullsized":"2024-09-20T22:44:53.143987+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:44:19.972742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:44:19.972742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:48.587176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1","version":"200'99","reported_seq":685,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.146011+0000","last_change":"2024-09-20T22:44:53.146406+0000","last_active":"2024-09-20T22:44:53.146011+0000","last_peered":"2024-09-20T22:44:53.146011+0000","last_clean":"2024-09-20T22:44:53.146011+0000","last_became_active":"2024-09-20T22:43:57.440095+0000","last_became_peered":"2024-09-20T22:43:57.440095+0000","last_unstale":"2024-09-20T22:44:53.146011+0000","last_undegraded":"2024-09-20T22:44:53.146011+0000","last_fullsized":"2024-09-20T22:44:53.146011+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"170'72","last_scrub_stamp":"2024-09-20T22:44:12.228784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:44:12.228784+0000","objects_scrubbed":3,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:20.887755+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.001935169,"stat_sum":{"num_bytes":7579957,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":14253,"num_write":11,"num_write_kb":13746,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"37","length":"1"},{"start":"3b","length":"1"}]},{"pgid":"1.7","version":"77'128","reported_seq":570,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.147010+0000","last_change":"2024-09-20T22:43:26.566566+0000","last_active":"2024-09-20T22:44:53.147010+0000","last_peered":"2024-09-20T22:44:53.147010+0000","last_clean":"2024-09-20T22:44:53.147010+0000","last_became_active":"2024-09-20T22:43:19.129015+0000","last_became_peered":"2024-09-20T22:43:19.129015+0000","last_unstale":"2024-09-20T22:44:53.147010+0000","last_undegraded":"2024-09-20T22:44:53.147010+0000","last_fullsized":"2024-09-20T22:44:53.147010+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:43:26.566436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:43:26.566436+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:54.611858+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"74'10","reported_seq":343,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.138974+0000","last_change":"2024-09-20T22:43:57.455836+0000","last_active":"2024-09-20T22:44:53.138974+0000","last_peered":"2024-09-20T22:44:53.138974+0000","last_clean":"2024-09-20T22:44:53.138974+0000","last_became_active":"2024-09-20T22:43:57.435599+0000","last_became_peered":"2024-09-20T22:43:57.435599+0000","last_unstale":"2024-09-20T22:44:53.138974+0000","last_undegraded":"2024-09-20T22:44:53.138974+0000","last_fullsized":"2024-09-20T22:44:53.138974+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-09-20T22:43:13.094419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:43:13.094419+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":314,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.618678+0000","last_change":"2024-09-20T22:42:39.595884+0000","last_active":"2024-09-20T22:44:53.618678+0000","last_peered":"2024-09-20T22:44:53.618678+0000","last_clean":"2024-09-20T22:44:53.618678+0000","last_became_active":"2024-09-20T22:41:38.155629+0000","last_became_peered":"2024-09-20T22:41:38.155629+0000","last_unstale":"2024-09-20T22:44:53.618678+0000","last_undegraded":"2024-09-20T22:44:53.618678+0000","last_fullsized":"2024-09-20T22:44:53.618678+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:42:39.595799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:42:39.595799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:54.631471+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"170'9","reported_seq":113,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.143567+0000","last_change":"2024-09-20T22:44:09.175786+0000","last_active":"2024-09-20T22:44:53.143567+0000","last_peered":"2024-09-20T22:44:53.143567+0000","last_clean":"2024-09-20T22:44:53.143567+0000","last_became_active":"2024-09-20T22:44:09.171546+0000","last_became_peered":"2024-09-20T22:44:09.171546+0000","last_unstale":"2024-09-20T22:44:53.143567+0000","last_undegraded":"2024-09-20T22:44:53.143567+0000","last_fullsized":"2024-09-20T22:44:53.143567+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-09-20T22:43:08.643587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:43:08.643587+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:55.019846+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":302,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.138491+0000","last_change":"2024-09-20T22:43:33.657671+0000","last_active":"2024-09-20T22:44:53.138491+0000","last_peered":"2024-09-20T22:44:53.138491+0000","last_clean":"2024-09-20T22:44:53.138491+0000","last_became_active":"2024-09-20T22:41:44.197619+0000","last_became_peered":"2024-09-20T22:41:44.197619+0000","last_unstale":"2024-09-20T22:44:53.138491+0000","last_undegraded":"2024-09-20T22:44:53.138491+0000","last_fullsized":"2024-09-20T22:44:53.138491+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:33.657587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:43:33.657587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.0","version":"182'69","reported_seq":698,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.617866+0000","last_change":"2024-09-20T22:44:53.617866+0000","last_active":"2024-09-20T22:44:53.617866+0000","last_peered":"2024-09-20T22:44:53.617866+0000","last_clean":"2024-09-20T22:44:53.617866+0000","last_became_active":"2024-09-20T22:43:37.136845+0000","last_became_peered":"2024-09-20T22:43:37.136845+0000","last_unstale":"2024-09-20T22:44:53.617866+0000","last_undegraded":"2024-09-20T22:44:53.617866+0000","last_fullsized":"2024-09-20T22:44:53.617866+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"145'58","last_scrub_stamp":"2024-09-20T22:43:45.705214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:43:45.705214+0000","objects_scrubbed":3,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:57.923934+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00073501999999999997,"stat_sum":{"num_bytes":3092488,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":11411,"num_write":5,"num_write_kb":1217,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"37","length":"1"},{"start":"3b","length":"1"}]},{"pgid":"1.6","version":"77'128","reported_seq":570,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.145642+0000","last_change":"2024-09-20T22:43:27.532172+0000","last_active":"2024-09-20T22:44:53.145642+0000","last_peered":"2024-09-20T22:44:53.145642+0000","last_clean":"2024-09-20T22:44:53.145642+0000","last_became_active":"2024-09-20T22:43:19.130331+0000","last_became_peered":"2024-09-20T22:43:19.130331+0000","last_unstale":"2024-09-20T22:44:53.145642+0000","last_undegraded":"2024-09-20T22:44:53.145642+0000","last_fullsized":"2024-09-20T22:44:53.145642+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:43:27.532085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:43:27.532085+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:54.611777+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"74'19","reported_seq":354,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.138849+0000","last_change":"2024-09-20T22:44:37.632952+0000","last_active":"2024-09-20T22:44:53.138849+0000","last_peered":"2024-09-20T22:44:53.138849+0000","last_clean":"2024-09-20T22:44:53.138849+0000","last_became_active":"2024-09-20T22:41:40.251866+0000","last_became_peered":"2024-09-20T22:41:40.251866+0000","last_unstale":"2024-09-20T22:44:53.138849+0000","last_undegraded":"2024-09-20T22:44:53.138849+0000","last_fullsized":"2024-09-20T22:44:53.138849+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'19","last_scrub_stamp":"2024-09-20T22:44:37.632834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:44:37.632834+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:52.348513+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":314,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.140107+0000","last_change":"2024-09-20T22:43:32.231716+0000","last_active":"2024-09-20T22:44:53.140107+0000","last_peered":"2024-09-20T22:44:53.140107+0000","last_clean":"2024-09-20T22:44:53.140107+0000","last_became_active":"2024-09-20T22:41:38.160832+0000","last_became_peered":"2024-09-20T22:41:38.160832+0000","last_unstale":"2024-09-20T22:44:53.140107+0000","last_undegraded":"2024-09-20T22:44:53.140107+0000","last_fullsized":"2024-09-20T22:44:53.140107+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:32.231644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:43:32.231644+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":306,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.147685+0000","last_change":"2024-09-20T22:44:19.441854+0000","last_active":"2024-09-20T22:44:53.147685+0000","last_peered":"2024-09-20T22:44:53.147685+0000","last_clean":"2024-09-20T22:44:53.147685+0000","last_became_active":"2024-09-20T22:41:42.184412+0000","last_became_peered":"2024-09-20T22:41:42.184412+0000","last_unstale":"2024-09-20T22:44:53.147685+0000","last_undegraded":"2024-09-20T22:44:53.147685+0000","last_fullsized":"2024-09-20T22:44:53.147685+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:44:19.441770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:44:19.441770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:34.420562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":302,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.139822+0000","last_change":"2024-09-20T22:43:53.247501+0000","last_active":"2024-09-20T22:44:53.139822+0000","last_peered":"2024-09-20T22:44:53.139822+0000","last_clean":"2024-09-20T22:44:53.139822+0000","last_became_active":"2024-09-20T22:41:44.830538+0000","last_became_peered":"2024-09-20T22:41:44.830538+0000","last_unstale":"2024-09-20T22:44:53.139822+0000","last_undegraded":"2024-09-20T22:44:53.139822+0000","last_fullsized":"2024-09-20T22:44:53.139822+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:53.247430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:43:53.247430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:03.338549+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.7","version":"198'178","reported_seq":797,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.149313+0000","last_change":"2024-09-20T22:44:53.149993+0000","last_active":"2024-09-20T22:44:53.149313+0000","last_peered":"2024-09-20T22:44:53.149313+0000","last_clean":"2024-09-20T22:44:53.149313+0000","last_became_active":"2024-09-20T22:43:37.132943+0000","last_became_peered":"2024-09-20T22:43:37.132943+0000","last_unstale":"2024-09-20T22:44:53.149313+0000","last_undegraded":"2024-09-20T22:44:53.149313+0000","last_fullsized":"2024-09-20T22:44:53.149313+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"147'119","last_scrub_stamp":"2024-09-20T22:43:47.407291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:43:47.407291+0000","objects_scrubbed":2,"log_size":178,"log_dups_size":0,"ondisk_log_size":178,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0025087019999999998,"stat_sum":{"num_bytes":3495856,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":48,"num_read_kb":52557,"num_write":35,"num_write_kb":12141,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"37","length":"1"},{"start":"3b","length":"1"}]},{"pgid":"1.1","version":"77'128","reported_seq":564,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.617536+0000","last_change":"2024-09-20T22:43:26.761137+0000","last_active":"2024-09-20T22:44:53.617536+0000","last_peered":"2024-09-20T22:44:53.617536+0000","last_clean":"2024-09-20T22:44:53.617536+0000","last_became_active":"2024-09-20T22:43:19.125806+0000","last_became_peered":"2024-09-20T22:43:19.125806+0000","last_unstale":"2024-09-20T22:44:53.617536+0000","last_undegraded":"2024-09-20T22:44:53.617536+0000","last_fullsized":"2024-09-20T22:44:53.617536+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:43:26.761032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:43:26.761032+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:54.631633+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"86'38","reported_seq":416,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.140010+0000","last_change":"2024-09-20T22:44:41.077590+0000","last_active":"2024-09-20T22:44:53.140010+0000","last_peered":"2024-09-20T22:44:53.140010+0000","last_clean":"2024-09-20T22:44:53.140010+0000","last_became_active":"2024-09-20T22:41:40.166958+0000","last_became_peered":"2024-09-20T22:41:40.166958+0000","last_unstale":"2024-09-20T22:44:53.140010+0000","last_undegraded":"2024-09-20T22:44:53.140010+0000","last_fullsized":"2024-09-20T22:44:53.140010+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'38","last_scrub_stamp":"2024-09-20T22:44:41.077475+0000","last_deep_scrub":"86'38","last_deep_scrub_stamp":"2024-09-20T22:42:43.935677+0000","last_clean_scrub_stamp":"2024-09-20T22:44:41.077475+0000","objects_scrubbed":10,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:46:05.497572+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":132,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.140027+0000","last_change":"2024-09-20T22:44:38.652538+0000","last_active":"2024-09-20T22:44:53.140027+0000","last_peered":"2024-09-20T22:44:53.140027+0000","last_clean":"2024-09-20T22:44:53.140027+0000","last_became_active":"2024-09-20T22:43:57.432267+0000","last_became_peered":"2024-09-20T22:43:57.432267+0000","last_unstale":"2024-09-20T22:44:53.140027+0000","last_undegraded":"2024-09-20T22:44:53.140027+0000","last_fullsized":"2024-09-20T22:44:53.140027+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:44:38.652446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:44:38.652446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:53.468832+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"69'2","reported_seq":332,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.617792+0000","last_change":"2024-09-20T22:44:35.668676+0000","last_active":"2024-09-20T22:44:53.617792+0000","last_peered":"2024-09-20T22:44:53.617792+0000","last_clean":"2024-09-20T22:44:53.617792+0000","last_became_active":"2024-09-20T22:41:37.148479+0000","last_became_peered":"2024-09-20T22:41:37.148479+0000","last_unstale":"2024-09-20T22:44:53.617792+0000","last_undegraded":"2024-09-20T22:44:53.617792+0000","last_fullsized":"2024-09-20T22:44:53.617792+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2024-09-20T22:44:35.668576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:36.130684+0000","last_clean_scrub_stamp":"2024-09-20T22:44:35.668576+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:39.072433+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00087280599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":312,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.140244+0000","last_change":"2024-09-20T22:44:11.765375+0000","last_active":"2024-09-20T22:44:53.140244+0000","last_peered":"2024-09-20T22:44:53.140244+0000","last_clean":"2024-09-20T22:44:53.140244+0000","last_became_active":"2024-09-20T22:43:57.431815+0000","last_became_peered":"2024-09-20T22:43:57.431815+0000","last_unstale":"2024-09-20T22:44:53.140244+0000","last_undegraded":"2024-09-20T22:44:53.140244+0000","last_fullsized":"2024-09-20T22:44:53.140244+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:44:11.765248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:44:11.765248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:34.701647+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":308,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.144783+0000","last_change":"2024-09-20T22:43:57.434389+0000","last_active":"2024-09-20T22:44:53.144783+0000","last_peered":"2024-09-20T22:44:53.144783+0000","last_clean":"2024-09-20T22:44:53.144783+0000","last_became_active":"2024-09-20T22:43:57.434044+0000","last_became_peered":"2024-09-20T22:43:57.434044+0000","last_unstale":"2024-09-20T22:44:53.144783+0000","last_undegraded":"2024-09-20T22:44:53.144783+0000","last_fullsized":"2024-09-20T22:44:53.144783+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:09.406141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:43:09.406141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:55.019831+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"210'50","reported_seq":675,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.141185+0000","last_change":"2024-09-20T22:44:53.141626+0000","last_active":"2024-09-20T22:44:53.141185+0000","last_peered":"2024-09-20T22:44:53.141185+0000","last_clean":"2024-09-20T22:44:53.141185+0000","last_became_active":"2024-09-20T22:43:57.429486+0000","last_became_peered":"2024-09-20T22:43:57.429486+0000","last_unstale":"2024-09-20T22:44:53.141185+0000","last_undegraded":"2024-09-20T22:44:53.141185+0000","last_fullsized":"2024-09-20T22:44:53.141185+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"145'39","last_scrub_stamp":"2024-09-20T22:43:47.802333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:43:47.802333+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:09.464700+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0019801369999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":20,"num_read_kb":28189,"num_write":6,"num_write_kb":1350,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"37","length":"1"},{"start":"3b","length":"1"}]},{"pgid":"1.2","version":"77'128","reported_seq":614,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.143950+0000","last_change":"2024-09-20T22:44:00.147275+0000","last_active":"2024-09-20T22:44:53.143950+0000","last_peered":"2024-09-20T22:44:53.143950+0000","last_clean":"2024-09-20T22:44:53.143950+0000","last_became_active":"2024-09-20T22:44:00.146907+0000","last_became_peered":"2024-09-20T22:44:00.146907+0000","last_unstale":"2024-09-20T22:44:53.143950+0000","last_undegraded":"2024-09-20T22:44:53.143950+0000","last_fullsized":"2024-09-20T22:44:53.143950+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:43:25.207385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:43:25.207385+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"74'13","reported_seq":337,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.142288+0000","last_change":"2024-09-20T22:43:31.240331+0000","last_active":"2024-09-20T22:44:53.142288+0000","last_peered":"2024-09-20T22:44:53.142288+0000","last_clean":"2024-09-20T22:44:53.142288+0000","last_became_active":"2024-09-20T22:41:40.169473+0000","last_became_peered":"2024-09-20T22:41:40.169473+0000","last_unstale":"2024-09-20T22:44:53.142288+0000","last_undegraded":"2024-09-20T22:44:53.142288+0000","last_fullsized":"2024-09-20T22:44:53.142288+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'13","last_scrub_stamp":"2024-09-20T22:43:31.240196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:43:31.240196+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:57.654965+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":322,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.140604+0000","last_change":"2024-09-20T22:44:35.131693+0000","last_active":"2024-09-20T22:44:53.140604+0000","last_peered":"2024-09-20T22:44:53.140604+0000","last_clean":"2024-09-20T22:44:53.140604+0000","last_became_active":"2024-09-20T22:41:38.156837+0000","last_became_peered":"2024-09-20T22:41:38.156837+0000","last_unstale":"2024-09-20T22:44:53.140604+0000","last_undegraded":"2024-09-20T22:44:53.140604+0000","last_fullsized":"2024-09-20T22:44:53.140604+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:44:35.131583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:44:35.131583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:47.311461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"67'1","reported_seq":330,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.150520+0000","last_change":"2024-09-20T22:44:39.441694+0000","last_active":"2024-09-20T22:44:53.150520+0000","last_peered":"2024-09-20T22:44:53.150520+0000","last_clean":"2024-09-20T22:44:53.150520+0000","last_became_active":"2024-09-20T22:41:37.148165+0000","last_became_peered":"2024-09-20T22:41:37.148165+0000","last_unstale":"2024-09-20T22:44:53.150520+0000","last_undegraded":"2024-09-20T22:44:53.150520+0000","last_fullsized":"2024-09-20T22:44:53.150520+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-09-20T22:44:39.441585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:36.130684+0000","last_clean_scrub_stamp":"2024-09-20T22:44:39.441585+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:40.621607+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001301459,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":305,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.151687+0000","last_change":"2024-09-20T22:44:14.284410+0000","last_active":"2024-09-20T22:44:53.151687+0000","last_peered":"2024-09-20T22:44:53.151687+0000","last_clean":"2024-09-20T22:44:53.151687+0000","last_became_active":"2024-09-20T22:41:42.185112+0000","last_became_peered":"2024-09-20T22:41:42.185112+0000","last_unstale":"2024-09-20T22:44:53.151687+0000","last_undegraded":"2024-09-20T22:44:53.151687+0000","last_fullsized":"2024-09-20T22:44:53.151687+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:44:14.284296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:44:14.284296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:23.561043+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":302,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.146320+0000","last_change":"2024-09-20T22:44:14.946132+0000","last_active":"2024-09-20T22:44:53.146320+0000","last_peered":"2024-09-20T22:44:53.146320+0000","last_clean":"2024-09-20T22:44:53.146320+0000","last_became_active":"2024-09-20T22:41:44.830515+0000","last_became_peered":"2024-09-20T22:41:44.830515+0000","last_unstale":"2024-09-20T22:44:53.146320+0000","last_undegraded":"2024-09-20T22:44:53.146320+0000","last_fullsized":"2024-09-20T22:44:53.146320+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:44:14.946052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:44:14.946052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:17.759544+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"200'30","reported_seq":641,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.146491+0000","last_change":"2024-09-20T22:44:53.147191+0000","last_active":"2024-09-20T22:44:53.146491+0000","last_peered":"2024-09-20T22:44:53.146491+0000","last_clean":"2024-09-20T22:44:53.146491+0000","last_became_active":"2024-09-20T22:43:37.133478+0000","last_became_peered":"2024-09-20T22:43:37.133478+0000","last_unstale":"2024-09-20T22:44:53.146491+0000","last_undegraded":"2024-09-20T22:44:53.146491+0000","last_fullsized":"2024-09-20T22:44:53.146491+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'3","last_scrub_stamp":"2024-09-20T22:43:49.197854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:43:49.197854+0000","objects_scrubbed":1,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:07.045813+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0022815869999999999,"stat_sum":{"num_bytes":3966452,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":16,"num_read_kb":12235,"num_write":19,"num_write_kb":11130,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"37","length":"1"},{"start":"3b","length":"1"}]},{"pgid":"1.3","version":"77'128","reported_seq":573,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.142794+0000","last_change":"2024-09-20T22:44:00.144446+0000","last_active":"2024-09-20T22:44:53.142794+0000","last_peered":"2024-09-20T22:44:53.142794+0000","last_clean":"2024-09-20T22:44:53.142794+0000","last_became_active":"2024-09-20T22:44:00.144099+0000","last_became_peered":"2024-09-20T22:44:00.144099+0000","last_unstale":"2024-09-20T22:44:53.142794+0000","last_undegraded":"2024-09-20T22:44:53.142794+0000","last_fullsized":"2024-09-20T22:44:53.142794+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:43:25.244624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:43:25.244624+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:54.143244+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"74'12","reported_seq":348,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.141780+0000","last_change":"2024-09-20T22:43:26.720587+0000","last_active":"2024-09-20T22:44:53.141780+0000","last_peered":"2024-09-20T22:44:53.141780+0000","last_clean":"2024-09-20T22:44:53.141780+0000","last_became_active":"2024-09-20T22:43:09.120425+0000","last_became_peered":"2024-09-20T22:43:09.120425+0000","last_unstale":"2024-09-20T22:44:53.141780+0000","last_undegraded":"2024-09-20T22:44:53.141780+0000","last_fullsized":"2024-09-20T22:44:53.141780+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-09-20T22:43:26.720499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:43:26.720499+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":320,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.138766+0000","last_change":"2024-09-20T22:44:19.834724+0000","last_active":"2024-09-20T22:44:53.138766+0000","last_peered":"2024-09-20T22:44:53.138766+0000","last_clean":"2024-09-20T22:44:53.138766+0000","last_became_active":"2024-09-20T22:43:57.421167+0000","last_became_peered":"2024-09-20T22:43:57.421167+0000","last_unstale":"2024-09-20T22:44:53.138766+0000","last_undegraded":"2024-09-20T22:44:53.138766+0000","last_fullsized":"2024-09-20T22:44:53.138766+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:44:19.834641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:44:19.834641+0000","last_clean_scrub_stamp":"2024-09-20T22:44:19.834641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:43.561314+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"75'5","reported_seq":515,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.145466+0000","last_change":"2024-09-20T22:44:13.885050+0000","last_active":"2024-09-20T22:44:53.145466+0000","last_peered":"2024-09-20T22:44:53.145466+0000","last_clean":"2024-09-20T22:44:53.145466+0000","last_became_active":"2024-09-20T22:41:37.148383+0000","last_became_peered":"2024-09-20T22:41:37.148383+0000","last_unstale":"2024-09-20T22:44:53.145466+0000","last_undegraded":"2024-09-20T22:44:53.145466+0000","last_fullsized":"2024-09-20T22:44:53.145466+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-09-20T22:44:13.884968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:36.130684+0000","last_clean_scrub_stamp":"2024-09-20T22:44:13.884968+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:22.123942+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00078631700000000003,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":190,"num_read_kb":185,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":306,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.618593+0000","last_change":"2024-09-20T22:44:38.788128+0000","last_active":"2024-09-20T22:44:53.618593+0000","last_peered":"2024-09-20T22:44:53.618593+0000","last_clean":"2024-09-20T22:44:53.618593+0000","last_became_active":"2024-09-20T22:41:42.187346+0000","last_became_peered":"2024-09-20T22:41:42.187346+0000","last_unstale":"2024-09-20T22:44:53.618593+0000","last_undegraded":"2024-09-20T22:44:53.618593+0000","last_fullsized":"2024-09-20T22:44:53.618593+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:44:38.788045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:44:38.788045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:46:03.203383+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":302,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.139588+0000","last_change":"2024-09-20T22:43:00.128772+0000","last_active":"2024-09-20T22:44:53.139588+0000","last_peered":"2024-09-20T22:44:53.139588+0000","last_clean":"2024-09-20T22:44:53.139588+0000","last_became_active":"2024-09-20T22:41:44.199046+0000","last_became_peered":"2024-09-20T22:41:44.199046+0000","last_unstale":"2024-09-20T22:44:53.139588+0000","last_undegraded":"2024-09-20T22:44:53.139588+0000","last_fullsized":"2024-09-20T22:44:53.139588+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:00.128705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:43:00.128705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.6","version":"150'60","reported_seq":678,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.142780+0000","last_change":"2024-09-20T22:44:53.143227+0000","last_active":"2024-09-20T22:44:53.142780+0000","last_peered":"2024-09-20T22:44:53.142780+0000","last_clean":"2024-09-20T22:44:53.142780+0000","last_became_active":"2024-09-20T22:43:37.136591+0000","last_became_peered":"2024-09-20T22:43:37.136591+0000","last_unstale":"2024-09-20T22:44:53.142780+0000","last_undegraded":"2024-09-20T22:44:53.142780+0000","last_fullsized":"2024-09-20T22:44:53.142780+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'60","last_scrub_stamp":"2024-09-20T22:43:49.735664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:43:49.735664+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:02.965738+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0017271319999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"37","length":"1"},{"start":"3b","length":"1"}]},{"pgid":"4.5","version":"159'17","reported_seq":396,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.146335+0000","last_change":"2024-09-20T22:43:57.462065+0000","last_active":"2024-09-20T22:44:53.146335+0000","last_peered":"2024-09-20T22:44:53.146335+0000","last_clean":"2024-09-20T22:44:53.146335+0000","last_became_active":"2024-09-20T22:43:57.433076+0000","last_became_peered":"2024-09-20T22:43:57.433076+0000","last_unstale":"2024-09-20T22:44:53.146335+0000","last_undegraded":"2024-09-20T22:44:53.146335+0000","last_fullsized":"2024-09-20T22:44:53.146335+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'16","last_scrub_stamp":"2024-09-20T22:43:56.348264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:43:56.348264+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:12.752488+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"67'1","reported_seq":312,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.144101+0000","last_change":"2024-09-20T22:41:38.159706+0000","last_active":"2024-09-20T22:44:53.144101+0000","last_peered":"2024-09-20T22:44:53.144101+0000","last_clean":"2024-09-20T22:44:53.144101+0000","last_became_active":"2024-09-20T22:41:38.158917+0000","last_became_peered":"2024-09-20T22:41:38.158917+0000","last_unstale":"2024-09-20T22:44:53.144101+0000","last_undegraded":"2024-09-20T22:44:53.144101+0000","last_fullsized":"2024-09-20T22:44:53.144101+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:55.019728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"77'128","reported_seq":605,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.145589+0000","last_change":"2024-09-20T22:44:35.887245+0000","last_active":"2024-09-20T22:44:53.145589+0000","last_peered":"2024-09-20T22:44:53.145589+0000","last_clean":"2024-09-20T22:44:53.145589+0000","last_became_active":"2024-09-20T22:43:19.126925+0000","last_became_peered":"2024-09-20T22:43:19.126925+0000","last_unstale":"2024-09-20T22:44:53.145589+0000","last_undegraded":"2024-09-20T22:44:53.145589+0000","last_fullsized":"2024-09-20T22:44:53.145589+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:44:35.887120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:44:35.887120+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:55.559505+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":306,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.144795+0000","last_change":"2024-09-20T22:43:52.163679+0000","last_active":"2024-09-20T22:44:53.144795+0000","last_peered":"2024-09-20T22:44:53.144795+0000","last_clean":"2024-09-20T22:44:53.144795+0000","last_became_active":"2024-09-20T22:41:42.184387+0000","last_became_peered":"2024-09-20T22:41:42.184387+0000","last_unstale":"2024-09-20T22:44:53.144795+0000","last_undegraded":"2024-09-20T22:44:53.144795+0000","last_fullsized":"2024-09-20T22:44:53.144795+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:52.163576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:43:52.163576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:57.444451+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":308,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.616848+0000","last_change":"2024-09-20T22:43:57.434803+0000","last_active":"2024-09-20T22:44:53.616848+0000","last_peered":"2024-09-20T22:44:53.616848+0000","last_clean":"2024-09-20T22:44:53.616848+0000","last_became_active":"2024-09-20T22:43:57.434435+0000","last_became_peered":"2024-09-20T22:43:57.434435+0000","last_unstale":"2024-09-20T22:44:53.616848+0000","last_undegraded":"2024-09-20T22:44:53.616848+0000","last_fullsized":"2024-09-20T22:44:53.616848+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:02.696105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:43:02.696105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:54.631580+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"77'128","reported_seq":571,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.142648+0000","last_change":"2024-09-20T22:43:29.253689+0000","last_active":"2024-09-20T22:44:53.142648+0000","last_peered":"2024-09-20T22:44:53.142648+0000","last_clean":"2024-09-20T22:44:53.142648+0000","last_became_active":"2024-09-20T22:43:24.124252+0000","last_became_peered":"2024-09-20T22:43:24.124252+0000","last_unstale":"2024-09-20T22:44:53.142648+0000","last_undegraded":"2024-09-20T22:44:53.142648+0000","last_fullsized":"2024-09-20T22:44:53.142648+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:43:29.253585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:43:29.253585+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.d","version":"194'71","reported_seq":686,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.145995+0000","last_change":"2024-09-20T22:44:53.146787+0000","last_active":"2024-09-20T22:44:53.145995+0000","last_peered":"2024-09-20T22:44:53.145995+0000","last_clean":"2024-09-20T22:44:53.145995+0000","last_became_active":"2024-09-20T22:42:52.410381+0000","last_became_peered":"2024-09-20T22:42:52.410381+0000","last_unstale":"2024-09-20T22:44:53.145995+0000","last_undegraded":"2024-09-20T22:44:53.145995+0000","last_fullsized":"2024-09-20T22:44:53.145995+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"194'71","last_scrub_stamp":"2024-09-20T22:44:40.529449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:44:40.529449+0000","objects_scrubbed":1,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:48.697832+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0016513789999999999,"stat_sum":{"num_bytes":2588245,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":52,"num_read_kb":49492,"num_write":25,"num_write_kb":7311,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"37","length":"1"},{"start":"3b","length":"1"}]},{"pgid":"4.e","version":"74'11","reported_seq":200,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.145979+0000","last_change":"2024-09-20T22:44:16.481158+0000","last_active":"2024-09-20T22:44:53.145979+0000","last_peered":"2024-09-20T22:44:53.145979+0000","last_clean":"2024-09-20T22:44:53.145979+0000","last_became_active":"2024-09-20T22:44:09.177976+0000","last_became_peered":"2024-09-20T22:44:09.177976+0000","last_unstale":"2024-09-20T22:44:53.145979+0000","last_undegraded":"2024-09-20T22:44:53.145979+0000","last_fullsized":"2024-09-20T22:44:53.145979+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-09-20T22:44:16.481063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:44:16.481063+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:45.292329+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":124,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.150461+0000","last_change":"2024-09-20T22:43:57.432786+0000","last_active":"2024-09-20T22:44:53.150461+0000","last_peered":"2024-09-20T22:44:53.150461+0000","last_clean":"2024-09-20T22:44:53.150461+0000","last_became_active":"2024-09-20T22:43:57.432492+0000","last_became_peered":"2024-09-20T22:43:57.432492+0000","last_unstale":"2024-09-20T22:44:53.150461+0000","last_undegraded":"2024-09-20T22:44:53.150461+0000","last_fullsized":"2024-09-20T22:44:53.150461+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:31.979152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:43:31.979152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:59.118070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":304,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.617078+0000","last_change":"2024-09-20T22:43:57.420183+0000","last_active":"2024-09-20T22:44:53.617078+0000","last_peered":"2024-09-20T22:44:53.617078+0000","last_clean":"2024-09-20T22:44:53.617078+0000","last_became_active":"2024-09-20T22:43:57.419756+0000","last_became_peered":"2024-09-20T22:43:57.419756+0000","last_unstale":"2024-09-20T22:44:53.617078+0000","last_undegraded":"2024-09-20T22:44:53.617078+0000","last_fullsized":"2024-09-20T22:44:53.617078+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:54.631387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":302,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.144117+0000","last_change":"2024-09-20T22:44:21.225215+0000","last_active":"2024-09-20T22:44:53.144117+0000","last_peered":"2024-09-20T22:44:53.144117+0000","last_clean":"2024-09-20T22:44:53.144117+0000","last_became_active":"2024-09-20T22:41:44.832009+0000","last_became_peered":"2024-09-20T22:41:44.832009+0000","last_unstale":"2024-09-20T22:44:53.144117+0000","last_undegraded":"2024-09-20T22:44:53.144117+0000","last_fullsized":"2024-09-20T22:44:53.144117+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:44:21.225119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:44:21.225119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:39.909132+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.e","version":"194'81","reported_seq":696,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.152009+0000","last_change":"2024-09-20T22:44:53.152181+0000","last_active":"2024-09-20T22:44:53.152009+0000","last_peered":"2024-09-20T22:44:53.152009+0000","last_clean":"2024-09-20T22:44:53.152009+0000","last_became_active":"2024-09-20T22:42:52.405039+0000","last_became_peered":"2024-09-20T22:42:52.405039+0000","last_unstale":"2024-09-20T22:44:53.152009+0000","last_undegraded":"2024-09-20T22:44:53.152009+0000","last_fullsized":"2024-09-20T22:44:53.152009+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"174'60","last_scrub_stamp":"2024-09-20T22:44:18.301776+0000","last_deep_scrub":"174'60","last_deep_scrub_stamp":"2024-09-20T22:44:18.301776+0000","last_clean_scrub_stamp":"2024-09-20T22:44:18.301776+0000","objects_scrubbed":2,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:40.208586+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0.000375485,"stat_sum":{"num_bytes":7176589,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":63,"num_read_kb":71358,"num_write":36,"num_write_kb":15332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"37","length":"1"},{"start":"3b","length":"1"}]},{"pgid":"1.8","version":"77'128","reported_seq":550,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.617103+0000","last_change":"2024-09-20T22:43:04.748279+0000","last_active":"2024-09-20T22:44:53.617103+0000","last_peered":"2024-09-20T22:44:53.617103+0000","last_clean":"2024-09-20T22:44:53.617103+0000","last_became_active":"2024-09-20T22:42:52.405594+0000","last_became_peered":"2024-09-20T22:42:52.405594+0000","last_unstale":"2024-09-20T22:44:53.617103+0000","last_undegraded":"2024-09-20T22:44:53.617103+0000","last_fullsized":"2024-09-20T22:44:53.617103+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:43:04.748176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:43:04.748176+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:54.631531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"74'17","reported_seq":197,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.151908+0000","last_change":"2024-09-20T22:44:21.261543+0000","last_active":"2024-09-20T22:44:53.151908+0000","last_peered":"2024-09-20T22:44:53.151908+0000","last_clean":"2024-09-20T22:44:53.151908+0000","last_became_active":"2024-09-20T22:43:57.432621+0000","last_became_peered":"2024-09-20T22:43:57.432621+0000","last_unstale":"2024-09-20T22:44:53.151908+0000","last_undegraded":"2024-09-20T22:44:53.151908+0000","last_fullsized":"2024-09-20T22:44:53.151908+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'17","last_scrub_stamp":"2024-09-20T22:44:21.261443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:44:21.261443+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:39.405257+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":28,"num_read_kb":18,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":320,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.148111+0000","last_change":"2024-09-20T22:43:57.429041+0000","last_active":"2024-09-20T22:44:53.148111+0000","last_peered":"2024-09-20T22:44:53.148111+0000","last_clean":"2024-09-20T22:44:53.148111+0000","last_became_active":"2024-09-20T22:43:57.425267+0000","last_became_peered":"2024-09-20T22:43:57.425267+0000","last_unstale":"2024-09-20T22:44:53.148111+0000","last_undegraded":"2024-09-20T22:44:53.148111+0000","last_fullsized":"2024-09-20T22:44:53.148111+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:35.492562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:35.492562+0000","last_clean_scrub_stamp":"2024-09-20T22:43:35.492562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:56.636092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":312,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.139689+0000","last_change":"2024-09-20T22:44:11.041771+0000","last_active":"2024-09-20T22:44:53.139689+0000","last_peered":"2024-09-20T22:44:53.139689+0000","last_clean":"2024-09-20T22:44:53.139689+0000","last_became_active":"2024-09-20T22:43:57.435048+0000","last_became_peered":"2024-09-20T22:43:57.435048+0000","last_unstale":"2024-09-20T22:44:53.139689+0000","last_undegraded":"2024-09-20T22:44:53.139689+0000","last_fullsized":"2024-09-20T22:44:53.139689+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:44:11.041699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:44:11.041699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:34.130343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":315,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.151575+0000","last_change":"2024-09-20T22:44:20.297496+0000","last_active":"2024-09-20T22:44:53.151575+0000","last_peered":"2024-09-20T22:44:53.151575+0000","last_clean":"2024-09-20T22:44:53.151575+0000","last_became_active":"2024-09-20T22:43:57.430630+0000","last_became_peered":"2024-09-20T22:43:57.430630+0000","last_unstale":"2024-09-20T22:44:53.151575+0000","last_undegraded":"2024-09-20T22:44:53.151575+0000","last_fullsized":"2024-09-20T22:44:53.151575+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:44:20.297424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:44:20.297424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:44.120776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"214'116","reported_seq":738,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.165442+0000","last_change":"2024-09-20T22:44:53.143815+0000","last_active":"2024-09-20T22:44:53.165442+0000","last_peered":"2024-09-20T22:44:53.165442+0000","last_clean":"2024-09-20T22:44:53.165442+0000","last_became_active":"2024-09-20T22:42:52.401210+0000","last_became_peered":"2024-09-20T22:42:52.401210+0000","last_unstale":"2024-09-20T22:44:53.165442+0000","last_undegraded":"2024-09-20T22:44:53.165442+0000","last_fullsized":"2024-09-20T22:44:53.165442+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"194'90","last_scrub_stamp":"2024-09-20T22:44:34.651203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:44:34.651203+0000","objects_scrubbed":3,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:46:04.251754+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.003476542,"stat_sum":{"num_bytes":10621628,"num_objects":5,"num_object_clones":2,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":80,"num_read_kb":66986,"num_write":61,"num_write_kb":23887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"37","length":"1"},{"start":"3b","length":"1"}]},{"pgid":"1.9","version":"77'128","reported_seq":591,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.145882+0000","last_change":"2024-09-20T22:43:03.239347+0000","last_active":"2024-09-20T22:44:53.145882+0000","last_peered":"2024-09-20T22:44:53.145882+0000","last_clean":"2024-09-20T22:44:53.145882+0000","last_became_active":"2024-09-20T22:42:52.400353+0000","last_became_peered":"2024-09-20T22:42:52.400353+0000","last_unstale":"2024-09-20T22:44:53.145882+0000","last_undegraded":"2024-09-20T22:44:53.145882+0000","last_fullsized":"2024-09-20T22:44:53.145882+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:43:03.239276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:43:03.239276+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"74'10","reported_seq":170,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.144582+0000","last_change":"2024-09-20T22:43:57.486043+0000","last_active":"2024-09-20T22:44:53.144582+0000","last_peered":"2024-09-20T22:44:53.144582+0000","last_clean":"2024-09-20T22:44:53.144582+0000","last_became_active":"2024-09-20T22:43:57.441016+0000","last_became_peered":"2024-09-20T22:43:57.441016+0000","last_unstale":"2024-09-20T22:44:53.144582+0000","last_undegraded":"2024-09-20T22:44:53.144582+0000","last_fullsized":"2024-09-20T22:44:53.144582+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-09-20T22:43:51.040007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:43:51.040007+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:58.060282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":320,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.145095+0000","last_change":"2024-09-20T22:43:57.441917+0000","last_active":"2024-09-20T22:44:53.145095+0000","last_peered":"2024-09-20T22:44:53.145095+0000","last_clean":"2024-09-20T22:44:53.145095+0000","last_became_active":"2024-09-20T22:43:57.441623+0000","last_became_peered":"2024-09-20T22:43:57.441623+0000","last_unstale":"2024-09-20T22:44:53.145095+0000","last_undegraded":"2024-09-20T22:44:53.145095+0000","last_fullsized":"2024-09-20T22:44:53.145095+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:05.431346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:43:05.431346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:55.019817+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"74'8","reported_seq":353,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.151965+0000","last_change":"2024-09-20T22:43:56.238237+0000","last_active":"2024-09-20T22:44:53.151965+0000","last_peered":"2024-09-20T22:44:53.151965+0000","last_clean":"2024-09-20T22:44:53.151965+0000","last_became_active":"2024-09-20T22:41:42.185863+0000","last_became_peered":"2024-09-20T22:41:42.185863+0000","last_unstale":"2024-09-20T22:44:53.151965+0000","last_undegraded":"2024-09-20T22:44:53.151965+0000","last_fullsized":"2024-09-20T22:44:53.151965+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-09-20T22:43:56.238114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:43:56.238114+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:23.984720+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":133,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.141378+0000","last_change":"2024-09-20T22:44:13.962056+0000","last_active":"2024-09-20T22:44:53.141378+0000","last_peered":"2024-09-20T22:44:53.141378+0000","last_clean":"2024-09-20T22:44:53.141378+0000","last_became_active":"2024-09-20T22:43:57.430067+0000","last_became_peered":"2024-09-20T22:43:57.430067+0000","last_unstale":"2024-09-20T22:44:53.141378+0000","last_undegraded":"2024-09-20T22:44:53.141378+0000","last_fullsized":"2024-09-20T22:44:53.141378+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:44:13.961983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:44:13.961983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:24.591102+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.e","version":"77'128","reported_seq":423,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.144419+0000","last_change":"2024-09-20T22:44:12.918980+0000","last_active":"2024-09-20T22:44:53.144419+0000","last_peered":"2024-09-20T22:44:53.144419+0000","last_clean":"2024-09-20T22:44:53.144419+0000","last_became_active":"2024-09-20T22:44:00.145891+0000","last_became_peered":"2024-09-20T22:44:00.145891+0000","last_unstale":"2024-09-20T22:44:53.144419+0000","last_undegraded":"2024-09-20T22:44:53.144419+0000","last_fullsized":"2024-09-20T22:44:53.144419+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:44:12.918885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:44:12.918885+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:14.421434+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.8","version":"213'51","reported_seq":673,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.150213+0000","last_change":"2024-09-20T22:44:53.150512+0000","last_active":"2024-09-20T22:44:53.150213+0000","last_peered":"2024-09-20T22:44:53.150213+0000","last_clean":"2024-09-20T22:44:53.150213+0000","last_became_active":"2024-09-20T22:43:57.431081+0000","last_became_peered":"2024-09-20T22:43:57.431081+0000","last_unstale":"2024-09-20T22:44:53.150213+0000","last_undegraded":"2024-09-20T22:44:53.150213+0000","last_fullsized":"2024-09-20T22:44:53.150213+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"143'41","last_scrub_stamp":"2024-09-20T22:43:46.243723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:43:46.243723+0000","objects_scrubbed":0,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00053571000000000001,"stat_sum":{"num_bytes":2117715,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":7,"num_write_kb":3247,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"37","length":"1"},{"start":"3b","length":"1"}]},{"pgid":"4.b","version":"74'9","reported_seq":342,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.138481+0000","last_change":"2024-09-20T22:43:57.476333+0000","last_active":"2024-09-20T22:44:53.138481+0000","last_peered":"2024-09-20T22:44:53.138481+0000","last_clean":"2024-09-20T22:44:53.138481+0000","last_became_active":"2024-09-20T22:43:57.442189+0000","last_became_peered":"2024-09-20T22:43:57.442189+0000","last_unstale":"2024-09-20T22:44:53.138481+0000","last_undegraded":"2024-09-20T22:44:53.138481+0000","last_fullsized":"2024-09-20T22:44:53.138481+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-09-20T22:43:00.749456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:43:00.749456+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":314,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.618380+0000","last_change":"2024-09-20T22:43:52.773066+0000","last_active":"2024-09-20T22:44:53.618380+0000","last_peered":"2024-09-20T22:44:53.618380+0000","last_clean":"2024-09-20T22:44:53.618380+0000","last_became_active":"2024-09-20T22:41:38.155953+0000","last_became_peered":"2024-09-20T22:41:38.155953+0000","last_unstale":"2024-09-20T22:44:53.618380+0000","last_undegraded":"2024-09-20T22:44:53.618380+0000","last_fullsized":"2024-09-20T22:44:53.618380+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:52.772982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:43:52.772982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:04.819342+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":311,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.150961+0000","last_change":"2024-09-20T22:43:57.440505+0000","last_active":"2024-09-20T22:44:53.150961+0000","last_peered":"2024-09-20T22:44:53.150961+0000","last_clean":"2024-09-20T22:44:53.150961+0000","last_became_active":"2024-09-20T22:43:57.440200+0000","last_became_peered":"2024-09-20T22:43:57.440200+0000","last_unstale":"2024-09-20T22:44:53.150961+0000","last_undegraded":"2024-09-20T22:44:53.150961+0000","last_fullsized":"2024-09-20T22:44:53.150961+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:55.274872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:43:55.274872+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:17.019595+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":302,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.138680+0000","last_change":"2024-09-20T22:44:22.818177+0000","last_active":"2024-09-20T22:44:53.138680+0000","last_peered":"2024-09-20T22:44:53.138680+0000","last_clean":"2024-09-20T22:44:53.138680+0000","last_became_active":"2024-09-20T22:41:44.196633+0000","last_became_peered":"2024-09-20T22:41:44.196633+0000","last_unstale":"2024-09-20T22:44:53.138680+0000","last_undegraded":"2024-09-20T22:44:53.138680+0000","last_fullsized":"2024-09-20T22:44:53.138680+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:44:22.818078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:44:22.818078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:47.784094+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.f","version":"77'128","reported_seq":587,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.148249+0000","last_change":"2024-09-20T22:44:00.141677+0000","last_active":"2024-09-20T22:44:53.148249+0000","last_peered":"2024-09-20T22:44:53.148249+0000","last_clean":"2024-09-20T22:44:53.148249+0000","last_became_active":"2024-09-20T22:44:00.141317+0000","last_became_peered":"2024-09-20T22:44:00.141317+0000","last_unstale":"2024-09-20T22:44:53.148249+0000","last_undegraded":"2024-09-20T22:44:53.148249+0000","last_fullsized":"2024-09-20T22:44:53.148249+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:43:30.421648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:43:30.421648+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:54.611809+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.9","version":"210'96","reported_seq":732,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.140309+0000","last_change":"2024-09-20T22:44:53.141262+0000","last_active":"2024-09-20T22:44:53.140309+0000","last_peered":"2024-09-20T22:44:53.140309+0000","last_clean":"2024-09-20T22:44:53.140309+0000","last_became_active":"2024-09-20T22:43:57.431328+0000","last_became_peered":"2024-09-20T22:43:57.431328+0000","last_unstale":"2024-09-20T22:44:53.140309+0000","last_undegraded":"2024-09-20T22:44:53.140309+0000","last_fullsized":"2024-09-20T22:44:53.140309+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"131'44","last_scrub_stamp":"2024-09-20T22:43:46.794463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:43:46.794463+0000","objects_scrubbed":1,"log_size":96,"log_dups_size":0,"ondisk_log_size":96,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0027075850000000002,"stat_sum":{"num_bytes":4874063,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":24,"num_read_kb":26899,"num_write":21,"num_write_kb":7692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":3815156,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"37","length":"1"},{"start":"3b","length":"1"}]},{"pgid":"4.a","version":"113'28","reported_seq":350,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.147733+0000","last_change":"2024-09-20T22:43:12.726018+0000","last_active":"2024-09-20T22:44:53.147733+0000","last_peered":"2024-09-20T22:44:53.147733+0000","last_clean":"2024-09-20T22:44:53.147733+0000","last_became_active":"2024-09-20T22:41:40.252011+0000","last_became_peered":"2024-09-20T22:41:40.252011+0000","last_unstale":"2024-09-20T22:44:53.147733+0000","last_undegraded":"2024-09-20T22:44:53.147733+0000","last_fullsized":"2024-09-20T22:44:53.147733+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'28","last_scrub_stamp":"2024-09-20T22:43:12.725900+0000","last_deep_scrub":"113'28","last_deep_scrub_stamp":"2024-09-20T22:43:12.725900+0000","last_clean_scrub_stamp":"2024-09-20T22:43:12.725900+0000","objects_scrubbed":9,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:54.611709+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":314,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.146750+0000","last_change":"2024-09-20T22:44:18.955889+0000","last_active":"2024-09-20T22:44:53.146750+0000","last_peered":"2024-09-20T22:44:53.146750+0000","last_clean":"2024-09-20T22:44:53.146750+0000","last_became_active":"2024-09-20T22:41:38.161524+0000","last_became_peered":"2024-09-20T22:41:38.161524+0000","last_unstale":"2024-09-20T22:44:53.146750+0000","last_undegraded":"2024-09-20T22:44:53.146750+0000","last_fullsized":"2024-09-20T22:44:53.146750+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:44:18.955801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:44:18.955801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:26.881892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":305,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.150774+0000","last_change":"2024-09-20T22:44:12.236220+0000","last_active":"2024-09-20T22:44:53.150774+0000","last_peered":"2024-09-20T22:44:53.150774+0000","last_clean":"2024-09-20T22:44:53.150774+0000","last_became_active":"2024-09-20T22:41:42.184342+0000","last_became_peered":"2024-09-20T22:41:42.184342+0000","last_unstale":"2024-09-20T22:44:53.150774+0000","last_undegraded":"2024-09-20T22:44:53.150774+0000","last_fullsized":"2024-09-20T22:44:53.150774+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:44:12.236096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:44:12.236096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:16.825489+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":310,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.145413+0000","last_change":"2024-09-20T22:44:42.956942+0000","last_active":"2024-09-20T22:44:53.145413+0000","last_peered":"2024-09-20T22:44:53.145413+0000","last_clean":"2024-09-20T22:44:53.145413+0000","last_became_active":"2024-09-20T22:41:44.199395+0000","last_became_peered":"2024-09-20T22:41:44.199395+0000","last_unstale":"2024-09-20T22:44:53.145413+0000","last_undegraded":"2024-09-20T22:44:53.145413+0000","last_fullsized":"2024-09-20T22:44:53.145413+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:44:42.956859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:44:42.956859+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:48.879336+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.c","version":"77'128","reported_seq":577,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.144701+0000","last_change":"2024-09-20T22:43:30.045082+0000","last_active":"2024-09-20T22:44:53.144701+0000","last_peered":"2024-09-20T22:44:53.144701+0000","last_clean":"2024-09-20T22:44:53.144701+0000","last_became_active":"2024-09-20T22:43:24.130949+0000","last_became_peered":"2024-09-20T22:43:24.130949+0000","last_unstale":"2024-09-20T22:44:53.144701+0000","last_undegraded":"2024-09-20T22:44:53.144701+0000","last_fullsized":"2024-09-20T22:44:53.144701+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:43:30.044975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:43:30.044975+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:55.019914+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"204'133","reported_seq":766,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.144400+0000","last_change":"2024-09-20T22:44:53.144707+0000","last_active":"2024-09-20T22:44:53.144400+0000","last_peered":"2024-09-20T22:44:53.144400+0000","last_clean":"2024-09-20T22:44:53.144400+0000","last_became_active":"2024-09-20T22:42:52.405085+0000","last_became_peered":"2024-09-20T22:42:52.405085+0000","last_unstale":"2024-09-20T22:44:53.144400+0000","last_undegraded":"2024-09-20T22:44:53.144400+0000","last_fullsized":"2024-09-20T22:44:53.144400+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:42:51.354519+0000","objects_scrubbed":0,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00083729099999999999,"stat_sum":{"num_bytes":9445567,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":123,"num_read_kb":130841,"num_write":83,"num_write_kb":30238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"37","length":"1"},{"start":"3b","length":"1"}]},{"pgid":"4.9","version":"74'12","reported_seq":186,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.139952+0000","last_change":"2024-09-20T22:44:53.049195+0000","last_active":"2024-09-20T22:44:53.139952+0000","last_peered":"2024-09-20T22:44:53.139952+0000","last_clean":"2024-09-20T22:44:53.139952+0000","last_became_active":"2024-09-20T22:43:57.442218+0000","last_became_peered":"2024-09-20T22:43:57.442218+0000","last_unstale":"2024-09-20T22:44:53.139952+0000","last_undegraded":"2024-09-20T22:44:53.139952+0000","last_fullsized":"2024-09-20T22:44:53.139952+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-09-20T22:44:53.049111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:44:53.049111+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:55.990924+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":21,"num_read_kb":12,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":328,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.145144+0000","last_change":"2024-09-20T22:44:20.970396+0000","last_active":"2024-09-20T22:44:53.145144+0000","last_peered":"2024-09-20T22:44:53.145144+0000","last_clean":"2024-09-20T22:44:53.145144+0000","last_became_active":"2024-09-20T22:43:57.419980+0000","last_became_peered":"2024-09-20T22:43:57.419980+0000","last_unstale":"2024-09-20T22:44:53.145144+0000","last_undegraded":"2024-09-20T22:44:53.145144+0000","last_fullsized":"2024-09-20T22:44:53.145144+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:44:20.970293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:44:20.970293+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:44.477869+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":313,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.150291+0000","last_change":"2024-09-20T22:44:37.379076+0000","last_active":"2024-09-20T22:44:53.150291+0000","last_peered":"2024-09-20T22:44:53.150291+0000","last_clean":"2024-09-20T22:44:53.150291+0000","last_became_active":"2024-09-20T22:41:42.181811+0000","last_became_peered":"2024-09-20T22:41:42.181811+0000","last_unstale":"2024-09-20T22:44:53.150291+0000","last_undegraded":"2024-09-20T22:44:53.150291+0000","last_fullsized":"2024-09-20T22:44:53.150291+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:44:37.379005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:44:37.379005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:44.358677+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":302,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.144127+0000","last_change":"2024-09-20T22:43:55.170518+0000","last_active":"2024-09-20T22:44:53.144127+0000","last_peered":"2024-09-20T22:44:53.144127+0000","last_clean":"2024-09-20T22:44:53.144127+0000","last_became_active":"2024-09-20T22:41:44.192609+0000","last_became_peered":"2024-09-20T22:41:44.192609+0000","last_unstale":"2024-09-20T22:44:53.144127+0000","last_undegraded":"2024-09-20T22:44:53.144127+0000","last_fullsized":"2024-09-20T22:44:53.144127+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:55.170379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:55.170379+0000","last_clean_scrub_stamp":"2024-09-20T22:43:55.170379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:04.790203+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.d","version":"77'128","reported_seq":499,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.144645+0000","last_change":"2024-09-20T22:44:00.147264+0000","last_active":"2024-09-20T22:44:53.144645+0000","last_peered":"2024-09-20T22:44:53.144645+0000","last_clean":"2024-09-20T22:44:53.144645+0000","last_became_active":"2024-09-20T22:44:00.146905+0000","last_became_peered":"2024-09-20T22:44:00.146905+0000","last_unstale":"2024-09-20T22:44:53.144645+0000","last_undegraded":"2024-09-20T22:44:53.144645+0000","last_fullsized":"2024-09-20T22:44:53.144645+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:43:30.175229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:43:30.175229+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:56.017421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.b","version":"210'130","reported_seq":756,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.142918+0000","last_change":"2024-09-20T22:44:53.143116+0000","last_active":"2024-09-20T22:44:53.142918+0000","last_peered":"2024-09-20T22:44:53.142918+0000","last_clean":"2024-09-20T22:44:53.142918+0000","last_became_active":"2024-09-20T22:42:52.409642+0000","last_became_peered":"2024-09-20T22:42:52.409642+0000","last_unstale":"2024-09-20T22:44:53.142918+0000","last_undegraded":"2024-09-20T22:44:53.142918+0000","last_fullsized":"2024-09-20T22:44:53.142918+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"194'116","last_scrub_stamp":"2024-09-20T22:44:37.097674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:44:37.097674+0000","objects_scrubbed":3,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:57.375658+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00064806599999999998,"stat_sum":{"num_bytes":8890936,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":86,"num_read_kb":78897,"num_write":76,"num_write_kb":27466,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"37","length":"1"},{"start":"3b","length":"1"}]},{"pgid":"4.8","version":"74'15","reported_seq":327,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.617273+0000","last_change":"2024-09-20T22:41:40.252427+0000","last_active":"2024-09-20T22:44:53.617273+0000","last_peered":"2024-09-20T22:44:53.617273+0000","last_clean":"2024-09-20T22:44:53.617273+0000","last_became_active":"2024-09-20T22:41:40.252113+0000","last_became_peered":"2024-09-20T22:41:40.252113+0000","last_unstale":"2024-09-20T22:44:53.617273+0000","last_undegraded":"2024-09-20T22:44:53.617273+0000","last_fullsized":"2024-09-20T22:44:53.617273+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:41:39.147434+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:54.631418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'2","reported_seq":338,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.146308+0000","last_change":"2024-09-20T22:44:16.936605+0000","last_active":"2024-09-20T22:44:53.146308+0000","last_peered":"2024-09-20T22:44:53.146308+0000","last_clean":"2024-09-20T22:44:53.146308+0000","last_became_active":"2024-09-20T22:43:57.422322+0000","last_became_peered":"2024-09-20T22:43:57.422322+0000","last_unstale":"2024-09-20T22:44:53.146308+0000","last_undegraded":"2024-09-20T22:44:53.146308+0000","last_fullsized":"2024-09-20T22:44:53.146308+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-09-20T22:44:16.936523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:44:16.936523+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:21.826669+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"159'9","reported_seq":372,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.145788+0000","last_change":"2024-09-20T22:44:36.904443+0000","last_active":"2024-09-20T22:44:53.145788+0000","last_peered":"2024-09-20T22:44:53.145788+0000","last_clean":"2024-09-20T22:44:53.145788+0000","last_became_active":"2024-09-20T22:43:57.420663+0000","last_became_peered":"2024-09-20T22:43:57.420663+0000","last_unstale":"2024-09-20T22:44:53.145788+0000","last_undegraded":"2024-09-20T22:44:53.145788+0000","last_fullsized":"2024-09-20T22:44:53.145788+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"159'9","last_scrub_stamp":"2024-09-20T22:44:36.904367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:44:36.904367+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:54.974970+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":294,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.617577+0000","last_change":"2024-09-20T22:41:44.831435+0000","last_active":"2024-09-20T22:44:53.617577+0000","last_peered":"2024-09-20T22:44:53.617577+0000","last_clean":"2024-09-20T22:44:53.617577+0000","last_became_active":"2024-09-20T22:41:44.830556+0000","last_became_peered":"2024-09-20T22:41:44.830556+0000","last_unstale":"2024-09-20T22:44:53.617577+0000","last_undegraded":"2024-09-20T22:44:53.617577+0000","last_fullsized":"2024-09-20T22:44:53.617577+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:54.631446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.14","version":"166'42","reported_seq":424,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.145238+0000","last_change":"2024-09-20T22:44:53.145999+0000","last_active":"2024-09-20T22:44:53.145238+0000","last_peered":"2024-09-20T22:44:53.145238+0000","last_clean":"2024-09-20T22:44:53.145238+0000","last_became_active":"2024-09-20T22:43:39.132459+0000","last_became_peered":"2024-09-20T22:43:39.132459+0000","last_unstale":"2024-09-20T22:44:53.145238+0000","last_undegraded":"2024-09-20T22:44:53.145238+0000","last_fullsized":"2024-09-20T22:44:53.145238+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":5,"last_scrub":"147'38","last_scrub_stamp":"2024-09-20T22:43:49.134297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:43:49.134297+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:58.327235+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0020644980000000001,"stat_sum":{"num_bytes":2806769,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":9,"num_read_kb":8509,"num_write":5,"num_write_kb":1512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"37","length":"1"},{"start":"3b","length":"1"}]},{"pgid":"1.12","version":"77'128","reported_seq":488,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.617694+0000","last_change":"2024-09-20T22:43:29.735118+0000","last_active":"2024-09-20T22:44:53.617694+0000","last_peered":"2024-09-20T22:44:53.617694+0000","last_clean":"2024-09-20T22:44:53.617694+0000","last_became_active":"2024-09-20T22:43:24.124301+0000","last_became_peered":"2024-09-20T22:43:24.124301+0000","last_unstale":"2024-09-20T22:44:53.617694+0000","last_undegraded":"2024-09-20T22:44:53.617694+0000","last_fullsized":"2024-09-20T22:44:53.617694+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:43:29.735047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:43:29.735047+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:55.469557+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":322,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.143943+0000","last_change":"2024-09-20T22:44:42.612217+0000","last_active":"2024-09-20T22:44:53.143943+0000","last_peered":"2024-09-20T22:44:53.143943+0000","last_clean":"2024-09-20T22:44:53.143943+0000","last_became_active":"2024-09-20T22:41:38.159139+0000","last_became_peered":"2024-09-20T22:41:38.159139+0000","last_unstale":"2024-09-20T22:44:53.143943+0000","last_undegraded":"2024-09-20T22:44:53.143943+0000","last_fullsized":"2024-09-20T22:44:53.143943+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:44:42.612144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:44:42.612144+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:46:11.366920+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"74'6","reported_seq":333,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.145369+0000","last_change":"2024-09-20T22:43:09.121694+0000","last_active":"2024-09-20T22:44:53.145369+0000","last_peered":"2024-09-20T22:44:53.145369+0000","last_clean":"2024-09-20T22:44:53.145369+0000","last_became_active":"2024-09-20T22:43:09.116887+0000","last_became_peered":"2024-09-20T22:43:09.116887+0000","last_unstale":"2024-09-20T22:44:53.145369+0000","last_undegraded":"2024-09-20T22:44:53.145369+0000","last_fullsized":"2024-09-20T22:44:53.145369+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'6","last_scrub_stamp":"2024-09-20T22:42:41.469214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:42:41.469214+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:55.019773+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":306,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.618396+0000","last_change":"2024-09-20T22:43:09.705715+0000","last_active":"2024-09-20T22:44:53.618396+0000","last_peered":"2024-09-20T22:44:53.618396+0000","last_clean":"2024-09-20T22:44:53.618396+0000","last_became_active":"2024-09-20T22:41:42.184930+0000","last_became_peered":"2024-09-20T22:41:42.184930+0000","last_unstale":"2024-09-20T22:44:53.618396+0000","last_undegraded":"2024-09-20T22:44:53.618396+0000","last_fullsized":"2024-09-20T22:44:53.618396+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:09.705632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:09.705632+0000","last_clean_scrub_stamp":"2024-09-20T22:43:09.705632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:54.631658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"74'1","reported_seq":313,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.145094+0000","last_change":"2024-09-20T22:43:57.435632+0000","last_active":"2024-09-20T22:44:53.145094+0000","last_peered":"2024-09-20T22:44:53.145094+0000","last_clean":"2024-09-20T22:44:53.145094+0000","last_became_active":"2024-09-20T22:43:57.419681+0000","last_became_peered":"2024-09-20T22:43:57.419681+0000","last_unstale":"2024-09-20T22:44:53.145094+0000","last_undegraded":"2024-09-20T22:44:53.145094+0000","last_fullsized":"2024-09-20T22:44:53.145094+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-09-20T22:43:33.180910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:43:33.180910+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:00.529090+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.15","version":"98'3","reported_seq":616,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.146254+0000","last_change":"2024-09-20T22:44:53.147390+0000","last_active":"2024-09-20T22:44:53.146254+0000","last_peered":"2024-09-20T22:44:53.146254+0000","last_clean":"2024-09-20T22:44:53.146254+0000","last_became_active":"2024-09-20T22:44:09.174719+0000","last_became_peered":"2024-09-20T22:44:09.174719+0000","last_unstale":"2024-09-20T22:44:53.146254+0000","last_undegraded":"2024-09-20T22:44:53.146254+0000","last_fullsized":"2024-09-20T22:44:53.146254+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":5,"last_scrub":"98'3","last_scrub_stamp":"2024-09-20T22:43:47.212572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:43:47.212572+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:00.877559+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0031220129999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"37","length":"1"},{"start":"3b","length":"1"}]},{"pgid":"1.13","version":"77'128","reported_seq":588,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.142545+0000","last_change":"2024-09-20T22:44:38.112816+0000","last_active":"2024-09-20T22:44:53.142545+0000","last_peered":"2024-09-20T22:44:53.142545+0000","last_clean":"2024-09-20T22:44:53.142545+0000","last_became_active":"2024-09-20T22:44:00.144081+0000","last_became_peered":"2024-09-20T22:44:00.144081+0000","last_unstale":"2024-09-20T22:44:53.142545+0000","last_undegraded":"2024-09-20T22:44:53.142545+0000","last_fullsized":"2024-09-20T22:44:53.142545+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:44:38.112742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:44:38.112742+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:42.803148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"74'9","reported_seq":334,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.141871+0000","last_change":"2024-09-20T22:44:52.644656+0000","last_active":"2024-09-20T22:44:53.141871+0000","last_peered":"2024-09-20T22:44:53.141871+0000","last_clean":"2024-09-20T22:44:53.141871+0000","last_became_active":"2024-09-20T22:41:40.253280+0000","last_became_peered":"2024-09-20T22:41:40.253280+0000","last_unstale":"2024-09-20T22:44:53.141871+0000","last_undegraded":"2024-09-20T22:44:53.141871+0000","last_fullsized":"2024-09-20T22:44:53.141871+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-09-20T22:44:52.644575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:44:52.644575+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:46:04.206183+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":320,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.146007+0000","last_change":"2024-09-20T22:44:11.942183+0000","last_active":"2024-09-20T22:44:53.146007+0000","last_peered":"2024-09-20T22:44:53.146007+0000","last_clean":"2024-09-20T22:44:53.146007+0000","last_became_active":"2024-09-20T22:43:57.419001+0000","last_became_peered":"2024-09-20T22:43:57.419001+0000","last_unstale":"2024-09-20T22:44:53.146007+0000","last_undegraded":"2024-09-20T22:44:53.146007+0000","last_fullsized":"2024-09-20T22:44:53.146007+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:44:11.942106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:44:11.942106+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:12.734421+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":306,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.145226+0000","last_change":"2024-09-20T22:43:32.057321+0000","last_active":"2024-09-20T22:44:53.145226+0000","last_peered":"2024-09-20T22:44:53.145226+0000","last_clean":"2024-09-20T22:44:53.145226+0000","last_became_active":"2024-09-20T22:41:42.187706+0000","last_became_peered":"2024-09-20T22:41:42.187706+0000","last_unstale":"2024-09-20T22:44:53.145226+0000","last_undegraded":"2024-09-20T22:44:53.145226+0000","last_fullsized":"2024-09-20T22:44:53.145226+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:32.057240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:43:32.057240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":307,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.150953+0000","last_change":"2024-09-20T22:43:57.428646+0000","last_active":"2024-09-20T22:44:53.150953+0000","last_peered":"2024-09-20T22:44:53.150953+0000","last_clean":"2024-09-20T22:44:53.150953+0000","last_became_active":"2024-09-20T22:43:57.428284+0000","last_became_peered":"2024-09-20T22:43:57.428284+0000","last_unstale":"2024-09-20T22:44:53.150953+0000","last_undegraded":"2024-09-20T22:44:53.150953+0000","last_fullsized":"2024-09-20T22:44:53.150953+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:08.232288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:43:08.232288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:54.488293+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.16","version":"210'47","reported_seq":577,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.141708+0000","last_change":"2024-09-20T22:44:53.142562+0000","last_active":"2024-09-20T22:44:53.141708+0000","last_peered":"2024-09-20T22:44:53.141708+0000","last_clean":"2024-09-20T22:44:53.141708+0000","last_became_active":"2024-09-20T22:43:57.423267+0000","last_became_peered":"2024-09-20T22:43:57.423267+0000","last_unstale":"2024-09-20T22:44:53.141708+0000","last_undegraded":"2024-09-20T22:44:53.141708+0000","last_fullsized":"2024-09-20T22:44:53.141708+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":5,"last_scrub":"136'41","last_scrub_stamp":"2024-09-20T22:44:09.743964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:44:09.743964+0000","objects_scrubbed":1,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:21.361236+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.001968418,"stat_sum":{"num_bytes":2218491,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":15653,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"37","length":"1"},{"start":"3b","length":"1"}]},{"pgid":"1.10","version":"77'128","reported_seq":496,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.616995+0000","last_change":"2024-09-20T22:44:34.630283+0000","last_active":"2024-09-20T22:44:53.616995+0000","last_peered":"2024-09-20T22:44:53.616995+0000","last_clean":"2024-09-20T22:44:53.616995+0000","last_became_active":"2024-09-20T22:43:24.125651+0000","last_became_peered":"2024-09-20T22:43:24.125651+0000","last_unstale":"2024-09-20T22:44:53.616995+0000","last_undegraded":"2024-09-20T22:44:53.616995+0000","last_fullsized":"2024-09-20T22:44:53.616995+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:44:34.630184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:44:34.630184+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:40.023975+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.15","version":"74'9","reported_seq":326,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.618601+0000","last_change":"2024-09-20T22:43:49.825694+0000","last_active":"2024-09-20T22:44:53.618601+0000","last_peered":"2024-09-20T22:44:53.618601+0000","last_clean":"2024-09-20T22:44:53.618601+0000","last_became_active":"2024-09-20T22:41:40.252486+0000","last_became_peered":"2024-09-20T22:41:40.252486+0000","last_unstale":"2024-09-20T22:44:53.618601+0000","last_undegraded":"2024-09-20T22:44:53.618601+0000","last_fullsized":"2024-09-20T22:44:53.618601+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-09-20T22:43:49.825573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:43:49.825573+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:17.986284+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":314,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.139339+0000","last_change":"2024-09-20T22:43:32.652702+0000","last_active":"2024-09-20T22:44:53.139339+0000","last_peered":"2024-09-20T22:44:53.139339+0000","last_clean":"2024-09-20T22:44:53.139339+0000","last_became_active":"2024-09-20T22:41:38.158146+0000","last_became_peered":"2024-09-20T22:41:38.158146+0000","last_unstale":"2024-09-20T22:44:53.139339+0000","last_undegraded":"2024-09-20T22:44:53.139339+0000","last_fullsized":"2024-09-20T22:44:53.139339+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:32.652587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:43:32.652587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"74'8","reported_seq":354,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.144323+0000","last_change":"2024-09-20T22:43:52.190172+0000","last_active":"2024-09-20T22:44:53.144323+0000","last_peered":"2024-09-20T22:44:53.144323+0000","last_clean":"2024-09-20T22:44:53.144323+0000","last_became_active":"2024-09-20T22:41:42.186352+0000","last_became_peered":"2024-09-20T22:41:42.186352+0000","last_unstale":"2024-09-20T22:44:53.144323+0000","last_undegraded":"2024-09-20T22:44:53.144323+0000","last_fullsized":"2024-09-20T22:44:53.144323+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-09-20T22:43:52.190097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:43:52.190097+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:14.977846+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":133,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.151531+0000","last_change":"2024-09-20T22:44:13.241839+0000","last_active":"2024-09-20T22:44:53.151531+0000","last_peered":"2024-09-20T22:44:53.151531+0000","last_clean":"2024-09-20T22:44:53.151531+0000","last_became_active":"2024-09-20T22:43:57.424515+0000","last_became_peered":"2024-09-20T22:43:57.424515+0000","last_unstale":"2024-09-20T22:44:53.151531+0000","last_undegraded":"2024-09-20T22:44:53.151531+0000","last_fullsized":"2024-09-20T22:44:53.151531+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:44:13.241769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:44:13.241769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:14.652647+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.17","version":"199'162","reported_seq":501,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.147264+0000","last_change":"2024-09-20T22:44:53.147654+0000","last_active":"2024-09-20T22:44:53.147264+0000","last_peered":"2024-09-20T22:44:53.147264+0000","last_clean":"2024-09-20T22:44:53.147264+0000","last_became_active":"2024-09-20T22:43:39.130801+0000","last_became_peered":"2024-09-20T22:43:39.130801+0000","last_unstale":"2024-09-20T22:44:53.147264+0000","last_undegraded":"2024-09-20T22:44:53.147264+0000","last_fullsized":"2024-09-20T22:44:53.147264+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":5,"last_scrub":"147'126","last_scrub_stamp":"2024-09-20T22:43:48.215007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:43:48.215007+0000","objects_scrubbed":3,"log_size":162,"log_dups_size":0,"ondisk_log_size":162,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.001920417,"stat_sum":{"num_bytes":6134555,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":44,"num_read_kb":56205,"num_write":20,"num_write_kb":10435,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"37","length":"1"},{"start":"3b","length":"1"}]},{"pgid":"1.11","version":"77'128","reported_seq":502,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.138851+0000","last_change":"2024-09-20T22:43:30.651129+0000","last_active":"2024-09-20T22:44:53.138851+0000","last_peered":"2024-09-20T22:44:53.138851+0000","last_clean":"2024-09-20T22:44:53.138851+0000","last_became_active":"2024-09-20T22:43:24.128884+0000","last_became_peered":"2024-09-20T22:43:24.128884+0000","last_unstale":"2024-09-20T22:44:53.138851+0000","last_undegraded":"2024-09-20T22:44:53.138851+0000","last_fullsized":"2024-09-20T22:44:53.138851+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:43:30.651054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:43:30.651054+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:55.194390+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"74'10","reported_seq":325,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.144867+0000","last_change":"2024-09-20T22:43:11.355914+0000","last_active":"2024-09-20T22:44:53.144867+0000","last_peered":"2024-09-20T22:44:53.144867+0000","last_clean":"2024-09-20T22:44:53.144867+0000","last_became_active":"2024-09-20T22:41:40.253614+0000","last_became_peered":"2024-09-20T22:41:40.253614+0000","last_unstale":"2024-09-20T22:44:53.144867+0000","last_undegraded":"2024-09-20T22:44:53.144867+0000","last_fullsized":"2024-09-20T22:44:53.144867+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-09-20T22:43:11.355829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:43:11.355829+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:55.019889+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":320,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.144276+0000","last_change":"2024-09-20T22:43:57.425304+0000","last_active":"2024-09-20T22:44:53.144276+0000","last_peered":"2024-09-20T22:44:53.144276+0000","last_clean":"2024-09-20T22:44:53.144276+0000","last_became_active":"2024-09-20T22:43:57.425004+0000","last_became_peered":"2024-09-20T22:43:57.425004+0000","last_unstale":"2024-09-20T22:44:53.144276+0000","last_undegraded":"2024-09-20T22:44:53.144276+0000","last_fullsized":"2024-09-20T22:44:53.144276+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:51.130212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:43:51.130212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:53.576047+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"74'8","reported_seq":357,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.618446+0000","last_change":"2024-09-20T22:42:59.718431+0000","last_active":"2024-09-20T22:44:53.618446+0000","last_peered":"2024-09-20T22:44:53.618446+0000","last_clean":"2024-09-20T22:44:53.618446+0000","last_became_active":"2024-09-20T22:41:42.189451+0000","last_became_peered":"2024-09-20T22:41:42.189451+0000","last_unstale":"2024-09-20T22:44:53.618446+0000","last_undegraded":"2024-09-20T22:44:53.618446+0000","last_fullsized":"2024-09-20T22:44:53.618446+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-09-20T22:42:59.718354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:42:59.718354+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:54.631502+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":302,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.141141+0000","last_change":"2024-09-20T22:42:58.680721+0000","last_active":"2024-09-20T22:44:53.141141+0000","last_peered":"2024-09-20T22:44:53.141141+0000","last_clean":"2024-09-20T22:44:53.141141+0000","last_became_active":"2024-09-20T22:41:44.191206+0000","last_became_peered":"2024-09-20T22:41:44.191206+0000","last_unstale":"2024-09-20T22:44:53.141141+0000","last_undegraded":"2024-09-20T22:44:53.141141+0000","last_fullsized":"2024-09-20T22:44:53.141141+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:42:58.680648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:42:58.680648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.10","version":"199'52","reported_seq":658,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.144336+0000","last_change":"2024-09-20T22:44:53.145154+0000","last_active":"2024-09-20T22:44:53.144336+0000","last_peered":"2024-09-20T22:44:53.144336+0000","last_clean":"2024-09-20T22:44:53.144336+0000","last_became_active":"2024-09-20T22:43:39.130391+0000","last_became_peered":"2024-09-20T22:43:39.130391+0000","last_unstale":"2024-09-20T22:44:53.144336+0000","last_undegraded":"2024-09-20T22:44:53.144336+0000","last_fullsized":"2024-09-20T22:44:53.144336+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":5,"last_scrub":"146'43","last_scrub_stamp":"2024-09-20T22:43:46.106059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:43:46.106059+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:54.305704+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00163781,"stat_sum":{"num_bytes":2537857,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":7491,"num_write":6,"num_write_kb":1265,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"37","length":"1"},{"start":"3b","length":"1"}]},{"pgid":"4.13","version":"74'11","reported_seq":174,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.144229+0000","last_change":"2024-09-20T22:43:57.446108+0000","last_active":"2024-09-20T22:44:53.144229+0000","last_peered":"2024-09-20T22:44:53.144229+0000","last_clean":"2024-09-20T22:44:53.144229+0000","last_became_active":"2024-09-20T22:43:57.416358+0000","last_became_peered":"2024-09-20T22:43:57.416358+0000","last_unstale":"2024-09-20T22:44:53.144229+0000","last_undegraded":"2024-09-20T22:44:53.144229+0000","last_fullsized":"2024-09-20T22:44:53.144229+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-09-20T22:43:33.923283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:43:33.923283+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":124,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.144979+0000","last_change":"2024-09-20T22:43:57.423148+0000","last_active":"2024-09-20T22:44:53.144979+0000","last_peered":"2024-09-20T22:44:53.144979+0000","last_clean":"2024-09-20T22:44:53.144979+0000","last_became_active":"2024-09-20T22:43:57.422556+0000","last_became_peered":"2024-09-20T22:43:57.422556+0000","last_unstale":"2024-09-20T22:44:53.144979+0000","last_undegraded":"2024-09-20T22:44:53.144979+0000","last_fullsized":"2024-09-20T22:44:53.144979+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:55.053048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:43:55.053048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:15.115356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":306,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.138760+0000","last_change":"2024-09-20T22:44:20.088194+0000","last_active":"2024-09-20T22:44:53.138760+0000","last_peered":"2024-09-20T22:44:53.138760+0000","last_clean":"2024-09-20T22:44:53.138760+0000","last_became_active":"2024-09-20T22:41:42.181736+0000","last_became_peered":"2024-09-20T22:41:42.181736+0000","last_unstale":"2024-09-20T22:44:53.138760+0000","last_undegraded":"2024-09-20T22:44:53.138760+0000","last_fullsized":"2024-09-20T22:44:53.138760+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:44:20.088125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:44:20.088125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:33.759142+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":302,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.143677+0000","last_change":"2024-09-20T22:43:53.201536+0000","last_active":"2024-09-20T22:44:53.143677+0000","last_peered":"2024-09-20T22:44:53.143677+0000","last_clean":"2024-09-20T22:44:53.143677+0000","last_became_active":"2024-09-20T22:41:44.192960+0000","last_became_peered":"2024-09-20T22:41:44.192960+0000","last_unstale":"2024-09-20T22:44:53.143677+0000","last_undegraded":"2024-09-20T22:44:53.143677+0000","last_fullsized":"2024-09-20T22:44:53.143677+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:53.201465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:43:53.201465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:09.466528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"135'39","reported_seq":397,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.145597+0000","last_change":"2024-09-20T22:44:53.146509+0000","last_active":"2024-09-20T22:44:53.145597+0000","last_peered":"2024-09-20T22:44:53.145597+0000","last_clean":"2024-09-20T22:44:53.145597+0000","last_became_active":"2024-09-20T22:43:39.130210+0000","last_became_peered":"2024-09-20T22:43:39.130210+0000","last_unstale":"2024-09-20T22:44:53.145597+0000","last_undegraded":"2024-09-20T22:44:53.145597+0000","last_fullsized":"2024-09-20T22:44:53.145597+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":5,"last_scrub":"135'39","last_scrub_stamp":"2024-09-20T22:43:46.228343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:43:46.228343+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:10.924615+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0022831280000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"37","length":"1"},{"start":"3b","length":"1"}]},{"pgid":"4.12","version":"74'9","reported_seq":326,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.139065+0000","last_change":"2024-09-20T22:43:28.210970+0000","last_active":"2024-09-20T22:44:53.139065+0000","last_peered":"2024-09-20T22:44:53.139065+0000","last_clean":"2024-09-20T22:44:53.139065+0000","last_became_active":"2024-09-20T22:41:40.164796+0000","last_became_peered":"2024-09-20T22:41:40.164796+0000","last_unstale":"2024-09-20T22:44:53.139065+0000","last_undegraded":"2024-09-20T22:44:53.139065+0000","last_fullsized":"2024-09-20T22:44:53.139065+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-09-20T22:43:28.210878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:43:28.210878+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":323,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.146580+0000","last_change":"2024-09-20T22:43:57.429374+0000","last_active":"2024-09-20T22:44:53.146580+0000","last_peered":"2024-09-20T22:44:53.146580+0000","last_clean":"2024-09-20T22:44:53.146580+0000","last_became_active":"2024-09-20T22:43:57.429046+0000","last_became_peered":"2024-09-20T22:43:57.429046+0000","last_unstale":"2024-09-20T22:44:53.146580+0000","last_undegraded":"2024-09-20T22:44:53.146580+0000","last_fullsized":"2024-09-20T22:44:53.146580+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:31.138067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:31.138067+0000","last_clean_scrub_stamp":"2024-09-20T22:43:31.138067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:56.064009+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":306,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.144227+0000","last_change":"2024-09-20T22:44:15.242983+0000","last_active":"2024-09-20T22:44:53.144227+0000","last_peered":"2024-09-20T22:44:53.144227+0000","last_clean":"2024-09-20T22:44:53.144227+0000","last_became_active":"2024-09-20T22:41:42.182966+0000","last_became_peered":"2024-09-20T22:41:42.182966+0000","last_unstale":"2024-09-20T22:44:53.144227+0000","last_undegraded":"2024-09-20T22:44:53.144227+0000","last_fullsized":"2024-09-20T22:44:53.144227+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:44:15.242881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:44:15.242881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:20.962494+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":310,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.139673+0000","last_change":"2024-09-20T22:44:39.660113+0000","last_active":"2024-09-20T22:44:53.139673+0000","last_peered":"2024-09-20T22:44:53.139673+0000","last_clean":"2024-09-20T22:44:53.139673+0000","last_became_active":"2024-09-20T22:41:44.194551+0000","last_became_peered":"2024-09-20T22:41:44.194551+0000","last_unstale":"2024-09-20T22:44:53.139673+0000","last_undegraded":"2024-09-20T22:44:53.139673+0000","last_fullsized":"2024-09-20T22:44:53.139673+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:44:39.660039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:44:39.660039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:50.176169+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.12","version":"210'198","reported_seq":564,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.147447+0000","last_change":"2024-09-20T22:44:53.149257+0000","last_active":"2024-09-20T22:44:53.147447+0000","last_peered":"2024-09-20T22:44:53.147447+0000","last_clean":"2024-09-20T22:44:53.147447+0000","last_became_active":"2024-09-20T22:43:39.135211+0000","last_became_peered":"2024-09-20T22:43:39.135211+0000","last_unstale":"2024-09-20T22:44:53.147447+0000","last_undegraded":"2024-09-20T22:44:53.147447+0000","last_fullsized":"2024-09-20T22:44:53.147447+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":5,"last_scrub":"147'95","last_scrub_stamp":"2024-09-20T22:43:46.419480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:43:46.419480+0000","objects_scrubbed":3,"log_size":198,"log_dups_size":0,"ondisk_log_size":198,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:15.961913+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0020038289999999999,"stat_sum":{"num_bytes":5882483,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":57,"num_read_kb":72901,"num_write":43,"num_write_kb":14081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"37","length":"1"},{"start":"3b","length":"1"}]},{"pgid":"1.14","version":"77'128","reported_seq":574,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.150818+0000","last_change":"2024-09-20T22:44:00.146043+0000","last_active":"2024-09-20T22:44:53.150818+0000","last_peered":"2024-09-20T22:44:53.150818+0000","last_clean":"2024-09-20T22:44:53.150818+0000","last_became_active":"2024-09-20T22:44:00.145687+0000","last_became_peered":"2024-09-20T22:44:00.145687+0000","last_unstale":"2024-09-20T22:44:53.150818+0000","last_undegraded":"2024-09-20T22:44:53.150818+0000","last_fullsized":"2024-09-20T22:44:53.150818+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:43:29.331919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:43:29.331919+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:54.488310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"74'11","reported_seq":119,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.617224+0000","last_change":"2024-09-20T22:44:14.705392+0000","last_active":"2024-09-20T22:44:53.617224+0000","last_peered":"2024-09-20T22:44:53.617224+0000","last_clean":"2024-09-20T22:44:53.617224+0000","last_became_active":"2024-09-20T22:44:09.170785+0000","last_became_peered":"2024-09-20T22:44:09.170785+0000","last_unstale":"2024-09-20T22:44:53.617224+0000","last_undegraded":"2024-09-20T22:44:53.617224+0000","last_fullsized":"2024-09-20T22:44:53.617224+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-09-20T22:44:14.705306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:44:14.705306+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:35.089571+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":314,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.618456+0000","last_change":"2024-09-20T22:43:06.775869+0000","last_active":"2024-09-20T22:44:53.618456+0000","last_peered":"2024-09-20T22:44:53.618456+0000","last_clean":"2024-09-20T22:44:53.618456+0000","last_became_active":"2024-09-20T22:41:38.158701+0000","last_became_peered":"2024-09-20T22:41:38.158701+0000","last_unstale":"2024-09-20T22:44:53.618456+0000","last_undegraded":"2024-09-20T22:44:53.618456+0000","last_fullsized":"2024-09-20T22:44:53.618456+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:06.775781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:06.775781+0000","last_clean_scrub_stamp":"2024-09-20T22:43:06.775781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:54.631556+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":312,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.144308+0000","last_change":"2024-09-20T22:43:57.419543+0000","last_active":"2024-09-20T22:44:53.144308+0000","last_peered":"2024-09-20T22:44:53.144308+0000","last_clean":"2024-09-20T22:44:53.144308+0000","last_became_active":"2024-09-20T22:43:57.419200+0000","last_became_peered":"2024-09-20T22:43:57.419200+0000","last_unstale":"2024-09-20T22:44:53.144308+0000","last_undegraded":"2024-09-20T22:44:53.144308+0000","last_fullsized":"2024-09-20T22:44:53.144308+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:50.174861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:50.174861+0000","last_clean_scrub_stamp":"2024-09-20T22:43:50.174861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:18.151735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":294,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.144555+0000","last_change":"2024-09-20T22:41:44.832293+0000","last_active":"2024-09-20T22:44:53.144555+0000","last_peered":"2024-09-20T22:44:53.144555+0000","last_clean":"2024-09-20T22:44:53.144555+0000","last_became_active":"2024-09-20T22:41:44.831967+0000","last_became_peered":"2024-09-20T22:41:44.831967+0000","last_unstale":"2024-09-20T22:44:53.144555+0000","last_undegraded":"2024-09-20T22:44:53.144555+0000","last_fullsized":"2024-09-20T22:44:53.144555+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:55.019699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"188'19","reported_seq":435,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.140369+0000","last_change":"2024-09-20T22:44:53.141091+0000","last_active":"2024-09-20T22:44:53.140369+0000","last_peered":"2024-09-20T22:44:53.140369+0000","last_clean":"2024-09-20T22:44:53.140369+0000","last_became_active":"2024-09-20T22:44:09.177614+0000","last_became_peered":"2024-09-20T22:44:09.177614+0000","last_unstale":"2024-09-20T22:44:53.140369+0000","last_undegraded":"2024-09-20T22:44:53.140369+0000","last_fullsized":"2024-09-20T22:44:53.140369+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":5,"last_scrub":"147'18","last_scrub_stamp":"2024-09-20T22:43:48.759471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:43:48.759471+0000","objects_scrubbed":1,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:06.937544+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0023375100000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":2020747,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"37","length":"1"},{"start":"3b","length":"1"}]},{"pgid":"4.10","version":"74'4","reported_seq":316,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.143714+0000","last_change":"2024-09-20T22:44:17.320732+0000","last_active":"2024-09-20T22:44:53.143714+0000","last_peered":"2024-09-20T22:44:53.143714+0000","last_clean":"2024-09-20T22:44:53.143714+0000","last_became_active":"2024-09-20T22:41:40.166123+0000","last_became_peered":"2024-09-20T22:41:40.166123+0000","last_unstale":"2024-09-20T22:44:53.143714+0000","last_undegraded":"2024-09-20T22:44:53.143714+0000","last_fullsized":"2024-09-20T22:44:53.143714+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'4","last_scrub_stamp":"2024-09-20T22:44:17.320608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:44:17.320608+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:17.670947+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":314,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.140842+0000","last_change":"2024-09-20T22:43:03.680628+0000","last_active":"2024-09-20T22:44:53.140842+0000","last_peered":"2024-09-20T22:44:53.140842+0000","last_clean":"2024-09-20T22:44:53.140842+0000","last_became_active":"2024-09-20T22:41:38.158347+0000","last_became_peered":"2024-09-20T22:41:38.158347+0000","last_unstale":"2024-09-20T22:44:53.140842+0000","last_undegraded":"2024-09-20T22:44:53.140842+0000","last_fullsized":"2024-09-20T22:44:53.140842+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:03.680528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:43:03.680528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":312,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.144398+0000","last_change":"2024-09-20T22:43:57.420261+0000","last_active":"2024-09-20T22:44:53.144398+0000","last_peered":"2024-09-20T22:44:53.144398+0000","last_clean":"2024-09-20T22:44:53.144398+0000","last_became_active":"2024-09-20T22:43:57.419908+0000","last_became_peered":"2024-09-20T22:43:57.419908+0000","last_unstale":"2024-09-20T22:44:53.144398+0000","last_undegraded":"2024-09-20T22:44:53.144398+0000","last_fullsized":"2024-09-20T22:44:53.144398+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:06.821310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:43:06.821310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:54.611743+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":308,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.143911+0000","last_change":"2024-09-20T22:43:57.425704+0000","last_active":"2024-09-20T22:44:53.143911+0000","last_peered":"2024-09-20T22:44:53.143911+0000","last_clean":"2024-09-20T22:44:53.143911+0000","last_became_active":"2024-09-20T22:43:57.424654+0000","last_became_peered":"2024-09-20T22:43:57.424654+0000","last_unstale":"2024-09-20T22:44:53.143911+0000","last_undegraded":"2024-09-20T22:44:53.143911+0000","last_fullsized":"2024-09-20T22:44:53.143911+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:10.233487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:10.233487+0000","last_clean_scrub_stamp":"2024-09-20T22:43:10.233487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":308,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.140612+0000","last_change":"2024-09-20T22:44:21.123125+0000","last_active":"2024-09-20T22:44:53.140612+0000","last_peered":"2024-09-20T22:44:53.140612+0000","last_clean":"2024-09-20T22:44:53.140612+0000","last_became_active":"2024-09-20T22:43:57.430933+0000","last_became_peered":"2024-09-20T22:43:57.430933+0000","last_unstale":"2024-09-20T22:44:53.140612+0000","last_undegraded":"2024-09-20T22:44:53.140612+0000","last_fullsized":"2024-09-20T22:44:53.140612+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:44:21.123026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:44:21.123026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:36.736823+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":322,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.144067+0000","last_change":"2024-09-20T22:44:10.267253+0000","last_active":"2024-09-20T22:44:53.144067+0000","last_peered":"2024-09-20T22:44:53.144067+0000","last_clean":"2024-09-20T22:44:53.144067+0000","last_became_active":"2024-09-20T22:41:38.154703+0000","last_became_peered":"2024-09-20T22:41:38.154703+0000","last_unstale":"2024-09-20T22:44:53.144067+0000","last_undegraded":"2024-09-20T22:44:53.144067+0000","last_fullsized":"2024-09-20T22:44:53.144067+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:44:10.267152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:44:10.267152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:28.386412+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"74'11","reported_seq":329,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.148518+0000","last_change":"2024-09-20T22:43:28.500040+0000","last_active":"2024-09-20T22:44:53.148518+0000","last_peered":"2024-09-20T22:44:53.148518+0000","last_clean":"2024-09-20T22:44:53.148518+0000","last_became_active":"2024-09-20T22:41:40.171256+0000","last_became_peered":"2024-09-20T22:41:40.171256+0000","last_unstale":"2024-09-20T22:44:53.148518+0000","last_undegraded":"2024-09-20T22:44:53.148518+0000","last_fullsized":"2024-09-20T22:44:53.148518+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-09-20T22:43:28.499961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:43:28.499961+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:44:54.611760+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":306,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-20T22:44:53.141491+0000","last_change":"2024-09-20T22:43:51.667414+0000","last_active":"2024-09-20T22:44:53.141491+0000","last_peered":"2024-09-20T22:44:53.141491+0000","last_clean":"2024-09-20T22:44:53.141491+0000","last_became_active":"2024-09-20T22:41:42.178157+0000","last_became_peered":"2024-09-20T22:41:42.178157+0000","last_unstale":"2024-09-20T22:44:53.141491+0000","last_undegraded":"2024-09-20T22:44:53.141491+0000","last_fullsized":"2024-09-20T22:44:53.141491+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:51.667341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:51.667341+0000","last_clean_scrub_stamp":"2024-09-20T22:43:51.667341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:07.821301+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":26,"stat_sum":{"num_bytes":118238153,"num_objects":48,"num_object_clones":10,"num_object_copies":144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":5,"num_read":994,"num_read_kb":1021251,"num_write":685,"num_write_kb":275412,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":5835903,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":305557504,"data_stored":315418691,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2319,"ondisk_log_size":2319,"up":78,"acting":78,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1261,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":69,"ondisk_log_size":69,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":686,"num_read_kb":443,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":1914,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":471040,"data_stored":12580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":427,"ondisk_log_size":427,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":5023,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":190,"num_read_kb":185,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2688,"ondisk_log_size":2688,"up":63,"acting":63,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070507,"num_pgs":77,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":86804,"kb_used_data":51452,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93658348,"statfs":{"total":95995035648,"available":95906148352,"internally_reserved":0,"allocated":52686848,"data_stored":53685026,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430257,"num_pgs":75,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":90500,"kb_used_data":55152,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93654652,"statfs":{"total":95995035648,"available":95902363648,"internally_reserved":0,"allocated":56475648,"data_stored":56594725,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691894,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":66092,"kb_used_data":34900,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93679060,"statfs":{"total":95995035648,"available":95927357440,"internally_reserved":0,"allocated":35737600,"data_stored":34450954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":0,"seq":133143986235,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182465,"num_pgs":83,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":88000,"kb_used_data":52644,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93657152,"statfs":{"total":95995035648,"available":95904923648,"internally_reserved":0,"allocated":53907456,"data_stored":53909802,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411398,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":77872,"kb_used_data":42520,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93667280,"statfs":{"total":95995035648,"available":95915294720,"internally_reserved":0,"allocated":43540480,"data_stored":43113396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574923,"num_pgs":79,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":61056,"kb_used_data":25708,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93684096,"statfs":{"total":95995035648,"available":95932514304,"internally_reserved":0,"allocated":26324992,"data_stored":25072962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738448,"num_pgs":79,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":77080,"kb_used_data":37564,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93668072,"statfs":{"total":95995035648,"available":95916105728,"internally_reserved":0,"allocated":38465536,"data_stored":39426661,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":36966400,"data_stored":39143043,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":23953408,"data_stored":23806698,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40935424,"data_stored":41577409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":53047296,"data_stored":55699092,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11284480,"data_stored":11212109,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":33140736,"data_stored":32921638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":53796864,"data_stored":54993278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":52432896,"data_stored":56065424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-20T22:44:54.845 INFO:tasks.thrashosds.thrasher:Setting 3.18 to [4, 1, 0, 7, 2, 3] 2024-09-20T22:44:54.845 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '3.18', '4', '1', '0', '7', '2', '3'] 2024-09-20T22:44:54.845 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd pg-upmap-items 3.18 4 1 0 7 2 3 2024-09-20T22:44:54.910 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:44:54.932 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:54.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:54.933 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:44:55.034 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:44:55.055 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:55.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:55.056 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:44:55.095 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-20T22:44:55.157 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:44:55.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:55.179 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:55.179 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:44:55.189 INFO:tasks.rados.rados.0.smithi084.stdout:645: done (2 left) 2024-09-20T22:44:55.189 INFO:tasks.rados.rados.0.smithi084.stdout:646: done (1 left) 2024-09-20T22:44:55.189 INFO:tasks.rados.rados.0.smithi084.stdout:648: done (0 left) 2024-09-20T22:44:55.189 INFO:tasks.rados.rados.0.smithi084.stdout:649: read oid 10 snap 55 2024-09-20T22:44:55.189 INFO:tasks.rados.rados.0.smithi084.stdout:649: expect deleted 2024-09-20T22:44:55.189 INFO:tasks.rados.rados.0.smithi084.stdout:650: snap_remove snap 59 2024-09-20T22:44:55.280 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:44:55.302 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:55.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:55.303 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:44:55.307 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:44:55.404 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:44:55.427 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:55.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:55.428 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:44:55.431 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:55 smithi084 ceph-mon[40024]: pgmap v223: 178 pgs: 1 active+clean+snaptrim, 177 active+clean; 113 MiB data, 535 MiB used, 625 GiB / 626 GiB avail; 5.0 MiB/s rd, 1.7 MiB/s wr, 11 op/s 2024-09-20T22:44:55.431 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:55 smithi084 ceph-mon[40024]: from='client.15189 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T22:44:55.431 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:55 smithi084 ceph-mon[40024]: 6.4 scrub starts 2024-09-20T22:44:55.431 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:55 smithi084 ceph-mon[40024]: 6.4 scrub ok 2024-09-20T22:44:55.431 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:55 smithi084 ceph-mon[40024]: 5.1a scrub starts 2024-09-20T22:44:55.431 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:55 smithi084 ceph-mon[40024]: 5.1a scrub ok 2024-09-20T22:44:55.431 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:55 smithi084 ceph-mon[40024]: osdmap e215: 8 total, 8 up, 7 in 2024-09-20T22:44:55.431 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:55 smithi084 ceph-mon[29774]: pgmap v223: 178 pgs: 1 active+clean+snaptrim, 177 active+clean; 113 MiB data, 535 MiB used, 625 GiB / 626 GiB avail; 5.0 MiB/s rd, 1.7 MiB/s wr, 11 op/s 2024-09-20T22:44:55.431 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:55 smithi084 ceph-mon[29774]: from='client.15189 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T22:44:55.431 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:55 smithi084 ceph-mon[29774]: 6.4 scrub starts 2024-09-20T22:44:55.432 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:55 smithi084 ceph-mon[29774]: 6.4 scrub ok 2024-09-20T22:44:55.432 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:55 smithi084 ceph-mon[29774]: 5.1a scrub starts 2024-09-20T22:44:55.432 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:55 smithi084 ceph-mon[29774]: 5.1a scrub ok 2024-09-20T22:44:55.432 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:55 smithi084 ceph-mon[29774]: osdmap e215: 8 total, 8 up, 7 in 2024-09-20T22:44:55.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:55 smithi154 ceph-mon[33264]: pgmap v223: 178 pgs: 1 active+clean+snaptrim, 177 active+clean; 113 MiB data, 535 MiB used, 625 GiB / 626 GiB avail; 5.0 MiB/s rd, 1.7 MiB/s wr, 11 op/s 2024-09-20T22:44:55.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:55 smithi154 ceph-mon[33264]: from='client.15189 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T22:44:55.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:55 smithi154 ceph-mon[33264]: 6.4 scrub starts 2024-09-20T22:44:55.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:55 smithi154 ceph-mon[33264]: 6.4 scrub ok 2024-09-20T22:44:55.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:55 smithi154 ceph-mon[33264]: 5.1a scrub starts 2024-09-20T22:44:55.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:55 smithi154 ceph-mon[33264]: 5.1a scrub ok 2024-09-20T22:44:55.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:55 smithi154 ceph-mon[33264]: osdmap e215: 8 total, 8 up, 7 in 2024-09-20T22:44:55.529 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:44:55.551 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:55.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:55.552 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:44:55.653 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:44:55.674 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:55.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:55.675 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:44:55.776 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:44:55.776 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:44:55.776 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:44:55.786 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-20T22:44:55.786 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2024-09-20T22:44:55.786 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2024-09-20T22:44:55.786 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2024-09-20T22:44:55.786 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-20T22:44:55.803 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:55.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:55.804 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:44:55.904 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:44:55.927 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:55.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:55.928 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:44:56.028 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:44:56.050 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:56.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:56.051 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:44:56.152 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:44:56.174 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:56.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:56.175 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:44:56.202 INFO:tasks.rados.rados.0.smithi084.stdout:650: done (1 left) 2024-09-20T22:44:56.202 INFO:tasks.rados.rados.0.smithi084.stdout:651: delete oid 7 current snap is 61 2024-09-20T22:44:56.204 INFO:tasks.rados.rados.0.smithi084.stdout:649: done (1 left) 2024-09-20T22:44:56.204 INFO:tasks.rados.rados.0.smithi084.stdout:651: done (0 left) 2024-09-20T22:44:56.204 INFO:tasks.rados.rados.0.smithi084.stdout:652: write oid 7 current snap is 61 2024-09-20T22:44:56.204 INFO:tasks.rados.rados.0.smithi084.stdout:652: seq_num 260 ranges {769820=720742,2114730=255090} 2024-09-20T22:44:56.218 INFO:tasks.rados.rados.0.smithi084.stdout:652: writing smithi08477203-7 from 769820 to 1490562 tid 1 2024-09-20T22:44:56.218 INFO:tasks.rados.rados.0.smithi084.stdout:652: writing smithi08477203-7 from 2114730 to 2369820 tid 2 2024-09-20T22:44:56.218 INFO:tasks.rados.rados.0.smithi084.stdout:653: rollback oid 25 current snap is 61 2024-09-20T22:44:56.218 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 25 to 56 2024-09-20T22:44:56.219 INFO:tasks.rados.rados.0.smithi084.stdout:652: finishing write tid 1 to smithi08477203-7 2024-09-20T22:44:56.219 INFO:tasks.rados.rados.0.smithi084.stdout:654: read oid 38 snap -1 2024-09-20T22:44:56.219 INFO:tasks.rados.rados.0.smithi084.stdout:654: expect deleted 2024-09-20T22:44:56.219 INFO:tasks.rados.rados.0.smithi084.stdout:655: copy_from oid 21 from oid 2 current snap is 61 2024-09-20T22:44:56.219 INFO:tasks.rados.rados.0.smithi084.stdout:656: copy_from oid 39 from oid 3 current snap is 61 2024-09-20T22:44:56.219 INFO:tasks.rados.rados.0.smithi084.stdout:657: snap_create 2024-09-20T22:44:56.220 INFO:tasks.rados.rados.0.smithi084.stdout:652: finishing write tid 2 to smithi08477203-7 2024-09-20T22:44:56.220 INFO:tasks.rados.rados.0.smithi084.stdout:652: finishing write tid 3 to smithi08477203-7 2024-09-20T22:44:56.220 INFO:tasks.rados.rados.0.smithi084.stdout:652: finishing write tid 4 to smithi08477203-7 2024-09-20T22:44:56.220 INFO:tasks.rados.rados.0.smithi084.stdout:652: oid 7 updating version 0 to 126 2024-09-20T22:44:56.220 INFO:tasks.rados.rados.0.smithi084.stdout:652: oid 7 version 126 is already newer than 125 2024-09-20T22:44:56.220 INFO:tasks.rados.rados.0.smithi084.stdout:652: oid 7 updating version 126 to 127 2024-09-20T22:44:56.220 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 7 v 127 (ObjNum 260 snap 61 seq_num 260) dirty exists 2024-09-20T22:44:56.220 INFO:tasks.rados.rados.0.smithi084.stdout:652: left oid 7 (ObjNum 260 snap 61 seq_num 260) 2024-09-20T22:44:56.224 INFO:tasks.rados.rados.0.smithi084.stdout:656: finishing copy_from racing read to smithi08477203-39 2024-09-20T22:44:56.225 INFO:tasks.rados.rados.0.smithi084.stdout:653: finishing rollback tid 0 to smithi08477203-25 2024-09-20T22:44:56.226 INFO:tasks.rados.rados.0.smithi084.stdout:655: finishing copy_from to smithi08477203-21 2024-09-20T22:44:56.226 INFO:tasks.rados.rados.0.smithi084.stdout:655: got expected ENOENT (src dne) 2024-09-20T22:44:56.226 INFO:tasks.rados.rados.0.smithi084.stdout:653: finishing rollback tid 1 to smithi08477203-25 2024-09-20T22:44:56.226 INFO:tasks.rados.rados.0.smithi084.stdout:655: finishing copy_from racing read to smithi08477203-21 2024-09-20T22:44:56.226 INFO:tasks.rados.rados.0.smithi084.stdout:653: finishing rollback tid 2 to smithi08477203-25 2024-09-20T22:44:56.227 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 25 v 290 (ObjNum 234 snap 51 seq_num 234) dirty exists 2024-09-20T22:44:56.229 INFO:tasks.rados.rados.0.smithi084.stdout:656: finishing copy_from to smithi08477203-39 2024-09-20T22:44:56.229 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 39 v 101 (ObjNum 245 snap 54 seq_num 245) dirty exists 2024-09-20T22:44:56.277 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:44:56.298 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:56.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:56.299 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:44:56.364 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:56 smithi084 ceph-mon[40024]: 4.12 scrub starts 2024-09-20T22:44:56.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:56 smithi084 ceph-mon[40024]: 4.12 scrub ok 2024-09-20T22:44:56.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:56 smithi084 ceph-mon[40024]: osdmap e216: 8 total, 8 up, 7 in 2024-09-20T22:44:56.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:56 smithi084 ceph-mon[40024]: 4.1c scrub starts 2024-09-20T22:44:56.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:56 smithi084 ceph-mon[40024]: 4.1c scrub ok 2024-09-20T22:44:56.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:56 smithi084 ceph-mon[29774]: 4.12 scrub starts 2024-09-20T22:44:56.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:56 smithi084 ceph-mon[29774]: 4.12 scrub ok 2024-09-20T22:44:56.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:56 smithi084 ceph-mon[29774]: osdmap e216: 8 total, 8 up, 7 in 2024-09-20T22:44:56.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:56 smithi084 ceph-mon[29774]: 4.1c scrub starts 2024-09-20T22:44:56.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:56 smithi084 ceph-mon[29774]: 4.1c scrub ok 2024-09-20T22:44:56.400 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:44:56.422 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:56.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:56.423 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:44:56.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:56 smithi154 ceph-mon[33264]: 4.12 scrub starts 2024-09-20T22:44:56.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:56 smithi154 ceph-mon[33264]: 4.12 scrub ok 2024-09-20T22:44:56.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:56 smithi154 ceph-mon[33264]: osdmap e216: 8 total, 8 up, 7 in 2024-09-20T22:44:56.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:56 smithi154 ceph-mon[33264]: 4.1c scrub starts 2024-09-20T22:44:56.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:56 smithi154 ceph-mon[33264]: 4.1c scrub ok 2024-09-20T22:44:56.524 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:44:56.545 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:56.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:56.546 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:44:56.647 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:44:56.667 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:56.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:56.668 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:44:56.769 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:44:56.790 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:56.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:56.791 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:44:56.892 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:44:56.914 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:56.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:56.914 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:44:57.015 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:44:57.036 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:57.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:57.037 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:44:57.138 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:44:57.159 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:57.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:57.160 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:44:57.195 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is unset 2024-09-20T22:44:57.198 INFO:tasks.rados.rados.0.smithi084.stdout:652: done (5 left) 2024-09-20T22:44:57.198 INFO:tasks.rados.rados.0.smithi084.stdout:653: done (4 left) 2024-09-20T22:44:57.199 INFO:tasks.rados.rados.0.smithi084.stdout:654: done (3 left) 2024-09-20T22:44:57.199 INFO:tasks.rados.rados.0.smithi084.stdout:655: done (2 left) 2024-09-20T22:44:57.199 INFO:tasks.rados.rados.0.smithi084.stdout:656: done (1 left) 2024-09-20T22:44:57.199 INFO:tasks.rados.rados.0.smithi084.stdout:657: done (0 left) 2024-09-20T22:44:57.199 INFO:tasks.rados.rados.0.smithi084.stdout:658: copy_from oid 1 from oid 42 current snap is 62 2024-09-20T22:44:57.199 INFO:tasks.rados.rados.0.smithi084.stdout:659: read oid 44 snap -1 2024-09-20T22:44:57.199 INFO:tasks.rados.rados.0.smithi084.stdout:659: expect (ObjNum 129 snap 23 seq_num 129) 2024-09-20T22:44:57.199 INFO:tasks.rados.rados.0.smithi084.stdout:660: snap_create 2024-09-20T22:44:57.201 INFO:tasks.rados.rados.0.smithi084.stdout:658: finishing copy_from to smithi08477203-1 2024-09-20T22:44:57.201 INFO:tasks.rados.rados.0.smithi084.stdout:658: got expected ENOENT (src dne) 2024-09-20T22:44:57.201 INFO:tasks.rados.rados.0.smithi084.stdout:658: finishing copy_from racing read to smithi08477203-1 2024-09-20T22:44:57.209 INFO:tasks.rados.rados.0.smithi084.stdout:659: expect (ObjNum 129 snap 23 seq_num 129) 2024-09-20T22:44:57.261 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:44:57.276 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-20T22:44:57.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:57.282 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:57.283 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:44:57.384 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:44:57.405 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:57.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:57.406 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:44:57.463 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:57 smithi084 ceph-mon[40024]: purged_snaps scrub starts 2024-09-20T22:44:57.463 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:57 smithi084 ceph-mon[40024]: purged_snaps scrub ok 2024-09-20T22:44:57.463 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:57 smithi084 ceph-mon[40024]: 5.e scrub starts 2024-09-20T22:44:57.463 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:57 smithi084 ceph-mon[40024]: 5.e scrub ok 2024-09-20T22:44:57.464 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:57 smithi084 ceph-mon[40024]: 4.a scrub starts 2024-09-20T22:44:57.464 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:57 smithi084 ceph-mon[40024]: 4.a scrub ok 2024-09-20T22:44:57.464 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:57 smithi084 ceph-mon[40024]: pgmap v226: 178 pgs: 1 active+clean+snaptrim, 177 active+clean; 113 MiB data, 535 MiB used, 625 GiB / 626 GiB avail; 5.0 MiB/s rd, 1.7 MiB/s wr, 11 op/s 2024-09-20T22:44:57.464 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:57 smithi084 ceph-mon[40024]: 4.f scrub starts 2024-09-20T22:44:57.464 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:57 smithi084 ceph-mon[40024]: 4.f scrub ok 2024-09-20T22:44:57.464 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:57 smithi084 ceph-mon[40024]: osdmap e217: 8 total, 8 up, 7 in 2024-09-20T22:44:57.464 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:57 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/1746452770' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:44:57.464 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:57 smithi084 ceph-mon[29774]: purged_snaps scrub starts 2024-09-20T22:44:57.464 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:57 smithi084 ceph-mon[29774]: purged_snaps scrub ok 2024-09-20T22:44:57.464 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:57 smithi084 ceph-mon[29774]: 5.e scrub starts 2024-09-20T22:44:57.464 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:57 smithi084 ceph-mon[29774]: 5.e scrub ok 2024-09-20T22:44:57.465 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:57 smithi084 ceph-mon[29774]: 4.a scrub starts 2024-09-20T22:44:57.465 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:57 smithi084 ceph-mon[29774]: 4.a scrub ok 2024-09-20T22:44:57.465 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:57 smithi084 ceph-mon[29774]: pgmap v226: 178 pgs: 1 active+clean+snaptrim, 177 active+clean; 113 MiB data, 535 MiB used, 625 GiB / 626 GiB avail; 5.0 MiB/s rd, 1.7 MiB/s wr, 11 op/s 2024-09-20T22:44:57.465 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:57 smithi084 ceph-mon[29774]: 4.f scrub starts 2024-09-20T22:44:57.465 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:57 smithi084 ceph-mon[29774]: 4.f scrub ok 2024-09-20T22:44:57.465 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:57 smithi084 ceph-mon[29774]: osdmap e217: 8 total, 8 up, 7 in 2024-09-20T22:44:57.465 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:57 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/1746452770' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:44:57.496 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.0 dump_historic_ops 2024-09-20T22:44:57.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:57 smithi154 ceph-mon[33264]: purged_snaps scrub starts 2024-09-20T22:44:57.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:57 smithi154 ceph-mon[33264]: purged_snaps scrub ok 2024-09-20T22:44:57.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:57 smithi154 ceph-mon[33264]: 5.e scrub starts 2024-09-20T22:44:57.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:57 smithi154 ceph-mon[33264]: 5.e scrub ok 2024-09-20T22:44:57.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:57 smithi154 ceph-mon[33264]: 4.a scrub starts 2024-09-20T22:44:57.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:57 smithi154 ceph-mon[33264]: 4.a scrub ok 2024-09-20T22:44:57.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:57 smithi154 ceph-mon[33264]: pgmap v226: 178 pgs: 1 active+clean+snaptrim, 177 active+clean; 113 MiB data, 535 MiB used, 625 GiB / 626 GiB avail; 5.0 MiB/s rd, 1.7 MiB/s wr, 11 op/s 2024-09-20T22:44:57.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:57 smithi154 ceph-mon[33264]: 4.f scrub starts 2024-09-20T22:44:57.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:57 smithi154 ceph-mon[33264]: 4.f scrub ok 2024-09-20T22:44:57.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:57 smithi154 ceph-mon[33264]: osdmap e217: 8 total, 8 up, 7 in 2024-09-20T22:44:57.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:57 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/1746452770' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:44:57.507 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:44:57.559 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:57.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:57.560 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:44:57.661 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:44:57.682 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:57.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:57.683 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:44:57.784 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:44:57.805 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:57.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:57.806 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:44:57.907 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:44:57.929 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:57.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:57.930 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:44:58.030 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:44:58.052 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:58.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:58.053 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:44:58.144 INFO:teuthology.orchestra.run.smithi084.stderr:set 3.18 pg_upmap_items mapping to [4->1,0->7,2->3] 2024-09-20T22:44:58.144 INFO:tasks.rados.rados.0.smithi084.stdout:658: done (2 left) 2024-09-20T22:44:58.144 INFO:tasks.rados.rados.0.smithi084.stdout:659: done (1 left) 2024-09-20T22:44:58.144 INFO:tasks.rados.rados.0.smithi084.stdout:660: done (0 left) 2024-09-20T22:44:58.144 INFO:tasks.rados.rados.0.smithi084.stdout:661: snap_create 2024-09-20T22:44:58.154 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:44:58.176 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:58.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:58.178 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:44:58.279 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:44:58.300 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:58.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:58.301 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:44:58.401 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:44:58.422 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:58.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:58.423 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:44:58.446 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:44:58 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: ::ffff:172.21.15.154 - - [20/Sep/2024:22:44:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-20T22:44:58.447 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:58 smithi084 ceph-mon[29774]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T22:44:58.447 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:58 smithi084 ceph-mon[29774]: Cluster is now healthy 2024-09-20T22:44:58.447 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:58 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/763443628' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "3.18", "id": [4, 1, 0, 7, 2, 3]}]: dispatch 2024-09-20T22:44:58.447 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:58 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "3.18", "id": [4, 1, 0, 7, 2, 3]}]: dispatch 2024-09-20T22:44:58.447 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:58 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/1746452770' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T22:44:58.447 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:58 smithi084 ceph-mon[29774]: osdmap e218: 8 total, 8 up, 7 in 2024-09-20T22:44:58.447 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:58 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "3.18", "id": [4, 1, 0, 7, 2, 3]}]': finished 2024-09-20T22:44:58.447 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:58 smithi084 ceph-mon[29774]: osdmap e219: 8 total, 8 up, 7 in 2024-09-20T22:44:58.447 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:58 smithi084 ceph-mon[40024]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T22:44:58.447 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:58 smithi084 ceph-mon[40024]: Cluster is now healthy 2024-09-20T22:44:58.447 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:58 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/763443628' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "3.18", "id": [4, 1, 0, 7, 2, 3]}]: dispatch 2024-09-20T22:44:58.448 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:58 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "3.18", "id": [4, 1, 0, 7, 2, 3]}]: dispatch 2024-09-20T22:44:58.448 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:58 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/1746452770' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T22:44:58.448 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:58 smithi084 ceph-mon[40024]: osdmap e218: 8 total, 8 up, 7 in 2024-09-20T22:44:58.448 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:58 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "3.18", "id": [4, 1, 0, 7, 2, 3]}]': finished 2024-09-20T22:44:58.448 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:58 smithi084 ceph-mon[40024]: osdmap e219: 8 total, 8 up, 7 in 2024-09-20T22:44:58.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:58 smithi154 ceph-mon[33264]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T22:44:58.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:58 smithi154 ceph-mon[33264]: Cluster is now healthy 2024-09-20T22:44:58.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:58 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/763443628' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "3.18", "id": [4, 1, 0, 7, 2, 3]}]: dispatch 2024-09-20T22:44:58.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:58 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "3.18", "id": [4, 1, 0, 7, 2, 3]}]: dispatch 2024-09-20T22:44:58.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:58 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/1746452770' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T22:44:58.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:58 smithi154 ceph-mon[33264]: osdmap e218: 8 total, 8 up, 7 in 2024-09-20T22:44:58.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:58 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "3.18", "id": [4, 1, 0, 7, 2, 3]}]': finished 2024-09-20T22:44:58.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:58 smithi154 ceph-mon[33264]: osdmap e219: 8 total, 8 up, 7 in 2024-09-20T22:44:58.524 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:44:58.545 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:58.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:58.546 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:44:58.647 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:44:58.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:58.669 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:58.669 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:44:58.770 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:44:58.791 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:58.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:58.792 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:44:58.805 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:44:58.892 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:44:58.913 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:58.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:58.914 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:44:59.015 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:44:59.036 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:44:59.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:59.037 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:44:59.138 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:44:59.148 INFO:tasks.rados.rados.0.smithi084.stdout:661: done (0 left) 2024-09-20T22:44:59.148 INFO:tasks.rados.rados.0.smithi084.stdout:662: read oid 27 snap -1 2024-09-20T22:44:59.148 INFO:tasks.rados.rados.0.smithi084.stdout:662: expect (ObjNum 234 snap 51 seq_num 234) 2024-09-20T22:44:59.148 INFO:tasks.rados.rados.0.smithi084.stdout:663: write oid 49 current snap is 64 2024-09-20T22:44:59.148 INFO:tasks.rados.rados.0.smithi084.stdout:663: seq_num 261 ranges {786627=795991,2173212=213415} 2024-09-20T22:44:59.156 INFO:tasks.rados.rados.0.smithi084.stdout:663: writing smithi08477203-49 from 786627 to 1582618 tid 1 2024-09-20T22:44:59.161 INFO:tasks.rados.rados.0.smithi084.stdout:663: writing smithi08477203-49 from 2173212 to 2386627 tid 2 2024-09-20T22:44:59.162 INFO:tasks.rados.rados.0.smithi084.stdout:662: expect (ObjNum 234 snap 51 seq_num 234) 2024-09-20T22:44:59.164 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:59.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:59.165 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:44:59.266 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:44:59.275 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:44:59.276 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:44:59.278 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:44:59.279 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:44:59.289 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:59.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:59.290 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:44:59.290 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:44:59.306 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:44:59.307 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:44:59.307 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:59 smithi084 ceph-mon[40024]: 4.b scrub starts 2024-09-20T22:44:59.307 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:59 smithi084 ceph-mon[40024]: 4.b scrub ok 2024-09-20T22:44:59.307 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:59 smithi084 ceph-mon[40024]: 5.f scrub starts 2024-09-20T22:44:59.307 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:59 smithi084 ceph-mon[40024]: 5.18 scrub starts 2024-09-20T22:44:59.307 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:59 smithi084 ceph-mon[40024]: 5.f scrub ok 2024-09-20T22:44:59.307 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:59 smithi084 ceph-mon[40024]: 5.18 scrub ok 2024-09-20T22:44:59.307 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:59 smithi084 ceph-mon[40024]: pgmap v229: 178 pgs: 1 active+clean+snaptrim, 177 active+clean; 113 MiB data, 535 MiB used, 625 GiB / 626 GiB avail 2024-09-20T22:44:59.308 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:59 smithi084 ceph-mon[40024]: 1.2 scrub starts 2024-09-20T22:44:59.308 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:59 smithi084 ceph-mon[40024]: 1.2 scrub ok 2024-09-20T22:44:59.308 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:59 smithi084 ceph-mon[40024]: 6.14 scrub starts 2024-09-20T22:44:59.308 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:59 smithi084 ceph-mon[40024]: 6.14 scrub ok 2024-09-20T22:44:59.308 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:59 smithi084 ceph-mon[40024]: 4.2 scrub starts 2024-09-20T22:44:59.308 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:59 smithi084 ceph-mon[40024]: 4.2 scrub ok 2024-09-20T22:44:59.308 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:44:59 smithi084 ceph-mon[40024]: osdmap e220: 8 total, 8 up, 7 in 2024-09-20T22:44:59.308 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:59 smithi084 ceph-mon[29774]: 4.b scrub starts 2024-09-20T22:44:59.308 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:59 smithi084 ceph-mon[29774]: 4.b scrub ok 2024-09-20T22:44:59.308 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:59 smithi084 ceph-mon[29774]: 5.f scrub starts 2024-09-20T22:44:59.309 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:59 smithi084 ceph-mon[29774]: 5.18 scrub starts 2024-09-20T22:44:59.309 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:59 smithi084 ceph-mon[29774]: 5.f scrub ok 2024-09-20T22:44:59.309 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:59 smithi084 ceph-mon[29774]: 5.18 scrub ok 2024-09-20T22:44:59.309 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:59 smithi084 ceph-mon[29774]: pgmap v229: 178 pgs: 1 active+clean+snaptrim, 177 active+clean; 113 MiB data, 535 MiB used, 625 GiB / 626 GiB avail 2024-09-20T22:44:59.309 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:59 smithi084 ceph-mon[29774]: 1.2 scrub starts 2024-09-20T22:44:59.309 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:59 smithi084 ceph-mon[29774]: 1.2 scrub ok 2024-09-20T22:44:59.309 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:59 smithi084 ceph-mon[29774]: 6.14 scrub starts 2024-09-20T22:44:59.309 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:59 smithi084 ceph-mon[29774]: 6.14 scrub ok 2024-09-20T22:44:59.309 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:59 smithi084 ceph-mon[29774]: 4.2 scrub starts 2024-09-20T22:44:59.309 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:59 smithi084 ceph-mon[29774]: 4.2 scrub ok 2024-09-20T22:44:59.309 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:44:59 smithi084 ceph-mon[29774]: osdmap e220: 8 total, 8 up, 7 in 2024-09-20T22:44:59.335 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-20T22:44:59.335 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T22:44:59.366 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:44:59.366 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:44:59.391 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:44:59.394 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:44:59.394 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:44:59.411 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:44:59.411 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:44:59.412 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:59.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:59.413 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:44:59.428 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:44:59.428 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:44:59.482 INFO:tasks.rados.rados.0.smithi084.stdout:663: finishing write tid 1 to smithi08477203-49 2024-09-20T22:44:59.482 INFO:tasks.rados.rados.0.smithi084.stdout:663: finishing write tid 2 to smithi08477203-49 2024-09-20T22:44:59.482 INFO:tasks.rados.rados.0.smithi084.stdout:662: done (1 left) 2024-09-20T22:44:59.482 INFO:tasks.rados.rados.0.smithi084.stdout:664: delete oid 15 current snap is 64 2024-09-20T22:44:59.482 INFO:tasks.rados.rados.0.smithi084.stdout:663: finishing write tid 3 to smithi08477203-49 2024-09-20T22:44:59.482 INFO:tasks.rados.rados.0.smithi084.stdout:663: finishing write tid 4 to smithi08477203-49 2024-09-20T22:44:59.482 INFO:tasks.rados.rados.0.smithi084.stdout:663: oid 49 updating version 0 to 202 2024-09-20T22:44:59.483 INFO:tasks.rados.rados.0.smithi084.stdout:663: oid 49 version 202 is already newer than 201 2024-09-20T22:44:59.483 INFO:tasks.rados.rados.0.smithi084.stdout:663: oid 49 version 202 is already newer than 200 2024-09-20T22:44:59.483 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 49 v 202 (ObjNum 261 snap 64 seq_num 261) dirty exists 2024-09-20T22:44:59.483 INFO:tasks.rados.rados.0.smithi084.stdout:663: left oid 49 (ObjNum 261 snap 64 seq_num 261) 2024-09-20T22:44:59.483 INFO:tasks.rados.rados.0.smithi084.stdout:663: done (1 left) 2024-09-20T22:44:59.483 INFO:tasks.rados.rados.0.smithi084.stdout:664: done (0 left) 2024-09-20T22:44:59.483 INFO:tasks.rados.rados.0.smithi084.stdout:665: rollback oid 17 current snap is 64 2024-09-20T22:44:59.483 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 17 to 63 2024-09-20T22:44:59.483 INFO:tasks.rados.rados.0.smithi084.stdout:666: snap_remove snap 61 2024-09-20T22:44:59.504 INFO:journalctl@ceph.iscsi.iscsi.a.smithi154.stdout:Sep 20 22:44:59 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-iscsi-iscsi-a[59730]: debug there is no tcmu-runner data available 2024-09-20T22:44:59.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:59 smithi154 ceph-mon[33264]: 4.b scrub starts 2024-09-20T22:44:59.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:59 smithi154 ceph-mon[33264]: 4.b scrub ok 2024-09-20T22:44:59.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:59 smithi154 ceph-mon[33264]: 5.f scrub starts 2024-09-20T22:44:59.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:59 smithi154 ceph-mon[33264]: 5.18 scrub starts 2024-09-20T22:44:59.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:59 smithi154 ceph-mon[33264]: 5.f scrub ok 2024-09-20T22:44:59.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:59 smithi154 ceph-mon[33264]: 5.18 scrub ok 2024-09-20T22:44:59.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:59 smithi154 ceph-mon[33264]: pgmap v229: 178 pgs: 1 active+clean+snaptrim, 177 active+clean; 113 MiB data, 535 MiB used, 625 GiB / 626 GiB avail 2024-09-20T22:44:59.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:59 smithi154 ceph-mon[33264]: 1.2 scrub starts 2024-09-20T22:44:59.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:59 smithi154 ceph-mon[33264]: 1.2 scrub ok 2024-09-20T22:44:59.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:59 smithi154 ceph-mon[33264]: 6.14 scrub starts 2024-09-20T22:44:59.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:59 smithi154 ceph-mon[33264]: 6.14 scrub ok 2024-09-20T22:44:59.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:59 smithi154 ceph-mon[33264]: 4.2 scrub starts 2024-09-20T22:44:59.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:59 smithi154 ceph-mon[33264]: 4.2 scrub ok 2024-09-20T22:44:59.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:44:59 smithi154 ceph-mon[33264]: osdmap e220: 8 total, 8 up, 7 in 2024-09-20T22:44:59.514 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:44:59.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:59.535 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:59.535 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:44:59.636 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:44:59.657 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:59.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:59.658 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:44:59.759 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:44:59.780 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:59.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:59.781 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:44:59.882 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:44:59.903 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:44:59.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:44:59.904 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:45:00.005 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:45:00.027 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:00.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:00.028 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:45:00.129 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:45:00.151 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:00.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:00.153 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:45:00.203 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd set noscrub 2024-09-20T22:45:00.211 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:00 smithi084 ceph-mon[29774]: 3.17 scrub starts 2024-09-20T22:45:00.211 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:00 smithi084 ceph-mon[29774]: 3.17 scrub ok 2024-09-20T22:45:00.211 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:00 smithi084 ceph-mon[29774]: 3.1e scrub starts 2024-09-20T22:45:00.211 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:00 smithi084 ceph-mon[29774]: 3.1e scrub ok 2024-09-20T22:45:00.212 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:00 smithi084 ceph-mon[29774]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:45:00.212 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:00 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/449838387' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:45:00.216 INFO:tasks.rados.rados.0.smithi084.stdout:666: done (1 left) 2024-09-20T22:45:00.216 INFO:tasks.rados.rados.0.smithi084.stdout:667: read oid 13 snap -1 2024-09-20T22:45:00.216 INFO:tasks.rados.rados.0.smithi084.stdout:667: expect deleted 2024-09-20T22:45:00.216 INFO:tasks.rados.rados.0.smithi084.stdout:668: snap_remove snap 56 2024-09-20T22:45:00.253 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:45:00.277 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:00.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:00.278 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:45:00.378 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:45:00.401 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:00.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:00.402 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:45:00.449 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-20T22:45:00.449 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:45:00.459 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:00 smithi084 ceph-mon[40024]: 3.17 scrub starts 2024-09-20T22:45:00.459 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:00 smithi084 ceph-mon[40024]: 3.17 scrub ok 2024-09-20T22:45:00.459 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:00 smithi084 ceph-mon[40024]: 3.1e scrub starts 2024-09-20T22:45:00.460 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:00 smithi084 ceph-mon[40024]: 3.1e scrub ok 2024-09-20T22:45:00.460 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:00 smithi084 ceph-mon[40024]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:45:00.460 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:00 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/449838387' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:45:00.502 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:45:00.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:00 smithi154 ceph-mon[33264]: 3.17 scrub starts 2024-09-20T22:45:00.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:00 smithi154 ceph-mon[33264]: 3.17 scrub ok 2024-09-20T22:45:00.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:00 smithi154 ceph-mon[33264]: 3.1e scrub starts 2024-09-20T22:45:00.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:00 smithi154 ceph-mon[33264]: 3.1e scrub ok 2024-09-20T22:45:00.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:00 smithi154 ceph-mon[33264]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:45:00.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:00 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/449838387' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:45:00.524 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:00.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:00.525 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:45:00.626 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:45:00.647 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:00.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:00.648 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:45:00.719 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:45:00.749 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:45:00.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:00.771 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:00.771 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:45:00.872 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:45:00.894 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:00.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:00.895 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:45:00.996 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:45:01.018 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:01.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:01.019 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:45:01.119 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:45:01.141 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:01.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:01.142 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:45:01.212 INFO:tasks.rados.rados.0.smithi084.stdout:668: done (2 left) 2024-09-20T22:45:01.212 INFO:tasks.rados.rados.0.smithi084.stdout:669: read oid 27 snap -1 2024-09-20T22:45:01.212 INFO:tasks.rados.rados.0.smithi084.stdout:669: expect (ObjNum 234 snap 51 seq_num 234) 2024-09-20T22:45:01.213 INFO:tasks.rados.rados.0.smithi084.stdout:670: copy_from oid 26 from oid 8 current snap is 64 2024-09-20T22:45:01.213 INFO:tasks.rados.rados.0.smithi084.stdout:665: finishing rollback tid 0 to smithi08477203-17 2024-09-20T22:45:01.213 INFO:tasks.rados.rados.0.smithi084.stdout:665: finishing rollback tid 1 to smithi08477203-17 2024-09-20T22:45:01.213 INFO:tasks.rados.rados.0.smithi084.stdout:665: finishing rollback tid 2 to smithi08477203-17 2024-09-20T22:45:01.213 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 17 v 85 (ObjNum 203 snap 41 seq_num 203) dirty exists 2024-09-20T22:45:01.213 INFO:tasks.rados.rados.0.smithi084.stdout:665: done (3 left) 2024-09-20T22:45:01.213 INFO:tasks.rados.rados.0.smithi084.stdout:667: done (2 left) 2024-09-20T22:45:01.213 INFO:tasks.rados.rados.0.smithi084.stdout:671: copy_from oid 50 from oid 11 current snap is 64 2024-09-20T22:45:01.213 INFO:tasks.rados.rados.0.smithi084.stdout:672: copy_from oid 28 from oid 20 current snap is 64 2024-09-20T22:45:01.213 INFO:tasks.rados.rados.0.smithi084.stdout:673: write (excl) oid 38 current snap is 64 2024-09-20T22:45:01.213 INFO:tasks.rados.rados.0.smithi084.stdout:673: seq_num 263 ranges {420241=546489,1373823=544414} 2024-09-20T22:45:01.219 INFO:tasks.rados.rados.0.smithi084.stdout:673: writing smithi08477203-38 from 420241 to 966730 tid 1 2024-09-20T22:45:01.226 INFO:tasks.rados.rados.0.smithi084.stdout:673: writing smithi08477203-38 from 1373823 to 1918237 tid 2 2024-09-20T22:45:01.226 INFO:tasks.rados.rados.0.smithi084.stdout:674: snap_create 2024-09-20T22:45:01.226 INFO:tasks.rados.rados.0.smithi084.stdout:672: finishing copy_from to smithi08477203-28 2024-09-20T22:45:01.226 INFO:tasks.rados.rados.0.smithi084.stdout:672: got expected ENOENT (src dne) 2024-09-20T22:45:01.227 INFO:tasks.rados.rados.0.smithi084.stdout:669: expect (ObjNum 234 snap 51 seq_num 234) 2024-09-20T22:45:01.243 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:45:01.264 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:01.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:01.265 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:45:01.320 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-20T22:45:01.365 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:45:01.387 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:01.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:01.388 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:45:01.472 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:01 smithi084 ceph-mon[40024]: pgmap v232: 178 pgs: 1 peering, 177 active+clean; 118 MiB data, 549 MiB used, 625 GiB / 626 GiB avail; 2.1 MiB/s rd, 756 KiB/s wr, 6 op/s 2024-09-20T22:45:01.473 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:01 smithi084 ceph-mon[40024]: 6.13 scrub starts 2024-09-20T22:45:01.473 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:01 smithi084 ceph-mon[40024]: 6.13 scrub ok 2024-09-20T22:45:01.473 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:01 smithi084 ceph-mon[40024]: osdmap e221: 8 total, 8 up, 7 in 2024-09-20T22:45:01.473 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:01 smithi084 ceph-mon[40024]: 5.11 scrub starts 2024-09-20T22:45:01.473 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:01 smithi084 ceph-mon[40024]: 5.11 scrub ok 2024-09-20T22:45:01.473 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:01 smithi084 ceph-mon[29774]: pgmap v232: 178 pgs: 1 peering, 177 active+clean; 118 MiB data, 549 MiB used, 625 GiB / 626 GiB avail; 2.1 MiB/s rd, 756 KiB/s wr, 6 op/s 2024-09-20T22:45:01.473 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:01 smithi084 ceph-mon[29774]: 6.13 scrub starts 2024-09-20T22:45:01.473 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:01 smithi084 ceph-mon[29774]: 6.13 scrub ok 2024-09-20T22:45:01.473 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:01 smithi084 ceph-mon[29774]: osdmap e221: 8 total, 8 up, 7 in 2024-09-20T22:45:01.473 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:01 smithi084 ceph-mon[29774]: 5.11 scrub starts 2024-09-20T22:45:01.473 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:01 smithi084 ceph-mon[29774]: 5.11 scrub ok 2024-09-20T22:45:01.489 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:45:01.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:01 smithi154 ceph-mon[33264]: pgmap v232: 178 pgs: 1 peering, 177 active+clean; 118 MiB data, 549 MiB used, 625 GiB / 626 GiB avail; 2.1 MiB/s rd, 756 KiB/s wr, 6 op/s 2024-09-20T22:45:01.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:01 smithi154 ceph-mon[33264]: 6.13 scrub starts 2024-09-20T22:45:01.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:01 smithi154 ceph-mon[33264]: 6.13 scrub ok 2024-09-20T22:45:01.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:01 smithi154 ceph-mon[33264]: osdmap e221: 8 total, 8 up, 7 in 2024-09-20T22:45:01.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:01 smithi154 ceph-mon[33264]: 5.11 scrub starts 2024-09-20T22:45:01.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:01 smithi154 ceph-mon[33264]: 5.11 scrub ok 2024-09-20T22:45:01.510 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:01.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:01.511 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:45:01.526 INFO:tasks.rados.rados.0.smithi084.stdout:672: finishing copy_from racing read to smithi08477203-28 2024-09-20T22:45:01.526 INFO:tasks.rados.rados.0.smithi084.stdout:670: finishing copy_from racing read to smithi08477203-26 2024-09-20T22:45:01.526 INFO:tasks.rados.rados.0.smithi084.stdout:673: finishing write tid 1 to smithi08477203-38 2024-09-20T22:45:01.526 INFO:tasks.rados.rados.0.smithi084.stdout:673: finishing write tid 2 to smithi08477203-38 2024-09-20T22:45:01.526 INFO:tasks.rados.rados.0.smithi084.stdout:673: finishing write tid 3 to smithi08477203-38 2024-09-20T22:45:01.526 INFO:tasks.rados.rados.0.smithi084.stdout:673: finishing write tid 4 to smithi08477203-38 2024-09-20T22:45:01.527 INFO:tasks.rados.rados.0.smithi084.stdout:673: oid 38 updating version 0 to 179 2024-09-20T22:45:01.527 INFO:tasks.rados.rados.0.smithi084.stdout:673: oid 38 updating version 179 to 181 2024-09-20T22:45:01.527 INFO:tasks.rados.rados.0.smithi084.stdout:673: oid 38 version 181 is already newer than 180 2024-09-20T22:45:01.527 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 38 v 181 (ObjNum 263 snap 64 seq_num 263) dirty exists 2024-09-20T22:45:01.527 INFO:tasks.rados.rados.0.smithi084.stdout:673: left oid 38 (ObjNum 263 snap 64 seq_num 263) 2024-09-20T22:45:01.527 INFO:tasks.rados.rados.0.smithi084.stdout:670: finishing copy_from to smithi08477203-26 2024-09-20T22:45:01.527 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 26 v 88 (ObjNum 164 snap 35 seq_num 164) dirty exists 2024-09-20T22:45:01.527 INFO:tasks.rados.rados.0.smithi084.stdout:671: finishing copy_from racing read to smithi08477203-50 2024-09-20T22:45:01.527 INFO:tasks.rados.rados.0.smithi084.stdout:671: finishing copy_from to smithi08477203-50 2024-09-20T22:45:01.527 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 50 v 101 (ObjNum 245 snap 54 seq_num 245) dirty exists 2024-09-20T22:45:01.612 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:45:01.633 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:01.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:01.634 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:45:01.735 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:45:01.756 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:01.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:01.756 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:45:01.857 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:45:01.864 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:45:01.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:01.880 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:01.880 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:45:01.981 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:45:02.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:02.002 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:02.003 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:45:02.050 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-20T22:45:02.103 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:45:02.125 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:02.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:02.126 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:45:02.220 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is set 2024-09-20T22:45:02.226 INFO:tasks.rados.rados.0.smithi084.stdout:669: done (5 left) 2024-09-20T22:45:02.226 INFO:tasks.rados.rados.0.smithi084.stdout:670: done (4 left) 2024-09-20T22:45:02.227 INFO:tasks.rados.rados.0.smithi084.stdout:671: done (3 left) 2024-09-20T22:45:02.227 INFO:tasks.rados.rados.0.smithi084.stdout:672: done (2 left) 2024-09-20T22:45:02.227 INFO:tasks.rados.rados.0.smithi084.stdout:673: done (1 left) 2024-09-20T22:45:02.227 INFO:tasks.rados.rados.0.smithi084.stdout:674: done (0 left) 2024-09-20T22:45:02.227 INFO:tasks.rados.rados.0.smithi084.stdout:675: write (excl) oid 24 current snap is 65 2024-09-20T22:45:02.227 INFO:tasks.rados.rados.0.smithi084.stdout:675: seq_num 264 ranges {437048=621738,1832305=404425} 2024-09-20T22:45:02.227 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:45:02.232 INFO:tasks.rados.rados.0.smithi084.stdout:675: writing smithi08477203-24 from 437048 to 1058786 tid 1 2024-09-20T22:45:02.241 INFO:tasks.rados.rados.0.smithi084.stdout:675: writing smithi08477203-24 from 1832305 to 2236730 tid 2 2024-09-20T22:45:02.241 INFO:tasks.rados.rados.0.smithi084.stdout:676: snap_remove snap 55 2024-09-20T22:45:02.250 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:02.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:02.251 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:45:02.283 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:02 smithi084 ceph-mon[40024]: 4.8 scrub starts 2024-09-20T22:45:02.283 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:02 smithi084 ceph-mon[40024]: 4.8 scrub ok 2024-09-20T22:45:02.283 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:02 smithi084 ceph-mon[40024]: 6.16 scrub starts 2024-09-20T22:45:02.283 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:02 smithi084 ceph-mon[40024]: 6.16 scrub ok 2024-09-20T22:45:02.283 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:02 smithi084 ceph-mon[40024]: osdmap e222: 8 total, 8 up, 7 in 2024-09-20T22:45:02.283 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:02 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/2171296309' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-20T22:45:02.283 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:02 smithi084 ceph-mon[40024]: 1.b deep-scrub starts 2024-09-20T22:45:02.283 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:02 smithi084 ceph-mon[40024]: 1.b deep-scrub ok 2024-09-20T22:45:02.283 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:02 smithi084 ceph-mon[29774]: 4.8 scrub starts 2024-09-20T22:45:02.284 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:02 smithi084 ceph-mon[29774]: 4.8 scrub ok 2024-09-20T22:45:02.284 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:02 smithi084 ceph-mon[29774]: 6.16 scrub starts 2024-09-20T22:45:02.284 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:02 smithi084 ceph-mon[29774]: 6.16 scrub ok 2024-09-20T22:45:02.284 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:02 smithi084 ceph-mon[29774]: osdmap e222: 8 total, 8 up, 7 in 2024-09-20T22:45:02.284 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:02 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/2171296309' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-20T22:45:02.284 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:02 smithi084 ceph-mon[29774]: 1.b deep-scrub starts 2024-09-20T22:45:02.284 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:02 smithi084 ceph-mon[29774]: 1.b deep-scrub ok 2024-09-20T22:45:02.351 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:45:02.373 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:02.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:02.374 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:45:02.381 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-20T22:45:02.381 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:45:02.474 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:45:02.496 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:02.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:02.497 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:45:02.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:02 smithi154 ceph-mon[33264]: 4.8 scrub starts 2024-09-20T22:45:02.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:02 smithi154 ceph-mon[33264]: 4.8 scrub ok 2024-09-20T22:45:02.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:02 smithi154 ceph-mon[33264]: 6.16 scrub starts 2024-09-20T22:45:02.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:02 smithi154 ceph-mon[33264]: 6.16 scrub ok 2024-09-20T22:45:02.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:02 smithi154 ceph-mon[33264]: osdmap e222: 8 total, 8 up, 7 in 2024-09-20T22:45:02.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:02 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/2171296309' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-20T22:45:02.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:02 smithi154 ceph-mon[33264]: 1.b deep-scrub starts 2024-09-20T22:45:02.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:02 smithi154 ceph-mon[33264]: 1.b deep-scrub ok 2024-09-20T22:45:02.598 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:45:02.620 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:02.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:02.621 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:45:02.721 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:45:02.743 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:02.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:02.744 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:45:02.845 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:45:02.866 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:02.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:02.867 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:45:02.968 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:45:02.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:02.993 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:02.994 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:45:03.002 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.1 dump_blocked_ops 2024-09-20T22:45:03.095 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:45:03.116 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:03.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:03.117 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:45:03.146 INFO:tasks.rados.rados.0.smithi084.stdout:676: done (1 left) 2024-09-20T22:45:03.146 INFO:tasks.rados.rados.0.smithi084.stdout:677: write oid 29 current snap is 65 2024-09-20T22:45:03.146 INFO:tasks.rados.rados.0.smithi084.stdout:677: seq_num 265 ranges {453855=696987,1890787=548083} 2024-09-20T22:45:03.155 INFO:tasks.rados.rados.0.smithi084.stdout:677: writing smithi08477203-29 from 453855 to 1150842 tid 1 2024-09-20T22:45:03.163 INFO:tasks.rados.rados.0.smithi084.stdout:677: writing smithi08477203-29 from 1890787 to 2438870 tid 2 2024-09-20T22:45:03.163 INFO:tasks.rados.rados.0.smithi084.stdout:678: delete oid 16 current snap is 65 2024-09-20T22:45:03.163 INFO:tasks.rados.rados.0.smithi084.stdout:675: finishing write tid 1 to smithi08477203-24 2024-09-20T22:45:03.164 INFO:tasks.rados.rados.0.smithi084.stdout:675: finishing write tid 2 to smithi08477203-24 2024-09-20T22:45:03.164 INFO:tasks.rados.rados.0.smithi084.stdout:675: finishing write tid 3 to smithi08477203-24 2024-09-20T22:45:03.164 INFO:tasks.rados.rados.0.smithi084.stdout:675: finishing write tid 4 to smithi08477203-24 2024-09-20T22:45:03.164 INFO:tasks.rados.rados.0.smithi084.stdout:675: oid 24 updating version 0 to 72 2024-09-20T22:45:03.164 INFO:tasks.rados.rados.0.smithi084.stdout:675: oid 24 version 72 is already newer than 70 2024-09-20T22:45:03.164 INFO:tasks.rados.rados.0.smithi084.stdout:675: oid 24 version 72 is already newer than 71 2024-09-20T22:45:03.164 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 24 v 72 (ObjNum 264 snap 65 seq_num 264) dirty exists 2024-09-20T22:45:03.164 INFO:tasks.rados.rados.0.smithi084.stdout:675: left oid 24 (ObjNum 264 snap 65 seq_num 264) 2024-09-20T22:45:03.164 INFO:tasks.rados.rados.0.smithi084.stdout:677: finishing write tid 1 to smithi08477203-29 2024-09-20T22:45:03.165 INFO:tasks.rados.rados.0.smithi084.stdout:675: done (2 left) 2024-09-20T22:45:03.165 INFO:tasks.rados.rados.0.smithi084.stdout:678: done (1 left) 2024-09-20T22:45:03.165 INFO:tasks.rados.rados.0.smithi084.stdout:679: rollback oid 32 current snap is 65 2024-09-20T22:45:03.165 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 32 to 63 2024-09-20T22:45:03.165 INFO:tasks.rados.rados.0.smithi084.stdout:680: write oid 21 current snap is 65 2024-09-20T22:45:03.165 INFO:tasks.rados.rados.0.smithi084.stdout:680: seq_num 267 ranges {487469=563838,1607751=551752} 2024-09-20T22:45:03.171 INFO:tasks.rados.rados.0.smithi084.stdout:680: writing smithi08477203-21 from 487469 to 1051307 tid 1 2024-09-20T22:45:03.177 INFO:tasks.rados.rados.0.smithi084.stdout:680: writing smithi08477203-21 from 1607751 to 2159503 tid 2 2024-09-20T22:45:03.177 INFO:tasks.rados.rados.0.smithi084.stdout:681: snap_create 2024-09-20T22:45:03.178 INFO:tasks.rados.rados.0.smithi084.stdout:677: finishing write tid 2 to smithi08477203-29 2024-09-20T22:45:03.178 INFO:tasks.rados.rados.0.smithi084.stdout:677: finishing write tid 3 to smithi08477203-29 2024-09-20T22:45:03.178 INFO:tasks.rados.rados.0.smithi084.stdout:679: finishing rollback tid 1 to smithi08477203-32 2024-09-20T22:45:03.178 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 32 v 0 (ObjNum 129 snap 0 seq_num 712571475) dirty dne 2024-09-20T22:45:03.178 INFO:tasks.rados.rados.0.smithi084.stdout:677: finishing write tid 4 to smithi08477203-29 2024-09-20T22:45:03.178 INFO:tasks.rados.rados.0.smithi084.stdout:677: oid 29 updating version 0 to 315 2024-09-20T22:45:03.178 INFO:tasks.rados.rados.0.smithi084.stdout:677: oid 29 version 315 is already newer than 306 2024-09-20T22:45:03.178 INFO:tasks.rados.rados.0.smithi084.stdout:677: oid 29 updating version 315 to 316 2024-09-20T22:45:03.178 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 29 v 316 (ObjNum 265 snap 65 seq_num 265) dirty exists 2024-09-20T22:45:03.178 INFO:tasks.rados.rados.0.smithi084.stdout:677: left oid 29 (ObjNum 265 snap 65 seq_num 265) 2024-09-20T22:45:03.178 INFO:tasks.rados.rados.0.smithi084.stdout:680: finishing write tid 1 to smithi08477203-21 2024-09-20T22:45:03.181 INFO:tasks.rados.rados.0.smithi084.stdout:680: finishing write tid 2 to smithi08477203-21 2024-09-20T22:45:03.181 INFO:tasks.rados.rados.0.smithi084.stdout:680: finishing write tid 3 to smithi08477203-21 2024-09-20T22:45:03.181 INFO:tasks.rados.rados.0.smithi084.stdout:680: finishing write tid 4 to smithi08477203-21 2024-09-20T22:45:03.181 INFO:tasks.rados.rados.0.smithi084.stdout:680: oid 21 updating version 0 to 212 2024-09-20T22:45:03.181 INFO:tasks.rados.rados.0.smithi084.stdout:680: oid 21 version 212 is already newer than 210 2024-09-20T22:45:03.181 INFO:tasks.rados.rados.0.smithi084.stdout:680: oid 21 version 212 is already newer than 211 2024-09-20T22:45:03.181 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 21 v 212 (ObjNum 267 snap 65 seq_num 267) dirty exists 2024-09-20T22:45:03.181 INFO:tasks.rados.rados.0.smithi084.stdout:680: left oid 21 (ObjNum 267 snap 65 seq_num 267) 2024-09-20T22:45:03.219 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:45:03.242 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:03.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:03.243 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:45:03.344 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:45:03.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:03.367 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:03.367 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:45:03.468 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:45:03.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:03.490 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:03.491 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:45:03.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:03 smithi154 ceph-mon[33264]: 6.a scrub starts 2024-09-20T22:45:03.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:03 smithi154 ceph-mon[33264]: 6.a scrub ok 2024-09-20T22:45:03.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:03 smithi154 ceph-mon[33264]: pgmap v235: 178 pgs: 1 peering, 177 active+clean; 118 MiB data, 549 MiB used, 625 GiB / 626 GiB avail; 2.1 MiB/s rd, 755 KiB/s wr, 6 op/s 2024-09-20T22:45:03.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:03 smithi154 ceph-mon[33264]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:45:03.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:03 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/2171296309' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T22:45:03.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:03 smithi154 ceph-mon[33264]: osdmap e223: 8 total, 8 up, 7 in 2024-09-20T22:45:03.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:03 smithi154 ceph-mon[33264]: purged_snaps scrub starts 2024-09-20T22:45:03.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:03 smithi154 ceph-mon[33264]: purged_snaps scrub ok 2024-09-20T22:45:03.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:03 smithi154 ceph-mon[33264]: 4.1f scrub starts 2024-09-20T22:45:03.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:03 smithi154 ceph-mon[33264]: osdmap e224: 8 total, 8 up, 7 in 2024-09-20T22:45:03.591 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:45:03.612 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:03.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:03.613 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:45:03.614 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:03 smithi084 ceph-mon[40024]: 6.a scrub starts 2024-09-20T22:45:03.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:03 smithi084 ceph-mon[40024]: 6.a scrub ok 2024-09-20T22:45:03.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:03 smithi084 ceph-mon[40024]: pgmap v235: 178 pgs: 1 peering, 177 active+clean; 118 MiB data, 549 MiB used, 625 GiB / 626 GiB avail; 2.1 MiB/s rd, 755 KiB/s wr, 6 op/s 2024-09-20T22:45:03.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:03 smithi084 ceph-mon[40024]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:45:03.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:03 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/2171296309' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T22:45:03.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:03 smithi084 ceph-mon[40024]: osdmap e223: 8 total, 8 up, 7 in 2024-09-20T22:45:03.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:03 smithi084 ceph-mon[40024]: purged_snaps scrub starts 2024-09-20T22:45:03.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:03 smithi084 ceph-mon[40024]: purged_snaps scrub ok 2024-09-20T22:45:03.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:03 smithi084 ceph-mon[40024]: 4.1f scrub starts 2024-09-20T22:45:03.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:03 smithi084 ceph-mon[40024]: osdmap e224: 8 total, 8 up, 7 in 2024-09-20T22:45:03.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:03 smithi084 ceph-mon[29774]: 6.a scrub starts 2024-09-20T22:45:03.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:03 smithi084 ceph-mon[29774]: 6.a scrub ok 2024-09-20T22:45:03.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:03 smithi084 ceph-mon[29774]: pgmap v235: 178 pgs: 1 peering, 177 active+clean; 118 MiB data, 549 MiB used, 625 GiB / 626 GiB avail; 2.1 MiB/s rd, 755 KiB/s wr, 6 op/s 2024-09-20T22:45:03.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:03 smithi084 ceph-mon[29774]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:45:03.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:03 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/2171296309' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T22:45:03.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:03 smithi084 ceph-mon[29774]: osdmap e223: 8 total, 8 up, 7 in 2024-09-20T22:45:03.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:03 smithi084 ceph-mon[29774]: purged_snaps scrub starts 2024-09-20T22:45:03.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:03 smithi084 ceph-mon[29774]: purged_snaps scrub ok 2024-09-20T22:45:03.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:03 smithi084 ceph-mon[29774]: 4.1f scrub starts 2024-09-20T22:45:03.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:03 smithi084 ceph-mon[29774]: osdmap e224: 8 total, 8 up, 7 in 2024-09-20T22:45:03.714 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:45:03.735 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:03.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:03.736 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:45:03.741 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:45:03.772 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:45:03.784 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 7] out_osds: [4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-20T22:45:03.784 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-20T22:45:03.784 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-09-20T22:45:03.784 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd dump -f json-pretty 2024-09-20T22:45:03.837 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:45:03.858 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:03.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:03.859 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:45:03.959 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:45:03.980 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:03.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:03.981 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:45:04.082 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:45:04.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:04.104 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:04.105 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:45:04.147 INFO:tasks.rados.rados.0.smithi084.stdout:677: done (3 left) 2024-09-20T22:45:04.147 INFO:tasks.rados.rados.0.smithi084.stdout:679: done (2 left) 2024-09-20T22:45:04.147 INFO:tasks.rados.rados.0.smithi084.stdout:680: done (1 left) 2024-09-20T22:45:04.148 INFO:tasks.rados.rados.0.smithi084.stdout:681: done (0 left) 2024-09-20T22:45:04.148 INFO:tasks.rados.rados.0.smithi084.stdout:682: read oid 1 snap -1 2024-09-20T22:45:04.148 INFO:tasks.rados.rados.0.smithi084.stdout:682: expect (ObjNum 237 snap 51 seq_num 237) 2024-09-20T22:45:04.148 INFO:tasks.rados.rados.0.smithi084.stdout:683: snap_create 2024-09-20T22:45:04.153 INFO:tasks.rados.rados.0.smithi084.stdout:682: expect (ObjNum 237 snap 51 seq_num 237) 2024-09-20T22:45:04.205 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:45:04.215 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:45:04.215 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:45:04.228 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:04.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:04.229 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:45:04.235 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:45:04.235 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:45:04.254 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:45:04.255 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:45:04.255 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:45:04.274 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-20T22:45:04.274 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T22:45:04.291 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:45:04.291 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:45:04.307 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:45:04.307 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:45:04.324 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:45:04.324 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:45:04.330 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:45:04.341 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:45:04.341 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:45:04.352 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:04.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:04.353 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:45:04.454 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:45:04.475 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:04.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:04.476 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:45:04.577 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:45:04.598 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:04.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:04.599 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:45:04.700 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:45:04.722 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:04.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:04.723 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:45:04.824 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:45:04.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:04.846 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:04.846 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:45:04.934 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-20T22:45:04.934 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2024-09-20T22:45:04.934 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2024-09-20T22:45:04.934 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2024-09-20T22:45:04.934 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-20T22:45:04.947 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:45:04.968 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:04.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:04.969 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:45:05.070 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:45:05.091 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:05.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:05.092 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:45:05.148 INFO:tasks.rados.rados.0.smithi084.stdout:682: done (1 left) 2024-09-20T22:45:05.148 INFO:tasks.rados.rados.0.smithi084.stdout:683: done (0 left) 2024-09-20T22:45:05.148 INFO:tasks.rados.rados.0.smithi084.stdout:684: snap_create 2024-09-20T22:45:05.193 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:45:05.214 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:05.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:05.215 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:45:05.316 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:45:05.337 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:05.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:05.338 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:45:05.438 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:45:05.460 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:05.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:05.461 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:45:05.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:05 smithi154 ceph-mon[33264]: pgmap v238: 178 pgs: 3 active+clean+snaptrim, 175 active+clean; 109 MiB data, 552 MiB used, 625 GiB / 626 GiB avail; 7.4 MiB/s rd, 2.5 MiB/s wr, 13 op/s 2024-09-20T22:45:05.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:05 smithi154 ceph-mon[33264]: osdmap e225: 8 total, 8 up, 7 in 2024-09-20T22:45:05.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:05 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/3000422478' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:45:05.538 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:05 smithi084 ceph-mon[40024]: pgmap v238: 178 pgs: 3 active+clean+snaptrim, 175 active+clean; 109 MiB data, 552 MiB used, 625 GiB / 626 GiB avail; 7.4 MiB/s rd, 2.5 MiB/s wr, 13 op/s 2024-09-20T22:45:05.538 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:05 smithi084 ceph-mon[40024]: osdmap e225: 8 total, 8 up, 7 in 2024-09-20T22:45:05.539 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:05 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/3000422478' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:45:05.539 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:05 smithi084 ceph-mon[29774]: pgmap v238: 178 pgs: 3 active+clean+snaptrim, 175 active+clean; 109 MiB data, 552 MiB used, 625 GiB / 626 GiB avail; 7.4 MiB/s rd, 2.5 MiB/s wr, 13 op/s 2024-09-20T22:45:05.539 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:05 smithi084 ceph-mon[29774]: osdmap e225: 8 total, 8 up, 7 in 2024-09-20T22:45:05.539 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:05 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/3000422478' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:45:05.562 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:45:05.564 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.1 dump_historic_ops 2024-09-20T22:45:05.583 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:05.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:05.584 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:45:05.684 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:45:05.685 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-20T22:45:05.706 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:05.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:05.707 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:45:05.808 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:45:05.829 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:05.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:05.830 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:45:05.931 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:45:05.952 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:05.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:05.953 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:45:06.054 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:45:06.075 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:06.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:06.076 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:45:06.149 INFO:tasks.rados.rados.0.smithi084.stdout:684: done (0 left) 2024-09-20T22:45:06.150 INFO:tasks.rados.rados.0.smithi084.stdout:685: snap_create 2024-09-20T22:45:06.177 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:45:06.200 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:06.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:06.201 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:45:06.268 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:06 smithi084 ceph-mon[40024]: osdmap e226: 8 total, 8 up, 7 in 2024-09-20T22:45:06.268 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:06 smithi084 ceph-mon[29774]: osdmap e226: 8 total, 8 up, 7 in 2024-09-20T22:45:06.302 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:45:06.323 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:06.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:06.323 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:45:06.424 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:45:06.448 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:06.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:06.449 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:45:06.477 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:45:06.487 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:45:06.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:06 smithi154 ceph-mon[33264]: osdmap e226: 8 total, 8 up, 7 in 2024-09-20T22:45:06.550 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:45:06.558 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-20T22:45:06.558 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-20T22:45:06.558 INFO:teuthology.orchestra.run.smithi084.stdout: "epoch": 227, 2024-09-20T22:45:06.558 INFO:teuthology.orchestra.run.smithi084.stdout: "fsid": "a9c0bc70-77a0-11ef-baf5-efdc52797490", 2024-09-20T22:45:06.558 INFO:teuthology.orchestra.run.smithi084.stdout: "created": "2024-09-20T22:36:30.240626+0000", 2024-09-20T22:45:06.558 INFO:teuthology.orchestra.run.smithi084.stdout: "modified": "2024-09-20T22:45:06.142462+0000", 2024-09-20T22:45:06.559 INFO:teuthology.orchestra.run.smithi084.stdout: "last_up_change": "2024-09-20T22:41:27.098565+0000", 2024-09-20T22:45:06.559 INFO:teuthology.orchestra.run.smithi084.stdout: "last_in_change": "2024-09-20T22:43:56.388784+0000", 2024-09-20T22:45:06.559 INFO:teuthology.orchestra.run.smithi084.stdout: "flags": "noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-20T22:45:06.559 INFO:teuthology.orchestra.run.smithi084.stdout: "flags_num": 5801984, 2024-09-20T22:45:06.559 INFO:teuthology.orchestra.run.smithi084.stdout: "flags_set": [ 2024-09-20T22:45:06.559 INFO:teuthology.orchestra.run.smithi084.stdout: "noscrub", 2024-09-20T22:45:06.559 INFO:teuthology.orchestra.run.smithi084.stdout: "pglog_hardlimit", 2024-09-20T22:45:06.559 INFO:teuthology.orchestra.run.smithi084.stdout: "purged_snapdirs", 2024-09-20T22:45:06.559 INFO:teuthology.orchestra.run.smithi084.stdout: "recovery_deletes", 2024-09-20T22:45:06.559 INFO:teuthology.orchestra.run.smithi084.stdout: "sortbitwise" 2024-09-20T22:45:06.559 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2024-09-20T22:45:06.559 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_version": 18, 2024-09-20T22:45:06.559 INFO:teuthology.orchestra.run.smithi084.stdout: "full_ratio": 0.94999998807907104, 2024-09-20T22:45:06.559 INFO:teuthology.orchestra.run.smithi084.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-20T22:45:06.560 INFO:teuthology.orchestra.run.smithi084.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-20T22:45:06.560 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_snapshot": "", 2024-09-20T22:45:06.560 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_max": 7, 2024-09-20T22:45:06.560 INFO:teuthology.orchestra.run.smithi084.stdout: "max_osd": 8, 2024-09-20T22:45:06.560 INFO:teuthology.orchestra.run.smithi084.stdout: "require_min_compat_client": "luminous", 2024-09-20T22:45:06.560 INFO:teuthology.orchestra.run.smithi084.stdout: "min_compat_client": "luminous", 2024-09-20T22:45:06.560 INFO:teuthology.orchestra.run.smithi084.stdout: "require_osd_release": "squid", 2024-09-20T22:45:06.560 INFO:teuthology.orchestra.run.smithi084.stdout: "allow_crimson": false, 2024-09-20T22:45:06.560 INFO:teuthology.orchestra.run.smithi084.stdout: "pools": [ 2024-09-20T22:45:06.560 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:06.560 INFO:teuthology.orchestra.run.smithi084.stdout: "pool": 1, 2024-09-20T22:45:06.560 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_name": ".mgr", 2024-09-20T22:45:06.560 INFO:teuthology.orchestra.run.smithi084.stdout: "create_time": "2024-09-20T22:39:21.984381+0000", 2024-09-20T22:45:06.560 INFO:teuthology.orchestra.run.smithi084.stdout: "flags": 1, 2024-09-20T22:45:06.561 INFO:teuthology.orchestra.run.smithi084.stdout: "flags_names": "hashpspool", 2024-09-20T22:45:06.561 INFO:teuthology.orchestra.run.smithi084.stdout: "type": 1, 2024-09-20T22:45:06.561 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 3, 2024-09-20T22:45:06.561 INFO:teuthology.orchestra.run.smithi084.stdout: "min_size": 2, 2024-09-20T22:45:06.561 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_rule": 0, 2024-09-20T22:45:06.561 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_count": 0, 2024-09-20T22:45:06.561 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_target": 0, 2024-09-20T22:45:06.561 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_barrier": 0, 2024-09-20T22:45:06.561 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-20T22:45:06.561 INFO:teuthology.orchestra.run.smithi084.stdout: "is_stretch_pool": false, 2024-09-20T22:45:06.561 INFO:teuthology.orchestra.run.smithi084.stdout: "object_hash": 2, 2024-09-20T22:45:06.561 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_autoscale_mode": "off", 2024-09-20T22:45:06.561 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num": 21, 2024-09-20T22:45:06.561 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num": 21, 2024-09-20T22:45:06.562 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num_target": 21, 2024-09-20T22:45:06.562 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_target": 21, 2024-09-20T22:45:06.562 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_pending": 21, 2024-09-20T22:45:06.562 INFO:teuthology.orchestra.run.smithi084.stdout: "last_pg_merge_meta": { 2024-09-20T22:45:06.562 INFO:teuthology.orchestra.run.smithi084.stdout: "source_pgid": "0.0", 2024-09-20T22:45:06.562 INFO:teuthology.orchestra.run.smithi084.stdout: "ready_epoch": 0, 2024-09-20T22:45:06.562 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_started": 0, 2024-09-20T22:45:06.562 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_clean": 0, 2024-09-20T22:45:06.562 INFO:teuthology.orchestra.run.smithi084.stdout: "source_version": "0'0", 2024-09-20T22:45:06.562 INFO:teuthology.orchestra.run.smithi084.stdout: "target_version": "0'0" 2024-09-20T22:45:06.562 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:06.562 INFO:teuthology.orchestra.run.smithi084.stdout: "last_change": "147", 2024-09-20T22:45:06.562 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend": "0", 2024-09-20T22:45:06.562 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-20T22:45:06.563 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_preluminous": "119", 2024-09-20T22:45:06.563 INFO:teuthology.orchestra.run.smithi084.stdout: "auid": 0, 2024-09-20T22:45:06.563 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_mode": "selfmanaged", 2024-09-20T22:45:06.563 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_seq": 0, 2024-09-20T22:45:06.563 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_epoch": 0, 2024-09-20T22:45:06.563 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_snaps": [], 2024-09-20T22:45:06.563 INFO:teuthology.orchestra.run.smithi084.stdout: "removed_snaps": "[]", 2024-09-20T22:45:06.563 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_bytes": 0, 2024-09-20T22:45:06.563 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_objects": 0, 2024-09-20T22:45:06.563 INFO:teuthology.orchestra.run.smithi084.stdout: "tiers": [], 2024-09-20T22:45:06.563 INFO:teuthology.orchestra.run.smithi084.stdout: "tier_of": -1, 2024-09-20T22:45:06.563 INFO:teuthology.orchestra.run.smithi084.stdout: "read_tier": -1, 2024-09-20T22:45:06.563 INFO:teuthology.orchestra.run.smithi084.stdout: "write_tier": -1, 2024-09-20T22:45:06.564 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_mode": "none", 2024-09-20T22:45:06.564 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_bytes": 0, 2024-09-20T22:45:06.564 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_objects": 0, 2024-09-20T22:45:06.564 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-20T22:45:06.564 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-20T22:45:06.564 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-20T22:45:06.564 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_flush_age": 0, 2024-09-20T22:45:06.564 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_evict_age": 0, 2024-09-20T22:45:06.564 INFO:teuthology.orchestra.run.smithi084.stdout: "erasure_code_profile": "", 2024-09-20T22:45:06.564 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_params": { 2024-09-20T22:45:06.564 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "none" 2024-09-20T22:45:06.564 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:06.564 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_period": 0, 2024-09-20T22:45:06.564 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_count": 0, 2024-09-20T22:45:06.565 INFO:teuthology.orchestra.run.smithi084.stdout: "use_gmt_hitset": true, 2024-09-20T22:45:06.565 INFO:teuthology.orchestra.run.smithi084.stdout: "min_read_recency_for_promote": 0, 2024-09-20T22:45:06.565 INFO:teuthology.orchestra.run.smithi084.stdout: "min_write_recency_for_promote": 0, 2024-09-20T22:45:06.565 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_grade_decay_rate": 0, 2024-09-20T22:45:06.565 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_search_last_n": 0, 2024-09-20T22:45:06.565 INFO:teuthology.orchestra.run.smithi084.stdout: "grade_table": [], 2024-09-20T22:45:06.565 INFO:teuthology.orchestra.run.smithi084.stdout: "stripe_width": 0, 2024-09-20T22:45:06.565 INFO:teuthology.orchestra.run.smithi084.stdout: "expected_num_objects": 0, 2024-09-20T22:45:06.565 INFO:teuthology.orchestra.run.smithi084.stdout: "fast_read": false, 2024-09-20T22:45:06.565 INFO:teuthology.orchestra.run.smithi084.stdout: "options": { 2024-09-20T22:45:06.565 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_max": 32, 2024-09-20T22:45:06.565 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_min": 1 2024-09-20T22:45:06.565 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:06.566 INFO:teuthology.orchestra.run.smithi084.stdout: "application_metadata": { 2024-09-20T22:45:06.566 INFO:teuthology.orchestra.run.smithi084.stdout: "mgr": {} 2024-09-20T22:45:06.566 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:06.566 INFO:teuthology.orchestra.run.smithi084.stdout: "read_balance": { 2024-09-20T22:45:06.566 INFO:teuthology.orchestra.run.smithi084.stdout: "score_type": "Fair distribution", 2024-09-20T22:45:06.566 INFO:teuthology.orchestra.run.smithi084.stdout: "score_acting": 1.8999999761581421, 2024-09-20T22:45:06.566 INFO:teuthology.orchestra.run.smithi084.stdout: "score_stable": 1.8999999761581421, 2024-09-20T22:45:06.566 INFO:teuthology.orchestra.run.smithi084.stdout: "optimal_score": 0.87999999523162842, 2024-09-20T22:45:06.566 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_acting": 1.6699999570846558, 2024-09-20T22:45:06.566 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_stable": 1.6699999570846558, 2024-09-20T22:45:06.566 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity_weighted": 1, 2024-09-20T22:45:06.566 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity": 1, 2024-09-20T22:45:06.566 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity_weighted": 1 2024-09-20T22:45:06.566 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:06.567 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:06.567 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:06.567 INFO:teuthology.orchestra.run.smithi084.stdout: "pool": 2, 2024-09-20T22:45:06.567 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_name": "datapool", 2024-09-20T22:45:06.567 INFO:teuthology.orchestra.run.smithi084.stdout: "create_time": "2024-09-20T22:41:35.820081+0000", 2024-09-20T22:45:06.567 INFO:teuthology.orchestra.run.smithi084.stdout: "flags": 8193, 2024-09-20T22:45:06.567 INFO:teuthology.orchestra.run.smithi084.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-20T22:45:06.567 INFO:teuthology.orchestra.run.smithi084.stdout: "type": 1, 2024-09-20T22:45:06.567 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 3, 2024-09-20T22:45:06.567 INFO:teuthology.orchestra.run.smithi084.stdout: "min_size": 2, 2024-09-20T22:45:06.567 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_rule": 0, 2024-09-20T22:45:06.567 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_count": 0, 2024-09-20T22:45:06.567 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_target": 0, 2024-09-20T22:45:06.568 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_barrier": 0, 2024-09-20T22:45:06.568 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-20T22:45:06.568 INFO:teuthology.orchestra.run.smithi084.stdout: "is_stretch_pool": false, 2024-09-20T22:45:06.568 INFO:teuthology.orchestra.run.smithi084.stdout: "object_hash": 2, 2024-09-20T22:45:06.568 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_autoscale_mode": "off", 2024-09-20T22:45:06.568 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num": 3, 2024-09-20T22:45:06.568 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num": 3, 2024-09-20T22:45:06.568 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num_target": 3, 2024-09-20T22:45:06.568 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_target": 3, 2024-09-20T22:45:06.568 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_pending": 3, 2024-09-20T22:45:06.568 INFO:teuthology.orchestra.run.smithi084.stdout: "last_pg_merge_meta": { 2024-09-20T22:45:06.568 INFO:teuthology.orchestra.run.smithi084.stdout: "source_pgid": "0.0", 2024-09-20T22:45:06.568 INFO:teuthology.orchestra.run.smithi084.stdout: "ready_epoch": 0, 2024-09-20T22:45:06.568 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_started": 0, 2024-09-20T22:45:06.569 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_clean": 0, 2024-09-20T22:45:06.569 INFO:teuthology.orchestra.run.smithi084.stdout: "source_version": "0'0", 2024-09-20T22:45:06.569 INFO:teuthology.orchestra.run.smithi084.stdout: "target_version": "0'0" 2024-09-20T22:45:06.569 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:06.569 INFO:teuthology.orchestra.run.smithi084.stdout: "last_change": "69", 2024-09-20T22:45:06.569 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend": "0", 2024-09-20T22:45:06.569 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-20T22:45:06.569 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_preluminous": "0", 2024-09-20T22:45:06.569 INFO:teuthology.orchestra.run.smithi084.stdout: "auid": 0, 2024-09-20T22:45:06.569 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_mode": "selfmanaged", 2024-09-20T22:45:06.569 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_seq": 2, 2024-09-20T22:45:06.569 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_epoch": 69, 2024-09-20T22:45:06.569 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_snaps": [], 2024-09-20T22:45:06.569 INFO:teuthology.orchestra.run.smithi084.stdout: "removed_snaps": "[]", 2024-09-20T22:45:06.570 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_bytes": 0, 2024-09-20T22:45:06.570 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_objects": 0, 2024-09-20T22:45:06.570 INFO:teuthology.orchestra.run.smithi084.stdout: "tiers": [], 2024-09-20T22:45:06.570 INFO:teuthology.orchestra.run.smithi084.stdout: "tier_of": -1, 2024-09-20T22:45:06.570 INFO:teuthology.orchestra.run.smithi084.stdout: "read_tier": -1, 2024-09-20T22:45:06.570 INFO:teuthology.orchestra.run.smithi084.stdout: "write_tier": -1, 2024-09-20T22:45:06.570 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_mode": "none", 2024-09-20T22:45:06.570 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_bytes": 0, 2024-09-20T22:45:06.570 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_objects": 0, 2024-09-20T22:45:06.570 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-20T22:45:06.570 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-20T22:45:06.570 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-20T22:45:06.570 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_flush_age": 0, 2024-09-20T22:45:06.571 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_evict_age": 0, 2024-09-20T22:45:06.571 INFO:teuthology.orchestra.run.smithi084.stdout: "erasure_code_profile": "", 2024-09-20T22:45:06.571 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_params": { 2024-09-20T22:45:06.571 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "none" 2024-09-20T22:45:06.571 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:06.571 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_period": 0, 2024-09-20T22:45:06.571 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_count": 0, 2024-09-20T22:45:06.571 INFO:teuthology.orchestra.run.smithi084.stdout: "use_gmt_hitset": true, 2024-09-20T22:45:06.571 INFO:teuthology.orchestra.run.smithi084.stdout: "min_read_recency_for_promote": 0, 2024-09-20T22:45:06.571 INFO:teuthology.orchestra.run.smithi084.stdout: "min_write_recency_for_promote": 0, 2024-09-20T22:45:06.571 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_grade_decay_rate": 0, 2024-09-20T22:45:06.571 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_search_last_n": 0, 2024-09-20T22:45:06.571 INFO:teuthology.orchestra.run.smithi084.stdout: "grade_table": [], 2024-09-20T22:45:06.571 INFO:teuthology.orchestra.run.smithi084.stdout: "stripe_width": 0, 2024-09-20T22:45:06.571 INFO:teuthology.orchestra.run.smithi084.stdout: "expected_num_objects": 0, 2024-09-20T22:45:06.572 INFO:teuthology.orchestra.run.smithi084.stdout: "fast_read": false, 2024-09-20T22:45:06.572 INFO:teuthology.orchestra.run.smithi084.stdout: "options": {}, 2024-09-20T22:45:06.572 INFO:teuthology.orchestra.run.smithi084.stdout: "application_metadata": { 2024-09-20T22:45:06.572 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd": {} 2024-09-20T22:45:06.572 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:06.572 INFO:teuthology.orchestra.run.smithi084.stdout: "read_balance": { 2024-09-20T22:45:06.572 INFO:teuthology.orchestra.run.smithi084.stdout: "score_type": "Fair distribution", 2024-09-20T22:45:06.572 INFO:teuthology.orchestra.run.smithi084.stdout: "score_acting": 2.6500000953674316, 2024-09-20T22:45:06.572 INFO:teuthology.orchestra.run.smithi084.stdout: "score_stable": 2.6500000953674316, 2024-09-20T22:45:06.572 INFO:teuthology.orchestra.run.smithi084.stdout: "optimal_score": 0.87999999523162842, 2024-09-20T22:45:06.572 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_acting": 2.3299999237060547, 2024-09-20T22:45:06.572 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_stable": 2.3299999237060547, 2024-09-20T22:45:06.572 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity_weighted": 1, 2024-09-20T22:45:06.572 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity": 1, 2024-09-20T22:45:06.573 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity_weighted": 1 2024-09-20T22:45:06.573 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:06.573 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:06.573 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:06.573 INFO:teuthology.orchestra.run.smithi084.stdout: "pool": 3, 2024-09-20T22:45:06.573 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_name": ".rgw.root", 2024-09-20T22:45:06.573 INFO:teuthology.orchestra.run.smithi084.stdout: "create_time": "2024-09-20T22:41:36.546389+0000", 2024-09-20T22:45:06.573 INFO:teuthology.orchestra.run.smithi084.stdout: "flags": 1, 2024-09-20T22:45:06.573 INFO:teuthology.orchestra.run.smithi084.stdout: "flags_names": "hashpspool", 2024-09-20T22:45:06.573 INFO:teuthology.orchestra.run.smithi084.stdout: "type": 1, 2024-09-20T22:45:06.573 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 3, 2024-09-20T22:45:06.573 INFO:teuthology.orchestra.run.smithi084.stdout: "min_size": 2, 2024-09-20T22:45:06.573 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_rule": 0, 2024-09-20T22:45:06.573 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_count": 0, 2024-09-20T22:45:06.574 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_target": 0, 2024-09-20T22:45:06.574 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_barrier": 0, 2024-09-20T22:45:06.574 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-20T22:45:06.574 INFO:teuthology.orchestra.run.smithi084.stdout: "is_stretch_pool": false, 2024-09-20T22:45:06.574 INFO:teuthology.orchestra.run.smithi084.stdout: "object_hash": 2, 2024-09-20T22:45:06.574 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_autoscale_mode": "off", 2024-09-20T22:45:06.574 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num": 32, 2024-09-20T22:45:06.574 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num": 32, 2024-09-20T22:45:06.574 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num_target": 32, 2024-09-20T22:45:06.574 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_target": 32, 2024-09-20T22:45:06.574 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_pending": 32, 2024-09-20T22:45:06.574 INFO:teuthology.orchestra.run.smithi084.stdout: "last_pg_merge_meta": { 2024-09-20T22:45:06.574 INFO:teuthology.orchestra.run.smithi084.stdout: "source_pgid": "0.0", 2024-09-20T22:45:06.574 INFO:teuthology.orchestra.run.smithi084.stdout: "ready_epoch": 0, 2024-09-20T22:45:06.575 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_started": 0, 2024-09-20T22:45:06.575 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_clean": 0, 2024-09-20T22:45:06.575 INFO:teuthology.orchestra.run.smithi084.stdout: "source_version": "0'0", 2024-09-20T22:45:06.575 INFO:teuthology.orchestra.run.smithi084.stdout: "target_version": "0'0" 2024-09-20T22:45:06.575 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:06.576 INFO:teuthology.orchestra.run.smithi084.stdout: "last_change": "68", 2024-09-20T22:45:06.576 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend": "0", 2024-09-20T22:45:06.576 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-20T22:45:06.576 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_preluminous": "0", 2024-09-20T22:45:06.576 INFO:teuthology.orchestra.run.smithi084.stdout: "auid": 0, 2024-09-20T22:45:06.576 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_mode": "selfmanaged", 2024-09-20T22:45:06.576 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_seq": 0, 2024-09-20T22:45:06.576 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_epoch": 0, 2024-09-20T22:45:06.576 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_snaps": [], 2024-09-20T22:45:06.576 INFO:teuthology.orchestra.run.smithi084.stdout: "removed_snaps": "[]", 2024-09-20T22:45:06.576 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_bytes": 0, 2024-09-20T22:45:06.576 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_objects": 0, 2024-09-20T22:45:06.576 INFO:teuthology.orchestra.run.smithi084.stdout: "tiers": [], 2024-09-20T22:45:06.576 INFO:teuthology.orchestra.run.smithi084.stdout: "tier_of": -1, 2024-09-20T22:45:06.576 INFO:teuthology.orchestra.run.smithi084.stdout: "read_tier": -1, 2024-09-20T22:45:06.577 INFO:teuthology.orchestra.run.smithi084.stdout: "write_tier": -1, 2024-09-20T22:45:06.577 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_mode": "none", 2024-09-20T22:45:06.577 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_bytes": 0, 2024-09-20T22:45:06.577 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_objects": 0, 2024-09-20T22:45:06.577 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-20T22:45:06.577 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-20T22:45:06.577 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-20T22:45:06.577 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_flush_age": 0, 2024-09-20T22:45:06.577 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_evict_age": 0, 2024-09-20T22:45:06.577 INFO:teuthology.orchestra.run.smithi084.stdout: "erasure_code_profile": "", 2024-09-20T22:45:06.577 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_params": { 2024-09-20T22:45:06.577 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "none" 2024-09-20T22:45:06.577 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:06.578 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_period": 0, 2024-09-20T22:45:06.578 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_count": 0, 2024-09-20T22:45:06.578 INFO:teuthology.orchestra.run.smithi084.stdout: "use_gmt_hitset": true, 2024-09-20T22:45:06.578 INFO:teuthology.orchestra.run.smithi084.stdout: "min_read_recency_for_promote": 0, 2024-09-20T22:45:06.578 INFO:teuthology.orchestra.run.smithi084.stdout: "min_write_recency_for_promote": 0, 2024-09-20T22:45:06.578 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_grade_decay_rate": 0, 2024-09-20T22:45:06.578 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_search_last_n": 0, 2024-09-20T22:45:06.578 INFO:teuthology.orchestra.run.smithi084.stdout: "grade_table": [], 2024-09-20T22:45:06.578 INFO:teuthology.orchestra.run.smithi084.stdout: "stripe_width": 0, 2024-09-20T22:45:06.578 INFO:teuthology.orchestra.run.smithi084.stdout: "expected_num_objects": 0, 2024-09-20T22:45:06.578 INFO:teuthology.orchestra.run.smithi084.stdout: "fast_read": false, 2024-09-20T22:45:06.578 INFO:teuthology.orchestra.run.smithi084.stdout: "options": {}, 2024-09-20T22:45:06.578 INFO:teuthology.orchestra.run.smithi084.stdout: "application_metadata": { 2024-09-20T22:45:06.578 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw": {} 2024-09-20T22:45:06.579 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:06.579 INFO:teuthology.orchestra.run.smithi084.stdout: "read_balance": { 2024-09-20T22:45:06.579 INFO:teuthology.orchestra.run.smithi084.stdout: "score_type": "Fair distribution", 2024-09-20T22:45:06.579 INFO:teuthology.orchestra.run.smithi084.stdout: "score_acting": 1.7400000095367432, 2024-09-20T22:45:06.579 INFO:teuthology.orchestra.run.smithi084.stdout: "score_stable": 1.7400000095367432, 2024-09-20T22:45:06.579 INFO:teuthology.orchestra.run.smithi084.stdout: "optimal_score": 0.87999999523162842, 2024-09-20T22:45:06.579 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_acting": 1.5299999713897705, 2024-09-20T22:45:06.579 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_stable": 1.5299999713897705, 2024-09-20T22:45:06.579 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity_weighted": 1, 2024-09-20T22:45:06.579 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity": 1, 2024-09-20T22:45:06.579 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity_weighted": 1 2024-09-20T22:45:06.579 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:06.579 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:06.579 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:06.580 INFO:teuthology.orchestra.run.smithi084.stdout: "pool": 4, 2024-09-20T22:45:06.580 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_name": "default.rgw.log", 2024-09-20T22:45:06.580 INFO:teuthology.orchestra.run.smithi084.stdout: "create_time": "2024-09-20T22:41:38.454640+0000", 2024-09-20T22:45:06.580 INFO:teuthology.orchestra.run.smithi084.stdout: "flags": 1, 2024-09-20T22:45:06.580 INFO:teuthology.orchestra.run.smithi084.stdout: "flags_names": "hashpspool", 2024-09-20T22:45:06.580 INFO:teuthology.orchestra.run.smithi084.stdout: "type": 1, 2024-09-20T22:45:06.580 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 3, 2024-09-20T22:45:06.580 INFO:teuthology.orchestra.run.smithi084.stdout: "min_size": 2, 2024-09-20T22:45:06.580 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_rule": 0, 2024-09-20T22:45:06.580 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_count": 0, 2024-09-20T22:45:06.580 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_target": 0, 2024-09-20T22:45:06.580 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_barrier": 0, 2024-09-20T22:45:06.580 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-20T22:45:06.580 INFO:teuthology.orchestra.run.smithi084.stdout: "is_stretch_pool": false, 2024-09-20T22:45:06.580 INFO:teuthology.orchestra.run.smithi084.stdout: "object_hash": 2, 2024-09-20T22:45:06.581 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_autoscale_mode": "off", 2024-09-20T22:45:06.581 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num": 32, 2024-09-20T22:45:06.581 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num": 32, 2024-09-20T22:45:06.581 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num_target": 32, 2024-09-20T22:45:06.581 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_target": 32, 2024-09-20T22:45:06.581 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_pending": 32, 2024-09-20T22:45:06.581 INFO:teuthology.orchestra.run.smithi084.stdout: "last_pg_merge_meta": { 2024-09-20T22:45:06.581 INFO:teuthology.orchestra.run.smithi084.stdout: "source_pgid": "0.0", 2024-09-20T22:45:06.581 INFO:teuthology.orchestra.run.smithi084.stdout: "ready_epoch": 0, 2024-09-20T22:45:06.581 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_started": 0, 2024-09-20T22:45:06.581 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_clean": 0, 2024-09-20T22:45:06.581 INFO:teuthology.orchestra.run.smithi084.stdout: "source_version": "0'0", 2024-09-20T22:45:06.581 INFO:teuthology.orchestra.run.smithi084.stdout: "target_version": "0'0" 2024-09-20T22:45:06.582 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:06.582 INFO:teuthology.orchestra.run.smithi084.stdout: "last_change": "70", 2024-09-20T22:45:06.582 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend": "0", 2024-09-20T22:45:06.582 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-20T22:45:06.582 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_preluminous": "0", 2024-09-20T22:45:06.582 INFO:teuthology.orchestra.run.smithi084.stdout: "auid": 0, 2024-09-20T22:45:06.582 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_mode": "selfmanaged", 2024-09-20T22:45:06.582 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_seq": 0, 2024-09-20T22:45:06.582 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_epoch": 0, 2024-09-20T22:45:06.582 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_snaps": [], 2024-09-20T22:45:06.582 INFO:teuthology.orchestra.run.smithi084.stdout: "removed_snaps": "[]", 2024-09-20T22:45:06.582 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_bytes": 0, 2024-09-20T22:45:06.582 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_objects": 0, 2024-09-20T22:45:06.583 INFO:teuthology.orchestra.run.smithi084.stdout: "tiers": [], 2024-09-20T22:45:06.583 INFO:teuthology.orchestra.run.smithi084.stdout: "tier_of": -1, 2024-09-20T22:45:06.583 INFO:teuthology.orchestra.run.smithi084.stdout: "read_tier": -1, 2024-09-20T22:45:06.583 INFO:teuthology.orchestra.run.smithi084.stdout: "write_tier": -1, 2024-09-20T22:45:06.583 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_mode": "none", 2024-09-20T22:45:06.583 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_bytes": 0, 2024-09-20T22:45:06.583 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_objects": 0, 2024-09-20T22:45:06.583 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-20T22:45:06.583 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-20T22:45:06.583 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-20T22:45:06.583 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_flush_age": 0, 2024-09-20T22:45:06.583 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_evict_age": 0, 2024-09-20T22:45:06.583 INFO:teuthology.orchestra.run.smithi084.stdout: "erasure_code_profile": "", 2024-09-20T22:45:06.583 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_params": { 2024-09-20T22:45:06.583 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "none" 2024-09-20T22:45:06.584 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:06.584 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_period": 0, 2024-09-20T22:45:06.584 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_count": 0, 2024-09-20T22:45:06.584 INFO:teuthology.orchestra.run.smithi084.stdout: "use_gmt_hitset": true, 2024-09-20T22:45:06.584 INFO:teuthology.orchestra.run.smithi084.stdout: "min_read_recency_for_promote": 0, 2024-09-20T22:45:06.584 INFO:teuthology.orchestra.run.smithi084.stdout: "min_write_recency_for_promote": 0, 2024-09-20T22:45:06.584 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_grade_decay_rate": 0, 2024-09-20T22:45:06.584 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_search_last_n": 0, 2024-09-20T22:45:06.584 INFO:teuthology.orchestra.run.smithi084.stdout: "grade_table": [], 2024-09-20T22:45:06.585 INFO:teuthology.orchestra.run.smithi084.stdout: "stripe_width": 0, 2024-09-20T22:45:06.585 INFO:teuthology.orchestra.run.smithi084.stdout: "expected_num_objects": 0, 2024-09-20T22:45:06.585 INFO:teuthology.orchestra.run.smithi084.stdout: "fast_read": false, 2024-09-20T22:45:06.585 INFO:teuthology.orchestra.run.smithi084.stdout: "options": {}, 2024-09-20T22:45:06.586 INFO:teuthology.orchestra.run.smithi084.stdout: "application_metadata": { 2024-09-20T22:45:06.586 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw": {} 2024-09-20T22:45:06.586 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:06.586 INFO:teuthology.orchestra.run.smithi084.stdout: "read_balance": { 2024-09-20T22:45:06.586 INFO:teuthology.orchestra.run.smithi084.stdout: "score_type": "Fair distribution", 2024-09-20T22:45:06.586 INFO:teuthology.orchestra.run.smithi084.stdout: "score_acting": 2.2400000095367432, 2024-09-20T22:45:06.586 INFO:teuthology.orchestra.run.smithi084.stdout: "score_stable": 2.2400000095367432, 2024-09-20T22:45:06.586 INFO:teuthology.orchestra.run.smithi084.stdout: "optimal_score": 0.87999999523162842, 2024-09-20T22:45:06.586 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_acting": 1.9700000286102295, 2024-09-20T22:45:06.586 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_stable": 1.9700000286102295, 2024-09-20T22:45:06.586 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity_weighted": 1, 2024-09-20T22:45:06.586 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity": 1, 2024-09-20T22:45:06.586 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity_weighted": 1 2024-09-20T22:45:06.586 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:06.586 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:06.587 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:06.587 INFO:teuthology.orchestra.run.smithi084.stdout: "pool": 5, 2024-09-20T22:45:06.587 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_name": "default.rgw.control", 2024-09-20T22:45:06.587 INFO:teuthology.orchestra.run.smithi084.stdout: "create_time": "2024-09-20T22:41:40.407129+0000", 2024-09-20T22:45:06.587 INFO:teuthology.orchestra.run.smithi084.stdout: "flags": 1, 2024-09-20T22:45:06.587 INFO:teuthology.orchestra.run.smithi084.stdout: "flags_names": "hashpspool", 2024-09-20T22:45:06.587 INFO:teuthology.orchestra.run.smithi084.stdout: "type": 1, 2024-09-20T22:45:06.587 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 3, 2024-09-20T22:45:06.587 INFO:teuthology.orchestra.run.smithi084.stdout: "min_size": 2, 2024-09-20T22:45:06.587 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_rule": 0, 2024-09-20T22:45:06.587 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_count": 0, 2024-09-20T22:45:06.587 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_target": 0, 2024-09-20T22:45:06.587 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_barrier": 0, 2024-09-20T22:45:06.587 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-20T22:45:06.588 INFO:teuthology.orchestra.run.smithi084.stdout: "is_stretch_pool": false, 2024-09-20T22:45:06.588 INFO:teuthology.orchestra.run.smithi084.stdout: "object_hash": 2, 2024-09-20T22:45:06.588 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_autoscale_mode": "off", 2024-09-20T22:45:06.588 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num": 32, 2024-09-20T22:45:06.588 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num": 32, 2024-09-20T22:45:06.588 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num_target": 32, 2024-09-20T22:45:06.588 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_target": 32, 2024-09-20T22:45:06.588 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_pending": 32, 2024-09-20T22:45:06.588 INFO:teuthology.orchestra.run.smithi084.stdout: "last_pg_merge_meta": { 2024-09-20T22:45:06.588 INFO:teuthology.orchestra.run.smithi084.stdout: "source_pgid": "0.0", 2024-09-20T22:45:06.588 INFO:teuthology.orchestra.run.smithi084.stdout: "ready_epoch": 0, 2024-09-20T22:45:06.588 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_started": 0, 2024-09-20T22:45:06.588 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_clean": 0, 2024-09-20T22:45:06.588 INFO:teuthology.orchestra.run.smithi084.stdout: "source_version": "0'0", 2024-09-20T22:45:06.588 INFO:teuthology.orchestra.run.smithi084.stdout: "target_version": "0'0" 2024-09-20T22:45:06.589 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:06.589 INFO:teuthology.orchestra.run.smithi084.stdout: "last_change": "72", 2024-09-20T22:45:06.589 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend": "0", 2024-09-20T22:45:06.589 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-20T22:45:06.589 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_preluminous": "0", 2024-09-20T22:45:06.589 INFO:teuthology.orchestra.run.smithi084.stdout: "auid": 0, 2024-09-20T22:45:06.589 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_mode": "selfmanaged", 2024-09-20T22:45:06.589 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_seq": 0, 2024-09-20T22:45:06.589 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_epoch": 0, 2024-09-20T22:45:06.589 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_snaps": [], 2024-09-20T22:45:06.589 INFO:teuthology.orchestra.run.smithi084.stdout: "removed_snaps": "[]", 2024-09-20T22:45:06.589 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_bytes": 0, 2024-09-20T22:45:06.589 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_objects": 0, 2024-09-20T22:45:06.589 INFO:teuthology.orchestra.run.smithi084.stdout: "tiers": [], 2024-09-20T22:45:06.589 INFO:teuthology.orchestra.run.smithi084.stdout: "tier_of": -1, 2024-09-20T22:45:06.589 INFO:teuthology.orchestra.run.smithi084.stdout: "read_tier": -1, 2024-09-20T22:45:06.590 INFO:teuthology.orchestra.run.smithi084.stdout: "write_tier": -1, 2024-09-20T22:45:06.590 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_mode": "none", 2024-09-20T22:45:06.590 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_bytes": 0, 2024-09-20T22:45:06.590 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_objects": 0, 2024-09-20T22:45:06.590 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-20T22:45:06.590 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-20T22:45:06.590 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-20T22:45:06.590 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_flush_age": 0, 2024-09-20T22:45:06.590 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_evict_age": 0, 2024-09-20T22:45:06.590 INFO:teuthology.orchestra.run.smithi084.stdout: "erasure_code_profile": "", 2024-09-20T22:45:06.590 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_params": { 2024-09-20T22:45:06.590 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "none" 2024-09-20T22:45:06.590 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:06.590 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_period": 0, 2024-09-20T22:45:06.591 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_count": 0, 2024-09-20T22:45:06.591 INFO:teuthology.orchestra.run.smithi084.stdout: "use_gmt_hitset": true, 2024-09-20T22:45:06.591 INFO:teuthology.orchestra.run.smithi084.stdout: "min_read_recency_for_promote": 0, 2024-09-20T22:45:06.591 INFO:teuthology.orchestra.run.smithi084.stdout: "min_write_recency_for_promote": 0, 2024-09-20T22:45:06.591 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_grade_decay_rate": 0, 2024-09-20T22:45:06.591 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_search_last_n": 0, 2024-09-20T22:45:06.591 INFO:teuthology.orchestra.run.smithi084.stdout: "grade_table": [], 2024-09-20T22:45:06.591 INFO:teuthology.orchestra.run.smithi084.stdout: "stripe_width": 0, 2024-09-20T22:45:06.591 INFO:teuthology.orchestra.run.smithi084.stdout: "expected_num_objects": 0, 2024-09-20T22:45:06.591 INFO:teuthology.orchestra.run.smithi084.stdout: "fast_read": false, 2024-09-20T22:45:06.591 INFO:teuthology.orchestra.run.smithi084.stdout: "options": {}, 2024-09-20T22:45:06.591 INFO:teuthology.orchestra.run.smithi084.stdout: "application_metadata": { 2024-09-20T22:45:06.591 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw": {} 2024-09-20T22:45:06.591 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:06.591 INFO:teuthology.orchestra.run.smithi084.stdout: "read_balance": { 2024-09-20T22:45:06.592 INFO:teuthology.orchestra.run.smithi084.stdout: "score_type": "Fair distribution", 2024-09-20T22:45:06.592 INFO:teuthology.orchestra.run.smithi084.stdout: "score_acting": 1.7400000095367432, 2024-09-20T22:45:06.592 INFO:teuthology.orchestra.run.smithi084.stdout: "score_stable": 1.7400000095367432, 2024-09-20T22:45:06.592 INFO:teuthology.orchestra.run.smithi084.stdout: "optimal_score": 0.87999999523162842, 2024-09-20T22:45:06.592 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_acting": 1.5299999713897705, 2024-09-20T22:45:06.592 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_stable": 1.5299999713897705, 2024-09-20T22:45:06.592 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity_weighted": 1, 2024-09-20T22:45:06.592 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity": 1, 2024-09-20T22:45:06.592 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity_weighted": 1 2024-09-20T22:45:06.592 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:06.592 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:06.592 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:06.592 INFO:teuthology.orchestra.run.smithi084.stdout: "pool": 6, 2024-09-20T22:45:06.593 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_name": "default.rgw.meta", 2024-09-20T22:45:06.593 INFO:teuthology.orchestra.run.smithi084.stdout: "create_time": "2024-09-20T22:41:42.476716+0000", 2024-09-20T22:45:06.593 INFO:teuthology.orchestra.run.smithi084.stdout: "flags": 1, 2024-09-20T22:45:06.593 INFO:teuthology.orchestra.run.smithi084.stdout: "flags_names": "hashpspool", 2024-09-20T22:45:06.593 INFO:teuthology.orchestra.run.smithi084.stdout: "type": 1, 2024-09-20T22:45:06.593 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 3, 2024-09-20T22:45:06.593 INFO:teuthology.orchestra.run.smithi084.stdout: "min_size": 2, 2024-09-20T22:45:06.593 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_rule": 0, 2024-09-20T22:45:06.593 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_count": 0, 2024-09-20T22:45:06.593 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_target": 0, 2024-09-20T22:45:06.593 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_barrier": 0, 2024-09-20T22:45:06.593 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-20T22:45:06.593 INFO:teuthology.orchestra.run.smithi084.stdout: "is_stretch_pool": false, 2024-09-20T22:45:06.594 INFO:teuthology.orchestra.run.smithi084.stdout: "object_hash": 2, 2024-09-20T22:45:06.594 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_autoscale_mode": "off", 2024-09-20T22:45:06.594 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num": 32, 2024-09-20T22:45:06.594 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num": 32, 2024-09-20T22:45:06.594 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num_target": 32, 2024-09-20T22:45:06.594 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_target": 32, 2024-09-20T22:45:06.594 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_pending": 32, 2024-09-20T22:45:06.594 INFO:teuthology.orchestra.run.smithi084.stdout: "last_pg_merge_meta": { 2024-09-20T22:45:06.594 INFO:teuthology.orchestra.run.smithi084.stdout: "source_pgid": "0.0", 2024-09-20T22:45:06.594 INFO:teuthology.orchestra.run.smithi084.stdout: "ready_epoch": 0, 2024-09-20T22:45:06.594 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_started": 0, 2024-09-20T22:45:06.594 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_clean": 0, 2024-09-20T22:45:06.594 INFO:teuthology.orchestra.run.smithi084.stdout: "source_version": "0'0", 2024-09-20T22:45:06.594 INFO:teuthology.orchestra.run.smithi084.stdout: "target_version": "0'0" 2024-09-20T22:45:06.595 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:06.595 INFO:teuthology.orchestra.run.smithi084.stdout: "last_change": "74", 2024-09-20T22:45:06.595 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend": "0", 2024-09-20T22:45:06.595 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-20T22:45:06.595 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_preluminous": "0", 2024-09-20T22:45:06.595 INFO:teuthology.orchestra.run.smithi084.stdout: "auid": 0, 2024-09-20T22:45:06.595 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_mode": "selfmanaged", 2024-09-20T22:45:06.595 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_seq": 0, 2024-09-20T22:45:06.595 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_epoch": 0, 2024-09-20T22:45:06.595 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_snaps": [], 2024-09-20T22:45:06.595 INFO:teuthology.orchestra.run.smithi084.stdout: "removed_snaps": "[]", 2024-09-20T22:45:06.595 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_bytes": 0, 2024-09-20T22:45:06.595 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_objects": 0, 2024-09-20T22:45:06.595 INFO:teuthology.orchestra.run.smithi084.stdout: "tiers": [], 2024-09-20T22:45:06.596 INFO:teuthology.orchestra.run.smithi084.stdout: "tier_of": -1, 2024-09-20T22:45:06.596 INFO:teuthology.orchestra.run.smithi084.stdout: "read_tier": -1, 2024-09-20T22:45:06.596 INFO:teuthology.orchestra.run.smithi084.stdout: "write_tier": -1, 2024-09-20T22:45:06.596 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_mode": "none", 2024-09-20T22:45:06.596 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_bytes": 0, 2024-09-20T22:45:06.596 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_objects": 0, 2024-09-20T22:45:06.596 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-20T22:45:06.596 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-20T22:45:06.596 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-20T22:45:06.596 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_flush_age": 0, 2024-09-20T22:45:06.596 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_evict_age": 0, 2024-09-20T22:45:06.596 INFO:teuthology.orchestra.run.smithi084.stdout: "erasure_code_profile": "", 2024-09-20T22:45:06.596 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_params": { 2024-09-20T22:45:06.596 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "none" 2024-09-20T22:45:06.597 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:06.597 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_period": 0, 2024-09-20T22:45:06.597 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_count": 0, 2024-09-20T22:45:06.597 INFO:teuthology.orchestra.run.smithi084.stdout: "use_gmt_hitset": true, 2024-09-20T22:45:06.597 INFO:teuthology.orchestra.run.smithi084.stdout: "min_read_recency_for_promote": 0, 2024-09-20T22:45:06.597 INFO:teuthology.orchestra.run.smithi084.stdout: "min_write_recency_for_promote": 0, 2024-09-20T22:45:06.597 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_grade_decay_rate": 0, 2024-09-20T22:45:06.597 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_search_last_n": 0, 2024-09-20T22:45:06.597 INFO:teuthology.orchestra.run.smithi084.stdout: "grade_table": [], 2024-09-20T22:45:06.597 INFO:teuthology.orchestra.run.smithi084.stdout: "stripe_width": 0, 2024-09-20T22:45:06.597 INFO:teuthology.orchestra.run.smithi084.stdout: "expected_num_objects": 0, 2024-09-20T22:45:06.597 INFO:teuthology.orchestra.run.smithi084.stdout: "fast_read": false, 2024-09-20T22:45:06.598 INFO:teuthology.orchestra.run.smithi084.stdout: "options": { 2024-09-20T22:45:06.598 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_autoscale_bias": 4 2024-09-20T22:45:06.598 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:06.598 INFO:teuthology.orchestra.run.smithi084.stdout: "application_metadata": { 2024-09-20T22:45:06.598 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw": {} 2024-09-20T22:45:06.598 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:06.598 INFO:teuthology.orchestra.run.smithi084.stdout: "read_balance": { 2024-09-20T22:45:06.598 INFO:teuthology.orchestra.run.smithi084.stdout: "score_type": "Fair distribution", 2024-09-20T22:45:06.598 INFO:teuthology.orchestra.run.smithi084.stdout: "score_acting": 1.9900000095367432, 2024-09-20T22:45:06.598 INFO:teuthology.orchestra.run.smithi084.stdout: "score_stable": 1.9900000095367432, 2024-09-20T22:45:06.598 INFO:teuthology.orchestra.run.smithi084.stdout: "optimal_score": 0.87999999523162842, 2024-09-20T22:45:06.598 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_acting": 1.75, 2024-09-20T22:45:06.598 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_stable": 1.75, 2024-09-20T22:45:06.599 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity_weighted": 1, 2024-09-20T22:45:06.599 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity": 1, 2024-09-20T22:45:06.599 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity_weighted": 1 2024-09-20T22:45:06.599 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:06.599 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:06.599 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:06.599 INFO:teuthology.orchestra.run.smithi084.stdout: "pool": 7, 2024-09-20T22:45:06.599 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_name": "unique_pool_0", 2024-09-20T22:45:06.599 INFO:teuthology.orchestra.run.smithi084.stdout: "create_time": "2024-09-20T22:42:51.354358+0000", 2024-09-20T22:45:06.599 INFO:teuthology.orchestra.run.smithi084.stdout: "flags": 8193, 2024-09-20T22:45:06.599 INFO:teuthology.orchestra.run.smithi084.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-20T22:45:06.599 INFO:teuthology.orchestra.run.smithi084.stdout: "type": 1, 2024-09-20T22:45:06.599 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 3, 2024-09-20T22:45:06.600 INFO:teuthology.orchestra.run.smithi084.stdout: "min_size": 2, 2024-09-20T22:45:06.600 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_rule": 0, 2024-09-20T22:45:06.600 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_count": 0, 2024-09-20T22:45:06.600 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_target": 0, 2024-09-20T22:45:06.600 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_barrier": 0, 2024-09-20T22:45:06.600 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-20T22:45:06.600 INFO:teuthology.orchestra.run.smithi084.stdout: "is_stretch_pool": false, 2024-09-20T22:45:06.600 INFO:teuthology.orchestra.run.smithi084.stdout: "object_hash": 2, 2024-09-20T22:45:06.600 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_autoscale_mode": "off", 2024-09-20T22:45:06.600 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num": 26, 2024-09-20T22:45:06.600 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num": 26, 2024-09-20T22:45:06.600 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num_target": 26, 2024-09-20T22:45:06.600 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_target": 26, 2024-09-20T22:45:06.601 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_pending": 26, 2024-09-20T22:45:06.601 INFO:teuthology.orchestra.run.smithi084.stdout: "last_pg_merge_meta": { 2024-09-20T22:45:06.601 INFO:teuthology.orchestra.run.smithi084.stdout: "source_pgid": "0.0", 2024-09-20T22:45:06.601 INFO:teuthology.orchestra.run.smithi084.stdout: "ready_epoch": 0, 2024-09-20T22:45:06.601 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_started": 0, 2024-09-20T22:45:06.601 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_clean": 0, 2024-09-20T22:45:06.601 INFO:teuthology.orchestra.run.smithi084.stdout: "source_version": "0'0", 2024-09-20T22:45:06.601 INFO:teuthology.orchestra.run.smithi084.stdout: "target_version": "0'0" 2024-09-20T22:45:06.601 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:06.601 INFO:teuthology.orchestra.run.smithi084.stdout: "last_change": "227", 2024-09-20T22:45:06.601 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend": "0", 2024-09-20T22:45:06.601 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-20T22:45:06.601 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_preluminous": "137", 2024-09-20T22:45:06.601 INFO:teuthology.orchestra.run.smithi084.stdout: "auid": 0, 2024-09-20T22:45:06.601 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_mode": "selfmanaged", 2024-09-20T22:45:06.602 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_seq": 69, 2024-09-20T22:45:06.602 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_epoch": 227, 2024-09-20T22:45:06.602 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_snaps": [], 2024-09-20T22:45:06.602 INFO:teuthology.orchestra.run.smithi084.stdout: "removed_snaps": "[]", 2024-09-20T22:45:06.602 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_bytes": 0, 2024-09-20T22:45:06.602 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_objects": 0, 2024-09-20T22:45:06.602 INFO:teuthology.orchestra.run.smithi084.stdout: "tiers": [], 2024-09-20T22:45:06.602 INFO:teuthology.orchestra.run.smithi084.stdout: "tier_of": -1, 2024-09-20T22:45:06.602 INFO:teuthology.orchestra.run.smithi084.stdout: "read_tier": -1, 2024-09-20T22:45:06.602 INFO:teuthology.orchestra.run.smithi084.stdout: "write_tier": -1, 2024-09-20T22:45:06.602 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_mode": "none", 2024-09-20T22:45:06.602 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_bytes": 0, 2024-09-20T22:45:06.602 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_objects": 0, 2024-09-20T22:45:06.603 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-20T22:45:06.603 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-20T22:45:06.603 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-20T22:45:06.603 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_flush_age": 0, 2024-09-20T22:45:06.603 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_evict_age": 0, 2024-09-20T22:45:06.603 INFO:teuthology.orchestra.run.smithi084.stdout: "erasure_code_profile": "", 2024-09-20T22:45:06.603 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_params": { 2024-09-20T22:45:06.603 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "none" 2024-09-20T22:45:06.603 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:06.603 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_period": 0, 2024-09-20T22:45:06.603 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_count": 0, 2024-09-20T22:45:06.603 INFO:teuthology.orchestra.run.smithi084.stdout: "use_gmt_hitset": true, 2024-09-20T22:45:06.603 INFO:teuthology.orchestra.run.smithi084.stdout: "min_read_recency_for_promote": 0, 2024-09-20T22:45:06.604 INFO:teuthology.orchestra.run.smithi084.stdout: "min_write_recency_for_promote": 0, 2024-09-20T22:45:06.604 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_grade_decay_rate": 0, 2024-09-20T22:45:06.604 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_search_last_n": 0, 2024-09-20T22:45:06.604 INFO:teuthology.orchestra.run.smithi084.stdout: "grade_table": [], 2024-09-20T22:45:06.604 INFO:teuthology.orchestra.run.smithi084.stdout: "stripe_width": 0, 2024-09-20T22:45:06.604 INFO:teuthology.orchestra.run.smithi084.stdout: "expected_num_objects": 0, 2024-09-20T22:45:06.604 INFO:teuthology.orchestra.run.smithi084.stdout: "fast_read": false, 2024-09-20T22:45:06.604 INFO:teuthology.orchestra.run.smithi084.stdout: "options": {}, 2024-09-20T22:45:06.604 INFO:teuthology.orchestra.run.smithi084.stdout: "application_metadata": { 2024-09-20T22:45:06.604 INFO:teuthology.orchestra.run.smithi084.stdout: "rados": {} 2024-09-20T22:45:06.604 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:06.604 INFO:teuthology.orchestra.run.smithi084.stdout: "read_balance": { 2024-09-20T22:45:06.604 INFO:teuthology.orchestra.run.smithi084.stdout: "score_type": "Fair distribution", 2024-09-20T22:45:06.604 INFO:teuthology.orchestra.run.smithi084.stdout: "score_acting": 2.1400001049041748, 2024-09-20T22:45:06.604 INFO:teuthology.orchestra.run.smithi084.stdout: "score_stable": 2.1400001049041748, 2024-09-20T22:45:06.605 INFO:teuthology.orchestra.run.smithi084.stdout: "optimal_score": 0.87999999523162842, 2024-09-20T22:45:06.605 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_acting": 1.8799999952316284, 2024-09-20T22:45:06.605 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_stable": 1.8799999952316284, 2024-09-20T22:45:06.605 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity_weighted": 1, 2024-09-20T22:45:06.605 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity": 1, 2024-09-20T22:45:06.605 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity_weighted": 1 2024-09-20T22:45:06.605 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:06.605 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:06.605 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2024-09-20T22:45:06.605 INFO:teuthology.orchestra.run.smithi084.stdout: "osds": [ 2024-09-20T22:45:06.605 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:06.605 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 0, 2024-09-20T22:45:06.605 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "ea36a489-6f91-4b2f-b784-b90506b9805d", 2024-09-20T22:45:06.605 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-09-20T22:45:06.606 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-09-20T22:45:06.606 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-09-20T22:45:06.606 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-09-20T22:45:06.606 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-09-20T22:45:06.606 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-09-20T22:45:06.606 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 8, 2024-09-20T22:45:06.606 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 169, 2024-09-20T22:45:06.606 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-09-20T22:45:06.606 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-09-20T22:45:06.606 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-09-20T22:45:06.606 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:45:06.606 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:06.606 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:45:06.606 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6801", 2024-09-20T22:45:06.607 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 268084626 2024-09-20T22:45:06.607 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:06.607 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:45:06.607 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:06.607 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-09-20T22:45:06.607 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:45:06.607 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:06.607 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:45:06.607 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6802", 2024-09-20T22:45:06.607 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 268084626 2024-09-20T22:45:06.607 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:06.607 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:45:06.607 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:06.607 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-09-20T22:45:06.607 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:45:06.608 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:06.608 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:45:06.608 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6804", 2024-09-20T22:45:06.608 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 268084626 2024-09-20T22:45:06.608 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:06.608 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:45:06.608 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:06.608 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-09-20T22:45:06.608 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:45:06.608 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:06.608 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:45:06.608 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6803", 2024-09-20T22:45:06.608 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 268084626 2024-09-20T22:45:06.609 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:06.609 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:45:06.609 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:06.609 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.84:6801/268084626", 2024-09-20T22:45:06.609 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.84:6802/268084626", 2024-09-20T22:45:06.609 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:06.609 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.84:6804/268084626", 2024-09-20T22:45:06.610 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.84:6803/268084626", 2024-09-20T22:45:06.610 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-09-20T22:45:06.610 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-09-20T22:45:06.610 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-09-20T22:45:06.610 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:45:06.610 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:06.610 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:06.610 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 1, 2024-09-20T22:45:06.610 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "c8649d2f-27ef-47f8-bc94-f1cd9e62bdb8", 2024-09-20T22:45:06.610 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-09-20T22:45:06.610 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-09-20T22:45:06.610 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-09-20T22:45:06.610 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-09-20T22:45:06.610 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-09-20T22:45:06.611 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-09-20T22:45:06.611 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 13, 2024-09-20T22:45:06.611 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 160, 2024-09-20T22:45:06.611 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-09-20T22:45:06.611 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-09-20T22:45:06.611 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-09-20T22:45:06.611 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:45:06.611 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:06.611 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:45:06.611 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6805", 2024-09-20T22:45:06.611 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 418374136 2024-09-20T22:45:06.611 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:06.611 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:45:06.611 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:06.611 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-09-20T22:45:06.612 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:45:06.612 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:06.612 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:45:06.612 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6806", 2024-09-20T22:45:06.612 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 418374136 2024-09-20T22:45:06.612 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:06.612 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:45:06.612 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:06.612 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-09-20T22:45:06.612 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:45:06.612 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:06.612 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:45:06.612 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6808", 2024-09-20T22:45:06.612 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 418374136 2024-09-20T22:45:06.612 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:06.613 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:45:06.613 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:06.613 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-09-20T22:45:06.613 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:45:06.613 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:06.613 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:45:06.613 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6807", 2024-09-20T22:45:06.613 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 418374136 2024-09-20T22:45:06.613 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:06.613 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:45:06.613 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:06.613 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.84:6805/418374136", 2024-09-20T22:45:06.613 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.84:6806/418374136", 2024-09-20T22:45:06.613 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.84:6808/418374136", 2024-09-20T22:45:06.613 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.84:6807/418374136", 2024-09-20T22:45:06.614 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-09-20T22:45:06.614 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-09-20T22:45:06.614 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-09-20T22:45:06.614 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:45:06.614 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:06.614 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:06.614 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 2, 2024-09-20T22:45:06.614 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "c728eb48-a0f0-4de4-a651-7f3bec867280", 2024-09-20T22:45:06.614 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-09-20T22:45:06.614 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-09-20T22:45:06.614 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-09-20T22:45:06.614 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-09-20T22:45:06.614 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-09-20T22:45:06.614 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-09-20T22:45:06.615 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 18, 2024-09-20T22:45:06.615 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 160, 2024-09-20T22:45:06.615 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-09-20T22:45:06.615 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-09-20T22:45:06.615 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-09-20T22:45:06.615 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:45:06.615 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:06.615 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:45:06.615 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6809", 2024-09-20T22:45:06.615 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3977168611 2024-09-20T22:45:06.615 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:06.615 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:45:06.615 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:06.615 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-09-20T22:45:06.616 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:45:06.616 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:06.616 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:45:06.616 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6810", 2024-09-20T22:45:06.616 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3977168611 2024-09-20T22:45:06.616 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:06.616 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:45:06.616 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:06.616 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-09-20T22:45:06.616 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:45:06.616 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:06.616 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:45:06.616 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6812", 2024-09-20T22:45:06.616 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3977168611 2024-09-20T22:45:06.617 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:06.617 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:45:06.617 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:06.617 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-09-20T22:45:06.617 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:45:06.617 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:06.617 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:45:06.617 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6811", 2024-09-20T22:45:06.617 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3977168611 2024-09-20T22:45:06.617 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:06.617 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:45:06.617 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:06.618 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.84:6809/3977168611", 2024-09-20T22:45:06.618 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.84:6810/3977168611", 2024-09-20T22:45:06.618 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.84:6812/3977168611", 2024-09-20T22:45:06.618 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.84:6811/3977168611", 2024-09-20T22:45:06.618 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-09-20T22:45:06.618 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-09-20T22:45:06.618 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-09-20T22:45:06.618 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:45:06.618 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:06.618 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:06.618 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 3, 2024-09-20T22:45:06.618 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "573f44eb-0853-4630-a7d8-9caaeb256f9b", 2024-09-20T22:45:06.618 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-09-20T22:45:06.619 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-09-20T22:45:06.619 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-09-20T22:45:06.619 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-09-20T22:45:06.619 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-09-20T22:45:06.619 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-09-20T22:45:06.619 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 25, 2024-09-20T22:45:06.619 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 219, 2024-09-20T22:45:06.619 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-09-20T22:45:06.619 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-09-20T22:45:06.619 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-09-20T22:45:06.619 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:45:06.619 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:06.619 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:45:06.619 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6813", 2024-09-20T22:45:06.619 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2192239945 2024-09-20T22:45:06.620 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:06.620 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:45:06.620 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:06.620 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-09-20T22:45:06.620 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:45:06.620 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:06.620 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:45:06.620 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6814", 2024-09-20T22:45:06.620 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2192239945 2024-09-20T22:45:06.620 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:06.620 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:45:06.620 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:06.620 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-09-20T22:45:06.620 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:45:06.621 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:06.621 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:45:06.621 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6816", 2024-09-20T22:45:06.621 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2192239945 2024-09-20T22:45:06.621 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:06.621 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:45:06.621 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:06.621 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-09-20T22:45:06.621 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:45:06.621 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:06.621 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:45:06.621 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6815", 2024-09-20T22:45:06.621 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2192239945 2024-09-20T22:45:06.621 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:06.622 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:45:06.622 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:06.622 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.84:6813/2192239945", 2024-09-20T22:45:06.622 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.84:6814/2192239945", 2024-09-20T22:45:06.622 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.84:6816/2192239945", 2024-09-20T22:45:06.622 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.84:6815/2192239945", 2024-09-20T22:45:06.622 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-09-20T22:45:06.622 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-09-20T22:45:06.622 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-09-20T22:45:06.622 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:45:06.622 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:06.622 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:06.623 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 4, 2024-09-20T22:45:06.623 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "a52c3b94-c5f3-490f-8ac8-09061b9b2abb", 2024-09-20T22:45:06.623 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-09-20T22:45:06.623 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 0, 2024-09-20T22:45:06.623 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 0, 2024-09-20T22:45:06.623 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-09-20T22:45:06.623 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-09-20T22:45:06.623 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-09-20T22:45:06.623 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 31, 2024-09-20T22:45:06.623 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 139, 2024-09-20T22:45:06.623 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-09-20T22:45:06.623 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-09-20T22:45:06.623 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-09-20T22:45:06.623 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:45:06.623 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:06.624 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:45:06.624 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.154:6800", 2024-09-20T22:45:06.624 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1793136002 2024-09-20T22:45:06.624 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:06.624 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:45:06.624 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:06.624 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-09-20T22:45:06.624 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:45:06.624 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:06.624 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:45:06.624 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.154:6801", 2024-09-20T22:45:06.624 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1793136002 2024-09-20T22:45:06.624 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:06.624 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:45:06.625 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:06.625 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-09-20T22:45:06.625 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:45:06.625 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:06.625 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:45:06.625 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.154:6803", 2024-09-20T22:45:06.625 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1793136002 2024-09-20T22:45:06.625 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:06.625 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:45:06.625 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:06.625 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-09-20T22:45:06.625 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:45:06.625 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:06.625 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:45:06.625 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.154:6802", 2024-09-20T22:45:06.625 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1793136002 2024-09-20T22:45:06.626 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:06.626 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:45:06.626 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:06.626 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.154:6800/1793136002", 2024-09-20T22:45:06.626 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.154:6801/1793136002", 2024-09-20T22:45:06.626 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.154:6803/1793136002", 2024-09-20T22:45:06.626 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.154:6802/1793136002", 2024-09-20T22:45:06.626 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-09-20T22:45:06.626 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-09-20T22:45:06.626 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-09-20T22:45:06.626 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:45:06.626 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:06.626 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:06.626 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 5, 2024-09-20T22:45:06.627 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "4fa0dc8e-25cc-46cd-96ff-33f20b6faec3", 2024-09-20T22:45:06.627 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-09-20T22:45:06.627 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-09-20T22:45:06.627 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-09-20T22:45:06.627 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-09-20T22:45:06.627 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-09-20T22:45:06.627 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-09-20T22:45:06.627 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 40, 2024-09-20T22:45:06.627 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 169, 2024-09-20T22:45:06.627 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-09-20T22:45:06.627 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-09-20T22:45:06.627 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-09-20T22:45:06.627 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:45:06.628 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:06.628 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:45:06.628 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.154:6804", 2024-09-20T22:45:06.628 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 493899927 2024-09-20T22:45:06.628 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:06.628 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:45:06.628 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:06.628 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-09-20T22:45:06.628 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:45:06.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:06.629 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:45:06.630 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:06.630 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:45:06.630 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.154:6805", 2024-09-20T22:45:06.630 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 493899927 2024-09-20T22:45:06.630 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:06.630 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:45:06.630 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:06.630 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-09-20T22:45:06.630 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:45:06.630 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:06.630 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:45:06.630 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.154:6807", 2024-09-20T22:45:06.631 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 493899927 2024-09-20T22:45:06.631 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:06.631 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:45:06.631 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:06.631 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-09-20T22:45:06.631 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:45:06.631 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:06.631 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:45:06.631 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.154:6806", 2024-09-20T22:45:06.631 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 493899927 2024-09-20T22:45:06.631 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:06.631 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:45:06.631 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:06.632 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.154:6804/493899927", 2024-09-20T22:45:06.632 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.154:6805/493899927", 2024-09-20T22:45:06.632 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.154:6807/493899927", 2024-09-20T22:45:06.632 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.154:6806/493899927", 2024-09-20T22:45:06.632 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-09-20T22:45:06.632 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-09-20T22:45:06.632 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-09-20T22:45:06.632 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:45:06.632 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:06.632 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:06.632 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 6, 2024-09-20T22:45:06.632 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "56788d74-810e-4827-a615-958c12f8a368", 2024-09-20T22:45:06.632 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-09-20T22:45:06.633 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-09-20T22:45:06.633 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-09-20T22:45:06.633 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-09-20T22:45:06.633 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-09-20T22:45:06.633 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-09-20T22:45:06.633 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 48, 2024-09-20T22:45:06.633 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 169, 2024-09-20T22:45:06.633 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-09-20T22:45:06.633 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-09-20T22:45:06.633 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-09-20T22:45:06.633 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:45:06.633 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:06.633 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:45:06.633 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.154:6808", 2024-09-20T22:45:06.633 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2375091033 2024-09-20T22:45:06.634 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:06.634 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:45:06.634 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:06.634 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-09-20T22:45:06.634 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:45:06.634 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:06.634 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:45:06.634 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.154:6809", 2024-09-20T22:45:06.634 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2375091033 2024-09-20T22:45:06.634 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:06.634 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:45:06.634 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:06.634 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-09-20T22:45:06.634 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:45:06.635 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:06.635 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:45:06.635 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.154:6811", 2024-09-20T22:45:06.635 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2375091033 2024-09-20T22:45:06.635 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:06.635 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:45:06.635 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:06.635 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-09-20T22:45:06.635 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:45:06.635 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:06.635 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:45:06.635 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.154:6810", 2024-09-20T22:45:06.635 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2375091033 2024-09-20T22:45:06.636 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:06.636 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:45:06.636 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:06.636 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.154:6808/2375091033", 2024-09-20T22:45:06.636 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.154:6809/2375091033", 2024-09-20T22:45:06.636 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.154:6811/2375091033", 2024-09-20T22:45:06.636 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.154:6810/2375091033", 2024-09-20T22:45:06.636 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-09-20T22:45:06.636 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-09-20T22:45:06.636 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-09-20T22:45:06.636 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:45:06.636 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:06.636 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:06.636 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 7, 2024-09-20T22:45:06.637 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "83503727-afc1-4b95-8b96-ecb63f3338c2", 2024-09-20T22:45:06.637 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-09-20T22:45:06.637 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-09-20T22:45:06.637 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-09-20T22:45:06.637 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-09-20T22:45:06.637 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-09-20T22:45:06.637 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-09-20T22:45:06.637 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 59, 2024-09-20T22:45:06.637 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 169, 2024-09-20T22:45:06.637 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-09-20T22:45:06.637 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-09-20T22:45:06.637 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-09-20T22:45:06.637 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:45:06.638 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:06.638 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:45:06.638 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.154:6812", 2024-09-20T22:45:06.638 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3717704491 2024-09-20T22:45:06.638 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:06.638 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:45:06.638 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:06.638 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-09-20T22:45:06.638 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:45:06.638 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:06.638 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:45:06.638 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.154:6813", 2024-09-20T22:45:06.638 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3717704491 2024-09-20T22:45:06.639 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:06.639 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:45:06.639 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:06.639 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-09-20T22:45:06.639 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:45:06.639 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:06.639 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:45:06.639 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.154:6815", 2024-09-20T22:45:06.639 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3717704491 2024-09-20T22:45:06.639 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:06.639 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:45:06.639 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:06.639 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-09-20T22:45:06.639 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:45:06.639 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:06.640 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:45:06.640 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.154:6814", 2024-09-20T22:45:06.640 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3717704491 2024-09-20T22:45:06.640 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:06.640 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:45:06.640 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:06.640 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.154:6812/3717704491", 2024-09-20T22:45:06.640 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.154:6813/3717704491", 2024-09-20T22:45:06.640 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.154:6815/3717704491", 2024-09-20T22:45:06.640 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.154:6814/3717704491", 2024-09-20T22:45:06.640 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-09-20T22:45:06.640 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-09-20T22:45:06.640 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-09-20T22:45:06.640 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:45:06.641 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:06.641 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2024-09-20T22:45:06.641 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_xinfo": [ 2024-09-20T22:45:06.641 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:06.641 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 0, 2024-09-20T22:45:06.641 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-09-20T22:45:06.641 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-09-20T22:45:06.641 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-09-20T22:45:06.641 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540701547738038271, 2024-09-20T22:45:06.641 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-09-20T22:45:06.641 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-09-20T22:43:38.373571+0000", 2024-09-20T22:45:06.641 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-09-20T22:45:06.641 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:06.642 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:06.642 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 1, 2024-09-20T22:45:06.642 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-09-20T22:45:06.642 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-09-20T22:45:06.642 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-09-20T22:45:06.642 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540701547738038271, 2024-09-20T22:45:06.642 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-09-20T22:45:06.642 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-09-20T22:44:01.955517+0000", 2024-09-20T22:45:06.642 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-09-20T22:45:06.642 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:06.642 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:06.642 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 2, 2024-09-20T22:45:06.642 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-09-20T22:45:06.643 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-09-20T22:45:06.643 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-09-20T22:45:06.643 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540701547738038271, 2024-09-20T22:45:06.643 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-09-20T22:45:06.643 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-09-20T22:44:49.887047+0000", 2024-09-20T22:45:06.643 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-09-20T22:45:06.643 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:06.643 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:06.643 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 3, 2024-09-20T22:45:06.643 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-09-20T22:45:06.643 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-09-20T22:45:06.643 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-09-20T22:45:06.643 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540701547738038271, 2024-09-20T22:45:06.643 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-09-20T22:45:06.643 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-09-20T22:44:55.581914+0000", 2024-09-20T22:45:06.644 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-09-20T22:45:06.644 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:06.644 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:06.644 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 4, 2024-09-20T22:45:06.644 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-09-20T22:45:06.644 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-09-20T22:45:06.644 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-09-20T22:45:06.644 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540701547738038271, 2024-09-20T22:45:06.644 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 65536, 2024-09-20T22:45:06.644 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-09-20T22:45:02.445007+0000", 2024-09-20T22:45:06.644 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-09-20T22:45:06.644 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:06.644 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:06.644 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 5, 2024-09-20T22:45:06.644 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-09-20T22:45:06.645 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-09-20T22:45:06.645 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-09-20T22:45:06.645 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540701547738038271, 2024-09-20T22:45:06.645 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-09-20T22:45:06.645 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-09-20T22:44:33.074031+0000", 2024-09-20T22:45:06.645 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-09-20T22:45:06.645 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:06.645 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:06.645 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 6, 2024-09-20T22:45:06.645 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-09-20T22:45:06.645 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-09-20T22:45:06.645 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-09-20T22:45:06.645 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540701547738038271, 2024-09-20T22:45:06.646 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-09-20T22:45:06.646 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-09-20T22:44:30.460653+0000", 2024-09-20T22:45:06.646 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-09-20T22:45:06.646 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:06.646 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:06.646 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 7, 2024-09-20T22:45:06.646 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-09-20T22:45:06.646 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-09-20T22:45:06.646 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-09-20T22:45:06.646 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540701547738038271, 2024-09-20T22:45:06.646 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-09-20T22:45:06.646 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-09-20T22:43:53.020113+0000", 2024-09-20T22:45:06.646 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-09-20T22:45:06.646 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:06.647 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2024-09-20T22:45:06.647 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_upmap": [], 2024-09-20T22:45:06.647 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_upmap_items": [ 2024-09-20T22:45:06.647 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:06.647 INFO:teuthology.orchestra.run.smithi084.stdout: "pgid": "3.18", 2024-09-20T22:45:06.647 INFO:teuthology.orchestra.run.smithi084.stdout: "mappings": [ 2024-09-20T22:45:06.647 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:06.647 INFO:teuthology.orchestra.run.smithi084.stdout: "from": 0, 2024-09-20T22:45:06.647 INFO:teuthology.orchestra.run.smithi084.stdout: "to": 7 2024-09-20T22:45:06.647 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:06.647 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:45:06.647 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:06.647 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:06.647 INFO:teuthology.orchestra.run.smithi084.stdout: "pgid": "4.6", 2024-09-20T22:45:06.647 INFO:teuthology.orchestra.run.smithi084.stdout: "mappings": [ 2024-09-20T22:45:06.648 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:06.648 INFO:teuthology.orchestra.run.smithi084.stdout: "from": 1, 2024-09-20T22:45:06.648 INFO:teuthology.orchestra.run.smithi084.stdout: "to": 5 2024-09-20T22:45:06.648 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:06.648 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:45:06.648 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:06.648 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:06.648 INFO:teuthology.orchestra.run.smithi084.stdout: "pgid": "4.e", 2024-09-20T22:45:06.648 INFO:teuthology.orchestra.run.smithi084.stdout: "mappings": [ 2024-09-20T22:45:06.648 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:06.648 INFO:teuthology.orchestra.run.smithi084.stdout: "from": 3, 2024-09-20T22:45:06.648 INFO:teuthology.orchestra.run.smithi084.stdout: "to": 2 2024-09-20T22:45:06.648 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:06.648 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:45:06.649 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:06.649 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:06.649 INFO:teuthology.orchestra.run.smithi084.stdout: "pgid": "4.11", 2024-09-20T22:45:06.649 INFO:teuthology.orchestra.run.smithi084.stdout: "mappings": [ 2024-09-20T22:45:06.649 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:06.649 INFO:teuthology.orchestra.run.smithi084.stdout: "from": 3, 2024-09-20T22:45:06.649 INFO:teuthology.orchestra.run.smithi084.stdout: "to": 5 2024-09-20T22:45:06.649 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:06.649 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:45:06.649 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:06.649 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:06.649 INFO:teuthology.orchestra.run.smithi084.stdout: "pgid": "4.17", 2024-09-20T22:45:06.649 INFO:teuthology.orchestra.run.smithi084.stdout: "mappings": [ 2024-09-20T22:45:06.650 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:06.650 INFO:teuthology.orchestra.run.smithi084.stdout: "from": 1, 2024-09-20T22:45:06.650 INFO:teuthology.orchestra.run.smithi084.stdout: "to": 2 2024-09-20T22:45:06.650 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:06.650 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:45:06.650 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:06.650 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:06.650 INFO:teuthology.orchestra.run.smithi084.stdout: "pgid": "4.1d", 2024-09-20T22:45:06.650 INFO:teuthology.orchestra.run.smithi084.stdout: "mappings": [ 2024-09-20T22:45:06.650 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:06.650 INFO:teuthology.orchestra.run.smithi084.stdout: "from": 1, 2024-09-20T22:45:06.650 INFO:teuthology.orchestra.run.smithi084.stdout: "to": 2 2024-09-20T22:45:06.650 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:06.650 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:45:06.651 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:06.651 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:06.651 INFO:teuthology.orchestra.run.smithi084.stdout: "pgid": "5.3", 2024-09-20T22:45:06.651 INFO:teuthology.orchestra.run.smithi084.stdout: "mappings": [ 2024-09-20T22:45:06.651 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:06.651 INFO:teuthology.orchestra.run.smithi084.stdout: "from": 0, 2024-09-20T22:45:06.651 INFO:teuthology.orchestra.run.smithi084.stdout: "to": 3 2024-09-20T22:45:06.651 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:06.651 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:45:06.651 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:06.651 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:06.651 INFO:teuthology.orchestra.run.smithi084.stdout: "pgid": "7.2", 2024-09-20T22:45:06.651 INFO:teuthology.orchestra.run.smithi084.stdout: "mappings": [ 2024-09-20T22:45:06.651 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:06.652 INFO:teuthology.orchestra.run.smithi084.stdout: "from": 3, 2024-09-20T22:45:06.652 INFO:teuthology.orchestra.run.smithi084.stdout: "to": 5 2024-09-20T22:45:06.652 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:06.652 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:45:06.652 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:06.652 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:06.652 INFO:teuthology.orchestra.run.smithi084.stdout: "pgid": "7.13", 2024-09-20T22:45:06.652 INFO:teuthology.orchestra.run.smithi084.stdout: "mappings": [ 2024-09-20T22:45:06.652 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:06.652 INFO:teuthology.orchestra.run.smithi084.stdout: "from": 7, 2024-09-20T22:45:06.652 INFO:teuthology.orchestra.run.smithi084.stdout: "to": 1 2024-09-20T22:45:06.652 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:06.652 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:45:06.652 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:06.653 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:06.653 INFO:teuthology.orchestra.run.smithi084.stdout: "pgid": "7.15", 2024-09-20T22:45:06.653 INFO:teuthology.orchestra.run.smithi084.stdout: "mappings": [ 2024-09-20T22:45:06.653 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:06.653 INFO:teuthology.orchestra.run.smithi084.stdout: "from": 2, 2024-09-20T22:45:06.653 INFO:teuthology.orchestra.run.smithi084.stdout: "to": 6 2024-09-20T22:45:06.653 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:06.653 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:45:06.653 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:06.653 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2024-09-20T22:45:06.653 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_upmap_primaries": [], 2024-09-20T22:45:06.653 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_temp": [], 2024-09-20T22:45:06.653 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_temp": [], 2024-09-20T22:45:06.653 INFO:teuthology.orchestra.run.smithi084.stdout: "blocklist": { 2024-09-20T22:45:06.654 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/542020545": "2024-09-21T22:42:07.596045+0000", 2024-09-20T22:45:06.654 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/25873236": "2024-09-21T22:36:54.917812+0000", 2024-09-20T22:45:06.654 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:6800/820908470": "2024-09-21T22:36:54.917812+0000", 2024-09-20T22:45:06.654 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/567614908": "2024-09-21T22:36:54.917812+0000", 2024-09-20T22:45:06.654 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/2963622459": "2024-09-21T22:42:07.596045+0000", 2024-09-20T22:45:06.654 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/1176849276": "2024-09-21T22:37:17.690104+0000", 2024-09-20T22:45:06.654 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:6800/2932420050": "2024-09-21T22:37:17.690104+0000", 2024-09-20T22:45:06.654 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/3879263635": "2024-09-21T22:37:17.690104+0000", 2024-09-20T22:45:06.654 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/969472279": "2024-09-21T22:36:54.917812+0000", 2024-09-20T22:45:06.654 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/1361143214": "2024-09-21T22:37:17.690104+0000", 2024-09-20T22:45:06.654 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/3353126955": "2024-09-21T22:42:07.596045+0000", 2024-09-20T22:45:06.654 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:6800/4024278029": "2024-09-21T22:42:07.596045+0000", 2024-09-20T22:45:06.654 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/547902262": "2024-09-21T22:42:07.596045+0000", 2024-09-20T22:45:06.655 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/1242929317": "2024-09-21T22:42:07.596045+0000" 2024-09-20T22:45:06.655 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:06.655 INFO:teuthology.orchestra.run.smithi084.stdout: "range_blocklist": {}, 2024-09-20T22:45:06.655 INFO:teuthology.orchestra.run.smithi084.stdout: "erasure_code_profiles": { 2024-09-20T22:45:06.655 INFO:teuthology.orchestra.run.smithi084.stdout: "default": { 2024-09-20T22:45:06.655 INFO:teuthology.orchestra.run.smithi084.stdout: "crush-failure-domain": "osd", 2024-09-20T22:45:06.655 INFO:teuthology.orchestra.run.smithi084.stdout: "k": "2", 2024-09-20T22:45:06.655 INFO:teuthology.orchestra.run.smithi084.stdout: "m": "1", 2024-09-20T22:45:06.655 INFO:teuthology.orchestra.run.smithi084.stdout: "plugin": "jerasure", 2024-09-20T22:45:06.655 INFO:teuthology.orchestra.run.smithi084.stdout: "technique": "reed_sol_van" 2024-09-20T22:45:06.655 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:06.655 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:06.655 INFO:teuthology.orchestra.run.smithi084.stdout: "removed_snaps_queue": [ 2024-09-20T22:45:06.655 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:06.656 INFO:teuthology.orchestra.run.smithi084.stdout: "pool": 7, 2024-09-20T22:45:06.656 INFO:teuthology.orchestra.run.smithi084.stdout: "snaps": [ 2024-09-20T22:45:06.656 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:06.656 INFO:teuthology.orchestra.run.smithi084.stdout: "begin": 57, 2024-09-20T22:45:06.656 INFO:teuthology.orchestra.run.smithi084.stdout: "length": 2 2024-09-20T22:45:06.656 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:06.656 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:06.656 INFO:teuthology.orchestra.run.smithi084.stdout: "begin": 63, 2024-09-20T22:45:06.656 INFO:teuthology.orchestra.run.smithi084.stdout: "length": 1 2024-09-20T22:45:06.656 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:06.656 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:45:06.656 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:06.656 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2024-09-20T22:45:06.656 INFO:teuthology.orchestra.run.smithi084.stdout: "new_removed_snaps": [], 2024-09-20T22:45:06.656 INFO:teuthology.orchestra.run.smithi084.stdout: "new_purged_snaps": [], 2024-09-20T22:45:06.656 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_node_flags": {}, 2024-09-20T22:45:06.657 INFO:teuthology.orchestra.run.smithi084.stdout: "device_class_flags": {}, 2024-09-20T22:45:06.657 INFO:teuthology.orchestra.run.smithi084.stdout: "stretch_mode": { 2024-09-20T22:45:06.657 INFO:teuthology.orchestra.run.smithi084.stdout: "stretch_mode_enabled": false, 2024-09-20T22:45:06.657 INFO:teuthology.orchestra.run.smithi084.stdout: "stretch_bucket_count": 0, 2024-09-20T22:45:06.657 INFO:teuthology.orchestra.run.smithi084.stdout: "degraded_stretch_mode": 0, 2024-09-20T22:45:06.657 INFO:teuthology.orchestra.run.smithi084.stdout: "recovering_stretch_mode": 0, 2024-09-20T22:45:06.657 INFO:teuthology.orchestra.run.smithi084.stdout: "stretch_mode_bucket": 0 2024-09-20T22:45:06.657 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:06.657 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-20T22:45:06.731 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:45:06.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:06.753 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:06.754 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:45:06.854 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:45:06.876 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:06.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:06.877 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:45:06.978 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:45:06.999 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:06.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:07.000 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:45:07.101 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:45:07.122 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:07.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:07.123 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:45:07.151 INFO:tasks.rados.rados.0.smithi084.stdout:685: done (0 left) 2024-09-20T22:45:07.151 INFO:tasks.rados.rados.0.smithi084.stdout:686: snap_remove snap 68 2024-09-20T22:45:07.223 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:45:07.245 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:07.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:07.246 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:45:07.347 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:45:07.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:07.369 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:07.369 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:45:07.470 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:45:07.477 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:07 smithi084 ceph-mon[40024]: pgmap v241: 178 pgs: 3 active+clean+snaptrim, 175 active+clean; 109 MiB data, 552 MiB used, 625 GiB / 626 GiB avail; 7.4 MiB/s rd, 2.5 MiB/s wr, 13 op/s 2024-09-20T22:45:07.477 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:07 smithi084 ceph-mon[40024]: purged_snaps scrub starts 2024-09-20T22:45:07.477 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:07 smithi084 ceph-mon[40024]: purged_snaps scrub ok 2024-09-20T22:45:07.477 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:07 smithi084 ceph-mon[40024]: osdmap e227: 8 total, 8 up, 7 in 2024-09-20T22:45:07.477 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:07 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/3674025470' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-20T22:45:07.477 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:07 smithi084 ceph-mon[40024]: osdmap e228: 8 total, 8 up, 7 in 2024-09-20T22:45:07.477 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:07 smithi084 ceph-mon[29774]: pgmap v241: 178 pgs: 3 active+clean+snaptrim, 175 active+clean; 109 MiB data, 552 MiB used, 625 GiB / 626 GiB avail; 7.4 MiB/s rd, 2.5 MiB/s wr, 13 op/s 2024-09-20T22:45:07.478 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:07 smithi084 ceph-mon[29774]: purged_snaps scrub starts 2024-09-20T22:45:07.478 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:07 smithi084 ceph-mon[29774]: purged_snaps scrub ok 2024-09-20T22:45:07.478 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:07 smithi084 ceph-mon[29774]: osdmap e227: 8 total, 8 up, 7 in 2024-09-20T22:45:07.478 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:07 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/3674025470' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-20T22:45:07.478 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:07 smithi084 ceph-mon[29774]: osdmap e228: 8 total, 8 up, 7 in 2024-09-20T22:45:07.492 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:07.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:07.493 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:45:07.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:07 smithi154 ceph-mon[33264]: pgmap v241: 178 pgs: 3 active+clean+snaptrim, 175 active+clean; 109 MiB data, 552 MiB used, 625 GiB / 626 GiB avail; 7.4 MiB/s rd, 2.5 MiB/s wr, 13 op/s 2024-09-20T22:45:07.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:07 smithi154 ceph-mon[33264]: purged_snaps scrub starts 2024-09-20T22:45:07.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:07 smithi154 ceph-mon[33264]: purged_snaps scrub ok 2024-09-20T22:45:07.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:07 smithi154 ceph-mon[33264]: osdmap e227: 8 total, 8 up, 7 in 2024-09-20T22:45:07.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:07 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/3674025470' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-20T22:45:07.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:07 smithi154 ceph-mon[33264]: osdmap e228: 8 total, 8 up, 7 in 2024-09-20T22:45:07.586 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-20T22:45:07.586 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2024-09-20T22:45:07.586 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2024-09-20T22:45:07.586 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [ 2024-09-20T22:45:07.586 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.586 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.14156.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-09-20T22:45:07.587 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:39:23.853477+0000", 2024-09-20T22:45:07.587 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 343.72151463500001, 2024-09-20T22:45:07.587 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0067540229999999996, 2024-09-20T22:45:07.587 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:45:07.587 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:45:07.587 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:45:07.587 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:45:07.587 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.14156", 2024-09-20T22:45:07.587 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/547902262", 2024-09-20T22:45:07.587 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 116 2024-09-20T22:45:07.587 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.587 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:45:07.587 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.587 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:45:07.588 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:39:23.853477+0000", 2024-09-20T22:45:07.588 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:45:07.588 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.588 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.588 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:45:07.588 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:39:23.853477+0000", 2024-09-20T22:45:07.588 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:45:07.588 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.588 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.588 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:45:07.588 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:39:23.853478+0000", 2024-09-20T22:45:07.588 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.2300000000000001e-07 2024-09-20T22:45:07.589 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.589 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.589 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:45:07.589 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:39:23.853481+0000", 2024-09-20T22:45:07.589 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.535e-06 2024-09-20T22:45:07.589 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.589 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.589 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:45:07.589 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:39:23.853487+0000", 2024-09-20T22:45:07.589 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.5650000000000001e-06 2024-09-20T22:45:07.589 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.589 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.589 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:45:07.589 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:39:23.853492+0000", 2024-09-20T22:45:07.590 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.1189999999999996e-06 2024-09-20T22:45:07.590 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.590 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.590 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:45:07.590 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:39:23.859190+0000", 2024-09-20T22:45:07.590 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0056979559999999997 2024-09-20T22:45:07.590 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.590 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.590 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:45:07.590 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:39:23.859257+0000", 2024-09-20T22:45:07.590 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.7382e-05 2024-09-20T22:45:07.590 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.590 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.591 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 0,2", 2024-09-20T22:45:07.591 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:39:23.859368+0000", 2024-09-20T22:45:07.591 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000110913 2024-09-20T22:45:07.591 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.591 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.591 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:45:07.591 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:39:23.859975+0000", 2024-09-20T22:45:07.591 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00060658399999999996 2024-09-20T22:45:07.591 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.591 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.591 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:45:07.591 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:39:23.860117+0000", 2024-09-20T22:45:07.591 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014242500000000001 2024-09-20T22:45:07.591 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.592 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.592 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:45:07.592 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:39:23.860156+0000", 2024-09-20T22:45:07.592 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.8887999999999998e-05 2024-09-20T22:45:07.592 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.592 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.592 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:45:07.592 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:39:23.860196+0000", 2024-09-20T22:45:07.592 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.9765999999999998e-05 2024-09-20T22:45:07.592 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.592 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.592 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:45:07.592 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:39:23.860231+0000", 2024-09-20T22:45:07.592 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.5367000000000002e-05 2024-09-20T22:45:07.593 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:07.593 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:45:07.593 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:07.593 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.593 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.593 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.14156.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-09-20T22:45:07.593 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:39:23.853500+0000", 2024-09-20T22:45:07.593 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 343.72149202399999, 2024-09-20T22:45:07.593 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.006861168, 2024-09-20T22:45:07.593 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:45:07.593 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:45:07.593 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:45:07.593 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:45:07.593 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.14156", 2024-09-20T22:45:07.594 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/547902262", 2024-09-20T22:45:07.594 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 117 2024-09-20T22:45:07.594 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.594 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:45:07.594 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.594 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:45:07.594 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:39:23.853500+0000", 2024-09-20T22:45:07.594 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:45:07.594 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.594 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.594 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:45:07.594 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:39:23.853500+0000", 2024-09-20T22:45:07.594 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:45:07.595 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.595 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.595 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:45:07.595 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:39:23.853500+0000", 2024-09-20T22:45:07.595 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.3000000000000001e-07 2024-09-20T22:45:07.595 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.595 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.595 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:45:07.595 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:39:23.853503+0000", 2024-09-20T22:45:07.595 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.3179999999999998e-06 2024-09-20T22:45:07.595 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.595 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.595 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:45:07.595 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:39:23.853509+0000", 2024-09-20T22:45:07.595 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.5400000000000003e-06 2024-09-20T22:45:07.596 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.596 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.596 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:45:07.596 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:39:23.853514+0000", 2024-09-20T22:45:07.596 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.5709999999999997e-06 2024-09-20T22:45:07.596 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.596 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.596 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:45:07.596 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:39:23.859552+0000", 2024-09-20T22:45:07.596 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0060371920000000003 2024-09-20T22:45:07.596 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.596 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.596 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:45:07.596 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:39:23.859594+0000", 2024-09-20T22:45:07.597 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.2218000000000001e-05 2024-09-20T22:45:07.597 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.597 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.597 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 0,2", 2024-09-20T22:45:07.597 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:39:23.859671+0000", 2024-09-20T22:45:07.597 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.7423999999999996e-05 2024-09-20T22:45:07.597 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.597 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.597 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:45:07.597 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:39:23.859977+0000", 2024-09-20T22:45:07.597 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00030561299999999999 2024-09-20T22:45:07.597 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.597 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.597 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:45:07.597 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:39:23.860281+0000", 2024-09-20T22:45:07.598 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00030366200000000001 2024-09-20T22:45:07.598 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.598 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.598 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:45:07.598 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:39:23.860311+0000", 2024-09-20T22:45:07.598 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.0499999999999999e-05 2024-09-20T22:45:07.598 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.598 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.598 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:45:07.598 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:39:23.860340+0000", 2024-09-20T22:45:07.598 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.8651000000000001e-05 2024-09-20T22:45:07.598 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.598 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.598 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:45:07.598 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:39:23.860361+0000", 2024-09-20T22:45:07.599 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.1049000000000001e-05 2024-09-20T22:45:07.599 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:07.599 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:45:07.599 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:07.599 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.599 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.599 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.14156.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2024-09-20T22:45:07.599 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:39:25.916568+0000", 2024-09-20T22:45:07.599 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 341.658423594, 2024-09-20T22:45:07.599 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0072203959999999996, 2024-09-20T22:45:07.599 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:45:07.599 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:45:07.599 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:45:07.599 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:45:07.600 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.14156", 2024-09-20T22:45:07.600 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/547902262", 2024-09-20T22:45:07.600 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 138 2024-09-20T22:45:07.600 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.600 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:45:07.600 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.600 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:45:07.600 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:39:25.916568+0000", 2024-09-20T22:45:07.600 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:45:07.600 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.600 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.600 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:45:07.600 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:39:25.916568+0000", 2024-09-20T22:45:07.601 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:45:07.601 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.601 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.601 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:45:07.601 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:39:25.916569+0000", 2024-09-20T22:45:07.601 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.0999999999999998e-07 2024-09-20T22:45:07.601 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.601 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.601 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:45:07.601 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:39:25.916573+0000", 2024-09-20T22:45:07.601 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.0740000000000003e-06 2024-09-20T22:45:07.601 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.601 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.601 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:45:07.602 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:39:25.916579+0000", 2024-09-20T22:45:07.602 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.2140000000000001e-06 2024-09-20T22:45:07.602 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.602 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.602 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:45:07.602 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:39:25.916586+0000", 2024-09-20T22:45:07.602 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.3540000000000002e-06 2024-09-20T22:45:07.602 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.602 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.602 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:45:07.602 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:39:25.921879+0000", 2024-09-20T22:45:07.602 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0052922009999999998 2024-09-20T22:45:07.602 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.602 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.603 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:45:07.603 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:39:25.921961+0000", 2024-09-20T22:45:07.603 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.2047999999999998e-05 2024-09-20T22:45:07.603 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.603 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.603 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 0,2", 2024-09-20T22:45:07.603 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:39:25.922062+0000", 2024-09-20T22:45:07.603 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00010108800000000001 2024-09-20T22:45:07.603 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.603 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.603 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:45:07.603 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:39:25.923164+0000", 2024-09-20T22:45:07.603 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0011025539999999999 2024-09-20T22:45:07.603 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.604 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.604 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:45:07.604 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:39:25.923202+0000", 2024-09-20T22:45:07.604 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.7837999999999999e-05 2024-09-20T22:45:07.604 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.604 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.604 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:45:07.604 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:39:25.923698+0000", 2024-09-20T22:45:07.604 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00049612600000000005 2024-09-20T22:45:07.604 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.604 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.604 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:45:07.604 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:39:25.923748+0000", 2024-09-20T22:45:07.604 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.0271e-05 2024-09-20T22:45:07.605 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.605 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.605 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:45:07.605 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:39:25.923788+0000", 2024-09-20T22:45:07.605 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.0018000000000002e-05 2024-09-20T22:45:07.605 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:07.605 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:45:07.605 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:07.605 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.605 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.605 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.14156.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2024-09-20T22:45:07.605 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:39:25.916596+0000", 2024-09-20T22:45:07.605 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 341.658395574, 2024-09-20T22:45:07.606 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0077891449999999999, 2024-09-20T22:45:07.606 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:45:07.606 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:45:07.606 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:45:07.606 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:45:07.606 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.14156", 2024-09-20T22:45:07.606 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/547902262", 2024-09-20T22:45:07.606 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 139 2024-09-20T22:45:07.606 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.606 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:45:07.606 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.606 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:45:07.607 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:39:25.916596+0000", 2024-09-20T22:45:07.607 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:45:07.607 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.607 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.607 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:45:07.607 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:39:25.916596+0000", 2024-09-20T22:45:07.607 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:45:07.607 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.607 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.607 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:45:07.607 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:39:25.916597+0000", 2024-09-20T22:45:07.607 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.8400000000000005e-07 2024-09-20T22:45:07.608 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.608 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.608 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:45:07.608 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:39:25.916601+0000", 2024-09-20T22:45:07.608 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.5329999999999996e-06 2024-09-20T22:45:07.608 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.608 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.608 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:45:07.608 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:39:25.916607+0000", 2024-09-20T22:45:07.608 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.1140000000000002e-06 2024-09-20T22:45:07.608 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.608 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.608 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:45:07.608 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:39:25.916612+0000", 2024-09-20T22:45:07.609 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.989e-06 2024-09-20T22:45:07.609 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.609 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.609 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:45:07.609 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:39:25.922396+0000", 2024-09-20T22:45:07.609 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0057839700000000003 2024-09-20T22:45:07.609 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.609 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.609 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:45:07.609 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:39:25.922489+0000", 2024-09-20T22:45:07.609 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.3128999999999996e-05 2024-09-20T22:45:07.609 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.609 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.609 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 0,2", 2024-09-20T22:45:07.610 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:39:25.922631+0000", 2024-09-20T22:45:07.610 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000141891 2024-09-20T22:45:07.610 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.610 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.610 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:45:07.610 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:39:25.924261+0000", 2024-09-20T22:45:07.610 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016297589999999999 2024-09-20T22:45:07.610 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.610 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.610 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:45:07.610 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:39:25.924298+0000", 2024-09-20T22:45:07.610 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.6619999999999998e-05 2024-09-20T22:45:07.610 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.611 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.611 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:45:07.611 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:39:25.924307+0000", 2024-09-20T22:45:07.611 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.9679999999999995e-06 2024-09-20T22:45:07.611 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.611 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.611 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:45:07.611 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:39:25.924347+0000", 2024-09-20T22:45:07.611 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.0002000000000001e-05 2024-09-20T22:45:07.611 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.611 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.611 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:45:07.611 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:39:25.924385+0000", 2024-09-20T22:45:07.611 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.8686000000000002e-05 2024-09-20T22:45:07.611 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:07.612 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:45:07.612 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:07.612 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.612 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.612 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.14156.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2024-09-20T22:45:07.612 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:39:25.916621+0000", 2024-09-20T22:45:07.612 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 341.65837092599998, 2024-09-20T22:45:07.612 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0080181760000000001, 2024-09-20T22:45:07.612 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:45:07.612 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:45:07.612 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:45:07.612 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:45:07.612 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.14156", 2024-09-20T22:45:07.612 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/547902262", 2024-09-20T22:45:07.613 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 140 2024-09-20T22:45:07.613 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.613 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:45:07.613 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.613 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:45:07.613 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:39:25.916621+0000", 2024-09-20T22:45:07.613 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:45:07.613 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.613 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.613 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:45:07.613 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:39:25.916621+0000", 2024-09-20T22:45:07.613 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:45:07.613 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.613 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.613 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:45:07.614 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:39:25.916621+0000", 2024-09-20T22:45:07.614 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.06e-07 2024-09-20T22:45:07.614 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.614 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.614 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:45:07.614 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:39:25.916711+0000", 2024-09-20T22:45:07.614 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.9435e-05 2024-09-20T22:45:07.614 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.614 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.614 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:45:07.614 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:39:25.916718+0000", 2024-09-20T22:45:07.614 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.5020000000000003e-06 2024-09-20T22:45:07.614 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.615 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.615 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:45:07.615 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:39:25.916724+0000", 2024-09-20T22:45:07.615 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.5550000000000002e-06 2024-09-20T22:45:07.615 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.615 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.615 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:45:07.615 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:39:25.923234+0000", 2024-09-20T22:45:07.615 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0065104910000000002 2024-09-20T22:45:07.615 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.615 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.615 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:45:07.615 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:39:25.923289+0000", 2024-09-20T22:45:07.615 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.5158000000000003e-05 2024-09-20T22:45:07.615 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.616 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.616 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 0,2", 2024-09-20T22:45:07.616 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:39:25.923363+0000", 2024-09-20T22:45:07.616 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.3852999999999999e-05 2024-09-20T22:45:07.616 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.616 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.616 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:45:07.616 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:39:25.924399+0000", 2024-09-20T22:45:07.616 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010355379999999999 2024-09-20T22:45:07.616 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.616 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.616 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:45:07.616 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:39:25.924442+0000", 2024-09-20T22:45:07.616 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.3717000000000002e-05 2024-09-20T22:45:07.617 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.617 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.617 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:45:07.617 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:39:25.924550+0000", 2024-09-20T22:45:07.617 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00010799899999999999 2024-09-20T22:45:07.617 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.617 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.617 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:45:07.617 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:39:25.924600+0000", 2024-09-20T22:45:07.617 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.9656e-05 2024-09-20T22:45:07.617 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.617 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.617 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:45:07.617 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:39:25.924639+0000", 2024-09-20T22:45:07.618 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.8766000000000001e-05 2024-09-20T22:45:07.618 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:07.618 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:45:07.618 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:07.618 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.618 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.618 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.14156.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2024-09-20T22:45:07.618 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:39:25.916734+0000", 2024-09-20T22:45:07.618 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 341.658257397, 2024-09-20T22:45:07.618 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0080427009999999993, 2024-09-20T22:45:07.618 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:45:07.618 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:45:07.618 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:45:07.618 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:45:07.619 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.14156", 2024-09-20T22:45:07.619 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/547902262", 2024-09-20T22:45:07.619 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 141 2024-09-20T22:45:07.619 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.619 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:45:07.619 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.619 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:45:07.619 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:39:25.916734+0000", 2024-09-20T22:45:07.619 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:45:07.619 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.619 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.619 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:45:07.619 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:39:25.916734+0000", 2024-09-20T22:45:07.619 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:45:07.619 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.620 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.620 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:45:07.620 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:39:25.916735+0000", 2024-09-20T22:45:07.620 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.0500000000000003e-07 2024-09-20T22:45:07.620 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.620 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.620 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:45:07.620 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:39:25.916742+0000", 2024-09-20T22:45:07.620 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.43e-06 2024-09-20T22:45:07.620 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.620 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.620 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:45:07.620 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:39:25.916748+0000", 2024-09-20T22:45:07.620 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.835e-06 2024-09-20T22:45:07.621 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.621 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.621 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:45:07.621 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:39:25.916753+0000", 2024-09-20T22:45:07.621 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.9799999999999998e-06 2024-09-20T22:45:07.621 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.621 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.621 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:45:07.621 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:39:25.923841+0000", 2024-09-20T22:45:07.621 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0070883550000000002 2024-09-20T22:45:07.621 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.621 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.621 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:45:07.621 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:39:25.923921+0000", 2024-09-20T22:45:07.622 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.9900000000000004e-05 2024-09-20T22:45:07.622 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.622 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.622 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 0,2", 2024-09-20T22:45:07.622 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:39:25.924007+0000", 2024-09-20T22:45:07.622 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.6148000000000003e-05 2024-09-20T22:45:07.622 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.622 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.622 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:45:07.622 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:39:25.924404+0000", 2024-09-20T22:45:07.622 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00039647199999999998 2024-09-20T22:45:07.622 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.622 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.622 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:45:07.623 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:39:25.924691+0000", 2024-09-20T22:45:07.623 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00028692700000000003 2024-09-20T22:45:07.623 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.623 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.623 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:45:07.623 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:39:25.924722+0000", 2024-09-20T22:45:07.623 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.1587000000000001e-05 2024-09-20T22:45:07.623 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.623 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.623 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:45:07.623 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:39:25.924751+0000", 2024-09-20T22:45:07.623 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.8782e-05 2024-09-20T22:45:07.623 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.623 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.624 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:45:07.624 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:39:25.924777+0000", 2024-09-20T22:45:07.624 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.5680000000000001e-05 2024-09-20T22:45:07.624 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:07.624 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:45:07.624 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:07.624 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.624 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.624 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.14478.0:43 4.2 4:4347d321:::bucket.sync-source-hints.:head [call,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-09-20T22:45:07.624 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:41:40.156893+0000", 2024-09-20T22:45:07.624 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 207.41809875199999, 2024-09-20T22:45:07.624 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011236520999999999, 2024-09-20T22:45:07.624 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:45:07.624 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:45:07.624 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "reached pg", 2024-09-20T22:45:07.625 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:45:07.625 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.14478", 2024-09-20T22:45:07.625 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/2994273271", 2024-09-20T22:45:07.625 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 43 2024-09-20T22:45:07.625 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.625 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:45:07.625 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.625 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:45:07.625 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:41:40.156893+0000", 2024-09-20T22:45:07.625 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:45:07.625 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.625 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.625 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:45:07.625 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:41:40.156893+0000", 2024-09-20T22:45:07.626 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:45:07.626 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.626 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.626 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:45:07.626 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:41:40.156894+0000", 2024-09-20T22:45:07.626 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.339e-06 2024-09-20T22:45:07.626 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.626 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.626 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:45:07.626 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:41:40.156910+0000", 2024-09-20T22:45:07.626 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.5466999999999999e-05 2024-09-20T22:45:07.626 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.626 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.626 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:45:07.627 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:41:40.156921+0000", 2024-09-20T22:45:07.627 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1375e-05 2024-09-20T22:45:07.627 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.627 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.627 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:45:07.627 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:41:40.156933+0000", 2024-09-20T22:45:07.627 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1765e-05 2024-09-20T22:45:07.627 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.627 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.627 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:45:07.627 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:41:40.156992+0000", 2024-09-20T22:45:07.627 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.9122000000000001e-05 2024-09-20T22:45:07.627 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.628 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.628 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-09-20T22:45:07.628 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:41:40.157002+0000", 2024-09-20T22:45:07.628 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.9369999999999999e-06 2024-09-20T22:45:07.628 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.628 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.628 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:45:07.628 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:41:40.168028+0000", 2024-09-20T22:45:07.628 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011025945000000001 2024-09-20T22:45:07.628 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.628 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.628 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:45:07.628 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:41:40.168129+0000", 2024-09-20T22:45:07.628 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00010157099999999999 2024-09-20T22:45:07.628 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:07.629 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:45:07.629 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:07.629 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.629 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.629 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.14496.0:43 4.2 4:4347d321:::bucket.sync-source-hints.:head [call,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-09-20T22:45:07.629 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:41:40.157112+0000", 2024-09-20T22:45:07.629 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 207.41787943400001, 2024-09-20T22:45:07.629 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010889953000000001, 2024-09-20T22:45:07.629 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:45:07.629 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:45:07.629 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "reached pg", 2024-09-20T22:45:07.629 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:45:07.629 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.14496", 2024-09-20T22:45:07.629 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3480761128", 2024-09-20T22:45:07.630 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 43 2024-09-20T22:45:07.630 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.630 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:45:07.630 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.630 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:45:07.630 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:41:40.157112+0000", 2024-09-20T22:45:07.630 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:45:07.630 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.630 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.630 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:45:07.630 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:41:40.157112+0000", 2024-09-20T22:45:07.630 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:45:07.630 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.630 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.631 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:45:07.631 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:41:40.157113+0000", 2024-09-20T22:45:07.631 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.235e-06 2024-09-20T22:45:07.631 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.631 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.631 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:45:07.631 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:41:40.157120+0000", 2024-09-20T22:45:07.631 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.0500000000000003e-06 2024-09-20T22:45:07.631 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.631 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.631 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:45:07.631 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:41:40.157130+0000", 2024-09-20T22:45:07.631 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.9639999999999998e-06 2024-09-20T22:45:07.631 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.632 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.632 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:45:07.632 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:41:40.157139+0000", 2024-09-20T22:45:07.632 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.2390000000000002e-06 2024-09-20T22:45:07.632 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.632 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.632 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:45:07.632 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:41:40.157204+0000", 2024-09-20T22:45:07.632 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.5517000000000003e-05 2024-09-20T22:45:07.632 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.632 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.632 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-09-20T22:45:07.632 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:41:40.157212+0000", 2024-09-20T22:45:07.632 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.3610000000000003e-06 2024-09-20T22:45:07.633 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.633 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.633 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:45:07.633 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:41:40.167213+0000", 2024-09-20T22:45:07.633 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010001856999999999 2024-09-20T22:45:07.633 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.633 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.633 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:45:07.633 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:41:40.168002+0000", 2024-09-20T22:45:07.633 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00078872999999999996 2024-09-20T22:45:07.633 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:07.633 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:45:07.633 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:07.633 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.633 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.634 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:321 7.b 7:df02c955:::smithi08477203-9:head [copy-from ver 3] snapc 5=[5] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-09-20T22:45:07.634 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:43:04.855135+0000", 2024-09-20T22:45:07.634 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 122.719856241, 2024-09-20T22:45:07.634 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013238773000000001, 2024-09-20T22:45:07.634 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:45:07.634 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:45:07.634 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:45:07.634 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:45:07.634 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:45:07.634 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:45:07.634 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 321 2024-09-20T22:45:07.634 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.635 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:45:07.635 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.635 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:45:07.635 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:04.855135+0000", 2024-09-20T22:45:07.635 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:45:07.635 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.635 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.635 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:45:07.635 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:04.855135+0000", 2024-09-20T22:45:07.635 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:45:07.635 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.635 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.635 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:45:07.635 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:04.855138+0000", 2024-09-20T22:45:07.636 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.5560000000000002e-06 2024-09-20T22:45:07.636 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.636 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.636 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:45:07.636 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:04.855147+0000", 2024-09-20T22:45:07.636 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.1649999999999995e-06 2024-09-20T22:45:07.636 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.636 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.636 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:45:07.636 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:04.855162+0000", 2024-09-20T22:45:07.636 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.4875000000000001e-05 2024-09-20T22:45:07.636 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.636 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.637 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:45:07.637 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:04.855171+0000", 2024-09-20T22:45:07.637 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.414e-06 2024-09-20T22:45:07.637 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.637 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.637 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:45:07.637 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:04.855256+0000", 2024-09-20T22:45:07.637 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.4640000000000003e-05 2024-09-20T22:45:07.637 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.637 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.637 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:45:07.637 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:04.855347+0000", 2024-09-20T22:45:07.637 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.0549000000000003e-05 2024-09-20T22:45:07.637 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.638 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.638 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5,6", 2024-09-20T22:45:07.638 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:04.861176+0000", 2024-09-20T22:45:07.638 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0058289450000000003 2024-09-20T22:45:07.638 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.638 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.638 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:45:07.638 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:04.864700+0000", 2024-09-20T22:45:07.638 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.003524396 2024-09-20T22:45:07.638 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.638 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.638 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:45:07.638 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:04.868245+0000", 2024-09-20T22:45:07.639 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0035450880000000001 2024-09-20T22:45:07.639 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.639 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.639 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:45:07.639 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:04.868314+0000", 2024-09-20T22:45:07.639 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.8731000000000003e-05 2024-09-20T22:45:07.639 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.639 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.639 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:45:07.639 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:04.868347+0000", 2024-09-20T22:45:07.639 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.3311e-05 2024-09-20T22:45:07.639 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.639 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.639 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:45:07.640 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:04.868374+0000", 2024-09-20T22:45:07.640 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.7103000000000001e-05 2024-09-20T22:45:07.640 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:07.640 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:45:07.640 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:07.640 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.640 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.640 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:322 7.b 7:df02c955:::smithi08477203-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e105)", 2024-09-20T22:45:07.640 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:43:04.855230+0000", 2024-09-20T22:45:07.640 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 122.71976146999999, 2024-09-20T22:45:07.640 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0089518659999999993, 2024-09-20T22:45:07.640 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:45:07.640 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:45:07.640 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-20T22:45:07.641 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:45:07.641 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:45:07.641 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:45:07.641 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 322 2024-09-20T22:45:07.641 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.641 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:45:07.641 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.641 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:45:07.641 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:45:07.643 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:04.855230+0000", 2024-09-20T22:45:07.643 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:45:07.643 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.643 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.643 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:45:07.643 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:04.855230+0000", 2024-09-20T22:45:07.643 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:45:07.643 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.643 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.643 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:45:07.644 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:04.855231+0000", 2024-09-20T22:45:07.644 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.0200000000000002e-07 2024-09-20T22:45:07.644 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.644 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.644 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:45:07.644 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:04.855234+0000", 2024-09-20T22:45:07.644 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.3459999999999998e-06 2024-09-20T22:45:07.644 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.644 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.644 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:45:07.644 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:04.855241+0000", 2024-09-20T22:45:07.644 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.9310000000000002e-06 2024-09-20T22:45:07.644 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.645 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.645 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:45:07.645 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:04.855247+0000", 2024-09-20T22:45:07.645 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.541e-06 2024-09-20T22:45:07.645 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.645 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.645 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:45:07.645 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:04.855502+0000", 2024-09-20T22:45:07.645 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00025502499999999998 2024-09-20T22:45:07.645 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.645 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.645 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for blocked object", 2024-09-20T22:45:07.645 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:04.855541+0000", 2024-09-20T22:45:07.645 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.9140999999999997e-05 2024-09-20T22:45:07.646 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.646 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.646 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:45:07.646 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:04.863944+0000", 2024-09-20T22:45:07.646 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0084030860000000006 2024-09-20T22:45:07.646 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.646 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.646 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:45:07.646 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:04.864062+0000", 2024-09-20T22:45:07.646 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00011835799999999999 2024-09-20T22:45:07.646 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.646 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.646 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:45:07.647 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:04.864182+0000", 2024-09-20T22:45:07.647 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000119836 2024-09-20T22:45:07.647 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:07.647 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:45:07.647 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:07.647 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.647 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.647 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24652.0:522 7.9 e124/94 7:9970b578:::smithi08477203-15:head v 124'40, mlcod=118'36)", 2024-09-20T22:45:07.647 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:43:23.137456+0000", 2024-09-20T22:45:07.647 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 104.437535372, 2024-09-20T22:45:07.647 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0091263230000000004, 2024-09-20T22:45:07.647 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:45:07.647 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:45:07.647 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:45:07.648 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:45:07.648 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.648 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:45:07.648 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:23.137456+0000", 2024-09-20T22:45:07.648 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:45:07.648 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.648 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.648 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:45:07.648 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:23.137456+0000", 2024-09-20T22:45:07.648 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:45:07.648 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.648 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.648 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:45:07.649 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:23.137459+0000", 2024-09-20T22:45:07.649 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.661e-06 2024-09-20T22:45:07.649 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.649 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.649 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:45:07.649 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:23.139409+0000", 2024-09-20T22:45:07.649 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0019496470000000001 2024-09-20T22:45:07.649 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.649 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.649 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:45:07.649 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:23.139437+0000", 2024-09-20T22:45:07.649 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.8337e-05 2024-09-20T22:45:07.649 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.649 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.650 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:45:07.650 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:23.139449+0000", 2024-09-20T22:45:07.650 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.169e-05 2024-09-20T22:45:07.650 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.650 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.650 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:45:07.650 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:23.139540+0000", 2024-09-20T22:45:07.650 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.1650999999999994e-05 2024-09-20T22:45:07.650 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.650 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.650 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:45:07.650 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:23.139563+0000", 2024-09-20T22:45:07.650 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.3186000000000001e-05 2024-09-20T22:45:07.650 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.651 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.651 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:45:07.651 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:23.146527+0000", 2024-09-20T22:45:07.651 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0069635870000000002 2024-09-20T22:45:07.651 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.651 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.651 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:45:07.651 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:23.146583+0000", 2024-09-20T22:45:07.651 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.5563999999999999e-05 2024-09-20T22:45:07.651 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:07.651 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:45:07.651 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:07.651 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.651 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.651 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(osd.6.0:220 7.d e137/94 7:b97fe42f:::smithi08477203-36:16 v 137'29, mlcod=135'25)", 2024-09-20T22:45:07.652 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:43:36.132464+0000", 2024-09-20T22:45:07.652 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 91.442527326999993, 2024-09-20T22:45:07.652 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0087387449999999992, 2024-09-20T22:45:07.652 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:45:07.652 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:45:07.652 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:45:07.652 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:45:07.652 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.652 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:45:07.652 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.132464+0000", 2024-09-20T22:45:07.652 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:45:07.652 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.652 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.652 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:45:07.653 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.132464+0000", 2024-09-20T22:45:07.653 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:45:07.653 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.653 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.653 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:45:07.653 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.132466+0000", 2024-09-20T22:45:07.653 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.217e-06 2024-09-20T22:45:07.653 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.653 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.653 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:45:07.653 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.132473+0000", 2024-09-20T22:45:07.653 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.5340000000000002e-06 2024-09-20T22:45:07.653 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.653 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.653 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:45:07.654 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.132486+0000", 2024-09-20T22:45:07.654 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3081999999999999e-05 2024-09-20T22:45:07.654 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.654 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.654 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:45:07.654 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.132496+0000", 2024-09-20T22:45:07.654 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0074e-05 2024-09-20T22:45:07.654 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.654 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.654 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:45:07.654 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.139264+0000", 2024-09-20T22:45:07.654 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0067681019999999998 2024-09-20T22:45:07.654 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.654 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.655 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:45:07.655 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.139288+0000", 2024-09-20T22:45:07.655 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.3638e-05 2024-09-20T22:45:07.655 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.655 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.655 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:45:07.655 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.141160+0000", 2024-09-20T22:45:07.655 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001872475 2024-09-20T22:45:07.655 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.655 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.655 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:45:07.655 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.141203+0000", 2024-09-20T22:45:07.655 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.2623000000000002e-05 2024-09-20T22:45:07.655 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:07.655 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:45:07.656 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:07.656 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.656 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.656 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(osd.6.0:250 7.d e143/94 7:b97fe42f:::smithi08477203-36:16 v 143'37, mlcod=141'33)", 2024-09-20T22:45:07.656 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:43:42.325595+0000", 2024-09-20T22:45:07.656 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 85.249396714, 2024-09-20T22:45:07.656 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0076303170000000002, 2024-09-20T22:45:07.656 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:45:07.656 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:45:07.656 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:45:07.656 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:45:07.656 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.656 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:45:07.656 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:42.325595+0000", 2024-09-20T22:45:07.656 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:45:07.657 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.657 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.657 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:45:07.657 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:42.325595+0000", 2024-09-20T22:45:07.657 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:45:07.657 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.657 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.657 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:45:07.657 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:42.325595+0000", 2024-09-20T22:45:07.657 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.32e-07 2024-09-20T22:45:07.657 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.657 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.657 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:45:07.657 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:42.325605+0000", 2024-09-20T22:45:07.658 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.6800000000000005e-06 2024-09-20T22:45:07.658 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.658 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.658 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:45:07.658 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:42.325614+0000", 2024-09-20T22:45:07.658 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.1800000000000002e-06 2024-09-20T22:45:07.658 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.658 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.658 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:45:07.658 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:42.325621+0000", 2024-09-20T22:45:07.658 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.4309999999999999e-06 2024-09-20T22:45:07.658 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.658 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.658 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:45:07.659 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:42.325689+0000", 2024-09-20T22:45:07.659 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.8183999999999996e-05 2024-09-20T22:45:07.659 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.659 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.659 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:45:07.659 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:42.325709+0000", 2024-09-20T22:45:07.659 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.0152999999999998e-05 2024-09-20T22:45:07.659 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.659 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.659 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:45:07.659 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:42.332817+0000", 2024-09-20T22:45:07.659 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0071076330000000004 2024-09-20T22:45:07.659 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.659 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.660 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:45:07.660 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:42.333225+0000", 2024-09-20T22:45:07.660 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00040862400000000001 2024-09-20T22:45:07.660 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:07.660 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:45:07.660 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:07.660 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.660 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.660 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(osd.6.0:251 7.d e143/94 7:b30e4612:::smithi08477203-42:f v 143'39, mlcod=141'33)", 2024-09-20T22:45:07.660 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:43:42.326221+0000", 2024-09-20T22:45:07.660 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 85.248770469999997, 2024-09-20T22:45:07.660 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0070737200000000004, 2024-09-20T22:45:07.660 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:45:07.660 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:45:07.660 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:45:07.661 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:45:07.661 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.661 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:45:07.661 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:42.326221+0000", 2024-09-20T22:45:07.661 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:45:07.661 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.661 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.661 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:45:07.661 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:42.326221+0000", 2024-09-20T22:45:07.661 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:45:07.661 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.661 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.661 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:45:07.661 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:42.326222+0000", 2024-09-20T22:45:07.662 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.44e-07 2024-09-20T22:45:07.662 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.662 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.662 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:45:07.662 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:42.326231+0000", 2024-09-20T22:45:07.662 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.3929999999999997e-06 2024-09-20T22:45:07.662 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.662 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.662 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:45:07.662 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:42.326242+0000", 2024-09-20T22:45:07.662 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1072e-05 2024-09-20T22:45:07.662 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.662 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.662 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:45:07.662 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:42.326249+0000", 2024-09-20T22:45:07.663 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.4060000000000002e-06 2024-09-20T22:45:07.663 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.663 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.663 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:45:07.663 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:42.332430+0000", 2024-09-20T22:45:07.663 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0061818139999999999 2024-09-20T22:45:07.663 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.663 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.663 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:45:07.663 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:42.332451+0000", 2024-09-20T22:45:07.663 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.0990999999999999e-05 2024-09-20T22:45:07.663 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.663 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.663 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:45:07.664 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:42.333237+0000", 2024-09-20T22:45:07.664 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00078589199999999995 2024-09-20T22:45:07.664 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.664 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.664 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:45:07.664 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:42.333295+0000", 2024-09-20T22:45:07.664 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.7608000000000001e-05 2024-09-20T22:45:07.664 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:07.664 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:45:07.664 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:07.664 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.664 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.664 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:885 7.b 7:d3ca3619:::smithi08477203-39:1c [read 0~3848770,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e155)", 2024-09-20T22:45:07.664 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:43:54.130576+0000", 2024-09-20T22:45:07.665 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 73.444415184999997, 2024-09-20T22:45:07.665 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.006868552, 2024-09-20T22:45:07.665 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:45:07.665 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:45:07.665 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-20T22:45:07.665 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:45:07.665 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:45:07.665 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:45:07.665 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 885 2024-09-20T22:45:07.665 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.665 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:45:07.665 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.665 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:45:07.665 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:54.130576+0000", 2024-09-20T22:45:07.665 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:45:07.666 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.666 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.666 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:45:07.666 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:54.130576+0000", 2024-09-20T22:45:07.666 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:45:07.666 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.666 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.666 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:45:07.666 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:54.130577+0000", 2024-09-20T22:45:07.666 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.1399999999999997e-07 2024-09-20T22:45:07.666 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.666 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.666 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:45:07.667 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:54.130580+0000", 2024-09-20T22:45:07.667 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.8219999999999998e-06 2024-09-20T22:45:07.667 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.667 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.667 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:45:07.667 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:54.130584+0000", 2024-09-20T22:45:07.667 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.7210000000000002e-06 2024-09-20T22:45:07.667 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.667 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.667 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:45:07.667 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:54.130588+0000", 2024-09-20T22:45:07.667 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.349e-06 2024-09-20T22:45:07.667 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.667 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.668 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:45:07.668 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:54.130681+0000", 2024-09-20T22:45:07.668 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.3256000000000003e-05 2024-09-20T22:45:07.668 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.668 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.668 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting_for_map not empty", 2024-09-20T22:45:07.668 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:54.130686+0000", 2024-09-20T22:45:07.668 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.3900000000000001e-06 2024-09-20T22:45:07.668 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.668 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.668 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:45:07.668 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:54.135443+0000", 2024-09-20T22:45:07.668 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0047561189999999996 2024-09-20T22:45:07.668 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.668 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.669 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:45:07.669 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:54.135502+0000", 2024-09-20T22:45:07.669 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.9713999999999998e-05 2024-09-20T22:45:07.669 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.669 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.669 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:45:07.669 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:54.137445+0000", 2024-09-20T22:45:07.669 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001942767 2024-09-20T22:45:07.669 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:07.669 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:45:07.669 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:07.669 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.669 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.669 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:886 7.b 7:d3ca3619:::smithi08477203-39:1c [sparse-read 0~1558754] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e155)", 2024-09-20T22:45:07.669 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:43:54.130606+0000", 2024-09-20T22:45:07.670 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 73.444386089999995, 2024-09-20T22:45:07.670 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0070683969999999997, 2024-09-20T22:45:07.670 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:45:07.670 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:45:07.670 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-20T22:45:07.670 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:45:07.670 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:45:07.670 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:45:07.670 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 886 2024-09-20T22:45:07.670 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.670 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:45:07.670 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.670 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:45:07.671 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:54.130606+0000", 2024-09-20T22:45:07.671 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:45:07.671 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.671 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.671 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:45:07.671 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:54.130606+0000", 2024-09-20T22:45:07.671 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:45:07.671 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.671 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.671 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:45:07.671 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:54.130606+0000", 2024-09-20T22:45:07.671 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.1800000000000002e-07 2024-09-20T22:45:07.671 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.671 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.672 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:45:07.672 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:54.130607+0000", 2024-09-20T22:45:07.672 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.415e-06 2024-09-20T22:45:07.672 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.672 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.672 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:45:07.672 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:54.130613+0000", 2024-09-20T22:45:07.672 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.3410000000000002e-06 2024-09-20T22:45:07.672 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.672 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.672 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:45:07.672 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:54.130621+0000", 2024-09-20T22:45:07.672 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.7879999999999995e-06 2024-09-20T22:45:07.672 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.672 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.673 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:45:07.673 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:54.130702+0000", 2024-09-20T22:45:07.673 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.0889000000000004e-05 2024-09-20T22:45:07.673 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.673 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.673 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting_for_map not empty", 2024-09-20T22:45:07.673 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:54.130705+0000", 2024-09-20T22:45:07.673 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.5720000000000001e-06 2024-09-20T22:45:07.673 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.673 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.673 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:45:07.673 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:54.137478+0000", 2024-09-20T22:45:07.673 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00677292 2024-09-20T22:45:07.673 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.674 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.674 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:45:07.674 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:54.137561+0000", 2024-09-20T22:45:07.674 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.3157000000000004e-05 2024-09-20T22:45:07.674 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.674 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.674 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:45:07.674 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:54.137674+0000", 2024-09-20T22:45:07.674 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000112997 2024-09-20T22:45:07.674 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:07.674 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:45:07.674 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:07.674 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.674 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.675 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:1034 7.b 7:d960cad4:::smithi08477203-18:head [copy-from ver 3] snapc 25=[25,24,23,21] ondisk+write+known_if_redirected+supports_pool_eio e166)", 2024-09-20T22:45:07.675 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:44:05.163199+0000", 2024-09-20T22:45:07.675 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 62.411792478999999, 2024-09-20T22:45:07.675 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0099447649999999995, 2024-09-20T22:45:07.675 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:45:07.675 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:45:07.675 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:45:07.675 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:45:07.675 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:45:07.675 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:45:07.675 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1034 2024-09-20T22:45:07.675 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.675 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:45:07.675 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.676 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:45:07.676 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:05.163199+0000", 2024-09-20T22:45:07.676 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:45:07.676 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.676 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.676 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:45:07.676 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:05.163199+0000", 2024-09-20T22:45:07.676 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:45:07.676 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.676 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.676 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:45:07.676 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:05.163201+0000", 2024-09-20T22:45:07.676 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.2919999999999998e-06 2024-09-20T22:45:07.676 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.676 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.677 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:45:07.677 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:05.163212+0000", 2024-09-20T22:45:07.677 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0669000000000001e-05 2024-09-20T22:45:07.677 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.677 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.677 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:45:07.677 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:05.163236+0000", 2024-09-20T22:45:07.677 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.3591000000000002e-05 2024-09-20T22:45:07.677 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.677 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.677 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:45:07.677 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:05.163247+0000", 2024-09-20T22:45:07.677 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1031e-05 2024-09-20T22:45:07.677 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.678 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.678 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:45:07.678 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:05.163310+0000", 2024-09-20T22:45:07.678 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.3006e-05 2024-09-20T22:45:07.678 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.678 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.678 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:45:07.678 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:05.163373+0000", 2024-09-20T22:45:07.678 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.2765e-05 2024-09-20T22:45:07.678 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.678 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.678 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5,6", 2024-09-20T22:45:07.679 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:05.166182+0000", 2024-09-20T22:45:07.679 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0028092080000000001 2024-09-20T22:45:07.679 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.679 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.679 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:45:07.679 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:05.169658+0000", 2024-09-20T22:45:07.679 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0034766390000000001 2024-09-20T22:45:07.679 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.679 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.679 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:45:07.679 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:05.173003+0000", 2024-09-20T22:45:07.679 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0033444740000000001 2024-09-20T22:45:07.679 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.680 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.680 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:45:07.680 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:05.173083+0000", 2024-09-20T22:45:07.680 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.9670000000000001e-05 2024-09-20T22:45:07.680 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.680 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.680 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:45:07.680 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:05.173117+0000", 2024-09-20T22:45:07.680 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.4740999999999998e-05 2024-09-20T22:45:07.680 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.680 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.680 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:45:07.680 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:05.173144+0000", 2024-09-20T22:45:07.680 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.6679e-05 2024-09-20T22:45:07.681 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:07.681 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:45:07.681 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:07.681 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.681 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.681 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(osd.3.0:230 7.1 e169/157 7:8422f82d:::smithi08477203-22:24 v 169'70, mlcod=164'66)", 2024-09-20T22:45:07.681 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:44:08.161278+0000", 2024-09-20T22:45:07.681 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 59.413713756999996, 2024-09-20T22:45:07.681 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0069932529999999996, 2024-09-20T22:45:07.681 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:45:07.681 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:45:07.681 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:45:07.681 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:45:07.682 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.682 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:45:07.682 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:08.161278+0000", 2024-09-20T22:45:07.682 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:45:07.682 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.682 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.682 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:45:07.682 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:08.161278+0000", 2024-09-20T22:45:07.682 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:45:07.682 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.682 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.682 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:45:07.682 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:08.161278+0000", 2024-09-20T22:45:07.683 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.22e-07 2024-09-20T22:45:07.683 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.683 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.683 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:45:07.683 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:08.161283+0000", 2024-09-20T22:45:07.683 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.0420000000000002e-06 2024-09-20T22:45:07.683 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.683 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.683 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:45:07.683 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:08.166799+0000", 2024-09-20T22:45:07.683 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0055158300000000002 2024-09-20T22:45:07.683 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.683 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.683 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:45:07.684 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:08.166807+0000", 2024-09-20T22:45:07.684 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.8110000000000003e-06 2024-09-20T22:45:07.684 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.684 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.684 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:45:07.684 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:08.167389+0000", 2024-09-20T22:45:07.684 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00058173600000000002 2024-09-20T22:45:07.684 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.684 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.684 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:45:07.684 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:08.167408+0000", 2024-09-20T22:45:07.684 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.9089999999999998e-05 2024-09-20T22:45:07.684 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.684 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.685 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:45:07.685 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:08.168238+0000", 2024-09-20T22:45:07.685 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00083036700000000004 2024-09-20T22:45:07.685 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.685 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.685 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:45:07.685 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:08.168271+0000", 2024-09-20T22:45:07.685 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.3055000000000001e-05 2024-09-20T22:45:07.685 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:07.685 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:45:07.685 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:07.685 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.685 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.685 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24652.0:1419 7.1 e189/157 7:84a866f5:::smithi08477203-27:head v 189'83, mlcod=179'77)", 2024-09-20T22:45:07.686 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:44:28.156327+0000", 2024-09-20T22:45:07.686 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 39.418664667000002, 2024-09-20T22:45:07.686 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011141458, 2024-09-20T22:45:07.686 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:45:07.686 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:45:07.686 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:45:07.686 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:45:07.686 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.686 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:45:07.686 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.156327+0000", 2024-09-20T22:45:07.686 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:45:07.686 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.686 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.687 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:45:07.687 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.156327+0000", 2024-09-20T22:45:07.687 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:45:07.687 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.687 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.687 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:45:07.687 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.156329+0000", 2024-09-20T22:45:07.687 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.5349999999999999e-06 2024-09-20T22:45:07.687 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.687 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.687 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:45:07.687 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.156339+0000", 2024-09-20T22:45:07.687 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0135999999999999e-05 2024-09-20T22:45:07.687 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.688 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.688 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:45:07.688 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.156359+0000", 2024-09-20T22:45:07.688 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.986e-05 2024-09-20T22:45:07.688 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.688 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.688 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:45:07.688 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.156368+0000", 2024-09-20T22:45:07.688 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.7680000000000003e-06 2024-09-20T22:45:07.688 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.688 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.688 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:45:07.688 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.156435+0000", 2024-09-20T22:45:07.688 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.6629000000000002e-05 2024-09-20T22:45:07.688 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.689 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.689 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:45:07.689 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.156458+0000", 2024-09-20T22:45:07.689 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.2891000000000002e-05 2024-09-20T22:45:07.689 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.689 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.689 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:45:07.689 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.167412+0000", 2024-09-20T22:45:07.689 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.01095456 2024-09-20T22:45:07.689 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.689 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.689 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:45:07.689 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.167468+0000", 2024-09-20T22:45:07.689 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.6079000000000003e-05 2024-09-20T22:45:07.690 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:07.690 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:45:07.690 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:07.690 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.690 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.690 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:1725 7.b 7:d3ca3619:::smithi08477203-39:head [copy-from ver 101] snapc 3e=[3e,3a,39] ondisk+write+known_if_redirected+supports_pool_eio e217)", 2024-09-20T22:45:07.690 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:44:56.216644+0000", 2024-09-20T22:45:07.690 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 11.358347554, 2024-09-20T22:45:07.690 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.012098411999999999, 2024-09-20T22:45:07.690 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:45:07.690 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:45:07.690 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:45:07.690 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:45:07.690 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:45:07.691 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:45:07.691 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1725 2024-09-20T22:45:07.691 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.691 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:45:07.691 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.691 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:45:07.691 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:56.216644+0000", 2024-09-20T22:45:07.691 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:45:07.691 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.691 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.691 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:45:07.691 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:56.216644+0000", 2024-09-20T22:45:07.691 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:45:07.691 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.692 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.692 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:45:07.692 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:56.216647+0000", 2024-09-20T22:45:07.692 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.5550000000000001e-06 2024-09-20T22:45:07.692 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.692 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.692 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:45:07.692 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:56.216655+0000", 2024-09-20T22:45:07.692 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.5359999999999992e-06 2024-09-20T22:45:07.692 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.692 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.692 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:45:07.692 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:56.216674+0000", 2024-09-20T22:45:07.692 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.8652999999999999e-05 2024-09-20T22:45:07.693 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.693 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.693 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:45:07.693 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:56.216684+0000", 2024-09-20T22:45:07.693 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.6889999999999999e-06 2024-09-20T22:45:07.693 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.693 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.693 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:45:07.693 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:56.216772+0000", 2024-09-20T22:45:07.693 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.8374000000000005e-05 2024-09-20T22:45:07.693 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.693 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.693 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:45:07.693 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:56.216835+0000", 2024-09-20T22:45:07.694 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.3528000000000006e-05 2024-09-20T22:45:07.694 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.694 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.694 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5,6", 2024-09-20T22:45:07.694 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:56.219251+0000", 2024-09-20T22:45:07.694 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002415426 2024-09-20T22:45:07.694 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.694 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.694 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:45:07.694 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:56.224154+0000", 2024-09-20T22:45:07.694 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0049032269999999996 2024-09-20T22:45:07.694 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.694 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.695 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:45:07.695 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:56.226093+0000", 2024-09-20T22:45:07.695 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0019389190000000001 2024-09-20T22:45:07.695 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.695 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.695 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:45:07.695 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:56.228669+0000", 2024-09-20T22:45:07.695 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002576138 2024-09-20T22:45:07.695 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.695 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.695 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:45:07.695 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:56.228709+0000", 2024-09-20T22:45:07.695 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.9845000000000002e-05 2024-09-20T22:45:07.695 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:07.695 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:07.696 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:45:07.696 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:56.228743+0000", 2024-09-20T22:45:07.696 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.3522000000000002e-05 2024-09-20T22:45:07.696 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:07.696 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:45:07.696 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:07.696 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:07.696 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:45:07.696 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-20T22:45:07.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:07.698 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:07.698 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:45:07.799 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:45:07.820 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:07.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:07.821 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:45:07.922 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:45:07.944 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:07.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:07.945 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:45:08.046 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:45:08.067 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:08.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:08.067 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:45:08.142 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is set 2024-09-20T22:45:08.148 INFO:tasks.rados.rados.0.smithi084.stdout:686: done (0 left) 2024-09-20T22:45:08.148 INFO:tasks.rados.rados.0.smithi084.stdout:687: rollback oid 33 current snap is 69 2024-09-20T22:45:08.148 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 33 to 60 2024-09-20T22:45:08.149 INFO:tasks.rados.rados.0.smithi084.stdout:688: copy_from oid 7 from oid 16 current snap is 69 2024-09-20T22:45:08.149 INFO:tasks.rados.rados.0.smithi084.stdout:689: write oid 38 current snap is 69 2024-09-20T22:45:08.149 INFO:tasks.rados.rados.0.smithi084.stdout:689: seq_num 268 ranges {504276=639087,1666233=695410} 2024-09-20T22:45:08.155 INFO:tasks.rados.rados.0.smithi084.stdout:689: writing smithi08477203-38 from 504276 to 1143363 tid 1 2024-09-20T22:45:08.163 INFO:tasks.rados.rados.0.smithi084.stdout:689: writing smithi08477203-38 from 1666233 to 2361643 tid 2 2024-09-20T22:45:08.163 INFO:tasks.rados.rados.0.smithi084.stdout:690: copy_from oid 41 from oid 15 current snap is 69 2024-09-20T22:45:08.163 INFO:tasks.rados.rados.0.smithi084.stdout:687: finishing rollback tid 0 to smithi08477203-33 2024-09-20T22:45:08.163 INFO:tasks.rados.rados.0.smithi084.stdout:691: write (excl) oid 26 current snap is 69 2024-09-20T22:45:08.163 INFO:tasks.rados.rados.0.smithi084.stdout:691: seq_num 269 ranges {521083=714336,1724715=555421} 2024-09-20T22:45:08.169 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:45:08.172 INFO:tasks.rados.rados.0.smithi084.stdout:691: writing smithi08477203-26 from 521083 to 1235419 tid 1 2024-09-20T22:45:08.178 INFO:tasks.rados.rados.0.smithi084.stdout:691: writing smithi08477203-26 from 1724715 to 2280136 tid 2 2024-09-20T22:45:08.178 INFO:tasks.rados.rados.0.smithi084.stdout:692: read oid 30 snap 65 2024-09-20T22:45:08.178 INFO:tasks.rados.rados.0.smithi084.stdout:692: expect deleted 2024-09-20T22:45:08.178 INFO:tasks.rados.rados.0.smithi084.stdout:693: rollback oid 40 current snap is 69 2024-09-20T22:45:08.178 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 40 to 66 2024-09-20T22:45:08.179 INFO:tasks.rados.rados.0.smithi084.stdout:687: finishing rollback tid 1 to smithi08477203-33 2024-09-20T22:45:08.179 INFO:tasks.rados.rados.0.smithi084.stdout:688: finishing copy_from to smithi08477203-7 2024-09-20T22:45:08.179 INFO:tasks.rados.rados.0.smithi084.stdout:688: got expected ENOENT (src dne) 2024-09-20T22:45:08.179 INFO:tasks.rados.rados.0.smithi084.stdout:687: finishing rollback tid 2 to smithi08477203-33 2024-09-20T22:45:08.179 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 33 v 137 (ObjNum 229 snap 48 seq_num 229) dirty exists 2024-09-20T22:45:08.179 INFO:tasks.rados.rados.0.smithi084.stdout:688: finishing copy_from racing read to smithi08477203-7 2024-09-20T22:45:08.179 INFO:tasks.rados.rados.0.smithi084.stdout:687: done (6 left) 2024-09-20T22:45:08.179 INFO:tasks.rados.rados.0.smithi084.stdout:688: done (5 left) 2024-09-20T22:45:08.179 INFO:tasks.rados.rados.0.smithi084.stdout:694: delete oid 14 current snap is 69 2024-09-20T22:45:08.179 INFO:tasks.rados.rados.0.smithi084.stdout:690: finishing copy_from to smithi08477203-41 2024-09-20T22:45:08.179 INFO:tasks.rados.rados.0.smithi084.stdout:690: got expected ENOENT (src dne) 2024-09-20T22:45:08.179 INFO:tasks.rados.rados.0.smithi084.stdout:690: finishing copy_from racing read to smithi08477203-41 2024-09-20T22:45:08.179 INFO:tasks.rados.rados.0.smithi084.stdout:689: finishing write tid 1 to smithi08477203-38 2024-09-20T22:45:08.179 INFO:tasks.rados.rados.0.smithi084.stdout:689: finishing write tid 2 to smithi08477203-38 2024-09-20T22:45:08.180 INFO:tasks.rados.rados.0.smithi084.stdout:689: finishing write tid 3 to smithi08477203-38 2024-09-20T22:45:08.180 INFO:tasks.rados.rados.0.smithi084.stdout:689: finishing write tid 4 to smithi08477203-38 2024-09-20T22:45:08.180 INFO:tasks.rados.rados.0.smithi084.stdout:689: oid 38 updating version 0 to 185 2024-09-20T22:45:08.180 INFO:tasks.rados.rados.0.smithi084.stdout:689: oid 38 version 185 is already newer than 183 2024-09-20T22:45:08.180 INFO:tasks.rados.rados.0.smithi084.stdout:689: oid 38 version 185 is already newer than 184 2024-09-20T22:45:08.180 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 38 v 185 (ObjNum 268 snap 69 seq_num 268) dirty exists 2024-09-20T22:45:08.180 INFO:tasks.rados.rados.0.smithi084.stdout:689: left oid 38 (ObjNum 268 snap 69 seq_num 268) 2024-09-20T22:45:08.187 INFO:tasks.rados.rados.0.smithi084.stdout:691: finishing write tid 1 to smithi08477203-26 2024-09-20T22:45:08.190 INFO:tasks.rados.rados.0.smithi084.stdout:691: finishing write tid 2 to smithi08477203-26 2024-09-20T22:45:08.190 INFO:tasks.rados.rados.0.smithi084.stdout:691: finishing write tid 3 to smithi08477203-26 2024-09-20T22:45:08.191 INFO:tasks.rados.rados.0.smithi084.stdout:689: done (5 left) 2024-09-20T22:45:08.191 INFO:tasks.rados.rados.0.smithi084.stdout:690: done (4 left) 2024-09-20T22:45:08.191 INFO:tasks.rados.rados.0.smithi084.stdout:692: done (3 left) 2024-09-20T22:45:08.191 INFO:tasks.rados.rados.0.smithi084.stdout:694: done (2 left) 2024-09-20T22:45:08.191 INFO:tasks.rados.rados.0.smithi084.stdout:695: read oid 30 snap -1 2024-09-20T22:45:08.191 INFO:tasks.rados.rados.0.smithi084.stdout:695: expect deleted 2024-09-20T22:45:08.191 INFO:tasks.rados.rados.0.smithi084.stdout:696: write oid 10 current snap is 69 2024-09-20T22:45:08.191 INFO:tasks.rados.rados.0.smithi084.stdout:693: finishing rollback tid 0 to smithi08477203-40 2024-09-20T22:45:08.191 INFO:tasks.rados.rados.0.smithi084.stdout:696: seq_num 271 ranges {554697=464834,1725326=559090} 2024-09-20T22:45:08.196 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:08.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:08.197 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:45:08.197 INFO:tasks.rados.rados.0.smithi084.stdout:696: writing smithi08477203-10 from 554697 to 1019531 tid 1 2024-09-20T22:45:08.204 INFO:tasks.rados.rados.0.smithi084.stdout:696: writing smithi08477203-10 from 1725326 to 2284416 tid 2 2024-09-20T22:45:08.204 INFO:tasks.rados.rados.0.smithi084.stdout:697: read oid 44 snap -1 2024-09-20T22:45:08.204 INFO:tasks.rados.rados.0.smithi084.stdout:697: expect (ObjNum 129 snap 23 seq_num 129) 2024-09-20T22:45:08.204 INFO:tasks.rados.rados.0.smithi084.stdout:698: read oid 39 snap -1 2024-09-20T22:45:08.204 INFO:tasks.rados.rados.0.smithi084.stdout:698: expect (ObjNum 245 snap 54 seq_num 245) 2024-09-20T22:45:08.204 INFO:tasks.rados.rados.0.smithi084.stdout:699: delete oid 29 current snap is 69 2024-09-20T22:45:08.204 INFO:tasks.rados.rados.0.smithi084.stdout:691: finishing write tid 4 to smithi08477203-26 2024-09-20T22:45:08.204 INFO:tasks.rados.rados.0.smithi084.stdout:691: oid 26 updating version 0 to 170 2024-09-20T22:45:08.205 INFO:tasks.rados.rados.0.smithi084.stdout:691: oid 26 version 170 is already newer than 169 2024-09-20T22:45:08.205 INFO:tasks.rados.rados.0.smithi084.stdout:691: oid 26 version 170 is already newer than 168 2024-09-20T22:45:08.205 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 26 v 170 (ObjNum 269 snap 69 seq_num 269) dirty exists 2024-09-20T22:45:08.205 INFO:tasks.rados.rados.0.smithi084.stdout:691: left oid 26 (ObjNum 269 snap 69 seq_num 269) 2024-09-20T22:45:08.205 INFO:tasks.rados.rados.0.smithi084.stdout:693: finishing rollback tid 1 to smithi08477203-40 2024-09-20T22:45:08.205 INFO:tasks.rados.rados.0.smithi084.stdout:693: finishing rollback tid 2 to smithi08477203-40 2024-09-20T22:45:08.205 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 40 v 141 (ObjNum 238 snap 52 seq_num 238) dirty exists 2024-09-20T22:45:08.205 INFO:tasks.rados.rados.0.smithi084.stdout:696: finishing write tid 1 to smithi08477203-10 2024-09-20T22:45:08.207 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-20T22:45:08.210 INFO:tasks.rados.rados.0.smithi084.stdout:696: finishing write tid 2 to smithi08477203-10 2024-09-20T22:45:08.211 INFO:tasks.rados.rados.0.smithi084.stdout:696: finishing write tid 3 to smithi08477203-10 2024-09-20T22:45:08.211 INFO:tasks.rados.rados.0.smithi084.stdout:696: finishing write tid 4 to smithi08477203-10 2024-09-20T22:45:08.211 INFO:tasks.rados.rados.0.smithi084.stdout:696: oid 10 updating version 0 to 135 2024-09-20T22:45:08.211 INFO:tasks.rados.rados.0.smithi084.stdout:696: oid 10 updating version 135 to 136 2024-09-20T22:45:08.211 INFO:tasks.rados.rados.0.smithi084.stdout:696: oid 10 version 136 is already newer than 134 2024-09-20T22:45:08.211 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 10 v 136 (ObjNum 271 snap 69 seq_num 271) dirty exists 2024-09-20T22:45:08.211 INFO:tasks.rados.rados.0.smithi084.stdout:696: left oid 10 (ObjNum 271 snap 69 seq_num 271) 2024-09-20T22:45:08.211 INFO:tasks.rados.rados.0.smithi084.stdout:697: expect (ObjNum 129 snap 23 seq_num 129) 2024-09-20T22:45:08.285 INFO:tasks.rados.rados.0.smithi084.stdout:698: expect (ObjNum 245 snap 54 seq_num 245) 2024-09-20T22:45:08.298 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:45:08.320 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:08.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:08.321 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:45:08.409 INFO:tasks.rados.rados.0.smithi084.stdout:691: done (6 left) 2024-09-20T22:45:08.409 INFO:tasks.rados.rados.0.smithi084.stdout:693: done (5 left) 2024-09-20T22:45:08.409 INFO:tasks.rados.rados.0.smithi084.stdout:695: done (4 left) 2024-09-20T22:45:08.410 INFO:tasks.rados.rados.0.smithi084.stdout:696: done (3 left) 2024-09-20T22:45:08.410 INFO:tasks.rados.rados.0.smithi084.stdout:697: done (2 left) 2024-09-20T22:45:08.410 INFO:tasks.rados.rados.0.smithi084.stdout:698: done (1 left) 2024-09-20T22:45:08.410 INFO:tasks.rados.rados.0.smithi084.stdout:699: done (0 left) 2024-09-20T22:45:08.410 INFO:tasks.rados.rados.0.smithi084.stdout:700: snap_remove snap 65 2024-09-20T22:45:08.421 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:45:08.443 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:08.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:08.444 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:45:08.477 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:45:08 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: ::ffff:172.21.15.154 - - [20/Sep/2024:22:45:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-20T22:45:08.477 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:08 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/3879972590' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:45:08.477 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:08 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:45:08.477 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:08 smithi084 ceph-mon[40024]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:45:08.477 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:08 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/3879972590' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T22:45:08.478 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:08 smithi084 ceph-mon[40024]: osdmap e229: 8 total, 8 up, 7 in 2024-09-20T22:45:08.478 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:08 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/3879972590' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:45:08.478 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:08 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:45:08.478 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:08 smithi084 ceph-mon[29774]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:45:08.478 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:08 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/3879972590' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T22:45:08.478 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:08 smithi084 ceph-mon[29774]: osdmap e229: 8 total, 8 up, 7 in 2024-09-20T22:45:08.503 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-20T22:45:08.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:08 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/3879972590' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:45:08.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:08 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:45:08.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:08 smithi154 ceph-mon[33264]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:45:08.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:08 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/3879972590' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T22:45:08.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:08 smithi154 ceph-mon[33264]: osdmap e229: 8 total, 8 up, 7 in 2024-09-20T22:45:08.545 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:45:08.567 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:08.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:08.568 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:45:08.669 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:45:08.690 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:08.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:08.691 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:45:08.792 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:45:08.814 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:08.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:08.815 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:45:08.915 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:45:08.938 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:08.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:08.939 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:45:09.040 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:45:09.061 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:09.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:09.061 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:45:09.162 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:45:09.168 INFO:tasks.rados.rados.0.smithi084.stdout:700: done (0 left) 2024-09-20T22:45:09.168 INFO:tasks.rados.rados.0.smithi084.stdout:701: write (excl) oid 18 current snap is 69 2024-09-20T22:45:09.168 INFO:tasks.rados.rados.0.smithi084.stdout:701: seq_num 273 ranges {588311=615332,1842290=562759} 2024-09-20T22:45:09.175 INFO:tasks.rados.rados.0.smithi084.stdout:701: writing smithi08477203-18 from 588311 to 1203643 tid 1 2024-09-20T22:45:09.182 INFO:tasks.rados.rados.0.smithi084.stdout:701: writing smithi08477203-18 from 1842290 to 2405049 tid 2 2024-09-20T22:45:09.182 INFO:tasks.rados.rados.0.smithi084.stdout:702: delete oid 3 current snap is 69 2024-09-20T22:45:09.187 INFO:tasks.rados.rados.0.smithi084.stdout:701: finishing write tid 1 to smithi08477203-18 2024-09-20T22:45:09.188 INFO:tasks.rados.rados.0.smithi084.stdout:702: done (1 left) 2024-09-20T22:45:09.188 INFO:tasks.rados.rados.0.smithi084.stdout:703: read oid 1 snap -1 2024-09-20T22:45:09.188 INFO:tasks.rados.rados.0.smithi084.stdout:703: expect (ObjNum 237 snap 51 seq_num 237) 2024-09-20T22:45:09.188 INFO:tasks.rados.rados.0.smithi084.stdout:704: write (excl) oid 29 current snap is 69 2024-09-20T22:45:09.188 INFO:tasks.rados.rados.0.smithi084.stdout:704: seq_num 275 ranges {621925=482183,1559254=566428} 2024-09-20T22:45:09.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:09.189 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:09.189 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:45:09.194 INFO:tasks.rados.rados.0.smithi084.stdout:704: writing smithi08477203-29 from 621925 to 1104108 tid 1 2024-09-20T22:45:09.199 INFO:tasks.rados.rados.0.smithi084.stdout:704: writing smithi08477203-29 from 1559254 to 2125682 tid 2 2024-09-20T22:45:09.199 INFO:tasks.rados.rados.0.smithi084.stdout:705: snap_remove snap 63 2024-09-20T22:45:09.290 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:45:09.311 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:09.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:09.312 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:45:09.323 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:09 smithi084 ceph-mon[40024]: pgmap v244: 178 pgs: 3 active+clean+snaptrim, 175 active+clean; 109 MiB data, 552 MiB used, 625 GiB / 626 GiB avail 2024-09-20T22:45:09.323 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:09 smithi084 ceph-mon[29774]: pgmap v244: 178 pgs: 3 active+clean+snaptrim, 175 active+clean; 109 MiB data, 552 MiB used, 625 GiB / 626 GiB avail 2024-09-20T22:45:09.412 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:45:09.435 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:09.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:09.435 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:45:09.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:09 smithi154 ceph-mon[33264]: pgmap v244: 178 pgs: 3 active+clean+snaptrim, 175 active+clean; 109 MiB data, 552 MiB used, 625 GiB / 626 GiB avail 2024-09-20T22:45:09.504 INFO:journalctl@ceph.iscsi.iscsi.a.smithi154.stdout:Sep 20 22:45:09 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-iscsi-iscsi-a[59730]: debug there is no tcmu-runner data available 2024-09-20T22:45:09.537 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:45:09.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:09.559 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:09.559 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:45:09.660 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:45:09.681 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:09.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:09.682 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:45:09.783 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:45:09.806 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:09.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:09.807 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:45:09.907 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:45:09.929 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:09.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:09.930 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:45:10.010 INFO:tasks.thrashosds.thrasher:j is {'epoch': 227, 'fsid': 'a9c0bc70-77a0-11ef-baf5-efdc52797490', 'created': '2024-09-20T22:36:30.240626+0000', 'modified': '2024-09-20T22:45:06.142462+0000', 'last_up_change': '2024-09-20T22:41:27.098565+0000', 'last_in_change': '2024-09-20T22:43:56.388784+0000', 'flags': 'noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5801984, 'flags_set': ['noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 18, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 7, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-20T22:39:21.984381+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 21, 'pg_placement_num': 21, 'pg_placement_num_target': 21, 'pg_num_target': 21, 'pg_num_pending': 21, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '147', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '119', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.899999976158142, 'score_stable': 1.899999976158142, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.6699999570846558, 'raw_score_stable': 1.6699999570846558, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-09-20T22:41:35.820081+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '69', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 69, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.6500000953674316, 'score_stable': 2.6500000953674316, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 2.3299999237060547, 'raw_score_stable': 2.3299999237060547, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-09-20T22:41:36.546389+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.7400000095367432, 'score_stable': 1.7400000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.5299999713897705, 'raw_score_stable': 1.5299999713897705, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-09-20T22:41:38.454640+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '70', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.240000009536743, 'score_stable': 2.240000009536743, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.9700000286102295, 'raw_score_stable': 1.9700000286102295, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-09-20T22:41:40.407129+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '72', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.7400000095367432, 'score_stable': 1.7400000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.5299999713897705, 'raw_score_stable': 1.5299999713897705, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-09-20T22:41:42.476716+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '74', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'unique_pool_0', 'create_time': '2024-09-20T22:42:51.354358+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 26, 'pg_placement_num': 26, 'pg_placement_num_target': 26, 'pg_num_target': 26, 'pg_num_pending': 26, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '227', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '137', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 69, 'snap_epoch': 227, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.140000104904175, 'score_stable': 2.140000104904175, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': 'ea36a489-6f91-4b2f-b784-b90506b9805d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 169, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6801', 'nonce': 268084626}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6802', 'nonce': 268084626}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6804', 'nonce': 268084626}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6803', 'nonce': 268084626}]}, 'public_addr': '172.21.15.84:6801/268084626', 'cluster_addr': '172.21.15.84:6802/268084626', 'heartbeat_back_addr': '172.21.15.84:6804/268084626', 'heartbeat_front_addr': '172.21.15.84:6803/268084626', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'c8649d2f-27ef-47f8-bc94-f1cd9e62bdb8', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 160, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6805', 'nonce': 418374136}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6806', 'nonce': 418374136}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6808', 'nonce': 418374136}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6807', 'nonce': 418374136}]}, 'public_addr': '172.21.15.84:6805/418374136', 'cluster_addr': '172.21.15.84:6806/418374136', 'heartbeat_back_addr': '172.21.15.84:6808/418374136', 'heartbeat_front_addr': '172.21.15.84:6807/418374136', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'c728eb48-a0f0-4de4-a651-7f3bec867280', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 160, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6809', 'nonce': 3977168611}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6810', 'nonce': 3977168611}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6812', 'nonce': 3977168611}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6811', 'nonce': 3977168611}]}, 'public_addr': '172.21.15.84:6809/3977168611', 'cluster_addr': '172.21.15.84:6810/3977168611', 'heartbeat_back_addr': '172.21.15.84:6812/3977168611', 'heartbeat_front_addr': '172.21.15.84:6811/3977168611', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '573f44eb-0853-4630-a7d8-9caaeb256f9b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 25, 'up_thru': 219, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6813', 'nonce': 2192239945}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6814', 'nonce': 2192239945}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6816', 'nonce': 2192239945}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6815', 'nonce': 2192239945}]}, 'public_addr': '172.21.15.84:6813/2192239945', 'cluster_addr': '172.21.15.84:6814/2192239945', 'heartbeat_back_addr': '172.21.15.84:6816/2192239945', 'heartbeat_front_addr': '172.21.15.84:6815/2192239945', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'a52c3b94-c5f3-490f-8ac8-09061b9b2abb', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 139, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6800', 'nonce': 1793136002}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6801', 'nonce': 1793136002}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6803', 'nonce': 1793136002}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6802', 'nonce': 1793136002}]}, 'public_addr': '172.21.15.154:6800/1793136002', 'cluster_addr': '172.21.15.154:6801/1793136002', 'heartbeat_back_addr': '172.21.15.154:6803/1793136002', 'heartbeat_front_addr': '172.21.15.154:6802/1793136002', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '4fa0dc8e-25cc-46cd-96ff-33f20b6faec3', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 40, 'up_thru': 169, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6804', 'nonce': 493899927}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6805', 'nonce': 493899927}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6807', 'nonce': 493899927}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6806', 'nonce': 493899927}]}, 'public_addr': '172.21.15.154:6804/493899927', 'cluster_addr': '172.21.15.154:6805/493899927', 'heartbeat_back_addr': '172.21.15.154:6807/493899927', 'heartbeat_front_addr': '172.21.15.154:6806/493899927', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '56788d74-810e-4827-a615-958c12f8a368', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 48, 'up_thru': 169, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6808', 'nonce': 2375091033}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6809', 'nonce': 2375091033}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6811', 'nonce': 2375091033}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6810', 'nonce': 2375091033}]}, 'public_addr': '172.21.15.154:6808/2375091033', 'cluster_addr': '172.21.15.154:6809/2375091033', 'heartbeat_back_addr': '172.21.15.154:6811/2375091033', 'heartbeat_front_addr': '172.21.15.154:6810/2375091033', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '83503727-afc1-4b95-8b96-ecb63f3338c2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 59, 'up_thru': 169, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6812', 'nonce': 3717704491}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6813', 'nonce': 3717704491}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6815', 'nonce': 3717704491}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6814', 'nonce': 3717704491}]}, 'public_addr': '172.21.15.154:6812/3717704491', 'cluster_addr': '172.21.15.154:6813/3717704491', 'heartbeat_back_addr': '172.21.15.154:6815/3717704491', 'heartbeat_front_addr': '172.21.15.154:6814/3717704491', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-20T22:43:38.373571+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-20T22:44:01.955517+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-20T22:44:49.887047+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-20T22:44:55.581914+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-09-20T22:45:02.445007+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-20T22:44:33.074031+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-20T22:44:30.460653+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-20T22:43:53.020113+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '3.18', 'mappings': [{'from': 0, 'to': 7}]}, {'pgid': '4.6', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.e', 'mappings': [{'from': 3, 'to': 2}]}, {'pgid': '4.11', 'mappings': [{'from': 3, 'to': 5}]}, {'pgid': '4.17', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.1d', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '5.3', 'mappings': [{'from': 0, 'to': 3}]}, {'pgid': '7.2', 'mappings': [{'from': 3, 'to': 5}]}, {'pgid': '7.13', 'mappings': [{'from': 7, 'to': 1}]}, {'pgid': '7.15', 'mappings': [{'from': 2, 'to': 6}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.84:0/542020545': '2024-09-21T22:42:07.596045+0000', '172.21.15.84:0/25873236': '2024-09-21T22:36:54.917812+0000', '172.21.15.84:6800/820908470': '2024-09-21T22:36:54.917812+0000', '172.21.15.84:0/567614908': '2024-09-21T22:36:54.917812+0000', '172.21.15.84:0/2963622459': '2024-09-21T22:42:07.596045+0000', '172.21.15.84:0/1176849276': '2024-09-21T22:37:17.690104+0000', '172.21.15.84:6800/2932420050': '2024-09-21T22:37:17.690104+0000', '172.21.15.84:0/3879263635': '2024-09-21T22:37:17.690104+0000', '172.21.15.84:0/969472279': '2024-09-21T22:36:54.917812+0000', '172.21.15.84:0/1361143214': '2024-09-21T22:37:17.690104+0000', '172.21.15.84:0/3353126955': '2024-09-21T22:42:07.596045+0000', '172.21.15.84:6800/4024278029': '2024-09-21T22:42:07.596045+0000', '172.21.15.84:0/547902262': '2024-09-21T22:42:07.596045+0000', '172.21.15.84:0/1242929317': '2024-09-21T22:42:07.596045+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [{'pool': 7, 'snaps': [{'begin': 57, 'length': 2}, {'begin': 63, 'length': 1}]}], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-20T22:45:10.010 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph pg dump --format=json 2024-09-20T22:45:10.022 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:45:10.030 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:45:10.031 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:45:10.054 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:10.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:10.055 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:45:10.156 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:45:10.178 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:10.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:10.179 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:45:10.181 INFO:tasks.rados.rados.0.smithi084.stdout:705: done (3 left) 2024-09-20T22:45:10.182 INFO:tasks.rados.rados.0.smithi084.stdout:706: write (excl) oid 8 current snap is 69 2024-09-20T22:45:10.182 INFO:tasks.rados.rados.0.smithi084.stdout:706: seq_num 276 ranges {638732=557432,1617736=710086} 2024-09-20T22:45:10.190 INFO:tasks.rados.rados.0.smithi084.stdout:706: writing smithi08477203-8 from 638732 to 1196164 tid 1 2024-09-20T22:45:10.198 INFO:tasks.rados.rados.0.smithi084.stdout:706: writing smithi08477203-8 from 1617736 to 2327822 tid 2 2024-09-20T22:45:10.198 INFO:tasks.rados.rados.0.smithi084.stdout:701: finishing write tid 2 to smithi08477203-18 2024-09-20T22:45:10.198 INFO:tasks.rados.rados.0.smithi084.stdout:701: finishing write tid 3 to smithi08477203-18 2024-09-20T22:45:10.198 INFO:tasks.rados.rados.0.smithi084.stdout:701: finishing write tid 4 to smithi08477203-18 2024-09-20T22:45:10.198 INFO:tasks.rados.rados.0.smithi084.stdout:701: oid 18 updating version 0 to 138 2024-09-20T22:45:10.198 INFO:tasks.rados.rados.0.smithi084.stdout:701: oid 18 updating version 138 to 140 2024-09-20T22:45:10.198 INFO:tasks.rados.rados.0.smithi084.stdout:701: oid 18 version 140 is already newer than 139 2024-09-20T22:45:10.198 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 18 v 140 (ObjNum 273 snap 69 seq_num 273) dirty exists 2024-09-20T22:45:10.198 INFO:tasks.rados.rados.0.smithi084.stdout:701: left oid 18 (ObjNum 273 snap 69 seq_num 273) 2024-09-20T22:45:10.199 INFO:tasks.rados.rados.0.smithi084.stdout:703: expect (ObjNum 237 snap 51 seq_num 237) 2024-09-20T22:45:10.280 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:45:10.301 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:10.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:10.302 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:45:10.329 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:10 smithi084 ceph-mon[40024]: purged_snaps scrub starts 2024-09-20T22:45:10.329 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:10 smithi084 ceph-mon[40024]: purged_snaps scrub ok 2024-09-20T22:45:10.329 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:10 smithi084 ceph-mon[40024]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:45:10.329 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:10 smithi084 ceph-mon[40024]: osdmap e230: 8 total, 8 up, 7 in 2024-09-20T22:45:10.329 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:10 smithi084 ceph-mon[29774]: purged_snaps scrub starts 2024-09-20T22:45:10.330 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:10 smithi084 ceph-mon[29774]: purged_snaps scrub ok 2024-09-20T22:45:10.330 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:10 smithi084 ceph-mon[29774]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:45:10.330 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:10 smithi084 ceph-mon[29774]: osdmap e230: 8 total, 8 up, 7 in 2024-09-20T22:45:10.403 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:45:10.429 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:10.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:10.430 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:45:10.444 INFO:tasks.rados.rados.0.smithi084.stdout:704: finishing write tid 1 to smithi08477203-29 2024-09-20T22:45:10.444 INFO:tasks.rados.rados.0.smithi084.stdout:704: finishing write tid 2 to smithi08477203-29 2024-09-20T22:45:10.444 INFO:tasks.rados.rados.0.smithi084.stdout:704: finishing write tid 3 to smithi08477203-29 2024-09-20T22:45:10.444 INFO:tasks.rados.rados.0.smithi084.stdout:704: finishing write tid 4 to smithi08477203-29 2024-09-20T22:45:10.444 INFO:tasks.rados.rados.0.smithi084.stdout:704: oid 29 updating version 0 to 322 2024-09-20T22:45:10.444 INFO:tasks.rados.rados.0.smithi084.stdout:704: oid 29 version 322 is already newer than 321 2024-09-20T22:45:10.444 INFO:tasks.rados.rados.0.smithi084.stdout:704: oid 29 updating version 322 to 323 2024-09-20T22:45:10.444 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 29 v 323 (ObjNum 275 snap 69 seq_num 275) dirty exists 2024-09-20T22:45:10.444 INFO:tasks.rados.rados.0.smithi084.stdout:704: left oid 29 (ObjNum 275 snap 69 seq_num 275) 2024-09-20T22:45:10.444 INFO:tasks.rados.rados.0.smithi084.stdout:706: finishing write tid 1 to smithi08477203-8 2024-09-20T22:45:10.444 INFO:tasks.rados.rados.0.smithi084.stdout:701: done (3 left) 2024-09-20T22:45:10.444 INFO:tasks.rados.rados.0.smithi084.stdout:703: done (2 left) 2024-09-20T22:45:10.445 INFO:tasks.rados.rados.0.smithi084.stdout:704: done (1 left) 2024-09-20T22:45:10.445 INFO:tasks.rados.rados.0.smithi084.stdout:707: read oid 16 snap 66 2024-09-20T22:45:10.445 INFO:tasks.rados.rados.0.smithi084.stdout:707: expect deleted 2024-09-20T22:45:10.445 INFO:tasks.rados.rados.0.smithi084.stdout:708: write (excl) oid 1 current snap is 69 2024-09-20T22:45:10.445 INFO:tasks.rados.rados.0.smithi084.stdout:706: finishing write tid 2 to smithi08477203-8 2024-09-20T22:45:10.445 INFO:tasks.rados.rados.0.smithi084.stdout:706: finishing write tid 3 to smithi08477203-8 2024-09-20T22:45:10.445 INFO:tasks.rados.rados.0.smithi084.stdout:708: seq_num 277 ranges {655539=632681,2076218=570097} 2024-09-20T22:45:10.450 INFO:tasks.rados.rados.0.smithi084.stdout:708: writing smithi08477203-1 from 655539 to 1288220 tid 1 2024-09-20T22:45:10.458 INFO:tasks.rados.rados.0.smithi084.stdout:708: writing smithi08477203-1 from 2076218 to 2646315 tid 2 2024-09-20T22:45:10.458 INFO:tasks.rados.rados.0.smithi084.stdout:709: write (excl) oid 24 current snap is 69 2024-09-20T22:45:10.458 INFO:tasks.rados.rados.0.smithi084.stdout:709: seq_num 278 ranges {672346=707930,2134700=537646} 2024-09-20T22:45:10.465 INFO:tasks.rados.rados.0.smithi084.stdout:709: writing smithi08477203-24 from 672346 to 1380276 tid 1 2024-09-20T22:45:10.473 INFO:tasks.rados.rados.0.smithi084.stdout:709: writing smithi08477203-24 from 2134700 to 2672346 tid 2 2024-09-20T22:45:10.473 INFO:tasks.rados.rados.0.smithi084.stdout:710: snap_create 2024-09-20T22:45:10.473 INFO:tasks.rados.rados.0.smithi084.stdout:706: finishing write tid 4 to smithi08477203-8 2024-09-20T22:45:10.473 INFO:tasks.rados.rados.0.smithi084.stdout:706: oid 8 updating version 0 to 103 2024-09-20T22:45:10.473 INFO:tasks.rados.rados.0.smithi084.stdout:706: oid 8 updating version 103 to 105 2024-09-20T22:45:10.473 INFO:tasks.rados.rados.0.smithi084.stdout:706: oid 8 version 105 is already newer than 104 2024-09-20T22:45:10.474 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 8 v 105 (ObjNum 276 snap 69 seq_num 276) dirty exists 2024-09-20T22:45:10.474 INFO:tasks.rados.rados.0.smithi084.stdout:706: left oid 8 (ObjNum 276 snap 69 seq_num 276) 2024-09-20T22:45:10.474 INFO:tasks.rados.rados.0.smithi084.stdout:708: finishing write tid 1 to smithi08477203-1 2024-09-20T22:45:10.474 INFO:tasks.rados.rados.0.smithi084.stdout:708: finishing write tid 2 to smithi08477203-1 2024-09-20T22:45:10.474 INFO:tasks.rados.rados.0.smithi084.stdout:708: finishing write tid 3 to smithi08477203-1 2024-09-20T22:45:10.474 INFO:tasks.rados.rados.0.smithi084.stdout:708: finishing write tid 4 to smithi08477203-1 2024-09-20T22:45:10.474 INFO:tasks.rados.rados.0.smithi084.stdout:708: oid 1 updating version 0 to 144 2024-09-20T22:45:10.474 INFO:tasks.rados.rados.0.smithi084.stdout:708: oid 1 updating version 144 to 146 2024-09-20T22:45:10.474 INFO:tasks.rados.rados.0.smithi084.stdout:708: oid 1 version 146 is already newer than 145 2024-09-20T22:45:10.474 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 1 v 146 (ObjNum 277 snap 69 seq_num 277) dirty exists 2024-09-20T22:45:10.474 INFO:tasks.rados.rados.0.smithi084.stdout:708: left oid 1 (ObjNum 277 snap 69 seq_num 277) 2024-09-20T22:45:10.474 INFO:tasks.rados.rados.0.smithi084.stdout:709: finishing write tid 1 to smithi08477203-24 2024-09-20T22:45:10.478 INFO:tasks.rados.rados.0.smithi084.stdout:709: finishing write tid 2 to smithi08477203-24 2024-09-20T22:45:10.478 INFO:tasks.rados.rados.0.smithi084.stdout:709: finishing write tid 3 to smithi08477203-24 2024-09-20T22:45:10.478 INFO:tasks.rados.rados.0.smithi084.stdout:709: finishing write tid 4 to smithi08477203-24 2024-09-20T22:45:10.478 INFO:tasks.rados.rados.0.smithi084.stdout:709: oid 24 updating version 0 to 75 2024-09-20T22:45:10.478 INFO:tasks.rados.rados.0.smithi084.stdout:709: oid 24 updating version 75 to 76 2024-09-20T22:45:10.478 INFO:tasks.rados.rados.0.smithi084.stdout:709: oid 24 version 76 is already newer than 74 2024-09-20T22:45:10.478 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 24 v 76 (ObjNum 278 snap 69 seq_num 278) dirty exists 2024-09-20T22:45:10.478 INFO:tasks.rados.rados.0.smithi084.stdout:709: left oid 24 (ObjNum 278 snap 69 seq_num 278) 2024-09-20T22:45:10.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:10 smithi154 ceph-mon[33264]: purged_snaps scrub starts 2024-09-20T22:45:10.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:10 smithi154 ceph-mon[33264]: purged_snaps scrub ok 2024-09-20T22:45:10.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:10 smithi154 ceph-mon[33264]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:45:10.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:10 smithi154 ceph-mon[33264]: osdmap e230: 8 total, 8 up, 7 in 2024-09-20T22:45:10.531 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:45:10.552 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:10.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:10.553 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:45:10.564 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:45:10.564 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:45:10.577 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:45:10.577 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:45:10.594 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:45:10.595 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:45:10.600 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:45:10.614 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-20T22:45:10.614 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T22:45:10.634 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:45:10.634 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:45:10.650 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:45:10.651 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:45:10.654 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:45:10.667 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:45:10.668 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:45:10.674 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:10.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:10.675 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:45:10.685 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:45:10.685 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:45:10.776 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:45:10.803 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:10.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:10.804 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:45:10.904 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:45:10.929 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:10.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:10.930 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:45:11.031 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:45:11.055 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:11.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:11.056 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:45:11.157 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:45:11.226 INFO:tasks.rados.rados.0.smithi084.stdout:706: done (4 left) 2024-09-20T22:45:11.226 INFO:tasks.rados.rados.0.smithi084.stdout:707: done (3 left) 2024-09-20T22:45:11.226 INFO:tasks.rados.rados.0.smithi084.stdout:708: done (2 left) 2024-09-20T22:45:11.226 INFO:tasks.rados.rados.0.smithi084.stdout:709: done (1 left) 2024-09-20T22:45:11.226 INFO:tasks.rados.rados.0.smithi084.stdout:710: done (0 left) 2024-09-20T22:45:11.227 INFO:tasks.rados.rados.0.smithi084.stdout:711: write oid 14 current snap is 70 2024-09-20T22:45:11.227 INFO:tasks.rados.rados.0.smithi084.stdout:711: seq_num 279 ranges {689153=783179,2076829=457413} 2024-09-20T22:45:11.227 INFO:tasks.rados.rados.0.smithi084.stdout:711: writing smithi08477203-14 from 689153 to 1472332 tid 1 2024-09-20T22:45:11.227 INFO:tasks.rados.rados.0.smithi084.stdout:711: writing smithi08477203-14 from 2076829 to 2534242 tid 2 2024-09-20T22:45:11.227 INFO:tasks.rados.rados.0.smithi084.stdout:711: finishing write tid 1 to smithi08477203-14 2024-09-20T22:45:11.227 INFO:tasks.rados.rados.0.smithi084.stdout:712: read oid 32 snap -1 2024-09-20T22:45:11.227 INFO:tasks.rados.rados.0.smithi084.stdout:712: expect deleted 2024-09-20T22:45:11.227 INFO:tasks.rados.rados.0.smithi084.stdout:713: read oid 39 snap -1 2024-09-20T22:45:11.227 INFO:tasks.rados.rados.0.smithi084.stdout:713: expect (ObjNum 245 snap 54 seq_num 245) 2024-09-20T22:45:11.228 INFO:tasks.rados.rados.0.smithi084.stdout:714: delete oid 26 current snap is 70 2024-09-20T22:45:11.228 INFO:tasks.rados.rados.0.smithi084.stdout:713: expect (ObjNum 245 snap 54 seq_num 245) 2024-09-20T22:45:11.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:11.255 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:11.255 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:45:11.327 INFO:tasks.rados.rados.0.smithi084.stdout:711: finishing write tid 2 to smithi08477203-14 2024-09-20T22:45:11.328 INFO:tasks.rados.rados.0.smithi084.stdout:711: finishing write tid 3 to smithi08477203-14 2024-09-20T22:45:11.328 INFO:tasks.rados.rados.0.smithi084.stdout:712: done (3 left) 2024-09-20T22:45:11.328 INFO:tasks.rados.rados.0.smithi084.stdout:713: done (2 left) 2024-09-20T22:45:11.328 INFO:tasks.rados.rados.0.smithi084.stdout:714: done (1 left) 2024-09-20T22:45:11.329 INFO:tasks.rados.rados.0.smithi084.stdout:715: read oid 11 snap 69 2024-09-20T22:45:11.329 INFO:tasks.rados.rados.0.smithi084.stdout:715: expect (ObjNum 245 snap 54 seq_num 245) 2024-09-20T22:45:11.329 INFO:tasks.rados.rados.0.smithi084.stdout:716: read oid 3 snap -1 2024-09-20T22:45:11.329 INFO:tasks.rados.rados.0.smithi084.stdout:716: expect deleted 2024-09-20T22:45:11.329 INFO:tasks.rados.rados.0.smithi084.stdout:717: snap_create 2024-09-20T22:45:11.329 INFO:tasks.rados.rados.0.smithi084.stdout:711: finishing write tid 4 to smithi08477203-14 2024-09-20T22:45:11.329 INFO:tasks.rados.rados.0.smithi084.stdout:711: oid 14 updating version 0 to 58 2024-09-20T22:45:11.330 INFO:tasks.rados.rados.0.smithi084.stdout:711: oid 14 updating version 58 to 59 2024-09-20T22:45:11.330 INFO:tasks.rados.rados.0.smithi084.stdout:711: oid 14 updating version 59 to 60 2024-09-20T22:45:11.330 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 14 v 60 (ObjNum 279 snap 70 seq_num 279) dirty exists 2024-09-20T22:45:11.330 INFO:tasks.rados.rados.0.smithi084.stdout:711: left oid 14 (ObjNum 279 snap 70 seq_num 279) 2024-09-20T22:45:11.330 INFO:tasks.rados.rados.0.smithi084.stdout:715: expect (ObjNum 245 snap 54 seq_num 245) 2024-09-20T22:45:11.350 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd unset noscrub 2024-09-20T22:45:11.356 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:45:11.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:11.381 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:11.381 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:45:11.483 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:45:11.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:11 smithi154 ceph-mon[33264]: pgmap v247: 178 pgs: 178 active+clean; 114 MiB data, 540 MiB used, 625 GiB / 626 GiB avail; 7.1 MiB/s rd, 886 KiB/s wr, 13 op/s 2024-09-20T22:45:11.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:11 smithi154 ceph-mon[33264]: osdmap e231: 8 total, 8 up, 7 in 2024-09-20T22:45:11.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:11 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/4184798319' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:45:11.506 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:11.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:11.507 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:45:11.553 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:11 smithi084 ceph-mon[29774]: pgmap v247: 178 pgs: 178 active+clean; 114 MiB data, 540 MiB used, 625 GiB / 626 GiB avail; 7.1 MiB/s rd, 886 KiB/s wr, 13 op/s 2024-09-20T22:45:11.554 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:11 smithi084 ceph-mon[29774]: osdmap e231: 8 total, 8 up, 7 in 2024-09-20T22:45:11.554 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:11 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/4184798319' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:45:11.554 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:11 smithi084 ceph-mon[40024]: pgmap v247: 178 pgs: 178 active+clean; 114 MiB data, 540 MiB used, 625 GiB / 626 GiB avail; 7.1 MiB/s rd, 886 KiB/s wr, 13 op/s 2024-09-20T22:45:11.554 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:11 smithi084 ceph-mon[40024]: osdmap e231: 8 total, 8 up, 7 in 2024-09-20T22:45:11.554 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:11 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/4184798319' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:45:11.608 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:45:11.630 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:11.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:11.631 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:45:11.731 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:45:11.756 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:11.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:11.757 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:45:11.858 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:45:11.878 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-20T22:45:11.878 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:45:11.881 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:11.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:11.882 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:45:11.983 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:45:12.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:12.010 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:12.010 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:45:12.111 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:45:12.137 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:12.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:12.138 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:45:12.191 INFO:tasks.rados.rados.0.smithi084.stdout:711: done (3 left) 2024-09-20T22:45:12.191 INFO:tasks.rados.rados.0.smithi084.stdout:715: done (2 left) 2024-09-20T22:45:12.191 INFO:tasks.rados.rados.0.smithi084.stdout:716: done (1 left) 2024-09-20T22:45:12.191 INFO:tasks.rados.rados.0.smithi084.stdout:717: done (0 left) 2024-09-20T22:45:12.191 INFO:tasks.rados.rados.0.smithi084.stdout:718: read oid 18 snap -1 2024-09-20T22:45:12.191 INFO:tasks.rados.rados.0.smithi084.stdout:718: expect (ObjNum 273 snap 69 seq_num 273) 2024-09-20T22:45:12.191 INFO:tasks.rados.rados.0.smithi084.stdout:719: write oid 16 current snap is 71 2024-09-20T22:45:12.191 INFO:tasks.rados.rados.0.smithi084.stdout:719: seq_num 281 ranges {722767=533677,1793793=461082} 2024-09-20T22:45:12.197 INFO:tasks.rados.rados.0.smithi084.stdout:719: writing smithi08477203-16 from 722767 to 1256444 tid 1 2024-09-20T22:45:12.204 INFO:tasks.rados.rados.0.smithi084.stdout:719: writing smithi08477203-16 from 1793793 to 2254875 tid 2 2024-09-20T22:45:12.204 INFO:tasks.rados.rados.0.smithi084.stdout:720: write oid 23 current snap is 71 2024-09-20T22:45:12.204 INFO:tasks.rados.rados.0.smithi084.stdout:720: seq_num 282 ranges {739574=725279,1968628=721093} 2024-09-20T22:45:12.212 INFO:tasks.rados.rados.0.smithi084.stdout:720: writing smithi08477203-23 from 739574 to 1464853 tid 1 2024-09-20T22:45:12.219 INFO:tasks.rados.rados.0.smithi084.stdout:720: writing smithi08477203-23 from 1968628 to 2689721 tid 2 2024-09-20T22:45:12.219 INFO:tasks.rados.rados.0.smithi084.stdout:721: delete oid 19 current snap is 71 2024-09-20T22:45:12.219 INFO:tasks.rados.rados.0.smithi084.stdout:718: expect (ObjNum 273 snap 69 seq_num 273) 2024-09-20T22:45:12.239 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:45:12.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:12.264 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:12.265 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:45:12.340 INFO:tasks.rados.rados.0.smithi084.stdout:719: finishing write tid 1 to smithi08477203-16 2024-09-20T22:45:12.340 INFO:tasks.rados.rados.0.smithi084.stdout:719: finishing write tid 2 to smithi08477203-16 2024-09-20T22:45:12.340 INFO:tasks.rados.rados.0.smithi084.stdout:719: finishing write tid 3 to smithi08477203-16 2024-09-20T22:45:12.340 INFO:tasks.rados.rados.0.smithi084.stdout:719: finishing write tid 4 to smithi08477203-16 2024-09-20T22:45:12.340 INFO:tasks.rados.rados.0.smithi084.stdout:719: oid 16 updating version 0 to 135 2024-09-20T22:45:12.340 INFO:tasks.rados.rados.0.smithi084.stdout:719: oid 16 version 135 is already newer than 134 2024-09-20T22:45:12.340 INFO:tasks.rados.rados.0.smithi084.stdout:719: oid 16 updating version 135 to 136 2024-09-20T22:45:12.340 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 16 v 136 (ObjNum 281 snap 71 seq_num 281) dirty exists 2024-09-20T22:45:12.341 INFO:tasks.rados.rados.0.smithi084.stdout:719: left oid 16 (ObjNum 281 snap 71 seq_num 281) 2024-09-20T22:45:12.341 INFO:tasks.rados.rados.0.smithi084.stdout:718: done (3 left) 2024-09-20T22:45:12.341 INFO:tasks.rados.rados.0.smithi084.stdout:719: done (2 left) 2024-09-20T22:45:12.341 INFO:tasks.rados.rados.0.smithi084.stdout:721: done (1 left) 2024-09-20T22:45:12.341 INFO:tasks.rados.rados.0.smithi084.stdout:722: read oid 18 snap -1 2024-09-20T22:45:12.341 INFO:tasks.rados.rados.0.smithi084.stdout:722: expect (ObjNum 273 snap 69 seq_num 273) 2024-09-20T22:45:12.341 INFO:tasks.rados.rados.0.smithi084.stdout:723: snap_remove snap 70 2024-09-20T22:45:12.341 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:45:12.365 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:45:12.390 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:12.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:12.391 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:45:12.477 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-20T22:45:12.477 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:12 smithi084 ceph-mon[29774]: osdmap e232: 8 total, 8 up, 7 in 2024-09-20T22:45:12.478 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:12 smithi084 ceph-mon[40024]: osdmap e232: 8 total, 8 up, 7 in 2024-09-20T22:45:12.484 INFO:teuthology.orchestra.run.smithi084.stderr:dumped all 2024-09-20T22:45:12.492 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:45:12.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:12 smithi154 ceph-mon[33264]: osdmap e232: 8 total, 8 up, 7 in 2024-09-20T22:45:12.513 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:12.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:12.514 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:45:12.615 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:45:12.636 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:12.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:12.637 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:45:12.732 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.2 dump_blocked_ops 2024-09-20T22:45:12.738 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:45:12.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:12.765 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:12.765 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:45:12.866 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:45:12.887 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:12.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:12.887 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:45:12.988 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:45:13.010 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:13.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:13.011 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:45:13.112 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:45:13.133 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:13.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:13.134 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:45:13.149 INFO:tasks.rados.rados.0.smithi084.stdout:723: done (2 left) 2024-09-20T22:45:13.149 INFO:tasks.rados.rados.0.smithi084.stdout:724: snap_remove snap 66 2024-09-20T22:45:13.235 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:45:13.246 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:45:13.259 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:13.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:13.259 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:45:13.360 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:45:13.364 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:13 smithi084 ceph-mon[29774]: pgmap v250: 178 pgs: 178 active+clean; 114 MiB data, 540 MiB used, 625 GiB / 626 GiB avail; 7.1 MiB/s rd, 886 KiB/s wr, 13 op/s 2024-09-20T22:45:13.364 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:13 smithi084 ceph-mon[29774]: osdmap e233: 8 total, 8 up, 7 in 2024-09-20T22:45:13.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:13 smithi084 ceph-mon[29774]: from='client.15249 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T22:45:13.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:13 smithi084 ceph-mon[29774]: osdmap e234: 8 total, 8 up, 7 in 2024-09-20T22:45:13.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:13 smithi084 ceph-mon[40024]: pgmap v250: 178 pgs: 178 active+clean; 114 MiB data, 540 MiB used, 625 GiB / 626 GiB avail; 7.1 MiB/s rd, 886 KiB/s wr, 13 op/s 2024-09-20T22:45:13.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:13 smithi084 ceph-mon[40024]: osdmap e233: 8 total, 8 up, 7 in 2024-09-20T22:45:13.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:13 smithi084 ceph-mon[40024]: from='client.15249 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T22:45:13.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:13 smithi084 ceph-mon[40024]: osdmap e234: 8 total, 8 up, 7 in 2024-09-20T22:45:13.386 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:13.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:13.387 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:45:13.419 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-20T22:45:13.488 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:45:13.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:13 smithi154 ceph-mon[33264]: pgmap v250: 178 pgs: 178 active+clean; 114 MiB data, 540 MiB used, 625 GiB / 626 GiB avail; 7.1 MiB/s rd, 886 KiB/s wr, 13 op/s 2024-09-20T22:45:13.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:13 smithi154 ceph-mon[33264]: osdmap e233: 8 total, 8 up, 7 in 2024-09-20T22:45:13.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:13 smithi154 ceph-mon[33264]: from='client.15249 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T22:45:13.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:13 smithi154 ceph-mon[33264]: osdmap e234: 8 total, 8 up, 7 in 2024-09-20T22:45:13.508 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:13.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:13.509 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:45:13.610 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:45:13.636 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:13.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:13.637 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:45:13.715 INFO:teuthology.orchestra.run.smithi084.stdout:{"pg_ready":true,"pg_map":{"version":251,"stamp":"2024-09-20T22:45:12.186020+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":119366254,"num_objects":251,"num_object_clones":15,"num_object_copies":753,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":251,"num_whiteouts":3,"num_read":1978,"num_read_kb":1089652,"num_write":1172,"num_write_kb":292150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":5838014,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5674,"ondisk_log_size":5674,"up":534,"acting":534,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":534,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":552612,"kb_used_data":300972,"kb_used_omap":10,"kb_used_meta":251445,"kb_avail":655663452,"statfs":{"total":671965249536,"available":671399374848,"internally_reserved":0,"allocated":308195328,"data_stored":345087701,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11157,"internal_metadata":257479787},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":5242986,"num_objects":7,"num_object_clones":6,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":2,"num_read":36,"num_read_kb":29109,"num_write":18,"num_write_kb":3546,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.043985"},"pg_stats":[{"pgid":"3.1f","version":"86'2","reported_seq":359,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.146680+0000","last_change":"2024-09-20T22:44:23.787416+0000","last_active":"2024-09-20T22:45:08.146680+0000","last_peered":"2024-09-20T22:45:08.146680+0000","last_clean":"2024-09-20T22:45:08.146680+0000","last_became_active":"2024-09-20T22:41:38.159085+0000","last_became_peered":"2024-09-20T22:41:38.159085+0000","last_unstale":"2024-09-20T22:45:08.146680+0000","last_undegraded":"2024-09-20T22:45:08.146680+0000","last_fullsized":"2024-09-20T22:45:08.146680+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'2","last_scrub_stamp":"2024-09-20T22:44:23.787342+0000","last_deep_scrub":"86'2","last_deep_scrub_stamp":"2024-09-20T22:42:39.664421+0000","last_clean_scrub_stamp":"2024-09-20T22:44:23.787342+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:45.057201+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"74'9","reported_seq":207,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.159284+0000","last_change":"2024-09-20T22:44:14.456654+0000","last_active":"2024-09-20T22:45:08.159284+0000","last_peered":"2024-09-20T22:45:08.159284+0000","last_clean":"2024-09-20T22:45:08.159284+0000","last_became_active":"2024-09-20T22:43:57.428786+0000","last_became_peered":"2024-09-20T22:43:57.428786+0000","last_unstale":"2024-09-20T22:45:08.159284+0000","last_undegraded":"2024-09-20T22:45:08.159284+0000","last_fullsized":"2024-09-20T22:45:08.159284+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-09-20T22:44:14.456573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:44:14.456573+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:30.414294+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":336,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.198644+0000","last_change":"2024-09-20T22:44:18.040704+0000","last_active":"2024-09-20T22:45:08.198644+0000","last_peered":"2024-09-20T22:45:08.198644+0000","last_clean":"2024-09-20T22:45:08.198644+0000","last_became_active":"2024-09-20T22:41:42.182024+0000","last_became_peered":"2024-09-20T22:41:42.182024+0000","last_unstale":"2024-09-20T22:45:08.198644+0000","last_undegraded":"2024-09-20T22:45:08.198644+0000","last_fullsized":"2024-09-20T22:45:08.198644+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:44:18.040631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:44:18.040631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:23.293547+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":163,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.151161+0000","last_change":"2024-09-20T22:44:40.746186+0000","last_active":"2024-09-20T22:45:08.151161+0000","last_peered":"2024-09-20T22:45:08.151161+0000","last_clean":"2024-09-20T22:45:08.151161+0000","last_became_active":"2024-09-20T22:43:57.429983+0000","last_became_peered":"2024-09-20T22:43:57.429983+0000","last_unstale":"2024-09-20T22:45:08.151161+0000","last_undegraded":"2024-09-20T22:45:08.151161+0000","last_fullsized":"2024-09-20T22:45:08.151161+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:44:40.746114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:44:40.746114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:40.841196+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":332,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.146686+0000","last_change":"2024-09-20T22:43:07.476040+0000","last_active":"2024-09-20T22:45:08.146686+0000","last_peered":"2024-09-20T22:45:08.146686+0000","last_clean":"2024-09-20T22:45:08.146686+0000","last_became_active":"2024-09-20T22:41:44.832099+0000","last_became_peered":"2024-09-20T22:41:44.832099+0000","last_unstale":"2024-09-20T22:45:08.146686+0000","last_undegraded":"2024-09-20T22:45:08.146686+0000","last_fullsized":"2024-09-20T22:45:08.146686+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:07.475964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:43:07.475964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:09.108000+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":344,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.147842+0000","last_change":"2024-09-20T22:44:58.994410+0000","last_active":"2024-09-20T22:45:08.147842+0000","last_peered":"2024-09-20T22:45:08.147842+0000","last_clean":"2024-09-20T22:45:08.147842+0000","last_became_active":"2024-09-20T22:41:38.157298+0000","last_became_peered":"2024-09-20T22:41:38.157298+0000","last_unstale":"2024-09-20T22:45:08.147842+0000","last_undegraded":"2024-09-20T22:45:08.147842+0000","last_fullsized":"2024-09-20T22:45:08.147842+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:44:58.994340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:44:58.994340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:46:26.252244+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"74'15","reported_seq":365,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.143853+0000","last_change":"2024-09-20T22:44:41.108291+0000","last_active":"2024-09-20T22:45:08.143853+0000","last_peered":"2024-09-20T22:45:08.143853+0000","last_clean":"2024-09-20T22:45:08.143853+0000","last_became_active":"2024-09-20T22:41:40.163242+0000","last_became_peered":"2024-09-20T22:41:40.163242+0000","last_unstale":"2024-09-20T22:45:08.143853+0000","last_undegraded":"2024-09-20T22:45:08.143853+0000","last_fullsized":"2024-09-20T22:45:08.143853+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-09-20T22:44:41.108205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:44:41.108205+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:50.242514+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":162,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.159443+0000","last_change":"2024-09-20T22:44:57.608536+0000","last_active":"2024-09-20T22:45:08.159443+0000","last_peered":"2024-09-20T22:45:08.159443+0000","last_clean":"2024-09-20T22:45:08.159443+0000","last_became_active":"2024-09-20T22:43:57.423685+0000","last_became_peered":"2024-09-20T22:43:57.423685+0000","last_unstale":"2024-09-20T22:45:08.159443+0000","last_undegraded":"2024-09-20T22:45:08.159443+0000","last_fullsized":"2024-09-20T22:45:08.159443+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:44:57.608471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:44:57.608471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:46:05.218588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.19","version":"224'141","reported_seq":591,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.159760+0000","last_change":"2024-09-20T22:45:08.160591+0000","last_active":"2024-09-20T22:45:08.159760+0000","last_peered":"2024-09-20T22:45:08.159760+0000","last_clean":"2024-09-20T22:45:08.159760+0000","last_became_active":"2024-09-20T22:43:39.130950+0000","last_became_peered":"2024-09-20T22:43:39.130950+0000","last_unstale":"2024-09-20T22:45:08.159760+0000","last_undegraded":"2024-09-20T22:45:08.159760+0000","last_fullsized":"2024-09-20T22:45:08.159760+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":5,"last_scrub":"143'50","last_scrub_stamp":"2024-09-20T22:43:45.392735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:43:45.392735+0000","objects_scrubbed":3,"log_size":141,"log_dups_size":0,"ondisk_log_size":141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00105148,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":35,"num_read_kb":29153,"num_write":35,"num_write_kb":14433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"39","length":"1"},{"start":"46","length":"1"}]},{"pgid":"3.1d","version":"0'0","reported_seq":350,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.151935+0000","last_change":"2024-09-20T22:43:57.430206+0000","last_active":"2024-09-20T22:45:08.151935+0000","last_peered":"2024-09-20T22:45:08.151935+0000","last_clean":"2024-09-20T22:45:08.151935+0000","last_became_active":"2024-09-20T22:43:57.429854+0000","last_became_peered":"2024-09-20T22:43:57.429854+0000","last_unstale":"2024-09-20T22:45:08.151935+0000","last_undegraded":"2024-09-20T22:45:08.151935+0000","last_fullsized":"2024-09-20T22:45:08.151935+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:34.681993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:43:34.681993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"74'9","reported_seq":199,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.146184+0000","last_change":"2024-09-20T22:43:57.488635+0000","last_active":"2024-09-20T22:45:08.146184+0000","last_peered":"2024-09-20T22:45:08.146184+0000","last_clean":"2024-09-20T22:45:08.146184+0000","last_became_active":"2024-09-20T22:43:57.445313+0000","last_became_peered":"2024-09-20T22:43:57.445313+0000","last_unstale":"2024-09-20T22:45:08.146184+0000","last_undegraded":"2024-09-20T22:45:08.146184+0000","last_fullsized":"2024-09-20T22:45:08.146184+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-09-20T22:43:30.938345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:43:30.938345+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:09.108254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":336,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.151186+0000","last_change":"2024-09-20T22:44:15.667340+0000","last_active":"2024-09-20T22:45:08.151186+0000","last_peered":"2024-09-20T22:45:08.151186+0000","last_clean":"2024-09-20T22:45:08.151186+0000","last_became_active":"2024-09-20T22:41:42.185538+0000","last_became_peered":"2024-09-20T22:41:42.185538+0000","last_unstale":"2024-09-20T22:45:08.151186+0000","last_undegraded":"2024-09-20T22:45:08.151186+0000","last_fullsized":"2024-09-20T22:45:08.151186+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:44:15.667266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:44:15.667266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:20.581167+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":332,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.144370+0000","last_change":"2024-09-20T22:44:20.811801+0000","last_active":"2024-09-20T22:45:08.144370+0000","last_peered":"2024-09-20T22:45:08.144370+0000","last_clean":"2024-09-20T22:45:08.144370+0000","last_became_active":"2024-09-20T22:41:44.197113+0000","last_became_peered":"2024-09-20T22:41:44.197113+0000","last_unstale":"2024-09-20T22:45:08.144370+0000","last_undegraded":"2024-09-20T22:45:08.144370+0000","last_fullsized":"2024-09-20T22:45:08.144370+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:44:20.811723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:44:20.811723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:38.989363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.18","version":"204'92","reported_seq":543,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.144671+0000","last_change":"2024-09-20T22:45:08.144944+0000","last_active":"2024-09-20T22:45:08.144671+0000","last_peered":"2024-09-20T22:45:08.144671+0000","last_clean":"2024-09-20T22:45:08.144671+0000","last_became_active":"2024-09-20T22:43:39.132159+0000","last_became_peered":"2024-09-20T22:43:39.132159+0000","last_unstale":"2024-09-20T22:45:08.144671+0000","last_undegraded":"2024-09-20T22:45:08.144671+0000","last_fullsized":"2024-09-20T22:45:08.144671+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":5,"last_scrub":"143'47","last_scrub_stamp":"2024-09-20T22:43:45.760674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:43:45.760674+0000","objects_scrubbed":2,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:12.797140+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00091684700000000004,"stat_sum":{"num_bytes":5361400,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":37,"num_read_kb":35570,"num_write":26,"num_write_kb":13129,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"39","length":"1"},{"start":"46","length":"1"}]},{"pgid":"3.1c","version":"0'0","reported_seq":350,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.151597+0000","last_change":"2024-09-20T22:43:57.430381+0000","last_active":"2024-09-20T22:45:08.151597+0000","last_peered":"2024-09-20T22:45:08.151597+0000","last_clean":"2024-09-20T22:45:08.151597+0000","last_became_active":"2024-09-20T22:43:57.430028+0000","last_became_peered":"2024-09-20T22:43:57.430028+0000","last_unstale":"2024-09-20T22:45:08.151597+0000","last_undegraded":"2024-09-20T22:45:08.151597+0000","last_fullsized":"2024-09-20T22:45:08.151597+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:00.713878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:43:00.713878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"181'7","reported_seq":199,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.147382+0000","last_change":"2024-09-20T22:44:20.259677+0000","last_active":"2024-09-20T22:45:08.147382+0000","last_peered":"2024-09-20T22:45:08.147382+0000","last_clean":"2024-09-20T22:45:08.147382+0000","last_became_active":"2024-09-20T22:43:57.441058+0000","last_became_peered":"2024-09-20T22:43:57.441058+0000","last_unstale":"2024-09-20T22:45:08.147382+0000","last_undegraded":"2024-09-20T22:45:08.147382+0000","last_fullsized":"2024-09-20T22:45:08.147382+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"181'7","last_scrub_stamp":"2024-09-20T22:44:20.259598+0000","last_deep_scrub":"181'7","last_deep_scrub_stamp":"2024-09-20T22:44:20.259598+0000","last_clean_scrub_stamp":"2024-09-20T22:44:20.259598+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:28.101261+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":6,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":353,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.156290+0000","last_change":"2024-09-20T22:44:54.057357+0000","last_active":"2024-09-20T22:45:08.156290+0000","last_peered":"2024-09-20T22:45:08.156290+0000","last_clean":"2024-09-20T22:45:08.156290+0000","last_became_active":"2024-09-20T22:43:57.419419+0000","last_became_peered":"2024-09-20T22:43:57.419419+0000","last_unstale":"2024-09-20T22:45:08.156290+0000","last_undegraded":"2024-09-20T22:45:08.156290+0000","last_fullsized":"2024-09-20T22:45:08.156290+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:44:54.057283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:44:54.057283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:46:09.354039+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":332,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.151046+0000","last_change":"2024-09-20T22:43:32.712751+0000","last_active":"2024-09-20T22:45:08.151046+0000","last_peered":"2024-09-20T22:45:08.151046+0000","last_clean":"2024-09-20T22:45:08.151046+0000","last_became_active":"2024-09-20T22:41:44.197927+0000","last_became_peered":"2024-09-20T22:41:44.197927+0000","last_unstale":"2024-09-20T22:45:08.151046+0000","last_undegraded":"2024-09-20T22:45:08.151046+0000","last_fullsized":"2024-09-20T22:45:08.151046+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:32.712676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:43:32.712676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":163,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.161374+0000","last_change":"2024-09-20T22:44:41.574836+0000","last_active":"2024-09-20T22:45:08.161374+0000","last_peered":"2024-09-20T22:45:08.161374+0000","last_clean":"2024-09-20T22:45:08.161374+0000","last_became_active":"2024-09-20T22:43:57.427971+0000","last_became_peered":"2024-09-20T22:43:57.427971+0000","last_unstale":"2024-09-20T22:45:08.161374+0000","last_undegraded":"2024-09-20T22:45:08.161374+0000","last_fullsized":"2024-09-20T22:45:08.161374+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:44:41.574713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:44:41.574713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:46:10.688311+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":350,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.145272+0000","last_change":"2024-09-20T22:44:18.785085+0000","last_active":"2024-09-20T22:45:08.145272+0000","last_peered":"2024-09-20T22:45:08.145272+0000","last_clean":"2024-09-20T22:45:08.145272+0000","last_became_active":"2024-09-20T22:43:57.432650+0000","last_became_peered":"2024-09-20T22:43:57.432650+0000","last_unstale":"2024-09-20T22:45:08.145272+0000","last_undegraded":"2024-09-20T22:45:08.145272+0000","last_fullsized":"2024-09-20T22:45:08.145272+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:44:18.784990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:44:18.784990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:47.957020+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"113'22","reported_seq":379,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.155552+0000","last_change":"2024-09-20T22:44:55.423258+0000","last_active":"2024-09-20T22:45:08.155552+0000","last_peered":"2024-09-20T22:45:08.155552+0000","last_clean":"2024-09-20T22:45:08.155552+0000","last_became_active":"2024-09-20T22:41:40.170131+0000","last_became_peered":"2024-09-20T22:41:40.170131+0000","last_unstale":"2024-09-20T22:45:08.155552+0000","last_undegraded":"2024-09-20T22:45:08.155552+0000","last_fullsized":"2024-09-20T22:45:08.155552+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'22","last_scrub_stamp":"2024-09-20T22:44:55.423175+0000","last_deep_scrub":"113'22","last_deep_scrub_stamp":"2024-09-20T22:43:12.191878+0000","last_clean_scrub_stamp":"2024-09-20T22:44:55.423175+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:46:10.184178+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":342,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.199086+0000","last_change":"2024-09-20T22:44:22.085304+0000","last_active":"2024-09-20T22:45:08.199086+0000","last_peered":"2024-09-20T22:45:08.199086+0000","last_clean":"2024-09-20T22:45:08.199086+0000","last_became_active":"2024-09-20T22:43:57.435288+0000","last_became_peered":"2024-09-20T22:43:57.435288+0000","last_unstale":"2024-09-20T22:45:08.199086+0000","last_undegraded":"2024-09-20T22:45:08.199086+0000","last_fullsized":"2024-09-20T22:45:08.199086+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:44:22.085224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:44:22.085224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:28.981956+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":324,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.148307+0000","last_change":"2024-09-20T22:41:44.832468+0000","last_active":"2024-09-20T22:45:08.148307+0000","last_peered":"2024-09-20T22:45:08.148307+0000","last_clean":"2024-09-20T22:45:08.148307+0000","last_became_active":"2024-09-20T22:41:44.832131+0000","last_became_peered":"2024-09-20T22:41:44.832131+0000","last_unstale":"2024-09-20T22:45:08.148307+0000","last_undegraded":"2024-09-20T22:45:08.148307+0000","last_fullsized":"2024-09-20T22:45:08.148307+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:09.107876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":162,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.198656+0000","last_change":"2024-09-20T22:44:12.001213+0000","last_active":"2024-09-20T22:45:08.198656+0000","last_peered":"2024-09-20T22:45:08.198656+0000","last_clean":"2024-09-20T22:45:08.198656+0000","last_became_active":"2024-09-20T22:43:57.435636+0000","last_became_peered":"2024-09-20T22:43:57.435636+0000","last_unstale":"2024-09-20T22:45:08.198656+0000","last_undegraded":"2024-09-20T22:45:08.198656+0000","last_fullsized":"2024-09-20T22:45:08.198656+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:44:12.001145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:44:12.001145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:14.872769+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"74'12","reported_seq":399,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.146428+0000","last_change":"2024-09-20T22:44:42.141018+0000","last_active":"2024-09-20T22:45:08.146428+0000","last_peered":"2024-09-20T22:45:08.146428+0000","last_clean":"2024-09-20T22:45:08.146428+0000","last_became_active":"2024-09-20T22:43:57.438162+0000","last_became_peered":"2024-09-20T22:43:57.438162+0000","last_unstale":"2024-09-20T22:45:08.146428+0000","last_undegraded":"2024-09-20T22:45:08.146428+0000","last_fullsized":"2024-09-20T22:45:08.146428+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-09-20T22:44:42.140915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:44:42.140915+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:57.947345+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":162,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.146681+0000","last_change":"2024-09-20T22:44:14.235908+0000","last_active":"2024-09-20T22:45:08.146681+0000","last_peered":"2024-09-20T22:45:08.146681+0000","last_clean":"2024-09-20T22:45:08.146681+0000","last_became_active":"2024-09-20T22:43:57.429179+0000","last_became_peered":"2024-09-20T22:43:57.429179+0000","last_unstale":"2024-09-20T22:45:08.146681+0000","last_undegraded":"2024-09-20T22:45:08.146681+0000","last_fullsized":"2024-09-20T22:45:08.146681+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:44:14.235804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:44:14.235804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:26.950979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"154'2","reported_seq":334,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.157137+0000","last_change":"2024-09-20T22:43:54.134185+0000","last_active":"2024-09-20T22:45:08.157137+0000","last_peered":"2024-09-20T22:45:08.157137+0000","last_clean":"2024-09-20T22:45:08.157137+0000","last_became_active":"2024-09-20T22:41:44.199822+0000","last_became_peered":"2024-09-20T22:41:44.199822+0000","last_unstale":"2024-09-20T22:45:08.157137+0000","last_undegraded":"2024-09-20T22:45:08.157137+0000","last_fullsized":"2024-09-20T22:45:08.157137+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"154'2","last_scrub_stamp":"2024-09-20T22:43:54.134066+0000","last_deep_scrub":"154'2","last_deep_scrub_stamp":"2024-09-20T22:43:54.134066+0000","last_clean_scrub_stamp":"2024-09-20T22:43:54.134066+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:16.124782+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"67'1","reported_seq":355,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.198854+0000","last_change":"2024-09-20T22:43:57.462130+0000","last_active":"2024-09-20T22:45:08.198854+0000","last_peered":"2024-09-20T22:45:08.198854+0000","last_clean":"2024-09-20T22:45:08.198854+0000","last_became_active":"2024-09-20T22:43:57.443155+0000","last_became_peered":"2024-09-20T22:43:57.443155+0000","last_unstale":"2024-09-20T22:45:08.198854+0000","last_undegraded":"2024-09-20T22:45:08.198854+0000","last_fullsized":"2024-09-20T22:45:08.198854+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-09-20T22:43:35.266242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:43:35.266242+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:12.054319+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"151'15","reported_seq":360,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.145537+0000","last_change":"2024-09-20T22:43:50.715845+0000","last_active":"2024-09-20T22:45:08.145537+0000","last_peered":"2024-09-20T22:45:08.145537+0000","last_clean":"2024-09-20T22:45:08.145537+0000","last_became_active":"2024-09-20T22:41:40.252001+0000","last_became_peered":"2024-09-20T22:41:40.252001+0000","last_unstale":"2024-09-20T22:45:08.145537+0000","last_undegraded":"2024-09-20T22:45:08.145537+0000","last_fullsized":"2024-09-20T22:45:08.145537+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"151'15","last_scrub_stamp":"2024-09-20T22:43:50.715761+0000","last_deep_scrub":"151'15","last_deep_scrub_stamp":"2024-09-20T22:43:50.715761+0000","last_clean_scrub_stamp":"2024-09-20T22:43:50.715761+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:08.787709+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"159'9","reported_seq":403,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.161574+0000","last_change":"2024-09-20T22:44:15.499231+0000","last_active":"2024-09-20T22:45:08.161574+0000","last_peered":"2024-09-20T22:45:08.161574+0000","last_clean":"2024-09-20T22:45:08.161574+0000","last_became_active":"2024-09-20T22:43:57.424879+0000","last_became_peered":"2024-09-20T22:43:57.424879+0000","last_unstale":"2024-09-20T22:45:08.161574+0000","last_undegraded":"2024-09-20T22:45:08.161574+0000","last_fullsized":"2024-09-20T22:45:08.161574+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"159'9","last_scrub_stamp":"2024-09-20T22:44:15.499150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:44:15.499150+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:39.142015+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.c","version":"229'318","reported_seq":1042,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.206419+0000","last_change":"2024-09-20T22:45:08.160804+0000","last_active":"2024-09-20T22:45:08.206419+0000","last_peered":"2024-09-20T22:45:08.206419+0000","last_clean":"2024-09-20T22:45:08.206419+0000","last_became_active":"2024-09-20T22:42:52.404027+0000","last_became_peered":"2024-09-20T22:42:52.404027+0000","last_unstale":"2024-09-20T22:45:08.206419+0000","last_undegraded":"2024-09-20T22:45:08.206419+0000","last_fullsized":"2024-09-20T22:45:08.206419+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"194'228","last_scrub_stamp":"2024-09-20T22:44:34.610972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:44:34.610972+0000","objects_scrubbed":5,"log_size":318,"log_dups_size":0,"ondisk_log_size":318,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:46:01.681065+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00048801600000000002,"stat_sum":{"num_bytes":13915825,"num_objects":8,"num_object_clones":3,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":1,"num_read":243,"num_read_kb":240008,"num_write":163,"num_write_kb":65627,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"39","length":"1"},{"start":"46","length":"1"}]},{"pgid":"1.a","version":"77'128","reported_seq":581,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.150886+0000","last_change":"2024-09-20T22:43:05.763879+0000","last_active":"2024-09-20T22:45:08.150886+0000","last_peered":"2024-09-20T22:45:08.150886+0000","last_clean":"2024-09-20T22:45:08.150886+0000","last_became_active":"2024-09-20T22:42:52.407791+0000","last_became_peered":"2024-09-20T22:42:52.407791+0000","last_unstale":"2024-09-20T22:45:08.150886+0000","last_undegraded":"2024-09-20T22:45:08.150886+0000","last_fullsized":"2024-09-20T22:45:08.150886+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:43:05.763796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:43:05.763796+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"74'15","reported_seq":395,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.199410+0000","last_change":"2024-09-20T22:44:55.972422+0000","last_active":"2024-09-20T22:45:08.199410+0000","last_peered":"2024-09-20T22:45:08.199410+0000","last_clean":"2024-09-20T22:45:08.199410+0000","last_became_active":"2024-09-20T22:43:57.441355+0000","last_became_peered":"2024-09-20T22:43:57.441355+0000","last_unstale":"2024-09-20T22:45:08.199410+0000","last_undegraded":"2024-09-20T22:45:08.199410+0000","last_fullsized":"2024-09-20T22:45:08.199410+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-09-20T22:44:55.972333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:44:55.972333+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:46:20.894884+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":336,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.144130+0000","last_change":"2024-09-20T22:41:38.157043+0000","last_active":"2024-09-20T22:45:08.144130+0000","last_peered":"2024-09-20T22:45:08.144130+0000","last_clean":"2024-09-20T22:45:08.144130+0000","last_became_active":"2024-09-20T22:41:38.156749+0000","last_became_peered":"2024-09-20T22:41:38.156749+0000","last_unstale":"2024-09-20T22:45:08.144130+0000","last_undegraded":"2024-09-20T22:45:08.144130+0000","last_fullsized":"2024-09-20T22:45:08.144130+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:09.107781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"159'9","reported_seq":206,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.151912+0000","last_change":"2024-09-20T22:44:55.604951+0000","last_active":"2024-09-20T22:45:08.151912+0000","last_peered":"2024-09-20T22:45:08.151912+0000","last_clean":"2024-09-20T22:45:08.151912+0000","last_became_active":"2024-09-20T22:43:57.425367+0000","last_became_peered":"2024-09-20T22:43:57.425367+0000","last_unstale":"2024-09-20T22:45:08.151912+0000","last_undegraded":"2024-09-20T22:45:08.151912+0000","last_fullsized":"2024-09-20T22:45:08.151912+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"159'9","last_scrub_stamp":"2024-09-20T22:44:55.604845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:44:55.604845+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:56.211983+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":332,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.151595+0000","last_change":"2024-09-20T22:43:53.790857+0000","last_active":"2024-09-20T22:45:08.151595+0000","last_peered":"2024-09-20T22:45:08.151595+0000","last_clean":"2024-09-20T22:45:08.151595+0000","last_became_active":"2024-09-20T22:41:44.198347+0000","last_became_peered":"2024-09-20T22:41:44.198347+0000","last_unstale":"2024-09-20T22:45:08.151595+0000","last_undegraded":"2024-09-20T22:45:08.151595+0000","last_fullsized":"2024-09-20T22:45:08.151595+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:53.790756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:43:53.790756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:14.003231+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3","version":"135'8","reported_seq":692,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.147487+0000","last_change":"2024-09-20T22:45:08.152820+0000","last_active":"2024-09-20T22:45:08.147487+0000","last_peered":"2024-09-20T22:45:08.147487+0000","last_clean":"2024-09-20T22:45:08.147487+0000","last_became_active":"2024-09-20T22:43:57.439135+0000","last_became_peered":"2024-09-20T22:43:57.439135+0000","last_unstale":"2024-09-20T22:45:08.147487+0000","last_undegraded":"2024-09-20T22:45:08.147487+0000","last_fullsized":"2024-09-20T22:45:08.147487+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'8","last_scrub_stamp":"2024-09-20T22:43:47.153362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:43:47.153362+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:14.838244+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0058667420000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"39","length":"1"},{"start":"46","length":"1"}]},{"pgid":"1.5","version":"77'128","reported_seq":453,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.155802+0000","last_change":"2024-09-20T22:44:10.229558+0000","last_active":"2024-09-20T22:45:08.155802+0000","last_peered":"2024-09-20T22:45:08.155802+0000","last_clean":"2024-09-20T22:45:08.155802+0000","last_became_active":"2024-09-20T22:44:00.145455+0000","last_became_peered":"2024-09-20T22:44:00.145455+0000","last_unstale":"2024-09-20T22:45:08.155802+0000","last_undegraded":"2024-09-20T22:45:08.155802+0000","last_fullsized":"2024-09-20T22:45:08.155802+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:44:10.229483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:44:10.229483+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:20.129048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,5],"acting":[2,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"74'18","reported_seq":372,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.143947+0000","last_change":"2024-09-20T22:44:23.220792+0000","last_active":"2024-09-20T22:45:08.143947+0000","last_peered":"2024-09-20T22:45:08.143947+0000","last_clean":"2024-09-20T22:45:08.143947+0000","last_became_active":"2024-09-20T22:41:40.251868+0000","last_became_peered":"2024-09-20T22:41:40.251868+0000","last_unstale":"2024-09-20T22:45:08.143947+0000","last_undegraded":"2024-09-20T22:45:08.143947+0000","last_fullsized":"2024-09-20T22:45:08.143947+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'18","last_scrub_stamp":"2024-09-20T22:44:23.220709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:44:23.220709+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:43.393391+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":344,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.148624+0000","last_change":"2024-09-20T22:43:51.170223+0000","last_active":"2024-09-20T22:45:08.148624+0000","last_peered":"2024-09-20T22:45:08.148624+0000","last_clean":"2024-09-20T22:45:08.148624+0000","last_became_active":"2024-09-20T22:41:38.161335+0000","last_became_peered":"2024-09-20T22:41:38.161335+0000","last_unstale":"2024-09-20T22:45:08.148624+0000","last_undegraded":"2024-09-20T22:45:08.148624+0000","last_fullsized":"2024-09-20T22:45:08.148624+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:51.170155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:43:51.170155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:12.305573+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":154,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.143679+0000","last_change":"2024-09-20T22:43:57.427579+0000","last_active":"2024-09-20T22:45:08.143679+0000","last_peered":"2024-09-20T22:45:08.143679+0000","last_clean":"2024-09-20T22:45:08.143679+0000","last_became_active":"2024-09-20T22:43:57.427287+0000","last_became_peered":"2024-09-20T22:43:57.427287+0000","last_unstale":"2024-09-20T22:45:08.143679+0000","last_undegraded":"2024-09-20T22:45:08.143679+0000","last_fullsized":"2024-09-20T22:45:08.143679+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:07.947945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:07.947945+0000","last_clean_scrub_stamp":"2024-09-20T22:43:07.947945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:09.108144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":155,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.146144+0000","last_change":"2024-09-20T22:43:57.435367+0000","last_active":"2024-09-20T22:45:08.146144+0000","last_peered":"2024-09-20T22:45:08.146144+0000","last_clean":"2024-09-20T22:45:08.146144+0000","last_became_active":"2024-09-20T22:43:57.435057+0000","last_became_peered":"2024-09-20T22:43:57.435057+0000","last_unstale":"2024-09-20T22:45:08.146144+0000","last_undegraded":"2024-09-20T22:45:08.146144+0000","last_fullsized":"2024-09-20T22:45:08.146144+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:53.036601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:53.036601+0000","last_clean_scrub_stamp":"2024-09-20T22:43:53.036601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:09.108323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2","version":"224'95","reported_seq":342,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.151355+0000","last_change":"2024-09-20T22:45:08.151707+0000","last_active":"2024-09-20T22:45:08.151355+0000","last_peered":"2024-09-20T22:45:08.151355+0000","last_clean":"2024-09-20T22:45:08.151355+0000","last_became_active":"2024-09-20T22:44:09.177625+0000","last_became_peered":"2024-09-20T22:44:09.177625+0000","last_unstale":"2024-09-20T22:45:08.151355+0000","last_undegraded":"2024-09-20T22:45:08.151355+0000","last_fullsized":"2024-09-20T22:45:08.151355+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"138'78","last_scrub_stamp":"2024-09-20T22:44:09.667741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:44:09.667741+0000","objects_scrubbed":1,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:30.941339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000670322,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":21,"num_read_kb":31713,"num_write":6,"num_write_kb":1390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"39","length":"1"},{"start":"46","length":"1"}]},{"pgid":"1.4","version":"77'128","reported_seq":594,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.146357+0000","last_change":"2024-09-20T22:43:26.151507+0000","last_active":"2024-09-20T22:45:08.146357+0000","last_peered":"2024-09-20T22:45:08.146357+0000","last_clean":"2024-09-20T22:45:08.146357+0000","last_became_active":"2024-09-20T22:43:19.125641+0000","last_became_peered":"2024-09-20T22:43:19.125641+0000","last_unstale":"2024-09-20T22:45:08.146357+0000","last_undegraded":"2024-09-20T22:45:08.146357+0000","last_fullsized":"2024-09-20T22:45:08.146357+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:43:26.151406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:43:26.151406+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:09.108117+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"74'14","reported_seq":218,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.151058+0000","last_change":"2024-09-20T22:44:17.668446+0000","last_active":"2024-09-20T22:45:08.151058+0000","last_peered":"2024-09-20T22:45:08.151058+0000","last_clean":"2024-09-20T22:45:08.151058+0000","last_became_active":"2024-09-20T22:43:57.422847+0000","last_became_peered":"2024-09-20T22:43:57.422847+0000","last_unstale":"2024-09-20T22:45:08.151058+0000","last_undegraded":"2024-09-20T22:45:08.151058+0000","last_fullsized":"2024-09-20T22:45:08.151058+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'14","last_scrub_stamp":"2024-09-20T22:44:17.668364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:44:17.668364+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:34.632610+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.6","version":"67'1","reported_seq":355,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.147173+0000","last_change":"2024-09-20T22:43:57.457010+0000","last_active":"2024-09-20T22:45:08.147173+0000","last_peered":"2024-09-20T22:45:08.147173+0000","last_clean":"2024-09-20T22:45:08.147173+0000","last_became_active":"2024-09-20T22:43:57.431232+0000","last_became_peered":"2024-09-20T22:43:57.431232+0000","last_unstale":"2024-09-20T22:45:08.147173+0000","last_undegraded":"2024-09-20T22:45:08.147173+0000","last_fullsized":"2024-09-20T22:45:08.147173+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-09-20T22:43:29.694945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:43:29.694945+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:08.787640+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"159'9","reported_seq":389,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.143684+0000","last_change":"2024-09-20T22:43:57.481621+0000","last_active":"2024-09-20T22:45:08.143684+0000","last_peered":"2024-09-20T22:45:08.143684+0000","last_clean":"2024-09-20T22:45:08.143684+0000","last_became_active":"2024-09-20T22:43:57.442440+0000","last_became_peered":"2024-09-20T22:43:57.442440+0000","last_unstale":"2024-09-20T22:45:08.143684+0000","last_undegraded":"2024-09-20T22:45:08.143684+0000","last_fullsized":"2024-09-20T22:45:08.143684+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:41:41.163059+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:09.107848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":332,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.156415+0000","last_change":"2024-09-20T22:44:19.972826+0000","last_active":"2024-09-20T22:45:08.156415+0000","last_peered":"2024-09-20T22:45:08.156415+0000","last_clean":"2024-09-20T22:45:08.156415+0000","last_became_active":"2024-09-20T22:41:44.831901+0000","last_became_peered":"2024-09-20T22:41:44.831901+0000","last_unstale":"2024-09-20T22:45:08.156415+0000","last_undegraded":"2024-09-20T22:45:08.156415+0000","last_fullsized":"2024-09-20T22:45:08.156415+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:44:19.972742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:44:19.972742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:48.587176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1","version":"200'99","reported_seq":760,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.146956+0000","last_change":"2024-09-20T22:45:08.147242+0000","last_active":"2024-09-20T22:45:08.146956+0000","last_peered":"2024-09-20T22:45:08.146956+0000","last_clean":"2024-09-20T22:45:08.146956+0000","last_became_active":"2024-09-20T22:43:57.440095+0000","last_became_peered":"2024-09-20T22:43:57.440095+0000","last_unstale":"2024-09-20T22:45:08.146956+0000","last_undegraded":"2024-09-20T22:45:08.146956+0000","last_fullsized":"2024-09-20T22:45:08.146956+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"170'72","last_scrub_stamp":"2024-09-20T22:44:12.228784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:44:12.228784+0000","objects_scrubbed":3,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:18.479241+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0037071349999999999,"stat_sum":{"num_bytes":7579957,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":29,"num_read_kb":44401,"num_write":11,"num_write_kb":13746,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"39","length":"1"},{"start":"46","length":"1"}]},{"pgid":"1.7","version":"77'128","reported_seq":601,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.161358+0000","last_change":"2024-09-20T22:43:26.566566+0000","last_active":"2024-09-20T22:45:08.161358+0000","last_peered":"2024-09-20T22:45:08.161358+0000","last_clean":"2024-09-20T22:45:08.161358+0000","last_became_active":"2024-09-20T22:43:19.129015+0000","last_became_peered":"2024-09-20T22:43:19.129015+0000","last_unstale":"2024-09-20T22:45:08.161358+0000","last_undegraded":"2024-09-20T22:45:08.161358+0000","last_fullsized":"2024-09-20T22:45:08.161358+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:43:26.566436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:43:26.566436+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:10.682109+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"74'10","reported_seq":381,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.198530+0000","last_change":"2024-09-20T22:44:59.049691+0000","last_active":"2024-09-20T22:45:08.198530+0000","last_peered":"2024-09-20T22:45:08.198530+0000","last_clean":"2024-09-20T22:45:08.198530+0000","last_became_active":"2024-09-20T22:43:57.435599+0000","last_became_peered":"2024-09-20T22:43:57.435599+0000","last_unstale":"2024-09-20T22:45:08.198530+0000","last_undegraded":"2024-09-20T22:45:08.198530+0000","last_fullsized":"2024-09-20T22:45:08.198530+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-09-20T22:44:59.049607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:44:59.049607+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:46:05.780194+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":344,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.152043+0000","last_change":"2024-09-20T22:42:39.595884+0000","last_active":"2024-09-20T22:45:08.152043+0000","last_peered":"2024-09-20T22:45:08.152043+0000","last_clean":"2024-09-20T22:45:08.152043+0000","last_became_active":"2024-09-20T22:41:38.155629+0000","last_became_peered":"2024-09-20T22:41:38.155629+0000","last_unstale":"2024-09-20T22:45:08.152043+0000","last_undegraded":"2024-09-20T22:45:08.152043+0000","last_fullsized":"2024-09-20T22:45:08.152043+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:42:39.595799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:42:39.595799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"170'9","reported_seq":146,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.146214+0000","last_change":"2024-09-20T22:44:09.175786+0000","last_active":"2024-09-20T22:45:08.146214+0000","last_peered":"2024-09-20T22:45:08.146214+0000","last_clean":"2024-09-20T22:45:08.146214+0000","last_became_active":"2024-09-20T22:44:09.171546+0000","last_became_peered":"2024-09-20T22:44:09.171546+0000","last_unstale":"2024-09-20T22:45:08.146214+0000","last_undegraded":"2024-09-20T22:45:08.146214+0000","last_fullsized":"2024-09-20T22:45:08.146214+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-09-20T22:43:08.643587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:43:08.643587+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:09.108085+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":332,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.144523+0000","last_change":"2024-09-20T22:43:33.657671+0000","last_active":"2024-09-20T22:45:08.144523+0000","last_peered":"2024-09-20T22:45:08.144523+0000","last_clean":"2024-09-20T22:45:08.144523+0000","last_became_active":"2024-09-20T22:41:44.197619+0000","last_became_peered":"2024-09-20T22:41:44.197619+0000","last_unstale":"2024-09-20T22:45:08.144523+0000","last_undegraded":"2024-09-20T22:45:08.144523+0000","last_fullsized":"2024-09-20T22:45:08.144523+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:33.657587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:43:33.657587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:08.787654+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.0","version":"223'72","reported_seq":771,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.150975+0000","last_change":"2024-09-20T22:45:08.151197+0000","last_active":"2024-09-20T22:45:08.150975+0000","last_peered":"2024-09-20T22:45:08.150975+0000","last_clean":"2024-09-20T22:45:08.150975+0000","last_became_active":"2024-09-20T22:43:37.136845+0000","last_became_peered":"2024-09-20T22:43:37.136845+0000","last_unstale":"2024-09-20T22:45:08.150975+0000","last_undegraded":"2024-09-20T22:45:08.150975+0000","last_fullsized":"2024-09-20T22:45:08.150975+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"145'58","last_scrub_stamp":"2024-09-20T22:43:45.705214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:43:45.705214+0000","objects_scrubbed":3,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00044961000000000003,"stat_sum":{"num_bytes":5529536,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":11412,"num_write":9,"num_write_kb":2220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"39","length":"1"},{"start":"46","length":"1"}]},{"pgid":"1.6","version":"77'128","reported_seq":601,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.161255+0000","last_change":"2024-09-20T22:43:27.532172+0000","last_active":"2024-09-20T22:45:08.161255+0000","last_peered":"2024-09-20T22:45:08.161255+0000","last_clean":"2024-09-20T22:45:08.161255+0000","last_became_active":"2024-09-20T22:43:19.130331+0000","last_became_peered":"2024-09-20T22:43:19.130331+0000","last_unstale":"2024-09-20T22:45:08.161255+0000","last_undegraded":"2024-09-20T22:45:08.161255+0000","last_fullsized":"2024-09-20T22:45:08.161255+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:43:27.532085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:43:27.532085+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:10.682077+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"74'19","reported_seq":384,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.145055+0000","last_change":"2024-09-20T22:44:37.632952+0000","last_active":"2024-09-20T22:45:08.145055+0000","last_peered":"2024-09-20T22:45:08.145055+0000","last_clean":"2024-09-20T22:45:08.145055+0000","last_became_active":"2024-09-20T22:41:40.251866+0000","last_became_peered":"2024-09-20T22:41:40.251866+0000","last_unstale":"2024-09-20T22:45:08.145055+0000","last_undegraded":"2024-09-20T22:45:08.145055+0000","last_fullsized":"2024-09-20T22:45:08.145055+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'19","last_scrub_stamp":"2024-09-20T22:44:37.632834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:44:37.632834+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:52.348513+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":344,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.198924+0000","last_change":"2024-09-20T22:43:32.231716+0000","last_active":"2024-09-20T22:45:08.198924+0000","last_peered":"2024-09-20T22:45:08.198924+0000","last_clean":"2024-09-20T22:45:08.198924+0000","last_became_active":"2024-09-20T22:41:38.160832+0000","last_became_peered":"2024-09-20T22:41:38.160832+0000","last_unstale":"2024-09-20T22:45:08.198924+0000","last_undegraded":"2024-09-20T22:45:08.198924+0000","last_fullsized":"2024-09-20T22:45:08.198924+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:32.231644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:43:32.231644+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:12.054283+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":336,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.160679+0000","last_change":"2024-09-20T22:44:19.441854+0000","last_active":"2024-09-20T22:45:08.160679+0000","last_peered":"2024-09-20T22:45:08.160679+0000","last_clean":"2024-09-20T22:45:08.160679+0000","last_became_active":"2024-09-20T22:41:42.184412+0000","last_became_peered":"2024-09-20T22:41:42.184412+0000","last_unstale":"2024-09-20T22:45:08.160679+0000","last_undegraded":"2024-09-20T22:45:08.160679+0000","last_fullsized":"2024-09-20T22:45:08.160679+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:44:19.441770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:44:19.441770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:34.420562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":332,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.198580+0000","last_change":"2024-09-20T22:43:53.247501+0000","last_active":"2024-09-20T22:45:08.198580+0000","last_peered":"2024-09-20T22:45:08.198580+0000","last_clean":"2024-09-20T22:45:08.198580+0000","last_became_active":"2024-09-20T22:41:44.830538+0000","last_became_peered":"2024-09-20T22:41:44.830538+0000","last_unstale":"2024-09-20T22:45:08.198580+0000","last_undegraded":"2024-09-20T22:45:08.198580+0000","last_fullsized":"2024-09-20T22:45:08.198580+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:53.247430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:43:53.247430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:12.054350+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.7","version":"229'185","reported_seq":875,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.171833+0000","last_change":"2024-09-20T22:45:08.164744+0000","last_active":"2024-09-20T22:45:08.171833+0000","last_peered":"2024-09-20T22:45:08.171833+0000","last_clean":"2024-09-20T22:45:08.171833+0000","last_became_active":"2024-09-20T22:43:37.132943+0000","last_became_peered":"2024-09-20T22:43:37.132943+0000","last_unstale":"2024-09-20T22:45:08.171833+0000","last_undegraded":"2024-09-20T22:45:08.171833+0000","last_fullsized":"2024-09-20T22:45:08.171833+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"147'119","last_scrub_stamp":"2024-09-20T22:43:47.407291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:43:47.407291+0000","objects_scrubbed":2,"log_size":185,"log_dups_size":0,"ondisk_log_size":185,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0031502359999999998,"stat_sum":{"num_bytes":7334629,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":51,"num_read_kb":52559,"num_write":43,"num_write_kb":14512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"39","length":"1"},{"start":"46","length":"1"}]},{"pgid":"1.1","version":"77'128","reported_seq":594,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.150865+0000","last_change":"2024-09-20T22:43:26.761137+0000","last_active":"2024-09-20T22:45:08.150865+0000","last_peered":"2024-09-20T22:45:08.150865+0000","last_clean":"2024-09-20T22:45:08.150865+0000","last_became_active":"2024-09-20T22:43:19.125806+0000","last_became_peered":"2024-09-20T22:43:19.125806+0000","last_unstale":"2024-09-20T22:45:08.150865+0000","last_undegraded":"2024-09-20T22:45:08.150865+0000","last_fullsized":"2024-09-20T22:45:08.150865+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:43:26.761032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:43:26.761032+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"86'38","reported_seq":449,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.197975+0000","last_change":"2024-09-20T22:44:41.077590+0000","last_active":"2024-09-20T22:45:08.197975+0000","last_peered":"2024-09-20T22:45:08.197975+0000","last_clean":"2024-09-20T22:45:08.197975+0000","last_became_active":"2024-09-20T22:41:40.166958+0000","last_became_peered":"2024-09-20T22:41:40.166958+0000","last_unstale":"2024-09-20T22:45:08.197975+0000","last_undegraded":"2024-09-20T22:45:08.197975+0000","last_fullsized":"2024-09-20T22:45:08.197975+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'38","last_scrub_stamp":"2024-09-20T22:44:41.077475+0000","last_deep_scrub":"86'38","last_deep_scrub_stamp":"2024-09-20T22:42:43.935677+0000","last_clean_scrub_stamp":"2024-09-20T22:44:41.077475+0000","objects_scrubbed":10,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:46:05.497572+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":162,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.145242+0000","last_change":"2024-09-20T22:44:38.652538+0000","last_active":"2024-09-20T22:45:08.145242+0000","last_peered":"2024-09-20T22:45:08.145242+0000","last_clean":"2024-09-20T22:45:08.145242+0000","last_became_active":"2024-09-20T22:43:57.432267+0000","last_became_peered":"2024-09-20T22:43:57.432267+0000","last_unstale":"2024-09-20T22:45:08.145242+0000","last_undegraded":"2024-09-20T22:45:08.145242+0000","last_fullsized":"2024-09-20T22:45:08.145242+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:44:38.652446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:44:38.652446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:53.468832+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"69'2","reported_seq":362,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.151617+0000","last_change":"2024-09-20T22:44:35.668676+0000","last_active":"2024-09-20T22:45:08.151617+0000","last_peered":"2024-09-20T22:45:08.151617+0000","last_clean":"2024-09-20T22:45:08.151617+0000","last_became_active":"2024-09-20T22:41:37.148479+0000","last_became_peered":"2024-09-20T22:41:37.148479+0000","last_unstale":"2024-09-20T22:45:08.151617+0000","last_undegraded":"2024-09-20T22:45:08.151617+0000","last_fullsized":"2024-09-20T22:45:08.151617+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2024-09-20T22:44:35.668576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:36.130684+0000","last_clean_scrub_stamp":"2024-09-20T22:44:35.668576+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:39.072433+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00087280599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":342,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.144894+0000","last_change":"2024-09-20T22:44:11.765375+0000","last_active":"2024-09-20T22:45:08.144894+0000","last_peered":"2024-09-20T22:45:08.144894+0000","last_clean":"2024-09-20T22:45:08.144894+0000","last_became_active":"2024-09-20T22:43:57.431815+0000","last_became_peered":"2024-09-20T22:43:57.431815+0000","last_unstale":"2024-09-20T22:45:08.144894+0000","last_undegraded":"2024-09-20T22:45:08.144894+0000","last_fullsized":"2024-09-20T22:45:08.144894+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:44:11.765248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:44:11.765248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:34.701647+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":338,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.146586+0000","last_change":"2024-09-20T22:43:57.434389+0000","last_active":"2024-09-20T22:45:08.146586+0000","last_peered":"2024-09-20T22:45:08.146586+0000","last_clean":"2024-09-20T22:45:08.146586+0000","last_became_active":"2024-09-20T22:43:57.434044+0000","last_became_peered":"2024-09-20T22:43:57.434044+0000","last_unstale":"2024-09-20T22:45:08.146586+0000","last_undegraded":"2024-09-20T22:45:08.146586+0000","last_fullsized":"2024-09-20T22:45:08.146586+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:09.406141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:43:09.406141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:09.108053+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"210'50","reported_seq":745,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.145032+0000","last_change":"2024-09-20T22:45:08.145214+0000","last_active":"2024-09-20T22:45:08.145032+0000","last_peered":"2024-09-20T22:45:08.145032+0000","last_clean":"2024-09-20T22:45:08.145032+0000","last_became_active":"2024-09-20T22:43:57.429486+0000","last_became_peered":"2024-09-20T22:43:57.429486+0000","last_unstale":"2024-09-20T22:45:08.145032+0000","last_undegraded":"2024-09-20T22:45:08.145032+0000","last_fullsized":"2024-09-20T22:45:08.145032+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"145'39","last_scrub_stamp":"2024-09-20T22:43:47.802333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:43:47.802333+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00071078500000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":20,"num_read_kb":28189,"num_write":6,"num_write_kb":1350,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"39","length":"1"},{"start":"46","length":"1"}]},{"pgid":"1.2","version":"77'128","reported_seq":652,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.156194+0000","last_change":"2024-09-20T22:44:58.121790+0000","last_active":"2024-09-20T22:45:08.156194+0000","last_peered":"2024-09-20T22:45:08.156194+0000","last_clean":"2024-09-20T22:45:08.156194+0000","last_became_active":"2024-09-20T22:44:00.146907+0000","last_became_peered":"2024-09-20T22:44:00.146907+0000","last_unstale":"2024-09-20T22:45:08.156194+0000","last_undegraded":"2024-09-20T22:45:08.156194+0000","last_fullsized":"2024-09-20T22:45:08.156194+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:44:58.121721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:44:58.121721+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:46:14.009238+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"74'13","reported_seq":367,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.197964+0000","last_change":"2024-09-20T22:43:31.240331+0000","last_active":"2024-09-20T22:45:08.197964+0000","last_peered":"2024-09-20T22:45:08.197964+0000","last_clean":"2024-09-20T22:45:08.197964+0000","last_became_active":"2024-09-20T22:41:40.169473+0000","last_became_peered":"2024-09-20T22:41:40.169473+0000","last_unstale":"2024-09-20T22:45:08.197964+0000","last_undegraded":"2024-09-20T22:45:08.197964+0000","last_fullsized":"2024-09-20T22:45:08.197964+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'13","last_scrub_stamp":"2024-09-20T22:43:31.240196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:43:31.240196+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:12.054336+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":352,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.199279+0000","last_change":"2024-09-20T22:44:35.131693+0000","last_active":"2024-09-20T22:45:08.199279+0000","last_peered":"2024-09-20T22:45:08.199279+0000","last_clean":"2024-09-20T22:45:08.199279+0000","last_became_active":"2024-09-20T22:41:38.156837+0000","last_became_peered":"2024-09-20T22:41:38.156837+0000","last_unstale":"2024-09-20T22:45:08.199279+0000","last_undegraded":"2024-09-20T22:45:08.199279+0000","last_fullsized":"2024-09-20T22:45:08.199279+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:44:35.131583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:44:35.131583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:47.311461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"67'1","reported_seq":360,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.154282+0000","last_change":"2024-09-20T22:44:39.441694+0000","last_active":"2024-09-20T22:45:08.154282+0000","last_peered":"2024-09-20T22:45:08.154282+0000","last_clean":"2024-09-20T22:45:08.154282+0000","last_became_active":"2024-09-20T22:41:37.148165+0000","last_became_peered":"2024-09-20T22:41:37.148165+0000","last_unstale":"2024-09-20T22:45:08.154282+0000","last_undegraded":"2024-09-20T22:45:08.154282+0000","last_fullsized":"2024-09-20T22:45:08.154282+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-09-20T22:44:39.441585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:36.130684+0000","last_clean_scrub_stamp":"2024-09-20T22:44:39.441585+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:40.621607+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001301459,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":335,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.156822+0000","last_change":"2024-09-20T22:44:14.284410+0000","last_active":"2024-09-20T22:45:08.156822+0000","last_peered":"2024-09-20T22:45:08.156822+0000","last_clean":"2024-09-20T22:45:08.156822+0000","last_became_active":"2024-09-20T22:41:42.185112+0000","last_became_peered":"2024-09-20T22:41:42.185112+0000","last_unstale":"2024-09-20T22:45:08.156822+0000","last_undegraded":"2024-09-20T22:45:08.156822+0000","last_fullsized":"2024-09-20T22:45:08.156822+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:44:14.284296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:44:14.284296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:23.561043+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":332,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.157305+0000","last_change":"2024-09-20T22:44:14.946132+0000","last_active":"2024-09-20T22:45:08.157305+0000","last_peered":"2024-09-20T22:45:08.157305+0000","last_clean":"2024-09-20T22:45:08.157305+0000","last_became_active":"2024-09-20T22:41:44.830515+0000","last_became_peered":"2024-09-20T22:41:44.830515+0000","last_unstale":"2024-09-20T22:45:08.157305+0000","last_undegraded":"2024-09-20T22:45:08.157305+0000","last_fullsized":"2024-09-20T22:45:08.157305+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:44:14.946052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:44:14.946052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:17.759544+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"224'32","reported_seq":712,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.157361+0000","last_change":"2024-09-20T22:45:08.157841+0000","last_active":"2024-09-20T22:45:08.157361+0000","last_peered":"2024-09-20T22:45:08.157361+0000","last_clean":"2024-09-20T22:45:08.157361+0000","last_became_active":"2024-09-20T22:43:37.133478+0000","last_became_peered":"2024-09-20T22:43:37.133478+0000","last_unstale":"2024-09-20T22:45:08.157361+0000","last_undegraded":"2024-09-20T22:45:08.157361+0000","last_fullsized":"2024-09-20T22:45:08.157361+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'3","last_scrub_stamp":"2024-09-20T22:43:49.197854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:43:49.197854+0000","objects_scrubbed":1,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:13.293991+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00069767899999999999,"stat_sum":{"num_bytes":3966452,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":17,"num_read_kb":12235,"num_write":20,"num_write_kb":11130,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"39","length":"1"},{"start":"46","length":"1"}]},{"pgid":"1.3","version":"77'128","reported_seq":603,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.199722+0000","last_change":"2024-09-20T22:44:00.144446+0000","last_active":"2024-09-20T22:45:08.199722+0000","last_peered":"2024-09-20T22:45:08.199722+0000","last_clean":"2024-09-20T22:45:08.199722+0000","last_became_active":"2024-09-20T22:44:00.144099+0000","last_became_peered":"2024-09-20T22:44:00.144099+0000","last_unstale":"2024-09-20T22:45:08.199722+0000","last_undegraded":"2024-09-20T22:45:08.199722+0000","last_fullsized":"2024-09-20T22:45:08.199722+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:43:25.244624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:43:25.244624+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:12.054302+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"74'12","reported_seq":378,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.145646+0000","last_change":"2024-09-20T22:43:26.720587+0000","last_active":"2024-09-20T22:45:08.145646+0000","last_peered":"2024-09-20T22:45:08.145646+0000","last_clean":"2024-09-20T22:45:08.145646+0000","last_became_active":"2024-09-20T22:43:09.120425+0000","last_became_peered":"2024-09-20T22:43:09.120425+0000","last_unstale":"2024-09-20T22:45:08.145646+0000","last_undegraded":"2024-09-20T22:45:08.145646+0000","last_fullsized":"2024-09-20T22:45:08.145646+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-09-20T22:43:26.720499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:43:26.720499+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:08.787605+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":350,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.144941+0000","last_change":"2024-09-20T22:44:19.834724+0000","last_active":"2024-09-20T22:45:08.144941+0000","last_peered":"2024-09-20T22:45:08.144941+0000","last_clean":"2024-09-20T22:45:08.144941+0000","last_became_active":"2024-09-20T22:43:57.421167+0000","last_became_peered":"2024-09-20T22:43:57.421167+0000","last_unstale":"2024-09-20T22:45:08.144941+0000","last_undegraded":"2024-09-20T22:45:08.144941+0000","last_fullsized":"2024-09-20T22:45:08.144941+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:44:19.834641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:44:19.834641+0000","last_clean_scrub_stamp":"2024-09-20T22:44:19.834641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:43.561314+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"75'5","reported_seq":560,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.157088+0000","last_change":"2024-09-20T22:44:13.885050+0000","last_active":"2024-09-20T22:45:08.157088+0000","last_peered":"2024-09-20T22:45:08.157088+0000","last_clean":"2024-09-20T22:45:08.157088+0000","last_became_active":"2024-09-20T22:41:37.148383+0000","last_became_peered":"2024-09-20T22:41:37.148383+0000","last_unstale":"2024-09-20T22:45:08.157088+0000","last_undegraded":"2024-09-20T22:45:08.157088+0000","last_fullsized":"2024-09-20T22:45:08.157088+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-09-20T22:44:13.884968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:36.130684+0000","last_clean_scrub_stamp":"2024-09-20T22:44:13.884968+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:22.123942+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00078631700000000003,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":205,"num_read_kb":200,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":336,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.152147+0000","last_change":"2024-09-20T22:44:38.788128+0000","last_active":"2024-09-20T22:45:08.152147+0000","last_peered":"2024-09-20T22:45:08.152147+0000","last_clean":"2024-09-20T22:45:08.152147+0000","last_became_active":"2024-09-20T22:41:42.187346+0000","last_became_peered":"2024-09-20T22:41:42.187346+0000","last_unstale":"2024-09-20T22:45:08.152147+0000","last_undegraded":"2024-09-20T22:45:08.152147+0000","last_fullsized":"2024-09-20T22:45:08.152147+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:44:38.788045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:44:38.788045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:46:03.203383+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":340,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.198610+0000","last_change":"2024-09-20T22:44:54.014154+0000","last_active":"2024-09-20T22:45:08.198610+0000","last_peered":"2024-09-20T22:45:08.198610+0000","last_clean":"2024-09-20T22:45:08.198610+0000","last_became_active":"2024-09-20T22:41:44.199046+0000","last_became_peered":"2024-09-20T22:41:44.199046+0000","last_unstale":"2024-09-20T22:45:08.198610+0000","last_undegraded":"2024-09-20T22:45:08.198610+0000","last_fullsized":"2024-09-20T22:45:08.198610+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:44:54.014085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:44:54.014085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:46:01.646305+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.6","version":"150'60","reported_seq":748,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.147673+0000","last_change":"2024-09-20T22:45:08.147882+0000","last_active":"2024-09-20T22:45:08.147673+0000","last_peered":"2024-09-20T22:45:08.147673+0000","last_clean":"2024-09-20T22:45:08.147673+0000","last_became_active":"2024-09-20T22:43:37.136591+0000","last_became_peered":"2024-09-20T22:43:37.136591+0000","last_unstale":"2024-09-20T22:45:08.147673+0000","last_undegraded":"2024-09-20T22:45:08.147673+0000","last_fullsized":"2024-09-20T22:45:08.147673+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'60","last_scrub_stamp":"2024-09-20T22:43:49.735664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:43:49.735664+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:09.316062+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0023279249999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"39","length":"1"},{"start":"46","length":"1"}]},{"pgid":"4.5","version":"159'17","reported_seq":429,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.161282+0000","last_change":"2024-09-20T22:43:57.462065+0000","last_active":"2024-09-20T22:45:08.161282+0000","last_peered":"2024-09-20T22:45:08.161282+0000","last_clean":"2024-09-20T22:45:08.161282+0000","last_became_active":"2024-09-20T22:43:57.433076+0000","last_became_peered":"2024-09-20T22:43:57.433076+0000","last_unstale":"2024-09-20T22:45:08.161282+0000","last_undegraded":"2024-09-20T22:45:08.161282+0000","last_fullsized":"2024-09-20T22:45:08.161282+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'16","last_scrub_stamp":"2024-09-20T22:43:56.348264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:43:56.348264+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:12.752488+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"67'1","reported_seq":342,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.146291+0000","last_change":"2024-09-20T22:41:38.159706+0000","last_active":"2024-09-20T22:45:08.146291+0000","last_peered":"2024-09-20T22:45:08.146291+0000","last_clean":"2024-09-20T22:45:08.146291+0000","last_became_active":"2024-09-20T22:41:38.158917+0000","last_became_peered":"2024-09-20T22:41:38.158917+0000","last_unstale":"2024-09-20T22:45:08.146291+0000","last_undegraded":"2024-09-20T22:45:08.146291+0000","last_fullsized":"2024-09-20T22:45:08.146291+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:41:37.135621+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:09.107816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"77'128","reported_seq":635,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.157248+0000","last_change":"2024-09-20T22:44:35.887245+0000","last_active":"2024-09-20T22:45:08.157248+0000","last_peered":"2024-09-20T22:45:08.157248+0000","last_clean":"2024-09-20T22:45:08.157248+0000","last_became_active":"2024-09-20T22:43:19.126925+0000","last_became_peered":"2024-09-20T22:43:19.126925+0000","last_unstale":"2024-09-20T22:45:08.157248+0000","last_undegraded":"2024-09-20T22:45:08.157248+0000","last_fullsized":"2024-09-20T22:45:08.157248+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:44:35.887120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:44:35.887120+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:55.559505+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":336,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.157294+0000","last_change":"2024-09-20T22:43:52.163679+0000","last_active":"2024-09-20T22:45:08.157294+0000","last_peered":"2024-09-20T22:45:08.157294+0000","last_clean":"2024-09-20T22:45:08.157294+0000","last_became_active":"2024-09-20T22:41:42.184387+0000","last_became_peered":"2024-09-20T22:41:42.184387+0000","last_unstale":"2024-09-20T22:45:08.157294+0000","last_undegraded":"2024-09-20T22:45:08.157294+0000","last_fullsized":"2024-09-20T22:45:08.157294+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:52.163576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:43:52.163576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:10.284322+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":338,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.152019+0000","last_change":"2024-09-20T22:43:57.434803+0000","last_active":"2024-09-20T22:45:08.152019+0000","last_peered":"2024-09-20T22:45:08.152019+0000","last_clean":"2024-09-20T22:45:08.152019+0000","last_became_active":"2024-09-20T22:43:57.434435+0000","last_became_peered":"2024-09-20T22:43:57.434435+0000","last_unstale":"2024-09-20T22:45:08.152019+0000","last_undegraded":"2024-09-20T22:45:08.152019+0000","last_fullsized":"2024-09-20T22:45:08.152019+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:02.696105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:43:02.696105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"77'128","reported_seq":609,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.199649+0000","last_change":"2024-09-20T22:45:02.051386+0000","last_active":"2024-09-20T22:45:08.199649+0000","last_peered":"2024-09-20T22:45:08.199649+0000","last_clean":"2024-09-20T22:45:08.199649+0000","last_became_active":"2024-09-20T22:43:24.124252+0000","last_became_peered":"2024-09-20T22:43:24.124252+0000","last_unstale":"2024-09-20T22:45:08.199649+0000","last_undegraded":"2024-09-20T22:45:08.199649+0000","last_fullsized":"2024-09-20T22:45:08.199649+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:45:02.051304+0000","last_deep_scrub":"77'128","last_deep_scrub_stamp":"2024-09-20T22:45:02.051304+0000","last_clean_scrub_stamp":"2024-09-20T22:45:02.051304+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:46:20.486842+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.d","version":"194'71","reported_seq":756,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.161504+0000","last_change":"2024-09-20T22:45:08.161504+0000","last_active":"2024-09-20T22:45:08.161504+0000","last_peered":"2024-09-20T22:45:08.161504+0000","last_clean":"2024-09-20T22:45:08.161504+0000","last_became_active":"2024-09-20T22:42:52.410381+0000","last_became_peered":"2024-09-20T22:42:52.410381+0000","last_unstale":"2024-09-20T22:45:08.161504+0000","last_undegraded":"2024-09-20T22:45:08.161504+0000","last_fullsized":"2024-09-20T22:45:08.161504+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"194'71","last_scrub_stamp":"2024-09-20T22:44:40.529449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:44:40.529449+0000","objects_scrubbed":1,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:55.271658+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00056110600000000002,"stat_sum":{"num_bytes":2588245,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":52,"num_read_kb":49492,"num_write":25,"num_write_kb":7311,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"39","length":"1"},{"start":"46","length":"1"}]},{"pgid":"4.e","version":"74'11","reported_seq":230,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.161363+0000","last_change":"2024-09-20T22:44:16.481158+0000","last_active":"2024-09-20T22:45:08.161363+0000","last_peered":"2024-09-20T22:45:08.161363+0000","last_clean":"2024-09-20T22:45:08.161363+0000","last_became_active":"2024-09-20T22:44:09.177976+0000","last_became_peered":"2024-09-20T22:44:09.177976+0000","last_unstale":"2024-09-20T22:45:08.161363+0000","last_undegraded":"2024-09-20T22:45:08.161363+0000","last_fullsized":"2024-09-20T22:45:08.161363+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-09-20T22:44:16.481063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:44:16.481063+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:45.292329+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":154,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.154140+0000","last_change":"2024-09-20T22:43:57.432786+0000","last_active":"2024-09-20T22:45:08.154140+0000","last_peered":"2024-09-20T22:45:08.154140+0000","last_clean":"2024-09-20T22:45:08.154140+0000","last_became_active":"2024-09-20T22:43:57.432492+0000","last_became_peered":"2024-09-20T22:43:57.432492+0000","last_unstale":"2024-09-20T22:45:08.154140+0000","last_undegraded":"2024-09-20T22:45:08.154140+0000","last_fullsized":"2024-09-20T22:45:08.154140+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:31.979152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:43:31.979152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:08.389370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":342,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.151898+0000","last_change":"2024-09-20T22:44:57.607725+0000","last_active":"2024-09-20T22:45:08.151898+0000","last_peered":"2024-09-20T22:45:08.151898+0000","last_clean":"2024-09-20T22:45:08.151898+0000","last_became_active":"2024-09-20T22:43:57.419756+0000","last_became_peered":"2024-09-20T22:43:57.419756+0000","last_unstale":"2024-09-20T22:45:08.151898+0000","last_undegraded":"2024-09-20T22:45:08.151898+0000","last_fullsized":"2024-09-20T22:45:08.151898+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:44:57.607616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:44:57.607616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:46:21.194848+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":332,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.146454+0000","last_change":"2024-09-20T22:44:21.225215+0000","last_active":"2024-09-20T22:45:08.146454+0000","last_peered":"2024-09-20T22:45:08.146454+0000","last_clean":"2024-09-20T22:45:08.146454+0000","last_became_active":"2024-09-20T22:41:44.832009+0000","last_became_peered":"2024-09-20T22:41:44.832009+0000","last_unstale":"2024-09-20T22:45:08.146454+0000","last_undegraded":"2024-09-20T22:45:08.146454+0000","last_fullsized":"2024-09-20T22:45:08.146454+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:44:21.225119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:44:21.225119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:39.909132+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.e","version":"224'91","reported_seq":775,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.157105+0000","last_change":"2024-09-20T22:45:08.157105+0000","last_active":"2024-09-20T22:45:08.157105+0000","last_peered":"2024-09-20T22:45:08.157105+0000","last_clean":"2024-09-20T22:45:08.157105+0000","last_became_active":"2024-09-20T22:42:52.405039+0000","last_became_peered":"2024-09-20T22:42:52.405039+0000","last_unstale":"2024-09-20T22:45:08.157105+0000","last_undegraded":"2024-09-20T22:45:08.157105+0000","last_fullsized":"2024-09-20T22:45:08.157105+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"174'60","last_scrub_stamp":"2024-09-20T22:44:18.301776+0000","last_deep_scrub":"174'60","last_deep_scrub_stamp":"2024-09-20T22:44:18.301776+0000","last_clean_scrub_stamp":"2024-09-20T22:44:18.301776+0000","objects_scrubbed":2,"log_size":91,"log_dups_size":0,"ondisk_log_size":91,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:24.823799+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0.000593565,"stat_sum":{"num_bytes":7176589,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":63,"num_read_kb":71358,"num_write":38,"num_write_kb":15332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"39","length":"1"},{"start":"46","length":"1"}]},{"pgid":"1.8","version":"77'128","reported_seq":581,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.150711+0000","last_change":"2024-09-20T22:43:04.748279+0000","last_active":"2024-09-20T22:45:08.150711+0000","last_peered":"2024-09-20T22:45:08.150711+0000","last_clean":"2024-09-20T22:45:08.150711+0000","last_became_active":"2024-09-20T22:42:52.405594+0000","last_became_peered":"2024-09-20T22:42:52.405594+0000","last_unstale":"2024-09-20T22:45:08.150711+0000","last_undegraded":"2024-09-20T22:45:08.150711+0000","last_fullsized":"2024-09-20T22:45:08.150711+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:43:04.748176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:43:04.748176+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"74'17","reported_seq":227,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.156000+0000","last_change":"2024-09-20T22:44:21.261543+0000","last_active":"2024-09-20T22:45:08.156000+0000","last_peered":"2024-09-20T22:45:08.156000+0000","last_clean":"2024-09-20T22:45:08.156000+0000","last_became_active":"2024-09-20T22:43:57.432621+0000","last_became_peered":"2024-09-20T22:43:57.432621+0000","last_unstale":"2024-09-20T22:45:08.156000+0000","last_undegraded":"2024-09-20T22:45:08.156000+0000","last_fullsized":"2024-09-20T22:45:08.156000+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'17","last_scrub_stamp":"2024-09-20T22:44:21.261443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:44:21.261443+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:39.405257+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":28,"num_read_kb":18,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":350,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.160934+0000","last_change":"2024-09-20T22:43:57.429041+0000","last_active":"2024-09-20T22:45:08.160934+0000","last_peered":"2024-09-20T22:45:08.160934+0000","last_clean":"2024-09-20T22:45:08.160934+0000","last_became_active":"2024-09-20T22:43:57.425267+0000","last_became_peered":"2024-09-20T22:43:57.425267+0000","last_unstale":"2024-09-20T22:45:08.160934+0000","last_undegraded":"2024-09-20T22:45:08.160934+0000","last_fullsized":"2024-09-20T22:45:08.160934+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:35.492562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:35.492562+0000","last_clean_scrub_stamp":"2024-09-20T22:43:35.492562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:10.682170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":342,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.198673+0000","last_change":"2024-09-20T22:44:11.041771+0000","last_active":"2024-09-20T22:45:08.198673+0000","last_peered":"2024-09-20T22:45:08.198673+0000","last_clean":"2024-09-20T22:45:08.198673+0000","last_became_active":"2024-09-20T22:43:57.435048+0000","last_became_peered":"2024-09-20T22:43:57.435048+0000","last_unstale":"2024-09-20T22:45:08.198673+0000","last_undegraded":"2024-09-20T22:45:08.198673+0000","last_fullsized":"2024-09-20T22:45:08.198673+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:44:11.041699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:44:11.041699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:34.130343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":345,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.156149+0000","last_change":"2024-09-20T22:44:20.297496+0000","last_active":"2024-09-20T22:45:08.156149+0000","last_peered":"2024-09-20T22:45:08.156149+0000","last_clean":"2024-09-20T22:45:08.156149+0000","last_became_active":"2024-09-20T22:43:57.430630+0000","last_became_peered":"2024-09-20T22:43:57.430630+0000","last_unstale":"2024-09-20T22:45:08.156149+0000","last_undegraded":"2024-09-20T22:45:08.156149+0000","last_fullsized":"2024-09-20T22:45:08.156149+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:44:20.297424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:44:20.297424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:44.120776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"224'139","reported_seq":829,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.160323+0000","last_change":"2024-09-20T22:45:08.147033+0000","last_active":"2024-09-20T22:45:08.160323+0000","last_peered":"2024-09-20T22:45:08.160323+0000","last_clean":"2024-09-20T22:45:08.160323+0000","last_became_active":"2024-09-20T22:42:52.401210+0000","last_became_peered":"2024-09-20T22:42:52.401210+0000","last_unstale":"2024-09-20T22:45:08.160323+0000","last_undegraded":"2024-09-20T22:45:08.160323+0000","last_fullsized":"2024-09-20T22:45:08.160323+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"194'90","last_scrub_stamp":"2024-09-20T22:44:34.651203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:44:34.651203+0000","objects_scrubbed":3,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:35.077957+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0019229570000000001,"stat_sum":{"num_bytes":9462440,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":82,"num_read_kb":66987,"num_write":66,"num_write_kb":24841,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"39","length":"1"},{"start":"46","length":"1"}]},{"pgid":"1.9","version":"77'128","reported_seq":622,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.156598+0000","last_change":"2024-09-20T22:43:03.239347+0000","last_active":"2024-09-20T22:45:08.156598+0000","last_peered":"2024-09-20T22:45:08.156598+0000","last_clean":"2024-09-20T22:45:08.156598+0000","last_became_active":"2024-09-20T22:42:52.400353+0000","last_became_peered":"2024-09-20T22:42:52.400353+0000","last_unstale":"2024-09-20T22:45:08.156598+0000","last_undegraded":"2024-09-20T22:45:08.156598+0000","last_fullsized":"2024-09-20T22:45:08.156598+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:43:03.239276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:43:03.239276+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:10.284259+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"74'10","reported_seq":200,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.149247+0000","last_change":"2024-09-20T22:43:57.486043+0000","last_active":"2024-09-20T22:45:08.149247+0000","last_peered":"2024-09-20T22:45:08.149247+0000","last_clean":"2024-09-20T22:45:08.149247+0000","last_became_active":"2024-09-20T22:43:57.441016+0000","last_became_peered":"2024-09-20T22:43:57.441016+0000","last_unstale":"2024-09-20T22:45:08.149247+0000","last_undegraded":"2024-09-20T22:45:08.149247+0000","last_fullsized":"2024-09-20T22:45:08.149247+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-09-20T22:43:51.040007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:43:51.040007+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:09.108352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":350,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.147476+0000","last_change":"2024-09-20T22:43:57.441917+0000","last_active":"2024-09-20T22:45:08.147476+0000","last_peered":"2024-09-20T22:45:08.147476+0000","last_clean":"2024-09-20T22:45:08.147476+0000","last_became_active":"2024-09-20T22:43:57.441623+0000","last_became_peered":"2024-09-20T22:43:57.441623+0000","last_unstale":"2024-09-20T22:45:08.147476+0000","last_undegraded":"2024-09-20T22:45:08.147476+0000","last_fullsized":"2024-09-20T22:45:08.147476+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:05.431346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:43:05.431346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:09.108027+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"74'8","reported_seq":386,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.155949+0000","last_change":"2024-09-20T22:43:56.238237+0000","last_active":"2024-09-20T22:45:08.155949+0000","last_peered":"2024-09-20T22:45:08.155949+0000","last_clean":"2024-09-20T22:45:08.155949+0000","last_became_active":"2024-09-20T22:41:42.185863+0000","last_became_peered":"2024-09-20T22:41:42.185863+0000","last_unstale":"2024-09-20T22:45:08.155949+0000","last_undegraded":"2024-09-20T22:45:08.155949+0000","last_fullsized":"2024-09-20T22:45:08.155949+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-09-20T22:43:56.238114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:43:56.238114+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:23.984720+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":163,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.198867+0000","last_change":"2024-09-20T22:44:13.962056+0000","last_active":"2024-09-20T22:45:08.198867+0000","last_peered":"2024-09-20T22:45:08.198867+0000","last_clean":"2024-09-20T22:45:08.198867+0000","last_became_active":"2024-09-20T22:43:57.430067+0000","last_became_peered":"2024-09-20T22:43:57.430067+0000","last_unstale":"2024-09-20T22:45:08.198867+0000","last_undegraded":"2024-09-20T22:45:08.198867+0000","last_fullsized":"2024-09-20T22:45:08.198867+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:44:13.961983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:44:13.961983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:24.591102+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.e","version":"77'128","reported_seq":453,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.156843+0000","last_change":"2024-09-20T22:44:12.918980+0000","last_active":"2024-09-20T22:45:08.156843+0000","last_peered":"2024-09-20T22:45:08.156843+0000","last_clean":"2024-09-20T22:45:08.156843+0000","last_became_active":"2024-09-20T22:44:00.145891+0000","last_became_peered":"2024-09-20T22:44:00.145891+0000","last_unstale":"2024-09-20T22:45:08.156843+0000","last_undegraded":"2024-09-20T22:45:08.156843+0000","last_fullsized":"2024-09-20T22:45:08.156843+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:44:12.918885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:44:12.918885+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:14.421434+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.8","version":"213'51","reported_seq":744,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.152727+0000","last_change":"2024-09-20T22:45:08.153188+0000","last_active":"2024-09-20T22:45:08.152727+0000","last_peered":"2024-09-20T22:45:08.152727+0000","last_clean":"2024-09-20T22:45:08.152727+0000","last_became_active":"2024-09-20T22:43:57.431081+0000","last_became_peered":"2024-09-20T22:43:57.431081+0000","last_unstale":"2024-09-20T22:45:08.152727+0000","last_undegraded":"2024-09-20T22:45:08.152727+0000","last_fullsized":"2024-09-20T22:45:08.152727+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"143'41","last_scrub_stamp":"2024-09-20T22:43:46.243723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:43:46.243723+0000","objects_scrubbed":0,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:12.167315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065495600000000005,"stat_sum":{"num_bytes":2117715,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":7,"num_write_kb":3247,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"39","length":"1"},{"start":"46","length":"1"}]},{"pgid":"4.b","version":"74'9","reported_seq":380,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.144815+0000","last_change":"2024-09-20T22:44:57.603747+0000","last_active":"2024-09-20T22:45:08.144815+0000","last_peered":"2024-09-20T22:45:08.144815+0000","last_clean":"2024-09-20T22:45:08.144815+0000","last_became_active":"2024-09-20T22:43:57.442189+0000","last_became_peered":"2024-09-20T22:43:57.442189+0000","last_unstale":"2024-09-20T22:45:08.144815+0000","last_undegraded":"2024-09-20T22:45:08.144815+0000","last_fullsized":"2024-09-20T22:45:08.144815+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-09-20T22:44:57.603671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:44:57.603671+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:46:05.790764+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":344,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.151692+0000","last_change":"2024-09-20T22:43:52.773066+0000","last_active":"2024-09-20T22:45:08.151692+0000","last_peered":"2024-09-20T22:45:08.151692+0000","last_clean":"2024-09-20T22:45:08.151692+0000","last_became_active":"2024-09-20T22:41:38.155953+0000","last_became_peered":"2024-09-20T22:41:38.155953+0000","last_unstale":"2024-09-20T22:45:08.151692+0000","last_undegraded":"2024-09-20T22:45:08.151692+0000","last_fullsized":"2024-09-20T22:45:08.151692+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:52.772982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:43:52.772982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":341,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.154493+0000","last_change":"2024-09-20T22:43:57.440505+0000","last_active":"2024-09-20T22:45:08.154493+0000","last_peered":"2024-09-20T22:45:08.154493+0000","last_clean":"2024-09-20T22:45:08.154493+0000","last_became_active":"2024-09-20T22:43:57.440200+0000","last_became_peered":"2024-09-20T22:43:57.440200+0000","last_unstale":"2024-09-20T22:45:08.154493+0000","last_undegraded":"2024-09-20T22:45:08.154493+0000","last_fullsized":"2024-09-20T22:45:08.154493+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:55.274872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:43:55.274872+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:17.019595+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":332,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.144852+0000","last_change":"2024-09-20T22:44:22.818177+0000","last_active":"2024-09-20T22:45:08.144852+0000","last_peered":"2024-09-20T22:45:08.144852+0000","last_clean":"2024-09-20T22:45:08.144852+0000","last_became_active":"2024-09-20T22:41:44.196633+0000","last_became_peered":"2024-09-20T22:41:44.196633+0000","last_unstale":"2024-09-20T22:45:08.144852+0000","last_undegraded":"2024-09-20T22:45:08.144852+0000","last_fullsized":"2024-09-20T22:45:08.144852+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:44:22.818078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:44:22.818078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:47.784094+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.f","version":"77'128","reported_seq":617,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.161640+0000","last_change":"2024-09-20T22:44:00.141677+0000","last_active":"2024-09-20T22:45:08.161640+0000","last_peered":"2024-09-20T22:45:08.161640+0000","last_clean":"2024-09-20T22:45:08.161640+0000","last_became_active":"2024-09-20T22:44:00.141317+0000","last_became_peered":"2024-09-20T22:44:00.141317+0000","last_unstale":"2024-09-20T22:45:08.161640+0000","last_undegraded":"2024-09-20T22:45:08.161640+0000","last_fullsized":"2024-09-20T22:45:08.161640+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:43:30.421648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:43:30.421648+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:10.682094+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.9","version":"210'96","reported_seq":804,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.144729+0000","last_change":"2024-09-20T22:45:08.145851+0000","last_active":"2024-09-20T22:45:08.144729+0000","last_peered":"2024-09-20T22:45:08.144729+0000","last_clean":"2024-09-20T22:45:08.144729+0000","last_became_active":"2024-09-20T22:43:57.431328+0000","last_became_peered":"2024-09-20T22:43:57.431328+0000","last_unstale":"2024-09-20T22:45:08.144729+0000","last_undegraded":"2024-09-20T22:45:08.144729+0000","last_fullsized":"2024-09-20T22:45:08.144729+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"131'44","last_scrub_stamp":"2024-09-20T22:43:46.794463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:43:46.794463+0000","objects_scrubbed":1,"log_size":96,"log_dups_size":0,"ondisk_log_size":96,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:14.620774+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0010718290000000001,"stat_sum":{"num_bytes":4874063,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":24,"num_read_kb":26899,"num_write":21,"num_write_kb":7692,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":3815156,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"39","length":"1"},{"start":"46","length":"1"}]},{"pgid":"4.a","version":"113'28","reported_seq":388,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.160767+0000","last_change":"2024-09-20T22:44:55.649681+0000","last_active":"2024-09-20T22:45:08.160767+0000","last_peered":"2024-09-20T22:45:08.160767+0000","last_clean":"2024-09-20T22:45:08.160767+0000","last_became_active":"2024-09-20T22:41:40.252011+0000","last_became_peered":"2024-09-20T22:41:40.252011+0000","last_unstale":"2024-09-20T22:45:08.160767+0000","last_undegraded":"2024-09-20T22:45:08.160767+0000","last_fullsized":"2024-09-20T22:45:08.160767+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'28","last_scrub_stamp":"2024-09-20T22:44:55.649589+0000","last_deep_scrub":"113'28","last_deep_scrub_stamp":"2024-09-20T22:43:12.725900+0000","last_clean_scrub_stamp":"2024-09-20T22:44:55.649589+0000","objects_scrubbed":9,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:59.901122+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":344,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.157494+0000","last_change":"2024-09-20T22:44:18.955889+0000","last_active":"2024-09-20T22:45:08.157494+0000","last_peered":"2024-09-20T22:45:08.157494+0000","last_clean":"2024-09-20T22:45:08.157494+0000","last_became_active":"2024-09-20T22:41:38.161524+0000","last_became_peered":"2024-09-20T22:41:38.161524+0000","last_unstale":"2024-09-20T22:45:08.157494+0000","last_undegraded":"2024-09-20T22:45:08.157494+0000","last_fullsized":"2024-09-20T22:45:08.157494+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:44:18.955801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:44:18.955801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:26.881892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":335,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.154651+0000","last_change":"2024-09-20T22:44:12.236220+0000","last_active":"2024-09-20T22:45:08.154651+0000","last_peered":"2024-09-20T22:45:08.154651+0000","last_clean":"2024-09-20T22:45:08.154651+0000","last_became_active":"2024-09-20T22:41:42.184342+0000","last_became_peered":"2024-09-20T22:41:42.184342+0000","last_unstale":"2024-09-20T22:45:08.154651+0000","last_undegraded":"2024-09-20T22:45:08.154651+0000","last_fullsized":"2024-09-20T22:45:08.154651+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:44:12.236096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:44:12.236096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:16.825489+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":340,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.157195+0000","last_change":"2024-09-20T22:44:42.956942+0000","last_active":"2024-09-20T22:45:08.157195+0000","last_peered":"2024-09-20T22:45:08.157195+0000","last_clean":"2024-09-20T22:45:08.157195+0000","last_became_active":"2024-09-20T22:41:44.199395+0000","last_became_peered":"2024-09-20T22:41:44.199395+0000","last_unstale":"2024-09-20T22:45:08.157195+0000","last_undegraded":"2024-09-20T22:45:08.157195+0000","last_fullsized":"2024-09-20T22:45:08.157195+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:44:42.956859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:44:42.956859+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:48.879336+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.c","version":"77'128","reported_seq":607,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.149348+0000","last_change":"2024-09-20T22:43:30.045082+0000","last_active":"2024-09-20T22:45:08.149348+0000","last_peered":"2024-09-20T22:45:08.149348+0000","last_clean":"2024-09-20T22:45:08.149348+0000","last_became_active":"2024-09-20T22:43:24.130949+0000","last_became_peered":"2024-09-20T22:43:24.130949+0000","last_unstale":"2024-09-20T22:45:08.149348+0000","last_undegraded":"2024-09-20T22:45:08.149348+0000","last_fullsized":"2024-09-20T22:45:08.149348+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:43:30.044975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:43:30.044975+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:09.108224+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"229'141","reported_seq":842,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.190431+0000","last_change":"2024-09-20T22:45:08.147402+0000","last_active":"2024-09-20T22:45:08.190431+0000","last_peered":"2024-09-20T22:45:08.190431+0000","last_clean":"2024-09-20T22:45:08.190431+0000","last_became_active":"2024-09-20T22:42:52.405085+0000","last_became_peered":"2024-09-20T22:42:52.405085+0000","last_unstale":"2024-09-20T22:45:08.190431+0000","last_undegraded":"2024-09-20T22:45:08.190431+0000","last_fullsized":"2024-09-20T22:45:08.190431+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:42:51.354519+0000","objects_scrubbed":0,"log_size":141,"log_dups_size":0,"ondisk_log_size":141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001390276,"stat_sum":{"num_bytes":9445567,"num_objects":5,"num_object_clones":2,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":123,"num_read_kb":130841,"num_write":87,"num_write_kb":30238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"39","length":"1"},{"start":"46","length":"1"}]},{"pgid":"4.9","version":"74'12","reported_seq":216,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.198756+0000","last_change":"2024-09-20T22:44:53.049195+0000","last_active":"2024-09-20T22:45:08.198756+0000","last_peered":"2024-09-20T22:45:08.198756+0000","last_clean":"2024-09-20T22:45:08.198756+0000","last_became_active":"2024-09-20T22:43:57.442218+0000","last_became_peered":"2024-09-20T22:43:57.442218+0000","last_unstale":"2024-09-20T22:45:08.198756+0000","last_undegraded":"2024-09-20T22:45:08.198756+0000","last_fullsized":"2024-09-20T22:45:08.198756+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-09-20T22:44:53.049111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:44:53.049111+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:55.990924+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":21,"num_read_kb":12,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":358,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.158090+0000","last_change":"2024-09-20T22:44:20.970396+0000","last_active":"2024-09-20T22:45:08.158090+0000","last_peered":"2024-09-20T22:45:08.158090+0000","last_clean":"2024-09-20T22:45:08.158090+0000","last_became_active":"2024-09-20T22:43:57.419980+0000","last_became_peered":"2024-09-20T22:43:57.419980+0000","last_unstale":"2024-09-20T22:45:08.158090+0000","last_undegraded":"2024-09-20T22:45:08.158090+0000","last_fullsized":"2024-09-20T22:45:08.158090+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:44:20.970293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:44:20.970293+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:44.477869+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":343,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.152875+0000","last_change":"2024-09-20T22:44:37.379076+0000","last_active":"2024-09-20T22:45:08.152875+0000","last_peered":"2024-09-20T22:45:08.152875+0000","last_clean":"2024-09-20T22:45:08.152875+0000","last_became_active":"2024-09-20T22:41:42.181811+0000","last_became_peered":"2024-09-20T22:41:42.181811+0000","last_unstale":"2024-09-20T22:45:08.152875+0000","last_undegraded":"2024-09-20T22:45:08.152875+0000","last_fullsized":"2024-09-20T22:45:08.152875+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:44:37.379005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:44:37.379005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:44.358677+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":332,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.147150+0000","last_change":"2024-09-20T22:43:55.170518+0000","last_active":"2024-09-20T22:45:08.147150+0000","last_peered":"2024-09-20T22:45:08.147150+0000","last_clean":"2024-09-20T22:45:08.147150+0000","last_became_active":"2024-09-20T22:41:44.192609+0000","last_became_peered":"2024-09-20T22:41:44.192609+0000","last_unstale":"2024-09-20T22:45:08.147150+0000","last_undegraded":"2024-09-20T22:45:08.147150+0000","last_fullsized":"2024-09-20T22:45:08.147150+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:55.170379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:55.170379+0000","last_clean_scrub_stamp":"2024-09-20T22:43:55.170379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:11.086854+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.d","version":"77'128","reported_seq":529,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.156882+0000","last_change":"2024-09-20T22:44:00.147264+0000","last_active":"2024-09-20T22:45:08.156882+0000","last_peered":"2024-09-20T22:45:08.156882+0000","last_clean":"2024-09-20T22:45:08.156882+0000","last_became_active":"2024-09-20T22:44:00.146905+0000","last_became_peered":"2024-09-20T22:44:00.146905+0000","last_unstale":"2024-09-20T22:45:08.156882+0000","last_undegraded":"2024-09-20T22:45:08.156882+0000","last_fullsized":"2024-09-20T22:45:08.156882+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:43:30.175229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:43:30.175229+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:10.284306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.b","version":"229'136","reported_seq":841,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.210400+0000","last_change":"2024-09-20T22:45:08.201395+0000","last_active":"2024-09-20T22:45:08.210400+0000","last_peered":"2024-09-20T22:45:08.210400+0000","last_clean":"2024-09-20T22:45:08.210400+0000","last_became_active":"2024-09-20T22:42:52.409642+0000","last_became_peered":"2024-09-20T22:42:52.409642+0000","last_unstale":"2024-09-20T22:45:08.210400+0000","last_undegraded":"2024-09-20T22:45:08.210400+0000","last_fullsized":"2024-09-20T22:45:08.210400+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"194'116","last_scrub_stamp":"2024-09-20T22:44:37.097674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:44:37.097674+0000","objects_scrubbed":3,"log_size":136,"log_dups_size":0,"ondisk_log_size":136,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:43.343526+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0019392019999999999,"stat_sum":{"num_bytes":11462440,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":105,"num_read_kb":97405,"num_write":81,"num_write_kb":30535,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"39","length":"1"},{"start":"46","length":"1"}]},{"pgid":"4.8","version":"74'15","reported_seq":365,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.150891+0000","last_change":"2024-09-20T22:45:00.648680+0000","last_active":"2024-09-20T22:45:08.150891+0000","last_peered":"2024-09-20T22:45:08.150891+0000","last_clean":"2024-09-20T22:45:08.150891+0000","last_became_active":"2024-09-20T22:41:40.252113+0000","last_became_peered":"2024-09-20T22:41:40.252113+0000","last_unstale":"2024-09-20T22:45:08.150891+0000","last_undegraded":"2024-09-20T22:45:08.150891+0000","last_fullsized":"2024-09-20T22:45:08.150891+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-09-20T22:45:00.648587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:45:00.648587+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:46:04.296690+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'2","reported_seq":368,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.158562+0000","last_change":"2024-09-20T22:44:16.936605+0000","last_active":"2024-09-20T22:45:08.158562+0000","last_peered":"2024-09-20T22:45:08.158562+0000","last_clean":"2024-09-20T22:45:08.158562+0000","last_became_active":"2024-09-20T22:43:57.422322+0000","last_became_peered":"2024-09-20T22:43:57.422322+0000","last_unstale":"2024-09-20T22:45:08.158562+0000","last_undegraded":"2024-09-20T22:45:08.158562+0000","last_fullsized":"2024-09-20T22:45:08.158562+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-09-20T22:44:16.936523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:44:16.936523+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:21.826669+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"159'9","reported_seq":405,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.156544+0000","last_change":"2024-09-20T22:44:36.904443+0000","last_active":"2024-09-20T22:45:08.156544+0000","last_peered":"2024-09-20T22:45:08.156544+0000","last_clean":"2024-09-20T22:45:08.156544+0000","last_became_active":"2024-09-20T22:43:57.420663+0000","last_became_peered":"2024-09-20T22:43:57.420663+0000","last_unstale":"2024-09-20T22:45:08.156544+0000","last_undegraded":"2024-09-20T22:45:08.156544+0000","last_fullsized":"2024-09-20T22:45:08.156544+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"159'9","last_scrub_stamp":"2024-09-20T22:44:36.904367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:44:36.904367+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:54.974970+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":332,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.151250+0000","last_change":"2024-09-20T22:45:01.616499+0000","last_active":"2024-09-20T22:45:08.151250+0000","last_peered":"2024-09-20T22:45:08.151250+0000","last_clean":"2024-09-20T22:45:08.151250+0000","last_became_active":"2024-09-20T22:41:44.830556+0000","last_became_peered":"2024-09-20T22:41:44.830556+0000","last_unstale":"2024-09-20T22:45:08.151250+0000","last_undegraded":"2024-09-20T22:45:08.151250+0000","last_fullsized":"2024-09-20T22:45:08.151250+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:45:01.616426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:45:01.616426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:46:16.580554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.14","version":"166'42","reported_seq":494,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.153212+0000","last_change":"2024-09-20T22:45:08.153212+0000","last_active":"2024-09-20T22:45:08.153212+0000","last_peered":"2024-09-20T22:45:08.153212+0000","last_clean":"2024-09-20T22:45:08.153212+0000","last_became_active":"2024-09-20T22:43:39.132459+0000","last_became_peered":"2024-09-20T22:43:39.132459+0000","last_unstale":"2024-09-20T22:45:08.153212+0000","last_undegraded":"2024-09-20T22:45:08.153212+0000","last_fullsized":"2024-09-20T22:45:08.153212+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":5,"last_scrub":"147'38","last_scrub_stamp":"2024-09-20T22:43:49.134297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:43:49.134297+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:14.826319+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0052646710000000003,"stat_sum":{"num_bytes":2806769,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":8509,"num_write":5,"num_write_kb":1512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"39","length":"1"},{"start":"46","length":"1"}]},{"pgid":"1.12","version":"77'128","reported_seq":518,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.151429+0000","last_change":"2024-09-20T22:43:29.735118+0000","last_active":"2024-09-20T22:45:08.151429+0000","last_peered":"2024-09-20T22:45:08.151429+0000","last_clean":"2024-09-20T22:45:08.151429+0000","last_became_active":"2024-09-20T22:43:24.124301+0000","last_became_peered":"2024-09-20T22:43:24.124301+0000","last_unstale":"2024-09-20T22:45:08.151429+0000","last_undegraded":"2024-09-20T22:45:08.151429+0000","last_fullsized":"2024-09-20T22:45:08.151429+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:43:29.735047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:43:29.735047+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":352,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.159584+0000","last_change":"2024-09-20T22:44:42.612217+0000","last_active":"2024-09-20T22:45:08.159584+0000","last_peered":"2024-09-20T22:45:08.159584+0000","last_clean":"2024-09-20T22:45:08.159584+0000","last_became_active":"2024-09-20T22:41:38.159139+0000","last_became_peered":"2024-09-20T22:41:38.159139+0000","last_unstale":"2024-09-20T22:45:08.159584+0000","last_undegraded":"2024-09-20T22:45:08.159584+0000","last_fullsized":"2024-09-20T22:45:08.159584+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:44:42.612144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:44:42.612144+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:46:11.366920+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"74'6","reported_seq":363,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.148531+0000","last_change":"2024-09-20T22:43:09.121694+0000","last_active":"2024-09-20T22:45:08.148531+0000","last_peered":"2024-09-20T22:45:08.148531+0000","last_clean":"2024-09-20T22:45:08.148531+0000","last_became_active":"2024-09-20T22:43:09.116887+0000","last_became_peered":"2024-09-20T22:43:09.116887+0000","last_unstale":"2024-09-20T22:45:08.148531+0000","last_undegraded":"2024-09-20T22:45:08.148531+0000","last_fullsized":"2024-09-20T22:45:08.148531+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'6","last_scrub_stamp":"2024-09-20T22:42:41.469214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:42:41.469214+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:09.107935+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":336,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.151978+0000","last_change":"2024-09-20T22:43:09.705715+0000","last_active":"2024-09-20T22:45:08.151978+0000","last_peered":"2024-09-20T22:45:08.151978+0000","last_clean":"2024-09-20T22:45:08.151978+0000","last_became_active":"2024-09-20T22:41:42.184930+0000","last_became_peered":"2024-09-20T22:41:42.184930+0000","last_unstale":"2024-09-20T22:45:08.151978+0000","last_undegraded":"2024-09-20T22:45:08.151978+0000","last_fullsized":"2024-09-20T22:45:08.151978+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:09.705632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:09.705632+0000","last_clean_scrub_stamp":"2024-09-20T22:43:09.705632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"74'1","reported_seq":343,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.157041+0000","last_change":"2024-09-20T22:43:57.435632+0000","last_active":"2024-09-20T22:45:08.157041+0000","last_peered":"2024-09-20T22:45:08.157041+0000","last_clean":"2024-09-20T22:45:08.157041+0000","last_became_active":"2024-09-20T22:43:57.419681+0000","last_became_peered":"2024-09-20T22:43:57.419681+0000","last_unstale":"2024-09-20T22:45:08.157041+0000","last_undegraded":"2024-09-20T22:45:08.157041+0000","last_fullsized":"2024-09-20T22:45:08.157041+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-09-20T22:43:33.180910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:43:33.180910+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:10.284360+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.15","version":"98'3","reported_seq":685,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.157000+0000","last_change":"2024-09-20T22:45:08.157731+0000","last_active":"2024-09-20T22:45:08.157000+0000","last_peered":"2024-09-20T22:45:08.157000+0000","last_clean":"2024-09-20T22:45:08.157000+0000","last_became_active":"2024-09-20T22:44:09.174719+0000","last_became_peered":"2024-09-20T22:44:09.174719+0000","last_unstale":"2024-09-20T22:45:08.157000+0000","last_undegraded":"2024-09-20T22:45:08.157000+0000","last_fullsized":"2024-09-20T22:45:08.157000+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":5,"last_scrub":"98'3","last_scrub_stamp":"2024-09-20T22:43:47.212572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:43:47.212572+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00098604199999999991,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"39","length":"1"},{"start":"46","length":"1"}]},{"pgid":"1.13","version":"77'128","reported_seq":618,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.199529+0000","last_change":"2024-09-20T22:44:38.112816+0000","last_active":"2024-09-20T22:45:08.199529+0000","last_peered":"2024-09-20T22:45:08.199529+0000","last_clean":"2024-09-20T22:45:08.199529+0000","last_became_active":"2024-09-20T22:44:00.144081+0000","last_became_peered":"2024-09-20T22:44:00.144081+0000","last_unstale":"2024-09-20T22:45:08.199529+0000","last_undegraded":"2024-09-20T22:45:08.199529+0000","last_fullsized":"2024-09-20T22:45:08.199529+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:44:38.112742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:44:38.112742+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:42.803148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"74'9","reported_seq":364,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.145774+0000","last_change":"2024-09-20T22:44:52.644656+0000","last_active":"2024-09-20T22:45:08.145774+0000","last_peered":"2024-09-20T22:45:08.145774+0000","last_clean":"2024-09-20T22:45:08.145774+0000","last_became_active":"2024-09-20T22:41:40.253280+0000","last_became_peered":"2024-09-20T22:41:40.253280+0000","last_unstale":"2024-09-20T22:45:08.145774+0000","last_undegraded":"2024-09-20T22:45:08.145774+0000","last_fullsized":"2024-09-20T22:45:08.145774+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-09-20T22:44:52.644575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:44:52.644575+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:46:04.206183+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":350,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.156693+0000","last_change":"2024-09-20T22:44:11.942183+0000","last_active":"2024-09-20T22:45:08.156693+0000","last_peered":"2024-09-20T22:45:08.156693+0000","last_clean":"2024-09-20T22:45:08.156693+0000","last_became_active":"2024-09-20T22:43:57.419001+0000","last_became_peered":"2024-09-20T22:43:57.419001+0000","last_unstale":"2024-09-20T22:45:08.156693+0000","last_undegraded":"2024-09-20T22:45:08.156693+0000","last_fullsized":"2024-09-20T22:45:08.156693+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:44:11.942106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:44:11.942106+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:12.734421+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":336,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.148447+0000","last_change":"2024-09-20T22:43:32.057321+0000","last_active":"2024-09-20T22:45:08.148447+0000","last_peered":"2024-09-20T22:45:08.148447+0000","last_clean":"2024-09-20T22:45:08.148447+0000","last_became_active":"2024-09-20T22:41:42.187706+0000","last_became_peered":"2024-09-20T22:41:42.187706+0000","last_unstale":"2024-09-20T22:45:08.148447+0000","last_undegraded":"2024-09-20T22:45:08.148447+0000","last_fullsized":"2024-09-20T22:45:08.148447+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:32.057240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:43:32.057240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:09.108295+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":345,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.155123+0000","last_change":"2024-09-20T22:44:58.459406+0000","last_active":"2024-09-20T22:45:08.155123+0000","last_peered":"2024-09-20T22:45:08.155123+0000","last_clean":"2024-09-20T22:45:08.155123+0000","last_became_active":"2024-09-20T22:43:57.428284+0000","last_became_peered":"2024-09-20T22:43:57.428284+0000","last_unstale":"2024-09-20T22:45:08.155123+0000","last_undegraded":"2024-09-20T22:45:08.155123+0000","last_fullsized":"2024-09-20T22:45:08.155123+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:44:58.459332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:44:58.459332+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:46:11.813722+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.16","version":"229'53","reported_seq":651,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.147397+0000","last_change":"2024-09-20T22:45:08.147397+0000","last_active":"2024-09-20T22:45:08.147397+0000","last_peered":"2024-09-20T22:45:08.147397+0000","last_clean":"2024-09-20T22:45:08.147397+0000","last_became_active":"2024-09-20T22:43:57.423267+0000","last_became_peered":"2024-09-20T22:43:57.423267+0000","last_unstale":"2024-09-20T22:45:08.147397+0000","last_undegraded":"2024-09-20T22:45:08.147397+0000","last_fullsized":"2024-09-20T22:45:08.147397+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":5,"last_scrub":"136'41","last_scrub_stamp":"2024-09-20T22:44:09.743964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:44:09.743964+0000","objects_scrubbed":1,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:10.429312+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.002425806,"stat_sum":{"num_bytes":2218491,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":17,"num_read_kb":15653,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"39","length":"1"},{"start":"46","length":"1"}]},{"pgid":"1.10","version":"77'128","reported_seq":526,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.150594+0000","last_change":"2024-09-20T22:44:34.630283+0000","last_active":"2024-09-20T22:45:08.150594+0000","last_peered":"2024-09-20T22:45:08.150594+0000","last_clean":"2024-09-20T22:45:08.150594+0000","last_became_active":"2024-09-20T22:43:24.125651+0000","last_became_peered":"2024-09-20T22:43:24.125651+0000","last_unstale":"2024-09-20T22:45:08.150594+0000","last_undegraded":"2024-09-20T22:45:08.150594+0000","last_fullsized":"2024-09-20T22:45:08.150594+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:44:34.630184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:44:34.630184+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:40.023975+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.15","version":"74'9","reported_seq":356,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.151920+0000","last_change":"2024-09-20T22:43:49.825694+0000","last_active":"2024-09-20T22:45:08.151920+0000","last_peered":"2024-09-20T22:45:08.151920+0000","last_clean":"2024-09-20T22:45:08.151920+0000","last_became_active":"2024-09-20T22:41:40.252486+0000","last_became_peered":"2024-09-20T22:41:40.252486+0000","last_unstale":"2024-09-20T22:45:08.151920+0000","last_undegraded":"2024-09-20T22:45:08.151920+0000","last_fullsized":"2024-09-20T22:45:08.151920+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-09-20T22:43:49.825573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:43:49.825573+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:17.986284+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":344,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.144483+0000","last_change":"2024-09-20T22:43:32.652702+0000","last_active":"2024-09-20T22:45:08.144483+0000","last_peered":"2024-09-20T22:45:08.144483+0000","last_clean":"2024-09-20T22:45:08.144483+0000","last_became_active":"2024-09-20T22:41:38.158146+0000","last_became_peered":"2024-09-20T22:41:38.158146+0000","last_unstale":"2024-09-20T22:45:08.144483+0000","last_undegraded":"2024-09-20T22:45:08.144483+0000","last_fullsized":"2024-09-20T22:45:08.144483+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:32.652587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:43:32.652587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:08.787623+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"74'8","reported_seq":387,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.148780+0000","last_change":"2024-09-20T22:43:52.190172+0000","last_active":"2024-09-20T22:45:08.148780+0000","last_peered":"2024-09-20T22:45:08.148780+0000","last_clean":"2024-09-20T22:45:08.148780+0000","last_became_active":"2024-09-20T22:41:42.186352+0000","last_became_peered":"2024-09-20T22:41:42.186352+0000","last_unstale":"2024-09-20T22:45:08.148780+0000","last_undegraded":"2024-09-20T22:45:08.148780+0000","last_fullsized":"2024-09-20T22:45:08.148780+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-09-20T22:43:52.190097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:43:52.190097+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:14.977846+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":163,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.155994+0000","last_change":"2024-09-20T22:44:13.241839+0000","last_active":"2024-09-20T22:45:08.155994+0000","last_peered":"2024-09-20T22:45:08.155994+0000","last_clean":"2024-09-20T22:45:08.155994+0000","last_became_active":"2024-09-20T22:43:57.424515+0000","last_became_peered":"2024-09-20T22:43:57.424515+0000","last_unstale":"2024-09-20T22:45:08.155994+0000","last_undegraded":"2024-09-20T22:45:08.155994+0000","last_fullsized":"2024-09-20T22:45:08.155994+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:44:13.241769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:44:13.241769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:14.652647+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.17","version":"229'170","reported_seq":578,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.190077+0000","last_change":"2024-09-20T22:45:08.158750+0000","last_active":"2024-09-20T22:45:08.190077+0000","last_peered":"2024-09-20T22:45:08.190077+0000","last_clean":"2024-09-20T22:45:08.190077+0000","last_became_active":"2024-09-20T22:43:39.130801+0000","last_became_peered":"2024-09-20T22:43:39.130801+0000","last_unstale":"2024-09-20T22:45:08.190077+0000","last_undegraded":"2024-09-20T22:45:08.190077+0000","last_fullsized":"2024-09-20T22:45:08.190077+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":5,"last_scrub":"147'126","last_scrub_stamp":"2024-09-20T22:43:48.215007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:43:48.215007+0000","objects_scrubbed":3,"log_size":170,"log_dups_size":0,"ondisk_log_size":170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.00053714500000000005,"stat_sum":{"num_bytes":10160660,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":47,"num_read_kb":56206,"num_write":25,"num_write_kb":14368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"39","length":"1"},{"start":"46","length":"1"}]},{"pgid":"1.11","version":"77'128","reported_seq":532,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.145029+0000","last_change":"2024-09-20T22:43:30.651129+0000","last_active":"2024-09-20T22:45:08.145029+0000","last_peered":"2024-09-20T22:45:08.145029+0000","last_clean":"2024-09-20T22:45:08.145029+0000","last_became_active":"2024-09-20T22:43:24.128884+0000","last_became_peered":"2024-09-20T22:43:24.128884+0000","last_unstale":"2024-09-20T22:45:08.145029+0000","last_undegraded":"2024-09-20T22:45:08.145029+0000","last_fullsized":"2024-09-20T22:45:08.145029+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:43:30.651054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:43:30.651054+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:08.787671+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"74'10","reported_seq":355,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.149502+0000","last_change":"2024-09-20T22:43:11.355914+0000","last_active":"2024-09-20T22:45:08.149502+0000","last_peered":"2024-09-20T22:45:08.149502+0000","last_clean":"2024-09-20T22:45:08.149502+0000","last_became_active":"2024-09-20T22:41:40.253614+0000","last_became_peered":"2024-09-20T22:41:40.253614+0000","last_unstale":"2024-09-20T22:45:08.149502+0000","last_undegraded":"2024-09-20T22:45:08.149502+0000","last_fullsized":"2024-09-20T22:45:08.149502+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-09-20T22:43:11.355829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:43:11.355829+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:09.108175+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":350,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.156724+0000","last_change":"2024-09-20T22:43:57.425304+0000","last_active":"2024-09-20T22:45:08.156724+0000","last_peered":"2024-09-20T22:45:08.156724+0000","last_clean":"2024-09-20T22:45:08.156724+0000","last_became_active":"2024-09-20T22:43:57.425004+0000","last_became_peered":"2024-09-20T22:43:57.425004+0000","last_unstale":"2024-09-20T22:45:08.156724+0000","last_undegraded":"2024-09-20T22:45:08.156724+0000","last_fullsized":"2024-09-20T22:45:08.156724+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:51.130212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:43:51.130212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:10.284290+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"74'8","reported_seq":390,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.151768+0000","last_change":"2024-09-20T22:42:59.718431+0000","last_active":"2024-09-20T22:45:08.151768+0000","last_peered":"2024-09-20T22:45:08.151768+0000","last_clean":"2024-09-20T22:45:08.151768+0000","last_became_active":"2024-09-20T22:41:42.189451+0000","last_became_peered":"2024-09-20T22:41:42.189451+0000","last_unstale":"2024-09-20T22:45:08.151768+0000","last_undegraded":"2024-09-20T22:45:08.151768+0000","last_fullsized":"2024-09-20T22:45:08.151768+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-09-20T22:42:59.718354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:42:59.718354+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":340,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.144996+0000","last_change":"2024-09-20T22:45:00.730113+0000","last_active":"2024-09-20T22:45:08.144996+0000","last_peered":"2024-09-20T22:45:08.144996+0000","last_clean":"2024-09-20T22:45:08.144996+0000","last_became_active":"2024-09-20T22:41:44.191206+0000","last_became_peered":"2024-09-20T22:41:44.191206+0000","last_unstale":"2024-09-20T22:45:08.144996+0000","last_undegraded":"2024-09-20T22:45:08.144996+0000","last_fullsized":"2024-09-20T22:45:08.144996+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:45:00.729995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:45:00.729995+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:46:05.652294+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.10","version":"224'56","reported_seq":731,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.146690+0000","last_change":"2024-09-20T22:45:08.153067+0000","last_active":"2024-09-20T22:45:08.146690+0000","last_peered":"2024-09-20T22:45:08.146690+0000","last_clean":"2024-09-20T22:45:08.146690+0000","last_became_active":"2024-09-20T22:43:39.130391+0000","last_became_peered":"2024-09-20T22:43:39.130391+0000","last_unstale":"2024-09-20T22:45:08.146690+0000","last_undegraded":"2024-09-20T22:45:08.146690+0000","last_fullsized":"2024-09-20T22:45:08.146690+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":5,"last_scrub":"146'43","last_scrub_stamp":"2024-09-20T22:43:46.106059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:43:46.106059+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0097005930000000004,"stat_sum":{"num_bytes":4655572,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7491,"num_write":7,"num_write_kb":3334,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"39","length":"1"},{"start":"46","length":"1"}]},{"pgid":"4.13","version":"74'11","reported_seq":204,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.160233+0000","last_change":"2024-09-20T22:43:57.446108+0000","last_active":"2024-09-20T22:45:08.160233+0000","last_peered":"2024-09-20T22:45:08.160233+0000","last_clean":"2024-09-20T22:45:08.160233+0000","last_became_active":"2024-09-20T22:43:57.416358+0000","last_became_peered":"2024-09-20T22:43:57.416358+0000","last_unstale":"2024-09-20T22:45:08.160233+0000","last_undegraded":"2024-09-20T22:45:08.160233+0000","last_fullsized":"2024-09-20T22:45:08.160233+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-09-20T22:43:33.923283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:43:33.923283+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:10.682157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":154,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.157381+0000","last_change":"2024-09-20T22:43:57.423148+0000","last_active":"2024-09-20T22:45:08.157381+0000","last_peered":"2024-09-20T22:45:08.157381+0000","last_clean":"2024-09-20T22:45:08.157381+0000","last_became_active":"2024-09-20T22:43:57.422556+0000","last_became_peered":"2024-09-20T22:43:57.422556+0000","last_unstale":"2024-09-20T22:45:08.157381+0000","last_undegraded":"2024-09-20T22:45:08.157381+0000","last_fullsized":"2024-09-20T22:45:08.157381+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:55.053048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:43:55.053048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:15.115356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":336,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.198421+0000","last_change":"2024-09-20T22:44:20.088194+0000","last_active":"2024-09-20T22:45:08.198421+0000","last_peered":"2024-09-20T22:45:08.198421+0000","last_clean":"2024-09-20T22:45:08.198421+0000","last_became_active":"2024-09-20T22:41:42.181736+0000","last_became_peered":"2024-09-20T22:41:42.181736+0000","last_unstale":"2024-09-20T22:45:08.198421+0000","last_undegraded":"2024-09-20T22:45:08.198421+0000","last_fullsized":"2024-09-20T22:45:08.198421+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:44:20.088125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:44:20.088125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:33.759142+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":332,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.143633+0000","last_change":"2024-09-20T22:43:53.201536+0000","last_active":"2024-09-20T22:45:08.143633+0000","last_peered":"2024-09-20T22:45:08.143633+0000","last_clean":"2024-09-20T22:45:08.143633+0000","last_became_active":"2024-09-20T22:41:44.192960+0000","last_became_peered":"2024-09-20T22:41:44.192960+0000","last_unstale":"2024-09-20T22:45:08.143633+0000","last_undegraded":"2024-09-20T22:45:08.143633+0000","last_fullsized":"2024-09-20T22:45:08.143633+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:53.201465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:43:53.201465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:09.466528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"135'39","reported_seq":466,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.156404+0000","last_change":"2024-09-20T22:45:08.157139+0000","last_active":"2024-09-20T22:45:08.156404+0000","last_peered":"2024-09-20T22:45:08.156404+0000","last_clean":"2024-09-20T22:45:08.156404+0000","last_became_active":"2024-09-20T22:43:39.130210+0000","last_became_peered":"2024-09-20T22:43:39.130210+0000","last_unstale":"2024-09-20T22:45:08.156404+0000","last_undegraded":"2024-09-20T22:45:08.156404+0000","last_fullsized":"2024-09-20T22:45:08.156404+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":5,"last_scrub":"135'39","last_scrub_stamp":"2024-09-20T22:43:46.228343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:43:46.228343+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00094867200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"39","length":"1"},{"start":"46","length":"1"}]},{"pgid":"4.12","version":"74'9","reported_seq":364,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.197553+0000","last_change":"2024-09-20T22:44:54.989495+0000","last_active":"2024-09-20T22:45:08.197553+0000","last_peered":"2024-09-20T22:45:08.197553+0000","last_clean":"2024-09-20T22:45:08.197553+0000","last_became_active":"2024-09-20T22:41:40.164796+0000","last_became_peered":"2024-09-20T22:41:40.164796+0000","last_unstale":"2024-09-20T22:45:08.197553+0000","last_undegraded":"2024-09-20T22:45:08.197553+0000","last_fullsized":"2024-09-20T22:45:08.197553+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-09-20T22:44:54.989403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:44:54.989403+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:46:19.125237+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":353,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.157405+0000","last_change":"2024-09-20T22:43:57.429374+0000","last_active":"2024-09-20T22:45:08.157405+0000","last_peered":"2024-09-20T22:45:08.157405+0000","last_clean":"2024-09-20T22:45:08.157405+0000","last_became_active":"2024-09-20T22:43:57.429046+0000","last_became_peered":"2024-09-20T22:43:57.429046+0000","last_unstale":"2024-09-20T22:45:08.157405+0000","last_undegraded":"2024-09-20T22:45:08.157405+0000","last_fullsized":"2024-09-20T22:45:08.157405+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:31.138067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:31.138067+0000","last_clean_scrub_stamp":"2024-09-20T22:43:31.138067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:10.284274+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":336,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.147243+0000","last_change":"2024-09-20T22:44:15.242983+0000","last_active":"2024-09-20T22:45:08.147243+0000","last_peered":"2024-09-20T22:45:08.147243+0000","last_clean":"2024-09-20T22:45:08.147243+0000","last_became_active":"2024-09-20T22:41:42.182966+0000","last_became_peered":"2024-09-20T22:41:42.182966+0000","last_unstale":"2024-09-20T22:45:08.147243+0000","last_undegraded":"2024-09-20T22:45:08.147243+0000","last_fullsized":"2024-09-20T22:45:08.147243+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:44:15.242881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:44:15.242881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:20.962494+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":340,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.144205+0000","last_change":"2024-09-20T22:44:39.660113+0000","last_active":"2024-09-20T22:45:08.144205+0000","last_peered":"2024-09-20T22:45:08.144205+0000","last_clean":"2024-09-20T22:45:08.144205+0000","last_became_active":"2024-09-20T22:41:44.194551+0000","last_became_peered":"2024-09-20T22:41:44.194551+0000","last_unstale":"2024-09-20T22:45:08.144205+0000","last_undegraded":"2024-09-20T22:45:08.144205+0000","last_fullsized":"2024-09-20T22:45:08.144205+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:44:39.660039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:44:39.660039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:50.176169+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.12","version":"224'212","reported_seq":648,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.160875+0000","last_change":"2024-09-20T22:45:08.161155+0000","last_active":"2024-09-20T22:45:08.160875+0000","last_peered":"2024-09-20T22:45:08.160875+0000","last_clean":"2024-09-20T22:45:08.160875+0000","last_became_active":"2024-09-20T22:43:39.135211+0000","last_became_peered":"2024-09-20T22:43:39.135211+0000","last_unstale":"2024-09-20T22:45:08.160875+0000","last_undegraded":"2024-09-20T22:45:08.160875+0000","last_fullsized":"2024-09-20T22:45:08.160875+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":5,"last_scrub":"147'95","last_scrub_stamp":"2024-09-20T22:43:46.419480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:43:46.419480+0000","objects_scrubbed":3,"log_size":212,"log_dups_size":0,"ondisk_log_size":212,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:10.807877+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00051570399999999997,"stat_sum":{"num_bytes":8113397,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":60,"num_read_kb":72903,"num_write":51,"num_write_kb":16158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"39","length":"1"},{"start":"46","length":"1"}]},{"pgid":"1.14","version":"77'128","reported_seq":604,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.154994+0000","last_change":"2024-09-20T22:44:00.146043+0000","last_active":"2024-09-20T22:45:08.154994+0000","last_peered":"2024-09-20T22:45:08.154994+0000","last_clean":"2024-09-20T22:45:08.154994+0000","last_became_active":"2024-09-20T22:44:00.145687+0000","last_became_peered":"2024-09-20T22:44:00.145687+0000","last_unstale":"2024-09-20T22:45:08.154994+0000","last_undegraded":"2024-09-20T22:45:08.154994+0000","last_fullsized":"2024-09-20T22:45:08.154994+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:43:29.331919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:43:29.331919+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:08.389339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"74'11","reported_seq":149,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.150779+0000","last_change":"2024-09-20T22:44:14.705392+0000","last_active":"2024-09-20T22:45:08.150779+0000","last_peered":"2024-09-20T22:45:08.150779+0000","last_clean":"2024-09-20T22:45:08.150779+0000","last_became_active":"2024-09-20T22:44:09.170785+0000","last_became_peered":"2024-09-20T22:44:09.170785+0000","last_unstale":"2024-09-20T22:45:08.150779+0000","last_undegraded":"2024-09-20T22:45:08.150779+0000","last_fullsized":"2024-09-20T22:45:08.150779+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-09-20T22:44:14.705306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:44:14.705306+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:35.089571+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":344,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.152067+0000","last_change":"2024-09-20T22:43:06.775869+0000","last_active":"2024-09-20T22:45:08.152067+0000","last_peered":"2024-09-20T22:45:08.152067+0000","last_clean":"2024-09-20T22:45:08.152067+0000","last_became_active":"2024-09-20T22:41:38.158701+0000","last_became_peered":"2024-09-20T22:41:38.158701+0000","last_unstale":"2024-09-20T22:45:08.152067+0000","last_undegraded":"2024-09-20T22:45:08.152067+0000","last_fullsized":"2024-09-20T22:45:08.152067+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:06.775781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:06.775781+0000","last_clean_scrub_stamp":"2024-09-20T22:43:06.775781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":342,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.155994+0000","last_change":"2024-09-20T22:43:57.419543+0000","last_active":"2024-09-20T22:45:08.155994+0000","last_peered":"2024-09-20T22:45:08.155994+0000","last_clean":"2024-09-20T22:45:08.155994+0000","last_became_active":"2024-09-20T22:43:57.419200+0000","last_became_peered":"2024-09-20T22:43:57.419200+0000","last_unstale":"2024-09-20T22:45:08.155994+0000","last_undegraded":"2024-09-20T22:45:08.155994+0000","last_fullsized":"2024-09-20T22:45:08.155994+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:50.174861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:50.174861+0000","last_clean_scrub_stamp":"2024-09-20T22:43:50.174861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:18.151735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":332,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.147407+0000","last_change":"2024-09-20T22:45:00.006980+0000","last_active":"2024-09-20T22:45:08.147407+0000","last_peered":"2024-09-20T22:45:08.147407+0000","last_clean":"2024-09-20T22:45:08.147407+0000","last_became_active":"2024-09-20T22:41:44.831967+0000","last_became_peered":"2024-09-20T22:41:44.831967+0000","last_unstale":"2024-09-20T22:45:08.147407+0000","last_undegraded":"2024-09-20T22:45:08.147407+0000","last_fullsized":"2024-09-20T22:45:08.147407+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:45:00.006909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:45:00.006909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:46:17.412244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"224'20","reported_seq":505,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.145570+0000","last_change":"2024-09-20T22:45:08.145927+0000","last_active":"2024-09-20T22:45:08.145570+0000","last_peered":"2024-09-20T22:45:08.145570+0000","last_clean":"2024-09-20T22:45:08.145570+0000","last_became_active":"2024-09-20T22:44:09.177614+0000","last_became_peered":"2024-09-20T22:44:09.177614+0000","last_unstale":"2024-09-20T22:45:08.145570+0000","last_undegraded":"2024-09-20T22:45:08.145570+0000","last_fullsized":"2024-09-20T22:45:08.145570+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":5,"last_scrub":"147'18","last_scrub_stamp":"2024-09-20T22:43:48.759471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:43:48.759471+0000","objects_scrubbed":1,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00072724699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":2020747,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"39","length":"1"},{"start":"46","length":"1"}]},{"pgid":"4.10","version":"74'4","reported_seq":346,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.143815+0000","last_change":"2024-09-20T22:44:17.320732+0000","last_active":"2024-09-20T22:45:08.143815+0000","last_peered":"2024-09-20T22:45:08.143815+0000","last_clean":"2024-09-20T22:45:08.143815+0000","last_became_active":"2024-09-20T22:41:40.166123+0000","last_became_peered":"2024-09-20T22:41:40.166123+0000","last_unstale":"2024-09-20T22:45:08.143815+0000","last_undegraded":"2024-09-20T22:45:08.143815+0000","last_fullsized":"2024-09-20T22:45:08.143815+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'4","last_scrub_stamp":"2024-09-20T22:44:17.320608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:44:17.320608+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:17.670947+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":352,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.146795+0000","last_change":"2024-09-20T22:44:58.632805+0000","last_active":"2024-09-20T22:45:08.146795+0000","last_peered":"2024-09-20T22:45:08.146795+0000","last_clean":"2024-09-20T22:45:08.146795+0000","last_became_active":"2024-09-20T22:41:38.158347+0000","last_became_peered":"2024-09-20T22:41:38.158347+0000","last_unstale":"2024-09-20T22:45:08.146795+0000","last_undegraded":"2024-09-20T22:45:08.146795+0000","last_fullsized":"2024-09-20T22:45:08.146795+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:44:58.632703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:44:58.632703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:46:09.773272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":350,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.159856+0000","last_change":"2024-09-20T22:45:00.680376+0000","last_active":"2024-09-20T22:45:08.159856+0000","last_peered":"2024-09-20T22:45:08.159856+0000","last_clean":"2024-09-20T22:45:08.159856+0000","last_became_active":"2024-09-20T22:43:57.419908+0000","last_became_peered":"2024-09-20T22:43:57.419908+0000","last_unstale":"2024-09-20T22:45:08.159856+0000","last_undegraded":"2024-09-20T22:45:08.159856+0000","last_fullsized":"2024-09-20T22:45:08.159856+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:45:00.680307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:45:00.680307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:46:24.048175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":338,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.156213+0000","last_change":"2024-09-20T22:43:57.425704+0000","last_active":"2024-09-20T22:45:08.156213+0000","last_peered":"2024-09-20T22:45:08.156213+0000","last_clean":"2024-09-20T22:45:08.156213+0000","last_became_active":"2024-09-20T22:43:57.424654+0000","last_became_peered":"2024-09-20T22:43:57.424654+0000","last_unstale":"2024-09-20T22:45:08.156213+0000","last_undegraded":"2024-09-20T22:45:08.156213+0000","last_fullsized":"2024-09-20T22:45:08.156213+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:10.233487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:10.233487+0000","last_clean_scrub_stamp":"2024-09-20T22:43:10.233487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:10.284238+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":338,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.198838+0000","last_change":"2024-09-20T22:44:21.123125+0000","last_active":"2024-09-20T22:45:08.198838+0000","last_peered":"2024-09-20T22:45:08.198838+0000","last_clean":"2024-09-20T22:45:08.198838+0000","last_became_active":"2024-09-20T22:43:57.430933+0000","last_became_peered":"2024-09-20T22:43:57.430933+0000","last_unstale":"2024-09-20T22:45:08.198838+0000","last_undegraded":"2024-09-20T22:45:08.198838+0000","last_fullsized":"2024-09-20T22:45:08.198838+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:44:21.123026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:44:21.123026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:36.736823+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":358,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.144040+0000","last_change":"2024-09-20T22:44:59.221701+0000","last_active":"2024-09-20T22:45:08.144040+0000","last_peered":"2024-09-20T22:45:08.144040+0000","last_clean":"2024-09-20T22:45:08.144040+0000","last_became_active":"2024-09-20T22:44:59.221296+0000","last_became_peered":"2024-09-20T22:44:59.221296+0000","last_unstale":"2024-09-20T22:45:08.144040+0000","last_undegraded":"2024-09-20T22:45:08.144040+0000","last_fullsized":"2024-09-20T22:45:08.144040+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:44:10.267152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:44:10.267152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:33.915070+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"74'11","reported_seq":362,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.161727+0000","last_change":"2024-09-20T22:45:03.155719+0000","last_active":"2024-09-20T22:45:08.161727+0000","last_peered":"2024-09-20T22:45:08.161727+0000","last_clean":"2024-09-20T22:45:08.161727+0000","last_became_active":"2024-09-20T22:41:40.171256+0000","last_became_peered":"2024-09-20T22:41:40.171256+0000","last_unstale":"2024-09-20T22:45:08.161727+0000","last_undegraded":"2024-09-20T22:45:08.161727+0000","last_fullsized":"2024-09-20T22:45:08.161727+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-09-20T22:43:28.499961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:43:28.499961+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":336,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-09-20T22:45:08.145395+0000","last_change":"2024-09-20T22:43:51.667414+0000","last_active":"2024-09-20T22:45:08.145395+0000","last_peered":"2024-09-20T22:45:08.145395+0000","last_clean":"2024-09-20T22:45:08.145395+0000","last_became_active":"2024-09-20T22:41:42.178157+0000","last_became_peered":"2024-09-20T22:41:42.178157+0000","last_unstale":"2024-09-20T22:45:08.145395+0000","last_undegraded":"2024-09-20T22:45:08.145395+0000","last_fullsized":"2024-09-20T22:45:08.145395+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:51.667341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:51.667341+0000","last_clean_scrub_stamp":"2024-09-20T22:43:51.667341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":26,"stat_sum":{"num_bytes":118769747,"num_objects":52,"num_object_clones":15,"num_object_copies":156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":3,"num_read":1063,"num_read_kb":1088985,"num_write":736,"num_write_kb":292105,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":5835903,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":306061312,"data_stored":356628752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2472,"ondisk_log_size":2472,"up":78,"acting":78,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1261,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":69,"ondisk_log_size":69,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":686,"num_read_kb":443,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":1914,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":471040,"data_stored":12580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":427,"ondisk_log_size":427,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":5023,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":205,"num_read_kb":200,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2688,"ondisk_log_size":2688,"up":63,"acting":63,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070510,"num_pgs":78,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":81988,"kb_used_data":46632,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93663164,"statfs":{"total":95995035648,"available":95911079936,"internally_reserved":0,"allocated":47751168,"data_stored":52801409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430260,"num_pgs":75,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":88852,"kb_used_data":49336,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93656300,"statfs":{"total":95995035648,"available":95904051200,"internally_reserved":0,"allocated":50520064,"data_stored":56917335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691897,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":70888,"kb_used_data":39692,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93674264,"statfs":{"total":95995035648,"available":95922446336,"internally_reserved":0,"allocated":40644608,"data_stored":48546766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":0,"seq":133143986238,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182468,"num_pgs":83,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":87852,"kb_used_data":52496,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93657300,"statfs":{"total":95995035648,"available":95905075200,"internally_reserved":0,"allocated":53755904,"data_stored":57431257,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411401,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":79780,"kb_used_data":44428,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93665372,"statfs":{"total":95995035648,"available":95913340928,"internally_reserved":0,"allocated":45494272,"data_stored":56007810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574926,"num_pgs":79,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":62968,"kb_used_data":27620,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93682184,"statfs":{"total":95995035648,"available":95930556416,"internally_reserved":0,"allocated":28282880,"data_stored":26974961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738451,"num_pgs":78,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":80284,"kb_used_data":40768,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93664868,"statfs":{"total":95995035648,"available":95912824832,"internally_reserved":0,"allocated":41746432,"data_stored":46408163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40239104,"data_stored":46413243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":26755072,"data_stored":26620430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":42704896,"data_stored":54359603,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":51970048,"data_stored":61763804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11284480,"data_stored":11212109,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":37851136,"data_stored":46897672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":50040832,"data_stored":57946121,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45215744,"data_stored":51415770,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-20T22:45:13.722 INFO:tasks.thrashosds.thrasher:Setting 7.17 to [7, 0, 4] 2024-09-20T22:45:13.722 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '7.17', '7', '0', '4'] 2024-09-20T22:45:13.723 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd pg-upmap 7.17 7 0 4 2024-09-20T22:45:13.738 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:45:13.764 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:13.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:13.765 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:45:13.865 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:45:13.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:13.887 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:13.887 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:45:13.988 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:45:14.014 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:14.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:14.015 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:45:14.115 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:45:14.136 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:14.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:14.137 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:45:14.152 INFO:tasks.rados.rados.0.smithi084.stdout:724: done (2 left) 2024-09-20T22:45:14.152 INFO:tasks.rados.rados.0.smithi084.stdout:725: rollback oid 39 current snap is 71 2024-09-20T22:45:14.152 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 39 to 69 2024-09-20T22:45:14.152 INFO:tasks.rados.rados.0.smithi084.stdout:726: write oid 1 current snap is 71 2024-09-20T22:45:14.152 INFO:tasks.rados.rados.0.smithi084.stdout:726: seq_num 284 ranges {773188=475777,1969239=724762} 2024-09-20T22:45:14.159 INFO:tasks.rados.rados.0.smithi084.stdout:726: writing smithi08477203-1 from 773188 to 1248965 tid 1 2024-09-20T22:45:14.169 INFO:tasks.rados.rados.0.smithi084.stdout:726: writing smithi08477203-1 from 1969239 to 2694001 tid 2 2024-09-20T22:45:14.169 INFO:tasks.rados.rados.0.smithi084.stdout:727: write (excl) oid 37 current snap is 71 2024-09-20T22:45:14.169 INFO:tasks.rados.rados.0.smithi084.stdout:720: finishing write tid 1 to smithi08477203-23 2024-09-20T22:45:14.169 INFO:tasks.rados.rados.0.smithi084.stdout:720: finishing write tid 2 to smithi08477203-23 2024-09-20T22:45:14.169 INFO:tasks.rados.rados.0.smithi084.stdout:720: finishing write tid 3 to smithi08477203-23 2024-09-20T22:45:14.169 INFO:tasks.rados.rados.0.smithi084.stdout:720: finishing write tid 4 to smithi08477203-23 2024-09-20T22:45:14.169 INFO:tasks.rados.rados.0.smithi084.stdout:720: oid 23 updating version 0 to 152 2024-09-20T22:45:14.169 INFO:tasks.rados.rados.0.smithi084.stdout:720: oid 23 version 152 is already newer than 151 2024-09-20T22:45:14.170 INFO:tasks.rados.rados.0.smithi084.stdout:720: oid 23 updating version 152 to 153 2024-09-20T22:45:14.170 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 23 v 153 (ObjNum 282 snap 71 seq_num 282) dirty exists 2024-09-20T22:45:14.170 INFO:tasks.rados.rados.0.smithi084.stdout:720: left oid 23 (ObjNum 282 snap 71 seq_num 282) 2024-09-20T22:45:14.170 INFO:tasks.rados.rados.0.smithi084.stdout:727: seq_num 285 ranges {789995=551026,2027721=468420} 2024-09-20T22:45:14.174 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is unset 2024-09-20T22:45:14.176 INFO:tasks.rados.rados.0.smithi084.stdout:727: writing smithi08477203-37 from 789995 to 1341021 tid 1 2024-09-20T22:45:14.183 INFO:tasks.rados.rados.0.smithi084.stdout:727: writing smithi08477203-37 from 2027721 to 2496141 tid 2 2024-09-20T22:45:14.183 INFO:tasks.rados.rados.0.smithi084.stdout:720: done (4 left) 2024-09-20T22:45:14.183 INFO:tasks.rados.rados.0.smithi084.stdout:728: delete oid 50 current snap is 71 2024-09-20T22:45:14.183 INFO:tasks.rados.rados.0.smithi084.stdout:722: expect (ObjNum 273 snap 69 seq_num 273) 2024-09-20T22:45:14.238 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:45:14.259 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:14.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:14.259 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:45:14.302 INFO:tasks.rados.rados.0.smithi084.stdout:725: finishing rollback tid 0 to smithi08477203-39 2024-09-20T22:45:14.303 INFO:tasks.rados.rados.0.smithi084.stdout:725: finishing rollback tid 1 to smithi08477203-39 2024-09-20T22:45:14.303 INFO:tasks.rados.rados.0.smithi084.stdout:725: finishing rollback tid 2 to smithi08477203-39 2024-09-20T22:45:14.303 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 39 v 154 (ObjNum 245 snap 54 seq_num 245) dirty exists 2024-09-20T22:45:14.303 INFO:tasks.rados.rados.0.smithi084.stdout:726: finishing write tid 1 to smithi08477203-1 2024-09-20T22:45:14.303 INFO:tasks.rados.rados.0.smithi084.stdout:726: finishing write tid 2 to smithi08477203-1 2024-09-20T22:45:14.303 INFO:tasks.rados.rados.0.smithi084.stdout:726: finishing write tid 3 to smithi08477203-1 2024-09-20T22:45:14.303 INFO:tasks.rados.rados.0.smithi084.stdout:726: finishing write tid 4 to smithi08477203-1 2024-09-20T22:45:14.303 INFO:tasks.rados.rados.0.smithi084.stdout:726: oid 1 updating version 0 to 157 2024-09-20T22:45:14.303 INFO:tasks.rados.rados.0.smithi084.stdout:726: oid 1 version 157 is already newer than 156 2024-09-20T22:45:14.303 INFO:tasks.rados.rados.0.smithi084.stdout:726: oid 1 updating version 157 to 158 2024-09-20T22:45:14.303 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 1 v 158 (ObjNum 284 snap 71 seq_num 284) dirty exists 2024-09-20T22:45:14.303 INFO:tasks.rados.rados.0.smithi084.stdout:726: left oid 1 (ObjNum 284 snap 71 seq_num 284) 2024-09-20T22:45:14.303 INFO:tasks.rados.rados.0.smithi084.stdout:727: finishing write tid 1 to smithi08477203-37 2024-09-20T22:45:14.304 INFO:tasks.rados.rados.0.smithi084.stdout:727: finishing write tid 2 to smithi08477203-37 2024-09-20T22:45:14.304 INFO:tasks.rados.rados.0.smithi084.stdout:727: finishing write tid 3 to smithi08477203-37 2024-09-20T22:45:14.304 INFO:tasks.rados.rados.0.smithi084.stdout:722: done (4 left) 2024-09-20T22:45:14.304 INFO:tasks.rados.rados.0.smithi084.stdout:725: done (3 left) 2024-09-20T22:45:14.304 INFO:tasks.rados.rados.0.smithi084.stdout:726: done (2 left) 2024-09-20T22:45:14.304 INFO:tasks.rados.rados.0.smithi084.stdout:728: done (1 left) 2024-09-20T22:45:14.304 INFO:tasks.rados.rados.0.smithi084.stdout:729: write (excl) oid 49 current snap is 71 2024-09-20T22:45:14.304 INFO:tasks.rados.rados.0.smithi084.stdout:727: finishing write tid 4 to smithi08477203-37 2024-09-20T22:45:14.304 INFO:tasks.rados.rados.0.smithi084.stdout:727: oid 37 updating version 0 to 141 2024-09-20T22:45:14.304 INFO:tasks.rados.rados.0.smithi084.stdout:727: oid 37 updating version 141 to 143 2024-09-20T22:45:14.304 INFO:tasks.rados.rados.0.smithi084.stdout:727: oid 37 version 143 is already newer than 142 2024-09-20T22:45:14.304 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 37 v 143 (ObjNum 285 snap 71 seq_num 285) dirty exists 2024-09-20T22:45:14.304 INFO:tasks.rados.rados.0.smithi084.stdout:727: left oid 37 (ObjNum 285 snap 71 seq_num 285) 2024-09-20T22:45:14.304 INFO:tasks.rados.rados.0.smithi084.stdout:729: seq_num 287 ranges {423609=701524,1628332=472089,2648987=174622} 2024-09-20T22:45:14.308 INFO:tasks.rados.rados.0.smithi084.stdout:729: writing smithi08477203-49 from 423609 to 1125133 tid 1 2024-09-20T22:45:14.313 INFO:tasks.rados.rados.0.smithi084.stdout:729: writing smithi08477203-49 from 1628332 to 2100421 tid 2 2024-09-20T22:45:14.318 INFO:tasks.rados.rados.0.smithi084.stdout:729: writing smithi08477203-49 from 2648987 to 2823609 tid 3 2024-09-20T22:45:14.318 INFO:tasks.rados.rados.0.smithi084.stdout:727: done (1 left) 2024-09-20T22:45:14.318 INFO:tasks.rados.rados.0.smithi084.stdout:730: read oid 4 snap 62 2024-09-20T22:45:14.318 INFO:tasks.rados.rados.0.smithi084.stdout:730: expect (ObjNum 241 snap 54 seq_num 241) 2024-09-20T22:45:14.318 INFO:tasks.rados.rados.0.smithi084.stdout:731: read oid 35 snap -1 2024-09-20T22:45:14.318 INFO:tasks.rados.rados.0.smithi084.stdout:731: expect (ObjNum 150 snap 30 seq_num 150) 2024-09-20T22:45:14.318 INFO:tasks.rados.rados.0.smithi084.stdout:732: snap_create 2024-09-20T22:45:14.319 INFO:tasks.rados.rados.0.smithi084.stdout:729: finishing write tid 1 to smithi08477203-49 2024-09-20T22:45:14.319 INFO:tasks.rados.rados.0.smithi084.stdout:729: finishing write tid 2 to smithi08477203-49 2024-09-20T22:45:14.320 INFO:tasks.rados.rados.0.smithi084.stdout:731: expect (ObjNum 150 snap 30 seq_num 150) 2024-09-20T22:45:14.361 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:45:14.385 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:14.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:14.386 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:45:14.453 INFO:tasks.rados.rados.0.smithi084.stdout:730: expect (ObjNum 241 snap 54 seq_num 241) 2024-09-20T22:45:14.487 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:45:14.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:14 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/956138767' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:45:14.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:14 smithi154 ceph-mon[33264]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:45:14.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:14 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/956138767' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T22:45:14.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:14 smithi154 ceph-mon[33264]: osdmap e235: 8 total, 8 up, 7 in 2024-09-20T22:45:14.508 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:14.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:14.509 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:45:14.510 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:14 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/956138767' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:45:14.510 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:14 smithi084 ceph-mon[40024]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:45:14.510 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:14 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/956138767' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T22:45:14.510 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:14 smithi084 ceph-mon[40024]: osdmap e235: 8 total, 8 up, 7 in 2024-09-20T22:45:14.510 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:14 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/956138767' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:45:14.510 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:14 smithi084 ceph-mon[29774]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:45:14.510 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:14 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/956138767' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T22:45:14.510 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:14 smithi084 ceph-mon[29774]: osdmap e235: 8 total, 8 up, 7 in 2024-09-20T22:45:14.556 INFO:tasks.rados.rados.0.smithi084.stdout:729: finishing write tid 3 to smithi08477203-49 2024-09-20T22:45:14.557 INFO:tasks.rados.rados.0.smithi084.stdout:729: finishing write tid 4 to smithi08477203-49 2024-09-20T22:45:14.557 INFO:tasks.rados.rados.0.smithi084.stdout:729: finishing write tid 5 to smithi08477203-49 2024-09-20T22:45:14.557 INFO:tasks.rados.rados.0.smithi084.stdout:729: oid 49 updating version 0 to 219 2024-09-20T22:45:14.557 INFO:tasks.rados.rados.0.smithi084.stdout:729: oid 49 updating version 219 to 221 2024-09-20T22:45:14.557 INFO:tasks.rados.rados.0.smithi084.stdout:729: oid 49 version 221 is already newer than 220 2024-09-20T22:45:14.557 INFO:tasks.rados.rados.0.smithi084.stdout:729: oid 49 version 221 is already newer than 218 2024-09-20T22:45:14.557 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 49 v 221 (ObjNum 287 snap 71 seq_num 287) dirty exists 2024-09-20T22:45:14.557 INFO:tasks.rados.rados.0.smithi084.stdout:729: left oid 49 (ObjNum 287 snap 71 seq_num 287) 2024-09-20T22:45:14.610 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:45:14.611 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-20T22:45:14.611 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:45:14.634 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:14.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:14.635 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:45:14.736 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:45:14.758 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:14.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:14.759 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:45:14.860 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:45:14.881 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:14.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:14.882 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:45:14.983 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:45:15.009 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:15.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:15.010 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:45:15.111 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:45:15.136 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:15.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:15.137 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:45:15.202 INFO:tasks.rados.rados.0.smithi084.stdout:729: done (3 left) 2024-09-20T22:45:15.202 INFO:tasks.rados.rados.0.smithi084.stdout:730: done (2 left) 2024-09-20T22:45:15.202 INFO:tasks.rados.rados.0.smithi084.stdout:731: done (1 left) 2024-09-20T22:45:15.202 INFO:tasks.rados.rados.0.smithi084.stdout:732: done (0 left) 2024-09-20T22:45:15.202 INFO:tasks.rados.rados.0.smithi084.stdout:733: write (excl) oid 30 current snap is 72 2024-09-20T22:45:15.202 INFO:tasks.rados.rados.0.smithi084.stdout:733: seq_num 288 ranges {440416=776773,1686814=732100} 2024-09-20T22:45:15.208 INFO:tasks.rados.rados.0.smithi084.stdout:733: writing smithi08477203-30 from 440416 to 1217189 tid 1 2024-09-20T22:45:15.216 INFO:tasks.rados.rados.0.smithi084.stdout:733: writing smithi08477203-30 from 1686814 to 2418914 tid 2 2024-09-20T22:45:15.216 INFO:tasks.rados.rados.0.smithi084.stdout:734: write (excl) oid 3 current snap is 72 2024-09-20T22:45:15.216 INFO:tasks.rados.rados.0.smithi084.stdout:733: finishing write tid 1 to smithi08477203-30 2024-09-20T22:45:15.216 INFO:tasks.rados.rados.0.smithi084.stdout:734: seq_num 289 ranges {457223=568375,1461649=475758,2420186=437037} 2024-09-20T22:45:15.221 INFO:tasks.rados.rados.0.smithi084.stdout:734: writing smithi08477203-3 from 457223 to 1025598 tid 1 2024-09-20T22:45:15.226 INFO:tasks.rados.rados.0.smithi084.stdout:734: writing smithi08477203-3 from 1461649 to 1937407 tid 2 2024-09-20T22:45:15.232 INFO:tasks.rados.rados.0.smithi084.stdout:734: writing smithi08477203-3 from 2420186 to 2857223 tid 3 2024-09-20T22:45:15.232 INFO:tasks.rados.rados.0.smithi084.stdout:735: delete oid 10 current snap is 72 2024-09-20T22:45:15.232 INFO:tasks.rados.rados.0.smithi084.stdout:733: finishing write tid 2 to smithi08477203-30 2024-09-20T22:45:15.232 INFO:tasks.rados.rados.0.smithi084.stdout:733: finishing write tid 3 to smithi08477203-30 2024-09-20T22:45:15.232 INFO:tasks.rados.rados.0.smithi084.stdout:733: finishing write tid 4 to smithi08477203-30 2024-09-20T22:45:15.232 INFO:tasks.rados.rados.0.smithi084.stdout:733: oid 30 updating version 0 to 143 2024-09-20T22:45:15.232 INFO:tasks.rados.rados.0.smithi084.stdout:733: oid 30 updating version 143 to 144 2024-09-20T22:45:15.232 INFO:tasks.rados.rados.0.smithi084.stdout:733: oid 30 version 144 is already newer than 142 2024-09-20T22:45:15.232 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 30 v 144 (ObjNum 288 snap 72 seq_num 288) dirty exists 2024-09-20T22:45:15.232 INFO:tasks.rados.rados.0.smithi084.stdout:733: left oid 30 (ObjNum 288 snap 72 seq_num 288) 2024-09-20T22:45:15.233 INFO:tasks.rados.rados.0.smithi084.stdout:734: finishing write tid 1 to smithi08477203-3 2024-09-20T22:45:15.233 INFO:tasks.rados.rados.0.smithi084.stdout:734: finishing write tid 2 to smithi08477203-3 2024-09-20T22:45:15.235 INFO:tasks.rados.rados.0.smithi084.stdout:733: done (2 left) 2024-09-20T22:45:15.235 INFO:tasks.rados.rados.0.smithi084.stdout:735: done (1 left) 2024-09-20T22:45:15.235 INFO:tasks.rados.rados.0.smithi084.stdout:736: read oid 17 snap -1 2024-09-20T22:45:15.235 INFO:tasks.rados.rados.0.smithi084.stdout:736: expect (ObjNum 203 snap 41 seq_num 203) 2024-09-20T22:45:15.235 INFO:tasks.rados.rados.0.smithi084.stdout:737: copy_from oid 50 from oid 42 current snap is 72 2024-09-20T22:45:15.235 INFO:tasks.rados.rados.0.smithi084.stdout:738: rollback oid 24 current snap is 72 2024-09-20T22:45:15.235 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 24 to 64 2024-09-20T22:45:15.235 INFO:tasks.rados.rados.0.smithi084.stdout:739: read oid 27 snap -1 2024-09-20T22:45:15.235 INFO:tasks.rados.rados.0.smithi084.stdout:739: expect (ObjNum 234 snap 51 seq_num 234) 2024-09-20T22:45:15.236 INFO:tasks.rados.rados.0.smithi084.stdout:740: snap_create 2024-09-20T22:45:15.236 INFO:tasks.rados.rados.0.smithi084.stdout:734: finishing write tid 3 to smithi08477203-3 2024-09-20T22:45:15.236 INFO:tasks.rados.rados.0.smithi084.stdout:734: finishing write tid 4 to smithi08477203-3 2024-09-20T22:45:15.236 INFO:tasks.rados.rados.0.smithi084.stdout:734: finishing write tid 5 to smithi08477203-3 2024-09-20T22:45:15.236 INFO:tasks.rados.rados.0.smithi084.stdout:734: oid 3 updating version 0 to 109 2024-09-20T22:45:15.236 INFO:tasks.rados.rados.0.smithi084.stdout:734: oid 3 updating version 109 to 110 2024-09-20T22:45:15.236 INFO:tasks.rados.rados.0.smithi084.stdout:734: oid 3 updating version 110 to 111 2024-09-20T22:45:15.236 INFO:tasks.rados.rados.0.smithi084.stdout:734: oid 3 updating version 111 to 112 2024-09-20T22:45:15.236 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 3 v 112 (ObjNum 289 snap 72 seq_num 289) dirty exists 2024-09-20T22:45:15.236 INFO:tasks.rados.rados.0.smithi084.stdout:734: left oid 3 (ObjNum 289 snap 72 seq_num 289) 2024-09-20T22:45:15.237 INFO:tasks.rados.rados.0.smithi084.stdout:737: finishing copy_from to smithi08477203-50 2024-09-20T22:45:15.237 INFO:tasks.rados.rados.0.smithi084.stdout:737: got expected ENOENT (src dne) 2024-09-20T22:45:15.237 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:45:15.238 INFO:tasks.rados.rados.0.smithi084.stdout:738: finishing rollback tid 0 to smithi08477203-24 2024-09-20T22:45:15.239 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:45:15.241 INFO:tasks.rados.rados.0.smithi084.stdout:739: expect (ObjNum 234 snap 51 seq_num 234) 2024-09-20T22:45:15.261 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:15.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:15.262 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:45:15.332 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:15 smithi084 ceph-mon[40024]: pgmap v253: 178 pgs: 1 active+clean+snaptrim, 177 active+clean; 128 MiB data, 574 MiB used, 625 GiB / 626 GiB avail; 15 MiB/s rd, 2.3 MiB/s wr, 24 op/s 2024-09-20T22:45:15.332 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:15 smithi084 ceph-mon[40024]: 7.15 scrub starts 2024-09-20T22:45:15.332 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:15 smithi084 ceph-mon[40024]: 7.15 scrub ok 2024-09-20T22:45:15.332 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:15 smithi084 ceph-mon[40024]: 3.1a scrub starts 2024-09-20T22:45:15.332 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:15 smithi084 ceph-mon[40024]: 3.1a scrub ok 2024-09-20T22:45:15.332 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:15 smithi084 ceph-mon[29774]: pgmap v253: 178 pgs: 1 active+clean+snaptrim, 177 active+clean; 128 MiB data, 574 MiB used, 625 GiB / 626 GiB avail; 15 MiB/s rd, 2.3 MiB/s wr, 24 op/s 2024-09-20T22:45:15.333 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:15 smithi084 ceph-mon[29774]: 7.15 scrub starts 2024-09-20T22:45:15.333 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:15 smithi084 ceph-mon[29774]: 7.15 scrub ok 2024-09-20T22:45:15.333 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:15 smithi084 ceph-mon[29774]: 3.1a scrub starts 2024-09-20T22:45:15.333 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:15 smithi084 ceph-mon[29774]: 3.1a scrub ok 2024-09-20T22:45:15.363 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:45:15.388 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:15.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:15.389 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:45:15.490 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:45:15.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:15 smithi154 ceph-mon[33264]: pgmap v253: 178 pgs: 1 active+clean+snaptrim, 177 active+clean; 128 MiB data, 574 MiB used, 625 GiB / 626 GiB avail; 15 MiB/s rd, 2.3 MiB/s wr, 24 op/s 2024-09-20T22:45:15.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:15 smithi154 ceph-mon[33264]: 7.15 scrub starts 2024-09-20T22:45:15.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:15 smithi154 ceph-mon[33264]: 7.15 scrub ok 2024-09-20T22:45:15.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:15 smithi154 ceph-mon[33264]: 3.1a scrub starts 2024-09-20T22:45:15.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:15 smithi154 ceph-mon[33264]: 3.1a scrub ok 2024-09-20T22:45:15.510 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:15.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:15.511 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:45:15.538 INFO:tasks.rados.rados.0.smithi084.stdout:737: finishing copy_from racing read to smithi08477203-50 2024-09-20T22:45:15.538 INFO:tasks.rados.rados.0.smithi084.stdout:738: finishing rollback tid 1 to smithi08477203-24 2024-09-20T22:45:15.538 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 24 v 81 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-09-20T22:45:15.538 INFO:tasks.rados.rados.0.smithi084.stdout:736: expect (ObjNum 203 snap 41 seq_num 203) 2024-09-20T22:45:15.613 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:45:15.637 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:15.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:15.638 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:45:15.739 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:45:15.763 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:15.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:15.764 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:45:15.817 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:45:15.844 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.2 dump_historic_ops 2024-09-20T22:45:15.865 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:45:15.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:15.888 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:15.888 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:45:15.989 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:45:16.010 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:16.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:16.011 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:45:16.112 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:45:16.136 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:16.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:16.137 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:45:16.208 INFO:tasks.rados.rados.0.smithi084.stdout:734: done (5 left) 2024-09-20T22:45:16.208 INFO:tasks.rados.rados.0.smithi084.stdout:736: done (4 left) 2024-09-20T22:45:16.208 INFO:tasks.rados.rados.0.smithi084.stdout:737: done (3 left) 2024-09-20T22:45:16.208 INFO:tasks.rados.rados.0.smithi084.stdout:738: done (2 left) 2024-09-20T22:45:16.208 INFO:tasks.rados.rados.0.smithi084.stdout:739: done (1 left) 2024-09-20T22:45:16.208 INFO:tasks.rados.rados.0.smithi084.stdout:740: done (0 left) 2024-09-20T22:45:16.208 INFO:tasks.rados.rados.0.smithi084.stdout:741: copy_from oid 6 from oid 24 current snap is 73 2024-09-20T22:45:16.208 INFO:tasks.rados.rados.0.smithi084.stdout:742: snap_create 2024-09-20T22:45:16.214 INFO:tasks.rados.rados.0.smithi084.stdout:741: finishing copy_from to smithi08477203-6 2024-09-20T22:45:16.214 INFO:tasks.rados.rados.0.smithi084.stdout:741: got expected ENOENT (src dne) 2024-09-20T22:45:16.214 INFO:tasks.rados.rados.0.smithi084.stdout:741: finishing copy_from racing read to smithi08477203-6 2024-09-20T22:45:16.238 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:45:16.260 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:16.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:16.260 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:45:16.343 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:16 smithi084 ceph-mon[40024]: 4.1f scrub starts 2024-09-20T22:45:16.343 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:16 smithi084 ceph-mon[40024]: 4.1f scrub ok 2024-09-20T22:45:16.343 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:16 smithi084 ceph-mon[40024]: 7.4 scrub starts 2024-09-20T22:45:16.343 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:16 smithi084 ceph-mon[40024]: 7.4 scrub ok 2024-09-20T22:45:16.343 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:16 smithi084 ceph-mon[40024]: osdmap e236: 8 total, 8 up, 7 in 2024-09-20T22:45:16.343 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:16 smithi084 ceph-mon[29774]: 4.1f scrub starts 2024-09-20T22:45:16.343 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:16 smithi084 ceph-mon[29774]: 4.1f scrub ok 2024-09-20T22:45:16.343 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:16 smithi084 ceph-mon[29774]: 7.4 scrub starts 2024-09-20T22:45:16.343 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:16 smithi084 ceph-mon[29774]: 7.4 scrub ok 2024-09-20T22:45:16.343 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:16 smithi084 ceph-mon[29774]: osdmap e236: 8 total, 8 up, 7 in 2024-09-20T22:45:16.362 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:45:16.388 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:16.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:16.389 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:45:16.490 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:45:16.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:16 smithi154 ceph-mon[33264]: 4.1f scrub starts 2024-09-20T22:45:16.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:16 smithi154 ceph-mon[33264]: 4.1f scrub ok 2024-09-20T22:45:16.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:16 smithi154 ceph-mon[33264]: 7.4 scrub starts 2024-09-20T22:45:16.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:16 smithi154 ceph-mon[33264]: 7.4 scrub ok 2024-09-20T22:45:16.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:16 smithi154 ceph-mon[33264]: osdmap e236: 8 total, 8 up, 7 in 2024-09-20T22:45:16.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:16.511 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:16.512 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:45:16.542 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:45:16.542 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:45:16.542 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:45:16.542 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-20T22:45:16.542 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:45:16.542 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:45:16.542 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:45:16.543 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:45:16.543 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T22:45:16.543 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:45:16.543 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:45:16.543 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:45:16.543 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:45:16.543 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:45:16.543 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:45:16.544 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:45:16.612 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:45:16.633 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:16.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:16.634 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:45:16.735 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:45:16.755 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:16.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:16.756 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:45:16.857 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:45:16.879 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:16.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:16.880 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:45:16.886 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:45:16.980 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:45:17.002 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:17.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:17.003 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:45:17.008 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-20T22:45:17.103 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:45:17.125 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:17.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:17.126 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:45:17.204 INFO:teuthology.orchestra.run.smithi084.stderr:set 7.17 pg_upmap mapping to [7,0,4] 2024-09-20T22:45:17.211 INFO:tasks.rados.rados.0.smithi084.stdout:741: done (1 left) 2024-09-20T22:45:17.211 INFO:tasks.rados.rados.0.smithi084.stdout:742: done (0 left) 2024-09-20T22:45:17.211 INFO:tasks.rados.rados.0.smithi084.stdout:743: write (excl) oid 13 current snap is 74 2024-09-20T22:45:17.212 INFO:tasks.rados.rados.0.smithi084.stdout:743: seq_num 291 ranges {490837=718873,1862260=479427,2875032=15805} 2024-09-20T22:45:17.217 INFO:tasks.rados.rados.0.smithi084.stdout:743: writing smithi08477203-13 from 490837 to 1209710 tid 1 2024-09-20T22:45:17.224 INFO:tasks.rados.rados.0.smithi084.stdout:743: writing smithi08477203-13 from 1862260 to 2341687 tid 2 2024-09-20T22:45:17.226 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:45:17.227 INFO:tasks.rados.rados.0.smithi084.stdout:743: writing smithi08477203-13 from 2875032 to 2890837 tid 3 2024-09-20T22:45:17.227 INFO:tasks.rados.rados.0.smithi084.stdout:744: rollback oid 12 current snap is 74 2024-09-20T22:45:17.228 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 12 to 64 2024-09-20T22:45:17.228 INFO:tasks.rados.rados.0.smithi084.stdout:743: finishing write tid 1 to smithi08477203-13 2024-09-20T22:45:17.228 INFO:tasks.rados.rados.0.smithi084.stdout:743: finishing write tid 2 to smithi08477203-13 2024-09-20T22:45:17.228 INFO:tasks.rados.rados.0.smithi084.stdout:745: delete oid 49 current snap is 74 2024-09-20T22:45:17.230 INFO:tasks.rados.rados.0.smithi084.stdout:745: done (2 left) 2024-09-20T22:45:17.230 INFO:tasks.rados.rados.0.smithi084.stdout:746: delete oid 50 current snap is 74 2024-09-20T22:45:17.231 INFO:tasks.rados.rados.0.smithi084.stdout:743: finishing write tid 3 to smithi08477203-13 2024-09-20T22:45:17.231 INFO:tasks.rados.rados.0.smithi084.stdout:744: finishing rollback tid 0 to smithi08477203-12 2024-09-20T22:45:17.232 INFO:tasks.rados.rados.0.smithi084.stdout:744: finishing rollback tid 1 to smithi08477203-12 2024-09-20T22:45:17.232 INFO:tasks.rados.rados.0.smithi084.stdout:743: finishing write tid 4 to smithi08477203-13 2024-09-20T22:45:17.232 INFO:tasks.rados.rados.0.smithi084.stdout:743: finishing write tid 5 to smithi08477203-13 2024-09-20T22:45:17.232 INFO:tasks.rados.rados.0.smithi084.stdout:743: oid 13 updating version 0 to 334 2024-09-20T22:45:17.232 INFO:tasks.rados.rados.0.smithi084.stdout:743: oid 13 updating version 334 to 336 2024-09-20T22:45:17.232 INFO:tasks.rados.rados.0.smithi084.stdout:743: oid 13 version 336 is already newer than 335 2024-09-20T22:45:17.232 INFO:tasks.rados.rados.0.smithi084.stdout:743: oid 13 updating version 336 to 337 2024-09-20T22:45:17.232 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 13 v 337 (ObjNum 291 snap 74 seq_num 291) dirty exists 2024-09-20T22:45:17.232 INFO:tasks.rados.rados.0.smithi084.stdout:743: left oid 13 (ObjNum 291 snap 74 seq_num 291) 2024-09-20T22:45:17.232 INFO:tasks.rados.rados.0.smithi084.stdout:744: finishing rollback tid 2 to smithi08477203-12 2024-09-20T22:45:17.232 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 12 v 182 (ObjNum 164 snap 35 seq_num 164) dirty exists 2024-09-20T22:45:17.232 INFO:tasks.rados.rados.0.smithi084.stdout:743: done (2 left) 2024-09-20T22:45:17.232 INFO:tasks.rados.rados.0.smithi084.stdout:744: done (1 left) 2024-09-20T22:45:17.233 INFO:tasks.rados.rados.0.smithi084.stdout:746: done (0 left) 2024-09-20T22:45:17.233 INFO:tasks.rados.rados.0.smithi084.stdout:747: read oid 5 snap -1 2024-09-20T22:45:17.233 INFO:tasks.rados.rados.0.smithi084.stdout:747: expect (ObjNum 50 snap 0 seq_num 50) 2024-09-20T22:45:17.233 INFO:tasks.rados.rados.0.smithi084.stdout:748: read oid 48 snap 67 2024-09-20T22:45:17.233 INFO:tasks.rados.rados.0.smithi084.stdout:748: expect (ObjNum 257 snap 60 seq_num 257) 2024-09-20T22:45:17.233 INFO:tasks.rados.rados.0.smithi084.stdout:749: write oid 41 current snap is 74 2024-09-20T22:45:17.233 INFO:tasks.rados.rados.0.smithi084.stdout:749: seq_num 294 ranges {541258=544620,1637706=626754,2840948=100310} 2024-09-20T22:45:17.238 INFO:tasks.rados.rados.0.smithi084.stdout:749: writing smithi08477203-41 from 541258 to 1085878 tid 1 2024-09-20T22:45:17.245 INFO:tasks.rados.rados.0.smithi084.stdout:749: writing smithi08477203-41 from 1637706 to 2264460 tid 2 2024-09-20T22:45:17.250 INFO:tasks.rados.rados.0.smithi084.stdout:749: writing smithi08477203-41 from 2840948 to 2941258 tid 3 2024-09-20T22:45:17.250 INFO:tasks.rados.rados.0.smithi084.stdout:750: snap_remove snap 64 2024-09-20T22:45:17.252 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:17.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:17.254 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:45:17.354 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:45:17.364 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:17 smithi084 ceph-mon[29774]: pgmap v256: 178 pgs: 1 active+clean+snaptrim, 177 active+clean; 128 MiB data, 574 MiB used, 625 GiB / 626 GiB avail; 15 MiB/s rd, 2.3 MiB/s wr, 24 op/s 2024-09-20T22:45:17.364 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:17 smithi084 ceph-mon[29774]: 7.12 scrub starts 2024-09-20T22:45:17.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:17 smithi084 ceph-mon[29774]: 7.12 scrub ok 2024-09-20T22:45:17.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:17 smithi084 ceph-mon[29774]: osdmap e237: 8 total, 8 up, 7 in 2024-09-20T22:45:17.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:17 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/844174825' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:45:17.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:17 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/4145173149' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "7.17", "id": [7, 0, 4]}]: dispatch 2024-09-20T22:45:17.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:17 smithi084 ceph-mon[40024]: pgmap v256: 178 pgs: 1 active+clean+snaptrim, 177 active+clean; 128 MiB data, 574 MiB used, 625 GiB / 626 GiB avail; 15 MiB/s rd, 2.3 MiB/s wr, 24 op/s 2024-09-20T22:45:17.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:17 smithi084 ceph-mon[40024]: 7.12 scrub starts 2024-09-20T22:45:17.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:17 smithi084 ceph-mon[40024]: 7.12 scrub ok 2024-09-20T22:45:17.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:17 smithi084 ceph-mon[40024]: osdmap e237: 8 total, 8 up, 7 in 2024-09-20T22:45:17.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:17 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/844174825' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:45:17.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:17 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/4145173149' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "7.17", "id": [7, 0, 4]}]: dispatch 2024-09-20T22:45:17.378 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:17.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:17.379 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:45:17.480 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:45:17.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:17.501 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:17.502 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:45:17.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:17 smithi154 ceph-mon[33264]: pgmap v256: 178 pgs: 1 active+clean+snaptrim, 177 active+clean; 128 MiB data, 574 MiB used, 625 GiB / 626 GiB avail; 15 MiB/s rd, 2.3 MiB/s wr, 24 op/s 2024-09-20T22:45:17.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:17 smithi154 ceph-mon[33264]: 7.12 scrub starts 2024-09-20T22:45:17.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:17 smithi154 ceph-mon[33264]: 7.12 scrub ok 2024-09-20T22:45:17.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:17 smithi154 ceph-mon[33264]: osdmap e237: 8 total, 8 up, 7 in 2024-09-20T22:45:17.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:17 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/844174825' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:45:17.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:17 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/4145173149' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "7.17", "id": [7, 0, 4]}]: dispatch 2024-09-20T22:45:17.603 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:45:17.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:17.624 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:17.625 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:45:17.725 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:45:17.746 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:17.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:17.748 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:45:17.848 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:45:17.870 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:17.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:17.871 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:45:17.972 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:45:17.998 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:17.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:17.999 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:45:18.100 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:45:18.121 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:18.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:18.122 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:45:18.149 INFO:tasks.rados.rados.0.smithi084.stdout:750: done (3 left) 2024-09-20T22:45:18.149 INFO:tasks.rados.rados.0.smithi084.stdout:751: snap_create 2024-09-20T22:45:18.149 INFO:tasks.rados.rados.0.smithi084.stdout:747: expect (ObjNum 50 snap 0 seq_num 50) 2024-09-20T22:45:18.223 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:45:18.243 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:18.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:18.244 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:45:18.282 INFO:tasks.rados.rados.0.smithi084.stdout:748: expect (ObjNum 257 snap 60 seq_num 257) 2024-09-20T22:45:18.345 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:45:18.366 INFO:tasks.rados.rados.0.smithi084.stdout:749: finishing write tid 1 to smithi08477203-41 2024-09-20T22:45:18.366 INFO:tasks.rados.rados.0.smithi084.stdout:749: finishing write tid 2 to smithi08477203-41 2024-09-20T22:45:18.366 INFO:tasks.rados.rados.0.smithi084.stdout:749: finishing write tid 3 to smithi08477203-41 2024-09-20T22:45:18.366 INFO:tasks.rados.rados.0.smithi084.stdout:749: finishing write tid 4 to smithi08477203-41 2024-09-20T22:45:18.366 INFO:tasks.rados.rados.0.smithi084.stdout:749: finishing write tid 5 to smithi08477203-41 2024-09-20T22:45:18.366 INFO:tasks.rados.rados.0.smithi084.stdout:749: oid 41 updating version 0 to 44 2024-09-20T22:45:18.366 INFO:tasks.rados.rados.0.smithi084.stdout:749: oid 41 updating version 44 to 45 2024-09-20T22:45:18.366 INFO:tasks.rados.rados.0.smithi084.stdout:749: oid 41 updating version 45 to 47 2024-09-20T22:45:18.367 INFO:tasks.rados.rados.0.smithi084.stdout:749: oid 41 version 47 is already newer than 46 2024-09-20T22:45:18.367 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 41 v 47 (ObjNum 294 snap 74 seq_num 294) dirty exists 2024-09-20T22:45:18.367 INFO:tasks.rados.rados.0.smithi084.stdout:749: left oid 41 (ObjNum 294 snap 74 seq_num 294) 2024-09-20T22:45:18.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:18.367 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:18.368 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:45:18.454 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:45:18 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: ::ffff:172.21.15.154 - - [20/Sep/2024:22:45:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-20T22:45:18.454 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:18 smithi084 ceph-mon[29774]: 7.7 scrub starts 2024-09-20T22:45:18.454 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:18 smithi084 ceph-mon[29774]: 7.7 scrub ok 2024-09-20T22:45:18.455 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:18 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/4145173149' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "7.17", "id": [7, 0, 4]}]': finished 2024-09-20T22:45:18.455 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:18 smithi084 ceph-mon[29774]: osdmap e238: 8 total, 8 up, 7 in 2024-09-20T22:45:18.455 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:18 smithi084 ceph-mon[29774]: 1.14 scrub starts 2024-09-20T22:45:18.455 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:18 smithi084 ceph-mon[29774]: 1.14 scrub ok 2024-09-20T22:45:18.455 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:18 smithi084 ceph-mon[29774]: 1.6 scrub starts 2024-09-20T22:45:18.455 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:18 smithi084 ceph-mon[29774]: 1.6 scrub ok 2024-09-20T22:45:18.455 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:18 smithi084 ceph-mon[29774]: osdmap e239: 8 total, 8 up, 7 in 2024-09-20T22:45:18.455 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:18 smithi084 ceph-mon[40024]: 7.7 scrub starts 2024-09-20T22:45:18.455 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:18 smithi084 ceph-mon[40024]: 7.7 scrub ok 2024-09-20T22:45:18.455 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:18 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/4145173149' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "7.17", "id": [7, 0, 4]}]': finished 2024-09-20T22:45:18.455 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:18 smithi084 ceph-mon[40024]: osdmap e238: 8 total, 8 up, 7 in 2024-09-20T22:45:18.455 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:18 smithi084 ceph-mon[40024]: 1.14 scrub starts 2024-09-20T22:45:18.455 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:18 smithi084 ceph-mon[40024]: 1.14 scrub ok 2024-09-20T22:45:18.456 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:18 smithi084 ceph-mon[40024]: 1.6 scrub starts 2024-09-20T22:45:18.456 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:18 smithi084 ceph-mon[40024]: 1.6 scrub ok 2024-09-20T22:45:18.456 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:18 smithi084 ceph-mon[40024]: osdmap e239: 8 total, 8 up, 7 in 2024-09-20T22:45:18.469 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:45:18.495 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:18.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:18.496 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:45:18.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:18 smithi154 ceph-mon[33264]: 7.7 scrub starts 2024-09-20T22:45:18.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:18 smithi154 ceph-mon[33264]: 7.7 scrub ok 2024-09-20T22:45:18.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:18 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/4145173149' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "7.17", "id": [7, 0, 4]}]': finished 2024-09-20T22:45:18.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:18 smithi154 ceph-mon[33264]: osdmap e238: 8 total, 8 up, 7 in 2024-09-20T22:45:18.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:18 smithi154 ceph-mon[33264]: 1.14 scrub starts 2024-09-20T22:45:18.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:18 smithi154 ceph-mon[33264]: 1.14 scrub ok 2024-09-20T22:45:18.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:18 smithi154 ceph-mon[33264]: 1.6 scrub starts 2024-09-20T22:45:18.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:18 smithi154 ceph-mon[33264]: 1.6 scrub ok 2024-09-20T22:45:18.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:18 smithi154 ceph-mon[33264]: osdmap e239: 8 total, 8 up, 7 in 2024-09-20T22:45:18.597 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:45:18.614 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-20T22:45:18.614 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2024-09-20T22:45:18.614 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2024-09-20T22:45:18.614 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [ 2024-09-20T22:45:18.614 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.614 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24652.0:726 7.19 e143/139 7:9eac4560:::smithi08477203-30:head v 143'50, mlcod=143'46)", 2024-09-20T22:45:18.614 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:43:42.333369+0000", 2024-09-20T22:45:18.614 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 95.790627349000005, 2024-09-20T22:45:18.614 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0099788589999999996, 2024-09-20T22:45:18.614 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:45:18.615 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:45:18.615 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:45:18.615 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:45:18.615 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.615 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:45:18.615 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:42.333369+0000", 2024-09-20T22:45:18.615 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:45:18.615 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.615 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.615 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:45:18.615 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:42.333369+0000", 2024-09-20T22:45:18.615 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:45:18.615 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.615 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.615 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:45:18.616 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:42.333370+0000", 2024-09-20T22:45:18.616 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.5000000000000003e-07 2024-09-20T22:45:18.616 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.616 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.616 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:45:18.616 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:42.336774+0000", 2024-09-20T22:45:18.616 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.003404387 2024-09-20T22:45:18.616 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.616 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.616 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:45:18.616 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:42.336784+0000", 2024-09-20T22:45:18.616 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0121e-05 2024-09-20T22:45:18.616 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.616 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.617 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:45:18.617 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:42.336791+0000", 2024-09-20T22:45:18.617 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.2099999999999996e-06 2024-09-20T22:45:18.617 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.617 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.617 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:45:18.617 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:42.336930+0000", 2024-09-20T22:45:18.617 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000138288 2024-09-20T22:45:18.617 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.617 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.617 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:45:18.617 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:42.336947+0000", 2024-09-20T22:45:18.617 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.7469000000000002e-05 2024-09-20T22:45:18.617 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.617 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.618 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:45:18.618 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:42.343300+0000", 2024-09-20T22:45:18.618 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.006352814 2024-09-20T22:45:18.618 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.618 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.618 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:45:18.618 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:42.343348+0000", 2024-09-20T22:45:18.618 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.8019999999999998e-05 2024-09-20T22:45:18.618 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:18.618 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:45:18.618 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:18.618 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.618 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.618 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24652.0:786 7.17 e145/139 7:ed54065f:::smithi08477203-12:head v 145'119, mlcod=143'117)", 2024-09-20T22:45:18.618 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:43:44.344970+0000", 2024-09-20T22:45:18.619 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 93.779026783000006, 2024-09-20T22:45:18.619 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010847977999999999, 2024-09-20T22:45:18.619 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:45:18.619 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:45:18.619 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:45:18.619 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:45:18.619 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.619 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:45:18.619 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:44.344970+0000", 2024-09-20T22:45:18.619 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:45:18.619 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.619 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.619 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:45:18.619 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:44.344970+0000", 2024-09-20T22:45:18.620 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:45:18.620 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.620 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.620 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:45:18.620 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:44.344970+0000", 2024-09-20T22:45:18.620 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.85e-07 2024-09-20T22:45:18.620 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.620 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.620 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:45:18.620 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:44.350418+0000", 2024-09-20T22:45:18.620 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0054472749999999997 2024-09-20T22:45:18.620 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.620 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.621 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:45:18.621 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:44.350434+0000", 2024-09-20T22:45:18.621 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.6113999999999999e-05 2024-09-20T22:45:18.621 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.621 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.621 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:45:18.621 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:44.350443+0000", 2024-09-20T22:45:18.621 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.6930000000000002e-06 2024-09-20T22:45:18.621 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.621 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.621 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:45:18.621 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:44.350504+0000", 2024-09-20T22:45:18.621 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.0541000000000002e-05 2024-09-20T22:45:18.621 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.621 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.622 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:45:18.622 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:44.350522+0000", 2024-09-20T22:45:18.622 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.8136999999999999e-05 2024-09-20T22:45:18.622 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.622 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.622 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:45:18.622 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:44.355760+0000", 2024-09-20T22:45:18.622 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0052376269999999999 2024-09-20T22:45:18.622 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.622 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.622 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:45:18.622 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:44.355818+0000", 2024-09-20T22:45:18.622 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.7806000000000002e-05 2024-09-20T22:45:18.622 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:18.623 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:45:18.623 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:18.623 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.623 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.623 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24652.0:894 7.a e155/94 7:50097c2d:::smithi08477203-23:head v 155'54, mlcod=143'50)", 2024-09-20T22:45:18.623 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:43:54.449836+0000", 2024-09-20T22:45:18.623 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 83.674160026999999, 2024-09-20T22:45:18.623 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010006717, 2024-09-20T22:45:18.623 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:45:18.623 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:45:18.623 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:45:18.623 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:45:18.623 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.623 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:45:18.624 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:54.449836+0000", 2024-09-20T22:45:18.624 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:45:18.624 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.624 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.624 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:45:18.624 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:54.449836+0000", 2024-09-20T22:45:18.624 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:45:18.624 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.624 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.624 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:45:18.624 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:54.449837+0000", 2024-09-20T22:45:18.624 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.7000000000000001e-07 2024-09-20T22:45:18.624 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.624 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.624 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:45:18.624 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:54.452193+0000", 2024-09-20T22:45:18.625 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0023568080000000002 2024-09-20T22:45:18.625 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.625 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.625 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:45:18.625 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:54.452234+0000", 2024-09-20T22:45:18.625 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.0847999999999999e-05 2024-09-20T22:45:18.625 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.625 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.625 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:45:18.625 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:54.452248+0000", 2024-09-20T22:45:18.625 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3718e-05 2024-09-20T22:45:18.625 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.625 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.625 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:45:18.625 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:54.452337+0000", 2024-09-20T22:45:18.626 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.8521000000000002e-05 2024-09-20T22:45:18.626 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.626 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.626 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:45:18.626 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:54.452364+0000", 2024-09-20T22:45:18.626 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.7436999999999998e-05 2024-09-20T22:45:18.626 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.626 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.626 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:45:18.626 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:54.459797+0000", 2024-09-20T22:45:18.626 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0074333419999999999 2024-09-20T22:45:18.626 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.626 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.626 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:45:18.627 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:54.459843+0000", 2024-09-20T22:45:18.627 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.5772999999999997e-05 2024-09-20T22:45:18.629 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:18.629 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:45:18.629 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:18.629 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.629 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.630 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:901 7.e 7:7c62ced4:::smithi08477203-17:head [copy-from ver 48] snapc 1e=[] ondisk+write+known_if_redirected+supports_pool_eio e156)", 2024-09-20T22:45:18.630 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:43:55.624762+0000", 2024-09-20T22:45:18.630 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 82.499233927999995, 2024-09-20T22:45:18.630 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.021835856000000001, 2024-09-20T22:45:18.630 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:45:18.630 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:45:18.630 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:45:18.630 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:45:18.630 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:45:18.630 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:45:18.630 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 901 2024-09-20T22:45:18.630 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.630 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:45:18.630 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.631 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:45:18.631 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.624762+0000", 2024-09-20T22:45:18.631 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:45:18.631 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.631 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.631 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:45:18.631 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.624762+0000", 2024-09-20T22:45:18.631 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:45:18.631 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.631 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.631 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:45:18.631 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.624765+0000", 2024-09-20T22:45:18.631 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.57e-06 2024-09-20T22:45:18.632 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.632 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.632 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:45:18.632 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.624776+0000", 2024-09-20T22:45:18.632 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.078e-05 2024-09-20T22:45:18.632 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.632 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.632 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:45:18.632 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.624803+0000", 2024-09-20T22:45:18.632 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.6769999999999999e-05 2024-09-20T22:45:18.632 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.632 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.632 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:45:18.632 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.624815+0000", 2024-09-20T22:45:18.633 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2459e-05 2024-09-20T22:45:18.633 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.633 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.633 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:45:18.633 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.624936+0000", 2024-09-20T22:45:18.633 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000120467 2024-09-20T22:45:18.633 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.633 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.633 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:45:18.633 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.625046+0000", 2024-09-20T22:45:18.633 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000110222 2024-09-20T22:45:18.633 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.633 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.633 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 0,5", 2024-09-20T22:45:18.634 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.635547+0000", 2024-09-20T22:45:18.634 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010501765999999999 2024-09-20T22:45:18.634 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.634 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.634 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:45:18.634 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.644332+0000", 2024-09-20T22:45:18.634 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0087847729999999992 2024-09-20T22:45:18.634 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.634 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.634 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:45:18.634 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.646168+0000", 2024-09-20T22:45:18.634 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0018360010000000001 2024-09-20T22:45:18.634 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.635 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.635 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:45:18.635 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.646512+0000", 2024-09-20T22:45:18.635 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000343269 2024-09-20T22:45:18.635 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.635 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.635 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:45:18.635 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.646550+0000", 2024-09-20T22:45:18.635 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.8383000000000002e-05 2024-09-20T22:45:18.635 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.635 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.635 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:45:18.635 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.646598+0000", 2024-09-20T22:45:18.635 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.8396000000000002e-05 2024-09-20T22:45:18.635 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:18.635 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:45:18.636 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:18.636 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.636 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.636 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:902 7.e 7:7c62ced4:::smithi08477203-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e156)", 2024-09-20T22:45:18.636 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:43:55.624853+0000", 2024-09-20T22:45:18.636 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 82.499143849999996, 2024-09-20T22:45:18.636 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.015323603999999999, 2024-09-20T22:45:18.636 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:45:18.636 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:45:18.636 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-20T22:45:18.636 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:45:18.636 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:45:18.636 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:45:18.636 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 902 2024-09-20T22:45:18.636 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.637 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:45:18.637 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.637 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:45:18.637 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.624853+0000", 2024-09-20T22:45:18.637 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:45:18.637 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.637 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.637 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:45:18.637 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.624853+0000", 2024-09-20T22:45:18.637 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:45:18.637 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.637 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.637 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:45:18.637 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.624853+0000", 2024-09-20T22:45:18.637 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.1399999999999997e-07 2024-09-20T22:45:18.638 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.638 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.638 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:45:18.638 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.624859+0000", 2024-09-20T22:45:18.638 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.9610000000000002e-06 2024-09-20T22:45:18.638 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.638 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.638 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:45:18.638 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.624887+0000", 2024-09-20T22:45:18.638 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.7597e-05 2024-09-20T22:45:18.638 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.638 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.638 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:45:18.639 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.624894+0000", 2024-09-20T22:45:18.639 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.469e-06 2024-09-20T22:45:18.639 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.639 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.639 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:45:18.639 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.625219+0000", 2024-09-20T22:45:18.639 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00032538499999999998 2024-09-20T22:45:18.639 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.639 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.639 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for blocked object", 2024-09-20T22:45:18.639 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.625258+0000", 2024-09-20T22:45:18.639 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.8467000000000003e-05 2024-09-20T22:45:18.639 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.640 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.640 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:45:18.640 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.640050+0000", 2024-09-20T22:45:18.640 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0147923 2024-09-20T22:45:18.640 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.640 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.640 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:45:18.640 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.640112+0000", 2024-09-20T22:45:18.640 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.1723000000000005e-05 2024-09-20T22:45:18.640 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.640 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.640 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:45:18.640 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.640176+0000", 2024-09-20T22:45:18.641 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.4188000000000003e-05 2024-09-20T22:45:18.641 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:18.641 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:45:18.641 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:18.641 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.641 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.641 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:1326 7.e 7:71b5c915:::smithi08477203-6:head [sparse-read 0~3233562] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e180)", 2024-09-20T22:45:18.641 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:44:19.253684+0000", 2024-09-20T22:45:18.641 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 58.870312319, 2024-09-20T22:45:18.641 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010161692, 2024-09-20T22:45:18.641 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:45:18.641 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:45:18.641 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-20T22:45:18.641 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:45:18.641 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:45:18.642 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:45:18.642 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1326 2024-09-20T22:45:18.642 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.642 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:45:18.642 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.642 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:45:18.642 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:19.253684+0000", 2024-09-20T22:45:18.642 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:45:18.642 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.642 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.642 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:45:18.642 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:19.253684+0000", 2024-09-20T22:45:18.642 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:45:18.642 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.643 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.643 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:45:18.643 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:19.253685+0000", 2024-09-20T22:45:18.643 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.4099999999999999e-07 2024-09-20T22:45:18.643 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.643 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.643 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:45:18.643 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:19.253687+0000", 2024-09-20T22:45:18.643 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.627e-06 2024-09-20T22:45:18.643 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.643 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.643 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:45:18.643 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:19.253693+0000", 2024-09-20T22:45:18.643 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.9800000000000003e-06 2024-09-20T22:45:18.643 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.644 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.644 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:45:18.644 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:19.253699+0000", 2024-09-20T22:45:18.644 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.5690000000000004e-06 2024-09-20T22:45:18.644 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.644 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.644 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:45:18.644 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:19.253765+0000", 2024-09-20T22:45:18.644 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.6643999999999994e-05 2024-09-20T22:45:18.644 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.644 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.644 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting_for_map not empty", 2024-09-20T22:45:18.644 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:19.253768+0000", 2024-09-20T22:45:18.644 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.4269999999999998e-06 2024-09-20T22:45:18.645 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.645 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.645 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:45:18.645 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:19.263384+0000", 2024-09-20T22:45:18.645 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0096165279999999992 2024-09-20T22:45:18.645 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.645 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.645 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:45:18.645 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:19.263489+0000", 2024-09-20T22:45:18.645 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00010485899999999999 2024-09-20T22:45:18.645 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.645 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.645 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:45:18.645 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:19.263846+0000", 2024-09-20T22:45:18.645 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00035661699999999999 2024-09-20T22:45:18.646 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:18.646 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:45:18.646 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:18.646 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.646 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.646 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:1327 7.e 7:71b5c915:::smithi08477203-6:head [sparse-read 0~3233562] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e180)", 2024-09-20T22:45:18.646 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:44:19.253715+0000", 2024-09-20T22:45:18.646 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 58.870281429000002, 2024-09-20T22:45:18.646 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010418802, 2024-09-20T22:45:18.646 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:45:18.646 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:45:18.646 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-20T22:45:18.646 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:45:18.646 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:45:18.647 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:45:18.647 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1327 2024-09-20T22:45:18.647 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.647 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:45:18.647 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.647 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:45:18.647 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:19.253715+0000", 2024-09-20T22:45:18.647 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:45:18.647 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.647 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.647 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:45:18.647 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:19.253715+0000", 2024-09-20T22:45:18.647 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:45:18.647 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.647 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.648 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:45:18.648 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:19.253715+0000", 2024-09-20T22:45:18.648 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.9200000000000002e-07 2024-09-20T22:45:18.648 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.648 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.648 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:45:18.648 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:19.253718+0000", 2024-09-20T22:45:18.648 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.2340000000000001e-06 2024-09-20T22:45:18.648 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.648 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.648 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:45:18.648 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:19.253723+0000", 2024-09-20T22:45:18.648 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.5269999999999998e-06 2024-09-20T22:45:18.648 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.648 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.649 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:45:18.649 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:19.253728+0000", 2024-09-20T22:45:18.649 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.1329999999999998e-06 2024-09-20T22:45:18.649 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.649 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.649 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:45:18.649 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:19.253782+0000", 2024-09-20T22:45:18.649 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.4221999999999997e-05 2024-09-20T22:45:18.649 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.649 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.649 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting_for_map not empty", 2024-09-20T22:45:18.649 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:19.253785+0000", 2024-09-20T22:45:18.649 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.3980000000000002e-06 2024-09-20T22:45:18.649 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.650 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.650 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:45:18.650 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:19.263880+0000", 2024-09-20T22:45:18.650 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010094736999999999 2024-09-20T22:45:18.650 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.650 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.650 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:45:18.650 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:19.263982+0000", 2024-09-20T22:45:18.650 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000102481 2024-09-20T22:45:18.650 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.650 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.650 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:45:18.650 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:19.264134+0000", 2024-09-20T22:45:18.650 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00015167800000000001 2024-09-20T22:45:18.651 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:18.651 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:45:18.651 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:18.651 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.651 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.651 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24652.0:1420 7.19 e189/139 7:9cc0b7e2:::smithi08477203-46:head v 189'99, mlcod=174'95)", 2024-09-20T22:45:18.651 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:44:28.150258+0000", 2024-09-20T22:45:18.651 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 49.973738154000003, 2024-09-20T22:45:18.651 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014180626, 2024-09-20T22:45:18.651 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:45:18.651 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:45:18.651 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:45:18.651 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:45:18.651 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.652 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:45:18.652 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.150258+0000", 2024-09-20T22:45:18.652 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:45:18.652 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.652 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.652 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:45:18.652 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.150258+0000", 2024-09-20T22:45:18.652 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:45:18.652 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.652 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.652 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:45:18.652 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.150259+0000", 2024-09-20T22:45:18.652 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.99e-07 2024-09-20T22:45:18.652 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.653 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.653 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:45:18.653 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.150674+0000", 2024-09-20T22:45:18.653 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00041528000000000002 2024-09-20T22:45:18.653 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.653 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.653 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:45:18.653 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.150688+0000", 2024-09-20T22:45:18.653 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3399e-05 2024-09-20T22:45:18.653 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.653 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.653 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:45:18.653 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.150699+0000", 2024-09-20T22:45:18.654 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1022999999999999e-05 2024-09-20T22:45:18.654 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.654 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.654 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:45:18.654 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.150772+0000", 2024-09-20T22:45:18.654 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.3455999999999996e-05 2024-09-20T22:45:18.654 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.654 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.654 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:45:18.654 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.150798+0000", 2024-09-20T22:45:18.654 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.5777000000000001e-05 2024-09-20T22:45:18.654 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.654 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.654 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:45:18.655 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.164390+0000", 2024-09-20T22:45:18.655 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013591789999999999 2024-09-20T22:45:18.655 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.655 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.655 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:45:18.655 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.164439+0000", 2024-09-20T22:45:18.655 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.9202000000000001e-05 2024-09-20T22:45:18.655 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:18.655 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:45:18.655 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:18.655 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.655 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.655 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24652.0:1421 7.19 e189/139 7:9cc0b7e2:::smithi08477203-46:head v 189'100, mlcod=174'95)", 2024-09-20T22:45:18.655 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:44:28.156805+0000", 2024-09-20T22:45:18.655 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 49.967191296000003, 2024-09-20T22:45:18.656 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013180661999999999, 2024-09-20T22:45:18.656 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:45:18.656 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:45:18.656 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:45:18.656 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:45:18.656 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.656 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:45:18.656 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.156805+0000", 2024-09-20T22:45:18.656 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:45:18.656 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.656 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.656 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:45:18.656 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.156805+0000", 2024-09-20T22:45:18.656 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:45:18.656 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.657 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.657 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:45:18.657 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.156806+0000", 2024-09-20T22:45:18.657 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.074e-06 2024-09-20T22:45:18.657 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.657 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.657 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:45:18.657 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.157126+0000", 2024-09-20T22:45:18.657 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00031991399999999997 2024-09-20T22:45:18.657 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.657 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.657 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:45:18.657 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.157136+0000", 2024-09-20T22:45:18.657 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.4499999999999993e-06 2024-09-20T22:45:18.657 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.658 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.658 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:45:18.658 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.157143+0000", 2024-09-20T22:45:18.658 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.7649999999999993e-06 2024-09-20T22:45:18.658 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.658 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.658 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:45:18.658 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.157188+0000", 2024-09-20T22:45:18.658 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.4885000000000001e-05 2024-09-20T22:45:18.658 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.658 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.658 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:45:18.658 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.157208+0000", 2024-09-20T22:45:18.658 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.0015000000000001e-05 2024-09-20T22:45:18.659 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.659 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.659 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:45:18.659 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.169936+0000", 2024-09-20T22:45:18.659 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.012728204 2024-09-20T22:45:18.659 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.659 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.659 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:45:18.659 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.169986+0000", 2024-09-20T22:45:18.659 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.9354999999999998e-05 2024-09-20T22:45:18.659 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:18.659 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:45:18.659 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:18.659 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.659 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.660 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:1531 7.e 7:7c62ced4:::smithi08477203-17:head [sparse-read 0~3411821] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e197)", 2024-09-20T22:45:18.660 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:44:36.199079+0000", 2024-09-20T22:45:18.660 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 41.924917284999999, 2024-09-20T22:45:18.660 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0099173630000000002, 2024-09-20T22:45:18.660 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:45:18.660 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:45:18.660 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-20T22:45:18.660 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:45:18.660 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:45:18.660 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:45:18.660 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1531 2024-09-20T22:45:18.660 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.660 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:45:18.660 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.661 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:45:18.661 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:36.199079+0000", 2024-09-20T22:45:18.661 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:45:18.661 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.661 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.661 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:45:18.661 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:36.199079+0000", 2024-09-20T22:45:18.661 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:45:18.661 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.661 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.661 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:45:18.661 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:36.199079+0000", 2024-09-20T22:45:18.661 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.4999999999999998e-07 2024-09-20T22:45:18.661 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.661 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.662 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:45:18.662 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:36.199082+0000", 2024-09-20T22:45:18.662 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.074e-06 2024-09-20T22:45:18.662 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.662 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.662 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:45:18.662 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:36.199086+0000", 2024-09-20T22:45:18.662 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.3560000000000003e-06 2024-09-20T22:45:18.662 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.662 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.662 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:45:18.662 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:36.199090+0000", 2024-09-20T22:45:18.663 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.2289999999999997e-06 2024-09-20T22:45:18.663 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.663 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.663 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:45:18.663 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:36.199230+0000", 2024-09-20T22:45:18.663 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013957299999999999 2024-09-20T22:45:18.663 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.663 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.663 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting_for_map not empty", 2024-09-20T22:45:18.663 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:36.199234+0000", 2024-09-20T22:45:18.663 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.031e-06 2024-09-20T22:45:18.663 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.663 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.663 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:45:18.663 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:36.208627+0000", 2024-09-20T22:45:18.664 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0093930260000000005 2024-09-20T22:45:18.664 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.664 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.664 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:45:18.664 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:36.208740+0000", 2024-09-20T22:45:18.664 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000113152 2024-09-20T22:45:18.664 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.664 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.664 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:45:18.664 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:36.208996+0000", 2024-09-20T22:45:18.664 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000256572 2024-09-20T22:45:18.664 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:18.664 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:45:18.664 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:18.664 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.664 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.665 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24652.0:1527 7.17 e197/139 7:ea720b85:::smithi08477203-26:head v 197'159, mlcod=187'156)", 2024-09-20T22:45:18.665 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:44:36.209366+0000", 2024-09-20T22:45:18.665 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 41.914630842999998, 2024-09-20T22:45:18.665 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010891473, 2024-09-20T22:45:18.665 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:45:18.665 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:45:18.665 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:45:18.665 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:45:18.665 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.665 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:45:18.665 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:36.209366+0000", 2024-09-20T22:45:18.665 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:45:18.665 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.665 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.665 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:45:18.665 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:36.209366+0000", 2024-09-20T22:45:18.666 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:45:18.666 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.666 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.666 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:45:18.666 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:36.209366+0000", 2024-09-20T22:45:18.666 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.0999999999999999e-07 2024-09-20T22:45:18.666 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.666 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.666 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:45:18.666 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:36.215620+0000", 2024-09-20T22:45:18.666 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0062539450000000003 2024-09-20T22:45:18.666 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.666 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.666 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:45:18.667 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:36.215649+0000", 2024-09-20T22:45:18.667 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.9037999999999999e-05 2024-09-20T22:45:18.667 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.667 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.667 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:45:18.667 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:36.215663+0000", 2024-09-20T22:45:18.667 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.4059999999999999e-05 2024-09-20T22:45:18.667 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.667 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.667 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:45:18.667 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:36.215772+0000", 2024-09-20T22:45:18.667 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000108773 2024-09-20T22:45:18.667 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.667 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.667 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:45:18.668 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:36.215801+0000", 2024-09-20T22:45:18.668 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.8711e-05 2024-09-20T22:45:18.668 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.668 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.668 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:45:18.668 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:36.220205+0000", 2024-09-20T22:45:18.668 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0044043989999999998 2024-09-20T22:45:18.668 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.668 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.668 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:45:18.668 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:36.220257+0000", 2024-09-20T22:45:18.668 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.2036999999999997e-05 2024-09-20T22:45:18.668 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:18.668 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:45:18.668 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:18.669 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.669 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.669 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24652.0:1542 7.12 e198/139 7:4b06cbed:::smithi08477203-49:head v 198'174, mlcod=197'171)", 2024-09-20T22:45:18.669 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:44:37.216841+0000", 2024-09-20T22:45:18.669 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 40.907154908999999, 2024-09-20T22:45:18.669 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010876451000000001, 2024-09-20T22:45:18.669 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:45:18.669 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:45:18.669 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:45:18.669 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:45:18.669 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.669 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:45:18.669 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:37.216841+0000", 2024-09-20T22:45:18.669 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:45:18.670 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.670 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.670 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:45:18.670 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:37.216841+0000", 2024-09-20T22:45:18.670 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:45:18.670 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.670 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.670 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:45:18.670 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:37.216842+0000", 2024-09-20T22:45:18.670 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.1399999999999997e-07 2024-09-20T22:45:18.670 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.670 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.670 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:45:18.670 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:37.221497+0000", 2024-09-20T22:45:18.670 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0046552140000000004 2024-09-20T22:45:18.671 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.671 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.671 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:45:18.671 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:37.221522+0000", 2024-09-20T22:45:18.671 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.4732999999999999e-05 2024-09-20T22:45:18.671 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.671 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.671 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:45:18.671 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:37.221536+0000", 2024-09-20T22:45:18.671 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.4382000000000001e-05 2024-09-20T22:45:18.671 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.671 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.671 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:45:18.671 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:37.221631+0000", 2024-09-20T22:45:18.671 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.4943000000000002e-05 2024-09-20T22:45:18.672 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.672 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.672 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:45:18.672 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:37.221662+0000", 2024-09-20T22:45:18.672 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.0480999999999998e-05 2024-09-20T22:45:18.672 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.672 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.672 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:45:18.672 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:37.227657+0000", 2024-09-20T22:45:18.672 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0059949970000000002 2024-09-20T22:45:18.672 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.672 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.672 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:45:18.672 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:37.227718+0000", 2024-09-20T22:45:18.672 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.1286999999999998e-05 2024-09-20T22:45:18.672 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:18.673 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:45:18.673 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:18.673 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.673 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.673 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24652.0:1654 7.19 e206/139 7:9970b578:::smithi08477203-15:head v 206'114, mlcod=205'111)", 2024-09-20T22:45:18.673 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:44:45.167994+0000", 2024-09-20T22:45:18.673 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 32.956002746000003, 2024-09-20T22:45:18.673 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.01345262, 2024-09-20T22:45:18.673 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:45:18.673 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:45:18.673 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:45:18.673 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:45:18.673 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.673 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:45:18.674 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:45.167994+0000", 2024-09-20T22:45:18.674 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:45:18.674 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.674 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.674 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:45:18.674 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:45.167994+0000", 2024-09-20T22:45:18.674 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:45:18.674 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.674 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.674 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:45:18.674 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:45.167996+0000", 2024-09-20T22:45:18.674 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.5100000000000001e-06 2024-09-20T22:45:18.674 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.674 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.674 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:45:18.675 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:45.175169+0000", 2024-09-20T22:45:18.675 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0071732740000000003 2024-09-20T22:45:18.675 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.675 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.675 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:45:18.675 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:45.175208+0000", 2024-09-20T22:45:18.675 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.8597000000000003e-05 2024-09-20T22:45:18.675 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.675 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.675 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:45:18.675 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:45.175223+0000", 2024-09-20T22:45:18.675 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.4627000000000001e-05 2024-09-20T22:45:18.675 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.675 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.676 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:45:18.676 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:45.175304+0000", 2024-09-20T22:45:18.676 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.1111000000000003e-05 2024-09-20T22:45:18.676 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.676 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.676 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:45:18.676 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:45.175336+0000", 2024-09-20T22:45:18.676 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.1945e-05 2024-09-20T22:45:18.676 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.676 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.676 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:45:18.676 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:45.181392+0000", 2024-09-20T22:45:18.676 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0060566630000000003 2024-09-20T22:45:18.676 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.676 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.677 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:45:18.677 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:45.181446+0000", 2024-09-20T22:45:18.677 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.3893000000000001e-05 2024-09-20T22:45:18.677 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:18.677 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:45:18.677 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:18.677 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.677 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.677 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24652.0:1799 7.17 e229/139 7:ea720b85:::smithi08477203-26:head v 229'168, mlcod=222'164)", 2024-09-20T22:45:18.677 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:45:08.171867+0000", 2024-09-20T22:45:18.677 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 9.9521295280000004, 2024-09-20T22:45:18.677 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014913454, 2024-09-20T22:45:18.677 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:45:18.677 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:45:18.677 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:45:18.678 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:45:18.678 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.678 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:45:18.678 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.171867+0000", 2024-09-20T22:45:18.678 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:45:18.678 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.678 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.678 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:45:18.678 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.171867+0000", 2024-09-20T22:45:18.678 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:45:18.678 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.678 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.678 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:45:18.678 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.171867+0000", 2024-09-20T22:45:18.678 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.3799999999999999e-07 2024-09-20T22:45:18.679 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.679 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.679 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:45:18.679 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.173016+0000", 2024-09-20T22:45:18.679 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0011485740000000001 2024-09-20T22:45:18.679 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.679 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.679 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:45:18.679 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.173024+0000", 2024-09-20T22:45:18.679 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.4780000000000008e-06 2024-09-20T22:45:18.679 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.679 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.679 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:45:18.679 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.173034+0000", 2024-09-20T22:45:18.679 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.7860000000000008e-06 2024-09-20T22:45:18.680 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.680 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.680 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:45:18.680 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.173098+0000", 2024-09-20T22:45:18.680 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.4405000000000004e-05 2024-09-20T22:45:18.680 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.680 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.680 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:45:18.680 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.173125+0000", 2024-09-20T22:45:18.680 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.6950000000000001e-05 2024-09-20T22:45:18.680 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.680 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.680 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:45:18.680 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.186711+0000", 2024-09-20T22:45:18.680 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.01358608 2024-09-20T22:45:18.681 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.681 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.681 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:45:18.681 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.186780+0000", 2024-09-20T22:45:18.681 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.8942999999999994e-05 2024-09-20T22:45:18.681 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:18.681 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:45:18.681 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:18.681 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.681 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.681 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24652.0:1800 7.17 e229/139 7:ea720b85:::smithi08477203-26:head v 229'169, mlcod=222'164)", 2024-09-20T22:45:18.681 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:45:08.178177+0000", 2024-09-20T22:45:18.681 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 9.9458197120000005, 2024-09-20T22:45:18.681 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011399595, 2024-09-20T22:45:18.681 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:45:18.682 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:45:18.682 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:45:18.682 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:45:18.682 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.682 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:45:18.682 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.178177+0000", 2024-09-20T22:45:18.682 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:45:18.682 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.682 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.682 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:45:18.682 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.178177+0000", 2024-09-20T22:45:18.682 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:45:18.682 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.682 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.682 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:45:18.683 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.178178+0000", 2024-09-20T22:45:18.683 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2389999999999999e-06 2024-09-20T22:45:18.683 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.683 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.683 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:45:18.683 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.179005+0000", 2024-09-20T22:45:18.683 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00082757900000000005 2024-09-20T22:45:18.683 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.683 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.683 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:45:18.683 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.179016+0000", 2024-09-20T22:45:18.683 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0404999999999999e-05 2024-09-20T22:45:18.683 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.683 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.683 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:45:18.684 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.179023+0000", 2024-09-20T22:45:18.684 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.2069999999999998e-06 2024-09-20T22:45:18.684 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.684 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.684 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:45:18.684 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.179088+0000", 2024-09-20T22:45:18.684 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.4783000000000006e-05 2024-09-20T22:45:18.684 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.684 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.684 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:45:18.684 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.179107+0000", 2024-09-20T22:45:18.684 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.9293999999999999e-05 2024-09-20T22:45:18.684 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.684 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.685 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:45:18.685 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.189518+0000", 2024-09-20T22:45:18.685 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010410739 2024-09-20T22:45:18.685 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.685 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.685 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:45:18.685 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.189576+0000", 2024-09-20T22:45:18.685 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.8349e-05 2024-09-20T22:45:18.685 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:18.685 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:45:18.685 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:18.685 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.685 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.685 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24652.0:1806 7.a e229/94 7:5aaef466:::smithi08477203-40:head v 229'139, mlcod=229'136)", 2024-09-20T22:45:18.686 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:45:08.178370+0000", 2024-09-20T22:45:18.686 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 9.9456265980000005, 2024-09-20T22:45:18.686 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011334828999999999, 2024-09-20T22:45:18.686 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:45:18.686 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:45:18.686 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:45:18.686 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:45:18.686 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.686 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:45:18.686 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.178370+0000", 2024-09-20T22:45:18.686 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:45:18.686 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.686 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.686 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:45:18.687 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.178370+0000", 2024-09-20T22:45:18.687 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:45:18.687 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.687 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.687 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:45:18.687 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.178371+0000", 2024-09-20T22:45:18.687 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.9599999999999998e-07 2024-09-20T22:45:18.687 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.687 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.687 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:45:18.687 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.178381+0000", 2024-09-20T22:45:18.687 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.8539999999999992e-06 2024-09-20T22:45:18.687 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.687 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.687 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:45:18.687 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.178398+0000", 2024-09-20T22:45:18.688 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.7303999999999999e-05 2024-09-20T22:45:18.688 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.688 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.688 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:45:18.688 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.178410+0000", 2024-09-20T22:45:18.688 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1827e-05 2024-09-20T22:45:18.688 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.688 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.688 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:45:18.688 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.178545+0000", 2024-09-20T22:45:18.688 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013498199999999999 2024-09-20T22:45:18.688 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.688 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.688 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:45:18.688 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.178593+0000", 2024-09-20T22:45:18.689 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.8167999999999997e-05 2024-09-20T22:45:18.689 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.689 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.689 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:45:18.689 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.189673+0000", 2024-09-20T22:45:18.689 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011080491 2024-09-20T22:45:18.689 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.689 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.689 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:45:18.689 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.189705+0000", 2024-09-20T22:45:18.689 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.1306999999999997e-05 2024-09-20T22:45:18.689 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:18.689 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:45:18.689 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:18.690 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.690 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.690 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24652.0:1809 7.16 e229/157 7:6885a65f:::smithi08477203-14:head v 229'53, mlcod=222'49)", 2024-09-20T22:45:18.690 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:45:08.178454+0000", 2024-09-20T22:45:18.690 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 9.9455419630000002, 2024-09-20T22:45:18.690 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011210886, 2024-09-20T22:45:18.690 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:45:18.690 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:45:18.690 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:45:18.690 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:45:18.690 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.690 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:45:18.690 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.178454+0000", 2024-09-20T22:45:18.690 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:45:18.690 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.691 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.691 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:45:18.691 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.178454+0000", 2024-09-20T22:45:18.691 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:45:18.691 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.691 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.691 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:45:18.691 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.178455+0000", 2024-09-20T22:45:18.691 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.05e-07 2024-09-20T22:45:18.691 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.691 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.691 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:45:18.691 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.178460+0000", 2024-09-20T22:45:18.692 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.4380000000000003e-06 2024-09-20T22:45:18.692 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.692 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.692 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:45:18.692 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.178469+0000", 2024-09-20T22:45:18.692 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.4649999999999994e-06 2024-09-20T22:45:18.692 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.692 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.692 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:45:18.692 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.178473+0000", 2024-09-20T22:45:18.692 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.9250000000000003e-06 2024-09-20T22:45:18.692 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.692 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.692 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:45:18.693 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.178705+0000", 2024-09-20T22:45:18.693 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00023114899999999999 2024-09-20T22:45:18.693 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.693 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.693 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:45:18.693 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.178735+0000", 2024-09-20T22:45:18.693 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.0889000000000002e-05 2024-09-20T22:45:18.693 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.693 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.693 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:45:18.693 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.189631+0000", 2024-09-20T22:45:18.693 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010895317 2024-09-20T22:45:18.693 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.693 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.694 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:45:18.694 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.189665+0000", 2024-09-20T22:45:18.694 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.4498000000000001e-05 2024-09-20T22:45:18.694 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:18.694 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:45:18.694 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:18.694 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.694 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.694 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24652.0:1807 7.a e229/94 7:5aaef466:::smithi08477203-40:head v 229'140, mlcod=229'136)", 2024-09-20T22:45:18.694 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:45:08.178940+0000", 2024-09-20T22:45:18.694 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 9.9450565570000009, 2024-09-20T22:45:18.694 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010891029999999999, 2024-09-20T22:45:18.694 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:45:18.694 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:45:18.695 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:45:18.695 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:45:18.695 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.695 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:45:18.695 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.178940+0000", 2024-09-20T22:45:18.695 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:45:18.695 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.695 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.695 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:45:18.695 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.178940+0000", 2024-09-20T22:45:18.695 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:45:18.695 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.695 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.695 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:45:18.695 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.178941+0000", 2024-09-20T22:45:18.695 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.75e-07 2024-09-20T22:45:18.695 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.696 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.696 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:45:18.696 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.178946+0000", 2024-09-20T22:45:18.696 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.8340000000000004e-06 2024-09-20T22:45:18.696 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.696 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.696 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:45:18.696 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.178959+0000", 2024-09-20T22:45:18.696 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3156000000000001e-05 2024-09-20T22:45:18.696 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.696 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.696 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:45:18.696 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.178970+0000", 2024-09-20T22:45:18.696 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0086e-05 2024-09-20T22:45:18.696 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.696 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.696 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:45:18.697 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.180323+0000", 2024-09-20T22:45:18.697 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001353808 2024-09-20T22:45:18.697 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.697 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.697 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:45:18.697 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.180345+0000", 2024-09-20T22:45:18.697 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.1409000000000001e-05 2024-09-20T22:45:18.697 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.697 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.697 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:45:18.697 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.189800+0000", 2024-09-20T22:45:18.697 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0094553980000000003 2024-09-20T22:45:18.697 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.697 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.697 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:45:18.698 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.189831+0000", 2024-09-20T22:45:18.698 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.0664000000000003e-05 2024-09-20T22:45:18.698 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:18.698 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:45:18.698 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:18.698 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.698 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.698 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24652.0:1801 7.17 e229/139 7:ea720b85:::smithi08477203-26:head v 229'170, mlcod=222'164)", 2024-09-20T22:45:18.698 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:45:08.179496+0000", 2024-09-20T22:45:18.698 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 9.944500626, 2024-09-20T22:45:18.698 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.01011184, 2024-09-20T22:45:18.698 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:45:18.698 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:45:18.698 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:45:18.698 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:45:18.698 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.699 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:45:18.699 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.179496+0000", 2024-09-20T22:45:18.699 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:45:18.699 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.699 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.699 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:45:18.699 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.179496+0000", 2024-09-20T22:45:18.699 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:45:18.699 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.699 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.699 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:45:18.699 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.179496+0000", 2024-09-20T22:45:18.699 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.2399999999999999e-07 2024-09-20T22:45:18.699 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.699 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.700 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:45:18.700 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.179502+0000", 2024-09-20T22:45:18.700 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.327e-06 2024-09-20T22:45:18.700 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.700 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.700 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:45:18.700 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.179515+0000", 2024-09-20T22:45:18.700 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3329999999999999e-05 2024-09-20T22:45:18.700 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.700 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.700 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:45:18.700 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.179522+0000", 2024-09-20T22:45:18.700 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.7120000000000003e-06 2024-09-20T22:45:18.700 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.700 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.700 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:45:18.701 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.179597+0000", 2024-09-20T22:45:18.701 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.5746000000000003e-05 2024-09-20T22:45:18.701 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.701 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.701 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:45:18.701 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.179616+0000", 2024-09-20T22:45:18.701 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.8904999999999999e-05 2024-09-20T22:45:18.701 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.701 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.701 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:45:18.701 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.189583+0000", 2024-09-20T22:45:18.701 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0099669979999999995 2024-09-20T22:45:18.701 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.701 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.701 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:45:18.701 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.189608+0000", 2024-09-20T22:45:18.702 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.4397999999999999e-05 2024-09-20T22:45:18.702 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:18.702 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:45:18.702 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:18.702 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.702 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.702 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24652.0:1808 7.a e229/94 7:5aaef466:::smithi08477203-40:head v 229'141, mlcod=229'136)", 2024-09-20T22:45:18.702 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:45:08.179805+0000", 2024-09-20T22:45:18.702 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 9.9441916409999997, 2024-09-20T22:45:18.702 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010048259, 2024-09-20T22:45:18.702 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:45:18.702 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:45:18.702 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:45:18.702 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:45:18.702 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.703 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:45:18.703 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.179805+0000", 2024-09-20T22:45:18.703 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:45:18.703 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.703 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.703 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:45:18.703 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.179805+0000", 2024-09-20T22:45:18.703 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:45:18.703 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.703 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.703 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:45:18.703 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.179806+0000", 2024-09-20T22:45:18.703 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.5600000000000005e-07 2024-09-20T22:45:18.704 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.704 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.704 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:45:18.704 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.179811+0000", 2024-09-20T22:45:18.704 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.7899999999999996e-06 2024-09-20T22:45:18.704 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.704 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.704 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:45:18.704 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.179820+0000", 2024-09-20T22:45:18.704 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.5629999999999992e-06 2024-09-20T22:45:18.704 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.704 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.704 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:45:18.704 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.179826+0000", 2024-09-20T22:45:18.704 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.5470000000000002e-06 2024-09-20T22:45:18.705 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.705 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.705 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:45:18.705 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.187155+0000", 2024-09-20T22:45:18.705 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0073287270000000002 2024-09-20T22:45:18.705 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.705 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.705 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:45:18.705 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.187179+0000", 2024-09-20T22:45:18.705 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.3592e-05 2024-09-20T22:45:18.705 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.705 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.705 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:45:18.705 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.189833+0000", 2024-09-20T22:45:18.705 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026542839999999998 2024-09-20T22:45:18.705 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:18.706 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:18.706 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:45:18.706 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.189853+0000", 2024-09-20T22:45:18.706 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.0000000000000002e-05 2024-09-20T22:45:18.706 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:18.706 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:45:18.706 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:18.706 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:18.706 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:45:18.706 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-20T22:45:18.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:18.707 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:18.707 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:45:18.808 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:45:18.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:18.834 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:18.835 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:45:18.923 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:45:18.935 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:45:18.960 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:18.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:18.961 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:45:19.061 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:45:19.082 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:19.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:19.083 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:45:19.153 INFO:tasks.rados.rados.0.smithi084.stdout:747: done (3 left) 2024-09-20T22:45:19.153 INFO:tasks.rados.rados.0.smithi084.stdout:748: done (2 left) 2024-09-20T22:45:19.153 INFO:tasks.rados.rados.0.smithi084.stdout:749: done (1 left) 2024-09-20T22:45:19.153 INFO:tasks.rados.rados.0.smithi084.stdout:751: done (0 left) 2024-09-20T22:45:19.153 INFO:tasks.rados.rados.0.smithi084.stdout:752: read oid 10 snap 60 2024-09-20T22:45:19.153 INFO:tasks.rados.rados.0.smithi084.stdout:752: expect deleted 2024-09-20T22:45:19.153 INFO:tasks.rados.rados.0.smithi084.stdout:753: write oid 26 current snap is 75 2024-09-20T22:45:19.153 INFO:tasks.rados.rados.0.smithi084.stdout:753: seq_num 295 ranges {558065=619869,1579835=486765,2468371=489694} 2024-09-20T22:45:19.158 INFO:tasks.rados.rados.0.smithi084.stdout:753: writing smithi08477203-26 from 558065 to 1177934 tid 1 2024-09-20T22:45:19.163 INFO:tasks.rados.rados.0.smithi084.stdout:753: writing smithi08477203-26 from 1579835 to 2066600 tid 2 2024-09-20T22:45:19.169 INFO:tasks.rados.rados.0.smithi084.stdout:753: writing smithi08477203-26 from 2468371 to 2958065 tid 3 2024-09-20T22:45:19.169 INFO:tasks.rados.rados.0.smithi084.stdout:754: rollback oid 48 current snap is 75 2024-09-20T22:45:19.169 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 48 to 60 2024-09-20T22:45:19.169 INFO:tasks.rados.rados.0.smithi084.stdout:753: finishing write tid 1 to smithi08477203-26 2024-09-20T22:45:19.169 INFO:tasks.rados.rados.0.smithi084.stdout:753: finishing write tid 2 to smithi08477203-26 2024-09-20T22:45:19.169 INFO:tasks.rados.rados.0.smithi084.stdout:752: done (2 left) 2024-09-20T22:45:19.169 INFO:tasks.rados.rados.0.smithi084.stdout:755: rollback oid 42 current snap is 75 2024-09-20T22:45:19.169 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 42 to 60 2024-09-20T22:45:19.170 INFO:tasks.rados.rados.0.smithi084.stdout:756: delete oid 12 current snap is 75 2024-09-20T22:45:19.173 INFO:tasks.rados.rados.0.smithi084.stdout:754: finishing rollback tid 0 to smithi08477203-48 2024-09-20T22:45:19.173 INFO:tasks.rados.rados.0.smithi084.stdout:753: finishing write tid 3 to smithi08477203-26 2024-09-20T22:45:19.173 INFO:tasks.rados.rados.0.smithi084.stdout:754: finishing rollback tid 1 to smithi08477203-48 2024-09-20T22:45:19.173 INFO:tasks.rados.rados.0.smithi084.stdout:753: finishing write tid 4 to smithi08477203-26 2024-09-20T22:45:19.174 INFO:tasks.rados.rados.0.smithi084.stdout:754: finishing rollback tid 2 to smithi08477203-48 2024-09-20T22:45:19.174 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 48 v 149 (ObjNum 257 snap 60 seq_num 257) dirty exists 2024-09-20T22:45:19.174 INFO:tasks.rados.rados.0.smithi084.stdout:753: finishing write tid 5 to smithi08477203-26 2024-09-20T22:45:19.174 INFO:tasks.rados.rados.0.smithi084.stdout:753: oid 26 updating version 0 to 189 2024-09-20T22:45:19.174 INFO:tasks.rados.rados.0.smithi084.stdout:753: oid 26 version 189 is already newer than 188 2024-09-20T22:45:19.174 INFO:tasks.rados.rados.0.smithi084.stdout:753: oid 26 version 189 is already newer than 187 2024-09-20T22:45:19.174 INFO:tasks.rados.rados.0.smithi084.stdout:753: oid 26 updating version 189 to 190 2024-09-20T22:45:19.174 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 26 v 190 (ObjNum 295 snap 75 seq_num 295) dirty exists 2024-09-20T22:45:19.175 INFO:tasks.rados.rados.0.smithi084.stdout:753: left oid 26 (ObjNum 295 snap 75 seq_num 295) 2024-09-20T22:45:19.175 INFO:tasks.rados.rados.0.smithi084.stdout:753: done (3 left) 2024-09-20T22:45:19.175 INFO:tasks.rados.rados.0.smithi084.stdout:754: done (2 left) 2024-09-20T22:45:19.175 INFO:tasks.rados.rados.0.smithi084.stdout:756: done (1 left) 2024-09-20T22:45:19.175 INFO:tasks.rados.rados.0.smithi084.stdout:757: rollback oid 18 current snap is 75 2024-09-20T22:45:19.175 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 18 to 69 2024-09-20T22:45:19.175 INFO:tasks.rados.rados.0.smithi084.stdout:758: delete oid 29 current snap is 75 2024-09-20T22:45:19.175 INFO:tasks.rados.rados.0.smithi084.stdout:755: finishing rollback tid 1 to smithi08477203-42 2024-09-20T22:45:19.175 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 42 v 0 (ObjNum 145 snap 0 seq_num 2090975744) dirty dne 2024-09-20T22:45:19.176 INFO:tasks.rados.rados.0.smithi084.stdout:755: done (2 left) 2024-09-20T22:45:19.176 INFO:tasks.rados.rados.0.smithi084.stdout:758: done (1 left) 2024-09-20T22:45:19.176 INFO:tasks.rados.rados.0.smithi084.stdout:759: write oid 10 current snap is 75 2024-09-20T22:45:19.176 INFO:tasks.rados.rados.0.smithi084.stdout:759: seq_num 298 ranges {608486=561969,1871634=634092,2950640=57846} 2024-09-20T22:45:19.182 INFO:tasks.rados.rados.0.smithi084.stdout:759: writing smithi08477203-10 from 608486 to 1170455 tid 1 2024-09-20T22:45:19.184 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:45:19.189 INFO:tasks.rados.rados.0.smithi084.stdout:759: writing smithi08477203-10 from 1871634 to 2505726 tid 2 2024-09-20T22:45:19.192 INFO:tasks.rados.rados.0.smithi084.stdout:759: writing smithi08477203-10 from 2950640 to 3008486 tid 3 2024-09-20T22:45:19.192 INFO:tasks.rados.rados.0.smithi084.stdout:760: read oid 45 snap -1 2024-09-20T22:45:19.192 INFO:tasks.rados.rados.0.smithi084.stdout:760: expect (ObjNum 184 snap 39 seq_num 184) 2024-09-20T22:45:19.193 INFO:tasks.rados.rados.0.smithi084.stdout:761: write (excl) oid 20 current snap is 75 2024-09-20T22:45:19.193 INFO:tasks.rados.rados.0.smithi084.stdout:757: finishing rollback tid 0 to smithi08477203-18 2024-09-20T22:45:19.193 INFO:tasks.rados.rados.0.smithi084.stdout:761: seq_num 299 ranges {625293=637218,1813763=494103,2978063=47230} 2024-09-20T22:45:19.199 INFO:tasks.rados.rados.0.smithi084.stdout:761: writing smithi08477203-20 from 625293 to 1262511 tid 1 2024-09-20T22:45:19.205 INFO:tasks.rados.rados.0.smithi084.stdout:761: writing smithi08477203-20 from 1813763 to 2307866 tid 2 2024-09-20T22:45:19.209 INFO:tasks.rados.rados.0.smithi084.stdout:761: writing smithi08477203-20 from 2978063 to 3025293 tid 3 2024-09-20T22:45:19.209 INFO:tasks.rados.rados.0.smithi084.stdout:762: snap_remove snap 74 2024-09-20T22:45:19.210 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:19.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:19.211 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:45:19.311 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:45:19.333 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:19.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:19.334 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:45:19.364 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:19 smithi084 ceph-mon[29774]: pgmap v259: 178 pgs: 1 active+clean+snaptrim, 177 active+clean; 128 MiB data, 574 MiB used, 625 GiB / 626 GiB avail 2024-09-20T22:45:19.364 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:19 smithi084 ceph-mon[29774]: 3.5 scrub starts 2024-09-20T22:45:19.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:19 smithi084 ceph-mon[29774]: 3.5 scrub ok 2024-09-20T22:45:19.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:19 smithi084 ceph-mon[29774]: osdmap e240: 8 total, 8 up, 7 in 2024-09-20T22:45:19.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:19 smithi084 ceph-mon[40024]: pgmap v259: 178 pgs: 1 active+clean+snaptrim, 177 active+clean; 128 MiB data, 574 MiB used, 625 GiB / 626 GiB avail 2024-09-20T22:45:19.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:19 smithi084 ceph-mon[40024]: 3.5 scrub starts 2024-09-20T22:45:19.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:19 smithi084 ceph-mon[40024]: 3.5 scrub ok 2024-09-20T22:45:19.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:19 smithi084 ceph-mon[40024]: osdmap e240: 8 total, 8 up, 7 in 2024-09-20T22:45:19.435 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:45:19.449 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-20T22:45:19.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:19.458 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:19.458 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:45:19.466 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-20T22:45:19.504 INFO:journalctl@ceph.iscsi.iscsi.a.smithi154.stdout:Sep 20 22:45:19 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-iscsi-iscsi-a[59730]: debug there is no tcmu-runner data available 2024-09-20T22:45:19.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:19 smithi154 ceph-mon[33264]: pgmap v259: 178 pgs: 1 active+clean+snaptrim, 177 active+clean; 128 MiB data, 574 MiB used, 625 GiB / 626 GiB avail 2024-09-20T22:45:19.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:19 smithi154 ceph-mon[33264]: 3.5 scrub starts 2024-09-20T22:45:19.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:19 smithi154 ceph-mon[33264]: 3.5 scrub ok 2024-09-20T22:45:19.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:19 smithi154 ceph-mon[33264]: osdmap e240: 8 total, 8 up, 7 in 2024-09-20T22:45:19.559 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:45:19.580 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:19.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:19.581 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:45:19.681 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:45:19.703 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:19.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:19.704 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:45:19.804 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:45:19.830 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:19.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:19.831 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:45:19.932 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:45:19.954 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:19.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:19.955 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:45:20.000 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:45:20.004 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:45:20.056 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:45:20.077 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:20.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:20.078 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:45:20.155 INFO:tasks.rados.rados.0.smithi084.stdout:762: done (4 left) 2024-09-20T22:45:20.155 INFO:tasks.rados.rados.0.smithi084.stdout:763: write oid 2 current snap is 75 2024-09-20T22:45:20.155 INFO:tasks.rados.rados.0.smithi084.stdout:763: seq_num 300 ranges {642100=712467,1872245=637761} 2024-09-20T22:45:20.164 INFO:tasks.rados.rados.0.smithi084.stdout:763: writing smithi08477203-2 from 642100 to 1354567 tid 1 2024-09-20T22:45:20.172 INFO:tasks.rados.rados.0.smithi084.stdout:763: writing smithi08477203-2 from 1872245 to 2510006 tid 2 2024-09-20T22:45:20.172 INFO:tasks.rados.rados.0.smithi084.stdout:764: delete oid 30 current snap is 75 2024-09-20T22:45:20.172 INFO:tasks.rados.rados.0.smithi084.stdout:757: finishing rollback tid 1 to smithi08477203-18 2024-09-20T22:45:20.172 INFO:tasks.rados.rados.0.smithi084.stdout:757: finishing rollback tid 2 to smithi08477203-18 2024-09-20T22:45:20.172 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 18 v 168 (ObjNum 273 snap 69 seq_num 273) dirty exists 2024-09-20T22:45:20.172 INFO:tasks.rados.rados.0.smithi084.stdout:759: finishing write tid 1 to smithi08477203-10 2024-09-20T22:45:20.173 INFO:tasks.rados.rados.0.smithi084.stdout:759: finishing write tid 2 to smithi08477203-10 2024-09-20T22:45:20.173 INFO:tasks.rados.rados.0.smithi084.stdout:759: finishing write tid 3 to smithi08477203-10 2024-09-20T22:45:20.173 INFO:tasks.rados.rados.0.smithi084.stdout:759: finishing write tid 4 to smithi08477203-10 2024-09-20T22:45:20.173 INFO:tasks.rados.rados.0.smithi084.stdout:759: finishing write tid 5 to smithi08477203-10 2024-09-20T22:45:20.173 INFO:tasks.rados.rados.0.smithi084.stdout:759: oid 10 updating version 0 to 172 2024-09-20T22:45:20.173 INFO:tasks.rados.rados.0.smithi084.stdout:759: oid 10 version 172 is already newer than 171 2024-09-20T22:45:20.173 INFO:tasks.rados.rados.0.smithi084.stdout:759: oid 10 version 172 is already newer than 169 2024-09-20T22:45:20.173 INFO:tasks.rados.rados.0.smithi084.stdout:759: oid 10 version 172 is already newer than 170 2024-09-20T22:45:20.173 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 10 v 172 (ObjNum 298 snap 75 seq_num 298) dirty exists 2024-09-20T22:45:20.173 INFO:tasks.rados.rados.0.smithi084.stdout:759: left oid 10 (ObjNum 298 snap 75 seq_num 298) 2024-09-20T22:45:20.173 INFO:tasks.rados.rados.0.smithi084.stdout:761: finishing write tid 1 to smithi08477203-20 2024-09-20T22:45:20.173 INFO:tasks.rados.rados.0.smithi084.stdout:760: expect (ObjNum 184 snap 39 seq_num 184) 2024-09-20T22:45:20.179 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:45:20.202 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:20.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:20.203 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:45:20.280 INFO:tasks.rados.rados.0.smithi084.stdout:761: finishing write tid 2 to smithi08477203-20 2024-09-20T22:45:20.280 INFO:tasks.rados.rados.0.smithi084.stdout:761: finishing write tid 3 to smithi08477203-20 2024-09-20T22:45:20.280 INFO:tasks.rados.rados.0.smithi084.stdout:757: done (5 left) 2024-09-20T22:45:20.280 INFO:tasks.rados.rados.0.smithi084.stdout:759: done (4 left) 2024-09-20T22:45:20.281 INFO:tasks.rados.rados.0.smithi084.stdout:760: done (3 left) 2024-09-20T22:45:20.281 INFO:tasks.rados.rados.0.smithi084.stdout:764: done (2 left) 2024-09-20T22:45:20.281 INFO:tasks.rados.rados.0.smithi084.stdout:765: write (excl) oid 19 current snap is 75 2024-09-20T22:45:20.281 INFO:tasks.rados.rados.0.smithi084.stdout:761: finishing write tid 4 to smithi08477203-20 2024-09-20T22:45:20.281 INFO:tasks.rados.rados.0.smithi084.stdout:761: finishing write tid 5 to smithi08477203-20 2024-09-20T22:45:20.281 INFO:tasks.rados.rados.0.smithi084.stdout:761: oid 20 updating version 0 to 62 2024-09-20T22:45:20.281 INFO:tasks.rados.rados.0.smithi084.stdout:761: oid 20 updating version 62 to 64 2024-09-20T22:45:20.281 INFO:tasks.rados.rados.0.smithi084.stdout:761: oid 20 version 64 is already newer than 61 2024-09-20T22:45:20.281 INFO:tasks.rados.rados.0.smithi084.stdout:761: oid 20 version 64 is already newer than 63 2024-09-20T22:45:20.281 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 20 v 64 (ObjNum 299 snap 75 seq_num 299) dirty exists 2024-09-20T22:45:20.281 INFO:tasks.rados.rados.0.smithi084.stdout:761: left oid 20 (ObjNum 299 snap 75 seq_num 299) 2024-09-20T22:45:20.281 INFO:tasks.rados.rados.0.smithi084.stdout:763: finishing write tid 1 to smithi08477203-2 2024-09-20T22:45:20.281 INFO:tasks.rados.rados.0.smithi084.stdout:765: seq_num 302 ranges {675714=462965,1589209=641430,2943979=131735} 2024-09-20T22:45:20.286 INFO:tasks.rados.rados.0.smithi084.stdout:765: writing smithi08477203-19 from 675714 to 1138679 tid 1 2024-09-20T22:45:20.292 INFO:tasks.rados.rados.0.smithi084.stdout:765: writing smithi08477203-19 from 1589209 to 2230639 tid 2 2024-09-20T22:45:20.298 INFO:tasks.rados.rados.0.smithi084.stdout:765: writing smithi08477203-19 from 2943979 to 3075714 tid 3 2024-09-20T22:45:20.298 INFO:tasks.rados.rados.0.smithi084.stdout:761: done (2 left) 2024-09-20T22:45:20.298 INFO:tasks.rados.rados.0.smithi084.stdout:766: delete oid 38 current snap is 75 2024-09-20T22:45:20.298 INFO:tasks.rados.rados.0.smithi084.stdout:763: finishing write tid 2 to smithi08477203-2 2024-09-20T22:45:20.298 INFO:tasks.rados.rados.0.smithi084.stdout:763: finishing write tid 3 to smithi08477203-2 2024-09-20T22:45:20.298 INFO:tasks.rados.rados.0.smithi084.stdout:763: finishing write tid 4 to smithi08477203-2 2024-09-20T22:45:20.298 INFO:tasks.rados.rados.0.smithi084.stdout:763: oid 2 updating version 0 to 96 2024-09-20T22:45:20.298 INFO:tasks.rados.rados.0.smithi084.stdout:763: oid 2 updating version 96 to 98 2024-09-20T22:45:20.298 INFO:tasks.rados.rados.0.smithi084.stdout:763: oid 2 version 98 is already newer than 97 2024-09-20T22:45:20.298 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 2 v 98 (ObjNum 300 snap 75 seq_num 300) dirty exists 2024-09-20T22:45:20.298 INFO:tasks.rados.rados.0.smithi084.stdout:763: left oid 2 (ObjNum 300 snap 75 seq_num 300) 2024-09-20T22:45:20.298 INFO:tasks.rados.rados.0.smithi084.stdout:765: finishing write tid 1 to smithi08477203-19 2024-09-20T22:45:20.299 INFO:tasks.rados.rados.0.smithi084.stdout:765: finishing write tid 2 to smithi08477203-19 2024-09-20T22:45:20.299 INFO:tasks.rados.rados.0.smithi084.stdout:763: done (2 left) 2024-09-20T22:45:20.300 INFO:tasks.rados.rados.0.smithi084.stdout:766: done (1 left) 2024-09-20T22:45:20.300 INFO:tasks.rados.rados.0.smithi084.stdout:767: read oid 7 snap -1 2024-09-20T22:45:20.300 INFO:tasks.rados.rados.0.smithi084.stdout:767: expect (ObjNum 260 snap 61 seq_num 260) 2024-09-20T22:45:20.300 INFO:tasks.rados.rados.0.smithi084.stdout:768: delete oid 27 current snap is 75 2024-09-20T22:45:20.300 INFO:tasks.rados.rados.0.smithi084.stdout:765: finishing write tid 3 to smithi08477203-19 2024-09-20T22:45:20.300 INFO:tasks.rados.rados.0.smithi084.stdout:765: finishing write tid 4 to smithi08477203-19 2024-09-20T22:45:20.300 INFO:tasks.rados.rados.0.smithi084.stdout:765: finishing write tid 5 to smithi08477203-19 2024-09-20T22:45:20.300 INFO:tasks.rados.rados.0.smithi084.stdout:765: oid 19 updating version 0 to 349 2024-09-20T22:45:20.300 INFO:tasks.rados.rados.0.smithi084.stdout:765: oid 19 version 349 is already newer than 346 2024-09-20T22:45:20.300 INFO:tasks.rados.rados.0.smithi084.stdout:765: oid 19 version 349 is already newer than 348 2024-09-20T22:45:20.300 INFO:tasks.rados.rados.0.smithi084.stdout:765: oid 19 version 349 is already newer than 347 2024-09-20T22:45:20.300 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 19 v 349 (ObjNum 302 snap 75 seq_num 302) dirty exists 2024-09-20T22:45:20.300 INFO:tasks.rados.rados.0.smithi084.stdout:765: left oid 19 (ObjNum 302 snap 75 seq_num 302) 2024-09-20T22:45:20.301 INFO:tasks.rados.rados.0.smithi084.stdout:765: done (2 left) 2024-09-20T22:45:20.301 INFO:tasks.rados.rados.0.smithi084.stdout:768: done (1 left) 2024-09-20T22:45:20.301 INFO:tasks.rados.rados.0.smithi084.stdout:769: rollback oid 3 current snap is 75 2024-09-20T22:45:20.301 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 3 to 69 2024-09-20T22:45:20.301 INFO:tasks.rados.rados.0.smithi084.stdout:770: read oid 38 snap -1 2024-09-20T22:45:20.301 INFO:tasks.rados.rados.0.smithi084.stdout:770: expect deleted 2024-09-20T22:45:20.301 INFO:tasks.rados.rados.0.smithi084.stdout:771: delete oid 48 current snap is 75 2024-09-20T22:45:20.303 INFO:tasks.rados.rados.0.smithi084.stdout:769: finishing rollback tid 0 to smithi08477203-3 2024-09-20T22:45:20.303 INFO:tasks.rados.rados.0.smithi084.stdout:769: finishing rollback tid 1 to smithi08477203-3 2024-09-20T22:45:20.303 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 3 v 119 (ObjNum 225 snap 0 seq_num 2090981616) dirty dne 2024-09-20T22:45:20.304 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:45:20.305 INFO:tasks.rados.rados.0.smithi084.stdout:767: expect (ObjNum 260 snap 61 seq_num 260) 2024-09-20T22:45:20.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:20.328 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:20.328 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:45:20.378 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:20 smithi084 ceph-mon[40024]: 3.8 scrub starts 2024-09-20T22:45:20.378 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:20 smithi084 ceph-mon[40024]: 3.8 scrub ok 2024-09-20T22:45:20.378 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:20 smithi084 ceph-mon[40024]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:45:20.378 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:20 smithi084 ceph-mon[40024]: 6.12 scrub starts 2024-09-20T22:45:20.378 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:20 smithi084 ceph-mon[40024]: 6.12 scrub ok 2024-09-20T22:45:20.378 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:20 smithi084 ceph-mon[40024]: 3.9 scrub starts 2024-09-20T22:45:20.379 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:20 smithi084 ceph-mon[40024]: 3.9 scrub ok 2024-09-20T22:45:20.379 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:20 smithi084 ceph-mon[40024]: osdmap e241: 8 total, 8 up, 7 in 2024-09-20T22:45:20.379 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:20 smithi084 ceph-mon[40024]: purged_snaps scrub starts 2024-09-20T22:45:20.379 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:20 smithi084 ceph-mon[40024]: purged_snaps scrub ok 2024-09-20T22:45:20.379 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:20 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/1946130109' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:45:20.379 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:20 smithi084 ceph-mon[29774]: 3.8 scrub starts 2024-09-20T22:45:20.379 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:20 smithi084 ceph-mon[29774]: 3.8 scrub ok 2024-09-20T22:45:20.379 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:20 smithi084 ceph-mon[29774]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:45:20.379 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:20 smithi084 ceph-mon[29774]: 6.12 scrub starts 2024-09-20T22:45:20.379 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:20 smithi084 ceph-mon[29774]: 6.12 scrub ok 2024-09-20T22:45:20.379 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:20 smithi084 ceph-mon[29774]: 3.9 scrub starts 2024-09-20T22:45:20.379 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:20 smithi084 ceph-mon[29774]: 3.9 scrub ok 2024-09-20T22:45:20.380 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:20 smithi084 ceph-mon[29774]: osdmap e241: 8 total, 8 up, 7 in 2024-09-20T22:45:20.380 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:20 smithi084 ceph-mon[29774]: purged_snaps scrub starts 2024-09-20T22:45:20.380 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:20 smithi084 ceph-mon[29774]: purged_snaps scrub ok 2024-09-20T22:45:20.380 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:20 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/1946130109' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:45:20.385 INFO:tasks.rados.rados.0.smithi084.stdout:767: done (3 left) 2024-09-20T22:45:20.385 INFO:tasks.rados.rados.0.smithi084.stdout:769: done (2 left) 2024-09-20T22:45:20.385 INFO:tasks.rados.rados.0.smithi084.stdout:770: done (1 left) 2024-09-20T22:45:20.386 INFO:tasks.rados.rados.0.smithi084.stdout:771: done (0 left) 2024-09-20T22:45:20.386 INFO:tasks.rados.rados.0.smithi084.stdout:772: copy_from oid 19 from oid 11 current snap is 75 2024-09-20T22:45:20.386 INFO:tasks.rados.rados.0.smithi084.stdout:773: rollback oid 44 current snap is 75 2024-09-20T22:45:20.386 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 44 to 73 2024-09-20T22:45:20.386 INFO:tasks.rados.rados.0.smithi084.stdout:774: write (excl) oid 14 current snap is 75 2024-09-20T22:45:20.386 INFO:tasks.rados.rados.0.smithi084.stdout:774: seq_num 306 ranges {742942=480314,1823137=648768} 2024-09-20T22:45:20.392 INFO:tasks.rados.rados.0.smithi084.stdout:774: writing smithi08477203-14 from 742942 to 1223256 tid 1 2024-09-20T22:45:20.400 INFO:tasks.rados.rados.0.smithi084.stdout:774: writing smithi08477203-14 from 1823137 to 2471905 tid 2 2024-09-20T22:45:20.400 INFO:tasks.rados.rados.0.smithi084.stdout:775: write (excl) oid 22 current snap is 75 2024-09-20T22:45:20.400 INFO:tasks.rados.rados.0.smithi084.stdout:773: finishing rollback tid 0 to smithi08477203-44 2024-09-20T22:45:20.400 INFO:tasks.rados.rados.0.smithi084.stdout:773: finishing rollback tid 1 to smithi08477203-44 2024-09-20T22:45:20.400 INFO:tasks.rados.rados.0.smithi084.stdout:773: finishing rollback tid 2 to smithi08477203-44 2024-09-20T22:45:20.400 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 44 v 353 (ObjNum 129 snap 23 seq_num 129) dirty exists 2024-09-20T22:45:20.400 INFO:tasks.rados.rados.0.smithi084.stdout:772: finishing copy_from racing read to smithi08477203-19 2024-09-20T22:45:20.400 INFO:tasks.rados.rados.0.smithi084.stdout:772: finishing copy_from to smithi08477203-19 2024-09-20T22:45:20.400 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 19 v 101 (ObjNum 245 snap 54 seq_num 245) dirty exists 2024-09-20T22:45:20.400 INFO:tasks.rados.rados.0.smithi084.stdout:774: finishing write tid 1 to smithi08477203-14 2024-09-20T22:45:20.401 INFO:tasks.rados.rados.0.smithi084.stdout:775: seq_num 307 ranges {759749=555563,1765266=792426,2964741=195008} 2024-09-20T22:45:20.407 INFO:tasks.rados.rados.0.smithi084.stdout:775: writing smithi08477203-22 from 759749 to 1315312 tid 1 2024-09-20T22:45:20.414 INFO:tasks.rados.rados.0.smithi084.stdout:775: writing smithi08477203-22 from 1765266 to 2557692 tid 2 2024-09-20T22:45:20.418 INFO:tasks.rados.rados.0.smithi084.stdout:775: writing smithi08477203-22 from 2964741 to 3159749 tid 3 2024-09-20T22:45:20.418 INFO:tasks.rados.rados.0.smithi084.stdout:772: done (3 left) 2024-09-20T22:45:20.418 INFO:tasks.rados.rados.0.smithi084.stdout:773: done (2 left) 2024-09-20T22:45:20.418 INFO:tasks.rados.rados.0.smithi084.stdout:776: copy_from oid 28 from oid 29 current snap is 75 2024-09-20T22:45:20.418 INFO:tasks.rados.rados.0.smithi084.stdout:774: finishing write tid 2 to smithi08477203-14 2024-09-20T22:45:20.419 INFO:tasks.rados.rados.0.smithi084.stdout:774: finishing write tid 3 to smithi08477203-14 2024-09-20T22:45:20.419 INFO:tasks.rados.rados.0.smithi084.stdout:774: finishing write tid 4 to smithi08477203-14 2024-09-20T22:45:20.419 INFO:tasks.rados.rados.0.smithi084.stdout:774: oid 14 updating version 0 to 68 2024-09-20T22:45:20.419 INFO:tasks.rados.rados.0.smithi084.stdout:774: oid 14 version 68 is already newer than 66 2024-09-20T22:45:20.419 INFO:tasks.rados.rados.0.smithi084.stdout:774: oid 14 version 68 is already newer than 67 2024-09-20T22:45:20.419 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 14 v 68 (ObjNum 306 snap 75 seq_num 306) dirty exists 2024-09-20T22:45:20.419 INFO:tasks.rados.rados.0.smithi084.stdout:774: left oid 14 (ObjNum 306 snap 75 seq_num 306) 2024-09-20T22:45:20.419 INFO:tasks.rados.rados.0.smithi084.stdout:775: finishing write tid 1 to smithi08477203-22 2024-09-20T22:45:20.419 INFO:tasks.rados.rados.0.smithi084.stdout:775: finishing write tid 2 to smithi08477203-22 2024-09-20T22:45:20.419 INFO:tasks.rados.rados.0.smithi084.stdout:774: done (2 left) 2024-09-20T22:45:20.419 INFO:tasks.rados.rados.0.smithi084.stdout:777: rollback oid 30 current snap is 75 2024-09-20T22:45:20.419 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 30 to 69 2024-09-20T22:45:20.419 INFO:tasks.rados.rados.0.smithi084.stdout:778: delete oid 45 current snap is 75 2024-09-20T22:45:20.420 INFO:tasks.rados.rados.0.smithi084.stdout:776: finishing copy_from to smithi08477203-28 2024-09-20T22:45:20.420 INFO:tasks.rados.rados.0.smithi084.stdout:776: got expected ENOENT (src dne) 2024-09-20T22:45:20.421 INFO:tasks.rados.rados.0.smithi084.stdout:776: finishing copy_from racing read to smithi08477203-28 2024-09-20T22:45:20.421 INFO:tasks.rados.rados.0.smithi084.stdout:777: finishing rollback tid 1 to smithi08477203-30 2024-09-20T22:45:20.421 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 30 v 146 (ObjNum 49 snap 0 seq_num 2085656144) dirty dne 2024-09-20T22:45:20.421 INFO:tasks.rados.rados.0.smithi084.stdout:775: finishing write tid 3 to smithi08477203-22 2024-09-20T22:45:20.421 INFO:tasks.rados.rados.0.smithi084.stdout:776: done (3 left) 2024-09-20T22:45:20.421 INFO:tasks.rados.rados.0.smithi084.stdout:777: done (2 left) 2024-09-20T22:45:20.421 INFO:tasks.rados.rados.0.smithi084.stdout:778: done (1 left) 2024-09-20T22:45:20.421 INFO:tasks.rados.rados.0.smithi084.stdout:779: write oid 6 current snap is 75 2024-09-20T22:45:20.421 INFO:tasks.rados.rados.0.smithi084.stdout:779: seq_num 309 ranges {793363=706061,2282230=796095} 2024-09-20T22:45:20.428 INFO:tasks.rados.rados.0.smithi084.stdout:779: writing smithi08477203-6 from 793363 to 1499424 tid 1 2024-09-20T22:45:20.429 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:45:20.438 INFO:tasks.rados.rados.0.smithi084.stdout:779: writing smithi08477203-6 from 2282230 to 3078325 tid 2 2024-09-20T22:45:20.438 INFO:tasks.rados.rados.0.smithi084.stdout:780: read oid 47 snap -1 2024-09-20T22:45:20.438 INFO:tasks.rados.rados.0.smithi084.stdout:780: expect (ObjNum 208 snap 42 seq_num 208) 2024-09-20T22:45:20.438 INFO:tasks.rados.rados.0.smithi084.stdout:781: read oid 18 snap 67 2024-09-20T22:45:20.438 INFO:tasks.rados.rados.0.smithi084.stdout:781: expect (ObjNum 167 snap 36 seq_num 167) 2024-09-20T22:45:20.438 INFO:tasks.rados.rados.0.smithi084.stdout:782: read oid 27 snap -1 2024-09-20T22:45:20.438 INFO:tasks.rados.rados.0.smithi084.stdout:782: expect deleted 2024-09-20T22:45:20.439 INFO:tasks.rados.rados.0.smithi084.stdout:783: read oid 37 snap -1 2024-09-20T22:45:20.439 INFO:tasks.rados.rados.0.smithi084.stdout:783: expect (ObjNum 285 snap 71 seq_num 285) 2024-09-20T22:45:20.439 INFO:tasks.rados.rados.0.smithi084.stdout:784: snap_create 2024-09-20T22:45:20.439 INFO:tasks.rados.rados.0.smithi084.stdout:775: finishing write tid 4 to smithi08477203-22 2024-09-20T22:45:20.439 INFO:tasks.rados.rados.0.smithi084.stdout:775: finishing write tid 5 to smithi08477203-22 2024-09-20T22:45:20.439 INFO:tasks.rados.rados.0.smithi084.stdout:775: oid 22 updating version 0 to 166 2024-09-20T22:45:20.439 INFO:tasks.rados.rados.0.smithi084.stdout:775: oid 22 version 166 is already newer than 164 2024-09-20T22:45:20.439 INFO:tasks.rados.rados.0.smithi084.stdout:775: oid 22 version 166 is already newer than 165 2024-09-20T22:45:20.439 INFO:tasks.rados.rados.0.smithi084.stdout:775: oid 22 version 166 is already newer than 163 2024-09-20T22:45:20.439 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 22 v 166 (ObjNum 307 snap 75 seq_num 307) dirty exists 2024-09-20T22:45:20.439 INFO:tasks.rados.rados.0.smithi084.stdout:775: left oid 22 (ObjNum 307 snap 75 seq_num 307) 2024-09-20T22:45:20.440 INFO:tasks.rados.rados.0.smithi084.stdout:779: finishing write tid 1 to smithi08477203-6 2024-09-20T22:45:20.440 INFO:tasks.rados.rados.0.smithi084.stdout:779: finishing write tid 2 to smithi08477203-6 2024-09-20T22:45:20.441 INFO:tasks.rados.rados.0.smithi084.stdout:779: finishing write tid 3 to smithi08477203-6 2024-09-20T22:45:20.441 INFO:tasks.rados.rados.0.smithi084.stdout:779: finishing write tid 4 to smithi08477203-6 2024-09-20T22:45:20.441 INFO:tasks.rados.rados.0.smithi084.stdout:779: oid 6 updating version 0 to 97 2024-09-20T22:45:20.441 INFO:tasks.rados.rados.0.smithi084.stdout:779: oid 6 version 97 is already newer than 95 2024-09-20T22:45:20.441 INFO:tasks.rados.rados.0.smithi084.stdout:779: oid 6 version 97 is already newer than 96 2024-09-20T22:45:20.442 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 6 v 97 (ObjNum 309 snap 75 seq_num 309) dirty exists 2024-09-20T22:45:20.442 INFO:tasks.rados.rados.0.smithi084.stdout:779: left oid 6 (ObjNum 309 snap 75 seq_num 309) 2024-09-20T22:45:20.445 INFO:tasks.rados.rados.0.smithi084.stdout:781: expect (ObjNum 167 snap 36 seq_num 167) 2024-09-20T22:45:20.456 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:20.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:20.457 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:45:20.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:20 smithi154 ceph-mon[33264]: 3.8 scrub starts 2024-09-20T22:45:20.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:20 smithi154 ceph-mon[33264]: 3.8 scrub ok 2024-09-20T22:45:20.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:20 smithi154 ceph-mon[33264]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:45:20.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:20 smithi154 ceph-mon[33264]: 6.12 scrub starts 2024-09-20T22:45:20.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:20 smithi154 ceph-mon[33264]: 6.12 scrub ok 2024-09-20T22:45:20.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:20 smithi154 ceph-mon[33264]: 3.9 scrub starts 2024-09-20T22:45:20.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:20 smithi154 ceph-mon[33264]: 3.9 scrub ok 2024-09-20T22:45:20.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:20 smithi154 ceph-mon[33264]: osdmap e241: 8 total, 8 up, 7 in 2024-09-20T22:45:20.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:20 smithi154 ceph-mon[33264]: purged_snaps scrub starts 2024-09-20T22:45:20.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:20 smithi154 ceph-mon[33264]: purged_snaps scrub ok 2024-09-20T22:45:20.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:20 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/1946130109' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:45:20.536 INFO:tasks.rados.rados.0.smithi084.stdout:783: expect (ObjNum 285 snap 71 seq_num 285) 2024-09-20T22:45:20.558 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:45:20.580 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:20.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:20.581 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:45:20.681 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:45:20.681 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:45:20.682 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:45:20.697 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:45:20.697 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:45:20.706 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:20.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:20.708 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:45:20.713 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:45:20.714 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:45:20.729 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-20T22:45:20.730 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T22:45:20.730 INFO:tasks.rados.rados.0.smithi084.stdout:780: expect (ObjNum 208 snap 42 seq_num 208) 2024-09-20T22:45:20.746 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:45:20.746 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:45:20.763 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:45:20.763 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:45:20.780 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:45:20.780 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:45:20.796 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:45:20.796 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:45:20.809 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:45:20.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:20.832 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:20.833 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:45:20.933 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:45:20.956 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:20.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:20.957 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:45:21.058 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:45:21.080 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:21.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:21.081 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:45:21.149 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is unset 2024-09-20T22:45:21.153 INFO:tasks.rados.rados.0.smithi084.stdout:775: done (6 left) 2024-09-20T22:45:21.153 INFO:tasks.rados.rados.0.smithi084.stdout:779: done (5 left) 2024-09-20T22:45:21.153 INFO:tasks.rados.rados.0.smithi084.stdout:780: done (4 left) 2024-09-20T22:45:21.153 INFO:tasks.rados.rados.0.smithi084.stdout:781: done (3 left) 2024-09-20T22:45:21.153 INFO:tasks.rados.rados.0.smithi084.stdout:782: done (2 left) 2024-09-20T22:45:21.153 INFO:tasks.rados.rados.0.smithi084.stdout:783: done (1 left) 2024-09-20T22:45:21.153 INFO:tasks.rados.rados.0.smithi084.stdout:784: done (0 left) 2024-09-20T22:45:21.153 INFO:tasks.rados.rados.0.smithi084.stdout:785: delete oid 5 current snap is 76 2024-09-20T22:45:21.155 INFO:tasks.rados.rados.0.smithi084.stdout:785: done (0 left) 2024-09-20T22:45:21.155 INFO:tasks.rados.rados.0.smithi084.stdout:786: rollback oid 43 current snap is 76 2024-09-20T22:45:21.155 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 43 to 73 2024-09-20T22:45:21.155 INFO:tasks.rados.rados.0.smithi084.stdout:787: write (excl) oid 28 current snap is 76 2024-09-20T22:45:21.155 INFO:tasks.rados.rados.0.smithi084.stdout:787: seq_num 311 ranges {426977=456559,1482841=799764,2958080=268897} 2024-09-20T22:45:21.160 INFO:tasks.rados.rados.0.smithi084.stdout:787: writing smithi08477203-28 from 426977 to 883536 tid 1 2024-09-20T22:45:21.168 INFO:tasks.rados.rados.0.smithi084.stdout:787: writing smithi08477203-28 from 1482841 to 2282605 tid 2 2024-09-20T22:45:21.173 INFO:tasks.rados.rados.0.smithi084.stdout:787: writing smithi08477203-28 from 2958080 to 3226977 tid 3 2024-09-20T22:45:21.174 INFO:tasks.rados.rados.0.smithi084.stdout:788: snap_create 2024-09-20T22:45:21.174 INFO:tasks.rados.rados.0.smithi084.stdout:786: finishing rollback tid 1 to smithi08477203-43 2024-09-20T22:45:21.174 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 43 v 0 (ObjNum 225 snap 0 seq_num 2091058144) dirty dne 2024-09-20T22:45:21.174 INFO:tasks.rados.rados.0.smithi084.stdout:787: finishing write tid 1 to smithi08477203-28 2024-09-20T22:45:21.174 INFO:tasks.rados.rados.0.smithi084.stdout:787: finishing write tid 2 to smithi08477203-28 2024-09-20T22:45:21.176 INFO:tasks.rados.rados.0.smithi084.stdout:787: finishing write tid 3 to smithi08477203-28 2024-09-20T22:45:21.176 INFO:tasks.rados.rados.0.smithi084.stdout:787: finishing write tid 4 to smithi08477203-28 2024-09-20T22:45:21.176 INFO:tasks.rados.rados.0.smithi084.stdout:787: finishing write tid 5 to smithi08477203-28 2024-09-20T22:45:21.176 INFO:tasks.rados.rados.0.smithi084.stdout:787: oid 28 updating version 0 to 357 2024-09-20T22:45:21.176 INFO:tasks.rados.rados.0.smithi084.stdout:787: oid 28 version 357 is already newer than 355 2024-09-20T22:45:21.176 INFO:tasks.rados.rados.0.smithi084.stdout:787: oid 28 version 357 is already newer than 356 2024-09-20T22:45:21.176 INFO:tasks.rados.rados.0.smithi084.stdout:787: oid 28 updating version 357 to 358 2024-09-20T22:45:21.176 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 28 v 358 (ObjNum 311 snap 76 seq_num 311) dirty exists 2024-09-20T22:45:21.176 INFO:tasks.rados.rados.0.smithi084.stdout:787: left oid 28 (ObjNum 311 snap 76 seq_num 311) 2024-09-20T22:45:21.182 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:45:21.185 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-20T22:45:21.185 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:45:21.203 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:21.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:21.204 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:45:21.305 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:45:21.327 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:21.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:21.328 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:45:21.428 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:45:21.450 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:21.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:21.451 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:45:21.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:21 smithi154 ceph-mon[33264]: pgmap v262: 178 pgs: 3 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 173 active+clean; 141 MiB data, 600 MiB used, 625 GiB / 626 GiB avail; 18 MiB/s rd, 2.2 MiB/s wr, 27 op/s 2024-09-20T22:45:21.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:21 smithi154 ceph-mon[33264]: 5.15 scrub starts 2024-09-20T22:45:21.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:21 smithi154 ceph-mon[33264]: 5.15 scrub ok 2024-09-20T22:45:21.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:21 smithi154 ceph-mon[33264]: 4.6 scrub starts 2024-09-20T22:45:21.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:21 smithi154 ceph-mon[33264]: 4.6 scrub ok 2024-09-20T22:45:21.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:21 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/1982805228' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:45:21.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:21 smithi154 ceph-mon[33264]: 1.f scrub starts 2024-09-20T22:45:21.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:21 smithi154 ceph-mon[33264]: 1.f scrub ok 2024-09-20T22:45:21.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:21 smithi154 ceph-mon[33264]: 3.4 scrub starts 2024-09-20T22:45:21.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:21 smithi154 ceph-mon[33264]: 3.4 scrub ok 2024-09-20T22:45:21.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:21 smithi154 ceph-mon[33264]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T22:45:21.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:21 smithi154 ceph-mon[33264]: Cluster is now healthy 2024-09-20T22:45:21.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:21 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/1946130109' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T22:45:21.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:21 smithi154 ceph-mon[33264]: osdmap e242: 8 total, 8 up, 7 in 2024-09-20T22:45:21.552 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:45:21.554 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:21 smithi084 ceph-mon[40024]: pgmap v262: 178 pgs: 3 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 173 active+clean; 141 MiB data, 600 MiB used, 625 GiB / 626 GiB avail; 18 MiB/s rd, 2.2 MiB/s wr, 27 op/s 2024-09-20T22:45:21.554 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:21 smithi084 ceph-mon[40024]: 5.15 scrub starts 2024-09-20T22:45:21.554 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:21 smithi084 ceph-mon[40024]: 5.15 scrub ok 2024-09-20T22:45:21.554 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:21 smithi084 ceph-mon[40024]: 4.6 scrub starts 2024-09-20T22:45:21.554 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:21 smithi084 ceph-mon[40024]: 4.6 scrub ok 2024-09-20T22:45:21.555 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:21 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/1982805228' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:45:21.555 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:21 smithi084 ceph-mon[40024]: 1.f scrub starts 2024-09-20T22:45:21.555 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:21 smithi084 ceph-mon[40024]: 1.f scrub ok 2024-09-20T22:45:21.555 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:21 smithi084 ceph-mon[40024]: 3.4 scrub starts 2024-09-20T22:45:21.555 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:21 smithi084 ceph-mon[40024]: 3.4 scrub ok 2024-09-20T22:45:21.555 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:21 smithi084 ceph-mon[40024]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T22:45:21.555 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:21 smithi084 ceph-mon[40024]: Cluster is now healthy 2024-09-20T22:45:21.555 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:21 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/1946130109' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T22:45:21.555 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:21 smithi084 ceph-mon[40024]: osdmap e242: 8 total, 8 up, 7 in 2024-09-20T22:45:21.556 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:21 smithi084 ceph-mon[29774]: pgmap v262: 178 pgs: 3 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 173 active+clean; 141 MiB data, 600 MiB used, 625 GiB / 626 GiB avail; 18 MiB/s rd, 2.2 MiB/s wr, 27 op/s 2024-09-20T22:45:21.556 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:21 smithi084 ceph-mon[29774]: 5.15 scrub starts 2024-09-20T22:45:21.556 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:21 smithi084 ceph-mon[29774]: 5.15 scrub ok 2024-09-20T22:45:21.556 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:21 smithi084 ceph-mon[29774]: 4.6 scrub starts 2024-09-20T22:45:21.556 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:21 smithi084 ceph-mon[29774]: 4.6 scrub ok 2024-09-20T22:45:21.557 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:21 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/1982805228' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:45:21.557 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:21 smithi084 ceph-mon[29774]: 1.f scrub starts 2024-09-20T22:45:21.557 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:21 smithi084 ceph-mon[29774]: 1.f scrub ok 2024-09-20T22:45:21.557 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:21 smithi084 ceph-mon[29774]: 3.4 scrub starts 2024-09-20T22:45:21.557 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:21 smithi084 ceph-mon[29774]: 3.4 scrub ok 2024-09-20T22:45:21.557 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:21 smithi084 ceph-mon[29774]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T22:45:21.557 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:21 smithi084 ceph-mon[29774]: Cluster is now healthy 2024-09-20T22:45:21.557 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:21 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/1946130109' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T22:45:21.557 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:21 smithi084 ceph-mon[29774]: osdmap e242: 8 total, 8 up, 7 in 2024-09-20T22:45:21.575 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:21.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:21.576 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:45:21.676 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:45:21.698 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:21.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:21.699 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:45:21.800 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:45:21.821 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:21.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:21.822 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:45:21.923 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:45:21.945 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:21.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:21.946 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:45:22.046 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:45:22.068 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:22.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:22.069 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:45:22.154 INFO:tasks.rados.rados.0.smithi084.stdout:786: done (2 left) 2024-09-20T22:45:22.155 INFO:tasks.rados.rados.0.smithi084.stdout:787: done (1 left) 2024-09-20T22:45:22.155 INFO:tasks.rados.rados.0.smithi084.stdout:788: done (0 left) 2024-09-20T22:45:22.155 INFO:tasks.rados.rados.0.smithi084.stdout:789: read oid 47 snap 72 2024-09-20T22:45:22.155 INFO:tasks.rados.rados.0.smithi084.stdout:789: expect (ObjNum 208 snap 42 seq_num 208) 2024-09-20T22:45:22.155 INFO:tasks.rados.rados.0.smithi084.stdout:790: write (excl) oid 12 current snap is 77 2024-09-20T22:45:22.155 INFO:tasks.rados.rados.0.smithi084.stdout:790: seq_num 312 ranges {443784=648161,1657676=659775,2818209=425575} 2024-09-20T22:45:22.161 INFO:tasks.rados.rados.0.smithi084.stdout:790: writing smithi08477203-12 from 443784 to 1091945 tid 1 2024-09-20T22:45:22.169 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:45:22.170 INFO:tasks.rados.rados.0.smithi084.stdout:790: writing smithi08477203-12 from 1657676 to 2317451 tid 2 2024-09-20T22:45:22.176 INFO:tasks.rados.rados.0.smithi084.stdout:790: writing smithi08477203-12 from 2818209 to 3243784 tid 3 2024-09-20T22:45:22.176 INFO:tasks.rados.rados.0.smithi084.stdout:791: read oid 29 snap -1 2024-09-20T22:45:22.176 INFO:tasks.rados.rados.0.smithi084.stdout:791: expect deleted 2024-09-20T22:45:22.176 INFO:tasks.rados.rados.0.smithi084.stdout:792: read oid 39 snap 60 2024-09-20T22:45:22.176 INFO:tasks.rados.rados.0.smithi084.stdout:792: expect deleted 2024-09-20T22:45:22.176 INFO:tasks.rados.rados.0.smithi084.stdout:793: snap_remove snap 76 2024-09-20T22:45:22.192 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:22.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:22.192 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:45:22.293 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:45:22.315 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:22.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:22.315 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:45:22.416 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:45:22.437 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:22.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:22.438 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:45:22.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:22 smithi154 ceph-mon[33264]: 7.8 scrub starts 2024-09-20T22:45:22.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:22 smithi154 ceph-mon[33264]: 7.8 scrub ok 2024-09-20T22:45:22.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:22 smithi154 ceph-mon[33264]: osdmap e243: 8 total, 8 up, 7 in 2024-09-20T22:45:22.539 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:45:22.560 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:22.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:22.561 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:45:22.584 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:22 smithi084 ceph-mon[40024]: 7.8 scrub starts 2024-09-20T22:45:22.584 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:22 smithi084 ceph-mon[40024]: 7.8 scrub ok 2024-09-20T22:45:22.584 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:22 smithi084 ceph-mon[40024]: osdmap e243: 8 total, 8 up, 7 in 2024-09-20T22:45:22.584 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:22 smithi084 ceph-mon[29774]: 7.8 scrub starts 2024-09-20T22:45:22.584 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:22 smithi084 ceph-mon[29774]: 7.8 scrub ok 2024-09-20T22:45:22.584 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:22 smithi084 ceph-mon[29774]: osdmap e243: 8 total, 8 up, 7 in 2024-09-20T22:45:22.662 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:45:22.682 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:22.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:22.683 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:45:22.784 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:45:22.810 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:22.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:22.811 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:45:22.912 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:45:22.938 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:22.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:22.939 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:45:23.040 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:45:23.061 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:23.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:23.062 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:45:23.153 INFO:tasks.rados.rados.0.smithi084.stdout:793: done (4 left) 2024-09-20T22:45:23.154 INFO:tasks.rados.rados.0.smithi084.stdout:794: rollback oid 4 current snap is 77 2024-09-20T22:45:23.154 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 4 to 72 2024-09-20T22:45:23.154 INFO:tasks.rados.rados.0.smithi084.stdout:795: read oid 15 snap -1 2024-09-20T22:45:23.154 INFO:tasks.rados.rados.0.smithi084.stdout:795: expect deleted 2024-09-20T22:45:23.154 INFO:tasks.rados.rados.0.smithi084.stdout:796: write oid 36 current snap is 77 2024-09-20T22:45:23.154 INFO:tasks.rados.rados.0.smithi084.stdout:796: seq_num 313 ranges {460591=723410,1716158=403433,2729279=440140} 2024-09-20T22:45:23.160 INFO:tasks.rados.rados.0.smithi084.stdout:796: writing smithi08477203-36 from 460591 to 1184001 tid 1 2024-09-20T22:45:23.163 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:45:23.166 INFO:tasks.rados.rados.0.smithi084.stdout:796: writing smithi08477203-36 from 1716158 to 2119591 tid 2 2024-09-20T22:45:23.173 INFO:tasks.rados.rados.0.smithi084.stdout:796: writing smithi08477203-36 from 2729279 to 3169419 tid 3 2024-09-20T22:45:23.173 INFO:tasks.rados.rados.0.smithi084.stdout:797: read oid 6 snap -1 2024-09-20T22:45:23.173 INFO:tasks.rados.rados.0.smithi084.stdout:797: expect (ObjNum 309 snap 75 seq_num 309) 2024-09-20T22:45:23.173 INFO:tasks.rados.rados.0.smithi084.stdout:798: delete oid 38 current snap is 77 2024-09-20T22:45:23.175 INFO:tasks.rados.rados.0.smithi084.stdout:790: finishing write tid 1 to smithi08477203-12 2024-09-20T22:45:23.175 INFO:tasks.rados.rados.0.smithi084.stdout:790: finishing write tid 2 to smithi08477203-12 2024-09-20T22:45:23.175 INFO:tasks.rados.rados.0.smithi084.stdout:789: expect (ObjNum 208 snap 42 seq_num 208) 2024-09-20T22:45:23.191 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:23.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:23.192 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:45:23.202 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.3 dump_blocked_ops 2024-09-20T22:45:23.293 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:45:23.317 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:23.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:23.318 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:45:23.409 INFO:tasks.rados.rados.0.smithi084.stdout:790: finishing write tid 3 to smithi08477203-12 2024-09-20T22:45:23.409 INFO:tasks.rados.rados.0.smithi084.stdout:790: finishing write tid 4 to smithi08477203-12 2024-09-20T22:45:23.410 INFO:tasks.rados.rados.0.smithi084.stdout:790: finishing write tid 5 to smithi08477203-12 2024-09-20T22:45:23.410 INFO:tasks.rados.rados.0.smithi084.stdout:790: oid 12 updating version 0 to 195 2024-09-20T22:45:23.410 INFO:tasks.rados.rados.0.smithi084.stdout:790: oid 12 updating version 195 to 198 2024-09-20T22:45:23.410 INFO:tasks.rados.rados.0.smithi084.stdout:790: oid 12 version 198 is already newer than 197 2024-09-20T22:45:23.410 INFO:tasks.rados.rados.0.smithi084.stdout:790: oid 12 version 198 is already newer than 196 2024-09-20T22:45:23.410 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 12 v 198 (ObjNum 312 snap 77 seq_num 312) dirty exists 2024-09-20T22:45:23.410 INFO:tasks.rados.rados.0.smithi084.stdout:790: left oid 12 (ObjNum 312 snap 77 seq_num 312) 2024-09-20T22:45:23.410 INFO:tasks.rados.rados.0.smithi084.stdout:794: finishing rollback tid 0 to smithi08477203-4 2024-09-20T22:45:23.410 INFO:tasks.rados.rados.0.smithi084.stdout:794: finishing rollback tid 1 to smithi08477203-4 2024-09-20T22:45:23.410 INFO:tasks.rados.rados.0.smithi084.stdout:789: done (8 left) 2024-09-20T22:45:23.410 INFO:tasks.rados.rados.0.smithi084.stdout:790: done (7 left) 2024-09-20T22:45:23.410 INFO:tasks.rados.rados.0.smithi084.stdout:791: done (6 left) 2024-09-20T22:45:23.410 INFO:tasks.rados.rados.0.smithi084.stdout:792: done (5 left) 2024-09-20T22:45:23.410 INFO:tasks.rados.rados.0.smithi084.stdout:795: done (4 left) 2024-09-20T22:45:23.410 INFO:tasks.rados.rados.0.smithi084.stdout:798: done (3 left) 2024-09-20T22:45:23.411 INFO:tasks.rados.rados.0.smithi084.stdout:799: snap_create 2024-09-20T22:45:23.411 INFO:tasks.rados.rados.0.smithi084.stdout:794: finishing rollback tid 2 to smithi08477203-4 2024-09-20T22:45:23.411 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 4 v 362 (ObjNum 241 snap 54 seq_num 241) dirty exists 2024-09-20T22:45:23.411 INFO:tasks.rados.rados.0.smithi084.stdout:796: finishing write tid 1 to smithi08477203-36 2024-09-20T22:45:23.411 INFO:tasks.rados.rados.0.smithi084.stdout:796: finishing write tid 2 to smithi08477203-36 2024-09-20T22:45:23.411 INFO:tasks.rados.rados.0.smithi084.stdout:796: finishing write tid 3 to smithi08477203-36 2024-09-20T22:45:23.411 INFO:tasks.rados.rados.0.smithi084.stdout:796: finishing write tid 4 to smithi08477203-36 2024-09-20T22:45:23.411 INFO:tasks.rados.rados.0.smithi084.stdout:796: finishing write tid 5 to smithi08477203-36 2024-09-20T22:45:23.411 INFO:tasks.rados.rados.0.smithi084.stdout:796: oid 36 updating version 0 to 77 2024-09-20T22:45:23.411 INFO:tasks.rados.rados.0.smithi084.stdout:796: oid 36 version 77 is already newer than 74 2024-09-20T22:45:23.411 INFO:tasks.rados.rados.0.smithi084.stdout:796: oid 36 version 77 is already newer than 76 2024-09-20T22:45:23.411 INFO:tasks.rados.rados.0.smithi084.stdout:796: oid 36 version 77 is already newer than 75 2024-09-20T22:45:23.411 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 36 v 77 (ObjNum 313 snap 77 seq_num 313) dirty exists 2024-09-20T22:45:23.411 INFO:tasks.rados.rados.0.smithi084.stdout:796: left oid 36 (ObjNum 313 snap 77 seq_num 313) 2024-09-20T22:45:23.411 INFO:tasks.rados.rados.0.smithi084.stdout:797: expect (ObjNum 309 snap 75 seq_num 309) 2024-09-20T22:45:23.419 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:45:23.441 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:23.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:23.442 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:45:23.488 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 7] out_osds: [4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-20T22:45:23.488 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-20T22:45:23.488 INFO:tasks.thrashosds.thrasher:Shrinking pool .mgr 2024-09-20T22:45:23.488 INFO:tasks.cephadm.ceph_manager.ceph:contract_pool .mgr by 10 min 8 2024-09-20T22:45:23.488 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph pg dump --format=json 2024-09-20T22:45:23.496 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:45:23.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:23 smithi154 ceph-mon[33264]: pgmap v265: 178 pgs: 3 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 173 active+clean; 141 MiB data, 600 MiB used, 625 GiB / 626 GiB avail; 18 MiB/s rd, 2.2 MiB/s wr, 27 op/s 2024-09-20T22:45:23.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:23 smithi154 ceph-mon[33264]: 3.2 scrub starts 2024-09-20T22:45:23.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:23 smithi154 ceph-mon[33264]: 3.2 scrub ok 2024-09-20T22:45:23.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:23 smithi154 ceph-mon[33264]: 6.17 scrub starts 2024-09-20T22:45:23.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:23 smithi154 ceph-mon[33264]: 6.17 scrub ok 2024-09-20T22:45:23.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:23 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:45:23.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:23 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:45:23.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:23 smithi154 ceph-mon[33264]: osdmap e244: 8 total, 8 up, 7 in 2024-09-20T22:45:23.510 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:23 smithi084 ceph-mon[40024]: pgmap v265: 178 pgs: 3 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 173 active+clean; 141 MiB data, 600 MiB used, 625 GiB / 626 GiB avail; 18 MiB/s rd, 2.2 MiB/s wr, 27 op/s 2024-09-20T22:45:23.510 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:23 smithi084 ceph-mon[40024]: 3.2 scrub starts 2024-09-20T22:45:23.510 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:23 smithi084 ceph-mon[40024]: 3.2 scrub ok 2024-09-20T22:45:23.511 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:23 smithi084 ceph-mon[40024]: 6.17 scrub starts 2024-09-20T22:45:23.511 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:23 smithi084 ceph-mon[40024]: 6.17 scrub ok 2024-09-20T22:45:23.511 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:23 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:45:23.511 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:23 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:45:23.511 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:23 smithi084 ceph-mon[40024]: osdmap e244: 8 total, 8 up, 7 in 2024-09-20T22:45:23.511 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:23 smithi084 ceph-mon[29774]: pgmap v265: 178 pgs: 3 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 173 active+clean; 141 MiB data, 600 MiB used, 625 GiB / 626 GiB avail; 18 MiB/s rd, 2.2 MiB/s wr, 27 op/s 2024-09-20T22:45:23.511 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:23 smithi084 ceph-mon[29774]: 3.2 scrub starts 2024-09-20T22:45:23.511 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:23 smithi084 ceph-mon[29774]: 3.2 scrub ok 2024-09-20T22:45:23.511 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:23 smithi084 ceph-mon[29774]: 6.17 scrub starts 2024-09-20T22:45:23.511 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:23 smithi084 ceph-mon[29774]: 6.17 scrub ok 2024-09-20T22:45:23.511 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:23 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:45:23.511 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:23 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:45:23.512 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:23 smithi084 ceph-mon[29774]: osdmap e244: 8 total, 8 up, 7 in 2024-09-20T22:45:23.543 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:45:23.564 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:23.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:23.565 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:45:23.666 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:45:23.689 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:23.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:23.690 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:45:23.790 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:45:23.811 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:23.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:23.812 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:45:23.913 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:45:23.934 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:23.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:23.935 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:45:23.985 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-20T22:45:24.035 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:45:24.056 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:24.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:24.057 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:45:24.077 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-20T22:45:24.077 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:45:24.158 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:45:24.180 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:24.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:24.181 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:45:24.252 INFO:tasks.rados.rados.0.smithi084.stdout:794: done (3 left) 2024-09-20T22:45:24.252 INFO:tasks.rados.rados.0.smithi084.stdout:796: done (2 left) 2024-09-20T22:45:24.252 INFO:tasks.rados.rados.0.smithi084.stdout:797: done (1 left) 2024-09-20T22:45:24.252 INFO:tasks.rados.rados.0.smithi084.stdout:799: done (0 left) 2024-09-20T22:45:24.252 INFO:tasks.rados.rados.0.smithi084.stdout:800: read oid 19 snap -1 2024-09-20T22:45:24.252 INFO:tasks.rados.rados.0.smithi084.stdout:800: expect (ObjNum 245 snap 54 seq_num 245) 2024-09-20T22:45:24.253 INFO:tasks.rados.rados.0.smithi084.stdout:801: read oid 42 snap -1 2024-09-20T22:45:24.253 INFO:tasks.rados.rados.0.smithi084.stdout:801: expect deleted 2024-09-20T22:45:24.253 INFO:tasks.rados.rados.0.smithi084.stdout:802: read oid 26 snap 75 2024-09-20T22:45:24.253 INFO:tasks.rados.rados.0.smithi084.stdout:802: expect (ObjNum 295 snap 75 seq_num 295) 2024-09-20T22:45:24.253 INFO:tasks.rados.rados.0.smithi084.stdout:803: rollback oid 3 current snap is 78 2024-09-20T22:45:24.253 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 3 to 62 2024-09-20T22:45:24.253 INFO:tasks.rados.rados.0.smithi084.stdout:804: copy_from oid 33 from oid 43 current snap is 78 2024-09-20T22:45:24.253 INFO:tasks.rados.rados.0.smithi084.stdout:805: copy_from oid 36 from oid 38 current snap is 78 2024-09-20T22:45:24.253 INFO:tasks.rados.rados.0.smithi084.stdout:806: write oid 17 current snap is 78 2024-09-20T22:45:24.253 INFO:tasks.rados.rados.0.smithi084.stdout:806: seq_num 315 ranges {494205=473908,1433122=407102,2384125=462684} 2024-09-20T22:45:24.257 INFO:tasks.rados.rados.0.smithi084.stdout:806: writing smithi08477203-17 from 494205 to 968113 tid 1 2024-09-20T22:45:24.263 INFO:tasks.rados.rados.0.smithi084.stdout:806: writing smithi08477203-17 from 1433122 to 1840224 tid 2 2024-09-20T22:45:24.269 INFO:tasks.rados.rados.0.smithi084.stdout:806: writing smithi08477203-17 from 2384125 to 2846809 tid 3 2024-09-20T22:45:24.269 INFO:tasks.rados.rados.0.smithi084.stdout:807: snap_remove snap 71 2024-09-20T22:45:24.283 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:45:24.306 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:24.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:24.307 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:45:24.364 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:24 smithi084 ceph-mon[29774]: 1.8 scrub starts 2024-09-20T22:45:24.364 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:24 smithi084 ceph-mon[29774]: 1.8 scrub ok 2024-09-20T22:45:24.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:24 smithi084 ceph-mon[29774]: 5.0 scrub starts 2024-09-20T22:45:24.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:24 smithi084 ceph-mon[29774]: 5.0 scrub ok 2024-09-20T22:45:24.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:24 smithi084 ceph-mon[29774]: 5.b scrub starts 2024-09-20T22:45:24.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:24 smithi084 ceph-mon[29774]: 5.b scrub ok 2024-09-20T22:45:24.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:24 smithi084 ceph-mon[29774]: 7.19 deep-scrub starts 2024-09-20T22:45:24.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:24 smithi084 ceph-mon[29774]: 7.19 deep-scrub ok 2024-09-20T22:45:24.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:24 smithi084 ceph-mon[40024]: 1.8 scrub starts 2024-09-20T22:45:24.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:24 smithi084 ceph-mon[40024]: 1.8 scrub ok 2024-09-20T22:45:24.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:24 smithi084 ceph-mon[40024]: 5.0 scrub starts 2024-09-20T22:45:24.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:24 smithi084 ceph-mon[40024]: 5.0 scrub ok 2024-09-20T22:45:24.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:24 smithi084 ceph-mon[40024]: 5.b scrub starts 2024-09-20T22:45:24.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:24 smithi084 ceph-mon[40024]: 5.b scrub ok 2024-09-20T22:45:24.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:24 smithi084 ceph-mon[40024]: 7.19 deep-scrub starts 2024-09-20T22:45:24.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:24 smithi084 ceph-mon[40024]: 7.19 deep-scrub ok 2024-09-20T22:45:24.408 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:45:24.429 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:24.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:24.430 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:45:24.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:24 smithi154 ceph-mon[33264]: 1.8 scrub starts 2024-09-20T22:45:24.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:24 smithi154 ceph-mon[33264]: 1.8 scrub ok 2024-09-20T22:45:24.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:24 smithi154 ceph-mon[33264]: 5.0 scrub starts 2024-09-20T22:45:24.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:24 smithi154 ceph-mon[33264]: 5.0 scrub ok 2024-09-20T22:45:24.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:24 smithi154 ceph-mon[33264]: 5.b scrub starts 2024-09-20T22:45:24.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:24 smithi154 ceph-mon[33264]: 5.b scrub ok 2024-09-20T22:45:24.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:24 smithi154 ceph-mon[33264]: 7.19 deep-scrub starts 2024-09-20T22:45:24.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:24 smithi154 ceph-mon[33264]: 7.19 deep-scrub ok 2024-09-20T22:45:24.531 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:45:24.556 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:24.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:24.557 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:45:24.610 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd set noscrub 2024-09-20T22:45:24.658 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:45:24.683 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:24.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:24.684 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:45:24.785 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:45:24.809 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:24.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:24.810 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:45:24.847 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.3 dump_historic_ops 2024-09-20T22:45:24.850 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:45:24.885 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:45:24.912 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:45:24.932 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:24.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:24.933 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:45:25.034 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:45:25.060 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:25.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:25.061 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:45:25.161 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:45:25.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:25.184 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:25.184 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:45:25.253 INFO:tasks.rados.rados.0.smithi084.stdout:807: done (7 left) 2024-09-20T22:45:25.254 INFO:tasks.rados.rados.0.smithi084.stdout:808: copy_from oid 11 from oid 21 current snap is 78 2024-09-20T22:45:25.254 INFO:tasks.rados.rados.0.smithi084.stdout:809: read oid 4 snap 69 2024-09-20T22:45:25.254 INFO:tasks.rados.rados.0.smithi084.stdout:809: expect (ObjNum 241 snap 54 seq_num 241) 2024-09-20T22:45:25.254 INFO:tasks.rados.rados.0.smithi084.stdout:810: snap_create 2024-09-20T22:45:25.254 INFO:tasks.rados.rados.0.smithi084.stdout:804: finishing copy_from to smithi08477203-33 2024-09-20T22:45:25.254 INFO:tasks.rados.rados.0.smithi084.stdout:804: got expected ENOENT (src dne) 2024-09-20T22:45:25.254 INFO:tasks.rados.rados.0.smithi084.stdout:804: finishing copy_from racing read to smithi08477203-33 2024-09-20T22:45:25.254 INFO:tasks.rados.rados.0.smithi084.stdout:803: finishing rollback tid 1 to smithi08477203-3 2024-09-20T22:45:25.254 INFO:tasks.rados.rados.0.smithi084.stdout:803: finishing rollback tid 2 to smithi08477203-3 2024-09-20T22:45:25.254 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 3 v 121 (ObjNum 245 snap 54 seq_num 245) dirty exists 2024-09-20T22:45:25.254 INFO:tasks.rados.rados.0.smithi084.stdout:805: finishing copy_from to smithi08477203-36 2024-09-20T22:45:25.254 INFO:tasks.rados.rados.0.smithi084.stdout:805: got expected ENOENT (src dne) 2024-09-20T22:45:25.255 INFO:tasks.rados.rados.0.smithi084.stdout:805: finishing copy_from racing read to smithi08477203-36 2024-09-20T22:45:25.255 INFO:tasks.rados.rados.0.smithi084.stdout:802: expect (ObjNum 295 snap 75 seq_num 295) 2024-09-20T22:45:25.285 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:45:25.306 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:25.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:25.307 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:45:25.353 INFO:tasks.rados.rados.0.smithi084.stdout:806: finishing write tid 1 to smithi08477203-17 2024-09-20T22:45:25.354 INFO:tasks.rados.rados.0.smithi084.stdout:806: finishing write tid 2 to smithi08477203-17 2024-09-20T22:45:25.354 INFO:tasks.rados.rados.0.smithi084.stdout:800: expect (ObjNum 245 snap 54 seq_num 245) 2024-09-20T22:45:25.408 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:45:25.430 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:25.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:25.431 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:45:25.485 INFO:tasks.rados.rados.0.smithi084.stdout:806: finishing write tid 3 to smithi08477203-17 2024-09-20T22:45:25.485 INFO:tasks.rados.rados.0.smithi084.stdout:806: finishing write tid 4 to smithi08477203-17 2024-09-20T22:45:25.485 INFO:tasks.rados.rados.0.smithi084.stdout:806: finishing write tid 5 to smithi08477203-17 2024-09-20T22:45:25.485 INFO:tasks.rados.rados.0.smithi084.stdout:806: oid 17 updating version 0 to 100 2024-09-20T22:45:25.485 INFO:tasks.rados.rados.0.smithi084.stdout:806: oid 17 updating version 100 to 102 2024-09-20T22:45:25.485 INFO:tasks.rados.rados.0.smithi084.stdout:806: oid 17 version 102 is already newer than 101 2024-09-20T22:45:25.485 INFO:tasks.rados.rados.0.smithi084.stdout:806: oid 17 version 102 is already newer than 99 2024-09-20T22:45:25.485 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 17 v 102 (ObjNum 315 snap 78 seq_num 315) dirty exists 2024-09-20T22:45:25.485 INFO:tasks.rados.rados.0.smithi084.stdout:806: left oid 17 (ObjNum 315 snap 78 seq_num 315) 2024-09-20T22:45:25.486 INFO:tasks.rados.rados.0.smithi084.stdout:809: expect (ObjNum 241 snap 54 seq_num 241) 2024-09-20T22:45:25.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:25 smithi154 ceph-mon[33264]: pgmap v268: 178 pgs: 2 active+clean+snaptrim, 176 active+clean; 167 MiB data, 648 MiB used, 625 GiB / 626 GiB avail; 24 MiB/s rd, 4.2 MiB/s wr, 39 op/s 2024-09-20T22:45:25.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:25 smithi154 ceph-mon[33264]: 3.12 scrub starts 2024-09-20T22:45:25.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:25 smithi154 ceph-mon[33264]: 3.12 scrub ok 2024-09-20T22:45:25.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:25 smithi154 ceph-mon[33264]: osdmap e245: 8 total, 8 up, 7 in 2024-09-20T22:45:25.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:25 smithi154 ceph-mon[33264]: 1.9 scrub starts 2024-09-20T22:45:25.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:25 smithi154 ceph-mon[33264]: 1.9 scrub ok 2024-09-20T22:45:25.532 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:45:25.556 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:25.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:25.557 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:45:25.578 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:25 smithi084 ceph-mon[40024]: pgmap v268: 178 pgs: 2 active+clean+snaptrim, 176 active+clean; 167 MiB data, 648 MiB used, 625 GiB / 626 GiB avail; 24 MiB/s rd, 4.2 MiB/s wr, 39 op/s 2024-09-20T22:45:25.578 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:25 smithi084 ceph-mon[40024]: 3.12 scrub starts 2024-09-20T22:45:25.579 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:25 smithi084 ceph-mon[40024]: 3.12 scrub ok 2024-09-20T22:45:25.579 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:25 smithi084 ceph-mon[40024]: osdmap e245: 8 total, 8 up, 7 in 2024-09-20T22:45:25.579 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:25 smithi084 ceph-mon[40024]: 1.9 scrub starts 2024-09-20T22:45:25.579 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:25 smithi084 ceph-mon[40024]: 1.9 scrub ok 2024-09-20T22:45:25.579 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:25 smithi084 ceph-mon[29774]: pgmap v268: 178 pgs: 2 active+clean+snaptrim, 176 active+clean; 167 MiB data, 648 MiB used, 625 GiB / 626 GiB avail; 24 MiB/s rd, 4.2 MiB/s wr, 39 op/s 2024-09-20T22:45:25.579 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:25 smithi084 ceph-mon[29774]: 3.12 scrub starts 2024-09-20T22:45:25.579 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:25 smithi084 ceph-mon[29774]: 3.12 scrub ok 2024-09-20T22:45:25.579 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:25 smithi084 ceph-mon[29774]: osdmap e245: 8 total, 8 up, 7 in 2024-09-20T22:45:25.579 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:25 smithi084 ceph-mon[29774]: 1.9 scrub starts 2024-09-20T22:45:25.579 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:25 smithi084 ceph-mon[29774]: 1.9 scrub ok 2024-09-20T22:45:25.589 INFO:tasks.rados.rados.0.smithi084.stdout:808: finishing copy_from racing read to smithi08477203-11 2024-09-20T22:45:25.589 INFO:tasks.rados.rados.0.smithi084.stdout:808: finishing copy_from to smithi08477203-11 2024-09-20T22:45:25.589 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 11 v 212 (ObjNum 267 snap 65 seq_num 267) dirty exists 2024-09-20T22:45:25.658 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:45:25.680 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:25.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:25.681 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:45:25.745 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-20T22:45:25.748 INFO:teuthology.orchestra.run.smithi084.stderr:dumped all 2024-09-20T22:45:25.782 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:45:25.804 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:25.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:25.805 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:45:25.906 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:45:25.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:25.927 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:25.927 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:45:26.028 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:45:26.054 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:26.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:26.055 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:45:26.156 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:45:26.177 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:26.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:26.178 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:45:26.251 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is set 2024-09-20T22:45:26.260 INFO:tasks.rados.rados.0.smithi084.stdout:800: done (9 left) 2024-09-20T22:45:26.260 INFO:tasks.rados.rados.0.smithi084.stdout:801: done (8 left) 2024-09-20T22:45:26.260 INFO:tasks.rados.rados.0.smithi084.stdout:802: done (7 left) 2024-09-20T22:45:26.260 INFO:tasks.rados.rados.0.smithi084.stdout:803: done (6 left) 2024-09-20T22:45:26.260 INFO:tasks.rados.rados.0.smithi084.stdout:804: done (5 left) 2024-09-20T22:45:26.260 INFO:tasks.rados.rados.0.smithi084.stdout:805: done (4 left) 2024-09-20T22:45:26.260 INFO:tasks.rados.rados.0.smithi084.stdout:806: done (3 left) 2024-09-20T22:45:26.260 INFO:tasks.rados.rados.0.smithi084.stdout:808: done (2 left) 2024-09-20T22:45:26.260 INFO:tasks.rados.rados.0.smithi084.stdout:809: done (1 left) 2024-09-20T22:45:26.260 INFO:tasks.rados.rados.0.smithi084.stdout:810: done (0 left) 2024-09-20T22:45:26.260 INFO:tasks.rados.rados.0.smithi084.stdout:811: read oid 38 snap -1 2024-09-20T22:45:26.260 INFO:tasks.rados.rados.0.smithi084.stdout:811: expect deleted 2024-09-20T22:45:26.261 INFO:tasks.rados.rados.0.smithi084.stdout:812: snap_remove snap 77 2024-09-20T22:45:26.279 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:45:26.303 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:26.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:26.304 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:45:26.405 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:45:26.409 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:26 smithi084 ceph-mon[40024]: 3.16 deep-scrub starts 2024-09-20T22:45:26.409 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:26 smithi084 ceph-mon[40024]: 3.16 deep-scrub ok 2024-09-20T22:45:26.409 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:26 smithi084 ceph-mon[40024]: 6.1f deep-scrub starts 2024-09-20T22:45:26.409 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:26 smithi084 ceph-mon[40024]: 6.1f deep-scrub ok 2024-09-20T22:45:26.409 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:26 smithi084 ceph-mon[40024]: osdmap e246: 8 total, 8 up, 7 in 2024-09-20T22:45:26.409 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:26 smithi084 ceph-mon[40024]: 5.a scrub starts 2024-09-20T22:45:26.409 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:26 smithi084 ceph-mon[40024]: 5.a scrub ok 2024-09-20T22:45:26.410 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:26 smithi084 ceph-mon[40024]: 1.7 scrub starts 2024-09-20T22:45:26.410 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:26 smithi084 ceph-mon[40024]: 1.7 scrub ok 2024-09-20T22:45:26.410 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:26 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/2338281916' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-20T22:45:26.410 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:26 smithi084 ceph-mon[29774]: 3.16 deep-scrub starts 2024-09-20T22:45:26.410 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:26 smithi084 ceph-mon[29774]: 3.16 deep-scrub ok 2024-09-20T22:45:26.410 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:26 smithi084 ceph-mon[29774]: 6.1f deep-scrub starts 2024-09-20T22:45:26.410 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:26 smithi084 ceph-mon[29774]: 6.1f deep-scrub ok 2024-09-20T22:45:26.410 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:26 smithi084 ceph-mon[29774]: osdmap e246: 8 total, 8 up, 7 in 2024-09-20T22:45:26.410 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:26 smithi084 ceph-mon[29774]: 5.a scrub starts 2024-09-20T22:45:26.410 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:26 smithi084 ceph-mon[29774]: 5.a scrub ok 2024-09-20T22:45:26.411 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:26 smithi084 ceph-mon[29774]: 1.7 scrub starts 2024-09-20T22:45:26.411 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:26 smithi084 ceph-mon[29774]: 1.7 scrub ok 2024-09-20T22:45:26.411 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:26 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/2338281916' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-20T22:45:26.428 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:26.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:26.429 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:45:26.436 INFO:teuthology.orchestra.run.smithi084.stdout:{"pg_ready":true,"pg_map":{"version":271,"stamp":"2024-09-20T22:45:25.670357+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":174888027,"num_objects":289,"num_object_clones":48,"num_object_copies":867,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":289,"num_whiteouts":11,"num_read":2199,"num_read_kb":1318878,"num_write":1319,"num_write_kb":327732,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":119,"num_bytes_recovered":5838014,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5953,"ondisk_log_size":5953,"up":534,"acting":534,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":534,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":663096,"kb_used_data":407308,"kb_used_omap":10,"kb_used_meta":255605,"kb_avail":655552968,"statfs":{"total":671965249536,"available":671286239232,"internally_reserved":0,"allocated":417083392,"data_stored":519779595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11157,"internal_metadata":261739627},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":27049633,"num_objects":18,"num_object_clones":15,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":5,"num_read":87,"num_read_kb":96301,"num_write":71,"num_write_kb":17333,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000992"},"pg_stats":[{"pgid":"3.1f","version":"86'2","reported_seq":389,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.150923+0000","last_change":"2024-09-20T22:44:23.787416+0000","last_active":"2024-09-20T22:45:23.150923+0000","last_peered":"2024-09-20T22:45:23.150923+0000","last_clean":"2024-09-20T22:45:23.150923+0000","last_became_active":"2024-09-20T22:41:38.159085+0000","last_became_peered":"2024-09-20T22:41:38.159085+0000","last_unstale":"2024-09-20T22:45:23.150923+0000","last_undegraded":"2024-09-20T22:45:23.150923+0000","last_fullsized":"2024-09-20T22:45:23.150923+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'2","last_scrub_stamp":"2024-09-20T22:44:23.787342+0000","last_deep_scrub":"86'2","last_deep_scrub_stamp":"2024-09-20T22:42:39.664421+0000","last_clean_scrub_stamp":"2024-09-20T22:44:23.787342+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:45.057201+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"74'9","reported_seq":237,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.155873+0000","last_change":"2024-09-20T22:44:14.456654+0000","last_active":"2024-09-20T22:45:23.155873+0000","last_peered":"2024-09-20T22:45:23.155873+0000","last_clean":"2024-09-20T22:45:23.155873+0000","last_became_active":"2024-09-20T22:43:57.428786+0000","last_became_peered":"2024-09-20T22:43:57.428786+0000","last_unstale":"2024-09-20T22:45:23.155873+0000","last_undegraded":"2024-09-20T22:45:23.155873+0000","last_fullsized":"2024-09-20T22:45:23.155873+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-09-20T22:44:14.456573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:44:14.456573+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:30.414294+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":366,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.149124+0000","last_change":"2024-09-20T22:44:18.040704+0000","last_active":"2024-09-20T22:45:23.149124+0000","last_peered":"2024-09-20T22:45:23.149124+0000","last_clean":"2024-09-20T22:45:23.149124+0000","last_became_active":"2024-09-20T22:41:42.182024+0000","last_became_peered":"2024-09-20T22:41:42.182024+0000","last_unstale":"2024-09-20T22:45:23.149124+0000","last_undegraded":"2024-09-20T22:45:23.149124+0000","last_fullsized":"2024-09-20T22:45:23.149124+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:44:18.040631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:44:18.040631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:23.293547+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":193,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.473916+0000","last_change":"2024-09-20T22:44:40.746186+0000","last_active":"2024-09-20T22:45:23.473916+0000","last_peered":"2024-09-20T22:45:23.473916+0000","last_clean":"2024-09-20T22:45:23.473916+0000","last_became_active":"2024-09-20T22:43:57.429983+0000","last_became_peered":"2024-09-20T22:43:57.429983+0000","last_unstale":"2024-09-20T22:45:23.473916+0000","last_undegraded":"2024-09-20T22:45:23.473916+0000","last_fullsized":"2024-09-20T22:45:23.473916+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:44:40.746114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:44:40.746114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:40.841196+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":362,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.160240+0000","last_change":"2024-09-20T22:43:07.476040+0000","last_active":"2024-09-20T22:45:23.160240+0000","last_peered":"2024-09-20T22:45:23.160240+0000","last_clean":"2024-09-20T22:45:23.160240+0000","last_became_active":"2024-09-20T22:41:44.832099+0000","last_became_peered":"2024-09-20T22:41:44.832099+0000","last_unstale":"2024-09-20T22:45:23.160240+0000","last_undegraded":"2024-09-20T22:45:23.160240+0000","last_fullsized":"2024-09-20T22:45:23.160240+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:07.475964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:43:07.475964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":374,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.165752+0000","last_change":"2024-09-20T22:44:58.994410+0000","last_active":"2024-09-20T22:45:23.165752+0000","last_peered":"2024-09-20T22:45:23.165752+0000","last_clean":"2024-09-20T22:45:23.165752+0000","last_became_active":"2024-09-20T22:41:38.157298+0000","last_became_peered":"2024-09-20T22:41:38.157298+0000","last_unstale":"2024-09-20T22:45:23.165752+0000","last_undegraded":"2024-09-20T22:45:23.165752+0000","last_fullsized":"2024-09-20T22:45:23.165752+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:44:58.994340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:44:58.994340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:46:26.252244+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"74'15","reported_seq":395,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.161559+0000","last_change":"2024-09-20T22:44:41.108291+0000","last_active":"2024-09-20T22:45:23.161559+0000","last_peered":"2024-09-20T22:45:23.161559+0000","last_clean":"2024-09-20T22:45:23.161559+0000","last_became_active":"2024-09-20T22:41:40.163242+0000","last_became_peered":"2024-09-20T22:41:40.163242+0000","last_unstale":"2024-09-20T22:45:23.161559+0000","last_undegraded":"2024-09-20T22:45:23.161559+0000","last_fullsized":"2024-09-20T22:45:23.161559+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-09-20T22:44:41.108205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:44:41.108205+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:50.242514+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":192,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.156091+0000","last_change":"2024-09-20T22:44:57.608536+0000","last_active":"2024-09-20T22:45:23.156091+0000","last_peered":"2024-09-20T22:45:23.156091+0000","last_clean":"2024-09-20T22:45:23.156091+0000","last_became_active":"2024-09-20T22:43:57.423685+0000","last_became_peered":"2024-09-20T22:43:57.423685+0000","last_unstale":"2024-09-20T22:45:23.156091+0000","last_undegraded":"2024-09-20T22:45:23.156091+0000","last_fullsized":"2024-09-20T22:45:23.156091+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:44:57.608471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:44:57.608471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:46:05.218588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.19","version":"241'147","reported_seq":670,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.156442+0000","last_change":"2024-09-20T22:45:23.157197+0000","last_active":"2024-09-20T22:45:23.156442+0000","last_peered":"2024-09-20T22:45:23.156442+0000","last_clean":"2024-09-20T22:45:23.156442+0000","last_became_active":"2024-09-20T22:43:39.130950+0000","last_became_peered":"2024-09-20T22:43:39.130950+0000","last_unstale":"2024-09-20T22:45:23.156442+0000","last_undegraded":"2024-09-20T22:45:23.156442+0000","last_fullsized":"2024-09-20T22:45:23.156442+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":5,"last_scrub":"143'50","last_scrub_stamp":"2024-09-20T22:43:45.392735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:43:45.392735+0000","objects_scrubbed":3,"log_size":147,"log_dups_size":0,"ondisk_log_size":147,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001412129,"stat_sum":{"num_bytes":2840416,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":37,"num_read_kb":29154,"num_write":40,"num_write_kb":15907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"42","length":"1"},{"start":"44","length":"1"},{"start":"4c","length":"1"},{"start":"4e","length":"1"}]},{"pgid":"3.1d","version":"0'0","reported_seq":380,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.474910+0000","last_change":"2024-09-20T22:43:57.430206+0000","last_active":"2024-09-20T22:45:23.474910+0000","last_peered":"2024-09-20T22:45:23.474910+0000","last_clean":"2024-09-20T22:45:23.474910+0000","last_became_active":"2024-09-20T22:43:57.429854+0000","last_became_peered":"2024-09-20T22:43:57.429854+0000","last_unstale":"2024-09-20T22:45:23.474910+0000","last_undegraded":"2024-09-20T22:45:23.474910+0000","last_fullsized":"2024-09-20T22:45:23.474910+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:34.681993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:43:34.681993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"74'9","reported_seq":229,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.160121+0000","last_change":"2024-09-20T22:43:57.488635+0000","last_active":"2024-09-20T22:45:23.160121+0000","last_peered":"2024-09-20T22:45:23.160121+0000","last_clean":"2024-09-20T22:45:23.160121+0000","last_became_active":"2024-09-20T22:43:57.445313+0000","last_became_peered":"2024-09-20T22:43:57.445313+0000","last_unstale":"2024-09-20T22:45:23.160121+0000","last_undegraded":"2024-09-20T22:45:23.160121+0000","last_fullsized":"2024-09-20T22:45:23.160121+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-09-20T22:43:30.938345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:43:30.938345+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":366,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.474398+0000","last_change":"2024-09-20T22:44:15.667340+0000","last_active":"2024-09-20T22:45:23.474398+0000","last_peered":"2024-09-20T22:45:23.474398+0000","last_clean":"2024-09-20T22:45:23.474398+0000","last_became_active":"2024-09-20T22:41:42.185538+0000","last_became_peered":"2024-09-20T22:41:42.185538+0000","last_unstale":"2024-09-20T22:45:23.474398+0000","last_undegraded":"2024-09-20T22:45:23.474398+0000","last_fullsized":"2024-09-20T22:45:23.474398+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:44:15.667266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:44:15.667266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":362,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.147690+0000","last_change":"2024-09-20T22:44:20.811801+0000","last_active":"2024-09-20T22:45:23.147690+0000","last_peered":"2024-09-20T22:45:23.147690+0000","last_clean":"2024-09-20T22:45:23.147690+0000","last_became_active":"2024-09-20T22:41:44.197113+0000","last_became_peered":"2024-09-20T22:41:44.197113+0000","last_unstale":"2024-09-20T22:45:23.147690+0000","last_undegraded":"2024-09-20T22:45:23.147690+0000","last_fullsized":"2024-09-20T22:45:23.147690+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:44:20.811723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:44:20.811723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:38.989363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.18","version":"242'94","reported_seq":623,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.147894+0000","last_change":"2024-09-20T22:45:23.148139+0000","last_active":"2024-09-20T22:45:23.147894+0000","last_peered":"2024-09-20T22:45:23.147894+0000","last_clean":"2024-09-20T22:45:23.147894+0000","last_became_active":"2024-09-20T22:43:39.132159+0000","last_became_peered":"2024-09-20T22:43:39.132159+0000","last_unstale":"2024-09-20T22:45:23.147894+0000","last_undegraded":"2024-09-20T22:45:23.147894+0000","last_fullsized":"2024-09-20T22:45:23.147894+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":5,"last_scrub":"143'47","last_scrub_stamp":"2024-09-20T22:43:45.760674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:43:45.760674+0000","objects_scrubbed":2,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.000537486,"stat_sum":{"num_bytes":5361400,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":55,"num_read_kb":59296,"num_write":27,"num_write_kb":13129,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"42","length":"1"},{"start":"44","length":"1"},{"start":"4c","length":"1"},{"start":"4e","length":"1"}]},{"pgid":"3.1c","version":"0'0","reported_seq":380,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.474606+0000","last_change":"2024-09-20T22:43:57.430381+0000","last_active":"2024-09-20T22:45:23.474606+0000","last_peered":"2024-09-20T22:45:23.474606+0000","last_clean":"2024-09-20T22:45:23.474606+0000","last_became_active":"2024-09-20T22:43:57.430028+0000","last_became_peered":"2024-09-20T22:43:57.430028+0000","last_unstale":"2024-09-20T22:45:23.474606+0000","last_undegraded":"2024-09-20T22:45:23.474606+0000","last_fullsized":"2024-09-20T22:45:23.474606+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:00.713878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:43:00.713878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"181'7","reported_seq":229,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.160463+0000","last_change":"2024-09-20T22:44:20.259677+0000","last_active":"2024-09-20T22:45:23.160463+0000","last_peered":"2024-09-20T22:45:23.160463+0000","last_clean":"2024-09-20T22:45:23.160463+0000","last_became_active":"2024-09-20T22:43:57.441058+0000","last_became_peered":"2024-09-20T22:43:57.441058+0000","last_unstale":"2024-09-20T22:45:23.160463+0000","last_undegraded":"2024-09-20T22:45:23.160463+0000","last_fullsized":"2024-09-20T22:45:23.160463+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"181'7","last_scrub_stamp":"2024-09-20T22:44:20.259598+0000","last_deep_scrub":"181'7","last_deep_scrub_stamp":"2024-09-20T22:44:20.259598+0000","last_clean_scrub_stamp":"2024-09-20T22:44:20.259598+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:28.101261+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":6,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":383,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.160135+0000","last_change":"2024-09-20T22:44:54.057357+0000","last_active":"2024-09-20T22:45:23.160135+0000","last_peered":"2024-09-20T22:45:23.160135+0000","last_clean":"2024-09-20T22:45:23.160135+0000","last_became_active":"2024-09-20T22:43:57.419419+0000","last_became_peered":"2024-09-20T22:43:57.419419+0000","last_unstale":"2024-09-20T22:45:23.160135+0000","last_undegraded":"2024-09-20T22:45:23.160135+0000","last_fullsized":"2024-09-20T22:45:23.160135+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:44:54.057283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:44:54.057283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:46:09.354039+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":362,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.473824+0000","last_change":"2024-09-20T22:43:32.712751+0000","last_active":"2024-09-20T22:45:23.473824+0000","last_peered":"2024-09-20T22:45:23.473824+0000","last_clean":"2024-09-20T22:45:23.473824+0000","last_became_active":"2024-09-20T22:41:44.197927+0000","last_became_peered":"2024-09-20T22:41:44.197927+0000","last_unstale":"2024-09-20T22:45:23.473824+0000","last_undegraded":"2024-09-20T22:45:23.473824+0000","last_fullsized":"2024-09-20T22:45:23.473824+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:32.712676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:43:32.712676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":193,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.157185+0000","last_change":"2024-09-20T22:44:41.574836+0000","last_active":"2024-09-20T22:45:23.157185+0000","last_peered":"2024-09-20T22:45:23.157185+0000","last_clean":"2024-09-20T22:45:23.157185+0000","last_became_active":"2024-09-20T22:43:57.427971+0000","last_became_peered":"2024-09-20T22:43:57.427971+0000","last_unstale":"2024-09-20T22:45:23.157185+0000","last_undegraded":"2024-09-20T22:45:23.157185+0000","last_fullsized":"2024-09-20T22:45:23.157185+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:44:41.574713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:44:41.574713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:46:10.688311+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":380,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.149263+0000","last_change":"2024-09-20T22:44:18.785085+0000","last_active":"2024-09-20T22:45:23.149263+0000","last_peered":"2024-09-20T22:45:23.149263+0000","last_clean":"2024-09-20T22:45:23.149263+0000","last_became_active":"2024-09-20T22:43:57.432650+0000","last_became_peered":"2024-09-20T22:43:57.432650+0000","last_unstale":"2024-09-20T22:45:23.149263+0000","last_undegraded":"2024-09-20T22:45:23.149263+0000","last_fullsized":"2024-09-20T22:45:23.149263+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:44:18.784990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:44:18.784990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:47.957020+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"113'22","reported_seq":409,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.178530+0000","last_change":"2024-09-20T22:44:55.423258+0000","last_active":"2024-09-20T22:45:23.178530+0000","last_peered":"2024-09-20T22:45:23.178530+0000","last_clean":"2024-09-20T22:45:23.178530+0000","last_became_active":"2024-09-20T22:41:40.170131+0000","last_became_peered":"2024-09-20T22:41:40.170131+0000","last_unstale":"2024-09-20T22:45:23.178530+0000","last_undegraded":"2024-09-20T22:45:23.178530+0000","last_fullsized":"2024-09-20T22:45:23.178530+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'22","last_scrub_stamp":"2024-09-20T22:44:55.423175+0000","last_deep_scrub":"113'22","last_deep_scrub_stamp":"2024-09-20T22:43:12.191878+0000","last_clean_scrub_stamp":"2024-09-20T22:44:55.423175+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:46:10.184178+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":372,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.150176+0000","last_change":"2024-09-20T22:44:22.085304+0000","last_active":"2024-09-20T22:45:23.150176+0000","last_peered":"2024-09-20T22:45:23.150176+0000","last_clean":"2024-09-20T22:45:23.150176+0000","last_became_active":"2024-09-20T22:43:57.435288+0000","last_became_peered":"2024-09-20T22:43:57.435288+0000","last_unstale":"2024-09-20T22:45:23.150176+0000","last_undegraded":"2024-09-20T22:45:23.150176+0000","last_fullsized":"2024-09-20T22:45:23.150176+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:44:22.085224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:44:22.085224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:28.981956+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":354,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.164871+0000","last_change":"2024-09-20T22:41:44.832468+0000","last_active":"2024-09-20T22:45:23.164871+0000","last_peered":"2024-09-20T22:45:23.164871+0000","last_clean":"2024-09-20T22:45:23.164871+0000","last_became_active":"2024-09-20T22:41:44.832131+0000","last_became_peered":"2024-09-20T22:41:44.832131+0000","last_unstale":"2024-09-20T22:45:23.164871+0000","last_undegraded":"2024-09-20T22:45:23.164871+0000","last_fullsized":"2024-09-20T22:45:23.164871+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:41:43.176430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":200,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.149827+0000","last_change":"2024-09-20T22:45:15.043547+0000","last_active":"2024-09-20T22:45:23.149827+0000","last_peered":"2024-09-20T22:45:23.149827+0000","last_clean":"2024-09-20T22:45:23.149827+0000","last_became_active":"2024-09-20T22:43:57.435636+0000","last_became_peered":"2024-09-20T22:43:57.435636+0000","last_unstale":"2024-09-20T22:45:23.149827+0000","last_undegraded":"2024-09-20T22:45:23.149827+0000","last_fullsized":"2024-09-20T22:45:23.149827+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:45:15.043443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:45:15.043443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:46:29.789629+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"74'12","reported_seq":429,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.159926+0000","last_change":"2024-09-20T22:44:42.141018+0000","last_active":"2024-09-20T22:45:23.159926+0000","last_peered":"2024-09-20T22:45:23.159926+0000","last_clean":"2024-09-20T22:45:23.159926+0000","last_became_active":"2024-09-20T22:43:57.438162+0000","last_became_peered":"2024-09-20T22:43:57.438162+0000","last_unstale":"2024-09-20T22:45:23.159926+0000","last_undegraded":"2024-09-20T22:45:23.159926+0000","last_fullsized":"2024-09-20T22:45:23.159926+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-09-20T22:44:42.140915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:44:42.140915+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:57.947345+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":192,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.161176+0000","last_change":"2024-09-20T22:44:14.235908+0000","last_active":"2024-09-20T22:45:23.161176+0000","last_peered":"2024-09-20T22:45:23.161176+0000","last_clean":"2024-09-20T22:45:23.161176+0000","last_became_active":"2024-09-20T22:43:57.429179+0000","last_became_peered":"2024-09-20T22:43:57.429179+0000","last_unstale":"2024-09-20T22:45:23.161176+0000","last_undegraded":"2024-09-20T22:45:23.161176+0000","last_fullsized":"2024-09-20T22:45:23.161176+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:44:14.235804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:44:14.235804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:26.950979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"154'2","reported_seq":364,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.160906+0000","last_change":"2024-09-20T22:43:54.134185+0000","last_active":"2024-09-20T22:45:23.160906+0000","last_peered":"2024-09-20T22:45:23.160906+0000","last_clean":"2024-09-20T22:45:23.160906+0000","last_became_active":"2024-09-20T22:41:44.199822+0000","last_became_peered":"2024-09-20T22:41:44.199822+0000","last_unstale":"2024-09-20T22:45:23.160906+0000","last_undegraded":"2024-09-20T22:45:23.160906+0000","last_fullsized":"2024-09-20T22:45:23.160906+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"154'2","last_scrub_stamp":"2024-09-20T22:43:54.134066+0000","last_deep_scrub":"154'2","last_deep_scrub_stamp":"2024-09-20T22:43:54.134066+0000","last_clean_scrub_stamp":"2024-09-20T22:43:54.134066+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"67'1","reported_seq":385,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.149214+0000","last_change":"2024-09-20T22:43:57.462130+0000","last_active":"2024-09-20T22:45:23.149214+0000","last_peered":"2024-09-20T22:45:23.149214+0000","last_clean":"2024-09-20T22:45:23.149214+0000","last_became_active":"2024-09-20T22:43:57.443155+0000","last_became_peered":"2024-09-20T22:43:57.443155+0000","last_unstale":"2024-09-20T22:45:23.149214+0000","last_undegraded":"2024-09-20T22:45:23.149214+0000","last_fullsized":"2024-09-20T22:45:23.149214+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-09-20T22:43:35.266242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:43:35.266242+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"151'15","reported_seq":390,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.151455+0000","last_change":"2024-09-20T22:43:50.715845+0000","last_active":"2024-09-20T22:45:23.151455+0000","last_peered":"2024-09-20T22:45:23.151455+0000","last_clean":"2024-09-20T22:45:23.151455+0000","last_became_active":"2024-09-20T22:41:40.252001+0000","last_became_peered":"2024-09-20T22:41:40.252001+0000","last_unstale":"2024-09-20T22:45:23.151455+0000","last_undegraded":"2024-09-20T22:45:23.151455+0000","last_fullsized":"2024-09-20T22:45:23.151455+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"151'15","last_scrub_stamp":"2024-09-20T22:43:50.715761+0000","last_deep_scrub":"151'15","last_deep_scrub_stamp":"2024-09-20T22:43:50.715761+0000","last_clean_scrub_stamp":"2024-09-20T22:43:50.715761+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"159'9","reported_seq":436,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.157908+0000","last_change":"2024-09-20T22:44:15.499231+0000","last_active":"2024-09-20T22:45:23.157908+0000","last_peered":"2024-09-20T22:45:23.157908+0000","last_clean":"2024-09-20T22:45:23.157908+0000","last_became_active":"2024-09-20T22:43:57.424879+0000","last_became_peered":"2024-09-20T22:43:57.424879+0000","last_unstale":"2024-09-20T22:45:23.157908+0000","last_undegraded":"2024-09-20T22:45:23.157908+0000","last_fullsized":"2024-09-20T22:45:23.157908+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"159'9","last_scrub_stamp":"2024-09-20T22:44:15.499150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:44:15.499150+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:39.142015+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.c","version":"244'362","reported_seq":1161,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.165684+0000","last_change":"2024-09-20T22:45:23.161309+0000","last_active":"2024-09-20T22:45:23.165684+0000","last_peered":"2024-09-20T22:45:23.165684+0000","last_clean":"2024-09-20T22:45:23.165684+0000","last_became_active":"2024-09-20T22:42:52.404027+0000","last_became_peered":"2024-09-20T22:42:52.404027+0000","last_unstale":"2024-09-20T22:45:23.165684+0000","last_undegraded":"2024-09-20T22:45:23.165684+0000","last_fullsized":"2024-09-20T22:45:23.165684+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"194'228","last_scrub_stamp":"2024-09-20T22:44:34.610972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:44:34.610972+0000","objects_scrubbed":5,"log_size":362,"log_dups_size":0,"ondisk_log_size":362,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:48.337213+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0041581589999999998,"stat_sum":{"num_bytes":24773279,"num_objects":14,"num_object_clones":7,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":1,"num_read":262,"num_read_kb":250031,"num_write":189,"num_write_kb":72609,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"42","length":"1"},{"start":"44","length":"1"},{"start":"4c","length":"1"},{"start":"4e","length":"1"}]},{"pgid":"1.a","version":"77'128","reported_seq":611,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.473798+0000","last_change":"2024-09-20T22:43:05.763879+0000","last_active":"2024-09-20T22:45:23.473798+0000","last_peered":"2024-09-20T22:45:23.473798+0000","last_clean":"2024-09-20T22:45:23.473798+0000","last_became_active":"2024-09-20T22:42:52.407791+0000","last_became_peered":"2024-09-20T22:42:52.407791+0000","last_unstale":"2024-09-20T22:45:23.473798+0000","last_undegraded":"2024-09-20T22:45:23.473798+0000","last_fullsized":"2024-09-20T22:45:23.473798+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:43:05.763796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:43:05.763796+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"74'15","reported_seq":425,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.150208+0000","last_change":"2024-09-20T22:44:55.972422+0000","last_active":"2024-09-20T22:45:23.150208+0000","last_peered":"2024-09-20T22:45:23.150208+0000","last_clean":"2024-09-20T22:45:23.150208+0000","last_became_active":"2024-09-20T22:43:57.441355+0000","last_became_peered":"2024-09-20T22:43:57.441355+0000","last_unstale":"2024-09-20T22:45:23.150208+0000","last_undegraded":"2024-09-20T22:45:23.150208+0000","last_fullsized":"2024-09-20T22:45:23.150208+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-09-20T22:44:55.972333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:44:55.972333+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:46:20.894884+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":374,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.163766+0000","last_change":"2024-09-20T22:45:19.027462+0000","last_active":"2024-09-20T22:45:23.163766+0000","last_peered":"2024-09-20T22:45:23.163766+0000","last_clean":"2024-09-20T22:45:23.163766+0000","last_became_active":"2024-09-20T22:41:38.156749+0000","last_became_peered":"2024-09-20T22:41:38.156749+0000","last_unstale":"2024-09-20T22:45:23.163766+0000","last_undegraded":"2024-09-20T22:45:23.163766+0000","last_fullsized":"2024-09-20T22:45:23.163766+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:45:19.027393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:45:19.027393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:46:31.029466+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"159'9","reported_seq":239,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.474732+0000","last_change":"2024-09-20T22:44:55.604951+0000","last_active":"2024-09-20T22:45:23.474732+0000","last_peered":"2024-09-20T22:45:23.474732+0000","last_clean":"2024-09-20T22:45:23.474732+0000","last_became_active":"2024-09-20T22:43:57.425367+0000","last_became_peered":"2024-09-20T22:43:57.425367+0000","last_unstale":"2024-09-20T22:45:23.474732+0000","last_undegraded":"2024-09-20T22:45:23.474732+0000","last_fullsized":"2024-09-20T22:45:23.474732+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"159'9","last_scrub_stamp":"2024-09-20T22:44:55.604845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:44:55.604845+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:56.211983+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":362,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.474511+0000","last_change":"2024-09-20T22:43:53.790857+0000","last_active":"2024-09-20T22:45:23.474511+0000","last_peered":"2024-09-20T22:45:23.474511+0000","last_clean":"2024-09-20T22:45:23.474511+0000","last_became_active":"2024-09-20T22:41:44.198347+0000","last_became_peered":"2024-09-20T22:41:44.198347+0000","last_unstale":"2024-09-20T22:45:23.474511+0000","last_undegraded":"2024-09-20T22:45:23.474511+0000","last_fullsized":"2024-09-20T22:45:23.474511+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:53.790756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:43:53.790756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3","version":"135'8","reported_seq":765,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.167486+0000","last_change":"2024-09-20T22:45:23.168785+0000","last_active":"2024-09-20T22:45:23.167486+0000","last_peered":"2024-09-20T22:45:23.167486+0000","last_clean":"2024-09-20T22:45:23.167486+0000","last_became_active":"2024-09-20T22:43:57.439135+0000","last_became_peered":"2024-09-20T22:43:57.439135+0000","last_unstale":"2024-09-20T22:45:23.167486+0000","last_undegraded":"2024-09-20T22:45:23.167486+0000","last_fullsized":"2024-09-20T22:45:23.167486+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'8","last_scrub_stamp":"2024-09-20T22:43:47.153362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:43:47.153362+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0085516979999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"42","length":"1"},{"start":"44","length":"1"},{"start":"4c","length":"1"},{"start":"4e","length":"1"}]},{"pgid":"1.5","version":"77'128","reported_seq":483,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.178720+0000","last_change":"2024-09-20T22:44:10.229558+0000","last_active":"2024-09-20T22:45:23.178720+0000","last_peered":"2024-09-20T22:45:23.178720+0000","last_clean":"2024-09-20T22:45:23.178720+0000","last_became_active":"2024-09-20T22:44:00.145455+0000","last_became_peered":"2024-09-20T22:44:00.145455+0000","last_unstale":"2024-09-20T22:45:23.178720+0000","last_undegraded":"2024-09-20T22:45:23.178720+0000","last_fullsized":"2024-09-20T22:45:23.178720+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:44:10.229483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:44:10.229483+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,5],"acting":[2,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"74'18","reported_seq":402,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.163464+0000","last_change":"2024-09-20T22:44:23.220792+0000","last_active":"2024-09-20T22:45:23.163464+0000","last_peered":"2024-09-20T22:45:23.163464+0000","last_clean":"2024-09-20T22:45:23.163464+0000","last_became_active":"2024-09-20T22:41:40.251868+0000","last_became_peered":"2024-09-20T22:41:40.251868+0000","last_unstale":"2024-09-20T22:45:23.163464+0000","last_undegraded":"2024-09-20T22:45:23.163464+0000","last_fullsized":"2024-09-20T22:45:23.163464+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'18","last_scrub_stamp":"2024-09-20T22:44:23.220709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:44:23.220709+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:43.393391+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":374,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.165322+0000","last_change":"2024-09-20T22:43:51.170223+0000","last_active":"2024-09-20T22:45:23.165322+0000","last_peered":"2024-09-20T22:45:23.165322+0000","last_clean":"2024-09-20T22:45:23.165322+0000","last_became_active":"2024-09-20T22:41:38.161335+0000","last_became_peered":"2024-09-20T22:41:38.161335+0000","last_unstale":"2024-09-20T22:45:23.165322+0000","last_undegraded":"2024-09-20T22:45:23.165322+0000","last_fullsized":"2024-09-20T22:45:23.165322+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:51.170155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:43:51.170155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":184,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.161225+0000","last_change":"2024-09-20T22:43:57.427579+0000","last_active":"2024-09-20T22:45:23.161225+0000","last_peered":"2024-09-20T22:45:23.161225+0000","last_clean":"2024-09-20T22:45:23.161225+0000","last_became_active":"2024-09-20T22:43:57.427287+0000","last_became_peered":"2024-09-20T22:43:57.427287+0000","last_unstale":"2024-09-20T22:45:23.161225+0000","last_undegraded":"2024-09-20T22:45:23.161225+0000","last_fullsized":"2024-09-20T22:45:23.161225+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:07.947945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:07.947945+0000","last_clean_scrub_stamp":"2024-09-20T22:43:07.947945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":185,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.160033+0000","last_change":"2024-09-20T22:43:57.435367+0000","last_active":"2024-09-20T22:45:23.160033+0000","last_peered":"2024-09-20T22:45:23.160033+0000","last_clean":"2024-09-20T22:45:23.160033+0000","last_became_active":"2024-09-20T22:43:57.435057+0000","last_became_peered":"2024-09-20T22:43:57.435057+0000","last_unstale":"2024-09-20T22:45:23.160033+0000","last_undegraded":"2024-09-20T22:45:23.160033+0000","last_fullsized":"2024-09-20T22:45:23.160033+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:53.036601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:53.036601+0000","last_clean_scrub_stamp":"2024-09-20T22:43:53.036601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2","version":"241'98","reported_seq":419,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.474239+0000","last_change":"2024-09-20T22:45:23.474686+0000","last_active":"2024-09-20T22:45:23.474239+0000","last_peered":"2024-09-20T22:45:23.474239+0000","last_clean":"2024-09-20T22:45:23.474239+0000","last_became_active":"2024-09-20T22:44:09.177625+0000","last_became_peered":"2024-09-20T22:44:09.177625+0000","last_unstale":"2024-09-20T22:45:23.474239+0000","last_undegraded":"2024-09-20T22:45:23.474239+0000","last_fullsized":"2024-09-20T22:45:23.474239+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"138'78","last_scrub_stamp":"2024-09-20T22:44:09.667741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:44:09.667741+0000","objects_scrubbed":1,"log_size":98,"log_dups_size":0,"ondisk_log_size":98,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:31.436753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00076417399999999995,"stat_sum":{"num_bytes":3042100,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":22,"num_read_kb":31714,"num_write":10,"num_write_kb":2709,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"42","length":"1"},{"start":"44","length":"1"},{"start":"4c","length":"1"},{"start":"4e","length":"1"}]},{"pgid":"1.4","version":"77'128","reported_seq":624,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.160977+0000","last_change":"2024-09-20T22:43:26.151507+0000","last_active":"2024-09-20T22:45:23.160977+0000","last_peered":"2024-09-20T22:45:23.160977+0000","last_clean":"2024-09-20T22:45:23.160977+0000","last_became_active":"2024-09-20T22:43:19.125641+0000","last_became_peered":"2024-09-20T22:43:19.125641+0000","last_unstale":"2024-09-20T22:45:23.160977+0000","last_undegraded":"2024-09-20T22:45:23.160977+0000","last_fullsized":"2024-09-20T22:45:23.160977+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:43:26.151406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:43:26.151406+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"74'14","reported_seq":248,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.473905+0000","last_change":"2024-09-20T22:44:17.668446+0000","last_active":"2024-09-20T22:45:23.473905+0000","last_peered":"2024-09-20T22:45:23.473905+0000","last_clean":"2024-09-20T22:45:23.473905+0000","last_became_active":"2024-09-20T22:43:57.422847+0000","last_became_peered":"2024-09-20T22:43:57.422847+0000","last_unstale":"2024-09-20T22:45:23.473905+0000","last_undegraded":"2024-09-20T22:45:23.473905+0000","last_fullsized":"2024-09-20T22:45:23.473905+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'14","last_scrub_stamp":"2024-09-20T22:44:17.668364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:44:17.668364+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:34.632610+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.6","version":"67'1","reported_seq":385,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.151747+0000","last_change":"2024-09-20T22:43:57.457010+0000","last_active":"2024-09-20T22:45:23.151747+0000","last_peered":"2024-09-20T22:45:23.151747+0000","last_clean":"2024-09-20T22:45:23.151747+0000","last_became_active":"2024-09-20T22:43:57.431232+0000","last_became_peered":"2024-09-20T22:43:57.431232+0000","last_unstale":"2024-09-20T22:45:23.151747+0000","last_undegraded":"2024-09-20T22:45:23.151747+0000","last_fullsized":"2024-09-20T22:45:23.151747+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-09-20T22:43:29.694945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:43:29.694945+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"159'9","reported_seq":430,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.162858+0000","last_change":"2024-09-20T22:45:23.017718+0000","last_active":"2024-09-20T22:45:23.162858+0000","last_peered":"2024-09-20T22:45:23.162858+0000","last_clean":"2024-09-20T22:45:23.162858+0000","last_became_active":"2024-09-20T22:43:57.442440+0000","last_became_peered":"2024-09-20T22:43:57.442440+0000","last_unstale":"2024-09-20T22:45:23.162858+0000","last_undegraded":"2024-09-20T22:45:23.162858+0000","last_fullsized":"2024-09-20T22:45:23.162858+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"159'9","last_scrub_stamp":"2024-09-20T22:45:23.017605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:45:23.017605+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:46:43.927062+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":362,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.159972+0000","last_change":"2024-09-20T22:44:19.972826+0000","last_active":"2024-09-20T22:45:23.159972+0000","last_peered":"2024-09-20T22:45:23.159972+0000","last_clean":"2024-09-20T22:45:23.159972+0000","last_became_active":"2024-09-20T22:41:44.831901+0000","last_became_peered":"2024-09-20T22:41:44.831901+0000","last_unstale":"2024-09-20T22:45:23.159972+0000","last_undegraded":"2024-09-20T22:45:23.159972+0000","last_fullsized":"2024-09-20T22:45:23.159972+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:44:19.972742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:44:19.972742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:48.587176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1","version":"241'106","reported_seq":842,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.169105+0000","last_change":"2024-09-20T22:45:23.169374+0000","last_active":"2024-09-20T22:45:23.169105+0000","last_peered":"2024-09-20T22:45:23.169105+0000","last_clean":"2024-09-20T22:45:23.169105+0000","last_became_active":"2024-09-20T22:43:57.440095+0000","last_became_peered":"2024-09-20T22:43:57.440095+0000","last_unstale":"2024-09-20T22:45:23.169105+0000","last_undegraded":"2024-09-20T22:45:23.169105+0000","last_fullsized":"2024-09-20T22:45:23.169105+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"170'72","last_scrub_stamp":"2024-09-20T22:44:12.228784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:44:12.228784+0000","objects_scrubbed":3,"log_size":106,"log_dups_size":0,"ondisk_log_size":106,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:25.062471+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00853487,"stat_sum":{"num_bytes":9122954,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":41,"num_read_kb":63464,"num_write":17,"num_write_kb":15254,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"42","length":"1"},{"start":"44","length":"1"},{"start":"4c","length":"1"},{"start":"4e","length":"1"}]},{"pgid":"1.7","version":"77'128","reported_seq":631,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.157596+0000","last_change":"2024-09-20T22:43:26.566566+0000","last_active":"2024-09-20T22:45:23.157596+0000","last_peered":"2024-09-20T22:45:23.157596+0000","last_clean":"2024-09-20T22:45:23.157596+0000","last_became_active":"2024-09-20T22:43:19.129015+0000","last_became_peered":"2024-09-20T22:43:19.129015+0000","last_unstale":"2024-09-20T22:45:23.157596+0000","last_undegraded":"2024-09-20T22:45:23.157596+0000","last_fullsized":"2024-09-20T22:45:23.157596+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:43:26.566436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:43:26.566436+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"74'10","reported_seq":411,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.149455+0000","last_change":"2024-09-20T22:44:59.049691+0000","last_active":"2024-09-20T22:45:23.149455+0000","last_peered":"2024-09-20T22:45:23.149455+0000","last_clean":"2024-09-20T22:45:23.149455+0000","last_became_active":"2024-09-20T22:43:57.435599+0000","last_became_peered":"2024-09-20T22:43:57.435599+0000","last_unstale":"2024-09-20T22:45:23.149455+0000","last_undegraded":"2024-09-20T22:45:23.149455+0000","last_fullsized":"2024-09-20T22:45:23.149455+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-09-20T22:44:59.049607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:44:59.049607+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:46:05.780194+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":382,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.474996+0000","last_change":"2024-09-20T22:45:17.702193+0000","last_active":"2024-09-20T22:45:23.474996+0000","last_peered":"2024-09-20T22:45:23.474996+0000","last_clean":"2024-09-20T22:45:23.474996+0000","last_became_active":"2024-09-20T22:41:38.155629+0000","last_became_peered":"2024-09-20T22:41:38.155629+0000","last_unstale":"2024-09-20T22:45:23.474996+0000","last_undegraded":"2024-09-20T22:45:23.474996+0000","last_fullsized":"2024-09-20T22:45:23.474996+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:45:17.702106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:45:17.702106+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:46:43.322898+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"170'9","reported_seq":179,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.159980+0000","last_change":"2024-09-20T22:44:09.175786+0000","last_active":"2024-09-20T22:45:23.159980+0000","last_peered":"2024-09-20T22:45:23.159980+0000","last_clean":"2024-09-20T22:45:23.159980+0000","last_became_active":"2024-09-20T22:44:09.171546+0000","last_became_peered":"2024-09-20T22:44:09.171546+0000","last_unstale":"2024-09-20T22:45:23.159980+0000","last_undegraded":"2024-09-20T22:45:23.159980+0000","last_fullsized":"2024-09-20T22:45:23.159980+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-09-20T22:43:08.643587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:43:08.643587+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":362,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.147814+0000","last_change":"2024-09-20T22:43:33.657671+0000","last_active":"2024-09-20T22:45:23.147814+0000","last_peered":"2024-09-20T22:45:23.147814+0000","last_clean":"2024-09-20T22:45:23.147814+0000","last_became_active":"2024-09-20T22:41:44.197619+0000","last_became_peered":"2024-09-20T22:41:44.197619+0000","last_unstale":"2024-09-20T22:45:23.147814+0000","last_undegraded":"2024-09-20T22:45:23.147814+0000","last_fullsized":"2024-09-20T22:45:23.147814+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:33.657587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:43:33.657587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.0","version":"241'83","reported_seq":856,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.473987+0000","last_change":"2024-09-20T22:45:23.474178+0000","last_active":"2024-09-20T22:45:23.473987+0000","last_peered":"2024-09-20T22:45:23.473987+0000","last_clean":"2024-09-20T22:45:23.473987+0000","last_became_active":"2024-09-20T22:43:37.136845+0000","last_became_peered":"2024-09-20T22:43:37.136845+0000","last_unstale":"2024-09-20T22:45:23.473987+0000","last_undegraded":"2024-09-20T22:45:23.473987+0000","last_fullsized":"2024-09-20T22:45:23.473987+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"145'58","last_scrub_stamp":"2024-09-20T22:43:45.705214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:43:45.705214+0000","objects_scrubbed":3,"log_size":83,"log_dups_size":0,"ondisk_log_size":83,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00046772200000000001,"stat_sum":{"num_bytes":6775112,"num_objects":5,"num_object_clones":3,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":2,"num_read":23,"num_read_kb":24717,"num_write":16,"num_write_kb":3438,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"42","length":"1"},{"start":"44","length":"1"},{"start":"4c","length":"1"},{"start":"4e","length":"1"}]},{"pgid":"1.6","version":"77'128","reported_seq":639,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.157030+0000","last_change":"2024-09-20T22:45:17.693086+0000","last_active":"2024-09-20T22:45:23.157030+0000","last_peered":"2024-09-20T22:45:23.157030+0000","last_clean":"2024-09-20T22:45:23.157030+0000","last_became_active":"2024-09-20T22:43:19.130331+0000","last_became_peered":"2024-09-20T22:43:19.130331+0000","last_unstale":"2024-09-20T22:45:23.157030+0000","last_undegraded":"2024-09-20T22:45:23.157030+0000","last_fullsized":"2024-09-20T22:45:23.157030+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:45:17.692996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:45:17.692996+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:46:42.614208+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"74'19","reported_seq":414,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.149019+0000","last_change":"2024-09-20T22:44:37.632952+0000","last_active":"2024-09-20T22:45:23.149019+0000","last_peered":"2024-09-20T22:45:23.149019+0000","last_clean":"2024-09-20T22:45:23.149019+0000","last_became_active":"2024-09-20T22:41:40.251866+0000","last_became_peered":"2024-09-20T22:41:40.251866+0000","last_unstale":"2024-09-20T22:45:23.149019+0000","last_undegraded":"2024-09-20T22:45:23.149019+0000","last_fullsized":"2024-09-20T22:45:23.149019+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'19","last_scrub_stamp":"2024-09-20T22:44:37.632834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:44:37.632834+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:52.348513+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":382,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.150499+0000","last_change":"2024-09-20T22:45:21.038355+0000","last_active":"2024-09-20T22:45:23.150499+0000","last_peered":"2024-09-20T22:45:23.150499+0000","last_clean":"2024-09-20T22:45:23.150499+0000","last_became_active":"2024-09-20T22:41:38.160832+0000","last_became_peered":"2024-09-20T22:41:38.160832+0000","last_unstale":"2024-09-20T22:45:23.150499+0000","last_undegraded":"2024-09-20T22:45:23.150499+0000","last_fullsized":"2024-09-20T22:45:23.150499+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:45:21.038284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:45:21.038284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:46:26.049904+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":366,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.156011+0000","last_change":"2024-09-20T22:44:19.441854+0000","last_active":"2024-09-20T22:45:23.156011+0000","last_peered":"2024-09-20T22:45:23.156011+0000","last_clean":"2024-09-20T22:45:23.156011+0000","last_became_active":"2024-09-20T22:41:42.184412+0000","last_became_peered":"2024-09-20T22:41:42.184412+0000","last_unstale":"2024-09-20T22:45:23.156011+0000","last_undegraded":"2024-09-20T22:45:23.156011+0000","last_fullsized":"2024-09-20T22:45:23.156011+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:44:19.441770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:44:19.441770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:34.420562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":362,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.149084+0000","last_change":"2024-09-20T22:43:53.247501+0000","last_active":"2024-09-20T22:45:23.149084+0000","last_peered":"2024-09-20T22:45:23.149084+0000","last_clean":"2024-09-20T22:45:23.149084+0000","last_became_active":"2024-09-20T22:41:44.830538+0000","last_became_peered":"2024-09-20T22:41:44.830538+0000","last_unstale":"2024-09-20T22:45:23.149084+0000","last_undegraded":"2024-09-20T22:45:23.149084+0000","last_fullsized":"2024-09-20T22:45:23.149084+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:53.247430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:43:53.247430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.7","version":"244'194","reported_seq":970,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.159980+0000","last_change":"2024-09-20T22:45:23.159980+0000","last_active":"2024-09-20T22:45:23.159980+0000","last_peered":"2024-09-20T22:45:23.159980+0000","last_clean":"2024-09-20T22:45:23.159980+0000","last_became_active":"2024-09-20T22:43:37.132943+0000","last_became_peered":"2024-09-20T22:43:37.132943+0000","last_unstale":"2024-09-20T22:45:23.159980+0000","last_undegraded":"2024-09-20T22:45:23.159980+0000","last_fullsized":"2024-09-20T22:45:23.159980+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"235'189","last_scrub_stamp":"2024-09-20T22:45:16.672762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:45:16.672762+0000","objects_scrubbed":3,"log_size":194,"log_dups_size":0,"ondisk_log_size":194,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:46:37.670765+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0019663760000000001,"stat_sum":{"num_bytes":7334629,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":68,"num_read_kb":82780,"num_write":44,"num_write_kb":14512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"42","length":"1"},{"start":"44","length":"1"},{"start":"4c","length":"1"},{"start":"4e","length":"1"}]},{"pgid":"1.1","version":"77'128","reported_seq":624,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.473631+0000","last_change":"2024-09-20T22:43:26.761137+0000","last_active":"2024-09-20T22:45:23.473631+0000","last_peered":"2024-09-20T22:45:23.473631+0000","last_clean":"2024-09-20T22:45:23.473631+0000","last_became_active":"2024-09-20T22:43:19.125806+0000","last_became_peered":"2024-09-20T22:43:19.125806+0000","last_unstale":"2024-09-20T22:45:23.473631+0000","last_undegraded":"2024-09-20T22:45:23.473631+0000","last_fullsized":"2024-09-20T22:45:23.473631+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:43:26.761032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:43:26.761032+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"86'38","reported_seq":482,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.150431+0000","last_change":"2024-09-20T22:44:41.077590+0000","last_active":"2024-09-20T22:45:23.150431+0000","last_peered":"2024-09-20T22:45:23.150431+0000","last_clean":"2024-09-20T22:45:23.150431+0000","last_became_active":"2024-09-20T22:41:40.166958+0000","last_became_peered":"2024-09-20T22:41:40.166958+0000","last_unstale":"2024-09-20T22:45:23.150431+0000","last_undegraded":"2024-09-20T22:45:23.150431+0000","last_fullsized":"2024-09-20T22:45:23.150431+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'38","last_scrub_stamp":"2024-09-20T22:44:41.077475+0000","last_deep_scrub":"86'38","last_deep_scrub_stamp":"2024-09-20T22:42:43.935677+0000","last_clean_scrub_stamp":"2024-09-20T22:44:41.077475+0000","objects_scrubbed":10,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:46:05.497572+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":192,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.149196+0000","last_change":"2024-09-20T22:44:38.652538+0000","last_active":"2024-09-20T22:45:23.149196+0000","last_peered":"2024-09-20T22:45:23.149196+0000","last_clean":"2024-09-20T22:45:23.149196+0000","last_became_active":"2024-09-20T22:43:57.432267+0000","last_became_peered":"2024-09-20T22:43:57.432267+0000","last_unstale":"2024-09-20T22:45:23.149196+0000","last_undegraded":"2024-09-20T22:45:23.149196+0000","last_fullsized":"2024-09-20T22:45:23.149196+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:44:38.652446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:44:38.652446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:53.468832+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"69'2","reported_seq":392,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.474366+0000","last_change":"2024-09-20T22:44:35.668676+0000","last_active":"2024-09-20T22:45:23.474366+0000","last_peered":"2024-09-20T22:45:23.474366+0000","last_clean":"2024-09-20T22:45:23.474366+0000","last_became_active":"2024-09-20T22:41:37.148479+0000","last_became_peered":"2024-09-20T22:41:37.148479+0000","last_unstale":"2024-09-20T22:45:23.474366+0000","last_undegraded":"2024-09-20T22:45:23.474366+0000","last_fullsized":"2024-09-20T22:45:23.474366+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2024-09-20T22:44:35.668576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:36.130684+0000","last_clean_scrub_stamp":"2024-09-20T22:44:35.668576+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:39.072433+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00087280599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":372,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.150545+0000","last_change":"2024-09-20T22:44:11.765375+0000","last_active":"2024-09-20T22:45:23.150545+0000","last_peered":"2024-09-20T22:45:23.150545+0000","last_clean":"2024-09-20T22:45:23.150545+0000","last_became_active":"2024-09-20T22:43:57.431815+0000","last_became_peered":"2024-09-20T22:43:57.431815+0000","last_unstale":"2024-09-20T22:45:23.150545+0000","last_undegraded":"2024-09-20T22:45:23.150545+0000","last_fullsized":"2024-09-20T22:45:23.150545+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:44:11.765248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:44:11.765248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:34.701647+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":368,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.164601+0000","last_change":"2024-09-20T22:43:57.434389+0000","last_active":"2024-09-20T22:45:23.164601+0000","last_peered":"2024-09-20T22:45:23.164601+0000","last_clean":"2024-09-20T22:45:23.164601+0000","last_became_active":"2024-09-20T22:43:57.434044+0000","last_became_peered":"2024-09-20T22:43:57.434044+0000","last_unstale":"2024-09-20T22:45:23.164601+0000","last_undegraded":"2024-09-20T22:45:23.164601+0000","last_fullsized":"2024-09-20T22:45:23.164601+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:09.406141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:43:09.406141+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"210'50","reported_seq":827,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.150076+0000","last_change":"2024-09-20T22:45:23.150248+0000","last_active":"2024-09-20T22:45:23.150076+0000","last_peered":"2024-09-20T22:45:23.150076+0000","last_clean":"2024-09-20T22:45:23.150076+0000","last_became_active":"2024-09-20T22:43:57.429486+0000","last_became_peered":"2024-09-20T22:43:57.429486+0000","last_unstale":"2024-09-20T22:45:23.150076+0000","last_undegraded":"2024-09-20T22:45:23.150076+0000","last_fullsized":"2024-09-20T22:45:23.150076+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"210'50","last_scrub_stamp":"2024-09-20T22:45:14.807813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:45:14.807813+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:46:33.039193+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00043259400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":20,"num_read_kb":28189,"num_write":6,"num_write_kb":1350,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"42","length":"1"},{"start":"44","length":"1"},{"start":"4c","length":"1"},{"start":"4e","length":"1"}]},{"pgid":"1.2","version":"77'128","reported_seq":682,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.159885+0000","last_change":"2024-09-20T22:44:58.121790+0000","last_active":"2024-09-20T22:45:23.159885+0000","last_peered":"2024-09-20T22:45:23.159885+0000","last_clean":"2024-09-20T22:45:23.159885+0000","last_became_active":"2024-09-20T22:44:00.146907+0000","last_became_peered":"2024-09-20T22:44:00.146907+0000","last_unstale":"2024-09-20T22:45:23.159885+0000","last_undegraded":"2024-09-20T22:45:23.159885+0000","last_fullsized":"2024-09-20T22:45:23.159885+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:44:58.121721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:44:58.121721+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:46:14.009238+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"74'13","reported_seq":397,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.150413+0000","last_change":"2024-09-20T22:43:31.240331+0000","last_active":"2024-09-20T22:45:23.150413+0000","last_peered":"2024-09-20T22:45:23.150413+0000","last_clean":"2024-09-20T22:45:23.150413+0000","last_became_active":"2024-09-20T22:41:40.169473+0000","last_became_peered":"2024-09-20T22:41:40.169473+0000","last_unstale":"2024-09-20T22:45:23.150413+0000","last_undegraded":"2024-09-20T22:45:23.150413+0000","last_fullsized":"2024-09-20T22:45:23.150413+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'13","last_scrub_stamp":"2024-09-20T22:43:31.240196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:43:31.240196+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":382,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.149492+0000","last_change":"2024-09-20T22:44:35.131693+0000","last_active":"2024-09-20T22:45:23.149492+0000","last_peered":"2024-09-20T22:45:23.149492+0000","last_clean":"2024-09-20T22:45:23.149492+0000","last_became_active":"2024-09-20T22:41:38.156837+0000","last_became_peered":"2024-09-20T22:41:38.156837+0000","last_unstale":"2024-09-20T22:45:23.149492+0000","last_undegraded":"2024-09-20T22:45:23.149492+0000","last_fullsized":"2024-09-20T22:45:23.149492+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:44:35.131583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:44:35.131583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:47.311461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"67'1","reported_seq":390,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.177728+0000","last_change":"2024-09-20T22:44:39.441694+0000","last_active":"2024-09-20T22:45:23.177728+0000","last_peered":"2024-09-20T22:45:23.177728+0000","last_clean":"2024-09-20T22:45:23.177728+0000","last_became_active":"2024-09-20T22:41:37.148165+0000","last_became_peered":"2024-09-20T22:41:37.148165+0000","last_unstale":"2024-09-20T22:45:23.177728+0000","last_undegraded":"2024-09-20T22:45:23.177728+0000","last_fullsized":"2024-09-20T22:45:23.177728+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-09-20T22:44:39.441585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:36.130684+0000","last_clean_scrub_stamp":"2024-09-20T22:44:39.441585+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:40.621607+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001301459,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":365,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.179394+0000","last_change":"2024-09-20T22:44:14.284410+0000","last_active":"2024-09-20T22:45:23.179394+0000","last_peered":"2024-09-20T22:45:23.179394+0000","last_clean":"2024-09-20T22:45:23.179394+0000","last_became_active":"2024-09-20T22:41:42.185112+0000","last_became_peered":"2024-09-20T22:41:42.185112+0000","last_unstale":"2024-09-20T22:45:23.179394+0000","last_undegraded":"2024-09-20T22:45:23.179394+0000","last_fullsized":"2024-09-20T22:45:23.179394+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:44:14.284296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:44:14.284296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:23.561043+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":362,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.162156+0000","last_change":"2024-09-20T22:44:14.946132+0000","last_active":"2024-09-20T22:45:23.162156+0000","last_peered":"2024-09-20T22:45:23.162156+0000","last_clean":"2024-09-20T22:45:23.162156+0000","last_became_active":"2024-09-20T22:41:44.830515+0000","last_became_peered":"2024-09-20T22:41:44.830515+0000","last_unstale":"2024-09-20T22:45:23.162156+0000","last_undegraded":"2024-09-20T22:45:23.162156+0000","last_fullsized":"2024-09-20T22:45:23.162156+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:44:14.946052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:44:14.946052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"239'39","reported_seq":793,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.162156+0000","last_change":"2024-09-20T22:45:23.162911+0000","last_active":"2024-09-20T22:45:23.162156+0000","last_peered":"2024-09-20T22:45:23.162156+0000","last_clean":"2024-09-20T22:45:23.162156+0000","last_became_active":"2024-09-20T22:43:37.133478+0000","last_became_peered":"2024-09-20T22:43:37.133478+0000","last_unstale":"2024-09-20T22:45:23.162156+0000","last_undegraded":"2024-09-20T22:45:23.162156+0000","last_fullsized":"2024-09-20T22:45:23.162156+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'3","last_scrub_stamp":"2024-09-20T22:43:49.197854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:43:49.197854+0000","objects_scrubbed":1,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00114061,"stat_sum":{"num_bytes":6689219,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":18,"num_read_kb":12236,"num_write":24,"num_write_kb":12103,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"42","length":"1"},{"start":"44","length":"1"},{"start":"4c","length":"1"},{"start":"4e","length":"1"}]},{"pgid":"1.3","version":"77'128","reported_seq":633,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.151159+0000","last_change":"2024-09-20T22:44:00.144446+0000","last_active":"2024-09-20T22:45:23.151159+0000","last_peered":"2024-09-20T22:45:23.151159+0000","last_clean":"2024-09-20T22:45:23.151159+0000","last_became_active":"2024-09-20T22:44:00.144099+0000","last_became_peered":"2024-09-20T22:44:00.144099+0000","last_unstale":"2024-09-20T22:45:23.151159+0000","last_undegraded":"2024-09-20T22:45:23.151159+0000","last_fullsized":"2024-09-20T22:45:23.151159+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:43:25.244624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:43:25.244624+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"74'12","reported_seq":416,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.151552+0000","last_change":"2024-09-20T22:45:19.776628+0000","last_active":"2024-09-20T22:45:23.151552+0000","last_peered":"2024-09-20T22:45:23.151552+0000","last_clean":"2024-09-20T22:45:23.151552+0000","last_became_active":"2024-09-20T22:43:09.120425+0000","last_became_peered":"2024-09-20T22:43:09.120425+0000","last_unstale":"2024-09-20T22:45:23.151552+0000","last_undegraded":"2024-09-20T22:45:23.151552+0000","last_fullsized":"2024-09-20T22:45:23.151552+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-09-20T22:45:19.776506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:45:19.776506+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:46:49.522686+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":380,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.148820+0000","last_change":"2024-09-20T22:44:19.834724+0000","last_active":"2024-09-20T22:45:23.148820+0000","last_peered":"2024-09-20T22:45:23.148820+0000","last_clean":"2024-09-20T22:45:23.148820+0000","last_became_active":"2024-09-20T22:43:57.421167+0000","last_became_peered":"2024-09-20T22:43:57.421167+0000","last_unstale":"2024-09-20T22:45:23.148820+0000","last_undegraded":"2024-09-20T22:45:23.148820+0000","last_fullsized":"2024-09-20T22:45:23.148820+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:44:19.834641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:44:19.834641+0000","last_clean_scrub_stamp":"2024-09-20T22:44:19.834641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:43.561314+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"75'5","reported_seq":605,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.161024+0000","last_change":"2024-09-20T22:44:13.885050+0000","last_active":"2024-09-20T22:45:23.161024+0000","last_peered":"2024-09-20T22:45:23.161024+0000","last_clean":"2024-09-20T22:45:23.161024+0000","last_became_active":"2024-09-20T22:41:37.148383+0000","last_became_peered":"2024-09-20T22:41:37.148383+0000","last_unstale":"2024-09-20T22:45:23.161024+0000","last_undegraded":"2024-09-20T22:45:23.161024+0000","last_fullsized":"2024-09-20T22:45:23.161024+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-09-20T22:44:13.884968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:36.130684+0000","last_clean_scrub_stamp":"2024-09-20T22:44:13.884968+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00078631700000000003,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":220,"num_read_kb":215,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":366,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.475206+0000","last_change":"2024-09-20T22:44:38.788128+0000","last_active":"2024-09-20T22:45:23.475206+0000","last_peered":"2024-09-20T22:45:23.475206+0000","last_clean":"2024-09-20T22:45:23.475206+0000","last_became_active":"2024-09-20T22:41:42.187346+0000","last_became_peered":"2024-09-20T22:41:42.187346+0000","last_unstale":"2024-09-20T22:45:23.475206+0000","last_undegraded":"2024-09-20T22:45:23.475206+0000","last_fullsized":"2024-09-20T22:45:23.475206+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:44:38.788045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:44:38.788045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:46:03.203383+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":370,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.150080+0000","last_change":"2024-09-20T22:44:54.014154+0000","last_active":"2024-09-20T22:45:23.150080+0000","last_peered":"2024-09-20T22:45:23.150080+0000","last_clean":"2024-09-20T22:45:23.150080+0000","last_became_active":"2024-09-20T22:41:44.199046+0000","last_became_peered":"2024-09-20T22:41:44.199046+0000","last_unstale":"2024-09-20T22:45:23.150080+0000","last_undegraded":"2024-09-20T22:45:23.150080+0000","last_fullsized":"2024-09-20T22:45:23.150080+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:44:54.014085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:44:54.014085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:46:01.646305+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.6","version":"240'64","reported_seq":826,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.152211+0000","last_change":"2024-09-20T22:45:23.152469+0000","last_active":"2024-09-20T22:45:23.152211+0000","last_peered":"2024-09-20T22:45:23.152211+0000","last_clean":"2024-09-20T22:45:23.152211+0000","last_became_active":"2024-09-20T22:43:37.136591+0000","last_became_peered":"2024-09-20T22:43:37.136591+0000","last_unstale":"2024-09-20T22:45:23.152211+0000","last_undegraded":"2024-09-20T22:45:23.152211+0000","last_fullsized":"2024-09-20T22:45:23.152211+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"150'60","last_scrub_stamp":"2024-09-20T22:43:49.735664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:43:49.735664+0000","objects_scrubbed":0,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00045895700000000002,"stat_sum":{"num_bytes":3025293,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":5,"num_write_kb":1153,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"42","length":"1"},{"start":"44","length":"1"},{"start":"4c","length":"1"},{"start":"4e","length":"1"}]},{"pgid":"4.5","version":"159'17","reported_seq":462,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.157165+0000","last_change":"2024-09-20T22:43:57.462065+0000","last_active":"2024-09-20T22:45:23.157165+0000","last_peered":"2024-09-20T22:45:23.157165+0000","last_clean":"2024-09-20T22:45:23.157165+0000","last_became_active":"2024-09-20T22:43:57.433076+0000","last_became_peered":"2024-09-20T22:43:57.433076+0000","last_unstale":"2024-09-20T22:45:23.157165+0000","last_undegraded":"2024-09-20T22:45:23.157165+0000","last_fullsized":"2024-09-20T22:45:23.157165+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'16","last_scrub_stamp":"2024-09-20T22:43:56.348264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:43:56.348264+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"67'1","reported_seq":380,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.160296+0000","last_change":"2024-09-20T22:45:22.053343+0000","last_active":"2024-09-20T22:45:23.160296+0000","last_peered":"2024-09-20T22:45:23.160296+0000","last_clean":"2024-09-20T22:45:23.160296+0000","last_became_active":"2024-09-20T22:41:38.158917+0000","last_became_peered":"2024-09-20T22:41:38.158917+0000","last_unstale":"2024-09-20T22:45:23.160296+0000","last_undegraded":"2024-09-20T22:45:23.160296+0000","last_fullsized":"2024-09-20T22:45:23.160296+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-09-20T22:45:22.053262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:45:22.053262+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:46:34.545771+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"77'128","reported_seq":665,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.161315+0000","last_change":"2024-09-20T22:44:35.887245+0000","last_active":"2024-09-20T22:45:23.161315+0000","last_peered":"2024-09-20T22:45:23.161315+0000","last_clean":"2024-09-20T22:45:23.161315+0000","last_became_active":"2024-09-20T22:43:19.126925+0000","last_became_peered":"2024-09-20T22:43:19.126925+0000","last_unstale":"2024-09-20T22:45:23.161315+0000","last_undegraded":"2024-09-20T22:45:23.161315+0000","last_fullsized":"2024-09-20T22:45:23.161315+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:44:35.887120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:44:35.887120+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:55.559505+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":366,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.161150+0000","last_change":"2024-09-20T22:43:52.163679+0000","last_active":"2024-09-20T22:45:23.161150+0000","last_peered":"2024-09-20T22:45:23.161150+0000","last_clean":"2024-09-20T22:45:23.161150+0000","last_became_active":"2024-09-20T22:41:42.184387+0000","last_became_peered":"2024-09-20T22:41:42.184387+0000","last_unstale":"2024-09-20T22:45:23.161150+0000","last_undegraded":"2024-09-20T22:45:23.161150+0000","last_fullsized":"2024-09-20T22:45:23.161150+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:52.163576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:43:52.163576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":368,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.474988+0000","last_change":"2024-09-20T22:43:57.434803+0000","last_active":"2024-09-20T22:45:23.474988+0000","last_peered":"2024-09-20T22:45:23.474988+0000","last_clean":"2024-09-20T22:45:23.474988+0000","last_became_active":"2024-09-20T22:43:57.434435+0000","last_became_peered":"2024-09-20T22:43:57.434435+0000","last_unstale":"2024-09-20T22:45:23.474988+0000","last_undegraded":"2024-09-20T22:45:23.474988+0000","last_fullsized":"2024-09-20T22:45:23.474988+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:02.696105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:43:02.696105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.b","version":"77'128","reported_seq":639,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.151075+0000","last_change":"2024-09-20T22:45:02.051386+0000","last_active":"2024-09-20T22:45:23.151075+0000","last_peered":"2024-09-20T22:45:23.151075+0000","last_clean":"2024-09-20T22:45:23.151075+0000","last_became_active":"2024-09-20T22:43:24.124252+0000","last_became_peered":"2024-09-20T22:43:24.124252+0000","last_unstale":"2024-09-20T22:45:23.151075+0000","last_undegraded":"2024-09-20T22:45:23.151075+0000","last_fullsized":"2024-09-20T22:45:23.151075+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:45:02.051304+0000","last_deep_scrub":"77'128","last_deep_scrub_stamp":"2024-09-20T22:45:02.051304+0000","last_clean_scrub_stamp":"2024-09-20T22:45:02.051304+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:46:20.486842+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.d","version":"244'77","reported_seq":835,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.191145+0000","last_change":"2024-09-20T22:45:23.157835+0000","last_active":"2024-09-20T22:45:23.191145+0000","last_peered":"2024-09-20T22:45:23.191145+0000","last_clean":"2024-09-20T22:45:23.191145+0000","last_became_active":"2024-09-20T22:42:52.410381+0000","last_became_peered":"2024-09-20T22:42:52.410381+0000","last_unstale":"2024-09-20T22:45:23.191145+0000","last_undegraded":"2024-09-20T22:45:23.191145+0000","last_fullsized":"2024-09-20T22:45:23.191145+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"194'71","last_scrub_stamp":"2024-09-20T22:44:40.529449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:44:40.529449+0000","objects_scrubbed":1,"log_size":77,"log_dups_size":0,"ondisk_log_size":77,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:52.550302+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00071469000000000003,"stat_sum":{"num_bytes":4387434,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":53,"num_read_kb":49493,"num_write":30,"num_write_kb":8842,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"42","length":"1"},{"start":"44","length":"1"},{"start":"4c","length":"1"},{"start":"4e","length":"1"}]},{"pgid":"4.e","version":"74'11","reported_seq":260,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.157339+0000","last_change":"2024-09-20T22:44:16.481158+0000","last_active":"2024-09-20T22:45:23.157339+0000","last_peered":"2024-09-20T22:45:23.157339+0000","last_clean":"2024-09-20T22:45:23.157339+0000","last_became_active":"2024-09-20T22:44:09.177976+0000","last_became_peered":"2024-09-20T22:44:09.177976+0000","last_unstale":"2024-09-20T22:45:23.157339+0000","last_undegraded":"2024-09-20T22:45:23.157339+0000","last_fullsized":"2024-09-20T22:45:23.157339+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-09-20T22:44:16.481063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:44:16.481063+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:45.292329+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":192,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.177617+0000","last_change":"2024-09-20T22:45:19.468033+0000","last_active":"2024-09-20T22:45:23.177617+0000","last_peered":"2024-09-20T22:45:23.177617+0000","last_clean":"2024-09-20T22:45:23.177617+0000","last_became_active":"2024-09-20T22:43:57.432492+0000","last_became_peered":"2024-09-20T22:43:57.432492+0000","last_unstale":"2024-09-20T22:45:23.177617+0000","last_undegraded":"2024-09-20T22:45:23.177617+0000","last_fullsized":"2024-09-20T22:45:23.177617+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:45:19.467967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:45:19.467967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:46:25.484754+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":372,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.474771+0000","last_change":"2024-09-20T22:44:57.607725+0000","last_active":"2024-09-20T22:45:23.474771+0000","last_peered":"2024-09-20T22:45:23.474771+0000","last_clean":"2024-09-20T22:45:23.474771+0000","last_became_active":"2024-09-20T22:43:57.419756+0000","last_became_peered":"2024-09-20T22:43:57.419756+0000","last_unstale":"2024-09-20T22:45:23.474771+0000","last_undegraded":"2024-09-20T22:45:23.474771+0000","last_fullsized":"2024-09-20T22:45:23.474771+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:44:57.607616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:44:57.607616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:46:21.194848+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":362,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.161082+0000","last_change":"2024-09-20T22:44:21.225215+0000","last_active":"2024-09-20T22:45:23.161082+0000","last_peered":"2024-09-20T22:45:23.161082+0000","last_clean":"2024-09-20T22:45:23.161082+0000","last_became_active":"2024-09-20T22:41:44.832009+0000","last_became_peered":"2024-09-20T22:41:44.832009+0000","last_unstale":"2024-09-20T22:45:23.161082+0000","last_undegraded":"2024-09-20T22:45:23.161082+0000","last_fullsized":"2024-09-20T22:45:23.161082+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:44:21.225119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:44:21.225119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:39.909132+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.e","version":"241'97","reported_seq":861,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.194007+0000","last_change":"2024-09-20T22:45:23.194286+0000","last_active":"2024-09-20T22:45:23.194007+0000","last_peered":"2024-09-20T22:45:23.194007+0000","last_clean":"2024-09-20T22:45:23.194007+0000","last_became_active":"2024-09-20T22:42:52.405039+0000","last_became_peered":"2024-09-20T22:42:52.405039+0000","last_unstale":"2024-09-20T22:45:23.194007+0000","last_undegraded":"2024-09-20T22:45:23.194007+0000","last_fullsized":"2024-09-20T22:45:23.194007+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"174'60","last_scrub_stamp":"2024-09-20T22:44:18.301776+0000","last_deep_scrub":"174'60","last_deep_scrub_stamp":"2024-09-20T22:44:18.301776+0000","last_clean_scrub_stamp":"2024-09-20T22:44:18.301776+0000","objects_scrubbed":2,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:33.912823+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0.015165435999999999,"stat_sum":{"num_bytes":8678745,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":81,"num_read_kb":97171,"num_write":42,"num_write_kb":16800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"42","length":"1"},{"start":"44","length":"1"},{"start":"4c","length":"1"},{"start":"4e","length":"1"}]},{"pgid":"1.8","version":"77'128","reported_seq":619,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.473607+0000","last_change":"2024-09-20T22:45:22.698507+0000","last_active":"2024-09-20T22:45:23.473607+0000","last_peered":"2024-09-20T22:45:23.473607+0000","last_clean":"2024-09-20T22:45:23.473607+0000","last_became_active":"2024-09-20T22:42:52.405594+0000","last_became_peered":"2024-09-20T22:42:52.405594+0000","last_unstale":"2024-09-20T22:45:23.473607+0000","last_undegraded":"2024-09-20T22:45:23.473607+0000","last_fullsized":"2024-09-20T22:45:23.473607+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:45:22.698390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:45:22.698390+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:46:25.962156+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"74'17","reported_seq":257,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.178868+0000","last_change":"2024-09-20T22:44:21.261543+0000","last_active":"2024-09-20T22:45:23.178868+0000","last_peered":"2024-09-20T22:45:23.178868+0000","last_clean":"2024-09-20T22:45:23.178868+0000","last_became_active":"2024-09-20T22:43:57.432621+0000","last_became_peered":"2024-09-20T22:43:57.432621+0000","last_unstale":"2024-09-20T22:45:23.178868+0000","last_undegraded":"2024-09-20T22:45:23.178868+0000","last_fullsized":"2024-09-20T22:45:23.178868+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'17","last_scrub_stamp":"2024-09-20T22:44:21.261443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:44:21.261443+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:39.405257+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":28,"num_read_kb":18,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":380,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.156437+0000","last_change":"2024-09-20T22:43:57.429041+0000","last_active":"2024-09-20T22:45:23.156437+0000","last_peered":"2024-09-20T22:45:23.156437+0000","last_clean":"2024-09-20T22:45:23.156437+0000","last_became_active":"2024-09-20T22:43:57.425267+0000","last_became_peered":"2024-09-20T22:43:57.425267+0000","last_unstale":"2024-09-20T22:45:23.156437+0000","last_undegraded":"2024-09-20T22:45:23.156437+0000","last_fullsized":"2024-09-20T22:45:23.156437+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:35.492562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:35.492562+0000","last_clean_scrub_stamp":"2024-09-20T22:43:35.492562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":372,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.150208+0000","last_change":"2024-09-20T22:44:11.041771+0000","last_active":"2024-09-20T22:45:23.150208+0000","last_peered":"2024-09-20T22:45:23.150208+0000","last_clean":"2024-09-20T22:45:23.150208+0000","last_became_active":"2024-09-20T22:43:57.435048+0000","last_became_peered":"2024-09-20T22:43:57.435048+0000","last_unstale":"2024-09-20T22:45:23.150208+0000","last_undegraded":"2024-09-20T22:45:23.150208+0000","last_fullsized":"2024-09-20T22:45:23.150208+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:44:11.041699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:44:11.041699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:34.130343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":375,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.179001+0000","last_change":"2024-09-20T22:44:20.297496+0000","last_active":"2024-09-20T22:45:23.179001+0000","last_peered":"2024-09-20T22:45:23.179001+0000","last_clean":"2024-09-20T22:45:23.179001+0000","last_became_active":"2024-09-20T22:43:57.430630+0000","last_became_peered":"2024-09-20T22:43:57.430630+0000","last_unstale":"2024-09-20T22:45:23.179001+0000","last_undegraded":"2024-09-20T22:45:23.179001+0000","last_fullsized":"2024-09-20T22:45:23.179001+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:44:20.297424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:44:20.297424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:44.120776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"241'152","reported_seq":923,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.151365+0000","last_change":"2024-09-20T22:45:23.151546+0000","last_active":"2024-09-20T22:45:23.151365+0000","last_peered":"2024-09-20T22:45:23.151365+0000","last_clean":"2024-09-20T22:45:23.151365+0000","last_became_active":"2024-09-20T22:42:52.401210+0000","last_became_peered":"2024-09-20T22:42:52.401210+0000","last_unstale":"2024-09-20T22:45:23.151365+0000","last_undegraded":"2024-09-20T22:45:23.151365+0000","last_fullsized":"2024-09-20T22:45:23.151365+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"194'90","last_scrub_stamp":"2024-09-20T22:44:34.651203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:44:34.651203+0000","objects_scrubbed":3,"log_size":152,"log_dups_size":0,"ondisk_log_size":152,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:49.260153+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00079766400000000001,"stat_sum":{"num_bytes":10775740,"num_objects":6,"num_object_clones":3,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":111,"num_read_kb":100655,"num_write":73,"num_write_kb":25838,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"42","length":"1"},{"start":"44","length":"1"},{"start":"4c","length":"1"},{"start":"4e","length":"1"}]},{"pgid":"1.9","version":"77'128","reported_seq":652,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.161370+0000","last_change":"2024-09-20T22:43:03.239347+0000","last_active":"2024-09-20T22:45:23.161370+0000","last_peered":"2024-09-20T22:45:23.161370+0000","last_clean":"2024-09-20T22:45:23.161370+0000","last_became_active":"2024-09-20T22:42:52.400353+0000","last_became_peered":"2024-09-20T22:42:52.400353+0000","last_unstale":"2024-09-20T22:45:23.161370+0000","last_undegraded":"2024-09-20T22:45:23.161370+0000","last_fullsized":"2024-09-20T22:45:23.161370+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:43:03.239276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:43:03.239276+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"74'10","reported_seq":230,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.161412+0000","last_change":"2024-09-20T22:43:57.486043+0000","last_active":"2024-09-20T22:45:23.161412+0000","last_peered":"2024-09-20T22:45:23.161412+0000","last_clean":"2024-09-20T22:45:23.161412+0000","last_became_active":"2024-09-20T22:43:57.441016+0000","last_became_peered":"2024-09-20T22:43:57.441016+0000","last_unstale":"2024-09-20T22:45:23.161412+0000","last_undegraded":"2024-09-20T22:45:23.161412+0000","last_fullsized":"2024-09-20T22:45:23.161412+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-09-20T22:43:51.040007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:43:51.040007+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":380,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.160516+0000","last_change":"2024-09-20T22:43:57.441917+0000","last_active":"2024-09-20T22:45:23.160516+0000","last_peered":"2024-09-20T22:45:23.160516+0000","last_clean":"2024-09-20T22:45:23.160516+0000","last_became_active":"2024-09-20T22:43:57.441623+0000","last_became_peered":"2024-09-20T22:43:57.441623+0000","last_unstale":"2024-09-20T22:45:23.160516+0000","last_undegraded":"2024-09-20T22:45:23.160516+0000","last_fullsized":"2024-09-20T22:45:23.160516+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:05.431346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:43:05.431346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"74'8","reported_seq":419,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.178787+0000","last_change":"2024-09-20T22:43:56.238237+0000","last_active":"2024-09-20T22:45:23.178787+0000","last_peered":"2024-09-20T22:45:23.178787+0000","last_clean":"2024-09-20T22:45:23.178787+0000","last_became_active":"2024-09-20T22:41:42.185863+0000","last_became_peered":"2024-09-20T22:41:42.185863+0000","last_unstale":"2024-09-20T22:45:23.178787+0000","last_undegraded":"2024-09-20T22:45:23.178787+0000","last_fullsized":"2024-09-20T22:45:23.178787+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-09-20T22:43:56.238114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:43:56.238114+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:23.984720+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":193,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.150009+0000","last_change":"2024-09-20T22:44:13.962056+0000","last_active":"2024-09-20T22:45:23.150009+0000","last_peered":"2024-09-20T22:45:23.150009+0000","last_clean":"2024-09-20T22:45:23.150009+0000","last_became_active":"2024-09-20T22:43:57.430067+0000","last_became_peered":"2024-09-20T22:43:57.430067+0000","last_unstale":"2024-09-20T22:45:23.150009+0000","last_undegraded":"2024-09-20T22:45:23.150009+0000","last_fullsized":"2024-09-20T22:45:23.150009+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:44:13.961983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:44:13.961983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:24.591102+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.e","version":"77'128","reported_seq":483,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.160872+0000","last_change":"2024-09-20T22:44:12.918980+0000","last_active":"2024-09-20T22:45:23.160872+0000","last_peered":"2024-09-20T22:45:23.160872+0000","last_clean":"2024-09-20T22:45:23.160872+0000","last_became_active":"2024-09-20T22:44:00.145891+0000","last_became_peered":"2024-09-20T22:44:00.145891+0000","last_unstale":"2024-09-20T22:45:23.160872+0000","last_undegraded":"2024-09-20T22:45:23.160872+0000","last_fullsized":"2024-09-20T22:45:23.160872+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:44:12.918885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:44:12.918885+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.8","version":"213'51","reported_seq":829,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.178101+0000","last_change":"2024-09-20T22:45:23.178277+0000","last_active":"2024-09-20T22:45:23.178101+0000","last_peered":"2024-09-20T22:45:23.178101+0000","last_clean":"2024-09-20T22:45:23.178101+0000","last_became_active":"2024-09-20T22:43:57.431081+0000","last_became_peered":"2024-09-20T22:43:57.431081+0000","last_unstale":"2024-09-20T22:45:23.178101+0000","last_undegraded":"2024-09-20T22:45:23.178101+0000","last_fullsized":"2024-09-20T22:45:23.178101+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'51","last_scrub_stamp":"2024-09-20T22:45:21.414332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:45:21.414332+0000","objects_scrubbed":1,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:46:23.073807+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00073161099999999996,"stat_sum":{"num_bytes":2117715,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":11,"num_read_kb":8281,"num_write":7,"num_write_kb":3247,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"42","length":"1"},{"start":"44","length":"1"},{"start":"4c","length":"1"},{"start":"4e","length":"1"}]},{"pgid":"4.b","version":"74'9","reported_seq":410,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.148616+0000","last_change":"2024-09-20T22:44:57.603747+0000","last_active":"2024-09-20T22:45:23.148616+0000","last_peered":"2024-09-20T22:45:23.148616+0000","last_clean":"2024-09-20T22:45:23.148616+0000","last_became_active":"2024-09-20T22:43:57.442189+0000","last_became_peered":"2024-09-20T22:43:57.442189+0000","last_unstale":"2024-09-20T22:45:23.148616+0000","last_undegraded":"2024-09-20T22:45:23.148616+0000","last_fullsized":"2024-09-20T22:45:23.148616+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-09-20T22:44:57.603671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:44:57.603671+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:46:05.790764+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":374,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.474731+0000","last_change":"2024-09-20T22:43:52.773066+0000","last_active":"2024-09-20T22:45:23.474731+0000","last_peered":"2024-09-20T22:45:23.474731+0000","last_clean":"2024-09-20T22:45:23.474731+0000","last_became_active":"2024-09-20T22:41:38.155953+0000","last_became_peered":"2024-09-20T22:41:38.155953+0000","last_unstale":"2024-09-20T22:45:23.474731+0000","last_undegraded":"2024-09-20T22:45:23.474731+0000","last_fullsized":"2024-09-20T22:45:23.474731+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:52.772982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:43:52.772982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":371,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.178467+0000","last_change":"2024-09-20T22:43:57.440505+0000","last_active":"2024-09-20T22:45:23.178467+0000","last_peered":"2024-09-20T22:45:23.178467+0000","last_clean":"2024-09-20T22:45:23.178467+0000","last_became_active":"2024-09-20T22:43:57.440200+0000","last_became_peered":"2024-09-20T22:43:57.440200+0000","last_unstale":"2024-09-20T22:45:23.178467+0000","last_undegraded":"2024-09-20T22:45:23.178467+0000","last_fullsized":"2024-09-20T22:45:23.178467+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:55.274872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:43:55.274872+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":362,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.148734+0000","last_change":"2024-09-20T22:44:22.818177+0000","last_active":"2024-09-20T22:45:23.148734+0000","last_peered":"2024-09-20T22:45:23.148734+0000","last_clean":"2024-09-20T22:45:23.148734+0000","last_became_active":"2024-09-20T22:41:44.196633+0000","last_became_peered":"2024-09-20T22:41:44.196633+0000","last_unstale":"2024-09-20T22:45:23.148734+0000","last_undegraded":"2024-09-20T22:45:23.148734+0000","last_fullsized":"2024-09-20T22:45:23.148734+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:44:22.818078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:44:22.818078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:47.784094+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.f","version":"77'128","reported_seq":655,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.157966+0000","last_change":"2024-09-20T22:45:20.754466+0000","last_active":"2024-09-20T22:45:23.157966+0000","last_peered":"2024-09-20T22:45:23.157966+0000","last_clean":"2024-09-20T22:45:23.157966+0000","last_became_active":"2024-09-20T22:44:00.141317+0000","last_became_peered":"2024-09-20T22:44:00.141317+0000","last_unstale":"2024-09-20T22:45:23.157966+0000","last_undegraded":"2024-09-20T22:45:23.157966+0000","last_fullsized":"2024-09-20T22:45:23.157966+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:45:20.754385+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:45:20.754385+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:46:42.151483+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.9","version":"241'119","reported_seq":897,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.148570+0000","last_change":"2024-09-20T22:45:23.149258+0000","last_active":"2024-09-20T22:45:23.148570+0000","last_peered":"2024-09-20T22:45:23.148570+0000","last_clean":"2024-09-20T22:45:23.148570+0000","last_became_active":"2024-09-20T22:43:57.431328+0000","last_became_peered":"2024-09-20T22:43:57.431328+0000","last_unstale":"2024-09-20T22:45:23.148570+0000","last_undegraded":"2024-09-20T22:45:23.148570+0000","last_fullsized":"2024-09-20T22:45:23.148570+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"131'44","last_scrub_stamp":"2024-09-20T22:43:46.794463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:43:46.794463+0000","objects_scrubbed":1,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00089643999999999998,"stat_sum":{"num_bytes":8998804,"num_objects":5,"num_object_clones":3,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":28,"num_read_kb":26901,"num_write":33,"num_write_kb":10379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":3815156,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"42","length":"1"},{"start":"44","length":"1"},{"start":"4c","length":"1"},{"start":"4e","length":"1"}]},{"pgid":"4.a","version":"113'28","reported_seq":418,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.156153+0000","last_change":"2024-09-20T22:44:55.649681+0000","last_active":"2024-09-20T22:45:23.156153+0000","last_peered":"2024-09-20T22:45:23.156153+0000","last_clean":"2024-09-20T22:45:23.156153+0000","last_became_active":"2024-09-20T22:41:40.252011+0000","last_became_peered":"2024-09-20T22:41:40.252011+0000","last_unstale":"2024-09-20T22:45:23.156153+0000","last_undegraded":"2024-09-20T22:45:23.156153+0000","last_fullsized":"2024-09-20T22:45:23.156153+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'28","last_scrub_stamp":"2024-09-20T22:44:55.649589+0000","last_deep_scrub":"113'28","last_deep_scrub_stamp":"2024-09-20T22:43:12.725900+0000","last_clean_scrub_stamp":"2024-09-20T22:44:55.649589+0000","objects_scrubbed":9,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:59.901122+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":374,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.162403+0000","last_change":"2024-09-20T22:44:18.955889+0000","last_active":"2024-09-20T22:45:23.162403+0000","last_peered":"2024-09-20T22:45:23.162403+0000","last_clean":"2024-09-20T22:45:23.162403+0000","last_became_active":"2024-09-20T22:41:38.161524+0000","last_became_peered":"2024-09-20T22:41:38.161524+0000","last_unstale":"2024-09-20T22:45:23.162403+0000","last_undegraded":"2024-09-20T22:45:23.162403+0000","last_fullsized":"2024-09-20T22:45:23.162403+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:44:18.955801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:44:18.955801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:26.881892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":373,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.475566+0000","last_change":"2024-09-20T22:45:23.475586+0000","last_active":"2024-09-20T22:45:23.475566+0000","last_peered":"2024-09-20T22:45:23.475566+0000","last_clean":"2024-09-20T22:45:23.475566+0000","last_became_active":"2024-09-20T22:41:42.184342+0000","last_became_peered":"2024-09-20T22:41:42.184342+0000","last_unstale":"2024-09-20T22:45:23.475566+0000","last_undegraded":"2024-09-20T22:45:23.475566+0000","last_fullsized":"2024-09-20T22:45:23.475566+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:45:23.475514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:45:23.475514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:46:37.363869+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":370,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.161154+0000","last_change":"2024-09-20T22:44:42.956942+0000","last_active":"2024-09-20T22:45:23.161154+0000","last_peered":"2024-09-20T22:45:23.161154+0000","last_clean":"2024-09-20T22:45:23.161154+0000","last_became_active":"2024-09-20T22:41:44.199395+0000","last_became_peered":"2024-09-20T22:41:44.199395+0000","last_unstale":"2024-09-20T22:45:23.161154+0000","last_undegraded":"2024-09-20T22:45:23.161154+0000","last_fullsized":"2024-09-20T22:45:23.161154+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:44:42.956859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:44:42.956859+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:48.879336+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.c","version":"77'128","reported_seq":637,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.161499+0000","last_change":"2024-09-20T22:43:30.045082+0000","last_active":"2024-09-20T22:45:23.161499+0000","last_peered":"2024-09-20T22:45:23.161499+0000","last_clean":"2024-09-20T22:45:23.161499+0000","last_became_active":"2024-09-20T22:43:24.130949+0000","last_became_peered":"2024-09-20T22:43:24.130949+0000","last_unstale":"2024-09-20T22:45:23.161499+0000","last_undegraded":"2024-09-20T22:45:23.161499+0000","last_fullsized":"2024-09-20T22:45:23.161499+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:43:30.044975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:43:30.044975+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"242'167","reported_seq":927,"reported_epoch":244,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T22:45:23.159769+0000","last_change":"2024-09-20T22:45:20.164566+0000","last_active":"2024-09-20T22:45:23.159769+0000","last_peered":"2024-09-20T22:45:23.159769+0000","last_clean":"2024-09-20T22:45:23.159769+0000","last_became_active":"2024-09-20T22:42:52.405085+0000","last_became_peered":"2024-09-20T22:42:52.405085+0000","last_unstale":"2024-09-20T22:45:23.159769+0000","last_undegraded":"2024-09-20T22:45:23.159769+0000","last_fullsized":"2024-09-20T22:45:23.159769+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:42:51.354519+0000","objects_scrubbed":0,"log_size":167,"log_dups_size":0,"ondisk_log_size":167,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.036834921,"stat_sum":{"num_bytes":10891939,"num_objects":6,"num_object_clones":3,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":124,"num_read_kb":130842,"num_write":91,"num_write_kb":31652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"44","length":"1"}]},{"pgid":"4.9","version":"74'12","reported_seq":246,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.149183+0000","last_change":"2024-09-20T22:44:53.049195+0000","last_active":"2024-09-20T22:45:23.149183+0000","last_peered":"2024-09-20T22:45:23.149183+0000","last_clean":"2024-09-20T22:45:23.149183+0000","last_became_active":"2024-09-20T22:43:57.442218+0000","last_became_peered":"2024-09-20T22:43:57.442218+0000","last_unstale":"2024-09-20T22:45:23.149183+0000","last_undegraded":"2024-09-20T22:45:23.149183+0000","last_fullsized":"2024-09-20T22:45:23.149183+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-09-20T22:44:53.049111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:44:53.049111+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:55.990924+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":21,"num_read_kb":12,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":388,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.161382+0000","last_change":"2024-09-20T22:44:20.970396+0000","last_active":"2024-09-20T22:45:23.161382+0000","last_peered":"2024-09-20T22:45:23.161382+0000","last_clean":"2024-09-20T22:45:23.161382+0000","last_became_active":"2024-09-20T22:43:57.419980+0000","last_became_peered":"2024-09-20T22:43:57.419980+0000","last_unstale":"2024-09-20T22:45:23.161382+0000","last_undegraded":"2024-09-20T22:45:23.161382+0000","last_fullsized":"2024-09-20T22:45:23.161382+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:44:20.970293+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:44:20.970293+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:44.477869+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":373,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.177889+0000","last_change":"2024-09-20T22:44:37.379076+0000","last_active":"2024-09-20T22:45:23.177889+0000","last_peered":"2024-09-20T22:45:23.177889+0000","last_clean":"2024-09-20T22:45:23.177889+0000","last_became_active":"2024-09-20T22:41:42.181811+0000","last_became_peered":"2024-09-20T22:41:42.181811+0000","last_unstale":"2024-09-20T22:45:23.177889+0000","last_undegraded":"2024-09-20T22:45:23.177889+0000","last_fullsized":"2024-09-20T22:45:23.177889+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:44:37.379005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:44:37.379005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:44.358677+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":362,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.160338+0000","last_change":"2024-09-20T22:43:55.170518+0000","last_active":"2024-09-20T22:45:23.160338+0000","last_peered":"2024-09-20T22:45:23.160338+0000","last_clean":"2024-09-20T22:45:23.160338+0000","last_became_active":"2024-09-20T22:41:44.192609+0000","last_became_peered":"2024-09-20T22:41:44.192609+0000","last_unstale":"2024-09-20T22:45:23.160338+0000","last_undegraded":"2024-09-20T22:45:23.160338+0000","last_fullsized":"2024-09-20T22:45:23.160338+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:55.170379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:55.170379+0000","last_clean_scrub_stamp":"2024-09-20T22:43:55.170379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.d","version":"77'128","reported_seq":559,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.161462+0000","last_change":"2024-09-20T22:44:00.147264+0000","last_active":"2024-09-20T22:45:23.161462+0000","last_peered":"2024-09-20T22:45:23.161462+0000","last_clean":"2024-09-20T22:45:23.161462+0000","last_became_active":"2024-09-20T22:44:00.146905+0000","last_became_peered":"2024-09-20T22:44:00.146905+0000","last_unstale":"2024-09-20T22:45:23.161462+0000","last_undegraded":"2024-09-20T22:45:23.161462+0000","last_fullsized":"2024-09-20T22:45:23.161462+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:43:30.175229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:43:30.175229+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.b","version":"243'176","reported_seq":959,"reported_epoch":244,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T22:45:23.150737+0000","last_change":"2024-09-20T22:45:22.165536+0000","last_active":"2024-09-20T22:45:23.150737+0000","last_peered":"2024-09-20T22:45:23.150737+0000","last_clean":"2024-09-20T22:45:23.150737+0000","last_became_active":"2024-09-20T22:42:52.409642+0000","last_became_peered":"2024-09-20T22:42:52.409642+0000","last_unstale":"2024-09-20T22:45:23.150737+0000","last_undegraded":"2024-09-20T22:45:23.150737+0000","last_fullsized":"2024-09-20T22:45:23.150737+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"194'116","last_scrub_stamp":"2024-09-20T22:44:37.097674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:44:37.097674+0000","objects_scrubbed":3,"log_size":176,"log_dups_size":0,"ondisk_log_size":176,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:55.412520+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.011591878999999999,"stat_sum":{"num_bytes":18131521,"num_objects":10,"num_object_clones":6,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":157,"num_read_kb":162503,"num_write":103,"num_write_kb":35260,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"42","length":"1"},{"start":"44","length":"1"}]},{"pgid":"4.8","version":"74'15","reported_seq":395,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.473770+0000","last_change":"2024-09-20T22:45:00.648680+0000","last_active":"2024-09-20T22:45:23.473770+0000","last_peered":"2024-09-20T22:45:23.473770+0000","last_clean":"2024-09-20T22:45:23.473770+0000","last_became_active":"2024-09-20T22:41:40.252113+0000","last_became_peered":"2024-09-20T22:41:40.252113+0000","last_unstale":"2024-09-20T22:45:23.473770+0000","last_undegraded":"2024-09-20T22:45:23.473770+0000","last_fullsized":"2024-09-20T22:45:23.473770+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-09-20T22:45:00.648587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:45:00.648587+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:46:04.296690+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'2","reported_seq":398,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.162441+0000","last_change":"2024-09-20T22:44:16.936605+0000","last_active":"2024-09-20T22:45:23.162441+0000","last_peered":"2024-09-20T22:45:23.162441+0000","last_clean":"2024-09-20T22:45:23.162441+0000","last_became_active":"2024-09-20T22:43:57.422322+0000","last_became_peered":"2024-09-20T22:43:57.422322+0000","last_unstale":"2024-09-20T22:45:23.162441+0000","last_undegraded":"2024-09-20T22:45:23.162441+0000","last_fullsized":"2024-09-20T22:45:23.162441+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-09-20T22:44:16.936523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:44:16.936523+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"159'9","reported_seq":438,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.161173+0000","last_change":"2024-09-20T22:44:36.904443+0000","last_active":"2024-09-20T22:45:23.161173+0000","last_peered":"2024-09-20T22:45:23.161173+0000","last_clean":"2024-09-20T22:45:23.161173+0000","last_became_active":"2024-09-20T22:43:57.420663+0000","last_became_peered":"2024-09-20T22:43:57.420663+0000","last_unstale":"2024-09-20T22:45:23.161173+0000","last_undegraded":"2024-09-20T22:45:23.161173+0000","last_fullsized":"2024-09-20T22:45:23.161173+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"159'9","last_scrub_stamp":"2024-09-20T22:44:36.904367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:44:36.904367+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:54.974970+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":362,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.474012+0000","last_change":"2024-09-20T22:45:01.616499+0000","last_active":"2024-09-20T22:45:23.474012+0000","last_peered":"2024-09-20T22:45:23.474012+0000","last_clean":"2024-09-20T22:45:23.474012+0000","last_became_active":"2024-09-20T22:41:44.830556+0000","last_became_peered":"2024-09-20T22:41:44.830556+0000","last_unstale":"2024-09-20T22:45:23.474012+0000","last_undegraded":"2024-09-20T22:45:23.474012+0000","last_fullsized":"2024-09-20T22:45:23.474012+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:45:01.616426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:45:01.616426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:46:16.580554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.14","version":"239'49","reported_seq":574,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.160325+0000","last_change":"2024-09-20T22:45:23.164586+0000","last_active":"2024-09-20T22:45:23.160325+0000","last_peered":"2024-09-20T22:45:23.160325+0000","last_clean":"2024-09-20T22:45:23.160325+0000","last_became_active":"2024-09-20T22:43:39.132459+0000","last_became_peered":"2024-09-20T22:43:39.132459+0000","last_unstale":"2024-09-20T22:45:23.160325+0000","last_undegraded":"2024-09-20T22:45:23.160325+0000","last_fullsized":"2024-09-20T22:45:23.160325+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":5,"last_scrub":"147'38","last_scrub_stamp":"2024-09-20T22:43:49.134297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:43:49.134297+0000","objects_scrubbed":0,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0023211500000000001,"stat_sum":{"num_bytes":4112632,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":8510,"num_write":10,"num_write_kb":2755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"42","length":"1"},{"start":"44","length":"1"},{"start":"4c","length":"1"},{"start":"4e","length":"1"}]},{"pgid":"1.12","version":"77'128","reported_seq":548,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.474230+0000","last_change":"2024-09-20T22:43:29.735118+0000","last_active":"2024-09-20T22:45:23.474230+0000","last_peered":"2024-09-20T22:45:23.474230+0000","last_clean":"2024-09-20T22:45:23.474230+0000","last_became_active":"2024-09-20T22:43:24.124301+0000","last_became_peered":"2024-09-20T22:43:24.124301+0000","last_unstale":"2024-09-20T22:45:23.474230+0000","last_undegraded":"2024-09-20T22:45:23.474230+0000","last_fullsized":"2024-09-20T22:45:23.474230+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:43:29.735047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:43:29.735047+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":382,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.156273+0000","last_change":"2024-09-20T22:44:42.612217+0000","last_active":"2024-09-20T22:45:23.156273+0000","last_peered":"2024-09-20T22:45:23.156273+0000","last_clean":"2024-09-20T22:45:23.156273+0000","last_became_active":"2024-09-20T22:41:38.159139+0000","last_became_peered":"2024-09-20T22:41:38.159139+0000","last_unstale":"2024-09-20T22:45:23.156273+0000","last_undegraded":"2024-09-20T22:45:23.156273+0000","last_fullsized":"2024-09-20T22:45:23.156273+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:44:42.612144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:44:42.612144+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:46:11.366920+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"74'6","reported_seq":393,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.165172+0000","last_change":"2024-09-20T22:43:09.121694+0000","last_active":"2024-09-20T22:45:23.165172+0000","last_peered":"2024-09-20T22:45:23.165172+0000","last_clean":"2024-09-20T22:45:23.165172+0000","last_became_active":"2024-09-20T22:43:09.116887+0000","last_became_peered":"2024-09-20T22:43:09.116887+0000","last_unstale":"2024-09-20T22:45:23.165172+0000","last_undegraded":"2024-09-20T22:45:23.165172+0000","last_fullsized":"2024-09-20T22:45:23.165172+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'6","last_scrub_stamp":"2024-09-20T22:42:41.469214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:42:41.469214+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":366,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.474831+0000","last_change":"2024-09-20T22:43:09.705715+0000","last_active":"2024-09-20T22:45:23.474831+0000","last_peered":"2024-09-20T22:45:23.474831+0000","last_clean":"2024-09-20T22:45:23.474831+0000","last_became_active":"2024-09-20T22:41:42.184930+0000","last_became_peered":"2024-09-20T22:41:42.184930+0000","last_unstale":"2024-09-20T22:45:23.474831+0000","last_undegraded":"2024-09-20T22:45:23.474831+0000","last_fullsized":"2024-09-20T22:45:23.474831+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:09.705632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:09.705632+0000","last_clean_scrub_stamp":"2024-09-20T22:43:09.705632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"74'1","reported_seq":373,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.161876+0000","last_change":"2024-09-20T22:43:57.435632+0000","last_active":"2024-09-20T22:45:23.161876+0000","last_peered":"2024-09-20T22:45:23.161876+0000","last_clean":"2024-09-20T22:45:23.161876+0000","last_became_active":"2024-09-20T22:43:57.419681+0000","last_became_peered":"2024-09-20T22:43:57.419681+0000","last_unstale":"2024-09-20T22:45:23.161876+0000","last_undegraded":"2024-09-20T22:45:23.161876+0000","last_fullsized":"2024-09-20T22:45:23.161876+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-09-20T22:43:33.180910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:43:33.180910+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.15","version":"98'3","reported_seq":766,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.161544+0000","last_change":"2024-09-20T22:45:23.162711+0000","last_active":"2024-09-20T22:45:23.161544+0000","last_peered":"2024-09-20T22:45:23.161544+0000","last_clean":"2024-09-20T22:45:23.161544+0000","last_became_active":"2024-09-20T22:44:09.174719+0000","last_became_peered":"2024-09-20T22:44:09.174719+0000","last_unstale":"2024-09-20T22:45:23.161544+0000","last_undegraded":"2024-09-20T22:45:23.161544+0000","last_fullsized":"2024-09-20T22:45:23.161544+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":5,"last_scrub":"98'3","last_scrub_stamp":"2024-09-20T22:45:14.323226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:45:14.323226+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:46:43.088333+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0015573480000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"42","length":"1"},{"start":"44","length":"1"},{"start":"4c","length":"1"},{"start":"4e","length":"1"}]},{"pgid":"1.13","version":"77'128","reported_seq":648,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.150947+0000","last_change":"2024-09-20T22:44:38.112816+0000","last_active":"2024-09-20T22:45:23.150947+0000","last_peered":"2024-09-20T22:45:23.150947+0000","last_clean":"2024-09-20T22:45:23.150947+0000","last_became_active":"2024-09-20T22:44:00.144081+0000","last_became_peered":"2024-09-20T22:44:00.144081+0000","last_unstale":"2024-09-20T22:45:23.150947+0000","last_undegraded":"2024-09-20T22:45:23.150947+0000","last_fullsized":"2024-09-20T22:45:23.150947+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:44:38.112742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:44:38.112742+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:42.803148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.16","version":"74'9","reported_seq":394,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.151650+0000","last_change":"2024-09-20T22:44:52.644656+0000","last_active":"2024-09-20T22:45:23.151650+0000","last_peered":"2024-09-20T22:45:23.151650+0000","last_clean":"2024-09-20T22:45:23.151650+0000","last_became_active":"2024-09-20T22:41:40.253280+0000","last_became_peered":"2024-09-20T22:41:40.253280+0000","last_unstale":"2024-09-20T22:45:23.151650+0000","last_undegraded":"2024-09-20T22:45:23.151650+0000","last_fullsized":"2024-09-20T22:45:23.151650+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-09-20T22:44:52.644575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:44:52.644575+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:46:04.206183+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":380,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.161605+0000","last_change":"2024-09-20T22:44:11.942183+0000","last_active":"2024-09-20T22:45:23.161605+0000","last_peered":"2024-09-20T22:45:23.161605+0000","last_clean":"2024-09-20T22:45:23.161605+0000","last_became_active":"2024-09-20T22:43:57.419001+0000","last_became_peered":"2024-09-20T22:43:57.419001+0000","last_unstale":"2024-09-20T22:45:23.161605+0000","last_undegraded":"2024-09-20T22:45:23.161605+0000","last_fullsized":"2024-09-20T22:45:23.161605+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:44:11.942106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:44:11.942106+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":366,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.164974+0000","last_change":"2024-09-20T22:43:32.057321+0000","last_active":"2024-09-20T22:45:23.164974+0000","last_peered":"2024-09-20T22:45:23.164974+0000","last_clean":"2024-09-20T22:45:23.164974+0000","last_became_active":"2024-09-20T22:41:42.187706+0000","last_became_peered":"2024-09-20T22:41:42.187706+0000","last_unstale":"2024-09-20T22:45:23.164974+0000","last_undegraded":"2024-09-20T22:45:23.164974+0000","last_fullsized":"2024-09-20T22:45:23.164974+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:32.057240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:43:32.057240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":375,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.178285+0000","last_change":"2024-09-20T22:44:58.459406+0000","last_active":"2024-09-20T22:45:23.178285+0000","last_peered":"2024-09-20T22:45:23.178285+0000","last_clean":"2024-09-20T22:45:23.178285+0000","last_became_active":"2024-09-20T22:43:57.428284+0000","last_became_peered":"2024-09-20T22:43:57.428284+0000","last_unstale":"2024-09-20T22:45:23.178285+0000","last_undegraded":"2024-09-20T22:45:23.178285+0000","last_fullsized":"2024-09-20T22:45:23.178285+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:44:58.459332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:44:58.459332+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:46:11.813722+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.16","version":"241'68","reported_seq":740,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.151312+0000","last_change":"2024-09-20T22:45:23.152144+0000","last_active":"2024-09-20T22:45:23.151312+0000","last_peered":"2024-09-20T22:45:23.151312+0000","last_clean":"2024-09-20T22:45:23.151312+0000","last_became_active":"2024-09-20T22:43:57.423267+0000","last_became_peered":"2024-09-20T22:43:57.423267+0000","last_unstale":"2024-09-20T22:45:23.151312+0000","last_undegraded":"2024-09-20T22:45:23.151312+0000","last_fullsized":"2024-09-20T22:45:23.151312+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":5,"last_scrub":"136'41","last_scrub_stamp":"2024-09-20T22:44:09.743964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:44:09.743964+0000","objects_scrubbed":1,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:24.615791+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0016122160000000001,"stat_sum":{"num_bytes":6490515,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":20,"num_read_kb":15655,"num_write":11,"num_write_kb":2316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"42","length":"1"},{"start":"44","length":"1"},{"start":"4c","length":"1"},{"start":"4e","length":"1"}]},{"pgid":"1.10","version":"77'128","reported_seq":556,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.473425+0000","last_change":"2024-09-20T22:44:34.630283+0000","last_active":"2024-09-20T22:45:23.473425+0000","last_peered":"2024-09-20T22:45:23.473425+0000","last_clean":"2024-09-20T22:45:23.473425+0000","last_became_active":"2024-09-20T22:43:24.125651+0000","last_became_peered":"2024-09-20T22:43:24.125651+0000","last_unstale":"2024-09-20T22:45:23.473425+0000","last_undegraded":"2024-09-20T22:45:23.473425+0000","last_fullsized":"2024-09-20T22:45:23.473425+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:44:34.630184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:44:34.630184+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:40.023975+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.15","version":"74'9","reported_seq":386,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.474832+0000","last_change":"2024-09-20T22:43:49.825694+0000","last_active":"2024-09-20T22:45:23.474832+0000","last_peered":"2024-09-20T22:45:23.474832+0000","last_clean":"2024-09-20T22:45:23.474832+0000","last_became_active":"2024-09-20T22:41:40.252486+0000","last_became_peered":"2024-09-20T22:41:40.252486+0000","last_unstale":"2024-09-20T22:45:23.474832+0000","last_undegraded":"2024-09-20T22:45:23.474832+0000","last_fullsized":"2024-09-20T22:45:23.474832+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-09-20T22:43:49.825573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:43:49.825573+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":374,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.148127+0000","last_change":"2024-09-20T22:43:32.652702+0000","last_active":"2024-09-20T22:45:23.148127+0000","last_peered":"2024-09-20T22:45:23.148127+0000","last_clean":"2024-09-20T22:45:23.148127+0000","last_became_active":"2024-09-20T22:41:38.158146+0000","last_became_peered":"2024-09-20T22:41:38.158146+0000","last_unstale":"2024-09-20T22:45:23.148127+0000","last_undegraded":"2024-09-20T22:45:23.148127+0000","last_fullsized":"2024-09-20T22:45:23.148127+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:32.652587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:43:32.652587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"74'8","reported_seq":420,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.161292+0000","last_change":"2024-09-20T22:43:52.190172+0000","last_active":"2024-09-20T22:45:23.161292+0000","last_peered":"2024-09-20T22:45:23.161292+0000","last_clean":"2024-09-20T22:45:23.161292+0000","last_became_active":"2024-09-20T22:41:42.186352+0000","last_became_peered":"2024-09-20T22:41:42.186352+0000","last_unstale":"2024-09-20T22:45:23.161292+0000","last_undegraded":"2024-09-20T22:45:23.161292+0000","last_fullsized":"2024-09-20T22:45:23.161292+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-09-20T22:43:52.190097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:43:52.190097+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":201,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.178975+0000","last_change":"2024-09-20T22:45:22.448183+0000","last_active":"2024-09-20T22:45:23.178975+0000","last_peered":"2024-09-20T22:45:23.178975+0000","last_clean":"2024-09-20T22:45:23.178975+0000","last_became_active":"2024-09-20T22:43:57.424515+0000","last_became_peered":"2024-09-20T22:43:57.424515+0000","last_unstale":"2024-09-20T22:45:23.178975+0000","last_undegraded":"2024-09-20T22:45:23.178975+0000","last_fullsized":"2024-09-20T22:45:23.178975+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:45:22.448107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:45:22.448107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:46:26.917309+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.17","version":"243'198","reported_seq":677,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.162690+0000","last_change":"2024-09-20T22:45:23.162946+0000","last_active":"2024-09-20T22:45:23.162690+0000","last_peered":"2024-09-20T22:45:23.162690+0000","last_clean":"2024-09-20T22:45:23.162690+0000","last_became_active":"2024-09-20T22:43:39.130801+0000","last_became_peered":"2024-09-20T22:43:39.130801+0000","last_unstale":"2024-09-20T22:45:23.162690+0000","last_undegraded":"2024-09-20T22:45:23.162690+0000","last_fullsized":"2024-09-20T22:45:23.162690+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":5,"last_scrub":"147'126","last_scrub_stamp":"2024-09-20T22:43:48.215007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:43:48.215007+0000","objects_scrubbed":3,"log_size":198,"log_dups_size":0,"ondisk_log_size":198,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.001132751,"stat_sum":{"num_bytes":16362509,"num_objects":6,"num_object_clones":4,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":50,"num_read_kb":56208,"num_write":39,"num_write_kb":17623,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"42","length":"1"},{"start":"44","length":"1"},{"start":"4c","length":"1"},{"start":"4e","length":"1"}]},{"pgid":"1.11","version":"77'128","reported_seq":562,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.148916+0000","last_change":"2024-09-20T22:43:30.651129+0000","last_active":"2024-09-20T22:45:23.148916+0000","last_peered":"2024-09-20T22:45:23.148916+0000","last_clean":"2024-09-20T22:45:23.148916+0000","last_became_active":"2024-09-20T22:43:24.128884+0000","last_became_peered":"2024-09-20T22:43:24.128884+0000","last_unstale":"2024-09-20T22:45:23.148916+0000","last_undegraded":"2024-09-20T22:45:23.148916+0000","last_fullsized":"2024-09-20T22:45:23.148916+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:43:30.651054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:43:30.651054+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.14","version":"74'10","reported_seq":385,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.161637+0000","last_change":"2024-09-20T22:43:11.355914+0000","last_active":"2024-09-20T22:45:23.161637+0000","last_peered":"2024-09-20T22:45:23.161637+0000","last_clean":"2024-09-20T22:45:23.161637+0000","last_became_active":"2024-09-20T22:41:40.253614+0000","last_became_peered":"2024-09-20T22:41:40.253614+0000","last_unstale":"2024-09-20T22:45:23.161637+0000","last_undegraded":"2024-09-20T22:45:23.161637+0000","last_fullsized":"2024-09-20T22:45:23.161637+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-09-20T22:43:11.355829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:43:11.355829+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":380,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.160324+0000","last_change":"2024-09-20T22:43:57.425304+0000","last_active":"2024-09-20T22:45:23.160324+0000","last_peered":"2024-09-20T22:45:23.160324+0000","last_clean":"2024-09-20T22:45:23.160324+0000","last_became_active":"2024-09-20T22:43:57.425004+0000","last_became_peered":"2024-09-20T22:43:57.425004+0000","last_unstale":"2024-09-20T22:45:23.160324+0000","last_undegraded":"2024-09-20T22:45:23.160324+0000","last_fullsized":"2024-09-20T22:45:23.160324+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:51.130212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:43:51.130212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"74'8","reported_seq":431,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.474652+0000","last_change":"2024-09-20T22:45:19.709069+0000","last_active":"2024-09-20T22:45:23.474652+0000","last_peered":"2024-09-20T22:45:23.474652+0000","last_clean":"2024-09-20T22:45:23.474652+0000","last_became_active":"2024-09-20T22:41:42.189451+0000","last_became_peered":"2024-09-20T22:41:42.189451+0000","last_unstale":"2024-09-20T22:45:23.474652+0000","last_undegraded":"2024-09-20T22:45:23.474652+0000","last_fullsized":"2024-09-20T22:45:23.474652+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-09-20T22:45:19.708971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:45:19.708971+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:46:22.586175+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":370,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.151147+0000","last_change":"2024-09-20T22:45:00.730113+0000","last_active":"2024-09-20T22:45:23.151147+0000","last_peered":"2024-09-20T22:45:23.151147+0000","last_clean":"2024-09-20T22:45:23.151147+0000","last_became_active":"2024-09-20T22:41:44.191206+0000","last_became_peered":"2024-09-20T22:41:44.191206+0000","last_unstale":"2024-09-20T22:45:23.151147+0000","last_undegraded":"2024-09-20T22:45:23.151147+0000","last_fullsized":"2024-09-20T22:45:23.151147+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:45:00.729995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:45:00.729995+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:46:05.652294+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.10","version":"240'63","reported_seq":810,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.169622+0000","last_change":"2024-09-20T22:45:23.169843+0000","last_active":"2024-09-20T22:45:23.169622+0000","last_peered":"2024-09-20T22:45:23.169622+0000","last_clean":"2024-09-20T22:45:23.169622+0000","last_became_active":"2024-09-20T22:43:39.130391+0000","last_became_peered":"2024-09-20T22:43:39.130391+0000","last_unstale":"2024-09-20T22:45:23.169622+0000","last_undegraded":"2024-09-20T22:45:23.169622+0000","last_fullsized":"2024-09-20T22:45:23.169622+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":5,"last_scrub":"146'43","last_scrub_stamp":"2024-09-20T22:43:46.106059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:43:46.106059+0000","objects_scrubbed":0,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.007192576,"stat_sum":{"num_bytes":4655572,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":11,"num_read_kb":7491,"num_write":8,"num_write_kb":3334,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"42","length":"1"},{"start":"44","length":"1"},{"start":"4c","length":"1"},{"start":"4e","length":"1"}]},{"pgid":"4.13","version":"74'11","reported_seq":234,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.156659+0000","last_change":"2024-09-20T22:43:57.446108+0000","last_active":"2024-09-20T22:45:23.156659+0000","last_peered":"2024-09-20T22:45:23.156659+0000","last_clean":"2024-09-20T22:45:23.156659+0000","last_became_active":"2024-09-20T22:43:57.416358+0000","last_became_peered":"2024-09-20T22:43:57.416358+0000","last_unstale":"2024-09-20T22:45:23.156659+0000","last_undegraded":"2024-09-20T22:45:23.156659+0000","last_fullsized":"2024-09-20T22:45:23.156659+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-09-20T22:43:33.923283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:43:33.923283+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":184,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.161316+0000","last_change":"2024-09-20T22:43:57.423148+0000","last_active":"2024-09-20T22:45:23.161316+0000","last_peered":"2024-09-20T22:45:23.161316+0000","last_clean":"2024-09-20T22:45:23.161316+0000","last_became_active":"2024-09-20T22:43:57.422556+0000","last_became_peered":"2024-09-20T22:43:57.422556+0000","last_unstale":"2024-09-20T22:45:23.161316+0000","last_undegraded":"2024-09-20T22:45:23.161316+0000","last_fullsized":"2024-09-20T22:45:23.161316+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:55.053048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:43:55.053048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":366,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.149344+0000","last_change":"2024-09-20T22:44:20.088194+0000","last_active":"2024-09-20T22:45:23.149344+0000","last_peered":"2024-09-20T22:45:23.149344+0000","last_clean":"2024-09-20T22:45:23.149344+0000","last_became_active":"2024-09-20T22:41:42.181736+0000","last_became_peered":"2024-09-20T22:41:42.181736+0000","last_unstale":"2024-09-20T22:45:23.149344+0000","last_undegraded":"2024-09-20T22:45:23.149344+0000","last_fullsized":"2024-09-20T22:45:23.149344+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:44:20.088125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:44:20.088125+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:33.759142+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":362,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.161144+0000","last_change":"2024-09-20T22:43:53.201536+0000","last_active":"2024-09-20T22:45:23.161144+0000","last_peered":"2024-09-20T22:45:23.161144+0000","last_clean":"2024-09-20T22:45:23.161144+0000","last_became_active":"2024-09-20T22:41:44.192960+0000","last_became_peered":"2024-09-20T22:41:44.192960+0000","last_unstale":"2024-09-20T22:45:23.161144+0000","last_undegraded":"2024-09-20T22:45:23.161144+0000","last_fullsized":"2024-09-20T22:45:23.161144+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:53.201465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:43:53.201465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"135'39","reported_seq":539,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.160916+0000","last_change":"2024-09-20T22:45:23.161988+0000","last_active":"2024-09-20T22:45:23.160916+0000","last_peered":"2024-09-20T22:45:23.160916+0000","last_clean":"2024-09-20T22:45:23.160916+0000","last_became_active":"2024-09-20T22:43:39.130210+0000","last_became_peered":"2024-09-20T22:43:39.130210+0000","last_unstale":"2024-09-20T22:45:23.160916+0000","last_undegraded":"2024-09-20T22:45:23.160916+0000","last_fullsized":"2024-09-20T22:45:23.160916+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":5,"last_scrub":"135'39","last_scrub_stamp":"2024-09-20T22:43:46.228343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:43:46.228343+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0014330219999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"42","length":"1"},{"start":"44","length":"1"},{"start":"4c","length":"1"},{"start":"4e","length":"1"}]},{"pgid":"4.12","version":"74'9","reported_seq":394,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.149506+0000","last_change":"2024-09-20T22:44:54.989495+0000","last_active":"2024-09-20T22:45:23.149506+0000","last_peered":"2024-09-20T22:45:23.149506+0000","last_clean":"2024-09-20T22:45:23.149506+0000","last_became_active":"2024-09-20T22:41:40.164796+0000","last_became_peered":"2024-09-20T22:41:40.164796+0000","last_unstale":"2024-09-20T22:45:23.149506+0000","last_undegraded":"2024-09-20T22:45:23.149506+0000","last_fullsized":"2024-09-20T22:45:23.149506+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-09-20T22:44:54.989403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:44:54.989403+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:46:19.125237+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":383,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.162319+0000","last_change":"2024-09-20T22:43:57.429374+0000","last_active":"2024-09-20T22:45:23.162319+0000","last_peered":"2024-09-20T22:45:23.162319+0000","last_clean":"2024-09-20T22:45:23.162319+0000","last_became_active":"2024-09-20T22:43:57.429046+0000","last_became_peered":"2024-09-20T22:43:57.429046+0000","last_unstale":"2024-09-20T22:45:23.162319+0000","last_undegraded":"2024-09-20T22:45:23.162319+0000","last_fullsized":"2024-09-20T22:45:23.162319+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:31.138067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:31.138067+0000","last_clean_scrub_stamp":"2024-09-20T22:43:31.138067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":366,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.160386+0000","last_change":"2024-09-20T22:44:15.242983+0000","last_active":"2024-09-20T22:45:23.160386+0000","last_peered":"2024-09-20T22:45:23.160386+0000","last_clean":"2024-09-20T22:45:23.160386+0000","last_became_active":"2024-09-20T22:41:42.182966+0000","last_became_peered":"2024-09-20T22:41:42.182966+0000","last_unstale":"2024-09-20T22:45:23.160386+0000","last_undegraded":"2024-09-20T22:45:23.160386+0000","last_fullsized":"2024-09-20T22:45:23.160386+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:44:15.242881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:44:15.242881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":370,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.147507+0000","last_change":"2024-09-20T22:44:39.660113+0000","last_active":"2024-09-20T22:45:23.147507+0000","last_peered":"2024-09-20T22:45:23.147507+0000","last_clean":"2024-09-20T22:45:23.147507+0000","last_became_active":"2024-09-20T22:41:44.194551+0000","last_became_peered":"2024-09-20T22:41:44.194551+0000","last_unstale":"2024-09-20T22:45:23.147507+0000","last_undegraded":"2024-09-20T22:45:23.147507+0000","last_fullsized":"2024-09-20T22:45:23.147507+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:44:39.660039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:44:39.660039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:50.176169+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.12","version":"239'227","reported_seq":741,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.156535+0000","last_change":"2024-09-20T22:45:23.156935+0000","last_active":"2024-09-20T22:45:23.156535+0000","last_peered":"2024-09-20T22:45:23.156535+0000","last_clean":"2024-09-20T22:45:23.156535+0000","last_became_active":"2024-09-20T22:43:39.135211+0000","last_became_peered":"2024-09-20T22:43:39.135211+0000","last_unstale":"2024-09-20T22:45:23.156535+0000","last_undegraded":"2024-09-20T22:45:23.156535+0000","last_fullsized":"2024-09-20T22:45:23.156535+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":5,"last_scrub":"235'221","last_scrub_stamp":"2024-09-20T22:45:15.702470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:45:15.702470+0000","objects_scrubbed":5,"log_size":227,"log_dups_size":0,"ondisk_log_size":227,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:46:45.294346+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00086531499999999996,"stat_sum":{"num_bytes":9723992,"num_objects":6,"num_object_clones":4,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":63,"num_read_kb":72904,"num_write":57,"num_write_kb":17477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"42","length":"1"},{"start":"44","length":"1"},{"start":"4c","length":"1"},{"start":"4e","length":"1"}]},{"pgid":"1.14","version":"77'128","reported_seq":642,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.178163+0000","last_change":"2024-09-20T22:45:17.527820+0000","last_active":"2024-09-20T22:45:23.178163+0000","last_peered":"2024-09-20T22:45:23.178163+0000","last_clean":"2024-09-20T22:45:23.178163+0000","last_became_active":"2024-09-20T22:44:00.145687+0000","last_became_peered":"2024-09-20T22:44:00.145687+0000","last_unstale":"2024-09-20T22:45:23.178163+0000","last_undegraded":"2024-09-20T22:45:23.178163+0000","last_fullsized":"2024-09-20T22:45:23.178163+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":119,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":5,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:45:17.527705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:45:17.527705+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:46:44.222620+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.11","version":"74'11","reported_seq":179,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.473475+0000","last_change":"2024-09-20T22:44:14.705392+0000","last_active":"2024-09-20T22:45:23.473475+0000","last_peered":"2024-09-20T22:45:23.473475+0000","last_clean":"2024-09-20T22:45:23.473475+0000","last_became_active":"2024-09-20T22:44:09.170785+0000","last_became_peered":"2024-09-20T22:44:09.170785+0000","last_unstale":"2024-09-20T22:45:23.473475+0000","last_undegraded":"2024-09-20T22:45:23.473475+0000","last_fullsized":"2024-09-20T22:45:23.473475+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-09-20T22:44:14.705306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:44:14.705306+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:35.089571+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":374,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.474933+0000","last_change":"2024-09-20T22:43:06.775869+0000","last_active":"2024-09-20T22:45:23.474933+0000","last_peered":"2024-09-20T22:45:23.474933+0000","last_clean":"2024-09-20T22:45:23.474933+0000","last_became_active":"2024-09-20T22:41:38.158701+0000","last_became_peered":"2024-09-20T22:41:38.158701+0000","last_unstale":"2024-09-20T22:45:23.474933+0000","last_undegraded":"2024-09-20T22:45:23.474933+0000","last_fullsized":"2024-09-20T22:45:23.474933+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:06.775781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:06.775781+0000","last_clean_scrub_stamp":"2024-09-20T22:43:06.775781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":372,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.160376+0000","last_change":"2024-09-20T22:43:57.419543+0000","last_active":"2024-09-20T22:45:23.160376+0000","last_peered":"2024-09-20T22:45:23.160376+0000","last_clean":"2024-09-20T22:45:23.160376+0000","last_became_active":"2024-09-20T22:43:57.419200+0000","last_became_peered":"2024-09-20T22:43:57.419200+0000","last_unstale":"2024-09-20T22:45:23.160376+0000","last_undegraded":"2024-09-20T22:45:23.160376+0000","last_fullsized":"2024-09-20T22:45:23.160376+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:50.174861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:50.174861+0000","last_clean_scrub_stamp":"2024-09-20T22:43:50.174861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":362,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.160486+0000","last_change":"2024-09-20T22:45:00.006980+0000","last_active":"2024-09-20T22:45:23.160486+0000","last_peered":"2024-09-20T22:45:23.160486+0000","last_clean":"2024-09-20T22:45:23.160486+0000","last_became_active":"2024-09-20T22:41:44.831967+0000","last_became_peered":"2024-09-20T22:41:44.831967+0000","last_unstale":"2024-09-20T22:45:23.160486+0000","last_undegraded":"2024-09-20T22:45:23.160486+0000","last_fullsized":"2024-09-20T22:45:23.160486+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:45:00.006909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:45:00.006909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:46:17.412244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"224'20","reported_seq":578,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.149367+0000","last_change":"2024-09-20T22:45:23.149553+0000","last_active":"2024-09-20T22:45:23.149367+0000","last_peered":"2024-09-20T22:45:23.149367+0000","last_clean":"2024-09-20T22:45:23.149367+0000","last_became_active":"2024-09-20T22:44:09.177614+0000","last_became_peered":"2024-09-20T22:44:09.177614+0000","last_unstale":"2024-09-20T22:45:23.149367+0000","last_undegraded":"2024-09-20T22:45:23.149367+0000","last_fullsized":"2024-09-20T22:45:23.149367+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":5,"last_scrub":"147'18","last_scrub_stamp":"2024-09-20T22:43:48.759471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:43:48.759471+0000","objects_scrubbed":1,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00072518799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":2020747,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"42","length":"1"},{"start":"44","length":"1"},{"start":"4c","length":"1"},{"start":"4e","length":"1"}]},{"pgid":"4.10","version":"74'4","reported_seq":376,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.162999+0000","last_change":"2024-09-20T22:44:17.320732+0000","last_active":"2024-09-20T22:45:23.162999+0000","last_peered":"2024-09-20T22:45:23.162999+0000","last_clean":"2024-09-20T22:45:23.162999+0000","last_became_active":"2024-09-20T22:41:40.166123+0000","last_became_peered":"2024-09-20T22:41:40.166123+0000","last_unstale":"2024-09-20T22:45:23.162999+0000","last_undegraded":"2024-09-20T22:45:23.162999+0000","last_fullsized":"2024-09-20T22:45:23.162999+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'4","last_scrub_stamp":"2024-09-20T22:44:17.320608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:44:17.320608+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":382,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.151021+0000","last_change":"2024-09-20T22:44:58.632805+0000","last_active":"2024-09-20T22:45:23.151021+0000","last_peered":"2024-09-20T22:45:23.151021+0000","last_clean":"2024-09-20T22:45:23.151021+0000","last_became_active":"2024-09-20T22:41:38.158347+0000","last_became_peered":"2024-09-20T22:41:38.158347+0000","last_unstale":"2024-09-20T22:45:23.151021+0000","last_undegraded":"2024-09-20T22:45:23.151021+0000","last_fullsized":"2024-09-20T22:45:23.151021+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:44:58.632703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:44:58.632703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:46:09.773272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":380,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.156060+0000","last_change":"2024-09-20T22:45:00.680376+0000","last_active":"2024-09-20T22:45:23.156060+0000","last_peered":"2024-09-20T22:45:23.156060+0000","last_clean":"2024-09-20T22:45:23.156060+0000","last_became_active":"2024-09-20T22:43:57.419908+0000","last_became_peered":"2024-09-20T22:43:57.419908+0000","last_unstale":"2024-09-20T22:45:23.156060+0000","last_undegraded":"2024-09-20T22:45:23.156060+0000","last_fullsized":"2024-09-20T22:45:23.156060+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:45:00.680307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:45:00.680307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:46:24.048175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":376,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.160000+0000","last_change":"2024-09-20T22:45:19.352251+0000","last_active":"2024-09-20T22:45:23.160000+0000","last_peered":"2024-09-20T22:45:23.160000+0000","last_clean":"2024-09-20T22:45:23.160000+0000","last_became_active":"2024-09-20T22:43:57.424654+0000","last_became_peered":"2024-09-20T22:43:57.424654+0000","last_unstale":"2024-09-20T22:45:23.160000+0000","last_undegraded":"2024-09-20T22:45:23.160000+0000","last_fullsized":"2024-09-20T22:45:23.160000+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:45:19.352162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:10.233487+0000","last_clean_scrub_stamp":"2024-09-20T22:45:19.352162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:46:48.139048+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":368,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.149892+0000","last_change":"2024-09-20T22:44:21.123125+0000","last_active":"2024-09-20T22:45:23.149892+0000","last_peered":"2024-09-20T22:45:23.149892+0000","last_clean":"2024-09-20T22:45:23.149892+0000","last_became_active":"2024-09-20T22:43:57.430933+0000","last_became_peered":"2024-09-20T22:43:57.430933+0000","last_unstale":"2024-09-20T22:45:23.149892+0000","last_undegraded":"2024-09-20T22:45:23.149892+0000","last_fullsized":"2024-09-20T22:45:23.149892+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:44:21.123026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:44:21.123026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:36.736823+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":388,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.163681+0000","last_change":"2024-09-20T22:44:59.221701+0000","last_active":"2024-09-20T22:45:23.163681+0000","last_peered":"2024-09-20T22:45:23.163681+0000","last_clean":"2024-09-20T22:45:23.163681+0000","last_became_active":"2024-09-20T22:44:59.221296+0000","last_became_peered":"2024-09-20T22:44:59.221296+0000","last_unstale":"2024-09-20T22:45:23.163681+0000","last_undegraded":"2024-09-20T22:45:23.163681+0000","last_fullsized":"2024-09-20T22:45:23.163681+0000","mapping_epoch":219,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:44:10.267152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:44:10.267152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:45:33.915070+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"74'11","reported_seq":400,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.158068+0000","last_change":"2024-09-20T22:45:14.702296+0000","last_active":"2024-09-20T22:45:23.158068+0000","last_peered":"2024-09-20T22:45:23.158068+0000","last_clean":"2024-09-20T22:45:23.158068+0000","last_became_active":"2024-09-20T22:41:40.171256+0000","last_became_peered":"2024-09-20T22:41:40.171256+0000","last_unstale":"2024-09-20T22:45:23.158068+0000","last_undegraded":"2024-09-20T22:45:23.158068+0000","last_fullsized":"2024-09-20T22:45:23.158068+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-09-20T22:45:14.702198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:45:14.702198+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:46:29.443493+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":366,"reported_epoch":244,"state":"active+clean","last_fresh":"2024-09-20T22:45:23.151298+0000","last_change":"2024-09-20T22:43:51.667414+0000","last_active":"2024-09-20T22:45:23.151298+0000","last_peered":"2024-09-20T22:45:23.151298+0000","last_clean":"2024-09-20T22:45:23.151298+0000","last_became_active":"2024-09-20T22:41:42.178157+0000","last_became_peered":"2024-09-20T22:41:42.178157+0000","last_unstale":"2024-09-20T22:45:23.151298+0000","last_undegraded":"2024-09-20T22:45:23.151298+0000","last_fullsized":"2024-09-20T22:45:23.151298+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:51.667341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:51.667341+0000","last_clean_scrub_stamp":"2024-09-20T22:43:51.667341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":26,"stat_sum":{"num_bytes":174291520,"num_objects":90,"num_object_clones":48,"num_object_copies":270,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":90,"num_whiteouts":11,"num_read":1269,"num_read_kb":1318196,"num_write":883,"num_write_kb":327687,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":5835903,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":412327936,"data_stored":528583676,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2751,"ondisk_log_size":2751,"up":78,"acting":78,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1261,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":69,"ondisk_log_size":69,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":686,"num_read_kb":443,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":98,"num_bytes_recovered":1914,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":471040,"data_stored":12580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":427,"ondisk_log_size":427,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":5023,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":220,"num_read_kb":215,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":21,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2688,"ondisk_log_size":2688,"up":63,"acting":63,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070513,"num_pgs":78,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":105068,"kb_used_data":65552,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93640084,"statfs":{"total":95995035648,"available":95887446016,"internally_reserved":0,"allocated":67125248,"data_stored":78685705,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":40434122},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430263,"num_pgs":75,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":106244,"kb_used_data":66728,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93638908,"statfs":{"total":95995035648,"available":95886241792,"internally_reserved":0,"allocated":68329472,"data_stored":87815105,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691900,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":85776,"kb_used_data":54588,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93659376,"statfs":{"total":95995035648,"available":95907201024,"internally_reserved":0,"allocated":55898112,"data_stored":75975214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":0,"seq":133143986241,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182471,"num_pgs":83,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":102728,"kb_used_data":67376,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93642424,"statfs":{"total":95995035648,"available":95889842176,"internally_reserved":0,"allocated":68993024,"data_stored":85384646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411404,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":94824,"kb_used_data":59472,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93650328,"statfs":{"total":95995035648,"available":95897935872,"internally_reserved":0,"allocated":60899328,"data_stored":76568457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574929,"num_pgs":79,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":75692,"kb_used_data":40344,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93669460,"statfs":{"total":95995035648,"available":95917527040,"internally_reserved":0,"allocated":41312256,"data_stored":50628232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738454,"num_pgs":78,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":92764,"kb_used_data":53248,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93652388,"statfs":{"total":95995035648,"available":95900045312,"internally_reserved":0,"allocated":54525952,"data_stored":64722236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53071872,"data_stored":64808968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":40153088,"data_stored":51042371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":57925632,"data_stored":74807265,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":65679360,"data_stored":85353021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11284480,"data_stored":11212109,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":52932608,"data_stored":74220755,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":66875392,"data_stored":89952106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":64405504,"data_stored":77187081,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-20T22:45:26.443 INFO:tasks.cephadm.ceph_manager.ceph:decrease pool size by 10 2024-09-20T22:45:26.443 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd pool set .mgr pg_num 11 2024-09-20T22:45:26.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:26 smithi154 ceph-mon[33264]: 3.16 deep-scrub starts 2024-09-20T22:45:26.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:26 smithi154 ceph-mon[33264]: 3.16 deep-scrub ok 2024-09-20T22:45:26.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:26 smithi154 ceph-mon[33264]: 6.1f deep-scrub starts 2024-09-20T22:45:26.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:26 smithi154 ceph-mon[33264]: 6.1f deep-scrub ok 2024-09-20T22:45:26.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:26 smithi154 ceph-mon[33264]: osdmap e246: 8 total, 8 up, 7 in 2024-09-20T22:45:26.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:26 smithi154 ceph-mon[33264]: 5.a scrub starts 2024-09-20T22:45:26.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:26 smithi154 ceph-mon[33264]: 5.a scrub ok 2024-09-20T22:45:26.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:26 smithi154 ceph-mon[33264]: 1.7 scrub starts 2024-09-20T22:45:26.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:26 smithi154 ceph-mon[33264]: 1.7 scrub ok 2024-09-20T22:45:26.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:26 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/2338281916' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-20T22:45:26.530 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:45:26.555 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:26.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:26.556 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:45:26.656 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:45:26.677 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:26.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:26.678 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:45:26.778 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:45:26.805 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:26.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:26.806 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:45:26.907 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:45:26.932 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:26.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:26.933 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:45:27.034 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:45:27.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:27.059 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:27.059 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:45:27.160 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:45:27.185 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:27.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:27.186 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:45:27.259 INFO:tasks.rados.rados.0.smithi084.stdout:812: done (1 left) 2024-09-20T22:45:27.259 INFO:tasks.rados.rados.0.smithi084.stdout:813: read oid 24 snap -1 2024-09-20T22:45:27.259 INFO:tasks.rados.rados.0.smithi084.stdout:813: expect deleted 2024-09-20T22:45:27.259 INFO:tasks.rados.rados.0.smithi084.stdout:814: snap_create 2024-09-20T22:45:27.286 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:45:27.314 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:27.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:27.315 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:45:27.316 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:45:27.330 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:45:27.340 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:45:27.416 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:45:27.441 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:27 smithi084 ceph-mon[40024]: pgmap v271: 178 pgs: 2 active+clean+snaptrim, 176 active+clean; 167 MiB data, 648 MiB used, 625 GiB / 626 GiB avail; 24 MiB/s rd, 4.2 MiB/s wr, 39 op/s 2024-09-20T22:45:27.441 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:27 smithi084 ceph-mon[40024]: from='client.15285 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T22:45:27.441 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:27 smithi084 ceph-mon[40024]: 3.6 scrub starts 2024-09-20T22:45:27.441 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:27 smithi084 ceph-mon[40024]: 3.6 scrub ok 2024-09-20T22:45:27.441 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:27 smithi084 ceph-mon[40024]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:45:27.441 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:27 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/2338281916' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T22:45:27.442 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:27 smithi084 ceph-mon[40024]: osdmap e247: 8 total, 8 up, 7 in 2024-09-20T22:45:27.442 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:27 smithi084 ceph-mon[40024]: 1.5 scrub starts 2024-09-20T22:45:27.442 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:27 smithi084 ceph-mon[29774]: pgmap v271: 178 pgs: 2 active+clean+snaptrim, 176 active+clean; 167 MiB data, 648 MiB used, 625 GiB / 626 GiB avail; 24 MiB/s rd, 4.2 MiB/s wr, 39 op/s 2024-09-20T22:45:27.442 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:27 smithi084 ceph-mon[29774]: from='client.15285 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T22:45:27.442 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:27 smithi084 ceph-mon[29774]: 3.6 scrub starts 2024-09-20T22:45:27.442 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:27 smithi084 ceph-mon[29774]: 3.6 scrub ok 2024-09-20T22:45:27.443 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:27 smithi084 ceph-mon[29774]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:45:27.443 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:27 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/2338281916' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T22:45:27.443 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:27 smithi084 ceph-mon[29774]: osdmap e247: 8 total, 8 up, 7 in 2024-09-20T22:45:27.443 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:27 smithi084 ceph-mon[29774]: 1.5 scrub starts 2024-09-20T22:45:27.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:27.443 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:27.443 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:45:27.544 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:45:27.565 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:27.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:27.566 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:45:27.667 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:45:27.674 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:27 smithi154 ceph-mon[33264]: pgmap v271: 178 pgs: 2 active+clean+snaptrim, 176 active+clean; 167 MiB data, 648 MiB used, 625 GiB / 626 GiB avail; 24 MiB/s rd, 4.2 MiB/s wr, 39 op/s 2024-09-20T22:45:27.674 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:27 smithi154 ceph-mon[33264]: from='client.15285 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T22:45:27.674 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:27 smithi154 ceph-mon[33264]: 3.6 scrub starts 2024-09-20T22:45:27.675 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:27 smithi154 ceph-mon[33264]: 3.6 scrub ok 2024-09-20T22:45:27.675 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:27 smithi154 ceph-mon[33264]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:45:27.675 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:27 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/2338281916' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T22:45:27.675 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:27 smithi154 ceph-mon[33264]: osdmap e247: 8 total, 8 up, 7 in 2024-09-20T22:45:27.675 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:27 smithi154 ceph-mon[33264]: 1.5 scrub starts 2024-09-20T22:45:27.687 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:27.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:27.688 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:45:27.788 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:45:27.810 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:27.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:27.811 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:45:27.875 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:45:27.875 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:45:27.889 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:45:27.890 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:45:27.907 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:45:27.907 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:45:27.911 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:45:27.923 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-20T22:45:27.923 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T22:45:27.932 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:27.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:27.933 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:45:27.940 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:45:27.941 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:45:27.958 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:45:27.959 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:45:27.975 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:45:27.975 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:45:27.991 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:45:27.992 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:45:28.034 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:45:28.055 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:28.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:28.056 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:45:28.154 INFO:tasks.rados.rados.0.smithi084.stdout:811: done (2 left) 2024-09-20T22:45:28.154 INFO:tasks.rados.rados.0.smithi084.stdout:813: done (1 left) 2024-09-20T22:45:28.155 INFO:tasks.rados.rados.0.smithi084.stdout:814: done (0 left) 2024-09-20T22:45:28.155 INFO:tasks.rados.rados.0.smithi084.stdout:815: rollback oid 25 current snap is 80 2024-09-20T22:45:28.155 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 25 to 73 2024-09-20T22:45:28.155 INFO:tasks.rados.rados.0.smithi084.stdout:816: rollback oid 19 current snap is 80 2024-09-20T22:45:28.155 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 19 to 69 2024-09-20T22:45:28.155 INFO:tasks.rados.rados.0.smithi084.stdout:817: read oid 39 snap 73 2024-09-20T22:45:28.155 INFO:tasks.rados.rados.0.smithi084.stdout:817: expect (ObjNum 245 snap 54 seq_num 245) 2024-09-20T22:45:28.155 INFO:tasks.rados.rados.0.smithi084.stdout:818: rollback oid 37 current snap is 80 2024-09-20T22:45:28.155 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 37 to 60 2024-09-20T22:45:28.155 INFO:tasks.rados.rados.0.smithi084.stdout:819: read oid 16 snap 72 2024-09-20T22:45:28.155 INFO:tasks.rados.rados.0.smithi084.stdout:819: expect (ObjNum 281 snap 71 seq_num 281) 2024-09-20T22:45:28.155 INFO:tasks.rados.rados.0.smithi084.stdout:820: snap_remove snap 75 2024-09-20T22:45:28.157 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:45:28.186 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:28.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:28.187 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:45:28.288 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:45:28.309 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:28.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:28.310 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:45:28.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:28 smithi084 ceph-mon[29774]: osdmap e248: 8 total, 8 up, 7 in 2024-09-20T22:45:28.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:28 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/1293145613' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:45:28.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:28 smithi084 ceph-mon[29774]: osdmap e249: 8 total, 8 up, 7 in 2024-09-20T22:45:28.365 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:45:28 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: ::ffff:172.21.15.154 - - [20/Sep/2024:22:45:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-20T22:45:28.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:28 smithi084 ceph-mon[40024]: osdmap e248: 8 total, 8 up, 7 in 2024-09-20T22:45:28.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:28 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/1293145613' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:45:28.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:28 smithi084 ceph-mon[40024]: osdmap e249: 8 total, 8 up, 7 in 2024-09-20T22:45:28.411 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:45:28.432 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:28.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:28.432 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:45:28.516 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-20T22:45:28.516 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2024-09-20T22:45:28.516 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2024-09-20T22:45:28.516 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [ 2024-09-20T22:45:28.516 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.517 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24652.0:336 7.7 e107/94 7:e67788e2:::smithi08477203-31:head v 107'45, mlcod=105'41)", 2024-09-20T22:45:28.517 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:43:06.546796+0000", 2024-09-20T22:45:28.517 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 141.96647568399999, 2024-09-20T22:45:28.517 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.015311521999999999, 2024-09-20T22:45:28.517 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:45:28.517 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:45:28.517 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:45:28.517 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:45:28.517 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.517 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:45:28.517 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:06.546796+0000", 2024-09-20T22:45:28.517 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:45:28.518 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.518 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.518 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:45:28.518 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:06.546796+0000", 2024-09-20T22:45:28.518 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:45:28.518 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.518 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.518 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:45:28.518 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:06.546796+0000", 2024-09-20T22:45:28.518 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.84e-07 2024-09-20T22:45:28.518 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.518 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.518 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:45:28.518 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:06.554340+0000", 2024-09-20T22:45:28.519 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0075444680000000004 2024-09-20T22:45:28.519 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.519 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.519 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:45:28.519 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:06.554366+0000", 2024-09-20T22:45:28.519 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.5531e-05 2024-09-20T22:45:28.519 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.519 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.519 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:45:28.519 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:06.554388+0000", 2024-09-20T22:45:28.519 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.2263e-05 2024-09-20T22:45:28.519 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.519 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.519 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:45:28.520 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:06.554465+0000", 2024-09-20T22:45:28.520 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.7296e-05 2024-09-20T22:45:28.520 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.520 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.520 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:45:28.520 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:06.554494+0000", 2024-09-20T22:45:28.520 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.8092000000000001e-05 2024-09-20T22:45:28.520 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.520 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.520 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:45:28.520 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:06.562070+0000", 2024-09-20T22:45:28.520 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0075764730000000002 2024-09-20T22:45:28.520 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.520 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.520 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:45:28.520 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:06.562107+0000", 2024-09-20T22:45:28.521 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.7115e-05 2024-09-20T22:45:28.521 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:28.521 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:45:28.521 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:28.521 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.521 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.521 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:554 7.a 7:5122292a:::smithi08477203-33:head [copy-from ver 6] snapc 13=[13,12,11,10,f,d] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-09-20T22:45:28.521 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:43:27.336824+0000", 2024-09-20T22:45:28.521 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 121.176447251, 2024-09-20T22:45:28.521 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.056403397000000001, 2024-09-20T22:45:28.521 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:45:28.521 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:45:28.521 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:45:28.521 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:45:28.521 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:45:28.521 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:45:28.522 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 554 2024-09-20T22:45:28.522 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.522 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:45:28.522 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.522 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:45:28.522 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:27.336824+0000", 2024-09-20T22:45:28.522 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:45:28.522 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.522 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.522 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:45:28.522 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:27.336824+0000", 2024-09-20T22:45:28.522 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:45:28.522 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.522 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.523 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:45:28.523 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:27.336826+0000", 2024-09-20T22:45:28.523 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.0339999999999999e-06 2024-09-20T22:45:28.523 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.523 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.523 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:45:28.523 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:27.336837+0000", 2024-09-20T22:45:28.523 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1243000000000001e-05 2024-09-20T22:45:28.523 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.523 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.523 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:45:28.523 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:27.336857+0000", 2024-09-20T22:45:28.523 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.9868999999999999e-05 2024-09-20T22:45:28.523 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.524 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.524 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:45:28.524 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:27.336869+0000", 2024-09-20T22:45:28.524 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1348999999999999e-05 2024-09-20T22:45:28.524 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.524 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.524 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:45:28.524 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:27.336946+0000", 2024-09-20T22:45:28.524 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.7683999999999997e-05 2024-09-20T22:45:28.524 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.524 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.524 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:45:28.524 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:27.337045+0000", 2024-09-20T22:45:28.524 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.9265999999999995e-05 2024-09-20T22:45:28.525 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.525 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.525 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 2,5", 2024-09-20T22:45:28.525 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:27.340603+0000", 2024-09-20T22:45:28.525 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.003557892 2024-09-20T22:45:28.525 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.525 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.525 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:45:28.525 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:27.343571+0000", 2024-09-20T22:45:28.525 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0029676609999999999 2024-09-20T22:45:28.525 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.525 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.525 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:45:28.525 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:27.346457+0000", 2024-09-20T22:45:28.525 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0028860040000000002 2024-09-20T22:45:28.526 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.526 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.526 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:45:28.526 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:27.393131+0000", 2024-09-20T22:45:28.526 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.046673813000000001 2024-09-20T22:45:28.526 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.526 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.526 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:45:28.526 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:27.393173+0000", 2024-09-20T22:45:28.526 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.2076999999999997e-05 2024-09-20T22:45:28.526 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.526 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.526 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:45:28.526 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:27.393227+0000", 2024-09-20T22:45:28.527 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.4505000000000001e-05 2024-09-20T22:45:28.527 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:28.527 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:45:28.527 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:28.527 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.527 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.527 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:663 7.12 7:4fa4b2b3:::smithi08477203-21:head [write 665577~781581,stat] snapc 17=[17,16,15,11,d] ondisk+write+known_if_redirected+supports_pool_eio e137)", 2024-09-20T22:45:28.527 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:43:36.161568+0000", 2024-09-20T22:45:28.527 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 112.351703074, 2024-09-20T22:45:28.527 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.99286350899999998, 2024-09-20T22:45:28.527 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:45:28.527 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:45:28.527 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:45:28.528 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:45:28.528 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:45:28.528 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:45:28.528 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 663 2024-09-20T22:45:28.528 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.528 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:45:28.528 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.528 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:45:28.528 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.161568+0000", 2024-09-20T22:45:28.528 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:45:28.528 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.528 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.528 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:45:28.528 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.161568+0000", 2024-09-20T22:45:28.529 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:45:28.529 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.529 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.529 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:45:28.529 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.161570+0000", 2024-09-20T22:45:28.529 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.0679999999999999e-06 2024-09-20T22:45:28.529 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.529 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.529 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:45:28.529 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.162310+0000", 2024-09-20T22:45:28.529 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00074005999999999996 2024-09-20T22:45:28.529 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.529 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.530 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:45:28.530 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.162349+0000", 2024-09-20T22:45:28.530 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.8778e-05 2024-09-20T22:45:28.530 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.530 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.530 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:45:28.531 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.162375+0000", 2024-09-20T22:45:28.531 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.6089e-05 2024-09-20T22:45:28.531 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.531 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.531 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:45:28.531 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.162434+0000", 2024-09-20T22:45:28.531 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.8764999999999997e-05 2024-09-20T22:45:28.531 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.531 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.531 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-09-20T22:45:28.532 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.162443+0000", 2024-09-20T22:45:28.532 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.054e-06 2024-09-20T22:45:28.532 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.532 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.532 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:45:28.532 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.143416+0000", 2024-09-20T22:45:28.532 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.98097291600000003 2024-09-20T22:45:28.532 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.532 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.532 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:45:28.532 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.143522+0000", 2024-09-20T22:45:28.532 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00010626 2024-09-20T22:45:28.532 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.532 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.533 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 1,2", 2024-09-20T22:45:28.533 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.143654+0000", 2024-09-20T22:45:28.533 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013218799999999999 2024-09-20T22:45:28.533 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.533 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.533 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:45:28.533 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.148086+0000", 2024-09-20T22:45:28.533 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0044314189999999998 2024-09-20T22:45:28.533 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.533 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.533 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:45:28.533 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.153546+0000", 2024-09-20T22:45:28.533 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0054599920000000003 2024-09-20T22:45:28.534 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.534 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.534 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:45:28.534 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.154317+0000", 2024-09-20T22:45:28.534 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00077125999999999996 2024-09-20T22:45:28.534 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.534 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.534 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:45:28.534 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.154380+0000", 2024-09-20T22:45:28.534 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.2526000000000004e-05 2024-09-20T22:45:28.534 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.534 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.534 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:45:28.535 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.154432+0000", 2024-09-20T22:45:28.535 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.2133999999999999e-05 2024-09-20T22:45:28.535 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:28.535 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:45:28.535 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:28.535 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.535 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.535 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:664 7.12 7:4fa4b2b3:::smithi08477203-21:head [write 2062560~563688] snapc 17=[17,16,15,11,d] ondisk+write+known_if_redirected+supports_pool_eio e137)", 2024-09-20T22:45:28.535 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:43:36.167947+0000", 2024-09-20T22:45:28.535 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 112.345324604, 2024-09-20T22:45:28.535 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.98665172599999995, 2024-09-20T22:45:28.535 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:45:28.535 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:45:28.535 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:45:28.536 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:45:28.536 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:45:28.536 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:45:28.536 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 664 2024-09-20T22:45:28.536 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.536 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:45:28.536 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.536 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:45:28.536 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.167947+0000", 2024-09-20T22:45:28.536 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:45:28.536 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.536 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.536 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:45:28.536 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.167947+0000", 2024-09-20T22:45:28.536 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:45:28.537 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.537 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.537 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:45:28.537 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.167948+0000", 2024-09-20T22:45:28.537 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.187e-06 2024-09-20T22:45:28.537 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.537 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.537 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:45:28.537 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.168193+0000", 2024-09-20T22:45:28.537 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00024496199999999999 2024-09-20T22:45:28.537 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.537 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.537 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:45:28.538 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.168203+0000", 2024-09-20T22:45:28.538 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0543999999999999e-05 2024-09-20T22:45:28.538 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.538 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.538 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:45:28.538 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.168209+0000", 2024-09-20T22:45:28.538 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.2839999999999997e-06 2024-09-20T22:45:28.538 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.538 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.538 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:45:28.538 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.168252+0000", 2024-09-20T22:45:28.538 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.3115000000000003e-05 2024-09-20T22:45:28.538 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.538 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.539 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-09-20T22:45:28.539 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.168257+0000", 2024-09-20T22:45:28.539 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.6860000000000003e-06 2024-09-20T22:45:28.539 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.539 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.539 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:45:28.539 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.144795+0000", 2024-09-20T22:45:28.539 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.97653758099999999 2024-09-20T22:45:28.539 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.539 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.539 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:45:28.539 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.144865+0000", 2024-09-20T22:45:28.539 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.9641000000000006e-05 2024-09-20T22:45:28.539 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.539 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.540 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 1,2", 2024-09-20T22:45:28.540 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.144986+0000", 2024-09-20T22:45:28.540 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000121248 2024-09-20T22:45:28.540 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.540 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.540 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:45:28.540 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.148091+0000", 2024-09-20T22:45:28.540 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0031050399999999999 2024-09-20T22:45:28.540 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.540 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.540 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:45:28.540 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.153603+0000", 2024-09-20T22:45:28.540 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0055118479999999997 2024-09-20T22:45:28.540 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.540 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.541 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:45:28.541 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.154508+0000", 2024-09-20T22:45:28.541 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00090530600000000004 2024-09-20T22:45:28.541 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.541 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.541 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:45:28.541 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.154560+0000", 2024-09-20T22:45:28.541 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.1474000000000002e-05 2024-09-20T22:45:28.541 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.541 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.541 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:45:28.541 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.154598+0000", 2024-09-20T22:45:28.541 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.8810000000000003e-05 2024-09-20T22:45:28.541 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:28.541 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:45:28.542 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:28.542 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.542 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.542 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:665 7.12 7:4fa4b2b3:::smithi08477203-21:head [write 3182305~643664] snapc 17=[17,16,15,11,d] ondisk+write+known_if_redirected+supports_pool_eio e137)", 2024-09-20T22:45:28.542 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:43:36.174520+0000", 2024-09-20T22:45:28.542 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 112.33875151399999, 2024-09-20T22:45:28.542 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.98020083400000002, 2024-09-20T22:45:28.542 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:45:28.542 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:45:28.542 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:45:28.542 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:45:28.542 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:45:28.542 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:45:28.543 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 665 2024-09-20T22:45:28.543 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.543 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:45:28.543 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.543 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:45:28.543 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174520+0000", 2024-09-20T22:45:28.543 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:45:28.543 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.543 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.543 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:45:28.543 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174520+0000", 2024-09-20T22:45:28.543 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:45:28.543 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.543 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.544 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:45:28.544 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174521+0000", 2024-09-20T22:45:28.544 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.234e-06 2024-09-20T22:45:28.544 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.544 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.544 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:45:28.544 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174816+0000", 2024-09-20T22:45:28.544 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000294682 2024-09-20T22:45:28.544 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.544 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.544 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:45:28.544 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174827+0000", 2024-09-20T22:45:28.544 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1025e-05 2024-09-20T22:45:28.545 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.545 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.545 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:45:28.545 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174832+0000", 2024-09-20T22:45:28.545 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.3990000000000003e-06 2024-09-20T22:45:28.545 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.545 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.545 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:45:28.545 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174868+0000", 2024-09-20T22:45:28.545 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.6225e-05 2024-09-20T22:45:28.545 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.545 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.545 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-09-20T22:45:28.546 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174873+0000", 2024-09-20T22:45:28.546 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.848e-06 2024-09-20T22:45:28.546 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.546 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.546 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:45:28.546 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.145844+0000", 2024-09-20T22:45:28.546 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.97097048900000005 2024-09-20T22:45:28.546 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.546 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.546 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:45:28.546 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.145917+0000", 2024-09-20T22:45:28.546 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.3248000000000001e-05 2024-09-20T22:45:28.546 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.546 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.546 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 1,2", 2024-09-20T22:45:28.547 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.146028+0000", 2024-09-20T22:45:28.547 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000110773 2024-09-20T22:45:28.547 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.547 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.547 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:45:28.547 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.148095+0000", 2024-09-20T22:45:28.547 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0020678179999999999 2024-09-20T22:45:28.547 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.547 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.547 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:45:28.547 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.153967+0000", 2024-09-20T22:45:28.547 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0058717350000000003 2024-09-20T22:45:28.547 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.547 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.547 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:45:28.548 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.154633+0000", 2024-09-20T22:45:28.548 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00066600600000000002 2024-09-20T22:45:28.548 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.548 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.548 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:45:28.548 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.154683+0000", 2024-09-20T22:45:28.548 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.9872e-05 2024-09-20T22:45:28.548 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.548 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.548 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:45:28.548 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.154721+0000", 2024-09-20T22:45:28.548 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.748e-05 2024-09-20T22:45:28.548 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:28.548 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:45:28.548 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:28.548 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.549 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.549 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:666 7.12 7:4fa4b2b3:::smithi08477203-21:head [setxattr _header (57) in=7b,truncate 3865577] snapc 17=[17,16,15,11,d] ondisk+write+known_if_redirected+supports_pool_eio e137)", 2024-09-20T22:45:28.549 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:43:36.174844+0000", 2024-09-20T22:45:28.549 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 112.338427315, 2024-09-20T22:45:28.549 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.98002428600000002, 2024-09-20T22:45:28.549 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:45:28.549 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:45:28.549 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:45:28.549 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:45:28.549 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:45:28.549 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:45:28.549 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 666 2024-09-20T22:45:28.549 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.549 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:45:28.550 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.550 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:45:28.550 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174844+0000", 2024-09-20T22:45:28.550 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:45:28.550 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.550 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.550 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:45:28.550 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174844+0000", 2024-09-20T22:45:28.550 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:45:28.550 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.550 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.550 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:45:28.550 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174845+0000", 2024-09-20T22:45:28.550 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.1999999999999999e-07 2024-09-20T22:45:28.551 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.551 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.551 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:45:28.551 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174853+0000", 2024-09-20T22:45:28.551 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.1950000000000003e-06 2024-09-20T22:45:28.551 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.551 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.551 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:45:28.551 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174858+0000", 2024-09-20T22:45:28.551 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.5650000000000001e-06 2024-09-20T22:45:28.551 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.551 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.551 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:45:28.551 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174863+0000", 2024-09-20T22:45:28.551 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.1649999999999997e-06 2024-09-20T22:45:28.552 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.552 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.552 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:45:28.552 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174915+0000", 2024-09-20T22:45:28.552 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.1045999999999999e-05 2024-09-20T22:45:28.552 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.552 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.552 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-09-20T22:45:28.552 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174921+0000", 2024-09-20T22:45:28.552 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.404e-06 2024-09-20T22:45:28.552 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.552 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.552 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:45:28.552 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.146528+0000", 2024-09-20T22:45:28.552 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.97160681199999999 2024-09-20T22:45:28.553 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.553 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.553 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:45:28.553 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.146881+0000", 2024-09-20T22:45:28.553 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00035348200000000001 2024-09-20T22:45:28.553 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.553 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.553 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 1,2", 2024-09-20T22:45:28.553 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.147011+0000", 2024-09-20T22:45:28.553 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013018900000000001 2024-09-20T22:45:28.553 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.553 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.553 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:45:28.553 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.148100+0000", 2024-09-20T22:45:28.553 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010881580000000001 2024-09-20T22:45:28.554 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.554 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.554 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:45:28.554 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.154278+0000", 2024-09-20T22:45:28.554 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0061781730000000003 2024-09-20T22:45:28.554 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.554 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.554 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:45:28.554 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.154764+0000", 2024-09-20T22:45:28.554 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00048589199999999998 2024-09-20T22:45:28.554 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.554 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.554 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:45:28.555 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.154815+0000", 2024-09-20T22:45:28.555 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.1428000000000003e-05 2024-09-20T22:45:28.555 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.555 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.555 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:45:28.555 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.154868+0000", 2024-09-20T22:45:28.555 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.3156999999999999e-05 2024-09-20T22:45:28.555 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:28.555 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:45:28.555 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:28.555 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.555 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.555 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:667 7.12 7:4fa4b2b3:::smithi08477203-21:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e137)", 2024-09-20T22:45:28.555 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:43:36.174872+0000", 2024-09-20T22:45:28.556 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 112.338399622, 2024-09-20T22:45:28.556 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.98024790299999998, 2024-09-20T22:45:28.556 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:45:28.556 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:45:28.556 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-20T22:45:28.556 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:45:28.556 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:45:28.556 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:45:28.556 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 667 2024-09-20T22:45:28.556 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.556 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:45:28.556 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.556 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:45:28.556 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174872+0000", 2024-09-20T22:45:28.557 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:45:28.557 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.557 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.557 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:45:28.557 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174872+0000", 2024-09-20T22:45:28.557 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:45:28.557 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.557 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.557 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:45:28.557 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174872+0000", 2024-09-20T22:45:28.557 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.2199999999999999e-07 2024-09-20T22:45:28.557 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.557 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.558 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:45:28.558 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174874+0000", 2024-09-20T22:45:28.558 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.187e-06 2024-09-20T22:45:28.558 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.558 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.558 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:45:28.558 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174881+0000", 2024-09-20T22:45:28.558 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.674e-06 2024-09-20T22:45:28.558 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.558 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.558 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:45:28.558 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174886+0000", 2024-09-20T22:45:28.558 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.2240000000000003e-06 2024-09-20T22:45:28.558 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.559 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.559 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:45:28.559 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174936+0000", 2024-09-20T22:45:28.559 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.9824999999999998e-05 2024-09-20T22:45:28.559 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.559 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.559 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-09-20T22:45:28.559 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174940+0000", 2024-09-20T22:45:28.559 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.5700000000000001e-06 2024-09-20T22:45:28.559 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.559 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.559 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:45:28.559 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.147523+0000", 2024-09-20T22:45:28.559 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.97258315399999995 2024-09-20T22:45:28.559 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.559 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.560 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for rw locks", 2024-09-20T22:45:28.560 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.147591+0000", 2024-09-20T22:45:28.560 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.7878000000000003e-05 2024-09-20T22:45:28.560 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.560 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.560 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:45:28.560 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.154925+0000", 2024-09-20T22:45:28.560 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0073344430000000004 2024-09-20T22:45:28.560 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.560 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.560 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:45:28.560 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.155020+0000", 2024-09-20T22:45:28.560 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.4767999999999998e-05 2024-09-20T22:45:28.560 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.560 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.560 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:45:28.561 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.155120+0000", 2024-09-20T22:45:28.561 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.9758000000000003e-05 2024-09-20T22:45:28.561 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:28.561 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:45:28.561 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:28.561 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.561 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.561 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:728 7.a 7:5122292a:::smithi08477203-33:head [delete] snapc 18=[17] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e143)", 2024-09-20T22:45:28.561 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:43:42.325019+0000", 2024-09-20T22:45:28.561 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 106.18825200400001, 2024-09-20T22:45:28.561 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.012596408999999999, 2024-09-20T22:45:28.561 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:45:28.561 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:45:28.561 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:45:28.562 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:45:28.562 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:45:28.562 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:45:28.562 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 728 2024-09-20T22:45:28.562 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.562 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:45:28.562 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.562 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:45:28.562 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:42.325019+0000", 2024-09-20T22:45:28.562 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:45:28.562 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.562 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.562 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:45:28.562 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:42.325019+0000", 2024-09-20T22:45:28.563 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:45:28.563 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.563 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.563 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:45:28.563 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:42.325021+0000", 2024-09-20T22:45:28.563 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.812e-06 2024-09-20T22:45:28.563 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.563 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.563 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:45:28.563 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:42.326485+0000", 2024-09-20T22:45:28.563 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0014643939999999999 2024-09-20T22:45:28.563 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.563 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.563 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:45:28.564 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:42.326507+0000", 2024-09-20T22:45:28.564 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.192e-05 2024-09-20T22:45:28.564 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.564 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.564 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:45:28.564 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:42.326517+0000", 2024-09-20T22:45:28.564 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.1400000000000006e-06 2024-09-20T22:45:28.564 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.564 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.564 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:45:28.564 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:42.326592+0000", 2024-09-20T22:45:28.564 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.5730999999999997e-05 2024-09-20T22:45:28.564 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.565 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.565 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op must wait for map", 2024-09-20T22:45:28.565 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:42.326599+0000", 2024-09-20T22:45:28.565 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.6800000000000004e-06 2024-09-20T22:45:28.565 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.565 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.565 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:45:28.565 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:42.334075+0000", 2024-09-20T22:45:28.565 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0074759780000000003 2024-09-20T22:45:28.565 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.565 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.565 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:45:28.565 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:42.334175+0000", 2024-09-20T22:45:28.565 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000100102 2024-09-20T22:45:28.566 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.566 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.566 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 2,5", 2024-09-20T22:45:28.566 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:42.334363+0000", 2024-09-20T22:45:28.566 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00018760700000000001 2024-09-20T22:45:28.566 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.566 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.566 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:45:28.566 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:42.336424+0000", 2024-09-20T22:45:28.566 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00206128 2024-09-20T22:45:28.566 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.566 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.566 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:45:28.566 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:42.336537+0000", 2024-09-20T22:45:28.566 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000113453 2024-09-20T22:45:28.567 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.567 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.567 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:45:28.567 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:42.337496+0000", 2024-09-20T22:45:28.567 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00095911200000000003 2024-09-20T22:45:28.567 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.567 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.567 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:45:28.567 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:42.337559+0000", 2024-09-20T22:45:28.567 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.2927000000000002e-05 2024-09-20T22:45:28.567 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.567 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.567 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:45:28.567 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:42.337616+0000", 2024-09-20T22:45:28.568 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.6273000000000001e-05 2024-09-20T22:45:28.568 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:28.568 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:45:28.568 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:28.568 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.568 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.568 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:894 7.a 7:50097c2d:::smithi08477203-23:head [copy-from ver 8] snapc 1e=[1c] ondisk+write+known_if_redirected+supports_pool_eio e155)", 2024-09-20T22:45:28.568 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:43:54.444916+0000", 2024-09-20T22:45:28.568 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 94.068354782, 2024-09-20T22:45:28.568 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.015193735, 2024-09-20T22:45:28.568 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:45:28.568 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:45:28.568 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:45:28.568 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:45:28.568 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:45:28.569 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:45:28.569 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 894 2024-09-20T22:45:28.569 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.569 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:45:28.569 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.569 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:45:28.569 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:54.444916+0000", 2024-09-20T22:45:28.569 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:45:28.569 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.569 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.569 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:45:28.569 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:54.444916+0000", 2024-09-20T22:45:28.569 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:45:28.570 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.570 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.570 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:45:28.570 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:54.444919+0000", 2024-09-20T22:45:28.570 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.1619999999999998e-06 2024-09-20T22:45:28.570 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.570 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.570 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:45:28.570 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:54.444943+0000", 2024-09-20T22:45:28.570 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.4128e-05 2024-09-20T22:45:28.570 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.570 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.570 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:45:28.570 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:54.444966+0000", 2024-09-20T22:45:28.570 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.3322000000000001e-05 2024-09-20T22:45:28.571 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.571 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.571 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:45:28.571 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:54.444975+0000", 2024-09-20T22:45:28.571 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.3489999999999998e-06 2024-09-20T22:45:28.571 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.571 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.571 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:45:28.571 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:54.445044+0000", 2024-09-20T22:45:28.571 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.8158999999999996e-05 2024-09-20T22:45:28.571 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.571 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.571 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:45:28.571 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:54.445116+0000", 2024-09-20T22:45:28.572 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.2557000000000003e-05 2024-09-20T22:45:28.572 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.572 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.572 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 2,5", 2024-09-20T22:45:28.572 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:54.449676+0000", 2024-09-20T22:45:28.572 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0045598569999999996 2024-09-20T22:45:28.572 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.572 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.572 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:45:28.572 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:54.459684+0000", 2024-09-20T22:45:28.572 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010008398999999999 2024-09-20T22:45:28.572 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.572 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.573 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:45:28.573 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:54.459996+0000", 2024-09-20T22:45:28.573 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00031162399999999998 2024-09-20T22:45:28.573 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.573 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.573 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:45:28.573 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:54.460030+0000", 2024-09-20T22:45:28.573 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.3770999999999999e-05 2024-09-20T22:45:28.573 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.573 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.573 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:45:28.573 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:54.460071+0000", 2024-09-20T22:45:28.573 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.1257000000000002e-05 2024-09-20T22:45:28.574 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.574 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.574 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:45:28.574 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:54.460110+0000", 2024-09-20T22:45:28.574 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.9150000000000003e-05 2024-09-20T22:45:28.574 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:28.574 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:45:28.574 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:28.574 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.574 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.574 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24652.0:947 7.c e159/94 7:309a425f:::smithi08477203-13:head v 159'151, mlcod=156'148)", 2024-09-20T22:45:28.574 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:43:58.152675+0000", 2024-09-20T22:45:28.574 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 90.360596107999996, 2024-09-20T22:45:28.574 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013285286, 2024-09-20T22:45:28.575 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:45:28.575 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:45:28.575 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:45:28.575 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:45:28.575 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.575 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:45:28.575 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:58.152675+0000", 2024-09-20T22:45:28.575 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:45:28.575 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.575 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.575 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:45:28.575 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:58.152675+0000", 2024-09-20T22:45:28.575 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:45:28.576 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.576 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.576 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:45:28.576 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:58.152677+0000", 2024-09-20T22:45:28.576 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3710000000000001e-06 2024-09-20T22:45:28.576 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.576 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.576 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:45:28.576 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:58.153173+0000", 2024-09-20T22:45:28.576 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00049638299999999998 2024-09-20T22:45:28.576 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.576 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.576 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:45:28.576 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:58.153189+0000", 2024-09-20T22:45:28.577 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.5834000000000001e-05 2024-09-20T22:45:28.577 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.577 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.577 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:45:28.577 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:58.153196+0000", 2024-09-20T22:45:28.577 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.4660000000000002e-06 2024-09-20T22:45:28.577 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.577 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.577 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:45:28.577 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:58.153252+0000", 2024-09-20T22:45:28.577 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.5615999999999997e-05 2024-09-20T22:45:28.577 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.577 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.577 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:45:28.577 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:58.153271+0000", 2024-09-20T22:45:28.578 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.9103e-05 2024-09-20T22:45:28.578 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.578 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.578 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:45:28.578 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:58.165894+0000", 2024-09-20T22:45:28.578 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.012623558999999999 2024-09-20T22:45:28.578 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.578 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.578 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:45:28.578 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:58.165960+0000", 2024-09-20T22:45:28.578 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.5953999999999999e-05 2024-09-20T22:45:28.578 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:28.578 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:45:28.578 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:28.579 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.579 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.579 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24652.0:948 7.c e159/94 7:309a425f:::smithi08477203-13:head v 159'152, mlcod=156'148)", 2024-09-20T22:45:28.579 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:43:58.153560+0000", 2024-09-20T22:45:28.579 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 90.359711551000004, 2024-09-20T22:45:28.579 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.012435414000000001, 2024-09-20T22:45:28.579 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:45:28.579 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:45:28.579 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:45:28.579 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:45:28.579 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.579 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:45:28.579 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:58.153560+0000", 2024-09-20T22:45:28.579 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:45:28.579 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.579 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.580 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:45:28.580 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:58.153560+0000", 2024-09-20T22:45:28.580 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:45:28.580 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.580 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.580 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:45:28.580 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:58.153561+0000", 2024-09-20T22:45:28.580 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.7499999999999999e-07 2024-09-20T22:45:28.580 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.580 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.580 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:45:28.580 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:58.153567+0000", 2024-09-20T22:45:28.580 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.0009999999999998e-06 2024-09-20T22:45:28.580 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.581 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.581 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:45:28.581 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:58.153576+0000", 2024-09-20T22:45:28.581 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.6739999999999992e-06 2024-09-20T22:45:28.581 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.581 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.581 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:45:28.581 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:58.153584+0000", 2024-09-20T22:45:28.581 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.3089999999999998e-06 2024-09-20T22:45:28.581 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.581 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.581 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:45:28.581 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:58.153705+0000", 2024-09-20T22:45:28.581 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000121499 2024-09-20T22:45:28.582 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.582 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.582 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:45:28.582 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:58.153728+0000", 2024-09-20T22:45:28.582 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.2529999999999999e-05 2024-09-20T22:45:28.582 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.582 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.582 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:45:28.582 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:58.165966+0000", 2024-09-20T22:45:28.582 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.012238847000000001 2024-09-20T22:45:28.582 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.582 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.582 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:45:28.582 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:58.165995+0000", 2024-09-20T22:45:28.583 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.8679000000000001e-05 2024-09-20T22:45:28.583 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:28.583 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:45:28.583 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:28.583 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.583 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.583 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:1022 7.a 7:5122292a:::smithi08477203-33:head [write 789962~534747,stat] snapc 25=[25,24,23,21] ondisk+write+known_if_redirected+supports_pool_eio e166)", 2024-09-20T22:45:28.583 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:44:05.138728+0000", 2024-09-20T22:45:28.583 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 83.374542829999996, 2024-09-20T22:45:28.583 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013886904, 2024-09-20T22:45:28.583 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:45:28.583 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:45:28.583 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:45:28.584 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:45:28.584 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:45:28.584 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:45:28.584 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1022 2024-09-20T22:45:28.584 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.584 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:45:28.584 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.584 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:45:28.584 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:05.138728+0000", 2024-09-20T22:45:28.584 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:45:28.584 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.584 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.584 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:45:28.584 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:05.138728+0000", 2024-09-20T22:45:28.584 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:45:28.585 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.585 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.585 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:45:28.585 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:05.138730+0000", 2024-09-20T22:45:28.585 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.702e-06 2024-09-20T22:45:28.585 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.585 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.585 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:45:28.585 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:45:28.587 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:05.139052+0000", 2024-09-20T22:45:28.587 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00032173400000000001 2024-09-20T22:45:28.587 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.587 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.587 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:45:28.587 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:05.139073+0000", 2024-09-20T22:45:28.587 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.0672000000000001e-05 2024-09-20T22:45:28.587 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.587 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.588 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:45:28.588 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:05.139084+0000", 2024-09-20T22:45:28.588 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1005000000000001e-05 2024-09-20T22:45:28.588 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.588 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.588 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:45:28.588 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:05.139155+0000", 2024-09-20T22:45:28.588 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.1521999999999997e-05 2024-09-20T22:45:28.588 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.588 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.588 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op must wait for map", 2024-09-20T22:45:28.588 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:05.139163+0000", 2024-09-20T22:45:28.588 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.537e-06 2024-09-20T22:45:28.588 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.589 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.589 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:45:28.589 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:05.144822+0000", 2024-09-20T22:45:28.589 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0056589409999999998 2024-09-20T22:45:28.589 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.589 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.589 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:45:28.589 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:05.144913+0000", 2024-09-20T22:45:28.589 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.1419e-05 2024-09-20T22:45:28.589 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.589 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.589 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 2,5", 2024-09-20T22:45:28.589 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:05.145052+0000", 2024-09-20T22:45:28.589 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013893900000000001 2024-09-20T22:45:28.590 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.590 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.590 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:45:28.590 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:05.148990+0000", 2024-09-20T22:45:28.590 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0039383250000000003 2024-09-20T22:45:28.590 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.590 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.590 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:45:28.590 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:05.151217+0000", 2024-09-20T22:45:28.590 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002226287 2024-09-20T22:45:28.590 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.590 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.590 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:45:28.590 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:05.152483+0000", 2024-09-20T22:45:28.590 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0012667679999999999 2024-09-20T22:45:28.591 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.591 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.591 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:45:28.591 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:05.152556+0000", 2024-09-20T22:45:28.591 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.2700000000000005e-05 2024-09-20T22:45:28.591 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.591 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.591 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:45:28.591 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:05.152615+0000", 2024-09-20T22:45:28.591 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.9352999999999999e-05 2024-09-20T22:45:28.591 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:28.591 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:45:28.591 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:28.591 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.592 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.592 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.14478.0:1259 5.3 5:c609908c:::notify.5:head [watch reconnect cookie 94843898679808 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e169)", 2024-09-20T22:45:28.592 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:44:08.230723+0000", 2024-09-20T22:45:28.592 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 80.282548317000007, 2024-09-20T22:45:28.592 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.94983577399999997, 2024-09-20T22:45:28.592 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:45:28.592 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:45:28.592 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-20T22:45:28.592 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:45:28.592 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.14478", 2024-09-20T22:45:28.592 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/2994273271", 2024-09-20T22:45:28.592 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1259 2024-09-20T22:45:28.592 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.592 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:45:28.593 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.593 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:45:28.593 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:08.230723+0000", 2024-09-20T22:45:28.593 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:45:28.593 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.593 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.593 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:45:28.593 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:08.230723+0000", 2024-09-20T22:45:28.593 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:45:28.593 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.593 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.593 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:45:28.593 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:08.230724+0000", 2024-09-20T22:45:28.593 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.077e-06 2024-09-20T22:45:28.593 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.594 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.594 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:45:28.594 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:08.230729+0000", 2024-09-20T22:45:28.594 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.9930000000000003e-06 2024-09-20T22:45:28.594 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.594 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.594 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:45:28.594 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:08.230741+0000", 2024-09-20T22:45:28.594 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1897e-05 2024-09-20T22:45:28.594 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.594 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.594 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:45:28.594 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:08.230747+0000", 2024-09-20T22:45:28.594 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.9830000000000001e-06 2024-09-20T22:45:28.595 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.595 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.595 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:45:28.595 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:08.230802+0000", 2024-09-20T22:45:28.595 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.4657000000000002e-05 2024-09-20T22:45:28.595 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.595 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.595 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-09-20T22:45:28.595 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:08.230811+0000", 2024-09-20T22:45:28.595 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.1160000000000005e-06 2024-09-20T22:45:28.595 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.595 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.595 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:45:28.595 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:09.171968+0000", 2024-09-20T22:45:28.596 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.94115747299999997 2024-09-20T22:45:28.596 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.596 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.596 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for missing object", 2024-09-20T22:45:28.596 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:09.172098+0000", 2024-09-20T22:45:28.596 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00012937499999999999 2024-09-20T22:45:28.596 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.596 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.596 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:45:28.596 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:09.174248+0000", 2024-09-20T22:45:28.596 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0021508959999999998 2024-09-20T22:45:28.596 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.596 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.596 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for rw locks", 2024-09-20T22:45:28.597 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:09.174341+0000", 2024-09-20T22:45:28.597 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.2906999999999997e-05 2024-09-20T22:45:28.597 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.597 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.597 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:45:28.597 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:09.175083+0000", 2024-09-20T22:45:28.597 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00074120100000000003 2024-09-20T22:45:28.597 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.597 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.597 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:45:28.597 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:09.175149+0000", 2024-09-20T22:45:28.597 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.6877000000000004e-05 2024-09-20T22:45:28.597 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.597 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.597 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:45:28.598 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:09.180559+0000", 2024-09-20T22:45:28.598 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0054093220000000003 2024-09-20T22:45:28.598 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:28.598 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:45:28.598 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:28.598 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.598 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.598 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:1417 7.1 7:84a866f5:::smithi08477203-27:head [append 0~0] snapc 2f=[2f] ondisk+write+known_if_redirected+supports_pool_eio e189)", 2024-09-20T22:45:28.598 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:44:28.137465+0000", 2024-09-20T22:45:28.598 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 60.375805958000001, 2024-09-20T22:45:28.598 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.020090858999999999, 2024-09-20T22:45:28.598 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:45:28.598 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:45:28.598 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:45:28.599 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:45:28.599 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:45:28.599 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:45:28.599 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1417 2024-09-20T22:45:28.599 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.599 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:45:28.599 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.599 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:45:28.599 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.137465+0000", 2024-09-20T22:45:28.599 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:45:28.599 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.599 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.599 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:45:28.599 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.137465+0000", 2024-09-20T22:45:28.599 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:45:28.600 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.600 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.600 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:45:28.600 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.137468+0000", 2024-09-20T22:45:28.600 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.7259999999999998e-06 2024-09-20T22:45:28.600 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.600 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.600 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:45:28.600 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.137481+0000", 2024-09-20T22:45:28.600 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2937e-05 2024-09-20T22:45:28.600 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.600 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.600 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:45:28.600 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.137528+0000", 2024-09-20T22:45:28.601 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.7011000000000001e-05 2024-09-20T22:45:28.601 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.601 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.601 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:45:28.601 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.137542+0000", 2024-09-20T22:45:28.601 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3529e-05 2024-09-20T22:45:28.601 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.601 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.601 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:45:28.601 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.137636+0000", 2024-09-20T22:45:28.601 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.4932999999999994e-05 2024-09-20T22:45:28.601 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.601 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.601 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op must wait for map", 2024-09-20T22:45:28.601 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.137645+0000", 2024-09-20T22:45:28.602 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.3180000000000007e-06 2024-09-20T22:45:28.602 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.602 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.602 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:45:28.602 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.143542+0000", 2024-09-20T22:45:28.602 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0058974459999999998 2024-09-20T22:45:28.602 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.602 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.602 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:45:28.602 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.143647+0000", 2024-09-20T22:45:28.602 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000105028 2024-09-20T22:45:28.602 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.602 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.602 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 1,6", 2024-09-20T22:45:28.603 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.143835+0000", 2024-09-20T22:45:28.603 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00018802599999999999 2024-09-20T22:45:28.603 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.603 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.603 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:45:28.603 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.155596+0000", 2024-09-20T22:45:28.603 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011760708 2024-09-20T22:45:28.603 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.603 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.603 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:45:28.603 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.155674+0000", 2024-09-20T22:45:28.603 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.7883e-05 2024-09-20T22:45:28.603 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.604 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.604 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:45:28.604 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.157440+0000", 2024-09-20T22:45:28.604 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001766014 2024-09-20T22:45:28.604 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.604 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.604 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:45:28.604 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.157505+0000", 2024-09-20T22:45:28.604 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.5252999999999993e-05 2024-09-20T22:45:28.604 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.604 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.604 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:45:28.604 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.157556+0000", 2024-09-20T22:45:28.605 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.1047000000000001e-05 2024-09-20T22:45:28.605 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:28.605 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:45:28.605 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:28.605 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.605 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.605 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:1418 7.1 7:84a866f5:::smithi08477203-27:head [rollback 2f] snapc 2f=[2f] ondisk+write+known_if_redirected+supports_pool_eio e189)", 2024-09-20T22:45:28.605 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:44:28.137587+0000", 2024-09-20T22:45:28.605 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 60.375684120000003, 2024-09-20T22:45:28.605 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.020550452, 2024-09-20T22:45:28.605 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:45:28.605 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:45:28.605 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:45:28.605 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:45:28.606 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:45:28.606 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:45:28.606 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1418 2024-09-20T22:45:28.606 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.606 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:45:28.606 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.606 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:45:28.606 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.137587+0000", 2024-09-20T22:45:28.606 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:45:28.606 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.606 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.606 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:45:28.606 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.137587+0000", 2024-09-20T22:45:28.607 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:45:28.607 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.607 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.607 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:45:28.607 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.137588+0000", 2024-09-20T22:45:28.607 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.4700000000000001e-07 2024-09-20T22:45:28.607 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.607 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.607 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:45:28.607 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.137591+0000", 2024-09-20T22:45:28.607 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.0000000000000001e-06 2024-09-20T22:45:28.607 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.607 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.607 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:45:28.608 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.137681+0000", 2024-09-20T22:45:28.608 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.0486000000000001e-05 2024-09-20T22:45:28.608 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.608 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.608 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:45:28.608 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.138018+0000", 2024-09-20T22:45:28.608 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00033709899999999999 2024-09-20T22:45:28.608 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.608 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.608 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:45:28.608 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.138109+0000", 2024-09-20T22:45:28.608 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.0538999999999995e-05 2024-09-20T22:45:28.608 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.608 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.609 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting_for_map not empty", 2024-09-20T22:45:28.609 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.138114+0000", 2024-09-20T22:45:28.609 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.6590000000000004e-06 2024-09-20T22:45:28.609 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.609 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.609 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:45:28.609 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.144398+0000", 2024-09-20T22:45:28.609 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0062832599999999997 2024-09-20T22:45:28.609 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.609 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.609 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:45:28.609 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.144483+0000", 2024-09-20T22:45:28.609 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.4772999999999996e-05 2024-09-20T22:45:28.609 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.609 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.610 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 1,6", 2024-09-20T22:45:28.610 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.144661+0000", 2024-09-20T22:45:28.610 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000178736 2024-09-20T22:45:28.610 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.610 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.610 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:45:28.610 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.155602+0000", 2024-09-20T22:45:28.610 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010940302000000001 2024-09-20T22:45:28.610 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.610 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.610 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:45:28.610 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.156162+0000", 2024-09-20T22:45:28.611 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00056014599999999997 2024-09-20T22:45:28.611 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.611 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.611 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:45:28.611 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.158054+0000", 2024-09-20T22:45:28.611 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00189223 2024-09-20T22:45:28.611 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.611 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.611 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:45:28.611 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.158100+0000", 2024-09-20T22:45:28.611 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.6279999999999997e-05 2024-09-20T22:45:28.611 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.611 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.611 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:45:28.611 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.158138+0000", 2024-09-20T22:45:28.611 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.7394999999999997e-05 2024-09-20T22:45:28.612 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:28.612 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:45:28.612 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:28.612 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.612 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.612 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:1419 7.1 7:84a866f5:::smithi08477203-27:head [append 0~0] snapc 2f=[2f] ondisk+write+known_if_redirected+supports_pool_eio e189)", 2024-09-20T22:45:28.612 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:44:28.138055+0000", 2024-09-20T22:45:28.612 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 60.375216678000001, 2024-09-20T22:45:28.612 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.030290338999999999, 2024-09-20T22:45:28.612 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:45:28.612 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:45:28.612 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:45:28.612 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:45:28.612 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:45:28.613 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:45:28.613 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1419 2024-09-20T22:45:28.613 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.613 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:45:28.613 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.613 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:45:28.613 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.138055+0000", 2024-09-20T22:45:28.613 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:45:28.613 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.613 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.613 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:45:28.613 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.138055+0000", 2024-09-20T22:45:28.613 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:45:28.614 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.614 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.614 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:45:28.614 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.138056+0000", 2024-09-20T22:45:28.614 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.13e-06 2024-09-20T22:45:28.614 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.614 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.614 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:45:28.614 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.138060+0000", 2024-09-20T22:45:28.614 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.0269999999999997e-06 2024-09-20T22:45:28.614 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.614 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.614 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:45:28.615 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.138070+0000", 2024-09-20T22:45:28.615 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.961e-06 2024-09-20T22:45:28.615 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.615 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.615 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:45:28.615 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.138076+0000", 2024-09-20T22:45:28.615 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.1800000000000001e-06 2024-09-20T22:45:28.615 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.615 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.615 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:45:28.615 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.138137+0000", 2024-09-20T22:45:28.615 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.1159000000000002e-05 2024-09-20T22:45:28.615 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.616 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.616 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting_for_map not empty", 2024-09-20T22:45:28.616 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.138141+0000", 2024-09-20T22:45:28.616 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.189e-06 2024-09-20T22:45:28.616 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.616 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.616 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:45:28.616 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.154982+0000", 2024-09-20T22:45:28.616 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.016840738000000001 2024-09-20T22:45:28.616 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.616 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.616 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:45:28.616 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.155089+0000", 2024-09-20T22:45:28.617 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00010653399999999999 2024-09-20T22:45:28.617 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.617 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.617 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 1,6", 2024-09-20T22:45:28.617 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.155320+0000", 2024-09-20T22:45:28.617 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00023196299999999999 2024-09-20T22:45:28.617 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.617 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.617 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:45:28.617 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.158224+0000", 2024-09-20T22:45:28.617 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00290361 2024-09-20T22:45:28.617 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.617 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.617 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:45:28.618 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.167686+0000", 2024-09-20T22:45:28.618 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0094622360000000006 2024-09-20T22:45:28.618 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.618 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.618 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:45:28.618 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.168183+0000", 2024-09-20T22:45:28.618 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00049655999999999995 2024-09-20T22:45:28.618 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.618 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.618 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:45:28.618 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.168273+0000", 2024-09-20T22:45:28.618 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.9758000000000004e-05 2024-09-20T22:45:28.618 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.619 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.619 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:45:28.619 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.168345+0000", 2024-09-20T22:45:28.619 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.2293999999999996e-05 2024-09-20T22:45:28.619 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:28.619 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:45:28.619 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:28.619 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.619 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.619 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24652.0:1652 7.c e206/94 7:31d0211f:::smithi08477203-25:head v 206'260, mlcod=206'258)", 2024-09-20T22:45:28.619 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:44:45.176127+0000", 2024-09-20T22:45:28.619 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 43.337144369999997, 2024-09-20T22:45:28.619 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013062403, 2024-09-20T22:45:28.619 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:45:28.620 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:45:28.620 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:45:28.620 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:45:28.620 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.620 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:45:28.620 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:45.176127+0000", 2024-09-20T22:45:28.620 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:45:28.620 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.620 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.620 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:45:28.620 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:45.176127+0000", 2024-09-20T22:45:28.620 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:45:28.620 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.620 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.621 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:45:28.621 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:45.176128+0000", 2024-09-20T22:45:28.621 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.559e-06 2024-09-20T22:45:28.621 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.621 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.621 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:45:28.621 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:45.183531+0000", 2024-09-20T22:45:28.621 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0074027750000000003 2024-09-20T22:45:28.621 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.621 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.621 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:45:28.621 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:45.183548+0000", 2024-09-20T22:45:28.622 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.6481e-05 2024-09-20T22:45:28.622 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.622 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.622 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:45:28.622 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:45.183558+0000", 2024-09-20T22:45:28.622 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0105000000000001e-05 2024-09-20T22:45:28.622 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.622 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.622 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:45:28.622 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:45.183627+0000", 2024-09-20T22:45:28.622 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.9586999999999996e-05 2024-09-20T22:45:28.622 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.622 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.623 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:45:28.623 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:45.183646+0000", 2024-09-20T22:45:28.623 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.8275e-05 2024-09-20T22:45:28.623 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.623 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.623 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:45:28.623 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:45.189146+0000", 2024-09-20T22:45:28.623 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0055007540000000001 2024-09-20T22:45:28.623 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.623 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.623 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:45:28.623 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:45.189189+0000", 2024-09-20T22:45:28.623 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.2867000000000001e-05 2024-09-20T22:45:28.623 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:28.623 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:45:28.624 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:28.624 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.624 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.624 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:1806 7.a 7:5aaef466:::smithi08477203-40:head [append 0~0] snapc 46=[45,44,43,42,41,40,3e] ondisk+write+known_if_redirected+supports_pool_eio e229)", 2024-09-20T22:45:28.624 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:45:08.177362+0000", 2024-09-20T22:45:28.624 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 20.335908786000001, 2024-09-20T22:45:28.624 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.012882621, 2024-09-20T22:45:28.624 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:45:28.624 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:45:28.624 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:45:28.624 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:45:28.624 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:45:28.624 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:45:28.624 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1806 2024-09-20T22:45:28.625 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.625 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:45:28.625 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.625 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:45:28.625 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.177362+0000", 2024-09-20T22:45:28.625 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:45:28.625 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.625 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.625 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:45:28.625 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.177362+0000", 2024-09-20T22:45:28.625 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:45:28.625 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.625 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.626 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:45:28.626 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.177365+0000", 2024-09-20T22:45:28.626 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.323e-06 2024-09-20T22:45:28.626 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.626 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.626 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:45:28.626 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.177374+0000", 2024-09-20T22:45:28.626 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.6560000000000004e-06 2024-09-20T22:45:28.626 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.626 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.626 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:45:28.626 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.177397+0000", 2024-09-20T22:45:28.626 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.2178999999999999e-05 2024-09-20T22:45:28.626 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.626 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.627 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:45:28.627 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.177408+0000", 2024-09-20T22:45:28.627 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1626e-05 2024-09-20T22:45:28.627 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.627 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.627 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:45:28.627 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.177484+0000", 2024-09-20T22:45:28.627 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.5253999999999995e-05 2024-09-20T22:45:28.627 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.627 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.627 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:45:28.627 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.177593+0000", 2024-09-20T22:45:28.627 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000109771 2024-09-20T22:45:28.627 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.628 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.628 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 2,5", 2024-09-20T22:45:28.628 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.177798+0000", 2024-09-20T22:45:28.628 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00020441 2024-09-20T22:45:28.628 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.628 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.628 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:45:28.628 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.181333+0000", 2024-09-20T22:45:28.628 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0035350160000000002 2024-09-20T22:45:28.628 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.628 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.628 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:45:28.628 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.186896+0000", 2024-09-20T22:45:28.628 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0055632269999999996 2024-09-20T22:45:28.629 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.629 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.629 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:45:28.629 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.190105+0000", 2024-09-20T22:45:28.629 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0032095539999999999 2024-09-20T22:45:28.629 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.629 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.629 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:45:28.629 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.190192+0000", 2024-09-20T22:45:28.629 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.6941999999999995e-05 2024-09-20T22:45:28.629 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.629 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.629 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:45:28.630 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.190245+0000", 2024-09-20T22:45:28.630 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.2663e-05 2024-09-20T22:45:28.630 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:28.630 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:45:28.630 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:28.630 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.630 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.630 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:1807 7.a 7:5aaef466:::smithi08477203-40:head [rollback 44] snapc 46=[45,44,43,42,41,40,3e] ondisk+write+known_if_redirected+supports_pool_eio e229)", 2024-09-20T22:45:28.630 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:45:08.177434+0000", 2024-09-20T22:45:28.630 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 20.33583703, 2024-09-20T22:45:28.630 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013035435, 2024-09-20T22:45:28.630 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:45:28.630 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:45:28.631 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:45:28.631 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:45:28.631 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:45:28.631 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:45:28.631 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1807 2024-09-20T22:45:28.631 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.631 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:45:28.631 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.631 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:45:28.631 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.177434+0000", 2024-09-20T22:45:28.631 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:45:28.631 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.631 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.631 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:45:28.632 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.177434+0000", 2024-09-20T22:45:28.632 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:45:28.632 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.632 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.632 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:45:28.632 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.177435+0000", 2024-09-20T22:45:28.632 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.7300000000000001e-07 2024-09-20T22:45:28.632 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.632 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.632 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:45:28.632 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.177437+0000", 2024-09-20T22:45:28.632 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.4789999999999999e-06 2024-09-20T22:45:28.632 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.632 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.633 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:45:28.633 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.177442+0000", 2024-09-20T22:45:28.633 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.2939999999999996e-06 2024-09-20T22:45:28.633 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.633 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.633 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:45:28.633 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.177448+0000", 2024-09-20T22:45:28.633 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.1749999999999996e-06 2024-09-20T22:45:28.633 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.633 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.633 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:45:28.633 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.178489+0000", 2024-09-20T22:45:28.633 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010416819999999999 2024-09-20T22:45:28.633 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.634 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.634 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:45:28.634 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.178564+0000", 2024-09-20T22:45:28.634 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.4709000000000006e-05 2024-09-20T22:45:28.634 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.634 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.634 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 2,5", 2024-09-20T22:45:28.634 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.178741+0000", 2024-09-20T22:45:28.634 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00017662600000000001 2024-09-20T22:45:28.634 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.634 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.634 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:45:28.634 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.181498+0000", 2024-09-20T22:45:28.634 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0027571890000000002 2024-09-20T22:45:28.635 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.635 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.635 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:45:28.635 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.189567+0000", 2024-09-20T22:45:28.635 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0080690789999999998 2024-09-20T22:45:28.635 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.635 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.635 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:45:28.635 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.190376+0000", 2024-09-20T22:45:28.635 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00080912099999999999 2024-09-20T22:45:28.635 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.635 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.635 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:45:28.635 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.190430+0000", 2024-09-20T22:45:28.635 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.3510000000000001e-05 2024-09-20T22:45:28.636 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.636 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.636 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:45:28.636 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.190470+0000", 2024-09-20T22:45:28.636 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.0098000000000001e-05 2024-09-20T22:45:28.636 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:28.636 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:45:28.636 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:28.636 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.636 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.636 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:1808 7.a 7:5aaef466:::smithi08477203-40:head [append 0~0] snapc 46=[45,44,43,42,41,40,3e] ondisk+write+known_if_redirected+supports_pool_eio e229)", 2024-09-20T22:45:28.636 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:45:08.177486+0000", 2024-09-20T22:45:28.637 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 20.335785735000002, 2024-09-20T22:45:28.637 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013133808, 2024-09-20T22:45:28.637 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:45:28.637 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:45:28.637 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:45:28.637 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:45:28.637 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:45:28.637 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:45:28.637 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1808 2024-09-20T22:45:28.637 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.637 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:45:28.637 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.637 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:45:28.638 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.177486+0000", 2024-09-20T22:45:28.638 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:45:28.638 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.638 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.638 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:45:28.638 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.177486+0000", 2024-09-20T22:45:28.638 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:45:28.638 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.638 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.638 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:45:28.638 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.177486+0000", 2024-09-20T22:45:28.638 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.9999999999999996e-07 2024-09-20T22:45:28.638 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.639 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.639 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:45:28.639 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.177491+0000", 2024-09-20T22:45:28.639 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.42e-06 2024-09-20T22:45:28.639 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.639 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.639 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:45:28.639 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.177502+0000", 2024-09-20T22:45:28.639 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1134999999999999e-05 2024-09-20T22:45:28.639 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.639 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.639 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:45:28.639 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.177510+0000", 2024-09-20T22:45:28.639 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.9959999999999993e-06 2024-09-20T22:45:28.640 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.640 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.640 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:45:28.640 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.179261+0000", 2024-09-20T22:45:28.640 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001750822 2024-09-20T22:45:28.640 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.640 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.640 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:45:28.640 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.179340+0000", 2024-09-20T22:45:28.640 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.9568999999999997e-05 2024-09-20T22:45:28.640 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.640 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.640 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 2,5", 2024-09-20T22:45:28.641 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.179486+0000", 2024-09-20T22:45:28.641 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014589900000000001 2024-09-20T22:45:28.641 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.641 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.641 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:45:28.641 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.181612+0000", 2024-09-20T22:45:28.641 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0021254080000000001 2024-09-20T22:45:28.641 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.641 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.641 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:45:28.641 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.189574+0000", 2024-09-20T22:45:28.641 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0079625960000000006 2024-09-20T22:45:28.641 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.641 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.641 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:45:28.642 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.190525+0000", 2024-09-20T22:45:28.642 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00095077499999999995 2024-09-20T22:45:28.642 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.642 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.642 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:45:28.642 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.190579+0000", 2024-09-20T22:45:28.642 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.4092999999999999e-05 2024-09-20T22:45:28.642 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:28.642 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:28.642 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:45:28.642 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.190619+0000", 2024-09-20T22:45:28.642 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.0295e-05 2024-09-20T22:45:28.642 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:28.643 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:45:28.643 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:28.643 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:28.643 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:45:28.643 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-20T22:45:28.658 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:28 smithi154 ceph-mon[33264]: osdmap e248: 8 total, 8 up, 7 in 2024-09-20T22:45:28.658 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:28 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/1293145613' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:45:28.658 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:28 smithi154 ceph-mon[33264]: osdmap e249: 8 total, 8 up, 7 in 2024-09-20T22:45:28.663 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:28.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:28.664 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:45:28.765 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:45:28.787 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:28.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:28.788 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:45:28.888 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:45:28.909 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:28.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:28.910 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:45:29.011 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:45:29.033 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:29.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:29.033 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:45:29.134 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:45:29.156 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:29.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:29.157 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:45:29.165 INFO:tasks.rados.rados.0.smithi084.stdout:820: done (5 left) 2024-09-20T22:45:29.165 INFO:tasks.rados.rados.0.smithi084.stdout:821: write oid 40 current snap is 80 2024-09-20T22:45:29.165 INFO:tasks.rados.rados.0.smithi084.stdout:821: seq_num 316 ranges {511012=549157,1775251=550760,3095195=215817} 2024-09-20T22:45:29.171 INFO:tasks.rados.rados.0.smithi084.stdout:821: writing smithi08477203-40 from 511012 to 1060169 tid 1 2024-09-20T22:45:29.178 INFO:tasks.rados.rados.0.smithi084.stdout:821: writing smithi08477203-40 from 1775251 to 2326011 tid 2 2024-09-20T22:45:29.184 INFO:tasks.rados.rados.0.smithi084.stdout:821: writing smithi08477203-40 from 3095195 to 3311012 tid 3 2024-09-20T22:45:29.184 INFO:tasks.rados.rados.0.smithi084.stdout:822: snap_create 2024-09-20T22:45:29.184 INFO:tasks.rados.rados.0.smithi084.stdout:818: finishing rollback tid 0 to smithi08477203-37 2024-09-20T22:45:29.184 INFO:tasks.rados.rados.0.smithi084.stdout:818: finishing rollback tid 1 to smithi08477203-37 2024-09-20T22:45:29.184 INFO:tasks.rados.rados.0.smithi084.stdout:818: finishing rollback tid 2 to smithi08477203-37 2024-09-20T22:45:29.184 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 37 v 158 (ObjNum 253 snap 59 seq_num 253) dirty exists 2024-09-20T22:45:29.184 INFO:tasks.rados.rados.0.smithi084.stdout:817: expect (ObjNum 245 snap 54 seq_num 245) 2024-09-20T22:45:29.208 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-20T22:45:29.258 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:45:29.282 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:29.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:29.283 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:45:29.310 INFO:tasks.rados.rados.0.smithi084.stdout:815: finishing rollback tid 0 to smithi08477203-25 2024-09-20T22:45:29.311 INFO:tasks.rados.rados.0.smithi084.stdout:815: finishing rollback tid 1 to smithi08477203-25 2024-09-20T22:45:29.311 INFO:tasks.rados.rados.0.smithi084.stdout:815: finishing rollback tid 2 to smithi08477203-25 2024-09-20T22:45:29.311 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 25 v 372 (ObjNum 234 snap 51 seq_num 234) dirty exists 2024-09-20T22:45:29.311 INFO:tasks.rados.rados.0.smithi084.stdout:816: finishing rollback tid 0 to smithi08477203-19 2024-09-20T22:45:29.311 INFO:tasks.rados.rados.0.smithi084.stdout:816: finishing rollback tid 1 to smithi08477203-19 2024-09-20T22:45:29.311 INFO:tasks.rados.rados.0.smithi084.stdout:816: finishing rollback tid 2 to smithi08477203-19 2024-09-20T22:45:29.311 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 19 v 376 (ObjNum 258 snap 60 seq_num 258) dirty exists 2024-09-20T22:45:29.311 INFO:tasks.rados.rados.0.smithi084.stdout:819: expect (ObjNum 281 snap 71 seq_num 281) 2024-09-20T22:45:29.384 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:45:29.405 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:29.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:29.406 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:45:29.408 INFO:tasks.rados.rados.0.smithi084.stdout:821: finishing write tid 1 to smithi08477203-40 2024-09-20T22:45:29.409 INFO:tasks.rados.rados.0.smithi084.stdout:821: finishing write tid 2 to smithi08477203-40 2024-09-20T22:45:29.409 INFO:tasks.rados.rados.0.smithi084.stdout:821: finishing write tid 3 to smithi08477203-40 2024-09-20T22:45:29.409 INFO:tasks.rados.rados.0.smithi084.stdout:821: finishing write tid 4 to smithi08477203-40 2024-09-20T22:45:29.409 INFO:tasks.rados.rados.0.smithi084.stdout:821: finishing write tid 5 to smithi08477203-40 2024-09-20T22:45:29.409 INFO:tasks.rados.rados.0.smithi084.stdout:821: oid 40 updating version 0 to 170 2024-09-20T22:45:29.409 INFO:tasks.rados.rados.0.smithi084.stdout:821: oid 40 version 170 is already newer than 169 2024-09-20T22:45:29.409 INFO:tasks.rados.rados.0.smithi084.stdout:821: oid 40 updating version 170 to 172 2024-09-20T22:45:29.409 INFO:tasks.rados.rados.0.smithi084.stdout:821: oid 40 version 172 is already newer than 171 2024-09-20T22:45:29.409 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 40 v 172 (ObjNum 316 snap 80 seq_num 316) dirty exists 2024-09-20T22:45:29.409 INFO:tasks.rados.rados.0.smithi084.stdout:821: left oid 40 (ObjNum 316 snap 80 seq_num 316) 2024-09-20T22:45:29.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:29 smithi154 ceph-mon[33264]: pgmap v274: 178 pgs: 2 active+clean+snaptrim, 176 active+clean; 167 MiB data, 648 MiB used, 625 GiB / 626 GiB avail 2024-09-20T22:45:29.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:29 smithi154 ceph-mon[33264]: osdmap e250: 8 total, 8 up, 7 in 2024-09-20T22:45:29.504 INFO:journalctl@ceph.iscsi.iscsi.a.smithi154.stdout:Sep 20 22:45:29 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-iscsi-iscsi-a[59730]: debug there is no tcmu-runner data available 2024-09-20T22:45:29.508 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:45:29.528 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:29.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:29.529 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:45:29.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:29 smithi084 ceph-mon[40024]: pgmap v274: 178 pgs: 2 active+clean+snaptrim, 176 active+clean; 167 MiB data, 648 MiB used, 625 GiB / 626 GiB avail 2024-09-20T22:45:29.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:29 smithi084 ceph-mon[40024]: osdmap e250: 8 total, 8 up, 7 in 2024-09-20T22:45:29.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:29 smithi084 ceph-mon[29774]: pgmap v274: 178 pgs: 2 active+clean+snaptrim, 176 active+clean; 167 MiB data, 648 MiB used, 625 GiB / 626 GiB avail 2024-09-20T22:45:29.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:29 smithi084 ceph-mon[29774]: osdmap e250: 8 total, 8 up, 7 in 2024-09-20T22:45:29.630 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:45:29.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:29.652 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:29.653 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:45:29.753 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:45:29.776 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:29.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:29.777 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:45:29.786 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-20T22:45:29.877 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:45:29.899 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:29.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:29.900 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:45:30.001 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:45:30.023 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:30.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:30.024 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:45:30.040 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:45:30.125 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:45:30.146 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:30.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:30.147 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:45:30.166 INFO:tasks.rados.rados.0.smithi084.stdout:815: done (6 left) 2024-09-20T22:45:30.166 INFO:tasks.rados.rados.0.smithi084.stdout:816: done (5 left) 2024-09-20T22:45:30.166 INFO:tasks.rados.rados.0.smithi084.stdout:817: done (4 left) 2024-09-20T22:45:30.166 INFO:tasks.rados.rados.0.smithi084.stdout:818: done (3 left) 2024-09-20T22:45:30.167 INFO:tasks.rados.rados.0.smithi084.stdout:819: done (2 left) 2024-09-20T22:45:30.167 INFO:tasks.rados.rados.0.smithi084.stdout:821: done (1 left) 2024-09-20T22:45:30.167 INFO:tasks.rados.rados.0.smithi084.stdout:822: done (0 left) 2024-09-20T22:45:30.167 INFO:tasks.rados.rados.0.smithi084.stdout:823: rollback oid 15 current snap is 81 2024-09-20T22:45:30.167 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 15 to 80 2024-09-20T22:45:30.167 INFO:tasks.rados.rados.0.smithi084.stdout:824: write oid 31 current snap is 81 2024-09-20T22:45:30.167 INFO:tasks.rados.rados.0.smithi084.stdout:824: seq_num 317 ranges {527819=624406,1833733=410771,2722618=601581} 2024-09-20T22:45:30.172 INFO:tasks.rados.rados.0.smithi084.stdout:824: writing smithi08477203-31 from 527819 to 1152225 tid 1 2024-09-20T22:45:30.178 INFO:tasks.rados.rados.0.smithi084.stdout:824: writing smithi08477203-31 from 1833733 to 2244504 tid 2 2024-09-20T22:45:30.184 INFO:tasks.rados.rados.0.smithi084.stdout:824: writing smithi08477203-31 from 2722618 to 3324199 tid 3 2024-09-20T22:45:30.185 INFO:tasks.rados.rados.0.smithi084.stdout:825: write (excl) oid 10 current snap is 81 2024-09-20T22:45:30.185 INFO:tasks.rados.rados.0.smithi084.stdout:823: finishing rollback tid 1 to smithi08477203-15 2024-09-20T22:45:30.185 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 15 v 0 (ObjNum 145 snap 0 seq_num 2090975744) dirty dne 2024-09-20T22:45:30.185 INFO:tasks.rados.rados.0.smithi084.stdout:825: seq_num 318 ranges {544626=699655,1892215=554429,3150041=194585} 2024-09-20T22:45:30.191 INFO:tasks.rados.rados.0.smithi084.stdout:825: writing smithi08477203-10 from 544626 to 1244281 tid 1 2024-09-20T22:45:30.198 INFO:tasks.rados.rados.0.smithi084.stdout:825: writing smithi08477203-10 from 1892215 to 2446644 tid 2 2024-09-20T22:45:30.204 INFO:tasks.rados.rados.0.smithi084.stdout:825: writing smithi08477203-10 from 3150041 to 3344626 tid 3 2024-09-20T22:45:30.204 INFO:tasks.rados.rados.0.smithi084.stdout:823: done (2 left) 2024-09-20T22:45:30.204 INFO:tasks.rados.rados.0.smithi084.stdout:826: rollback oid 13 current snap is 81 2024-09-20T22:45:30.204 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 13 to 67 2024-09-20T22:45:30.204 INFO:tasks.rados.rados.0.smithi084.stdout:824: finishing write tid 1 to smithi08477203-31 2024-09-20T22:45:30.204 INFO:tasks.rados.rados.0.smithi084.stdout:824: finishing write tid 2 to smithi08477203-31 2024-09-20T22:45:30.205 INFO:tasks.rados.rados.0.smithi084.stdout:824: finishing write tid 3 to smithi08477203-31 2024-09-20T22:45:30.205 INFO:tasks.rados.rados.0.smithi084.stdout:824: finishing write tid 4 to smithi08477203-31 2024-09-20T22:45:30.205 INFO:tasks.rados.rados.0.smithi084.stdout:824: finishing write tid 5 to smithi08477203-31 2024-09-20T22:45:30.205 INFO:tasks.rados.rados.0.smithi084.stdout:824: oid 31 updating version 0 to 200 2024-09-20T22:45:30.205 INFO:tasks.rados.rados.0.smithi084.stdout:824: oid 31 version 200 is already newer than 198 2024-09-20T22:45:30.205 INFO:tasks.rados.rados.0.smithi084.stdout:824: oid 31 version 200 is already newer than 197 2024-09-20T22:45:30.205 INFO:tasks.rados.rados.0.smithi084.stdout:824: oid 31 version 200 is already newer than 199 2024-09-20T22:45:30.205 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 31 v 200 (ObjNum 317 snap 81 seq_num 317) dirty exists 2024-09-20T22:45:30.205 INFO:tasks.rados.rados.0.smithi084.stdout:824: left oid 31 (ObjNum 317 snap 81 seq_num 317) 2024-09-20T22:45:30.205 INFO:tasks.rados.rados.0.smithi084.stdout:825: finishing write tid 1 to smithi08477203-10 2024-09-20T22:45:30.205 INFO:tasks.rados.rados.0.smithi084.stdout:825: finishing write tid 2 to smithi08477203-10 2024-09-20T22:45:30.205 INFO:tasks.rados.rados.0.smithi084.stdout:824: done (2 left) 2024-09-20T22:45:30.205 INFO:tasks.rados.rados.0.smithi084.stdout:827: read oid 17 snap -1 2024-09-20T22:45:30.205 INFO:tasks.rados.rados.0.smithi084.stdout:827: expect (ObjNum 315 snap 78 seq_num 315) 2024-09-20T22:45:30.206 INFO:tasks.rados.rados.0.smithi084.stdout:828: read oid 18 snap 79 2024-09-20T22:45:30.206 INFO:tasks.rados.rados.0.smithi084.stdout:828: expect (ObjNum 273 snap 69 seq_num 273) 2024-09-20T22:45:30.206 INFO:tasks.rados.rados.0.smithi084.stdout:829: delete oid 44 current snap is 81 2024-09-20T22:45:30.206 INFO:tasks.rados.rados.0.smithi084.stdout:826: finishing rollback tid 0 to smithi08477203-13 2024-09-20T22:45:30.207 INFO:tasks.rados.rados.0.smithi084.stdout:826: finishing rollback tid 1 to smithi08477203-13 2024-09-20T22:45:30.207 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 13 v 385 (ObjNum 289 snap 0 seq_num 0) dirty dne 2024-09-20T22:45:30.207 INFO:tasks.rados.rados.0.smithi084.stdout:826: done (4 left) 2024-09-20T22:45:30.207 INFO:tasks.rados.rados.0.smithi084.stdout:829: done (3 left) 2024-09-20T22:45:30.207 INFO:tasks.rados.rados.0.smithi084.stdout:830: read oid 12 snap -1 2024-09-20T22:45:30.207 INFO:tasks.rados.rados.0.smithi084.stdout:830: expect (ObjNum 312 snap 77 seq_num 312) 2024-09-20T22:45:30.207 INFO:tasks.rados.rados.0.smithi084.stdout:831: snap_remove snap 72 2024-09-20T22:45:30.248 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:45:30.270 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:30.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:30.271 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:45:30.370 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:30 smithi084 ceph-mon[40024]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:45:30.370 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:30 smithi084 ceph-mon[40024]: osdmap e251: 8 total, 8 up, 7 in 2024-09-20T22:45:30.370 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:30 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/3075469601' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-09-20T22:45:30.370 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:30 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-09-20T22:45:30.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:30 smithi084 ceph-mon[29774]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:45:30.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:30 smithi084 ceph-mon[29774]: osdmap e251: 8 total, 8 up, 7 in 2024-09-20T22:45:30.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:30 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/3075469601' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-09-20T22:45:30.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:30 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-09-20T22:45:30.371 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:45:30.392 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:30.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:30.393 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:45:30.412 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:30 smithi154 ceph-mon[33264]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:45:30.412 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:30 smithi154 ceph-mon[33264]: osdmap e251: 8 total, 8 up, 7 in 2024-09-20T22:45:30.412 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:30 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/3075469601' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-09-20T22:45:30.412 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:30 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-09-20T22:45:30.426 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:45:30.494 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:45:30.514 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-20T22:45:30.514 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [], 2024-09-20T22:45:30.514 INFO:teuthology.orchestra.run.smithi154.stdout: "num_ops": 0 2024-09-20T22:45:30.515 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-20T22:45:30.515 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:30.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:30.516 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:45:30.617 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:45:30.638 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:30.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:30.639 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:45:30.740 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:45:30.761 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:30.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:30.762 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:45:30.863 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:45:30.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:30.885 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:30.886 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:45:30.986 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:45:31.008 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:31.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:31.009 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:45:31.109 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:45:31.130 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:31.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:31.131 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:45:31.164 INFO:tasks.rados.rados.0.smithi084.stdout:831: done (4 left) 2024-09-20T22:45:31.165 INFO:tasks.rados.rados.0.smithi084.stdout:832: snap_create 2024-09-20T22:45:31.165 INFO:tasks.rados.rados.0.smithi084.stdout:825: finishing write tid 3 to smithi08477203-10 2024-09-20T22:45:31.165 INFO:tasks.rados.rados.0.smithi084.stdout:825: finishing write tid 4 to smithi08477203-10 2024-09-20T22:45:31.165 INFO:tasks.rados.rados.0.smithi084.stdout:825: finishing write tid 5 to smithi08477203-10 2024-09-20T22:45:31.165 INFO:tasks.rados.rados.0.smithi084.stdout:825: oid 10 updating version 0 to 183 2024-09-20T22:45:31.165 INFO:tasks.rados.rados.0.smithi084.stdout:825: oid 10 version 183 is already newer than 182 2024-09-20T22:45:31.165 INFO:tasks.rados.rados.0.smithi084.stdout:825: oid 10 updating version 183 to 184 2024-09-20T22:45:31.165 INFO:tasks.rados.rados.0.smithi084.stdout:825: oid 10 updating version 184 to 185 2024-09-20T22:45:31.165 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 10 v 185 (ObjNum 318 snap 81 seq_num 318) dirty exists 2024-09-20T22:45:31.165 INFO:tasks.rados.rados.0.smithi084.stdout:825: left oid 10 (ObjNum 318 snap 81 seq_num 318) 2024-09-20T22:45:31.165 INFO:tasks.rados.rados.0.smithi084.stdout:828: expect (ObjNum 273 snap 69 seq_num 273) 2024-09-20T22:45:31.166 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is set 2024-09-20T22:45:31.166 INFO:teuthology.orchestra.run.smithi084.stderr:set pool 1 pg_num to 11 2024-09-20T22:45:31.232 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:45:31.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:31.253 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:31.254 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:45:31.287 INFO:tasks.rados.rados.0.smithi084.stdout:827: expect (ObjNum 315 snap 78 seq_num 315) 2024-09-20T22:45:31.311 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.4 dump_blocked_ops 2024-09-20T22:45:31.355 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:45:31.376 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:31.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:31.377 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:45:31.417 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:31 smithi084 ceph-mon[40024]: pgmap v277: 178 pgs: 178 active+clean; 175 MiB data, 668 MiB used, 625 GiB / 626 GiB avail; 13 MiB/s rd, 936 KiB/s wr, 16 op/s 2024-09-20T22:45:31.418 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:31 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/3618561712' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:45:31.418 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:31 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:45:31.418 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:31 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-09-20T22:45:31.418 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:31 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T22:45:31.418 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:31 smithi084 ceph-mon[40024]: osdmap e252: 8 total, 8 up, 7 in 2024-09-20T22:45:31.418 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:31 smithi084 ceph-mon[29774]: pgmap v277: 178 pgs: 178 active+clean; 175 MiB data, 668 MiB used, 625 GiB / 626 GiB avail; 13 MiB/s rd, 936 KiB/s wr, 16 op/s 2024-09-20T22:45:31.418 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:31 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/3618561712' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:45:31.418 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:31 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:45:31.418 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:31 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-09-20T22:45:31.419 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:31 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T22:45:31.419 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:31 smithi084 ceph-mon[29774]: osdmap e252: 8 total, 8 up, 7 in 2024-09-20T22:45:31.478 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:45:31.500 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:31.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:31.501 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:45:31.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:31 smithi154 ceph-mon[33264]: pgmap v277: 178 pgs: 178 active+clean; 175 MiB data, 668 MiB used, 625 GiB / 626 GiB avail; 13 MiB/s rd, 936 KiB/s wr, 16 op/s 2024-09-20T22:45:31.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:31 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/3618561712' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:45:31.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:31 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:45:31.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:31 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-09-20T22:45:31.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:31 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T22:45:31.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:31 smithi154 ceph-mon[33264]: osdmap e252: 8 total, 8 up, 7 in 2024-09-20T22:45:31.534 INFO:tasks.rados.rados.0.smithi084.stdout:830: expect (ObjNum 312 snap 77 seq_num 312) 2024-09-20T22:45:31.558 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:45:31.602 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:45:31.623 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:31.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:31.624 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:45:31.725 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:45:31.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:31.747 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:31.747 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:45:31.848 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:45:31.870 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:31.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:31.871 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:45:31.971 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:45:31.992 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:31.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:31.993 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:45:32.044 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-20T22:45:32.045 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [], 2024-09-20T22:45:32.045 INFO:teuthology.orchestra.run.smithi154.stdout: "complaint_time": 30, 2024-09-20T22:45:32.045 INFO:teuthology.orchestra.run.smithi154.stdout: "num_blocked_ops": 0 2024-09-20T22:45:32.045 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-20T22:45:32.094 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:45:32.116 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:32.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:32.117 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:45:32.170 INFO:tasks.rados.rados.0.smithi084.stdout:825: done (4 left) 2024-09-20T22:45:32.170 INFO:tasks.rados.rados.0.smithi084.stdout:827: done (3 left) 2024-09-20T22:45:32.171 INFO:tasks.rados.rados.0.smithi084.stdout:828: done (2 left) 2024-09-20T22:45:32.171 INFO:tasks.rados.rados.0.smithi084.stdout:830: done (1 left) 2024-09-20T22:45:32.171 INFO:tasks.rados.rados.0.smithi084.stdout:832: done (0 left) 2024-09-20T22:45:32.171 INFO:tasks.rados.rados.0.smithi084.stdout:833: write (excl) oid 46 current snap is 82 2024-09-20T22:45:32.171 INFO:tasks.rados.rados.0.smithi084.stdout:833: seq_num 320 ranges {578240=566506,1609179=558098,2804887=551750} 2024-09-20T22:45:32.177 INFO:tasks.rados.rados.0.smithi084.stdout:833: writing smithi08477203-46 from 578240 to 1144746 tid 1 2024-09-20T22:45:32.182 INFO:tasks.rados.rados.0.smithi084.stdout:833: writing smithi08477203-46 from 1609179 to 2167277 tid 2 2024-09-20T22:45:32.189 INFO:tasks.rados.rados.0.smithi084.stdout:833: writing smithi08477203-46 from 2804887 to 3356637 tid 3 2024-09-20T22:45:32.189 INFO:tasks.rados.rados.0.smithi084.stdout:834: copy_from oid 23 from oid 45 current snap is 82 2024-09-20T22:45:32.189 INFO:tasks.rados.rados.0.smithi084.stdout:833: finishing write tid 1 to smithi08477203-46 2024-09-20T22:45:32.189 INFO:tasks.rados.rados.0.smithi084.stdout:833: finishing write tid 2 to smithi08477203-46 2024-09-20T22:45:32.190 INFO:tasks.rados.rados.0.smithi084.stdout:835: snap_remove snap 62 2024-09-20T22:45:32.218 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:45:32.239 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:32.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:32.240 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:45:32.340 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:45:32.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:32.362 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:32.362 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:45:32.400 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-20T22:45:32.425 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:32 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-09-20T22:45:32.425 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:32 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-09-20T22:45:32.425 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:32 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-09-20T22:45:32.425 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:32 smithi084 ceph-mon[40024]: osdmap e253: 8 total, 8 up, 7 in 2024-09-20T22:45:32.425 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:32 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-09-20T22:45:32.425 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:32 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-09-20T22:45:32.426 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:32 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-09-20T22:45:32.426 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:32 smithi084 ceph-mon[29774]: osdmap e253: 8 total, 8 up, 7 in 2024-09-20T22:45:32.463 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:45:32.484 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:32.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:32.485 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:45:32.586 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:45:32.608 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:32.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:32.609 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:45:32.653 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:45:32.709 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:45:32.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:32.732 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:32.732 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:45:32.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:32 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-09-20T22:45:32.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:32 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-09-20T22:45:32.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:32 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-09-20T22:45:32.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:32 smithi154 ceph-mon[33264]: osdmap e253: 8 total, 8 up, 7 in 2024-09-20T22:45:32.833 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:45:32.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:32.855 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:32.855 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:45:32.895 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.4 dump_historic_ops 2024-09-20T22:45:32.956 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:45:32.999 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:33.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:33.000 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:45:33.101 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:45:33.123 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:33.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:33.124 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:45:33.141 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:45:33.142 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:45:33.157 INFO:tasks.rados.rados.0.smithi084.stdout:835: done (2 left) 2024-09-20T22:45:33.157 INFO:tasks.rados.rados.0.smithi084.stdout:836: rollback oid 10 current snap is 82 2024-09-20T22:45:33.157 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 10 to 69 2024-09-20T22:45:33.157 INFO:tasks.rados.rados.0.smithi084.stdout:837: write (excl) oid 21 current snap is 82 2024-09-20T22:45:33.157 INFO:tasks.rados.rados.0.smithi084.stdout:837: seq_num 321 ranges {595047=641755,1667661=418109,2548663=763022} 2024-09-20T22:45:33.161 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:45:33.161 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:45:33.164 INFO:tasks.rados.rados.0.smithi084.stdout:837: writing smithi08477203-21 from 595047 to 1236802 tid 1 2024-09-20T22:45:33.170 INFO:tasks.rados.rados.0.smithi084.stdout:837: writing smithi08477203-21 from 1667661 to 2085770 tid 2 2024-09-20T22:45:33.178 INFO:tasks.rados.rados.0.smithi084.stdout:837: writing smithi08477203-21 from 2548663 to 3311685 tid 3 2024-09-20T22:45:33.181 INFO:tasks.rados.rados.0.smithi084.stdout:838: copy_from oid 15 from oid 22 current snap is 82 2024-09-20T22:45:33.181 INFO:tasks.rados.rados.0.smithi084.stdout:834: finishing copy_from to smithi08477203-23 2024-09-20T22:45:33.181 INFO:tasks.rados.rados.0.smithi084.stdout:834: got expected ENOENT (src dne) 2024-09-20T22:45:33.181 INFO:tasks.rados.rados.0.smithi084.stdout:834: finishing copy_from racing read to smithi08477203-23 2024-09-20T22:45:33.181 INFO:tasks.rados.rados.0.smithi084.stdout:833: finishing write tid 3 to smithi08477203-46 2024-09-20T22:45:33.182 INFO:tasks.rados.rados.0.smithi084.stdout:833: finishing write tid 4 to smithi08477203-46 2024-09-20T22:45:33.182 INFO:tasks.rados.rados.0.smithi084.stdout:833: finishing write tid 5 to smithi08477203-46 2024-09-20T22:45:33.182 INFO:tasks.rados.rados.0.smithi084.stdout:833: oid 46 updating version 0 to 151 2024-09-20T22:45:33.182 INFO:tasks.rados.rados.0.smithi084.stdout:833: oid 46 updating version 151 to 153 2024-09-20T22:45:33.182 INFO:tasks.rados.rados.0.smithi084.stdout:833: oid 46 version 153 is already newer than 152 2024-09-20T22:45:33.182 INFO:tasks.rados.rados.0.smithi084.stdout:833: oid 46 version 153 is already newer than 150 2024-09-20T22:45:33.183 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 46 v 153 (ObjNum 320 snap 82 seq_num 320) dirty exists 2024-09-20T22:45:33.183 INFO:tasks.rados.rados.0.smithi084.stdout:833: left oid 46 (ObjNum 320 snap 82 seq_num 320) 2024-09-20T22:45:33.183 INFO:tasks.rados.rados.0.smithi084.stdout:833: done (4 left) 2024-09-20T22:45:33.183 INFO:tasks.rados.rados.0.smithi084.stdout:834: done (3 left) 2024-09-20T22:45:33.183 INFO:tasks.rados.rados.0.smithi084.stdout:839: read oid 8 snap 81 2024-09-20T22:45:33.183 INFO:tasks.rados.rados.0.smithi084.stdout:839: expect (ObjNum 276 snap 69 seq_num 276) 2024-09-20T22:45:33.183 INFO:tasks.rados.rados.0.smithi084.stdout:840: rollback oid 39 current snap is 82 2024-09-20T22:45:33.184 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 39 to 73 2024-09-20T22:45:33.184 INFO:tasks.rados.rados.0.smithi084.stdout:836: finishing rollback tid 0 to smithi08477203-10 2024-09-20T22:45:33.184 INFO:tasks.rados.rados.0.smithi084.stdout:836: finishing rollback tid 1 to smithi08477203-10 2024-09-20T22:45:33.184 INFO:tasks.rados.rados.0.smithi084.stdout:836: finishing rollback tid 2 to smithi08477203-10 2024-09-20T22:45:33.185 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 10 v 191 (ObjNum 271 snap 69 seq_num 271) dirty exists 2024-09-20T22:45:33.185 INFO:tasks.rados.rados.0.smithi084.stdout:836: done (4 left) 2024-09-20T22:45:33.185 INFO:tasks.rados.rados.0.smithi084.stdout:841: read oid 45 snap -1 2024-09-20T22:45:33.185 INFO:tasks.rados.rados.0.smithi084.stdout:841: expect deleted 2024-09-20T22:45:33.185 INFO:tasks.rados.rados.0.smithi084.stdout:842: write (excl) oid 35 current snap is 82 2024-09-20T22:45:33.185 INFO:tasks.rados.rados.0.smithi084.stdout:842: seq_num 322 ranges {611854=717004,2126143=561767,3259733=152121} 2024-09-20T22:45:33.187 INFO:tasks.rados.rados.0.smithi084.stdout:842: writing smithi08477203-35 from 611854 to 1328858 tid 1 2024-09-20T22:45:33.195 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:45:33.195 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:45:33.196 INFO:tasks.rados.rados.0.smithi084.stdout:842: writing smithi08477203-35 from 2126143 to 2687910 tid 2 2024-09-20T22:45:33.197 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:45:33.200 INFO:tasks.rados.rados.0.smithi084.stdout:842: writing smithi08477203-35 from 3259733 to 3411854 tid 3 2024-09-20T22:45:33.200 INFO:tasks.rados.rados.0.smithi084.stdout:843: write oid 44 current snap is 82 2024-09-20T22:45:33.200 INFO:tasks.rados.rados.0.smithi084.stdout:843: seq_num 323 ranges {628661=792253,2184625=421778,3403509=25152} 2024-09-20T22:45:33.209 INFO:tasks.rados.rados.0.smithi084.stdout:843: writing smithi08477203-44 from 628661 to 1420914 tid 1 2024-09-20T22:45:33.217 INFO:tasks.rados.rados.0.smithi084.stdout:843: writing smithi08477203-44 from 2184625 to 2606403 tid 2 2024-09-20T22:45:33.222 INFO:tasks.rados.rados.0.smithi084.stdout:843: writing smithi08477203-44 from 3403509 to 3428661 tid 3 2024-09-20T22:45:33.222 INFO:tasks.rados.rados.0.smithi084.stdout:844: snap_create 2024-09-20T22:45:33.222 INFO:tasks.rados.rados.0.smithi084.stdout:840: finishing rollback tid 0 to smithi08477203-39 2024-09-20T22:45:33.222 INFO:tasks.rados.rados.0.smithi084.stdout:839: expect (ObjNum 276 snap 69 seq_num 276) 2024-09-20T22:45:33.223 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-20T22:45:33.223 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T22:45:33.225 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:45:33.240 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:45:33.240 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:45:33.246 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:33.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:33.247 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:45:33.257 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:45:33.257 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:45:33.273 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:45:33.273 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:45:33.291 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:45:33.291 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:45:33.300 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:33 smithi084 ceph-mon[40024]: pgmap v280: 178 pgs: 178 active+clean; 175 MiB data, 668 MiB used, 625 GiB / 626 GiB avail; 13 MiB/s rd, 936 KiB/s wr, 16 op/s 2024-09-20T22:45:33.300 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:33 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/3220423549' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:45:33.300 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:33 smithi084 ceph-mon[40024]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:45:33.300 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:33 smithi084 ceph-mon[40024]: osdmap e254: 8 total, 8 up, 7 in 2024-09-20T22:45:33.301 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:33 smithi084 ceph-mon[29774]: pgmap v280: 178 pgs: 178 active+clean; 175 MiB data, 668 MiB used, 625 GiB / 626 GiB avail; 13 MiB/s rd, 936 KiB/s wr, 16 op/s 2024-09-20T22:45:33.301 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:33 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/3220423549' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:45:33.301 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:33 smithi084 ceph-mon[29774]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:45:33.301 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:33 smithi084 ceph-mon[29774]: osdmap e254: 8 total, 8 up, 7 in 2024-09-20T22:45:33.346 INFO:tasks.rados.rados.0.smithi084.stdout:840: finishing rollback tid 1 to smithi08477203-39 2024-09-20T22:45:33.347 INFO:tasks.rados.rados.0.smithi084.stdout:840: finishing rollback tid 2 to smithi08477203-39 2024-09-20T22:45:33.347 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 39 v 199 (ObjNum 245 snap 54 seq_num 245) dirty exists 2024-09-20T22:45:33.347 INFO:tasks.rados.rados.0.smithi084.stdout:837: finishing write tid 1 to smithi08477203-21 2024-09-20T22:45:33.347 INFO:tasks.rados.rados.0.smithi084.stdout:837: finishing write tid 2 to smithi08477203-21 2024-09-20T22:45:33.347 INFO:tasks.rados.rados.0.smithi084.stdout:837: finishing write tid 3 to smithi08477203-21 2024-09-20T22:45:33.347 INFO:tasks.rados.rados.0.smithi084.stdout:837: finishing write tid 4 to smithi08477203-21 2024-09-20T22:45:33.347 INFO:tasks.rados.rados.0.smithi084.stdout:837: finishing write tid 5 to smithi08477203-21 2024-09-20T22:45:33.347 INFO:tasks.rados.rados.0.smithi084.stdout:837: oid 21 updating version 0 to 237 2024-09-20T22:45:33.347 INFO:tasks.rados.rados.0.smithi084.stdout:837: oid 21 updating version 237 to 239 2024-09-20T22:45:33.347 INFO:tasks.rados.rados.0.smithi084.stdout:837: oid 21 updating version 239 to 240 2024-09-20T22:45:33.347 INFO:tasks.rados.rados.0.smithi084.stdout:837: oid 21 version 240 is already newer than 238 2024-09-20T22:45:33.347 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 21 v 240 (ObjNum 321 snap 82 seq_num 321) dirty exists 2024-09-20T22:45:33.348 INFO:tasks.rados.rados.0.smithi084.stdout:837: left oid 21 (ObjNum 321 snap 82 seq_num 321) 2024-09-20T22:45:33.348 INFO:tasks.rados.rados.0.smithi084.stdout:838: finishing copy_from racing read to smithi08477203-15 2024-09-20T22:45:33.348 INFO:tasks.rados.rados.0.smithi084.stdout:842: finishing write tid 1 to smithi08477203-35 2024-09-20T22:45:33.348 INFO:tasks.rados.rados.0.smithi084.stdout:838: finishing copy_from to smithi08477203-15 2024-09-20T22:45:33.348 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 15 v 166 (ObjNum 307 snap 75 seq_num 307) dirty exists 2024-09-20T22:45:33.348 INFO:tasks.rados.rados.0.smithi084.stdout:842: finishing write tid 2 to smithi08477203-35 2024-09-20T22:45:33.348 INFO:tasks.rados.rados.0.smithi084.stdout:842: finishing write tid 3 to smithi08477203-35 2024-09-20T22:45:33.348 INFO:tasks.rados.rados.0.smithi084.stdout:842: finishing write tid 4 to smithi08477203-35 2024-09-20T22:45:33.348 INFO:tasks.rados.rados.0.smithi084.stdout:842: finishing write tid 5 to smithi08477203-35 2024-09-20T22:45:33.348 INFO:tasks.rados.rados.0.smithi084.stdout:842: oid 35 updating version 0 to 156 2024-09-20T22:45:33.348 INFO:tasks.rados.rados.0.smithi084.stdout:842: oid 35 updating version 156 to 158 2024-09-20T22:45:33.348 INFO:tasks.rados.rados.0.smithi084.stdout:842: oid 35 updating version 158 to 159 2024-09-20T22:45:33.348 INFO:tasks.rados.rados.0.smithi084.stdout:842: oid 35 version 159 is already newer than 157 2024-09-20T22:45:33.348 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 35 v 159 (ObjNum 322 snap 82 seq_num 322) dirty exists 2024-09-20T22:45:33.349 INFO:tasks.rados.rados.0.smithi084.stdout:842: left oid 35 (ObjNum 322 snap 82 seq_num 322) 2024-09-20T22:45:33.349 INFO:tasks.rados.rados.0.smithi084.stdout:843: finishing write tid 1 to smithi08477203-44 2024-09-20T22:45:33.349 INFO:tasks.rados.rados.0.smithi084.stdout:843: finishing write tid 2 to smithi08477203-44 2024-09-20T22:45:33.349 INFO:tasks.rados.rados.0.smithi084.stdout:843: finishing write tid 3 to smithi08477203-44 2024-09-20T22:45:33.349 INFO:tasks.rados.rados.0.smithi084.stdout:843: finishing write tid 4 to smithi08477203-44 2024-09-20T22:45:33.349 INFO:tasks.rados.rados.0.smithi084.stdout:843: finishing write tid 5 to smithi08477203-44 2024-09-20T22:45:33.349 INFO:tasks.rados.rados.0.smithi084.stdout:843: oid 44 updating version 0 to 406 2024-09-20T22:45:33.349 INFO:tasks.rados.rados.0.smithi084.stdout:843: oid 44 updating version 406 to 409 2024-09-20T22:45:33.349 INFO:tasks.rados.rados.0.smithi084.stdout:843: oid 44 version 409 is already newer than 408 2024-09-20T22:45:33.349 INFO:tasks.rados.rados.0.smithi084.stdout:843: oid 44 version 409 is already newer than 407 2024-09-20T22:45:33.349 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 44 v 409 (ObjNum 323 snap 82 seq_num 323) dirty exists 2024-09-20T22:45:33.349 INFO:tasks.rados.rados.0.smithi084.stdout:843: left oid 44 (ObjNum 323 snap 82 seq_num 323) 2024-09-20T22:45:33.350 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:45:33.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:33.372 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:33.372 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:45:33.473 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:45:33.494 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:33.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:33.494 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:45:33.595 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:45:33.599 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:33 smithi154 ceph-mon[33264]: pgmap v280: 178 pgs: 178 active+clean; 175 MiB data, 668 MiB used, 625 GiB / 626 GiB avail; 13 MiB/s rd, 936 KiB/s wr, 16 op/s 2024-09-20T22:45:33.599 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:33 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/3220423549' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:45:33.599 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:33 smithi154 ceph-mon[33264]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:45:33.599 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:33 smithi154 ceph-mon[33264]: osdmap e254: 8 total, 8 up, 7 in 2024-09-20T22:45:33.617 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:33.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:33.618 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:45:33.710 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-20T22:45:33.711 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:45:33.718 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:45:33.740 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:33.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:33.740 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:45:33.841 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:45:33.863 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:33.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:33.864 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:45:33.965 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:45:33.986 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:33.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:33.987 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:45:34.087 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:45:34.109 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:34.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:34.110 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:45:34.162 INFO:tasks.rados.rados.0.smithi084.stdout:837: done (7 left) 2024-09-20T22:45:34.162 INFO:tasks.rados.rados.0.smithi084.stdout:838: done (6 left) 2024-09-20T22:45:34.162 INFO:tasks.rados.rados.0.smithi084.stdout:839: done (5 left) 2024-09-20T22:45:34.162 INFO:tasks.rados.rados.0.smithi084.stdout:840: done (4 left) 2024-09-20T22:45:34.162 INFO:tasks.rados.rados.0.smithi084.stdout:841: done (3 left) 2024-09-20T22:45:34.162 INFO:tasks.rados.rados.0.smithi084.stdout:842: done (2 left) 2024-09-20T22:45:34.162 INFO:tasks.rados.rados.0.smithi084.stdout:843: done (1 left) 2024-09-20T22:45:34.162 INFO:tasks.rados.rados.0.smithi084.stdout:844: done (0 left) 2024-09-20T22:45:34.162 INFO:tasks.rados.rados.0.smithi084.stdout:845: read oid 1 snap -1 2024-09-20T22:45:34.162 INFO:tasks.rados.rados.0.smithi084.stdout:845: expect (ObjNum 284 snap 71 seq_num 284) 2024-09-20T22:45:34.162 INFO:tasks.rados.rados.0.smithi084.stdout:846: read oid 37 snap 60 2024-09-20T22:45:34.162 INFO:tasks.rados.rados.0.smithi084.stdout:846: expect (ObjNum 253 snap 59 seq_num 253) 2024-09-20T22:45:34.163 INFO:tasks.rados.rados.0.smithi084.stdout:847: read oid 42 snap -1 2024-09-20T22:45:34.163 INFO:tasks.rados.rados.0.smithi084.stdout:847: expect deleted 2024-09-20T22:45:34.163 INFO:tasks.rados.rados.0.smithi084.stdout:848: copy_from oid 50 from oid 29 current snap is 83 2024-09-20T22:45:34.163 INFO:tasks.rados.rados.0.smithi084.stdout:849: rollback oid 32 current snap is 83 2024-09-20T22:45:34.163 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 32 to 69 2024-09-20T22:45:34.163 INFO:tasks.rados.rados.0.smithi084.stdout:850: read oid 40 snap -1 2024-09-20T22:45:34.163 INFO:tasks.rados.rados.0.smithi084.stdout:850: expect (ObjNum 316 snap 80 seq_num 316) 2024-09-20T22:45:34.163 INFO:tasks.rados.rados.0.smithi084.stdout:851: snap_remove snap 82 2024-09-20T22:45:34.180 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd unset noscrub 2024-09-20T22:45:34.210 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:45:34.244 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:34.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:34.245 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:45:34.346 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:45:34.368 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:34.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:34.369 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:45:34.430 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:45:34.470 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:45:34.491 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-20T22:45:34.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:34.494 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:34.494 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:45:34.595 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:45:34.617 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:34.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:34.618 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:45:34.718 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:45:34.739 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:45:34.740 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:34.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:34.741 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:45:34.842 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:45:34.863 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:34.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:34.864 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:45:34.965 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:45:34.986 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:34.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:34.987 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:45:35.088 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:45:35.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:35.109 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:35.109 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:45:35.171 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is unset 2024-09-20T22:45:35.171 INFO:tasks.rados.rados.0.smithi084.stdout:851: done (6 left) 2024-09-20T22:45:35.171 INFO:tasks.rados.rados.0.smithi084.stdout:852: rollback oid 9 current snap is 83 2024-09-20T22:45:35.171 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 9 to 78 2024-09-20T22:45:35.171 INFO:tasks.rados.rados.0.smithi084.stdout:848: finishing copy_from to smithi08477203-50 2024-09-20T22:45:35.171 INFO:tasks.rados.rados.0.smithi084.stdout:848: got expected ENOENT (src dne) 2024-09-20T22:45:35.171 INFO:tasks.rados.rados.0.smithi084.stdout:848: finishing copy_from racing read to smithi08477203-50 2024-09-20T22:45:35.171 INFO:tasks.rados.rados.0.smithi084.stdout:845: expect (ObjNum 284 snap 71 seq_num 284) 2024-09-20T22:45:35.211 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:45:35.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:35.232 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:35.232 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:45:35.240 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-20T22:45:35.240 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:45:35.333 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:45:35.353 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:35.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:35.354 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:45:35.387 INFO:tasks.rados.rados.0.smithi084.stdout:850: expect (ObjNum 316 snap 80 seq_num 316) 2024-09-20T22:45:35.455 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:45:35.471 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:35 smithi084 ceph-mon[40024]: pgmap v283: 178 pgs: 3 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 10 remapped+peering, 161 active+clean; 198 MiB data, 696 MiB used, 625 GiB / 626 GiB avail; 7.5 MiB/s rd, 3.3 MiB/s wr, 31 op/s; 39 KiB/s, 0 objects/s recovering 2024-09-20T22:45:35.471 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:35 smithi084 ceph-mon[40024]: osdmap e255: 8 total, 8 up, 7 in 2024-09-20T22:45:35.471 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:35 smithi084 ceph-mon[40024]: Health check failed: Reduced data availability: 4 pgs inactive, 6 pgs peering (PG_AVAILABILITY) 2024-09-20T22:45:35.471 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:35 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/476319430' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:45:35.472 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:35 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:45:35.472 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:35 smithi084 ceph-mon[29774]: pgmap v283: 178 pgs: 3 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 10 remapped+peering, 161 active+clean; 198 MiB data, 696 MiB used, 625 GiB / 626 GiB avail; 7.5 MiB/s rd, 3.3 MiB/s wr, 31 op/s; 39 KiB/s, 0 objects/s recovering 2024-09-20T22:45:35.472 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:35 smithi084 ceph-mon[29774]: osdmap e255: 8 total, 8 up, 7 in 2024-09-20T22:45:35.472 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:35 smithi084 ceph-mon[29774]: Health check failed: Reduced data availability: 4 pgs inactive, 6 pgs peering (PG_AVAILABILITY) 2024-09-20T22:45:35.472 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:35 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/476319430' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:45:35.472 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:35 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:45:35.476 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:35.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:35.477 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:45:35.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:35 smithi154 ceph-mon[33264]: pgmap v283: 178 pgs: 3 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 10 remapped+peering, 161 active+clean; 198 MiB data, 696 MiB used, 625 GiB / 626 GiB avail; 7.5 MiB/s rd, 3.3 MiB/s wr, 31 op/s; 39 KiB/s, 0 objects/s recovering 2024-09-20T22:45:35.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:35 smithi154 ceph-mon[33264]: osdmap e255: 8 total, 8 up, 7 in 2024-09-20T22:45:35.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:35 smithi154 ceph-mon[33264]: Health check failed: Reduced data availability: 4 pgs inactive, 6 pgs peering (PG_AVAILABILITY) 2024-09-20T22:45:35.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:35 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/476319430' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:45:35.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:35 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:45:35.578 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:45:35.600 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:35.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:35.602 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:45:35.621 INFO:tasks.rados.rados.0.smithi084.stdout:846: expect (ObjNum 253 snap 59 seq_num 253) 2024-09-20T22:45:35.702 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:45:35.724 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:35.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:35.725 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:45:35.752 INFO:tasks.rados.rados.0.smithi084.stdout:849: finishing rollback tid 1 to smithi08477203-32 2024-09-20T22:45:35.752 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 32 v 0 (ObjNum 2090720704 snap 21870 seq_num 304) dirty dne 2024-09-20T22:45:35.752 INFO:tasks.rados.rados.0.smithi084.stdout:852: finishing rollback tid 1 to smithi08477203-9 2024-09-20T22:45:35.752 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 9 v 0 (ObjNum 129 snap 0 seq_num 803183952) dirty dne 2024-09-20T22:45:35.752 INFO:tasks.rados.rados.0.smithi084.stdout:845: done (6 left) 2024-09-20T22:45:35.752 INFO:tasks.rados.rados.0.smithi084.stdout:846: done (5 left) 2024-09-20T22:45:35.752 INFO:tasks.rados.rados.0.smithi084.stdout:847: done (4 left) 2024-09-20T22:45:35.752 INFO:tasks.rados.rados.0.smithi084.stdout:848: done (3 left) 2024-09-20T22:45:35.752 INFO:tasks.rados.rados.0.smithi084.stdout:849: done (2 left) 2024-09-20T22:45:35.752 INFO:tasks.rados.rados.0.smithi084.stdout:850: done (1 left) 2024-09-20T22:45:35.752 INFO:tasks.rados.rados.0.smithi084.stdout:852: done (0 left) 2024-09-20T22:45:35.752 INFO:tasks.rados.rados.0.smithi084.stdout:853: rollback oid 44 current snap is 83 2024-09-20T22:45:35.753 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 44 to 67 2024-09-20T22:45:35.753 INFO:tasks.rados.rados.0.smithi084.stdout:854: read oid 27 snap -1 2024-09-20T22:45:35.753 INFO:tasks.rados.rados.0.smithi084.stdout:854: expect deleted 2024-09-20T22:45:35.753 INFO:tasks.rados.rados.0.smithi084.stdout:855: write oid 2 current snap is 83 2024-09-20T22:45:35.753 INFO:tasks.rados.rados.0.smithi084.stdout:855: seq_num 324 ranges {645468=467502,1726754=565436,2798226=596838} 2024-09-20T22:45:35.757 INFO:tasks.rados.rados.0.smithi084.stdout:855: writing smithi08477203-2 from 645468 to 1112970 tid 1 2024-09-20T22:45:35.764 INFO:tasks.rados.rados.0.smithi084.stdout:855: writing smithi08477203-2 from 1726754 to 2292190 tid 2 2024-09-20T22:45:35.771 INFO:tasks.rados.rados.0.smithi084.stdout:855: writing smithi08477203-2 from 2798226 to 3395064 tid 3 2024-09-20T22:45:35.771 INFO:tasks.rados.rados.0.smithi084.stdout:856: read oid 11 snap -1 2024-09-20T22:45:35.771 INFO:tasks.rados.rados.0.smithi084.stdout:856: expect (ObjNum 267 snap 65 seq_num 267) 2024-09-20T22:45:35.772 INFO:tasks.rados.rados.0.smithi084.stdout:857: read oid 49 snap 60 2024-09-20T22:45:35.772 INFO:tasks.rados.rados.0.smithi084.stdout:857: expect (ObjNum 242 snap 54 seq_num 242) 2024-09-20T22:45:35.772 INFO:tasks.rados.rados.0.smithi084.stdout:858: snap_create 2024-09-20T22:45:35.772 INFO:tasks.rados.rados.0.smithi084.stdout:853: finishing rollback tid 0 to smithi08477203-44 2024-09-20T22:45:35.772 INFO:tasks.rados.rados.0.smithi084.stdout:853: finishing rollback tid 1 to smithi08477203-44 2024-09-20T22:45:35.772 INFO:tasks.rados.rados.0.smithi084.stdout:853: finishing rollback tid 2 to smithi08477203-44 2024-09-20T22:45:35.772 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 44 v 412 (ObjNum 129 snap 23 seq_num 129) dirty exists 2024-09-20T22:45:35.772 INFO:tasks.rados.rados.0.smithi084.stdout:855: finishing write tid 1 to smithi08477203-2 2024-09-20T22:45:35.777 INFO:tasks.rados.rados.0.smithi084.stdout:856: expect (ObjNum 267 snap 65 seq_num 267) 2024-09-20T22:45:35.825 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:45:35.846 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:35.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:35.847 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:45:35.904 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.5 dump_blocked_ops 2024-09-20T22:45:35.934 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-20T22:45:35.948 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:45:35.966 INFO:tasks.rados.rados.0.smithi084.stdout:855: finishing write tid 2 to smithi08477203-2 2024-09-20T22:45:35.967 INFO:tasks.rados.rados.0.smithi084.stdout:855: finishing write tid 3 to smithi08477203-2 2024-09-20T22:45:35.967 INFO:tasks.rados.rados.0.smithi084.stdout:855: finishing write tid 4 to smithi08477203-2 2024-09-20T22:45:35.967 INFO:tasks.rados.rados.0.smithi084.stdout:855: finishing write tid 5 to smithi08477203-2 2024-09-20T22:45:35.967 INFO:tasks.rados.rados.0.smithi084.stdout:855: oid 2 updating version 0 to 101 2024-09-20T22:45:35.967 INFO:tasks.rados.rados.0.smithi084.stdout:855: oid 2 updating version 101 to 103 2024-09-20T22:45:35.967 INFO:tasks.rados.rados.0.smithi084.stdout:855: oid 2 version 103 is already newer than 102 2024-09-20T22:45:35.967 INFO:tasks.rados.rados.0.smithi084.stdout:855: oid 2 version 103 is already newer than 100 2024-09-20T22:45:35.967 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 2 v 103 (ObjNum 324 snap 83 seq_num 324) dirty exists 2024-09-20T22:45:35.967 INFO:tasks.rados.rados.0.smithi084.stdout:855: left oid 2 (ObjNum 324 snap 83 seq_num 324) 2024-09-20T22:45:35.967 INFO:tasks.rados.rados.0.smithi084.stdout:857: expect (ObjNum 242 snap 54 seq_num 242) 2024-09-20T22:45:35.988 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:35.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:35.989 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:45:36.090 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:45:36.113 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:36.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:36.114 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:45:36.150 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:45:36.176 INFO:tasks.rados.rados.0.smithi084.stdout:853: done (5 left) 2024-09-20T22:45:36.176 INFO:tasks.rados.rados.0.smithi084.stdout:854: done (4 left) 2024-09-20T22:45:36.176 INFO:tasks.rados.rados.0.smithi084.stdout:855: done (3 left) 2024-09-20T22:45:36.176 INFO:tasks.rados.rados.0.smithi084.stdout:856: done (2 left) 2024-09-20T22:45:36.176 INFO:tasks.rados.rados.0.smithi084.stdout:857: done (1 left) 2024-09-20T22:45:36.176 INFO:tasks.rados.rados.0.smithi084.stdout:858: done (0 left) 2024-09-20T22:45:36.176 INFO:tasks.rados.rados.0.smithi084.stdout:859: read oid 34 snap -1 2024-09-20T22:45:36.176 INFO:tasks.rados.rados.0.smithi084.stdout:859: expect deleted 2024-09-20T22:45:36.176 INFO:tasks.rados.rados.0.smithi084.stdout:860: read oid 35 snap 67 2024-09-20T22:45:36.176 INFO:tasks.rados.rados.0.smithi084.stdout:860: expect (ObjNum 150 snap 30 seq_num 150) 2024-09-20T22:45:36.176 INFO:tasks.rados.rados.0.smithi084.stdout:861: delete oid 24 current snap is 84 2024-09-20T22:45:36.185 INFO:tasks.rados.rados.0.smithi084.stdout:860: expect (ObjNum 150 snap 30 seq_num 150) 2024-09-20T22:45:36.193 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:45:36.215 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:45:36.236 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:36.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:36.237 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:45:36.318 INFO:tasks.rados.rados.0.smithi084.stdout:859: done (2 left) 2024-09-20T22:45:36.318 INFO:tasks.rados.rados.0.smithi084.stdout:860: done (1 left) 2024-09-20T22:45:36.318 INFO:tasks.rados.rados.0.smithi084.stdout:861: done (0 left) 2024-09-20T22:45:36.318 INFO:tasks.rados.rados.0.smithi084.stdout:862: rollback oid 37 current snap is 84 2024-09-20T22:45:36.318 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 37 to 81 2024-09-20T22:45:36.318 INFO:tasks.rados.rados.0.smithi084.stdout:863: read oid 19 snap 67 2024-09-20T22:45:36.318 INFO:tasks.rados.rados.0.smithi084.stdout:863: expect (ObjNum 258 snap 60 seq_num 258) 2024-09-20T22:45:36.318 INFO:tasks.rados.rados.0.smithi084.stdout:864: read oid 34 snap -1 2024-09-20T22:45:36.318 INFO:tasks.rados.rados.0.smithi084.stdout:864: expect deleted 2024-09-20T22:45:36.319 INFO:tasks.rados.rados.0.smithi084.stdout:865: copy_from oid 31 from oid 50 current snap is 84 2024-09-20T22:45:36.319 INFO:tasks.rados.rados.0.smithi084.stdout:866: rollback oid 7 current snap is 84 2024-09-20T22:45:36.319 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 7 to 73 2024-09-20T22:45:36.319 INFO:tasks.rados.rados.0.smithi084.stdout:867: read oid 27 snap -1 2024-09-20T22:45:36.319 INFO:tasks.rados.rados.0.smithi084.stdout:867: expect deleted 2024-09-20T22:45:36.319 INFO:tasks.rados.rados.0.smithi084.stdout:868: read oid 46 snap 69 2024-09-20T22:45:36.319 INFO:tasks.rados.rados.0.smithi084.stdout:868: expect deleted 2024-09-20T22:45:36.319 INFO:tasks.rados.rados.0.smithi084.stdout:869: write oid 26 current snap is 84 2024-09-20T22:45:36.319 INFO:tasks.rados.rados.0.smithi084.stdout:869: seq_num 326 ranges {679082=618000,1843718=569105,2853072=626010} 2024-09-20T22:45:36.325 INFO:tasks.rados.rados.0.smithi084.stdout:869: writing smithi08477203-26 from 679082 to 1297082 tid 1 2024-09-20T22:45:36.332 INFO:tasks.rados.rados.0.smithi084.stdout:869: writing smithi08477203-26 from 1843718 to 2412823 tid 2 2024-09-20T22:45:36.338 INFO:tasks.rados.rados.0.smithi084.stdout:869: writing smithi08477203-26 from 2853072 to 3479082 tid 3 2024-09-20T22:45:36.338 INFO:tasks.rados.rados.0.smithi084.stdout:870: write (excl) oid 8 current snap is 84 2024-09-20T22:45:36.338 INFO:tasks.rados.rados.0.smithi084.stdout:870: seq_num 327 ranges {695889=409602,1618553=712763,2996848=499041} 2024-09-20T22:45:36.338 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:45:36.344 INFO:tasks.rados.rados.0.smithi084.stdout:870: writing smithi08477203-8 from 695889 to 1105491 tid 1 2024-09-20T22:45:36.351 INFO:tasks.rados.rados.0.smithi084.stdout:870: writing smithi08477203-8 from 1618553 to 2331316 tid 2 2024-09-20T22:45:36.357 INFO:tasks.rados.rados.0.smithi084.stdout:870: writing smithi08477203-8 from 2996848 to 3495889 tid 3 2024-09-20T22:45:36.357 INFO:tasks.rados.rados.0.smithi084.stdout:871: write oid 48 current snap is 84 2024-09-20T22:45:36.357 INFO:tasks.rados.rados.0.smithi084.stdout:871: seq_num 328 ranges {712696=484851,1960682=572774,3307918=204778} 2024-09-20T22:45:36.360 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:36.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:36.361 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:45:36.363 INFO:tasks.rados.rados.0.smithi084.stdout:871: writing smithi08477203-48 from 712696 to 1197547 tid 1 2024-09-20T22:45:36.371 INFO:tasks.rados.rados.0.smithi084.stdout:871: writing smithi08477203-48 from 1960682 to 2533456 tid 2 2024-09-20T22:45:36.377 INFO:tasks.rados.rados.0.smithi084.stdout:871: writing smithi08477203-48 from 3307918 to 3512696 tid 3 2024-09-20T22:45:36.378 INFO:tasks.rados.rados.0.smithi084.stdout:872: copy_from oid 44 from oid 39 current snap is 84 2024-09-20T22:45:36.378 INFO:tasks.rados.rados.0.smithi084.stdout:873: read oid 42 snap -1 2024-09-20T22:45:36.378 INFO:tasks.rados.rados.0.smithi084.stdout:873: expect deleted 2024-09-20T22:45:36.378 INFO:tasks.rados.rados.0.smithi084.stdout:874: read oid 14 snap -1 2024-09-20T22:45:36.378 INFO:tasks.rados.rados.0.smithi084.stdout:874: expect (ObjNum 306 snap 75 seq_num 306) 2024-09-20T22:45:36.378 INFO:tasks.rados.rados.0.smithi084.stdout:875: read oid 23 snap 80 2024-09-20T22:45:36.378 INFO:tasks.rados.rados.0.smithi084.stdout:875: expect (ObjNum 282 snap 71 seq_num 282) 2024-09-20T22:45:36.378 INFO:tasks.rados.rados.0.smithi084.stdout:876: read oid 38 snap -1 2024-09-20T22:45:36.378 INFO:tasks.rados.rados.0.smithi084.stdout:876: expect deleted 2024-09-20T22:45:36.378 INFO:tasks.rados.rados.0.smithi084.stdout:877: read oid 13 snap -1 2024-09-20T22:45:36.378 INFO:tasks.rados.rados.0.smithi084.stdout:877: expect deleted 2024-09-20T22:45:36.378 INFO:tasks.rados.rados.0.smithi084.stdout: waiting on 16 2024-09-20T22:45:36.378 INFO:tasks.rados.rados.0.smithi084.stdout:865: finishing copy_from to smithi08477203-31 2024-09-20T22:45:36.378 INFO:tasks.rados.rados.0.smithi084.stdout:865: got expected ENOENT (src dne) 2024-09-20T22:45:36.379 INFO:tasks.rados.rados.0.smithi084.stdout:865: finishing copy_from racing read to smithi08477203-31 2024-09-20T22:45:36.379 INFO:tasks.rados.rados.0.smithi084.stdout:862: finishing rollback tid 0 to smithi08477203-37 2024-09-20T22:45:36.379 INFO:tasks.rados.rados.0.smithi084.stdout:862: finishing rollback tid 1 to smithi08477203-37 2024-09-20T22:45:36.379 INFO:tasks.rados.rados.0.smithi084.stdout:864: done (15 left) 2024-09-20T22:45:36.379 INFO:tasks.rados.rados.0.smithi084.stdout:865: done (14 left) 2024-09-20T22:45:36.379 INFO:tasks.rados.rados.0.smithi084.stdout:867: done (13 left) 2024-09-20T22:45:36.379 INFO:tasks.rados.rados.0.smithi084.stdout:868: done (12 left) 2024-09-20T22:45:36.379 INFO:tasks.rados.rados.0.smithi084.stdout:878: write (excl) oid 31 current snap is 84 2024-09-20T22:45:36.379 INFO:tasks.rados.rados.0.smithi084.stdout:878: seq_num 329 ranges {729503=560100,2019164=716432,3335341=194162} 2024-09-20T22:45:36.384 INFO:tasks.rados.rados.0.smithi084.stdout:878: writing smithi08477203-31 from 729503 to 1289603 tid 1 2024-09-20T22:45:36.393 INFO:tasks.rados.rados.0.smithi084.stdout:878: writing smithi08477203-31 from 2019164 to 2735596 tid 2 2024-09-20T22:45:36.397 INFO:tasks.rados.rados.0.smithi084.stdout:878: writing smithi08477203-31 from 3335341 to 3529503 tid 3 2024-09-20T22:45:36.398 INFO:tasks.rados.rados.0.smithi084.stdout:879: write oid 20 current snap is 84 2024-09-20T22:45:36.398 INFO:tasks.rados.rados.0.smithi084.stdout:862: finishing rollback tid 2 to smithi08477203-37 2024-09-20T22:45:36.398 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 37 v 172 (ObjNum 253 snap 59 seq_num 253) dirty exists 2024-09-20T22:45:36.398 INFO:tasks.rados.rados.0.smithi084.stdout:879: seq_num 330 ranges {746310=635349,2077646=576443,3362764=183546} 2024-09-20T22:45:36.405 INFO:tasks.rados.rados.0.smithi084.stdout:879: writing smithi08477203-20 from 746310 to 1381659 tid 1 2024-09-20T22:45:36.412 INFO:tasks.rados.rados.0.smithi084.stdout:879: writing smithi08477203-20 from 2077646 to 2654089 tid 2 2024-09-20T22:45:36.417 INFO:tasks.rados.rados.0.smithi084.stdout:879: writing smithi08477203-20 from 3362764 to 3546310 tid 3 2024-09-20T22:45:36.417 INFO:tasks.rados.rados.0.smithi084.stdout:862: done (13 left) 2024-09-20T22:45:36.417 INFO:tasks.rados.rados.0.smithi084.stdout:880: copy_from oid 4 from oid 40 current snap is 84 2024-09-20T22:45:36.417 INFO:tasks.rados.rados.0.smithi084.stdout:866: finishing rollback tid 0 to smithi08477203-7 2024-09-20T22:45:36.417 INFO:tasks.rados.rados.0.smithi084.stdout:881: copy_from oid 6 from oid 50 current snap is 84 2024-09-20T22:45:36.417 INFO:tasks.rados.rados.0.smithi084.stdout:866: finishing rollback tid 1 to smithi08477203-7 2024-09-20T22:45:36.417 INFO:tasks.rados.rados.0.smithi084.stdout:866: finishing rollback tid 2 to smithi08477203-7 2024-09-20T22:45:36.418 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 7 v 176 (ObjNum 260 snap 61 seq_num 260) dirty exists 2024-09-20T22:45:36.418 INFO:tasks.rados.rados.0.smithi084.stdout:863: expect (ObjNum 258 snap 60 seq_num 258) 2024-09-20T22:45:36.432 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:36 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T22:45:36.432 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:36 smithi154 ceph-mon[33264]: osdmap e256: 8 total, 8 up, 7 in 2024-09-20T22:45:36.432 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:36 smithi154 ceph-mon[33264]: 7.11 scrub starts 2024-09-20T22:45:36.432 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:36 smithi154 ceph-mon[33264]: 7.11 scrub ok 2024-09-20T22:45:36.432 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:36 smithi154 ceph-mon[33264]: 5.6 scrub starts 2024-09-20T22:45:36.432 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:36 smithi154 ceph-mon[33264]: 5.6 scrub ok 2024-09-20T22:45:36.459 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:36 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T22:45:36.459 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:36 smithi084 ceph-mon[40024]: osdmap e256: 8 total, 8 up, 7 in 2024-09-20T22:45:36.459 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:36 smithi084 ceph-mon[40024]: 7.11 scrub starts 2024-09-20T22:45:36.459 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:36 smithi084 ceph-mon[40024]: 7.11 scrub ok 2024-09-20T22:45:36.459 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:36 smithi084 ceph-mon[40024]: 5.6 scrub starts 2024-09-20T22:45:36.459 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:36 smithi084 ceph-mon[40024]: 5.6 scrub ok 2024-09-20T22:45:36.459 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:36 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T22:45:36.460 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:36 smithi084 ceph-mon[29774]: osdmap e256: 8 total, 8 up, 7 in 2024-09-20T22:45:36.460 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:36 smithi084 ceph-mon[29774]: 7.11 scrub starts 2024-09-20T22:45:36.460 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:36 smithi084 ceph-mon[29774]: 7.11 scrub ok 2024-09-20T22:45:36.460 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:36 smithi084 ceph-mon[29774]: 5.6 scrub starts 2024-09-20T22:45:36.460 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:36 smithi084 ceph-mon[29774]: 5.6 scrub ok 2024-09-20T22:45:36.462 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:45:36.483 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:36.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:36.484 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:45:36.564 INFO:tasks.rados.rados.0.smithi084.stdout:869: finishing write tid 1 to smithi08477203-26 2024-09-20T22:45:36.564 INFO:tasks.rados.rados.0.smithi084.stdout:869: finishing write tid 2 to smithi08477203-26 2024-09-20T22:45:36.564 INFO:tasks.rados.rados.0.smithi084.stdout:863: done (14 left) 2024-09-20T22:45:36.564 INFO:tasks.rados.rados.0.smithi084.stdout:866: done (13 left) 2024-09-20T22:45:36.564 INFO:tasks.rados.rados.0.smithi084.stdout:882: snap_remove snap 67 2024-09-20T22:45:36.586 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:45:36.607 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:36.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:36.608 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:45:36.642 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-20T22:45:36.642 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:45:36.709 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:45:36.731 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:36.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:36.732 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:45:36.751 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:45:36.751 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:45:36.766 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:45:36.766 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:45:36.782 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:45:36.782 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:45:36.799 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-20T22:45:36.799 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T22:45:36.815 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:45:36.815 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:45:36.832 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:45:36.832 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:45:36.832 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:45:36.849 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:45:36.849 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:45:36.853 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:36.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:36.854 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:45:36.866 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:45:36.866 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:45:36.956 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:45:36.977 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:36.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:36.979 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:45:37.079 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:45:37.100 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:37.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:37.102 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:45:37.175 INFO:tasks.rados.rados.0.smithi084.stdout:882: done (13 left) 2024-09-20T22:45:37.175 INFO:tasks.rados.rados.0.smithi084.stdout:883: read oid 46 snap -1 2024-09-20T22:45:37.175 INFO:tasks.rados.rados.0.smithi084.stdout:883: expect (ObjNum 320 snap 82 seq_num 320) 2024-09-20T22:45:37.176 INFO:tasks.rados.rados.0.smithi084.stdout:884: copy_from oid 16 from oid 18 current snap is 84 2024-09-20T22:45:37.176 INFO:tasks.rados.rados.0.smithi084.stdout:885: rollback oid 11 current snap is 84 2024-09-20T22:45:37.176 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 11 to 60 2024-09-20T22:45:37.176 INFO:tasks.rados.rados.0.smithi084.stdout: waiting on 16 2024-09-20T22:45:37.177 INFO:tasks.rados.rados.0.smithi084.stdout:869: finishing write tid 3 to smithi08477203-26 2024-09-20T22:45:37.177 INFO:tasks.rados.rados.0.smithi084.stdout:869: finishing write tid 4 to smithi08477203-26 2024-09-20T22:45:37.177 INFO:tasks.rados.rados.0.smithi084.stdout:869: finishing write tid 5 to smithi08477203-26 2024-09-20T22:45:37.177 INFO:tasks.rados.rados.0.smithi084.stdout:869: oid 26 updating version 0 to 211 2024-09-20T22:45:37.178 INFO:tasks.rados.rados.0.smithi084.stdout:869: oid 26 version 211 is already newer than 210 2024-09-20T22:45:37.178 INFO:tasks.rados.rados.0.smithi084.stdout:869: oid 26 version 211 is already newer than 208 2024-09-20T22:45:37.178 INFO:tasks.rados.rados.0.smithi084.stdout:869: oid 26 version 211 is already newer than 209 2024-09-20T22:45:37.178 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 26 v 211 (ObjNum 326 snap 84 seq_num 326) dirty exists 2024-09-20T22:45:37.178 INFO:tasks.rados.rados.0.smithi084.stdout:869: left oid 26 (ObjNum 326 snap 84 seq_num 326) 2024-09-20T22:45:37.178 INFO:tasks.rados.rados.0.smithi084.stdout:870: finishing write tid 1 to smithi08477203-8 2024-09-20T22:45:37.178 INFO:tasks.rados.rados.0.smithi084.stdout:870: finishing write tid 2 to smithi08477203-8 2024-09-20T22:45:37.178 INFO:tasks.rados.rados.0.smithi084.stdout:870: finishing write tid 3 to smithi08477203-8 2024-09-20T22:45:37.178 INFO:tasks.rados.rados.0.smithi084.stdout:870: finishing write tid 4 to smithi08477203-8 2024-09-20T22:45:37.178 INFO:tasks.rados.rados.0.smithi084.stdout:870: finishing write tid 5 to smithi08477203-8 2024-09-20T22:45:37.178 INFO:tasks.rados.rados.0.smithi084.stdout:870: oid 8 updating version 0 to 132 2024-09-20T22:45:37.178 INFO:tasks.rados.rados.0.smithi084.stdout:870: oid 8 version 132 is already newer than 130 2024-09-20T22:45:37.179 INFO:tasks.rados.rados.0.smithi084.stdout:870: oid 8 version 132 is already newer than 129 2024-09-20T22:45:37.179 INFO:tasks.rados.rados.0.smithi084.stdout:870: oid 8 version 132 is already newer than 131 2024-09-20T22:45:37.179 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 8 v 132 (ObjNum 327 snap 84 seq_num 327) dirty exists 2024-09-20T22:45:37.179 INFO:tasks.rados.rados.0.smithi084.stdout:870: left oid 8 (ObjNum 327 snap 84 seq_num 327) 2024-09-20T22:45:37.179 INFO:tasks.rados.rados.0.smithi084.stdout:871: finishing write tid 1 to smithi08477203-48 2024-09-20T22:45:37.179 INFO:tasks.rados.rados.0.smithi084.stdout:871: finishing write tid 2 to smithi08477203-48 2024-09-20T22:45:37.179 INFO:tasks.rados.rados.0.smithi084.stdout:871: finishing write tid 3 to smithi08477203-48 2024-09-20T22:45:37.179 INFO:tasks.rados.rados.0.smithi084.stdout:871: finishing write tid 4 to smithi08477203-48 2024-09-20T22:45:37.179 INFO:tasks.rados.rados.0.smithi084.stdout:871: finishing write tid 5 to smithi08477203-48 2024-09-20T22:45:37.179 INFO:tasks.rados.rados.0.smithi084.stdout:871: oid 48 updating version 0 to 180 2024-09-20T22:45:37.179 INFO:tasks.rados.rados.0.smithi084.stdout:871: oid 48 version 180 is already newer than 177 2024-09-20T22:45:37.179 INFO:tasks.rados.rados.0.smithi084.stdout:871: oid 48 version 180 is already newer than 179 2024-09-20T22:45:37.179 INFO:tasks.rados.rados.0.smithi084.stdout:871: oid 48 version 180 is already newer than 178 2024-09-20T22:45:37.179 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 48 v 180 (ObjNum 328 snap 84 seq_num 328) dirty exists 2024-09-20T22:45:37.180 INFO:tasks.rados.rados.0.smithi084.stdout:871: left oid 48 (ObjNum 328 snap 84 seq_num 328) 2024-09-20T22:45:37.180 INFO:tasks.rados.rados.0.smithi084.stdout:874: expect (ObjNum 306 snap 75 seq_num 306) 2024-09-20T22:45:37.202 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:45:37.223 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:37.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:37.224 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:45:37.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:37 smithi154 ceph-mon[33264]: pgmap v286: 178 pgs: 3 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 10 remapped+peering, 161 active+clean; 198 MiB data, 696 MiB used, 625 GiB / 626 GiB avail; 7.5 MiB/s rd, 3.3 MiB/s wr, 31 op/s; 39 KiB/s, 0 objects/s recovering 2024-09-20T22:45:37.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:37 smithi154 ceph-mon[33264]: 7.13 scrub starts 2024-09-20T22:45:37.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:37 smithi154 ceph-mon[33264]: 7.13 scrub ok 2024-09-20T22:45:37.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:37 smithi154 ceph-mon[33264]: 4.1b scrub starts 2024-09-20T22:45:37.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:37 smithi154 ceph-mon[33264]: 4.1b scrub ok 2024-09-20T22:45:37.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:37 smithi154 ceph-mon[33264]: osdmap e257: 8 total, 8 up, 7 in 2024-09-20T22:45:37.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:37 smithi154 ceph-mon[33264]: 1.10 scrub starts 2024-09-20T22:45:37.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:37 smithi154 ceph-mon[33264]: 1.10 scrub ok 2024-09-20T22:45:37.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:37 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/1858133773' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:45:37.324 INFO:tasks.rados.rados.0.smithi084.stdout:875: expect (ObjNum 282 snap 71 seq_num 282) 2024-09-20T22:45:37.324 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:45:37.348 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:37.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:37.349 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:45:37.359 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.5 dump_historic_ops 2024-09-20T22:45:37.450 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:45:37.471 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:37.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:37.472 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:45:37.475 INFO:tasks.rados.rados.0.smithi084.stdout:872: finishing copy_from racing read to smithi08477203-44 2024-09-20T22:45:37.475 INFO:tasks.rados.rados.0.smithi084.stdout:878: finishing write tid 1 to smithi08477203-31 2024-09-20T22:45:37.475 INFO:tasks.rados.rados.0.smithi084.stdout:872: finishing copy_from to smithi08477203-44 2024-09-20T22:45:37.475 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 44 v 199 (ObjNum 245 snap 54 seq_num 245) dirty exists 2024-09-20T22:45:37.475 INFO:tasks.rados.rados.0.smithi084.stdout:878: finishing write tid 2 to smithi08477203-31 2024-09-20T22:45:37.475 INFO:tasks.rados.rados.0.smithi084.stdout:878: finishing write tid 3 to smithi08477203-31 2024-09-20T22:45:37.476 INFO:tasks.rados.rados.0.smithi084.stdout:878: finishing write tid 4 to smithi08477203-31 2024-09-20T22:45:37.476 INFO:tasks.rados.rados.0.smithi084.stdout:878: finishing write tid 5 to smithi08477203-31 2024-09-20T22:45:37.476 INFO:tasks.rados.rados.0.smithi084.stdout:878: oid 31 updating version 0 to 207 2024-09-20T22:45:37.476 INFO:tasks.rados.rados.0.smithi084.stdout:878: oid 31 version 207 is already newer than 205 2024-09-20T22:45:37.476 INFO:tasks.rados.rados.0.smithi084.stdout:878: oid 31 version 207 is already newer than 206 2024-09-20T22:45:37.476 INFO:tasks.rados.rados.0.smithi084.stdout:878: oid 31 version 207 is already newer than 204 2024-09-20T22:45:37.476 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 31 v 207 (ObjNum 329 snap 84 seq_num 329) dirty exists 2024-09-20T22:45:37.476 INFO:tasks.rados.rados.0.smithi084.stdout:878: left oid 31 (ObjNum 329 snap 84 seq_num 329) 2024-09-20T22:45:37.476 INFO:tasks.rados.rados.0.smithi084.stdout:879: finishing write tid 1 to smithi08477203-20 2024-09-20T22:45:37.476 INFO:tasks.rados.rados.0.smithi084.stdout:879: finishing write tid 2 to smithi08477203-20 2024-09-20T22:45:37.476 INFO:tasks.rados.rados.0.smithi084.stdout:869: done (15 left) 2024-09-20T22:45:37.476 INFO:tasks.rados.rados.0.smithi084.stdout:870: done (14 left) 2024-09-20T22:45:37.476 INFO:tasks.rados.rados.0.smithi084.stdout:871: done (13 left) 2024-09-20T22:45:37.476 INFO:tasks.rados.rados.0.smithi084.stdout:872: done (12 left) 2024-09-20T22:45:37.477 INFO:tasks.rados.rados.0.smithi084.stdout:873: done (11 left) 2024-09-20T22:45:37.477 INFO:tasks.rados.rados.0.smithi084.stdout:874: done (10 left) 2024-09-20T22:45:37.477 INFO:tasks.rados.rados.0.smithi084.stdout:875: done (9 left) 2024-09-20T22:45:37.477 INFO:tasks.rados.rados.0.smithi084.stdout:876: done (8 left) 2024-09-20T22:45:37.477 INFO:tasks.rados.rados.0.smithi084.stdout:877: done (7 left) 2024-09-20T22:45:37.477 INFO:tasks.rados.rados.0.smithi084.stdout:878: done (6 left) 2024-09-20T22:45:37.477 INFO:tasks.rados.rados.0.smithi084.stdout:886: read oid 44 snap -1 2024-09-20T22:45:37.477 INFO:tasks.rados.rados.0.smithi084.stdout:886: expect (ObjNum 245 snap 54 seq_num 245) 2024-09-20T22:45:37.477 INFO:tasks.rados.rados.0.smithi084.stdout:887: snap_create 2024-09-20T22:45:37.477 INFO:tasks.rados.rados.0.smithi084.stdout:879: finishing write tid 3 to smithi08477203-20 2024-09-20T22:45:37.477 INFO:tasks.rados.rados.0.smithi084.stdout:881: finishing copy_from to smithi08477203-6 2024-09-20T22:45:37.477 INFO:tasks.rados.rados.0.smithi084.stdout:881: got expected ENOENT (src dne) 2024-09-20T22:45:37.477 INFO:tasks.rados.rados.0.smithi084.stdout:881: finishing copy_from racing read to smithi08477203-6 2024-09-20T22:45:37.477 INFO:tasks.rados.rados.0.smithi084.stdout:879: finishing write tid 4 to smithi08477203-20 2024-09-20T22:45:37.477 INFO:tasks.rados.rados.0.smithi084.stdout:879: finishing write tid 5 to smithi08477203-20 2024-09-20T22:45:37.478 INFO:tasks.rados.rados.0.smithi084.stdout:879: oid 20 updating version 0 to 68 2024-09-20T22:45:37.478 INFO:tasks.rados.rados.0.smithi084.stdout:879: oid 20 version 68 is already newer than 66 2024-09-20T22:45:37.478 INFO:tasks.rados.rados.0.smithi084.stdout:879: oid 20 version 68 is already newer than 67 2024-09-20T22:45:37.478 INFO:tasks.rados.rados.0.smithi084.stdout:879: oid 20 updating version 68 to 69 2024-09-20T22:45:37.478 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 20 v 69 (ObjNum 330 snap 84 seq_num 330) dirty exists 2024-09-20T22:45:37.478 INFO:tasks.rados.rados.0.smithi084.stdout:879: left oid 20 (ObjNum 330 snap 84 seq_num 330) 2024-09-20T22:45:37.478 INFO:tasks.rados.rados.0.smithi084.stdout:880: finishing copy_from racing read to smithi08477203-4 2024-09-20T22:45:37.478 INFO:tasks.rados.rados.0.smithi084.stdout:880: finishing copy_from to smithi08477203-4 2024-09-20T22:45:37.478 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 4 v 172 (ObjNum 316 snap 80 seq_num 316) dirty exists 2024-09-20T22:45:37.478 INFO:tasks.rados.rados.0.smithi084.stdout:885: finishing rollback tid 0 to smithi08477203-11 2024-09-20T22:45:37.478 INFO:tasks.rados.rados.0.smithi084.stdout:885: finishing rollback tid 1 to smithi08477203-11 2024-09-20T22:45:37.478 INFO:tasks.rados.rados.0.smithi084.stdout:885: finishing rollback tid 2 to smithi08477203-11 2024-09-20T22:45:37.478 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 11 v 215 (ObjNum 245 snap 54 seq_num 245) dirty exists 2024-09-20T22:45:37.478 INFO:tasks.rados.rados.0.smithi084.stdout:883: expect (ObjNum 320 snap 82 seq_num 320) 2024-09-20T22:45:37.506 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:37 smithi084 ceph-mon[40024]: pgmap v286: 178 pgs: 3 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 10 remapped+peering, 161 active+clean; 198 MiB data, 696 MiB used, 625 GiB / 626 GiB avail; 7.5 MiB/s rd, 3.3 MiB/s wr, 31 op/s; 39 KiB/s, 0 objects/s recovering 2024-09-20T22:45:37.506 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:37 smithi084 ceph-mon[40024]: 7.13 scrub starts 2024-09-20T22:45:37.506 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:37 smithi084 ceph-mon[40024]: 7.13 scrub ok 2024-09-20T22:45:37.506 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:37 smithi084 ceph-mon[40024]: 4.1b scrub starts 2024-09-20T22:45:37.506 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:37 smithi084 ceph-mon[40024]: 4.1b scrub ok 2024-09-20T22:45:37.506 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:37 smithi084 ceph-mon[40024]: osdmap e257: 8 total, 8 up, 7 in 2024-09-20T22:45:37.506 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:37 smithi084 ceph-mon[40024]: 1.10 scrub starts 2024-09-20T22:45:37.506 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:37 smithi084 ceph-mon[40024]: 1.10 scrub ok 2024-09-20T22:45:37.506 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:37 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/1858133773' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:45:37.507 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:37 smithi084 ceph-mon[29774]: pgmap v286: 178 pgs: 3 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 10 remapped+peering, 161 active+clean; 198 MiB data, 696 MiB used, 625 GiB / 626 GiB avail; 7.5 MiB/s rd, 3.3 MiB/s wr, 31 op/s; 39 KiB/s, 0 objects/s recovering 2024-09-20T22:45:37.507 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:37 smithi084 ceph-mon[29774]: 7.13 scrub starts 2024-09-20T22:45:37.507 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:37 smithi084 ceph-mon[29774]: 7.13 scrub ok 2024-09-20T22:45:37.507 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:37 smithi084 ceph-mon[29774]: 4.1b scrub starts 2024-09-20T22:45:37.507 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:37 smithi084 ceph-mon[29774]: 4.1b scrub ok 2024-09-20T22:45:37.507 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:37 smithi084 ceph-mon[29774]: osdmap e257: 8 total, 8 up, 7 in 2024-09-20T22:45:37.507 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:37 smithi084 ceph-mon[29774]: 1.10 scrub starts 2024-09-20T22:45:37.507 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:37 smithi084 ceph-mon[29774]: 1.10 scrub ok 2024-09-20T22:45:37.507 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:37 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/1858133773' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:45:37.518 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 7] out_osds: [4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-20T22:45:37.519 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-20T22:45:37.519 INFO:tasks.thrashosds.thrasher:Removing osd 0, in_osds are: [0, 1, 2, 3, 5, 6, 7] 2024-09-20T22:45:37.519 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd out 0 2024-09-20T22:45:37.572 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:45:37.585 INFO:tasks.rados.rados.0.smithi084.stdout:884: finishing copy_from racing read to smithi08477203-16 2024-09-20T22:45:37.585 INFO:tasks.rados.rados.0.smithi084.stdout:884: finishing copy_from to smithi08477203-16 2024-09-20T22:45:37.585 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 16 v 168 (ObjNum 273 snap 69 seq_num 273) dirty exists 2024-09-20T22:45:37.585 INFO:tasks.rados.rados.0.smithi084.stdout:886: expect (ObjNum 245 snap 54 seq_num 245) 2024-09-20T22:45:37.594 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:37.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:37.595 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:45:37.650 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:45:37.695 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:45:37.717 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:37.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:37.718 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:45:37.768 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:45:37.803 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-20T22:45:37.819 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:45:37.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:37.842 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:37.842 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:45:37.943 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:45:37.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:37.965 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:37.966 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:45:38.066 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:45:38.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:38.089 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:38.090 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:45:38.163 INFO:tasks.rados.rados.0.smithi084.stdout:879: done (7 left) 2024-09-20T22:45:38.163 INFO:tasks.rados.rados.0.smithi084.stdout:880: done (6 left) 2024-09-20T22:45:38.163 INFO:tasks.rados.rados.0.smithi084.stdout:881: done (5 left) 2024-09-20T22:45:38.163 INFO:tasks.rados.rados.0.smithi084.stdout:883: done (4 left) 2024-09-20T22:45:38.164 INFO:tasks.rados.rados.0.smithi084.stdout:884: done (3 left) 2024-09-20T22:45:38.164 INFO:tasks.rados.rados.0.smithi084.stdout:885: done (2 left) 2024-09-20T22:45:38.164 INFO:tasks.rados.rados.0.smithi084.stdout:886: done (1 left) 2024-09-20T22:45:38.164 INFO:tasks.rados.rados.0.smithi084.stdout:887: done (0 left) 2024-09-20T22:45:38.164 INFO:tasks.rados.rados.0.smithi084.stdout:888: rollback oid 15 current snap is 85 2024-09-20T22:45:38.164 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 15 to 79 2024-09-20T22:45:38.164 INFO:tasks.rados.rados.0.smithi084.stdout:889: delete oid 24 current snap is 85 2024-09-20T22:45:38.172 INFO:tasks.rados.rados.0.smithi084.stdout:889: done (1 left) 2024-09-20T22:45:38.172 INFO:tasks.rados.rados.0.smithi084.stdout:890: write oid 21 current snap is 85 2024-09-20T22:45:38.172 INFO:tasks.rados.rados.0.smithi084.stdout:890: seq_num 332 ranges {779924=785847,2078257=580112,3301257=278667} 2024-09-20T22:45:38.178 INFO:tasks.rados.rados.0.smithi084.stdout:890: writing smithi08477203-21 from 779924 to 1565771 tid 1 2024-09-20T22:45:38.185 INFO:tasks.rados.rados.0.smithi084.stdout:890: writing smithi08477203-21 from 2078257 to 2658369 tid 2 2024-09-20T22:45:38.190 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:45:38.191 INFO:tasks.rados.rados.0.smithi084.stdout:890: writing smithi08477203-21 from 3301257 to 3579924 tid 3 2024-09-20T22:45:38.191 INFO:tasks.rados.rados.0.smithi084.stdout:891: read oid 23 snap -1 2024-09-20T22:45:38.191 INFO:tasks.rados.rados.0.smithi084.stdout:891: expect (ObjNum 282 snap 71 seq_num 282) 2024-09-20T22:45:38.191 INFO:tasks.rados.rados.0.smithi084.stdout:892: write (excl) oid 41 current snap is 85 2024-09-20T22:45:38.192 INFO:tasks.rados.rados.0.smithi084.stdout:888: finishing rollback tid 0 to smithi08477203-15 2024-09-20T22:45:38.192 INFO:tasks.rados.rados.0.smithi084.stdout:892: seq_num 333 ranges {796731=461096,1736739=723770,2928680=668051} 2024-09-20T22:45:38.198 INFO:tasks.rados.rados.0.smithi084.stdout:892: writing smithi08477203-41 from 796731 to 1257827 tid 1 2024-09-20T22:45:38.204 INFO:tasks.rados.rados.0.smithi084.stdout:892: writing smithi08477203-41 from 1736739 to 2460509 tid 2 2024-09-20T22:45:38.211 INFO:tasks.rados.rados.0.smithi084.stdout:892: writing smithi08477203-41 from 2928680 to 3596731 tid 3 2024-09-20T22:45:38.211 INFO:tasks.rados.rados.0.smithi084.stdout:893: snap_remove snap 73 2024-09-20T22:45:38.216 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:38.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:38.217 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:45:38.282 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-20T22:45:38.283 INFO:teuthology.orchestra.run.smithi154.stdout: "size": 20, 2024-09-20T22:45:38.283 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 600, 2024-09-20T22:45:38.283 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [ 2024-09-20T22:45:38.283 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.283 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24334.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-09-20T22:45:38.283 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:41:40.156321+0000", 2024-09-20T22:45:38.283 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 238.12316570799999, 2024-09-20T22:45:38.283 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.012506929, 2024-09-20T22:45:38.283 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:45:38.283 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:45:38.283 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:45:38.283 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:45:38.284 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24334", 2024-09-20T22:45:38.284 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/343304630", 2024-09-20T22:45:38.284 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 7 2024-09-20T22:45:38.284 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.284 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:45:38.284 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.284 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:45:38.284 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:40.156321+0000", 2024-09-20T22:45:38.284 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:45:38.284 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.284 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.284 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:45:38.284 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:40.156321+0000", 2024-09-20T22:45:38.284 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:45:38.284 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.285 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.285 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:45:38.285 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:40.156323+0000", 2024-09-20T22:45:38.285 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9700000000000002e-06 2024-09-20T22:45:38.286 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.286 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.286 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:45:38.286 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:40.156330+0000", 2024-09-20T22:45:38.286 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.9809999999999997e-06 2024-09-20T22:45:38.286 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.286 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.286 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:45:38.286 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:40.156348+0000", 2024-09-20T22:45:38.286 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.7938e-05 2024-09-20T22:45:38.286 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.286 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.286 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:45:38.287 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:40.156362+0000", 2024-09-20T22:45:38.287 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3641e-05 2024-09-20T22:45:38.287 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.287 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.287 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:45:38.287 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:40.156430+0000", 2024-09-20T22:45:38.287 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.7543e-05 2024-09-20T22:45:38.287 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.287 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.287 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:45:38.287 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:40.156543+0000", 2024-09-20T22:45:38.287 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00011390100000000001 2024-09-20T22:45:38.287 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.287 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.288 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 1,6", 2024-09-20T22:45:38.288 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:40.156712+0000", 2024-09-20T22:45:38.288 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00016825499999999999 2024-09-20T22:45:38.288 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.288 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.288 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:45:38.288 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:40.157640+0000", 2024-09-20T22:45:38.288 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00092863699999999995 2024-09-20T22:45:38.288 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.288 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.288 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:45:38.288 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:40.159647+0000", 2024-09-20T22:45:38.288 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0020068320000000001 2024-09-20T22:45:38.288 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.288 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.288 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:45:38.288 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:40.168687+0000", 2024-09-20T22:45:38.289 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0090397749999999999 2024-09-20T22:45:38.289 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.289 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.289 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:45:38.289 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:40.168768+0000", 2024-09-20T22:45:38.289 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.0819e-05 2024-09-20T22:45:38.289 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.289 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.289 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:45:38.289 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:40.168828+0000", 2024-09-20T22:45:38.289 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.0637000000000002e-05 2024-09-20T22:45:38.289 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:45:38.289 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:45:38.289 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:45:38.290 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.290 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.290 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "MOSDPGPush(1.4 93/92 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 77'128, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@77'128, size: 327680, copy_subset: [(0, 327680)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 327680, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-20T22:45:38.290 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:42:50.646393+0000", 2024-09-20T22:45:38.290 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 167.633094269, 2024-09-20T22:45:38.290 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.033640915, 2024-09-20T22:45:38.290 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:45:38.290 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:45:38.290 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:45:38.290 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:45:38.290 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.290 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:45:38.290 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:42:50.646393+0000", 2024-09-20T22:45:38.291 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:45:38.291 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.291 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.291 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:45:38.291 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:42:50.646393+0000", 2024-09-20T22:45:38.291 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:45:38.291 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.291 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.291 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:45:38.291 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:42:50.646393+0000", 2024-09-20T22:45:38.291 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.23e-07 2024-09-20T22:45:38.291 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.291 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.291 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:45:38.292 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:42:50.679431+0000", 2024-09-20T22:45:38.292 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.033038349000000002 2024-09-20T22:45:38.292 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.292 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.292 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:45:38.292 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:42:50.679478+0000", 2024-09-20T22:45:38.292 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.6828e-05 2024-09-20T22:45:38.292 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.292 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.292 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:45:38.292 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:42:50.679492+0000", 2024-09-20T22:45:38.292 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4062e-05 2024-09-20T22:45:38.292 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.292 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.292 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:45:38.293 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:42:50.679593+0000", 2024-09-20T22:45:38.293 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000100865 2024-09-20T22:45:38.293 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.293 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.293 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:45:38.293 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:42:50.679604+0000", 2024-09-20T22:45:38.293 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0348e-05 2024-09-20T22:45:38.293 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.293 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.293 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:45:38.293 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:42:50.680034+0000", 2024-09-20T22:45:38.293 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00043023999999999997 2024-09-20T22:45:38.293 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:45:38.293 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:45:38.294 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:45:38.294 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.294 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.294 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:321 7.b e105/94 7:df02c955:::smithi08477203-9:head v 105'17, mlcod=98'14)", 2024-09-20T22:45:38.294 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:04.861367+0000", 2024-09-20T22:45:38.294 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 153.418120378, 2024-09-20T22:45:38.294 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0067649240000000003, 2024-09-20T22:45:38.294 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:45:38.294 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:45:38.294 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:45:38.294 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:45:38.294 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.294 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:45:38.294 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:04.861367+0000", 2024-09-20T22:45:38.294 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:45:38.295 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.295 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.295 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:45:38.295 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:04.861367+0000", 2024-09-20T22:45:38.295 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:45:38.295 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.295 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.295 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:45:38.295 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:04.861369+0000", 2024-09-20T22:45:38.295 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.8470000000000001e-06 2024-09-20T22:45:38.295 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.295 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.295 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:45:38.295 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:04.865306+0000", 2024-09-20T22:45:38.296 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0039377969999999998 2024-09-20T22:45:38.296 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.296 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.296 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:45:38.296 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:04.865326+0000", 2024-09-20T22:45:38.296 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9865e-05 2024-09-20T22:45:38.296 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.296 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.296 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:45:38.297 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:04.865337+0000", 2024-09-20T22:45:38.297 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0961000000000001e-05 2024-09-20T22:45:38.297 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.297 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.297 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:45:38.297 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:04.865404+0000", 2024-09-20T22:45:38.297 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.6869e-05 2024-09-20T22:45:38.297 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.297 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.297 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:45:38.298 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:04.865422+0000", 2024-09-20T22:45:38.298 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.7824000000000001e-05 2024-09-20T22:45:38.298 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.298 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.298 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:45:38.298 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:04.868076+0000", 2024-09-20T22:45:38.298 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0026539789999999999 2024-09-20T22:45:38.298 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.298 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.298 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:45:38.298 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:04.868132+0000", 2024-09-20T22:45:38.298 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.5782000000000002e-05 2024-09-20T22:45:38.298 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:45:38.298 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:45:38.298 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:45:38.299 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.299 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.299 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:419 7.0 7:05bb442d:::smithi08477203-24:head [write 460525~690852] snapc a=[9] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-09-20T22:45:38.299 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:14.165736+0000", 2024-09-20T22:45:38.299 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 144.11375137499999, 2024-09-20T22:45:38.299 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0072523500000000003, 2024-09-20T22:45:38.299 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:45:38.299 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:45:38.299 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:45:38.299 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:45:38.299 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:45:38.299 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:45:38.299 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 419 2024-09-20T22:45:38.299 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.300 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:45:38.300 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.300 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:45:38.300 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:14.165736+0000", 2024-09-20T22:45:38.300 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:45:38.300 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.300 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.300 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:45:38.300 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:14.165736+0000", 2024-09-20T22:45:38.300 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:45:38.300 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.300 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.300 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:45:38.300 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:14.165737+0000", 2024-09-20T22:45:38.300 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.615e-06 2024-09-20T22:45:38.301 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.301 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.301 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:45:38.301 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:14.166920+0000", 2024-09-20T22:45:38.301 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001182535 2024-09-20T22:45:38.301 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.301 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.301 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:45:38.301 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:14.166933+0000", 2024-09-20T22:45:38.301 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3263e-05 2024-09-20T22:45:38.301 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.301 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.301 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:45:38.301 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:14.166942+0000", 2024-09-20T22:45:38.302 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.3329999999999997e-06 2024-09-20T22:45:38.302 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.302 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.302 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:45:38.302 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:14.167001+0000", 2024-09-20T22:45:38.302 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.9305000000000002e-05 2024-09-20T22:45:38.302 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.302 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.302 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op must wait for map", 2024-09-20T22:45:38.303 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:14.167008+0000", 2024-09-20T22:45:38.303 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.5580000000000003e-06 2024-09-20T22:45:38.303 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.303 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.303 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:45:38.303 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:14.169081+0000", 2024-09-20T22:45:38.303 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0020726270000000001 2024-09-20T22:45:38.303 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.303 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.303 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:45:38.303 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:14.169179+0000", 2024-09-20T22:45:38.303 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.7613000000000002e-05 2024-09-20T22:45:38.303 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.304 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.304 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 1,3", 2024-09-20T22:45:38.304 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:14.169364+0000", 2024-09-20T22:45:38.304 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000185642 2024-09-20T22:45:38.304 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.304 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.304 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:45:38.304 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:14.171014+0000", 2024-09-20T22:45:38.304 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0016497129999999999 2024-09-20T22:45:38.304 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.304 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.304 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:45:38.304 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:14.172374+0000", 2024-09-20T22:45:38.304 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0013601240000000001 2024-09-20T22:45:38.304 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.305 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.305 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:45:38.305 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:14.172925+0000", 2024-09-20T22:45:38.305 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00055099499999999998 2024-09-20T22:45:38.305 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.305 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.305 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:45:38.305 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:14.172959+0000", 2024-09-20T22:45:38.305 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.4069000000000002e-05 2024-09-20T22:45:38.305 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.305 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.305 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:45:38.305 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:14.172988+0000", 2024-09-20T22:45:38.305 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.8958e-05 2024-09-20T22:45:38.306 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:45:38.306 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:45:38.306 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:45:38.306 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.306 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.306 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:551 7.0 7:05bb442d:::smithi08477203-24:head [read 0~3260525,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e128)", 2024-09-20T22:45:38.306 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:27.143449+0000", 2024-09-20T22:45:38.306 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 131.13603828199999, 2024-09-20T22:45:38.306 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0071533109999999999, 2024-09-20T22:45:38.306 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:45:38.306 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:45:38.306 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:45:38.306 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:45:38.306 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:45:38.307 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:45:38.307 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 551 2024-09-20T22:45:38.307 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.307 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:45:38.307 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.307 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:45:38.307 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.143449+0000", 2024-09-20T22:45:38.307 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:45:38.307 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.307 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.307 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:45:38.307 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.143449+0000", 2024-09-20T22:45:38.307 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:45:38.307 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.308 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.308 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:45:38.308 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.143449+0000", 2024-09-20T22:45:38.308 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.2000000000000001e-07 2024-09-20T22:45:38.308 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.308 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.308 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:45:38.308 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.143456+0000", 2024-09-20T22:45:38.308 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.8859999999999998e-06 2024-09-20T22:45:38.308 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.308 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.309 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:45:38.309 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.143462+0000", 2024-09-20T22:45:38.309 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.5940000000000001e-06 2024-09-20T22:45:38.309 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.309 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.309 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:45:38.309 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.143465+0000", 2024-09-20T22:45:38.310 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.631e-06 2024-09-20T22:45:38.310 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.310 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.310 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:45:38.310 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.143568+0000", 2024-09-20T22:45:38.311 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000103 2024-09-20T22:45:38.311 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.311 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.311 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting_for_map not empty", 2024-09-20T22:45:38.311 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.143576+0000", 2024-09-20T22:45:38.312 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.2699999999999999e-06 2024-09-20T22:45:38.312 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.312 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.312 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:45:38.312 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.149401+0000", 2024-09-20T22:45:38.312 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0058254489999999999 2024-09-20T22:45:38.313 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.313 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.313 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:45:38.313 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.149466+0000", 2024-09-20T22:45:38.313 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.4772999999999998e-05 2024-09-20T22:45:38.313 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.313 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.314 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:45:38.314 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.150602+0000", 2024-09-20T22:45:38.314 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0011363879999999999 2024-09-20T22:45:38.314 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:45:38.314 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:45:38.314 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:45:38.315 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.315 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.315 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:552 7.0 7:05bb442d:::smithi08477203-24:head [sparse-read 0~1736210] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e128)", 2024-09-20T22:45:38.315 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:27.143470+0000", 2024-09-20T22:45:38.315 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 131.136017006, 2024-09-20T22:45:38.315 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0075036570000000004, 2024-09-20T22:45:38.316 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:45:38.316 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:45:38.316 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:45:38.316 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:45:38.316 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:45:38.316 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:45:38.316 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 552 2024-09-20T22:45:38.317 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.317 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:45:38.317 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.317 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:45:38.317 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.143470+0000", 2024-09-20T22:45:38.317 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:45:38.317 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.318 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.318 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:45:38.318 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.143470+0000", 2024-09-20T22:45:38.318 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:45:38.318 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.318 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.319 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:45:38.319 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.143470+0000", 2024-09-20T22:45:38.319 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.84e-07 2024-09-20T22:45:38.319 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.319 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.319 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:45:38.319 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.143472+0000", 2024-09-20T22:45:38.320 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4139999999999999e-06 2024-09-20T22:45:38.320 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.320 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.320 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:45:38.320 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.143475+0000", 2024-09-20T22:45:38.320 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.1360000000000001e-06 2024-09-20T22:45:38.321 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.321 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.321 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:45:38.321 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.143478+0000", 2024-09-20T22:45:38.321 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.9000000000000002e-06 2024-09-20T22:45:38.321 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.321 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.322 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:45:38.322 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.143601+0000", 2024-09-20T22:45:38.322 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00012325199999999999 2024-09-20T22:45:38.322 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.322 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.322 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting_for_map not empty", 2024-09-20T22:45:38.322 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.143605+0000", 2024-09-20T22:45:38.323 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.5540000000000002e-06 2024-09-20T22:45:38.323 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.323 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.323 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:45:38.324 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.150656+0000", 2024-09-20T22:45:38.324 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0070513290000000003 2024-09-20T22:45:38.324 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.324 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.324 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:45:38.324 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.150787+0000", 2024-09-20T22:45:38.325 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00013061099999999999 2024-09-20T22:45:38.325 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.325 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.325 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:45:38.325 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.150974+0000", 2024-09-20T22:45:38.325 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00018717699999999999 2024-09-20T22:45:38.326 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:45:38.326 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:45:38.326 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:45:38.326 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.326 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.326 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:554 7.a e128/94 7:5122292a:::smithi08477203-33:head v 128'32, mlcod=119'28)", 2024-09-20T22:45:38.327 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:27.341250+0000", 2024-09-20T22:45:38.327 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 130.93823668499999, 2024-09-20T22:45:38.327 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.051661319999999997, 2024-09-20T22:45:38.327 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:45:38.327 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:45:38.327 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:45:38.328 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:45:38.328 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.328 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:45:38.328 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.341250+0000", 2024-09-20T22:45:38.328 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:45:38.328 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.329 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.329 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:45:38.329 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.341250+0000", 2024-09-20T22:45:38.329 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:45:38.329 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.329 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.330 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:45:38.330 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.341251+0000", 2024-09-20T22:45:38.330 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.7299999999999996e-07 2024-09-20T22:45:38.330 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.330 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.330 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:45:38.331 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.390395+0000", 2024-09-20T22:45:38.331 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.049143539999999999 2024-09-20T22:45:38.331 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.331 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.331 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:45:38.331 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.390413+0000", 2024-09-20T22:45:38.331 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.8709999999999999e-05 2024-09-20T22:45:38.332 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.332 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.332 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:45:38.332 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.390423+0000", 2024-09-20T22:45:38.332 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.4660000000000005e-06 2024-09-20T22:45:38.332 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.332 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.333 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:45:38.333 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.390484+0000", 2024-09-20T22:45:38.333 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.0851999999999999e-05 2024-09-20T22:45:38.333 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.333 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.333 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:45:38.333 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.390504+0000", 2024-09-20T22:45:38.333 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9976999999999999e-05 2024-09-20T22:45:38.333 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.334 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.334 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:45:38.334 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.392875+0000", 2024-09-20T22:45:38.334 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0023713710000000002 2024-09-20T22:45:38.334 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.334 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.334 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:45:38.334 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.392912+0000", 2024-09-20T22:45:38.335 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.6831e-05 2024-09-20T22:45:38.335 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:45:38.335 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:45:38.335 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:45:38.335 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.335 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.335 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:786 7.17 e145/139 7:ed54065f:::smithi08477203-12:head v 145'119, mlcod=143'117)", 2024-09-20T22:45:38.335 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:44.345115+0000", 2024-09-20T22:45:38.336 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 113.934372203, 2024-09-20T22:45:38.336 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0098654899999999993, 2024-09-20T22:45:38.336 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:45:38.336 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:45:38.336 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:45:38.336 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:45:38.336 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.336 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:45:38.337 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.345115+0000", 2024-09-20T22:45:38.337 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:45:38.337 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.337 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.337 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:45:38.337 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.345115+0000", 2024-09-20T22:45:38.337 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:45:38.337 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.337 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.337 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:45:38.337 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.345116+0000", 2024-09-20T22:45:38.337 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.8299999999999995e-07 2024-09-20T22:45:38.337 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.337 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.337 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:45:38.337 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.347955+0000", 2024-09-20T22:45:38.338 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.002839028 2024-09-20T22:45:38.338 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.338 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.338 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:45:38.338 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.347977+0000", 2024-09-20T22:45:38.338 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.1897999999999999e-05 2024-09-20T22:45:38.338 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.338 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.338 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:45:38.338 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.347992+0000", 2024-09-20T22:45:38.338 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4761e-05 2024-09-20T22:45:38.338 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.339 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.339 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:45:38.339 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.348072+0000", 2024-09-20T22:45:38.339 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.0586000000000004e-05 2024-09-20T22:45:38.339 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.339 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.339 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:45:38.339 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.348097+0000", 2024-09-20T22:45:38.339 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.5026000000000001e-05 2024-09-20T22:45:38.339 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.339 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.339 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:45:38.339 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.354924+0000", 2024-09-20T22:45:38.339 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0068271240000000004 2024-09-20T22:45:38.339 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.340 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.340 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:45:38.340 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.354980+0000", 2024-09-20T22:45:38.340 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.6184000000000003e-05 2024-09-20T22:45:38.340 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:45:38.340 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:45:38.340 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:45:38.340 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.340 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.340 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:804 7.17 e145/139 7:ea720b85:::smithi08477203-26:head v 145'120, mlcod=143'118)", 2024-09-20T22:45:38.340 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:44.456907+0000", 2024-09-20T22:45:38.340 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 113.822580449, 2024-09-20T22:45:38.340 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0070184640000000003, 2024-09-20T22:45:38.341 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:45:38.341 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:45:38.341 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:45:38.341 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:45:38.341 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.341 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:45:38.341 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.456907+0000", 2024-09-20T22:45:38.341 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:45:38.341 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.341 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.341 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:45:38.341 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.456907+0000", 2024-09-20T22:45:38.341 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:45:38.341 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.341 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.342 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:45:38.342 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.456909+0000", 2024-09-20T22:45:38.342 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.384e-06 2024-09-20T22:45:38.342 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.342 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.342 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:45:38.342 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.458084+0000", 2024-09-20T22:45:38.342 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00117508 2024-09-20T22:45:38.342 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.342 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.342 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:45:38.342 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.458121+0000", 2024-09-20T22:45:38.342 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.6711999999999997e-05 2024-09-20T22:45:38.342 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.343 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.343 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:45:38.343 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.458135+0000", 2024-09-20T22:45:38.343 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4508999999999999e-05 2024-09-20T22:45:38.343 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.343 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.343 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:45:38.343 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.458226+0000", 2024-09-20T22:45:38.343 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.0107999999999999e-05 2024-09-20T22:45:38.343 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.343 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.343 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:45:38.343 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.458246+0000", 2024-09-20T22:45:38.343 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.0891999999999999e-05 2024-09-20T22:45:38.344 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.344 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.344 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:45:38.344 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.463881+0000", 2024-09-20T22:45:38.344 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0056345859999999996 2024-09-20T22:45:38.344 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.344 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.344 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:45:38.344 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.463925+0000", 2024-09-20T22:45:38.344 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.4193000000000002e-05 2024-09-20T22:45:38.344 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:45:38.344 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:45:38.344 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:45:38.345 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.345 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.345 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:894 7.a e155/94 7:50097c2d:::smithi08477203-23:head v 155'54, mlcod=143'50)", 2024-09-20T22:45:38.345 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:54.450745+0000", 2024-09-20T22:45:38.345 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 103.828742115, 2024-09-20T22:45:38.345 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0091901029999999998, 2024-09-20T22:45:38.345 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:45:38.345 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:45:38.345 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:45:38.345 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:45:38.345 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.345 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:45:38.345 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:54.450745+0000", 2024-09-20T22:45:38.346 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:45:38.346 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.346 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.346 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:45:38.346 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:54.450745+0000", 2024-09-20T22:45:38.346 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:45:38.346 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.346 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.346 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:45:38.346 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:54.450745+0000", 2024-09-20T22:45:38.346 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.4200000000000002e-07 2024-09-20T22:45:38.346 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.346 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.346 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:45:38.347 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:54.456336+0000", 2024-09-20T22:45:38.347 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.005590589 2024-09-20T22:45:38.347 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.347 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.347 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:45:38.347 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:54.456353+0000", 2024-09-20T22:45:38.347 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.7187e-05 2024-09-20T22:45:38.347 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.347 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.347 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:45:38.348 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:54.456362+0000", 2024-09-20T22:45:38.348 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.7409999999999998e-06 2024-09-20T22:45:38.348 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.348 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.348 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:45:38.348 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:54.456418+0000", 2024-09-20T22:45:38.348 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.6431000000000002e-05 2024-09-20T22:45:38.348 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.348 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.348 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:45:38.348 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:54.456438+0000", 2024-09-20T22:45:38.348 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9718999999999999e-05 2024-09-20T22:45:38.348 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.348 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.348 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:45:38.349 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:54.459899+0000", 2024-09-20T22:45:38.349 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.003461254 2024-09-20T22:45:38.350 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.350 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.350 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:45:38.350 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:54.459935+0000", 2024-09-20T22:45:38.350 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.5939999999999998e-05 2024-09-20T22:45:38.350 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:45:38.350 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:45:38.350 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:45:38.351 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.351 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.351 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:901 7.e e156/94 7:7c62ced4:::smithi08477203-17:head v 156'42, mlcod=154'40)", 2024-09-20T22:45:38.351 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:55.635897+0000", 2024-09-20T22:45:38.351 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 102.643589722, 2024-09-20T22:45:38.351 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0082336760000000005, 2024-09-20T22:45:38.351 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:45:38.351 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:45:38.351 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:45:38.351 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:45:38.351 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.351 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:45:38.351 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:55.635897+0000", 2024-09-20T22:45:38.351 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:45:38.351 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.351 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.352 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:45:38.352 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:55.635897+0000", 2024-09-20T22:45:38.352 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:45:38.352 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.352 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.352 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:45:38.352 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:55.635898+0000", 2024-09-20T22:45:38.352 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.23e-07 2024-09-20T22:45:38.352 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.352 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.352 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:45:38.352 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:55.640084+0000", 2024-09-20T22:45:38.352 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0041865499999999998 2024-09-20T22:45:38.352 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.353 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.353 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:45:38.353 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:55.640118+0000", 2024-09-20T22:45:38.353 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.4093e-05 2024-09-20T22:45:38.353 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.353 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.353 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:45:38.353 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:55.640129+0000", 2024-09-20T22:45:38.353 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0281e-05 2024-09-20T22:45:38.353 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.353 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.353 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:45:38.353 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:55.640230+0000", 2024-09-20T22:45:38.353 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000101538 2024-09-20T22:45:38.353 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.354 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.354 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:45:38.354 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:55.640251+0000", 2024-09-20T22:45:38.354 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.1293000000000001e-05 2024-09-20T22:45:38.354 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.354 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.354 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:45:38.354 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:55.644088+0000", 2024-09-20T22:45:38.354 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.003836178 2024-09-20T22:45:38.354 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.354 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.354 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:45:38.354 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:55.644131+0000", 2024-09-20T22:45:38.354 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.3519999999999997e-05 2024-09-20T22:45:38.355 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:45:38.355 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:45:38.355 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:45:38.355 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.355 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.355 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:1034 7.b e166/94 7:d960cad4:::smithi08477203-18:head v 166'64, mlcod=163'61)", 2024-09-20T22:45:38.355 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:44:05.166370+0000", 2024-09-20T22:45:38.355 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 93.113116829999996, 2024-09-20T22:45:38.355 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0065985330000000002, 2024-09-20T22:45:38.355 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:45:38.355 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:45:38.355 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:45:38.355 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:45:38.355 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.355 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:45:38.356 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:05.166370+0000", 2024-09-20T22:45:38.356 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:45:38.356 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.356 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.356 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:45:38.356 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:05.166370+0000", 2024-09-20T22:45:38.356 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:45:38.356 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.356 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.356 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:45:38.356 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:05.166371+0000", 2024-09-20T22:45:38.356 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.2900000000000003e-07 2024-09-20T22:45:38.356 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.356 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.356 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:45:38.357 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:05.170473+0000", 2024-09-20T22:45:38.357 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0041020919999999999 2024-09-20T22:45:38.357 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.357 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.357 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:45:38.357 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:05.170486+0000", 2024-09-20T22:45:38.357 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2847000000000001e-05 2024-09-20T22:45:38.357 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.357 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.357 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:45:38.357 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:05.170493+0000", 2024-09-20T22:45:38.357 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.6630000000000004e-06 2024-09-20T22:45:38.357 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.357 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.358 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:45:38.358 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:05.170546+0000", 2024-09-20T22:45:38.358 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.3547000000000001e-05 2024-09-20T22:45:38.358 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.358 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.358 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:45:38.358 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:05.170562+0000", 2024-09-20T22:45:38.358 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.5515e-05 2024-09-20T22:45:38.358 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.358 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.358 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:45:38.358 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:05.172925+0000", 2024-09-20T22:45:38.358 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.002363633 2024-09-20T22:45:38.358 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.359 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.359 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:45:38.359 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:05.172969+0000", 2024-09-20T22:45:38.359 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.3606999999999998e-05 2024-09-20T22:45:38.359 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:45:38.359 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:45:38.359 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:45:38.359 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.359 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.359 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:1316 7.0 7:05bb442d:::smithi08477203-24:head [stat,delete] snapc 2b=[2b,2a] ondisk+write+known_if_redirected+supports_pool_eio e180)", 2024-09-20T22:45:38.359 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:44:19.225492+0000", 2024-09-20T22:45:38.359 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 79.053995150999995, 2024-09-20T22:45:38.359 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0083403250000000009, 2024-09-20T22:45:38.359 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:45:38.359 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:45:38.360 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:45:38.360 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:45:38.360 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:45:38.360 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:45:38.360 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 1316 2024-09-20T22:45:38.360 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.360 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:45:38.360 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.360 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:45:38.360 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:19.225492+0000", 2024-09-20T22:45:38.360 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:45:38.360 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.360 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.360 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:45:38.361 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:19.225492+0000", 2024-09-20T22:45:38.361 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:45:38.361 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.361 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.361 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:45:38.361 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:19.225495+0000", 2024-09-20T22:45:38.361 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.96e-06 2024-09-20T22:45:38.361 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.361 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.361 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:45:38.361 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:19.225508+0000", 2024-09-20T22:45:38.361 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3343000000000001e-05 2024-09-20T22:45:38.362 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.362 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.362 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:45:38.362 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:19.225542+0000", 2024-09-20T22:45:38.362 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.3973000000000002e-05 2024-09-20T22:45:38.362 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.362 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.362 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:45:38.362 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:19.225570+0000", 2024-09-20T22:45:38.362 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.7713999999999999e-05 2024-09-20T22:45:38.362 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.362 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.362 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:45:38.363 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:19.225655+0000", 2024-09-20T22:45:38.363 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.4944999999999994e-05 2024-09-20T22:45:38.363 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.363 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.363 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op must wait for map", 2024-09-20T22:45:38.363 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:19.225660+0000", 2024-09-20T22:45:38.363 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.5470000000000004e-06 2024-09-20T22:45:38.363 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.363 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.363 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:45:38.363 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:19.232618+0000", 2024-09-20T22:45:38.363 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0069571069999999997 2024-09-20T22:45:38.363 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.363 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.364 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:45:38.364 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:19.232717+0000", 2024-09-20T22:45:38.364 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.9306000000000001e-05 2024-09-20T22:45:38.364 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.364 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.364 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:45:38.364 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:19.233832+0000", 2024-09-20T22:45:38.364 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0011154299999999999 2024-09-20T22:45:38.364 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:45:38.364 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:45:38.364 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:45:38.364 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.364 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.364 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:1527 7.17 e197/139 7:ea720b85:::smithi08477203-26:head v 197'159, mlcod=187'156)", 2024-09-20T22:45:38.364 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:44:36.209560+0000", 2024-09-20T22:45:38.365 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 62.069927524000001, 2024-09-20T22:45:38.365 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.010263069999999999, 2024-09-20T22:45:38.365 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:45:38.365 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:45:38.365 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:45:38.365 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:45:38.365 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.365 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:45:38.365 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.209560+0000", 2024-09-20T22:45:38.365 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:45:38.365 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.366 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.366 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:45:38.366 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.209560+0000", 2024-09-20T22:45:38.366 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:45:38.366 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.366 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.366 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:45:38.366 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.209560+0000", 2024-09-20T22:45:38.366 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.0800000000000001e-07 2024-09-20T22:45:38.366 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.366 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.366 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:45:38.367 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.212844+0000", 2024-09-20T22:45:38.367 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.003283907 2024-09-20T22:45:38.367 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.367 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.367 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:45:38.367 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.212867+0000", 2024-09-20T22:45:38.367 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.3229000000000001e-05 2024-09-20T22:45:38.367 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.367 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.367 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:45:38.368 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.212878+0000", 2024-09-20T22:45:38.368 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0998e-05 2024-09-20T22:45:38.368 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.368 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.368 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:45:38.368 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.212971+0000", 2024-09-20T22:45:38.368 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.2872000000000001e-05 2024-09-20T22:45:38.368 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.368 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.368 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:45:38.368 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.212996+0000", 2024-09-20T22:45:38.368 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.4921000000000001e-05 2024-09-20T22:45:38.368 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.368 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.368 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:45:38.369 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.219758+0000", 2024-09-20T22:45:38.369 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0067625139999999999 2024-09-20T22:45:38.369 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.369 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.369 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:45:38.369 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.219823+0000", 2024-09-20T22:45:38.369 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.4420999999999999e-05 2024-09-20T22:45:38.369 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:45:38.369 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:45:38.369 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:45:38.369 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.369 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.369 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:1550 7.17 e198/139 7:ed54065f:::smithi08477203-12:head v 198'160, mlcod=187'157)", 2024-09-20T22:45:38.369 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:44:37.246938+0000", 2024-09-20T22:45:38.369 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 61.032548785000003, 2024-09-20T22:45:38.369 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.006923935, 2024-09-20T22:45:38.370 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:45:38.370 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:45:38.370 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:45:38.371 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:45:38.372 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:45:38.372 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:45:38.372 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.372 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:45:38.372 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:37.246938+0000", 2024-09-20T22:45:38.372 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:45:38.372 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.372 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.372 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:45:38.372 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:37.246938+0000", 2024-09-20T22:45:38.372 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:45:38.372 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.372 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.372 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:45:38.373 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:37.246939+0000", 2024-09-20T22:45:38.373 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.1299999999999999e-07 2024-09-20T22:45:38.373 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.373 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.373 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:45:38.373 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:37.248285+0000", 2024-09-20T22:45:38.373 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0013466369999999999 2024-09-20T22:45:38.373 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.373 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.373 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:45:38.373 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:37.248309+0000", 2024-09-20T22:45:38.373 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.4088e-05 2024-09-20T22:45:38.373 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.373 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.373 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:45:38.374 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:37.248322+0000", 2024-09-20T22:45:38.374 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2329e-05 2024-09-20T22:45:38.374 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.374 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.374 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:45:38.374 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:37.248425+0000", 2024-09-20T22:45:38.374 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00010300400000000001 2024-09-20T22:45:38.374 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.374 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.374 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:45:38.374 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:37.248443+0000", 2024-09-20T22:45:38.374 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.8522e-05 2024-09-20T22:45:38.374 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.374 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.375 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:45:38.375 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:37.253782+0000", 2024-09-20T22:45:38.375 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.005339122 2024-09-20T22:45:38.375 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.375 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.375 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:45:38.375 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:37.253862+0000", 2024-09-20T22:45:38.375 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.0019999999999996e-05 2024-09-20T22:45:38.375 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:45:38.375 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:45:38.375 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:45:38.375 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.375 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.375 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:1665 7.0 7:064f66ab:::smithi08477203-45:head [copy-from ver 0] snapc 3a=[3a,39,37] ondisk+write+known_if_redirected+supports_pool_eio e209)", 2024-09-20T22:45:38.375 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:44:48.141884+0000", 2024-09-20T22:45:38.376 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 50.137603581, 2024-09-20T22:45:38.376 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0074924060000000001, 2024-09-20T22:45:38.376 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:45:38.376 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:45:38.376 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:45:38.376 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:45:38.376 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:45:38.376 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:45:38.376 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 1665 2024-09-20T22:45:38.376 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.376 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:45:38.376 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.376 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:45:38.376 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:48.141884+0000", 2024-09-20T22:45:38.376 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:45:38.376 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.377 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.377 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:45:38.377 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:48.141884+0000", 2024-09-20T22:45:38.377 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:45:38.377 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.377 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.377 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:45:38.377 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:48.141886+0000", 2024-09-20T22:45:38.377 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.1579999999999999e-06 2024-09-20T22:45:38.377 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.377 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.377 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:45:38.378 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:48.141894+0000", 2024-09-20T22:45:38.378 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.7530000000000007e-06 2024-09-20T22:45:38.378 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.378 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.378 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:45:38.378 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:48.141911+0000", 2024-09-20T22:45:38.378 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.6759e-05 2024-09-20T22:45:38.378 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.378 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.378 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:45:38.378 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:48.141921+0000", 2024-09-20T22:45:38.378 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0209e-05 2024-09-20T22:45:38.378 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.379 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.379 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:45:38.379 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:48.141987+0000", 2024-09-20T22:45:38.379 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.5953999999999999e-05 2024-09-20T22:45:38.379 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.379 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.379 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op must wait for map", 2024-09-20T22:45:38.379 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:48.141995+0000", 2024-09-20T22:45:38.379 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.3749999999999997e-06 2024-09-20T22:45:38.379 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.379 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.379 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:45:38.380 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:48.143638+0000", 2024-09-20T22:45:38.380 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001642798 2024-09-20T22:45:38.380 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.380 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.380 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:45:38.380 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:48.143738+0000", 2024-09-20T22:45:38.380 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000100723 2024-09-20T22:45:38.380 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.380 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.380 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:45:38.380 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:48.149376+0000", 2024-09-20T22:45:38.380 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0056376769999999998 2024-09-20T22:45:38.381 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:45:38.381 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:45:38.381 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:45:38.381 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.381 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.381 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:1666 7.0 7:064f66ab:::smithi08477203-45:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e209)", 2024-09-20T22:45:38.381 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:44:48.141945+0000", 2024-09-20T22:45:38.381 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 50.137542467000003, 2024-09-20T22:45:38.381 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0076299330000000002, 2024-09-20T22:45:38.381 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:45:38.381 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:45:38.381 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:45:38.381 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:45:38.382 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:45:38.382 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:45:38.382 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 1666 2024-09-20T22:45:38.382 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.382 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:45:38.382 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.382 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:45:38.382 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:48.141945+0000", 2024-09-20T22:45:38.382 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:45:38.382 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.382 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.382 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:45:38.383 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:48.141945+0000", 2024-09-20T22:45:38.383 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:45:38.383 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.383 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.383 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:45:38.383 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:48.141945+0000", 2024-09-20T22:45:38.383 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.1900000000000002e-07 2024-09-20T22:45:38.383 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.383 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.383 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:45:38.383 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:48.141949+0000", 2024-09-20T22:45:38.383 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.19e-06 2024-09-20T22:45:38.384 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.384 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.384 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:45:38.384 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:48.141957+0000", 2024-09-20T22:45:38.384 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.0579999999999998e-06 2024-09-20T22:45:38.384 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.384 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.384 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:45:38.384 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:48.141963+0000", 2024-09-20T22:45:38.384 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.9920000000000004e-06 2024-09-20T22:45:38.384 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.384 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.384 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:45:38.384 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:48.142051+0000", 2024-09-20T22:45:38.385 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.8295999999999996e-05 2024-09-20T22:45:38.385 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.385 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.385 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting_for_map not empty", 2024-09-20T22:45:38.385 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:48.142058+0000", 2024-09-20T22:45:38.385 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.7989999999999996e-06 2024-09-20T22:45:38.385 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.385 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.385 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:45:38.385 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:48.143995+0000", 2024-09-20T22:45:38.385 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001937624 2024-09-20T22:45:38.385 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.385 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.386 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for blocked object", 2024-09-20T22:45:38.386 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:48.144501+0000", 2024-09-20T22:45:38.386 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00050535399999999996 2024-09-20T22:45:38.386 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.386 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.386 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:45:38.386 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:48.149458+0000", 2024-09-20T22:45:38.386 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0049577830000000003 2024-09-20T22:45:38.386 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.386 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.386 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:45:38.386 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:48.149513+0000", 2024-09-20T22:45:38.386 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.5012000000000001e-05 2024-09-20T22:45:38.386 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.387 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.387 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:45:38.387 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:48.149575+0000", 2024-09-20T22:45:38.387 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.1205999999999996e-05 2024-09-20T22:45:38.387 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:45:38.387 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:45:38.387 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:45:38.387 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.387 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.387 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:1725 7.b e217/94 7:d3ca3619:::smithi08477203-39:head v 217'131, mlcod=208'128)", 2024-09-20T22:45:38.387 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:44:56.219920+0000", 2024-09-20T22:45:38.387 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 42.059567508000001, 2024-09-20T22:45:38.388 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0060624069999999997, 2024-09-20T22:45:38.388 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:45:38.388 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:45:38.388 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:45:38.388 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:45:38.388 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.388 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:45:38.388 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:56.219920+0000", 2024-09-20T22:45:38.388 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:45:38.388 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.388 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.388 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:45:38.388 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:56.219920+0000", 2024-09-20T22:45:38.388 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:45:38.389 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.389 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.389 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:45:38.389 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:56.219920+0000", 2024-09-20T22:45:38.389 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.4700000000000001e-07 2024-09-20T22:45:38.389 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.389 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.389 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:45:38.389 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:56.223560+0000", 2024-09-20T22:45:38.389 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0036394320000000002 2024-09-20T22:45:38.389 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.389 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.389 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:45:38.390 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:56.223568+0000", 2024-09-20T22:45:38.390 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.5990000000000002e-06 2024-09-20T22:45:38.390 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.390 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.390 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:45:38.390 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:56.223576+0000", 2024-09-20T22:45:38.390 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.3699999999999997e-06 2024-09-20T22:45:38.390 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.390 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.390 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:45:38.390 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:56.223628+0000", 2024-09-20T22:45:38.390 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.2859999999999999e-05 2024-09-20T22:45:38.390 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.390 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.390 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:45:38.391 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:56.223649+0000", 2024-09-20T22:45:38.391 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.0095999999999999e-05 2024-09-20T22:45:38.391 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.391 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.391 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:45:38.391 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:56.225942+0000", 2024-09-20T22:45:38.391 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0022932759999999999 2024-09-20T22:45:38.391 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.391 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.391 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:45:38.391 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:56.225982+0000", 2024-09-20T22:45:38.391 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.0227e-05 2024-09-20T22:45:38.391 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:45:38.391 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:45:38.391 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:45:38.392 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.392 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.392 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:1799 7.17 e229/139 7:ea720b85:::smithi08477203-26:head v 229'168, mlcod=222'164)", 2024-09-20T22:45:38.392 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:45:08.172029+0000", 2024-09-20T22:45:38.392 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 30.107458213000001, 2024-09-20T22:45:38.392 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0063782379999999996, 2024-09-20T22:45:38.392 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:45:38.392 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:45:38.392 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:45:38.392 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:45:38.392 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.392 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:45:38.392 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.172029+0000", 2024-09-20T22:45:38.392 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:45:38.392 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.393 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.393 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:45:38.393 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.172029+0000", 2024-09-20T22:45:38.393 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:45:38.393 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.393 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.393 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:45:38.393 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.172029+0000", 2024-09-20T22:45:38.393 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.1100000000000002e-07 2024-09-20T22:45:38.393 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.393 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.393 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:45:38.393 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.172410+0000", 2024-09-20T22:45:38.393 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00038106799999999998 2024-09-20T22:45:38.393 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.394 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.394 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:45:38.394 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.172421+0000", 2024-09-20T22:45:38.394 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0431999999999999e-05 2024-09-20T22:45:38.394 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.394 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.394 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:45:38.394 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.172427+0000", 2024-09-20T22:45:38.394 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.0689999999999998e-06 2024-09-20T22:45:38.394 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.394 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.394 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:45:38.394 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.172478+0000", 2024-09-20T22:45:38.394 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.1427000000000001e-05 2024-09-20T22:45:38.394 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.395 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.395 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:45:38.395 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.172498+0000", 2024-09-20T22:45:38.395 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9398999999999998e-05 2024-09-20T22:45:38.395 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.395 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.395 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:45:38.395 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.178365+0000", 2024-09-20T22:45:38.395 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0058675400000000001 2024-09-20T22:45:38.395 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.395 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.395 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:45:38.395 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.178407+0000", 2024-09-20T22:45:38.395 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.1992e-05 2024-09-20T22:45:38.395 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:45:38.396 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:45:38.396 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:45:38.396 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.396 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.396 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:1825 7.b e230/94 7:d960cad4:::smithi08477203-18:head v 230'138, mlcod=229'135)", 2024-09-20T22:45:38.396 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:45:09.177560+0000", 2024-09-20T22:45:38.396 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 29.101927660000001, 2024-09-20T22:45:38.396 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0086763780000000002, 2024-09-20T22:45:38.396 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:45:38.396 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:45:38.396 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:45:38.396 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:45:38.396 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.396 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:45:38.397 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:09.177560+0000", 2024-09-20T22:45:38.397 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:45:38.397 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.397 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.397 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:45:38.397 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:09.177560+0000", 2024-09-20T22:45:38.397 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:45:38.397 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.397 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.397 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:45:38.397 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:09.177560+0000", 2024-09-20T22:45:38.397 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.0599999999999999e-07 2024-09-20T22:45:38.397 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.397 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.398 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:45:38.398 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:09.178522+0000", 2024-09-20T22:45:38.398 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00096277000000000005 2024-09-20T22:45:38.398 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.398 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.398 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:45:38.398 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:09.178536+0000", 2024-09-20T22:45:38.398 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3886999999999999e-05 2024-09-20T22:45:38.398 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.398 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.398 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:45:38.398 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:09.178547+0000", 2024-09-20T22:45:38.398 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0526999999999999e-05 2024-09-20T22:45:38.398 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.399 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.399 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:45:38.399 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:09.178614+0000", 2024-09-20T22:45:38.399 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.7547999999999998e-05 2024-09-20T22:45:38.399 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.399 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.399 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:45:38.399 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:09.178638+0000", 2024-09-20T22:45:38.399 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.3190999999999999e-05 2024-09-20T22:45:38.399 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.399 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.399 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:45:38.399 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:09.186193+0000", 2024-09-20T22:45:38.399 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0075549559999999998 2024-09-20T22:45:38.399 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:38.400 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:38.400 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:45:38.400 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:09.186236+0000", 2024-09-20T22:45:38.400 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.3293e-05 2024-09-20T22:45:38.400 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:45:38.400 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:45:38.400 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:45:38.400 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:45:38.400 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:45:38.400 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-20T22:45:38.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:38.401 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:38.402 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:45:38.435 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:45:38 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: ::ffff:172.21.15.154 - - [20/Sep/2024:22:45:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-20T22:45:38.436 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:38 smithi084 ceph-mon[40024]: 7.9 scrub starts 2024-09-20T22:45:38.436 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:38 smithi084 ceph-mon[40024]: 7.9 scrub ok 2024-09-20T22:45:38.436 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:38 smithi084 ceph-mon[40024]: 1.11 scrub starts 2024-09-20T22:45:38.436 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:38 smithi084 ceph-mon[40024]: 1.11 scrub ok 2024-09-20T22:45:38.436 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:38 smithi084 ceph-mon[40024]: 1.e scrub starts 2024-09-20T22:45:38.436 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:38 smithi084 ceph-mon[40024]: 1.e scrub ok 2024-09-20T22:45:38.436 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:38 smithi084 ceph-mon[40024]: 3.19 scrub starts 2024-09-20T22:45:38.437 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:38 smithi084 ceph-mon[40024]: 3.19 scrub ok 2024-09-20T22:45:38.437 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:38 smithi084 ceph-mon[40024]: osdmap e258: 8 total, 8 up, 7 in 2024-09-20T22:45:38.437 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:38 smithi084 ceph-mon[40024]: 1.12 scrub starts 2024-09-20T22:45:38.437 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:38 smithi084 ceph-mon[40024]: 1.12 scrub ok 2024-09-20T22:45:38.437 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:38 smithi084 ceph-mon[40024]: 1.d scrub starts 2024-09-20T22:45:38.437 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:38 smithi084 ceph-mon[40024]: 1.d scrub ok 2024-09-20T22:45:38.437 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:38 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:45:38.437 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:38 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:45:38.437 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:38 smithi084 ceph-mon[40024]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:45:38.437 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:38 smithi084 ceph-mon[40024]: osdmap e259: 8 total, 8 up, 7 in 2024-09-20T22:45:38.437 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:38 smithi084 ceph-mon[29774]: 7.9 scrub starts 2024-09-20T22:45:38.437 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:38 smithi084 ceph-mon[29774]: 7.9 scrub ok 2024-09-20T22:45:38.438 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:38 smithi084 ceph-mon[29774]: 1.11 scrub starts 2024-09-20T22:45:38.438 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:38 smithi084 ceph-mon[29774]: 1.11 scrub ok 2024-09-20T22:45:38.438 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:38 smithi084 ceph-mon[29774]: 1.e scrub starts 2024-09-20T22:45:38.438 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:38 smithi084 ceph-mon[29774]: 1.e scrub ok 2024-09-20T22:45:38.438 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:38 smithi084 ceph-mon[29774]: 3.19 scrub starts 2024-09-20T22:45:38.438 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:38 smithi084 ceph-mon[29774]: 3.19 scrub ok 2024-09-20T22:45:38.438 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:38 smithi084 ceph-mon[29774]: osdmap e258: 8 total, 8 up, 7 in 2024-09-20T22:45:38.438 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:38 smithi084 ceph-mon[29774]: 1.12 scrub starts 2024-09-20T22:45:38.438 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:38 smithi084 ceph-mon[29774]: 1.12 scrub ok 2024-09-20T22:45:38.438 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:38 smithi084 ceph-mon[29774]: 1.d scrub starts 2024-09-20T22:45:38.438 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:38 smithi084 ceph-mon[29774]: 1.d scrub ok 2024-09-20T22:45:38.438 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:38 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:45:38.438 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:38 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:45:38.438 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:38 smithi084 ceph-mon[29774]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:45:38.439 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:38 smithi084 ceph-mon[29774]: osdmap e259: 8 total, 8 up, 7 in 2024-09-20T22:45:38.502 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:45:38.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:38 smithi154 ceph-mon[33264]: 7.9 scrub starts 2024-09-20T22:45:38.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:38 smithi154 ceph-mon[33264]: 7.9 scrub ok 2024-09-20T22:45:38.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:38 smithi154 ceph-mon[33264]: 1.11 scrub starts 2024-09-20T22:45:38.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:38 smithi154 ceph-mon[33264]: 1.11 scrub ok 2024-09-20T22:45:38.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:38 smithi154 ceph-mon[33264]: 1.e scrub starts 2024-09-20T22:45:38.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:38 smithi154 ceph-mon[33264]: 1.e scrub ok 2024-09-20T22:45:38.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:38 smithi154 ceph-mon[33264]: 3.19 scrub starts 2024-09-20T22:45:38.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:38 smithi154 ceph-mon[33264]: 3.19 scrub ok 2024-09-20T22:45:38.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:38 smithi154 ceph-mon[33264]: osdmap e258: 8 total, 8 up, 7 in 2024-09-20T22:45:38.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:38 smithi154 ceph-mon[33264]: 1.12 scrub starts 2024-09-20T22:45:38.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:38 smithi154 ceph-mon[33264]: 1.12 scrub ok 2024-09-20T22:45:38.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:38 smithi154 ceph-mon[33264]: 1.d scrub starts 2024-09-20T22:45:38.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:38 smithi154 ceph-mon[33264]: 1.d scrub ok 2024-09-20T22:45:38.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:38 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:45:38.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:38 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:45:38.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:38 smithi154 ceph-mon[33264]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:45:38.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:38 smithi154 ceph-mon[33264]: osdmap e259: 8 total, 8 up, 7 in 2024-09-20T22:45:38.524 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:38.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:38.525 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:45:38.626 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:45:38.647 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:38.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:38.647 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:45:38.715 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:38 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mon-a[29770]: 2024-09-20T22:45:38.472+0000 7f26ad5e8640 -1 mon.a@0(leader).osd e259 definitely_dead 0 2024-09-20T22:45:38.748 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:45:38.769 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:38.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:38.770 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:45:38.871 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:45:38.892 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:38.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:38.893 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:45:38.993 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:45:39.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:39.015 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:39.015 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:45:39.116 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:45:39.137 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:39.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:39.139 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:45:39.186 INFO:teuthology.orchestra.run.smithi084.stderr:marked out osd.0. 2024-09-20T22:45:39.193 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is unset 2024-09-20T22:45:39.195 INFO:tasks.rados.rados.0.smithi084.stdout:893: done (4 left) 2024-09-20T22:45:39.196 INFO:tasks.rados.rados.0.smithi084.stdout:894: read oid 28 snap 69 2024-09-20T22:45:39.196 INFO:tasks.rados.rados.0.smithi084.stdout:894: expect deleted 2024-09-20T22:45:39.196 INFO:tasks.rados.rados.0.smithi084.stdout:895: read oid 33 snap -1 2024-09-20T22:45:39.196 INFO:tasks.rados.rados.0.smithi084.stdout:895: expect (ObjNum 229 snap 48 seq_num 229) 2024-09-20T22:45:39.196 INFO:tasks.rados.rados.0.smithi084.stdout:896: delete oid 45 current snap is 85 2024-09-20T22:45:39.196 INFO:tasks.rados.rados.0.smithi084.stdout:888: finishing rollback tid 1 to smithi08477203-15 2024-09-20T22:45:39.196 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 15 v 168 (ObjNum 49 snap 0 seq_num 1953066355) dirty dne 2024-09-20T22:45:39.197 INFO:tasks.rados.rados.0.smithi084.stdout:890: finishing write tid 1 to smithi08477203-21 2024-09-20T22:45:39.197 INFO:tasks.rados.rados.0.smithi084.stdout:890: finishing write tid 2 to smithi08477203-21 2024-09-20T22:45:39.197 INFO:tasks.rados.rados.0.smithi084.stdout:890: finishing write tid 3 to smithi08477203-21 2024-09-20T22:45:39.197 INFO:tasks.rados.rados.0.smithi084.stdout:890: finishing write tid 4 to smithi08477203-21 2024-09-20T22:45:39.197 INFO:tasks.rados.rados.0.smithi084.stdout:890: finishing write tid 5 to smithi08477203-21 2024-09-20T22:45:39.197 INFO:tasks.rados.rados.0.smithi084.stdout:890: oid 21 updating version 0 to 246 2024-09-20T22:45:39.198 INFO:tasks.rados.rados.0.smithi084.stdout:890: oid 21 updating version 246 to 247 2024-09-20T22:45:39.198 INFO:tasks.rados.rados.0.smithi084.stdout:890: oid 21 updating version 247 to 249 2024-09-20T22:45:39.198 INFO:tasks.rados.rados.0.smithi084.stdout:890: oid 21 version 249 is already newer than 248 2024-09-20T22:45:39.198 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 21 v 249 (ObjNum 332 snap 85 seq_num 332) dirty exists 2024-09-20T22:45:39.198 INFO:tasks.rados.rados.0.smithi084.stdout:890: left oid 21 (ObjNum 332 snap 85 seq_num 332) 2024-09-20T22:45:39.198 INFO:tasks.rados.rados.0.smithi084.stdout:891: expect (ObjNum 282 snap 71 seq_num 282) 2024-09-20T22:45:39.240 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-20T22:45:39.240 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:45:39.261 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:39.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:39.262 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:45:39.362 INFO:tasks.rados.rados.0.smithi084.stdout:892: finishing write tid 1 to smithi08477203-41 2024-09-20T22:45:39.363 INFO:tasks.rados.rados.0.smithi084.stdout:892: finishing write tid 2 to smithi08477203-41 2024-09-20T22:45:39.363 INFO:tasks.rados.rados.0.smithi084.stdout:892: finishing write tid 3 to smithi08477203-41 2024-09-20T22:45:39.363 INFO:tasks.rados.rados.0.smithi084.stdout:892: finishing write tid 4 to smithi08477203-41 2024-09-20T22:45:39.363 INFO:tasks.rados.rados.0.smithi084.stdout:892: finishing write tid 5 to smithi08477203-41 2024-09-20T22:45:39.363 INFO:tasks.rados.rados.0.smithi084.stdout:892: oid 41 updating version 0 to 58 2024-09-20T22:45:39.363 INFO:tasks.rados.rados.0.smithi084.stdout:892: oid 41 updating version 58 to 60 2024-09-20T22:45:39.363 INFO:tasks.rados.rados.0.smithi084.stdout:892: oid 41 version 60 is already newer than 57 2024-09-20T22:45:39.363 INFO:tasks.rados.rados.0.smithi084.stdout:892: oid 41 version 60 is already newer than 59 2024-09-20T22:45:39.363 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 41 v 60 (ObjNum 333 snap 85 seq_num 333) dirty exists 2024-09-20T22:45:39.363 INFO:tasks.rados.rados.0.smithi084.stdout:892: left oid 41 (ObjNum 333 snap 85 seq_num 333) 2024-09-20T22:45:39.363 INFO:tasks.rados.rados.0.smithi084.stdout:888: done (6 left) 2024-09-20T22:45:39.363 INFO:tasks.rados.rados.0.smithi084.stdout:890: done (5 left) 2024-09-20T22:45:39.363 INFO:tasks.rados.rados.0.smithi084.stdout:891: done (4 left) 2024-09-20T22:45:39.364 INFO:tasks.rados.rados.0.smithi084.stdout:892: done (3 left) 2024-09-20T22:45:39.364 INFO:tasks.rados.rados.0.smithi084.stdout:896: done (2 left) 2024-09-20T22:45:39.364 INFO:tasks.rados.rados.0.smithi084.stdout:897: copy_from oid 32 from oid 19 current snap is 85 2024-09-20T22:45:39.364 INFO:tasks.rados.rados.0.smithi084.stdout:895: expect (ObjNum 229 snap 48 seq_num 229) 2024-09-20T22:45:39.364 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:45:39.384 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:39.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:39.385 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:45:39.459 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:39 smithi084 ceph-mon[40024]: pgmap v289: 178 pgs: 3 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 10 remapped+peering, 161 active+clean; 198 MiB data, 696 MiB used, 625 GiB / 626 GiB avail 2024-09-20T22:45:39.459 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:39 smithi084 ceph-mon[40024]: 1.a scrub starts 2024-09-20T22:45:39.459 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:39 smithi084 ceph-mon[40024]: 1.a scrub ok 2024-09-20T22:45:39.459 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:39 smithi084 ceph-mon[40024]: 4.13 scrub starts 2024-09-20T22:45:39.459 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:39 smithi084 ceph-mon[40024]: 4.13 scrub ok 2024-09-20T22:45:39.459 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:39 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/1461080971' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["0"]}]: dispatch 2024-09-20T22:45:39.459 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:39 smithi084 ceph-mon[40024]: Client client.admin marked osd.0 out, while it was still marked up 2024-09-20T22:45:39.459 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:39 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/1993296054' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:45:39.459 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:39 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:45:39.460 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:39 smithi084 ceph-mon[29774]: pgmap v289: 178 pgs: 3 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 10 remapped+peering, 161 active+clean; 198 MiB data, 696 MiB used, 625 GiB / 626 GiB avail 2024-09-20T22:45:39.460 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:39 smithi084 ceph-mon[29774]: 1.a scrub starts 2024-09-20T22:45:39.460 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:39 smithi084 ceph-mon[29774]: 1.a scrub ok 2024-09-20T22:45:39.460 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:39 smithi084 ceph-mon[29774]: 4.13 scrub starts 2024-09-20T22:45:39.460 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:39 smithi084 ceph-mon[29774]: 4.13 scrub ok 2024-09-20T22:45:39.460 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:39 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/1461080971' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["0"]}]: dispatch 2024-09-20T22:45:39.460 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:39 smithi084 ceph-mon[29774]: Client client.admin marked osd.0 out, while it was still marked up 2024-09-20T22:45:39.460 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:39 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/1993296054' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:45:39.460 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:39 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:45:39.485 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:45:39.486 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:45:39.504 INFO:journalctl@ceph.iscsi.iscsi.a.smithi154.stdout:Sep 20 22:45:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-iscsi-iscsi-a[59730]: debug there is no tcmu-runner data available 2024-09-20T22:45:39.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:39 smithi154 ceph-mon[33264]: pgmap v289: 178 pgs: 3 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 10 remapped+peering, 161 active+clean; 198 MiB data, 696 MiB used, 625 GiB / 626 GiB avail 2024-09-20T22:45:39.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:39 smithi154 ceph-mon[33264]: 1.a scrub starts 2024-09-20T22:45:39.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:39 smithi154 ceph-mon[33264]: 1.a scrub ok 2024-09-20T22:45:39.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:39 smithi154 ceph-mon[33264]: 4.13 scrub starts 2024-09-20T22:45:39.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:39 smithi154 ceph-mon[33264]: 4.13 scrub ok 2024-09-20T22:45:39.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:39 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/1461080971' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["0"]}]: dispatch 2024-09-20T22:45:39.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:39 smithi154 ceph-mon[33264]: Client client.admin marked osd.0 out, while it was still marked up 2024-09-20T22:45:39.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:39 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/1993296054' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:45:39.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:39 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:45:39.507 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:39.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:39.508 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:45:39.534 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-20T22:45:39.565 INFO:tasks.rados.rados.0.smithi084.stdout:894: done (2 left) 2024-09-20T22:45:39.565 INFO:tasks.rados.rados.0.smithi084.stdout:895: done (1 left) 2024-09-20T22:45:39.565 INFO:tasks.rados.rados.0.smithi084.stdout:898: write (excl) oid 25 current snap is 85 2024-09-20T22:45:39.565 INFO:tasks.rados.rados.0.smithi084.stdout:898: seq_num 335 ranges {430345=727947,1570056=727439,2699879=469889} 2024-09-20T22:45:39.571 INFO:tasks.rados.rados.0.smithi084.stdout:898: writing smithi08477203-25 from 430345 to 1158292 tid 1 2024-09-20T22:45:39.577 INFO:tasks.rados.rados.0.smithi084.stdout:898: writing smithi08477203-25 from 1570056 to 2297495 tid 2 2024-09-20T22:45:39.583 INFO:tasks.rados.rados.0.smithi084.stdout:898: writing smithi08477203-25 from 2699879 to 3169768 tid 3 2024-09-20T22:45:39.583 INFO:tasks.rados.rados.0.smithi084.stdout:899: copy_from oid 48 from oid 27 current snap is 85 2024-09-20T22:45:39.583 INFO:tasks.rados.rados.0.smithi084.stdout:898: finishing write tid 1 to smithi08477203-25 2024-09-20T22:45:39.583 INFO:tasks.rados.rados.0.smithi084.stdout:898: finishing write tid 2 to smithi08477203-25 2024-09-20T22:45:39.583 INFO:tasks.rados.rados.0.smithi084.stdout:900: rollback oid 46 current snap is 85 2024-09-20T22:45:39.583 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 46 to 60 2024-09-20T22:45:39.583 INFO:tasks.rados.rados.0.smithi084.stdout:901: write oid 45 current snap is 85 2024-09-20T22:45:39.583 INFO:tasks.rados.rados.0.smithi084.stdout:901: seq_num 336 ranges {447152=403196,1512185=587450,2727302=681161} 2024-09-20T22:45:39.587 INFO:tasks.rados.rados.0.smithi084.stdout:901: writing smithi08477203-45 from 447152 to 850348 tid 1 2024-09-20T22:45:39.594 INFO:tasks.rados.rados.0.smithi084.stdout:901: writing smithi08477203-45 from 1512185 to 2099635 tid 2 2024-09-20T22:45:39.602 INFO:tasks.rados.rados.0.smithi084.stdout:901: writing smithi08477203-45 from 2727302 to 3408463 tid 3 2024-09-20T22:45:39.602 INFO:tasks.rados.rados.0.smithi084.stdout:902: delete oid 16 current snap is 85 2024-09-20T22:45:39.602 INFO:tasks.rados.rados.0.smithi084.stdout:900: finishing rollback tid 0 to smithi08477203-46 2024-09-20T22:45:39.602 INFO:tasks.rados.rados.0.smithi084.stdout:900: finishing rollback tid 1 to smithi08477203-46 2024-09-20T22:45:39.602 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 46 v 170 (ObjNum 129 snap 0 seq_num 710361031) dirty dne 2024-09-20T22:45:39.602 INFO:tasks.rados.rados.0.smithi084.stdout:898: finishing write tid 3 to smithi08477203-25 2024-09-20T22:45:39.602 INFO:tasks.rados.rados.0.smithi084.stdout:898: finishing write tid 4 to smithi08477203-25 2024-09-20T22:45:39.602 INFO:tasks.rados.rados.0.smithi084.stdout:898: finishing write tid 5 to smithi08477203-25 2024-09-20T22:45:39.603 INFO:tasks.rados.rados.0.smithi084.stdout:898: oid 25 updating version 0 to 436 2024-09-20T22:45:39.603 INFO:tasks.rados.rados.0.smithi084.stdout:898: oid 25 updating version 436 to 439 2024-09-20T22:45:39.603 INFO:tasks.rados.rados.0.smithi084.stdout:898: oid 25 version 439 is already newer than 438 2024-09-20T22:45:39.603 INFO:tasks.rados.rados.0.smithi084.stdout:898: oid 25 version 439 is already newer than 437 2024-09-20T22:45:39.603 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 25 v 439 (ObjNum 335 snap 85 seq_num 335) dirty exists 2024-09-20T22:45:39.603 INFO:tasks.rados.rados.0.smithi084.stdout:898: left oid 25 (ObjNum 335 snap 85 seq_num 335) 2024-09-20T22:45:39.603 INFO:tasks.rados.rados.0.smithi084.stdout:901: finishing write tid 1 to smithi08477203-45 2024-09-20T22:45:39.603 INFO:tasks.rados.rados.0.smithi084.stdout:901: finishing write tid 2 to smithi08477203-45 2024-09-20T22:45:39.606 INFO:tasks.rados.rados.0.smithi084.stdout:901: finishing write tid 3 to smithi08477203-45 2024-09-20T22:45:39.606 INFO:tasks.rados.rados.0.smithi084.stdout:901: finishing write tid 4 to smithi08477203-45 2024-09-20T22:45:39.606 INFO:tasks.rados.rados.0.smithi084.stdout:901: finishing write tid 5 to smithi08477203-45 2024-09-20T22:45:39.606 INFO:tasks.rados.rados.0.smithi084.stdout:901: oid 45 updating version 0 to 102 2024-09-20T22:45:39.606 INFO:tasks.rados.rados.0.smithi084.stdout:901: oid 45 version 102 is already newer than 101 2024-09-20T22:45:39.606 INFO:tasks.rados.rados.0.smithi084.stdout:901: oid 45 updating version 102 to 103 2024-09-20T22:45:39.606 INFO:tasks.rados.rados.0.smithi084.stdout:901: oid 45 updating version 103 to 104 2024-09-20T22:45:39.606 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 45 v 104 (ObjNum 336 snap 85 seq_num 336) dirty exists 2024-09-20T22:45:39.606 INFO:tasks.rados.rados.0.smithi084.stdout:901: left oid 45 (ObjNum 336 snap 85 seq_num 336) 2024-09-20T22:45:39.609 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:45:39.630 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:39.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:39.630 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:45:39.731 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:45:39.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:39.753 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:39.753 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:45:39.853 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:45:39.875 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:39.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:39.876 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:45:39.977 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:45:39.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:39.999 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:39.999 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:45:40.075 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-20T22:45:40.075 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [], 2024-09-20T22:45:40.075 INFO:teuthology.orchestra.run.smithi154.stdout: "num_ops": 0 2024-09-20T22:45:40.075 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-20T22:45:40.101 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:45:40.151 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:40.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:40.152 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:45:40.246 INFO:tasks.rados.rados.0.smithi084.stdout:897: finishing copy_from racing read to smithi08477203-32 2024-09-20T22:45:40.249 INFO:tasks.rados.rados.0.smithi084.stdout:898: done (5 left) 2024-09-20T22:45:40.249 INFO:tasks.rados.rados.0.smithi084.stdout:900: done (4 left) 2024-09-20T22:45:40.249 INFO:tasks.rados.rados.0.smithi084.stdout:901: done (3 left) 2024-09-20T22:45:40.249 INFO:tasks.rados.rados.0.smithi084.stdout:902: done (2 left) 2024-09-20T22:45:40.249 INFO:tasks.rados.rados.0.smithi084.stdout:903: snap_remove snap 81 2024-09-20T22:45:40.253 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:45:40.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:40 smithi154 ceph-mon[33264]: 3.15 scrub starts 2024-09-20T22:45:40.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:40 smithi154 ceph-mon[33264]: 3.15 scrub ok 2024-09-20T22:45:40.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:40 smithi154 ceph-mon[33264]: 1.3 scrub starts 2024-09-20T22:45:40.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:40 smithi154 ceph-mon[33264]: 1.3 scrub ok 2024-09-20T22:45:40.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:40 smithi154 ceph-mon[33264]: 4.17 scrub starts 2024-09-20T22:45:40.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:40 smithi154 ceph-mon[33264]: 4.17 scrub ok 2024-09-20T22:45:40.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:40 smithi154 ceph-mon[33264]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:45:40.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:40 smithi154 ceph-mon[33264]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T22:45:40.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:40 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/1461080971' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["0"]}]': finished 2024-09-20T22:45:40.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:40 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T22:45:40.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:40 smithi154 ceph-mon[33264]: osdmap e260: 8 total, 8 up, 6 in 2024-09-20T22:45:40.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:40 smithi154 ceph-mon[33264]: 3.13 scrub starts 2024-09-20T22:45:40.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:40 smithi154 ceph-mon[33264]: 3.13 scrub ok 2024-09-20T22:45:40.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:40 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "20"}]: dispatch 2024-09-20T22:45:40.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:40 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "20"}]: dispatch 2024-09-20T22:45:40.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:40 smithi154 ceph-mon[33264]: 6.1 scrub starts 2024-09-20T22:45:40.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:40 smithi154 ceph-mon[33264]: 6.1 scrub ok 2024-09-20T22:45:40.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:40 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "20"}]': finished 2024-09-20T22:45:40.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:40 smithi154 ceph-mon[33264]: osdmap e261: 8 total, 8 up, 6 in 2024-09-20T22:45:40.280 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:40.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:40.281 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:45:40.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:40 smithi084 ceph-mon[29774]: 3.15 scrub starts 2024-09-20T22:45:40.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:40 smithi084 ceph-mon[29774]: 3.15 scrub ok 2024-09-20T22:45:40.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:40 smithi084 ceph-mon[29774]: 1.3 scrub starts 2024-09-20T22:45:40.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:40 smithi084 ceph-mon[29774]: 1.3 scrub ok 2024-09-20T22:45:40.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:40 smithi084 ceph-mon[29774]: 4.17 scrub starts 2024-09-20T22:45:40.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:40 smithi084 ceph-mon[29774]: 4.17 scrub ok 2024-09-20T22:45:40.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:40 smithi084 ceph-mon[29774]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:45:40.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:40 smithi084 ceph-mon[29774]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T22:45:40.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:40 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/1461080971' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["0"]}]': finished 2024-09-20T22:45:40.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:40 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T22:45:40.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:40 smithi084 ceph-mon[29774]: osdmap e260: 8 total, 8 up, 6 in 2024-09-20T22:45:40.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:40 smithi084 ceph-mon[29774]: 3.13 scrub starts 2024-09-20T22:45:40.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:40 smithi084 ceph-mon[29774]: 3.13 scrub ok 2024-09-20T22:45:40.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:40 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "20"}]: dispatch 2024-09-20T22:45:40.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:40 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "20"}]: dispatch 2024-09-20T22:45:40.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:40 smithi084 ceph-mon[29774]: 6.1 scrub starts 2024-09-20T22:45:40.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:40 smithi084 ceph-mon[29774]: 6.1 scrub ok 2024-09-20T22:45:40.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:40 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "20"}]': finished 2024-09-20T22:45:40.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:40 smithi084 ceph-mon[29774]: osdmap e261: 8 total, 8 up, 6 in 2024-09-20T22:45:40.367 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:40 smithi084 ceph-mon[40024]: 3.15 scrub starts 2024-09-20T22:45:40.367 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:40 smithi084 ceph-mon[40024]: 3.15 scrub ok 2024-09-20T22:45:40.367 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:40 smithi084 ceph-mon[40024]: 1.3 scrub starts 2024-09-20T22:45:40.367 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:40 smithi084 ceph-mon[40024]: 1.3 scrub ok 2024-09-20T22:45:40.367 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:40 smithi084 ceph-mon[40024]: 4.17 scrub starts 2024-09-20T22:45:40.367 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:40 smithi084 ceph-mon[40024]: 4.17 scrub ok 2024-09-20T22:45:40.367 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:40 smithi084 ceph-mon[40024]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:45:40.367 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:40 smithi084 ceph-mon[40024]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T22:45:40.367 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:40 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/1461080971' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["0"]}]': finished 2024-09-20T22:45:40.367 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:40 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T22:45:40.367 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:40 smithi084 ceph-mon[40024]: osdmap e260: 8 total, 8 up, 6 in 2024-09-20T22:45:40.367 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:40 smithi084 ceph-mon[40024]: 3.13 scrub starts 2024-09-20T22:45:40.367 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:40 smithi084 ceph-mon[40024]: 3.13 scrub ok 2024-09-20T22:45:40.368 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:40 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "20"}]: dispatch 2024-09-20T22:45:40.368 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:40 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "20"}]: dispatch 2024-09-20T22:45:40.368 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:40 smithi084 ceph-mon[40024]: 6.1 scrub starts 2024-09-20T22:45:40.368 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:40 smithi084 ceph-mon[40024]: 6.1 scrub ok 2024-09-20T22:45:40.368 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:40 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "20"}]': finished 2024-09-20T22:45:40.368 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:40 smithi084 ceph-mon[40024]: osdmap e261: 8 total, 8 up, 6 in 2024-09-20T22:45:40.382 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:45:40.403 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:40.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:40.404 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:45:40.505 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:45:40.526 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:40.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:40.527 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:45:40.621 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:45:40.628 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:45:40.649 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:40.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:40.650 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:45:40.738 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.6 dump_blocked_ops 2024-09-20T22:45:40.751 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:45:40.802 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:40.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:40.803 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:45:40.903 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:45:40.926 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:40.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:40.926 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:45:40.984 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:45:41.027 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:45:41.049 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:41.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:41.050 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:45:41.078 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:45:41.078 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:45:41.078 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:45:41.078 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:45:41.093 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:45:41.093 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:45:41.109 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-20T22:45:41.109 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T22:45:41.125 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:45:41.125 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:45:41.141 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:45:41.141 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:45:41.151 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:45:41.158 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:45:41.158 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:45:41.172 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:41.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:41.173 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:45:41.174 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:45:41.174 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:45:41.197 INFO:tasks.rados.rados.0.smithi084.stdout:903: done (2 left) 2024-09-20T22:45:41.197 INFO:tasks.rados.rados.0.smithi084.stdout:904: write oid 45 current snap is 85 2024-09-20T22:45:41.197 INFO:tasks.rados.rados.0.smithi084.stdout:904: seq_num 338 ranges {480766=553694,1629149=591119,2782148=420058} 2024-09-20T22:45:41.204 INFO:tasks.rados.rados.0.smithi084.stdout:904: writing smithi08477203-45 from 480766 to 1034460 tid 1 2024-09-20T22:45:41.211 INFO:tasks.rados.rados.0.smithi084.stdout:904: writing smithi08477203-45 from 1629149 to 2220268 tid 2 2024-09-20T22:45:41.218 INFO:tasks.rados.rados.0.smithi084.stdout:904: writing smithi08477203-45 from 2782148 to 3202206 tid 3 2024-09-20T22:45:41.218 INFO:tasks.rados.rados.0.smithi084.stdout:905: read oid 46 snap 78 2024-09-20T22:45:41.218 INFO:tasks.rados.rados.0.smithi084.stdout:905: expect deleted 2024-09-20T22:45:41.218 INFO:tasks.rados.rados.0.smithi084.stdout:906: snap_remove snap 83 2024-09-20T22:45:41.274 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:45:41.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:41.297 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:41.297 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:45:41.364 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:41 smithi084 ceph-mon[40024]: pgmap v292: 178 pgs: 3 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 171 active+clean; 216 MiB data, 641 MiB used, 536 GiB / 536 GiB avail; 30 MiB/s rd, 4.9 MiB/s wr, 33 op/s; 41 KiB/s, 0 objects/s recovering 2024-09-20T22:45:41.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:41 smithi084 ceph-mon[40024]: 6.1b scrub starts 2024-09-20T22:45:41.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:41 smithi084 ceph-mon[40024]: 6.1b scrub ok 2024-09-20T22:45:41.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:41 smithi084 ceph-mon[40024]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs inactive, 6 pgs peering) 2024-09-20T22:45:41.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:41 smithi084 ceph-mon[40024]: Cluster is now healthy 2024-09-20T22:45:41.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:41 smithi084 ceph-mon[40024]: purged_snaps scrub starts 2024-09-20T22:45:41.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:41 smithi084 ceph-mon[40024]: purged_snaps scrub ok 2024-09-20T22:45:41.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:41 smithi084 ceph-mon[40024]: 5.d deep-scrub starts 2024-09-20T22:45:41.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:41 smithi084 ceph-mon[40024]: 5.d deep-scrub ok 2024-09-20T22:45:41.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:41 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/2482975250' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:45:41.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:41 smithi084 ceph-mon[40024]: osdmap e262: 8 total, 8 up, 6 in 2024-09-20T22:45:41.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:41 smithi084 ceph-mon[29774]: pgmap v292: 178 pgs: 3 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 171 active+clean; 216 MiB data, 641 MiB used, 536 GiB / 536 GiB avail; 30 MiB/s rd, 4.9 MiB/s wr, 33 op/s; 41 KiB/s, 0 objects/s recovering 2024-09-20T22:45:41.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:41 smithi084 ceph-mon[29774]: 6.1b scrub starts 2024-09-20T22:45:41.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:41 smithi084 ceph-mon[29774]: 6.1b scrub ok 2024-09-20T22:45:41.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:41 smithi084 ceph-mon[29774]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs inactive, 6 pgs peering) 2024-09-20T22:45:41.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:41 smithi084 ceph-mon[29774]: Cluster is now healthy 2024-09-20T22:45:41.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:41 smithi084 ceph-mon[29774]: purged_snaps scrub starts 2024-09-20T22:45:41.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:41 smithi084 ceph-mon[29774]: purged_snaps scrub ok 2024-09-20T22:45:41.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:41 smithi084 ceph-mon[29774]: 5.d deep-scrub starts 2024-09-20T22:45:41.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:41 smithi084 ceph-mon[29774]: 5.d deep-scrub ok 2024-09-20T22:45:41.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:41 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/2482975250' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:45:41.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:41 smithi084 ceph-mon[29774]: osdmap e262: 8 total, 8 up, 6 in 2024-09-20T22:45:41.394 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:41 smithi154 ceph-mon[33264]: pgmap v292: 178 pgs: 3 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 171 active+clean; 216 MiB data, 641 MiB used, 536 GiB / 536 GiB avail; 30 MiB/s rd, 4.9 MiB/s wr, 33 op/s; 41 KiB/s, 0 objects/s recovering 2024-09-20T22:45:41.394 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:41 smithi154 ceph-mon[33264]: 6.1b scrub starts 2024-09-20T22:45:41.394 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:41 smithi154 ceph-mon[33264]: 6.1b scrub ok 2024-09-20T22:45:41.394 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:41 smithi154 ceph-mon[33264]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs inactive, 6 pgs peering) 2024-09-20T22:45:41.394 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:41 smithi154 ceph-mon[33264]: Cluster is now healthy 2024-09-20T22:45:41.394 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:41 smithi154 ceph-mon[33264]: purged_snaps scrub starts 2024-09-20T22:45:41.394 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:41 smithi154 ceph-mon[33264]: purged_snaps scrub ok 2024-09-20T22:45:41.394 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:41 smithi154 ceph-mon[33264]: 5.d deep-scrub starts 2024-09-20T22:45:41.394 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:41 smithi154 ceph-mon[33264]: 5.d deep-scrub ok 2024-09-20T22:45:41.394 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:41 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/2482975250' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:45:41.394 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:41 smithi154 ceph-mon[33264]: osdmap e262: 8 total, 8 up, 6 in 2024-09-20T22:45:41.398 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:45:41.420 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:41.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:41.421 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:45:41.494 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-20T22:45:41.494 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:45:41.522 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:45:41.542 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:41.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:41.543 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:45:41.644 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:45:41.666 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:41.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:41.667 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:45:41.768 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:45:41.788 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:41.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:41.789 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:45:41.890 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:45:41.911 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:41.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:41.912 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:45:42.013 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:45:42.215 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd set noscrub 2024-09-20T22:45:42.215 INFO:tasks.rados.rados.0.smithi084.stdout:906: done (4 left) 2024-09-20T22:45:42.215 INFO:tasks.rados.rados.0.smithi084.stdout:907: snap_remove snap 80 2024-09-20T22:45:42.219 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.6 dump_historic_ops 2024-09-20T22:45:42.238 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:42.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:42.239 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:45:42.340 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:45:42.361 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:42.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:42.362 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:45:42.461 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:45:42.463 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:45:42.466 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:45:42.484 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:42.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:42.485 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:45:42.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:42 smithi154 ceph-mon[33264]: 7.5 scrub starts 2024-09-20T22:45:42.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:42 smithi154 ceph-mon[33264]: 7.5 scrub ok 2024-09-20T22:45:42.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:42 smithi154 ceph-mon[33264]: 1.5 deep-scrub starts 2024-09-20T22:45:42.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:42 smithi154 ceph-mon[33264]: 1.5 deep-scrub ok 2024-09-20T22:45:42.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:42 smithi154 ceph-mon[33264]: osdmap e263: 8 total, 8 up, 6 in 2024-09-20T22:45:42.584 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:42 smithi084 ceph-mon[29774]: 7.5 scrub starts 2024-09-20T22:45:42.584 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:42 smithi084 ceph-mon[29774]: 7.5 scrub ok 2024-09-20T22:45:42.584 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:42 smithi084 ceph-mon[29774]: 1.5 deep-scrub starts 2024-09-20T22:45:42.584 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:42 smithi084 ceph-mon[29774]: 1.5 deep-scrub ok 2024-09-20T22:45:42.584 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:42 smithi084 ceph-mon[29774]: osdmap e263: 8 total, 8 up, 6 in 2024-09-20T22:45:42.584 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:42 smithi084 ceph-mon[40024]: 7.5 scrub starts 2024-09-20T22:45:42.584 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:42 smithi084 ceph-mon[40024]: 7.5 scrub ok 2024-09-20T22:45:42.584 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:42 smithi084 ceph-mon[40024]: 1.5 deep-scrub starts 2024-09-20T22:45:42.584 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:42 smithi084 ceph-mon[40024]: 1.5 deep-scrub ok 2024-09-20T22:45:42.584 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:42 smithi084 ceph-mon[40024]: osdmap e263: 8 total, 8 up, 6 in 2024-09-20T22:45:42.586 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:45:42.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:42.608 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:42.609 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:45:42.709 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:45:42.730 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:42.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:42.731 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:45:42.831 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:45:42.852 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:42.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:42.853 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:45:42.918 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-20T22:45:42.918 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:45:42.953 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:45:42.974 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:42.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:42.975 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:45:43.076 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:45:43.097 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:43.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:43.098 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:45:43.162 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is set 2024-09-20T22:45:43.162 INFO:tasks.rados.rados.0.smithi084.stdout:907: done (4 left) 2024-09-20T22:45:43.162 INFO:tasks.rados.rados.0.smithi084.stdout:908: snap_create 2024-09-20T22:45:43.163 INFO:tasks.rados.rados.0.smithi084.stdout:897: finishing copy_from to smithi08477203-32 2024-09-20T22:45:43.163 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 32 v 376 (ObjNum 258 snap 60 seq_num 258) dirty exists 2024-09-20T22:45:43.163 INFO:tasks.rados.rados.0.smithi084.stdout:904: finishing write tid 1 to smithi08477203-45 2024-09-20T22:45:43.163 INFO:tasks.rados.rados.0.smithi084.stdout:899: finishing copy_from to smithi08477203-48 2024-09-20T22:45:43.163 INFO:tasks.rados.rados.0.smithi084.stdout:899: got expected ENOENT (src dne) 2024-09-20T22:45:43.163 INFO:tasks.rados.rados.0.smithi084.stdout:899: finishing copy_from racing read to smithi08477203-48 2024-09-20T22:45:43.163 INFO:tasks.rados.rados.0.smithi084.stdout:904: finishing write tid 2 to smithi08477203-45 2024-09-20T22:45:43.163 INFO:tasks.rados.rados.0.smithi084.stdout:904: finishing write tid 3 to smithi08477203-45 2024-09-20T22:45:43.163 INFO:tasks.rados.rados.0.smithi084.stdout:904: finishing write tid 4 to smithi08477203-45 2024-09-20T22:45:43.163 INFO:tasks.rados.rados.0.smithi084.stdout:904: finishing write tid 5 to smithi08477203-45 2024-09-20T22:45:43.163 INFO:tasks.rados.rados.0.smithi084.stdout:904: oid 45 updating version 0 to 105 2024-09-20T22:45:43.163 INFO:tasks.rados.rados.0.smithi084.stdout:904: oid 45 updating version 105 to 106 2024-09-20T22:45:43.163 INFO:tasks.rados.rados.0.smithi084.stdout:904: oid 45 updating version 106 to 107 2024-09-20T22:45:43.163 INFO:tasks.rados.rados.0.smithi084.stdout:904: oid 45 updating version 107 to 108 2024-09-20T22:45:43.164 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 45 v 108 (ObjNum 338 snap 85 seq_num 338) dirty exists 2024-09-20T22:45:43.164 INFO:tasks.rados.rados.0.smithi084.stdout:904: left oid 45 (ObjNum 338 snap 85 seq_num 338) 2024-09-20T22:45:43.199 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:45:43.220 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:43.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:43.221 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:45:43.321 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:45:43.342 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:43.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:43.343 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:45:43.444 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:45:43.465 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:43.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:43.466 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:45:43.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:43 smithi154 ceph-mon[33264]: pgmap v295: 178 pgs: 3 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 171 active+clean; 216 MiB data, 641 MiB used, 536 GiB / 536 GiB avail; 30 MiB/s rd, 4.9 MiB/s wr, 33 op/s; 41 KiB/s, 0 objects/s recovering 2024-09-20T22:45:43.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:43 smithi154 ceph-mon[33264]: 6.7 scrub starts 2024-09-20T22:45:43.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:43 smithi154 ceph-mon[33264]: 6.7 scrub ok 2024-09-20T22:45:43.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:43 smithi154 ceph-mon[33264]: 5.4 scrub starts 2024-09-20T22:45:43.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:43 smithi154 ceph-mon[33264]: 5.4 scrub ok 2024-09-20T22:45:43.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:43 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/924587017' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-20T22:45:43.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:43 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-20T22:45:43.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:43 smithi154 ceph-mon[33264]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:45:43.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:43 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T22:45:43.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:43 smithi154 ceph-mon[33264]: osdmap e264: 8 total, 8 up, 6 in 2024-09-20T22:45:43.541 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:43 smithi084 ceph-mon[29774]: pgmap v295: 178 pgs: 3 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 171 active+clean; 216 MiB data, 641 MiB used, 536 GiB / 536 GiB avail; 30 MiB/s rd, 4.9 MiB/s wr, 33 op/s; 41 KiB/s, 0 objects/s recovering 2024-09-20T22:45:43.541 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:43 smithi084 ceph-mon[29774]: 6.7 scrub starts 2024-09-20T22:45:43.541 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:43 smithi084 ceph-mon[29774]: 6.7 scrub ok 2024-09-20T22:45:43.541 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:43 smithi084 ceph-mon[29774]: 5.4 scrub starts 2024-09-20T22:45:43.541 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:43 smithi084 ceph-mon[29774]: 5.4 scrub ok 2024-09-20T22:45:43.541 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:43 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/924587017' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-20T22:45:43.541 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:43 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-20T22:45:43.541 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:43 smithi084 ceph-mon[29774]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:45:43.542 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:43 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T22:45:43.542 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:43 smithi084 ceph-mon[29774]: osdmap e264: 8 total, 8 up, 6 in 2024-09-20T22:45:43.542 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:43 smithi084 ceph-mon[40024]: pgmap v295: 178 pgs: 3 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 171 active+clean; 216 MiB data, 641 MiB used, 536 GiB / 536 GiB avail; 30 MiB/s rd, 4.9 MiB/s wr, 33 op/s; 41 KiB/s, 0 objects/s recovering 2024-09-20T22:45:43.542 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:43 smithi084 ceph-mon[40024]: 6.7 scrub starts 2024-09-20T22:45:43.542 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:43 smithi084 ceph-mon[40024]: 6.7 scrub ok 2024-09-20T22:45:43.542 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:43 smithi084 ceph-mon[40024]: 5.4 scrub starts 2024-09-20T22:45:43.542 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:43 smithi084 ceph-mon[40024]: 5.4 scrub ok 2024-09-20T22:45:43.542 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:43 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/924587017' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-20T22:45:43.542 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:43 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-20T22:45:43.542 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:43 smithi084 ceph-mon[40024]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:45:43.542 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:43 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T22:45:43.543 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:43 smithi084 ceph-mon[40024]: osdmap e264: 8 total, 8 up, 6 in 2024-09-20T22:45:43.567 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:45:43.587 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:43.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:43.588 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:45:43.658 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-20T22:45:43.689 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:45:43.710 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:43.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:43.710 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:45:43.802 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-20T22:45:43.811 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:45:43.832 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:43.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:43.832 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:45:43.914 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:45:43.933 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:45:43.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:43.954 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:43.954 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:45:44.055 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:45:44.058 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:45:44.076 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:44.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:44.077 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:45:44.162 INFO:tasks.rados.rados.0.smithi084.stdout:897: done (4 left) 2024-09-20T22:45:44.162 INFO:tasks.rados.rados.0.smithi084.stdout:899: done (3 left) 2024-09-20T22:45:44.162 INFO:tasks.rados.rados.0.smithi084.stdout:904: done (2 left) 2024-09-20T22:45:44.162 INFO:tasks.rados.rados.0.smithi084.stdout:905: done (1 left) 2024-09-20T22:45:44.163 INFO:tasks.rados.rados.0.smithi084.stdout:908: done (0 left) 2024-09-20T22:45:44.163 INFO:tasks.rados.rados.0.smithi084.stdout:909: read oid 33 snap -1 2024-09-20T22:45:44.163 INFO:tasks.rados.rados.0.smithi084.stdout:909: expect (ObjNum 229 snap 48 seq_num 229) 2024-09-20T22:45:44.163 INFO:tasks.rados.rados.0.smithi084.stdout:910: read oid 14 snap -1 2024-09-20T22:45:44.163 INFO:tasks.rados.rados.0.smithi084.stdout:910: expect (ObjNum 306 snap 75 seq_num 306) 2024-09-20T22:45:44.163 INFO:tasks.rados.rados.0.smithi084.stdout:911: write (excl) oid 21 current snap is 86 2024-09-20T22:45:44.163 INFO:tasks.rados.rados.0.smithi084.stdout:911: seq_num 339 ranges {497573=628943,1687631=734777,3093218=604355} 2024-09-20T22:45:44.169 INFO:tasks.rados.rados.0.smithi084.stdout:911: writing smithi08477203-21 from 497573 to 1126516 tid 1 2024-09-20T22:45:44.176 INFO:tasks.rados.rados.0.smithi084.stdout:911: writing smithi08477203-21 from 1687631 to 2422408 tid 2 2024-09-20T22:45:44.177 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:45:44.184 INFO:tasks.rados.rados.0.smithi084.stdout:911: writing smithi08477203-21 from 3093218 to 3697573 tid 3 2024-09-20T22:45:44.184 INFO:tasks.rados.rados.0.smithi084.stdout:912: rollback oid 2 current snap is 86 2024-09-20T22:45:44.184 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 2 to 60 2024-09-20T22:45:44.184 INFO:tasks.rados.rados.0.smithi084.stdout:909: expect (ObjNum 229 snap 48 seq_num 229) 2024-09-20T22:45:44.200 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:44.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:44.201 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:45:44.301 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:45:44.318 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:44 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:45:44.319 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:44 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "19"}]: dispatch 2024-09-20T22:45:44.319 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:44 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "19"}]: dispatch 2024-09-20T22:45:44.319 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:44 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:45:44.319 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:44 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:45:44.319 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:44 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:45:44.319 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:44 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "19"}]': finished 2024-09-20T22:45:44.319 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:44 smithi154 ceph-mon[33264]: osdmap e265: 8 total, 8 up, 6 in 2024-09-20T22:45:44.323 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:44.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:44.324 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:45:44.390 INFO:tasks.rados.rados.0.smithi084.stdout:910: expect (ObjNum 306 snap 75 seq_num 306) 2024-09-20T22:45:44.419 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-20T22:45:44.419 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:45:44.425 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:45:44.446 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:44 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:45:44.446 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:44 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "19"}]: dispatch 2024-09-20T22:45:44.446 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:44 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "19"}]: dispatch 2024-09-20T22:45:44.446 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:44 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:45:44.446 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:44 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:45:44.446 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:44 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:45:44.446 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:44 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "19"}]': finished 2024-09-20T22:45:44.446 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:44 smithi084 ceph-mon[40024]: osdmap e265: 8 total, 8 up, 6 in 2024-09-20T22:45:44.446 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:44 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:45:44.447 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:44 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "19"}]: dispatch 2024-09-20T22:45:44.447 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:44 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "19"}]: dispatch 2024-09-20T22:45:44.447 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:44 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:45:44.447 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:44 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:45:44.447 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:44 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:45:44.447 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:44 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "19"}]': finished 2024-09-20T22:45:44.447 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:44 smithi084 ceph-mon[29774]: osdmap e265: 8 total, 8 up, 6 in 2024-09-20T22:45:44.447 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:44.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:44.448 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:45:44.536 INFO:tasks.rados.rados.0.smithi084.stdout:911: finishing write tid 1 to smithi08477203-21 2024-09-20T22:45:44.536 INFO:tasks.rados.rados.0.smithi084.stdout:911: finishing write tid 2 to smithi08477203-21 2024-09-20T22:45:44.536 INFO:tasks.rados.rados.0.smithi084.stdout:912: finishing rollback tid 0 to smithi08477203-2 2024-09-20T22:45:44.536 INFO:tasks.rados.rados.0.smithi084.stdout:912: finishing rollback tid 1 to smithi08477203-2 2024-09-20T22:45:44.537 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 2 v 110 (ObjNum 129 snap 0 seq_num 2090735872) dirty dne 2024-09-20T22:45:44.537 INFO:tasks.rados.rados.0.smithi084.stdout:909: done (3 left) 2024-09-20T22:45:44.537 INFO:tasks.rados.rados.0.smithi084.stdout:910: done (2 left) 2024-09-20T22:45:44.537 INFO:tasks.rados.rados.0.smithi084.stdout:912: done (1 left) 2024-09-20T22:45:44.537 INFO:tasks.rados.rados.0.smithi084.stdout:913: read oid 44 snap 69 2024-09-20T22:45:44.537 INFO:tasks.rados.rados.0.smithi084.stdout:913: expect (ObjNum 129 snap 23 seq_num 129) 2024-09-20T22:45:44.537 INFO:tasks.rados.rados.0.smithi084.stdout:914: rollback oid 2 current snap is 86 2024-09-20T22:45:44.537 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 2 to 79 2024-09-20T22:45:44.537 INFO:tasks.rados.rados.0.smithi084.stdout:911: finishing write tid 3 to smithi08477203-21 2024-09-20T22:45:44.537 INFO:tasks.rados.rados.0.smithi084.stdout:911: finishing write tid 4 to smithi08477203-21 2024-09-20T22:45:44.537 INFO:tasks.rados.rados.0.smithi084.stdout:911: finishing write tid 5 to smithi08477203-21 2024-09-20T22:45:44.537 INFO:tasks.rados.rados.0.smithi084.stdout:911: oid 21 updating version 0 to 257 2024-09-20T22:45:44.537 INFO:tasks.rados.rados.0.smithi084.stdout:911: oid 21 updating version 257 to 258 2024-09-20T22:45:44.537 INFO:tasks.rados.rados.0.smithi084.stdout:911: oid 21 updating version 258 to 259 2024-09-20T22:45:44.538 INFO:tasks.rados.rados.0.smithi084.stdout:911: oid 21 updating version 259 to 260 2024-09-20T22:45:44.538 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 21 v 260 (ObjNum 339 snap 86 seq_num 339) dirty exists 2024-09-20T22:45:44.538 INFO:tasks.rados.rados.0.smithi084.stdout:911: left oid 21 (ObjNum 339 snap 86 seq_num 339) 2024-09-20T22:45:44.538 INFO:tasks.rados.rados.0.smithi084.stdout:911: done (2 left) 2024-09-20T22:45:44.538 INFO:tasks.rados.rados.0.smithi084.stdout:915: snap_create 2024-09-20T22:45:44.538 INFO:tasks.rados.rados.0.smithi084.stdout:914: finishing rollback tid 1 to smithi08477203-2 2024-09-20T22:45:44.538 INFO:tasks.rados.rados.0.smithi084.stdout:914: finishing rollback tid 2 to smithi08477203-2 2024-09-20T22:45:44.538 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 2 v 112 (ObjNum 300 snap 75 seq_num 300) dirty exists 2024-09-20T22:45:44.544 INFO:tasks.rados.rados.0.smithi084.stdout:913: expect (ObjNum 129 snap 23 seq_num 129) 2024-09-20T22:45:44.545 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:45:44.545 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:45:44.549 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:45:44.562 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:45:44.563 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:45:44.573 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:44.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:44.574 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:45:44.583 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:45:44.584 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:45:44.599 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-20T22:45:44.600 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T22:45:44.616 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:45:44.616 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:45:44.632 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:45:44.633 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:45:44.649 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:45:44.649 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:45:44.665 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:45:44.665 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:45:44.675 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:45:44.697 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:44.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:44.698 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:45:44.798 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:45:44.820 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:44.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:44.821 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:45:44.921 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:45:44.943 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:44.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:44.944 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:45:45.045 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:45:45.067 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:45.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:45.068 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:45:45.164 INFO:tasks.rados.rados.0.smithi084.stdout:913: done (2 left) 2024-09-20T22:45:45.164 INFO:tasks.rados.rados.0.smithi084.stdout:914: done (1 left) 2024-09-20T22:45:45.164 INFO:tasks.rados.rados.0.smithi084.stdout:915: done (0 left) 2024-09-20T22:45:45.164 INFO:tasks.rados.rados.0.smithi084.stdout:916: snap_create 2024-09-20T22:45:45.168 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:45:45.190 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:45.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:45.191 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:45:45.292 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:45:45.313 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:45.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:45.314 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:45:45.415 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:45:45.438 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:45.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:45.439 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:45:45.442 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.7 dump_blocked_ops 2024-09-20T22:45:45.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:45 smithi154 ceph-mon[33264]: pgmap v298: 177 pgs: 4 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 168 active+clean; 213 MiB data, 733 MiB used, 536 GiB / 536 GiB avail; 3.1 MiB/s rd, 1.8 MiB/s wr, 8 op/s; 17 MiB/s, 52 objects/s recovering 2024-09-20T22:45:45.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:45 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/1672305235' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:45:45.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:45 smithi154 ceph-mon[33264]: osdmap e266: 8 total, 8 up, 6 in 2024-09-20T22:45:45.539 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:45:45.560 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 5, 6, 7] out_osds: [4, 0] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-20T22:45:45.560 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-20T22:45:45.560 INFO:tasks.thrashosds.thrasher:inject_pause on osd.1 2024-09-20T22:45:45.560 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-20T22:45:45.561 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-20T22:45:45.561 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.1 config set filestore_inject_stall 3 2024-09-20T22:45:45.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:45.562 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:45.562 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:45:45.614 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:45 smithi084 ceph-mon[29774]: pgmap v298: 177 pgs: 4 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 168 active+clean; 213 MiB data, 733 MiB used, 536 GiB / 536 GiB avail; 3.1 MiB/s rd, 1.8 MiB/s wr, 8 op/s; 17 MiB/s, 52 objects/s recovering 2024-09-20T22:45:45.614 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:45 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/1672305235' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:45:45.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:45 smithi084 ceph-mon[29774]: osdmap e266: 8 total, 8 up, 6 in 2024-09-20T22:45:45.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:45 smithi084 ceph-mon[40024]: pgmap v298: 177 pgs: 4 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 168 active+clean; 213 MiB data, 733 MiB used, 536 GiB / 536 GiB avail; 3.1 MiB/s rd, 1.8 MiB/s wr, 8 op/s; 17 MiB/s, 52 objects/s recovering 2024-09-20T22:45:45.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:45 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/1672305235' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:45:45.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:45 smithi084 ceph-mon[40024]: osdmap e266: 8 total, 8 up, 6 in 2024-09-20T22:45:45.663 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:45:45.686 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:45.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:45.688 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:45:45.688 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:45:45.789 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:45:45.811 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:45.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:45.811 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:45:45.848 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:45:45.867 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-20T22:45:45.912 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:45:45.934 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:45.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:45.935 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:45:46.036 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:45:46.057 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:46.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:46.058 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:45:46.143 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-20T22:45:46.144 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [], 2024-09-20T22:45:46.144 INFO:teuthology.orchestra.run.smithi154.stdout: "complaint_time": 30, 2024-09-20T22:45:46.144 INFO:teuthology.orchestra.run.smithi154.stdout: "num_blocked_ops": 0 2024-09-20T22:45:46.144 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-20T22:45:46.159 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:45:46.163 INFO:tasks.rados.rados.0.smithi084.stdout:916: done (0 left) 2024-09-20T22:45:46.163 INFO:tasks.rados.rados.0.smithi084.stdout:917: write oid 13 current snap is 88 2024-09-20T22:45:46.163 INFO:tasks.rados.rados.0.smithi084.stdout:917: seq_num 340 ranges {514380=704192,1629760=478435,2604288=442602} 2024-09-20T22:45:46.169 INFO:tasks.rados.rados.0.smithi084.stdout:917: writing smithi08477203-13 from 514380 to 1218572 tid 1 2024-09-20T22:45:46.174 INFO:tasks.rados.rados.0.smithi084.stdout:917: writing smithi08477203-13 from 1629760 to 2108195 tid 2 2024-09-20T22:45:46.180 INFO:tasks.rados.rados.0.smithi084.stdout:917: writing smithi08477203-13 from 2604288 to 3046890 tid 3 2024-09-20T22:45:46.180 INFO:tasks.rados.rados.0.smithi084.stdout:918: snap_remove snap 87 2024-09-20T22:45:46.186 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:46.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:46.188 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:45:46.288 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:45:46.310 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:46.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:46.311 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:45:46.345 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-20T22:45:46.345 INFO:teuthology.orchestra.run.smithi084.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-20T22:45:46.345 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-20T22:45:46.412 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:45:46.433 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:46.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:46.434 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:45:46.534 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:45:46.556 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:46.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:46.557 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:45:46.589 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:45:46.658 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:45:46.681 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:46.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:46.682 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:45:46.783 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:45:46.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:46.805 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:46.805 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:45:46.877 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.7 dump_historic_ops 2024-09-20T22:45:46.906 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:45:46.927 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:46.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:46.928 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:45:47.029 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:45:47.051 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:47.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:47.052 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:45:47.121 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:45:47.153 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:45:47.165 INFO:tasks.rados.rados.0.smithi084.stdout:918: done (1 left) 2024-09-20T22:45:47.165 INFO:tasks.rados.rados.0.smithi084.stdout:919: write (excl) oid 10 current snap is 88 2024-09-20T22:45:47.165 INFO:tasks.rados.rados.0.smithi084.stdout:919: seq_num 341 ranges {531187=779441,2088242=738446,3431711=299476} 2024-09-20T22:45:47.173 INFO:tasks.rados.rados.0.smithi084.stdout:919: writing smithi08477203-10 from 531187 to 1310628 tid 1 2024-09-20T22:45:47.178 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:47.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:47.179 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:45:47.181 INFO:tasks.rados.rados.0.smithi084.stdout:919: writing smithi08477203-10 from 2088242 to 2826688 tid 2 2024-09-20T22:45:47.187 INFO:tasks.rados.rados.0.smithi084.stdout:919: writing smithi08477203-10 from 3431711 to 3731187 tid 3 2024-09-20T22:45:47.187 INFO:tasks.rados.rados.0.smithi084.stdout:920: write (excl) oid 22 current snap is 88 2024-09-20T22:45:47.187 INFO:tasks.rados.rados.0.smithi084.stdout:920: seq_num 342 ranges {547994=454690,1746724=482104,2659134=465146,3605976=142018} 2024-09-20T22:45:47.192 INFO:tasks.rados.rados.0.smithi084.stdout:920: writing smithi08477203-22 from 547994 to 1002684 tid 1 2024-09-20T22:45:47.199 INFO:tasks.rados.rados.0.smithi084.stdout:920: writing smithi08477203-22 from 1746724 to 2228828 tid 2 2024-09-20T22:45:47.204 INFO:tasks.rados.rados.0.smithi084.stdout:920: writing smithi08477203-22 from 2659134 to 3124280 tid 3 2024-09-20T22:45:47.207 INFO:tasks.rados.rados.0.smithi084.stdout:920: writing smithi08477203-22 from 3605976 to 3747994 tid 4 2024-09-20T22:45:47.207 INFO:tasks.rados.rados.0.smithi084.stdout:921: snap_remove snap 79 2024-09-20T22:45:47.279 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:45:47.291 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:47 smithi084 ceph-mon[40024]: pgmap v301: 177 pgs: 4 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 168 active+clean; 213 MiB data, 733 MiB used, 536 GiB / 536 GiB avail; 3.1 MiB/s rd, 1.8 MiB/s wr, 8 op/s; 17 MiB/s, 52 objects/s recovering 2024-09-20T22:45:47.291 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:47 smithi084 ceph-mon[40024]: osdmap e267: 8 total, 8 up, 6 in 2024-09-20T22:45:47.291 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:47 smithi084 ceph-mon[29774]: pgmap v301: 177 pgs: 4 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 168 active+clean; 213 MiB data, 733 MiB used, 536 GiB / 536 GiB avail; 3.1 MiB/s rd, 1.8 MiB/s wr, 8 op/s; 17 MiB/s, 52 objects/s recovering 2024-09-20T22:45:47.291 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:47 smithi084 ceph-mon[29774]: osdmap e267: 8 total, 8 up, 6 in 2024-09-20T22:45:47.300 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:47.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:47.301 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:45:47.401 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:45:47.423 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:47.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:47.423 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:45:47.437 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:47 smithi154 ceph-mon[33264]: pgmap v301: 177 pgs: 4 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 168 active+clean; 213 MiB data, 733 MiB used, 536 GiB / 536 GiB avail; 3.1 MiB/s rd, 1.8 MiB/s wr, 8 op/s; 17 MiB/s, 52 objects/s recovering 2024-09-20T22:45:47.437 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:47 smithi154 ceph-mon[33264]: osdmap e267: 8 total, 8 up, 6 in 2024-09-20T22:45:47.455 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-20T22:45:47.524 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:45:47.545 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:47.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:47.545 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:45:47.639 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-20T22:45:47.639 INFO:teuthology.orchestra.run.smithi154.stdout: "size": 20, 2024-09-20T22:45:47.639 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 600, 2024-09-20T22:45:47.639 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [ 2024-09-20T22:45:47.639 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.640 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:786 7.17 7:ed54065f:::smithi08477203-12:head [copy-from ver 61] snapc 19=[17] ondisk+write+known_if_redirected+supports_pool_eio e145)", 2024-09-20T22:45:47.640 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:44.333196+0000", 2024-09-20T22:45:47.640 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 123.303116708, 2024-09-20T22:45:47.640 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.023328503, 2024-09-20T22:45:47.640 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:45:47.640 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:45:47.640 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:45:47.640 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:45:47.640 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:45:47.640 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:45:47.640 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 786 2024-09-20T22:45:47.640 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.640 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:45:47.640 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.641 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:45:47.641 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.333196+0000", 2024-09-20T22:45:47.641 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:45:47.641 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.641 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.641 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:45:47.641 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.333196+0000", 2024-09-20T22:45:47.641 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:45:47.641 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.641 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.641 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:45:47.641 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.333197+0000", 2024-09-20T22:45:47.641 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.486e-06 2024-09-20T22:45:47.641 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.642 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.642 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:45:47.642 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.333207+0000", 2024-09-20T22:45:47.642 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0307e-05 2024-09-20T22:45:47.642 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.642 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.642 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:45:47.642 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.333220+0000", 2024-09-20T22:45:47.642 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.306e-05 2024-09-20T22:45:47.642 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.642 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.642 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:45:47.642 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.333229+0000", 2024-09-20T22:45:47.642 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.3450000000000006e-06 2024-09-20T22:45:47.643 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.643 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.643 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:45:47.643 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.333292+0000", 2024-09-20T22:45:47.643 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.3137000000000002e-05 2024-09-20T22:45:47.643 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.643 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.643 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op must wait for map", 2024-09-20T22:45:47.643 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.333298+0000", 2024-09-20T22:45:47.643 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.7780000000000004e-06 2024-09-20T22:45:47.643 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.643 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.643 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:45:47.643 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.336639+0000", 2024-09-20T22:45:47.644 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0033414479999999999 2024-09-20T22:45:47.644 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.644 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.644 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:45:47.644 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.336729+0000", 2024-09-20T22:45:47.644 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.9888000000000004e-05 2024-09-20T22:45:47.644 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.644 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.644 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 2,5", 2024-09-20T22:45:47.644 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.344876+0000", 2024-09-20T22:45:47.644 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0081473989999999996 2024-09-20T22:45:47.644 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.644 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.644 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:45:47.644 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.355138+0000", 2024-09-20T22:45:47.645 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.010261462000000001 2024-09-20T22:45:47.645 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.645 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.645 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:45:47.645 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.355972+0000", 2024-09-20T22:45:47.645 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00083377899999999999 2024-09-20T22:45:47.645 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.645 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.645 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:45:47.645 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.356451+0000", 2024-09-20T22:45:47.645 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00047985200000000001 2024-09-20T22:45:47.645 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.645 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.645 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:45:47.646 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.356486+0000", 2024-09-20T22:45:47.646 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.4653000000000002e-05 2024-09-20T22:45:47.646 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.646 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.646 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:45:47.646 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.356524+0000", 2024-09-20T22:45:47.646 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.7908999999999999e-05 2024-09-20T22:45:47.646 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:45:47.646 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:45:47.646 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:45:47.646 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.646 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.646 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:787 7.17 7:ed54065f:::smithi08477203-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e145)", 2024-09-20T22:45:47.646 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:44.333248+0000", 2024-09-20T22:45:47.646 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 123.303064613, 2024-09-20T22:45:47.647 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.016881184, 2024-09-20T22:45:47.647 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:45:47.647 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:45:47.647 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:45:47.647 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:45:47.647 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:45:47.647 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:45:47.647 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 787 2024-09-20T22:45:47.647 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.647 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:45:47.647 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.647 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:45:47.647 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.333248+0000", 2024-09-20T22:45:47.648 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:45:47.648 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.648 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.648 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:45:47.648 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.333248+0000", 2024-09-20T22:45:47.648 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:45:47.648 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.648 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.648 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:45:47.648 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.333248+0000", 2024-09-20T22:45:47.648 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.1399999999999997e-07 2024-09-20T22:45:47.648 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.648 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.648 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:45:47.648 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.333250+0000", 2024-09-20T22:45:47.649 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.8199999999999999e-06 2024-09-20T22:45:47.649 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.649 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.649 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:45:47.649 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.333257+0000", 2024-09-20T22:45:47.649 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.43e-06 2024-09-20T22:45:47.649 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.649 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.649 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:45:47.649 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.333261+0000", 2024-09-20T22:45:47.649 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.3670000000000001e-06 2024-09-20T22:45:47.649 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.649 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.649 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:45:47.650 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.333351+0000", 2024-09-20T22:45:47.650 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.0486000000000001e-05 2024-09-20T22:45:47.650 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.650 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.650 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting_for_map not empty", 2024-09-20T22:45:47.650 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.333357+0000", 2024-09-20T22:45:47.650 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.3520000000000003e-06 2024-09-20T22:45:47.650 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.650 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.650 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:45:47.650 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.336912+0000", 2024-09-20T22:45:47.650 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.003554612 2024-09-20T22:45:47.650 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.650 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.651 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for blocked object", 2024-09-20T22:45:47.651 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.336957+0000", 2024-09-20T22:45:47.651 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.4919000000000001e-05 2024-09-20T22:45:47.651 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.651 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.651 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:45:47.651 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.349989+0000", 2024-09-20T22:45:47.651 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.01303238 2024-09-20T22:45:47.651 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.651 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.651 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:45:47.651 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.350058+0000", 2024-09-20T22:45:47.651 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.8418999999999997e-05 2024-09-20T22:45:47.651 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.652 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.652 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:45:47.652 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.350129+0000", 2024-09-20T22:45:47.652 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.0984999999999999e-05 2024-09-20T22:45:47.652 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:45:47.652 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:45:47.652 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:45:47.652 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.652 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.652 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:804 7.17 7:ea720b85:::smithi08477203-26:head [copy-from ver 61] snapc 19=[17] ondisk+write+known_if_redirected+supports_pool_eio e145)", 2024-09-20T22:45:47.652 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:44.451241+0000", 2024-09-20T22:45:47.652 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 123.185070778, 2024-09-20T22:45:47.652 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.012865619999999999, 2024-09-20T22:45:47.652 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:45:47.653 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:45:47.653 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:45:47.653 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:45:47.653 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:45:47.654 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:45:47.654 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:45:47.654 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 804 2024-09-20T22:45:47.654 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.654 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:45:47.655 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.655 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:45:47.655 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.451241+0000", 2024-09-20T22:45:47.655 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:45:47.655 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.655 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.655 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:45:47.655 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.451241+0000", 2024-09-20T22:45:47.655 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:45:47.655 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.655 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.655 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:45:47.655 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.451244+0000", 2024-09-20T22:45:47.656 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.2689999999999998e-06 2024-09-20T22:45:47.656 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.656 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.656 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:45:47.656 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.451255+0000", 2024-09-20T22:45:47.656 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1151e-05 2024-09-20T22:45:47.656 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.656 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.656 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:45:47.656 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.451274+0000", 2024-09-20T22:45:47.656 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9389999999999999e-05 2024-09-20T22:45:47.656 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.656 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.656 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:45:47.656 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.451286+0000", 2024-09-20T22:45:47.657 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1703e-05 2024-09-20T22:45:47.657 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.657 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.657 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:45:47.657 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.451361+0000", 2024-09-20T22:45:47.657 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.5310999999999998e-05 2024-09-20T22:45:47.657 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.657 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.657 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:45:47.657 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.451462+0000", 2024-09-20T22:45:47.657 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000100809 2024-09-20T22:45:47.657 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.657 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.657 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 2,5", 2024-09-20T22:45:47.657 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.456603+0000", 2024-09-20T22:45:47.658 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0051411169999999997 2024-09-20T22:45:47.658 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.658 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.658 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:45:47.658 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.463017+0000", 2024-09-20T22:45:47.658 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.006413307 2024-09-20T22:45:47.658 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.658 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.658 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:45:47.658 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.463754+0000", 2024-09-20T22:45:47.658 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00073758000000000005 2024-09-20T22:45:47.658 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.658 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.658 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:45:47.659 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.464036+0000", 2024-09-20T22:45:47.659 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00028171000000000001 2024-09-20T22:45:47.659 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.659 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.659 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:45:47.659 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.464076+0000", 2024-09-20T22:45:47.659 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.9736e-05 2024-09-20T22:45:47.659 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.659 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.659 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:45:47.659 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.464107+0000", 2024-09-20T22:45:47.659 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.1537e-05 2024-09-20T22:45:47.659 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:45:47.659 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:45:47.659 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:45:47.660 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.660 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.660 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:878 7.17 7:ea720b85:::smithi08477203-26:head [read 0~2369787,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e154)", 2024-09-20T22:45:47.660 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:53.126973+0000", 2024-09-20T22:45:47.660 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 114.50933923300001, 2024-09-20T22:45:47.660 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.010984944, 2024-09-20T22:45:47.660 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:45:47.660 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:45:47.660 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:45:47.660 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:45:47.660 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:45:47.660 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:45:47.660 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 878 2024-09-20T22:45:47.660 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.661 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:45:47.661 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.661 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:45:47.661 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:53.126973+0000", 2024-09-20T22:45:47.661 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:45:47.661 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.661 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.661 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:45:47.661 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:53.126973+0000", 2024-09-20T22:45:47.661 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:45:47.661 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.661 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.661 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:45:47.661 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:53.126974+0000", 2024-09-20T22:45:47.661 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.2200000000000004e-07 2024-09-20T22:45:47.662 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.662 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.662 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:45:47.662 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:53.126990+0000", 2024-09-20T22:45:47.662 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.6597999999999999e-05 2024-09-20T22:45:47.662 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.662 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.662 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:45:47.662 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:53.126998+0000", 2024-09-20T22:45:47.662 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.038e-06 2024-09-20T22:45:47.662 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.662 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.662 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:45:47.662 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:53.127008+0000", 2024-09-20T22:45:47.663 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0108999999999999e-05 2024-09-20T22:45:47.663 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.663 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.663 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:45:47.663 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:53.127089+0000", 2024-09-20T22:45:47.663 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.0863000000000002e-05 2024-09-20T22:45:47.663 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.663 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.663 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting_for_map not empty", 2024-09-20T22:45:47.663 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:53.127095+0000", 2024-09-20T22:45:47.663 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.806e-06 2024-09-20T22:45:47.663 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.663 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.663 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:45:47.664 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:53.135275+0000", 2024-09-20T22:45:47.664 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0081799060000000007 2024-09-20T22:45:47.664 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.664 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.664 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:45:47.664 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:53.135352+0000", 2024-09-20T22:45:47.664 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.6641e-05 2024-09-20T22:45:47.664 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.664 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.664 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:45:47.664 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:53.137958+0000", 2024-09-20T22:45:47.664 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0026063610000000002 2024-09-20T22:45:47.664 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:45:47.664 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:45:47.664 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:45:47.665 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.665 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.665 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:1527 7.17 7:ea720b85:::smithi08477203-26:head [copy-from ver 83] snapc 33=[33] ondisk+write+known_if_redirected+supports_pool_eio e197)", 2024-09-20T22:45:47.665 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:44:36.198921+0000", 2024-09-20T22:45:47.665 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 71.437391453999993, 2024-09-20T22:45:47.665 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.021621703999999999, 2024-09-20T22:45:47.665 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:45:47.665 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:45:47.665 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:45:47.665 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:45:47.665 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:45:47.665 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:45:47.665 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 1527 2024-09-20T22:45:47.665 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.666 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:45:47.666 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.666 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:45:47.666 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.198921+0000", 2024-09-20T22:45:47.666 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:45:47.666 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.666 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.666 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:45:47.666 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.198921+0000", 2024-09-20T22:45:47.666 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:45:47.666 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.666 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.666 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:45:47.666 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.198923+0000", 2024-09-20T22:45:47.667 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.0080000000000001e-06 2024-09-20T22:45:47.667 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.667 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.667 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:45:47.667 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.198934+0000", 2024-09-20T22:45:47.667 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0923e-05 2024-09-20T22:45:47.667 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.667 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.667 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:45:47.667 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.198955+0000", 2024-09-20T22:45:47.667 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.0806e-05 2024-09-20T22:45:47.667 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.667 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.667 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:45:47.667 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.198966+0000", 2024-09-20T22:45:47.668 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1779e-05 2024-09-20T22:45:47.668 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.668 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.668 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:45:47.668 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.199051+0000", 2024-09-20T22:45:47.668 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.4357999999999994e-05 2024-09-20T22:45:47.668 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.668 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.668 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op must wait for map", 2024-09-20T22:45:47.668 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.199058+0000", 2024-09-20T22:45:47.668 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.8530000000000003e-06 2024-09-20T22:45:47.668 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.668 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.668 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:45:47.669 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.202420+0000", 2024-09-20T22:45:47.669 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0033623350000000002 2024-09-20T22:45:47.669 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.669 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.669 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:45:47.669 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.202498+0000", 2024-09-20T22:45:47.669 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.8316e-05 2024-09-20T22:45:47.669 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.669 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.669 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 2,5", 2024-09-20T22:45:47.669 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.209140+0000", 2024-09-20T22:45:47.669 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0066415759999999997 2024-09-20T22:45:47.669 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.669 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.670 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:45:47.670 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.214931+0000", 2024-09-20T22:45:47.670 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0057916520000000004 2024-09-20T22:45:47.670 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.670 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.670 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:45:47.670 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.220009+0000", 2024-09-20T22:45:47.670 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0050773750000000003 2024-09-20T22:45:47.670 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.670 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.670 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:45:47.670 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.220417+0000", 2024-09-20T22:45:47.670 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00040778599999999999 2024-09-20T22:45:47.670 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.671 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.671 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:45:47.671 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.220479+0000", 2024-09-20T22:45:47.671 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.1977000000000006e-05 2024-09-20T22:45:47.671 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.671 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.671 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:45:47.671 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.220542+0000", 2024-09-20T22:45:47.671 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.3960000000000004e-05 2024-09-20T22:45:47.671 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:45:47.671 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:45:47.671 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:45:47.671 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.671 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.672 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:1528 7.17 7:ea720b85:::smithi08477203-26:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e197)", 2024-09-20T22:45:47.672 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:44:36.198994+0000", 2024-09-20T22:45:47.672 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 71.437318727000005, 2024-09-20T22:45:47.672 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.015013924, 2024-09-20T22:45:47.672 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:45:47.672 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:45:47.672 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:45:47.672 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:45:47.672 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:45:47.672 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:45:47.672 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 1528 2024-09-20T22:45:47.672 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.672 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:45:47.672 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.673 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:45:47.673 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.198994+0000", 2024-09-20T22:45:47.673 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:45:47.673 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.673 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.673 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:45:47.673 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.198994+0000", 2024-09-20T22:45:47.673 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:45:47.673 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.673 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.673 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:45:47.673 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.198994+0000", 2024-09-20T22:45:47.673 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.8100000000000003e-07 2024-09-20T22:45:47.673 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.674 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.674 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:45:47.674 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.198996+0000", 2024-09-20T22:45:47.674 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.3e-06 2024-09-20T22:45:47.674 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.674 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.674 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:45:47.674 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.199008+0000", 2024-09-20T22:45:47.674 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1603999999999999e-05 2024-09-20T22:45:47.674 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.674 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.674 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:45:47.674 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.199013+0000", 2024-09-20T22:45:47.674 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.4500000000000003e-06 2024-09-20T22:45:47.675 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.675 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.675 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:45:47.675 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.199120+0000", 2024-09-20T22:45:47.675 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000106772 2024-09-20T22:45:47.675 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.675 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.675 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting_for_map not empty", 2024-09-20T22:45:47.675 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.199126+0000", 2024-09-20T22:45:47.675 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.8320000000000002e-06 2024-09-20T22:45:47.675 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.675 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.675 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:45:47.675 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.202625+0000", 2024-09-20T22:45:47.676 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0034986309999999999 2024-09-20T22:45:47.676 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.676 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.676 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for blocked object", 2024-09-20T22:45:47.676 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.202667+0000", 2024-09-20T22:45:47.676 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.2759000000000002e-05 2024-09-20T22:45:47.676 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.676 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.676 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:45:47.676 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.213787+0000", 2024-09-20T22:45:47.676 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.011119377 2024-09-20T22:45:47.676 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.676 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.676 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:45:47.676 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.213902+0000", 2024-09-20T22:45:47.676 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000115369 2024-09-20T22:45:47.677 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.677 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.677 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:45:47.677 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.214007+0000", 2024-09-20T22:45:47.677 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000105349 2024-09-20T22:45:47.677 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:45:47.677 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:45:47.677 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:45:47.677 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.677 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.677 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:1525 7.c e197/94 7:3f865ef5:::smithi08477203-28:head v 197'230, mlcod=194'226)", 2024-09-20T22:45:47.677 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:44:36.215464+0000", 2024-09-20T22:45:47.677 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 71.420848023999994, 2024-09-20T22:45:47.677 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.011855274000000001, 2024-09-20T22:45:47.678 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:45:47.678 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:45:47.678 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:45:47.678 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:45:47.678 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.678 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:45:47.678 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.215464+0000", 2024-09-20T22:45:47.678 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:45:47.678 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.678 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.678 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:45:47.678 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.215464+0000", 2024-09-20T22:45:47.678 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:45:47.678 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.679 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.679 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:45:47.679 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.215465+0000", 2024-09-20T22:45:47.679 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.5100000000000001e-07 2024-09-20T22:45:47.679 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.679 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.679 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:45:47.679 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.218161+0000", 2024-09-20T22:45:47.679 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0026963909999999998 2024-09-20T22:45:47.679 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.679 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.679 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:45:47.679 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.218182+0000", 2024-09-20T22:45:47.679 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.0582999999999999e-05 2024-09-20T22:45:47.680 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.680 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.680 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:45:47.680 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.218192+0000", 2024-09-20T22:45:47.680 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0441999999999999e-05 2024-09-20T22:45:47.680 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.680 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.680 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:45:47.680 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.218251+0000", 2024-09-20T22:45:47.680 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.8884000000000001e-05 2024-09-20T22:45:47.680 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.680 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.680 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:45:47.680 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.218270+0000", 2024-09-20T22:45:47.680 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9324e-05 2024-09-20T22:45:47.681 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.681 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.681 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:45:47.681 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.227272+0000", 2024-09-20T22:45:47.681 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0090018509999999999 2024-09-20T22:45:47.681 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.681 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.681 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:45:47.681 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.227319+0000", 2024-09-20T22:45:47.681 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.7447999999999997e-05 2024-09-20T22:45:47.681 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:45:47.681 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:45:47.681 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:45:47.681 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.682 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.682 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:1550 7.17 7:ed54065f:::smithi08477203-12:head [copy-from ver 88] snapc 34=[34,33] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2024-09-20T22:45:47.682 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:44:37.238376+0000", 2024-09-20T22:45:47.682 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 70.397935814999997, 2024-09-20T22:45:47.682 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.016634917999999999, 2024-09-20T22:45:47.682 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:45:47.682 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:45:47.682 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:45:47.682 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:45:47.682 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:45:47.682 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:45:47.682 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 1550 2024-09-20T22:45:47.682 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.682 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:45:47.683 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.683 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:45:47.683 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:37.238376+0000", 2024-09-20T22:45:47.683 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:45:47.683 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.683 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.683 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:45:47.683 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:37.238376+0000", 2024-09-20T22:45:47.683 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:45:47.683 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.683 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.683 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:45:47.683 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:37.238379+0000", 2024-09-20T22:45:47.683 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.7989999999999998e-06 2024-09-20T22:45:47.683 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.684 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.684 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:45:47.684 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:37.238408+0000", 2024-09-20T22:45:47.684 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.8980999999999999e-05 2024-09-20T22:45:47.684 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.684 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.684 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:45:47.684 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:37.238433+0000", 2024-09-20T22:45:47.684 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.5185999999999999e-05 2024-09-20T22:45:47.684 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.684 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.684 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:45:47.684 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:37.238446+0000", 2024-09-20T22:45:47.684 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2883e-05 2024-09-20T22:45:47.684 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.685 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.685 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:45:47.685 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:37.238523+0000", 2024-09-20T22:45:47.685 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.6396999999999994e-05 2024-09-20T22:45:47.685 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.685 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.685 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:45:47.685 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:37.238618+0000", 2024-09-20T22:45:47.685 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.5049000000000004e-05 2024-09-20T22:45:47.685 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.685 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.685 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 2,5", 2024-09-20T22:45:47.686 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:37.246556+0000", 2024-09-20T22:45:47.686 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0079378090000000005 2024-09-20T22:45:47.686 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.686 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.686 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:45:47.686 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:37.254025+0000", 2024-09-20T22:45:47.686 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0074697139999999997 2024-09-20T22:45:47.686 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.686 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.686 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:45:47.686 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:37.254033+0000", 2024-09-20T22:45:47.686 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.6979999999999998e-06 2024-09-20T22:45:47.686 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.687 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.687 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:45:47.687 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:37.254919+0000", 2024-09-20T22:45:47.687 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00088564700000000004 2024-09-20T22:45:47.687 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.687 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.687 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:45:47.687 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:37.254958+0000", 2024-09-20T22:45:47.687 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.9515999999999999e-05 2024-09-20T22:45:47.687 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.687 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.687 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:45:47.687 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:37.255011+0000", 2024-09-20T22:45:47.687 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.3239000000000003e-05 2024-09-20T22:45:47.688 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:45:47.688 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:45:47.688 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:45:47.688 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.688 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.688 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:1551 7.17 7:ed54065f:::smithi08477203-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e198)", 2024-09-20T22:45:47.688 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:44:37.238470+0000", 2024-09-20T22:45:47.688 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 70.397842639000004, 2024-09-20T22:45:47.688 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.011771032000000001, 2024-09-20T22:45:47.688 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:45:47.688 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:45:47.688 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:45:47.688 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:45:47.688 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:45:47.689 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:45:47.689 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 1551 2024-09-20T22:45:47.689 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.689 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:45:47.689 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.689 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:45:47.689 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:37.238470+0000", 2024-09-20T22:45:47.689 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:45:47.689 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.689 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.689 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:45:47.689 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:37.238470+0000", 2024-09-20T22:45:47.689 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:45:47.689 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.690 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.690 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:45:47.690 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:37.238470+0000", 2024-09-20T22:45:47.690 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.6899999999999997e-07 2024-09-20T22:45:47.690 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.690 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.690 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:45:47.690 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:37.238473+0000", 2024-09-20T22:45:47.690 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.7709999999999998e-06 2024-09-20T22:45:47.690 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.690 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.690 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:45:47.690 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:37.238489+0000", 2024-09-20T22:45:47.690 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.5591999999999999e-05 2024-09-20T22:45:47.691 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.691 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.691 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:45:47.691 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:37.238495+0000", 2024-09-20T22:45:47.691 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.1750000000000002e-06 2024-09-20T22:45:47.691 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.691 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.691 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:45:47.691 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:37.238802+0000", 2024-09-20T22:45:47.691 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00030682799999999998 2024-09-20T22:45:47.691 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.691 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.691 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for blocked object", 2024-09-20T22:45:47.691 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:37.238851+0000", 2024-09-20T22:45:47.692 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.9199000000000001e-05 2024-09-20T22:45:47.692 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.692 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.692 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:45:47.692 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:37.250099+0000", 2024-09-20T22:45:47.692 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.011247872000000001 2024-09-20T22:45:47.692 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.692 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.692 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:45:47.692 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:37.250166+0000", 2024-09-20T22:45:47.692 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.6990999999999996e-05 2024-09-20T22:45:47.692 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.692 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.692 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:45:47.693 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:37.250241+0000", 2024-09-20T22:45:47.693 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.5035000000000002e-05 2024-09-20T22:45:47.693 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:45:47.693 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:45:47.693 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:45:47.693 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.693 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.693 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:1570 7.5 7:a160b5d4:::smithi08477203-16:head [write 766452~565707] snapc 34=[] ondisk+write+known_if_redirected+supports_pool_eio e200)", 2024-09-20T22:45:47.693 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:44:39.233104+0000", 2024-09-20T22:45:47.693 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 68.403208203999995, 2024-09-20T22:45:47.693 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.010902642000000001, 2024-09-20T22:45:47.693 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:45:47.693 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:45:47.693 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:45:47.693 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:45:47.694 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:45:47.694 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:45:47.694 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 1570 2024-09-20T22:45:47.694 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.694 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:45:47.694 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.694 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:45:47.694 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:39.233104+0000", 2024-09-20T22:45:47.694 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:45:47.694 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.694 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.694 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:45:47.694 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:39.233104+0000", 2024-09-20T22:45:47.694 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:45:47.695 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.695 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.695 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:45:47.695 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:39.233107+0000", 2024-09-20T22:45:47.695 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.4729999999999999e-06 2024-09-20T22:45:47.695 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.695 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.695 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:45:47.695 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:39.234121+0000", 2024-09-20T22:45:47.695 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0010140730000000001 2024-09-20T22:45:47.695 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.695 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.695 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:45:47.695 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:39.234134+0000", 2024-09-20T22:45:47.696 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3893000000000001e-05 2024-09-20T22:45:47.696 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.696 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.696 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:45:47.696 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:39.234145+0000", 2024-09-20T22:45:47.696 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0349e-05 2024-09-20T22:45:47.696 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.696 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.696 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:45:47.696 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:39.234203+0000", 2024-09-20T22:45:47.696 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.7880000000000001e-05 2024-09-20T22:45:47.696 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.696 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.696 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op must wait for map", 2024-09-20T22:45:47.697 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:39.234209+0000", 2024-09-20T22:45:47.697 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.9039999999999997e-06 2024-09-20T22:45:47.697 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.697 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.697 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:45:47.697 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:39.236992+0000", 2024-09-20T22:45:47.697 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0027830960000000001 2024-09-20T22:45:47.697 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.697 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.697 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:45:47.697 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:39.237088+0000", 2024-09-20T22:45:47.697 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.6750999999999996e-05 2024-09-20T22:45:47.697 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.697 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.697 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 0,2", 2024-09-20T22:45:47.698 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:39.237229+0000", 2024-09-20T22:45:47.698 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00014063699999999999 2024-09-20T22:45:47.698 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.698 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.698 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:45:47.698 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:39.243760+0000", 2024-09-20T22:45:47.698 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0065313510000000003 2024-09-20T22:45:47.698 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.698 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.698 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:45:47.698 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:39.243813+0000", 2024-09-20T22:45:47.698 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.2228999999999997e-05 2024-09-20T22:45:47.698 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.699 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.699 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:45:47.699 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:39.243892+0000", 2024-09-20T22:45:47.699 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.9003000000000002e-05 2024-09-20T22:45:47.699 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.699 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.699 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:45:47.699 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:39.243952+0000", 2024-09-20T22:45:47.699 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.0544000000000002e-05 2024-09-20T22:45:47.699 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.699 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.699 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:45:47.699 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:39.244007+0000", 2024-09-20T22:45:47.699 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.4459000000000002e-05 2024-09-20T22:45:47.699 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:45:47.700 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:45:47.700 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:45:47.700 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.700 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.700 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:1652 7.c e206/94 7:31d0211f:::smithi08477203-25:head v 206'260, mlcod=206'258)", 2024-09-20T22:45:47.700 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:44:45.175893+0000", 2024-09-20T22:45:47.700 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 62.460419641000001, 2024-09-20T22:45:47.700 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.010529134000000001, 2024-09-20T22:45:47.700 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:45:47.700 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:45:47.700 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:45:47.700 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:45:47.700 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.700 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:45:47.701 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:45.175893+0000", 2024-09-20T22:45:47.701 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:45:47.701 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.701 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.701 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:45:47.701 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:45.175893+0000", 2024-09-20T22:45:47.701 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:45:47.701 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.701 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.701 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:45:47.701 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:45.175895+0000", 2024-09-20T22:45:47.701 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.0080000000000001e-06 2024-09-20T22:45:47.701 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.701 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.701 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:45:47.702 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:45.178758+0000", 2024-09-20T22:45:47.702 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0028635980000000002 2024-09-20T22:45:47.702 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.702 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.702 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:45:47.702 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:45.178782+0000", 2024-09-20T22:45:47.702 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.3740999999999999e-05 2024-09-20T22:45:47.702 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.702 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.702 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:45:47.702 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:45.178795+0000", 2024-09-20T22:45:47.702 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2859e-05 2024-09-20T22:45:47.702 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.702 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.703 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:45:47.703 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:45.178891+0000", 2024-09-20T22:45:47.703 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.5952999999999994e-05 2024-09-20T22:45:47.703 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.703 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.703 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:45:47.703 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:45.178931+0000", 2024-09-20T22:45:47.703 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.9990999999999997e-05 2024-09-20T22:45:47.703 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.703 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.703 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:45:47.703 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:45.186379+0000", 2024-09-20T22:45:47.703 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0074482230000000003 2024-09-20T22:45:47.703 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.703 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.704 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:45:47.704 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:45.186422+0000", 2024-09-20T22:45:47.704 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.2760999999999999e-05 2024-09-20T22:45:47.704 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:45:47.704 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:45:47.704 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:45:47.704 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.704 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.704 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:1799 7.17 7:ea720b85:::smithi08477203-26:head [write 521083~714336,stat] snapc 46=[45,44,43,42,41,40,3e] ondisk+write+known_if_redirected+supports_pool_eio e229)", 2024-09-20T22:45:47.704 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:45:08.170663+0000", 2024-09-20T22:45:47.704 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 39.465649169999999, 2024-09-20T22:45:47.704 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.016368226999999999, 2024-09-20T22:45:47.704 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:45:47.704 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:45:47.705 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:45:47.705 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:45:47.705 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:45:47.705 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:45:47.705 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 1799 2024-09-20T22:45:47.705 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.705 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:45:47.705 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.705 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:45:47.705 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.170663+0000", 2024-09-20T22:45:47.705 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:45:47.705 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.705 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.705 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:45:47.706 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.170663+0000", 2024-09-20T22:45:47.706 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:45:47.706 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.706 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.706 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:45:47.706 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.170664+0000", 2024-09-20T22:45:47.706 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.085e-06 2024-09-20T22:45:47.706 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.706 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.706 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:45:47.706 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.171481+0000", 2024-09-20T22:45:47.706 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00081653900000000002 2024-09-20T22:45:47.706 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.706 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.707 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:45:47.707 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.171492+0000", 2024-09-20T22:45:47.707 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1565e-05 2024-09-20T22:45:47.707 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.707 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.707 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:45:47.707 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.171498+0000", 2024-09-20T22:45:47.707 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.925e-06 2024-09-20T22:45:47.707 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.707 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.707 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:45:47.707 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.171547+0000", 2024-09-20T22:45:47.707 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.8916999999999999e-05 2024-09-20T22:45:47.707 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.708 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.708 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:45:47.708 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.171611+0000", 2024-09-20T22:45:47.708 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.3838999999999996e-05 2024-09-20T22:45:47.708 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.708 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.708 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 2,5", 2024-09-20T22:45:47.708 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.171740+0000", 2024-09-20T22:45:47.708 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00012881500000000001 2024-09-20T22:45:47.708 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.708 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.708 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:45:47.708 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.173862+0000", 2024-09-20T22:45:47.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:47.710 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0021225649999999999 2024-09-20T22:45:47.710 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.710 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.710 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:45:47.711 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.179252+0000", 2024-09-20T22:45:47.711 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0053895490000000004 2024-09-20T22:45:47.711 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.711 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.711 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:45:47.711 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.186956+0000", 2024-09-20T22:45:47.711 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0077038050000000002 2024-09-20T22:45:47.711 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.711 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.711 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:45:47.711 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.186993+0000", 2024-09-20T22:45:47.711 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.7141000000000002e-05 2024-09-20T22:45:47.711 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.711 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.712 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:45:47.712 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.187031+0000", 2024-09-20T22:45:47.712 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.8482000000000002e-05 2024-09-20T22:45:47.712 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:45:47.712 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:45:47.712 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:45:47.712 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.712 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.712 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:1800 7.17 7:ea720b85:::smithi08477203-26:head [write 1724715~555421] snapc 46=[45,44,43,42,41,40,3e] ondisk+write+known_if_redirected+supports_pool_eio e229)", 2024-09-20T22:45:47.712 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:45:08.177237+0000", 2024-09-20T22:45:47.712 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 39.459075544999997, 2024-09-20T22:45:47.712 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.012771694, 2024-09-20T22:45:47.712 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:45:47.712 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:45:47.713 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:45:47.713 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:45:47.713 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:45:47.713 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:45:47.713 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 1800 2024-09-20T22:45:47.713 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.713 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:45:47.713 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.713 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:45:47.713 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.177237+0000", 2024-09-20T22:45:47.713 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:45:47.713 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.713 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.713 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:45:47.713 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.177237+0000", 2024-09-20T22:45:47.714 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:45:47.714 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.714 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.714 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:45:47.714 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.177238+0000", 2024-09-20T22:45:47.714 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.6080000000000001e-06 2024-09-20T22:45:47.714 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.714 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.714 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:45:47.714 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.177844+0000", 2024-09-20T22:45:47.714 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00060579999999999998 2024-09-20T22:45:47.714 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.714 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.714 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:45:47.715 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.177857+0000", 2024-09-20T22:45:47.715 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2529e-05 2024-09-20T22:45:47.715 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.715 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.715 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:45:47.715 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.177864+0000", 2024-09-20T22:45:47.715 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.2470000000000002e-06 2024-09-20T22:45:47.715 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.715 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.715 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:45:47.715 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.177908+0000", 2024-09-20T22:45:47.715 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.4329999999999997e-05 2024-09-20T22:45:47.715 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.715 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.715 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:45:47.716 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.177977+0000", 2024-09-20T22:45:47.716 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.8598999999999998e-05 2024-09-20T22:45:47.716 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.716 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.716 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 2,5", 2024-09-20T22:45:47.716 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.178079+0000", 2024-09-20T22:45:47.716 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000102074 2024-09-20T22:45:47.716 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.716 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.716 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:45:47.716 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.181043+0000", 2024-09-20T22:45:47.716 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0029639810000000001 2024-09-20T22:45:47.716 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.716 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.716 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:45:47.717 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.181481+0000", 2024-09-20T22:45:47.717 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00043855899999999999 2024-09-20T22:45:47.717 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.717 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.717 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:45:47.717 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.189928+0000", 2024-09-20T22:45:47.717 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0084463149999999994 2024-09-20T22:45:47.717 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.717 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.717 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:45:47.717 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.189968+0000", 2024-09-20T22:45:47.717 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.0268999999999997e-05 2024-09-20T22:45:47.717 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.717 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.718 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:45:47.718 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.190008+0000", 2024-09-20T22:45:47.718 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.0383000000000003e-05 2024-09-20T22:45:47.718 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:45:47.718 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:45:47.718 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:45:47.718 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.718 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.718 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:1801 7.17 7:ea720b85:::smithi08477203-26:head [setxattr _header (57) in=7b,truncate 2521083] snapc 46=[45,44,43,42,41,40,3e] ondisk+write+known_if_redirected+supports_pool_eio e229)", 2024-09-20T22:45:47.718 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:45:08.177878+0000", 2024-09-20T22:45:47.718 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 39.458434293000003, 2024-09-20T22:45:47.718 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.012232588000000001, 2024-09-20T22:45:47.718 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:45:47.719 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:45:47.719 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:45:47.719 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:45:47.719 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:45:47.719 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:45:47.719 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 1801 2024-09-20T22:45:47.719 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.719 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:45:47.719 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.719 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:45:47.719 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.177878+0000", 2024-09-20T22:45:47.719 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:45:47.719 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.719 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.719 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:45:47.720 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.177878+0000", 2024-09-20T22:45:47.720 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:45:47.720 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.720 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.720 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:45:47.720 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.177878+0000", 2024-09-20T22:45:47.720 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.3399999999999999e-07 2024-09-20T22:45:47.720 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.720 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.720 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:45:47.720 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.177886+0000", 2024-09-20T22:45:47.720 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.9410000000000007e-06 2024-09-20T22:45:47.720 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.720 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.721 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:45:47.721 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.177891+0000", 2024-09-20T22:45:47.721 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.741e-06 2024-09-20T22:45:47.721 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.721 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.721 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:45:47.721 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.177895+0000", 2024-09-20T22:45:47.721 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.7400000000000002e-06 2024-09-20T22:45:47.721 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.721 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.721 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:45:47.721 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.178777+0000", 2024-09-20T22:45:47.721 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00088171899999999999 2024-09-20T22:45:47.721 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.721 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.722 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:45:47.722 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.178862+0000", 2024-09-20T22:45:47.722 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.5223999999999996e-05 2024-09-20T22:45:47.722 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.722 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.722 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 2,5", 2024-09-20T22:45:47.722 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.178966+0000", 2024-09-20T22:45:47.722 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000104044 2024-09-20T22:45:47.722 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.722 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.722 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:45:47.722 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.181156+0000", 2024-09-20T22:45:47.722 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0021905359999999999 2024-09-20T22:45:47.722 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.723 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.723 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:45:47.723 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.181519+0000", 2024-09-20T22:45:47.723 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00036243100000000003 2024-09-20T22:45:47.723 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.723 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.723 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:45:47.723 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.190041+0000", 2024-09-20T22:45:47.723 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0085219200000000005 2024-09-20T22:45:47.723 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.723 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.723 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:45:47.723 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.190076+0000", 2024-09-20T22:45:47.723 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.5091e-05 2024-09-20T22:45:47.724 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.724 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.724 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:45:47.724 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.190111+0000", 2024-09-20T22:45:47.724 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.4666999999999999e-05 2024-09-20T22:45:47.724 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:45:47.724 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:45:47.724 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:45:47.724 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.724 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.724 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:1802 7.17 7:ea720b85:::smithi08477203-26:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e229)", 2024-09-20T22:45:47.724 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:45:08.177901+0000", 2024-09-20T22:45:47.724 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 39.458411601000002, 2024-09-20T22:45:47.724 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.012470591, 2024-09-20T22:45:47.725 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:45:47.725 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:45:47.725 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:45:47.725 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:45:47.725 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:45:47.725 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:45:47.725 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 1802 2024-09-20T22:45:47.725 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.725 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:45:47.725 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.725 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:45:47.725 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.177901+0000", 2024-09-20T22:45:47.725 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:45:47.725 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.725 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.726 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:45:47.726 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.177901+0000", 2024-09-20T22:45:47.726 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:45:47.726 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.726 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.726 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:45:47.726 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.177901+0000", 2024-09-20T22:45:47.726 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.2899999999999999e-07 2024-09-20T22:45:47.726 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.726 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.726 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:45:47.726 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.177902+0000", 2024-09-20T22:45:47.726 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.452e-06 2024-09-20T22:45:47.726 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.727 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.727 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:45:47.727 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.177906+0000", 2024-09-20T22:45:47.727 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.788e-06 2024-09-20T22:45:47.727 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.727 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.727 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:45:47.727 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.177909+0000", 2024-09-20T22:45:47.727 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.9840000000000001e-06 2024-09-20T22:45:47.727 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.727 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.727 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:45:47.727 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.179283+0000", 2024-09-20T22:45:47.728 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001373733 2024-09-20T22:45:47.728 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.728 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.728 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for rw locks", 2024-09-20T22:45:47.728 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.179364+0000", 2024-09-20T22:45:47.728 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.1209000000000001e-05 2024-09-20T22:45:47.728 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.728 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.728 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:45:47.728 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.190140+0000", 2024-09-20T22:45:47.728 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.010775527 2024-09-20T22:45:47.728 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.728 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.728 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:45:47.728 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.190186+0000", 2024-09-20T22:45:47.729 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.5853000000000003e-05 2024-09-20T22:45:47.729 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.729 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.729 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:45:47.729 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.190371+0000", 2024-09-20T22:45:47.729 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000185716 2024-09-20T22:45:47.729 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:45:47.729 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:45:47.729 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:45:47.729 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.729 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.729 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:2318 7.5 7:a160b5d4:::smithi08477203-16:head [copy-from ver 168] snapc 55=[55,53,52,51,50,4b,47,3e] ondisk+write+known_if_redirected+supports_pool_eio e258)", 2024-09-20T22:45:47.729 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:45:37.175863+0000", 2024-09-20T22:45:47.730 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 10.460449095, 2024-09-20T22:45:47.730 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.030719883999999999, 2024-09-20T22:45:47.730 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:45:47.730 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:45:47.730 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:45:47.730 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:45:47.730 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:45:47.730 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:45:47.730 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 2318 2024-09-20T22:45:47.730 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.730 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:45:47.730 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.730 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:45:47.730 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.175863+0000", 2024-09-20T22:45:47.730 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:45:47.731 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.731 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.731 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:45:47.731 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.175863+0000", 2024-09-20T22:45:47.731 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:45:47.731 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.731 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.731 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:45:47.731 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.175865+0000", 2024-09-20T22:45:47.731 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.488e-06 2024-09-20T22:45:47.731 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.731 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.731 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:45:47.731 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.175871+0000", 2024-09-20T22:45:47.732 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.438e-06 2024-09-20T22:45:47.732 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.732 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.732 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:45:47.732 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.175881+0000", 2024-09-20T22:45:47.732 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.6430000000000007e-06 2024-09-20T22:45:47.732 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.732 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.732 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:45:47.732 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.175886+0000", 2024-09-20T22:45:47.732 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.3059999999999997e-06 2024-09-20T22:45:47.732 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.732 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.732 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:45:47.733 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.175944+0000", 2024-09-20T22:45:47.733 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.7982999999999997e-05 2024-09-20T22:45:47.733 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.733 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.733 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op must wait for map", 2024-09-20T22:45:47.733 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.175951+0000", 2024-09-20T22:45:47.733 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.9280000000000004e-06 2024-09-20T22:45:47.733 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.733 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.733 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:45:47.733 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.179794+0000", 2024-09-20T22:45:47.733 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0038435209999999999 2024-09-20T22:45:47.733 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.733 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.733 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:45:47.734 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.179940+0000", 2024-09-20T22:45:47.734 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00014550000000000001 2024-09-20T22:45:47.734 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.734 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.734 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 0,2", 2024-09-20T22:45:47.734 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.190992+0000", 2024-09-20T22:45:47.734 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.011051587999999999 2024-09-20T22:45:47.734 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.734 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.734 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:45:47.734 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.195070+0000", 2024-09-20T22:45:47.734 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0040784929999999999 2024-09-20T22:45:47.734 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.734 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.734 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:45:47.735 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.202908+0000", 2024-09-20T22:45:47.735 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0078376690000000002 2024-09-20T22:45:47.735 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.735 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.735 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:45:47.735 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.206517+0000", 2024-09-20T22:45:47.735 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0036088880000000002 2024-09-20T22:45:47.735 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.735 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.735 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:45:47.735 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.206553+0000", 2024-09-20T22:45:47.735 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.6363999999999999e-05 2024-09-20T22:45:47.735 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.735 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.735 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:45:47.736 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.206583+0000", 2024-09-20T22:45:47.736 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.0074999999999999e-05 2024-09-20T22:45:47.736 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:45:47.736 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:45:47.736 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:45:47.736 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.736 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.736 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:2319 7.5 7:a160b5d4:::smithi08477203-16:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e258)", 2024-09-20T22:45:47.736 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:45:37.175983+0000", 2024-09-20T22:45:47.736 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 10.460329339999999, 2024-09-20T22:45:47.736 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.018323547999999999, 2024-09-20T22:45:47.736 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:45:47.736 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:45:47.736 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:45:47.737 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:45:47.737 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:45:47.737 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:45:47.737 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 2319 2024-09-20T22:45:47.737 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.737 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:45:47.737 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.737 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:45:47.737 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.175983+0000", 2024-09-20T22:45:47.737 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:45:47.737 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.737 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.737 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:45:47.737 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.175983+0000", 2024-09-20T22:45:47.737 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:45:47.738 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.738 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.738 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:45:47.738 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.175984+0000", 2024-09-20T22:45:47.738 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.1099999999999995e-07 2024-09-20T22:45:47.738 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.738 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.738 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:45:47.738 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.175996+0000", 2024-09-20T22:45:47.738 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2121e-05 2024-09-20T22:45:47.738 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.738 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.738 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:45:47.738 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.176019+0000", 2024-09-20T22:45:47.739 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.3465999999999999e-05 2024-09-20T22:45:47.739 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.739 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.739 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:45:47.739 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.176026+0000", 2024-09-20T22:45:47.739 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.8190000000000003e-06 2024-09-20T22:45:47.739 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.739 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.739 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:45:47.739 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.176110+0000", 2024-09-20T22:45:47.739 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.3967000000000004e-05 2024-09-20T22:45:47.739 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.739 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.739 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting_for_map not empty", 2024-09-20T22:45:47.740 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.176116+0000", 2024-09-20T22:45:47.740 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.0249999999999999e-06 2024-09-20T22:45:47.740 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.740 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.740 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:45:47.740 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.180106+0000", 2024-09-20T22:45:47.740 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0039898980000000004 2024-09-20T22:45:47.740 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.740 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.740 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for blocked object", 2024-09-20T22:45:47.740 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.180135+0000", 2024-09-20T22:45:47.740 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.9595999999999999e-05 2024-09-20T22:45:47.740 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.740 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.741 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:45:47.741 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.194106+0000", 2024-09-20T22:45:47.741 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.013970761999999999 2024-09-20T22:45:47.741 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.741 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.741 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:45:47.741 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.194203+0000", 2024-09-20T22:45:47.741 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.7211000000000002e-05 2024-09-20T22:45:47.741 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.741 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.741 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:45:47.741 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.194306+0000", 2024-09-20T22:45:47.741 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00010307200000000001 2024-09-20T22:45:47.741 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:45:47.742 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:45:47.742 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:45:47.742 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.742 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.742 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(osd.6.0:726 7.19 e260/139 7:9eac4560:::smithi08477203-30:4c v 260'159, mlcod=259'156)", 2024-09-20T22:45:47.742 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:45:39.194116+0000", 2024-09-20T22:45:47.742 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 8.4421957580000004, 2024-09-20T22:45:47.742 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.011958139, 2024-09-20T22:45:47.742 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:45:47.742 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:45:47.742 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:45:47.742 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:45:47.742 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.742 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:45:47.743 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.194116+0000", 2024-09-20T22:45:47.743 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:45:47.743 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.743 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.743 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:45:47.743 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.194116+0000", 2024-09-20T22:45:47.743 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:45:47.743 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.743 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.743 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:45:47.743 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.194117+0000", 2024-09-20T22:45:47.743 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.63e-07 2024-09-20T22:45:47.743 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.743 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.744 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:45:47.744 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.194123+0000", 2024-09-20T22:45:47.744 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.7520000000000002e-06 2024-09-20T22:45:47.744 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.744 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.744 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:45:47.744 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.194128+0000", 2024-09-20T22:45:47.744 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.5269999999999998e-06 2024-09-20T22:45:47.744 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.744 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.744 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:45:47.744 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.194138+0000", 2024-09-20T22:45:47.744 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.6299999999999993e-06 2024-09-20T22:45:47.744 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.745 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.745 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:45:47.745 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.194215+0000", 2024-09-20T22:45:47.745 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.6860999999999995e-05 2024-09-20T22:45:47.745 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.745 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.745 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:45:47.745 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.194238+0000", 2024-09-20T22:45:47.745 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.3125999999999998e-05 2024-09-20T22:45:47.745 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.745 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.745 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:45:47.745 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.206030+0000", 2024-09-20T22:45:47.745 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.011792408000000001 2024-09-20T22:45:47.746 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.746 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.746 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:45:47.746 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.206075+0000", 2024-09-20T22:45:47.746 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.4372000000000001e-05 2024-09-20T22:45:47.746 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:45:47.746 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:45:47.746 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:45:47.746 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.746 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.746 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24415.0:230 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch (1) in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e259)", 2024-09-20T22:45:47.746 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:45:39.207410+0000", 2024-09-20T22:45:47.746 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 8.4289022679999999, 2024-09-20T22:45:47.746 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0072298289999999, 2024-09-20T22:45:47.746 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:45:47.747 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:45:47.747 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:45:47.747 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:45:47.747 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24415", 2024-09-20T22:45:47.747 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/2056244102", 2024-09-20T22:45:47.747 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 230 2024-09-20T22:45:47.747 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.747 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:45:47.747 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.747 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:45:47.747 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.207410+0000", 2024-09-20T22:45:47.747 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:45:47.747 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.747 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.748 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:45:47.748 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.207410+0000", 2024-09-20T22:45:47.748 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:45:47.748 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.748 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.748 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:45:47.748 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.207412+0000", 2024-09-20T22:45:47.748 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.7519999999999999e-06 2024-09-20T22:45:47.748 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.748 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.748 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:45:47.748 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.207419+0000", 2024-09-20T22:45:47.748 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.097e-06 2024-09-20T22:45:47.748 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.748 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.748 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:45:47.749 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.207432+0000", 2024-09-20T22:45:47.749 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2719e-05 2024-09-20T22:45:47.749 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.749 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.749 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:45:47.749 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.207464+0000", 2024-09-20T22:45:47.749 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.2820000000000001e-05 2024-09-20T22:45:47.749 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.749 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.749 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:45:47.749 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.207670+0000", 2024-09-20T22:45:47.749 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000205486 2024-09-20T22:45:47.749 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.749 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.750 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-20T22:45:47.750 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.207687+0000", 2024-09-20T22:45:47.750 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.7107000000000001e-05 2024-09-20T22:45:47.750 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.750 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.750 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:45:47.750 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:40.212672+0000", 2024-09-20T22:45:47.750 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.004984562 2024-09-20T22:45:47.750 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.750 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.750 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:45:47.750 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:40.213576+0000", 2024-09-20T22:45:47.750 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00090404799999999996 2024-09-20T22:45:47.750 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.751 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.751 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:45:47.751 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:40.214640+0000", 2024-09-20T22:45:47.751 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0010642379999999999 2024-09-20T22:45:47.751 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:45:47.751 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:45:47.751 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:45:47.751 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.751 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.751 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:2367 7.5 7:a160b5d4:::smithi08477203-16:head [stat,delete] snapc 56=[56,55,53,52,51,50,47,3e] ondisk+write+known_if_redirected+supports_pool_eio e260)", 2024-09-20T22:45:47.751 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:45:39.601299+0000", 2024-09-20T22:45:47.751 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 8.0350128470000008, 2024-09-20T22:45:47.751 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.64714523300000004, 2024-09-20T22:45:47.752 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:45:47.752 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:45:47.752 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:45:47.752 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:45:47.752 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:45:47.752 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:45:47.752 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 2367 2024-09-20T22:45:47.752 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.752 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:45:47.752 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.752 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:45:47.752 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.601299+0000", 2024-09-20T22:45:47.752 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:45:47.752 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.752 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.753 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:45:47.753 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.601299+0000", 2024-09-20T22:45:47.753 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:45:47.753 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.753 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.753 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:45:47.753 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.601301+0000", 2024-09-20T22:45:47.753 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3850000000000001e-06 2024-09-20T22:45:47.753 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.753 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.753 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:45:47.753 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.601306+0000", 2024-09-20T22:45:47.753 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.2730000000000002e-06 2024-09-20T22:45:47.753 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.753 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.754 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:45:47.754 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.601318+0000", 2024-09-20T22:45:47.754 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2148e-05 2024-09-20T22:45:47.754 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.754 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.754 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:45:47.754 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.601324+0000", 2024-09-20T22:45:47.754 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.852e-06 2024-09-20T22:45:47.754 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.754 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.754 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:45:47.754 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.601383+0000", 2024-09-20T22:45:47.754 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.8570000000000003e-05 2024-09-20T22:45:47.754 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.755 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.755 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-20T22:45:47.755 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.601391+0000", 2024-09-20T22:45:47.755 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.5290000000000008e-06 2024-09-20T22:45:47.755 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.755 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.755 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:45:47.755 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:40.221227+0000", 2024-09-20T22:45:47.755 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.619835515 2024-09-20T22:45:47.755 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.755 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.755 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for degraded object", 2024-09-20T22:45:47.755 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:40.223919+0000", 2024-09-20T22:45:47.755 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.002692168 2024-09-20T22:45:47.756 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.756 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.756 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:45:47.756 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:40.244536+0000", 2024-09-20T22:45:47.756 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.020617613 2024-09-20T22:45:47.756 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.756 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.756 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:45:47.756 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:40.244653+0000", 2024-09-20T22:45:47.756 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000116796 2024-09-20T22:45:47.756 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.756 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.756 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 1,2", 2024-09-20T22:45:47.756 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:40.244832+0000", 2024-09-20T22:45:47.756 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00017833500000000001 2024-09-20T22:45:47.757 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.757 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.757 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:45:47.757 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:40.245494+0000", 2024-09-20T22:45:47.757 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00066241399999999995 2024-09-20T22:45:47.757 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.757 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.757 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:45:47.757 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:40.246501+0000", 2024-09-20T22:45:47.757 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0010066299999999999 2024-09-20T22:45:47.757 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.757 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.757 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:45:47.757 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:40.248326+0000", 2024-09-20T22:45:47.757 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001824895 2024-09-20T22:45:47.758 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.758 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.758 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:45:47.758 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:40.248370+0000", 2024-09-20T22:45:47.758 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.4298999999999997e-05 2024-09-20T22:45:47.758 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:45:47.758 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:45:47.758 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:45:47.758 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:40.248445+0000", 2024-09-20T22:45:47.758 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.4810999999999999e-05 2024-09-20T22:45:47.758 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:45:47.758 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:45:47.758 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:45:47.758 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:45:47.759 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:45:47.759 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-20T22:45:47.759 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:47.759 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:45:47.759 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:45:47.860 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:45:47.883 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:47.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:47.884 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:45:47.984 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:45:48.006 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:48.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:48.007 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:45:48.108 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:45:48.129 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:48.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:48.130 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:45:48.155 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is set 2024-09-20T22:45:48.158 INFO:tasks.rados.rados.0.smithi084.stdout:921: done (3 left) 2024-09-20T22:45:48.159 INFO:tasks.rados.rados.0.smithi084.stdout:922: read oid 12 snap -1 2024-09-20T22:45:48.159 INFO:tasks.rados.rados.0.smithi084.stdout:922: expect (ObjNum 312 snap 77 seq_num 312) 2024-09-20T22:45:48.159 INFO:tasks.rados.rados.0.smithi084.stdout:923: snap_remove snap 60 2024-09-20T22:45:48.231 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:45:48.253 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:48.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:48.254 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:45:48.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:48 smithi154 ceph-mon[33264]: osdmap e268: 8 total, 8 up, 6 in 2024-09-20T22:45:48.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:48 smithi154 ceph-mon[33264]: purged_snaps scrub starts 2024-09-20T22:45:48.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:48 smithi154 ceph-mon[33264]: purged_snaps scrub ok 2024-09-20T22:45:48.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:48 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "18"}]: dispatch 2024-09-20T22:45:48.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:48 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "18"}]: dispatch 2024-09-20T22:45:48.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:48 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/2380006037' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:45:48.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:48 smithi154 ceph-mon[33264]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:45:48.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:48 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "18"}]': finished 2024-09-20T22:45:48.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:48 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/2380006037' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T22:45:48.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:48 smithi154 ceph-mon[33264]: osdmap e269: 8 total, 8 up, 6 in 2024-09-20T22:45:48.354 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:45:48.364 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:45:48 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: ::ffff:172.21.15.154 - - [20/Sep/2024:22:45:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-20T22:45:48.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:48 smithi084 ceph-mon[29774]: osdmap e268: 8 total, 8 up, 6 in 2024-09-20T22:45:48.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:48 smithi084 ceph-mon[29774]: purged_snaps scrub starts 2024-09-20T22:45:48.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:48 smithi084 ceph-mon[29774]: purged_snaps scrub ok 2024-09-20T22:45:48.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:48 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "18"}]: dispatch 2024-09-20T22:45:48.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:48 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "18"}]: dispatch 2024-09-20T22:45:48.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:48 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/2380006037' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:45:48.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:48 smithi084 ceph-mon[29774]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:45:48.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:48 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "18"}]': finished 2024-09-20T22:45:48.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:48 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/2380006037' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T22:45:48.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:48 smithi084 ceph-mon[29774]: osdmap e269: 8 total, 8 up, 6 in 2024-09-20T22:45:48.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:48 smithi084 ceph-mon[40024]: osdmap e268: 8 total, 8 up, 6 in 2024-09-20T22:45:48.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:48 smithi084 ceph-mon[40024]: purged_snaps scrub starts 2024-09-20T22:45:48.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:48 smithi084 ceph-mon[40024]: purged_snaps scrub ok 2024-09-20T22:45:48.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:48 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "18"}]: dispatch 2024-09-20T22:45:48.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:48 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "18"}]: dispatch 2024-09-20T22:45:48.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:48 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/2380006037' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:45:48.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:48 smithi084 ceph-mon[40024]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:45:48.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:48 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "18"}]': finished 2024-09-20T22:45:48.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:48 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/2380006037' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T22:45:48.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:48 smithi084 ceph-mon[40024]: osdmap e269: 8 total, 8 up, 6 in 2024-09-20T22:45:48.375 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:48.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:48.376 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:45:48.476 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:45:48.497 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:48.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:48.498 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:45:48.530 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:45:48.530 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:45:48.547 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:45:48.548 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:45:48.563 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:45:48.563 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:45:48.579 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-20T22:45:48.579 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T22:45:48.595 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:45:48.595 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:45:48.600 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:45:48.612 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:45:48.612 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:45:48.620 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:48.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:48.621 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:45:48.629 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:45:48.629 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:45:48.646 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:45:48.646 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:45:48.722 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:45:48.743 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:48.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:48.743 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:45:48.777 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-20T22:45:48.844 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:45:48.865 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:48.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:48.866 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:45:48.966 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:45:48.987 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:48.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:48.988 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:45:49.088 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:45:49.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:49.111 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:49.111 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:45:49.165 INFO:tasks.rados.rados.0.smithi084.stdout:923: done (4 left) 2024-09-20T22:45:49.165 INFO:tasks.rados.rados.0.smithi084.stdout:924: write oid 1 current snap is 88 2024-09-20T22:45:49.165 INFO:tasks.rados.rados.0.smithi084.stdout:924: seq_num 343 ranges {564801=646292,1921559=742115,3202910=561891} 2024-09-20T22:45:49.172 INFO:tasks.rados.rados.0.smithi084.stdout:924: writing smithi08477203-1 from 564801 to 1211093 tid 1 2024-09-20T22:45:49.181 INFO:tasks.rados.rados.0.smithi084.stdout:924: writing smithi08477203-1 from 1921559 to 2663674 tid 2 2024-09-20T22:45:49.187 INFO:tasks.rados.rados.0.smithi084.stdout:924: writing smithi08477203-1 from 3202910 to 3764801 tid 3 2024-09-20T22:45:49.187 INFO:tasks.rados.rados.0.smithi084.stdout:925: snap_remove snap 86 2024-09-20T22:45:49.212 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:45:49.233 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:49.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:49.234 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:45:49.335 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:45:49.356 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:49.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:49.357 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:45:49.457 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:45:49.480 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:49.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:49.481 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:45:49.504 INFO:journalctl@ceph.iscsi.iscsi.a.smithi154.stdout:Sep 20 22:45:49 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-iscsi-iscsi-a[59730]: debug there is no tcmu-runner data available 2024-09-20T22:45:49.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:49 smithi154 ceph-mon[33264]: pgmap v304: 176 pgs: 4 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 167 active+clean; 213 MiB data, 733 MiB used, 536 GiB / 536 GiB avail 2024-09-20T22:45:49.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:49 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/3723415884' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:45:49.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:49 smithi154 ceph-mon[33264]: osdmap e270: 8 total, 8 up, 6 in 2024-09-20T22:45:49.582 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:45:49.583 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:49 smithi084 ceph-mon[40024]: pgmap v304: 176 pgs: 4 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 167 active+clean; 213 MiB data, 733 MiB used, 536 GiB / 536 GiB avail 2024-09-20T22:45:49.583 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:49 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/3723415884' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:45:49.583 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:49 smithi084 ceph-mon[40024]: osdmap e270: 8 total, 8 up, 6 in 2024-09-20T22:45:49.583 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:49 smithi084 ceph-mon[29774]: pgmap v304: 176 pgs: 4 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 167 active+clean; 213 MiB data, 733 MiB used, 536 GiB / 536 GiB avail 2024-09-20T22:45:49.583 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:49 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/3723415884' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:45:49.584 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:49 smithi084 ceph-mon[29774]: osdmap e270: 8 total, 8 up, 6 in 2024-09-20T22:45:49.603 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:49.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:49.604 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:45:49.705 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:45:49.726 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:49.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:49.727 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:45:49.828 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:45:49.850 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:49.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:49.851 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:45:49.939 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:45:49.952 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:45:49.974 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:49.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:49.975 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:45:50.076 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:45:50.097 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:50.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:50.098 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:45:50.167 INFO:tasks.rados.rados.0.smithi084.stdout:925: done (5 left) 2024-09-20T22:45:50.167 INFO:tasks.rados.rados.0.smithi084.stdout:926: rollback oid 18 current snap is 88 2024-09-20T22:45:50.167 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 18 to 85 2024-09-20T22:45:50.167 INFO:tasks.rados.rados.0.smithi084.stdout:917: finishing write tid 1 to smithi08477203-13 2024-09-20T22:45:50.167 INFO:tasks.rados.rados.0.smithi084.stdout:917: finishing write tid 2 to smithi08477203-13 2024-09-20T22:45:50.167 INFO:tasks.rados.rados.0.smithi084.stdout:917: finishing write tid 3 to smithi08477203-13 2024-09-20T22:45:50.167 INFO:tasks.rados.rados.0.smithi084.stdout:917: finishing write tid 4 to smithi08477203-13 2024-09-20T22:45:50.167 INFO:tasks.rados.rados.0.smithi084.stdout:917: finishing write tid 5 to smithi08477203-13 2024-09-20T22:45:50.167 INFO:tasks.rados.rados.0.smithi084.stdout:917: oid 13 updating version 0 to 461 2024-09-20T22:45:50.167 INFO:tasks.rados.rados.0.smithi084.stdout:917: oid 13 version 461 is already newer than 458 2024-09-20T22:45:50.167 INFO:tasks.rados.rados.0.smithi084.stdout:917: oid 13 version 461 is already newer than 460 2024-09-20T22:45:50.167 INFO:tasks.rados.rados.0.smithi084.stdout:917: oid 13 version 461 is already newer than 459 2024-09-20T22:45:50.167 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 13 v 461 (ObjNum 340 snap 88 seq_num 340) dirty exists 2024-09-20T22:45:50.168 INFO:tasks.rados.rados.0.smithi084.stdout:917: left oid 13 (ObjNum 340 snap 88 seq_num 340) 2024-09-20T22:45:50.168 INFO:tasks.rados.rados.0.smithi084.stdout:919: finishing write tid 1 to smithi08477203-10 2024-09-20T22:45:50.168 INFO:tasks.rados.rados.0.smithi084.stdout:919: finishing write tid 2 to smithi08477203-10 2024-09-20T22:45:50.168 INFO:tasks.rados.rados.0.smithi084.stdout:919: finishing write tid 3 to smithi08477203-10 2024-09-20T22:45:50.168 INFO:tasks.rados.rados.0.smithi084.stdout:919: finishing write tid 4 to smithi08477203-10 2024-09-20T22:45:50.168 INFO:tasks.rados.rados.0.smithi084.stdout:919: finishing write tid 5 to smithi08477203-10 2024-09-20T22:45:50.168 INFO:tasks.rados.rados.0.smithi084.stdout:919: oid 10 updating version 0 to 220 2024-09-20T22:45:50.168 INFO:tasks.rados.rados.0.smithi084.stdout:919: oid 10 updating version 220 to 221 2024-09-20T22:45:50.168 INFO:tasks.rados.rados.0.smithi084.stdout:919: oid 10 updating version 221 to 222 2024-09-20T22:45:50.168 INFO:tasks.rados.rados.0.smithi084.stdout:919: oid 10 updating version 222 to 223 2024-09-20T22:45:50.168 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 10 v 223 (ObjNum 341 snap 88 seq_num 341) dirty exists 2024-09-20T22:45:50.168 INFO:tasks.rados.rados.0.smithi084.stdout:919: left oid 10 (ObjNum 341 snap 88 seq_num 341) 2024-09-20T22:45:50.168 INFO:tasks.rados.rados.0.smithi084.stdout:920: finishing write tid 1 to smithi08477203-22 2024-09-20T22:45:50.168 INFO:tasks.rados.rados.0.smithi084.stdout:920: finishing write tid 2 to smithi08477203-22 2024-09-20T22:45:50.169 INFO:tasks.rados.rados.0.smithi084.stdout:920: finishing write tid 3 to smithi08477203-22 2024-09-20T22:45:50.169 INFO:tasks.rados.rados.0.smithi084.stdout:920: finishing write tid 4 to smithi08477203-22 2024-09-20T22:45:50.169 INFO:tasks.rados.rados.0.smithi084.stdout:920: finishing write tid 5 to smithi08477203-22 2024-09-20T22:45:50.169 INFO:tasks.rados.rados.0.smithi084.stdout:920: finishing write tid 6 to smithi08477203-22 2024-09-20T22:45:50.169 INFO:tasks.rados.rados.0.smithi084.stdout:920: oid 22 updating version 0 to 168 2024-09-20T22:45:50.169 INFO:tasks.rados.rados.0.smithi084.stdout:920: oid 22 updating version 168 to 169 2024-09-20T22:45:50.169 INFO:tasks.rados.rados.0.smithi084.stdout:920: oid 22 version 169 is already newer than 167 2024-09-20T22:45:50.169 INFO:tasks.rados.rados.0.smithi084.stdout:920: oid 22 updating version 169 to 170 2024-09-20T22:45:50.169 INFO:tasks.rados.rados.0.smithi084.stdout:920: oid 22 updating version 170 to 171 2024-09-20T22:45:50.169 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 22 v 171 (ObjNum 342 snap 88 seq_num 342) dirty exists 2024-09-20T22:45:50.169 INFO:tasks.rados.rados.0.smithi084.stdout:920: left oid 22 (ObjNum 342 snap 88 seq_num 342) 2024-09-20T22:45:50.169 INFO:tasks.rados.rados.0.smithi084.stdout:922: expect (ObjNum 312 snap 77 seq_num 312) 2024-09-20T22:45:50.199 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:45:50.220 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:50.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:50.220 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:45:50.276 INFO:tasks.rados.rados.0.smithi084.stdout:924: finishing write tid 1 to smithi08477203-1 2024-09-20T22:45:50.277 INFO:tasks.rados.rados.0.smithi084.stdout:924: finishing write tid 2 to smithi08477203-1 2024-09-20T22:45:50.277 INFO:tasks.rados.rados.0.smithi084.stdout:917: done (5 left) 2024-09-20T22:45:50.277 INFO:tasks.rados.rados.0.smithi084.stdout:919: done (4 left) 2024-09-20T22:45:50.277 INFO:tasks.rados.rados.0.smithi084.stdout:920: done (3 left) 2024-09-20T22:45:50.277 INFO:tasks.rados.rados.0.smithi084.stdout:922: done (2 left) 2024-09-20T22:45:50.277 INFO:tasks.rados.rados.0.smithi084.stdout:927: read oid 7 snap -1 2024-09-20T22:45:50.277 INFO:tasks.rados.rados.0.smithi084.stdout:927: expect (ObjNum 260 snap 61 seq_num 260) 2024-09-20T22:45:50.277 INFO:tasks.rados.rados.0.smithi084.stdout:928: write oid 11 current snap is 88 2024-09-20T22:45:50.277 INFO:tasks.rados.rados.0.smithi084.stdout:924: finishing write tid 3 to smithi08477203-1 2024-09-20T22:45:50.277 INFO:tasks.rados.rados.0.smithi084.stdout:924: finishing write tid 4 to smithi08477203-1 2024-09-20T22:45:50.277 INFO:tasks.rados.rados.0.smithi084.stdout:928: seq_num 344 ranges {581608=721541,1863688=485773,3113980=604043} 2024-09-20T22:45:50.295 INFO:tasks.rados.rados.0.smithi084.stdout:928: writing smithi08477203-11 from 581608 to 1303149 tid 1 2024-09-20T22:45:50.295 INFO:tasks.rados.rados.0.smithi084.stdout:928: writing smithi08477203-11 from 1863688 to 2349461 tid 2 2024-09-20T22:45:50.299 INFO:tasks.rados.rados.0.smithi084.stdout:928: writing smithi08477203-11 from 3113980 to 3718023 tid 3 2024-09-20T22:45:50.299 INFO:tasks.rados.rados.0.smithi084.stdout:929: snap_remove snap 84 2024-09-20T22:45:50.321 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:45:50.343 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:50.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:50.344 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:45:50.445 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:45:50.466 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:50.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:50.467 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:45:50.479 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:50 smithi084 ceph-mon[40024]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:45:50.479 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:50 smithi084 ceph-mon[40024]: osdmap e271: 8 total, 8 up, 6 in 2024-09-20T22:45:50.479 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:50 smithi084 ceph-mon[29774]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:45:50.479 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:50 smithi084 ceph-mon[29774]: osdmap e271: 8 total, 8 up, 6 in 2024-09-20T22:45:50.494 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-20T22:45:50.494 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:45:50.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:50 smithi154 ceph-mon[33264]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:45:50.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:50 smithi154 ceph-mon[33264]: osdmap e271: 8 total, 8 up, 6 in 2024-09-20T22:45:50.567 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:45:50.589 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:50.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:50.589 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:45:50.690 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:45:50.712 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:50.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:50.713 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:45:50.814 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:45:50.835 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:50.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:50.836 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:45:50.937 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:45:50.959 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:50.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:50.960 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:45:51.060 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:45:51.087 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:51.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:51.087 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:45:51.173 INFO:tasks.rados.rados.0.smithi084.stdout:929: done (4 left) 2024-09-20T22:45:51.173 INFO:tasks.rados.rados.0.smithi084.stdout:930: copy_from oid 6 from oid 27 current snap is 88 2024-09-20T22:45:51.173 INFO:tasks.rados.rados.0.smithi084.stdout:924: finishing write tid 5 to smithi08477203-1 2024-09-20T22:45:51.173 INFO:tasks.rados.rados.0.smithi084.stdout:924: oid 1 updating version 0 to 230 2024-09-20T22:45:51.173 INFO:tasks.rados.rados.0.smithi084.stdout:924: oid 1 updating version 230 to 231 2024-09-20T22:45:51.173 INFO:tasks.rados.rados.0.smithi084.stdout:924: oid 1 version 231 is already newer than 229 2024-09-20T22:45:51.174 INFO:tasks.rados.rados.0.smithi084.stdout:924: oid 1 updating version 231 to 232 2024-09-20T22:45:51.174 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 1 v 232 (ObjNum 343 snap 88 seq_num 343) dirty exists 2024-09-20T22:45:51.174 INFO:tasks.rados.rados.0.smithi084.stdout:924: left oid 1 (ObjNum 343 snap 88 seq_num 343) 2024-09-20T22:45:51.174 INFO:tasks.rados.rados.0.smithi084.stdout:926: finishing rollback tid 0 to smithi08477203-18 2024-09-20T22:45:51.174 INFO:tasks.rados.rados.0.smithi084.stdout:926: finishing rollback tid 1 to smithi08477203-18 2024-09-20T22:45:51.174 INFO:tasks.rados.rados.0.smithi084.stdout:926: finishing rollback tid 2 to smithi08477203-18 2024-09-20T22:45:51.174 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 18 v 236 (ObjNum 273 snap 69 seq_num 273) dirty exists 2024-09-20T22:45:51.174 INFO:tasks.rados.rados.0.smithi084.stdout:927: expect (ObjNum 260 snap 61 seq_num 260) 2024-09-20T22:45:51.188 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:45:51.211 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:51.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:51.212 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:45:51.213 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.0 dump_blocked_ops 2024-09-20T22:45:51.224 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd unset noscrub 2024-09-20T22:45:51.269 INFO:tasks.rados.rados.0.smithi084.stdout:928: finishing write tid 1 to smithi08477203-11 2024-09-20T22:45:51.270 INFO:tasks.rados.rados.0.smithi084.stdout:928: finishing write tid 2 to smithi08477203-11 2024-09-20T22:45:51.270 INFO:tasks.rados.rados.0.smithi084.stdout:928: finishing write tid 3 to smithi08477203-11 2024-09-20T22:45:51.270 INFO:tasks.rados.rados.0.smithi084.stdout:924: done (4 left) 2024-09-20T22:45:51.270 INFO:tasks.rados.rados.0.smithi084.stdout:926: done (3 left) 2024-09-20T22:45:51.270 INFO:tasks.rados.rados.0.smithi084.stdout:927: done (2 left) 2024-09-20T22:45:51.270 INFO:tasks.rados.rados.0.smithi084.stdout:931: write oid 22 current snap is 88 2024-09-20T22:45:51.270 INFO:tasks.rados.rados.0.smithi084.stdout:928: finishing write tid 4 to smithi08477203-11 2024-09-20T22:45:51.270 INFO:tasks.rados.rados.0.smithi084.stdout:928: finishing write tid 5 to smithi08477203-11 2024-09-20T22:45:51.270 INFO:tasks.rados.rados.0.smithi084.stdout:928: oid 11 updating version 0 to 219 2024-09-20T22:45:51.270 INFO:tasks.rados.rados.0.smithi084.stdout:928: oid 11 version 219 is already newer than 217 2024-09-20T22:45:51.270 INFO:tasks.rados.rados.0.smithi084.stdout:928: oid 11 version 219 is already newer than 216 2024-09-20T22:45:51.271 INFO:tasks.rados.rados.0.smithi084.stdout:928: oid 11 version 219 is already newer than 218 2024-09-20T22:45:51.271 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 11 v 219 (ObjNum 344 snap 88 seq_num 344) dirty exists 2024-09-20T22:45:51.271 INFO:tasks.rados.rados.0.smithi084.stdout:928: left oid 11 (ObjNum 344 snap 88 seq_num 344) 2024-09-20T22:45:51.271 INFO:tasks.rados.rados.0.smithi084.stdout:931: seq_num 345 ranges {598415=796790,1922170=745784,3141403=415315} 2024-09-20T22:45:51.277 INFO:tasks.rados.rados.0.smithi084.stdout:931: writing smithi08477203-22 from 598415 to 1395205 tid 1 2024-09-20T22:45:51.284 INFO:tasks.rados.rados.0.smithi084.stdout:931: writing smithi08477203-22 from 1922170 to 2667954 tid 2 2024-09-20T22:45:51.289 INFO:tasks.rados.rados.0.smithi084.stdout:931: writing smithi08477203-22 from 3141403 to 3556718 tid 3 2024-09-20T22:45:51.289 INFO:tasks.rados.rados.0.smithi084.stdout:928: done (2 left) 2024-09-20T22:45:51.289 INFO:tasks.rados.rados.0.smithi084.stdout:932: delete oid 15 current snap is 88 2024-09-20T22:45:51.289 INFO:tasks.rados.rados.0.smithi084.stdout:930: finishing copy_from to smithi08477203-6 2024-09-20T22:45:51.289 INFO:tasks.rados.rados.0.smithi084.stdout:930: got expected ENOENT (src dne) 2024-09-20T22:45:51.289 INFO:tasks.rados.rados.0.smithi084.stdout:930: finishing copy_from racing read to smithi08477203-6 2024-09-20T22:45:51.290 INFO:tasks.rados.rados.0.smithi084.stdout:931: finishing write tid 1 to smithi08477203-22 2024-09-20T22:45:51.290 INFO:tasks.rados.rados.0.smithi084.stdout:931: finishing write tid 2 to smithi08477203-22 2024-09-20T22:45:51.292 INFO:tasks.rados.rados.0.smithi084.stdout:930: done (2 left) 2024-09-20T22:45:51.292 INFO:tasks.rados.rados.0.smithi084.stdout:932: done (1 left) 2024-09-20T22:45:51.292 INFO:tasks.rados.rados.0.smithi084.stdout:933: read oid 48 snap 69 2024-09-20T22:45:51.292 INFO:tasks.rados.rados.0.smithi084.stdout:933: expect (ObjNum 257 snap 60 seq_num 257) 2024-09-20T22:45:51.292 INFO:tasks.rados.rados.0.smithi084.stdout:934: read oid 28 snap -1 2024-09-20T22:45:51.292 INFO:tasks.rados.rados.0.smithi084.stdout:934: expect (ObjNum 311 snap 76 seq_num 311) 2024-09-20T22:45:51.292 INFO:tasks.rados.rados.0.smithi084.stdout:931: finishing write tid 3 to smithi08477203-22 2024-09-20T22:45:51.292 INFO:tasks.rados.rados.0.smithi084.stdout:935: read oid 26 snap -1 2024-09-20T22:45:51.292 INFO:tasks.rados.rados.0.smithi084.stdout:935: expect (ObjNum 326 snap 84 seq_num 326) 2024-09-20T22:45:51.292 INFO:tasks.rados.rados.0.smithi084.stdout:936: snap_create 2024-09-20T22:45:51.293 INFO:tasks.rados.rados.0.smithi084.stdout:931: finishing write tid 4 to smithi08477203-22 2024-09-20T22:45:51.293 INFO:tasks.rados.rados.0.smithi084.stdout:931: finishing write tid 5 to smithi08477203-22 2024-09-20T22:45:51.293 INFO:tasks.rados.rados.0.smithi084.stdout:931: oid 22 updating version 0 to 175 2024-09-20T22:45:51.293 INFO:tasks.rados.rados.0.smithi084.stdout:931: oid 22 version 175 is already newer than 173 2024-09-20T22:45:51.293 INFO:tasks.rados.rados.0.smithi084.stdout:931: oid 22 version 175 is already newer than 172 2024-09-20T22:45:51.293 INFO:tasks.rados.rados.0.smithi084.stdout:931: oid 22 version 175 is already newer than 174 2024-09-20T22:45:51.293 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 22 v 175 (ObjNum 345 snap 88 seq_num 345) dirty exists 2024-09-20T22:45:51.293 INFO:tasks.rados.rados.0.smithi084.stdout:931: left oid 22 (ObjNum 345 snap 88 seq_num 345) 2024-09-20T22:45:51.294 INFO:tasks.rados.rados.0.smithi084.stdout:933: expect (ObjNum 257 snap 60 seq_num 257) 2024-09-20T22:45:51.313 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:45:51.335 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:51.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:51.336 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:45:51.381 INFO:tasks.rados.rados.0.smithi084.stdout:934: expect (ObjNum 311 snap 76 seq_num 311) 2024-09-20T22:45:51.437 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:45:51.459 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:51.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:51.460 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:45:51.495 INFO:tasks.rados.rados.0.smithi084.stdout:935: expect (ObjNum 326 snap 84 seq_num 326) 2024-09-20T22:45:51.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:51 smithi154 ceph-mon[33264]: pgmap v307: 176 pgs: 2 peering, 5 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 164 active+clean; 224 MiB data, 740 MiB used, 536 GiB / 536 GiB avail; 10 MiB/s rd, 1.7 MiB/s wr, 16 op/s 2024-09-20T22:45:51.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:51 smithi154 ceph-mon[33264]: osdmap e272: 8 total, 8 up, 6 in 2024-09-20T22:45:51.526 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:45:51.538 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:45:51.561 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:45:51.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:51.586 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:51.586 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:45:51.614 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:51 smithi084 ceph-mon[40024]: pgmap v307: 176 pgs: 2 peering, 5 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 164 active+clean; 224 MiB data, 740 MiB used, 536 GiB / 536 GiB avail; 10 MiB/s rd, 1.7 MiB/s wr, 16 op/s 2024-09-20T22:45:51.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:51 smithi084 ceph-mon[40024]: osdmap e272: 8 total, 8 up, 6 in 2024-09-20T22:45:51.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:51 smithi084 ceph-mon[29774]: pgmap v307: 176 pgs: 2 peering, 5 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 164 active+clean; 224 MiB data, 740 MiB used, 536 GiB / 536 GiB avail; 10 MiB/s rd, 1.7 MiB/s wr, 16 op/s 2024-09-20T22:45:51.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:51 smithi084 ceph-mon[29774]: osdmap e272: 8 total, 8 up, 6 in 2024-09-20T22:45:51.687 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:45:51.708 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:51.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:51.709 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:45:51.810 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:45:51.833 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:51.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:51.834 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:45:51.894 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-20T22:45:51.935 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:45:51.957 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:51.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:51.958 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:45:52.059 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:45:52.064 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 5, 6, 7] out_osds: [4, 0] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-20T22:45:52.065 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-20T22:45:52.065 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-09-20T22:45:52.065 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph tell osd.1 'trim stale osdmaps' 2024-09-20T22:45:52.081 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:52.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:52.082 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:45:52.183 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:45:52.209 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:52.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:52.211 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:45:52.213 INFO:tasks.rados.rados.0.smithi084.stdout:931: done (4 left) 2024-09-20T22:45:52.213 INFO:tasks.rados.rados.0.smithi084.stdout:933: done (3 left) 2024-09-20T22:45:52.213 INFO:tasks.rados.rados.0.smithi084.stdout:934: done (2 left) 2024-09-20T22:45:52.213 INFO:tasks.rados.rados.0.smithi084.stdout:935: done (1 left) 2024-09-20T22:45:52.213 INFO:tasks.rados.rados.0.smithi084.stdout:936: done (0 left) 2024-09-20T22:45:52.213 INFO:tasks.rados.rados.0.smithi084.stdout:937: write (excl) oid 16 current snap is 89 2024-09-20T22:45:52.213 INFO:tasks.rados.rados.0.smithi084.stdout:937: seq_num 347 ranges {632029=547288,1639134=749453,2796249=437859,3653524=178505} 2024-09-20T22:45:52.220 INFO:tasks.rados.rados.0.smithi084.stdout:937: writing smithi08477203-16 from 632029 to 1179317 tid 1 2024-09-20T22:45:52.228 INFO:tasks.rados.rados.0.smithi084.stdout:937: writing smithi08477203-16 from 1639134 to 2388587 tid 2 2024-09-20T22:45:52.234 INFO:tasks.rados.rados.0.smithi084.stdout:937: writing smithi08477203-16 from 2796249 to 3234108 tid 3 2024-09-20T22:45:52.238 INFO:tasks.rados.rados.0.smithi084.stdout:937: writing smithi08477203-16 from 3653524 to 3832029 tid 4 2024-09-20T22:45:52.238 INFO:tasks.rados.rados.0.smithi084.stdout:938: write (excl) oid 25 current snap is 89 2024-09-20T22:45:52.238 INFO:tasks.rados.rados.0.smithi084.stdout:937: finishing write tid 1 to smithi08477203-16 2024-09-20T22:45:52.238 INFO:tasks.rados.rados.0.smithi084.stdout:937: finishing write tid 2 to smithi08477203-16 2024-09-20T22:45:52.238 INFO:tasks.rados.rados.0.smithi084.stdout:937: finishing write tid 3 to smithi08477203-16 2024-09-20T22:45:52.238 INFO:tasks.rados.rados.0.smithi084.stdout:938: seq_num 348 ranges {648836=622537,1981263=493111,3107319=741517} 2024-09-20T22:45:52.246 INFO:tasks.rados.rados.0.smithi084.stdout:938: writing smithi08477203-25 from 648836 to 1271373 tid 1 2024-09-20T22:45:52.253 INFO:tasks.rados.rados.0.smithi084.stdout:938: writing smithi08477203-25 from 1981263 to 2474374 tid 2 2024-09-20T22:45:52.261 INFO:tasks.rados.rados.0.smithi084.stdout:938: writing smithi08477203-25 from 3107319 to 3848836 tid 3 2024-09-20T22:45:52.261 INFO:tasks.rados.rados.0.smithi084.stdout:939: delete oid 29 current snap is 89 2024-09-20T22:45:52.261 INFO:tasks.rados.rados.0.smithi084.stdout:937: finishing write tid 4 to smithi08477203-16 2024-09-20T22:45:52.261 INFO:tasks.rados.rados.0.smithi084.stdout:937: finishing write tid 5 to smithi08477203-16 2024-09-20T22:45:52.262 INFO:tasks.rados.rados.0.smithi084.stdout:937: finishing write tid 6 to smithi08477203-16 2024-09-20T22:45:52.262 INFO:tasks.rados.rados.0.smithi084.stdout:937: oid 16 updating version 0 to 174 2024-09-20T22:45:52.262 INFO:tasks.rados.rados.0.smithi084.stdout:937: oid 16 version 174 is already newer than 171 2024-09-20T22:45:52.262 INFO:tasks.rados.rados.0.smithi084.stdout:937: oid 16 version 174 is already newer than 172 2024-09-20T22:45:52.262 INFO:tasks.rados.rados.0.smithi084.stdout:937: oid 16 version 174 is already newer than 173 2024-09-20T22:45:52.262 INFO:tasks.rados.rados.0.smithi084.stdout:937: oid 16 version 174 is already newer than 170 2024-09-20T22:45:52.262 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 16 v 174 (ObjNum 347 snap 89 seq_num 347) dirty exists 2024-09-20T22:45:52.262 INFO:tasks.rados.rados.0.smithi084.stdout:937: left oid 16 (ObjNum 347 snap 89 seq_num 347) 2024-09-20T22:45:52.262 INFO:tasks.rados.rados.0.smithi084.stdout:938: finishing write tid 1 to smithi08477203-25 2024-09-20T22:45:52.262 INFO:tasks.rados.rados.0.smithi084.stdout:938: finishing write tid 2 to smithi08477203-25 2024-09-20T22:45:52.266 INFO:tasks.rados.rados.0.smithi084.stdout:938: finishing write tid 3 to smithi08477203-25 2024-09-20T22:45:52.266 INFO:tasks.rados.rados.0.smithi084.stdout:938: finishing write tid 4 to smithi08477203-25 2024-09-20T22:45:52.266 INFO:tasks.rados.rados.0.smithi084.stdout:938: finishing write tid 5 to smithi08477203-25 2024-09-20T22:45:52.266 INFO:tasks.rados.rados.0.smithi084.stdout:938: oid 25 updating version 0 to 485 2024-09-20T22:45:52.266 INFO:tasks.rados.rados.0.smithi084.stdout:938: oid 25 updating version 485 to 488 2024-09-20T22:45:52.266 INFO:tasks.rados.rados.0.smithi084.stdout:938: oid 25 version 488 is already newer than 486 2024-09-20T22:45:52.266 INFO:tasks.rados.rados.0.smithi084.stdout:938: oid 25 version 488 is already newer than 487 2024-09-20T22:45:52.266 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 25 v 488 (ObjNum 348 snap 89 seq_num 348) dirty exists 2024-09-20T22:45:52.266 INFO:tasks.rados.rados.0.smithi084.stdout:938: left oid 25 (ObjNum 348 snap 89 seq_num 348) 2024-09-20T22:45:52.267 INFO:tasks.rados.rados.0.smithi084.stdout:937: done (2 left) 2024-09-20T22:45:52.267 INFO:tasks.rados.rados.0.smithi084.stdout:938: done (1 left) 2024-09-20T22:45:52.267 INFO:tasks.rados.rados.0.smithi084.stdout:939: done (0 left) 2024-09-20T22:45:52.267 INFO:tasks.rados.rados.0.smithi084.stdout:940: snap_create 2024-09-20T22:45:52.279 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-20T22:45:52.279 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:45:52.311 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:45:52.333 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:52.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:52.334 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:45:52.435 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:45:52.460 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:52.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:52.461 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:45:52.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:52 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "17"}]: dispatch 2024-09-20T22:45:52.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:52 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "17"}]: dispatch 2024-09-20T22:45:52.561 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:45:52.583 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:52.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:52.584 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:45:52.614 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:52 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "17"}]: dispatch 2024-09-20T22:45:52.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:52 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "17"}]: dispatch 2024-09-20T22:45:52.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:52 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "17"}]: dispatch 2024-09-20T22:45:52.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:52 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "17"}]: dispatch 2024-09-20T22:45:52.684 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:45:52.710 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:52.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:52.711 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:45:52.812 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:45:52.835 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:52.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:52.836 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:45:52.937 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:45:52.958 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:52.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:52.959 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:45:53.059 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:45:53.085 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:53.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:53.086 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:45:53.151 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:45:53.156 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is unset 2024-09-20T22:45:53.165 INFO:tasks.rados.rados.0.smithi084.stdout:940: done (0 left) 2024-09-20T22:45:53.165 INFO:tasks.rados.rados.0.smithi084.stdout:941: read oid 2 snap -1 2024-09-20T22:45:53.165 INFO:tasks.rados.rados.0.smithi084.stdout:941: expect (ObjNum 300 snap 75 seq_num 300) 2024-09-20T22:45:53.165 INFO:tasks.rados.rados.0.smithi084.stdout:942: delete oid 20 current snap is 90 2024-09-20T22:45:53.173 INFO:tasks.rados.rados.0.smithi084.stdout:942: done (1 left) 2024-09-20T22:45:53.173 INFO:tasks.rados.rados.0.smithi084.stdout:943: copy_from oid 38 from oid 19 current snap is 90 2024-09-20T22:45:53.173 INFO:tasks.rados.rados.0.smithi084.stdout:944: read oid 20 snap -1 2024-09-20T22:45:53.173 INFO:tasks.rados.rados.0.smithi084.stdout:944: expect deleted 2024-09-20T22:45:53.174 INFO:tasks.rados.rados.0.smithi084.stdout:945: read oid 6 snap 89 2024-09-20T22:45:53.174 INFO:tasks.rados.rados.0.smithi084.stdout:945: expect (ObjNum 309 snap 75 seq_num 309) 2024-09-20T22:45:53.174 INFO:tasks.rados.rados.0.smithi084.stdout:946: delete oid 4 current snap is 90 2024-09-20T22:45:53.176 INFO:tasks.rados.rados.0.smithi084.stdout:941: expect (ObjNum 300 snap 75 seq_num 300) 2024-09-20T22:45:53.187 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:45:53.197 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.0 dump_historic_ops 2024-09-20T22:45:53.210 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:53.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:53.211 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:45:53.288 INFO:tasks.rados.rados.0.smithi084.stdout:945: expect (ObjNum 309 snap 75 seq_num 309) 2024-09-20T22:45:53.312 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:45:53.335 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:53.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:53.336 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:45:53.436 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:45:53.458 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:53.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:53.459 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:45:53.494 INFO:tasks.rados.rados.0.smithi084.stdout:943: finishing copy_from racing read to smithi08477203-38 2024-09-20T22:45:53.494 INFO:tasks.rados.rados.0.smithi084.stdout:943: finishing copy_from to smithi08477203-38 2024-09-20T22:45:53.494 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 38 v 376 (ObjNum 258 snap 60 seq_num 258) dirty exists 2024-09-20T22:45:53.494 INFO:tasks.rados.rados.0.smithi084.stdout:941: done (4 left) 2024-09-20T22:45:53.494 INFO:tasks.rados.rados.0.smithi084.stdout:943: done (3 left) 2024-09-20T22:45:53.495 INFO:tasks.rados.rados.0.smithi084.stdout:944: done (2 left) 2024-09-20T22:45:53.495 INFO:tasks.rados.rados.0.smithi084.stdout:945: done (1 left) 2024-09-20T22:45:53.495 INFO:tasks.rados.rados.0.smithi084.stdout:946: done (0 left) 2024-09-20T22:45:53.495 INFO:tasks.rados.rados.0.smithi084.stdout:947: read oid 19 snap -1 2024-09-20T22:45:53.495 INFO:tasks.rados.rados.0.smithi084.stdout:947: expect (ObjNum 258 snap 60 seq_num 258) 2024-09-20T22:45:53.495 INFO:tasks.rados.rados.0.smithi084.stdout:948: snap_remove snap 89 2024-09-20T22:45:53.498 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:53 smithi084 ceph-mon[40024]: pgmap v310: 175 pgs: 1 peering, 5 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 164 active+clean; 224 MiB data, 740 MiB used, 536 GiB / 536 GiB avail; 10 MiB/s rd, 1.7 MiB/s wr, 16 op/s 2024-09-20T22:45:53.498 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:53 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "17"}]': finished 2024-09-20T22:45:53.498 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:53 smithi084 ceph-mon[40024]: osdmap e273: 8 total, 8 up, 6 in 2024-09-20T22:45:53.498 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:53 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:45:53.498 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:53 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:45:53.498 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:53 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/1276807170' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:45:53.498 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:53 smithi084 ceph-mon[40024]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:45:53.498 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:53 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/1276807170' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T22:45:53.498 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:53 smithi084 ceph-mon[40024]: osdmap e274: 8 total, 8 up, 6 in 2024-09-20T22:45:53.499 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:53 smithi084 ceph-mon[29774]: pgmap v310: 175 pgs: 1 peering, 5 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 164 active+clean; 224 MiB data, 740 MiB used, 536 GiB / 536 GiB avail; 10 MiB/s rd, 1.7 MiB/s wr, 16 op/s 2024-09-20T22:45:53.499 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:53 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "17"}]': finished 2024-09-20T22:45:53.499 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:53 smithi084 ceph-mon[29774]: osdmap e273: 8 total, 8 up, 6 in 2024-09-20T22:45:53.499 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:53 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:45:53.499 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:53 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:45:53.499 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:53 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/1276807170' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:45:53.499 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:53 smithi084 ceph-mon[29774]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:45:53.499 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:53 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/1276807170' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T22:45:53.499 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:53 smithi084 ceph-mon[29774]: osdmap e274: 8 total, 8 up, 6 in 2024-09-20T22:45:53.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:53 smithi154 ceph-mon[33264]: pgmap v310: 175 pgs: 1 peering, 5 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 164 active+clean; 224 MiB data, 740 MiB used, 536 GiB / 536 GiB avail; 10 MiB/s rd, 1.7 MiB/s wr, 16 op/s 2024-09-20T22:45:53.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:53 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "17"}]': finished 2024-09-20T22:45:53.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:53 smithi154 ceph-mon[33264]: osdmap e273: 8 total, 8 up, 6 in 2024-09-20T22:45:53.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:53 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:45:53.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:53 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:45:53.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:53 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/1276807170' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:45:53.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:53 smithi154 ceph-mon[33264]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:45:53.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:53 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/1276807170' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T22:45:53.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:53 smithi154 ceph-mon[33264]: osdmap e274: 8 total, 8 up, 6 in 2024-09-20T22:45:53.560 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:45:53.581 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:53.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:53.582 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:45:53.683 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:45:53.704 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:53.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:53.705 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:45:53.806 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:45:53.826 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:53.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:53.827 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:45:53.928 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:45:53.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:53.950 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:53.950 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:45:54.051 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:45:54.073 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:54.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:54.074 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:45:54.162 INFO:tasks.rados.rados.0.smithi084.stdout:948: done (1 left) 2024-09-20T22:45:54.162 INFO:tasks.rados.rados.0.smithi084.stdout:949: write oid 9 current snap is 90 2024-09-20T22:45:54.162 INFO:tasks.rados.rados.0.smithi084.stdout:949: seq_num 352 ranges {716064=639886,1815191=500449,2817011=526925} 2024-09-20T22:45:54.170 INFO:tasks.rados.rados.0.smithi084.stdout:949: writing smithi08477203-9 from 716064 to 1355950 tid 1 2024-09-20T22:45:54.174 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:45:54.176 INFO:tasks.rados.rados.0.smithi084.stdout:949: writing smithi08477203-9 from 1815191 to 2315640 tid 2 2024-09-20T22:45:54.181 INFO:tasks.rados.rados.0.smithi084.stdout:949: writing smithi08477203-9 from 2817011 to 3343936 tid 3 2024-09-20T22:45:54.182 INFO:tasks.rados.rados.0.smithi084.stdout:947: expect (ObjNum 258 snap 60 seq_num 258) 2024-09-20T22:45:54.196 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:54.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:54.197 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:45:54.297 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:45:54.318 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:54.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:54.319 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:45:54.326 INFO:tasks.rados.rados.0.smithi084.stdout:949: finishing write tid 1 to smithi08477203-9 2024-09-20T22:45:54.326 INFO:tasks.rados.rados.0.smithi084.stdout:949: finishing write tid 2 to smithi08477203-9 2024-09-20T22:45:54.326 INFO:tasks.rados.rados.0.smithi084.stdout:949: finishing write tid 3 to smithi08477203-9 2024-09-20T22:45:54.326 INFO:tasks.rados.rados.0.smithi084.stdout:949: finishing write tid 4 to smithi08477203-9 2024-09-20T22:45:54.326 INFO:tasks.rados.rados.0.smithi084.stdout:950: read oid 39 snap -1 2024-09-20T22:45:54.326 INFO:tasks.rados.rados.0.smithi084.stdout:950: expect (ObjNum 245 snap 54 seq_num 245) 2024-09-20T22:45:54.327 INFO:tasks.rados.rados.0.smithi084.stdout:947: done (2 left) 2024-09-20T22:45:54.327 INFO:tasks.rados.rados.0.smithi084.stdout:951: rollback oid 45 current snap is 90 2024-09-20T22:45:54.327 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 45 to 88 2024-09-20T22:45:54.327 INFO:tasks.rados.rados.0.smithi084.stdout:949: finishing write tid 5 to smithi08477203-9 2024-09-20T22:45:54.327 INFO:tasks.rados.rados.0.smithi084.stdout:949: oid 9 updating version 0 to 245 2024-09-20T22:45:54.327 INFO:tasks.rados.rados.0.smithi084.stdout:949: oid 9 updating version 245 to 246 2024-09-20T22:45:54.327 INFO:tasks.rados.rados.0.smithi084.stdout:949: oid 9 updating version 246 to 247 2024-09-20T22:45:54.327 INFO:tasks.rados.rados.0.smithi084.stdout:949: oid 9 updating version 247 to 248 2024-09-20T22:45:54.327 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 9 v 248 (ObjNum 352 snap 90 seq_num 352) dirty exists 2024-09-20T22:45:54.327 INFO:tasks.rados.rados.0.smithi084.stdout:949: left oid 9 (ObjNum 352 snap 90 seq_num 352) 2024-09-20T22:45:54.327 INFO:tasks.rados.rados.0.smithi084.stdout:949: done (2 left) 2024-09-20T22:45:54.327 INFO:tasks.rados.rados.0.smithi084.stdout:952: read oid 2 snap -1 2024-09-20T22:45:54.328 INFO:tasks.rados.rados.0.smithi084.stdout:952: expect (ObjNum 300 snap 75 seq_num 300) 2024-09-20T22:45:54.328 INFO:tasks.rados.rados.0.smithi084.stdout:953: write (excl) oid 14 current snap is 90 2024-09-20T22:45:54.328 INFO:tasks.rados.rados.0.smithi084.stdout:953: seq_num 353 ranges {732871=715135,1873673=644107,3244434=688437} 2024-09-20T22:45:54.334 INFO:tasks.rados.rados.0.smithi084.stdout:953: writing smithi08477203-14 from 732871 to 1448006 tid 1 2024-09-20T22:45:54.340 INFO:tasks.rados.rados.0.smithi084.stdout:953: writing smithi08477203-14 from 1873673 to 2517780 tid 2 2024-09-20T22:45:54.348 INFO:tasks.rados.rados.0.smithi084.stdout:953: writing smithi08477203-14 from 3244434 to 3932871 tid 3 2024-09-20T22:45:54.348 INFO:tasks.rados.rados.0.smithi084.stdout:954: delete oid 33 current snap is 90 2024-09-20T22:45:54.348 INFO:tasks.rados.rados.0.smithi084.stdout:950: expect (ObjNum 245 snap 54 seq_num 245) 2024-09-20T22:45:54.400 INFO:teuthology.orchestra.run.smithi084.stderr:no valid command found; 10 closest matches: 2024-09-20T22:45:54.401 INFO:teuthology.orchestra.run.smithi084.stderr:0 2024-09-20T22:45:54.401 INFO:teuthology.orchestra.run.smithi084.stderr:1 2024-09-20T22:45:54.401 INFO:teuthology.orchestra.run.smithi084.stderr:2 2024-09-20T22:45:54.401 INFO:teuthology.orchestra.run.smithi084.stderr:abort 2024-09-20T22:45:54.401 INFO:teuthology.orchestra.run.smithi084.stderr:assert 2024-09-20T22:45:54.401 INFO:teuthology.orchestra.run.smithi084.stderr:bench [] [] [] [] 2024-09-20T22:45:54.401 INFO:teuthology.orchestra.run.smithi084.stderr:bluefs debug_inject_read_zeros 2024-09-20T22:45:54.401 INFO:teuthology.orchestra.run.smithi084.stderr:bluefs files list 2024-09-20T22:45:54.401 INFO:teuthology.orchestra.run.smithi084.stderr:bluefs stats 2024-09-20T22:45:54.401 INFO:teuthology.orchestra.run.smithi084.stderr:bluestore allocator dump block 2024-09-20T22:45:54.401 INFO:teuthology.orchestra.run.smithi084.stderr:Error EINVAL: invalid command 2024-09-20T22:45:54.402 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:54 smithi084 ceph-mon[40024]: 7.e scrub starts 2024-09-20T22:45:54.402 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:54 smithi084 ceph-mon[40024]: 7.e scrub ok 2024-09-20T22:45:54.402 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:54 smithi084 ceph-mon[40024]: osdmap e275: 8 total, 8 up, 6 in 2024-09-20T22:45:54.402 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:54 smithi084 ceph-mon[29774]: 7.e scrub starts 2024-09-20T22:45:54.402 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:54 smithi084 ceph-mon[29774]: 7.e scrub ok 2024-09-20T22:45:54.402 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:54 smithi084 ceph-mon[29774]: osdmap e275: 8 total, 8 up, 6 in 2024-09-20T22:45:54.406 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:45:54.420 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:45:54.445 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:54.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:54.446 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:45:54.473 INFO:tasks.rados.rados.0.smithi084.stdout:951: finishing rollback tid 0 to smithi08477203-45 2024-09-20T22:45:54.474 INFO:tasks.rados.rados.0.smithi084.stdout:951: finishing rollback tid 1 to smithi08477203-45 2024-09-20T22:45:54.474 INFO:tasks.rados.rados.0.smithi084.stdout:951: finishing rollback tid 2 to smithi08477203-45 2024-09-20T22:45:54.474 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 45 v 114 (ObjNum 338 snap 85 seq_num 338) dirty exists 2024-09-20T22:45:54.474 INFO:tasks.rados.rados.0.smithi084.stdout:952: expect (ObjNum 300 snap 75 seq_num 300) 2024-09-20T22:45:54.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:54 smithi154 ceph-mon[33264]: 7.e scrub starts 2024-09-20T22:45:54.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:54 smithi154 ceph-mon[33264]: 7.e scrub ok 2024-09-20T22:45:54.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:54 smithi154 ceph-mon[33264]: osdmap e275: 8 total, 8 up, 6 in 2024-09-20T22:45:54.546 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:45:54.571 INFO:tasks.rados.rados.0.smithi084.stdout:953: finishing write tid 1 to smithi08477203-14 2024-09-20T22:45:54.571 INFO:tasks.rados.rados.0.smithi084.stdout:953: finishing write tid 2 to smithi08477203-14 2024-09-20T22:45:54.571 INFO:tasks.rados.rados.0.smithi084.stdout:953: finishing write tid 3 to smithi08477203-14 2024-09-20T22:45:54.571 INFO:tasks.rados.rados.0.smithi084.stdout:953: finishing write tid 4 to smithi08477203-14 2024-09-20T22:45:54.571 INFO:tasks.rados.rados.0.smithi084.stdout:953: finishing write tid 5 to smithi08477203-14 2024-09-20T22:45:54.571 INFO:tasks.rados.rados.0.smithi084.stdout:953: oid 14 updating version 0 to 83 2024-09-20T22:45:54.571 INFO:tasks.rados.rados.0.smithi084.stdout:953: oid 14 updating version 83 to 84 2024-09-20T22:45:54.571 INFO:tasks.rados.rados.0.smithi084.stdout:953: oid 14 version 84 is already newer than 82 2024-09-20T22:45:54.571 INFO:tasks.rados.rados.0.smithi084.stdout:953: oid 14 updating version 84 to 85 2024-09-20T22:45:54.571 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 14 v 85 (ObjNum 353 snap 90 seq_num 353) dirty exists 2024-09-20T22:45:54.571 INFO:tasks.rados.rados.0.smithi084.stdout:953: left oid 14 (ObjNum 353 snap 90 seq_num 353) 2024-09-20T22:45:54.571 INFO:tasks.rados.rados.0.smithi084.stdout:950: done (4 left) 2024-09-20T22:45:54.571 INFO:tasks.rados.rados.0.smithi084.stdout:951: done (3 left) 2024-09-20T22:45:54.572 INFO:tasks.rados.rados.0.smithi084.stdout:952: done (2 left) 2024-09-20T22:45:54.572 INFO:tasks.rados.rados.0.smithi084.stdout:953: done (1 left) 2024-09-20T22:45:54.572 INFO:tasks.rados.rados.0.smithi084.stdout:954: done (0 left) 2024-09-20T22:45:54.572 INFO:tasks.rados.rados.0.smithi084.stdout:955: rollback oid 40 current snap is 90 2024-09-20T22:45:54.572 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 40 to 88 2024-09-20T22:45:54.572 INFO:tasks.rados.rados.0.smithi084.stdout:956: snap_remove snap 85 2024-09-20T22:45:54.572 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:54.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:54.573 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:45:54.674 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:45:54.675 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:45:54.697 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:54.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:54.698 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:45:54.798 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:45:54.824 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:54.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:54.825 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:45:54.926 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:45:54.952 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:54.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:54.953 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:45:55.053 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:45:55.080 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:55.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:55.081 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:45:55.125 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-20T22:45:55.125 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-20T22:45:55.125 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph tell osd.2 'trim stale osdmaps' 2024-09-20T22:45:55.168 INFO:tasks.rados.rados.0.smithi084.stdout:956: done (1 left) 2024-09-20T22:45:55.168 INFO:tasks.rados.rados.0.smithi084.stdout:957: rollback oid 7 current snap is 90 2024-09-20T22:45:55.168 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 7 to 88 2024-09-20T22:45:55.168 INFO:tasks.rados.rados.0.smithi084.stdout:955: finishing rollback tid 0 to smithi08477203-40 2024-09-20T22:45:55.168 INFO:tasks.rados.rados.0.smithi084.stdout:955: finishing rollback tid 1 to smithi08477203-40 2024-09-20T22:45:55.168 INFO:tasks.rados.rados.0.smithi084.stdout:955: finishing rollback tid 2 to smithi08477203-40 2024-09-20T22:45:55.168 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 40 v 198 (ObjNum 316 snap 80 seq_num 316) dirty exists 2024-09-20T22:45:55.169 INFO:tasks.rados.rados.0.smithi084.stdout:955: done (1 left) 2024-09-20T22:45:55.169 INFO:tasks.rados.rados.0.smithi084.stdout:958: write oid 3 current snap is 90 2024-09-20T22:45:55.169 INFO:tasks.rados.rados.0.smithi084.stdout:958: seq_num 355 ranges {766485=465633,1990637=647776,3299280=667205} 2024-09-20T22:45:55.175 INFO:tasks.rados.rados.0.smithi084.stdout:958: writing smithi08477203-3 from 766485 to 1232118 tid 1 2024-09-20T22:45:55.181 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:45:55.183 INFO:tasks.rados.rados.0.smithi084.stdout:958: writing smithi08477203-3 from 1990637 to 2638413 tid 2 2024-09-20T22:45:55.191 INFO:tasks.rados.rados.0.smithi084.stdout:958: writing smithi08477203-3 from 3299280 to 3966485 tid 3 2024-09-20T22:45:55.191 INFO:tasks.rados.rados.0.smithi084.stdout:959: snap_create 2024-09-20T22:45:55.191 INFO:tasks.rados.rados.0.smithi084.stdout:957: finishing rollback tid 0 to smithi08477203-7 2024-09-20T22:45:55.191 INFO:tasks.rados.rados.0.smithi084.stdout:957: finishing rollback tid 1 to smithi08477203-7 2024-09-20T22:45:55.191 INFO:tasks.rados.rados.0.smithi084.stdout:957: finishing rollback tid 2 to smithi08477203-7 2024-09-20T22:45:55.191 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 7 v 202 (ObjNum 260 snap 61 seq_num 260) dirty exists 2024-09-20T22:45:55.191 INFO:tasks.rados.rados.0.smithi084.stdout:958: finishing write tid 1 to smithi08477203-3 2024-09-20T22:45:55.191 INFO:tasks.rados.rados.0.smithi084.stdout:958: finishing write tid 2 to smithi08477203-3 2024-09-20T22:45:55.194 INFO:tasks.rados.rados.0.smithi084.stdout:958: finishing write tid 3 to smithi08477203-3 2024-09-20T22:45:55.194 INFO:tasks.rados.rados.0.smithi084.stdout:958: finishing write tid 4 to smithi08477203-3 2024-09-20T22:45:55.194 INFO:tasks.rados.rados.0.smithi084.stdout:958: finishing write tid 5 to smithi08477203-3 2024-09-20T22:45:55.194 INFO:tasks.rados.rados.0.smithi084.stdout:958: oid 3 updating version 0 to 151 2024-09-20T22:45:55.194 INFO:tasks.rados.rados.0.smithi084.stdout:958: oid 3 version 151 is already newer than 150 2024-09-20T22:45:55.195 INFO:tasks.rados.rados.0.smithi084.stdout:958: oid 3 version 151 is already newer than 148 2024-09-20T22:45:55.195 INFO:tasks.rados.rados.0.smithi084.stdout:958: oid 3 version 151 is already newer than 149 2024-09-20T22:45:55.195 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 3 v 151 (ObjNum 355 snap 90 seq_num 355) dirty exists 2024-09-20T22:45:55.195 INFO:tasks.rados.rados.0.smithi084.stdout:958: left oid 3 (ObjNum 355 snap 90 seq_num 355) 2024-09-20T22:45:55.205 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:55.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:55.206 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:45:55.307 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:45:55.325 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:55 smithi084 ceph-mon[40024]: 7.17 scrub starts 2024-09-20T22:45:55.325 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:55 smithi084 ceph-mon[40024]: 7.17 scrub ok 2024-09-20T22:45:55.325 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:55 smithi084 ceph-mon[40024]: pgmap v313: 175 pgs: 2 clean+premerge+peered, 4 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 166 active+clean; 201 MiB data, 697 MiB used, 536 GiB / 536 GiB avail; 17 MiB/s rd, 2.8 MiB/s wr, 25 op/s 2024-09-20T22:45:55.325 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:55 smithi084 ceph-mon[40024]: 3.1c scrub starts 2024-09-20T22:45:55.325 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:55 smithi084 ceph-mon[40024]: 3.1c scrub ok 2024-09-20T22:45:55.325 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:55 smithi084 ceph-mon[40024]: 1.4 scrub starts 2024-09-20T22:45:55.325 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:55 smithi084 ceph-mon[40024]: 1.4 scrub ok 2024-09-20T22:45:55.325 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:55 smithi084 ceph-mon[40024]: 4.d scrub starts 2024-09-20T22:45:55.325 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:55 smithi084 ceph-mon[40024]: 4.d scrub ok 2024-09-20T22:45:55.325 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:55 smithi084 ceph-mon[40024]: osdmap e276: 8 total, 8 up, 6 in 2024-09-20T22:45:55.326 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:55 smithi084 ceph-mon[29774]: 7.17 scrub starts 2024-09-20T22:45:55.326 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:55 smithi084 ceph-mon[29774]: 7.17 scrub ok 2024-09-20T22:45:55.326 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:55 smithi084 ceph-mon[29774]: pgmap v313: 175 pgs: 2 clean+premerge+peered, 4 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 166 active+clean; 201 MiB data, 697 MiB used, 536 GiB / 536 GiB avail; 17 MiB/s rd, 2.8 MiB/s wr, 25 op/s 2024-09-20T22:45:55.326 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:55 smithi084 ceph-mon[29774]: 3.1c scrub starts 2024-09-20T22:45:55.326 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:55 smithi084 ceph-mon[29774]: 3.1c scrub ok 2024-09-20T22:45:55.326 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:55 smithi084 ceph-mon[29774]: 1.4 scrub starts 2024-09-20T22:45:55.326 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:55 smithi084 ceph-mon[29774]: 1.4 scrub ok 2024-09-20T22:45:55.326 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:55 smithi084 ceph-mon[29774]: 4.d scrub starts 2024-09-20T22:45:55.326 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:55 smithi084 ceph-mon[29774]: 4.d scrub ok 2024-09-20T22:45:55.326 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:55 smithi084 ceph-mon[29774]: osdmap e276: 8 total, 8 up, 6 in 2024-09-20T22:45:55.331 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:55.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:55.332 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:45:55.432 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:45:55.454 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:55.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:55.455 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:45:55.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:55 smithi154 ceph-mon[33264]: 7.17 scrub starts 2024-09-20T22:45:55.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:55 smithi154 ceph-mon[33264]: 7.17 scrub ok 2024-09-20T22:45:55.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:55 smithi154 ceph-mon[33264]: pgmap v313: 175 pgs: 2 clean+premerge+peered, 4 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 166 active+clean; 201 MiB data, 697 MiB used, 536 GiB / 536 GiB avail; 17 MiB/s rd, 2.8 MiB/s wr, 25 op/s 2024-09-20T22:45:55.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:55 smithi154 ceph-mon[33264]: 3.1c scrub starts 2024-09-20T22:45:55.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:55 smithi154 ceph-mon[33264]: 3.1c scrub ok 2024-09-20T22:45:55.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:55 smithi154 ceph-mon[33264]: 1.4 scrub starts 2024-09-20T22:45:55.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:55 smithi154 ceph-mon[33264]: 1.4 scrub ok 2024-09-20T22:45:55.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:55 smithi154 ceph-mon[33264]: 4.d scrub starts 2024-09-20T22:45:55.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:55 smithi154 ceph-mon[33264]: 4.d scrub ok 2024-09-20T22:45:55.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:55 smithi154 ceph-mon[33264]: osdmap e276: 8 total, 8 up, 6 in 2024-09-20T22:45:55.555 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:45:55.577 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:55.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:55.578 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:45:55.679 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:45:55.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:55.705 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:55.705 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:45:55.793 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:45:55.793 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:45:55.806 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:45:55.810 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:45:55.810 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:45:55.826 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:45:55.826 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:45:55.830 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:55.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:55.831 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:45:55.842 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-20T22:45:55.842 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T22:45:55.859 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:45:55.859 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:45:55.876 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:45:55.876 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:45:55.893 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:45:55.894 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:45:55.909 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-20T22:45:55.912 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:45:55.912 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:45:55.933 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:45:55.975 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:55.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:55.976 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:45:56.076 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:45:56.102 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:56.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:56.103 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:45:56.166 INFO:tasks.rados.rados.0.smithi084.stdout:957: done (2 left) 2024-09-20T22:45:56.167 INFO:tasks.rados.rados.0.smithi084.stdout:958: done (1 left) 2024-09-20T22:45:56.167 INFO:tasks.rados.rados.0.smithi084.stdout:959: done (0 left) 2024-09-20T22:45:56.167 INFO:tasks.rados.rados.0.smithi084.stdout:960: delete oid 8 current snap is 91 2024-09-20T22:45:56.169 INFO:tasks.rados.rados.0.smithi084.stdout:960: done (0 left) 2024-09-20T22:45:56.169 INFO:tasks.rados.rados.0.smithi084.stdout:961: delete oid 12 current snap is 91 2024-09-20T22:45:56.173 INFO:tasks.rados.rados.0.smithi084.stdout:961: done (0 left) 2024-09-20T22:45:56.173 INFO:tasks.rados.rados.0.smithi084.stdout:962: delete oid 7 current snap is 91 2024-09-20T22:45:56.178 INFO:tasks.rados.rados.0.smithi084.stdout:962: done (0 left) 2024-09-20T22:45:56.178 INFO:tasks.rados.rados.0.smithi084.stdout:963: snap_create 2024-09-20T22:45:56.204 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:45:56.231 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:56.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:56.232 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:45:56.301 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-20T22:45:56.301 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2024-09-20T22:45:56.301 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2024-09-20T22:45:56.301 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [ 2024-09-20T22:45:56.301 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.301 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.14460.0:1 3.12 3:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-20T22:45:56.301 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:41:37.966675+0000", 2024-09-20T22:45:56.301 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 258.05773963299998, 2024-09-20T22:45:56.301 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.19206436900000001, 2024-09-20T22:45:56.301 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:45:56.301 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:45:56.301 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "reached pg", 2024-09-20T22:45:56.302 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:45:56.302 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.14460", 2024-09-20T22:45:56.302 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/1273482728", 2024-09-20T22:45:56.302 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1 2024-09-20T22:45:56.302 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.302 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:45:56.302 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.302 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:45:56.302 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:41:37.966675+0000", 2024-09-20T22:45:56.302 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:45:56.302 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.302 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.302 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:45:56.303 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:41:37.966675+0000", 2024-09-20T22:45:56.303 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:45:56.303 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.303 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.303 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:45:56.303 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:41:37.966682+0000", 2024-09-20T22:45:56.303 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.3100000000000003e-06 2024-09-20T22:45:56.303 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.303 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.303 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:45:56.303 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:41:37.966688+0000", 2024-09-20T22:45:56.303 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.1209999999999998e-06 2024-09-20T22:45:56.303 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.303 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.304 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:45:56.304 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:41:37.966698+0000", 2024-09-20T22:45:56.304 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0220999999999999e-05 2024-09-20T22:45:56.304 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.304 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.304 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:45:56.304 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:41:37.966706+0000", 2024-09-20T22:45:56.304 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.4460000000000001e-06 2024-09-20T22:45:56.304 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.304 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.304 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:45:56.304 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:41:37.966763+0000", 2024-09-20T22:45:56.304 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.7025999999999999e-05 2024-09-20T22:45:56.304 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.305 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.305 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-09-20T22:45:56.305 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:41:37.966771+0000", 2024-09-20T22:45:56.305 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.9540000000000004e-06 2024-09-20T22:45:56.305 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.305 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.305 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:45:56.305 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:41:38.158654+0000", 2024-09-20T22:45:56.305 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.19188287000000001 2024-09-20T22:45:56.305 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.305 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.305 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:45:56.305 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:41:38.158740+0000", 2024-09-20T22:45:56.305 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.5420999999999994e-05 2024-09-20T22:45:56.305 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:56.306 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:45:56.306 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:56.306 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.306 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.306 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24652.0:336 7.7 e107/94 7:e67788e2:::smithi08477203-31:head v 107'45, mlcod=105'41)", 2024-09-20T22:45:56.306 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:43:06.546789+0000", 2024-09-20T22:45:56.306 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 169.477625859, 2024-09-20T22:45:56.306 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013900732000000001, 2024-09-20T22:45:56.306 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:45:56.306 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:45:56.306 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:45:56.306 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:45:56.306 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.307 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:45:56.307 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:06.546789+0000", 2024-09-20T22:45:56.307 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:45:56.307 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.307 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.307 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:45:56.307 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:06.546789+0000", 2024-09-20T22:45:56.307 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:45:56.307 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.307 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.307 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:45:56.307 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:06.546789+0000", 2024-09-20T22:45:56.307 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.1500000000000001e-07 2024-09-20T22:45:56.307 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.308 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.308 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:45:56.308 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:06.554560+0000", 2024-09-20T22:45:56.308 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0077711159999999998 2024-09-20T22:45:56.308 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.308 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.308 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:45:56.308 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:06.554577+0000", 2024-09-20T22:45:56.308 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.6509000000000001e-05 2024-09-20T22:45:56.308 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.308 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.308 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:45:56.308 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:06.554586+0000", 2024-09-20T22:45:56.308 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.7089999999999997e-06 2024-09-20T22:45:56.309 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.309 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.309 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:45:56.309 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:06.554645+0000", 2024-09-20T22:45:56.309 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.8202999999999998e-05 2024-09-20T22:45:56.309 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.309 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.309 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:45:56.309 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:06.554661+0000", 2024-09-20T22:45:56.309 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.6099e-05 2024-09-20T22:45:56.309 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.309 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.309 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:45:56.310 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:06.560643+0000", 2024-09-20T22:45:56.310 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0059824750000000001 2024-09-20T22:45:56.310 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.310 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.310 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:45:56.310 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:06.560690+0000", 2024-09-20T22:45:56.310 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.6406000000000003e-05 2024-09-20T22:45:56.310 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:56.310 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:45:56.310 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:56.310 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.311 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.311 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:353 7.f 7:f93102e8:::smithi08477203-37:head [copy-from ver 3] snapc 6=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-09-20T22:45:56.311 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:43:07.813602+0000", 2024-09-20T22:45:56.311 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 168.21081322500001, 2024-09-20T22:45:56.311 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014089683, 2024-09-20T22:45:56.311 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:45:56.311 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:45:56.311 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:45:56.311 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:45:56.311 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:45:56.311 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:45:56.311 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 353 2024-09-20T22:45:56.311 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.312 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:45:56.312 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.312 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:45:56.312 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.813602+0000", 2024-09-20T22:45:56.312 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:45:56.312 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.312 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.312 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:45:56.312 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.813602+0000", 2024-09-20T22:45:56.312 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:45:56.312 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.312 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.312 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:45:56.312 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.813604+0000", 2024-09-20T22:45:56.313 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.9939999999999999e-06 2024-09-20T22:45:56.313 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.313 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.313 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:45:56.313 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.813612+0000", 2024-09-20T22:45:56.313 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.3690000000000007e-06 2024-09-20T22:45:56.313 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.313 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.313 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:45:56.313 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.813631+0000", 2024-09-20T22:45:56.313 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.8589000000000001e-05 2024-09-20T22:45:56.313 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.313 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.314 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:45:56.314 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.813642+0000", 2024-09-20T22:45:56.314 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1766999999999999e-05 2024-09-20T22:45:56.314 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.314 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.314 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:45:56.314 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.813711+0000", 2024-09-20T22:45:56.314 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.9175000000000001e-05 2024-09-20T22:45:56.314 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.314 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.314 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:45:56.314 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.813811+0000", 2024-09-20T22:45:56.314 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.9672000000000004e-05 2024-09-20T22:45:56.314 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.315 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.315 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 3,7", 2024-09-20T22:45:56.315 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.817022+0000", 2024-09-20T22:45:56.315 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00321135 2024-09-20T22:45:56.315 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.315 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.315 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:45:56.315 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.820378+0000", 2024-09-20T22:45:56.315 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.003355224 2024-09-20T22:45:56.315 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.315 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.315 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:45:56.315 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.820397+0000", 2024-09-20T22:45:56.315 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.8899999999999999e-05 2024-09-20T22:45:56.316 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.316 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.316 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:45:56.316 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.827619+0000", 2024-09-20T22:45:56.316 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.007222816 2024-09-20T22:45:56.316 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.316 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.316 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:45:56.316 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.827653+0000", 2024-09-20T22:45:56.316 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.392e-05 2024-09-20T22:45:56.316 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.316 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.316 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:45:56.316 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.827691+0000", 2024-09-20T22:45:56.316 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.7907000000000001e-05 2024-09-20T22:45:56.317 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:56.317 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:45:56.317 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:56.317 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.317 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.317 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:359 7.9 7:91f2d47c:::smithi08477203-8:head [copy-from ver 6] snapc 6=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-09-20T22:45:56.317 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:43:07.829163+0000", 2024-09-20T22:45:56.317 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 168.19525220599999, 2024-09-20T22:45:56.317 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011899852000000001, 2024-09-20T22:45:56.317 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:45:56.317 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:45:56.317 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:45:56.317 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:45:56.317 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:45:56.317 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:45:56.318 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 359 2024-09-20T22:45:56.318 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.318 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:45:56.318 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.318 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:45:56.318 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.829163+0000", 2024-09-20T22:45:56.318 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:45:56.318 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.318 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.318 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:45:56.318 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.829163+0000", 2024-09-20T22:45:56.318 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:45:56.318 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.319 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.319 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:45:56.319 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.829163+0000", 2024-09-20T22:45:56.319 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.6199999999999999e-07 2024-09-20T22:45:56.319 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.319 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.319 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:45:56.319 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.829165+0000", 2024-09-20T22:45:56.319 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.7209999999999999e-06 2024-09-20T22:45:56.319 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.319 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.319 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:45:56.320 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.829169+0000", 2024-09-20T22:45:56.320 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.121e-06 2024-09-20T22:45:56.320 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.320 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.320 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:45:56.320 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.829172+0000", 2024-09-20T22:45:56.320 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.8839999999999998e-06 2024-09-20T22:45:56.320 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.320 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.320 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:45:56.320 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.829211+0000", 2024-09-20T22:45:56.320 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.9551000000000001e-05 2024-09-20T22:45:56.320 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.320 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.321 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:45:56.321 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.829294+0000", 2024-09-20T22:45:56.321 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.2548999999999999e-05 2024-09-20T22:45:56.321 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.321 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.321 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 1,4", 2024-09-20T22:45:56.321 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.834578+0000", 2024-09-20T22:45:56.321 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.005284179 2024-09-20T22:45:56.321 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.321 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.321 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:45:56.321 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.840537+0000", 2024-09-20T22:45:56.321 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0059587670000000002 2024-09-20T22:45:56.321 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.322 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.322 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:45:56.322 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.840753+0000", 2024-09-20T22:45:56.322 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00021661799999999999 2024-09-20T22:45:56.322 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.322 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.322 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:45:56.322 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.840999+0000", 2024-09-20T22:45:56.322 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000246009 2024-09-20T22:45:56.322 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.322 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.322 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:45:56.322 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.841032+0000", 2024-09-20T22:45:56.322 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.2567999999999997e-05 2024-09-20T22:45:56.322 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.323 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.323 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:45:56.323 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.841062+0000", 2024-09-20T22:45:56.323 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.0623000000000001e-05 2024-09-20T22:45:56.323 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:56.323 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:45:56.323 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:56.323 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.323 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.323 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:397 7.9 7:91f2d47c:::smithi08477203-8:head [copy-from ver 3] snapc 8=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-09-20T22:45:56.323 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:43:11.361736+0000", 2024-09-20T22:45:56.323 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 164.662679287, 2024-09-20T22:45:56.323 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011019584000000001, 2024-09-20T22:45:56.323 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:45:56.324 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:45:56.324 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:45:56.324 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:45:56.324 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:45:56.324 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:45:56.324 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 397 2024-09-20T22:45:56.324 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.324 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:45:56.324 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.324 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:45:56.324 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:11.361736+0000", 2024-09-20T22:45:56.324 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:45:56.324 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.325 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.325 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:45:56.325 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:11.361736+0000", 2024-09-20T22:45:56.325 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:45:56.325 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.325 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.325 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:45:56.325 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:11.361738+0000", 2024-09-20T22:45:56.325 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.2120000000000002e-06 2024-09-20T22:45:56.325 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.325 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.325 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:45:56.325 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:11.361746+0000", 2024-09-20T22:45:56.325 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.4749999999999993e-06 2024-09-20T22:45:56.325 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.326 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.326 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:45:56.326 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:11.361766+0000", 2024-09-20T22:45:56.326 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.9519999999999999e-05 2024-09-20T22:45:56.326 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.326 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.326 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:45:56.326 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:11.361776+0000", 2024-09-20T22:45:56.326 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0719e-05 2024-09-20T22:45:56.326 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.326 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.326 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:45:56.326 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:11.361847+0000", 2024-09-20T22:45:56.326 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.0829999999999998e-05 2024-09-20T22:45:56.327 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.327 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.327 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:45:56.327 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:11.361951+0000", 2024-09-20T22:45:56.327 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000103538 2024-09-20T22:45:56.327 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.327 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.327 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 1,4", 2024-09-20T22:45:56.327 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:11.366189+0000", 2024-09-20T22:45:56.327 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0042376990000000002 2024-09-20T22:45:56.327 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.327 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.327 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:45:56.327 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:11.372340+0000", 2024-09-20T22:45:56.328 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0061511650000000001 2024-09-20T22:45:56.328 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.328 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.328 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:45:56.328 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:11.372390+0000", 2024-09-20T22:45:56.328 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.0210999999999997e-05 2024-09-20T22:45:56.328 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.328 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.328 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:45:56.328 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:11.372645+0000", 2024-09-20T22:45:56.328 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000255233 2024-09-20T22:45:56.328 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.328 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.328 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:45:56.328 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:11.372707+0000", 2024-09-20T22:45:56.329 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.1427999999999995e-05 2024-09-20T22:45:56.329 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.329 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.329 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:45:56.329 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:11.372755+0000", 2024-09-20T22:45:56.329 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.8554000000000003e-05 2024-09-20T22:45:56.329 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:56.329 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:45:56.329 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:56.329 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.329 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.329 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:522 7.9 7:9970b578:::smithi08477203-15:head [copy-from ver 28] snapc f=[f,d] ondisk+write+known_if_redirected+supports_pool_eio e124)", 2024-09-20T22:45:56.329 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:43:23.126433+0000", 2024-09-20T22:45:56.330 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 152.89798207699999, 2024-09-20T22:45:56.330 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.027726022999999999, 2024-09-20T22:45:56.330 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:45:56.330 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:45:56.330 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:45:56.330 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:45:56.330 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:45:56.330 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:45:56.330 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 522 2024-09-20T22:45:56.330 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.330 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:45:56.330 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.330 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:45:56.330 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:23.126433+0000", 2024-09-20T22:45:56.331 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:45:56.331 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.331 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.331 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:45:56.331 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:23.126433+0000", 2024-09-20T22:45:56.331 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:45:56.331 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.331 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.331 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:45:56.331 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:23.126434+0000", 2024-09-20T22:45:56.331 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3200000000000001e-06 2024-09-20T22:45:56.331 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.331 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.332 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:45:56.332 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:23.126438+0000", 2024-09-20T22:45:56.332 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.6959999999999998e-06 2024-09-20T22:45:56.332 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.332 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.332 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:45:56.332 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:23.126448+0000", 2024-09-20T22:45:56.332 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0436e-05 2024-09-20T22:45:56.332 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.332 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.332 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:45:56.332 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:23.126455+0000", 2024-09-20T22:45:56.332 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.7399999999999998e-06 2024-09-20T22:45:56.332 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.332 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.333 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:45:56.333 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:23.126505+0000", 2024-09-20T22:45:56.333 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.9622e-05 2024-09-20T22:45:56.333 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.333 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.333 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:45:56.333 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:23.126598+0000", 2024-09-20T22:45:56.333 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.3905000000000003e-05 2024-09-20T22:45:56.333 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.333 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.333 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 1,4", 2024-09-20T22:45:56.333 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:23.136852+0000", 2024-09-20T22:45:56.333 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010253372 2024-09-20T22:45:56.333 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.334 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.334 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:45:56.334 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:23.146781+0000", 2024-09-20T22:45:56.334 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0099292019999999998 2024-09-20T22:45:56.334 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.334 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.334 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:45:56.334 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:23.146838+0000", 2024-09-20T22:45:56.334 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.7046999999999998e-05 2024-09-20T22:45:56.334 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.334 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.334 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:45:56.334 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:23.154054+0000", 2024-09-20T22:45:56.335 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0072155910000000004 2024-09-20T22:45:56.335 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.335 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.335 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:45:56.335 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:23.154100+0000", 2024-09-20T22:45:56.335 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.6613999999999998e-05 2024-09-20T22:45:56.335 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.335 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.335 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:45:56.335 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:23.154159+0000", 2024-09-20T22:45:56.335 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.8477999999999998e-05 2024-09-20T22:45:56.335 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:56.335 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:45:56.335 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:56.336 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.336 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.336 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:523 7.9 7:9970b578:::smithi08477203-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e124)", 2024-09-20T22:45:56.336 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:43:23.126473+0000", 2024-09-20T22:45:56.336 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 152.897941962, 2024-09-20T22:45:56.336 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.015897446999999999, 2024-09-20T22:45:56.336 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:45:56.336 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:45:56.336 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-20T22:45:56.336 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:45:56.336 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:45:56.336 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:45:56.336 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 523 2024-09-20T22:45:56.336 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.336 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:45:56.337 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.337 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:45:56.337 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:23.126473+0000", 2024-09-20T22:45:56.337 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:45:56.337 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.337 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.337 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:45:56.337 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:23.126473+0000", 2024-09-20T22:45:56.337 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:45:56.337 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.337 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.337 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:45:56.337 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:23.126473+0000", 2024-09-20T22:45:56.337 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.5599999999999995e-07 2024-09-20T22:45:56.337 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.338 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.338 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:45:56.338 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:23.126476+0000", 2024-09-20T22:45:56.338 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.357e-06 2024-09-20T22:45:56.338 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.338 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.338 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:45:56.338 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:23.126482+0000", 2024-09-20T22:45:56.338 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.5899999999999996e-06 2024-09-20T22:45:56.338 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.338 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.338 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:45:56.338 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:23.126488+0000", 2024-09-20T22:45:56.338 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.6219999999999997e-06 2024-09-20T22:45:56.338 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.339 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.339 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:45:56.339 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:23.126755+0000", 2024-09-20T22:45:56.339 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026716100000000003 2024-09-20T22:45:56.339 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.339 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.339 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for blocked object", 2024-09-20T22:45:56.339 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:23.126808+0000", 2024-09-20T22:45:56.339 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.2859999999999999e-05 2024-09-20T22:45:56.339 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.339 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.339 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:45:56.339 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:23.142223+0000", 2024-09-20T22:45:56.339 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.015414773999999999 2024-09-20T22:45:56.339 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.340 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.340 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:45:56.340 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:23.142296+0000", 2024-09-20T22:45:56.340 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.3394999999999998e-05 2024-09-20T22:45:56.340 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.340 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.340 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:45:56.340 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:23.142370+0000", 2024-09-20T22:45:56.340 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.4132000000000001e-05 2024-09-20T22:45:56.340 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:56.340 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:45:56.340 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:56.340 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.340 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.341 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:792 7.f 7:f93102e8:::smithi08477203-37:head [write 618524~710869,stat] snapc 19=[17] ondisk+write+known_if_redirected+supports_pool_eio e145)", 2024-09-20T22:45:56.341 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:43:44.346227+0000", 2024-09-20T22:45:56.341 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 131.67818754499999, 2024-09-20T22:45:56.341 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010284069, 2024-09-20T22:45:56.341 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:45:56.341 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:45:56.341 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:45:56.341 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:45:56.341 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:45:56.341 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:45:56.341 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 792 2024-09-20T22:45:56.341 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.341 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:45:56.341 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.341 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:45:56.342 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:44.346227+0000", 2024-09-20T22:45:56.342 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:45:56.342 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.342 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.342 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:45:56.342 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:44.346227+0000", 2024-09-20T22:45:56.342 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:45:56.342 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.342 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.342 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:45:56.342 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:44.346229+0000", 2024-09-20T22:45:56.342 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.9290000000000001e-06 2024-09-20T22:45:56.342 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.342 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.343 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:45:56.343 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:44.346616+0000", 2024-09-20T22:45:56.343 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00038728800000000001 2024-09-20T22:45:56.343 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.343 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.343 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:45:56.343 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:44.346635+0000", 2024-09-20T22:45:56.343 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.8380999999999999e-05 2024-09-20T22:45:56.343 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.343 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.343 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:45:56.343 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:44.346646+0000", 2024-09-20T22:45:56.343 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1106e-05 2024-09-20T22:45:56.343 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.343 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.344 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:45:56.344 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:44.346716+0000", 2024-09-20T22:45:56.344 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.0067000000000005e-05 2024-09-20T22:45:56.344 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.344 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.344 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:45:56.344 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:44.346832+0000", 2024-09-20T22:45:56.344 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000116221 2024-09-20T22:45:56.344 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.344 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.344 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 3,7", 2024-09-20T22:45:56.344 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:44.347050+0000", 2024-09-20T22:45:56.344 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000218011 2024-09-20T22:45:56.344 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.345 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.345 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:45:56.345 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:44.348787+0000", 2024-09-20T22:45:56.345 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00173689 2024-09-20T22:45:56.345 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.345 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.345 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:45:56.345 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:44.349085+0000", 2024-09-20T22:45:56.345 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00029818299999999999 2024-09-20T22:45:56.345 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.345 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.345 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:45:56.345 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:44.356381+0000", 2024-09-20T22:45:56.345 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0072956920000000003 2024-09-20T22:45:56.346 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.346 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.346 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:45:56.346 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:44.356449+0000", 2024-09-20T22:45:56.346 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.8219000000000005e-05 2024-09-20T22:45:56.346 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.346 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.346 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:45:56.346 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:44.356511+0000", 2024-09-20T22:45:56.346 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.2082000000000006e-05 2024-09-20T22:45:56.346 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:56.346 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:45:56.346 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:56.346 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.346 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.347 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24652.0:901 7.e e156/94 7:7c62ced4:::smithi08477203-17:head v 156'42, mlcod=154'40)", 2024-09-20T22:45:56.347 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:43:55.635862+0000", 2024-09-20T22:45:56.347 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 120.38855266, 2024-09-20T22:45:56.347 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010520952, 2024-09-20T22:45:56.347 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:45:56.347 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:45:56.347 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:45:56.347 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:45:56.347 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.347 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:45:56.347 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.635862+0000", 2024-09-20T22:45:56.347 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:45:56.347 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.347 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.348 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:45:56.348 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.635862+0000", 2024-09-20T22:45:56.348 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:45:56.348 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.348 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.348 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:45:56.348 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.635863+0000", 2024-09-20T22:45:56.348 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.7000000000000005e-07 2024-09-20T22:45:56.348 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.348 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.348 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:45:56.348 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.638378+0000", 2024-09-20T22:45:56.348 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025154499999999998 2024-09-20T22:45:56.348 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.348 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.349 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:45:56.349 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.638408+0000", 2024-09-20T22:45:56.349 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.0247999999999999e-05 2024-09-20T22:45:56.349 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.349 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.349 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:45:56.349 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.638420+0000", 2024-09-20T22:45:56.349 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1566000000000001e-05 2024-09-20T22:45:56.349 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.349 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.349 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:45:56.349 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.638504+0000", 2024-09-20T22:45:56.349 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.4320999999999994e-05 2024-09-20T22:45:56.349 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.349 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.350 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:45:56.350 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.638530+0000", 2024-09-20T22:45:56.350 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.5321999999999999e-05 2024-09-20T22:45:56.350 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.350 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.350 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:45:56.350 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.646340+0000", 2024-09-20T22:45:56.350 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0078100009999999996 2024-09-20T22:45:56.350 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.350 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.350 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:45:56.350 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.646383+0000", 2024-09-20T22:45:56.350 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.3473999999999997e-05 2024-09-20T22:45:56.350 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:56.351 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:45:56.351 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:56.351 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.351 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.351 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:928 7.9 7:91f2d47c:::smithi08477203-8:head [stat,delete] snapc 1f=[] ondisk+write+known_if_redirected+supports_pool_eio e158)", 2024-09-20T22:45:56.351 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:43:57.433849+0000", 2024-09-20T22:45:56.351 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 118.590565962, 2024-09-20T22:45:56.351 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.012508692, 2024-09-20T22:45:56.351 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:45:56.351 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:45:56.351 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-20T22:45:56.351 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:45:56.351 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:45:56.351 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:45:56.352 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 928 2024-09-20T22:45:56.352 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.352 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:45:56.352 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.352 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:45:56.352 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:57.433849+0000", 2024-09-20T22:45:56.352 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:45:56.352 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.352 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.352 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:45:56.352 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:57.433849+0000", 2024-09-20T22:45:56.352 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:45:56.352 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.352 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.352 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:45:56.353 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:57.433850+0000", 2024-09-20T22:45:56.353 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.4670000000000001e-06 2024-09-20T22:45:56.353 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.353 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.353 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:45:56.353 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:57.433855+0000", 2024-09-20T22:45:56.353 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.4399999999999998e-06 2024-09-20T22:45:56.353 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.353 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.353 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:45:56.353 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:57.433865+0000", 2024-09-20T22:45:56.353 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.8779999999999993e-06 2024-09-20T22:45:56.353 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.353 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.353 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:45:56.354 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:57.433872+0000", 2024-09-20T22:45:56.354 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.2350000000000002e-06 2024-09-20T22:45:56.354 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.354 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.354 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:45:56.354 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:57.434024+0000", 2024-09-20T22:45:56.354 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000152091 2024-09-20T22:45:56.354 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.354 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.354 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for missing object", 2024-09-20T22:45:56.354 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:57.434140+0000", 2024-09-20T22:45:56.354 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000116242 2024-09-20T22:45:56.354 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.354 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.355 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:45:56.355 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:57.438648+0000", 2024-09-20T22:45:56.355 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0045080349999999996 2024-09-20T22:45:56.355 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.355 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.355 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:45:56.355 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:57.438752+0000", 2024-09-20T22:45:56.355 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00010382699999999999 2024-09-20T22:45:56.355 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.355 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.355 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:45:56.355 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:57.446358+0000", 2024-09-20T22:45:56.355 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0076054770000000002 2024-09-20T22:45:56.355 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:56.355 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:45:56.356 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:56.356 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.356 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.356 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:950 7.18 7:1cd232b2:::smithi08477203-35:head [copy-from ver 6] snapc 20=[20] ondisk+write+known_if_redirected+supports_pool_eio e159)", 2024-09-20T22:45:56.356 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:43:58.151870+0000", 2024-09-20T22:45:56.356 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 117.87254460299999, 2024-09-20T22:45:56.356 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.01686789, 2024-09-20T22:45:56.356 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:45:56.356 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:45:56.356 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:45:56.356 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:45:56.356 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:45:56.356 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:45:56.356 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 950 2024-09-20T22:45:56.357 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.357 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:45:56.357 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.357 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:45:56.357 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:58.151870+0000", 2024-09-20T22:45:56.357 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:45:56.357 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.357 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.357 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:45:56.357 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:58.151870+0000", 2024-09-20T22:45:56.357 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:45:56.357 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.357 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.357 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:45:56.358 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:58.151873+0000", 2024-09-20T22:45:56.358 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.3140000000000002e-06 2024-09-20T22:45:56.358 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.358 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.358 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:45:56.358 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:58.151896+0000", 2024-09-20T22:45:56.358 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.3788e-05 2024-09-20T22:45:56.358 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.358 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.358 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:45:56.358 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:58.151934+0000", 2024-09-20T22:45:56.358 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.7873999999999997e-05 2024-09-20T22:45:56.358 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.358 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.359 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:45:56.359 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:58.151956+0000", 2024-09-20T22:45:56.359 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.1523e-05 2024-09-20T22:45:56.359 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.359 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.359 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:45:56.359 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:58.152044+0000", 2024-09-20T22:45:56.359 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.8732000000000004e-05 2024-09-20T22:45:56.359 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.359 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.359 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:45:56.359 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:58.152152+0000", 2024-09-20T22:45:56.359 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000107769 2024-09-20T22:45:56.359 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.360 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.360 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 6,7", 2024-09-20T22:45:56.360 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:58.158276+0000", 2024-09-20T22:45:56.360 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0061237970000000003 2024-09-20T22:45:56.360 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.360 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.360 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:45:56.360 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:58.166555+0000", 2024-09-20T22:45:56.360 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0082786119999999994 2024-09-20T22:45:56.360 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.360 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.360 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:45:56.360 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:58.168271+0000", 2024-09-20T22:45:56.360 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001716062 2024-09-20T22:45:56.360 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.361 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.361 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:45:56.361 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:58.168658+0000", 2024-09-20T22:45:56.361 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000387563 2024-09-20T22:45:56.361 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.361 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.361 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:45:56.361 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:58.168693+0000", 2024-09-20T22:45:56.361 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.4870999999999999e-05 2024-09-20T22:45:56.361 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.361 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.361 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:45:56.361 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:58.168738+0000", 2024-09-20T22:45:56.361 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.4984999999999997e-05 2024-09-20T22:45:56.362 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:56.362 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:45:56.362 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:56.362 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.362 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.362 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:1075 7.9 7:907aec0e:::smithi08477203-3:head [append 0~0] snapc 26=[26,24] ondisk+write+known_if_redirected+supports_pool_eio e170)", 2024-09-20T22:45:56.362 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:44:09.184670+0000", 2024-09-20T22:45:56.362 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 106.83974509399999, 2024-09-20T22:45:56.362 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011407349000000001, 2024-09-20T22:45:56.362 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:45:56.362 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:45:56.362 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:45:56.362 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:45:56.362 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:45:56.363 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:45:56.363 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1075 2024-09-20T22:45:56.363 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.363 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:45:56.363 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.363 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:45:56.363 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:09.184670+0000", 2024-09-20T22:45:56.363 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:45:56.363 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.363 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.363 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:45:56.363 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:09.184670+0000", 2024-09-20T22:45:56.363 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:45:56.363 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.363 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.364 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:45:56.364 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:09.184670+0000", 2024-09-20T22:45:56.364 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.5499999999999998e-07 2024-09-20T22:45:56.364 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.364 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.364 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:45:56.364 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:09.184673+0000", 2024-09-20T22:45:56.364 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.7480000000000002e-06 2024-09-20T22:45:56.364 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.364 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.364 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:45:56.364 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:09.184691+0000", 2024-09-20T22:45:56.364 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.8219e-05 2024-09-20T22:45:56.364 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.364 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.365 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:45:56.365 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:09.184697+0000", 2024-09-20T22:45:56.365 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.8819999999999998e-06 2024-09-20T22:45:56.365 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.365 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.365 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:45:56.365 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:09.190938+0000", 2024-09-20T22:45:56.365 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0062412930000000002 2024-09-20T22:45:56.365 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.365 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.365 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:45:56.366 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:09.191026+0000", 2024-09-20T22:45:56.366 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.7445e-05 2024-09-20T22:45:56.366 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.366 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.366 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 1,7", 2024-09-20T22:45:56.366 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:09.191258+0000", 2024-09-20T22:45:56.366 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000232413 2024-09-20T22:45:56.366 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.366 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.366 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:45:56.366 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:09.192005+0000", 2024-09-20T22:45:56.366 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00074707500000000004 2024-09-20T22:45:56.366 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.366 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.367 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:45:56.367 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:09.193303+0000", 2024-09-20T22:45:56.367 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0012979630000000001 2024-09-20T22:45:56.367 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.367 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.367 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:45:56.367 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:09.195258+0000", 2024-09-20T22:45:56.367 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0019544689999999999 2024-09-20T22:45:56.367 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.367 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.367 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:45:56.367 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:09.195912+0000", 2024-09-20T22:45:56.367 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00065425800000000001 2024-09-20T22:45:56.367 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.367 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.368 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:45:56.368 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:09.196077+0000", 2024-09-20T22:45:56.368 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00016502899999999999 2024-09-20T22:45:56.368 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:56.368 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:45:56.368 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:56.368 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.368 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.368 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:1079 7.f 7:fa55e5ae:::smithi08477203-7:head [copy-from ver 87] snapc 26=[26,24] ondisk+write+known_if_redirected+supports_pool_eio e170)", 2024-09-20T22:45:56.368 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:44:09.184851+0000", 2024-09-20T22:45:56.368 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 106.83956353799999, 2024-09-20T22:45:56.368 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.028346902, 2024-09-20T22:45:56.368 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:45:56.368 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:45:56.369 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:45:56.369 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:45:56.369 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:45:56.369 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:45:56.369 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1079 2024-09-20T22:45:56.369 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.369 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:45:56.369 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.369 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:45:56.369 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:09.184851+0000", 2024-09-20T22:45:56.369 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:45:56.369 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.369 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.369 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:45:56.369 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:09.184851+0000", 2024-09-20T22:45:56.370 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:45:56.370 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.370 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.370 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:45:56.370 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:09.184852+0000", 2024-09-20T22:45:56.370 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.3399999999999999e-07 2024-09-20T22:45:56.370 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.370 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.370 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:45:56.370 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:09.184888+0000", 2024-09-20T22:45:56.370 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.6112000000000003e-05 2024-09-20T22:45:56.370 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.370 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.370 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:45:56.371 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:09.184895+0000", 2024-09-20T22:45:56.371 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.8539999999999999e-06 2024-09-20T22:45:56.371 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.371 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.371 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:45:56.371 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:09.184926+0000", 2024-09-20T22:45:56.371 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.1293e-05 2024-09-20T22:45:56.371 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.371 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.371 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:45:56.371 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:09.188919+0000", 2024-09-20T22:45:56.371 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0039926830000000003 2024-09-20T22:45:56.371 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.371 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.371 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:45:56.372 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:09.189012+0000", 2024-09-20T22:45:56.372 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.3690999999999995e-05 2024-09-20T22:45:56.372 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.372 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.372 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 3,7", 2024-09-20T22:45:56.372 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:09.205339+0000", 2024-09-20T22:45:56.372 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.016326092 2024-09-20T22:45:56.372 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.372 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.372 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:45:56.372 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:09.209753+0000", 2024-09-20T22:45:56.372 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0044144789999999998 2024-09-20T22:45:56.372 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.372 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.373 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:45:56.373 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:09.212376+0000", 2024-09-20T22:45:56.373 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026232180000000001 2024-09-20T22:45:56.373 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.373 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.373 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:45:56.373 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:09.213093+0000", 2024-09-20T22:45:56.373 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00071704400000000001 2024-09-20T22:45:56.373 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.373 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.373 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:45:56.373 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:09.213141+0000", 2024-09-20T22:45:56.373 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.7991999999999997e-05 2024-09-20T22:45:56.373 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.373 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.374 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:45:56.374 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:09.213198+0000", 2024-09-20T22:45:56.374 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.6910000000000002e-05 2024-09-20T22:45:56.374 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:56.374 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:45:56.374 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:56.374 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.374 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.374 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:1080 7.f 7:fa55e5ae:::smithi08477203-7:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e170)", 2024-09-20T22:45:56.374 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:44:09.184935+0000", 2024-09-20T22:45:56.374 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 106.839479846, 2024-09-20T22:45:56.374 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.024005572999999999, 2024-09-20T22:45:56.374 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:45:56.374 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:45:56.375 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-20T22:45:56.375 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:45:56.375 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:45:56.375 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:45:56.375 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1080 2024-09-20T22:45:56.375 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.375 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:45:56.375 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.375 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:45:56.375 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:09.184935+0000", 2024-09-20T22:45:56.375 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:45:56.375 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.375 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.375 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:45:56.376 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:09.184935+0000", 2024-09-20T22:45:56.376 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:45:56.376 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.376 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.376 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:45:56.376 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:09.184936+0000", 2024-09-20T22:45:56.376 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.6599999999999996e-07 2024-09-20T22:45:56.376 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.376 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.376 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:45:56.376 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:09.184938+0000", 2024-09-20T22:45:56.376 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.5359999999999999e-06 2024-09-20T22:45:56.376 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.376 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.376 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:45:56.377 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:09.184971+0000", 2024-09-20T22:45:56.377 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.2848000000000001e-05 2024-09-20T22:45:56.377 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.377 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.377 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:45:56.377 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:09.184977+0000", 2024-09-20T22:45:56.377 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.0859999999999998e-06 2024-09-20T22:45:56.377 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.377 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.377 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:45:56.377 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:09.189687+0000", 2024-09-20T22:45:56.377 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0047097700000000003 2024-09-20T22:45:56.377 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.377 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.378 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for blocked object", 2024-09-20T22:45:56.378 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:09.189852+0000", 2024-09-20T22:45:56.378 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000165243 2024-09-20T22:45:56.378 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.378 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.378 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:45:56.378 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:09.208617+0000", 2024-09-20T22:45:56.378 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.018765190000000001 2024-09-20T22:45:56.378 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.378 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.378 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:45:56.378 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:09.208772+0000", 2024-09-20T22:45:56.378 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000154318 2024-09-20T22:45:56.378 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.378 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.379 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:45:56.379 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:09.208941+0000", 2024-09-20T22:45:56.379 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000169016 2024-09-20T22:45:56.379 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:56.379 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:45:56.379 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:56.379 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.379 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.379 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24652.0:1542 7.12 e198/139 7:4b06cbed:::smithi08477203-49:head v 198'174, mlcod=197'171)", 2024-09-20T22:45:56.379 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:44:37.216845+0000", 2024-09-20T22:45:56.379 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 78.807569646999994, 2024-09-20T22:45:56.379 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011189424999999999, 2024-09-20T22:45:56.379 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:45:56.379 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:45:56.379 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:45:56.380 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:45:56.380 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.380 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:45:56.380 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:37.216845+0000", 2024-09-20T22:45:56.380 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:45:56.380 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.380 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.380 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:45:56.380 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:37.216845+0000", 2024-09-20T22:45:56.380 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:45:56.380 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.380 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.380 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:45:56.380 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:37.216846+0000", 2024-09-20T22:45:56.380 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.4900000000000001e-07 2024-09-20T22:45:56.381 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.381 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.381 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:45:56.381 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:37.221689+0000", 2024-09-20T22:45:56.381 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0048433640000000002 2024-09-20T22:45:56.381 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.381 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.381 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:45:56.381 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:37.221706+0000", 2024-09-20T22:45:56.381 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.7405999999999999e-05 2024-09-20T22:45:56.381 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.381 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.381 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:45:56.381 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:37.221716+0000", 2024-09-20T22:45:56.381 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.4290000000000007e-06 2024-09-20T22:45:56.382 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.382 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.382 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:45:56.382 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:37.221789+0000", 2024-09-20T22:45:56.382 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.2809e-05 2024-09-20T22:45:56.382 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.382 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.382 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:45:56.382 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:37.221814+0000", 2024-09-20T22:45:56.382 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.5235000000000001e-05 2024-09-20T22:45:56.382 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.382 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.382 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:45:56.382 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:37.227978+0000", 2024-09-20T22:45:56.383 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.006163656 2024-09-20T22:45:56.383 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.383 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.383 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:45:56.383 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:37.228035+0000", 2024-09-20T22:45:56.383 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.7077000000000003e-05 2024-09-20T22:45:56.383 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:56.383 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:45:56.383 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:56.383 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.383 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.383 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:1583 7.18 7:1e9e2524:::smithi08477203-5:head [copy-from ver 154] snapc 35=[35] ondisk+write+known_if_redirected+supports_pool_eio e201)", 2024-09-20T22:45:56.383 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:44:40.229595+0000", 2024-09-20T22:45:56.383 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 75.794819602999993, 2024-09-20T22:45:56.383 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.026717541000000001, 2024-09-20T22:45:56.384 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:45:56.384 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:45:56.384 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:45:56.384 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:45:56.384 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:45:56.384 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:45:56.384 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1583 2024-09-20T22:45:56.384 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.384 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:45:56.384 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.384 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:45:56.384 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:40.229595+0000", 2024-09-20T22:45:56.384 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:45:56.384 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.384 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.385 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:45:56.385 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:40.229595+0000", 2024-09-20T22:45:56.385 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:45:56.385 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.385 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.385 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:45:56.385 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:40.229596+0000", 2024-09-20T22:45:56.385 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.68e-07 2024-09-20T22:45:56.385 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.385 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.385 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:45:56.385 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:40.229600+0000", 2024-09-20T22:45:56.385 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.805e-06 2024-09-20T22:45:56.385 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.386 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.386 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:45:56.386 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:40.229606+0000", 2024-09-20T22:45:56.386 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.4300000000000003e-06 2024-09-20T22:45:56.386 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.386 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.386 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:45:56.386 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:40.229611+0000", 2024-09-20T22:45:56.386 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.0250000000000002e-06 2024-09-20T22:45:56.386 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.386 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.386 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:45:56.386 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:40.229661+0000", 2024-09-20T22:45:56.386 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.0343999999999998e-05 2024-09-20T22:45:56.386 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.387 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.387 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:45:56.387 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:40.229737+0000", 2024-09-20T22:45:56.387 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.5444000000000005e-05 2024-09-20T22:45:56.387 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.387 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.387 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 6,7", 2024-09-20T22:45:56.387 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:40.244365+0000", 2024-09-20T22:45:56.387 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014628238999999999 2024-09-20T22:45:56.387 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.387 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.387 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:45:56.387 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:40.249487+0000", 2024-09-20T22:45:56.387 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0051221799999999996 2024-09-20T22:45:56.387 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.388 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.388 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:45:56.388 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:40.252864+0000", 2024-09-20T22:45:56.388 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0033763830000000002 2024-09-20T22:45:56.388 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.388 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.388 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:45:56.388 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:40.256237+0000", 2024-09-20T22:45:56.388 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0033731550000000001 2024-09-20T22:45:56.388 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.388 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.388 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:45:56.388 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:40.256277+0000", 2024-09-20T22:45:56.388 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.0092999999999997e-05 2024-09-20T22:45:56.389 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.389 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.389 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:45:56.389 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:40.256313+0000", 2024-09-20T22:45:56.389 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.5874999999999998e-05 2024-09-20T22:45:56.389 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:56.389 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:45:56.389 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:56.389 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.389 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.389 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:1584 7.18 7:1e9e2524:::smithi08477203-5:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e201)", 2024-09-20T22:45:56.389 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:44:40.229623+0000", 2024-09-20T22:45:56.389 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 75.794791908999997, 2024-09-20T22:45:56.389 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.019097652, 2024-09-20T22:45:56.389 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:45:56.390 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:45:56.390 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-20T22:45:56.390 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:45:56.390 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:45:56.390 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:45:56.390 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1584 2024-09-20T22:45:56.390 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.390 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:45:56.390 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.390 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:45:56.390 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:40.229623+0000", 2024-09-20T22:45:56.390 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:45:56.390 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.390 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.390 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:45:56.391 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:40.229623+0000", 2024-09-20T22:45:56.391 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:45:56.391 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.391 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.391 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:45:56.391 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:40.229623+0000", 2024-09-20T22:45:56.391 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.7300000000000002e-07 2024-09-20T22:45:56.391 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.391 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.391 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:45:56.391 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:40.229625+0000", 2024-09-20T22:45:56.391 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.199e-06 2024-09-20T22:45:56.391 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.391 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.392 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:45:56.392 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:40.229631+0000", 2024-09-20T22:45:56.392 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.1599999999999997e-06 2024-09-20T22:45:56.392 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.392 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.392 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:45:56.392 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:40.229636+0000", 2024-09-20T22:45:56.392 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.7780000000000004e-06 2024-09-20T22:45:56.392 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.392 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.392 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:45:56.392 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:40.229830+0000", 2024-09-20T22:45:56.392 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00019321 2024-09-20T22:45:56.392 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.392 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.393 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for blocked object", 2024-09-20T22:45:56.393 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:40.229857+0000", 2024-09-20T22:45:56.393 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.7030999999999999e-05 2024-09-20T22:45:56.393 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.393 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.393 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:45:56.393 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:40.248577+0000", 2024-09-20T22:45:56.393 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.018720554 2024-09-20T22:45:56.393 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.393 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.393 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:45:56.393 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:40.248643+0000", 2024-09-20T22:45:56.393 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.5361e-05 2024-09-20T22:45:56.393 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.393 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.394 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:45:56.394 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:40.248721+0000", 2024-09-20T22:45:56.394 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.7985999999999995e-05 2024-09-20T22:45:56.394 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:56.394 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:45:56.394 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:56.394 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.394 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.394 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:1791 7.f 7:fa55e5ae:::smithi08477203-7:head [copy-from ver 0] snapc 46=[45,44,43,42,41,40,3e] ondisk+write+known_if_redirected+supports_pool_eio e229)", 2024-09-20T22:45:56.394 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:45:08.148595+0000", 2024-09-20T22:45:56.394 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 47.875820161, 2024-09-20T22:45:56.394 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011493375, 2024-09-20T22:45:56.394 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:45:56.394 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:45:56.395 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-20T22:45:56.395 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:45:56.395 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:45:56.395 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:45:56.395 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1791 2024-09-20T22:45:56.395 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.395 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:45:56.395 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.395 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:45:56.395 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.148595+0000", 2024-09-20T22:45:56.395 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:45:56.395 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.395 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.395 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:45:56.395 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.148595+0000", 2024-09-20T22:45:56.396 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:45:56.396 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.396 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.396 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:45:56.396 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.148597+0000", 2024-09-20T22:45:56.396 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.9649999999999998e-06 2024-09-20T22:45:56.396 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.396 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.396 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:45:56.396 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.148603+0000", 2024-09-20T22:45:56.396 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.2079999999999997e-06 2024-09-20T22:45:56.396 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.396 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.396 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:45:56.397 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.148615+0000", 2024-09-20T22:45:56.397 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1938e-05 2024-09-20T22:45:56.397 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.397 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.397 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:45:56.397 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.148623+0000", 2024-09-20T22:45:56.397 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.2930000000000001e-06 2024-09-20T22:45:56.397 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.397 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.397 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:45:56.397 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.148707+0000", 2024-09-20T22:45:56.397 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.3805000000000001e-05 2024-09-20T22:45:56.397 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.397 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.397 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:45:56.397 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.148816+0000", 2024-09-20T22:45:56.398 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000109048 2024-09-20T22:45:56.398 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.398 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.398 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:45:56.398 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.160088+0000", 2024-09-20T22:45:56.398 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011272117999999999 2024-09-20T22:45:56.398 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:56.398 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:45:56.398 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:56.398 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.398 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.398 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:1792 7.f 7:fa55e5ae:::smithi08477203-7:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e229)", 2024-09-20T22:45:56.398 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:45:08.148653+0000", 2024-09-20T22:45:56.398 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 47.875762295999998, 2024-09-20T22:45:56.399 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011699416000000001, 2024-09-20T22:45:56.399 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:45:56.399 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:45:56.399 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-20T22:45:56.399 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:45:56.399 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:45:56.399 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:45:56.399 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1792 2024-09-20T22:45:56.399 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.399 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:45:56.399 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.399 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:45:56.399 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.148653+0000", 2024-09-20T22:45:56.399 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:45:56.399 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.400 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.400 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:45:56.400 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.148653+0000", 2024-09-20T22:45:56.400 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:45:56.400 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.400 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.400 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:45:56.400 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.148653+0000", 2024-09-20T22:45:56.400 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.2699999999999999e-07 2024-09-20T22:45:56.400 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.400 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.400 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:45:56.400 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.148656+0000", 2024-09-20T22:45:56.400 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.7630000000000001e-06 2024-09-20T22:45:56.400 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.401 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.401 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:45:56.401 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.148668+0000", 2024-09-20T22:45:56.401 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2515e-05 2024-09-20T22:45:56.401 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.401 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.401 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:45:56.401 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.148679+0000", 2024-09-20T22:45:56.401 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0200999999999999e-05 2024-09-20T22:45:56.401 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.401 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.401 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:45:56.401 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.149057+0000", 2024-09-20T22:45:56.401 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00037865700000000002 2024-09-20T22:45:56.401 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.402 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.402 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for blocked object", 2024-09-20T22:45:56.402 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.149130+0000", 2024-09-20T22:45:56.402 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.2409000000000004e-05 2024-09-20T22:45:56.402 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.402 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.402 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:45:56.402 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.160178+0000", 2024-09-20T22:45:56.402 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011047916 2024-09-20T22:45:56.402 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.402 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.402 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:45:56.402 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.160274+0000", 2024-09-20T22:45:56.402 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.6138999999999996e-05 2024-09-20T22:45:56.403 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.403 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.403 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:45:56.403 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.160352+0000", 2024-09-20T22:45:56.403 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.8189000000000006e-05 2024-09-20T22:45:56.403 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:56.403 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:45:56.403 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:56.403 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.403 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.403 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:1809 7.16 7:6885a65f:::smithi08477203-14:head [delete] snapc 46=[45,44,43,42,41,40,3e] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e229)", 2024-09-20T22:45:56.403 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:45:08.177381+0000", 2024-09-20T22:45:56.403 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 47.847034055000002, 2024-09-20T22:45:56.403 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.01285646, 2024-09-20T22:45:56.403 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:45:56.404 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:45:56.404 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:45:56.404 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:45:56.404 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:45:56.404 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:45:56.404 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1809 2024-09-20T22:45:56.404 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.404 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:45:56.404 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.404 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:45:56.404 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.177381+0000", 2024-09-20T22:45:56.404 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:45:56.404 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.404 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.404 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:45:56.405 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.177381+0000", 2024-09-20T22:45:56.405 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:45:56.405 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.405 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.405 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:45:56.405 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.177383+0000", 2024-09-20T22:45:56.405 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.1399999999999998e-06 2024-09-20T22:45:56.405 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.405 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.405 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:45:56.405 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.177391+0000", 2024-09-20T22:45:56.405 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.7319999999999998e-06 2024-09-20T22:45:56.405 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.406 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.406 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:45:56.406 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.177415+0000", 2024-09-20T22:45:56.406 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.3879999999999998e-05 2024-09-20T22:45:56.406 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.406 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.406 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:45:56.406 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.177425+0000", 2024-09-20T22:45:56.406 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0509999999999999e-05 2024-09-20T22:45:56.406 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.406 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.406 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:45:56.406 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.177494+0000", 2024-09-20T22:45:56.407 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.8544e-05 2024-09-20T22:45:56.407 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.407 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.407 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:45:56.407 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.177590+0000", 2024-09-20T22:45:56.407 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.6729000000000002e-05 2024-09-20T22:45:56.407 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.407 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.407 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 2,7", 2024-09-20T22:45:56.407 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.177798+0000", 2024-09-20T22:45:56.407 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000207398 2024-09-20T22:45:56.407 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.407 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.407 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:45:56.407 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.179598+0000", 2024-09-20T22:45:56.408 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0018002160000000001 2024-09-20T22:45:56.408 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.408 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.408 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:45:56.408 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.186392+0000", 2024-09-20T22:45:56.408 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.006794423 2024-09-20T22:45:56.408 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.408 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.408 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:45:56.408 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.190106+0000", 2024-09-20T22:45:56.408 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0037136109999999999 2024-09-20T22:45:56.408 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.408 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.408 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:45:56.409 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.190180+0000", 2024-09-20T22:45:56.409 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.3831000000000005e-05 2024-09-20T22:45:56.409 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:45:56.409 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:45:56.409 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:45:56.409 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.190237+0000", 2024-09-20T22:45:56.409 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:56 smithi084 ceph-mon[40024]: 5.16 scrub starts 2024-09-20T22:45:56.409 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:56 smithi084 ceph-mon[40024]: 5.16 scrub ok 2024-09-20T22:45:56.409 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:56 smithi084 ceph-mon[40024]: 1.c scrub starts 2024-09-20T22:45:56.409 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:56 smithi084 ceph-mon[40024]: 1.c scrub ok 2024-09-20T22:45:56.410 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:56 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "16"}]: dispatch 2024-09-20T22:45:56.410 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:56 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "16"}]: dispatch 2024-09-20T22:45:56.410 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:56 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/690808353' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:45:56.410 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:56 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "16"}]': finished 2024-09-20T22:45:56.410 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:56 smithi084 ceph-mon[40024]: osdmap e277: 8 total, 8 up, 6 in 2024-09-20T22:45:56.410 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.7445999999999999e-05 2024-09-20T22:45:56.410 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:56.410 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:45:56.411 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:56.411 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:45:56.411 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:45:56.411 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-20T22:45:56.411 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:56 smithi084 ceph-mon[29774]: 5.16 scrub starts 2024-09-20T22:45:56.411 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:56 smithi084 ceph-mon[29774]: 5.16 scrub ok 2024-09-20T22:45:56.411 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:56 smithi084 ceph-mon[29774]: 1.c scrub starts 2024-09-20T22:45:56.411 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:56 smithi084 ceph-mon[29774]: 1.c scrub ok 2024-09-20T22:45:56.411 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:56 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "16"}]: dispatch 2024-09-20T22:45:56.411 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:56 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "16"}]: dispatch 2024-09-20T22:45:56.411 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:56 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/690808353' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:45:56.411 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:56 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "16"}]': finished 2024-09-20T22:45:56.412 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:56 smithi084 ceph-mon[29774]: osdmap e277: 8 total, 8 up, 6 in 2024-09-20T22:45:56.412 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:45:56.433 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:56.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:56.434 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:45:56.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:56 smithi154 ceph-mon[33264]: 5.16 scrub starts 2024-09-20T22:45:56.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:56 smithi154 ceph-mon[33264]: 5.16 scrub ok 2024-09-20T22:45:56.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:56 smithi154 ceph-mon[33264]: 1.c scrub starts 2024-09-20T22:45:56.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:56 smithi154 ceph-mon[33264]: 1.c scrub ok 2024-09-20T22:45:56.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:56 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "16"}]: dispatch 2024-09-20T22:45:56.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:56 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "16"}]: dispatch 2024-09-20T22:45:56.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:56 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/690808353' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:45:56.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:56 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "16"}]': finished 2024-09-20T22:45:56.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:56 smithi154 ceph-mon[33264]: osdmap e277: 8 total, 8 up, 6 in 2024-09-20T22:45:56.534 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:45:56.555 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:56.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:56.556 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:45:56.657 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:45:56.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:56.679 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:56.679 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:45:56.780 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:45:56.805 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:56.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:56.806 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:45:56.907 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:45:56.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:56.929 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:56.929 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:45:57.030 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:45:57.045 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:45:57.055 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:57.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:57.056 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:45:57.157 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:45:57.171 INFO:tasks.rados.rados.0.smithi084.stdout:963: done (0 left) 2024-09-20T22:45:57.171 INFO:tasks.rados.rados.0.smithi084.stdout:964: copy_from oid 22 from oid 47 current snap is 92 2024-09-20T22:45:57.171 INFO:tasks.rados.rados.0.smithi084.stdout:965: read oid 27 snap 91 2024-09-20T22:45:57.171 INFO:tasks.rados.rados.0.smithi084.stdout:965: expect deleted 2024-09-20T22:45:57.171 INFO:tasks.rados.rados.0.smithi084.stdout:966: write (excl) oid 44 current snap is 92 2024-09-20T22:45:57.171 INFO:tasks.rados.rados.0.smithi084.stdout:966: seq_num 359 ranges {433713=482982,1424565=609148} 2024-09-20T22:45:57.178 INFO:tasks.rados.rados.0.smithi084.stdout:966: writing smithi08477203-44 from 433713 to 916695 tid 1 2024-09-20T22:45:57.184 INFO:tasks.rados.rados.0.smithi084.stdout:966: writing smithi08477203-44 from 1424565 to 2033713 tid 2 2024-09-20T22:45:57.185 INFO:tasks.rados.rados.0.smithi084.stdout:967: write oid 10 current snap is 92 2024-09-20T22:45:57.185 INFO:tasks.rados.rados.0.smithi084.stdout:967: seq_num 360 ranges {450520=558231,1483047=515125} 2024-09-20T22:45:57.186 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:57.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:57.187 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:45:57.190 INFO:tasks.rados.rados.0.smithi084.stdout:967: writing smithi08477203-10 from 450520 to 1008751 tid 1 2024-09-20T22:45:57.196 INFO:tasks.rados.rados.0.smithi084.stdout:967: writing smithi08477203-10 from 1483047 to 1998172 tid 2 2024-09-20T22:45:57.196 INFO:tasks.rados.rados.0.smithi084.stdout:968: snap_remove snap 88 2024-09-20T22:45:57.287 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:45:57.309 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:57.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:57.310 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:45:57.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:57 smithi084 ceph-mon[29774]: pgmap v316: 174 pgs: 1 clean+premerge+peered, 4 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 166 active+clean; 201 MiB data, 697 MiB used, 536 GiB / 536 GiB avail; 17 MiB/s rd, 2.8 MiB/s wr, 25 op/s 2024-09-20T22:45:57.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:57 smithi084 ceph-mon[29774]: 5.1 scrub starts 2024-09-20T22:45:57.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:57 smithi084 ceph-mon[29774]: 5.1 scrub ok 2024-09-20T22:45:57.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:57 smithi084 ceph-mon[29774]: osdmap e278: 8 total, 8 up, 6 in 2024-09-20T22:45:57.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:57 smithi084 ceph-mon[40024]: pgmap v316: 174 pgs: 1 clean+premerge+peered, 4 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 166 active+clean; 201 MiB data, 697 MiB used, 536 GiB / 536 GiB avail; 17 MiB/s rd, 2.8 MiB/s wr, 25 op/s 2024-09-20T22:45:57.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:57 smithi084 ceph-mon[40024]: 5.1 scrub starts 2024-09-20T22:45:57.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:57 smithi084 ceph-mon[40024]: 5.1 scrub ok 2024-09-20T22:45:57.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:57 smithi084 ceph-mon[40024]: osdmap e278: 8 total, 8 up, 6 in 2024-09-20T22:45:57.411 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:45:57.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:57.433 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:57.433 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:45:57.493 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-20T22:45:57.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:57 smithi154 ceph-mon[33264]: pgmap v316: 174 pgs: 1 clean+premerge+peered, 4 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 166 active+clean; 201 MiB data, 697 MiB used, 536 GiB / 536 GiB avail; 17 MiB/s rd, 2.8 MiB/s wr, 25 op/s 2024-09-20T22:45:57.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:57 smithi154 ceph-mon[33264]: 5.1 scrub starts 2024-09-20T22:45:57.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:57 smithi154 ceph-mon[33264]: 5.1 scrub ok 2024-09-20T22:45:57.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:57 smithi154 ceph-mon[33264]: osdmap e278: 8 total, 8 up, 6 in 2024-09-20T22:45:57.534 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:45:57.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:57.558 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:57.559 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:45:57.659 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:45:57.681 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:57.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:57.682 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:45:57.782 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:45:57.803 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:57.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:57.804 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:45:57.905 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:45:57.926 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:57.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:57.927 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:45:57.945 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:45:57.964 INFO:teuthology.orchestra.run.smithi084.stderr:no valid command found; 10 closest matches: 2024-09-20T22:45:57.964 INFO:teuthology.orchestra.run.smithi084.stderr:0 2024-09-20T22:45:57.964 INFO:teuthology.orchestra.run.smithi084.stderr:1 2024-09-20T22:45:57.964 INFO:teuthology.orchestra.run.smithi084.stderr:2 2024-09-20T22:45:57.964 INFO:teuthology.orchestra.run.smithi084.stderr:abort 2024-09-20T22:45:57.964 INFO:teuthology.orchestra.run.smithi084.stderr:assert 2024-09-20T22:45:57.964 INFO:teuthology.orchestra.run.smithi084.stderr:bench [] [] [] [] 2024-09-20T22:45:57.964 INFO:teuthology.orchestra.run.smithi084.stderr:bluefs debug_inject_read_zeros 2024-09-20T22:45:57.964 INFO:teuthology.orchestra.run.smithi084.stderr:bluefs files list 2024-09-20T22:45:57.964 INFO:teuthology.orchestra.run.smithi084.stderr:bluefs stats 2024-09-20T22:45:57.965 INFO:teuthology.orchestra.run.smithi084.stderr:bluestore allocator dump block 2024-09-20T22:45:57.965 INFO:teuthology.orchestra.run.smithi084.stderr:Error EINVAL: invalid command 2024-09-20T22:45:58.028 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:45:58.051 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:58.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:58.052 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:45:58.153 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:45:58.164 INFO:tasks.rados.rados.0.smithi084.stdout:968: done (4 left) 2024-09-20T22:45:58.164 INFO:tasks.rados.rados.0.smithi084.stdout:969: snap_remove snap 69 2024-09-20T22:45:58.180 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:58.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:58.181 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:45:58.282 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:45:58.303 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:58.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:58.304 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:45:58.404 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:45:58.439 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:58.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:58.440 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:45:58.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:58 smithi154 ceph-mon[33264]: 3.b scrub starts 2024-09-20T22:45:58.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:58 smithi154 ceph-mon[33264]: 3.b scrub ok 2024-09-20T22:45:58.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:58 smithi154 ceph-mon[33264]: osdmap e279: 8 total, 8 up, 6 in 2024-09-20T22:45:58.526 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:58 smithi084 ceph-mon[40024]: 3.b scrub starts 2024-09-20T22:45:58.526 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:58 smithi084 ceph-mon[40024]: 3.b scrub ok 2024-09-20T22:45:58.526 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:58 smithi084 ceph-mon[40024]: osdmap e279: 8 total, 8 up, 6 in 2024-09-20T22:45:58.526 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:58 smithi084 ceph-mon[29774]: 3.b scrub starts 2024-09-20T22:45:58.527 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:58 smithi084 ceph-mon[29774]: 3.b scrub ok 2024-09-20T22:45:58.527 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:58 smithi084 ceph-mon[29774]: osdmap e279: 8 total, 8 up, 6 in 2024-09-20T22:45:58.527 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:45:58 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: ::ffff:172.21.15.154 - - [20/Sep/2024:22:45:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-20T22:45:58.540 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:45:58.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:58.563 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:58.563 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:45:58.656 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:45:58.664 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:45:58.686 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:58.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:58.687 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:45:58.788 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:45:58.809 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:58.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:58.810 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:45:58.911 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:45:58.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:58.934 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:58.934 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:45:59.035 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:45:59.056 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:59.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:59.057 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:45:59.059 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-20T22:45:59.139 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-20T22:45:59.139 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2024-09-20T22:45:59.139 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2024-09-20T22:45:59.139 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-20T22:45:59.159 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:45:59.164 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is unset 2024-09-20T22:45:59.165 INFO:tasks.rados.rados.0.smithi084.stdout:969: done (4 left) 2024-09-20T22:45:59.165 INFO:tasks.rados.rados.0.smithi084.stdout:970: snap_create 2024-09-20T22:45:59.165 INFO:tasks.rados.rados.0.smithi084.stdout:966: finishing write tid 1 to smithi08477203-44 2024-09-20T22:45:59.165 INFO:tasks.rados.rados.0.smithi084.stdout:964: finishing copy_from racing read to smithi08477203-22 2024-09-20T22:45:59.165 INFO:tasks.rados.rados.0.smithi084.stdout:964: finishing copy_from to smithi08477203-22 2024-09-20T22:45:59.165 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 22 v 174 (ObjNum 208 snap 42 seq_num 208) dirty exists 2024-09-20T22:45:59.165 INFO:tasks.rados.rados.0.smithi084.stdout:966: finishing write tid 2 to smithi08477203-44 2024-09-20T22:45:59.165 INFO:tasks.rados.rados.0.smithi084.stdout:966: finishing write tid 3 to smithi08477203-44 2024-09-20T22:45:59.165 INFO:tasks.rados.rados.0.smithi084.stdout:967: finishing write tid 1 to smithi08477203-10 2024-09-20T22:45:59.166 INFO:tasks.rados.rados.0.smithi084.stdout:966: finishing write tid 4 to smithi08477203-44 2024-09-20T22:45:59.166 INFO:tasks.rados.rados.0.smithi084.stdout:966: oid 44 updating version 0 to 501 2024-09-20T22:45:59.166 INFO:tasks.rados.rados.0.smithi084.stdout:966: oid 44 version 501 is already newer than 499 2024-09-20T22:45:59.166 INFO:tasks.rados.rados.0.smithi084.stdout:966: oid 44 version 501 is already newer than 500 2024-09-20T22:45:59.166 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 44 v 501 (ObjNum 359 snap 92 seq_num 359) dirty exists 2024-09-20T22:45:59.166 INFO:tasks.rados.rados.0.smithi084.stdout:966: left oid 44 (ObjNum 359 snap 92 seq_num 359) 2024-09-20T22:45:59.166 INFO:tasks.rados.rados.0.smithi084.stdout:967: finishing write tid 2 to smithi08477203-10 2024-09-20T22:45:59.166 INFO:tasks.rados.rados.0.smithi084.stdout:967: finishing write tid 3 to smithi08477203-10 2024-09-20T22:45:59.166 INFO:tasks.rados.rados.0.smithi084.stdout:967: finishing write tid 4 to smithi08477203-10 2024-09-20T22:45:59.166 INFO:tasks.rados.rados.0.smithi084.stdout:967: oid 10 updating version 0 to 258 2024-09-20T22:45:59.166 INFO:tasks.rados.rados.0.smithi084.stdout:967: oid 10 version 258 is already newer than 256 2024-09-20T22:45:59.166 INFO:tasks.rados.rados.0.smithi084.stdout:967: oid 10 version 258 is already newer than 257 2024-09-20T22:45:59.167 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 10 v 258 (ObjNum 360 snap 92 seq_num 360) dirty exists 2024-09-20T22:45:59.167 INFO:tasks.rados.rados.0.smithi084.stdout:967: left oid 10 (ObjNum 360 snap 92 seq_num 360) 2024-09-20T22:45:59.191 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:59.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:59.192 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:45:59.293 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:45:59.314 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:59.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:59.315 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:45:59.364 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:59 smithi084 ceph-mon[40024]: pgmap v319: 174 pgs: 1 clean+premerge+peered, 4 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 166 active+clean; 201 MiB data, 697 MiB used, 536 GiB / 536 GiB avail 2024-09-20T22:45:59.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:59 smithi084 ceph-mon[40024]: 5.3 scrub starts 2024-09-20T22:45:59.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:59 smithi084 ceph-mon[40024]: 5.3 scrub ok 2024-09-20T22:45:59.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:59 smithi084 ceph-mon[40024]: purged_snaps scrub starts 2024-09-20T22:45:59.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:59 smithi084 ceph-mon[40024]: purged_snaps scrub ok 2024-09-20T22:45:59.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:59 smithi084 ceph-mon[40024]: 6.6 scrub starts 2024-09-20T22:45:59.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:59 smithi084 ceph-mon[40024]: 6.6 scrub ok 2024-09-20T22:45:59.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:59 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/2452826550' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:45:59.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:59 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:45:59.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:59 smithi084 ceph-mon[40024]: 6.f scrub starts 2024-09-20T22:45:59.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:59 smithi084 ceph-mon[40024]: 6.f scrub ok 2024-09-20T22:45:59.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:59 smithi084 ceph-mon[40024]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T22:45:59.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:59 smithi084 ceph-mon[40024]: Cluster is now healthy 2024-09-20T22:45:59.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:59 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T22:45:59.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:45:59 smithi084 ceph-mon[40024]: osdmap e280: 8 total, 8 up, 6 in 2024-09-20T22:45:59.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:59 smithi084 ceph-mon[29774]: pgmap v319: 174 pgs: 1 clean+premerge+peered, 4 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 166 active+clean; 201 MiB data, 697 MiB used, 536 GiB / 536 GiB avail 2024-09-20T22:45:59.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:59 smithi084 ceph-mon[29774]: 5.3 scrub starts 2024-09-20T22:45:59.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:59 smithi084 ceph-mon[29774]: 5.3 scrub ok 2024-09-20T22:45:59.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:59 smithi084 ceph-mon[29774]: purged_snaps scrub starts 2024-09-20T22:45:59.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:59 smithi084 ceph-mon[29774]: purged_snaps scrub ok 2024-09-20T22:45:59.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:59 smithi084 ceph-mon[29774]: 6.6 scrub starts 2024-09-20T22:45:59.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:59 smithi084 ceph-mon[29774]: 6.6 scrub ok 2024-09-20T22:45:59.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:59 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/2452826550' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:45:59.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:59 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:45:59.367 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:59 smithi084 ceph-mon[29774]: 6.f scrub starts 2024-09-20T22:45:59.367 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:59 smithi084 ceph-mon[29774]: 6.f scrub ok 2024-09-20T22:45:59.367 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:59 smithi084 ceph-mon[29774]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T22:45:59.367 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:59 smithi084 ceph-mon[29774]: Cluster is now healthy 2024-09-20T22:45:59.367 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:59 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T22:45:59.367 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:45:59 smithi084 ceph-mon[29774]: osdmap e280: 8 total, 8 up, 6 in 2024-09-20T22:45:59.416 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:45:59.438 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:59.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:59.439 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:45:59.504 INFO:journalctl@ceph.iscsi.iscsi.a.smithi154.stdout:Sep 20 22:45:59 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-iscsi-iscsi-a[59730]: debug there is no tcmu-runner data available 2024-09-20T22:45:59.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:59 smithi154 ceph-mon[33264]: pgmap v319: 174 pgs: 1 clean+premerge+peered, 4 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 166 active+clean; 201 MiB data, 697 MiB used, 536 GiB / 536 GiB avail 2024-09-20T22:45:59.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:59 smithi154 ceph-mon[33264]: 5.3 scrub starts 2024-09-20T22:45:59.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:59 smithi154 ceph-mon[33264]: 5.3 scrub ok 2024-09-20T22:45:59.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:59 smithi154 ceph-mon[33264]: purged_snaps scrub starts 2024-09-20T22:45:59.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:59 smithi154 ceph-mon[33264]: purged_snaps scrub ok 2024-09-20T22:45:59.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:59 smithi154 ceph-mon[33264]: 6.6 scrub starts 2024-09-20T22:45:59.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:59 smithi154 ceph-mon[33264]: 6.6 scrub ok 2024-09-20T22:45:59.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:59 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/2452826550' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:45:59.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:59 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:45:59.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:59 smithi154 ceph-mon[33264]: 6.f scrub starts 2024-09-20T22:45:59.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:59 smithi154 ceph-mon[33264]: 6.f scrub ok 2024-09-20T22:45:59.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:59 smithi154 ceph-mon[33264]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T22:45:59.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:59 smithi154 ceph-mon[33264]: Cluster is now healthy 2024-09-20T22:45:59.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:59 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T22:45:59.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:45:59 smithi154 ceph-mon[33264]: osdmap e280: 8 total, 8 up, 6 in 2024-09-20T22:45:59.540 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:45:59.562 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:59.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:59.563 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:45:59.663 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:45:59.685 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:59.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:59.686 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:45:59.786 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.1 dump_blocked_ops 2024-09-20T22:45:59.786 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:45:59.807 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:45:59.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:59.808 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:45:59.909 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:45:59.931 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:45:59.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:45:59.932 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:46:00.033 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:46:00.054 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:00.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:00.056 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:46:00.157 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:46:00.164 INFO:tasks.rados.rados.0.smithi084.stdout:964: done (4 left) 2024-09-20T22:46:00.165 INFO:tasks.rados.rados.0.smithi084.stdout:965: done (3 left) 2024-09-20T22:46:00.165 INFO:tasks.rados.rados.0.smithi084.stdout:966: done (2 left) 2024-09-20T22:46:00.165 INFO:tasks.rados.rados.0.smithi084.stdout:967: done (1 left) 2024-09-20T22:46:00.165 INFO:tasks.rados.rados.0.smithi084.stdout:970: done (0 left) 2024-09-20T22:46:00.165 INFO:tasks.rados.rados.0.smithi084.stdout:971: write (excl) oid 15 current snap is 93 2024-09-20T22:46:00.165 INFO:tasks.rados.rados.0.smithi084.stdout:971: seq_num 361 ranges {467327=633480,1825176=242151} 2024-09-20T22:46:00.170 INFO:tasks.rados.rados.0.smithi084.stdout:971: writing smithi08477203-15 from 467327 to 1100807 tid 1 2024-09-20T22:46:00.177 INFO:tasks.rados.rados.0.smithi084.stdout:971: writing smithi08477203-15 from 1825176 to 2067327 tid 2 2024-09-20T22:46:00.177 INFO:tasks.rados.rados.0.smithi084.stdout:972: read oid 42 snap -1 2024-09-20T22:46:00.177 INFO:tasks.rados.rados.0.smithi084.stdout:972: expect deleted 2024-09-20T22:46:00.177 INFO:tasks.rados.rados.0.smithi084.stdout:973: write oid 41 current snap is 93 2024-09-20T22:46:00.177 INFO:tasks.rados.rados.0.smithi084.stdout:973: seq_num 362 ranges {484134=708729,1883658=200476} 2024-09-20T22:46:00.182 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:00.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:00.183 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:46:00.184 INFO:tasks.rados.rados.0.smithi084.stdout:973: writing smithi08477203-41 from 484134 to 1192863 tid 1 2024-09-20T22:46:00.189 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-20T22:46:00.190 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-20T22:46:00.190 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph tell osd.3 'trim stale osdmaps' 2024-09-20T22:46:00.191 INFO:tasks.rados.rados.0.smithi084.stdout:973: writing smithi08477203-41 from 1883658 to 2084134 tid 2 2024-09-20T22:46:00.191 INFO:tasks.rados.rados.0.smithi084.stdout:974: read oid 43 snap -1 2024-09-20T22:46:00.191 INFO:tasks.rados.rados.0.smithi084.stdout:974: expect deleted 2024-09-20T22:46:00.191 INFO:tasks.rados.rados.0.smithi084.stdout:975: write oid 49 current snap is 93 2024-09-20T22:46:00.191 INFO:tasks.rados.rados.0.smithi084.stdout:975: seq_num 363 ranges {500941=783978,1942140=158801} 2024-09-20T22:46:00.196 INFO:tasks.rados.rados.0.smithi084.stdout:975: writing smithi08477203-49 from 500941 to 1284919 tid 1 2024-09-20T22:46:00.202 INFO:tasks.rados.rados.0.smithi084.stdout:975: writing smithi08477203-49 from 1942140 to 2100941 tid 2 2024-09-20T22:46:00.202 INFO:tasks.rados.rados.0.smithi084.stdout:976: read oid 37 snap -1 2024-09-20T22:46:00.202 INFO:tasks.rados.rados.0.smithi084.stdout:976: expect (ObjNum 253 snap 59 seq_num 253) 2024-09-20T22:46:00.202 INFO:tasks.rados.rados.0.smithi084.stdout:971: finishing write tid 1 to smithi08477203-15 2024-09-20T22:46:00.202 INFO:tasks.rados.rados.0.smithi084.stdout:971: finishing write tid 2 to smithi08477203-15 2024-09-20T22:46:00.202 INFO:tasks.rados.rados.0.smithi084.stdout:971: finishing write tid 3 to smithi08477203-15 2024-09-20T22:46:00.202 INFO:tasks.rados.rados.0.smithi084.stdout:971: finishing write tid 4 to smithi08477203-15 2024-09-20T22:46:00.203 INFO:tasks.rados.rados.0.smithi084.stdout:971: oid 15 updating version 0 to 173 2024-09-20T22:46:00.203 INFO:tasks.rados.rados.0.smithi084.stdout:971: oid 15 updating version 173 to 174 2024-09-20T22:46:00.203 INFO:tasks.rados.rados.0.smithi084.stdout:971: oid 15 version 174 is already newer than 172 2024-09-20T22:46:00.203 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 15 v 174 (ObjNum 361 snap 93 seq_num 361) dirty exists 2024-09-20T22:46:00.203 INFO:tasks.rados.rados.0.smithi084.stdout:971: left oid 15 (ObjNum 361 snap 93 seq_num 361) 2024-09-20T22:46:00.203 INFO:tasks.rados.rados.0.smithi084.stdout:973: finishing write tid 1 to smithi08477203-41 2024-09-20T22:46:00.203 INFO:tasks.rados.rados.0.smithi084.stdout:973: finishing write tid 2 to smithi08477203-41 2024-09-20T22:46:00.203 INFO:tasks.rados.rados.0.smithi084.stdout:973: finishing write tid 3 to smithi08477203-41 2024-09-20T22:46:00.203 INFO:tasks.rados.rados.0.smithi084.stdout:973: finishing write tid 4 to smithi08477203-41 2024-09-20T22:46:00.203 INFO:tasks.rados.rados.0.smithi084.stdout:973: oid 41 updating version 0 to 71 2024-09-20T22:46:00.203 INFO:tasks.rados.rados.0.smithi084.stdout:973: oid 41 version 71 is already newer than 70 2024-09-20T22:46:00.203 INFO:tasks.rados.rados.0.smithi084.stdout:973: oid 41 updating version 71 to 72 2024-09-20T22:46:00.203 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 41 v 72 (ObjNum 362 snap 93 seq_num 362) dirty exists 2024-09-20T22:46:00.203 INFO:tasks.rados.rados.0.smithi084.stdout:973: left oid 41 (ObjNum 362 snap 93 seq_num 362) 2024-09-20T22:46:00.204 INFO:tasks.rados.rados.0.smithi084.stdout:975: finishing write tid 1 to smithi08477203-49 2024-09-20T22:46:00.204 INFO:tasks.rados.rados.0.smithi084.stdout:971: done (5 left) 2024-09-20T22:46:00.204 INFO:tasks.rados.rados.0.smithi084.stdout:972: done (4 left) 2024-09-20T22:46:00.204 INFO:tasks.rados.rados.0.smithi084.stdout:973: done (3 left) 2024-09-20T22:46:00.204 INFO:tasks.rados.rados.0.smithi084.stdout:974: done (2 left) 2024-09-20T22:46:00.204 INFO:tasks.rados.rados.0.smithi084.stdout:977: rollback oid 3 current snap is 93 2024-09-20T22:46:00.204 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 3 to 91 2024-09-20T22:46:00.204 INFO:tasks.rados.rados.0.smithi084.stdout:978: copy_from oid 50 from oid 18 current snap is 93 2024-09-20T22:46:00.204 INFO:tasks.rados.rados.0.smithi084.stdout:979: delete oid 5 current snap is 93 2024-09-20T22:46:00.204 INFO:tasks.rados.rados.0.smithi084.stdout:975: finishing write tid 2 to smithi08477203-49 2024-09-20T22:46:00.205 INFO:tasks.rados.rados.0.smithi084.stdout:975: finishing write tid 3 to smithi08477203-49 2024-09-20T22:46:00.205 INFO:tasks.rados.rados.0.smithi084.stdout:979: done (4 left) 2024-09-20T22:46:00.205 INFO:tasks.rados.rados.0.smithi084.stdout:980: snap_remove snap 78 2024-09-20T22:46:00.283 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:46:00.304 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:00.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:00.305 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:46:00.406 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:46:00.428 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:00.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:00.429 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:46:00.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:00 smithi154 ceph-mon[33264]: 4.1e scrub starts 2024-09-20T22:46:00.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:00 smithi154 ceph-mon[33264]: 4.1e scrub ok 2024-09-20T22:46:00.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:00 smithi154 ceph-mon[33264]: 1.1 scrub starts 2024-09-20T22:46:00.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:00 smithi154 ceph-mon[33264]: 1.1 scrub ok 2024-09-20T22:46:00.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:00 smithi154 ceph-mon[33264]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:46:00.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:00 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "15"}]: dispatch 2024-09-20T22:46:00.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:00 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "15"}]: dispatch 2024-09-20T22:46:00.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:00 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "15"}]': finished 2024-09-20T22:46:00.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:00 smithi154 ceph-mon[33264]: osdmap e281: 8 total, 8 up, 6 in 2024-09-20T22:46:00.530 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:46:00.539 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:00 smithi084 ceph-mon[40024]: 4.1e scrub starts 2024-09-20T22:46:00.539 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:00 smithi084 ceph-mon[40024]: 4.1e scrub ok 2024-09-20T22:46:00.540 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:00 smithi084 ceph-mon[40024]: 1.1 scrub starts 2024-09-20T22:46:00.540 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:00 smithi084 ceph-mon[40024]: 1.1 scrub ok 2024-09-20T22:46:00.540 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:00 smithi084 ceph-mon[40024]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:46:00.540 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:00 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "15"}]: dispatch 2024-09-20T22:46:00.540 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:00 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "15"}]: dispatch 2024-09-20T22:46:00.540 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:00 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "15"}]': finished 2024-09-20T22:46:00.540 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:00 smithi084 ceph-mon[40024]: osdmap e281: 8 total, 8 up, 6 in 2024-09-20T22:46:00.540 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:00 smithi084 ceph-mon[29774]: 4.1e scrub starts 2024-09-20T22:46:00.540 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:00 smithi084 ceph-mon[29774]: 4.1e scrub ok 2024-09-20T22:46:00.540 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:00 smithi084 ceph-mon[29774]: 1.1 scrub starts 2024-09-20T22:46:00.540 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:00 smithi084 ceph-mon[29774]: 1.1 scrub ok 2024-09-20T22:46:00.541 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:00 smithi084 ceph-mon[29774]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:46:00.541 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:00 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "15"}]: dispatch 2024-09-20T22:46:00.541 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:00 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "15"}]: dispatch 2024-09-20T22:46:00.541 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:00 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "15"}]': finished 2024-09-20T22:46:00.541 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:00 smithi084 ceph-mon[29774]: osdmap e281: 8 total, 8 up, 6 in 2024-09-20T22:46:00.552 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:00.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:00.553 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:46:00.653 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:46:00.675 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:00.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:00.676 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:46:00.777 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:46:00.798 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:00.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:00.799 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:46:00.889 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:46:00.897 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:46:00.900 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:46:00.926 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:46:00.926 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:00.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:00.927 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:46:01.028 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:46:01.050 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:01.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:01.051 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:46:01.152 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:46:01.165 INFO:tasks.rados.rados.0.smithi084.stdout:980: done (4 left) 2024-09-20T22:46:01.165 INFO:tasks.rados.rados.0.smithi084.stdout:981: read oid 41 snap -1 2024-09-20T22:46:01.165 INFO:tasks.rados.rados.0.smithi084.stdout:981: expect (ObjNum 362 snap 93 seq_num 362) 2024-09-20T22:46:01.165 INFO:tasks.rados.rados.0.smithi084.stdout:982: delete oid 6 current snap is 93 2024-09-20T22:46:01.165 INFO:tasks.rados.rados.0.smithi084.stdout:975: finishing write tid 4 to smithi08477203-49 2024-09-20T22:46:01.166 INFO:tasks.rados.rados.0.smithi084.stdout:975: oid 49 updating version 0 to 277 2024-09-20T22:46:01.166 INFO:tasks.rados.rados.0.smithi084.stdout:975: oid 49 version 277 is already newer than 275 2024-09-20T22:46:01.166 INFO:tasks.rados.rados.0.smithi084.stdout:975: oid 49 version 277 is already newer than 276 2024-09-20T22:46:01.166 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 49 v 277 (ObjNum 363 snap 93 seq_num 363) dirty exists 2024-09-20T22:46:01.166 INFO:tasks.rados.rados.0.smithi084.stdout:975: left oid 49 (ObjNum 363 snap 93 seq_num 363) 2024-09-20T22:46:01.166 INFO:tasks.rados.rados.0.smithi084.stdout:977: finishing rollback tid 0 to smithi08477203-3 2024-09-20T22:46:01.166 INFO:tasks.rados.rados.0.smithi084.stdout:977: finishing rollback tid 1 to smithi08477203-3 2024-09-20T22:46:01.166 INFO:tasks.rados.rados.0.smithi084.stdout:977: finishing rollback tid 2 to smithi08477203-3 2024-09-20T22:46:01.166 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 3 v 163 (ObjNum 355 snap 90 seq_num 355) dirty exists 2024-09-20T22:46:01.166 INFO:tasks.rados.rados.0.smithi084.stdout:976: expect (ObjNum 253 snap 59 seq_num 253) 2024-09-20T22:46:01.178 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:01.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:01.179 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:46:01.280 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:46:01.301 INFO:tasks.rados.rados.0.smithi084.stdout:978: finishing copy_from racing read to smithi08477203-50 2024-09-20T22:46:01.301 INFO:tasks.rados.rados.0.smithi084.stdout:978: finishing copy_from to smithi08477203-50 2024-09-20T22:46:01.301 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 50 v 236 (ObjNum 273 snap 69 seq_num 273) dirty exists 2024-09-20T22:46:01.301 INFO:tasks.rados.rados.0.smithi084.stdout:975: done (5 left) 2024-09-20T22:46:01.301 INFO:tasks.rados.rados.0.smithi084.stdout:976: done (4 left) 2024-09-20T22:46:01.301 INFO:tasks.rados.rados.0.smithi084.stdout:977: done (3 left) 2024-09-20T22:46:01.301 INFO:tasks.rados.rados.0.smithi084.stdout:978: done (2 left) 2024-09-20T22:46:01.301 INFO:tasks.rados.rados.0.smithi084.stdout:982: done (1 left) 2024-09-20T22:46:01.301 INFO:tasks.rados.rados.0.smithi084.stdout:983: snap_remove snap 91 2024-09-20T22:46:01.302 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:01.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:01.303 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:46:01.381 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:01 smithi084 ceph-mon[40024]: 6.15 scrub starts 2024-09-20T22:46:01.381 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:01 smithi084 ceph-mon[40024]: 6.15 scrub ok 2024-09-20T22:46:01.381 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:01 smithi084 ceph-mon[40024]: pgmap v322: 173 pgs: 1 peering, 6 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 161 active+clean; 204 MiB data, 709 MiB used, 536 GiB / 536 GiB avail; 5.8 MiB/s rd, 2.7 MiB/s wr, 16 op/s 2024-09-20T22:46:01.381 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:01 smithi084 ceph-mon[40024]: 2.1 scrub starts 2024-09-20T22:46:01.381 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:01 smithi084 ceph-mon[40024]: 2.1 scrub ok 2024-09-20T22:46:01.381 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:01 smithi084 ceph-mon[40024]: 3.a scrub starts 2024-09-20T22:46:01.381 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:01 smithi084 ceph-mon[40024]: 3.a scrub ok 2024-09-20T22:46:01.381 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:01 smithi084 ceph-mon[40024]: osdmap e282: 8 total, 8 up, 6 in 2024-09-20T22:46:01.382 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:01 smithi084 ceph-mon[29774]: 6.15 scrub starts 2024-09-20T22:46:01.382 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:01 smithi084 ceph-mon[29774]: 6.15 scrub ok 2024-09-20T22:46:01.382 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:01 smithi084 ceph-mon[29774]: pgmap v322: 173 pgs: 1 peering, 6 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 161 active+clean; 204 MiB data, 709 MiB used, 536 GiB / 536 GiB avail; 5.8 MiB/s rd, 2.7 MiB/s wr, 16 op/s 2024-09-20T22:46:01.382 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:01 smithi084 ceph-mon[29774]: 2.1 scrub starts 2024-09-20T22:46:01.382 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:01 smithi084 ceph-mon[29774]: 2.1 scrub ok 2024-09-20T22:46:01.382 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:01 smithi084 ceph-mon[29774]: 3.a scrub starts 2024-09-20T22:46:01.382 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:01 smithi084 ceph-mon[29774]: 3.a scrub ok 2024-09-20T22:46:01.382 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:01 smithi084 ceph-mon[29774]: osdmap e282: 8 total, 8 up, 6 in 2024-09-20T22:46:01.404 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:46:01.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:01.427 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:01.427 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:46:01.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:01 smithi154 ceph-mon[33264]: 6.15 scrub starts 2024-09-20T22:46:01.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:01 smithi154 ceph-mon[33264]: 6.15 scrub ok 2024-09-20T22:46:01.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:01 smithi154 ceph-mon[33264]: pgmap v322: 173 pgs: 1 peering, 6 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 161 active+clean; 204 MiB data, 709 MiB used, 536 GiB / 536 GiB avail; 5.8 MiB/s rd, 2.7 MiB/s wr, 16 op/s 2024-09-20T22:46:01.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:01 smithi154 ceph-mon[33264]: 2.1 scrub starts 2024-09-20T22:46:01.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:01 smithi154 ceph-mon[33264]: 2.1 scrub ok 2024-09-20T22:46:01.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:01 smithi154 ceph-mon[33264]: 3.a scrub starts 2024-09-20T22:46:01.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:01 smithi154 ceph-mon[33264]: 3.a scrub ok 2024-09-20T22:46:01.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:01 smithi154 ceph-mon[33264]: osdmap e282: 8 total, 8 up, 6 in 2024-09-20T22:46:01.528 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:46:01.552 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:01.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:01.553 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:46:01.564 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-20T22:46:01.565 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2024-09-20T22:46:01.565 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2024-09-20T22:46:01.565 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2024-09-20T22:46:01.565 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-20T22:46:01.654 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:46:01.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:01.678 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:01.678 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:46:01.773 INFO:teuthology.orchestra.run.smithi084.stderr:no valid command found; 10 closest matches: 2024-09-20T22:46:01.773 INFO:teuthology.orchestra.run.smithi084.stderr:0 2024-09-20T22:46:01.773 INFO:teuthology.orchestra.run.smithi084.stderr:1 2024-09-20T22:46:01.773 INFO:teuthology.orchestra.run.smithi084.stderr:2 2024-09-20T22:46:01.773 INFO:teuthology.orchestra.run.smithi084.stderr:abort 2024-09-20T22:46:01.773 INFO:teuthology.orchestra.run.smithi084.stderr:assert 2024-09-20T22:46:01.773 INFO:teuthology.orchestra.run.smithi084.stderr:bench [] [] [] [] 2024-09-20T22:46:01.773 INFO:teuthology.orchestra.run.smithi084.stderr:bluefs debug_inject_read_zeros 2024-09-20T22:46:01.773 INFO:teuthology.orchestra.run.smithi084.stderr:bluefs files list 2024-09-20T22:46:01.773 INFO:teuthology.orchestra.run.smithi084.stderr:bluefs stats 2024-09-20T22:46:01.774 INFO:teuthology.orchestra.run.smithi084.stderr:bluestore allocator dump block 2024-09-20T22:46:01.774 INFO:teuthology.orchestra.run.smithi084.stderr:Error EINVAL: invalid command 2024-09-20T22:46:01.779 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:46:01.801 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:01.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:01.802 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:46:01.903 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:46:01.924 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:01.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:01.925 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:46:02.025 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:46:02.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:02.048 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:02.048 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:46:02.149 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:46:02.167 INFO:tasks.rados.rados.0.smithi084.stdout:983: done (1 left) 2024-09-20T22:46:02.167 INFO:tasks.rados.rados.0.smithi084.stdout:984: delete oid 11 current snap is 93 2024-09-20T22:46:02.167 INFO:tasks.rados.rados.0.smithi084.stdout:981: expect (ObjNum 362 snap 93 seq_num 362) 2024-09-20T22:46:02.175 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:02.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:02.176 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:46:02.277 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:46:02.298 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:02.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:02.300 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:46:02.317 INFO:tasks.rados.rados.0.smithi084.stdout:981: done (1 left) 2024-09-20T22:46:02.317 INFO:tasks.rados.rados.0.smithi084.stdout:984: done (0 left) 2024-09-20T22:46:02.318 INFO:tasks.rados.rados.0.smithi084.stdout:985: snap_create 2024-09-20T22:46:02.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:02 smithi084 ceph-mon[29774]: 3.1d deep-scrub starts 2024-09-20T22:46:02.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:02 smithi084 ceph-mon[29774]: 3.1d deep-scrub ok 2024-09-20T22:46:02.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:02 smithi084 ceph-mon[29774]: 5.8 deep-scrub starts 2024-09-20T22:46:02.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:02 smithi084 ceph-mon[29774]: 5.8 deep-scrub ok 2024-09-20T22:46:02.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:02 smithi084 ceph-mon[29774]: 7.18 scrub starts 2024-09-20T22:46:02.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:02 smithi084 ceph-mon[29774]: 7.18 scrub ok 2024-09-20T22:46:02.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:02 smithi084 ceph-mon[29774]: osdmap e283: 8 total, 8 up, 6 in 2024-09-20T22:46:02.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:02 smithi084 ceph-mon[40024]: 3.1d deep-scrub starts 2024-09-20T22:46:02.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:02 smithi084 ceph-mon[40024]: 3.1d deep-scrub ok 2024-09-20T22:46:02.367 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:02 smithi084 ceph-mon[40024]: 5.8 deep-scrub starts 2024-09-20T22:46:02.367 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:02 smithi084 ceph-mon[40024]: 5.8 deep-scrub ok 2024-09-20T22:46:02.367 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:02 smithi084 ceph-mon[40024]: 7.18 scrub starts 2024-09-20T22:46:02.367 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:02 smithi084 ceph-mon[40024]: 7.18 scrub ok 2024-09-20T22:46:02.367 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:02 smithi084 ceph-mon[40024]: osdmap e283: 8 total, 8 up, 6 in 2024-09-20T22:46:02.402 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:46:02.423 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:02.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:02.424 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:46:02.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:02 smithi154 ceph-mon[33264]: 3.1d deep-scrub starts 2024-09-20T22:46:02.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:02 smithi154 ceph-mon[33264]: 3.1d deep-scrub ok 2024-09-20T22:46:02.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:02 smithi154 ceph-mon[33264]: 5.8 deep-scrub starts 2024-09-20T22:46:02.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:02 smithi154 ceph-mon[33264]: 5.8 deep-scrub ok 2024-09-20T22:46:02.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:02 smithi154 ceph-mon[33264]: 7.18 scrub starts 2024-09-20T22:46:02.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:02 smithi154 ceph-mon[33264]: 7.18 scrub ok 2024-09-20T22:46:02.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:02 smithi154 ceph-mon[33264]: osdmap e283: 8 total, 8 up, 6 in 2024-09-20T22:46:02.525 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:46:02.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:02.547 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:02.548 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:46:02.648 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:46:02.669 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:02.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:02.670 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:46:02.771 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:46:02.795 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:02.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:02.796 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:46:02.858 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd set noscrub 2024-09-20T22:46:02.896 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:46:02.902 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.1 dump_historic_ops 2024-09-20T22:46:02.963 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:02.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:02.964 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:46:03.065 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:46:03.091 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:03.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:03.092 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:46:03.163 INFO:tasks.rados.rados.0.smithi084.stdout:985: done (0 left) 2024-09-20T22:46:03.163 INFO:tasks.rados.rados.0.smithi084.stdout:986: copy_from oid 17 from oid 11 current snap is 94 2024-09-20T22:46:03.163 INFO:tasks.rados.rados.0.smithi084.stdout:987: delete oid 3 current snap is 94 2024-09-20T22:46:03.168 INFO:tasks.rados.rados.0.smithi084.stdout:986: finishing copy_from to smithi08477203-17 2024-09-20T22:46:03.168 INFO:tasks.rados.rados.0.smithi084.stdout:986: got expected ENOENT (src dne) 2024-09-20T22:46:03.169 INFO:tasks.rados.rados.0.smithi084.stdout:986: finishing copy_from racing read to smithi08477203-17 2024-09-20T22:46:03.170 INFO:tasks.rados.rados.0.smithi084.stdout:986: done (1 left) 2024-09-20T22:46:03.170 INFO:tasks.rados.rados.0.smithi084.stdout:987: done (0 left) 2024-09-20T22:46:03.170 INFO:tasks.rados.rados.0.smithi084.stdout:988: write (excl) oid 28 current snap is 94 2024-09-20T22:46:03.171 INFO:tasks.rados.rados.0.smithi084.stdout:988: seq_num 368 ranges {584976=476576,1834550=350426} 2024-09-20T22:46:03.176 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:46:03.176 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:46:03.176 INFO:tasks.rados.rados.0.smithi084.stdout:988: writing smithi08477203-28 from 584976 to 1061552 tid 1 2024-09-20T22:46:03.183 INFO:tasks.rados.rados.0.smithi084.stdout:988: writing smithi08477203-28 from 1834550 to 2184976 tid 2 2024-09-20T22:46:03.183 INFO:tasks.rados.rados.0.smithi084.stdout:989: copy_from oid 9 from oid 5 current snap is 94 2024-09-20T22:46:03.183 INFO:tasks.rados.rados.0.smithi084.stdout:988: finishing write tid 1 to smithi08477203-28 2024-09-20T22:46:03.183 INFO:tasks.rados.rados.0.smithi084.stdout:990: snap_create 2024-09-20T22:46:03.183 INFO:tasks.rados.rados.0.smithi084.stdout:989: finishing copy_from to smithi08477203-9 2024-09-20T22:46:03.183 INFO:tasks.rados.rados.0.smithi084.stdout:989: got expected ENOENT (src dne) 2024-09-20T22:46:03.184 INFO:tasks.rados.rados.0.smithi084.stdout:989: finishing copy_from racing read to smithi08477203-9 2024-09-20T22:46:03.185 INFO:tasks.rados.rados.0.smithi084.stdout:988: finishing write tid 2 to smithi08477203-28 2024-09-20T22:46:03.185 INFO:tasks.rados.rados.0.smithi084.stdout:988: finishing write tid 3 to smithi08477203-28 2024-09-20T22:46:03.185 INFO:tasks.rados.rados.0.smithi084.stdout:988: finishing write tid 4 to smithi08477203-28 2024-09-20T22:46:03.185 INFO:tasks.rados.rados.0.smithi084.stdout:988: oid 28 updating version 0 to 533 2024-09-20T22:46:03.185 INFO:tasks.rados.rados.0.smithi084.stdout:988: oid 28 version 533 is already newer than 532 2024-09-20T22:46:03.186 INFO:tasks.rados.rados.0.smithi084.stdout:988: oid 28 version 533 is already newer than 531 2024-09-20T22:46:03.186 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 28 v 533 (ObjNum 368 snap 94 seq_num 368) dirty exists 2024-09-20T22:46:03.186 INFO:tasks.rados.rados.0.smithi084.stdout:988: left oid 28 (ObjNum 368 snap 94 seq_num 368) 2024-09-20T22:46:03.193 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:46:03.194 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:46:03.194 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:46:03.214 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:46:03.215 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:46:03.216 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:03.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:03.218 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:46:03.230 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-20T22:46:03.230 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T22:46:03.248 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:46:03.248 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:46:03.266 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:46:03.266 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:46:03.287 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:46:03.287 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:46:03.311 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:46:03.311 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:46:03.318 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:46:03.339 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:03.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:03.340 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:46:03.441 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:46:03.445 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:03 smithi084 ceph-mon[40024]: pgmap v325: 173 pgs: 1 peering, 6 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 161 active+clean; 204 MiB data, 709 MiB used, 536 GiB / 536 GiB avail; 5.8 MiB/s rd, 2.7 MiB/s wr, 16 op/s 2024-09-20T22:46:03.445 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:03 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/3527000577' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:46:03.445 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:03 smithi084 ceph-mon[40024]: osdmap e284: 8 total, 8 up, 6 in 2024-09-20T22:46:03.445 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:03 smithi084 ceph-mon[29774]: pgmap v325: 173 pgs: 1 peering, 6 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 161 active+clean; 204 MiB data, 709 MiB used, 536 GiB / 536 GiB avail; 5.8 MiB/s rd, 2.7 MiB/s wr, 16 op/s 2024-09-20T22:46:03.445 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:03 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/3527000577' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:46:03.445 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:03 smithi084 ceph-mon[29774]: osdmap e284: 8 total, 8 up, 6 in 2024-09-20T22:46:03.464 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:03.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:03.465 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:46:03.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:03 smithi154 ceph-mon[33264]: pgmap v325: 173 pgs: 1 peering, 6 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 161 active+clean; 204 MiB data, 709 MiB used, 536 GiB / 536 GiB avail; 5.8 MiB/s rd, 2.7 MiB/s wr, 16 op/s 2024-09-20T22:46:03.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:03 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/3527000577' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:46:03.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:03 smithi154 ceph-mon[33264]: osdmap e284: 8 total, 8 up, 6 in 2024-09-20T22:46:03.566 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:46:03.587 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:03.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:03.588 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:46:03.679 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-20T22:46:03.679 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-20T22:46:03.679 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph tell osd.5 'trim stale osdmaps' 2024-09-20T22:46:03.690 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:46:03.712 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:03.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:03.713 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:46:03.814 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:46:03.836 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:03.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:03.837 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:46:03.938 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:46:03.951 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:46:03.951 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:46:03.960 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:03.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:03.961 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:46:04.062 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:46:04.083 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:04.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:04.084 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:46:04.165 INFO:tasks.rados.rados.0.smithi084.stdout:988: done (2 left) 2024-09-20T22:46:04.166 INFO:tasks.rados.rados.0.smithi084.stdout:989: done (1 left) 2024-09-20T22:46:04.166 INFO:tasks.rados.rados.0.smithi084.stdout:990: done (0 left) 2024-09-20T22:46:04.166 INFO:tasks.rados.rados.0.smithi084.stdout:991: copy_from oid 37 from oid 31 current snap is 95 2024-09-20T22:46:04.166 INFO:tasks.rados.rados.0.smithi084.stdout:992: read oid 19 snap -1 2024-09-20T22:46:04.166 INFO:tasks.rados.rados.0.smithi084.stdout:992: expect (ObjNum 258 snap 60 seq_num 258) 2024-09-20T22:46:04.166 INFO:tasks.rados.rados.0.smithi084.stdout:993: snap_create 2024-09-20T22:46:04.178 INFO:tasks.rados.rados.0.smithi084.stdout:992: expect (ObjNum 258 snap 60 seq_num 258) 2024-09-20T22:46:04.185 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:46:04.207 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:04.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:04.208 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:46:04.309 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:46:04.326 INFO:tasks.rados.rados.0.smithi084.stdout:991: finishing copy_from racing read to smithi08477203-37 2024-09-20T22:46:04.327 INFO:tasks.rados.rados.0.smithi084.stdout:991: finishing copy_from to smithi08477203-37 2024-09-20T22:46:04.327 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 37 v 207 (ObjNum 329 snap 84 seq_num 329) dirty exists 2024-09-20T22:46:04.330 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:04.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:04.331 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:46:04.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:04 smithi084 ceph-mon[40024]: 7.0 scrub starts 2024-09-20T22:46:04.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:04 smithi084 ceph-mon[40024]: 7.0 scrub ok 2024-09-20T22:46:04.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:04 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "14"}]: dispatch 2024-09-20T22:46:04.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:04 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "14"}]: dispatch 2024-09-20T22:46:04.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:04 smithi084 ceph-mon[40024]: 4.7 scrub starts 2024-09-20T22:46:04.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:04 smithi084 ceph-mon[40024]: 4.7 scrub ok 2024-09-20T22:46:04.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:04 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "14"}]': finished 2024-09-20T22:46:04.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:04 smithi084 ceph-mon[40024]: osdmap e285: 8 total, 8 up, 6 in 2024-09-20T22:46:04.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:04 smithi084 ceph-mon[29774]: 7.0 scrub starts 2024-09-20T22:46:04.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:04 smithi084 ceph-mon[29774]: 7.0 scrub ok 2024-09-20T22:46:04.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:04 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "14"}]: dispatch 2024-09-20T22:46:04.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:04 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "14"}]: dispatch 2024-09-20T22:46:04.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:04 smithi084 ceph-mon[29774]: 4.7 scrub starts 2024-09-20T22:46:04.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:04 smithi084 ceph-mon[29774]: 4.7 scrub ok 2024-09-20T22:46:04.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:04 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "14"}]': finished 2024-09-20T22:46:04.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:04 smithi084 ceph-mon[29774]: osdmap e285: 8 total, 8 up, 6 in 2024-09-20T22:46:04.431 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:46:04.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:04.453 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:04.454 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:46:04.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:04 smithi154 ceph-mon[33264]: 7.0 scrub starts 2024-09-20T22:46:04.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:04 smithi154 ceph-mon[33264]: 7.0 scrub ok 2024-09-20T22:46:04.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:04 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "14"}]: dispatch 2024-09-20T22:46:04.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:04 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "14"}]: dispatch 2024-09-20T22:46:04.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:04 smithi154 ceph-mon[33264]: 4.7 scrub starts 2024-09-20T22:46:04.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:04 smithi154 ceph-mon[33264]: 4.7 scrub ok 2024-09-20T22:46:04.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:04 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "14"}]': finished 2024-09-20T22:46:04.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:04 smithi154 ceph-mon[33264]: osdmap e285: 8 total, 8 up, 6 in 2024-09-20T22:46:04.554 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:46:04.576 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:04.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:04.577 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:46:04.678 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:46:04.698 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:04.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:04.699 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:46:04.800 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:46:04.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:04.822 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:04.822 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:46:04.923 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:46:04.948 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:04.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:04.949 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:46:05.017 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-20T22:46:05.017 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:46:05.050 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:46:05.073 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:05.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:05.074 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:46:05.167 INFO:tasks.rados.rados.0.smithi084.stdout:991: done (2 left) 2024-09-20T22:46:05.167 INFO:tasks.rados.rados.0.smithi084.stdout:992: done (1 left) 2024-09-20T22:46:05.167 INFO:tasks.rados.rados.0.smithi084.stdout:993: done (0 left) 2024-09-20T22:46:05.167 INFO:tasks.rados.rados.0.smithi084.stdout:994: rollback oid 27 current snap is 96 2024-09-20T22:46:05.167 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 27 to 92 2024-09-20T22:46:05.167 INFO:tasks.rados.rados.0.smithi084.stdout:995: snap_remove snap 95 2024-09-20T22:46:05.175 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:46:05.197 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:05.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:05.198 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:46:05.299 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:46:05.320 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:05.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:05.321 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:46:05.422 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:46:05.446 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:05.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:05.447 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:46:05.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:05 smithi154 ceph-mon[33264]: 7.6 scrub starts 2024-09-20T22:46:05.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:05 smithi154 ceph-mon[33264]: 7.6 scrub ok 2024-09-20T22:46:05.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:05 smithi154 ceph-mon[33264]: pgmap v328: 172 pgs: 6 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 162 active+clean; 133 MiB data, 567 MiB used, 536 GiB / 536 GiB avail; 5.0 MiB/s rd, 1.5 MiB/s wr, 13 op/s 2024-09-20T22:46:05.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:05 smithi154 ceph-mon[33264]: 4.5 scrub starts 2024-09-20T22:46:05.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:05 smithi154 ceph-mon[33264]: 4.5 scrub ok 2024-09-20T22:46:05.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:05 smithi154 ceph-mon[33264]: osdmap e286: 8 total, 8 up, 6 in 2024-09-20T22:46:05.542 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:05 smithi084 ceph-mon[40024]: 7.6 scrub starts 2024-09-20T22:46:05.542 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:05 smithi084 ceph-mon[40024]: 7.6 scrub ok 2024-09-20T22:46:05.542 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:05 smithi084 ceph-mon[40024]: pgmap v328: 172 pgs: 6 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 162 active+clean; 133 MiB data, 567 MiB used, 536 GiB / 536 GiB avail; 5.0 MiB/s rd, 1.5 MiB/s wr, 13 op/s 2024-09-20T22:46:05.542 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:05 smithi084 ceph-mon[40024]: 4.5 scrub starts 2024-09-20T22:46:05.542 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:05 smithi084 ceph-mon[40024]: 4.5 scrub ok 2024-09-20T22:46:05.542 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:05 smithi084 ceph-mon[40024]: osdmap e286: 8 total, 8 up, 6 in 2024-09-20T22:46:05.542 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:05 smithi084 ceph-mon[29774]: 7.6 scrub starts 2024-09-20T22:46:05.543 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:05 smithi084 ceph-mon[29774]: 7.6 scrub ok 2024-09-20T22:46:05.543 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:05 smithi084 ceph-mon[29774]: pgmap v328: 172 pgs: 6 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 162 active+clean; 133 MiB data, 567 MiB used, 536 GiB / 536 GiB avail; 5.0 MiB/s rd, 1.5 MiB/s wr, 13 op/s 2024-09-20T22:46:05.543 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:05 smithi084 ceph-mon[29774]: 4.5 scrub starts 2024-09-20T22:46:05.543 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:05 smithi084 ceph-mon[29774]: 4.5 scrub ok 2024-09-20T22:46:05.543 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:05 smithi084 ceph-mon[29774]: osdmap e286: 8 total, 8 up, 6 in 2024-09-20T22:46:05.547 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:46:05.569 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:05.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:05.570 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:46:05.671 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:46:05.693 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:05.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:05.694 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:46:05.795 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:46:05.817 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:05.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:05.818 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:46:05.893 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-20T22:46:05.899 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:46:05.919 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:46:05.941 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:05.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:05.942 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:46:06.042 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:46:06.064 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:06.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:06.065 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:46:06.165 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:46:06.169 INFO:tasks.rados.rados.0.smithi084.stdout:995: done (1 left) 2024-09-20T22:46:06.169 INFO:tasks.rados.rados.0.smithi084.stdout:996: rollback oid 35 current snap is 96 2024-09-20T22:46:06.169 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 35 to 94 2024-09-20T22:46:06.169 INFO:tasks.rados.rados.0.smithi084.stdout:997: rollback oid 24 current snap is 96 2024-09-20T22:46:06.169 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 24 to 94 2024-09-20T22:46:06.169 INFO:tasks.rados.rados.0.smithi084.stdout:998: write oid 25 current snap is 96 2024-09-20T22:46:06.169 INFO:tasks.rados.rados.0.smithi084.stdout:998: seq_num 369 ranges {601783=551825,1776679=425104} 2024-09-20T22:46:06.169 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is set 2024-09-20T22:46:06.175 INFO:tasks.rados.rados.0.smithi084.stdout:998: writing smithi08477203-25 from 601783 to 1153608 tid 1 2024-09-20T22:46:06.181 INFO:tasks.rados.rados.0.smithi084.stdout:998: writing smithi08477203-25 from 1776679 to 2201783 tid 2 2024-09-20T22:46:06.181 INFO:tasks.rados.rados.0.smithi084.stdout:999: snap_remove snap 93 2024-09-20T22:46:06.193 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:06.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:06.194 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:46:06.295 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:46:06.318 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:06.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:06.319 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:46:06.372 INFO:teuthology.orchestra.run.smithi084.stderr:no valid command found; 10 closest matches: 2024-09-20T22:46:06.373 INFO:teuthology.orchestra.run.smithi084.stderr:0 2024-09-20T22:46:06.373 INFO:teuthology.orchestra.run.smithi084.stderr:1 2024-09-20T22:46:06.373 INFO:teuthology.orchestra.run.smithi084.stderr:2 2024-09-20T22:46:06.373 INFO:teuthology.orchestra.run.smithi084.stderr:abort 2024-09-20T22:46:06.373 INFO:teuthology.orchestra.run.smithi084.stderr:assert 2024-09-20T22:46:06.373 INFO:teuthology.orchestra.run.smithi084.stderr:bench [] [] [] [] 2024-09-20T22:46:06.373 INFO:teuthology.orchestra.run.smithi084.stderr:bluefs debug_inject_read_zeros 2024-09-20T22:46:06.373 INFO:teuthology.orchestra.run.smithi084.stderr:bluefs files list 2024-09-20T22:46:06.373 INFO:teuthology.orchestra.run.smithi084.stderr:bluefs stats 2024-09-20T22:46:06.373 INFO:teuthology.orchestra.run.smithi084.stderr:bluestore allocator dump block 2024-09-20T22:46:06.373 INFO:teuthology.orchestra.run.smithi084.stderr:Error EINVAL: invalid command 2024-09-20T22:46:06.374 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:06 smithi084 ceph-mon[40024]: 3.14 scrub starts 2024-09-20T22:46:06.374 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:06 smithi084 ceph-mon[40024]: 3.14 scrub ok 2024-09-20T22:46:06.374 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:06 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/467222618' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-20T22:46:06.374 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:06 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-20T22:46:06.374 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:06 smithi084 ceph-mon[40024]: 5.5 scrub starts 2024-09-20T22:46:06.374 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:06 smithi084 ceph-mon[40024]: 5.5 scrub ok 2024-09-20T22:46:06.374 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:06 smithi084 ceph-mon[40024]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:46:06.374 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:06 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T22:46:06.374 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:06 smithi084 ceph-mon[40024]: osdmap e287: 8 total, 8 up, 6 in 2024-09-20T22:46:06.375 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:06 smithi084 ceph-mon[29774]: 3.14 scrub starts 2024-09-20T22:46:06.375 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:06 smithi084 ceph-mon[29774]: 3.14 scrub ok 2024-09-20T22:46:06.375 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:06 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/467222618' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-20T22:46:06.375 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:06 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-20T22:46:06.375 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:06 smithi084 ceph-mon[29774]: 5.5 scrub starts 2024-09-20T22:46:06.375 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:06 smithi084 ceph-mon[29774]: 5.5 scrub ok 2024-09-20T22:46:06.375 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:06 smithi084 ceph-mon[29774]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:46:06.375 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:06 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T22:46:06.375 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:06 smithi084 ceph-mon[29774]: osdmap e287: 8 total, 8 up, 6 in 2024-09-20T22:46:06.392 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-20T22:46:06.419 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:46:06.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:06.456 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:06.456 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:46:06.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:06 smithi154 ceph-mon[33264]: 3.14 scrub starts 2024-09-20T22:46:06.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:06 smithi154 ceph-mon[33264]: 3.14 scrub ok 2024-09-20T22:46:06.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:06 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/467222618' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-20T22:46:06.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:06 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-20T22:46:06.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:06 smithi154 ceph-mon[33264]: 5.5 scrub starts 2024-09-20T22:46:06.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:06 smithi154 ceph-mon[33264]: 5.5 scrub ok 2024-09-20T22:46:06.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:06 smithi154 ceph-mon[33264]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:46:06.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:06 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T22:46:06.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:06 smithi154 ceph-mon[33264]: osdmap e287: 8 total, 8 up, 6 in 2024-09-20T22:46:06.557 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:46:06.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:06.582 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:06.582 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:46:06.683 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:46:06.704 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:06.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:06.705 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:46:06.805 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:46:06.826 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:06.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:06.827 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:46:06.928 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:46:06.954 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:06.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:06.955 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:46:07.026 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-20T22:46:07.026 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-20T22:46:07.027 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph tell osd.6 'trim stale osdmaps' 2024-09-20T22:46:07.055 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:46:07.081 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:07.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:07.082 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:46:07.168 INFO:tasks.rados.rados.0.smithi084.stdout:999: done (4 left) 2024-09-20T22:46:07.168 INFO:tasks.rados.rados.0.smithi084.stdout:1000: delete oid 43 current snap is 96 2024-09-20T22:46:07.168 INFO:tasks.rados.rados.0.smithi084.stdout:994: finishing rollback tid 1 to smithi08477203-27 2024-09-20T22:46:07.168 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 27 v 0 (ObjNum 145 snap 0 seq_num 2090634080) dirty dne 2024-09-20T22:46:07.168 INFO:tasks.rados.rados.0.smithi084.stdout:997: finishing rollback tid 1 to smithi08477203-24 2024-09-20T22:46:07.168 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 24 v 0 (ObjNum 65 snap 0 seq_num 2) dirty dne 2024-09-20T22:46:07.168 INFO:tasks.rados.rados.0.smithi084.stdout:996: finishing rollback tid 0 to smithi08477203-35 2024-09-20T22:46:07.168 INFO:tasks.rados.rados.0.smithi084.stdout:996: finishing rollback tid 1 to smithi08477203-35 2024-09-20T22:46:07.168 INFO:tasks.rados.rados.0.smithi084.stdout:996: finishing rollback tid 2 to smithi08477203-35 2024-09-20T22:46:07.169 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 35 v 162 (ObjNum 322 snap 82 seq_num 322) dirty exists 2024-09-20T22:46:07.169 INFO:tasks.rados.rados.0.smithi084.stdout:998: finishing write tid 1 to smithi08477203-25 2024-09-20T22:46:07.169 INFO:tasks.rados.rados.0.smithi084.stdout:998: finishing write tid 2 to smithi08477203-25 2024-09-20T22:46:07.169 INFO:tasks.rados.rados.0.smithi084.stdout:998: finishing write tid 3 to smithi08477203-25 2024-09-20T22:46:07.169 INFO:tasks.rados.rados.0.smithi084.stdout:998: finishing write tid 4 to smithi08477203-25 2024-09-20T22:46:07.169 INFO:tasks.rados.rados.0.smithi084.stdout:998: oid 25 updating version 0 to 536 2024-09-20T22:46:07.169 INFO:tasks.rados.rados.0.smithi084.stdout:998: oid 25 updating version 536 to 537 2024-09-20T22:46:07.169 INFO:tasks.rados.rados.0.smithi084.stdout:998: oid 25 version 537 is already newer than 535 2024-09-20T22:46:07.169 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 25 v 537 (ObjNum 369 snap 96 seq_num 369) dirty exists 2024-09-20T22:46:07.169 INFO:tasks.rados.rados.0.smithi084.stdout:998: left oid 25 (ObjNum 369 snap 96 seq_num 369) 2024-09-20T22:46:07.174 INFO:tasks.rados.rados.0.smithi084.stdout:994: done (4 left) 2024-09-20T22:46:07.174 INFO:tasks.rados.rados.0.smithi084.stdout:996: done (3 left) 2024-09-20T22:46:07.174 INFO:tasks.rados.rados.0.smithi084.stdout:997: done (2 left) 2024-09-20T22:46:07.174 INFO:tasks.rados.rados.0.smithi084.stdout:998: done (1 left) 2024-09-20T22:46:07.174 INFO:tasks.rados.rados.0.smithi084.stdout:1000: done (0 left) 2024-09-20T22:46:07.174 INFO:tasks.rados.rados.0.smithi084.stdout:1001: snap_remove snap 90 2024-09-20T22:46:07.183 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:46:07.207 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:07.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:07.208 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:46:07.309 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:46:07.331 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:07.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:07.332 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:46:07.432 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:46:07.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:07.454 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:07.454 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:46:07.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:07 smithi154 ceph-mon[33264]: 7.16 scrub starts 2024-09-20T22:46:07.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:07 smithi154 ceph-mon[33264]: 7.16 scrub ok 2024-09-20T22:46:07.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:07 smithi154 ceph-mon[33264]: pgmap v331: 172 pgs: 6 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 162 active+clean; 133 MiB data, 567 MiB used, 536 GiB / 536 GiB avail; 5.0 MiB/s rd, 1.5 MiB/s wr, 13 op/s 2024-09-20T22:46:07.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:07 smithi154 ceph-mon[33264]: 6.19 scrub starts 2024-09-20T22:46:07.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:07 smithi154 ceph-mon[33264]: 6.19 scrub ok 2024-09-20T22:46:07.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:07 smithi154 ceph-mon[33264]: osdmap e288: 8 total, 8 up, 6 in 2024-09-20T22:46:07.535 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:07 smithi084 ceph-mon[40024]: 7.16 scrub starts 2024-09-20T22:46:07.536 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:07 smithi084 ceph-mon[40024]: 7.16 scrub ok 2024-09-20T22:46:07.536 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:07 smithi084 ceph-mon[40024]: pgmap v331: 172 pgs: 6 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 162 active+clean; 133 MiB data, 567 MiB used, 536 GiB / 536 GiB avail; 5.0 MiB/s rd, 1.5 MiB/s wr, 13 op/s 2024-09-20T22:46:07.536 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:07 smithi084 ceph-mon[40024]: 6.19 scrub starts 2024-09-20T22:46:07.536 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:07 smithi084 ceph-mon[40024]: 6.19 scrub ok 2024-09-20T22:46:07.536 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:07 smithi084 ceph-mon[40024]: osdmap e288: 8 total, 8 up, 6 in 2024-09-20T22:46:07.536 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:07 smithi084 ceph-mon[29774]: 7.16 scrub starts 2024-09-20T22:46:07.536 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:07 smithi084 ceph-mon[29774]: 7.16 scrub ok 2024-09-20T22:46:07.536 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:07 smithi084 ceph-mon[29774]: pgmap v331: 172 pgs: 6 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 162 active+clean; 133 MiB data, 567 MiB used, 536 GiB / 536 GiB avail; 5.0 MiB/s rd, 1.5 MiB/s wr, 13 op/s 2024-09-20T22:46:07.536 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:07 smithi084 ceph-mon[29774]: 6.19 scrub starts 2024-09-20T22:46:07.536 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:07 smithi084 ceph-mon[29774]: 6.19 scrub ok 2024-09-20T22:46:07.536 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:07 smithi084 ceph-mon[29774]: osdmap e288: 8 total, 8 up, 6 in 2024-09-20T22:46:07.555 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:46:07.582 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:07.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:07.583 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:46:07.667 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:46:07.676 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:46:07.684 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:46:07.710 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:07.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:07.711 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:46:07.812 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:46:07.836 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:07.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:07.837 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:46:07.938 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:46:07.963 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:07.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:07.964 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:46:08.065 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:46:08.089 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:08.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:08.090 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:46:08.175 INFO:tasks.rados.rados.0.smithi084.stdout:1001: done (0 left) 2024-09-20T22:46:08.175 INFO:tasks.rados.rados.0.smithi084.stdout:1002: snap_create 2024-09-20T22:46:08.190 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:46:08.212 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:08.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:08.213 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:46:08.314 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:46:08.336 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:08.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:08.337 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:46:08.364 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:46:08 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: ::ffff:172.21.15.154 - - [20/Sep/2024:22:46:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-20T22:46:08.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:08 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "13"}]: dispatch 2024-09-20T22:46:08.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:08 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "13"}]: dispatch 2024-09-20T22:46:08.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:08 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:46:08.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:08 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "13"}]': finished 2024-09-20T22:46:08.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:08 smithi084 ceph-mon[40024]: osdmap e289: 8 total, 8 up, 6 in 2024-09-20T22:46:08.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:08 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.a", "id": [6, 2]}]: dispatch 2024-09-20T22:46:08.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:08 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.a", "id": [6, 2]}]: dispatch 2024-09-20T22:46:08.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:08 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "13"}]: dispatch 2024-09-20T22:46:08.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:08 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "13"}]: dispatch 2024-09-20T22:46:08.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:08 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:46:08.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:08 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "13"}]': finished 2024-09-20T22:46:08.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:08 smithi084 ceph-mon[29774]: osdmap e289: 8 total, 8 up, 6 in 2024-09-20T22:46:08.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:08 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.a", "id": [6, 2]}]: dispatch 2024-09-20T22:46:08.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:08 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.a", "id": [6, 2]}]: dispatch 2024-09-20T22:46:08.403 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-20T22:46:08.403 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:46:08.437 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:46:08.459 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:08.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:08.460 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:46:08.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:08 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "13"}]: dispatch 2024-09-20T22:46:08.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:08 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "13"}]: dispatch 2024-09-20T22:46:08.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:08 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:46:08.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:08 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "13"}]': finished 2024-09-20T22:46:08.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:08 smithi154 ceph-mon[33264]: osdmap e289: 8 total, 8 up, 6 in 2024-09-20T22:46:08.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:08 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.a", "id": [6, 2]}]: dispatch 2024-09-20T22:46:08.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:08 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.a", "id": [6, 2]}]: dispatch 2024-09-20T22:46:08.561 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:46:08.583 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:08.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:08.584 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:46:08.685 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:46:08.707 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:08.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:08.708 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:46:08.808 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:46:08.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:08.830 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:08.831 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:46:08.931 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:46:08.953 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:08.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:08.954 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:46:09.055 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:46:09.077 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:09.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:09.077 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:46:09.159 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.2 dump_blocked_ops 2024-09-20T22:46:09.178 INFO:tasks.rados.rados.0.smithi084.stdout:1002: done (0 left) 2024-09-20T22:46:09.178 INFO:tasks.rados.rados.0.smithi084.stdout:1003: delete oid 14 current snap is 97 2024-09-20T22:46:09.178 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:46:09.185 INFO:tasks.rados.rados.0.smithi084.stdout:1003: done (0 left) 2024-09-20T22:46:09.185 INFO:tasks.rados.rados.0.smithi084.stdout:1004: delete oid 43 current snap is 97 2024-09-20T22:46:09.187 INFO:tasks.rados.rados.0.smithi084.stdout:1004: done (0 left) 2024-09-20T22:46:09.187 INFO:tasks.rados.rados.0.smithi084.stdout:1005: snap_remove snap 96 2024-09-20T22:46:09.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:09.202 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:09.202 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:46:09.302 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:46:09.323 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:09.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:09.324 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:46:09.351 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:46:09.351 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:46:09.370 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:46:09.370 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:46:09.388 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:46:09.389 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:46:09.389 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:09 smithi084 ceph-mon[40024]: pgmap v334: 171 pgs: 6 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 161 active+clean; 133 MiB data, 567 MiB used, 536 GiB / 536 GiB avail 2024-09-20T22:46:09.389 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:09 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.a", "id": [6, 2]}]': finished 2024-09-20T22:46:09.389 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:09 smithi084 ceph-mon[40024]: osdmap e290: 8 total, 8 up, 6 in 2024-09-20T22:46:09.389 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:09 smithi084 ceph-mon[29774]: pgmap v334: 171 pgs: 6 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 161 active+clean; 133 MiB data, 567 MiB used, 536 GiB / 536 GiB avail 2024-09-20T22:46:09.389 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:09 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.a", "id": [6, 2]}]': finished 2024-09-20T22:46:09.389 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:09 smithi084 ceph-mon[29774]: osdmap e290: 8 total, 8 up, 6 in 2024-09-20T22:46:09.407 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-20T22:46:09.407 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T22:46:09.425 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:46:09.428 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:46:09.431 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:46:09.432 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:46:09.447 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:09.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:09.448 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:46:09.449 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:46:09.450 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:46:09.462 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:46:09.468 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:46:09.469 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:46:09.485 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:46:09.486 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:46:09.504 INFO:journalctl@ceph.iscsi.iscsi.a.smithi154.stdout:Sep 20 22:46:09 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-iscsi-iscsi-a[59730]: debug there is no tcmu-runner data available 2024-09-20T22:46:09.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:09 smithi154 ceph-mon[33264]: pgmap v334: 171 pgs: 6 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 161 active+clean; 133 MiB data, 567 MiB used, 536 GiB / 536 GiB avail 2024-09-20T22:46:09.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:09 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.a", "id": [6, 2]}]': finished 2024-09-20T22:46:09.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:09 smithi154 ceph-mon[33264]: osdmap e290: 8 total, 8 up, 6 in 2024-09-20T22:46:09.549 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:46:09.570 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:09.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:09.571 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:46:09.613 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-20T22:46:09.671 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:46:09.692 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:09.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:09.693 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:46:09.793 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:46:09.815 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:09.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:09.816 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:46:09.916 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:46:09.938 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:09.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:09.938 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:46:10.039 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:46:10.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:10.061 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:10.061 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:46:10.161 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:46:10.177 INFO:tasks.rados.rados.0.smithi084.stdout:1005: done (0 left) 2024-09-20T22:46:10.177 INFO:tasks.rados.rados.0.smithi084.stdout:1006: write (excl) oid 35 current snap is 97 2024-09-20T22:46:10.177 INFO:tasks.rados.rados.0.smithi084.stdout:1006: seq_num 373 ranges {669011=569174,2010607=258404} 2024-09-20T22:46:10.183 INFO:tasks.rados.rados.0.smithi084.stdout:1006: writing smithi08477203-35 from 669011 to 1238185 tid 1 2024-09-20T22:46:10.186 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:10.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:10.187 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:46:10.189 INFO:tasks.rados.rados.0.smithi084.stdout:1006: writing smithi08477203-35 from 2010607 to 2269011 tid 2 2024-09-20T22:46:10.190 INFO:tasks.rados.rados.0.smithi084.stdout:1007: snap_create 2024-09-20T22:46:10.190 INFO:tasks.rados.rados.0.smithi084.stdout:1006: finishing write tid 1 to smithi08477203-35 2024-09-20T22:46:10.192 INFO:tasks.rados.rados.0.smithi084.stdout:1006: finishing write tid 2 to smithi08477203-35 2024-09-20T22:46:10.192 INFO:tasks.rados.rados.0.smithi084.stdout:1006: finishing write tid 3 to smithi08477203-35 2024-09-20T22:46:10.192 INFO:tasks.rados.rados.0.smithi084.stdout:1006: finishing write tid 4 to smithi08477203-35 2024-09-20T22:46:10.192 INFO:tasks.rados.rados.0.smithi084.stdout:1006: oid 35 updating version 0 to 165 2024-09-20T22:46:10.192 INFO:tasks.rados.rados.0.smithi084.stdout:1006: oid 35 version 165 is already newer than 164 2024-09-20T22:46:10.192 INFO:tasks.rados.rados.0.smithi084.stdout:1006: oid 35 version 165 is already newer than 163 2024-09-20T22:46:10.192 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 35 v 165 (ObjNum 373 snap 97 seq_num 373) dirty exists 2024-09-20T22:46:10.192 INFO:tasks.rados.rados.0.smithi084.stdout:1006: left oid 35 (ObjNum 373 snap 97 seq_num 373) 2024-09-20T22:46:10.288 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:46:10.309 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:10.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:10.310 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:46:10.356 INFO:teuthology.orchestra.run.smithi084.stderr:no valid command found; 10 closest matches: 2024-09-20T22:46:10.356 INFO:teuthology.orchestra.run.smithi084.stderr:0 2024-09-20T22:46:10.356 INFO:teuthology.orchestra.run.smithi084.stderr:1 2024-09-20T22:46:10.356 INFO:teuthology.orchestra.run.smithi084.stderr:2 2024-09-20T22:46:10.356 INFO:teuthology.orchestra.run.smithi084.stderr:abort 2024-09-20T22:46:10.357 INFO:teuthology.orchestra.run.smithi084.stderr:assert 2024-09-20T22:46:10.357 INFO:teuthology.orchestra.run.smithi084.stderr:bench [] [] [] [] 2024-09-20T22:46:10.357 INFO:teuthology.orchestra.run.smithi084.stderr:bluefs debug_inject_read_zeros 2024-09-20T22:46:10.357 INFO:teuthology.orchestra.run.smithi084.stderr:bluefs files list 2024-09-20T22:46:10.357 INFO:teuthology.orchestra.run.smithi084.stderr:bluefs stats 2024-09-20T22:46:10.357 INFO:teuthology.orchestra.run.smithi084.stderr:bluestore allocator dump block 2024-09-20T22:46:10.357 INFO:teuthology.orchestra.run.smithi084.stderr:Error EINVAL: invalid command 2024-09-20T22:46:10.410 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:46:10.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:10.434 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:10.434 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:46:10.535 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:46:10.556 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:10.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:10.556 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:46:10.614 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:10 smithi084 ceph-mon[29774]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:46:10.614 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:10 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/576464128' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:46:10.614 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:10 smithi084 ceph-mon[29774]: osdmap e291: 8 total, 8 up, 6 in 2024-09-20T22:46:10.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:10 smithi084 ceph-mon[40024]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:46:10.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:10 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/576464128' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:46:10.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:10 smithi084 ceph-mon[40024]: osdmap e291: 8 total, 8 up, 6 in 2024-09-20T22:46:10.657 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:46:10.679 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:10.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:10.680 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:46:10.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:10 smithi154 ceph-mon[33264]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:46:10.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:10 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/576464128' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:46:10.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:10 smithi154 ceph-mon[33264]: osdmap e291: 8 total, 8 up, 6 in 2024-09-20T22:46:10.781 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:46:10.803 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:10.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:10.804 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:46:10.866 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-20T22:46:10.866 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2024-09-20T22:46:10.866 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2024-09-20T22:46:10.866 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2024-09-20T22:46:10.867 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-20T22:46:10.904 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:46:10.925 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:10.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:10.926 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:46:11.027 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:46:11.048 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:11.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:11.049 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:46:11.150 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:46:11.171 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:11.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:11.172 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:46:11.177 INFO:tasks.rados.rados.0.smithi084.stdout:1006: done (1 left) 2024-09-20T22:46:11.178 INFO:tasks.rados.rados.0.smithi084.stdout:1007: done (0 left) 2024-09-20T22:46:11.178 INFO:tasks.rados.rados.0.smithi084.stdout:1008: snap_create 2024-09-20T22:46:11.272 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:46:11.294 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:11.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:11.295 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:46:11.396 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:46:11.417 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:11.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:11.418 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:46:11.445 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:11 smithi084 ceph-mon[40024]: pgmap v337: 171 pgs: 2 peering, 4 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 161 active+clean; 124 MiB data, 547 MiB used, 536 GiB / 536 GiB avail; 3.3 MiB/s rd, 1.1 MiB/s wr, 5 op/s 2024-09-20T22:46:11.446 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:11 smithi084 ceph-mon[40024]: osdmap e292: 8 total, 8 up, 6 in 2024-09-20T22:46:11.446 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:11 smithi084 ceph-mon[29774]: pgmap v337: 171 pgs: 2 peering, 4 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 161 active+clean; 124 MiB data, 547 MiB used, 536 GiB / 536 GiB avail; 3.3 MiB/s rd, 1.1 MiB/s wr, 5 op/s 2024-09-20T22:46:11.446 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:11 smithi084 ceph-mon[29774]: osdmap e292: 8 total, 8 up, 6 in 2024-09-20T22:46:11.519 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:46:11.540 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:11.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:11.541 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:46:11.642 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:46:11.663 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:11.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:11.664 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:46:11.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:11 smithi154 ceph-mon[33264]: pgmap v337: 171 pgs: 2 peering, 4 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 161 active+clean; 124 MiB data, 547 MiB used, 536 GiB / 536 GiB avail; 3.3 MiB/s rd, 1.1 MiB/s wr, 5 op/s 2024-09-20T22:46:11.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:11 smithi154 ceph-mon[33264]: osdmap e292: 8 total, 8 up, 6 in 2024-09-20T22:46:11.765 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:46:11.786 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:11.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:11.787 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:46:11.888 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:46:11.891 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-20T22:46:11.891 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-20T22:46:11.891 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph tell osd.7 'trim stale osdmaps' 2024-09-20T22:46:11.909 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:11.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:11.910 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:46:12.011 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:46:12.032 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:12.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:12.033 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:46:12.134 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:46:12.157 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:12.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:12.158 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:46:12.182 INFO:tasks.rados.rados.0.smithi084.stdout:1008: done (0 left) 2024-09-20T22:46:12.182 INFO:tasks.rados.rados.0.smithi084.stdout:1009: write oid 41 current snap is 99 2024-09-20T22:46:12.183 INFO:tasks.rados.rados.0.smithi084.stdout:1009: seq_num 374 ranges {685818=644423,2069089=216729} 2024-09-20T22:46:12.190 INFO:tasks.rados.rados.0.smithi084.stdout:1009: writing smithi08477203-41 from 685818 to 1330241 tid 1 2024-09-20T22:46:12.195 INFO:tasks.rados.rados.0.smithi084.stdout:1009: writing smithi08477203-41 from 2069089 to 2285818 tid 2 2024-09-20T22:46:12.195 INFO:tasks.rados.rados.0.smithi084.stdout:1010: write (excl) oid 32 current snap is 99 2024-09-20T22:46:12.195 INFO:tasks.rados.rados.0.smithi084.stdout:1010: seq_num 375 ranges {702625=719672,2127571=175054} 2024-09-20T22:46:12.203 INFO:tasks.rados.rados.0.smithi084.stdout:1010: writing smithi08477203-32 from 702625 to 1422297 tid 1 2024-09-20T22:46:12.208 INFO:tasks.rados.rados.0.smithi084.stdout:1010: writing smithi08477203-32 from 2127571 to 2302625 tid 2 2024-09-20T22:46:12.208 INFO:tasks.rados.rados.0.smithi084.stdout:1011: read oid 7 snap -1 2024-09-20T22:46:12.209 INFO:tasks.rados.rados.0.smithi084.stdout:1011: expect deleted 2024-09-20T22:46:12.209 INFO:tasks.rados.rados.0.smithi084.stdout:1012: snap_remove snap 98 2024-09-20T22:46:12.259 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:46:12.281 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:12.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:12.282 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:46:12.361 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.2 dump_historic_ops 2024-09-20T22:46:12.375 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:46:12.383 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:46:12.405 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:12.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:12.406 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:46:12.507 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:46:12.529 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:12.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:12.530 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:46:12.536 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:12 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "12"}]: dispatch 2024-09-20T22:46:12.536 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:12 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "12"}]: dispatch 2024-09-20T22:46:12.536 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:12 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "12"}]': finished 2024-09-20T22:46:12.536 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:12 smithi084 ceph-mon[40024]: osdmap e293: 8 total, 8 up, 6 in 2024-09-20T22:46:12.536 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:12 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "12"}]: dispatch 2024-09-20T22:46:12.537 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:12 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "12"}]: dispatch 2024-09-20T22:46:12.537 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:12 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "12"}]': finished 2024-09-20T22:46:12.537 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:12 smithi084 ceph-mon[29774]: osdmap e293: 8 total, 8 up, 6 in 2024-09-20T22:46:12.630 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:46:12.652 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:12.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:12.653 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:46:12.754 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:46:12.842 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:12 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "12"}]: dispatch 2024-09-20T22:46:12.842 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:12 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "12"}]: dispatch 2024-09-20T22:46:12.842 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:12 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "12"}]': finished 2024-09-20T22:46:12.842 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:12 smithi154 ceph-mon[33264]: osdmap e293: 8 total, 8 up, 6 in 2024-09-20T22:46:12.863 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:12.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:12.864 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:46:12.888 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:46:12.888 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:46:12.965 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:46:12.988 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:12.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:12.989 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:46:13.090 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:46:13.112 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:13.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:13.113 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:46:13.168 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is set 2024-09-20T22:46:13.181 INFO:tasks.rados.rados.0.smithi084.stdout:1012: done (3 left) 2024-09-20T22:46:13.181 INFO:tasks.rados.rados.0.smithi084.stdout:1013: read oid 42 snap 94 2024-09-20T22:46:13.181 INFO:tasks.rados.rados.0.smithi084.stdout:1013: expect deleted 2024-09-20T22:46:13.181 INFO:tasks.rados.rados.0.smithi084.stdout:1014: snap_remove snap 92 2024-09-20T22:46:13.214 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:46:13.234 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:13.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:13.235 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:46:13.336 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:46:13.357 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:13.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:13.358 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:46:13.402 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:13 smithi084 ceph-mon[40024]: pgmap v340: 170 pgs: 1 peering, 4 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 161 active+clean; 124 MiB data, 547 MiB used, 536 GiB / 536 GiB avail; 3.3 MiB/s rd, 1.1 MiB/s wr, 5 op/s 2024-09-20T22:46:13.402 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:13 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/158468808' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:46:13.402 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:13 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:46:13.402 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:13 smithi084 ceph-mon[40024]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:46:13.402 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:13 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T22:46:13.402 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:13 smithi084 ceph-mon[40024]: osdmap e294: 8 total, 8 up, 6 in 2024-09-20T22:46:13.403 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:13 smithi084 ceph-mon[29774]: pgmap v340: 170 pgs: 1 peering, 4 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 161 active+clean; 124 MiB data, 547 MiB used, 536 GiB / 536 GiB avail; 3.3 MiB/s rd, 1.1 MiB/s wr, 5 op/s 2024-09-20T22:46:13.403 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:13 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/158468808' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:46:13.403 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:13 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:46:13.403 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:13 smithi084 ceph-mon[29774]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:46:13.403 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:13 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T22:46:13.403 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:13 smithi084 ceph-mon[29774]: osdmap e294: 8 total, 8 up, 6 in 2024-09-20T22:46:13.460 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:46:13.472 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-20T22:46:13.481 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:13.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:13.482 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:46:13.583 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:46:13.604 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:13.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:13.605 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:46:13.706 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:46:13.727 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:13.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:13.728 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:46:13.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:13 smithi154 ceph-mon[33264]: pgmap v340: 170 pgs: 1 peering, 4 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 161 active+clean; 124 MiB data, 547 MiB used, 536 GiB / 536 GiB avail; 3.3 MiB/s rd, 1.1 MiB/s wr, 5 op/s 2024-09-20T22:46:13.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:13 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/158468808' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:46:13.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:13 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:46:13.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:13 smithi154 ceph-mon[33264]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:46:13.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:13 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T22:46:13.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:13 smithi154 ceph-mon[33264]: osdmap e294: 8 total, 8 up, 6 in 2024-09-20T22:46:13.780 INFO:teuthology.orchestra.run.smithi084.stderr:no valid command found; 10 closest matches: 2024-09-20T22:46:13.780 INFO:teuthology.orchestra.run.smithi084.stderr:0 2024-09-20T22:46:13.780 INFO:teuthology.orchestra.run.smithi084.stderr:1 2024-09-20T22:46:13.780 INFO:teuthology.orchestra.run.smithi084.stderr:2 2024-09-20T22:46:13.780 INFO:teuthology.orchestra.run.smithi084.stderr:abort 2024-09-20T22:46:13.780 INFO:teuthology.orchestra.run.smithi084.stderr:assert 2024-09-20T22:46:13.780 INFO:teuthology.orchestra.run.smithi084.stderr:bench [] [] [] [] 2024-09-20T22:46:13.780 INFO:teuthology.orchestra.run.smithi084.stderr:bluefs debug_inject_read_zeros 2024-09-20T22:46:13.780 INFO:teuthology.orchestra.run.smithi084.stderr:bluefs files list 2024-09-20T22:46:13.780 INFO:teuthology.orchestra.run.smithi084.stderr:bluefs stats 2024-09-20T22:46:13.780 INFO:teuthology.orchestra.run.smithi084.stderr:bluestore allocator dump block 2024-09-20T22:46:13.780 INFO:teuthology.orchestra.run.smithi084.stderr:Error EINVAL: invalid command 2024-09-20T22:46:13.829 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:46:13.850 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:13.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:13.851 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:46:13.952 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:46:13.973 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:13.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:13.974 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:46:14.075 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:46:14.097 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:14.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:14.098 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:46:14.170 INFO:tasks.rados.rados.0.smithi084.stdout:1014: done (4 left) 2024-09-20T22:46:14.170 INFO:tasks.rados.rados.0.smithi084.stdout:1015: delete oid 14 current snap is 99 2024-09-20T22:46:14.170 INFO:tasks.rados.rados.0.smithi084.stdout:1009: finishing write tid 1 to smithi08477203-41 2024-09-20T22:46:14.170 INFO:tasks.rados.rados.0.smithi084.stdout:1009: finishing write tid 2 to smithi08477203-41 2024-09-20T22:46:14.170 INFO:tasks.rados.rados.0.smithi084.stdout:1009: finishing write tid 3 to smithi08477203-41 2024-09-20T22:46:14.170 INFO:tasks.rados.rados.0.smithi084.stdout:1009: finishing write tid 4 to smithi08477203-41 2024-09-20T22:46:14.170 INFO:tasks.rados.rados.0.smithi084.stdout:1009: oid 41 updating version 0 to 79 2024-09-20T22:46:14.171 INFO:tasks.rados.rados.0.smithi084.stdout:1009: oid 41 updating version 79 to 80 2024-09-20T22:46:14.171 INFO:tasks.rados.rados.0.smithi084.stdout:1009: oid 41 version 80 is already newer than 78 2024-09-20T22:46:14.171 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 41 v 80 (ObjNum 374 snap 99 seq_num 374) dirty exists 2024-09-20T22:46:14.171 INFO:tasks.rados.rados.0.smithi084.stdout:1009: left oid 41 (ObjNum 374 snap 99 seq_num 374) 2024-09-20T22:46:14.171 INFO:tasks.rados.rados.0.smithi084.stdout:1010: finishing write tid 1 to smithi08477203-32 2024-09-20T22:46:14.171 INFO:tasks.rados.rados.0.smithi084.stdout:1010: finishing write tid 2 to smithi08477203-32 2024-09-20T22:46:14.171 INFO:tasks.rados.rados.0.smithi084.stdout:1010: finishing write tid 3 to smithi08477203-32 2024-09-20T22:46:14.171 INFO:tasks.rados.rados.0.smithi084.stdout:1010: finishing write tid 4 to smithi08477203-32 2024-09-20T22:46:14.171 INFO:tasks.rados.rados.0.smithi084.stdout:1010: oid 32 updating version 0 to 377 2024-09-20T22:46:14.171 INFO:tasks.rados.rados.0.smithi084.stdout:1010: oid 32 updating version 377 to 379 2024-09-20T22:46:14.171 INFO:tasks.rados.rados.0.smithi084.stdout:1010: oid 32 version 379 is already newer than 378 2024-09-20T22:46:14.171 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 32 v 379 (ObjNum 375 snap 99 seq_num 375) dirty exists 2024-09-20T22:46:14.172 INFO:tasks.rados.rados.0.smithi084.stdout:1010: left oid 32 (ObjNum 375 snap 99 seq_num 375) 2024-09-20T22:46:14.175 INFO:tasks.rados.rados.0.smithi084.stdout:1009: done (4 left) 2024-09-20T22:46:14.175 INFO:tasks.rados.rados.0.smithi084.stdout:1010: done (3 left) 2024-09-20T22:46:14.175 INFO:tasks.rados.rados.0.smithi084.stdout:1011: done (2 left) 2024-09-20T22:46:14.175 INFO:tasks.rados.rados.0.smithi084.stdout:1013: done (1 left) 2024-09-20T22:46:14.175 INFO:tasks.rados.rados.0.smithi084.stdout:1015: done (0 left) 2024-09-20T22:46:14.175 INFO:tasks.rados.rados.0.smithi084.stdout:1016: write oid 31 current snap is 99 2024-09-20T22:46:14.175 INFO:tasks.rados.rados.0.smithi084.stdout:1016: seq_num 377 ranges {736239=470170,1728182=571782} 2024-09-20T22:46:14.181 INFO:tasks.rados.rados.0.smithi084.stdout:1016: writing smithi08477203-31 from 736239 to 1206409 tid 1 2024-09-20T22:46:14.188 INFO:tasks.rados.rados.0.smithi084.stdout:1016: writing smithi08477203-31 from 1728182 to 2299964 tid 2 2024-09-20T22:46:14.188 INFO:tasks.rados.rados.0.smithi084.stdout:1017: copy_from oid 29 from oid 21 current snap is 99 2024-09-20T22:46:14.189 INFO:tasks.rados.rados.0.smithi084.stdout:1018: snap_remove snap 94 2024-09-20T22:46:14.199 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:46:14.221 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:14.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:14.222 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:46:14.322 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:46:14.344 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:14.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:14.345 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:46:14.446 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:46:14.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:14.469 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:14.469 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:46:14.544 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-20T22:46:14.544 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-20T22:46:14.570 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:46:14.592 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:14.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:14.593 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:46:14.693 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:46:14.715 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:14.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:14.716 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:46:14.817 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:46:14.839 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:14.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:14.840 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:46:14.941 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:46:14.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:14.964 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:14.964 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:46:15.065 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:46:15.086 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:15.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:15.087 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:46:15.172 INFO:tasks.rados.rados.0.smithi084.stdout:1018: done (2 left) 2024-09-20T22:46:15.172 INFO:tasks.rados.rados.0.smithi084.stdout:1019: read oid 34 snap -1 2024-09-20T22:46:15.172 INFO:tasks.rados.rados.0.smithi084.stdout:1019: expect deleted 2024-09-20T22:46:15.172 INFO:tasks.rados.rados.0.smithi084.stdout:1020: write (excl) oid 33 current snap is 99 2024-09-20T22:46:15.172 INFO:tasks.rados.rados.0.smithi084.stdout:1016: finishing write tid 1 to smithi08477203-31 2024-09-20T22:46:15.172 INFO:tasks.rados.rados.0.smithi084.stdout:1020: seq_num 378 ranges {753046=545419,1786664=431793} 2024-09-20T22:46:15.179 INFO:tasks.rados.rados.0.smithi084.stdout:1020: writing smithi08477203-33 from 753046 to 1298465 tid 1 2024-09-20T22:46:15.185 INFO:tasks.rados.rados.0.smithi084.stdout:1020: writing smithi08477203-33 from 1786664 to 2218457 tid 2 2024-09-20T22:46:15.185 INFO:tasks.rados.rados.0.smithi084.stdout:1021: snap_create 2024-09-20T22:46:15.186 INFO:tasks.rados.rados.0.smithi084.stdout:1016: finishing write tid 2 to smithi08477203-31 2024-09-20T22:46:15.186 INFO:tasks.rados.rados.0.smithi084.stdout:1016: finishing write tid 3 to smithi08477203-31 2024-09-20T22:46:15.186 INFO:tasks.rados.rados.0.smithi084.stdout:1016: finishing write tid 4 to smithi08477203-31 2024-09-20T22:46:15.186 INFO:tasks.rados.rados.0.smithi084.stdout:1016: oid 31 updating version 0 to 379 2024-09-20T22:46:15.186 INFO:tasks.rados.rados.0.smithi084.stdout:1016: oid 31 version 379 is already newer than 378 2024-09-20T22:46:15.186 INFO:tasks.rados.rados.0.smithi084.stdout:1016: oid 31 version 379 is already newer than 377 2024-09-20T22:46:15.186 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 31 v 379 (ObjNum 377 snap 99 seq_num 377) dirty exists 2024-09-20T22:46:15.186 INFO:tasks.rados.rados.0.smithi084.stdout:1016: left oid 31 (ObjNum 377 snap 99 seq_num 377) 2024-09-20T22:46:15.186 INFO:tasks.rados.rados.0.smithi084.stdout:1017: finishing copy_from racing read to smithi08477203-29 2024-09-20T22:46:15.186 INFO:tasks.rados.rados.0.smithi084.stdout:1017: finishing copy_from to smithi08477203-29 2024-09-20T22:46:15.186 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 29 v 260 (ObjNum 339 snap 86 seq_num 339) dirty exists 2024-09-20T22:46:15.186 INFO:tasks.rados.rados.0.smithi084.stdout:1020: finishing write tid 1 to smithi08477203-33 2024-09-20T22:46:15.187 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:46:15.188 INFO:tasks.rados.rados.0.smithi084.stdout:1020: finishing write tid 2 to smithi08477203-33 2024-09-20T22:46:15.188 INFO:tasks.rados.rados.0.smithi084.stdout:1020: finishing write tid 3 to smithi08477203-33 2024-09-20T22:46:15.188 INFO:tasks.rados.rados.0.smithi084.stdout:1020: finishing write tid 4 to smithi08477203-33 2024-09-20T22:46:15.188 INFO:tasks.rados.rados.0.smithi084.stdout:1020: oid 33 updating version 0 to 213 2024-09-20T22:46:15.188 INFO:tasks.rados.rados.0.smithi084.stdout:1020: oid 33 version 213 is already newer than 212 2024-09-20T22:46:15.189 INFO:tasks.rados.rados.0.smithi084.stdout:1020: oid 33 version 213 is already newer than 211 2024-09-20T22:46:15.189 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 33 v 213 (ObjNum 378 snap 99 seq_num 378) dirty exists 2024-09-20T22:46:15.189 INFO:tasks.rados.rados.0.smithi084.stdout:1020: left oid 33 (ObjNum 378 snap 99 seq_num 378) 2024-09-20T22:46:15.209 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-20T22:46:15.210 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2024-09-20T22:46:15.210 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2024-09-20T22:46:15.210 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [ 2024-09-20T22:46:15.210 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.210 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24652.0:786 7.17 e145/139 7:ed54065f:::smithi08477203-12:head v 145'119, mlcod=143'117)", 2024-09-20T22:46:15.210 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:43:44.344970+0000", 2024-09-20T22:46:15.210 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 150.86218398299999, 2024-09-20T22:46:15.210 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010847977999999999, 2024-09-20T22:46:15.210 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:46:15.210 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:46:15.210 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:46:15.210 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:46:15.210 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.210 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:46:15.211 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:44.344970+0000", 2024-09-20T22:46:15.211 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:46:15.211 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.211 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.211 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:46:15.211 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:44.344970+0000", 2024-09-20T22:46:15.211 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:46:15.211 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.211 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.211 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:46:15.211 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:44.344970+0000", 2024-09-20T22:46:15.211 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.85e-07 2024-09-20T22:46:15.211 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.212 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.212 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:46:15.212 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:44.350418+0000", 2024-09-20T22:46:15.212 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0054472749999999997 2024-09-20T22:46:15.212 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.212 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.212 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:46:15.212 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:44.350434+0000", 2024-09-20T22:46:15.212 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.6113999999999999e-05 2024-09-20T22:46:15.212 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.212 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.212 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:46:15.212 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:44.350443+0000", 2024-09-20T22:46:15.212 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.6930000000000002e-06 2024-09-20T22:46:15.213 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.213 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.213 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:46:15.213 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:44.350504+0000", 2024-09-20T22:46:15.213 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.0541000000000002e-05 2024-09-20T22:46:15.213 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.213 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.213 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:46:15.213 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:44.350522+0000", 2024-09-20T22:46:15.213 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.8136999999999999e-05 2024-09-20T22:46:15.213 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.213 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.213 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:46:15.214 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:44.355760+0000", 2024-09-20T22:46:15.214 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0052376269999999999 2024-09-20T22:46:15.214 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.214 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.214 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:46:15.214 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:44.355818+0000", 2024-09-20T22:46:15.214 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.7806000000000002e-05 2024-09-20T22:46:15.214 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:46:15.214 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:46:15.214 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:46:15.214 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.214 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.214 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:901 7.e 7:7c62ced4:::smithi08477203-17:head [copy-from ver 48] snapc 1e=[] ondisk+write+known_if_redirected+supports_pool_eio e156)", 2024-09-20T22:46:15.214 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:43:55.624762+0000", 2024-09-20T22:46:15.215 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 139.58239112800001, 2024-09-20T22:46:15.215 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.021835856000000001, 2024-09-20T22:46:15.215 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:46:15.215 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:46:15.215 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:46:15.215 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:46:15.215 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:46:15.215 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:46:15.215 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 901 2024-09-20T22:46:15.215 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.215 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:46:15.215 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.215 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:46:15.215 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.624762+0000", 2024-09-20T22:46:15.215 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:46:15.215 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.216 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.216 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:46:15.216 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.624762+0000", 2024-09-20T22:46:15.216 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:46:15.216 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.216 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.216 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:46:15.216 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.624765+0000", 2024-09-20T22:46:15.216 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.57e-06 2024-09-20T22:46:15.216 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.216 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.216 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:46:15.216 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.624776+0000", 2024-09-20T22:46:15.216 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.078e-05 2024-09-20T22:46:15.216 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.217 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.217 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:46:15.217 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.624803+0000", 2024-09-20T22:46:15.217 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.6769999999999999e-05 2024-09-20T22:46:15.217 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.217 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.217 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:46:15.217 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.624815+0000", 2024-09-20T22:46:15.217 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2459e-05 2024-09-20T22:46:15.217 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.217 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.217 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:46:15.218 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.624936+0000", 2024-09-20T22:46:15.218 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000120467 2024-09-20T22:46:15.218 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.218 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.218 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:46:15.218 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.625046+0000", 2024-09-20T22:46:15.218 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000110222 2024-09-20T22:46:15.218 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.218 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.218 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 0,5", 2024-09-20T22:46:15.218 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.635547+0000", 2024-09-20T22:46:15.218 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010501765999999999 2024-09-20T22:46:15.218 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.219 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.219 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:46:15.219 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.644332+0000", 2024-09-20T22:46:15.219 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0087847729999999992 2024-09-20T22:46:15.219 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.219 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.219 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:46:15.219 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.646168+0000", 2024-09-20T22:46:15.219 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0018360010000000001 2024-09-20T22:46:15.219 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.219 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.219 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:46:15.219 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.646512+0000", 2024-09-20T22:46:15.219 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000343269 2024-09-20T22:46:15.220 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.220 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.220 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:46:15.220 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.646550+0000", 2024-09-20T22:46:15.220 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.8383000000000002e-05 2024-09-20T22:46:15.220 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.220 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.220 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:46:15.220 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.646598+0000", 2024-09-20T22:46:15.220 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.8396000000000002e-05 2024-09-20T22:46:15.220 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:46:15.220 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:46:15.220 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:46:15.220 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.221 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.221 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:902 7.e 7:7c62ced4:::smithi08477203-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e156)", 2024-09-20T22:46:15.221 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:43:55.624853+0000", 2024-09-20T22:46:15.221 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 139.58230105000001, 2024-09-20T22:46:15.221 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.015323603999999999, 2024-09-20T22:46:15.221 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:46:15.221 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:46:15.221 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-20T22:46:15.221 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:46:15.221 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:46:15.221 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:46:15.221 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 902 2024-09-20T22:46:15.221 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.222 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:46:15.222 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.222 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:46:15.222 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.624853+0000", 2024-09-20T22:46:15.222 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:46:15.222 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.222 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.222 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:46:15.222 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.624853+0000", 2024-09-20T22:46:15.222 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:46:15.222 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.222 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.222 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:46:15.222 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.624853+0000", 2024-09-20T22:46:15.222 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.1399999999999997e-07 2024-09-20T22:46:15.222 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.223 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.223 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:46:15.223 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.624859+0000", 2024-09-20T22:46:15.223 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.9610000000000002e-06 2024-09-20T22:46:15.223 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.223 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.223 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:46:15.223 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.624887+0000", 2024-09-20T22:46:15.223 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.7597e-05 2024-09-20T22:46:15.223 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.223 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.223 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:46:15.223 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.624894+0000", 2024-09-20T22:46:15.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:15.224 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.469e-06 2024-09-20T22:46:15.225 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.225 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.225 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:46:15.225 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.625219+0000", 2024-09-20T22:46:15.225 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00032538499999999998 2024-09-20T22:46:15.225 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.225 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.225 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for blocked object", 2024-09-20T22:46:15.225 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.625258+0000", 2024-09-20T22:46:15.225 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.8467000000000003e-05 2024-09-20T22:46:15.225 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.225 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.225 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:46:15.226 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.640050+0000", 2024-09-20T22:46:15.226 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0147923 2024-09-20T22:46:15.226 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.226 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.226 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:46:15.226 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.640112+0000", 2024-09-20T22:46:15.226 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.1723000000000005e-05 2024-09-20T22:46:15.226 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.226 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.226 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:46:15.226 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.640176+0000", 2024-09-20T22:46:15.226 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.4188000000000003e-05 2024-09-20T22:46:15.226 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:46:15.227 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:46:15.227 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:46:15.227 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.227 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.227 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24652.0:1420 7.19 e189/139 7:9cc0b7e2:::smithi08477203-46:head v 189'99, mlcod=174'95)", 2024-09-20T22:46:15.227 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:44:28.150258+0000", 2024-09-20T22:46:15.227 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 107.05689535400001, 2024-09-20T22:46:15.227 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014180626, 2024-09-20T22:46:15.227 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:46:15.227 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:46:15.227 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:46:15.227 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:46:15.227 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.227 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:46:15.227 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.150258+0000", 2024-09-20T22:46:15.228 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:46:15.228 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.228 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.228 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:46:15.228 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.150258+0000", 2024-09-20T22:46:15.228 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:46:15.228 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.228 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.228 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:46:15.228 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.150259+0000", 2024-09-20T22:46:15.228 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.99e-07 2024-09-20T22:46:15.228 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.228 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.229 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:46:15.229 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.150674+0000", 2024-09-20T22:46:15.229 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00041528000000000002 2024-09-20T22:46:15.229 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.229 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.229 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:46:15.229 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.150688+0000", 2024-09-20T22:46:15.229 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3399e-05 2024-09-20T22:46:15.229 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.229 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.229 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:46:15.229 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.150699+0000", 2024-09-20T22:46:15.229 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1022999999999999e-05 2024-09-20T22:46:15.230 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.230 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.230 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:46:15.230 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.150772+0000", 2024-09-20T22:46:15.230 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.3455999999999996e-05 2024-09-20T22:46:15.230 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.230 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.230 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:46:15.230 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.150798+0000", 2024-09-20T22:46:15.230 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.5777000000000001e-05 2024-09-20T22:46:15.230 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.230 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.230 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:46:15.230 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.164390+0000", 2024-09-20T22:46:15.231 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013591789999999999 2024-09-20T22:46:15.231 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.231 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.231 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:46:15.231 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.164439+0000", 2024-09-20T22:46:15.231 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.9202000000000001e-05 2024-09-20T22:46:15.231 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:46:15.231 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:46:15.231 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:46:15.231 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.231 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.231 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24652.0:1421 7.19 e189/139 7:9cc0b7e2:::smithi08477203-46:head v 189'100, mlcod=174'95)", 2024-09-20T22:46:15.231 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:44:28.156805+0000", 2024-09-20T22:46:15.232 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 107.050348496, 2024-09-20T22:46:15.232 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013180661999999999, 2024-09-20T22:46:15.232 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:46:15.232 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:46:15.232 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:46:15.232 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:46:15.232 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.232 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:46:15.232 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.156805+0000", 2024-09-20T22:46:15.232 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:46:15.232 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.232 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.232 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:46:15.233 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.156805+0000", 2024-09-20T22:46:15.233 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:46:15.233 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.233 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.233 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:46:15.233 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.156806+0000", 2024-09-20T22:46:15.233 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.074e-06 2024-09-20T22:46:15.233 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.233 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.233 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:46:15.233 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.157126+0000", 2024-09-20T22:46:15.233 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00031991399999999997 2024-09-20T22:46:15.233 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.233 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.234 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:46:15.234 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.157136+0000", 2024-09-20T22:46:15.234 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.4499999999999993e-06 2024-09-20T22:46:15.234 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.234 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.234 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:46:15.234 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.157143+0000", 2024-09-20T22:46:15.234 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.7649999999999993e-06 2024-09-20T22:46:15.234 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.234 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.234 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:46:15.234 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.157188+0000", 2024-09-20T22:46:15.234 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.4885000000000001e-05 2024-09-20T22:46:15.235 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.235 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.235 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:46:15.235 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.157208+0000", 2024-09-20T22:46:15.235 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.0015000000000001e-05 2024-09-20T22:46:15.235 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.235 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.235 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:46:15.235 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.169936+0000", 2024-09-20T22:46:15.235 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.012728204 2024-09-20T22:46:15.235 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.235 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.235 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:46:15.235 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.169986+0000", 2024-09-20T22:46:15.236 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.9354999999999998e-05 2024-09-20T22:46:15.236 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:46:15.236 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:46:15.236 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:46:15.236 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.236 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.236 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24652.0:1527 7.17 e197/139 7:ea720b85:::smithi08477203-26:head v 197'159, mlcod=187'156)", 2024-09-20T22:46:15.236 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:44:36.209366+0000", 2024-09-20T22:46:15.236 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 98.997788043, 2024-09-20T22:46:15.236 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010891473, 2024-09-20T22:46:15.236 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:46:15.236 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:46:15.236 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:46:15.237 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:46:15.237 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.237 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:46:15.237 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:36.209366+0000", 2024-09-20T22:46:15.237 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:46:15.237 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.237 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.237 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:46:15.237 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:36.209366+0000", 2024-09-20T22:46:15.237 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:46:15.237 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.237 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.237 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:46:15.237 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:36.209366+0000", 2024-09-20T22:46:15.238 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.0999999999999999e-07 2024-09-20T22:46:15.238 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.238 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.238 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:46:15.238 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:36.215620+0000", 2024-09-20T22:46:15.238 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0062539450000000003 2024-09-20T22:46:15.238 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.238 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.238 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:46:15.238 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:36.215649+0000", 2024-09-20T22:46:15.238 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.9037999999999999e-05 2024-09-20T22:46:15.238 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.238 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.238 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:46:15.238 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:36.215663+0000", 2024-09-20T22:46:15.239 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.4059999999999999e-05 2024-09-20T22:46:15.239 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.239 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.239 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:46:15.239 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:36.215772+0000", 2024-09-20T22:46:15.239 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000108773 2024-09-20T22:46:15.239 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.239 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.239 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:46:15.239 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:36.215801+0000", 2024-09-20T22:46:15.239 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.8711e-05 2024-09-20T22:46:15.239 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.240 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.240 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:46:15.240 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:36.220205+0000", 2024-09-20T22:46:15.240 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0044043989999999998 2024-09-20T22:46:15.240 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.240 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.240 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:46:15.240 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:36.220257+0000", 2024-09-20T22:46:15.240 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.2036999999999997e-05 2024-09-20T22:46:15.240 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:46:15.240 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:46:15.240 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:46:15.240 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.240 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.241 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24652.0:1542 7.12 e198/139 7:4b06cbed:::smithi08477203-49:head v 198'174, mlcod=197'171)", 2024-09-20T22:46:15.241 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:44:37.216841+0000", 2024-09-20T22:46:15.241 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 97.990312109000001, 2024-09-20T22:46:15.241 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010876451000000001, 2024-09-20T22:46:15.241 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:46:15.241 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:46:15.241 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:46:15.241 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:46:15.241 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.241 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:46:15.241 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:37.216841+0000", 2024-09-20T22:46:15.241 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:46:15.242 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.242 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.242 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:46:15.242 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:37.216841+0000", 2024-09-20T22:46:15.242 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:46:15.242 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.242 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.242 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:46:15.242 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:37.216842+0000", 2024-09-20T22:46:15.242 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.1399999999999997e-07 2024-09-20T22:46:15.242 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.242 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.242 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:46:15.243 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:37.221497+0000", 2024-09-20T22:46:15.243 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0046552140000000004 2024-09-20T22:46:15.243 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.243 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.243 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:46:15.243 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:37.221522+0000", 2024-09-20T22:46:15.243 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.4732999999999999e-05 2024-09-20T22:46:15.243 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.243 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.243 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:46:15.243 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:37.221536+0000", 2024-09-20T22:46:15.243 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.4382000000000001e-05 2024-09-20T22:46:15.243 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.243 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.244 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:46:15.244 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:37.221631+0000", 2024-09-20T22:46:15.244 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.4943000000000002e-05 2024-09-20T22:46:15.244 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.244 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.244 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:46:15.244 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:37.221662+0000", 2024-09-20T22:46:15.244 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.0480999999999998e-05 2024-09-20T22:46:15.244 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.244 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.244 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:46:15.244 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:37.227657+0000", 2024-09-20T22:46:15.244 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0059949970000000002 2024-09-20T22:46:15.244 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.244 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.245 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:46:15.245 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:37.227718+0000", 2024-09-20T22:46:15.245 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.1286999999999998e-05 2024-09-20T22:46:15.245 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:46:15.245 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:46:15.245 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:46:15.245 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.245 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.245 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24652.0:1654 7.19 e206/139 7:9970b578:::smithi08477203-15:head v 206'114, mlcod=205'111)", 2024-09-20T22:46:15.245 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:44:45.167994+0000", 2024-09-20T22:46:15.245 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 90.039159945999998, 2024-09-20T22:46:15.245 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.01345262, 2024-09-20T22:46:15.245 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:46:15.245 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:46:15.246 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:46:15.246 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:46:15.246 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.246 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:46:15.246 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:45.167994+0000", 2024-09-20T22:46:15.246 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:46:15.246 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.246 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.246 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:46:15.246 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:45.167994+0000", 2024-09-20T22:46:15.246 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:46:15.246 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.246 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.246 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:46:15.247 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:45.167996+0000", 2024-09-20T22:46:15.247 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.5100000000000001e-06 2024-09-20T22:46:15.247 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.247 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.247 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:46:15.247 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:45.175169+0000", 2024-09-20T22:46:15.247 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0071732740000000003 2024-09-20T22:46:15.247 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.247 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.247 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:46:15.247 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:45.175208+0000", 2024-09-20T22:46:15.247 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.8597000000000003e-05 2024-09-20T22:46:15.247 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.247 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.248 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:46:15.248 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:45.175223+0000", 2024-09-20T22:46:15.248 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.4627000000000001e-05 2024-09-20T22:46:15.248 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.248 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.248 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:46:15.248 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:45.175304+0000", 2024-09-20T22:46:15.248 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.1111000000000003e-05 2024-09-20T22:46:15.248 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.248 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.248 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:46:15.248 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:45.175336+0000", 2024-09-20T22:46:15.248 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.1945e-05 2024-09-20T22:46:15.248 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.248 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.248 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:46:15.248 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:45.181392+0000", 2024-09-20T22:46:15.249 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0060566630000000003 2024-09-20T22:46:15.249 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.249 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.249 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:46:15.249 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:45.181446+0000", 2024-09-20T22:46:15.249 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.3893000000000001e-05 2024-09-20T22:46:15.249 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:46:15.249 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:46:15.249 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:46:15.249 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.249 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.249 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24652.0:1799 7.17 e229/139 7:ea720b85:::smithi08477203-26:head v 229'168, mlcod=222'164)", 2024-09-20T22:46:15.249 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:45:08.171867+0000", 2024-09-20T22:46:15.249 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 67.035286728000003, 2024-09-20T22:46:15.249 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014913454, 2024-09-20T22:46:15.250 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:46:15.250 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:46:15.250 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:46:15.250 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:46:15.250 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.250 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:46:15.250 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.171867+0000", 2024-09-20T22:46:15.250 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:46:15.250 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.250 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.250 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:46:15.250 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.171867+0000", 2024-09-20T22:46:15.250 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:46:15.251 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.251 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.251 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:46:15.251 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.171867+0000", 2024-09-20T22:46:15.251 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.3799999999999999e-07 2024-09-20T22:46:15.251 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.251 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.251 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:46:15.251 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.173016+0000", 2024-09-20T22:46:15.251 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0011485740000000001 2024-09-20T22:46:15.251 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.251 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.251 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:46:15.252 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.173024+0000", 2024-09-20T22:46:15.252 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.4780000000000008e-06 2024-09-20T22:46:15.252 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.252 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.252 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:46:15.252 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.173034+0000", 2024-09-20T22:46:15.252 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.7860000000000008e-06 2024-09-20T22:46:15.252 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.252 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.252 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:46:15.252 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.173098+0000", 2024-09-20T22:46:15.252 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.4405000000000004e-05 2024-09-20T22:46:15.252 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.253 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.253 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:46:15.253 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.173125+0000", 2024-09-20T22:46:15.253 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.6950000000000001e-05 2024-09-20T22:46:15.253 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.253 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.253 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:46:15.253 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.186711+0000", 2024-09-20T22:46:15.253 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.01358608 2024-09-20T22:46:15.253 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.253 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.253 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:46:15.253 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.186780+0000", 2024-09-20T22:46:15.253 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.8942999999999994e-05 2024-09-20T22:46:15.254 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:46:15.254 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:46:15.254 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:46:15.254 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.254 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.254 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24652.0:1800 7.17 e229/139 7:ea720b85:::smithi08477203-26:head v 229'169, mlcod=222'164)", 2024-09-20T22:46:15.254 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:45:08.178177+0000", 2024-09-20T22:46:15.254 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 67.028976912000005, 2024-09-20T22:46:15.254 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011399595, 2024-09-20T22:46:15.254 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:46:15.254 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:46:15.254 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:46:15.254 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:46:15.254 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.255 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:46:15.255 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.178177+0000", 2024-09-20T22:46:15.255 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:46:15.255 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.255 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.255 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:46:15.255 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.178177+0000", 2024-09-20T22:46:15.255 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:46:15.255 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.255 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.255 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:46:15.255 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.178178+0000", 2024-09-20T22:46:15.256 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2389999999999999e-06 2024-09-20T22:46:15.256 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.256 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.256 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:46:15.256 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.179005+0000", 2024-09-20T22:46:15.256 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00082757900000000005 2024-09-20T22:46:15.256 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.256 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.256 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:46:15.256 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.179016+0000", 2024-09-20T22:46:15.256 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0404999999999999e-05 2024-09-20T22:46:15.256 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.256 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.257 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:46:15.257 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.179023+0000", 2024-09-20T22:46:15.257 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.2069999999999998e-06 2024-09-20T22:46:15.257 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.257 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.257 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:46:15.257 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.179088+0000", 2024-09-20T22:46:15.257 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.4783000000000006e-05 2024-09-20T22:46:15.257 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.257 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.257 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:46:15.257 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.179107+0000", 2024-09-20T22:46:15.257 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.9293999999999999e-05 2024-09-20T22:46:15.257 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.258 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.258 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:46:15.258 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.189518+0000", 2024-09-20T22:46:15.258 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010410739 2024-09-20T22:46:15.258 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.258 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.258 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:46:15.258 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.189576+0000", 2024-09-20T22:46:15.258 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.8349e-05 2024-09-20T22:46:15.258 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:46:15.258 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:46:15.258 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:46:15.258 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.259 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.259 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24652.0:1806 7.a e229/94 7:5aaef466:::smithi08477203-40:head v 229'139, mlcod=229'136)", 2024-09-20T22:46:15.259 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:45:08.178370+0000", 2024-09-20T22:46:15.259 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 67.028783798000006, 2024-09-20T22:46:15.259 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011334828999999999, 2024-09-20T22:46:15.259 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:46:15.259 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:46:15.259 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:46:15.259 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:46:15.259 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.259 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:46:15.259 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.178370+0000", 2024-09-20T22:46:15.259 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:46:15.259 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.260 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.260 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:46:15.260 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.178370+0000", 2024-09-20T22:46:15.260 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:46:15.260 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.260 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.260 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:46:15.260 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.178371+0000", 2024-09-20T22:46:15.260 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.9599999999999998e-07 2024-09-20T22:46:15.260 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.260 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.260 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:46:15.260 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.178381+0000", 2024-09-20T22:46:15.260 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.8539999999999992e-06 2024-09-20T22:46:15.260 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.261 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.261 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:46:15.261 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.178398+0000", 2024-09-20T22:46:15.261 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.7303999999999999e-05 2024-09-20T22:46:15.261 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.261 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.261 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:46:15.261 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.178410+0000", 2024-09-20T22:46:15.261 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1827e-05 2024-09-20T22:46:15.261 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.261 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.261 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:46:15.261 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.178545+0000", 2024-09-20T22:46:15.261 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013498199999999999 2024-09-20T22:46:15.262 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.262 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.262 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:46:15.262 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.178593+0000", 2024-09-20T22:46:15.262 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.8167999999999997e-05 2024-09-20T22:46:15.262 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.262 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.262 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:46:15.262 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.189673+0000", 2024-09-20T22:46:15.262 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011080491 2024-09-20T22:46:15.262 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.262 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.262 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:46:15.262 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.189705+0000", 2024-09-20T22:46:15.263 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.1306999999999997e-05 2024-09-20T22:46:15.263 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:46:15.263 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:46:15.263 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:46:15.263 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.263 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.263 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24652.0:1809 7.16 e229/157 7:6885a65f:::smithi08477203-14:head v 229'53, mlcod=222'49)", 2024-09-20T22:46:15.263 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:45:08.178454+0000", 2024-09-20T22:46:15.263 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 67.028699162999999, 2024-09-20T22:46:15.263 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011210886, 2024-09-20T22:46:15.263 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:46:15.263 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:46:15.263 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:46:15.264 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:46:15.264 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.264 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:46:15.264 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.178454+0000", 2024-09-20T22:46:15.264 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:46:15.264 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.264 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.264 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:46:15.264 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.178454+0000", 2024-09-20T22:46:15.264 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:46:15.264 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.264 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.264 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:46:15.264 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.178455+0000", 2024-09-20T22:46:15.265 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.05e-07 2024-09-20T22:46:15.265 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.265 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.265 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:46:15.265 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.178460+0000", 2024-09-20T22:46:15.265 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.4380000000000003e-06 2024-09-20T22:46:15.265 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.265 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.265 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:46:15.265 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.178469+0000", 2024-09-20T22:46:15.265 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.4649999999999994e-06 2024-09-20T22:46:15.265 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.265 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.265 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:46:15.265 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.178473+0000", 2024-09-20T22:46:15.266 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.9250000000000003e-06 2024-09-20T22:46:15.266 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.266 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.266 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:46:15.266 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.178705+0000", 2024-09-20T22:46:15.266 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00023114899999999999 2024-09-20T22:46:15.266 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.266 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.266 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:46:15.266 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.178735+0000", 2024-09-20T22:46:15.266 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.0889000000000002e-05 2024-09-20T22:46:15.266 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.266 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.266 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:46:15.267 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.189631+0000", 2024-09-20T22:46:15.267 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010895317 2024-09-20T22:46:15.267 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.267 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.267 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:46:15.267 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.189665+0000", 2024-09-20T22:46:15.267 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.4498000000000001e-05 2024-09-20T22:46:15.267 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:46:15.267 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:46:15.267 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:46:15.267 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.267 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.268 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:15.268 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:46:15.269 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24652.0:1807 7.a e229/94 7:5aaef466:::smithi08477203-40:head v 229'140, mlcod=229'136)", 2024-09-20T22:46:15.269 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:45:08.178940+0000", 2024-09-20T22:46:15.269 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 67.028213757000003, 2024-09-20T22:46:15.269 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010891029999999999, 2024-09-20T22:46:15.269 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:46:15.269 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:46:15.269 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:46:15.269 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:46:15.269 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.269 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:46:15.269 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.178940+0000", 2024-09-20T22:46:15.269 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:46:15.270 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.270 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.270 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:46:15.270 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.178940+0000", 2024-09-20T22:46:15.270 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:46:15.270 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.270 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.270 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:46:15.270 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.178941+0000", 2024-09-20T22:46:15.270 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.75e-07 2024-09-20T22:46:15.270 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.270 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.271 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:46:15.271 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.178946+0000", 2024-09-20T22:46:15.271 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.8340000000000004e-06 2024-09-20T22:46:15.271 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.271 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.271 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:46:15.271 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.178959+0000", 2024-09-20T22:46:15.271 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3156000000000001e-05 2024-09-20T22:46:15.271 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.271 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.271 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:46:15.272 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.178970+0000", 2024-09-20T22:46:15.272 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0086e-05 2024-09-20T22:46:15.272 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.272 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.272 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:46:15.272 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.180323+0000", 2024-09-20T22:46:15.272 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001353808 2024-09-20T22:46:15.272 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.272 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.272 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:46:15.272 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.180345+0000", 2024-09-20T22:46:15.272 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.1409000000000001e-05 2024-09-20T22:46:15.272 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.272 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.273 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:46:15.273 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.189800+0000", 2024-09-20T22:46:15.273 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0094553980000000003 2024-09-20T22:46:15.273 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.273 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.273 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:46:15.273 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.189831+0000", 2024-09-20T22:46:15.273 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.0664000000000003e-05 2024-09-20T22:46:15.273 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:46:15.273 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:46:15.273 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:46:15.273 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.273 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.274 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:2321 7.8 7:12e9c8db:::smithi08477203-11:head [rollback 3e] snapc 55=[55,53,52,51,50,4b,47,3e] ondisk+write+known_if_redirected+supports_pool_eio e258)", 2024-09-20T22:46:15.274 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:45:37.176065+0000", 2024-09-20T22:46:15.274 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 38.031088853999997, 2024-09-20T22:46:15.274 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.012489495, 2024-09-20T22:46:15.274 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:46:15.274 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:46:15.274 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:46:15.274 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:46:15.274 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:46:15.274 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:46:15.274 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 2321 2024-09-20T22:46:15.274 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.274 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:46:15.274 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.275 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:46:15.275 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.176065+0000", 2024-09-20T22:46:15.275 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:46:15.275 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.275 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.275 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:46:15.275 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.176065+0000", 2024-09-20T22:46:15.275 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:46:15.275 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.275 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.275 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:46:15.275 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.176065+0000", 2024-09-20T22:46:15.275 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.3e-07 2024-09-20T22:46:15.276 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.276 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.276 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:46:15.276 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.176069+0000", 2024-09-20T22:46:15.276 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.8010000000000001e-06 2024-09-20T22:46:15.276 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.276 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.276 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:46:15.276 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.176076+0000", 2024-09-20T22:46:15.276 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.6599999999999998e-06 2024-09-20T22:46:15.276 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.276 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.276 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:46:15.277 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.176086+0000", 2024-09-20T22:46:15.277 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.9720000000000004e-06 2024-09-20T22:46:15.277 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.277 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.277 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:46:15.277 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.176134+0000", 2024-09-20T22:46:15.277 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.7778999999999998e-05 2024-09-20T22:46:15.277 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.277 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.277 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting_for_map not empty", 2024-09-20T22:46:15.277 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.176139+0000", 2024-09-20T22:46:15.277 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.7699999999999998e-06 2024-09-20T22:46:15.277 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.277 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.278 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:46:15.278 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.181825+0000", 2024-09-20T22:46:15.278 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.005685921 2024-09-20T22:46:15.278 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.278 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.278 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:46:15.278 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.181919+0000", 2024-09-20T22:46:15.278 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.3630999999999999e-05 2024-09-20T22:46:15.278 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.278 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.278 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 3,7", 2024-09-20T22:46:15.279 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.182118+0000", 2024-09-20T22:46:15.279 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00019958500000000001 2024-09-20T22:46:15.279 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.279 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.279 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:46:15.279 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.185052+0000", 2024-09-20T22:46:15.279 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0029336620000000001 2024-09-20T22:46:15.279 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.279 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.279 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:46:15.279 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.185756+0000", 2024-09-20T22:46:15.279 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00070376499999999997 2024-09-20T22:46:15.279 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.279 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.280 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:46:15.280 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.188120+0000", 2024-09-20T22:46:15.280 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0023645739999999999 2024-09-20T22:46:15.280 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.280 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.280 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:46:15.280 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.188207+0000", 2024-09-20T22:46:15.280 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.6562000000000002e-05 2024-09-20T22:46:15.280 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.280 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.280 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:46:15.280 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.188554+0000", 2024-09-20T22:46:15.280 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00034718299999999999 2024-09-20T22:46:15.280 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:46:15.280 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:46:15.281 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:46:15.281 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.281 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.281 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:2322 7.8 7:12e9c8db:::smithi08477203-11:head [append 0~0] snapc 55=[55,53,52,51,50,4b,47,3e] ondisk+write+known_if_redirected+supports_pool_eio e258)", 2024-09-20T22:46:15.281 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:45:37.176152+0000", 2024-09-20T22:46:15.281 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 38.031001938000003, 2024-09-20T22:46:15.281 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013033472000000001, 2024-09-20T22:46:15.281 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:46:15.281 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:46:15.281 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:46:15.281 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:46:15.281 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:46:15.281 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:46:15.281 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 2322 2024-09-20T22:46:15.281 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.282 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:46:15.282 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.282 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:46:15.282 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.176152+0000", 2024-09-20T22:46:15.282 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:46:15.282 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.282 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.282 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:46:15.282 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.176152+0000", 2024-09-20T22:46:15.282 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:46:15.282 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.282 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.282 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:46:15.282 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.176152+0000", 2024-09-20T22:46:15.283 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.8899999999999999e-07 2024-09-20T22:46:15.283 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.283 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.283 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:46:15.283 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.176156+0000", 2024-09-20T22:46:15.283 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.2669999999999998e-06 2024-09-20T22:46:15.283 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.283 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.283 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:46:15.283 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.176162+0000", 2024-09-20T22:46:15.283 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.1589999999999998e-06 2024-09-20T22:46:15.283 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.283 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.283 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:46:15.283 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.176167+0000", 2024-09-20T22:46:15.284 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.3040000000000004e-06 2024-09-20T22:46:15.284 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.284 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.284 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:46:15.284 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.176212+0000", 2024-09-20T22:46:15.284 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.5448999999999998e-05 2024-09-20T22:46:15.284 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.284 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.284 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting_for_map not empty", 2024-09-20T22:46:15.284 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.176218+0000", 2024-09-20T22:46:15.284 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.5520000000000004e-06 2024-09-20T22:46:15.284 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.284 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.284 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:46:15.285 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.182721+0000", 2024-09-20T22:46:15.285 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0065028550000000001 2024-09-20T22:46:15.285 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.285 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.285 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:46:15.285 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.182787+0000", 2024-09-20T22:46:15.285 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.5995999999999996e-05 2024-09-20T22:46:15.285 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.285 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.285 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 3,7", 2024-09-20T22:46:15.285 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.183002+0000", 2024-09-20T22:46:15.285 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00021516699999999999 2024-09-20T22:46:15.285 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.285 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.286 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:46:15.286 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.185137+0000", 2024-09-20T22:46:15.286 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0021350570000000001 2024-09-20T22:46:15.286 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.286 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.286 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:46:15.286 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.185875+0000", 2024-09-20T22:46:15.286 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00073794799999999999 2024-09-20T22:46:15.286 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.286 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.286 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:46:15.286 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.188620+0000", 2024-09-20T22:46:15.287 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0027446229999999999 2024-09-20T22:46:15.287 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.287 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.287 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:46:15.287 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.189088+0000", 2024-09-20T22:46:15.287 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00046794499999999999 2024-09-20T22:46:15.287 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.287 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.287 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:46:15.287 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.189185+0000", 2024-09-20T22:46:15.287 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.7560999999999996e-05 2024-09-20T22:46:15.287 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:46:15.287 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:46:15.287 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:46:15.287 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.288 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.288 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24652.0:2318 7.5 e258/137 7:a160b5d4:::smithi08477203-16:head v 258'43, mlcod=239'39)", 2024-09-20T22:46:15.288 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:45:37.192419+0000", 2024-09-20T22:46:15.288 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 38.014734640999997, 2024-09-20T22:46:15.288 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013974723, 2024-09-20T22:46:15.288 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:46:15.288 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:46:15.288 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:46:15.288 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:46:15.288 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.288 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:46:15.288 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.192419+0000", 2024-09-20T22:46:15.288 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:46:15.288 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.289 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.289 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:46:15.289 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.192419+0000", 2024-09-20T22:46:15.289 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:46:15.289 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.289 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.289 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:46:15.289 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.192420+0000", 2024-09-20T22:46:15.289 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.3799999999999997e-07 2024-09-20T22:46:15.289 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.289 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.289 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:46:15.290 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.204088+0000", 2024-09-20T22:46:15.290 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011668883 2024-09-20T22:46:15.290 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.290 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.290 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:46:15.290 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.204104+0000", 2024-09-20T22:46:15.290 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.5437000000000001e-05 2024-09-20T22:46:15.290 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.290 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.290 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:46:15.290 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.204112+0000", 2024-09-20T22:46:15.290 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.2519999999999999e-06 2024-09-20T22:46:15.290 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.291 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.291 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:46:15.291 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.204168+0000", 2024-09-20T22:46:15.291 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.5977999999999998e-05 2024-09-20T22:46:15.291 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.291 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.291 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:46:15.291 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.204188+0000", 2024-09-20T22:46:15.291 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.9400999999999999e-05 2024-09-20T22:46:15.291 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.291 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.291 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:46:15.291 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.206340+0000", 2024-09-20T22:46:15.292 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0021527159999999998 2024-09-20T22:46:15.292 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.292 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.292 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:46:15.292 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.206394+0000", 2024-09-20T22:46:15.292 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.3418000000000003e-05 2024-09-20T22:46:15.292 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:46:15.292 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:46:15.292 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:46:15.292 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.292 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.292 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(osd.6.0:726 7.19 e260/139 7:9eac4560:::smithi08477203-30:4c v 260'159, mlcod=259'156)", 2024-09-20T22:46:15.292 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:45:39.197025+0000", 2024-09-20T22:46:15.292 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 36.010128731000002, 2024-09-20T22:46:15.293 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.018150454, 2024-09-20T22:46:15.293 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:46:15.293 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:46:15.293 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:46:15.293 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:46:15.293 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.293 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:46:15.293 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.197025+0000", 2024-09-20T22:46:15.293 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:46:15.293 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.293 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.293 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:46:15.293 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.197025+0000", 2024-09-20T22:46:15.293 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:46:15.294 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.294 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.294 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:46:15.294 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.197025+0000", 2024-09-20T22:46:15.294 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.7599999999999998e-07 2024-09-20T22:46:15.294 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.294 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.294 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:46:15.294 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.197033+0000", 2024-09-20T22:46:15.294 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.9119999999999998e-06 2024-09-20T22:46:15.294 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.294 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.294 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:46:15.294 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.197042+0000", 2024-09-20T22:46:15.295 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.3349999999999996e-06 2024-09-20T22:46:15.295 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.295 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.295 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:46:15.295 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.197050+0000", 2024-09-20T22:46:15.295 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.7230000000000004e-06 2024-09-20T22:46:15.295 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.295 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.295 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:46:15.295 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.200756+0000", 2024-09-20T22:46:15.295 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0037059749999999998 2024-09-20T22:46:15.295 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.295 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.295 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:46:15.296 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.200778+0000", 2024-09-20T22:46:15.296 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.2140000000000001e-05 2024-09-20T22:46:15.296 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.296 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.296 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:46:15.296 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.215131+0000", 2024-09-20T22:46:15.296 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014353054000000001 2024-09-20T22:46:15.296 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.296 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.296 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:46:15.296 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.215175+0000", 2024-09-20T22:46:15.296 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.4039000000000003e-05 2024-09-20T22:46:15.296 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:46:15.296 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:46:15.296 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:46:15.297 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.297 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.297 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(osd.7.0:152 7.5 e269/260 7:a160b5d4:::smithi08477203-16:55 v 269'55, mlcod=263'51)", 2024-09-20T22:46:15.297 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:45:48.173893+0000", 2024-09-20T22:46:15.297 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 27.033260989999999, 2024-09-20T22:46:15.297 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014936935, 2024-09-20T22:46:15.297 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:46:15.297 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:46:15.297 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:46:15.297 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:46:15.297 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.297 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:46:15.297 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.173893+0000", 2024-09-20T22:46:15.298 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:46:15.298 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.298 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.298 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:46:15.298 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.173893+0000", 2024-09-20T22:46:15.298 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:46:15.298 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.298 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.298 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:46:15.298 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.173893+0000", 2024-09-20T22:46:15.298 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.8800000000000002e-07 2024-09-20T22:46:15.298 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.299 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.299 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:46:15.299 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.173916+0000", 2024-09-20T22:46:15.299 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.2889999999999999e-05 2024-09-20T22:46:15.299 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.299 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.299 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:46:15.299 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.173924+0000", 2024-09-20T22:46:15.299 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.4420000000000001e-06 2024-09-20T22:46:15.299 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.299 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.299 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:46:15.299 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.173932+0000", 2024-09-20T22:46:15.300 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.5299999999999996e-06 2024-09-20T22:46:15.300 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.300 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.300 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:46:15.300 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.174011+0000", 2024-09-20T22:46:15.300 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.9342e-05 2024-09-20T22:46:15.300 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.300 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.300 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:46:15.300 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.174047+0000", 2024-09-20T22:46:15.300 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.5237999999999997e-05 2024-09-20T22:46:15.300 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.300 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.301 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:46:15.301 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.188782+0000", 2024-09-20T22:46:15.301 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014735599 2024-09-20T22:46:15.301 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.301 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.301 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:46:15.301 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.188830+0000", 2024-09-20T22:46:15.301 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.7206000000000002e-05 2024-09-20T22:46:15.301 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:46:15.301 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:46:15.301 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:46:15.301 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.301 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.301 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(osd.7.0:153 7.17 e269/139 7:ea720b85:::smithi08477203-26:55 v 269'225, mlcod=263'221)", 2024-09-20T22:46:15.302 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:45:48.173991+0000", 2024-09-20T22:46:15.302 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 27.033162509, 2024-09-20T22:46:15.302 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013064868, 2024-09-20T22:46:15.302 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:46:15.302 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:46:15.302 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:46:15.302 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:46:15.302 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.302 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:46:15.302 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.173991+0000", 2024-09-20T22:46:15.302 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:46:15.302 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.302 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.302 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:46:15.303 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.173991+0000", 2024-09-20T22:46:15.303 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:46:15.303 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.303 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.303 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:46:15.303 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.173991+0000", 2024-09-20T22:46:15.303 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.9999999999999999e-07 2024-09-20T22:46:15.303 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.303 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.303 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:46:15.303 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.173996+0000", 2024-09-20T22:46:15.303 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.3080000000000001e-06 2024-09-20T22:46:15.303 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.303 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.304 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:46:15.304 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.174001+0000", 2024-09-20T22:46:15.304 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.203e-06 2024-09-20T22:46:15.304 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.304 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.304 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:46:15.304 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.174005+0000", 2024-09-20T22:46:15.304 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.4880000000000001e-06 2024-09-20T22:46:15.304 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.304 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.304 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:46:15.304 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.175600+0000", 2024-09-20T22:46:15.304 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015942459999999999 2024-09-20T22:46:15.304 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.305 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.305 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:46:15.305 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.175631+0000", 2024-09-20T22:46:15.305 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.1816000000000002e-05 2024-09-20T22:46:15.305 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.305 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.305 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:46:15.305 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.187003+0000", 2024-09-20T22:46:15.305 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011372006 2024-09-20T22:46:15.305 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.305 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.305 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:46:15.305 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.187056+0000", 2024-09-20T22:46:15.305 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.2500999999999998e-05 2024-09-20T22:46:15.305 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:46:15.306 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:46:15.306 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:46:15.306 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.306 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.306 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(osd.6.0:767 7.12 e269/263 7:4fa4b2b3:::smithi08477203-21:53 v 269'262, mlcod=265'259)", 2024-09-20T22:46:15.306 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:45:48.175130+0000", 2024-09-20T22:46:15.306 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 27.03202332, 2024-09-20T22:46:15.306 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013842727000000001, 2024-09-20T22:46:15.306 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:46:15.306 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:46:15.306 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:46:15.306 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:46:15.307 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.307 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:46:15.307 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.175130+0000", 2024-09-20T22:46:15.307 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:46:15.307 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.307 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.307 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:46:15.307 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.175130+0000", 2024-09-20T22:46:15.307 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:46:15.307 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.307 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.307 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:46:15.307 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.175130+0000", 2024-09-20T22:46:15.307 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.98e-07 2024-09-20T22:46:15.308 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.308 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.308 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:46:15.308 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.175139+0000", 2024-09-20T22:46:15.308 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.5909999999999996e-06 2024-09-20T22:46:15.308 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.308 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.308 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:46:15.308 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.175144+0000", 2024-09-20T22:46:15.308 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.361e-06 2024-09-20T22:46:15.308 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.308 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.308 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:46:15.309 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.175156+0000", 2024-09-20T22:46:15.309 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1236999999999999e-05 2024-09-20T22:46:15.309 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.309 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.309 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:46:15.309 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.176359+0000", 2024-09-20T22:46:15.309 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0012036060000000001 2024-09-20T22:46:15.309 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.309 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.309 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:46:15.309 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.176380+0000", 2024-09-20T22:46:15.309 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.0497999999999999e-05 2024-09-20T22:46:15.309 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.310 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.310 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:46:15.310 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.188936+0000", 2024-09-20T22:46:15.310 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.012556286999999999 2024-09-20T22:46:15.310 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:15.310 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:15.310 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:46:15.310 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.188973+0000", 2024-09-20T22:46:15.310 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.6949000000000001e-05 2024-09-20T22:46:15.310 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:46:15.310 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:46:15.310 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:46:15.310 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:46:15.310 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:46:15.311 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-20T22:46:15.368 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:46:15.390 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:15.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:15.391 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:46:15.492 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:46:15.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:15 smithi154 ceph-mon[33264]: pgmap v343: 170 pgs: 3 active+clean+snaptrim, 2 clean+premerge+peered, 1 active+clean+snaptrim_wait, 164 active+clean; 113 MiB data, 521 MiB used, 536 GiB / 536 GiB avail; 2.0 KiB/s rd, 631 KiB/s wr, 5 op/s 2024-09-20T22:46:15.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:15 smithi154 ceph-mon[33264]: osdmap e295: 8 total, 8 up, 6 in 2024-09-20T22:46:15.512 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:15.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:15.513 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:46:15.614 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:46:15.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:15 smithi084 ceph-mon[40024]: pgmap v343: 170 pgs: 3 active+clean+snaptrim, 2 clean+premerge+peered, 1 active+clean+snaptrim_wait, 164 active+clean; 113 MiB data, 521 MiB used, 536 GiB / 536 GiB avail; 2.0 KiB/s rd, 631 KiB/s wr, 5 op/s 2024-09-20T22:46:15.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:15 smithi084 ceph-mon[40024]: osdmap e295: 8 total, 8 up, 6 in 2024-09-20T22:46:15.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:15 smithi084 ceph-mon[29774]: pgmap v343: 170 pgs: 3 active+clean+snaptrim, 2 clean+premerge+peered, 1 active+clean+snaptrim_wait, 164 active+clean; 113 MiB data, 521 MiB used, 536 GiB / 536 GiB avail; 2.0 KiB/s rd, 631 KiB/s wr, 5 op/s 2024-09-20T22:46:15.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:15 smithi084 ceph-mon[29774]: osdmap e295: 8 total, 8 up, 6 in 2024-09-20T22:46:15.635 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:15.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:15.636 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:46:15.737 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:46:15.757 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:15.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:15.759 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:46:15.859 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:46:15.886 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:15.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:15.887 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:46:15.988 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:46:16.013 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:16.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:16.014 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:46:16.102 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-20T22:46:16.114 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:46:16.140 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:16.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:16.141 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:46:16.184 INFO:tasks.rados.rados.0.smithi084.stdout:1016: done (4 left) 2024-09-20T22:46:16.184 INFO:tasks.rados.rados.0.smithi084.stdout:1017: done (3 left) 2024-09-20T22:46:16.184 INFO:tasks.rados.rados.0.smithi084.stdout:1019: done (2 left) 2024-09-20T22:46:16.185 INFO:tasks.rados.rados.0.smithi084.stdout:1020: done (1 left) 2024-09-20T22:46:16.185 INFO:tasks.rados.rados.0.smithi084.stdout:1021: done (0 left) 2024-09-20T22:46:16.185 INFO:tasks.rados.rados.0.smithi084.stdout:1022: rollback oid 37 current snap is 100 2024-09-20T22:46:16.185 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 37 to 97 2024-09-20T22:46:16.185 INFO:tasks.rados.rados.0.smithi084.stdout:1023: read oid 32 snap -1 2024-09-20T22:46:16.185 INFO:tasks.rados.rados.0.smithi084.stdout:1023: expect (ObjNum 375 snap 99 seq_num 375) 2024-09-20T22:46:16.185 INFO:tasks.rados.rados.0.smithi084.stdout:1024: rollback oid 25 current snap is 100 2024-09-20T22:46:16.185 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 25 to 97 2024-09-20T22:46:16.185 INFO:tasks.rados.rados.0.smithi084.stdout:1025: write (excl) oid 29 current snap is 100 2024-09-20T22:46:16.185 INFO:tasks.rados.rados.0.smithi084.stdout:1025: seq_num 379 ranges {769853=620668,1845146=524707} 2024-09-20T22:46:16.189 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:46:16.192 INFO:tasks.rados.rados.0.smithi084.stdout:1025: writing smithi08477203-29 from 769853 to 1390521 tid 1 2024-09-20T22:46:16.198 INFO:tasks.rados.rados.0.smithi084.stdout:1025: writing smithi08477203-29 from 1845146 to 2369853 tid 2 2024-09-20T22:46:16.198 INFO:tasks.rados.rados.0.smithi084.stdout:1026: write oid 20 current snap is 100 2024-09-20T22:46:16.198 INFO:tasks.rados.rados.0.smithi084.stdout:1026: seq_num 380 ranges {786660=695917,1903628=435462} 2024-09-20T22:46:16.206 INFO:tasks.rados.rados.0.smithi084.stdout:1026: writing smithi08477203-20 from 786660 to 1482577 tid 1 2024-09-20T22:46:16.211 INFO:tasks.rados.rados.0.smithi084.stdout:1026: writing smithi08477203-20 from 1903628 to 2339090 tid 2 2024-09-20T22:46:16.211 INFO:tasks.rados.rados.0.smithi084.stdout:1027: delete oid 26 current snap is 100 2024-09-20T22:46:16.212 INFO:tasks.rados.rados.0.smithi084.stdout:1023: expect (ObjNum 375 snap 99 seq_num 375) 2024-09-20T22:46:16.242 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:46:16.263 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:16.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:16.264 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:46:16.364 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:16 smithi084 ceph-mon[40024]: osdmap e296: 8 total, 8 up, 6 in 2024-09-20T22:46:16.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:16 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-09-20T22:46:16.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:16 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-09-20T22:46:16.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:16 smithi084 ceph-mon[29774]: osdmap e296: 8 total, 8 up, 6 in 2024-09-20T22:46:16.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:16 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-09-20T22:46:16.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:16 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-09-20T22:46:16.365 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:46:16.370 INFO:tasks.rados.rados.0.smithi084.stdout:1024: finishing rollback tid 0 to smithi08477203-25 2024-09-20T22:46:16.370 INFO:tasks.rados.rados.0.smithi084.stdout:1024: finishing rollback tid 1 to smithi08477203-25 2024-09-20T22:46:16.371 INFO:tasks.rados.rados.0.smithi084.stdout:1024: finishing rollback tid 2 to smithi08477203-25 2024-09-20T22:46:16.371 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 25 v 558 (ObjNum 369 snap 96 seq_num 369) dirty exists 2024-09-20T22:46:16.371 INFO:tasks.rados.rados.0.smithi084.stdout:1022: finishing rollback tid 0 to smithi08477203-37 2024-09-20T22:46:16.371 INFO:tasks.rados.rados.0.smithi084.stdout:1022: finishing rollback tid 1 to smithi08477203-37 2024-09-20T22:46:16.371 INFO:tasks.rados.rados.0.smithi084.stdout:1022: finishing rollback tid 2 to smithi08477203-37 2024-09-20T22:46:16.371 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 37 v 228 (ObjNum 329 snap 84 seq_num 329) dirty exists 2024-09-20T22:46:16.371 INFO:tasks.rados.rados.0.smithi084.stdout:1025: finishing write tid 1 to smithi08477203-29 2024-09-20T22:46:16.371 INFO:tasks.rados.rados.0.smithi084.stdout:1022: done (5 left) 2024-09-20T22:46:16.371 INFO:tasks.rados.rados.0.smithi084.stdout:1023: done (4 left) 2024-09-20T22:46:16.371 INFO:tasks.rados.rados.0.smithi084.stdout:1024: done (3 left) 2024-09-20T22:46:16.371 INFO:tasks.rados.rados.0.smithi084.stdout:1027: done (2 left) 2024-09-20T22:46:16.371 INFO:tasks.rados.rados.0.smithi084.stdout:1028: write oid 50 current snap is 100 2024-09-20T22:46:16.371 INFO:tasks.rados.rados.0.smithi084.stdout:1025: finishing write tid 2 to smithi08477203-29 2024-09-20T22:46:16.372 INFO:tasks.rados.rados.0.smithi084.stdout:1025: finishing write tid 3 to smithi08477203-29 2024-09-20T22:46:16.372 INFO:tasks.rados.rados.0.smithi084.stdout:1025: finishing write tid 4 to smithi08477203-29 2024-09-20T22:46:16.372 INFO:tasks.rados.rados.0.smithi084.stdout:1025: oid 29 updating version 0 to 562 2024-09-20T22:46:16.372 INFO:tasks.rados.rados.0.smithi084.stdout:1025: oid 29 version 562 is already newer than 560 2024-09-20T22:46:16.372 INFO:tasks.rados.rados.0.smithi084.stdout:1025: oid 29 version 562 is already newer than 561 2024-09-20T22:46:16.372 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 29 v 562 (ObjNum 379 snap 100 seq_num 379) dirty exists 2024-09-20T22:46:16.372 INFO:tasks.rados.rados.0.smithi084.stdout:1025: left oid 29 (ObjNum 379 snap 100 seq_num 379) 2024-09-20T22:46:16.372 INFO:tasks.rados.rados.0.smithi084.stdout:1026: finishing write tid 1 to smithi08477203-20 2024-09-20T22:46:16.372 INFO:tasks.rados.rados.0.smithi084.stdout:1026: finishing write tid 2 to smithi08477203-20 2024-09-20T22:46:16.372 INFO:tasks.rados.rados.0.smithi084.stdout:1028: seq_num 382 ranges {420274=562768,1620592=439131} 2024-09-20T22:46:16.375 INFO:tasks.rados.rados.0.smithi084.stdout:1028: writing smithi08477203-50 from 420274 to 983042 tid 1 2024-09-20T22:46:16.382 INFO:tasks.rados.rados.0.smithi084.stdout:1028: writing smithi08477203-50 from 1620592 to 2059723 tid 2 2024-09-20T22:46:16.382 INFO:tasks.rados.rados.0.smithi084.stdout:1025: done (2 left) 2024-09-20T22:46:16.382 INFO:tasks.rados.rados.0.smithi084.stdout:1029: copy_from oid 47 from oid 27 current snap is 100 2024-09-20T22:46:16.382 INFO:tasks.rados.rados.0.smithi084.stdout:1026: finishing write tid 3 to smithi08477203-20 2024-09-20T22:46:16.382 INFO:tasks.rados.rados.0.smithi084.stdout:1026: finishing write tid 4 to smithi08477203-20 2024-09-20T22:46:16.382 INFO:tasks.rados.rados.0.smithi084.stdout:1026: oid 20 updating version 0 to 88 2024-09-20T22:46:16.382 INFO:tasks.rados.rados.0.smithi084.stdout:1026: oid 20 updating version 88 to 90 2024-09-20T22:46:16.382 INFO:tasks.rados.rados.0.smithi084.stdout:1026: oid 20 version 90 is already newer than 89 2024-09-20T22:46:16.382 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 20 v 90 (ObjNum 380 snap 100 seq_num 380) dirty exists 2024-09-20T22:46:16.383 INFO:tasks.rados.rados.0.smithi084.stdout:1026: left oid 20 (ObjNum 380 snap 100 seq_num 380) 2024-09-20T22:46:16.383 INFO:tasks.rados.rados.0.smithi084.stdout:1028: finishing write tid 1 to smithi08477203-50 2024-09-20T22:46:16.383 INFO:tasks.rados.rados.0.smithi084.stdout:1026: done (2 left) 2024-09-20T22:46:16.383 INFO:tasks.rados.rados.0.smithi084.stdout:1030: copy_from oid 16 from oid 5 current snap is 100 2024-09-20T22:46:16.383 INFO:tasks.rados.rados.0.smithi084.stdout:1031: rollback oid 29 current snap is 100 2024-09-20T22:46:16.383 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 29 to 99 2024-09-20T22:46:16.383 INFO:tasks.rados.rados.0.smithi084.stdout:1032: write (excl) oid 10 current snap is 100 2024-09-20T22:46:16.383 INFO:tasks.rados.rados.0.smithi084.stdout:1032: seq_num 383 ranges {437081=638017,1679074=582789} 2024-09-20T22:46:16.387 INFO:tasks.rados.rados.0.smithi084.stdout:1032: writing smithi08477203-10 from 437081 to 1075098 tid 1 2024-09-20T22:46:16.390 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:16.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:16.391 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:46:16.395 INFO:tasks.rados.rados.0.smithi084.stdout:1032: writing smithi08477203-10 from 1679074 to 2261863 tid 2 2024-09-20T22:46:16.396 INFO:tasks.rados.rados.0.smithi084.stdout:1033: rollback oid 43 current snap is 100 2024-09-20T22:46:16.396 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 43 to 99 2024-09-20T22:46:16.396 INFO:tasks.rados.rados.0.smithi084.stdout:1029: finishing copy_from to smithi08477203-47 2024-09-20T22:46:16.396 INFO:tasks.rados.rados.0.smithi084.stdout:1029: got expected ENOENT (src dne) 2024-09-20T22:46:16.396 INFO:tasks.rados.rados.0.smithi084.stdout:1030: finishing copy_from to smithi08477203-16 2024-09-20T22:46:16.396 INFO:tasks.rados.rados.0.smithi084.stdout:1030: got expected ENOENT (src dne) 2024-09-20T22:46:16.396 INFO:tasks.rados.rados.0.smithi084.stdout:1029: finishing copy_from racing read to smithi08477203-47 2024-09-20T22:46:16.396 INFO:tasks.rados.rados.0.smithi084.stdout:1030: finishing copy_from racing read to smithi08477203-16 2024-09-20T22:46:16.396 INFO:tasks.rados.rados.0.smithi084.stdout:1028: finishing write tid 2 to smithi08477203-50 2024-09-20T22:46:16.396 INFO:tasks.rados.rados.0.smithi084.stdout:1028: finishing write tid 3 to smithi08477203-50 2024-09-20T22:46:16.396 INFO:tasks.rados.rados.0.smithi084.stdout:1031: finishing rollback tid 0 to smithi08477203-29 2024-09-20T22:46:16.397 INFO:tasks.rados.rados.0.smithi084.stdout:1028: finishing write tid 4 to smithi08477203-50 2024-09-20T22:46:16.397 INFO:tasks.rados.rados.0.smithi084.stdout:1028: oid 50 updating version 0 to 239 2024-09-20T22:46:16.397 INFO:tasks.rados.rados.0.smithi084.stdout:1028: oid 50 version 239 is already newer than 238 2024-09-20T22:46:16.397 INFO:tasks.rados.rados.0.smithi084.stdout:1028: oid 50 version 239 is already newer than 237 2024-09-20T22:46:16.397 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 50 v 239 (ObjNum 382 snap 100 seq_num 382) dirty exists 2024-09-20T22:46:16.397 INFO:tasks.rados.rados.0.smithi084.stdout:1028: left oid 50 (ObjNum 382 snap 100 seq_num 382) 2024-09-20T22:46:16.397 INFO:tasks.rados.rados.0.smithi084.stdout:1031: finishing rollback tid 1 to smithi08477203-29 2024-09-20T22:46:16.397 INFO:tasks.rados.rados.0.smithi084.stdout:1031: finishing rollback tid 2 to smithi08477203-29 2024-09-20T22:46:16.397 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 29 v 565 (ObjNum 339 snap 86 seq_num 339) dirty exists 2024-09-20T22:46:16.397 INFO:tasks.rados.rados.0.smithi084.stdout:1032: finishing write tid 1 to smithi08477203-10 2024-09-20T22:46:16.397 INFO:tasks.rados.rados.0.smithi084.stdout:1028: done (5 left) 2024-09-20T22:46:16.397 INFO:tasks.rados.rados.0.smithi084.stdout:1029: done (4 left) 2024-09-20T22:46:16.397 INFO:tasks.rados.rados.0.smithi084.stdout:1030: done (3 left) 2024-09-20T22:46:16.397 INFO:tasks.rados.rados.0.smithi084.stdout:1031: done (2 left) 2024-09-20T22:46:16.398 INFO:tasks.rados.rados.0.smithi084.stdout:1034: read oid 47 snap -1 2024-09-20T22:46:16.398 INFO:tasks.rados.rados.0.smithi084.stdout:1034: expect (ObjNum 208 snap 42 seq_num 208) 2024-09-20T22:46:16.398 INFO:tasks.rados.rados.0.smithi084.stdout:1035: rollback oid 6 current snap is 100 2024-09-20T22:46:16.398 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 6 to 99 2024-09-20T22:46:16.398 INFO:tasks.rados.rados.0.smithi084.stdout:1036: read oid 11 snap -1 2024-09-20T22:46:16.398 INFO:tasks.rados.rados.0.smithi084.stdout:1036: expect deleted 2024-09-20T22:46:16.398 INFO:tasks.rados.rados.0.smithi084.stdout:1037: read oid 13 snap 99 2024-09-20T22:46:16.398 INFO:tasks.rados.rados.0.smithi084.stdout:1037: expect (ObjNum 340 snap 88 seq_num 340) 2024-09-20T22:46:16.398 INFO:tasks.rados.rados.0.smithi084.stdout:1038: copy_from oid 3 from oid 46 current snap is 100 2024-09-20T22:46:16.398 INFO:tasks.rados.rados.0.smithi084.stdout:1039: rollback oid 38 current snap is 100 2024-09-20T22:46:16.398 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 38 to 99 2024-09-20T22:46:16.398 INFO:tasks.rados.rados.0.smithi084.stdout:1040: snap_remove snap 97 2024-09-20T22:46:16.492 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:46:16.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:16 smithi154 ceph-mon[33264]: osdmap e296: 8 total, 8 up, 6 in 2024-09-20T22:46:16.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:16 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-09-20T22:46:16.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:16 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-09-20T22:46:16.515 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:16.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:16.516 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:46:16.616 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:46:16.639 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:16.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:16.640 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:46:16.652 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:46:16.652 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:46:16.671 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:46:16.672 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:46:16.675 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:46:16.688 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:46:16.688 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:46:16.710 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-20T22:46:16.711 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T22:46:16.727 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:46:16.727 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:46:16.741 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:46:16.744 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:46:16.744 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:46:16.760 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:46:16.761 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:46:16.762 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:16.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:16.763 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:46:16.777 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:46:16.778 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:46:16.863 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:46:16.887 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:16.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:16.888 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:46:16.950 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd unset noscrub 2024-09-20T22:46:16.988 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:46:17.038 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:17.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:17.039 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:46:17.140 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:46:17.165 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:17.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:17.166 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:46:17.184 INFO:tasks.rados.rados.0.smithi084.stdout:1040: done (8 left) 2024-09-20T22:46:17.184 INFO:tasks.rados.rados.0.smithi084.stdout:1041: rollback oid 34 current snap is 100 2024-09-20T22:46:17.184 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 34 to 99 2024-09-20T22:46:17.184 INFO:tasks.rados.rados.0.smithi084.stdout:1035: finishing rollback tid 1 to smithi08477203-6 2024-09-20T22:46:17.184 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 6 v 0 (ObjNum 129 snap 0 seq_num 2090892904) dirty dne 2024-09-20T22:46:17.184 INFO:tasks.rados.rados.0.smithi084.stdout:1038: finishing copy_from to smithi08477203-3 2024-09-20T22:46:17.185 INFO:tasks.rados.rados.0.smithi084.stdout:1038: got expected ENOENT (src dne) 2024-09-20T22:46:17.185 INFO:tasks.rados.rados.0.smithi084.stdout:1038: finishing copy_from racing read to smithi08477203-3 2024-09-20T22:46:17.185 INFO:tasks.rados.rados.0.smithi084.stdout:1032: finishing write tid 2 to smithi08477203-10 2024-09-20T22:46:17.185 INFO:tasks.rados.rados.0.smithi084.stdout:1032: finishing write tid 3 to smithi08477203-10 2024-09-20T22:46:17.185 INFO:tasks.rados.rados.0.smithi084.stdout:1032: finishing write tid 4 to smithi08477203-10 2024-09-20T22:46:17.185 INFO:tasks.rados.rados.0.smithi084.stdout:1032: oid 10 updating version 0 to 282 2024-09-20T22:46:17.185 INFO:tasks.rados.rados.0.smithi084.stdout:1032: oid 10 updating version 282 to 283 2024-09-20T22:46:17.185 INFO:tasks.rados.rados.0.smithi084.stdout:1032: oid 10 updating version 283 to 284 2024-09-20T22:46:17.185 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 10 v 284 (ObjNum 383 snap 100 seq_num 383) dirty exists 2024-09-20T22:46:17.185 INFO:tasks.rados.rados.0.smithi084.stdout:1032: left oid 10 (ObjNum 383 snap 100 seq_num 383) 2024-09-20T22:46:17.185 INFO:tasks.rados.rados.0.smithi084.stdout:1033: finishing rollback tid 1 to smithi08477203-43 2024-09-20T22:46:17.185 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 43 v 0 (ObjNum 544235886 snap 668211 seq_num 304) dirty dne 2024-09-20T22:46:17.185 INFO:tasks.rados.rados.0.smithi084.stdout:1034: expect (ObjNum 208 snap 42 seq_num 208) 2024-09-20T22:46:17.267 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:46:17.291 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:17.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:17.292 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:46:17.364 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:17 smithi084 ceph-mon[40024]: pgmap v346: 169 pgs: 3 active+clean+snaptrim, 1 clean+premerge+peered, 1 active+clean+snaptrim_wait, 164 active+clean; 113 MiB data, 521 MiB used, 536 GiB / 536 GiB avail; 2.0 KiB/s rd, 631 KiB/s wr, 5 op/s 2024-09-20T22:46:17.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:17 smithi084 ceph-mon[40024]: purged_snaps scrub starts 2024-09-20T22:46:17.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:17 smithi084 ceph-mon[40024]: purged_snaps scrub ok 2024-09-20T22:46:17.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:17 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-09-20T22:46:17.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:17 smithi084 ceph-mon[40024]: osdmap e297: 8 total, 8 up, 6 in 2024-09-20T22:46:17.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:17 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/2047488264' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:46:17.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:17 smithi084 ceph-mon[40024]: purged_snaps scrub starts 2024-09-20T22:46:17.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:17 smithi084 ceph-mon[40024]: purged_snaps scrub ok 2024-09-20T22:46:17.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:17 smithi084 ceph-mon[29774]: pgmap v346: 169 pgs: 3 active+clean+snaptrim, 1 clean+premerge+peered, 1 active+clean+snaptrim_wait, 164 active+clean; 113 MiB data, 521 MiB used, 536 GiB / 536 GiB avail; 2.0 KiB/s rd, 631 KiB/s wr, 5 op/s 2024-09-20T22:46:17.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:17 smithi084 ceph-mon[29774]: purged_snaps scrub starts 2024-09-20T22:46:17.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:17 smithi084 ceph-mon[29774]: purged_snaps scrub ok 2024-09-20T22:46:17.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:17 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-09-20T22:46:17.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:17 smithi084 ceph-mon[29774]: osdmap e297: 8 total, 8 up, 6 in 2024-09-20T22:46:17.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:17 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/2047488264' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:46:17.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:17 smithi084 ceph-mon[29774]: purged_snaps scrub starts 2024-09-20T22:46:17.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:17 smithi084 ceph-mon[29774]: purged_snaps scrub ok 2024-09-20T22:46:17.392 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:46:17.411 INFO:tasks.rados.rados.0.smithi084.stdout:1039: finishing rollback tid 0 to smithi08477203-38 2024-09-20T22:46:17.411 INFO:tasks.rados.rados.0.smithi084.stdout:1039: finishing rollback tid 1 to smithi08477203-38 2024-09-20T22:46:17.411 INFO:tasks.rados.rados.0.smithi084.stdout:1037: expect (ObjNum 340 snap 88 seq_num 340) 2024-09-20T22:46:17.415 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:17.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:17.416 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:46:17.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:17 smithi154 ceph-mon[33264]: pgmap v346: 169 pgs: 3 active+clean+snaptrim, 1 clean+premerge+peered, 1 active+clean+snaptrim_wait, 164 active+clean; 113 MiB data, 521 MiB used, 536 GiB / 536 GiB avail; 2.0 KiB/s rd, 631 KiB/s wr, 5 op/s 2024-09-20T22:46:17.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:17 smithi154 ceph-mon[33264]: purged_snaps scrub starts 2024-09-20T22:46:17.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:17 smithi154 ceph-mon[33264]: purged_snaps scrub ok 2024-09-20T22:46:17.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:17 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-09-20T22:46:17.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:17 smithi154 ceph-mon[33264]: osdmap e297: 8 total, 8 up, 6 in 2024-09-20T22:46:17.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:17 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/2047488264' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:46:17.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:17 smithi154 ceph-mon[33264]: purged_snaps scrub starts 2024-09-20T22:46:17.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:17 smithi154 ceph-mon[33264]: purged_snaps scrub ok 2024-09-20T22:46:17.517 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:46:17.530 INFO:tasks.rados.rados.0.smithi084.stdout:1039: finishing rollback tid 2 to smithi08477203-38 2024-09-20T22:46:17.530 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 38 v 382 (ObjNum 258 snap 60 seq_num 258) dirty exists 2024-09-20T22:46:17.530 INFO:tasks.rados.rados.0.smithi084.stdout:1041: finishing rollback tid 1 to smithi08477203-34 2024-09-20T22:46:17.530 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 34 v 0 (ObjNum 129 snap 0 seq_num 712520430) dirty dne 2024-09-20T22:46:17.530 INFO:tasks.rados.rados.0.smithi084.stdout:1032: done (8 left) 2024-09-20T22:46:17.530 INFO:tasks.rados.rados.0.smithi084.stdout:1033: done (7 left) 2024-09-20T22:46:17.530 INFO:tasks.rados.rados.0.smithi084.stdout:1034: done (6 left) 2024-09-20T22:46:17.530 INFO:tasks.rados.rados.0.smithi084.stdout:1035: done (5 left) 2024-09-20T22:46:17.530 INFO:tasks.rados.rados.0.smithi084.stdout:1036: done (4 left) 2024-09-20T22:46:17.530 INFO:tasks.rados.rados.0.smithi084.stdout:1037: done (3 left) 2024-09-20T22:46:17.530 INFO:tasks.rados.rados.0.smithi084.stdout:1038: done (2 left) 2024-09-20T22:46:17.530 INFO:tasks.rados.rados.0.smithi084.stdout:1039: done (1 left) 2024-09-20T22:46:17.531 INFO:tasks.rados.rados.0.smithi084.stdout:1041: done (0 left) 2024-09-20T22:46:17.531 INFO:tasks.rados.rados.0.smithi084.stdout:1042: write (excl) oid 3 current snap is 100 2024-09-20T22:46:17.531 INFO:tasks.rados.rados.0.smithi084.stdout:1042: seq_num 384 ranges {453888=713266,1737556=442800} 2024-09-20T22:46:17.536 INFO:tasks.rados.rados.0.smithi084.stdout:1042: writing smithi08477203-3 from 453888 to 1167154 tid 1 2024-09-20T22:46:17.541 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:17.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:17.542 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:46:17.542 INFO:tasks.rados.rados.0.smithi084.stdout:1042: writing smithi08477203-3 from 1737556 to 2180356 tid 2 2024-09-20T22:46:17.542 INFO:tasks.rados.rados.0.smithi084.stdout:1043: snap_remove snap 99 2024-09-20T22:46:17.642 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:46:17.663 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:17.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:17.664 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:46:17.765 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:46:17.786 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:17.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:17.787 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:46:17.824 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-20T22:46:17.824 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:46:17.888 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:46:17.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:17.910 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:17.910 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:46:18.011 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:46:18.031 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:18.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:18.032 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:46:18.133 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:46:18.159 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:18.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:18.160 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:46:18.168 INFO:tasks.rados.rados.0.smithi084.stdout:1043: done (1 left) 2024-09-20T22:46:18.168 INFO:tasks.rados.rados.0.smithi084.stdout:1044: read oid 41 snap -1 2024-09-20T22:46:18.168 INFO:tasks.rados.rados.0.smithi084.stdout:1044: expect (ObjNum 374 snap 99 seq_num 374) 2024-09-20T22:46:18.168 INFO:tasks.rados.rados.0.smithi084.stdout:1045: delete oid 42 current snap is 100 2024-09-20T22:46:18.168 INFO:tasks.rados.rados.0.smithi084.stdout:1042: finishing write tid 1 to smithi08477203-3 2024-09-20T22:46:18.168 INFO:tasks.rados.rados.0.smithi084.stdout:1042: finishing write tid 2 to smithi08477203-3 2024-09-20T22:46:18.168 INFO:tasks.rados.rados.0.smithi084.stdout:1042: finishing write tid 3 to smithi08477203-3 2024-09-20T22:46:18.168 INFO:tasks.rados.rados.0.smithi084.stdout:1042: finishing write tid 4 to smithi08477203-3 2024-09-20T22:46:18.169 INFO:tasks.rados.rados.0.smithi084.stdout:1042: oid 3 updating version 0 to 180 2024-09-20T22:46:18.169 INFO:tasks.rados.rados.0.smithi084.stdout:1042: oid 3 updating version 180 to 182 2024-09-20T22:46:18.169 INFO:tasks.rados.rados.0.smithi084.stdout:1042: oid 3 version 182 is already newer than 181 2024-09-20T22:46:18.169 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 3 v 182 (ObjNum 384 snap 100 seq_num 384) dirty exists 2024-09-20T22:46:18.169 INFO:tasks.rados.rados.0.smithi084.stdout:1042: left oid 3 (ObjNum 384 snap 100 seq_num 384) 2024-09-20T22:46:18.175 INFO:tasks.rados.rados.0.smithi084.stdout:1042: done (2 left) 2024-09-20T22:46:18.175 INFO:tasks.rados.rados.0.smithi084.stdout:1045: done (1 left) 2024-09-20T22:46:18.175 INFO:tasks.rados.rados.0.smithi084.stdout:1046: write oid 30 current snap is 100 2024-09-20T22:46:18.175 INFO:tasks.rados.rados.0.smithi084.stdout:1046: seq_num 386 ranges {487502=463764,1738167=446469} 2024-09-20T22:46:18.181 INFO:tasks.rados.rados.0.smithi084.stdout:1046: writing smithi08477203-30 from 487502 to 951266 tid 1 2024-09-20T22:46:18.188 INFO:tasks.rados.rados.0.smithi084.stdout:1046: writing smithi08477203-30 from 1738167 to 2184636 tid 2 2024-09-20T22:46:18.188 INFO:tasks.rados.rados.0.smithi084.stdout:1047: read oid 2 snap -1 2024-09-20T22:46:18.188 INFO:tasks.rados.rados.0.smithi084.stdout:1047: expect (ObjNum 300 snap 75 seq_num 300) 2024-09-20T22:46:18.188 INFO:tasks.rados.rados.0.smithi084.stdout:1048: write oid 21 current snap is 100 2024-09-20T22:46:18.189 INFO:tasks.rados.rados.0.smithi084.stdout:1048: seq_num 387 ranges {504309=539013,1796649=590127} 2024-09-20T22:46:18.194 INFO:tasks.rados.rados.0.smithi084.stdout:1048: writing smithi08477203-21 from 504309 to 1043322 tid 1 2024-09-20T22:46:18.202 INFO:tasks.rados.rados.0.smithi084.stdout:1048: writing smithi08477203-21 from 1796649 to 2386776 tid 2 2024-09-20T22:46:18.202 INFO:tasks.rados.rados.0.smithi084.stdout:1049: read oid 40 snap -1 2024-09-20T22:46:18.202 INFO:tasks.rados.rados.0.smithi084.stdout:1049: expect (ObjNum 316 snap 80 seq_num 316) 2024-09-20T22:46:18.202 INFO:tasks.rados.rados.0.smithi084.stdout:1050: snap_create 2024-09-20T22:46:18.202 INFO:tasks.rados.rados.0.smithi084.stdout:1044: expect (ObjNum 374 snap 99 seq_num 374) 2024-09-20T22:46:18.261 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:46:18.282 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:18.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:18.283 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:46:18.384 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:46:18.386 INFO:tasks.rados.rados.0.smithi084.stdout:1046: finishing write tid 1 to smithi08477203-30 2024-09-20T22:46:18.386 INFO:tasks.rados.rados.0.smithi084.stdout:1046: finishing write tid 2 to smithi08477203-30 2024-09-20T22:46:18.386 INFO:tasks.rados.rados.0.smithi084.stdout:1046: finishing write tid 3 to smithi08477203-30 2024-09-20T22:46:18.386 INFO:tasks.rados.rados.0.smithi084.stdout:1046: finishing write tid 4 to smithi08477203-30 2024-09-20T22:46:18.386 INFO:tasks.rados.rados.0.smithi084.stdout:1046: oid 30 updating version 0 to 177 2024-09-20T22:46:18.386 INFO:tasks.rados.rados.0.smithi084.stdout:1046: oid 30 version 177 is already newer than 176 2024-09-20T22:46:18.386 INFO:tasks.rados.rados.0.smithi084.stdout:1046: oid 30 version 177 is already newer than 175 2024-09-20T22:46:18.386 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 30 v 177 (ObjNum 386 snap 100 seq_num 386) dirty exists 2024-09-20T22:46:18.386 INFO:tasks.rados.rados.0.smithi084.stdout:1046: left oid 30 (ObjNum 386 snap 100 seq_num 386) 2024-09-20T22:46:18.386 INFO:tasks.rados.rados.0.smithi084.stdout:1047: expect (ObjNum 300 snap 75 seq_num 300) 2024-09-20T22:46:18.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:18.408 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:18.409 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:46:18.483 INFO:tasks.rados.rados.0.smithi084.stdout:1048: finishing write tid 1 to smithi08477203-21 2024-09-20T22:46:18.484 INFO:tasks.rados.rados.0.smithi084.stdout:1048: finishing write tid 2 to smithi08477203-21 2024-09-20T22:46:18.484 INFO:tasks.rados.rados.0.smithi084.stdout:1048: finishing write tid 3 to smithi08477203-21 2024-09-20T22:46:18.484 INFO:tasks.rados.rados.0.smithi084.stdout:1048: finishing write tid 4 to smithi08477203-21 2024-09-20T22:46:18.484 INFO:tasks.rados.rados.0.smithi084.stdout:1048: oid 21 updating version 0 to 281 2024-09-20T22:46:18.484 INFO:tasks.rados.rados.0.smithi084.stdout:1048: oid 21 updating version 281 to 282 2024-09-20T22:46:18.484 INFO:tasks.rados.rados.0.smithi084.stdout:1048: oid 21 updating version 282 to 283 2024-09-20T22:46:18.484 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 21 v 283 (ObjNum 387 snap 100 seq_num 387) dirty exists 2024-09-20T22:46:18.484 INFO:tasks.rados.rados.0.smithi084.stdout:1048: left oid 21 (ObjNum 387 snap 100 seq_num 387) 2024-09-20T22:46:18.484 INFO:tasks.rados.rados.0.smithi084.stdout:1049: expect (ObjNum 316 snap 80 seq_num 316) 2024-09-20T22:46:18.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:18 smithi154 ceph-mon[33264]: osdmap e298: 8 total, 8 up, 6 in 2024-09-20T22:46:18.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:18 smithi154 ceph-mon[33264]: osdmap e299: 8 total, 8 up, 6 in 2024-09-20T22:46:18.509 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:46:18.532 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:18.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:18.533 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:46:18.570 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:46:18 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: ::ffff:172.21.15.154 - - [20/Sep/2024:22:46:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-20T22:46:18.571 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:18 smithi084 ceph-mon[40024]: osdmap e298: 8 total, 8 up, 6 in 2024-09-20T22:46:18.571 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:18 smithi084 ceph-mon[40024]: osdmap e299: 8 total, 8 up, 6 in 2024-09-20T22:46:18.571 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:18 smithi084 ceph-mon[29774]: osdmap e298: 8 total, 8 up, 6 in 2024-09-20T22:46:18.571 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:18 smithi084 ceph-mon[29774]: osdmap e299: 8 total, 8 up, 6 in 2024-09-20T22:46:18.596 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.3 dump_blocked_ops 2024-09-20T22:46:18.634 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:46:18.635 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:46:18.656 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:18.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:18.657 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:46:18.758 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:46:18.782 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:18.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:18.783 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:46:18.884 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:46:18.907 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:18.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:18.908 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:46:19.009 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:46:19.035 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:19.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:19.036 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:46:19.137 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:46:19.144 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:46:19.161 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:19.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:19.162 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:46:19.173 INFO:tasks.rados.rados.0.smithi084.stdout:1044: done (5 left) 2024-09-20T22:46:19.173 INFO:tasks.rados.rados.0.smithi084.stdout:1046: done (4 left) 2024-09-20T22:46:19.173 INFO:tasks.rados.rados.0.smithi084.stdout:1047: done (3 left) 2024-09-20T22:46:19.173 INFO:tasks.rados.rados.0.smithi084.stdout:1048: done (2 left) 2024-09-20T22:46:19.174 INFO:tasks.rados.rados.0.smithi084.stdout:1049: done (1 left) 2024-09-20T22:46:19.174 INFO:tasks.rados.rados.0.smithi084.stdout:1050: done (0 left) 2024-09-20T22:46:19.174 INFO:tasks.rados.rados.0.smithi084.stdout:1051: rollback oid 35 current snap is 101 2024-09-20T22:46:19.174 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 35 to 100 2024-09-20T22:46:19.174 INFO:tasks.rados.rados.0.smithi084.stdout:1052: delete oid 16 current snap is 101 2024-09-20T22:46:19.179 INFO:tasks.rados.rados.0.smithi084.stdout:1051: finishing rollback tid 0 to smithi08477203-35 2024-09-20T22:46:19.180 INFO:tasks.rados.rados.0.smithi084.stdout:1051: finishing rollback tid 1 to smithi08477203-35 2024-09-20T22:46:19.180 INFO:tasks.rados.rados.0.smithi084.stdout:1051: finishing rollback tid 2 to smithi08477203-35 2024-09-20T22:46:19.180 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 35 v 168 (ObjNum 373 snap 97 seq_num 373) dirty exists 2024-09-20T22:46:19.182 INFO:tasks.rados.rados.0.smithi084.stdout:1051: done (1 left) 2024-09-20T22:46:19.182 INFO:tasks.rados.rados.0.smithi084.stdout:1052: done (0 left) 2024-09-20T22:46:19.182 INFO:tasks.rados.rados.0.smithi084.stdout:1053: write oid 18 current snap is 101 2024-09-20T22:46:19.182 INFO:tasks.rados.rados.0.smithi084.stdout:1053: seq_num 389 ranges {537923=405864,1513613=593796} 2024-09-20T22:46:19.187 INFO:tasks.rados.rados.0.smithi084.stdout:1053: writing smithi08477203-18 from 537923 to 943787 tid 1 2024-09-20T22:46:19.194 INFO:tasks.rados.rados.0.smithi084.stdout:1053: writing smithi08477203-18 from 1513613 to 2107409 tid 2 2024-09-20T22:46:19.194 INFO:tasks.rados.rados.0.smithi084.stdout:1054: snap_remove snap 100 2024-09-20T22:46:19.263 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:46:19.286 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:19.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:19.287 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:46:19.388 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:46:19.410 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:19.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:19.411 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:46:19.494 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:19 smithi084 ceph-mon[40024]: pgmap v349: 169 pgs: 3 active+clean+snaptrim, 1 clean+premerge+peered, 1 active+clean+snaptrim_wait, 164 active+clean; 113 MiB data, 521 MiB used, 536 GiB / 536 GiB avail 2024-09-20T22:46:19.494 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:19 smithi084 ceph-mon[40024]: osdmap e300: 8 total, 8 up, 6 in 2024-09-20T22:46:19.494 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:19 smithi084 ceph-mon[29774]: pgmap v349: 169 pgs: 3 active+clean+snaptrim, 1 clean+premerge+peered, 1 active+clean+snaptrim_wait, 164 active+clean; 113 MiB data, 521 MiB used, 536 GiB / 536 GiB avail 2024-09-20T22:46:19.494 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:19 smithi084 ceph-mon[29774]: osdmap e300: 8 total, 8 up, 6 in 2024-09-20T22:46:19.504 INFO:journalctl@ceph.iscsi.iscsi.a.smithi154.stdout:Sep 20 22:46:19 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-iscsi-iscsi-a[59730]: debug there is no tcmu-runner data available 2024-09-20T22:46:19.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:19 smithi154 ceph-mon[33264]: pgmap v349: 169 pgs: 3 active+clean+snaptrim, 1 clean+premerge+peered, 1 active+clean+snaptrim_wait, 164 active+clean; 113 MiB data, 521 MiB used, 536 GiB / 536 GiB avail 2024-09-20T22:46:19.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:19 smithi154 ceph-mon[33264]: osdmap e300: 8 total, 8 up, 6 in 2024-09-20T22:46:19.512 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:46:19.535 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:19.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:19.536 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:46:19.545 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 5, 6, 7] out_osds: [4, 0] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-20T22:46:19.546 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-20T22:46:19.546 INFO:tasks.thrashosds.thrasher:inject_pause on osd.1 2024-09-20T22:46:19.546 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-20T22:46:19.546 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-20T22:46:19.546 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.1 config set filestore_inject_stall 3 2024-09-20T22:46:19.593 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-20T22:46:19.593 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:46:19.608 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-20T22:46:19.636 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:46:19.660 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:19.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:19.661 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:46:19.761 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:46:19.785 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:19.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:19.786 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:46:19.887 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:46:19.908 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:19.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:19.909 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:46:20.010 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:46:20.031 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:20.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:20.032 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:46:20.133 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:46:20.154 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:20.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:20.155 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:46:20.199 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is unset 2024-09-20T22:46:20.208 INFO:tasks.rados.rados.0.smithi084.stdout:1054: done (1 left) 2024-09-20T22:46:20.208 INFO:tasks.rados.rados.0.smithi084.stdout:1055: rollback oid 40 current snap is 101 2024-09-20T22:46:20.208 INFO:tasks.rados.rados.0.smithi084.stdout:1055: done (1 left) 2024-09-20T22:46:20.208 INFO:tasks.rados.rados.0.smithi084.stdout:1056: write (excl) oid 26 current snap is 101 2024-09-20T22:46:20.208 INFO:tasks.rados.rados.0.smithi084.stdout:1053: finishing write tid 1 to smithi08477203-18 2024-09-20T22:46:20.208 INFO:tasks.rados.rados.0.smithi084.stdout:1056: seq_num 390 ranges {554730=481113,1572095=737454} 2024-09-20T22:46:20.214 INFO:tasks.rados.rados.0.smithi084.stdout:1056: writing smithi08477203-26 from 554730 to 1035843 tid 1 2024-09-20T22:46:20.222 INFO:tasks.rados.rados.0.smithi084.stdout:1056: writing smithi08477203-26 from 1572095 to 2309549 tid 2 2024-09-20T22:46:20.222 INFO:tasks.rados.rados.0.smithi084.stdout:1053: finishing write tid 2 to smithi08477203-18 2024-09-20T22:46:20.222 INFO:tasks.rados.rados.0.smithi084.stdout:1053: finishing write tid 3 to smithi08477203-18 2024-09-20T22:46:20.222 INFO:tasks.rados.rados.0.smithi084.stdout:1053: finishing write tid 4 to smithi08477203-18 2024-09-20T22:46:20.223 INFO:tasks.rados.rados.0.smithi084.stdout:1053: oid 18 updating version 0 to 290 2024-09-20T22:46:20.223 INFO:tasks.rados.rados.0.smithi084.stdout:1053: oid 18 version 290 is already newer than 289 2024-09-20T22:46:20.223 INFO:tasks.rados.rados.0.smithi084.stdout:1053: oid 18 version 290 is already newer than 288 2024-09-20T22:46:20.223 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 18 v 290 (ObjNum 389 snap 101 seq_num 389) dirty exists 2024-09-20T22:46:20.223 INFO:tasks.rados.rados.0.smithi084.stdout:1053: left oid 18 (ObjNum 389 snap 101 seq_num 389) 2024-09-20T22:46:20.223 INFO:tasks.rados.rados.0.smithi084.stdout:1053: done (1 left) 2024-09-20T22:46:20.223 INFO:tasks.rados.rados.0.smithi084.stdout:1057: read oid 30 snap -1 2024-09-20T22:46:20.223 INFO:tasks.rados.rados.0.smithi084.stdout:1057: expect (ObjNum 386 snap 100 seq_num 386) 2024-09-20T22:46:20.223 INFO:tasks.rados.rados.0.smithi084.stdout:1058: write oid 10 current snap is 101 2024-09-20T22:46:20.223 INFO:tasks.rados.rados.0.smithi084.stdout:1058: seq_num 391 ranges {571537=556362,1630577=597465} 2024-09-20T22:46:20.229 INFO:tasks.rados.rados.0.smithi084.stdout:1058: writing smithi08477203-10 from 571537 to 1127899 tid 1 2024-09-20T22:46:20.236 INFO:tasks.rados.rados.0.smithi084.stdout:1058: writing smithi08477203-10 from 1630577 to 2228042 tid 2 2024-09-20T22:46:20.236 INFO:tasks.rados.rados.0.smithi084.stdout:1059: copy_from oid 5 from oid 32 current snap is 101 2024-09-20T22:46:20.236 INFO:tasks.rados.rados.0.smithi084.stdout:1056: finishing write tid 1 to smithi08477203-26 2024-09-20T22:46:20.236 INFO:tasks.rados.rados.0.smithi084.stdout:1056: finishing write tid 2 to smithi08477203-26 2024-09-20T22:46:20.236 INFO:tasks.rados.rados.0.smithi084.stdout:1056: finishing write tid 3 to smithi08477203-26 2024-09-20T22:46:20.236 INFO:tasks.rados.rados.0.smithi084.stdout:1056: finishing write tid 4 to smithi08477203-26 2024-09-20T22:46:20.236 INFO:tasks.rados.rados.0.smithi084.stdout:1056: oid 26 updating version 0 to 251 2024-09-20T22:46:20.236 INFO:tasks.rados.rados.0.smithi084.stdout:1056: oid 26 updating version 251 to 252 2024-09-20T22:46:20.236 INFO:tasks.rados.rados.0.smithi084.stdout:1056: oid 26 version 252 is already newer than 250 2024-09-20T22:46:20.236 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 26 v 252 (ObjNum 390 snap 101 seq_num 390) dirty exists 2024-09-20T22:46:20.237 INFO:tasks.rados.rados.0.smithi084.stdout:1056: left oid 26 (ObjNum 390 snap 101 seq_num 390) 2024-09-20T22:46:20.237 INFO:tasks.rados.rados.0.smithi084.stdout:1058: finishing write tid 1 to smithi08477203-10 2024-09-20T22:46:20.237 INFO:tasks.rados.rados.0.smithi084.stdout:1056: done (3 left) 2024-09-20T22:46:20.237 INFO:tasks.rados.rados.0.smithi084.stdout:1060: read oid 37 snap -1 2024-09-20T22:46:20.237 INFO:tasks.rados.rados.0.smithi084.stdout:1060: expect (ObjNum 329 snap 84 seq_num 329) 2024-09-20T22:46:20.237 INFO:tasks.rados.rados.0.smithi084.stdout:1061: rollback oid 21 current snap is 101 2024-09-20T22:46:20.237 INFO:tasks.rados.rados.0.smithi084.stdout:1061: done (4 left) 2024-09-20T22:46:20.237 INFO:tasks.rados.rados.0.smithi084.stdout:1062: read oid 36 snap -1 2024-09-20T22:46:20.237 INFO:tasks.rados.rados.0.smithi084.stdout:1062: expect (ObjNum 313 snap 77 seq_num 313) 2024-09-20T22:46:20.237 INFO:tasks.rados.rados.0.smithi084.stdout:1063: write oid 22 current snap is 101 2024-09-20T22:46:20.237 INFO:tasks.rados.rados.0.smithi084.stdout:1063: seq_num 392 ranges {588344=631611,1689059=741123} 2024-09-20T22:46:20.243 INFO:tasks.rados.rados.0.smithi084.stdout:1063: writing smithi08477203-22 from 588344 to 1219955 tid 1 2024-09-20T22:46:20.251 INFO:tasks.rados.rados.0.smithi084.stdout:1063: writing smithi08477203-22 from 1689059 to 2430182 tid 2 2024-09-20T22:46:20.251 INFO:tasks.rados.rados.0.smithi084.stdout:1064: rollback oid 43 current snap is 101 2024-09-20T22:46:20.251 INFO:tasks.rados.rados.0.smithi084.stdout:1064: done (6 left) 2024-09-20T22:46:20.251 INFO:tasks.rados.rados.0.smithi084.stdout:1065: write oid 18 current snap is 101 2024-09-20T22:46:20.251 INFO:tasks.rados.rados.0.smithi084.stdout:1065: seq_num 393 ranges {605151=706860,2031188=573963} 2024-09-20T22:46:20.256 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:46:20.259 INFO:tasks.rados.rados.0.smithi084.stdout:1065: writing smithi08477203-18 from 605151 to 1312011 tid 1 2024-09-20T22:46:20.265 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:20 smithi084 ceph-mon[29774]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:46:20.265 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:20 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/2021203486' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:46:20.265 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:20 smithi084 ceph-mon[40024]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:46:20.265 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:20 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/2021203486' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:46:20.266 INFO:tasks.rados.rados.0.smithi084.stdout:1065: writing smithi08477203-18 from 2031188 to 2605151 tid 2 2024-09-20T22:46:20.267 INFO:tasks.rados.rados.0.smithi084.stdout:1066: read oid 26 snap -1 2024-09-20T22:46:20.267 INFO:tasks.rados.rados.0.smithi084.stdout:1066: expect (ObjNum 390 snap 101 seq_num 390) 2024-09-20T22:46:20.267 INFO:tasks.rados.rados.0.smithi084.stdout:1067: read oid 17 snap -1 2024-09-20T22:46:20.267 INFO:tasks.rados.rados.0.smithi084.stdout:1067: expect (ObjNum 315 snap 78 seq_num 315) 2024-09-20T22:46:20.267 INFO:tasks.rados.rados.0.smithi084.stdout:1068: delete oid 28 current snap is 101 2024-09-20T22:46:20.267 INFO:tasks.rados.rados.0.smithi084.stdout:1057: expect (ObjNum 386 snap 100 seq_num 386) 2024-09-20T22:46:20.282 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:20.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:20.283 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:46:20.296 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.3 dump_historic_ops 2024-09-20T22:46:20.354 INFO:tasks.rados.rados.0.smithi084.stdout:1058: finishing write tid 2 to smithi08477203-10 2024-09-20T22:46:20.354 INFO:tasks.rados.rados.0.smithi084.stdout:1058: finishing write tid 3 to smithi08477203-10 2024-09-20T22:46:20.354 INFO:tasks.rados.rados.0.smithi084.stdout:1057: done (9 left) 2024-09-20T22:46:20.355 INFO:tasks.rados.rados.0.smithi084.stdout:1068: done (8 left) 2024-09-20T22:46:20.355 INFO:tasks.rados.rados.0.smithi084.stdout:1069: copy_from oid 34 from oid 44 current snap is 101 2024-09-20T22:46:20.355 INFO:tasks.rados.rados.0.smithi084.stdout:1058: finishing write tid 4 to smithi08477203-10 2024-09-20T22:46:20.355 INFO:tasks.rados.rados.0.smithi084.stdout:1058: oid 10 updating version 0 to 295 2024-09-20T22:46:20.355 INFO:tasks.rados.rados.0.smithi084.stdout:1058: oid 10 version 295 is already newer than 294 2024-09-20T22:46:20.355 INFO:tasks.rados.rados.0.smithi084.stdout:1058: oid 10 version 295 is already newer than 293 2024-09-20T22:46:20.355 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 10 v 295 (ObjNum 391 snap 101 seq_num 391) dirty exists 2024-09-20T22:46:20.355 INFO:tasks.rados.rados.0.smithi084.stdout:1058: left oid 10 (ObjNum 391 snap 101 seq_num 391) 2024-09-20T22:46:20.355 INFO:tasks.rados.rados.0.smithi084.stdout:1062: expect (ObjNum 313 snap 77 seq_num 313) 2024-09-20T22:46:20.383 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:46:20.404 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:20.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:20.405 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:46:20.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:20 smithi154 ceph-mon[33264]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:46:20.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:20 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/2021203486' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:46:20.506 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:46:20.510 INFO:tasks.rados.rados.0.smithi084.stdout:1059: finishing copy_from racing read to smithi08477203-5 2024-09-20T22:46:20.510 INFO:tasks.rados.rados.0.smithi084.stdout:1060: expect (ObjNum 329 snap 84 seq_num 329) 2024-09-20T22:46:20.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:20.528 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:20.528 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:46:20.629 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:46:20.652 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:20.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:20.653 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:46:20.666 INFO:tasks.rados.rados.0.smithi084.stdout:1063: finishing write tid 1 to smithi08477203-22 2024-09-20T22:46:20.666 INFO:tasks.rados.rados.0.smithi084.stdout:1058: done (8 left) 2024-09-20T22:46:20.667 INFO:tasks.rados.rados.0.smithi084.stdout:1060: done (7 left) 2024-09-20T22:46:20.667 INFO:tasks.rados.rados.0.smithi084.stdout:1062: done (6 left) 2024-09-20T22:46:20.667 INFO:tasks.rados.rados.0.smithi084.stdout:1070: read oid 16 snap -1 2024-09-20T22:46:20.667 INFO:tasks.rados.rados.0.smithi084.stdout:1070: expect deleted 2024-09-20T22:46:20.667 INFO:tasks.rados.rados.0.smithi084.stdout:1071: write (excl) oid 33 current snap is 101 2024-09-20T22:46:20.667 INFO:tasks.rados.rados.0.smithi084.stdout:1059: finishing copy_from to smithi08477203-5 2024-09-20T22:46:20.667 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 5 v 379 (ObjNum 375 snap 99 seq_num 375) dirty exists 2024-09-20T22:46:20.667 INFO:tasks.rados.rados.0.smithi084.stdout:1063: finishing write tid 2 to smithi08477203-22 2024-09-20T22:46:20.667 INFO:tasks.rados.rados.0.smithi084.stdout:1063: finishing write tid 3 to smithi08477203-22 2024-09-20T22:46:20.667 INFO:tasks.rados.rados.0.smithi084.stdout:1063: finishing write tid 4 to smithi08477203-22 2024-09-20T22:46:20.667 INFO:tasks.rados.rados.0.smithi084.stdout:1063: oid 22 updating version 0 to 178 2024-09-20T22:46:20.667 INFO:tasks.rados.rados.0.smithi084.stdout:1063: oid 22 version 178 is already newer than 177 2024-09-20T22:46:20.667 INFO:tasks.rados.rados.0.smithi084.stdout:1063: oid 22 version 178 is already newer than 176 2024-09-20T22:46:20.667 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 22 v 178 (ObjNum 392 snap 101 seq_num 392) dirty exists 2024-09-20T22:46:20.667 INFO:tasks.rados.rados.0.smithi084.stdout:1063: left oid 22 (ObjNum 392 snap 101 seq_num 392) 2024-09-20T22:46:20.668 INFO:tasks.rados.rados.0.smithi084.stdout:1065: finishing write tid 1 to smithi08477203-18 2024-09-20T22:46:20.668 INFO:tasks.rados.rados.0.smithi084.stdout:1065: finishing write tid 2 to smithi08477203-18 2024-09-20T22:46:20.668 INFO:tasks.rados.rados.0.smithi084.stdout:1065: finishing write tid 3 to smithi08477203-18 2024-09-20T22:46:20.668 INFO:tasks.rados.rados.0.smithi084.stdout:1065: finishing write tid 4 to smithi08477203-18 2024-09-20T22:46:20.668 INFO:tasks.rados.rados.0.smithi084.stdout:1065: oid 18 updating version 0 to 296 2024-09-20T22:46:20.668 INFO:tasks.rados.rados.0.smithi084.stdout:1065: oid 18 updating version 296 to 297 2024-09-20T22:46:20.668 INFO:tasks.rados.rados.0.smithi084.stdout:1065: oid 18 updating version 297 to 298 2024-09-20T22:46:20.668 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 18 v 298 (ObjNum 393 snap 101 seq_num 393) dirty exists 2024-09-20T22:46:20.668 INFO:tasks.rados.rados.0.smithi084.stdout:1065: left oid 18 (ObjNum 393 snap 101 seq_num 393) 2024-09-20T22:46:20.668 INFO:tasks.rados.rados.0.smithi084.stdout:1067: expect (ObjNum 315 snap 78 seq_num 315) 2024-09-20T22:46:20.753 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:46:20.776 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:20.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:20.777 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:46:20.878 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:46:20.900 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:20.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:20.901 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:46:20.913 INFO:tasks.rados.rados.0.smithi084.stdout:1066: expect (ObjNum 390 snap 101 seq_num 390) 2024-09-20T22:46:20.914 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:46:20.918 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:46:20.999 INFO:tasks.rados.rados.0.smithi084.stdout:1069: finishing copy_from racing read to smithi08477203-34 2024-09-20T22:46:20.999 INFO:tasks.rados.rados.0.smithi084.stdout:1069: finishing copy_from to smithi08477203-34 2024-09-20T22:46:20.999 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 34 v 501 (ObjNum 359 snap 92 seq_num 359) dirty exists 2024-09-20T22:46:20.999 INFO:tasks.rados.rados.0.smithi084.stdout:1071: seq_num 395 ranges {638765=457358,1748152=604803} 2024-09-20T22:46:21.002 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:46:21.005 INFO:tasks.rados.rados.0.smithi084.stdout:1071: writing smithi08477203-33 from 638765 to 1096123 tid 1 2024-09-20T22:46:21.013 INFO:tasks.rados.rados.0.smithi084.stdout:1071: writing smithi08477203-33 from 1748152 to 2352955 tid 2 2024-09-20T22:46:21.013 INFO:tasks.rados.rados.0.smithi084.stdout:1059: done (7 left) 2024-09-20T22:46:21.013 INFO:tasks.rados.rados.0.smithi084.stdout:1063: done (6 left) 2024-09-20T22:46:21.013 INFO:tasks.rados.rados.0.smithi084.stdout:1065: done (5 left) 2024-09-20T22:46:21.013 INFO:tasks.rados.rados.0.smithi084.stdout:1066: done (4 left) 2024-09-20T22:46:21.013 INFO:tasks.rados.rados.0.smithi084.stdout:1067: done (3 left) 2024-09-20T22:46:21.013 INFO:tasks.rados.rados.0.smithi084.stdout:1069: done (2 left) 2024-09-20T22:46:21.013 INFO:tasks.rados.rados.0.smithi084.stdout:1070: done (1 left) 2024-09-20T22:46:21.013 INFO:tasks.rados.rados.0.smithi084.stdout:1072: rollback oid 2 current snap is 101 2024-09-20T22:46:21.014 INFO:tasks.rados.rados.0.smithi084.stdout:1072: done (1 left) 2024-09-20T22:46:21.014 INFO:tasks.rados.rados.0.smithi084.stdout:1073: rollback oid 48 current snap is 101 2024-09-20T22:46:21.014 INFO:tasks.rados.rados.0.smithi084.stdout:1073: done (1 left) 2024-09-20T22:46:21.014 INFO:tasks.rados.rados.0.smithi084.stdout:1074: rollback oid 8 current snap is 101 2024-09-20T22:46:21.014 INFO:tasks.rados.rados.0.smithi084.stdout:1074: done (1 left) 2024-09-20T22:46:21.014 INFO:tasks.rados.rados.0.smithi084.stdout:1075: write (excl) oid 28 current snap is 101 2024-09-20T22:46:21.014 INFO:tasks.rados.rados.0.smithi084.stdout:1071: finishing write tid 1 to smithi08477203-33 2024-09-20T22:46:21.014 INFO:tasks.rados.rados.0.smithi084.stdout:1075: seq_num 396 ranges {655572=648960,1922987=732585} 2024-09-20T22:46:21.020 INFO:tasks.rados.rados.0.smithi084.stdout:1075: writing smithi08477203-28 from 655572 to 1304532 tid 1 2024-09-20T22:46:21.024 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:21.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:21.025 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:46:21.030 INFO:tasks.rados.rados.0.smithi084.stdout:1075: writing smithi08477203-28 from 1922987 to 2655572 tid 2 2024-09-20T22:46:21.031 INFO:tasks.rados.rados.0.smithi084.stdout:1076: rollback oid 17 current snap is 101 2024-09-20T22:46:21.031 INFO:tasks.rados.rados.0.smithi084.stdout:1076: done (2 left) 2024-09-20T22:46:21.031 INFO:tasks.rados.rados.0.smithi084.stdout:1077: write oid 38 current snap is 101 2024-09-20T22:46:21.031 INFO:tasks.rados.rados.0.smithi084.stdout:1071: finishing write tid 2 to smithi08477203-33 2024-09-20T22:46:21.031 INFO:tasks.rados.rados.0.smithi084.stdout:1071: finishing write tid 3 to smithi08477203-33 2024-09-20T22:46:21.031 INFO:tasks.rados.rados.0.smithi084.stdout:1071: finishing write tid 4 to smithi08477203-33 2024-09-20T22:46:21.031 INFO:tasks.rados.rados.0.smithi084.stdout:1071: oid 33 updating version 0 to 214 2024-09-20T22:46:21.031 INFO:tasks.rados.rados.0.smithi084.stdout:1071: oid 33 updating version 214 to 216 2024-09-20T22:46:21.031 INFO:tasks.rados.rados.0.smithi084.stdout:1071: oid 33 version 216 is already newer than 215 2024-09-20T22:46:21.031 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 33 v 216 (ObjNum 395 snap 101 seq_num 395) dirty exists 2024-09-20T22:46:21.031 INFO:tasks.rados.rados.0.smithi084.stdout:1071: left oid 33 (ObjNum 395 snap 101 seq_num 395) 2024-09-20T22:46:21.032 INFO:tasks.rados.rados.0.smithi084.stdout:1075: finishing write tid 1 to smithi08477203-28 2024-09-20T22:46:21.032 INFO:tasks.rados.rados.0.smithi084.stdout:1077: seq_num 397 ranges {672379=724209,1865116=608472} 2024-09-20T22:46:21.035 INFO:tasks.rados.rados.0.smithi084.stdout:1077: writing smithi08477203-38 from 672379 to 1396588 tid 1 2024-09-20T22:46:21.042 INFO:tasks.rados.rados.0.smithi084.stdout:1077: writing smithi08477203-38 from 1865116 to 2473588 tid 2 2024-09-20T22:46:21.042 INFO:tasks.rados.rados.0.smithi084.stdout:1071: done (2 left) 2024-09-20T22:46:21.042 INFO:tasks.rados.rados.0.smithi084.stdout:1078: write oid 23 current snap is 101 2024-09-20T22:46:21.042 INFO:tasks.rados.rados.0.smithi084.stdout:1075: finishing write tid 2 to smithi08477203-28 2024-09-20T22:46:21.042 INFO:tasks.rados.rados.0.smithi084.stdout:1075: finishing write tid 3 to smithi08477203-28 2024-09-20T22:46:21.042 INFO:tasks.rados.rados.0.smithi084.stdout:1075: finishing write tid 4 to smithi08477203-28 2024-09-20T22:46:21.042 INFO:tasks.rados.rados.0.smithi084.stdout:1075: oid 28 updating version 0 to 574 2024-09-20T22:46:21.042 INFO:tasks.rados.rados.0.smithi084.stdout:1075: oid 28 updating version 574 to 575 2024-09-20T22:46:21.042 INFO:tasks.rados.rados.0.smithi084.stdout:1075: oid 28 version 575 is already newer than 573 2024-09-20T22:46:21.043 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 28 v 575 (ObjNum 396 snap 101 seq_num 396) dirty exists 2024-09-20T22:46:21.043 INFO:tasks.rados.rados.0.smithi084.stdout:1075: left oid 28 (ObjNum 396 snap 101 seq_num 396) 2024-09-20T22:46:21.043 INFO:tasks.rados.rados.0.smithi084.stdout:1078: seq_num 398 ranges {689186=799458,1923598=752130} 2024-09-20T22:46:21.050 INFO:tasks.rados.rados.0.smithi084.stdout:1078: writing smithi08477203-23 from 689186 to 1488644 tid 1 2024-09-20T22:46:21.057 INFO:tasks.rados.rados.0.smithi084.stdout:1078: writing smithi08477203-23 from 1923598 to 2675728 tid 2 2024-09-20T22:46:21.057 INFO:tasks.rados.rados.0.smithi084.stdout:1075: done (2 left) 2024-09-20T22:46:21.057 INFO:tasks.rados.rados.0.smithi084.stdout:1079: delete oid 49 current snap is 101 2024-09-20T22:46:21.057 INFO:tasks.rados.rados.0.smithi084.stdout:1077: finishing write tid 1 to smithi08477203-38 2024-09-20T22:46:21.057 INFO:tasks.rados.rados.0.smithi084.stdout:1077: finishing write tid 2 to smithi08477203-38 2024-09-20T22:46:21.057 INFO:tasks.rados.rados.0.smithi084.stdout:1077: finishing write tid 3 to smithi08477203-38 2024-09-20T22:46:21.057 INFO:tasks.rados.rados.0.smithi084.stdout:1077: finishing write tid 4 to smithi08477203-38 2024-09-20T22:46:21.057 INFO:tasks.rados.rados.0.smithi084.stdout:1077: oid 38 updating version 0 to 385 2024-09-20T22:46:21.058 INFO:tasks.rados.rados.0.smithi084.stdout:1077: oid 38 version 385 is already newer than 384 2024-09-20T22:46:21.058 INFO:tasks.rados.rados.0.smithi084.stdout:1077: oid 38 version 385 is already newer than 383 2024-09-20T22:46:21.058 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 38 v 385 (ObjNum 397 snap 101 seq_num 397) dirty exists 2024-09-20T22:46:21.058 INFO:tasks.rados.rados.0.smithi084.stdout:1077: left oid 38 (ObjNum 397 snap 101 seq_num 397) 2024-09-20T22:46:21.058 INFO:tasks.rados.rados.0.smithi084.stdout:1078: finishing write tid 1 to smithi08477203-23 2024-09-20T22:46:21.058 INFO:tasks.rados.rados.0.smithi084.stdout:1077: done (2 left) 2024-09-20T22:46:21.058 INFO:tasks.rados.rados.0.smithi084.stdout:1079: done (1 left) 2024-09-20T22:46:21.059 INFO:tasks.rados.rados.0.smithi084.stdout:1080: write (excl) oid 29 current snap is 101 2024-09-20T22:46:21.059 INFO:tasks.rados.rados.0.smithi084.stdout:1080: seq_num 400 ranges {722800=549956,2040562=682238} 2024-09-20T22:46:21.065 INFO:tasks.rados.rados.0.smithi084.stdout:1080: writing smithi08477203-29 from 722800 to 1272756 tid 1 2024-09-20T22:46:21.074 INFO:tasks.rados.rados.0.smithi084.stdout:1080: writing smithi08477203-29 from 2040562 to 2722800 tid 2 2024-09-20T22:46:21.074 INFO:tasks.rados.rados.0.smithi084.stdout:1081: rollback oid 46 current snap is 101 2024-09-20T22:46:21.074 INFO:tasks.rados.rados.0.smithi084.stdout:1081: done (2 left) 2024-09-20T22:46:21.074 INFO:tasks.rados.rados.0.smithi084.stdout:1082: copy_from oid 22 from oid 10 current snap is 101 2024-09-20T22:46:21.074 INFO:tasks.rados.rados.0.smithi084.stdout:1078: finishing write tid 2 to smithi08477203-23 2024-09-20T22:46:21.074 INFO:tasks.rados.rados.0.smithi084.stdout:1078: finishing write tid 3 to smithi08477203-23 2024-09-20T22:46:21.074 INFO:tasks.rados.rados.0.smithi084.stdout:1078: finishing write tid 4 to smithi08477203-23 2024-09-20T22:46:21.074 INFO:tasks.rados.rados.0.smithi084.stdout:1078: oid 23 updating version 0 to 217 2024-09-20T22:46:21.074 INFO:tasks.rados.rados.0.smithi084.stdout:1078: oid 23 updating version 217 to 218 2024-09-20T22:46:21.074 INFO:tasks.rados.rados.0.smithi084.stdout:1078: oid 23 updating version 218 to 219 2024-09-20T22:46:21.075 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 23 v 219 (ObjNum 398 snap 101 seq_num 398) dirty exists 2024-09-20T22:46:21.075 INFO:tasks.rados.rados.0.smithi084.stdout:1078: left oid 23 (ObjNum 398 snap 101 seq_num 398) 2024-09-20T22:46:21.075 INFO:tasks.rados.rados.0.smithi084.stdout:1080: finishing write tid 1 to smithi08477203-29 2024-09-20T22:46:21.075 INFO:tasks.rados.rados.0.smithi084.stdout:1078: done (2 left) 2024-09-20T22:46:21.075 INFO:tasks.rados.rados.0.smithi084.stdout:1083: rollback oid 6 current snap is 101 2024-09-20T22:46:21.075 INFO:tasks.rados.rados.0.smithi084.stdout:1083: done (2 left) 2024-09-20T22:46:21.075 INFO:tasks.rados.rados.0.smithi084.stdout:1084: rollback oid 31 current snap is 101 2024-09-20T22:46:21.075 INFO:tasks.rados.rados.0.smithi084.stdout:1084: done (2 left) 2024-09-20T22:46:21.075 INFO:tasks.rados.rados.0.smithi084.stdout:1085: write oid 32 current snap is 101 2024-09-20T22:46:21.075 INFO:tasks.rados.rados.0.smithi084.stdout:1085: seq_num 401 ranges {739607=625205,1982691=499457} 2024-09-20T22:46:21.081 INFO:tasks.rados.rados.0.smithi084.stdout:1085: writing smithi08477203-32 from 739607 to 1364812 tid 1 2024-09-20T22:46:21.087 INFO:tasks.rados.rados.0.smithi084.stdout:1085: writing smithi08477203-32 from 1982691 to 2482148 tid 2 2024-09-20T22:46:21.088 INFO:tasks.rados.rados.0.smithi084.stdout:1086: write (excl) oid 13 current snap is 101 2024-09-20T22:46:21.088 INFO:tasks.rados.rados.0.smithi084.stdout:1086: seq_num 402 ranges {756414=700454,2041173=715241} 2024-09-20T22:46:21.095 INFO:tasks.rados.rados.0.smithi084.stdout:1086: writing smithi08477203-13 from 756414 to 1456868 tid 1 2024-09-20T22:46:21.103 INFO:tasks.rados.rados.0.smithi084.stdout:1086: writing smithi08477203-13 from 2041173 to 2756414 tid 2 2024-09-20T22:46:21.103 INFO:tasks.rados.rados.0.smithi084.stdout:1082: finishing copy_from racing read to smithi08477203-22 2024-09-20T22:46:21.103 INFO:tasks.rados.rados.0.smithi084.stdout:1087: read oid 21 snap -1 2024-09-20T22:46:21.103 INFO:tasks.rados.rados.0.smithi084.stdout:1087: expect (ObjNum 387 snap 100 seq_num 387) 2024-09-20T22:46:21.103 INFO:tasks.rados.rados.0.smithi084.stdout:1088: write oid 1 current snap is 101 2024-09-20T22:46:21.104 INFO:tasks.rados.rados.0.smithi084.stdout:1080: finishing write tid 2 to smithi08477203-29 2024-09-20T22:46:21.104 INFO:tasks.rados.rados.0.smithi084.stdout:1088: seq_num 403 ranges {773221=492056,1816008=503126} 2024-09-20T22:46:21.110 INFO:tasks.rados.rados.0.smithi084.stdout:1088: writing smithi08477203-1 from 773221 to 1265277 tid 1 2024-09-20T22:46:21.116 INFO:tasks.rados.rados.0.smithi084.stdout:1088: writing smithi08477203-1 from 1816008 to 2319134 tid 2 2024-09-20T22:46:21.116 INFO:tasks.rados.rados.0.smithi084.stdout:1089: copy_from oid 3 from oid 45 current snap is 101 2024-09-20T22:46:21.116 INFO:tasks.rados.rados.0.smithi084.stdout:1080: finishing write tid 3 to smithi08477203-29 2024-09-20T22:46:21.116 INFO:tasks.rados.rados.0.smithi084.stdout:1080: finishing write tid 4 to smithi08477203-29 2024-09-20T22:46:21.116 INFO:tasks.rados.rados.0.smithi084.stdout:1080: oid 29 updating version 0 to 577 2024-09-20T22:46:21.116 INFO:tasks.rados.rados.0.smithi084.stdout:1080: oid 29 version 577 is already newer than 576 2024-09-20T22:46:21.116 INFO:tasks.rados.rados.0.smithi084.stdout:1080: oid 29 updating version 577 to 578 2024-09-20T22:46:21.116 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 29 v 578 (ObjNum 400 snap 101 seq_num 400) dirty exists 2024-09-20T22:46:21.117 INFO:tasks.rados.rados.0.smithi084.stdout:1080: left oid 29 (ObjNum 400 snap 101 seq_num 400) 2024-09-20T22:46:21.117 INFO:tasks.rados.rados.0.smithi084.stdout:1082: finishing copy_from to smithi08477203-22 2024-09-20T22:46:21.117 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 22 v 295 (ObjNum 391 snap 101 seq_num 391) dirty exists 2024-09-20T22:46:21.117 INFO:tasks.rados.rados.0.smithi084.stdout:1085: finishing write tid 1 to smithi08477203-32 2024-09-20T22:46:21.117 INFO:tasks.rados.rados.0.smithi084.stdout:1085: finishing write tid 2 to smithi08477203-32 2024-09-20T22:46:21.117 INFO:tasks.rados.rados.0.smithi084.stdout:1085: finishing write tid 3 to smithi08477203-32 2024-09-20T22:46:21.117 INFO:tasks.rados.rados.0.smithi084.stdout:1080: done (6 left) 2024-09-20T22:46:21.117 INFO:tasks.rados.rados.0.smithi084.stdout:1082: done (5 left) 2024-09-20T22:46:21.117 INFO:tasks.rados.rados.0.smithi084.stdout:1090: rollback oid 29 current snap is 101 2024-09-20T22:46:21.117 INFO:tasks.rados.rados.0.smithi084.stdout:1090: done (5 left) 2024-09-20T22:46:21.117 INFO:tasks.rados.rados.0.smithi084.stdout:1091: write (excl) oid 50 current snap is 101 2024-09-20T22:46:21.117 INFO:tasks.rados.rados.0.smithi084.stdout:1085: finishing write tid 4 to smithi08477203-32 2024-09-20T22:46:21.117 INFO:tasks.rados.rados.0.smithi084.stdout:1085: oid 32 updating version 0 to 381 2024-09-20T22:46:21.117 INFO:tasks.rados.rados.0.smithi084.stdout:1085: oid 32 updating version 381 to 382 2024-09-20T22:46:21.117 INFO:tasks.rados.rados.0.smithi084.stdout:1085: oid 32 version 382 is already newer than 380 2024-09-20T22:46:21.118 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 32 v 382 (ObjNum 401 snap 101 seq_num 401) dirty exists 2024-09-20T22:46:21.118 INFO:tasks.rados.rados.0.smithi084.stdout:1085: left oid 32 (ObjNum 401 snap 101 seq_num 401) 2024-09-20T22:46:21.118 INFO:tasks.rados.rados.0.smithi084.stdout:1086: finishing write tid 1 to smithi08477203-13 2024-09-20T22:46:21.118 INFO:tasks.rados.rados.0.smithi084.stdout:1091: seq_num 404 ranges {790028=567305,1874490=763137} 2024-09-20T22:46:21.123 INFO:tasks.rados.rados.0.smithi084.stdout:1091: writing smithi08477203-50 from 790028 to 1357333 tid 1 2024-09-20T22:46:21.126 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:46:21.131 INFO:tasks.rados.rados.0.smithi084.stdout:1091: writing smithi08477203-50 from 1874490 to 2637627 tid 2 2024-09-20T22:46:21.131 INFO:tasks.rados.rados.0.smithi084.stdout:1085: done (5 left) 2024-09-20T22:46:21.131 INFO:tasks.rados.rados.0.smithi084.stdout:1092: copy_from oid 46 from oid 11 current snap is 101 2024-09-20T22:46:21.131 INFO:tasks.rados.rados.0.smithi084.stdout:1087: expect (ObjNum 387 snap 100 seq_num 387) 2024-09-20T22:46:21.149 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:21.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:21.150 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:46:21.250 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:46:21.271 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:21.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:21.272 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:46:21.364 INFO:tasks.rados.rados.0.smithi084.stdout:1086: finishing write tid 2 to smithi08477203-13 2024-09-20T22:46:21.364 INFO:tasks.rados.rados.0.smithi084.stdout:1086: finishing write tid 3 to smithi08477203-13 2024-09-20T22:46:21.364 INFO:tasks.rados.rados.0.smithi084.stdout:1086: finishing write tid 4 to smithi08477203-13 2024-09-20T22:46:21.364 INFO:tasks.rados.rados.0.smithi084.stdout:1086: oid 13 updating version 0 to 580 2024-09-20T22:46:21.364 INFO:tasks.rados.rados.0.smithi084.stdout:1086: oid 13 version 580 is already newer than 579 2024-09-20T22:46:21.364 INFO:tasks.rados.rados.0.smithi084.stdout:1086: oid 13 updating version 580 to 581 2024-09-20T22:46:21.364 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 13 v 581 (ObjNum 402 snap 101 seq_num 402) dirty exists 2024-09-20T22:46:21.364 INFO:tasks.rados.rados.0.smithi084.stdout:1086: left oid 13 (ObjNum 402 snap 101 seq_num 402) 2024-09-20T22:46:21.364 INFO:tasks.rados.rados.0.smithi084.stdout:1088: finishing write tid 1 to smithi08477203-1 2024-09-20T22:46:21.365 INFO:tasks.rados.rados.0.smithi084.stdout:1086: done (5 left) 2024-09-20T22:46:21.365 INFO:tasks.rados.rados.0.smithi084.stdout:1087: done (4 left) 2024-09-20T22:46:21.365 INFO:tasks.rados.rados.0.smithi084.stdout:1093: write (excl) oid 12 current snap is 101 2024-09-20T22:46:21.365 INFO:tasks.rados.rados.0.smithi084.stdout:1088: finishing write tid 2 to smithi08477203-1 2024-09-20T22:46:21.365 INFO:tasks.rados.rados.0.smithi084.stdout:1093: seq_num 405 ranges {406835=642554,1532972=506795,2772312=34523} 2024-09-20T22:46:21.369 INFO:tasks.rados.rados.0.smithi084.stdout:1093: writing smithi08477203-12 from 406835 to 1049389 tid 1 2024-09-20T22:46:21.373 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:46:21.375 INFO:tasks.rados.rados.0.smithi084.stdout:1093: writing smithi08477203-12 from 1532972 to 2039767 tid 2 2024-09-20T22:46:21.379 INFO:tasks.rados.rados.0.smithi084.stdout:1093: writing smithi08477203-12 from 2772312 to 2806835 tid 3 2024-09-20T22:46:21.379 INFO:tasks.rados.rados.0.smithi084.stdout:1094: write (excl) oid 16 current snap is 101 2024-09-20T22:46:21.379 INFO:tasks.rados.rados.0.smithi084.stdout:1094: seq_num 406 ranges {423642=717803,1875101=766806} 2024-09-20T22:46:21.385 INFO:tasks.rados.rados.0.smithi084.stdout:1094: writing smithi08477203-16 from 423642 to 1141445 tid 1 2024-09-20T22:46:21.394 INFO:tasks.rados.rados.0.smithi084.stdout:1094: writing smithi08477203-16 from 1875101 to 2641907 tid 2 2024-09-20T22:46:21.394 INFO:tasks.rados.rados.0.smithi084.stdout:1095: snap_create 2024-09-20T22:46:21.394 INFO:tasks.rados.rados.0.smithi084.stdout:1088: finishing write tid 3 to smithi08477203-1 2024-09-20T22:46:21.394 INFO:tasks.rados.rados.0.smithi084.stdout:1088: finishing write tid 4 to smithi08477203-1 2024-09-20T22:46:21.394 INFO:tasks.rados.rados.0.smithi084.stdout:1088: oid 1 updating version 0 to 299 2024-09-20T22:46:21.394 INFO:tasks.rados.rados.0.smithi084.stdout:1088: oid 1 updating version 299 to 300 2024-09-20T22:46:21.395 INFO:tasks.rados.rados.0.smithi084.stdout:1088: oid 1 updating version 300 to 301 2024-09-20T22:46:21.395 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 1 v 301 (ObjNum 403 snap 101 seq_num 403) dirty exists 2024-09-20T22:46:21.395 INFO:tasks.rados.rados.0.smithi084.stdout:1088: left oid 1 (ObjNum 403 snap 101 seq_num 403) 2024-09-20T22:46:21.395 INFO:tasks.rados.rados.0.smithi084.stdout:1091: finishing write tid 1 to smithi08477203-50 2024-09-20T22:46:21.395 INFO:tasks.rados.rados.0.smithi084.stdout:1089: finishing copy_from racing read to smithi08477203-3 2024-09-20T22:46:21.395 INFO:tasks.rados.rados.0.smithi084.stdout:1092: finishing copy_from to smithi08477203-46 2024-09-20T22:46:21.395 INFO:tasks.rados.rados.0.smithi084.stdout:1092: got expected ENOENT (src dne) 2024-09-20T22:46:21.395 INFO:tasks.rados.rados.0.smithi084.stdout:1092: finishing copy_from racing read to smithi08477203-46 2024-09-20T22:46:21.395 INFO:tasks.rados.rados.0.smithi084.stdout:1091: finishing write tid 2 to smithi08477203-50 2024-09-20T22:46:21.395 INFO:tasks.rados.rados.0.smithi084.stdout:1091: finishing write tid 3 to smithi08477203-50 2024-09-20T22:46:21.395 INFO:tasks.rados.rados.0.smithi084.stdout:1091: finishing write tid 4 to smithi08477203-50 2024-09-20T22:46:21.395 INFO:tasks.rados.rados.0.smithi084.stdout:1091: oid 50 updating version 0 to 240 2024-09-20T22:46:21.396 INFO:tasks.rados.rados.0.smithi084.stdout:1091: oid 50 updating version 240 to 241 2024-09-20T22:46:21.396 INFO:tasks.rados.rados.0.smithi084.stdout:1091: oid 50 updating version 241 to 242 2024-09-20T22:46:21.396 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 50 v 242 (ObjNum 404 snap 101 seq_num 404) dirty exists 2024-09-20T22:46:21.396 INFO:tasks.rados.rados.0.smithi084.stdout:1091: left oid 50 (ObjNum 404 snap 101 seq_num 404) 2024-09-20T22:46:21.396 INFO:tasks.rados.rados.0.smithi084.stdout:1089: finishing copy_from to smithi08477203-3 2024-09-20T22:46:21.396 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 3 v 114 (ObjNum 338 snap 85 seq_num 338) dirty exists 2024-09-20T22:46:21.396 INFO:tasks.rados.rados.0.smithi084.stdout:1093: finishing write tid 1 to smithi08477203-12 2024-09-20T22:46:21.396 INFO:tasks.rados.rados.0.smithi084.stdout:1093: finishing write tid 2 to smithi08477203-12 2024-09-20T22:46:21.396 INFO:tasks.rados.rados.0.smithi084.stdout:1093: finishing write tid 3 to smithi08477203-12 2024-09-20T22:46:21.396 INFO:tasks.rados.rados.0.smithi084.stdout:1093: finishing write tid 4 to smithi08477203-12 2024-09-20T22:46:21.396 INFO:tasks.rados.rados.0.smithi084.stdout:1093: finishing write tid 5 to smithi08477203-12 2024-09-20T22:46:21.396 INFO:tasks.rados.rados.0.smithi084.stdout:1093: oid 12 updating version 0 to 255 2024-09-20T22:46:21.396 INFO:tasks.rados.rados.0.smithi084.stdout:1093: oid 12 version 255 is already newer than 254 2024-09-20T22:46:21.396 INFO:tasks.rados.rados.0.smithi084.stdout:1093: oid 12 version 255 is already newer than 253 2024-09-20T22:46:21.397 INFO:tasks.rados.rados.0.smithi084.stdout:1093: oid 12 updating version 255 to 256 2024-09-20T22:46:21.397 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 12 v 256 (ObjNum 405 snap 101 seq_num 405) dirty exists 2024-09-20T22:46:21.397 INFO:tasks.rados.rados.0.smithi084.stdout:1093: left oid 12 (ObjNum 405 snap 101 seq_num 405) 2024-09-20T22:46:21.397 INFO:tasks.rados.rados.0.smithi084.stdout:1094: finishing write tid 1 to smithi08477203-16 2024-09-20T22:46:21.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:21.398 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:21.398 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:46:21.399 INFO:tasks.rados.rados.0.smithi084.stdout:1094: finishing write tid 2 to smithi08477203-16 2024-09-20T22:46:21.399 INFO:tasks.rados.rados.0.smithi084.stdout:1094: finishing write tid 3 to smithi08477203-16 2024-09-20T22:46:21.399 INFO:tasks.rados.rados.0.smithi084.stdout:1094: finishing write tid 4 to smithi08477203-16 2024-09-20T22:46:21.399 INFO:tasks.rados.rados.0.smithi084.stdout:1094: oid 16 updating version 0 to 177 2024-09-20T22:46:21.399 INFO:tasks.rados.rados.0.smithi084.stdout:1094: oid 16 updating version 177 to 178 2024-09-20T22:46:21.399 INFO:tasks.rados.rados.0.smithi084.stdout:1094: oid 16 version 178 is already newer than 176 2024-09-20T22:46:21.399 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 16 v 178 (ObjNum 406 snap 101 seq_num 406) dirty exists 2024-09-20T22:46:21.399 INFO:tasks.rados.rados.0.smithi084.stdout:1094: left oid 16 (ObjNum 406 snap 101 seq_num 406) 2024-09-20T22:46:21.448 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:21 smithi084 ceph-mon[29774]: pgmap v352: 168 pgs: 4 active+clean+snaptrim, 1 peering, 6 active+clean+snaptrim_wait, 157 active+clean; 97 MiB data, 461 MiB used, 536 GiB / 536 GiB avail; 17 MiB/s rd, 3.2 MiB/s wr, 29 op/s; 80 KiB/s, 0 objects/s recovering 2024-09-20T22:46:21.448 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:21 smithi084 ceph-mon[29774]: purged_snaps scrub starts 2024-09-20T22:46:21.448 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:21 smithi084 ceph-mon[29774]: purged_snaps scrub ok 2024-09-20T22:46:21.448 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:21 smithi084 ceph-mon[29774]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:46:21.448 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:21 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/2021203486' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T22:46:21.449 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:21 smithi084 ceph-mon[29774]: osdmap e301: 8 total, 8 up, 6 in 2024-09-20T22:46:21.449 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:21 smithi084 ceph-mon[40024]: pgmap v352: 168 pgs: 4 active+clean+snaptrim, 1 peering, 6 active+clean+snaptrim_wait, 157 active+clean; 97 MiB data, 461 MiB used, 536 GiB / 536 GiB avail; 17 MiB/s rd, 3.2 MiB/s wr, 29 op/s; 80 KiB/s, 0 objects/s recovering 2024-09-20T22:46:21.449 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:21 smithi084 ceph-mon[40024]: purged_snaps scrub starts 2024-09-20T22:46:21.449 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:21 smithi084 ceph-mon[40024]: purged_snaps scrub ok 2024-09-20T22:46:21.449 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:21 smithi084 ceph-mon[40024]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:46:21.449 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:21 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/2021203486' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T22:46:21.449 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:21 smithi084 ceph-mon[40024]: osdmap e301: 8 total, 8 up, 6 in 2024-09-20T22:46:21.498 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:46:21.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:21 smithi154 ceph-mon[33264]: pgmap v352: 168 pgs: 4 active+clean+snaptrim, 1 peering, 6 active+clean+snaptrim_wait, 157 active+clean; 97 MiB data, 461 MiB used, 536 GiB / 536 GiB avail; 17 MiB/s rd, 3.2 MiB/s wr, 29 op/s; 80 KiB/s, 0 objects/s recovering 2024-09-20T22:46:21.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:21 smithi154 ceph-mon[33264]: purged_snaps scrub starts 2024-09-20T22:46:21.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:21 smithi154 ceph-mon[33264]: purged_snaps scrub ok 2024-09-20T22:46:21.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:21 smithi154 ceph-mon[33264]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:46:21.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:21 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/2021203486' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T22:46:21.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:21 smithi154 ceph-mon[33264]: osdmap e301: 8 total, 8 up, 6 in 2024-09-20T22:46:21.519 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:21.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:21.520 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:46:21.621 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:46:21.643 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:21.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:21.644 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:46:21.702 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-20T22:46:21.702 INFO:teuthology.orchestra.run.smithi084.stdout: "success": "" 2024-09-20T22:46:21.703 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-20T22:46:21.744 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:46:21.765 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:21.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:21.766 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:46:21.811 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:46:21.811 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:46:21.811 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:46:21.811 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-20T22:46:21.811 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:46:21.811 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:46:21.811 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:46:21.811 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:46:21.812 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:46:21.812 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:46:21.812 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T22:46:21.812 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:46:21.812 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:46:21.812 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:46:21.812 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:46:21.812 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:46:21.868 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:46:21.890 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:21.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:21.891 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:46:21.991 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:46:22.012 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:22.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:22.013 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:46:22.114 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:46:22.134 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:22.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:22.135 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:46:22.210 INFO:tasks.rados.rados.0.smithi084.stdout:1088: done (6 left) 2024-09-20T22:46:22.210 INFO:tasks.rados.rados.0.smithi084.stdout:1089: done (5 left) 2024-09-20T22:46:22.210 INFO:tasks.rados.rados.0.smithi084.stdout:1091: done (4 left) 2024-09-20T22:46:22.210 INFO:tasks.rados.rados.0.smithi084.stdout:1092: done (3 left) 2024-09-20T22:46:22.210 INFO:tasks.rados.rados.0.smithi084.stdout:1093: done (2 left) 2024-09-20T22:46:22.210 INFO:tasks.rados.rados.0.smithi084.stdout:1094: done (1 left) 2024-09-20T22:46:22.210 INFO:tasks.rados.rados.0.smithi084.stdout:1095: done (0 left) 2024-09-20T22:46:22.210 INFO:tasks.rados.rados.0.smithi084.stdout:1096: rollback oid 44 current snap is 102 2024-09-20T22:46:22.210 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 44 to 101 2024-09-20T22:46:22.210 INFO:tasks.rados.rados.0.smithi084.stdout:1097: read oid 19 snap 101 2024-09-20T22:46:22.210 INFO:tasks.rados.rados.0.smithi084.stdout:1097: expect (ObjNum 258 snap 60 seq_num 258) 2024-09-20T22:46:22.210 INFO:tasks.rados.rados.0.smithi084.stdout:1098: read oid 2 snap -1 2024-09-20T22:46:22.210 INFO:tasks.rados.rados.0.smithi084.stdout:1098: expect (ObjNum 300 snap 75 seq_num 300) 2024-09-20T22:46:22.211 INFO:tasks.rados.rados.0.smithi084.stdout:1099: rollback oid 14 current snap is 102 2024-09-20T22:46:22.211 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 14 to 101 2024-09-20T22:46:22.211 INFO:tasks.rados.rados.0.smithi084.stdout:1100: snap_create 2024-09-20T22:46:22.215 INFO:tasks.rados.rados.0.smithi084.stdout:1099: finishing rollback tid 1 to smithi08477203-14 2024-09-20T22:46:22.216 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 14 v 0 (ObjNum 129 snap 0 seq_num 2090802960) dirty dne 2024-09-20T22:46:22.218 INFO:tasks.rados.rados.0.smithi084.stdout:1096: finishing rollback tid 0 to smithi08477203-44 2024-09-20T22:46:22.224 INFO:tasks.rados.rados.0.smithi084.stdout:1098: expect (ObjNum 300 snap 75 seq_num 300) 2024-09-20T22:46:22.236 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:46:22.259 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:22.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:22.260 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:46:22.328 INFO:tasks.rados.rados.0.smithi084.stdout:1097: expect (ObjNum 258 snap 60 seq_num 258) 2024-09-20T22:46:22.361 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:46:22.382 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:22.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:22.383 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:46:22.471 INFO:tasks.rados.rados.0.smithi084.stdout:1096: finishing rollback tid 1 to smithi08477203-44 2024-09-20T22:46:22.471 INFO:tasks.rados.rados.0.smithi084.stdout:1096: finishing rollback tid 2 to smithi08477203-44 2024-09-20T22:46:22.471 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 44 v 585 (ObjNum 359 snap 92 seq_num 359) dirty exists 2024-09-20T22:46:22.484 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:46:22.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:22 smithi154 ceph-mon[33264]: 5.1e scrub starts 2024-09-20T22:46:22.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:22 smithi154 ceph-mon[33264]: 5.1e scrub ok 2024-09-20T22:46:22.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:22 smithi154 ceph-mon[33264]: 4.18 scrub starts 2024-09-20T22:46:22.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:22 smithi154 ceph-mon[33264]: 4.18 scrub ok 2024-09-20T22:46:22.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:22 smithi154 ceph-mon[33264]: 4.16 scrub starts 2024-09-20T22:46:22.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:22 smithi154 ceph-mon[33264]: 4.16 scrub ok 2024-09-20T22:46:22.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:22 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/251528281' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:46:22.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:22.506 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:22.506 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:46:22.582 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:22 smithi084 ceph-mon[29774]: 5.1e scrub starts 2024-09-20T22:46:22.582 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:22 smithi084 ceph-mon[29774]: 5.1e scrub ok 2024-09-20T22:46:22.582 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:22 smithi084 ceph-mon[29774]: 4.18 scrub starts 2024-09-20T22:46:22.583 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:22 smithi084 ceph-mon[29774]: 4.18 scrub ok 2024-09-20T22:46:22.583 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:22 smithi084 ceph-mon[29774]: 4.16 scrub starts 2024-09-20T22:46:22.583 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:22 smithi084 ceph-mon[29774]: 4.16 scrub ok 2024-09-20T22:46:22.583 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:22 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/251528281' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:46:22.583 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:22 smithi084 ceph-mon[40024]: 5.1e scrub starts 2024-09-20T22:46:22.583 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:22 smithi084 ceph-mon[40024]: 5.1e scrub ok 2024-09-20T22:46:22.583 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:22 smithi084 ceph-mon[40024]: 4.18 scrub starts 2024-09-20T22:46:22.583 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:22 smithi084 ceph-mon[40024]: 4.18 scrub ok 2024-09-20T22:46:22.583 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:22 smithi084 ceph-mon[40024]: 4.16 scrub starts 2024-09-20T22:46:22.583 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:22 smithi084 ceph-mon[40024]: 4.16 scrub ok 2024-09-20T22:46:22.583 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:22 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/251528281' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:46:22.607 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:46:22.628 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:22.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:22.629 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:46:22.729 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:46:22.750 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:22.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:22.751 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:46:22.851 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:46:22.873 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:22.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:22.874 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:46:22.897 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-20T22:46:22.975 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:46:23.001 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:23.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:23.002 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:46:23.058 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:46:23.102 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:46:23.127 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:23.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:23.128 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:46:23.173 INFO:tasks.rados.rados.0.smithi084.stdout:1096: done (4 left) 2024-09-20T22:46:23.173 INFO:tasks.rados.rados.0.smithi084.stdout:1097: done (3 left) 2024-09-20T22:46:23.173 INFO:tasks.rados.rados.0.smithi084.stdout:1098: done (2 left) 2024-09-20T22:46:23.173 INFO:tasks.rados.rados.0.smithi084.stdout:1099: done (1 left) 2024-09-20T22:46:23.173 INFO:tasks.rados.rados.0.smithi084.stdout:1100: done (0 left) 2024-09-20T22:46:23.173 INFO:tasks.rados.rados.0.smithi084.stdout:1101: read oid 28 snap -1 2024-09-20T22:46:23.173 INFO:tasks.rados.rados.0.smithi084.stdout:1101: expect (ObjNum 396 snap 101 seq_num 396) 2024-09-20T22:46:23.173 INFO:tasks.rados.rados.0.smithi084.stdout:1102: delete oid 35 current snap is 103 2024-09-20T22:46:23.175 INFO:tasks.rados.rados.0.smithi084.stdout:1102: done (1 left) 2024-09-20T22:46:23.175 INFO:tasks.rados.rados.0.smithi084.stdout:1103: rollback oid 48 current snap is 103 2024-09-20T22:46:23.175 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 48 to 101 2024-09-20T22:46:23.175 INFO:tasks.rados.rados.0.smithi084.stdout:1104: delete oid 15 current snap is 103 2024-09-20T22:46:23.181 INFO:tasks.rados.rados.0.smithi084.stdout:1103: finishing rollback tid 0 to smithi08477203-48 2024-09-20T22:46:23.182 INFO:tasks.rados.rados.0.smithi084.stdout:1103: finishing rollback tid 1 to smithi08477203-48 2024-09-20T22:46:23.182 INFO:tasks.rados.rados.0.smithi084.stdout:1103: finishing rollback tid 2 to smithi08477203-48 2024-09-20T22:46:23.182 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 48 v 234 (ObjNum 328 snap 84 seq_num 328) dirty exists 2024-09-20T22:46:23.185 INFO:tasks.rados.rados.0.smithi084.stdout:1101: expect (ObjNum 396 snap 101 seq_num 396) 2024-09-20T22:46:23.229 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:46:23.252 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:23.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:23.253 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:46:23.270 INFO:tasks.rados.rados.0.smithi084.stdout:1101: done (2 left) 2024-09-20T22:46:23.271 INFO:tasks.rados.rados.0.smithi084.stdout:1103: done (1 left) 2024-09-20T22:46:23.271 INFO:tasks.rados.rados.0.smithi084.stdout:1104: done (0 left) 2024-09-20T22:46:23.271 INFO:tasks.rados.rados.0.smithi084.stdout:1105: write oid 35 current snap is 103 2024-09-20T22:46:23.271 INFO:tasks.rados.rados.0.smithi084.stdout:1105: seq_num 409 ranges {474063=543550,1650547=514133,2765651=108412} 2024-09-20T22:46:23.276 INFO:tasks.rados.rados.0.smithi084.stdout:1105: writing smithi08477203-35 from 474063 to 1017613 tid 1 2024-09-20T22:46:23.282 INFO:tasks.rados.rados.0.smithi084.stdout:1105: writing smithi08477203-35 from 1650547 to 2164680 tid 2 2024-09-20T22:46:23.286 INFO:tasks.rados.rados.0.smithi084.stdout:1105: writing smithi08477203-35 from 2765651 to 2874063 tid 3 2024-09-20T22:46:23.286 INFO:tasks.rados.rados.0.smithi084.stdout:1106: read oid 32 snap 101 2024-09-20T22:46:23.286 INFO:tasks.rados.rados.0.smithi084.stdout:1106: expect (ObjNum 401 snap 101 seq_num 401) 2024-09-20T22:46:23.286 INFO:tasks.rados.rados.0.smithi084.stdout:1107: copy_from oid 50 from oid 3 current snap is 103 2024-09-20T22:46:23.286 INFO:tasks.rados.rados.0.smithi084.stdout:1105: finishing write tid 1 to smithi08477203-35 2024-09-20T22:46:23.286 INFO:tasks.rados.rados.0.smithi084.stdout:1105: finishing write tid 2 to smithi08477203-35 2024-09-20T22:46:23.287 INFO:tasks.rados.rados.0.smithi084.stdout:1108: rollback oid 16 current snap is 103 2024-09-20T22:46:23.287 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 16 to 101 2024-09-20T22:46:23.287 INFO:tasks.rados.rados.0.smithi084.stdout:1109: snap_create 2024-09-20T22:46:23.288 INFO:tasks.rados.rados.0.smithi084.stdout:1105: finishing write tid 3 to smithi08477203-35 2024-09-20T22:46:23.288 INFO:tasks.rados.rados.0.smithi084.stdout:1105: finishing write tid 4 to smithi08477203-35 2024-09-20T22:46:23.288 INFO:tasks.rados.rados.0.smithi084.stdout:1105: finishing write tid 5 to smithi08477203-35 2024-09-20T22:46:23.288 INFO:tasks.rados.rados.0.smithi084.stdout:1105: oid 35 updating version 0 to 383 2024-09-20T22:46:23.288 INFO:tasks.rados.rados.0.smithi084.stdout:1105: oid 35 version 383 is already newer than 382 2024-09-20T22:46:23.289 INFO:tasks.rados.rados.0.smithi084.stdout:1105: oid 35 updating version 383 to 384 2024-09-20T22:46:23.289 INFO:tasks.rados.rados.0.smithi084.stdout:1105: oid 35 version 384 is already newer than 381 2024-09-20T22:46:23.289 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 35 v 384 (ObjNum 409 snap 103 seq_num 409) dirty exists 2024-09-20T22:46:23.289 INFO:tasks.rados.rados.0.smithi084.stdout:1105: left oid 35 (ObjNum 409 snap 103 seq_num 409) 2024-09-20T22:46:23.289 INFO:tasks.rados.rados.0.smithi084.stdout:1106: expect (ObjNum 401 snap 101 seq_num 401) 2024-09-20T22:46:23.354 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:46:23.375 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:23.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:23.377 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:46:23.473 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:23 smithi084 ceph-mon[29774]: 6.1c scrub starts 2024-09-20T22:46:23.474 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:23 smithi084 ceph-mon[29774]: 6.1c scrub ok 2024-09-20T22:46:23.474 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:23 smithi084 ceph-mon[29774]: pgmap v354: 168 pgs: 4 active+clean+snaptrim, 1 peering, 6 active+clean+snaptrim_wait, 157 active+clean; 97 MiB data, 461 MiB used, 536 GiB / 536 GiB avail; 15 MiB/s rd, 2.8 MiB/s wr, 25 op/s; 71 KiB/s, 0 objects/s recovering 2024-09-20T22:46:23.474 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:23 smithi084 ceph-mon[29774]: 5.1f scrub starts 2024-09-20T22:46:23.474 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:23 smithi084 ceph-mon[29774]: 5.1f scrub ok 2024-09-20T22:46:23.474 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:23 smithi084 ceph-mon[29774]: 7.2 scrub starts 2024-09-20T22:46:23.474 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:23 smithi084 ceph-mon[29774]: 7.2 scrub ok 2024-09-20T22:46:23.474 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:23 smithi084 ceph-mon[29774]: osdmap e302: 8 total, 8 up, 6 in 2024-09-20T22:46:23.474 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:23 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:46:23.474 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:23 smithi084 ceph-mon[29774]: osdmap e303: 8 total, 8 up, 6 in 2024-09-20T22:46:23.474 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:23 smithi084 ceph-mon[40024]: 6.1c scrub starts 2024-09-20T22:46:23.475 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:23 smithi084 ceph-mon[40024]: 6.1c scrub ok 2024-09-20T22:46:23.475 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:23 smithi084 ceph-mon[40024]: pgmap v354: 168 pgs: 4 active+clean+snaptrim, 1 peering, 6 active+clean+snaptrim_wait, 157 active+clean; 97 MiB data, 461 MiB used, 536 GiB / 536 GiB avail; 15 MiB/s rd, 2.8 MiB/s wr, 25 op/s; 71 KiB/s, 0 objects/s recovering 2024-09-20T22:46:23.475 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:23 smithi084 ceph-mon[40024]: 5.1f scrub starts 2024-09-20T22:46:23.475 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:23 smithi084 ceph-mon[40024]: 5.1f scrub ok 2024-09-20T22:46:23.475 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:23 smithi084 ceph-mon[40024]: 7.2 scrub starts 2024-09-20T22:46:23.475 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:23 smithi084 ceph-mon[40024]: 7.2 scrub ok 2024-09-20T22:46:23.475 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:23 smithi084 ceph-mon[40024]: osdmap e302: 8 total, 8 up, 6 in 2024-09-20T22:46:23.475 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:23 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:46:23.475 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:23 smithi084 ceph-mon[40024]: osdmap e303: 8 total, 8 up, 6 in 2024-09-20T22:46:23.477 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:46:23.498 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:23.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:23.499 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:46:23.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:23 smithi154 ceph-mon[33264]: 6.1c scrub starts 2024-09-20T22:46:23.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:23 smithi154 ceph-mon[33264]: 6.1c scrub ok 2024-09-20T22:46:23.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:23 smithi154 ceph-mon[33264]: pgmap v354: 168 pgs: 4 active+clean+snaptrim, 1 peering, 6 active+clean+snaptrim_wait, 157 active+clean; 97 MiB data, 461 MiB used, 536 GiB / 536 GiB avail; 15 MiB/s rd, 2.8 MiB/s wr, 25 op/s; 71 KiB/s, 0 objects/s recovering 2024-09-20T22:46:23.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:23 smithi154 ceph-mon[33264]: 5.1f scrub starts 2024-09-20T22:46:23.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:23 smithi154 ceph-mon[33264]: 5.1f scrub ok 2024-09-20T22:46:23.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:23 smithi154 ceph-mon[33264]: 7.2 scrub starts 2024-09-20T22:46:23.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:23 smithi154 ceph-mon[33264]: 7.2 scrub ok 2024-09-20T22:46:23.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:23 smithi154 ceph-mon[33264]: osdmap e302: 8 total, 8 up, 6 in 2024-09-20T22:46:23.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:23 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:46:23.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:23 smithi154 ceph-mon[33264]: osdmap e303: 8 total, 8 up, 6 in 2024-09-20T22:46:23.510 INFO:tasks.rados.rados.0.smithi084.stdout:1108: finishing rollback tid 0 to smithi08477203-16 2024-09-20T22:46:23.510 INFO:tasks.rados.rados.0.smithi084.stdout:1108: finishing rollback tid 1 to smithi08477203-16 2024-09-20T22:46:23.510 INFO:tasks.rados.rados.0.smithi084.stdout:1108: finishing rollback tid 2 to smithi08477203-16 2024-09-20T22:46:23.510 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 16 v 181 (ObjNum 406 snap 101 seq_num 406) dirty exists 2024-09-20T22:46:23.510 INFO:tasks.rados.rados.0.smithi084.stdout:1107: finishing copy_from racing read to smithi08477203-50 2024-09-20T22:46:23.510 INFO:tasks.rados.rados.0.smithi084.stdout:1107: finishing copy_from to smithi08477203-50 2024-09-20T22:46:23.510 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 50 v 114 (ObjNum 338 snap 85 seq_num 338) dirty exists 2024-09-20T22:46:23.576 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-20T22:46:23.576 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2024-09-20T22:46:23.576 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2024-09-20T22:46:23.577 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [ 2024-09-20T22:46:23.577 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.577 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24652.0:336 7.7 e107/94 7:e67788e2:::smithi08477203-31:head v 107'45, mlcod=105'41)", 2024-09-20T22:46:23.577 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:43:06.546796+0000", 2024-09-20T22:46:23.577 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 197.026608915, 2024-09-20T22:46:23.577 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.015311521999999999, 2024-09-20T22:46:23.577 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:46:23.577 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:46:23.577 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:46:23.577 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:46:23.577 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.577 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:46:23.577 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:06.546796+0000", 2024-09-20T22:46:23.577 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:46:23.577 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.578 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.578 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:46:23.578 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:06.546796+0000", 2024-09-20T22:46:23.578 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:46:23.578 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.578 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.578 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:46:23.578 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:06.546796+0000", 2024-09-20T22:46:23.578 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.84e-07 2024-09-20T22:46:23.578 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.578 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.578 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:46:23.578 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:06.554340+0000", 2024-09-20T22:46:23.579 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0075444680000000004 2024-09-20T22:46:23.579 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.579 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.579 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:46:23.579 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:06.554366+0000", 2024-09-20T22:46:23.579 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.5531e-05 2024-09-20T22:46:23.579 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.579 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.579 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:46:23.579 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:06.554388+0000", 2024-09-20T22:46:23.579 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.2263e-05 2024-09-20T22:46:23.579 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.579 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.580 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:46:23.580 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:06.554465+0000", 2024-09-20T22:46:23.580 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.7296e-05 2024-09-20T22:46:23.580 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.580 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.580 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:46:23.580 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:06.554494+0000", 2024-09-20T22:46:23.580 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.8092000000000001e-05 2024-09-20T22:46:23.580 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.580 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.580 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:46:23.580 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:06.562070+0000", 2024-09-20T22:46:23.580 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0075764730000000002 2024-09-20T22:46:23.581 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.581 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.581 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:46:23.581 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:06.562107+0000", 2024-09-20T22:46:23.581 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.7115e-05 2024-09-20T22:46:23.581 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:46:23.581 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:46:23.581 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:46:23.581 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.581 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.581 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:554 7.a 7:5122292a:::smithi08477203-33:head [copy-from ver 6] snapc 13=[13,12,11,10,f,d] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-09-20T22:46:23.581 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:43:27.336824+0000", 2024-09-20T22:46:23.581 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 176.23658048199999, 2024-09-20T22:46:23.581 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.056403397000000001, 2024-09-20T22:46:23.581 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:46:23.581 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:46:23.582 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:46:23.582 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:46:23.582 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:46:23.582 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:46:23.582 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 554 2024-09-20T22:46:23.582 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.582 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:46:23.582 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.582 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:46:23.582 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:27.336824+0000", 2024-09-20T22:46:23.582 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:46:23.582 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.582 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.583 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:46:23.583 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:27.336824+0000", 2024-09-20T22:46:23.583 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:46:23.583 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.583 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.583 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:46:23.583 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:27.336826+0000", 2024-09-20T22:46:23.583 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.0339999999999999e-06 2024-09-20T22:46:23.583 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.583 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.583 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:46:23.583 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:27.336837+0000", 2024-09-20T22:46:23.583 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1243000000000001e-05 2024-09-20T22:46:23.584 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.584 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.584 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:46:23.584 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:27.336857+0000", 2024-09-20T22:46:23.584 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.9868999999999999e-05 2024-09-20T22:46:23.584 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.584 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.584 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:46:23.584 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:27.336869+0000", 2024-09-20T22:46:23.584 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1348999999999999e-05 2024-09-20T22:46:23.584 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.584 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.584 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:46:23.585 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:27.336946+0000", 2024-09-20T22:46:23.585 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.7683999999999997e-05 2024-09-20T22:46:23.585 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.585 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.585 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:46:23.585 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:27.337045+0000", 2024-09-20T22:46:23.585 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.9265999999999995e-05 2024-09-20T22:46:23.585 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.585 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.585 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 2,5", 2024-09-20T22:46:23.585 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:27.340603+0000", 2024-09-20T22:46:23.585 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.003557892 2024-09-20T22:46:23.586 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.586 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.586 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:46:23.586 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:27.343571+0000", 2024-09-20T22:46:23.586 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0029676609999999999 2024-09-20T22:46:23.586 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.586 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.586 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:46:23.586 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:27.346457+0000", 2024-09-20T22:46:23.586 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0028860040000000002 2024-09-20T22:46:23.586 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.586 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.587 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:46:23.587 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:27.393131+0000", 2024-09-20T22:46:23.587 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.046673813000000001 2024-09-20T22:46:23.587 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.587 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.587 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:46:23.587 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:27.393173+0000", 2024-09-20T22:46:23.587 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.2076999999999997e-05 2024-09-20T22:46:23.587 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.587 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.587 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:46:23.587 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:27.393227+0000", 2024-09-20T22:46:23.588 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.4505000000000001e-05 2024-09-20T22:46:23.588 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:46:23.588 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:46:23.588 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:46:23.588 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.588 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.588 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:663 7.12 7:4fa4b2b3:::smithi08477203-21:head [write 665577~781581,stat] snapc 17=[17,16,15,11,d] ondisk+write+known_if_redirected+supports_pool_eio e137)", 2024-09-20T22:46:23.588 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:43:36.161568+0000", 2024-09-20T22:46:23.588 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 167.41183630500001, 2024-09-20T22:46:23.588 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.99286350899999998, 2024-09-20T22:46:23.588 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:46:23.588 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:46:23.588 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:46:23.588 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:46:23.588 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:46:23.589 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:46:23.589 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 663 2024-09-20T22:46:23.589 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.589 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:46:23.589 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.589 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:46:23.589 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.161568+0000", 2024-09-20T22:46:23.589 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:46:23.589 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.589 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.589 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:46:23.589 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.161568+0000", 2024-09-20T22:46:23.589 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:46:23.589 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.590 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.590 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:46:23.590 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.161570+0000", 2024-09-20T22:46:23.590 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.0679999999999999e-06 2024-09-20T22:46:23.590 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.590 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.590 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:46:23.590 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.162310+0000", 2024-09-20T22:46:23.590 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00074005999999999996 2024-09-20T22:46:23.590 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.590 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.590 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:46:23.590 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.162349+0000", 2024-09-20T22:46:23.590 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.8778e-05 2024-09-20T22:46:23.591 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.591 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.591 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:46:23.591 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.162375+0000", 2024-09-20T22:46:23.592 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.6089e-05 2024-09-20T22:46:23.592 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.592 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.592 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:46:23.592 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.162434+0000", 2024-09-20T22:46:23.592 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.8764999999999997e-05 2024-09-20T22:46:23.592 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.592 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.592 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-09-20T22:46:23.592 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.162443+0000", 2024-09-20T22:46:23.592 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.054e-06 2024-09-20T22:46:23.592 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.592 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.592 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:46:23.593 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.143416+0000", 2024-09-20T22:46:23.593 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.98097291600000003 2024-09-20T22:46:23.593 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.593 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.593 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:46:23.593 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.143522+0000", 2024-09-20T22:46:23.593 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00010626 2024-09-20T22:46:23.593 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.593 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.593 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 1,2", 2024-09-20T22:46:23.593 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.143654+0000", 2024-09-20T22:46:23.593 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013218799999999999 2024-09-20T22:46:23.593 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.593 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.594 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:46:23.594 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.148086+0000", 2024-09-20T22:46:23.594 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0044314189999999998 2024-09-20T22:46:23.594 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.594 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.594 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:46:23.594 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.153546+0000", 2024-09-20T22:46:23.594 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0054599920000000003 2024-09-20T22:46:23.594 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.594 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.594 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:46:23.594 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.154317+0000", 2024-09-20T22:46:23.594 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00077125999999999996 2024-09-20T22:46:23.595 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.595 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.595 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:46:23.595 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.154380+0000", 2024-09-20T22:46:23.595 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.2526000000000004e-05 2024-09-20T22:46:23.595 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.595 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.595 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:46:23.595 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.154432+0000", 2024-09-20T22:46:23.595 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.2133999999999999e-05 2024-09-20T22:46:23.595 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:46:23.595 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:46:23.595 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:46:23.595 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.596 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.596 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:664 7.12 7:4fa4b2b3:::smithi08477203-21:head [write 2062560~563688] snapc 17=[17,16,15,11,d] ondisk+write+known_if_redirected+supports_pool_eio e137)", 2024-09-20T22:46:23.596 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:43:36.167947+0000", 2024-09-20T22:46:23.596 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 167.40545783499999, 2024-09-20T22:46:23.596 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.98665172599999995, 2024-09-20T22:46:23.596 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:46:23.596 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:46:23.596 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:46:23.596 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:46:23.596 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:46:23.596 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:46:23.596 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 664 2024-09-20T22:46:23.596 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.596 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:46:23.597 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.597 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:46:23.597 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.167947+0000", 2024-09-20T22:46:23.597 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:46:23.597 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.597 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.597 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:46:23.597 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.167947+0000", 2024-09-20T22:46:23.597 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:46:23.597 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.597 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.597 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:46:23.597 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.167948+0000", 2024-09-20T22:46:23.598 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.187e-06 2024-09-20T22:46:23.598 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.598 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.598 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:46:23.598 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.168193+0000", 2024-09-20T22:46:23.598 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00024496199999999999 2024-09-20T22:46:23.598 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.598 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.598 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:46:23.598 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.168203+0000", 2024-09-20T22:46:23.598 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0543999999999999e-05 2024-09-20T22:46:23.598 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.598 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.599 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:46:23.599 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.168209+0000", 2024-09-20T22:46:23.599 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.2839999999999997e-06 2024-09-20T22:46:23.599 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.599 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.599 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:46:23.599 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.168252+0000", 2024-09-20T22:46:23.599 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.3115000000000003e-05 2024-09-20T22:46:23.599 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.599 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.599 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-09-20T22:46:23.599 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.168257+0000", 2024-09-20T22:46:23.599 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.6860000000000003e-06 2024-09-20T22:46:23.599 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.600 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.600 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:46:23.600 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.144795+0000", 2024-09-20T22:46:23.600 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.97653758099999999 2024-09-20T22:46:23.600 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.600 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.600 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:46:23.600 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.144865+0000", 2024-09-20T22:46:23.600 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.9641000000000006e-05 2024-09-20T22:46:23.600 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.600 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.600 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 1,2", 2024-09-20T22:46:23.600 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.144986+0000", 2024-09-20T22:46:23.600 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000121248 2024-09-20T22:46:23.600 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.600 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.601 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:46:23.601 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.148091+0000", 2024-09-20T22:46:23.601 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0031050399999999999 2024-09-20T22:46:23.601 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.601 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.601 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:46:23.601 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.153603+0000", 2024-09-20T22:46:23.601 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0055118479999999997 2024-09-20T22:46:23.601 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.601 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.601 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:46:23.601 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.154508+0000", 2024-09-20T22:46:23.601 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00090530600000000004 2024-09-20T22:46:23.601 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.602 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.602 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:46:23.602 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.154560+0000", 2024-09-20T22:46:23.602 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.1474000000000002e-05 2024-09-20T22:46:23.602 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.602 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.602 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:46:23.602 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.154598+0000", 2024-09-20T22:46:23.602 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.8810000000000003e-05 2024-09-20T22:46:23.602 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:46:23.602 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:46:23.602 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:46:23.602 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.602 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.603 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:665 7.12 7:4fa4b2b3:::smithi08477203-21:head [write 3182305~643664] snapc 17=[17,16,15,11,d] ondisk+write+known_if_redirected+supports_pool_eio e137)", 2024-09-20T22:46:23.603 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:43:36.174520+0000", 2024-09-20T22:46:23.603 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 167.398884745, 2024-09-20T22:46:23.603 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.98020083400000002, 2024-09-20T22:46:23.603 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:46:23.603 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:46:23.603 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:46:23.603 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:46:23.603 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:46:23.603 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:46:23.603 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 665 2024-09-20T22:46:23.603 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.603 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:46:23.603 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.604 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:46:23.604 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174520+0000", 2024-09-20T22:46:23.604 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:46:23.604 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.604 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.604 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:46:23.604 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174520+0000", 2024-09-20T22:46:23.604 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:46:23.604 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.604 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.604 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:46:23.604 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174521+0000", 2024-09-20T22:46:23.604 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.234e-06 2024-09-20T22:46:23.604 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.605 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.605 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:46:23.605 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174816+0000", 2024-09-20T22:46:23.605 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000294682 2024-09-20T22:46:23.605 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.605 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.605 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:46:23.605 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174827+0000", 2024-09-20T22:46:23.605 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1025e-05 2024-09-20T22:46:23.605 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.605 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.605 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:46:23.605 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174832+0000", 2024-09-20T22:46:23.605 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.3990000000000003e-06 2024-09-20T22:46:23.605 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.605 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.606 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:46:23.606 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174868+0000", 2024-09-20T22:46:23.606 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.6225e-05 2024-09-20T22:46:23.606 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.606 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.606 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-09-20T22:46:23.606 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174873+0000", 2024-09-20T22:46:23.606 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.848e-06 2024-09-20T22:46:23.606 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.606 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.606 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:46:23.606 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.145844+0000", 2024-09-20T22:46:23.606 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.97097048900000005 2024-09-20T22:46:23.607 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.607 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.607 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:46:23.607 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.145917+0000", 2024-09-20T22:46:23.607 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.3248000000000001e-05 2024-09-20T22:46:23.607 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.607 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.607 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 1,2", 2024-09-20T22:46:23.607 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.146028+0000", 2024-09-20T22:46:23.607 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000110773 2024-09-20T22:46:23.607 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.607 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.608 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:46:23.608 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.148095+0000", 2024-09-20T22:46:23.608 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0020678179999999999 2024-09-20T22:46:23.608 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.608 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.608 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:46:23.608 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.153967+0000", 2024-09-20T22:46:23.608 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0058717350000000003 2024-09-20T22:46:23.608 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.608 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.608 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:46:23.609 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.154633+0000", 2024-09-20T22:46:23.609 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00066600600000000002 2024-09-20T22:46:23.609 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.609 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.609 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:46:23.609 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.154683+0000", 2024-09-20T22:46:23.609 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.9872e-05 2024-09-20T22:46:23.609 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.609 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.609 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:46:23.610 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.154721+0000", 2024-09-20T22:46:23.610 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.748e-05 2024-09-20T22:46:23.610 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:46:23.610 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:46:23.610 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:46:23.610 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.610 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.610 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:666 7.12 7:4fa4b2b3:::smithi08477203-21:head [setxattr _header (57) in=7b,truncate 3865577] snapc 17=[17,16,15,11,d] ondisk+write+known_if_redirected+supports_pool_eio e137)", 2024-09-20T22:46:23.610 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:43:36.174844+0000", 2024-09-20T22:46:23.610 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 167.398560546, 2024-09-20T22:46:23.610 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.98002428600000002, 2024-09-20T22:46:23.610 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:46:23.610 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:46:23.610 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:46:23.611 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:46:23.611 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:46:23.611 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:46:23.611 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 666 2024-09-20T22:46:23.611 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.611 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:46:23.611 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.611 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:46:23.611 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174844+0000", 2024-09-20T22:46:23.611 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:46:23.611 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.611 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.611 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:46:23.611 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174844+0000", 2024-09-20T22:46:23.611 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:46:23.612 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.612 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.612 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:46:23.612 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174845+0000", 2024-09-20T22:46:23.612 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.1999999999999999e-07 2024-09-20T22:46:23.612 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.612 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.612 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:46:23.612 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174853+0000", 2024-09-20T22:46:23.612 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.1950000000000003e-06 2024-09-20T22:46:23.612 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.612 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.612 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:46:23.612 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174858+0000", 2024-09-20T22:46:23.613 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.5650000000000001e-06 2024-09-20T22:46:23.613 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.613 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.613 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:46:23.613 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174863+0000", 2024-09-20T22:46:23.613 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.1649999999999997e-06 2024-09-20T22:46:23.613 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.614 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.614 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:46:23.614 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174915+0000", 2024-09-20T22:46:23.614 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.1045999999999999e-05 2024-09-20T22:46:23.614 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.614 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.614 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-09-20T22:46:23.614 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174921+0000", 2024-09-20T22:46:23.615 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.404e-06 2024-09-20T22:46:23.615 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.615 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.615 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:46:23.615 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.146528+0000", 2024-09-20T22:46:23.615 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.97160681199999999 2024-09-20T22:46:23.615 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.615 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.615 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:46:23.615 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.146881+0000", 2024-09-20T22:46:23.616 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00035348200000000001 2024-09-20T22:46:23.616 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.616 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.616 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 1,2", 2024-09-20T22:46:23.616 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.147011+0000", 2024-09-20T22:46:23.616 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013018900000000001 2024-09-20T22:46:23.616 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.616 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.617 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:46:23.617 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.148100+0000", 2024-09-20T22:46:23.617 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010881580000000001 2024-09-20T22:46:23.617 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.617 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.617 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:46:23.617 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.154278+0000", 2024-09-20T22:46:23.617 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0061781730000000003 2024-09-20T22:46:23.617 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.617 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.617 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:46:23.617 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.154764+0000", 2024-09-20T22:46:23.617 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00048589199999999998 2024-09-20T22:46:23.617 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.617 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.618 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:46:23.618 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.154815+0000", 2024-09-20T22:46:23.618 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.1428000000000003e-05 2024-09-20T22:46:23.618 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.618 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.618 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:46:23.618 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.154868+0000", 2024-09-20T22:46:23.618 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.3156999999999999e-05 2024-09-20T22:46:23.618 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:46:23.618 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:46:23.618 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:46:23.618 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.618 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.618 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:667 7.12 7:4fa4b2b3:::smithi08477203-21:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e137)", 2024-09-20T22:46:23.618 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:43:36.174872+0000", 2024-09-20T22:46:23.619 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 167.39853285300001, 2024-09-20T22:46:23.619 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.98024790299999998, 2024-09-20T22:46:23.619 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:46:23.619 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:46:23.619 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-20T22:46:23.619 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:46:23.619 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:46:23.619 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:46:23.619 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 667 2024-09-20T22:46:23.619 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.619 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:46:23.619 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.619 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:46:23.619 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174872+0000", 2024-09-20T22:46:23.619 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:46:23.620 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.620 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.620 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:46:23.620 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174872+0000", 2024-09-20T22:46:23.620 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:46:23.620 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.620 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.620 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:46:23.620 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174872+0000", 2024-09-20T22:46:23.620 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.2199999999999999e-07 2024-09-20T22:46:23.620 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.620 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.620 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:46:23.620 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174874+0000", 2024-09-20T22:46:23.620 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.187e-06 2024-09-20T22:46:23.621 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.621 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.621 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:46:23.621 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174881+0000", 2024-09-20T22:46:23.621 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.674e-06 2024-09-20T22:46:23.621 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.621 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.621 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:46:23.621 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174886+0000", 2024-09-20T22:46:23.621 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.2240000000000003e-06 2024-09-20T22:46:23.621 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.621 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.621 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:46:23.622 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174936+0000", 2024-09-20T22:46:23.622 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.9824999999999998e-05 2024-09-20T22:46:23.622 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.622 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.622 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-09-20T22:46:23.622 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174940+0000", 2024-09-20T22:46:23.622 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.5700000000000001e-06 2024-09-20T22:46:23.622 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.622 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.622 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:46:23.622 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.147523+0000", 2024-09-20T22:46:23.622 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.97258315399999995 2024-09-20T22:46:23.622 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.622 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.623 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for rw locks", 2024-09-20T22:46:23.623 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.147591+0000", 2024-09-20T22:46:23.623 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.7878000000000003e-05 2024-09-20T22:46:23.623 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.623 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.623 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:46:23.623 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.154925+0000", 2024-09-20T22:46:23.623 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0073344430000000004 2024-09-20T22:46:23.623 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.623 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.623 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:46:23.623 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.155020+0000", 2024-09-20T22:46:23.623 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.4767999999999998e-05 2024-09-20T22:46:23.623 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.624 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.624 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:46:23.624 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.155120+0000", 2024-09-20T22:46:23.624 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.9758000000000003e-05 2024-09-20T22:46:23.624 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:46:23.624 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:46:23.624 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:46:23.624 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.624 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.625 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.14478.0:1259 5.3 5:c609908c:::notify.5:head [watch reconnect cookie 94843898679808 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e169)", 2024-09-20T22:46:23.625 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:44:08.230723+0000", 2024-09-20T22:46:23.625 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 135.342681548, 2024-09-20T22:46:23.625 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.94983577399999997, 2024-09-20T22:46:23.625 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:46:23.625 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:46:23.625 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-20T22:46:23.625 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:46:23.625 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.14478", 2024-09-20T22:46:23.626 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/2994273271", 2024-09-20T22:46:23.626 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1259 2024-09-20T22:46:23.626 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.626 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:46:23.626 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.626 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:46:23.626 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:08.230723+0000", 2024-09-20T22:46:23.626 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:46:23.627 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.627 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.627 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:46:23.627 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:08.230723+0000", 2024-09-20T22:46:23.627 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:46:23.627 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.627 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.627 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:46:23.628 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:08.230724+0000", 2024-09-20T22:46:23.628 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.077e-06 2024-09-20T22:46:23.628 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.628 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.628 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:46:23.628 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:08.230729+0000", 2024-09-20T22:46:23.628 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.9930000000000003e-06 2024-09-20T22:46:23.628 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.628 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.629 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:46:23.629 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:08.230741+0000", 2024-09-20T22:46:23.629 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1897e-05 2024-09-20T22:46:23.629 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.629 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.629 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:46:23.629 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:08.230747+0000", 2024-09-20T22:46:23.629 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.9830000000000001e-06 2024-09-20T22:46:23.629 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.630 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.630 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:46:23.630 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:08.230802+0000", 2024-09-20T22:46:23.630 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.4657000000000002e-05 2024-09-20T22:46:23.630 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.630 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.630 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-09-20T22:46:23.630 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:08.230811+0000", 2024-09-20T22:46:23.631 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.1160000000000005e-06 2024-09-20T22:46:23.631 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.631 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.631 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:46:23.631 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:09.171968+0000", 2024-09-20T22:46:23.631 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.94115747299999997 2024-09-20T22:46:23.631 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.631 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.631 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for missing object", 2024-09-20T22:46:23.632 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:09.172098+0000", 2024-09-20T22:46:23.632 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00012937499999999999 2024-09-20T22:46:23.632 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.632 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.632 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:46:23.632 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:09.174248+0000", 2024-09-20T22:46:23.632 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0021508959999999998 2024-09-20T22:46:23.632 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.633 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.633 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for rw locks", 2024-09-20T22:46:23.633 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:09.174341+0000", 2024-09-20T22:46:23.633 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.2906999999999997e-05 2024-09-20T22:46:23.633 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.633 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.633 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:46:23.633 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:09.175083+0000", 2024-09-20T22:46:23.633 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00074120100000000003 2024-09-20T22:46:23.634 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.634 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.634 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:46:23.634 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:09.175149+0000", 2024-09-20T22:46:23.634 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.6877000000000004e-05 2024-09-20T22:46:23.634 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.634 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.634 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:46:23.634 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:09.180559+0000", 2024-09-20T22:46:23.635 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0054093220000000003 2024-09-20T22:46:23.635 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:46:23.635 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:46:23.635 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:46:23.635 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.635 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.635 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:1417 7.1 7:84a866f5:::smithi08477203-27:head [append 0~0] snapc 2f=[2f] ondisk+write+known_if_redirected+supports_pool_eio e189)", 2024-09-20T22:46:23.635 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:44:28.137465+0000", 2024-09-20T22:46:23.635 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 115.435939189, 2024-09-20T22:46:23.636 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.020090858999999999, 2024-09-20T22:46:23.636 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:46:23.636 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:46:23.636 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:46:23.636 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:46:23.636 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:46:23.636 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:46:23.636 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1417 2024-09-20T22:46:23.636 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.637 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:46:23.637 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.637 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:46:23.637 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.137465+0000", 2024-09-20T22:46:23.637 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:46:23.637 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.637 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.637 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:46:23.637 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.137465+0000", 2024-09-20T22:46:23.638 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:46:23.638 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.638 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.638 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:46:23.638 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.137468+0000", 2024-09-20T22:46:23.638 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.7259999999999998e-06 2024-09-20T22:46:23.638 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.638 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.638 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:46:23.638 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.137481+0000", 2024-09-20T22:46:23.639 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2937e-05 2024-09-20T22:46:23.639 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.639 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.639 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:46:23.639 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.137528+0000", 2024-09-20T22:46:23.639 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.7011000000000001e-05 2024-09-20T22:46:23.639 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.639 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.639 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:46:23.639 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.137542+0000", 2024-09-20T22:46:23.639 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3529e-05 2024-09-20T22:46:23.639 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.639 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.640 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:46:23.640 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.137636+0000", 2024-09-20T22:46:23.640 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.4932999999999994e-05 2024-09-20T22:46:23.640 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.640 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.640 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op must wait for map", 2024-09-20T22:46:23.640 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.137645+0000", 2024-09-20T22:46:23.640 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:46:23.642 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.3180000000000007e-06 2024-09-20T22:46:23.642 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.642 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.642 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:46:23.642 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.143542+0000", 2024-09-20T22:46:23.642 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0058974459999999998 2024-09-20T22:46:23.642 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.642 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.642 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:46:23.642 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.143647+0000", 2024-09-20T22:46:23.643 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000105028 2024-09-20T22:46:23.643 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.643 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.643 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 1,6", 2024-09-20T22:46:23.643 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.143835+0000", 2024-09-20T22:46:23.643 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00018802599999999999 2024-09-20T22:46:23.643 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.643 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.643 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:46:23.643 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.155596+0000", 2024-09-20T22:46:23.643 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011760708 2024-09-20T22:46:23.643 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.643 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.644 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:46:23.644 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.155674+0000", 2024-09-20T22:46:23.644 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.7883e-05 2024-09-20T22:46:23.644 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.644 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.644 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:46:23.644 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.157440+0000", 2024-09-20T22:46:23.644 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001766014 2024-09-20T22:46:23.644 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.644 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.644 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:46:23.644 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.157505+0000", 2024-09-20T22:46:23.644 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.5252999999999993e-05 2024-09-20T22:46:23.644 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.644 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.645 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:46:23.645 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.157556+0000", 2024-09-20T22:46:23.645 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.1047000000000001e-05 2024-09-20T22:46:23.645 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:46:23.645 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:46:23.645 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:46:23.645 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.645 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.645 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:1418 7.1 7:84a866f5:::smithi08477203-27:head [rollback 2f] snapc 2f=[2f] ondisk+write+known_if_redirected+supports_pool_eio e189)", 2024-09-20T22:46:23.645 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:44:28.137587+0000", 2024-09-20T22:46:23.645 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 115.435817351, 2024-09-20T22:46:23.645 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.020550452, 2024-09-20T22:46:23.645 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:46:23.645 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:46:23.645 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:46:23.646 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:46:23.646 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:46:23.646 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:46:23.646 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1418 2024-09-20T22:46:23.646 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.646 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:46:23.646 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.646 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:46:23.646 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.137587+0000", 2024-09-20T22:46:23.646 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:46:23.646 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.646 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.646 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:46:23.646 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.137587+0000", 2024-09-20T22:46:23.646 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:46:23.647 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.647 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.647 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:46:23.647 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.137588+0000", 2024-09-20T22:46:23.647 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.4700000000000001e-07 2024-09-20T22:46:23.647 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.647 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.647 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:46:23.647 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.137591+0000", 2024-09-20T22:46:23.647 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.0000000000000001e-06 2024-09-20T22:46:23.647 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.647 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.647 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:46:23.647 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.137681+0000", 2024-09-20T22:46:23.648 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.0486000000000001e-05 2024-09-20T22:46:23.648 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.648 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.648 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:46:23.648 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.138018+0000", 2024-09-20T22:46:23.648 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00033709899999999999 2024-09-20T22:46:23.648 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.648 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.648 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:46:23.648 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.138109+0000", 2024-09-20T22:46:23.648 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.0538999999999995e-05 2024-09-20T22:46:23.648 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.648 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.648 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting_for_map not empty", 2024-09-20T22:46:23.649 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.138114+0000", 2024-09-20T22:46:23.649 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.6590000000000004e-06 2024-09-20T22:46:23.649 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.649 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.649 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:46:23.649 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.144398+0000", 2024-09-20T22:46:23.649 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0062832599999999997 2024-09-20T22:46:23.649 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.649 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.649 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:46:23.649 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.144483+0000", 2024-09-20T22:46:23.649 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.4772999999999996e-05 2024-09-20T22:46:23.649 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.649 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.650 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 1,6", 2024-09-20T22:46:23.650 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.144661+0000", 2024-09-20T22:46:23.650 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000178736 2024-09-20T22:46:23.650 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.650 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.650 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:46:23.650 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.155602+0000", 2024-09-20T22:46:23.650 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010940302000000001 2024-09-20T22:46:23.650 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.650 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.650 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:46:23.650 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.156162+0000", 2024-09-20T22:46:23.650 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00056014599999999997 2024-09-20T22:46:23.650 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.650 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.651 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:46:23.651 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.158054+0000", 2024-09-20T22:46:23.651 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00189223 2024-09-20T22:46:23.651 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.651 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.651 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:46:23.651 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.158100+0000", 2024-09-20T22:46:23.651 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.6279999999999997e-05 2024-09-20T22:46:23.651 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.651 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.651 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:46:23.651 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.158138+0000", 2024-09-20T22:46:23.651 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.7394999999999997e-05 2024-09-20T22:46:23.651 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:46:23.651 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:46:23.652 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:46:23.652 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.652 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.652 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:1419 7.1 7:84a866f5:::smithi08477203-27:head [append 0~0] snapc 2f=[2f] ondisk+write+known_if_redirected+supports_pool_eio e189)", 2024-09-20T22:46:23.652 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:44:28.138055+0000", 2024-09-20T22:46:23.652 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 115.435349909, 2024-09-20T22:46:23.652 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.030290338999999999, 2024-09-20T22:46:23.652 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:46:23.652 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:46:23.652 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:46:23.652 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:46:23.652 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:46:23.652 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:46:23.652 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1419 2024-09-20T22:46:23.653 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.653 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:46:23.653 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.653 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:46:23.653 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.138055+0000", 2024-09-20T22:46:23.653 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:46:23.653 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.653 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.653 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:46:23.653 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.138055+0000", 2024-09-20T22:46:23.653 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:46:23.653 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.653 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.653 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:46:23.654 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.138056+0000", 2024-09-20T22:46:23.654 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.13e-06 2024-09-20T22:46:23.654 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.654 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.654 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:46:23.654 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.138060+0000", 2024-09-20T22:46:23.654 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.0269999999999997e-06 2024-09-20T22:46:23.654 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.654 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.654 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:46:23.654 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.138070+0000", 2024-09-20T22:46:23.654 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.961e-06 2024-09-20T22:46:23.654 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.655 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.655 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:46:23.655 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.138076+0000", 2024-09-20T22:46:23.655 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.1800000000000001e-06 2024-09-20T22:46:23.655 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.655 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.655 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:46:23.655 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.138137+0000", 2024-09-20T22:46:23.655 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.1159000000000002e-05 2024-09-20T22:46:23.655 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.655 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.655 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting_for_map not empty", 2024-09-20T22:46:23.655 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.138141+0000", 2024-09-20T22:46:23.655 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.189e-06 2024-09-20T22:46:23.655 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.656 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.656 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:46:23.656 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.154982+0000", 2024-09-20T22:46:23.656 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.016840738000000001 2024-09-20T22:46:23.656 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.656 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.656 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:46:23.656 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.155089+0000", 2024-09-20T22:46:23.656 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00010653399999999999 2024-09-20T22:46:23.656 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.656 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.656 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 1,6", 2024-09-20T22:46:23.656 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.155320+0000", 2024-09-20T22:46:23.656 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00023196299999999999 2024-09-20T22:46:23.656 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.657 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.657 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:46:23.657 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.158224+0000", 2024-09-20T22:46:23.657 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00290361 2024-09-20T22:46:23.657 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.657 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.657 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:46:23.657 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.167686+0000", 2024-09-20T22:46:23.657 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0094622360000000006 2024-09-20T22:46:23.657 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.657 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.657 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:46:23.657 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.168183+0000", 2024-09-20T22:46:23.657 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00049655999999999995 2024-09-20T22:46:23.657 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.658 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.658 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:46:23.658 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.168273+0000", 2024-09-20T22:46:23.658 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.9758000000000004e-05 2024-09-20T22:46:23.658 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.658 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.658 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:46:23.658 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.168345+0000", 2024-09-20T22:46:23.658 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.2293999999999996e-05 2024-09-20T22:46:23.658 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:46:23.658 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:46:23.658 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:46:23.658 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.658 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.658 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:2347 7.a 7:5122292a:::smithi08477203-33:head [sparse-read 0~2590699,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e260)", 2024-09-20T22:46:23.659 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:45:39.196440+0000", 2024-09-20T22:46:23.659 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 44.376964780999998, 2024-09-20T22:46:23.659 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.026701163, 2024-09-20T22:46:23.659 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:46:23.659 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:46:23.659 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-20T22:46:23.659 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:46:23.659 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:46:23.659 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:46:23.659 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 2347 2024-09-20T22:46:23.659 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.659 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:46:23.659 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.659 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:46:23.660 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.196440+0000", 2024-09-20T22:46:23.660 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:46:23.660 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.660 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.660 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:46:23.660 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.196440+0000", 2024-09-20T22:46:23.660 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:46:23.660 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.660 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.660 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:46:23.660 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.196441+0000", 2024-09-20T22:46:23.660 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.714e-06 2024-09-20T22:46:23.660 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.660 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.661 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:46:23.661 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.196449+0000", 2024-09-20T22:46:23.661 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.8730000000000007e-06 2024-09-20T22:46:23.661 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.661 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.661 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:46:23.661 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.196462+0000", 2024-09-20T22:46:23.661 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2649e-05 2024-09-20T22:46:23.661 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.661 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.661 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:46:23.661 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.196470+0000", 2024-09-20T22:46:23.661 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.4989999999999994e-06 2024-09-20T22:46:23.661 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.662 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.662 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:46:23.662 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.212172+0000", 2024-09-20T22:46:23.662 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.015701335 2024-09-20T22:46:23.662 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.662 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.662 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:46:23.662 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.212270+0000", 2024-09-20T22:46:23.662 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.8257999999999993e-05 2024-09-20T22:46:23.662 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.662 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.662 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:46:23.662 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.223141+0000", 2024-09-20T22:46:23.662 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010870835000000001 2024-09-20T22:46:23.663 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:46:23.663 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:46:23.663 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:46:23.663 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.663 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.663 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:2348 7.a 7:5122292a:::smithi08477203-33:head [read 0~3848803,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e260)", 2024-09-20T22:46:23.663 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:45:39.196485+0000", 2024-09-20T22:46:23.663 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 44.376919307999998, 2024-09-20T22:46:23.663 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0284703, 2024-09-20T22:46:23.663 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:46:23.663 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:46:23.663 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-20T22:46:23.663 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:46:23.663 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:46:23.664 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:46:23.664 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 2348 2024-09-20T22:46:23.664 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.664 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:46:23.664 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.664 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:46:23.664 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.196485+0000", 2024-09-20T22:46:23.664 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:46:23.664 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.664 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.664 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:46:23.664 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.196485+0000", 2024-09-20T22:46:23.664 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:46:23.664 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.665 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.665 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:46:23.665 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.196486+0000", 2024-09-20T22:46:23.665 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.9599999999999999e-07 2024-09-20T22:46:23.665 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.665 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.665 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:46:23.665 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.196489+0000", 2024-09-20T22:46:23.665 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.5470000000000001e-06 2024-09-20T22:46:23.665 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.665 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.665 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:46:23.665 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.196496+0000", 2024-09-20T22:46:23.665 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.3570000000000003e-06 2024-09-20T22:46:23.666 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.666 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.666 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:46:23.666 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.196501+0000", 2024-09-20T22:46:23.666 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.6239999999999999e-06 2024-09-20T22:46:23.666 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.666 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.666 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:46:23.666 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.223176+0000", 2024-09-20T22:46:23.666 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.026675285999999999 2024-09-20T22:46:23.666 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.666 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.666 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:46:23.666 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.223274+0000", 2024-09-20T22:46:23.666 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.7009999999999994e-05 2024-09-20T22:46:23.667 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.667 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.667 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:46:23.667 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.224955+0000", 2024-09-20T22:46:23.667 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00168198 2024-09-20T22:46:23.667 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:46:23.667 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:46:23.667 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:46:23.667 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.667 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.667 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:2349 7.a 7:5122292a:::smithi08477203-33:head [sparse-read 0~2590699] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e260)", 2024-09-20T22:46:23.667 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:45:39.196508+0000", 2024-09-20T22:46:23.667 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 44.376896021999997, 2024-09-20T22:46:23.667 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.028731293000000001, 2024-09-20T22:46:23.668 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:46:23.668 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:46:23.668 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-20T22:46:23.668 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:46:23.668 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:46:23.668 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:46:23.668 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 2349 2024-09-20T22:46:23.668 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.668 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:46:23.668 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.668 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:46:23.668 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.196508+0000", 2024-09-20T22:46:23.668 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:46:23.668 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.668 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.669 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:46:23.669 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.196508+0000", 2024-09-20T22:46:23.669 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:46:23.669 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.669 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.669 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:46:23.669 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.196509+0000", 2024-09-20T22:46:23.669 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.08e-07 2024-09-20T22:46:23.669 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.669 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.669 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:46:23.669 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.196511+0000", 2024-09-20T22:46:23.670 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.142e-06 2024-09-20T22:46:23.670 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.670 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.670 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:46:23.670 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.196517+0000", 2024-09-20T22:46:23.670 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.6330000000000001e-06 2024-09-20T22:46:23.670 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.670 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.670 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:46:23.670 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.196522+0000", 2024-09-20T22:46:23.670 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.3530000000000002e-06 2024-09-20T22:46:23.670 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.670 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.670 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:46:23.671 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.224990+0000", 2024-09-20T22:46:23.671 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.028467864999999998 2024-09-20T22:46:23.671 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.671 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.671 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:46:23.671 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.225088+0000", 2024-09-20T22:46:23.671 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.8281000000000003e-05 2024-09-20T22:46:23.671 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.671 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.671 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:46:23.671 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.225240+0000", 2024-09-20T22:46:23.671 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00015161100000000001 2024-09-20T22:46:23.671 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:46:23.672 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:46:23.672 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:46:23.672 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.672 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.672 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(osd.6.0:727 7.c e260/94 7:318c7512:::smithi08477203-44:4c v 260'428, mlcod=258'424)", 2024-09-20T22:46:23.672 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:45:39.202413+0000", 2024-09-20T22:46:23.672 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 44.370991482999997, 2024-09-20T22:46:23.672 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.022852897, 2024-09-20T22:46:23.672 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:46:23.672 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:46:23.672 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:46:23.672 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:46:23.672 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.672 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:46:23.673 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.202413+0000", 2024-09-20T22:46:23.673 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:46:23.673 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.673 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.673 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:46:23.673 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.202413+0000", 2024-09-20T22:46:23.673 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:46:23.673 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.673 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.673 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:46:23.673 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.202414+0000", 2024-09-20T22:46:23.673 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.9299999999999997e-07 2024-09-20T22:46:23.673 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.673 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.674 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:46:23.674 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.202421+0000", 2024-09-20T22:46:23.674 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.8109999999999997e-06 2024-09-20T22:46:23.674 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.674 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.674 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:46:23.674 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.202433+0000", 2024-09-20T22:46:23.674 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2411e-05 2024-09-20T22:46:23.674 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.674 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.674 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:46:23.674 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.202442+0000", 2024-09-20T22:46:23.674 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.7269999999999996e-06 2024-09-20T22:46:23.675 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.675 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.675 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:46:23.675 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.221764+0000", 2024-09-20T22:46:23.675 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.019322068000000001 2024-09-20T22:46:23.675 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.675 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.675 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:46:23.675 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.221786+0000", 2024-09-20T22:46:23.675 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.2167e-05 2024-09-20T22:46:23.675 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.675 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.675 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:46:23.675 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.225217+0000", 2024-09-20T22:46:23.675 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0034313709999999999 2024-09-20T22:46:23.676 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.676 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.676 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:46:23.676 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.225266+0000", 2024-09-20T22:46:23.676 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.8649000000000001e-05 2024-09-20T22:46:23.676 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:46:23.676 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:46:23.676 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:46:23.676 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.676 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.676 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(osd.6.0:728 7.c e260/94 7:31d0211f:::smithi08477203-25:51 v 260'430, mlcod=258'424)", 2024-09-20T22:46:23.676 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:45:39.202454+0000", 2024-09-20T22:46:23.676 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 44.370950620000002, 2024-09-20T22:46:23.676 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.022837705, 2024-09-20T22:46:23.676 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:46:23.677 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:46:23.677 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:46:23.677 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:46:23.677 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.677 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:46:23.677 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.202454+0000", 2024-09-20T22:46:23.677 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:46:23.677 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.677 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.677 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:46:23.677 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.202454+0000", 2024-09-20T22:46:23.677 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:46:23.677 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.677 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.678 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:46:23.678 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.202454+0000", 2024-09-20T22:46:23.678 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.9999999999999999e-07 2024-09-20T22:46:23.678 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.678 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.678 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:46:23.678 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.202464+0000", 2024-09-20T22:46:23.678 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.8179999999999999e-06 2024-09-20T22:46:23.678 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.678 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.678 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:46:23.678 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.202469+0000", 2024-09-20T22:46:23.678 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.1089999999999997e-06 2024-09-20T22:46:23.679 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.679 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.679 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:46:23.679 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.202473+0000", 2024-09-20T22:46:23.679 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.0940000000000001e-06 2024-09-20T22:46:23.679 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.679 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.679 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:46:23.679 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.222250+0000", 2024-09-20T22:46:23.679 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.019776353999999999 2024-09-20T22:46:23.679 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.679 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.679 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:46:23.679 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.222268+0000", 2024-09-20T22:46:23.679 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.8342000000000002e-05 2024-09-20T22:46:23.680 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.680 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.680 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:46:23.680 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.225268+0000", 2024-09-20T22:46:23.680 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00300038 2024-09-20T22:46:23.680 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.680 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.680 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:46:23.680 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.225292+0000", 2024-09-20T22:46:23.680 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.3308000000000001e-05 2024-09-20T22:46:23.680 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:46:23.680 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:46:23.680 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:46:23.680 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.681 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.681 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:2525 7.1 7:8422f82d:::smithi08477203-22:head [copy-from ver 174] snapc 5d=[5d,5c,5a,50,47] ondisk+write+known_if_redirected+supports_pool_eio e278)", 2024-09-20T22:46:23.681 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:45:57.170348+0000", 2024-09-20T22:46:23.681 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 26.403056508999999, 2024-09-20T22:46:23.681 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.023333486000000001, 2024-09-20T22:46:23.681 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:46:23.681 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:46:23.681 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:46:23.681 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:46:23.681 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:46:23.681 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:46:23.681 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 2525 2024-09-20T22:46:23.681 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.681 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:46:23.681 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.682 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:46:23.682 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:57.170348+0000", 2024-09-20T22:46:23.682 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:46:23.682 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.682 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.682 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:46:23.682 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:57.170348+0000", 2024-09-20T22:46:23.682 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:46:23.682 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.682 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.682 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:46:23.682 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:57.170350+0000", 2024-09-20T22:46:23.682 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.1509999999999998e-06 2024-09-20T22:46:23.683 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.683 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.683 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:46:23.683 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:57.170361+0000", 2024-09-20T22:46:23.683 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0492999999999999e-05 2024-09-20T22:46:23.683 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.683 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.683 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:46:23.683 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:57.170381+0000", 2024-09-20T22:46:23.683 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.0509999999999998e-05 2024-09-20T22:46:23.683 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.683 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.683 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:46:23.683 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:57.170393+0000", 2024-09-20T22:46:23.684 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2009e-05 2024-09-20T22:46:23.684 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.684 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.684 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:46:23.684 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:57.170465+0000", 2024-09-20T22:46:23.684 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.1927000000000004e-05 2024-09-20T22:46:23.684 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.684 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.684 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op must wait for map", 2024-09-20T22:46:23.684 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:57.170473+0000", 2024-09-20T22:46:23.684 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.1890000000000007e-06 2024-09-20T22:46:23.684 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.684 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.684 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:46:23.685 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:57.172855+0000", 2024-09-20T22:46:23.685 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002382168 2024-09-20T22:46:23.685 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.685 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.685 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:46:23.685 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:57.172967+0000", 2024-09-20T22:46:23.685 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00011108299999999999 2024-09-20T22:46:23.685 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.685 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.685 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 1,6", 2024-09-20T22:46:23.686 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:57.181533+0000", 2024-09-20T22:46:23.686 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0085667499999999997 2024-09-20T22:46:23.686 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.686 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.686 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:46:23.686 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:57.191599+0000", 2024-09-20T22:46:23.686 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010065795000000001 2024-09-20T22:46:23.686 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.686 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.686 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:46:23.686 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:57.193469+0000", 2024-09-20T22:46:23.686 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0018695199999999999 2024-09-20T22:46:23.687 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.687 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.687 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:46:23.687 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:57.193550+0000", 2024-09-20T22:46:23.687 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.1891000000000006e-05 2024-09-20T22:46:23.687 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.687 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.687 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:46:23.687 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:57.193625+0000", 2024-09-20T22:46:23.687 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.4679999999999996e-05 2024-09-20T22:46:23.687 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.687 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.688 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:46:23.688 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:57.193681+0000", 2024-09-20T22:46:23.688 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.6320000000000003e-05 2024-09-20T22:46:23.688 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:46:23.688 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:46:23.688 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:46:23.688 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.688 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.688 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:2526 7.1 7:8422f82d:::smithi08477203-22:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e278)", 2024-09-20T22:46:23.688 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:45:57.170418+0000", 2024-09-20T22:46:23.688 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 26.402986560999999, 2024-09-20T22:46:23.688 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.016999675999999998, 2024-09-20T22:46:23.688 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:46:23.688 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:46:23.689 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-20T22:46:23.689 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:46:23.689 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:46:23.689 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:46:23.689 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 2526 2024-09-20T22:46:23.689 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.689 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:46:23.689 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.689 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:46:23.689 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:57.170418+0000", 2024-09-20T22:46:23.689 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:46:23.689 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.689 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.689 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:46:23.690 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:57.170418+0000", 2024-09-20T22:46:23.690 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:46:23.690 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.690 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.690 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:46:23.690 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:57.170419+0000", 2024-09-20T22:46:23.690 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.8299999999999997e-07 2024-09-20T22:46:23.690 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.690 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.690 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:46:23.690 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:57.170422+0000", 2024-09-20T22:46:23.690 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.3100000000000001e-06 2024-09-20T22:46:23.690 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.690 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.690 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:46:23.691 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:57.170431+0000", 2024-09-20T22:46:23.691 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.6889999999999993e-06 2024-09-20T22:46:23.691 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.691 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.691 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:46:23.691 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:57.170436+0000", 2024-09-20T22:46:23.691 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.8289999999999996e-06 2024-09-20T22:46:23.691 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.691 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.691 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:46:23.691 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:57.170544+0000", 2024-09-20T22:46:23.691 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00010765200000000001 2024-09-20T22:46:23.691 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.691 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.691 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting_for_map not empty", 2024-09-20T22:46:23.692 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:57.170551+0000", 2024-09-20T22:46:23.692 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.6460000000000004e-06 2024-09-20T22:46:23.692 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.692 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.692 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:46:23.692 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:57.173174+0000", 2024-09-20T22:46:23.692 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026231399999999999 2024-09-20T22:46:23.692 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.692 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.692 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for blocked object", 2024-09-20T22:46:23.692 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:57.173219+0000", 2024-09-20T22:46:23.692 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.5225000000000002e-05 2024-09-20T22:46:23.692 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.692 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.692 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:46:23.693 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:57.187179+0000", 2024-09-20T22:46:23.693 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013959608 2024-09-20T22:46:23.693 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.693 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.693 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:46:23.693 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:57.187299+0000", 2024-09-20T22:46:23.693 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000120703 2024-09-20T22:46:23.693 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.693 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.693 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:46:23.693 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:57.187418+0000", 2024-09-20T22:46:23.693 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000118291 2024-09-20T22:46:23.693 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:46:23.693 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:46:23.694 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:46:23.694 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.694 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.694 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:2565 7.10 7:0f524219:::smithi08477203-50:head [copy-from ver 236] snapc 5e=[5e,5d,5c,50] ondisk+write+known_if_redirected+supports_pool_eio e281)", 2024-09-20T22:46:23.694 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:46:00.201569+0000", 2024-09-20T22:46:23.694 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 23.371835547, 2024-09-20T22:46:23.694 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.018538422999999998, 2024-09-20T22:46:23.694 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:46:23.694 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:46:23.694 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:46:23.694 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:46:23.694 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:46:23.694 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:46:23.694 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 2565 2024-09-20T22:46:23.695 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.695 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:46:23.695 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.695 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:46:23.695 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:00.201569+0000", 2024-09-20T22:46:23.695 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:46:23.695 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.695 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.695 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:46:23.695 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:00.201569+0000", 2024-09-20T22:46:23.695 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:46:23.695 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.695 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.695 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:46:23.696 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:00.201570+0000", 2024-09-20T22:46:23.696 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1829999999999999e-06 2024-09-20T22:46:23.696 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.696 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.696 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:46:23.696 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:00.201576+0000", 2024-09-20T22:46:23.696 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.2079999999999997e-06 2024-09-20T22:46:23.696 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.696 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.696 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:46:23.696 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:00.201587+0000", 2024-09-20T22:46:23.696 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0601999999999999e-05 2024-09-20T22:46:23.696 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.696 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.696 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:46:23.697 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:00.201613+0000", 2024-09-20T22:46:23.697 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.5919e-05 2024-09-20T22:46:23.697 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.697 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.697 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:46:23.697 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:00.201699+0000", 2024-09-20T22:46:23.697 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.5796000000000003e-05 2024-09-20T22:46:23.697 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.697 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.697 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:46:23.697 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:00.201819+0000", 2024-09-20T22:46:23.697 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000120166 2024-09-20T22:46:23.697 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.697 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.697 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 1,2", 2024-09-20T22:46:23.698 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:00.209541+0000", 2024-09-20T22:46:23.698 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0077224670000000002 2024-09-20T22:46:23.698 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.698 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.698 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:46:23.698 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:00.219712+0000", 2024-09-20T22:46:23.698 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010170937999999999 2024-09-20T22:46:23.698 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.698 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.698 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:46:23.698 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:00.219805+0000", 2024-09-20T22:46:23.698 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.2285000000000002e-05 2024-09-20T22:46:23.698 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.698 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.698 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:46:23.699 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:00.219977+0000", 2024-09-20T22:46:23.699 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000172702 2024-09-20T22:46:23.699 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.699 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.699 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:46:23.699 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:00.220045+0000", 2024-09-20T22:46:23.699 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.7812999999999996e-05 2024-09-20T22:46:23.699 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.699 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.699 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:46:23.699 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:00.220107+0000", 2024-09-20T22:46:23.699 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.2343999999999998e-05 2024-09-20T22:46:23.699 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:46:23.699 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:46:23.699 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:46:23.700 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.700 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.700 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:2853 7.10 7:0f524219:::smithi08477203-50:head [copy-from ver 114] snapc 68=[68,67] ondisk+write+known_if_redirected+supports_pool_eio e303)", 2024-09-20T22:46:23.700 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:46:23.285507+0000", 2024-09-20T22:46:23.700 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 0.28789777999999999, 2024-09-20T22:46:23.700 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.023440631, 2024-09-20T22:46:23.700 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:46:23.700 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:46:23.700 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:46:23.700 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:46:23.700 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:46:23.700 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:46:23.700 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 2853 2024-09-20T22:46:23.700 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.700 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:46:23.700 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.701 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:46:23.701 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.285507+0000", 2024-09-20T22:46:23.701 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:46:23.701 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.701 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.701 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:46:23.701 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.285507+0000", 2024-09-20T22:46:23.701 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:46:23.701 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.701 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.701 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:46:23.701 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.285508+0000", 2024-09-20T22:46:23.701 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.491e-06 2024-09-20T22:46:23.701 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.701 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.702 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:46:23.702 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.285514+0000", 2024-09-20T22:46:23.702 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.502e-06 2024-09-20T22:46:23.702 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.702 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.702 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:46:23.702 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.285526+0000", 2024-09-20T22:46:23.702 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2639e-05 2024-09-20T22:46:23.702 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.702 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.702 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:46:23.702 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.285535+0000", 2024-09-20T22:46:23.702 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.4689999999999997e-06 2024-09-20T22:46:23.702 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.703 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.703 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:46:23.703 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.285591+0000", 2024-09-20T22:46:23.703 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.6407999999999999e-05 2024-09-20T22:46:23.703 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.703 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.703 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:46:23.703 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.285681+0000", 2024-09-20T22:46:23.703 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.9845000000000005e-05 2024-09-20T22:46:23.703 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.703 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.703 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 1,2", 2024-09-20T22:46:23.703 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.293719+0000", 2024-09-20T22:46:23.703 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0080383950000000003 2024-09-20T22:46:23.704 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.704 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.704 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:46:23.704 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.307398+0000", 2024-09-20T22:46:23.704 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013678074 2024-09-20T22:46:23.704 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.704 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.704 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:46:23.704 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.308451+0000", 2024-09-20T22:46:23.704 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001052975 2024-09-20T22:46:23.704 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.704 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.704 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:46:23.704 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.308811+0000", 2024-09-20T22:46:23.704 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00036041399999999998 2024-09-20T22:46:23.705 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.705 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.705 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:46:23.705 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.308878+0000", 2024-09-20T22:46:23.705 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.7114999999999997e-05 2024-09-20T22:46:23.705 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:46:23.705 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:46:23.705 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:46:23.705 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.308947+0000", 2024-09-20T22:46:23.705 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.9303999999999999e-05 2024-09-20T22:46:23.705 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:46:23.705 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:46:23.705 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:46:23.705 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:46:23.706 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:46:23.706 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-20T22:46:23.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:23.707 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:23.707 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:46:23.808 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:46:23.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:23.830 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:23.830 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:46:23.870 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:46:23.931 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:46:23.952 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:23.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:23.953 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:46:24.053 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:46:24.080 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:24.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:24.081 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:46:24.181 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:46:24.202 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:24.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:24.203 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:46:24.218 INFO:tasks.rados.rados.0.smithi084.stdout:1105: done (4 left) 2024-09-20T22:46:24.219 INFO:tasks.rados.rados.0.smithi084.stdout:1106: done (3 left) 2024-09-20T22:46:24.219 INFO:tasks.rados.rados.0.smithi084.stdout:1107: done (2 left) 2024-09-20T22:46:24.219 INFO:tasks.rados.rados.0.smithi084.stdout:1108: done (1 left) 2024-09-20T22:46:24.219 INFO:tasks.rados.rados.0.smithi084.stdout:1109: done (0 left) 2024-09-20T22:46:24.219 INFO:tasks.rados.rados.0.smithi084.stdout:1110: snap_create 2024-09-20T22:46:24.304 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:46:24.329 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:24.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:24.330 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:46:24.431 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:46:24.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:24.453 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:24.453 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:46:24.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:24 smithi154 ceph-mon[33264]: 2.0 scrub starts 2024-09-20T22:46:24.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:24 smithi154 ceph-mon[33264]: 2.0 scrub ok 2024-09-20T22:46:24.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:24 smithi154 ceph-mon[33264]: 5.19 scrub starts 2024-09-20T22:46:24.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:24 smithi154 ceph-mon[33264]: 5.19 scrub ok 2024-09-20T22:46:24.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:24 smithi154 ceph-mon[33264]: 7.a scrub starts 2024-09-20T22:46:24.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:24 smithi154 ceph-mon[33264]: 7.a scrub ok 2024-09-20T22:46:24.547 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:24 smithi084 ceph-mon[40024]: 2.0 scrub starts 2024-09-20T22:46:24.547 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:24 smithi084 ceph-mon[40024]: 2.0 scrub ok 2024-09-20T22:46:24.548 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:24 smithi084 ceph-mon[40024]: 5.19 scrub starts 2024-09-20T22:46:24.548 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:24 smithi084 ceph-mon[40024]: 5.19 scrub ok 2024-09-20T22:46:24.548 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:24 smithi084 ceph-mon[40024]: 7.a scrub starts 2024-09-20T22:46:24.548 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:24 smithi084 ceph-mon[40024]: 7.a scrub ok 2024-09-20T22:46:24.548 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:24 smithi084 ceph-mon[29774]: 2.0 scrub starts 2024-09-20T22:46:24.548 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:24 smithi084 ceph-mon[29774]: 2.0 scrub ok 2024-09-20T22:46:24.548 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:24 smithi084 ceph-mon[29774]: 5.19 scrub starts 2024-09-20T22:46:24.548 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:24 smithi084 ceph-mon[29774]: 5.19 scrub ok 2024-09-20T22:46:24.548 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:24 smithi084 ceph-mon[29774]: 7.a scrub starts 2024-09-20T22:46:24.548 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:24 smithi084 ceph-mon[29774]: 7.a scrub ok 2024-09-20T22:46:24.554 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:46:24.575 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:24.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:24.576 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:46:24.578 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-20T22:46:24.677 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:46:24.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:24.699 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:24.700 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:46:24.800 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:46:24.822 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:24.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:24.823 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:46:24.840 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:46:24.924 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:46:24.945 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:24.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:24.946 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:46:25.043 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-20T22:46:25.047 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:46:25.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:25.108 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:25.108 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:46:25.209 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:46:25.223 INFO:tasks.rados.rados.0.smithi084.stdout:1110: done (0 left) 2024-09-20T22:46:25.223 INFO:tasks.rados.rados.0.smithi084.stdout:1111: rollback oid 43 current snap is 105 2024-09-20T22:46:25.223 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 43 to 103 2024-09-20T22:46:25.223 INFO:tasks.rados.rados.0.smithi084.stdout:1112: delete oid 9 current snap is 105 2024-09-20T22:46:25.230 INFO:tasks.rados.rados.0.smithi084.stdout:1111: finishing rollback tid 1 to smithi08477203-43 2024-09-20T22:46:25.231 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 43 v 0 (ObjNum 129 snap 0 seq_num 2090759256) dirty dne 2024-09-20T22:46:25.232 INFO:tasks.rados.rados.0.smithi084.stdout:1111: done (1 left) 2024-09-20T22:46:25.232 INFO:tasks.rados.rados.0.smithi084.stdout:1112: done (0 left) 2024-09-20T22:46:25.232 INFO:tasks.rados.rados.0.smithi084.stdout:1113: write (excl) oid 13 current snap is 105 2024-09-20T22:46:25.232 INFO:tasks.rados.rados.0.smithi084.stdout:1113: seq_num 411 ranges {507677=410401,1367511=517802,2420497=487180} 2024-09-20T22:46:25.234 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:25.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:25.235 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:46:25.237 INFO:tasks.rados.rados.0.smithi084.stdout:1113: writing smithi08477203-13 from 507677 to 918078 tid 1 2024-09-20T22:46:25.243 INFO:tasks.rados.rados.0.smithi084.stdout:1113: writing smithi08477203-13 from 1367511 to 1885313 tid 2 2024-09-20T22:46:25.249 INFO:tasks.rados.rados.0.smithi084.stdout:1113: writing smithi08477203-13 from 2420497 to 2907677 tid 3 2024-09-20T22:46:25.249 INFO:tasks.rados.rados.0.smithi084.stdout:1114: read oid 50 snap 104 2024-09-20T22:46:25.249 INFO:tasks.rados.rados.0.smithi084.stdout:1114: expect (ObjNum 338 snap 85 seq_num 338) 2024-09-20T22:46:25.249 INFO:tasks.rados.rados.0.smithi084.stdout:1115: read oid 35 snap 101 2024-09-20T22:46:25.249 INFO:tasks.rados.rados.0.smithi084.stdout:1115: expect (ObjNum 373 snap 97 seq_num 373) 2024-09-20T22:46:25.249 INFO:tasks.rados.rados.0.smithi084.stdout:1116: write oid 37 current snap is 105 2024-09-20T22:46:25.249 INFO:tasks.rados.rados.0.smithi084.stdout:1113: finishing write tid 1 to smithi08477203-13 2024-09-20T22:46:25.249 INFO:tasks.rados.rados.0.smithi084.stdout:1116: seq_num 412 ranges {524484=485650,1425993=661460,2847920=76564} 2024-09-20T22:46:25.255 INFO:tasks.rados.rados.0.smithi084.stdout:1116: writing smithi08477203-37 from 524484 to 1010134 tid 1 2024-09-20T22:46:25.261 INFO:tasks.rados.rados.0.smithi084.stdout:1116: writing smithi08477203-37 from 1425993 to 2087453 tid 2 2024-09-20T22:46:25.267 INFO:tasks.rados.rados.0.smithi084.stdout:1116: writing smithi08477203-37 from 2847920 to 2924484 tid 3 2024-09-20T22:46:25.267 INFO:tasks.rados.rados.0.smithi084.stdout:1117: delete oid 25 current snap is 105 2024-09-20T22:46:25.267 INFO:tasks.rados.rados.0.smithi084.stdout:1113: finishing write tid 2 to smithi08477203-13 2024-09-20T22:46:25.267 INFO:tasks.rados.rados.0.smithi084.stdout:1113: finishing write tid 3 to smithi08477203-13 2024-09-20T22:46:25.268 INFO:tasks.rados.rados.0.smithi084.stdout:1113: finishing write tid 4 to smithi08477203-13 2024-09-20T22:46:25.268 INFO:tasks.rados.rados.0.smithi084.stdout:1113: finishing write tid 5 to smithi08477203-13 2024-09-20T22:46:25.268 INFO:tasks.rados.rados.0.smithi084.stdout:1113: oid 13 updating version 0 to 588 2024-09-20T22:46:25.268 INFO:tasks.rados.rados.0.smithi084.stdout:1113: oid 13 updating version 588 to 589 2024-09-20T22:46:25.268 INFO:tasks.rados.rados.0.smithi084.stdout:1113: oid 13 version 589 is already newer than 587 2024-09-20T22:46:25.268 INFO:tasks.rados.rados.0.smithi084.stdout:1113: oid 13 updating version 589 to 590 2024-09-20T22:46:25.268 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 13 v 590 (ObjNum 411 snap 105 seq_num 411) dirty exists 2024-09-20T22:46:25.268 INFO:tasks.rados.rados.0.smithi084.stdout:1113: left oid 13 (ObjNum 411 snap 105 seq_num 411) 2024-09-20T22:46:25.268 INFO:tasks.rados.rados.0.smithi084.stdout:1114: expect (ObjNum 338 snap 85 seq_num 338) 2024-09-20T22:46:25.310 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:46:25.336 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:46:25.349 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-20T22:46:25.349 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [], 2024-09-20T22:46:25.349 INFO:teuthology.orchestra.run.smithi154.stdout: "num_ops": 0 2024-09-20T22:46:25.349 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-20T22:46:25.357 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:25.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:25.357 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:46:25.364 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:25 smithi084 ceph-mon[29774]: pgmap v357: 168 pgs: 168 active+clean; 106 MiB data, 473 MiB used, 536 GiB / 536 GiB avail; 24 MiB/s rd, 7.5 MiB/s wr, 46 op/s 2024-09-20T22:46:25.364 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:25 smithi084 ceph-mon[29774]: osdmap e304: 8 total, 8 up, 6 in 2024-09-20T22:46:25.364 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:25 smithi084 ceph-mon[29774]: 4.14 scrub starts 2024-09-20T22:46:25.364 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:25 smithi084 ceph-mon[29774]: 4.14 scrub ok 2024-09-20T22:46:25.364 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:25 smithi084 ceph-mon[29774]: 6.e scrub starts 2024-09-20T22:46:25.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:25 smithi084 ceph-mon[29774]: 6.e scrub ok 2024-09-20T22:46:25.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:25 smithi084 ceph-mon[40024]: pgmap v357: 168 pgs: 168 active+clean; 106 MiB data, 473 MiB used, 536 GiB / 536 GiB avail; 24 MiB/s rd, 7.5 MiB/s wr, 46 op/s 2024-09-20T22:46:25.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:25 smithi084 ceph-mon[40024]: osdmap e304: 8 total, 8 up, 6 in 2024-09-20T22:46:25.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:25 smithi084 ceph-mon[40024]: 4.14 scrub starts 2024-09-20T22:46:25.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:25 smithi084 ceph-mon[40024]: 4.14 scrub ok 2024-09-20T22:46:25.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:25 smithi084 ceph-mon[40024]: 6.e scrub starts 2024-09-20T22:46:25.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:25 smithi084 ceph-mon[40024]: 6.e scrub ok 2024-09-20T22:46:25.448 INFO:tasks.rados.rados.0.smithi084.stdout:1115: expect (ObjNum 373 snap 97 seq_num 373) 2024-09-20T22:46:25.458 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:46:25.481 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:25.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:25.482 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:46:25.500 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:25 smithi154 ceph-mon[33264]: pgmap v357: 168 pgs: 168 active+clean; 106 MiB data, 473 MiB used, 536 GiB / 536 GiB avail; 24 MiB/s rd, 7.5 MiB/s wr, 46 op/s 2024-09-20T22:46:25.500 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:25 smithi154 ceph-mon[33264]: osdmap e304: 8 total, 8 up, 6 in 2024-09-20T22:46:25.500 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:25 smithi154 ceph-mon[33264]: 4.14 scrub starts 2024-09-20T22:46:25.500 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:25 smithi154 ceph-mon[33264]: 4.14 scrub ok 2024-09-20T22:46:25.500 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:25 smithi154 ceph-mon[33264]: 6.e scrub starts 2024-09-20T22:46:25.500 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:25 smithi154 ceph-mon[33264]: 6.e scrub ok 2024-09-20T22:46:25.583 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:46:25.601 INFO:tasks.rados.rados.0.smithi084.stdout:1116: finishing write tid 1 to smithi08477203-37 2024-09-20T22:46:25.601 INFO:tasks.rados.rados.0.smithi084.stdout:1116: finishing write tid 2 to smithi08477203-37 2024-09-20T22:46:25.601 INFO:tasks.rados.rados.0.smithi084.stdout:1113: done (4 left) 2024-09-20T22:46:25.601 INFO:tasks.rados.rados.0.smithi084.stdout:1114: done (3 left) 2024-09-20T22:46:25.601 INFO:tasks.rados.rados.0.smithi084.stdout:1115: done (2 left) 2024-09-20T22:46:25.601 INFO:tasks.rados.rados.0.smithi084.stdout:1117: done (1 left) 2024-09-20T22:46:25.602 INFO:tasks.rados.rados.0.smithi084.stdout:1118: read oid 8 snap -1 2024-09-20T22:46:25.602 INFO:tasks.rados.rados.0.smithi084.stdout:1118: expect deleted 2024-09-20T22:46:25.602 INFO:tasks.rados.rados.0.smithi084.stdout:1119: write (excl) oid 23 current snap is 105 2024-09-20T22:46:25.602 INFO:tasks.rados.rados.0.smithi084.stdout:1116: finishing write tid 3 to smithi08477203-37 2024-09-20T22:46:25.602 INFO:tasks.rados.rados.0.smithi084.stdout:1119: seq_num 414 ranges {558098=636148,1826604=665129} 2024-09-20T22:46:25.604 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:25.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:25.605 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:46:25.608 INFO:tasks.rados.rados.0.smithi084.stdout:1119: writing smithi08477203-23 from 558098 to 1194246 tid 1 2024-09-20T22:46:25.616 INFO:tasks.rados.rados.0.smithi084.stdout:1119: writing smithi08477203-23 from 1826604 to 2491733 tid 2 2024-09-20T22:46:25.616 INFO:tasks.rados.rados.0.smithi084.stdout:1120: read oid 10 snap -1 2024-09-20T22:46:25.616 INFO:tasks.rados.rados.0.smithi084.stdout:1120: expect (ObjNum 391 snap 101 seq_num 391) 2024-09-20T22:46:25.616 INFO:tasks.rados.rados.0.smithi084.stdout:1121: write (excl) oid 17 current snap is 105 2024-09-20T22:46:25.616 INFO:tasks.rados.rados.0.smithi084.stdout:1116: finishing write tid 4 to smithi08477203-37 2024-09-20T22:46:25.616 INFO:tasks.rados.rados.0.smithi084.stdout:1116: finishing write tid 5 to smithi08477203-37 2024-09-20T22:46:25.616 INFO:tasks.rados.rados.0.smithi084.stdout:1116: oid 37 updating version 0 to 238 2024-09-20T22:46:25.616 INFO:tasks.rados.rados.0.smithi084.stdout:1116: oid 37 version 238 is already newer than 236 2024-09-20T22:46:25.616 INFO:tasks.rados.rados.0.smithi084.stdout:1116: oid 37 version 238 is already newer than 237 2024-09-20T22:46:25.616 INFO:tasks.rados.rados.0.smithi084.stdout:1116: oid 37 updating version 238 to 239 2024-09-20T22:46:25.616 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 37 v 239 (ObjNum 412 snap 105 seq_num 412) dirty exists 2024-09-20T22:46:25.616 INFO:tasks.rados.rados.0.smithi084.stdout:1116: left oid 37 (ObjNum 412 snap 105 seq_num 412) 2024-09-20T22:46:25.617 INFO:tasks.rados.rados.0.smithi084.stdout:1121: seq_num 415 ranges {574905=711397,1885086=525140,2930189=44716} 2024-09-20T22:46:25.623 INFO:tasks.rados.rados.0.smithi084.stdout:1121: writing smithi08477203-17 from 574905 to 1286302 tid 1 2024-09-20T22:46:25.630 INFO:tasks.rados.rados.0.smithi084.stdout:1121: writing smithi08477203-17 from 1885086 to 2410226 tid 2 2024-09-20T22:46:25.633 INFO:tasks.rados.rados.0.smithi084.stdout:1121: writing smithi08477203-17 from 2930189 to 2974905 tid 3 2024-09-20T22:46:25.633 INFO:tasks.rados.rados.0.smithi084.stdout:1116: done (4 left) 2024-09-20T22:46:25.633 INFO:tasks.rados.rados.0.smithi084.stdout:1122: write (excl) oid 30 current snap is 105 2024-09-20T22:46:25.633 INFO:tasks.rados.rados.0.smithi084.stdout:1122: seq_num 416 ranges {591712=786646,1943568=668798} 2024-09-20T22:46:25.640 INFO:tasks.rados.rados.0.smithi084.stdout:1122: writing smithi08477203-30 from 591712 to 1378358 tid 1 2024-09-20T22:46:25.648 INFO:tasks.rados.rados.0.smithi084.stdout:1122: writing smithi08477203-30 from 1943568 to 2612366 tid 2 2024-09-20T22:46:25.648 INFO:tasks.rados.rados.0.smithi084.stdout:1123: write (excl) oid 40 current snap is 105 2024-09-20T22:46:25.648 INFO:tasks.rados.rados.0.smithi084.stdout:1123: seq_num 417 ranges {608519=461895,1602050=528809,2585035=423484} 2024-09-20T22:46:25.653 INFO:tasks.rados.rados.0.smithi084.stdout:1123: writing smithi08477203-40 from 608519 to 1070414 tid 1 2024-09-20T22:46:25.659 INFO:tasks.rados.rados.0.smithi084.stdout:1123: writing smithi08477203-40 from 1602050 to 2130859 tid 2 2024-09-20T22:46:25.664 INFO:tasks.rados.rados.0.smithi084.stdout:1123: writing smithi08477203-40 from 2585035 to 3008519 tid 3 2024-09-20T22:46:25.665 INFO:tasks.rados.rados.0.smithi084.stdout:1124: copy_from oid 36 from oid 19 current snap is 105 2024-09-20T22:46:25.665 INFO:tasks.rados.rados.0.smithi084.stdout:1119: finishing write tid 1 to smithi08477203-23 2024-09-20T22:46:25.665 INFO:tasks.rados.rados.0.smithi084.stdout:1119: finishing write tid 2 to smithi08477203-23 2024-09-20T22:46:25.665 INFO:tasks.rados.rados.0.smithi084.stdout:1119: finishing write tid 3 to smithi08477203-23 2024-09-20T22:46:25.665 INFO:tasks.rados.rados.0.smithi084.stdout:1118: done (6 left) 2024-09-20T22:46:25.665 INFO:tasks.rados.rados.0.smithi084.stdout:1125: read oid 49 snap -1 2024-09-20T22:46:25.665 INFO:tasks.rados.rados.0.smithi084.stdout:1125: expect deleted 2024-09-20T22:46:25.665 INFO:tasks.rados.rados.0.smithi084.stdout:1126: copy_from oid 35 from oid 39 current snap is 105 2024-09-20T22:46:25.665 INFO:tasks.rados.rados.0.smithi084.stdout:1119: finishing write tid 4 to smithi08477203-23 2024-09-20T22:46:25.665 INFO:tasks.rados.rados.0.smithi084.stdout:1119: oid 23 updating version 0 to 221 2024-09-20T22:46:25.665 INFO:tasks.rados.rados.0.smithi084.stdout:1119: oid 23 updating version 221 to 223 2024-09-20T22:46:25.665 INFO:tasks.rados.rados.0.smithi084.stdout:1119: oid 23 version 223 is already newer than 222 2024-09-20T22:46:25.665 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 23 v 223 (ObjNum 414 snap 105 seq_num 414) dirty exists 2024-09-20T22:46:25.665 INFO:tasks.rados.rados.0.smithi084.stdout:1119: left oid 23 (ObjNum 414 snap 105 seq_num 414) 2024-09-20T22:46:25.666 INFO:tasks.rados.rados.0.smithi084.stdout:1120: expect (ObjNum 391 snap 101 seq_num 391) 2024-09-20T22:46:25.706 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:46:25.728 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:25.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:25.729 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:46:25.805 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:46:25.805 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:46:25.822 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:46:25.822 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:46:25.829 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:46:25.838 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:46:25.838 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:46:25.843 INFO:tasks.rados.rados.0.smithi084.stdout:1121: finishing write tid 1 to smithi08477203-17 2024-09-20T22:46:25.843 INFO:tasks.rados.rados.0.smithi084.stdout:1121: finishing write tid 2 to smithi08477203-17 2024-09-20T22:46:25.843 INFO:tasks.rados.rados.0.smithi084.stdout:1121: finishing write tid 3 to smithi08477203-17 2024-09-20T22:46:25.843 INFO:tasks.rados.rados.0.smithi084.stdout:1121: finishing write tid 4 to smithi08477203-17 2024-09-20T22:46:25.843 INFO:tasks.rados.rados.0.smithi084.stdout:1121: finishing write tid 5 to smithi08477203-17 2024-09-20T22:46:25.843 INFO:tasks.rados.rados.0.smithi084.stdout:1121: oid 17 updating version 0 to 144 2024-09-20T22:46:25.843 INFO:tasks.rados.rados.0.smithi084.stdout:1121: oid 17 updating version 144 to 147 2024-09-20T22:46:25.843 INFO:tasks.rados.rados.0.smithi084.stdout:1121: oid 17 version 147 is already newer than 145 2024-09-20T22:46:25.843 INFO:tasks.rados.rados.0.smithi084.stdout:1121: oid 17 version 147 is already newer than 146 2024-09-20T22:46:25.843 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 17 v 147 (ObjNum 415 snap 105 seq_num 415) dirty exists 2024-09-20T22:46:25.843 INFO:tasks.rados.rados.0.smithi084.stdout:1121: left oid 17 (ObjNum 415 snap 105 seq_num 415) 2024-09-20T22:46:25.844 INFO:tasks.rados.rados.0.smithi084.stdout:1122: finishing write tid 1 to smithi08477203-30 2024-09-20T22:46:25.844 INFO:tasks.rados.rados.0.smithi084.stdout:1119: done (7 left) 2024-09-20T22:46:25.844 INFO:tasks.rados.rados.0.smithi084.stdout:1120: done (6 left) 2024-09-20T22:46:25.844 INFO:tasks.rados.rados.0.smithi084.stdout:1121: done (5 left) 2024-09-20T22:46:25.844 INFO:tasks.rados.rados.0.smithi084.stdout:1127: write oid 43 current snap is 105 2024-09-20T22:46:25.844 INFO:tasks.rados.rados.0.smithi084.stdout:1122: finishing write tid 2 to smithi08477203-30 2024-09-20T22:46:25.844 INFO:tasks.rados.rados.0.smithi084.stdout:1122: finishing write tid 3 to smithi08477203-30 2024-09-20T22:46:25.844 INFO:tasks.rados.rados.0.smithi084.stdout:1127: seq_num 418 ranges {625326=537144,1944179=672467} 2024-09-20T22:46:25.849 INFO:tasks.rados.rados.0.smithi084.stdout:1127: writing smithi08477203-43 from 625326 to 1162470 tid 1 2024-09-20T22:46:25.850 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:25.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:25.851 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:46:25.854 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-20T22:46:25.855 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T22:46:25.857 INFO:tasks.rados.rados.0.smithi084.stdout:1127: writing smithi08477203-43 from 1944179 to 2616646 tid 2 2024-09-20T22:46:25.858 INFO:tasks.rados.rados.0.smithi084.stdout:1128: rollback oid 42 current snap is 105 2024-09-20T22:46:25.858 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 42 to 102 2024-09-20T22:46:25.858 INFO:tasks.rados.rados.0.smithi084.stdout:1122: finishing write tid 4 to smithi08477203-30 2024-09-20T22:46:25.858 INFO:tasks.rados.rados.0.smithi084.stdout:1122: oid 30 updating version 0 to 181 2024-09-20T22:46:25.858 INFO:tasks.rados.rados.0.smithi084.stdout:1122: oid 30 updating version 181 to 183 2024-09-20T22:46:25.858 INFO:tasks.rados.rados.0.smithi084.stdout:1122: oid 30 version 183 is already newer than 182 2024-09-20T22:46:25.858 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 30 v 183 (ObjNum 416 snap 105 seq_num 416) dirty exists 2024-09-20T22:46:25.858 INFO:tasks.rados.rados.0.smithi084.stdout:1122: left oid 30 (ObjNum 416 snap 105 seq_num 416) 2024-09-20T22:46:25.858 INFO:tasks.rados.rados.0.smithi084.stdout:1123: finishing write tid 1 to smithi08477203-40 2024-09-20T22:46:25.858 INFO:tasks.rados.rados.0.smithi084.stdout:1122: done (6 left) 2024-09-20T22:46:25.858 INFO:tasks.rados.rados.0.smithi084.stdout:1129: write oid 10 current snap is 105 2024-09-20T22:46:25.858 INFO:tasks.rados.rados.0.smithi084.stdout:1129: seq_num 419 ranges {642133=728746,2119014=532478} 2024-09-20T22:46:25.864 INFO:tasks.rados.rados.0.smithi084.stdout:1129: writing smithi08477203-10 from 642133 to 1370879 tid 1 2024-09-20T22:46:25.873 INFO:tasks.rados.rados.0.smithi084.stdout:1129: writing smithi08477203-10 from 2119014 to 2651492 tid 2 2024-09-20T22:46:25.873 INFO:tasks.rados.rados.0.smithi084.stdout:1130: copy_from oid 47 from oid 26 current snap is 105 2024-09-20T22:46:25.873 INFO:tasks.rados.rados.0.smithi084.stdout:1123: finishing write tid 2 to smithi08477203-40 2024-09-20T22:46:25.873 INFO:tasks.rados.rados.0.smithi084.stdout:1123: finishing write tid 3 to smithi08477203-40 2024-09-20T22:46:25.873 INFO:tasks.rados.rados.0.smithi084.stdout:1123: finishing write tid 4 to smithi08477203-40 2024-09-20T22:46:25.873 INFO:tasks.rados.rados.0.smithi084.stdout:1123: finishing write tid 5 to smithi08477203-40 2024-09-20T22:46:25.874 INFO:tasks.rados.rados.0.smithi084.stdout:1123: oid 40 updating version 0 to 227 2024-09-20T22:46:25.874 INFO:tasks.rados.rados.0.smithi084.stdout:1123: oid 40 updating version 227 to 228 2024-09-20T22:46:25.874 INFO:tasks.rados.rados.0.smithi084.stdout:1123: oid 40 version 228 is already newer than 225 2024-09-20T22:46:25.874 INFO:tasks.rados.rados.0.smithi084.stdout:1123: oid 40 version 228 is already newer than 226 2024-09-20T22:46:25.874 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 40 v 228 (ObjNum 417 snap 105 seq_num 417) dirty exists 2024-09-20T22:46:25.874 INFO:tasks.rados.rados.0.smithi084.stdout:1123: left oid 40 (ObjNum 417 snap 105 seq_num 417) 2024-09-20T22:46:25.874 INFO:tasks.rados.rados.0.smithi084.stdout:1124: finishing copy_from racing read to smithi08477203-36 2024-09-20T22:46:25.874 INFO:tasks.rados.rados.0.smithi084.stdout:1126: finishing copy_from racing read to smithi08477203-35 2024-09-20T22:46:25.874 INFO:tasks.rados.rados.0.smithi084.stdout:1126: finishing copy_from to smithi08477203-35 2024-09-20T22:46:25.874 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 35 v 199 (ObjNum 245 snap 54 seq_num 245) dirty exists 2024-09-20T22:46:25.874 INFO:tasks.rados.rados.0.smithi084.stdout:1124: finishing copy_from to smithi08477203-36 2024-09-20T22:46:25.874 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 36 v 376 (ObjNum 258 snap 60 seq_num 258) dirty exists 2024-09-20T22:46:25.874 INFO:tasks.rados.rados.0.smithi084.stdout:1127: finishing write tid 1 to smithi08477203-43 2024-09-20T22:46:25.875 INFO:tasks.rados.rados.0.smithi084.stdout:1128: finishing rollback tid 1 to smithi08477203-42 2024-09-20T22:46:25.875 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 42 v 0 (ObjNum 803183840 snap 32761 seq_num 304) dirty dne 2024-09-20T22:46:25.875 INFO:tasks.rados.rados.0.smithi084.stdout:1127: finishing write tid 2 to smithi08477203-43 2024-09-20T22:46:25.875 INFO:tasks.rados.rados.0.smithi084.stdout:1127: finishing write tid 3 to smithi08477203-43 2024-09-20T22:46:25.875 INFO:tasks.rados.rados.0.smithi084.stdout:1127: finishing write tid 4 to smithi08477203-43 2024-09-20T22:46:25.875 INFO:tasks.rados.rados.0.smithi084.stdout:1127: oid 43 updating version 0 to 56 2024-09-20T22:46:25.875 INFO:tasks.rados.rados.0.smithi084.stdout:1127: oid 43 updating version 56 to 57 2024-09-20T22:46:25.875 INFO:tasks.rados.rados.0.smithi084.stdout:1127: oid 43 version 57 is already newer than 55 2024-09-20T22:46:25.875 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 43 v 57 (ObjNum 418 snap 105 seq_num 418) dirty exists 2024-09-20T22:46:25.875 INFO:tasks.rados.rados.0.smithi084.stdout:1127: left oid 43 (ObjNum 418 snap 105 seq_num 418) 2024-09-20T22:46:25.875 INFO:tasks.rados.rados.0.smithi084.stdout:1129: finishing write tid 1 to smithi08477203-10 2024-09-20T22:46:25.875 INFO:tasks.rados.rados.0.smithi084.stdout:1123: done (7 left) 2024-09-20T22:46:25.876 INFO:tasks.rados.rados.0.smithi084.stdout:1124: done (6 left) 2024-09-20T22:46:25.876 INFO:tasks.rados.rados.0.smithi084.stdout:1125: done (5 left) 2024-09-20T22:46:25.876 INFO:tasks.rados.rados.0.smithi084.stdout:1126: done (4 left) 2024-09-20T22:46:25.876 INFO:tasks.rados.rados.0.smithi084.stdout:1127: done (3 left) 2024-09-20T22:46:25.876 INFO:tasks.rados.rados.0.smithi084.stdout:1128: done (2 left) 2024-09-20T22:46:25.876 INFO:tasks.rados.rados.0.smithi084.stdout:1131: write oid 31 current snap is 105 2024-09-20T22:46:25.876 INFO:tasks.rados.rados.0.smithi084.stdout:1131: seq_num 420 ranges {658940=403995,1777496=676136,2950951=107989} 2024-09-20T22:46:25.876 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:46:25.876 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:46:25.878 INFO:tasks.rados.rados.0.smithi084.stdout:1131: writing smithi08477203-31 from 658940 to 1062935 tid 1 2024-09-20T22:46:25.885 INFO:tasks.rados.rados.0.smithi084.stdout:1131: writing smithi08477203-31 from 1777496 to 2453632 tid 2 2024-09-20T22:46:25.889 INFO:tasks.rados.rados.0.smithi084.stdout:1131: writing smithi08477203-31 from 2950951 to 3058940 tid 3 2024-09-20T22:46:25.890 INFO:tasks.rados.rados.0.smithi084.stdout:1132: write (excl) oid 4 current snap is 105 2024-09-20T22:46:25.890 INFO:tasks.rados.rados.0.smithi084.stdout:1129: finishing write tid 2 to smithi08477203-10 2024-09-20T22:46:25.890 INFO:tasks.rados.rados.0.smithi084.stdout:1129: finishing write tid 3 to smithi08477203-10 2024-09-20T22:46:25.890 INFO:tasks.rados.rados.0.smithi084.stdout:1129: finishing write tid 4 to smithi08477203-10 2024-09-20T22:46:25.890 INFO:tasks.rados.rados.0.smithi084.stdout:1129: oid 10 updating version 0 to 309 2024-09-20T22:46:25.890 INFO:tasks.rados.rados.0.smithi084.stdout:1129: oid 10 version 309 is already newer than 307 2024-09-20T22:46:25.890 INFO:tasks.rados.rados.0.smithi084.stdout:1129: oid 10 version 309 is already newer than 308 2024-09-20T22:46:25.890 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 10 v 309 (ObjNum 419 snap 105 seq_num 419) dirty exists 2024-09-20T22:46:25.890 INFO:tasks.rados.rados.0.smithi084.stdout:1129: left oid 10 (ObjNum 419 snap 105 seq_num 419) 2024-09-20T22:46:25.890 INFO:tasks.rados.rados.0.smithi084.stdout:1130: finishing copy_from racing read to smithi08477203-47 2024-09-20T22:46:25.890 INFO:tasks.rados.rados.0.smithi084.stdout:1130: finishing copy_from to smithi08477203-47 2024-09-20T22:46:25.890 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 47 v 252 (ObjNum 390 snap 101 seq_num 390) dirty exists 2024-09-20T22:46:25.890 INFO:tasks.rados.rados.0.smithi084.stdout:1131: finishing write tid 1 to smithi08477203-31 2024-09-20T22:46:25.890 INFO:tasks.rados.rados.0.smithi084.stdout:1132: seq_num 421 ranges {675747=479244,1835978=536147} 2024-09-20T22:46:25.895 INFO:tasks.rados.rados.0.smithi084.stdout:1132: writing smithi08477203-4 from 675747 to 1154991 tid 1 2024-09-20T22:46:25.895 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:46:25.896 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:46:25.903 INFO:tasks.rados.rados.0.smithi084.stdout:1132: writing smithi08477203-4 from 1835978 to 2372125 tid 2 2024-09-20T22:46:25.903 INFO:tasks.rados.rados.0.smithi084.stdout:1129: done (3 left) 2024-09-20T22:46:25.903 INFO:tasks.rados.rados.0.smithi084.stdout:1130: done (2 left) 2024-09-20T22:46:25.903 INFO:tasks.rados.rados.0.smithi084.stdout:1133: read oid 35 snap 104 2024-09-20T22:46:25.903 INFO:tasks.rados.rados.0.smithi084.stdout:1133: expect (ObjNum 409 snap 103 seq_num 409) 2024-09-20T22:46:25.903 INFO:tasks.rados.rados.0.smithi084.stdout:1134: copy_from oid 28 from oid 18 current snap is 105 2024-09-20T22:46:25.903 INFO:tasks.rados.rados.0.smithi084.stdout:1131: finishing write tid 2 to smithi08477203-31 2024-09-20T22:46:25.903 INFO:tasks.rados.rados.0.smithi084.stdout:1131: finishing write tid 3 to smithi08477203-31 2024-09-20T22:46:25.903 INFO:tasks.rados.rados.0.smithi084.stdout:1131: finishing write tid 4 to smithi08477203-31 2024-09-20T22:46:25.903 INFO:tasks.rados.rados.0.smithi084.stdout:1131: finishing write tid 5 to smithi08477203-31 2024-09-20T22:46:25.904 INFO:tasks.rados.rados.0.smithi084.stdout:1131: oid 31 updating version 0 to 389 2024-09-20T22:46:25.904 INFO:tasks.rados.rados.0.smithi084.stdout:1131: oid 31 version 389 is already newer than 388 2024-09-20T22:46:25.904 INFO:tasks.rados.rados.0.smithi084.stdout:1131: oid 31 version 389 is already newer than 386 2024-09-20T22:46:25.904 INFO:tasks.rados.rados.0.smithi084.stdout:1131: oid 31 version 389 is already newer than 387 2024-09-20T22:46:25.904 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 31 v 389 (ObjNum 420 snap 105 seq_num 420) dirty exists 2024-09-20T22:46:25.904 INFO:tasks.rados.rados.0.smithi084.stdout:1131: left oid 31 (ObjNum 420 snap 105 seq_num 420) 2024-09-20T22:46:25.904 INFO:tasks.rados.rados.0.smithi084.stdout:1132: finishing write tid 1 to smithi08477203-4 2024-09-20T22:46:25.904 INFO:tasks.rados.rados.0.smithi084.stdout:1131: done (3 left) 2024-09-20T22:46:25.904 INFO:tasks.rados.rados.0.smithi084.stdout:1135: snap_create 2024-09-20T22:46:25.908 INFO:tasks.rados.rados.0.smithi084.stdout:1132: finishing write tid 2 to smithi08477203-4 2024-09-20T22:46:25.910 INFO:tasks.rados.rados.0.smithi084.stdout:1132: finishing write tid 3 to smithi08477203-4 2024-09-20T22:46:25.911 INFO:tasks.rados.rados.0.smithi084.stdout:1133: expect (ObjNum 409 snap 103 seq_num 409) 2024-09-20T22:46:25.915 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:46:25.915 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:46:25.931 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:46:25.931 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:46:25.952 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:46:25.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:25.973 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:25.973 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:46:26.010 INFO:tasks.rados.rados.0.smithi084.stdout:1132: finishing write tid 4 to smithi08477203-4 2024-09-20T22:46:26.010 INFO:tasks.rados.rados.0.smithi084.stdout:1132: oid 4 updating version 0 to 594 2024-09-20T22:46:26.010 INFO:tasks.rados.rados.0.smithi084.stdout:1132: oid 4 version 594 is already newer than 593 2024-09-20T22:46:26.010 INFO:tasks.rados.rados.0.smithi084.stdout:1132: oid 4 updating version 594 to 595 2024-09-20T22:46:26.010 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 4 v 595 (ObjNum 421 snap 105 seq_num 421) dirty exists 2024-09-20T22:46:26.010 INFO:tasks.rados.rados.0.smithi084.stdout:1132: left oid 4 (ObjNum 421 snap 105 seq_num 421) 2024-09-20T22:46:26.010 INFO:tasks.rados.rados.0.smithi084.stdout:1134: finishing copy_from racing read to smithi08477203-28 2024-09-20T22:46:26.010 INFO:tasks.rados.rados.0.smithi084.stdout:1134: finishing copy_from to smithi08477203-28 2024-09-20T22:46:26.010 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 28 v 298 (ObjNum 393 snap 101 seq_num 393) dirty exists 2024-09-20T22:46:26.074 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:46:26.095 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:26.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:26.096 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:46:26.131 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.4 dump_blocked_ops 2024-09-20T22:46:26.197 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:46:26.215 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is unset 2024-09-20T22:46:26.220 INFO:tasks.rados.rados.0.smithi084.stdout:1132: done (3 left) 2024-09-20T22:46:26.220 INFO:tasks.rados.rados.0.smithi084.stdout:1133: done (2 left) 2024-09-20T22:46:26.220 INFO:tasks.rados.rados.0.smithi084.stdout:1134: done (1 left) 2024-09-20T22:46:26.220 INFO:tasks.rados.rados.0.smithi084.stdout:1135: done (0 left) 2024-09-20T22:46:26.220 INFO:tasks.rados.rados.0.smithi084.stdout:1136: delete oid 49 current snap is 106 2024-09-20T22:46:26.225 INFO:tasks.rados.rados.0.smithi084.stdout:1136: done (0 left) 2024-09-20T22:46:26.225 INFO:tasks.rados.rados.0.smithi084.stdout:1137: snap_create 2024-09-20T22:46:26.240 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:26.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:26.240 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:46:26.341 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:46:26.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:26.363 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:26.364 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:46:26.432 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:46:26.464 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:46:26.472 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:26 smithi084 ceph-mon[40024]: 3.c scrub starts 2024-09-20T22:46:26.472 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:26 smithi084 ceph-mon[40024]: 3.c scrub ok 2024-09-20T22:46:26.472 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:26 smithi084 ceph-mon[40024]: osdmap e305: 8 total, 8 up, 6 in 2024-09-20T22:46:26.472 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:26 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/67567188' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:46:26.472 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:26 smithi084 ceph-mon[40024]: 4.1c scrub starts 2024-09-20T22:46:26.472 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:26 smithi084 ceph-mon[40024]: 4.1c scrub ok 2024-09-20T22:46:26.472 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:26 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/100527557' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:46:26.473 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:26 smithi084 ceph-mon[40024]: 5.12 scrub starts 2024-09-20T22:46:26.473 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:26 smithi084 ceph-mon[40024]: 5.12 scrub ok 2024-09-20T22:46:26.473 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:26 smithi084 ceph-mon[40024]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T22:46:26.473 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:26 smithi084 ceph-mon[40024]: Cluster is now healthy 2024-09-20T22:46:26.473 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:26 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/67567188' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T22:46:26.473 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:26 smithi084 ceph-mon[40024]: osdmap e306: 8 total, 8 up, 6 in 2024-09-20T22:46:26.473 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:26 smithi084 ceph-mon[29774]: 3.c scrub starts 2024-09-20T22:46:26.473 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:26 smithi084 ceph-mon[29774]: 3.c scrub ok 2024-09-20T22:46:26.473 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:26 smithi084 ceph-mon[29774]: osdmap e305: 8 total, 8 up, 6 in 2024-09-20T22:46:26.473 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:26 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/67567188' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:46:26.473 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:26 smithi084 ceph-mon[29774]: 4.1c scrub starts 2024-09-20T22:46:26.473 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:26 smithi084 ceph-mon[29774]: 4.1c scrub ok 2024-09-20T22:46:26.474 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:26 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/100527557' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:46:26.474 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:26 smithi084 ceph-mon[29774]: 5.12 scrub starts 2024-09-20T22:46:26.474 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:26 smithi084 ceph-mon[29774]: 5.12 scrub ok 2024-09-20T22:46:26.474 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:26 smithi084 ceph-mon[29774]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T22:46:26.474 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:26 smithi084 ceph-mon[29774]: Cluster is now healthy 2024-09-20T22:46:26.474 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:26 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/67567188' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T22:46:26.474 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:26 smithi084 ceph-mon[29774]: osdmap e306: 8 total, 8 up, 6 in 2024-09-20T22:46:26.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:26.487 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:26.487 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:46:26.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:26 smithi154 ceph-mon[33264]: 3.c scrub starts 2024-09-20T22:46:26.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:26 smithi154 ceph-mon[33264]: 3.c scrub ok 2024-09-20T22:46:26.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:26 smithi154 ceph-mon[33264]: osdmap e305: 8 total, 8 up, 6 in 2024-09-20T22:46:26.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:26 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/67567188' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:46:26.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:26 smithi154 ceph-mon[33264]: 4.1c scrub starts 2024-09-20T22:46:26.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:26 smithi154 ceph-mon[33264]: 4.1c scrub ok 2024-09-20T22:46:26.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:26 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/100527557' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:46:26.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:26 smithi154 ceph-mon[33264]: 5.12 scrub starts 2024-09-20T22:46:26.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:26 smithi154 ceph-mon[33264]: 5.12 scrub ok 2024-09-20T22:46:26.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:26 smithi154 ceph-mon[33264]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T22:46:26.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:26 smithi154 ceph-mon[33264]: Cluster is now healthy 2024-09-20T22:46:26.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:26 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/67567188' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T22:46:26.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:26 smithi154 ceph-mon[33264]: osdmap e306: 8 total, 8 up, 6 in 2024-09-20T22:46:26.587 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:46:26.611 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:26.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:26.612 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:46:26.712 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:46:26.733 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:26.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:26.734 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:46:26.835 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:46:26.857 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:26.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:26.858 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:46:26.959 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:46:26.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:26.981 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:26.981 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:46:27.081 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:46:27.082 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-20T22:46:27.082 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:46:27.104 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:27.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:27.105 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:46:27.206 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:46:27.228 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:27.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:27.229 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:46:27.231 INFO:tasks.rados.rados.0.smithi084.stdout:1137: done (0 left) 2024-09-20T22:46:27.231 INFO:tasks.rados.rados.0.smithi084.stdout:1138: snap_create 2024-09-20T22:46:27.323 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:27 smithi084 ceph-mon[40024]: pgmap v360: 168 pgs: 168 active+clean; 106 MiB data, 473 MiB used, 536 GiB / 536 GiB avail; 27 MiB/s rd, 8.5 MiB/s wr, 52 op/s 2024-09-20T22:46:27.323 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:27 smithi084 ceph-mon[40024]: 6.d scrub starts 2024-09-20T22:46:27.323 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:27 smithi084 ceph-mon[40024]: 6.d scrub ok 2024-09-20T22:46:27.323 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:27 smithi084 ceph-mon[40024]: osdmap e307: 8 total, 8 up, 6 in 2024-09-20T22:46:27.324 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:27 smithi084 ceph-mon[29774]: pgmap v360: 168 pgs: 168 active+clean; 106 MiB data, 473 MiB used, 536 GiB / 536 GiB avail; 27 MiB/s rd, 8.5 MiB/s wr, 52 op/s 2024-09-20T22:46:27.324 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:27 smithi084 ceph-mon[29774]: 6.d scrub starts 2024-09-20T22:46:27.324 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:27 smithi084 ceph-mon[29774]: 6.d scrub ok 2024-09-20T22:46:27.324 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:27 smithi084 ceph-mon[29774]: osdmap e307: 8 total, 8 up, 6 in 2024-09-20T22:46:27.330 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:46:27.352 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:27.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:27.353 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:46:27.454 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:46:27.475 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:27.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:27.476 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:46:27.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:27 smithi154 ceph-mon[33264]: pgmap v360: 168 pgs: 168 active+clean; 106 MiB data, 473 MiB used, 536 GiB / 536 GiB avail; 27 MiB/s rd, 8.5 MiB/s wr, 52 op/s 2024-09-20T22:46:27.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:27 smithi154 ceph-mon[33264]: 6.d scrub starts 2024-09-20T22:46:27.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:27 smithi154 ceph-mon[33264]: 6.d scrub ok 2024-09-20T22:46:27.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:27 smithi154 ceph-mon[33264]: osdmap e307: 8 total, 8 up, 6 in 2024-09-20T22:46:27.577 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:46:27.598 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:27.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:27.599 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:46:27.609 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 5, 6, 7] out_osds: [4, 0] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-20T22:46:27.609 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-20T22:46:27.610 INFO:tasks.thrashosds.thrasher:reset_purged_snaps_last 2024-09-20T22:46:27.610 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph tell osd.1 reset_purged_snaps_last 2024-09-20T22:46:27.699 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:46:27.722 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:27.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:27.723 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:46:27.734 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.4 dump_historic_ops 2024-09-20T22:46:27.824 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:46:27.846 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:27.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:27.847 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:46:27.885 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:46:27.948 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:46:27.971 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:27.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:27.972 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:46:28.022 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:46:28.073 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:46:28.094 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:28.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:28.095 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:46:28.180 INFO:tasks.rados.rados.0.smithi084.stdout:1138: done (0 left) 2024-09-20T22:46:28.180 INFO:tasks.rados.rados.0.smithi084.stdout:1139: read oid 40 snap -1 2024-09-20T22:46:28.180 INFO:tasks.rados.rados.0.smithi084.stdout:1139: expect (ObjNum 417 snap 105 seq_num 417) 2024-09-20T22:46:28.180 INFO:tasks.rados.rados.0.smithi084.stdout:1140: write (excl) oid 10 current snap is 108 2024-09-20T22:46:28.181 INFO:tasks.rados.rados.0.smithi084.stdout:1140: seq_num 423 ranges {709361=629742,1836589=539816,3033220=76141} 2024-09-20T22:46:28.189 INFO:tasks.rados.rados.0.smithi084.stdout:1140: writing smithi08477203-10 from 709361 to 1339103 tid 1 2024-09-20T22:46:28.194 INFO:tasks.rados.rados.0.smithi084.stdout:1140: writing smithi08477203-10 from 1836589 to 2376405 tid 2 2024-09-20T22:46:28.196 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:46:28.199 INFO:tasks.rados.rados.0.smithi084.stdout:1140: writing smithi08477203-10 from 3033220 to 3109361 tid 3 2024-09-20T22:46:28.199 INFO:tasks.rados.rados.0.smithi084.stdout:1141: snap_create 2024-09-20T22:46:28.199 INFO:tasks.rados.rados.0.smithi084.stdout:1139: expect (ObjNum 417 snap 105 seq_num 417) 2024-09-20T22:46:28.219 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:28.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:28.220 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:46:28.321 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:46:28.341 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:28 smithi084 ceph-mon[40024]: 4.15 scrub starts 2024-09-20T22:46:28.341 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:28 smithi084 ceph-mon[40024]: 4.15 scrub ok 2024-09-20T22:46:28.341 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:28 smithi084 ceph-mon[40024]: osdmap e308: 8 total, 8 up, 6 in 2024-09-20T22:46:28.342 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:46:28 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: ::ffff:172.21.15.154 - - [20/Sep/2024:22:46:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-20T22:46:28.342 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:28 smithi084 ceph-mon[29774]: 4.15 scrub starts 2024-09-20T22:46:28.342 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:28 smithi084 ceph-mon[29774]: 4.15 scrub ok 2024-09-20T22:46:28.342 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:28 smithi084 ceph-mon[29774]: osdmap e308: 8 total, 8 up, 6 in 2024-09-20T22:46:28.342 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:28.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:28.343 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:46:28.426 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:28 smithi154 ceph-mon[33264]: 4.15 scrub starts 2024-09-20T22:46:28.426 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:28 smithi154 ceph-mon[33264]: 4.15 scrub ok 2024-09-20T22:46:28.426 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:28 smithi154 ceph-mon[33264]: osdmap e308: 8 total, 8 up, 6 in 2024-09-20T22:46:28.444 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:46:28.462 INFO:tasks.rados.rados.0.smithi084.stdout:1140: finishing write tid 1 to smithi08477203-10 2024-09-20T22:46:28.462 INFO:tasks.rados.rados.0.smithi084.stdout:1140: finishing write tid 2 to smithi08477203-10 2024-09-20T22:46:28.462 INFO:tasks.rados.rados.0.smithi084.stdout:1140: finishing write tid 3 to smithi08477203-10 2024-09-20T22:46:28.462 INFO:tasks.rados.rados.0.smithi084.stdout:1140: finishing write tid 4 to smithi08477203-10 2024-09-20T22:46:28.463 INFO:tasks.rados.rados.0.smithi084.stdout:1140: finishing write tid 5 to smithi08477203-10 2024-09-20T22:46:28.463 INFO:tasks.rados.rados.0.smithi084.stdout:1140: oid 10 updating version 0 to 311 2024-09-20T22:46:28.463 INFO:tasks.rados.rados.0.smithi084.stdout:1140: oid 10 updating version 311 to 312 2024-09-20T22:46:28.463 INFO:tasks.rados.rados.0.smithi084.stdout:1140: oid 10 updating version 312 to 313 2024-09-20T22:46:28.463 INFO:tasks.rados.rados.0.smithi084.stdout:1140: oid 10 updating version 313 to 314 2024-09-20T22:46:28.463 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 10 v 314 (ObjNum 423 snap 108 seq_num 423) dirty exists 2024-09-20T22:46:28.463 INFO:tasks.rados.rados.0.smithi084.stdout:1140: left oid 10 (ObjNum 423 snap 108 seq_num 423) 2024-09-20T22:46:28.466 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:28.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:28.467 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:46:28.568 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:46:28.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:28.590 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:28.590 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:46:28.691 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:46:28.695 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-20T22:46:28.695 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:46:28.713 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:28.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:28.714 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:46:28.814 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:46:28.836 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:28.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:28.837 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:46:28.937 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:46:28.959 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:28.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:28.960 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:46:28.969 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph tell osd.2 reset_purged_snaps_last 2024-09-20T22:46:29.014 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd set noscrub 2024-09-20T22:46:29.061 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:46:29.083 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:29.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:29.084 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:46:29.185 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:46:29.192 INFO:tasks.rados.rados.0.smithi084.stdout:1139: done (2 left) 2024-09-20T22:46:29.192 INFO:tasks.rados.rados.0.smithi084.stdout:1140: done (1 left) 2024-09-20T22:46:29.192 INFO:tasks.rados.rados.0.smithi084.stdout:1141: done (0 left) 2024-09-20T22:46:29.192 INFO:tasks.rados.rados.0.smithi084.stdout:1142: write (excl) oid 48 current snap is 109 2024-09-20T22:46:29.192 INFO:tasks.rados.rados.0.smithi084.stdout:1142: seq_num 424 ranges {726168=704991,1895071=683474} 2024-09-20T22:46:29.200 INFO:tasks.rados.rados.0.smithi084.stdout:1142: writing smithi08477203-48 from 726168 to 1431159 tid 1 2024-09-20T22:46:29.208 INFO:tasks.rados.rados.0.smithi084.stdout:1142: writing smithi08477203-48 from 1895071 to 2578545 tid 2 2024-09-20T22:46:29.208 INFO:tasks.rados.rados.0.smithi084.stdout:1143: read oid 20 snap 102 2024-09-20T22:46:29.208 INFO:tasks.rados.rados.0.smithi084.stdout:1143: expect (ObjNum 380 snap 100 seq_num 380) 2024-09-20T22:46:29.208 INFO:tasks.rados.rados.0.smithi084.stdout:1144: read oid 41 snap -1 2024-09-20T22:46:29.208 INFO:tasks.rados.rados.0.smithi084.stdout:1144: expect (ObjNum 374 snap 99 seq_num 374) 2024-09-20T22:46:29.208 INFO:tasks.rados.rados.0.smithi084.stdout:1145: write (excl) oid 28 current snap is 109 2024-09-20T22:46:29.209 INFO:tasks.rados.rados.0.smithi084.stdout:1145: seq_num 425 ranges {742975=780240,1953553=427132,2971713=171262} 2024-09-20T22:46:29.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:29.209 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:29.210 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:46:29.217 INFO:tasks.rados.rados.0.smithi084.stdout:1145: writing smithi08477203-28 from 742975 to 1523215 tid 1 2024-09-20T22:46:29.221 INFO:tasks.rados.rados.0.smithi084.stdout:1145: writing smithi08477203-28 from 1953553 to 2380685 tid 2 2024-09-20T22:46:29.226 INFO:tasks.rados.rados.0.smithi084.stdout:1145: writing smithi08477203-28 from 2971713 to 3142975 tid 3 2024-09-20T22:46:29.226 INFO:tasks.rados.rados.0.smithi084.stdout:1146: write oid 17 current snap is 109 2024-09-20T22:46:29.226 INFO:tasks.rados.rados.0.smithi084.stdout:1146: seq_num 426 ranges {759782=571842,2012035=687143} 2024-09-20T22:46:29.233 INFO:tasks.rados.rados.0.smithi084.stdout:1146: writing smithi08477203-17 from 759782 to 1331624 tid 1 2024-09-20T22:46:29.242 INFO:tasks.rados.rados.0.smithi084.stdout:1146: writing smithi08477203-17 from 2012035 to 2699178 tid 2 2024-09-20T22:46:29.242 INFO:tasks.rados.rados.0.smithi084.stdout:1144: expect (ObjNum 374 snap 99 seq_num 374) 2024-09-20T22:46:29.283 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:46:29.310 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:46:29.329 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:46:29.333 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:29.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:29.334 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:46:29.350 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-20T22:46:29.355 INFO:journalctl@ceph.iscsi.iscsi.a.smithi154.stdout:Sep 20 22:46:29 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-iscsi-iscsi-a[59730]: debug there is no tcmu-runner data available 2024-09-20T22:46:29.355 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:29 smithi154 ceph-mon[33264]: 6.5 scrub starts 2024-09-20T22:46:29.355 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:29 smithi154 ceph-mon[33264]: 6.5 scrub ok 2024-09-20T22:46:29.355 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:29 smithi154 ceph-mon[33264]: pgmap v363: 168 pgs: 168 active+clean; 106 MiB data, 473 MiB used, 536 GiB / 536 GiB avail 2024-09-20T22:46:29.355 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:29 smithi154 ceph-mon[33264]: osdmap e309: 8 total, 8 up, 6 in 2024-09-20T22:46:29.381 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-20T22:46:29.434 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:46:29.441 INFO:tasks.rados.rados.0.smithi084.stdout:1142: finishing write tid 1 to smithi08477203-48 2024-09-20T22:46:29.442 INFO:tasks.rados.rados.0.smithi084.stdout:1142: finishing write tid 2 to smithi08477203-48 2024-09-20T22:46:29.442 INFO:tasks.rados.rados.0.smithi084.stdout:1142: finishing write tid 3 to smithi08477203-48 2024-09-20T22:46:29.442 INFO:tasks.rados.rados.0.smithi084.stdout:1142: finishing write tid 4 to smithi08477203-48 2024-09-20T22:46:29.442 INFO:tasks.rados.rados.0.smithi084.stdout:1142: oid 48 updating version 0 to 241 2024-09-20T22:46:29.442 INFO:tasks.rados.rados.0.smithi084.stdout:1142: oid 48 updating version 241 to 243 2024-09-20T22:46:29.442 INFO:tasks.rados.rados.0.smithi084.stdout:1142: oid 48 version 243 is already newer than 242 2024-09-20T22:46:29.442 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 48 v 243 (ObjNum 424 snap 109 seq_num 424) dirty exists 2024-09-20T22:46:29.442 INFO:tasks.rados.rados.0.smithi084.stdout:1142: left oid 48 (ObjNum 424 snap 109 seq_num 424) 2024-09-20T22:46:29.442 INFO:tasks.rados.rados.0.smithi084.stdout:1143: expect (ObjNum 380 snap 100 seq_num 380) 2024-09-20T22:46:29.455 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:29.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:29.456 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:46:29.521 INFO:tasks.rados.rados.0.smithi084.stdout:1145: finishing write tid 1 to smithi08477203-28 2024-09-20T22:46:29.521 INFO:tasks.rados.rados.0.smithi084.stdout:1147: read oid 15 snap 106 2024-09-20T22:46:29.521 INFO:tasks.rados.rados.0.smithi084.stdout:1147: expect deleted 2024-09-20T22:46:29.521 INFO:tasks.rados.rados.0.smithi084.stdout:1142: done (5 left) 2024-09-20T22:46:29.522 INFO:tasks.rados.rados.0.smithi084.stdout:1143: done (4 left) 2024-09-20T22:46:29.522 INFO:tasks.rados.rados.0.smithi084.stdout:1144: done (3 left) 2024-09-20T22:46:29.522 INFO:tasks.rados.rados.0.smithi084.stdout:1148: read oid 3 snap -1 2024-09-20T22:46:29.522 INFO:tasks.rados.rados.0.smithi084.stdout:1148: expect (ObjNum 338 snap 85 seq_num 338) 2024-09-20T22:46:29.522 INFO:tasks.rados.rados.0.smithi084.stdout:1149: snap_remove snap 101 2024-09-20T22:46:29.548 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:29 smithi084 ceph-mon[40024]: 6.5 scrub starts 2024-09-20T22:46:29.548 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:29 smithi084 ceph-mon[40024]: 6.5 scrub ok 2024-09-20T22:46:29.548 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:29 smithi084 ceph-mon[40024]: pgmap v363: 168 pgs: 168 active+clean; 106 MiB data, 473 MiB used, 536 GiB / 536 GiB avail 2024-09-20T22:46:29.548 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:29 smithi084 ceph-mon[40024]: osdmap e309: 8 total, 8 up, 6 in 2024-09-20T22:46:29.548 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:29 smithi084 ceph-mon[29774]: 6.5 scrub starts 2024-09-20T22:46:29.548 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:29 smithi084 ceph-mon[29774]: 6.5 scrub ok 2024-09-20T22:46:29.548 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:29 smithi084 ceph-mon[29774]: pgmap v363: 168 pgs: 168 active+clean; 106 MiB data, 473 MiB used, 536 GiB / 536 GiB avail 2024-09-20T22:46:29.548 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:29 smithi084 ceph-mon[29774]: osdmap e309: 8 total, 8 up, 6 in 2024-09-20T22:46:29.557 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:46:29.579 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:29.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:29.580 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:46:29.630 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:46:29.681 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:46:29.702 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:29.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:29.703 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:46:29.804 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:46:29.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:29.826 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:29.827 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:46:29.927 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:46:29.985 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:29.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:29.986 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:46:30.087 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:46:30.108 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:30.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:30.109 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:46:30.109 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-20T22:46:30.109 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:46:30.210 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:46:30.232 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:30.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:30.233 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:46:30.256 INFO:tasks.rados.rados.0.smithi084.stdout:1149: done (4 left) 2024-09-20T22:46:30.256 INFO:tasks.rados.rados.0.smithi084.stdout:1150: delete oid 31 current snap is 109 2024-09-20T22:46:30.257 INFO:tasks.rados.rados.0.smithi084.stdout:1145: finishing write tid 2 to smithi08477203-28 2024-09-20T22:46:30.257 INFO:tasks.rados.rados.0.smithi084.stdout:1145: finishing write tid 3 to smithi08477203-28 2024-09-20T22:46:30.257 INFO:tasks.rados.rados.0.smithi084.stdout:1145: finishing write tid 4 to smithi08477203-28 2024-09-20T22:46:30.257 INFO:tasks.rados.rados.0.smithi084.stdout:1145: finishing write tid 5 to smithi08477203-28 2024-09-20T22:46:30.257 INFO:tasks.rados.rados.0.smithi084.stdout:1145: oid 28 updating version 0 to 601 2024-09-20T22:46:30.257 INFO:tasks.rados.rados.0.smithi084.stdout:1145: oid 28 version 601 is already newer than 600 2024-09-20T22:46:30.257 INFO:tasks.rados.rados.0.smithi084.stdout:1145: oid 28 version 601 is already newer than 599 2024-09-20T22:46:30.257 INFO:tasks.rados.rados.0.smithi084.stdout:1145: oid 28 updating version 601 to 602 2024-09-20T22:46:30.257 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 28 v 602 (ObjNum 425 snap 109 seq_num 425) dirty exists 2024-09-20T22:46:30.257 INFO:tasks.rados.rados.0.smithi084.stdout:1145: left oid 28 (ObjNum 425 snap 109 seq_num 425) 2024-09-20T22:46:30.257 INFO:tasks.rados.rados.0.smithi084.stdout:1146: finishing write tid 1 to smithi08477203-17 2024-09-20T22:46:30.257 INFO:tasks.rados.rados.0.smithi084.stdout:1146: finishing write tid 2 to smithi08477203-17 2024-09-20T22:46:30.257 INFO:tasks.rados.rados.0.smithi084.stdout:1146: finishing write tid 3 to smithi08477203-17 2024-09-20T22:46:30.257 INFO:tasks.rados.rados.0.smithi084.stdout:1146: finishing write tid 4 to smithi08477203-17 2024-09-20T22:46:30.258 INFO:tasks.rados.rados.0.smithi084.stdout:1146: oid 17 updating version 0 to 150 2024-09-20T22:46:30.258 INFO:tasks.rados.rados.0.smithi084.stdout:1146: oid 17 updating version 150 to 151 2024-09-20T22:46:30.258 INFO:tasks.rados.rados.0.smithi084.stdout:1146: oid 17 version 151 is already newer than 149 2024-09-20T22:46:30.258 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 17 v 151 (ObjNum 426 snap 109 seq_num 426) dirty exists 2024-09-20T22:46:30.258 INFO:tasks.rados.rados.0.smithi084.stdout:1146: left oid 17 (ObjNum 426 snap 109 seq_num 426) 2024-09-20T22:46:30.258 INFO:tasks.rados.rados.0.smithi084.stdout:1148: expect (ObjNum 338 snap 85 seq_num 338) 2024-09-20T22:46:30.333 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:46:30.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:30.356 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:30.357 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:46:30.403 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:30 smithi154 ceph-mon[33264]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:46:30.404 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:30 smithi154 ceph-mon[33264]: 6.18 scrub starts 2024-09-20T22:46:30.404 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:30 smithi154 ceph-mon[33264]: 6.18 scrub ok 2024-09-20T22:46:30.404 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:30 smithi154 ceph-mon[33264]: 6.14 scrub starts 2024-09-20T22:46:30.404 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:30 smithi154 ceph-mon[33264]: 6.14 scrub ok 2024-09-20T22:46:30.433 INFO:tasks.rados.rados.0.smithi084.stdout:1145: done (4 left) 2024-09-20T22:46:30.433 INFO:tasks.rados.rados.0.smithi084.stdout:1146: done (3 left) 2024-09-20T22:46:30.433 INFO:tasks.rados.rados.0.smithi084.stdout:1147: done (2 left) 2024-09-20T22:46:30.433 INFO:tasks.rados.rados.0.smithi084.stdout:1148: done (1 left) 2024-09-20T22:46:30.433 INFO:tasks.rados.rados.0.smithi084.stdout:1150: done (0 left) 2024-09-20T22:46:30.433 INFO:tasks.rados.rados.0.smithi084.stdout:1151: copy_from oid 22 from oid 10 current snap is 109 2024-09-20T22:46:30.434 INFO:tasks.rados.rados.0.smithi084.stdout:1152: read oid 18 snap 104 2024-09-20T22:46:30.434 INFO:tasks.rados.rados.0.smithi084.stdout:1152: expect (ObjNum 393 snap 101 seq_num 393) 2024-09-20T22:46:30.434 INFO:tasks.rados.rados.0.smithi084.stdout:1153: snap_create 2024-09-20T22:46:30.436 INFO:tasks.rados.rados.0.smithi084.stdout:1152: expect (ObjNum 393 snap 101 seq_num 393) 2024-09-20T22:46:30.457 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:46:30.477 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:30.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:30.478 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:46:30.567 INFO:tasks.rados.rados.0.smithi084.stdout:1151: finishing copy_from racing read to smithi08477203-22 2024-09-20T22:46:30.567 INFO:tasks.rados.rados.0.smithi084.stdout:1151: finishing copy_from to smithi08477203-22 2024-09-20T22:46:30.567 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 22 v 314 (ObjNum 423 snap 108 seq_num 423) dirty exists 2024-09-20T22:46:30.579 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:46:30.601 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:30.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:30.602 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:46:30.614 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:30 smithi084 ceph-mon[29774]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:46:30.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:30 smithi084 ceph-mon[29774]: 6.18 scrub starts 2024-09-20T22:46:30.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:30 smithi084 ceph-mon[29774]: 6.18 scrub ok 2024-09-20T22:46:30.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:30 smithi084 ceph-mon[29774]: 6.14 scrub starts 2024-09-20T22:46:30.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:30 smithi084 ceph-mon[29774]: 6.14 scrub ok 2024-09-20T22:46:30.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:30 smithi084 ceph-mon[40024]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:46:30.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:30 smithi084 ceph-mon[40024]: 6.18 scrub starts 2024-09-20T22:46:30.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:30 smithi084 ceph-mon[40024]: 6.18 scrub ok 2024-09-20T22:46:30.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:30 smithi084 ceph-mon[40024]: 6.14 scrub starts 2024-09-20T22:46:30.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:30 smithi084 ceph-mon[40024]: 6.14 scrub ok 2024-09-20T22:46:30.703 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:46:30.726 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:30.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:30.727 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:46:30.735 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.5 dump_blocked_ops 2024-09-20T22:46:30.827 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:46:30.846 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph tell osd.3 reset_purged_snaps_last 2024-09-20T22:46:30.849 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:30.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:30.850 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:46:30.951 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:46:30.974 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:30.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:30.975 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:46:31.021 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:46:31.076 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:46:31.098 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:31.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:31.099 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:46:31.198 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:46:31.199 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:46:31.223 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:31.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:31.225 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:46:31.259 INFO:tasks.rados.rados.0.smithi084.stdout:1151: done (2 left) 2024-09-20T22:46:31.259 INFO:tasks.rados.rados.0.smithi084.stdout:1152: done (1 left) 2024-09-20T22:46:31.260 INFO:tasks.rados.rados.0.smithi084.stdout:1153: done (0 left) 2024-09-20T22:46:31.260 INFO:tasks.rados.rados.0.smithi084.stdout:1154: rollback oid 37 current snap is 110 2024-09-20T22:46:31.260 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 37 to 104 2024-09-20T22:46:31.260 INFO:tasks.rados.rados.0.smithi084.stdout:1155: snap_remove snap 109 2024-09-20T22:46:31.325 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:46:31.346 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:31.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:31.347 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:46:31.370 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:31 smithi084 ceph-mon[40024]: 5.10 scrub starts 2024-09-20T22:46:31.371 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:31 smithi084 ceph-mon[40024]: 5.10 scrub ok 2024-09-20T22:46:31.371 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:31 smithi084 ceph-mon[40024]: pgmap v366: 168 pgs: 168 active+clean; 134 MiB data, 524 MiB used, 536 GiB / 536 GiB avail; 16 MiB/s rd, 5.6 MiB/s wr, 32 op/s 2024-09-20T22:46:31.371 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:31 smithi084 ceph-mon[40024]: osdmap e310: 8 total, 8 up, 6 in 2024-09-20T22:46:31.371 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:31 smithi084 ceph-mon[40024]: 3.9 scrub starts 2024-09-20T22:46:31.371 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:31 smithi084 ceph-mon[40024]: 3.9 scrub ok 2024-09-20T22:46:31.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:31 smithi084 ceph-mon[29774]: 5.10 scrub starts 2024-09-20T22:46:31.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:31 smithi084 ceph-mon[29774]: 5.10 scrub ok 2024-09-20T22:46:31.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:31 smithi084 ceph-mon[29774]: pgmap v366: 168 pgs: 168 active+clean; 134 MiB data, 524 MiB used, 536 GiB / 536 GiB avail; 16 MiB/s rd, 5.6 MiB/s wr, 32 op/s 2024-09-20T22:46:31.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:31 smithi084 ceph-mon[29774]: osdmap e310: 8 total, 8 up, 6 in 2024-09-20T22:46:31.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:31 smithi084 ceph-mon[29774]: 3.9 scrub starts 2024-09-20T22:46:31.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:31 smithi084 ceph-mon[29774]: 3.9 scrub ok 2024-09-20T22:46:31.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:31 smithi154 ceph-mon[33264]: 5.10 scrub starts 2024-09-20T22:46:31.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:31 smithi154 ceph-mon[33264]: 5.10 scrub ok 2024-09-20T22:46:31.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:31 smithi154 ceph-mon[33264]: pgmap v366: 168 pgs: 168 active+clean; 134 MiB data, 524 MiB used, 536 GiB / 536 GiB avail; 16 MiB/s rd, 5.6 MiB/s wr, 32 op/s 2024-09-20T22:46:31.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:31 smithi154 ceph-mon[33264]: osdmap e310: 8 total, 8 up, 6 in 2024-09-20T22:46:31.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:31 smithi154 ceph-mon[33264]: 3.9 scrub starts 2024-09-20T22:46:31.392 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:31 smithi154 ceph-mon[33264]: 3.9 scrub ok 2024-09-20T22:46:31.449 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:46:31.470 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:31.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:31.471 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:46:31.492 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-20T22:46:31.492 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:46:31.572 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:46:31.593 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:31.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:31.594 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:46:31.695 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:46:31.717 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:31.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:31.717 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:46:31.818 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:46:31.839 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:31.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:31.840 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:46:31.940 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:46:31.962 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:31.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:31.963 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:46:32.064 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:46:32.087 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:32.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:32.088 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:46:32.107 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.5 dump_historic_ops 2024-09-20T22:46:32.189 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:46:32.211 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:32.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:32.212 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:46:32.260 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is set 2024-09-20T22:46:32.264 INFO:tasks.rados.rados.0.smithi084.stdout:1155: done (1 left) 2024-09-20T22:46:32.265 INFO:tasks.rados.rados.0.smithi084.stdout:1156: write (excl) oid 23 current snap is 110 2024-09-20T22:46:32.265 INFO:tasks.rados.rados.0.smithi084.stdout:1154: finishing rollback tid 0 to smithi08477203-37 2024-09-20T22:46:32.265 INFO:tasks.rados.rados.0.smithi084.stdout:1154: finishing rollback tid 1 to smithi08477203-37 2024-09-20T22:46:32.265 INFO:tasks.rados.rados.0.smithi084.stdout:1154: finishing rollback tid 2 to smithi08477203-37 2024-09-20T22:46:32.265 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 37 v 247 (ObjNum 329 snap 84 seq_num 329) dirty exists 2024-09-20T22:46:32.265 INFO:tasks.rados.rados.0.smithi084.stdout:1156: seq_num 428 ranges {793396=722340,2128999=690812} 2024-09-20T22:46:32.273 INFO:tasks.rados.rados.0.smithi084.stdout:1156: writing smithi08477203-23 from 793396 to 1515736 tid 1 2024-09-20T22:46:32.281 INFO:tasks.rados.rados.0.smithi084.stdout:1156: writing smithi08477203-23 from 2128999 to 2819811 tid 2 2024-09-20T22:46:32.281 INFO:tasks.rados.rados.0.smithi084.stdout:1154: done (1 left) 2024-09-20T22:46:32.281 INFO:tasks.rados.rados.0.smithi084.stdout:1157: snap_remove snap 103 2024-09-20T22:46:32.313 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:46:32.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:32.335 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:32.335 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:46:32.364 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:32 smithi084 ceph-mon[40024]: 3.d scrub starts 2024-09-20T22:46:32.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:32 smithi084 ceph-mon[40024]: 3.d scrub ok 2024-09-20T22:46:32.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:32 smithi084 ceph-mon[40024]: 4.11 scrub starts 2024-09-20T22:46:32.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:32 smithi084 ceph-mon[40024]: 4.11 scrub ok 2024-09-20T22:46:32.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:32 smithi084 ceph-mon[40024]: osdmap e311: 8 total, 8 up, 6 in 2024-09-20T22:46:32.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:32 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/3369493194' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-20T22:46:32.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:32 smithi084 ceph-mon[29774]: 3.d scrub starts 2024-09-20T22:46:32.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:32 smithi084 ceph-mon[29774]: 3.d scrub ok 2024-09-20T22:46:32.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:32 smithi084 ceph-mon[29774]: 4.11 scrub starts 2024-09-20T22:46:32.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:32 smithi084 ceph-mon[29774]: 4.11 scrub ok 2024-09-20T22:46:32.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:32 smithi084 ceph-mon[29774]: osdmap e311: 8 total, 8 up, 6 in 2024-09-20T22:46:32.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:32 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/3369493194' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-20T22:46:32.400 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:46:32.436 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:46:32.454 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph tell osd.5 reset_purged_snaps_last 2024-09-20T22:46:32.457 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:32.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:32.458 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:46:32.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:32 smithi154 ceph-mon[33264]: 3.d scrub starts 2024-09-20T22:46:32.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:32 smithi154 ceph-mon[33264]: 3.d scrub ok 2024-09-20T22:46:32.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:32 smithi154 ceph-mon[33264]: 4.11 scrub starts 2024-09-20T22:46:32.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:32 smithi154 ceph-mon[33264]: 4.11 scrub ok 2024-09-20T22:46:32.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:32 smithi154 ceph-mon[33264]: osdmap e311: 8 total, 8 up, 6 in 2024-09-20T22:46:32.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:32 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/3369493194' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-20T22:46:32.559 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:46:32.579 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:32.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:32.580 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:46:32.681 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:46:32.686 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:46:32.702 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:32.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:32.703 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:46:32.803 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:46:32.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:32.825 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:32.826 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:46:32.873 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-20T22:46:32.873 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:46:32.926 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:46:32.947 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:32.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:32.948 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:46:33.049 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:46:33.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:33.071 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:33.071 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:46:33.172 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:46:33.193 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:33.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:33.195 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:46:33.275 INFO:tasks.rados.rados.0.smithi084.stdout:1157: done (1 left) 2024-09-20T22:46:33.276 INFO:tasks.rados.rados.0.smithi084.stdout:1158: write oid 41 current snap is 110 2024-09-20T22:46:33.276 INFO:tasks.rados.rados.0.smithi084.stdout:1156: finishing write tid 1 to smithi08477203-23 2024-09-20T22:46:33.276 INFO:tasks.rados.rados.0.smithi084.stdout:1158: seq_num 429 ranges {410203=797589,1787481=434470,2681405=528798} 2024-09-20T22:46:33.283 INFO:tasks.rados.rados.0.smithi084.stdout:1158: writing smithi08477203-41 from 410203 to 1207792 tid 1 2024-09-20T22:46:33.289 INFO:tasks.rados.rados.0.smithi084.stdout:1158: writing smithi08477203-41 from 1787481 to 2221951 tid 2 2024-09-20T22:46:33.294 INFO:tasks.rados.rados.0.smithi084.stdout:1158: writing smithi08477203-41 from 2681405 to 3210203 tid 3 2024-09-20T22:46:33.294 INFO:tasks.rados.rados.0.smithi084.stdout:1159: snap_remove snap 107 2024-09-20T22:46:33.295 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:46:33.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:33.316 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:33.316 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:46:33.417 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:46:33.438 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:33.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:33.439 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:46:33.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:33 smithi154 ceph-mon[33264]: pgmap v369: 168 pgs: 168 active+clean; 134 MiB data, 524 MiB used, 536 GiB / 536 GiB avail; 16 MiB/s rd, 5.6 MiB/s wr, 32 op/s 2024-09-20T22:46:33.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:33 smithi154 ceph-mon[33264]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:46:33.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:33 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/3369493194' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T22:46:33.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:33 smithi154 ceph-mon[33264]: osdmap e312: 8 total, 8 up, 6 in 2024-09-20T22:46:33.509 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:46:33.510 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:46:33.516 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:46:33.527 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:46:33.528 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:46:33.528 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:33 smithi084 ceph-mon[40024]: pgmap v369: 168 pgs: 168 active+clean; 134 MiB data, 524 MiB used, 536 GiB / 536 GiB avail; 16 MiB/s rd, 5.6 MiB/s wr, 32 op/s 2024-09-20T22:46:33.528 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:33 smithi084 ceph-mon[40024]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:46:33.528 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:33 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/3369493194' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T22:46:33.528 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:33 smithi084 ceph-mon[40024]: osdmap e312: 8 total, 8 up, 6 in 2024-09-20T22:46:33.528 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:33 smithi084 ceph-mon[29774]: pgmap v369: 168 pgs: 168 active+clean; 134 MiB data, 524 MiB used, 536 GiB / 536 GiB avail; 16 MiB/s rd, 5.6 MiB/s wr, 32 op/s 2024-09-20T22:46:33.529 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:33 smithi084 ceph-mon[29774]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:46:33.529 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:33 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/3369493194' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T22:46:33.529 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:33 smithi084 ceph-mon[29774]: osdmap e312: 8 total, 8 up, 6 in 2024-09-20T22:46:33.540 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:46:33.544 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:46:33.545 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:46:33.561 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-20T22:46:33.561 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T22:46:33.562 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:33.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:33.563 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:46:33.578 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:46:33.578 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:46:33.594 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:46:33.594 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:46:33.611 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:46:33.611 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:46:33.623 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-20T22:46:33.627 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:46:33.627 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:46:33.664 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:46:33.687 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:33.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:33.688 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:46:33.788 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:46:33.810 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:33.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:33.811 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:46:33.869 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:46:33.913 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:46:33.934 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:33.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:33.935 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:46:34.036 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:46:34.058 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:34.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:34.059 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:46:34.159 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:46:34.180 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:34.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:34.181 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:46:34.280 INFO:tasks.rados.rados.0.smithi084.stdout:1159: done (2 left) 2024-09-20T22:46:34.280 INFO:tasks.rados.rados.0.smithi084.stdout:1160: copy_from oid 15 from oid 32 current snap is 110 2024-09-20T22:46:34.280 INFO:tasks.rados.rados.0.smithi084.stdout:1156: finishing write tid 2 to smithi08477203-23 2024-09-20T22:46:34.280 INFO:tasks.rados.rados.0.smithi084.stdout:1156: finishing write tid 3 to smithi08477203-23 2024-09-20T22:46:34.280 INFO:tasks.rados.rados.0.smithi084.stdout:1156: finishing write tid 4 to smithi08477203-23 2024-09-20T22:46:34.280 INFO:tasks.rados.rados.0.smithi084.stdout:1156: oid 23 updating version 0 to 234 2024-09-20T22:46:34.280 INFO:tasks.rados.rados.0.smithi084.stdout:1156: oid 23 updating version 234 to 236 2024-09-20T22:46:34.280 INFO:tasks.rados.rados.0.smithi084.stdout:1156: oid 23 version 236 is already newer than 235 2024-09-20T22:46:34.280 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 23 v 236 (ObjNum 428 snap 110 seq_num 428) dirty exists 2024-09-20T22:46:34.280 INFO:tasks.rados.rados.0.smithi084.stdout:1156: left oid 23 (ObjNum 428 snap 110 seq_num 428) 2024-09-20T22:46:34.280 INFO:tasks.rados.rados.0.smithi084.stdout:1158: finishing write tid 1 to smithi08477203-41 2024-09-20T22:46:34.281 INFO:tasks.rados.rados.0.smithi084.stdout:1158: finishing write tid 2 to smithi08477203-41 2024-09-20T22:46:34.281 INFO:tasks.rados.rados.0.smithi084.stdout:1158: finishing write tid 3 to smithi08477203-41 2024-09-20T22:46:34.281 INFO:tasks.rados.rados.0.smithi084.stdout:1158: finishing write tid 4 to smithi08477203-41 2024-09-20T22:46:34.281 INFO:tasks.rados.rados.0.smithi084.stdout:1158: finishing write tid 5 to smithi08477203-41 2024-09-20T22:46:34.281 INFO:tasks.rados.rados.0.smithi084.stdout:1158: oid 41 updating version 0 to 88 2024-09-20T22:46:34.281 INFO:tasks.rados.rados.0.smithi084.stdout:1158: oid 41 updating version 88 to 91 2024-09-20T22:46:34.281 INFO:tasks.rados.rados.0.smithi084.stdout:1158: oid 41 version 91 is already newer than 89 2024-09-20T22:46:34.281 INFO:tasks.rados.rados.0.smithi084.stdout:1158: oid 41 version 91 is already newer than 90 2024-09-20T22:46:34.281 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 41 v 91 (ObjNum 429 snap 110 seq_num 429) dirty exists 2024-09-20T22:46:34.281 INFO:tasks.rados.rados.0.smithi084.stdout:1158: left oid 41 (ObjNum 429 snap 110 seq_num 429) 2024-09-20T22:46:34.281 INFO:tasks.rados.rados.0.smithi084.stdout:1156: done (2 left) 2024-09-20T22:46:34.281 INFO:tasks.rados.rados.0.smithi084.stdout:1158: done (1 left) 2024-09-20T22:46:34.281 INFO:tasks.rados.rados.0.smithi084.stdout:1161: delete oid 14 current snap is 110 2024-09-20T22:46:34.282 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:46:34.282 INFO:tasks.rados.rados.0.smithi084.stdout:1161: done (1 left) 2024-09-20T22:46:34.282 INFO:tasks.rados.rados.0.smithi084.stdout:1162: snap_create 2024-09-20T22:46:34.293 INFO:tasks.rados.rados.0.smithi084.stdout:1160: finishing copy_from racing read to smithi08477203-15 2024-09-20T22:46:34.296 INFO:tasks.rados.rados.0.smithi084.stdout:1160: finishing copy_from to smithi08477203-15 2024-09-20T22:46:34.297 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 15 v 382 (ObjNum 401 snap 101 seq_num 401) dirty exists 2024-09-20T22:46:34.304 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:34.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:34.305 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:46:34.364 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:34 smithi084 ceph-mon[40024]: osdmap e313: 8 total, 8 up, 6 in 2024-09-20T22:46:34.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:34 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/4110321814' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:46:34.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:34 smithi084 ceph-mon[29774]: osdmap e313: 8 total, 8 up, 6 in 2024-09-20T22:46:34.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:34 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/4110321814' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:46:34.406 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:46:34.407 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-20T22:46:34.407 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [], 2024-09-20T22:46:34.408 INFO:teuthology.orchestra.run.smithi154.stdout: "num_ops": 0 2024-09-20T22:46:34.408 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-20T22:46:34.429 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:34.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:34.430 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:46:34.531 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:46:34.552 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:34.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:34.553 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:46:34.654 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:46:34.674 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:34.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:34.675 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:46:34.722 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:34 smithi154 ceph-mon[33264]: osdmap e313: 8 total, 8 up, 6 in 2024-09-20T22:46:34.722 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:34 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/4110321814' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:46:34.775 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:46:34.797 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:34.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:34.798 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:46:34.899 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:46:34.921 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:34.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:34.922 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:46:35.022 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:46:35.043 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:35.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:35.044 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:46:35.145 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:46:35.166 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.6 dump_blocked_ops 2024-09-20T22:46:35.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:35.167 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:35.167 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:46:35.257 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph tell osd.6 reset_purged_snaps_last 2024-09-20T22:46:35.268 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:46:35.286 INFO:tasks.rados.rados.0.smithi084.stdout:1160: done (1 left) 2024-09-20T22:46:35.286 INFO:tasks.rados.rados.0.smithi084.stdout:1162: done (0 left) 2024-09-20T22:46:35.286 INFO:tasks.rados.rados.0.smithi084.stdout:1163: delete oid 37 current snap is 111 2024-09-20T22:46:35.290 INFO:tasks.rados.rados.0.smithi084.stdout:1163: done (0 left) 2024-09-20T22:46:35.290 INFO:tasks.rados.rados.0.smithi084.stdout:1164: read oid 28 snap 104 2024-09-20T22:46:35.290 INFO:tasks.rados.rados.0.smithi084.stdout:1164: expect (ObjNum 396 snap 101 seq_num 396) 2024-09-20T22:46:35.290 INFO:tasks.rados.rados.0.smithi084.stdout:1165: snap_create 2024-09-20T22:46:35.295 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:35.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:35.296 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:46:35.299 INFO:tasks.rados.rados.0.smithi084.stdout:1164: expect (ObjNum 396 snap 101 seq_num 396) 2024-09-20T22:46:35.316 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-20T22:46:35.396 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:46:35.417 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:35.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:35.418 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:46:35.423 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:46:35.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:35 smithi154 ceph-mon[33264]: pgmap v372: 168 pgs: 4 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 161 active+clean; 147 MiB data, 552 MiB used, 536 GiB / 536 GiB avail; 10 MiB/s rd, 2.5 MiB/s wr, 18 op/s 2024-09-20T22:46:35.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:35 smithi154 ceph-mon[33264]: osdmap e314: 8 total, 8 up, 6 in 2024-09-20T22:46:35.518 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:46:35.520 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:46:35.539 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:35.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:35.540 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:46:35.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:35 smithi084 ceph-mon[40024]: pgmap v372: 168 pgs: 4 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 161 active+clean; 147 MiB data, 552 MiB used, 536 GiB / 536 GiB avail; 10 MiB/s rd, 2.5 MiB/s wr, 18 op/s 2024-09-20T22:46:35.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:35 smithi084 ceph-mon[40024]: osdmap e314: 8 total, 8 up, 6 in 2024-09-20T22:46:35.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:35 smithi084 ceph-mon[29774]: pgmap v372: 168 pgs: 4 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 161 active+clean; 147 MiB data, 552 MiB used, 536 GiB / 536 GiB avail; 10 MiB/s rd, 2.5 MiB/s wr, 18 op/s 2024-09-20T22:46:35.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:35 smithi084 ceph-mon[29774]: osdmap e314: 8 total, 8 up, 6 in 2024-09-20T22:46:35.641 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:46:35.661 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:35.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:35.662 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:46:35.763 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:46:35.785 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:35.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:35.786 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:46:35.868 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-20T22:46:35.868 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [], 2024-09-20T22:46:35.868 INFO:teuthology.orchestra.run.smithi154.stdout: "complaint_time": 30, 2024-09-20T22:46:35.868 INFO:teuthology.orchestra.run.smithi154.stdout: "num_blocked_ops": 0 2024-09-20T22:46:35.868 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-20T22:46:35.887 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:46:35.908 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:35.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:35.909 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:46:36.006 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:46:36.010 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:46:36.033 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:36.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:36.034 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:46:36.135 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:46:36.156 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:36.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:36.156 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:46:36.257 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:46:36.278 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:36.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:36.279 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:46:36.288 INFO:tasks.rados.rados.0.smithi084.stdout:1164: done (1 left) 2024-09-20T22:46:36.289 INFO:tasks.rados.rados.0.smithi084.stdout:1165: done (0 left) 2024-09-20T22:46:36.289 INFO:tasks.rados.rados.0.smithi084.stdout:1166: delete oid 44 current snap is 112 2024-09-20T22:46:36.293 INFO:tasks.rados.rados.0.smithi084.stdout:1166: done (0 left) 2024-09-20T22:46:36.294 INFO:tasks.rados.rados.0.smithi084.stdout:1167: rollback oid 32 current snap is 112 2024-09-20T22:46:36.294 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 32 to 111 2024-09-20T22:46:36.294 INFO:tasks.rados.rados.0.smithi084.stdout:1168: write (excl) oid 24 current snap is 112 2024-09-20T22:46:36.294 INFO:tasks.rados.rados.0.smithi084.stdout:1168: seq_num 433 ranges {477431=698585,1905056=441808,3074744=202687} 2024-09-20T22:46:36.300 INFO:tasks.rados.rados.0.smithi084.stdout:1168: writing smithi08477203-24 from 477431 to 1176016 tid 1 2024-09-20T22:46:36.307 INFO:tasks.rados.rados.0.smithi084.stdout:1168: writing smithi08477203-24 from 1905056 to 2346864 tid 2 2024-09-20T22:46:36.312 INFO:tasks.rados.rados.0.smithi084.stdout:1168: writing smithi08477203-24 from 3074744 to 3277431 tid 3 2024-09-20T22:46:36.312 INFO:tasks.rados.rados.0.smithi084.stdout:1169: write (excl) oid 12 current snap is 112 2024-09-20T22:46:36.312 INFO:tasks.rados.rados.0.smithi084.stdout:1167: finishing rollback tid 0 to smithi08477203-32 2024-09-20T22:46:36.312 INFO:tasks.rados.rados.0.smithi084.stdout:1167: finishing rollback tid 1 to smithi08477203-32 2024-09-20T22:46:36.312 INFO:tasks.rados.rados.0.smithi084.stdout:1167: finishing rollback tid 2 to smithi08477203-32 2024-09-20T22:46:36.312 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 32 v 385 (ObjNum 401 snap 101 seq_num 401) dirty exists 2024-09-20T22:46:36.312 INFO:tasks.rados.rados.0.smithi084.stdout:1168: finishing write tid 1 to smithi08477203-24 2024-09-20T22:46:36.313 INFO:tasks.rados.rados.0.smithi084.stdout:1168: finishing write tid 2 to smithi08477203-24 2024-09-20T22:46:36.313 INFO:tasks.rados.rados.0.smithi084.stdout:1169: seq_num 434 ranges {494238=490187,1563538=701819,2818520=475718} 2024-09-20T22:46:36.317 INFO:tasks.rados.rados.0.smithi084.stdout:1169: writing smithi08477203-12 from 494238 to 984425 tid 1 2024-09-20T22:46:36.324 INFO:tasks.rados.rados.0.smithi084.stdout:1169: writing smithi08477203-12 from 1563538 to 2265357 tid 2 2024-09-20T22:46:36.332 INFO:tasks.rados.rados.0.smithi084.stdout:1169: writing smithi08477203-12 from 2818520 to 3294238 tid 3 2024-09-20T22:46:36.332 INFO:tasks.rados.rados.0.smithi084.stdout:1167: done (2 left) 2024-09-20T22:46:36.332 INFO:tasks.rados.rados.0.smithi084.stdout:1170: copy_from oid 28 from oid 29 current snap is 112 2024-09-20T22:46:36.332 INFO:tasks.rados.rados.0.smithi084.stdout:1168: finishing write tid 3 to smithi08477203-24 2024-09-20T22:46:36.332 INFO:tasks.rados.rados.0.smithi084.stdout:1168: finishing write tid 4 to smithi08477203-24 2024-09-20T22:46:36.332 INFO:tasks.rados.rados.0.smithi084.stdout:1168: finishing write tid 5 to smithi08477203-24 2024-09-20T22:46:36.332 INFO:tasks.rados.rados.0.smithi084.stdout:1168: oid 24 updating version 0 to 125 2024-09-20T22:46:36.332 INFO:tasks.rados.rados.0.smithi084.stdout:1168: oid 24 updating version 125 to 126 2024-09-20T22:46:36.332 INFO:tasks.rados.rados.0.smithi084.stdout:1168: oid 24 version 126 is already newer than 124 2024-09-20T22:46:36.332 INFO:tasks.rados.rados.0.smithi084.stdout:1168: oid 24 version 126 is already newer than 123 2024-09-20T22:46:36.332 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 24 v 126 (ObjNum 433 snap 112 seq_num 433) dirty exists 2024-09-20T22:46:36.332 INFO:tasks.rados.rados.0.smithi084.stdout:1168: left oid 24 (ObjNum 433 snap 112 seq_num 433) 2024-09-20T22:46:36.332 INFO:tasks.rados.rados.0.smithi084.stdout:1169: finishing write tid 1 to smithi08477203-12 2024-09-20T22:46:36.333 INFO:tasks.rados.rados.0.smithi084.stdout:1169: finishing write tid 2 to smithi08477203-12 2024-09-20T22:46:36.333 INFO:tasks.rados.rados.0.smithi084.stdout:1168: done (2 left) 2024-09-20T22:46:36.333 INFO:tasks.rados.rados.0.smithi084.stdout:1171: read oid 36 snap -1 2024-09-20T22:46:36.333 INFO:tasks.rados.rados.0.smithi084.stdout:1171: expect (ObjNum 258 snap 60 seq_num 258) 2024-09-20T22:46:36.333 INFO:tasks.rados.rados.0.smithi084.stdout:1172: write oid 16 current snap is 112 2024-09-20T22:46:36.333 INFO:tasks.rados.rados.0.smithi084.stdout:1172: seq_num 435 ranges {511045=565436,1622020=445477,2729590=556855} 2024-09-20T22:46:36.336 INFO:tasks.rados.rados.0.smithi084.stdout:1172: writing smithi08477203-16 from 511045 to 1076481 tid 1 2024-09-20T22:46:36.341 INFO:tasks.rados.rados.0.smithi084.stdout:1172: writing smithi08477203-16 from 1622020 to 2067497 tid 2 2024-09-20T22:46:36.349 INFO:tasks.rados.rados.0.smithi084.stdout:1172: writing smithi08477203-16 from 2729590 to 3286445 tid 3 2024-09-20T22:46:36.349 INFO:tasks.rados.rados.0.smithi084.stdout:1173: snap_create 2024-09-20T22:46:36.349 INFO:tasks.rados.rados.0.smithi084.stdout:1169: finishing write tid 3 to smithi08477203-12 2024-09-20T22:46:36.349 INFO:tasks.rados.rados.0.smithi084.stdout:1169: finishing write tid 4 to smithi08477203-12 2024-09-20T22:46:36.349 INFO:tasks.rados.rados.0.smithi084.stdout:1169: finishing write tid 5 to smithi08477203-12 2024-09-20T22:46:36.349 INFO:tasks.rados.rados.0.smithi084.stdout:1169: oid 12 updating version 0 to 258 2024-09-20T22:46:36.349 INFO:tasks.rados.rados.0.smithi084.stdout:1169: oid 12 updating version 258 to 261 2024-09-20T22:46:36.349 INFO:tasks.rados.rados.0.smithi084.stdout:1169: oid 12 version 261 is already newer than 260 2024-09-20T22:46:36.349 INFO:tasks.rados.rados.0.smithi084.stdout:1169: oid 12 version 261 is already newer than 259 2024-09-20T22:46:36.349 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 12 v 261 (ObjNum 434 snap 112 seq_num 434) dirty exists 2024-09-20T22:46:36.349 INFO:tasks.rados.rados.0.smithi084.stdout:1169: left oid 12 (ObjNum 434 snap 112 seq_num 434) 2024-09-20T22:46:36.349 INFO:tasks.rados.rados.0.smithi084.stdout:1172: finishing write tid 1 to smithi08477203-16 2024-09-20T22:46:36.349 INFO:tasks.rados.rados.0.smithi084.stdout:1171: expect (ObjNum 258 snap 60 seq_num 258) 2024-09-20T22:46:36.380 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:46:36.400 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:36.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:36.401 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:46:36.428 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-20T22:46:36.497 INFO:tasks.rados.rados.0.smithi084.stdout:1170: finishing copy_from racing read to smithi08477203-28 2024-09-20T22:46:36.497 INFO:tasks.rados.rados.0.smithi084.stdout:1172: finishing write tid 2 to smithi08477203-16 2024-09-20T22:46:36.497 INFO:tasks.rados.rados.0.smithi084.stdout:1170: finishing copy_from to smithi08477203-28 2024-09-20T22:46:36.497 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 28 v 578 (ObjNum 400 snap 101 seq_num 400) dirty exists 2024-09-20T22:46:36.497 INFO:tasks.rados.rados.0.smithi084.stdout:1172: finishing write tid 3 to smithi08477203-16 2024-09-20T22:46:36.497 INFO:tasks.rados.rados.0.smithi084.stdout:1172: finishing write tid 4 to smithi08477203-16 2024-09-20T22:46:36.498 INFO:tasks.rados.rados.0.smithi084.stdout:1172: finishing write tid 5 to smithi08477203-16 2024-09-20T22:46:36.498 INFO:tasks.rados.rados.0.smithi084.stdout:1172: oid 16 updating version 0 to 183 2024-09-20T22:46:36.498 INFO:tasks.rados.rados.0.smithi084.stdout:1172: oid 16 updating version 183 to 184 2024-09-20T22:46:36.498 INFO:tasks.rados.rados.0.smithi084.stdout:1172: oid 16 version 184 is already newer than 182 2024-09-20T22:46:36.498 INFO:tasks.rados.rados.0.smithi084.stdout:1172: oid 16 updating version 184 to 185 2024-09-20T22:46:36.498 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 16 v 185 (ObjNum 435 snap 112 seq_num 435) dirty exists 2024-09-20T22:46:36.498 INFO:tasks.rados.rados.0.smithi084.stdout:1172: left oid 16 (ObjNum 435 snap 112 seq_num 435) 2024-09-20T22:46:36.502 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:46:36.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:36 smithi154 ceph-mon[33264]: osdmap e315: 8 total, 8 up, 6 in 2024-09-20T22:46:36.524 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:36.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:36.525 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:46:36.565 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.6 dump_historic_ops 2024-09-20T22:46:36.583 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:36 smithi084 ceph-mon[40024]: osdmap e315: 8 total, 8 up, 6 in 2024-09-20T22:46:36.583 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:36 smithi084 ceph-mon[29774]: osdmap e315: 8 total, 8 up, 6 in 2024-09-20T22:46:36.626 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:46:36.670 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:36.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:36.671 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:46:36.771 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:46:36.793 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:36.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:36.794 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:46:36.851 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:46:36.895 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:46:36.910 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph tell osd.7 reset_purged_snaps_last 2024-09-20T22:46:36.917 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:36.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:36.918 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:46:37.018 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:46:37.039 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:37.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:37.040 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:46:37.141 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:46:37.162 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:37.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:37.163 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:46:37.244 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:46:37.249 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:46:37.264 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:46:37.289 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:37.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:37.290 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:46:37.293 INFO:tasks.rados.rados.0.smithi084.stdout:1169: done (4 left) 2024-09-20T22:46:37.293 INFO:tasks.rados.rados.0.smithi084.stdout:1170: done (3 left) 2024-09-20T22:46:37.293 INFO:tasks.rados.rados.0.smithi084.stdout:1171: done (2 left) 2024-09-20T22:46:37.293 INFO:tasks.rados.rados.0.smithi084.stdout:1172: done (1 left) 2024-09-20T22:46:37.294 INFO:tasks.rados.rados.0.smithi084.stdout:1173: done (0 left) 2024-09-20T22:46:37.294 INFO:tasks.rados.rados.0.smithi084.stdout:1174: copy_from oid 6 from oid 33 current snap is 113 2024-09-20T22:46:37.294 INFO:tasks.rados.rados.0.smithi084.stdout:1175: snap_remove snap 104 2024-09-20T22:46:37.331 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-20T22:46:37.331 INFO:teuthology.orchestra.run.smithi154.stdout: "size": 20, 2024-09-20T22:46:37.331 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 600, 2024-09-20T22:46:37.331 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [ 2024-09-20T22:46:37.331 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.331 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:336 7.7 7:e67788e2:::smithi08477203-31:head [copy-from ver 30] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-09-20T22:46:37.331 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:06.542956+0000", 2024-09-20T22:46:37.332 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 210.784983963, 2024-09-20T22:46:37.332 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.019630893999999999, 2024-09-20T22:46:37.332 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:46:37.332 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:46:37.332 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:46:37.332 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:46:37.332 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:46:37.332 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:46:37.332 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 336 2024-09-20T22:46:37.332 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.332 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:46:37.332 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.332 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:46:37.332 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:06.542956+0000", 2024-09-20T22:46:37.333 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:46:37.333 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.333 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.333 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:46:37.333 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:06.542956+0000", 2024-09-20T22:46:37.333 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:46:37.333 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.333 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.333 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:46:37.333 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:06.542957+0000", 2024-09-20T22:46:37.333 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2580000000000001e-06 2024-09-20T22:46:37.333 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.333 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.333 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:46:37.334 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:06.542967+0000", 2024-09-20T22:46:37.334 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0701e-05 2024-09-20T22:46:37.334 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.334 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.334 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:46:37.334 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:06.542987+0000", 2024-09-20T22:46:37.334 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9760000000000001e-05 2024-09-20T22:46:37.334 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.334 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.334 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:46:37.334 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:06.543002+0000", 2024-09-20T22:46:37.334 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4578e-05 2024-09-20T22:46:37.334 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.335 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.335 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:46:37.335 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:06.543070+0000", 2024-09-20T22:46:37.335 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.8263999999999996e-05 2024-09-20T22:46:37.335 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.335 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.335 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:46:37.335 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:06.543159+0000", 2024-09-20T22:46:37.335 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.9241999999999997e-05 2024-09-20T22:46:37.335 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.335 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.335 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 0,3", 2024-09-20T22:46:37.335 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:06.546618+0000", 2024-09-20T22:46:37.335 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0034581949999999998 2024-09-20T22:46:37.335 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.336 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.336 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:46:37.336 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:06.551199+0000", 2024-09-20T22:46:37.336 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0045811309999999996 2024-09-20T22:46:37.336 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.336 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.336 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:46:37.336 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:06.560992+0000", 2024-09-20T22:46:37.336 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0097937270000000003 2024-09-20T22:46:37.336 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.336 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.336 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:46:37.336 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:06.562419+0000", 2024-09-20T22:46:37.336 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0014269040000000001 2024-09-20T22:46:37.337 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.337 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.337 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:46:37.337 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:06.562515+0000", 2024-09-20T22:46:37.337 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.5544999999999994e-05 2024-09-20T22:46:37.337 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.337 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.337 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:46:37.337 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:06.562586+0000", 2024-09-20T22:46:37.337 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.1588999999999995e-05 2024-09-20T22:46:37.337 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:46:37.337 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:46:37.337 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:46:37.337 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.338 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.338 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:540 7.c 7:39b84aab:::smithi08477203-19:head [copy-from ver 26] snapc 13=[13,12,11,10,f,d] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-09-20T22:46:37.338 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:27.123339+0000", 2024-09-20T22:46:37.338 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 190.20460096100001, 2024-09-20T22:46:37.338 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.017326617999999998, 2024-09-20T22:46:37.338 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:46:37.338 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:46:37.338 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:46:37.338 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:46:37.338 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:46:37.338 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:46:37.338 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 540 2024-09-20T22:46:37.339 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.339 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:46:37.339 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.339 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:46:37.339 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.123339+0000", 2024-09-20T22:46:37.339 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:46:37.339 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.339 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.339 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:46:37.339 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.123339+0000", 2024-09-20T22:46:37.339 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:46:37.339 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.339 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.339 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:46:37.340 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.123341+0000", 2024-09-20T22:46:37.340 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.3099999999999999e-06 2024-09-20T22:46:37.340 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.340 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.340 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:46:37.340 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.123351+0000", 2024-09-20T22:46:37.340 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0185e-05 2024-09-20T22:46:37.340 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.340 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.340 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:46:37.340 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.123372+0000", 2024-09-20T22:46:37.340 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.143e-05 2024-09-20T22:46:37.340 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.340 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.341 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:46:37.341 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.123385+0000", 2024-09-20T22:46:37.341 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2279000000000001e-05 2024-09-20T22:46:37.341 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.341 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.341 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:46:37.341 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.123467+0000", 2024-09-20T22:46:37.341 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.2218999999999993e-05 2024-09-20T22:46:37.341 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.341 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.341 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op must wait for map", 2024-09-20T22:46:37.341 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.123474+0000", 2024-09-20T22:46:37.341 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.323e-06 2024-09-20T22:46:37.341 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.342 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.342 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:46:37.342 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.125408+0000", 2024-09-20T22:46:37.342 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001933883 2024-09-20T22:46:37.342 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.342 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.342 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:46:37.342 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.125464+0000", 2024-09-20T22:46:37.342 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.5649000000000002e-05 2024-09-20T22:46:37.342 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.342 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.342 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 3,7", 2024-09-20T22:46:37.342 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.128846+0000", 2024-09-20T22:46:37.342 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0033820999999999999 2024-09-20T22:46:37.343 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.343 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.343 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:46:37.343 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.134732+0000", 2024-09-20T22:46:37.343 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0058861310000000002 2024-09-20T22:46:37.343 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.343 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.343 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:46:37.343 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.137919+0000", 2024-09-20T22:46:37.343 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0031871080000000001 2024-09-20T22:46:37.343 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.343 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.343 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:46:37.343 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.140579+0000", 2024-09-20T22:46:37.343 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0026593979999999999 2024-09-20T22:46:37.344 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.344 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.344 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:46:37.344 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.140618+0000", 2024-09-20T22:46:37.344 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.9286000000000003e-05 2024-09-20T22:46:37.344 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.344 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.344 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:46:37.344 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.140665+0000", 2024-09-20T22:46:37.344 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.7317000000000001e-05 2024-09-20T22:46:37.344 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:46:37.344 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:46:37.344 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:46:37.344 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.344 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.345 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:654 7.7 7:ea720b85:::smithi08477203-26:head [append 0~0] snapc 17=[17,16,15,13,11,d] ondisk+write+known_if_redirected+supports_pool_eio e136)", 2024-09-20T22:46:37.345 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:35.126065+0000", 2024-09-20T22:46:37.345 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 182.201874033, 2024-09-20T22:46:37.345 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.069757472000000001, 2024-09-20T22:46:37.345 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:46:37.345 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:46:37.345 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:46:37.345 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:46:37.345 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:46:37.345 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:46:37.345 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 654 2024-09-20T22:46:37.345 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.345 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:46:37.346 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.346 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:46:37.346 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.126065+0000", 2024-09-20T22:46:37.346 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:46:37.346 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.346 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.346 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:46:37.346 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.126065+0000", 2024-09-20T22:46:37.346 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:46:37.346 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.346 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.346 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:46:37.346 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.126067+0000", 2024-09-20T22:46:37.346 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.6139999999999999e-06 2024-09-20T22:46:37.346 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.347 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.347 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:46:37.347 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.126074+0000", 2024-09-20T22:46:37.347 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.1509999999999998e-06 2024-09-20T22:46:37.347 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.347 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.347 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:46:37.347 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.126090+0000", 2024-09-20T22:46:37.347 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.6001000000000001e-05 2024-09-20T22:46:37.347 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.347 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.347 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:46:37.347 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.126098+0000", 2024-09-20T22:46:37.347 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.2379999999999997e-06 2024-09-20T22:46:37.348 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.348 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.348 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:46:37.348 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.126155+0000", 2024-09-20T22:46:37.348 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.6279000000000001e-05 2024-09-20T22:46:37.348 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.348 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.348 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op must wait for map", 2024-09-20T22:46:37.348 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.126160+0000", 2024-09-20T22:46:37.348 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.8110000000000002e-06 2024-09-20T22:46:37.348 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.348 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.348 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:46:37.349 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.193660+0000", 2024-09-20T22:46:37.349 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.067499997000000006 2024-09-20T22:46:37.349 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.349 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.349 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:46:37.349 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.193751+0000", 2024-09-20T22:46:37.349 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.1571999999999997e-05 2024-09-20T22:46:37.349 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.349 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.349 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 0,3", 2024-09-20T22:46:37.349 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.193941+0000", 2024-09-20T22:46:37.349 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00018952699999999999 2024-09-20T22:46:37.349 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.349 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.350 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:46:37.350 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.195649+0000", 2024-09-20T22:46:37.350 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001707966 2024-09-20T22:46:37.350 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.350 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.350 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:46:37.350 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.195707+0000", 2024-09-20T22:46:37.350 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.8649e-05 2024-09-20T22:46:37.350 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.350 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.350 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:46:37.350 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.195722+0000", 2024-09-20T22:46:37.350 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4309e-05 2024-09-20T22:46:37.350 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.351 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.351 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:46:37.351 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.195773+0000", 2024-09-20T22:46:37.351 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.1368e-05 2024-09-20T22:46:37.351 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.351 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.351 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:46:37.351 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.195823+0000", 2024-09-20T22:46:37.351 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.9990000000000001e-05 2024-09-20T22:46:37.351 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:46:37.351 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:46:37.351 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:46:37.351 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.351 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.352 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:655 7.7 7:ea720b85:::smithi08477203-26:head [rollback 16] snapc 17=[17,16,15,13,11,d] ondisk+write+known_if_redirected+supports_pool_eio e136)", 2024-09-20T22:46:37.352 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:35.126118+0000", 2024-09-20T22:46:37.352 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 182.20182177999999, 2024-09-20T22:46:37.352 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.069926453, 2024-09-20T22:46:37.352 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:46:37.352 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:46:37.352 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:46:37.352 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:46:37.352 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:46:37.352 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:46:37.352 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 655 2024-09-20T22:46:37.352 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.352 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:46:37.353 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.353 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:46:37.353 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.126118+0000", 2024-09-20T22:46:37.353 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:46:37.353 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.353 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.353 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:46:37.353 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.126118+0000", 2024-09-20T22:46:37.353 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:46:37.353 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.353 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.353 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:46:37.353 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.126118+0000", 2024-09-20T22:46:37.353 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.3000000000000001e-07 2024-09-20T22:46:37.354 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.354 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.354 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:46:37.354 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.126123+0000", 2024-09-20T22:46:37.354 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.7650000000000001e-06 2024-09-20T22:46:37.354 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.354 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.354 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:46:37.354 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.126141+0000", 2024-09-20T22:46:37.354 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.7538e-05 2024-09-20T22:46:37.354 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.354 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.354 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:46:37.354 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.126150+0000", 2024-09-20T22:46:37.355 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.3549999999999994e-06 2024-09-20T22:46:37.355 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.355 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.355 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:46:37.355 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.126204+0000", 2024-09-20T22:46:37.355 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.4289999999999997e-05 2024-09-20T22:46:37.355 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.355 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.355 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting_for_map not empty", 2024-09-20T22:46:37.355 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.126210+0000", 2024-09-20T22:46:37.355 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.361e-06 2024-09-20T22:46:37.355 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.355 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.356 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:46:37.356 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.194528+0000", 2024-09-20T22:46:37.356 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.068318913999999994 2024-09-20T22:46:37.356 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.356 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.356 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:46:37.356 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.194614+0000", 2024-09-20T22:46:37.356 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.5280999999999999e-05 2024-09-20T22:46:37.356 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.356 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.356 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 0,3", 2024-09-20T22:46:37.356 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.194759+0000", 2024-09-20T22:46:37.356 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000145678 2024-09-20T22:46:37.356 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.357 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.357 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:46:37.357 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.195836+0000", 2024-09-20T22:46:37.357 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001076474 2024-09-20T22:46:37.357 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.357 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.357 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:46:37.357 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.195884+0000", 2024-09-20T22:46:37.357 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.8421000000000003e-05 2024-09-20T22:46:37.357 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.357 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.357 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:46:37.357 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.195947+0000", 2024-09-20T22:46:37.357 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.2522999999999998e-05 2024-09-20T22:46:37.358 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.358 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.358 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:46:37.358 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.196003+0000", 2024-09-20T22:46:37.358 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.5911000000000001e-05 2024-09-20T22:46:37.358 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.358 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.358 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:46:37.358 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.196044+0000", 2024-09-20T22:46:37.358 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.1412000000000003e-05 2024-09-20T22:46:37.358 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:46:37.358 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:46:37.358 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:46:37.358 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.359 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.360 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:656 7.7 7:ea720b85:::smithi08477203-26:head [append 0~0] snapc 17=[17,16,15,13,11,d] ondisk+write+known_if_redirected+supports_pool_eio e136)", 2024-09-20T22:46:37.360 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:35.126159+0000", 2024-09-20T22:46:37.360 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 182.20178094600001, 2024-09-20T22:46:37.360 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.070101607999999996, 2024-09-20T22:46:37.360 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:46:37.360 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:46:37.360 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:46:37.360 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:46:37.361 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:46:37.361 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:46:37.361 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 656 2024-09-20T22:46:37.361 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.361 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:46:37.361 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.361 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:46:37.361 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.126159+0000", 2024-09-20T22:46:37.361 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:46:37.361 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.361 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.361 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:46:37.361 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.126159+0000", 2024-09-20T22:46:37.361 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:46:37.361 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.362 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.362 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:46:37.362 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.126159+0000", 2024-09-20T22:46:37.362 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.1500000000000005e-07 2024-09-20T22:46:37.362 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.362 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.362 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:46:37.362 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.126168+0000", 2024-09-20T22:46:37.362 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.4600000000000003e-06 2024-09-20T22:46:37.362 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.362 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.362 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:46:37.362 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.126173+0000", 2024-09-20T22:46:37.362 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.7089999999999999e-06 2024-09-20T22:46:37.363 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.363 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.363 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:46:37.363 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.126177+0000", 2024-09-20T22:46:37.363 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.9040000000000002e-06 2024-09-20T22:46:37.363 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.363 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.363 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:46:37.363 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.126232+0000", 2024-09-20T22:46:37.363 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.5337999999999998e-05 2024-09-20T22:46:37.363 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.363 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.363 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting_for_map not empty", 2024-09-20T22:46:37.363 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.126238+0000", 2024-09-20T22:46:37.364 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.2390000000000002e-06 2024-09-20T22:46:37.364 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.364 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.364 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:46:37.364 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.195116+0000", 2024-09-20T22:46:37.364 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.068878012000000002 2024-09-20T22:46:37.364 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.364 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.364 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:46:37.364 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.195197+0000", 2024-09-20T22:46:37.364 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.0964999999999995e-05 2024-09-20T22:46:37.364 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.364 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.364 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 0,3", 2024-09-20T22:46:37.364 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.195338+0000", 2024-09-20T22:46:37.365 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000141006 2024-09-20T22:46:37.365 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.365 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.365 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:46:37.365 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.195897+0000", 2024-09-20T22:46:37.365 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00055979999999999995 2024-09-20T22:46:37.365 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.365 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.365 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:46:37.365 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.196102+0000", 2024-09-20T22:46:37.365 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00020405099999999999 2024-09-20T22:46:37.365 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.365 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.365 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:46:37.366 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.196160+0000", 2024-09-20T22:46:37.366 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.8677000000000001e-05 2024-09-20T22:46:37.366 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.366 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.366 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:46:37.366 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.196214+0000", 2024-09-20T22:46:37.366 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.414e-05 2024-09-20T22:46:37.366 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.366 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.366 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:46:37.366 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.196260+0000", 2024-09-20T22:46:37.366 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.5791999999999998e-05 2024-09-20T22:46:37.366 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:46:37.366 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:46:37.366 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:46:37.367 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.367 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.367 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:726 7.19 7:9eac4560:::smithi08477203-30:head [copy-from ver 81] snapc 18=[17] ondisk+write+known_if_redirected+supports_pool_eio e143)", 2024-09-20T22:46:37.367 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:42.325306+0000", 2024-09-20T22:46:37.367 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 175.002633076, 2024-09-20T22:46:37.367 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.018367953999999999, 2024-09-20T22:46:37.367 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:46:37.367 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:46:37.367 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:46:37.367 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:46:37.367 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:46:37.367 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:46:37.367 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 726 2024-09-20T22:46:37.367 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.368 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:46:37.368 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.368 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:46:37.368 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.325306+0000", 2024-09-20T22:46:37.368 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:46:37.368 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.368 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.368 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:46:37.368 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.325306+0000", 2024-09-20T22:46:37.368 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:46:37.368 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.368 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.368 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:46:37.368 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.325307+0000", 2024-09-20T22:46:37.368 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.4400000000000001e-07 2024-09-20T22:46:37.369 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.369 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.369 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:46:37.369 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.325309+0000", 2024-09-20T22:46:37.369 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.5500000000000001e-06 2024-09-20T22:46:37.369 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.369 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.369 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:46:37.369 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.325316+0000", 2024-09-20T22:46:37.369 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.4640000000000003e-06 2024-09-20T22:46:37.369 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.369 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.369 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:46:37.369 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.325321+0000", 2024-09-20T22:46:37.370 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.977e-06 2024-09-20T22:46:37.370 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.370 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.370 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:46:37.370 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.325460+0000", 2024-09-20T22:46:37.370 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00013921199999999999 2024-09-20T22:46:37.370 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.370 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.370 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:46:37.370 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.325532+0000", 2024-09-20T22:46:37.370 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.2355999999999996e-05 2024-09-20T22:46:37.370 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.370 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.371 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 2,7", 2024-09-20T22:46:37.371 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.332251+0000", 2024-09-20T22:46:37.371 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0067184000000000002 2024-09-20T22:46:37.371 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.371 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.371 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:46:37.371 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.342590+0000", 2024-09-20T22:46:37.371 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.010339355 2024-09-20T22:46:37.371 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.371 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.371 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:46:37.371 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.342631+0000", 2024-09-20T22:46:37.371 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.1053999999999997e-05 2024-09-20T22:46:37.371 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.371 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.372 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:46:37.372 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.343558+0000", 2024-09-20T22:46:37.372 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00092629099999999999 2024-09-20T22:46:37.372 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.372 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.372 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:46:37.372 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.343620+0000", 2024-09-20T22:46:37.372 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.2836999999999995e-05 2024-09-20T22:46:37.372 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.372 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.372 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:46:37.372 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.343674+0000", 2024-09-20T22:46:37.372 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.4014000000000002e-05 2024-09-20T22:46:37.372 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:46:37.373 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:46:37.373 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:46:37.373 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.373 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.373 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:921 7.19 7:9eac4560:::smithi08477203-30:head [write 487436~547559,stat] snapc 1f=[] ondisk+write+known_if_redirected+supports_pool_eio e158)", 2024-09-20T22:46:37.373 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:57.424404+0000", 2024-09-20T22:46:37.373 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 159.90353523499999, 2024-09-20T22:46:37.373 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.018385357000000001, 2024-09-20T22:46:37.373 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:46:37.373 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:46:37.373 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:46:37.373 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:46:37.373 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:46:37.374 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:46:37.374 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 921 2024-09-20T22:46:37.374 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.374 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:46:37.374 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.374 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:46:37.374 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:57.424404+0000", 2024-09-20T22:46:37.374 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:46:37.374 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.374 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.374 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:46:37.374 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:57.424404+0000", 2024-09-20T22:46:37.374 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:46:37.374 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.375 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.375 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:46:37.375 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:57.424406+0000", 2024-09-20T22:46:37.375 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.6169999999999999e-06 2024-09-20T22:46:37.375 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.375 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.375 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:46:37.375 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:57.428411+0000", 2024-09-20T22:46:37.375 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0040051109999999996 2024-09-20T22:46:37.375 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.375 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.375 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:46:37.375 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:57.428423+0000", 2024-09-20T22:46:37.375 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1590000000000001e-05 2024-09-20T22:46:37.376 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.376 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.376 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:46:37.376 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:57.428432+0000", 2024-09-20T22:46:37.376 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.781e-06 2024-09-20T22:46:37.376 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.376 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.376 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:46:37.376 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:57.431029+0000", 2024-09-20T22:46:37.376 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0025962139999999999 2024-09-20T22:46:37.376 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.378 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.378 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:46:37.378 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:57.431170+0000", 2024-09-20T22:46:37.378 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00014111199999999999 2024-09-20T22:46:37.378 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.378 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.378 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 2,7", 2024-09-20T22:46:37.378 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:57.431332+0000", 2024-09-20T22:46:37.378 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00016258599999999999 2024-09-20T22:46:37.379 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.379 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.379 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:46:37.379 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:57.433472+0000", 2024-09-20T22:46:37.379 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0021397120000000002 2024-09-20T22:46:37.379 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.379 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.379 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:46:37.379 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:57.437960+0000", 2024-09-20T22:46:37.379 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0044878410000000002 2024-09-20T22:46:37.379 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.379 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.379 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:46:37.379 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:57.442690+0000", 2024-09-20T22:46:37.380 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0047299250000000003 2024-09-20T22:46:37.380 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.380 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.380 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:46:37.380 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:57.442743+0000", 2024-09-20T22:46:37.380 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.3368999999999997e-05 2024-09-20T22:46:37.380 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.380 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.380 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:46:37.380 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:57.442790+0000", 2024-09-20T22:46:37.380 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.6499000000000003e-05 2024-09-20T22:46:37.380 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:46:37.380 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:46:37.381 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:46:37.381 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.381 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.381 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:1017 7.19 7:9cc0b7e2:::smithi08477203-46:head [copy-from ver 99] snapc 25=[25,24,23,21] ondisk+write+known_if_redirected+supports_pool_eio e166)", 2024-09-20T22:46:37.381 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:44:05.130713+0000", 2024-09-20T22:46:37.381 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 152.19722639899999, 2024-09-20T22:46:37.381 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.021441722999999999, 2024-09-20T22:46:37.381 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:46:37.381 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:46:37.381 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:46:37.381 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:46:37.381 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:46:37.381 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:46:37.382 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 1017 2024-09-20T22:46:37.382 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.382 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:46:37.382 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.382 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:46:37.382 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:05.130713+0000", 2024-09-20T22:46:37.382 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:46:37.382 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.382 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.382 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:46:37.382 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:05.130713+0000", 2024-09-20T22:46:37.382 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:46:37.382 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.383 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.383 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:46:37.383 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:05.130715+0000", 2024-09-20T22:46:37.383 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.66e-06 2024-09-20T22:46:37.383 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.383 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.383 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:46:37.383 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:05.130721+0000", 2024-09-20T22:46:37.383 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.0430000000000004e-06 2024-09-20T22:46:37.383 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.383 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.383 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:46:37.383 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:05.130732+0000", 2024-09-20T22:46:37.383 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1494999999999999e-05 2024-09-20T22:46:37.384 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.384 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.384 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:46:37.384 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:05.130748+0000", 2024-09-20T22:46:37.384 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.5520999999999999e-05 2024-09-20T22:46:37.384 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.384 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.384 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:46:37.384 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:05.130827+0000", 2024-09-20T22:46:37.384 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.8990000000000001e-05 2024-09-20T22:46:37.384 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.384 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.384 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op must wait for map", 2024-09-20T22:46:37.384 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:05.130834+0000", 2024-09-20T22:46:37.384 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.097e-06 2024-09-20T22:46:37.385 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.385 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.385 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:46:37.385 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:05.133176+0000", 2024-09-20T22:46:37.385 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.002341704 2024-09-20T22:46:37.385 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.385 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.385 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:46:37.385 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:05.133285+0000", 2024-09-20T22:46:37.385 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000109473 2024-09-20T22:46:37.385 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.385 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.385 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 2,7", 2024-09-20T22:46:37.385 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:05.142516+0000", 2024-09-20T22:46:37.386 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0092311130000000009 2024-09-20T22:46:37.386 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.386 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.386 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:46:37.386 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:05.150724+0000", 2024-09-20T22:46:37.386 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0082083139999999995 2024-09-20T22:46:37.386 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.386 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.386 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:46:37.386 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:05.150825+0000", 2024-09-20T22:46:37.386 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000100431 2024-09-20T22:46:37.386 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.386 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.386 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:46:37.387 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:05.152061+0000", 2024-09-20T22:46:37.387 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001235912 2024-09-20T22:46:37.387 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.387 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.387 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:46:37.387 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:05.152108+0000", 2024-09-20T22:46:37.387 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.7068999999999999e-05 2024-09-20T22:46:37.387 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.387 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.387 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:46:37.387 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:05.152155+0000", 2024-09-20T22:46:37.387 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.6900999999999997e-05 2024-09-20T22:46:37.387 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:46:37.387 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:46:37.387 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:46:37.388 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.388 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.388 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:1054 7.12 7:4fa4b2b3:::smithi08477203-21:head [copy-from ver 67] snapc 25=[25,24,23,21] ondisk+write+known_if_redirected+supports_pool_eio e166)", 2024-09-20T22:46:37.388 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:44:05.634022+0000", 2024-09-20T22:46:37.388 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 151.693917121, 2024-09-20T22:46:37.388 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.019656218, 2024-09-20T22:46:37.388 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:46:37.388 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:46:37.388 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:46:37.388 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:46:37.388 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:46:37.388 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:46:37.388 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 1054 2024-09-20T22:46:37.388 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.389 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:46:37.389 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.389 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:46:37.389 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:05.634022+0000", 2024-09-20T22:46:37.389 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:46:37.389 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.389 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.389 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:46:37.389 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:05.634022+0000", 2024-09-20T22:46:37.389 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:46:37.389 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.389 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.389 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:46:37.389 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:05.634024+0000", 2024-09-20T22:46:37.390 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.013e-06 2024-09-20T22:46:37.390 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.390 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.390 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:46:37.390 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:05.634035+0000", 2024-09-20T22:46:37.390 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0644e-05 2024-09-20T22:46:37.390 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.390 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.390 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:46:37.390 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:05.634061+0000", 2024-09-20T22:46:37.390 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.6352000000000001e-05 2024-09-20T22:46:37.390 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.390 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.390 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:46:37.391 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:05.634081+0000", 2024-09-20T22:46:37.391 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9347e-05 2024-09-20T22:46:37.391 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.391 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.391 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:46:37.391 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:05.634257+0000", 2024-09-20T22:46:37.391 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000176365 2024-09-20T22:46:37.391 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.391 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.391 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:46:37.391 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:05.634379+0000", 2024-09-20T22:46:37.391 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000122173 2024-09-20T22:46:37.391 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.391 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.392 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 0,2", 2024-09-20T22:46:37.392 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:05.643905+0000", 2024-09-20T22:46:37.392 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0095258679999999998 2024-09-20T22:46:37.392 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.392 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.392 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:46:37.392 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:05.647750+0000", 2024-09-20T22:46:37.392 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.003844564 2024-09-20T22:46:37.392 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.392 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.392 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:46:37.392 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:05.650803+0000", 2024-09-20T22:46:37.392 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.003053114 2024-09-20T22:46:37.392 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.392 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.393 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:46:37.393 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:05.653610+0000", 2024-09-20T22:46:37.393 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0028071459999999999 2024-09-20T22:46:37.393 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.393 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.393 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:46:37.393 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:05.653645+0000", 2024-09-20T22:46:37.393 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.5231000000000002e-05 2024-09-20T22:46:37.393 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.393 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.393 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:46:37.394 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:05.653679+0000", 2024-09-20T22:46:37.394 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.3401000000000001e-05 2024-09-20T22:46:37.394 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:46:37.394 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:46:37.394 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:46:37.394 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.394 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.394 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:1420 7.19 7:9cc0b7e2:::smithi08477203-46:head [write 480733~537415,stat] snapc 2f=[2f] ondisk+write+known_if_redirected+supports_pool_eio e189)", 2024-09-20T22:46:37.394 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:44:28.143096+0000", 2024-09-20T22:46:37.394 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 129.184843602, 2024-09-20T22:46:37.394 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.021594787000000001, 2024-09-20T22:46:37.394 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:46:37.394 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:46:37.395 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:46:37.395 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:46:37.395 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:46:37.395 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:46:37.395 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 1420 2024-09-20T22:46:37.395 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.395 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:46:37.395 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.395 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:46:37.395 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:28.143096+0000", 2024-09-20T22:46:37.395 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:46:37.395 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.395 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.395 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:46:37.396 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:28.143096+0000", 2024-09-20T22:46:37.396 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:46:37.396 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.396 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.396 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:46:37.396 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:28.143098+0000", 2024-09-20T22:46:37.396 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.1840000000000002e-06 2024-09-20T22:46:37.396 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.396 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.396 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:46:37.396 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:28.143663+0000", 2024-09-20T22:46:37.396 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00056486000000000004 2024-09-20T22:46:37.396 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.396 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.397 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:46:37.397 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:28.143684+0000", 2024-09-20T22:46:37.397 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.0792e-05 2024-09-20T22:46:37.397 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.397 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.397 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:46:37.397 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:28.143697+0000", 2024-09-20T22:46:37.397 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.292e-05 2024-09-20T22:46:37.397 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.397 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.397 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:46:37.397 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:28.143788+0000", 2024-09-20T22:46:37.397 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.1218999999999995e-05 2024-09-20T22:46:37.398 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.398 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.398 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op must wait for map", 2024-09-20T22:46:37.398 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:28.143798+0000", 2024-09-20T22:46:37.398 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.7999999999999993e-06 2024-09-20T22:46:37.398 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.398 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.398 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:46:37.398 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:28.148428+0000", 2024-09-20T22:46:37.398 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0046303780000000001 2024-09-20T22:46:37.398 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.398 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.398 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:46:37.398 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:28.148611+0000", 2024-09-20T22:46:37.399 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00018319699999999999 2024-09-20T22:46:37.399 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.399 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.399 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 2,7", 2024-09-20T22:46:37.399 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:28.148807+0000", 2024-09-20T22:46:37.399 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00019560399999999999 2024-09-20T22:46:37.399 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.399 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.399 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:46:37.399 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:28.157814+0000", 2024-09-20T22:46:37.399 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0090068449999999994 2024-09-20T22:46:37.399 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.399 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.399 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:46:37.400 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:28.158746+0000", 2024-09-20T22:46:37.400 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000932076 2024-09-20T22:46:37.400 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.400 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.400 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:46:37.400 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:28.164580+0000", 2024-09-20T22:46:37.400 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0058337850000000002 2024-09-20T22:46:37.400 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.400 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.400 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:46:37.400 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:28.164638+0000", 2024-09-20T22:46:37.400 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.8100999999999998e-05 2024-09-20T22:46:37.400 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.400 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.400 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:46:37.401 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:28.164691+0000", 2024-09-20T22:46:37.401 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.3025999999999997e-05 2024-09-20T22:46:37.401 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:46:37.401 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:46:37.401 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:46:37.401 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.401 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.401 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:1525 7.c 7:3f865ef5:::smithi08477203-28:head [copy-from ver 127] snapc 33=[33] ondisk+write+known_if_redirected+supports_pool_eio e197)", 2024-09-20T22:46:37.401 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:44:36.198921+0000", 2024-09-20T22:46:37.401 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 121.129018698, 2024-09-20T22:46:37.401 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.028621220999999999, 2024-09-20T22:46:37.401 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:46:37.401 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:46:37.401 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:46:37.402 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:46:37.402 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:46:37.402 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:46:37.402 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 1525 2024-09-20T22:46:37.402 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.402 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:46:37.402 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.402 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:46:37.402 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.198921+0000", 2024-09-20T22:46:37.402 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:46:37.402 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.402 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.402 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:46:37.403 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.198921+0000", 2024-09-20T22:46:37.403 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:46:37.403 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.403 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.403 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:46:37.403 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.198923+0000", 2024-09-20T22:46:37.403 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.1090000000000001e-06 2024-09-20T22:46:37.403 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.403 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.403 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:46:37.403 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.198934+0000", 2024-09-20T22:46:37.403 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0837e-05 2024-09-20T22:46:37.403 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.403 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.403 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:46:37.404 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.198955+0000", 2024-09-20T22:46:37.404 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.0789e-05 2024-09-20T22:46:37.404 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.404 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.404 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:46:37.404 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.198966+0000", 2024-09-20T22:46:37.404 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1778e-05 2024-09-20T22:46:37.404 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.404 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.404 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:46:37.404 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.199046+0000", 2024-09-20T22:46:37.404 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.9567999999999994e-05 2024-09-20T22:46:37.404 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.405 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.405 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op must wait for map", 2024-09-20T22:46:37.405 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.199054+0000", 2024-09-20T22:46:37.405 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.9349999999999994e-06 2024-09-20T22:46:37.405 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.405 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.405 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:46:37.405 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.201428+0000", 2024-09-20T22:46:37.405 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0023744389999999999 2024-09-20T22:46:37.405 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.405 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.405 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:46:37.405 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.201527+0000", 2024-09-20T22:46:37.405 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.8554000000000005e-05 2024-09-20T22:46:37.406 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.406 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.406 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 3,7", 2024-09-20T22:46:37.406 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.215277+0000", 2024-09-20T22:46:37.406 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.013750326 2024-09-20T22:46:37.406 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.406 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.406 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:46:37.406 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.226629+0000", 2024-09-20T22:46:37.406 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.011351781999999999 2024-09-20T22:46:37.406 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.406 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.406 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:46:37.406 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.227366+0000", 2024-09-20T22:46:37.407 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00073709199999999996 2024-09-20T22:46:37.407 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.407 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.407 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:46:37.407 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.227450+0000", 2024-09-20T22:46:37.407 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.3902999999999999e-05 2024-09-20T22:46:37.407 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.407 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.407 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:46:37.407 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.227489+0000", 2024-09-20T22:46:37.407 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.9103000000000002e-05 2024-09-20T22:46:37.407 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.407 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.407 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:46:37.407 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.227542+0000", 2024-09-20T22:46:37.408 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.3006e-05 2024-09-20T22:46:37.408 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:46:37.408 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:46:37.408 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:46:37.408 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.408 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.408 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:1526 7.c 7:3f865ef5:::smithi08477203-28:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e197)", 2024-09-20T22:46:37.408 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:44:36.198993+0000", 2024-09-20T22:46:37.408 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 121.128946332, 2024-09-20T22:46:37.408 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.022582641000000001, 2024-09-20T22:46:37.408 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:46:37.408 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:46:37.408 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:46:37.408 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:46:37.409 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:46:37.409 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:46:37.409 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 1526 2024-09-20T22:46:37.409 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.409 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:46:37.409 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.409 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:46:37.409 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.198993+0000", 2024-09-20T22:46:37.409 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:46:37.409 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.409 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.409 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:46:37.409 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.198993+0000", 2024-09-20T22:46:37.409 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:46:37.409 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.410 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.410 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:46:37.410 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.198994+0000", 2024-09-20T22:46:37.410 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.2e-07 2024-09-20T22:46:37.410 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.410 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.410 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:46:37.410 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.198996+0000", 2024-09-20T22:46:37.410 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.6410000000000002e-06 2024-09-20T22:46:37.410 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.410 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.410 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:46:37.410 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.199005+0000", 2024-09-20T22:46:37.411 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.8079999999999994e-06 2024-09-20T22:46:37.411 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.411 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.411 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:46:37.411 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.199011+0000", 2024-09-20T22:46:37.411 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.7010000000000001e-06 2024-09-20T22:46:37.411 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.411 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.411 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:46:37.411 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.199112+0000", 2024-09-20T22:46:37.411 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000101335 2024-09-20T22:46:37.411 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.411 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.411 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting_for_map not empty", 2024-09-20T22:46:37.411 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.199117+0000", 2024-09-20T22:46:37.412 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.2979999999999999e-06 2024-09-20T22:46:37.412 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.412 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.412 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:46:37.412 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.201667+0000", 2024-09-20T22:46:37.412 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.002549922 2024-09-20T22:46:37.412 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.412 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.412 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for blocked object", 2024-09-20T22:46:37.412 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.201708+0000", 2024-09-20T22:46:37.412 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.0772999999999997e-05 2024-09-20T22:46:37.412 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.412 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.412 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:46:37.412 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.221443+0000", 2024-09-20T22:46:37.413 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.019735229 2024-09-20T22:46:37.413 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.413 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.413 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:46:37.413 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.221509+0000", 2024-09-20T22:46:37.413 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.5618999999999996e-05 2024-09-20T22:46:37.413 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.413 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.413 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:46:37.413 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.221576+0000", 2024-09-20T22:46:37.413 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.6795e-05 2024-09-20T22:46:37.413 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:46:37.413 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:46:37.413 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:46:37.414 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.414 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.414 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:1542 7.12 7:4b06cbed:::smithi08477203-49:head [copy-from ver 69] snapc 34=[34,33] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2024-09-20T22:46:37.414 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:44:37.209101+0000", 2024-09-20T22:46:37.414 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 120.118838289, 2024-09-20T22:46:37.414 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.019142659999999999, 2024-09-20T22:46:37.414 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:46:37.414 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:46:37.414 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:46:37.414 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:46:37.414 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:46:37.414 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:46:37.414 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 1542 2024-09-20T22:46:37.414 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.414 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:46:37.415 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.415 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:46:37.415 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:37.209101+0000", 2024-09-20T22:46:37.415 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:46:37.415 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.415 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.415 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:46:37.415 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:37.209101+0000", 2024-09-20T22:46:37.415 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:46:37.415 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.415 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.415 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:46:37.415 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:37.209103+0000", 2024-09-20T22:46:37.416 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.454e-06 2024-09-20T22:46:37.416 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.416 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.416 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:46:37.416 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:37.209111+0000", 2024-09-20T22:46:37.416 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.1089999999999998e-06 2024-09-20T22:46:37.416 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.416 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.416 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:46:37.416 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:37.209130+0000", 2024-09-20T22:46:37.416 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.8933e-05 2024-09-20T22:46:37.416 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.416 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.417 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:46:37.417 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:37.209142+0000", 2024-09-20T22:46:37.417 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1843e-05 2024-09-20T22:46:37.417 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.417 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.417 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:46:37.417 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:37.209255+0000", 2024-09-20T22:46:37.417 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000113204 2024-09-20T22:46:37.417 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.417 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.417 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op must wait for map", 2024-09-20T22:46:37.417 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:37.209266+0000", 2024-09-20T22:46:37.417 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1070000000000001e-05 2024-09-20T22:46:37.417 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.417 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.418 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:46:37.418 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:37.210962+0000", 2024-09-20T22:46:37.418 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001696442 2024-09-20T22:46:37.418 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.418 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.418 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:46:37.418 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:37.211044+0000", 2024-09-20T22:46:37.418 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.2007000000000003e-05 2024-09-20T22:46:37.418 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.418 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.418 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 0,2", 2024-09-20T22:46:37.418 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:37.216569+0000", 2024-09-20T22:46:37.418 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.005525126 2024-09-20T22:46:37.419 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.419 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.419 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:46:37.419 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:37.221473+0000", 2024-09-20T22:46:37.419 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0049038739999999999 2024-09-20T22:46:37.419 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.419 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.419 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:46:37.419 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:37.227857+0000", 2024-09-20T22:46:37.419 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0063841109999999996 2024-09-20T22:46:37.419 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.419 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.419 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:46:37.419 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:37.228154+0000", 2024-09-20T22:46:37.420 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00029620999999999998 2024-09-20T22:46:37.420 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.420 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.420 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:46:37.420 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:37.228193+0000", 2024-09-20T22:46:37.420 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.9097000000000002e-05 2024-09-20T22:46:37.420 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.420 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.420 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:46:37.420 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:37.228244+0000", 2024-09-20T22:46:37.420 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.1180000000000001e-05 2024-09-20T22:46:37.420 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:46:37.420 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:46:37.420 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:46:37.421 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.421 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.421 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:1652 7.c 7:31d0211f:::smithi08477203-25:head [copy-from ver 161] snapc 38=[38,37] ondisk+write+known_if_redirected+supports_pool_eio e206)", 2024-09-20T22:46:37.421 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:44:45.159981+0000", 2024-09-20T22:46:37.421 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 112.167958298, 2024-09-20T22:46:37.421 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.029481621, 2024-09-20T22:46:37.421 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:46:37.421 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:46:37.421 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:46:37.421 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:46:37.421 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:46:37.421 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:46:37.421 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 1652 2024-09-20T22:46:37.421 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.422 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:46:37.422 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.422 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:46:37.422 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:45.159981+0000", 2024-09-20T22:46:37.422 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:46:37.422 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.422 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.422 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:46:37.422 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:45.159981+0000", 2024-09-20T22:46:37.422 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:46:37.422 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.422 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.422 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:46:37.422 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:45.159982+0000", 2024-09-20T22:46:37.423 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.8999999999999995e-07 2024-09-20T22:46:37.423 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.423 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.423 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:46:37.423 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:45.159986+0000", 2024-09-20T22:46:37.423 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.732e-06 2024-09-20T22:46:37.423 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.423 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.423 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:46:37.423 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:45.159996+0000", 2024-09-20T22:46:37.423 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.9639999999999998e-06 2024-09-20T22:46:37.423 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.423 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.423 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:46:37.423 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:45.160001+0000", 2024-09-20T22:46:37.424 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.8010000000000003e-06 2024-09-20T22:46:37.424 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.424 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.424 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:46:37.424 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:45.165664+0000", 2024-09-20T22:46:37.424 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0056630869999999998 2024-09-20T22:46:37.424 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.424 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.424 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:46:37.424 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:45.165710+0000", 2024-09-20T22:46:37.424 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.6371e-05 2024-09-20T22:46:37.424 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.424 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.424 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 3,7", 2024-09-20T22:46:37.425 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:45.175729+0000", 2024-09-20T22:46:37.425 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.010018750999999999 2024-09-20T22:46:37.425 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.425 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.425 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:46:37.425 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:45.186279+0000", 2024-09-20T22:46:37.425 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.010550241 2024-09-20T22:46:37.425 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.425 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.425 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:46:37.425 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:45.186520+0000", 2024-09-20T22:46:37.425 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000241394 2024-09-20T22:46:37.425 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.425 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.426 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:46:37.426 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:45.189408+0000", 2024-09-20T22:46:37.426 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0028872849999999999 2024-09-20T22:46:37.426 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.426 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.426 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:46:37.426 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:45.189440+0000", 2024-09-20T22:46:37.426 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.2341000000000001e-05 2024-09-20T22:46:37.426 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.426 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.426 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:46:37.426 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:45.189463+0000", 2024-09-20T22:46:37.426 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.2764000000000001e-05 2024-09-20T22:46:37.426 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:46:37.427 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:46:37.427 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:46:37.427 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.427 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.427 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:1653 7.c 7:31d0211f:::smithi08477203-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e206)", 2024-09-20T22:46:37.427 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:44:45.160009+0000", 2024-09-20T22:46:37.427 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 112.16793010000001, 2024-09-20T22:46:37.427 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.020255434999999999, 2024-09-20T22:46:37.427 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:46:37.427 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:46:37.427 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:46:37.427 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:46:37.427 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:46:37.427 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:46:37.428 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 1653 2024-09-20T22:46:37.428 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.428 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:46:37.428 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.428 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:46:37.428 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:45.160009+0000", 2024-09-20T22:46:37.428 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:46:37.428 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.428 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.428 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:46:37.428 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:45.160009+0000", 2024-09-20T22:46:37.428 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:46:37.428 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.428 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.428 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:46:37.429 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:45.160010+0000", 2024-09-20T22:46:37.429 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.5100000000000001e-07 2024-09-20T22:46:37.429 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.429 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.429 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:46:37.429 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:45.160011+0000", 2024-09-20T22:46:37.429 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.547e-06 2024-09-20T22:46:37.429 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.429 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.429 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:46:37.429 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:45.160015+0000", 2024-09-20T22:46:37.429 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.3749999999999999e-06 2024-09-20T22:46:37.429 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.429 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.430 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:46:37.430 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:45.160018+0000", 2024-09-20T22:46:37.430 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.1250000000000001e-06 2024-09-20T22:46:37.430 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.430 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.430 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:46:37.431 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:37 smithi084 ceph-mon[40024]: pgmap v375: 168 pgs: 4 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 161 active+clean; 147 MiB data, 552 MiB used, 536 GiB / 536 GiB avail; 10 MiB/s rd, 2.5 MiB/s wr, 18 op/s 2024-09-20T22:46:37.431 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:37 smithi084 ceph-mon[40024]: osdmap e316: 8 total, 8 up, 6 in 2024-09-20T22:46:37.432 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:37 smithi084 ceph-mon[29774]: pgmap v375: 168 pgs: 4 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 161 active+clean; 147 MiB data, 552 MiB used, 536 GiB / 536 GiB avail; 10 MiB/s rd, 2.5 MiB/s wr, 18 op/s 2024-09-20T22:46:37.432 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:37 smithi084 ceph-mon[29774]: osdmap e316: 8 total, 8 up, 6 in 2024-09-20T22:46:37.432 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:46:37.432 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:45.165846+0000", 2024-09-20T22:46:37.432 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0058279270000000001 2024-09-20T22:46:37.432 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.432 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.432 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for blocked object", 2024-09-20T22:46:37.433 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:45.165873+0000", 2024-09-20T22:46:37.433 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.7059999999999998e-05 2024-09-20T22:46:37.433 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.433 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.433 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:46:37.433 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:45.180130+0000", 2024-09-20T22:46:37.433 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.014256998999999999 2024-09-20T22:46:37.433 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.433 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.433 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:46:37.433 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:45.180199+0000", 2024-09-20T22:46:37.433 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.9550999999999999e-05 2024-09-20T22:46:37.433 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.434 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.434 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:46:37.434 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:45.180265+0000", 2024-09-20T22:46:37.434 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.5500000000000006e-05 2024-09-20T22:46:37.434 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:46:37.434 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:46:37.434 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:46:37.434 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.434 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.434 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:1654 7.19 7:9970b578:::smithi08477203-15:head [copy-from ver 110] snapc 38=[38,37] ondisk+write+known_if_redirected+supports_pool_eio e206)", 2024-09-20T22:46:37.434 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:44:45.160022+0000", 2024-09-20T22:46:37.434 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 112.16791769300001, 2024-09-20T22:46:37.434 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.021661590000000001, 2024-09-20T22:46:37.434 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:46:37.435 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:46:37.435 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:46:37.435 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:46:37.435 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:46:37.435 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:46:37.435 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 1654 2024-09-20T22:46:37.435 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.435 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:46:37.435 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.435 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:46:37.435 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:45.160022+0000", 2024-09-20T22:46:37.435 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:46:37.435 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.436 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.436 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:46:37.436 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:45.160022+0000", 2024-09-20T22:46:37.436 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:46:37.436 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.436 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.436 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:46:37.436 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:45.160022+0000", 2024-09-20T22:46:37.436 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.4299999999999999e-07 2024-09-20T22:46:37.436 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.436 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.436 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:46:37.436 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:45.160024+0000", 2024-09-20T22:46:37.437 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.018e-06 2024-09-20T22:46:37.437 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.437 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.437 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:46:37.437 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:45.160027+0000", 2024-09-20T22:46:37.437 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.9100000000000001e-06 2024-09-20T22:46:37.437 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.437 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.437 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:46:37.437 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:45.160030+0000", 2024-09-20T22:46:37.437 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.7140000000000002e-06 2024-09-20T22:46:37.437 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.437 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.437 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:46:37.437 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:45.160078+0000", 2024-09-20T22:46:37.438 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.7933000000000003e-05 2024-09-20T22:46:37.438 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.438 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.438 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:46:37.438 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:45.160133+0000", 2024-09-20T22:46:37.438 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.5823999999999999e-05 2024-09-20T22:46:37.438 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.438 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.438 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 2,7", 2024-09-20T22:46:37.438 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:45.167724+0000", 2024-09-20T22:46:37.438 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0075904060000000001 2024-09-20T22:46:37.438 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.438 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.438 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:46:37.439 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:45.177349+0000", 2024-09-20T22:46:37.439 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0096249809999999995 2024-09-20T22:46:37.439 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.439 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.439 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:46:37.439 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:45.178986+0000", 2024-09-20T22:46:37.439 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00163722 2024-09-20T22:46:37.439 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.439 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.439 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:46:37.439 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:45.181613+0000", 2024-09-20T22:46:37.439 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.002626888 2024-09-20T22:46:37.439 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.439 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.440 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:46:37.440 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:45.181650+0000", 2024-09-20T22:46:37.440 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.7249000000000002e-05 2024-09-20T22:46:37.440 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.440 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.440 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:46:37.440 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:45.181683+0000", 2024-09-20T22:46:37.440 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.3204000000000003e-05 2024-09-20T22:46:37.440 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:46:37.440 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:46:37.440 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:46:37.440 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.440 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.441 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:2179 7.12 7:4fa4b2b3:::smithi08477203-21:head [write 595047~641755,stat] snapc 53=[53,52,51,50,4b,47,45,3e] ondisk+write+known_if_redirected+supports_pool_eio e254)", 2024-09-20T22:46:37.441 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:45:33.166066+0000", 2024-09-20T22:46:37.441 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 64.161873705000005, 2024-09-20T22:46:37.441 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.020018068999999999, 2024-09-20T22:46:37.441 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:46:37.441 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:46:37.441 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:46:37.441 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:46:37.441 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:46:37.441 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:46:37.441 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 2179 2024-09-20T22:46:37.441 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.441 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:46:37.441 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.442 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:46:37.442 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:33.166066+0000", 2024-09-20T22:46:37.442 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:46:37.442 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.442 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.442 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:46:37.442 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:33.166066+0000", 2024-09-20T22:46:37.442 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:46:37.442 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.442 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.442 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:46:37.442 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:33.166067+0000", 2024-09-20T22:46:37.442 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.6500000000000001e-06 2024-09-20T22:46:37.442 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.443 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.443 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:46:37.443 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:33.167423+0000", 2024-09-20T22:46:37.443 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0013558999999999999 2024-09-20T22:46:37.443 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.443 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.443 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:46:37.443 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:33.167433+0000", 2024-09-20T22:46:37.443 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.4790000000000002e-06 2024-09-20T22:46:37.443 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.443 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.443 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:46:37.443 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:33.167442+0000", 2024-09-20T22:46:37.443 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.1439999999999992e-06 2024-09-20T22:46:37.444 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.444 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.444 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:46:37.444 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:33.167627+0000", 2024-09-20T22:46:37.444 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00018459100000000001 2024-09-20T22:46:37.444 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.444 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.444 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:46:37.444 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:33.167690+0000", 2024-09-20T22:46:37.444 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.3806000000000005e-05 2024-09-20T22:46:37.444 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.444 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.444 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 0,2", 2024-09-20T22:46:37.444 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:33.167834+0000", 2024-09-20T22:46:37.445 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00014365600000000001 2024-09-20T22:46:37.445 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.445 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.445 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:46:37.445 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:33.169264+0000", 2024-09-20T22:46:37.445 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0014299040000000001 2024-09-20T22:46:37.445 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.445 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.445 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:46:37.445 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:33.175836+0000", 2024-09-20T22:46:37.445 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0065721750000000004 2024-09-20T22:46:37.445 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.445 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.445 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:46:37.446 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:33.185986+0000", 2024-09-20T22:46:37.446 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.010149696 2024-09-20T22:46:37.446 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.446 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.446 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:46:37.446 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:33.186040+0000", 2024-09-20T22:46:37.446 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.4601000000000001e-05 2024-09-20T22:46:37.446 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.446 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.446 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:46:37.446 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:33.186084+0000", 2024-09-20T22:46:37.446 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.3467000000000002e-05 2024-09-20T22:46:37.446 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:46:37.446 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:46:37.447 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:46:37.447 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.447 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.447 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:2184 7.19 7:9970b578:::smithi08477203-15:head [copy-from ver 166] snapc 53=[53,52,51,50,4b,47,45,3e] ondisk+write+known_if_redirected+supports_pool_eio e254)", 2024-09-20T22:46:37.447 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:45:33.179233+0000", 2024-09-20T22:46:37.447 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 64.148706133000005, 2024-09-20T22:46:37.447 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.017462780000000001, 2024-09-20T22:46:37.447 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:46:37.447 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:46:37.447 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:46:37.447 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:46:37.447 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:46:37.447 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:46:37.447 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 2184 2024-09-20T22:46:37.448 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.448 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:46:37.448 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.448 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:46:37.448 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:33.179233+0000", 2024-09-20T22:46:37.448 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:46:37.448 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.448 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.448 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:46:37.448 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:33.179233+0000", 2024-09-20T22:46:37.448 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:46:37.448 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.448 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.448 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:46:37.449 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:33.179234+0000", 2024-09-20T22:46:37.449 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.15e-07 2024-09-20T22:46:37.449 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.449 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.449 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:46:37.449 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:33.179236+0000", 2024-09-20T22:46:37.449 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.4660000000000002e-06 2024-09-20T22:46:37.449 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.449 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.449 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:46:37.449 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:33.179241+0000", 2024-09-20T22:46:37.449 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.8960000000000002e-06 2024-09-20T22:46:37.449 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.449 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.450 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:46:37.450 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:33.179245+0000", 2024-09-20T22:46:37.450 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.0300000000000004e-06 2024-09-20T22:46:37.450 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.450 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.450 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:46:37.450 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:33.179318+0000", 2024-09-20T22:46:37.450 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.2366999999999993e-05 2024-09-20T22:46:37.450 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.450 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.450 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:46:37.450 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:33.179388+0000", 2024-09-20T22:46:37.450 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.0335999999999999e-05 2024-09-20T22:46:37.451 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.451 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.451 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 2,7", 2024-09-20T22:46:37.451 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:33.187886+0000", 2024-09-20T22:46:37.451 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0084980839999999995 2024-09-20T22:46:37.451 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.451 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.451 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:46:37.451 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:33.194176+0000", 2024-09-20T22:46:37.451 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0062899690000000003 2024-09-20T22:46:37.451 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.451 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.451 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:46:37.451 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:33.196397+0000", 2024-09-20T22:46:37.451 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0022213710000000002 2024-09-20T22:46:37.452 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.452 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.452 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:46:37.452 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:33.196614+0000", 2024-09-20T22:46:37.452 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00021659399999999999 2024-09-20T22:46:37.452 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.452 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.452 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:46:37.452 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:33.196653+0000", 2024-09-20T22:46:37.452 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.9329999999999998e-05 2024-09-20T22:46:37.452 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.452 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.452 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:46:37.452 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:33.196696+0000", 2024-09-20T22:46:37.453 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.2922e-05 2024-09-20T22:46:37.453 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:46:37.453 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:46:37.453 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:46:37.453 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.453 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.453 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:2636 7.c 7:35bf785a:::smithi08477203-29:head [copy-from ver 260] snapc 64=[63,60] ondisk+write+known_if_redirected+supports_pool_eio e295)", 2024-09-20T22:46:37.453 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:46:14.189237+0000", 2024-09-20T22:46:37.453 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 23.138702351999999, 2024-09-20T22:46:37.453 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.01875288, 2024-09-20T22:46:37.453 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:46:37.453 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:46:37.453 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:46:37.453 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:46:37.454 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:46:37.454 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:46:37.454 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 2636 2024-09-20T22:46:37.454 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.454 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:46:37.454 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.454 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:46:37.454 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:14.189237+0000", 2024-09-20T22:46:37.454 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:46:37.454 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.454 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.454 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:46:37.454 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:14.189237+0000", 2024-09-20T22:46:37.454 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:46:37.455 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.455 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.455 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:46:37.455 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:14.189238+0000", 2024-09-20T22:46:37.455 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.8000000000000001e-07 2024-09-20T22:46:37.455 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.455 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.455 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:46:37.455 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:14.189239+0000", 2024-09-20T22:46:37.455 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.939e-06 2024-09-20T22:46:37.455 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.455 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.455 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:46:37.455 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:14.189243+0000", 2024-09-20T22:46:37.455 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.2540000000000001e-06 2024-09-20T22:46:37.456 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.456 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.456 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:46:37.456 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:14.189245+0000", 2024-09-20T22:46:37.456 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.6879999999999999e-06 2024-09-20T22:46:37.456 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.456 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.456 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:46:37.456 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:14.189288+0000", 2024-09-20T22:46:37.456 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.3024e-05 2024-09-20T22:46:37.456 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.456 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.456 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:46:37.456 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:14.189342+0000", 2024-09-20T22:46:37.457 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.3229000000000001e-05 2024-09-20T22:46:37.457 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.457 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.457 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 3,7", 2024-09-20T22:46:37.457 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:14.196998+0000", 2024-09-20T22:46:37.457 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0076564099999999998 2024-09-20T22:46:37.457 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.457 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.457 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:46:37.457 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:14.206729+0000", 2024-09-20T22:46:37.457 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0097304560000000002 2024-09-20T22:46:37.457 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.457 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.457 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:46:37.457 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:14.207550+0000", 2024-09-20T22:46:37.458 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00082114700000000004 2024-09-20T22:46:37.458 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.458 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.458 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:46:37.458 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:14.207871+0000", 2024-09-20T22:46:37.458 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00032147099999999999 2024-09-20T22:46:37.458 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.458 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.458 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:46:37.458 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:14.207939+0000", 2024-09-20T22:46:37.458 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.7900999999999999e-05 2024-09-20T22:46:37.458 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.458 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.458 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:46:37.459 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:14.207990+0000", 2024-09-20T22:46:37.459 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.0980999999999999e-05 2024-09-20T22:46:37.459 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:46:37.459 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:46:37.459 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:46:37.459 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.459 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.459 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:2996 7.19 7:9970b578:::smithi08477203-15:head [copy-from ver 382] snapc 6f=[6e,6c,6b,6a,68] ondisk+write+known_if_redirected+supports_pool_eio e314)", 2024-09-20T22:46:37.459 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:46:34.278928+0000", 2024-09-20T22:46:37.459 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 3.049011776, 2024-09-20T22:46:37.459 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.017309241, 2024-09-20T22:46:37.459 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:46:37.459 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:46:37.459 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:46:37.460 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:46:37.460 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:46:37.460 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:46:37.460 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 2996 2024-09-20T22:46:37.460 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.460 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:46:37.460 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.460 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:46:37.460 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:34.278928+0000", 2024-09-20T22:46:37.460 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:46:37.460 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.460 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.460 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:46:37.460 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:34.278928+0000", 2024-09-20T22:46:37.461 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:46:37.461 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.461 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.461 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:46:37.461 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:34.278930+0000", 2024-09-20T22:46:37.461 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9489999999999999e-06 2024-09-20T22:46:37.461 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.461 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.461 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:46:37.461 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:34.278936+0000", 2024-09-20T22:46:37.461 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.3969999999999999e-06 2024-09-20T22:46:37.461 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.461 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.461 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:46:37.461 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:34.278951+0000", 2024-09-20T22:46:37.462 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4612e-05 2024-09-20T22:46:37.462 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.462 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.462 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:46:37.462 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:34.278961+0000", 2024-09-20T22:46:37.462 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0556e-05 2024-09-20T22:46:37.462 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.462 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.462 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:46:37.462 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:34.279035+0000", 2024-09-20T22:46:37.462 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.3539000000000002e-05 2024-09-20T22:46:37.462 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.462 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.463 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op must wait for map", 2024-09-20T22:46:37.463 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:34.279044+0000", 2024-09-20T22:46:37.463 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.9309999999999997e-06 2024-09-20T22:46:37.463 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.463 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.463 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:46:37.463 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:34.282109+0000", 2024-09-20T22:46:37.463 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0030655919999999998 2024-09-20T22:46:37.463 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.463 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.463 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:46:37.463 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:34.282209+0000", 2024-09-20T22:46:37.463 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.9637999999999997e-05 2024-09-20T22:46:37.463 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.464 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.464 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 2,7", 2024-09-20T22:46:37.464 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:34.288474+0000", 2024-09-20T22:46:37.464 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0062651149999999999 2024-09-20T22:46:37.464 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.464 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.464 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:46:37.464 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:34.294267+0000", 2024-09-20T22:46:37.464 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0057930719999999998 2024-09-20T22:46:37.464 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.464 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.464 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:46:37.464 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:34.295923+0000", 2024-09-20T22:46:37.465 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001655608 2024-09-20T22:46:37.465 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.465 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.465 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:46:37.465 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:34.296170+0000", 2024-09-20T22:46:37.465 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00024765300000000002 2024-09-20T22:46:37.465 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.465 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.465 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:46:37.465 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:34.296207+0000", 2024-09-20T22:46:37.465 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.7073999999999998e-05 2024-09-20T22:46:37.465 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:46:37.465 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:46:37.465 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:46:37.466 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:34.296237+0000", 2024-09-20T22:46:37.466 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.9505e-05 2024-09-20T22:46:37.466 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:46:37.466 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:46:37.466 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:46:37.466 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:46:37.466 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:46:37.466 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-20T22:46:37.487 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:37.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:37.488 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:46:37.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:37 smithi154 ceph-mon[33264]: pgmap v375: 168 pgs: 4 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 161 active+clean; 147 MiB data, 552 MiB used, 536 GiB / 536 GiB avail; 10 MiB/s rd, 2.5 MiB/s wr, 18 op/s 2024-09-20T22:46:37.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:37 smithi154 ceph-mon[33264]: osdmap e316: 8 total, 8 up, 6 in 2024-09-20T22:46:37.589 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:46:37.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:37.611 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:37.611 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:46:37.712 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:46:37.735 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:37.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:37.736 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:46:37.837 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:46:37.858 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:37.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:37.859 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:46:37.924 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:46:37.925 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:46:37.940 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:46:37.940 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:46:37.956 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:46:37.956 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:46:37.959 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:46:37.973 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-20T22:46:37.973 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T22:46:37.980 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:37.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:37.981 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:46:37.990 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:46:37.990 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:46:38.006 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:46:38.007 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:46:38.023 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:46:38.023 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:46:38.039 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:46:38.039 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:46:38.082 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:46:38.104 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:38.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:38.105 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:46:38.169 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is set 2024-09-20T22:46:38.177 INFO:tasks.rados.rados.0.smithi084.stdout:1175: done (1 left) 2024-09-20T22:46:38.177 INFO:tasks.rados.rados.0.smithi084.stdout:1176: snap_remove snap 106 2024-09-20T22:46:38.205 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:46:38.226 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:38.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:38.227 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:46:38.310 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-20T22:46:38.328 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:46:38.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:38.375 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:38.375 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:46:38.476 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:46:38.498 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:38.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:38.499 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:46:38.558 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:46:38.600 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:46:38.614 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:46:38 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: ::ffff:172.21.15.154 - - [20/Sep/2024:22:46:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-20T22:46:38.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:38 smithi084 ceph-mon[40024]: purged_snaps scrub starts 2024-09-20T22:46:38.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:38 smithi084 ceph-mon[40024]: purged_snaps scrub ok 2024-09-20T22:46:38.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:38 smithi084 ceph-mon[40024]: osdmap e317: 8 total, 8 up, 6 in 2024-09-20T22:46:38.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:38 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/2852141822' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:46:38.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:38 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:46:38.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:38 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/1423222801' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:46:38.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:38 smithi084 ceph-mon[40024]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:46:38.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:38 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/2852141822' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T22:46:38.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:38 smithi084 ceph-mon[40024]: osdmap e318: 8 total, 8 up, 6 in 2024-09-20T22:46:38.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:38 smithi084 ceph-mon[29774]: purged_snaps scrub starts 2024-09-20T22:46:38.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:38 smithi084 ceph-mon[29774]: purged_snaps scrub ok 2024-09-20T22:46:38.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:38 smithi084 ceph-mon[29774]: osdmap e317: 8 total, 8 up, 6 in 2024-09-20T22:46:38.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:38 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/2852141822' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:46:38.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:38 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:46:38.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:38 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/1423222801' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:46:38.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:38 smithi084 ceph-mon[29774]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:46:38.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:38 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/2852141822' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T22:46:38.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:38 smithi084 ceph-mon[29774]: osdmap e318: 8 total, 8 up, 6 in 2024-09-20T22:46:38.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:38.624 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:38.624 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:46:38.680 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:38 smithi154 ceph-mon[33264]: purged_snaps scrub starts 2024-09-20T22:46:38.680 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:38 smithi154 ceph-mon[33264]: purged_snaps scrub ok 2024-09-20T22:46:38.680 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:38 smithi154 ceph-mon[33264]: osdmap e317: 8 total, 8 up, 6 in 2024-09-20T22:46:38.680 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:38 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/2852141822' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:46:38.680 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:38 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:46:38.680 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:38 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/1423222801' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:46:38.680 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:38 smithi154 ceph-mon[33264]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:46:38.680 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:38 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/2852141822' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T22:46:38.680 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:38 smithi154 ceph-mon[33264]: osdmap e318: 8 total, 8 up, 6 in 2024-09-20T22:46:38.725 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:46:38.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:38.747 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:38.747 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:46:38.848 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:46:38.869 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:38.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:38.870 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:46:38.971 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:46:38.992 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:38.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:38.993 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:46:39.029 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-20T22:46:39.029 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:46:39.095 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:46:39.117 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:39.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:39.119 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:46:39.178 INFO:tasks.rados.rados.0.smithi084.stdout:1176: done (1 left) 2024-09-20T22:46:39.178 INFO:tasks.rados.rados.0.smithi084.stdout:1177: read oid 20 snap -1 2024-09-20T22:46:39.178 INFO:tasks.rados.rados.0.smithi084.stdout:1177: expect (ObjNum 380 snap 100 seq_num 380) 2024-09-20T22:46:39.178 INFO:tasks.rados.rados.0.smithi084.stdout:1178: snap_remove snap 111 2024-09-20T22:46:39.219 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:46:39.242 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:39.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:39.243 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:46:39.254 INFO:journalctl@ceph.iscsi.iscsi.a.smithi154.stdout:Sep 20 22:46:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-iscsi-iscsi-a[59730]: debug there is no tcmu-runner data available 2024-09-20T22:46:39.343 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:46:39.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:39.366 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:39.366 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:46:39.467 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:46:39.489 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:39.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:39.490 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:46:39.528 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:39 smithi084 ceph-mon[29774]: pgmap v378: 168 pgs: 4 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 161 active+clean; 147 MiB data, 552 MiB used, 536 GiB / 536 GiB avail 2024-09-20T22:46:39.528 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:39 smithi084 ceph-mon[29774]: osdmap e319: 8 total, 8 up, 6 in 2024-09-20T22:46:39.528 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:39 smithi084 ceph-mon[40024]: pgmap v378: 168 pgs: 4 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 161 active+clean; 147 MiB data, 552 MiB used, 536 GiB / 536 GiB avail 2024-09-20T22:46:39.528 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:39 smithi084 ceph-mon[40024]: osdmap e319: 8 total, 8 up, 6 in 2024-09-20T22:46:39.591 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:46:39.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:39.614 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:39.614 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:46:39.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:39 smithi154 ceph-mon[33264]: pgmap v378: 168 pgs: 4 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 161 active+clean; 147 MiB data, 552 MiB used, 536 GiB / 536 GiB avail 2024-09-20T22:46:39.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:39 smithi154 ceph-mon[33264]: osdmap e319: 8 total, 8 up, 6 in 2024-09-20T22:46:39.669 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.7 dump_blocked_ops 2024-09-20T22:46:39.714 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:46:39.736 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:39.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:39.737 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:46:39.838 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:46:39.860 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:39.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:39.861 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:46:39.919 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:46:39.962 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:46:39.985 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:39.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:39.986 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:46:40.087 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:46:40.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:40.110 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:40.111 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:46:40.177 INFO:tasks.rados.rados.0.smithi084.stdout:1178: done (2 left) 2024-09-20T22:46:40.177 INFO:tasks.rados.rados.0.smithi084.stdout:1179: read oid 37 snap -1 2024-09-20T22:46:40.177 INFO:tasks.rados.rados.0.smithi084.stdout:1179: expect deleted 2024-09-20T22:46:40.177 INFO:tasks.rados.rados.0.smithi084.stdout:1180: delete oid 19 current snap is 113 2024-09-20T22:46:40.178 INFO:tasks.rados.rados.0.smithi084.stdout:1174: finishing copy_from racing read to smithi08477203-6 2024-09-20T22:46:40.178 INFO:tasks.rados.rados.0.smithi084.stdout:1174: finishing copy_from to smithi08477203-6 2024-09-20T22:46:40.178 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 6 v 216 (ObjNum 395 snap 101 seq_num 395) dirty exists 2024-09-20T22:46:40.178 INFO:tasks.rados.rados.0.smithi084.stdout:1177: expect (ObjNum 380 snap 100 seq_num 380) 2024-09-20T22:46:40.211 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:46:40.232 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:40.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:40.233 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:46:40.257 INFO:tasks.rados.rados.0.smithi084.stdout:1174: done (3 left) 2024-09-20T22:46:40.257 INFO:tasks.rados.rados.0.smithi084.stdout:1177: done (2 left) 2024-09-20T22:46:40.257 INFO:tasks.rados.rados.0.smithi084.stdout:1179: done (1 left) 2024-09-20T22:46:40.257 INFO:tasks.rados.rados.0.smithi084.stdout:1180: done (0 left) 2024-09-20T22:46:40.257 INFO:tasks.rados.rados.0.smithi084.stdout:1181: snap_remove snap 108 2024-09-20T22:46:40.334 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:46:40.355 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:40.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:40.356 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:46:40.382 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-20T22:46:40.382 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:46:40.457 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:46:40.477 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:40.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:40.478 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:46:40.579 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:46:40.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:40.601 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:40.602 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:46:40.614 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:40 smithi084 ceph-mon[29774]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:46:40.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:40 smithi084 ceph-mon[29774]: osdmap e320: 8 total, 8 up, 6 in 2024-09-20T22:46:40.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:40 smithi084 ceph-mon[40024]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:46:40.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:40 smithi084 ceph-mon[40024]: osdmap e320: 8 total, 8 up, 6 in 2024-09-20T22:46:40.694 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:40 smithi154 ceph-mon[33264]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:46:40.694 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:40 smithi154 ceph-mon[33264]: osdmap e320: 8 total, 8 up, 6 in 2024-09-20T22:46:40.703 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:46:40.724 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:40.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:40.725 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:46:40.826 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:46:40.848 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:40.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:40.849 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:46:40.857 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd unset noscrub 2024-09-20T22:46:40.950 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:46:40.972 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:40.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:40.973 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:46:41.074 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:46:41.091 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.7 dump_historic_ops 2024-09-20T22:46:41.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:41.096 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:41.096 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:46:41.136 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:46:41.177 INFO:tasks.rados.rados.0.smithi084.stdout:1181: done (0 left) 2024-09-20T22:46:41.177 INFO:tasks.rados.rados.0.smithi084.stdout:1182: snap_remove snap 102 2024-09-20T22:46:41.197 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:46:41.217 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:41.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:41.218 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:46:41.319 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:46:41.342 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:41.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:41.343 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:46:41.344 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:46:41.444 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:46:41.464 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:41 smithi154 ceph-mon[33264]: pgmap v381: 168 pgs: 4 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 160 active+clean; 158 MiB data, 597 MiB used, 536 GiB / 536 GiB avail; 4.7 MiB/s rd, 3.0 MiB/s wr, 13 op/s 2024-09-20T22:46:41.464 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:41 smithi154 ceph-mon[33264]: osdmap e321: 8 total, 8 up, 6 in 2024-09-20T22:46:41.485 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:41.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:41.486 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:46:41.584 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:41 smithi084 ceph-mon[40024]: pgmap v381: 168 pgs: 4 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 160 active+clean; 158 MiB data, 597 MiB used, 536 GiB / 536 GiB avail; 4.7 MiB/s rd, 3.0 MiB/s wr, 13 op/s 2024-09-20T22:46:41.584 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:41 smithi084 ceph-mon[40024]: osdmap e321: 8 total, 8 up, 6 in 2024-09-20T22:46:41.584 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:41 smithi084 ceph-mon[29774]: pgmap v381: 168 pgs: 4 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 160 active+clean; 158 MiB data, 597 MiB used, 536 GiB / 536 GiB avail; 4.7 MiB/s rd, 3.0 MiB/s wr, 13 op/s 2024-09-20T22:46:41.584 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:41 smithi084 ceph-mon[29774]: osdmap e321: 8 total, 8 up, 6 in 2024-09-20T22:46:41.586 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:46:41.610 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:41.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:41.611 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:46:41.712 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:46:41.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:41.734 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:41.734 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:46:41.798 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-20T22:46:41.798 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:46:41.835 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:46:41.855 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:41.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:41.856 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:46:41.957 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:46:41.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:41.980 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:41.980 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:46:42.003 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-20T22:46:42.080 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:46:42.102 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:42.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:42.102 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:46:42.175 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is unset 2024-09-20T22:46:42.180 INFO:tasks.rados.rados.0.smithi084.stdout:1182: done (0 left) 2024-09-20T22:46:42.180 INFO:tasks.rados.rados.0.smithi084.stdout:1183: read oid 34 snap -1 2024-09-20T22:46:42.180 INFO:tasks.rados.rados.0.smithi084.stdout:1183: expect (ObjNum 359 snap 92 seq_num 359) 2024-09-20T22:46:42.180 INFO:tasks.rados.rados.0.smithi084.stdout:1184: copy_from oid 39 from oid 38 current snap is 113 2024-09-20T22:46:42.180 INFO:tasks.rados.rados.0.smithi084.stdout:1185: read oid 29 snap -1 2024-09-20T22:46:42.181 INFO:tasks.rados.rados.0.smithi084.stdout:1185: expect (ObjNum 400 snap 101 seq_num 400) 2024-09-20T22:46:42.181 INFO:tasks.rados.rados.0.smithi084.stdout:1186: snap_create 2024-09-20T22:46:42.195 INFO:tasks.rados.rados.0.smithi084.stdout:1185: expect (ObjNum 400 snap 101 seq_num 400) 2024-09-20T22:46:42.203 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:46:42.225 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:42.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:42.226 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:46:42.327 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:46:42.348 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:42.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:42.349 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:46:42.410 INFO:tasks.rados.rados.0.smithi084.stdout:1184: finishing copy_from racing read to smithi08477203-39 2024-09-20T22:46:42.410 INFO:tasks.rados.rados.0.smithi084.stdout:1183: expect (ObjNum 359 snap 92 seq_num 359) 2024-09-20T22:46:42.450 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:46:42.473 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:42.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:42.474 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:46:42.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:42 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/4144978452' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:46:42.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:42 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/4144978452' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T22:46:42.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:42 smithi154 ceph-mon[33264]: osdmap e322: 8 total, 8 up, 6 in 2024-09-20T22:46:42.509 INFO:tasks.rados.rados.0.smithi084.stdout:1184: finishing copy_from to smithi08477203-39 2024-09-20T22:46:42.509 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 39 v 385 (ObjNum 397 snap 101 seq_num 397) dirty exists 2024-09-20T22:46:42.532 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-20T22:46:42.571 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:42 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/4144978452' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:46:42.571 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:42 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/4144978452' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T22:46:42.571 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:42 smithi084 ceph-mon[40024]: osdmap e322: 8 total, 8 up, 6 in 2024-09-20T22:46:42.571 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:42 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/4144978452' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:46:42.571 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:42 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/4144978452' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T22:46:42.572 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:42 smithi084 ceph-mon[29774]: osdmap e322: 8 total, 8 up, 6 in 2024-09-20T22:46:42.575 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:46:42.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:42.597 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:42.598 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:46:42.699 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:46:42.720 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:42.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:42.721 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:46:42.821 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:46:42.844 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:42.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:42.845 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:46:42.908 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:46:42.929 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:46:42.946 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:46:42.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:42.968 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:42.969 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:46:43.069 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:46:43.090 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:43.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:43.091 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:46:43.177 INFO:tasks.rados.rados.0.smithi084.stdout:1183: done (3 left) 2024-09-20T22:46:43.177 INFO:tasks.rados.rados.0.smithi084.stdout:1184: done (2 left) 2024-09-20T22:46:43.178 INFO:tasks.rados.rados.0.smithi084.stdout:1185: done (1 left) 2024-09-20T22:46:43.178 INFO:tasks.rados.rados.0.smithi084.stdout:1186: done (0 left) 2024-09-20T22:46:43.178 INFO:tasks.rados.rados.0.smithi084.stdout:1187: snap_create 2024-09-20T22:46:43.192 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:46:43.214 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:43.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:43.215 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:46:43.315 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:46:43.337 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:43.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:43.339 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:46:43.424 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-20T22:46:43.424 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:46:43.440 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:46:43.472 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:43.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:43.473 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:46:43.574 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:46:43.595 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:43.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:43.596 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:46:43.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:43 smithi084 ceph-mon[29774]: pgmap v384: 168 pgs: 4 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 160 active+clean; 158 MiB data, 597 MiB used, 536 GiB / 536 GiB avail; 4.7 MiB/s rd, 3.0 MiB/s wr, 13 op/s 2024-09-20T22:46:43.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:43 smithi084 ceph-mon[29774]: 3.1f scrub starts 2024-09-20T22:46:43.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:43 smithi084 ceph-mon[29774]: 3.1f scrub ok 2024-09-20T22:46:43.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:43 smithi084 ceph-mon[29774]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:46:43.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:43 smithi084 ceph-mon[29774]: osdmap e323: 8 total, 8 up, 6 in 2024-09-20T22:46:43.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:43 smithi084 ceph-mon[40024]: pgmap v384: 168 pgs: 4 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 160 active+clean; 158 MiB data, 597 MiB used, 536 GiB / 536 GiB avail; 4.7 MiB/s rd, 3.0 MiB/s wr, 13 op/s 2024-09-20T22:46:43.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:43 smithi084 ceph-mon[40024]: 3.1f scrub starts 2024-09-20T22:46:43.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:43 smithi084 ceph-mon[40024]: 3.1f scrub ok 2024-09-20T22:46:43.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:43 smithi084 ceph-mon[40024]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:46:43.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:43 smithi084 ceph-mon[40024]: osdmap e323: 8 total, 8 up, 6 in 2024-09-20T22:46:43.696 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:46:43.718 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:43.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:43.719 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:46:43.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:43 smithi154 ceph-mon[33264]: pgmap v384: 168 pgs: 4 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 160 active+clean; 158 MiB data, 597 MiB used, 536 GiB / 536 GiB avail; 4.7 MiB/s rd, 3.0 MiB/s wr, 13 op/s 2024-09-20T22:46:43.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:43 smithi154 ceph-mon[33264]: 3.1f scrub starts 2024-09-20T22:46:43.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:43 smithi154 ceph-mon[33264]: 3.1f scrub ok 2024-09-20T22:46:43.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:43 smithi154 ceph-mon[33264]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:46:43.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:43 smithi154 ceph-mon[33264]: osdmap e323: 8 total, 8 up, 6 in 2024-09-20T22:46:43.820 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:46:43.841 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:43.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:43.842 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:46:43.943 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:46:43.964 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:43.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:43.965 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:46:44.043 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.0 dump_blocked_ops 2024-09-20T22:46:44.065 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:46:44.107 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:44.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:44.108 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:46:44.182 INFO:tasks.rados.rados.0.smithi084.stdout:1187: done (0 left) 2024-09-20T22:46:44.182 INFO:tasks.rados.rados.0.smithi084.stdout:1188: rollback oid 32 current snap is 115 2024-09-20T22:46:44.182 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 32 to 113 2024-09-20T22:46:44.182 INFO:tasks.rados.rados.0.smithi084.stdout:1189: read oid 19 snap -1 2024-09-20T22:46:44.182 INFO:tasks.rados.rados.0.smithi084.stdout:1189: expect deleted 2024-09-20T22:46:44.182 INFO:tasks.rados.rados.0.smithi084.stdout:1190: copy_from oid 10 from oid 11 current snap is 115 2024-09-20T22:46:44.183 INFO:tasks.rados.rados.0.smithi084.stdout:1191: snap_remove snap 112 2024-09-20T22:46:44.209 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:46:44.235 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:44.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:44.236 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:46:44.337 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:46:44.359 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:44.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:44.360 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:46:44.461 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:46:44.482 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:44.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:44.483 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:46:44.530 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 5, 6, 7] out_osds: [4, 0] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-20T22:46:44.530 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-20T22:46:44.530 INFO:tasks.thrashosds.thrasher:reset_purged_snaps_last 2024-09-20T22:46:44.531 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph tell osd.1 reset_purged_snaps_last 2024-09-20T22:46:44.567 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:46:44.567 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:46:44.584 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:46:44.597 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:46:44.598 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:44 smithi084 ceph-mon[29774]: 5.2 scrub starts 2024-09-20T22:46:44.598 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:44 smithi084 ceph-mon[29774]: 5.2 scrub ok 2024-09-20T22:46:44.598 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:44 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:46:44.598 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:44 smithi084 ceph-mon[29774]: osdmap e324: 8 total, 8 up, 6 in 2024-09-20T22:46:44.599 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:46:44.599 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:46:44.599 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:44 smithi084 ceph-mon[40024]: 5.2 scrub starts 2024-09-20T22:46:44.599 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:44 smithi084 ceph-mon[40024]: 5.2 scrub ok 2024-09-20T22:46:44.599 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:44 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:46:44.599 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:44 smithi084 ceph-mon[40024]: osdmap e324: 8 total, 8 up, 6 in 2024-09-20T22:46:44.610 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:44.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:44.611 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:46:44.619 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:46:44.619 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:46:44.646 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-20T22:46:44.647 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T22:46:44.663 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:46:44.663 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:46:44.680 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:46:44.680 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:46:44.697 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:46:44.697 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:46:44.711 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:46:44.714 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:46:44.714 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:46:44.732 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:44.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:44.733 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:46:44.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:44 smithi154 ceph-mon[33264]: 5.2 scrub starts 2024-09-20T22:46:44.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:44 smithi154 ceph-mon[33264]: 5.2 scrub ok 2024-09-20T22:46:44.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:44 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:46:44.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:44 smithi154 ceph-mon[33264]: osdmap e324: 8 total, 8 up, 6 in 2024-09-20T22:46:44.834 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:46:44.859 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:44.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:44.860 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:46:44.861 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-20T22:46:44.961 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:46:44.982 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:44.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:44.983 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:46:45.084 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:46:45.105 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:45.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:45.106 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:46:45.183 INFO:tasks.rados.rados.0.smithi084.stdout:1191: done (3 left) 2024-09-20T22:46:45.184 INFO:tasks.rados.rados.0.smithi084.stdout:1192: write oid 12 current snap is 115 2024-09-20T22:46:45.184 INFO:tasks.rados.rados.0.smithi084.stdout:1190: finishing copy_from to smithi08477203-10 2024-09-20T22:46:45.184 INFO:tasks.rados.rados.0.smithi084.stdout:1190: got expected ENOENT (src dne) 2024-09-20T22:46:45.184 INFO:tasks.rados.rados.0.smithi084.stdout:1188: finishing rollback tid 0 to smithi08477203-32 2024-09-20T22:46:45.184 INFO:tasks.rados.rados.0.smithi084.stdout:1190: finishing copy_from racing read to smithi08477203-10 2024-09-20T22:46:45.184 INFO:tasks.rados.rados.0.smithi084.stdout:1188: finishing rollback tid 1 to smithi08477203-32 2024-09-20T22:46:45.184 INFO:tasks.rados.rados.0.smithi084.stdout:1188: finishing rollback tid 2 to smithi08477203-32 2024-09-20T22:46:45.184 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 32 v 388 (ObjNum 401 snap 101 seq_num 401) dirty exists 2024-09-20T22:46:45.184 INFO:tasks.rados.rados.0.smithi084.stdout:1192: seq_num 437 ranges {544659=715934,1738984=449146,2784436=560223} 2024-09-20T22:46:45.191 INFO:tasks.rados.rados.0.smithi084.stdout:1192: writing smithi08477203-12 from 544659 to 1260593 tid 1 2024-09-20T22:46:45.196 INFO:tasks.rados.rados.0.smithi084.stdout:1192: writing smithi08477203-12 from 1738984 to 2188130 tid 2 2024-09-20T22:46:45.203 INFO:tasks.rados.rados.0.smithi084.stdout:1192: writing smithi08477203-12 from 2784436 to 3344659 tid 3 2024-09-20T22:46:45.203 INFO:tasks.rados.rados.0.smithi084.stdout:1188: done (3 left) 2024-09-20T22:46:45.203 INFO:tasks.rados.rados.0.smithi084.stdout:1189: done (2 left) 2024-09-20T22:46:45.203 INFO:tasks.rados.rados.0.smithi084.stdout:1190: done (1 left) 2024-09-20T22:46:45.203 INFO:tasks.rados.rados.0.smithi084.stdout:1193: read oid 43 snap 110 2024-09-20T22:46:45.203 INFO:tasks.rados.rados.0.smithi084.stdout:1193: expect (ObjNum 418 snap 105 seq_num 418) 2024-09-20T22:46:45.203 INFO:tasks.rados.rados.0.smithi084.stdout:1194: write oid 8 current snap is 115 2024-09-20T22:46:45.203 INFO:tasks.rados.rados.0.smithi084.stdout:1192: finishing write tid 1 to smithi08477203-12 2024-09-20T22:46:45.204 INFO:tasks.rados.rados.0.smithi084.stdout:1192: finishing write tid 2 to smithi08477203-12 2024-09-20T22:46:45.204 INFO:tasks.rados.rados.0.smithi084.stdout:1194: seq_num 438 ranges {561466=791183,2081113=592804,3095506=265960} 2024-09-20T22:46:45.207 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:46:45.210 INFO:tasks.rados.rados.0.smithi084.stdout:1194: writing smithi08477203-8 from 561466 to 1352649 tid 1 2024-09-20T22:46:45.217 INFO:tasks.rados.rados.0.smithi084.stdout:1194: writing smithi08477203-8 from 2081113 to 2673917 tid 2 2024-09-20T22:46:45.221 INFO:tasks.rados.rados.0.smithi084.stdout:1194: writing smithi08477203-8 from 3095506 to 3361466 tid 3 2024-09-20T22:46:45.221 INFO:tasks.rados.rados.0.smithi084.stdout:1195: copy_from oid 46 from oid 9 current snap is 115 2024-09-20T22:46:45.221 INFO:tasks.rados.rados.0.smithi084.stdout:1193: expect (ObjNum 418 snap 105 seq_num 418) 2024-09-20T22:46:45.228 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:45.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:45.229 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:46:45.322 INFO:tasks.rados.rados.0.smithi084.stdout:1192: finishing write tid 3 to smithi08477203-12 2024-09-20T22:46:45.322 INFO:tasks.rados.rados.0.smithi084.stdout:1192: finishing write tid 4 to smithi08477203-12 2024-09-20T22:46:45.322 INFO:tasks.rados.rados.0.smithi084.stdout:1192: finishing write tid 5 to smithi08477203-12 2024-09-20T22:46:45.323 INFO:tasks.rados.rados.0.smithi084.stdout:1192: oid 12 updating version 0 to 276 2024-09-20T22:46:45.323 INFO:tasks.rados.rados.0.smithi084.stdout:1192: oid 12 version 276 is already newer than 275 2024-09-20T22:46:45.323 INFO:tasks.rados.rados.0.smithi084.stdout:1192: oid 12 version 276 is already newer than 273 2024-09-20T22:46:45.323 INFO:tasks.rados.rados.0.smithi084.stdout:1192: oid 12 version 276 is already newer than 274 2024-09-20T22:46:45.323 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 12 v 276 (ObjNum 437 snap 115 seq_num 437) dirty exists 2024-09-20T22:46:45.323 INFO:tasks.rados.rados.0.smithi084.stdout:1192: left oid 12 (ObjNum 437 snap 115 seq_num 437) 2024-09-20T22:46:45.323 INFO:tasks.rados.rados.0.smithi084.stdout:1194: finishing write tid 1 to smithi08477203-8 2024-09-20T22:46:45.323 INFO:tasks.rados.rados.0.smithi084.stdout:1192: done (3 left) 2024-09-20T22:46:45.323 INFO:tasks.rados.rados.0.smithi084.stdout:1193: done (2 left) 2024-09-20T22:46:45.323 INFO:tasks.rados.rados.0.smithi084.stdout:1196: delete oid 49 current snap is 115 2024-09-20T22:46:45.323 INFO:tasks.rados.rados.0.smithi084.stdout:1194: finishing write tid 2 to smithi08477203-8 2024-09-20T22:46:45.323 INFO:tasks.rados.rados.0.smithi084.stdout:1195: finishing copy_from to smithi08477203-46 2024-09-20T22:46:45.323 INFO:tasks.rados.rados.0.smithi084.stdout:1195: got expected ENOENT (src dne) 2024-09-20T22:46:45.323 INFO:tasks.rados.rados.0.smithi084.stdout:1195: finishing copy_from racing read to smithi08477203-46 2024-09-20T22:46:45.324 INFO:tasks.rados.rados.0.smithi084.stdout:1194: finishing write tid 3 to smithi08477203-8 2024-09-20T22:46:45.324 INFO:tasks.rados.rados.0.smithi084.stdout:1194: finishing write tid 4 to smithi08477203-8 2024-09-20T22:46:45.324 INFO:tasks.rados.rados.0.smithi084.stdout:1194: finishing write tid 5 to smithi08477203-8 2024-09-20T22:46:45.324 INFO:tasks.rados.rados.0.smithi084.stdout:1194: oid 8 updating version 0 to 184 2024-09-20T22:46:45.324 INFO:tasks.rados.rados.0.smithi084.stdout:1194: oid 8 updating version 184 to 187 2024-09-20T22:46:45.324 INFO:tasks.rados.rados.0.smithi084.stdout:1194: oid 8 version 187 is already newer than 185 2024-09-20T22:46:45.324 INFO:tasks.rados.rados.0.smithi084.stdout:1194: oid 8 version 187 is already newer than 186 2024-09-20T22:46:45.324 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 8 v 187 (ObjNum 438 snap 115 seq_num 438) dirty exists 2024-09-20T22:46:45.324 INFO:tasks.rados.rados.0.smithi084.stdout:1194: left oid 8 (ObjNum 438 snap 115 seq_num 438) 2024-09-20T22:46:45.325 INFO:tasks.rados.rados.0.smithi084.stdout:1194: done (2 left) 2024-09-20T22:46:45.325 INFO:tasks.rados.rados.0.smithi084.stdout:1195: done (1 left) 2024-09-20T22:46:45.325 INFO:tasks.rados.rados.0.smithi084.stdout:1196: done (0 left) 2024-09-20T22:46:45.325 INFO:tasks.rados.rados.0.smithi084.stdout:1197: write oid 41 current snap is 115 2024-09-20T22:46:45.325 INFO:tasks.rados.rados.0.smithi084.stdout:1197: seq_num 440 ranges {595080=541681,1798077=596473,3150352=244728} 2024-09-20T22:46:45.330 INFO:tasks.rados.rados.0.smithi084.stdout:1197: writing smithi08477203-41 from 595080 to 1136761 tid 1 2024-09-20T22:46:45.330 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:46:45.337 INFO:tasks.rados.rados.0.smithi084.stdout:1197: writing smithi08477203-41 from 1798077 to 2394550 tid 2 2024-09-20T22:46:45.343 INFO:tasks.rados.rados.0.smithi084.stdout:1197: writing smithi08477203-41 from 3150352 to 3395080 tid 3 2024-09-20T22:46:45.343 INFO:tasks.rados.rados.0.smithi084.stdout:1198: snap_create 2024-09-20T22:46:45.343 INFO:tasks.rados.rados.0.smithi084.stdout:1197: finishing write tid 1 to smithi08477203-41 2024-09-20T22:46:45.343 INFO:tasks.rados.rados.0.smithi084.stdout:1197: finishing write tid 2 to smithi08477203-41 2024-09-20T22:46:45.345 INFO:tasks.rados.rados.0.smithi084.stdout:1197: finishing write tid 3 to smithi08477203-41 2024-09-20T22:46:45.345 INFO:tasks.rados.rados.0.smithi084.stdout:1197: finishing write tid 4 to smithi08477203-41 2024-09-20T22:46:45.345 INFO:tasks.rados.rados.0.smithi084.stdout:1197: finishing write tid 5 to smithi08477203-41 2024-09-20T22:46:45.345 INFO:tasks.rados.rados.0.smithi084.stdout:1197: oid 41 updating version 0 to 100 2024-09-20T22:46:45.345 INFO:tasks.rados.rados.0.smithi084.stdout:1197: oid 41 version 100 is already newer than 99 2024-09-20T22:46:45.346 INFO:tasks.rados.rados.0.smithi084.stdout:1197: oid 41 version 100 is already newer than 98 2024-09-20T22:46:45.346 INFO:tasks.rados.rados.0.smithi084.stdout:1197: oid 41 version 100 is already newer than 97 2024-09-20T22:46:45.346 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 41 v 100 (ObjNum 440 snap 115 seq_num 440) dirty exists 2024-09-20T22:46:45.346 INFO:tasks.rados.rados.0.smithi084.stdout:1197: left oid 41 (ObjNum 440 snap 115 seq_num 440) 2024-09-20T22:46:45.352 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:45.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:45.354 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:46:45.454 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:46:45.476 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:45.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:45.477 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:46:45.520 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:45 smithi084 ceph-mon[40024]: pgmap v387: 168 pgs: 6 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 158 active+clean; 130 MiB data, 542 MiB used, 536 GiB / 536 GiB avail; 7.2 MiB/s rd, 652 KiB/s wr, 8 op/s 2024-09-20T22:46:45.520 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:45 smithi084 ceph-mon[40024]: 6.0 scrub starts 2024-09-20T22:46:45.520 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:45 smithi084 ceph-mon[40024]: 6.0 scrub ok 2024-09-20T22:46:45.520 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:45 smithi084 ceph-mon[40024]: 2.2 scrub starts 2024-09-20T22:46:45.520 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:45 smithi084 ceph-mon[40024]: 2.2 scrub ok 2024-09-20T22:46:45.520 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:45 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:46:45.521 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:45 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:46:45.521 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:45 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:46:45.521 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:45 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/1886808800' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:46:45.521 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:45 smithi084 ceph-mon[40024]: osdmap e325: 8 total, 8 up, 6 in 2024-09-20T22:46:45.521 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:45 smithi084 ceph-mon[29774]: pgmap v387: 168 pgs: 6 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 158 active+clean; 130 MiB data, 542 MiB used, 536 GiB / 536 GiB avail; 7.2 MiB/s rd, 652 KiB/s wr, 8 op/s 2024-09-20T22:46:45.521 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:45 smithi084 ceph-mon[29774]: 6.0 scrub starts 2024-09-20T22:46:45.521 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:45 smithi084 ceph-mon[29774]: 6.0 scrub ok 2024-09-20T22:46:45.521 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:45 smithi084 ceph-mon[29774]: 2.2 scrub starts 2024-09-20T22:46:45.521 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:45 smithi084 ceph-mon[29774]: 2.2 scrub ok 2024-09-20T22:46:45.521 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:45 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:46:45.521 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:45 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:46:45.521 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:45 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:46:45.522 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:45 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/1886808800' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:46:45.522 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:45 smithi084 ceph-mon[29774]: osdmap e325: 8 total, 8 up, 6 in 2024-09-20T22:46:45.577 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:46:45.598 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:45.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:45.599 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:46:45.699 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:46:45.721 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:45.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:45.722 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:46:45.751 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-20T22:46:45.751 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2024-09-20T22:46:45.751 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2024-09-20T22:46:45.751 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2024-09-20T22:46:45.751 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-20T22:46:45.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:45 smithi154 ceph-mon[33264]: pgmap v387: 168 pgs: 6 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 158 active+clean; 130 MiB data, 542 MiB used, 536 GiB / 536 GiB avail; 7.2 MiB/s rd, 652 KiB/s wr, 8 op/s 2024-09-20T22:46:45.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:45 smithi154 ceph-mon[33264]: 6.0 scrub starts 2024-09-20T22:46:45.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:45 smithi154 ceph-mon[33264]: 6.0 scrub ok 2024-09-20T22:46:45.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:45 smithi154 ceph-mon[33264]: 2.2 scrub starts 2024-09-20T22:46:45.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:45 smithi154 ceph-mon[33264]: 2.2 scrub ok 2024-09-20T22:46:45.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:45 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:46:45.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:45 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:46:45.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:45 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:46:45.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:45 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/1886808800' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:46:45.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:45 smithi154 ceph-mon[33264]: osdmap e325: 8 total, 8 up, 6 in 2024-09-20T22:46:45.823 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:46:45.844 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:45.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:45.845 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:46:45.946 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:46:45.968 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:45.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:45.969 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:46:46.070 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:46:46.091 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:46.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:46.092 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:46:46.193 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:46:46.217 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:46.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:46.218 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:46:46.319 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:46:46.335 INFO:tasks.rados.rados.0.smithi084.stdout:1197: done (1 left) 2024-09-20T22:46:46.336 INFO:tasks.rados.rados.0.smithi084.stdout:1198: done (0 left) 2024-09-20T22:46:46.336 INFO:tasks.rados.rados.0.smithi084.stdout:1199: read oid 37 snap -1 2024-09-20T22:46:46.336 INFO:tasks.rados.rados.0.smithi084.stdout:1199: expect deleted 2024-09-20T22:46:46.336 INFO:tasks.rados.rados.0.smithi084.stdout:1200: read oid 48 snap 113 2024-09-20T22:46:46.336 INFO:tasks.rados.rados.0.smithi084.stdout:1200: expect (ObjNum 424 snap 109 seq_num 424) 2024-09-20T22:46:46.336 INFO:tasks.rados.rados.0.smithi084.stdout:1201: delete oid 44 current snap is 116 2024-09-20T22:46:46.340 INFO:tasks.rados.rados.0.smithi084.stdout:1199: done (2 left) 2024-09-20T22:46:46.340 INFO:tasks.rados.rados.0.smithi084.stdout:1201: done (1 left) 2024-09-20T22:46:46.340 INFO:tasks.rados.rados.0.smithi084.stdout:1202: read oid 8 snap -1 2024-09-20T22:46:46.340 INFO:tasks.rados.rados.0.smithi084.stdout:1202: expect (ObjNum 438 snap 115 seq_num 438) 2024-09-20T22:46:46.341 INFO:tasks.rados.rados.0.smithi084.stdout:1203: write oid 7 current snap is 116 2024-09-20T22:46:46.341 INFO:tasks.rados.rados.0.smithi084.stdout:1203: seq_num 442 ranges {628694=408532,1515041=600142,2805198=552112} 2024-09-20T22:46:46.346 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:46.346 INFO:tasks.rados.rados.0.smithi084.stdout:1203: writing smithi08477203-7 from 628694 to 1037226 tid 1 2024-09-20T22:46:46.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:46.347 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:46:46.352 INFO:tasks.rados.rados.0.smithi084.stdout:1203: writing smithi08477203-7 from 1515041 to 2115183 tid 2 2024-09-20T22:46:46.359 INFO:tasks.rados.rados.0.smithi084.stdout:1203: writing smithi08477203-7 from 2805198 to 3357310 tid 3 2024-09-20T22:46:46.359 INFO:tasks.rados.rados.0.smithi084.stdout:1204: copy_from oid 4 from oid 40 current snap is 116 2024-09-20T22:46:46.359 INFO:tasks.rados.rados.0.smithi084.stdout:1200: expect (ObjNum 424 snap 109 seq_num 424) 2024-09-20T22:46:46.448 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:46:46.472 INFO:tasks.rados.rados.0.smithi084.stdout:1202: expect (ObjNum 438 snap 115 seq_num 438) 2024-09-20T22:46:46.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:46.474 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:46.474 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:46:46.523 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.0 dump_historic_ops 2024-09-20T22:46:46.548 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:46:46.575 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:46:46.588 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:46:46.597 INFO:tasks.rados.rados.0.smithi084.stdout:1203: finishing write tid 1 to smithi08477203-7 2024-09-20T22:46:46.597 INFO:tasks.rados.rados.0.smithi084.stdout:1203: finishing write tid 2 to smithi08477203-7 2024-09-20T22:46:46.597 INFO:tasks.rados.rados.0.smithi084.stdout:1203: finishing write tid 3 to smithi08477203-7 2024-09-20T22:46:46.597 INFO:tasks.rados.rados.0.smithi084.stdout:1203: finishing write tid 4 to smithi08477203-7 2024-09-20T22:46:46.597 INFO:tasks.rados.rados.0.smithi084.stdout:1203: finishing write tid 5 to smithi08477203-7 2024-09-20T22:46:46.597 INFO:tasks.rados.rados.0.smithi084.stdout:1203: oid 7 updating version 0 to 269 2024-09-20T22:46:46.598 INFO:tasks.rados.rados.0.smithi084.stdout:1203: oid 7 version 269 is already newer than 266 2024-09-20T22:46:46.598 INFO:tasks.rados.rados.0.smithi084.stdout:1203: oid 7 version 269 is already newer than 268 2024-09-20T22:46:46.598 INFO:tasks.rados.rados.0.smithi084.stdout:1203: oid 7 version 269 is already newer than 267 2024-09-20T22:46:46.598 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 7 v 269 (ObjNum 442 snap 116 seq_num 442) dirty exists 2024-09-20T22:46:46.598 INFO:tasks.rados.rados.0.smithi084.stdout:1203: left oid 7 (ObjNum 442 snap 116 seq_num 442) 2024-09-20T22:46:46.598 INFO:tasks.rados.rados.0.smithi084.stdout:1204: finishing copy_from racing read to smithi08477203-4 2024-09-20T22:46:46.598 INFO:tasks.rados.rados.0.smithi084.stdout:1204: finishing copy_from to smithi08477203-4 2024-09-20T22:46:46.598 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 4 v 228 (ObjNum 417 snap 105 seq_num 417) dirty exists 2024-09-20T22:46:46.598 INFO:tasks.rados.rados.0.smithi084.stdout:1200: done (3 left) 2024-09-20T22:46:46.598 INFO:tasks.rados.rados.0.smithi084.stdout:1202: done (2 left) 2024-09-20T22:46:46.598 INFO:tasks.rados.rados.0.smithi084.stdout:1203: done (1 left) 2024-09-20T22:46:46.598 INFO:tasks.rados.rados.0.smithi084.stdout:1204: done (0 left) 2024-09-20T22:46:46.598 INFO:tasks.rados.rados.0.smithi084.stdout:1205: copy_from oid 44 from oid 37 current snap is 116 2024-09-20T22:46:46.598 INFO:tasks.rados.rados.0.smithi084.stdout:1206: snap_create 2024-09-20T22:46:46.599 INFO:tasks.rados.rados.0.smithi084.stdout:1205: finishing copy_from to smithi08477203-44 2024-09-20T22:46:46.599 INFO:tasks.rados.rados.0.smithi084.stdout:1205: got expected ENOENT (src dne) 2024-09-20T22:46:46.599 INFO:tasks.rados.rados.0.smithi084.stdout:1205: finishing copy_from racing read to smithi08477203-44 2024-09-20T22:46:46.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:46.600 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:46.600 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:46:46.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:46 smithi084 ceph-mon[40024]: 4.1a scrub starts 2024-09-20T22:46:46.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:46 smithi084 ceph-mon[40024]: 4.1a scrub ok 2024-09-20T22:46:46.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:46 smithi084 ceph-mon[40024]: 6.1a scrub starts 2024-09-20T22:46:46.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:46 smithi084 ceph-mon[40024]: 6.1a scrub ok 2024-09-20T22:46:46.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:46 smithi084 ceph-mon[40024]: 7.8 scrub starts 2024-09-20T22:46:46.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:46 smithi084 ceph-mon[40024]: 7.8 scrub ok 2024-09-20T22:46:46.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:46 smithi084 ceph-mon[40024]: purged_snaps scrub starts 2024-09-20T22:46:46.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:46 smithi084 ceph-mon[40024]: purged_snaps scrub ok 2024-09-20T22:46:46.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:46 smithi084 ceph-mon[29774]: 4.1a scrub starts 2024-09-20T22:46:46.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:46 smithi084 ceph-mon[29774]: 4.1a scrub ok 2024-09-20T22:46:46.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:46 smithi084 ceph-mon[29774]: 6.1a scrub starts 2024-09-20T22:46:46.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:46 smithi084 ceph-mon[29774]: 6.1a scrub ok 2024-09-20T22:46:46.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:46 smithi084 ceph-mon[29774]: 7.8 scrub starts 2024-09-20T22:46:46.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:46 smithi084 ceph-mon[29774]: 7.8 scrub ok 2024-09-20T22:46:46.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:46 smithi084 ceph-mon[29774]: purged_snaps scrub starts 2024-09-20T22:46:46.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:46 smithi084 ceph-mon[29774]: purged_snaps scrub ok 2024-09-20T22:46:46.700 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:46:46.723 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:46.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:46.724 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:46:46.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:46 smithi154 ceph-mon[33264]: 4.1a scrub starts 2024-09-20T22:46:46.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:46 smithi154 ceph-mon[33264]: 4.1a scrub ok 2024-09-20T22:46:46.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:46 smithi154 ceph-mon[33264]: 6.1a scrub starts 2024-09-20T22:46:46.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:46 smithi154 ceph-mon[33264]: 6.1a scrub ok 2024-09-20T22:46:46.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:46 smithi154 ceph-mon[33264]: 7.8 scrub starts 2024-09-20T22:46:46.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:46 smithi154 ceph-mon[33264]: 7.8 scrub ok 2024-09-20T22:46:46.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:46 smithi154 ceph-mon[33264]: purged_snaps scrub starts 2024-09-20T22:46:46.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:46 smithi154 ceph-mon[33264]: purged_snaps scrub ok 2024-09-20T22:46:46.825 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:46:46.850 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:46.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:46.851 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:46:46.952 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:46:46.977 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:46.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:46.978 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:46:47.079 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:46:47.100 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:47.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:47.101 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:46:47.202 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:46:47.228 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:47.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:47.229 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:46:47.329 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:46:47.343 INFO:tasks.rados.rados.0.smithi084.stdout:1205: done (1 left) 2024-09-20T22:46:47.343 INFO:tasks.rados.rados.0.smithi084.stdout:1206: done (0 left) 2024-09-20T22:46:47.343 INFO:tasks.rados.rados.0.smithi084.stdout:1207: write (excl) oid 21 current snap is 117 2024-09-20T22:46:47.343 INFO:tasks.rados.rados.0.smithi084.stdout:1207: seq_num 443 ranges {645501=483781,1573523=460153,2548974=763384} 2024-09-20T22:46:47.349 INFO:tasks.rados.rados.0.smithi084.stdout:1207: writing smithi08477203-21 from 645501 to 1129282 tid 1 2024-09-20T22:46:47.354 INFO:tasks.rados.rados.0.smithi084.stdout:1207: writing smithi08477203-21 from 1573523 to 2033676 tid 2 2024-09-20T22:46:47.356 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:47.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:47.357 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:46:47.361 INFO:tasks.rados.rados.0.smithi084.stdout:1207: writing smithi08477203-21 from 2548974 to 3312358 tid 3 2024-09-20T22:46:47.361 INFO:tasks.rados.rados.0.smithi084.stdout:1208: write (excl) oid 42 current snap is 117 2024-09-20T22:46:47.362 INFO:tasks.rados.rados.0.smithi084.stdout:1207: finishing write tid 1 to smithi08477203-21 2024-09-20T22:46:47.362 INFO:tasks.rados.rados.0.smithi084.stdout:1208: seq_num 444 ranges {662308=559030,1632005=603811,2860044=602264} 2024-09-20T22:46:47.368 INFO:tasks.rados.rados.0.smithi084.stdout:1208: writing smithi08477203-42 from 662308 to 1221338 tid 1 2024-09-20T22:46:47.375 INFO:tasks.rados.rados.0.smithi084.stdout:1208: writing smithi08477203-42 from 1632005 to 2235816 tid 2 2024-09-20T22:46:47.384 INFO:tasks.rados.rados.0.smithi084.stdout:1208: writing smithi08477203-42 from 2860044 to 3462308 tid 3 2024-09-20T22:46:47.384 INFO:tasks.rados.rados.0.smithi084.stdout:1209: write oid 18 current snap is 117 2024-09-20T22:46:47.384 INFO:tasks.rados.rados.0.smithi084.stdout:1207: finishing write tid 2 to smithi08477203-21 2024-09-20T22:46:47.384 INFO:tasks.rados.rados.0.smithi084.stdout:1209: seq_num 445 ranges {679115=634279,2090487=463822,3003820=475295} 2024-09-20T22:46:47.391 INFO:tasks.rados.rados.0.smithi084.stdout:1209: writing smithi08477203-18 from 679115 to 1313394 tid 1 2024-09-20T22:46:47.398 INFO:tasks.rados.rados.0.smithi084.stdout:1209: writing smithi08477203-18 from 2090487 to 2554309 tid 2 2024-09-20T22:46:47.404 INFO:tasks.rados.rados.0.smithi084.stdout:1209: writing smithi08477203-18 from 3003820 to 3479115 tid 3 2024-09-20T22:46:47.404 INFO:tasks.rados.rados.0.smithi084.stdout:1210: read oid 27 snap -1 2024-09-20T22:46:47.404 INFO:tasks.rados.rados.0.smithi084.stdout:1210: expect deleted 2024-09-20T22:46:47.404 INFO:tasks.rados.rados.0.smithi084.stdout:1211: snap_create 2024-09-20T22:46:47.404 INFO:tasks.rados.rados.0.smithi084.stdout:1207: finishing write tid 3 to smithi08477203-21 2024-09-20T22:46:47.404 INFO:tasks.rados.rados.0.smithi084.stdout:1207: finishing write tid 4 to smithi08477203-21 2024-09-20T22:46:47.404 INFO:tasks.rados.rados.0.smithi084.stdout:1207: finishing write tid 5 to smithi08477203-21 2024-09-20T22:46:47.404 INFO:tasks.rados.rados.0.smithi084.stdout:1207: oid 21 updating version 0 to 503 2024-09-20T22:46:47.404 INFO:tasks.rados.rados.0.smithi084.stdout:1207: oid 21 updating version 503 to 505 2024-09-20T22:46:47.404 INFO:tasks.rados.rados.0.smithi084.stdout:1207: oid 21 updating version 505 to 506 2024-09-20T22:46:47.404 INFO:tasks.rados.rados.0.smithi084.stdout:1207: oid 21 version 506 is already newer than 504 2024-09-20T22:46:47.405 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 21 v 506 (ObjNum 443 snap 117 seq_num 443) dirty exists 2024-09-20T22:46:47.405 INFO:tasks.rados.rados.0.smithi084.stdout:1207: left oid 21 (ObjNum 443 snap 117 seq_num 443) 2024-09-20T22:46:47.405 INFO:tasks.rados.rados.0.smithi084.stdout:1208: finishing write tid 1 to smithi08477203-42 2024-09-20T22:46:47.405 INFO:tasks.rados.rados.0.smithi084.stdout:1208: finishing write tid 2 to smithi08477203-42 2024-09-20T22:46:47.405 INFO:tasks.rados.rados.0.smithi084.stdout:1208: finishing write tid 3 to smithi08477203-42 2024-09-20T22:46:47.405 INFO:tasks.rados.rados.0.smithi084.stdout:1208: finishing write tid 4 to smithi08477203-42 2024-09-20T22:46:47.405 INFO:tasks.rados.rados.0.smithi084.stdout:1208: finishing write tid 5 to smithi08477203-42 2024-09-20T22:46:47.405 INFO:tasks.rados.rados.0.smithi084.stdout:1208: oid 42 updating version 0 to 377 2024-09-20T22:46:47.405 INFO:tasks.rados.rados.0.smithi084.stdout:1208: oid 42 updating version 377 to 380 2024-09-20T22:46:47.405 INFO:tasks.rados.rados.0.smithi084.stdout:1208: oid 42 version 380 is already newer than 379 2024-09-20T22:46:47.405 INFO:tasks.rados.rados.0.smithi084.stdout:1208: oid 42 version 380 is already newer than 378 2024-09-20T22:46:47.405 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 42 v 380 (ObjNum 444 snap 117 seq_num 444) dirty exists 2024-09-20T22:46:47.405 INFO:tasks.rados.rados.0.smithi084.stdout:1208: left oid 42 (ObjNum 444 snap 117 seq_num 444) 2024-09-20T22:46:47.406 INFO:tasks.rados.rados.0.smithi084.stdout:1209: finishing write tid 1 to smithi08477203-18 2024-09-20T22:46:47.406 INFO:tasks.rados.rados.0.smithi084.stdout:1209: finishing write tid 2 to smithi08477203-18 2024-09-20T22:46:47.406 INFO:tasks.rados.rados.0.smithi084.stdout:1209: finishing write tid 3 to smithi08477203-18 2024-09-20T22:46:47.406 INFO:tasks.rados.rados.0.smithi084.stdout:1209: finishing write tid 4 to smithi08477203-18 2024-09-20T22:46:47.407 INFO:tasks.rados.rados.0.smithi084.stdout:1209: finishing write tid 5 to smithi08477203-18 2024-09-20T22:46:47.407 INFO:tasks.rados.rados.0.smithi084.stdout:1209: oid 18 updating version 0 to 387 2024-09-20T22:46:47.407 INFO:tasks.rados.rados.0.smithi084.stdout:1209: oid 18 version 387 is already newer than 386 2024-09-20T22:46:47.407 INFO:tasks.rados.rados.0.smithi084.stdout:1209: oid 18 updating version 387 to 388 2024-09-20T22:46:47.407 INFO:tasks.rados.rados.0.smithi084.stdout:1209: oid 18 updating version 388 to 389 2024-09-20T22:46:47.407 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 18 v 389 (ObjNum 445 snap 117 seq_num 445) dirty exists 2024-09-20T22:46:47.407 INFO:tasks.rados.rados.0.smithi084.stdout:1209: left oid 18 (ObjNum 445 snap 117 seq_num 445) 2024-09-20T22:46:47.459 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:46:47.481 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:47.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:47.482 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:46:47.544 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-20T22:46:47.565 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:47 smithi084 ceph-mon[40024]: pgmap v390: 168 pgs: 6 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 158 active+clean; 130 MiB data, 542 MiB used, 536 GiB / 536 GiB avail; 7.2 MiB/s rd, 652 KiB/s wr, 8 op/s 2024-09-20T22:46:47.565 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:47 smithi084 ceph-mon[40024]: osdmap e326: 8 total, 8 up, 6 in 2024-09-20T22:46:47.566 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:47 smithi084 ceph-mon[29774]: pgmap v390: 168 pgs: 6 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 158 active+clean; 130 MiB data, 542 MiB used, 536 GiB / 536 GiB avail; 7.2 MiB/s rd, 652 KiB/s wr, 8 op/s 2024-09-20T22:46:47.566 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:47 smithi084 ceph-mon[29774]: osdmap e326: 8 total, 8 up, 6 in 2024-09-20T22:46:47.583 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:46:47.605 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:47.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:47.607 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:46:47.707 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:46:47.730 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:47.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:47.731 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:46:47.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:47 smithi154 ceph-mon[33264]: pgmap v390: 168 pgs: 6 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 158 active+clean; 130 MiB data, 542 MiB used, 536 GiB / 536 GiB avail; 7.2 MiB/s rd, 652 KiB/s wr, 8 op/s 2024-09-20T22:46:47.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:47 smithi154 ceph-mon[33264]: osdmap e326: 8 total, 8 up, 6 in 2024-09-20T22:46:47.832 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:46:47.853 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:47.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:47.854 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:46:47.955 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:46:47.976 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:47.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:47.977 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:46:48.067 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph tell osd.2 reset_purged_snaps_last 2024-09-20T22:46:48.078 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:46:48.134 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:48.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:48.135 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:46:48.157 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:46:48.162 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:46:48.173 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is unset 2024-09-20T22:46:48.181 INFO:tasks.rados.rados.0.smithi084.stdout:1207: done (4 left) 2024-09-20T22:46:48.181 INFO:tasks.rados.rados.0.smithi084.stdout:1208: done (3 left) 2024-09-20T22:46:48.181 INFO:tasks.rados.rados.0.smithi084.stdout:1209: done (2 left) 2024-09-20T22:46:48.181 INFO:tasks.rados.rados.0.smithi084.stdout:1210: done (1 left) 2024-09-20T22:46:48.181 INFO:tasks.rados.rados.0.smithi084.stdout:1211: done (0 left) 2024-09-20T22:46:48.181 INFO:tasks.rados.rados.0.smithi084.stdout:1212: write oid 10 current snap is 118 2024-09-20T22:46:48.181 INFO:tasks.rados.rados.0.smithi084.stdout:1212: seq_num 446 ranges {695922=709528,2032616=607480,3198537=297385} 2024-09-20T22:46:48.191 INFO:tasks.rados.rados.0.smithi084.stdout:1212: writing smithi08477203-10 from 695922 to 1405450 tid 1 2024-09-20T22:46:48.204 INFO:tasks.rados.rados.0.smithi084.stdout:1212: writing smithi08477203-10 from 2032616 to 2640096 tid 2 2024-09-20T22:46:48.210 INFO:tasks.rados.rados.0.smithi084.stdout:1212: writing smithi08477203-10 from 3198537 to 3495922 tid 3 2024-09-20T22:46:48.210 INFO:tasks.rados.rados.0.smithi084.stdout:1213: read oid 34 snap -1 2024-09-20T22:46:48.210 INFO:tasks.rados.rados.0.smithi084.stdout:1213: expect (ObjNum 359 snap 92 seq_num 359) 2024-09-20T22:46:48.210 INFO:tasks.rados.rados.0.smithi084.stdout:1214: copy_from oid 30 from oid 18 current snap is 118 2024-09-20T22:46:48.211 INFO:tasks.rados.rados.0.smithi084.stdout:1215: copy_from oid 2 from oid 17 current snap is 118 2024-09-20T22:46:48.211 INFO:tasks.rados.rados.0.smithi084.stdout:1216: copy_from oid 33 from oid 38 current snap is 118 2024-09-20T22:46:48.211 INFO:tasks.rados.rados.0.smithi084.stdout:1217: snap_create 2024-09-20T22:46:48.220 INFO:tasks.rados.rados.0.smithi084.stdout:1213: expect (ObjNum 359 snap 92 seq_num 359) 2024-09-20T22:46:48.235 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:46:48.257 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:48.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:48.258 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:46:48.322 INFO:tasks.rados.rados.0.smithi084.stdout:1212: finishing write tid 1 to smithi08477203-10 2024-09-20T22:46:48.322 INFO:tasks.rados.rados.0.smithi084.stdout:1212: finishing write tid 2 to smithi08477203-10 2024-09-20T22:46:48.322 INFO:tasks.rados.rados.0.smithi084.stdout:1212: finishing write tid 3 to smithi08477203-10 2024-09-20T22:46:48.322 INFO:tasks.rados.rados.0.smithi084.stdout:1212: finishing write tid 4 to smithi08477203-10 2024-09-20T22:46:48.322 INFO:tasks.rados.rados.0.smithi084.stdout:1212: finishing write tid 5 to smithi08477203-10 2024-09-20T22:46:48.322 INFO:tasks.rados.rados.0.smithi084.stdout:1212: oid 10 updating version 0 to 393 2024-09-20T22:46:48.322 INFO:tasks.rados.rados.0.smithi084.stdout:1212: oid 10 version 393 is already newer than 392 2024-09-20T22:46:48.322 INFO:tasks.rados.rados.0.smithi084.stdout:1212: oid 10 version 393 is already newer than 391 2024-09-20T22:46:48.322 INFO:tasks.rados.rados.0.smithi084.stdout:1212: oid 10 version 393 is already newer than 390 2024-09-20T22:46:48.322 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 10 v 393 (ObjNum 446 snap 118 seq_num 446) dirty exists 2024-09-20T22:46:48.322 INFO:tasks.rados.rados.0.smithi084.stdout:1212: left oid 10 (ObjNum 446 snap 118 seq_num 446) 2024-09-20T22:46:48.323 INFO:tasks.rados.rados.0.smithi084.stdout:1216: finishing copy_from racing read to smithi08477203-33 2024-09-20T22:46:48.323 INFO:tasks.rados.rados.0.smithi084.stdout:1214: finishing copy_from racing read to smithi08477203-30 2024-09-20T22:46:48.323 INFO:tasks.rados.rados.0.smithi084.stdout:1215: finishing copy_from racing read to smithi08477203-2 2024-09-20T22:46:48.323 INFO:tasks.rados.rados.0.smithi084.stdout:1214: finishing copy_from to smithi08477203-30 2024-09-20T22:46:48.323 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 30 v 389 (ObjNum 445 snap 117 seq_num 445) dirty exists 2024-09-20T22:46:48.323 INFO:tasks.rados.rados.0.smithi084.stdout:1216: finishing copy_from to smithi08477203-33 2024-09-20T22:46:48.323 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 33 v 385 (ObjNum 397 snap 101 seq_num 397) dirty exists 2024-09-20T22:46:48.323 INFO:tasks.rados.rados.0.smithi084.stdout:1215: finishing copy_from to smithi08477203-2 2024-09-20T22:46:48.323 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 2 v 151 (ObjNum 426 snap 109 seq_num 426) dirty exists 2024-09-20T22:46:48.344 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:46:48 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: ::ffff:172.21.15.154 - - [20/Sep/2024:22:46:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-20T22:46:48.358 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:46:48.379 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:48.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:48.380 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:46:48.481 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:46:48.502 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:48.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:48.503 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:46:48.604 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:46:48.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:48 smithi084 ceph-mon[40024]: 3.11 scrub starts 2024-09-20T22:46:48.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:48 smithi084 ceph-mon[40024]: 3.11 scrub ok 2024-09-20T22:46:48.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:48 smithi084 ceph-mon[40024]: osdmap e327: 8 total, 8 up, 6 in 2024-09-20T22:46:48.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:48 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/598770706' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:46:48.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:48 smithi084 ceph-mon[40024]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T22:46:48.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:48 smithi084 ceph-mon[40024]: Cluster is now healthy 2024-09-20T22:46:48.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:48 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/598770706' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T22:46:48.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:48 smithi084 ceph-mon[40024]: osdmap e328: 8 total, 8 up, 6 in 2024-09-20T22:46:48.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:48 smithi084 ceph-mon[29774]: 3.11 scrub starts 2024-09-20T22:46:48.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:48 smithi084 ceph-mon[29774]: 3.11 scrub ok 2024-09-20T22:46:48.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:48 smithi084 ceph-mon[29774]: osdmap e327: 8 total, 8 up, 6 in 2024-09-20T22:46:48.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:48 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/598770706' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:46:48.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:48 smithi084 ceph-mon[29774]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T22:46:48.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:48 smithi084 ceph-mon[29774]: Cluster is now healthy 2024-09-20T22:46:48.617 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:48 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/598770706' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T22:46:48.617 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:48 smithi084 ceph-mon[29774]: osdmap e328: 8 total, 8 up, 6 in 2024-09-20T22:46:48.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:48.629 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:48.629 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:46:48.730 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:46:48.753 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:48.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:48.754 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:46:48.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:48 smithi154 ceph-mon[33264]: 3.11 scrub starts 2024-09-20T22:46:48.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:48 smithi154 ceph-mon[33264]: 3.11 scrub ok 2024-09-20T22:46:48.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:48 smithi154 ceph-mon[33264]: osdmap e327: 8 total, 8 up, 6 in 2024-09-20T22:46:48.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:48 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/598770706' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:46:48.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:48 smithi154 ceph-mon[33264]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T22:46:48.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:48 smithi154 ceph-mon[33264]: Cluster is now healthy 2024-09-20T22:46:48.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:48 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/598770706' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T22:46:48.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:48 smithi154 ceph-mon[33264]: osdmap e328: 8 total, 8 up, 6 in 2024-09-20T22:46:48.791 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:46:48.855 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:46:48.881 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:48.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:48.882 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:46:48.982 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:46:49.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:49.005 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:49.005 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:46:49.106 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:46:49.131 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:49.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:49.132 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:46:49.190 INFO:tasks.rados.rados.0.smithi084.stdout:1212: done (5 left) 2024-09-20T22:46:49.190 INFO:tasks.rados.rados.0.smithi084.stdout:1213: done (4 left) 2024-09-20T22:46:49.190 INFO:tasks.rados.rados.0.smithi084.stdout:1214: done (3 left) 2024-09-20T22:46:49.190 INFO:tasks.rados.rados.0.smithi084.stdout:1215: done (2 left) 2024-09-20T22:46:49.190 INFO:tasks.rados.rados.0.smithi084.stdout:1216: done (1 left) 2024-09-20T22:46:49.190 INFO:tasks.rados.rados.0.smithi084.stdout:1217: done (0 left) 2024-09-20T22:46:49.190 INFO:tasks.rados.rados.0.smithi084.stdout:1218: snap_remove snap 117 2024-09-20T22:46:49.233 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:46:49.258 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:49.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:49.259 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:46:49.360 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:46:49.385 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:49.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:49.386 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:46:49.487 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:46:49.504 INFO:journalctl@ceph.iscsi.iscsi.a.smithi154.stdout:Sep 20 22:46:49 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-iscsi-iscsi-a[59730]: debug there is no tcmu-runner data available 2024-09-20T22:46:49.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:49 smithi154 ceph-mon[33264]: pgmap v393: 168 pgs: 6 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 158 active+clean; 130 MiB data, 542 MiB used, 536 GiB / 536 GiB avail 2024-09-20T22:46:49.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:49 smithi154 ceph-mon[33264]: 5.17 scrub starts 2024-09-20T22:46:49.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:49 smithi154 ceph-mon[33264]: 5.17 scrub ok 2024-09-20T22:46:49.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:49 smithi154 ceph-mon[33264]: 4.1 scrub starts 2024-09-20T22:46:49.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:49 smithi154 ceph-mon[33264]: 4.1 scrub ok 2024-09-20T22:46:49.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:49 smithi154 ceph-mon[33264]: osdmap e329: 8 total, 8 up, 6 in 2024-09-20T22:46:49.511 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:49.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:49.512 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:46:49.612 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:46:49.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:49 smithi084 ceph-mon[29774]: pgmap v393: 168 pgs: 6 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 158 active+clean; 130 MiB data, 542 MiB used, 536 GiB / 536 GiB avail 2024-09-20T22:46:49.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:49 smithi084 ceph-mon[29774]: 5.17 scrub starts 2024-09-20T22:46:49.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:49 smithi084 ceph-mon[29774]: 5.17 scrub ok 2024-09-20T22:46:49.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:49 smithi084 ceph-mon[29774]: 4.1 scrub starts 2024-09-20T22:46:49.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:49 smithi084 ceph-mon[29774]: 4.1 scrub ok 2024-09-20T22:46:49.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:49 smithi084 ceph-mon[29774]: osdmap e329: 8 total, 8 up, 6 in 2024-09-20T22:46:49.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:49 smithi084 ceph-mon[40024]: pgmap v393: 168 pgs: 6 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 158 active+clean; 130 MiB data, 542 MiB used, 536 GiB / 536 GiB avail 2024-09-20T22:46:49.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:49 smithi084 ceph-mon[40024]: 5.17 scrub starts 2024-09-20T22:46:49.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:49 smithi084 ceph-mon[40024]: 5.17 scrub ok 2024-09-20T22:46:49.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:49 smithi084 ceph-mon[40024]: 4.1 scrub starts 2024-09-20T22:46:49.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:49 smithi084 ceph-mon[40024]: 4.1 scrub ok 2024-09-20T22:46:49.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:49 smithi084 ceph-mon[40024]: osdmap e329: 8 total, 8 up, 6 in 2024-09-20T22:46:49.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:49.635 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:49.635 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:46:49.736 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:46:49.761 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:49.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:49.762 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:46:49.863 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:46:49.888 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:49.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:49.890 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:46:49.938 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:46:49.938 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:46:49.954 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:46:49.955 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:46:49.970 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:46:49.971 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:46:49.986 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-20T22:46:49.986 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T22:46:49.991 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:46:50.003 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:46:50.003 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:46:50.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:50.016 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:50.016 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:46:50.020 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:46:50.020 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:46:50.037 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:46:50.037 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:46:50.054 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:46:50.055 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:46:50.117 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:46:50.141 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:50.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:50.142 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:46:50.188 INFO:tasks.rados.rados.0.smithi084.stdout:1218: done (0 left) 2024-09-20T22:46:50.188 INFO:tasks.rados.rados.0.smithi084.stdout:1219: read oid 34 snap -1 2024-09-20T22:46:50.189 INFO:tasks.rados.rados.0.smithi084.stdout:1219: expect (ObjNum 359 snap 92 seq_num 359) 2024-09-20T22:46:50.189 INFO:tasks.rados.rados.0.smithi084.stdout:1220: rollback oid 28 current snap is 119 2024-09-20T22:46:50.189 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 28 to 118 2024-09-20T22:46:50.189 INFO:tasks.rados.rados.0.smithi084.stdout:1221: read oid 23 snap -1 2024-09-20T22:46:50.189 INFO:tasks.rados.rados.0.smithi084.stdout:1221: expect (ObjNum 428 snap 110 seq_num 428) 2024-09-20T22:46:50.189 INFO:tasks.rados.rados.0.smithi084.stdout:1222: write oid 30 current snap is 119 2024-09-20T22:46:50.189 INFO:tasks.rados.rados.0.smithi084.stdout:1222: seq_num 447 ranges {712729=784777,2091098=467491,3342313=170416} 2024-09-20T22:46:50.198 INFO:tasks.rados.rados.0.smithi084.stdout:1222: writing smithi08477203-30 from 712729 to 1497506 tid 1 2024-09-20T22:46:50.205 INFO:tasks.rados.rados.0.smithi084.stdout:1222: writing smithi08477203-30 from 2091098 to 2558589 tid 2 2024-09-20T22:46:50.211 INFO:tasks.rados.rados.0.smithi084.stdout:1222: writing smithi08477203-30 from 3342313 to 3512729 tid 3 2024-09-20T22:46:50.211 INFO:tasks.rados.rados.0.smithi084.stdout:1223: snap_create 2024-09-20T22:46:50.211 INFO:tasks.rados.rados.0.smithi084.stdout:1219: expect (ObjNum 359 snap 92 seq_num 359) 2024-09-20T22:46:50.242 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:46:50.263 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:50.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:50.264 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:46:50.309 INFO:tasks.rados.rados.0.smithi084.stdout:1220: finishing rollback tid 0 to smithi08477203-28 2024-09-20T22:46:50.309 INFO:tasks.rados.rados.0.smithi084.stdout:1220: finishing rollback tid 1 to smithi08477203-28 2024-09-20T22:46:50.309 INFO:tasks.rados.rados.0.smithi084.stdout:1220: finishing rollback tid 2 to smithi08477203-28 2024-09-20T22:46:50.309 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 28 v 665 (ObjNum 400 snap 101 seq_num 400) dirty exists 2024-09-20T22:46:50.309 INFO:tasks.rados.rados.0.smithi084.stdout:1221: expect (ObjNum 428 snap 110 seq_num 428) 2024-09-20T22:46:50.365 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:46:50.387 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:50.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:50.388 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:46:50.488 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:46:50.494 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:50 smithi084 ceph-mon[29774]: 3.f scrub starts 2024-09-20T22:46:50.494 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:50 smithi084 ceph-mon[29774]: 3.f scrub ok 2024-09-20T22:46:50.494 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:50 smithi084 ceph-mon[29774]: 5.a scrub starts 2024-09-20T22:46:50.494 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:50 smithi084 ceph-mon[29774]: 5.a scrub ok 2024-09-20T22:46:50.494 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:50 smithi084 ceph-mon[29774]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:46:50.494 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:50 smithi084 ceph-mon[29774]: 4.e scrub starts 2024-09-20T22:46:50.495 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:50 smithi084 ceph-mon[29774]: 4.e scrub ok 2024-09-20T22:46:50.495 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:50 smithi084 ceph-mon[29774]: 5.1d scrub starts 2024-09-20T22:46:50.495 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:50 smithi084 ceph-mon[29774]: 5.1d scrub ok 2024-09-20T22:46:50.495 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:50 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/3468814953' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:46:50.495 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:50 smithi084 ceph-mon[29774]: osdmap e330: 8 total, 8 up, 6 in 2024-09-20T22:46:50.495 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:50 smithi084 ceph-mon[40024]: 3.f scrub starts 2024-09-20T22:46:50.495 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:50 smithi084 ceph-mon[40024]: 3.f scrub ok 2024-09-20T22:46:50.495 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:50 smithi084 ceph-mon[40024]: 5.a scrub starts 2024-09-20T22:46:50.495 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:50 smithi084 ceph-mon[40024]: 5.a scrub ok 2024-09-20T22:46:50.495 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:50 smithi084 ceph-mon[40024]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:46:50.495 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:50 smithi084 ceph-mon[40024]: 4.e scrub starts 2024-09-20T22:46:50.495 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:50 smithi084 ceph-mon[40024]: 4.e scrub ok 2024-09-20T22:46:50.496 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:50 smithi084 ceph-mon[40024]: 5.1d scrub starts 2024-09-20T22:46:50.496 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:50 smithi084 ceph-mon[40024]: 5.1d scrub ok 2024-09-20T22:46:50.496 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:50 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/3468814953' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:46:50.496 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:50 smithi084 ceph-mon[40024]: osdmap e330: 8 total, 8 up, 6 in 2024-09-20T22:46:50.511 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:50.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:50.512 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:46:50.522 INFO:tasks.rados.rados.0.smithi084.stdout:1222: finishing write tid 1 to smithi08477203-30 2024-09-20T22:46:50.522 INFO:tasks.rados.rados.0.smithi084.stdout:1222: finishing write tid 2 to smithi08477203-30 2024-09-20T22:46:50.522 INFO:tasks.rados.rados.0.smithi084.stdout:1222: finishing write tid 3 to smithi08477203-30 2024-09-20T22:46:50.522 INFO:tasks.rados.rados.0.smithi084.stdout:1222: finishing write tid 4 to smithi08477203-30 2024-09-20T22:46:50.523 INFO:tasks.rados.rados.0.smithi084.stdout:1222: finishing write tid 5 to smithi08477203-30 2024-09-20T22:46:50.523 INFO:tasks.rados.rados.0.smithi084.stdout:1222: oid 30 updating version 0 to 390 2024-09-20T22:46:50.523 INFO:tasks.rados.rados.0.smithi084.stdout:1222: oid 30 updating version 390 to 392 2024-09-20T22:46:50.523 INFO:tasks.rados.rados.0.smithi084.stdout:1222: oid 30 updating version 392 to 393 2024-09-20T22:46:50.523 INFO:tasks.rados.rados.0.smithi084.stdout:1222: oid 30 version 393 is already newer than 391 2024-09-20T22:46:50.523 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 30 v 393 (ObjNum 447 snap 119 seq_num 447) dirty exists 2024-09-20T22:46:50.523 INFO:tasks.rados.rados.0.smithi084.stdout:1222: left oid 30 (ObjNum 447 snap 119 seq_num 447) 2024-09-20T22:46:50.592 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-20T22:46:50.592 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:46:50.613 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:46:50.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:50.636 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:50.636 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:46:50.737 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:46:50.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:50 smithi154 ceph-mon[33264]: 3.f scrub starts 2024-09-20T22:46:50.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:50 smithi154 ceph-mon[33264]: 3.f scrub ok 2024-09-20T22:46:50.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:50 smithi154 ceph-mon[33264]: 5.a scrub starts 2024-09-20T22:46:50.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:50 smithi154 ceph-mon[33264]: 5.a scrub ok 2024-09-20T22:46:50.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:50 smithi154 ceph-mon[33264]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:46:50.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:50 smithi154 ceph-mon[33264]: 4.e scrub starts 2024-09-20T22:46:50.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:50 smithi154 ceph-mon[33264]: 4.e scrub ok 2024-09-20T22:46:50.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:50 smithi154 ceph-mon[33264]: 5.1d scrub starts 2024-09-20T22:46:50.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:50 smithi154 ceph-mon[33264]: 5.1d scrub ok 2024-09-20T22:46:50.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:50 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/3468814953' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:46:50.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:50 smithi154 ceph-mon[33264]: osdmap e330: 8 total, 8 up, 6 in 2024-09-20T22:46:50.760 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:50.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:50.761 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:46:50.780 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd set noscrub 2024-09-20T22:46:50.862 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:46:50.885 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:50.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:50.886 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:46:50.987 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:46:51.008 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:51.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:51.009 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:46:51.110 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:46:51.133 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:51.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:51.134 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:46:51.190 INFO:tasks.rados.rados.0.smithi084.stdout:1219: done (4 left) 2024-09-20T22:46:51.190 INFO:tasks.rados.rados.0.smithi084.stdout:1220: done (3 left) 2024-09-20T22:46:51.190 INFO:tasks.rados.rados.0.smithi084.stdout:1221: done (2 left) 2024-09-20T22:46:51.190 INFO:tasks.rados.rados.0.smithi084.stdout:1222: done (1 left) 2024-09-20T22:46:51.191 INFO:tasks.rados.rados.0.smithi084.stdout:1223: done (0 left) 2024-09-20T22:46:51.191 INFO:tasks.rados.rados.0.smithi084.stdout:1224: write oid 14 current snap is 120 2024-09-20T22:46:51.191 INFO:tasks.rados.rados.0.smithi084.stdout:1224: seq_num 448 ranges {729536=460026,1749580=611149,2853383=452450} 2024-09-20T22:46:51.197 INFO:tasks.rados.rados.0.smithi084.stdout:1224: writing smithi08477203-14 from 729536 to 1189562 tid 1 2024-09-20T22:46:51.204 INFO:tasks.rados.rados.0.smithi084.stdout:1224: writing smithi08477203-14 from 1749580 to 2360729 tid 2 2024-09-20T22:46:51.210 INFO:tasks.rados.rados.0.smithi084.stdout:1224: writing smithi08477203-14 from 2853383 to 3305833 tid 3 2024-09-20T22:46:51.210 INFO:tasks.rados.rados.0.smithi084.stdout:1225: copy_from oid 6 from oid 24 current snap is 120 2024-09-20T22:46:51.210 INFO:tasks.rados.rados.0.smithi084.stdout:1224: finishing write tid 1 to smithi08477203-14 2024-09-20T22:46:51.210 INFO:tasks.rados.rados.0.smithi084.stdout:1224: finishing write tid 2 to smithi08477203-14 2024-09-20T22:46:51.210 INFO:tasks.rados.rados.0.smithi084.stdout:1226: read oid 12 snap -1 2024-09-20T22:46:51.210 INFO:tasks.rados.rados.0.smithi084.stdout:1226: expect (ObjNum 437 snap 115 seq_num 437) 2024-09-20T22:46:51.210 INFO:tasks.rados.rados.0.smithi084.stdout:1227: read oid 39 snap -1 2024-09-20T22:46:51.210 INFO:tasks.rados.rados.0.smithi084.stdout:1227: expect (ObjNum 397 snap 101 seq_num 397) 2024-09-20T22:46:51.210 INFO:tasks.rados.rados.0.smithi084.stdout:1228: write oid 20 current snap is 120 2024-09-20T22:46:51.210 INFO:tasks.rados.rados.0.smithi084.stdout:1228: seq_num 449 ranges {746343=651628,1924415=754807,3397159=149184} 2024-09-20T22:46:51.220 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-20T22:46:51.220 INFO:tasks.rados.rados.0.smithi084.stdout:1228: writing smithi08477203-20 from 746343 to 1397971 tid 1 2024-09-20T22:46:51.227 INFO:tasks.rados.rados.0.smithi084.stdout:1228: writing smithi08477203-20 from 1924415 to 2679222 tid 2 2024-09-20T22:46:51.232 INFO:tasks.rados.rados.0.smithi084.stdout:1228: writing smithi08477203-20 from 3397159 to 3546343 tid 3 2024-09-20T22:46:51.233 INFO:tasks.rados.rados.0.smithi084.stdout:1229: snap_remove snap 116 2024-09-20T22:46:51.235 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:46:51.257 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:51.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:51.258 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:46:51.358 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:46:51.379 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:51.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:51.380 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:46:51.481 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:46:51.502 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:51.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:51.503 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:46:51.604 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:46:51.614 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:51 smithi084 ceph-mon[29774]: pgmap v396: 168 pgs: 4 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 162 active+clean; 158 MiB data, 567 MiB used, 536 GiB / 536 GiB avail; 9.2 MiB/s rd, 6.0 MiB/s wr, 24 op/s 2024-09-20T22:46:51.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:51 smithi084 ceph-mon[29774]: 5.d deep-scrub starts 2024-09-20T22:46:51.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:51 smithi084 ceph-mon[29774]: 5.d deep-scrub ok 2024-09-20T22:46:51.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:51 smithi084 ceph-mon[29774]: osdmap e331: 8 total, 8 up, 6 in 2024-09-20T22:46:51.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:51 smithi084 ceph-mon[40024]: pgmap v396: 168 pgs: 4 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 162 active+clean; 158 MiB data, 567 MiB used, 536 GiB / 536 GiB avail; 9.2 MiB/s rd, 6.0 MiB/s wr, 24 op/s 2024-09-20T22:46:51.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:51 smithi084 ceph-mon[40024]: 5.d deep-scrub starts 2024-09-20T22:46:51.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:51 smithi084 ceph-mon[40024]: 5.d deep-scrub ok 2024-09-20T22:46:51.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:51 smithi084 ceph-mon[40024]: osdmap e331: 8 total, 8 up, 6 in 2024-09-20T22:46:51.627 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:51.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:51.628 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:46:51.728 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:46:51.751 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:51.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:51.752 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:46:51.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:51 smithi154 ceph-mon[33264]: pgmap v396: 168 pgs: 4 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 162 active+clean; 158 MiB data, 567 MiB used, 536 GiB / 536 GiB avail; 9.2 MiB/s rd, 6.0 MiB/s wr, 24 op/s 2024-09-20T22:46:51.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:51 smithi154 ceph-mon[33264]: 5.d deep-scrub starts 2024-09-20T22:46:51.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:51 smithi154 ceph-mon[33264]: 5.d deep-scrub ok 2024-09-20T22:46:51.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:51 smithi154 ceph-mon[33264]: osdmap e331: 8 total, 8 up, 6 in 2024-09-20T22:46:51.756 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph tell osd.3 reset_purged_snaps_last 2024-09-20T22:46:51.854 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:46:51.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:51.877 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:51.877 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:46:51.978 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:46:51.999 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:52.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:52.000 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:46:52.101 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:46:52.122 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:52.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:52.123 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:46:52.191 INFO:tasks.rados.rados.0.smithi084.stdout:1229: done (5 left) 2024-09-20T22:46:52.191 INFO:tasks.rados.rados.0.smithi084.stdout:1230: delete oid 18 current snap is 120 2024-09-20T22:46:52.191 INFO:tasks.rados.rados.0.smithi084.stdout:1227: expect (ObjNum 397 snap 101 seq_num 397) 2024-09-20T22:46:52.224 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:46:52.249 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:52.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:52.250 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:46:52.253 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:46:52.253 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:46:52.259 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:46:52.351 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:46:52.372 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:52.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:52.373 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:46:52.377 INFO:tasks.rados.rados.0.smithi084.stdout:1224: finishing write tid 3 to smithi08477203-14 2024-09-20T22:46:52.377 INFO:tasks.rados.rados.0.smithi084.stdout:1224: finishing write tid 4 to smithi08477203-14 2024-09-20T22:46:52.377 INFO:tasks.rados.rados.0.smithi084.stdout:1227: done (5 left) 2024-09-20T22:46:52.377 INFO:tasks.rados.rados.0.smithi084.stdout:1230: done (4 left) 2024-09-20T22:46:52.377 INFO:tasks.rados.rados.0.smithi084.stdout:1231: rollback oid 10 current snap is 120 2024-09-20T22:46:52.377 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 10 to 105 2024-09-20T22:46:52.378 INFO:tasks.rados.rados.0.smithi084.stdout:1224: finishing write tid 5 to smithi08477203-14 2024-09-20T22:46:52.378 INFO:tasks.rados.rados.0.smithi084.stdout:1224: oid 14 updating version 0 to 105 2024-09-20T22:46:52.378 INFO:tasks.rados.rados.0.smithi084.stdout:1224: oid 14 version 105 is already newer than 103 2024-09-20T22:46:52.378 INFO:tasks.rados.rados.0.smithi084.stdout:1224: oid 14 version 105 is already newer than 104 2024-09-20T22:46:52.378 INFO:tasks.rados.rados.0.smithi084.stdout:1224: oid 14 version 105 is already newer than 102 2024-09-20T22:46:52.378 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 14 v 105 (ObjNum 448 snap 120 seq_num 448) dirty exists 2024-09-20T22:46:52.378 INFO:tasks.rados.rados.0.smithi084.stdout:1224: left oid 14 (ObjNum 448 snap 120 seq_num 448) 2024-09-20T22:46:52.378 INFO:tasks.rados.rados.0.smithi084.stdout:1225: finishing copy_from racing read to smithi08477203-6 2024-09-20T22:46:52.378 INFO:tasks.rados.rados.0.smithi084.stdout:1226: expect (ObjNum 437 snap 115 seq_num 437) 2024-09-20T22:46:52.474 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:46:52.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:52.496 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:52.496 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:46:52.565 INFO:tasks.rados.rados.0.smithi084.stdout:1228: finishing write tid 1 to smithi08477203-20 2024-09-20T22:46:52.565 INFO:tasks.rados.rados.0.smithi084.stdout:1225: finishing copy_from to smithi08477203-6 2024-09-20T22:46:52.565 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 6 v 126 (ObjNum 433 snap 112 seq_num 433) dirty exists 2024-09-20T22:46:52.565 INFO:tasks.rados.rados.0.smithi084.stdout:1228: finishing write tid 2 to smithi08477203-20 2024-09-20T22:46:52.565 INFO:tasks.rados.rados.0.smithi084.stdout:1224: done (4 left) 2024-09-20T22:46:52.565 INFO:tasks.rados.rados.0.smithi084.stdout:1225: done (3 left) 2024-09-20T22:46:52.565 INFO:tasks.rados.rados.0.smithi084.stdout:1226: done (2 left) 2024-09-20T22:46:52.565 INFO:tasks.rados.rados.0.smithi084.stdout:1232: snap_create 2024-09-20T22:46:52.565 INFO:tasks.rados.rados.0.smithi084.stdout:1228: finishing write tid 3 to smithi08477203-20 2024-09-20T22:46:52.565 INFO:tasks.rados.rados.0.smithi084.stdout:1228: finishing write tid 4 to smithi08477203-20 2024-09-20T22:46:52.565 INFO:tasks.rados.rados.0.smithi084.stdout:1228: finishing write tid 5 to smithi08477203-20 2024-09-20T22:46:52.566 INFO:tasks.rados.rados.0.smithi084.stdout:1228: oid 20 updating version 0 to 95 2024-09-20T22:46:52.566 INFO:tasks.rados.rados.0.smithi084.stdout:1228: oid 20 version 95 is already newer than 93 2024-09-20T22:46:52.566 INFO:tasks.rados.rados.0.smithi084.stdout:1228: oid 20 version 95 is already newer than 92 2024-09-20T22:46:52.566 INFO:tasks.rados.rados.0.smithi084.stdout:1228: oid 20 version 95 is already newer than 94 2024-09-20T22:46:52.566 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 20 v 95 (ObjNum 449 snap 120 seq_num 449) dirty exists 2024-09-20T22:46:52.566 INFO:tasks.rados.rados.0.smithi084.stdout:1228: left oid 20 (ObjNum 449 snap 120 seq_num 449) 2024-09-20T22:46:52.566 INFO:tasks.rados.rados.0.smithi084.stdout:1231: finishing rollback tid 0 to smithi08477203-10 2024-09-20T22:46:52.566 INFO:tasks.rados.rados.0.smithi084.stdout:1231: finishing rollback tid 1 to smithi08477203-10 2024-09-20T22:46:52.566 INFO:tasks.rados.rados.0.smithi084.stdout:1231: finishing rollback tid 2 to smithi08477203-10 2024-09-20T22:46:52.566 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 10 v 397 (ObjNum 419 snap 105 seq_num 419) dirty exists 2024-09-20T22:46:52.597 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:46:52.618 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:52.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:52.619 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:46:52.720 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:46:52.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:52.742 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:52.742 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:46:52.843 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:46:52.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:52.865 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:52.865 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:46:52.928 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-20T22:46:52.928 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:46:52.966 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:46:52.987 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:52.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:52.988 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:46:53.089 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:46:53.110 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:53.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:53.111 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:46:53.182 INFO:tasks.rados.rados.0.smithi084.stdout:1228: done (2 left) 2024-09-20T22:46:53.182 INFO:tasks.rados.rados.0.smithi084.stdout:1231: done (1 left) 2024-09-20T22:46:53.182 INFO:tasks.rados.rados.0.smithi084.stdout:1232: done (0 left) 2024-09-20T22:46:53.182 INFO:tasks.rados.rados.0.smithi084.stdout:1233: read oid 32 snap 119 2024-09-20T22:46:53.182 INFO:tasks.rados.rados.0.smithi084.stdout:1233: expect (ObjNum 401 snap 101 seq_num 401) 2024-09-20T22:46:53.182 INFO:tasks.rados.rados.0.smithi084.stdout:1234: write (excl) oid 16 current snap is 121 2024-09-20T22:46:53.183 INFO:tasks.rados.rados.0.smithi084.stdout:1234: seq_num 451 ranges {779957=402126,1925026=758476,3335652=244305} 2024-09-20T22:46:53.188 INFO:tasks.rados.rados.0.smithi084.stdout:1234: writing smithi08477203-16 from 779957 to 1182083 tid 1 2024-09-20T22:46:53.197 INFO:tasks.rados.rados.0.smithi084.stdout:1234: writing smithi08477203-16 from 1925026 to 2683502 tid 2 2024-09-20T22:46:53.203 INFO:tasks.rados.rados.0.smithi084.stdout:1234: writing smithi08477203-16 from 3335652 to 3579957 tid 3 2024-09-20T22:46:53.203 INFO:tasks.rados.rados.0.smithi084.stdout:1235: read oid 21 snap 113 2024-09-20T22:46:53.203 INFO:tasks.rados.rados.0.smithi084.stdout:1235: expect (ObjNum 387 snap 100 seq_num 387) 2024-09-20T22:46:53.204 INFO:tasks.rados.rados.0.smithi084.stdout:1236: write (excl) oid 5 current snap is 121 2024-09-20T22:46:53.204 INFO:tasks.rados.rados.0.smithi084.stdout:1236: seq_num 452 ranges {796764=477375,1983508=618487,3363075=233689} 2024-09-20T22:46:53.210 INFO:tasks.rados.rados.0.smithi084.stdout:1236: writing smithi08477203-5 from 796764 to 1274139 tid 1 2024-09-20T22:46:53.212 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:46:53.219 INFO:tasks.rados.rados.0.smithi084.stdout:1236: writing smithi08477203-5 from 1983508 to 2601995 tid 2 2024-09-20T22:46:53.225 INFO:tasks.rados.rados.0.smithi084.stdout:1236: writing smithi08477203-5 from 3363075 to 3596764 tid 3 2024-09-20T22:46:53.225 INFO:tasks.rados.rados.0.smithi084.stdout:1237: read oid 29 snap -1 2024-09-20T22:46:53.225 INFO:tasks.rados.rados.0.smithi084.stdout:1237: expect (ObjNum 400 snap 101 seq_num 400) 2024-09-20T22:46:53.225 INFO:tasks.rados.rados.0.smithi084.stdout:1238: read oid 37 snap -1 2024-09-20T22:46:53.225 INFO:tasks.rados.rados.0.smithi084.stdout:1238: expect deleted 2024-09-20T22:46:53.225 INFO:tasks.rados.rados.0.smithi084.stdout:1239: snap_create 2024-09-20T22:46:53.225 INFO:tasks.rados.rados.0.smithi084.stdout:1233: expect (ObjNum 401 snap 101 seq_num 401) 2024-09-20T22:46:53.240 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:53.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:53.242 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:46:53.342 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:46:53.382 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:53.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:53.383 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:46:53.438 INFO:tasks.rados.rados.0.smithi084.stdout:1234: finishing write tid 1 to smithi08477203-16 2024-09-20T22:46:53.438 INFO:tasks.rados.rados.0.smithi084.stdout:1234: finishing write tid 2 to smithi08477203-16 2024-09-20T22:46:53.438 INFO:tasks.rados.rados.0.smithi084.stdout:1234: finishing write tid 3 to smithi08477203-16 2024-09-20T22:46:53.438 INFO:tasks.rados.rados.0.smithi084.stdout:1234: finishing write tid 4 to smithi08477203-16 2024-09-20T22:46:53.438 INFO:tasks.rados.rados.0.smithi084.stdout:1234: finishing write tid 5 to smithi08477203-16 2024-09-20T22:46:53.438 INFO:tasks.rados.rados.0.smithi084.stdout:1234: oid 16 updating version 0 to 188 2024-09-20T22:46:53.438 INFO:tasks.rados.rados.0.smithi084.stdout:1234: oid 16 version 188 is already newer than 186 2024-09-20T22:46:53.438 INFO:tasks.rados.rados.0.smithi084.stdout:1234: oid 16 updating version 188 to 189 2024-09-20T22:46:53.438 INFO:tasks.rados.rados.0.smithi084.stdout:1234: oid 16 version 189 is already newer than 187 2024-09-20T22:46:53.438 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 16 v 189 (ObjNum 451 snap 121 seq_num 451) dirty exists 2024-09-20T22:46:53.439 INFO:tasks.rados.rados.0.smithi084.stdout:1234: left oid 16 (ObjNum 451 snap 121 seq_num 451) 2024-09-20T22:46:53.439 INFO:tasks.rados.rados.0.smithi084.stdout:1235: expect (ObjNum 387 snap 100 seq_num 387) 2024-09-20T22:46:53.474 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:53 smithi084 ceph-mon[40024]: 6.9 scrub starts 2024-09-20T22:46:53.474 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:53 smithi084 ceph-mon[40024]: 6.9 scrub ok 2024-09-20T22:46:53.475 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:53 smithi084 ceph-mon[40024]: pgmap v399: 168 pgs: 4 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 162 active+clean; 158 MiB data, 567 MiB used, 536 GiB / 536 GiB avail; 9.2 MiB/s rd, 6.0 MiB/s wr, 24 op/s 2024-09-20T22:46:53.475 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:53 smithi084 ceph-mon[40024]: 7.d scrub starts 2024-09-20T22:46:53.475 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:53 smithi084 ceph-mon[40024]: 7.d scrub ok 2024-09-20T22:46:53.475 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:53 smithi084 ceph-mon[40024]: osdmap e332: 8 total, 8 up, 6 in 2024-09-20T22:46:53.475 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:53 smithi084 ceph-mon[40024]: 5.1b scrub starts 2024-09-20T22:46:53.475 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:53 smithi084 ceph-mon[40024]: 5.1b scrub ok 2024-09-20T22:46:53.475 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:53 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:46:53.475 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:53 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:46:53.475 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:53 smithi084 ceph-mon[40024]: osdmap e333: 8 total, 8 up, 6 in 2024-09-20T22:46:53.475 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:53 smithi084 ceph-mon[29774]: 6.9 scrub starts 2024-09-20T22:46:53.475 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:53 smithi084 ceph-mon[29774]: 6.9 scrub ok 2024-09-20T22:46:53.475 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:53 smithi084 ceph-mon[29774]: pgmap v399: 168 pgs: 4 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 162 active+clean; 158 MiB data, 567 MiB used, 536 GiB / 536 GiB avail; 9.2 MiB/s rd, 6.0 MiB/s wr, 24 op/s 2024-09-20T22:46:53.475 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:53 smithi084 ceph-mon[29774]: 7.d scrub starts 2024-09-20T22:46:53.476 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:53 smithi084 ceph-mon[29774]: 7.d scrub ok 2024-09-20T22:46:53.476 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:53 smithi084 ceph-mon[29774]: osdmap e332: 8 total, 8 up, 6 in 2024-09-20T22:46:53.476 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:53 smithi084 ceph-mon[29774]: 5.1b scrub starts 2024-09-20T22:46:53.476 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:53 smithi084 ceph-mon[29774]: 5.1b scrub ok 2024-09-20T22:46:53.476 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:53 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:46:53.476 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:53 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:46:53.476 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:53 smithi084 ceph-mon[29774]: osdmap e333: 8 total, 8 up, 6 in 2024-09-20T22:46:53.484 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:46:53.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:53 smithi154 ceph-mon[33264]: 6.9 scrub starts 2024-09-20T22:46:53.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:53 smithi154 ceph-mon[33264]: 6.9 scrub ok 2024-09-20T22:46:53.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:53 smithi154 ceph-mon[33264]: pgmap v399: 168 pgs: 4 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 162 active+clean; 158 MiB data, 567 MiB used, 536 GiB / 536 GiB avail; 9.2 MiB/s rd, 6.0 MiB/s wr, 24 op/s 2024-09-20T22:46:53.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:53 smithi154 ceph-mon[33264]: 7.d scrub starts 2024-09-20T22:46:53.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:53 smithi154 ceph-mon[33264]: 7.d scrub ok 2024-09-20T22:46:53.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:53 smithi154 ceph-mon[33264]: osdmap e332: 8 total, 8 up, 6 in 2024-09-20T22:46:53.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:53 smithi154 ceph-mon[33264]: 5.1b scrub starts 2024-09-20T22:46:53.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:53 smithi154 ceph-mon[33264]: 5.1b scrub ok 2024-09-20T22:46:53.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:53 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:46:53.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:53 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:46:53.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:53 smithi154 ceph-mon[33264]: osdmap e333: 8 total, 8 up, 6 in 2024-09-20T22:46:53.506 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:53.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:53.507 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:46:53.608 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:46:53.629 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:53.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:53.630 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:46:53.671 INFO:tasks.rados.rados.0.smithi084.stdout:1236: finishing write tid 1 to smithi08477203-5 2024-09-20T22:46:53.672 INFO:tasks.rados.rados.0.smithi084.stdout:1236: finishing write tid 2 to smithi08477203-5 2024-09-20T22:46:53.672 INFO:tasks.rados.rados.0.smithi084.stdout:1237: expect (ObjNum 400 snap 101 seq_num 400) 2024-09-20T22:46:53.731 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:46:53.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:53.753 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:53.754 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:46:53.854 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:46:53.876 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:53.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:53.877 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:46:53.888 INFO:tasks.rados.rados.0.smithi084.stdout:1236: finishing write tid 3 to smithi08477203-5 2024-09-20T22:46:53.888 INFO:tasks.rados.rados.0.smithi084.stdout:1236: finishing write tid 4 to smithi08477203-5 2024-09-20T22:46:53.888 INFO:tasks.rados.rados.0.smithi084.stdout:1236: finishing write tid 5 to smithi08477203-5 2024-09-20T22:46:53.888 INFO:tasks.rados.rados.0.smithi084.stdout:1236: oid 5 updating version 0 to 386 2024-09-20T22:46:53.888 INFO:tasks.rados.rados.0.smithi084.stdout:1236: oid 5 updating version 386 to 388 2024-09-20T22:46:53.888 INFO:tasks.rados.rados.0.smithi084.stdout:1236: oid 5 version 388 is already newer than 385 2024-09-20T22:46:53.888 INFO:tasks.rados.rados.0.smithi084.stdout:1236: oid 5 version 388 is already newer than 387 2024-09-20T22:46:53.888 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 5 v 388 (ObjNum 452 snap 121 seq_num 452) dirty exists 2024-09-20T22:46:53.888 INFO:tasks.rados.rados.0.smithi084.stdout:1236: left oid 5 (ObjNum 452 snap 121 seq_num 452) 2024-09-20T22:46:53.978 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:46:53.999 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:54.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:54.000 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:46:54.088 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.1 dump_blocked_ops 2024-09-20T22:46:54.101 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:46:54.122 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:54.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:54.123 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:46:54.164 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-20T22:46:54.201 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is set 2024-09-20T22:46:54.205 INFO:tasks.rados.rados.0.smithi084.stdout:1233: done (6 left) 2024-09-20T22:46:54.205 INFO:tasks.rados.rados.0.smithi084.stdout:1234: done (5 left) 2024-09-20T22:46:54.205 INFO:tasks.rados.rados.0.smithi084.stdout:1235: done (4 left) 2024-09-20T22:46:54.205 INFO:tasks.rados.rados.0.smithi084.stdout:1236: done (3 left) 2024-09-20T22:46:54.205 INFO:tasks.rados.rados.0.smithi084.stdout:1237: done (2 left) 2024-09-20T22:46:54.205 INFO:tasks.rados.rados.0.smithi084.stdout:1238: done (1 left) 2024-09-20T22:46:54.205 INFO:tasks.rados.rados.0.smithi084.stdout:1239: done (0 left) 2024-09-20T22:46:54.206 INFO:tasks.rados.rados.0.smithi084.stdout:1240: read oid 2 snap -1 2024-09-20T22:46:54.206 INFO:tasks.rados.rados.0.smithi084.stdout:1240: expect (ObjNum 426 snap 109 seq_num 426) 2024-09-20T22:46:54.206 INFO:tasks.rados.rados.0.smithi084.stdout:1241: rollback oid 33 current snap is 122 2024-09-20T22:46:54.206 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 33 to 110 2024-09-20T22:46:54.206 INFO:tasks.rados.rados.0.smithi084.stdout:1242: read oid 10 snap -1 2024-09-20T22:46:54.206 INFO:tasks.rados.rados.0.smithi084.stdout:1242: expect (ObjNum 419 snap 105 seq_num 419) 2024-09-20T22:46:54.206 INFO:tasks.rados.rados.0.smithi084.stdout:1243: snap_create 2024-09-20T22:46:54.217 INFO:tasks.rados.rados.0.smithi084.stdout:1241: finishing rollback tid 0 to smithi08477203-33 2024-09-20T22:46:54.217 INFO:tasks.rados.rados.0.smithi084.stdout:1241: finishing rollback tid 1 to smithi08477203-33 2024-09-20T22:46:54.217 INFO:tasks.rados.rados.0.smithi084.stdout:1241: finishing rollback tid 2 to smithi08477203-33 2024-09-20T22:46:54.217 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 33 v 388 (ObjNum 395 snap 101 seq_num 395) dirty exists 2024-09-20T22:46:54.217 INFO:tasks.rados.rados.0.smithi084.stdout:1242: expect (ObjNum 419 snap 105 seq_num 419) 2024-09-20T22:46:54.224 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:46:54.246 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:54.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:54.247 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:46:54.348 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:46:54.364 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:54 smithi084 ceph-mon[29774]: 3.e deep-scrub starts 2024-09-20T22:46:54.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:54 smithi084 ceph-mon[29774]: 3.e deep-scrub ok 2024-09-20T22:46:54.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:54 smithi084 ceph-mon[29774]: 7.c scrub starts 2024-09-20T22:46:54.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:54 smithi084 ceph-mon[29774]: 7.c scrub ok 2024-09-20T22:46:54.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:54 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/3149951882' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-20T22:46:54.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:54 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-20T22:46:54.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:54 smithi084 ceph-mon[40024]: 3.e deep-scrub starts 2024-09-20T22:46:54.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:54 smithi084 ceph-mon[40024]: 3.e deep-scrub ok 2024-09-20T22:46:54.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:54 smithi084 ceph-mon[40024]: 7.c scrub starts 2024-09-20T22:46:54.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:54 smithi084 ceph-mon[40024]: 7.c scrub ok 2024-09-20T22:46:54.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:54 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/3149951882' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-20T22:46:54.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:54 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-20T22:46:54.369 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:54.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:54.370 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:46:54.402 INFO:tasks.rados.rados.0.smithi084.stdout:1240: expect (ObjNum 426 snap 109 seq_num 426) 2024-09-20T22:46:54.471 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:46:54.492 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:54.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:54.492 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:46:54.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:54 smithi154 ceph-mon[33264]: 3.e deep-scrub starts 2024-09-20T22:46:54.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:54 smithi154 ceph-mon[33264]: 3.e deep-scrub ok 2024-09-20T22:46:54.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:54 smithi154 ceph-mon[33264]: 7.c scrub starts 2024-09-20T22:46:54.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:54 smithi154 ceph-mon[33264]: 7.c scrub ok 2024-09-20T22:46:54.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:54 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/3149951882' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-20T22:46:54.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:54 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-20T22:46:54.593 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:46:54.616 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:54.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:54.617 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:46:54.717 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:46:54.740 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:54.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:54.741 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:46:54.800 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph tell osd.5 reset_purged_snaps_last 2024-09-20T22:46:54.841 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:46:54.863 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:54.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:54.864 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:46:54.965 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:46:54.988 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:54.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:54.989 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:46:55.089 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:46:55.094 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:46:55.094 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:46:55.096 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:46:55.114 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:55.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:55.116 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:46:55.208 INFO:tasks.rados.rados.0.smithi084.stdout:1240: done (3 left) 2024-09-20T22:46:55.208 INFO:tasks.rados.rados.0.smithi084.stdout:1241: done (2 left) 2024-09-20T22:46:55.208 INFO:tasks.rados.rados.0.smithi084.stdout:1242: done (1 left) 2024-09-20T22:46:55.208 INFO:tasks.rados.rados.0.smithi084.stdout:1243: done (0 left) 2024-09-20T22:46:55.208 INFO:tasks.rados.rados.0.smithi084.stdout:1244: write (excl) oid 16 current snap is 123 2024-09-20T22:46:55.208 INFO:tasks.rados.rados.0.smithi084.stdout:1244: seq_num 453 ranges {413571=552624,1641990=762145,2990498=425163} 2024-09-20T22:46:55.213 INFO:tasks.rados.rados.0.smithi084.stdout:1244: writing smithi08477203-16 from 413571 to 966195 tid 1 2024-09-20T22:46:55.216 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:46:55.222 INFO:tasks.rados.rados.0.smithi084.stdout:1244: writing smithi08477203-16 from 1641990 to 2404135 tid 2 2024-09-20T22:46:55.228 INFO:tasks.rados.rados.0.smithi084.stdout:1244: writing smithi08477203-16 from 2990498 to 3415661 tid 3 2024-09-20T22:46:55.229 INFO:tasks.rados.rados.0.smithi084.stdout:1245: rollback oid 5 current snap is 123 2024-09-20T22:46:55.229 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 5 to 105 2024-09-20T22:46:55.229 INFO:tasks.rados.rados.0.smithi084.stdout:1244: finishing write tid 1 to smithi08477203-16 2024-09-20T22:46:55.229 INFO:tasks.rados.rados.0.smithi084.stdout:1244: finishing write tid 2 to smithi08477203-16 2024-09-20T22:46:55.229 INFO:tasks.rados.rados.0.smithi084.stdout:1246: write (excl) oid 25 current snap is 123 2024-09-20T22:46:55.229 INFO:tasks.rados.rados.0.smithi084.stdout:1246: seq_num 454 ranges {430378=627873,1700472=622156,3017921=612457} 2024-09-20T22:46:55.234 INFO:tasks.rados.rados.0.smithi084.stdout:1246: writing smithi08477203-25 from 430378 to 1058251 tid 1 2024-09-20T22:46:55.240 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:55.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:55.241 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:46:55.241 INFO:tasks.rados.rados.0.smithi084.stdout:1246: writing smithi08477203-25 from 1700472 to 2322628 tid 2 2024-09-20T22:46:55.249 INFO:tasks.rados.rados.0.smithi084.stdout:1246: writing smithi08477203-25 from 3017921 to 3630378 tid 3 2024-09-20T22:46:55.249 INFO:tasks.rados.rados.0.smithi084.stdout:1247: snap_create 2024-09-20T22:46:55.249 INFO:tasks.rados.rados.0.smithi084.stdout:1244: finishing write tid 3 to smithi08477203-16 2024-09-20T22:46:55.249 INFO:tasks.rados.rados.0.smithi084.stdout:1244: finishing write tid 4 to smithi08477203-16 2024-09-20T22:46:55.249 INFO:tasks.rados.rados.0.smithi084.stdout:1244: finishing write tid 5 to smithi08477203-16 2024-09-20T22:46:55.249 INFO:tasks.rados.rados.0.smithi084.stdout:1244: oid 16 updating version 0 to 191 2024-09-20T22:46:55.249 INFO:tasks.rados.rados.0.smithi084.stdout:1244: oid 16 version 191 is already newer than 190 2024-09-20T22:46:55.249 INFO:tasks.rados.rados.0.smithi084.stdout:1244: oid 16 updating version 191 to 192 2024-09-20T22:46:55.250 INFO:tasks.rados.rados.0.smithi084.stdout:1244: oid 16 updating version 192 to 193 2024-09-20T22:46:55.250 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 16 v 193 (ObjNum 453 snap 123 seq_num 453) dirty exists 2024-09-20T22:46:55.250 INFO:tasks.rados.rados.0.smithi084.stdout:1244: left oid 16 (ObjNum 453 snap 123 seq_num 453) 2024-09-20T22:46:55.250 INFO:tasks.rados.rados.0.smithi084.stdout:1245: finishing rollback tid 0 to smithi08477203-5 2024-09-20T22:46:55.250 INFO:tasks.rados.rados.0.smithi084.stdout:1245: finishing rollback tid 1 to smithi08477203-5 2024-09-20T22:46:55.250 INFO:tasks.rados.rados.0.smithi084.stdout:1245: finishing rollback tid 2 to smithi08477203-5 2024-09-20T22:46:55.250 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 5 v 391 (ObjNum 375 snap 99 seq_num 375) dirty exists 2024-09-20T22:46:55.250 INFO:tasks.rados.rados.0.smithi084.stdout:1246: finishing write tid 1 to smithi08477203-25 2024-09-20T22:46:55.250 INFO:tasks.rados.rados.0.smithi084.stdout:1246: finishing write tid 2 to smithi08477203-25 2024-09-20T22:46:55.251 INFO:tasks.rados.rados.0.smithi084.stdout:1246: finishing write tid 3 to smithi08477203-25 2024-09-20T22:46:55.251 INFO:tasks.rados.rados.0.smithi084.stdout:1246: finishing write tid 4 to smithi08477203-25 2024-09-20T22:46:55.251 INFO:tasks.rados.rados.0.smithi084.stdout:1246: finishing write tid 5 to smithi08477203-25 2024-09-20T22:46:55.252 INFO:tasks.rados.rados.0.smithi084.stdout:1246: oid 25 updating version 0 to 671 2024-09-20T22:46:55.252 INFO:tasks.rados.rados.0.smithi084.stdout:1246: oid 25 version 671 is already newer than 669 2024-09-20T22:46:55.252 INFO:tasks.rados.rados.0.smithi084.stdout:1246: oid 25 version 671 is already newer than 670 2024-09-20T22:46:55.252 INFO:tasks.rados.rados.0.smithi084.stdout:1246: oid 25 version 671 is already newer than 668 2024-09-20T22:46:55.252 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 25 v 671 (ObjNum 454 snap 123 seq_num 454) dirty exists 2024-09-20T22:46:55.252 INFO:tasks.rados.rados.0.smithi084.stdout:1246: left oid 25 (ObjNum 454 snap 123 seq_num 454) 2024-09-20T22:46:55.341 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:46:55.365 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:55.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:55.366 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:46:55.467 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:46:55.479 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:55 smithi084 ceph-mon[40024]: pgmap v402: 168 pgs: 3 active+clean+snaptrim, 165 active+clean; 173 MiB data, 622 MiB used, 536 GiB / 536 GiB avail; 20 MiB/s rd, 2.5 MiB/s wr, 25 op/s 2024-09-20T22:46:55.479 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:55 smithi084 ceph-mon[40024]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:46:55.479 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:55 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T22:46:55.480 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:55 smithi084 ceph-mon[40024]: osdmap e334: 8 total, 8 up, 6 in 2024-09-20T22:46:55.480 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:55 smithi084 ceph-mon[29774]: pgmap v402: 168 pgs: 3 active+clean+snaptrim, 165 active+clean; 173 MiB data, 622 MiB used, 536 GiB / 536 GiB avail; 20 MiB/s rd, 2.5 MiB/s wr, 25 op/s 2024-09-20T22:46:55.480 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:55 smithi084 ceph-mon[29774]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:46:55.480 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:55 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T22:46:55.480 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:55 smithi084 ceph-mon[29774]: osdmap e334: 8 total, 8 up, 6 in 2024-09-20T22:46:55.490 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:55.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:55.491 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:46:55.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:55 smithi154 ceph-mon[33264]: pgmap v402: 168 pgs: 3 active+clean+snaptrim, 165 active+clean; 173 MiB data, 622 MiB used, 536 GiB / 536 GiB avail; 20 MiB/s rd, 2.5 MiB/s wr, 25 op/s 2024-09-20T22:46:55.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:55 smithi154 ceph-mon[33264]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:46:55.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:55 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T22:46:55.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:55 smithi154 ceph-mon[33264]: osdmap e334: 8 total, 8 up, 6 in 2024-09-20T22:46:55.592 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:46:55.615 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:55.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:55.616 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:46:55.716 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:46:55.741 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:55.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:55.742 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:46:55.843 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:46:55.868 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:55.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:55.869 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:46:55.920 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:46:55.921 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:46:55.937 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:46:55.937 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:46:55.953 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:46:55.953 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:46:55.968 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-20T22:46:55.969 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T22:46:55.970 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:46:55.986 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:46:55.986 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:46:55.995 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:55.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:55.996 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:46:56.003 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:46:56.003 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:46:56.020 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:46:56.020 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:46:56.036 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:46:56.037 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:46:56.097 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:46:56.122 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:56.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:56.123 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:46:56.212 INFO:tasks.rados.rados.0.smithi084.stdout:1244: done (3 left) 2024-09-20T22:46:56.213 INFO:tasks.rados.rados.0.smithi084.stdout:1245: done (2 left) 2024-09-20T22:46:56.213 INFO:tasks.rados.rados.0.smithi084.stdout:1246: done (1 left) 2024-09-20T22:46:56.213 INFO:tasks.rados.rados.0.smithi084.stdout:1247: done (0 left) 2024-09-20T22:46:56.213 INFO:tasks.rados.rados.0.smithi084.stdout:1248: snap_create 2024-09-20T22:46:56.223 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:46:56.248 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:56.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:56.249 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:46:56.349 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:46:56.364 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:56 smithi084 ceph-mon[40024]: osdmap e335: 8 total, 8 up, 6 in 2024-09-20T22:46:56.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:56 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/1685339181' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:46:56.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:56 smithi084 ceph-mon[29774]: osdmap e335: 8 total, 8 up, 6 in 2024-09-20T22:46:56.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:56 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/1685339181' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:46:56.374 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:56.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:56.375 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:46:56.476 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:46:56.498 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:56.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:56.499 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:46:56.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:56 smithi154 ceph-mon[33264]: osdmap e335: 8 total, 8 up, 6 in 2024-09-20T22:46:56.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:56 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/1685339181' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:46:56.600 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:46:56.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:56.627 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:56.628 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:46:56.729 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:46:56.755 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:56.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:56.756 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:46:56.856 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:46:56.881 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:56.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:56.882 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:46:56.983 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:46:57.009 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:57.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:57.010 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:46:57.111 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:46:57.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:57.137 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:57.137 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:46:57.214 INFO:tasks.rados.rados.0.smithi084.stdout:1248: done (0 left) 2024-09-20T22:46:57.214 INFO:tasks.rados.rados.0.smithi084.stdout:1249: copy_from oid 40 from oid 35 current snap is 125 2024-09-20T22:46:57.214 INFO:tasks.rados.rados.0.smithi084.stdout:1250: rollback oid 13 current snap is 125 2024-09-20T22:46:57.214 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 13 to 118 2024-09-20T22:46:57.214 INFO:tasks.rados.rados.0.smithi084.stdout:1251: copy_from oid 14 from oid 8 current snap is 125 2024-09-20T22:46:57.214 INFO:tasks.rados.rados.0.smithi084.stdout:1252: snap_remove snap 110 2024-09-20T22:46:57.238 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:46:57.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:57.261 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:57.262 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:46:57.360 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:57 smithi084 ceph-mon[40024]: pgmap v405: 168 pgs: 3 active+clean+snaptrim, 165 active+clean; 173 MiB data, 622 MiB used, 536 GiB / 536 GiB avail; 20 MiB/s rd, 2.5 MiB/s wr, 25 op/s 2024-09-20T22:46:57.361 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:57 smithi084 ceph-mon[40024]: osdmap e336: 8 total, 8 up, 6 in 2024-09-20T22:46:57.361 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:57 smithi084 ceph-mon[29774]: pgmap v405: 168 pgs: 3 active+clean+snaptrim, 165 active+clean; 173 MiB data, 622 MiB used, 536 GiB / 536 GiB avail; 20 MiB/s rd, 2.5 MiB/s wr, 25 op/s 2024-09-20T22:46:57.361 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:57 smithi084 ceph-mon[29774]: osdmap e336: 8 total, 8 up, 6 in 2024-09-20T22:46:57.363 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:46:57.386 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:57.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:57.387 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:46:57.488 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:46:57.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:57 smithi154 ceph-mon[33264]: pgmap v405: 168 pgs: 3 active+clean+snaptrim, 165 active+clean; 173 MiB data, 622 MiB used, 536 GiB / 536 GiB avail; 20 MiB/s rd, 2.5 MiB/s wr, 25 op/s 2024-09-20T22:46:57.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:57 smithi154 ceph-mon[33264]: osdmap e336: 8 total, 8 up, 6 in 2024-09-20T22:46:57.512 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:57.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:57.512 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:46:57.613 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:46:57.635 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:57.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:57.636 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:46:57.736 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:46:57.757 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:57.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:57.759 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:46:57.860 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:46:57.882 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:57.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:57.883 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:46:57.942 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph tell osd.6 reset_purged_snaps_last 2024-09-20T22:46:57.983 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:46:58.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:58.008 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:58.008 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:46:58.098 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-20T22:46:58.098 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2024-09-20T22:46:58.098 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2024-09-20T22:46:58.098 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2024-09-20T22:46:58.098 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-20T22:46:58.109 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:46:58.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:58.131 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:58.131 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:46:58.182 INFO:tasks.rados.rados.0.smithi084.stdout:1252: done (3 left) 2024-09-20T22:46:58.182 INFO:tasks.rados.rados.0.smithi084.stdout:1253: copy_from oid 27 from oid 45 current snap is 125 2024-09-20T22:46:58.182 INFO:tasks.rados.rados.0.smithi084.stdout:1250: finishing rollback tid 0 to smithi08477203-13 2024-09-20T22:46:58.182 INFO:tasks.rados.rados.0.smithi084.stdout:1250: finishing rollback tid 1 to smithi08477203-13 2024-09-20T22:46:58.182 INFO:tasks.rados.rados.0.smithi084.stdout:1250: finishing rollback tid 2 to smithi08477203-13 2024-09-20T22:46:58.182 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 13 v 675 (ObjNum 411 snap 105 seq_num 411) dirty exists 2024-09-20T22:46:58.182 INFO:tasks.rados.rados.0.smithi084.stdout:1249: finishing copy_from racing read to smithi08477203-40 2024-09-20T22:46:58.182 INFO:tasks.rados.rados.0.smithi084.stdout:1249: finishing copy_from to smithi08477203-40 2024-09-20T22:46:58.182 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 40 v 199 (ObjNum 245 snap 54 seq_num 245) dirty exists 2024-09-20T22:46:58.183 INFO:tasks.rados.rados.0.smithi084.stdout:1251: finishing copy_from racing read to smithi08477203-14 2024-09-20T22:46:58.183 INFO:tasks.rados.rados.0.smithi084.stdout:1251: finishing copy_from to smithi08477203-14 2024-09-20T22:46:58.183 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 14 v 187 (ObjNum 438 snap 115 seq_num 438) dirty exists 2024-09-20T22:46:58.183 INFO:tasks.rados.rados.0.smithi084.stdout:1249: done (3 left) 2024-09-20T22:46:58.183 INFO:tasks.rados.rados.0.smithi084.stdout:1250: done (2 left) 2024-09-20T22:46:58.183 INFO:tasks.rados.rados.0.smithi084.stdout:1251: done (1 left) 2024-09-20T22:46:58.183 INFO:tasks.rados.rados.0.smithi084.stdout:1254: copy_from oid 18 from oid 47 current snap is 125 2024-09-20T22:46:58.183 INFO:tasks.rados.rados.0.smithi084.stdout:1255: read oid 50 snap -1 2024-09-20T22:46:58.183 INFO:tasks.rados.rados.0.smithi084.stdout:1255: expect (ObjNum 338 snap 85 seq_num 338) 2024-09-20T22:46:58.183 INFO:tasks.rados.rados.0.smithi084.stdout:1256: read oid 48 snap 113 2024-09-20T22:46:58.183 INFO:tasks.rados.rados.0.smithi084.stdout:1256: expect (ObjNum 424 snap 109 seq_num 424) 2024-09-20T22:46:58.183 INFO:tasks.rados.rados.0.smithi084.stdout:1257: read oid 25 snap 114 2024-09-20T22:46:58.183 INFO:tasks.rados.rados.0.smithi084.stdout:1257: expect deleted 2024-09-20T22:46:58.183 INFO:tasks.rados.rados.0.smithi084.stdout:1258: snap_remove snap 115 2024-09-20T22:46:58.232 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:46:58.253 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:58.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:58.254 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:46:58.355 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:46:58.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:58.377 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:58.377 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:46:58.478 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:46:58.500 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:58.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:58.501 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:46:58.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:58 smithi154 ceph-mon[33264]: osdmap e337: 8 total, 8 up, 6 in 2024-09-20T22:46:58.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:58 smithi154 ceph-mon[33264]: osdmap e338: 8 total, 8 up, 6 in 2024-09-20T22:46:58.542 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:58 smithi084 ceph-mon[40024]: osdmap e337: 8 total, 8 up, 6 in 2024-09-20T22:46:58.542 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:58 smithi084 ceph-mon[40024]: osdmap e338: 8 total, 8 up, 6 in 2024-09-20T22:46:58.542 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:46:58 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: ::ffff:172.21.15.154 - - [20/Sep/2024:22:46:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-20T22:46:58.542 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:58 smithi084 ceph-mon[29774]: osdmap e337: 8 total, 8 up, 6 in 2024-09-20T22:46:58.543 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:58 smithi084 ceph-mon[29774]: osdmap e338: 8 total, 8 up, 6 in 2024-09-20T22:46:58.602 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:46:58.623 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:58.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:58.624 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:46:58.688 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.1 dump_historic_ops 2024-09-20T22:46:58.724 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:46:58.746 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:58.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:58.747 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:46:58.848 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:46:58.869 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:58.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:58.870 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:46:58.971 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:46:58.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:58.994 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:58.994 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:46:59.095 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:46:59.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:59.118 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:59.118 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:46:59.188 INFO:tasks.rados.rados.0.smithi084.stdout:1258: done (5 left) 2024-09-20T22:46:59.188 INFO:tasks.rados.rados.0.smithi084.stdout:1259: write (excl) oid 14 current snap is 125 2024-09-20T22:46:59.188 INFO:tasks.rados.rados.0.smithi084.stdout:1259: seq_num 455 ranges {447185=703122,1642601=765814,2928991=447707} 2024-09-20T22:46:59.194 INFO:tasks.rados.rados.0.smithi084.stdout:1259: writing smithi08477203-14 from 447185 to 1150307 tid 1 2024-09-20T22:46:59.206 INFO:tasks.rados.rados.0.smithi084.stdout:1259: writing smithi08477203-14 from 1642601 to 2408415 tid 2 2024-09-20T22:46:59.212 INFO:tasks.rados.rados.0.smithi084.stdout:1259: writing smithi08477203-14 from 2928991 to 3376698 tid 3 2024-09-20T22:46:59.212 INFO:tasks.rados.rados.0.smithi084.stdout:1260: delete oid 36 current snap is 125 2024-09-20T22:46:59.212 INFO:tasks.rados.rados.0.smithi084.stdout:1255: expect (ObjNum 338 snap 85 seq_num 338) 2024-09-20T22:46:59.219 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:46:59.240 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:46:59.245 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:59.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:59.246 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:46:59.277 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:46:59.347 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:46:59.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:59.370 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:59.370 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:46:59.390 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-20T22:46:59.390 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:59 smithi084 ceph-mon[29774]: pgmap v408: 168 pgs: 3 active+clean+snaptrim, 165 active+clean; 173 MiB data, 622 MiB used, 536 GiB / 536 GiB avail 2024-09-20T22:46:59.391 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:46:59 smithi084 ceph-mon[29774]: osdmap e339: 8 total, 8 up, 6 in 2024-09-20T22:46:59.391 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:59 smithi084 ceph-mon[40024]: pgmap v408: 168 pgs: 3 active+clean+snaptrim, 165 active+clean; 173 MiB data, 622 MiB used, 536 GiB / 536 GiB avail 2024-09-20T22:46:59.391 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:46:59 smithi084 ceph-mon[40024]: osdmap e339: 8 total, 8 up, 6 in 2024-09-20T22:46:59.394 INFO:tasks.rados.rados.0.smithi084.stdout:1254: finishing copy_from racing read to smithi08477203-18 2024-09-20T22:46:59.394 INFO:tasks.rados.rados.0.smithi084.stdout:1256: expect (ObjNum 424 snap 109 seq_num 424) 2024-09-20T22:46:59.471 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:46:59.492 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:59.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:59.493 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:46:59.504 INFO:journalctl@ceph.iscsi.iscsi.a.smithi154.stdout:Sep 20 22:46:59 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-iscsi-iscsi-a[59730]: debug there is no tcmu-runner data available 2024-09-20T22:46:59.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:59 smithi154 ceph-mon[33264]: pgmap v408: 168 pgs: 3 active+clean+snaptrim, 165 active+clean; 173 MiB data, 622 MiB used, 536 GiB / 536 GiB avail 2024-09-20T22:46:59.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:46:59 smithi154 ceph-mon[33264]: osdmap e339: 8 total, 8 up, 6 in 2024-09-20T22:46:59.514 INFO:tasks.rados.rados.0.smithi084.stdout:1253: finishing copy_from racing read to smithi08477203-27 2024-09-20T22:46:59.514 INFO:tasks.rados.rados.0.smithi084.stdout:1254: finishing copy_from to smithi08477203-18 2024-09-20T22:46:59.514 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 18 v 252 (ObjNum 390 snap 101 seq_num 390) dirty exists 2024-09-20T22:46:59.514 INFO:tasks.rados.rados.0.smithi084.stdout:1254: done (6 left) 2024-09-20T22:46:59.514 INFO:tasks.rados.rados.0.smithi084.stdout:1255: done (5 left) 2024-09-20T22:46:59.514 INFO:tasks.rados.rados.0.smithi084.stdout:1256: done (4 left) 2024-09-20T22:46:59.514 INFO:tasks.rados.rados.0.smithi084.stdout:1257: done (3 left) 2024-09-20T22:46:59.514 INFO:tasks.rados.rados.0.smithi084.stdout:1260: done (2 left) 2024-09-20T22:46:59.515 INFO:tasks.rados.rados.0.smithi084.stdout:1261: copy_from oid 26 from oid 46 current snap is 125 2024-09-20T22:46:59.515 INFO:tasks.rados.rados.0.smithi084.stdout:1253: finishing copy_from to smithi08477203-27 2024-09-20T22:46:59.515 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 27 v 114 (ObjNum 338 snap 85 seq_num 338) dirty exists 2024-09-20T22:46:59.515 INFO:tasks.rados.rados.0.smithi084.stdout:1259: finishing write tid 1 to smithi08477203-14 2024-09-20T22:46:59.515 INFO:tasks.rados.rados.0.smithi084.stdout:1259: finishing write tid 2 to smithi08477203-14 2024-09-20T22:46:59.515 INFO:tasks.rados.rados.0.smithi084.stdout:1259: finishing write tid 3 to smithi08477203-14 2024-09-20T22:46:59.515 INFO:tasks.rados.rados.0.smithi084.stdout:1259: finishing write tid 4 to smithi08477203-14 2024-09-20T22:46:59.515 INFO:tasks.rados.rados.0.smithi084.stdout:1259: finishing write tid 5 to smithi08477203-14 2024-09-20T22:46:59.515 INFO:tasks.rados.rados.0.smithi084.stdout:1259: oid 14 updating version 0 to 191 2024-09-20T22:46:59.515 INFO:tasks.rados.rados.0.smithi084.stdout:1259: oid 14 version 191 is already newer than 188 2024-09-20T22:46:59.515 INFO:tasks.rados.rados.0.smithi084.stdout:1259: oid 14 version 191 is already newer than 189 2024-09-20T22:46:59.515 INFO:tasks.rados.rados.0.smithi084.stdout:1259: oid 14 version 191 is already newer than 190 2024-09-20T22:46:59.515 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 14 v 191 (ObjNum 455 snap 125 seq_num 455) dirty exists 2024-09-20T22:46:59.515 INFO:tasks.rados.rados.0.smithi084.stdout:1259: left oid 14 (ObjNum 455 snap 125 seq_num 455) 2024-09-20T22:46:59.516 INFO:tasks.rados.rados.0.smithi084.stdout:1253: done (2 left) 2024-09-20T22:46:59.516 INFO:tasks.rados.rados.0.smithi084.stdout:1259: done (1 left) 2024-09-20T22:46:59.516 INFO:tasks.rados.rados.0.smithi084.stdout:1262: snap_create 2024-09-20T22:46:59.516 INFO:tasks.rados.rados.0.smithi084.stdout:1261: finishing copy_from to smithi08477203-26 2024-09-20T22:46:59.516 INFO:tasks.rados.rados.0.smithi084.stdout:1261: got expected ENOENT (src dne) 2024-09-20T22:46:59.516 INFO:tasks.rados.rados.0.smithi084.stdout:1261: finishing copy_from racing read to smithi08477203-26 2024-09-20T22:46:59.595 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:46:59.616 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:59.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:59.617 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:46:59.717 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:46:59.738 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:46:59.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:59.739 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:46:59.840 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:46:59.862 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:59.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:59.863 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:46:59.964 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:46:59.986 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:46:59.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:46:59.987 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:47:00.087 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:47:00.109 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:00.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:00.110 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:47:00.211 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:47:00.233 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:00.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:00.234 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:47:00.247 INFO:tasks.rados.rados.0.smithi084.stdout:1261: done (1 left) 2024-09-20T22:47:00.247 INFO:tasks.rados.rados.0.smithi084.stdout:1262: done (0 left) 2024-09-20T22:47:00.247 INFO:tasks.rados.rados.0.smithi084.stdout:1263: write (excl) oid 32 current snap is 126 2024-09-20T22:47:00.247 INFO:tasks.rados.rados.0.smithi084.stdout:1263: seq_num 457 ranges {480799=569973,1475918=769483,2700190=586604} 2024-09-20T22:47:00.253 INFO:tasks.rados.rados.0.smithi084.stdout:1263: writing smithi08477203-32 from 480799 to 1050772 tid 1 2024-09-20T22:47:00.260 INFO:tasks.rados.rados.0.smithi084.stdout:1263: writing smithi08477203-32 from 1475918 to 2245401 tid 2 2024-09-20T22:47:00.266 INFO:tasks.rados.rados.0.smithi084.stdout:1263: writing smithi08477203-32 from 2700190 to 3286794 tid 3 2024-09-20T22:47:00.266 INFO:tasks.rados.rados.0.smithi084.stdout:1264: read oid 34 snap -1 2024-09-20T22:47:00.266 INFO:tasks.rados.rados.0.smithi084.stdout:1264: expect (ObjNum 359 snap 92 seq_num 359) 2024-09-20T22:47:00.266 INFO:tasks.rados.rados.0.smithi084.stdout:1265: write (excl) oid 25 current snap is 126 2024-09-20T22:47:00.266 INFO:tasks.rados.rados.0.smithi084.stdout:1263: finishing write tid 1 to smithi08477203-32 2024-09-20T22:47:00.266 INFO:tasks.rados.rados.0.smithi084.stdout:1265: seq_num 458 ranges {497606=645222,1934400=629494,3243966=453640} 2024-09-20T22:47:00.272 INFO:tasks.rados.rados.0.smithi084.stdout:1265: writing smithi08477203-25 from 497606 to 1142828 tid 1 2024-09-20T22:47:00.280 INFO:tasks.rados.rados.0.smithi084.stdout:1265: writing smithi08477203-25 from 1934400 to 2563894 tid 2 2024-09-20T22:47:00.286 INFO:tasks.rados.rados.0.smithi084.stdout:1265: writing smithi08477203-25 from 3243966 to 3697606 tid 3 2024-09-20T22:47:00.286 INFO:tasks.rados.rados.0.smithi084.stdout:1263: finishing write tid 2 to smithi08477203-32 2024-09-20T22:47:00.287 INFO:tasks.rados.rados.0.smithi084.stdout:1266: read oid 15 snap -1 2024-09-20T22:47:00.287 INFO:tasks.rados.rados.0.smithi084.stdout:1266: expect (ObjNum 401 snap 101 seq_num 401) 2024-09-20T22:47:00.287 INFO:tasks.rados.rados.0.smithi084.stdout:1263: finishing write tid 3 to smithi08477203-32 2024-09-20T22:47:00.287 INFO:tasks.rados.rados.0.smithi084.stdout:1263: finishing write tid 4 to smithi08477203-32 2024-09-20T22:47:00.287 INFO:tasks.rados.rados.0.smithi084.stdout:1263: finishing write tid 5 to smithi08477203-32 2024-09-20T22:47:00.287 INFO:tasks.rados.rados.0.smithi084.stdout:1263: oid 32 updating version 0 to 392 2024-09-20T22:47:00.287 INFO:tasks.rados.rados.0.smithi084.stdout:1263: oid 32 version 392 is already newer than 391 2024-09-20T22:47:00.287 INFO:tasks.rados.rados.0.smithi084.stdout:1263: oid 32 version 392 is already newer than 389 2024-09-20T22:47:00.287 INFO:tasks.rados.rados.0.smithi084.stdout:1263: oid 32 version 392 is already newer than 390 2024-09-20T22:47:00.287 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 32 v 392 (ObjNum 457 snap 126 seq_num 457) dirty exists 2024-09-20T22:47:00.287 INFO:tasks.rados.rados.0.smithi084.stdout:1263: left oid 32 (ObjNum 457 snap 126 seq_num 457) 2024-09-20T22:47:00.287 INFO:tasks.rados.rados.0.smithi084.stdout:1264: expect (ObjNum 359 snap 92 seq_num 359) 2024-09-20T22:47:00.335 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:47:00.356 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:00.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:00.357 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:47:00.364 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:00 smithi084 ceph-mon[40024]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:47:00.364 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:00 smithi084 ceph-mon[29774]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:47:00.388 INFO:tasks.rados.rados.0.smithi084.stdout:1265: finishing write tid 1 to smithi08477203-25 2024-09-20T22:47:00.389 INFO:tasks.rados.rados.0.smithi084.stdout:1265: finishing write tid 2 to smithi08477203-25 2024-09-20T22:47:00.389 INFO:tasks.rados.rados.0.smithi084.stdout:1267: read oid 26 snap -1 2024-09-20T22:47:00.389 INFO:tasks.rados.rados.0.smithi084.stdout:1267: expect (ObjNum 390 snap 101 seq_num 390) 2024-09-20T22:47:00.389 INFO:tasks.rados.rados.0.smithi084.stdout:1263: done (4 left) 2024-09-20T22:47:00.389 INFO:tasks.rados.rados.0.smithi084.stdout:1264: done (3 left) 2024-09-20T22:47:00.389 INFO:tasks.rados.rados.0.smithi084.stdout:1268: delete oid 5 current snap is 126 2024-09-20T22:47:00.389 INFO:tasks.rados.rados.0.smithi084.stdout:1265: finishing write tid 3 to smithi08477203-25 2024-09-20T22:47:00.389 INFO:tasks.rados.rados.0.smithi084.stdout:1265: finishing write tid 4 to smithi08477203-25 2024-09-20T22:47:00.389 INFO:tasks.rados.rados.0.smithi084.stdout:1265: finishing write tid 5 to smithi08477203-25 2024-09-20T22:47:00.389 INFO:tasks.rados.rados.0.smithi084.stdout:1265: oid 25 updating version 0 to 693 2024-09-20T22:47:00.389 INFO:tasks.rados.rados.0.smithi084.stdout:1265: oid 25 updating version 693 to 696 2024-09-20T22:47:00.389 INFO:tasks.rados.rados.0.smithi084.stdout:1265: oid 25 version 696 is already newer than 695 2024-09-20T22:47:00.389 INFO:tasks.rados.rados.0.smithi084.stdout:1265: oid 25 version 696 is already newer than 694 2024-09-20T22:47:00.390 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 25 v 696 (ObjNum 458 snap 126 seq_num 458) dirty exists 2024-09-20T22:47:00.390 INFO:tasks.rados.rados.0.smithi084.stdout:1265: left oid 25 (ObjNum 458 snap 126 seq_num 458) 2024-09-20T22:47:00.390 INFO:tasks.rados.rados.0.smithi084.stdout:1266: expect (ObjNum 401 snap 101 seq_num 401) 2024-09-20T22:47:00.436 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph tell osd.7 reset_purged_snaps_last 2024-09-20T22:47:00.457 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:47:00.497 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:00.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:00.498 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:47:00.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:00 smithi154 ceph-mon[33264]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:47:00.599 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:47:00.602 INFO:tasks.rados.rados.0.smithi084.stdout:1265: done (3 left) 2024-09-20T22:47:00.602 INFO:tasks.rados.rados.0.smithi084.stdout:1266: done (2 left) 2024-09-20T22:47:00.603 INFO:tasks.rados.rados.0.smithi084.stdout:1268: done (1 left) 2024-09-20T22:47:00.603 INFO:tasks.rados.rados.0.smithi084.stdout:1269: copy_from oid 4 from oid 8 current snap is 126 2024-09-20T22:47:00.603 INFO:tasks.rados.rados.0.smithi084.stdout:1267: expect (ObjNum 390 snap 101 seq_num 390) 2024-09-20T22:47:00.621 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:00.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:00.622 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:47:00.687 INFO:tasks.rados.rados.0.smithi084.stdout:1269: finishing copy_from racing read to smithi08477203-4 2024-09-20T22:47:00.687 INFO:tasks.rados.rados.0.smithi084.stdout:1269: finishing copy_from to smithi08477203-4 2024-09-20T22:47:00.687 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 4 v 187 (ObjNum 438 snap 115 seq_num 438) dirty exists 2024-09-20T22:47:00.687 INFO:tasks.rados.rados.0.smithi084.stdout:1267: done (1 left) 2024-09-20T22:47:00.687 INFO:tasks.rados.rados.0.smithi084.stdout:1269: done (0 left) 2024-09-20T22:47:00.688 INFO:tasks.rados.rados.0.smithi084.stdout:1270: write (excl) oid 38 current snap is 126 2024-09-20T22:47:00.688 INFO:tasks.rados.rados.0.smithi084.stdout:1270: seq_num 460 ranges {531220=795720,1935011=516810,3066106=420420} 2024-09-20T22:47:00.694 INFO:tasks.rados.rados.0.smithi084.stdout:1270: writing smithi08477203-38 from 531220 to 1326940 tid 1 2024-09-20T22:47:00.701 INFO:tasks.rados.rados.0.smithi084.stdout:1270: writing smithi08477203-38 from 1935011 to 2451821 tid 2 2024-09-20T22:47:00.707 INFO:tasks.rados.rados.0.smithi084.stdout:1270: writing smithi08477203-38 from 3066106 to 3486526 tid 3 2024-09-20T22:47:00.710 INFO:tasks.rados.rados.0.smithi084.stdout:1271: read oid 46 snap 114 2024-09-20T22:47:00.710 INFO:tasks.rados.rados.0.smithi084.stdout:1271: expect deleted 2024-09-20T22:47:00.710 INFO:tasks.rados.rados.0.smithi084.stdout:1272: copy_from oid 28 from oid 47 current snap is 126 2024-09-20T22:47:00.710 INFO:tasks.rados.rados.0.smithi084.stdout:1270: finishing write tid 1 to smithi08477203-38 2024-09-20T22:47:00.710 INFO:tasks.rados.rados.0.smithi084.stdout:1273: copy_from oid 11 from oid 13 current snap is 126 2024-09-20T22:47:00.710 INFO:tasks.rados.rados.0.smithi084.stdout:1270: finishing write tid 2 to smithi08477203-38 2024-09-20T22:47:00.710 INFO:tasks.rados.rados.0.smithi084.stdout:1274: snap_remove snap 123 2024-09-20T22:47:00.722 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:47:00.744 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:00.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:00.745 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:47:00.846 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:47:00.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:00.869 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:00.869 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:47:00.968 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-20T22:47:00.968 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2024-09-20T22:47:00.968 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2024-09-20T22:47:00.968 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [ 2024-09-20T22:47:00.968 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:00.968 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.14478.0:43 4.2 4:4347d321:::bucket.sync-source-hints.:head [call,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e69)", 2024-09-20T22:47:00.968 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:41:40.156893+0000", 2024-09-20T22:47:00.969 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 320.801278459, 2024-09-20T22:47:00.969 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011236520999999999, 2024-09-20T22:47:00.969 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:47:00.969 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:47:00.969 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "reached pg", 2024-09-20T22:47:00.969 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:47:00.969 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.14478", 2024-09-20T22:47:00.969 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/2994273271", 2024-09-20T22:47:00.969 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 43 2024-09-20T22:47:00.969 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:00.969 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:47:00.969 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:00.969 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:47:00.969 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:41:40.156893+0000", 2024-09-20T22:47:00.970 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:47:00.970 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:00.970 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:00.970 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:47:00.970 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:41:40.156893+0000", 2024-09-20T22:47:00.970 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:47:00.970 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:00.970 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:00.970 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:47:00.970 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:41:40.156894+0000", 2024-09-20T22:47:00.970 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.339e-06 2024-09-20T22:47:00.970 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:00.970 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:00.971 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:47:00.971 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:41:40.156910+0000", 2024-09-20T22:47:00.971 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.5466999999999999e-05 2024-09-20T22:47:00.971 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:00.971 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:00.971 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:47:00.971 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:41:40.156921+0000", 2024-09-20T22:47:00.971 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1375e-05 2024-09-20T22:47:00.971 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:00.971 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:00.971 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:47:00.971 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:41:40.156933+0000", 2024-09-20T22:47:00.971 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1765e-05 2024-09-20T22:47:00.972 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:00.972 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:00.972 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:47:00.972 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:41:40.156992+0000", 2024-09-20T22:47:00.972 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.9122000000000001e-05 2024-09-20T22:47:00.972 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:00.972 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:00.972 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-09-20T22:47:00.972 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:41:40.157002+0000", 2024-09-20T22:47:00.972 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.9369999999999999e-06 2024-09-20T22:47:00.972 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:00.972 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:00.972 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:47:00.973 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:41:40.168028+0000", 2024-09-20T22:47:00.973 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011025945000000001 2024-09-20T22:47:00.973 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:00.973 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:00.973 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:47:00.973 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:41:40.168129+0000", 2024-09-20T22:47:00.973 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00010157099999999999 2024-09-20T22:47:00.973 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:47:00.973 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:47:00.973 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:47:00.973 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:00.973 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:00.974 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:321 7.b 7:df02c955:::smithi08477203-9:head [copy-from ver 3] snapc 5=[5] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-09-20T22:47:00.974 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:43:04.855135+0000", 2024-09-20T22:47:00.974 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 236.10303594800001, 2024-09-20T22:47:00.974 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013238773000000001, 2024-09-20T22:47:00.974 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:47:00.974 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:47:00.975 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:47:00.975 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:47:00.975 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:47:00.975 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:47:00.975 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 321 2024-09-20T22:47:00.975 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:00.975 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:47:00.975 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:00.975 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:47:00.975 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:04.855135+0000", 2024-09-20T22:47:00.975 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:47:00.975 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:00.975 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:00.975 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:47:00.976 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:04.855135+0000", 2024-09-20T22:47:00.976 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:47:00.976 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:00.976 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:00.976 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:47:00.976 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:04.855138+0000", 2024-09-20T22:47:00.976 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.5560000000000002e-06 2024-09-20T22:47:00.976 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:00.976 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:00.976 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:47:00.976 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:04.855147+0000", 2024-09-20T22:47:00.977 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.1649999999999995e-06 2024-09-20T22:47:00.977 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:00.977 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:00.977 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:47:00.977 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:04.855162+0000", 2024-09-20T22:47:00.977 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.4875000000000001e-05 2024-09-20T22:47:00.977 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:00.977 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:00.977 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:47:00.977 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:04.855171+0000", 2024-09-20T22:47:00.977 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.414e-06 2024-09-20T22:47:00.977 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:00.977 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:00.977 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:47:00.978 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:04.855256+0000", 2024-09-20T22:47:00.978 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.4640000000000003e-05 2024-09-20T22:47:00.978 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:00.978 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:00.978 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:47:00.978 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:04.855347+0000", 2024-09-20T22:47:00.978 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.0549000000000003e-05 2024-09-20T22:47:00.978 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:00.978 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:00.978 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5,6", 2024-09-20T22:47:00.978 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:04.861176+0000", 2024-09-20T22:47:00.978 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0058289450000000003 2024-09-20T22:47:00.978 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:00.979 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:00.979 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:47:00.979 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:04.864700+0000", 2024-09-20T22:47:00.979 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.003524396 2024-09-20T22:47:00.979 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:00.979 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:00.979 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:47:00.979 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:04.868245+0000", 2024-09-20T22:47:00.979 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0035450880000000001 2024-09-20T22:47:00.979 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:00.979 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:00.979 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:47:00.979 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:04.868314+0000", 2024-09-20T22:47:00.979 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.8731000000000003e-05 2024-09-20T22:47:00.980 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:00.980 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:00.980 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:47:00.980 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:04.868347+0000", 2024-09-20T22:47:00.980 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.3311e-05 2024-09-20T22:47:00.980 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:00.980 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:00.980 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:47:00.980 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:04.868374+0000", 2024-09-20T22:47:00.980 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.7103000000000001e-05 2024-09-20T22:47:00.980 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:47:00.980 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:47:00.980 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:47:00.980 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:00.980 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:00.981 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:1725 7.b 7:d3ca3619:::smithi08477203-39:head [copy-from ver 101] snapc 3e=[3e,3a,39] ondisk+write+known_if_redirected+supports_pool_eio e217)", 2024-09-20T22:47:00.981 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:44:56.216644+0000", 2024-09-20T22:47:00.981 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 124.741527261, 2024-09-20T22:47:00.981 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.012098411999999999, 2024-09-20T22:47:00.981 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:47:00.981 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:47:00.981 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:47:00.981 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:47:00.981 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:47:00.981 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:47:00.981 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1725 2024-09-20T22:47:00.981 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:00.982 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:47:00.982 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:00.982 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:47:00.982 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:56.216644+0000", 2024-09-20T22:47:00.982 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:47:00.982 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:00.982 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:00.982 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:47:00.982 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:56.216644+0000", 2024-09-20T22:47:00.982 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:47:00.982 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:00.982 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:00.983 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:47:00.983 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:56.216647+0000", 2024-09-20T22:47:00.983 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.5550000000000001e-06 2024-09-20T22:47:00.983 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:00.983 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:00.983 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:47:00.983 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:56.216655+0000", 2024-09-20T22:47:00.983 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.5359999999999992e-06 2024-09-20T22:47:00.983 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:00.983 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:00.983 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:47:00.983 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:56.216674+0000", 2024-09-20T22:47:00.983 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.8652999999999999e-05 2024-09-20T22:47:00.983 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:00.984 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:00.984 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:47:00.984 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:56.216684+0000", 2024-09-20T22:47:00.984 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.6889999999999999e-06 2024-09-20T22:47:00.984 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:00.984 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:00.984 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:47:00.984 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:56.216772+0000", 2024-09-20T22:47:00.984 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.8374000000000005e-05 2024-09-20T22:47:00.984 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:00.984 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:00.984 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:47:00.985 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:56.216835+0000", 2024-09-20T22:47:00.985 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.3528000000000006e-05 2024-09-20T22:47:00.985 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:00.985 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:00.985 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5,6", 2024-09-20T22:47:00.985 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:56.219251+0000", 2024-09-20T22:47:00.985 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002415426 2024-09-20T22:47:00.985 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:00.985 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:00.985 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:47:00.986 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:56.224154+0000", 2024-09-20T22:47:00.986 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0049032269999999996 2024-09-20T22:47:00.986 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:00.986 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:00.986 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:47:00.986 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:56.226093+0000", 2024-09-20T22:47:00.986 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0019389190000000001 2024-09-20T22:47:00.986 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:00.986 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:00.986 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:47:00.986 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:56.228669+0000", 2024-09-20T22:47:00.986 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002576138 2024-09-20T22:47:00.986 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:00.986 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:00.987 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:47:00.987 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:56.228709+0000", 2024-09-20T22:47:00.987 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.9845000000000002e-05 2024-09-20T22:47:00.987 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:00.987 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:00.987 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:47:00.987 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:56.228743+0000", 2024-09-20T22:47:00.987 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.3522000000000002e-05 2024-09-20T22:47:00.987 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:47:00.987 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:47:00.987 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:47:00.987 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:00.987 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:00.987 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:1825 7.b 7:d960cad4:::smithi08477203-18:head [write 588311~615332,stat] snapc 46=[45,44,42,41,40,3e] ondisk+write+known_if_redirected+supports_pool_eio e230)", 2024-09-20T22:47:00.988 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:45:09.174640+0000", 2024-09-20T22:47:00.988 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 111.783531192, 2024-09-20T22:47:00.988 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.012012688000000001, 2024-09-20T22:47:00.988 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:47:00.988 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:47:00.988 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:47:00.988 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:47:00.988 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:47:00.988 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:47:00.988 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1825 2024-09-20T22:47:00.988 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:00.988 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:47:00.988 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:00.988 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:47:00.989 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:09.174640+0000", 2024-09-20T22:47:00.989 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:47:00.989 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:00.989 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:00.989 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:47:00.989 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:09.174640+0000", 2024-09-20T22:47:00.989 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:47:00.989 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:00.989 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:00.989 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:47:00.989 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:09.174642+0000", 2024-09-20T22:47:00.989 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.5290000000000001e-06 2024-09-20T22:47:00.989 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:00.989 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:00.990 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:47:00.990 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:09.174940+0000", 2024-09-20T22:47:00.990 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000298215 2024-09-20T22:47:00.990 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:00.990 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:00.990 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:47:00.990 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:09.174950+0000", 2024-09-20T22:47:00.990 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0484e-05 2024-09-20T22:47:00.990 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:00.990 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:00.990 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:47:00.990 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:09.174957+0000", 2024-09-20T22:47:00.990 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.1809999999999998e-06 2024-09-20T22:47:00.991 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:00.991 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:00.991 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:47:00.991 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:09.175007+0000", 2024-09-20T22:47:00.991 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.0207000000000002e-05 2024-09-20T22:47:00.991 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:00.991 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:00.991 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op must wait for map", 2024-09-20T22:47:00.991 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:09.175011+0000", 2024-09-20T22:47:00.991 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.5379999999999996e-06 2024-09-20T22:47:00.991 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:00.991 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:00.991 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:47:00.991 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:09.177248+0000", 2024-09-20T22:47:00.992 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0022370150000000002 2024-09-20T22:47:00.992 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:00.992 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:00.992 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:47:00.992 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:09.177321+0000", 2024-09-20T22:47:00.992 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.2707999999999996e-05 2024-09-20T22:47:00.992 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:00.992 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:00.992 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5,6", 2024-09-20T22:47:00.992 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:09.177438+0000", 2024-09-20T22:47:00.992 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000116937 2024-09-20T22:47:00.992 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:00.992 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:00.992 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:47:00.993 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:09.179277+0000", 2024-09-20T22:47:00.993 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001839491 2024-09-20T22:47:00.993 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:00.993 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:00.993 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:47:00.993 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:09.186413+0000", 2024-09-20T22:47:00.993 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.007135064 2024-09-20T22:47:00.993 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:00.993 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:00.993 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:47:00.993 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:09.186554+0000", 2024-09-20T22:47:00.993 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000141657 2024-09-20T22:47:00.993 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:00.993 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:00.994 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:47:00.994 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:09.186606+0000", 2024-09-20T22:47:00.994 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.1792999999999997e-05 2024-09-20T22:47:00.994 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:00.994 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:00.994 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:47:00.994 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:09.186653+0000", 2024-09-20T22:47:00.994 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.6869000000000001e-05 2024-09-20T22:47:00.994 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:47:00.994 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:47:00.994 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:47:00.994 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:00.994 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:00.995 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:2127 7.b 7:d3ca3619:::smithi08477203-39:4b [sparse-read 0~2117715,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e249)", 2024-09-20T22:47:00.995 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:45:28.154111+0000", 2024-09-20T22:47:00.995 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 92.804059981999998, 2024-09-20T22:47:00.995 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011535606, 2024-09-20T22:47:00.995 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:47:00.995 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:47:00.995 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-20T22:47:00.995 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:47:00.996 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:47:00.996 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:47:00.996 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 2127 2024-09-20T22:47:00.996 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:00.996 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:47:00.996 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:00.996 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:47:00.996 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:28.154111+0000", 2024-09-20T22:47:00.996 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:47:00.996 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:00.996 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:00.996 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:47:00.996 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:28.154111+0000", 2024-09-20T22:47:00.996 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:47:00.997 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:00.997 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:00.997 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:47:00.997 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:28.154113+0000", 2024-09-20T22:47:00.997 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.7940000000000001e-06 2024-09-20T22:47:00.997 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:00.997 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:00.997 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:47:00.997 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:28.154121+0000", 2024-09-20T22:47:00.997 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.2190000000000004e-06 2024-09-20T22:47:00.997 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:00.997 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:00.997 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:47:00.997 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:28.154136+0000", 2024-09-20T22:47:00.997 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.4623e-05 2024-09-20T22:47:00.998 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:00.998 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:00.998 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:47:00.998 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:28.154145+0000", 2024-09-20T22:47:00.998 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.7879999999999995e-06 2024-09-20T22:47:00.998 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:00.998 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:00.998 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:47:00.998 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:28.154197+0000", 2024-09-20T22:47:00.998 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.1857999999999997e-05 2024-09-20T22:47:00.998 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:00.998 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:00.998 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op must wait for map", 2024-09-20T22:47:00.999 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:28.154203+0000", 2024-09-20T22:47:00.999 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.6409999999999996e-06 2024-09-20T22:47:00.999 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:00.999 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:00.999 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:47:00.999 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:28.159330+0000", 2024-09-20T22:47:00.999 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0051264700000000002 2024-09-20T22:47:00.999 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:00.999 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:00.999 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:47:00.999 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:28.159443+0000", 2024-09-20T22:47:00.999 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000112794 2024-09-20T22:47:00.999 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:00.999 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.000 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:47:01.000 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:28.165647+0000", 2024-09-20T22:47:01.000 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0062044190000000001 2024-09-20T22:47:01.000 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:47:01.000 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:47:01.000 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:47:01.000 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.000 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.000 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:2128 7.b 7:d3ca3619:::smithi08477203-39:4b [read 0~2117715,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e249)", 2024-09-20T22:47:01.000 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:45:28.154162+0000", 2024-09-20T22:47:01.000 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 92.804008988999996, 2024-09-20T22:47:01.000 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011909497, 2024-09-20T22:47:01.001 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:47:01.001 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:47:01.001 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-20T22:47:01.001 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:47:01.001 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:47:01.001 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:47:01.001 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 2128 2024-09-20T22:47:01.001 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.001 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:47:01.001 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.001 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:47:01.001 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:28.154162+0000", 2024-09-20T22:47:01.001 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:47:01.001 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.002 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.002 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:47:01.002 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:28.154162+0000", 2024-09-20T22:47:01.002 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:47:01.002 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.002 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.002 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:47:01.002 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:28.154163+0000", 2024-09-20T22:47:01.002 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.2300000000000001e-07 2024-09-20T22:47:01.002 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.002 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.002 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:47:01.002 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:28.154167+0000", 2024-09-20T22:47:01.002 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.6789999999999998e-06 2024-09-20T22:47:01.003 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.003 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.003 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:47:01.003 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:28.154174+0000", 2024-09-20T22:47:01.003 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.0099999999999998e-06 2024-09-20T22:47:01.003 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.003 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.003 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:47:01.003 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:28.154179+0000", 2024-09-20T22:47:01.003 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.8239999999999996e-06 2024-09-20T22:47:01.003 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.003 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.003 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:47:01.003 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:28.154246+0000", 2024-09-20T22:47:01.004 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.6817999999999996e-05 2024-09-20T22:47:01.004 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.004 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.004 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting_for_map not empty", 2024-09-20T22:47:01.004 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:28.154251+0000", 2024-09-20T22:47:01.004 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.4299999999999999e-06 2024-09-20T22:47:01.004 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.004 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.004 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:47:01.004 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:28.165689+0000", 2024-09-20T22:47:01.004 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011437932 2024-09-20T22:47:01.004 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.004 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.004 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:47:01.005 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:28.165788+0000", 2024-09-20T22:47:01.005 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.9778999999999995e-05 2024-09-20T22:47:01.005 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.005 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.005 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:47:01.005 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:28.166072+0000", 2024-09-20T22:47:01.005 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00028350199999999998 2024-09-20T22:47:01.005 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:47:01.005 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:47:01.005 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:47:01.005 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.005 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.005 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:2129 7.b 7:d3ca3619:::smithi08477203-39:4b [read 0~2117715,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e249)", 2024-09-20T22:47:01.006 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:45:28.154188+0000", 2024-09-20T22:47:01.006 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 92.803983489999993, 2024-09-20T22:47:01.006 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.012397672, 2024-09-20T22:47:01.006 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:47:01.006 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:47:01.006 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-20T22:47:01.006 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:47:01.006 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:47:01.006 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:47:01.006 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 2129 2024-09-20T22:47:01.006 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.006 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:47:01.007 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.007 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:47:01.007 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:28.154188+0000", 2024-09-20T22:47:01.007 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:47:01.007 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.007 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.007 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:47:01.007 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:28.154188+0000", 2024-09-20T22:47:01.007 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:47:01.007 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.007 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.007 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:47:01.007 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:28.154188+0000", 2024-09-20T22:47:01.007 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.46e-07 2024-09-20T22:47:01.007 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.008 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.008 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:47:01.008 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:28.154191+0000", 2024-09-20T22:47:01.008 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.7140000000000002e-06 2024-09-20T22:47:01.008 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.008 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.008 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:47:01.008 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:28.154197+0000", 2024-09-20T22:47:01.008 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.4110000000000001e-06 2024-09-20T22:47:01.008 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.008 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.008 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:47:01.009 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:28.154203+0000", 2024-09-20T22:47:01.009 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.711e-06 2024-09-20T22:47:01.009 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.009 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.009 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:47:01.009 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:28.154269+0000", 2024-09-20T22:47:01.009 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.5382999999999994e-05 2024-09-20T22:47:01.009 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.009 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.009 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting_for_map not empty", 2024-09-20T22:47:01.009 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:28.154273+0000", 2024-09-20T22:47:01.009 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.6770000000000003e-06 2024-09-20T22:47:01.009 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.009 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.009 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:47:01.010 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:28.166111+0000", 2024-09-20T22:47:01.010 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011837537 2024-09-20T22:47:01.010 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.010 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.010 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:47:01.010 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:28.166221+0000", 2024-09-20T22:47:01.010 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00011047 2024-09-20T22:47:01.010 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.010 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.010 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:47:01.010 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:28.166586+0000", 2024-09-20T22:47:01.010 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000364323 2024-09-20T22:47:01.010 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:47:01.011 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:47:01.011 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:47:01.011 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.011 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.011 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:2351 7.13 7:cae5792a:::smithi08477203-32:head [copy-from ver 376] snapc 56=[56,55,53,52,51,50,47,3e] ondisk+write+known_if_redirected+supports_pool_eio e260)", 2024-09-20T22:47:01.011 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:45:39.362090+0000", 2024-09-20T22:47:01.011 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 81.596081564000002, 2024-09-20T22:47:01.011 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.88649603300000002, 2024-09-20T22:47:01.011 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:47:01.011 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:47:01.011 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:47:01.011 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:47:01.011 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:47:01.011 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:47:01.011 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 2351 2024-09-20T22:47:01.012 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.012 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:47:01.012 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.012 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:47:01.012 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.362090+0000", 2024-09-20T22:47:01.012 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:47:01.012 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.012 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.012 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:47:01.013 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:47:01.013 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.362090+0000", 2024-09-20T22:47:01.013 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:47:01.013 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.013 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.013 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:47:01.014 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.362091+0000", 2024-09-20T22:47:01.014 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.5570000000000001e-06 2024-09-20T22:47:01.014 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.014 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.014 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:47:01.014 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.362099+0000", 2024-09-20T22:47:01.014 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.7030000000000006e-06 2024-09-20T22:47:01.014 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.014 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.014 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:47:01.014 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.362113+0000", 2024-09-20T22:47:01.014 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3492999999999999e-05 2024-09-20T22:47:01.014 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.015 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.015 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:47:01.015 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.362119+0000", 2024-09-20T22:47:01.015 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.7700000000000004e-06 2024-09-20T22:47:01.015 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.015 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.015 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:47:01.015 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.362190+0000", 2024-09-20T22:47:01.015 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.0576999999999999e-05 2024-09-20T22:47:01.015 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.015 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.015 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-09-20T22:47:01.015 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.362203+0000", 2024-09-20T22:47:01.015 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3484e-05 2024-09-20T22:47:01.016 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.016 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.016 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:47:01.016 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:40.226702+0000", 2024-09-20T22:47:01.016 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.86449815799999996 2024-09-20T22:47:01.016 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.016 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.016 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for missing object", 2024-09-20T22:47:01.016 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:40.227275+0000", 2024-09-20T22:47:01.016 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00057360899999999999 2024-09-20T22:47:01.017 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.017 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.017 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:47:01.017 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:40.232018+0000", 2024-09-20T22:47:01.017 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0047427839999999999 2024-09-20T22:47:01.017 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.017 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.017 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:47:01.017 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:40.232136+0000", 2024-09-20T22:47:01.017 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000118068 2024-09-20T22:47:01.017 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.017 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.017 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 2,5", 2024-09-20T22:47:01.017 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:40.240483+0000", 2024-09-20T22:47:01.018 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0083470390000000005 2024-09-20T22:47:01.018 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.018 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.018 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:47:01.018 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:40.245573+0000", 2024-09-20T22:47:01.018 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00508998 2024-09-20T22:47:01.018 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.018 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.018 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:47:01.018 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:40.248086+0000", 2024-09-20T22:47:01.018 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025128870000000001 2024-09-20T22:47:01.018 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.018 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.019 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:47:01.019 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:40.248398+0000", 2024-09-20T22:47:01.019 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00031257600000000001 2024-09-20T22:47:01.019 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.019 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.019 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:47:01.019 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:40.248476+0000", 2024-09-20T22:47:01.019 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.7491999999999996e-05 2024-09-20T22:47:01.019 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.019 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.019 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:47:01.019 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:40.248586+0000", 2024-09-20T22:47:01.019 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000109856 2024-09-20T22:47:01.020 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:47:01.020 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:47:01.020 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:47:01.020 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.020 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.020 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:2352 7.13 7:cae5792a:::smithi08477203-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e260)", 2024-09-20T22:47:01.020 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:45:39.362137+0000", 2024-09-20T22:47:01.020 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 81.596034696999993, 2024-09-20T22:47:01.020 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.883015891, 2024-09-20T22:47:01.020 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:47:01.020 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:47:01.020 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-20T22:47:01.020 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:47:01.021 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:47:01.021 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:47:01.021 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 2352 2024-09-20T22:47:01.021 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.021 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:47:01.021 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.021 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:47:01.021 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.362137+0000", 2024-09-20T22:47:01.021 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:47:01.021 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.021 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.021 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:47:01.021 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.362137+0000", 2024-09-20T22:47:01.021 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:47:01.021 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.022 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.022 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:47:01.022 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.362137+0000", 2024-09-20T22:47:01.022 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.6300000000000001e-07 2024-09-20T22:47:01.022 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.022 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.022 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:47:01.022 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.362139+0000", 2024-09-20T22:47:01.022 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.7460000000000001e-06 2024-09-20T22:47:01.022 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.022 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.022 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:47:01.023 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.362144+0000", 2024-09-20T22:47:01.023 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.1080000000000001e-06 2024-09-20T22:47:01.023 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.023 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.023 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:47:01.023 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.362147+0000", 2024-09-20T22:47:01.023 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.2109999999999998e-06 2024-09-20T22:47:01.023 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.023 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.023 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:47:01.023 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.362233+0000", 2024-09-20T22:47:01.023 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.5736999999999996e-05 2024-09-20T22:47:01.023 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.023 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.024 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-09-20T22:47:01.024 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.362242+0000", 2024-09-20T22:47:01.024 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.8179999999999993e-06 2024-09-20T22:47:01.024 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.024 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.024 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:47:01.024 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:40.227702+0000", 2024-09-20T22:47:01.024 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.86546061299999999 2024-09-20T22:47:01.024 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.024 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.024 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for missing object", 2024-09-20T22:47:01.024 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:40.227751+0000", 2024-09-20T22:47:01.024 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.9128000000000001e-05 2024-09-20T22:47:01.024 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.025 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.025 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:47:01.025 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:40.232300+0000", 2024-09-20T22:47:01.025 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0045490260000000003 2024-09-20T22:47:01.025 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.025 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.025 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for blocked object", 2024-09-20T22:47:01.025 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:40.232354+0000", 2024-09-20T22:47:01.025 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.3183000000000002e-05 2024-09-20T22:47:01.025 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.025 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.025 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:47:01.025 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:40.244838+0000", 2024-09-20T22:47:01.025 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.012484004 2024-09-20T22:47:01.026 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.026 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.026 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:47:01.026 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:40.244974+0000", 2024-09-20T22:47:01.026 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013651399999999999 2024-09-20T22:47:01.026 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.026 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.026 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:47:01.026 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:40.245153+0000", 2024-09-20T22:47:01.026 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00017844000000000001 2024-09-20T22:47:01.026 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:47:01.027 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:47:01.027 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:47:01.027 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.027 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.027 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(osd.5.0:136 7.2 e269/169 7:45b54b6b:::smithi08477203-2:54 v 269'114, mlcod=265'111)", 2024-09-20T22:47:01.027 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:45:48.170083+0000", 2024-09-20T22:47:01.027 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 72.788088783999996, 2024-09-20T22:47:01.027 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.017111619000000002, 2024-09-20T22:47:01.027 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:47:01.027 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:47:01.027 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:47:01.027 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:47:01.027 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.027 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:47:01.028 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:47:01.029 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.170083+0000", 2024-09-20T22:47:01.029 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:47:01.029 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.029 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.029 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:47:01.029 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.170083+0000", 2024-09-20T22:47:01.029 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:47:01.029 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.029 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.029 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:47:01.029 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.170083+0000", 2024-09-20T22:47:01.030 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.2500000000000001e-07 2024-09-20T22:47:01.030 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.030 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.030 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:47:01.030 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.170096+0000", 2024-09-20T22:47:01.030 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3101999999999999e-05 2024-09-20T22:47:01.030 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.030 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.030 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:47:01.030 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.170101+0000", 2024-09-20T22:47:01.030 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.1379999999999998e-06 2024-09-20T22:47:01.030 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.030 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.030 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:47:01.031 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.170111+0000", 2024-09-20T22:47:01.031 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0305e-05 2024-09-20T22:47:01.031 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.031 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.031 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:47:01.031 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.173555+0000", 2024-09-20T22:47:01.031 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0034440389999999999 2024-09-20T22:47:01.031 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.031 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.031 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:47:01.031 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.173580+0000", 2024-09-20T22:47:01.031 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.4142e-05 2024-09-20T22:47:01.031 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.031 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.032 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:47:01.032 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.187133+0000", 2024-09-20T22:47:01.032 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013553170999999999 2024-09-20T22:47:01.032 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.032 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.032 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:47:01.032 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.187194+0000", 2024-09-20T22:47:01.032 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.1396999999999995e-05 2024-09-20T22:47:01.032 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:47:01.032 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:47:01.032 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:47:01.032 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.032 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.032 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24652.0:2525 7.1 e278/157 7:8422f82d:::smithi08477203-22:head v 278'142, mlcod=272'139)", 2024-09-20T22:47:01.033 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:45:57.181674+0000", 2024-09-20T22:47:01.033 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 63.776497546000002, 2024-09-20T22:47:01.033 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011755127000000001, 2024-09-20T22:47:01.033 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:47:01.033 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:47:01.033 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:47:01.033 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:47:01.033 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.033 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:47:01.033 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:57.181674+0000", 2024-09-20T22:47:01.033 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:47:01.033 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.034 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.034 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:47:01.034 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:57.181674+0000", 2024-09-20T22:47:01.034 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:47:01.034 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.034 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.034 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:47:01.034 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:57.181674+0000", 2024-09-20T22:47:01.034 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.8099999999999999e-07 2024-09-20T22:47:01.034 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.034 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.034 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:47:01.034 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:57.184381+0000", 2024-09-20T22:47:01.035 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0027067860000000001 2024-09-20T22:47:01.035 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.035 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.035 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:47:01.035 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:57.184411+0000", 2024-09-20T22:47:01.035 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.9921999999999999e-05 2024-09-20T22:47:01.035 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.035 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.035 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:47:01.035 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:57.184426+0000", 2024-09-20T22:47:01.035 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.5384e-05 2024-09-20T22:47:01.035 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.035 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.035 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:47:01.036 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:57.184539+0000", 2024-09-20T22:47:01.036 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00011305 2024-09-20T22:47:01.036 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.036 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.036 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:47:01.036 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:57.184574+0000", 2024-09-20T22:47:01.036 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.4446000000000002e-05 2024-09-20T22:47:01.036 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.036 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.036 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:47:01.036 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:57.193390+0000", 2024-09-20T22:47:01.036 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0088158799999999999 2024-09-20T22:47:01.036 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.037 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.037 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:47:01.037 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:57.193429+0000", 2024-09-20T22:47:01.037 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.9378000000000001e-05 2024-09-20T22:47:01.037 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:47:01.037 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:47:01.037 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:47:01.037 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.037 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.037 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24652.0:2853 7.10 e303/139 7:0f524219:::smithi08477203-50:head v 303'85, mlcod=301'82)", 2024-09-20T22:47:01.037 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:46:23.293845+0000", 2024-09-20T22:47:01.037 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 37.664326844999998, 2024-09-20T22:47:01.038 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014852432, 2024-09-20T22:47:01.038 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:47:01.038 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:47:01.038 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:47:01.038 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:47:01.038 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.038 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:47:01.038 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.293845+0000", 2024-09-20T22:47:01.038 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:47:01.038 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.038 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.038 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:47:01.038 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.293845+0000", 2024-09-20T22:47:01.038 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:47:01.039 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.039 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.039 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:47:01.039 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.293846+0000", 2024-09-20T22:47:01.039 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.133e-06 2024-09-20T22:47:01.039 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.039 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.039 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:47:01.039 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.297227+0000", 2024-09-20T22:47:01.039 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0033811480000000001 2024-09-20T22:47:01.039 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.039 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.039 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:47:01.039 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.297254+0000", 2024-09-20T22:47:01.039 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.7223999999999999e-05 2024-09-20T22:47:01.040 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.040 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.040 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:47:01.040 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.297266+0000", 2024-09-20T22:47:01.040 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1744e-05 2024-09-20T22:47:01.040 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.040 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.040 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:47:01.040 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.297359+0000", 2024-09-20T22:47:01.040 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.3098999999999998e-05 2024-09-20T22:47:01.040 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.041 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.041 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:47:01.041 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.297384+0000", 2024-09-20T22:47:01.041 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.5358999999999999e-05 2024-09-20T22:47:01.041 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.041 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.041 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:47:01.041 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.308654+0000", 2024-09-20T22:47:01.041 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011269823999999999 2024-09-20T22:47:01.042 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.042 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.042 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:47:01.042 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.308697+0000", 2024-09-20T22:47:01.042 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.2901000000000001e-05 2024-09-20T22:47:01.042 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:47:01.042 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:47:01.042 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:47:01.043 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.043 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.043 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:3124 7.b 7:df7b2b73:::smithi08477203-10:head [write 695922~709528] snapc 77=[77,76,75,74,73,70,6b] ondisk+write+known_if_redirected+supports_pool_eio e328)", 2024-09-20T22:47:01.043 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:46:48.190758+0000", 2024-09-20T22:47:01.043 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 12.76741359, 2024-09-20T22:47:01.043 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.030625260000000001, 2024-09-20T22:47:01.043 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:47:01.043 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:47:01.043 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:47:01.044 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:47:01.044 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:47:01.044 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:47:01.044 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 3124 2024-09-20T22:47:01.044 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.044 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:47:01.044 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.044 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:47:01.044 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.190758+0000", 2024-09-20T22:47:01.045 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:47:01.045 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.045 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.045 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:47:01.045 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.190758+0000", 2024-09-20T22:47:01.045 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:47:01.045 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.045 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.045 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:47:01.046 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.190760+0000", 2024-09-20T22:47:01.046 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.0109999999999999e-06 2024-09-20T22:47:01.046 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.046 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.046 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:47:01.046 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.191218+0000", 2024-09-20T22:47:01.046 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00045814 2024-09-20T22:47:01.046 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.046 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.047 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:47:01.047 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.191242+0000", 2024-09-20T22:47:01.047 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.4264e-05 2024-09-20T22:47:01.047 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.047 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.047 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:47:01.047 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.191256+0000", 2024-09-20T22:47:01.047 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.345e-05 2024-09-20T22:47:01.048 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.048 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.048 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:47:01.048 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.191362+0000", 2024-09-20T22:47:01.048 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00010649699999999999 2024-09-20T22:47:01.048 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.048 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.048 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op must wait for map", 2024-09-20T22:47:01.048 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.191370+0000", 2024-09-20T22:47:01.049 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.9049999999999997e-06 2024-09-20T22:47:01.049 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.049 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.049 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:47:01.049 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.202536+0000", 2024-09-20T22:47:01.049 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011165552 2024-09-20T22:47:01.049 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.049 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.050 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:47:01.050 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.202637+0000", 2024-09-20T22:47:01.050 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000101648 2024-09-20T22:47:01.050 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.050 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.050 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5,6", 2024-09-20T22:47:01.050 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.202824+0000", 2024-09-20T22:47:01.050 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00018637699999999999 2024-09-20T22:47:01.050 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.051 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.051 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:47:01.051 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.211036+0000", 2024-09-20T22:47:01.051 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0082124829999999996 2024-09-20T22:47:01.051 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.051 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.051 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:47:01.051 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.217521+0000", 2024-09-20T22:47:01.051 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0064849080000000002 2024-09-20T22:47:01.052 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.052 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.052 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:47:01.052 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.221251+0000", 2024-09-20T22:47:01.052 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0037300200000000001 2024-09-20T22:47:01.052 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.052 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.052 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:47:01.053 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.221325+0000", 2024-09-20T22:47:01.053 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.3737000000000002e-05 2024-09-20T22:47:01.053 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.053 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.053 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:47:01.053 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.221383+0000", 2024-09-20T22:47:01.053 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.8267999999999998e-05 2024-09-20T22:47:01.053 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:47:01.053 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:47:01.054 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:47:01.054 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.054 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.054 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:3125 7.b 7:df7b2b73:::smithi08477203-10:head [write 2032616~607480] snapc 77=[77,76,75,74,73,70,6b] ondisk+write+known_if_redirected+supports_pool_eio e328)", 2024-09-20T22:47:01.054 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:46:48.204698+0000", 2024-09-20T22:47:01.054 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 12.75347382, 2024-09-20T22:47:01.054 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.016777378999999999, 2024-09-20T22:47:01.054 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:47:01.055 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:47:01.055 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:47:01.055 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:47:01.055 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:47:01.055 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:47:01.055 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 3125 2024-09-20T22:47:01.055 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.055 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:47:01.055 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.055 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:47:01.055 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.204698+0000", 2024-09-20T22:47:01.055 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:47:01.056 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.056 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.056 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:47:01.056 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.204698+0000", 2024-09-20T22:47:01.056 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:47:01.056 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.056 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.056 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:47:01.056 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.204700+0000", 2024-09-20T22:47:01.056 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.1890000000000001e-06 2024-09-20T22:47:01.056 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.056 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.056 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:47:01.056 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.204872+0000", 2024-09-20T22:47:01.057 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00017232799999999999 2024-09-20T22:47:01.057 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.057 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.057 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:47:01.057 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.204885+0000", 2024-09-20T22:47:01.057 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3135e-05 2024-09-20T22:47:01.057 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.057 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.057 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:47:01.057 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.204896+0000", 2024-09-20T22:47:01.057 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1048e-05 2024-09-20T22:47:01.057 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.057 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.057 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:47:01.058 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.205292+0000", 2024-09-20T22:47:01.058 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00039570499999999999 2024-09-20T22:47:01.058 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.058 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.058 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:47:01.058 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.205404+0000", 2024-09-20T22:47:01.058 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000112224 2024-09-20T22:47:01.058 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.058 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.058 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5,6", 2024-09-20T22:47:01.058 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.206058+0000", 2024-09-20T22:47:01.058 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00065375100000000005 2024-09-20T22:47:01.058 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.059 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.059 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:47:01.059 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.211942+0000", 2024-09-20T22:47:01.059 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0058842690000000001 2024-09-20T22:47:01.059 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.059 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.059 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:47:01.059 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.217575+0000", 2024-09-20T22:47:01.059 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.005633201 2024-09-20T22:47:01.059 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.059 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.059 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:47:01.059 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.221393+0000", 2024-09-20T22:47:01.059 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00381797 2024-09-20T22:47:01.059 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.060 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.060 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:47:01.060 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.221438+0000", 2024-09-20T22:47:01.060 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.4347000000000001e-05 2024-09-20T22:47:01.060 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.060 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.060 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:47:01.060 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.221475+0000", 2024-09-20T22:47:01.060 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.7212000000000002e-05 2024-09-20T22:47:01.060 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:47:01.060 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:47:01.060 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:47:01.060 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.060 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.061 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:3126 7.b 7:df7b2b73:::smithi08477203-10:head [write 3198537~297385] snapc 77=[77,76,75,74,73,70,6b] ondisk+write+known_if_redirected+supports_pool_eio e328)", 2024-09-20T22:47:01.061 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:46:48.209654+0000", 2024-09-20T22:47:01.061 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 12.748517144000001, 2024-09-20T22:47:01.061 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.012100431, 2024-09-20T22:47:01.061 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:47:01.061 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:47:01.061 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:47:01.061 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:47:01.061 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:47:01.061 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:47:01.061 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 3126 2024-09-20T22:47:01.061 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.061 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:47:01.061 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.061 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:47:01.062 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.209654+0000", 2024-09-20T22:47:01.062 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:47:01.062 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.062 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.062 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:47:01.062 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.209654+0000", 2024-09-20T22:47:01.062 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:47:01.062 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.062 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.062 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:47:01.062 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.209657+0000", 2024-09-20T22:47:01.062 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.3499999999999999e-06 2024-09-20T22:47:01.062 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.063 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.063 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:47:01.063 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.209848+0000", 2024-09-20T22:47:01.063 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00019115399999999999 2024-09-20T22:47:01.063 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.063 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.063 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:47:01.063 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.209862+0000", 2024-09-20T22:47:01.063 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.4409999999999999e-05 2024-09-20T22:47:01.063 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.063 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.063 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:47:01.063 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.209874+0000", 2024-09-20T22:47:01.063 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.206e-05 2024-09-20T22:47:01.064 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.064 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.064 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:47:01.064 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.210012+0000", 2024-09-20T22:47:01.064 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013812699999999999 2024-09-20T22:47:01.064 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.064 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.064 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:47:01.064 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.210126+0000", 2024-09-20T22:47:01.064 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00011349999999999999 2024-09-20T22:47:01.064 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.064 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.064 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5,6", 2024-09-20T22:47:01.064 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.210291+0000", 2024-09-20T22:47:01.065 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000164938 2024-09-20T22:47:01.065 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.065 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.065 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:47:01.065 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.217632+0000", 2024-09-20T22:47:01.065 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0073417150000000004 2024-09-20T22:47:01.065 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.065 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.065 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:47:01.065 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.217690+0000", 2024-09-20T22:47:01.065 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.8028000000000001e-05 2024-09-20T22:47:01.065 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.065 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.065 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:47:01.066 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.221671+0000", 2024-09-20T22:47:01.066 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0039809320000000004 2024-09-20T22:47:01.066 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.066 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.066 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:47:01.066 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.221719+0000", 2024-09-20T22:47:01.066 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.7537999999999998e-05 2024-09-20T22:47:01.066 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.066 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.066 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:47:01.066 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.221755+0000", 2024-09-20T22:47:01.066 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.5679000000000002e-05 2024-09-20T22:47:01.067 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:47:01.067 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:47:01.067 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:47:01.067 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.067 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.067 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:3127 7.b 7:df7b2b73:::smithi08477203-10:head [setxattr _header (58) in=7b,truncate 3495922] snapc 77=[77,76,75,74,73,70,6b] ondisk+write+known_if_redirected+supports_pool_eio e328)", 2024-09-20T22:47:01.067 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:46:48.209902+0000", 2024-09-20T22:47:01.067 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 12.748268925, 2024-09-20T22:47:01.067 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011952539, 2024-09-20T22:47:01.067 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:47:01.067 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:47:01.068 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:47:01.068 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:47:01.068 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:47:01.068 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:47:01.068 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 3127 2024-09-20T22:47:01.068 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.068 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:47:01.068 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.069 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:47:01.069 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.209902+0000", 2024-09-20T22:47:01.069 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:47:01.069 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.069 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.069 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:47:01.069 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.209902+0000", 2024-09-20T22:47:01.069 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:47:01.069 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.069 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.069 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:47:01.069 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.209903+0000", 2024-09-20T22:47:01.070 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.8999999999999996e-07 2024-09-20T22:47:01.070 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.070 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.070 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:47:01.070 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.209908+0000", 2024-09-20T22:47:01.070 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.057e-06 2024-09-20T22:47:01.070 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.070 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.070 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:47:01.070 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.209916+0000", 2024-09-20T22:47:01.071 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.4590000000000001e-06 2024-09-20T22:47:01.071 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.071 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.071 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:47:01.071 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.209922+0000", 2024-09-20T22:47:01.071 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.7880000000000001e-06 2024-09-20T22:47:01.071 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.071 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.071 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:47:01.071 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.211097+0000", 2024-09-20T22:47:01.071 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001174568 2024-09-20T22:47:01.072 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.072 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.072 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:47:01.072 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.211193+0000", 2024-09-20T22:47:01.072 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.5991999999999998e-05 2024-09-20T22:47:01.072 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.072 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.072 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5,6", 2024-09-20T22:47:01.072 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.211349+0000", 2024-09-20T22:47:01.072 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000156362 2024-09-20T22:47:01.072 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.072 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.072 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:47:01.072 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.217753+0000", 2024-09-20T22:47:01.072 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0064041280000000003 2024-09-20T22:47:01.073 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.073 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.073 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:47:01.073 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.217799+0000", 2024-09-20T22:47:01.073 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.5781000000000001e-05 2024-09-20T22:47:01.073 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.073 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.073 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:47:01.073 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.221762+0000", 2024-09-20T22:47:01.073 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0039626059999999996 2024-09-20T22:47:01.073 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.074 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.074 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:47:01.074 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.221805+0000", 2024-09-20T22:47:01.074 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.2750000000000002e-05 2024-09-20T22:47:01.074 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.074 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.074 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:47:01.074 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.221855+0000", 2024-09-20T22:47:01.074 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.0358000000000001e-05 2024-09-20T22:47:01.074 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:47:01.074 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:47:01.074 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:47:01.074 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.074 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.075 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:3128 7.b 7:df7b2b73:::smithi08477203-10:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e328)", 2024-09-20T22:47:01.075 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:46:48.209932+0000", 2024-09-20T22:47:01.075 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 12.748239584, 2024-09-20T22:47:01.075 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.012172217000000001, 2024-09-20T22:47:01.075 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:47:01.075 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:47:01.075 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-20T22:47:01.075 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:47:01.075 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:47:01.075 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:47:01.075 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 3128 2024-09-20T22:47:01.075 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.075 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:47:01.075 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.076 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:47:01.076 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.209932+0000", 2024-09-20T22:47:01.076 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:47:01.076 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.076 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.076 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:47:01.076 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.209932+0000", 2024-09-20T22:47:01.076 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:47:01.076 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.076 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.076 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:47:01.076 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.209932+0000", 2024-09-20T22:47:01.077 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.4199999999999996e-07 2024-09-20T22:47:01.077 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.077 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.077 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:47:01.077 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.209935+0000", 2024-09-20T22:47:01.077 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.401e-06 2024-09-20T22:47:01.077 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.077 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.077 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:47:01.077 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.209955+0000", 2024-09-20T22:47:01.077 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.9885e-05 2024-09-20T22:47:01.077 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.077 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.077 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:47:01.078 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.209961+0000", 2024-09-20T22:47:01.078 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.224e-06 2024-09-20T22:47:01.078 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.078 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.078 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:47:01.078 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.211721+0000", 2024-09-20T22:47:01.078 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0017597979999999999 2024-09-20T22:47:01.078 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.078 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.078 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for rw locks", 2024-09-20T22:47:01.078 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.211869+0000", 2024-09-20T22:47:01.078 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014819400000000001 2024-09-20T22:47:01.078 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.078 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.079 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:47:01.079 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.221887+0000", 2024-09-20T22:47:01.079 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010018605 2024-09-20T22:47:01.079 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.079 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.079 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:47:01.079 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.221996+0000", 2024-09-20T22:47:01.079 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00010880899999999999 2024-09-20T22:47:01.079 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.079 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.079 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:47:01.079 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.222104+0000", 2024-09-20T22:47:01.080 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000107759 2024-09-20T22:47:01.080 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:47:01.080 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:47:01.080 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:47:01.080 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.080 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.080 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:3237 7.b 7:d960cad4:::smithi08477203-18:head [copy-from ver 252] snapc 7e=[7e,7d,7c,7b,7a,79,78,75,74,73,6b] ondisk+write+known_if_redirected+supports_pool_eio e338)", 2024-09-20T22:47:01.080 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:46:58.180925+0000", 2024-09-20T22:47:01.080 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 2.7772462610000002, 2024-09-20T22:47:01.080 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.029255882, 2024-09-20T22:47:01.080 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:47:01.080 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:47:01.080 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:47:01.081 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:47:01.081 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:47:01.081 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:47:01.081 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 3237 2024-09-20T22:47:01.081 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.081 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:47:01.081 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.081 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:47:01.081 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.180925+0000", 2024-09-20T22:47:01.081 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:47:01.081 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.081 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.081 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:47:01.081 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.180925+0000", 2024-09-20T22:47:01.082 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:47:01.082 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.082 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.082 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:47:01.082 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.180927+0000", 2024-09-20T22:47:01.082 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.945e-06 2024-09-20T22:47:01.082 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.082 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.082 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:47:01.082 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.180933+0000", 2024-09-20T22:47:01.082 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.7400000000000001e-06 2024-09-20T22:47:01.082 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.083 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.083 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:47:01.083 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.180947+0000", 2024-09-20T22:47:01.083 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3947999999999999e-05 2024-09-20T22:47:01.083 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.083 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.083 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:47:01.083 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.180954+0000", 2024-09-20T22:47:01.083 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.25e-06 2024-09-20T22:47:01.083 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.083 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.083 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:47:01.083 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.181002+0000", 2024-09-20T22:47:01.084 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.8331000000000002e-05 2024-09-20T22:47:01.084 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.084 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.084 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:47:01.084 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.181101+0000", 2024-09-20T22:47:01.084 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.8553000000000003e-05 2024-09-20T22:47:01.084 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.084 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.084 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5,6", 2024-09-20T22:47:01.084 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.196353+0000", 2024-09-20T22:47:01.084 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.015251789 2024-09-20T22:47:01.084 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.084 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.084 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:47:01.085 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.202094+0000", 2024-09-20T22:47:01.085 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0057409829999999999 2024-09-20T22:47:01.085 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.085 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.085 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:47:01.085 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.202682+0000", 2024-09-20T22:47:01.085 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000587898 2024-09-20T22:47:01.085 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.085 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.085 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:47:01.085 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.210104+0000", 2024-09-20T22:47:01.085 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0074227900000000003 2024-09-20T22:47:01.085 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.085 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.086 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:47:01.086 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.210145+0000", 2024-09-20T22:47:01.086 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.1050999999999997e-05 2024-09-20T22:47:01.086 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.086 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.086 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:47:01.086 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.210181+0000", 2024-09-20T22:47:01.086 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.5604e-05 2024-09-20T22:47:01.086 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:47:01.086 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:47:01.086 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:47:01.086 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.086 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.087 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:3238 7.b 7:d960cad4:::smithi08477203-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e338)", 2024-09-20T22:47:01.087 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:46:58.180973+0000", 2024-09-20T22:47:01.087 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 2.777198576, 2024-09-20T22:47:01.087 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.020749782000000001, 2024-09-20T22:47:01.087 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:47:01.087 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:47:01.087 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-20T22:47:01.087 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:47:01.087 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:47:01.087 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:47:01.087 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 3238 2024-09-20T22:47:01.088 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.088 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:47:01.088 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.088 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:47:01.088 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.180973+0000", 2024-09-20T22:47:01.088 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:47:01.088 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.088 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.088 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:47:01.088 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.180973+0000", 2024-09-20T22:47:01.088 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:47:01.089 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.089 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.089 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:47:01.089 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.180973+0000", 2024-09-20T22:47:01.089 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.4000000000000002e-07 2024-09-20T22:47:01.089 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.089 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.089 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:47:01.089 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.180976+0000", 2024-09-20T22:47:01.089 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.576e-06 2024-09-20T22:47:01.089 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.089 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.089 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:47:01.089 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.180982+0000", 2024-09-20T22:47:01.089 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.0700000000000003e-06 2024-09-20T22:47:01.090 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.090 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.090 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:47:01.090 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.180988+0000", 2024-09-20T22:47:01.090 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.5929999999999996e-06 2024-09-20T22:47:01.090 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.090 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.090 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:47:01.090 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.182094+0000", 2024-09-20T22:47:01.090 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001106604 2024-09-20T22:47:01.090 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.090 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.090 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for blocked object", 2024-09-20T22:47:01.090 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.182223+0000", 2024-09-20T22:47:01.091 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000129152 2024-09-20T22:47:01.091 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.091 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.091 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:47:01.091 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.201400+0000", 2024-09-20T22:47:01.091 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.019176192000000002 2024-09-20T22:47:01.091 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.091 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.091 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:47:01.091 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.201563+0000", 2024-09-20T22:47:01.091 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00016328999999999999 2024-09-20T22:47:01.092 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.092 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.092 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:47:01.092 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.201723+0000", 2024-09-20T22:47:01.092 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000159765 2024-09-20T22:47:01.092 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:47:01.092 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:47:01.092 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:47:01.092 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.092 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.092 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24652.0:3235 7.1 e338/157 7:84a866f5:::smithi08477203-27:head v 338'167, mlcod=320'164)", 2024-09-20T22:47:01.092 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:46:58.205374+0000", 2024-09-20T22:47:01.092 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 2.7527977039999998, 2024-09-20T22:47:01.092 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.019862464999999999, 2024-09-20T22:47:01.093 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:47:01.093 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:47:01.093 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:47:01.093 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:47:01.093 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.093 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:47:01.093 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.205374+0000", 2024-09-20T22:47:01.093 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:47:01.093 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.093 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.093 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:47:01.093 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.205374+0000", 2024-09-20T22:47:01.093 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:47:01.093 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.094 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.094 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:47:01.094 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.205374+0000", 2024-09-20T22:47:01.094 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.3200000000000001e-07 2024-09-20T22:47:01.094 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.094 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.094 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:47:01.094 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.208223+0000", 2024-09-20T22:47:01.094 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0028490820000000002 2024-09-20T22:47:01.094 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.094 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.094 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:47:01.094 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.208249+0000", 2024-09-20T22:47:01.094 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.584e-05 2024-09-20T22:47:01.095 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.095 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.095 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:47:01.095 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.208262+0000", 2024-09-20T22:47:01.095 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3166999999999999e-05 2024-09-20T22:47:01.095 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.095 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.095 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:47:01.095 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.208344+0000", 2024-09-20T22:47:01.095 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.1567e-05 2024-09-20T22:47:01.095 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.096 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.096 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:47:01.096 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.208370+0000", 2024-09-20T22:47:01.096 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.5973999999999999e-05 2024-09-20T22:47:01.096 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.096 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.096 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:47:01.096 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.225185+0000", 2024-09-20T22:47:01.096 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.016814976999999998 2024-09-20T22:47:01.096 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:01.096 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:01.096 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:47:01.096 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.225236+0000", 2024-09-20T22:47:01.096 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.1526000000000001e-05 2024-09-20T22:47:01.096 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:47:01.097 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:47:01.097 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:47:01.097 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:47:01.097 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:47:01.097 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-20T22:47:01.117 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:01.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:01.118 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:47:01.157 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-20T22:47:01.208 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:47:01.219 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:47:01.242 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:01.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:01.243 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:47:01.254 INFO:tasks.rados.rados.0.smithi084.stdout:1274: done (4 left) 2024-09-20T22:47:01.254 INFO:tasks.rados.rados.0.smithi084.stdout:1275: write (excl) oid 4 current snap is 126 2024-09-20T22:47:01.254 INFO:tasks.rados.rados.0.smithi084.stdout:1275: seq_num 461 ranges {548027=470969,1593493=776821,3093529=631692} 2024-09-20T22:47:01.260 INFO:tasks.rados.rados.0.smithi084.stdout:1275: writing smithi08477203-4 from 548027 to 1018996 tid 1 2024-09-20T22:47:01.267 INFO:tasks.rados.rados.0.smithi084.stdout:1275: writing smithi08477203-4 from 1593493 to 2370314 tid 2 2024-09-20T22:47:01.276 INFO:tasks.rados.rados.0.smithi084.stdout:1275: writing smithi08477203-4 from 3093529 to 3725221 tid 3 2024-09-20T22:47:01.277 INFO:tasks.rados.rados.0.smithi084.stdout:1276: write oid 50 current snap is 126 2024-09-20T22:47:01.277 INFO:tasks.rados.rados.0.smithi084.stdout:1276: seq_num 462 ranges {564834=546218,1651975=520479,2720952=559317} 2024-09-20T22:47:01.283 INFO:tasks.rados.rados.0.smithi084.stdout:1276: writing smithi08477203-50 from 564834 to 1111052 tid 1 2024-09-20T22:47:01.289 INFO:tasks.rados.rados.0.smithi084.stdout:1276: writing smithi08477203-50 from 1651975 to 2172454 tid 2 2024-09-20T22:47:01.295 INFO:tasks.rados.rados.0.smithi084.stdout:1276: writing smithi08477203-50 from 2720952 to 3280269 tid 3 2024-09-20T22:47:01.295 INFO:tasks.rados.rados.0.smithi084.stdout:1277: snap_create 2024-09-20T22:47:01.295 INFO:tasks.rados.rados.0.smithi084.stdout:1270: finishing write tid 3 to smithi08477203-38 2024-09-20T22:47:01.295 INFO:tasks.rados.rados.0.smithi084.stdout:1270: finishing write tid 4 to smithi08477203-38 2024-09-20T22:47:01.296 INFO:tasks.rados.rados.0.smithi084.stdout:1270: finishing write tid 5 to smithi08477203-38 2024-09-20T22:47:01.296 INFO:tasks.rados.rados.0.smithi084.stdout:1270: oid 38 updating version 0 to 391 2024-09-20T22:47:01.296 INFO:tasks.rados.rados.0.smithi084.stdout:1270: oid 38 updating version 391 to 393 2024-09-20T22:47:01.296 INFO:tasks.rados.rados.0.smithi084.stdout:1270: oid 38 updating version 393 to 394 2024-09-20T22:47:01.296 INFO:tasks.rados.rados.0.smithi084.stdout:1270: oid 38 version 394 is already newer than 392 2024-09-20T22:47:01.296 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 38 v 394 (ObjNum 460 snap 126 seq_num 460) dirty exists 2024-09-20T22:47:01.296 INFO:tasks.rados.rados.0.smithi084.stdout:1270: left oid 38 (ObjNum 460 snap 126 seq_num 460) 2024-09-20T22:47:01.296 INFO:tasks.rados.rados.0.smithi084.stdout:1272: finishing copy_from racing read to smithi08477203-28 2024-09-20T22:47:01.296 INFO:tasks.rados.rados.0.smithi084.stdout:1272: finishing copy_from to smithi08477203-28 2024-09-20T22:47:01.296 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 28 v 252 (ObjNum 390 snap 101 seq_num 390) dirty exists 2024-09-20T22:47:01.296 INFO:tasks.rados.rados.0.smithi084.stdout:1273: finishing copy_from racing read to smithi08477203-11 2024-09-20T22:47:01.296 INFO:tasks.rados.rados.0.smithi084.stdout:1273: finishing copy_from to smithi08477203-11 2024-09-20T22:47:01.296 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 11 v 675 (ObjNum 411 snap 105 seq_num 411) dirty exists 2024-09-20T22:47:01.297 INFO:tasks.rados.rados.0.smithi084.stdout:1275: finishing write tid 1 to smithi08477203-4 2024-09-20T22:47:01.297 INFO:tasks.rados.rados.0.smithi084.stdout:1275: finishing write tid 2 to smithi08477203-4 2024-09-20T22:47:01.297 INFO:tasks.rados.rados.0.smithi084.stdout:1275: finishing write tid 3 to smithi08477203-4 2024-09-20T22:47:01.297 INFO:tasks.rados.rados.0.smithi084.stdout:1275: finishing write tid 4 to smithi08477203-4 2024-09-20T22:47:01.297 INFO:tasks.rados.rados.0.smithi084.stdout:1275: finishing write tid 5 to smithi08477203-4 2024-09-20T22:47:01.297 INFO:tasks.rados.rados.0.smithi084.stdout:1275: oid 4 updating version 0 to 711 2024-09-20T22:47:01.297 INFO:tasks.rados.rados.0.smithi084.stdout:1275: oid 4 version 711 is already newer than 701 2024-09-20T22:47:01.297 INFO:tasks.rados.rados.0.smithi084.stdout:1275: oid 4 version 711 is already newer than 706 2024-09-20T22:47:01.297 INFO:tasks.rados.rados.0.smithi084.stdout:1275: oid 4 updating version 711 to 712 2024-09-20T22:47:01.297 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 4 v 712 (ObjNum 461 snap 126 seq_num 461) dirty exists 2024-09-20T22:47:01.297 INFO:tasks.rados.rados.0.smithi084.stdout:1275: left oid 4 (ObjNum 461 snap 126 seq_num 461) 2024-09-20T22:47:01.297 INFO:tasks.rados.rados.0.smithi084.stdout:1276: finishing write tid 1 to smithi08477203-50 2024-09-20T22:47:01.297 INFO:tasks.rados.rados.0.smithi084.stdout:1276: finishing write tid 2 to smithi08477203-50 2024-09-20T22:47:01.298 INFO:tasks.rados.rados.0.smithi084.stdout:1276: finishing write tid 3 to smithi08477203-50 2024-09-20T22:47:01.298 INFO:tasks.rados.rados.0.smithi084.stdout:1276: finishing write tid 4 to smithi08477203-50 2024-09-20T22:47:01.298 INFO:tasks.rados.rados.0.smithi084.stdout:1276: finishing write tid 5 to smithi08477203-50 2024-09-20T22:47:01.298 INFO:tasks.rados.rados.0.smithi084.stdout:1276: oid 50 updating version 0 to 243 2024-09-20T22:47:01.298 INFO:tasks.rados.rados.0.smithi084.stdout:1276: oid 50 updating version 243 to 244 2024-09-20T22:47:01.298 INFO:tasks.rados.rados.0.smithi084.stdout:1276: oid 50 updating version 244 to 246 2024-09-20T22:47:01.299 INFO:tasks.rados.rados.0.smithi084.stdout:1276: oid 50 version 246 is already newer than 245 2024-09-20T22:47:01.299 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 50 v 246 (ObjNum 462 snap 126 seq_num 462) dirty exists 2024-09-20T22:47:01.299 INFO:tasks.rados.rados.0.smithi084.stdout:1276: left oid 50 (ObjNum 462 snap 126 seq_num 462) 2024-09-20T22:47:01.343 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:47:01.365 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:01.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:01.366 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:47:01.467 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:47:01.488 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:01.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:01.489 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:47:01.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:01 smithi154 ceph-mon[33264]: pgmap v411: 168 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 162 active+clean; 189 MiB data, 650 MiB used, 536 GiB / 536 GiB avail; 14 MiB/s rd, 3.7 MiB/s wr, 16 op/s 2024-09-20T22:47:01.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:01 smithi154 ceph-mon[33264]: osdmap e340: 8 total, 8 up, 6 in 2024-09-20T22:47:01.564 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:01 smithi084 ceph-mon[40024]: pgmap v411: 168 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 162 active+clean; 189 MiB data, 650 MiB used, 536 GiB / 536 GiB avail; 14 MiB/s rd, 3.7 MiB/s wr, 16 op/s 2024-09-20T22:47:01.564 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:01 smithi084 ceph-mon[40024]: osdmap e340: 8 total, 8 up, 6 in 2024-09-20T22:47:01.564 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:01 smithi084 ceph-mon[29774]: pgmap v411: 168 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 162 active+clean; 189 MiB data, 650 MiB used, 536 GiB / 536 GiB avail; 14 MiB/s rd, 3.7 MiB/s wr, 16 op/s 2024-09-20T22:47:01.564 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:01 smithi084 ceph-mon[29774]: osdmap e340: 8 total, 8 up, 6 in 2024-09-20T22:47:01.590 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:47:01.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:01.612 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:01.612 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:47:01.712 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:47:01.734 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:01.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:01.735 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:47:01.835 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:47:01.856 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:01.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:01.857 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:47:01.958 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:47:01.979 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:01.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:01.980 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:47:02.081 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:47:02.104 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:02.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:02.105 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:47:02.121 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:47:02.121 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:47:02.139 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:47:02.139 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:47:02.155 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:47:02.155 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:47:02.170 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-20T22:47:02.171 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T22:47:02.187 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:47:02.187 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:47:02.206 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:47:02.206 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:47:02.206 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:47:02.222 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:47:02.223 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:47:02.228 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:02.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:02.230 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:47:02.240 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:47:02.240 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:47:02.257 INFO:tasks.rados.rados.0.smithi084.stdout:1270: done (6 left) 2024-09-20T22:47:02.258 INFO:tasks.rados.rados.0.smithi084.stdout:1271: done (5 left) 2024-09-20T22:47:02.258 INFO:tasks.rados.rados.0.smithi084.stdout:1272: done (4 left) 2024-09-20T22:47:02.258 INFO:tasks.rados.rados.0.smithi084.stdout:1273: done (3 left) 2024-09-20T22:47:02.258 INFO:tasks.rados.rados.0.smithi084.stdout:1275: done (2 left) 2024-09-20T22:47:02.258 INFO:tasks.rados.rados.0.smithi084.stdout:1276: done (1 left) 2024-09-20T22:47:02.258 INFO:tasks.rados.rados.0.smithi084.stdout:1277: done (0 left) 2024-09-20T22:47:02.259 INFO:tasks.rados.rados.0.smithi084.stdout:1278: write (excl) oid 28 current snap is 127 2024-09-20T22:47:02.259 INFO:tasks.rados.rados.0.smithi084.stdout:1278: seq_num 463 ranges {581641=621467,1994104=780490,3432022=349619} 2024-09-20T22:47:02.263 INFO:tasks.rados.rados.0.smithi084.stdout:1278: writing smithi08477203-28 from 581641 to 1203108 tid 1 2024-09-20T22:47:02.272 INFO:tasks.rados.rados.0.smithi084.stdout:1278: writing smithi08477203-28 from 1994104 to 2774594 tid 2 2024-09-20T22:47:02.278 INFO:tasks.rados.rados.0.smithi084.stdout:1278: writing smithi08477203-28 from 3432022 to 3781641 tid 3 2024-09-20T22:47:02.278 INFO:tasks.rados.rados.0.smithi084.stdout:1279: rollback oid 20 current snap is 127 2024-09-20T22:47:02.278 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 20 to 125 2024-09-20T22:47:02.278 INFO:tasks.rados.rados.0.smithi084.stdout:1278: finishing write tid 1 to smithi08477203-28 2024-09-20T22:47:02.278 INFO:tasks.rados.rados.0.smithi084.stdout:1278: finishing write tid 2 to smithi08477203-28 2024-09-20T22:47:02.279 INFO:tasks.rados.rados.0.smithi084.stdout:1280: write (excl) oid 1 current snap is 127 2024-09-20T22:47:02.279 INFO:tasks.rados.rados.0.smithi084.stdout:1280: seq_num 464 ranges {598448=413069,1768939=524148,2775798=581861,3781488=16960} 2024-09-20T22:47:02.284 INFO:tasks.rados.rados.0.smithi084.stdout:1280: writing smithi08477203-1 from 598448 to 1011517 tid 1 2024-09-20T22:47:02.291 INFO:tasks.rados.rados.0.smithi084.stdout:1280: writing smithi08477203-1 from 1768939 to 2293087 tid 2 2024-09-20T22:47:02.297 INFO:tasks.rados.rados.0.smithi084.stdout:1280: writing smithi08477203-1 from 2775798 to 3357659 tid 3 2024-09-20T22:47:02.300 INFO:tasks.rados.rados.0.smithi084.stdout:1280: writing smithi08477203-1 from 3781488 to 3798448 tid 4 2024-09-20T22:47:02.300 INFO:tasks.rados.rados.0.smithi084.stdout:1281: rollback oid 30 current snap is 127 2024-09-20T22:47:02.300 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 30 to 113 2024-09-20T22:47:02.300 INFO:tasks.rados.rados.0.smithi084.stdout:1278: finishing write tid 3 to smithi08477203-28 2024-09-20T22:47:02.300 INFO:tasks.rados.rados.0.smithi084.stdout:1279: finishing rollback tid 0 to smithi08477203-20 2024-09-20T22:47:02.300 INFO:tasks.rados.rados.0.smithi084.stdout:1279: finishing rollback tid 1 to smithi08477203-20 2024-09-20T22:47:02.300 INFO:tasks.rados.rados.0.smithi084.stdout:1278: finishing write tid 4 to smithi08477203-28 2024-09-20T22:47:02.300 INFO:tasks.rados.rados.0.smithi084.stdout:1279: finishing rollback tid 2 to smithi08477203-20 2024-09-20T22:47:02.300 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 20 v 105 (ObjNum 449 snap 120 seq_num 449) dirty exists 2024-09-20T22:47:02.300 INFO:tasks.rados.rados.0.smithi084.stdout:1278: finishing write tid 5 to smithi08477203-28 2024-09-20T22:47:02.300 INFO:tasks.rados.rados.0.smithi084.stdout:1278: oid 28 updating version 0 to 717 2024-09-20T22:47:02.301 INFO:tasks.rados.rados.0.smithi084.stdout:1278: oid 28 version 717 is already newer than 716 2024-09-20T22:47:02.301 INFO:tasks.rados.rados.0.smithi084.stdout:1278: oid 28 version 717 is already newer than 715 2024-09-20T22:47:02.301 INFO:tasks.rados.rados.0.smithi084.stdout:1278: oid 28 version 717 is already newer than 714 2024-09-20T22:47:02.301 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 28 v 717 (ObjNum 463 snap 127 seq_num 463) dirty exists 2024-09-20T22:47:02.301 INFO:tasks.rados.rados.0.smithi084.stdout:1278: left oid 28 (ObjNum 463 snap 127 seq_num 463) 2024-09-20T22:47:02.301 INFO:tasks.rados.rados.0.smithi084.stdout:1280: finishing write tid 1 to smithi08477203-1 2024-09-20T22:47:02.301 INFO:tasks.rados.rados.0.smithi084.stdout:1280: finishing write tid 2 to smithi08477203-1 2024-09-20T22:47:02.301 INFO:tasks.rados.rados.0.smithi084.stdout:1278: done (3 left) 2024-09-20T22:47:02.301 INFO:tasks.rados.rados.0.smithi084.stdout:1279: done (2 left) 2024-09-20T22:47:02.301 INFO:tasks.rados.rados.0.smithi084.stdout:1282: delete oid 19 current snap is 127 2024-09-20T22:47:02.302 INFO:tasks.rados.rados.0.smithi084.stdout:1280: finishing write tid 3 to smithi08477203-1 2024-09-20T22:47:02.302 INFO:tasks.rados.rados.0.smithi084.stdout:1280: finishing write tid 4 to smithi08477203-1 2024-09-20T22:47:02.302 INFO:tasks.rados.rados.0.smithi084.stdout:1280: finishing write tid 5 to smithi08477203-1 2024-09-20T22:47:02.302 INFO:tasks.rados.rados.0.smithi084.stdout:1280: finishing write tid 6 to smithi08477203-1 2024-09-20T22:47:02.302 INFO:tasks.rados.rados.0.smithi084.stdout:1280: oid 1 updating version 0 to 402 2024-09-20T22:47:02.302 INFO:tasks.rados.rados.0.smithi084.stdout:1280: oid 1 version 402 is already newer than 401 2024-09-20T22:47:02.302 INFO:tasks.rados.rados.0.smithi084.stdout:1280: oid 1 version 402 is already newer than 398 2024-09-20T22:47:02.302 INFO:tasks.rados.rados.0.smithi084.stdout:1280: oid 1 version 402 is already newer than 400 2024-09-20T22:47:02.302 INFO:tasks.rados.rados.0.smithi084.stdout:1280: oid 1 version 402 is already newer than 399 2024-09-20T22:47:02.302 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 1 v 402 (ObjNum 464 snap 127 seq_num 464) dirty exists 2024-09-20T22:47:02.302 INFO:tasks.rados.rados.0.smithi084.stdout:1280: left oid 1 (ObjNum 464 snap 127 seq_num 464) 2024-09-20T22:47:02.303 INFO:tasks.rados.rados.0.smithi084.stdout:1281: finishing rollback tid 0 to smithi08477203-30 2024-09-20T22:47:02.303 INFO:tasks.rados.rados.0.smithi084.stdout:1280: done (2 left) 2024-09-20T22:47:02.303 INFO:tasks.rados.rados.0.smithi084.stdout:1282: done (1 left) 2024-09-20T22:47:02.303 INFO:tasks.rados.rados.0.smithi084.stdout:1283: snap_create 2024-09-20T22:47:02.303 INFO:tasks.rados.rados.0.smithi084.stdout:1281: finishing rollback tid 1 to smithi08477203-30 2024-09-20T22:47:02.304 INFO:tasks.rados.rados.0.smithi084.stdout:1281: finishing rollback tid 2 to smithi08477203-30 2024-09-20T22:47:02.304 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 30 v 396 (ObjNum 416 snap 105 seq_num 416) dirty exists 2024-09-20T22:47:02.330 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:47:02.351 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:02.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:02.355 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:47:02.455 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:47:02.477 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:02.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:02.478 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:47:02.579 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:47:02.600 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:02.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:02.601 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:47:02.614 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:02 smithi084 ceph-mon[40024]: osdmap e341: 8 total, 8 up, 6 in 2024-09-20T22:47:02.614 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:02 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/2721937994' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:47:02.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:02 smithi084 ceph-mon[29774]: osdmap e341: 8 total, 8 up, 6 in 2024-09-20T22:47:02.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:02 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/2721937994' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:47:02.702 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:47:02.723 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:02.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:02.724 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:47:02.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:02 smithi154 ceph-mon[33264]: osdmap e341: 8 total, 8 up, 6 in 2024-09-20T22:47:02.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:02 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/2721937994' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:47:02.825 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:47:02.837 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-20T22:47:02.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:02.848 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:02.848 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:47:02.949 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:47:02.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:02.971 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:02.971 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:47:03.072 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:47:03.093 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:03.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:03.094 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:47:03.181 INFO:tasks.rados.rados.0.smithi084.stdout:1281: done (1 left) 2024-09-20T22:47:03.181 INFO:tasks.rados.rados.0.smithi084.stdout:1283: done (0 left) 2024-09-20T22:47:03.181 INFO:tasks.rados.rados.0.smithi084.stdout:1284: rollback oid 48 current snap is 128 2024-09-20T22:47:03.181 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 48 to 126 2024-09-20T22:47:03.181 INFO:tasks.rados.rados.0.smithi084.stdout:1285: rollback oid 2 current snap is 128 2024-09-20T22:47:03.181 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 2 to 126 2024-09-20T22:47:03.181 INFO:tasks.rados.rados.0.smithi084.stdout:1286: write (excl) oid 45 current snap is 128 2024-09-20T22:47:03.181 INFO:tasks.rados.rados.0.smithi084.stdout:1286: seq_num 466 ranges {632062=563567,1885903=527817,2830644=720758} 2024-09-20T22:47:03.188 INFO:tasks.rados.rados.0.smithi084.stdout:1286: writing smithi08477203-45 from 632062 to 1195629 tid 1 2024-09-20T22:47:03.194 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:47:03.195 INFO:tasks.rados.rados.0.smithi084.stdout:1286: writing smithi08477203-45 from 1885903 to 2413720 tid 2 2024-09-20T22:47:03.202 INFO:tasks.rados.rados.0.smithi084.stdout:1286: writing smithi08477203-45 from 2830644 to 3551402 tid 3 2024-09-20T22:47:03.202 INFO:tasks.rados.rados.0.smithi084.stdout:1287: write oid 16 current snap is 128 2024-09-20T22:47:03.202 INFO:tasks.rados.rados.0.smithi084.stdout:1284: finishing rollback tid 0 to smithi08477203-48 2024-09-20T22:47:03.202 INFO:tasks.rados.rados.0.smithi084.stdout:1287: seq_num 467 ranges {648869=638816,1828032=787828,3141714=532030} 2024-09-20T22:47:03.209 INFO:tasks.rados.rados.0.smithi084.stdout:1287: writing smithi08477203-16 from 648869 to 1287685 tid 1 2024-09-20T22:47:03.216 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:03.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:03.217 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:47:03.217 INFO:tasks.rados.rados.0.smithi084.stdout:1287: writing smithi08477203-16 from 1828032 to 2615860 tid 2 2024-09-20T22:47:03.223 INFO:tasks.rados.rados.0.smithi084.stdout:1287: writing smithi08477203-16 from 3141714 to 3673744 tid 3 2024-09-20T22:47:03.223 INFO:tasks.rados.rados.0.smithi084.stdout:1288: read oid 3 snap -1 2024-09-20T22:47:03.223 INFO:tasks.rados.rados.0.smithi084.stdout:1288: expect (ObjNum 338 snap 85 seq_num 338) 2024-09-20T22:47:03.223 INFO:tasks.rados.rados.0.smithi084.stdout:1289: rollback oid 42 current snap is 128 2024-09-20T22:47:03.223 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 42 to 122 2024-09-20T22:47:03.224 INFO:tasks.rados.rados.0.smithi084.stdout:1284: finishing rollback tid 1 to smithi08477203-48 2024-09-20T22:47:03.224 INFO:tasks.rados.rados.0.smithi084.stdout:1284: finishing rollback tid 2 to smithi08477203-48 2024-09-20T22:47:03.224 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 48 v 279 (ObjNum 424 snap 109 seq_num 424) dirty exists 2024-09-20T22:47:03.224 INFO:tasks.rados.rados.0.smithi084.stdout:1285: finishing rollback tid 0 to smithi08477203-2 2024-09-20T22:47:03.224 INFO:tasks.rados.rados.0.smithi084.stdout:1285: finishing rollback tid 1 to smithi08477203-2 2024-09-20T22:47:03.224 INFO:tasks.rados.rados.0.smithi084.stdout:1284: done (5 left) 2024-09-20T22:47:03.224 INFO:tasks.rados.rados.0.smithi084.stdout:1290: snap_create 2024-09-20T22:47:03.224 INFO:tasks.rados.rados.0.smithi084.stdout:1285: finishing rollback tid 2 to smithi08477203-2 2024-09-20T22:47:03.224 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 2 v 154 (ObjNum 426 snap 109 seq_num 426) dirty exists 2024-09-20T22:47:03.224 INFO:tasks.rados.rados.0.smithi084.stdout:1286: finishing write tid 1 to smithi08477203-45 2024-09-20T22:47:03.224 INFO:tasks.rados.rados.0.smithi084.stdout:1286: finishing write tid 2 to smithi08477203-45 2024-09-20T22:47:03.224 INFO:tasks.rados.rados.0.smithi084.stdout:1286: finishing write tid 3 to smithi08477203-45 2024-09-20T22:47:03.224 INFO:tasks.rados.rados.0.smithi084.stdout:1286: finishing write tid 4 to smithi08477203-45 2024-09-20T22:47:03.224 INFO:tasks.rados.rados.0.smithi084.stdout:1286: finishing write tid 5 to smithi08477203-45 2024-09-20T22:47:03.225 INFO:tasks.rados.rados.0.smithi084.stdout:1286: oid 45 updating version 0 to 129 2024-09-20T22:47:03.225 INFO:tasks.rados.rados.0.smithi084.stdout:1286: oid 45 updating version 129 to 130 2024-09-20T22:47:03.225 INFO:tasks.rados.rados.0.smithi084.stdout:1286: oid 45 updating version 130 to 131 2024-09-20T22:47:03.225 INFO:tasks.rados.rados.0.smithi084.stdout:1286: oid 45 version 131 is already newer than 128 2024-09-20T22:47:03.225 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 45 v 131 (ObjNum 466 snap 128 seq_num 466) dirty exists 2024-09-20T22:47:03.225 INFO:tasks.rados.rados.0.smithi084.stdout:1286: left oid 45 (ObjNum 466 snap 128 seq_num 466) 2024-09-20T22:47:03.225 INFO:tasks.rados.rados.0.smithi084.stdout:1287: finishing write tid 1 to smithi08477203-16 2024-09-20T22:47:03.225 INFO:tasks.rados.rados.0.smithi084.stdout:1287: finishing write tid 2 to smithi08477203-16 2024-09-20T22:47:03.226 INFO:tasks.rados.rados.0.smithi084.stdout:1288: expect (ObjNum 338 snap 85 seq_num 338) 2024-09-20T22:47:03.317 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:47:03.338 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:03.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:03.339 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:47:03.403 INFO:tasks.rados.rados.0.smithi084.stdout:1287: finishing write tid 3 to smithi08477203-16 2024-09-20T22:47:03.404 INFO:tasks.rados.rados.0.smithi084.stdout:1287: finishing write tid 4 to smithi08477203-16 2024-09-20T22:47:03.404 INFO:tasks.rados.rados.0.smithi084.stdout:1287: finishing write tid 5 to smithi08477203-16 2024-09-20T22:47:03.404 INFO:tasks.rados.rados.0.smithi084.stdout:1287: oid 16 updating version 0 to 196 2024-09-20T22:47:03.404 INFO:tasks.rados.rados.0.smithi084.stdout:1287: oid 16 updating version 196 to 197 2024-09-20T22:47:03.404 INFO:tasks.rados.rados.0.smithi084.stdout:1287: oid 16 version 197 is already newer than 195 2024-09-20T22:47:03.404 INFO:tasks.rados.rados.0.smithi084.stdout:1287: oid 16 version 197 is already newer than 194 2024-09-20T22:47:03.404 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 16 v 197 (ObjNum 467 snap 128 seq_num 467) dirty exists 2024-09-20T22:47:03.404 INFO:tasks.rados.rados.0.smithi084.stdout:1287: left oid 16 (ObjNum 467 snap 128 seq_num 467) 2024-09-20T22:47:03.404 INFO:tasks.rados.rados.0.smithi084.stdout:1289: finishing rollback tid 0 to smithi08477203-42 2024-09-20T22:47:03.404 INFO:tasks.rados.rados.0.smithi084.stdout:1289: finishing rollback tid 1 to smithi08477203-42 2024-09-20T22:47:03.404 INFO:tasks.rados.rados.0.smithi084.stdout:1289: finishing rollback tid 2 to smithi08477203-42 2024-09-20T22:47:03.404 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 42 v 384 (ObjNum 444 snap 117 seq_num 444) dirty exists 2024-09-20T22:47:03.440 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:47:03.462 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:03.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:03.463 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:47:03.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:03 smithi154 ceph-mon[33264]: pgmap v414: 168 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 162 active+clean; 189 MiB data, 650 MiB used, 536 GiB / 536 GiB avail; 14 MiB/s rd, 3.7 MiB/s wr, 16 op/s 2024-09-20T22:47:03.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:03 smithi154 ceph-mon[33264]: osdmap e342: 8 total, 8 up, 6 in 2024-09-20T22:47:03.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:03 smithi154 ceph-mon[33264]: osdmap e343: 8 total, 8 up, 6 in 2024-09-20T22:47:03.529 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:03 smithi084 ceph-mon[40024]: pgmap v414: 168 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 162 active+clean; 189 MiB data, 650 MiB used, 536 GiB / 536 GiB avail; 14 MiB/s rd, 3.7 MiB/s wr, 16 op/s 2024-09-20T22:47:03.529 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:03 smithi084 ceph-mon[40024]: osdmap e342: 8 total, 8 up, 6 in 2024-09-20T22:47:03.529 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:03 smithi084 ceph-mon[40024]: osdmap e343: 8 total, 8 up, 6 in 2024-09-20T22:47:03.529 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:03 smithi084 ceph-mon[29774]: pgmap v414: 168 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 162 active+clean; 189 MiB data, 650 MiB used, 536 GiB / 536 GiB avail; 14 MiB/s rd, 3.7 MiB/s wr, 16 op/s 2024-09-20T22:47:03.529 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:03 smithi084 ceph-mon[29774]: osdmap e342: 8 total, 8 up, 6 in 2024-09-20T22:47:03.529 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:03 smithi084 ceph-mon[29774]: osdmap e343: 8 total, 8 up, 6 in 2024-09-20T22:47:03.563 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:47:03.585 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:03.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:03.586 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:47:03.686 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:47:03.708 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:03.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:03.709 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:47:03.810 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:47:03.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:03.831 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:03.831 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:47:03.932 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:47:03.954 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:03.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:03.955 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:47:04.055 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:47:04.077 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:04.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:04.078 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:47:04.137 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:47:04.179 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:47:04.184 INFO:tasks.rados.rados.0.smithi084.stdout:1285: done (5 left) 2024-09-20T22:47:04.184 INFO:tasks.rados.rados.0.smithi084.stdout:1286: done (4 left) 2024-09-20T22:47:04.184 INFO:tasks.rados.rados.0.smithi084.stdout:1287: done (3 left) 2024-09-20T22:47:04.184 INFO:tasks.rados.rados.0.smithi084.stdout:1288: done (2 left) 2024-09-20T22:47:04.184 INFO:tasks.rados.rados.0.smithi084.stdout:1289: done (1 left) 2024-09-20T22:47:04.184 INFO:tasks.rados.rados.0.smithi084.stdout:1290: done (0 left) 2024-09-20T22:47:04.184 INFO:tasks.rados.rados.0.smithi084.stdout:1291: write oid 22 current snap is 129 2024-09-20T22:47:04.184 INFO:tasks.rados.rados.0.smithi084.stdout:1291: seq_num 468 ranges {665676=714065,1886514=531486,3169137=696539} 2024-09-20T22:47:04.191 INFO:tasks.rados.rados.0.smithi084.stdout:1291: writing smithi08477203-22 from 665676 to 1379741 tid 1 2024-09-20T22:47:04.198 INFO:tasks.rados.rados.0.smithi084.stdout:1291: writing smithi08477203-22 from 1886514 to 2418000 tid 2 2024-09-20T22:47:04.202 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:04.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:04.203 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:47:04.206 INFO:tasks.rados.rados.0.smithi084.stdout:1291: writing smithi08477203-22 from 3169137 to 3865676 tid 3 2024-09-20T22:47:04.207 INFO:tasks.rados.rados.0.smithi084.stdout:1291: finishing write tid 1 to smithi08477203-22 2024-09-20T22:47:04.207 INFO:tasks.rados.rados.0.smithi084.stdout:1292: read oid 1 snap -1 2024-09-20T22:47:04.207 INFO:tasks.rados.rados.0.smithi084.stdout:1292: expect (ObjNum 464 snap 127 seq_num 464) 2024-09-20T22:47:04.207 INFO:tasks.rados.rados.0.smithi084.stdout:1293: snap_create 2024-09-20T22:47:04.207 INFO:tasks.rados.rados.0.smithi084.stdout:1291: finishing write tid 2 to smithi08477203-22 2024-09-20T22:47:04.209 INFO:tasks.rados.rados.0.smithi084.stdout:1291: finishing write tid 3 to smithi08477203-22 2024-09-20T22:47:04.209 INFO:tasks.rados.rados.0.smithi084.stdout:1291: finishing write tid 4 to smithi08477203-22 2024-09-20T22:47:04.210 INFO:tasks.rados.rados.0.smithi084.stdout:1291: finishing write tid 5 to smithi08477203-22 2024-09-20T22:47:04.210 INFO:tasks.rados.rados.0.smithi084.stdout:1291: oid 22 updating version 0 to 317 2024-09-20T22:47:04.210 INFO:tasks.rados.rados.0.smithi084.stdout:1291: oid 22 version 317 is already newer than 316 2024-09-20T22:47:04.210 INFO:tasks.rados.rados.0.smithi084.stdout:1291: oid 22 updating version 317 to 318 2024-09-20T22:47:04.210 INFO:tasks.rados.rados.0.smithi084.stdout:1291: oid 22 version 318 is already newer than 315 2024-09-20T22:47:04.210 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 22 v 318 (ObjNum 468 snap 129 seq_num 468) dirty exists 2024-09-20T22:47:04.210 INFO:tasks.rados.rados.0.smithi084.stdout:1291: left oid 22 (ObjNum 468 snap 129 seq_num 468) 2024-09-20T22:47:04.212 INFO:tasks.rados.rados.0.smithi084.stdout:1292: expect (ObjNum 464 snap 127 seq_num 464) 2024-09-20T22:47:04.304 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:47:04.326 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:04.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:04.327 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:47:04.428 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:47:04.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:04.450 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:04.450 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:47:04.551 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:47:04.572 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:04.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:04.573 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:47:04.626 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:47:04.674 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:47:04.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:04.697 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:04.698 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:47:04.799 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:47:04.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:04.822 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:04.822 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:47:04.923 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:47:04.945 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:04.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:04.946 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:47:05.046 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:47:05.068 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:05.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:05.069 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:47:05.133 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-20T22:47:05.134 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:47:05.170 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:47:05.187 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is set 2024-09-20T22:47:05.192 INFO:tasks.rados.rados.0.smithi084.stdout:1291: done (2 left) 2024-09-20T22:47:05.192 INFO:tasks.rados.rados.0.smithi084.stdout:1292: done (1 left) 2024-09-20T22:47:05.192 INFO:tasks.rados.rados.0.smithi084.stdout:1293: done (0 left) 2024-09-20T22:47:05.192 INFO:tasks.rados.rados.0.smithi084.stdout:1294: delete oid 2 current snap is 130 2024-09-20T22:47:05.195 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:05.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:05.196 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:47:05.196 INFO:tasks.rados.rados.0.smithi084.stdout:1294: done (0 left) 2024-09-20T22:47:05.197 INFO:tasks.rados.rados.0.smithi084.stdout:1295: copy_from oid 27 from oid 19 current snap is 130 2024-09-20T22:47:05.197 INFO:tasks.rados.rados.0.smithi084.stdout:1296: write (excl) oid 44 current snap is 130 2024-09-20T22:47:05.197 INFO:tasks.rados.rados.0.smithi084.stdout:1296: seq_num 470 ranges {699290=464563,1603478=535155,2823983=765846} 2024-09-20T22:47:05.202 INFO:tasks.rados.rados.0.smithi084.stdout:1296: writing smithi08477203-44 from 699290 to 1163853 tid 1 2024-09-20T22:47:05.208 INFO:tasks.rados.rados.0.smithi084.stdout:1296: writing smithi08477203-44 from 1603478 to 2138633 tid 2 2024-09-20T22:47:05.217 INFO:tasks.rados.rados.0.smithi084.stdout:1296: writing smithi08477203-44 from 2823983 to 3589829 tid 3 2024-09-20T22:47:05.217 INFO:tasks.rados.rados.0.smithi084.stdout:1297: read oid 37 snap -1 2024-09-20T22:47:05.218 INFO:tasks.rados.rados.0.smithi084.stdout:1297: expect deleted 2024-09-20T22:47:05.218 INFO:tasks.rados.rados.0.smithi084.stdout:1295: finishing copy_from to smithi08477203-27 2024-09-20T22:47:05.218 INFO:tasks.rados.rados.0.smithi084.stdout:1295: got expected ENOENT (src dne) 2024-09-20T22:47:05.218 INFO:tasks.rados.rados.0.smithi084.stdout:1298: read oid 42 snap 127 2024-09-20T22:47:05.218 INFO:tasks.rados.rados.0.smithi084.stdout:1298: expect (ObjNum 444 snap 117 seq_num 444) 2024-09-20T22:47:05.218 INFO:tasks.rados.rados.0.smithi084.stdout:1299: copy_from oid 12 from oid 31 current snap is 130 2024-09-20T22:47:05.218 INFO:tasks.rados.rados.0.smithi084.stdout:1295: finishing copy_from racing read to smithi08477203-27 2024-09-20T22:47:05.218 INFO:tasks.rados.rados.0.smithi084.stdout:1296: finishing write tid 1 to smithi08477203-44 2024-09-20T22:47:05.218 INFO:tasks.rados.rados.0.smithi084.stdout:1296: finishing write tid 2 to smithi08477203-44 2024-09-20T22:47:05.218 INFO:tasks.rados.rados.0.smithi084.stdout:1295: done (4 left) 2024-09-20T22:47:05.218 INFO:tasks.rados.rados.0.smithi084.stdout:1300: read oid 25 snap 129 2024-09-20T22:47:05.218 INFO:tasks.rados.rados.0.smithi084.stdout:1300: expect (ObjNum 458 snap 126 seq_num 458) 2024-09-20T22:47:05.218 INFO:tasks.rados.rados.0.smithi084.stdout:1301: snap_remove snap 122 2024-09-20T22:47:05.297 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:47:05.318 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:05.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:05.319 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:47:05.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:05 smithi084 ceph-mon[40024]: pgmap v417: 168 pgs: 2 active+clean+snaptrim, 166 active+clean; 217 MiB data, 706 MiB used, 536 GiB / 536 GiB avail; 12 MiB/s rd, 6.4 MiB/s wr, 31 op/s 2024-09-20T22:47:05.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:05 smithi084 ceph-mon[40024]: osdmap e344: 8 total, 8 up, 6 in 2024-09-20T22:47:05.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:05 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/1376922721' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:47:05.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:05 smithi084 ceph-mon[29774]: pgmap v417: 168 pgs: 2 active+clean+snaptrim, 166 active+clean; 217 MiB data, 706 MiB used, 536 GiB / 536 GiB avail; 12 MiB/s rd, 6.4 MiB/s wr, 31 op/s 2024-09-20T22:47:05.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:05 smithi084 ceph-mon[29774]: osdmap e344: 8 total, 8 up, 6 in 2024-09-20T22:47:05.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:05 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/1376922721' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:47:05.420 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:47:05.441 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:05.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:05.442 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:47:05.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:05 smithi154 ceph-mon[33264]: pgmap v417: 168 pgs: 2 active+clean+snaptrim, 166 active+clean; 217 MiB data, 706 MiB used, 536 GiB / 536 GiB avail; 12 MiB/s rd, 6.4 MiB/s wr, 31 op/s 2024-09-20T22:47:05.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:05 smithi154 ceph-mon[33264]: osdmap e344: 8 total, 8 up, 6 in 2024-09-20T22:47:05.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:05 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/1376922721' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:47:05.543 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:47:05.564 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:05.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:05.565 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:47:05.665 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:47:05.687 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:05.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:05.688 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:47:05.789 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:47:05.809 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:05.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:05.810 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:47:05.911 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:47:05.933 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:05.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:05.934 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:47:06.034 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:47:06.056 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:06.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:06.057 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:47:06.157 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:47:06.158 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-20T22:47:06.184 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:06.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:06.185 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:47:06.197 INFO:tasks.rados.rados.0.smithi084.stdout:1301: done (5 left) 2024-09-20T22:47:06.197 INFO:tasks.rados.rados.0.smithi084.stdout:1302: rollback oid 3 current snap is 130 2024-09-20T22:47:06.197 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 3 to 127 2024-09-20T22:47:06.197 INFO:tasks.rados.rados.0.smithi084.stdout:1299: finishing copy_from to smithi08477203-12 2024-09-20T22:47:06.197 INFO:tasks.rados.rados.0.smithi084.stdout:1299: got expected ENOENT (src dne) 2024-09-20T22:47:06.197 INFO:tasks.rados.rados.0.smithi084.stdout:1299: finishing copy_from racing read to smithi08477203-12 2024-09-20T22:47:06.197 INFO:tasks.rados.rados.0.smithi084.stdout:1298: expect (ObjNum 444 snap 117 seq_num 444) 2024-09-20T22:47:06.285 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:47:06.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:06.307 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:06.307 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:47:06.313 INFO:tasks.rados.rados.0.smithi084.stdout:1296: finishing write tid 3 to smithi08477203-44 2024-09-20T22:47:06.313 INFO:tasks.rados.rados.0.smithi084.stdout:1296: finishing write tid 4 to smithi08477203-44 2024-09-20T22:47:06.313 INFO:tasks.rados.rados.0.smithi084.stdout:1296: finishing write tid 5 to smithi08477203-44 2024-09-20T22:47:06.313 INFO:tasks.rados.rados.0.smithi084.stdout:1296: oid 44 updating version 0 to 721 2024-09-20T22:47:06.313 INFO:tasks.rados.rados.0.smithi084.stdout:1296: oid 44 version 721 is already newer than 720 2024-09-20T22:47:06.313 INFO:tasks.rados.rados.0.smithi084.stdout:1296: oid 44 updating version 721 to 722 2024-09-20T22:47:06.314 INFO:tasks.rados.rados.0.smithi084.stdout:1296: oid 44 version 722 is already newer than 719 2024-09-20T22:47:06.314 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 44 v 722 (ObjNum 470 snap 130 seq_num 470) dirty exists 2024-09-20T22:47:06.314 INFO:tasks.rados.rados.0.smithi084.stdout:1296: left oid 44 (ObjNum 470 snap 130 seq_num 470) 2024-09-20T22:47:06.314 INFO:tasks.rados.rados.0.smithi084.stdout:1300: expect (ObjNum 458 snap 126 seq_num 458) 2024-09-20T22:47:06.408 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:47:06.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:06.430 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:06.430 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:47:06.478 INFO:tasks.rados.rados.0.smithi084.stdout:1302: finishing rollback tid 0 to smithi08477203-3 2024-09-20T22:47:06.478 INFO:tasks.rados.rados.0.smithi084.stdout:1302: finishing rollback tid 1 to smithi08477203-3 2024-09-20T22:47:06.479 INFO:tasks.rados.rados.0.smithi084.stdout:1302: finishing rollback tid 2 to smithi08477203-3 2024-09-20T22:47:06.479 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 3 v 191 (ObjNum 338 snap 85 seq_num 338) dirty exists 2024-09-20T22:47:06.479 INFO:tasks.rados.rados.0.smithi084.stdout:1296: done (5 left) 2024-09-20T22:47:06.479 INFO:tasks.rados.rados.0.smithi084.stdout:1297: done (4 left) 2024-09-20T22:47:06.479 INFO:tasks.rados.rados.0.smithi084.stdout:1298: done (3 left) 2024-09-20T22:47:06.479 INFO:tasks.rados.rados.0.smithi084.stdout:1299: done (2 left) 2024-09-20T22:47:06.479 INFO:tasks.rados.rados.0.smithi084.stdout:1300: done (1 left) 2024-09-20T22:47:06.479 INFO:tasks.rados.rados.0.smithi084.stdout:1302: done (0 left) 2024-09-20T22:47:06.479 INFO:tasks.rados.rados.0.smithi084.stdout:1303: write (excl) oid 31 current snap is 130 2024-09-20T22:47:06.479 INFO:tasks.rados.rados.0.smithi084.stdout:1303: seq_num 471 ranges {716097=539812,1945607=795166,3535053=381044} 2024-09-20T22:47:06.485 INFO:tasks.rados.rados.0.smithi084.stdout:1303: writing smithi08477203-31 from 716097 to 1255909 tid 1 2024-09-20T22:47:06.493 INFO:tasks.rados.rados.0.smithi084.stdout:1303: writing smithi08477203-31 from 1945607 to 2740773 tid 2 2024-09-20T22:47:06.500 INFO:tasks.rados.rados.0.smithi084.stdout:1303: writing smithi08477203-31 from 3535053 to 3916097 tid 3 2024-09-20T22:47:06.500 INFO:tasks.rados.rados.0.smithi084.stdout:1304: snap_remove snap 121 2024-09-20T22:47:06.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:06 smithi154 ceph-mon[33264]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:47:06.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:06 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/1376922721' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T22:47:06.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:06 smithi154 ceph-mon[33264]: osdmap e345: 8 total, 8 up, 6 in 2024-09-20T22:47:06.532 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:47:06.553 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:06.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:06.554 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:47:06.614 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:06 smithi084 ceph-mon[29774]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:47:06.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:06 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/1376922721' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T22:47:06.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:06 smithi084 ceph-mon[29774]: osdmap e345: 8 total, 8 up, 6 in 2024-09-20T22:47:06.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:06 smithi084 ceph-mon[40024]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:47:06.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:06 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/1376922721' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T22:47:06.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:06 smithi084 ceph-mon[40024]: osdmap e345: 8 total, 8 up, 6 in 2024-09-20T22:47:06.641 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.2 dump_blocked_ops 2024-09-20T22:47:06.655 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:47:06.708 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:06.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:06.709 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:47:06.761 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:47:06.810 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:47:06.832 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:06.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:06.834 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:47:06.934 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:47:06.955 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:06.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:06.956 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:47:07.057 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:47:07.079 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:07.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:07.080 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:47:07.181 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:47:07.202 INFO:tasks.rados.rados.0.smithi084.stdout:1304: done (1 left) 2024-09-20T22:47:07.202 INFO:tasks.rados.rados.0.smithi084.stdout:1305: write oid 18 current snap is 130 2024-09-20T22:47:07.202 INFO:tasks.rados.rados.0.smithi084.stdout:1305: seq_num 472 ranges {732904=731414,2120442=538824,3278829=504743} 2024-09-20T22:47:07.203 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:07.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:07.204 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:47:07.210 INFO:tasks.rados.rados.0.smithi084.stdout:1305: writing smithi08477203-18 from 732904 to 1464318 tid 1 2024-09-20T22:47:07.216 INFO:tasks.rados.rados.0.smithi084.stdout:1305: writing smithi08477203-18 from 2120442 to 2659266 tid 2 2024-09-20T22:47:07.224 INFO:tasks.rados.rados.0.smithi084.stdout:1305: writing smithi08477203-18 from 3278829 to 3783572 tid 3 2024-09-20T22:47:07.224 INFO:tasks.rados.rados.0.smithi084.stdout:1306: copy_from oid 3 from oid 10 current snap is 130 2024-09-20T22:47:07.224 INFO:tasks.rados.rados.0.smithi084.stdout:1303: finishing write tid 1 to smithi08477203-31 2024-09-20T22:47:07.224 INFO:tasks.rados.rados.0.smithi084.stdout:1303: finishing write tid 2 to smithi08477203-31 2024-09-20T22:47:07.224 INFO:tasks.rados.rados.0.smithi084.stdout:1303: finishing write tid 3 to smithi08477203-31 2024-09-20T22:47:07.224 INFO:tasks.rados.rados.0.smithi084.stdout:1303: finishing write tid 4 to smithi08477203-31 2024-09-20T22:47:07.224 INFO:tasks.rados.rados.0.smithi084.stdout:1303: finishing write tid 5 to smithi08477203-31 2024-09-20T22:47:07.224 INFO:tasks.rados.rados.0.smithi084.stdout:1303: oid 31 updating version 0 to 397 2024-09-20T22:47:07.224 INFO:tasks.rados.rados.0.smithi084.stdout:1303: oid 31 version 397 is already newer than 395 2024-09-20T22:47:07.224 INFO:tasks.rados.rados.0.smithi084.stdout:1303: oid 31 version 397 is already newer than 396 2024-09-20T22:47:07.225 INFO:tasks.rados.rados.0.smithi084.stdout:1303: oid 31 updating version 397 to 398 2024-09-20T22:47:07.225 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 31 v 398 (ObjNum 471 snap 130 seq_num 471) dirty exists 2024-09-20T22:47:07.225 INFO:tasks.rados.rados.0.smithi084.stdout:1303: left oid 31 (ObjNum 471 snap 130 seq_num 471) 2024-09-20T22:47:07.225 INFO:tasks.rados.rados.0.smithi084.stdout:1305: finishing write tid 1 to smithi08477203-18 2024-09-20T22:47:07.225 INFO:tasks.rados.rados.0.smithi084.stdout:1305: finishing write tid 2 to smithi08477203-18 2024-09-20T22:47:07.225 INFO:tasks.rados.rados.0.smithi084.stdout:1303: done (2 left) 2024-09-20T22:47:07.225 INFO:tasks.rados.rados.0.smithi084.stdout:1307: read oid 13 snap -1 2024-09-20T22:47:07.225 INFO:tasks.rados.rados.0.smithi084.stdout:1307: expect (ObjNum 411 snap 105 seq_num 411) 2024-09-20T22:47:07.225 INFO:tasks.rados.rados.0.smithi084.stdout:1308: copy_from oid 42 from oid 6 current snap is 130 2024-09-20T22:47:07.225 INFO:tasks.rados.rados.0.smithi084.stdout:1309: snap_create 2024-09-20T22:47:07.229 INFO:tasks.rados.rados.0.smithi084.stdout:1305: finishing write tid 3 to smithi08477203-18 2024-09-20T22:47:07.229 INFO:tasks.rados.rados.0.smithi084.stdout:1305: finishing write tid 4 to smithi08477203-18 2024-09-20T22:47:07.229 INFO:tasks.rados.rados.0.smithi084.stdout:1305: finishing write tid 5 to smithi08477203-18 2024-09-20T22:47:07.229 INFO:tasks.rados.rados.0.smithi084.stdout:1305: oid 18 updating version 0 to 404 2024-09-20T22:47:07.229 INFO:tasks.rados.rados.0.smithi084.stdout:1305: oid 18 version 404 is already newer than 403 2024-09-20T22:47:07.229 INFO:tasks.rados.rados.0.smithi084.stdout:1305: oid 18 updating version 404 to 406 2024-09-20T22:47:07.230 INFO:tasks.rados.rados.0.smithi084.stdout:1305: oid 18 version 406 is already newer than 405 2024-09-20T22:47:07.230 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 18 v 406 (ObjNum 472 snap 130 seq_num 472) dirty exists 2024-09-20T22:47:07.230 INFO:tasks.rados.rados.0.smithi084.stdout:1305: left oid 18 (ObjNum 472 snap 130 seq_num 472) 2024-09-20T22:47:07.233 INFO:tasks.rados.rados.0.smithi084.stdout:1307: expect (ObjNum 411 snap 105 seq_num 411) 2024-09-20T22:47:07.291 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:47:07.291 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:47:07.304 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:47:07.310 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:47:07.310 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:47:07.323 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:47:07.325 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:07.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:07.326 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:47:07.330 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:47:07.330 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:47:07.348 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-20T22:47:07.349 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T22:47:07.369 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:47:07.369 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:47:07.385 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:47:07.385 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:47:07.397 INFO:tasks.rados.rados.0.smithi084.stdout:1306: finishing copy_from racing read to smithi08477203-3 2024-09-20T22:47:07.397 INFO:tasks.rados.rados.0.smithi084.stdout:1308: finishing copy_from racing read to smithi08477203-42 2024-09-20T22:47:07.397 INFO:tasks.rados.rados.0.smithi084.stdout:1306: finishing copy_from to smithi08477203-3 2024-09-20T22:47:07.398 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 3 v 397 (ObjNum 419 snap 105 seq_num 419) dirty exists 2024-09-20T22:47:07.398 INFO:tasks.rados.rados.0.smithi084.stdout:1308: finishing copy_from to smithi08477203-42 2024-09-20T22:47:07.398 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 42 v 126 (ObjNum 433 snap 112 seq_num 433) dirty exists 2024-09-20T22:47:07.402 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:47:07.403 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:47:07.419 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:47:07.419 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:47:07.427 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:47:07.429 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:07 smithi084 ceph-mon[29774]: pgmap v420: 168 pgs: 2 active+clean+snaptrim, 166 active+clean; 217 MiB data, 706 MiB used, 536 GiB / 536 GiB avail; 12 MiB/s rd, 6.4 MiB/s wr, 31 op/s 2024-09-20T22:47:07.429 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:07 smithi084 ceph-mon[29774]: purged_snaps scrub starts 2024-09-20T22:47:07.430 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:07 smithi084 ceph-mon[29774]: purged_snaps scrub ok 2024-09-20T22:47:07.430 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:07 smithi084 ceph-mon[29774]: osdmap e346: 8 total, 8 up, 6 in 2024-09-20T22:47:07.430 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:07 smithi084 ceph-mon[29774]: purged_snaps scrub starts 2024-09-20T22:47:07.430 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:07 smithi084 ceph-mon[29774]: purged_snaps scrub ok 2024-09-20T22:47:07.430 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:07 smithi084 ceph-mon[40024]: pgmap v420: 168 pgs: 2 active+clean+snaptrim, 166 active+clean; 217 MiB data, 706 MiB used, 536 GiB / 536 GiB avail; 12 MiB/s rd, 6.4 MiB/s wr, 31 op/s 2024-09-20T22:47:07.430 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:07 smithi084 ceph-mon[40024]: purged_snaps scrub starts 2024-09-20T22:47:07.430 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:07 smithi084 ceph-mon[40024]: purged_snaps scrub ok 2024-09-20T22:47:07.430 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:07 smithi084 ceph-mon[40024]: osdmap e346: 8 total, 8 up, 6 in 2024-09-20T22:47:07.431 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:07 smithi084 ceph-mon[40024]: purged_snaps scrub starts 2024-09-20T22:47:07.431 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:07 smithi084 ceph-mon[40024]: purged_snaps scrub ok 2024-09-20T22:47:07.448 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:07.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:07.449 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:47:07.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:07 smithi154 ceph-mon[33264]: pgmap v420: 168 pgs: 2 active+clean+snaptrim, 166 active+clean; 217 MiB data, 706 MiB used, 536 GiB / 536 GiB avail; 12 MiB/s rd, 6.4 MiB/s wr, 31 op/s 2024-09-20T22:47:07.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:07 smithi154 ceph-mon[33264]: purged_snaps scrub starts 2024-09-20T22:47:07.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:07 smithi154 ceph-mon[33264]: purged_snaps scrub ok 2024-09-20T22:47:07.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:07 smithi154 ceph-mon[33264]: osdmap e346: 8 total, 8 up, 6 in 2024-09-20T22:47:07.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:07 smithi154 ceph-mon[33264]: purged_snaps scrub starts 2024-09-20T22:47:07.507 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:07 smithi154 ceph-mon[33264]: purged_snaps scrub ok 2024-09-20T22:47:07.550 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:47:07.571 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:07.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:07.572 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:47:07.673 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:47:07.695 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:07.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:07.696 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:47:07.797 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:47:07.818 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:07.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:07.819 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:47:07.920 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:47:07.941 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:07.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:07.942 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:47:08.043 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:47:08.064 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:08.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:08.066 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:47:08.167 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:47:08.181 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd unset noscrub 2024-09-20T22:47:08.183 INFO:tasks.rados.rados.0.smithi084.stdout:1305: done (4 left) 2024-09-20T22:47:08.183 INFO:tasks.rados.rados.0.smithi084.stdout:1306: done (3 left) 2024-09-20T22:47:08.183 INFO:tasks.rados.rados.0.smithi084.stdout:1307: done (2 left) 2024-09-20T22:47:08.183 INFO:tasks.rados.rados.0.smithi084.stdout:1308: done (1 left) 2024-09-20T22:47:08.183 INFO:tasks.rados.rados.0.smithi084.stdout:1309: done (0 left) 2024-09-20T22:47:08.184 INFO:tasks.rados.rados.0.smithi084.stdout:1310: snap_remove snap 128 2024-09-20T22:47:08.191 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:08.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:08.192 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:47:08.293 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:47:08.314 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:08.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:08.315 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:47:08.416 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:47:08.438 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:08.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:08.439 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:47:08.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:08 smithi154 ceph-mon[33264]: osdmap e347: 8 total, 8 up, 6 in 2024-09-20T22:47:08.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:08 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/1898877597' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:47:08.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:08 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:47:08.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:08 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:47:08.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:08 smithi154 ceph-mon[33264]: osdmap e348: 8 total, 8 up, 6 in 2024-09-20T22:47:08.518 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:08 smithi084 ceph-mon[40024]: osdmap e347: 8 total, 8 up, 6 in 2024-09-20T22:47:08.518 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:08 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/1898877597' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:47:08.518 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:08 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:47:08.518 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:08 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:47:08.518 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:08 smithi084 ceph-mon[40024]: osdmap e348: 8 total, 8 up, 6 in 2024-09-20T22:47:08.518 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:47:08 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: ::ffff:172.21.15.154 - - [20/Sep/2024:22:47:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-20T22:47:08.519 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:08 smithi084 ceph-mon[29774]: osdmap e347: 8 total, 8 up, 6 in 2024-09-20T22:47:08.519 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:08 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/1898877597' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:47:08.519 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:08 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:47:08.519 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:08 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:47:08.519 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:08 smithi084 ceph-mon[29774]: osdmap e348: 8 total, 8 up, 6 in 2024-09-20T22:47:08.519 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-20T22:47:08.519 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2024-09-20T22:47:08.519 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2024-09-20T22:47:08.519 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2024-09-20T22:47:08.520 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-20T22:47:08.540 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:47:08.554 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 5, 6, 7] out_osds: [4, 0] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-20T22:47:08.555 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-20T22:47:08.555 INFO:tasks.thrashosds.thrasher:inject_pause on osd.3 2024-09-20T22:47:08.555 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-20T22:47:08.555 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-20T22:47:08.555 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.3 config set filestore_inject_stall 3 2024-09-20T22:47:08.561 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:08.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:08.562 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:47:08.663 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:47:08.684 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:08.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:08.686 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:47:08.786 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:47:08.808 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:08.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:08.809 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:47:08.910 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:47:08.932 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:08.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:08.933 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:47:09.034 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:47:09.055 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:09.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:09.056 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:47:09.157 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:47:09.167 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.2 dump_historic_ops 2024-09-20T22:47:09.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:09.182 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:09.182 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:47:09.192 INFO:tasks.rados.rados.0.smithi084.stdout:1310: done (0 left) 2024-09-20T22:47:09.192 INFO:tasks.rados.rados.0.smithi084.stdout:1311: snap_remove snap 118 2024-09-20T22:47:09.240 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:47:09.283 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:47:09.306 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:09.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:09.307 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:47:09.392 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:09 smithi084 ceph-mon[40024]: pgmap v423: 168 pgs: 2 active+clean+snaptrim, 166 active+clean; 217 MiB data, 706 MiB used, 536 GiB / 536 GiB avail 2024-09-20T22:47:09.392 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:09 smithi084 ceph-mon[40024]: osdmap e349: 8 total, 8 up, 6 in 2024-09-20T22:47:09.393 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:09 smithi084 ceph-mon[29774]: pgmap v423: 168 pgs: 2 active+clean+snaptrim, 166 active+clean; 217 MiB data, 706 MiB used, 536 GiB / 536 GiB avail 2024-09-20T22:47:09.393 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:09 smithi084 ceph-mon[29774]: osdmap e349: 8 total, 8 up, 6 in 2024-09-20T22:47:09.407 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:47:09.430 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:09.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:09.431 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:47:09.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:09 smithi154 ceph-mon[33264]: pgmap v423: 168 pgs: 2 active+clean+snaptrim, 166 active+clean; 217 MiB data, 706 MiB used, 536 GiB / 536 GiB avail 2024-09-20T22:47:09.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:09 smithi154 ceph-mon[33264]: osdmap e349: 8 total, 8 up, 6 in 2024-09-20T22:47:09.504 INFO:journalctl@ceph.iscsi.iscsi.a.smithi154.stdout:Sep 20 22:47:09 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-iscsi-iscsi-a[59730]: debug there is no tcmu-runner data available 2024-09-20T22:47:09.532 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:47:09.555 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:09.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:09.556 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:47:09.657 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:47:09.679 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:09.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:09.680 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:47:09.703 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:47:09.703 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-20T22:47:09.703 INFO:teuthology.orchestra.run.smithi084.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-20T22:47:09.704 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-20T22:47:09.781 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:47:09.803 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:09.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:09.805 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:47:09.905 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:47:09.922 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:47:09.928 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:09.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:09.929 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:47:10.029 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:47:10.051 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:10.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:10.052 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:47:10.153 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:47:10.176 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:10.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:10.177 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:47:10.222 INFO:tasks.rados.rados.0.smithi084.stdout:1311: done (0 left) 2024-09-20T22:47:10.222 INFO:tasks.rados.rados.0.smithi084.stdout:1312: write oid 1 current snap is 131 2024-09-20T22:47:10.222 INFO:tasks.rados.rados.0.smithi084.stdout:1312: seq_num 473 ranges {749711=406663,1778924=682482,3189899=716015} 2024-09-20T22:47:10.228 INFO:tasks.rados.rados.0.smithi084.stdout:1312: writing smithi08477203-1 from 749711 to 1156374 tid 1 2024-09-20T22:47:10.235 INFO:tasks.rados.rados.0.smithi084.stdout:1312: writing smithi08477203-1 from 1778924 to 2461406 tid 2 2024-09-20T22:47:10.244 INFO:tasks.rados.rados.0.smithi084.stdout:1312: writing smithi08477203-1 from 3189899 to 3905914 tid 3 2024-09-20T22:47:10.244 INFO:tasks.rados.rados.0.smithi084.stdout:1313: snap_create 2024-09-20T22:47:10.244 INFO:tasks.rados.rados.0.smithi084.stdout:1312: finishing write tid 1 to smithi08477203-1 2024-09-20T22:47:10.244 INFO:tasks.rados.rados.0.smithi084.stdout:1312: finishing write tid 2 to smithi08477203-1 2024-09-20T22:47:10.248 INFO:tasks.rados.rados.0.smithi084.stdout:1312: finishing write tid 3 to smithi08477203-1 2024-09-20T22:47:10.248 INFO:tasks.rados.rados.0.smithi084.stdout:1312: finishing write tid 4 to smithi08477203-1 2024-09-20T22:47:10.248 INFO:tasks.rados.rados.0.smithi084.stdout:1312: finishing write tid 5 to smithi08477203-1 2024-09-20T22:47:10.248 INFO:tasks.rados.rados.0.smithi084.stdout:1312: oid 1 updating version 0 to 410 2024-09-20T22:47:10.248 INFO:tasks.rados.rados.0.smithi084.stdout:1312: oid 1 version 410 is already newer than 408 2024-09-20T22:47:10.248 INFO:tasks.rados.rados.0.smithi084.stdout:1312: oid 1 version 410 is already newer than 409 2024-09-20T22:47:10.248 INFO:tasks.rados.rados.0.smithi084.stdout:1312: oid 1 version 410 is already newer than 407 2024-09-20T22:47:10.248 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 1 v 410 (ObjNum 473 snap 131 seq_num 473) dirty exists 2024-09-20T22:47:10.248 INFO:tasks.rados.rados.0.smithi084.stdout:1312: left oid 1 (ObjNum 473 snap 131 seq_num 473) 2024-09-20T22:47:10.279 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:47:10.297 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-20T22:47:10.301 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:10.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:10.302 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:47:10.319 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:10 smithi084 ceph-mon[40024]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:47:10.319 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:10 smithi084 ceph-mon[40024]: osdmap e350: 8 total, 8 up, 6 in 2024-09-20T22:47:10.319 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:10 smithi084 ceph-mon[29774]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:47:10.320 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:10 smithi084 ceph-mon[29774]: osdmap e350: 8 total, 8 up, 6 in 2024-09-20T22:47:10.402 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:47:10.423 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:10.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:10.424 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:47:10.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:10 smithi154 ceph-mon[33264]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:47:10.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:10 smithi154 ceph-mon[33264]: osdmap e350: 8 total, 8 up, 6 in 2024-09-20T22:47:10.526 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:47:10.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:10.549 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:10.549 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:47:10.650 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:47:10.673 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:10.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:10.674 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:47:10.775 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:47:10.797 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:10.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:10.798 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:47:10.899 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:47:10.922 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:10.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:10.923 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:47:11.024 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:47:11.047 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:11.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:11.048 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:47:11.148 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:47:11.170 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:11.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:11.171 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:47:11.243 INFO:tasks.rados.rados.0.smithi084.stdout:1312: done (1 left) 2024-09-20T22:47:11.243 INFO:tasks.rados.rados.0.smithi084.stdout:1313: done (0 left) 2024-09-20T22:47:11.243 INFO:tasks.rados.rados.0.smithi084.stdout:1314: read oid 7 snap -1 2024-09-20T22:47:11.243 INFO:tasks.rados.rados.0.smithi084.stdout:1314: expect (ObjNum 442 snap 116 seq_num 442) 2024-09-20T22:47:11.243 INFO:tasks.rados.rados.0.smithi084.stdout:1315: read oid 14 snap -1 2024-09-20T22:47:11.243 INFO:tasks.rados.rados.0.smithi084.stdout:1315: expect (ObjNum 455 snap 125 seq_num 455) 2024-09-20T22:47:11.244 INFO:tasks.rados.rados.0.smithi084.stdout:1316: read oid 3 snap -1 2024-09-20T22:47:11.244 INFO:tasks.rados.rados.0.smithi084.stdout:1316: expect (ObjNum 419 snap 105 seq_num 419) 2024-09-20T22:47:11.244 INFO:tasks.rados.rados.0.smithi084.stdout:1317: rollback oid 8 current snap is 132 2024-09-20T22:47:11.244 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 8 to 125 2024-09-20T22:47:11.244 INFO:tasks.rados.rados.0.smithi084.stdout:1318: snap_create 2024-09-20T22:47:11.251 INFO:tasks.rados.rados.0.smithi084.stdout:1316: expect (ObjNum 419 snap 105 seq_num 419) 2024-09-20T22:47:11.272 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:47:11.292 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:11.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:11.294 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:47:11.299 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-20T22:47:11.299 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2024-09-20T22:47:11.299 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2024-09-20T22:47:11.299 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [ 2024-09-20T22:47:11.299 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.300 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:901 7.e 7:7c62ced4:::smithi08477203-17:head [copy-from ver 48] snapc 1e=[] ondisk+write+known_if_redirected+supports_pool_eio e156)", 2024-09-20T22:47:11.300 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:43:55.624762+0000", 2024-09-20T22:47:11.300 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 195.67153392, 2024-09-20T22:47:11.300 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.021835856000000001, 2024-09-20T22:47:11.300 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:47:11.300 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:47:11.300 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:47:11.300 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:47:11.300 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:47:11.300 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:47:11.300 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 901 2024-09-20T22:47:11.300 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.300 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:47:11.300 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.301 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:47:11.301 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.624762+0000", 2024-09-20T22:47:11.301 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:47:11.301 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.301 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.301 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:47:11.301 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.624762+0000", 2024-09-20T22:47:11.301 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:47:11.301 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.301 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.301 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:47:11.301 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.624765+0000", 2024-09-20T22:47:11.301 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.57e-06 2024-09-20T22:47:11.301 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.302 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.302 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:47:11.302 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.624776+0000", 2024-09-20T22:47:11.302 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.078e-05 2024-09-20T22:47:11.302 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.302 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.302 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:47:11.302 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.624803+0000", 2024-09-20T22:47:11.302 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.6769999999999999e-05 2024-09-20T22:47:11.302 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.302 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.302 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:47:11.302 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.624815+0000", 2024-09-20T22:47:11.303 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2459e-05 2024-09-20T22:47:11.303 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.303 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.303 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:47:11.303 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.624936+0000", 2024-09-20T22:47:11.303 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000120467 2024-09-20T22:47:11.303 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.303 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.303 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:47:11.303 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.625046+0000", 2024-09-20T22:47:11.303 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000110222 2024-09-20T22:47:11.303 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.303 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.303 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 0,5", 2024-09-20T22:47:11.303 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.635547+0000", 2024-09-20T22:47:11.304 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010501765999999999 2024-09-20T22:47:11.304 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.304 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.304 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:47:11.304 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.644332+0000", 2024-09-20T22:47:11.304 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0087847729999999992 2024-09-20T22:47:11.304 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.304 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.304 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:47:11.304 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.646168+0000", 2024-09-20T22:47:11.304 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0018360010000000001 2024-09-20T22:47:11.304 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.304 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.304 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:47:11.305 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.646512+0000", 2024-09-20T22:47:11.305 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000343269 2024-09-20T22:47:11.305 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.305 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.305 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:47:11.305 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.646550+0000", 2024-09-20T22:47:11.305 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.8383000000000002e-05 2024-09-20T22:47:11.305 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.305 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.305 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:47:11.305 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.646598+0000", 2024-09-20T22:47:11.305 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.8396000000000002e-05 2024-09-20T22:47:11.305 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:47:11.305 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:47:11.306 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:47:11.306 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.306 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.306 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:902 7.e 7:7c62ced4:::smithi08477203-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e156)", 2024-09-20T22:47:11.306 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:43:55.624853+0000", 2024-09-20T22:47:11.306 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 195.671443842, 2024-09-20T22:47:11.306 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.015323603999999999, 2024-09-20T22:47:11.306 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:47:11.306 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:47:11.306 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-20T22:47:11.306 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:47:11.306 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:47:11.306 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:47:11.306 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 902 2024-09-20T22:47:11.307 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.307 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:47:11.307 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.307 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:47:11.307 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.624853+0000", 2024-09-20T22:47:11.308 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:47:11.308 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.308 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.308 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:47:11.308 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.624853+0000", 2024-09-20T22:47:11.308 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:47:11.308 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.308 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.308 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:47:11.308 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.624853+0000", 2024-09-20T22:47:11.308 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.1399999999999997e-07 2024-09-20T22:47:11.308 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.308 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.309 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:47:11.309 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.624859+0000", 2024-09-20T22:47:11.309 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.9610000000000002e-06 2024-09-20T22:47:11.309 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.310 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.310 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:47:11.310 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.624887+0000", 2024-09-20T22:47:11.310 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.7597e-05 2024-09-20T22:47:11.310 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.310 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.310 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:47:11.310 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.624894+0000", 2024-09-20T22:47:11.310 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.469e-06 2024-09-20T22:47:11.310 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.310 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.310 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:47:11.310 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.625219+0000", 2024-09-20T22:47:11.310 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00032538499999999998 2024-09-20T22:47:11.311 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.311 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.311 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for blocked object", 2024-09-20T22:47:11.311 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.625258+0000", 2024-09-20T22:47:11.311 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.8467000000000003e-05 2024-09-20T22:47:11.311 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.311 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.311 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:47:11.311 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.640050+0000", 2024-09-20T22:47:11.311 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0147923 2024-09-20T22:47:11.311 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.311 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.311 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:47:11.311 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.640112+0000", 2024-09-20T22:47:11.311 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.1723000000000005e-05 2024-09-20T22:47:11.312 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.312 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.312 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:47:11.312 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.640176+0000", 2024-09-20T22:47:11.312 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.4188000000000003e-05 2024-09-20T22:47:11.312 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:47:11.312 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:47:11.312 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:47:11.312 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.312 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.312 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24652.0:1420 7.19 e189/139 7:9cc0b7e2:::smithi08477203-46:head v 189'99, mlcod=174'95)", 2024-09-20T22:47:11.312 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:44:28.150258+0000", 2024-09-20T22:47:11.312 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 163.146038146, 2024-09-20T22:47:11.312 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014180626, 2024-09-20T22:47:11.313 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:47:11.313 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:47:11.313 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:47:11.313 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:47:11.313 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.313 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:47:11.313 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.150258+0000", 2024-09-20T22:47:11.313 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:47:11.313 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.313 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.313 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:47:11.313 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.150258+0000", 2024-09-20T22:47:11.313 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:47:11.313 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.313 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.314 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:47:11.314 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.150259+0000", 2024-09-20T22:47:11.314 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.99e-07 2024-09-20T22:47:11.314 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.314 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.314 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:47:11.314 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.150674+0000", 2024-09-20T22:47:11.314 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00041528000000000002 2024-09-20T22:47:11.314 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.314 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.314 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:47:11.314 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.150688+0000", 2024-09-20T22:47:11.314 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3399e-05 2024-09-20T22:47:11.315 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.315 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.315 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:47:11.315 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.150699+0000", 2024-09-20T22:47:11.315 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1022999999999999e-05 2024-09-20T22:47:11.315 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.315 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.315 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:47:11.315 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.150772+0000", 2024-09-20T22:47:11.315 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.3455999999999996e-05 2024-09-20T22:47:11.315 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.315 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.315 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:47:11.315 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.150798+0000", 2024-09-20T22:47:11.316 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.5777000000000001e-05 2024-09-20T22:47:11.316 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.316 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.316 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:47:11.316 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.164390+0000", 2024-09-20T22:47:11.316 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013591789999999999 2024-09-20T22:47:11.316 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.316 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.316 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:47:11.316 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.164439+0000", 2024-09-20T22:47:11.316 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.9202000000000001e-05 2024-09-20T22:47:11.316 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:47:11.316 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:47:11.316 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:47:11.317 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.317 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.317 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24652.0:1421 7.19 e189/139 7:9cc0b7e2:::smithi08477203-46:head v 189'100, mlcod=174'95)", 2024-09-20T22:47:11.317 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:44:28.156805+0000", 2024-09-20T22:47:11.317 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 163.13949128799999, 2024-09-20T22:47:11.317 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013180661999999999, 2024-09-20T22:47:11.317 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:47:11.317 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:47:11.317 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:47:11.317 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:47:11.317 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.317 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:47:11.317 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.156805+0000", 2024-09-20T22:47:11.317 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:47:11.318 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.318 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.318 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:47:11.318 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.156805+0000", 2024-09-20T22:47:11.318 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:47:11.318 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.318 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.318 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:47:11.318 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.156806+0000", 2024-09-20T22:47:11.318 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.074e-06 2024-09-20T22:47:11.318 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.318 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.318 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:47:11.318 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.157126+0000", 2024-09-20T22:47:11.319 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00031991399999999997 2024-09-20T22:47:11.319 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.319 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.319 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:47:11.319 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.157136+0000", 2024-09-20T22:47:11.319 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.4499999999999993e-06 2024-09-20T22:47:11.319 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.319 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.319 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:47:11.319 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.157143+0000", 2024-09-20T22:47:11.319 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.7649999999999993e-06 2024-09-20T22:47:11.319 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.319 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.319 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:47:11.319 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.157188+0000", 2024-09-20T22:47:11.320 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.4885000000000001e-05 2024-09-20T22:47:11.320 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.320 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.320 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:47:11.320 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.157208+0000", 2024-09-20T22:47:11.320 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.0015000000000001e-05 2024-09-20T22:47:11.320 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.320 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.320 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:47:11.320 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.169936+0000", 2024-09-20T22:47:11.320 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.012728204 2024-09-20T22:47:11.320 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.320 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.320 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:47:11.320 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.169986+0000", 2024-09-20T22:47:11.321 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.9354999999999998e-05 2024-09-20T22:47:11.321 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:47:11.321 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:47:11.321 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:47:11.321 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.321 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.321 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24652.0:1654 7.19 e206/139 7:9970b578:::smithi08477203-15:head v 206'114, mlcod=205'111)", 2024-09-20T22:47:11.321 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:44:45.167994+0000", 2024-09-20T22:47:11.321 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 146.128302738, 2024-09-20T22:47:11.321 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.01345262, 2024-09-20T22:47:11.321 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:47:11.321 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:47:11.321 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:47:11.321 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:47:11.321 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.322 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:47:11.323 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:45.167994+0000", 2024-09-20T22:47:11.323 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:47:11.323 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.323 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.323 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:47:11.323 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:45.167994+0000", 2024-09-20T22:47:11.323 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:47:11.323 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.323 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.323 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:47:11.323 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:45.167996+0000", 2024-09-20T22:47:11.324 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.5100000000000001e-06 2024-09-20T22:47:11.324 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.324 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.324 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:47:11.324 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:45.175169+0000", 2024-09-20T22:47:11.324 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0071732740000000003 2024-09-20T22:47:11.324 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.324 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.324 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:47:11.324 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:45.175208+0000", 2024-09-20T22:47:11.324 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.8597000000000003e-05 2024-09-20T22:47:11.324 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.324 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.325 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:47:11.325 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:45.175223+0000", 2024-09-20T22:47:11.325 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.4627000000000001e-05 2024-09-20T22:47:11.325 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.325 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.325 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:47:11.325 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:45.175304+0000", 2024-09-20T22:47:11.325 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.1111000000000003e-05 2024-09-20T22:47:11.325 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.325 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.325 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:47:11.325 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:45.175336+0000", 2024-09-20T22:47:11.325 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.1945e-05 2024-09-20T22:47:11.326 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.326 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.326 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:47:11.326 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:45.181392+0000", 2024-09-20T22:47:11.326 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0060566630000000003 2024-09-20T22:47:11.326 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.326 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.326 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:47:11.326 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:45.181446+0000", 2024-09-20T22:47:11.326 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.3893000000000001e-05 2024-09-20T22:47:11.326 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:47:11.326 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:47:11.327 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:47:11.327 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.327 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.327 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24652.0:1799 7.17 e229/139 7:ea720b85:::smithi08477203-26:head v 229'168, mlcod=222'164)", 2024-09-20T22:47:11.327 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:45:08.171867+0000", 2024-09-20T22:47:11.327 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 123.12442952000001, 2024-09-20T22:47:11.327 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014913454, 2024-09-20T22:47:11.327 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:47:11.327 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:47:11.327 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:47:11.327 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:47:11.327 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.327 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:47:11.327 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.171867+0000", 2024-09-20T22:47:11.327 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:47:11.328 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.328 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.328 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:47:11.328 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.171867+0000", 2024-09-20T22:47:11.328 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:47:11.328 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.328 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.328 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:47:11.328 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.171867+0000", 2024-09-20T22:47:11.328 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.3799999999999999e-07 2024-09-20T22:47:11.328 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.328 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.328 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:47:11.328 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.173016+0000", 2024-09-20T22:47:11.328 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0011485740000000001 2024-09-20T22:47:11.329 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.329 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.329 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:47:11.329 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.173024+0000", 2024-09-20T22:47:11.329 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.4780000000000008e-06 2024-09-20T22:47:11.329 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.329 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.329 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:47:11.329 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.173034+0000", 2024-09-20T22:47:11.329 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.7860000000000008e-06 2024-09-20T22:47:11.329 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.329 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.329 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:47:11.329 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.173098+0000", 2024-09-20T22:47:11.331 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.4405000000000004e-05 2024-09-20T22:47:11.331 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.331 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.331 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:47:11.331 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.173125+0000", 2024-09-20T22:47:11.331 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.6950000000000001e-05 2024-09-20T22:47:11.331 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.331 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.331 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:47:11.331 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.186711+0000", 2024-09-20T22:47:11.332 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.01358608 2024-09-20T22:47:11.332 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.332 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.332 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:47:11.332 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.186780+0000", 2024-09-20T22:47:11.332 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.8942999999999994e-05 2024-09-20T22:47:11.332 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:47:11.332 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:47:11.332 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:47:11.332 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.332 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.332 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:2321 7.8 7:12e9c8db:::smithi08477203-11:head [rollback 3e] snapc 55=[55,53,52,51,50,4b,47,3e] ondisk+write+known_if_redirected+supports_pool_eio e258)", 2024-09-20T22:47:11.332 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:45:37.176065+0000", 2024-09-20T22:47:11.333 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 94.120231645999993, 2024-09-20T22:47:11.333 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.012489495, 2024-09-20T22:47:11.333 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:47:11.333 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:47:11.333 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:47:11.333 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:47:11.333 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:47:11.333 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:47:11.333 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 2321 2024-09-20T22:47:11.333 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.333 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:47:11.333 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.333 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:47:11.333 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.176065+0000", 2024-09-20T22:47:11.334 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:47:11.334 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.334 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.334 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:47:11.334 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.176065+0000", 2024-09-20T22:47:11.334 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:47:11.334 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.334 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.334 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:47:11.334 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.176065+0000", 2024-09-20T22:47:11.334 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.3e-07 2024-09-20T22:47:11.334 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.334 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.334 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:47:11.335 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.176069+0000", 2024-09-20T22:47:11.335 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.8010000000000001e-06 2024-09-20T22:47:11.335 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.335 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.335 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:47:11.335 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.176076+0000", 2024-09-20T22:47:11.335 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.6599999999999998e-06 2024-09-20T22:47:11.335 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.335 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.335 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:47:11.335 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.176086+0000", 2024-09-20T22:47:11.335 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.9720000000000004e-06 2024-09-20T22:47:11.335 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.335 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.336 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:47:11.336 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.176134+0000", 2024-09-20T22:47:11.336 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.7778999999999998e-05 2024-09-20T22:47:11.336 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.336 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.336 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting_for_map not empty", 2024-09-20T22:47:11.336 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.176139+0000", 2024-09-20T22:47:11.336 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.7699999999999998e-06 2024-09-20T22:47:11.336 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.336 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.336 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:47:11.336 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.181825+0000", 2024-09-20T22:47:11.336 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.005685921 2024-09-20T22:47:11.336 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.337 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.337 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:47:11.337 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.181919+0000", 2024-09-20T22:47:11.337 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.3630999999999999e-05 2024-09-20T22:47:11.337 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.337 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.337 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 3,7", 2024-09-20T22:47:11.337 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.182118+0000", 2024-09-20T22:47:11.337 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00019958500000000001 2024-09-20T22:47:11.337 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.337 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.337 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:47:11.337 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.185052+0000", 2024-09-20T22:47:11.337 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0029336620000000001 2024-09-20T22:47:11.338 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.338 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.338 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:47:11.338 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.185756+0000", 2024-09-20T22:47:11.338 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00070376499999999997 2024-09-20T22:47:11.338 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.338 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.338 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:47:11.338 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.188120+0000", 2024-09-20T22:47:11.338 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0023645739999999999 2024-09-20T22:47:11.338 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.338 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.338 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:47:11.338 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.188207+0000", 2024-09-20T22:47:11.339 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.6562000000000002e-05 2024-09-20T22:47:11.339 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.339 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.339 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:47:11.339 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.188554+0000", 2024-09-20T22:47:11.339 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00034718299999999999 2024-09-20T22:47:11.339 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:47:11.340 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:47:11.340 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:47:11.340 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.340 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.340 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:2322 7.8 7:12e9c8db:::smithi08477203-11:head [append 0~0] snapc 55=[55,53,52,51,50,4b,47,3e] ondisk+write+known_if_redirected+supports_pool_eio e258)", 2024-09-20T22:47:11.340 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:45:37.176152+0000", 2024-09-20T22:47:11.340 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 94.120144730000007, 2024-09-20T22:47:11.340 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013033472000000001, 2024-09-20T22:47:11.340 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:47:11.340 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:47:11.340 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:47:11.340 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:47:11.340 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:47:11.341 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:47:11.341 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 2322 2024-09-20T22:47:11.341 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.341 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:47:11.341 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.341 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:47:11.341 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.176152+0000", 2024-09-20T22:47:11.341 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:47:11.341 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.341 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.341 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:47:11.341 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.176152+0000", 2024-09-20T22:47:11.341 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:47:11.341 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.342 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.342 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:47:11.342 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.176152+0000", 2024-09-20T22:47:11.342 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.8899999999999999e-07 2024-09-20T22:47:11.342 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.342 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.342 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:47:11.342 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.176156+0000", 2024-09-20T22:47:11.342 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.2669999999999998e-06 2024-09-20T22:47:11.342 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.342 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.342 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:47:11.342 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.176162+0000", 2024-09-20T22:47:11.343 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.1589999999999998e-06 2024-09-20T22:47:11.343 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.343 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.343 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:47:11.343 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.176167+0000", 2024-09-20T22:47:11.343 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.3040000000000004e-06 2024-09-20T22:47:11.343 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.343 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.343 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:47:11.343 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.176212+0000", 2024-09-20T22:47:11.343 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.5448999999999998e-05 2024-09-20T22:47:11.343 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.343 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.343 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting_for_map not empty", 2024-09-20T22:47:11.344 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.176218+0000", 2024-09-20T22:47:11.344 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.5520000000000004e-06 2024-09-20T22:47:11.344 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.344 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.344 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:47:11.344 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.182721+0000", 2024-09-20T22:47:11.344 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0065028550000000001 2024-09-20T22:47:11.344 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.344 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.344 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:47:11.344 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.182787+0000", 2024-09-20T22:47:11.344 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.5995999999999996e-05 2024-09-20T22:47:11.344 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.345 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.345 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 3,7", 2024-09-20T22:47:11.345 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.183002+0000", 2024-09-20T22:47:11.345 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00021516699999999999 2024-09-20T22:47:11.345 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.345 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.345 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:47:11.345 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.185137+0000", 2024-09-20T22:47:11.345 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0021350570000000001 2024-09-20T22:47:11.345 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.345 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.345 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:47:11.345 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.185875+0000", 2024-09-20T22:47:11.345 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00073794799999999999 2024-09-20T22:47:11.346 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.346 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.346 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:47:11.346 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.188620+0000", 2024-09-20T22:47:11.346 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0027446229999999999 2024-09-20T22:47:11.346 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.346 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.346 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:47:11.346 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.189088+0000", 2024-09-20T22:47:11.346 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00046794499999999999 2024-09-20T22:47:11.346 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.346 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.346 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:47:11.347 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.189185+0000", 2024-09-20T22:47:11.347 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.7560999999999996e-05 2024-09-20T22:47:11.347 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:47:11.347 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:47:11.347 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:47:11.347 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.347 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.347 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24652.0:2318 7.5 e258/137 7:a160b5d4:::smithi08477203-16:head v 258'43, mlcod=239'39)", 2024-09-20T22:47:11.347 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:45:37.192419+0000", 2024-09-20T22:47:11.347 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 94.103877432999994, 2024-09-20T22:47:11.347 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013974723, 2024-09-20T22:47:11.347 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:47:11.347 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:47:11.347 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:47:11.348 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:47:11.348 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.348 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:47:11.348 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.192419+0000", 2024-09-20T22:47:11.348 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:47:11.348 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.348 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.348 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:47:11.348 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.192419+0000", 2024-09-20T22:47:11.348 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:47:11.348 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.348 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.348 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:47:11.349 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.192420+0000", 2024-09-20T22:47:11.349 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.3799999999999997e-07 2024-09-20T22:47:11.349 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.349 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.349 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:47:11.349 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.204088+0000", 2024-09-20T22:47:11.349 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011668883 2024-09-20T22:47:11.349 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.349 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.349 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:47:11.349 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.204104+0000", 2024-09-20T22:47:11.349 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.5437000000000001e-05 2024-09-20T22:47:11.350 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.350 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.350 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:47:11.350 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.204112+0000", 2024-09-20T22:47:11.350 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.2519999999999999e-06 2024-09-20T22:47:11.350 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.350 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.350 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:47:11.350 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.204168+0000", 2024-09-20T22:47:11.350 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.5977999999999998e-05 2024-09-20T22:47:11.350 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.350 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.350 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:47:11.351 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.204188+0000", 2024-09-20T22:47:11.351 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.9400999999999999e-05 2024-09-20T22:47:11.351 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.351 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.351 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:47:11.351 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.206340+0000", 2024-09-20T22:47:11.351 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0021527159999999998 2024-09-20T22:47:11.351 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.351 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.351 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:47:11.351 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.206394+0000", 2024-09-20T22:47:11.351 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.3418000000000003e-05 2024-09-20T22:47:11.352 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:47:11.352 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:47:11.352 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:47:11.352 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.352 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.352 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(osd.6.0:726 7.19 e260/139 7:9eac4560:::smithi08477203-30:4c v 260'159, mlcod=259'156)", 2024-09-20T22:47:11.352 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:45:39.197025+0000", 2024-09-20T22:47:11.352 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 92.099271522999999, 2024-09-20T22:47:11.352 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.018150454, 2024-09-20T22:47:11.352 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:47:11.352 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:47:11.352 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:47:11.352 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:47:11.353 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.353 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:47:11.353 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.197025+0000", 2024-09-20T22:47:11.353 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:47:11.353 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.353 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.353 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:47:11.353 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.197025+0000", 2024-09-20T22:47:11.353 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:47:11.353 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.353 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.353 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:47:11.353 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.197025+0000", 2024-09-20T22:47:11.354 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.7599999999999998e-07 2024-09-20T22:47:11.354 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.354 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.354 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:47:11.354 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.197033+0000", 2024-09-20T22:47:11.354 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.9119999999999998e-06 2024-09-20T22:47:11.354 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.354 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.354 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:47:11.354 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.197042+0000", 2024-09-20T22:47:11.354 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.3349999999999996e-06 2024-09-20T22:47:11.354 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.354 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.355 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:47:11.355 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.197050+0000", 2024-09-20T22:47:11.355 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.7230000000000004e-06 2024-09-20T22:47:11.355 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.355 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.355 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:47:11.355 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.200756+0000", 2024-09-20T22:47:11.355 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0037059749999999998 2024-09-20T22:47:11.355 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.355 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.355 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:47:11.355 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.200778+0000", 2024-09-20T22:47:11.356 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.2140000000000001e-05 2024-09-20T22:47:11.356 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.356 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.356 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:47:11.356 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.215131+0000", 2024-09-20T22:47:11.356 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014353054000000001 2024-09-20T22:47:11.356 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.356 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.356 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:47:11.356 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.215175+0000", 2024-09-20T22:47:11.356 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.4039000000000003e-05 2024-09-20T22:47:11.356 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:47:11.357 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:47:11.357 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:47:11.357 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.357 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.357 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(osd.7.0:152 7.5 e269/260 7:a160b5d4:::smithi08477203-16:55 v 269'55, mlcod=263'51)", 2024-09-20T22:47:11.357 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:45:48.173893+0000", 2024-09-20T22:47:11.357 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 83.122403782000006, 2024-09-20T22:47:11.357 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014936935, 2024-09-20T22:47:11.357 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:47:11.357 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:47:11.357 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:47:11.357 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:47:11.357 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.357 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:47:11.358 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.173893+0000", 2024-09-20T22:47:11.358 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:47:11.358 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.358 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.358 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:47:11.358 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.173893+0000", 2024-09-20T22:47:11.358 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:47:11.358 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.358 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.358 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:47:11.358 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.173893+0000", 2024-09-20T22:47:11.358 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.8800000000000002e-07 2024-09-20T22:47:11.358 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.359 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.359 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:47:11.359 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.173916+0000", 2024-09-20T22:47:11.359 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.2889999999999999e-05 2024-09-20T22:47:11.359 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.359 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.359 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:47:11.359 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.173924+0000", 2024-09-20T22:47:11.359 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.4420000000000001e-06 2024-09-20T22:47:11.359 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.359 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.359 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:47:11.359 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.173932+0000", 2024-09-20T22:47:11.359 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.5299999999999996e-06 2024-09-20T22:47:11.359 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.360 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.360 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:47:11.360 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.174011+0000", 2024-09-20T22:47:11.360 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.9342e-05 2024-09-20T22:47:11.360 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.360 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.360 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:47:11.360 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.174047+0000", 2024-09-20T22:47:11.360 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.5237999999999997e-05 2024-09-20T22:47:11.360 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.360 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.360 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:47:11.360 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.188782+0000", 2024-09-20T22:47:11.361 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014735599 2024-09-20T22:47:11.361 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.361 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.361 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:47:11.361 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.188830+0000", 2024-09-20T22:47:11.361 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.7206000000000002e-05 2024-09-20T22:47:11.361 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:47:11.361 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:47:11.361 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:47:11.361 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.361 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.361 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(osd.7.0:153 7.17 e269/139 7:ea720b85:::smithi08477203-26:55 v 269'225, mlcod=263'221)", 2024-09-20T22:47:11.362 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:45:48.173991+0000", 2024-09-20T22:47:11.362 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 83.122305300999997, 2024-09-20T22:47:11.362 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013064868, 2024-09-20T22:47:11.362 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:47:11.362 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:47:11.362 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:47:11.362 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:47:11.362 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.362 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:47:11.362 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.173991+0000", 2024-09-20T22:47:11.362 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:47:11.362 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.362 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.363 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:47:11.363 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.173991+0000", 2024-09-20T22:47:11.363 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:47:11.363 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.363 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.363 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:47:11.363 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.173991+0000", 2024-09-20T22:47:11.363 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.9999999999999999e-07 2024-09-20T22:47:11.363 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.363 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.363 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:47:11.363 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.173996+0000", 2024-09-20T22:47:11.363 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.3080000000000001e-06 2024-09-20T22:47:11.363 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.363 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.364 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:47:11.364 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.174001+0000", 2024-09-20T22:47:11.364 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.203e-06 2024-09-20T22:47:11.364 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.364 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.364 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:47:11.364 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.174005+0000", 2024-09-20T22:47:11.364 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.4880000000000001e-06 2024-09-20T22:47:11.364 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.364 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.364 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:47:11.364 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.175600+0000", 2024-09-20T22:47:11.364 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015942459999999999 2024-09-20T22:47:11.364 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.365 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.365 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:47:11.365 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.175631+0000", 2024-09-20T22:47:11.365 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.1816000000000002e-05 2024-09-20T22:47:11.365 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.365 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.365 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:47:11.365 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.187003+0000", 2024-09-20T22:47:11.365 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011372006 2024-09-20T22:47:11.365 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.365 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.365 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:47:11.365 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.187056+0000", 2024-09-20T22:47:11.366 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.2500999999999998e-05 2024-09-20T22:47:11.366 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:47:11.366 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:47:11.366 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:47:11.366 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.366 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.366 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(osd.6.0:767 7.12 e269/263 7:4fa4b2b3:::smithi08477203-21:53 v 269'262, mlcod=265'259)", 2024-09-20T22:47:11.366 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:45:48.175130+0000", 2024-09-20T22:47:11.366 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 83.121166111999997, 2024-09-20T22:47:11.366 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013842727000000001, 2024-09-20T22:47:11.366 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:47:11.366 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:47:11.366 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:47:11.367 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:47:11.367 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.367 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:47:11.367 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.175130+0000", 2024-09-20T22:47:11.367 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:47:11.367 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.367 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.367 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:47:11.367 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.175130+0000", 2024-09-20T22:47:11.367 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:47:11.367 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.367 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.367 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:47:11.367 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.175130+0000", 2024-09-20T22:47:11.368 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.98e-07 2024-09-20T22:47:11.368 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.368 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.368 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:47:11.368 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.175139+0000", 2024-09-20T22:47:11.368 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.5909999999999996e-06 2024-09-20T22:47:11.368 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.368 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.368 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:47:11.368 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.175144+0000", 2024-09-20T22:47:11.368 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.361e-06 2024-09-20T22:47:11.368 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.368 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.368 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:47:11.368 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.175156+0000", 2024-09-20T22:47:11.369 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1236999999999999e-05 2024-09-20T22:47:11.369 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.369 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.369 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:47:11.369 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.176359+0000", 2024-09-20T22:47:11.369 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0012036060000000001 2024-09-20T22:47:11.369 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.369 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.369 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:47:11.369 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.176380+0000", 2024-09-20T22:47:11.369 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.0497999999999999e-05 2024-09-20T22:47:11.369 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.369 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.369 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:47:11.369 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.188936+0000", 2024-09-20T22:47:11.370 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.012556286999999999 2024-09-20T22:47:11.370 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.370 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.370 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:47:11.370 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.188973+0000", 2024-09-20T22:47:11.370 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.6949000000000001e-05 2024-09-20T22:47:11.370 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:47:11.370 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:47:11.370 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:47:11.370 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.370 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.370 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24652.0:2853 7.10 e303/139 7:0f524219:::smithi08477203-50:head v 303'85, mlcod=301'82)", 2024-09-20T22:47:11.370 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:46:23.293837+0000", 2024-09-20T22:47:11.370 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 48.002459397999999, 2024-09-20T22:47:11.371 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013368555000000001, 2024-09-20T22:47:11.371 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:47:11.371 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:47:11.371 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:47:11.371 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:47:11.371 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.371 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:47:11.371 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.293837+0000", 2024-09-20T22:47:11.371 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:47:11.371 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.371 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.371 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:47:11.371 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.293837+0000", 2024-09-20T22:47:11.372 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:47:11.372 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.372 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.372 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:47:11.372 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.293839+0000", 2024-09-20T22:47:11.372 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.7239999999999999e-06 2024-09-20T22:47:11.372 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.372 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.372 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:47:11.372 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.297226+0000", 2024-09-20T22:47:11.372 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.003387583 2024-09-20T22:47:11.372 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.372 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.373 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:47:11.373 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.297253+0000", 2024-09-20T22:47:11.373 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.6849e-05 2024-09-20T22:47:11.373 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.373 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.373 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:47:11.373 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.297266+0000", 2024-09-20T22:47:11.373 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2481e-05 2024-09-20T22:47:11.373 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.373 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.373 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:47:11.373 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.297362+0000", 2024-09-20T22:47:11.373 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.6347000000000004e-05 2024-09-20T22:47:11.374 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.374 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.374 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:47:11.374 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.297394+0000", 2024-09-20T22:47:11.374 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.1888999999999999e-05 2024-09-20T22:47:11.374 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.374 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.374 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:47:11.374 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.307147+0000", 2024-09-20T22:47:11.374 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0097533719999999997 2024-09-20T22:47:11.374 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.374 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.374 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:47:11.374 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.307206+0000", 2024-09-20T22:47:11.375 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.8310000000000002e-05 2024-09-20T22:47:11.375 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:47:11.375 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:47:11.375 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:47:11.375 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.375 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.375 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:3030 7.e 7:71b5c915:::smithi08477203-6:head [copy-from ver 216] snapc 72=[72,71,70,6e,6c,6b,6a,68] ondisk+write+known_if_redirected+supports_pool_eio e317)", 2024-09-20T22:47:11.375 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:46:37.293029+0000", 2024-09-20T22:47:11.375 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 34.003267039999997, 2024-09-20T22:47:11.375 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.019929916999999998, 2024-09-20T22:47:11.375 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:47:11.375 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:47:11.375 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:47:11.376 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:47:11.376 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:47:11.376 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:47:11.376 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 3030 2024-09-20T22:47:11.376 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.376 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:47:11.376 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.376 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:47:11.376 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.293029+0000", 2024-09-20T22:47:11.376 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:47:11.376 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.376 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.376 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:47:11.377 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.293029+0000", 2024-09-20T22:47:11.377 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:47:11.377 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.377 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.377 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:47:11.377 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.293031+0000", 2024-09-20T22:47:11.377 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.4190000000000001e-06 2024-09-20T22:47:11.377 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.377 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.377 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:47:11.377 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.293040+0000", 2024-09-20T22:47:11.377 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.6039999999999999e-06 2024-09-20T22:47:11.377 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.378 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.378 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:47:11.378 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.293054+0000", 2024-09-20T22:47:11.378 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.4014e-05 2024-09-20T22:47:11.378 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.378 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.378 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:47:11.378 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.293062+0000", 2024-09-20T22:47:11.378 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.8820000000000001e-06 2024-09-20T22:47:11.378 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.378 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.378 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:47:11.378 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.293121+0000", 2024-09-20T22:47:11.379 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.8977999999999997e-05 2024-09-20T22:47:11.379 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.379 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.379 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op must wait for map", 2024-09-20T22:47:11.379 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.293125+0000", 2024-09-20T22:47:11.379 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.2089999999999999e-06 2024-09-20T22:47:11.379 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.379 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.379 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:47:11.379 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.298805+0000", 2024-09-20T22:47:11.380 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0056790870000000002 2024-09-20T22:47:11.380 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.380 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.380 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:47:11.380 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.298886+0000", 2024-09-20T22:47:11.380 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.1363e-05 2024-09-20T22:47:11.380 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.380 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.380 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 3,5", 2024-09-20T22:47:11.380 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.305883+0000", 2024-09-20T22:47:11.380 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0069966159999999998 2024-09-20T22:47:11.380 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.380 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.381 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:47:11.381 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.312556+0000", 2024-09-20T22:47:11.381 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0066737009999999998 2024-09-20T22:47:11.381 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.381 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.381 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:47:11.381 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.312587+0000", 2024-09-20T22:47:11.381 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.0286000000000001e-05 2024-09-20T22:47:11.381 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.381 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.381 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:47:11.381 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.312890+0000", 2024-09-20T22:47:11.381 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00030374800000000001 2024-09-20T22:47:11.382 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.382 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.382 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:47:11.382 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.312932+0000", 2024-09-20T22:47:11.382 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.1621e-05 2024-09-20T22:47:11.382 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.382 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.382 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:47:11.382 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.312959+0000", 2024-09-20T22:47:11.382 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.7389000000000002e-05 2024-09-20T22:47:11.382 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:47:11.382 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:47:11.382 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:47:11.383 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.383 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.383 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:3031 7.e 7:71b5c915:::smithi08477203-6:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e317)", 2024-09-20T22:47:11.383 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:46:37.293082+0000", 2024-09-20T22:47:11.383 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 34.003214661999998, 2024-09-20T22:47:11.383 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.016134757, 2024-09-20T22:47:11.383 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:47:11.383 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:47:11.383 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-20T22:47:11.383 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:47:11.383 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:47:11.383 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:47:11.383 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 3031 2024-09-20T22:47:11.383 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.384 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:47:11.384 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.384 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:47:11.384 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.293082+0000", 2024-09-20T22:47:11.384 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:47:11.384 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.384 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.384 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:47:11.384 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.293082+0000", 2024-09-20T22:47:11.385 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:47:11.385 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.385 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.385 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:47:11.385 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.293082+0000", 2024-09-20T22:47:11.385 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.1799999999999995e-07 2024-09-20T22:47:11.385 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.385 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.385 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:47:11.385 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.293085+0000", 2024-09-20T22:47:11.385 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.937e-06 2024-09-20T22:47:11.385 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.385 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.385 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:47:11.386 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.293093+0000", 2024-09-20T22:47:11.386 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.554e-06 2024-09-20T22:47:11.386 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.386 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.386 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:47:11.386 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.293098+0000", 2024-09-20T22:47:11.386 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.643e-06 2024-09-20T22:47:11.386 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.386 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.386 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:47:11.386 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.293164+0000", 2024-09-20T22:47:11.386 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.5806e-05 2024-09-20T22:47:11.386 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.387 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.387 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting_for_map not empty", 2024-09-20T22:47:11.387 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.293168+0000", 2024-09-20T22:47:11.387 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.2459999999999999e-06 2024-09-20T22:47:11.387 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.387 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.387 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:47:11.387 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.299033+0000", 2024-09-20T22:47:11.387 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0058656699999999999 2024-09-20T22:47:11.387 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.387 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.387 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for blocked object", 2024-09-20T22:47:11.387 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.299069+0000", 2024-09-20T22:47:11.388 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.5392999999999998e-05 2024-09-20T22:47:11.388 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.388 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.388 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:47:11.388 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.309088+0000", 2024-09-20T22:47:11.388 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010019287999999999 2024-09-20T22:47:11.388 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.388 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.388 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:47:11.388 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.309150+0000", 2024-09-20T22:47:11.388 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.2008000000000007e-05 2024-09-20T22:47:11.388 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.388 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.388 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:47:11.389 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.309216+0000", 2024-09-20T22:47:11.389 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.6594000000000006e-05 2024-09-20T22:47:11.389 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:47:11.389 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:47:11.389 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:47:11.389 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.389 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.389 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24652.0:3134 7.2 e328/169 7:45b54b6b:::smithi08477203-2:head v 328'122, mlcod=276'118)", 2024-09-20T22:47:11.389 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:46:48.229313+0000", 2024-09-20T22:47:11.389 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 23.066983722, 2024-09-20T22:47:11.389 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.01241282, 2024-09-20T22:47:11.389 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:47:11.389 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:47:11.389 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:47:11.390 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:47:11.390 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.390 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:47:11.390 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.229313+0000", 2024-09-20T22:47:11.390 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:47:11.390 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.390 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.390 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:47:11.390 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.229313+0000", 2024-09-20T22:47:11.390 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:47:11.390 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.390 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.390 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:47:11.391 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.229313+0000", 2024-09-20T22:47:11.391 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.0499999999999999e-07 2024-09-20T22:47:11.391 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.391 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.391 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:47:11.391 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.236508+0000", 2024-09-20T22:47:11.391 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0071947310000000002 2024-09-20T22:47:11.391 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.391 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.391 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:47:11.391 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.236528+0000", 2024-09-20T22:47:11.391 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.9919999999999999e-05 2024-09-20T22:47:11.391 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.391 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.392 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:47:11.392 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.236539+0000", 2024-09-20T22:47:11.392 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1327e-05 2024-09-20T22:47:11.392 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.392 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.392 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:47:11.392 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.236610+0000", 2024-09-20T22:47:11.392 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.1433999999999994e-05 2024-09-20T22:47:11.392 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.392 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.392 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:47:11.392 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.236637+0000", 2024-09-20T22:47:11.392 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.6648999999999998e-05 2024-09-20T22:47:11.392 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.392 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.393 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:47:11.393 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.241650+0000", 2024-09-20T22:47:11.393 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.005013053 2024-09-20T22:47:11.393 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.393 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.393 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:47:11.393 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.241725+0000", 2024-09-20T22:47:11.393 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.5401000000000005e-05 2024-09-20T22:47:11.393 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:47:11.393 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:47:11.393 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:47:11.393 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.393 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.394 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:3290 7.8 7:12e9c8db:::smithi08477203-11:head [copy-from ver 675] snapc 7f=[7f,7e,7d,7c,7b,7a,79,78,74,73,6b] ondisk+write+known_if_redirected+supports_pool_eio e340)", 2024-09-20T22:47:11.394 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:47:00.706846+0000", 2024-09-20T22:47:11.394 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 10.589450803, 2024-09-20T22:47:11.394 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.018408746, 2024-09-20T22:47:11.394 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:47:11.394 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:47:11.394 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:47:11.394 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:47:11.394 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:47:11.394 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:47:11.394 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 3290 2024-09-20T22:47:11.394 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.394 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:47:11.394 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.395 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:47:11.395 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.706846+0000", 2024-09-20T22:47:11.395 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:47:11.395 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.395 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.395 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:47:11.395 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.706846+0000", 2024-09-20T22:47:11.395 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:47:11.395 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.395 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.395 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:47:11.395 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.706848+0000", 2024-09-20T22:47:11.395 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.2340000000000001e-06 2024-09-20T22:47:11.395 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.396 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.396 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:47:11.396 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.706859+0000", 2024-09-20T22:47:11.396 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1001e-05 2024-09-20T22:47:11.396 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.396 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.396 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:47:11.396 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.706879+0000", 2024-09-20T22:47:11.396 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.0556000000000001e-05 2024-09-20T22:47:11.396 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.396 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.396 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:47:11.396 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.706890+0000", 2024-09-20T22:47:11.396 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0328e-05 2024-09-20T22:47:11.397 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.397 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.397 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:47:11.397 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.706973+0000", 2024-09-20T22:47:11.397 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.3574999999999999e-05 2024-09-20T22:47:11.397 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.397 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.397 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:47:11.397 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.707042+0000", 2024-09-20T22:47:11.397 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.8987999999999997e-05 2024-09-20T22:47:11.397 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.397 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.397 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 3,7", 2024-09-20T22:47:11.397 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.717491+0000", 2024-09-20T22:47:11.398 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010448618 2024-09-20T22:47:11.398 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.398 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.398 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:47:11.398 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.722875+0000", 2024-09-20T22:47:11.398 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0053846190000000002 2024-09-20T22:47:11.398 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.398 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.398 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:47:11.398 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.724841+0000", 2024-09-20T22:47:11.398 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0019651579999999998 2024-09-20T22:47:11.398 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.398 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.399 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:47:11.399 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.725168+0000", 2024-09-20T22:47:11.399 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00032734499999999999 2024-09-20T22:47:11.399 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.399 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.399 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:47:11.399 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.725218+0000", 2024-09-20T22:47:11.399 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.0123000000000001e-05 2024-09-20T22:47:11.399 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.399 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.399 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:47:11.399 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.725254+0000", 2024-09-20T22:47:11.399 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.6201000000000001e-05 2024-09-20T22:47:11.399 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:47:11.400 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:47:11.400 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:47:11.400 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.400 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.400 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:3291 7.8 7:12e9c8db:::smithi08477203-11:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e340)", 2024-09-20T22:47:11.400 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:47:00.706927+0000", 2024-09-20T22:47:11.400 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 10.589369164000001, 2024-09-20T22:47:11.400 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014159255000000001, 2024-09-20T22:47:11.400 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:47:11.400 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:47:11.400 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-20T22:47:11.400 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:47:11.400 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:47:11.400 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:47:11.401 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 3291 2024-09-20T22:47:11.401 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.401 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:47:11.401 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.401 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:47:11.401 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.706927+0000", 2024-09-20T22:47:11.401 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:47:11.401 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.401 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.401 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:47:11.401 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.706927+0000", 2024-09-20T22:47:11.401 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:47:11.401 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.401 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.402 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:47:11.402 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.706928+0000", 2024-09-20T22:47:11.402 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.4099999999999999e-07 2024-09-20T22:47:11.402 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.402 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.402 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:47:11.402 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.706941+0000", 2024-09-20T22:47:11.402 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3335e-05 2024-09-20T22:47:11.402 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.402 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.402 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:47:11.402 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.706948+0000", 2024-09-20T22:47:11.402 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.3350000000000001e-06 2024-09-20T22:47:11.402 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.403 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.403 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:47:11.403 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.706954+0000", 2024-09-20T22:47:11.403 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.9190000000000004e-06 2024-09-20T22:47:11.403 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.403 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.403 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:47:11.403 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.707166+0000", 2024-09-20T22:47:11.403 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00021159 2024-09-20T22:47:11.403 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.403 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.403 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for blocked object", 2024-09-20T22:47:11.403 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.707217+0000", 2024-09-20T22:47:11.403 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.1167e-05 2024-09-20T22:47:11.404 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.404 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.404 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:47:11.404 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.720939+0000", 2024-09-20T22:47:11.404 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013721458000000001 2024-09-20T22:47:11.404 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.404 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.404 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:47:11.404 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.721010+0000", 2024-09-20T22:47:11.404 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.1570999999999996e-05 2024-09-20T22:47:11.404 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.404 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.404 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:47:11.404 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.721086+0000", 2024-09-20T22:47:11.405 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.6339000000000002e-05 2024-09-20T22:47:11.405 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:47:11.405 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:47:11.405 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:47:11.405 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.405 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.405 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(osd.5.0:167 7.2 e347/169 7:45b54b6b:::smithi08477203-2:81 v 347'140, mlcod=345'136)", 2024-09-20T22:47:11.405 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:47:08.204691+0000", 2024-09-20T22:47:11.405 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 3.0916050629999998, 2024-09-20T22:47:11.405 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.01335918, 2024-09-20T22:47:11.405 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:47:11.405 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:47:11.406 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:47:11.406 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:47:11.406 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.406 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:47:11.406 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:08.204691+0000", 2024-09-20T22:47:11.406 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:47:11.406 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.406 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.406 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:47:11.406 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:08.204691+0000", 2024-09-20T22:47:11.406 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:47:11.406 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.406 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.407 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:47:11.407 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:08.204692+0000", 2024-09-20T22:47:11.407 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.5400000000000001e-07 2024-09-20T22:47:11.407 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.407 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.407 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:47:11.407 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:08.204697+0000", 2024-09-20T22:47:11.407 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.614e-06 2024-09-20T22:47:11.407 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.407 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.407 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:47:11.407 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:08.204705+0000", 2024-09-20T22:47:11.407 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.8259999999999993e-06 2024-09-20T22:47:11.407 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.407 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.408 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:47:11.408 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:08.204715+0000", 2024-09-20T22:47:11.408 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.8279999999999998e-06 2024-09-20T22:47:11.408 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.408 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.408 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:47:11.408 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:08.204801+0000", 2024-09-20T22:47:11.408 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.6263999999999999e-05 2024-09-20T22:47:11.408 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.408 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.408 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:47:11.408 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:08.204823+0000", 2024-09-20T22:47:11.408 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.1971999999999999e-05 2024-09-20T22:47:11.408 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.409 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.409 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:47:11.409 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:08.217961+0000", 2024-09-20T22:47:11.409 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013137618 2024-09-20T22:47:11.409 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:11.409 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:11.409 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:47:11.409 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:08.218050+0000", 2024-09-20T22:47:11.409 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.9504000000000002e-05 2024-09-20T22:47:11.409 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:47:11.409 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:47:11.409 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:47:11.409 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:47:11.409 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:47:11.409 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-20T22:47:11.410 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:47:11.429 INFO:tasks.rados.rados.0.smithi084.stdout:1317: finishing rollback tid 0 to smithi08477203-8 2024-09-20T22:47:11.429 INFO:tasks.rados.rados.0.smithi084.stdout:1317: finishing rollback tid 1 to smithi08477203-8 2024-09-20T22:47:11.429 INFO:tasks.rados.rados.0.smithi084.stdout:1317: finishing rollback tid 2 to smithi08477203-8 2024-09-20T22:47:11.429 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 8 v 400 (ObjNum 438 snap 115 seq_num 438) dirty exists 2024-09-20T22:47:11.429 INFO:tasks.rados.rados.0.smithi084.stdout:1314: expect (ObjNum 442 snap 116 seq_num 442) 2024-09-20T22:47:11.431 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:11.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:11.432 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:47:11.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:11 smithi154 ceph-mon[33264]: pgmap v426: 168 pgs: 5 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 157 active+clean; 232 MiB data, 745 MiB used, 536 GiB / 536 GiB avail; 9.7 MiB/s rd, 3.2 MiB/s wr, 17 op/s 2024-09-20T22:47:11.533 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:47:11.542 INFO:tasks.rados.rados.0.smithi084.stdout:1315: expect (ObjNum 455 snap 125 seq_num 455) 2024-09-20T22:47:11.553 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:11.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:11.554 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:47:11.614 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:11 smithi084 ceph-mon[40024]: pgmap v426: 168 pgs: 5 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 157 active+clean; 232 MiB data, 745 MiB used, 536 GiB / 536 GiB avail; 9.7 MiB/s rd, 3.2 MiB/s wr, 17 op/s 2024-09-20T22:47:11.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:11 smithi084 ceph-mon[29774]: pgmap v426: 168 pgs: 5 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 157 active+clean; 232 MiB data, 745 MiB used, 536 GiB / 536 GiB avail; 9.7 MiB/s rd, 3.2 MiB/s wr, 17 op/s 2024-09-20T22:47:11.655 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:47:11.676 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:11.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:11.677 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:47:11.778 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:47:11.803 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:11.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:11.804 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:47:11.905 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:47:11.926 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:11.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:11.927 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:47:12.028 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:47:12.053 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:12.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:12.054 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:47:12.157 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:47:12.178 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:12.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:12.179 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:47:12.250 INFO:tasks.rados.rados.0.smithi084.stdout:1314: done (4 left) 2024-09-20T22:47:12.250 INFO:tasks.rados.rados.0.smithi084.stdout:1315: done (3 left) 2024-09-20T22:47:12.250 INFO:tasks.rados.rados.0.smithi084.stdout:1316: done (2 left) 2024-09-20T22:47:12.250 INFO:tasks.rados.rados.0.smithi084.stdout:1317: done (1 left) 2024-09-20T22:47:12.250 INFO:tasks.rados.rados.0.smithi084.stdout:1318: done (0 left) 2024-09-20T22:47:12.250 INFO:tasks.rados.rados.0.smithi084.stdout:1319: write (excl) oid 47 current snap is 133 2024-09-20T22:47:12.251 INFO:tasks.rados.rados.0.smithi084.stdout:1319: seq_num 474 ranges {766518=481912,1837406=542493,2933675=527287,3876584=89934} 2024-09-20T22:47:12.251 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is unset 2024-09-20T22:47:12.256 INFO:tasks.rados.rados.0.smithi084.stdout:1319: writing smithi08477203-47 from 766518 to 1248430 tid 1 2024-09-20T22:47:12.263 INFO:tasks.rados.rados.0.smithi084.stdout:1319: writing smithi08477203-47 from 1837406 to 2379899 tid 2 2024-09-20T22:47:12.269 INFO:tasks.rados.rados.0.smithi084.stdout:1319: writing smithi08477203-47 from 2933675 to 3460962 tid 3 2024-09-20T22:47:12.273 INFO:tasks.rados.rados.0.smithi084.stdout:1319: writing smithi08477203-47 from 3876584 to 3966518 tid 4 2024-09-20T22:47:12.273 INFO:tasks.rados.rados.0.smithi084.stdout:1320: delete oid 38 current snap is 133 2024-09-20T22:47:12.273 INFO:tasks.rados.rados.0.smithi084.stdout:1319: finishing write tid 1 to smithi08477203-47 2024-09-20T22:47:12.273 INFO:tasks.rados.rados.0.smithi084.stdout:1319: finishing write tid 2 to smithi08477203-47 2024-09-20T22:47:12.274 INFO:tasks.rados.rados.0.smithi084.stdout:1319: finishing write tid 3 to smithi08477203-47 2024-09-20T22:47:12.275 INFO:tasks.rados.rados.0.smithi084.stdout:1319: finishing write tid 4 to smithi08477203-47 2024-09-20T22:47:12.275 INFO:tasks.rados.rados.0.smithi084.stdout:1319: finishing write tid 5 to smithi08477203-47 2024-09-20T22:47:12.275 INFO:tasks.rados.rados.0.smithi084.stdout:1319: finishing write tid 6 to smithi08477203-47 2024-09-20T22:47:12.275 INFO:tasks.rados.rados.0.smithi084.stdout:1319: oid 47 updating version 0 to 399 2024-09-20T22:47:12.275 INFO:tasks.rados.rados.0.smithi084.stdout:1319: oid 47 updating version 399 to 403 2024-09-20T22:47:12.275 INFO:tasks.rados.rados.0.smithi084.stdout:1319: oid 47 version 403 is already newer than 400 2024-09-20T22:47:12.275 INFO:tasks.rados.rados.0.smithi084.stdout:1319: oid 47 version 403 is already newer than 401 2024-09-20T22:47:12.276 INFO:tasks.rados.rados.0.smithi084.stdout:1319: oid 47 version 403 is already newer than 402 2024-09-20T22:47:12.276 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 47 v 403 (ObjNum 474 snap 133 seq_num 474) dirty exists 2024-09-20T22:47:12.276 INFO:tasks.rados.rados.0.smithi084.stdout:1319: left oid 47 (ObjNum 474 snap 133 seq_num 474) 2024-09-20T22:47:12.277 INFO:tasks.rados.rados.0.smithi084.stdout:1319: done (1 left) 2024-09-20T22:47:12.277 INFO:tasks.rados.rados.0.smithi084.stdout:1320: done (0 left) 2024-09-20T22:47:12.277 INFO:tasks.rados.rados.0.smithi084.stdout:1321: rollback oid 12 current snap is 133 2024-09-20T22:47:12.277 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 12 to 114 2024-09-20T22:47:12.277 INFO:tasks.rados.rados.0.smithi084.stdout:1322: write (excl) oid 39 current snap is 133 2024-09-20T22:47:12.277 INFO:tasks.rados.rados.0.smithi084.stdout:1322: seq_num 476 ranges {400132=632410,1438017=546162} 2024-09-20T22:47:12.280 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:47:12.282 INFO:tasks.rados.rados.0.smithi084.stdout:1322: writing smithi08477203-39 from 400132 to 1032542 tid 1 2024-09-20T22:47:12.288 INFO:tasks.rados.rados.0.smithi084.stdout:1322: writing smithi08477203-39 from 1438017 to 1984179 tid 2 2024-09-20T22:47:12.288 INFO:tasks.rados.rados.0.smithi084.stdout:1323: read oid 13 snap 127 2024-09-20T22:47:12.288 INFO:tasks.rados.rados.0.smithi084.stdout:1323: expect (ObjNum 411 snap 105 seq_num 411) 2024-09-20T22:47:12.289 INFO:tasks.rados.rados.0.smithi084.stdout:1324: write (excl) oid 40 current snap is 133 2024-09-20T22:47:12.289 INFO:tasks.rados.rados.0.smithi084.stdout:1321: finishing rollback tid 0 to smithi08477203-12 2024-09-20T22:47:12.289 INFO:tasks.rados.rados.0.smithi084.stdout:1324: seq_num 477 ranges {416939=707659,1896499=120440} 2024-09-20T22:47:12.294 INFO:tasks.rados.rados.0.smithi084.stdout:1324: writing smithi08477203-40 from 416939 to 1124598 tid 1 2024-09-20T22:47:12.300 INFO:tasks.rados.rados.0.smithi084.stdout:1324: writing smithi08477203-40 from 1896499 to 2016939 tid 2 2024-09-20T22:47:12.300 INFO:tasks.rados.rados.0.smithi084.stdout:1325: snap_create 2024-09-20T22:47:12.300 INFO:tasks.rados.rados.0.smithi084.stdout:1321: finishing rollback tid 1 to smithi08477203-12 2024-09-20T22:47:12.300 INFO:tasks.rados.rados.0.smithi084.stdout:1321: finishing rollback tid 2 to smithi08477203-12 2024-09-20T22:47:12.300 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 12 v 282 (ObjNum 434 snap 112 seq_num 434) dirty exists 2024-09-20T22:47:12.300 INFO:tasks.rados.rados.0.smithi084.stdout:1322: finishing write tid 1 to smithi08477203-39 2024-09-20T22:47:12.300 INFO:tasks.rados.rados.0.smithi084.stdout:1322: finishing write tid 2 to smithi08477203-39 2024-09-20T22:47:12.300 INFO:tasks.rados.rados.0.smithi084.stdout:1322: finishing write tid 3 to smithi08477203-39 2024-09-20T22:47:12.301 INFO:tasks.rados.rados.0.smithi084.stdout:1322: finishing write tid 4 to smithi08477203-39 2024-09-20T22:47:12.301 INFO:tasks.rados.rados.0.smithi084.stdout:1322: oid 39 updating version 0 to 413 2024-09-20T22:47:12.301 INFO:tasks.rados.rados.0.smithi084.stdout:1322: oid 39 version 413 is already newer than 412 2024-09-20T22:47:12.301 INFO:tasks.rados.rados.0.smithi084.stdout:1322: oid 39 version 413 is already newer than 411 2024-09-20T22:47:12.301 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 39 v 413 (ObjNum 476 snap 133 seq_num 476) dirty exists 2024-09-20T22:47:12.301 INFO:tasks.rados.rados.0.smithi084.stdout:1322: left oid 39 (ObjNum 476 snap 133 seq_num 476) 2024-09-20T22:47:12.301 INFO:tasks.rados.rados.0.smithi084.stdout:1323: expect (ObjNum 411 snap 105 seq_num 411) 2024-09-20T22:47:12.306 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:12.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:12.307 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:47:12.328 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-20T22:47:12.408 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:47:12.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:12.430 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:12.430 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:47:12.456 INFO:tasks.rados.rados.0.smithi084.stdout:1324: finishing write tid 1 to smithi08477203-40 2024-09-20T22:47:12.456 INFO:tasks.rados.rados.0.smithi084.stdout:1324: finishing write tid 2 to smithi08477203-40 2024-09-20T22:47:12.456 INFO:tasks.rados.rados.0.smithi084.stdout:1324: finishing write tid 3 to smithi08477203-40 2024-09-20T22:47:12.456 INFO:tasks.rados.rados.0.smithi084.stdout:1324: finishing write tid 4 to smithi08477203-40 2024-09-20T22:47:12.456 INFO:tasks.rados.rados.0.smithi084.stdout:1324: oid 40 updating version 0 to 389 2024-09-20T22:47:12.456 INFO:tasks.rados.rados.0.smithi084.stdout:1324: oid 40 updating version 389 to 391 2024-09-20T22:47:12.456 INFO:tasks.rados.rados.0.smithi084.stdout:1324: oid 40 version 391 is already newer than 390 2024-09-20T22:47:12.457 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 40 v 391 (ObjNum 477 snap 133 seq_num 477) dirty exists 2024-09-20T22:47:12.457 INFO:tasks.rados.rados.0.smithi084.stdout:1324: left oid 40 (ObjNum 477 snap 133 seq_num 477) 2024-09-20T22:47:12.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:12 smithi154 ceph-mon[33264]: osdmap e351: 8 total, 8 up, 6 in 2024-09-20T22:47:12.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:12 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/1370019515' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:47:12.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:12 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:47:12.532 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:47:12.557 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:12.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:12.558 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:47:12.614 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:12 smithi084 ceph-mon[40024]: osdmap e351: 8 total, 8 up, 6 in 2024-09-20T22:47:12.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:12 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/1370019515' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:47:12.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:12 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:47:12.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:12 smithi084 ceph-mon[29774]: osdmap e351: 8 total, 8 up, 6 in 2024-09-20T22:47:12.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:12 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/1370019515' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:47:12.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:12 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:47:12.658 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:47:12.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:12.681 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:12.681 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:47:12.782 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:47:12.803 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:12.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:12.804 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:47:12.905 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:47:12.931 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:12.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:12.932 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:47:13.033 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:47:13.059 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:13.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:13.060 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:47:13.093 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:47:13.160 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:47:13.182 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:13.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:13.183 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:47:13.187 INFO:tasks.rados.rados.0.smithi084.stdout:1321: done (4 left) 2024-09-20T22:47:13.187 INFO:tasks.rados.rados.0.smithi084.stdout:1322: done (3 left) 2024-09-20T22:47:13.187 INFO:tasks.rados.rados.0.smithi084.stdout:1323: done (2 left) 2024-09-20T22:47:13.188 INFO:tasks.rados.rados.0.smithi084.stdout:1324: done (1 left) 2024-09-20T22:47:13.188 INFO:tasks.rados.rados.0.smithi084.stdout:1325: done (0 left) 2024-09-20T22:47:13.188 INFO:tasks.rados.rados.0.smithi084.stdout:1326: rollback oid 10 current snap is 134 2024-09-20T22:47:13.188 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 10 to 133 2024-09-20T22:47:13.188 INFO:tasks.rados.rados.0.smithi084.stdout:1327: snap_create 2024-09-20T22:47:13.191 INFO:tasks.rados.rados.0.smithi084.stdout:1326: finishing rollback tid 0 to smithi08477203-10 2024-09-20T22:47:13.194 INFO:tasks.rados.rados.0.smithi084.stdout:1326: finishing rollback tid 1 to smithi08477203-10 2024-09-20T22:47:13.194 INFO:tasks.rados.rados.0.smithi084.stdout:1326: finishing rollback tid 2 to smithi08477203-10 2024-09-20T22:47:13.194 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 10 v 416 (ObjNum 419 snap 105 seq_num 419) dirty exists 2024-09-20T22:47:13.284 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:47:13.309 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:13.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:13.310 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:47:13.348 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:13 smithi084 ceph-mon[40024]: pgmap v429: 168 pgs: 5 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 157 active+clean; 232 MiB data, 745 MiB used, 536 GiB / 536 GiB avail; 9.7 MiB/s rd, 3.2 MiB/s wr, 17 op/s 2024-09-20T22:47:13.348 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:13 smithi084 ceph-mon[40024]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:47:13.348 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:13 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T22:47:13.348 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:13 smithi084 ceph-mon[40024]: osdmap e352: 8 total, 8 up, 6 in 2024-09-20T22:47:13.348 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:13 smithi084 ceph-mon[40024]: 3.10 scrub starts 2024-09-20T22:47:13.348 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:13 smithi084 ceph-mon[40024]: 3.10 scrub ok 2024-09-20T22:47:13.348 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:13 smithi084 ceph-mon[40024]: osdmap e353: 8 total, 8 up, 6 in 2024-09-20T22:47:13.349 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:13 smithi084 ceph-mon[29774]: pgmap v429: 168 pgs: 5 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 157 active+clean; 232 MiB data, 745 MiB used, 536 GiB / 536 GiB avail; 9.7 MiB/s rd, 3.2 MiB/s wr, 17 op/s 2024-09-20T22:47:13.349 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:13 smithi084 ceph-mon[29774]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:47:13.349 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:13 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T22:47:13.349 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:13 smithi084 ceph-mon[29774]: osdmap e352: 8 total, 8 up, 6 in 2024-09-20T22:47:13.349 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:13 smithi084 ceph-mon[29774]: 3.10 scrub starts 2024-09-20T22:47:13.349 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:13 smithi084 ceph-mon[29774]: 3.10 scrub ok 2024-09-20T22:47:13.349 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:13 smithi084 ceph-mon[29774]: osdmap e353: 8 total, 8 up, 6 in 2024-09-20T22:47:13.410 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:47:13.432 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:13.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:13.433 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:47:13.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:13 smithi154 ceph-mon[33264]: pgmap v429: 168 pgs: 5 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 157 active+clean; 232 MiB data, 745 MiB used, 536 GiB / 536 GiB avail; 9.7 MiB/s rd, 3.2 MiB/s wr, 17 op/s 2024-09-20T22:47:13.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:13 smithi154 ceph-mon[33264]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:47:13.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:13 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T22:47:13.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:13 smithi154 ceph-mon[33264]: osdmap e352: 8 total, 8 up, 6 in 2024-09-20T22:47:13.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:13 smithi154 ceph-mon[33264]: 3.10 scrub starts 2024-09-20T22:47:13.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:13 smithi154 ceph-mon[33264]: 3.10 scrub ok 2024-09-20T22:47:13.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:13 smithi154 ceph-mon[33264]: osdmap e353: 8 total, 8 up, 6 in 2024-09-20T22:47:13.534 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:47:13.556 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:13.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:13.557 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:47:13.584 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:47:13.584 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:47:13.599 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:47:13.602 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:47:13.602 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:47:13.620 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:47:13.620 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:47:13.637 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-20T22:47:13.637 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T22:47:13.654 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:47:13.655 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:47:13.658 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:47:13.671 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:47:13.672 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:47:13.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:13.685 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:13.686 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:47:13.688 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:47:13.688 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:47:13.706 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:47:13.706 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:47:13.786 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:47:13.812 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:13.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:13.813 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:47:13.914 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:47:13.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:13.948 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:13.948 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:47:14.048 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:47:14.074 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:14.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:14.075 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:47:14.176 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:47:14.235 INFO:tasks.rados.rados.0.smithi084.stdout:1326: done (1 left) 2024-09-20T22:47:14.236 INFO:tasks.rados.rados.0.smithi084.stdout:1327: done (0 left) 2024-09-20T22:47:14.236 INFO:tasks.rados.rados.0.smithi084.stdout:1328: rollback oid 20 current snap is 135 2024-09-20T22:47:14.236 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 20 to 127 2024-09-20T22:47:14.236 INFO:tasks.rados.rados.0.smithi084.stdout:1329: rollback oid 29 current snap is 135 2024-09-20T22:47:14.236 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 29 to 124 2024-09-20T22:47:14.236 INFO:tasks.rados.rados.0.smithi084.stdout:1330: write (excl) oid 37 current snap is 135 2024-09-20T22:47:14.236 INFO:tasks.rados.rados.0.smithi084.stdout:1330: seq_num 478 ranges {433746=782908,1954981=78765} 2024-09-20T22:47:14.236 INFO:tasks.rados.rados.0.smithi084.stdout:1330: writing smithi08477203-37 from 433746 to 1216654 tid 1 2024-09-20T22:47:14.236 INFO:tasks.rados.rados.0.smithi084.stdout:1330: writing smithi08477203-37 from 1954981 to 2033746 tid 2 2024-09-20T22:47:14.236 INFO:tasks.rados.rados.0.smithi084.stdout:1331: write oid 8 current snap is 135 2024-09-20T22:47:14.236 INFO:tasks.rados.rados.0.smithi084.stdout:1329: finishing rollback tid 0 to smithi08477203-29 2024-09-20T22:47:14.237 INFO:tasks.rados.rados.0.smithi084.stdout:1331: seq_num 479 ranges {450553=574510,1613463=437090} 2024-09-20T22:47:14.237 INFO:tasks.rados.rados.0.smithi084.stdout:1331: writing smithi08477203-8 from 450553 to 1025063 tid 1 2024-09-20T22:47:14.237 INFO:tasks.rados.rados.0.smithi084.stdout:1331: writing smithi08477203-8 from 1613463 to 2050553 tid 2 2024-09-20T22:47:14.237 INFO:tasks.rados.rados.0.smithi084.stdout:1332: read oid 28 snap -1 2024-09-20T22:47:14.237 INFO:tasks.rados.rados.0.smithi084.stdout:1332: expect (ObjNum 463 snap 127 seq_num 463) 2024-09-20T22:47:14.237 INFO:tasks.rados.rados.0.smithi084.stdout:1333: read oid 50 snap 126 2024-09-20T22:47:14.237 INFO:tasks.rados.rados.0.smithi084.stdout:1333: expect (ObjNum 462 snap 126 seq_num 462) 2024-09-20T22:47:14.237 INFO:tasks.rados.rados.0.smithi084.stdout:1334: delete oid 34 current snap is 135 2024-09-20T22:47:14.237 INFO:tasks.rados.rados.0.smithi084.stdout:1328: finishing rollback tid 0 to smithi08477203-20 2024-09-20T22:47:14.237 INFO:tasks.rados.rados.0.smithi084.stdout:1329: finishing rollback tid 1 to smithi08477203-29 2024-09-20T22:47:14.237 INFO:tasks.rados.rados.0.smithi084.stdout:1329: finishing rollback tid 2 to smithi08477203-29 2024-09-20T22:47:14.237 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 29 v 744 (ObjNum 400 snap 101 seq_num 400) dirty exists 2024-09-20T22:47:14.237 INFO:tasks.rados.rados.0.smithi084.stdout:1328: finishing rollback tid 1 to smithi08477203-20 2024-09-20T22:47:14.238 INFO:tasks.rados.rados.0.smithi084.stdout:1328: finishing rollback tid 2 to smithi08477203-20 2024-09-20T22:47:14.238 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 20 v 115 (ObjNum 449 snap 120 seq_num 449) dirty exists 2024-09-20T22:47:14.238 INFO:tasks.rados.rados.0.smithi084.stdout:1330: finishing write tid 1 to smithi08477203-37 2024-09-20T22:47:14.238 INFO:tasks.rados.rados.0.smithi084.stdout:1330: finishing write tid 2 to smithi08477203-37 2024-09-20T22:47:14.238 INFO:tasks.rados.rados.0.smithi084.stdout:1330: finishing write tid 3 to smithi08477203-37 2024-09-20T22:47:14.238 INFO:tasks.rados.rados.0.smithi084.stdout:1330: finishing write tid 4 to smithi08477203-37 2024-09-20T22:47:14.238 INFO:tasks.rados.rados.0.smithi084.stdout:1330: oid 37 updating version 0 to 286 2024-09-20T22:47:14.238 INFO:tasks.rados.rados.0.smithi084.stdout:1330: oid 37 version 286 is already newer than 285 2024-09-20T22:47:14.238 INFO:tasks.rados.rados.0.smithi084.stdout:1330: oid 37 version 286 is already newer than 284 2024-09-20T22:47:14.238 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 37 v 286 (ObjNum 478 snap 135 seq_num 478) dirty exists 2024-09-20T22:47:14.238 INFO:tasks.rados.rados.0.smithi084.stdout:1330: left oid 37 (ObjNum 478 snap 135 seq_num 478) 2024-09-20T22:47:14.239 INFO:tasks.rados.rados.0.smithi084.stdout:1331: finishing write tid 1 to smithi08477203-8 2024-09-20T22:47:14.239 INFO:tasks.rados.rados.0.smithi084.stdout:1328: done (6 left) 2024-09-20T22:47:14.239 INFO:tasks.rados.rados.0.smithi084.stdout:1329: done (5 left) 2024-09-20T22:47:14.239 INFO:tasks.rados.rados.0.smithi084.stdout:1330: done (4 left) 2024-09-20T22:47:14.239 INFO:tasks.rados.rados.0.smithi084.stdout:1334: done (3 left) 2024-09-20T22:47:14.239 INFO:tasks.rados.rados.0.smithi084.stdout:1335: read oid 36 snap -1 2024-09-20T22:47:14.239 INFO:tasks.rados.rados.0.smithi084.stdout:1335: expect deleted 2024-09-20T22:47:14.239 INFO:tasks.rados.rados.0.smithi084.stdout:1336: delete oid 9 current snap is 135 2024-09-20T22:47:14.239 INFO:tasks.rados.rados.0.smithi084.stdout:1331: finishing write tid 2 to smithi08477203-8 2024-09-20T22:47:14.239 INFO:tasks.rados.rados.0.smithi084.stdout:1331: finishing write tid 3 to smithi08477203-8 2024-09-20T22:47:14.239 INFO:tasks.rados.rados.0.smithi084.stdout:1331: finishing write tid 4 to smithi08477203-8 2024-09-20T22:47:14.239 INFO:tasks.rados.rados.0.smithi084.stdout:1331: oid 8 updating version 0 to 403 2024-09-20T22:47:14.239 INFO:tasks.rados.rados.0.smithi084.stdout:1331: oid 8 version 403 is already newer than 402 2024-09-20T22:47:14.239 INFO:tasks.rados.rados.0.smithi084.stdout:1331: oid 8 version 403 is already newer than 401 2024-09-20T22:47:14.240 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 8 v 403 (ObjNum 479 snap 135 seq_num 479) dirty exists 2024-09-20T22:47:14.240 INFO:tasks.rados.rados.0.smithi084.stdout:1331: left oid 8 (ObjNum 479 snap 135 seq_num 479) 2024-09-20T22:47:14.240 INFO:tasks.rados.rados.0.smithi084.stdout:1333: expect (ObjNum 462 snap 126 seq_num 462) 2024-09-20T22:47:14.255 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:14.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:14.256 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:47:14.356 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:47:14.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:14.378 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:14.378 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:47:14.461 INFO:tasks.rados.rados.0.smithi084.stdout:1332: expect (ObjNum 463 snap 127 seq_num 463) 2024-09-20T22:47:14.479 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:47:14.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:14.501 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:14.501 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:47:14.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:14 smithi154 ceph-mon[33264]: 4.3 scrub starts 2024-09-20T22:47:14.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:14 smithi154 ceph-mon[33264]: 4.3 scrub ok 2024-09-20T22:47:14.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:14 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/2122064201' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:47:14.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:14 smithi154 ceph-mon[33264]: 7.4 scrub starts 2024-09-20T22:47:14.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:14 smithi154 ceph-mon[33264]: 7.4 scrub ok 2024-09-20T22:47:14.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:14 smithi154 ceph-mon[33264]: 4.a scrub starts 2024-09-20T22:47:14.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:14 smithi154 ceph-mon[33264]: 4.a scrub ok 2024-09-20T22:47:14.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:14 smithi154 ceph-mon[33264]: osdmap e354: 8 total, 8 up, 6 in 2024-09-20T22:47:14.517 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:14 smithi084 ceph-mon[29774]: 4.3 scrub starts 2024-09-20T22:47:14.517 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:14 smithi084 ceph-mon[29774]: 4.3 scrub ok 2024-09-20T22:47:14.517 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:14 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/2122064201' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:47:14.517 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:14 smithi084 ceph-mon[29774]: 7.4 scrub starts 2024-09-20T22:47:14.517 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:14 smithi084 ceph-mon[29774]: 7.4 scrub ok 2024-09-20T22:47:14.517 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:14 smithi084 ceph-mon[29774]: 4.a scrub starts 2024-09-20T22:47:14.517 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:14 smithi084 ceph-mon[29774]: 4.a scrub ok 2024-09-20T22:47:14.517 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:14 smithi084 ceph-mon[29774]: osdmap e354: 8 total, 8 up, 6 in 2024-09-20T22:47:14.518 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:14 smithi084 ceph-mon[40024]: 4.3 scrub starts 2024-09-20T22:47:14.518 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:14 smithi084 ceph-mon[40024]: 4.3 scrub ok 2024-09-20T22:47:14.518 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:14 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/2122064201' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:47:14.518 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:14 smithi084 ceph-mon[40024]: 7.4 scrub starts 2024-09-20T22:47:14.518 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:14 smithi084 ceph-mon[40024]: 7.4 scrub ok 2024-09-20T22:47:14.518 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:14 smithi084 ceph-mon[40024]: 4.a scrub starts 2024-09-20T22:47:14.518 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:14 smithi084 ceph-mon[40024]: 4.a scrub ok 2024-09-20T22:47:14.518 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:14 smithi084 ceph-mon[40024]: osdmap e354: 8 total, 8 up, 6 in 2024-09-20T22:47:14.602 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:47:14.621 INFO:tasks.rados.rados.0.smithi084.stdout:1331: done (4 left) 2024-09-20T22:47:14.621 INFO:tasks.rados.rados.0.smithi084.stdout:1332: done (3 left) 2024-09-20T22:47:14.621 INFO:tasks.rados.rados.0.smithi084.stdout:1333: done (2 left) 2024-09-20T22:47:14.621 INFO:tasks.rados.rados.0.smithi084.stdout:1335: done (1 left) 2024-09-20T22:47:14.621 INFO:tasks.rados.rados.0.smithi084.stdout:1336: done (0 left) 2024-09-20T22:47:14.621 INFO:tasks.rados.rados.0.smithi084.stdout:1337: delete oid 16 current snap is 135 2024-09-20T22:47:14.622 INFO:tasks.rados.rados.0.smithi084.stdout:1337: done (0 left) 2024-09-20T22:47:14.623 INFO:tasks.rados.rados.0.smithi084.stdout:1338: write oid 21 current snap is 135 2024-09-20T22:47:14.623 INFO:tasks.rados.rados.0.smithi084.stdout:1338: seq_num 483 ranges {517781=475506,1731038=386743} 2024-09-20T22:47:14.628 INFO:tasks.rados.rados.0.smithi084.stdout:1338: writing smithi08477203-21 from 517781 to 993287 tid 1 2024-09-20T22:47:14.628 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:14.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:14.629 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:47:14.634 INFO:tasks.rados.rados.0.smithi084.stdout:1338: writing smithi08477203-21 from 1731038 to 2117781 tid 2 2024-09-20T22:47:14.634 INFO:tasks.rados.rados.0.smithi084.stdout:1339: snap_remove snap 124 2024-09-20T22:47:14.725 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-20T22:47:14.725 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:47:14.731 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:47:14.755 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:14.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:14.756 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:47:14.857 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:47:14.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:14.880 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:14.880 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:47:14.981 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:47:15.007 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:15.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:15.008 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:47:15.109 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:47:15.126 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-20T22:47:15.131 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:15.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:15.132 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:47:15.233 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:47:15.260 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:15.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:15.261 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:47:15.264 INFO:tasks.rados.rados.0.smithi084.stdout:1339: done (1 left) 2024-09-20T22:47:15.264 INFO:tasks.rados.rados.0.smithi084.stdout:1340: rollback oid 22 current snap is 135 2024-09-20T22:47:15.264 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 22 to 133 2024-09-20T22:47:15.264 INFO:tasks.rados.rados.0.smithi084.stdout:1338: finishing write tid 1 to smithi08477203-21 2024-09-20T22:47:15.264 INFO:tasks.rados.rados.0.smithi084.stdout:1338: finishing write tid 2 to smithi08477203-21 2024-09-20T22:47:15.264 INFO:tasks.rados.rados.0.smithi084.stdout:1338: finishing write tid 3 to smithi08477203-21 2024-09-20T22:47:15.264 INFO:tasks.rados.rados.0.smithi084.stdout:1338: finishing write tid 4 to smithi08477203-21 2024-09-20T22:47:15.264 INFO:tasks.rados.rados.0.smithi084.stdout:1338: oid 21 updating version 0 to 509 2024-09-20T22:47:15.264 INFO:tasks.rados.rados.0.smithi084.stdout:1338: oid 21 updating version 509 to 510 2024-09-20T22:47:15.264 INFO:tasks.rados.rados.0.smithi084.stdout:1338: oid 21 version 510 is already newer than 508 2024-09-20T22:47:15.264 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 21 v 510 (ObjNum 483 snap 135 seq_num 483) dirty exists 2024-09-20T22:47:15.264 INFO:tasks.rados.rados.0.smithi084.stdout:1338: left oid 21 (ObjNum 483 snap 135 seq_num 483) 2024-09-20T22:47:15.265 INFO:tasks.rados.rados.0.smithi084.stdout:1338: done (1 left) 2024-09-20T22:47:15.265 INFO:tasks.rados.rados.0.smithi084.stdout:1341: read oid 45 snap -1 2024-09-20T22:47:15.265 INFO:tasks.rados.rados.0.smithi084.stdout:1341: expect (ObjNum 466 snap 128 seq_num 466) 2024-09-20T22:47:15.265 INFO:tasks.rados.rados.0.smithi084.stdout:1342: read oid 44 snap -1 2024-09-20T22:47:15.266 INFO:tasks.rados.rados.0.smithi084.stdout:1342: expect (ObjNum 470 snap 130 seq_num 470) 2024-09-20T22:47:15.266 INFO:tasks.rados.rados.0.smithi084.stdout:1343: copy_from oid 40 from oid 21 current snap is 135 2024-09-20T22:47:15.266 INFO:tasks.rados.rados.0.smithi084.stdout:1344: copy_from oid 16 from oid 10 current snap is 135 2024-09-20T22:47:15.266 INFO:tasks.rados.rados.0.smithi084.stdout:1345: write (excl) oid 6 current snap is 135 2024-09-20T22:47:15.266 INFO:tasks.rados.rados.0.smithi084.stdout:1345: seq_num 484 ranges {534588=550755,1789520=345068} 2024-09-20T22:47:15.270 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:15 smithi084 ceph-mon[40024]: 3.1b scrub starts 2024-09-20T22:47:15.270 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:15 smithi084 ceph-mon[40024]: 3.1b scrub ok 2024-09-20T22:47:15.270 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:15 smithi084 ceph-mon[40024]: pgmap v432: 168 pgs: 4 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 160 active+clean; 238 MiB data, 761 MiB used, 536 GiB / 536 GiB avail; 13 MiB/s rd, 1.3 MiB/s wr, 18 op/s 2024-09-20T22:47:15.271 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:15 smithi084 ceph-mon[29774]: 3.1b scrub starts 2024-09-20T22:47:15.271 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:15 smithi084 ceph-mon[29774]: 3.1b scrub ok 2024-09-20T22:47:15.271 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:15 smithi084 ceph-mon[29774]: pgmap v432: 168 pgs: 4 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 160 active+clean; 238 MiB data, 761 MiB used, 536 GiB / 536 GiB avail; 13 MiB/s rd, 1.3 MiB/s wr, 18 op/s 2024-09-20T22:47:15.273 INFO:tasks.rados.rados.0.smithi084.stdout:1345: writing smithi08477203-6 from 534588 to 1085343 tid 1 2024-09-20T22:47:15.282 INFO:tasks.rados.rados.0.smithi084.stdout:1345: writing smithi08477203-6 from 1789520 to 2134588 tid 2 2024-09-20T22:47:15.282 INFO:tasks.rados.rados.0.smithi084.stdout:1340: finishing rollback tid 0 to smithi08477203-22 2024-09-20T22:47:15.282 INFO:tasks.rados.rados.0.smithi084.stdout:1340: finishing rollback tid 1 to smithi08477203-22 2024-09-20T22:47:15.282 INFO:tasks.rados.rados.0.smithi084.stdout:1340: finishing rollback tid 2 to smithi08477203-22 2024-09-20T22:47:15.282 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 22 v 321 (ObjNum 468 snap 129 seq_num 468) dirty exists 2024-09-20T22:47:15.282 INFO:tasks.rados.rados.0.smithi084.stdout:1345: finishing write tid 1 to smithi08477203-6 2024-09-20T22:47:15.282 INFO:tasks.rados.rados.0.smithi084.stdout:1340: done (5 left) 2024-09-20T22:47:15.282 INFO:tasks.rados.rados.0.smithi084.stdout:1346: read oid 7 snap -1 2024-09-20T22:47:15.282 INFO:tasks.rados.rados.0.smithi084.stdout:1346: expect (ObjNum 442 snap 116 seq_num 442) 2024-09-20T22:47:15.283 INFO:tasks.rados.rados.0.smithi084.stdout:1347: snap_remove snap 119 2024-09-20T22:47:15.314 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.3 dump_blocked_ops 2024-09-20T22:47:15.361 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:47:15.383 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:15.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:15.384 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:47:15.398 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:47:15.485 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:47:15.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:15 smithi154 ceph-mon[33264]: 3.1b scrub starts 2024-09-20T22:47:15.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:15 smithi154 ceph-mon[33264]: 3.1b scrub ok 2024-09-20T22:47:15.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:15 smithi154 ceph-mon[33264]: pgmap v432: 168 pgs: 4 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 160 active+clean; 238 MiB data, 761 MiB used, 536 GiB / 536 GiB avail; 13 MiB/s rd, 1.3 MiB/s wr, 18 op/s 2024-09-20T22:47:15.507 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:15.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:15.508 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:47:15.608 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:47:15.631 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:15.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:15.632 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:47:15.669 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 5, 6, 7] out_osds: [4, 0] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-20T22:47:15.670 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd reweight 1 1 2024-09-20T22:47:15.733 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:47:15.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:15.755 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:15.755 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:47:15.856 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:47:15.880 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:15.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:15.881 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:47:15.889 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:47:15.982 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:47:16.006 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:16.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:16.007 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:47:16.108 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:47:16.131 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:16.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:16.132 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:47:16.233 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:47:16.254 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:16.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:16.254 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:47:16.262 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is unset 2024-09-20T22:47:16.267 INFO:tasks.rados.rados.0.smithi084.stdout:1347: done (6 left) 2024-09-20T22:47:16.267 INFO:tasks.rados.rados.0.smithi084.stdout:1348: copy_from oid 12 from oid 37 current snap is 135 2024-09-20T22:47:16.268 INFO:tasks.rados.rados.0.smithi084.stdout:1344: finishing copy_from racing read to smithi08477203-16 2024-09-20T22:47:16.268 INFO:tasks.rados.rados.0.smithi084.stdout:1341: expect (ObjNum 466 snap 128 seq_num 466) 2024-09-20T22:47:16.355 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:47:16.370 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:16 smithi084 ceph-mon[40024]: osdmap e355: 8 total, 8 up, 6 in 2024-09-20T22:47:16.371 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:16 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/282712493' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:47:16.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:16 smithi084 ceph-mon[29774]: osdmap e355: 8 total, 8 up, 6 in 2024-09-20T22:47:16.371 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:16 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/282712493' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:47:16.377 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-20T22:47:16.377 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:47:16.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:16.378 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:16.378 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:47:16.479 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:47:16.500 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:16.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:16.501 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:47:16.506 INFO:tasks.rados.rados.0.smithi084.stdout:1342: expect (ObjNum 470 snap 130 seq_num 470) 2024-09-20T22:47:16.520 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-20T22:47:16.601 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:47:16.624 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:16.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:16.625 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:47:16.635 INFO:tasks.rados.rados.0.smithi084.stdout:1343: finishing copy_from racing read to smithi08477203-40 2024-09-20T22:47:16.635 INFO:tasks.rados.rados.0.smithi084.stdout:1346: expect (ObjNum 442 snap 116 seq_num 442) 2024-09-20T22:47:16.725 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:47:16.748 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:16.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:16.749 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:47:16.750 INFO:tasks.rados.rados.0.smithi084.stdout:1345: finishing write tid 2 to smithi08477203-6 2024-09-20T22:47:16.750 INFO:tasks.rados.rados.0.smithi084.stdout:1345: finishing write tid 3 to smithi08477203-6 2024-09-20T22:47:16.751 INFO:tasks.rados.rados.0.smithi084.stdout:1341: done (6 left) 2024-09-20T22:47:16.751 INFO:tasks.rados.rados.0.smithi084.stdout:1342: done (5 left) 2024-09-20T22:47:16.751 INFO:tasks.rados.rados.0.smithi084.stdout:1346: done (4 left) 2024-09-20T22:47:16.751 INFO:tasks.rados.rados.0.smithi084.stdout:1349: read oid 14 snap -1 2024-09-20T22:47:16.751 INFO:tasks.rados.rados.0.smithi084.stdout:1349: expect (ObjNum 455 snap 125 seq_num 455) 2024-09-20T22:47:16.751 INFO:tasks.rados.rados.0.smithi084.stdout:1350: copy_from oid 3 from oid 43 current snap is 135 2024-09-20T22:47:16.751 INFO:tasks.rados.rados.0.smithi084.stdout:1345: finishing write tid 4 to smithi08477203-6 2024-09-20T22:47:16.751 INFO:tasks.rados.rados.0.smithi084.stdout:1345: oid 6 updating version 0 to 218 2024-09-20T22:47:16.751 INFO:tasks.rados.rados.0.smithi084.stdout:1345: oid 6 updating version 218 to 219 2024-09-20T22:47:16.751 INFO:tasks.rados.rados.0.smithi084.stdout:1345: oid 6 version 219 is already newer than 217 2024-09-20T22:47:16.751 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 6 v 219 (ObjNum 484 snap 135 seq_num 484) dirty exists 2024-09-20T22:47:16.751 INFO:tasks.rados.rados.0.smithi084.stdout:1345: left oid 6 (ObjNum 484 snap 135 seq_num 484) 2024-09-20T22:47:16.751 INFO:tasks.rados.rados.0.smithi084.stdout:1343: finishing copy_from to smithi08477203-40 2024-09-20T22:47:16.752 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 40 v 510 (ObjNum 483 snap 135 seq_num 483) dirty exists 2024-09-20T22:47:16.752 INFO:tasks.rados.rados.0.smithi084.stdout:1344: finishing copy_from to smithi08477203-16 2024-09-20T22:47:16.752 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 16 v 416 (ObjNum 419 snap 105 seq_num 419) dirty exists 2024-09-20T22:47:16.752 INFO:tasks.rados.rados.0.smithi084.stdout:1348: finishing copy_from racing read to smithi08477203-12 2024-09-20T22:47:16.752 INFO:tasks.rados.rados.0.smithi084.stdout:1348: finishing copy_from to smithi08477203-12 2024-09-20T22:47:16.752 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 12 v 286 (ObjNum 478 snap 135 seq_num 478) dirty exists 2024-09-20T22:47:16.752 INFO:tasks.rados.rados.0.smithi084.stdout:1343: done (5 left) 2024-09-20T22:47:16.752 INFO:tasks.rados.rados.0.smithi084.stdout:1344: done (4 left) 2024-09-20T22:47:16.752 INFO:tasks.rados.rados.0.smithi084.stdout:1345: done (3 left) 2024-09-20T22:47:16.752 INFO:tasks.rados.rados.0.smithi084.stdout:1348: done (2 left) 2024-09-20T22:47:16.752 INFO:tasks.rados.rados.0.smithi084.stdout:1351: snap_create 2024-09-20T22:47:16.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:16 smithi154 ceph-mon[33264]: osdmap e355: 8 total, 8 up, 6 in 2024-09-20T22:47:16.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:16 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/282712493' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:47:16.757 INFO:tasks.rados.rados.0.smithi084.stdout:1350: finishing copy_from racing read to smithi08477203-3 2024-09-20T22:47:16.761 INFO:tasks.rados.rados.0.smithi084.stdout:1349: expect (ObjNum 455 snap 125 seq_num 455) 2024-09-20T22:47:16.850 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:47:16.871 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:16.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:16.872 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:47:16.874 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:47:16.924 INFO:tasks.rados.rados.0.smithi084.stdout:1350: finishing copy_from to smithi08477203-3 2024-09-20T22:47:16.924 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 3 v 57 (ObjNum 418 snap 105 seq_num 418) dirty exists 2024-09-20T22:47:16.973 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:47:16.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:16.995 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:16.995 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:47:17.096 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:47:17.117 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:17.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:17.118 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:47:17.219 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:47:17.241 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:17.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:17.242 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:47:17.272 INFO:tasks.rados.rados.0.smithi084.stdout:1349: done (2 left) 2024-09-20T22:47:17.273 INFO:tasks.rados.rados.0.smithi084.stdout:1350: done (1 left) 2024-09-20T22:47:17.273 INFO:tasks.rados.rados.0.smithi084.stdout:1351: done (0 left) 2024-09-20T22:47:17.273 INFO:tasks.rados.rados.0.smithi084.stdout:1352: delete oid 13 current snap is 136 2024-09-20T22:47:17.273 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:17 smithi084 ceph-mon[29774]: pgmap v435: 168 pgs: 4 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 160 active+clean; 238 MiB data, 761 MiB used, 536 GiB / 536 GiB avail; 13 MiB/s rd, 1.3 MiB/s wr, 18 op/s 2024-09-20T22:47:17.280 INFO:tasks.rados.rados.0.smithi084.stdout:1352: done (0 left) 2024-09-20T22:47:17.280 INFO:tasks.rados.rados.0.smithi084.stdout:1353: snap_remove snap 129 2024-09-20T22:47:17.343 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:47:17.364 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:17.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:17.365 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:47:17.466 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:47:17.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:17.488 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:17.488 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:47:17.560 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:17 smithi084 ceph-mon[40024]: pgmap v435: 168 pgs: 4 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 160 active+clean; 238 MiB data, 761 MiB used, 536 GiB / 536 GiB avail; 13 MiB/s rd, 1.3 MiB/s wr, 18 op/s 2024-09-20T22:47:17.560 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:17 smithi084 ceph-mon[40024]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T22:47:17.560 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:17 smithi084 ceph-mon[40024]: Cluster is now healthy 2024-09-20T22:47:17.560 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:17 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/282712493' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T22:47:17.560 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:17 smithi084 ceph-mon[40024]: osdmap e356: 8 total, 8 up, 6 in 2024-09-20T22:47:17.560 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:17 smithi084 ceph-mon[40024]: 6.3 scrub starts 2024-09-20T22:47:17.560 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:17 smithi084 ceph-mon[40024]: 6.3 scrub ok 2024-09-20T22:47:17.560 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:17 smithi084 ceph-mon[40024]: 3.3 scrub starts 2024-09-20T22:47:17.561 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:17 smithi084 ceph-mon[40024]: 3.3 scrub ok 2024-09-20T22:47:17.561 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:17 smithi084 ceph-mon[29774]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T22:47:17.561 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:17 smithi084 ceph-mon[29774]: Cluster is now healthy 2024-09-20T22:47:17.561 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:17 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/282712493' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T22:47:17.561 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:17 smithi084 ceph-mon[29774]: osdmap e356: 8 total, 8 up, 6 in 2024-09-20T22:47:17.561 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:17 smithi084 ceph-mon[29774]: 6.3 scrub starts 2024-09-20T22:47:17.561 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:17 smithi084 ceph-mon[29774]: 6.3 scrub ok 2024-09-20T22:47:17.561 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:17 smithi084 ceph-mon[29774]: 3.3 scrub starts 2024-09-20T22:47:17.562 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:17 smithi084 ceph-mon[29774]: 3.3 scrub ok 2024-09-20T22:47:17.590 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:47:17.613 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:17.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:17.614 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:47:17.715 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:47:17.736 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:17.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:17.737 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:47:17.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:17 smithi154 ceph-mon[33264]: pgmap v435: 168 pgs: 4 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 160 active+clean; 238 MiB data, 761 MiB used, 536 GiB / 536 GiB avail; 13 MiB/s rd, 1.3 MiB/s wr, 18 op/s 2024-09-20T22:47:17.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:17 smithi154 ceph-mon[33264]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T22:47:17.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:17 smithi154 ceph-mon[33264]: Cluster is now healthy 2024-09-20T22:47:17.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:17 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/282712493' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T22:47:17.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:17 smithi154 ceph-mon[33264]: osdmap e356: 8 total, 8 up, 6 in 2024-09-20T22:47:17.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:17 smithi154 ceph-mon[33264]: 6.3 scrub starts 2024-09-20T22:47:17.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:17 smithi154 ceph-mon[33264]: 6.3 scrub ok 2024-09-20T22:47:17.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:17 smithi154 ceph-mon[33264]: 3.3 scrub starts 2024-09-20T22:47:17.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:17 smithi154 ceph-mon[33264]: 3.3 scrub ok 2024-09-20T22:47:17.837 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:47:17.861 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:17.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:17.862 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:47:17.963 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:47:17.986 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:17.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:17.987 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:47:18.088 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:47:18.109 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:18.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:18.110 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:47:18.185 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.1 to 1 (10000) 2024-09-20T22:47:18.186 INFO:tasks.rados.rados.0.smithi084.stdout:1353: done (0 left) 2024-09-20T22:47:18.186 INFO:tasks.rados.rados.0.smithi084.stdout:1354: delete oid 43 current snap is 136 2024-09-20T22:47:18.187 INFO:tasks.rados.rados.0.smithi084.stdout:1354: done (0 left) 2024-09-20T22:47:18.187 INFO:tasks.rados.rados.0.smithi084.stdout:1355: read oid 5 snap -1 2024-09-20T22:47:18.188 INFO:tasks.rados.rados.0.smithi084.stdout:1355: expect deleted 2024-09-20T22:47:18.188 INFO:tasks.rados.rados.0.smithi084.stdout:1356: snap_remove snap 127 2024-09-20T22:47:18.211 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:47:18.236 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:18.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:18.237 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:47:18.338 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:47:18.351 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.3 dump_historic_ops 2024-09-20T22:47:18.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:18.362 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:18.362 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:47:18.463 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:47:18.485 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:18.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:18.485 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:47:18.586 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:47:18.610 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:18.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:18.611 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:47:18.614 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:47:18 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: ::ffff:172.21.15.154 - - [20/Sep/2024:22:47:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-20T22:47:18.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:18 smithi084 ceph-mon[40024]: osdmap e357: 8 total, 8 up, 6 in 2024-09-20T22:47:18.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:18 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/3370799755' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-09-20T22:47:18.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:18 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-09-20T22:47:18.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:18 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-20T22:47:18.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:18 smithi084 ceph-mon[40024]: osdmap e358: 8 total, 8 up, 6 in 2024-09-20T22:47:18.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:18 smithi084 ceph-mon[29774]: osdmap e357: 8 total, 8 up, 6 in 2024-09-20T22:47:18.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:18 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/3370799755' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-09-20T22:47:18.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:18 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-09-20T22:47:18.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:18 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-20T22:47:18.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:18 smithi084 ceph-mon[29774]: osdmap e358: 8 total, 8 up, 6 in 2024-09-20T22:47:18.712 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:47:18.734 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:18.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:18.735 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:47:18.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:18 smithi154 ceph-mon[33264]: osdmap e357: 8 total, 8 up, 6 in 2024-09-20T22:47:18.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:18 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/3370799755' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-09-20T22:47:18.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:18 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-09-20T22:47:18.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:18 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-20T22:47:18.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:18 smithi154 ceph-mon[33264]: osdmap e358: 8 total, 8 up, 6 in 2024-09-20T22:47:18.836 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:47:18.857 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:18.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:18.858 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:47:18.959 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:47:18.972 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd reweight 2 1 2024-09-20T22:47:18.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:18.981 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:18.982 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:47:19.022 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:47:19.034 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:47:19.082 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:47:19.103 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:19.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:19.104 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:47:19.187 INFO:tasks.rados.rados.0.smithi084.stdout:1356: done (1 left) 2024-09-20T22:47:19.187 INFO:tasks.rados.rados.0.smithi084.stdout:1357: read oid 42 snap -1 2024-09-20T22:47:19.187 INFO:tasks.rados.rados.0.smithi084.stdout:1357: expect (ObjNum 433 snap 112 seq_num 433) 2024-09-20T22:47:19.187 INFO:tasks.rados.rados.0.smithi084.stdout:1358: write (excl) oid 6 current snap is 136 2024-09-20T22:47:19.187 INFO:tasks.rados.rados.0.smithi084.stdout:1358: seq_num 487 ranges {585009=492855,1564966=620043} 2024-09-20T22:47:19.194 INFO:tasks.rados.rados.0.smithi084.stdout:1358: writing smithi08477203-6 from 585009 to 1077864 tid 1 2024-09-20T22:47:19.201 INFO:tasks.rados.rados.0.smithi084.stdout:1358: writing smithi08477203-6 from 1564966 to 2185009 tid 2 2024-09-20T22:47:19.201 INFO:tasks.rados.rados.0.smithi084.stdout:1359: snap_create 2024-09-20T22:47:19.201 INFO:tasks.rados.rados.0.smithi084.stdout:1358: finishing write tid 1 to smithi08477203-6 2024-09-20T22:47:19.204 INFO:tasks.rados.rados.0.smithi084.stdout:1358: finishing write tid 2 to smithi08477203-6 2024-09-20T22:47:19.204 INFO:tasks.rados.rados.0.smithi084.stdout:1358: finishing write tid 3 to smithi08477203-6 2024-09-20T22:47:19.204 INFO:tasks.rados.rados.0.smithi084.stdout:1358: finishing write tid 4 to smithi08477203-6 2024-09-20T22:47:19.204 INFO:tasks.rados.rados.0.smithi084.stdout:1358: oid 6 updating version 0 to 220 2024-09-20T22:47:19.204 INFO:tasks.rados.rados.0.smithi084.stdout:1358: oid 6 updating version 220 to 222 2024-09-20T22:47:19.205 INFO:tasks.rados.rados.0.smithi084.stdout:1358: oid 6 version 222 is already newer than 221 2024-09-20T22:47:19.205 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 6 v 222 (ObjNum 487 snap 136 seq_num 487) dirty exists 2024-09-20T22:47:19.205 INFO:tasks.rados.rados.0.smithi084.stdout:1358: left oid 6 (ObjNum 487 snap 136 seq_num 487) 2024-09-20T22:47:19.205 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:47:19.206 INFO:tasks.rados.rados.0.smithi084.stdout:1357: expect (ObjNum 433 snap 112 seq_num 433) 2024-09-20T22:47:19.225 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:19.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:19.226 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:47:19.327 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:47:19.352 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:19.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:19.353 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:47:19.403 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd set noscrub 2024-09-20T22:47:19.454 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:47:19.475 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:19.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:19.476 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:47:19.504 INFO:journalctl@ceph.iscsi.iscsi.a.smithi154.stdout:Sep 20 22:47:19 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-iscsi-iscsi-a[59730]: debug there is no tcmu-runner data available 2024-09-20T22:47:19.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:19 smithi154 ceph-mon[33264]: pgmap v438: 168 pgs: 4 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 160 active+clean; 238 MiB data, 761 MiB used, 536 GiB / 536 GiB avail 2024-09-20T22:47:19.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:19 smithi154 ceph-mon[33264]: 6.2 scrub starts 2024-09-20T22:47:19.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:19 smithi154 ceph-mon[33264]: 6.2 scrub ok 2024-09-20T22:47:19.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:19 smithi154 ceph-mon[33264]: 6.8 deep-scrub starts 2024-09-20T22:47:19.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:19 smithi154 ceph-mon[33264]: 6.8 deep-scrub ok 2024-09-20T22:47:19.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:19 smithi154 ceph-mon[33264]: osdmap e359: 8 total, 8 up, 6 in 2024-09-20T22:47:19.540 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:47:19.541 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:47:19.541 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:19 smithi084 ceph-mon[40024]: pgmap v438: 168 pgs: 4 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 160 active+clean; 238 MiB data, 761 MiB used, 536 GiB / 536 GiB avail 2024-09-20T22:47:19.541 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:19 smithi084 ceph-mon[40024]: 6.2 scrub starts 2024-09-20T22:47:19.541 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:19 smithi084 ceph-mon[40024]: 6.2 scrub ok 2024-09-20T22:47:19.541 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:19 smithi084 ceph-mon[40024]: 6.8 deep-scrub starts 2024-09-20T22:47:19.541 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:19 smithi084 ceph-mon[40024]: 6.8 deep-scrub ok 2024-09-20T22:47:19.541 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:19 smithi084 ceph-mon[40024]: osdmap e359: 8 total, 8 up, 6 in 2024-09-20T22:47:19.542 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:19 smithi084 ceph-mon[29774]: pgmap v438: 168 pgs: 4 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 160 active+clean; 238 MiB data, 761 MiB used, 536 GiB / 536 GiB avail 2024-09-20T22:47:19.542 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:19 smithi084 ceph-mon[29774]: 6.2 scrub starts 2024-09-20T22:47:19.542 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:19 smithi084 ceph-mon[29774]: 6.2 scrub ok 2024-09-20T22:47:19.542 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:19 smithi084 ceph-mon[29774]: 6.8 deep-scrub starts 2024-09-20T22:47:19.542 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:19 smithi084 ceph-mon[29774]: 6.8 deep-scrub ok 2024-09-20T22:47:19.542 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:19 smithi084 ceph-mon[29774]: osdmap e359: 8 total, 8 up, 6 in 2024-09-20T22:47:19.547 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:47:19.547 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:47:19.575 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:47:19.575 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:47:19.577 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:47:19.591 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:47:19.594 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-20T22:47:19.594 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T22:47:19.600 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:19.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:19.601 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:47:19.615 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:47:19.616 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:47:19.632 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:47:19.633 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:47:19.649 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:47:19.649 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:47:19.666 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:47:19.666 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:47:19.703 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:47:19.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:19.729 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:19.729 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:47:19.830 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:47:19.854 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:19.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:19.855 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:47:19.956 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:47:19.980 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:19.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:19.981 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:47:20.082 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:47:20.106 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:20.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:20.107 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:47:20.190 INFO:tasks.rados.rados.0.smithi084.stdout:1355: done (3 left) 2024-09-20T22:47:20.191 INFO:tasks.rados.rados.0.smithi084.stdout:1357: done (2 left) 2024-09-20T22:47:20.191 INFO:tasks.rados.rados.0.smithi084.stdout:1358: done (1 left) 2024-09-20T22:47:20.191 INFO:tasks.rados.rados.0.smithi084.stdout:1359: done (0 left) 2024-09-20T22:47:20.191 INFO:tasks.rados.rados.0.smithi084.stdout:1360: read oid 9 snap -1 2024-09-20T22:47:20.191 INFO:tasks.rados.rados.0.smithi084.stdout:1360: expect deleted 2024-09-20T22:47:20.191 INFO:tasks.rados.rados.0.smithi084.stdout:1361: write oid 6 current snap is 137 2024-09-20T22:47:20.191 INFO:tasks.rados.rados.0.smithi084.stdout:1361: seq_num 488 ranges {601816=568104,1623448=451823} 2024-09-20T22:47:20.196 INFO:tasks.rados.rados.0.smithi084.stdout:1361: writing smithi08477203-6 from 601816 to 1169920 tid 1 2024-09-20T22:47:20.203 INFO:tasks.rados.rados.0.smithi084.stdout:1361: writing smithi08477203-6 from 1623448 to 2075271 tid 2 2024-09-20T22:47:20.203 INFO:tasks.rados.rados.0.smithi084.stdout:1360: done (1 left) 2024-09-20T22:47:20.203 INFO:tasks.rados.rados.0.smithi084.stdout:1362: rollback oid 9 current snap is 137 2024-09-20T22:47:20.203 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 9 to 105 2024-09-20T22:47:20.203 INFO:tasks.rados.rados.0.smithi084.stdout:1363: write (excl) oid 38 current snap is 137 2024-09-20T22:47:20.203 INFO:tasks.rados.rados.0.smithi084.stdout:1363: seq_num 489 ranges {618623=643353,1681930=536693} 2024-09-20T22:47:20.208 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:47:20.209 INFO:tasks.rados.rados.0.smithi084.stdout:1363: writing smithi08477203-38 from 618623 to 1261976 tid 1 2024-09-20T22:47:20.215 INFO:tasks.rados.rados.0.smithi084.stdout:1363: writing smithi08477203-38 from 1681930 to 2218623 tid 2 2024-09-20T22:47:20.216 INFO:tasks.rados.rados.0.smithi084.stdout:1364: snap_create 2024-09-20T22:47:20.216 INFO:tasks.rados.rados.0.smithi084.stdout:1361: finishing write tid 1 to smithi08477203-6 2024-09-20T22:47:20.216 INFO:tasks.rados.rados.0.smithi084.stdout:1362: finishing rollback tid 1 to smithi08477203-9 2024-09-20T22:47:20.216 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 9 v 0 (ObjNum 65 snap 0 seq_num 1953066355) dirty dne 2024-09-20T22:47:20.216 INFO:tasks.rados.rados.0.smithi084.stdout:1361: finishing write tid 2 to smithi08477203-6 2024-09-20T22:47:20.216 INFO:tasks.rados.rados.0.smithi084.stdout:1361: finishing write tid 3 to smithi08477203-6 2024-09-20T22:47:20.217 INFO:tasks.rados.rados.0.smithi084.stdout:1361: finishing write tid 4 to smithi08477203-6 2024-09-20T22:47:20.217 INFO:tasks.rados.rados.0.smithi084.stdout:1361: oid 6 updating version 0 to 223 2024-09-20T22:47:20.217 INFO:tasks.rados.rados.0.smithi084.stdout:1361: oid 6 updating version 223 to 224 2024-09-20T22:47:20.217 INFO:tasks.rados.rados.0.smithi084.stdout:1361: oid 6 updating version 224 to 225 2024-09-20T22:47:20.217 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 6 v 225 (ObjNum 488 snap 137 seq_num 488) dirty exists 2024-09-20T22:47:20.217 INFO:tasks.rados.rados.0.smithi084.stdout:1361: left oid 6 (ObjNum 488 snap 137 seq_num 488) 2024-09-20T22:47:20.219 INFO:tasks.rados.rados.0.smithi084.stdout:1363: finishing write tid 1 to smithi08477203-38 2024-09-20T22:47:20.219 INFO:tasks.rados.rados.0.smithi084.stdout:1363: finishing write tid 2 to smithi08477203-38 2024-09-20T22:47:20.219 INFO:tasks.rados.rados.0.smithi084.stdout:1363: finishing write tid 3 to smithi08477203-38 2024-09-20T22:47:20.219 INFO:tasks.rados.rados.0.smithi084.stdout:1363: finishing write tid 4 to smithi08477203-38 2024-09-20T22:47:20.219 INFO:tasks.rados.rados.0.smithi084.stdout:1363: oid 38 updating version 0 to 406 2024-09-20T22:47:20.220 INFO:tasks.rados.rados.0.smithi084.stdout:1363: oid 38 updating version 406 to 407 2024-09-20T22:47:20.220 INFO:tasks.rados.rados.0.smithi084.stdout:1363: oid 38 version 407 is already newer than 405 2024-09-20T22:47:20.220 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 38 v 407 (ObjNum 489 snap 137 seq_num 489) dirty exists 2024-09-20T22:47:20.220 INFO:tasks.rados.rados.0.smithi084.stdout:1363: left oid 38 (ObjNum 489 snap 137 seq_num 489) 2024-09-20T22:47:20.235 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:20.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:20.236 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:47:20.288 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-20T22:47:20.288 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2024-09-20T22:47:20.288 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2024-09-20T22:47:20.288 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [ 2024-09-20T22:47:20.288 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.289 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:554 7.a 7:5122292a:::smithi08477203-33:head [copy-from ver 6] snapc 13=[13,12,11,10,f,d] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-09-20T22:47:20.289 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:43:27.336824+0000", 2024-09-20T22:47:20.289 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 232.94852082099999, 2024-09-20T22:47:20.289 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.056403397000000001, 2024-09-20T22:47:20.289 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:47:20.289 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:47:20.289 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:47:20.289 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:47:20.289 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:47:20.289 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:47:20.289 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 554 2024-09-20T22:47:20.289 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.289 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:47:20.289 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.289 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:47:20.290 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:27.336824+0000", 2024-09-20T22:47:20.290 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:47:20.290 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.290 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.290 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:47:20.290 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:27.336824+0000", 2024-09-20T22:47:20.290 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:47:20.290 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.290 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.290 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:47:20.290 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:27.336826+0000", 2024-09-20T22:47:20.290 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.0339999999999999e-06 2024-09-20T22:47:20.290 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.290 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.291 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:47:20.291 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:27.336837+0000", 2024-09-20T22:47:20.291 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1243000000000001e-05 2024-09-20T22:47:20.291 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.291 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.291 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:47:20.291 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:27.336857+0000", 2024-09-20T22:47:20.291 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.9868999999999999e-05 2024-09-20T22:47:20.291 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.291 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.291 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:47:20.291 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:27.336869+0000", 2024-09-20T22:47:20.291 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1348999999999999e-05 2024-09-20T22:47:20.292 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.292 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.292 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:47:20.292 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:27.336946+0000", 2024-09-20T22:47:20.292 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.7683999999999997e-05 2024-09-20T22:47:20.292 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.292 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.292 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:47:20.292 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:27.337045+0000", 2024-09-20T22:47:20.292 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.9265999999999995e-05 2024-09-20T22:47:20.293 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.293 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.293 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 2,5", 2024-09-20T22:47:20.293 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:27.340603+0000", 2024-09-20T22:47:20.293 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.003557892 2024-09-20T22:47:20.293 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.293 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.293 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:47:20.293 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:27.343571+0000", 2024-09-20T22:47:20.293 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0029676609999999999 2024-09-20T22:47:20.293 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.293 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.293 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:47:20.293 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:27.346457+0000", 2024-09-20T22:47:20.294 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0028860040000000002 2024-09-20T22:47:20.294 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.294 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.294 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:47:20.294 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:27.393131+0000", 2024-09-20T22:47:20.294 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.046673813000000001 2024-09-20T22:47:20.294 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.294 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.294 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:47:20.294 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:27.393173+0000", 2024-09-20T22:47:20.294 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.2076999999999997e-05 2024-09-20T22:47:20.294 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.294 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.295 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:47:20.295 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:27.393227+0000", 2024-09-20T22:47:20.295 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.4505000000000001e-05 2024-09-20T22:47:20.295 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:47:20.295 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:47:20.295 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:47:20.295 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.295 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.295 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:663 7.12 7:4fa4b2b3:::smithi08477203-21:head [write 665577~781581,stat] snapc 17=[17,16,15,11,d] ondisk+write+known_if_redirected+supports_pool_eio e137)", 2024-09-20T22:47:20.295 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:43:36.161568+0000", 2024-09-20T22:47:20.295 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 224.123776644, 2024-09-20T22:47:20.295 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.99286350899999998, 2024-09-20T22:47:20.295 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:47:20.296 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:47:20.296 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:47:20.296 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:47:20.296 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:47:20.296 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:47:20.296 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 663 2024-09-20T22:47:20.296 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.296 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:47:20.296 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.296 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:47:20.296 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.161568+0000", 2024-09-20T22:47:20.296 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:47:20.296 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.296 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.297 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:47:20.297 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.161568+0000", 2024-09-20T22:47:20.297 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:47:20.297 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.297 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.297 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:47:20.297 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.161570+0000", 2024-09-20T22:47:20.297 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.0679999999999999e-06 2024-09-20T22:47:20.297 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.297 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.297 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:47:20.297 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.162310+0000", 2024-09-20T22:47:20.297 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00074005999999999996 2024-09-20T22:47:20.297 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.298 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.298 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:47:20.298 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.162349+0000", 2024-09-20T22:47:20.298 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.8778e-05 2024-09-20T22:47:20.298 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.298 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.298 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:47:20.298 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.162375+0000", 2024-09-20T22:47:20.298 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.6089e-05 2024-09-20T22:47:20.298 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.298 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.298 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:47:20.298 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.162434+0000", 2024-09-20T22:47:20.299 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.8764999999999997e-05 2024-09-20T22:47:20.299 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.299 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.299 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-09-20T22:47:20.299 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.162443+0000", 2024-09-20T22:47:20.299 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.054e-06 2024-09-20T22:47:20.299 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.299 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.299 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:47:20.299 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.143416+0000", 2024-09-20T22:47:20.299 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.98097291600000003 2024-09-20T22:47:20.299 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.299 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.299 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:47:20.299 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.143522+0000", 2024-09-20T22:47:20.300 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00010626 2024-09-20T22:47:20.300 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.300 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.300 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 1,2", 2024-09-20T22:47:20.300 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.143654+0000", 2024-09-20T22:47:20.300 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013218799999999999 2024-09-20T22:47:20.300 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.300 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.300 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:47:20.300 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.148086+0000", 2024-09-20T22:47:20.300 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0044314189999999998 2024-09-20T22:47:20.300 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.300 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.301 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:47:20.301 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.153546+0000", 2024-09-20T22:47:20.301 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0054599920000000003 2024-09-20T22:47:20.301 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.301 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.301 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:47:20.301 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.154317+0000", 2024-09-20T22:47:20.301 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00077125999999999996 2024-09-20T22:47:20.301 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.301 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.301 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:47:20.301 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.154380+0000", 2024-09-20T22:47:20.301 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.2526000000000004e-05 2024-09-20T22:47:20.301 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.301 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.302 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:47:20.302 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.154432+0000", 2024-09-20T22:47:20.302 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.2133999999999999e-05 2024-09-20T22:47:20.302 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:47:20.302 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:47:20.302 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:47:20.303 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.303 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.303 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:664 7.12 7:4fa4b2b3:::smithi08477203-21:head [write 2062560~563688] snapc 17=[17,16,15,11,d] ondisk+write+known_if_redirected+supports_pool_eio e137)", 2024-09-20T22:47:20.304 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:43:36.167947+0000", 2024-09-20T22:47:20.304 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 224.11739817399999, 2024-09-20T22:47:20.304 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.98665172599999995, 2024-09-20T22:47:20.304 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:47:20.304 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:47:20.304 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:47:20.305 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:47:20.305 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:47:20.305 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:47:20.305 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 664 2024-09-20T22:47:20.305 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.305 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:47:20.305 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.305 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:47:20.306 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.167947+0000", 2024-09-20T22:47:20.306 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:47:20.306 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.306 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.306 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:47:20.306 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.167947+0000", 2024-09-20T22:47:20.306 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:47:20.306 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.306 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.306 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:47:20.306 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.167948+0000", 2024-09-20T22:47:20.306 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.187e-06 2024-09-20T22:47:20.306 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.306 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.307 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:47:20.307 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.168193+0000", 2024-09-20T22:47:20.307 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00024496199999999999 2024-09-20T22:47:20.307 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.307 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.307 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:47:20.307 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.168203+0000", 2024-09-20T22:47:20.307 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0543999999999999e-05 2024-09-20T22:47:20.307 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.307 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.307 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:47:20.307 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.168209+0000", 2024-09-20T22:47:20.307 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.2839999999999997e-06 2024-09-20T22:47:20.307 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.308 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.308 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:47:20.308 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.168252+0000", 2024-09-20T22:47:20.308 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.3115000000000003e-05 2024-09-20T22:47:20.308 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.308 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.308 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-09-20T22:47:20.308 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.168257+0000", 2024-09-20T22:47:20.308 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.6860000000000003e-06 2024-09-20T22:47:20.309 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.309 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.309 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:47:20.309 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.144795+0000", 2024-09-20T22:47:20.309 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.97653758099999999 2024-09-20T22:47:20.309 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.309 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.309 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:47:20.309 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.144865+0000", 2024-09-20T22:47:20.309 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.9641000000000006e-05 2024-09-20T22:47:20.309 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.309 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.309 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 1,2", 2024-09-20T22:47:20.310 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.144986+0000", 2024-09-20T22:47:20.310 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000121248 2024-09-20T22:47:20.310 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.310 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.310 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:47:20.310 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.148091+0000", 2024-09-20T22:47:20.310 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0031050399999999999 2024-09-20T22:47:20.310 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.310 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.310 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:47:20.310 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.153603+0000", 2024-09-20T22:47:20.310 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0055118479999999997 2024-09-20T22:47:20.310 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.311 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.311 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:47:20.311 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.154508+0000", 2024-09-20T22:47:20.311 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00090530600000000004 2024-09-20T22:47:20.311 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.311 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.311 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:47:20.311 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.154560+0000", 2024-09-20T22:47:20.311 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.1474000000000002e-05 2024-09-20T22:47:20.311 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.311 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.311 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:47:20.311 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.154598+0000", 2024-09-20T22:47:20.311 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.8810000000000003e-05 2024-09-20T22:47:20.311 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:47:20.312 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:47:20.312 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:47:20.312 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.312 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.312 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:665 7.12 7:4fa4b2b3:::smithi08477203-21:head [write 3182305~643664] snapc 17=[17,16,15,11,d] ondisk+write+known_if_redirected+supports_pool_eio e137)", 2024-09-20T22:47:20.312 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:43:36.174520+0000", 2024-09-20T22:47:20.312 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 224.110825084, 2024-09-20T22:47:20.312 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.98020083400000002, 2024-09-20T22:47:20.312 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:47:20.312 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:47:20.312 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:47:20.312 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:47:20.312 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:47:20.312 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:47:20.313 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 665 2024-09-20T22:47:20.313 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.313 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:47:20.313 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.313 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:47:20.313 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174520+0000", 2024-09-20T22:47:20.313 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:47:20.313 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.313 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.313 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:47:20.313 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174520+0000", 2024-09-20T22:47:20.313 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:47:20.313 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.313 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.313 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:47:20.314 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174521+0000", 2024-09-20T22:47:20.314 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.234e-06 2024-09-20T22:47:20.314 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.314 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.314 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:47:20.314 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174816+0000", 2024-09-20T22:47:20.314 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000294682 2024-09-20T22:47:20.314 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.314 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.314 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:47:20.314 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174827+0000", 2024-09-20T22:47:20.314 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1025e-05 2024-09-20T22:47:20.314 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.314 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.315 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:47:20.315 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174832+0000", 2024-09-20T22:47:20.315 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.3990000000000003e-06 2024-09-20T22:47:20.315 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.315 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.315 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:47:20.315 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174868+0000", 2024-09-20T22:47:20.315 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.6225e-05 2024-09-20T22:47:20.315 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.315 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.315 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-09-20T22:47:20.315 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174873+0000", 2024-09-20T22:47:20.315 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.848e-06 2024-09-20T22:47:20.315 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.316 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.316 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:47:20.316 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.145844+0000", 2024-09-20T22:47:20.316 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.97097048900000005 2024-09-20T22:47:20.316 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.316 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.316 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:47:20.316 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.145917+0000", 2024-09-20T22:47:20.316 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.3248000000000001e-05 2024-09-20T22:47:20.316 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.316 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.316 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 1,2", 2024-09-20T22:47:20.316 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.146028+0000", 2024-09-20T22:47:20.316 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000110773 2024-09-20T22:47:20.316 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.317 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.317 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:47:20.317 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.148095+0000", 2024-09-20T22:47:20.317 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0020678179999999999 2024-09-20T22:47:20.317 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.317 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.317 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:47:20.317 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.153967+0000", 2024-09-20T22:47:20.317 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0058717350000000003 2024-09-20T22:47:20.317 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.317 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.317 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:47:20.317 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.154633+0000", 2024-09-20T22:47:20.317 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00066600600000000002 2024-09-20T22:47:20.318 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.318 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.318 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:47:20.318 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.154683+0000", 2024-09-20T22:47:20.318 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.9872e-05 2024-09-20T22:47:20.318 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.318 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.318 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:47:20.318 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.154721+0000", 2024-09-20T22:47:20.318 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.748e-05 2024-09-20T22:47:20.318 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:47:20.318 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:47:20.318 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:47:20.318 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.319 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.319 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:666 7.12 7:4fa4b2b3:::smithi08477203-21:head [setxattr _header (57) in=7b,truncate 3865577] snapc 17=[17,16,15,11,d] ondisk+write+known_if_redirected+supports_pool_eio e137)", 2024-09-20T22:47:20.319 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:43:36.174844+0000", 2024-09-20T22:47:20.319 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 224.11050088499999, 2024-09-20T22:47:20.319 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.98002428600000002, 2024-09-20T22:47:20.319 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:47:20.319 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:47:20.319 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:47:20.319 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:47:20.319 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:47:20.319 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:47:20.319 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 666 2024-09-20T22:47:20.319 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.319 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:47:20.319 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.320 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:47:20.320 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174844+0000", 2024-09-20T22:47:20.320 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:47:20.320 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.320 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.320 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:47:20.320 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174844+0000", 2024-09-20T22:47:20.320 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:47:20.320 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.320 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.320 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:47:20.320 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174845+0000", 2024-09-20T22:47:20.320 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.1999999999999999e-07 2024-09-20T22:47:20.320 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.321 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.321 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:47:20.321 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174853+0000", 2024-09-20T22:47:20.321 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.1950000000000003e-06 2024-09-20T22:47:20.321 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.321 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.321 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:47:20.321 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174858+0000", 2024-09-20T22:47:20.321 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.5650000000000001e-06 2024-09-20T22:47:20.321 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.321 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.321 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:47:20.321 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174863+0000", 2024-09-20T22:47:20.321 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.1649999999999997e-06 2024-09-20T22:47:20.321 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.322 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.322 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:47:20.322 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174915+0000", 2024-09-20T22:47:20.322 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.1045999999999999e-05 2024-09-20T22:47:20.322 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.322 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.322 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-09-20T22:47:20.322 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174921+0000", 2024-09-20T22:47:20.322 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.404e-06 2024-09-20T22:47:20.322 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.322 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.322 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:47:20.322 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.146528+0000", 2024-09-20T22:47:20.322 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.97160681199999999 2024-09-20T22:47:20.323 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.323 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.323 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:47:20.323 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.146881+0000", 2024-09-20T22:47:20.323 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00035348200000000001 2024-09-20T22:47:20.323 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.323 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.323 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 1,2", 2024-09-20T22:47:20.323 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.147011+0000", 2024-09-20T22:47:20.323 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013018900000000001 2024-09-20T22:47:20.323 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.323 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.323 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:47:20.323 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.148100+0000", 2024-09-20T22:47:20.324 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010881580000000001 2024-09-20T22:47:20.324 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.324 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.324 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:47:20.324 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.154278+0000", 2024-09-20T22:47:20.324 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0061781730000000003 2024-09-20T22:47:20.324 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.325 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.325 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:47:20.325 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.154764+0000", 2024-09-20T22:47:20.325 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00048589199999999998 2024-09-20T22:47:20.325 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.325 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.326 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:47:20.326 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.154815+0000", 2024-09-20T22:47:20.326 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.1428000000000003e-05 2024-09-20T22:47:20.326 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.326 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.326 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:47:20.326 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.154868+0000", 2024-09-20T22:47:20.327 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.3156999999999999e-05 2024-09-20T22:47:20.327 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:47:20.327 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:47:20.327 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:47:20.327 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.327 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.327 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:667 7.12 7:4fa4b2b3:::smithi08477203-21:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e137)", 2024-09-20T22:47:20.328 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:43:36.174872+0000", 2024-09-20T22:47:20.328 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 224.110473192, 2024-09-20T22:47:20.328 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.98024790299999998, 2024-09-20T22:47:20.328 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:47:20.328 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:47:20.328 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-20T22:47:20.328 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:47:20.328 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:47:20.328 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:47:20.328 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 667 2024-09-20T22:47:20.329 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.329 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:47:20.329 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.329 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:47:20.329 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174872+0000", 2024-09-20T22:47:20.329 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:47:20.329 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.329 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.329 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:47:20.329 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174872+0000", 2024-09-20T22:47:20.329 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:47:20.329 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.329 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.329 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:47:20.330 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174872+0000", 2024-09-20T22:47:20.330 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.2199999999999999e-07 2024-09-20T22:47:20.330 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.330 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.330 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:47:20.330 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174874+0000", 2024-09-20T22:47:20.330 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.187e-06 2024-09-20T22:47:20.330 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.330 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.330 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:47:20.330 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174881+0000", 2024-09-20T22:47:20.330 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.674e-06 2024-09-20T22:47:20.330 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.330 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.331 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:47:20.331 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174886+0000", 2024-09-20T22:47:20.331 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.2240000000000003e-06 2024-09-20T22:47:20.331 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.331 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.331 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:47:20.331 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174936+0000", 2024-09-20T22:47:20.331 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.9824999999999998e-05 2024-09-20T22:47:20.331 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.331 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.331 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-09-20T22:47:20.331 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174940+0000", 2024-09-20T22:47:20.331 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.5700000000000001e-06 2024-09-20T22:47:20.331 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.331 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.332 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:47:20.332 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.147523+0000", 2024-09-20T22:47:20.332 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.97258315399999995 2024-09-20T22:47:20.332 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.332 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.332 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for rw locks", 2024-09-20T22:47:20.332 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.147591+0000", 2024-09-20T22:47:20.332 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.7878000000000003e-05 2024-09-20T22:47:20.332 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.332 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.332 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:47:20.332 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.154925+0000", 2024-09-20T22:47:20.332 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0073344430000000004 2024-09-20T22:47:20.332 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.333 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.333 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:47:20.333 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.155020+0000", 2024-09-20T22:47:20.333 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.4767999999999998e-05 2024-09-20T22:47:20.333 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.333 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.333 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:47:20.333 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.155120+0000", 2024-09-20T22:47:20.333 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.9758000000000003e-05 2024-09-20T22:47:20.333 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:47:20.333 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:47:20.333 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:47:20.333 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.333 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.333 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.14478.0:1259 5.3 5:c609908c:::notify.5:head [watch reconnect cookie 94843898679808 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e169)", 2024-09-20T22:47:20.334 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:44:08.230723+0000", 2024-09-20T22:47:20.334 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 192.054621887, 2024-09-20T22:47:20.334 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.94983577399999997, 2024-09-20T22:47:20.334 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:47:20.334 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:47:20.334 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-20T22:47:20.334 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:47:20.334 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.14478", 2024-09-20T22:47:20.335 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/2994273271", 2024-09-20T22:47:20.335 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1259 2024-09-20T22:47:20.335 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.335 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:47:20.335 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.335 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:47:20.335 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:08.230723+0000", 2024-09-20T22:47:20.335 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:47:20.335 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.335 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.335 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:47:20.335 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:08.230723+0000", 2024-09-20T22:47:20.335 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:47:20.336 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.336 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.336 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:47:20.336 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:08.230724+0000", 2024-09-20T22:47:20.336 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.077e-06 2024-09-20T22:47:20.336 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.336 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.336 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:47:20.336 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:08.230729+0000", 2024-09-20T22:47:20.336 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.9930000000000003e-06 2024-09-20T22:47:20.336 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.336 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.336 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:47:20.336 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:08.230741+0000", 2024-09-20T22:47:20.336 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1897e-05 2024-09-20T22:47:20.337 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.337 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.337 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:47:20.337 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:08.230747+0000", 2024-09-20T22:47:20.337 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.9830000000000001e-06 2024-09-20T22:47:20.337 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.337 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.337 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:47:20.337 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:08.230802+0000", 2024-09-20T22:47:20.337 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.4657000000000002e-05 2024-09-20T22:47:20.337 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.337 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.337 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-09-20T22:47:20.337 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:08.230811+0000", 2024-09-20T22:47:20.338 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.1160000000000005e-06 2024-09-20T22:47:20.338 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.338 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.338 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:47:20.338 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:09.171968+0000", 2024-09-20T22:47:20.338 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.94115747299999997 2024-09-20T22:47:20.338 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.338 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.338 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for missing object", 2024-09-20T22:47:20.338 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:09.172098+0000", 2024-09-20T22:47:20.338 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00012937499999999999 2024-09-20T22:47:20.338 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.339 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.339 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:47:20.339 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:09.174248+0000", 2024-09-20T22:47:20.339 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0021508959999999998 2024-09-20T22:47:20.339 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.339 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.339 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for rw locks", 2024-09-20T22:47:20.339 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:09.174341+0000", 2024-09-20T22:47:20.339 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.2906999999999997e-05 2024-09-20T22:47:20.339 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.340 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.340 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:47:20.340 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:09.175083+0000", 2024-09-20T22:47:20.340 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00074120100000000003 2024-09-20T22:47:20.340 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.340 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.340 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:47:20.340 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:09.175149+0000", 2024-09-20T22:47:20.340 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.6877000000000004e-05 2024-09-20T22:47:20.340 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.341 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.341 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:47:20.341 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:09.180559+0000", 2024-09-20T22:47:20.341 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0054093220000000003 2024-09-20T22:47:20.341 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:47:20.341 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:47:20.341 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:47:20.341 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.341 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.341 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:1417 7.1 7:84a866f5:::smithi08477203-27:head [append 0~0] snapc 2f=[2f] ondisk+write+known_if_redirected+supports_pool_eio e189)", 2024-09-20T22:47:20.342 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:44:28.137465+0000", 2024-09-20T22:47:20.342 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 172.147879528, 2024-09-20T22:47:20.343 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.020090858999999999, 2024-09-20T22:47:20.343 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:47:20.343 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:47:20.343 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:47:20.343 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:47:20.343 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:47:20.343 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:47:20.343 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1417 2024-09-20T22:47:20.343 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.343 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:47:20.343 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.343 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:47:20.343 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.137465+0000", 2024-09-20T22:47:20.344 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:47:20.344 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.344 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.344 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:47:20.344 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.137465+0000", 2024-09-20T22:47:20.344 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:47:20.344 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.344 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.344 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:47:20.344 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.137468+0000", 2024-09-20T22:47:20.344 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.7259999999999998e-06 2024-09-20T22:47:20.344 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.344 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.344 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:47:20.345 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.137481+0000", 2024-09-20T22:47:20.345 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2937e-05 2024-09-20T22:47:20.345 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.345 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.345 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:47:20.345 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.137528+0000", 2024-09-20T22:47:20.345 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.7011000000000001e-05 2024-09-20T22:47:20.345 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.345 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.345 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:47:20.345 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.137542+0000", 2024-09-20T22:47:20.345 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3529e-05 2024-09-20T22:47:20.345 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.346 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.346 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:47:20.346 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.137636+0000", 2024-09-20T22:47:20.346 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.4932999999999994e-05 2024-09-20T22:47:20.346 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.346 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.346 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op must wait for map", 2024-09-20T22:47:20.346 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.137645+0000", 2024-09-20T22:47:20.346 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.3180000000000007e-06 2024-09-20T22:47:20.346 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.346 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.346 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:47:20.346 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.143542+0000", 2024-09-20T22:47:20.346 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0058974459999999998 2024-09-20T22:47:20.347 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.347 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.347 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:47:20.347 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.143647+0000", 2024-09-20T22:47:20.347 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000105028 2024-09-20T22:47:20.347 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.347 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.347 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 1,6", 2024-09-20T22:47:20.347 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.143835+0000", 2024-09-20T22:47:20.347 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00018802599999999999 2024-09-20T22:47:20.347 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.347 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.347 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:47:20.347 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.155596+0000", 2024-09-20T22:47:20.347 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011760708 2024-09-20T22:47:20.348 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.348 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.348 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:47:20.348 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.155674+0000", 2024-09-20T22:47:20.348 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.7883e-05 2024-09-20T22:47:20.348 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.348 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.348 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:47:20.348 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.157440+0000", 2024-09-20T22:47:20.348 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001766014 2024-09-20T22:47:20.348 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.348 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.348 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:47:20.348 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.157505+0000", 2024-09-20T22:47:20.349 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.5252999999999993e-05 2024-09-20T22:47:20.349 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.349 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.349 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:47:20.349 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.157556+0000", 2024-09-20T22:47:20.349 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.1047000000000001e-05 2024-09-20T22:47:20.349 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:47:20.349 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:47:20.349 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:47:20.349 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.349 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.349 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:1418 7.1 7:84a866f5:::smithi08477203-27:head [rollback 2f] snapc 2f=[2f] ondisk+write+known_if_redirected+supports_pool_eio e189)", 2024-09-20T22:47:20.349 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:44:28.137587+0000", 2024-09-20T22:47:20.349 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 172.14775768999999, 2024-09-20T22:47:20.350 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.020550452, 2024-09-20T22:47:20.350 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:47:20.350 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:47:20.350 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:47:20.350 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:47:20.350 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:47:20.350 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:47:20.350 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1418 2024-09-20T22:47:20.350 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.350 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:47:20.350 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.350 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:47:20.350 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.137587+0000", 2024-09-20T22:47:20.350 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:47:20.351 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.351 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.351 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:47:20.351 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.137587+0000", 2024-09-20T22:47:20.351 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:47:20.351 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.351 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.351 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:47:20.351 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.137588+0000", 2024-09-20T22:47:20.351 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.4700000000000001e-07 2024-09-20T22:47:20.351 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.351 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.351 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:47:20.351 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.137591+0000", 2024-09-20T22:47:20.352 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.0000000000000001e-06 2024-09-20T22:47:20.352 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.352 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.352 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:47:20.352 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.137681+0000", 2024-09-20T22:47:20.352 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.0486000000000001e-05 2024-09-20T22:47:20.352 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.352 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.352 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:47:20.352 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.138018+0000", 2024-09-20T22:47:20.352 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00033709899999999999 2024-09-20T22:47:20.352 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.352 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.352 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:47:20.353 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.138109+0000", 2024-09-20T22:47:20.353 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.0538999999999995e-05 2024-09-20T22:47:20.353 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.353 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.353 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting_for_map not empty", 2024-09-20T22:47:20.353 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.138114+0000", 2024-09-20T22:47:20.353 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.6590000000000004e-06 2024-09-20T22:47:20.353 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.353 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.353 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:47:20.353 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.144398+0000", 2024-09-20T22:47:20.353 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0062832599999999997 2024-09-20T22:47:20.353 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.353 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.354 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:47:20.354 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.144483+0000", 2024-09-20T22:47:20.354 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.4772999999999996e-05 2024-09-20T22:47:20.354 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.354 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.354 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 1,6", 2024-09-20T22:47:20.354 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.144661+0000", 2024-09-20T22:47:20.354 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000178736 2024-09-20T22:47:20.354 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.354 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.354 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:47:20.354 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.155602+0000", 2024-09-20T22:47:20.354 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010940302000000001 2024-09-20T22:47:20.354 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.354 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.355 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:47:20.355 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.156162+0000", 2024-09-20T22:47:20.355 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00056014599999999997 2024-09-20T22:47:20.355 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.355 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.355 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:47:20.355 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.158054+0000", 2024-09-20T22:47:20.355 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00189223 2024-09-20T22:47:20.355 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.355 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.355 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:47:20.355 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.158100+0000", 2024-09-20T22:47:20.355 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.6279999999999997e-05 2024-09-20T22:47:20.355 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.356 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.356 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:47:20.356 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.158138+0000", 2024-09-20T22:47:20.356 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.7394999999999997e-05 2024-09-20T22:47:20.356 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:47:20.356 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:47:20.356 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:47:20.356 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.356 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.356 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:1419 7.1 7:84a866f5:::smithi08477203-27:head [append 0~0] snapc 2f=[2f] ondisk+write+known_if_redirected+supports_pool_eio e189)", 2024-09-20T22:47:20.356 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:44:28.138055+0000", 2024-09-20T22:47:20.356 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 172.14729024799999, 2024-09-20T22:47:20.356 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.030290338999999999, 2024-09-20T22:47:20.357 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:47:20.357 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:47:20.357 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:47:20.357 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:47:20.357 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:47:20.357 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:47:20.357 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1419 2024-09-20T22:47:20.357 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.357 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:47:20.357 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.357 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:47:20.357 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.138055+0000", 2024-09-20T22:47:20.357 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:47:20.357 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.358 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.358 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:47:20.358 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.138055+0000", 2024-09-20T22:47:20.358 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:47:20.358 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.358 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.358 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:47:20.358 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.138056+0000", 2024-09-20T22:47:20.358 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.13e-06 2024-09-20T22:47:20.358 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.358 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.358 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:47:20.358 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.138060+0000", 2024-09-20T22:47:20.359 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.0269999999999997e-06 2024-09-20T22:47:20.359 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.359 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.359 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:47:20.359 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.138070+0000", 2024-09-20T22:47:20.359 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.961e-06 2024-09-20T22:47:20.359 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.359 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.359 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:47:20.359 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.138076+0000", 2024-09-20T22:47:20.359 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.1800000000000001e-06 2024-09-20T22:47:20.359 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.359 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.360 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:47:20.360 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.138137+0000", 2024-09-20T22:47:20.360 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.1159000000000002e-05 2024-09-20T22:47:20.360 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.360 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.360 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting_for_map not empty", 2024-09-20T22:47:20.360 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.138141+0000", 2024-09-20T22:47:20.360 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.189e-06 2024-09-20T22:47:20.360 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.360 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.360 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:47:20.360 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.154982+0000", 2024-09-20T22:47:20.360 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.016840738000000001 2024-09-20T22:47:20.360 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.360 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.361 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:47:20.361 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.155089+0000", 2024-09-20T22:47:20.361 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00010653399999999999 2024-09-20T22:47:20.361 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.361 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.361 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 1,6", 2024-09-20T22:47:20.361 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.155320+0000", 2024-09-20T22:47:20.361 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00023196299999999999 2024-09-20T22:47:20.361 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.361 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.361 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:47:20.361 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.158224+0000", 2024-09-20T22:47:20.361 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00290361 2024-09-20T22:47:20.361 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.362 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.362 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:47:20.362 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.167686+0000", 2024-09-20T22:47:20.362 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0094622360000000006 2024-09-20T22:47:20.362 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.362 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.362 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:47:20.362 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.168183+0000", 2024-09-20T22:47:20.362 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00049655999999999995 2024-09-20T22:47:20.362 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.362 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.362 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:47:20.362 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.168273+0000", 2024-09-20T22:47:20.362 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.9758000000000004e-05 2024-09-20T22:47:20.363 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.363 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.363 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:47:20.363 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.168345+0000", 2024-09-20T22:47:20.363 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.2293999999999996e-05 2024-09-20T22:47:20.363 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:47:20.363 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:47:20.363 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:47:20.363 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.363 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.363 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:2347 7.a 7:5122292a:::smithi08477203-33:head [sparse-read 0~2590699,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e260)", 2024-09-20T22:47:20.363 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:45:39.196440+0000", 2024-09-20T22:47:20.364 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 101.08890512000001, 2024-09-20T22:47:20.364 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.026701163, 2024-09-20T22:47:20.364 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:47:20.364 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:47:20.364 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-20T22:47:20.364 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:47:20.364 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:47:20.366 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:47:20.366 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:47:20.366 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 2347 2024-09-20T22:47:20.366 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.366 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:47:20.366 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.366 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:47:20.366 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.196440+0000", 2024-09-20T22:47:20.366 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:47:20.366 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.366 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.367 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:47:20.367 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.196440+0000", 2024-09-20T22:47:20.367 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:47:20.367 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.367 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.367 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:47:20.367 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.196441+0000", 2024-09-20T22:47:20.367 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.714e-06 2024-09-20T22:47:20.367 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.367 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.367 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:47:20.367 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.196449+0000", 2024-09-20T22:47:20.367 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.8730000000000007e-06 2024-09-20T22:47:20.367 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.368 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.368 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:47:20.368 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.196462+0000", 2024-09-20T22:47:20.368 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2649e-05 2024-09-20T22:47:20.368 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.368 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.368 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:47:20.368 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.196470+0000", 2024-09-20T22:47:20.368 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.4989999999999994e-06 2024-09-20T22:47:20.368 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.368 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.368 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:47:20.368 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.212172+0000", 2024-09-20T22:47:20.368 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.015701335 2024-09-20T22:47:20.369 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.369 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.369 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:47:20.369 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.212270+0000", 2024-09-20T22:47:20.369 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.8257999999999993e-05 2024-09-20T22:47:20.369 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.369 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.369 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:47:20.369 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.223141+0000", 2024-09-20T22:47:20.369 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010870835000000001 2024-09-20T22:47:20.369 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:47:20.369 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:47:20.369 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:47:20.369 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.370 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.370 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:2348 7.a 7:5122292a:::smithi08477203-33:head [read 0~3848803,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e260)", 2024-09-20T22:47:20.370 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:45:39.196485+0000", 2024-09-20T22:47:20.370 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 101.08885964700001, 2024-09-20T22:47:20.370 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0284703, 2024-09-20T22:47:20.370 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:47:20.370 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:47:20.370 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-20T22:47:20.370 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:47:20.370 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:47:20.370 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:47:20.370 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 2348 2024-09-20T22:47:20.370 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.370 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:47:20.371 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.371 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:47:20.371 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.196485+0000", 2024-09-20T22:47:20.371 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:47:20.371 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.371 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.371 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:47:20.371 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.196485+0000", 2024-09-20T22:47:20.371 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:47:20.371 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.371 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.371 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:47:20.371 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.196486+0000", 2024-09-20T22:47:20.371 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.9599999999999999e-07 2024-09-20T22:47:20.371 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.372 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.372 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:47:20.372 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.196489+0000", 2024-09-20T22:47:20.372 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.5470000000000001e-06 2024-09-20T22:47:20.372 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.372 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.372 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:47:20.372 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.196496+0000", 2024-09-20T22:47:20.372 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.3570000000000003e-06 2024-09-20T22:47:20.372 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.372 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.372 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:47:20.372 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.196501+0000", 2024-09-20T22:47:20.373 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.6239999999999999e-06 2024-09-20T22:47:20.373 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.373 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.373 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:47:20.373 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.223176+0000", 2024-09-20T22:47:20.373 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.026675285999999999 2024-09-20T22:47:20.373 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.373 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.373 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:47:20.373 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.223274+0000", 2024-09-20T22:47:20.373 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.7009999999999994e-05 2024-09-20T22:47:20.373 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.373 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.373 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:47:20.374 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.224955+0000", 2024-09-20T22:47:20.374 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00168198 2024-09-20T22:47:20.374 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:47:20.374 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:47:20.374 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:47:20.374 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.374 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.374 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:2349 7.a 7:5122292a:::smithi08477203-33:head [sparse-read 0~2590699] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e260)", 2024-09-20T22:47:20.374 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:45:39.196508+0000", 2024-09-20T22:47:20.374 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 101.08883636100001, 2024-09-20T22:47:20.374 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.028731293000000001, 2024-09-20T22:47:20.374 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:47:20.374 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:47:20.374 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-20T22:47:20.374 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:47:20.375 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:47:20.375 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:47:20.375 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 2349 2024-09-20T22:47:20.375 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.375 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:47:20.375 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.375 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:47:20.375 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.196508+0000", 2024-09-20T22:47:20.375 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:47:20.375 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.375 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.375 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:47:20.375 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.196508+0000", 2024-09-20T22:47:20.375 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:47:20.375 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.376 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.376 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:47:20.376 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.196509+0000", 2024-09-20T22:47:20.376 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.08e-07 2024-09-20T22:47:20.376 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.376 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.376 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:47:20.376 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.196511+0000", 2024-09-20T22:47:20.376 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.142e-06 2024-09-20T22:47:20.376 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.376 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.376 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:47:20.376 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.196517+0000", 2024-09-20T22:47:20.376 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.6330000000000001e-06 2024-09-20T22:47:20.377 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.377 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.377 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:47:20.377 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.196522+0000", 2024-09-20T22:47:20.377 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.3530000000000002e-06 2024-09-20T22:47:20.377 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.377 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.377 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:47:20.377 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.224990+0000", 2024-09-20T22:47:20.377 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.028467864999999998 2024-09-20T22:47:20.377 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.377 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.377 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:47:20.377 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.225088+0000", 2024-09-20T22:47:20.377 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.8281000000000003e-05 2024-09-20T22:47:20.378 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.378 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.378 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:47:20.378 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.225240+0000", 2024-09-20T22:47:20.378 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00015161100000000001 2024-09-20T22:47:20.378 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:47:20.378 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:47:20.378 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:47:20.378 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.378 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.378 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(osd.6.0:727 7.c e260/94 7:318c7512:::smithi08477203-44:4c v 260'428, mlcod=258'424)", 2024-09-20T22:47:20.378 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:45:39.202413+0000", 2024-09-20T22:47:20.378 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 101.08293182200001, 2024-09-20T22:47:20.378 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.022852897, 2024-09-20T22:47:20.378 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:47:20.379 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:47:20.379 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:47:20.379 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:47:20.379 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.379 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:47:20.379 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.202413+0000", 2024-09-20T22:47:20.379 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:47:20.379 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.379 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.379 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:47:20.379 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.202413+0000", 2024-09-20T22:47:20.379 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:47:20.379 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.379 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.379 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:47:20.380 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.202414+0000", 2024-09-20T22:47:20.380 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.9299999999999997e-07 2024-09-20T22:47:20.380 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.380 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.380 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:47:20.380 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.202421+0000", 2024-09-20T22:47:20.380 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.8109999999999997e-06 2024-09-20T22:47:20.380 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.380 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.380 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:47:20.380 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.202433+0000", 2024-09-20T22:47:20.380 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2411e-05 2024-09-20T22:47:20.380 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.380 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.381 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:47:20.381 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.202442+0000", 2024-09-20T22:47:20.381 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.7269999999999996e-06 2024-09-20T22:47:20.381 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.381 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.381 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:47:20.381 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.221764+0000", 2024-09-20T22:47:20.381 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.019322068000000001 2024-09-20T22:47:20.381 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.381 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.381 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:47:20.381 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.221786+0000", 2024-09-20T22:47:20.381 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.2167e-05 2024-09-20T22:47:20.381 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.381 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.382 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:47:20.382 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.225217+0000", 2024-09-20T22:47:20.382 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0034313709999999999 2024-09-20T22:47:20.382 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.382 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.382 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:47:20.382 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.225266+0000", 2024-09-20T22:47:20.382 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.8649000000000001e-05 2024-09-20T22:47:20.382 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:47:20.382 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:47:20.382 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:47:20.382 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.382 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.382 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(osd.6.0:728 7.c e260/94 7:31d0211f:::smithi08477203-25:51 v 260'430, mlcod=258'424)", 2024-09-20T22:47:20.382 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:45:39.202454+0000", 2024-09-20T22:47:20.383 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 101.082890959, 2024-09-20T22:47:20.383 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.022837705, 2024-09-20T22:47:20.383 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:47:20.383 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:47:20.383 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:47:20.383 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:47:20.383 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.383 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:47:20.383 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.202454+0000", 2024-09-20T22:47:20.383 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:47:20.383 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.383 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.383 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:47:20.383 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.202454+0000", 2024-09-20T22:47:20.383 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:47:20.384 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.384 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.384 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:47:20.384 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.202454+0000", 2024-09-20T22:47:20.384 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.9999999999999999e-07 2024-09-20T22:47:20.384 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.384 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.384 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:47:20.384 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.202464+0000", 2024-09-20T22:47:20.384 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.8179999999999999e-06 2024-09-20T22:47:20.384 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.384 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.384 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:47:20.384 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.202469+0000", 2024-09-20T22:47:20.385 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.1089999999999997e-06 2024-09-20T22:47:20.385 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.385 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.385 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:47:20.385 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.202473+0000", 2024-09-20T22:47:20.385 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.0940000000000001e-06 2024-09-20T22:47:20.385 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.385 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.385 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:47:20.385 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.222250+0000", 2024-09-20T22:47:20.385 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.019776353999999999 2024-09-20T22:47:20.385 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.385 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.385 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:47:20.386 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.222268+0000", 2024-09-20T22:47:20.386 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.8342000000000002e-05 2024-09-20T22:47:20.386 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.386 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.386 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:47:20.386 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.225268+0000", 2024-09-20T22:47:20.386 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00300038 2024-09-20T22:47:20.386 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.386 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.386 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:47:20.386 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.225292+0000", 2024-09-20T22:47:20.386 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.3308000000000001e-05 2024-09-20T22:47:20.386 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:47:20.386 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:47:20.387 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:47:20.387 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.387 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.387 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:2525 7.1 7:8422f82d:::smithi08477203-22:head [copy-from ver 174] snapc 5d=[5d,5c,5a,50,47] ondisk+write+known_if_redirected+supports_pool_eio e278)", 2024-09-20T22:47:20.387 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:45:57.170348+0000", 2024-09-20T22:47:20.387 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 83.114996848000004, 2024-09-20T22:47:20.387 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.023333486000000001, 2024-09-20T22:47:20.387 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:47:20.387 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:47:20.387 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:47:20.387 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:47:20.387 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:47:20.387 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:47:20.387 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 2525 2024-09-20T22:47:20.388 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.388 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:47:20.388 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.388 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:47:20.388 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:57.170348+0000", 2024-09-20T22:47:20.388 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:47:20.388 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.388 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.388 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:47:20.388 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:57.170348+0000", 2024-09-20T22:47:20.388 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:47:20.388 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.388 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.388 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:47:20.388 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:57.170350+0000", 2024-09-20T22:47:20.389 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.1509999999999998e-06 2024-09-20T22:47:20.389 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.389 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.389 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:47:20.389 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:57.170361+0000", 2024-09-20T22:47:20.389 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0492999999999999e-05 2024-09-20T22:47:20.389 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.389 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.389 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:47:20.389 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:57.170381+0000", 2024-09-20T22:47:20.389 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.0509999999999998e-05 2024-09-20T22:47:20.389 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.389 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.389 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:47:20.390 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:57.170393+0000", 2024-09-20T22:47:20.390 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2009e-05 2024-09-20T22:47:20.390 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.390 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.390 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:47:20.390 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:57.170465+0000", 2024-09-20T22:47:20.390 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.1927000000000004e-05 2024-09-20T22:47:20.390 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.390 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.390 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op must wait for map", 2024-09-20T22:47:20.390 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:57.170473+0000", 2024-09-20T22:47:20.390 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.1890000000000007e-06 2024-09-20T22:47:20.390 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.391 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.391 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:47:20.391 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:57.172855+0000", 2024-09-20T22:47:20.391 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002382168 2024-09-20T22:47:20.391 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.391 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.391 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:47:20.391 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:57.172967+0000", 2024-09-20T22:47:20.391 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00011108299999999999 2024-09-20T22:47:20.391 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.391 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.391 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 1,6", 2024-09-20T22:47:20.391 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:57.181533+0000", 2024-09-20T22:47:20.391 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0085667499999999997 2024-09-20T22:47:20.391 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.392 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.392 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:47:20.392 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:57.191599+0000", 2024-09-20T22:47:20.392 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010065795000000001 2024-09-20T22:47:20.392 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.392 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.392 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:47:20.392 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:57.193469+0000", 2024-09-20T22:47:20.392 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0018695199999999999 2024-09-20T22:47:20.392 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.392 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.392 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:47:20.392 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:57.193550+0000", 2024-09-20T22:47:20.392 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.1891000000000006e-05 2024-09-20T22:47:20.392 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.393 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.393 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:47:20.393 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:57.193625+0000", 2024-09-20T22:47:20.393 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.4679999999999996e-05 2024-09-20T22:47:20.393 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.393 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.393 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:47:20.393 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:57.193681+0000", 2024-09-20T22:47:20.393 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.6320000000000003e-05 2024-09-20T22:47:20.393 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:47:20.393 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:47:20.393 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:47:20.393 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.393 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.393 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:2853 7.10 7:0f524219:::smithi08477203-50:head [copy-from ver 114] snapc 68=[68,67] ondisk+write+known_if_redirected+supports_pool_eio e303)", 2024-09-20T22:47:20.394 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:46:23.285507+0000", 2024-09-20T22:47:20.394 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 56.999838119000003, 2024-09-20T22:47:20.394 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.023440631, 2024-09-20T22:47:20.394 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:47:20.394 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:47:20.394 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:47:20.394 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:47:20.394 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:47:20.394 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:47:20.394 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 2853 2024-09-20T22:47:20.394 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.394 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:47:20.394 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.394 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:47:20.395 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.285507+0000", 2024-09-20T22:47:20.395 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:47:20.395 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.395 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.395 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:47:20.395 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.285507+0000", 2024-09-20T22:47:20.395 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:47:20.395 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.395 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.395 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:47:20.395 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.285508+0000", 2024-09-20T22:47:20.395 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.491e-06 2024-09-20T22:47:20.395 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.395 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.395 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:47:20.396 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.285514+0000", 2024-09-20T22:47:20.396 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.502e-06 2024-09-20T22:47:20.396 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.396 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.396 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:47:20.396 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.285526+0000", 2024-09-20T22:47:20.396 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2639e-05 2024-09-20T22:47:20.396 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.396 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.396 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:47:20.396 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.285535+0000", 2024-09-20T22:47:20.396 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.4689999999999997e-06 2024-09-20T22:47:20.396 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.396 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.397 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:47:20.397 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.285591+0000", 2024-09-20T22:47:20.397 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.6407999999999999e-05 2024-09-20T22:47:20.397 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.397 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.397 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:47:20.397 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.285681+0000", 2024-09-20T22:47:20.397 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.9845000000000005e-05 2024-09-20T22:47:20.397 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.397 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.397 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 1,2", 2024-09-20T22:47:20.397 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.293719+0000", 2024-09-20T22:47:20.397 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0080383950000000003 2024-09-20T22:47:20.397 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.397 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.398 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:47:20.398 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.307398+0000", 2024-09-20T22:47:20.398 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013678074 2024-09-20T22:47:20.398 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.398 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.398 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:47:20.398 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.308451+0000", 2024-09-20T22:47:20.398 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001052975 2024-09-20T22:47:20.398 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.398 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.398 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:47:20.398 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.308811+0000", 2024-09-20T22:47:20.398 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00036041399999999998 2024-09-20T22:47:20.398 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.398 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.398 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:47:20.399 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.308878+0000", 2024-09-20T22:47:20.399 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.7114999999999997e-05 2024-09-20T22:47:20.399 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.399 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.399 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:47:20.399 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.308947+0000", 2024-09-20T22:47:20.399 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.9303999999999999e-05 2024-09-20T22:47:20.399 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:47:20.399 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:47:20.399 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:47:20.399 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.399 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.399 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:3136 7.a 7:5122292a:::smithi08477203-33:head [copy-from ver 385] snapc 77=[77,76,75,74,73,70,6b] ondisk+write+known_if_redirected+supports_pool_eio e328)", 2024-09-20T22:47:20.399 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:46:48.210350+0000", 2024-09-20T22:47:20.400 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 32.074994988, 2024-09-20T22:47:20.400 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.030150626999999999, 2024-09-20T22:47:20.400 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:47:20.400 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:47:20.400 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:47:20.400 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:47:20.400 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:47:20.400 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:47:20.400 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 3136 2024-09-20T22:47:20.400 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.400 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:47:20.400 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.400 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:47:20.400 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.210350+0000", 2024-09-20T22:47:20.400 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:47:20.401 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.401 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.401 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:47:20.401 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.210350+0000", 2024-09-20T22:47:20.401 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:47:20.401 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.401 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.401 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:47:20.401 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.210352+0000", 2024-09-20T22:47:20.401 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.889e-06 2024-09-20T22:47:20.401 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.401 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.401 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:47:20.401 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.210360+0000", 2024-09-20T22:47:20.402 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.8890000000000002e-06 2024-09-20T22:47:20.402 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.402 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.402 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:47:20.402 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.210372+0000", 2024-09-20T22:47:20.402 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2662e-05 2024-09-20T22:47:20.402 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.402 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.402 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:47:20.402 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.210381+0000", 2024-09-20T22:47:20.402 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.8589999999999994e-06 2024-09-20T22:47:20.402 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.402 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.403 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:47:20.403 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.210470+0000", 2024-09-20T22:47:20.403 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.9034000000000002e-05 2024-09-20T22:47:20.403 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.403 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.403 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:47:20.403 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.210568+0000", 2024-09-20T22:47:20.403 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.7919999999999998e-05 2024-09-20T22:47:20.403 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.403 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.403 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 2,5", 2024-09-20T22:47:20.403 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.218491+0000", 2024-09-20T22:47:20.403 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0079229490000000003 2024-09-20T22:47:20.403 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.403 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.404 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:47:20.404 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.224053+0000", 2024-09-20T22:47:20.404 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0055624330000000003 2024-09-20T22:47:20.404 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.404 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.404 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:47:20.404 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.227143+0000", 2024-09-20T22:47:20.404 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0030896360000000002 2024-09-20T22:47:20.404 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.404 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.404 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:47:20.404 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.240355+0000", 2024-09-20T22:47:20.404 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013211624999999999 2024-09-20T22:47:20.404 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.405 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.405 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:47:20.405 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.240439+0000", 2024-09-20T22:47:20.405 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.4009000000000002e-05 2024-09-20T22:47:20.405 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.405 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.405 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:47:20.405 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.240500+0000", 2024-09-20T22:47:20.405 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.1722000000000003e-05 2024-09-20T22:47:20.405 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:47:20.405 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:47:20.405 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:47:20.405 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.405 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.406 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:3235 7.1 7:84a866f5:::smithi08477203-27:head [copy-from ver 114] snapc 7e=[7e,7d,7c,7b,7a,79,78,75,74,73,6b] ondisk+write+known_if_redirected+supports_pool_eio e338)", 2024-09-20T22:47:20.406 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:46:58.180892+0000", 2024-09-20T22:47:20.406 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 22.104452765000001, 2024-09-20T22:47:20.406 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.044574093000000002, 2024-09-20T22:47:20.406 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:47:20.406 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:47:20.406 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:47:20.406 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:47:20.406 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:47:20.406 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:47:20.406 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 3235 2024-09-20T22:47:20.406 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.406 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:47:20.406 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.407 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:47:20.407 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.180892+0000", 2024-09-20T22:47:20.407 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:47:20.407 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.407 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.407 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:47:20.407 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.180892+0000", 2024-09-20T22:47:20.407 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:47:20.407 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.407 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.407 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:47:20.407 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.180894+0000", 2024-09-20T22:47:20.407 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.1789999999999998e-06 2024-09-20T22:47:20.407 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.408 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.408 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:47:20.408 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.180915+0000", 2024-09-20T22:47:20.408 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.0859000000000001e-05 2024-09-20T22:47:20.408 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.408 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.408 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:47:20.408 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.180937+0000", 2024-09-20T22:47:20.408 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.2357999999999998e-05 2024-09-20T22:47:20.408 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.408 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.408 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:47:20.408 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.180947+0000", 2024-09-20T22:47:20.408 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.8090000000000004e-06 2024-09-20T22:47:20.408 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.409 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.409 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:47:20.409 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.181042+0000", 2024-09-20T22:47:20.409 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.4327999999999995e-05 2024-09-20T22:47:20.409 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.409 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.409 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op must wait for map", 2024-09-20T22:47:20.409 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.181049+0000", 2024-09-20T22:47:20.409 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.1640000000000004e-06 2024-09-20T22:47:20.409 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.409 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.409 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:47:20.409 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.185933+0000", 2024-09-20T22:47:20.409 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.004884645 2024-09-20T22:47:20.410 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.410 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.410 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:47:20.410 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.186177+0000", 2024-09-20T22:47:20.410 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000243107 2024-09-20T22:47:20.410 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.410 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.410 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 1,6", 2024-09-20T22:47:20.410 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.204218+0000", 2024-09-20T22:47:20.410 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.018041805000000001 2024-09-20T22:47:20.410 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.410 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.411 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:47:20.411 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.218275+0000", 2024-09-20T22:47:20.411 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014056528 2024-09-20T22:47:20.411 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.411 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.411 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:47:20.411 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.224283+0000", 2024-09-20T22:47:20.411 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0060084860000000004 2024-09-20T22:47:20.411 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.411 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.411 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:47:20.411 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.225379+0000", 2024-09-20T22:47:20.411 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010960830000000001 2024-09-20T22:47:20.411 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.412 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.412 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:47:20.412 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.225429+0000", 2024-09-20T22:47:20.412 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.9440000000000001e-05 2024-09-20T22:47:20.412 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.412 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.412 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:47:20.412 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.225466+0000", 2024-09-20T22:47:20.412 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.7302000000000003e-05 2024-09-20T22:47:20.412 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:47:20.412 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:47:20.412 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:47:20.412 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.412 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.412 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:3236 7.1 7:84a866f5:::smithi08477203-27:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e338)", 2024-09-20T22:47:20.413 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:46:58.180973+0000", 2024-09-20T22:47:20.413 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 22.104372142999999, 2024-09-20T22:47:20.413 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.028358540000000002, 2024-09-20T22:47:20.413 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:47:20.413 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:47:20.413 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-20T22:47:20.413 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:47:20.413 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:47:20.413 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:47:20.413 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 3236 2024-09-20T22:47:20.413 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.413 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:47:20.413 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.413 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:47:20.413 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.180973+0000", 2024-09-20T22:47:20.414 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:47:20.414 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.414 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.414 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:47:20.414 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.180973+0000", 2024-09-20T22:47:20.414 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:47:20.414 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.414 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.414 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:47:20.414 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.180973+0000", 2024-09-20T22:47:20.414 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.9100000000000004e-07 2024-09-20T22:47:20.414 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.414 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.415 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:47:20.415 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.180976+0000", 2024-09-20T22:47:20.415 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.118e-06 2024-09-20T22:47:20.415 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.415 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.415 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:47:20.415 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.180984+0000", 2024-09-20T22:47:20.415 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.43e-06 2024-09-20T22:47:20.415 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.415 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.415 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:47:20.415 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.180990+0000", 2024-09-20T22:47:20.415 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.1109999999999996e-06 2024-09-20T22:47:20.415 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.415 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.416 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:47:20.416 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.181102+0000", 2024-09-20T22:47:20.416 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000112193 2024-09-20T22:47:20.416 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.416 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.416 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting_for_map not empty", 2024-09-20T22:47:20.416 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.181107+0000", 2024-09-20T22:47:20.416 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.0490000000000003e-06 2024-09-20T22:47:20.416 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.416 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.416 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:47:20.416 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.186302+0000", 2024-09-20T22:47:20.416 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0051950240000000003 2024-09-20T22:47:20.417 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.417 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.417 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for blocked object", 2024-09-20T22:47:20.417 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.186344+0000", 2024-09-20T22:47:20.417 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.2168e-05 2024-09-20T22:47:20.417 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.417 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.417 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:47:20.417 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.209115+0000", 2024-09-20T22:47:20.417 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.022770291000000002 2024-09-20T22:47:20.417 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.417 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.418 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:47:20.418 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.209217+0000", 2024-09-20T22:47:20.418 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000102192 2024-09-20T22:47:20.418 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:47:20.418 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:47:20.418 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:47:20.418 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.209331+0000", 2024-09-20T22:47:20.418 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000114373 2024-09-20T22:47:20.418 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:47:20.418 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:47:20.418 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:47:20.418 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:47:20.419 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:47:20.419 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-20T22:47:20.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:20.420 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:20.420 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:47:20.521 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:47:20.544 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:20.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:20.545 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:47:20.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:20 smithi084 ceph-mon[40024]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:47:20.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:20 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/891822532' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:47:20.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:20 smithi084 ceph-mon[40024]: 6.1d scrub starts 2024-09-20T22:47:20.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:20 smithi084 ceph-mon[40024]: 6.1d scrub ok 2024-09-20T22:47:20.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:20 smithi084 ceph-mon[40024]: 5.18 scrub starts 2024-09-20T22:47:20.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:20 smithi084 ceph-mon[40024]: 5.18 scrub ok 2024-09-20T22:47:20.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:20 smithi084 ceph-mon[40024]: 6.17 scrub starts 2024-09-20T22:47:20.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:20 smithi084 ceph-mon[40024]: 6.17 scrub ok 2024-09-20T22:47:20.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:20 smithi084 ceph-mon[40024]: osdmap e360: 8 total, 8 up, 6 in 2024-09-20T22:47:20.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:20 smithi084 ceph-mon[29774]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:47:20.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:20 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/891822532' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:47:20.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:20 smithi084 ceph-mon[29774]: 6.1d scrub starts 2024-09-20T22:47:20.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:20 smithi084 ceph-mon[29774]: 6.1d scrub ok 2024-09-20T22:47:20.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:20 smithi084 ceph-mon[29774]: 5.18 scrub starts 2024-09-20T22:47:20.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:20 smithi084 ceph-mon[29774]: 5.18 scrub ok 2024-09-20T22:47:20.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:20 smithi084 ceph-mon[29774]: 6.17 scrub starts 2024-09-20T22:47:20.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:20 smithi084 ceph-mon[29774]: 6.17 scrub ok 2024-09-20T22:47:20.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:20 smithi084 ceph-mon[29774]: osdmap e360: 8 total, 8 up, 6 in 2024-09-20T22:47:20.646 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:47:20.669 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:20.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:20.670 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:47:20.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:20 smithi154 ceph-mon[33264]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:47:20.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:20 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/891822532' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:47:20.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:20 smithi154 ceph-mon[33264]: 6.1d scrub starts 2024-09-20T22:47:20.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:20 smithi154 ceph-mon[33264]: 6.1d scrub ok 2024-09-20T22:47:20.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:20 smithi154 ceph-mon[33264]: 5.18 scrub starts 2024-09-20T22:47:20.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:20 smithi154 ceph-mon[33264]: 5.18 scrub ok 2024-09-20T22:47:20.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:20 smithi154 ceph-mon[33264]: 6.17 scrub starts 2024-09-20T22:47:20.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:20 smithi154 ceph-mon[33264]: 6.17 scrub ok 2024-09-20T22:47:20.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:20 smithi154 ceph-mon[33264]: osdmap e360: 8 total, 8 up, 6 in 2024-09-20T22:47:20.770 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:47:20.792 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:20.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:20.793 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:47:20.894 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:47:20.917 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:20.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:20.918 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:47:21.019 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:47:21.042 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:21.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:21.043 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:47:21.144 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:47:21.168 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:21.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:21.169 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:47:21.191 INFO:tasks.rados.rados.0.smithi084.stdout:1361: done (3 left) 2024-09-20T22:47:21.191 INFO:tasks.rados.rados.0.smithi084.stdout:1362: done (2 left) 2024-09-20T22:47:21.191 INFO:tasks.rados.rados.0.smithi084.stdout:1363: done (1 left) 2024-09-20T22:47:21.191 INFO:tasks.rados.rados.0.smithi084.stdout:1364: done (0 left) 2024-09-20T22:47:21.191 INFO:tasks.rados.rados.0.smithi084.stdout:1365: delete oid 34 current snap is 138 2024-09-20T22:47:21.195 INFO:tasks.rados.rados.0.smithi084.stdout:1365: done (0 left) 2024-09-20T22:47:21.195 INFO:tasks.rados.rados.0.smithi084.stdout:1366: rollback oid 43 current snap is 138 2024-09-20T22:47:21.195 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 43 to 120 2024-09-20T22:47:21.195 INFO:tasks.rados.rados.0.smithi084.stdout:1367: delete oid 1 current snap is 138 2024-09-20T22:47:21.199 INFO:tasks.rados.rados.0.smithi084.stdout:1366: finishing rollback tid 1 to smithi08477203-43 2024-09-20T22:47:21.199 INFO:tasks.rados.rados.0.smithi084.stdout:1366: finishing rollback tid 2 to smithi08477203-43 2024-09-20T22:47:21.199 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 43 v 61 (ObjNum 418 snap 105 seq_num 418) dirty exists 2024-09-20T22:47:21.200 INFO:tasks.rados.rados.0.smithi084.stdout:1366: done (1 left) 2024-09-20T22:47:21.200 INFO:tasks.rados.rados.0.smithi084.stdout:1367: done (0 left) 2024-09-20T22:47:21.200 INFO:tasks.rados.rados.0.smithi084.stdout:1368: write (excl) oid 30 current snap is 138 2024-09-20T22:47:21.200 INFO:tasks.rados.rados.0.smithi084.stdout:1368: seq_num 492 ranges {669044=469100,1741023=459161} 2024-09-20T22:47:21.206 INFO:tasks.rados.rados.0.smithi084.stdout:1368: writing smithi08477203-30 from 669044 to 1138144 tid 1 2024-09-20T22:47:21.212 INFO:tasks.rados.rados.0.smithi084.stdout:1368: writing smithi08477203-30 from 1741023 to 2200184 tid 2 2024-09-20T22:47:21.212 INFO:tasks.rados.rados.0.smithi084.stdout:1369: copy_from oid 3 from oid 13 current snap is 138 2024-09-20T22:47:21.213 INFO:tasks.rados.rados.0.smithi084.stdout:1368: finishing write tid 1 to smithi08477203-30 2024-09-20T22:47:21.213 INFO:tasks.rados.rados.0.smithi084.stdout:1370: write oid 49 current snap is 138 2024-09-20T22:47:21.213 INFO:tasks.rados.rados.0.smithi084.stdout:1370: seq_num 493 ranges {685851=544349,1799505=486346} 2024-09-20T22:47:21.219 INFO:tasks.rados.rados.0.smithi084.stdout:1370: writing smithi08477203-49 from 685851 to 1230200 tid 1 2024-09-20T22:47:21.225 INFO:tasks.rados.rados.0.smithi084.stdout:1370: writing smithi08477203-49 from 1799505 to 2285851 tid 2 2024-09-20T22:47:21.225 INFO:tasks.rados.rados.0.smithi084.stdout:1371: write oid 2 current snap is 138 2024-09-20T22:47:21.225 INFO:tasks.rados.rados.0.smithi084.stdout:1369: finishing copy_from to smithi08477203-3 2024-09-20T22:47:21.225 INFO:tasks.rados.rados.0.smithi084.stdout:1369: got expected ENOENT (src dne) 2024-09-20T22:47:21.225 INFO:tasks.rados.rados.0.smithi084.stdout:1369: finishing copy_from racing read to smithi08477203-3 2024-09-20T22:47:21.225 INFO:tasks.rados.rados.0.smithi084.stdout:1368: finishing write tid 2 to smithi08477203-30 2024-09-20T22:47:21.225 INFO:tasks.rados.rados.0.smithi084.stdout:1371: seq_num 494 ranges {702658=619598,1857987=444671} 2024-09-20T22:47:21.232 INFO:tasks.rados.rados.0.smithi084.stdout:1371: writing smithi08477203-2 from 702658 to 1322256 tid 1 2024-09-20T22:47:21.238 INFO:tasks.rados.rados.0.smithi084.stdout:1371: writing smithi08477203-2 from 1857987 to 2302658 tid 2 2024-09-20T22:47:21.238 INFO:tasks.rados.rados.0.smithi084.stdout:1369: done (3 left) 2024-09-20T22:47:21.238 INFO:tasks.rados.rados.0.smithi084.stdout:1372: snap_create 2024-09-20T22:47:21.238 INFO:tasks.rados.rados.0.smithi084.stdout:1368: finishing write tid 3 to smithi08477203-30 2024-09-20T22:47:21.238 INFO:tasks.rados.rados.0.smithi084.stdout:1368: finishing write tid 4 to smithi08477203-30 2024-09-20T22:47:21.239 INFO:tasks.rados.rados.0.smithi084.stdout:1368: oid 30 updating version 0 to 399 2024-09-20T22:47:21.239 INFO:tasks.rados.rados.0.smithi084.stdout:1368: oid 30 version 399 is already newer than 398 2024-09-20T22:47:21.239 INFO:tasks.rados.rados.0.smithi084.stdout:1368: oid 30 version 399 is already newer than 397 2024-09-20T22:47:21.239 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 30 v 399 (ObjNum 492 snap 138 seq_num 492) dirty exists 2024-09-20T22:47:21.239 INFO:tasks.rados.rados.0.smithi084.stdout:1368: left oid 30 (ObjNum 492 snap 138 seq_num 492) 2024-09-20T22:47:21.239 INFO:tasks.rados.rados.0.smithi084.stdout:1370: finishing write tid 1 to smithi08477203-49 2024-09-20T22:47:21.239 INFO:tasks.rados.rados.0.smithi084.stdout:1370: finishing write tid 2 to smithi08477203-49 2024-09-20T22:47:21.239 INFO:tasks.rados.rados.0.smithi084.stdout:1370: finishing write tid 3 to smithi08477203-49 2024-09-20T22:47:21.239 INFO:tasks.rados.rados.0.smithi084.stdout:1370: finishing write tid 4 to smithi08477203-49 2024-09-20T22:47:21.239 INFO:tasks.rados.rados.0.smithi084.stdout:1370: oid 49 updating version 0 to 513 2024-09-20T22:47:21.239 INFO:tasks.rados.rados.0.smithi084.stdout:1370: oid 49 version 513 is already newer than 512 2024-09-20T22:47:21.239 INFO:tasks.rados.rados.0.smithi084.stdout:1370: oid 49 version 513 is already newer than 511 2024-09-20T22:47:21.239 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 49 v 513 (ObjNum 493 snap 138 seq_num 493) dirty exists 2024-09-20T22:47:21.239 INFO:tasks.rados.rados.0.smithi084.stdout:1370: left oid 49 (ObjNum 493 snap 138 seq_num 493) 2024-09-20T22:47:21.239 INFO:tasks.rados.rados.0.smithi084.stdout:1371: finishing write tid 1 to smithi08477203-2 2024-09-20T22:47:21.241 INFO:tasks.rados.rados.0.smithi084.stdout:1371: finishing write tid 2 to smithi08477203-2 2024-09-20T22:47:21.241 INFO:tasks.rados.rados.0.smithi084.stdout:1371: finishing write tid 3 to smithi08477203-2 2024-09-20T22:47:21.241 INFO:tasks.rados.rados.0.smithi084.stdout:1371: finishing write tid 4 to smithi08477203-2 2024-09-20T22:47:21.241 INFO:tasks.rados.rados.0.smithi084.stdout:1371: oid 2 updating version 0 to 157 2024-09-20T22:47:21.241 INFO:tasks.rados.rados.0.smithi084.stdout:1371: oid 2 version 157 is already newer than 156 2024-09-20T22:47:21.241 INFO:tasks.rados.rados.0.smithi084.stdout:1371: oid 2 updating version 157 to 158 2024-09-20T22:47:21.241 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 2 v 158 (ObjNum 494 snap 138 seq_num 494) dirty exists 2024-09-20T22:47:21.241 INFO:tasks.rados.rados.0.smithi084.stdout:1371: left oid 2 (ObjNum 494 snap 138 seq_num 494) 2024-09-20T22:47:21.268 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-20T22:47:21.270 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:47:21.329 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:21.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:21.330 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:47:21.431 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:47:21.454 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:21.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:21.455 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:47:21.520 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:47:21.555 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:47:21.577 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:21.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:21.578 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:47:21.586 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:21 smithi084 ceph-mon[40024]: pgmap v441: 168 pgs: 6 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 156 active+clean; 251 MiB data, 801 MiB used, 536 GiB / 536 GiB avail; 22 MiB/s rd, 3.3 MiB/s wr, 24 op/s 2024-09-20T22:47:21.586 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:21 smithi084 ceph-mon[40024]: 5.e scrub starts 2024-09-20T22:47:21.586 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:21 smithi084 ceph-mon[40024]: 5.e scrub ok 2024-09-20T22:47:21.586 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:21 smithi084 ceph-mon[40024]: osdmap e361: 8 total, 8 up, 6 in 2024-09-20T22:47:21.586 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:21 smithi084 ceph-mon[29774]: pgmap v441: 168 pgs: 6 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 156 active+clean; 251 MiB data, 801 MiB used, 536 GiB / 536 GiB avail; 22 MiB/s rd, 3.3 MiB/s wr, 24 op/s 2024-09-20T22:47:21.586 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:21 smithi084 ceph-mon[29774]: 5.e scrub starts 2024-09-20T22:47:21.586 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:21 smithi084 ceph-mon[29774]: 5.e scrub ok 2024-09-20T22:47:21.586 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:21 smithi084 ceph-mon[29774]: osdmap e361: 8 total, 8 up, 6 in 2024-09-20T22:47:21.641 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:21 smithi154 ceph-mon[33264]: pgmap v441: 168 pgs: 6 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 156 active+clean; 251 MiB data, 801 MiB used, 536 GiB / 536 GiB avail; 22 MiB/s rd, 3.3 MiB/s wr, 24 op/s 2024-09-20T22:47:21.641 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:21 smithi154 ceph-mon[33264]: 5.e scrub starts 2024-09-20T22:47:21.641 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:21 smithi154 ceph-mon[33264]: 5.e scrub ok 2024-09-20T22:47:21.641 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:21 smithi154 ceph-mon[33264]: osdmap e361: 8 total, 8 up, 6 in 2024-09-20T22:47:21.679 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:47:21.700 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:21.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:21.701 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:47:21.802 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:47:21.824 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:21.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:21.826 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:47:21.927 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:47:21.949 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:21.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:21.951 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:47:22.009 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-20T22:47:22.009 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [], 2024-09-20T22:47:22.009 INFO:teuthology.orchestra.run.smithi154.stdout: "num_ops": 0 2024-09-20T22:47:22.009 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-20T22:47:22.052 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:47:22.073 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:22.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:22.074 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:47:22.175 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:47:22.194 INFO:tasks.rados.rados.0.smithi084.stdout:1368: done (3 left) 2024-09-20T22:47:22.194 INFO:tasks.rados.rados.0.smithi084.stdout:1370: done (2 left) 2024-09-20T22:47:22.194 INFO:tasks.rados.rados.0.smithi084.stdout:1371: done (1 left) 2024-09-20T22:47:22.194 INFO:tasks.rados.rados.0.smithi084.stdout:1372: done (0 left) 2024-09-20T22:47:22.195 INFO:tasks.rados.rados.0.smithi084.stdout:1373: rollback oid 27 current snap is 139 2024-09-20T22:47:22.195 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 27 to 114 2024-09-20T22:47:22.195 INFO:tasks.rados.rados.0.smithi084.stdout:1374: read oid 44 snap -1 2024-09-20T22:47:22.195 INFO:tasks.rados.rados.0.smithi084.stdout:1374: expect (ObjNum 470 snap 130 seq_num 470) 2024-09-20T22:47:22.195 INFO:tasks.rados.rados.0.smithi084.stdout:1375: write (excl) oid 49 current snap is 139 2024-09-20T22:47:22.195 INFO:tasks.rados.rados.0.smithi084.stdout:1375: seq_num 495 ranges {719465=411200,1632822=686643} 2024-09-20T22:47:22.197 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:22.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:22.198 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:47:22.200 INFO:tasks.rados.rados.0.smithi084.stdout:1375: writing smithi08477203-49 from 719465 to 1130665 tid 1 2024-09-20T22:47:22.207 INFO:tasks.rados.rados.0.smithi084.stdout:1375: writing smithi08477203-49 from 1632822 to 2319465 tid 2 2024-09-20T22:47:22.207 INFO:tasks.rados.rados.0.smithi084.stdout:1376: read oid 10 snap 125 2024-09-20T22:47:22.207 INFO:tasks.rados.rados.0.smithi084.stdout:1376: expect (ObjNum 419 snap 105 seq_num 419) 2024-09-20T22:47:22.208 INFO:tasks.rados.rados.0.smithi084.stdout:1377: write oid 11 current snap is 139 2024-09-20T22:47:22.208 INFO:tasks.rados.rados.0.smithi084.stdout:1373: finishing rollback tid 0 to smithi08477203-27 2024-09-20T22:47:22.208 INFO:tasks.rados.rados.0.smithi084.stdout:1373: finishing rollback tid 1 to smithi08477203-27 2024-09-20T22:47:22.208 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 27 v 323 (ObjNum 803183840 snap 32761 seq_num 304) dirty dne 2024-09-20T22:47:22.208 INFO:tasks.rados.rados.0.smithi084.stdout:1377: seq_num 496 ranges {736272=486449,1974951=361321} 2024-09-20T22:47:22.214 INFO:tasks.rados.rados.0.smithi084.stdout:1377: writing smithi08477203-11 from 736272 to 1222721 tid 1 2024-09-20T22:47:22.220 INFO:tasks.rados.rados.0.smithi084.stdout:1377: writing smithi08477203-11 from 1974951 to 2336272 tid 2 2024-09-20T22:47:22.220 INFO:tasks.rados.rados.0.smithi084.stdout:1373: done (4 left) 2024-09-20T22:47:22.220 INFO:tasks.rados.rados.0.smithi084.stdout:1378: copy_from oid 34 from oid 22 current snap is 139 2024-09-20T22:47:22.220 INFO:tasks.rados.rados.0.smithi084.stdout:1374: expect (ObjNum 470 snap 130 seq_num 470) 2024-09-20T22:47:22.299 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:47:22.321 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:22.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:22.322 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:47:22.345 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:22 smithi084 ceph-mon[29774]: 4.c scrub starts 2024-09-20T22:47:22.345 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:22 smithi084 ceph-mon[29774]: 4.c scrub ok 2024-09-20T22:47:22.345 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:22 smithi084 ceph-mon[29774]: 5.7 scrub starts 2024-09-20T22:47:22.345 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:22 smithi084 ceph-mon[29774]: 5.7 scrub ok 2024-09-20T22:47:22.345 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:22 smithi084 ceph-mon[29774]: 6.1e scrub starts 2024-09-20T22:47:22.345 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:22 smithi084 ceph-mon[29774]: 6.1e scrub ok 2024-09-20T22:47:22.345 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:22 smithi084 ceph-mon[29774]: osdmap e362: 8 total, 8 up, 6 in 2024-09-20T22:47:22.345 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:22 smithi084 ceph-mon[40024]: 4.c scrub starts 2024-09-20T22:47:22.345 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:22 smithi084 ceph-mon[40024]: 4.c scrub ok 2024-09-20T22:47:22.346 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:22 smithi084 ceph-mon[40024]: 5.7 scrub starts 2024-09-20T22:47:22.346 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:22 smithi084 ceph-mon[40024]: 5.7 scrub ok 2024-09-20T22:47:22.346 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:22 smithi084 ceph-mon[40024]: 6.1e scrub starts 2024-09-20T22:47:22.346 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:22 smithi084 ceph-mon[40024]: 6.1e scrub ok 2024-09-20T22:47:22.346 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:22 smithi084 ceph-mon[40024]: osdmap e362: 8 total, 8 up, 6 in 2024-09-20T22:47:22.354 INFO:tasks.rados.rados.0.smithi084.stdout:1374: done (4 left) 2024-09-20T22:47:22.354 INFO:tasks.rados.rados.0.smithi084.stdout:1379: copy_from oid 47 from oid 45 current snap is 139 2024-09-20T22:47:22.354 INFO:tasks.rados.rados.0.smithi084.stdout:1375: finishing write tid 1 to smithi08477203-49 2024-09-20T22:47:22.354 INFO:tasks.rados.rados.0.smithi084.stdout:1375: finishing write tid 2 to smithi08477203-49 2024-09-20T22:47:22.354 INFO:tasks.rados.rados.0.smithi084.stdout:1375: finishing write tid 3 to smithi08477203-49 2024-09-20T22:47:22.354 INFO:tasks.rados.rados.0.smithi084.stdout:1375: finishing write tid 4 to smithi08477203-49 2024-09-20T22:47:22.354 INFO:tasks.rados.rados.0.smithi084.stdout:1375: oid 49 updating version 0 to 515 2024-09-20T22:47:22.354 INFO:tasks.rados.rados.0.smithi084.stdout:1375: oid 49 updating version 515 to 516 2024-09-20T22:47:22.354 INFO:tasks.rados.rados.0.smithi084.stdout:1375: oid 49 version 516 is already newer than 514 2024-09-20T22:47:22.355 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 49 v 516 (ObjNum 495 snap 139 seq_num 495) dirty exists 2024-09-20T22:47:22.355 INFO:tasks.rados.rados.0.smithi084.stdout:1375: left oid 49 (ObjNum 495 snap 139 seq_num 495) 2024-09-20T22:47:22.355 INFO:tasks.rados.rados.0.smithi084.stdout:1376: expect (ObjNum 419 snap 105 seq_num 419) 2024-09-20T22:47:22.389 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:47:22.422 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:47:22.444 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:22.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:22.445 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:47:22.530 INFO:tasks.rados.rados.0.smithi084.stdout:1377: finishing write tid 1 to smithi08477203-11 2024-09-20T22:47:22.530 INFO:tasks.rados.rados.0.smithi084.stdout:1377: finishing write tid 2 to smithi08477203-11 2024-09-20T22:47:22.530 INFO:tasks.rados.rados.0.smithi084.stdout:1377: finishing write tid 3 to smithi08477203-11 2024-09-20T22:47:22.530 INFO:tasks.rados.rados.0.smithi084.stdout:1377: finishing write tid 4 to smithi08477203-11 2024-09-20T22:47:22.530 INFO:tasks.rados.rados.0.smithi084.stdout:1377: oid 11 updating version 0 to 678 2024-09-20T22:47:22.530 INFO:tasks.rados.rados.0.smithi084.stdout:1377: oid 11 version 678 is already newer than 676 2024-09-20T22:47:22.530 INFO:tasks.rados.rados.0.smithi084.stdout:1377: oid 11 version 678 is already newer than 677 2024-09-20T22:47:22.530 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 11 v 678 (ObjNum 496 snap 139 seq_num 496) dirty exists 2024-09-20T22:47:22.530 INFO:tasks.rados.rados.0.smithi084.stdout:1377: left oid 11 (ObjNum 496 snap 139 seq_num 496) 2024-09-20T22:47:22.531 INFO:tasks.rados.rados.0.smithi084.stdout:1378: finishing copy_from racing read to smithi08477203-34 2024-09-20T22:47:22.531 INFO:tasks.rados.rados.0.smithi084.stdout:1378: finishing copy_from to smithi08477203-34 2024-09-20T22:47:22.531 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 34 v 321 (ObjNum 468 snap 129 seq_num 468) dirty exists 2024-09-20T22:47:22.531 INFO:tasks.rados.rados.0.smithi084.stdout:1375: done (4 left) 2024-09-20T22:47:22.531 INFO:tasks.rados.rados.0.smithi084.stdout:1376: done (3 left) 2024-09-20T22:47:22.531 INFO:tasks.rados.rados.0.smithi084.stdout:1377: done (2 left) 2024-09-20T22:47:22.531 INFO:tasks.rados.rados.0.smithi084.stdout:1378: done (1 left) 2024-09-20T22:47:22.531 INFO:tasks.rados.rados.0.smithi084.stdout:1380: read oid 40 snap 137 2024-09-20T22:47:22.531 INFO:tasks.rados.rados.0.smithi084.stdout:1380: expect (ObjNum 483 snap 135 seq_num 483) 2024-09-20T22:47:22.531 INFO:tasks.rados.rados.0.smithi084.stdout:1381: read oid 19 snap -1 2024-09-20T22:47:22.531 INFO:tasks.rados.rados.0.smithi084.stdout:1381: expect deleted 2024-09-20T22:47:22.531 INFO:tasks.rados.rados.0.smithi084.stdout:1382: rollback oid 26 current snap is 139 2024-09-20T22:47:22.531 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 26 to 133 2024-09-20T22:47:22.531 INFO:tasks.rados.rados.0.smithi084.stdout:1379: finishing copy_from racing read to smithi08477203-47 2024-09-20T22:47:22.531 INFO:tasks.rados.rados.0.smithi084.stdout:1379: finishing copy_from to smithi08477203-47 2024-09-20T22:47:22.532 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 47 v 131 (ObjNum 466 snap 128 seq_num 466) dirty exists 2024-09-20T22:47:22.532 INFO:tasks.rados.rados.0.smithi084.stdout:1379: done (3 left) 2024-09-20T22:47:22.532 INFO:tasks.rados.rados.0.smithi084.stdout:1383: copy_from oid 36 from oid 41 current snap is 139 2024-09-20T22:47:22.532 INFO:tasks.rados.rados.0.smithi084.stdout:1384: read oid 24 snap -1 2024-09-20T22:47:22.532 INFO:tasks.rados.rados.0.smithi084.stdout:1384: expect (ObjNum 433 snap 112 seq_num 433) 2024-09-20T22:47:22.532 INFO:tasks.rados.rados.0.smithi084.stdout:1385: snap_create 2024-09-20T22:47:22.533 INFO:tasks.rados.rados.0.smithi084.stdout:1382: finishing rollback tid 0 to smithi08477203-26 2024-09-20T22:47:22.533 INFO:tasks.rados.rados.0.smithi084.stdout:1382: finishing rollback tid 1 to smithi08477203-26 2024-09-20T22:47:22.533 INFO:tasks.rados.rados.0.smithi084.stdout:1382: finishing rollback tid 2 to smithi08477203-26 2024-09-20T22:47:22.533 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 26 v 296 (ObjNum 390 snap 101 seq_num 390) dirty exists 2024-09-20T22:47:22.534 INFO:tasks.rados.rados.0.smithi084.stdout:1380: expect (ObjNum 483 snap 135 seq_num 483) 2024-09-20T22:47:22.546 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:47:22.566 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:22.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:22.567 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:47:22.666 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:22 smithi154 ceph-mon[33264]: 4.c scrub starts 2024-09-20T22:47:22.667 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:22 smithi154 ceph-mon[33264]: 4.c scrub ok 2024-09-20T22:47:22.667 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:22 smithi154 ceph-mon[33264]: 5.7 scrub starts 2024-09-20T22:47:22.667 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:22 smithi154 ceph-mon[33264]: 5.7 scrub ok 2024-09-20T22:47:22.667 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:22 smithi154 ceph-mon[33264]: 6.1e scrub starts 2024-09-20T22:47:22.667 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:22 smithi154 ceph-mon[33264]: 6.1e scrub ok 2024-09-20T22:47:22.667 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:22 smithi154 ceph-mon[33264]: osdmap e362: 8 total, 8 up, 6 in 2024-09-20T22:47:22.668 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:47:22.689 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:22.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:22.690 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:47:22.702 INFO:tasks.rados.rados.0.smithi084.stdout:1383: finishing copy_from racing read to smithi08477203-36 2024-09-20T22:47:22.702 INFO:tasks.rados.rados.0.smithi084.stdout:1384: expect (ObjNum 433 snap 112 seq_num 433) 2024-09-20T22:47:22.719 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.4 dump_blocked_ops 2024-09-20T22:47:22.791 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:47:22.812 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:22.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:22.813 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:47:22.813 INFO:tasks.rados.rados.0.smithi084.stdout:1383: finishing copy_from to smithi08477203-36 2024-09-20T22:47:22.813 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 36 v 100 (ObjNum 440 snap 115 seq_num 440) dirty exists 2024-09-20T22:47:22.914 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:47:22.937 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:22.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:22.938 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:47:22.972 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:47:23.039 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:47:23.060 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:23.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:23.061 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:47:23.161 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:47:23.183 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:23.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:23.184 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:47:23.185 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is set 2024-09-20T22:47:23.192 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.2 to 1 (10000) 2024-09-20T22:47:23.192 INFO:tasks.rados.rados.0.smithi084.stdout:1380: done (5 left) 2024-09-20T22:47:23.193 INFO:tasks.rados.rados.0.smithi084.stdout:1381: done (4 left) 2024-09-20T22:47:23.193 INFO:tasks.rados.rados.0.smithi084.stdout:1382: done (3 left) 2024-09-20T22:47:23.193 INFO:tasks.rados.rados.0.smithi084.stdout:1383: done (2 left) 2024-09-20T22:47:23.193 INFO:tasks.rados.rados.0.smithi084.stdout:1384: done (1 left) 2024-09-20T22:47:23.193 INFO:tasks.rados.rados.0.smithi084.stdout:1385: done (0 left) 2024-09-20T22:47:23.193 INFO:tasks.rados.rados.0.smithi084.stdout:1386: read oid 40 snap -1 2024-09-20T22:47:23.193 INFO:tasks.rados.rados.0.smithi084.stdout:1386: expect (ObjNum 483 snap 135 seq_num 483) 2024-09-20T22:47:23.193 INFO:tasks.rados.rados.0.smithi084.stdout:1387: copy_from oid 1 from oid 49 current snap is 140 2024-09-20T22:47:23.193 INFO:tasks.rados.rados.0.smithi084.stdout:1388: read oid 37 snap -1 2024-09-20T22:47:23.193 INFO:tasks.rados.rados.0.smithi084.stdout:1388: expect (ObjNum 478 snap 135 seq_num 478) 2024-09-20T22:47:23.193 INFO:tasks.rados.rados.0.smithi084.stdout:1389: rollback oid 39 current snap is 140 2024-09-20T22:47:23.193 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 39 to 126 2024-09-20T22:47:23.193 INFO:tasks.rados.rados.0.smithi084.stdout:1390: snap_create 2024-09-20T22:47:23.202 INFO:tasks.rados.rados.0.smithi084.stdout:1389: finishing rollback tid 0 to smithi08477203-39 2024-09-20T22:47:23.202 INFO:tasks.rados.rados.0.smithi084.stdout:1389: finishing rollback tid 1 to smithi08477203-39 2024-09-20T22:47:23.203 INFO:tasks.rados.rados.0.smithi084.stdout:1389: finishing rollback tid 2 to smithi08477203-39 2024-09-20T22:47:23.203 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 39 v 420 (ObjNum 397 snap 101 seq_num 397) dirty exists 2024-09-20T22:47:23.204 INFO:tasks.rados.rados.0.smithi084.stdout:1386: expect (ObjNum 483 snap 135 seq_num 483) 2024-09-20T22:47:23.285 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:47:23.307 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:23.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:23.308 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:47:23.375 INFO:tasks.rados.rados.0.smithi084.stdout:1388: expect (ObjNum 478 snap 135 seq_num 478) 2024-09-20T22:47:23.409 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:47:23.412 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:23 smithi154 ceph-mon[33264]: pgmap v444: 168 pgs: 6 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 156 active+clean; 251 MiB data, 801 MiB used, 536 GiB / 536 GiB avail; 22 MiB/s rd, 3.3 MiB/s wr, 24 op/s 2024-09-20T22:47:23.412 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:23 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/61573007' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-09-20T22:47:23.413 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:23 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-09-20T22:47:23.413 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:23 smithi154 ceph-mon[33264]: purged_snaps scrub starts 2024-09-20T22:47:23.413 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:23 smithi154 ceph-mon[33264]: purged_snaps scrub ok 2024-09-20T22:47:23.413 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:23 smithi154 ceph-mon[33264]: 5.6 deep-scrub starts 2024-09-20T22:47:23.413 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:23 smithi154 ceph-mon[33264]: 5.6 deep-scrub ok 2024-09-20T22:47:23.413 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:23 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:47:23.413 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:23 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:47:23.413 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:23 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/2184237591' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-20T22:47:23.413 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:23 smithi154 ceph-mon[33264]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:47:23.413 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:23 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-20T22:47:23.414 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:23 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/2184237591' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T22:47:23.414 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:23 smithi154 ceph-mon[33264]: osdmap e363: 8 total, 8 up, 6 in 2024-09-20T22:47:23.432 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:23.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:23.433 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:47:23.442 INFO:tasks.rados.rados.0.smithi084.stdout:1387: finishing copy_from racing read to smithi08477203-1 2024-09-20T22:47:23.442 INFO:tasks.rados.rados.0.smithi084.stdout:1387: finishing copy_from to smithi08477203-1 2024-09-20T22:47:23.442 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 1 v 516 (ObjNum 495 snap 139 seq_num 495) dirty exists 2024-09-20T22:47:23.520 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-20T22:47:23.520 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [], 2024-09-20T22:47:23.520 INFO:teuthology.orchestra.run.smithi154.stdout: "complaint_time": 30, 2024-09-20T22:47:23.521 INFO:teuthology.orchestra.run.smithi154.stdout: "num_blocked_ops": 0 2024-09-20T22:47:23.521 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-20T22:47:23.534 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:47:23.544 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:23 smithi084 ceph-mon[40024]: pgmap v444: 168 pgs: 6 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 156 active+clean; 251 MiB data, 801 MiB used, 536 GiB / 536 GiB avail; 22 MiB/s rd, 3.3 MiB/s wr, 24 op/s 2024-09-20T22:47:23.544 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:23 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/61573007' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-09-20T22:47:23.544 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:23 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-09-20T22:47:23.544 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:23 smithi084 ceph-mon[40024]: purged_snaps scrub starts 2024-09-20T22:47:23.545 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:23 smithi084 ceph-mon[40024]: purged_snaps scrub ok 2024-09-20T22:47:23.545 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:23 smithi084 ceph-mon[40024]: 5.6 deep-scrub starts 2024-09-20T22:47:23.545 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:23 smithi084 ceph-mon[40024]: 5.6 deep-scrub ok 2024-09-20T22:47:23.545 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:23 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:47:23.545 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:23 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:47:23.545 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:23 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/2184237591' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-20T22:47:23.545 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:23 smithi084 ceph-mon[40024]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:47:23.545 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:23 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-20T22:47:23.545 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:23 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/2184237591' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T22:47:23.545 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:23 smithi084 ceph-mon[40024]: osdmap e363: 8 total, 8 up, 6 in 2024-09-20T22:47:23.545 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:23 smithi084 ceph-mon[29774]: pgmap v444: 168 pgs: 6 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 156 active+clean; 251 MiB data, 801 MiB used, 536 GiB / 536 GiB avail; 22 MiB/s rd, 3.3 MiB/s wr, 24 op/s 2024-09-20T22:47:23.546 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:23 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/61573007' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-09-20T22:47:23.546 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:23 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-09-20T22:47:23.546 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:23 smithi084 ceph-mon[29774]: purged_snaps scrub starts 2024-09-20T22:47:23.546 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:23 smithi084 ceph-mon[29774]: purged_snaps scrub ok 2024-09-20T22:47:23.546 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:23 smithi084 ceph-mon[29774]: 5.6 deep-scrub starts 2024-09-20T22:47:23.546 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:23 smithi084 ceph-mon[29774]: 5.6 deep-scrub ok 2024-09-20T22:47:23.546 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:23 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:47:23.546 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:23 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:47:23.546 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:23 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/2184237591' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-20T22:47:23.546 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:23 smithi084 ceph-mon[29774]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:47:23.546 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:23 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-20T22:47:23.546 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:23 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/2184237591' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T22:47:23.546 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:23 smithi084 ceph-mon[29774]: osdmap e363: 8 total, 8 up, 6 in 2024-09-20T22:47:23.556 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:23.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:23.557 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:47:23.657 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:47:23.680 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:23.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:23.681 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:47:23.781 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:47:23.803 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:23.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:23.804 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:47:23.904 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:47:23.927 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:23.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:23.928 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:47:24.028 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:47:24.049 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:24.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:24.050 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:47:24.094 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-20T22:47:24.150 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:47:24.171 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:24.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:24.172 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:47:24.203 INFO:tasks.rados.rados.0.smithi084.stdout:1386: done (4 left) 2024-09-20T22:47:24.203 INFO:tasks.rados.rados.0.smithi084.stdout:1387: done (3 left) 2024-09-20T22:47:24.203 INFO:tasks.rados.rados.0.smithi084.stdout:1388: done (2 left) 2024-09-20T22:47:24.203 INFO:tasks.rados.rados.0.smithi084.stdout:1389: done (1 left) 2024-09-20T22:47:24.203 INFO:tasks.rados.rados.0.smithi084.stdout:1390: done (0 left) 2024-09-20T22:47:24.203 INFO:tasks.rados.rados.0.smithi084.stdout:1391: read oid 24 snap 134 2024-09-20T22:47:24.203 INFO:tasks.rados.rados.0.smithi084.stdout:1391: expect (ObjNum 433 snap 112 seq_num 433) 2024-09-20T22:47:24.203 INFO:tasks.rados.rados.0.smithi084.stdout:1392: snap_create 2024-09-20T22:47:24.209 INFO:tasks.rados.rados.0.smithi084.stdout:1391: expect (ObjNum 433 snap 112 seq_num 433) 2024-09-20T22:47:24.273 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:47:24.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:24.295 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:24.295 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:47:24.335 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.4 dump_historic_ops 2024-09-20T22:47:24.396 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:47:24.418 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:24.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:24.419 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:47:24.520 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:47:24.541 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:24.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:24.542 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:47:24.583 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:47:24.600 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:24 smithi084 ceph-mon[40024]: 1.0 deep-scrub starts 2024-09-20T22:47:24.600 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:24 smithi084 ceph-mon[40024]: 1.0 deep-scrub ok 2024-09-20T22:47:24.600 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:24 smithi084 ceph-mon[40024]: 7.3 scrub starts 2024-09-20T22:47:24.600 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:24 smithi084 ceph-mon[40024]: 7.3 scrub ok 2024-09-20T22:47:24.600 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:24 smithi084 ceph-mon[40024]: osdmap e364: 8 total, 8 up, 6 in 2024-09-20T22:47:24.601 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:24 smithi084 ceph-mon[29774]: 1.0 deep-scrub starts 2024-09-20T22:47:24.601 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:24 smithi084 ceph-mon[29774]: 1.0 deep-scrub ok 2024-09-20T22:47:24.601 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:24 smithi084 ceph-mon[29774]: 7.3 scrub starts 2024-09-20T22:47:24.601 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:24 smithi084 ceph-mon[29774]: 7.3 scrub ok 2024-09-20T22:47:24.601 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:24 smithi084 ceph-mon[29774]: osdmap e364: 8 total, 8 up, 6 in 2024-09-20T22:47:24.626 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd reweight 3 1 2024-09-20T22:47:24.642 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:47:24.664 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:24.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:24.665 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:47:24.700 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:47:24.704 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:24 smithi154 ceph-mon[33264]: 1.0 deep-scrub starts 2024-09-20T22:47:24.705 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:24 smithi154 ceph-mon[33264]: 1.0 deep-scrub ok 2024-09-20T22:47:24.705 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:24 smithi154 ceph-mon[33264]: 7.3 scrub starts 2024-09-20T22:47:24.705 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:24 smithi154 ceph-mon[33264]: 7.3 scrub ok 2024-09-20T22:47:24.705 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:24 smithi154 ceph-mon[33264]: osdmap e364: 8 total, 8 up, 6 in 2024-09-20T22:47:24.765 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:47:24.786 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:24.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:24.787 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:47:24.888 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:47:24.909 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:24.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:24.910 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:47:25.010 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:47:25.031 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:25.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:25.032 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:47:25.096 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-20T22:47:25.096 INFO:teuthology.orchestra.run.smithi154.stdout: "size": 20, 2024-09-20T22:47:25.096 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 600, 2024-09-20T22:47:25.096 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [ 2024-09-20T22:47:25.096 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.096 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.14478.0:53 5.e 5:7759931f:::notify.3:head [watch watch cookie 94843898686720] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2024-09-20T22:47:25.096 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:41:42.194704+0000", 2024-09-20T22:47:25.096 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 342.89865735299998, 2024-09-20T22:47:25.097 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0018746430000000001, 2024-09-20T22:47:25.097 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:47:25.097 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:47:25.097 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:47:25.097 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:47:25.097 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.14478", 2024-09-20T22:47:25.097 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/2994273271", 2024-09-20T22:47:25.097 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 53 2024-09-20T22:47:25.097 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.097 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:47:25.097 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.097 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:47:25.097 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:42.194704+0000", 2024-09-20T22:47:25.097 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:47:25.098 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.098 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.098 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:47:25.098 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:42.194704+0000", 2024-09-20T22:47:25.098 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:47:25.098 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.098 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.098 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:47:25.098 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:42.194705+0000", 2024-09-20T22:47:25.098 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.9800000000000003e-07 2024-09-20T22:47:25.098 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.098 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.098 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:47:25.098 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:42.194708+0000", 2024-09-20T22:47:25.099 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.2279999999999998e-06 2024-09-20T22:47:25.099 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.099 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.099 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:47:25.099 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:42.194715+0000", 2024-09-20T22:47:25.099 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.9299999999999997e-06 2024-09-20T22:47:25.099 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.099 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.099 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:47:25.099 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:42.194720+0000", 2024-09-20T22:47:25.099 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.8230000000000002e-06 2024-09-20T22:47:25.099 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.099 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.100 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:47:25.100 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:42.195053+0000", 2024-09-20T22:47:25.100 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000333312 2024-09-20T22:47:25.100 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.100 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.100 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for rw locks", 2024-09-20T22:47:25.100 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:42.195134+0000", 2024-09-20T22:47:25.100 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.1036000000000001e-05 2024-09-20T22:47:25.100 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.100 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.100 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:47:25.100 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:42.195679+0000", 2024-09-20T22:47:25.100 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00054494600000000004 2024-09-20T22:47:25.100 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.101 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.101 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:47:25.101 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:42.195732+0000", 2024-09-20T22:47:25.101 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.2605999999999997e-05 2024-09-20T22:47:25.101 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.101 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.101 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 0,5", 2024-09-20T22:47:25.101 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:42.195869+0000", 2024-09-20T22:47:25.101 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00013703899999999999 2024-09-20T22:47:25.101 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.101 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.101 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:47:25.101 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:42.196212+0000", 2024-09-20T22:47:25.101 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00034263300000000001 2024-09-20T22:47:25.102 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.102 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.102 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:47:25.102 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:42.196356+0000", 2024-09-20T22:47:25.102 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00014463799999999999 2024-09-20T22:47:25.102 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.102 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.102 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:47:25.102 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:42.196509+0000", 2024-09-20T22:47:25.102 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000152893 2024-09-20T22:47:25.102 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.102 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.102 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:47:25.102 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:42.196539+0000", 2024-09-20T22:47:25.103 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.9876e-05 2024-09-20T22:47:25.103 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.103 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.103 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:47:25.103 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:42.196579+0000", 2024-09-20T22:47:25.103 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.9984999999999997e-05 2024-09-20T22:47:25.103 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:47:25.103 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:47:25.103 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:47:25.103 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.103 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.103 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.14478.0:206 4.18 4:1e85421f:gc::gc.1:head [create,call version.check_conds in=18b,call rgw_gc.rgw_gc_queue_init in=23b,call version.set in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-09-20T22:47:25.103 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:41:45.190343+0000", 2024-09-20T22:47:25.103 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 339.903018874, 2024-09-20T22:47:25.103 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0022015450000000001, 2024-09-20T22:47:25.104 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:47:25.104 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:47:25.104 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:47:25.104 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:47:25.104 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.14478", 2024-09-20T22:47:25.104 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/2994273271", 2024-09-20T22:47:25.104 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 206 2024-09-20T22:47:25.104 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.104 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:47:25.104 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.104 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:47:25.104 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:45.190343+0000", 2024-09-20T22:47:25.104 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:47:25.104 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.105 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.105 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:47:25.105 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:45.190343+0000", 2024-09-20T22:47:25.105 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:47:25.105 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.105 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.105 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:47:25.105 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:45.190345+0000", 2024-09-20T22:47:25.105 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.843e-06 2024-09-20T22:47:25.105 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.105 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.105 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:47:25.105 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:45.190351+0000", 2024-09-20T22:47:25.105 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.7750000000000004e-06 2024-09-20T22:47:25.106 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.106 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.106 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:47:25.106 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:45.190365+0000", 2024-09-20T22:47:25.106 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4123e-05 2024-09-20T22:47:25.106 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.106 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.106 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:47:25.106 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:45.190373+0000", 2024-09-20T22:47:25.106 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.0199999999999997e-06 2024-09-20T22:47:25.106 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.106 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.106 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:47:25.107 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:45.190438+0000", 2024-09-20T22:47:25.107 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.5446999999999999e-05 2024-09-20T22:47:25.107 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.107 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.107 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:47:25.107 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:45.190562+0000", 2024-09-20T22:47:25.107 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00012417999999999999 2024-09-20T22:47:25.107 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.107 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.107 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 1,6", 2024-09-20T22:47:25.107 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:45.190826+0000", 2024-09-20T22:47:25.107 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00026341199999999998 2024-09-20T22:47:25.107 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.107 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.107 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:47:25.108 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:45.191210+0000", 2024-09-20T22:47:25.108 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00038417799999999999 2024-09-20T22:47:25.108 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.108 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.108 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:47:25.108 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:45.191549+0000", 2024-09-20T22:47:25.108 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00033966500000000001 2024-09-20T22:47:25.108 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.108 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.108 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:47:25.108 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:45.192471+0000", 2024-09-20T22:47:25.108 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00092185200000000002 2024-09-20T22:47:25.108 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.108 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.109 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:47:25.109 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:45.192520+0000", 2024-09-20T22:47:25.109 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.8600999999999998e-05 2024-09-20T22:47:25.109 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.109 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.109 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:47:25.109 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:45.192544+0000", 2024-09-20T22:47:25.109 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.4448999999999999e-05 2024-09-20T22:47:25.109 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:47:25.109 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:47:25.109 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:47:25.109 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.109 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.109 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(osd.0.0:116 7.9 e106/94 7:9eac4560:::smithi08477203-30:5 v 106'27, mlcod=105'24)", 2024-09-20T22:47:25.110 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:05.532754+0000", 2024-09-20T22:47:25.110 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 259.56060788799999, 2024-09-20T22:47:25.110 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0023975530000000002, 2024-09-20T22:47:25.110 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:47:25.110 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:47:25.110 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:47:25.110 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:47:25.110 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.110 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:47:25.110 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:05.532754+0000", 2024-09-20T22:47:25.110 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:47:25.110 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.110 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.110 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:47:25.111 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:05.532754+0000", 2024-09-20T22:47:25.111 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:47:25.111 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.111 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.111 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:47:25.111 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:05.532754+0000", 2024-09-20T22:47:25.111 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.4999999999999999e-07 2024-09-20T22:47:25.111 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.111 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.111 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:47:25.111 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:05.532758+0000", 2024-09-20T22:47:25.111 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.0389999999999998e-06 2024-09-20T22:47:25.111 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.111 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.111 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:47:25.112 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:05.532776+0000", 2024-09-20T22:47:25.112 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.8045000000000001e-05 2024-09-20T22:47:25.112 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.112 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.112 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:47:25.112 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:05.532783+0000", 2024-09-20T22:47:25.112 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.3300000000000001e-06 2024-09-20T22:47:25.112 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.112 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.112 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:47:25.112 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:05.532851+0000", 2024-09-20T22:47:25.112 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.7939000000000001e-05 2024-09-20T22:47:25.112 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.112 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.113 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:47:25.113 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:05.532865+0000", 2024-09-20T22:47:25.113 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3416e-05 2024-09-20T22:47:25.113 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.113 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.113 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:47:25.113 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:05.535106+0000", 2024-09-20T22:47:25.113 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0022412159999999999 2024-09-20T22:47:25.113 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.113 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.113 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:47:25.113 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:05.535151+0000", 2024-09-20T22:47:25.113 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.5318000000000002e-05 2024-09-20T22:47:25.113 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:47:25.113 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:47:25.114 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:47:25.114 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.114 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.114 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:324 7.8 e106/94 7:12e9c8db:::smithi08477203-11:head v 106'10, mlcod=98'8)", 2024-09-20T22:47:25.114 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:05.548487+0000", 2024-09-20T22:47:25.114 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 259.54487487699998, 2024-09-20T22:47:25.114 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.002619522, 2024-09-20T22:47:25.114 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:47:25.114 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:47:25.114 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:47:25.114 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:47:25.114 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.114 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:47:25.114 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:05.548487+0000", 2024-09-20T22:47:25.115 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:47:25.115 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.115 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.115 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:47:25.115 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:05.548487+0000", 2024-09-20T22:47:25.115 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:47:25.115 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.115 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.115 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:47:25.115 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:05.548487+0000", 2024-09-20T22:47:25.115 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.7300000000000001e-07 2024-09-20T22:47:25.115 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.115 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.115 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:47:25.115 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:05.550209+0000", 2024-09-20T22:47:25.116 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0017213370000000001 2024-09-20T22:47:25.116 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.116 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.116 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:47:25.116 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:05.550218+0000", 2024-09-20T22:47:25.116 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.9189999999999994e-06 2024-09-20T22:47:25.116 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.116 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.116 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:47:25.116 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:05.550224+0000", 2024-09-20T22:47:25.116 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.5260000000000001e-06 2024-09-20T22:47:25.116 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.116 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.116 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:47:25.117 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:05.550267+0000", 2024-09-20T22:47:25.117 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.3231e-05 2024-09-20T22:47:25.117 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.117 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.117 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:47:25.117 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:05.550283+0000", 2024-09-20T22:47:25.117 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.5421999999999999e-05 2024-09-20T22:47:25.117 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.117 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.117 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:47:25.117 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:05.551070+0000", 2024-09-20T22:47:25.117 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00078768700000000002 2024-09-20T22:47:25.117 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.117 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.117 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:47:25.118 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:05.551106+0000", 2024-09-20T22:47:25.118 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.5926999999999997e-05 2024-09-20T22:47:25.118 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:47:25.118 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:47:25.118 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:47:25.118 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.118 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.118 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:325 7.8 e106/94 7:12e9c8db:::smithi08477203-11:head v 106'11, mlcod=98'9)", 2024-09-20T22:47:25.118 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:05.554854+0000", 2024-09-20T22:47:25.118 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 259.53850713600002, 2024-09-20T22:47:25.118 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001914212, 2024-09-20T22:47:25.118 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:47:25.118 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:47:25.118 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:47:25.119 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:47:25.119 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.119 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:47:25.119 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:05.554854+0000", 2024-09-20T22:47:25.119 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:47:25.119 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.119 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.119 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:47:25.119 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:05.554854+0000", 2024-09-20T22:47:25.119 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:47:25.119 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.119 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.119 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:47:25.119 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:05.554855+0000", 2024-09-20T22:47:25.120 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.3300000000000001e-07 2024-09-20T22:47:25.120 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.120 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.120 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:47:25.120 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:05.555828+0000", 2024-09-20T22:47:25.120 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00097240100000000002 2024-09-20T22:47:25.120 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.120 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.120 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:47:25.120 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:05.555840+0000", 2024-09-20T22:47:25.120 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2711000000000001e-05 2024-09-20T22:47:25.120 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.120 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.121 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:47:25.121 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:05.555847+0000", 2024-09-20T22:47:25.121 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.4119999999999998e-06 2024-09-20T22:47:25.121 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.121 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.121 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:47:25.121 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:05.555898+0000", 2024-09-20T22:47:25.121 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.1107999999999999e-05 2024-09-20T22:47:25.121 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.121 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.121 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:47:25.121 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:05.555914+0000", 2024-09-20T22:47:25.121 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.5977999999999998e-05 2024-09-20T22:47:25.121 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.122 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.122 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:47:25.122 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:05.556720+0000", 2024-09-20T22:47:25.122 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00080616200000000002 2024-09-20T22:47:25.122 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.122 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.122 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:47:25.122 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:05.556769+0000", 2024-09-20T22:47:25.122 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.8606999999999997e-05 2024-09-20T22:47:25.122 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:47:25.122 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:47:25.122 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:47:25.122 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.122 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.123 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "MOSDScrubReserve(5.11,QREQUEST,e:107,from:6,reservation_nonce:2)", 2024-09-20T22:47:25.123 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:06.817932+0000", 2024-09-20T22:47:25.123 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 258.275429927, 2024-09-20T22:47:25.123 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0020011370000000001, 2024-09-20T22:47:25.123 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:47:25.123 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:47:25.123 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:47:25.123 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:47:25.123 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.123 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:47:25.123 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:06.817932+0000", 2024-09-20T22:47:25.123 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:47:25.123 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.124 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.124 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:47:25.124 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:06.817932+0000", 2024-09-20T22:47:25.125 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:47:25.125 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.125 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.125 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:47:25.125 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:06.817933+0000", 2024-09-20T22:47:25.125 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.251e-06 2024-09-20T22:47:25.125 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.125 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.125 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:47:25.125 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:06.817939+0000", 2024-09-20T22:47:25.125 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.2879999999999998e-06 2024-09-20T22:47:25.125 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.125 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.125 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:47:25.126 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:06.817953+0000", 2024-09-20T22:47:25.126 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3757999999999999e-05 2024-09-20T22:47:25.126 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.126 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.126 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:47:25.126 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:06.817968+0000", 2024-09-20T22:47:25.126 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.5184000000000001e-05 2024-09-20T22:47:25.126 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.126 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.126 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:47:25.126 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:06.818023+0000", 2024-09-20T22:47:25.126 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.4833999999999997e-05 2024-09-20T22:47:25.126 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.127 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.127 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op must wait for map", 2024-09-20T22:47:25.127 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:06.818029+0000", 2024-09-20T22:47:25.127 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.7899999999999996e-06 2024-09-20T22:47:25.127 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.127 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.127 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:47:25.127 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:06.819889+0000", 2024-09-20T22:47:25.127 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001860163 2024-09-20T22:47:25.127 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.127 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.127 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:47:25.127 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:06.819899+0000", 2024-09-20T22:47:25.127 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0451999999999999e-05 2024-09-20T22:47:25.128 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.128 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.128 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:47:25.128 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:06.819933+0000", 2024-09-20T22:47:25.128 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.3417000000000002e-05 2024-09-20T22:47:25.128 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:47:25.128 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:47:25.128 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:47:25.128 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.128 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.128 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:359 7.9 e108/94 7:91f2d47c:::smithi08477203-8:head v 108'28, mlcod=105'25)", 2024-09-20T22:47:25.128 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:07.834938+0000", 2024-09-20T22:47:25.128 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 257.25842323000001, 2024-09-20T22:47:25.128 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0054540810000000004, 2024-09-20T22:47:25.129 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:47:25.129 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:47:25.129 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:47:25.129 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:47:25.129 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.129 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:47:25.129 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:07.834938+0000", 2024-09-20T22:47:25.129 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:47:25.129 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.129 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.129 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:47:25.129 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:07.834938+0000", 2024-09-20T22:47:25.129 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:47:25.129 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.130 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.130 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:47:25.130 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:07.834939+0000", 2024-09-20T22:47:25.130 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.2399999999999998e-07 2024-09-20T22:47:25.130 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.130 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.130 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:47:25.130 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:07.837461+0000", 2024-09-20T22:47:25.130 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0025222769999999999 2024-09-20T22:47:25.130 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.130 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.130 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:47:25.130 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:07.837473+0000", 2024-09-20T22:47:25.130 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1422e-05 2024-09-20T22:47:25.131 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.131 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.131 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:47:25.131 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:07.837479+0000", 2024-09-20T22:47:25.131 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.083e-06 2024-09-20T22:47:25.131 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.131 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.131 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:47:25.131 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:07.837561+0000", 2024-09-20T22:47:25.131 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.2243999999999994e-05 2024-09-20T22:47:25.131 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.131 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.131 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:47:25.131 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:07.837586+0000", 2024-09-20T22:47:25.132 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.4638999999999999e-05 2024-09-20T22:47:25.132 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.132 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.132 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:47:25.132 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:07.840355+0000", 2024-09-20T22:47:25.132 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.002769681 2024-09-20T22:47:25.132 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.132 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.132 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:47:25.132 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:07.840392+0000", 2024-09-20T22:47:25.132 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.7211e-05 2024-09-20T22:47:25.132 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:47:25.132 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:47:25.132 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:47:25.132 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.133 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.133 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:379 7.1 7:8422f82d:::smithi08477203-22:head [write 742876~447756,stat] snapc 8=[7] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-09-20T22:47:25.133 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:10.210918+0000", 2024-09-20T22:47:25.133 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 254.882443309, 2024-09-20T22:47:25.133 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0030494419999999999, 2024-09-20T22:47:25.133 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:47:25.133 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:47:25.133 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:47:25.133 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:47:25.133 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:47:25.133 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:47:25.133 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 379 2024-09-20T22:47:25.133 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.133 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:47:25.133 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.134 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:47:25.134 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:10.210918+0000", 2024-09-20T22:47:25.134 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:47:25.134 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.134 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.134 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:47:25.134 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:10.210918+0000", 2024-09-20T22:47:25.134 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:47:25.134 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.134 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.134 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:47:25.134 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:10.210920+0000", 2024-09-20T22:47:25.134 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.8419999999999999e-06 2024-09-20T22:47:25.134 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.134 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.135 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:47:25.135 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:10.211774+0000", 2024-09-20T22:47:25.135 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00085377000000000001 2024-09-20T22:47:25.135 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.135 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.135 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:47:25.135 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:10.211791+0000", 2024-09-20T22:47:25.135 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.7186000000000001e-05 2024-09-20T22:47:25.135 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.135 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.135 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:47:25.135 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:10.211799+0000", 2024-09-20T22:47:25.135 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.9179999999999994e-06 2024-09-20T22:47:25.135 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.136 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.136 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:47:25.136 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:10.211870+0000", 2024-09-20T22:47:25.136 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.1271000000000003e-05 2024-09-20T22:47:25.136 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.136 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.136 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:47:25.136 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:10.211939+0000", 2024-09-20T22:47:25.136 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.8245999999999997e-05 2024-09-20T22:47:25.136 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.136 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.136 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 3,6", 2024-09-20T22:47:25.136 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:10.212067+0000", 2024-09-20T22:47:25.136 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000127995 2024-09-20T22:47:25.136 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.137 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.137 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:47:25.137 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:10.213450+0000", 2024-09-20T22:47:25.137 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001383205 2024-09-20T22:47:25.137 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.137 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.137 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:47:25.137 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:10.213457+0000", 2024-09-20T22:47:25.137 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.3159999999999999e-06 2024-09-20T22:47:25.137 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.137 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.137 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:47:25.137 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:10.213884+0000", 2024-09-20T22:47:25.137 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000426444 2024-09-20T22:47:25.137 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.138 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.138 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:47:25.138 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:10.213920+0000", 2024-09-20T22:47:25.138 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.6665000000000002e-05 2024-09-20T22:47:25.138 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.138 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.138 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:47:25.138 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:10.213968+0000", 2024-09-20T22:47:25.138 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.7583999999999997e-05 2024-09-20T22:47:25.138 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:47:25.138 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:47:25.138 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:47:25.138 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.139 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.139 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:380 7.1 7:8422f82d:::smithi08477203-22:head [write 1729599~575687] snapc 8=[7] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-09-20T22:47:25.139 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:10.216967+0000", 2024-09-20T22:47:25.139 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 254.87639417899999, 2024-09-20T22:47:25.139 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0023329090000000002, 2024-09-20T22:47:25.139 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:47:25.139 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:47:25.139 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:47:25.139 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:47:25.139 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:47:25.139 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:47:25.139 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 380 2024-09-20T22:47:25.139 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.139 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:47:25.139 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.140 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:47:25.140 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:10.216967+0000", 2024-09-20T22:47:25.140 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:47:25.140 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.140 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.140 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:47:25.140 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:10.216967+0000", 2024-09-20T22:47:25.140 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:47:25.140 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.140 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.140 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:47:25.140 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:10.216969+0000", 2024-09-20T22:47:25.140 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.111e-06 2024-09-20T22:47:25.140 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.141 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.141 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:47:25.141 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:10.217597+0000", 2024-09-20T22:47:25.141 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00062867999999999995 2024-09-20T22:47:25.141 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.141 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.141 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:47:25.141 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:10.217610+0000", 2024-09-20T22:47:25.141 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2842e-05 2024-09-20T22:47:25.141 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.141 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.141 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:47:25.141 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:10.217616+0000", 2024-09-20T22:47:25.142 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.4360000000000001e-06 2024-09-20T22:47:25.142 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.142 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.142 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:47:25.142 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:10.217655+0000", 2024-09-20T22:47:25.142 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.9032000000000002e-05 2024-09-20T22:47:25.142 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.142 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.142 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:47:25.142 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:10.217708+0000", 2024-09-20T22:47:25.142 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.3022999999999997e-05 2024-09-20T22:47:25.142 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.142 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.142 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 3,6", 2024-09-20T22:47:25.143 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:10.217792+0000", 2024-09-20T22:47:25.143 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.4425000000000006e-05 2024-09-20T22:47:25.143 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.143 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.143 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:47:25.143 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:10.218924+0000", 2024-09-20T22:47:25.143 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00113226 2024-09-20T22:47:25.143 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.143 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.143 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:47:25.143 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:10.219144+0000", 2024-09-20T22:47:25.143 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000219583 2024-09-20T22:47:25.143 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.143 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.143 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:47:25.144 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:10.219224+0000", 2024-09-20T22:47:25.144 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.0269e-05 2024-09-20T22:47:25.144 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.144 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.144 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:47:25.144 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:10.219258+0000", 2024-09-20T22:47:25.144 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.3843000000000001e-05 2024-09-20T22:47:25.144 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.144 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.144 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:47:25.144 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:10.219300+0000", 2024-09-20T22:47:25.144 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.2404999999999998e-05 2024-09-20T22:47:25.144 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:47:25.144 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:47:25.144 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:47:25.145 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.145 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.145 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:383 7.1 7:8422f82d:::smithi08477203-22:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e111)", 2024-09-20T22:47:25.145 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:10.221599+0000", 2024-09-20T22:47:25.145 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 254.87176287400001, 2024-09-20T22:47:25.145 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0016595360000000001, 2024-09-20T22:47:25.145 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:47:25.145 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:47:25.145 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:47:25.145 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:47:25.145 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:47:25.145 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:47:25.145 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 383 2024-09-20T22:47:25.145 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.146 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:47:25.146 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.146 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:47:25.146 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:10.221599+0000", 2024-09-20T22:47:25.146 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:47:25.146 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.146 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.146 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:47:25.146 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:10.221599+0000", 2024-09-20T22:47:25.146 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:47:25.146 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.146 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.146 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:47:25.146 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:10.221599+0000", 2024-09-20T22:47:25.147 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.7599999999999997e-07 2024-09-20T22:47:25.147 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.147 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.147 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:47:25.147 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:10.221602+0000", 2024-09-20T22:47:25.147 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.3879999999999998e-06 2024-09-20T22:47:25.147 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.147 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.147 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:47:25.147 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:10.221608+0000", 2024-09-20T22:47:25.147 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.9900000000000002e-06 2024-09-20T22:47:25.147 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.147 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.147 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:47:25.147 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:10.221614+0000", 2024-09-20T22:47:25.148 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.032e-06 2024-09-20T22:47:25.148 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.148 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.148 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:47:25.148 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:10.222474+0000", 2024-09-20T22:47:25.148 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00086004699999999996 2024-09-20T22:47:25.148 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.148 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.148 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for rw locks", 2024-09-20T22:47:25.148 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:10.222519+0000", 2024-09-20T22:47:25.148 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.5050999999999999e-05 2024-09-20T22:47:25.148 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.148 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.148 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:47:25.148 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:10.223040+0000", 2024-09-20T22:47:25.149 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00052144099999999998 2024-09-20T22:47:25.149 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.149 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.149 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:47:25.149 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:10.223082+0000", 2024-09-20T22:47:25.149 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.1777999999999999e-05 2024-09-20T22:47:25.149 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.149 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.149 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:47:25.149 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:10.223258+0000", 2024-09-20T22:47:25.149 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00017633300000000001 2024-09-20T22:47:25.149 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:47:25.149 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:47:25.149 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:47:25.149 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.150 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.150 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:397 7.9 e112/94 7:91f2d47c:::smithi08477203-8:head v 112'29, mlcod=106'27)", 2024-09-20T22:47:25.150 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:11.366618+0000", 2024-09-20T22:47:25.150 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 253.726743285, 2024-09-20T22:47:25.150 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.005929686, 2024-09-20T22:47:25.150 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:47:25.150 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:47:25.150 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:47:25.150 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:47:25.150 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.150 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:47:25.150 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:11.366618+0000", 2024-09-20T22:47:25.150 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:47:25.150 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.151 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.151 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:47:25.151 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:11.366618+0000", 2024-09-20T22:47:25.151 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:47:25.151 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.151 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.151 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:47:25.151 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:11.366619+0000", 2024-09-20T22:47:25.151 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.67e-07 2024-09-20T22:47:25.151 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.151 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.151 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:47:25.151 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:11.370298+0000", 2024-09-20T22:47:25.151 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0036796559999999999 2024-09-20T22:47:25.152 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.152 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.152 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:47:25.152 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:11.370311+0000", 2024-09-20T22:47:25.152 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3230999999999999e-05 2024-09-20T22:47:25.152 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.152 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.152 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:47:25.152 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:11.370318+0000", 2024-09-20T22:47:25.152 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.3990000000000001e-06 2024-09-20T22:47:25.152 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.152 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.152 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:47:25.152 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:11.370380+0000", 2024-09-20T22:47:25.152 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.2385999999999995e-05 2024-09-20T22:47:25.153 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.153 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.153 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:47:25.153 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:11.370403+0000", 2024-09-20T22:47:25.153 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.2707000000000001e-05 2024-09-20T22:47:25.153 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.153 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.153 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:47:25.153 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:11.372510+0000", 2024-09-20T22:47:25.153 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0021069370000000001 2024-09-20T22:47:25.153 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.153 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.154 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:47:25.154 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:11.372548+0000", 2024-09-20T22:47:25.154 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.8203e-05 2024-09-20T22:47:25.154 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:47:25.154 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:47:25.154 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:47:25.154 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.154 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.154 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:429 7.9 e115/94 7:91f2d47c:::smithi08477203-8:head v 115'31, mlcod=108'28)", 2024-09-20T22:47:25.154 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:14.204349+0000", 2024-09-20T22:47:25.154 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 250.889013081, 2024-09-20T22:47:25.154 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001754469, 2024-09-20T22:47:25.154 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:47:25.155 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:47:25.155 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:47:25.155 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:47:25.155 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.155 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:47:25.155 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:14.204349+0000", 2024-09-20T22:47:25.155 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:47:25.155 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.155 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.155 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:47:25.155 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:14.204349+0000", 2024-09-20T22:47:25.155 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:47:25.155 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.155 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.155 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:47:25.156 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:14.204349+0000", 2024-09-20T22:47:25.156 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.73e-07 2024-09-20T22:47:25.156 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.156 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.156 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:47:25.156 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:14.204888+0000", 2024-09-20T22:47:25.156 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00053951200000000002 2024-09-20T22:47:25.156 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.156 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.156 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:47:25.156 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:14.204902+0000", 2024-09-20T22:47:25.156 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3295000000000001e-05 2024-09-20T22:47:25.156 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.157 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.157 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:47:25.157 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:14.204909+0000", 2024-09-20T22:47:25.157 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.3930000000000002e-06 2024-09-20T22:47:25.157 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.157 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.157 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:47:25.157 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:14.204972+0000", 2024-09-20T22:47:25.157 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.2954999999999996e-05 2024-09-20T22:47:25.157 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.157 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.157 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:47:25.157 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:14.204996+0000", 2024-09-20T22:47:25.157 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.3879999999999998e-05 2024-09-20T22:47:25.157 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.158 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.158 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:47:25.158 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:14.206071+0000", 2024-09-20T22:47:25.158 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001075073 2024-09-20T22:47:25.158 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.158 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.158 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:47:25.158 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:14.206103+0000", 2024-09-20T22:47:25.158 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.2187999999999997e-05 2024-09-20T22:47:25.158 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:47:25.158 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:47:25.158 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:47:25.158 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.158 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.159 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:522 7.9 e124/94 7:9970b578:::smithi08477203-15:head v 124'40, mlcod=118'36)", 2024-09-20T22:47:25.159 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:23.137032+0000", 2024-09-20T22:47:25.159 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 241.95632989999999, 2024-09-20T22:47:25.159 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0095727380000000008, 2024-09-20T22:47:25.159 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:47:25.159 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:47:25.159 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:47:25.159 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:47:25.159 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.159 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:47:25.159 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:23.137032+0000", 2024-09-20T22:47:25.159 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:47:25.159 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.159 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.159 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:47:25.160 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:23.137032+0000", 2024-09-20T22:47:25.160 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:47:25.160 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.160 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.160 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:47:25.160 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:23.137033+0000", 2024-09-20T22:47:25.160 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.034e-06 2024-09-20T22:47:25.160 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.160 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.160 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:47:25.160 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:23.140240+0000", 2024-09-20T22:47:25.160 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0032075419999999999 2024-09-20T22:47:25.160 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.160 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.161 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:47:25.161 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:23.140261+0000", 2024-09-20T22:47:25.161 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.1155999999999999e-05 2024-09-20T22:47:25.161 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.161 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.161 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:47:25.161 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:23.140271+0000", 2024-09-20T22:47:25.161 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.6970000000000005e-06 2024-09-20T22:47:25.161 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.161 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.161 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:47:25.161 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:23.140339+0000", 2024-09-20T22:47:25.161 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.7964000000000002e-05 2024-09-20T22:47:25.161 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.161 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.162 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:47:25.162 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:23.140361+0000", 2024-09-20T22:47:25.162 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.196e-05 2024-09-20T22:47:25.162 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.162 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.162 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:47:25.162 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:23.146563+0000", 2024-09-20T22:47:25.162 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0062014879999999998 2024-09-20T22:47:25.162 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.162 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.162 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:47:25.162 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:23.146604+0000", 2024-09-20T22:47:25.162 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.1897000000000002e-05 2024-09-20T22:47:25.162 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:47:25.163 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:47:25.163 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:47:25.163 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.163 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.163 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "MOSDScrubReserve(1.2,QREQUEST,e:126,from:7,reservation_nonce:3)", 2024-09-20T22:47:25.163 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:25.203729+0000", 2024-09-20T22:47:25.163 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 239.88963216799999, 2024-09-20T22:47:25.163 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0020268320000000001, 2024-09-20T22:47:25.163 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:47:25.163 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:47:25.163 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:47:25.163 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:47:25.163 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.163 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:47:25.164 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:25.203729+0000", 2024-09-20T22:47:25.164 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:47:25.164 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.164 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.164 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:47:25.164 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:25.203729+0000", 2024-09-20T22:47:25.164 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:47:25.164 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.164 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.164 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:47:25.164 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:25.203731+0000", 2024-09-20T22:47:25.164 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1710000000000001e-06 2024-09-20T22:47:25.164 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.164 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.165 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:47:25.165 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:25.203736+0000", 2024-09-20T22:47:25.165 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.6210000000000001e-06 2024-09-20T22:47:25.165 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.165 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.165 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:47:25.165 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:25.203751+0000", 2024-09-20T22:47:25.165 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4423000000000001e-05 2024-09-20T22:47:25.165 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.165 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.165 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:47:25.165 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:25.203758+0000", 2024-09-20T22:47:25.165 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.1110000000000002e-06 2024-09-20T22:47:25.165 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.166 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.166 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:47:25.166 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:25.203819+0000", 2024-09-20T22:47:25.166 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.1476999999999994e-05 2024-09-20T22:47:25.166 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.166 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.166 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op must wait for map", 2024-09-20T22:47:25.166 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:25.203828+0000", 2024-09-20T22:47:25.166 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.5490000000000006e-06 2024-09-20T22:47:25.166 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.166 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.166 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:47:25.166 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:25.205701+0000", 2024-09-20T22:47:25.166 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0018732 2024-09-20T22:47:25.167 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.167 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.167 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:47:25.167 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:25.205718+0000", 2024-09-20T22:47:25.167 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.6790999999999999e-05 2024-09-20T22:47:25.167 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.167 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.167 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:47:25.167 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:25.205756+0000", 2024-09-20T22:47:25.167 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.8488999999999997e-05 2024-09-20T22:47:25.167 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:47:25.167 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:47:25.167 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:47:25.167 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.168 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.168 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:639 7.3 e135/94 7:cae5792a:::smithi08477203-32:head v 135'5, mlcod=98'2)", 2024-09-20T22:47:25.168 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:34.157877+0000", 2024-09-20T22:47:25.168 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 230.935484127, 2024-09-20T22:47:25.168 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0019217240000000001, 2024-09-20T22:47:25.168 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:47:25.168 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:47:25.168 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:47:25.168 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:47:25.168 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.168 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:47:25.168 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:34.157877+0000", 2024-09-20T22:47:25.168 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:47:25.168 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.169 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.169 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:47:25.169 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:34.157877+0000", 2024-09-20T22:47:25.169 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:47:25.169 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.169 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.169 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:47:25.169 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:34.157878+0000", 2024-09-20T22:47:25.169 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.9999999999999995e-07 2024-09-20T22:47:25.169 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.169 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.169 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:47:25.169 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:34.158752+0000", 2024-09-20T22:47:25.169 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00087380499999999996 2024-09-20T22:47:25.169 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.170 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.170 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:47:25.170 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:34.158765+0000", 2024-09-20T22:47:25.170 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2320999999999999e-05 2024-09-20T22:47:25.170 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.170 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.170 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:47:25.170 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:34.158771+0000", 2024-09-20T22:47:25.170 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.438e-06 2024-09-20T22:47:25.170 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.170 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.170 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:47:25.170 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:34.158826+0000", 2024-09-20T22:47:25.170 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.4663000000000002e-05 2024-09-20T22:47:25.171 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.171 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.171 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:47:25.171 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:34.158844+0000", 2024-09-20T22:47:25.171 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.8158000000000001e-05 2024-09-20T22:47:25.171 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.171 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.171 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:47:25.171 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:34.159770+0000", 2024-09-20T22:47:25.171 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00092567699999999997 2024-09-20T22:47:25.171 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.171 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.171 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:47:25.171 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:34.159799+0000", 2024-09-20T22:47:25.172 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.9662000000000002e-05 2024-09-20T22:47:25.172 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:47:25.172 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:47:25.172 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:47:25.172 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.172 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.172 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:745 7.1 7:8422f82d:::smithi08477203-22:head [append 0~0] snapc 18=[17] ondisk+write+known_if_redirected+supports_pool_eio e143)", 2024-09-20T22:47:25.172 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:42.616952+0000", 2024-09-20T22:47:25.172 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 222.47640950799999, 2024-09-20T22:47:25.172 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0025256609999999998, 2024-09-20T22:47:25.172 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:47:25.172 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:47:25.172 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:47:25.172 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:47:25.173 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:47:25.173 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:47:25.173 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 745 2024-09-20T22:47:25.173 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.173 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:47:25.173 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.173 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:47:25.173 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.616952+0000", 2024-09-20T22:47:25.173 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:47:25.173 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.173 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.173 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:47:25.173 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.616952+0000", 2024-09-20T22:47:25.173 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:47:25.174 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.174 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.174 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:47:25.174 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.616954+0000", 2024-09-20T22:47:25.174 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.449e-06 2024-09-20T22:47:25.174 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.174 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.174 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:47:25.174 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.616961+0000", 2024-09-20T22:47:25.174 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.1250000000000004e-06 2024-09-20T22:47:25.174 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.174 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.174 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:47:25.174 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.616975+0000", 2024-09-20T22:47:25.175 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4669e-05 2024-09-20T22:47:25.175 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.175 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.175 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:47:25.175 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.616983+0000", 2024-09-20T22:47:25.175 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.3769999999999998e-06 2024-09-20T22:47:25.175 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.175 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.175 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:47:25.175 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.617040+0000", 2024-09-20T22:47:25.175 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.7147e-05 2024-09-20T22:47:25.175 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.175 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.176 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:47:25.176 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.617152+0000", 2024-09-20T22:47:25.176 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000111809 2024-09-20T22:47:25.176 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.176 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.176 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 3,6", 2024-09-20T22:47:25.176 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.617272+0000", 2024-09-20T22:47:25.176 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000120677 2024-09-20T22:47:25.176 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.176 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.176 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:47:25.176 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.619251+0000", 2024-09-20T22:47:25.176 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001978914 2024-09-20T22:47:25.176 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.177 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.177 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:47:25.177 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.619298+0000", 2024-09-20T22:47:25.177 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.7067000000000002e-05 2024-09-20T22:47:25.177 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.177 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.177 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:47:25.177 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.619391+0000", 2024-09-20T22:47:25.177 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.2793000000000004e-05 2024-09-20T22:47:25.177 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.177 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.177 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:47:25.177 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.619425+0000", 2024-09-20T22:47:25.177 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.3626e-05 2024-09-20T22:47:25.177 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.178 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.178 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:47:25.178 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.619478+0000", 2024-09-20T22:47:25.178 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.3007999999999998e-05 2024-09-20T22:47:25.178 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:47:25.178 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:47:25.178 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:47:25.178 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.178 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.178 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:746 7.1 7:8422f82d:::smithi08477203-22:head [rollback 17] snapc 18=[17] ondisk+write+known_if_redirected+supports_pool_eio e143)", 2024-09-20T22:47:25.178 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:42.617006+0000", 2024-09-20T22:47:25.178 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 222.47635598100001, 2024-09-20T22:47:25.178 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0025901660000000001, 2024-09-20T22:47:25.178 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:47:25.179 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:47:25.179 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:47:25.179 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:47:25.179 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:47:25.179 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:47:25.179 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 746 2024-09-20T22:47:25.179 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.179 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:47:25.179 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.179 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:47:25.179 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.617006+0000", 2024-09-20T22:47:25.179 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:47:25.179 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.179 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.180 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:47:25.180 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.617006+0000", 2024-09-20T22:47:25.180 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:47:25.180 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.180 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.180 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:47:25.180 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.617006+0000", 2024-09-20T22:47:25.180 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.4499999999999997e-07 2024-09-20T22:47:25.180 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.180 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.180 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:47:25.180 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.617009+0000", 2024-09-20T22:47:25.180 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.0120000000000001e-06 2024-09-20T22:47:25.180 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.180 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.181 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:47:25.181 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.617022+0000", 2024-09-20T22:47:25.181 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2468e-05 2024-09-20T22:47:25.181 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.181 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.181 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:47:25.181 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.617028+0000", 2024-09-20T22:47:25.181 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.9190000000000004e-06 2024-09-20T22:47:25.181 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.181 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.181 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:47:25.181 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.617952+0000", 2024-09-20T22:47:25.181 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000924488 2024-09-20T22:47:25.182 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.182 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.182 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:47:25.182 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.618027+0000", 2024-09-20T22:47:25.182 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.5164000000000001e-05 2024-09-20T22:47:25.182 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.182 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.182 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 3,6", 2024-09-20T22:47:25.182 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.618216+0000", 2024-09-20T22:47:25.182 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00018830799999999999 2024-09-20T22:47:25.182 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.183 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.183 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:47:25.183 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.619377+0000", 2024-09-20T22:47:25.183 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0011610909999999999 2024-09-20T22:47:25.183 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.183 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.183 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:47:25.183 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.619527+0000", 2024-09-20T22:47:25.183 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00015048500000000001 2024-09-20T22:47:25.183 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.183 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.183 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:47:25.183 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.619544+0000", 2024-09-20T22:47:25.183 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.6691e-05 2024-09-20T22:47:25.183 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.184 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.184 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:47:25.184 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.619573+0000", 2024-09-20T22:47:25.184 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.8895e-05 2024-09-20T22:47:25.184 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.184 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.184 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:47:25.184 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.619596+0000", 2024-09-20T22:47:25.184 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.3099999999999999e-05 2024-09-20T22:47:25.184 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:47:25.184 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:47:25.184 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:47:25.184 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.184 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.185 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:747 7.1 7:8422f82d:::smithi08477203-22:head [append 0~0] snapc 18=[17] ondisk+write+known_if_redirected+supports_pool_eio e143)", 2024-09-20T22:47:25.185 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:42.617038+0000", 2024-09-20T22:47:25.185 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 222.47632377100001, 2024-09-20T22:47:25.185 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0026729190000000002, 2024-09-20T22:47:25.185 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:47:25.185 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:47:25.185 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:47:25.185 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:47:25.185 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:47:25.185 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:47:25.185 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 747 2024-09-20T22:47:25.185 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.185 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:47:25.186 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.186 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:47:25.186 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.617038+0000", 2024-09-20T22:47:25.186 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:47:25.186 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.186 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.186 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:47:25.186 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.617038+0000", 2024-09-20T22:47:25.186 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:47:25.186 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.186 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.186 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:47:25.186 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.617038+0000", 2024-09-20T22:47:25.186 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.8500000000000002e-07 2024-09-20T22:47:25.187 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.187 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.187 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:47:25.187 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.617041+0000", 2024-09-20T22:47:25.187 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.6800000000000002e-06 2024-09-20T22:47:25.187 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.187 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.187 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:47:25.187 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.617047+0000", 2024-09-20T22:47:25.187 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.7699999999999998e-06 2024-09-20T22:47:25.187 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.187 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.187 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:47:25.187 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.617052+0000", 2024-09-20T22:47:25.188 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.9069999999999998e-06 2024-09-20T22:47:25.188 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.188 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.188 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:47:25.188 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.618852+0000", 2024-09-20T22:47:25.188 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001800458 2024-09-20T22:47:25.188 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.188 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.188 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:47:25.188 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.618940+0000", 2024-09-20T22:47:25.188 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.8208000000000007e-05 2024-09-20T22:47:25.188 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.188 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.188 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 3,6", 2024-09-20T22:47:25.188 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.619042+0000", 2024-09-20T22:47:25.189 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000101364 2024-09-20T22:47:25.189 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.189 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.189 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:47:25.189 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.619601+0000", 2024-09-20T22:47:25.189 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00055948900000000004 2024-09-20T22:47:25.189 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.189 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.189 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:47:25.189 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.619634+0000", 2024-09-20T22:47:25.189 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.2325e-05 2024-09-20T22:47:25.189 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.189 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.189 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:47:25.189 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.619661+0000", 2024-09-20T22:47:25.189 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.7169999999999999e-05 2024-09-20T22:47:25.190 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.190 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.190 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:47:25.190 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.619690+0000", 2024-09-20T22:47:25.190 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.8903000000000001e-05 2024-09-20T22:47:25.190 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.190 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.190 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:47:25.190 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.619711+0000", 2024-09-20T22:47:25.190 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.1160000000000001e-05 2024-09-20T22:47:25.190 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:47:25.190 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:47:25.190 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:47:25.190 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.190 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.191 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:769 7.4 e143/137 7:24b1a612:::smithi08477203-43:head v 143'35, mlcod=140'32)", 2024-09-20T22:47:25.191 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:43.082896+0000", 2024-09-20T22:47:25.191 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 222.01046592599999, 2024-09-20T22:47:25.191 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0016841919999999999, 2024-09-20T22:47:25.191 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:47:25.191 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:47:25.191 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:47:25.191 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:47:25.191 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.191 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:47:25.191 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:43.082896+0000", 2024-09-20T22:47:25.191 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:47:25.191 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.191 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.192 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:47:25.192 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:43.082896+0000", 2024-09-20T22:47:25.192 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:47:25.192 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.192 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.192 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:47:25.192 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:43.082898+0000", 2024-09-20T22:47:25.192 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.12e-06 2024-09-20T22:47:25.192 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.192 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.192 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:47:25.192 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:43.083692+0000", 2024-09-20T22:47:25.192 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00079414999999999998 2024-09-20T22:47:25.192 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.193 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.193 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:47:25.193 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:43.083715+0000", 2024-09-20T22:47:25.193 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.3456999999999999e-05 2024-09-20T22:47:25.193 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.193 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.193 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:47:25.194 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:47:25.194 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:43.083727+0000", 2024-09-20T22:47:25.194 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1488000000000001e-05 2024-09-20T22:47:25.194 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.194 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.194 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:47:25.194 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:43.083799+0000", 2024-09-20T22:47:25.194 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.2434000000000005e-05 2024-09-20T22:47:25.194 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.194 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.194 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:47:25.195 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:43.083832+0000", 2024-09-20T22:47:25.195 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.3093000000000003e-05 2024-09-20T22:47:25.195 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.195 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.195 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:47:25.195 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:43.084544+0000", 2024-09-20T22:47:25.195 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00071157800000000004 2024-09-20T22:47:25.195 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.195 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.195 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:47:25.195 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:43.084580+0000", 2024-09-20T22:47:25.195 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.5871999999999998e-05 2024-09-20T22:47:25.195 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:47:25.195 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:47:25.196 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:47:25.196 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.196 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.196 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(osd.2.18:3 7.1 7:84a866f5:::smithi08477203-27:head [assert-version v48,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e156)", 2024-09-20T22:47:25.196 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:55.627602+0000", 2024-09-20T22:47:25.196 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 209.465759249, 2024-09-20T22:47:25.196 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0032967339999999999, 2024-09-20T22:47:25.196 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:47:25.196 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:47:25.196 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:47:25.196 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:47:25.196 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.196 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:47:25.196 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:55.627602+0000", 2024-09-20T22:47:25.196 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:47:25.197 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.197 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.197 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:47:25.197 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:55.627602+0000", 2024-09-20T22:47:25.197 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:47:25.197 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.197 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.197 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:47:25.197 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:55.627604+0000", 2024-09-20T22:47:25.197 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.703e-06 2024-09-20T22:47:25.197 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.197 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.197 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:47:25.197 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:55.627622+0000", 2024-09-20T22:47:25.198 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.7683999999999999e-05 2024-09-20T22:47:25.198 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.198 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.198 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:47:25.198 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:55.627632+0000", 2024-09-20T22:47:25.198 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0719e-05 2024-09-20T22:47:25.198 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.198 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.198 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:47:25.198 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:55.627644+0000", 2024-09-20T22:47:25.198 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1097000000000001e-05 2024-09-20T22:47:25.198 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.198 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.198 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:47:25.199 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:55.627739+0000", 2024-09-20T22:47:25.199 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.5031000000000005e-05 2024-09-20T22:47:25.199 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.199 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.199 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:47:25.199 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:55.627869+0000", 2024-09-20T22:47:25.199 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00012995500000000001 2024-09-20T22:47:25.199 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:25.199 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:25.199 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:47:25.199 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:55.630899+0000", 2024-09-20T22:47:25.199 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0030305449999999999 2024-09-20T22:47:25.199 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:47:25.199 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:47:25.200 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:47:25.200 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:47:25.200 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:47:25.200 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-20T22:47:25.200 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:47:25.201 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:47:25.205 INFO:tasks.rados.rados.0.smithi084.stdout:1391: done (1 left) 2024-09-20T22:47:25.205 INFO:tasks.rados.rados.0.smithi084.stdout:1392: done (0 left) 2024-09-20T22:47:25.205 INFO:tasks.rados.rados.0.smithi084.stdout:1393: write (excl) oid 19 current snap is 142 2024-09-20T22:47:25.205 INFO:tasks.rados.rados.0.smithi084.stdout:1393: seq_num 497 ranges {753079=561698,2033433=319646} 2024-09-20T22:47:25.212 INFO:tasks.rados.rados.0.smithi084.stdout:1393: writing smithi08477203-19 from 753079 to 1314777 tid 1 2024-09-20T22:47:25.213 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:47:25.213 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:47:25.218 INFO:tasks.rados.rados.0.smithi084.stdout:1393: writing smithi08477203-19 from 2033433 to 2353079 tid 2 2024-09-20T22:47:25.218 INFO:tasks.rados.rados.0.smithi084.stdout:1394: snap_remove snap 141 2024-09-20T22:47:25.223 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:25.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:25.224 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:47:25.232 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:47:25.232 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:47:25.234 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:47:25.249 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-20T22:47:25.249 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T22:47:25.266 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:47:25.267 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:47:25.283 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:47:25.283 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:47:25.302 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:47:25.302 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:47:25.318 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:47:25.318 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:47:25.325 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:47:25.345 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:25.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:25.346 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:47:25.447 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:47:25.468 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:25.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:25.469 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:47:25.481 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:25 smithi084 ceph-mon[40024]: pgmap v447: 168 pgs: 4 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 160 active+clean; 278 MiB data, 858 MiB used, 536 GiB / 536 GiB avail; 19 MiB/s rd, 5.2 MiB/s wr, 31 op/s 2024-09-20T22:47:25.481 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:25 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/2695519828' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:47:25.481 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:25 smithi084 ceph-mon[40024]: osdmap e365: 8 total, 8 up, 6 in 2024-09-20T22:47:25.481 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:25 smithi084 ceph-mon[29774]: pgmap v447: 168 pgs: 4 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 160 active+clean; 278 MiB data, 858 MiB used, 536 GiB / 536 GiB avail; 19 MiB/s rd, 5.2 MiB/s wr, 31 op/s 2024-09-20T22:47:25.481 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:25 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/2695519828' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:47:25.481 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:25 smithi084 ceph-mon[29774]: osdmap e365: 8 total, 8 up, 6 in 2024-09-20T22:47:25.570 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:47:25.590 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:25.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:25.592 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:47:25.692 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:47:25.713 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:25.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:25.714 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:47:25.741 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:25 smithi154 ceph-mon[33264]: pgmap v447: 168 pgs: 4 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 160 active+clean; 278 MiB data, 858 MiB used, 536 GiB / 536 GiB avail; 19 MiB/s rd, 5.2 MiB/s wr, 31 op/s 2024-09-20T22:47:25.741 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:25 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/2695519828' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:47:25.741 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:25 smithi154 ceph-mon[33264]: osdmap e365: 8 total, 8 up, 6 in 2024-09-20T22:47:25.815 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:47:25.836 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:25.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:25.837 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:47:25.938 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:47:25.938 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-20T22:47:25.961 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:25.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:25.962 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:47:26.031 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-20T22:47:26.063 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:47:26.085 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:26.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:26.086 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:47:26.186 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:47:26.201 INFO:tasks.rados.rados.0.smithi084.stdout:1394: done (1 left) 2024-09-20T22:47:26.201 INFO:tasks.rados.rados.0.smithi084.stdout:1395: write oid 1 current snap is 142 2024-09-20T22:47:26.201 INFO:tasks.rados.rados.0.smithi084.stdout:1395: seq_num 498 ranges {769886=636947,2091915=277971} 2024-09-20T22:47:26.208 INFO:tasks.rados.rados.0.smithi084.stdout:1395: writing smithi08477203-1 from 769886 to 1406833 tid 1 2024-09-20T22:47:26.210 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:26.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:26.212 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:47:26.214 INFO:tasks.rados.rados.0.smithi084.stdout:1395: writing smithi08477203-1 from 2091915 to 2369886 tid 2 2024-09-20T22:47:26.215 INFO:tasks.rados.rados.0.smithi084.stdout:1396: snap_create 2024-09-20T22:47:26.215 INFO:tasks.rados.rados.0.smithi084.stdout:1393: finishing write tid 1 to smithi08477203-19 2024-09-20T22:47:26.215 INFO:tasks.rados.rados.0.smithi084.stdout:1393: finishing write tid 2 to smithi08477203-19 2024-09-20T22:47:26.215 INFO:tasks.rados.rados.0.smithi084.stdout:1393: finishing write tid 3 to smithi08477203-19 2024-09-20T22:47:26.215 INFO:tasks.rados.rados.0.smithi084.stdout:1393: finishing write tid 4 to smithi08477203-19 2024-09-20T22:47:26.215 INFO:tasks.rados.rados.0.smithi084.stdout:1393: oid 19 updating version 0 to 773 2024-09-20T22:47:26.215 INFO:tasks.rados.rados.0.smithi084.stdout:1393: oid 19 updating version 773 to 774 2024-09-20T22:47:26.215 INFO:tasks.rados.rados.0.smithi084.stdout:1393: oid 19 updating version 774 to 775 2024-09-20T22:47:26.215 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 19 v 775 (ObjNum 497 snap 142 seq_num 497) dirty exists 2024-09-20T22:47:26.215 INFO:tasks.rados.rados.0.smithi084.stdout:1393: left oid 19 (ObjNum 497 snap 142 seq_num 497) 2024-09-20T22:47:26.215 INFO:tasks.rados.rados.0.smithi084.stdout:1395: finishing write tid 1 to smithi08477203-1 2024-09-20T22:47:26.217 INFO:tasks.rados.rados.0.smithi084.stdout:1395: finishing write tid 2 to smithi08477203-1 2024-09-20T22:47:26.217 INFO:tasks.rados.rados.0.smithi084.stdout:1395: finishing write tid 3 to smithi08477203-1 2024-09-20T22:47:26.218 INFO:tasks.rados.rados.0.smithi084.stdout:1395: finishing write tid 4 to smithi08477203-1 2024-09-20T22:47:26.218 INFO:tasks.rados.rados.0.smithi084.stdout:1395: oid 1 updating version 0 to 518 2024-09-20T22:47:26.218 INFO:tasks.rados.rados.0.smithi084.stdout:1395: oid 1 updating version 518 to 519 2024-09-20T22:47:26.218 INFO:tasks.rados.rados.0.smithi084.stdout:1395: oid 1 version 519 is already newer than 517 2024-09-20T22:47:26.218 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 1 v 519 (ObjNum 498 snap 142 seq_num 498) dirty exists 2024-09-20T22:47:26.218 INFO:tasks.rados.rados.0.smithi084.stdout:1395: left oid 1 (ObjNum 498 snap 142 seq_num 498) 2024-09-20T22:47:26.278 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:47:26.313 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:47:26.335 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:26.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:26.336 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:47:26.437 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:47:26.459 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:26.460 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:47:26.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:26.460 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:47:26.561 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:47:26.582 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:26.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:26.584 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:47:26.684 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:47:26.706 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:26.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:26.707 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:47:26.757 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-20T22:47:26.758 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:47:26.808 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:47:26.830 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:26.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:26.831 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:47:26.932 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:47:26.953 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:26.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:26.954 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:47:27.055 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:47:27.076 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:27.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:27.077 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:47:27.178 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:47:27.199 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:27.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:27.201 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:47:27.202 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.3 to 1 (10000) 2024-09-20T22:47:27.206 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is set 2024-09-20T22:47:27.207 INFO:tasks.rados.rados.0.smithi084.stdout:1393: done (2 left) 2024-09-20T22:47:27.207 INFO:tasks.rados.rados.0.smithi084.stdout:1395: done (1 left) 2024-09-20T22:47:27.207 INFO:tasks.rados.rados.0.smithi084.stdout:1396: done (0 left) 2024-09-20T22:47:27.207 INFO:tasks.rados.rados.0.smithi084.stdout:1397: delete oid 26 current snap is 143 2024-09-20T22:47:27.211 INFO:tasks.rados.rados.0.smithi084.stdout:1397: done (0 left) 2024-09-20T22:47:27.211 INFO:tasks.rados.rados.0.smithi084.stdout:1398: delete oid 5 current snap is 143 2024-09-20T22:47:27.215 INFO:tasks.rados.rados.0.smithi084.stdout:1398: done (0 left) 2024-09-20T22:47:27.215 INFO:tasks.rados.rados.0.smithi084.stdout:1399: rollback oid 35 current snap is 143 2024-09-20T22:47:27.215 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 35 to 136 2024-09-20T22:47:27.215 INFO:tasks.rados.rados.0.smithi084.stdout:1400: write (excl) oid 21 current snap is 143 2024-09-20T22:47:27.215 INFO:tasks.rados.rados.0.smithi084.stdout:1400: seq_num 501 ranges {420307=462694,1351008=617495} 2024-09-20T22:47:27.220 INFO:tasks.rados.rados.0.smithi084.stdout:1400: writing smithi08477203-21 from 420307 to 883001 tid 1 2024-09-20T22:47:27.227 INFO:tasks.rados.rados.0.smithi084.stdout:1400: writing smithi08477203-21 from 1351008 to 1968503 tid 2 2024-09-20T22:47:27.227 INFO:tasks.rados.rados.0.smithi084.stdout:1401: copy_from oid 28 from oid 9 current snap is 143 2024-09-20T22:47:27.227 INFO:tasks.rados.rados.0.smithi084.stdout:1399: finishing rollback tid 0 to smithi08477203-35 2024-09-20T22:47:27.227 INFO:tasks.rados.rados.0.smithi084.stdout:1399: finishing rollback tid 1 to smithi08477203-35 2024-09-20T22:47:27.227 INFO:tasks.rados.rados.0.smithi084.stdout:1399: finishing rollback tid 2 to smithi08477203-35 2024-09-20T22:47:27.227 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 35 v 395 (ObjNum 245 snap 54 seq_num 245) dirty exists 2024-09-20T22:47:27.227 INFO:tasks.rados.rados.0.smithi084.stdout:1400: finishing write tid 1 to smithi08477203-21 2024-09-20T22:47:27.227 INFO:tasks.rados.rados.0.smithi084.stdout:1399: done (2 left) 2024-09-20T22:47:27.228 INFO:tasks.rados.rados.0.smithi084.stdout:1402: write oid 26 current snap is 143 2024-09-20T22:47:27.228 INFO:tasks.rados.rados.0.smithi084.stdout:1402: seq_num 502 ranges {437114=654296,1525843=477506} 2024-09-20T22:47:27.233 INFO:tasks.rados.rados.0.smithi084.stdout:1402: writing smithi08477203-26 from 437114 to 1091410 tid 1 2024-09-20T22:47:27.239 INFO:tasks.rados.rados.0.smithi084.stdout:1402: writing smithi08477203-26 from 1525843 to 2003349 tid 2 2024-09-20T22:47:27.239 INFO:tasks.rados.rados.0.smithi084.stdout:1403: delete oid 12 current snap is 143 2024-09-20T22:47:27.239 INFO:tasks.rados.rados.0.smithi084.stdout:1400: finishing write tid 2 to smithi08477203-21 2024-09-20T22:47:27.239 INFO:tasks.rados.rados.0.smithi084.stdout:1400: finishing write tid 3 to smithi08477203-21 2024-09-20T22:47:27.240 INFO:tasks.rados.rados.0.smithi084.stdout:1400: finishing write tid 4 to smithi08477203-21 2024-09-20T22:47:27.240 INFO:tasks.rados.rados.0.smithi084.stdout:1400: oid 21 updating version 0 to 519 2024-09-20T22:47:27.240 INFO:tasks.rados.rados.0.smithi084.stdout:1400: oid 21 version 519 is already newer than 517 2024-09-20T22:47:27.240 INFO:tasks.rados.rados.0.smithi084.stdout:1400: oid 21 version 519 is already newer than 518 2024-09-20T22:47:27.240 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 21 v 519 (ObjNum 501 snap 143 seq_num 501) dirty exists 2024-09-20T22:47:27.240 INFO:tasks.rados.rados.0.smithi084.stdout:1400: left oid 21 (ObjNum 501 snap 143 seq_num 501) 2024-09-20T22:47:27.240 INFO:tasks.rados.rados.0.smithi084.stdout:1401: finishing copy_from to smithi08477203-28 2024-09-20T22:47:27.240 INFO:tasks.rados.rados.0.smithi084.stdout:1401: got expected ENOENT (src dne) 2024-09-20T22:47:27.240 INFO:tasks.rados.rados.0.smithi084.stdout:1401: finishing copy_from racing read to smithi08477203-28 2024-09-20T22:47:27.240 INFO:tasks.rados.rados.0.smithi084.stdout:1402: finishing write tid 1 to smithi08477203-26 2024-09-20T22:47:27.243 INFO:tasks.rados.rados.0.smithi084.stdout:1402: finishing write tid 2 to smithi08477203-26 2024-09-20T22:47:27.243 INFO:tasks.rados.rados.0.smithi084.stdout:1402: finishing write tid 3 to smithi08477203-26 2024-09-20T22:47:27.243 INFO:tasks.rados.rados.0.smithi084.stdout:1402: finishing write tid 4 to smithi08477203-26 2024-09-20T22:47:27.243 INFO:tasks.rados.rados.0.smithi084.stdout:1402: oid 26 updating version 0 to 301 2024-09-20T22:47:27.243 INFO:tasks.rados.rados.0.smithi084.stdout:1402: oid 26 version 301 is already newer than 300 2024-09-20T22:47:27.243 INFO:tasks.rados.rados.0.smithi084.stdout:1402: oid 26 version 301 is already newer than 299 2024-09-20T22:47:27.243 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 26 v 301 (ObjNum 502 snap 143 seq_num 502) dirty exists 2024-09-20T22:47:27.243 INFO:tasks.rados.rados.0.smithi084.stdout:1402: left oid 26 (ObjNum 502 snap 143 seq_num 502) 2024-09-20T22:47:27.243 INFO:tasks.rados.rados.0.smithi084.stdout:1400: done (3 left) 2024-09-20T22:47:27.243 INFO:tasks.rados.rados.0.smithi084.stdout:1401: done (2 left) 2024-09-20T22:47:27.243 INFO:tasks.rados.rados.0.smithi084.stdout:1402: done (1 left) 2024-09-20T22:47:27.243 INFO:tasks.rados.rados.0.smithi084.stdout:1403: done (0 left) 2024-09-20T22:47:27.244 INFO:tasks.rados.rados.0.smithi084.stdout:1404: read oid 14 snap -1 2024-09-20T22:47:27.244 INFO:tasks.rados.rados.0.smithi084.stdout:1404: expect (ObjNum 455 snap 125 seq_num 455) 2024-09-20T22:47:27.244 INFO:tasks.rados.rados.0.smithi084.stdout:1405: write (excl) oid 34 current snap is 143 2024-09-20T22:47:27.244 INFO:tasks.rados.rados.0.smithi084.stdout:1405: seq_num 504 ranges {470728=404794,1526454=481175} 2024-09-20T22:47:27.247 INFO:tasks.rados.rados.0.smithi084.stdout:1405: writing smithi08477203-34 from 470728 to 875522 tid 1 2024-09-20T22:47:27.253 INFO:tasks.rados.rados.0.smithi084.stdout:1405: writing smithi08477203-34 from 1526454 to 2007629 tid 2 2024-09-20T22:47:27.254 INFO:tasks.rados.rados.0.smithi084.stdout:1406: read oid 15 snap -1 2024-09-20T22:47:27.254 INFO:tasks.rados.rados.0.smithi084.stdout:1406: expect (ObjNum 401 snap 101 seq_num 401) 2024-09-20T22:47:27.254 INFO:tasks.rados.rados.0.smithi084.stdout:1407: write (excl) oid 50 current snap is 143 2024-09-20T22:47:27.254 INFO:tasks.rados.rados.0.smithi084.stdout:1407: seq_num 505 ranges {487535=480043,1584936=624833} 2024-09-20T22:47:27.258 INFO:tasks.rados.rados.0.smithi084.stdout:1407: writing smithi08477203-50 from 487535 to 967578 tid 1 2024-09-20T22:47:27.266 INFO:tasks.rados.rados.0.smithi084.stdout:1407: writing smithi08477203-50 from 1584936 to 2209769 tid 2 2024-09-20T22:47:27.266 INFO:tasks.rados.rados.0.smithi084.stdout:1408: rollback oid 19 current snap is 143 2024-09-20T22:47:27.266 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 19 to 113 2024-09-20T22:47:27.266 INFO:tasks.rados.rados.0.smithi084.stdout:1405: finishing write tid 1 to smithi08477203-34 2024-09-20T22:47:27.266 INFO:tasks.rados.rados.0.smithi084.stdout:1409: write (excl) oid 43 current snap is 143 2024-09-20T22:47:27.266 INFO:tasks.rados.rados.0.smithi084.stdout:1404: expect (ObjNum 455 snap 125 seq_num 455) 2024-09-20T22:47:27.302 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:47:27.322 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:27.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:27.323 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:47:27.345 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:27 smithi154 ceph-mon[33264]: pgmap v450: 168 pgs: 4 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 160 active+clean; 278 MiB data, 858 MiB used, 536 GiB / 536 GiB avail; 19 MiB/s rd, 5.2 MiB/s wr, 31 op/s 2024-09-20T22:47:27.345 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:27 smithi154 ceph-mon[33264]: osdmap e366: 8 total, 8 up, 6 in 2024-09-20T22:47:27.345 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:27 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/2582790615' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-09-20T22:47:27.345 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:27 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/942228466' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:47:27.346 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:27 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:47:27.372 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.5 dump_blocked_ops 2024-09-20T22:47:27.424 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:47:27.430 INFO:tasks.rados.rados.0.smithi084.stdout:1406: expect (ObjNum 401 snap 101 seq_num 401) 2024-09-20T22:47:27.445 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:27.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:27.446 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:47:27.450 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:27 smithi084 ceph-mon[40024]: pgmap v450: 168 pgs: 4 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 160 active+clean; 278 MiB data, 858 MiB used, 536 GiB / 536 GiB avail; 19 MiB/s rd, 5.2 MiB/s wr, 31 op/s 2024-09-20T22:47:27.450 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:27 smithi084 ceph-mon[40024]: osdmap e366: 8 total, 8 up, 6 in 2024-09-20T22:47:27.450 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:27 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/2582790615' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-09-20T22:47:27.450 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:27 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/942228466' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:47:27.450 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:27 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:47:27.451 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:27 smithi084 ceph-mon[29774]: pgmap v450: 168 pgs: 4 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 160 active+clean; 278 MiB data, 858 MiB used, 536 GiB / 536 GiB avail; 19 MiB/s rd, 5.2 MiB/s wr, 31 op/s 2024-09-20T22:47:27.451 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:27 smithi084 ceph-mon[29774]: osdmap e366: 8 total, 8 up, 6 in 2024-09-20T22:47:27.451 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:27 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/2582790615' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-09-20T22:47:27.451 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:27 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/942228466' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:47:27.451 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:27 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:47:27.547 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:47:27.568 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:27.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:27.569 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:47:27.621 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:47:27.631 INFO:tasks.rados.rados.0.smithi084.stdout:1405: finishing write tid 2 to smithi08477203-34 2024-09-20T22:47:27.632 INFO:tasks.rados.rados.0.smithi084.stdout:1405: finishing write tid 3 to smithi08477203-34 2024-09-20T22:47:27.632 INFO:tasks.rados.rados.0.smithi084.stdout:1409: seq_num 506 ranges {504342=555292,1643418=484844} 2024-09-20T22:47:27.637 INFO:tasks.rados.rados.0.smithi084.stdout:1409: writing smithi08477203-43 from 504342 to 1059634 tid 1 2024-09-20T22:47:27.644 INFO:tasks.rados.rados.0.smithi084.stdout:1409: writing smithi08477203-43 from 1643418 to 2128262 tid 2 2024-09-20T22:47:27.644 INFO:tasks.rados.rados.0.smithi084.stdout:1404: done (5 left) 2024-09-20T22:47:27.644 INFO:tasks.rados.rados.0.smithi084.stdout:1406: done (4 left) 2024-09-20T22:47:27.644 INFO:tasks.rados.rados.0.smithi084.stdout:1410: read oid 39 snap -1 2024-09-20T22:47:27.645 INFO:tasks.rados.rados.0.smithi084.stdout:1410: expect (ObjNum 397 snap 101 seq_num 397) 2024-09-20T22:47:27.645 INFO:tasks.rados.rados.0.smithi084.stdout:1411: snap_create 2024-09-20T22:47:27.645 INFO:tasks.rados.rados.0.smithi084.stdout:1405: finishing write tid 4 to smithi08477203-34 2024-09-20T22:47:27.645 INFO:tasks.rados.rados.0.smithi084.stdout:1405: oid 34 updating version 0 to 521 2024-09-20T22:47:27.645 INFO:tasks.rados.rados.0.smithi084.stdout:1405: oid 34 version 521 is already newer than 520 2024-09-20T22:47:27.645 INFO:tasks.rados.rados.0.smithi084.stdout:1405: oid 34 updating version 521 to 522 2024-09-20T22:47:27.645 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 34 v 522 (ObjNum 504 snap 143 seq_num 504) dirty exists 2024-09-20T22:47:27.645 INFO:tasks.rados.rados.0.smithi084.stdout:1405: left oid 34 (ObjNum 504 snap 143 seq_num 504) 2024-09-20T22:47:27.645 INFO:tasks.rados.rados.0.smithi084.stdout:1407: finishing write tid 1 to smithi08477203-50 2024-09-20T22:47:27.645 INFO:tasks.rados.rados.0.smithi084.stdout:1408: finishing rollback tid 0 to smithi08477203-19 2024-09-20T22:47:27.646 INFO:tasks.rados.rados.0.smithi084.stdout:1408: finishing rollback tid 1 to smithi08477203-19 2024-09-20T22:47:27.646 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 19 v 778 (ObjNum 33 snap 0 seq_num 709432509) dirty dne 2024-09-20T22:47:27.646 INFO:tasks.rados.rados.0.smithi084.stdout:1407: finishing write tid 2 to smithi08477203-50 2024-09-20T22:47:27.646 INFO:tasks.rados.rados.0.smithi084.stdout:1407: finishing write tid 3 to smithi08477203-50 2024-09-20T22:47:27.646 INFO:tasks.rados.rados.0.smithi084.stdout:1407: finishing write tid 4 to smithi08477203-50 2024-09-20T22:47:27.646 INFO:tasks.rados.rados.0.smithi084.stdout:1407: oid 50 updating version 0 to 249 2024-09-20T22:47:27.646 INFO:tasks.rados.rados.0.smithi084.stdout:1407: oid 50 version 249 is already newer than 248 2024-09-20T22:47:27.646 INFO:tasks.rados.rados.0.smithi084.stdout:1407: oid 50 version 249 is already newer than 247 2024-09-20T22:47:27.646 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 50 v 249 (ObjNum 505 snap 143 seq_num 505) dirty exists 2024-09-20T22:47:27.647 INFO:tasks.rados.rados.0.smithi084.stdout:1407: left oid 50 (ObjNum 505 snap 143 seq_num 505) 2024-09-20T22:47:27.647 INFO:tasks.rados.rados.0.smithi084.stdout:1409: finishing write tid 1 to smithi08477203-43 2024-09-20T22:47:27.648 INFO:tasks.rados.rados.0.smithi084.stdout:1409: finishing write tid 2 to smithi08477203-43 2024-09-20T22:47:27.648 INFO:tasks.rados.rados.0.smithi084.stdout:1409: finishing write tid 3 to smithi08477203-43 2024-09-20T22:47:27.648 INFO:tasks.rados.rados.0.smithi084.stdout:1409: finishing write tid 4 to smithi08477203-43 2024-09-20T22:47:27.648 INFO:tasks.rados.rados.0.smithi084.stdout:1409: oid 43 updating version 0 to 67 2024-09-20T22:47:27.648 INFO:tasks.rados.rados.0.smithi084.stdout:1409: oid 43 version 67 is already newer than 66 2024-09-20T22:47:27.648 INFO:tasks.rados.rados.0.smithi084.stdout:1409: oid 43 version 67 is already newer than 65 2024-09-20T22:47:27.648 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 43 v 67 (ObjNum 506 snap 143 seq_num 506) dirty exists 2024-09-20T22:47:27.648 INFO:tasks.rados.rados.0.smithi084.stdout:1409: left oid 43 (ObjNum 506 snap 143 seq_num 506) 2024-09-20T22:47:27.649 INFO:tasks.rados.rados.0.smithi084.stdout:1410: expect (ObjNum 397 snap 101 seq_num 397) 2024-09-20T22:47:27.670 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:47:27.691 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:27.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:27.692 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:47:27.793 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:47:27.813 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:27.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:27.814 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:47:27.915 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:47:27.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:27.938 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:27.938 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:47:28.039 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:47:28.060 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:28.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:28.061 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:47:28.092 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-20T22:47:28.092 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:47:28.161 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:47:28.182 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:28.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:28.183 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:47:28.188 INFO:tasks.rados.rados.0.smithi084.stdout:1405: done (5 left) 2024-09-20T22:47:28.189 INFO:tasks.rados.rados.0.smithi084.stdout:1407: done (4 left) 2024-09-20T22:47:28.189 INFO:tasks.rados.rados.0.smithi084.stdout:1408: done (3 left) 2024-09-20T22:47:28.189 INFO:tasks.rados.rados.0.smithi084.stdout:1409: done (2 left) 2024-09-20T22:47:28.189 INFO:tasks.rados.rados.0.smithi084.stdout:1410: done (1 left) 2024-09-20T22:47:28.189 INFO:tasks.rados.rados.0.smithi084.stdout:1411: done (0 left) 2024-09-20T22:47:28.189 INFO:tasks.rados.rados.0.smithi084.stdout:1412: read oid 6 snap 139 2024-09-20T22:47:28.189 INFO:tasks.rados.rados.0.smithi084.stdout:1412: expect (ObjNum 488 snap 137 seq_num 488) 2024-09-20T22:47:28.189 INFO:tasks.rados.rados.0.smithi084.stdout:1413: snap_remove snap 135 2024-09-20T22:47:28.197 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-20T22:47:28.246 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd reweight 5 1 2024-09-20T22:47:28.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:28 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/2582790615' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-20T22:47:28.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:28 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T22:47:28.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:28 smithi154 ceph-mon[33264]: osdmap e367: 8 total, 8 up, 6 in 2024-09-20T22:47:28.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:28 smithi154 ceph-mon[33264]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:47:28.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:28 smithi154 ceph-mon[33264]: osdmap e368: 8 total, 8 up, 6 in 2024-09-20T22:47:28.284 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:47:28.306 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:28.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:28.307 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:47:28.407 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:47:28.428 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:28.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:28.429 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:47:28.530 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:47:28.551 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:28.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:28.552 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:47:28.615 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:47:28 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: ::ffff:172.21.15.154 - - [20/Sep/2024:22:47:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-20T22:47:28.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:28 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/2582790615' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-20T22:47:28.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:28 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T22:47:28.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:28 smithi084 ceph-mon[29774]: osdmap e367: 8 total, 8 up, 6 in 2024-09-20T22:47:28.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:28 smithi084 ceph-mon[29774]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:47:28.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:28 smithi084 ceph-mon[29774]: osdmap e368: 8 total, 8 up, 6 in 2024-09-20T22:47:28.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:28 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/2582790615' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-20T22:47:28.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:28 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T22:47:28.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:28 smithi084 ceph-mon[40024]: osdmap e367: 8 total, 8 up, 6 in 2024-09-20T22:47:28.617 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:28 smithi084 ceph-mon[40024]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:47:28.617 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:28 smithi084 ceph-mon[40024]: osdmap e368: 8 total, 8 up, 6 in 2024-09-20T22:47:28.653 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:47:28.674 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:28.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:28.675 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:47:28.733 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.5 dump_historic_ops 2024-09-20T22:47:28.776 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:47:28.786 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:47:28.798 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:28.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:28.799 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:47:28.900 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:47:28.922 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:28.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:28.923 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:47:28.980 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:47:29.023 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:47:29.045 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:29.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:29.046 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:47:29.061 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:47:29.147 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:47:29.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:29.170 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:29.170 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:47:29.197 INFO:tasks.rados.rados.0.smithi084.stdout:1413: done (1 left) 2024-09-20T22:47:29.197 INFO:tasks.rados.rados.0.smithi084.stdout:1414: write (excl) oid 2 current snap is 144 2024-09-20T22:47:29.198 INFO:tasks.rados.rados.0.smithi084.stdout:1414: seq_num 507 ranges {521149=630541,1701900=628502} 2024-09-20T22:47:29.204 INFO:tasks.rados.rados.0.smithi084.stdout:1414: writing smithi08477203-2 from 521149 to 1151690 tid 1 2024-09-20T22:47:29.212 INFO:tasks.rados.rados.0.smithi084.stdout:1414: writing smithi08477203-2 from 1701900 to 2330402 tid 2 2024-09-20T22:47:29.212 INFO:tasks.rados.rados.0.smithi084.stdout:1415: read oid 47 snap -1 2024-09-20T22:47:29.213 INFO:tasks.rados.rados.0.smithi084.stdout:1415: expect (ObjNum 466 snap 128 seq_num 466) 2024-09-20T22:47:29.213 INFO:tasks.rados.rados.0.smithi084.stdout:1416: write (excl) oid 35 current snap is 144 2024-09-20T22:47:29.213 INFO:tasks.rados.rados.0.smithi084.stdout:1412: expect (ObjNum 488 snap 137 seq_num 488) 2024-09-20T22:47:29.270 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:47:29.292 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:29.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:29.294 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:47:29.317 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:47:29.317 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:47:29.318 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:29 smithi084 ceph-mon[40024]: pgmap v453: 168 pgs: 4 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 160 active+clean; 278 MiB data, 858 MiB used, 536 GiB / 536 GiB avail 2024-09-20T22:47:29.318 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:29 smithi084 ceph-mon[40024]: osdmap e369: 8 total, 8 up, 6 in 2024-09-20T22:47:29.318 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:29 smithi084 ceph-mon[29774]: pgmap v453: 168 pgs: 4 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 160 active+clean; 278 MiB data, 858 MiB used, 536 GiB / 536 GiB avail 2024-09-20T22:47:29.318 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:29 smithi084 ceph-mon[29774]: osdmap e369: 8 total, 8 up, 6 in 2024-09-20T22:47:29.345 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:47:29.346 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:47:29.369 INFO:journalctl@ceph.iscsi.iscsi.a.smithi154.stdout:Sep 20 22:47:29 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-iscsi-iscsi-a[59730]: debug there is no tcmu-runner data available 2024-09-20T22:47:29.369 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:29 smithi154 ceph-mon[33264]: pgmap v453: 168 pgs: 4 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 160 active+clean; 278 MiB data, 858 MiB used, 536 GiB / 536 GiB avail 2024-09-20T22:47:29.369 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:29 smithi154 ceph-mon[33264]: osdmap e369: 8 total, 8 up, 6 in 2024-09-20T22:47:29.371 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:47:29.372 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:47:29.380 INFO:tasks.rados.rados.0.smithi084.stdout:1416: seq_num 508 ranges {537956=705790,1644029=772160} 2024-09-20T22:47:29.387 INFO:tasks.rados.rados.0.smithi084.stdout:1416: writing smithi08477203-35 from 537956 to 1243746 tid 1 2024-09-20T22:47:29.388 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-20T22:47:29.388 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T22:47:29.394 INFO:tasks.rados.rados.0.smithi084.stdout:1416: writing smithi08477203-35 from 1644029 to 2416189 tid 2 2024-09-20T22:47:29.394 INFO:tasks.rados.rados.0.smithi084.stdout:1412: done (3 left) 2024-09-20T22:47:29.394 INFO:tasks.rados.rados.0.smithi084.stdout:1417: read oid 45 snap -1 2024-09-20T22:47:29.394 INFO:tasks.rados.rados.0.smithi084.stdout:1417: expect (ObjNum 466 snap 128 seq_num 466) 2024-09-20T22:47:29.394 INFO:tasks.rados.rados.0.smithi084.stdout:1418: snap_remove snap 142 2024-09-20T22:47:29.394 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:47:29.405 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:47:29.406 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:47:29.418 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:29.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:29.419 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:47:29.422 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:47:29.422 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:47:29.438 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:47:29.438 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:47:29.456 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:47:29.457 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:47:29.475 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-20T22:47:29.475 INFO:teuthology.orchestra.run.smithi154.stdout: "size": 20, 2024-09-20T22:47:29.475 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 600, 2024-09-20T22:47:29.475 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [ 2024-09-20T22:47:29.476 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.476 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24334.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-09-20T22:47:29.476 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:41:40.156321+0000", 2024-09-20T22:47:29.476 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 349.31628864599998, 2024-09-20T22:47:29.476 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.012506929, 2024-09-20T22:47:29.476 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:47:29.476 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:47:29.477 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:47:29.477 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:47:29.477 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24334", 2024-09-20T22:47:29.477 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/343304630", 2024-09-20T22:47:29.477 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 7 2024-09-20T22:47:29.477 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.477 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:47:29.477 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.477 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:47:29.478 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:40.156321+0000", 2024-09-20T22:47:29.478 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:47:29.478 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.478 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.478 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:47:29.478 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:40.156321+0000", 2024-09-20T22:47:29.478 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:47:29.478 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.478 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.478 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:47:29.478 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:40.156323+0000", 2024-09-20T22:47:29.478 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9700000000000002e-06 2024-09-20T22:47:29.479 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.479 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.479 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:47:29.479 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:40.156330+0000", 2024-09-20T22:47:29.479 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.9809999999999997e-06 2024-09-20T22:47:29.479 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.479 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.479 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:47:29.479 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:40.156348+0000", 2024-09-20T22:47:29.479 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.7938e-05 2024-09-20T22:47:29.479 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.479 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.479 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:47:29.479 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:40.156362+0000", 2024-09-20T22:47:29.480 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3641e-05 2024-09-20T22:47:29.480 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.480 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.480 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:47:29.480 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:40.156430+0000", 2024-09-20T22:47:29.480 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.7543e-05 2024-09-20T22:47:29.480 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.480 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.480 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:47:29.480 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:40.156543+0000", 2024-09-20T22:47:29.480 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00011390100000000001 2024-09-20T22:47:29.480 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.480 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.480 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 1,6", 2024-09-20T22:47:29.481 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:40.156712+0000", 2024-09-20T22:47:29.481 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00016825499999999999 2024-09-20T22:47:29.481 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.481 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.481 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:47:29.481 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:40.157640+0000", 2024-09-20T22:47:29.481 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00092863699999999995 2024-09-20T22:47:29.481 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.481 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.481 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:47:29.481 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:40.159647+0000", 2024-09-20T22:47:29.481 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0020068320000000001 2024-09-20T22:47:29.481 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.482 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.482 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:47:29.482 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:40.168687+0000", 2024-09-20T22:47:29.482 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0090397749999999999 2024-09-20T22:47:29.482 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.482 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.482 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:47:29.482 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:40.168768+0000", 2024-09-20T22:47:29.482 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.0819e-05 2024-09-20T22:47:29.482 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.482 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.482 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:47:29.482 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:40.168828+0000", 2024-09-20T22:47:29.483 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.0637000000000002e-05 2024-09-20T22:47:29.483 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:47:29.483 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:47:29.483 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:47:29.483 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.483 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.483 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "MOSDPGPush(1.4 93/92 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 77'128, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@77'128, size: 327680, copy_subset: [(0, 327680)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 327680, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-20T22:47:29.483 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:42:50.646393+0000", 2024-09-20T22:47:29.483 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 278.82621720700001, 2024-09-20T22:47:29.484 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.033640915, 2024-09-20T22:47:29.484 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:47:29.484 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:47:29.484 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:47:29.484 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:47:29.484 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.484 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:47:29.484 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:42:50.646393+0000", 2024-09-20T22:47:29.484 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:47:29.484 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.484 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.484 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:47:29.485 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:42:50.646393+0000", 2024-09-20T22:47:29.485 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:47:29.485 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.485 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.485 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:47:29.485 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:42:50.646393+0000", 2024-09-20T22:47:29.485 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.23e-07 2024-09-20T22:47:29.485 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.485 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.485 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:47:29.485 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:42:50.679431+0000", 2024-09-20T22:47:29.485 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.033038349000000002 2024-09-20T22:47:29.485 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.486 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.486 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:47:29.486 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:42:50.679478+0000", 2024-09-20T22:47:29.486 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.6828e-05 2024-09-20T22:47:29.486 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.486 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.486 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:47:29.486 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:42:50.679492+0000", 2024-09-20T22:47:29.486 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4062e-05 2024-09-20T22:47:29.486 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.486 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.486 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:47:29.486 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:42:50.679593+0000", 2024-09-20T22:47:29.486 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000100865 2024-09-20T22:47:29.487 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.487 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.487 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:47:29.487 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:42:50.679604+0000", 2024-09-20T22:47:29.487 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0348e-05 2024-09-20T22:47:29.487 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.487 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.487 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:47:29.487 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:42:50.680034+0000", 2024-09-20T22:47:29.487 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00043023999999999997 2024-09-20T22:47:29.487 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:47:29.487 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:47:29.487 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:47:29.488 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.488 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.488 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:419 7.0 7:05bb442d:::smithi08477203-24:head [write 460525~690852] snapc a=[9] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-09-20T22:47:29.488 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:14.165736+0000", 2024-09-20T22:47:29.488 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 255.30687431300001, 2024-09-20T22:47:29.488 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0072523500000000003, 2024-09-20T22:47:29.488 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:47:29.488 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:47:29.488 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:47:29.488 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:47:29.488 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:47:29.488 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:47:29.488 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 419 2024-09-20T22:47:29.488 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.489 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:47:29.489 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.489 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:47:29.489 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:14.165736+0000", 2024-09-20T22:47:29.489 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:47:29.489 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.489 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.489 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:47:29.489 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:14.165736+0000", 2024-09-20T22:47:29.489 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:47:29.489 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.490 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.490 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:47:29.490 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:14.165737+0000", 2024-09-20T22:47:29.490 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.615e-06 2024-09-20T22:47:29.490 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.490 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.490 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:47:29.490 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:14.166920+0000", 2024-09-20T22:47:29.490 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001182535 2024-09-20T22:47:29.490 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.490 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.491 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:47:29.491 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:14.166933+0000", 2024-09-20T22:47:29.491 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3263e-05 2024-09-20T22:47:29.491 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.491 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.491 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:47:29.491 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:14.166942+0000", 2024-09-20T22:47:29.491 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.3329999999999997e-06 2024-09-20T22:47:29.491 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.491 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.491 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:47:29.492 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:14.167001+0000", 2024-09-20T22:47:29.492 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.9305000000000002e-05 2024-09-20T22:47:29.492 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.492 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.492 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op must wait for map", 2024-09-20T22:47:29.492 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:14.167008+0000", 2024-09-20T22:47:29.492 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.5580000000000003e-06 2024-09-20T22:47:29.492 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.492 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.492 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:47:29.492 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:14.169081+0000", 2024-09-20T22:47:29.492 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0020726270000000001 2024-09-20T22:47:29.493 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.493 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.493 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:47:29.493 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:14.169179+0000", 2024-09-20T22:47:29.493 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.7613000000000002e-05 2024-09-20T22:47:29.493 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.493 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.493 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 1,3", 2024-09-20T22:47:29.493 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:14.169364+0000", 2024-09-20T22:47:29.493 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000185642 2024-09-20T22:47:29.493 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.493 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.493 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:47:29.493 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:14.171014+0000", 2024-09-20T22:47:29.494 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0016497129999999999 2024-09-20T22:47:29.494 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.494 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.494 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:47:29.494 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:14.172374+0000", 2024-09-20T22:47:29.494 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0013601240000000001 2024-09-20T22:47:29.494 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.494 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.494 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:47:29.494 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:14.172925+0000", 2024-09-20T22:47:29.494 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00055099499999999998 2024-09-20T22:47:29.494 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.495 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.495 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:47:29.495 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:14.172959+0000", 2024-09-20T22:47:29.495 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.4069000000000002e-05 2024-09-20T22:47:29.495 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.495 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.495 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:47:29.495 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:14.172988+0000", 2024-09-20T22:47:29.495 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.8958e-05 2024-09-20T22:47:29.495 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:47:29.495 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:47:29.495 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:47:29.495 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.496 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.496 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:552 7.0 7:05bb442d:::smithi08477203-24:head [sparse-read 0~1736210] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e128)", 2024-09-20T22:47:29.496 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:27.143470+0000", 2024-09-20T22:47:29.496 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 242.32913994399999, 2024-09-20T22:47:29.496 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0075036570000000004, 2024-09-20T22:47:29.496 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:47:29.496 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:47:29.497 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:47:29.497 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:47:29.497 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:47:29.497 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:47:29.497 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 552 2024-09-20T22:47:29.498 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.498 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:47:29.498 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.498 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:47:29.498 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.143470+0000", 2024-09-20T22:47:29.498 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:47:29.498 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.498 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.498 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:47:29.498 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.143470+0000", 2024-09-20T22:47:29.499 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:47:29.499 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.499 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.499 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:47:29.499 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.143470+0000", 2024-09-20T22:47:29.499 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.84e-07 2024-09-20T22:47:29.499 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.499 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.499 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:47:29.499 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.143472+0000", 2024-09-20T22:47:29.499 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4139999999999999e-06 2024-09-20T22:47:29.499 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.499 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.499 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:47:29.499 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.143475+0000", 2024-09-20T22:47:29.500 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.1360000000000001e-06 2024-09-20T22:47:29.500 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.500 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.500 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:47:29.500 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.143478+0000", 2024-09-20T22:47:29.500 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.9000000000000002e-06 2024-09-20T22:47:29.500 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.500 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.500 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:47:29.500 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.143601+0000", 2024-09-20T22:47:29.500 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00012325199999999999 2024-09-20T22:47:29.500 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.500 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.500 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting_for_map not empty", 2024-09-20T22:47:29.501 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.143605+0000", 2024-09-20T22:47:29.501 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.5540000000000002e-06 2024-09-20T22:47:29.501 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.501 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.501 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:47:29.501 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.150656+0000", 2024-09-20T22:47:29.501 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0070513290000000003 2024-09-20T22:47:29.501 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.501 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.501 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:47:29.501 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.150787+0000", 2024-09-20T22:47:29.501 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00013061099999999999 2024-09-20T22:47:29.501 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.501 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.501 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:47:29.502 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.150974+0000", 2024-09-20T22:47:29.502 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00018717699999999999 2024-09-20T22:47:29.502 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:47:29.502 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:47:29.502 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:47:29.502 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.502 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.502 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:554 7.a e128/94 7:5122292a:::smithi08477203-33:head v 128'32, mlcod=119'28)", 2024-09-20T22:47:29.502 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:27.341250+0000", 2024-09-20T22:47:29.502 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 242.13135962300001, 2024-09-20T22:47:29.502 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.051661319999999997, 2024-09-20T22:47:29.502 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:47:29.502 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:47:29.502 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:47:29.503 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:47:29.503 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.503 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:47:29.503 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.341250+0000", 2024-09-20T22:47:29.503 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:47:29.503 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.503 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.503 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:47:29.503 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.341250+0000", 2024-09-20T22:47:29.503 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:47:29.503 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.503 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.503 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:47:29.503 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.341251+0000", 2024-09-20T22:47:29.504 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.7299999999999996e-07 2024-09-20T22:47:29.504 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.504 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.504 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:47:29.504 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.390395+0000", 2024-09-20T22:47:29.504 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.049143539999999999 2024-09-20T22:47:29.504 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.504 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.504 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:47:29.504 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.390413+0000", 2024-09-20T22:47:29.504 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.8709999999999999e-05 2024-09-20T22:47:29.504 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.504 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.504 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:47:29.504 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.390423+0000", 2024-09-20T22:47:29.505 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.4660000000000005e-06 2024-09-20T22:47:29.505 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.505 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.505 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:47:29.505 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.390484+0000", 2024-09-20T22:47:29.505 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.0851999999999999e-05 2024-09-20T22:47:29.505 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.505 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.505 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:47:29.505 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.390504+0000", 2024-09-20T22:47:29.506 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9976999999999999e-05 2024-09-20T22:47:29.506 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.506 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.506 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:47:29.506 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.392875+0000", 2024-09-20T22:47:29.506 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0023713710000000002 2024-09-20T22:47:29.506 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.506 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.506 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:47:29.506 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.392912+0000", 2024-09-20T22:47:29.506 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.6831e-05 2024-09-20T22:47:29.507 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:47:29.507 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:47:29.508 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:47:29.508 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.508 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.508 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:786 7.17 e145/139 7:ed54065f:::smithi08477203-12:head v 145'119, mlcod=143'117)", 2024-09-20T22:47:29.508 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:44.345115+0000", 2024-09-20T22:47:29.508 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 225.127495141, 2024-09-20T22:47:29.508 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0098654899999999993, 2024-09-20T22:47:29.508 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:47:29.508 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:47:29.508 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:47:29.508 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:47:29.508 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.508 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:47:29.508 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.345115+0000", 2024-09-20T22:47:29.508 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:47:29.509 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.509 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.509 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:47:29.509 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.345115+0000", 2024-09-20T22:47:29.509 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:47:29.509 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.509 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.509 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:47:29.509 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.345116+0000", 2024-09-20T22:47:29.509 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.8299999999999995e-07 2024-09-20T22:47:29.509 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.509 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.509 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:47:29.509 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.347955+0000", 2024-09-20T22:47:29.510 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.002839028 2024-09-20T22:47:29.510 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.510 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.510 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:47:29.510 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.347977+0000", 2024-09-20T22:47:29.510 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.1897999999999999e-05 2024-09-20T22:47:29.510 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.510 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.510 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:47:29.510 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.347992+0000", 2024-09-20T22:47:29.510 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4761e-05 2024-09-20T22:47:29.510 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.510 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.510 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:47:29.511 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.348072+0000", 2024-09-20T22:47:29.511 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.0586000000000004e-05 2024-09-20T22:47:29.511 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.511 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.511 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:47:29.511 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.348097+0000", 2024-09-20T22:47:29.511 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.5026000000000001e-05 2024-09-20T22:47:29.511 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.511 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.511 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:47:29.511 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.354924+0000", 2024-09-20T22:47:29.511 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0068271240000000004 2024-09-20T22:47:29.511 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.511 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.512 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:47:29.512 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.354980+0000", 2024-09-20T22:47:29.512 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.6184000000000003e-05 2024-09-20T22:47:29.512 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:47:29.512 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:47:29.512 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:47:29.512 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.512 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.512 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:894 7.a e155/94 7:50097c2d:::smithi08477203-23:head v 155'54, mlcod=143'50)", 2024-09-20T22:47:29.512 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:54.450745+0000", 2024-09-20T22:47:29.512 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 215.021865053, 2024-09-20T22:47:29.512 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0091901029999999998, 2024-09-20T22:47:29.512 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:47:29.513 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:47:29.513 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:47:29.513 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:47:29.513 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.513 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:47:29.513 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:54.450745+0000", 2024-09-20T22:47:29.513 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:47:29.513 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.513 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.513 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:47:29.513 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:54.450745+0000", 2024-09-20T22:47:29.513 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:47:29.513 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.513 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.514 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:47:29.514 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:54.450745+0000", 2024-09-20T22:47:29.514 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.4200000000000002e-07 2024-09-20T22:47:29.514 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.514 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.514 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:47:29.514 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:54.456336+0000", 2024-09-20T22:47:29.514 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.005590589 2024-09-20T22:47:29.514 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.514 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.514 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:47:29.514 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:54.456353+0000", 2024-09-20T22:47:29.515 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.7187e-05 2024-09-20T22:47:29.515 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.515 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.515 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:47:29.515 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:54.456362+0000", 2024-09-20T22:47:29.515 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.7409999999999998e-06 2024-09-20T22:47:29.515 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.515 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.515 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:47:29.515 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:54.456418+0000", 2024-09-20T22:47:29.515 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.6431000000000002e-05 2024-09-20T22:47:29.516 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.516 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.516 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:47:29.516 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:54.456438+0000", 2024-09-20T22:47:29.516 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9718999999999999e-05 2024-09-20T22:47:29.516 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.516 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.516 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:47:29.516 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:54.459899+0000", 2024-09-20T22:47:29.516 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.003461254 2024-09-20T22:47:29.516 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.516 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.516 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:47:29.516 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:54.459935+0000", 2024-09-20T22:47:29.517 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.5939999999999998e-05 2024-09-20T22:47:29.517 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:47:29.517 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:47:29.517 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:47:29.517 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.517 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.517 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:901 7.e e156/94 7:7c62ced4:::smithi08477203-17:head v 156'42, mlcod=154'40)", 2024-09-20T22:47:29.517 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:55.635897+0000", 2024-09-20T22:47:29.517 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 213.83671265999999, 2024-09-20T22:47:29.517 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0082336760000000005, 2024-09-20T22:47:29.517 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:47:29.517 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:47:29.517 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:47:29.517 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:47:29.517 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.517 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:47:29.518 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:55.635897+0000", 2024-09-20T22:47:29.518 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:47:29.518 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.518 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.518 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:47:29.518 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:55.635897+0000", 2024-09-20T22:47:29.518 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:47:29.518 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.518 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.518 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:47:29.518 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:55.635898+0000", 2024-09-20T22:47:29.518 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.23e-07 2024-09-20T22:47:29.518 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.518 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.518 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:47:29.519 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:55.640084+0000", 2024-09-20T22:47:29.519 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0041865499999999998 2024-09-20T22:47:29.519 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.519 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.519 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:47:29.519 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:55.640118+0000", 2024-09-20T22:47:29.519 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.4093e-05 2024-09-20T22:47:29.519 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.519 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.519 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:47:29.519 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:55.640129+0000", 2024-09-20T22:47:29.519 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0281e-05 2024-09-20T22:47:29.519 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.519 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.519 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:47:29.520 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:55.640230+0000", 2024-09-20T22:47:29.520 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000101538 2024-09-20T22:47:29.520 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.520 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.521 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:47:29.521 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:55.640251+0000", 2024-09-20T22:47:29.521 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.1293000000000001e-05 2024-09-20T22:47:29.521 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.521 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.521 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:47:29.521 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:55.644088+0000", 2024-09-20T22:47:29.521 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.003836178 2024-09-20T22:47:29.521 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.521 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.521 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:47:29.521 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:55.644131+0000", 2024-09-20T22:47:29.521 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.3519999999999997e-05 2024-09-20T22:47:29.522 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:47:29.522 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:47:29.522 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:47:29.522 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.522 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.522 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:1316 7.0 7:05bb442d:::smithi08477203-24:head [stat,delete] snapc 2b=[2b,2a] ondisk+write+known_if_redirected+supports_pool_eio e180)", 2024-09-20T22:47:29.522 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:44:19.225492+0000", 2024-09-20T22:47:29.522 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 190.247118089, 2024-09-20T22:47:29.522 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0083403250000000009, 2024-09-20T22:47:29.522 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:47:29.522 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:47:29.522 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:47:29.522 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:47:29.522 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:47:29.523 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:47:29.523 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 1316 2024-09-20T22:47:29.523 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.523 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:47:29.523 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.523 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:47:29.523 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:19.225492+0000", 2024-09-20T22:47:29.523 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:47:29.524 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.524 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.524 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:47:29.524 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:19.225492+0000", 2024-09-20T22:47:29.524 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:47:29.524 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.524 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.524 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:47:29.524 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:19.225495+0000", 2024-09-20T22:47:29.524 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.96e-06 2024-09-20T22:47:29.525 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.525 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.525 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:47:29.525 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:19.225508+0000", 2024-09-20T22:47:29.525 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3343000000000001e-05 2024-09-20T22:47:29.525 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.525 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.525 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:47:29.525 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:19.225542+0000", 2024-09-20T22:47:29.525 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.3973000000000002e-05 2024-09-20T22:47:29.525 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.525 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.525 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:47:29.525 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:19.225570+0000", 2024-09-20T22:47:29.525 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.7713999999999999e-05 2024-09-20T22:47:29.526 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.526 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.526 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:47:29.526 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:19.225655+0000", 2024-09-20T22:47:29.526 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.4944999999999994e-05 2024-09-20T22:47:29.526 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.526 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.526 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op must wait for map", 2024-09-20T22:47:29.526 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:19.225660+0000", 2024-09-20T22:47:29.526 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.5470000000000004e-06 2024-09-20T22:47:29.526 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.527 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.527 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:47:29.527 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:19.232618+0000", 2024-09-20T22:47:29.527 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0069571069999999997 2024-09-20T22:47:29.527 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.527 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.527 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:47:29.527 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:19.232717+0000", 2024-09-20T22:47:29.527 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.9306000000000001e-05 2024-09-20T22:47:29.527 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.527 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.527 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:47:29.527 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:19.233832+0000", 2024-09-20T22:47:29.527 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0011154299999999999 2024-09-20T22:47:29.528 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:47:29.528 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:47:29.528 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:47:29.528 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.528 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.528 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:1527 7.17 e197/139 7:ea720b85:::smithi08477203-26:head v 197'159, mlcod=187'156)", 2024-09-20T22:47:29.528 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:44:36.209560+0000", 2024-09-20T22:47:29.528 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 173.263050462, 2024-09-20T22:47:29.528 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.010263069999999999, 2024-09-20T22:47:29.528 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:47:29.528 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:47:29.528 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:47:29.528 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:47:29.529 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.529 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:47:29.529 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.209560+0000", 2024-09-20T22:47:29.529 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:47:29.529 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.529 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.529 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:47:29.529 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.209560+0000", 2024-09-20T22:47:29.529 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:47:29.529 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.529 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.529 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:47:29.529 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.209560+0000", 2024-09-20T22:47:29.529 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.0800000000000001e-07 2024-09-20T22:47:29.530 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.530 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.530 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:47:29.530 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.212844+0000", 2024-09-20T22:47:29.530 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.003283907 2024-09-20T22:47:29.530 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.530 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.530 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:47:29.530 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.212867+0000", 2024-09-20T22:47:29.530 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.3229000000000001e-05 2024-09-20T22:47:29.530 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.530 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.530 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:47:29.530 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.212878+0000", 2024-09-20T22:47:29.531 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0998e-05 2024-09-20T22:47:29.531 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.531 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.531 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:47:29.531 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.212971+0000", 2024-09-20T22:47:29.531 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.2872000000000001e-05 2024-09-20T22:47:29.531 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.531 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.531 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:47:29.531 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.212996+0000", 2024-09-20T22:47:29.531 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.4921000000000001e-05 2024-09-20T22:47:29.531 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.531 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.531 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:47:29.532 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.219758+0000", 2024-09-20T22:47:29.532 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0067625139999999999 2024-09-20T22:47:29.532 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.532 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.532 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:47:29.532 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.219823+0000", 2024-09-20T22:47:29.532 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.4420999999999999e-05 2024-09-20T22:47:29.532 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:47:29.532 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:47:29.532 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:47:29.532 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.532 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.532 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:1665 7.0 7:064f66ab:::smithi08477203-45:head [copy-from ver 0] snapc 3a=[3a,39,37] ondisk+write+known_if_redirected+supports_pool_eio e209)", 2024-09-20T22:47:29.532 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:44:48.141884+0000", 2024-09-20T22:47:29.533 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 161.330726519, 2024-09-20T22:47:29.533 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0074924060000000001, 2024-09-20T22:47:29.533 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:47:29.533 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:47:29.533 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:47:29.533 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:47:29.533 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:47:29.533 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:47:29.533 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 1665 2024-09-20T22:47:29.533 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.533 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:47:29.533 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.533 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:47:29.533 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:48.141884+0000", 2024-09-20T22:47:29.533 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:47:29.534 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.534 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.534 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:47:29.534 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:48.141884+0000", 2024-09-20T22:47:29.534 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:47:29.534 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.534 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.534 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:47:29.534 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:48.141886+0000", 2024-09-20T22:47:29.534 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.1579999999999999e-06 2024-09-20T22:47:29.534 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.534 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.534 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:47:29.534 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:48.141894+0000", 2024-09-20T22:47:29.535 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.7530000000000007e-06 2024-09-20T22:47:29.535 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.535 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.535 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:47:29.535 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:48.141911+0000", 2024-09-20T22:47:29.535 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.6759e-05 2024-09-20T22:47:29.535 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.535 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.535 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:47:29.535 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:48.141921+0000", 2024-09-20T22:47:29.535 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0209e-05 2024-09-20T22:47:29.535 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.536 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.536 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:47:29.536 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:48.141987+0000", 2024-09-20T22:47:29.536 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.5953999999999999e-05 2024-09-20T22:47:29.536 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.536 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.536 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op must wait for map", 2024-09-20T22:47:29.536 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:48.141995+0000", 2024-09-20T22:47:29.536 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.3749999999999997e-06 2024-09-20T22:47:29.536 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.536 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.536 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:47:29.537 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:48.143638+0000", 2024-09-20T22:47:29.537 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001642798 2024-09-20T22:47:29.537 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.537 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.537 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:47:29.537 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:48.143738+0000", 2024-09-20T22:47:29.537 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000100723 2024-09-20T22:47:29.537 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.537 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.537 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:47:29.537 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:48.149376+0000", 2024-09-20T22:47:29.537 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0056376769999999998 2024-09-20T22:47:29.537 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:47:29.538 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:47:29.538 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:47:29.538 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.538 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.538 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:1666 7.0 7:064f66ab:::smithi08477203-45:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e209)", 2024-09-20T22:47:29.538 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:44:48.141945+0000", 2024-09-20T22:47:29.538 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 161.33066540499999, 2024-09-20T22:47:29.538 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0076299330000000002, 2024-09-20T22:47:29.538 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:47:29.538 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:47:29.538 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:47:29.538 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:47:29.539 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:47:29.539 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:47:29.539 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 1666 2024-09-20T22:47:29.539 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.539 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:47:29.539 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.539 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:47:29.539 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:48.141945+0000", 2024-09-20T22:47:29.539 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:47:29.539 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.539 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.539 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:47:29.539 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:48.141945+0000", 2024-09-20T22:47:29.539 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:47:29.540 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.540 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.540 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:47:29.540 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:48.141945+0000", 2024-09-20T22:47:29.540 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.1900000000000002e-07 2024-09-20T22:47:29.540 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.540 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.540 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:47:29.540 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:48.141949+0000", 2024-09-20T22:47:29.540 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.19e-06 2024-09-20T22:47:29.540 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.540 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.540 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:47:29.541 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:48.141957+0000", 2024-09-20T22:47:29.541 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.0579999999999998e-06 2024-09-20T22:47:29.541 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.541 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.541 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:47:29.541 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:48.141963+0000", 2024-09-20T22:47:29.541 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.9920000000000004e-06 2024-09-20T22:47:29.541 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.541 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.541 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:47:29.541 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:48.142051+0000", 2024-09-20T22:47:29.541 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.8295999999999996e-05 2024-09-20T22:47:29.541 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.541 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.542 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting_for_map not empty", 2024-09-20T22:47:29.542 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:48.142058+0000", 2024-09-20T22:47:29.542 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.7989999999999996e-06 2024-09-20T22:47:29.542 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.542 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.542 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:47:29.542 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:48.143995+0000", 2024-09-20T22:47:29.542 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001937624 2024-09-20T22:47:29.542 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.542 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.542 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for blocked object", 2024-09-20T22:47:29.542 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:48.144501+0000", 2024-09-20T22:47:29.542 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00050535399999999996 2024-09-20T22:47:29.542 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.543 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.543 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:47:29.543 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:48.149458+0000", 2024-09-20T22:47:29.543 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0049577830000000003 2024-09-20T22:47:29.543 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.543 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.543 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:47:29.543 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:48.149513+0000", 2024-09-20T22:47:29.543 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.5012000000000001e-05 2024-09-20T22:47:29.543 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.543 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.543 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:47:29.543 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:48.149575+0000", 2024-09-20T22:47:29.543 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.1205999999999996e-05 2024-09-20T22:47:29.544 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:47:29.544 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:47:29.544 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:47:29.544 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.544 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.544 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:1825 7.b e230/94 7:d960cad4:::smithi08477203-18:head v 230'138, mlcod=229'135)", 2024-09-20T22:47:29.544 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:45:09.177560+0000", 2024-09-20T22:47:29.544 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 140.29505059799999, 2024-09-20T22:47:29.544 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0086763780000000002, 2024-09-20T22:47:29.544 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:47:29.544 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:47:29.544 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:47:29.544 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:47:29.544 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.545 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:47:29.545 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:09.177560+0000", 2024-09-20T22:47:29.545 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:47:29.545 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.545 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.545 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:47:29.545 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:09.177560+0000", 2024-09-20T22:47:29.545 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:47:29.545 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.545 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.545 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:47:29.545 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:09.177560+0000", 2024-09-20T22:47:29.545 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.0599999999999999e-07 2024-09-20T22:47:29.545 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.545 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.545 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:47:29.546 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:09.178522+0000", 2024-09-20T22:47:29.546 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00096277000000000005 2024-09-20T22:47:29.546 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.546 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.546 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:47:29.546 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:09.178536+0000", 2024-09-20T22:47:29.546 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3886999999999999e-05 2024-09-20T22:47:29.546 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.546 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.546 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:47:29.547 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:09.178547+0000", 2024-09-20T22:47:29.547 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0526999999999999e-05 2024-09-20T22:47:29.547 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.547 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.547 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:47:29.547 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:09.178614+0000", 2024-09-20T22:47:29.547 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.7547999999999998e-05 2024-09-20T22:47:29.547 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.547 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.547 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:47:29.547 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:09.178638+0000", 2024-09-20T22:47:29.547 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.3190999999999999e-05 2024-09-20T22:47:29.547 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.547 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.548 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:47:29.548 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:09.186193+0000", 2024-09-20T22:47:29.548 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0075549559999999998 2024-09-20T22:47:29.548 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.548 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.548 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:47:29.548 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:09.186236+0000", 2024-09-20T22:47:29.548 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.3293e-05 2024-09-20T22:47:29.548 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:47:29.548 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:47:29.548 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:47:29.548 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.548 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.548 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:2350 7.0 7:064f66ab:::smithi08477203-45:head [stat,delete] snapc 56=[56,55,53,52,51,50,47,3e] ondisk+write+known_if_redirected+supports_pool_eio e260)", 2024-09-20T22:47:29.548 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:45:39.199566+0000", 2024-09-20T22:47:29.549 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 110.27304403300001, 2024-09-20T22:47:29.549 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.027897148, 2024-09-20T22:47:29.549 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:47:29.549 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:47:29.549 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:47:29.549 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:47:29.549 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:47:29.549 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:47:29.549 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 2350 2024-09-20T22:47:29.549 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.549 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:47:29.549 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.549 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:47:29.549 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.199566+0000", 2024-09-20T22:47:29.549 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:47:29.549 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.550 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.550 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:47:29.550 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.199566+0000", 2024-09-20T22:47:29.550 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:47:29.550 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.550 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.550 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:47:29.550 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.199568+0000", 2024-09-20T22:47:29.550 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9290000000000001e-06 2024-09-20T22:47:29.550 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.551 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.551 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:47:29.551 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.199574+0000", 2024-09-20T22:47:29.551 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.3409999999999999e-06 2024-09-20T22:47:29.551 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.551 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.551 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:47:29.551 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.199584+0000", 2024-09-20T22:47:29.551 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0091e-05 2024-09-20T22:47:29.551 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.551 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.551 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:47:29.552 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.199593+0000", 2024-09-20T22:47:29.552 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.1079999999999993e-06 2024-09-20T22:47:29.552 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.552 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.552 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:47:29.552 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.199661+0000", 2024-09-20T22:47:29.552 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.8767000000000001e-05 2024-09-20T22:47:29.552 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.552 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.552 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:47:29.552 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.199765+0000", 2024-09-20T22:47:29.552 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000103886 2024-09-20T22:47:29.552 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.552 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.553 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:47:29.553 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.227463+0000", 2024-09-20T22:47:29.553 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.027698026000000001 2024-09-20T22:47:29.553 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:47:29.553 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:47:29.553 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:47:29.553 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.553 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.553 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:2834 7.2 7:45b54b6b:::smithi08477203-2:head [sparse-read 0~1350228] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e302)", 2024-09-20T22:47:29.553 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:46:22.209076+0000", 2024-09-20T22:47:29.553 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 67.263534358000001, 2024-09-20T22:47:29.553 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.007179314, 2024-09-20T22:47:29.553 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:47:29.554 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:47:29.554 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:47:29.554 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:47:29.554 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:47:29.554 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:47:29.554 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 2834 2024-09-20T22:47:29.554 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.554 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:47:29.554 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.554 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:47:29.554 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:22.209076+0000", 2024-09-20T22:47:29.554 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:47:29.554 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.554 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.555 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:47:29.555 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:22.209076+0000", 2024-09-20T22:47:29.555 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:47:29.555 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.555 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.555 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:47:29.555 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:22.209077+0000", 2024-09-20T22:47:29.555 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.7599999999999996e-07 2024-09-20T22:47:29.555 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.555 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.556 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:47:29.556 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:22.209080+0000", 2024-09-20T22:47:29.556 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.8290000000000001e-06 2024-09-20T22:47:29.556 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.556 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.556 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:47:29.556 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:22.209105+0000", 2024-09-20T22:47:29.556 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.4511999999999998e-05 2024-09-20T22:47:29.556 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.556 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.557 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:47:29.557 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:22.209112+0000", 2024-09-20T22:47:29.557 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.0589999999999997e-06 2024-09-20T22:47:29.557 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.557 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.557 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:47:29.557 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:22.209171+0000", 2024-09-20T22:47:29.557 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.9116000000000001e-05 2024-09-20T22:47:29.557 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.557 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.557 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting_for_map not empty", 2024-09-20T22:47:29.557 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:22.209176+0000", 2024-09-20T22:47:29.557 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.7700000000000001e-06 2024-09-20T22:47:29.557 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.558 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.558 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:47:29.558 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:22.216068+0000", 2024-09-20T22:47:29.558 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0068917259999999999 2024-09-20T22:47:29.558 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.558 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.558 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:47:29.558 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:22.216130+0000", 2024-09-20T22:47:29.558 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.2080000000000002e-05 2024-09-20T22:47:29.558 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.558 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.558 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:47:29.558 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:22.216255+0000", 2024-09-20T22:47:29.559 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000125446 2024-09-20T22:47:29.559 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:47:29.559 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:47:29.559 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:47:29.559 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.559 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.559 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:3134 7.2 7:45b54b6b:::smithi08477203-2:head [copy-from ver 151] snapc 77=[77,76,75,74,73,70,6b] ondisk+write+known_if_redirected+supports_pool_eio e328)", 2024-09-20T22:47:29.559 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:46:48.211103+0000", 2024-09-20T22:47:29.561 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 41.261507526000003, 2024-09-20T22:47:29.561 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.030812519, 2024-09-20T22:47:29.561 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:47:29.562 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:47:29.562 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:47:29.562 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:47:29.562 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:47:29.562 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:47:29.562 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 3134 2024-09-20T22:47:29.562 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.562 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:47:29.562 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.562 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:47:29.562 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.211103+0000", 2024-09-20T22:47:29.562 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:47:29.562 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.562 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.563 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:47:29.563 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.211103+0000", 2024-09-20T22:47:29.563 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:47:29.563 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.563 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.563 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:47:29.563 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.211105+0000", 2024-09-20T22:47:29.563 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.3369999999999998e-06 2024-09-20T22:47:29.563 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.563 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.563 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:47:29.563 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.211111+0000", 2024-09-20T22:47:29.563 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.2759999999999997e-06 2024-09-20T22:47:29.563 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.564 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.564 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:47:29.564 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.211123+0000", 2024-09-20T22:47:29.564 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2119999999999999e-05 2024-09-20T22:47:29.564 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.564 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.564 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:47:29.565 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.211131+0000", 2024-09-20T22:47:29.565 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.2899999999999997e-06 2024-09-20T22:47:29.565 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.565 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.565 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:47:29.565 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.211213+0000", 2024-09-20T22:47:29.565 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.2756999999999994e-05 2024-09-20T22:47:29.566 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.566 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.566 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:47:29.566 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.211310+0000", 2024-09-20T22:47:29.566 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.6644999999999994e-05 2024-09-20T22:47:29.566 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.566 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.566 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 1,2", 2024-09-20T22:47:29.566 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.229109+0000", 2024-09-20T22:47:29.566 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.017798587000000001 2024-09-20T22:47:29.566 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.566 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.566 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:47:29.566 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.240265+0000", 2024-09-20T22:47:29.567 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.011156716000000001 2024-09-20T22:47:29.567 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.567 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.567 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:47:29.567 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.241608+0000", 2024-09-20T22:47:29.567 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001343033 2024-09-20T22:47:29.567 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.568 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.568 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:47:29.568 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.241831+0000", 2024-09-20T22:47:29.568 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000223148 2024-09-20T22:47:29.568 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.568 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.568 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:47:29.568 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.241885+0000", 2024-09-20T22:47:29.568 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.3161000000000001e-05 2024-09-20T22:47:29.568 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.568 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.568 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:47:29.568 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.241915+0000", 2024-09-20T22:47:29.568 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.0448999999999999e-05 2024-09-20T22:47:29.568 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:47:29.569 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:47:29.569 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:47:29.569 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.569 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.569 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:3135 7.2 7:45b54b6b:::smithi08477203-2:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e328)", 2024-09-20T22:47:29.569 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:46:48.211153+0000", 2024-09-20T22:47:29.569 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 41.261457554000003, 2024-09-20T22:47:29.569 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.022731534000000001, 2024-09-20T22:47:29.569 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:47:29.569 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:47:29.569 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:47:29.569 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:47:29.569 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:47:29.569 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:47:29.570 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 3135 2024-09-20T22:47:29.570 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.570 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:47:29.570 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.570 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:47:29.570 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.211153+0000", 2024-09-20T22:47:29.570 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:47:29.570 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.570 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.570 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:47:29.570 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.211153+0000", 2024-09-20T22:47:29.570 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:47:29.570 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.570 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.571 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:47:29.571 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.211153+0000", 2024-09-20T22:47:29.571 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.1600000000000001e-07 2024-09-20T22:47:29.571 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.571 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.571 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:47:29.571 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.211157+0000", 2024-09-20T22:47:29.571 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.5470000000000001e-06 2024-09-20T22:47:29.571 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.571 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.571 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:47:29.571 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.211164+0000", 2024-09-20T22:47:29.571 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.2799999999999998e-06 2024-09-20T22:47:29.572 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.572 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.572 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:47:29.572 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.211170+0000", 2024-09-20T22:47:29.572 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.0730000000000001e-06 2024-09-20T22:47:29.572 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.572 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.572 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:47:29.572 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.211634+0000", 2024-09-20T22:47:29.572 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00046401800000000002 2024-09-20T22:47:29.572 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.572 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.572 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for blocked object", 2024-09-20T22:47:29.572 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.211690+0000", 2024-09-20T22:47:29.572 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.5587999999999997e-05 2024-09-20T22:47:29.573 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.573 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.573 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:47:29.573 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.233669+0000", 2024-09-20T22:47:29.573 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.021979240000000001 2024-09-20T22:47:29.573 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.573 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.573 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:47:29.573 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.233762+0000", 2024-09-20T22:47:29.573 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.2658999999999995e-05 2024-09-20T22:47:29.573 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.573 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.573 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:47:29.573 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.233884+0000", 2024-09-20T22:47:29.573 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00012241300000000001 2024-09-20T22:47:29.574 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:47:29.574 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:47:29.574 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:47:29.574 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.574 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.574 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:3136 7.a e328/94 7:5122292a:::smithi08477203-33:head v 328'252, mlcod=322'248)", 2024-09-20T22:47:29.574 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:46:48.223492+0000", 2024-09-20T22:47:29.574 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 41.249117843000001, 2024-09-20T22:47:29.574 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.016662746999999999, 2024-09-20T22:47:29.574 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:47:29.574 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:47:29.574 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:47:29.574 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:47:29.574 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.574 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:47:29.575 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.223492+0000", 2024-09-20T22:47:29.575 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:47:29.575 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.575 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.575 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:47:29.575 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.223492+0000", 2024-09-20T22:47:29.575 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:47:29.575 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.575 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.575 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:47:29.575 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.223493+0000", 2024-09-20T22:47:29.576 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.4299999999999998e-07 2024-09-20T22:47:29.576 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.576 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.576 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:47:29.576 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.228060+0000", 2024-09-20T22:47:29.576 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0045669350000000003 2024-09-20T22:47:29.576 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.576 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.576 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:47:29.576 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.228107+0000", 2024-09-20T22:47:29.576 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.6932999999999999e-05 2024-09-20T22:47:29.576 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.576 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.576 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:47:29.576 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.228127+0000", 2024-09-20T22:47:29.577 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.0777000000000001e-05 2024-09-20T22:47:29.577 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.577 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.577 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:47:29.577 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.228206+0000", 2024-09-20T22:47:29.577 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.8660999999999998e-05 2024-09-20T22:47:29.577 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.577 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.577 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:47:29.577 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.228232+0000", 2024-09-20T22:47:29.577 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.6199000000000001e-05 2024-09-20T22:47:29.578 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.578 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.578 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:47:29.578 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.240103+0000", 2024-09-20T22:47:29.578 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.011871008000000001 2024-09-20T22:47:29.578 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.578 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.578 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:47:29.578 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.240155+0000", 2024-09-20T22:47:29.578 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.1790999999999999e-05 2024-09-20T22:47:29.578 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:47:29.578 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:47:29.578 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:47:29.578 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.579 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.579 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:3232 7.16 e337/260 7:6885a65f:::smithi08477203-14:head v 337'107, mlcod=331'104)", 2024-09-20T22:47:29.579 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:46:57.231995+0000", 2024-09-20T22:47:29.579 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 32.240614889, 2024-09-20T22:47:29.579 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0090885140000000007, 2024-09-20T22:47:29.579 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:47:29.579 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:47:29.579 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:47:29.579 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:47:29.579 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.579 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:47:29.579 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.231995+0000", 2024-09-20T22:47:29.579 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:47:29.579 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.580 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.580 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:47:29.580 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.231995+0000", 2024-09-20T22:47:29.580 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:47:29.580 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.580 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.580 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:47:29.580 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.231996+0000", 2024-09-20T22:47:29.580 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.5799999999999998e-07 2024-09-20T22:47:29.580 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.580 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.580 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:47:29.580 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.234590+0000", 2024-09-20T22:47:29.580 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.002594482 2024-09-20T22:47:29.580 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.581 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.581 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:47:29.581 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.234617+0000", 2024-09-20T22:47:29.581 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.6855e-05 2024-09-20T22:47:29.581 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.581 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.581 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:47:29.581 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.234636+0000", 2024-09-20T22:47:29.581 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.8289000000000001e-05 2024-09-20T22:47:29.581 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.581 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.581 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:47:29.581 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.234729+0000", 2024-09-20T22:47:29.581 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.3207000000000004e-05 2024-09-20T22:47:29.581 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.581 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.582 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:47:29.582 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.234758+0000", 2024-09-20T22:47:29.582 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.8994999999999999e-05 2024-09-20T22:47:29.582 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.582 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.582 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:47:29.582 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.241031+0000", 2024-09-20T22:47:29.582 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0062733279999999999 2024-09-20T22:47:29.582 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.582 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.582 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:47:29.582 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.241084+0000", 2024-09-20T22:47:29.582 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.2599999999999998e-05 2024-09-20T22:47:29.582 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:47:29.582 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:47:29.583 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:47:29.583 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.583 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.583 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:3589 7.b e363/94 7:dc95255e:::smithi08477203-1:head v 363'419, mlcod=363'417)", 2024-09-20T22:47:29.583 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:47:23.205271+0000", 2024-09-20T22:47:29.583 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 6.2673388619999999, 2024-09-20T22:47:29.583 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.014301911, 2024-09-20T22:47:29.583 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:47:29.583 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:47:29.583 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:47:29.583 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:47:29.583 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.583 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:47:29.583 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:23.205271+0000", 2024-09-20T22:47:29.584 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:47:29.584 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.584 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.584 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:47:29.584 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:23.205271+0000", 2024-09-20T22:47:29.584 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:47:29.584 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.584 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.584 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:47:29.584 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:23.205273+0000", 2024-09-20T22:47:29.584 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.409e-06 2024-09-20T22:47:29.584 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.584 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.584 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:47:29.585 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:23.210044+0000", 2024-09-20T22:47:29.585 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0047712609999999997 2024-09-20T22:47:29.585 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.585 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.585 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:47:29.585 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:23.210064+0000", 2024-09-20T22:47:29.585 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.0421e-05 2024-09-20T22:47:29.585 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.585 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.585 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:47:29.585 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:23.210074+0000", 2024-09-20T22:47:29.585 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.4329999999999993e-06 2024-09-20T22:47:29.585 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.585 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.586 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:47:29.586 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:23.210130+0000", 2024-09-20T22:47:29.586 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.6286000000000003e-05 2024-09-20T22:47:29.586 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.586 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.586 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:47:29.586 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:23.210148+0000", 2024-09-20T22:47:29.586 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.7895999999999999e-05 2024-09-20T22:47:29.586 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.586 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.586 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:47:29.586 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:23.219532+0000", 2024-09-20T22:47:29.586 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.009383677 2024-09-20T22:47:29.586 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:29.587 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:29.587 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:47:29.587 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:23.219573+0000", 2024-09-20T22:47:29.587 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.1527999999999999e-05 2024-09-20T22:47:29.587 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:47:29.587 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:47:29.587 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:47:29.587 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:47:29.587 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:47:29.587 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-20T22:47:29.587 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:47:29.608 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:29.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:29.609 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:47:29.710 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:47:29.731 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:29.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:29.732 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:47:29.833 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:47:29.854 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:29.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:29.855 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:47:29.956 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:47:29.977 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:29.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:29.978 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:47:30.079 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:47:30.100 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:30.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:30.101 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:47:30.201 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:47:30.223 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:30.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:30.224 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:47:30.230 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.5 to 1 (10000) 2024-09-20T22:47:30.237 INFO:tasks.rados.rados.0.smithi084.stdout:1418: done (4 left) 2024-09-20T22:47:30.237 INFO:tasks.rados.rados.0.smithi084.stdout:1419: copy_from oid 16 from oid 23 current snap is 144 2024-09-20T22:47:30.237 INFO:tasks.rados.rados.0.smithi084.stdout:1420: delete oid 37 current snap is 144 2024-09-20T22:47:30.237 INFO:tasks.rados.rados.0.smithi084.stdout:1414: finishing write tid 1 to smithi08477203-2 2024-09-20T22:47:30.237 INFO:tasks.rados.rados.0.smithi084.stdout:1414: finishing write tid 2 to smithi08477203-2 2024-09-20T22:47:30.237 INFO:tasks.rados.rados.0.smithi084.stdout:1414: finishing write tid 3 to smithi08477203-2 2024-09-20T22:47:30.237 INFO:tasks.rados.rados.0.smithi084.stdout:1414: finishing write tid 4 to smithi08477203-2 2024-09-20T22:47:30.237 INFO:tasks.rados.rados.0.smithi084.stdout:1414: oid 2 updating version 0 to 160 2024-09-20T22:47:30.237 INFO:tasks.rados.rados.0.smithi084.stdout:1414: oid 2 updating version 160 to 161 2024-09-20T22:47:30.237 INFO:tasks.rados.rados.0.smithi084.stdout:1414: oid 2 version 161 is already newer than 159 2024-09-20T22:47:30.237 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 2 v 161 (ObjNum 507 snap 144 seq_num 507) dirty exists 2024-09-20T22:47:30.238 INFO:tasks.rados.rados.0.smithi084.stdout:1414: left oid 2 (ObjNum 507 snap 144 seq_num 507) 2024-09-20T22:47:30.238 INFO:tasks.rados.rados.0.smithi084.stdout:1415: expect (ObjNum 466 snap 128 seq_num 466) 2024-09-20T22:47:30.324 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:47:30.345 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:30.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:30.346 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:47:30.361 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-20T22:47:30.447 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:47:30.468 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:30.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:30.469 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:47:30.485 INFO:tasks.rados.rados.0.smithi084.stdout:1416: finishing write tid 1 to smithi08477203-35 2024-09-20T22:47:30.485 INFO:tasks.rados.rados.0.smithi084.stdout:1416: finishing write tid 2 to smithi08477203-35 2024-09-20T22:47:30.485 INFO:tasks.rados.rados.0.smithi084.stdout:1416: finishing write tid 3 to smithi08477203-35 2024-09-20T22:47:30.485 INFO:tasks.rados.rados.0.smithi084.stdout:1416: finishing write tid 4 to smithi08477203-35 2024-09-20T22:47:30.485 INFO:tasks.rados.rados.0.smithi084.stdout:1416: oid 35 updating version 0 to 398 2024-09-20T22:47:30.485 INFO:tasks.rados.rados.0.smithi084.stdout:1416: oid 35 version 398 is already newer than 396 2024-09-20T22:47:30.485 INFO:tasks.rados.rados.0.smithi084.stdout:1416: oid 35 version 398 is already newer than 397 2024-09-20T22:47:30.485 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 35 v 398 (ObjNum 508 snap 144 seq_num 508) dirty exists 2024-09-20T22:47:30.485 INFO:tasks.rados.rados.0.smithi084.stdout:1416: left oid 35 (ObjNum 508 snap 144 seq_num 508) 2024-09-20T22:47:30.486 INFO:tasks.rados.rados.0.smithi084.stdout:1417: expect (ObjNum 466 snap 128 seq_num 466) 2024-09-20T22:47:30.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:30 smithi154 ceph-mon[33264]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:47:30.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:30 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/813868920' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:47:30.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:30 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/3152112480' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-20T22:47:30.570 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:47:30.592 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:30.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:30.593 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:47:30.603 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:47:30.614 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:30 smithi084 ceph-mon[40024]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:47:30.614 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:30 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/813868920' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:47:30.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:30 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/3152112480' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-20T22:47:30.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:30 smithi084 ceph-mon[29774]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:47:30.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:30 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/813868920' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:47:30.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:30 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/3152112480' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-20T22:47:30.693 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:47:30.714 INFO:tasks.rados.rados.0.smithi084.stdout:1419: finishing copy_from racing read to smithi08477203-16 2024-09-20T22:47:30.715 INFO:tasks.rados.rados.0.smithi084.stdout:1419: finishing copy_from to smithi08477203-16 2024-09-20T22:47:30.715 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 16 v 236 (ObjNum 428 snap 110 seq_num 428) dirty exists 2024-09-20T22:47:30.715 INFO:tasks.rados.rados.0.smithi084.stdout:1414: done (5 left) 2024-09-20T22:47:30.715 INFO:tasks.rados.rados.0.smithi084.stdout:1415: done (4 left) 2024-09-20T22:47:30.715 INFO:tasks.rados.rados.0.smithi084.stdout:1416: done (3 left) 2024-09-20T22:47:30.715 INFO:tasks.rados.rados.0.smithi084.stdout:1417: done (2 left) 2024-09-20T22:47:30.715 INFO:tasks.rados.rados.0.smithi084.stdout:1419: done (1 left) 2024-09-20T22:47:30.715 INFO:tasks.rados.rados.0.smithi084.stdout:1420: done (0 left) 2024-09-20T22:47:30.715 INFO:tasks.rados.rados.0.smithi084.stdout:1421: rollback oid 35 current snap is 144 2024-09-20T22:47:30.715 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 35 to 134 2024-09-20T22:47:30.715 INFO:tasks.rados.rados.0.smithi084.stdout:1422: snap_remove snap 114 2024-09-20T22:47:30.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:30.716 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:30.717 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:47:30.755 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd unset noscrub 2024-09-20T22:47:30.818 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:47:30.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:30.840 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:30.840 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:47:30.941 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:47:30.962 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:30.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:30.963 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:47:31.022 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-20T22:47:31.023 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [], 2024-09-20T22:47:31.023 INFO:teuthology.orchestra.run.smithi154.stdout: "num_ops": 0 2024-09-20T22:47:31.023 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-20T22:47:31.052 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd reweight 6 1 2024-09-20T22:47:31.063 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:47:31.085 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:31.085 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:47:31.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:31.086 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:47:31.187 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:47:31.209 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:31.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:31.210 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:47:31.245 INFO:tasks.rados.rados.0.smithi084.stdout:1422: done (1 left) 2024-09-20T22:47:31.245 INFO:tasks.rados.rados.0.smithi084.stdout:1423: read oid 28 snap -1 2024-09-20T22:47:31.245 INFO:tasks.rados.rados.0.smithi084.stdout:1423: expect (ObjNum 463 snap 127 seq_num 463) 2024-09-20T22:47:31.246 INFO:tasks.rados.rados.0.smithi084.stdout:1424: delete oid 8 current snap is 144 2024-09-20T22:47:31.246 INFO:tasks.rados.rados.0.smithi084.stdout:1421: finishing rollback tid 0 to smithi08477203-35 2024-09-20T22:47:31.246 INFO:tasks.rados.rados.0.smithi084.stdout:1421: finishing rollback tid 1 to smithi08477203-35 2024-09-20T22:47:31.246 INFO:tasks.rados.rados.0.smithi084.stdout:1421: finishing rollback tid 2 to smithi08477203-35 2024-09-20T22:47:31.246 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 35 v 401 (ObjNum 245 snap 54 seq_num 245) dirty exists 2024-09-20T22:47:31.247 INFO:tasks.rados.rados.0.smithi084.stdout:1421: done (2 left) 2024-09-20T22:47:31.247 INFO:tasks.rados.rados.0.smithi084.stdout:1424: done (1 left) 2024-09-20T22:47:31.247 INFO:tasks.rados.rados.0.smithi084.stdout:1425: rollback oid 45 current snap is 144 2024-09-20T22:47:31.248 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 45 to 134 2024-09-20T22:47:31.248 INFO:tasks.rados.rados.0.smithi084.stdout:1426: copy_from oid 16 from oid 27 current snap is 144 2024-09-20T22:47:31.248 INFO:tasks.rados.rados.0.smithi084.stdout:1427: snap_remove snap 105 2024-09-20T22:47:31.310 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:47:31.332 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:31.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:31.333 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:47:31.335 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:31 smithi084 ceph-mon[40024]: pgmap v456: 168 pgs: 168 active+clean; 287 MiB data, 867 MiB used, 536 GiB / 536 GiB avail; 13 MiB/s rd, 1.7 MiB/s wr, 23 op/s 2024-09-20T22:47:31.335 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:31 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/3152112480' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-20T22:47:31.335 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:31 smithi084 ceph-mon[40024]: osdmap e370: 8 total, 8 up, 6 in 2024-09-20T22:47:31.335 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:31 smithi084 ceph-mon[29774]: pgmap v456: 168 pgs: 168 active+clean; 287 MiB data, 867 MiB used, 536 GiB / 536 GiB avail; 13 MiB/s rd, 1.7 MiB/s wr, 23 op/s 2024-09-20T22:47:31.336 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:31 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/3152112480' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-20T22:47:31.336 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:31 smithi084 ceph-mon[29774]: osdmap e370: 8 total, 8 up, 6 in 2024-09-20T22:47:31.434 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:47:31.456 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:31.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:31.457 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:47:31.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:31 smithi154 ceph-mon[33264]: pgmap v456: 168 pgs: 168 active+clean; 287 MiB data, 867 MiB used, 536 GiB / 536 GiB avail; 13 MiB/s rd, 1.7 MiB/s wr, 23 op/s 2024-09-20T22:47:31.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:31 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/3152112480' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-20T22:47:31.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:31 smithi154 ceph-mon[33264]: osdmap e370: 8 total, 8 up, 6 in 2024-09-20T22:47:31.558 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:47:31.579 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:31.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:31.580 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:47:31.663 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.6 dump_blocked_ops 2024-09-20T22:47:31.681 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:47:31.703 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:31.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:31.704 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:47:31.737 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:47:31.805 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:47:31.826 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:31.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:31.827 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:47:31.907 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:47:31.928 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:47:31.949 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:31.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:31.950 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:47:32.051 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:47:32.073 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:32.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:32.073 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:47:32.174 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:47:32.196 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:32.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:32.197 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:47:32.252 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is unset 2024-09-20T22:47:32.252 INFO:tasks.rados.rados.0.smithi084.stdout:1427: done (3 left) 2024-09-20T22:47:32.253 INFO:tasks.rados.rados.0.smithi084.stdout:1428: read oid 22 snap -1 2024-09-20T22:47:32.253 INFO:tasks.rados.rados.0.smithi084.stdout:1428: expect (ObjNum 468 snap 129 seq_num 468) 2024-09-20T22:47:32.253 INFO:tasks.rados.rados.0.smithi084.stdout:1429: snap_create 2024-09-20T22:47:32.253 INFO:tasks.rados.rados.0.smithi084.stdout:1426: finishing copy_from to smithi08477203-16 2024-09-20T22:47:32.253 INFO:tasks.rados.rados.0.smithi084.stdout:1426: got expected ENOENT (src dne) 2024-09-20T22:47:32.253 INFO:tasks.rados.rados.0.smithi084.stdout:1426: finishing copy_from racing read to smithi08477203-16 2024-09-20T22:47:32.253 INFO:tasks.rados.rados.0.smithi084.stdout:1425: finishing rollback tid 0 to smithi08477203-45 2024-09-20T22:47:32.253 INFO:tasks.rados.rados.0.smithi084.stdout:1425: finishing rollback tid 1 to smithi08477203-45 2024-09-20T22:47:32.253 INFO:tasks.rados.rados.0.smithi084.stdout:1425: finishing rollback tid 2 to smithi08477203-45 2024-09-20T22:47:32.253 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 45 v 149 (ObjNum 466 snap 128 seq_num 466) dirty exists 2024-09-20T22:47:32.254 INFO:tasks.rados.rados.0.smithi084.stdout:1423: expect (ObjNum 463 snap 127 seq_num 463) 2024-09-20T22:47:32.298 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:47:32.321 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:32.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:32.322 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:47:32.340 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:32 smithi154 ceph-mon[33264]: osdmap e371: 8 total, 8 up, 6 in 2024-09-20T22:47:32.340 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:32 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/3610566784' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:47:32.340 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:32 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:47:32.409 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-20T22:47:32.422 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:47:32.423 INFO:tasks.rados.rados.0.smithi084.stdout:1428: expect (ObjNum 468 snap 129 seq_num 468) 2024-09-20T22:47:32.441 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-20T22:47:32.442 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [], 2024-09-20T22:47:32.442 INFO:teuthology.orchestra.run.smithi154.stdout: "complaint_time": 30, 2024-09-20T22:47:32.442 INFO:teuthology.orchestra.run.smithi154.stdout: "num_blocked_ops": 0 2024-09-20T22:47:32.442 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-20T22:47:32.443 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:32.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:32.444 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:47:32.544 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:47:32.566 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:32.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:32.567 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:47:32.581 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:32 smithi084 ceph-mon[40024]: osdmap e371: 8 total, 8 up, 6 in 2024-09-20T22:47:32.582 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:32 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/3610566784' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:47:32.582 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:32 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:47:32.582 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:32 smithi084 ceph-mon[29774]: osdmap e371: 8 total, 8 up, 6 in 2024-09-20T22:47:32.582 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:32 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/3610566784' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:47:32.582 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:32 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:47:32.668 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:47:32.689 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:32.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:32.690 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:47:32.791 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:47:32.815 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:32.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:32.816 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:47:32.917 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:47:32.939 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:32.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:32.940 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:47:33.041 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:47:33.064 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:33.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:33.065 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:47:33.088 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.6 dump_historic_ops 2024-09-20T22:47:33.166 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:47:33.186 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.6 to 1 (10000) 2024-09-20T22:47:33.190 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:33.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:33.191 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:47:33.193 INFO:tasks.rados.rados.0.smithi084.stdout:1423: done (4 left) 2024-09-20T22:47:33.193 INFO:tasks.rados.rados.0.smithi084.stdout:1425: done (3 left) 2024-09-20T22:47:33.193 INFO:tasks.rados.rados.0.smithi084.stdout:1426: done (2 left) 2024-09-20T22:47:33.193 INFO:tasks.rados.rados.0.smithi084.stdout:1428: done (1 left) 2024-09-20T22:47:33.193 INFO:tasks.rados.rados.0.smithi084.stdout:1429: done (0 left) 2024-09-20T22:47:33.193 INFO:tasks.rados.rados.0.smithi084.stdout:1430: write oid 41 current snap is 145 2024-09-20T22:47:33.193 INFO:tasks.rados.rados.0.smithi084.stdout:1430: seq_num 511 ranges {588377=647890,1935828=635840} 2024-09-20T22:47:33.200 INFO:tasks.rados.rados.0.smithi084.stdout:1430: writing smithi08477203-41 from 588377 to 1236267 tid 1 2024-09-20T22:47:33.207 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:47:33.208 INFO:tasks.rados.rados.0.smithi084.stdout:1430: writing smithi08477203-41 from 1935828 to 2571668 tid 2 2024-09-20T22:47:33.208 INFO:tasks.rados.rados.0.smithi084.stdout:1431: snap_create 2024-09-20T22:47:33.210 INFO:tasks.rados.rados.0.smithi084.stdout:1430: finishing write tid 1 to smithi08477203-41 2024-09-20T22:47:33.211 INFO:tasks.rados.rados.0.smithi084.stdout:1430: finishing write tid 2 to smithi08477203-41 2024-09-20T22:47:33.211 INFO:tasks.rados.rados.0.smithi084.stdout:1430: finishing write tid 3 to smithi08477203-41 2024-09-20T22:47:33.211 INFO:tasks.rados.rados.0.smithi084.stdout:1430: finishing write tid 4 to smithi08477203-41 2024-09-20T22:47:33.211 INFO:tasks.rados.rados.0.smithi084.stdout:1430: oid 41 updating version 0 to 107 2024-09-20T22:47:33.212 INFO:tasks.rados.rados.0.smithi084.stdout:1430: oid 41 updating version 107 to 108 2024-09-20T22:47:33.212 INFO:tasks.rados.rados.0.smithi084.stdout:1430: oid 41 version 108 is already newer than 106 2024-09-20T22:47:33.212 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 41 v 108 (ObjNum 511 snap 145 seq_num 511) dirty exists 2024-09-20T22:47:33.212 INFO:tasks.rados.rados.0.smithi084.stdout:1430: left oid 41 (ObjNum 511 snap 145 seq_num 511) 2024-09-20T22:47:33.291 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:47:33.315 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:33.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:33.316 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:47:33.342 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:47:33.364 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:33 smithi084 ceph-mon[40024]: pgmap v459: 168 pgs: 168 active+clean; 287 MiB data, 867 MiB used, 536 GiB / 536 GiB avail; 13 MiB/s rd, 1.7 MiB/s wr, 23 op/s 2024-09-20T22:47:33.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:33 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T22:47:33.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:33 smithi084 ceph-mon[40024]: osdmap e372: 8 total, 8 up, 6 in 2024-09-20T22:47:33.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:33 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/1829037129' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-09-20T22:47:33.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:33 smithi084 ceph-mon[40024]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:47:33.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:33 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/1829037129' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-20T22:47:33.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:33 smithi084 ceph-mon[40024]: osdmap e373: 8 total, 8 up, 6 in 2024-09-20T22:47:33.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:33 smithi084 ceph-mon[29774]: pgmap v459: 168 pgs: 168 active+clean; 287 MiB data, 867 MiB used, 536 GiB / 536 GiB avail; 13 MiB/s rd, 1.7 MiB/s wr, 23 op/s 2024-09-20T22:47:33.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:33 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T22:47:33.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:33 smithi084 ceph-mon[29774]: osdmap e372: 8 total, 8 up, 6 in 2024-09-20T22:47:33.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:33 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/1829037129' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-09-20T22:47:33.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:33 smithi084 ceph-mon[29774]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:47:33.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:33 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/1829037129' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-20T22:47:33.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:33 smithi084 ceph-mon[29774]: osdmap e373: 8 total, 8 up, 6 in 2024-09-20T22:47:33.397 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:33 smithi154 ceph-mon[33264]: pgmap v459: 168 pgs: 168 active+clean; 287 MiB data, 867 MiB used, 536 GiB / 536 GiB avail; 13 MiB/s rd, 1.7 MiB/s wr, 23 op/s 2024-09-20T22:47:33.397 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:33 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T22:47:33.397 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:33 smithi154 ceph-mon[33264]: osdmap e372: 8 total, 8 up, 6 in 2024-09-20T22:47:33.397 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:33 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/1829037129' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-09-20T22:47:33.397 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:33 smithi154 ceph-mon[33264]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:47:33.398 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:33 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/1829037129' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-20T22:47:33.398 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:33 smithi154 ceph-mon[33264]: osdmap e373: 8 total, 8 up, 6 in 2024-09-20T22:47:33.417 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:47:33.439 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:33.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:33.440 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:47:33.541 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:47:33.562 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:33.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:33.563 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:47:33.663 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:47:33.685 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:33.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:33.687 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:47:33.787 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:47:33.809 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:33.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:33.810 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:47:33.836 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-20T22:47:33.836 INFO:teuthology.orchestra.run.smithi154.stdout: "size": 20, 2024-09-20T22:47:33.836 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 600, 2024-09-20T22:47:33.836 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [ 2024-09-20T22:47:33.836 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.836 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:336 7.7 7:e67788e2:::smithi08477203-31:head [copy-from ver 30] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-09-20T22:47:33.836 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:06.542956+0000", 2024-09-20T22:47:33.836 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 267.289944918, 2024-09-20T22:47:33.836 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.019630893999999999, 2024-09-20T22:47:33.836 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:47:33.836 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:47:33.836 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:47:33.837 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:47:33.837 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:47:33.837 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:47:33.837 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 336 2024-09-20T22:47:33.837 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.837 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:47:33.837 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.837 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:47:33.837 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:06.542956+0000", 2024-09-20T22:47:33.837 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:47:33.837 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.837 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.837 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:47:33.837 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:06.542956+0000", 2024-09-20T22:47:33.838 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:47:33.838 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.838 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.838 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:47:33.838 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:06.542957+0000", 2024-09-20T22:47:33.838 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2580000000000001e-06 2024-09-20T22:47:33.838 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.838 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.838 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:47:33.838 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:06.542967+0000", 2024-09-20T22:47:33.838 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0701e-05 2024-09-20T22:47:33.838 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.838 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.839 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:47:33.839 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:06.542987+0000", 2024-09-20T22:47:33.839 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9760000000000001e-05 2024-09-20T22:47:33.839 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.839 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.839 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:47:33.839 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:06.543002+0000", 2024-09-20T22:47:33.839 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4578e-05 2024-09-20T22:47:33.839 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.839 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.839 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:47:33.839 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:06.543070+0000", 2024-09-20T22:47:33.839 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.8263999999999996e-05 2024-09-20T22:47:33.840 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.840 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.840 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:47:33.840 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:06.543159+0000", 2024-09-20T22:47:33.840 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.9241999999999997e-05 2024-09-20T22:47:33.840 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.840 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.840 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 0,3", 2024-09-20T22:47:33.840 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:06.546618+0000", 2024-09-20T22:47:33.840 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0034581949999999998 2024-09-20T22:47:33.840 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.840 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.840 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:47:33.840 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:06.551199+0000", 2024-09-20T22:47:33.841 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0045811309999999996 2024-09-20T22:47:33.841 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.841 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.841 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:47:33.841 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:06.560992+0000", 2024-09-20T22:47:33.841 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0097937270000000003 2024-09-20T22:47:33.841 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.841 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.841 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:47:33.841 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:06.562419+0000", 2024-09-20T22:47:33.841 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0014269040000000001 2024-09-20T22:47:33.841 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.841 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.842 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:47:33.842 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:06.562515+0000", 2024-09-20T22:47:33.842 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.5544999999999994e-05 2024-09-20T22:47:33.842 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.842 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.842 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:47:33.842 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:06.562586+0000", 2024-09-20T22:47:33.842 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.1588999999999995e-05 2024-09-20T22:47:33.842 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:47:33.842 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:47:33.842 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:47:33.842 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.842 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.842 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:654 7.7 7:ea720b85:::smithi08477203-26:head [append 0~0] snapc 17=[17,16,15,13,11,d] ondisk+write+known_if_redirected+supports_pool_eio e136)", 2024-09-20T22:47:33.843 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:35.126065+0000", 2024-09-20T22:47:33.843 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 238.706834988, 2024-09-20T22:47:33.843 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.069757472000000001, 2024-09-20T22:47:33.843 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:47:33.843 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:47:33.843 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:47:33.843 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:47:33.843 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:47:33.843 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:47:33.843 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 654 2024-09-20T22:47:33.843 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.843 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:47:33.843 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.843 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:47:33.844 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.126065+0000", 2024-09-20T22:47:33.844 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:47:33.844 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.844 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.844 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:47:33.844 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.126065+0000", 2024-09-20T22:47:33.844 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:47:33.844 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.844 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.844 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:47:33.844 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.126067+0000", 2024-09-20T22:47:33.844 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.6139999999999999e-06 2024-09-20T22:47:33.844 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.844 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.844 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:47:33.845 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.126074+0000", 2024-09-20T22:47:33.845 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.1509999999999998e-06 2024-09-20T22:47:33.845 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.845 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.845 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:47:33.845 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.126090+0000", 2024-09-20T22:47:33.845 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.6001000000000001e-05 2024-09-20T22:47:33.845 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.845 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.845 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:47:33.845 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.126098+0000", 2024-09-20T22:47:33.845 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.2379999999999997e-06 2024-09-20T22:47:33.845 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.845 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.846 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:47:33.846 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.126155+0000", 2024-09-20T22:47:33.846 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.6279000000000001e-05 2024-09-20T22:47:33.846 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.846 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.846 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op must wait for map", 2024-09-20T22:47:33.846 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.126160+0000", 2024-09-20T22:47:33.846 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.8110000000000002e-06 2024-09-20T22:47:33.846 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.846 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.846 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:47:33.846 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.193660+0000", 2024-09-20T22:47:33.846 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.067499997000000006 2024-09-20T22:47:33.846 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.847 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.847 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:47:33.847 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.193751+0000", 2024-09-20T22:47:33.847 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.1571999999999997e-05 2024-09-20T22:47:33.847 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.847 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.847 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 0,3", 2024-09-20T22:47:33.847 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.193941+0000", 2024-09-20T22:47:33.847 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00018952699999999999 2024-09-20T22:47:33.847 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.847 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.847 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:47:33.847 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.195649+0000", 2024-09-20T22:47:33.847 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001707966 2024-09-20T22:47:33.848 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.848 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.848 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:47:33.848 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.195707+0000", 2024-09-20T22:47:33.848 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.8649e-05 2024-09-20T22:47:33.848 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.848 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.848 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:47:33.848 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.195722+0000", 2024-09-20T22:47:33.848 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4309e-05 2024-09-20T22:47:33.848 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.848 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.848 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:47:33.848 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.195773+0000", 2024-09-20T22:47:33.849 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.1368e-05 2024-09-20T22:47:33.849 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.849 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.849 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:47:33.849 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.195823+0000", 2024-09-20T22:47:33.849 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.9990000000000001e-05 2024-09-20T22:47:33.849 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:47:33.849 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:47:33.849 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:47:33.849 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.849 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.849 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:655 7.7 7:ea720b85:::smithi08477203-26:head [rollback 16] snapc 17=[17,16,15,13,11,d] ondisk+write+known_if_redirected+supports_pool_eio e136)", 2024-09-20T22:47:33.850 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:35.126118+0000", 2024-09-20T22:47:33.850 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 238.70678273499999, 2024-09-20T22:47:33.850 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.069926453, 2024-09-20T22:47:33.850 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:47:33.850 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:47:33.851 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:47:33.851 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:47:33.851 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:47:33.851 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:47:33.851 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 655 2024-09-20T22:47:33.851 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.851 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:47:33.851 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.851 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:47:33.851 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.126118+0000", 2024-09-20T22:47:33.851 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:47:33.851 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.851 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.852 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:47:33.852 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.126118+0000", 2024-09-20T22:47:33.852 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:47:33.852 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.852 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.852 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:47:33.852 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.126118+0000", 2024-09-20T22:47:33.852 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.3000000000000001e-07 2024-09-20T22:47:33.852 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.852 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.852 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:47:33.852 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.126123+0000", 2024-09-20T22:47:33.852 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.7650000000000001e-06 2024-09-20T22:47:33.852 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.853 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.853 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:47:33.853 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.126141+0000", 2024-09-20T22:47:33.853 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.7538e-05 2024-09-20T22:47:33.853 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.853 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.853 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:47:33.853 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.126150+0000", 2024-09-20T22:47:33.853 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.3549999999999994e-06 2024-09-20T22:47:33.853 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.853 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.853 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:47:33.853 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.126204+0000", 2024-09-20T22:47:33.853 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.4289999999999997e-05 2024-09-20T22:47:33.854 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.854 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.854 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting_for_map not empty", 2024-09-20T22:47:33.854 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.126210+0000", 2024-09-20T22:47:33.854 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.361e-06 2024-09-20T22:47:33.854 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.854 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.854 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:47:33.854 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.194528+0000", 2024-09-20T22:47:33.854 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.068318913999999994 2024-09-20T22:47:33.854 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.854 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.854 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:47:33.855 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.194614+0000", 2024-09-20T22:47:33.855 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.5280999999999999e-05 2024-09-20T22:47:33.855 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.855 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.855 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 0,3", 2024-09-20T22:47:33.855 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.194759+0000", 2024-09-20T22:47:33.855 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000145678 2024-09-20T22:47:33.855 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.855 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.855 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:47:33.855 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.195836+0000", 2024-09-20T22:47:33.855 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001076474 2024-09-20T22:47:33.855 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.855 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.856 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:47:33.856 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.195884+0000", 2024-09-20T22:47:33.856 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.8421000000000003e-05 2024-09-20T22:47:33.856 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.856 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.856 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:47:33.856 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.195947+0000", 2024-09-20T22:47:33.856 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.2522999999999998e-05 2024-09-20T22:47:33.856 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.856 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.856 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:47:33.856 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.196003+0000", 2024-09-20T22:47:33.856 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.5911000000000001e-05 2024-09-20T22:47:33.856 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.856 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.857 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:47:33.857 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.196044+0000", 2024-09-20T22:47:33.857 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.1412000000000003e-05 2024-09-20T22:47:33.857 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:47:33.857 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:47:33.857 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:47:33.857 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.857 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.857 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:656 7.7 7:ea720b85:::smithi08477203-26:head [append 0~0] snapc 17=[17,16,15,13,11,d] ondisk+write+known_if_redirected+supports_pool_eio e136)", 2024-09-20T22:47:33.857 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:35.126159+0000", 2024-09-20T22:47:33.857 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 238.70674190099999, 2024-09-20T22:47:33.857 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.070101607999999996, 2024-09-20T22:47:33.857 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:47:33.857 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:47:33.858 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:47:33.858 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:47:33.858 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:47:33.858 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:47:33.858 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 656 2024-09-20T22:47:33.858 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.858 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:47:33.858 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.858 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:47:33.858 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.126159+0000", 2024-09-20T22:47:33.858 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:47:33.858 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.858 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.859 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:47:33.859 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.126159+0000", 2024-09-20T22:47:33.859 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:47:33.859 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.859 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.859 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:47:33.859 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.126159+0000", 2024-09-20T22:47:33.859 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.1500000000000005e-07 2024-09-20T22:47:33.859 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.859 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.859 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:47:33.859 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.126168+0000", 2024-09-20T22:47:33.859 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.4600000000000003e-06 2024-09-20T22:47:33.859 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.860 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.860 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:47:33.860 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.126173+0000", 2024-09-20T22:47:33.860 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.7089999999999999e-06 2024-09-20T22:47:33.860 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.860 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.860 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:47:33.860 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.126177+0000", 2024-09-20T22:47:33.860 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.9040000000000002e-06 2024-09-20T22:47:33.860 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.860 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.860 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:47:33.860 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.126232+0000", 2024-09-20T22:47:33.860 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.5337999999999998e-05 2024-09-20T22:47:33.861 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.861 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.861 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting_for_map not empty", 2024-09-20T22:47:33.861 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.126238+0000", 2024-09-20T22:47:33.861 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.2390000000000002e-06 2024-09-20T22:47:33.861 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.861 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.861 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:47:33.861 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.195116+0000", 2024-09-20T22:47:33.861 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.068878012000000002 2024-09-20T22:47:33.861 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.861 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.861 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:47:33.861 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.195197+0000", 2024-09-20T22:47:33.862 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.0964999999999995e-05 2024-09-20T22:47:33.862 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.862 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.862 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 0,3", 2024-09-20T22:47:33.862 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.195338+0000", 2024-09-20T22:47:33.862 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000141006 2024-09-20T22:47:33.862 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.862 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.862 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:47:33.862 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.195897+0000", 2024-09-20T22:47:33.862 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00055979999999999995 2024-09-20T22:47:33.862 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.862 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.863 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:47:33.863 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.196102+0000", 2024-09-20T22:47:33.863 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00020405099999999999 2024-09-20T22:47:33.863 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.863 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.863 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:47:33.863 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.196160+0000", 2024-09-20T22:47:33.863 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.8677000000000001e-05 2024-09-20T22:47:33.863 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.863 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.863 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:47:33.863 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.196214+0000", 2024-09-20T22:47:33.863 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.414e-05 2024-09-20T22:47:33.863 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.863 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.864 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:47:33.864 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.196260+0000", 2024-09-20T22:47:33.864 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.5791999999999998e-05 2024-09-20T22:47:33.864 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:47:33.864 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:47:33.864 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:47:33.864 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.864 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.864 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:921 7.19 7:9eac4560:::smithi08477203-30:head [write 487436~547559,stat] snapc 1f=[] ondisk+write+known_if_redirected+supports_pool_eio e158)", 2024-09-20T22:47:33.864 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:57.424404+0000", 2024-09-20T22:47:33.864 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 216.40849618999999, 2024-09-20T22:47:33.864 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.018385357000000001, 2024-09-20T22:47:33.864 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:47:33.864 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:47:33.865 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:47:33.865 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:47:33.865 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:47:33.865 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:47:33.865 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 921 2024-09-20T22:47:33.865 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.865 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:47:33.865 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.865 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:47:33.865 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:57.424404+0000", 2024-09-20T22:47:33.865 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:47:33.865 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.865 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.865 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:47:33.866 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:57.424404+0000", 2024-09-20T22:47:33.866 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:47:33.866 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.866 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.866 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:47:33.866 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:57.424406+0000", 2024-09-20T22:47:33.866 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.6169999999999999e-06 2024-09-20T22:47:33.866 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.866 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.866 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:47:33.866 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:57.428411+0000", 2024-09-20T22:47:33.866 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0040051109999999996 2024-09-20T22:47:33.866 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.866 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.867 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:47:33.867 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:57.428423+0000", 2024-09-20T22:47:33.867 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1590000000000001e-05 2024-09-20T22:47:33.867 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.867 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.867 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:47:33.867 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:57.428432+0000", 2024-09-20T22:47:33.867 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.781e-06 2024-09-20T22:47:33.867 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.867 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.867 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:47:33.867 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:57.431029+0000", 2024-09-20T22:47:33.867 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0025962139999999999 2024-09-20T22:47:33.867 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.868 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.868 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:47:33.868 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:57.431170+0000", 2024-09-20T22:47:33.868 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00014111199999999999 2024-09-20T22:47:33.868 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.868 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.868 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 2,7", 2024-09-20T22:47:33.868 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:57.431332+0000", 2024-09-20T22:47:33.868 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00016258599999999999 2024-09-20T22:47:33.868 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.868 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.868 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:47:33.868 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:57.433472+0000", 2024-09-20T22:47:33.868 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0021397120000000002 2024-09-20T22:47:33.868 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.869 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.869 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:47:33.869 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:57.437960+0000", 2024-09-20T22:47:33.869 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0044878410000000002 2024-09-20T22:47:33.869 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.869 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.869 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:47:33.869 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:57.442690+0000", 2024-09-20T22:47:33.869 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0047299250000000003 2024-09-20T22:47:33.869 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.869 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.869 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:47:33.869 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:57.442743+0000", 2024-09-20T22:47:33.869 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.3368999999999997e-05 2024-09-20T22:47:33.870 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.870 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.870 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:47:33.870 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:57.442790+0000", 2024-09-20T22:47:33.870 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.6499000000000003e-05 2024-09-20T22:47:33.870 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:47:33.870 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:47:33.870 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:47:33.870 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.870 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.870 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:1017 7.19 7:9cc0b7e2:::smithi08477203-46:head [copy-from ver 99] snapc 25=[25,24,23,21] ondisk+write+known_if_redirected+supports_pool_eio e166)", 2024-09-20T22:47:33.870 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:44:05.130713+0000", 2024-09-20T22:47:33.870 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 208.70218735399999, 2024-09-20T22:47:33.871 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.021441722999999999, 2024-09-20T22:47:33.871 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:47:33.871 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:47:33.871 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:47:33.871 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:47:33.871 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:47:33.871 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:47:33.871 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 1017 2024-09-20T22:47:33.871 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.871 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:47:33.871 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.871 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:47:33.871 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:05.130713+0000", 2024-09-20T22:47:33.871 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:47:33.871 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.872 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.872 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:47:33.872 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:05.130713+0000", 2024-09-20T22:47:33.872 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:47:33.872 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.872 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.872 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:47:33.872 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:05.130715+0000", 2024-09-20T22:47:33.872 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.66e-06 2024-09-20T22:47:33.872 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.872 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.872 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:47:33.872 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:05.130721+0000", 2024-09-20T22:47:33.872 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.0430000000000004e-06 2024-09-20T22:47:33.873 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.873 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.873 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:47:33.873 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:05.130732+0000", 2024-09-20T22:47:33.873 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1494999999999999e-05 2024-09-20T22:47:33.873 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.873 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.873 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:47:33.873 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:05.130748+0000", 2024-09-20T22:47:33.873 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.5520999999999999e-05 2024-09-20T22:47:33.873 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.873 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.873 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:47:33.873 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:05.130827+0000", 2024-09-20T22:47:33.874 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.8990000000000001e-05 2024-09-20T22:47:33.874 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.874 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.874 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op must wait for map", 2024-09-20T22:47:33.874 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:05.130834+0000", 2024-09-20T22:47:33.874 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.097e-06 2024-09-20T22:47:33.874 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.874 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.874 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:47:33.874 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:05.133176+0000", 2024-09-20T22:47:33.874 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.002341704 2024-09-20T22:47:33.874 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.874 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.874 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:47:33.875 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:05.133285+0000", 2024-09-20T22:47:33.875 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000109473 2024-09-20T22:47:33.875 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.875 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.875 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 2,7", 2024-09-20T22:47:33.875 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:05.142516+0000", 2024-09-20T22:47:33.875 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0092311130000000009 2024-09-20T22:47:33.875 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.875 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.875 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:47:33.875 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:05.150724+0000", 2024-09-20T22:47:33.875 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0082083139999999995 2024-09-20T22:47:33.875 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.875 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.876 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:47:33.876 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:05.150825+0000", 2024-09-20T22:47:33.876 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000100431 2024-09-20T22:47:33.876 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.876 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.876 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:47:33.876 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:05.152061+0000", 2024-09-20T22:47:33.876 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001235912 2024-09-20T22:47:33.876 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.876 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.876 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:47:33.876 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:05.152108+0000", 2024-09-20T22:47:33.876 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.7068999999999999e-05 2024-09-20T22:47:33.876 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.877 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.877 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:47:33.877 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:05.152155+0000", 2024-09-20T22:47:33.877 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.6900999999999997e-05 2024-09-20T22:47:33.877 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:47:33.877 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:47:33.877 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:47:33.877 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.877 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.877 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:1054 7.12 7:4fa4b2b3:::smithi08477203-21:head [copy-from ver 67] snapc 25=[25,24,23,21] ondisk+write+known_if_redirected+supports_pool_eio e166)", 2024-09-20T22:47:33.877 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:44:05.634022+0000", 2024-09-20T22:47:33.877 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 208.198878076, 2024-09-20T22:47:33.877 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.019656218, 2024-09-20T22:47:33.877 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:47:33.878 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:47:33.878 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:47:33.878 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:47:33.878 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:47:33.878 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:47:33.878 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 1054 2024-09-20T22:47:33.878 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.878 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:47:33.878 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.878 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:47:33.878 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:05.634022+0000", 2024-09-20T22:47:33.878 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:47:33.878 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.878 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.879 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:47:33.879 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:05.634022+0000", 2024-09-20T22:47:33.879 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:47:33.879 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.879 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.879 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:47:33.879 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:05.634024+0000", 2024-09-20T22:47:33.879 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.013e-06 2024-09-20T22:47:33.879 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.879 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.879 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:47:33.879 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:05.634035+0000", 2024-09-20T22:47:33.879 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0644e-05 2024-09-20T22:47:33.879 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.879 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.880 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:47:33.880 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:05.634061+0000", 2024-09-20T22:47:33.880 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.6352000000000001e-05 2024-09-20T22:47:33.880 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.880 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.880 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:47:33.880 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:05.634081+0000", 2024-09-20T22:47:33.880 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9347e-05 2024-09-20T22:47:33.880 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.880 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.880 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:47:33.880 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:05.634257+0000", 2024-09-20T22:47:33.880 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000176365 2024-09-20T22:47:33.880 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.881 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.881 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:47:33.881 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:05.634379+0000", 2024-09-20T22:47:33.881 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000122173 2024-09-20T22:47:33.881 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.881 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.881 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 0,2", 2024-09-20T22:47:33.881 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:05.643905+0000", 2024-09-20T22:47:33.881 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0095258679999999998 2024-09-20T22:47:33.881 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.881 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.881 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:47:33.881 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:05.647750+0000", 2024-09-20T22:47:33.881 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.003844564 2024-09-20T22:47:33.882 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.882 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.882 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:47:33.882 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:05.650803+0000", 2024-09-20T22:47:33.882 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.003053114 2024-09-20T22:47:33.882 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.882 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.882 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:47:33.882 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:05.653610+0000", 2024-09-20T22:47:33.882 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0028071459999999999 2024-09-20T22:47:33.882 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.882 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.882 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:47:33.882 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:05.653645+0000", 2024-09-20T22:47:33.883 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.5231000000000002e-05 2024-09-20T22:47:33.883 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.883 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.883 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:47:33.883 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:05.653679+0000", 2024-09-20T22:47:33.883 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.3401000000000001e-05 2024-09-20T22:47:33.883 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:47:33.883 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:47:33.883 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:47:33.883 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.883 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.883 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:1420 7.19 7:9cc0b7e2:::smithi08477203-46:head [write 480733~537415,stat] snapc 2f=[2f] ondisk+write+known_if_redirected+supports_pool_eio e189)", 2024-09-20T22:47:33.883 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:44:28.143096+0000", 2024-09-20T22:47:33.883 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 185.689804557, 2024-09-20T22:47:33.883 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.021594787000000001, 2024-09-20T22:47:33.884 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:47:33.884 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:47:33.884 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:47:33.884 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:47:33.884 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:47:33.884 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:47:33.884 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 1420 2024-09-20T22:47:33.884 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.884 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:47:33.884 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.884 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:47:33.884 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:28.143096+0000", 2024-09-20T22:47:33.884 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:47:33.884 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.885 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.885 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:47:33.885 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:28.143096+0000", 2024-09-20T22:47:33.885 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:47:33.885 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.885 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.885 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:47:33.885 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:28.143098+0000", 2024-09-20T22:47:33.885 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.1840000000000002e-06 2024-09-20T22:47:33.885 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.885 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.885 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:47:33.885 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:28.143663+0000", 2024-09-20T22:47:33.885 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00056486000000000004 2024-09-20T22:47:33.885 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.886 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.886 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:47:33.886 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:28.143684+0000", 2024-09-20T22:47:33.886 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.0792e-05 2024-09-20T22:47:33.886 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.886 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.886 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:47:33.886 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:28.143697+0000", 2024-09-20T22:47:33.886 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.292e-05 2024-09-20T22:47:33.886 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.886 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.886 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:47:33.886 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:28.143788+0000", 2024-09-20T22:47:33.886 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.1218999999999995e-05 2024-09-20T22:47:33.887 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.887 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.887 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op must wait for map", 2024-09-20T22:47:33.887 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:28.143798+0000", 2024-09-20T22:47:33.887 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.7999999999999993e-06 2024-09-20T22:47:33.887 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.887 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.887 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:47:33.887 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:28.148428+0000", 2024-09-20T22:47:33.887 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0046303780000000001 2024-09-20T22:47:33.887 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.887 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.887 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:47:33.887 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:28.148611+0000", 2024-09-20T22:47:33.887 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00018319699999999999 2024-09-20T22:47:33.888 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.888 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.888 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 2,7", 2024-09-20T22:47:33.888 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:28.148807+0000", 2024-09-20T22:47:33.888 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00019560399999999999 2024-09-20T22:47:33.888 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.888 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.888 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:47:33.888 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:28.157814+0000", 2024-09-20T22:47:33.888 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0090068449999999994 2024-09-20T22:47:33.888 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.888 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.888 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:47:33.888 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:28.158746+0000", 2024-09-20T22:47:33.888 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000932076 2024-09-20T22:47:33.889 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.889 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.889 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:47:33.889 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:28.164580+0000", 2024-09-20T22:47:33.889 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0058337850000000002 2024-09-20T22:47:33.889 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.889 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.889 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:47:33.889 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:28.164638+0000", 2024-09-20T22:47:33.889 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.8100999999999998e-05 2024-09-20T22:47:33.889 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.889 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.889 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:47:33.889 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:28.164691+0000", 2024-09-20T22:47:33.890 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.3025999999999997e-05 2024-09-20T22:47:33.890 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:47:33.890 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:47:33.890 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:47:33.890 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.890 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.890 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:1525 7.c 7:3f865ef5:::smithi08477203-28:head [copy-from ver 127] snapc 33=[33] ondisk+write+known_if_redirected+supports_pool_eio e197)", 2024-09-20T22:47:33.890 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:44:36.198921+0000", 2024-09-20T22:47:33.890 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 177.63397965300001, 2024-09-20T22:47:33.890 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.028621220999999999, 2024-09-20T22:47:33.890 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:47:33.890 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:47:33.890 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:47:33.890 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:47:33.890 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:47:33.890 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:47:33.891 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 1525 2024-09-20T22:47:33.891 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.891 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:47:33.891 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.891 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:47:33.891 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.198921+0000", 2024-09-20T22:47:33.891 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:47:33.891 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.891 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.891 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:47:33.891 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.198921+0000", 2024-09-20T22:47:33.891 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:47:33.891 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.891 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.892 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:47:33.892 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.198923+0000", 2024-09-20T22:47:33.892 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.1090000000000001e-06 2024-09-20T22:47:33.892 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.892 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.892 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:47:33.892 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.198934+0000", 2024-09-20T22:47:33.892 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0837e-05 2024-09-20T22:47:33.892 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.892 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.892 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:47:33.892 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.198955+0000", 2024-09-20T22:47:33.892 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.0789e-05 2024-09-20T22:47:33.893 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.893 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.893 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:47:33.893 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.198966+0000", 2024-09-20T22:47:33.893 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1778e-05 2024-09-20T22:47:33.893 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.893 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.893 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:47:33.893 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.199046+0000", 2024-09-20T22:47:33.893 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.9567999999999994e-05 2024-09-20T22:47:33.893 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.893 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.893 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op must wait for map", 2024-09-20T22:47:33.893 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.199054+0000", 2024-09-20T22:47:33.894 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.9349999999999994e-06 2024-09-20T22:47:33.894 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.894 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.894 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:47:33.894 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.201428+0000", 2024-09-20T22:47:33.894 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0023744389999999999 2024-09-20T22:47:33.894 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.894 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.894 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:47:33.894 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.201527+0000", 2024-09-20T22:47:33.894 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.8554000000000005e-05 2024-09-20T22:47:33.894 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.894 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.894 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 3,7", 2024-09-20T22:47:33.895 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.215277+0000", 2024-09-20T22:47:33.895 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.013750326 2024-09-20T22:47:33.895 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.895 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.895 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:47:33.895 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.226629+0000", 2024-09-20T22:47:33.895 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.011351781999999999 2024-09-20T22:47:33.895 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.895 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.895 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:47:33.895 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.227366+0000", 2024-09-20T22:47:33.895 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00073709199999999996 2024-09-20T22:47:33.895 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.895 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.896 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:47:33.896 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.227450+0000", 2024-09-20T22:47:33.896 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.3902999999999999e-05 2024-09-20T22:47:33.896 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.896 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.896 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:47:33.896 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.227489+0000", 2024-09-20T22:47:33.896 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.9103000000000002e-05 2024-09-20T22:47:33.896 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.896 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.896 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:47:33.896 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.227542+0000", 2024-09-20T22:47:33.896 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.3006e-05 2024-09-20T22:47:33.896 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:47:33.896 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:47:33.897 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:47:33.897 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.897 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.897 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:1526 7.c 7:3f865ef5:::smithi08477203-28:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e197)", 2024-09-20T22:47:33.897 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:44:36.198993+0000", 2024-09-20T22:47:33.897 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 177.633907287, 2024-09-20T22:47:33.897 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.022582641000000001, 2024-09-20T22:47:33.897 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:47:33.897 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:47:33.897 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:47:33.897 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:47:33.897 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:47:33.897 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:47:33.897 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 1526 2024-09-20T22:47:33.897 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.898 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:47:33.898 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.898 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:47:33.898 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.198993+0000", 2024-09-20T22:47:33.898 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:47:33.898 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.898 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.898 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:47:33.898 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.198993+0000", 2024-09-20T22:47:33.898 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:47:33.898 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.898 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.898 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:47:33.898 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.198994+0000", 2024-09-20T22:47:33.899 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.2e-07 2024-09-20T22:47:33.899 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.899 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.899 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:47:33.899 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.198996+0000", 2024-09-20T22:47:33.899 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.6410000000000002e-06 2024-09-20T22:47:33.899 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.899 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.899 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:47:33.899 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.199005+0000", 2024-09-20T22:47:33.899 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.8079999999999994e-06 2024-09-20T22:47:33.899 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.899 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.899 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:47:33.899 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.199011+0000", 2024-09-20T22:47:33.900 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.7010000000000001e-06 2024-09-20T22:47:33.900 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.900 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.900 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:47:33.900 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.199112+0000", 2024-09-20T22:47:33.900 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000101335 2024-09-20T22:47:33.900 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.900 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.900 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting_for_map not empty", 2024-09-20T22:47:33.900 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.199117+0000", 2024-09-20T22:47:33.900 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.2979999999999999e-06 2024-09-20T22:47:33.900 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.900 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.900 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:47:33.901 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.201667+0000", 2024-09-20T22:47:33.901 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.002549922 2024-09-20T22:47:33.901 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.901 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.901 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for blocked object", 2024-09-20T22:47:33.901 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.201708+0000", 2024-09-20T22:47:33.901 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.0772999999999997e-05 2024-09-20T22:47:33.901 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.901 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.901 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:47:33.901 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.221443+0000", 2024-09-20T22:47:33.901 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.019735229 2024-09-20T22:47:33.901 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.901 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.901 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:47:33.902 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.221509+0000", 2024-09-20T22:47:33.902 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.5618999999999996e-05 2024-09-20T22:47:33.902 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.902 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.902 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:47:33.902 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.221576+0000", 2024-09-20T22:47:33.902 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.6795e-05 2024-09-20T22:47:33.902 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:47:33.902 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:47:33.902 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:47:33.902 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.902 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.902 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:1542 7.12 7:4b06cbed:::smithi08477203-49:head [copy-from ver 69] snapc 34=[34,33] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2024-09-20T22:47:33.902 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:44:37.209101+0000", 2024-09-20T22:47:33.902 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 176.623799244, 2024-09-20T22:47:33.903 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.019142659999999999, 2024-09-20T22:47:33.903 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:47:33.903 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:47:33.903 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:47:33.903 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:47:33.903 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:47:33.903 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:47:33.903 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 1542 2024-09-20T22:47:33.903 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.903 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:47:33.903 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.903 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:47:33.903 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:37.209101+0000", 2024-09-20T22:47:33.903 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:47:33.903 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.904 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.904 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:47:33.904 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:37.209101+0000", 2024-09-20T22:47:33.904 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:47:33.904 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.904 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.904 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:47:33.904 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:37.209103+0000", 2024-09-20T22:47:33.904 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.454e-06 2024-09-20T22:47:33.904 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.905 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.905 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:47:33.905 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:37.209111+0000", 2024-09-20T22:47:33.905 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.1089999999999998e-06 2024-09-20T22:47:33.905 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.906 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.906 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:47:33.906 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:37.209130+0000", 2024-09-20T22:47:33.906 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.8933e-05 2024-09-20T22:47:33.906 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.906 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.906 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:47:33.906 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:37.209142+0000", 2024-09-20T22:47:33.906 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1843e-05 2024-09-20T22:47:33.906 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.906 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.906 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:47:33.906 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:37.209255+0000", 2024-09-20T22:47:33.906 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000113204 2024-09-20T22:47:33.907 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.907 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.907 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op must wait for map", 2024-09-20T22:47:33.907 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:37.209266+0000", 2024-09-20T22:47:33.907 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1070000000000001e-05 2024-09-20T22:47:33.907 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.907 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.907 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:47:33.907 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:37.210962+0000", 2024-09-20T22:47:33.907 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001696442 2024-09-20T22:47:33.907 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.907 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.907 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:47:33.907 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:37.211044+0000", 2024-09-20T22:47:33.908 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.2007000000000003e-05 2024-09-20T22:47:33.908 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.908 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.908 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 0,2", 2024-09-20T22:47:33.908 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:37.216569+0000", 2024-09-20T22:47:33.908 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.005525126 2024-09-20T22:47:33.908 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.908 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.908 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:47:33.908 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:37.221473+0000", 2024-09-20T22:47:33.908 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0049038739999999999 2024-09-20T22:47:33.908 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.908 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.909 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:47:33.909 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:37.227857+0000", 2024-09-20T22:47:33.909 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0063841109999999996 2024-09-20T22:47:33.909 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.909 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.909 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:47:33.909 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:37.228154+0000", 2024-09-20T22:47:33.909 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00029620999999999998 2024-09-20T22:47:33.909 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.909 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.909 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:47:33.909 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:37.228193+0000", 2024-09-20T22:47:33.909 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.9097000000000002e-05 2024-09-20T22:47:33.909 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.909 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.910 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:47:33.910 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:37.228244+0000", 2024-09-20T22:47:33.910 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.1180000000000001e-05 2024-09-20T22:47:33.910 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:47:33.910 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:47:33.910 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:47:33.910 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.910 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.910 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:1652 7.c 7:31d0211f:::smithi08477203-25:head [copy-from ver 161] snapc 38=[38,37] ondisk+write+known_if_redirected+supports_pool_eio e206)", 2024-09-20T22:47:33.910 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:44:45.159981+0000", 2024-09-20T22:47:33.910 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 168.672919253, 2024-09-20T22:47:33.910 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.029481621, 2024-09-20T22:47:33.910 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:47:33.910 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:47:33.910 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:47:33.911 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:47:33.911 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:47:33.911 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:47:33.911 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 1652 2024-09-20T22:47:33.911 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.911 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:47:33.911 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.911 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:47:33.911 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:45.159981+0000", 2024-09-20T22:47:33.911 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:47:33.911 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.911 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.911 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:47:33.911 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:45.159981+0000", 2024-09-20T22:47:33.912 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:47:33.912 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.912 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.912 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:47:33.912 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:45.159982+0000", 2024-09-20T22:47:33.912 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.8999999999999995e-07 2024-09-20T22:47:33.912 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.912 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.912 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:47:33.912 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:45.159986+0000", 2024-09-20T22:47:33.912 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.732e-06 2024-09-20T22:47:33.912 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.912 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.912 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:47:33.913 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:45.159996+0000", 2024-09-20T22:47:33.913 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.9639999999999998e-06 2024-09-20T22:47:33.913 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.913 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.913 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:47:33.913 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:45.160001+0000", 2024-09-20T22:47:33.913 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.8010000000000003e-06 2024-09-20T22:47:33.913 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.913 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.913 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:47:33.913 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:45.165664+0000", 2024-09-20T22:47:33.913 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0056630869999999998 2024-09-20T22:47:33.913 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.913 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.914 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:47:33.914 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:45.165710+0000", 2024-09-20T22:47:33.914 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.6371e-05 2024-09-20T22:47:33.914 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.914 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.914 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 3,7", 2024-09-20T22:47:33.914 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:45.175729+0000", 2024-09-20T22:47:33.914 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.010018750999999999 2024-09-20T22:47:33.914 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.914 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.914 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:47:33.914 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:45.186279+0000", 2024-09-20T22:47:33.914 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.010550241 2024-09-20T22:47:33.914 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.915 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.915 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:47:33.915 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:45.186520+0000", 2024-09-20T22:47:33.915 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000241394 2024-09-20T22:47:33.915 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.915 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.915 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:47:33.915 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:45.189408+0000", 2024-09-20T22:47:33.915 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0028872849999999999 2024-09-20T22:47:33.915 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.915 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.915 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:47:33.915 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:45.189440+0000", 2024-09-20T22:47:33.915 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.2341000000000001e-05 2024-09-20T22:47:33.915 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.916 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.916 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:47:33.916 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:45.189463+0000", 2024-09-20T22:47:33.916 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.2764000000000001e-05 2024-09-20T22:47:33.916 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:47:33.916 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:47:33.916 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:47:33.916 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.916 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.916 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:1653 7.c 7:31d0211f:::smithi08477203-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e206)", 2024-09-20T22:47:33.916 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:44:45.160009+0000", 2024-09-20T22:47:33.916 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 168.67289105500001, 2024-09-20T22:47:33.916 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.020255434999999999, 2024-09-20T22:47:33.916 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:47:33.916 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:47:33.917 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:47:33.917 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:47:33.917 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:47:33.917 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:47:33.917 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 1653 2024-09-20T22:47:33.917 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.917 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:47:33.917 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.917 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:47:33.917 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:45.160009+0000", 2024-09-20T22:47:33.917 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:47:33.917 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.917 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.917 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:47:33.917 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:45.160009+0000", 2024-09-20T22:47:33.918 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:47:33.918 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.918 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.918 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:47:33.918 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:45.160010+0000", 2024-09-20T22:47:33.918 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.5100000000000001e-07 2024-09-20T22:47:33.918 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.918 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.918 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:47:33.918 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:45.160011+0000", 2024-09-20T22:47:33.918 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.547e-06 2024-09-20T22:47:33.918 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.918 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.918 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:47:33.918 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:45.160015+0000", 2024-09-20T22:47:33.919 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.3749999999999999e-06 2024-09-20T22:47:33.919 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.919 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.919 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:47:33.919 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:45.160018+0000", 2024-09-20T22:47:33.919 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.1250000000000001e-06 2024-09-20T22:47:33.919 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.919 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.919 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:47:33.919 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:45.165846+0000", 2024-09-20T22:47:33.919 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0058279270000000001 2024-09-20T22:47:33.919 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.919 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.919 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for blocked object", 2024-09-20T22:47:33.920 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:45.165873+0000", 2024-09-20T22:47:33.920 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.7059999999999998e-05 2024-09-20T22:47:33.920 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.920 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.920 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:47:33.920 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:45.180130+0000", 2024-09-20T22:47:33.920 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.014256998999999999 2024-09-20T22:47:33.920 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.920 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.920 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:47:33.920 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:45.180199+0000", 2024-09-20T22:47:33.920 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.9550999999999999e-05 2024-09-20T22:47:33.920 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.920 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.921 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:47:33.921 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:45.180265+0000", 2024-09-20T22:47:33.921 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.5500000000000006e-05 2024-09-20T22:47:33.921 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:47:33.921 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:47:33.921 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:47:33.921 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.921 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.921 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:1654 7.19 7:9970b578:::smithi08477203-15:head [copy-from ver 110] snapc 38=[38,37] ondisk+write+known_if_redirected+supports_pool_eio e206)", 2024-09-20T22:47:33.921 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:44:45.160022+0000", 2024-09-20T22:47:33.921 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 168.67287864799999, 2024-09-20T22:47:33.921 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.021661590000000001, 2024-09-20T22:47:33.921 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:47:33.921 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:47:33.922 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:47:33.922 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:47:33.922 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:47:33.922 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:47:33.922 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 1654 2024-09-20T22:47:33.922 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.922 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:47:33.922 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.922 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:47:33.922 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:45.160022+0000", 2024-09-20T22:47:33.922 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:47:33.922 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.922 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.922 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:47:33.923 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:45.160022+0000", 2024-09-20T22:47:33.923 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:47:33.923 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.923 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.923 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:47:33.923 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:45.160022+0000", 2024-09-20T22:47:33.923 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.4299999999999999e-07 2024-09-20T22:47:33.923 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.923 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.923 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:47:33.923 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:45.160024+0000", 2024-09-20T22:47:33.923 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.018e-06 2024-09-20T22:47:33.923 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.923 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.924 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:47:33.924 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:45.160027+0000", 2024-09-20T22:47:33.924 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.9100000000000001e-06 2024-09-20T22:47:33.924 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.924 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.924 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:47:33.924 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:45.160030+0000", 2024-09-20T22:47:33.924 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.7140000000000002e-06 2024-09-20T22:47:33.924 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.924 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.924 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:47:33.924 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:45.160078+0000", 2024-09-20T22:47:33.924 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.7933000000000003e-05 2024-09-20T22:47:33.924 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.924 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.925 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:47:33.925 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:45.160133+0000", 2024-09-20T22:47:33.925 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.5823999999999999e-05 2024-09-20T22:47:33.925 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.925 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.925 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 2,7", 2024-09-20T22:47:33.925 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:45.167724+0000", 2024-09-20T22:47:33.925 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0075904060000000001 2024-09-20T22:47:33.925 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.925 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.925 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:47:33.925 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:45.177349+0000", 2024-09-20T22:47:33.925 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0096249809999999995 2024-09-20T22:47:33.925 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.925 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.926 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:47:33.926 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:45.178986+0000", 2024-09-20T22:47:33.926 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00163722 2024-09-20T22:47:33.926 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.926 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.926 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:47:33.926 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:45.181613+0000", 2024-09-20T22:47:33.926 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.002626888 2024-09-20T22:47:33.926 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.926 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.926 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:47:33.926 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:45.181650+0000", 2024-09-20T22:47:33.926 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.7249000000000002e-05 2024-09-20T22:47:33.926 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.927 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.927 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:47:33.927 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:45.181683+0000", 2024-09-20T22:47:33.927 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.3204000000000003e-05 2024-09-20T22:47:33.927 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:47:33.927 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:47:33.927 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:47:33.927 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.927 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.927 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:2179 7.12 7:4fa4b2b3:::smithi08477203-21:head [write 595047~641755,stat] snapc 53=[53,52,51,50,4b,47,45,3e] ondisk+write+known_if_redirected+supports_pool_eio e254)", 2024-09-20T22:47:33.927 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:45:33.166066+0000", 2024-09-20T22:47:33.927 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 120.66683466000001, 2024-09-20T22:47:33.927 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.020018068999999999, 2024-09-20T22:47:33.927 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:47:33.927 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:47:33.928 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:47:33.928 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:47:33.928 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:47:33.928 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:47:33.928 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 2179 2024-09-20T22:47:33.928 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.928 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:47:33.928 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.928 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:47:33.928 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:33.166066+0000", 2024-09-20T22:47:33.928 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:47:33.928 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.928 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.928 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:47:33.929 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:33.166066+0000", 2024-09-20T22:47:33.929 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:47:33.929 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.929 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.929 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:47:33.929 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:33.166067+0000", 2024-09-20T22:47:33.929 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.6500000000000001e-06 2024-09-20T22:47:33.929 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.929 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.929 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:47:33.929 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:33.167423+0000", 2024-09-20T22:47:33.929 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0013558999999999999 2024-09-20T22:47:33.929 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.929 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.929 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:47:33.930 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:33.167433+0000", 2024-09-20T22:47:33.930 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.4790000000000002e-06 2024-09-20T22:47:33.930 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.930 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.930 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:47:33.930 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:33.167442+0000", 2024-09-20T22:47:33.930 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.1439999999999992e-06 2024-09-20T22:47:33.930 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.930 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.930 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:47:33.930 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:33.167627+0000", 2024-09-20T22:47:33.930 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00018459100000000001 2024-09-20T22:47:33.930 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.930 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.930 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:47:33.931 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:33.167690+0000", 2024-09-20T22:47:33.931 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.3806000000000005e-05 2024-09-20T22:47:33.931 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.931 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.931 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 0,2", 2024-09-20T22:47:33.931 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:33.167834+0000", 2024-09-20T22:47:33.931 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00014365600000000001 2024-09-20T22:47:33.931 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.931 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.931 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:47:33.931 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:33.169264+0000", 2024-09-20T22:47:33.931 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0014299040000000001 2024-09-20T22:47:33.931 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.931 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.932 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:47:33.932 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:33.175836+0000", 2024-09-20T22:47:33.932 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0065721750000000004 2024-09-20T22:47:33.932 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.932 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.932 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:47:33.932 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:33.185986+0000", 2024-09-20T22:47:33.932 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.010149696 2024-09-20T22:47:33.932 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.932 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.932 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:47:33.932 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:33.186040+0000", 2024-09-20T22:47:33.932 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.4601000000000001e-05 2024-09-20T22:47:33.932 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.933 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.933 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:47:33.933 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:33.186084+0000", 2024-09-20T22:47:33.933 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.3467000000000002e-05 2024-09-20T22:47:33.933 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:47:33.933 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:47:33.933 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:47:33.933 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.933 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.933 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:2636 7.c 7:35bf785a:::smithi08477203-29:head [copy-from ver 260] snapc 64=[63,60] ondisk+write+known_if_redirected+supports_pool_eio e295)", 2024-09-20T22:47:33.933 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:46:14.189237+0000", 2024-09-20T22:47:33.933 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 79.643663306999997, 2024-09-20T22:47:33.933 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.01875288, 2024-09-20T22:47:33.934 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:47:33.934 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:47:33.934 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:47:33.934 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:47:33.934 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:47:33.934 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:47:33.934 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 2636 2024-09-20T22:47:33.934 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.934 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:47:33.934 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.934 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:47:33.935 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:14.189237+0000", 2024-09-20T22:47:33.935 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:47:33.935 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.935 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.935 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:47:33.935 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:14.189237+0000", 2024-09-20T22:47:33.935 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:47:33.935 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.935 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.935 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:47:33.935 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:14.189238+0000", 2024-09-20T22:47:33.935 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.8000000000000001e-07 2024-09-20T22:47:33.935 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.935 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.936 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:47:33.936 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:14.189239+0000", 2024-09-20T22:47:33.936 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.939e-06 2024-09-20T22:47:33.936 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.936 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.936 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:47:33.936 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:14.189243+0000", 2024-09-20T22:47:33.936 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.2540000000000001e-06 2024-09-20T22:47:33.936 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.936 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.936 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:47:33.936 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:14.189245+0000", 2024-09-20T22:47:33.936 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.6879999999999999e-06 2024-09-20T22:47:33.936 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.937 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.937 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:47:33.937 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:14.189288+0000", 2024-09-20T22:47:33.937 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.3024e-05 2024-09-20T22:47:33.937 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.937 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.937 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:47:33.937 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:14.189342+0000", 2024-09-20T22:47:33.937 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.3229000000000001e-05 2024-09-20T22:47:33.937 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.937 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.937 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 3,7", 2024-09-20T22:47:33.937 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:14.196998+0000", 2024-09-20T22:47:33.938 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0076564099999999998 2024-09-20T22:47:33.938 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.938 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.938 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:47:33.938 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:14.206729+0000", 2024-09-20T22:47:33.938 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0097304560000000002 2024-09-20T22:47:33.938 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.938 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.938 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:47:33.938 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:14.207550+0000", 2024-09-20T22:47:33.938 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00082114700000000004 2024-09-20T22:47:33.938 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.938 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.938 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:47:33.939 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:14.207871+0000", 2024-09-20T22:47:33.939 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00032147099999999999 2024-09-20T22:47:33.939 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.939 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.939 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:47:33.939 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:14.207939+0000", 2024-09-20T22:47:33.939 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.7900999999999999e-05 2024-09-20T22:47:33.939 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.939 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.939 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:47:33.939 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:14.207990+0000", 2024-09-20T22:47:33.939 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.0980999999999999e-05 2024-09-20T22:47:33.939 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:47:33.939 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:47:33.940 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:47:33.940 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.940 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.940 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:3132 7.19 7:9eac4560:::smithi08477203-30:head [copy-from ver 389] snapc 77=[77,76,75,74,73,70,6b] ondisk+write+known_if_redirected+supports_pool_eio e328)", 2024-09-20T22:47:33.940 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:46:48.209876+0000", 2024-09-20T22:47:33.940 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 45.623024512999997, 2024-09-20T22:47:33.940 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.030505467000000001, 2024-09-20T22:47:33.940 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:47:33.940 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:47:33.940 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:47:33.940 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:47:33.940 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:47:33.940 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:47:33.940 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 3132 2024-09-20T22:47:33.940 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.941 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:47:33.941 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.941 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:47:33.941 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.209876+0000", 2024-09-20T22:47:33.941 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:47:33.941 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.941 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.941 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:47:33.941 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.209876+0000", 2024-09-20T22:47:33.941 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:47:33.941 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.941 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.941 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:47:33.941 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.209876+0000", 2024-09-20T22:47:33.942 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.7e-07 2024-09-20T22:47:33.942 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.942 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.942 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:47:33.942 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.209880+0000", 2024-09-20T22:47:33.942 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.4149999999999999e-06 2024-09-20T22:47:33.942 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.942 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.942 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:47:33.942 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.209886+0000", 2024-09-20T22:47:33.942 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.5819999999999998e-06 2024-09-20T22:47:33.942 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.942 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.942 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:47:33.943 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.209893+0000", 2024-09-20T22:47:33.943 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.1820000000000003e-06 2024-09-20T22:47:33.943 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.943 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.943 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:47:33.943 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.209945+0000", 2024-09-20T22:47:33.943 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.2528999999999998e-05 2024-09-20T22:47:33.943 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.943 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.943 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:47:33.943 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.210028+0000", 2024-09-20T22:47:33.943 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.3067999999999998e-05 2024-09-20T22:47:33.943 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.943 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.944 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 2,7", 2024-09-20T22:47:33.944 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.223189+0000", 2024-09-20T22:47:33.944 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.013160579 2024-09-20T22:47:33.944 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.944 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.944 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:47:33.944 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.232498+0000", 2024-09-20T22:47:33.944 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0093088760000000007 2024-09-20T22:47:33.944 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.944 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.944 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:47:33.944 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.240058+0000", 2024-09-20T22:47:33.944 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0075599930000000001 2024-09-20T22:47:33.944 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.944 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.944 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:47:33.945 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.240300+0000", 2024-09-20T22:47:33.945 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00024190999999999999 2024-09-20T22:47:33.945 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.945 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.945 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:47:33.945 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.240343+0000", 2024-09-20T22:47:33.945 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.2972000000000001e-05 2024-09-20T22:47:33.945 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.945 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.945 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:47:33.945 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.240381+0000", 2024-09-20T22:47:33.945 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.8890999999999997e-05 2024-09-20T22:47:33.946 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:47:33.946 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:47:33.946 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:47:33.946 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.946 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.946 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:3133 7.19 7:9eac4560:::smithi08477203-30:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e328)", 2024-09-20T22:47:33.946 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:46:48.209912+0000", 2024-09-20T22:47:33.946 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 45.622988747999997, 2024-09-20T22:47:33.946 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.020991129000000001, 2024-09-20T22:47:33.946 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:47:33.946 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:47:33.946 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:47:33.946 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:47:33.946 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:47:33.947 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:47:33.947 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 3133 2024-09-20T22:47:33.947 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.947 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:47:33.947 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.947 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:47:33.947 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.209912+0000", 2024-09-20T22:47:33.947 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:47:33.947 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.947 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.947 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:47:33.947 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.209912+0000", 2024-09-20T22:47:33.947 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:47:33.947 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.948 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.948 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:47:33.948 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.209912+0000", 2024-09-20T22:47:33.948 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.0399999999999996e-07 2024-09-20T22:47:33.948 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.948 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.948 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:47:33.948 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.209915+0000", 2024-09-20T22:47:33.948 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.8890000000000002e-06 2024-09-20T22:47:33.948 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.948 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.948 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:47:33.948 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.209922+0000", 2024-09-20T22:47:33.948 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.153e-06 2024-09-20T22:47:33.948 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.949 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.949 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:47:33.949 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.209929+0000", 2024-09-20T22:47:33.949 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.5400000000000001e-06 2024-09-20T22:47:33.949 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.949 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.949 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:47:33.949 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.210307+0000", 2024-09-20T22:47:33.949 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00037839900000000002 2024-09-20T22:47:33.949 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.949 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.949 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for blocked object", 2024-09-20T22:47:33.949 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.210386+0000", 2024-09-20T22:47:33.949 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.8536999999999997e-05 2024-09-20T22:47:33.950 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.950 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.950 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:47:33.950 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.230565+0000", 2024-09-20T22:47:33.950 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.020179389999999998 2024-09-20T22:47:33.950 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.950 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.950 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:47:33.950 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.230784+0000", 2024-09-20T22:47:33.950 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00021929 2024-09-20T22:47:33.950 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.950 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.950 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:47:33.950 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.230903+0000", 2024-09-20T22:47:33.951 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00011832699999999999 2024-09-20T22:47:33.951 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:47:33.951 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:47:33.951 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:47:33.951 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.951 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.951 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:3567 7.12 7:4d20232a:::smithi08477203-34:head [copy-from ver 321] snapc 8c=[8c,8b,8a,89,88,87,86,85,84,80,7f,7a,74,73,6b] ondisk+write+known_if_redirected+supports_pool_eio e362)", 2024-09-20T22:47:33.951 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:47:22.219645+0000", 2024-09-20T22:47:33.951 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 11.613255389999999, 2024-09-20T22:47:33.951 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.021532453, 2024-09-20T22:47:33.951 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:47:33.951 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:47:33.951 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:47:33.951 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:47:33.952 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:47:33.952 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:47:33.952 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 3567 2024-09-20T22:47:33.952 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.952 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:47:33.952 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.952 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:47:33.952 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:22.219645+0000", 2024-09-20T22:47:33.952 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:47:33.952 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.952 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.952 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:47:33.952 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:22.219645+0000", 2024-09-20T22:47:33.952 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:47:33.953 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.953 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.953 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:47:33.953 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:22.219647+0000", 2024-09-20T22:47:33.953 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.3649999999999998e-06 2024-09-20T22:47:33.953 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.953 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.953 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:47:33.953 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:22.219665+0000", 2024-09-20T22:47:33.953 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.7368e-05 2024-09-20T22:47:33.953 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.953 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.953 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:47:33.953 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:22.219685+0000", 2024-09-20T22:47:33.953 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9771000000000001e-05 2024-09-20T22:47:33.954 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.954 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.954 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:47:33.954 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:22.219695+0000", 2024-09-20T22:47:33.954 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0074e-05 2024-09-20T22:47:33.954 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.954 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.954 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:47:33.954 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:22.219752+0000", 2024-09-20T22:47:33.954 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.7633000000000002e-05 2024-09-20T22:47:33.954 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.954 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.954 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:47:33.955 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:22.219833+0000", 2024-09-20T22:47:33.955 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.0370999999999993e-05 2024-09-20T22:47:33.955 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.955 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.955 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 2,7", 2024-09-20T22:47:33.955 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:22.230376+0000", 2024-09-20T22:47:33.955 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.010543703 2024-09-20T22:47:33.955 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.955 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.955 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:47:33.955 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:22.240312+0000", 2024-09-20T22:47:33.955 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.009935213 2024-09-20T22:47:33.955 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.955 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.956 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:47:33.956 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:22.240836+0000", 2024-09-20T22:47:33.956 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00052436400000000004 2024-09-20T22:47:33.956 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.956 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.956 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:47:33.956 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:22.241104+0000", 2024-09-20T22:47:33.956 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00026767000000000002 2024-09-20T22:47:33.956 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.956 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.956 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:47:33.956 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:22.241142+0000", 2024-09-20T22:47:33.956 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.8071999999999997e-05 2024-09-20T22:47:33.956 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.956 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.957 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:47:33.957 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:22.241178+0000", 2024-09-20T22:47:33.957 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.5849000000000002e-05 2024-09-20T22:47:33.957 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:47:33.957 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:47:33.957 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:47:33.957 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.957 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.957 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:3580 7.d 7:b97fe42f:::smithi08477203-36:head [copy-from ver 100] snapc 8c=[8c,8b,8a,89,88,87,86,85,84,80,7f,7a,74,73,6b] ondisk+write+known_if_redirected+supports_pool_eio e362)", 2024-09-20T22:47:33.957 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:47:22.529195+0000", 2024-09-20T22:47:33.957 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 11.303705676, 2024-09-20T22:47:33.957 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.023634354, 2024-09-20T22:47:33.957 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:47:33.958 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:47:33.958 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:47:33.958 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:47:33.958 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:47:33.958 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:47:33.958 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 3580 2024-09-20T22:47:33.958 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.958 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:47:33.958 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.958 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:47:33.958 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:22.529195+0000", 2024-09-20T22:47:33.958 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:47:33.958 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.958 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.959 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:47:33.959 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:22.529195+0000", 2024-09-20T22:47:33.959 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:47:33.959 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.959 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.959 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:47:33.959 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:22.529195+0000", 2024-09-20T22:47:33.959 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.3000000000000001e-07 2024-09-20T22:47:33.959 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.959 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.959 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:47:33.959 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:22.529199+0000", 2024-09-20T22:47:33.959 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.416e-06 2024-09-20T22:47:33.959 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.959 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.960 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:47:33.960 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:22.529204+0000", 2024-09-20T22:47:33.960 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.6319999999999996e-06 2024-09-20T22:47:33.960 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.960 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.960 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:47:33.960 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:22.529209+0000", 2024-09-20T22:47:33.960 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.5850000000000001e-06 2024-09-20T22:47:33.960 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.960 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.960 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:47:33.960 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:22.529345+0000", 2024-09-20T22:47:33.960 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00013665899999999999 2024-09-20T22:47:33.960 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.960 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.961 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:47:33.961 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:22.529428+0000", 2024-09-20T22:47:33.961 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.2408000000000001e-05 2024-09-20T22:47:33.961 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.961 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.961 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 1,3", 2024-09-20T22:47:33.961 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:22.536949+0000", 2024-09-20T22:47:33.961 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0075212220000000002 2024-09-20T22:47:33.961 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.961 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.961 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:47:33.961 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:22.539734+0000", 2024-09-20T22:47:33.961 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0027850829999999998 2024-09-20T22:47:33.961 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.961 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.962 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:47:33.962 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:22.552673+0000", 2024-09-20T22:47:33.962 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.012938586 2024-09-20T22:47:33.962 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.962 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.962 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:47:33.962 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:22.552718+0000", 2024-09-20T22:47:33.962 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.5071999999999998e-05 2024-09-20T22:47:33.962 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.962 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.962 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:47:33.962 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:22.552764+0000", 2024-09-20T22:47:33.962 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.6570999999999998e-05 2024-09-20T22:47:33.962 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:47:33.962 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:47:33.963 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:47:33.963 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:22.552829+0000", 2024-09-20T22:47:33.963 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.4690000000000006e-05 2024-09-20T22:47:33.963 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:47:33.963 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:47:33.963 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:47:33.963 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:47:33.963 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:47:33.963 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-20T22:47:33.963 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:47:33.984 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd reweight 7 1 2024-09-20T22:47:33.985 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:33.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:33.986 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:47:34.088 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:47:34.109 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:34.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:34.110 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:47:34.194 INFO:tasks.rados.rados.0.smithi084.stdout:1430: done (1 left) 2024-09-20T22:47:34.194 INFO:tasks.rados.rados.0.smithi084.stdout:1431: done (0 left) 2024-09-20T22:47:34.195 INFO:tasks.rados.rados.0.smithi084.stdout:1432: read oid 23 snap -1 2024-09-20T22:47:34.195 INFO:tasks.rados.rados.0.smithi084.stdout:1432: expect (ObjNum 428 snap 110 seq_num 428) 2024-09-20T22:47:34.195 INFO:tasks.rados.rados.0.smithi084.stdout:1433: read oid 8 snap -1 2024-09-20T22:47:34.195 INFO:tasks.rados.rados.0.smithi084.stdout:1433: expect deleted 2024-09-20T22:47:34.195 INFO:tasks.rados.rados.0.smithi084.stdout:1434: snap_create 2024-09-20T22:47:34.200 INFO:tasks.rados.rados.0.smithi084.stdout:1432: expect (ObjNum 428 snap 110 seq_num 428) 2024-09-20T22:47:34.211 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:47:34.232 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:34.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:34.233 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:47:34.334 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:47:34.356 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:34.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:34.357 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:47:34.457 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:47:34.459 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:47:34.459 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:47:34.459 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:47:34.459 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:47:34.476 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:47:34.476 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:47:34.479 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:34.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:34.480 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:47:34.492 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-20T22:47:34.492 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T22:47:34.509 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:47:34.510 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:47:34.528 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:47:34.528 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:47:34.547 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:47:34.547 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:47:34.548 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:47:34.565 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:47:34.565 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:47:34.581 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:47:34.605 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:34.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:34.606 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:47:34.706 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:47:34.728 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:34.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:34.729 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:47:34.811 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-20T22:47:34.830 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:47:34.852 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:34.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:34.853 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:47:34.953 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:47:34.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:34.976 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:34.976 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:47:35.056 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:47:35.077 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:47:35.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:35.099 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:35.099 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:47:35.111 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-20T22:47:35.193 INFO:tasks.rados.rados.0.smithi084.stdout:1432: done (2 left) 2024-09-20T22:47:35.193 INFO:tasks.rados.rados.0.smithi084.stdout:1433: done (1 left) 2024-09-20T22:47:35.193 INFO:tasks.rados.rados.0.smithi084.stdout:1434: done (0 left) 2024-09-20T22:47:35.193 INFO:tasks.rados.rados.0.smithi084.stdout:1435: snap_remove snap 120 2024-09-20T22:47:35.200 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:47:35.222 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:35.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:35.223 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:47:35.324 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:47:35.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:35.346 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:35.347 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:47:35.447 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:47:35.463 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:35 smithi084 ceph-mon[40024]: pgmap v462: 168 pgs: 6 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 156 active+clean; 276 MiB data, 855 MiB used, 536 GiB / 536 GiB avail; 15 MiB/s rd, 1.7 MiB/s wr, 16 op/s 2024-09-20T22:47:35.463 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:35 smithi084 ceph-mon[40024]: 7.7 scrub starts 2024-09-20T22:47:35.463 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:35 smithi084 ceph-mon[40024]: 7.7 scrub ok 2024-09-20T22:47:35.463 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:35 smithi084 ceph-mon[40024]: osdmap e374: 8 total, 8 up, 6 in 2024-09-20T22:47:35.463 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:35 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/1818238233' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:47:35.464 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:35 smithi084 ceph-mon[29774]: pgmap v462: 168 pgs: 6 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 156 active+clean; 276 MiB data, 855 MiB used, 536 GiB / 536 GiB avail; 15 MiB/s rd, 1.7 MiB/s wr, 16 op/s 2024-09-20T22:47:35.464 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:35 smithi084 ceph-mon[29774]: 7.7 scrub starts 2024-09-20T22:47:35.464 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:35 smithi084 ceph-mon[29774]: 7.7 scrub ok 2024-09-20T22:47:35.464 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:35 smithi084 ceph-mon[29774]: osdmap e374: 8 total, 8 up, 6 in 2024-09-20T22:47:35.464 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:35 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/1818238233' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:47:35.464 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:35 smithi154 ceph-mon[33264]: pgmap v462: 168 pgs: 6 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 156 active+clean; 276 MiB data, 855 MiB used, 536 GiB / 536 GiB avail; 15 MiB/s rd, 1.7 MiB/s wr, 16 op/s 2024-09-20T22:47:35.464 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:35 smithi154 ceph-mon[33264]: 7.7 scrub starts 2024-09-20T22:47:35.464 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:35 smithi154 ceph-mon[33264]: 7.7 scrub ok 2024-09-20T22:47:35.464 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:35 smithi154 ceph-mon[33264]: osdmap e374: 8 total, 8 up, 6 in 2024-09-20T22:47:35.465 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:35 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/1818238233' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:47:35.470 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:35.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:35.471 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:47:35.565 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-20T22:47:35.565 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:47:35.571 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:47:35.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:35.593 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:35.594 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:47:35.695 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:47:35.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:35.718 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:35.718 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:47:35.785 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:47:35.819 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:47:35.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:35.841 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:35.841 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:47:35.942 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:47:35.963 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:35.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:35.964 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:47:36.064 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:47:36.085 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:36.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:36.086 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:47:36.187 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:47:36.206 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.7 to 1 (10000) 2024-09-20T22:47:36.210 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:36.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:36.211 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:47:36.212 INFO:tasks.rados.rados.0.smithi084.stdout:1435: done (0 left) 2024-09-20T22:47:36.212 INFO:tasks.rados.rados.0.smithi084.stdout:1436: read oid 26 snap -1 2024-09-20T22:47:36.212 INFO:tasks.rados.rados.0.smithi084.stdout:1436: expect (ObjNum 502 snap 143 seq_num 502) 2024-09-20T22:47:36.212 INFO:tasks.rados.rados.0.smithi084.stdout:1437: read oid 13 snap -1 2024-09-20T22:47:36.212 INFO:tasks.rados.rados.0.smithi084.stdout:1437: expect deleted 2024-09-20T22:47:36.212 INFO:tasks.rados.rados.0.smithi084.stdout:1438: copy_from oid 17 from oid 46 current snap is 147 2024-09-20T22:47:36.212 INFO:tasks.rados.rados.0.smithi084.stdout:1439: delete oid 14 current snap is 147 2024-09-20T22:47:36.216 INFO:tasks.rados.rados.0.smithi084.stdout:1438: finishing copy_from to smithi08477203-17 2024-09-20T22:47:36.216 INFO:tasks.rados.rados.0.smithi084.stdout:1438: got expected ENOENT (src dne) 2024-09-20T22:47:36.216 INFO:tasks.rados.rados.0.smithi084.stdout:1438: finishing copy_from racing read to smithi08477203-17 2024-09-20T22:47:36.223 INFO:tasks.rados.rados.0.smithi084.stdout:1437: done (3 left) 2024-09-20T22:47:36.223 INFO:tasks.rados.rados.0.smithi084.stdout:1438: done (2 left) 2024-09-20T22:47:36.223 INFO:tasks.rados.rados.0.smithi084.stdout:1439: done (1 left) 2024-09-20T22:47:36.223 INFO:tasks.rados.rados.0.smithi084.stdout:1440: copy_from oid 13 from oid 21 current snap is 147 2024-09-20T22:47:36.223 INFO:tasks.rados.rados.0.smithi084.stdout:1441: read oid 29 snap -1 2024-09-20T22:47:36.223 INFO:tasks.rados.rados.0.smithi084.stdout:1441: expect (ObjNum 400 snap 101 seq_num 400) 2024-09-20T22:47:36.223 INFO:tasks.rados.rados.0.smithi084.stdout:1442: delete oid 19 current snap is 147 2024-09-20T22:47:36.225 INFO:tasks.rados.rados.0.smithi084.stdout:1436: expect (ObjNum 502 snap 143 seq_num 502) 2024-09-20T22:47:36.312 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:47:36.317 INFO:tasks.rados.rados.0.smithi084.stdout:1440: finishing copy_from racing read to smithi08477203-13 2024-09-20T22:47:36.317 INFO:tasks.rados.rados.0.smithi084.stdout:1441: expect (ObjNum 400 snap 101 seq_num 400) 2024-09-20T22:47:36.322 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.7 dump_blocked_ops 2024-09-20T22:47:36.334 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:36.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:36.334 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:47:36.364 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:36 smithi084 ceph-mon[40024]: osdmap e375: 8 total, 8 up, 6 in 2024-09-20T22:47:36.364 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:36 smithi084 ceph-mon[40024]: 4.8 scrub starts 2024-09-20T22:47:36.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:36 smithi084 ceph-mon[40024]: 4.8 scrub ok 2024-09-20T22:47:36.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:36 smithi084 ceph-mon[40024]: 7.15 scrub starts 2024-09-20T22:47:36.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:36 smithi084 ceph-mon[40024]: 7.15 scrub ok 2024-09-20T22:47:36.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:36 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/174430488' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-09-20T22:47:36.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:36 smithi084 ceph-mon[29774]: osdmap e375: 8 total, 8 up, 6 in 2024-09-20T22:47:36.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:36 smithi084 ceph-mon[29774]: 4.8 scrub starts 2024-09-20T22:47:36.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:36 smithi084 ceph-mon[29774]: 4.8 scrub ok 2024-09-20T22:47:36.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:36 smithi084 ceph-mon[29774]: 7.15 scrub starts 2024-09-20T22:47:36.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:36 smithi084 ceph-mon[29774]: 7.15 scrub ok 2024-09-20T22:47:36.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:36 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/174430488' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-09-20T22:47:36.435 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:47:36.456 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:36.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:36.457 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:47:36.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:36 smithi154 ceph-mon[33264]: osdmap e375: 8 total, 8 up, 6 in 2024-09-20T22:47:36.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:36 smithi154 ceph-mon[33264]: 4.8 scrub starts 2024-09-20T22:47:36.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:36 smithi154 ceph-mon[33264]: 4.8 scrub ok 2024-09-20T22:47:36.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:36 smithi154 ceph-mon[33264]: 7.15 scrub starts 2024-09-20T22:47:36.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:36 smithi154 ceph-mon[33264]: 7.15 scrub ok 2024-09-20T22:47:36.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:36 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/174430488' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-09-20T22:47:36.533 INFO:tasks.rados.rados.0.smithi084.stdout:1440: finishing copy_from to smithi08477203-13 2024-09-20T22:47:36.533 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 13 v 519 (ObjNum 501 snap 143 seq_num 501) dirty exists 2024-09-20T22:47:36.533 INFO:tasks.rados.rados.0.smithi084.stdout:1436: done (3 left) 2024-09-20T22:47:36.533 INFO:tasks.rados.rados.0.smithi084.stdout:1440: done (2 left) 2024-09-20T22:47:36.533 INFO:tasks.rados.rados.0.smithi084.stdout:1441: done (1 left) 2024-09-20T22:47:36.533 INFO:tasks.rados.rados.0.smithi084.stdout:1442: done (0 left) 2024-09-20T22:47:36.534 INFO:tasks.rados.rados.0.smithi084.stdout:1443: rollback oid 5 current snap is 147 2024-09-20T22:47:36.534 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 5 to 145 2024-09-20T22:47:36.534 INFO:tasks.rados.rados.0.smithi084.stdout:1444: rollback oid 48 current snap is 147 2024-09-20T22:47:36.534 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 48 to 139 2024-09-20T22:47:36.534 INFO:tasks.rados.rados.0.smithi084.stdout:1445: snap_remove snap 130 2024-09-20T22:47:36.557 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:47:36.567 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:47:36.579 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:36.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:36.580 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:47:36.681 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:47:36.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:36.702 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:36.702 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:47:36.803 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:47:36.824 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:36.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:36.825 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:47:36.926 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:47:36.947 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:36.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:36.949 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:47:36.955 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-09-20T22:47:36.955 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph pg dump --format=json 2024-09-20T22:47:37.030 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-20T22:47:37.030 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:47:37.049 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:47:37.071 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:37.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:37.072 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:47:37.173 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:47:37.194 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:37.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:37.195 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:47:37.212 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is unset 2024-09-20T22:47:37.217 INFO:tasks.rados.rados.0.smithi084.stdout:1445: done (2 left) 2024-09-20T22:47:37.217 INFO:tasks.rados.rados.0.smithi084.stdout:1446: snap_create 2024-09-20T22:47:37.217 INFO:tasks.rados.rados.0.smithi084.stdout:1443: finishing rollback tid 1 to smithi08477203-5 2024-09-20T22:47:37.217 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 5 v 392 (ObjNum 129 snap 0 seq_num 2091830368) dirty dne 2024-09-20T22:47:37.218 INFO:tasks.rados.rados.0.smithi084.stdout:1444: finishing rollback tid 0 to smithi08477203-48 2024-09-20T22:47:37.218 INFO:tasks.rados.rados.0.smithi084.stdout:1444: finishing rollback tid 1 to smithi08477203-48 2024-09-20T22:47:37.218 INFO:tasks.rados.rados.0.smithi084.stdout:1444: finishing rollback tid 2 to smithi08477203-48 2024-09-20T22:47:37.218 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 48 v 300 (ObjNum 424 snap 109 seq_num 424) dirty exists 2024-09-20T22:47:37.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:37 smithi154 ceph-mon[33264]: pgmap v465: 168 pgs: 6 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 156 active+clean; 276 MiB data, 855 MiB used, 536 GiB / 536 GiB avail; 15 MiB/s rd, 1.7 MiB/s wr, 16 op/s 2024-09-20T22:47:37.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:37 smithi154 ceph-mon[33264]: 5.b scrub starts 2024-09-20T22:47:37.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:37 smithi154 ceph-mon[33264]: 5.b scrub ok 2024-09-20T22:47:37.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:37 smithi154 ceph-mon[33264]: 6.11 scrub starts 2024-09-20T22:47:37.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:37 smithi154 ceph-mon[33264]: 6.11 scrub ok 2024-09-20T22:47:37.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:37 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/174430488' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-20T22:47:37.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:37 smithi154 ceph-mon[33264]: osdmap e376: 8 total, 8 up, 6 in 2024-09-20T22:47:37.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:37 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/2198925689' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:47:37.296 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:47:37.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:37.319 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:37.319 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:47:37.420 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:47:37.421 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:47:37.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:37.442 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:37.442 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:47:37.512 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-20T22:47:37.513 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:37 smithi084 ceph-mon[40024]: pgmap v465: 168 pgs: 6 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 156 active+clean; 276 MiB data, 855 MiB used, 536 GiB / 536 GiB avail; 15 MiB/s rd, 1.7 MiB/s wr, 16 op/s 2024-09-20T22:47:37.513 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:37 smithi084 ceph-mon[40024]: 5.b scrub starts 2024-09-20T22:47:37.513 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:37 smithi084 ceph-mon[40024]: 5.b scrub ok 2024-09-20T22:47:37.513 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:37 smithi084 ceph-mon[40024]: 6.11 scrub starts 2024-09-20T22:47:37.514 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:37 smithi084 ceph-mon[40024]: 6.11 scrub ok 2024-09-20T22:47:37.514 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:37 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/174430488' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-20T22:47:37.514 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:37 smithi084 ceph-mon[40024]: osdmap e376: 8 total, 8 up, 6 in 2024-09-20T22:47:37.514 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:37 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/2198925689' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:47:37.514 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:37 smithi084 ceph-mon[29774]: pgmap v465: 168 pgs: 6 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 156 active+clean; 276 MiB data, 855 MiB used, 536 GiB / 536 GiB avail; 15 MiB/s rd, 1.7 MiB/s wr, 16 op/s 2024-09-20T22:47:37.514 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:37 smithi084 ceph-mon[29774]: 5.b scrub starts 2024-09-20T22:47:37.514 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:37 smithi084 ceph-mon[29774]: 5.b scrub ok 2024-09-20T22:47:37.514 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:37 smithi084 ceph-mon[29774]: 6.11 scrub starts 2024-09-20T22:47:37.515 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:37 smithi084 ceph-mon[29774]: 6.11 scrub ok 2024-09-20T22:47:37.515 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:37 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/174430488' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-20T22:47:37.515 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:37 smithi084 ceph-mon[29774]: osdmap e376: 8 total, 8 up, 6 in 2024-09-20T22:47:37.515 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:37 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/2198925689' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:47:37.543 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:47:37.576 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:37.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:37.577 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:47:37.678 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:47:37.699 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:37.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:37.700 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:47:37.713 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.7 dump_historic_ops 2024-09-20T22:47:37.801 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:47:37.823 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:37.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:37.824 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:47:37.925 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:47:37.949 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:37.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:37.950 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:47:37.962 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:47:38.051 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:47:38.073 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:38.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:38.074 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:47:38.175 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:47:38.191 INFO:tasks.rados.rados.0.smithi084.stdout:1443: done (2 left) 2024-09-20T22:47:38.191 INFO:tasks.rados.rados.0.smithi084.stdout:1444: done (1 left) 2024-09-20T22:47:38.192 INFO:tasks.rados.rados.0.smithi084.stdout:1446: done (0 left) 2024-09-20T22:47:38.192 INFO:tasks.rados.rados.0.smithi084.stdout:1447: snap_remove snap 143 2024-09-20T22:47:38.198 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:38.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:38.199 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:47:38.300 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:47:38.301 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:47:38.322 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:38.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:38.323 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:47:38.350 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:38 smithi154 ceph-mon[33264]: 5.c scrub starts 2024-09-20T22:47:38.350 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:38 smithi154 ceph-mon[33264]: 5.c scrub ok 2024-09-20T22:47:38.350 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:38 smithi154 ceph-mon[33264]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T22:47:38.350 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:38 smithi154 ceph-mon[33264]: Cluster is now healthy 2024-09-20T22:47:38.350 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:38 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/2198925689' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T22:47:38.350 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:38 smithi154 ceph-mon[33264]: osdmap e377: 8 total, 8 up, 6 in 2024-09-20T22:47:38.350 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:38 smithi154 ceph-mon[33264]: 6.10 scrub starts 2024-09-20T22:47:38.350 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:38 smithi154 ceph-mon[33264]: 6.10 scrub ok 2024-09-20T22:47:38.350 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:38 smithi154 ceph-mon[33264]: 5.9 scrub starts 2024-09-20T22:47:38.351 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:38 smithi154 ceph-mon[33264]: 5.9 scrub ok 2024-09-20T22:47:38.351 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:38 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:47:38.351 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:38 smithi154 ceph-mon[33264]: osdmap e378: 8 total, 8 up, 6 in 2024-09-20T22:47:38.423 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:47:38.424 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:47:38 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: ::ffff:172.21.15.154 - - [20/Sep/2024:22:47:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-20T22:47:38.424 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:38 smithi084 ceph-mon[40024]: 5.c scrub starts 2024-09-20T22:47:38.424 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:38 smithi084 ceph-mon[40024]: 5.c scrub ok 2024-09-20T22:47:38.424 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:38 smithi084 ceph-mon[40024]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T22:47:38.424 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:38 smithi084 ceph-mon[40024]: Cluster is now healthy 2024-09-20T22:47:38.425 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:38 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/2198925689' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T22:47:38.425 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:38 smithi084 ceph-mon[40024]: osdmap e377: 8 total, 8 up, 6 in 2024-09-20T22:47:38.425 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:38 smithi084 ceph-mon[40024]: 6.10 scrub starts 2024-09-20T22:47:38.425 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:38 smithi084 ceph-mon[40024]: 6.10 scrub ok 2024-09-20T22:47:38.425 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:38 smithi084 ceph-mon[40024]: 5.9 scrub starts 2024-09-20T22:47:38.425 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:38 smithi084 ceph-mon[40024]: 5.9 scrub ok 2024-09-20T22:47:38.425 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:38 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:47:38.425 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:38 smithi084 ceph-mon[40024]: osdmap e378: 8 total, 8 up, 6 in 2024-09-20T22:47:38.425 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:38 smithi084 ceph-mon[29774]: 5.c scrub starts 2024-09-20T22:47:38.425 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:38 smithi084 ceph-mon[29774]: 5.c scrub ok 2024-09-20T22:47:38.425 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:38 smithi084 ceph-mon[29774]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T22:47:38.426 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:38 smithi084 ceph-mon[29774]: Cluster is now healthy 2024-09-20T22:47:38.426 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:38 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/2198925689' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T22:47:38.426 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:38 smithi084 ceph-mon[29774]: osdmap e377: 8 total, 8 up, 6 in 2024-09-20T22:47:38.426 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:38 smithi084 ceph-mon[29774]: 6.10 scrub starts 2024-09-20T22:47:38.426 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:38 smithi084 ceph-mon[29774]: 6.10 scrub ok 2024-09-20T22:47:38.426 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:38 smithi084 ceph-mon[29774]: 5.9 scrub starts 2024-09-20T22:47:38.426 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:38 smithi084 ceph-mon[29774]: 5.9 scrub ok 2024-09-20T22:47:38.426 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:38 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:47:38.426 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:38 smithi084 ceph-mon[29774]: osdmap e378: 8 total, 8 up, 6 in 2024-09-20T22:47:38.446 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:38.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:38.447 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:47:38.451 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-20T22:47:38.451 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:47:38.548 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:47:38.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:38.570 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:38.570 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:47:38.670 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:47:38.692 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:38.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:38.693 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:47:38.751 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-20T22:47:38.754 INFO:teuthology.orchestra.run.smithi084.stderr:dumped all 2024-09-20T22:47:38.794 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:47:38.816 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:38.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:38.817 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:47:38.917 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:47:38.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:38.941 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:38.941 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:47:38.956 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:47:38.956 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:47:38.969 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:47:38.970 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:47:38.985 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:47:38.985 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:47:39.002 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-20T22:47:39.002 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T22:47:39.019 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:47:39.019 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:47:39.036 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:47:39.036 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:47:39.041 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-20T22:47:39.041 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:47:39.053 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:47:39.053 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:47:39.062 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:39.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:39.064 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:47:39.070 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:47:39.071 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:47:39.164 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:47:39.186 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:39.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:39.187 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:47:39.200 INFO:tasks.rados.rados.0.smithi084.stdout:1447: done (0 left) 2024-09-20T22:47:39.200 INFO:tasks.rados.rados.0.smithi084.stdout:1448: delete oid 35 current snap is 148 2024-09-20T22:47:39.207 INFO:tasks.rados.rados.0.smithi084.stdout:1448: done (0 left) 2024-09-20T22:47:39.207 INFO:tasks.rados.rados.0.smithi084.stdout:1449: rollback oid 30 current snap is 148 2024-09-20T22:47:39.207 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 30 to 136 2024-09-20T22:47:39.207 INFO:tasks.rados.rados.0.smithi084.stdout:1450: write oid 50 current snap is 148 2024-09-20T22:47:39.207 INFO:tasks.rados.rados.0.smithi084.stdout:1450: seq_num 515 ranges {655605=548886,1653403=643178} 2024-09-20T22:47:39.214 INFO:tasks.rados.rados.0.smithi084.stdout:1450: writing smithi08477203-50 from 655605 to 1204491 tid 1 2024-09-20T22:47:39.221 INFO:tasks.rados.rados.0.smithi084.stdout:1450: writing smithi08477203-50 from 1653403 to 2296581 tid 2 2024-09-20T22:47:39.221 INFO:tasks.rados.rados.0.smithi084.stdout:1451: write oid 10 current snap is 148 2024-09-20T22:47:39.221 INFO:tasks.rados.rados.0.smithi084.stdout:1449: finishing rollback tid 0 to smithi08477203-30 2024-09-20T22:47:39.221 INFO:tasks.rados.rados.0.smithi084.stdout:1451: seq_num 516 ranges {672412=624135,1995532=676880} 2024-09-20T22:47:39.225 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:39 smithi084 ceph-mon[29774]: pgmap v468: 168 pgs: 6 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 156 active+clean; 276 MiB data, 855 MiB used, 536 GiB / 536 GiB avail 2024-09-20T22:47:39.225 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:39 smithi084 ceph-mon[29774]: 5.11 scrub starts 2024-09-20T22:47:39.225 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:39 smithi084 ceph-mon[29774]: 5.11 scrub ok 2024-09-20T22:47:39.225 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:39 smithi084 ceph-mon[29774]: 6.b scrub starts 2024-09-20T22:47:39.225 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:39 smithi084 ceph-mon[29774]: 6.b scrub ok 2024-09-20T22:47:39.225 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:39 smithi084 ceph-mon[29774]: 4.d scrub starts 2024-09-20T22:47:39.225 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:39 smithi084 ceph-mon[29774]: 4.d scrub ok 2024-09-20T22:47:39.225 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:39 smithi084 ceph-mon[29774]: 5.f scrub starts 2024-09-20T22:47:39.225 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:39 smithi084 ceph-mon[29774]: 5.f scrub ok 2024-09-20T22:47:39.225 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:39 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/2167592965' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:47:39.225 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:39 smithi084 ceph-mon[29774]: osdmap e379: 8 total, 8 up, 6 in 2024-09-20T22:47:39.228 INFO:tasks.rados.rados.0.smithi084.stdout:1451: writing smithi08477203-10 from 672412 to 1296547 tid 1 2024-09-20T22:47:39.236 INFO:tasks.rados.rados.0.smithi084.stdout:1451: writing smithi08477203-10 from 1995532 to 2672412 tid 2 2024-09-20T22:47:39.236 INFO:tasks.rados.rados.0.smithi084.stdout:1452: snap_create 2024-09-20T22:47:39.236 INFO:tasks.rados.rados.0.smithi084.stdout:1449: finishing rollback tid 1 to smithi08477203-30 2024-09-20T22:47:39.236 INFO:tasks.rados.rados.0.smithi084.stdout:1449: finishing rollback tid 2 to smithi08477203-30 2024-09-20T22:47:39.237 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 30 v 402 (ObjNum 416 snap 105 seq_num 416) dirty exists 2024-09-20T22:47:39.237 INFO:tasks.rados.rados.0.smithi084.stdout:1450: finishing write tid 1 to smithi08477203-50 2024-09-20T22:47:39.237 INFO:tasks.rados.rados.0.smithi084.stdout:1450: finishing write tid 2 to smithi08477203-50 2024-09-20T22:47:39.237 INFO:tasks.rados.rados.0.smithi084.stdout:1450: finishing write tid 3 to smithi08477203-50 2024-09-20T22:47:39.237 INFO:tasks.rados.rados.0.smithi084.stdout:1450: finishing write tid 4 to smithi08477203-50 2024-09-20T22:47:39.237 INFO:tasks.rados.rados.0.smithi084.stdout:1450: oid 50 updating version 0 to 250 2024-09-20T22:47:39.237 INFO:tasks.rados.rados.0.smithi084.stdout:1450: oid 50 updating version 250 to 251 2024-09-20T22:47:39.237 INFO:tasks.rados.rados.0.smithi084.stdout:1450: oid 50 updating version 251 to 252 2024-09-20T22:47:39.237 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 50 v 252 (ObjNum 515 snap 148 seq_num 515) dirty exists 2024-09-20T22:47:39.237 INFO:tasks.rados.rados.0.smithi084.stdout:1450: left oid 50 (ObjNum 515 snap 148 seq_num 515) 2024-09-20T22:47:39.237 INFO:tasks.rados.rados.0.smithi084.stdout:1451: finishing write tid 1 to smithi08477203-10 2024-09-20T22:47:39.240 INFO:tasks.rados.rados.0.smithi084.stdout:1451: finishing write tid 2 to smithi08477203-10 2024-09-20T22:47:39.240 INFO:tasks.rados.rados.0.smithi084.stdout:1451: finishing write tid 3 to smithi08477203-10 2024-09-20T22:47:39.240 INFO:tasks.rados.rados.0.smithi084.stdout:1451: finishing write tid 4 to smithi08477203-10 2024-09-20T22:47:39.240 INFO:tasks.rados.rados.0.smithi084.stdout:1451: oid 10 updating version 0 to 520 2024-09-20T22:47:39.240 INFO:tasks.rados.rados.0.smithi084.stdout:1451: oid 10 updating version 520 to 522 2024-09-20T22:47:39.240 INFO:tasks.rados.rados.0.smithi084.stdout:1451: oid 10 version 522 is already newer than 521 2024-09-20T22:47:39.240 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 10 v 522 (ObjNum 516 snap 148 seq_num 516) dirty exists 2024-09-20T22:47:39.240 INFO:tasks.rados.rados.0.smithi084.stdout:1451: left oid 10 (ObjNum 516 snap 148 seq_num 516) 2024-09-20T22:47:39.288 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:47:39.309 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:39.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:39.310 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:47:39.411 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:47:39.433 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:39.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:39.434 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:47:39.504 INFO:journalctl@ceph.iscsi.iscsi.a.smithi154.stdout:Sep 20 22:47:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-iscsi-iscsi-a[59730]: debug there is no tcmu-runner data available 2024-09-20T22:47:39.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:39 smithi154 ceph-mon[33264]: pgmap v468: 168 pgs: 6 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 156 active+clean; 276 MiB data, 855 MiB used, 536 GiB / 536 GiB avail 2024-09-20T22:47:39.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:39 smithi154 ceph-mon[33264]: 5.11 scrub starts 2024-09-20T22:47:39.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:39 smithi154 ceph-mon[33264]: 5.11 scrub ok 2024-09-20T22:47:39.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:39 smithi154 ceph-mon[33264]: 6.b scrub starts 2024-09-20T22:47:39.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:39 smithi154 ceph-mon[33264]: 6.b scrub ok 2024-09-20T22:47:39.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:39 smithi154 ceph-mon[33264]: 4.d scrub starts 2024-09-20T22:47:39.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:39 smithi154 ceph-mon[33264]: 4.d scrub ok 2024-09-20T22:47:39.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:39 smithi154 ceph-mon[33264]: 5.f scrub starts 2024-09-20T22:47:39.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:39 smithi154 ceph-mon[33264]: 5.f scrub ok 2024-09-20T22:47:39.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:39 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/2167592965' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:47:39.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:39 smithi154 ceph-mon[33264]: osdmap e379: 8 total, 8 up, 6 in 2024-09-20T22:47:39.534 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:47:39.539 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:39 smithi084 ceph-mon[40024]: pgmap v468: 168 pgs: 6 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 156 active+clean; 276 MiB data, 855 MiB used, 536 GiB / 536 GiB avail 2024-09-20T22:47:39.539 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:39 smithi084 ceph-mon[40024]: 5.11 scrub starts 2024-09-20T22:47:39.539 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:39 smithi084 ceph-mon[40024]: 5.11 scrub ok 2024-09-20T22:47:39.539 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:39 smithi084 ceph-mon[40024]: 6.b scrub starts 2024-09-20T22:47:39.539 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:39 smithi084 ceph-mon[40024]: 6.b scrub ok 2024-09-20T22:47:39.539 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:39 smithi084 ceph-mon[40024]: 4.d scrub starts 2024-09-20T22:47:39.539 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:39 smithi084 ceph-mon[40024]: 4.d scrub ok 2024-09-20T22:47:39.539 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:39 smithi084 ceph-mon[40024]: 5.f scrub starts 2024-09-20T22:47:39.539 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:39 smithi084 ceph-mon[40024]: 5.f scrub ok 2024-09-20T22:47:39.539 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:39 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/2167592965' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:47:39.539 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:39 smithi084 ceph-mon[40024]: osdmap e379: 8 total, 8 up, 6 in 2024-09-20T22:47:39.556 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:39.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:39.557 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:47:39.658 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:47:39.679 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:39.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:39.680 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:47:39.781 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:47:39.834 INFO:teuthology.orchestra.run.smithi084.stdout:{"pg_ready":true,"pg_map":{"version":469,"stamp":"2024-09-20T22:47:38.187485+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":289088788,"num_objects":335,"num_object_clones":89,"num_object_copies":1005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":335,"num_whiteouts":6,"num_read":3605,"num_read_kb":2642674,"num_write":2159,"num_write_kb":653898,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":329,"num_bytes_recovered":77274003,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6156,"ondisk_log_size":6156,"up":504,"acting":504,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":504,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":562470912,"kb_used":875208,"kb_used_data":688064,"kb_used_omap":9,"kb_used_meta":186998,"kb_avail":561595704,"statfs":{"total":575970213888,"available":575074000896,"internally_reserved":0,"allocated":704577536,"data_stored":937726185,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9562,"internal_metadata":191486630},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.996887"},"pg_stats":[{"pgid":"3.1f","version":"86'2","reported_seq":306,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-20T22:47:33.208489+0000","last_change":"2024-09-20T22:46:42.206673+0000","last_active":"2024-09-20T22:47:33.208489+0000","last_peered":"2024-09-20T22:47:33.208489+0000","last_clean":"2024-09-20T22:47:33.208489+0000","last_became_active":"2024-09-20T22:45:40.207225+0000","last_became_peered":"2024-09-20T22:45:40.207225+0000","last_unstale":"2024-09-20T22:47:33.208489+0000","last_undegraded":"2024-09-20T22:47:33.208489+0000","last_fullsized":"2024-09-20T22:47:33.208489+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'2","last_scrub_stamp":"2024-09-20T22:46:42.206600+0000","last_deep_scrub":"86'2","last_deep_scrub_stamp":"2024-09-20T22:42:39.664421+0000","last_clean_scrub_stamp":"2024-09-20T22:46:42.206600+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:47:54.236322+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":436,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"74'9","reported_seq":503,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-20T22:47:33.535799+0000","last_change":"2024-09-20T22:46:20.889599+0000","last_active":"2024-09-20T22:47:33.535799+0000","last_peered":"2024-09-20T22:47:33.535799+0000","last_clean":"2024-09-20T22:47:33.535799+0000","last_became_active":"2024-09-20T22:43:57.428786+0000","last_became_peered":"2024-09-20T22:43:57.428786+0000","last_unstale":"2024-09-20T22:47:33.535799+0000","last_undegraded":"2024-09-20T22:47:33.535799+0000","last_fullsized":"2024-09-20T22:47:33.535799+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-09-20T22:46:20.889510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:46:20.889510+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:47:36.787279+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":632,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-20T22:47:33.239057+0000","last_change":"2024-09-20T22:46:22.955529+0000","last_active":"2024-09-20T22:47:33.239057+0000","last_peered":"2024-09-20T22:47:33.239057+0000","last_clean":"2024-09-20T22:47:33.239057+0000","last_became_active":"2024-09-20T22:41:42.182024+0000","last_became_peered":"2024-09-20T22:41:42.182024+0000","last_unstale":"2024-09-20T22:47:33.239057+0000","last_undegraded":"2024-09-20T22:47:33.239057+0000","last_fullsized":"2024-09-20T22:47:33.239057+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:46:22.955460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:46:22.955460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:47:34.698768+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":460,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-20T22:47:33.207199+0000","last_change":"2024-09-20T22:46:45.177175+0000","last_active":"2024-09-20T22:47:33.207199+0000","last_peered":"2024-09-20T22:47:33.207199+0000","last_clean":"2024-09-20T22:47:33.207199+0000","last_became_active":"2024-09-20T22:43:57.429983+0000","last_became_peered":"2024-09-20T22:43:57.429983+0000","last_unstale":"2024-09-20T22:47:33.207199+0000","last_undegraded":"2024-09-20T22:47:33.207199+0000","last_fullsized":"2024-09-20T22:47:33.207199+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:46:45.177105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:46:45.177105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:13.549053+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":628,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-20T22:47:33.203367+0000","last_change":"2024-09-20T22:45:40.053329+0000","last_active":"2024-09-20T22:47:33.203367+0000","last_peered":"2024-09-20T22:47:33.203367+0000","last_clean":"2024-09-20T22:47:33.203367+0000","last_became_active":"2024-09-20T22:41:44.832099+0000","last_became_peered":"2024-09-20T22:41:44.832099+0000","last_unstale":"2024-09-20T22:47:33.203367+0000","last_undegraded":"2024-09-20T22:47:33.203367+0000","last_fullsized":"2024-09-20T22:47:33.203367+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:45:40.053263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:45:40.053263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:47:35.121414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":632,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-20T22:47:33.204982+0000","last_change":"2024-09-20T22:44:58.994410+0000","last_active":"2024-09-20T22:47:33.204982+0000","last_peered":"2024-09-20T22:47:33.204982+0000","last_clean":"2024-09-20T22:47:33.204982+0000","last_became_active":"2024-09-20T22:41:38.157298+0000","last_became_peered":"2024-09-20T22:41:38.157298+0000","last_unstale":"2024-09-20T22:47:33.204982+0000","last_undegraded":"2024-09-20T22:47:33.204982+0000","last_fullsized":"2024-09-20T22:47:33.204982+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:44:58.994340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:44:58.994340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:47:35.121299+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"74'15","reported_seq":675,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-20T22:47:33.203358+0000","last_change":"2024-09-20T22:45:40.242495+0000","last_active":"2024-09-20T22:47:33.203358+0000","last_peered":"2024-09-20T22:47:33.203358+0000","last_clean":"2024-09-20T22:47:33.203358+0000","last_became_active":"2024-09-20T22:45:40.214879+0000","last_became_peered":"2024-09-20T22:45:40.214879+0000","last_unstale":"2024-09-20T22:47:33.203358+0000","last_undegraded":"2024-09-20T22:47:33.203358+0000","last_fullsized":"2024-09-20T22:47:33.203358+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-09-20T22:44:41.108205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:44:41.108205+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:47:35.121273+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":458,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-20T22:47:33.535648+0000","last_change":"2024-09-20T22:47:19.805833+0000","last_active":"2024-09-20T22:47:33.535648+0000","last_peered":"2024-09-20T22:47:33.535648+0000","last_clean":"2024-09-20T22:47:33.535648+0000","last_became_active":"2024-09-20T22:43:57.423685+0000","last_became_peered":"2024-09-20T22:43:57.423685+0000","last_unstale":"2024-09-20T22:47:33.535648+0000","last_undegraded":"2024-09-20T22:47:33.535648+0000","last_fullsized":"2024-09-20T22:47:33.535648+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:47:19.805701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:47:19.805701+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:41.119667+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.19","version":"372'235","reported_seq":1401,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-20T22:47:33.536897+0000","last_change":"2024-09-20T22:47:32.805168+0000","last_active":"2024-09-20T22:47:33.536897+0000","last_peered":"2024-09-20T22:47:33.536897+0000","last_clean":"2024-09-20T22:47:33.536897+0000","last_became_active":"2024-09-20T22:43:39.130950+0000","last_became_peered":"2024-09-20T22:43:39.130950+0000","last_unstale":"2024-09-20T22:47:33.536897+0000","last_undegraded":"2024-09-20T22:47:33.536897+0000","last_fullsized":"2024-09-20T22:47:33.536897+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":5,"last_scrub":"241'147","last_scrub_stamp":"2024-09-20T22:45:23.839574+0000","last_deep_scrub":"241'147","last_deep_scrub_stamp":"2024-09-20T22:45:23.839574+0000","last_clean_scrub_stamp":"2024-09-20T22:45:23.839574+0000","objects_scrubbed":2,"log_size":235,"log_dups_size":0,"ondisk_log_size":235,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.0034669499999999999,"stat_sum":{"num_bytes":13164021,"num_objects":5,"num_object_clones":3,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":87,"num_read_kb":82678,"num_write":76,"num_write_kb":32174,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"6b","length":"1"},{"start":"74","length":"1"},{"start":"89","length":"1"},{"start":"90","length":"1"}]},{"pgid":"3.1d","version":"0'0","reported_seq":647,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-20T22:47:33.207964+0000","last_change":"2024-09-20T22:46:01.000010+0000","last_active":"2024-09-20T22:47:33.207964+0000","last_peered":"2024-09-20T22:47:33.207964+0000","last_clean":"2024-09-20T22:47:33.207964+0000","last_became_active":"2024-09-20T22:43:57.429854+0000","last_became_peered":"2024-09-20T22:43:57.429854+0000","last_unstale":"2024-09-20T22:47:33.207964+0000","last_undegraded":"2024-09-20T22:47:33.207964+0000","last_fullsized":"2024-09-20T22:47:33.207964+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:46:00.999939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:46:00.999939+0000","last_clean_scrub_stamp":"2024-09-20T22:46:00.999939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:47:34.304639+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"74'9","reported_seq":511,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-20T22:47:33.203095+0000","last_change":"2024-09-20T22:46:45.096042+0000","last_active":"2024-09-20T22:47:33.203095+0000","last_peered":"2024-09-20T22:47:33.203095+0000","last_clean":"2024-09-20T22:47:33.203095+0000","last_became_active":"2024-09-20T22:45:40.220428+0000","last_became_peered":"2024-09-20T22:45:40.220428+0000","last_unstale":"2024-09-20T22:47:33.203095+0000","last_undegraded":"2024-09-20T22:47:33.203095+0000","last_fullsized":"2024-09-20T22:47:33.203095+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-09-20T22:46:45.095957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:46:45.095957+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:06.043333+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":639,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-20T22:47:33.207454+0000","last_change":"2024-09-20T22:46:52.239575+0000","last_active":"2024-09-20T22:47:33.207454+0000","last_peered":"2024-09-20T22:47:33.207454+0000","last_clean":"2024-09-20T22:47:33.207454+0000","last_became_active":"2024-09-20T22:45:40.213185+0000","last_became_peered":"2024-09-20T22:45:40.213185+0000","last_unstale":"2024-09-20T22:47:33.207454+0000","last_undegraded":"2024-09-20T22:47:33.207454+0000","last_fullsized":"2024-09-20T22:47:33.207454+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:46:52.239508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:46:52.239508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:05.847887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":243,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-20T22:47:33.239274+0000","last_change":"2024-09-20T22:46:29.801951+0000","last_active":"2024-09-20T22:47:33.239274+0000","last_peered":"2024-09-20T22:47:33.239274+0000","last_clean":"2024-09-20T22:47:33.239274+0000","last_became_active":"2024-09-20T22:45:40.215834+0000","last_became_peered":"2024-09-20T22:45:40.215834+0000","last_unstale":"2024-09-20T22:47:33.239274+0000","last_undegraded":"2024-09-20T22:47:33.239274+0000","last_fullsized":"2024-09-20T22:47:33.239274+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:46:29.801854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:46:29.801854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:47:49.174382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.18","version":"372'217","reported_seq":1404,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-20T22:47:33.537473+0000","last_change":"2024-09-20T22:47:32.807526+0000","last_active":"2024-09-20T22:47:33.537473+0000","last_peered":"2024-09-20T22:47:33.537473+0000","last_clean":"2024-09-20T22:47:33.537473+0000","last_became_active":"2024-09-20T22:45:43.170008+0000","last_became_peered":"2024-09-20T22:45:43.170008+0000","last_unstale":"2024-09-20T22:47:33.537473+0000","last_undegraded":"2024-09-20T22:47:33.537473+0000","last_fullsized":"2024-09-20T22:47:33.537473+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":5,"last_scrub":"282'128","last_scrub_stamp":"2024-09-20T22:46:01.882247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:46:01.882247+0000","objects_scrubbed":1,"log_size":217,"log_dups_size":0,"ondisk_log_size":217,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0054549619999999998,"stat_sum":{"num_bytes":6396308,"num_objects":6,"num_object_clones":4,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":93,"num_read_kb":94824,"num_write":64,"num_write_kb":23540,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":6792292,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"6b","length":"1"},{"start":"74","length":"1"},{"start":"89","length":"1"},{"start":"90","length":"1"}]},{"pgid":"3.1c","version":"0'0","reported_seq":647,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-20T22:47:33.207853+0000","last_change":"2024-09-20T22:45:53.934021+0000","last_active":"2024-09-20T22:47:33.207853+0000","last_peered":"2024-09-20T22:47:33.207853+0000","last_clean":"2024-09-20T22:47:33.207853+0000","last_became_active":"2024-09-20T22:43:57.430028+0000","last_became_peered":"2024-09-20T22:43:57.430028+0000","last_unstale":"2024-09-20T22:47:33.207853+0000","last_undegraded":"2024-09-20T22:47:33.207853+0000","last_fullsized":"2024-09-20T22:47:33.207853+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:45:53.933937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:45:53.933937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:47:34.304610+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"181'7","reported_seq":495,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-20T22:47:33.203631+0000","last_change":"2024-09-20T22:45:36.106966+0000","last_active":"2024-09-20T22:47:33.203631+0000","last_peered":"2024-09-20T22:47:33.203631+0000","last_clean":"2024-09-20T22:47:33.203631+0000","last_became_active":"2024-09-20T22:43:57.441058+0000","last_became_peered":"2024-09-20T22:43:57.441058+0000","last_unstale":"2024-09-20T22:47:33.203631+0000","last_undegraded":"2024-09-20T22:47:33.203631+0000","last_fullsized":"2024-09-20T22:47:33.203631+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"181'7","last_scrub_stamp":"2024-09-20T22:45:36.106880+0000","last_deep_scrub":"181'7","last_deep_scrub_stamp":"2024-09-20T22:44:20.259598+0000","last_clean_scrub_stamp":"2024-09-20T22:45:36.106880+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:47:35.121372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":6,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":648,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-20T22:47:33.413367+0000","last_change":"2024-09-20T22:45:40.211399+0000","last_active":"2024-09-20T22:47:33.413367+0000","last_peered":"2024-09-20T22:47:33.413367+0000","last_clean":"2024-09-20T22:47:33.413367+0000","last_became_active":"2024-09-20T22:45:40.210886+0000","last_became_peered":"2024-09-20T22:45:40.210886+0000","last_unstale":"2024-09-20T22:47:33.413367+0000","last_undegraded":"2024-09-20T22:47:33.413367+0000","last_fullsized":"2024-09-20T22:47:33.413367+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:44:54.057283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:44:54.057283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:47:37.372978+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":629,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-20T22:47:33.207159+0000","last_change":"2024-09-20T22:46:06.068842+0000","last_active":"2024-09-20T22:47:33.207159+0000","last_peered":"2024-09-20T22:47:33.207159+0000","last_clean":"2024-09-20T22:47:33.207159+0000","last_became_active":"2024-09-20T22:41:44.197927+0000","last_became_peered":"2024-09-20T22:41:44.197927+0000","last_unstale":"2024-09-20T22:47:33.207159+0000","last_undegraded":"2024-09-20T22:47:33.207159+0000","last_fullsized":"2024-09-20T22:47:33.207159+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:46:06.068745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:46:06.068745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:47:34.304672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":459,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-20T22:47:33.535857+0000","last_change":"2024-09-20T22:47:21.819643+0000","last_active":"2024-09-20T22:47:33.535857+0000","last_peered":"2024-09-20T22:47:33.535857+0000","last_clean":"2024-09-20T22:47:33.535857+0000","last_became_active":"2024-09-20T22:43:57.427971+0000","last_became_peered":"2024-09-20T22:43:57.427971+0000","last_unstale":"2024-09-20T22:47:33.535857+0000","last_undegraded":"2024-09-20T22:47:33.535857+0000","last_fullsized":"2024-09-20T22:47:33.535857+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:47:21.819572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:47:21.819572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:30.659501+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":434,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-20T22:47:33.413532+0000","last_change":"2024-09-20T22:47:13.413986+0000","last_active":"2024-09-20T22:47:33.413532+0000","last_peered":"2024-09-20T22:47:33.413532+0000","last_clean":"2024-09-20T22:47:33.413532+0000","last_became_active":"2024-09-20T22:45:40.229557+0000","last_became_peered":"2024-09-20T22:45:40.229557+0000","last_unstale":"2024-09-20T22:47:33.413532+0000","last_undegraded":"2024-09-20T22:47:33.413532+0000","last_fullsized":"2024-09-20T22:47:33.413532+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:47:13.413912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:47:13.413912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:19.554644+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1c","version":"113'22","reported_seq":676,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-20T22:47:33.189372+0000","last_change":"2024-09-20T22:46:25.731055+0000","last_active":"2024-09-20T22:47:33.189372+0000","last_peered":"2024-09-20T22:47:33.189372+0000","last_clean":"2024-09-20T22:47:33.189372+0000","last_became_active":"2024-09-20T22:41:40.170131+0000","last_became_peered":"2024-09-20T22:41:40.170131+0000","last_unstale":"2024-09-20T22:47:33.189372+0000","last_undegraded":"2024-09-20T22:47:33.189372+0000","last_fullsized":"2024-09-20T22:47:33.189372+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'22","last_scrub_stamp":"2024-09-20T22:46:25.730972+0000","last_deep_scrub":"113'22","last_deep_scrub_stamp":"2024-09-20T22:43:12.191878+0000","last_clean_scrub_stamp":"2024-09-20T22:46:25.730972+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:47:33.996947+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":644,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-20T22:47:33.239750+0000","last_change":"2024-09-20T22:46:49.872553+0000","last_active":"2024-09-20T22:47:33.239750+0000","last_peered":"2024-09-20T22:47:33.239750+0000","last_clean":"2024-09-20T22:47:33.239750+0000","last_became_active":"2024-09-20T22:45:40.221969+0000","last_became_peered":"2024-09-20T22:45:40.221969+0000","last_unstale":"2024-09-20T22:47:33.239750+0000","last_undegraded":"2024-09-20T22:47:33.239750+0000","last_fullsized":"2024-09-20T22:47:33.239750+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:46:49.872479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:46:49.872479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:47:58.447433+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":620,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-20T22:47:33.204673+0000","last_change":"2024-09-20T22:45:25.002270+0000","last_active":"2024-09-20T22:47:33.204673+0000","last_peered":"2024-09-20T22:47:33.204673+0000","last_clean":"2024-09-20T22:47:33.204673+0000","last_became_active":"2024-09-20T22:41:44.832131+0000","last_became_peered":"2024-09-20T22:41:44.832131+0000","last_unstale":"2024-09-20T22:47:33.204673+0000","last_undegraded":"2024-09-20T22:47:33.204673+0000","last_fullsized":"2024-09-20T22:47:33.204673+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:45:25.002199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:45:25.002199+0000","last_clean_scrub_stamp":"2024-09-20T22:45:25.002199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:47:35.121342+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":464,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-20T22:47:33.239726+0000","last_change":"2024-09-20T22:45:40.217601+0000","last_active":"2024-09-20T22:47:33.239726+0000","last_peered":"2024-09-20T22:47:33.239726+0000","last_clean":"2024-09-20T22:47:33.239726+0000","last_became_active":"2024-09-20T22:45:40.217251+0000","last_became_peered":"2024-09-20T22:45:40.217251+0000","last_unstale":"2024-09-20T22:47:33.239726+0000","last_undegraded":"2024-09-20T22:47:33.239726+0000","last_fullsized":"2024-09-20T22:47:33.239726+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:45:15.043443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:45:15.043443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:47:34.698473+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"74'12","reported_seq":687,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-20T22:47:33.204588+0000","last_change":"2024-09-20T22:44:42.141018+0000","last_active":"2024-09-20T22:47:33.204588+0000","last_peered":"2024-09-20T22:47:33.204588+0000","last_clean":"2024-09-20T22:47:33.204588+0000","last_became_active":"2024-09-20T22:43:57.438162+0000","last_became_peered":"2024-09-20T22:43:57.438162+0000","last_unstale":"2024-09-20T22:47:33.204588+0000","last_undegraded":"2024-09-20T22:47:33.204588+0000","last_fullsized":"2024-09-20T22:47:33.204588+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-09-20T22:44:42.140915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:44:42.140915+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:47:35.121258+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":450,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-20T22:47:33.204111+0000","last_change":"2024-09-20T22:44:14.235908+0000","last_active":"2024-09-20T22:47:33.204111+0000","last_peered":"2024-09-20T22:47:33.204111+0000","last_clean":"2024-09-20T22:47:33.204111+0000","last_became_active":"2024-09-20T22:43:57.429179+0000","last_became_peered":"2024-09-20T22:43:57.429179+0000","last_unstale":"2024-09-20T22:47:33.204111+0000","last_undegraded":"2024-09-20T22:47:33.204111+0000","last_fullsized":"2024-09-20T22:47:33.204111+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:44:14.235804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:44:14.235804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:47:35.121166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"154'2","reported_seq":631,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-20T22:47:33.413587+0000","last_change":"2024-09-20T22:46:21.446679+0000","last_active":"2024-09-20T22:47:33.413587+0000","last_peered":"2024-09-20T22:47:33.413587+0000","last_clean":"2024-09-20T22:47:33.413587+0000","last_became_active":"2024-09-20T22:41:44.199822+0000","last_became_peered":"2024-09-20T22:41:44.199822+0000","last_unstale":"2024-09-20T22:47:33.413587+0000","last_undegraded":"2024-09-20T22:47:33.413587+0000","last_fullsized":"2024-09-20T22:47:33.413587+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"154'2","last_scrub_stamp":"2024-09-20T22:46:21.446609+0000","last_deep_scrub":"154'2","last_deep_scrub_stamp":"2024-09-20T22:43:54.134066+0000","last_clean_scrub_stamp":"2024-09-20T22:46:21.446609+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:47:37.373451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"67'1","reported_seq":651,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-20T22:47:33.240169+0000","last_change":"2024-09-20T22:45:36.976765+0000","last_active":"2024-09-20T22:47:33.240169+0000","last_peered":"2024-09-20T22:47:33.240169+0000","last_clean":"2024-09-20T22:47:33.240169+0000","last_became_active":"2024-09-20T22:43:57.443155+0000","last_became_peered":"2024-09-20T22:43:57.443155+0000","last_unstale":"2024-09-20T22:47:33.240169+0000","last_undegraded":"2024-09-20T22:47:33.240169+0000","last_fullsized":"2024-09-20T22:47:33.240169+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-09-20T22:45:36.976689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:45:36.976689+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:47:34.698709+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"151'15","reported_seq":450,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-20T22:47:33.413798+0000","last_change":"2024-09-20T22:45:58.439899+0000","last_active":"2024-09-20T22:47:33.413798+0000","last_peered":"2024-09-20T22:47:33.413798+0000","last_clean":"2024-09-20T22:47:33.413798+0000","last_became_active":"2024-09-20T22:45:40.212043+0000","last_became_peered":"2024-09-20T22:45:40.212043+0000","last_unstale":"2024-09-20T22:47:33.413798+0000","last_undegraded":"2024-09-20T22:47:33.413798+0000","last_fullsized":"2024-09-20T22:47:33.413798+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"151'15","last_scrub_stamp":"2024-09-20T22:45:58.439782+0000","last_deep_scrub":"151'15","last_deep_scrub_stamp":"2024-09-20T22:43:50.715761+0000","last_clean_scrub_stamp":"2024-09-20T22:45:58.439782+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:47:37.373481+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"159'9","reported_seq":728,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-20T22:47:33.536199+0000","last_change":"2024-09-20T22:46:21.897778+0000","last_active":"2024-09-20T22:47:33.536199+0000","last_peered":"2024-09-20T22:47:33.536199+0000","last_clean":"2024-09-20T22:47:33.536199+0000","last_became_active":"2024-09-20T22:43:57.424879+0000","last_became_peered":"2024-09-20T22:43:57.424879+0000","last_unstale":"2024-09-20T22:47:33.536199+0000","last_undegraded":"2024-09-20T22:47:33.536199+0000","last_fullsized":"2024-09-20T22:47:33.536199+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"159'9","last_scrub_stamp":"2024-09-20T22:46:21.897701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:46:21.897701+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.c","version":"372'802","reported_seq":2243,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-20T22:47:33.537957+0000","last_change":"2024-09-20T22:47:32.810665+0000","last_active":"2024-09-20T22:47:33.537957+0000","last_peered":"2024-09-20T22:47:33.537957+0000","last_clean":"2024-09-20T22:47:33.537957+0000","last_became_active":"2024-09-20T22:42:52.404027+0000","last_became_peered":"2024-09-20T22:42:52.404027+0000","last_unstale":"2024-09-20T22:47:33.537957+0000","last_undegraded":"2024-09-20T22:47:33.537957+0000","last_fullsized":"2024-09-20T22:47:33.537957+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"332'667","last_scrub_stamp":"2024-09-20T22:46:52.845442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:46:52.845442+0000","objects_scrubbed":12,"log_size":802,"log_dups_size":0,"ondisk_log_size":802,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:01.375911+0000","scrub_duration":3,"objects_trimmed":4,"snaptrim_duration":0.0092785279999999994,"stat_sum":{"num_bytes":32407328,"num_objects":15,"num_object_clones":9,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":1,"num_read":487,"num_read_kb":481315,"num_write":317,"num_write_kb":123200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"6b","length":"1"},{"start":"74","length":"1"},{"start":"89","length":"1"},{"start":"90","length":"1"}]},{"pgid":"1.a","version":"77'128","reported_seq":900,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-20T22:47:33.207080+0000","last_change":"2024-09-20T22:45:37.851690+0000","last_active":"2024-09-20T22:47:33.207080+0000","last_peered":"2024-09-20T22:47:33.207080+0000","last_clean":"2024-09-20T22:47:33.207080+0000","last_became_active":"2024-09-20T22:42:52.407791+0000","last_became_peered":"2024-09-20T22:42:52.407791+0000","last_unstale":"2024-09-20T22:47:33.207080+0000","last_undegraded":"2024-09-20T22:47:33.207080+0000","last_fullsized":"2024-09-20T22:47:33.207080+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:45:37.851582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:45:37.851582+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:47:34.304552+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"74'15","reported_seq":683,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-20T22:47:33.240016+0000","last_change":"2024-09-20T22:44:55.972422+0000","last_active":"2024-09-20T22:47:33.240016+0000","last_peered":"2024-09-20T22:47:33.240016+0000","last_clean":"2024-09-20T22:47:33.240016+0000","last_became_active":"2024-09-20T22:43:57.441355+0000","last_became_peered":"2024-09-20T22:43:57.441355+0000","last_unstale":"2024-09-20T22:47:33.240016+0000","last_undegraded":"2024-09-20T22:47:33.240016+0000","last_fullsized":"2024-09-20T22:47:33.240016+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-09-20T22:44:55.972333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:44:55.972333+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:47:34.698504+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":632,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-20T22:47:33.203213+0000","last_change":"2024-09-20T22:45:19.027462+0000","last_active":"2024-09-20T22:47:33.203213+0000","last_peered":"2024-09-20T22:47:33.203213+0000","last_clean":"2024-09-20T22:47:33.203213+0000","last_became_active":"2024-09-20T22:41:38.156749+0000","last_became_peered":"2024-09-20T22:41:38.156749+0000","last_unstale":"2024-09-20T22:47:33.203213+0000","last_undegraded":"2024-09-20T22:47:33.203213+0000","last_fullsized":"2024-09-20T22:47:33.203213+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:45:19.027393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:45:19.027393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:47:35.121313+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"264'10","reported_seq":542,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-20T22:47:33.208282+0000","last_change":"2024-09-20T22:47:20.300876+0000","last_active":"2024-09-20T22:47:33.208282+0000","last_peered":"2024-09-20T22:47:33.208282+0000","last_clean":"2024-09-20T22:47:33.208282+0000","last_became_active":"2024-09-20T22:45:40.213956+0000","last_became_peered":"2024-09-20T22:45:40.213956+0000","last_unstale":"2024-09-20T22:47:33.208282+0000","last_undegraded":"2024-09-20T22:47:33.208282+0000","last_fullsized":"2024-09-20T22:47:33.208282+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"264'10","last_scrub_stamp":"2024-09-20T22:47:20.300787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:47:20.300787+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:25.147757+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":635,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-20T22:47:33.207676+0000","last_change":"2024-09-20T22:46:26.147242+0000","last_active":"2024-09-20T22:47:33.207676+0000","last_peered":"2024-09-20T22:47:33.207676+0000","last_clean":"2024-09-20T22:47:33.207676+0000","last_became_active":"2024-09-20T22:45:40.224443+0000","last_became_peered":"2024-09-20T22:45:40.224443+0000","last_unstale":"2024-09-20T22:47:33.207676+0000","last_undegraded":"2024-09-20T22:47:33.207676+0000","last_fullsized":"2024-09-20T22:47:33.207676+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:46:26.147167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:46:26.147167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:47:34.299765+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3","version":"135'8","reported_seq":1395,"reported_epoch":373,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-20T22:47:33.211128+0000","last_change":"2024-09-20T22:47:33.211166+0000","last_active":"2024-09-20T22:47:33.211128+0000","last_peered":"2024-09-20T22:47:33.211128+0000","last_clean":"2024-09-20T22:47:33.211128+0000","last_became_active":"2024-09-20T22:43:57.439135+0000","last_became_peered":"2024-09-20T22:43:57.439135+0000","last_unstale":"2024-09-20T22:47:33.211128+0000","last_undegraded":"2024-09-20T22:47:33.211128+0000","last_fullsized":"2024-09-20T22:47:33.211128+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'8","last_scrub_stamp":"2024-09-20T22:47:23.137750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:47:23.137750+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:43.202372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0078887799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"6b","length":"1"},{"start":"89","length":"1"},{"start":"90","length":"1"}]},{"pgid":"1.5","version":"77'128","reported_seq":785,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-20T22:47:33.189136+0000","last_change":"2024-09-20T22:46:11.182702+0000","last_active":"2024-09-20T22:47:33.189136+0000","last_peered":"2024-09-20T22:47:33.189136+0000","last_clean":"2024-09-20T22:47:33.189136+0000","last_became_active":"2024-09-20T22:46:11.182471+0000","last_became_peered":"2024-09-20T22:46:11.182471+0000","last_unstale":"2024-09-20T22:47:33.189136+0000","last_undegraded":"2024-09-20T22:47:33.189136+0000","last_fullsized":"2024-09-20T22:47:33.189136+0000","mapping_epoch":160,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:45:41.608075+0000","last_deep_scrub":"77'128","last_deep_scrub_stamp":"2024-09-20T22:45:41.608075+0000","last_clean_scrub_stamp":"2024-09-20T22:45:41.608075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:47:33.996891+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,5],"acting":[2,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"74'18","reported_seq":684,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-20T22:47:33.203431+0000","last_change":"2024-09-20T22:45:40.246576+0000","last_active":"2024-09-20T22:47:33.203431+0000","last_peered":"2024-09-20T22:47:33.203431+0000","last_clean":"2024-09-20T22:47:33.203431+0000","last_became_active":"2024-09-20T22:45:40.215766+0000","last_became_peered":"2024-09-20T22:45:40.215766+0000","last_unstale":"2024-09-20T22:47:33.203431+0000","last_undegraded":"2024-09-20T22:47:33.203431+0000","last_fullsized":"2024-09-20T22:47:33.203431+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'18","last_scrub_stamp":"2024-09-20T22:44:23.220709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:44:23.220709+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:47:35.121152+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":638,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-20T22:47:33.205070+0000","last_change":"2024-09-20T22:45:40.213320+0000","last_active":"2024-09-20T22:47:33.205070+0000","last_peered":"2024-09-20T22:47:33.205070+0000","last_clean":"2024-09-20T22:47:33.205070+0000","last_became_active":"2024-09-20T22:45:40.213020+0000","last_became_peered":"2024-09-20T22:45:40.213020+0000","last_unstale":"2024-09-20T22:47:33.205070+0000","last_undegraded":"2024-09-20T22:47:33.205070+0000","last_fullsized":"2024-09-20T22:47:33.205070+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:51.170155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:43:51.170155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:47:35.121077+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":456,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-20T22:47:33.202952+0000","last_change":"2024-09-20T22:45:56.199496+0000","last_active":"2024-09-20T22:47:33.202952+0000","last_peered":"2024-09-20T22:47:33.202952+0000","last_clean":"2024-09-20T22:47:33.202952+0000","last_became_active":"2024-09-20T22:45:40.214328+0000","last_became_peered":"2024-09-20T22:45:40.214328+0000","last_unstale":"2024-09-20T22:47:33.202952+0000","last_undegraded":"2024-09-20T22:47:33.202952+0000","last_fullsized":"2024-09-20T22:47:33.202952+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:45:56.199432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:07.947945+0000","last_clean_scrub_stamp":"2024-09-20T22:45:56.199432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:47:35.121427+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":451,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-20T22:47:33.203000+0000","last_change":"2024-09-20T22:47:18.179688+0000","last_active":"2024-09-20T22:47:33.203000+0000","last_peered":"2024-09-20T22:47:33.203000+0000","last_clean":"2024-09-20T22:47:33.203000+0000","last_became_active":"2024-09-20T22:43:57.435057+0000","last_became_peered":"2024-09-20T22:43:57.435057+0000","last_unstale":"2024-09-20T22:47:33.203000+0000","last_undegraded":"2024-09-20T22:47:33.203000+0000","last_fullsized":"2024-09-20T22:47:33.203000+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:47:18.179575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:53.036601+0000","last_clean_scrub_stamp":"2024-09-20T22:47:18.179575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:25.505641+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2","version":"371'163","reported_seq":1126,"reported_epoch":373,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T22:47:33.207183+0000","last_change":"2024-09-20T22:47:32.242750+0000","last_active":"2024-09-20T22:47:33.207183+0000","last_peered":"2024-09-20T22:47:33.207183+0000","last_clean":"2024-09-20T22:47:33.207183+0000","last_became_active":"2024-09-20T22:44:09.177625+0000","last_became_peered":"2024-09-20T22:44:09.177625+0000","last_unstale":"2024-09-20T22:47:33.207183+0000","last_undegraded":"2024-09-20T22:47:33.207183+0000","last_fullsized":"2024-09-20T22:47:33.207183+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"282'120","last_scrub_stamp":"2024-09-20T22:46:22.099642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:46:22.099642+0000","objects_scrubbed":1,"log_size":163,"log_dups_size":0,"ondisk_log_size":163,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":1.0105230540000001,"stat_sum":{"num_bytes":9954330,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":69,"num_read_kb":83024,"num_write":30,"num_write_kb":9659,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"89","length":"1"},{"start":"90","length":"1"}]},{"pgid":"1.4","version":"77'128","reported_seq":932,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-20T22:47:33.204200+0000","last_change":"2024-09-20T22:46:15.181632+0000","last_active":"2024-09-20T22:47:33.204200+0000","last_peered":"2024-09-20T22:47:33.204200+0000","last_clean":"2024-09-20T22:47:33.204200+0000","last_became_active":"2024-09-20T22:46:15.181305+0000","last_became_peered":"2024-09-20T22:46:15.181305+0000","last_unstale":"2024-09-20T22:47:33.204200+0000","last_undegraded":"2024-09-20T22:47:33.204200+0000","last_fullsized":"2024-09-20T22:47:33.204200+0000","mapping_epoch":94,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:45:54.131305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:45:54.131305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":59,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:47:35.121456+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"74'14","reported_seq":537,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-20T22:47:33.207201+0000","last_change":"2024-09-20T22:46:48.202431+0000","last_active":"2024-09-20T22:47:33.207201+0000","last_peered":"2024-09-20T22:47:33.207201+0000","last_clean":"2024-09-20T22:47:33.207201+0000","last_became_active":"2024-09-20T22:45:40.213664+0000","last_became_peered":"2024-09-20T22:45:40.213664+0000","last_unstale":"2024-09-20T22:47:33.207201+0000","last_undegraded":"2024-09-20T22:47:33.207201+0000","last_fullsized":"2024-09-20T22:47:33.207201+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'14","last_scrub_stamp":"2024-09-20T22:46:48.202311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:46:48.202311+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:14.529508+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,7],"acting":[5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.6","version":"67'1","reported_seq":247,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-20T22:47:33.240019+0000","last_change":"2024-09-20T22:45:40.246043+0000","last_active":"2024-09-20T22:47:33.240019+0000","last_peered":"2024-09-20T22:47:33.240019+0000","last_clean":"2024-09-20T22:47:33.240019+0000","last_became_active":"2024-09-20T22:45:40.224257+0000","last_became_peered":"2024-09-20T22:45:40.224257+0000","last_unstale":"2024-09-20T22:47:33.240019+0000","last_undegraded":"2024-09-20T22:47:33.240019+0000","last_fullsized":"2024-09-20T22:47:33.240019+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-09-20T22:45:25.781996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:45:25.781996+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:47:34.698551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"264'10","reported_seq":724,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-20T22:47:33.203693+0000","last_change":"2024-09-20T22:45:40.270343+0000","last_active":"2024-09-20T22:47:33.203693+0000","last_peered":"2024-09-20T22:47:33.203693+0000","last_clean":"2024-09-20T22:47:33.203693+0000","last_became_active":"2024-09-20T22:45:40.222219+0000","last_became_peered":"2024-09-20T22:45:40.222219+0000","last_unstale":"2024-09-20T22:47:33.203693+0000","last_undegraded":"2024-09-20T22:47:33.203693+0000","last_fullsized":"2024-09-20T22:47:33.203693+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"159'9","last_scrub_stamp":"2024-09-20T22:45:23.017605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:45:23.017605+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:47:35.121388+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":629,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-20T22:47:33.413308+0000","last_change":"2024-09-20T22:47:16.381511+0000","last_active":"2024-09-20T22:47:33.413308+0000","last_peered":"2024-09-20T22:47:33.413308+0000","last_clean":"2024-09-20T22:47:33.413308+0000","last_became_active":"2024-09-20T22:41:44.831901+0000","last_became_peered":"2024-09-20T22:41:44.831901+0000","last_unstale":"2024-09-20T22:47:33.413308+0000","last_undegraded":"2024-09-20T22:47:33.413308+0000","last_fullsized":"2024-09-20T22:47:33.413308+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:47:16.381390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:47:16.381390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:23.066386+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1","version":"371'193","reported_seq":1553,"reported_epoch":373,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-20T22:47:33.243595+0000","last_change":"2024-09-20T22:47:33.243631+0000","last_active":"2024-09-20T22:47:33.243595+0000","last_peered":"2024-09-20T22:47:33.243595+0000","last_clean":"2024-09-20T22:47:33.243595+0000","last_became_active":"2024-09-20T22:43:57.440095+0000","last_became_peered":"2024-09-20T22:43:57.440095+0000","last_unstale":"2024-09-20T22:47:33.243595+0000","last_undegraded":"2024-09-20T22:47:33.243595+0000","last_fullsized":"2024-09-20T22:47:33.243595+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"170'72","last_scrub_stamp":"2024-09-20T22:44:12.228784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:44:12.228784+0000","objects_scrubbed":3,"log_size":193,"log_dups_size":0,"ondisk_log_size":193,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.040982708,"stat_sum":{"num_bytes":11363530,"num_objects":6,"num_object_clones":4,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":59,"num_read_kb":78458,"num_write":45,"num_write_kb":34476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"74","length":"1"},{"start":"89","length":"1"},{"start":"90","length":"1"}]},{"pgid":"1.7","version":"77'128","reported_seq":929,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-20T22:47:33.537324+0000","last_change":"2024-09-20T22:46:03.170000+0000","last_active":"2024-09-20T22:47:33.537324+0000","last_peered":"2024-09-20T22:47:33.537324+0000","last_clean":"2024-09-20T22:47:33.537324+0000","last_became_active":"2024-09-20T22:46:03.169596+0000","last_became_peered":"2024-09-20T22:46:03.169596+0000","last_unstale":"2024-09-20T22:47:33.537324+0000","last_undegraded":"2024-09-20T22:47:33.537324+0000","last_fullsized":"2024-09-20T22:47:33.537324+0000","mapping_epoch":94,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:45:25.813923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:45:25.813923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:47:36.787063+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"74'10","reported_seq":687,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-20T22:47:33.239020+0000","last_change":"2024-09-20T22:45:40.249290+0000","last_active":"2024-09-20T22:47:33.239020+0000","last_peered":"2024-09-20T22:47:33.239020+0000","last_clean":"2024-09-20T22:47:33.239020+0000","last_became_active":"2024-09-20T22:45:40.221018+0000","last_became_peered":"2024-09-20T22:45:40.221018+0000","last_unstale":"2024-09-20T22:47:33.239020+0000","last_undegraded":"2024-09-20T22:47:33.239020+0000","last_fullsized":"2024-09-20T22:47:33.239020+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-09-20T22:44:59.049607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:44:59.049607+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:47:34.698519+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":641,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-20T22:47:33.208048+0000","last_change":"2024-09-20T22:45:17.702193+0000","last_active":"2024-09-20T22:47:33.208048+0000","last_peered":"2024-09-20T22:47:33.208048+0000","last_clean":"2024-09-20T22:47:33.208048+0000","last_became_active":"2024-09-20T22:41:38.155629+0000","last_became_peered":"2024-09-20T22:41:38.155629+0000","last_unstale":"2024-09-20T22:47:33.208048+0000","last_undegraded":"2024-09-20T22:47:33.208048+0000","last_fullsized":"2024-09-20T22:47:33.208048+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:45:17.702106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:45:17.702106+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:47:34.304522+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"264'10","reported_seq":290,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-20T22:47:33.537189+0000","last_change":"2024-09-20T22:45:57.889712+0000","last_active":"2024-09-20T22:47:33.537189+0000","last_peered":"2024-09-20T22:47:33.537189+0000","last_clean":"2024-09-20T22:47:33.537189+0000","last_became_active":"2024-09-20T22:45:40.211664+0000","last_became_peered":"2024-09-20T22:45:40.211664+0000","last_unstale":"2024-09-20T22:47:33.537189+0000","last_undegraded":"2024-09-20T22:47:33.537189+0000","last_fullsized":"2024-09-20T22:47:33.537189+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"264'10","last_scrub_stamp":"2024-09-20T22:45:57.889636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:45:57.889636+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:47:36.787198+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":243,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-20T22:47:33.203692+0000","last_change":"2024-09-20T22:46:44.117883+0000","last_active":"2024-09-20T22:47:33.203692+0000","last_peered":"2024-09-20T22:47:33.203692+0000","last_clean":"2024-09-20T22:47:33.203692+0000","last_became_active":"2024-09-20T22:45:40.218780+0000","last_became_peered":"2024-09-20T22:45:40.218780+0000","last_unstale":"2024-09-20T22:47:33.203692+0000","last_undegraded":"2024-09-20T22:47:33.203692+0000","last_fullsized":"2024-09-20T22:47:33.203692+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:46:44.117761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:46:44.117761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:47:45.344447+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.0","version":"373'151","reported_seq":1572,"reported_epoch":373,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T22:47:33.237264+0000","last_change":"2024-09-20T22:47:33.237264+0000","last_active":"2024-09-20T22:47:33.237264+0000","last_peered":"2024-09-20T22:47:33.237264+0000","last_clean":"2024-09-20T22:47:33.237264+0000","last_became_active":"2024-09-20T22:43:37.136845+0000","last_became_peered":"2024-09-20T22:43:37.136845+0000","last_unstale":"2024-09-20T22:47:33.237264+0000","last_undegraded":"2024-09-20T22:47:33.237264+0000","last_fullsized":"2024-09-20T22:47:33.237264+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"280'122","last_scrub_stamp":"2024-09-20T22:46:03.006423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:46:03.006423+0000","objects_scrubbed":1,"log_size":151,"log_dups_size":0,"ondisk_log_size":151,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.030478344000000001,"stat_sum":{"num_bytes":8921635,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":59,"num_read_kb":57291,"num_write":40,"num_write_kb":9686,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"74","length":"1"},{"start":"89","length":"1"},{"start":"90","length":"1"}]},{"pgid":"1.6","version":"77'128","reported_seq":936,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-20T22:47:33.537405+0000","last_change":"2024-09-20T22:46:07.176473+0000","last_active":"2024-09-20T22:47:33.537405+0000","last_peered":"2024-09-20T22:47:33.537405+0000","last_clean":"2024-09-20T22:47:33.537405+0000","last_became_active":"2024-09-20T22:46:07.175920+0000","last_became_peered":"2024-09-20T22:46:07.175920+0000","last_unstale":"2024-09-20T22:47:33.537405+0000","last_undegraded":"2024-09-20T22:47:33.537405+0000","last_fullsized":"2024-09-20T22:47:33.537405+0000","mapping_epoch":263,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":288,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:45:17.692996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:45:17.692996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:47:36.787023+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"74'19","reported_seq":317,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-20T22:47:33.207581+0000","last_change":"2024-09-20T22:47:13.403322+0000","last_active":"2024-09-20T22:47:33.207581+0000","last_peered":"2024-09-20T22:47:33.207581+0000","last_clean":"2024-09-20T22:47:33.207581+0000","last_became_active":"2024-09-20T22:45:40.229189+0000","last_became_peered":"2024-09-20T22:45:40.229189+0000","last_unstale":"2024-09-20T22:47:33.207581+0000","last_undegraded":"2024-09-20T22:47:33.207581+0000","last_fullsized":"2024-09-20T22:47:33.207581+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'19","last_scrub_stamp":"2024-09-20T22:47:13.403224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:47:13.403224+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:43.024760+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":38,"num_read_kb":24,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":640,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-20T22:47:33.240002+0000","last_change":"2024-09-20T22:45:21.038355+0000","last_active":"2024-09-20T22:47:33.240002+0000","last_peered":"2024-09-20T22:47:33.240002+0000","last_clean":"2024-09-20T22:47:33.240002+0000","last_became_active":"2024-09-20T22:41:38.160832+0000","last_became_peered":"2024-09-20T22:41:38.160832+0000","last_unstale":"2024-09-20T22:47:33.240002+0000","last_undegraded":"2024-09-20T22:47:33.240002+0000","last_fullsized":"2024-09-20T22:47:33.240002+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:45:21.038284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:45:21.038284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:47:34.698533+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":638,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-20T22:47:33.537958+0000","last_change":"2024-09-20T22:46:43.827791+0000","last_active":"2024-09-20T22:47:33.537958+0000","last_peered":"2024-09-20T22:47:33.537958+0000","last_clean":"2024-09-20T22:47:33.537958+0000","last_became_active":"2024-09-20T22:45:40.212686+0000","last_became_peered":"2024-09-20T22:45:40.212686+0000","last_unstale":"2024-09-20T22:47:33.537958+0000","last_undegraded":"2024-09-20T22:47:33.537958+0000","last_fullsized":"2024-09-20T22:47:33.537958+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:46:43.827680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:46:43.827680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:47:58.640025+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":628,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-20T22:47:33.238888+0000","last_change":"2024-09-20T22:45:39.938227+0000","last_active":"2024-09-20T22:47:33.238888+0000","last_peered":"2024-09-20T22:47:33.238888+0000","last_clean":"2024-09-20T22:47:33.238888+0000","last_became_active":"2024-09-20T22:41:44.830538+0000","last_became_peered":"2024-09-20T22:41:44.830538+0000","last_unstale":"2024-09-20T22:47:33.238888+0000","last_undegraded":"2024-09-20T22:47:33.238888+0000","last_fullsized":"2024-09-20T22:47:33.238888+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:45:39.938119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:45:39.938119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:47:34.698593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.7","version":"372'324","reported_seq":1781,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-20T22:47:33.537622+0000","last_change":"2024-09-20T22:47:32.813704+0000","last_active":"2024-09-20T22:47:33.537622+0000","last_peered":"2024-09-20T22:47:33.537622+0000","last_clean":"2024-09-20T22:47:33.537622+0000","last_became_active":"2024-09-20T22:45:43.169140+0000","last_became_peered":"2024-09-20T22:45:43.169140+0000","last_unstale":"2024-09-20T22:47:33.537622+0000","last_undegraded":"2024-09-20T22:47:33.537622+0000","last_fullsized":"2024-09-20T22:47:33.537622+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"235'189","last_scrub_stamp":"2024-09-20T22:45:16.672762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:45:16.672762+0000","objects_scrubbed":3,"log_size":324,"log_dups_size":0,"ondisk_log_size":324,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":1,"snaptrim_duration":0.011208515,"stat_sum":{"num_bytes":20001455,"num_objects":7,"num_object_clones":4,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":106,"num_read_kb":114984,"num_write":94,"num_write_kb":35683,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":8139149,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"6b","length":"1"},{"start":"74","length":"1"},{"start":"89","length":"1"},{"start":"90","length":"1"}]},{"pgid":"1.1","version":"77'128","reported_seq":930,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-20T22:47:33.207000+0000","last_change":"2024-09-20T22:45:58.908515+0000","last_active":"2024-09-20T22:47:33.207000+0000","last_peered":"2024-09-20T22:47:33.207000+0000","last_clean":"2024-09-20T22:47:33.207000+0000","last_became_active":"2024-09-20T22:45:55.169241+0000","last_became_peered":"2024-09-20T22:45:55.169241+0000","last_unstale":"2024-09-20T22:47:33.207000+0000","last_undegraded":"2024-09-20T22:47:33.207000+0000","last_fullsized":"2024-09-20T22:47:33.207000+0000","mapping_epoch":263,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:45:58.908433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:45:58.908433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:47:34.304625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"86'38","reported_seq":766,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-20T22:47:33.239905+0000","last_change":"2024-09-20T22:44:41.077590+0000","last_active":"2024-09-20T22:47:33.239905+0000","last_peered":"2024-09-20T22:47:33.239905+0000","last_clean":"2024-09-20T22:47:33.239905+0000","last_became_active":"2024-09-20T22:41:40.166958+0000","last_became_peered":"2024-09-20T22:41:40.166958+0000","last_unstale":"2024-09-20T22:47:33.239905+0000","last_undegraded":"2024-09-20T22:47:33.239905+0000","last_fullsized":"2024-09-20T22:47:33.239905+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'38","last_scrub_stamp":"2024-09-20T22:44:41.077475+0000","last_deep_scrub":"86'38","last_deep_scrub_stamp":"2024-09-20T22:42:43.935677+0000","last_clean_scrub_stamp":"2024-09-20T22:44:41.077475+0000","objects_scrubbed":10,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:47:34.698442+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":245,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-20T22:47:33.537250+0000","last_change":"2024-09-20T22:47:16.817569+0000","last_active":"2024-09-20T22:47:33.537250+0000","last_peered":"2024-09-20T22:47:33.537250+0000","last_clean":"2024-09-20T22:47:33.537250+0000","last_became_active":"2024-09-20T22:45:40.204848+0000","last_became_peered":"2024-09-20T22:45:40.204848+0000","last_unstale":"2024-09-20T22:47:33.537250+0000","last_undegraded":"2024-09-20T22:47:33.537250+0000","last_fullsized":"2024-09-20T22:47:33.537250+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:47:16.817468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:47:16.817468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:38.645748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"69'2","reported_seq":659,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-20T22:47:33.207530+0000","last_change":"2024-09-20T22:46:44.210668+0000","last_active":"2024-09-20T22:47:33.207530+0000","last_peered":"2024-09-20T22:47:33.207530+0000","last_clean":"2024-09-20T22:47:33.207530+0000","last_became_active":"2024-09-20T22:41:37.148479+0000","last_became_peered":"2024-09-20T22:41:37.148479+0000","last_unstale":"2024-09-20T22:47:33.207530+0000","last_undegraded":"2024-09-20T22:47:33.207530+0000","last_fullsized":"2024-09-20T22:47:33.207530+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2024-09-20T22:46:44.210559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:36.130684+0000","last_clean_scrub_stamp":"2024-09-20T22:46:44.210559+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:09.950285+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00087280599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":426,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-20T22:47:33.239557+0000","last_change":"2024-09-20T22:46:06.104856+0000","last_active":"2024-09-20T22:47:33.239557+0000","last_peered":"2024-09-20T22:47:33.239557+0000","last_clean":"2024-09-20T22:47:33.239557+0000","last_became_active":"2024-09-20T22:45:40.207256+0000","last_became_peered":"2024-09-20T22:45:40.207256+0000","last_unstale":"2024-09-20T22:47:33.239557+0000","last_undegraded":"2024-09-20T22:47:33.239557+0000","last_fullsized":"2024-09-20T22:47:33.239557+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:46:06.104786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:46:06.104786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:47:34.698739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":634,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-20T22:47:33.204594+0000","last_change":"2024-09-20T22:45:58.213051+0000","last_active":"2024-09-20T22:47:33.204594+0000","last_peered":"2024-09-20T22:47:33.204594+0000","last_clean":"2024-09-20T22:47:33.204594+0000","last_became_active":"2024-09-20T22:43:57.434044+0000","last_became_peered":"2024-09-20T22:43:57.434044+0000","last_unstale":"2024-09-20T22:47:33.204594+0000","last_undegraded":"2024-09-20T22:47:33.204594+0000","last_fullsized":"2024-09-20T22:47:33.204594+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:45:58.212988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:45:58.212988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:47:35.121441+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"370'71","reported_seq":1247,"reported_epoch":373,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-20T22:47:33.239539+0000","last_change":"2024-09-20T22:47:33.197492+0000","last_active":"2024-09-20T22:47:33.239539+0000","last_peered":"2024-09-20T22:47:33.239539+0000","last_clean":"2024-09-20T22:47:33.239539+0000","last_became_active":"2024-09-20T22:45:40.223468+0000","last_became_peered":"2024-09-20T22:45:40.223468+0000","last_unstale":"2024-09-20T22:47:33.239539+0000","last_undegraded":"2024-09-20T22:47:33.239539+0000","last_fullsized":"2024-09-20T22:47:33.239539+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"305'57","last_scrub_stamp":"2024-09-20T22:47:13.636468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:47:13.636468+0000","objects_scrubbed":1,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:23.544800+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.94771171700000001,"stat_sum":{"num_bytes":7090788,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":31,"num_read_kb":31741,"num_write":16,"num_write_kb":3549,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"89","length":"1"},{"start":"90","length":"1"}]},{"pgid":"1.2","version":"77'128","reported_seq":980,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-20T22:47:33.413243+0000","last_change":"2024-09-20T22:45:51.178542+0000","last_active":"2024-09-20T22:47:33.413243+0000","last_peered":"2024-09-20T22:47:33.413243+0000","last_clean":"2024-09-20T22:47:33.413243+0000","last_became_active":"2024-09-20T22:45:51.178152+0000","last_became_peered":"2024-09-20T22:45:51.178152+0000","last_unstale":"2024-09-20T22:47:33.413243+0000","last_undegraded":"2024-09-20T22:47:33.413243+0000","last_fullsized":"2024-09-20T22:47:33.413243+0000","mapping_epoch":263,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:44:58.121721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:44:58.121721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:47:37.373032+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"74'13","reported_seq":681,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-20T22:47:33.240059+0000","last_change":"2024-09-20T22:46:04.145988+0000","last_active":"2024-09-20T22:47:33.240059+0000","last_peered":"2024-09-20T22:47:33.240059+0000","last_clean":"2024-09-20T22:47:33.240059+0000","last_became_active":"2024-09-20T22:45:40.221991+0000","last_became_peered":"2024-09-20T22:45:40.221991+0000","last_unstale":"2024-09-20T22:47:33.240059+0000","last_undegraded":"2024-09-20T22:47:33.240059+0000","last_fullsized":"2024-09-20T22:47:33.240059+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'13","last_scrub_stamp":"2024-09-20T22:46:04.145892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:46:04.145892+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:47:34.698801+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":640,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-20T22:47:33.238732+0000","last_change":"2024-09-20T22:44:35.131693+0000","last_active":"2024-09-20T22:47:33.238732+0000","last_peered":"2024-09-20T22:47:33.238732+0000","last_clean":"2024-09-20T22:47:33.238732+0000","last_became_active":"2024-09-20T22:41:38.156837+0000","last_became_peered":"2024-09-20T22:41:38.156837+0000","last_unstale":"2024-09-20T22:47:33.238732+0000","last_undegraded":"2024-09-20T22:47:33.238732+0000","last_fullsized":"2024-09-20T22:47:33.238732+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:44:35.131583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:44:35.131583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:47:34.698396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"67'1","reported_seq":667,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-20T22:47:33.188818+0000","last_change":"2024-09-20T22:46:00.792036+0000","last_active":"2024-09-20T22:47:33.188818+0000","last_peered":"2024-09-20T22:47:33.188818+0000","last_clean":"2024-09-20T22:47:33.188818+0000","last_became_active":"2024-09-20T22:45:40.223966+0000","last_became_peered":"2024-09-20T22:45:40.223966+0000","last_unstale":"2024-09-20T22:47:33.188818+0000","last_undegraded":"2024-09-20T22:47:33.188818+0000","last_fullsized":"2024-09-20T22:47:33.188818+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-09-20T22:46:00.791909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:36.130684+0000","last_clean_scrub_stamp":"2024-09-20T22:46:00.791909+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:47:33.996860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001301459,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":640,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-20T22:47:33.190340+0000","last_change":"2024-09-20T22:47:22.881180+0000","last_active":"2024-09-20T22:47:33.190340+0000","last_peered":"2024-09-20T22:47:33.190340+0000","last_clean":"2024-09-20T22:47:33.190340+0000","last_became_active":"2024-09-20T22:41:42.185112+0000","last_became_peered":"2024-09-20T22:41:42.185112+0000","last_unstale":"2024-09-20T22:47:33.190340+0000","last_undegraded":"2024-09-20T22:47:33.190340+0000","last_fullsized":"2024-09-20T22:47:33.190340+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:47:22.881109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:47:22.881109+0000","last_clean_scrub_stamp":"2024-09-20T22:47:22.881109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:46.218094+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":629,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-20T22:47:33.413959+0000","last_change":"2024-09-20T22:46:27.448645+0000","last_active":"2024-09-20T22:47:33.413959+0000","last_peered":"2024-09-20T22:47:33.413959+0000","last_clean":"2024-09-20T22:47:33.413959+0000","last_became_active":"2024-09-20T22:41:44.830515+0000","last_became_peered":"2024-09-20T22:41:44.830515+0000","last_unstale":"2024-09-20T22:47:33.413959+0000","last_undegraded":"2024-09-20T22:47:33.413959+0000","last_fullsized":"2024-09-20T22:47:33.413959+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:46:27.448572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:46:27.448572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:47:37.458021+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"372'136","reported_seq":1548,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-20T22:47:33.413719+0000","last_change":"2024-09-20T22:47:32.377666+0000","last_active":"2024-09-20T22:47:33.413719+0000","last_peered":"2024-09-20T22:47:33.413719+0000","last_clean":"2024-09-20T22:47:33.413719+0000","last_became_active":"2024-09-20T22:45:40.220614+0000","last_became_peered":"2024-09-20T22:45:40.220614+0000","last_unstale":"2024-09-20T22:47:33.413719+0000","last_undegraded":"2024-09-20T22:47:33.413719+0000","last_fullsized":"2024-09-20T22:47:33.413719+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"262'49","last_scrub_stamp":"2024-09-20T22:45:41.346463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:45:41.346463+0000","objects_scrubbed":4,"log_size":136,"log_dups_size":0,"ondisk_log_size":136,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":1,"snaptrim_duration":0.0024683909999999999,"stat_sum":{"num_bytes":14113470,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":23854,"num_write":62,"num_write_kb":30562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":5530930,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"6b","length":"1"},{"start":"74","length":"1"},{"start":"89","length":"1"},{"start":"90","length":"1"}]},{"pgid":"1.3","version":"77'128","reported_seq":941,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-20T22:47:33.239378+0000","last_change":"2024-09-20T22:46:19.180360+0000","last_active":"2024-09-20T22:47:33.239378+0000","last_peered":"2024-09-20T22:47:33.239378+0000","last_clean":"2024-09-20T22:47:33.239378+0000","last_became_active":"2024-09-20T22:46:19.180012+0000","last_became_peered":"2024-09-20T22:46:19.180012+0000","last_unstale":"2024-09-20T22:47:33.239378+0000","last_undegraded":"2024-09-20T22:47:33.239378+0000","last_fullsized":"2024-09-20T22:47:33.239378+0000","mapping_epoch":160,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:45:38.952841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:45:38.952841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:47:34.698623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"74'12","reported_seq":358,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-20T22:47:33.208148+0000","last_change":"2024-09-20T22:45:40.231346+0000","last_active":"2024-09-20T22:47:33.208148+0000","last_peered":"2024-09-20T22:47:33.208148+0000","last_clean":"2024-09-20T22:47:33.208148+0000","last_became_active":"2024-09-20T22:45:40.214689+0000","last_became_peered":"2024-09-20T22:45:40.214689+0000","last_unstale":"2024-09-20T22:47:33.208148+0000","last_undegraded":"2024-09-20T22:47:33.208148+0000","last_fullsized":"2024-09-20T22:47:33.208148+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-09-20T22:45:19.776506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:45:19.776506+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:47:34.304467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":426,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-20T22:47:33.203275+0000","last_change":"2024-09-20T22:45:40.213422+0000","last_active":"2024-09-20T22:47:33.203275+0000","last_peered":"2024-09-20T22:47:33.203275+0000","last_clean":"2024-09-20T22:47:33.203275+0000","last_became_active":"2024-09-20T22:45:40.213120+0000","last_became_peered":"2024-09-20T22:45:40.213120+0000","last_unstale":"2024-09-20T22:47:33.203275+0000","last_undegraded":"2024-09-20T22:47:33.203275+0000","last_fullsized":"2024-09-20T22:47:33.203275+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:44:19.834641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:44:19.834641+0000","last_clean_scrub_stamp":"2024-09-20T22:44:19.834641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:47:35.121220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"75'5","reported_seq":1011,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-20T22:47:33.413137+0000","last_change":"2024-09-20T22:46:22.436542+0000","last_active":"2024-09-20T22:47:33.413137+0000","last_peered":"2024-09-20T22:47:33.413137+0000","last_clean":"2024-09-20T22:47:33.413137+0000","last_became_active":"2024-09-20T22:45:40.212288+0000","last_became_peered":"2024-09-20T22:45:40.212288+0000","last_unstale":"2024-09-20T22:47:33.413137+0000","last_undegraded":"2024-09-20T22:47:33.413137+0000","last_fullsized":"2024-09-20T22:47:33.413137+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-09-20T22:46:22.436424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:36.130684+0000","last_clean_scrub_stamp":"2024-09-20T22:46:22.436424+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:47:50.777035+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00078631700000000003,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":348,"num_read_kb":343,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":639,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-20T22:47:33.208473+0000","last_change":"2024-09-20T22:47:21.268561+0000","last_active":"2024-09-20T22:47:33.208473+0000","last_peered":"2024-09-20T22:47:33.208473+0000","last_clean":"2024-09-20T22:47:33.208473+0000","last_became_active":"2024-09-20T22:45:40.229194+0000","last_became_peered":"2024-09-20T22:45:40.229194+0000","last_unstale":"2024-09-20T22:47:33.208473+0000","last_undegraded":"2024-09-20T22:47:33.208473+0000","last_fullsized":"2024-09-20T22:47:33.208473+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:47:21.268493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:47:21.268493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:37.727493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":628,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-20T22:47:33.239113+0000","last_change":"2024-09-20T22:44:54.014154+0000","last_active":"2024-09-20T22:47:33.239113+0000","last_peered":"2024-09-20T22:47:33.239113+0000","last_clean":"2024-09-20T22:47:33.239113+0000","last_became_active":"2024-09-20T22:41:44.199046+0000","last_became_peered":"2024-09-20T22:41:44.199046+0000","last_unstale":"2024-09-20T22:47:33.239113+0000","last_undegraded":"2024-09-20T22:47:33.239113+0000","last_fullsized":"2024-09-20T22:47:33.239113+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:44:54.014085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:44:54.014085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:47:34.698426+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.6","version":"372'127","reported_seq":1534,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-20T22:47:33.414403+0000","last_change":"2024-09-20T22:47:32.379333+0000","last_active":"2024-09-20T22:47:33.414403+0000","last_peered":"2024-09-20T22:47:33.414403+0000","last_clean":"2024-09-20T22:47:33.414403+0000","last_became_active":"2024-09-20T22:45:40.217779+0000","last_became_peered":"2024-09-20T22:45:40.217779+0000","last_unstale":"2024-09-20T22:47:33.414403+0000","last_undegraded":"2024-09-20T22:47:33.414403+0000","last_fullsized":"2024-09-20T22:47:33.414403+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"282'87","last_scrub_stamp":"2024-09-20T22:46:03.391903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:46:03.391903+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0041335720000000003,"stat_sum":{"num_bytes":4660216,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":21,"num_read_kb":17312,"num_write":24,"num_write_kb":5144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":2573889,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"6b","length":"1"},{"start":"74","length":"1"},{"start":"89","length":"1"},{"start":"90","length":"1"}]},{"pgid":"4.5","version":"264'18","reported_seq":778,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-20T22:47:33.537555+0000","last_change":"2024-09-20T22:46:04.901692+0000","last_active":"2024-09-20T22:47:33.537555+0000","last_peered":"2024-09-20T22:47:33.537555+0000","last_clean":"2024-09-20T22:47:33.537555+0000","last_became_active":"2024-09-20T22:45:40.220283+0000","last_became_peered":"2024-09-20T22:45:40.220283+0000","last_unstale":"2024-09-20T22:47:33.537555+0000","last_undegraded":"2024-09-20T22:47:33.537555+0000","last_fullsized":"2024-09-20T22:47:33.537555+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"264'18","last_scrub_stamp":"2024-09-20T22:46:04.901561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:46:04.901561+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:47:36.787315+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"67'1","reported_seq":638,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-20T22:47:33.203248+0000","last_change":"2024-09-20T22:45:22.053343+0000","last_active":"2024-09-20T22:47:33.203248+0000","last_peered":"2024-09-20T22:47:33.203248+0000","last_clean":"2024-09-20T22:47:33.203248+0000","last_became_active":"2024-09-20T22:41:38.158917+0000","last_became_peered":"2024-09-20T22:41:38.158917+0000","last_unstale":"2024-09-20T22:47:33.203248+0000","last_undegraded":"2024-09-20T22:47:33.203248+0000","last_fullsized":"2024-09-20T22:47:33.203248+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-09-20T22:45:22.053262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:45:22.053262+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:47:35.121328+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"77'128","reported_seq":972,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-20T22:47:33.413396+0000","last_change":"2024-09-20T22:47:22.417229+0000","last_active":"2024-09-20T22:47:33.413396+0000","last_peered":"2024-09-20T22:47:33.413396+0000","last_clean":"2024-09-20T22:47:33.413396+0000","last_became_active":"2024-09-20T22:45:59.171055+0000","last_became_peered":"2024-09-20T22:45:59.171055+0000","last_unstale":"2024-09-20T22:47:33.413396+0000","last_undegraded":"2024-09-20T22:47:33.413396+0000","last_fullsized":"2024-09-20T22:47:33.413396+0000","mapping_epoch":263,"log_start":"77'128","ondisk_log_start":"77'128","created":19,"last_epoch_clean":280,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:47:22.417099+0000","last_deep_scrub":"77'128","last_deep_scrub_stamp":"2024-09-20T22:47:22.417099+0000","last_clean_scrub_stamp":"2024-09-20T22:47:22.417099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:37.110257+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":633,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-20T22:47:33.413721+0000","last_change":"2024-09-20T22:45:42.357247+0000","last_active":"2024-09-20T22:47:33.413721+0000","last_peered":"2024-09-20T22:47:33.413721+0000","last_clean":"2024-09-20T22:47:33.413721+0000","last_became_active":"2024-09-20T22:41:42.184387+0000","last_became_peered":"2024-09-20T22:41:42.184387+0000","last_unstale":"2024-09-20T22:47:33.413721+0000","last_undegraded":"2024-09-20T22:47:33.413721+0000","last_fullsized":"2024-09-20T22:47:33.413721+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:45:42.357173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:45:42.357173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:47:37.373354+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":635,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-20T22:47:33.208323+0000","last_change":"2024-09-20T22:45:41.820396+0000","last_active":"2024-09-20T22:47:33.208323+0000","last_peered":"2024-09-20T22:47:33.208323+0000","last_clean":"2024-09-20T22:47:33.208323+0000","last_became_active":"2024-09-20T22:43:57.434435+0000","last_became_peered":"2024-09-20T22:43:57.434435+0000","last_unstale":"2024-09-20T22:47:33.208323+0000","last_undegraded":"2024-09-20T22:47:33.208323+0000","last_fullsized":"2024-09-20T22:47:33.208323+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:45:41.820321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:45:41.820321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:47:34.304582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"372'150","reported_seq":1576,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-20T22:47:33.537364+0000","last_change":"2024-09-20T22:47:32.811178+0000","last_active":"2024-09-20T22:47:33.537364+0000","last_peered":"2024-09-20T22:47:33.537364+0000","last_clean":"2024-09-20T22:47:33.537364+0000","last_became_active":"2024-09-20T22:45:40.219625+0000","last_became_peered":"2024-09-20T22:45:40.219625+0000","last_unstale":"2024-09-20T22:47:33.537364+0000","last_undegraded":"2024-09-20T22:47:33.537364+0000","last_fullsized":"2024-09-20T22:47:33.537364+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"327'109","last_scrub_stamp":"2024-09-20T22:46:51.808234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:46:51.808234+0000","objects_scrubbed":2,"log_size":150,"log_dups_size":0,"ondisk_log_size":150,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:02.022453+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0089020569999999997,"stat_sum":{"num_bytes":12471025,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":92,"num_read_kb":93497,"num_write":41,"num_write_kb":19366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2755788,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"6b","length":"1"},{"start":"74","length":"1"},{"start":"89","length":"1"},{"start":"90","length":"1"}]},{"pgid":"4.e","version":"74'11","reported_seq":526,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-20T22:47:33.537258+0000","last_change":"2024-09-20T22:46:49.809263+0000","last_active":"2024-09-20T22:47:33.537258+0000","last_peered":"2024-09-20T22:47:33.537258+0000","last_clean":"2024-09-20T22:47:33.537258+0000","last_became_active":"2024-09-20T22:44:09.177976+0000","last_became_peered":"2024-09-20T22:44:09.177976+0000","last_unstale":"2024-09-20T22:47:33.537258+0000","last_undegraded":"2024-09-20T22:47:33.537258+0000","last_fullsized":"2024-09-20T22:47:33.537258+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-09-20T22:46:49.809151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:46:49.809151+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:47:57.133405+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":459,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-20T22:47:33.188607+0000","last_change":"2024-09-20T22:46:30.871847+0000","last_active":"2024-09-20T22:47:33.188607+0000","last_peered":"2024-09-20T22:47:33.188607+0000","last_clean":"2024-09-20T22:47:33.188607+0000","last_became_active":"2024-09-20T22:43:57.432492+0000","last_became_peered":"2024-09-20T22:43:57.432492+0000","last_unstale":"2024-09-20T22:47:33.188607+0000","last_undegraded":"2024-09-20T22:47:33.188607+0000","last_fullsized":"2024-09-20T22:47:33.188607+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:46:30.871771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:46:30.871771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:00.060628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":631,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-20T22:47:33.208269+0000","last_change":"2024-09-20T22:44:57.607725+0000","last_active":"2024-09-20T22:47:33.208269+0000","last_peered":"2024-09-20T22:47:33.208269+0000","last_clean":"2024-09-20T22:47:33.208269+0000","last_became_active":"2024-09-20T22:43:57.419756+0000","last_became_peered":"2024-09-20T22:43:57.419756+0000","last_unstale":"2024-09-20T22:47:33.208269+0000","last_undegraded":"2024-09-20T22:47:33.208269+0000","last_fullsized":"2024-09-20T22:47:33.208269+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:44:57.607616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:44:57.607616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:47:34.304451+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":620,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-20T22:47:33.204135+0000","last_change":"2024-09-20T22:44:21.225215+0000","last_active":"2024-09-20T22:47:33.204135+0000","last_peered":"2024-09-20T22:47:33.204135+0000","last_clean":"2024-09-20T22:47:33.204135+0000","last_became_active":"2024-09-20T22:41:44.832009+0000","last_became_peered":"2024-09-20T22:41:44.832009+0000","last_unstale":"2024-09-20T22:47:33.204135+0000","last_undegraded":"2024-09-20T22:47:33.204135+0000","last_fullsized":"2024-09-20T22:47:33.204135+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:44:21.225119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:44:21.225119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:47:35.121233+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.e","version":"372'198","reported_seq":1639,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-20T22:47:33.190040+0000","last_change":"2024-09-20T22:47:32.381020+0000","last_active":"2024-09-20T22:47:33.190040+0000","last_peered":"2024-09-20T22:47:33.190040+0000","last_clean":"2024-09-20T22:47:33.190040+0000","last_became_active":"2024-09-20T22:45:43.167013+0000","last_became_peered":"2024-09-20T22:45:43.167013+0000","last_unstale":"2024-09-20T22:47:33.190040+0000","last_undegraded":"2024-09-20T22:47:33.190040+0000","last_fullsized":"2024-09-20T22:47:33.190040+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"270'130","last_scrub_stamp":"2024-09-20T22:45:53.692660+0000","last_deep_scrub":"174'60","last_deep_scrub_stamp":"2024-09-20T22:44:18.301776+0000","last_clean_scrub_stamp":"2024-09-20T22:45:53.692660+0000","objects_scrubbed":4,"log_size":198,"log_dups_size":0,"ondisk_log_size":198,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":1,"snaptrim_duration":0.001713617,"stat_sum":{"num_bytes":11283510,"num_objects":5,"num_object_clones":3,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":129,"num_read_kb":145192,"num_write":71,"num_write_kb":29334,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":9591801,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"6b","length":"1"},{"start":"74","length":"1"},{"start":"89","length":"1"},{"start":"90","length":"1"}]},{"pgid":"1.8","version":"77'128","reported_seq":900,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-20T22:47:33.206544+0000","last_change":"2024-09-20T22:45:22.698507+0000","last_active":"2024-09-20T22:47:33.206544+0000","last_peered":"2024-09-20T22:47:33.206544+0000","last_clean":"2024-09-20T22:47:33.206544+0000","last_became_active":"2024-09-20T22:42:52.405594+0000","last_became_peered":"2024-09-20T22:42:52.405594+0000","last_unstale":"2024-09-20T22:47:33.206544+0000","last_undegraded":"2024-09-20T22:47:33.206544+0000","last_fullsized":"2024-09-20T22:47:33.206544+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:45:22.698390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:45:22.698390+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:47:34.304537+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"74'17","reported_seq":524,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-20T22:47:33.189394+0000","last_change":"2024-09-20T22:45:54.729590+0000","last_active":"2024-09-20T22:47:33.189394+0000","last_peered":"2024-09-20T22:47:33.189394+0000","last_clean":"2024-09-20T22:47:33.189394+0000","last_became_active":"2024-09-20T22:43:57.432621+0000","last_became_peered":"2024-09-20T22:43:57.432621+0000","last_unstale":"2024-09-20T22:47:33.189394+0000","last_undegraded":"2024-09-20T22:47:33.189394+0000","last_fullsized":"2024-09-20T22:47:33.189394+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'17","last_scrub_stamp":"2024-09-20T22:45:54.729502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:45:54.729502+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:47:33.996845+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":28,"num_read_kb":18,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":175,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-20T22:47:33.189844+0000","last_change":"2024-09-20T22:46:10.291173+0000","last_active":"2024-09-20T22:47:33.189844+0000","last_peered":"2024-09-20T22:47:33.189844+0000","last_clean":"2024-09-20T22:47:33.189844+0000","last_became_active":"2024-09-20T22:46:10.290822+0000","last_became_peered":"2024-09-20T22:46:10.290822+0000","last_unstale":"2024-09-20T22:47:33.189844+0000","last_undegraded":"2024-09-20T22:47:33.189844+0000","last_fullsized":"2024-09-20T22:47:33.189844+0000","mapping_epoch":290,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:46:00.879802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:35.492562+0000","last_clean_scrub_stamp":"2024-09-20T22:46:00.879802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:47:33.996929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":636,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-20T22:47:33.239226+0000","last_change":"2024-09-20T22:45:40.208509+0000","last_active":"2024-09-20T22:47:33.239226+0000","last_peered":"2024-09-20T22:47:33.239226+0000","last_clean":"2024-09-20T22:47:33.239226+0000","last_became_active":"2024-09-20T22:45:40.208186+0000","last_became_peered":"2024-09-20T22:45:40.208186+0000","last_unstale":"2024-09-20T22:47:33.239226+0000","last_undegraded":"2024-09-20T22:47:33.239226+0000","last_fullsized":"2024-09-20T22:47:33.239226+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:44:11.041699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:44:11.041699+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:47:34.698381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":642,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-20T22:47:33.190673+0000","last_change":"2024-09-20T22:45:58.798673+0000","last_active":"2024-09-20T22:47:33.190673+0000","last_peered":"2024-09-20T22:47:33.190673+0000","last_clean":"2024-09-20T22:47:33.190673+0000","last_became_active":"2024-09-20T22:43:57.430630+0000","last_became_peered":"2024-09-20T22:43:57.430630+0000","last_unstale":"2024-09-20T22:47:33.190673+0000","last_undegraded":"2024-09-20T22:47:33.190673+0000","last_fullsized":"2024-09-20T22:47:33.190673+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:45:58.798603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:45:58.798603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:47:33.996874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"370'292","reported_seq":1735,"reported_epoch":373,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-20T22:47:33.204473+0000","last_change":"2024-09-20T22:47:31.240244+0000","last_active":"2024-09-20T22:47:33.204473+0000","last_peered":"2024-09-20T22:47:33.204473+0000","last_clean":"2024-09-20T22:47:33.204473+0000","last_became_active":"2024-09-20T22:45:43.170143+0000","last_became_peered":"2024-09-20T22:45:43.170143+0000","last_unstale":"2024-09-20T22:47:33.204473+0000","last_undegraded":"2024-09-20T22:47:33.204473+0000","last_fullsized":"2024-09-20T22:47:33.204473+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"194'90","last_scrub_stamp":"2024-09-20T22:44:34.651203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:44:34.651203+0000","objects_scrubbed":3,"log_size":292,"log_dups_size":0,"ondisk_log_size":292,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.97074054799999998,"stat_sum":{"num_bytes":13288085,"num_objects":7,"num_object_clones":4,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":207,"num_read_kb":214522,"num_write":116,"num_write_kb":35440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":10116431,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"89","length":"1"},{"start":"90","length":"1"}]},{"pgid":"1.9","version":"77'128","reported_seq":941,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-20T22:47:33.413174+0000","last_change":"2024-09-20T22:45:24.337974+0000","last_active":"2024-09-20T22:47:33.413174+0000","last_peered":"2024-09-20T22:47:33.413174+0000","last_clean":"2024-09-20T22:47:33.413174+0000","last_became_active":"2024-09-20T22:42:52.400353+0000","last_became_peered":"2024-09-20T22:42:52.400353+0000","last_unstale":"2024-09-20T22:47:33.413174+0000","last_undegraded":"2024-09-20T22:47:33.413174+0000","last_fullsized":"2024-09-20T22:47:33.413174+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:45:24.337881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:45:24.337881+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:47:37.373166+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"74'10","reported_seq":496,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-20T22:47:33.203948+0000","last_change":"2024-09-20T22:47:21.092403+0000","last_active":"2024-09-20T22:47:33.203948+0000","last_peered":"2024-09-20T22:47:33.203948+0000","last_clean":"2024-09-20T22:47:33.203948+0000","last_became_active":"2024-09-20T22:43:57.441016+0000","last_became_peered":"2024-09-20T22:43:57.441016+0000","last_unstale":"2024-09-20T22:47:33.203948+0000","last_undegraded":"2024-09-20T22:47:33.203948+0000","last_fullsized":"2024-09-20T22:47:33.203948+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-09-20T22:47:21.092309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:47:21.092309+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:34.540078+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":652,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-20T22:47:33.203842+0000","last_change":"2024-09-20T22:45:57.194785+0000","last_active":"2024-09-20T22:47:33.203842+0000","last_peered":"2024-09-20T22:47:33.203842+0000","last_clean":"2024-09-20T22:47:33.203842+0000","last_became_active":"2024-09-20T22:45:40.228762+0000","last_became_peered":"2024-09-20T22:45:40.228762+0000","last_unstale":"2024-09-20T22:47:33.203842+0000","last_undegraded":"2024-09-20T22:47:33.203842+0000","last_fullsized":"2024-09-20T22:47:33.203842+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:45:57.194715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:45:57.194715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:47:35.121401+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"261'9","reported_seq":721,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-20T22:47:33.189309+0000","last_change":"2024-09-20T22:46:50.974814+0000","last_active":"2024-09-20T22:47:33.189309+0000","last_peered":"2024-09-20T22:47:33.189309+0000","last_clean":"2024-09-20T22:47:33.189309+0000","last_became_active":"2024-09-20T22:41:42.185863+0000","last_became_peered":"2024-09-20T22:41:42.185863+0000","last_unstale":"2024-09-20T22:47:33.189309+0000","last_undegraded":"2024-09-20T22:47:33.189309+0000","last_fullsized":"2024-09-20T22:47:33.189309+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'9","last_scrub_stamp":"2024-09-20T22:46:50.974735+0000","last_deep_scrub":"261'9","last_deep_scrub_stamp":"2024-09-20T22:46:50.974735+0000","last_clean_scrub_stamp":"2024-09-20T22:46:50.974735+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:47:51.558873+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":459,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-20T22:47:33.239838+0000","last_change":"2024-09-20T22:46:24.906585+0000","last_active":"2024-09-20T22:47:33.239838+0000","last_peered":"2024-09-20T22:47:33.239838+0000","last_clean":"2024-09-20T22:47:33.239838+0000","last_became_active":"2024-09-20T22:43:57.430067+0000","last_became_peered":"2024-09-20T22:43:57.430067+0000","last_unstale":"2024-09-20T22:47:33.239838+0000","last_undegraded":"2024-09-20T22:47:33.239838+0000","last_fullsized":"2024-09-20T22:47:33.239838+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:46:24.906516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:46:24.906516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:47:54.740972+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.8","version":"369'105","reported_seq":1520,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-20T22:47:33.188145+0000","last_change":"2024-09-20T22:47:32.377793+0000","last_active":"2024-09-20T22:47:33.188145+0000","last_peered":"2024-09-20T22:47:33.188145+0000","last_clean":"2024-09-20T22:47:33.188145+0000","last_became_active":"2024-09-20T22:43:57.431081+0000","last_became_peered":"2024-09-20T22:43:57.431081+0000","last_unstale":"2024-09-20T22:47:33.188145+0000","last_undegraded":"2024-09-20T22:47:33.188145+0000","last_fullsized":"2024-09-20T22:47:33.188145+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'98","last_scrub_stamp":"2024-09-20T22:46:45.991157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:46:45.991157+0000","objects_scrubbed":0,"log_size":105,"log_dups_size":0,"ondisk_log_size":105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:47:55.414465+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00065412900000000004,"stat_sum":{"num_bytes":3755447,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":25,"num_read_kb":20437,"num_write":21,"num_write_kb":11116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"6b","length":"1"},{"start":"74","length":"1"},{"start":"89","length":"1"},{"start":"90","length":"1"}]},{"pgid":"4.b","version":"74'9","reported_seq":280,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-20T22:47:33.239187+0000","last_change":"2024-09-20T22:45:40.256230+0000","last_active":"2024-09-20T22:47:33.239187+0000","last_peered":"2024-09-20T22:47:33.239187+0000","last_clean":"2024-09-20T22:47:33.239187+0000","last_became_active":"2024-09-20T22:45:40.222847+0000","last_became_peered":"2024-09-20T22:45:40.222847+0000","last_unstale":"2024-09-20T22:47:33.239187+0000","last_undegraded":"2024-09-20T22:47:33.239187+0000","last_fullsized":"2024-09-20T22:47:33.239187+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-09-20T22:44:57.603671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:44:57.603671+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:47:34.698458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,5],"acting":[1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":641,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-20T22:47:33.207884+0000","last_change":"2024-09-20T22:46:25.141094+0000","last_active":"2024-09-20T22:47:33.207884+0000","last_peered":"2024-09-20T22:47:33.207884+0000","last_clean":"2024-09-20T22:47:33.207884+0000","last_became_active":"2024-09-20T22:41:38.155953+0000","last_became_peered":"2024-09-20T22:41:38.155953+0000","last_unstale":"2024-09-20T22:47:33.207884+0000","last_undegraded":"2024-09-20T22:47:33.207884+0000","last_fullsized":"2024-09-20T22:47:33.207884+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:46:25.141028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:46:25.141028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:47:38.610184+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":652,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-20T22:47:33.188719+0000","last_change":"2024-09-20T22:46:49.061935+0000","last_active":"2024-09-20T22:47:33.188719+0000","last_peered":"2024-09-20T22:47:33.188719+0000","last_clean":"2024-09-20T22:47:33.188719+0000","last_became_active":"2024-09-20T22:45:40.212560+0000","last_became_peered":"2024-09-20T22:45:40.212560+0000","last_unstale":"2024-09-20T22:47:33.188719+0000","last_undegraded":"2024-09-20T22:47:33.188719+0000","last_fullsized":"2024-09-20T22:47:33.188719+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:46:49.061850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:46:49.061850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:16.726402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":244,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-20T22:47:33.413414+0000","last_change":"2024-09-20T22:46:51.469673+0000","last_active":"2024-09-20T22:47:33.413414+0000","last_peered":"2024-09-20T22:47:33.413414+0000","last_clean":"2024-09-20T22:47:33.413414+0000","last_became_active":"2024-09-20T22:45:40.221355+0000","last_became_peered":"2024-09-20T22:45:40.221355+0000","last_unstale":"2024-09-20T22:47:33.413414+0000","last_undegraded":"2024-09-20T22:47:33.413414+0000","last_fullsized":"2024-09-20T22:47:33.413414+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:46:51.469567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:46:51.469567+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:07.052829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.9","version":"371'222","reported_seq":1665,"reported_epoch":373,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-20T22:47:33.240863+0000","last_change":"2024-09-20T22:47:33.240885+0000","last_active":"2024-09-20T22:47:33.240863+0000","last_peered":"2024-09-20T22:47:33.240863+0000","last_clean":"2024-09-20T22:47:33.240863+0000","last_became_active":"2024-09-20T22:45:43.166728+0000","last_became_peered":"2024-09-20T22:45:43.166728+0000","last_unstale":"2024-09-20T22:47:33.240863+0000","last_undegraded":"2024-09-20T22:47:33.240863+0000","last_fullsized":"2024-09-20T22:47:33.240863+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"257'132","last_scrub_stamp":"2024-09-20T22:45:36.793712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:45:36.793712+0000","objects_scrubbed":6,"log_size":222,"log_dups_size":0,"ondisk_log_size":222,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0019813550000000002,"stat_sum":{"num_bytes":14121291,"num_objects":7,"num_object_clones":5,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":79,"num_read_kb":87324,"num_write":69,"num_write_kb":26957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":13891316,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,6],"acting":[1,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"74","length":"1"},{"start":"89","length":"1"},{"start":"90","length":"1"}]},{"pgid":"4.a","version":"113'28","reported_seq":684,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-20T22:47:33.538072+0000","last_change":"2024-09-20T22:47:13.842452+0000","last_active":"2024-09-20T22:47:33.538072+0000","last_peered":"2024-09-20T22:47:33.538072+0000","last_clean":"2024-09-20T22:47:33.538072+0000","last_became_active":"2024-09-20T22:41:40.252011+0000","last_became_peered":"2024-09-20T22:41:40.252011+0000","last_unstale":"2024-09-20T22:47:33.538072+0000","last_undegraded":"2024-09-20T22:47:33.538072+0000","last_fullsized":"2024-09-20T22:47:33.538072+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'28","last_scrub_stamp":"2024-09-20T22:47:13.842316+0000","last_deep_scrub":"113'28","last_deep_scrub_stamp":"2024-09-20T22:43:12.725900+0000","last_clean_scrub_stamp":"2024-09-20T22:47:13.842316+0000","objects_scrubbed":9,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:31.564433+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":641,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-20T22:47:33.414339+0000","last_change":"2024-09-20T22:46:30.430426+0000","last_active":"2024-09-20T22:47:33.414339+0000","last_peered":"2024-09-20T22:47:33.414339+0000","last_clean":"2024-09-20T22:47:33.414339+0000","last_became_active":"2024-09-20T22:41:38.161524+0000","last_became_peered":"2024-09-20T22:41:38.161524+0000","last_unstale":"2024-09-20T22:47:33.414339+0000","last_undegraded":"2024-09-20T22:47:33.414339+0000","last_fullsized":"2024-09-20T22:47:33.414339+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:46:30.430353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:46:30.430353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:47:53.016290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":638,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-20T22:47:33.188927+0000","last_change":"2024-09-20T22:45:40.214399+0000","last_active":"2024-09-20T22:47:33.188927+0000","last_peered":"2024-09-20T22:47:33.188927+0000","last_clean":"2024-09-20T22:47:33.188927+0000","last_became_active":"2024-09-20T22:45:40.214075+0000","last_became_peered":"2024-09-20T22:45:40.214075+0000","last_unstale":"2024-09-20T22:47:33.188927+0000","last_undegraded":"2024-09-20T22:47:33.188927+0000","last_fullsized":"2024-09-20T22:47:33.188927+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:45:23.475514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:45:23.475514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:47:33.996805+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":637,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-20T22:47:33.413172+0000","last_change":"2024-09-20T22:47:18.329286+0000","last_active":"2024-09-20T22:47:33.413172+0000","last_peered":"2024-09-20T22:47:33.413172+0000","last_clean":"2024-09-20T22:47:33.413172+0000","last_became_active":"2024-09-20T22:41:44.199395+0000","last_became_peered":"2024-09-20T22:41:44.199395+0000","last_unstale":"2024-09-20T22:47:33.413172+0000","last_undegraded":"2024-09-20T22:47:33.413172+0000","last_fullsized":"2024-09-20T22:47:33.413172+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:47:18.329209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:47:18.329209+0000","last_clean_scrub_stamp":"2024-09-20T22:47:18.329209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:22.373197+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.a","version":"373'298","reported_seq":1717,"reported_epoch":373,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T22:47:33.212804+0000","last_change":"2024-09-20T22:47:33.211245+0000","last_active":"2024-09-20T22:47:33.212804+0000","last_peered":"2024-09-20T22:47:33.212804+0000","last_clean":"2024-09-20T22:47:33.212804+0000","last_became_active":"2024-09-20T22:42:52.405085+0000","last_became_peered":"2024-09-20T22:42:52.405085+0000","last_unstale":"2024-09-20T22:47:33.212804+0000","last_undegraded":"2024-09-20T22:47:33.212804+0000","last_fullsized":"2024-09-20T22:47:33.212804+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"301'219","last_scrub_stamp":"2024-09-20T22:46:23.260041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:46:23.260041+0000","objects_scrubbed":3,"log_size":298,"log_dups_size":0,"ondisk_log_size":298,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:47:49.716021+0000","scrub_duration":2,"objects_trimmed":1,"snaptrim_duration":0.97005414899999998,"stat_sum":{"num_bytes":19215419,"num_objects":8,"num_object_clones":5,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":230,"num_read_kb":256829,"num_write":133,"num_write_kb":48041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"89","length":"1"},{"start":"90","length":"1"}]},{"pgid":"4.9","version":"74'12","reported_seq":504,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-20T22:47:33.239213+0000","last_change":"2024-09-20T22:44:53.049195+0000","last_active":"2024-09-20T22:47:33.239213+0000","last_peered":"2024-09-20T22:47:33.239213+0000","last_clean":"2024-09-20T22:47:33.239213+0000","last_became_active":"2024-09-20T22:43:57.442218+0000","last_became_peered":"2024-09-20T22:43:57.442218+0000","last_unstale":"2024-09-20T22:47:33.239213+0000","last_undegraded":"2024-09-20T22:47:33.239213+0000","last_fullsized":"2024-09-20T22:47:33.239213+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-09-20T22:44:53.049111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:44:53.049111+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:47:34.698412+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":21,"num_read_kb":12,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":661,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-20T22:47:33.413655+0000","last_change":"2024-09-20T22:46:52.466011+0000","last_active":"2024-09-20T22:47:33.413655+0000","last_peered":"2024-09-20T22:47:33.413655+0000","last_clean":"2024-09-20T22:47:33.413655+0000","last_became_active":"2024-09-20T22:45:40.212393+0000","last_became_peered":"2024-09-20T22:45:40.212393+0000","last_unstale":"2024-09-20T22:47:33.413655+0000","last_undegraded":"2024-09-20T22:47:33.413655+0000","last_fullsized":"2024-09-20T22:47:33.413655+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:46:52.465940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:46:52.465940+0000","last_clean_scrub_stamp":"2024-09-20T22:46:52.465940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:16.208150+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":646,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-20T22:47:33.188263+0000","last_change":"2024-09-20T22:46:01.828155+0000","last_active":"2024-09-20T22:47:33.188263+0000","last_peered":"2024-09-20T22:47:33.188263+0000","last_clean":"2024-09-20T22:47:33.188263+0000","last_became_active":"2024-09-20T22:45:40.205381+0000","last_became_peered":"2024-09-20T22:45:40.205381+0000","last_unstale":"2024-09-20T22:47:33.188263+0000","last_undegraded":"2024-09-20T22:47:33.188263+0000","last_fullsized":"2024-09-20T22:47:33.188263+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:46:01.828080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:46:01.828080+0000","last_clean_scrub_stamp":"2024-09-20T22:46:01.828080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":620,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-20T22:47:33.203455+0000","last_change":"2024-09-20T22:43:55.170518+0000","last_active":"2024-09-20T22:47:33.203455+0000","last_peered":"2024-09-20T22:47:33.203455+0000","last_clean":"2024-09-20T22:47:33.203455+0000","last_became_active":"2024-09-20T22:41:44.192609+0000","last_became_peered":"2024-09-20T22:41:44.192609+0000","last_unstale":"2024-09-20T22:47:33.203455+0000","last_undegraded":"2024-09-20T22:47:33.203455+0000","last_fullsized":"2024-09-20T22:47:33.203455+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:55.170379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:55.170379+0000","last_clean_scrub_stamp":"2024-09-20T22:43:55.170379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:47:35.121062+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"372'437","reported_seq":1863,"reported_epoch":373,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T22:47:33.239688+0000","last_change":"2024-09-20T22:47:33.197539+0000","last_active":"2024-09-20T22:47:33.239688+0000","last_peered":"2024-09-20T22:47:33.239688+0000","last_clean":"2024-09-20T22:47:33.239688+0000","last_became_active":"2024-09-20T22:42:52.409642+0000","last_became_peered":"2024-09-20T22:42:52.409642+0000","last_unstale":"2024-09-20T22:47:33.239688+0000","last_undegraded":"2024-09-20T22:47:33.239688+0000","last_fullsized":"2024-09-20T22:47:33.239688+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"194'116","last_scrub_stamp":"2024-09-20T22:44:37.097674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:44:37.097674+0000","objects_scrubbed":3,"log_size":437,"log_dups_size":0,"ondisk_log_size":437,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.96103275200000005,"stat_sum":{"num_bytes":31152027,"num_objects":16,"num_object_clones":12,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":281,"num_read_kb":284687,"num_write":208,"num_write_kb":68549,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"89","length":"1"},{"start":"90","length":"1"}]},{"pgid":"4.8","version":"74'15","reported_seq":654,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-20T22:47:33.206776+0000","last_change":"2024-09-20T22:45:00.648680+0000","last_active":"2024-09-20T22:47:33.206776+0000","last_peered":"2024-09-20T22:47:33.206776+0000","last_clean":"2024-09-20T22:47:33.206776+0000","last_became_active":"2024-09-20T22:41:40.252113+0000","last_became_peered":"2024-09-20T22:41:40.252113+0000","last_unstale":"2024-09-20T22:47:33.206776+0000","last_undegraded":"2024-09-20T22:47:33.206776+0000","last_fullsized":"2024-09-20T22:47:33.206776+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-09-20T22:45:00.648587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:45:00.648587+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:47:34.304420+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'2","reported_seq":677,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-20T22:47:33.414609+0000","last_change":"2024-09-20T22:46:48.473576+0000","last_active":"2024-09-20T22:47:33.414609+0000","last_peered":"2024-09-20T22:47:33.414609+0000","last_clean":"2024-09-20T22:47:33.414609+0000","last_became_active":"2024-09-20T22:45:40.213804+0000","last_became_peered":"2024-09-20T22:45:40.213804+0000","last_unstale":"2024-09-20T22:47:33.414609+0000","last_undegraded":"2024-09-20T22:47:33.414609+0000","last_fullsized":"2024-09-20T22:47:33.414609+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-09-20T22:46:48.473494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:46:48.473494+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:12.933141+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"159'9","reported_seq":723,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-20T22:47:33.413541+0000","last_change":"2024-09-20T22:44:36.904443+0000","last_active":"2024-09-20T22:47:33.413541+0000","last_peered":"2024-09-20T22:47:33.413541+0000","last_clean":"2024-09-20T22:47:33.413541+0000","last_became_active":"2024-09-20T22:43:57.420663+0000","last_became_peered":"2024-09-20T22:43:57.420663+0000","last_unstale":"2024-09-20T22:47:33.413541+0000","last_undegraded":"2024-09-20T22:47:33.413541+0000","last_fullsized":"2024-09-20T22:47:33.413541+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"159'9","last_scrub_stamp":"2024-09-20T22:44:36.904367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:44:36.904367+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:47:37.372954+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":627,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-20T22:47:33.207319+0000","last_change":"2024-09-20T22:45:40.217012+0000","last_active":"2024-09-20T22:47:33.207319+0000","last_peered":"2024-09-20T22:47:33.207319+0000","last_clean":"2024-09-20T22:47:33.207319+0000","last_became_active":"2024-09-20T22:45:40.216593+0000","last_became_peered":"2024-09-20T22:45:40.216593+0000","last_unstale":"2024-09-20T22:47:33.207319+0000","last_undegraded":"2024-09-20T22:47:33.207319+0000","last_fullsized":"2024-09-20T22:47:33.207319+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:45:01.616426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:45:01.616426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:47:34.304481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.14","version":"373'110","reported_seq":1264,"reported_epoch":373,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T22:47:33.243671+0000","last_change":"2024-09-20T22:47:33.243671+0000","last_active":"2024-09-20T22:47:33.243671+0000","last_peered":"2024-09-20T22:47:33.243671+0000","last_clean":"2024-09-20T22:47:33.243671+0000","last_became_active":"2024-09-20T22:43:39.132459+0000","last_became_peered":"2024-09-20T22:43:39.132459+0000","last_unstale":"2024-09-20T22:47:33.243671+0000","last_undegraded":"2024-09-20T22:47:33.243671+0000","last_fullsized":"2024-09-20T22:47:33.243671+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":5,"last_scrub":"147'38","last_scrub_stamp":"2024-09-20T22:43:49.134297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:43:49.134297+0000","objects_scrubbed":0,"log_size":110,"log_dups_size":0,"ondisk_log_size":110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":1,"snaptrim_duration":0.96987785900000001,"stat_sum":{"num_bytes":5876815,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":44,"num_read_kb":36718,"num_write":37,"num_write_kb":10623,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"89","length":"1"},{"start":"90","length":"1"}]},{"pgid":"3.10","version":"0'0","reported_seq":654,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-20T22:47:33.537699+0000","last_change":"2024-09-20T22:47:12.820395+0000","last_active":"2024-09-20T22:47:33.537699+0000","last_peered":"2024-09-20T22:47:33.537699+0000","last_clean":"2024-09-20T22:47:33.537699+0000","last_became_active":"2024-09-20T22:45:40.212409+0000","last_became_peered":"2024-09-20T22:45:40.212409+0000","last_unstale":"2024-09-20T22:47:33.537699+0000","last_undegraded":"2024-09-20T22:47:33.537699+0000","last_fullsized":"2024-09-20T22:47:33.537699+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:47:12.820304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:47:12.820304+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:28.477165+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"74'6","reported_seq":659,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-20T22:47:33.204868+0000","last_change":"2024-09-20T22:45:39.079425+0000","last_active":"2024-09-20T22:47:33.204868+0000","last_peered":"2024-09-20T22:47:33.204868+0000","last_clean":"2024-09-20T22:47:33.204868+0000","last_became_active":"2024-09-20T22:43:09.116887+0000","last_became_peered":"2024-09-20T22:43:09.116887+0000","last_unstale":"2024-09-20T22:47:33.204868+0000","last_undegraded":"2024-09-20T22:47:33.204868+0000","last_fullsized":"2024-09-20T22:47:33.204868+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'6","last_scrub_stamp":"2024-09-20T22:45:39.079346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:45:39.079346+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:47:35.121357+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":633,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-20T22:47:33.208342+0000","last_change":"2024-09-20T22:45:54.949745+0000","last_active":"2024-09-20T22:47:33.208342+0000","last_peered":"2024-09-20T22:47:33.208342+0000","last_clean":"2024-09-20T22:47:33.208342+0000","last_became_active":"2024-09-20T22:41:42.184930+0000","last_became_peered":"2024-09-20T22:41:42.184930+0000","last_unstale":"2024-09-20T22:47:33.208342+0000","last_undegraded":"2024-09-20T22:47:33.208342+0000","last_fullsized":"2024-09-20T22:47:33.208342+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:45:54.949661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:09.705632+0000","last_clean_scrub_stamp":"2024-09-20T22:45:54.949661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:47:34.304595+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"74'1","reported_seq":640,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-20T22:47:33.413937+0000","last_change":"2024-09-20T22:45:59.486156+0000","last_active":"2024-09-20T22:47:33.413937+0000","last_peered":"2024-09-20T22:47:33.413937+0000","last_clean":"2024-09-20T22:47:33.413937+0000","last_became_active":"2024-09-20T22:43:57.419681+0000","last_became_peered":"2024-09-20T22:43:57.419681+0000","last_unstale":"2024-09-20T22:47:33.413937+0000","last_undegraded":"2024-09-20T22:47:33.413937+0000","last_fullsized":"2024-09-20T22:47:33.413937+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-09-20T22:45:59.486075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:45:59.486075+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:47:37.373328+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.15","version":"98'3","reported_seq":1393,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-20T22:47:33.413770+0000","last_change":"2024-09-20T22:47:32.375869+0000","last_active":"2024-09-20T22:47:33.413770+0000","last_peered":"2024-09-20T22:47:33.413770+0000","last_clean":"2024-09-20T22:47:33.413770+0000","last_became_active":"2024-09-20T22:44:09.174719+0000","last_became_peered":"2024-09-20T22:44:09.174719+0000","last_unstale":"2024-09-20T22:47:33.413770+0000","last_undegraded":"2024-09-20T22:47:33.413770+0000","last_fullsized":"2024-09-20T22:47:33.413770+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":5,"last_scrub":"98'3","last_scrub_stamp":"2024-09-20T22:45:14.323226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:45:14.323226+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00089293200000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"6b","length":"1"},{"start":"74","length":"1"},{"start":"89","length":"1"},{"start":"90","length":"1"}]},{"pgid":"4.16","version":"74'9","reported_seq":288,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-20T22:47:33.204730+0000","last_change":"2024-09-20T22:46:21.269606+0000","last_active":"2024-09-20T22:47:33.204730+0000","last_peered":"2024-09-20T22:47:33.204730+0000","last_clean":"2024-09-20T22:47:33.204730+0000","last_became_active":"2024-09-20T22:45:40.218216+0000","last_became_peered":"2024-09-20T22:45:40.218216+0000","last_unstale":"2024-09-20T22:47:33.204730+0000","last_undegraded":"2024-09-20T22:47:33.204730+0000","last_fullsized":"2024-09-20T22:47:33.204730+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-09-20T22:46:21.269487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:46:21.269487+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:47:42.406334+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":653,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-20T22:47:33.413053+0000","last_change":"2024-09-20T22:46:46.438664+0000","last_active":"2024-09-20T22:47:33.413053+0000","last_peered":"2024-09-20T22:47:33.413053+0000","last_clean":"2024-09-20T22:47:33.413053+0000","last_became_active":"2024-09-20T22:45:40.215995+0000","last_became_peered":"2024-09-20T22:45:40.215995+0000","last_unstale":"2024-09-20T22:47:33.413053+0000","last_undegraded":"2024-09-20T22:47:33.413053+0000","last_fullsized":"2024-09-20T22:47:33.413053+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:46:46.438556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:46:46.438556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:47:47.974917+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":632,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-20T22:47:33.204693+0000","last_change":"2024-09-20T22:46:48.012442+0000","last_active":"2024-09-20T22:47:33.204693+0000","last_peered":"2024-09-20T22:47:33.204693+0000","last_clean":"2024-09-20T22:47:33.204693+0000","last_became_active":"2024-09-20T22:41:42.187706+0000","last_became_peered":"2024-09-20T22:41:42.187706+0000","last_unstale":"2024-09-20T22:47:33.204693+0000","last_undegraded":"2024-09-20T22:47:33.204693+0000","last_fullsized":"2024-09-20T22:47:33.204693+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:46:48.012372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:46:48.012372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:11.078815+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":648,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-20T22:47:33.188946+0000","last_change":"2024-09-20T22:46:29.870305+0000","last_active":"2024-09-20T22:47:33.188946+0000","last_peered":"2024-09-20T22:47:33.188946+0000","last_clean":"2024-09-20T22:47:33.188946+0000","last_became_active":"2024-09-20T22:45:40.217798+0000","last_became_peered":"2024-09-20T22:45:40.217798+0000","last_unstale":"2024-09-20T22:47:33.188946+0000","last_undegraded":"2024-09-20T22:47:33.188946+0000","last_fullsized":"2024-09-20T22:47:33.188946+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:46:29.870228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:46:29.870228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:47:59.415198+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.16","version":"355'119","reported_seq":1452,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-20T22:47:33.414550+0000","last_change":"2024-09-20T22:47:32.378191+0000","last_active":"2024-09-20T22:47:33.414550+0000","last_peered":"2024-09-20T22:47:33.414550+0000","last_clean":"2024-09-20T22:47:33.414550+0000","last_became_active":"2024-09-20T22:45:40.220472+0000","last_became_peered":"2024-09-20T22:45:40.220472+0000","last_unstale":"2024-09-20T22:47:33.414550+0000","last_undegraded":"2024-09-20T22:47:33.414550+0000","last_fullsized":"2024-09-20T22:47:33.414550+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":5,"last_scrub":"282'91","last_scrub_stamp":"2024-09-20T22:46:05.393044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:46:05.393044+0000","objects_scrubbed":1,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0026042000000000001,"stat_sum":{"num_bytes":7176721,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":67,"num_read_kb":74958,"num_write":28,"num_write_kb":10963,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":3271548,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"6b","length":"1"},{"start":"74","length":"1"},{"start":"89","length":"1"},{"start":"90","length":"1"}]},{"pgid":"4.15","version":"74'9","reported_seq":653,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-20T22:47:33.207946+0000","last_change":"2024-09-20T22:46:27.123331+0000","last_active":"2024-09-20T22:47:33.207946+0000","last_peered":"2024-09-20T22:47:33.207946+0000","last_clean":"2024-09-20T22:47:33.207946+0000","last_became_active":"2024-09-20T22:41:40.252486+0000","last_became_peered":"2024-09-20T22:41:40.252486+0000","last_unstale":"2024-09-20T22:47:33.207946+0000","last_undegraded":"2024-09-20T22:47:33.207946+0000","last_fullsized":"2024-09-20T22:47:33.207946+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-09-20T22:46:27.123248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:46:27.123248+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:47:50.439941+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":236,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-20T22:47:33.413511+0000","last_change":"2024-09-20T22:45:40.212127+0000","last_active":"2024-09-20T22:47:33.413511+0000","last_peered":"2024-09-20T22:47:33.413511+0000","last_clean":"2024-09-20T22:47:33.413511+0000","last_became_active":"2024-09-20T22:45:40.211736+0000","last_became_peered":"2024-09-20T22:45:40.211736+0000","last_unstale":"2024-09-20T22:47:33.413511+0000","last_undegraded":"2024-09-20T22:47:33.413511+0000","last_fullsized":"2024-09-20T22:47:33.413511+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:45:23.789053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:45:23.789053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:47:37.373056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.14","version":"74'8","reported_seq":704,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-20T22:47:33.204087+0000","last_change":"2024-09-20T22:43:52.190172+0000","last_active":"2024-09-20T22:47:33.204087+0000","last_peered":"2024-09-20T22:47:33.204087+0000","last_clean":"2024-09-20T22:47:33.204087+0000","last_became_active":"2024-09-20T22:41:42.186352+0000","last_became_peered":"2024-09-20T22:41:42.186352+0000","last_unstale":"2024-09-20T22:47:33.204087+0000","last_undegraded":"2024-09-20T22:47:33.204087+0000","last_fullsized":"2024-09-20T22:47:33.204087+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-09-20T22:43:52.190097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:43:52.190097+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:47:35.121104+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":474,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-20T22:47:33.190668+0000","last_change":"2024-09-20T22:47:19.828610+0000","last_active":"2024-09-20T22:47:33.190668+0000","last_peered":"2024-09-20T22:47:33.190668+0000","last_clean":"2024-09-20T22:47:33.190668+0000","last_became_active":"2024-09-20T22:45:40.214556+0000","last_became_peered":"2024-09-20T22:45:40.214556+0000","last_unstale":"2024-09-20T22:47:33.190668+0000","last_undegraded":"2024-09-20T22:47:33.190668+0000","last_fullsized":"2024-09-20T22:47:33.190668+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:47:19.828541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:47:19.828541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:26.698779+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.17","version":"372'319","reported_seq":1445,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-20T22:47:33.414458+0000","last_change":"2024-09-20T22:47:32.377774+0000","last_active":"2024-09-20T22:47:33.414458+0000","last_peered":"2024-09-20T22:47:33.414458+0000","last_clean":"2024-09-20T22:47:33.414458+0000","last_became_active":"2024-09-20T22:43:39.130801+0000","last_became_peered":"2024-09-20T22:43:39.130801+0000","last_unstale":"2024-09-20T22:47:33.414458+0000","last_undegraded":"2024-09-20T22:47:33.414458+0000","last_fullsized":"2024-09-20T22:47:33.414458+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":5,"last_scrub":"270'229","last_scrub_stamp":"2024-09-20T22:45:53.389694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:45:53.389694+0000","objects_scrubbed":5,"log_size":319,"log_dups_size":0,"ondisk_log_size":319,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":1,"snaptrim_duration":0.0028481869999999999,"stat_sum":{"num_bytes":13720013,"num_objects":8,"num_object_clones":6,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":1,"num_read":123,"num_read_kb":134683,"num_write":76,"num_write_kb":28153,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"6b","length":"1"},{"start":"74","length":"1"},{"start":"89","length":"1"},{"start":"90","length":"1"}]},{"pgid":"4.14","version":"74'10","reported_seq":651,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-20T22:47:33.202604+0000","last_change":"2024-09-20T22:46:24.269736+0000","last_active":"2024-09-20T22:47:33.202604+0000","last_peered":"2024-09-20T22:47:33.202604+0000","last_clean":"2024-09-20T22:47:33.202604+0000","last_became_active":"2024-09-20T22:41:40.253614+0000","last_became_peered":"2024-09-20T22:41:40.253614+0000","last_unstale":"2024-09-20T22:47:33.202604+0000","last_undegraded":"2024-09-20T22:47:33.202604+0000","last_fullsized":"2024-09-20T22:47:33.202604+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-09-20T22:46:24.269655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:46:24.269655+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:47:35.121490+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":647,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-20T22:47:33.413704+0000","last_change":"2024-09-20T22:45:39.355210+0000","last_active":"2024-09-20T22:47:33.413704+0000","last_peered":"2024-09-20T22:47:33.413704+0000","last_clean":"2024-09-20T22:47:33.413704+0000","last_became_active":"2024-09-20T22:43:57.425004+0000","last_became_peered":"2024-09-20T22:43:57.425004+0000","last_unstale":"2024-09-20T22:47:33.413704+0000","last_undegraded":"2024-09-20T22:47:33.413704+0000","last_fullsized":"2024-09-20T22:47:33.413704+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:45:39.355129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:45:39.355129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:47:37.373134+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"264'9","reported_seq":726,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-20T22:47:33.207846+0000","last_change":"2024-09-20T22:45:40.260046+0000","last_active":"2024-09-20T22:47:33.207846+0000","last_peered":"2024-09-20T22:47:33.207846+0000","last_clean":"2024-09-20T22:47:33.207846+0000","last_became_active":"2024-09-20T22:45:40.224430+0000","last_became_peered":"2024-09-20T22:45:40.224430+0000","last_unstale":"2024-09-20T22:47:33.207846+0000","last_undegraded":"2024-09-20T22:47:33.207846+0000","last_fullsized":"2024-09-20T22:47:33.207846+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-09-20T22:45:19.708971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:45:19.708971+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:47:34.304568+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":236,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-20T22:47:33.414247+0000","last_change":"2024-09-20T22:45:40.218103+0000","last_active":"2024-09-20T22:47:33.414247+0000","last_peered":"2024-09-20T22:47:33.414247+0000","last_clean":"2024-09-20T22:47:33.414247+0000","last_became_active":"2024-09-20T22:45:40.217739+0000","last_became_peered":"2024-09-20T22:45:40.217739+0000","last_unstale":"2024-09-20T22:47:33.414247+0000","last_undegraded":"2024-09-20T22:47:33.414247+0000","last_fullsized":"2024-09-20T22:47:33.414247+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:45:00.729995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:45:00.729995+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:47:37.373003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.10","version":"370'110","reported_seq":1486,"reported_epoch":373,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-20T22:47:33.210783+0000","last_change":"2024-09-20T22:47:33.210827+0000","last_active":"2024-09-20T22:47:33.210783+0000","last_peered":"2024-09-20T22:47:33.210783+0000","last_clean":"2024-09-20T22:47:33.210783+0000","last_became_active":"2024-09-20T22:43:39.130391+0000","last_became_peered":"2024-09-20T22:43:39.130391+0000","last_unstale":"2024-09-20T22:47:33.210783+0000","last_undegraded":"2024-09-20T22:47:33.210783+0000","last_fullsized":"2024-09-20T22:47:33.210783+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":5,"last_scrub":"146'43","last_scrub_stamp":"2024-09-20T22:43:46.106059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:43:46.106059+0000","objects_scrubbed":0,"log_size":110,"log_dups_size":0,"ondisk_log_size":110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0067757080000000001,"stat_sum":{"num_bytes":6495724,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":47,"num_read_kb":65519,"num_write":27,"num_write_kb":14407,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"89","length":"1"},{"start":"90","length":"1"}]},{"pgid":"4.13","version":"74'11","reported_seq":500,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-20T22:47:33.535744+0000","last_change":"2024-09-20T22:45:37.940866+0000","last_active":"2024-09-20T22:47:33.535744+0000","last_peered":"2024-09-20T22:47:33.535744+0000","last_clean":"2024-09-20T22:47:33.535744+0000","last_became_active":"2024-09-20T22:43:57.416358+0000","last_became_peered":"2024-09-20T22:43:57.416358+0000","last_unstale":"2024-09-20T22:47:33.535744+0000","last_undegraded":"2024-09-20T22:47:33.535744+0000","last_fullsized":"2024-09-20T22:47:33.535744+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-09-20T22:45:37.940753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:45:37.940753+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:47:36.787162+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":451,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-20T22:47:33.413915+0000","last_change":"2024-09-20T22:46:04.391377+0000","last_active":"2024-09-20T22:47:33.413915+0000","last_peered":"2024-09-20T22:47:33.413915+0000","last_clean":"2024-09-20T22:47:33.413915+0000","last_became_active":"2024-09-20T22:43:57.422556+0000","last_became_peered":"2024-09-20T22:43:57.422556+0000","last_unstale":"2024-09-20T22:47:33.413915+0000","last_undegraded":"2024-09-20T22:47:33.413915+0000","last_fullsized":"2024-09-20T22:47:33.413915+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:46:04.391306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:46:04.391306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:47:37.373465+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":632,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-20T22:47:33.238929+0000","last_change":"2024-09-20T22:46:25.881993+0000","last_active":"2024-09-20T22:47:33.238929+0000","last_peered":"2024-09-20T22:47:33.238929+0000","last_clean":"2024-09-20T22:47:33.238929+0000","last_became_active":"2024-09-20T22:41:42.181736+0000","last_became_peered":"2024-09-20T22:41:42.181736+0000","last_unstale":"2024-09-20T22:47:33.238929+0000","last_undegraded":"2024-09-20T22:47:33.238929+0000","last_fullsized":"2024-09-20T22:47:33.238929+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:46:25.881921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:46:25.881921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:47:35.703311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":626,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-20T22:47:33.202847+0000","last_change":"2024-09-20T22:45:40.223432+0000","last_active":"2024-09-20T22:47:33.202847+0000","last_peered":"2024-09-20T22:47:33.202847+0000","last_clean":"2024-09-20T22:47:33.202847+0000","last_became_active":"2024-09-20T22:45:40.222560+0000","last_became_peered":"2024-09-20T22:45:40.222560+0000","last_unstale":"2024-09-20T22:47:33.202847+0000","last_undegraded":"2024-09-20T22:47:33.202847+0000","last_fullsized":"2024-09-20T22:47:33.202847+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:53.201465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:43:53.201465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:47:35.121024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"135'39","reported_seq":1174,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-20T22:47:33.414043+0000","last_change":"2024-09-20T22:47:32.375573+0000","last_active":"2024-09-20T22:47:33.414043+0000","last_peered":"2024-09-20T22:47:33.414043+0000","last_clean":"2024-09-20T22:47:33.414043+0000","last_became_active":"2024-09-20T22:43:39.130210+0000","last_became_peered":"2024-09-20T22:43:39.130210+0000","last_unstale":"2024-09-20T22:47:33.414043+0000","last_undegraded":"2024-09-20T22:47:33.414043+0000","last_fullsized":"2024-09-20T22:47:33.414043+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":5,"last_scrub":"135'39","last_scrub_stamp":"2024-09-20T22:45:35.341125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:45:35.341125+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00066917400000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"6b","length":"1"},{"start":"74","length":"1"},{"start":"89","length":"1"},{"start":"90","length":"1"}]},{"pgid":"4.12","version":"74'9","reported_seq":652,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-20T22:47:33.239054+0000","last_change":"2024-09-20T22:44:54.989495+0000","last_active":"2024-09-20T22:47:33.239054+0000","last_peered":"2024-09-20T22:47:33.239054+0000","last_clean":"2024-09-20T22:47:33.239054+0000","last_became_active":"2024-09-20T22:41:40.164796+0000","last_became_peered":"2024-09-20T22:41:40.164796+0000","last_unstale":"2024-09-20T22:47:33.239054+0000","last_undegraded":"2024-09-20T22:47:33.239054+0000","last_fullsized":"2024-09-20T22:47:33.239054+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-09-20T22:44:54.989403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:44:54.989403+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:47:34.698489+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":656,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-20T22:47:33.414166+0000","last_change":"2024-09-20T22:45:40.219795+0000","last_active":"2024-09-20T22:47:33.414166+0000","last_peered":"2024-09-20T22:47:33.414166+0000","last_clean":"2024-09-20T22:47:33.414166+0000","last_became_active":"2024-09-20T22:45:40.219482+0000","last_became_peered":"2024-09-20T22:45:40.219482+0000","last_unstale":"2024-09-20T22:47:33.414166+0000","last_undegraded":"2024-09-20T22:47:33.414166+0000","last_fullsized":"2024-09-20T22:47:33.414166+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:45:38.371521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:31.138067+0000","last_clean_scrub_stamp":"2024-09-20T22:45:38.371521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:47:37.373230+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":630,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-20T22:47:33.203537+0000","last_change":"2024-09-20T22:45:40.229442+0000","last_active":"2024-09-20T22:47:33.203537+0000","last_peered":"2024-09-20T22:47:33.203537+0000","last_clean":"2024-09-20T22:47:33.203537+0000","last_became_active":"2024-09-20T22:45:40.228813+0000","last_became_peered":"2024-09-20T22:45:40.228813+0000","last_unstale":"2024-09-20T22:47:33.203537+0000","last_undegraded":"2024-09-20T22:47:33.203537+0000","last_fullsized":"2024-09-20T22:47:33.203537+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:44:15.242881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:44:15.242881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:47:35.121189+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":236,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-20T22:47:33.206651+0000","last_change":"2024-09-20T22:45:40.224832+0000","last_active":"2024-09-20T22:47:33.206651+0000","last_peered":"2024-09-20T22:47:33.206651+0000","last_clean":"2024-09-20T22:47:33.206651+0000","last_became_active":"2024-09-20T22:45:40.224506+0000","last_became_peered":"2024-09-20T22:45:40.224506+0000","last_unstale":"2024-09-20T22:47:33.206651+0000","last_undegraded":"2024-09-20T22:47:33.206651+0000","last_fullsized":"2024-09-20T22:47:33.206651+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:44:39.660039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:44:39.660039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:47:34.304437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.12","version":"372'351","reported_seq":1574,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-20T22:47:33.537926+0000","last_change":"2024-09-20T22:47:32.809712+0000","last_active":"2024-09-20T22:47:33.537926+0000","last_peered":"2024-09-20T22:47:33.537926+0000","last_clean":"2024-09-20T22:47:33.537926+0000","last_became_active":"2024-09-20T22:45:43.172023+0000","last_became_peered":"2024-09-20T22:45:43.172023+0000","last_unstale":"2024-09-20T22:47:33.537926+0000","last_undegraded":"2024-09-20T22:47:33.537926+0000","last_fullsized":"2024-09-20T22:47:33.537926+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":5,"last_scrub":"235'221","last_scrub_stamp":"2024-09-20T22:45:15.702470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:45:15.702470+0000","objects_scrubbed":5,"log_size":351,"log_dups_size":0,"ondisk_log_size":351,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.007735904,"stat_sum":{"num_bytes":16803451,"num_objects":9,"num_object_clones":6,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":137,"num_read_kb":130360,"num_write":109,"num_write_kb":37094,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":12258040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"6b","length":"1"},{"start":"74","length":"1"},{"start":"89","length":"1"},{"start":"90","length":"1"}]},{"pgid":"4.11","version":"74'11","reported_seq":446,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-20T22:47:33.206905+0000","last_change":"2024-09-20T22:46:31.254470+0000","last_active":"2024-09-20T22:47:33.206905+0000","last_peered":"2024-09-20T22:47:33.206905+0000","last_clean":"2024-09-20T22:47:33.206905+0000","last_became_active":"2024-09-20T22:44:09.170785+0000","last_became_peered":"2024-09-20T22:44:09.170785+0000","last_unstale":"2024-09-20T22:47:33.206905+0000","last_undegraded":"2024-09-20T22:47:33.206905+0000","last_fullsized":"2024-09-20T22:47:33.206905+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-09-20T22:46:31.254376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:46:31.254376+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:00.231236+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":641,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-20T22:47:33.208431+0000","last_change":"2024-09-20T22:45:24.765509+0000","last_active":"2024-09-20T22:47:33.208431+0000","last_peered":"2024-09-20T22:47:33.208431+0000","last_clean":"2024-09-20T22:47:33.208431+0000","last_became_active":"2024-09-20T22:41:38.158701+0000","last_became_peered":"2024-09-20T22:41:38.158701+0000","last_unstale":"2024-09-20T22:47:33.208431+0000","last_undegraded":"2024-09-20T22:47:33.208431+0000","last_fullsized":"2024-09-20T22:47:33.208431+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:45:24.765414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:45:24.765414+0000","last_clean_scrub_stamp":"2024-09-20T22:45:24.765414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:47:34.304509+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":645,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-20T22:47:33.412930+0000","last_change":"2024-09-20T22:46:29.466430+0000","last_active":"2024-09-20T22:47:33.412930+0000","last_peered":"2024-09-20T22:47:33.412930+0000","last_clean":"2024-09-20T22:47:33.412930+0000","last_became_active":"2024-09-20T22:45:40.210241+0000","last_became_peered":"2024-09-20T22:45:40.210241+0000","last_unstale":"2024-09-20T22:47:33.412930+0000","last_undegraded":"2024-09-20T22:47:33.412930+0000","last_fullsized":"2024-09-20T22:47:33.412930+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:46:29.466361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:50.174861+0000","last_clean_scrub_stamp":"2024-09-20T22:46:29.466361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:47:35.399992+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":626,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-20T22:47:33.203736+0000","last_change":"2024-09-20T22:45:40.218749+0000","last_active":"2024-09-20T22:47:33.203736+0000","last_peered":"2024-09-20T22:47:33.203736+0000","last_clean":"2024-09-20T22:47:33.203736+0000","last_became_active":"2024-09-20T22:45:40.218262+0000","last_became_peered":"2024-09-20T22:45:40.218262+0000","last_unstale":"2024-09-20T22:47:33.203736+0000","last_undegraded":"2024-09-20T22:47:33.203736+0000","last_fullsized":"2024-09-20T22:47:33.203736+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:45:00.006909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:45:00.006909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:47:35.121286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"373'72","reported_seq":931,"reported_epoch":373,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T22:47:33.240935+0000","last_change":"2024-09-20T22:47:33.240935+0000","last_active":"2024-09-20T22:47:33.240935+0000","last_peered":"2024-09-20T22:47:33.240935+0000","last_clean":"2024-09-20T22:47:33.240935+0000","last_became_active":"2024-09-20T22:45:40.223628+0000","last_became_peered":"2024-09-20T22:45:40.223628+0000","last_unstale":"2024-09-20T22:47:33.240935+0000","last_undegraded":"2024-09-20T22:47:33.240935+0000","last_fullsized":"2024-09-20T22:47:33.240935+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":5,"last_scrub":"255'21","last_scrub_stamp":"2024-09-20T22:45:35.767459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:45:35.767459+0000","objects_scrubbed":1,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":1,"snaptrim_duration":0.96048160500000002,"stat_sum":{"num_bytes":5059672,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":32,"num_read_kb":31660,"num_write":19,"num_write_kb":6137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":2020747,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"89","length":"1"},{"start":"90","length":"1"}]},{"pgid":"4.10","version":"74'4","reported_seq":634,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-20T22:47:33.203520+0000","last_change":"2024-09-20T22:44:17.320732+0000","last_active":"2024-09-20T22:47:33.203520+0000","last_peered":"2024-09-20T22:47:33.203520+0000","last_clean":"2024-09-20T22:47:33.203520+0000","last_became_active":"2024-09-20T22:41:40.166123+0000","last_became_peered":"2024-09-20T22:41:40.166123+0000","last_unstale":"2024-09-20T22:47:33.203520+0000","last_undegraded":"2024-09-20T22:47:33.203520+0000","last_fullsized":"2024-09-20T22:47:33.203520+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'4","last_scrub_stamp":"2024-09-20T22:44:17.320608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:44:17.320608+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:47:35.121136+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":236,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-20T22:47:33.208525+0000","last_change":"2024-09-20T22:45:40.209386+0000","last_active":"2024-09-20T22:47:33.208525+0000","last_peered":"2024-09-20T22:47:33.208525+0000","last_clean":"2024-09-20T22:47:33.208525+0000","last_became_active":"2024-09-20T22:45:40.209090+0000","last_became_peered":"2024-09-20T22:45:40.209090+0000","last_unstale":"2024-09-20T22:47:33.208525+0000","last_undegraded":"2024-09-20T22:47:33.208525+0000","last_fullsized":"2024-09-20T22:47:33.208525+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:44:58.632703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:44:58.632703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:47:34.304495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":644,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-20T22:47:33.537024+0000","last_change":"2024-09-20T22:45:40.205509+0000","last_active":"2024-09-20T22:47:33.537024+0000","last_peered":"2024-09-20T22:47:33.537024+0000","last_clean":"2024-09-20T22:47:33.537024+0000","last_became_active":"2024-09-20T22:45:40.205010+0000","last_became_peered":"2024-09-20T22:45:40.205010+0000","last_unstale":"2024-09-20T22:47:33.537024+0000","last_undegraded":"2024-09-20T22:47:33.537024+0000","last_fullsized":"2024-09-20T22:47:33.537024+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:45:00.680307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:45:00.680307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:47:36.786813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":635,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-20T22:47:33.413289+0000","last_change":"2024-09-20T22:45:19.352251+0000","last_active":"2024-09-20T22:47:33.413289+0000","last_peered":"2024-09-20T22:47:33.413289+0000","last_clean":"2024-09-20T22:47:33.413289+0000","last_became_active":"2024-09-20T22:43:57.424654+0000","last_became_peered":"2024-09-20T22:43:57.424654+0000","last_unstale":"2024-09-20T22:47:33.413289+0000","last_undegraded":"2024-09-20T22:47:33.413289+0000","last_fullsized":"2024-09-20T22:47:33.413289+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:45:19.352162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:10.233487+0000","last_clean_scrub_stamp":"2024-09-20T22:45:19.352162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:47:37.373254+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":634,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-20T22:47:33.239455+0000","last_change":"2024-09-20T22:47:19.596358+0000","last_active":"2024-09-20T22:47:33.239455+0000","last_peered":"2024-09-20T22:47:33.239455+0000","last_clean":"2024-09-20T22:47:33.239455+0000","last_became_active":"2024-09-20T22:43:57.430933+0000","last_became_peered":"2024-09-20T22:43:57.430933+0000","last_unstale":"2024-09-20T22:47:33.239455+0000","last_undegraded":"2024-09-20T22:47:33.239455+0000","last_fullsized":"2024-09-20T22:47:33.239455+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:47:19.596232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:47:19.596232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:41.558738+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":652,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-20T22:47:33.203298+0000","last_change":"2024-09-20T22:45:40.221677+0000","last_active":"2024-09-20T22:47:33.203298+0000","last_peered":"2024-09-20T22:47:33.203298+0000","last_clean":"2024-09-20T22:47:33.203298+0000","last_became_active":"2024-09-20T22:45:40.221329+0000","last_became_peered":"2024-09-20T22:45:40.221329+0000","last_unstale":"2024-09-20T22:47:33.203298+0000","last_undegraded":"2024-09-20T22:47:33.203298+0000","last_fullsized":"2024-09-20T22:47:33.203298+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:44:10.267152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:44:10.267152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:47:35.121121+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"74'11","reported_seq":658,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-20T22:47:33.537489+0000","last_change":"2024-09-20T22:45:14.702296+0000","last_active":"2024-09-20T22:47:33.537489+0000","last_peered":"2024-09-20T22:47:33.537489+0000","last_clean":"2024-09-20T22:47:33.537489+0000","last_became_active":"2024-09-20T22:41:40.171256+0000","last_became_peered":"2024-09-20T22:41:40.171256+0000","last_unstale":"2024-09-20T22:47:33.537489+0000","last_undegraded":"2024-09-20T22:47:33.537489+0000","last_fullsized":"2024-09-20T22:47:33.537489+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-09-20T22:45:14.702198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:45:14.702198+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:47:36.786934+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":244,"reported_epoch":373,"state":"active+clean","last_fresh":"2024-09-20T22:47:33.414112+0000","last_change":"2024-09-20T22:46:20.485525+0000","last_active":"2024-09-20T22:47:33.414112+0000","last_peered":"2024-09-20T22:47:33.414112+0000","last_clean":"2024-09-20T22:47:33.414112+0000","last_became_active":"2024-09-20T22:45:40.221497+0000","last_became_peered":"2024-09-20T22:45:40.221497+0000","last_unstale":"2024-09-20T22:47:33.414112+0000","last_undegraded":"2024-09-20T22:47:33.414112+0000","last_fullsized":"2024-09-20T22:47:33.414112+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:46:20.485453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:51.667341+0000","last_clean_scrub_stamp":"2024-09-20T22:46:20.485453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:47:44.676326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":26,"stat_sum":{"num_bytes":288492281,"num_objects":136,"num_object_clones":89,"num_object_copies":408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":136,"num_whiteouts":6,"num_read":2550,"num_read_kb":2641867,"num_write":1723,"num_write_kb":653853,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":152,"num_bytes_recovered":76941931,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":755855360,"data_stored":1015314514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5252,"ondisk_log_size":5252,"up":78,"acting":78,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1261,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74,"ondisk_log_size":74,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":683,"num_read_kb":440,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":159,"num_bytes_recovered":3278,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":524288,"data_stored":14054,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":428,"ondisk_log_size":428,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":5597,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":348,"num_read_kb":343,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":384,"ondisk_log_size":384,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070539,"num_pgs":87,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":138152,"kb_used_data":106960,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93607000,"statfs":{"total":95995035648,"available":95853568000,"internally_reserved":0,"allocated":109527040,"data_stored":145209668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430289,"num_pgs":78,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":169932,"kb_used_data":138736,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93575220,"statfs":{"total":95995035648,"available":95821025280,"internally_reserved":0,"allocated":142065664,"data_stored":190827357,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691926,"num_pgs":88,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":147048,"kb_used_data":103372,"kb_used_omap":1,"kb_used_meta":43646,"kb_avail":93598104,"statfs":{"total":95995035648,"available":95844458496,"internally_reserved":0,"allocated":105852928,"data_stored":148554561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":44693957},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":0,"seq":133143986267,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182497,"num_pgs":89,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":147988,"kb_used_data":120964,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93597164,"statfs":{"total":95995035648,"available":95843495936,"internally_reserved":0,"allocated":123867136,"data_stored":161082842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411430,"num_pgs":77,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":130804,"kb_used_data":103776,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93614348,"statfs":{"total":95995035648,"available":95861092352,"internally_reserved":0,"allocated":106266624,"data_stored":136270475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574955,"num_pgs":85,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":141284,"kb_used_data":114256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93603868,"statfs":{"total":95995035648,"available":95850360832,"internally_reserved":0,"allocated":116998144,"data_stored":155781282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":34359738480,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":2668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":68685824,"data_stored":96463859,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":112975872,"data_stored":150307052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102424576,"data_stored":129726407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":119668736,"data_stored":156691949,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11284480,"data_stored":11212109,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":102891520,"data_stored":146559555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":131080192,"data_stored":180867831,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":106844160,"data_stored":143485752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-20T22:47:39.842 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph pg dump --format=json 2024-09-20T22:47:39.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:39.896 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:39.897 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:47:39.924 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd set noscrub 2024-09-20T22:47:39.998 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:47:40.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:40.022 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:40.023 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:47:40.123 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:47:40.144 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:40.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:40.145 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:47:40.231 INFO:tasks.rados.rados.0.smithi084.stdout:1449: done (3 left) 2024-09-20T22:47:40.232 INFO:tasks.rados.rados.0.smithi084.stdout:1450: done (2 left) 2024-09-20T22:47:40.232 INFO:tasks.rados.rados.0.smithi084.stdout:1451: done (1 left) 2024-09-20T22:47:40.232 INFO:tasks.rados.rados.0.smithi084.stdout:1452: done (0 left) 2024-09-20T22:47:40.232 INFO:tasks.rados.rados.0.smithi084.stdout:1453: snap_create 2024-09-20T22:47:40.246 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:47:40.267 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:40.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:40.268 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:47:40.369 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:47:40.393 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:40.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:40.394 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:47:40.434 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:47:40.444 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:47:40.473 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:47:40.495 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:47:40.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:40 smithi154 ceph-mon[33264]: purged_snaps scrub starts 2024-09-20T22:47:40.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:40 smithi154 ceph-mon[33264]: purged_snaps scrub ok 2024-09-20T22:47:40.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:40 smithi154 ceph-mon[33264]: from='client.15735 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T22:47:40.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:40 smithi154 ceph-mon[33264]: 4.1f scrub starts 2024-09-20T22:47:40.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:40 smithi154 ceph-mon[33264]: 4.1f scrub ok 2024-09-20T22:47:40.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:40 smithi154 ceph-mon[33264]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:47:40.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:40 smithi154 ceph-mon[33264]: purged_snaps scrub starts 2024-09-20T22:47:40.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:40 smithi154 ceph-mon[33264]: purged_snaps scrub ok 2024-09-20T22:47:40.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:40 smithi154 ceph-mon[33264]: 4.6 scrub starts 2024-09-20T22:47:40.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:40 smithi154 ceph-mon[33264]: 4.6 scrub ok 2024-09-20T22:47:40.518 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:40.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:40.519 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:47:40.561 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:40 smithi084 ceph-mon[40024]: purged_snaps scrub starts 2024-09-20T22:47:40.561 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:40 smithi084 ceph-mon[40024]: purged_snaps scrub ok 2024-09-20T22:47:40.561 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:40 smithi084 ceph-mon[40024]: from='client.15735 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T22:47:40.561 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:40 smithi084 ceph-mon[40024]: 4.1f scrub starts 2024-09-20T22:47:40.561 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:40 smithi084 ceph-mon[40024]: 4.1f scrub ok 2024-09-20T22:47:40.562 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:40 smithi084 ceph-mon[40024]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:47:40.562 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:40 smithi084 ceph-mon[40024]: purged_snaps scrub starts 2024-09-20T22:47:40.562 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:40 smithi084 ceph-mon[40024]: purged_snaps scrub ok 2024-09-20T22:47:40.562 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:40 smithi084 ceph-mon[40024]: 4.6 scrub starts 2024-09-20T22:47:40.562 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:40 smithi084 ceph-mon[40024]: 4.6 scrub ok 2024-09-20T22:47:40.562 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:40 smithi084 ceph-mon[29774]: purged_snaps scrub starts 2024-09-20T22:47:40.563 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:40 smithi084 ceph-mon[29774]: purged_snaps scrub ok 2024-09-20T22:47:40.563 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:40 smithi084 ceph-mon[29774]: from='client.15735 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T22:47:40.563 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:40 smithi084 ceph-mon[29774]: 4.1f scrub starts 2024-09-20T22:47:40.563 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:40 smithi084 ceph-mon[29774]: 4.1f scrub ok 2024-09-20T22:47:40.563 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:40 smithi084 ceph-mon[29774]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:47:40.563 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:40 smithi084 ceph-mon[29774]: purged_snaps scrub starts 2024-09-20T22:47:40.564 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:40 smithi084 ceph-mon[29774]: purged_snaps scrub ok 2024-09-20T22:47:40.564 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:40 smithi084 ceph-mon[29774]: 4.6 scrub starts 2024-09-20T22:47:40.564 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:40 smithi084 ceph-mon[29774]: 4.6 scrub ok 2024-09-20T22:47:40.619 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:47:40.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:40.643 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:40.643 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:47:40.744 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:47:40.767 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:40.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:40.768 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:47:40.869 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:47:40.892 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:40.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:40.893 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:47:40.994 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:47:41.017 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:41.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:41.018 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:47:41.119 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:47:41.141 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:41.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:41.142 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:47:41.232 INFO:tasks.rados.rados.0.smithi084.stdout:1453: done (0 left) 2024-09-20T22:47:41.233 INFO:tasks.rados.rados.0.smithi084.stdout:1454: delete oid 40 current snap is 150 2024-09-20T22:47:41.239 INFO:tasks.rados.rados.0.smithi084.stdout:1454: done (0 left) 2024-09-20T22:47:41.239 INFO:tasks.rados.rados.0.smithi084.stdout:1455: snap_remove snap 131 2024-09-20T22:47:41.242 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:47:41.268 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:41.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:41.269 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:47:41.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:41 smithi084 ceph-mon[29774]: pgmap v471: 168 pgs: 6 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 157 active+clean; 265 MiB data, 820 MiB used, 536 GiB / 536 GiB avail; 10 MiB/s rd, 591 KiB/s wr, 9 op/s 2024-09-20T22:47:41.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:41 smithi084 ceph-mon[29774]: 3.7 scrub starts 2024-09-20T22:47:41.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:41 smithi084 ceph-mon[29774]: 3.7 scrub ok 2024-09-20T22:47:41.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:41 smithi084 ceph-mon[29774]: osdmap e380: 8 total, 8 up, 6 in 2024-09-20T22:47:41.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:41 smithi084 ceph-mon[29774]: 5.1a deep-scrub starts 2024-09-20T22:47:41.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:41 smithi084 ceph-mon[29774]: 5.1a deep-scrub ok 2024-09-20T22:47:41.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:41 smithi084 ceph-mon[29774]: 2.1 deep-scrub starts 2024-09-20T22:47:41.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:41 smithi084 ceph-mon[29774]: 2.1 deep-scrub ok 2024-09-20T22:47:41.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:41 smithi084 ceph-mon[40024]: pgmap v471: 168 pgs: 6 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 157 active+clean; 265 MiB data, 820 MiB used, 536 GiB / 536 GiB avail; 10 MiB/s rd, 591 KiB/s wr, 9 op/s 2024-09-20T22:47:41.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:41 smithi084 ceph-mon[40024]: 3.7 scrub starts 2024-09-20T22:47:41.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:41 smithi084 ceph-mon[40024]: 3.7 scrub ok 2024-09-20T22:47:41.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:41 smithi084 ceph-mon[40024]: osdmap e380: 8 total, 8 up, 6 in 2024-09-20T22:47:41.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:41 smithi084 ceph-mon[40024]: 5.1a deep-scrub starts 2024-09-20T22:47:41.367 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:41 smithi084 ceph-mon[40024]: 5.1a deep-scrub ok 2024-09-20T22:47:41.367 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:41 smithi084 ceph-mon[40024]: 2.1 deep-scrub starts 2024-09-20T22:47:41.367 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:41 smithi084 ceph-mon[40024]: 2.1 deep-scrub ok 2024-09-20T22:47:41.370 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:47:41.410 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:41.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:41.411 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:47:41.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:41 smithi154 ceph-mon[33264]: pgmap v471: 168 pgs: 6 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 157 active+clean; 265 MiB data, 820 MiB used, 536 GiB / 536 GiB avail; 10 MiB/s rd, 591 KiB/s wr, 9 op/s 2024-09-20T22:47:41.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:41 smithi154 ceph-mon[33264]: 3.7 scrub starts 2024-09-20T22:47:41.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:41 smithi154 ceph-mon[33264]: 3.7 scrub ok 2024-09-20T22:47:41.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:41 smithi154 ceph-mon[33264]: osdmap e380: 8 total, 8 up, 6 in 2024-09-20T22:47:41.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:41 smithi154 ceph-mon[33264]: 5.1a deep-scrub starts 2024-09-20T22:47:41.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:41 smithi154 ceph-mon[33264]: 5.1a deep-scrub ok 2024-09-20T22:47:41.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:41 smithi154 ceph-mon[33264]: 2.1 deep-scrub starts 2024-09-20T22:47:41.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:41 smithi154 ceph-mon[33264]: 2.1 deep-scrub ok 2024-09-20T22:47:41.512 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:47:41.533 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:41.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:41.534 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:47:41.635 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:47:41.656 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:41.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:41.657 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:47:41.751 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-20T22:47:41.754 INFO:teuthology.orchestra.run.smithi084.stderr:dumped all 2024-09-20T22:47:41.759 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:47:41.785 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:41.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:41.786 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:47:41.820 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-20T22:47:41.820 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2024-09-20T22:47:41.820 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2024-09-20T22:47:41.820 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-20T22:47:41.887 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:47:41.908 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:41.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:41.909 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:47:42.010 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:47:42.031 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:42.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:42.032 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:47:42.133 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:47:42.154 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:42.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:42.155 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:47:42.238 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is set 2024-09-20T22:47:42.243 INFO:tasks.rados.rados.0.smithi084.stdout:1455: done (0 left) 2024-09-20T22:47:42.244 INFO:tasks.rados.rados.0.smithi084.stdout:1456: snap_create 2024-09-20T22:47:42.256 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:47:42.280 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:42.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:42.281 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:47:42.334 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-20T22:47:42.382 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:47:42.407 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:42.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:42.408 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:47:42.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:42 smithi154 ceph-mon[33264]: osdmap e381: 8 total, 8 up, 6 in 2024-09-20T22:47:42.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:42 smithi154 ceph-mon[33264]: 6.a scrub starts 2024-09-20T22:47:42.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:42 smithi154 ceph-mon[33264]: 6.a scrub ok 2024-09-20T22:47:42.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:42 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/2881538832' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-20T22:47:42.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:42 smithi154 ceph-mon[33264]: 7.e scrub starts 2024-09-20T22:47:42.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:42 smithi154 ceph-mon[33264]: 7.e scrub ok 2024-09-20T22:47:42.508 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:42 smithi084 ceph-mon[29774]: osdmap e381: 8 total, 8 up, 6 in 2024-09-20T22:47:42.508 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:42 smithi084 ceph-mon[29774]: 6.a scrub starts 2024-09-20T22:47:42.508 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:42 smithi084 ceph-mon[29774]: 6.a scrub ok 2024-09-20T22:47:42.508 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:42 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/2881538832' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-20T22:47:42.508 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:42 smithi084 ceph-mon[29774]: 7.e scrub starts 2024-09-20T22:47:42.508 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:42 smithi084 ceph-mon[29774]: 7.e scrub ok 2024-09-20T22:47:42.509 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:42 smithi084 ceph-mon[40024]: osdmap e381: 8 total, 8 up, 6 in 2024-09-20T22:47:42.509 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:42 smithi084 ceph-mon[40024]: 6.a scrub starts 2024-09-20T22:47:42.509 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:42 smithi084 ceph-mon[40024]: 6.a scrub ok 2024-09-20T22:47:42.509 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:42 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/2881538832' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-20T22:47:42.509 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:42 smithi084 ceph-mon[40024]: 7.e scrub starts 2024-09-20T22:47:42.509 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:42 smithi084 ceph-mon[40024]: 7.e scrub ok 2024-09-20T22:47:42.509 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:47:42.531 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:42.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:42.532 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:47:42.546 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.0 dump_blocked_ops 2024-09-20T22:47:42.633 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:47:42.658 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:42.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:42.659 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:47:42.760 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:47:42.785 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:42.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:42.786 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:47:42.887 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:47:42.912 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:42.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:42.914 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:47:43.014 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:47:43.039 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:43.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:43.040 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:47:43.141 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:47:43.165 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:43.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:43.166 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:47:43.254 INFO:tasks.rados.rados.0.smithi084.stdout:1456: done (0 left) 2024-09-20T22:47:43.254 INFO:tasks.rados.rados.0.smithi084.stdout:1457: rollback oid 39 current snap is 151 2024-09-20T22:47:43.254 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 39 to 134 2024-09-20T22:47:43.254 INFO:tasks.rados.rados.0.smithi084.stdout:1458: copy_from oid 28 from oid 41 current snap is 151 2024-09-20T22:47:43.254 INFO:tasks.rados.rados.0.smithi084.stdout:1459: write oid 31 current snap is 151 2024-09-20T22:47:43.255 INFO:tasks.rados.rados.0.smithi084.stdout:1459: seq_num 518 ranges {706026=490986,1828849=790505} 2024-09-20T22:47:43.261 INFO:tasks.rados.rados.0.smithi084.stdout:1459: writing smithi08477203-31 from 706026 to 1197012 tid 1 2024-09-20T22:47:43.266 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:47:43.269 INFO:tasks.rados.rados.0.smithi084.stdout:1459: writing smithi08477203-31 from 1828849 to 2619354 tid 2 2024-09-20T22:47:43.269 INFO:tasks.rados.rados.0.smithi084.stdout:1460: read oid 38 snap -1 2024-09-20T22:47:43.269 INFO:tasks.rados.rados.0.smithi084.stdout:1460: expect (ObjNum 489 snap 137 seq_num 489) 2024-09-20T22:47:43.269 INFO:tasks.rados.rados.0.smithi084.stdout:1461: write oid 29 current snap is 151 2024-09-20T22:47:43.269 INFO:tasks.rados.rados.0.smithi084.stdout:1461: seq_num 519 ranges {722833=566235,1887331=650516} 2024-09-20T22:47:43.276 INFO:tasks.rados.rados.0.smithi084.stdout:1461: writing smithi08477203-29 from 722833 to 1289068 tid 1 2024-09-20T22:47:43.284 INFO:tasks.rados.rados.0.smithi084.stdout:1461: writing smithi08477203-29 from 1887331 to 2537847 tid 2 2024-09-20T22:47:43.284 INFO:tasks.rados.rados.0.smithi084.stdout:1462: copy_from oid 32 from oid 47 current snap is 151 2024-09-20T22:47:43.284 INFO:tasks.rados.rados.0.smithi084.stdout:1457: finishing rollback tid 0 to smithi08477203-39 2024-09-20T22:47:43.284 INFO:tasks.rados.rados.0.smithi084.stdout:1457: finishing rollback tid 1 to smithi08477203-39 2024-09-20T22:47:43.284 INFO:tasks.rados.rados.0.smithi084.stdout:1457: finishing rollback tid 2 to smithi08477203-39 2024-09-20T22:47:43.284 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 39 v 525 (ObjNum 476 snap 133 seq_num 476) dirty exists 2024-09-20T22:47:43.284 INFO:tasks.rados.rados.0.smithi084.stdout:1459: finishing write tid 1 to smithi08477203-31 2024-09-20T22:47:43.284 INFO:tasks.rados.rados.0.smithi084.stdout:1458: finishing copy_from racing read to smithi08477203-28 2024-09-20T22:47:43.284 INFO:tasks.rados.rados.0.smithi084.stdout:1457: done (5 left) 2024-09-20T22:47:43.284 INFO:tasks.rados.rados.0.smithi084.stdout:1463: write oid 12 current snap is 151 2024-09-20T22:47:43.284 INFO:tasks.rados.rados.0.smithi084.stdout:1463: seq_num 520 ranges {739640=641484,1829460=794174} 2024-09-20T22:47:43.291 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:43.292 INFO:tasks.rados.rados.0.smithi084.stdout:1463: writing smithi08477203-12 from 739640 to 1381124 tid 1 2024-09-20T22:47:43.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:43.292 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:47:43.299 INFO:tasks.rados.rados.0.smithi084.stdout:1463: writing smithi08477203-12 from 1829460 to 2623634 tid 2 2024-09-20T22:47:43.299 INFO:tasks.rados.rados.0.smithi084.stdout:1464: snap_remove snap 149 2024-09-20T22:47:43.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:43 smithi084 ceph-mon[29774]: 6.16 scrub starts 2024-09-20T22:47:43.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:43 smithi084 ceph-mon[29774]: 6.16 scrub ok 2024-09-20T22:47:43.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:43 smithi084 ceph-mon[29774]: pgmap v474: 168 pgs: 6 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 157 active+clean; 265 MiB data, 820 MiB used, 536 GiB / 536 GiB avail; 10 MiB/s rd, 591 KiB/s wr, 9 op/s 2024-09-20T22:47:43.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:43 smithi084 ceph-mon[29774]: from='client.15753 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T22:47:43.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:43 smithi084 ceph-mon[29774]: 7.12 scrub starts 2024-09-20T22:47:43.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:43 smithi084 ceph-mon[29774]: 7.12 scrub ok 2024-09-20T22:47:43.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:43 smithi084 ceph-mon[29774]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:47:43.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:43 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/2881538832' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T22:47:43.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:43 smithi084 ceph-mon[29774]: osdmap e382: 8 total, 8 up, 6 in 2024-09-20T22:47:43.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:43 smithi084 ceph-mon[29774]: purged_snaps scrub starts 2024-09-20T22:47:43.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:43 smithi084 ceph-mon[29774]: purged_snaps scrub ok 2024-09-20T22:47:43.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:43 smithi084 ceph-mon[40024]: 6.16 scrub starts 2024-09-20T22:47:43.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:43 smithi084 ceph-mon[40024]: 6.16 scrub ok 2024-09-20T22:47:43.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:43 smithi084 ceph-mon[40024]: pgmap v474: 168 pgs: 6 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 157 active+clean; 265 MiB data, 820 MiB used, 536 GiB / 536 GiB avail; 10 MiB/s rd, 591 KiB/s wr, 9 op/s 2024-09-20T22:47:43.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:43 smithi084 ceph-mon[40024]: from='client.15753 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T22:47:43.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:43 smithi084 ceph-mon[40024]: 7.12 scrub starts 2024-09-20T22:47:43.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:43 smithi084 ceph-mon[40024]: 7.12 scrub ok 2024-09-20T22:47:43.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:43 smithi084 ceph-mon[40024]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:47:43.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:43 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/2881538832' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T22:47:43.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:43 smithi084 ceph-mon[40024]: osdmap e382: 8 total, 8 up, 6 in 2024-09-20T22:47:43.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:43 smithi084 ceph-mon[40024]: purged_snaps scrub starts 2024-09-20T22:47:43.367 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:43 smithi084 ceph-mon[40024]: purged_snaps scrub ok 2024-09-20T22:47:43.393 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:47:43.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:43.415 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:43.415 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:47:43.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:43 smithi154 ceph-mon[33264]: 6.16 scrub starts 2024-09-20T22:47:43.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:43 smithi154 ceph-mon[33264]: 6.16 scrub ok 2024-09-20T22:47:43.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:43 smithi154 ceph-mon[33264]: pgmap v474: 168 pgs: 6 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 157 active+clean; 265 MiB data, 820 MiB used, 536 GiB / 536 GiB avail; 10 MiB/s rd, 591 KiB/s wr, 9 op/s 2024-09-20T22:47:43.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:43 smithi154 ceph-mon[33264]: from='client.15753 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T22:47:43.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:43 smithi154 ceph-mon[33264]: 7.12 scrub starts 2024-09-20T22:47:43.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:43 smithi154 ceph-mon[33264]: 7.12 scrub ok 2024-09-20T22:47:43.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:43 smithi154 ceph-mon[33264]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:47:43.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:43 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/2881538832' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T22:47:43.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:43 smithi154 ceph-mon[33264]: osdmap e382: 8 total, 8 up, 6 in 2024-09-20T22:47:43.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:43 smithi154 ceph-mon[33264]: purged_snaps scrub starts 2024-09-20T22:47:43.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:43 smithi154 ceph-mon[33264]: purged_snaps scrub ok 2024-09-20T22:47:43.516 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:47:43.538 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:43.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:43.539 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:47:43.545 INFO:teuthology.orchestra.run.smithi084.stdout:{"pg_ready":true,"pg_map":{"version":474,"stamp":"2024-09-20T22:47:41.700971+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":277412359,"num_objects":331,"num_object_clones":85,"num_object_copies":993,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":331,"num_whiteouts":6,"num_read":3641,"num_read_kb":2685326,"num_write":2163,"num_write_kb":656262,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":329,"num_bytes_recovered":77274003,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6261,"ondisk_log_size":6261,"up":504,"acting":504,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":504,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":562470912,"kb_used":839672,"kb_used_data":652508,"kb_used_omap":9,"kb_used_meta":186998,"kb_avail":561631240,"statfs":{"total":575970213888,"available":575110389760,"internally_reserved":0,"allocated":668168192,"data_stored":891984085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9562,"internal_metadata":191486630},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-11676429,"num_objects":-4,"num_object_clones":-4,"num_object_copies":-12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4,"num_whiteouts":0,"num_read":36,"num_read_kb":42652,"num_write":4,"num_write_kb":2364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000724"},"pg_stats":[{"pgid":"3.1f","version":"86'2","reported_seq":316,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-20T22:47:38.221976+0000","last_change":"2024-09-20T22:46:42.206673+0000","last_active":"2024-09-20T22:47:38.221976+0000","last_peered":"2024-09-20T22:47:38.221976+0000","last_clean":"2024-09-20T22:47:38.221976+0000","last_became_active":"2024-09-20T22:45:40.207225+0000","last_became_peered":"2024-09-20T22:45:40.207225+0000","last_unstale":"2024-09-20T22:47:38.221976+0000","last_undegraded":"2024-09-20T22:47:38.221976+0000","last_fullsized":"2024-09-20T22:47:38.221976+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'2","last_scrub_stamp":"2024-09-20T22:46:42.206600+0000","last_deep_scrub":"86'2","last_deep_scrub_stamp":"2024-09-20T22:42:39.664421+0000","last_clean_scrub_stamp":"2024-09-20T22:46:42.206600+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:47:54.236322+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":436,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"74'9","reported_seq":513,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-20T22:47:38.286047+0000","last_change":"2024-09-20T22:46:20.889599+0000","last_active":"2024-09-20T22:47:38.286047+0000","last_peered":"2024-09-20T22:47:38.286047+0000","last_clean":"2024-09-20T22:47:38.286047+0000","last_became_active":"2024-09-20T22:43:57.428786+0000","last_became_peered":"2024-09-20T22:43:57.428786+0000","last_unstale":"2024-09-20T22:47:38.286047+0000","last_undegraded":"2024-09-20T22:47:38.286047+0000","last_fullsized":"2024-09-20T22:47:38.286047+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-09-20T22:46:20.889510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:46:20.889510+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":642,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-20T22:47:38.188957+0000","last_change":"2024-09-20T22:46:22.955529+0000","last_active":"2024-09-20T22:47:38.188957+0000","last_peered":"2024-09-20T22:47:38.188957+0000","last_clean":"2024-09-20T22:47:38.188957+0000","last_became_active":"2024-09-20T22:41:42.182024+0000","last_became_peered":"2024-09-20T22:41:42.182024+0000","last_unstale":"2024-09-20T22:47:38.188957+0000","last_undegraded":"2024-09-20T22:47:38.188957+0000","last_fullsized":"2024-09-20T22:47:38.188957+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:46:22.955460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:46:22.955460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":470,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-20T22:47:38.220699+0000","last_change":"2024-09-20T22:46:45.177175+0000","last_active":"2024-09-20T22:47:38.220699+0000","last_peered":"2024-09-20T22:47:38.220699+0000","last_clean":"2024-09-20T22:47:38.220699+0000","last_became_active":"2024-09-20T22:43:57.429983+0000","last_became_peered":"2024-09-20T22:43:57.429983+0000","last_unstale":"2024-09-20T22:47:38.220699+0000","last_undegraded":"2024-09-20T22:47:38.220699+0000","last_fullsized":"2024-09-20T22:47:38.220699+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:46:45.177105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:46:45.177105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:13.549053+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":638,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-20T22:47:38.507047+0000","last_change":"2024-09-20T22:45:40.053329+0000","last_active":"2024-09-20T22:47:38.507047+0000","last_peered":"2024-09-20T22:47:38.507047+0000","last_clean":"2024-09-20T22:47:38.507047+0000","last_became_active":"2024-09-20T22:41:44.832099+0000","last_became_peered":"2024-09-20T22:41:44.832099+0000","last_unstale":"2024-09-20T22:47:38.507047+0000","last_undegraded":"2024-09-20T22:47:38.507047+0000","last_fullsized":"2024-09-20T22:47:38.507047+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:45:40.053263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:45:40.053263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":642,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-20T22:47:38.508092+0000","last_change":"2024-09-20T22:44:58.994410+0000","last_active":"2024-09-20T22:47:38.508092+0000","last_peered":"2024-09-20T22:47:38.508092+0000","last_clean":"2024-09-20T22:47:38.508092+0000","last_became_active":"2024-09-20T22:41:38.157298+0000","last_became_peered":"2024-09-20T22:41:38.157298+0000","last_unstale":"2024-09-20T22:47:38.508092+0000","last_undegraded":"2024-09-20T22:47:38.508092+0000","last_fullsized":"2024-09-20T22:47:38.508092+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:44:58.994340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:44:58.994340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"74'15","reported_seq":685,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-20T22:47:38.507681+0000","last_change":"2024-09-20T22:45:40.242495+0000","last_active":"2024-09-20T22:47:38.507681+0000","last_peered":"2024-09-20T22:47:38.507681+0000","last_clean":"2024-09-20T22:47:38.507681+0000","last_became_active":"2024-09-20T22:45:40.214879+0000","last_became_peered":"2024-09-20T22:45:40.214879+0000","last_unstale":"2024-09-20T22:47:38.507681+0000","last_undegraded":"2024-09-20T22:47:38.507681+0000","last_fullsized":"2024-09-20T22:47:38.507681+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-09-20T22:44:41.108205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:44:41.108205+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":468,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-20T22:47:38.286204+0000","last_change":"2024-09-20T22:47:19.805833+0000","last_active":"2024-09-20T22:47:38.286204+0000","last_peered":"2024-09-20T22:47:38.286204+0000","last_clean":"2024-09-20T22:47:38.286204+0000","last_became_active":"2024-09-20T22:43:57.423685+0000","last_became_peered":"2024-09-20T22:43:57.423685+0000","last_unstale":"2024-09-20T22:47:38.286204+0000","last_undegraded":"2024-09-20T22:47:38.286204+0000","last_fullsized":"2024-09-20T22:47:38.286204+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:47:19.805701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:47:19.805701+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:41.119667+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.19","version":"377'239","reported_seq":1428,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-20T22:47:38.286378+0000","last_change":"2024-09-20T22:47:37.255143+0000","last_active":"2024-09-20T22:47:38.286378+0000","last_peered":"2024-09-20T22:47:38.286378+0000","last_clean":"2024-09-20T22:47:38.286378+0000","last_became_active":"2024-09-20T22:43:39.130950+0000","last_became_peered":"2024-09-20T22:43:39.130950+0000","last_unstale":"2024-09-20T22:47:38.286378+0000","last_undegraded":"2024-09-20T22:47:38.286378+0000","last_fullsized":"2024-09-20T22:47:38.286378+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":5,"last_scrub":"241'147","last_scrub_stamp":"2024-09-20T22:45:23.839574+0000","last_deep_scrub":"241'147","last_deep_scrub_stamp":"2024-09-20T22:45:23.839574+0000","last_clean_scrub_stamp":"2024-09-20T22:45:23.839574+0000","objects_scrubbed":2,"log_size":239,"log_dups_size":0,"ondisk_log_size":239,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.003997062,"stat_sum":{"num_bytes":13164021,"num_objects":5,"num_object_clones":3,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":87,"num_read_kb":82678,"num_write":76,"num_write_kb":32174,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"6b","length":"1"},{"start":"74","length":"1"},{"start":"7a","length":"1"},{"start":"84","length":"1"}]},{"pgid":"3.1d","version":"0'0","reported_seq":657,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-20T22:47:38.221491+0000","last_change":"2024-09-20T22:46:01.000010+0000","last_active":"2024-09-20T22:47:38.221491+0000","last_peered":"2024-09-20T22:47:38.221491+0000","last_clean":"2024-09-20T22:47:38.221491+0000","last_became_active":"2024-09-20T22:43:57.429854+0000","last_became_peered":"2024-09-20T22:43:57.429854+0000","last_unstale":"2024-09-20T22:47:38.221491+0000","last_undegraded":"2024-09-20T22:47:38.221491+0000","last_fullsized":"2024-09-20T22:47:38.221491+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:46:00.999939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:46:00.999939+0000","last_clean_scrub_stamp":"2024-09-20T22:46:00.999939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"74'9","reported_seq":521,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-20T22:47:38.506958+0000","last_change":"2024-09-20T22:46:45.096042+0000","last_active":"2024-09-20T22:47:38.506958+0000","last_peered":"2024-09-20T22:47:38.506958+0000","last_clean":"2024-09-20T22:47:38.506958+0000","last_became_active":"2024-09-20T22:45:40.220428+0000","last_became_peered":"2024-09-20T22:45:40.220428+0000","last_unstale":"2024-09-20T22:47:38.506958+0000","last_undegraded":"2024-09-20T22:47:38.506958+0000","last_fullsized":"2024-09-20T22:47:38.506958+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-09-20T22:46:45.095957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:46:45.095957+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:06.043333+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":649,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-20T22:47:38.221132+0000","last_change":"2024-09-20T22:46:52.239575+0000","last_active":"2024-09-20T22:47:38.221132+0000","last_peered":"2024-09-20T22:47:38.221132+0000","last_clean":"2024-09-20T22:47:38.221132+0000","last_became_active":"2024-09-20T22:45:40.213185+0000","last_became_peered":"2024-09-20T22:45:40.213185+0000","last_unstale":"2024-09-20T22:47:38.221132+0000","last_undegraded":"2024-09-20T22:47:38.221132+0000","last_fullsized":"2024-09-20T22:47:38.221132+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:46:52.239508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:46:52.239508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:05.847887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":253,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-20T22:47:38.188853+0000","last_change":"2024-09-20T22:46:29.801951+0000","last_active":"2024-09-20T22:47:38.188853+0000","last_peered":"2024-09-20T22:47:38.188853+0000","last_clean":"2024-09-20T22:47:38.188853+0000","last_became_active":"2024-09-20T22:45:40.215834+0000","last_became_peered":"2024-09-20T22:45:40.215834+0000","last_unstale":"2024-09-20T22:47:38.188853+0000","last_undegraded":"2024-09-20T22:47:38.188853+0000","last_fullsized":"2024-09-20T22:47:38.188853+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:46:29.801854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:46:29.801854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:47:49.174382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.18","version":"377'224","reported_seq":1433,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-20T22:47:38.286045+0000","last_change":"2024-09-20T22:47:37.254975+0000","last_active":"2024-09-20T22:47:38.286045+0000","last_peered":"2024-09-20T22:47:38.286045+0000","last_clean":"2024-09-20T22:47:38.286045+0000","last_became_active":"2024-09-20T22:45:43.170008+0000","last_became_peered":"2024-09-20T22:45:43.170008+0000","last_unstale":"2024-09-20T22:47:38.286045+0000","last_undegraded":"2024-09-20T22:47:38.286045+0000","last_fullsized":"2024-09-20T22:47:38.286045+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":5,"last_scrub":"282'128","last_scrub_stamp":"2024-09-20T22:46:01.882247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:46:01.882247+0000","objects_scrubbed":1,"log_size":224,"log_dups_size":0,"ondisk_log_size":224,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0041553450000000004,"stat_sum":{"num_bytes":6396308,"num_objects":6,"num_object_clones":4,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":93,"num_read_kb":94824,"num_write":64,"num_write_kb":23540,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":6792292,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"6b","length":"1"},{"start":"74","length":"1"},{"start":"7a","length":"1"},{"start":"84","length":"1"}]},{"pgid":"3.1c","version":"0'0","reported_seq":657,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-20T22:47:38.221234+0000","last_change":"2024-09-20T22:45:53.934021+0000","last_active":"2024-09-20T22:47:38.221234+0000","last_peered":"2024-09-20T22:47:38.221234+0000","last_clean":"2024-09-20T22:47:38.221234+0000","last_became_active":"2024-09-20T22:43:57.430028+0000","last_became_peered":"2024-09-20T22:43:57.430028+0000","last_unstale":"2024-09-20T22:47:38.221234+0000","last_undegraded":"2024-09-20T22:47:38.221234+0000","last_fullsized":"2024-09-20T22:47:38.221234+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:45:53.933937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:45:53.933937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"181'7","reported_seq":505,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-20T22:47:38.507191+0000","last_change":"2024-09-20T22:45:36.106966+0000","last_active":"2024-09-20T22:47:38.507191+0000","last_peered":"2024-09-20T22:47:38.507191+0000","last_clean":"2024-09-20T22:47:38.507191+0000","last_became_active":"2024-09-20T22:43:57.441058+0000","last_became_peered":"2024-09-20T22:43:57.441058+0000","last_unstale":"2024-09-20T22:47:38.507191+0000","last_undegraded":"2024-09-20T22:47:38.507191+0000","last_fullsized":"2024-09-20T22:47:38.507191+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"181'7","last_scrub_stamp":"2024-09-20T22:45:36.106880+0000","last_deep_scrub":"181'7","last_deep_scrub_stamp":"2024-09-20T22:44:20.259598+0000","last_clean_scrub_stamp":"2024-09-20T22:45:36.106880+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":6,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":658,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-20T22:47:38.394407+0000","last_change":"2024-09-20T22:45:40.211399+0000","last_active":"2024-09-20T22:47:38.394407+0000","last_peered":"2024-09-20T22:47:38.394407+0000","last_clean":"2024-09-20T22:47:38.394407+0000","last_became_active":"2024-09-20T22:45:40.210886+0000","last_became_peered":"2024-09-20T22:45:40.210886+0000","last_unstale":"2024-09-20T22:47:38.394407+0000","last_undegraded":"2024-09-20T22:47:38.394407+0000","last_fullsized":"2024-09-20T22:47:38.394407+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:44:54.057283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:44:54.057283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":639,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-20T22:47:38.220721+0000","last_change":"2024-09-20T22:46:06.068842+0000","last_active":"2024-09-20T22:47:38.220721+0000","last_peered":"2024-09-20T22:47:38.220721+0000","last_clean":"2024-09-20T22:47:38.220721+0000","last_became_active":"2024-09-20T22:41:44.197927+0000","last_became_peered":"2024-09-20T22:41:44.197927+0000","last_unstale":"2024-09-20T22:47:38.220721+0000","last_undegraded":"2024-09-20T22:47:38.220721+0000","last_fullsized":"2024-09-20T22:47:38.220721+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:46:06.068745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:46:06.068745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":469,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-20T22:47:38.286850+0000","last_change":"2024-09-20T22:47:21.819643+0000","last_active":"2024-09-20T22:47:38.286850+0000","last_peered":"2024-09-20T22:47:38.286850+0000","last_clean":"2024-09-20T22:47:38.286850+0000","last_became_active":"2024-09-20T22:43:57.427971+0000","last_became_peered":"2024-09-20T22:43:57.427971+0000","last_unstale":"2024-09-20T22:47:38.286850+0000","last_undegraded":"2024-09-20T22:47:38.286850+0000","last_fullsized":"2024-09-20T22:47:38.286850+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:47:21.819572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:47:21.819572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:30.659501+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":444,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-20T22:47:38.394599+0000","last_change":"2024-09-20T22:47:13.413986+0000","last_active":"2024-09-20T22:47:38.394599+0000","last_peered":"2024-09-20T22:47:38.394599+0000","last_clean":"2024-09-20T22:47:38.394599+0000","last_became_active":"2024-09-20T22:45:40.229557+0000","last_became_peered":"2024-09-20T22:45:40.229557+0000","last_unstale":"2024-09-20T22:47:38.394599+0000","last_undegraded":"2024-09-20T22:47:38.394599+0000","last_fullsized":"2024-09-20T22:47:38.394599+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:47:13.413912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:47:13.413912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:19.554644+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1c","version":"113'22","reported_seq":686,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-20T22:47:38.284170+0000","last_change":"2024-09-20T22:46:25.731055+0000","last_active":"2024-09-20T22:47:38.284170+0000","last_peered":"2024-09-20T22:47:38.284170+0000","last_clean":"2024-09-20T22:47:38.284170+0000","last_became_active":"2024-09-20T22:41:40.170131+0000","last_became_peered":"2024-09-20T22:41:40.170131+0000","last_unstale":"2024-09-20T22:47:38.284170+0000","last_undegraded":"2024-09-20T22:47:38.284170+0000","last_fullsized":"2024-09-20T22:47:38.284170+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'22","last_scrub_stamp":"2024-09-20T22:46:25.730972+0000","last_deep_scrub":"113'22","last_deep_scrub_stamp":"2024-09-20T22:43:12.191878+0000","last_clean_scrub_stamp":"2024-09-20T22:46:25.730972+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":654,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-20T22:47:38.190832+0000","last_change":"2024-09-20T22:46:49.872553+0000","last_active":"2024-09-20T22:47:38.190832+0000","last_peered":"2024-09-20T22:47:38.190832+0000","last_clean":"2024-09-20T22:47:38.190832+0000","last_became_active":"2024-09-20T22:45:40.221969+0000","last_became_peered":"2024-09-20T22:45:40.221969+0000","last_unstale":"2024-09-20T22:47:38.190832+0000","last_undegraded":"2024-09-20T22:47:38.190832+0000","last_fullsized":"2024-09-20T22:47:38.190832+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:46:49.872479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:46:49.872479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:47:58.447433+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":630,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-20T22:47:38.508210+0000","last_change":"2024-09-20T22:45:25.002270+0000","last_active":"2024-09-20T22:47:38.508210+0000","last_peered":"2024-09-20T22:47:38.508210+0000","last_clean":"2024-09-20T22:47:38.508210+0000","last_became_active":"2024-09-20T22:41:44.832131+0000","last_became_peered":"2024-09-20T22:41:44.832131+0000","last_unstale":"2024-09-20T22:47:38.508210+0000","last_undegraded":"2024-09-20T22:47:38.508210+0000","last_fullsized":"2024-09-20T22:47:38.508210+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:45:25.002199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:45:25.002199+0000","last_clean_scrub_stamp":"2024-09-20T22:45:25.002199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":474,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-20T22:47:38.189338+0000","last_change":"2024-09-20T22:45:40.217601+0000","last_active":"2024-09-20T22:47:38.189338+0000","last_peered":"2024-09-20T22:47:38.189338+0000","last_clean":"2024-09-20T22:47:38.189338+0000","last_became_active":"2024-09-20T22:45:40.217251+0000","last_became_peered":"2024-09-20T22:45:40.217251+0000","last_unstale":"2024-09-20T22:47:38.189338+0000","last_undegraded":"2024-09-20T22:47:38.189338+0000","last_fullsized":"2024-09-20T22:47:38.189338+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:45:15.043443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:45:15.043443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"74'12","reported_seq":697,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-20T22:47:38.507689+0000","last_change":"2024-09-20T22:44:42.141018+0000","last_active":"2024-09-20T22:47:38.507689+0000","last_peered":"2024-09-20T22:47:38.507689+0000","last_clean":"2024-09-20T22:47:38.507689+0000","last_became_active":"2024-09-20T22:43:57.438162+0000","last_became_peered":"2024-09-20T22:43:57.438162+0000","last_unstale":"2024-09-20T22:47:38.507689+0000","last_undegraded":"2024-09-20T22:47:38.507689+0000","last_fullsized":"2024-09-20T22:47:38.507689+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-09-20T22:44:42.140915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:44:42.140915+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":460,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-20T22:47:38.507850+0000","last_change":"2024-09-20T22:44:14.235908+0000","last_active":"2024-09-20T22:47:38.507850+0000","last_peered":"2024-09-20T22:47:38.507850+0000","last_clean":"2024-09-20T22:47:38.507850+0000","last_became_active":"2024-09-20T22:43:57.429179+0000","last_became_peered":"2024-09-20T22:43:57.429179+0000","last_unstale":"2024-09-20T22:47:38.507850+0000","last_undegraded":"2024-09-20T22:47:38.507850+0000","last_fullsized":"2024-09-20T22:47:38.507850+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:44:14.235804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:44:14.235804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"154'2","reported_seq":641,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-20T22:47:38.395064+0000","last_change":"2024-09-20T22:46:21.446679+0000","last_active":"2024-09-20T22:47:38.395064+0000","last_peered":"2024-09-20T22:47:38.395064+0000","last_clean":"2024-09-20T22:47:38.395064+0000","last_became_active":"2024-09-20T22:41:44.199822+0000","last_became_peered":"2024-09-20T22:41:44.199822+0000","last_unstale":"2024-09-20T22:47:38.395064+0000","last_undegraded":"2024-09-20T22:47:38.395064+0000","last_fullsized":"2024-09-20T22:47:38.395064+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"154'2","last_scrub_stamp":"2024-09-20T22:46:21.446609+0000","last_deep_scrub":"154'2","last_deep_scrub_stamp":"2024-09-20T22:43:54.134066+0000","last_clean_scrub_stamp":"2024-09-20T22:46:21.446609+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"67'1","reported_seq":661,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-20T22:47:38.189130+0000","last_change":"2024-09-20T22:45:36.976765+0000","last_active":"2024-09-20T22:47:38.189130+0000","last_peered":"2024-09-20T22:47:38.189130+0000","last_clean":"2024-09-20T22:47:38.189130+0000","last_became_active":"2024-09-20T22:43:57.443155+0000","last_became_peered":"2024-09-20T22:43:57.443155+0000","last_unstale":"2024-09-20T22:47:38.189130+0000","last_undegraded":"2024-09-20T22:47:38.189130+0000","last_fullsized":"2024-09-20T22:47:38.189130+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-09-20T22:45:36.976689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:45:36.976689+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"151'15","reported_seq":460,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-20T22:47:38.394756+0000","last_change":"2024-09-20T22:45:58.439899+0000","last_active":"2024-09-20T22:47:38.394756+0000","last_peered":"2024-09-20T22:47:38.394756+0000","last_clean":"2024-09-20T22:47:38.394756+0000","last_became_active":"2024-09-20T22:45:40.212043+0000","last_became_peered":"2024-09-20T22:45:40.212043+0000","last_unstale":"2024-09-20T22:47:38.394756+0000","last_undegraded":"2024-09-20T22:47:38.394756+0000","last_fullsized":"2024-09-20T22:47:38.394756+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"151'15","last_scrub_stamp":"2024-09-20T22:45:58.439782+0000","last_deep_scrub":"151'15","last_deep_scrub_stamp":"2024-09-20T22:43:50.715761+0000","last_clean_scrub_stamp":"2024-09-20T22:45:58.439782+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"159'9","reported_seq":739,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-20T22:47:38.287791+0000","last_change":"2024-09-20T22:46:21.897778+0000","last_active":"2024-09-20T22:47:38.287791+0000","last_peered":"2024-09-20T22:47:38.287791+0000","last_clean":"2024-09-20T22:47:38.287791+0000","last_became_active":"2024-09-20T22:43:57.424879+0000","last_became_peered":"2024-09-20T22:43:57.424879+0000","last_unstale":"2024-09-20T22:47:38.287791+0000","last_undegraded":"2024-09-20T22:47:38.287791+0000","last_fullsized":"2024-09-20T22:47:38.287791+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"159'9","last_scrub_stamp":"2024-09-20T22:46:21.897701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:46:21.897701+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.c","version":"377'816","reported_seq":2281,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-20T22:47:38.286646+0000","last_change":"2024-09-20T22:47:37.258224+0000","last_active":"2024-09-20T22:47:38.286646+0000","last_peered":"2024-09-20T22:47:38.286646+0000","last_clean":"2024-09-20T22:47:38.286646+0000","last_became_active":"2024-09-20T22:42:52.404027+0000","last_became_peered":"2024-09-20T22:42:52.404027+0000","last_unstale":"2024-09-20T22:47:38.286646+0000","last_undegraded":"2024-09-20T22:47:38.286646+0000","last_fullsized":"2024-09-20T22:47:38.286646+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"332'667","last_scrub_stamp":"2024-09-20T22:46:52.845442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:46:52.845442+0000","objects_scrubbed":12,"log_size":816,"log_dups_size":0,"ondisk_log_size":816,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:14.963545+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0049756369999999998,"stat_sum":{"num_bytes":34827635,"num_objects":15,"num_object_clones":9,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":497,"num_read_kb":494614,"num_write":318,"num_write_kb":125564,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"6b","length":"1"},{"start":"74","length":"1"},{"start":"7a","length":"1"},{"start":"84","length":"1"}]},{"pgid":"1.a","version":"77'128","reported_seq":910,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-20T22:47:38.220588+0000","last_change":"2024-09-20T22:45:37.851690+0000","last_active":"2024-09-20T22:47:38.220588+0000","last_peered":"2024-09-20T22:47:38.220588+0000","last_clean":"2024-09-20T22:47:38.220588+0000","last_became_active":"2024-09-20T22:42:52.407791+0000","last_became_peered":"2024-09-20T22:42:52.407791+0000","last_unstale":"2024-09-20T22:47:38.220588+0000","last_undegraded":"2024-09-20T22:47:38.220588+0000","last_fullsized":"2024-09-20T22:47:38.220588+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:45:37.851582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:45:37.851582+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"74'15","reported_seq":693,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-20T22:47:38.191199+0000","last_change":"2024-09-20T22:44:55.972422+0000","last_active":"2024-09-20T22:47:38.191199+0000","last_peered":"2024-09-20T22:47:38.191199+0000","last_clean":"2024-09-20T22:47:38.191199+0000","last_became_active":"2024-09-20T22:43:57.441355+0000","last_became_peered":"2024-09-20T22:43:57.441355+0000","last_unstale":"2024-09-20T22:47:38.191199+0000","last_undegraded":"2024-09-20T22:47:38.191199+0000","last_fullsized":"2024-09-20T22:47:38.191199+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-09-20T22:44:55.972333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:44:55.972333+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":642,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-20T22:47:38.506036+0000","last_change":"2024-09-20T22:45:19.027462+0000","last_active":"2024-09-20T22:47:38.506036+0000","last_peered":"2024-09-20T22:47:38.506036+0000","last_clean":"2024-09-20T22:47:38.506036+0000","last_became_active":"2024-09-20T22:41:38.156749+0000","last_became_peered":"2024-09-20T22:41:38.156749+0000","last_unstale":"2024-09-20T22:47:38.506036+0000","last_undegraded":"2024-09-20T22:47:38.506036+0000","last_fullsized":"2024-09-20T22:47:38.506036+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:45:19.027393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:45:19.027393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"264'10","reported_seq":553,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-20T22:47:38.221764+0000","last_change":"2024-09-20T22:47:20.300876+0000","last_active":"2024-09-20T22:47:38.221764+0000","last_peered":"2024-09-20T22:47:38.221764+0000","last_clean":"2024-09-20T22:47:38.221764+0000","last_became_active":"2024-09-20T22:45:40.213956+0000","last_became_peered":"2024-09-20T22:45:40.213956+0000","last_unstale":"2024-09-20T22:47:38.221764+0000","last_undegraded":"2024-09-20T22:47:38.221764+0000","last_fullsized":"2024-09-20T22:47:38.221764+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"264'10","last_scrub_stamp":"2024-09-20T22:47:20.300787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:47:20.300787+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:25.147757+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":645,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-20T22:47:38.221087+0000","last_change":"2024-09-20T22:46:26.147242+0000","last_active":"2024-09-20T22:47:38.221087+0000","last_peered":"2024-09-20T22:47:38.221087+0000","last_clean":"2024-09-20T22:47:38.221087+0000","last_became_active":"2024-09-20T22:45:40.224443+0000","last_became_peered":"2024-09-20T22:45:40.224443+0000","last_unstale":"2024-09-20T22:47:38.221087+0000","last_undegraded":"2024-09-20T22:47:38.221087+0000","last_fullsized":"2024-09-20T22:47:38.221087+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:46:26.147167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:46:26.147167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3","version":"135'8","reported_seq":1414,"reported_epoch":378,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-20T22:47:38.506911+0000","last_change":"2024-09-20T22:47:37.248187+0000","last_active":"2024-09-20T22:47:38.506911+0000","last_peered":"2024-09-20T22:47:38.506911+0000","last_clean":"2024-09-20T22:47:38.506911+0000","last_became_active":"2024-09-20T22:43:57.439135+0000","last_became_peered":"2024-09-20T22:43:57.439135+0000","last_unstale":"2024-09-20T22:47:38.506911+0000","last_undegraded":"2024-09-20T22:47:38.506911+0000","last_fullsized":"2024-09-20T22:47:38.506911+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'8","last_scrub_stamp":"2024-09-20T22:47:23.137750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:47:23.137750+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:47.291292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.029843445,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"6b","length":"1"},{"start":"74","length":"1"}]},{"pgid":"1.5","version":"77'128","reported_seq":795,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-20T22:47:38.284174+0000","last_change":"2024-09-20T22:46:11.182702+0000","last_active":"2024-09-20T22:47:38.284174+0000","last_peered":"2024-09-20T22:47:38.284174+0000","last_clean":"2024-09-20T22:47:38.284174+0000","last_became_active":"2024-09-20T22:46:11.182471+0000","last_became_peered":"2024-09-20T22:46:11.182471+0000","last_unstale":"2024-09-20T22:47:38.284174+0000","last_undegraded":"2024-09-20T22:47:38.284174+0000","last_fullsized":"2024-09-20T22:47:38.284174+0000","mapping_epoch":160,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":292,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:45:41.608075+0000","last_deep_scrub":"77'128","last_deep_scrub_stamp":"2024-09-20T22:45:41.608075+0000","last_clean_scrub_stamp":"2024-09-20T22:45:41.608075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,5],"acting":[2,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"74'18","reported_seq":694,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-20T22:47:38.506152+0000","last_change":"2024-09-20T22:45:40.246576+0000","last_active":"2024-09-20T22:47:38.506152+0000","last_peered":"2024-09-20T22:47:38.506152+0000","last_clean":"2024-09-20T22:47:38.506152+0000","last_became_active":"2024-09-20T22:45:40.215766+0000","last_became_peered":"2024-09-20T22:45:40.215766+0000","last_unstale":"2024-09-20T22:47:38.506152+0000","last_undegraded":"2024-09-20T22:47:38.506152+0000","last_fullsized":"2024-09-20T22:47:38.506152+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'18","last_scrub_stamp":"2024-09-20T22:44:23.220709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:44:23.220709+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":648,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-20T22:47:38.508469+0000","last_change":"2024-09-20T22:45:40.213320+0000","last_active":"2024-09-20T22:47:38.508469+0000","last_peered":"2024-09-20T22:47:38.508469+0000","last_clean":"2024-09-20T22:47:38.508469+0000","last_became_active":"2024-09-20T22:45:40.213020+0000","last_became_peered":"2024-09-20T22:45:40.213020+0000","last_unstale":"2024-09-20T22:47:38.508469+0000","last_undegraded":"2024-09-20T22:47:38.508469+0000","last_fullsized":"2024-09-20T22:47:38.508469+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:43:51.170155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:43:51.170155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":466,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-20T22:47:38.507433+0000","last_change":"2024-09-20T22:45:56.199496+0000","last_active":"2024-09-20T22:47:38.507433+0000","last_peered":"2024-09-20T22:47:38.507433+0000","last_clean":"2024-09-20T22:47:38.507433+0000","last_became_active":"2024-09-20T22:45:40.214328+0000","last_became_peered":"2024-09-20T22:45:40.214328+0000","last_unstale":"2024-09-20T22:47:38.507433+0000","last_undegraded":"2024-09-20T22:47:38.507433+0000","last_fullsized":"2024-09-20T22:47:38.507433+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:45:56.199432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:07.947945+0000","last_clean_scrub_stamp":"2024-09-20T22:45:56.199432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":461,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-20T22:47:38.506855+0000","last_change":"2024-09-20T22:47:18.179688+0000","last_active":"2024-09-20T22:47:38.506855+0000","last_peered":"2024-09-20T22:47:38.506855+0000","last_clean":"2024-09-20T22:47:38.506855+0000","last_became_active":"2024-09-20T22:43:57.435057+0000","last_became_peered":"2024-09-20T22:43:57.435057+0000","last_unstale":"2024-09-20T22:47:38.506855+0000","last_undegraded":"2024-09-20T22:47:38.506855+0000","last_fullsized":"2024-09-20T22:47:38.506855+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:47:18.179575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:53.036601+0000","last_clean_scrub_stamp":"2024-09-20T22:47:18.179575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:25.505641+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,6],"acting":[3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2","version":"376'167","reported_seq":1159,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-20T22:47:38.247891+0000","last_change":"2024-09-20T22:47:38.248229+0000","last_active":"2024-09-20T22:47:38.247891+0000","last_peered":"2024-09-20T22:47:38.247891+0000","last_clean":"2024-09-20T22:47:38.247891+0000","last_became_active":"2024-09-20T22:44:09.177625+0000","last_became_peered":"2024-09-20T22:44:09.177625+0000","last_unstale":"2024-09-20T22:47:38.247891+0000","last_undegraded":"2024-09-20T22:47:38.247891+0000","last_fullsized":"2024-09-20T22:47:38.247891+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"282'120","last_scrub_stamp":"2024-09-20T22:46:22.099642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:46:22.099642+0000","objects_scrubbed":1,"log_size":167,"log_dups_size":0,"ondisk_log_size":167,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.027420458000000002,"stat_sum":{"num_bytes":9954330,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":69,"num_read_kb":83024,"num_write":30,"num_write_kb":9659,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"6b","length":"1"},{"start":"74","length":"1"},{"start":"7a","length":"1"},{"start":"84","length":"1"}]},{"pgid":"1.4","version":"77'128","reported_seq":942,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-20T22:47:38.507947+0000","last_change":"2024-09-20T22:46:15.181632+0000","last_active":"2024-09-20T22:47:38.507947+0000","last_peered":"2024-09-20T22:47:38.507947+0000","last_clean":"2024-09-20T22:47:38.507947+0000","last_became_active":"2024-09-20T22:46:15.181305+0000","last_became_peered":"2024-09-20T22:46:15.181305+0000","last_unstale":"2024-09-20T22:47:38.507947+0000","last_undegraded":"2024-09-20T22:47:38.507947+0000","last_fullsized":"2024-09-20T22:47:38.507947+0000","mapping_epoch":94,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:45:54.131305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:45:54.131305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":59,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"74'14","reported_seq":547,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-20T22:47:38.220737+0000","last_change":"2024-09-20T22:46:48.202431+0000","last_active":"2024-09-20T22:47:38.220737+0000","last_peered":"2024-09-20T22:47:38.220737+0000","last_clean":"2024-09-20T22:47:38.220737+0000","last_became_active":"2024-09-20T22:45:40.213664+0000","last_became_peered":"2024-09-20T22:45:40.213664+0000","last_unstale":"2024-09-20T22:47:38.220737+0000","last_undegraded":"2024-09-20T22:47:38.220737+0000","last_fullsized":"2024-09-20T22:47:38.220737+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'14","last_scrub_stamp":"2024-09-20T22:46:48.202311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:46:48.202311+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:14.529508+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,7],"acting":[5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.6","version":"67'1","reported_seq":257,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-20T22:47:38.191045+0000","last_change":"2024-09-20T22:45:40.246043+0000","last_active":"2024-09-20T22:47:38.191045+0000","last_peered":"2024-09-20T22:47:38.191045+0000","last_clean":"2024-09-20T22:47:38.191045+0000","last_became_active":"2024-09-20T22:45:40.224257+0000","last_became_peered":"2024-09-20T22:45:40.224257+0000","last_unstale":"2024-09-20T22:47:38.191045+0000","last_undegraded":"2024-09-20T22:47:38.191045+0000","last_fullsized":"2024-09-20T22:47:38.191045+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-09-20T22:45:25.781996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:45:25.781996+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"264'10","reported_seq":735,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-20T22:47:38.506282+0000","last_change":"2024-09-20T22:45:40.270343+0000","last_active":"2024-09-20T22:47:38.506282+0000","last_peered":"2024-09-20T22:47:38.506282+0000","last_clean":"2024-09-20T22:47:38.506282+0000","last_became_active":"2024-09-20T22:45:40.222219+0000","last_became_peered":"2024-09-20T22:45:40.222219+0000","last_unstale":"2024-09-20T22:47:38.506282+0000","last_undegraded":"2024-09-20T22:47:38.506282+0000","last_fullsized":"2024-09-20T22:47:38.506282+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"159'9","last_scrub_stamp":"2024-09-20T22:45:23.017605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:45:23.017605+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":639,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-20T22:47:38.394319+0000","last_change":"2024-09-20T22:47:16.381511+0000","last_active":"2024-09-20T22:47:38.394319+0000","last_peered":"2024-09-20T22:47:38.394319+0000","last_clean":"2024-09-20T22:47:38.394319+0000","last_became_active":"2024-09-20T22:41:44.831901+0000","last_became_peered":"2024-09-20T22:41:44.831901+0000","last_unstale":"2024-09-20T22:47:38.394319+0000","last_undegraded":"2024-09-20T22:47:38.394319+0000","last_fullsized":"2024-09-20T22:47:38.394319+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:47:16.381390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:47:16.381390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:23.066386+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1","version":"377'195","reported_seq":1571,"reported_epoch":378,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T22:47:38.506437+0000","last_change":"2024-09-20T22:47:37.248267+0000","last_active":"2024-09-20T22:47:38.506437+0000","last_peered":"2024-09-20T22:47:38.506437+0000","last_clean":"2024-09-20T22:47:38.506437+0000","last_became_active":"2024-09-20T22:43:57.440095+0000","last_became_peered":"2024-09-20T22:43:57.440095+0000","last_unstale":"2024-09-20T22:47:38.506437+0000","last_undegraded":"2024-09-20T22:47:38.506437+0000","last_fullsized":"2024-09-20T22:47:38.506437+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"170'72","last_scrub_stamp":"2024-09-20T22:44:12.228784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:44:12.228784+0000","objects_scrubbed":3,"log_size":195,"log_dups_size":0,"ondisk_log_size":195,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.040982708,"stat_sum":{"num_bytes":8791993,"num_objects":5,"num_object_clones":3,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":59,"num_read_kb":78458,"num_write":45,"num_write_kb":34476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"74","length":"1"}]},{"pgid":"1.7","version":"77'128","reported_seq":939,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-20T22:47:38.287624+0000","last_change":"2024-09-20T22:46:03.170000+0000","last_active":"2024-09-20T22:47:38.287624+0000","last_peered":"2024-09-20T22:47:38.287624+0000","last_clean":"2024-09-20T22:47:38.287624+0000","last_became_active":"2024-09-20T22:46:03.169596+0000","last_became_peered":"2024-09-20T22:46:03.169596+0000","last_unstale":"2024-09-20T22:47:38.287624+0000","last_undegraded":"2024-09-20T22:47:38.287624+0000","last_fullsized":"2024-09-20T22:47:38.287624+0000","mapping_epoch":94,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:45:25.813923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:45:25.813923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"74'10","reported_seq":697,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-20T22:47:38.189116+0000","last_change":"2024-09-20T22:45:40.249290+0000","last_active":"2024-09-20T22:47:38.189116+0000","last_peered":"2024-09-20T22:47:38.189116+0000","last_clean":"2024-09-20T22:47:38.189116+0000","last_became_active":"2024-09-20T22:45:40.221018+0000","last_became_peered":"2024-09-20T22:45:40.221018+0000","last_unstale":"2024-09-20T22:47:38.189116+0000","last_undegraded":"2024-09-20T22:47:38.189116+0000","last_fullsized":"2024-09-20T22:47:38.189116+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-09-20T22:44:59.049607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:44:59.049607+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":651,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-20T22:47:38.221614+0000","last_change":"2024-09-20T22:45:17.702193+0000","last_active":"2024-09-20T22:47:38.221614+0000","last_peered":"2024-09-20T22:47:38.221614+0000","last_clean":"2024-09-20T22:47:38.221614+0000","last_became_active":"2024-09-20T22:41:38.155629+0000","last_became_peered":"2024-09-20T22:41:38.155629+0000","last_unstale":"2024-09-20T22:47:38.221614+0000","last_undegraded":"2024-09-20T22:47:38.221614+0000","last_fullsized":"2024-09-20T22:47:38.221614+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:45:17.702106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:45:17.702106+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"264'10","reported_seq":301,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-20T22:47:38.286488+0000","last_change":"2024-09-20T22:45:57.889712+0000","last_active":"2024-09-20T22:47:38.286488+0000","last_peered":"2024-09-20T22:47:38.286488+0000","last_clean":"2024-09-20T22:47:38.286488+0000","last_became_active":"2024-09-20T22:45:40.211664+0000","last_became_peered":"2024-09-20T22:45:40.211664+0000","last_unstale":"2024-09-20T22:47:38.286488+0000","last_undegraded":"2024-09-20T22:47:38.286488+0000","last_fullsized":"2024-09-20T22:47:38.286488+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"264'10","last_scrub_stamp":"2024-09-20T22:45:57.889636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:45:57.889636+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":253,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-20T22:47:38.506285+0000","last_change":"2024-09-20T22:46:44.117883+0000","last_active":"2024-09-20T22:47:38.506285+0000","last_peered":"2024-09-20T22:47:38.506285+0000","last_clean":"2024-09-20T22:47:38.506285+0000","last_became_active":"2024-09-20T22:45:40.218780+0000","last_became_peered":"2024-09-20T22:45:40.218780+0000","last_unstale":"2024-09-20T22:47:38.506285+0000","last_undegraded":"2024-09-20T22:47:38.506285+0000","last_fullsized":"2024-09-20T22:47:38.506285+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:46:44.117761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:46:44.117761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:47:45.344447+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.0","version":"377'155","reported_seq":1599,"reported_epoch":378,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T22:47:38.220564+0000","last_change":"2024-09-20T22:47:38.218855+0000","last_active":"2024-09-20T22:47:38.220564+0000","last_peered":"2024-09-20T22:47:38.220564+0000","last_clean":"2024-09-20T22:47:38.220564+0000","last_became_active":"2024-09-20T22:43:37.136845+0000","last_became_peered":"2024-09-20T22:43:37.136845+0000","last_unstale":"2024-09-20T22:47:38.220564+0000","last_undegraded":"2024-09-20T22:47:38.220564+0000","last_fullsized":"2024-09-20T22:47:38.220564+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"280'122","last_scrub_stamp":"2024-09-20T22:46:03.006423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:46:03.006423+0000","objects_scrubbed":1,"log_size":155,"log_dups_size":0,"ondisk_log_size":155,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.972896329,"stat_sum":{"num_bytes":8921635,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":59,"num_read_kb":57291,"num_write":40,"num_write_kb":9686,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"6b","length":"1"},{"start":"74","length":"1"},{"start":"7a","length":"1"}]},{"pgid":"1.6","version":"77'128","reported_seq":946,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-20T22:47:38.286758+0000","last_change":"2024-09-20T22:46:07.176473+0000","last_active":"2024-09-20T22:47:38.286758+0000","last_peered":"2024-09-20T22:47:38.286758+0000","last_clean":"2024-09-20T22:47:38.286758+0000","last_became_active":"2024-09-20T22:46:07.175920+0000","last_became_peered":"2024-09-20T22:46:07.175920+0000","last_unstale":"2024-09-20T22:47:38.286758+0000","last_undegraded":"2024-09-20T22:47:38.286758+0000","last_fullsized":"2024-09-20T22:47:38.286758+0000","mapping_epoch":263,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":288,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:45:17.692996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:45:17.692996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"74'19","reported_seq":327,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-20T22:47:38.221234+0000","last_change":"2024-09-20T22:47:13.403322+0000","last_active":"2024-09-20T22:47:38.221234+0000","last_peered":"2024-09-20T22:47:38.221234+0000","last_clean":"2024-09-20T22:47:38.221234+0000","last_became_active":"2024-09-20T22:45:40.229189+0000","last_became_peered":"2024-09-20T22:45:40.229189+0000","last_unstale":"2024-09-20T22:47:38.221234+0000","last_undegraded":"2024-09-20T22:47:38.221234+0000","last_fullsized":"2024-09-20T22:47:38.221234+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'19","last_scrub_stamp":"2024-09-20T22:47:13.403224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:47:13.403224+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:43.024760+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":38,"num_read_kb":24,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":650,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-20T22:47:38.190492+0000","last_change":"2024-09-20T22:45:21.038355+0000","last_active":"2024-09-20T22:47:38.190492+0000","last_peered":"2024-09-20T22:47:38.190492+0000","last_clean":"2024-09-20T22:47:38.190492+0000","last_became_active":"2024-09-20T22:41:38.160832+0000","last_became_peered":"2024-09-20T22:41:38.160832+0000","last_unstale":"2024-09-20T22:47:38.190492+0000","last_undegraded":"2024-09-20T22:47:38.190492+0000","last_fullsized":"2024-09-20T22:47:38.190492+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:45:21.038284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:45:21.038284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":648,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-20T22:47:38.286963+0000","last_change":"2024-09-20T22:46:43.827791+0000","last_active":"2024-09-20T22:47:38.286963+0000","last_peered":"2024-09-20T22:47:38.286963+0000","last_clean":"2024-09-20T22:47:38.286963+0000","last_became_active":"2024-09-20T22:45:40.212686+0000","last_became_peered":"2024-09-20T22:45:40.212686+0000","last_unstale":"2024-09-20T22:47:38.286963+0000","last_undegraded":"2024-09-20T22:47:38.286963+0000","last_fullsized":"2024-09-20T22:47:38.286963+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:46:43.827680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:46:43.827680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:47:58.640025+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":638,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-20T22:47:38.188842+0000","last_change":"2024-09-20T22:45:39.938227+0000","last_active":"2024-09-20T22:47:38.188842+0000","last_peered":"2024-09-20T22:47:38.188842+0000","last_clean":"2024-09-20T22:47:38.188842+0000","last_became_active":"2024-09-20T22:41:44.830538+0000","last_became_peered":"2024-09-20T22:41:44.830538+0000","last_unstale":"2024-09-20T22:47:38.188842+0000","last_undegraded":"2024-09-20T22:47:38.188842+0000","last_fullsized":"2024-09-20T22:47:38.188842+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:45:39.938119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:45:39.938119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.7","version":"377'332","reported_seq":1818,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-20T22:47:38.287788+0000","last_change":"2024-09-20T22:47:37.258731+0000","last_active":"2024-09-20T22:47:38.287788+0000","last_peered":"2024-09-20T22:47:38.287788+0000","last_clean":"2024-09-20T22:47:38.287788+0000","last_became_active":"2024-09-20T22:45:43.169140+0000","last_became_peered":"2024-09-20T22:45:43.169140+0000","last_unstale":"2024-09-20T22:47:38.287788+0000","last_undegraded":"2024-09-20T22:47:38.287788+0000","last_fullsized":"2024-09-20T22:47:38.287788+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"372'324","last_scrub_stamp":"2024-09-20T22:47:33.829464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:47:33.829464+0000","objects_scrubbed":7,"log_size":332,"log_dups_size":0,"ondisk_log_size":332,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:52.987197+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0047492749999999999,"stat_sum":{"num_bytes":20001455,"num_objects":7,"num_object_clones":4,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":106,"num_read_kb":114984,"num_write":94,"num_write_kb":35683,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":8139149,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"6b","length":"1"},{"start":"74","length":"1"},{"start":"7a","length":"1"},{"start":"84","length":"1"}]},{"pgid":"1.1","version":"77'128","reported_seq":940,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-20T22:47:38.220555+0000","last_change":"2024-09-20T22:45:58.908515+0000","last_active":"2024-09-20T22:47:38.220555+0000","last_peered":"2024-09-20T22:47:38.220555+0000","last_clean":"2024-09-20T22:47:38.220555+0000","last_became_active":"2024-09-20T22:45:55.169241+0000","last_became_peered":"2024-09-20T22:45:55.169241+0000","last_unstale":"2024-09-20T22:47:38.220555+0000","last_undegraded":"2024-09-20T22:47:38.220555+0000","last_fullsized":"2024-09-20T22:47:38.220555+0000","mapping_epoch":263,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":276,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:45:58.908433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:45:58.908433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"86'38","reported_seq":777,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-20T22:47:38.190401+0000","last_change":"2024-09-20T22:44:41.077590+0000","last_active":"2024-09-20T22:47:38.190401+0000","last_peered":"2024-09-20T22:47:38.190401+0000","last_clean":"2024-09-20T22:47:38.190401+0000","last_became_active":"2024-09-20T22:41:40.166958+0000","last_became_peered":"2024-09-20T22:41:40.166958+0000","last_unstale":"2024-09-20T22:47:38.190401+0000","last_undegraded":"2024-09-20T22:47:38.190401+0000","last_fullsized":"2024-09-20T22:47:38.190401+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'38","last_scrub_stamp":"2024-09-20T22:44:41.077475+0000","last_deep_scrub":"86'38","last_deep_scrub_stamp":"2024-09-20T22:42:43.935677+0000","last_clean_scrub_stamp":"2024-09-20T22:44:41.077475+0000","objects_scrubbed":10,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":255,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-20T22:47:38.286587+0000","last_change":"2024-09-20T22:47:16.817569+0000","last_active":"2024-09-20T22:47:38.286587+0000","last_peered":"2024-09-20T22:47:38.286587+0000","last_clean":"2024-09-20T22:47:38.286587+0000","last_became_active":"2024-09-20T22:45:40.204848+0000","last_became_peered":"2024-09-20T22:45:40.204848+0000","last_unstale":"2024-09-20T22:47:38.286587+0000","last_undegraded":"2024-09-20T22:47:38.286587+0000","last_fullsized":"2024-09-20T22:47:38.286587+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:47:16.817468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:47:16.817468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:38.645748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"69'2","reported_seq":669,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-20T22:47:38.221135+0000","last_change":"2024-09-20T22:46:44.210668+0000","last_active":"2024-09-20T22:47:38.221135+0000","last_peered":"2024-09-20T22:47:38.221135+0000","last_clean":"2024-09-20T22:47:38.221135+0000","last_became_active":"2024-09-20T22:41:37.148479+0000","last_became_peered":"2024-09-20T22:41:37.148479+0000","last_unstale":"2024-09-20T22:47:38.221135+0000","last_undegraded":"2024-09-20T22:47:38.221135+0000","last_fullsized":"2024-09-20T22:47:38.221135+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2024-09-20T22:46:44.210559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:36.130684+0000","last_clean_scrub_stamp":"2024-09-20T22:46:44.210559+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:09.950285+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00087280599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":436,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-20T22:47:38.190711+0000","last_change":"2024-09-20T22:46:06.104856+0000","last_active":"2024-09-20T22:47:38.190711+0000","last_peered":"2024-09-20T22:47:38.190711+0000","last_clean":"2024-09-20T22:47:38.190711+0000","last_became_active":"2024-09-20T22:45:40.207256+0000","last_became_peered":"2024-09-20T22:45:40.207256+0000","last_unstale":"2024-09-20T22:47:38.190711+0000","last_undegraded":"2024-09-20T22:47:38.190711+0000","last_fullsized":"2024-09-20T22:47:38.190711+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:46:06.104786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:46:06.104786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":644,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-20T22:47:38.507944+0000","last_change":"2024-09-20T22:45:58.213051+0000","last_active":"2024-09-20T22:47:38.507944+0000","last_peered":"2024-09-20T22:47:38.507944+0000","last_clean":"2024-09-20T22:47:38.507944+0000","last_became_active":"2024-09-20T22:43:57.434044+0000","last_became_peered":"2024-09-20T22:43:57.434044+0000","last_unstale":"2024-09-20T22:47:38.507944+0000","last_undegraded":"2024-09-20T22:47:38.507944+0000","last_fullsized":"2024-09-20T22:47:38.507944+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:45:58.212988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:45:58.212988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"375'73","reported_seq":1268,"reported_epoch":378,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-20T22:47:38.190338+0000","last_change":"2024-09-20T22:47:37.244160+0000","last_active":"2024-09-20T22:47:38.190338+0000","last_peered":"2024-09-20T22:47:38.190338+0000","last_clean":"2024-09-20T22:47:38.190338+0000","last_became_active":"2024-09-20T22:45:40.223468+0000","last_became_peered":"2024-09-20T22:45:40.223468+0000","last_unstale":"2024-09-20T22:47:38.190338+0000","last_undegraded":"2024-09-20T22:47:38.190338+0000","last_fullsized":"2024-09-20T22:47:38.190338+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"305'57","last_scrub_stamp":"2024-09-20T22:47:13.636468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:47:13.636468+0000","objects_scrubbed":1,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:34.245288+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.029455342999999998,"stat_sum":{"num_bytes":7090788,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":31,"num_read_kb":31741,"num_write":16,"num_write_kb":3549,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"6b","length":"1"}]},{"pgid":"1.2","version":"77'128","reported_seq":990,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-20T22:47:38.394211+0000","last_change":"2024-09-20T22:45:51.178542+0000","last_active":"2024-09-20T22:47:38.394211+0000","last_peered":"2024-09-20T22:47:38.394211+0000","last_clean":"2024-09-20T22:47:38.394211+0000","last_became_active":"2024-09-20T22:45:51.178152+0000","last_became_peered":"2024-09-20T22:45:51.178152+0000","last_unstale":"2024-09-20T22:47:38.394211+0000","last_undegraded":"2024-09-20T22:47:38.394211+0000","last_fullsized":"2024-09-20T22:47:38.394211+0000","mapping_epoch":263,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:44:58.121721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:44:58.121721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"74'13","reported_seq":691,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-20T22:47:38.191219+0000","last_change":"2024-09-20T22:46:04.145988+0000","last_active":"2024-09-20T22:47:38.191219+0000","last_peered":"2024-09-20T22:47:38.191219+0000","last_clean":"2024-09-20T22:47:38.191219+0000","last_became_active":"2024-09-20T22:45:40.221991+0000","last_became_peered":"2024-09-20T22:45:40.221991+0000","last_unstale":"2024-09-20T22:47:38.191219+0000","last_undegraded":"2024-09-20T22:47:38.191219+0000","last_fullsized":"2024-09-20T22:47:38.191219+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'13","last_scrub_stamp":"2024-09-20T22:46:04.145892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:46:04.145892+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":650,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-20T22:47:38.188362+0000","last_change":"2024-09-20T22:44:35.131693+0000","last_active":"2024-09-20T22:47:38.188362+0000","last_peered":"2024-09-20T22:47:38.188362+0000","last_clean":"2024-09-20T22:47:38.188362+0000","last_became_active":"2024-09-20T22:41:38.156837+0000","last_became_peered":"2024-09-20T22:41:38.156837+0000","last_unstale":"2024-09-20T22:47:38.188362+0000","last_undegraded":"2024-09-20T22:47:38.188362+0000","last_fullsized":"2024-09-20T22:47:38.188362+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:44:35.131583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:44:35.131583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"67'1","reported_seq":677,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-20T22:47:38.283557+0000","last_change":"2024-09-20T22:46:00.792036+0000","last_active":"2024-09-20T22:47:38.283557+0000","last_peered":"2024-09-20T22:47:38.283557+0000","last_clean":"2024-09-20T22:47:38.283557+0000","last_became_active":"2024-09-20T22:45:40.223966+0000","last_became_peered":"2024-09-20T22:45:40.223966+0000","last_unstale":"2024-09-20T22:47:38.283557+0000","last_undegraded":"2024-09-20T22:47:38.283557+0000","last_fullsized":"2024-09-20T22:47:38.283557+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-09-20T22:46:00.791909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:36.130684+0000","last_clean_scrub_stamp":"2024-09-20T22:46:00.791909+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001301459,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":650,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-20T22:47:38.284807+0000","last_change":"2024-09-20T22:47:22.881180+0000","last_active":"2024-09-20T22:47:38.284807+0000","last_peered":"2024-09-20T22:47:38.284807+0000","last_clean":"2024-09-20T22:47:38.284807+0000","last_became_active":"2024-09-20T22:41:42.185112+0000","last_became_peered":"2024-09-20T22:41:42.185112+0000","last_unstale":"2024-09-20T22:47:38.284807+0000","last_undegraded":"2024-09-20T22:47:38.284807+0000","last_fullsized":"2024-09-20T22:47:38.284807+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:47:22.881109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:47:22.881109+0000","last_clean_scrub_stamp":"2024-09-20T22:47:22.881109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:46.218094+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":639,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-20T22:47:38.395663+0000","last_change":"2024-09-20T22:46:27.448645+0000","last_active":"2024-09-20T22:47:38.395663+0000","last_peered":"2024-09-20T22:47:38.395663+0000","last_clean":"2024-09-20T22:47:38.395663+0000","last_became_active":"2024-09-20T22:41:44.830515+0000","last_became_peered":"2024-09-20T22:41:44.830515+0000","last_unstale":"2024-09-20T22:47:38.395663+0000","last_undegraded":"2024-09-20T22:47:38.395663+0000","last_fullsized":"2024-09-20T22:47:38.395663+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:46:27.448572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:46:27.448572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"377'140","reported_seq":1575,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-20T22:47:38.395595+0000","last_change":"2024-09-20T22:47:37.251593+0000","last_active":"2024-09-20T22:47:38.395595+0000","last_peered":"2024-09-20T22:47:38.395595+0000","last_clean":"2024-09-20T22:47:38.395595+0000","last_became_active":"2024-09-20T22:45:40.220614+0000","last_became_peered":"2024-09-20T22:45:40.220614+0000","last_unstale":"2024-09-20T22:47:38.395595+0000","last_undegraded":"2024-09-20T22:47:38.395595+0000","last_fullsized":"2024-09-20T22:47:38.395595+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"262'49","last_scrub_stamp":"2024-09-20T22:45:41.346463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:45:41.346463+0000","objects_scrubbed":4,"log_size":140,"log_dups_size":0,"ondisk_log_size":140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.002534096,"stat_sum":{"num_bytes":14113470,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":23854,"num_write":62,"num_write_kb":30562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":5530930,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"6b","length":"1"},{"start":"74","length":"1"},{"start":"7a","length":"1"},{"start":"84","length":"1"}]},{"pgid":"1.3","version":"77'128","reported_seq":951,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-20T22:47:38.189460+0000","last_change":"2024-09-20T22:46:19.180360+0000","last_active":"2024-09-20T22:47:38.189460+0000","last_peered":"2024-09-20T22:47:38.189460+0000","last_clean":"2024-09-20T22:47:38.189460+0000","last_became_active":"2024-09-20T22:46:19.180012+0000","last_became_peered":"2024-09-20T22:46:19.180012+0000","last_unstale":"2024-09-20T22:47:38.189460+0000","last_undegraded":"2024-09-20T22:47:38.189460+0000","last_fullsized":"2024-09-20T22:47:38.189460+0000","mapping_epoch":160,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":300,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:45:38.952841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:45:38.952841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"74'12","reported_seq":368,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-20T22:47:38.221586+0000","last_change":"2024-09-20T22:45:40.231346+0000","last_active":"2024-09-20T22:47:38.221586+0000","last_peered":"2024-09-20T22:47:38.221586+0000","last_clean":"2024-09-20T22:47:38.221586+0000","last_became_active":"2024-09-20T22:45:40.214689+0000","last_became_peered":"2024-09-20T22:45:40.214689+0000","last_unstale":"2024-09-20T22:47:38.221586+0000","last_undegraded":"2024-09-20T22:47:38.221586+0000","last_fullsized":"2024-09-20T22:47:38.221586+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-09-20T22:45:19.776506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:45:19.776506+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":436,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-20T22:47:38.507619+0000","last_change":"2024-09-20T22:45:40.213422+0000","last_active":"2024-09-20T22:47:38.507619+0000","last_peered":"2024-09-20T22:47:38.507619+0000","last_clean":"2024-09-20T22:47:38.507619+0000","last_became_active":"2024-09-20T22:45:40.213120+0000","last_became_peered":"2024-09-20T22:45:40.213120+0000","last_unstale":"2024-09-20T22:47:38.507619+0000","last_undegraded":"2024-09-20T22:47:38.507619+0000","last_fullsized":"2024-09-20T22:47:38.507619+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:44:19.834641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:44:19.834641+0000","last_clean_scrub_stamp":"2024-09-20T22:44:19.834641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"75'5","reported_seq":1027,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-20T22:47:38.394325+0000","last_change":"2024-09-20T22:46:22.436542+0000","last_active":"2024-09-20T22:47:38.394325+0000","last_peered":"2024-09-20T22:47:38.394325+0000","last_clean":"2024-09-20T22:47:38.394325+0000","last_became_active":"2024-09-20T22:45:40.212288+0000","last_became_peered":"2024-09-20T22:45:40.212288+0000","last_unstale":"2024-09-20T22:47:38.394325+0000","last_undegraded":"2024-09-20T22:47:38.394325+0000","last_fullsized":"2024-09-20T22:47:38.394325+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-09-20T22:46:22.436424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:36.130684+0000","last_clean_scrub_stamp":"2024-09-20T22:46:22.436424+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:47:50.777035+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00078631700000000003,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":353,"num_read_kb":348,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":649,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-20T22:47:38.221955+0000","last_change":"2024-09-20T22:47:21.268561+0000","last_active":"2024-09-20T22:47:38.221955+0000","last_peered":"2024-09-20T22:47:38.221955+0000","last_clean":"2024-09-20T22:47:38.221955+0000","last_became_active":"2024-09-20T22:45:40.229194+0000","last_became_peered":"2024-09-20T22:45:40.229194+0000","last_unstale":"2024-09-20T22:47:38.221955+0000","last_undegraded":"2024-09-20T22:47:38.221955+0000","last_fullsized":"2024-09-20T22:47:38.221955+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:47:21.268493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:47:21.268493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:37.727493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":638,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-20T22:47:38.189951+0000","last_change":"2024-09-20T22:44:54.014154+0000","last_active":"2024-09-20T22:47:38.189951+0000","last_peered":"2024-09-20T22:47:38.189951+0000","last_clean":"2024-09-20T22:47:38.189951+0000","last_became_active":"2024-09-20T22:41:44.199046+0000","last_became_peered":"2024-09-20T22:41:44.199046+0000","last_unstale":"2024-09-20T22:47:38.189951+0000","last_undegraded":"2024-09-20T22:47:38.189951+0000","last_fullsized":"2024-09-20T22:47:38.189951+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:44:54.014085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:44:54.014085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.6","version":"377'131","reported_seq":1561,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-20T22:47:38.395317+0000","last_change":"2024-09-20T22:47:37.254380+0000","last_active":"2024-09-20T22:47:38.395317+0000","last_peered":"2024-09-20T22:47:38.395317+0000","last_clean":"2024-09-20T22:47:38.395317+0000","last_became_active":"2024-09-20T22:45:40.217779+0000","last_became_peered":"2024-09-20T22:45:40.217779+0000","last_unstale":"2024-09-20T22:47:38.395317+0000","last_undegraded":"2024-09-20T22:47:38.395317+0000","last_fullsized":"2024-09-20T22:47:38.395317+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"282'87","last_scrub_stamp":"2024-09-20T22:46:03.391903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:46:03.391903+0000","objects_scrubbed":0,"log_size":131,"log_dups_size":0,"ondisk_log_size":131,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0050434679999999997,"stat_sum":{"num_bytes":4660216,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":21,"num_read_kb":17312,"num_write":24,"num_write_kb":5144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":2573889,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"6b","length":"1"},{"start":"74","length":"1"},{"start":"7a","length":"1"},{"start":"84","length":"1"}]},{"pgid":"4.5","version":"264'18","reported_seq":789,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-20T22:47:38.287347+0000","last_change":"2024-09-20T22:46:04.901692+0000","last_active":"2024-09-20T22:47:38.287347+0000","last_peered":"2024-09-20T22:47:38.287347+0000","last_clean":"2024-09-20T22:47:38.287347+0000","last_became_active":"2024-09-20T22:45:40.220283+0000","last_became_peered":"2024-09-20T22:45:40.220283+0000","last_unstale":"2024-09-20T22:47:38.287347+0000","last_undegraded":"2024-09-20T22:47:38.287347+0000","last_fullsized":"2024-09-20T22:47:38.287347+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"264'18","last_scrub_stamp":"2024-09-20T22:46:04.901561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:46:04.901561+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"67'1","reported_seq":648,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-20T22:47:38.507016+0000","last_change":"2024-09-20T22:45:22.053343+0000","last_active":"2024-09-20T22:47:38.507016+0000","last_peered":"2024-09-20T22:47:38.507016+0000","last_clean":"2024-09-20T22:47:38.507016+0000","last_became_active":"2024-09-20T22:41:38.158917+0000","last_became_peered":"2024-09-20T22:41:38.158917+0000","last_unstale":"2024-09-20T22:47:38.507016+0000","last_undegraded":"2024-09-20T22:47:38.507016+0000","last_fullsized":"2024-09-20T22:47:38.507016+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-09-20T22:45:22.053262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:45:22.053262+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"77'128","reported_seq":982,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-20T22:47:38.394404+0000","last_change":"2024-09-20T22:47:22.417229+0000","last_active":"2024-09-20T22:47:38.394404+0000","last_peered":"2024-09-20T22:47:38.394404+0000","last_clean":"2024-09-20T22:47:38.394404+0000","last_became_active":"2024-09-20T22:45:59.171055+0000","last_became_peered":"2024-09-20T22:45:59.171055+0000","last_unstale":"2024-09-20T22:47:38.394404+0000","last_undegraded":"2024-09-20T22:47:38.394404+0000","last_fullsized":"2024-09-20T22:47:38.394404+0000","mapping_epoch":263,"log_start":"77'128","ondisk_log_start":"77'128","created":19,"last_epoch_clean":280,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:47:22.417099+0000","last_deep_scrub":"77'128","last_deep_scrub_stamp":"2024-09-20T22:47:22.417099+0000","last_clean_scrub_stamp":"2024-09-20T22:47:22.417099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:37.110257+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":643,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-20T22:47:38.395206+0000","last_change":"2024-09-20T22:45:42.357247+0000","last_active":"2024-09-20T22:47:38.395206+0000","last_peered":"2024-09-20T22:47:38.395206+0000","last_clean":"2024-09-20T22:47:38.395206+0000","last_became_active":"2024-09-20T22:41:42.184387+0000","last_became_peered":"2024-09-20T22:41:42.184387+0000","last_unstale":"2024-09-20T22:47:38.395206+0000","last_undegraded":"2024-09-20T22:47:38.395206+0000","last_fullsized":"2024-09-20T22:47:38.395206+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:45:42.357173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:45:42.357173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":645,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-20T22:47:38.221842+0000","last_change":"2024-09-20T22:45:41.820396+0000","last_active":"2024-09-20T22:47:38.221842+0000","last_peered":"2024-09-20T22:47:38.221842+0000","last_clean":"2024-09-20T22:47:38.221842+0000","last_became_active":"2024-09-20T22:43:57.434435+0000","last_became_peered":"2024-09-20T22:43:57.434435+0000","last_unstale":"2024-09-20T22:47:38.221842+0000","last_undegraded":"2024-09-20T22:47:38.221842+0000","last_fullsized":"2024-09-20T22:47:38.221842+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:45:41.820321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:45:41.820321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"376'154","reported_seq":1602,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-20T22:47:38.287385+0000","last_change":"2024-09-20T22:47:37.255413+0000","last_active":"2024-09-20T22:47:38.287385+0000","last_peered":"2024-09-20T22:47:38.287385+0000","last_clean":"2024-09-20T22:47:38.287385+0000","last_became_active":"2024-09-20T22:45:40.219625+0000","last_became_peered":"2024-09-20T22:45:40.219625+0000","last_unstale":"2024-09-20T22:47:38.287385+0000","last_undegraded":"2024-09-20T22:47:38.287385+0000","last_fullsized":"2024-09-20T22:47:38.287385+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"327'109","last_scrub_stamp":"2024-09-20T22:46:51.808234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:46:51.808234+0000","objects_scrubbed":2,"log_size":154,"log_dups_size":0,"ondisk_log_size":154,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:47:59.179934+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0029827740000000001,"stat_sum":{"num_bytes":12471025,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":92,"num_read_kb":93497,"num_write":41,"num_write_kb":19366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2755788,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"6b","length":"1"},{"start":"74","length":"1"},{"start":"7a","length":"1"},{"start":"84","length":"1"}]},{"pgid":"4.e","version":"74'11","reported_seq":536,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-20T22:47:38.286926+0000","last_change":"2024-09-20T22:46:49.809263+0000","last_active":"2024-09-20T22:47:38.286926+0000","last_peered":"2024-09-20T22:47:38.286926+0000","last_clean":"2024-09-20T22:47:38.286926+0000","last_became_active":"2024-09-20T22:44:09.177976+0000","last_became_peered":"2024-09-20T22:44:09.177976+0000","last_unstale":"2024-09-20T22:47:38.286926+0000","last_undegraded":"2024-09-20T22:47:38.286926+0000","last_fullsized":"2024-09-20T22:47:38.286926+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-09-20T22:46:49.809151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:46:49.809151+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:47:57.133405+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":469,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-20T22:47:38.283421+0000","last_change":"2024-09-20T22:46:30.871847+0000","last_active":"2024-09-20T22:47:38.283421+0000","last_peered":"2024-09-20T22:47:38.283421+0000","last_clean":"2024-09-20T22:47:38.283421+0000","last_became_active":"2024-09-20T22:43:57.432492+0000","last_became_peered":"2024-09-20T22:43:57.432492+0000","last_unstale":"2024-09-20T22:47:38.283421+0000","last_undegraded":"2024-09-20T22:47:38.283421+0000","last_fullsized":"2024-09-20T22:47:38.283421+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:46:30.871771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:46:30.871771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:00.060628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":649,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-20T22:47:38.288983+0000","last_change":"2024-09-20T22:47:38.289013+0000","last_active":"2024-09-20T22:47:38.288983+0000","last_peered":"2024-09-20T22:47:38.288983+0000","last_clean":"2024-09-20T22:47:38.288983+0000","last_became_active":"2024-09-20T22:43:57.419756+0000","last_became_peered":"2024-09-20T22:43:57.419756+0000","last_unstale":"2024-09-20T22:47:38.288983+0000","last_undegraded":"2024-09-20T22:47:38.288983+0000","last_fullsized":"2024-09-20T22:47:38.288983+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:47:38.288926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:47:38.288926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:58.948654+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":630,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-20T22:47:38.507909+0000","last_change":"2024-09-20T22:44:21.225215+0000","last_active":"2024-09-20T22:47:38.507909+0000","last_peered":"2024-09-20T22:47:38.507909+0000","last_clean":"2024-09-20T22:47:38.507909+0000","last_became_active":"2024-09-20T22:41:44.832009+0000","last_became_peered":"2024-09-20T22:41:44.832009+0000","last_unstale":"2024-09-20T22:47:38.507909+0000","last_undegraded":"2024-09-20T22:47:38.507909+0000","last_fullsized":"2024-09-20T22:47:38.507909+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:44:21.225119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:44:21.225119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.e","version":"377'202","reported_seq":1667,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-20T22:47:38.284502+0000","last_change":"2024-09-20T22:47:37.249816+0000","last_active":"2024-09-20T22:47:38.284502+0000","last_peered":"2024-09-20T22:47:38.284502+0000","last_clean":"2024-09-20T22:47:38.284502+0000","last_became_active":"2024-09-20T22:45:43.167013+0000","last_became_peered":"2024-09-20T22:45:43.167013+0000","last_unstale":"2024-09-20T22:47:38.284502+0000","last_undegraded":"2024-09-20T22:47:38.284502+0000","last_fullsized":"2024-09-20T22:47:38.284502+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"270'130","last_scrub_stamp":"2024-09-20T22:45:53.692660+0000","last_deep_scrub":"174'60","last_deep_scrub_stamp":"2024-09-20T22:44:18.301776+0000","last_clean_scrub_stamp":"2024-09-20T22:45:53.692660+0000","objects_scrubbed":4,"log_size":202,"log_dups_size":0,"ondisk_log_size":202,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0022308390000000001,"stat_sum":{"num_bytes":11283510,"num_objects":5,"num_object_clones":3,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":130,"num_read_kb":145192,"num_write":71,"num_write_kb":29334,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":9591801,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"6b","length":"1"},{"start":"74","length":"1"},{"start":"7a","length":"1"},{"start":"84","length":"1"}]},{"pgid":"1.8","version":"77'128","reported_seq":910,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-20T22:47:38.220409+0000","last_change":"2024-09-20T22:45:22.698507+0000","last_active":"2024-09-20T22:47:38.220409+0000","last_peered":"2024-09-20T22:47:38.220409+0000","last_clean":"2024-09-20T22:47:38.220409+0000","last_became_active":"2024-09-20T22:42:52.405594+0000","last_became_peered":"2024-09-20T22:42:52.405594+0000","last_unstale":"2024-09-20T22:47:38.220409+0000","last_undegraded":"2024-09-20T22:47:38.220409+0000","last_fullsized":"2024-09-20T22:47:38.220409+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:45:22.698390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:45:22.698390+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"74'17","reported_seq":542,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-20T22:47:38.284280+0000","last_change":"2024-09-20T22:47:38.052271+0000","last_active":"2024-09-20T22:47:38.284280+0000","last_peered":"2024-09-20T22:47:38.284280+0000","last_clean":"2024-09-20T22:47:38.284280+0000","last_became_active":"2024-09-20T22:43:57.432621+0000","last_became_peered":"2024-09-20T22:43:57.432621+0000","last_unstale":"2024-09-20T22:47:38.284280+0000","last_undegraded":"2024-09-20T22:47:38.284280+0000","last_fullsized":"2024-09-20T22:47:38.284280+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'17","last_scrub_stamp":"2024-09-20T22:47:38.052166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:47:38.052166+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:47.679970+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":28,"num_read_kb":18,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":185,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-20T22:47:38.283586+0000","last_change":"2024-09-20T22:46:10.291173+0000","last_active":"2024-09-20T22:47:38.283586+0000","last_peered":"2024-09-20T22:47:38.283586+0000","last_clean":"2024-09-20T22:47:38.283586+0000","last_became_active":"2024-09-20T22:46:10.290822+0000","last_became_peered":"2024-09-20T22:46:10.290822+0000","last_unstale":"2024-09-20T22:47:38.283586+0000","last_undegraded":"2024-09-20T22:47:38.283586+0000","last_fullsized":"2024-09-20T22:47:38.283586+0000","mapping_epoch":290,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:46:00.879802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:35.492562+0000","last_clean_scrub_stamp":"2024-09-20T22:46:00.879802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":654,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-20T22:47:38.189456+0000","last_change":"2024-09-20T22:47:36.658732+0000","last_active":"2024-09-20T22:47:38.189456+0000","last_peered":"2024-09-20T22:47:38.189456+0000","last_clean":"2024-09-20T22:47:38.189456+0000","last_became_active":"2024-09-20T22:45:40.208186+0000","last_became_peered":"2024-09-20T22:45:40.208186+0000","last_unstale":"2024-09-20T22:47:38.189456+0000","last_undegraded":"2024-09-20T22:47:38.189456+0000","last_fullsized":"2024-09-20T22:47:38.189456+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:47:36.658662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:47:36.658662+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:56.434460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":652,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-20T22:47:38.284468+0000","last_change":"2024-09-20T22:45:58.798673+0000","last_active":"2024-09-20T22:47:38.284468+0000","last_peered":"2024-09-20T22:47:38.284468+0000","last_clean":"2024-09-20T22:47:38.284468+0000","last_became_active":"2024-09-20T22:43:57.430630+0000","last_became_peered":"2024-09-20T22:43:57.430630+0000","last_unstale":"2024-09-20T22:47:38.284468+0000","last_undegraded":"2024-09-20T22:47:38.284468+0000","last_fullsized":"2024-09-20T22:47:38.284468+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:45:58.798603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:45:58.798603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"376'300","reported_seq":1760,"reported_epoch":378,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-20T22:47:38.508194+0000","last_change":"2024-09-20T22:47:37.247943+0000","last_active":"2024-09-20T22:47:38.508194+0000","last_peered":"2024-09-20T22:47:38.508194+0000","last_clean":"2024-09-20T22:47:38.508194+0000","last_became_active":"2024-09-20T22:45:43.170143+0000","last_became_peered":"2024-09-20T22:45:43.170143+0000","last_unstale":"2024-09-20T22:47:38.508194+0000","last_undegraded":"2024-09-20T22:47:38.508194+0000","last_fullsized":"2024-09-20T22:47:38.508194+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"194'90","last_scrub_stamp":"2024-09-20T22:44:34.651203+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:44:34.651203+0000","objects_scrubbed":3,"log_size":300,"log_dups_size":0,"ondisk_log_size":300,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.028646438,"stat_sum":{"num_bytes":8588608,"num_objects":6,"num_object_clones":3,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":207,"num_read_kb":214522,"num_write":118,"num_write_kb":35440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":10116431,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"6b","length":"1"}]},{"pgid":"1.9","version":"77'128","reported_seq":951,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-20T22:47:38.394259+0000","last_change":"2024-09-20T22:45:24.337974+0000","last_active":"2024-09-20T22:47:38.394259+0000","last_peered":"2024-09-20T22:47:38.394259+0000","last_clean":"2024-09-20T22:47:38.394259+0000","last_became_active":"2024-09-20T22:42:52.400353+0000","last_became_peered":"2024-09-20T22:42:52.400353+0000","last_unstale":"2024-09-20T22:47:38.394259+0000","last_undegraded":"2024-09-20T22:47:38.394259+0000","last_fullsized":"2024-09-20T22:47:38.394259+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:45:24.337881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:45:24.337881+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"74'10","reported_seq":506,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-20T22:47:38.507637+0000","last_change":"2024-09-20T22:47:21.092403+0000","last_active":"2024-09-20T22:47:38.507637+0000","last_peered":"2024-09-20T22:47:38.507637+0000","last_clean":"2024-09-20T22:47:38.507637+0000","last_became_active":"2024-09-20T22:43:57.441016+0000","last_became_peered":"2024-09-20T22:43:57.441016+0000","last_unstale":"2024-09-20T22:47:38.507637+0000","last_undegraded":"2024-09-20T22:47:38.507637+0000","last_fullsized":"2024-09-20T22:47:38.507637+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-09-20T22:47:21.092309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:47:21.092309+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:34.540078+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":662,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-20T22:47:38.507297+0000","last_change":"2024-09-20T22:45:57.194785+0000","last_active":"2024-09-20T22:47:38.507297+0000","last_peered":"2024-09-20T22:47:38.507297+0000","last_clean":"2024-09-20T22:47:38.507297+0000","last_became_active":"2024-09-20T22:45:40.228762+0000","last_became_peered":"2024-09-20T22:45:40.228762+0000","last_unstale":"2024-09-20T22:47:38.507297+0000","last_undegraded":"2024-09-20T22:47:38.507297+0000","last_fullsized":"2024-09-20T22:47:38.507297+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:45:57.194715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:45:57.194715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"261'9","reported_seq":732,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-20T22:47:38.284212+0000","last_change":"2024-09-20T22:46:50.974814+0000","last_active":"2024-09-20T22:47:38.284212+0000","last_peered":"2024-09-20T22:47:38.284212+0000","last_clean":"2024-09-20T22:47:38.284212+0000","last_became_active":"2024-09-20T22:41:42.185863+0000","last_became_peered":"2024-09-20T22:41:42.185863+0000","last_unstale":"2024-09-20T22:47:38.284212+0000","last_undegraded":"2024-09-20T22:47:38.284212+0000","last_fullsized":"2024-09-20T22:47:38.284212+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'9","last_scrub_stamp":"2024-09-20T22:46:50.974735+0000","last_deep_scrub":"261'9","last_deep_scrub_stamp":"2024-09-20T22:46:50.974735+0000","last_clean_scrub_stamp":"2024-09-20T22:46:50.974735+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:47:51.558873+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":469,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-20T22:47:38.190968+0000","last_change":"2024-09-20T22:46:24.906585+0000","last_active":"2024-09-20T22:47:38.190968+0000","last_peered":"2024-09-20T22:47:38.190968+0000","last_clean":"2024-09-20T22:47:38.190968+0000","last_became_active":"2024-09-20T22:43:57.430067+0000","last_became_peered":"2024-09-20T22:43:57.430067+0000","last_unstale":"2024-09-20T22:47:38.190968+0000","last_undegraded":"2024-09-20T22:47:38.190968+0000","last_fullsized":"2024-09-20T22:47:38.190968+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:46:24.906516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:46:24.906516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:47:54.740972+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.8","version":"377'107","reported_seq":1545,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-20T22:47:38.283256+0000","last_change":"2024-09-20T22:47:37.247732+0000","last_active":"2024-09-20T22:47:38.283256+0000","last_peered":"2024-09-20T22:47:38.283256+0000","last_clean":"2024-09-20T22:47:38.283256+0000","last_became_active":"2024-09-20T22:43:57.431081+0000","last_became_peered":"2024-09-20T22:43:57.431081+0000","last_unstale":"2024-09-20T22:47:38.283256+0000","last_undegraded":"2024-09-20T22:47:38.283256+0000","last_fullsized":"2024-09-20T22:47:38.283256+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'98","last_scrub_stamp":"2024-09-20T22:46:45.991157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:46:45.991157+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:09.521829+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0019299639999999999,"stat_sum":{"num_bytes":3755447,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":25,"num_read_kb":20437,"num_write":21,"num_write_kb":11116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"6b","length":"1"},{"start":"74","length":"1"},{"start":"7a","length":"1"},{"start":"84","length":"1"}]},{"pgid":"4.b","version":"74'9","reported_seq":290,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-20T22:47:38.189388+0000","last_change":"2024-09-20T22:45:40.256230+0000","last_active":"2024-09-20T22:47:38.189388+0000","last_peered":"2024-09-20T22:47:38.189388+0000","last_clean":"2024-09-20T22:47:38.189388+0000","last_became_active":"2024-09-20T22:45:40.222847+0000","last_became_peered":"2024-09-20T22:45:40.222847+0000","last_unstale":"2024-09-20T22:47:38.189388+0000","last_undegraded":"2024-09-20T22:47:38.189388+0000","last_fullsized":"2024-09-20T22:47:38.189388+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-09-20T22:44:57.603671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:44:57.603671+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,5],"acting":[1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":651,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-20T22:47:38.221328+0000","last_change":"2024-09-20T22:46:25.141094+0000","last_active":"2024-09-20T22:47:38.221328+0000","last_peered":"2024-09-20T22:47:38.221328+0000","last_clean":"2024-09-20T22:47:38.221328+0000","last_became_active":"2024-09-20T22:41:38.155953+0000","last_became_peered":"2024-09-20T22:41:38.155953+0000","last_unstale":"2024-09-20T22:47:38.221328+0000","last_undegraded":"2024-09-20T22:47:38.221328+0000","last_fullsized":"2024-09-20T22:47:38.221328+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:46:25.141028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:46:25.141028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:47:38.610184+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":662,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-20T22:47:38.283748+0000","last_change":"2024-09-20T22:46:49.061935+0000","last_active":"2024-09-20T22:47:38.283748+0000","last_peered":"2024-09-20T22:47:38.283748+0000","last_clean":"2024-09-20T22:47:38.283748+0000","last_became_active":"2024-09-20T22:45:40.212560+0000","last_became_peered":"2024-09-20T22:45:40.212560+0000","last_unstale":"2024-09-20T22:47:38.283748+0000","last_undegraded":"2024-09-20T22:47:38.283748+0000","last_fullsized":"2024-09-20T22:47:38.283748+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:46:49.061850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:46:49.061850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:16.726402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":254,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-20T22:47:38.394428+0000","last_change":"2024-09-20T22:46:51.469673+0000","last_active":"2024-09-20T22:47:38.394428+0000","last_peered":"2024-09-20T22:47:38.394428+0000","last_clean":"2024-09-20T22:47:38.394428+0000","last_became_active":"2024-09-20T22:45:40.221355+0000","last_became_peered":"2024-09-20T22:45:40.221355+0000","last_unstale":"2024-09-20T22:47:38.394428+0000","last_undegraded":"2024-09-20T22:47:38.394428+0000","last_fullsized":"2024-09-20T22:47:38.394428+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:46:51.469567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:46:51.469567+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:07.052829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.9","version":"376'224","reported_seq":1683,"reported_epoch":378,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T22:47:38.188937+0000","last_change":"2024-09-20T22:47:37.203427+0000","last_active":"2024-09-20T22:47:38.188937+0000","last_peered":"2024-09-20T22:47:38.188937+0000","last_clean":"2024-09-20T22:47:38.188937+0000","last_became_active":"2024-09-20T22:45:43.166728+0000","last_became_peered":"2024-09-20T22:45:43.166728+0000","last_unstale":"2024-09-20T22:47:38.188937+0000","last_undegraded":"2024-09-20T22:47:38.188937+0000","last_fullsized":"2024-09-20T22:47:38.188937+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":0,"last_scrub":"257'132","last_scrub_stamp":"2024-09-20T22:45:36.793712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:45:36.793712+0000","objects_scrubbed":6,"log_size":224,"log_dups_size":0,"ondisk_log_size":224,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0019813550000000002,"stat_sum":{"num_bytes":14121291,"num_objects":7,"num_object_clones":5,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":79,"num_read_kb":87324,"num_write":69,"num_write_kb":26957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":13891316,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,6],"acting":[1,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"74","length":"1"}]},{"pgid":"4.a","version":"113'28","reported_seq":694,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-20T22:47:38.287000+0000","last_change":"2024-09-20T22:47:13.842452+0000","last_active":"2024-09-20T22:47:38.287000+0000","last_peered":"2024-09-20T22:47:38.287000+0000","last_clean":"2024-09-20T22:47:38.287000+0000","last_became_active":"2024-09-20T22:41:40.252011+0000","last_became_peered":"2024-09-20T22:41:40.252011+0000","last_unstale":"2024-09-20T22:47:38.287000+0000","last_undegraded":"2024-09-20T22:47:38.287000+0000","last_fullsized":"2024-09-20T22:47:38.287000+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'28","last_scrub_stamp":"2024-09-20T22:47:13.842316+0000","last_deep_scrub":"113'28","last_deep_scrub_stamp":"2024-09-20T22:43:12.725900+0000","last_clean_scrub_stamp":"2024-09-20T22:47:13.842316+0000","objects_scrubbed":9,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:31.564433+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":651,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-20T22:47:38.395892+0000","last_change":"2024-09-20T22:46:30.430426+0000","last_active":"2024-09-20T22:47:38.395892+0000","last_peered":"2024-09-20T22:47:38.395892+0000","last_clean":"2024-09-20T22:47:38.395892+0000","last_became_active":"2024-09-20T22:41:38.161524+0000","last_became_peered":"2024-09-20T22:41:38.161524+0000","last_unstale":"2024-09-20T22:47:38.395892+0000","last_undegraded":"2024-09-20T22:47:38.395892+0000","last_fullsized":"2024-09-20T22:47:38.395892+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:46:30.430353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:46:30.430353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:47:53.016290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":656,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-20T22:47:38.283857+0000","last_change":"2024-09-20T22:47:36.012227+0000","last_active":"2024-09-20T22:47:38.283857+0000","last_peered":"2024-09-20T22:47:38.283857+0000","last_clean":"2024-09-20T22:47:38.283857+0000","last_became_active":"2024-09-20T22:45:40.214075+0000","last_became_peered":"2024-09-20T22:45:40.214075+0000","last_unstale":"2024-09-20T22:47:38.283857+0000","last_undegraded":"2024-09-20T22:47:38.283857+0000","last_fullsized":"2024-09-20T22:47:38.283857+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:47:36.012147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:47:36.012147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:48.377187+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":647,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-20T22:47:38.394270+0000","last_change":"2024-09-20T22:47:18.329286+0000","last_active":"2024-09-20T22:47:38.394270+0000","last_peered":"2024-09-20T22:47:38.394270+0000","last_clean":"2024-09-20T22:47:38.394270+0000","last_became_active":"2024-09-20T22:41:44.199395+0000","last_became_peered":"2024-09-20T22:41:44.199395+0000","last_unstale":"2024-09-20T22:47:38.394270+0000","last_undegraded":"2024-09-20T22:47:38.394270+0000","last_fullsized":"2024-09-20T22:47:38.394270+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:47:18.329209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:47:18.329209+0000","last_clean_scrub_stamp":"2024-09-20T22:47:18.329209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:22.373197+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.a","version":"374'300","reported_seq":1740,"reported_epoch":378,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-20T22:47:38.506822+0000","last_change":"2024-09-20T22:47:37.215133+0000","last_active":"2024-09-20T22:47:38.506822+0000","last_peered":"2024-09-20T22:47:38.506822+0000","last_clean":"2024-09-20T22:47:38.506822+0000","last_became_active":"2024-09-20T22:42:52.405085+0000","last_became_peered":"2024-09-20T22:42:52.405085+0000","last_unstale":"2024-09-20T22:47:38.506822+0000","last_undegraded":"2024-09-20T22:47:38.506822+0000","last_fullsized":"2024-09-20T22:47:38.506822+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"301'219","last_scrub_stamp":"2024-09-20T22:46:23.260041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:46:23.260041+0000","objects_scrubbed":3,"log_size":300,"log_dups_size":0,"ondisk_log_size":300,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:47:52.517002+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":1.006227196,"stat_sum":{"num_bytes":19215419,"num_objects":8,"num_object_clones":5,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":240,"num_read_kb":275547,"num_write":133,"num_write_kb":48041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"6b","length":"1"}]},{"pgid":"4.9","version":"74'12","reported_seq":514,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-20T22:47:38.189040+0000","last_change":"2024-09-20T22:44:53.049195+0000","last_active":"2024-09-20T22:47:38.189040+0000","last_peered":"2024-09-20T22:47:38.189040+0000","last_clean":"2024-09-20T22:47:38.189040+0000","last_became_active":"2024-09-20T22:43:57.442218+0000","last_became_peered":"2024-09-20T22:43:57.442218+0000","last_unstale":"2024-09-20T22:47:38.189040+0000","last_undegraded":"2024-09-20T22:47:38.189040+0000","last_fullsized":"2024-09-20T22:47:38.189040+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-09-20T22:44:53.049111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:44:53.049111+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":21,"num_read_kb":12,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":671,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-20T22:47:38.394733+0000","last_change":"2024-09-20T22:46:52.466011+0000","last_active":"2024-09-20T22:47:38.394733+0000","last_peered":"2024-09-20T22:47:38.394733+0000","last_clean":"2024-09-20T22:47:38.394733+0000","last_became_active":"2024-09-20T22:45:40.212393+0000","last_became_peered":"2024-09-20T22:45:40.212393+0000","last_unstale":"2024-09-20T22:47:38.394733+0000","last_undegraded":"2024-09-20T22:47:38.394733+0000","last_fullsized":"2024-09-20T22:47:38.394733+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:46:52.465940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:46:52.465940+0000","last_clean_scrub_stamp":"2024-09-20T22:46:52.465940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:16.208150+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":656,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-20T22:47:38.283363+0000","last_change":"2024-09-20T22:46:01.828155+0000","last_active":"2024-09-20T22:47:38.283363+0000","last_peered":"2024-09-20T22:47:38.283363+0000","last_clean":"2024-09-20T22:47:38.283363+0000","last_became_active":"2024-09-20T22:45:40.205381+0000","last_became_peered":"2024-09-20T22:45:40.205381+0000","last_unstale":"2024-09-20T22:47:38.283363+0000","last_undegraded":"2024-09-20T22:47:38.283363+0000","last_fullsized":"2024-09-20T22:47:38.283363+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:46:01.828080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:46:01.828080+0000","last_clean_scrub_stamp":"2024-09-20T22:46:01.828080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":638,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-20T22:47:38.507097+0000","last_change":"2024-09-20T22:47:38.019602+0000","last_active":"2024-09-20T22:47:38.507097+0000","last_peered":"2024-09-20T22:47:38.507097+0000","last_clean":"2024-09-20T22:47:38.507097+0000","last_became_active":"2024-09-20T22:41:44.192609+0000","last_became_peered":"2024-09-20T22:41:44.192609+0000","last_unstale":"2024-09-20T22:47:38.507097+0000","last_undegraded":"2024-09-20T22:47:38.507097+0000","last_fullsized":"2024-09-20T22:47:38.507097+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:47:38.019533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:55.170379+0000","last_clean_scrub_stamp":"2024-09-20T22:47:38.019533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:49:06.773525+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"375'441","reported_seq":1884,"reported_epoch":378,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-20T22:47:38.189746+0000","last_change":"2024-09-20T22:47:37.203405+0000","last_active":"2024-09-20T22:47:38.189746+0000","last_peered":"2024-09-20T22:47:38.189746+0000","last_clean":"2024-09-20T22:47:38.189746+0000","last_became_active":"2024-09-20T22:42:52.409642+0000","last_became_peered":"2024-09-20T22:42:52.409642+0000","last_unstale":"2024-09-20T22:47:38.189746+0000","last_undegraded":"2024-09-20T22:47:38.189746+0000","last_fullsized":"2024-09-20T22:47:38.189746+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"194'116","last_scrub_stamp":"2024-09-20T22:44:37.097674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:44:37.097674+0000","objects_scrubbed":3,"log_size":441,"log_dups_size":0,"ondisk_log_size":441,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.98383390100000001,"stat_sum":{"num_bytes":27855841,"num_objects":14,"num_object_clones":10,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":281,"num_read_kb":284687,"num_write":208,"num_write_kb":68549,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"6b","length":"1"}]},{"pgid":"4.8","version":"74'15","reported_seq":672,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-20T22:47:38.220714+0000","last_change":"2024-09-20T22:47:35.313505+0000","last_active":"2024-09-20T22:47:38.220714+0000","last_peered":"2024-09-20T22:47:38.220714+0000","last_clean":"2024-09-20T22:47:38.220714+0000","last_became_active":"2024-09-20T22:41:40.252113+0000","last_became_peered":"2024-09-20T22:41:40.252113+0000","last_unstale":"2024-09-20T22:47:38.220714+0000","last_undegraded":"2024-09-20T22:47:38.220714+0000","last_fullsized":"2024-09-20T22:47:38.220714+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-09-20T22:47:35.313413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:47:35.313413+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:52.070278+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'2","reported_seq":687,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-20T22:47:38.395479+0000","last_change":"2024-09-20T22:46:48.473576+0000","last_active":"2024-09-20T22:47:38.395479+0000","last_peered":"2024-09-20T22:47:38.395479+0000","last_clean":"2024-09-20T22:47:38.395479+0000","last_became_active":"2024-09-20T22:45:40.213804+0000","last_became_peered":"2024-09-20T22:45:40.213804+0000","last_unstale":"2024-09-20T22:47:38.395479+0000","last_undegraded":"2024-09-20T22:47:38.395479+0000","last_fullsized":"2024-09-20T22:47:38.395479+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-09-20T22:46:48.473494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:46:48.473494+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:12.933141+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"159'9","reported_seq":742,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-20T22:47:38.394516+0000","last_change":"2024-09-20T22:47:37.382293+0000","last_active":"2024-09-20T22:47:38.394516+0000","last_peered":"2024-09-20T22:47:38.394516+0000","last_clean":"2024-09-20T22:47:38.394516+0000","last_became_active":"2024-09-20T22:43:57.420663+0000","last_became_peered":"2024-09-20T22:43:57.420663+0000","last_unstale":"2024-09-20T22:47:38.394516+0000","last_undegraded":"2024-09-20T22:47:38.394516+0000","last_fullsized":"2024-09-20T22:47:38.394516+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"159'9","last_scrub_stamp":"2024-09-20T22:47:37.382220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:47:37.382220+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:54.564884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":637,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-20T22:47:38.220790+0000","last_change":"2024-09-20T22:45:40.217012+0000","last_active":"2024-09-20T22:47:38.220790+0000","last_peered":"2024-09-20T22:47:38.220790+0000","last_clean":"2024-09-20T22:47:38.220790+0000","last_became_active":"2024-09-20T22:45:40.216593+0000","last_became_peered":"2024-09-20T22:45:40.216593+0000","last_unstale":"2024-09-20T22:47:38.220790+0000","last_undegraded":"2024-09-20T22:47:38.220790+0000","last_fullsized":"2024-09-20T22:47:38.220790+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:45:01.616426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:45:01.616426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.14","version":"373'110","reported_seq":1282,"reported_epoch":378,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-20T22:47:38.508129+0000","last_change":"2024-09-20T22:47:35.245149+0000","last_active":"2024-09-20T22:47:38.508129+0000","last_peered":"2024-09-20T22:47:38.508129+0000","last_clean":"2024-09-20T22:47:38.508129+0000","last_became_active":"2024-09-20T22:43:39.132459+0000","last_became_peered":"2024-09-20T22:43:39.132459+0000","last_unstale":"2024-09-20T22:47:38.508129+0000","last_undegraded":"2024-09-20T22:47:38.508129+0000","last_fullsized":"2024-09-20T22:47:38.508129+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":5,"last_scrub":"147'38","last_scrub_stamp":"2024-09-20T22:43:49.134297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:43:49.134297+0000","objects_scrubbed":0,"log_size":110,"log_dups_size":0,"ondisk_log_size":110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.029112550000000001,"stat_sum":{"num_bytes":5876815,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":44,"num_read_kb":36718,"num_write":37,"num_write_kb":10623,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"6b","length":"1"}]},{"pgid":"3.10","version":"0'0","reported_seq":664,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-20T22:47:38.286268+0000","last_change":"2024-09-20T22:47:12.820395+0000","last_active":"2024-09-20T22:47:38.286268+0000","last_peered":"2024-09-20T22:47:38.286268+0000","last_clean":"2024-09-20T22:47:38.286268+0000","last_became_active":"2024-09-20T22:45:40.212409+0000","last_became_peered":"2024-09-20T22:45:40.212409+0000","last_unstale":"2024-09-20T22:47:38.286268+0000","last_undegraded":"2024-09-20T22:47:38.286268+0000","last_fullsized":"2024-09-20T22:47:38.286268+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:47:12.820304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:47:12.820304+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:28.477165+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"74'6","reported_seq":669,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-20T22:47:38.508404+0000","last_change":"2024-09-20T22:45:39.079425+0000","last_active":"2024-09-20T22:47:38.508404+0000","last_peered":"2024-09-20T22:47:38.508404+0000","last_clean":"2024-09-20T22:47:38.508404+0000","last_became_active":"2024-09-20T22:43:09.116887+0000","last_became_peered":"2024-09-20T22:43:09.116887+0000","last_unstale":"2024-09-20T22:47:38.508404+0000","last_undegraded":"2024-09-20T22:47:38.508404+0000","last_fullsized":"2024-09-20T22:47:38.508404+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'6","last_scrub_stamp":"2024-09-20T22:45:39.079346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:45:39.079346+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":643,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-20T22:47:38.221810+0000","last_change":"2024-09-20T22:45:54.949745+0000","last_active":"2024-09-20T22:47:38.221810+0000","last_peered":"2024-09-20T22:47:38.221810+0000","last_clean":"2024-09-20T22:47:38.221810+0000","last_became_active":"2024-09-20T22:41:42.184930+0000","last_became_peered":"2024-09-20T22:41:42.184930+0000","last_unstale":"2024-09-20T22:47:38.221810+0000","last_undegraded":"2024-09-20T22:47:38.221810+0000","last_fullsized":"2024-09-20T22:47:38.221810+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:45:54.949661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:09.705632+0000","last_clean_scrub_stamp":"2024-09-20T22:45:54.949661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"74'1","reported_seq":650,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-20T22:47:38.395568+0000","last_change":"2024-09-20T22:45:59.486156+0000","last_active":"2024-09-20T22:47:38.395568+0000","last_peered":"2024-09-20T22:47:38.395568+0000","last_clean":"2024-09-20T22:47:38.395568+0000","last_became_active":"2024-09-20T22:43:57.419681+0000","last_became_peered":"2024-09-20T22:43:57.419681+0000","last_unstale":"2024-09-20T22:47:38.395568+0000","last_undegraded":"2024-09-20T22:47:38.395568+0000","last_fullsized":"2024-09-20T22:47:38.395568+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-09-20T22:45:59.486075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:45:59.486075+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.15","version":"98'3","reported_seq":1424,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-20T22:47:38.395068+0000","last_change":"2024-09-20T22:47:37.249775+0000","last_active":"2024-09-20T22:47:38.395068+0000","last_peered":"2024-09-20T22:47:38.395068+0000","last_clean":"2024-09-20T22:47:38.395068+0000","last_became_active":"2024-09-20T22:44:09.174719+0000","last_became_peered":"2024-09-20T22:44:09.174719+0000","last_unstale":"2024-09-20T22:47:38.395068+0000","last_undegraded":"2024-09-20T22:47:38.395068+0000","last_fullsized":"2024-09-20T22:47:38.395068+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":5,"last_scrub":"98'3","last_scrub_stamp":"2024-09-20T22:47:35.357009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:47:35.357009+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:52.223276+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00086515400000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"6b","length":"1"},{"start":"74","length":"1"},{"start":"7a","length":"1"},{"start":"84","length":"1"}]},{"pgid":"4.16","version":"74'9","reported_seq":298,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-20T22:47:38.508031+0000","last_change":"2024-09-20T22:46:21.269606+0000","last_active":"2024-09-20T22:47:38.508031+0000","last_peered":"2024-09-20T22:47:38.508031+0000","last_clean":"2024-09-20T22:47:38.508031+0000","last_became_active":"2024-09-20T22:45:40.218216+0000","last_became_peered":"2024-09-20T22:45:40.218216+0000","last_unstale":"2024-09-20T22:47:38.508031+0000","last_undegraded":"2024-09-20T22:47:38.508031+0000","last_fullsized":"2024-09-20T22:47:38.508031+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-09-20T22:46:21.269487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:46:21.269487+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:47:42.406334+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":663,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-20T22:47:38.394071+0000","last_change":"2024-09-20T22:46:46.438664+0000","last_active":"2024-09-20T22:47:38.394071+0000","last_peered":"2024-09-20T22:47:38.394071+0000","last_clean":"2024-09-20T22:47:38.394071+0000","last_became_active":"2024-09-20T22:45:40.215995+0000","last_became_peered":"2024-09-20T22:45:40.215995+0000","last_unstale":"2024-09-20T22:47:38.394071+0000","last_undegraded":"2024-09-20T22:47:38.394071+0000","last_fullsized":"2024-09-20T22:47:38.394071+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:46:46.438556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:46:46.438556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:47:47.974917+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":642,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-20T22:47:38.508289+0000","last_change":"2024-09-20T22:46:48.012442+0000","last_active":"2024-09-20T22:47:38.508289+0000","last_peered":"2024-09-20T22:47:38.508289+0000","last_clean":"2024-09-20T22:47:38.508289+0000","last_became_active":"2024-09-20T22:41:42.187706+0000","last_became_peered":"2024-09-20T22:41:42.187706+0000","last_unstale":"2024-09-20T22:47:38.508289+0000","last_undegraded":"2024-09-20T22:47:38.508289+0000","last_fullsized":"2024-09-20T22:47:38.508289+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:46:48.012372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:46:48.012372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:11.078815+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":658,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-20T22:47:38.283940+0000","last_change":"2024-09-20T22:46:29.870305+0000","last_active":"2024-09-20T22:47:38.283940+0000","last_peered":"2024-09-20T22:47:38.283940+0000","last_clean":"2024-09-20T22:47:38.283940+0000","last_became_active":"2024-09-20T22:45:40.217798+0000","last_became_peered":"2024-09-20T22:45:40.217798+0000","last_unstale":"2024-09-20T22:47:38.283940+0000","last_undegraded":"2024-09-20T22:47:38.283940+0000","last_fullsized":"2024-09-20T22:47:38.283940+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:46:29.870228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:46:29.870228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:47:59.415198+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.16","version":"377'125","reported_seq":1480,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-20T22:47:38.395695+0000","last_change":"2024-09-20T22:47:37.254676+0000","last_active":"2024-09-20T22:47:38.395695+0000","last_peered":"2024-09-20T22:47:38.395695+0000","last_clean":"2024-09-20T22:47:38.395695+0000","last_became_active":"2024-09-20T22:45:40.220472+0000","last_became_peered":"2024-09-20T22:45:40.220472+0000","last_unstale":"2024-09-20T22:47:38.395695+0000","last_undegraded":"2024-09-20T22:47:38.395695+0000","last_fullsized":"2024-09-20T22:47:38.395695+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":5,"last_scrub":"282'91","last_scrub_stamp":"2024-09-20T22:46:05.393044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:46:05.393044+0000","objects_scrubbed":1,"log_size":125,"log_dups_size":0,"ondisk_log_size":125,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0050037340000000001,"stat_sum":{"num_bytes":3647185,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":68,"num_read_kb":74958,"num_write":29,"num_write_kb":10963,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":3271548,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"6b","length":"1"},{"start":"74","length":"1"},{"start":"7a","length":"1"},{"start":"84","length":"1"}]},{"pgid":"4.15","version":"74'9","reported_seq":663,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-20T22:47:38.221480+0000","last_change":"2024-09-20T22:46:27.123331+0000","last_active":"2024-09-20T22:47:38.221480+0000","last_peered":"2024-09-20T22:47:38.221480+0000","last_clean":"2024-09-20T22:47:38.221480+0000","last_became_active":"2024-09-20T22:41:40.252486+0000","last_became_peered":"2024-09-20T22:41:40.252486+0000","last_unstale":"2024-09-20T22:47:38.221480+0000","last_undegraded":"2024-09-20T22:47:38.221480+0000","last_fullsized":"2024-09-20T22:47:38.221480+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-09-20T22:46:27.123248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:46:27.123248+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:47:50.439941+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":246,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-20T22:47:38.394527+0000","last_change":"2024-09-20T22:45:40.212127+0000","last_active":"2024-09-20T22:47:38.394527+0000","last_peered":"2024-09-20T22:47:38.394527+0000","last_clean":"2024-09-20T22:47:38.394527+0000","last_became_active":"2024-09-20T22:45:40.211736+0000","last_became_peered":"2024-09-20T22:45:40.211736+0000","last_unstale":"2024-09-20T22:47:38.394527+0000","last_undegraded":"2024-09-20T22:47:38.394527+0000","last_fullsized":"2024-09-20T22:47:38.394527+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:45:23.789053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:45:23.789053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.14","version":"74'8","reported_seq":715,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-20T22:47:38.507791+0000","last_change":"2024-09-20T22:43:52.190172+0000","last_active":"2024-09-20T22:47:38.507791+0000","last_peered":"2024-09-20T22:47:38.507791+0000","last_clean":"2024-09-20T22:47:38.507791+0000","last_became_active":"2024-09-20T22:41:42.186352+0000","last_became_peered":"2024-09-20T22:41:42.186352+0000","last_unstale":"2024-09-20T22:47:38.507791+0000","last_undegraded":"2024-09-20T22:47:38.507791+0000","last_fullsized":"2024-09-20T22:47:38.507791+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-09-20T22:43:52.190097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:43:52.190097+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":484,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-20T22:47:38.284480+0000","last_change":"2024-09-20T22:47:19.828610+0000","last_active":"2024-09-20T22:47:38.284480+0000","last_peered":"2024-09-20T22:47:38.284480+0000","last_clean":"2024-09-20T22:47:38.284480+0000","last_became_active":"2024-09-20T22:45:40.214556+0000","last_became_peered":"2024-09-20T22:45:40.214556+0000","last_unstale":"2024-09-20T22:47:38.284480+0000","last_undegraded":"2024-09-20T22:47:38.284480+0000","last_fullsized":"2024-09-20T22:47:38.284480+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:47:19.828541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:47:19.828541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:26.698779+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.17","version":"377'327","reported_seq":1477,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-20T22:47:38.395281+0000","last_change":"2024-09-20T22:47:37.252332+0000","last_active":"2024-09-20T22:47:38.395281+0000","last_peered":"2024-09-20T22:47:38.395281+0000","last_clean":"2024-09-20T22:47:38.395281+0000","last_became_active":"2024-09-20T22:43:39.130801+0000","last_became_peered":"2024-09-20T22:43:39.130801+0000","last_unstale":"2024-09-20T22:47:38.395281+0000","last_undegraded":"2024-09-20T22:47:38.395281+0000","last_fullsized":"2024-09-20T22:47:38.395281+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":5,"last_scrub":"270'229","last_scrub_stamp":"2024-09-20T22:45:53.389694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:45:53.389694+0000","objects_scrubbed":5,"log_size":327,"log_dups_size":0,"ondisk_log_size":327,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0035181349999999999,"stat_sum":{"num_bytes":13720013,"num_objects":8,"num_object_clones":6,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":1,"num_read":132,"num_read_kb":145313,"num_write":76,"num_write_kb":28153,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"6b","length":"1"},{"start":"74","length":"1"},{"start":"7a","length":"1"},{"start":"84","length":"1"}]},{"pgid":"4.14","version":"74'10","reported_seq":661,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-20T22:47:38.507427+0000","last_change":"2024-09-20T22:46:24.269736+0000","last_active":"2024-09-20T22:47:38.507427+0000","last_peered":"2024-09-20T22:47:38.507427+0000","last_clean":"2024-09-20T22:47:38.507427+0000","last_became_active":"2024-09-20T22:41:40.253614+0000","last_became_peered":"2024-09-20T22:41:40.253614+0000","last_unstale":"2024-09-20T22:47:38.507427+0000","last_undegraded":"2024-09-20T22:47:38.507427+0000","last_fullsized":"2024-09-20T22:47:38.507427+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-09-20T22:46:24.269655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:46:24.269655+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":657,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-20T22:47:38.394813+0000","last_change":"2024-09-20T22:45:39.355210+0000","last_active":"2024-09-20T22:47:38.394813+0000","last_peered":"2024-09-20T22:47:38.394813+0000","last_clean":"2024-09-20T22:47:38.394813+0000","last_became_active":"2024-09-20T22:43:57.425004+0000","last_became_peered":"2024-09-20T22:43:57.425004+0000","last_unstale":"2024-09-20T22:47:38.394813+0000","last_undegraded":"2024-09-20T22:47:38.394813+0000","last_fullsized":"2024-09-20T22:47:38.394813+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:45:39.355129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:45:39.355129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"264'9","reported_seq":737,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-20T22:47:38.221374+0000","last_change":"2024-09-20T22:45:40.260046+0000","last_active":"2024-09-20T22:47:38.221374+0000","last_peered":"2024-09-20T22:47:38.221374+0000","last_clean":"2024-09-20T22:47:38.221374+0000","last_became_active":"2024-09-20T22:45:40.224430+0000","last_became_peered":"2024-09-20T22:45:40.224430+0000","last_unstale":"2024-09-20T22:47:38.221374+0000","last_undegraded":"2024-09-20T22:47:38.221374+0000","last_fullsized":"2024-09-20T22:47:38.221374+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-09-20T22:45:19.708971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:45:19.708971+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":246,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-20T22:47:38.395520+0000","last_change":"2024-09-20T22:45:40.218103+0000","last_active":"2024-09-20T22:47:38.395520+0000","last_peered":"2024-09-20T22:47:38.395520+0000","last_clean":"2024-09-20T22:47:38.395520+0000","last_became_active":"2024-09-20T22:45:40.217739+0000","last_became_peered":"2024-09-20T22:45:40.217739+0000","last_unstale":"2024-09-20T22:47:38.395520+0000","last_undegraded":"2024-09-20T22:47:38.395520+0000","last_fullsized":"2024-09-20T22:47:38.395520+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:45:00.729995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:45:00.729995+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.10","version":"376'112","reported_seq":1504,"reported_epoch":378,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T22:47:38.506508+0000","last_change":"2024-09-20T22:47:37.215217+0000","last_active":"2024-09-20T22:47:38.506508+0000","last_peered":"2024-09-20T22:47:38.506508+0000","last_clean":"2024-09-20T22:47:38.506508+0000","last_became_active":"2024-09-20T22:43:39.130391+0000","last_became_peered":"2024-09-20T22:43:39.130391+0000","last_unstale":"2024-09-20T22:47:38.506508+0000","last_undegraded":"2024-09-20T22:47:38.506508+0000","last_fullsized":"2024-09-20T22:47:38.506508+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":5,"last_scrub":"146'43","last_scrub_stamp":"2024-09-20T22:43:46.106059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:43:46.106059+0000","objects_scrubbed":0,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0067757080000000001,"stat_sum":{"num_bytes":6495724,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":47,"num_read_kb":65519,"num_write":27,"num_write_kb":14407,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.13","version":"74'11","reported_seq":510,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-20T22:47:38.286549+0000","last_change":"2024-09-20T22:45:37.940866+0000","last_active":"2024-09-20T22:47:38.286549+0000","last_peered":"2024-09-20T22:47:38.286549+0000","last_clean":"2024-09-20T22:47:38.286549+0000","last_became_active":"2024-09-20T22:43:57.416358+0000","last_became_peered":"2024-09-20T22:43:57.416358+0000","last_unstale":"2024-09-20T22:47:38.286549+0000","last_undegraded":"2024-09-20T22:47:38.286549+0000","last_fullsized":"2024-09-20T22:47:38.286549+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-09-20T22:45:37.940753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:45:37.940753+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":461,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-20T22:47:38.395369+0000","last_change":"2024-09-20T22:46:04.391377+0000","last_active":"2024-09-20T22:47:38.395369+0000","last_peered":"2024-09-20T22:47:38.395369+0000","last_clean":"2024-09-20T22:47:38.395369+0000","last_became_active":"2024-09-20T22:43:57.422556+0000","last_became_peered":"2024-09-20T22:43:57.422556+0000","last_unstale":"2024-09-20T22:47:38.395369+0000","last_undegraded":"2024-09-20T22:47:38.395369+0000","last_fullsized":"2024-09-20T22:47:38.395369+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:46:04.391306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:46:04.391306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":642,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-20T22:47:38.189033+0000","last_change":"2024-09-20T22:46:25.881993+0000","last_active":"2024-09-20T22:47:38.189033+0000","last_peered":"2024-09-20T22:47:38.189033+0000","last_clean":"2024-09-20T22:47:38.189033+0000","last_became_active":"2024-09-20T22:41:42.181736+0000","last_became_peered":"2024-09-20T22:41:42.181736+0000","last_unstale":"2024-09-20T22:47:38.189033+0000","last_undegraded":"2024-09-20T22:47:38.189033+0000","last_fullsized":"2024-09-20T22:47:38.189033+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:46:25.881921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:46:25.881921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":644,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-20T22:47:38.507342+0000","last_change":"2024-09-20T22:47:36.066586+0000","last_active":"2024-09-20T22:47:38.507342+0000","last_peered":"2024-09-20T22:47:38.507342+0000","last_clean":"2024-09-20T22:47:38.507342+0000","last_became_active":"2024-09-20T22:45:40.222560+0000","last_became_peered":"2024-09-20T22:45:40.222560+0000","last_unstale":"2024-09-20T22:47:38.507342+0000","last_undegraded":"2024-09-20T22:47:38.507342+0000","last_fullsized":"2024-09-20T22:47:38.507342+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:47:36.066480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:47:36.066480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:51.026441+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"135'39","reported_seq":1197,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-20T22:47:38.394945+0000","last_change":"2024-09-20T22:47:37.249432+0000","last_active":"2024-09-20T22:47:38.394945+0000","last_peered":"2024-09-20T22:47:38.394945+0000","last_clean":"2024-09-20T22:47:38.394945+0000","last_became_active":"2024-09-20T22:43:39.130210+0000","last_became_peered":"2024-09-20T22:43:39.130210+0000","last_unstale":"2024-09-20T22:47:38.394945+0000","last_undegraded":"2024-09-20T22:47:38.394945+0000","last_fullsized":"2024-09-20T22:47:38.394945+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":5,"last_scrub":"135'39","last_scrub_stamp":"2024-09-20T22:45:35.341125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:45:35.341125+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00052468100000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"6b","length":"1"},{"start":"74","length":"1"},{"start":"7a","length":"1"},{"start":"84","length":"1"}]},{"pgid":"4.12","version":"74'9","reported_seq":662,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-20T22:47:38.189223+0000","last_change":"2024-09-20T22:44:54.989495+0000","last_active":"2024-09-20T22:47:38.189223+0000","last_peered":"2024-09-20T22:47:38.189223+0000","last_clean":"2024-09-20T22:47:38.189223+0000","last_became_active":"2024-09-20T22:41:40.164796+0000","last_became_peered":"2024-09-20T22:41:40.164796+0000","last_unstale":"2024-09-20T22:47:38.189223+0000","last_undegraded":"2024-09-20T22:47:38.189223+0000","last_fullsized":"2024-09-20T22:47:38.189223+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-09-20T22:44:54.989403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:44:54.989403+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":666,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-20T22:47:38.395794+0000","last_change":"2024-09-20T22:45:40.219795+0000","last_active":"2024-09-20T22:47:38.395794+0000","last_peered":"2024-09-20T22:47:38.395794+0000","last_clean":"2024-09-20T22:47:38.395794+0000","last_became_active":"2024-09-20T22:45:40.219482+0000","last_became_peered":"2024-09-20T22:45:40.219482+0000","last_unstale":"2024-09-20T22:47:38.395794+0000","last_undegraded":"2024-09-20T22:47:38.395794+0000","last_fullsized":"2024-09-20T22:47:38.395794+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:45:38.371521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:31.138067+0000","last_clean_scrub_stamp":"2024-09-20T22:45:38.371521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":640,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-20T22:47:38.507134+0000","last_change":"2024-09-20T22:45:40.229442+0000","last_active":"2024-09-20T22:47:38.507134+0000","last_peered":"2024-09-20T22:47:38.507134+0000","last_clean":"2024-09-20T22:47:38.507134+0000","last_became_active":"2024-09-20T22:45:40.228813+0000","last_became_peered":"2024-09-20T22:45:40.228813+0000","last_unstale":"2024-09-20T22:47:38.507134+0000","last_undegraded":"2024-09-20T22:47:38.507134+0000","last_fullsized":"2024-09-20T22:47:38.507134+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:44:15.242881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:44:15.242881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":254,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-20T22:47:38.220589+0000","last_change":"2024-09-20T22:47:37.309942+0000","last_active":"2024-09-20T22:47:38.220589+0000","last_peered":"2024-09-20T22:47:38.220589+0000","last_clean":"2024-09-20T22:47:38.220589+0000","last_became_active":"2024-09-20T22:45:40.224506+0000","last_became_peered":"2024-09-20T22:45:40.224506+0000","last_unstale":"2024-09-20T22:47:38.220589+0000","last_undegraded":"2024-09-20T22:47:38.220589+0000","last_fullsized":"2024-09-20T22:47:38.220589+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:47:37.309842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:47:37.309842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:55.778741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.12","version":"377'359","reported_seq":1604,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-20T22:47:38.286787+0000","last_change":"2024-09-20T22:47:37.257008+0000","last_active":"2024-09-20T22:47:38.286787+0000","last_peered":"2024-09-20T22:47:38.286787+0000","last_clean":"2024-09-20T22:47:38.286787+0000","last_became_active":"2024-09-20T22:45:43.172023+0000","last_became_peered":"2024-09-20T22:45:43.172023+0000","last_unstale":"2024-09-20T22:47:38.286787+0000","last_undegraded":"2024-09-20T22:47:38.286787+0000","last_fullsized":"2024-09-20T22:47:38.286787+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":264,"parent":"0.0","parent_split_bits":5,"last_scrub":"235'221","last_scrub_stamp":"2024-09-20T22:45:15.702470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:45:15.702470+0000","objects_scrubbed":5,"log_size":359,"log_dups_size":0,"ondisk_log_size":359,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0057263050000000001,"stat_sum":{"num_bytes":16803451,"num_objects":9,"num_object_clones":6,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":137,"num_read_kb":130360,"num_write":109,"num_write_kb":37094,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":12258040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"6b","length":"1"},{"start":"74","length":"1"},{"start":"7a","length":"1"},{"start":"84","length":"1"}]},{"pgid":"4.11","version":"74'11","reported_seq":456,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-20T22:47:38.220379+0000","last_change":"2024-09-20T22:46:31.254470+0000","last_active":"2024-09-20T22:47:38.220379+0000","last_peered":"2024-09-20T22:47:38.220379+0000","last_clean":"2024-09-20T22:47:38.220379+0000","last_became_active":"2024-09-20T22:44:09.170785+0000","last_became_peered":"2024-09-20T22:44:09.170785+0000","last_unstale":"2024-09-20T22:47:38.220379+0000","last_undegraded":"2024-09-20T22:47:38.220379+0000","last_fullsized":"2024-09-20T22:47:38.220379+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-09-20T22:46:31.254376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:46:31.254376+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:00.231236+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":651,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-20T22:47:38.221903+0000","last_change":"2024-09-20T22:45:24.765509+0000","last_active":"2024-09-20T22:47:38.221903+0000","last_peered":"2024-09-20T22:47:38.221903+0000","last_clean":"2024-09-20T22:47:38.221903+0000","last_became_active":"2024-09-20T22:41:38.158701+0000","last_became_peered":"2024-09-20T22:41:38.158701+0000","last_unstale":"2024-09-20T22:47:38.221903+0000","last_undegraded":"2024-09-20T22:47:38.221903+0000","last_fullsized":"2024-09-20T22:47:38.221903+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:45:24.765414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:45:24.765414+0000","last_clean_scrub_stamp":"2024-09-20T22:45:24.765414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":655,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-20T22:47:38.394005+0000","last_change":"2024-09-20T22:46:29.466430+0000","last_active":"2024-09-20T22:47:38.394005+0000","last_peered":"2024-09-20T22:47:38.394005+0000","last_clean":"2024-09-20T22:47:38.394005+0000","last_became_active":"2024-09-20T22:45:40.210241+0000","last_became_peered":"2024-09-20T22:45:40.210241+0000","last_unstale":"2024-09-20T22:47:38.394005+0000","last_undegraded":"2024-09-20T22:47:38.394005+0000","last_fullsized":"2024-09-20T22:47:38.394005+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:46:29.466361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:50.174861+0000","last_clean_scrub_stamp":"2024-09-20T22:46:29.466361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":636,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-20T22:47:38.507220+0000","last_change":"2024-09-20T22:45:40.218749+0000","last_active":"2024-09-20T22:47:38.507220+0000","last_peered":"2024-09-20T22:47:38.507220+0000","last_clean":"2024-09-20T22:47:38.507220+0000","last_became_active":"2024-09-20T22:45:40.218262+0000","last_became_peered":"2024-09-20T22:45:40.218262+0000","last_unstale":"2024-09-20T22:47:38.507220+0000","last_undegraded":"2024-09-20T22:47:38.507220+0000","last_fullsized":"2024-09-20T22:47:38.507220+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:45:00.006909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:45:00.006909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"377'74","reported_seq":952,"reported_epoch":378,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T22:47:38.189959+0000","last_change":"2024-09-20T22:47:37.244194+0000","last_active":"2024-09-20T22:47:38.189959+0000","last_peered":"2024-09-20T22:47:38.189959+0000","last_clean":"2024-09-20T22:47:38.189959+0000","last_became_active":"2024-09-20T22:45:40.223628+0000","last_became_peered":"2024-09-20T22:45:40.223628+0000","last_unstale":"2024-09-20T22:47:38.189959+0000","last_undegraded":"2024-09-20T22:47:38.189959+0000","last_fullsized":"2024-09-20T22:47:38.189959+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":5,"last_scrub":"255'21","last_scrub_stamp":"2024-09-20T22:45:35.767459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:45:35.767459+0000","objects_scrubbed":1,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.050228736000000003,"stat_sum":{"num_bytes":5059672,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":32,"num_read_kb":31660,"num_write":19,"num_write_kb":6137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":2020747,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"6b","length":"1"}]},{"pgid":"4.10","version":"74'4","reported_seq":644,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-20T22:47:38.506196+0000","last_change":"2024-09-20T22:44:17.320732+0000","last_active":"2024-09-20T22:47:38.506196+0000","last_peered":"2024-09-20T22:47:38.506196+0000","last_clean":"2024-09-20T22:47:38.506196+0000","last_became_active":"2024-09-20T22:41:40.166123+0000","last_became_peered":"2024-09-20T22:41:40.166123+0000","last_unstale":"2024-09-20T22:47:38.506196+0000","last_undegraded":"2024-09-20T22:47:38.506196+0000","last_fullsized":"2024-09-20T22:47:38.506196+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'4","last_scrub_stamp":"2024-09-20T22:44:17.320608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:44:17.320608+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":246,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-20T22:47:38.222007+0000","last_change":"2024-09-20T22:45:40.209386+0000","last_active":"2024-09-20T22:47:38.222007+0000","last_peered":"2024-09-20T22:47:38.222007+0000","last_clean":"2024-09-20T22:47:38.222007+0000","last_became_active":"2024-09-20T22:45:40.209090+0000","last_became_peered":"2024-09-20T22:45:40.209090+0000","last_unstale":"2024-09-20T22:47:38.222007+0000","last_undegraded":"2024-09-20T22:47:38.222007+0000","last_fullsized":"2024-09-20T22:47:38.222007+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:44:58.632703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:44:58.632703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":662,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-20T22:47:38.287432+0000","last_change":"2024-09-20T22:47:37.826112+0000","last_active":"2024-09-20T22:47:38.287432+0000","last_peered":"2024-09-20T22:47:38.287432+0000","last_clean":"2024-09-20T22:47:38.287432+0000","last_became_active":"2024-09-20T22:45:40.205010+0000","last_became_peered":"2024-09-20T22:45:40.205010+0000","last_unstale":"2024-09-20T22:47:38.287432+0000","last_undegraded":"2024-09-20T22:47:38.287432+0000","last_fullsized":"2024-09-20T22:47:38.287432+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:47:37.826041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:47:37.826041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:57.152641+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":645,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-20T22:47:38.394306+0000","last_change":"2024-09-20T22:45:19.352251+0000","last_active":"2024-09-20T22:47:38.394306+0000","last_peered":"2024-09-20T22:47:38.394306+0000","last_clean":"2024-09-20T22:47:38.394306+0000","last_became_active":"2024-09-20T22:43:57.424654+0000","last_became_peered":"2024-09-20T22:43:57.424654+0000","last_unstale":"2024-09-20T22:47:38.394306+0000","last_undegraded":"2024-09-20T22:47:38.394306+0000","last_fullsized":"2024-09-20T22:47:38.394306+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:45:19.352162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:10.233487+0000","last_clean_scrub_stamp":"2024-09-20T22:45:19.352162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":644,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-20T22:47:38.190651+0000","last_change":"2024-09-20T22:47:19.596358+0000","last_active":"2024-09-20T22:47:38.190651+0000","last_peered":"2024-09-20T22:47:38.190651+0000","last_clean":"2024-09-20T22:47:38.190651+0000","last_became_active":"2024-09-20T22:43:57.430933+0000","last_became_peered":"2024-09-20T22:43:57.430933+0000","last_unstale":"2024-09-20T22:47:38.190651+0000","last_undegraded":"2024-09-20T22:47:38.190651+0000","last_fullsized":"2024-09-20T22:47:38.190651+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:47:19.596232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:47:19.596232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:41.558738+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":662,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-20T22:47:38.506040+0000","last_change":"2024-09-20T22:45:40.221677+0000","last_active":"2024-09-20T22:47:38.506040+0000","last_peered":"2024-09-20T22:47:38.506040+0000","last_clean":"2024-09-20T22:47:38.506040+0000","last_became_active":"2024-09-20T22:45:40.221329+0000","last_became_peered":"2024-09-20T22:45:40.221329+0000","last_unstale":"2024-09-20T22:47:38.506040+0000","last_undegraded":"2024-09-20T22:47:38.506040+0000","last_fullsized":"2024-09-20T22:47:38.506040+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:44:10.267152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:44:10.267152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"74'11","reported_seq":668,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-20T22:47:38.287902+0000","last_change":"2024-09-20T22:45:14.702296+0000","last_active":"2024-09-20T22:47:38.287902+0000","last_peered":"2024-09-20T22:47:38.287902+0000","last_clean":"2024-09-20T22:47:38.287902+0000","last_became_active":"2024-09-20T22:41:40.171256+0000","last_became_peered":"2024-09-20T22:41:40.171256+0000","last_unstale":"2024-09-20T22:47:38.287902+0000","last_undegraded":"2024-09-20T22:47:38.287902+0000","last_fullsized":"2024-09-20T22:47:38.287902+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-09-20T22:45:14.702198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:45:14.702198+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":254,"reported_epoch":378,"state":"active+clean","last_fresh":"2024-09-20T22:47:38.395442+0000","last_change":"2024-09-20T22:46:20.485525+0000","last_active":"2024-09-20T22:47:38.395442+0000","last_peered":"2024-09-20T22:47:38.395442+0000","last_clean":"2024-09-20T22:47:38.395442+0000","last_became_active":"2024-09-20T22:45:40.221497+0000","last_became_peered":"2024-09-20T22:45:40.221497+0000","last_unstale":"2024-09-20T22:47:38.395442+0000","last_undegraded":"2024-09-20T22:47:38.395442+0000","last_fullsized":"2024-09-20T22:47:38.395442+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:46:20.485453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:51.667341+0000","last_clean_scrub_stamp":"2024-09-20T22:46:20.485453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:47:44.676326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":26,"stat_sum":{"num_bytes":276815852,"num_objects":132,"num_object_clones":85,"num_object_copies":396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":132,"num_whiteouts":6,"num_read":2581,"num_read_kb":2684514,"num_write":1727,"num_write_kb":656217,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":152,"num_bytes_recovered":76941931,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":731820032,"data_stored":989275225,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5357,"ondisk_log_size":5357,"up":78,"acting":78,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1261,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74,"ondisk_log_size":74,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":683,"num_read_kb":440,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":159,"num_bytes_recovered":3278,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":524288,"data_stored":14054,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":428,"ondisk_log_size":428,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":5597,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":353,"num_read_kb":348,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":384,"ondisk_log_size":384,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070540,"num_pgs":87,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":134968,"kb_used_data":103780,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93610184,"statfs":{"total":95995035648,"available":95856828416,"internally_reserved":0,"allocated":106270720,"data_stored":143359091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430290,"num_pgs":78,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":159340,"kb_used_data":128144,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93585812,"statfs":{"total":95995035648,"available":95831871488,"internally_reserved":0,"allocated":131219456,"data_stored":177931134,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691927,"num_pgs":88,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":139428,"kb_used_data":95756,"kb_used_omap":1,"kb_used_meta":43646,"kb_avail":93605724,"statfs":{"total":95995035648,"available":95852261376,"internally_reserved":0,"allocated":98054144,"data_stored":140063679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":44693957},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":0,"seq":133143986268,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182498,"num_pgs":89,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":142592,"kb_used_data":115560,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93602560,"statfs":{"total":95995035648,"available":95849021440,"internally_reserved":0,"allocated":118333440,"data_stored":155781017,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411431,"num_pgs":77,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":128164,"kb_used_data":101124,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93616988,"statfs":{"total":95995035648,"available":95863795712,"internally_reserved":0,"allocated":103550976,"data_stored":129899057,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574956,"num_pgs":85,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":135180,"kb_used_data":108144,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93609972,"statfs":{"total":95995035648,"available":95856611328,"internally_reserved":0,"allocated":110739456,"data_stored":144950107,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":34359738481,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":2668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":68685824,"data_stored":96463859,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":108281856,"data_stored":143515557,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":100888576,"data_stored":128202782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":115314688,"data_stored":153755769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":11284480,"data_stored":11212109,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":95031296,"data_stored":138031022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":128806912,"data_stored":176496643,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":103526400,"data_stored":141597484,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-20T22:47:43.551 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-09-20T22:47:43.551 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-20T22:47:43.552 INFO:tasks.thrashosds.thrasher:Adding osd 0 2024-09-20T22:47:43.552 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd in 0 2024-09-20T22:47:43.640 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:47:43.663 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:43.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:43.664 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:47:43.764 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:47:43.785 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:43.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:43.786 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:47:43.887 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:47:43.911 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:43.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:43.911 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:47:44.012 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:47:44.023 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:47:44.036 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:44.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:44.037 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:47:44.138 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:47:44.162 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:44.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:44.163 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:47:44.242 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:47:44.264 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:47:44.265 INFO:tasks.rados.rados.0.smithi084.stdout:1464: done (6 left) 2024-09-20T22:47:44.265 INFO:tasks.rados.rados.0.smithi084.stdout:1465: write oid 16 current snap is 151 2024-09-20T22:47:44.266 INFO:tasks.rados.rados.0.smithi084.stdout:1465: seq_num 521 ranges {756447=716733,1887942=537832} 2024-09-20T22:47:44.274 INFO:tasks.rados.rados.0.smithi084.stdout:1465: writing smithi08477203-16 from 756447 to 1473180 tid 1 2024-09-20T22:47:44.281 INFO:tasks.rados.rados.0.smithi084.stdout:1465: writing smithi08477203-16 from 1887942 to 2425774 tid 2 2024-09-20T22:47:44.281 INFO:tasks.rados.rados.0.smithi084.stdout:1459: finishing write tid 2 to smithi08477203-31 2024-09-20T22:47:44.281 INFO:tasks.rados.rados.0.smithi084.stdout:1459: finishing write tid 3 to smithi08477203-31 2024-09-20T22:47:44.281 INFO:tasks.rados.rados.0.smithi084.stdout:1459: finishing write tid 4 to smithi08477203-31 2024-09-20T22:47:44.281 INFO:tasks.rados.rados.0.smithi084.stdout:1459: oid 31 updating version 0 to 410 2024-09-20T22:47:44.281 INFO:tasks.rados.rados.0.smithi084.stdout:1459: oid 31 version 410 is already newer than 409 2024-09-20T22:47:44.281 INFO:tasks.rados.rados.0.smithi084.stdout:1459: oid 31 version 410 is already newer than 408 2024-09-20T22:47:44.282 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 31 v 410 (ObjNum 518 snap 151 seq_num 518) dirty exists 2024-09-20T22:47:44.282 INFO:tasks.rados.rados.0.smithi084.stdout:1459: left oid 31 (ObjNum 518 snap 151 seq_num 518) 2024-09-20T22:47:44.282 INFO:tasks.rados.rados.0.smithi084.stdout:1458: finishing copy_from to smithi08477203-28 2024-09-20T22:47:44.282 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 28 v 108 (ObjNum 511 snap 145 seq_num 511) dirty exists 2024-09-20T22:47:44.282 INFO:tasks.rados.rados.0.smithi084.stdout:1461: finishing write tid 1 to smithi08477203-29 2024-09-20T22:47:44.282 INFO:tasks.rados.rados.0.smithi084.stdout:1460: expect (ObjNum 489 snap 137 seq_num 489) 2024-09-20T22:47:44.283 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:44 smithi084 ceph-mon[40024]: osdmap e383: 8 total, 8 up, 6 in 2024-09-20T22:47:44.284 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:44 smithi084 ceph-mon[29774]: osdmap e383: 8 total, 8 up, 6 in 2024-09-20T22:47:44.289 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:44.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:44.290 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:47:44.363 INFO:tasks.rados.rados.0.smithi084.stdout:1461: finishing write tid 2 to smithi08477203-29 2024-09-20T22:47:44.363 INFO:tasks.rados.rados.0.smithi084.stdout:1461: finishing write tid 3 to smithi08477203-29 2024-09-20T22:47:44.363 INFO:tasks.rados.rados.0.smithi084.stdout:1461: finishing write tid 4 to smithi08477203-29 2024-09-20T22:47:44.363 INFO:tasks.rados.rados.0.smithi084.stdout:1461: oid 29 updating version 0 to 824 2024-09-20T22:47:44.363 INFO:tasks.rados.rados.0.smithi084.stdout:1461: oid 29 updating version 824 to 825 2024-09-20T22:47:44.363 INFO:tasks.rados.rados.0.smithi084.stdout:1461: oid 29 updating version 825 to 826 2024-09-20T22:47:44.363 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 29 v 826 (ObjNum 519 snap 151 seq_num 519) dirty exists 2024-09-20T22:47:44.363 INFO:tasks.rados.rados.0.smithi084.stdout:1461: left oid 29 (ObjNum 519 snap 151 seq_num 519) 2024-09-20T22:47:44.363 INFO:tasks.rados.rados.0.smithi084.stdout:1462: finishing copy_from racing read to smithi08477203-32 2024-09-20T22:47:44.363 INFO:tasks.rados.rados.0.smithi084.stdout:1463: finishing write tid 1 to smithi08477203-12 2024-09-20T22:47:44.363 INFO:tasks.rados.rados.0.smithi084.stdout:1462: finishing copy_from to smithi08477203-32 2024-09-20T22:47:44.363 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 32 v 131 (ObjNum 466 snap 128 seq_num 466) dirty exists 2024-09-20T22:47:44.364 INFO:tasks.rados.rados.0.smithi084.stdout:1463: finishing write tid 2 to smithi08477203-12 2024-09-20T22:47:44.364 INFO:tasks.rados.rados.0.smithi084.stdout:1458: done (6 left) 2024-09-20T22:47:44.364 INFO:tasks.rados.rados.0.smithi084.stdout:1459: done (5 left) 2024-09-20T22:47:44.364 INFO:tasks.rados.rados.0.smithi084.stdout:1460: done (4 left) 2024-09-20T22:47:44.364 INFO:tasks.rados.rados.0.smithi084.stdout:1461: done (3 left) 2024-09-20T22:47:44.364 INFO:tasks.rados.rados.0.smithi084.stdout:1462: done (2 left) 2024-09-20T22:47:44.364 INFO:tasks.rados.rados.0.smithi084.stdout:1466: delete oid 42 current snap is 151 2024-09-20T22:47:44.364 INFO:tasks.rados.rados.0.smithi084.stdout:1463: finishing write tid 3 to smithi08477203-12 2024-09-20T22:47:44.364 INFO:tasks.rados.rados.0.smithi084.stdout:1463: finishing write tid 4 to smithi08477203-12 2024-09-20T22:47:44.364 INFO:tasks.rados.rados.0.smithi084.stdout:1463: oid 12 updating version 0 to 332 2024-09-20T22:47:44.364 INFO:tasks.rados.rados.0.smithi084.stdout:1463: oid 12 updating version 332 to 334 2024-09-20T22:47:44.364 INFO:tasks.rados.rados.0.smithi084.stdout:1463: oid 12 version 334 is already newer than 333 2024-09-20T22:47:44.364 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 12 v 334 (ObjNum 520 snap 151 seq_num 520) dirty exists 2024-09-20T22:47:44.364 INFO:tasks.rados.rados.0.smithi084.stdout:1463: left oid 12 (ObjNum 520 snap 151 seq_num 520) 2024-09-20T22:47:44.365 INFO:tasks.rados.rados.0.smithi084.stdout:1465: finishing write tid 1 to smithi08477203-16 2024-09-20T22:47:44.365 INFO:tasks.rados.rados.0.smithi084.stdout:1465: finishing write tid 2 to smithi08477203-16 2024-09-20T22:47:44.365 INFO:tasks.rados.rados.0.smithi084.stdout:1465: finishing write tid 3 to smithi08477203-16 2024-09-20T22:47:44.365 INFO:tasks.rados.rados.0.smithi084.stdout:1465: finishing write tid 4 to smithi08477203-16 2024-09-20T22:47:44.365 INFO:tasks.rados.rados.0.smithi084.stdout:1465: oid 16 updating version 0 to 419 2024-09-20T22:47:44.365 INFO:tasks.rados.rados.0.smithi084.stdout:1465: oid 16 version 419 is already newer than 417 2024-09-20T22:47:44.365 INFO:tasks.rados.rados.0.smithi084.stdout:1465: oid 16 version 419 is already newer than 418 2024-09-20T22:47:44.365 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 16 v 419 (ObjNum 521 snap 151 seq_num 521) dirty exists 2024-09-20T22:47:44.365 INFO:tasks.rados.rados.0.smithi084.stdout:1465: left oid 16 (ObjNum 521 snap 151 seq_num 521) 2024-09-20T22:47:44.368 INFO:tasks.rados.rados.0.smithi084.stdout:1463: done (2 left) 2024-09-20T22:47:44.369 INFO:tasks.rados.rados.0.smithi084.stdout:1465: done (1 left) 2024-09-20T22:47:44.369 INFO:tasks.rados.rados.0.smithi084.stdout:1466: done (0 left) 2024-09-20T22:47:44.369 INFO:tasks.rados.rados.0.smithi084.stdout:1467: write oid 29 current snap is 151 2024-09-20T22:47:44.369 INFO:tasks.rados.rados.0.smithi084.stdout:1467: seq_num 523 ranges {790061=467231,2004906=541501} 2024-09-20T22:47:44.375 INFO:tasks.rados.rados.0.smithi084.stdout:1467: writing smithi08477203-29 from 790061 to 1257292 tid 1 2024-09-20T22:47:44.382 INFO:tasks.rados.rados.0.smithi084.stdout:1467: writing smithi08477203-29 from 2004906 to 2546407 tid 2 2024-09-20T22:47:44.382 INFO:tasks.rados.rados.0.smithi084.stdout:1468: read oid 16 snap 138 2024-09-20T22:47:44.382 INFO:tasks.rados.rados.0.smithi084.stdout:1468: expect (ObjNum 419 snap 105 seq_num 419) 2024-09-20T22:47:44.382 INFO:tasks.rados.rados.0.smithi084.stdout:1469: delete oid 22 current snap is 151 2024-09-20T22:47:44.382 INFO:tasks.rados.rados.0.smithi084.stdout:1467: finishing write tid 1 to smithi08477203-29 2024-09-20T22:47:44.384 INFO:tasks.rados.rados.0.smithi084.stdout:1469: done (2 left) 2024-09-20T22:47:44.384 INFO:tasks.rados.rados.0.smithi084.stdout:1470: write oid 39 current snap is 151 2024-09-20T22:47:44.384 INFO:tasks.rados.rados.0.smithi084.stdout:1470: seq_num 525 ranges {423675=734082,1721870=545170,2717777=105898} 2024-09-20T22:47:44.391 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:47:44.392 INFO:tasks.rados.rados.0.smithi084.stdout:1470: writing smithi08477203-39 from 423675 to 1157757 tid 1 2024-09-20T22:47:44.398 INFO:tasks.rados.rados.0.smithi084.stdout:1470: writing smithi08477203-39 from 1721870 to 2267040 tid 2 2024-09-20T22:47:44.401 INFO:tasks.rados.rados.0.smithi084.stdout:1470: writing smithi08477203-39 from 2717777 to 2823675 tid 3 2024-09-20T22:47:44.401 INFO:tasks.rados.rados.0.smithi084.stdout:1471: read oid 14 snap -1 2024-09-20T22:47:44.401 INFO:tasks.rados.rados.0.smithi084.stdout:1471: expect deleted 2024-09-20T22:47:44.401 INFO:tasks.rados.rados.0.smithi084.stdout:1472: write oid 43 current snap is 151 2024-09-20T22:47:44.401 INFO:tasks.rados.rados.0.smithi084.stdout:1467: finishing write tid 2 to smithi08477203-29 2024-09-20T22:47:44.401 INFO:tasks.rados.rados.0.smithi084.stdout:1467: finishing write tid 3 to smithi08477203-29 2024-09-20T22:47:44.402 INFO:tasks.rados.rados.0.smithi084.stdout:1467: finishing write tid 4 to smithi08477203-29 2024-09-20T22:47:44.402 INFO:tasks.rados.rados.0.smithi084.stdout:1467: oid 29 updating version 0 to 832 2024-09-20T22:47:44.402 INFO:tasks.rados.rados.0.smithi084.stdout:1467: oid 29 updating version 832 to 833 2024-09-20T22:47:44.402 INFO:tasks.rados.rados.0.smithi084.stdout:1467: oid 29 version 833 is already newer than 831 2024-09-20T22:47:44.402 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 29 v 833 (ObjNum 523 snap 151 seq_num 523) dirty exists 2024-09-20T22:47:44.402 INFO:tasks.rados.rados.0.smithi084.stdout:1467: left oid 29 (ObjNum 523 snap 151 seq_num 523) 2024-09-20T22:47:44.402 INFO:tasks.rados.rados.0.smithi084.stdout:1468: expect (ObjNum 419 snap 105 seq_num 419) 2024-09-20T22:47:44.414 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:44.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:44.415 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:47:44.515 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:47:44.539 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:44.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:44.540 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:47:44.572 INFO:tasks.rados.rados.0.smithi084.stdout:1470: finishing write tid 1 to smithi08477203-39 2024-09-20T22:47:44.572 INFO:tasks.rados.rados.0.smithi084.stdout:1470: finishing write tid 2 to smithi08477203-39 2024-09-20T22:47:44.572 INFO:tasks.rados.rados.0.smithi084.stdout:1472: seq_num 526 ranges {440482=409331,1380352=405181,2461553=378929} 2024-09-20T22:47:44.576 INFO:tasks.rados.rados.0.smithi084.stdout:1472: writing smithi08477203-43 from 440482 to 849813 tid 1 2024-09-20T22:47:44.581 INFO:tasks.rados.rados.0.smithi084.stdout:1472: writing smithi08477203-43 from 1380352 to 1785533 tid 2 2024-09-20T22:47:44.588 INFO:tasks.rados.rados.0.smithi084.stdout:1472: writing smithi08477203-43 from 2461553 to 2840482 tid 3 2024-09-20T22:47:44.588 INFO:tasks.rados.rados.0.smithi084.stdout:1467: done (4 left) 2024-09-20T22:47:44.588 INFO:tasks.rados.rados.0.smithi084.stdout:1468: done (3 left) 2024-09-20T22:47:44.588 INFO:tasks.rados.rados.0.smithi084.stdout:1473: delete oid 19 current snap is 151 2024-09-20T22:47:44.588 INFO:tasks.rados.rados.0.smithi084.stdout:1470: finishing write tid 3 to smithi08477203-39 2024-09-20T22:47:44.588 INFO:tasks.rados.rados.0.smithi084.stdout:1470: finishing write tid 4 to smithi08477203-39 2024-09-20T22:47:44.588 INFO:tasks.rados.rados.0.smithi084.stdout:1470: finishing write tid 5 to smithi08477203-39 2024-09-20T22:47:44.588 INFO:tasks.rados.rados.0.smithi084.stdout:1470: oid 39 updating version 0 to 526 2024-09-20T22:47:44.588 INFO:tasks.rados.rados.0.smithi084.stdout:1470: oid 39 updating version 526 to 529 2024-09-20T22:47:44.588 INFO:tasks.rados.rados.0.smithi084.stdout:1470: oid 39 version 529 is already newer than 527 2024-09-20T22:47:44.589 INFO:tasks.rados.rados.0.smithi084.stdout:1470: oid 39 version 529 is already newer than 528 2024-09-20T22:47:44.589 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 39 v 529 (ObjNum 525 snap 151 seq_num 525) dirty exists 2024-09-20T22:47:44.589 INFO:tasks.rados.rados.0.smithi084.stdout:1470: left oid 39 (ObjNum 525 snap 151 seq_num 525) 2024-09-20T22:47:44.589 INFO:tasks.rados.rados.0.smithi084.stdout:1472: finishing write tid 1 to smithi08477203-43 2024-09-20T22:47:44.589 INFO:tasks.rados.rados.0.smithi084.stdout:1472: finishing write tid 2 to smithi08477203-43 2024-09-20T22:47:44.590 INFO:tasks.rados.rados.0.smithi084.stdout:1472: finishing write tid 3 to smithi08477203-43 2024-09-20T22:47:44.590 INFO:tasks.rados.rados.0.smithi084.stdout:1472: finishing write tid 4 to smithi08477203-43 2024-09-20T22:47:44.590 INFO:tasks.rados.rados.0.smithi084.stdout:1470: done (3 left) 2024-09-20T22:47:44.590 INFO:tasks.rados.rados.0.smithi084.stdout:1471: done (2 left) 2024-09-20T22:47:44.590 INFO:tasks.rados.rados.0.smithi084.stdout:1473: done (1 left) 2024-09-20T22:47:44.590 INFO:tasks.rados.rados.0.smithi084.stdout:1474: snap_create 2024-09-20T22:47:44.591 INFO:tasks.rados.rados.0.smithi084.stdout:1472: finishing write tid 5 to smithi08477203-43 2024-09-20T22:47:44.591 INFO:tasks.rados.rados.0.smithi084.stdout:1472: oid 43 updating version 0 to 80 2024-09-20T22:47:44.591 INFO:tasks.rados.rados.0.smithi084.stdout:1472: oid 43 version 80 is already newer than 78 2024-09-20T22:47:44.591 INFO:tasks.rados.rados.0.smithi084.stdout:1472: oid 43 version 80 is already newer than 79 2024-09-20T22:47:44.591 INFO:tasks.rados.rados.0.smithi084.stdout:1472: oid 43 version 80 is already newer than 77 2024-09-20T22:47:44.591 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 43 v 80 (ObjNum 526 snap 151 seq_num 526) dirty exists 2024-09-20T22:47:44.591 INFO:tasks.rados.rados.0.smithi084.stdout:1472: left oid 43 (ObjNum 526 snap 151 seq_num 526) 2024-09-20T22:47:44.640 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:47:44.662 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:44.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:44.663 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:47:44.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:44 smithi154 ceph-mon[33264]: osdmap e383: 8 total, 8 up, 6 in 2024-09-20T22:47:44.764 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:47:44.785 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:44.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:44.786 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:47:44.825 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-20T22:47:44.826 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:47:44.887 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:47:44.910 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:44.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:44.911 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:47:45.012 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:47:45.033 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:45.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:45.034 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:47:45.134 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:47:45.156 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:45.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:45.157 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:47:45.258 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:47:45.266 INFO:tasks.rados.rados.0.smithi084.stdout:1472: done (1 left) 2024-09-20T22:47:45.266 INFO:tasks.rados.rados.0.smithi084.stdout:1474: done (0 left) 2024-09-20T22:47:45.266 INFO:tasks.rados.rados.0.smithi084.stdout:1475: read oid 18 snap 133 2024-09-20T22:47:45.266 INFO:tasks.rados.rados.0.smithi084.stdout:1475: expect (ObjNum 472 snap 130 seq_num 472) 2024-09-20T22:47:45.266 INFO:tasks.rados.rados.0.smithi084.stdout:1476: write oid 7 current snap is 152 2024-09-20T22:47:45.266 INFO:tasks.rados.rados.0.smithi084.stdout:1476: seq_num 528 ranges {474096=559829,1780963=408850,2800046=74050} 2024-09-20T22:47:45.272 INFO:tasks.rados.rados.0.smithi084.stdout:1476: writing smithi08477203-7 from 474096 to 1033925 tid 1 2024-09-20T22:47:45.279 INFO:tasks.rados.rados.0.smithi084.stdout:1476: writing smithi08477203-7 from 1780963 to 2189813 tid 2 2024-09-20T22:47:45.284 INFO:tasks.rados.rados.0.smithi084.stdout:1476: writing smithi08477203-7 from 2800046 to 2874096 tid 3 2024-09-20T22:47:45.284 INFO:tasks.rados.rados.0.smithi084.stdout:1477: copy_from oid 6 from oid 26 current snap is 152 2024-09-20T22:47:45.284 INFO:tasks.rados.rados.0.smithi084.stdout:1475: expect (ObjNum 472 snap 130 seq_num 472) 2024-09-20T22:47:45.287 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:45.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:45.289 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:47:45.389 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:47:45.411 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:45.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:45.412 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:47:45.476 INFO:tasks.rados.rados.0.smithi084.stdout:1476: finishing write tid 1 to smithi08477203-7 2024-09-20T22:47:45.476 INFO:tasks.rados.rados.0.smithi084.stdout:1476: finishing write tid 2 to smithi08477203-7 2024-09-20T22:47:45.476 INFO:tasks.rados.rados.0.smithi084.stdout:1476: finishing write tid 3 to smithi08477203-7 2024-09-20T22:47:45.476 INFO:tasks.rados.rados.0.smithi084.stdout:1476: finishing write tid 4 to smithi08477203-7 2024-09-20T22:47:45.476 INFO:tasks.rados.rados.0.smithi084.stdout:1476: finishing write tid 5 to smithi08477203-7 2024-09-20T22:47:45.476 INFO:tasks.rados.rados.0.smithi084.stdout:1476: oid 7 updating version 0 to 305 2024-09-20T22:47:45.476 INFO:tasks.rados.rados.0.smithi084.stdout:1476: oid 7 version 305 is already newer than 304 2024-09-20T22:47:45.477 INFO:tasks.rados.rados.0.smithi084.stdout:1476: oid 7 updating version 305 to 307 2024-09-20T22:47:45.477 INFO:tasks.rados.rados.0.smithi084.stdout:1476: oid 7 version 307 is already newer than 306 2024-09-20T22:47:45.477 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 7 v 307 (ObjNum 528 snap 152 seq_num 528) dirty exists 2024-09-20T22:47:45.477 INFO:tasks.rados.rados.0.smithi084.stdout:1476: left oid 7 (ObjNum 528 snap 152 seq_num 528) 2024-09-20T22:47:45.477 INFO:tasks.rados.rados.0.smithi084.stdout:1477: finishing copy_from racing read to smithi08477203-6 2024-09-20T22:47:45.477 INFO:tasks.rados.rados.0.smithi084.stdout:1475: done (2 left) 2024-09-20T22:47:45.477 INFO:tasks.rados.rados.0.smithi084.stdout:1476: done (1 left) 2024-09-20T22:47:45.477 INFO:tasks.rados.rados.0.smithi084.stdout:1478: read oid 22 snap -1 2024-09-20T22:47:45.477 INFO:tasks.rados.rados.0.smithi084.stdout:1478: expect deleted 2024-09-20T22:47:45.477 INFO:tasks.rados.rados.0.smithi084.stdout:1479: read oid 42 snap -1 2024-09-20T22:47:45.477 INFO:tasks.rados.rados.0.smithi084.stdout:1479: expect deleted 2024-09-20T22:47:45.477 INFO:tasks.rados.rados.0.smithi084.stdout:1480: rollback oid 38 current snap is 152 2024-09-20T22:47:45.477 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 38 to 138 2024-09-20T22:47:45.477 INFO:tasks.rados.rados.0.smithi084.stdout:1477: finishing copy_from to smithi08477203-6 2024-09-20T22:47:45.478 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 6 v 301 (ObjNum 502 snap 143 seq_num 502) dirty exists 2024-09-20T22:47:45.478 INFO:tasks.rados.rados.0.smithi084.stdout:1477: done (3 left) 2024-09-20T22:47:45.478 INFO:tasks.rados.rados.0.smithi084.stdout:1481: write oid 45 current snap is 152 2024-09-20T22:47:45.478 INFO:tasks.rados.rados.0.smithi084.stdout:1481: seq_num 529 ranges {490903=635078,1839445=552508} 2024-09-20T22:47:45.482 INFO:tasks.rados.rados.0.smithi084.stdout:1481: writing smithi08477203-45 from 490903 to 1125981 tid 1 2024-09-20T22:47:45.489 INFO:tasks.rados.rados.0.smithi084.stdout:1481: writing smithi08477203-45 from 1839445 to 2391953 tid 2 2024-09-20T22:47:45.489 INFO:tasks.rados.rados.0.smithi084.stdout:1482: copy_from oid 43 from oid 9 current snap is 152 2024-09-20T22:47:45.489 INFO:tasks.rados.rados.0.smithi084.stdout:1480: finishing rollback tid 0 to smithi08477203-38 2024-09-20T22:47:45.489 INFO:tasks.rados.rados.0.smithi084.stdout:1480: finishing rollback tid 1 to smithi08477203-38 2024-09-20T22:47:45.490 INFO:tasks.rados.rados.0.smithi084.stdout:1480: finishing rollback tid 2 to smithi08477203-38 2024-09-20T22:47:45.490 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 38 v 413 (ObjNum 489 snap 137 seq_num 489) dirty exists 2024-09-20T22:47:45.490 INFO:tasks.rados.rados.0.smithi084.stdout:1481: finishing write tid 1 to smithi08477203-45 2024-09-20T22:47:45.490 INFO:tasks.rados.rados.0.smithi084.stdout:1478: done (4 left) 2024-09-20T22:47:45.490 INFO:tasks.rados.rados.0.smithi084.stdout:1479: done (3 left) 2024-09-20T22:47:45.490 INFO:tasks.rados.rados.0.smithi084.stdout:1480: done (2 left) 2024-09-20T22:47:45.490 INFO:tasks.rados.rados.0.smithi084.stdout:1483: delete oid 34 current snap is 152 2024-09-20T22:47:45.490 INFO:tasks.rados.rados.0.smithi084.stdout:1482: finishing copy_from to smithi08477203-43 2024-09-20T22:47:45.491 INFO:tasks.rados.rados.0.smithi084.stdout:1482: got expected ENOENT (src dne) 2024-09-20T22:47:45.491 INFO:tasks.rados.rados.0.smithi084.stdout:1482: finishing copy_from racing read to smithi08477203-43 2024-09-20T22:47:45.491 INFO:tasks.rados.rados.0.smithi084.stdout:1482: done (2 left) 2024-09-20T22:47:45.491 INFO:tasks.rados.rados.0.smithi084.stdout:1483: done (1 left) 2024-09-20T22:47:45.491 INFO:tasks.rados.rados.0.smithi084.stdout:1484: snap_create 2024-09-20T22:47:45.492 INFO:tasks.rados.rados.0.smithi084.stdout:1481: finishing write tid 2 to smithi08477203-45 2024-09-20T22:47:45.492 INFO:tasks.rados.rados.0.smithi084.stdout:1481: finishing write tid 3 to smithi08477203-45 2024-09-20T22:47:45.492 INFO:tasks.rados.rados.0.smithi084.stdout:1481: finishing write tid 4 to smithi08477203-45 2024-09-20T22:47:45.492 INFO:tasks.rados.rados.0.smithi084.stdout:1481: oid 45 updating version 0 to 162 2024-09-20T22:47:45.492 INFO:tasks.rados.rados.0.smithi084.stdout:1481: oid 45 version 162 is already newer than 161 2024-09-20T22:47:45.492 INFO:tasks.rados.rados.0.smithi084.stdout:1481: oid 45 updating version 162 to 163 2024-09-20T22:47:45.492 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 45 v 163 (ObjNum 529 snap 152 seq_num 529) dirty exists 2024-09-20T22:47:45.493 INFO:tasks.rados.rados.0.smithi084.stdout:1481: left oid 45 (ObjNum 529 snap 152 seq_num 529) 2024-09-20T22:47:45.515 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:47:45.537 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:45.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:45.538 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:47:45.572 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:45 smithi084 ceph-mon[40024]: pgmap v477: 168 pgs: 6 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 157 active+clean; 278 MiB data, 818 MiB used, 536 GiB / 536 GiB avail; 2.4 MiB/s rd, 3.1 MiB/s wr, 12 op/s 2024-09-20T22:47:45.572 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:45 smithi084 ceph-mon[40024]: osdmap e384: 8 total, 8 up, 6 in 2024-09-20T22:47:45.572 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:45 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:47:45.572 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:45 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/3311271393' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:47:45.572 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:45 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:47:45.573 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:45 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:47:45.573 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:45 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:47:45.573 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:45 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:47:45.573 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:45 smithi084 ceph-mon[29774]: pgmap v477: 168 pgs: 6 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 157 active+clean; 278 MiB data, 818 MiB used, 536 GiB / 536 GiB avail; 2.4 MiB/s rd, 3.1 MiB/s wr, 12 op/s 2024-09-20T22:47:45.573 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:45 smithi084 ceph-mon[29774]: osdmap e384: 8 total, 8 up, 6 in 2024-09-20T22:47:45.573 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:45 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:47:45.573 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:45 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/3311271393' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:47:45.573 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:45 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:47:45.573 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:45 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:47:45.573 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:45 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:47:45.574 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:45 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:47:45.601 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.0 dump_historic_ops 2024-09-20T22:47:45.638 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:47:45.665 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:45.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:45.666 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:47:45.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:45 smithi154 ceph-mon[33264]: pgmap v477: 168 pgs: 6 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 157 active+clean; 278 MiB data, 818 MiB used, 536 GiB / 536 GiB avail; 2.4 MiB/s rd, 3.1 MiB/s wr, 12 op/s 2024-09-20T22:47:45.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:45 smithi154 ceph-mon[33264]: osdmap e384: 8 total, 8 up, 6 in 2024-09-20T22:47:45.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:45 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:47:45.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:45 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/3311271393' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:47:45.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:45 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:47:45.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:45 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:47:45.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:45 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:47:45.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:45 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:47:45.766 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:47:45.787 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:45.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:45.788 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:47:45.872 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:47:45.872 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:47:45.872 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:47:45.872 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-20T22:47:45.872 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:47:45.873 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:47:45.873 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:47:45.873 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:47:45.873 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:47:45.873 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T22:47:45.873 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:47:45.873 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:47:45.874 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:47:45.874 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:47:45.874 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:47:45.874 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:47:45.889 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:47:45.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:45.911 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:45.912 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:47:46.012 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:47:46.034 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:46.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:46.035 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:47:46.052 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-20T22:47:46.135 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:47:46.157 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:46.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:46.158 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:47:46.258 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:47:46.273 INFO:tasks.rados.rados.0.smithi084.stdout:1481: done (1 left) 2024-09-20T22:47:46.273 INFO:tasks.rados.rados.0.smithi084.stdout:1484: done (0 left) 2024-09-20T22:47:46.273 INFO:tasks.rados.rados.0.smithi084.stdout:1485: rollback oid 30 current snap is 153 2024-09-20T22:47:46.273 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 30 to 125 2024-09-20T22:47:46.273 INFO:tasks.rados.rados.0.smithi084.stdout:1486: snap_create 2024-09-20T22:47:46.277 INFO:tasks.rados.rados.0.smithi084.stdout:1485: finishing rollback tid 0 to smithi08477203-30 2024-09-20T22:47:46.278 INFO:tasks.rados.rados.0.smithi084.stdout:1485: finishing rollback tid 1 to smithi08477203-30 2024-09-20T22:47:46.278 INFO:tasks.rados.rados.0.smithi084.stdout:1485: finishing rollback tid 2 to smithi08477203-30 2024-09-20T22:47:46.278 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 30 v 405 (ObjNum 447 snap 119 seq_num 447) dirty exists 2024-09-20T22:47:46.281 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:46.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:46.282 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:47:46.383 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:47:46.405 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:46.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:46.406 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:47:46.506 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:47:46.510 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:46 smithi084 ceph-mon[40024]: osdmap e385: 8 total, 8 up, 6 in 2024-09-20T22:47:46.510 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:46 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:47:46.510 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:46 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:47:46.510 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:46 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:47:46.511 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:46 smithi084 ceph-mon[29774]: osdmap e385: 8 total, 8 up, 6 in 2024-09-20T22:47:46.511 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:46 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:47:46.511 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:46 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:47:46.511 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:46 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:47:46.528 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:46.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:46.529 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:47:46.629 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:47:46.635 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:47:46.639 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:47:46.651 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:46.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:46.652 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:47:46.671 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:47:46.753 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:47:46.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:46 smithi154 ceph-mon[33264]: osdmap e385: 8 total, 8 up, 6 in 2024-09-20T22:47:46.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:46 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:47:46.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:46 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:47:46.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:46 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:47:46.776 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:46.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:46.777 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:47:46.877 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:47:46.898 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:46.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:46.899 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:47:46.999 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:47:47.023 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:47.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:47.024 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:47:47.124 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:47:47.158 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:47.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:47.159 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:47:47.260 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:47:47.272 INFO:tasks.rados.rados.0.smithi084.stdout:1485: done (1 left) 2024-09-20T22:47:47.272 INFO:tasks.rados.rados.0.smithi084.stdout:1486: done (0 left) 2024-09-20T22:47:47.272 INFO:tasks.rados.rados.0.smithi084.stdout:1487: read oid 34 snap -1 2024-09-20T22:47:47.272 INFO:tasks.rados.rados.0.smithi084.stdout:1487: expect deleted 2024-09-20T22:47:47.272 INFO:tasks.rados.rados.0.smithi084.stdout:1488: write (excl) oid 12 current snap is 154 2024-09-20T22:47:47.272 INFO:tasks.rados.rados.0.smithi084.stdout:1488: seq_num 531 ranges {524517=785576,1956409=556177} 2024-09-20T22:47:47.279 INFO:tasks.rados.rados.0.smithi084.stdout:1488: writing smithi08477203-12 from 524517 to 1310093 tid 1 2024-09-20T22:47:47.286 INFO:tasks.rados.rados.0.smithi084.stdout:1488: writing smithi08477203-12 from 1956409 to 2512586 tid 2 2024-09-20T22:47:47.286 INFO:tasks.rados.rados.0.smithi084.stdout:1489: delete oid 17 current snap is 154 2024-09-20T22:47:47.288 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:47.289 INFO:tasks.rados.rados.0.smithi084.stdout:1487: done (2 left) 2024-09-20T22:47:47.289 INFO:tasks.rados.rados.0.smithi084.stdout:1489: done (1 left) 2024-09-20T22:47:47.289 INFO:tasks.rados.rados.0.smithi084.stdout:1490: read oid 37 snap 140 2024-09-20T22:47:47.289 INFO:tasks.rados.rados.0.smithi084.stdout:1490: expect (ObjNum 478 snap 135 seq_num 478) 2024-09-20T22:47:47.289 INFO:tasks.rados.rados.0.smithi084.stdout:1491: delete oid 26 current snap is 154 2024-09-20T22:47:47.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:47.290 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:47:47.291 INFO:tasks.rados.rados.0.smithi084.stdout:1488: finishing write tid 1 to smithi08477203-12 2024-09-20T22:47:47.291 INFO:tasks.rados.rados.0.smithi084.stdout:1488: finishing write tid 2 to smithi08477203-12 2024-09-20T22:47:47.292 INFO:tasks.rados.rados.0.smithi084.stdout:1488: finishing write tid 3 to smithi08477203-12 2024-09-20T22:47:47.292 INFO:tasks.rados.rados.0.smithi084.stdout:1488: finishing write tid 4 to smithi08477203-12 2024-09-20T22:47:47.292 INFO:tasks.rados.rados.0.smithi084.stdout:1488: oid 12 updating version 0 to 336 2024-09-20T22:47:47.292 INFO:tasks.rados.rados.0.smithi084.stdout:1488: oid 12 updating version 336 to 337 2024-09-20T22:47:47.292 INFO:tasks.rados.rados.0.smithi084.stdout:1488: oid 12 updating version 337 to 338 2024-09-20T22:47:47.292 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 12 v 338 (ObjNum 531 snap 154 seq_num 531) dirty exists 2024-09-20T22:47:47.292 INFO:tasks.rados.rados.0.smithi084.stdout:1488: left oid 12 (ObjNum 531 snap 154 seq_num 531) 2024-09-20T22:47:47.292 INFO:tasks.rados.rados.0.smithi084.stdout:1488: done (2 left) 2024-09-20T22:47:47.293 INFO:tasks.rados.rados.0.smithi084.stdout:1491: done (1 left) 2024-09-20T22:47:47.293 INFO:tasks.rados.rados.0.smithi084.stdout:1492: snap_create 2024-09-20T22:47:47.295 INFO:tasks.rados.rados.0.smithi084.stdout:1490: expect (ObjNum 478 snap 135 seq_num 478) 2024-09-20T22:47:47.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:47 smithi084 ceph-mon[40024]: pgmap v480: 168 pgs: 6 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 157 active+clean; 278 MiB data, 818 MiB used, 536 GiB / 536 GiB avail; 2.4 MiB/s rd, 3.1 MiB/s wr, 12 op/s 2024-09-20T22:47:47.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:47 smithi084 ceph-mon[40024]: purged_snaps scrub starts 2024-09-20T22:47:47.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:47 smithi084 ceph-mon[40024]: purged_snaps scrub ok 2024-09-20T22:47:47.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:47 smithi084 ceph-mon[40024]: osdmap e386: 8 total, 8 up, 6 in 2024-09-20T22:47:47.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:47 smithi084 ceph-mon[29774]: pgmap v480: 168 pgs: 6 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 157 active+clean; 278 MiB data, 818 MiB used, 536 GiB / 536 GiB avail; 2.4 MiB/s rd, 3.1 MiB/s wr, 12 op/s 2024-09-20T22:47:47.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:47 smithi084 ceph-mon[29774]: purged_snaps scrub starts 2024-09-20T22:47:47.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:47 smithi084 ceph-mon[29774]: purged_snaps scrub ok 2024-09-20T22:47:47.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:47 smithi084 ceph-mon[29774]: osdmap e386: 8 total, 8 up, 6 in 2024-09-20T22:47:47.391 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:47:47.413 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:47.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:47.414 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:47:47.509 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-20T22:47:47.509 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:47:47.515 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:47:47.540 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:47.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:47.541 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:47:47.642 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:47:47.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:47.664 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:47.664 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:47:47.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:47 smithi154 ceph-mon[33264]: pgmap v480: 168 pgs: 6 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 157 active+clean; 278 MiB data, 818 MiB used, 536 GiB / 536 GiB avail; 2.4 MiB/s rd, 3.1 MiB/s wr, 12 op/s 2024-09-20T22:47:47.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:47 smithi154 ceph-mon[33264]: purged_snaps scrub starts 2024-09-20T22:47:47.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:47 smithi154 ceph-mon[33264]: purged_snaps scrub ok 2024-09-20T22:47:47.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:47 smithi154 ceph-mon[33264]: osdmap e386: 8 total, 8 up, 6 in 2024-09-20T22:47:47.765 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:47:47.787 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:47.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:47.788 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:47:47.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:47 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mon-a[29770]: 2024-09-20T22:47:47.536+0000 7f26ad5e8640 -1 mon.a@0(leader).osd e387 definitely_dead 0 2024-09-20T22:47:47.888 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:47:47.911 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:47.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:47.912 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:47:48.013 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:47:48.036 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:48.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:48.037 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:47:48.137 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:47:48.160 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:48.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:48.161 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:47:48.191 INFO:teuthology.orchestra.run.smithi084.stderr:marked in osd.0. 2024-09-20T22:47:48.196 INFO:tasks.rados.rados.0.smithi084.stdout:1490: done (1 left) 2024-09-20T22:47:48.196 INFO:tasks.rados.rados.0.smithi084.stdout:1492: done (0 left) 2024-09-20T22:47:48.196 INFO:tasks.rados.rados.0.smithi084.stdout:1493: snap_create 2024-09-20T22:47:48.262 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:47:48.272 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:47:48 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: ::ffff:172.21.15.154 - - [20/Sep/2024:22:47:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-20T22:47:48.285 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:48.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:48.286 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:47:48.303 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-20T22:47:48.386 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:47:48.409 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:48.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:48.410 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:47:48.511 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:47:48.533 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:48.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:48.533 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:47:48.537 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-20T22:47:48.559 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:48 smithi084 ceph-mon[29774]: osdmap e387: 8 total, 8 up, 6 in 2024-09-20T22:47:48.559 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:48 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/1148062085' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["0"]}]: dispatch 2024-09-20T22:47:48.559 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:48 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/1148062085' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["0"]}]': finished 2024-09-20T22:47:48.559 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:48 smithi084 ceph-mon[29774]: osdmap e388: 8 total, 8 up, 7 in 2024-09-20T22:47:48.560 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:48 smithi084 ceph-mon[40024]: osdmap e387: 8 total, 8 up, 6 in 2024-09-20T22:47:48.560 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:48 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/1148062085' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["0"]}]: dispatch 2024-09-20T22:47:48.560 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:48 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/1148062085' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["0"]}]': finished 2024-09-20T22:47:48.560 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:48 smithi084 ceph-mon[40024]: osdmap e388: 8 total, 8 up, 7 in 2024-09-20T22:47:48.634 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:47:48.656 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:48.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:48.658 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:47:48.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:48 smithi154 ceph-mon[33264]: osdmap e387: 8 total, 8 up, 6 in 2024-09-20T22:47:48.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:48 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/1148062085' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["0"]}]: dispatch 2024-09-20T22:47:48.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:48 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/1148062085' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["0"]}]': finished 2024-09-20T22:47:48.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:48 smithi154 ceph-mon[33264]: osdmap e388: 8 total, 8 up, 7 in 2024-09-20T22:47:48.759 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:47:48.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:48.781 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:48.781 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:47:48.882 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:47:48.905 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:48.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:48.906 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:47:48.945 INFO:tasks.thrashosds.thrasher:Added osd 0 2024-09-20T22:47:49.007 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:47:49.029 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:49.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:49.030 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:47:49.131 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:47:49.154 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:49.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:49.155 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:47:49.202 INFO:tasks.rados.rados.0.smithi084.stdout:1493: done (0 left) 2024-09-20T22:47:49.202 INFO:tasks.rados.rados.0.smithi084.stdout:1494: read oid 31 snap -1 2024-09-20T22:47:49.202 INFO:tasks.rados.rados.0.smithi084.stdout:1494: expect (ObjNum 518 snap 151 seq_num 518) 2024-09-20T22:47:49.202 INFO:tasks.rados.rados.0.smithi084.stdout:1495: copy_from oid 2 from oid 26 current snap is 156 2024-09-20T22:47:49.202 INFO:tasks.rados.rados.0.smithi084.stdout:1496: read oid 3 snap -1 2024-09-20T22:47:49.202 INFO:tasks.rados.rados.0.smithi084.stdout:1496: expect (ObjNum 418 snap 105 seq_num 418) 2024-09-20T22:47:49.202 INFO:tasks.rados.rados.0.smithi084.stdout:1497: read oid 46 snap -1 2024-09-20T22:47:49.203 INFO:tasks.rados.rados.0.smithi084.stdout:1497: expect deleted 2024-09-20T22:47:49.203 INFO:tasks.rados.rados.0.smithi084.stdout:1498: rollback oid 29 current snap is 156 2024-09-20T22:47:49.203 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 29 to 133 2024-09-20T22:47:49.203 INFO:tasks.rados.rados.0.smithi084.stdout:1499: snap_remove snap 151 2024-09-20T22:47:49.256 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:47:49.278 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:49.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:49.279 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:47:49.380 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:47:49.403 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:49.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:49.404 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:47:49.471 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:47:49.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:49 smithi154 ceph-mon[33264]: pgmap v483: 168 pgs: 6 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 157 active+clean; 278 MiB data, 818 MiB used, 536 GiB / 536 GiB avail 2024-09-20T22:47:49.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:49 smithi154 ceph-mon[33264]: osdmap e389: 8 total, 8 up, 7 in 2024-09-20T22:47:49.504 INFO:journalctl@ceph.iscsi.iscsi.a.smithi154.stdout:Sep 20 22:47:49 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-iscsi-iscsi-a[59730]: debug there is no tcmu-runner data available 2024-09-20T22:47:49.505 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:47:49.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:49.527 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:49.528 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:47:49.597 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:49 smithi084 ceph-mon[40024]: pgmap v483: 168 pgs: 6 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 157 active+clean; 278 MiB data, 818 MiB used, 536 GiB / 536 GiB avail 2024-09-20T22:47:49.597 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:49 smithi084 ceph-mon[40024]: osdmap e389: 8 total, 8 up, 7 in 2024-09-20T22:47:49.597 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:49 smithi084 ceph-mon[29774]: pgmap v483: 168 pgs: 6 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 157 active+clean; 278 MiB data, 818 MiB used, 536 GiB / 536 GiB avail 2024-09-20T22:47:49.597 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:49 smithi084 ceph-mon[29774]: osdmap e389: 8 total, 8 up, 7 in 2024-09-20T22:47:49.629 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:47:49.651 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:49.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:49.652 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:47:49.699 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:47:49.752 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:47:49.774 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:49.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:49.775 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:47:49.876 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:47:49.897 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:49.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:49.898 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:47:49.999 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:47:50.022 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:50.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:50.023 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:47:50.025 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-20T22:47:50.025 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:47:50.123 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:47:50.145 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:50.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:50.146 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:47:50.202 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is set 2024-09-20T22:47:50.202 INFO:tasks.rados.rados.0.smithi084.stdout:1499: done (5 left) 2024-09-20T22:47:50.202 INFO:tasks.rados.rados.0.smithi084.stdout:1500: snap_create 2024-09-20T22:47:50.203 INFO:tasks.rados.rados.0.smithi084.stdout:1495: finishing copy_from to smithi08477203-2 2024-09-20T22:47:50.203 INFO:tasks.rados.rados.0.smithi084.stdout:1495: got expected ENOENT (src dne) 2024-09-20T22:47:50.203 INFO:tasks.rados.rados.0.smithi084.stdout:1495: finishing copy_from racing read to smithi08477203-2 2024-09-20T22:47:50.203 INFO:tasks.rados.rados.0.smithi084.stdout:1498: finishing rollback tid 0 to smithi08477203-29 2024-09-20T22:47:50.203 INFO:tasks.rados.rados.0.smithi084.stdout:1498: finishing rollback tid 1 to smithi08477203-29 2024-09-20T22:47:50.203 INFO:tasks.rados.rados.0.smithi084.stdout:1498: finishing rollback tid 2 to smithi08477203-29 2024-09-20T22:47:50.203 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 29 v 838 (ObjNum 400 snap 101 seq_num 400) dirty exists 2024-09-20T22:47:50.227 INFO:tasks.rados.rados.0.smithi084.stdout:1496: expect (ObjNum 418 snap 105 seq_num 418) 2024-09-20T22:47:50.247 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:47:50.273 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:50.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:50.274 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:47:50.356 INFO:tasks.rados.rados.0.smithi084.stdout:1494: expect (ObjNum 518 snap 151 seq_num 518) 2024-09-20T22:47:50.375 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:47:50.398 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:50.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:50.399 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:47:50.500 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:47:50.521 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:50.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:50.522 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:47:50.527 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:50 smithi084 ceph-mon[40024]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:47:50.527 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:50 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/2686340301' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:47:50.527 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:50 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:47:50.527 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:50 smithi084 ceph-mon[40024]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:47:50.527 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:50 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T22:47:50.527 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:50 smithi084 ceph-mon[40024]: osdmap e390: 8 total, 8 up, 7 in 2024-09-20T22:47:50.527 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:50 smithi084 ceph-mon[40024]: 1.6 starting backfill to osd.0 from (0'0,0'0] MAX to 77'128 2024-09-20T22:47:50.527 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:50 smithi084 ceph-mon[29774]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:47:50.527 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:50 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/2686340301' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:47:50.528 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:50 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:47:50.528 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:50 smithi084 ceph-mon[29774]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:47:50.528 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:50 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T22:47:50.528 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:50 smithi084 ceph-mon[29774]: osdmap e390: 8 total, 8 up, 7 in 2024-09-20T22:47:50.528 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:50 smithi084 ceph-mon[29774]: 1.6 starting backfill to osd.0 from (0'0,0'0] MAX to 77'128 2024-09-20T22:47:50.622 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:47:50.644 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:50.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:50.645 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:47:50.715 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:47:50.715 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:47:50.732 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:47:50.733 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:47:50.746 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:47:50.749 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:47:50.749 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:47:50.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:50 smithi154 ceph-mon[33264]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:47:50.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:50 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/2686340301' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:47:50.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:50 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:47:50.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:50 smithi154 ceph-mon[33264]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:47:50.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:50 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T22:47:50.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:50 smithi154 ceph-mon[33264]: osdmap e390: 8 total, 8 up, 7 in 2024-09-20T22:47:50.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:50 smithi154 ceph-mon[33264]: 1.6 starting backfill to osd.0 from (0'0,0'0] MAX to 77'128 2024-09-20T22:47:50.766 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-20T22:47:50.766 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T22:47:50.768 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:50.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:50.769 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:47:50.783 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:47:50.783 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:47:50.799 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:47:50.800 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:47:50.816 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:47:50.816 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:47:50.833 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:47:50.833 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:47:50.869 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:47:50.891 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:50.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:50.892 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:47:50.993 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:47:51.014 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:51.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:51.015 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:47:51.116 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:47:51.137 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:51.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:51.138 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:47:51.204 INFO:tasks.rados.rados.0.smithi084.stdout:1494: done (5 left) 2024-09-20T22:47:51.204 INFO:tasks.rados.rados.0.smithi084.stdout:1495: done (4 left) 2024-09-20T22:47:51.204 INFO:tasks.rados.rados.0.smithi084.stdout:1496: done (3 left) 2024-09-20T22:47:51.205 INFO:tasks.rados.rados.0.smithi084.stdout:1497: done (2 left) 2024-09-20T22:47:51.205 INFO:tasks.rados.rados.0.smithi084.stdout:1498: done (1 left) 2024-09-20T22:47:51.205 INFO:tasks.rados.rados.0.smithi084.stdout:1500: done (0 left) 2024-09-20T22:47:51.205 INFO:tasks.rados.rados.0.smithi084.stdout:1501: copy_from oid 21 from oid 46 current snap is 157 2024-09-20T22:47:51.205 INFO:tasks.rados.rados.0.smithi084.stdout:1502: snap_create 2024-09-20T22:47:51.239 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:47:51.260 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:51.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:51.261 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:47:51.362 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:47:51.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:51 smithi084 ceph-mon[40024]: pgmap v486: 168 pgs: 5 unknown, 71 peering, 2 active+clean+snaptrim, 90 active+clean; 253 MiB data, 904 MiB used, 625 GiB / 626 GiB avail 2024-09-20T22:47:51.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:51 smithi084 ceph-mon[40024]: 1.0 starting backfill to osd.0 from (0'0,0'0] MAX to 77'128 2024-09-20T22:47:51.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:51 smithi084 ceph-mon[40024]: 1.2 starting backfill to osd.0 from (0'0,0'0] MAX to 77'128 2024-09-20T22:47:51.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:51 smithi084 ceph-mon[40024]: 1.1 starting backfill to osd.0 from (0'0,0'0] MAX to 77'128 2024-09-20T22:47:51.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:51 smithi084 ceph-mon[40024]: Health check failed: Reduced data availability: 12 pgs inactive, 15 pgs peering (PG_AVAILABILITY) 2024-09-20T22:47:51.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:51 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/110949944' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:47:51.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:51 smithi084 ceph-mon[40024]: osdmap e391: 8 total, 8 up, 7 in 2024-09-20T22:47:51.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:51 smithi084 ceph-mon[29774]: pgmap v486: 168 pgs: 5 unknown, 71 peering, 2 active+clean+snaptrim, 90 active+clean; 253 MiB data, 904 MiB used, 625 GiB / 626 GiB avail 2024-09-20T22:47:51.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:51 smithi084 ceph-mon[29774]: 1.0 starting backfill to osd.0 from (0'0,0'0] MAX to 77'128 2024-09-20T22:47:51.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:51 smithi084 ceph-mon[29774]: 1.2 starting backfill to osd.0 from (0'0,0'0] MAX to 77'128 2024-09-20T22:47:51.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:51 smithi084 ceph-mon[29774]: 1.1 starting backfill to osd.0 from (0'0,0'0] MAX to 77'128 2024-09-20T22:47:51.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:51 smithi084 ceph-mon[29774]: Health check failed: Reduced data availability: 12 pgs inactive, 15 pgs peering (PG_AVAILABILITY) 2024-09-20T22:47:51.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:51 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/110949944' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:47:51.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:51 smithi084 ceph-mon[29774]: osdmap e391: 8 total, 8 up, 7 in 2024-09-20T22:47:51.384 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:51.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:51.385 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:47:51.485 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:47:51.507 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:51.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:51.507 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:47:51.608 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:47:51.630 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:51.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:51.631 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:47:51.731 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:47:51.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:51 smithi154 ceph-mon[33264]: pgmap v486: 168 pgs: 5 unknown, 71 peering, 2 active+clean+snaptrim, 90 active+clean; 253 MiB data, 904 MiB used, 625 GiB / 626 GiB avail 2024-09-20T22:47:51.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:51 smithi154 ceph-mon[33264]: 1.0 starting backfill to osd.0 from (0'0,0'0] MAX to 77'128 2024-09-20T22:47:51.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:51 smithi154 ceph-mon[33264]: 1.2 starting backfill to osd.0 from (0'0,0'0] MAX to 77'128 2024-09-20T22:47:51.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:51 smithi154 ceph-mon[33264]: 1.1 starting backfill to osd.0 from (0'0,0'0] MAX to 77'128 2024-09-20T22:47:51.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:51 smithi154 ceph-mon[33264]: Health check failed: Reduced data availability: 12 pgs inactive, 15 pgs peering (PG_AVAILABILITY) 2024-09-20T22:47:51.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:51 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/110949944' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:47:51.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:51 smithi154 ceph-mon[33264]: osdmap e391: 8 total, 8 up, 7 in 2024-09-20T22:47:51.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:51.756 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:51.756 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:47:51.859 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:47:51.880 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:51.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:51.881 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:47:51.982 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:47:52.004 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:52.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:52.005 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:47:52.106 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:47:52.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:52.130 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:52.130 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:47:52.202 INFO:tasks.rados.rados.0.smithi084.stdout:1502: done (1 left) 2024-09-20T22:47:52.202 INFO:tasks.rados.rados.0.smithi084.stdout:1503: read oid 38 snap -1 2024-09-20T22:47:52.202 INFO:tasks.rados.rados.0.smithi084.stdout:1503: expect (ObjNum 489 snap 137 seq_num 489) 2024-09-20T22:47:52.203 INFO:tasks.rados.rados.0.smithi084.stdout:1504: read oid 32 snap -1 2024-09-20T22:47:52.203 INFO:tasks.rados.rados.0.smithi084.stdout:1504: expect (ObjNum 466 snap 128 seq_num 466) 2024-09-20T22:47:52.203 INFO:tasks.rados.rados.0.smithi084.stdout:1505: delete oid 19 current snap is 158 2024-09-20T22:47:52.209 INFO:tasks.rados.rados.0.smithi084.stdout:1505: done (3 left) 2024-09-20T22:47:52.210 INFO:tasks.rados.rados.0.smithi084.stdout:1506: snap_create 2024-09-20T22:47:52.212 INFO:tasks.rados.rados.0.smithi084.stdout:1501: finishing copy_from to smithi08477203-21 2024-09-20T22:47:52.212 INFO:tasks.rados.rados.0.smithi084.stdout:1501: got expected ENOENT (src dne) 2024-09-20T22:47:52.212 INFO:tasks.rados.rados.0.smithi084.stdout:1501: finishing copy_from racing read to smithi08477203-21 2024-09-20T22:47:52.216 INFO:tasks.rados.rados.0.smithi084.stdout:1504: expect (ObjNum 466 snap 128 seq_num 466) 2024-09-20T22:47:52.231 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:47:52.252 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:52.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:52.253 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:47:52.354 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:47:52.375 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:52.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:52.376 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:47:52.448 INFO:tasks.rados.rados.0.smithi084.stdout:1503: expect (ObjNum 489 snap 137 seq_num 489) 2024-09-20T22:47:52.476 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:47:52.497 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:52.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:52.498 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:47:52.570 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.1 dump_blocked_ops 2024-09-20T22:47:52.599 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:47:52.633 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:52.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:52.634 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:47:52.735 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:47:52.761 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:52.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:52.762 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:47:52.818 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:47:52.863 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:47:52.888 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:52.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:52.889 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:47:52.990 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:47:53.012 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:53.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:53.013 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:47:53.114 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:47:53.135 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:53.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:53.136 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:47:53.215 INFO:tasks.rados.rados.0.smithi084.stdout:1501: done (3 left) 2024-09-20T22:47:53.215 INFO:tasks.rados.rados.0.smithi084.stdout:1503: done (2 left) 2024-09-20T22:47:53.215 INFO:tasks.rados.rados.0.smithi084.stdout:1504: done (1 left) 2024-09-20T22:47:53.216 INFO:tasks.rados.rados.0.smithi084.stdout:1506: done (0 left) 2024-09-20T22:47:53.216 INFO:tasks.rados.rados.0.smithi084.stdout:1507: read oid 22 snap -1 2024-09-20T22:47:53.216 INFO:tasks.rados.rados.0.smithi084.stdout:1507: expect deleted 2024-09-20T22:47:53.216 INFO:tasks.rados.rados.0.smithi084.stdout:1508: read oid 23 snap -1 2024-09-20T22:47:53.216 INFO:tasks.rados.rados.0.smithi084.stdout:1508: expect (ObjNum 428 snap 110 seq_num 428) 2024-09-20T22:47:53.216 INFO:tasks.rados.rados.0.smithi084.stdout:1509: read oid 30 snap 152 2024-09-20T22:47:53.216 INFO:tasks.rados.rados.0.smithi084.stdout:1509: expect (ObjNum 416 snap 105 seq_num 416) 2024-09-20T22:47:53.216 INFO:tasks.rados.rados.0.smithi084.stdout:1510: write (excl) oid 18 current snap is 159 2024-09-20T22:47:53.216 INFO:tasks.rados.rados.0.smithi084.stdout:1510: seq_num 535 ranges {591745=402925,1790337=563515} 2024-09-20T22:47:53.221 INFO:tasks.rados.rados.0.smithi084.stdout:1510: writing smithi08477203-18 from 591745 to 994670 tid 1 2024-09-20T22:47:53.229 INFO:tasks.rados.rados.0.smithi084.stdout:1510: writing smithi08477203-18 from 1790337 to 2353852 tid 2 2024-09-20T22:47:53.229 INFO:tasks.rados.rados.0.smithi084.stdout:1511: delete oid 39 current snap is 159 2024-09-20T22:47:53.229 INFO:tasks.rados.rados.0.smithi084.stdout:1508: expect (ObjNum 428 snap 110 seq_num 428) 2024-09-20T22:47:53.237 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:47:53.260 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:53.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:53.261 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:47:53.348 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-20T22:47:53.348 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2024-09-20T22:47:53.348 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2024-09-20T22:47:53.348 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2024-09-20T22:47:53.348 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-20T22:47:53.362 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:47:53.385 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:53.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:53.386 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:47:53.447 INFO:tasks.rados.rados.0.smithi084.stdout:1510: finishing write tid 1 to smithi08477203-18 2024-09-20T22:47:53.447 INFO:tasks.rados.rados.0.smithi084.stdout:1509: expect (ObjNum 416 snap 105 seq_num 416) 2024-09-20T22:47:53.479 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd unset noscrub 2024-09-20T22:47:53.487 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:47:53.501 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:53 smithi084 ceph-mon[40024]: pgmap v489: 168 pgs: 5 unknown, 71 peering, 2 active+clean+snaptrim, 90 active+clean; 253 MiB data, 904 MiB used, 625 GiB / 626 GiB avail 2024-09-20T22:47:53.502 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:53 smithi084 ceph-mon[40024]: osdmap e392: 8 total, 8 up, 7 in 2024-09-20T22:47:53.502 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:53 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:47:53.502 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:53 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:47:53.502 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:53 smithi084 ceph-mon[29774]: pgmap v489: 168 pgs: 5 unknown, 71 peering, 2 active+clean+snaptrim, 90 active+clean; 253 MiB data, 904 MiB used, 625 GiB / 626 GiB avail 2024-09-20T22:47:53.502 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:53 smithi084 ceph-mon[29774]: osdmap e392: 8 total, 8 up, 7 in 2024-09-20T22:47:53.502 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:53 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:47:53.502 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:53 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:47:53.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:53 smithi154 ceph-mon[33264]: pgmap v489: 168 pgs: 5 unknown, 71 peering, 2 active+clean+snaptrim, 90 active+clean; 253 MiB data, 904 MiB used, 625 GiB / 626 GiB avail 2024-09-20T22:47:53.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:53 smithi154 ceph-mon[33264]: osdmap e392: 8 total, 8 up, 7 in 2024-09-20T22:47:53.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:53 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:47:53.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:53 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:47:53.511 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:53.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:53.512 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:47:53.584 INFO:tasks.rados.rados.0.smithi084.stdout:1510: finishing write tid 2 to smithi08477203-18 2024-09-20T22:47:53.584 INFO:tasks.rados.rados.0.smithi084.stdout:1510: finishing write tid 3 to smithi08477203-18 2024-09-20T22:47:53.584 INFO:tasks.rados.rados.0.smithi084.stdout:1510: finishing write tid 4 to smithi08477203-18 2024-09-20T22:47:53.584 INFO:tasks.rados.rados.0.smithi084.stdout:1510: oid 18 updating version 0 to 532 2024-09-20T22:47:53.584 INFO:tasks.rados.rados.0.smithi084.stdout:1510: oid 18 version 532 is already newer than 531 2024-09-20T22:47:53.584 INFO:tasks.rados.rados.0.smithi084.stdout:1510: oid 18 version 532 is already newer than 530 2024-09-20T22:47:53.585 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 18 v 532 (ObjNum 535 snap 159 seq_num 535) dirty exists 2024-09-20T22:47:53.585 INFO:tasks.rados.rados.0.smithi084.stdout:1510: left oid 18 (ObjNum 535 snap 159 seq_num 535) 2024-09-20T22:47:53.585 INFO:tasks.rados.rados.0.smithi084.stdout:1507: done (4 left) 2024-09-20T22:47:53.585 INFO:tasks.rados.rados.0.smithi084.stdout:1508: done (3 left) 2024-09-20T22:47:53.585 INFO:tasks.rados.rados.0.smithi084.stdout:1509: done (2 left) 2024-09-20T22:47:53.585 INFO:tasks.rados.rados.0.smithi084.stdout:1510: done (1 left) 2024-09-20T22:47:53.585 INFO:tasks.rados.rados.0.smithi084.stdout:1511: done (0 left) 2024-09-20T22:47:53.585 INFO:tasks.rados.rados.0.smithi084.stdout:1512: snap_remove snap 137 2024-09-20T22:47:53.612 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:47:53.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:53.637 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:53.637 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:47:53.738 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:47:53.759 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:53.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:53.760 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:47:53.861 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:47:53.885 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:53.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:53.886 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:47:53.946 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 5, 6, 7, 0] out_osds: [4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-20T22:47:53.946 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-20T22:47:53.946 INFO:tasks.thrashosds.thrasher:inject_pause on osd.6 2024-09-20T22:47:53.946 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-20T22:47:53.947 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-20T22:47:53.947 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.6 config set filestore_inject_stall 3 2024-09-20T22:47:53.987 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:47:54.013 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-20T22:47:54.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:54.014 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:54.014 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:47:54.028 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.1 dump_historic_ops 2024-09-20T22:47:54.062 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:47:54.115 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:47:54.137 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:54.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:54.139 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:47:54.197 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:47:54.222 INFO:tasks.rados.rados.0.smithi084.stdout:1512: done (0 left) 2024-09-20T22:47:54.222 INFO:tasks.rados.rados.0.smithi084.stdout:1513: rollback oid 10 current snap is 159 2024-09-20T22:47:54.222 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 10 to 158 2024-09-20T22:47:54.222 INFO:tasks.rados.rados.0.smithi084.stdout:1514: snap_remove snap 138 2024-09-20T22:47:54.240 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:47:54.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:54 smithi154 ceph-mon[33264]: osdmap e393: 8 total, 8 up, 7 in 2024-09-20T22:47:54.261 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:54.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:54.262 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:47:54.325 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:54 smithi084 ceph-mon[29774]: osdmap e393: 8 total, 8 up, 7 in 2024-09-20T22:47:54.325 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:54 smithi084 ceph-mon[40024]: osdmap e393: 8 total, 8 up, 7 in 2024-09-20T22:47:54.362 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:47:54.399 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:54.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:54.400 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:47:54.501 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:47:54.522 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:54.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:54.523 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:47:54.624 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:47:54.630 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:47:54.630 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:47:54.648 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:54.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:54.649 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:47:54.707 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-20T22:47:54.707 INFO:teuthology.orchestra.run.smithi154.stdout: "success": "" 2024-09-20T22:47:54.707 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-20T22:47:54.749 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:47:54.770 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:54.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:54.771 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:47:54.872 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:47:54.895 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:54.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:54.896 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:47:54.997 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:47:55.019 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:55.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:55.020 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:47:55.121 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:47:55.142 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:55.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:55.143 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:47:55.220 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is unset 2024-09-20T22:47:55.222 INFO:tasks.rados.rados.0.smithi084.stdout:1514: done (1 left) 2024-09-20T22:47:55.222 INFO:tasks.rados.rados.0.smithi084.stdout:1515: rollback oid 16 current snap is 159 2024-09-20T22:47:55.222 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 16 to 136 2024-09-20T22:47:55.222 INFO:tasks.rados.rados.0.smithi084.stdout:1513: finishing rollback tid 0 to smithi08477203-10 2024-09-20T22:47:55.222 INFO:tasks.rados.rados.0.smithi084.stdout:1513: finishing rollback tid 1 to smithi08477203-10 2024-09-20T22:47:55.222 INFO:tasks.rados.rados.0.smithi084.stdout:1513: finishing rollback tid 2 to smithi08477203-10 2024-09-20T22:47:55.222 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 10 v 536 (ObjNum 516 snap 148 seq_num 516) dirty exists 2024-09-20T22:47:55.222 INFO:tasks.rados.rados.0.smithi084.stdout:1513: done (1 left) 2024-09-20T22:47:55.222 INFO:tasks.rados.rados.0.smithi084.stdout:1516: snap_create 2024-09-20T22:47:55.232 INFO:tasks.rados.rados.0.smithi084.stdout:1515: finishing rollback tid 0 to smithi08477203-16 2024-09-20T22:47:55.232 INFO:tasks.rados.rados.0.smithi084.stdout:1515: finishing rollback tid 1 to smithi08477203-16 2024-09-20T22:47:55.233 INFO:tasks.rados.rados.0.smithi084.stdout:1515: finishing rollback tid 2 to smithi08477203-16 2024-09-20T22:47:55.233 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 16 v 422 (ObjNum 419 snap 105 seq_num 419) dirty exists 2024-09-20T22:47:55.244 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:47:55.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:55 smithi154 ceph-mon[33264]: pgmap v492: 168 pgs: 168 active+clean; 292 MiB data, 898 MiB used, 625 GiB / 626 GiB avail; 137 MiB/s rd, 25 MiB/s wr, 211 op/s; 36 MiB/s, 93 objects/s recovering 2024-09-20T22:47:55.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:55 smithi154 ceph-mon[33264]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 12 pgs inactive, 15 pgs peering) 2024-09-20T22:47:55.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:55 smithi154 ceph-mon[33264]: osdmap e394: 8 total, 8 up, 7 in 2024-09-20T22:47:55.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:55 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/2534590068' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:47:55.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:55.267 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:55.267 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:47:55.368 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:47:55.380 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-20T22:47:55.380 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:47:55.389 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:55.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:55.390 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:47:55.490 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:47:55.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:55.512 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:55.513 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:47:55.613 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:47:55.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:55 smithi084 ceph-mon[40024]: pgmap v492: 168 pgs: 168 active+clean; 292 MiB data, 898 MiB used, 625 GiB / 626 GiB avail; 137 MiB/s rd, 25 MiB/s wr, 211 op/s; 36 MiB/s, 93 objects/s recovering 2024-09-20T22:47:55.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:55 smithi084 ceph-mon[40024]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 12 pgs inactive, 15 pgs peering) 2024-09-20T22:47:55.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:55 smithi084 ceph-mon[40024]: osdmap e394: 8 total, 8 up, 7 in 2024-09-20T22:47:55.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:55 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/2534590068' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:47:55.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:55 smithi084 ceph-mon[29774]: pgmap v492: 168 pgs: 168 active+clean; 292 MiB data, 898 MiB used, 625 GiB / 626 GiB avail; 137 MiB/s rd, 25 MiB/s wr, 211 op/s; 36 MiB/s, 93 objects/s recovering 2024-09-20T22:47:55.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:55 smithi084 ceph-mon[29774]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 12 pgs inactive, 15 pgs peering) 2024-09-20T22:47:55.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:55 smithi084 ceph-mon[29774]: osdmap e394: 8 total, 8 up, 7 in 2024-09-20T22:47:55.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:55 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/2534590068' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:47:55.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:55.636 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:55.636 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:47:55.736 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:47:55.758 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:55.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:55.759 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:47:55.860 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:47:55.881 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:55.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:55.882 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:47:55.983 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:47:56.005 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:56.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:56.006 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:47:56.105 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-20T22:47:56.106 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:47:56.128 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:56.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:56.129 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:47:56.210 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:47:56.210 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:47:56.211 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:47:56.211 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-20T22:47:56.211 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:47:56.211 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:47:56.211 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:47:56.211 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:47:56.211 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:47:56.211 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:47:56.211 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T22:47:56.211 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:47:56.212 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:47:56.212 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:47:56.212 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:47:56.212 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:47:56.227 INFO:tasks.rados.rados.0.smithi084.stdout:1515: done (1 left) 2024-09-20T22:47:56.227 INFO:tasks.rados.rados.0.smithi084.stdout:1516: done (0 left) 2024-09-20T22:47:56.227 INFO:tasks.rados.rados.0.smithi084.stdout:1517: snap_create 2024-09-20T22:47:56.229 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:47:56.251 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:56.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:56.252 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:47:56.352 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:47:56.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:56 smithi084 ceph-mon[40024]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:47:56.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:56 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/2534590068' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T22:47:56.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:56 smithi084 ceph-mon[40024]: osdmap e395: 8 total, 8 up, 7 in 2024-09-20T22:47:56.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:56 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/3200951572' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:47:56.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:56 smithi084 ceph-mon[29774]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:47:56.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:56 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/2534590068' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T22:47:56.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:56 smithi084 ceph-mon[29774]: osdmap e395: 8 total, 8 up, 7 in 2024-09-20T22:47:56.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:56 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/3200951572' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:47:56.373 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:56.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:56.374 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:47:56.475 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:47:56.496 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:56.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:56.497 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:47:56.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:56 smithi154 ceph-mon[33264]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:47:56.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:56 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/2534590068' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T22:47:56.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:56 smithi154 ceph-mon[33264]: osdmap e395: 8 total, 8 up, 7 in 2024-09-20T22:47:56.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:56 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/3200951572' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:47:56.598 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:47:56.619 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:56.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:56.620 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:47:56.721 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:47:56.742 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:56.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:56.743 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:47:56.844 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:47:56.865 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:56.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:56.866 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:47:56.966 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:47:56.988 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:56.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:56.989 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:47:57.081 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:47:57.089 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:47:57.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:57.111 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:57.111 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:47:57.212 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:47:57.235 INFO:tasks.rados.rados.0.smithi084.stdout:1517: done (0 left) 2024-09-20T22:47:57.235 INFO:tasks.rados.rados.0.smithi084.stdout:1518: delete oid 5 current snap is 161 2024-09-20T22:47:57.236 INFO:tasks.rados.rados.0.smithi084.stdout:1518: done (0 left) 2024-09-20T22:47:57.236 INFO:tasks.rados.rados.0.smithi084.stdout:1519: snap_create 2024-09-20T22:47:57.239 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:57.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:57.240 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:47:57.341 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:47:57.363 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:57.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:57.364 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:47:57.364 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:57 smithi084 ceph-mon[40024]: 3.0 scrub starts 2024-09-20T22:47:57.364 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:57 smithi084 ceph-mon[40024]: 3.0 scrub ok 2024-09-20T22:47:57.364 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:57 smithi084 ceph-mon[40024]: 3.1 scrub starts 2024-09-20T22:47:57.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:57 smithi084 ceph-mon[40024]: 3.1 scrub ok 2024-09-20T22:47:57.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:57 smithi084 ceph-mon[40024]: pgmap v495: 168 pgs: 168 active+clean; 292 MiB data, 898 MiB used, 625 GiB / 626 GiB avail; 137 MiB/s rd, 25 MiB/s wr, 211 op/s; 36 MiB/s, 93 objects/s recovering 2024-09-20T22:47:57.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:57 smithi084 ceph-mon[40024]: 5.14 scrub starts 2024-09-20T22:47:57.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:57 smithi084 ceph-mon[40024]: 5.14 scrub ok 2024-09-20T22:47:57.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:57 smithi084 ceph-mon[40024]: osdmap e396: 8 total, 8 up, 7 in 2024-09-20T22:47:57.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:57 smithi084 ceph-mon[29774]: 3.0 scrub starts 2024-09-20T22:47:57.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:57 smithi084 ceph-mon[29774]: 3.0 scrub ok 2024-09-20T22:47:57.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:57 smithi084 ceph-mon[29774]: 3.1 scrub starts 2024-09-20T22:47:57.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:57 smithi084 ceph-mon[29774]: 3.1 scrub ok 2024-09-20T22:47:57.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:57 smithi084 ceph-mon[29774]: pgmap v495: 168 pgs: 168 active+clean; 292 MiB data, 898 MiB used, 625 GiB / 626 GiB avail; 137 MiB/s rd, 25 MiB/s wr, 211 op/s; 36 MiB/s, 93 objects/s recovering 2024-09-20T22:47:57.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:57 smithi084 ceph-mon[29774]: 5.14 scrub starts 2024-09-20T22:47:57.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:57 smithi084 ceph-mon[29774]: 5.14 scrub ok 2024-09-20T22:47:57.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:57 smithi084 ceph-mon[29774]: osdmap e396: 8 total, 8 up, 7 in 2024-09-20T22:47:57.464 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:47:57.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:57.487 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:57.487 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:47:57.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:57 smithi154 ceph-mon[33264]: 3.0 scrub starts 2024-09-20T22:47:57.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:57 smithi154 ceph-mon[33264]: 3.0 scrub ok 2024-09-20T22:47:57.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:57 smithi154 ceph-mon[33264]: 3.1 scrub starts 2024-09-20T22:47:57.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:57 smithi154 ceph-mon[33264]: 3.1 scrub ok 2024-09-20T22:47:57.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:57 smithi154 ceph-mon[33264]: pgmap v495: 168 pgs: 168 active+clean; 292 MiB data, 898 MiB used, 625 GiB / 626 GiB avail; 137 MiB/s rd, 25 MiB/s wr, 211 op/s; 36 MiB/s, 93 objects/s recovering 2024-09-20T22:47:57.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:57 smithi154 ceph-mon[33264]: 5.14 scrub starts 2024-09-20T22:47:57.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:57 smithi154 ceph-mon[33264]: 5.14 scrub ok 2024-09-20T22:47:57.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:57 smithi154 ceph-mon[33264]: osdmap e396: 8 total, 8 up, 7 in 2024-09-20T22:47:57.588 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:47:57.609 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:57.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:57.610 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:47:57.711 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:47:57.732 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:57.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:57.733 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:47:57.834 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:47:57.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:57.856 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:57.856 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:47:57.957 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:47:57.978 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:57.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:57.979 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:47:58.080 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:47:58.097 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-20T22:47:58.098 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:47:58.102 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:58.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:58.103 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:47:58.197 INFO:tasks.rados.rados.0.smithi084.stdout:1519: done (0 left) 2024-09-20T22:47:58.197 INFO:tasks.rados.rados.0.smithi084.stdout:1520: read oid 18 snap -1 2024-09-20T22:47:58.198 INFO:tasks.rados.rados.0.smithi084.stdout:1520: expect (ObjNum 535 snap 159 seq_num 535) 2024-09-20T22:47:58.198 INFO:tasks.rados.rados.0.smithi084.stdout:1521: snap_create 2024-09-20T22:47:58.201 INFO:tasks.rados.rados.0.smithi084.stdout:1520: expect (ObjNum 535 snap 159 seq_num 535) 2024-09-20T22:47:58.205 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:47:58.229 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:58.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:58.230 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:47:58.331 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:47:58.352 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:58.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:58.353 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:47:58.454 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:47:58.475 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:58.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:58.476 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:47:58.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:58 smithi154 ceph-mon[33264]: 1.2 scrub starts 2024-09-20T22:47:58.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:58 smithi154 ceph-mon[33264]: 1.2 scrub ok 2024-09-20T22:47:58.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:58 smithi154 ceph-mon[33264]: 7.19 scrub starts 2024-09-20T22:47:58.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:58 smithi154 ceph-mon[33264]: 7.19 scrub ok 2024-09-20T22:47:58.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:58 smithi154 ceph-mon[33264]: osdmap e397: 8 total, 8 up, 7 in 2024-09-20T22:47:58.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:58 smithi154 ceph-mon[33264]: osdmap e398: 8 total, 8 up, 7 in 2024-09-20T22:47:58.577 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:47:58.598 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:58.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:58.599 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:47:58.615 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:47:58 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: ::ffff:172.21.15.154 - - [20/Sep/2024:22:47:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-20T22:47:58.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:58 smithi084 ceph-mon[40024]: 1.2 scrub starts 2024-09-20T22:47:58.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:58 smithi084 ceph-mon[40024]: 1.2 scrub ok 2024-09-20T22:47:58.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:58 smithi084 ceph-mon[40024]: 7.19 scrub starts 2024-09-20T22:47:58.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:58 smithi084 ceph-mon[40024]: 7.19 scrub ok 2024-09-20T22:47:58.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:58 smithi084 ceph-mon[40024]: osdmap e397: 8 total, 8 up, 7 in 2024-09-20T22:47:58.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:58 smithi084 ceph-mon[40024]: osdmap e398: 8 total, 8 up, 7 in 2024-09-20T22:47:58.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:58 smithi084 ceph-mon[29774]: 1.2 scrub starts 2024-09-20T22:47:58.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:58 smithi084 ceph-mon[29774]: 1.2 scrub ok 2024-09-20T22:47:58.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:58 smithi084 ceph-mon[29774]: 7.19 scrub starts 2024-09-20T22:47:58.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:58 smithi084 ceph-mon[29774]: 7.19 scrub ok 2024-09-20T22:47:58.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:58 smithi084 ceph-mon[29774]: osdmap e397: 8 total, 8 up, 7 in 2024-09-20T22:47:58.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:58 smithi084 ceph-mon[29774]: osdmap e398: 8 total, 8 up, 7 in 2024-09-20T22:47:58.700 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:47:58.721 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:58.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:58.722 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:47:58.822 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:47:58.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:58.850 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:58.850 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:47:58.947 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.2 dump_blocked_ops 2024-09-20T22:47:58.951 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:47:58.976 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:58.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:58.978 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:47:59.012 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-20T22:47:59.078 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:47:59.103 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:59.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:59.104 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:47:59.205 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:47:59.206 INFO:tasks.rados.rados.0.smithi084.stdout:1520: done (1 left) 2024-09-20T22:47:59.206 INFO:tasks.rados.rados.0.smithi084.stdout:1521: done (0 left) 2024-09-20T22:47:59.206 INFO:tasks.rados.rados.0.smithi084.stdout:1522: snap_remove snap 150 2024-09-20T22:47:59.212 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:47:59.231 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:59.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:59.232 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:47:59.237 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:59 smithi084 ceph-mon[40024]: 7.f scrub starts 2024-09-20T22:47:59.237 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:59 smithi084 ceph-mon[40024]: 7.f scrub ok 2024-09-20T22:47:59.237 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:59 smithi084 ceph-mon[29774]: 7.f scrub starts 2024-09-20T22:47:59.237 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:59 smithi084 ceph-mon[29774]: 7.f scrub ok 2024-09-20T22:47:59.237 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:59 smithi084 ceph-mon[29774]: pgmap v498: 168 pgs: 168 active+clean; 292 MiB data, 898 MiB used, 625 GiB / 626 GiB avail 2024-09-20T22:47:59.237 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:59 smithi084 ceph-mon[29774]: 4.9 scrub starts 2024-09-20T22:47:59.237 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:59 smithi084 ceph-mon[29774]: 4.9 scrub ok 2024-09-20T22:47:59.333 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:47:59.357 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:59.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:59.358 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:47:59.459 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:47:59.482 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:59.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:59.484 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:47:59.491 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:59 smithi084 ceph-mon[40024]: pgmap v498: 168 pgs: 168 active+clean; 292 MiB data, 898 MiB used, 625 GiB / 626 GiB avail 2024-09-20T22:47:59.491 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:59 smithi084 ceph-mon[40024]: 4.9 scrub starts 2024-09-20T22:47:59.491 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:59 smithi084 ceph-mon[40024]: 4.9 scrub ok 2024-09-20T22:47:59.491 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:59 smithi084 ceph-mon[40024]: 5.15 scrub starts 2024-09-20T22:47:59.491 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:59 smithi084 ceph-mon[40024]: 5.15 scrub ok 2024-09-20T22:47:59.491 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:59 smithi084 ceph-mon[40024]: 6.f scrub starts 2024-09-20T22:47:59.491 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:59 smithi084 ceph-mon[40024]: 6.f scrub ok 2024-09-20T22:47:59.491 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:47:59 smithi084 ceph-mon[40024]: osdmap e399: 8 total, 8 up, 7 in 2024-09-20T22:47:59.491 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:59 smithi084 ceph-mon[29774]: 5.15 scrub starts 2024-09-20T22:47:59.492 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:59 smithi084 ceph-mon[29774]: 5.15 scrub ok 2024-09-20T22:47:59.492 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:59 smithi084 ceph-mon[29774]: 6.f scrub starts 2024-09-20T22:47:59.492 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:59 smithi084 ceph-mon[29774]: 6.f scrub ok 2024-09-20T22:47:59.492 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:47:59 smithi084 ceph-mon[29774]: osdmap e399: 8 total, 8 up, 7 in 2024-09-20T22:47:59.504 INFO:journalctl@ceph.iscsi.iscsi.a.smithi154.stdout:Sep 20 22:47:59 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-iscsi-iscsi-a[59730]: debug there is no tcmu-runner data available 2024-09-20T22:47:59.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:59 smithi154 ceph-mon[33264]: 7.f scrub starts 2024-09-20T22:47:59.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:59 smithi154 ceph-mon[33264]: 7.f scrub ok 2024-09-20T22:47:59.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:59 smithi154 ceph-mon[33264]: pgmap v498: 168 pgs: 168 active+clean; 292 MiB data, 898 MiB used, 625 GiB / 626 GiB avail 2024-09-20T22:47:59.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:59 smithi154 ceph-mon[33264]: 4.9 scrub starts 2024-09-20T22:47:59.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:59 smithi154 ceph-mon[33264]: 4.9 scrub ok 2024-09-20T22:47:59.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:59 smithi154 ceph-mon[33264]: 5.15 scrub starts 2024-09-20T22:47:59.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:59 smithi154 ceph-mon[33264]: 5.15 scrub ok 2024-09-20T22:47:59.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:59 smithi154 ceph-mon[33264]: 6.f scrub starts 2024-09-20T22:47:59.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:59 smithi154 ceph-mon[33264]: 6.f scrub ok 2024-09-20T22:47:59.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:47:59 smithi154 ceph-mon[33264]: osdmap e399: 8 total, 8 up, 7 in 2024-09-20T22:47:59.583 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-20T22:47:59.584 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:47:59.607 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:59.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:59.608 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:47:59.707 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-20T22:47:59.708 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:47:59.709 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:47:59.713 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:47:59.732 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:47:59.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:59.733 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:47:59.834 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:47:59.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:59.856 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:59.856 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:47:59.957 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:47:59.981 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:47:59.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:47:59.982 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:48:00.083 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:48:00.109 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:00.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:00.110 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:48:00.202 INFO:tasks.rados.rados.0.smithi084.stdout:1522: done (0 left) 2024-09-20T22:48:00.202 INFO:tasks.rados.rados.0.smithi084.stdout:1523: read oid 20 snap -1 2024-09-20T22:48:00.202 INFO:tasks.rados.rados.0.smithi084.stdout:1523: expect (ObjNum 449 snap 120 seq_num 449) 2024-09-20T22:48:00.202 INFO:tasks.rados.rados.0.smithi084.stdout:1524: write (excl) oid 37 current snap is 163 2024-09-20T22:48:00.202 INFO:tasks.rados.rados.0.smithi084.stdout:1524: seq_num 538 ranges {642166=628672,1849430=710842} 2024-09-20T22:48:00.209 INFO:tasks.rados.rados.0.smithi084.stdout:1524: writing smithi08477203-37 from 642166 to 1270838 tid 1 2024-09-20T22:48:00.210 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:48:00.218 INFO:tasks.rados.rados.0.smithi084.stdout:1524: writing smithi08477203-37 from 1849430 to 2560272 tid 2 2024-09-20T22:48:00.218 INFO:tasks.rados.rados.0.smithi084.stdout:1525: rollback oid 1 current snap is 163 2024-09-20T22:48:00.218 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 1 to 146 2024-09-20T22:48:00.218 INFO:tasks.rados.rados.0.smithi084.stdout:1523: expect (ObjNum 449 snap 120 seq_num 449) 2024-09-20T22:48:00.235 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:00.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:00.236 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:48:00.328 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:00 smithi084 ceph-mon[40024]: 4.e scrub starts 2024-09-20T22:48:00.329 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:00 smithi084 ceph-mon[40024]: 4.e scrub ok 2024-09-20T22:48:00.329 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:00 smithi084 ceph-mon[40024]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:48:00.329 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:00 smithi084 ceph-mon[40024]: 6.4 scrub starts 2024-09-20T22:48:00.329 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:00 smithi084 ceph-mon[40024]: 6.4 scrub ok 2024-09-20T22:48:00.329 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:00 smithi084 ceph-mon[40024]: osdmap e400: 8 total, 8 up, 7 in 2024-09-20T22:48:00.329 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:00 smithi084 ceph-mon[29774]: 4.e scrub starts 2024-09-20T22:48:00.329 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:00 smithi084 ceph-mon[29774]: 4.e scrub ok 2024-09-20T22:48:00.329 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:00 smithi084 ceph-mon[29774]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:48:00.329 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:00 smithi084 ceph-mon[29774]: 6.4 scrub starts 2024-09-20T22:48:00.329 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:00 smithi084 ceph-mon[29774]: 6.4 scrub ok 2024-09-20T22:48:00.329 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:00 smithi084 ceph-mon[29774]: osdmap e400: 8 total, 8 up, 7 in 2024-09-20T22:48:00.337 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:48:00.358 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:00.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:00.359 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:48:00.378 INFO:tasks.rados.rados.0.smithi084.stdout:1524: finishing write tid 1 to smithi08477203-37 2024-09-20T22:48:00.378 INFO:tasks.rados.rados.0.smithi084.stdout:1523: done (2 left) 2024-09-20T22:48:00.378 INFO:tasks.rados.rados.0.smithi084.stdout:1526: delete oid 9 current snap is 163 2024-09-20T22:48:00.378 INFO:tasks.rados.rados.0.smithi084.stdout:1525: finishing rollback tid 0 to smithi08477203-1 2024-09-20T22:48:00.379 INFO:tasks.rados.rados.0.smithi084.stdout:1525: finishing rollback tid 1 to smithi08477203-1 2024-09-20T22:48:00.379 INFO:tasks.rados.rados.0.smithi084.stdout:1524: finishing write tid 2 to smithi08477203-37 2024-09-20T22:48:00.379 INFO:tasks.rados.rados.0.smithi084.stdout:1524: finishing write tid 3 to smithi08477203-37 2024-09-20T22:48:00.379 INFO:tasks.rados.rados.0.smithi084.stdout:1525: finishing rollback tid 2 to smithi08477203-1 2024-09-20T22:48:00.379 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 1 v 539 (ObjNum 498 snap 142 seq_num 498) dirty exists 2024-09-20T22:48:00.379 INFO:tasks.rados.rados.0.smithi084.stdout:1524: finishing write tid 4 to smithi08477203-37 2024-09-20T22:48:00.379 INFO:tasks.rados.rados.0.smithi084.stdout:1524: oid 37 updating version 0 to 329 2024-09-20T22:48:00.379 INFO:tasks.rados.rados.0.smithi084.stdout:1524: oid 37 updating version 329 to 330 2024-09-20T22:48:00.379 INFO:tasks.rados.rados.0.smithi084.stdout:1524: oid 37 version 330 is already newer than 328 2024-09-20T22:48:00.379 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 37 v 330 (ObjNum 538 snap 163 seq_num 538) dirty exists 2024-09-20T22:48:00.379 INFO:tasks.rados.rados.0.smithi084.stdout:1524: left oid 37 (ObjNum 538 snap 163 seq_num 538) 2024-09-20T22:48:00.380 INFO:tasks.rados.rados.0.smithi084.stdout:1524: done (2 left) 2024-09-20T22:48:00.380 INFO:tasks.rados.rados.0.smithi084.stdout:1525: done (1 left) 2024-09-20T22:48:00.380 INFO:tasks.rados.rados.0.smithi084.stdout:1526: done (0 left) 2024-09-20T22:48:00.380 INFO:tasks.rados.rados.0.smithi084.stdout:1527: snap_remove snap 160 2024-09-20T22:48:00.460 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:48:00.466 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 5, 6, 7, 0] out_osds: [4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-20T22:48:00.466 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-20T22:48:00.466 INFO:tasks.thrashosds.thrasher:Adding osd 4 2024-09-20T22:48:00.467 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd in 4 2024-09-20T22:48:00.486 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:00.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:00.487 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:48:00.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:00 smithi154 ceph-mon[33264]: 4.e scrub starts 2024-09-20T22:48:00.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:00 smithi154 ceph-mon[33264]: 4.e scrub ok 2024-09-20T22:48:00.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:00 smithi154 ceph-mon[33264]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:48:00.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:00 smithi154 ceph-mon[33264]: 6.4 scrub starts 2024-09-20T22:48:00.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:00 smithi154 ceph-mon[33264]: 6.4 scrub ok 2024-09-20T22:48:00.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:00 smithi154 ceph-mon[33264]: osdmap e400: 8 total, 8 up, 7 in 2024-09-20T22:48:00.587 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.2 dump_historic_ops 2024-09-20T22:48:00.588 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:48:00.608 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:00.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:00.610 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:48:00.711 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:48:00.736 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:00.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:00.738 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:48:00.839 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:48:00.864 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:00.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:00.865 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:48:00.872 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:48:00.874 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:48:00.880 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:48:00.966 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:48:00.987 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:00.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:00.988 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:48:01.089 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:48:01.115 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:01.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:01.116 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:48:01.217 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:48:01.238 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:01.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:01.239 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:48:01.254 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is unset 2024-09-20T22:48:01.255 INFO:tasks.rados.rados.0.smithi084.stdout:1527: done (0 left) 2024-09-20T22:48:01.255 INFO:tasks.rados.rados.0.smithi084.stdout:1528: snap_create 2024-09-20T22:48:01.340 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:48:01.365 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:01.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:01.366 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:48:01.467 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:48:01.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:01.492 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:01.492 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:48:01.593 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:48:01.614 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:01 smithi084 ceph-mon[40024]: pgmap v501: 168 pgs: 2 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 163 active+clean; 294 MiB data, 897 MiB used, 625 GiB / 626 GiB avail; 2.7 MiB/s rd, 0 B/s wr, 4 op/s 2024-09-20T22:48:01.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:01 smithi084 ceph-mon[40024]: 4.4 scrub starts 2024-09-20T22:48:01.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:01 smithi084 ceph-mon[40024]: 4.4 scrub ok 2024-09-20T22:48:01.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:01 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/3012973394' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:48:01.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:01 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:48:01.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:01 smithi084 ceph-mon[29774]: pgmap v501: 168 pgs: 2 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 163 active+clean; 294 MiB data, 897 MiB used, 625 GiB / 626 GiB avail; 2.7 MiB/s rd, 0 B/s wr, 4 op/s 2024-09-20T22:48:01.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:01 smithi084 ceph-mon[29774]: 4.4 scrub starts 2024-09-20T22:48:01.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:01 smithi084 ceph-mon[29774]: 4.4 scrub ok 2024-09-20T22:48:01.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:01 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/3012973394' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:48:01.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:01 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:48:01.616 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:01.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:01.617 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:48:01.718 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:48:01.741 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:01.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:01.742 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:48:01.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:01 smithi154 ceph-mon[33264]: pgmap v501: 168 pgs: 2 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 163 active+clean; 294 MiB data, 897 MiB used, 625 GiB / 626 GiB avail; 2.7 MiB/s rd, 0 B/s wr, 4 op/s 2024-09-20T22:48:01.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:01 smithi154 ceph-mon[33264]: 4.4 scrub starts 2024-09-20T22:48:01.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:01 smithi154 ceph-mon[33264]: 4.4 scrub ok 2024-09-20T22:48:01.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:01 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/3012973394' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:48:01.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:01 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:48:01.843 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:48:01.867 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:01.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:01.868 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:48:01.969 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:48:01.992 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:01.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:01.993 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:48:02.094 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:48:02.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:02.120 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:02.120 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:48:02.201 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:48:02.201 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:48:02.216 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:48:02.217 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:48:02.221 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:48:02.233 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:48:02.233 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:48:02.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:02.244 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:02.244 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:48:02.250 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-20T22:48:02.250 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T22:48:02.265 INFO:tasks.rados.rados.0.smithi084.stdout:1528: done (0 left) 2024-09-20T22:48:02.265 INFO:tasks.rados.rados.0.smithi084.stdout:1529: write (excl) oid 6 current snap is 164 2024-09-20T22:48:02.265 INFO:tasks.rados.rados.0.smithi084.stdout:1529: seq_num 540 ranges {675780=495523,1682747=714511,3012769=63011} 2024-09-20T22:48:02.270 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:48:02.270 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:48:02.272 INFO:tasks.rados.rados.0.smithi084.stdout:1529: writing smithi08477203-6 from 675780 to 1171303 tid 1 2024-09-20T22:48:02.279 INFO:tasks.rados.rados.0.smithi084.stdout:1529: writing smithi08477203-6 from 1682747 to 2397258 tid 2 2024-09-20T22:48:02.284 INFO:tasks.rados.rados.0.smithi084.stdout:1529: writing smithi08477203-6 from 3012769 to 3075780 tid 3 2024-09-20T22:48:02.284 INFO:tasks.rados.rados.0.smithi084.stdout:1530: write (excl) oid 2 current snap is 164 2024-09-20T22:48:02.284 INFO:tasks.rados.rados.0.smithi084.stdout:1530: seq_num 541 ranges {692587=570772,2024876=574522,3040192=52395} 2024-09-20T22:48:02.290 INFO:tasks.rados.rados.0.smithi084.stdout:1530: writing smithi08477203-2 from 692587 to 1263359 tid 1 2024-09-20T22:48:02.291 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:48:02.291 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:48:02.298 INFO:tasks.rados.rados.0.smithi084.stdout:1530: writing smithi08477203-2 from 2024876 to 2599398 tid 2 2024-09-20T22:48:02.302 INFO:tasks.rados.rados.0.smithi084.stdout:1530: writing smithi08477203-2 from 3040192 to 3092587 tid 3 2024-09-20T22:48:02.302 INFO:tasks.rados.rados.0.smithi084.stdout:1531: delete oid 22 current snap is 164 2024-09-20T22:48:02.302 INFO:tasks.rados.rados.0.smithi084.stdout:1529: finishing write tid 1 to smithi08477203-6 2024-09-20T22:48:02.302 INFO:tasks.rados.rados.0.smithi084.stdout:1529: finishing write tid 2 to smithi08477203-6 2024-09-20T22:48:02.302 INFO:tasks.rados.rados.0.smithi084.stdout:1529: finishing write tid 3 to smithi08477203-6 2024-09-20T22:48:02.302 INFO:tasks.rados.rados.0.smithi084.stdout:1529: finishing write tid 4 to smithi08477203-6 2024-09-20T22:48:02.302 INFO:tasks.rados.rados.0.smithi084.stdout:1529: finishing write tid 5 to smithi08477203-6 2024-09-20T22:48:02.302 INFO:tasks.rados.rados.0.smithi084.stdout:1529: oid 6 updating version 0 to 304 2024-09-20T22:48:02.302 INFO:tasks.rados.rados.0.smithi084.stdout:1529: oid 6 version 304 is already newer than 303 2024-09-20T22:48:02.302 INFO:tasks.rados.rados.0.smithi084.stdout:1529: oid 6 updating version 304 to 305 2024-09-20T22:48:02.302 INFO:tasks.rados.rados.0.smithi084.stdout:1529: oid 6 updating version 305 to 306 2024-09-20T22:48:02.302 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 6 v 306 (ObjNum 540 snap 164 seq_num 540) dirty exists 2024-09-20T22:48:02.303 INFO:tasks.rados.rados.0.smithi084.stdout:1529: left oid 6 (ObjNum 540 snap 164 seq_num 540) 2024-09-20T22:48:02.303 INFO:tasks.rados.rados.0.smithi084.stdout:1530: finishing write tid 1 to smithi08477203-2 2024-09-20T22:48:02.303 INFO:tasks.rados.rados.0.smithi084.stdout:1530: finishing write tid 2 to smithi08477203-2 2024-09-20T22:48:02.303 INFO:tasks.rados.rados.0.smithi084.stdout:1530: finishing write tid 3 to smithi08477203-2 2024-09-20T22:48:02.304 INFO:tasks.rados.rados.0.smithi084.stdout:1530: finishing write tid 4 to smithi08477203-2 2024-09-20T22:48:02.305 INFO:tasks.rados.rados.0.smithi084.stdout:1530: finishing write tid 5 to smithi08477203-2 2024-09-20T22:48:02.305 INFO:tasks.rados.rados.0.smithi084.stdout:1530: oid 2 updating version 0 to 175 2024-09-20T22:48:02.305 INFO:tasks.rados.rados.0.smithi084.stdout:1530: oid 2 updating version 175 to 176 2024-09-20T22:48:02.305 INFO:tasks.rados.rados.0.smithi084.stdout:1530: oid 2 version 176 is already newer than 173 2024-09-20T22:48:02.305 INFO:tasks.rados.rados.0.smithi084.stdout:1530: oid 2 version 176 is already newer than 174 2024-09-20T22:48:02.305 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 2 v 176 (ObjNum 541 snap 164 seq_num 541) dirty exists 2024-09-20T22:48:02.305 INFO:tasks.rados.rados.0.smithi084.stdout:1530: left oid 2 (ObjNum 541 snap 164 seq_num 541) 2024-09-20T22:48:02.306 INFO:tasks.rados.rados.0.smithi084.stdout:1529: done (2 left) 2024-09-20T22:48:02.306 INFO:tasks.rados.rados.0.smithi084.stdout:1530: done (1 left) 2024-09-20T22:48:02.306 INFO:tasks.rados.rados.0.smithi084.stdout:1531: done (0 left) 2024-09-20T22:48:02.306 INFO:tasks.rados.rados.0.smithi084.stdout:1532: read oid 49 snap -1 2024-09-20T22:48:02.306 INFO:tasks.rados.rados.0.smithi084.stdout:1532: expect (ObjNum 495 snap 139 seq_num 495) 2024-09-20T22:48:02.306 INFO:tasks.rados.rados.0.smithi084.stdout:1533: write oid 6 current snap is 164 2024-09-20T22:48:02.306 INFO:tasks.rados.rados.0.smithi084.stdout:1533: seq_num 543 ranges {726201=721270,2141840=578191} 2024-09-20T22:48:02.310 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:48:02.311 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:48:02.313 INFO:tasks.rados.rados.0.smithi084.stdout:1533: writing smithi08477203-6 from 726201 to 1447471 tid 1 2024-09-20T22:48:02.320 INFO:tasks.rados.rados.0.smithi084.stdout:1533: writing smithi08477203-6 from 2141840 to 2720031 tid 2 2024-09-20T22:48:02.320 INFO:tasks.rados.rados.0.smithi084.stdout:1534: rollback oid 12 current snap is 164 2024-09-20T22:48:02.320 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 12 to 140 2024-09-20T22:48:02.320 INFO:tasks.rados.rados.0.smithi084.stdout:1532: expect (ObjNum 495 snap 139 seq_num 495) 2024-09-20T22:48:02.330 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:48:02.330 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:48:02.345 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:48:02.369 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:02.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:02.370 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:48:02.418 INFO:tasks.rados.rados.0.smithi084.stdout:1533: finishing write tid 1 to smithi08477203-6 2024-09-20T22:48:02.418 INFO:tasks.rados.rados.0.smithi084.stdout:1532: done (2 left) 2024-09-20T22:48:02.419 INFO:tasks.rados.rados.0.smithi084.stdout:1535: read oid 45 snap -1 2024-09-20T22:48:02.419 INFO:tasks.rados.rados.0.smithi084.stdout:1535: expect (ObjNum 529 snap 152 seq_num 529) 2024-09-20T22:48:02.419 INFO:tasks.rados.rados.0.smithi084.stdout:1536: write oid 10 current snap is 164 2024-09-20T22:48:02.419 INFO:tasks.rados.rados.0.smithi084.stdout:1533: finishing write tid 2 to smithi08477203-6 2024-09-20T22:48:02.419 INFO:tasks.rados.rados.0.smithi084.stdout:1533: finishing write tid 3 to smithi08477203-6 2024-09-20T22:48:02.419 INFO:tasks.rados.rados.0.smithi084.stdout:1536: seq_num 544 ranges {743008=796519,2200322=721849} 2024-09-20T22:48:02.428 INFO:tasks.rados.rados.0.smithi084.stdout:1536: writing smithi08477203-10 from 743008 to 1539527 tid 1 2024-09-20T22:48:02.438 INFO:tasks.rados.rados.0.smithi084.stdout:1536: writing smithi08477203-10 from 2200322 to 2922171 tid 2 2024-09-20T22:48:02.438 INFO:tasks.rados.rados.0.smithi084.stdout:1537: read oid 42 snap -1 2024-09-20T22:48:02.438 INFO:tasks.rados.rados.0.smithi084.stdout:1537: expect deleted 2024-09-20T22:48:02.438 INFO:tasks.rados.rados.0.smithi084.stdout:1533: finishing write tid 4 to smithi08477203-6 2024-09-20T22:48:02.438 INFO:tasks.rados.rados.0.smithi084.stdout:1533: oid 6 updating version 0 to 308 2024-09-20T22:48:02.438 INFO:tasks.rados.rados.0.smithi084.stdout:1533: oid 6 updating version 308 to 309 2024-09-20T22:48:02.438 INFO:tasks.rados.rados.0.smithi084.stdout:1533: oid 6 version 309 is already newer than 307 2024-09-20T22:48:02.439 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 6 v 309 (ObjNum 543 snap 164 seq_num 543) dirty exists 2024-09-20T22:48:02.439 INFO:tasks.rados.rados.0.smithi084.stdout:1533: left oid 6 (ObjNum 543 snap 164 seq_num 543) 2024-09-20T22:48:02.439 INFO:tasks.rados.rados.0.smithi084.stdout:1534: finishing rollback tid 0 to smithi08477203-12 2024-09-20T22:48:02.439 INFO:tasks.rados.rados.0.smithi084.stdout:1534: finishing rollback tid 1 to smithi08477203-12 2024-09-20T22:48:02.439 INFO:tasks.rados.rados.0.smithi084.stdout:1538: read oid 25 snap -1 2024-09-20T22:48:02.439 INFO:tasks.rados.rados.0.smithi084.stdout:1538: expect (ObjNum 458 snap 126 seq_num 458) 2024-09-20T22:48:02.439 INFO:tasks.rados.rados.0.smithi084.stdout:1533: done (5 left) 2024-09-20T22:48:02.439 INFO:tasks.rados.rados.0.smithi084.stdout:1539: rollback oid 19 current snap is 164 2024-09-20T22:48:02.439 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 19 to 157 2024-09-20T22:48:02.439 INFO:tasks.rados.rados.0.smithi084.stdout:1534: finishing rollback tid 2 to smithi08477203-12 2024-09-20T22:48:02.439 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 12 v 358 (ObjNum 478 snap 135 seq_num 478) dirty exists 2024-09-20T22:48:02.439 INFO:tasks.rados.rados.0.smithi084.stdout:1535: expect (ObjNum 529 snap 152 seq_num 529) 2024-09-20T22:48:02.447 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-20T22:48:02.447 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2024-09-20T22:48:02.447 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2024-09-20T22:48:02.447 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [ 2024-09-20T22:48:02.447 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.448 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:901 7.e 7:7c62ced4:::smithi08477203-17:head [copy-from ver 48] snapc 1e=[] ondisk+write+known_if_redirected+supports_pool_eio e156)", 2024-09-20T22:48:02.448 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:43:55.624762+0000", 2024-09-20T22:48:02.448 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 246.81997597200001, 2024-09-20T22:48:02.448 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.021835856000000001, 2024-09-20T22:48:02.448 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:48:02.448 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:48:02.448 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:48:02.448 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:48:02.448 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:48:02.448 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:48:02.448 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 901 2024-09-20T22:48:02.448 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.448 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:48:02.449 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.449 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:48:02.449 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.624762+0000", 2024-09-20T22:48:02.449 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:48:02.449 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.449 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.449 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:48:02.449 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.624762+0000", 2024-09-20T22:48:02.449 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:48:02.449 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.449 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.449 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:48:02.449 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.624765+0000", 2024-09-20T22:48:02.449 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.57e-06 2024-09-20T22:48:02.450 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.450 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.450 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:48:02.450 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.624776+0000", 2024-09-20T22:48:02.450 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.078e-05 2024-09-20T22:48:02.450 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.450 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.450 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:48:02.450 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.624803+0000", 2024-09-20T22:48:02.450 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.6769999999999999e-05 2024-09-20T22:48:02.450 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.450 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.450 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:48:02.451 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.624815+0000", 2024-09-20T22:48:02.451 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2459e-05 2024-09-20T22:48:02.451 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.451 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.451 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:48:02.451 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.624936+0000", 2024-09-20T22:48:02.451 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000120467 2024-09-20T22:48:02.451 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.451 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.451 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:48:02.451 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.625046+0000", 2024-09-20T22:48:02.451 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000110222 2024-09-20T22:48:02.451 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.452 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.452 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 0,5", 2024-09-20T22:48:02.452 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.635547+0000", 2024-09-20T22:48:02.452 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010501765999999999 2024-09-20T22:48:02.452 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.452 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.452 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:48:02.452 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.644332+0000", 2024-09-20T22:48:02.452 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0087847729999999992 2024-09-20T22:48:02.452 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.452 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.452 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:48:02.452 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.646168+0000", 2024-09-20T22:48:02.452 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0018360010000000001 2024-09-20T22:48:02.453 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.453 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.453 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:48:02.453 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.646512+0000", 2024-09-20T22:48:02.453 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000343269 2024-09-20T22:48:02.453 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.453 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.453 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:48:02.453 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.646550+0000", 2024-09-20T22:48:02.453 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.8383000000000002e-05 2024-09-20T22:48:02.453 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.453 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.454 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:48:02.454 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.646598+0000", 2024-09-20T22:48:02.454 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.8396000000000002e-05 2024-09-20T22:48:02.454 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:02.454 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:48:02.454 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:02.454 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.454 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.454 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:902 7.e 7:7c62ced4:::smithi08477203-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e156)", 2024-09-20T22:48:02.454 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:43:55.624853+0000", 2024-09-20T22:48:02.454 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 246.81988589400001, 2024-09-20T22:48:02.454 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.015323603999999999, 2024-09-20T22:48:02.454 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:48:02.454 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:48:02.455 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-20T22:48:02.455 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:48:02.455 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:48:02.455 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:48:02.455 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 902 2024-09-20T22:48:02.455 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.455 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:48:02.455 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.455 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:48:02.455 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.624853+0000", 2024-09-20T22:48:02.455 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:48:02.455 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.455 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.455 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:48:02.456 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.624853+0000", 2024-09-20T22:48:02.456 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:48:02.456 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.456 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.456 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:48:02.456 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.624853+0000", 2024-09-20T22:48:02.456 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.1399999999999997e-07 2024-09-20T22:48:02.456 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.456 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.456 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:48:02.456 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.624859+0000", 2024-09-20T22:48:02.456 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.9610000000000002e-06 2024-09-20T22:48:02.457 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.457 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.457 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:48:02.457 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.624887+0000", 2024-09-20T22:48:02.457 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.7597e-05 2024-09-20T22:48:02.457 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.457 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.457 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:48:02.457 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.624894+0000", 2024-09-20T22:48:02.457 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.469e-06 2024-09-20T22:48:02.457 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.457 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.457 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:48:02.457 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.625219+0000", 2024-09-20T22:48:02.458 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00032538499999999998 2024-09-20T22:48:02.458 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.458 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.458 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for blocked object", 2024-09-20T22:48:02.458 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.625258+0000", 2024-09-20T22:48:02.458 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.8467000000000003e-05 2024-09-20T22:48:02.458 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.458 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.458 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:48:02.458 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.640050+0000", 2024-09-20T22:48:02.458 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0147923 2024-09-20T22:48:02.458 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.458 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.459 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:48:02.459 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.640112+0000", 2024-09-20T22:48:02.459 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.1723000000000005e-05 2024-09-20T22:48:02.459 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.459 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.459 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:48:02.459 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.640176+0000", 2024-09-20T22:48:02.459 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.4188000000000003e-05 2024-09-20T22:48:02.459 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:02.459 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:48:02.459 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:02.459 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.459 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.459 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24652.0:1420 7.19 e189/139 7:9cc0b7e2:::smithi08477203-46:head v 189'99, mlcod=174'95)", 2024-09-20T22:48:02.460 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:44:28.150258+0000", 2024-09-20T22:48:02.460 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 214.294480198, 2024-09-20T22:48:02.460 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014180626, 2024-09-20T22:48:02.460 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:48:02.460 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:48:02.460 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:48:02.460 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:48:02.460 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.460 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:48:02.460 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.150258+0000", 2024-09-20T22:48:02.460 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:48:02.460 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.460 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.461 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:48:02.461 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.150258+0000", 2024-09-20T22:48:02.461 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:48:02.461 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.461 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.461 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:48:02.461 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.150259+0000", 2024-09-20T22:48:02.461 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.99e-07 2024-09-20T22:48:02.461 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.461 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.461 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:48:02.461 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.150674+0000", 2024-09-20T22:48:02.461 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00041528000000000002 2024-09-20T22:48:02.461 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.462 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.462 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:48:02.462 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.150688+0000", 2024-09-20T22:48:02.462 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3399e-05 2024-09-20T22:48:02.463 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.463 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.463 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:48:02.463 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.150699+0000", 2024-09-20T22:48:02.463 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1022999999999999e-05 2024-09-20T22:48:02.463 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.463 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.463 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:48:02.463 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.150772+0000", 2024-09-20T22:48:02.463 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.3455999999999996e-05 2024-09-20T22:48:02.463 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.464 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.464 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:48:02.464 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.150798+0000", 2024-09-20T22:48:02.464 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.5777000000000001e-05 2024-09-20T22:48:02.464 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.464 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.464 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:48:02.464 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.164390+0000", 2024-09-20T22:48:02.464 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013591789999999999 2024-09-20T22:48:02.464 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.464 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.464 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:48:02.464 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.164439+0000", 2024-09-20T22:48:02.464 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.9202000000000001e-05 2024-09-20T22:48:02.465 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:02.465 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:48:02.465 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:02.465 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.465 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.465 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24652.0:1421 7.19 e189/139 7:9cc0b7e2:::smithi08477203-46:head v 189'100, mlcod=174'95)", 2024-09-20T22:48:02.465 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:44:28.156805+0000", 2024-09-20T22:48:02.465 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 214.28793334, 2024-09-20T22:48:02.465 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013180661999999999, 2024-09-20T22:48:02.465 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:48:02.465 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:48:02.465 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:48:02.465 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:48:02.465 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.466 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:48:02.466 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.156805+0000", 2024-09-20T22:48:02.466 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:48:02.466 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.466 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.466 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:48:02.466 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.156805+0000", 2024-09-20T22:48:02.466 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:48:02.466 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.466 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.466 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:48:02.466 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.156806+0000", 2024-09-20T22:48:02.466 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.074e-06 2024-09-20T22:48:02.467 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.467 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.467 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:48:02.467 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.157126+0000", 2024-09-20T22:48:02.467 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00031991399999999997 2024-09-20T22:48:02.467 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.467 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.467 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:48:02.467 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.157136+0000", 2024-09-20T22:48:02.467 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.4499999999999993e-06 2024-09-20T22:48:02.467 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.467 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.467 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:48:02.467 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.157143+0000", 2024-09-20T22:48:02.468 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.7649999999999993e-06 2024-09-20T22:48:02.468 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.468 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.468 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:48:02.468 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.157188+0000", 2024-09-20T22:48:02.468 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.4885000000000001e-05 2024-09-20T22:48:02.468 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.468 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.468 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:48:02.468 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.157208+0000", 2024-09-20T22:48:02.468 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.0015000000000001e-05 2024-09-20T22:48:02.468 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.468 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.468 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:48:02.469 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.169936+0000", 2024-09-20T22:48:02.469 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.012728204 2024-09-20T22:48:02.469 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.469 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.469 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:48:02.469 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.169986+0000", 2024-09-20T22:48:02.469 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.9354999999999998e-05 2024-09-20T22:48:02.469 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:02.469 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:48:02.469 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:02.469 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.469 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.469 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24652.0:1654 7.19 e206/139 7:9970b578:::smithi08477203-15:head v 206'114, mlcod=205'111)", 2024-09-20T22:48:02.469 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:44:45.167994+0000", 2024-09-20T22:48:02.469 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 197.27674479000001, 2024-09-20T22:48:02.470 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.01345262, 2024-09-20T22:48:02.470 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:48:02.470 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:48:02.470 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:48:02.470 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:48:02.470 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.470 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:48:02.470 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:45.167994+0000", 2024-09-20T22:48:02.470 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:48:02.470 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.470 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.470 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:48:02.470 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:45.167994+0000", 2024-09-20T22:48:02.471 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:48:02.471 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.471 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.471 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:48:02.471 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:45.167996+0000", 2024-09-20T22:48:02.471 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.5100000000000001e-06 2024-09-20T22:48:02.471 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.471 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.471 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:48:02.471 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:45.175169+0000", 2024-09-20T22:48:02.471 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0071732740000000003 2024-09-20T22:48:02.471 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.471 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.472 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:48:02.472 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:45.175208+0000", 2024-09-20T22:48:02.472 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.8597000000000003e-05 2024-09-20T22:48:02.472 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.472 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.472 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:48:02.472 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:45.175223+0000", 2024-09-20T22:48:02.472 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.4627000000000001e-05 2024-09-20T22:48:02.472 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.472 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.472 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:48:02.472 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:45.175304+0000", 2024-09-20T22:48:02.472 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.1111000000000003e-05 2024-09-20T22:48:02.472 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.473 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.473 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:48:02.473 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:45.175336+0000", 2024-09-20T22:48:02.473 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.1945e-05 2024-09-20T22:48:02.473 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.473 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.473 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:48:02.473 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:45.181392+0000", 2024-09-20T22:48:02.473 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0060566630000000003 2024-09-20T22:48:02.473 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.473 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.473 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:48:02.473 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:45.181446+0000", 2024-09-20T22:48:02.473 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.3893000000000001e-05 2024-09-20T22:48:02.474 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:02.474 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:48:02.474 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:02.474 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.474 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.474 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24652.0:1799 7.17 e229/139 7:ea720b85:::smithi08477203-26:head v 229'168, mlcod=222'164)", 2024-09-20T22:48:02.474 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:45:08.171867+0000", 2024-09-20T22:48:02.474 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 174.27287157200001, 2024-09-20T22:48:02.474 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014913454, 2024-09-20T22:48:02.474 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:48:02.474 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:48:02.474 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:48:02.474 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:48:02.474 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.475 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:48:02.475 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.171867+0000", 2024-09-20T22:48:02.475 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:48:02.475 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.475 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.475 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:48:02.475 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.171867+0000", 2024-09-20T22:48:02.475 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:48:02.475 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.475 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.475 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:48:02.475 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.171867+0000", 2024-09-20T22:48:02.475 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.3799999999999999e-07 2024-09-20T22:48:02.475 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.476 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.476 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:48:02.476 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.173016+0000", 2024-09-20T22:48:02.476 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0011485740000000001 2024-09-20T22:48:02.476 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.476 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.476 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:48:02.476 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.173024+0000", 2024-09-20T22:48:02.476 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.4780000000000008e-06 2024-09-20T22:48:02.476 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.476 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.476 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:48:02.476 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.173034+0000", 2024-09-20T22:48:02.476 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.7860000000000008e-06 2024-09-20T22:48:02.476 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.477 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.477 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:48:02.477 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.173098+0000", 2024-09-20T22:48:02.477 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.4405000000000004e-05 2024-09-20T22:48:02.477 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.477 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.477 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:48:02.477 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.173125+0000", 2024-09-20T22:48:02.477 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.6950000000000001e-05 2024-09-20T22:48:02.477 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.477 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.477 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:48:02.477 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.186711+0000", 2024-09-20T22:48:02.478 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.01358608 2024-09-20T22:48:02.478 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.478 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.478 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:48:02.478 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.186780+0000", 2024-09-20T22:48:02.478 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.8942999999999994e-05 2024-09-20T22:48:02.478 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:02.478 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:48:02.478 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:02.478 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.478 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.478 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:2321 7.8 7:12e9c8db:::smithi08477203-11:head [rollback 3e] snapc 55=[55,53,52,51,50,4b,47,3e] ondisk+write+known_if_redirected+supports_pool_eio e258)", 2024-09-20T22:48:02.478 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:45:37.176065+0000", 2024-09-20T22:48:02.478 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 145.26867369799999, 2024-09-20T22:48:02.479 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.012489495, 2024-09-20T22:48:02.479 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:48:02.479 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:48:02.479 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:48:02.479 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:48:02.479 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:48:02.479 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:48:02.479 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 2321 2024-09-20T22:48:02.479 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.479 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:48:02.479 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.479 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:48:02.479 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.176065+0000", 2024-09-20T22:48:02.480 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:48:02.480 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.480 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.480 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:48:02.480 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.176065+0000", 2024-09-20T22:48:02.480 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:48:02.480 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.480 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.480 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:48:02.480 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.176065+0000", 2024-09-20T22:48:02.480 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.3e-07 2024-09-20T22:48:02.480 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.480 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.480 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:48:02.480 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.176069+0000", 2024-09-20T22:48:02.481 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.8010000000000001e-06 2024-09-20T22:48:02.481 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.481 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.481 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:48:02.481 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.176076+0000", 2024-09-20T22:48:02.481 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.6599999999999998e-06 2024-09-20T22:48:02.481 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.481 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.481 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:48:02.481 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.176086+0000", 2024-09-20T22:48:02.481 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.9720000000000004e-06 2024-09-20T22:48:02.481 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.481 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.481 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:48:02.482 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.176134+0000", 2024-09-20T22:48:02.482 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.7778999999999998e-05 2024-09-20T22:48:02.482 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.482 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.482 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting_for_map not empty", 2024-09-20T22:48:02.482 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.176139+0000", 2024-09-20T22:48:02.482 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.7699999999999998e-06 2024-09-20T22:48:02.482 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.482 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.482 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:48:02.482 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.181825+0000", 2024-09-20T22:48:02.482 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.005685921 2024-09-20T22:48:02.482 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.482 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.483 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:48:02.483 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.181919+0000", 2024-09-20T22:48:02.483 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.3630999999999999e-05 2024-09-20T22:48:02.483 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.483 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.483 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 3,7", 2024-09-20T22:48:02.483 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.182118+0000", 2024-09-20T22:48:02.483 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00019958500000000001 2024-09-20T22:48:02.483 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.483 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.483 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:48:02.483 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.185052+0000", 2024-09-20T22:48:02.483 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0029336620000000001 2024-09-20T22:48:02.484 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.484 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.484 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:48:02.484 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.185756+0000", 2024-09-20T22:48:02.484 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00070376499999999997 2024-09-20T22:48:02.484 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.484 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.484 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:48:02.484 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.188120+0000", 2024-09-20T22:48:02.484 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0023645739999999999 2024-09-20T22:48:02.484 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.484 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.485 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:48:02.485 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.188207+0000", 2024-09-20T22:48:02.485 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.6562000000000002e-05 2024-09-20T22:48:02.485 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.485 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.485 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:48:02.485 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.188554+0000", 2024-09-20T22:48:02.485 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00034718299999999999 2024-09-20T22:48:02.485 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:02.486 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:48:02.486 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:02.486 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.486 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.486 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:2322 7.8 7:12e9c8db:::smithi08477203-11:head [append 0~0] snapc 55=[55,53,52,51,50,4b,47,3e] ondisk+write+known_if_redirected+supports_pool_eio e258)", 2024-09-20T22:48:02.486 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:45:37.176152+0000", 2024-09-20T22:48:02.486 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 145.268586782, 2024-09-20T22:48:02.486 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013033472000000001, 2024-09-20T22:48:02.486 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:48:02.486 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:48:02.486 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:48:02.486 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:48:02.486 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:48:02.487 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:48:02.487 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 2322 2024-09-20T22:48:02.487 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.487 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:48:02.487 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.487 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:48:02.487 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.176152+0000", 2024-09-20T22:48:02.487 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:48:02.487 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.487 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.487 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:48:02.487 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.176152+0000", 2024-09-20T22:48:02.487 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:48:02.487 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.488 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.488 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:48:02.488 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.176152+0000", 2024-09-20T22:48:02.488 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.8899999999999999e-07 2024-09-20T22:48:02.488 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.488 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.488 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:48:02.488 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.176156+0000", 2024-09-20T22:48:02.488 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.2669999999999998e-06 2024-09-20T22:48:02.488 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.488 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.489 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:48:02.489 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.176162+0000", 2024-09-20T22:48:02.489 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.1589999999999998e-06 2024-09-20T22:48:02.489 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.489 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.489 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:48:02.489 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.176167+0000", 2024-09-20T22:48:02.489 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.3040000000000004e-06 2024-09-20T22:48:02.489 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.489 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.489 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:48:02.489 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.176212+0000", 2024-09-20T22:48:02.490 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.5448999999999998e-05 2024-09-20T22:48:02.490 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.490 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.490 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting_for_map not empty", 2024-09-20T22:48:02.490 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.176218+0000", 2024-09-20T22:48:02.490 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.5520000000000004e-06 2024-09-20T22:48:02.490 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.490 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.490 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:48:02.490 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.182721+0000", 2024-09-20T22:48:02.490 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0065028550000000001 2024-09-20T22:48:02.490 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.490 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.490 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:48:02.491 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.182787+0000", 2024-09-20T22:48:02.491 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.5995999999999996e-05 2024-09-20T22:48:02.491 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.491 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.491 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 3,7", 2024-09-20T22:48:02.491 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.183002+0000", 2024-09-20T22:48:02.491 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00021516699999999999 2024-09-20T22:48:02.491 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.491 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.491 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:48:02.491 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.185137+0000", 2024-09-20T22:48:02.491 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0021350570000000001 2024-09-20T22:48:02.491 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.491 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.492 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:48:02.492 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.185875+0000", 2024-09-20T22:48:02.492 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00073794799999999999 2024-09-20T22:48:02.492 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.492 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.492 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:48:02.492 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.188620+0000", 2024-09-20T22:48:02.492 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0027446229999999999 2024-09-20T22:48:02.492 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.492 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.492 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:48:02.492 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.189088+0000", 2024-09-20T22:48:02.492 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00046794499999999999 2024-09-20T22:48:02.493 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.493 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.493 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:48:02.493 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.189185+0000", 2024-09-20T22:48:02.493 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.7560999999999996e-05 2024-09-20T22:48:02.493 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:02.493 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:48:02.493 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:02.493 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.493 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.493 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24652.0:2318 7.5 e258/137 7:a160b5d4:::smithi08477203-16:head v 258'43, mlcod=239'39)", 2024-09-20T22:48:02.493 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:45:37.192419+0000", 2024-09-20T22:48:02.493 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 145.25231948499999, 2024-09-20T22:48:02.494 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013974723, 2024-09-20T22:48:02.494 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:48:02.494 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:48:02.494 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:48:02.494 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:48:02.494 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.494 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:48:02.494 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.192419+0000", 2024-09-20T22:48:02.494 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:48:02.494 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.494 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.494 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:48:02.494 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.192419+0000", 2024-09-20T22:48:02.495 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:48:02.495 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.495 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.495 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:48:02.495 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.192420+0000", 2024-09-20T22:48:02.495 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.3799999999999997e-07 2024-09-20T22:48:02.495 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.495 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.495 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:48:02.495 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.204088+0000", 2024-09-20T22:48:02.495 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011668883 2024-09-20T22:48:02.495 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.495 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.495 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:48:02.496 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.204104+0000", 2024-09-20T22:48:02.496 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.5437000000000001e-05 2024-09-20T22:48:02.496 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.496 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.496 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:48:02.496 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.204112+0000", 2024-09-20T22:48:02.496 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.2519999999999999e-06 2024-09-20T22:48:02.496 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.496 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.496 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:48:02.496 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.204168+0000", 2024-09-20T22:48:02.496 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.5977999999999998e-05 2024-09-20T22:48:02.497 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.497 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.497 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:48:02.497 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.204188+0000", 2024-09-20T22:48:02.497 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.9400999999999999e-05 2024-09-20T22:48:02.497 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.497 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.497 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:48:02.497 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.206340+0000", 2024-09-20T22:48:02.497 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0021527159999999998 2024-09-20T22:48:02.497 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.497 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.497 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:48:02.497 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.206394+0000", 2024-09-20T22:48:02.497 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.3418000000000003e-05 2024-09-20T22:48:02.498 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:02.498 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:48:02.498 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:02.498 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.498 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.498 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(osd.6.0:726 7.19 e260/139 7:9eac4560:::smithi08477203-30:4c v 260'159, mlcod=259'156)", 2024-09-20T22:48:02.498 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:45:39.197025+0000", 2024-09-20T22:48:02.498 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 143.24771357500001, 2024-09-20T22:48:02.498 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.018150454, 2024-09-20T22:48:02.498 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:48:02.498 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:48:02.498 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:48:02.498 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:48:02.499 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.499 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:48:02.499 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.197025+0000", 2024-09-20T22:48:02.499 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:48:02.499 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.499 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.499 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:48:02.499 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.197025+0000", 2024-09-20T22:48:02.499 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:48:02.499 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.499 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.499 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:48:02.499 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.197025+0000", 2024-09-20T22:48:02.500 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.7599999999999998e-07 2024-09-20T22:48:02.500 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.500 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.500 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:48:02.500 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.197033+0000", 2024-09-20T22:48:02.500 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.9119999999999998e-06 2024-09-20T22:48:02.500 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.500 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.500 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:48:02.500 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.197042+0000", 2024-09-20T22:48:02.500 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.3349999999999996e-06 2024-09-20T22:48:02.500 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.500 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.500 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:48:02.501 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.197050+0000", 2024-09-20T22:48:02.501 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.7230000000000004e-06 2024-09-20T22:48:02.501 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.501 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.501 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:48:02.501 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.200756+0000", 2024-09-20T22:48:02.501 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0037059749999999998 2024-09-20T22:48:02.501 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.501 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.501 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:48:02.501 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.200778+0000", 2024-09-20T22:48:02.501 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.2140000000000001e-05 2024-09-20T22:48:02.501 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.501 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.502 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:48:02.502 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.215131+0000", 2024-09-20T22:48:02.502 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014353054000000001 2024-09-20T22:48:02.502 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.502 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.502 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:48:02.502 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.215175+0000", 2024-09-20T22:48:02.502 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.4039000000000003e-05 2024-09-20T22:48:02.502 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:02.502 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:48:02.502 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:02.502 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.502 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.503 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(osd.7.0:152 7.5 e269/260 7:a160b5d4:::smithi08477203-16:55 v 269'55, mlcod=263'51)", 2024-09-20T22:48:02.503 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:45:48.173893+0000", 2024-09-20T22:48:02.503 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 134.270845834, 2024-09-20T22:48:02.503 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014936935, 2024-09-20T22:48:02.503 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:48:02.503 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:48:02.503 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:48:02.503 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:48:02.503 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.503 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:48:02.503 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.173893+0000", 2024-09-20T22:48:02.503 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:48:02.503 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.504 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.504 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:48:02.504 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.173893+0000", 2024-09-20T22:48:02.504 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:48:02.504 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.504 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.504 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:48:02.504 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.173893+0000", 2024-09-20T22:48:02.504 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.8800000000000002e-07 2024-09-20T22:48:02.504 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.504 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.504 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:48:02.504 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.173916+0000", 2024-09-20T22:48:02.505 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.2889999999999999e-05 2024-09-20T22:48:02.505 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.505 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.505 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:48:02.505 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.173924+0000", 2024-09-20T22:48:02.505 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.4420000000000001e-06 2024-09-20T22:48:02.505 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.505 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.505 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:48:02.505 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.173932+0000", 2024-09-20T22:48:02.505 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.5299999999999996e-06 2024-09-20T22:48:02.505 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.505 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.505 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:48:02.506 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.174011+0000", 2024-09-20T22:48:02.506 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.9342e-05 2024-09-20T22:48:02.506 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.506 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.506 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:48:02.506 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.174047+0000", 2024-09-20T22:48:02.506 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.5237999999999997e-05 2024-09-20T22:48:02.506 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.506 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.506 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:48:02.506 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.188782+0000", 2024-09-20T22:48:02.506 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014735599 2024-09-20T22:48:02.506 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.507 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.507 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:48:02.507 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.188830+0000", 2024-09-20T22:48:02.507 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.7206000000000002e-05 2024-09-20T22:48:02.507 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:02.507 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:48:02.507 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:02.507 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.507 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.507 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(osd.7.0:153 7.17 e269/139 7:ea720b85:::smithi08477203-26:55 v 269'225, mlcod=263'221)", 2024-09-20T22:48:02.507 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:45:48.173991+0000", 2024-09-20T22:48:02.508 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 134.27074735299999, 2024-09-20T22:48:02.508 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013064868, 2024-09-20T22:48:02.508 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:48:02.508 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:48:02.508 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:48:02.508 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:48:02.508 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.508 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:48:02.508 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.173991+0000", 2024-09-20T22:48:02.508 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:48:02.508 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.508 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.508 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:48:02.508 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.173991+0000", 2024-09-20T22:48:02.509 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:48:02.509 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.509 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.509 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:48:02.509 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.173991+0000", 2024-09-20T22:48:02.509 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.9999999999999999e-07 2024-09-20T22:48:02.509 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.509 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.509 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:48:02.509 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.173996+0000", 2024-09-20T22:48:02.509 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.3080000000000001e-06 2024-09-20T22:48:02.509 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.509 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.509 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:48:02.510 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.174001+0000", 2024-09-20T22:48:02.510 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.203e-06 2024-09-20T22:48:02.510 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.510 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.510 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:48:02.510 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.174005+0000", 2024-09-20T22:48:02.510 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.4880000000000001e-06 2024-09-20T22:48:02.510 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.510 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.510 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:48:02.510 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.175600+0000", 2024-09-20T22:48:02.510 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015942459999999999 2024-09-20T22:48:02.510 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.510 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.510 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:48:02.511 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.175631+0000", 2024-09-20T22:48:02.511 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.1816000000000002e-05 2024-09-20T22:48:02.511 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.511 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.511 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:48:02.511 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.187003+0000", 2024-09-20T22:48:02.511 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011372006 2024-09-20T22:48:02.511 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.511 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.511 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:48:02.511 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.187056+0000", 2024-09-20T22:48:02.511 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.2500999999999998e-05 2024-09-20T22:48:02.511 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:02.511 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:48:02.512 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:02.512 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.512 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.512 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(osd.6.0:767 7.12 e269/263 7:4fa4b2b3:::smithi08477203-21:53 v 269'262, mlcod=265'259)", 2024-09-20T22:48:02.512 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:45:48.175130+0000", 2024-09-20T22:48:02.512 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 134.269608164, 2024-09-20T22:48:02.512 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013842727000000001, 2024-09-20T22:48:02.512 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:48:02.512 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:48:02.512 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:48:02.512 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:48:02.512 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.512 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:48:02.512 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.175130+0000", 2024-09-20T22:48:02.513 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:48:02.513 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.513 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.513 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:48:02.513 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.175130+0000", 2024-09-20T22:48:02.514 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:48:02.514 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.514 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.514 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:48:02.514 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.175130+0000", 2024-09-20T22:48:02.514 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.98e-07 2024-09-20T22:48:02.514 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.514 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.514 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:48:02.514 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.175139+0000", 2024-09-20T22:48:02.514 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.5909999999999996e-06 2024-09-20T22:48:02.514 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.514 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.514 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:48:02.515 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.175144+0000", 2024-09-20T22:48:02.515 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.361e-06 2024-09-20T22:48:02.515 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.515 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.515 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:48:02.515 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.175156+0000", 2024-09-20T22:48:02.515 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1236999999999999e-05 2024-09-20T22:48:02.515 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.515 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.515 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:48:02.515 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.176359+0000", 2024-09-20T22:48:02.515 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0012036060000000001 2024-09-20T22:48:02.515 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.515 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.516 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:48:02.516 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.176380+0000", 2024-09-20T22:48:02.516 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.0497999999999999e-05 2024-09-20T22:48:02.516 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.516 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.516 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:48:02.516 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.188936+0000", 2024-09-20T22:48:02.516 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.012556286999999999 2024-09-20T22:48:02.516 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.516 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.516 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:48:02.516 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.188973+0000", 2024-09-20T22:48:02.517 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.6949000000000001e-05 2024-09-20T22:48:02.517 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:02.517 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:48:02.517 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:02.517 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.517 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.517 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24652.0:2853 7.10 e303/139 7:0f524219:::smithi08477203-50:head v 303'85, mlcod=301'82)", 2024-09-20T22:48:02.517 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:46:23.293837+0000", 2024-09-20T22:48:02.517 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 99.150901450000006, 2024-09-20T22:48:02.517 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013368555000000001, 2024-09-20T22:48:02.517 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:48:02.517 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:48:02.517 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:48:02.517 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:48:02.518 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.518 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:48:02.518 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.293837+0000", 2024-09-20T22:48:02.518 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:48:02.518 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.518 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.518 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:48:02.518 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.293837+0000", 2024-09-20T22:48:02.518 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:48:02.518 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.518 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.518 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:48:02.518 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.293839+0000", 2024-09-20T22:48:02.518 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.7239999999999999e-06 2024-09-20T22:48:02.519 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.519 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.519 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:48:02.519 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.297226+0000", 2024-09-20T22:48:02.519 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.003387583 2024-09-20T22:48:02.519 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.519 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.519 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:48:02.519 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.297253+0000", 2024-09-20T22:48:02.519 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.6849e-05 2024-09-20T22:48:02.519 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.519 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.519 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:48:02.519 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.297266+0000", 2024-09-20T22:48:02.520 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2481e-05 2024-09-20T22:48:02.520 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.520 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.520 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:48:02.520 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.297362+0000", 2024-09-20T22:48:02.520 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.6347000000000004e-05 2024-09-20T22:48:02.520 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.520 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.520 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:48:02.520 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.297394+0000", 2024-09-20T22:48:02.520 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.1888999999999999e-05 2024-09-20T22:48:02.520 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.520 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.520 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:48:02.520 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.307147+0000", 2024-09-20T22:48:02.521 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0097533719999999997 2024-09-20T22:48:02.521 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.521 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.521 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:48:02.521 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.307206+0000", 2024-09-20T22:48:02.521 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.8310000000000002e-05 2024-09-20T22:48:02.521 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:48:02.522 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:02.522 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:48:02.523 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:02.523 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.523 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.523 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:3030 7.e 7:71b5c915:::smithi08477203-6:head [copy-from ver 216] snapc 72=[72,71,70,6e,6c,6b,6a,68] ondisk+write+known_if_redirected+supports_pool_eio e317)", 2024-09-20T22:48:02.523 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:46:37.293029+0000", 2024-09-20T22:48:02.523 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 85.151709092000004, 2024-09-20T22:48:02.523 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.019929916999999998, 2024-09-20T22:48:02.523 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:48:02.523 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:48:02.523 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:48:02.523 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:48:02.523 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:48:02.523 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:48:02.523 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 3030 2024-09-20T22:48:02.524 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.524 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:48:02.524 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.524 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:48:02.524 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.293029+0000", 2024-09-20T22:48:02.524 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:48:02.524 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.524 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.524 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:48:02.524 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.293029+0000", 2024-09-20T22:48:02.524 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:48:02.524 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.524 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.524 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:48:02.525 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.293031+0000", 2024-09-20T22:48:02.525 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.4190000000000001e-06 2024-09-20T22:48:02.525 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.525 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.525 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:48:02.525 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.293040+0000", 2024-09-20T22:48:02.525 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.6039999999999999e-06 2024-09-20T22:48:02.525 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.525 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.525 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:48:02.525 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.293054+0000", 2024-09-20T22:48:02.525 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.4014e-05 2024-09-20T22:48:02.525 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.525 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.526 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:48:02.526 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.293062+0000", 2024-09-20T22:48:02.526 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.8820000000000001e-06 2024-09-20T22:48:02.526 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.526 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.526 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:48:02.526 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.293121+0000", 2024-09-20T22:48:02.526 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.8977999999999997e-05 2024-09-20T22:48:02.526 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.526 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.526 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op must wait for map", 2024-09-20T22:48:02.526 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.293125+0000", 2024-09-20T22:48:02.526 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.2089999999999999e-06 2024-09-20T22:48:02.527 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.527 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.527 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:48:02.527 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.298805+0000", 2024-09-20T22:48:02.527 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0056790870000000002 2024-09-20T22:48:02.527 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.527 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.527 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:48:02.527 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.298886+0000", 2024-09-20T22:48:02.527 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.1363e-05 2024-09-20T22:48:02.527 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.527 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.527 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 3,5", 2024-09-20T22:48:02.527 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.305883+0000", 2024-09-20T22:48:02.527 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0069966159999999998 2024-09-20T22:48:02.528 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.528 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.528 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:48:02.528 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.312556+0000", 2024-09-20T22:48:02.528 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0066737009999999998 2024-09-20T22:48:02.528 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.528 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.528 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:48:02.528 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.312587+0000", 2024-09-20T22:48:02.528 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.0286000000000001e-05 2024-09-20T22:48:02.528 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.528 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.528 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:48:02.528 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.312890+0000", 2024-09-20T22:48:02.529 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00030374800000000001 2024-09-20T22:48:02.529 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.529 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.529 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:48:02.529 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.312932+0000", 2024-09-20T22:48:02.529 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.1621e-05 2024-09-20T22:48:02.529 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.529 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.529 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:48:02.529 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.312959+0000", 2024-09-20T22:48:02.529 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.7389000000000002e-05 2024-09-20T22:48:02.529 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:02.529 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:48:02.529 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:02.529 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.530 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.530 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:3031 7.e 7:71b5c915:::smithi08477203-6:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e317)", 2024-09-20T22:48:02.530 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:46:37.293082+0000", 2024-09-20T22:48:02.530 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 85.151656713999998, 2024-09-20T22:48:02.530 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.016134757, 2024-09-20T22:48:02.530 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:48:02.530 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:48:02.530 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-20T22:48:02.530 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:48:02.530 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:48:02.530 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:48:02.530 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 3031 2024-09-20T22:48:02.530 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.531 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:48:02.531 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.531 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:48:02.531 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.293082+0000", 2024-09-20T22:48:02.531 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:48:02.531 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.531 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.531 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:48:02.531 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.293082+0000", 2024-09-20T22:48:02.531 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:48:02.531 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.531 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.531 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:48:02.531 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.293082+0000", 2024-09-20T22:48:02.531 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.1799999999999995e-07 2024-09-20T22:48:02.532 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.532 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.532 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:48:02.532 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.293085+0000", 2024-09-20T22:48:02.532 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.937e-06 2024-09-20T22:48:02.532 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.532 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.532 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:48:02.532 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.293093+0000", 2024-09-20T22:48:02.532 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.554e-06 2024-09-20T22:48:02.532 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.532 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.532 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:48:02.532 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.293098+0000", 2024-09-20T22:48:02.533 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.643e-06 2024-09-20T22:48:02.533 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.533 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.533 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:48:02.533 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.293164+0000", 2024-09-20T22:48:02.533 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.5806e-05 2024-09-20T22:48:02.533 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.533 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.533 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting_for_map not empty", 2024-09-20T22:48:02.533 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.293168+0000", 2024-09-20T22:48:02.533 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.2459999999999999e-06 2024-09-20T22:48:02.533 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.533 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.534 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:48:02.534 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.299033+0000", 2024-09-20T22:48:02.534 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0058656699999999999 2024-09-20T22:48:02.534 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.534 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.534 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for blocked object", 2024-09-20T22:48:02.534 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.299069+0000", 2024-09-20T22:48:02.534 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.5392999999999998e-05 2024-09-20T22:48:02.534 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.534 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.534 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:48:02.534 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.309088+0000", 2024-09-20T22:48:02.534 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010019287999999999 2024-09-20T22:48:02.535 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.535 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.535 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:48:02.535 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.309150+0000", 2024-09-20T22:48:02.535 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.2008000000000007e-05 2024-09-20T22:48:02.535 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.535 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.535 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:48:02.535 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.309216+0000", 2024-09-20T22:48:02.535 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.6594000000000006e-05 2024-09-20T22:48:02.535 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:02.535 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:48:02.535 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:02.535 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.535 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.536 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:3290 7.8 7:12e9c8db:::smithi08477203-11:head [copy-from ver 675] snapc 7f=[7f,7e,7d,7c,7b,7a,79,78,74,73,6b] ondisk+write+known_if_redirected+supports_pool_eio e340)", 2024-09-20T22:48:02.536 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:47:00.706846+0000", 2024-09-20T22:48:02.536 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 61.737892854999998, 2024-09-20T22:48:02.536 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.018408746, 2024-09-20T22:48:02.536 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:48:02.536 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:48:02.536 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:48:02.536 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:48:02.536 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:48:02.536 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:48:02.536 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 3290 2024-09-20T22:48:02.536 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.536 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:48:02.536 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.537 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:48:02.537 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.706846+0000", 2024-09-20T22:48:02.537 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:48:02.537 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.537 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.537 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:48:02.537 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.706846+0000", 2024-09-20T22:48:02.537 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:48:02.537 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.537 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.537 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:48:02.537 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.706848+0000", 2024-09-20T22:48:02.537 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.2340000000000001e-06 2024-09-20T22:48:02.537 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.538 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.538 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:48:02.538 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.706859+0000", 2024-09-20T22:48:02.538 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1001e-05 2024-09-20T22:48:02.538 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.538 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.538 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:48:02.538 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.706879+0000", 2024-09-20T22:48:02.538 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.0556000000000001e-05 2024-09-20T22:48:02.538 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.538 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.538 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:48:02.538 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.706890+0000", 2024-09-20T22:48:02.538 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0328e-05 2024-09-20T22:48:02.539 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.539 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.539 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:48:02.539 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.706973+0000", 2024-09-20T22:48:02.539 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.3574999999999999e-05 2024-09-20T22:48:02.539 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.539 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.539 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:48:02.539 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.707042+0000", 2024-09-20T22:48:02.539 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.8987999999999997e-05 2024-09-20T22:48:02.539 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.539 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.539 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 3,7", 2024-09-20T22:48:02.540 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.717491+0000", 2024-09-20T22:48:02.540 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010448618 2024-09-20T22:48:02.540 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.540 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.540 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:48:02.540 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.722875+0000", 2024-09-20T22:48:02.540 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0053846190000000002 2024-09-20T22:48:02.540 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.540 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.540 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:48:02.540 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.724841+0000", 2024-09-20T22:48:02.540 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0019651579999999998 2024-09-20T22:48:02.540 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.540 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.541 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:48:02.541 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.725168+0000", 2024-09-20T22:48:02.541 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00032734499999999999 2024-09-20T22:48:02.541 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.541 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.541 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:48:02.541 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.725218+0000", 2024-09-20T22:48:02.541 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.0123000000000001e-05 2024-09-20T22:48:02.541 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.541 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.541 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:48:02.541 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.725254+0000", 2024-09-20T22:48:02.541 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.6201000000000001e-05 2024-09-20T22:48:02.541 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:02.542 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:48:02.542 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:02.542 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.542 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.542 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:3291 7.8 7:12e9c8db:::smithi08477203-11:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e340)", 2024-09-20T22:48:02.542 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:47:00.706927+0000", 2024-09-20T22:48:02.542 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 61.737811215999997, 2024-09-20T22:48:02.542 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014159255000000001, 2024-09-20T22:48:02.542 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:48:02.542 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:48:02.542 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-20T22:48:02.542 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:48:02.542 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:48:02.542 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:48:02.543 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 3291 2024-09-20T22:48:02.543 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.543 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:48:02.543 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.543 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:48:02.543 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.706927+0000", 2024-09-20T22:48:02.543 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:48:02.543 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.543 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.543 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:48:02.543 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.706927+0000", 2024-09-20T22:48:02.543 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:48:02.543 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.543 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.543 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:48:02.544 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.706928+0000", 2024-09-20T22:48:02.544 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.4099999999999999e-07 2024-09-20T22:48:02.544 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.544 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.544 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:48:02.544 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.706941+0000", 2024-09-20T22:48:02.544 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3335e-05 2024-09-20T22:48:02.544 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.544 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.544 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:48:02.544 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.706948+0000", 2024-09-20T22:48:02.544 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.3350000000000001e-06 2024-09-20T22:48:02.544 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.544 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.545 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:48:02.545 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.706954+0000", 2024-09-20T22:48:02.545 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.9190000000000004e-06 2024-09-20T22:48:02.545 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.545 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.545 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:48:02.545 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.707166+0000", 2024-09-20T22:48:02.545 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00021159 2024-09-20T22:48:02.545 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.545 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.545 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for blocked object", 2024-09-20T22:48:02.545 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.707217+0000", 2024-09-20T22:48:02.545 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.1167e-05 2024-09-20T22:48:02.545 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.545 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.546 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:48:02.546 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.720939+0000", 2024-09-20T22:48:02.546 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013721458000000001 2024-09-20T22:48:02.546 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.546 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.546 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:48:02.546 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.721010+0000", 2024-09-20T22:48:02.546 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.1570999999999996e-05 2024-09-20T22:48:02.546 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.546 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.546 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:48:02.546 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.721086+0000", 2024-09-20T22:48:02.546 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.6339000000000002e-05 2024-09-20T22:48:02.546 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:02.547 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:48:02.547 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:02.547 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.547 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.547 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(osd.5.0:167 7.2 e347/169 7:45b54b6b:::smithi08477203-2:81 v 347'140, mlcod=345'136)", 2024-09-20T22:48:02.547 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:47:08.204691+0000", 2024-09-20T22:48:02.547 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 54.240047115000003, 2024-09-20T22:48:02.547 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.01335918, 2024-09-20T22:48:02.547 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:48:02.547 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:48:02.547 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:48:02.547 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:48:02.547 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.547 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:48:02.548 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:08.204691+0000", 2024-09-20T22:48:02.548 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:48:02.548 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.548 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.548 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:48:02.548 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:08.204691+0000", 2024-09-20T22:48:02.548 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:48:02.548 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.548 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.548 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:48:02.548 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:08.204692+0000", 2024-09-20T22:48:02.548 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.5400000000000001e-07 2024-09-20T22:48:02.548 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.550 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.550 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:48:02.550 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:08.204697+0000", 2024-09-20T22:48:02.550 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.614e-06 2024-09-20T22:48:02.550 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.550 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.550 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:48:02.550 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:08.204705+0000", 2024-09-20T22:48:02.551 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.8259999999999993e-06 2024-09-20T22:48:02.551 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.551 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.551 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:48:02.551 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:08.204715+0000", 2024-09-20T22:48:02.551 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.8279999999999998e-06 2024-09-20T22:48:02.551 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.551 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.551 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:48:02.551 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:08.204801+0000", 2024-09-20T22:48:02.551 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.6263999999999999e-05 2024-09-20T22:48:02.551 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.551 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.551 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:48:02.551 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:08.204823+0000", 2024-09-20T22:48:02.552 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.1971999999999999e-05 2024-09-20T22:48:02.552 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.552 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.552 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:48:02.552 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:08.217961+0000", 2024-09-20T22:48:02.552 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013137618 2024-09-20T22:48:02.552 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.552 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.552 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:48:02.552 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:08.218050+0000", 2024-09-20T22:48:02.552 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.9504000000000002e-05 2024-09-20T22:48:02.552 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:02.552 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:48:02.553 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:02.553 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.553 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.553 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24652.0:3760 7.13 e383/260 7:cae5792a:::smithi08477203-32:head v 383'78, mlcod=377'74)", 2024-09-20T22:48:02.553 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:47:43.291645+0000", 2024-09-20T22:48:02.553 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 19.153093243000001, 2024-09-20T22:48:02.553 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.012696648, 2024-09-20T22:48:02.553 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:48:02.553 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:48:02.553 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:48:02.553 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:48:02.553 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.553 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:48:02.553 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:43.291645+0000", 2024-09-20T22:48:02.554 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:48:02.554 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.554 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.554 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:48:02.554 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:43.291645+0000", 2024-09-20T22:48:02.554 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:48:02.554 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.554 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.554 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:48:02.554 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:43.291645+0000", 2024-09-20T22:48:02.554 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.0800000000000001e-07 2024-09-20T22:48:02.554 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.554 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.554 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:48:02.555 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:43.294764+0000", 2024-09-20T22:48:02.555 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0031183420000000001 2024-09-20T22:48:02.555 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.555 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.555 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:48:02.555 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:43.294777+0000", 2024-09-20T22:48:02.555 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3465999999999999e-05 2024-09-20T22:48:02.555 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.555 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.555 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:48:02.555 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:43.294788+0000", 2024-09-20T22:48:02.555 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0662999999999999e-05 2024-09-20T22:48:02.555 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.555 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.555 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:48:02.556 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:43.294952+0000", 2024-09-20T22:48:02.556 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00016369799999999999 2024-09-20T22:48:02.556 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.556 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.556 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:48:02.556 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:43.294978+0000", 2024-09-20T22:48:02.556 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.6414000000000002e-05 2024-09-20T22:48:02.556 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.556 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.556 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:48:02.556 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:43.304313+0000", 2024-09-20T22:48:02.556 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0093352039999999997 2024-09-20T22:48:02.556 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:02.556 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:02.557 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:48:02.557 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:43.304342+0000", 2024-09-20T22:48:02.557 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.8552999999999999e-05 2024-09-20T22:48:02.557 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:02.557 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:48:02.557 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:02.557 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:02.557 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:48:02.557 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-20T22:48:02.557 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:02 smithi084 ceph-mon[40024]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T22:48:02.558 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:02 smithi084 ceph-mon[40024]: Cluster is now healthy 2024-09-20T22:48:02.558 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:02 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T22:48:02.558 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:02 smithi084 ceph-mon[40024]: osdmap e401: 8 total, 8 up, 7 in 2024-09-20T22:48:02.558 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:02 smithi084 ceph-mon[40024]: 3.13 scrub starts 2024-09-20T22:48:02.558 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:02 smithi084 ceph-mon[40024]: 3.13 scrub ok 2024-09-20T22:48:02.558 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:02 smithi084 ceph-mon[40024]: 1.6 scrub starts 2024-09-20T22:48:02.558 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:02 smithi084 ceph-mon[40024]: 1.6 scrub ok 2024-09-20T22:48:02.558 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:02 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/107400826' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:48:02.558 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:02 smithi084 ceph-mon[29774]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T22:48:02.559 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:02 smithi084 ceph-mon[29774]: Cluster is now healthy 2024-09-20T22:48:02.559 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:02 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T22:48:02.559 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:02 smithi084 ceph-mon[29774]: osdmap e401: 8 total, 8 up, 7 in 2024-09-20T22:48:02.559 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:02 smithi084 ceph-mon[29774]: 3.13 scrub starts 2024-09-20T22:48:02.559 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:02 smithi084 ceph-mon[29774]: 3.13 scrub ok 2024-09-20T22:48:02.559 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:02 smithi084 ceph-mon[29774]: 1.6 scrub starts 2024-09-20T22:48:02.559 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:02 smithi084 ceph-mon[29774]: 1.6 scrub ok 2024-09-20T22:48:02.559 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:02 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/107400826' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:48:02.580 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:02.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:02.581 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:48:02.667 INFO:tasks.rados.rados.0.smithi084.stdout:1536: finishing write tid 1 to smithi08477203-10 2024-09-20T22:48:02.667 INFO:tasks.rados.rados.0.smithi084.stdout:1534: done (5 left) 2024-09-20T22:48:02.667 INFO:tasks.rados.rados.0.smithi084.stdout:1535: done (4 left) 2024-09-20T22:48:02.667 INFO:tasks.rados.rados.0.smithi084.stdout:1537: done (3 left) 2024-09-20T22:48:02.667 INFO:tasks.rados.rados.0.smithi084.stdout:1540: snap_create 2024-09-20T22:48:02.667 INFO:tasks.rados.rados.0.smithi084.stdout:1536: finishing write tid 2 to smithi08477203-10 2024-09-20T22:48:02.667 INFO:tasks.rados.rados.0.smithi084.stdout:1536: finishing write tid 3 to smithi08477203-10 2024-09-20T22:48:02.667 INFO:tasks.rados.rados.0.smithi084.stdout:1536: finishing write tid 4 to smithi08477203-10 2024-09-20T22:48:02.667 INFO:tasks.rados.rados.0.smithi084.stdout:1536: oid 10 updating version 0 to 541 2024-09-20T22:48:02.667 INFO:tasks.rados.rados.0.smithi084.stdout:1536: oid 10 updating version 541 to 542 2024-09-20T22:48:02.668 INFO:tasks.rados.rados.0.smithi084.stdout:1536: oid 10 version 542 is already newer than 540 2024-09-20T22:48:02.668 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 10 v 542 (ObjNum 544 snap 164 seq_num 544) dirty exists 2024-09-20T22:48:02.668 INFO:tasks.rados.rados.0.smithi084.stdout:1536: left oid 10 (ObjNum 544 snap 164 seq_num 544) 2024-09-20T22:48:02.668 INFO:tasks.rados.rados.0.smithi084.stdout:1538: expect (ObjNum 458 snap 126 seq_num 458) 2024-09-20T22:48:02.682 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:48:02.707 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:02.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:02.708 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:48:02.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:02 smithi154 ceph-mon[33264]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T22:48:02.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:02 smithi154 ceph-mon[33264]: Cluster is now healthy 2024-09-20T22:48:02.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:02 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T22:48:02.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:02 smithi154 ceph-mon[33264]: osdmap e401: 8 total, 8 up, 7 in 2024-09-20T22:48:02.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:02 smithi154 ceph-mon[33264]: 3.13 scrub starts 2024-09-20T22:48:02.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:02 smithi154 ceph-mon[33264]: 3.13 scrub ok 2024-09-20T22:48:02.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:02 smithi154 ceph-mon[33264]: 1.6 scrub starts 2024-09-20T22:48:02.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:02 smithi154 ceph-mon[33264]: 1.6 scrub ok 2024-09-20T22:48:02.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:02 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/107400826' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:48:02.809 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:48:02.829 INFO:tasks.rados.rados.0.smithi084.stdout:1539: finishing rollback tid 1 to smithi08477203-19 2024-09-20T22:48:02.829 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 19 v 0 (ObjNum 33 snap 0 seq_num 803183856) dirty dne 2024-09-20T22:48:02.829 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:02.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:02.830 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:48:02.931 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:48:02.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:02.958 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:02.959 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:48:03.059 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:48:03.081 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:03.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:03.082 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:48:03.182 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:48:03.200 INFO:tasks.rados.rados.0.smithi084.stdout:1536: done (3 left) 2024-09-20T22:48:03.200 INFO:tasks.rados.rados.0.smithi084.stdout:1538: done (2 left) 2024-09-20T22:48:03.200 INFO:tasks.rados.rados.0.smithi084.stdout:1539: done (1 left) 2024-09-20T22:48:03.200 INFO:tasks.rados.rados.0.smithi084.stdout:1540: done (0 left) 2024-09-20T22:48:03.200 INFO:tasks.rados.rados.0.smithi084.stdout:1541: copy_from oid 31 from oid 9 current snap is 165 2024-09-20T22:48:03.200 INFO:tasks.rados.rados.0.smithi084.stdout:1542: write (excl) oid 4 current snap is 165 2024-09-20T22:48:03.201 INFO:tasks.rados.rados.0.smithi084.stdout:1542: seq_num 545 ranges {759815=471768,1742451=465507,2917178=242637} 2024-09-20T22:48:03.206 INFO:tasks.rados.rados.0.smithi084.stdout:1542: writing smithi08477203-4 from 759815 to 1231583 tid 1 2024-09-20T22:48:03.211 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:03.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:03.213 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:48:03.213 INFO:tasks.rados.rados.0.smithi084.stdout:1542: writing smithi08477203-4 from 1742451 to 2207958 tid 2 2024-09-20T22:48:03.219 INFO:tasks.rados.rados.0.smithi084.stdout:1542: writing smithi08477203-4 from 2917178 to 3159815 tid 3 2024-09-20T22:48:03.219 INFO:tasks.rados.rados.0.smithi084.stdout:1543: read oid 43 snap -1 2024-09-20T22:48:03.219 INFO:tasks.rados.rados.0.smithi084.stdout:1543: expect (ObjNum 526 snap 151 seq_num 526) 2024-09-20T22:48:03.219 INFO:tasks.rados.rados.0.smithi084.stdout:1544: snap_remove snap 159 2024-09-20T22:48:03.313 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:48:03.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:03.338 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:03.338 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:48:03.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:03 smithi084 ceph-mon[40024]: pgmap v504: 168 pgs: 2 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 163 active+clean; 294 MiB data, 897 MiB used, 625 GiB / 626 GiB avail; 2.7 MiB/s rd, 0 B/s wr, 4 op/s 2024-09-20T22:48:03.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:03 smithi084 ceph-mon[40024]: 1.5 scrub starts 2024-09-20T22:48:03.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:03 smithi084 ceph-mon[40024]: 1.5 scrub ok 2024-09-20T22:48:03.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:03 smithi084 ceph-mon[40024]: osdmap e402: 8 total, 8 up, 7 in 2024-09-20T22:48:03.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:03 smithi084 ceph-mon[40024]: 7.11 scrub starts 2024-09-20T22:48:03.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:03 smithi084 ceph-mon[40024]: 7.11 scrub ok 2024-09-20T22:48:03.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:03 smithi084 ceph-mon[40024]: 3.16 scrub starts 2024-09-20T22:48:03.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:03 smithi084 ceph-mon[40024]: 3.16 scrub ok 2024-09-20T22:48:03.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:03 smithi084 ceph-mon[40024]: 1.7 scrub starts 2024-09-20T22:48:03.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:03 smithi084 ceph-mon[40024]: 1.7 scrub ok 2024-09-20T22:48:03.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:03 smithi084 ceph-mon[40024]: osdmap e403: 8 total, 8 up, 7 in 2024-09-20T22:48:03.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:03 smithi084 ceph-mon[29774]: pgmap v504: 168 pgs: 2 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 163 active+clean; 294 MiB data, 897 MiB used, 625 GiB / 626 GiB avail; 2.7 MiB/s rd, 0 B/s wr, 4 op/s 2024-09-20T22:48:03.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:03 smithi084 ceph-mon[29774]: 1.5 scrub starts 2024-09-20T22:48:03.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:03 smithi084 ceph-mon[29774]: 1.5 scrub ok 2024-09-20T22:48:03.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:03 smithi084 ceph-mon[29774]: osdmap e402: 8 total, 8 up, 7 in 2024-09-20T22:48:03.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:03 smithi084 ceph-mon[29774]: 7.11 scrub starts 2024-09-20T22:48:03.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:03 smithi084 ceph-mon[29774]: 7.11 scrub ok 2024-09-20T22:48:03.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:03 smithi084 ceph-mon[29774]: 3.16 scrub starts 2024-09-20T22:48:03.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:03 smithi084 ceph-mon[29774]: 3.16 scrub ok 2024-09-20T22:48:03.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:03 smithi084 ceph-mon[29774]: 1.7 scrub starts 2024-09-20T22:48:03.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:03 smithi084 ceph-mon[29774]: 1.7 scrub ok 2024-09-20T22:48:03.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:03 smithi084 ceph-mon[29774]: osdmap e403: 8 total, 8 up, 7 in 2024-09-20T22:48:03.439 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:48:03.460 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:03.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:03.461 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:48:03.562 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:48:03.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:03.587 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:03.587 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:48:03.687 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:48:03.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:03.709 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:03.709 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:48:03.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:03 smithi154 ceph-mon[33264]: pgmap v504: 168 pgs: 2 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 163 active+clean; 294 MiB data, 897 MiB used, 625 GiB / 626 GiB avail; 2.7 MiB/s rd, 0 B/s wr, 4 op/s 2024-09-20T22:48:03.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:03 smithi154 ceph-mon[33264]: 1.5 scrub starts 2024-09-20T22:48:03.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:03 smithi154 ceph-mon[33264]: 1.5 scrub ok 2024-09-20T22:48:03.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:03 smithi154 ceph-mon[33264]: osdmap e402: 8 total, 8 up, 7 in 2024-09-20T22:48:03.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:03 smithi154 ceph-mon[33264]: 7.11 scrub starts 2024-09-20T22:48:03.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:03 smithi154 ceph-mon[33264]: 7.11 scrub ok 2024-09-20T22:48:03.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:03 smithi154 ceph-mon[33264]: 3.16 scrub starts 2024-09-20T22:48:03.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:03 smithi154 ceph-mon[33264]: 3.16 scrub ok 2024-09-20T22:48:03.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:03 smithi154 ceph-mon[33264]: 1.7 scrub starts 2024-09-20T22:48:03.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:03 smithi154 ceph-mon[33264]: 1.7 scrub ok 2024-09-20T22:48:03.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:03 smithi154 ceph-mon[33264]: osdmap e403: 8 total, 8 up, 7 in 2024-09-20T22:48:03.809 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:48:03.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:03.832 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:03.832 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:48:03.933 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:48:03.959 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:03.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:03.960 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:48:03.997 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd set noscrub 2024-09-20T22:48:04.060 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:48:04.080 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:04.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:04.081 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:48:04.182 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:48:04.204 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:04.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:04.205 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:48:04.205 INFO:teuthology.orchestra.run.smithi084.stderr:marked in osd.4. 2024-09-20T22:48:04.225 INFO:tasks.rados.rados.0.smithi084.stdout:1544: done (3 left) 2024-09-20T22:48:04.225 INFO:tasks.rados.rados.0.smithi084.stdout:1545: write (excl) oid 47 current snap is 165 2024-09-20T22:48:04.225 INFO:tasks.rados.rados.0.smithi084.stdout:1545: seq_num 546 ranges {776622=547017,1800933=725518,2944601=232021} 2024-09-20T22:48:04.235 INFO:tasks.rados.rados.0.smithi084.stdout:1545: writing smithi08477203-47 from 776622 to 1323639 tid 1 2024-09-20T22:48:04.242 INFO:tasks.rados.rados.0.smithi084.stdout:1545: writing smithi08477203-47 from 1800933 to 2526451 tid 2 2024-09-20T22:48:04.246 INFO:tasks.rados.rados.0.smithi084.stdout:1545: writing smithi08477203-47 from 2944601 to 3176622 tid 3 2024-09-20T22:48:04.246 INFO:tasks.rados.rados.0.smithi084.stdout:1542: finishing write tid 1 to smithi08477203-4 2024-09-20T22:48:04.247 INFO:tasks.rados.rados.0.smithi084.stdout:1541: finishing copy_from to smithi08477203-31 2024-09-20T22:48:04.247 INFO:tasks.rados.rados.0.smithi084.stdout:1541: got expected ENOENT (src dne) 2024-09-20T22:48:04.247 INFO:tasks.rados.rados.0.smithi084.stdout:1541: finishing copy_from racing read to smithi08477203-31 2024-09-20T22:48:04.247 INFO:tasks.rados.rados.0.smithi084.stdout:1542: finishing write tid 2 to smithi08477203-4 2024-09-20T22:48:04.247 INFO:tasks.rados.rados.0.smithi084.stdout:1542: finishing write tid 3 to smithi08477203-4 2024-09-20T22:48:04.247 INFO:tasks.rados.rados.0.smithi084.stdout:1542: finishing write tid 4 to smithi08477203-4 2024-09-20T22:48:04.247 INFO:tasks.rados.rados.0.smithi084.stdout:1542: finishing write tid 5 to smithi08477203-4 2024-09-20T22:48:04.247 INFO:tasks.rados.rados.0.smithi084.stdout:1542: oid 4 updating version 0 to 858 2024-09-20T22:48:04.247 INFO:tasks.rados.rados.0.smithi084.stdout:1542: oid 4 version 858 is already newer than 855 2024-09-20T22:48:04.247 INFO:tasks.rados.rados.0.smithi084.stdout:1542: oid 4 version 858 is already newer than 857 2024-09-20T22:48:04.247 INFO:tasks.rados.rados.0.smithi084.stdout:1542: oid 4 version 858 is already newer than 856 2024-09-20T22:48:04.247 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 4 v 858 (ObjNum 545 snap 165 seq_num 545) dirty exists 2024-09-20T22:48:04.247 INFO:tasks.rados.rados.0.smithi084.stdout:1542: left oid 4 (ObjNum 545 snap 165 seq_num 545) 2024-09-20T22:48:04.247 INFO:tasks.rados.rados.0.smithi084.stdout:1543: expect (ObjNum 526 snap 151 seq_num 526) 2024-09-20T22:48:04.306 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:48:04.309 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:04 smithi084 ceph-mon[40024]: 7.14 scrub starts 2024-09-20T22:48:04.309 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:04 smithi084 ceph-mon[40024]: 7.14 scrub ok 2024-09-20T22:48:04.309 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:04 smithi084 ceph-mon[40024]: 1.9 scrub starts 2024-09-20T22:48:04.309 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:04 smithi084 ceph-mon[40024]: 1.9 scrub ok 2024-09-20T22:48:04.310 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:04 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/2279245935' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["4"]}]: dispatch 2024-09-20T22:48:04.310 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:04 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/2279245935' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["4"]}]': finished 2024-09-20T22:48:04.310 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:04 smithi084 ceph-mon[40024]: osdmap e404: 8 total, 8 up, 8 in 2024-09-20T22:48:04.310 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:04 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mon-a[29770]: 2024-09-20T22:48:04.040+0000 7f26ad5e8640 -1 mon.a@0(leader).osd e403 definitely_dead 0 2024-09-20T22:48:04.310 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:04 smithi084 ceph-mon[29774]: 7.14 scrub starts 2024-09-20T22:48:04.310 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:04 smithi084 ceph-mon[29774]: 7.14 scrub ok 2024-09-20T22:48:04.310 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:04 smithi084 ceph-mon[29774]: 1.9 scrub starts 2024-09-20T22:48:04.310 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:04 smithi084 ceph-mon[29774]: 1.9 scrub ok 2024-09-20T22:48:04.310 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:04 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/2279245935' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["4"]}]: dispatch 2024-09-20T22:48:04.310 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:04 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/2279245935' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["4"]}]': finished 2024-09-20T22:48:04.311 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:04 smithi084 ceph-mon[29774]: osdmap e404: 8 total, 8 up, 8 in 2024-09-20T22:48:04.329 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:04.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:04.330 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:48:04.342 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-20T22:48:04.404 INFO:tasks.rados.rados.0.smithi084.stdout:1545: finishing write tid 1 to smithi08477203-47 2024-09-20T22:48:04.404 INFO:tasks.rados.rados.0.smithi084.stdout:1545: finishing write tid 2 to smithi08477203-47 2024-09-20T22:48:04.404 INFO:tasks.rados.rados.0.smithi084.stdout:1545: finishing write tid 3 to smithi08477203-47 2024-09-20T22:48:04.404 INFO:tasks.rados.rados.0.smithi084.stdout:1545: finishing write tid 4 to smithi08477203-47 2024-09-20T22:48:04.404 INFO:tasks.rados.rados.0.smithi084.stdout:1545: finishing write tid 5 to smithi08477203-47 2024-09-20T22:48:04.404 INFO:tasks.rados.rados.0.smithi084.stdout:1545: oid 47 updating version 0 to 416 2024-09-20T22:48:04.404 INFO:tasks.rados.rados.0.smithi084.stdout:1545: oid 47 updating version 416 to 417 2024-09-20T22:48:04.404 INFO:tasks.rados.rados.0.smithi084.stdout:1545: oid 47 version 417 is already newer than 414 2024-09-20T22:48:04.404 INFO:tasks.rados.rados.0.smithi084.stdout:1545: oid 47 version 417 is already newer than 415 2024-09-20T22:48:04.404 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 47 v 417 (ObjNum 546 snap 165 seq_num 546) dirty exists 2024-09-20T22:48:04.404 INFO:tasks.rados.rados.0.smithi084.stdout:1545: left oid 47 (ObjNum 546 snap 165 seq_num 546) 2024-09-20T22:48:04.405 INFO:tasks.rados.rados.0.smithi084.stdout:1541: done (3 left) 2024-09-20T22:48:04.405 INFO:tasks.rados.rados.0.smithi084.stdout:1542: done (2 left) 2024-09-20T22:48:04.405 INFO:tasks.rados.rados.0.smithi084.stdout:1543: done (1 left) 2024-09-20T22:48:04.405 INFO:tasks.rados.rados.0.smithi084.stdout:1545: done (0 left) 2024-09-20T22:48:04.405 INFO:tasks.rados.rados.0.smithi084.stdout:1546: snap_create 2024-09-20T22:48:04.431 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:48:04.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:04.454 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:04.454 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:48:04.555 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:48:04.562 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:48:04.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:04.580 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:04.580 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:48:04.682 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:48:04.703 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:04.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:04.704 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:48:04.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:04 smithi154 ceph-mon[33264]: 7.14 scrub starts 2024-09-20T22:48:04.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:04 smithi154 ceph-mon[33264]: 7.14 scrub ok 2024-09-20T22:48:04.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:04 smithi154 ceph-mon[33264]: 1.9 scrub starts 2024-09-20T22:48:04.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:04 smithi154 ceph-mon[33264]: 1.9 scrub ok 2024-09-20T22:48:04.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:04 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/2279245935' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["4"]}]: dispatch 2024-09-20T22:48:04.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:04 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/2279245935' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["4"]}]': finished 2024-09-20T22:48:04.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:04 smithi154 ceph-mon[33264]: osdmap e404: 8 total, 8 up, 8 in 2024-09-20T22:48:04.805 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:48:04.830 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:04.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:04.831 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:48:04.932 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:48:04.940 INFO:tasks.thrashosds.thrasher:Added osd 4 2024-09-20T22:48:04.956 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:04.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:04.957 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:48:05.058 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:48:05.079 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:05.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:05.080 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:48:05.181 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:48:05.207 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:05.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:05.208 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:48:05.216 INFO:tasks.rados.rados.0.smithi084.stdout:1546: done (0 left) 2024-09-20T22:48:05.217 INFO:tasks.rados.rados.0.smithi084.stdout:1547: snap_create 2024-09-20T22:48:05.309 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:48:05.330 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:05.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:05.331 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:48:05.432 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:48:05.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:05.455 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:05.455 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:48:05.465 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:48:05.556 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:48:05.578 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:05.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:05.579 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:48:05.591 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:05 smithi084 ceph-mon[40024]: 4.b scrub starts 2024-09-20T22:48:05.591 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:05 smithi084 ceph-mon[40024]: 4.b scrub ok 2024-09-20T22:48:05.591 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:05 smithi084 ceph-mon[40024]: pgmap v507: 168 pgs: 1 active+clean+snaptrim, 167 active+clean; 305 MiB data, 921 MiB used, 625 GiB / 626 GiB avail; 15 MiB/s rd, 1.9 MiB/s wr, 22 op/s 2024-09-20T22:48:05.591 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:05 smithi084 ceph-mon[40024]: 4.12 deep-scrub starts 2024-09-20T22:48:05.591 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:05 smithi084 ceph-mon[40024]: 4.12 deep-scrub ok 2024-09-20T22:48:05.591 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:05 smithi084 ceph-mon[40024]: 3.5 scrub starts 2024-09-20T22:48:05.591 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:05 smithi084 ceph-mon[40024]: 3.5 scrub ok 2024-09-20T22:48:05.591 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:05 smithi084 ceph-mon[40024]: osdmap e405: 8 total, 8 up, 8 in 2024-09-20T22:48:05.592 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:05 smithi084 ceph-mon[29774]: 4.b scrub starts 2024-09-20T22:48:05.592 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:05 smithi084 ceph-mon[29774]: 4.b scrub ok 2024-09-20T22:48:05.592 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:05 smithi084 ceph-mon[29774]: pgmap v507: 168 pgs: 1 active+clean+snaptrim, 167 active+clean; 305 MiB data, 921 MiB used, 625 GiB / 626 GiB avail; 15 MiB/s rd, 1.9 MiB/s wr, 22 op/s 2024-09-20T22:48:05.592 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:05 smithi084 ceph-mon[29774]: 4.12 deep-scrub starts 2024-09-20T22:48:05.592 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:05 smithi084 ceph-mon[29774]: 4.12 deep-scrub ok 2024-09-20T22:48:05.592 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:05 smithi084 ceph-mon[29774]: 3.5 scrub starts 2024-09-20T22:48:05.592 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:05 smithi084 ceph-mon[29774]: 3.5 scrub ok 2024-09-20T22:48:05.592 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:05 smithi084 ceph-mon[29774]: osdmap e405: 8 total, 8 up, 8 in 2024-09-20T22:48:05.680 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:48:05.692 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-20T22:48:05.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:05.702 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:05.703 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:48:05.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:05 smithi154 ceph-mon[33264]: 4.b scrub starts 2024-09-20T22:48:05.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:05 smithi154 ceph-mon[33264]: 4.b scrub ok 2024-09-20T22:48:05.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:05 smithi154 ceph-mon[33264]: pgmap v507: 168 pgs: 1 active+clean+snaptrim, 167 active+clean; 305 MiB data, 921 MiB used, 625 GiB / 626 GiB avail; 15 MiB/s rd, 1.9 MiB/s wr, 22 op/s 2024-09-20T22:48:05.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:05 smithi154 ceph-mon[33264]: 4.12 deep-scrub starts 2024-09-20T22:48:05.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:05 smithi154 ceph-mon[33264]: 4.12 deep-scrub ok 2024-09-20T22:48:05.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:05 smithi154 ceph-mon[33264]: 3.5 scrub starts 2024-09-20T22:48:05.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:05 smithi154 ceph-mon[33264]: 3.5 scrub ok 2024-09-20T22:48:05.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:05 smithi154 ceph-mon[33264]: osdmap e405: 8 total, 8 up, 8 in 2024-09-20T22:48:05.803 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:48:05.824 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:05.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:05.825 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:48:05.926 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:48:05.947 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:05.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:05.948 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:48:06.005 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-20T22:48:06.006 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2024-09-20T22:48:06.006 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2024-09-20T22:48:06.006 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-20T22:48:06.049 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:48:06.071 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:06.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:06.071 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:48:06.172 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:48:06.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:06.200 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:06.200 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:48:06.213 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is set 2024-09-20T22:48:06.215 INFO:tasks.rados.rados.0.smithi084.stdout:1547: done (0 left) 2024-09-20T22:48:06.215 INFO:tasks.rados.rados.0.smithi084.stdout:1548: copy_from oid 7 from oid 29 current snap is 167 2024-09-20T22:48:06.215 INFO:tasks.rados.rados.0.smithi084.stdout:1549: copy_from oid 34 from oid 9 current snap is 167 2024-09-20T22:48:06.215 INFO:tasks.rados.rados.0.smithi084.stdout:1550: delete oid 20 current snap is 167 2024-09-20T22:48:06.218 INFO:tasks.rados.rados.0.smithi084.stdout:1549: finishing copy_from to smithi08477203-34 2024-09-20T22:48:06.218 INFO:tasks.rados.rados.0.smithi084.stdout:1549: got expected ENOENT (src dne) 2024-09-20T22:48:06.218 INFO:tasks.rados.rados.0.smithi084.stdout:1549: finishing copy_from racing read to smithi08477203-34 2024-09-20T22:48:06.220 INFO:tasks.rados.rados.0.smithi084.stdout:1549: done (2 left) 2024-09-20T22:48:06.221 INFO:tasks.rados.rados.0.smithi084.stdout:1550: done (1 left) 2024-09-20T22:48:06.221 INFO:tasks.rados.rados.0.smithi084.stdout:1551: delete oid 5 current snap is 167 2024-09-20T22:48:06.224 INFO:tasks.rados.rados.0.smithi084.stdout:1551: done (1 left) 2024-09-20T22:48:06.224 INFO:tasks.rados.rados.0.smithi084.stdout:1552: write (excl) oid 6 current snap is 167 2024-09-20T22:48:06.225 INFO:tasks.rados.rados.0.smithi084.stdout:1552: seq_num 549 ranges {427043=489117,1576379=472845,2626870=600173} 2024-09-20T22:48:06.230 INFO:tasks.rados.rados.0.smithi084.stdout:1552: writing smithi08477203-6 from 427043 to 916160 tid 1 2024-09-20T22:48:06.237 INFO:tasks.rados.rados.0.smithi084.stdout:1552: writing smithi08477203-6 from 1576379 to 2049224 tid 2 2024-09-20T22:48:06.245 INFO:tasks.rados.rados.0.smithi084.stdout:1552: writing smithi08477203-6 from 2626870 to 3227043 tid 3 2024-09-20T22:48:06.245 INFO:tasks.rados.rados.0.smithi084.stdout:1553: snap_create 2024-09-20T22:48:06.245 INFO:tasks.rados.rados.0.smithi084.stdout:1548: finishing copy_from racing read to smithi08477203-7 2024-09-20T22:48:06.245 INFO:tasks.rados.rados.0.smithi084.stdout:1552: finishing write tid 1 to smithi08477203-6 2024-09-20T22:48:06.245 INFO:tasks.rados.rados.0.smithi084.stdout:1548: finishing copy_from to smithi08477203-7 2024-09-20T22:48:06.245 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 7 v 838 (ObjNum 400 snap 101 seq_num 400) dirty exists 2024-09-20T22:48:06.245 INFO:tasks.rados.rados.0.smithi084.stdout:1552: finishing write tid 2 to smithi08477203-6 2024-09-20T22:48:06.249 INFO:tasks.rados.rados.0.smithi084.stdout:1552: finishing write tid 3 to smithi08477203-6 2024-09-20T22:48:06.249 INFO:tasks.rados.rados.0.smithi084.stdout:1552: finishing write tid 4 to smithi08477203-6 2024-09-20T22:48:06.250 INFO:tasks.rados.rados.0.smithi084.stdout:1552: finishing write tid 5 to smithi08477203-6 2024-09-20T22:48:06.250 INFO:tasks.rados.rados.0.smithi084.stdout:1552: oid 6 updating version 0 to 310 2024-09-20T22:48:06.250 INFO:tasks.rados.rados.0.smithi084.stdout:1552: oid 6 updating version 310 to 313 2024-09-20T22:48:06.250 INFO:tasks.rados.rados.0.smithi084.stdout:1552: oid 6 version 313 is already newer than 312 2024-09-20T22:48:06.250 INFO:tasks.rados.rados.0.smithi084.stdout:1552: oid 6 version 313 is already newer than 311 2024-09-20T22:48:06.251 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 6 v 313 (ObjNum 549 snap 167 seq_num 549) dirty exists 2024-09-20T22:48:06.251 INFO:tasks.rados.rados.0.smithi084.stdout:1552: left oid 6 (ObjNum 549 snap 167 seq_num 549) 2024-09-20T22:48:06.301 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:48:06.326 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:06.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:06.327 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:48:06.428 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:48:06.449 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:06.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:06.450 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:48:06.551 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:48:06.561 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:06 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/2188410868' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-20T22:48:06.561 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:06 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-20T22:48:06.561 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:06 smithi084 ceph-mon[29774]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:48:06.561 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:06 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T22:48:06.562 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:06 smithi084 ceph-mon[29774]: osdmap e406: 8 total, 8 up, 8 in 2024-09-20T22:48:06.562 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:06 smithi084 ceph-mon[29774]: 1.3 starting backfill to osd.4 from (0'0,0'0] MAX to 77'128 2024-09-20T22:48:06.562 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:06 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/2188410868' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-20T22:48:06.562 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:06 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-20T22:48:06.562 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:06 smithi084 ceph-mon[40024]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:48:06.562 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:06 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T22:48:06.562 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:06 smithi084 ceph-mon[40024]: osdmap e406: 8 total, 8 up, 8 in 2024-09-20T22:48:06.562 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:06 smithi084 ceph-mon[40024]: 1.3 starting backfill to osd.4 from (0'0,0'0] MAX to 77'128 2024-09-20T22:48:06.572 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:06.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:06.573 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:48:06.674 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:48:06.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:06.696 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:06.697 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:48:06.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:06 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/2188410868' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-20T22:48:06.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:06 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-20T22:48:06.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:06 smithi154 ceph-mon[33264]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:48:06.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:06 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T22:48:06.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:06 smithi154 ceph-mon[33264]: osdmap e406: 8 total, 8 up, 8 in 2024-09-20T22:48:06.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:06 smithi154 ceph-mon[33264]: 1.3 starting backfill to osd.4 from (0'0,0'0] MAX to 77'128 2024-09-20T22:48:06.797 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:48:06.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:06.819 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:06.820 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:48:06.920 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:48:06.941 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:06.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:06.942 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:48:06.964 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.3 dump_blocked_ops 2024-09-20T22:48:07.029 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:48:07.043 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:48:07.064 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:07.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:07.065 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:48:07.166 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:48:07.187 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:07.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:07.188 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:48:07.221 INFO:tasks.rados.rados.0.smithi084.stdout:1548: done (2 left) 2024-09-20T22:48:07.221 INFO:tasks.rados.rados.0.smithi084.stdout:1552: done (1 left) 2024-09-20T22:48:07.221 INFO:tasks.rados.rados.0.smithi084.stdout:1553: done (0 left) 2024-09-20T22:48:07.221 INFO:tasks.rados.rados.0.smithi084.stdout:1554: copy_from oid 46 from oid 1 current snap is 168 2024-09-20T22:48:07.221 INFO:tasks.rados.rados.0.smithi084.stdout:1555: delete oid 8 current snap is 168 2024-09-20T22:48:07.230 INFO:tasks.rados.rados.0.smithi084.stdout:1554: finishing copy_from racing read to smithi08477203-46 2024-09-20T22:48:07.233 INFO:tasks.rados.rados.0.smithi084.stdout:1554: finishing copy_from to smithi08477203-46 2024-09-20T22:48:07.233 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 46 v 539 (ObjNum 498 snap 142 seq_num 498) dirty exists 2024-09-20T22:48:07.289 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:48:07.310 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:07.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:07.311 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:48:07.412 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:48:07.434 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:07.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:07.435 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:48:07.535 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:48:07.549 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:07 smithi084 ceph-mon[40024]: pgmap v510: 168 pgs: 1 active+clean+snaptrim, 167 active+clean; 305 MiB data, 921 MiB used, 625 GiB / 626 GiB avail; 15 MiB/s rd, 1.9 MiB/s wr, 22 op/s 2024-09-20T22:48:07.549 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:07 smithi084 ceph-mon[40024]: 3.a scrub starts 2024-09-20T22:48:07.549 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:07 smithi084 ceph-mon[40024]: 3.a scrub ok 2024-09-20T22:48:07.549 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:07 smithi084 ceph-mon[40024]: 1.2 starting backfill to osd.4 from (0'0,0'0] MAX to 77'128 2024-09-20T22:48:07.549 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:07 smithi084 ceph-mon[40024]: 1.5 starting backfill to osd.4 from (0'0,0'0] MAX to 77'128 2024-09-20T22:48:07.549 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:07 smithi084 ceph-mon[40024]: purged_snaps scrub starts 2024-09-20T22:48:07.549 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:07 smithi084 ceph-mon[40024]: purged_snaps scrub ok 2024-09-20T22:48:07.549 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:07 smithi084 ceph-mon[40024]: osdmap e407: 8 total, 8 up, 8 in 2024-09-20T22:48:07.549 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:07 smithi084 ceph-mon[29774]: pgmap v510: 168 pgs: 1 active+clean+snaptrim, 167 active+clean; 305 MiB data, 921 MiB used, 625 GiB / 626 GiB avail; 15 MiB/s rd, 1.9 MiB/s wr, 22 op/s 2024-09-20T22:48:07.550 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:07 smithi084 ceph-mon[29774]: 3.a scrub starts 2024-09-20T22:48:07.550 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:07 smithi084 ceph-mon[29774]: 3.a scrub ok 2024-09-20T22:48:07.550 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:07 smithi084 ceph-mon[29774]: 1.2 starting backfill to osd.4 from (0'0,0'0] MAX to 77'128 2024-09-20T22:48:07.550 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:07 smithi084 ceph-mon[29774]: 1.5 starting backfill to osd.4 from (0'0,0'0] MAX to 77'128 2024-09-20T22:48:07.550 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:07 smithi084 ceph-mon[29774]: purged_snaps scrub starts 2024-09-20T22:48:07.550 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:07 smithi084 ceph-mon[29774]: purged_snaps scrub ok 2024-09-20T22:48:07.550 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:07 smithi084 ceph-mon[29774]: osdmap e407: 8 total, 8 up, 8 in 2024-09-20T22:48:07.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:07.557 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:07.557 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:48:07.658 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:48:07.679 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:07.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:07.680 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:48:07.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:07 smithi154 ceph-mon[33264]: pgmap v510: 168 pgs: 1 active+clean+snaptrim, 167 active+clean; 305 MiB data, 921 MiB used, 625 GiB / 626 GiB avail; 15 MiB/s rd, 1.9 MiB/s wr, 22 op/s 2024-09-20T22:48:07.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:07 smithi154 ceph-mon[33264]: 3.a scrub starts 2024-09-20T22:48:07.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:07 smithi154 ceph-mon[33264]: 3.a scrub ok 2024-09-20T22:48:07.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:07 smithi154 ceph-mon[33264]: 1.2 starting backfill to osd.4 from (0'0,0'0] MAX to 77'128 2024-09-20T22:48:07.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:07 smithi154 ceph-mon[33264]: 1.5 starting backfill to osd.4 from (0'0,0'0] MAX to 77'128 2024-09-20T22:48:07.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:07 smithi154 ceph-mon[33264]: purged_snaps scrub starts 2024-09-20T22:48:07.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:07 smithi154 ceph-mon[33264]: purged_snaps scrub ok 2024-09-20T22:48:07.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:07 smithi154 ceph-mon[33264]: osdmap e407: 8 total, 8 up, 8 in 2024-09-20T22:48:07.781 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:48:07.802 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:07.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:07.803 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:48:07.904 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:48:07.925 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:07.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:07.926 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:48:08.026 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:48:08.048 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:08.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:08.049 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:48:08.069 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:48:08.069 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:48:08.070 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:48:08.070 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:48:08.087 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:48:08.087 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:48:08.108 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-20T22:48:08.108 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T22:48:08.109 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:48:08.125 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:48:08.126 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:48:08.144 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:48:08.144 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:48:08.149 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:48:08.161 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:48:08.161 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:48:08.170 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:08.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:08.170 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:48:08.178 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:48:08.178 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:48:08.214 INFO:tasks.rados.rados.0.smithi084.stdout:1554: done (1 left) 2024-09-20T22:48:08.214 INFO:tasks.rados.rados.0.smithi084.stdout:1555: done (0 left) 2024-09-20T22:48:08.214 INFO:tasks.rados.rados.0.smithi084.stdout:1556: read oid 31 snap -1 2024-09-20T22:48:08.214 INFO:tasks.rados.rados.0.smithi084.stdout:1556: expect (ObjNum 518 snap 151 seq_num 518) 2024-09-20T22:48:08.214 INFO:tasks.rados.rados.0.smithi084.stdout:1557: write (excl) oid 32 current snap is 168 2024-09-20T22:48:08.214 INFO:tasks.rados.rados.0.smithi084.stdout:1557: seq_num 551 ranges {460657=639615,1693343=476514,2681716=578941} 2024-09-20T22:48:08.220 INFO:tasks.rados.rados.0.smithi084.stdout:1557: writing smithi08477203-32 from 460657 to 1100272 tid 1 2024-09-20T22:48:08.226 INFO:tasks.rados.rados.0.smithi084.stdout:1557: writing smithi08477203-32 from 1693343 to 2169857 tid 2 2024-09-20T22:48:08.233 INFO:tasks.rados.rados.0.smithi084.stdout:1557: writing smithi08477203-32 from 2681716 to 3260657 tid 3 2024-09-20T22:48:08.233 INFO:tasks.rados.rados.0.smithi084.stdout:1558: copy_from oid 33 from oid 25 current snap is 168 2024-09-20T22:48:08.233 INFO:tasks.rados.rados.0.smithi084.stdout:1556: expect (ObjNum 518 snap 151 seq_num 518) 2024-09-20T22:48:08.272 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:48:08.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:08.295 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:08.295 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:48:08.364 INFO:tasks.rados.rados.0.smithi084.stdout:1556: done (2 left) 2024-09-20T22:48:08.365 INFO:tasks.rados.rados.0.smithi084.stdout:1559: copy_from oid 23 from oid 36 current snap is 168 2024-09-20T22:48:08.365 INFO:tasks.rados.rados.0.smithi084.stdout:1560: rollback oid 27 current snap is 168 2024-09-20T22:48:08.365 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 27 to 155 2024-09-20T22:48:08.365 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:48:08 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: ::ffff:172.21.15.154 - - [20/Sep/2024:22:48:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-20T22:48:08.365 INFO:tasks.rados.rados.0.smithi084.stdout:1561: snap_remove snap 125 2024-09-20T22:48:08.396 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:48:08.417 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:08.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:08.418 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:48:08.519 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:48:08.540 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:08.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:08.541 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:48:08.642 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:48:08.663 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:08.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:08.664 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:48:08.765 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:48:08.786 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:08.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:08.787 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:48:08.887 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:48:08.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:08.910 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:08.910 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:48:09.011 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:48:09.033 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:09.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:09.034 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:48:09.135 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:48:09.156 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:09.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:09.157 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:48:09.213 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-20T22:48:09.213 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:48:09.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:08 smithi154 ceph-mon[33264]: pgmap v513: 168 pgs: 1 active+clean+snaptrim, 167 active+clean; 305 MiB data, 921 MiB used, 625 GiB / 626 GiB avail 2024-09-20T22:48:09.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:08 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:48:09.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:08 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:48:09.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:08 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/379947085' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:48:09.252 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:08 smithi154 ceph-mon[33264]: osdmap e408: 8 total, 8 up, 8 in 2024-09-20T22:48:09.258 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:48:09.279 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:09.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:09.280 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:48:09.292 INFO:tasks.rados.rados.0.smithi084.stdout:1561: done (4 left) 2024-09-20T22:48:09.292 INFO:tasks.rados.rados.0.smithi084.stdout:1562: write (excl) oid 34 current snap is 168 2024-09-20T22:48:09.292 INFO:tasks.rados.rados.0.smithi084.stdout:1562: seq_num 552 ranges {477464=714864,1751825=736525,3225492=51972} 2024-09-20T22:48:09.299 INFO:tasks.rados.rados.0.smithi084.stdout:1562: writing smithi08477203-34 from 477464 to 1192328 tid 1 2024-09-20T22:48:09.307 INFO:tasks.rados.rados.0.smithi084.stdout:1562: writing smithi08477203-34 from 1751825 to 2488350 tid 2 2024-09-20T22:48:09.313 INFO:tasks.rados.rados.0.smithi084.stdout:1562: writing smithi08477203-34 from 3225492 to 3277464 tid 3 2024-09-20T22:48:09.313 INFO:tasks.rados.rados.0.smithi084.stdout:1563: read oid 47 snap -1 2024-09-20T22:48:09.313 INFO:tasks.rados.rados.0.smithi084.stdout:1563: expect (ObjNum 546 snap 165 seq_num 546) 2024-09-20T22:48:09.313 INFO:tasks.rados.rados.0.smithi084.stdout:1564: copy_from oid 18 from oid 50 current snap is 168 2024-09-20T22:48:09.313 INFO:tasks.rados.rados.0.smithi084.stdout:1557: finishing write tid 1 to smithi08477203-32 2024-09-20T22:48:09.313 INFO:tasks.rados.rados.0.smithi084.stdout:1557: finishing write tid 2 to smithi08477203-32 2024-09-20T22:48:09.313 INFO:tasks.rados.rados.0.smithi084.stdout:1557: finishing write tid 3 to smithi08477203-32 2024-09-20T22:48:09.313 INFO:tasks.rados.rados.0.smithi084.stdout:1557: finishing write tid 4 to smithi08477203-32 2024-09-20T22:48:09.313 INFO:tasks.rados.rados.0.smithi084.stdout:1557: finishing write tid 5 to smithi08477203-32 2024-09-20T22:48:09.313 INFO:tasks.rados.rados.0.smithi084.stdout:1557: oid 32 updating version 0 to 395 2024-09-20T22:48:09.313 INFO:tasks.rados.rados.0.smithi084.stdout:1557: oid 32 version 395 is already newer than 394 2024-09-20T22:48:09.313 INFO:tasks.rados.rados.0.smithi084.stdout:1557: oid 32 version 395 is already newer than 393 2024-09-20T22:48:09.314 INFO:tasks.rados.rados.0.smithi084.stdout:1557: oid 32 updating version 395 to 396 2024-09-20T22:48:09.314 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 32 v 396 (ObjNum 551 snap 168 seq_num 551) dirty exists 2024-09-20T22:48:09.314 INFO:tasks.rados.rados.0.smithi084.stdout:1557: left oid 32 (ObjNum 551 snap 168 seq_num 551) 2024-09-20T22:48:09.314 INFO:tasks.rados.rados.0.smithi084.stdout:1558: finishing copy_from racing read to smithi08477203-33 2024-09-20T22:48:09.314 INFO:tasks.rados.rados.0.smithi084.stdout:1558: finishing copy_from to smithi08477203-33 2024-09-20T22:48:09.314 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 33 v 696 (ObjNum 458 snap 126 seq_num 458) dirty exists 2024-09-20T22:48:09.314 INFO:tasks.rados.rados.0.smithi084.stdout:1557: done (6 left) 2024-09-20T22:48:09.314 INFO:tasks.rados.rados.0.smithi084.stdout:1558: done (5 left) 2024-09-20T22:48:09.314 INFO:tasks.rados.rados.0.smithi084.stdout:1565: read oid 1 snap -1 2024-09-20T22:48:09.314 INFO:tasks.rados.rados.0.smithi084.stdout:1565: expect (ObjNum 498 snap 142 seq_num 498) 2024-09-20T22:48:09.314 INFO:tasks.rados.rados.0.smithi084.stdout:1566: snap_create 2024-09-20T22:48:09.314 INFO:tasks.rados.rados.0.smithi084.stdout:1560: finishing rollback tid 1 to smithi08477203-27 2024-09-20T22:48:09.314 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 27 v 323 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-09-20T22:48:09.314 INFO:tasks.rados.rados.0.smithi084.stdout:1559: finishing copy_from racing read to smithi08477203-23 2024-09-20T22:48:09.315 INFO:tasks.rados.rados.0.smithi084.stdout:1559: finishing copy_from to smithi08477203-23 2024-09-20T22:48:09.315 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 23 v 100 (ObjNum 440 snap 115 seq_num 440) dirty exists 2024-09-20T22:48:09.315 INFO:tasks.rados.rados.0.smithi084.stdout:1562: finishing write tid 1 to smithi08477203-34 2024-09-20T22:48:09.315 INFO:tasks.rados.rados.0.smithi084.stdout:1562: finishing write tid 2 to smithi08477203-34 2024-09-20T22:48:09.315 INFO:tasks.rados.rados.0.smithi084.stdout:1562: finishing write tid 3 to smithi08477203-34 2024-09-20T22:48:09.316 INFO:tasks.rados.rados.0.smithi084.stdout:1562: finishing write tid 4 to smithi08477203-34 2024-09-20T22:48:09.316 INFO:tasks.rados.rados.0.smithi084.stdout:1562: finishing write tid 5 to smithi08477203-34 2024-09-20T22:48:09.316 INFO:tasks.rados.rados.0.smithi084.stdout:1562: oid 34 updating version 0 to 526 2024-09-20T22:48:09.316 INFO:tasks.rados.rados.0.smithi084.stdout:1562: oid 34 version 526 is already newer than 524 2024-09-20T22:48:09.316 INFO:tasks.rados.rados.0.smithi084.stdout:1562: oid 34 version 526 is already newer than 525 2024-09-20T22:48:09.316 INFO:tasks.rados.rados.0.smithi084.stdout:1562: oid 34 updating version 526 to 527 2024-09-20T22:48:09.316 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 34 v 527 (ObjNum 552 snap 168 seq_num 552) dirty exists 2024-09-20T22:48:09.316 INFO:tasks.rados.rados.0.smithi084.stdout:1562: left oid 34 (ObjNum 552 snap 168 seq_num 552) 2024-09-20T22:48:09.316 INFO:tasks.rados.rados.0.smithi084.stdout:1565: expect (ObjNum 498 snap 142 seq_num 498) 2024-09-20T22:48:09.364 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:08 smithi084 ceph-mon[29774]: pgmap v513: 168 pgs: 1 active+clean+snaptrim, 167 active+clean; 305 MiB data, 921 MiB used, 625 GiB / 626 GiB avail 2024-09-20T22:48:09.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:08 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:48:09.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:08 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:48:09.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:08 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/379947085' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:48:09.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:08 smithi084 ceph-mon[29774]: osdmap e408: 8 total, 8 up, 8 in 2024-09-20T22:48:09.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:08 smithi084 ceph-mon[40024]: pgmap v513: 168 pgs: 1 active+clean+snaptrim, 167 active+clean; 305 MiB data, 921 MiB used, 625 GiB / 626 GiB avail 2024-09-20T22:48:09.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:08 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:48:09.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:08 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:48:09.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:08 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/379947085' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:48:09.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:08 smithi084 ceph-mon[40024]: osdmap e408: 8 total, 8 up, 8 in 2024-09-20T22:48:09.381 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:48:09.403 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:09.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:09.404 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:48:09.433 INFO:tasks.rados.rados.0.smithi084.stdout:1564: finishing copy_from racing read to smithi08477203-18 2024-09-20T22:48:09.433 INFO:tasks.rados.rados.0.smithi084.stdout:1563: expect (ObjNum 546 snap 165 seq_num 546) 2024-09-20T22:48:09.504 INFO:journalctl@ceph.iscsi.iscsi.a.smithi154.stdout:Sep 20 22:48:09 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-iscsi-iscsi-a[59730]: debug there is no tcmu-runner data available 2024-09-20T22:48:09.505 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:48:09.526 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:09.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:09.527 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:48:09.552 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-20T22:48:09.628 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:48:09.651 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:09.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:09.652 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:48:09.664 INFO:tasks.rados.rados.0.smithi084.stdout:1564: finishing copy_from to smithi08477203-18 2024-09-20T22:48:09.664 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 18 v 252 (ObjNum 515 snap 148 seq_num 515) dirty exists 2024-09-20T22:48:09.753 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:48:09.773 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:09.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:09.774 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:48:09.875 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:48:09.896 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:09.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:09.897 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:48:09.926 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.3 dump_historic_ops 2024-09-20T22:48:09.941 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 5, 6, 7, 0, 4] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-20T22:48:09.941 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-20T22:48:09.941 INFO:tasks.thrashosds.thrasher:Shrinking pool unique_pool_0 2024-09-20T22:48:09.941 INFO:tasks.cephadm.ceph_manager.ceph:contract_pool unique_pool_0 by 10 min 8 2024-09-20T22:48:09.942 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph pg dump --format=json 2024-09-20T22:48:09.998 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:48:09.998 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:48:10.058 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:10.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:10.059 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:48:10.160 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:48:10.183 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:10.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:10.184 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:48:10.285 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:48:10.300 INFO:tasks.rados.rados.0.smithi084.stdout:1559: done (6 left) 2024-09-20T22:48:10.300 INFO:tasks.rados.rados.0.smithi084.stdout:1560: done (5 left) 2024-09-20T22:48:10.300 INFO:tasks.rados.rados.0.smithi084.stdout:1562: done (4 left) 2024-09-20T22:48:10.300 INFO:tasks.rados.rados.0.smithi084.stdout:1563: done (3 left) 2024-09-20T22:48:10.301 INFO:tasks.rados.rados.0.smithi084.stdout:1564: done (2 left) 2024-09-20T22:48:10.301 INFO:tasks.rados.rados.0.smithi084.stdout:1565: done (1 left) 2024-09-20T22:48:10.301 INFO:tasks.rados.rados.0.smithi084.stdout:1566: done (0 left) 2024-09-20T22:48:10.301 INFO:tasks.rados.rados.0.smithi084.stdout:1567: rollback oid 40 current snap is 169 2024-09-20T22:48:10.301 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 40 to 165 2024-09-20T22:48:10.301 INFO:tasks.rados.rados.0.smithi084.stdout:1568: rollback oid 1 current snap is 169 2024-09-20T22:48:10.301 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 1 to 140 2024-09-20T22:48:10.301 INFO:tasks.rados.rados.0.smithi084.stdout:1569: read oid 23 snap -1 2024-09-20T22:48:10.301 INFO:tasks.rados.rados.0.smithi084.stdout:1569: expect (ObjNum 440 snap 115 seq_num 440) 2024-09-20T22:48:10.301 INFO:tasks.rados.rados.0.smithi084.stdout:1570: copy_from oid 46 from oid 44 current snap is 169 2024-09-20T22:48:10.301 INFO:tasks.rados.rados.0.smithi084.stdout:1571: read oid 43 snap -1 2024-09-20T22:48:10.301 INFO:tasks.rados.rados.0.smithi084.stdout:1571: expect (ObjNum 526 snap 151 seq_num 526) 2024-09-20T22:48:10.302 INFO:tasks.rados.rados.0.smithi084.stdout:1572: delete oid 3 current snap is 169 2024-09-20T22:48:10.309 INFO:tasks.rados.rados.0.smithi084.stdout:1568: finishing rollback tid 0 to smithi08477203-1 2024-09-20T22:48:10.309 INFO:tasks.rados.rados.0.smithi084.stdout:1568: finishing rollback tid 1 to smithi08477203-1 2024-09-20T22:48:10.309 INFO:tasks.rados.rados.0.smithi084.stdout:1568: finishing rollback tid 2 to smithi08477203-1 2024-09-20T22:48:10.310 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 1 v 545 (ObjNum 495 snap 139 seq_num 495) dirty exists 2024-09-20T22:48:10.313 INFO:tasks.rados.rados.0.smithi084.stdout:1571: expect (ObjNum 526 snap 151 seq_num 526) 2024-09-20T22:48:10.319 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:10.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:10.320 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:48:10.421 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:10 smithi084 ceph-mon[40024]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:48:10.421 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:10 smithi084 ceph-mon[40024]: osdmap e409: 8 total, 8 up, 8 in 2024-09-20T22:48:10.421 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:10 smithi084 ceph-mon[29774]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:48:10.421 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:10 smithi084 ceph-mon[29774]: osdmap e409: 8 total, 8 up, 8 in 2024-09-20T22:48:10.421 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:48:10.442 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:10.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:10.444 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:48:10.471 INFO:tasks.rados.rados.0.smithi084.stdout:1569: expect (ObjNum 440 snap 115 seq_num 440) 2024-09-20T22:48:10.544 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:48:10.574 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:48:10.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:10.576 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:10.576 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:48:10.579 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:48:10.677 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:48:10.698 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:10.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:10.699 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:48:10.750 INFO:tasks.rados.rados.0.smithi084.stdout:1567: finishing rollback tid 1 to smithi08477203-40 2024-09-20T22:48:10.750 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 40 v 511 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-09-20T22:48:10.750 INFO:tasks.rados.rados.0.smithi084.stdout:1570: finishing copy_from racing read to smithi08477203-46 2024-09-20T22:48:10.750 INFO:tasks.rados.rados.0.smithi084.stdout:1570: finishing copy_from to smithi08477203-46 2024-09-20T22:48:10.750 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 46 v 722 (ObjNum 470 snap 130 seq_num 470) dirty exists 2024-09-20T22:48:10.750 INFO:tasks.rados.rados.0.smithi084.stdout:1567: done (5 left) 2024-09-20T22:48:10.750 INFO:tasks.rados.rados.0.smithi084.stdout:1568: done (4 left) 2024-09-20T22:48:10.750 INFO:tasks.rados.rados.0.smithi084.stdout:1569: done (3 left) 2024-09-20T22:48:10.750 INFO:tasks.rados.rados.0.smithi084.stdout:1570: done (2 left) 2024-09-20T22:48:10.751 INFO:tasks.rados.rados.0.smithi084.stdout:1571: done (1 left) 2024-09-20T22:48:10.751 INFO:tasks.rados.rados.0.smithi084.stdout:1572: done (0 left) 2024-09-20T22:48:10.751 INFO:tasks.rados.rados.0.smithi084.stdout:1573: snap_remove snap 166 2024-09-20T22:48:10.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:10 smithi154 ceph-mon[33264]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:48:10.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:10 smithi154 ceph-mon[33264]: osdmap e409: 8 total, 8 up, 8 in 2024-09-20T22:48:10.800 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:48:10.826 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:10.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:10.827 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:48:10.928 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:48:10.951 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:10.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:10.952 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:48:11.041 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-20T22:48:11.052 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:48:11.077 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:11.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:11.078 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:48:11.179 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:48:11.205 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:11.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:11.206 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:48:11.299 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is set 2024-09-20T22:48:11.303 INFO:tasks.rados.rados.0.smithi084.stdout:1573: done (0 left) 2024-09-20T22:48:11.303 INFO:tasks.rados.rados.0.smithi084.stdout:1574: write oid 42 current snap is 169 2024-09-20T22:48:11.303 INFO:tasks.rados.rados.0.smithi084.stdout:1574: seq_num 554 ranges {511078=465362,1752436=740194,3163985=147093} 2024-09-20T22:48:11.307 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:48:11.309 INFO:tasks.rados.rados.0.smithi084.stdout:1574: writing smithi08477203-42 from 511078 to 976440 tid 1 2024-09-20T22:48:11.317 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:11 smithi084 ceph-mon[40024]: pgmap v516: 168 pgs: 168 active+clean; 321 MiB data, 1018 MiB used, 714 GiB / 715 GiB avail; 541 KiB/s rd, 4.0 MiB/s wr, 8 op/s; 11 MiB/s, 34 objects/s recovering 2024-09-20T22:48:11.317 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:11 smithi084 ceph-mon[40024]: osdmap e410: 8 total, 8 up, 8 in 2024-09-20T22:48:11.317 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:11 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/1869011604' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:48:11.318 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:11 smithi084 ceph-mon[29774]: pgmap v516: 168 pgs: 168 active+clean; 321 MiB data, 1018 MiB used, 714 GiB / 715 GiB avail; 541 KiB/s rd, 4.0 MiB/s wr, 8 op/s; 11 MiB/s, 34 objects/s recovering 2024-09-20T22:48:11.318 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:11 smithi084 ceph-mon[29774]: osdmap e410: 8 total, 8 up, 8 in 2024-09-20T22:48:11.318 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:11 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/1869011604' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:48:11.318 INFO:tasks.rados.rados.0.smithi084.stdout:1574: writing smithi08477203-42 from 1752436 to 2492630 tid 2 2024-09-20T22:48:11.324 INFO:tasks.rados.rados.0.smithi084.stdout:1574: writing smithi08477203-42 from 3163985 to 3311078 tid 3 2024-09-20T22:48:11.324 INFO:tasks.rados.rados.0.smithi084.stdout:1575: read oid 32 snap -1 2024-09-20T22:48:11.324 INFO:tasks.rados.rados.0.smithi084.stdout:1575: expect (ObjNum 551 snap 168 seq_num 551) 2024-09-20T22:48:11.324 INFO:tasks.rados.rados.0.smithi084.stdout:1576: copy_from oid 3 from oid 21 current snap is 169 2024-09-20T22:48:11.324 INFO:tasks.rados.rados.0.smithi084.stdout:1574: finishing write tid 1 to smithi08477203-42 2024-09-20T22:48:11.324 INFO:tasks.rados.rados.0.smithi084.stdout:1574: finishing write tid 2 to smithi08477203-42 2024-09-20T22:48:11.324 INFO:tasks.rados.rados.0.smithi084.stdout:1577: read oid 40 snap -1 2024-09-20T22:48:11.324 INFO:tasks.rados.rados.0.smithi084.stdout:1577: expect deleted 2024-09-20T22:48:11.324 INFO:tasks.rados.rados.0.smithi084.stdout:1578: read oid 10 snap -1 2024-09-20T22:48:11.324 INFO:tasks.rados.rados.0.smithi084.stdout:1578: expect (ObjNum 544 snap 164 seq_num 544) 2024-09-20T22:48:11.324 INFO:tasks.rados.rados.0.smithi084.stdout:1579: copy_from oid 28 from oid 16 current snap is 169 2024-09-20T22:48:11.325 INFO:tasks.rados.rados.0.smithi084.stdout:1580: delete oid 7 current snap is 169 2024-09-20T22:48:11.327 INFO:tasks.rados.rados.0.smithi084.stdout:1577: done (6 left) 2024-09-20T22:48:11.327 INFO:tasks.rados.rados.0.smithi084.stdout:1580: done (5 left) 2024-09-20T22:48:11.327 INFO:tasks.rados.rados.0.smithi084.stdout:1581: write oid 11 current snap is 169 2024-09-20T22:48:11.327 INFO:tasks.rados.rados.0.smithi084.stdout:1574: finishing write tid 3 to smithi08477203-42 2024-09-20T22:48:11.327 INFO:tasks.rados.rados.0.smithi084.stdout:1581: seq_num 556 ranges {544692=732213,1985753=743863,3335184=9508} 2024-09-20T22:48:11.333 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:11.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:11.334 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:48:11.336 INFO:tasks.rados.rados.0.smithi084.stdout:1581: writing smithi08477203-11 from 544692 to 1276905 tid 1 2024-09-20T22:48:11.344 INFO:tasks.rados.rados.0.smithi084.stdout:1581: writing smithi08477203-11 from 1985753 to 2729616 tid 2 2024-09-20T22:48:11.348 INFO:tasks.rados.rados.0.smithi084.stdout:1581: writing smithi08477203-11 from 3335184 to 3344692 tid 3 2024-09-20T22:48:11.348 INFO:tasks.rados.rados.0.smithi084.stdout:1582: write (excl) oid 49 current snap is 169 2024-09-20T22:48:11.348 INFO:tasks.rados.rados.0.smithi084.stdout:1582: seq_num 557 ranges {561499=407462,1527882=487521,2729901=557217} 2024-09-20T22:48:11.353 INFO:tasks.rados.rados.0.smithi084.stdout:1582: writing smithi08477203-49 from 561499 to 968961 tid 1 2024-09-20T22:48:11.360 INFO:tasks.rados.rados.0.smithi084.stdout:1582: writing smithi08477203-49 from 1527882 to 2015403 tid 2 2024-09-20T22:48:11.367 INFO:tasks.rados.rados.0.smithi084.stdout:1582: writing smithi08477203-49 from 2729901 to 3287118 tid 3 2024-09-20T22:48:11.367 INFO:tasks.rados.rados.0.smithi084.stdout:1583: read oid 18 snap 126 2024-09-20T22:48:11.367 INFO:tasks.rados.rados.0.smithi084.stdout:1583: expect (ObjNum 390 snap 101 seq_num 390) 2024-09-20T22:48:11.367 INFO:tasks.rados.rados.0.smithi084.stdout:1584: read oid 2 snap 162 2024-09-20T22:48:11.367 INFO:tasks.rados.rados.0.smithi084.stdout:1584: expect (ObjNum 507 snap 144 seq_num 507) 2024-09-20T22:48:11.367 INFO:tasks.rados.rados.0.smithi084.stdout:1585: delete oid 7 current snap is 169 2024-09-20T22:48:11.367 INFO:tasks.rados.rados.0.smithi084.stdout:1574: finishing write tid 4 to smithi08477203-42 2024-09-20T22:48:11.368 INFO:tasks.rados.rados.0.smithi084.stdout:1574: finishing write tid 5 to smithi08477203-42 2024-09-20T22:48:11.368 INFO:tasks.rados.rados.0.smithi084.stdout:1574: oid 42 updating version 0 to 386 2024-09-20T22:48:11.368 INFO:tasks.rados.rados.0.smithi084.stdout:1574: oid 42 updating version 386 to 388 2024-09-20T22:48:11.368 INFO:tasks.rados.rados.0.smithi084.stdout:1574: oid 42 updating version 388 to 389 2024-09-20T22:48:11.368 INFO:tasks.rados.rados.0.smithi084.stdout:1574: oid 42 version 389 is already newer than 387 2024-09-20T22:48:11.368 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 42 v 389 (ObjNum 554 snap 169 seq_num 554) dirty exists 2024-09-20T22:48:11.368 INFO:tasks.rados.rados.0.smithi084.stdout:1574: left oid 42 (ObjNum 554 snap 169 seq_num 554) 2024-09-20T22:48:11.368 INFO:tasks.rados.rados.0.smithi084.stdout:1575: expect (ObjNum 551 snap 168 seq_num 551) 2024-09-20T22:48:11.435 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:48:11.457 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:11.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:11.458 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:48:11.493 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-20T22:48:11.493 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:48:11.500 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-20T22:48:11.502 INFO:teuthology.orchestra.run.smithi084.stderr:dumped all 2024-09-20T22:48:11.559 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:48:11.566 INFO:tasks.rados.rados.0.smithi084.stdout:1579: finishing copy_from racing read to smithi08477203-28 2024-09-20T22:48:11.566 INFO:tasks.rados.rados.0.smithi084.stdout:1578: expect (ObjNum 544 snap 164 seq_num 544) 2024-09-20T22:48:11.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:11.583 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:11.583 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:48:11.684 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:48:11.708 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:11.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:11.709 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:48:11.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:11 smithi154 ceph-mon[33264]: pgmap v516: 168 pgs: 168 active+clean; 321 MiB data, 1018 MiB used, 714 GiB / 715 GiB avail; 541 KiB/s rd, 4.0 MiB/s wr, 8 op/s; 11 MiB/s, 34 objects/s recovering 2024-09-20T22:48:11.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:11 smithi154 ceph-mon[33264]: osdmap e410: 8 total, 8 up, 8 in 2024-09-20T22:48:11.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:11 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/1869011604' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:48:11.779 INFO:tasks.rados.rados.0.smithi084.stdout:1576: finishing copy_from racing read to smithi08477203-3 2024-09-20T22:48:11.779 INFO:tasks.rados.rados.0.smithi084.stdout:1579: finishing copy_from to smithi08477203-28 2024-09-20T22:48:11.779 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 28 v 422 (ObjNum 419 snap 105 seq_num 419) dirty exists 2024-09-20T22:48:11.780 INFO:tasks.rados.rados.0.smithi084.stdout:1581: finishing write tid 1 to smithi08477203-11 2024-09-20T22:48:11.780 INFO:tasks.rados.rados.0.smithi084.stdout:1581: finishing write tid 2 to smithi08477203-11 2024-09-20T22:48:11.780 INFO:tasks.rados.rados.0.smithi084.stdout:1574: done (9 left) 2024-09-20T22:48:11.780 INFO:tasks.rados.rados.0.smithi084.stdout:1575: done (8 left) 2024-09-20T22:48:11.780 INFO:tasks.rados.rados.0.smithi084.stdout:1578: done (7 left) 2024-09-20T22:48:11.780 INFO:tasks.rados.rados.0.smithi084.stdout:1579: done (6 left) 2024-09-20T22:48:11.780 INFO:tasks.rados.rados.0.smithi084.stdout:1585: done (5 left) 2024-09-20T22:48:11.780 INFO:tasks.rados.rados.0.smithi084.stdout:1586: snap_remove snap 136 2024-09-20T22:48:11.809 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:48:11.832 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:11.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:11.833 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:48:11.934 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:48:11.957 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:11.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:11.958 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:48:12.058 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:48:12.080 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:12.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:12.081 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:48:12.182 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:48:12.205 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:12.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:12.206 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:48:12.305 INFO:tasks.rados.rados.0.smithi084.stdout:1586: done (5 left) 2024-09-20T22:48:12.305 INFO:tasks.rados.rados.0.smithi084.stdout:1587: write (excl) oid 41 current snap is 169 2024-09-20T22:48:12.305 INFO:tasks.rados.rados.0.smithi084.stdout:1576: finishing copy_from to smithi08477203-3 2024-09-20T22:48:12.306 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 3 v 519 (ObjNum 501 snap 143 seq_num 501) dirty exists 2024-09-20T22:48:12.306 INFO:tasks.rados.rados.0.smithi084.stdout:1581: finishing write tid 3 to smithi08477203-11 2024-09-20T22:48:12.306 INFO:tasks.rados.rados.0.smithi084.stdout:1581: finishing write tid 4 to smithi08477203-11 2024-09-20T22:48:12.306 INFO:tasks.rados.rados.0.smithi084.stdout:1587: seq_num 559 ranges {595113=557960,1644846=491190,2784747=610366} 2024-09-20T22:48:12.307 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:48:12.312 INFO:tasks.rados.rados.0.smithi084.stdout:1587: writing smithi08477203-41 from 595113 to 1153073 tid 1 2024-09-20T22:48:12.318 INFO:tasks.rados.rados.0.smithi084.stdout:1587: writing smithi08477203-41 from 1644846 to 2136036 tid 2 2024-09-20T22:48:12.325 INFO:tasks.rados.rados.0.smithi084.stdout:1587: writing smithi08477203-41 from 2784747 to 3395113 tid 3 2024-09-20T22:48:12.325 INFO:tasks.rados.rados.0.smithi084.stdout:1576: done (5 left) 2024-09-20T22:48:12.326 INFO:tasks.rados.rados.0.smithi084.stdout:1588: write (excl) oid 5 current snap is 169 2024-09-20T22:48:12.326 INFO:tasks.rados.rados.0.smithi084.stdout:1581: finishing write tid 5 to smithi08477203-11 2024-09-20T22:48:12.326 INFO:tasks.rados.rados.0.smithi084.stdout:1581: oid 11 updating version 0 to 681 2024-09-20T22:48:12.326 INFO:tasks.rados.rados.0.smithi084.stdout:1581: oid 11 updating version 681 to 682 2024-09-20T22:48:12.326 INFO:tasks.rados.rados.0.smithi084.stdout:1581: oid 11 version 682 is already newer than 680 2024-09-20T22:48:12.326 INFO:tasks.rados.rados.0.smithi084.stdout:1581: oid 11 version 682 is already newer than 679 2024-09-20T22:48:12.326 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 11 v 682 (ObjNum 556 snap 169 seq_num 556) dirty exists 2024-09-20T22:48:12.326 INFO:tasks.rados.rados.0.smithi084.stdout:1581: left oid 11 (ObjNum 556 snap 169 seq_num 556) 2024-09-20T22:48:12.326 INFO:tasks.rados.rados.0.smithi084.stdout:1582: finishing write tid 1 to smithi08477203-49 2024-09-20T22:48:12.326 INFO:tasks.rados.rados.0.smithi084.stdout:1582: finishing write tid 2 to smithi08477203-49 2024-09-20T22:48:12.326 INFO:tasks.rados.rados.0.smithi084.stdout:1588: seq_num 560 ranges {611920=633209,1703328=634848,2812170=507386} 2024-09-20T22:48:12.332 INFO:tasks.rados.rados.0.smithi084.stdout:1588: writing smithi08477203-5 from 611920 to 1245129 tid 1 2024-09-20T22:48:12.334 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:12.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:12.335 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:48:12.338 INFO:tasks.rados.rados.0.smithi084.stdout:1588: writing smithi08477203-5 from 1703328 to 2338176 tid 2 2024-09-20T22:48:12.344 INFO:tasks.rados.rados.0.smithi084.stdout:1588: writing smithi08477203-5 from 2812170 to 3319556 tid 3 2024-09-20T22:48:12.344 INFO:tasks.rados.rados.0.smithi084.stdout:1581: done (5 left) 2024-09-20T22:48:12.344 INFO:tasks.rados.rados.0.smithi084.stdout:1589: rollback oid 36 current snap is 169 2024-09-20T22:48:12.344 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 36 to 147 2024-09-20T22:48:12.344 INFO:tasks.rados.rados.0.smithi084.stdout:1582: finishing write tid 3 to smithi08477203-49 2024-09-20T22:48:12.344 INFO:tasks.rados.rados.0.smithi084.stdout:1582: finishing write tid 4 to smithi08477203-49 2024-09-20T22:48:12.345 INFO:tasks.rados.rados.0.smithi084.stdout:1582: finishing write tid 5 to smithi08477203-49 2024-09-20T22:48:12.345 INFO:tasks.rados.rados.0.smithi084.stdout:1582: oid 49 updating version 0 to 528 2024-09-20T22:48:12.345 INFO:tasks.rados.rados.0.smithi084.stdout:1582: oid 49 updating version 528 to 529 2024-09-20T22:48:12.345 INFO:tasks.rados.rados.0.smithi084.stdout:1582: oid 49 updating version 529 to 530 2024-09-20T22:48:12.345 INFO:tasks.rados.rados.0.smithi084.stdout:1582: oid 49 updating version 530 to 531 2024-09-20T22:48:12.345 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 49 v 531 (ObjNum 557 snap 169 seq_num 557) dirty exists 2024-09-20T22:48:12.345 INFO:tasks.rados.rados.0.smithi084.stdout:1582: left oid 49 (ObjNum 557 snap 169 seq_num 557) 2024-09-20T22:48:12.345 INFO:tasks.rados.rados.0.smithi084.stdout:1583: expect (ObjNum 390 snap 101 seq_num 390) 2024-09-20T22:48:12.425 INFO:tasks.rados.rados.0.smithi084.stdout:1584: expect (ObjNum 507 snap 144 seq_num 507) 2024-09-20T22:48:12.436 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:48:12.440 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:12 smithi084 ceph-mon[40024]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:48:12.440 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:12 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/1869011604' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T22:48:12.440 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:12 smithi084 ceph-mon[40024]: osdmap e411: 8 total, 8 up, 8 in 2024-09-20T22:48:12.440 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:12 smithi084 ceph-mon[40024]: from='client.25730 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T22:48:12.440 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:12 smithi084 ceph-mon[29774]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:48:12.440 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:12 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/1869011604' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T22:48:12.440 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:12 smithi084 ceph-mon[29774]: osdmap e411: 8 total, 8 up, 8 in 2024-09-20T22:48:12.440 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:12 smithi084 ceph-mon[29774]: from='client.25730 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T22:48:12.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:12.458 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:12.458 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:48:12.540 INFO:tasks.rados.rados.0.smithi084.stdout:1587: finishing write tid 1 to smithi08477203-41 2024-09-20T22:48:12.540 INFO:tasks.rados.rados.0.smithi084.stdout:1587: finishing write tid 2 to smithi08477203-41 2024-09-20T22:48:12.541 INFO:tasks.rados.rados.0.smithi084.stdout:1587: finishing write tid 3 to smithi08477203-41 2024-09-20T22:48:12.541 INFO:tasks.rados.rados.0.smithi084.stdout:1587: finishing write tid 4 to smithi08477203-41 2024-09-20T22:48:12.541 INFO:tasks.rados.rados.0.smithi084.stdout:1587: finishing write tid 5 to smithi08477203-41 2024-09-20T22:48:12.541 INFO:tasks.rados.rados.0.smithi084.stdout:1587: oid 41 updating version 0 to 121 2024-09-20T22:48:12.541 INFO:tasks.rados.rados.0.smithi084.stdout:1587: oid 41 version 121 is already newer than 120 2024-09-20T22:48:12.541 INFO:tasks.rados.rados.0.smithi084.stdout:1587: oid 41 updating version 121 to 123 2024-09-20T22:48:12.541 INFO:tasks.rados.rados.0.smithi084.stdout:1587: oid 41 version 123 is already newer than 122 2024-09-20T22:48:12.541 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 41 v 123 (ObjNum 559 snap 169 seq_num 559) dirty exists 2024-09-20T22:48:12.541 INFO:tasks.rados.rados.0.smithi084.stdout:1587: left oid 41 (ObjNum 559 snap 169 seq_num 559) 2024-09-20T22:48:12.541 INFO:tasks.rados.rados.0.smithi084.stdout:1588: finishing write tid 1 to smithi08477203-5 2024-09-20T22:48:12.541 INFO:tasks.rados.rados.0.smithi084.stdout:1582: done (5 left) 2024-09-20T22:48:12.541 INFO:tasks.rados.rados.0.smithi084.stdout:1583: done (4 left) 2024-09-20T22:48:12.541 INFO:tasks.rados.rados.0.smithi084.stdout:1584: done (3 left) 2024-09-20T22:48:12.542 INFO:tasks.rados.rados.0.smithi084.stdout:1587: done (2 left) 2024-09-20T22:48:12.542 INFO:tasks.rados.rados.0.smithi084.stdout:1590: copy_from oid 20 from oid 48 current snap is 169 2024-09-20T22:48:12.542 INFO:tasks.rados.rados.0.smithi084.stdout:1588: finishing write tid 2 to smithi08477203-5 2024-09-20T22:48:12.542 INFO:tasks.rados.rados.0.smithi084.stdout:1589: finishing rollback tid 0 to smithi08477203-36 2024-09-20T22:48:12.542 INFO:tasks.rados.rados.0.smithi084.stdout:1588: finishing write tid 3 to smithi08477203-5 2024-09-20T22:48:12.542 INFO:tasks.rados.rados.0.smithi084.stdout:1589: finishing rollback tid 1 to smithi08477203-36 2024-09-20T22:48:12.542 INFO:tasks.rados.rados.0.smithi084.stdout:1588: finishing write tid 4 to smithi08477203-5 2024-09-20T22:48:12.542 INFO:tasks.rados.rados.0.smithi084.stdout:1588: finishing write tid 5 to smithi08477203-5 2024-09-20T22:48:12.542 INFO:tasks.rados.rados.0.smithi084.stdout:1588: oid 5 updating version 0 to 404 2024-09-20T22:48:12.542 INFO:tasks.rados.rados.0.smithi084.stdout:1588: oid 5 updating version 404 to 406 2024-09-20T22:48:12.542 INFO:tasks.rados.rados.0.smithi084.stdout:1588: oid 5 version 406 is already newer than 403 2024-09-20T22:48:12.542 INFO:tasks.rados.rados.0.smithi084.stdout:1588: oid 5 version 406 is already newer than 405 2024-09-20T22:48:12.542 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 5 v 406 (ObjNum 560 snap 169 seq_num 560) dirty exists 2024-09-20T22:48:12.542 INFO:tasks.rados.rados.0.smithi084.stdout:1588: left oid 5 (ObjNum 560 snap 169 seq_num 560) 2024-09-20T22:48:12.543 INFO:tasks.rados.rados.0.smithi084.stdout:1589: finishing rollback tid 2 to smithi08477203-36 2024-09-20T22:48:12.543 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 36 v 392 (ObjNum 440 snap 115 seq_num 440) dirty exists 2024-09-20T22:48:12.543 INFO:tasks.rados.rados.0.smithi084.stdout:1588: done (2 left) 2024-09-20T22:48:12.543 INFO:tasks.rados.rados.0.smithi084.stdout:1589: done (1 left) 2024-09-20T22:48:12.543 INFO:tasks.rados.rados.0.smithi084.stdout:1591: delete oid 43 current snap is 169 2024-09-20T22:48:12.543 INFO:tasks.rados.rados.0.smithi084.stdout:1591: done (1 left) 2024-09-20T22:48:12.543 INFO:tasks.rados.rados.0.smithi084.stdout:1592: write oid 6 current snap is 169 2024-09-20T22:48:12.543 INFO:tasks.rados.rados.0.smithi084.stdout:1592: seq_num 562 ranges {645534=783707,2103939=638517,3150663=294871} 2024-09-20T22:48:12.550 INFO:tasks.rados.rados.0.smithi084.stdout:1592: writing smithi08477203-6 from 645534 to 1429241 tid 1 2024-09-20T22:48:12.557 INFO:tasks.rados.rados.0.smithi084.stdout:1592: writing smithi08477203-6 from 2103939 to 2742456 tid 2 2024-09-20T22:48:12.559 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:48:12.561 INFO:tasks.rados.rados.0.smithi084.stdout:1592: writing smithi08477203-6 from 3150663 to 3445534 tid 3 2024-09-20T22:48:12.561 INFO:tasks.rados.rados.0.smithi084.stdout:1593: snap_create 2024-09-20T22:48:12.561 INFO:tasks.rados.rados.0.smithi084.stdout:1590: finishing copy_from racing read to smithi08477203-20 2024-09-20T22:48:12.561 INFO:tasks.rados.rados.0.smithi084.stdout:1590: finishing copy_from to smithi08477203-20 2024-09-20T22:48:12.561 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 20 v 300 (ObjNum 424 snap 109 seq_num 424) dirty exists 2024-09-20T22:48:12.562 INFO:tasks.rados.rados.0.smithi084.stdout:1592: finishing write tid 1 to smithi08477203-6 2024-09-20T22:48:12.562 INFO:tasks.rados.rados.0.smithi084.stdout:1592: finishing write tid 2 to smithi08477203-6 2024-09-20T22:48:12.563 INFO:tasks.rados.rados.0.smithi084.stdout:1592: finishing write tid 3 to smithi08477203-6 2024-09-20T22:48:12.563 INFO:tasks.rados.rados.0.smithi084.stdout:1592: finishing write tid 4 to smithi08477203-6 2024-09-20T22:48:12.563 INFO:tasks.rados.rados.0.smithi084.stdout:1592: finishing write tid 5 to smithi08477203-6 2024-09-20T22:48:12.563 INFO:tasks.rados.rados.0.smithi084.stdout:1592: oid 6 updating version 0 to 316 2024-09-20T22:48:12.563 INFO:tasks.rados.rados.0.smithi084.stdout:1592: oid 6 version 316 is already newer than 315 2024-09-20T22:48:12.563 INFO:tasks.rados.rados.0.smithi084.stdout:1592: oid 6 version 316 is already newer than 314 2024-09-20T22:48:12.563 INFO:tasks.rados.rados.0.smithi084.stdout:1592: oid 6 updating version 316 to 317 2024-09-20T22:48:12.563 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 6 v 317 (ObjNum 562 snap 169 seq_num 562) dirty exists 2024-09-20T22:48:12.564 INFO:tasks.rados.rados.0.smithi084.stdout:1592: left oid 6 (ObjNum 562 snap 169 seq_num 562) 2024-09-20T22:48:12.581 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:12.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:12.582 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:48:12.683 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:48:12.704 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:12.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:12.705 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:48:12.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:12 smithi154 ceph-mon[33264]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:48:12.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:12 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/1869011604' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T22:48:12.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:12 smithi154 ceph-mon[33264]: osdmap e411: 8 total, 8 up, 8 in 2024-09-20T22:48:12.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:12 smithi154 ceph-mon[33264]: from='client.25730 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T22:48:12.806 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:48:12.827 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:12.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:12.828 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:48:12.928 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:48:12.951 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:12.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:12.952 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:48:13.053 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:48:13.077 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:13.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:13.078 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:48:13.132 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-20T22:48:13.179 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:48:13.202 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:13.203 INFO:tasks.rados.rados.0.smithi084.stdout:1590: done (2 left) 2024-09-20T22:48:13.203 INFO:tasks.rados.rados.0.smithi084.stdout:1592: done (1 left) 2024-09-20T22:48:13.203 INFO:tasks.rados.rados.0.smithi084.stdout:1593: done (0 left) 2024-09-20T22:48:13.203 INFO:tasks.rados.rados.0.smithi084.stdout:1594: snap_remove snap 162 2024-09-20T22:48:13.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:13.204 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:48:13.304 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:48:13.326 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:13.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:13.327 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:48:13.364 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:13 smithi084 ceph-mon[40024]: pgmap v519: 168 pgs: 168 active+clean; 321 MiB data, 1018 MiB used, 714 GiB / 715 GiB avail; 541 KiB/s rd, 4.0 MiB/s wr, 8 op/s; 11 MiB/s, 34 objects/s recovering 2024-09-20T22:48:13.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:13 smithi084 ceph-mon[40024]: osdmap e412: 8 total, 8 up, 8 in 2024-09-20T22:48:13.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:13 smithi084 ceph-mon[40024]: osdmap e413: 8 total, 8 up, 8 in 2024-09-20T22:48:13.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:13 smithi084 ceph-mon[29774]: pgmap v519: 168 pgs: 168 active+clean; 321 MiB data, 1018 MiB used, 714 GiB / 715 GiB avail; 541 KiB/s rd, 4.0 MiB/s wr, 8 op/s; 11 MiB/s, 34 objects/s recovering 2024-09-20T22:48:13.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:13 smithi084 ceph-mon[29774]: osdmap e412: 8 total, 8 up, 8 in 2024-09-20T22:48:13.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:13 smithi084 ceph-mon[29774]: osdmap e413: 8 total, 8 up, 8 in 2024-09-20T22:48:13.398 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:48:13.427 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:48:13.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:13.450 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:13.450 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:48:13.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:13 smithi154 ceph-mon[33264]: pgmap v519: 168 pgs: 168 active+clean; 321 MiB data, 1018 MiB used, 714 GiB / 715 GiB avail; 541 KiB/s rd, 4.0 MiB/s wr, 8 op/s; 11 MiB/s, 34 objects/s recovering 2024-09-20T22:48:13.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:13 smithi154 ceph-mon[33264]: osdmap e412: 8 total, 8 up, 8 in 2024-09-20T22:48:13.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:13 smithi154 ceph-mon[33264]: osdmap e413: 8 total, 8 up, 8 in 2024-09-20T22:48:13.551 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:48:13.572 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:13.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:13.573 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:48:13.674 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:48:13.696 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:13.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:13.697 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:48:13.798 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:48:13.820 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:13.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:13.821 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:48:13.858 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:48:13.868 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-20T22:48:13.868 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:48:13.885 INFO:teuthology.orchestra.run.smithi084.stdout:{"pg_ready":true,"pg_map":{"version":518,"stamp":"2024-09-20T22:48:11.294865+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":336665204,"num_objects":369,"num_object_clones":122,"num_object_copies":1107,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":369,"num_whiteouts":13,"num_read":3844,"num_read_kb":2872116,"num_write":2294,"num_write_kb":704553,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":756,"num_bytes_recovered":237995937,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6739,"ondisk_log_size":6739,"up":504,"acting":504,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":515,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":1042348,"kb_used_data":801124,"kb_used_omap":12,"kb_used_meta":241011,"kb_avail":748918868,"statfs":{"total":767960285184,"available":766892920832,"internally_reserved":0,"allocated":820350976,"data_stored":1136565610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":246795824},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":16947133,"num_objects":8,"num_object_clones":7,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":1,"num_read":16,"num_read_kb":2164,"num_write":20,"num_write_kb":16556,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":137,"num_bytes_recovered":47185692,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.081248"},"pg_stats":[{"pgid":"3.1f","version":"86'2","reported_seq":53,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.198292+0000","last_change":"2024-09-20T22:47:49.237762+0000","last_active":"2024-09-20T22:48:08.198292+0000","last_peered":"2024-09-20T22:48:08.198292+0000","last_clean":"2024-09-20T22:48:08.198292+0000","last_became_active":"2024-09-20T22:47:49.222114+0000","last_became_peered":"2024-09-20T22:47:49.222114+0000","last_unstale":"2024-09-20T22:48:08.198292+0000","last_undegraded":"2024-09-20T22:48:08.198292+0000","last_fullsized":"2024-09-20T22:48:08.198292+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'2","last_scrub_stamp":"2024-09-20T22:46:42.206600+0000","last_deep_scrub":"86'2","last_deep_scrub_stamp":"2024-09-20T22:42:39.664421+0000","last_clean_scrub_stamp":"2024-09-20T22:46:42.206600+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:11.508639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":436,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"74'9","reported_seq":27,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.198827+0000","last_change":"2024-09-20T22:48:05.283966+0000","last_active":"2024-09-20T22:48:08.198827+0000","last_peered":"2024-09-20T22:48:08.198827+0000","last_clean":"2024-09-20T22:48:08.198827+0000","last_became_active":"2024-09-20T22:48:05.248980+0000","last_became_peered":"2024-09-20T22:48:05.248980+0000","last_unstale":"2024-09-20T22:48:08.198827+0000","last_undegraded":"2024-09-20T22:48:08.198827+0000","last_fullsized":"2024-09-20T22:48:08.198827+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-09-20T22:46:20.889510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:46:20.889510+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:12.869765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":702,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.202934+0000","last_change":"2024-09-20T22:46:22.955529+0000","last_active":"2024-09-20T22:48:08.202934+0000","last_peered":"2024-09-20T22:48:08.202934+0000","last_clean":"2024-09-20T22:48:08.202934+0000","last_became_active":"2024-09-20T22:41:42.182024+0000","last_became_peered":"2024-09-20T22:41:42.182024+0000","last_unstale":"2024-09-20T22:48:08.202934+0000","last_undegraded":"2024-09-20T22:48:08.202934+0000","last_fullsized":"2024-09-20T22:48:08.202934+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:46:22.955460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:46:22.955460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":17,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.199333+0000","last_change":"2024-09-20T22:48:05.248713+0000","last_active":"2024-09-20T22:48:08.199333+0000","last_peered":"2024-09-20T22:48:08.199333+0000","last_clean":"2024-09-20T22:48:08.199333+0000","last_became_active":"2024-09-20T22:48:05.247569+0000","last_became_peered":"2024-09-20T22:48:05.247569+0000","last_unstale":"2024-09-20T22:48:08.199333+0000","last_undegraded":"2024-09-20T22:48:08.199333+0000","last_fullsized":"2024-09-20T22:48:08.199333+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:46:45.177105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:46:45.177105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:12.869749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":698,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.205476+0000","last_change":"2024-09-20T22:45:40.053329+0000","last_active":"2024-09-20T22:48:08.205476+0000","last_peered":"2024-09-20T22:48:08.205476+0000","last_clean":"2024-09-20T22:48:08.205476+0000","last_became_active":"2024-09-20T22:41:44.832099+0000","last_became_peered":"2024-09-20T22:41:44.832099+0000","last_unstale":"2024-09-20T22:48:08.205476+0000","last_undegraded":"2024-09-20T22:48:08.205476+0000","last_fullsized":"2024-09-20T22:48:08.205476+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:45:40.053263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:45:40.053263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":703,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.207239+0000","last_change":"2024-09-20T22:44:58.994410+0000","last_active":"2024-09-20T22:48:08.207239+0000","last_peered":"2024-09-20T22:48:08.207239+0000","last_clean":"2024-09-20T22:48:08.207239+0000","last_became_active":"2024-09-20T22:41:38.157298+0000","last_became_peered":"2024-09-20T22:41:38.157298+0000","last_unstale":"2024-09-20T22:48:08.207239+0000","last_undegraded":"2024-09-20T22:48:08.207239+0000","last_fullsized":"2024-09-20T22:48:08.207239+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:44:58.994340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:44:58.994340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"74'15","reported_seq":767,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.206684+0000","last_change":"2024-09-20T22:47:49.246946+0000","last_active":"2024-09-20T22:48:08.206684+0000","last_peered":"2024-09-20T22:48:08.206684+0000","last_clean":"2024-09-20T22:48:08.206684+0000","last_became_active":"2024-09-20T22:47:49.218798+0000","last_became_peered":"2024-09-20T22:47:49.218798+0000","last_unstale":"2024-09-20T22:48:08.206684+0000","last_undegraded":"2024-09-20T22:48:08.206684+0000","last_fullsized":"2024-09-20T22:48:08.206684+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-09-20T22:44:41.108205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:44:41.108205+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":17,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.198686+0000","last_change":"2024-09-20T22:48:05.247938+0000","last_active":"2024-09-20T22:48:08.198686+0000","last_peered":"2024-09-20T22:48:08.198686+0000","last_clean":"2024-09-20T22:48:08.198686+0000","last_became_active":"2024-09-20T22:48:05.247636+0000","last_became_peered":"2024-09-20T22:48:05.247636+0000","last_unstale":"2024-09-20T22:48:08.198686+0000","last_undegraded":"2024-09-20T22:48:08.198686+0000","last_fullsized":"2024-09-20T22:48:08.198686+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:47:19.805701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:47:19.805701+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:23.203738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.19","version":"407'258","reported_seq":1582,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.204245+0000","last_change":"2024-09-20T22:48:04.230452+0000","last_active":"2024-09-20T22:48:08.204245+0000","last_peered":"2024-09-20T22:48:08.204245+0000","last_clean":"2024-09-20T22:48:08.204245+0000","last_became_active":"2024-09-20T22:43:39.130950+0000","last_became_peered":"2024-09-20T22:43:39.130950+0000","last_unstale":"2024-09-20T22:48:08.204245+0000","last_undegraded":"2024-09-20T22:48:08.204245+0000","last_fullsized":"2024-09-20T22:48:08.204245+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":5,"last_scrub":"395'255","last_scrub_stamp":"2024-09-20T22:47:57.056233+0000","last_deep_scrub":"241'147","last_deep_scrub_stamp":"2024-09-20T22:45:23.839574+0000","last_clean_scrub_stamp":"2024-09-20T22:47:57.056233+0000","objects_scrubbed":7,"log_size":258,"log_dups_size":0,"ondisk_log_size":258,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:49:25.126717+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0044246859999999997,"stat_sum":{"num_bytes":20697565,"num_objects":8,"num_object_clones":5,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":97,"num_read_kb":96094,"num_write":81,"num_write_kb":34489,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"98","length":"1"},{"start":"a1","length":"2"}]},{"pgid":"3.1d","version":"0'0","reported_seq":723,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.242340+0000","last_change":"2024-09-20T22:48:05.242386+0000","last_active":"2024-09-20T22:48:08.242340+0000","last_peered":"2024-09-20T22:48:08.242340+0000","last_clean":"2024-09-20T22:48:08.242340+0000","last_became_active":"2024-09-20T22:48:05.242059+0000","last_became_peered":"2024-09-20T22:48:05.242059+0000","last_unstale":"2024-09-20T22:48:08.242340+0000","last_undegraded":"2024-09-20T22:48:08.242340+0000","last_fullsized":"2024-09-20T22:48:08.242340+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:46:00.999939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:46:00.999939+0000","last_clean_scrub_stamp":"2024-09-20T22:46:00.999939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:11.497598+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"74'9","reported_seq":27,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.199420+0000","last_change":"2024-09-20T22:48:05.259778+0000","last_active":"2024-09-20T22:48:08.199420+0000","last_peered":"2024-09-20T22:48:08.199420+0000","last_clean":"2024-09-20T22:48:08.199420+0000","last_became_active":"2024-09-20T22:48:05.233429+0000","last_became_peered":"2024-09-20T22:48:05.233429+0000","last_unstale":"2024-09-20T22:48:08.199420+0000","last_undegraded":"2024-09-20T22:48:08.199420+0000","last_fullsized":"2024-09-20T22:48:08.199420+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-09-20T22:46:45.095957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:46:45.095957+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:12.869794+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":715,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.242289+0000","last_change":"2024-09-20T22:47:49.228490+0000","last_active":"2024-09-20T22:48:08.242289+0000","last_peered":"2024-09-20T22:48:08.242289+0000","last_clean":"2024-09-20T22:48:08.242289+0000","last_became_active":"2024-09-20T22:47:49.227379+0000","last_became_peered":"2024-09-20T22:47:49.227379+0000","last_unstale":"2024-09-20T22:48:08.242289+0000","last_undegraded":"2024-09-20T22:48:08.242289+0000","last_fullsized":"2024-09-20T22:48:08.242289+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:46:52.239508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:46:52.239508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:20.193806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":49,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.196489+0000","last_change":"2024-09-20T22:47:49.225833+0000","last_active":"2024-09-20T22:48:08.196489+0000","last_peered":"2024-09-20T22:48:08.196489+0000","last_clean":"2024-09-20T22:48:08.196489+0000","last_became_active":"2024-09-20T22:47:49.225515+0000","last_became_peered":"2024-09-20T22:47:49.225515+0000","last_unstale":"2024-09-20T22:48:08.196489+0000","last_undegraded":"2024-09-20T22:48:08.196489+0000","last_fullsized":"2024-09-20T22:48:08.196489+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:46:29.801854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:46:29.801854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:11.508596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.18","version":"406'236","reported_seq":1545,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.196745+0000","last_change":"2024-09-20T22:48:04.217398+0000","last_active":"2024-09-20T22:48:08.196745+0000","last_peered":"2024-09-20T22:48:08.196745+0000","last_clean":"2024-09-20T22:48:08.196745+0000","last_became_active":"2024-09-20T22:47:52.206113+0000","last_became_peered":"2024-09-20T22:47:52.206113+0000","last_unstale":"2024-09-20T22:48:08.196745+0000","last_undegraded":"2024-09-20T22:48:08.196745+0000","last_fullsized":"2024-09-20T22:48:08.196745+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":5,"last_scrub":"282'128","last_scrub_stamp":"2024-09-20T22:46:01.882247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:46:01.882247+0000","objects_scrubbed":1,"log_size":236,"log_dups_size":0,"ondisk_log_size":236,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.010514107999999999,"stat_sum":{"num_bytes":6396308,"num_objects":6,"num_object_clones":4,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":2,"num_read":93,"num_read_kb":94824,"num_write":65,"num_write_kb":23540,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":13188600,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"98","length":"1"},{"start":"a1","length":"2"}]},{"pgid":"3.1c","version":"0'0","reported_seq":723,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.241856+0000","last_change":"2024-09-20T22:48:05.242381+0000","last_active":"2024-09-20T22:48:08.241856+0000","last_peered":"2024-09-20T22:48:08.241856+0000","last_clean":"2024-09-20T22:48:08.241856+0000","last_became_active":"2024-09-20T22:48:05.242084+0000","last_became_peered":"2024-09-20T22:48:05.242084+0000","last_unstale":"2024-09-20T22:48:08.241856+0000","last_undegraded":"2024-09-20T22:48:08.241856+0000","last_fullsized":"2024-09-20T22:48:08.241856+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:45:53.933937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:45:53.933937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:11.497548+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"181'7","reported_seq":23,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.199901+0000","last_change":"2024-09-20T22:48:05.287983+0000","last_active":"2024-09-20T22:48:08.199901+0000","last_peered":"2024-09-20T22:48:08.199901+0000","last_clean":"2024-09-20T22:48:08.199901+0000","last_became_active":"2024-09-20T22:48:05.249763+0000","last_became_peered":"2024-09-20T22:48:05.249763+0000","last_unstale":"2024-09-20T22:48:08.199901+0000","last_undegraded":"2024-09-20T22:48:08.199901+0000","last_fullsized":"2024-09-20T22:48:08.199901+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"181'7","last_scrub_stamp":"2024-09-20T22:45:36.106880+0000","last_deep_scrub":"181'7","last_deep_scrub_stamp":"2024-09-20T22:44:20.259598+0000","last_clean_scrub_stamp":"2024-09-20T22:45:36.106880+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:12.869675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":6,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":737,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.200473+0000","last_change":"2024-09-20T22:48:05.238275+0000","last_active":"2024-09-20T22:48:08.200473+0000","last_peered":"2024-09-20T22:48:08.200473+0000","last_clean":"2024-09-20T22:48:08.200473+0000","last_became_active":"2024-09-20T22:48:05.237791+0000","last_became_peered":"2024-09-20T22:48:05.237791+0000","last_unstale":"2024-09-20T22:48:08.200473+0000","last_undegraded":"2024-09-20T22:48:08.200473+0000","last_fullsized":"2024-09-20T22:48:08.200473+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:47:40.361958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:47:40.361958+0000","last_clean_scrub_stamp":"2024-09-20T22:47:40.361958+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:56.944095+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":699,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.240251+0000","last_change":"2024-09-20T22:46:06.068842+0000","last_active":"2024-09-20T22:48:08.240251+0000","last_peered":"2024-09-20T22:48:08.240251+0000","last_clean":"2024-09-20T22:48:08.240251+0000","last_became_active":"2024-09-20T22:41:44.197927+0000","last_became_peered":"2024-09-20T22:41:44.197927+0000","last_unstale":"2024-09-20T22:48:08.240251+0000","last_undegraded":"2024-09-20T22:48:08.240251+0000","last_fullsized":"2024-09-20T22:48:08.240251+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:46:06.068745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:46:06.068745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:11.497611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":17,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.201540+0000","last_change":"2024-09-20T22:48:05.244188+0000","last_active":"2024-09-20T22:48:08.201540+0000","last_peered":"2024-09-20T22:48:08.201540+0000","last_clean":"2024-09-20T22:48:08.201540+0000","last_became_active":"2024-09-20T22:48:05.243877+0000","last_became_peered":"2024-09-20T22:48:05.243877+0000","last_unstale":"2024-09-20T22:48:08.201540+0000","last_undegraded":"2024-09-20T22:48:08.201540+0000","last_fullsized":"2024-09-20T22:48:08.201540+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:47:21.819572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:47:21.819572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:48.818606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":55,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.198762+0000","last_change":"2024-09-20T22:48:05.234091+0000","last_active":"2024-09-20T22:48:08.198762+0000","last_peered":"2024-09-20T22:48:08.198762+0000","last_clean":"2024-09-20T22:48:08.198762+0000","last_became_active":"2024-09-20T22:48:05.233687+0000","last_became_peered":"2024-09-20T22:48:05.233687+0000","last_unstale":"2024-09-20T22:48:08.198762+0000","last_undegraded":"2024-09-20T22:48:08.198762+0000","last_fullsized":"2024-09-20T22:48:08.198762+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:47:13.413912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:47:13.413912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:13.625333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"113'22","reported_seq":746,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.198519+0000","last_change":"2024-09-20T22:46:25.731055+0000","last_active":"2024-09-20T22:48:08.198519+0000","last_peered":"2024-09-20T22:48:08.198519+0000","last_clean":"2024-09-20T22:48:08.198519+0000","last_became_active":"2024-09-20T22:41:40.170131+0000","last_became_peered":"2024-09-20T22:41:40.170131+0000","last_unstale":"2024-09-20T22:48:08.198519+0000","last_undegraded":"2024-09-20T22:48:08.198519+0000","last_fullsized":"2024-09-20T22:48:08.198519+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'22","last_scrub_stamp":"2024-09-20T22:46:25.730972+0000","last_deep_scrub":"113'22","last_deep_scrub_stamp":"2024-09-20T22:43:12.191878+0000","last_clean_scrub_stamp":"2024-09-20T22:46:25.730972+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:13.135178+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":726,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.203091+0000","last_change":"2024-09-20T22:48:05.249765+0000","last_active":"2024-09-20T22:48:08.203091+0000","last_peered":"2024-09-20T22:48:08.203091+0000","last_clean":"2024-09-20T22:48:08.203091+0000","last_became_active":"2024-09-20T22:48:05.249417+0000","last_became_peered":"2024-09-20T22:48:05.249417+0000","last_unstale":"2024-09-20T22:48:08.203091+0000","last_undegraded":"2024-09-20T22:48:08.203091+0000","last_fullsized":"2024-09-20T22:48:08.203091+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:46:49.872479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:46:49.872479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":691,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.208046+0000","last_change":"2024-09-20T22:45:25.002270+0000","last_active":"2024-09-20T22:48:08.208046+0000","last_peered":"2024-09-20T22:48:08.208046+0000","last_clean":"2024-09-20T22:48:08.208046+0000","last_became_active":"2024-09-20T22:41:44.832131+0000","last_became_peered":"2024-09-20T22:41:44.832131+0000","last_unstale":"2024-09-20T22:48:08.208046+0000","last_undegraded":"2024-09-20T22:48:08.208046+0000","last_fullsized":"2024-09-20T22:48:08.208046+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:45:25.002199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:45:25.002199+0000","last_clean_scrub_stamp":"2024-09-20T22:45:25.002199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":17,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.199587+0000","last_change":"2024-09-20T22:48:05.251150+0000","last_active":"2024-09-20T22:48:08.199587+0000","last_peered":"2024-09-20T22:48:08.199587+0000","last_clean":"2024-09-20T22:48:08.199587+0000","last_became_active":"2024-09-20T22:48:05.250581+0000","last_became_peered":"2024-09-20T22:48:05.250581+0000","last_unstale":"2024-09-20T22:48:08.199587+0000","last_undegraded":"2024-09-20T22:48:08.199587+0000","last_fullsized":"2024-09-20T22:48:08.199587+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:45:15.043443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:45:15.043443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:12.869659+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"74'12","reported_seq":775,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.205866+0000","last_change":"2024-09-20T22:48:05.258994+0000","last_active":"2024-09-20T22:48:08.205866+0000","last_peered":"2024-09-20T22:48:08.205866+0000","last_clean":"2024-09-20T22:48:08.205866+0000","last_became_active":"2024-09-20T22:48:05.233983+0000","last_became_peered":"2024-09-20T22:48:05.233983+0000","last_unstale":"2024-09-20T22:48:08.205866+0000","last_undegraded":"2024-09-20T22:48:08.205866+0000","last_fullsized":"2024-09-20T22:48:08.205866+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-09-20T22:44:42.140915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:44:42.140915+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":17,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.199108+0000","last_change":"2024-09-20T22:48:05.250865+0000","last_active":"2024-09-20T22:48:08.199108+0000","last_peered":"2024-09-20T22:48:08.199108+0000","last_clean":"2024-09-20T22:48:08.199108+0000","last_became_active":"2024-09-20T22:48:05.250319+0000","last_became_peered":"2024-09-20T22:48:05.250319+0000","last_unstale":"2024-09-20T22:48:08.199108+0000","last_undegraded":"2024-09-20T22:48:08.199108+0000","last_fullsized":"2024-09-20T22:48:08.199108+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:44:14.235804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:44:14.235804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:12.869619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"154'2","reported_seq":700,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.201619+0000","last_change":"2024-09-20T22:46:21.446679+0000","last_active":"2024-09-20T22:48:08.201619+0000","last_peered":"2024-09-20T22:48:08.201619+0000","last_clean":"2024-09-20T22:48:08.201619+0000","last_became_active":"2024-09-20T22:41:44.199822+0000","last_became_peered":"2024-09-20T22:41:44.199822+0000","last_unstale":"2024-09-20T22:48:08.201619+0000","last_undegraded":"2024-09-20T22:48:08.201619+0000","last_fullsized":"2024-09-20T22:48:08.201619+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"154'2","last_scrub_stamp":"2024-09-20T22:46:21.446609+0000","last_deep_scrub":"154'2","last_deep_scrub_stamp":"2024-09-20T22:43:54.134066+0000","last_clean_scrub_stamp":"2024-09-20T22:46:21.446609+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:12.326133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"67'1","reported_seq":731,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.203093+0000","last_change":"2024-09-20T22:48:05.280350+0000","last_active":"2024-09-20T22:48:08.203093+0000","last_peered":"2024-09-20T22:48:08.203093+0000","last_clean":"2024-09-20T22:48:08.203093+0000","last_became_active":"2024-09-20T22:48:05.249578+0000","last_became_peered":"2024-09-20T22:48:05.249578+0000","last_unstale":"2024-09-20T22:48:08.203093+0000","last_undegraded":"2024-09-20T22:48:08.203093+0000","last_fullsized":"2024-09-20T22:48:08.203093+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-09-20T22:45:36.976689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:45:36.976689+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"151'15","reported_seq":61,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.199357+0000","last_change":"2024-09-20T22:47:49.273137+0000","last_active":"2024-09-20T22:48:08.199357+0000","last_peered":"2024-09-20T22:48:08.199357+0000","last_clean":"2024-09-20T22:48:08.199357+0000","last_became_active":"2024-09-20T22:47:49.232181+0000","last_became_peered":"2024-09-20T22:47:49.232181+0000","last_unstale":"2024-09-20T22:48:08.199357+0000","last_undegraded":"2024-09-20T22:48:08.199357+0000","last_fullsized":"2024-09-20T22:48:08.199357+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"151'15","last_scrub_stamp":"2024-09-20T22:45:58.439782+0000","last_deep_scrub":"151'15","last_deep_scrub_stamp":"2024-09-20T22:43:50.715761+0000","last_clean_scrub_stamp":"2024-09-20T22:45:58.439782+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:11.508519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"408'10","reported_seq":816,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.204942+0000","last_change":"2024-09-20T22:48:05.259617+0000","last_active":"2024-09-20T22:48:08.204942+0000","last_peered":"2024-09-20T22:48:08.204942+0000","last_clean":"2024-09-20T22:48:08.204942+0000","last_became_active":"2024-09-20T22:48:05.244270+0000","last_became_peered":"2024-09-20T22:48:05.244270+0000","last_unstale":"2024-09-20T22:48:08.204942+0000","last_undegraded":"2024-09-20T22:48:08.204942+0000","last_fullsized":"2024-09-20T22:48:08.204942+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"159'9","last_scrub_stamp":"2024-09-20T22:46:21.897701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:46:21.897701+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.c","version":"404'860","reported_seq":2451,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.205072+0000","last_change":"2024-09-20T22:48:04.238399+0000","last_active":"2024-09-20T22:48:08.205072+0000","last_peered":"2024-09-20T22:48:08.205072+0000","last_clean":"2024-09-20T22:48:08.205072+0000","last_became_active":"2024-09-20T22:42:52.404027+0000","last_became_peered":"2024-09-20T22:42:52.404027+0000","last_unstale":"2024-09-20T22:48:08.205072+0000","last_undegraded":"2024-09-20T22:48:08.205072+0000","last_fullsized":"2024-09-20T22:48:08.205072+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"332'667","last_scrub_stamp":"2024-09-20T22:46:52.845442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:46:52.845442+0000","objects_scrubbed":12,"log_size":860,"log_dups_size":0,"ondisk_log_size":860,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:18.144669+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.013545458,"stat_sum":{"num_bytes":41113807,"num_objects":19,"num_object_clones":13,"num_object_copies":57,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":510,"num_read_kb":506249,"num_write":334,"num_write_kb":131420,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"98","length":"1"},{"start":"a1","length":"2"}]},{"pgid":"1.a","version":"77'128","reported_seq":970,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.241441+0000","last_change":"2024-09-20T22:45:37.851690+0000","last_active":"2024-09-20T22:48:08.241441+0000","last_peered":"2024-09-20T22:48:08.241441+0000","last_clean":"2024-09-20T22:48:08.241441+0000","last_became_active":"2024-09-20T22:42:52.407791+0000","last_became_peered":"2024-09-20T22:42:52.407791+0000","last_unstale":"2024-09-20T22:48:08.241441+0000","last_undegraded":"2024-09-20T22:48:08.241441+0000","last_fullsized":"2024-09-20T22:48:08.241441+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:45:37.851582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:45:37.851582+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:11.497519+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"74'15","reported_seq":775,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.203856+0000","last_change":"2024-09-20T22:48:05.280673+0000","last_active":"2024-09-20T22:48:08.203856+0000","last_peered":"2024-09-20T22:48:08.203856+0000","last_clean":"2024-09-20T22:48:08.203856+0000","last_became_active":"2024-09-20T22:48:05.250308+0000","last_became_peered":"2024-09-20T22:48:05.250308+0000","last_unstale":"2024-09-20T22:48:08.203856+0000","last_undegraded":"2024-09-20T22:48:08.203856+0000","last_fullsized":"2024-09-20T22:48:08.203856+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-09-20T22:44:55.972333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:44:55.972333+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":702,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.205171+0000","last_change":"2024-09-20T22:45:19.027462+0000","last_active":"2024-09-20T22:48:08.205171+0000","last_peered":"2024-09-20T22:48:08.205171+0000","last_clean":"2024-09-20T22:48:08.205171+0000","last_became_active":"2024-09-20T22:41:38.156749+0000","last_became_peered":"2024-09-20T22:41:38.156749+0000","last_unstale":"2024-09-20T22:48:08.205171+0000","last_undegraded":"2024-09-20T22:48:08.205171+0000","last_fullsized":"2024-09-20T22:48:08.205171+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:45:19.027393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:45:19.027393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"408'12","reported_seq":22,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.276719+0000","last_change":"2024-09-20T22:48:05.251167+0000","last_active":"2024-09-20T22:48:08.276719+0000","last_peered":"2024-09-20T22:48:08.276719+0000","last_clean":"2024-09-20T22:48:08.276719+0000","last_became_active":"2024-09-20T22:48:05.242914+0000","last_became_peered":"2024-09-20T22:48:05.242914+0000","last_unstale":"2024-09-20T22:48:08.276719+0000","last_undegraded":"2024-09-20T22:48:08.276719+0000","last_fullsized":"2024-09-20T22:48:08.276719+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"264'10","last_scrub_stamp":"2024-09-20T22:47:20.300787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:47:20.300787+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:35.056072+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":711,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.241776+0000","last_change":"2024-09-20T22:47:49.227726+0000","last_active":"2024-09-20T22:48:08.241776+0000","last_peered":"2024-09-20T22:48:08.241776+0000","last_clean":"2024-09-20T22:48:08.241776+0000","last_became_active":"2024-09-20T22:47:49.227039+0000","last_became_peered":"2024-09-20T22:47:49.227039+0000","last_unstale":"2024-09-20T22:48:08.241776+0000","last_undegraded":"2024-09-20T22:48:08.241776+0000","last_fullsized":"2024-09-20T22:48:08.241776+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:46:26.147167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:46:26.147167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:11.497657+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3","version":"135'8","reported_seq":1550,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.205343+0000","last_change":"2024-09-20T22:48:05.235978+0000","last_active":"2024-09-20T22:48:08.205343+0000","last_peered":"2024-09-20T22:48:08.205343+0000","last_clean":"2024-09-20T22:48:08.205343+0000","last_became_active":"2024-09-20T22:48:05.234893+0000","last_became_peered":"2024-09-20T22:48:05.234893+0000","last_unstale":"2024-09-20T22:48:08.205343+0000","last_undegraded":"2024-09-20T22:48:08.205343+0000","last_fullsized":"2024-09-20T22:48:08.205343+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'8","last_scrub_stamp":"2024-09-20T22:47:23.137750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:47:23.137750+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:28.392054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065514900000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"98","length":"1"},{"start":"a1","length":"2"}]},{"pgid":"1.5","version":"77'128","reported_seq":875,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.209292+0000","last_change":"2024-09-20T22:48:08.207463+0000","last_active":"2024-09-20T22:48:08.209292+0000","last_peered":"2024-09-20T22:48:08.209292+0000","last_clean":"2024-09-20T22:48:08.209292+0000","last_became_active":"2024-09-20T22:48:08.207018+0000","last_became_peered":"2024-09-20T22:48:08.207018+0000","last_unstale":"2024-09-20T22:48:08.209292+0000","last_undegraded":"2024-09-20T22:48:08.209292+0000","last_fullsized":"2024-09-20T22:48:08.209292+0000","mapping_epoch":407,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":408,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:48:02.056211+0000","last_deep_scrub":"77'128","last_deep_scrub_stamp":"2024-09-20T22:45:41.608075+0000","last_clean_scrub_stamp":"2024-09-20T22:48:02.056211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:49:15.339453+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"74'18","reported_seq":779,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.205396+0000","last_change":"2024-09-20T22:47:49.265801+0000","last_active":"2024-09-20T22:48:08.205396+0000","last_peered":"2024-09-20T22:48:08.205396+0000","last_clean":"2024-09-20T22:48:08.205396+0000","last_became_active":"2024-09-20T22:47:49.217605+0000","last_became_peered":"2024-09-20T22:47:49.217605+0000","last_unstale":"2024-09-20T22:48:08.205396+0000","last_undegraded":"2024-09-20T22:48:08.205396+0000","last_fullsized":"2024-09-20T22:48:08.205396+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'18","last_scrub_stamp":"2024-09-20T22:44:23.220709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:44:23.220709+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":722,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.208561+0000","last_change":"2024-09-20T22:47:49.227431+0000","last_active":"2024-09-20T22:48:08.208561+0000","last_peered":"2024-09-20T22:48:08.208561+0000","last_clean":"2024-09-20T22:48:08.208561+0000","last_became_active":"2024-09-20T22:47:49.227069+0000","last_became_peered":"2024-09-20T22:47:49.227069+0000","last_unstale":"2024-09-20T22:48:08.208561+0000","last_undegraded":"2024-09-20T22:48:08.208561+0000","last_fullsized":"2024-09-20T22:48:08.208561+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:47:39.982616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:47:39.982616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:59.960690+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":17,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.201646+0000","last_change":"2024-09-20T22:48:05.231356+0000","last_active":"2024-09-20T22:48:08.201646+0000","last_peered":"2024-09-20T22:48:08.201646+0000","last_clean":"2024-09-20T22:48:08.201646+0000","last_became_active":"2024-09-20T22:48:05.230003+0000","last_became_peered":"2024-09-20T22:48:05.230003+0000","last_unstale":"2024-09-20T22:48:08.201646+0000","last_undegraded":"2024-09-20T22:48:08.201646+0000","last_fullsized":"2024-09-20T22:48:08.201646+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:45:56.199432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:07.947945+0000","last_clean_scrub_stamp":"2024-09-20T22:45:56.199432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:12.869720+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":17,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.199505+0000","last_change":"2024-09-20T22:48:05.240266+0000","last_active":"2024-09-20T22:48:08.199505+0000","last_peered":"2024-09-20T22:48:08.199505+0000","last_clean":"2024-09-20T22:48:08.199505+0000","last_became_active":"2024-09-20T22:48:05.239917+0000","last_became_peered":"2024-09-20T22:48:05.239917+0000","last_unstale":"2024-09-20T22:48:08.199505+0000","last_undegraded":"2024-09-20T22:48:08.199505+0000","last_fullsized":"2024-09-20T22:48:08.199505+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:47:18.179575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:53.036601+0000","last_clean_scrub_stamp":"2024-09-20T22:47:18.179575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:25.671647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2","version":"404'178","reported_seq":1296,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.241392+0000","last_change":"2024-09-20T22:48:06.236085+0000","last_active":"2024-09-20T22:48:08.241392+0000","last_peered":"2024-09-20T22:48:08.241392+0000","last_clean":"2024-09-20T22:48:08.241392+0000","last_became_active":"2024-09-20T22:44:09.177625+0000","last_became_peered":"2024-09-20T22:44:09.177625+0000","last_unstale":"2024-09-20T22:48:08.241392+0000","last_undegraded":"2024-09-20T22:48:08.241392+0000","last_fullsized":"2024-09-20T22:48:08.241392+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"282'120","last_scrub_stamp":"2024-09-20T22:46:22.099642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:46:22.099642+0000","objects_scrubbed":1,"log_size":178,"log_dups_size":0,"ondisk_log_size":178,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.017464948000000001,"stat_sum":{"num_bytes":11592813,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":72,"num_read_kb":83025,"num_write":35,"num_write_kb":10831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"98","length":"1"},{"start":"a1","length":"2"}]},{"pgid":"1.4","version":"77'128","reported_seq":1002,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.207714+0000","last_change":"2024-09-20T22:46:15.181632+0000","last_active":"2024-09-20T22:48:08.207714+0000","last_peered":"2024-09-20T22:48:08.207714+0000","last_clean":"2024-09-20T22:48:08.207714+0000","last_became_active":"2024-09-20T22:46:15.181305+0000","last_became_peered":"2024-09-20T22:46:15.181305+0000","last_unstale":"2024-09-20T22:48:08.207714+0000","last_undegraded":"2024-09-20T22:48:08.207714+0000","last_fullsized":"2024-09-20T22:48:08.207714+0000","mapping_epoch":94,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:45:54.131305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:45:54.131305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":59,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"74'14","reported_seq":33,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.201320+0000","last_change":"2024-09-20T22:48:05.262315+0000","last_active":"2024-09-20T22:48:08.201320+0000","last_peered":"2024-09-20T22:48:08.201320+0000","last_clean":"2024-09-20T22:48:08.201320+0000","last_became_active":"2024-09-20T22:48:05.235812+0000","last_became_peered":"2024-09-20T22:48:05.235812+0000","last_unstale":"2024-09-20T22:48:08.201320+0000","last_undegraded":"2024-09-20T22:48:08.201320+0000","last_fullsized":"2024-09-20T22:48:08.201320+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'14","last_scrub_stamp":"2024-09-20T22:46:48.202311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:46:48.202311+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:18.094672+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"67'1","reported_seq":63,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.199261+0000","last_change":"2024-09-20T22:48:05.252943+0000","last_active":"2024-09-20T22:48:08.199261+0000","last_peered":"2024-09-20T22:48:08.199261+0000","last_clean":"2024-09-20T22:48:08.199261+0000","last_became_active":"2024-09-20T22:48:05.237425+0000","last_became_peered":"2024-09-20T22:48:05.237425+0000","last_unstale":"2024-09-20T22:48:08.199261+0000","last_undegraded":"2024-09-20T22:48:08.199261+0000","last_fullsized":"2024-09-20T22:48:08.199261+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-09-20T22:45:25.781996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:45:25.781996+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:11.508505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"408'12","reported_seq":822,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.205668+0000","last_change":"2024-09-20T22:48:05.247422+0000","last_active":"2024-09-20T22:48:08.205668+0000","last_peered":"2024-09-20T22:48:08.205668+0000","last_clean":"2024-09-20T22:48:08.205668+0000","last_became_active":"2024-09-20T22:48:05.234607+0000","last_became_peered":"2024-09-20T22:48:05.234607+0000","last_unstale":"2024-09-20T22:48:08.205668+0000","last_undegraded":"2024-09-20T22:48:08.205668+0000","last_fullsized":"2024-09-20T22:48:08.205668+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"159'9","last_scrub_stamp":"2024-09-20T22:45:23.017605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:45:23.017605+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":699,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.200741+0000","last_change":"2024-09-20T22:47:16.381511+0000","last_active":"2024-09-20T22:48:08.200741+0000","last_peered":"2024-09-20T22:48:08.200741+0000","last_clean":"2024-09-20T22:48:08.200741+0000","last_became_active":"2024-09-20T22:41:44.831901+0000","last_became_peered":"2024-09-20T22:41:44.831901+0000","last_unstale":"2024-09-20T22:48:08.200741+0000","last_undegraded":"2024-09-20T22:48:08.200741+0000","last_fullsized":"2024-09-20T22:48:08.200741+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:47:16.381390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:47:16.381390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:23.066386+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1","version":"408'215","reported_seq":1705,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.365939+0000","last_change":"2024-09-20T22:48:08.210277+0000","last_active":"2024-09-20T22:48:08.365939+0000","last_peered":"2024-09-20T22:48:08.365939+0000","last_clean":"2024-09-20T22:48:08.365939+0000","last_became_active":"2024-09-20T22:48:08.208698+0000","last_became_peered":"2024-09-20T22:48:08.208698+0000","last_unstale":"2024-09-20T22:48:08.365939+0000","last_undegraded":"2024-09-20T22:48:08.365939+0000","last_fullsized":"2024-09-20T22:48:08.365939+0000","mapping_epoch":407,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":408,"parent":"0.0","parent_split_bits":0,"last_scrub":"170'72","last_scrub_stamp":"2024-09-20T22:44:12.228784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:44:12.228784+0000","objects_scrubbed":3,"log_size":215,"log_dups_size":0,"ondisk_log_size":215,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00053706200000000004,"stat_sum":{"num_bytes":8791993,"num_objects":6,"num_object_clones":4,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":2,"num_read":60,"num_read_kb":78458,"num_write":46,"num_write_kb":34476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":8732217,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"98","length":"1"},{"start":"a1","length":"2"}]},{"pgid":"1.7","version":"77'128","reported_seq":1007,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.204539+0000","last_change":"2024-09-20T22:48:03.022941+0000","last_active":"2024-09-20T22:48:08.204539+0000","last_peered":"2024-09-20T22:48:08.204539+0000","last_clean":"2024-09-20T22:48:08.204539+0000","last_became_active":"2024-09-20T22:46:03.169596+0000","last_became_peered":"2024-09-20T22:46:03.169596+0000","last_unstale":"2024-09-20T22:48:08.204539+0000","last_undegraded":"2024-09-20T22:48:08.204539+0000","last_fullsized":"2024-09-20T22:48:08.204539+0000","mapping_epoch":94,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:48:03.022838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:48:03.022838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:49:29.905235+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"74'10","reported_seq":794,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.204052+0000","last_change":"2024-09-20T22:48:05.277360+0000","last_active":"2024-09-20T22:48:08.204052+0000","last_peered":"2024-09-20T22:48:08.204052+0000","last_clean":"2024-09-20T22:48:08.204052+0000","last_became_active":"2024-09-20T22:48:05.251200+0000","last_became_peered":"2024-09-20T22:48:05.251200+0000","last_unstale":"2024-09-20T22:48:08.204052+0000","last_undegraded":"2024-09-20T22:48:08.204052+0000","last_fullsized":"2024-09-20T22:48:08.204052+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-09-20T22:44:59.049607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:44:59.049607+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":720,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.242409+0000","last_change":"2024-09-20T22:48:04.520131+0000","last_active":"2024-09-20T22:48:08.242409+0000","last_peered":"2024-09-20T22:48:08.242409+0000","last_clean":"2024-09-20T22:48:08.242409+0000","last_became_active":"2024-09-20T22:41:38.155629+0000","last_became_peered":"2024-09-20T22:41:38.155629+0000","last_unstale":"2024-09-20T22:48:08.242409+0000","last_undegraded":"2024-09-20T22:48:08.242409+0000","last_fullsized":"2024-09-20T22:48:08.242409+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:48:04.520053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:48:04.520053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:49:32.650911+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"389'11","reported_seq":58,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.198736+0000","last_change":"2024-09-20T22:47:49.246973+0000","last_active":"2024-09-20T22:48:08.198736+0000","last_peered":"2024-09-20T22:48:08.198736+0000","last_clean":"2024-09-20T22:48:08.198736+0000","last_became_active":"2024-09-20T22:47:49.227264+0000","last_became_peered":"2024-09-20T22:47:49.227264+0000","last_unstale":"2024-09-20T22:48:08.198736+0000","last_undegraded":"2024-09-20T22:48:08.198736+0000","last_fullsized":"2024-09-20T22:48:08.198736+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"264'10","last_scrub_stamp":"2024-09-20T22:45:57.889636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:45:57.889636+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:11.508569+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":49,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.196329+0000","last_change":"2024-09-20T22:47:49.232739+0000","last_active":"2024-09-20T22:48:08.196329+0000","last_peered":"2024-09-20T22:48:08.196329+0000","last_clean":"2024-09-20T22:48:08.196329+0000","last_became_active":"2024-09-20T22:47:49.232380+0000","last_became_peered":"2024-09-20T22:47:49.232380+0000","last_unstale":"2024-09-20T22:48:08.196329+0000","last_undegraded":"2024-09-20T22:48:08.196329+0000","last_fullsized":"2024-09-20T22:48:08.196329+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:46:44.117761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:46:44.117761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:11.508652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.0","version":"400'171","reported_seq":1745,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.240917+0000","last_change":"2024-09-20T22:48:04.211401+0000","last_active":"2024-09-20T22:48:08.240917+0000","last_peered":"2024-09-20T22:48:08.240917+0000","last_clean":"2024-09-20T22:48:08.240917+0000","last_became_active":"2024-09-20T22:43:37.136845+0000","last_became_peered":"2024-09-20T22:43:37.136845+0000","last_unstale":"2024-09-20T22:48:08.240917+0000","last_undegraded":"2024-09-20T22:48:08.240917+0000","last_fullsized":"2024-09-20T22:48:08.240917+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"280'122","last_scrub_stamp":"2024-09-20T22:46:03.006423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:46:03.006423+0000","objects_scrubbed":1,"log_size":171,"log_dups_size":0,"ondisk_log_size":171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00067178500000000005,"stat_sum":{"num_bytes":10109221,"num_objects":5,"num_object_clones":3,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":70,"num_read_kb":74240,"num_write":44,"num_write_kb":10847,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"98","length":"1"},{"start":"a1","length":"2"}]},{"pgid":"1.6","version":"77'128","reported_seq":1029,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.204240+0000","last_change":"2024-09-20T22:48:02.056439+0000","last_active":"2024-09-20T22:48:08.204240+0000","last_peered":"2024-09-20T22:48:08.204240+0000","last_clean":"2024-09-20T22:48:08.204240+0000","last_became_active":"2024-09-20T22:47:52.212190+0000","last_became_peered":"2024-09-20T22:47:52.212190+0000","last_unstale":"2024-09-20T22:48:08.204240+0000","last_undegraded":"2024-09-20T22:48:08.204240+0000","last_fullsized":"2024-09-20T22:48:08.204240+0000","mapping_epoch":391,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:48:02.056333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:48:02.056333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:49:14.033978+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"74'19","reported_seq":67,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.198570+0000","last_change":"2024-09-20T22:47:49.249445+0000","last_active":"2024-09-20T22:48:08.198570+0000","last_peered":"2024-09-20T22:48:08.198570+0000","last_clean":"2024-09-20T22:48:08.198570+0000","last_became_active":"2024-09-20T22:47:49.222009+0000","last_became_peered":"2024-09-20T22:47:49.222009+0000","last_unstale":"2024-09-20T22:48:08.198570+0000","last_undegraded":"2024-09-20T22:48:08.198570+0000","last_fullsized":"2024-09-20T22:48:08.198570+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'19","last_scrub_stamp":"2024-09-20T22:47:13.403224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:47:13.403224+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:25.688374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":38,"num_read_kb":24,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":711,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.202622+0000","last_change":"2024-09-20T22:45:21.038355+0000","last_active":"2024-09-20T22:48:08.202622+0000","last_peered":"2024-09-20T22:48:08.202622+0000","last_clean":"2024-09-20T22:48:08.202622+0000","last_became_active":"2024-09-20T22:41:38.160832+0000","last_became_peered":"2024-09-20T22:41:38.160832+0000","last_unstale":"2024-09-20T22:48:08.202622+0000","last_undegraded":"2024-09-20T22:48:08.202622+0000","last_fullsized":"2024-09-20T22:48:08.202622+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:45:21.038284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:45:21.038284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":715,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.204494+0000","last_change":"2024-09-20T22:47:49.214380+0000","last_active":"2024-09-20T22:48:08.204494+0000","last_peered":"2024-09-20T22:48:08.204494+0000","last_clean":"2024-09-20T22:48:08.204494+0000","last_became_active":"2024-09-20T22:47:49.214055+0000","last_became_peered":"2024-09-20T22:47:49.214055+0000","last_unstale":"2024-09-20T22:48:08.204494+0000","last_undegraded":"2024-09-20T22:48:08.204494+0000","last_fullsized":"2024-09-20T22:48:08.204494+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:46:43.827680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:46:43.827680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:08.907647+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":699,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.202840+0000","last_change":"2024-09-20T22:45:39.938227+0000","last_active":"2024-09-20T22:48:08.202840+0000","last_peered":"2024-09-20T22:48:08.202840+0000","last_clean":"2024-09-20T22:48:08.202840+0000","last_became_active":"2024-09-20T22:41:44.830538+0000","last_became_peered":"2024-09-20T22:41:44.830538+0000","last_unstale":"2024-09-20T22:48:08.202840+0000","last_undegraded":"2024-09-20T22:48:08.202840+0000","last_fullsized":"2024-09-20T22:48:08.202840+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:45:39.938119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:45:39.938119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.7","version":"404'369","reported_seq":2061,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.217797+0000","last_change":"2024-09-20T22:48:04.226711+0000","last_active":"2024-09-20T22:48:08.217797+0000","last_peered":"2024-09-20T22:48:08.217797+0000","last_clean":"2024-09-20T22:48:08.217797+0000","last_became_active":"2024-09-20T22:47:52.214221+0000","last_became_peered":"2024-09-20T22:47:52.214221+0000","last_unstale":"2024-09-20T22:48:08.217797+0000","last_undegraded":"2024-09-20T22:48:08.217797+0000","last_fullsized":"2024-09-20T22:48:08.217797+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"372'324","last_scrub_stamp":"2024-09-20T22:47:33.829464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:47:33.829464+0000","objects_scrubbed":7,"log_size":369,"log_dups_size":0,"ondisk_log_size":369,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:50.975445+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0061896540000000002,"stat_sum":{"num_bytes":22787502,"num_objects":10,"num_object_clones":7,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":144,"num_read_kb":151503,"num_write":105,"num_write_kb":38406,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":24551664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"98","length":"1"},{"start":"a1","length":"2"}]},{"pgid":"1.1","version":"77'128","reported_seq":1018,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.240382+0000","last_change":"2024-09-20T22:47:52.208511+0000","last_active":"2024-09-20T22:48:08.240382+0000","last_peered":"2024-09-20T22:48:08.240382+0000","last_clean":"2024-09-20T22:48:08.240382+0000","last_became_active":"2024-09-20T22:47:52.208102+0000","last_became_peered":"2024-09-20T22:47:52.208102+0000","last_unstale":"2024-09-20T22:48:08.240382+0000","last_undegraded":"2024-09-20T22:48:08.240382+0000","last_fullsized":"2024-09-20T22:48:08.240382+0000","mapping_epoch":391,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:45:58.908433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:45:58.908433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:11.497563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"86'38","reported_seq":852,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.202531+0000","last_change":"2024-09-20T22:48:00.380100+0000","last_active":"2024-09-20T22:48:08.202531+0000","last_peered":"2024-09-20T22:48:08.202531+0000","last_clean":"2024-09-20T22:48:08.202531+0000","last_became_active":"2024-09-20T22:41:40.166958+0000","last_became_peered":"2024-09-20T22:41:40.166958+0000","last_unstale":"2024-09-20T22:48:08.202531+0000","last_undegraded":"2024-09-20T22:48:08.202531+0000","last_fullsized":"2024-09-20T22:48:08.202531+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'38","last_scrub_stamp":"2024-09-20T22:48:00.379987+0000","last_deep_scrub":"86'38","last_deep_scrub_stamp":"2024-09-20T22:42:43.935677+0000","last_clean_scrub_stamp":"2024-09-20T22:48:00.379987+0000","objects_scrubbed":10,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:49:30.129855+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":17,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.199806+0000","last_change":"2024-09-20T22:48:05.238748+0000","last_active":"2024-09-20T22:48:08.199806+0000","last_peered":"2024-09-20T22:48:08.199806+0000","last_clean":"2024-09-20T22:48:08.199806+0000","last_became_active":"2024-09-20T22:48:05.237756+0000","last_became_peered":"2024-09-20T22:48:05.237756+0000","last_unstale":"2024-09-20T22:48:08.199806+0000","last_undegraded":"2024-09-20T22:48:08.199806+0000","last_fullsized":"2024-09-20T22:48:08.199806+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:47:16.817468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:47:16.817468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:41.426857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"69'2","reported_seq":730,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.241773+0000","last_change":"2024-09-20T22:46:44.210668+0000","last_active":"2024-09-20T22:48:08.241773+0000","last_peered":"2024-09-20T22:48:08.241773+0000","last_clean":"2024-09-20T22:48:08.241773+0000","last_became_active":"2024-09-20T22:41:37.148479+0000","last_became_peered":"2024-09-20T22:41:37.148479+0000","last_unstale":"2024-09-20T22:48:08.241773+0000","last_undegraded":"2024-09-20T22:48:08.241773+0000","last_fullsized":"2024-09-20T22:48:08.241773+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2024-09-20T22:46:44.210559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:36.130684+0000","last_clean_scrub_stamp":"2024-09-20T22:46:44.210559+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:09.950285+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00087280599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":55,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.198270+0000","last_change":"2024-09-20T22:48:05.236164+0000","last_active":"2024-09-20T22:48:08.198270+0000","last_peered":"2024-09-20T22:48:08.198270+0000","last_clean":"2024-09-20T22:48:08.198270+0000","last_became_active":"2024-09-20T22:48:05.233519+0000","last_became_peered":"2024-09-20T22:48:05.233519+0000","last_unstale":"2024-09-20T22:48:08.198270+0000","last_undegraded":"2024-09-20T22:48:08.198270+0000","last_fullsized":"2024-09-20T22:48:08.198270+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:46:06.104786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:46:06.104786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:11.508583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":710,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.206984+0000","last_change":"2024-09-20T22:48:05.234499+0000","last_active":"2024-09-20T22:48:08.206984+0000","last_peered":"2024-09-20T22:48:08.206984+0000","last_clean":"2024-09-20T22:48:08.206984+0000","last_became_active":"2024-09-20T22:48:05.234193+0000","last_became_peered":"2024-09-20T22:48:05.234193+0000","last_unstale":"2024-09-20T22:48:08.206984+0000","last_undegraded":"2024-09-20T22:48:08.206984+0000","last_fullsized":"2024-09-20T22:48:08.206984+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:45:58.212988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:45:58.212988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"400'86","reported_seq":138,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.197488+0000","last_change":"2024-09-20T22:48:05.293279+0000","last_active":"2024-09-20T22:48:08.197488+0000","last_peered":"2024-09-20T22:48:08.197488+0000","last_clean":"2024-09-20T22:48:08.197488+0000","last_became_active":"2024-09-20T22:48:05.242705+0000","last_became_peered":"2024-09-20T22:48:05.242705+0000","last_unstale":"2024-09-20T22:48:08.197488+0000","last_undegraded":"2024-09-20T22:48:08.197488+0000","last_fullsized":"2024-09-20T22:48:08.197488+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"305'57","last_scrub_stamp":"2024-09-20T22:47:13.636468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:47:13.636468+0000","objects_scrubbed":1,"log_size":86,"log_dups_size":0,"ondisk_log_size":86,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:35.691465+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00041129399999999998,"stat_sum":{"num_bytes":8284229,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":33,"num_read_kb":31742,"num_write":21,"num_write_kb":4716,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":10294275,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"98","length":"1"},{"start":"a1","length":"2"}]},{"pgid":"1.2","version":"77'128","reported_seq":1087,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.203458+0000","last_change":"2024-09-20T22:48:08.203458+0000","last_active":"2024-09-20T22:48:08.203458+0000","last_peered":"2024-09-20T22:48:08.203458+0000","last_clean":"2024-09-20T22:48:08.203458+0000","last_became_active":"2024-09-20T22:48:08.203028+0000","last_became_peered":"2024-09-20T22:48:08.203028+0000","last_unstale":"2024-09-20T22:48:08.203458+0000","last_undegraded":"2024-09-20T22:48:08.203458+0000","last_fullsized":"2024-09-20T22:48:08.203458+0000","mapping_epoch":407,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":408,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:47:56.410772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:47:56.410772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:49:17.378865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"74'13","reported_seq":770,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.203997+0000","last_change":"2024-09-20T22:47:49.267177+0000","last_active":"2024-09-20T22:48:08.203997+0000","last_peered":"2024-09-20T22:48:08.203997+0000","last_clean":"2024-09-20T22:48:08.203997+0000","last_became_active":"2024-09-20T22:47:49.228702+0000","last_became_peered":"2024-09-20T22:47:49.228702+0000","last_unstale":"2024-09-20T22:48:08.203997+0000","last_undegraded":"2024-09-20T22:48:08.203997+0000","last_fullsized":"2024-09-20T22:48:08.203997+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'13","last_scrub_stamp":"2024-09-20T22:46:04.145892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:46:04.145892+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":719,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.201052+0000","last_change":"2024-09-20T22:47:55.478609+0000","last_active":"2024-09-20T22:48:08.201052+0000","last_peered":"2024-09-20T22:48:08.201052+0000","last_clean":"2024-09-20T22:48:08.201052+0000","last_became_active":"2024-09-20T22:41:38.156837+0000","last_became_peered":"2024-09-20T22:41:38.156837+0000","last_unstale":"2024-09-20T22:48:08.201052+0000","last_undegraded":"2024-09-20T22:48:08.201052+0000","last_fullsized":"2024-09-20T22:48:08.201052+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:47:55.478501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:47:55.478501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:49:23.720355+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"380'2","reported_seq":757,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.197490+0000","last_change":"2024-09-20T22:47:49.233750+0000","last_active":"2024-09-20T22:48:08.197490+0000","last_peered":"2024-09-20T22:48:08.197490+0000","last_clean":"2024-09-20T22:48:08.197490+0000","last_became_active":"2024-09-20T22:47:49.222812+0000","last_became_peered":"2024-09-20T22:47:49.222812+0000","last_unstale":"2024-09-20T22:48:08.197490+0000","last_undegraded":"2024-09-20T22:48:08.197490+0000","last_fullsized":"2024-09-20T22:48:08.197490+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"380'2","last_scrub_stamp":"2024-09-20T22:47:40.983039+0000","last_deep_scrub":"380'2","last_deep_scrub_stamp":"2024-09-20T22:47:40.983039+0000","last_clean_scrub_stamp":"2024-09-20T22:47:40.983039+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:59.344462+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.001301459,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":711,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.200347+0000","last_change":"2024-09-20T22:47:22.881180+0000","last_active":"2024-09-20T22:48:08.200347+0000","last_peered":"2024-09-20T22:48:08.200347+0000","last_clean":"2024-09-20T22:48:08.200347+0000","last_became_active":"2024-09-20T22:41:42.185112+0000","last_became_peered":"2024-09-20T22:41:42.185112+0000","last_unstale":"2024-09-20T22:48:08.200347+0000","last_undegraded":"2024-09-20T22:48:08.200347+0000","last_fullsized":"2024-09-20T22:48:08.200347+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:47:22.881109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:47:22.881109+0000","last_clean_scrub_stamp":"2024-09-20T22:47:22.881109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:46.218094+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":699,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.201414+0000","last_change":"2024-09-20T22:46:27.448645+0000","last_active":"2024-09-20T22:48:08.201414+0000","last_peered":"2024-09-20T22:48:08.201414+0000","last_clean":"2024-09-20T22:48:08.201414+0000","last_became_active":"2024-09-20T22:41:44.830515+0000","last_became_peered":"2024-09-20T22:41:44.830515+0000","last_unstale":"2024-09-20T22:48:08.201414+0000","last_undegraded":"2024-09-20T22:48:08.201414+0000","last_fullsized":"2024-09-20T22:48:08.201414+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:46:27.448572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:46:27.448572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:12.326178+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"400'158","reported_seq":1756,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.201141+0000","last_change":"2024-09-20T22:48:04.223130+0000","last_active":"2024-09-20T22:48:08.201141+0000","last_peered":"2024-09-20T22:48:08.201141+0000","last_clean":"2024-09-20T22:48:08.201141+0000","last_became_active":"2024-09-20T22:47:52.206033+0000","last_became_peered":"2024-09-20T22:47:52.206033+0000","last_unstale":"2024-09-20T22:48:08.201141+0000","last_undegraded":"2024-09-20T22:48:08.201141+0000","last_fullsized":"2024-09-20T22:48:08.201141+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"262'49","last_scrub_stamp":"2024-09-20T22:45:41.346463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:45:41.346463+0000","objects_scrubbed":4,"log_size":158,"log_dups_size":0,"ondisk_log_size":158,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:12.326038+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0087985819999999992,"stat_sum":{"num_bytes":18410168,"num_objects":7,"num_object_clones":6,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":55,"num_read_kb":38714,"num_write":68,"num_write_kb":31788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":19673726,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"98","length":"1"},{"start":"a1","length":"2"}]},{"pgid":"1.3","version":"77'128","reported_seq":1026,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.208749+0000","last_change":"2024-09-20T22:48:08.208749+0000","last_active":"2024-09-20T22:48:08.208749+0000","last_peered":"2024-09-20T22:48:08.208749+0000","last_clean":"2024-09-20T22:48:08.208749+0000","last_became_active":"2024-09-20T22:48:08.208402+0000","last_became_peered":"2024-09-20T22:48:08.208402+0000","last_unstale":"2024-09-20T22:48:08.208749+0000","last_undegraded":"2024-09-20T22:48:08.208749+0000","last_fullsized":"2024-09-20T22:48:08.208749+0000","mapping_epoch":407,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":408,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:45:38.952841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:45:38.952841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"74'12","reported_seq":63,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.199386+0000","last_change":"2024-09-20T22:47:49.246080+0000","last_active":"2024-09-20T22:48:08.199386+0000","last_peered":"2024-09-20T22:48:08.199386+0000","last_clean":"2024-09-20T22:48:08.199386+0000","last_became_active":"2024-09-20T22:47:49.224223+0000","last_became_peered":"2024-09-20T22:47:49.224223+0000","last_unstale":"2024-09-20T22:48:08.199386+0000","last_undegraded":"2024-09-20T22:48:08.199386+0000","last_fullsized":"2024-09-20T22:48:08.199386+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-09-20T22:47:39.293455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:47:39.293455+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:39.767613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":63,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.196543+0000","last_change":"2024-09-20T22:48:05.233412+0000","last_active":"2024-09-20T22:48:08.196543+0000","last_peered":"2024-09-20T22:48:08.196543+0000","last_clean":"2024-09-20T22:48:08.196543+0000","last_became_active":"2024-09-20T22:48:05.233093+0000","last_became_peered":"2024-09-20T22:48:05.233093+0000","last_unstale":"2024-09-20T22:48:08.196543+0000","last_undegraded":"2024-09-20T22:48:08.196543+0000","last_fullsized":"2024-09-20T22:48:08.196543+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:47:55.576301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:44:19.834641+0000","last_clean_scrub_stamp":"2024-09-20T22:47:55.576301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:49:20.279866+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"75'5","reported_seq":1125,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.199369+0000","last_change":"2024-09-20T22:47:49.235292+0000","last_active":"2024-09-20T22:48:08.199369+0000","last_peered":"2024-09-20T22:48:08.199369+0000","last_clean":"2024-09-20T22:48:08.199369+0000","last_became_active":"2024-09-20T22:47:49.214414+0000","last_became_peered":"2024-09-20T22:47:49.214414+0000","last_unstale":"2024-09-20T22:48:08.199369+0000","last_undegraded":"2024-09-20T22:48:08.199369+0000","last_fullsized":"2024-09-20T22:48:08.199369+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-09-20T22:46:22.436424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:36.130684+0000","last_clean_scrub_stamp":"2024-09-20T22:46:22.436424+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:12.326164+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00078631700000000003,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":381,"num_read_kb":376,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":715,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.242728+0000","last_change":"2024-09-20T22:47:49.227940+0000","last_active":"2024-09-20T22:48:08.242728+0000","last_peered":"2024-09-20T22:48:08.242728+0000","last_clean":"2024-09-20T22:48:08.242728+0000","last_became_active":"2024-09-20T22:47:49.227371+0000","last_became_peered":"2024-09-20T22:47:49.227371+0000","last_unstale":"2024-09-20T22:48:08.242728+0000","last_undegraded":"2024-09-20T22:48:08.242728+0000","last_fullsized":"2024-09-20T22:48:08.242728+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:47:21.268493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:47:21.268493+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:31.211295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":707,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.202186+0000","last_change":"2024-09-20T22:47:59.388634+0000","last_active":"2024-09-20T22:48:08.202186+0000","last_peered":"2024-09-20T22:48:08.202186+0000","last_clean":"2024-09-20T22:48:08.202186+0000","last_became_active":"2024-09-20T22:41:44.199046+0000","last_became_peered":"2024-09-20T22:41:44.199046+0000","last_unstale":"2024-09-20T22:48:08.202186+0000","last_undegraded":"2024-09-20T22:48:08.202186+0000","last_fullsized":"2024-09-20T22:48:08.202186+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:47:59.388571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:47:59.388571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:49:09.987768+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.6","version":"406'135","reported_seq":1666,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.199142+0000","last_change":"2024-09-20T22:48:04.223375+0000","last_active":"2024-09-20T22:48:08.199142+0000","last_peered":"2024-09-20T22:48:08.199142+0000","last_clean":"2024-09-20T22:48:08.199142+0000","last_became_active":"2024-09-20T22:47:52.216302+0000","last_became_peered":"2024-09-20T22:47:52.216302+0000","last_unstale":"2024-09-20T22:48:08.199142+0000","last_undegraded":"2024-09-20T22:48:08.199142+0000","last_fullsized":"2024-09-20T22:48:08.199142+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"282'87","last_scrub_stamp":"2024-09-20T22:46:03.391903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:46:03.391903+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0016840640000000001,"stat_sum":{"num_bytes":4660216,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":30,"num_read_kb":32795,"num_write":25,"num_write_kb":5144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":5260887,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"98","length":"1"},{"start":"a1","length":"2"}]},{"pgid":"4.5","version":"408'20","reported_seq":904,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.204531+0000","last_change":"2024-09-20T22:48:05.267009+0000","last_active":"2024-09-20T22:48:08.204531+0000","last_peered":"2024-09-20T22:48:08.204531+0000","last_clean":"2024-09-20T22:48:08.204531+0000","last_became_active":"2024-09-20T22:48:05.243523+0000","last_became_peered":"2024-09-20T22:48:05.243523+0000","last_unstale":"2024-09-20T22:48:08.204531+0000","last_undegraded":"2024-09-20T22:48:08.204531+0000","last_fullsized":"2024-09-20T22:48:08.204531+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"264'18","last_scrub_stamp":"2024-09-20T22:46:04.901561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:46:04.901561+0000","objects_scrubbed":8,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"67'1","reported_seq":709,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.206506+0000","last_change":"2024-09-20T22:45:22.053343+0000","last_active":"2024-09-20T22:48:08.206506+0000","last_peered":"2024-09-20T22:48:08.206506+0000","last_clean":"2024-09-20T22:48:08.206506+0000","last_became_active":"2024-09-20T22:41:38.158917+0000","last_became_peered":"2024-09-20T22:41:38.158917+0000","last_unstale":"2024-09-20T22:48:08.206506+0000","last_undegraded":"2024-09-20T22:48:08.206506+0000","last_fullsized":"2024-09-20T22:48:08.206506+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-09-20T22:45:22.053262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:45:22.053262+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"77'128","reported_seq":1059,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.199488+0000","last_change":"2024-09-20T22:47:52.206552+0000","last_active":"2024-09-20T22:48:08.199488+0000","last_peered":"2024-09-20T22:48:08.199488+0000","last_clean":"2024-09-20T22:48:08.199488+0000","last_became_active":"2024-09-20T22:47:52.206169+0000","last_became_peered":"2024-09-20T22:47:52.206169+0000","last_unstale":"2024-09-20T22:48:08.199488+0000","last_undegraded":"2024-09-20T22:48:08.199488+0000","last_fullsized":"2024-09-20T22:48:08.199488+0000","mapping_epoch":391,"log_start":"77'128","ondisk_log_start":"77'128","created":19,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:47:22.417099+0000","last_deep_scrub":"77'128","last_deep_scrub_stamp":"2024-09-20T22:47:22.417099+0000","last_clean_scrub_stamp":"2024-09-20T22:47:22.417099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:36.734565+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":703,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.201779+0000","last_change":"2024-09-20T22:45:42.357247+0000","last_active":"2024-09-20T22:48:08.201779+0000","last_peered":"2024-09-20T22:48:08.201779+0000","last_clean":"2024-09-20T22:48:08.201779+0000","last_became_active":"2024-09-20T22:41:42.184387+0000","last_became_peered":"2024-09-20T22:41:42.184387+0000","last_unstale":"2024-09-20T22:48:08.201779+0000","last_undegraded":"2024-09-20T22:48:08.201779+0000","last_fullsized":"2024-09-20T22:48:08.201779+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:45:42.357173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:45:42.357173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:12.326117+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":711,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.242211+0000","last_change":"2024-09-20T22:48:05.242010+0000","last_active":"2024-09-20T22:48:08.242211+0000","last_peered":"2024-09-20T22:48:08.242211+0000","last_clean":"2024-09-20T22:48:08.242211+0000","last_became_active":"2024-09-20T22:48:05.241602+0000","last_became_peered":"2024-09-20T22:48:05.241602+0000","last_unstale":"2024-09-20T22:48:08.242211+0000","last_undegraded":"2024-09-20T22:48:08.242211+0000","last_fullsized":"2024-09-20T22:48:08.242211+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:45:41.820321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:45:41.820321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:11.497503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"400'162","reported_seq":1771,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.204262+0000","last_change":"2024-09-20T22:48:04.226624+0000","last_active":"2024-09-20T22:48:08.204262+0000","last_peered":"2024-09-20T22:48:08.204262+0000","last_clean":"2024-09-20T22:48:08.204262+0000","last_became_active":"2024-09-20T22:47:52.208200+0000","last_became_peered":"2024-09-20T22:47:52.208200+0000","last_unstale":"2024-09-20T22:48:08.204262+0000","last_undegraded":"2024-09-20T22:48:08.204262+0000","last_fullsized":"2024-09-20T22:48:08.204262+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"327'109","last_scrub_stamp":"2024-09-20T22:46:51.808234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:46:51.808234+0000","objects_scrubbed":2,"log_size":162,"log_dups_size":0,"ondisk_log_size":162,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0051233090000000004,"stat_sum":{"num_bytes":12471025,"num_objects":5,"num_object_clones":3,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":93,"num_read_kb":93497,"num_write":42,"num_write_kb":19366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":13529610,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"98","length":"1"},{"start":"a1","length":"2"}]},{"pgid":"4.e","version":"74'11","reported_seq":29,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.201618+0000","last_change":"2024-09-20T22:48:05.271944+0000","last_active":"2024-09-20T22:48:08.201618+0000","last_peered":"2024-09-20T22:48:08.201618+0000","last_clean":"2024-09-20T22:48:08.201618+0000","last_became_active":"2024-09-20T22:48:05.248271+0000","last_became_peered":"2024-09-20T22:48:05.248271+0000","last_unstale":"2024-09-20T22:48:08.201618+0000","last_undegraded":"2024-09-20T22:48:08.201618+0000","last_fullsized":"2024-09-20T22:48:08.201618+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-09-20T22:47:59.092235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:47:59.092235+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:49:11.673300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":17,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.199977+0000","last_change":"2024-09-20T22:48:05.255396+0000","last_active":"2024-09-20T22:48:08.199977+0000","last_peered":"2024-09-20T22:48:08.199977+0000","last_clean":"2024-09-20T22:48:08.199977+0000","last_became_active":"2024-09-20T22:48:05.254688+0000","last_became_peered":"2024-09-20T22:48:05.254688+0000","last_unstale":"2024-09-20T22:48:08.199977+0000","last_undegraded":"2024-09-20T22:48:08.199977+0000","last_fullsized":"2024-09-20T22:48:08.199977+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:46:30.871771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:46:30.871771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:12.869734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":715,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.242108+0000","last_change":"2024-09-20T22:48:05.234120+0000","last_active":"2024-09-20T22:48:08.242108+0000","last_peered":"2024-09-20T22:48:08.242108+0000","last_clean":"2024-09-20T22:48:08.242108+0000","last_became_active":"2024-09-20T22:48:05.233778+0000","last_became_peered":"2024-09-20T22:48:05.233778+0000","last_unstale":"2024-09-20T22:48:08.242108+0000","last_undegraded":"2024-09-20T22:48:08.242108+0000","last_fullsized":"2024-09-20T22:48:08.242108+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:47:38.288926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:47:38.288926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:49:08.085240+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":690,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.207702+0000","last_change":"2024-09-20T22:44:21.225215+0000","last_active":"2024-09-20T22:48:08.207702+0000","last_peered":"2024-09-20T22:48:08.207702+0000","last_clean":"2024-09-20T22:48:08.207702+0000","last_became_active":"2024-09-20T22:41:44.832009+0000","last_became_peered":"2024-09-20T22:41:44.832009+0000","last_unstale":"2024-09-20T22:48:08.207702+0000","last_undegraded":"2024-09-20T22:48:08.207702+0000","last_fullsized":"2024-09-20T22:48:08.207702+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:44:21.225119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:44:21.225119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.e","version":"406'239","reported_seq":1881,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.199795+0000","last_change":"2024-09-20T22:48:04.228426+0000","last_active":"2024-09-20T22:48:08.199795+0000","last_peered":"2024-09-20T22:48:08.199795+0000","last_clean":"2024-09-20T22:48:08.199795+0000","last_became_active":"2024-09-20T22:47:52.209452+0000","last_became_peered":"2024-09-20T22:47:52.209452+0000","last_unstale":"2024-09-20T22:48:08.199795+0000","last_undegraded":"2024-09-20T22:48:08.199795+0000","last_fullsized":"2024-09-20T22:48:08.199795+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"377'202","last_scrub_stamp":"2024-09-20T22:47:42.000554+0000","last_deep_scrub":"174'60","last_deep_scrub_stamp":"2024-09-20T22:44:18.301776+0000","last_clean_scrub_stamp":"2024-09-20T22:47:42.000554+0000","objects_scrubbed":5,"log_size":239,"log_dups_size":0,"ondisk_log_size":239,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:49:01.460374+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.013122778999999999,"stat_sum":{"num_bytes":17497904,"num_objects":9,"num_object_clones":7,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":1,"num_read":137,"num_read_kb":145195,"num_write":87,"num_write_kb":35755,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":23110741,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"98","length":"1"},{"start":"a1","length":"2"}]},{"pgid":"1.8","version":"77'128","reported_seq":970,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.240902+0000","last_change":"2024-09-20T22:45:22.698507+0000","last_active":"2024-09-20T22:48:08.240902+0000","last_peered":"2024-09-20T22:48:08.240902+0000","last_clean":"2024-09-20T22:48:08.240902+0000","last_became_active":"2024-09-20T22:42:52.405594+0000","last_became_peered":"2024-09-20T22:42:52.405594+0000","last_unstale":"2024-09-20T22:48:08.240902+0000","last_undegraded":"2024-09-20T22:48:08.240902+0000","last_fullsized":"2024-09-20T22:48:08.240902+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:45:22.698390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:45:22.698390+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:11.497486+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"74'17","reported_seq":35,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.200100+0000","last_change":"2024-09-20T22:48:05.273509+0000","last_active":"2024-09-20T22:48:08.200100+0000","last_peered":"2024-09-20T22:48:08.200100+0000","last_clean":"2024-09-20T22:48:08.200100+0000","last_became_active":"2024-09-20T22:48:05.247971+0000","last_became_peered":"2024-09-20T22:48:05.247971+0000","last_unstale":"2024-09-20T22:48:08.200100+0000","last_undegraded":"2024-09-20T22:48:08.200100+0000","last_fullsized":"2024-09-20T22:48:08.200100+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'17","last_scrub_stamp":"2024-09-20T22:47:38.052166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:47:38.052166+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:59.439253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":28,"num_read_kb":18,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":259,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.198993+0000","last_change":"2024-09-20T22:48:06.135135+0000","last_active":"2024-09-20T22:48:08.198993+0000","last_peered":"2024-09-20T22:48:08.198993+0000","last_clean":"2024-09-20T22:48:08.198993+0000","last_became_active":"2024-09-20T22:48:05.240481+0000","last_became_peered":"2024-09-20T22:48:05.240481+0000","last_unstale":"2024-09-20T22:48:08.198993+0000","last_undegraded":"2024-09-20T22:48:08.198993+0000","last_fullsized":"2024-09-20T22:48:08.198993+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:48:06.135068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:35.492562+0000","last_clean_scrub_stamp":"2024-09-20T22:48:06.135068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:49:31.308860+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":726,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.202281+0000","last_change":"2024-09-20T22:48:05.252451+0000","last_active":"2024-09-20T22:48:08.202281+0000","last_peered":"2024-09-20T22:48:08.202281+0000","last_clean":"2024-09-20T22:48:08.202281+0000","last_became_active":"2024-09-20T22:48:05.250175+0000","last_became_peered":"2024-09-20T22:48:05.250175+0000","last_unstale":"2024-09-20T22:48:08.202281+0000","last_undegraded":"2024-09-20T22:48:08.202281+0000","last_fullsized":"2024-09-20T22:48:08.202281+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:47:36.658662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:47:36.658662+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:50.405263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":726,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.198198+0000","last_change":"2024-09-20T22:48:05.239302+0000","last_active":"2024-09-20T22:48:08.198198+0000","last_peered":"2024-09-20T22:48:08.198198+0000","last_clean":"2024-09-20T22:48:08.198198+0000","last_became_active":"2024-09-20T22:48:05.239006+0000","last_became_peered":"2024-09-20T22:48:05.239006+0000","last_unstale":"2024-09-20T22:48:08.198198+0000","last_undegraded":"2024-09-20T22:48:08.198198+0000","last_fullsized":"2024-09-20T22:48:08.198198+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:47:58.989773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:47:58.989773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:49:24.841257+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"406'332","reported_seq":1938,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.198429+0000","last_change":"2024-09-20T22:48:04.220828+0000","last_active":"2024-09-20T22:48:08.198429+0000","last_peered":"2024-09-20T22:48:08.198429+0000","last_clean":"2024-09-20T22:48:08.198429+0000","last_became_active":"2024-09-20T22:47:52.216198+0000","last_became_peered":"2024-09-20T22:47:52.216198+0000","last_unstale":"2024-09-20T22:48:08.198429+0000","last_undegraded":"2024-09-20T22:48:08.198429+0000","last_fullsized":"2024-09-20T22:48:08.198429+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"395'325","last_scrub_stamp":"2024-09-20T22:47:57.577936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:47:57.577936+0000","objects_scrubbed":7,"log_size":332,"log_dups_size":0,"ondisk_log_size":332,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:49:23.348309+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.000450121,"stat_sum":{"num_bytes":15396303,"num_objects":8,"num_object_clones":5,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":211,"num_read_kb":214524,"num_write":128,"num_write_kb":40428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":53,"num_bytes_recovered":14983556,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"98","length":"1"},{"start":"a1","length":"2"}]},{"pgid":"1.9","version":"77'128","reported_seq":1018,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.199183+0000","last_change":"2024-09-20T22:48:03.252010+0000","last_active":"2024-09-20T22:48:08.199183+0000","last_peered":"2024-09-20T22:48:08.199183+0000","last_clean":"2024-09-20T22:48:08.199183+0000","last_became_active":"2024-09-20T22:42:52.400353+0000","last_became_peered":"2024-09-20T22:42:52.400353+0000","last_unstale":"2024-09-20T22:48:08.199183+0000","last_undegraded":"2024-09-20T22:48:08.199183+0000","last_fullsized":"2024-09-20T22:48:08.199183+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:48:03.251924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:48:03.251924+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:49:08.807948+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"74'10","reported_seq":29,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.199200+0000","last_change":"2024-09-20T22:48:05.262827+0000","last_active":"2024-09-20T22:48:08.199200+0000","last_peered":"2024-09-20T22:48:08.199200+0000","last_clean":"2024-09-20T22:48:08.199200+0000","last_became_active":"2024-09-20T22:48:05.242811+0000","last_became_peered":"2024-09-20T22:48:05.242811+0000","last_unstale":"2024-09-20T22:48:08.199200+0000","last_undegraded":"2024-09-20T22:48:08.199200+0000","last_fullsized":"2024-09-20T22:48:08.199200+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-09-20T22:47:21.092309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:47:21.092309+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:36.393442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":734,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.206711+0000","last_change":"2024-09-20T22:48:05.234296+0000","last_active":"2024-09-20T22:48:08.206711+0000","last_peered":"2024-09-20T22:48:08.206711+0000","last_clean":"2024-09-20T22:48:08.206711+0000","last_became_active":"2024-09-20T22:48:05.233999+0000","last_became_peered":"2024-09-20T22:48:05.233999+0000","last_unstale":"2024-09-20T22:48:08.206711+0000","last_undegraded":"2024-09-20T22:48:08.206711+0000","last_fullsized":"2024-09-20T22:48:08.206711+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:45:57.194715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:45:57.194715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"261'9","reported_seq":798,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.199568+0000","last_change":"2024-09-20T22:46:50.974814+0000","last_active":"2024-09-20T22:48:08.199568+0000","last_peered":"2024-09-20T22:48:08.199568+0000","last_clean":"2024-09-20T22:48:08.199568+0000","last_became_active":"2024-09-20T22:41:42.185863+0000","last_became_peered":"2024-09-20T22:41:42.185863+0000","last_unstale":"2024-09-20T22:48:08.199568+0000","last_undegraded":"2024-09-20T22:48:08.199568+0000","last_fullsized":"2024-09-20T22:48:08.199568+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'9","last_scrub_stamp":"2024-09-20T22:46:50.974735+0000","last_deep_scrub":"261'9","last_deep_scrub_stamp":"2024-09-20T22:46:50.974735+0000","last_clean_scrub_stamp":"2024-09-20T22:46:50.974735+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:13.135231+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":17,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.201315+0000","last_change":"2024-09-20T22:48:05.250642+0000","last_active":"2024-09-20T22:48:08.201315+0000","last_peered":"2024-09-20T22:48:08.201315+0000","last_clean":"2024-09-20T22:48:08.201315+0000","last_became_active":"2024-09-20T22:48:05.250284+0000","last_became_peered":"2024-09-20T22:48:05.250284+0000","last_unstale":"2024-09-20T22:48:08.201315+0000","last_undegraded":"2024-09-20T22:48:08.201315+0000","last_fullsized":"2024-09-20T22:48:08.201315+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:46:24.906516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:46:24.906516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:12.869779+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.8","version":"395'113","reported_seq":1699,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.204271+0000","last_change":"2024-09-20T22:48:08.204271+0000","last_active":"2024-09-20T22:48:08.204271+0000","last_peered":"2024-09-20T22:48:08.204271+0000","last_clean":"2024-09-20T22:48:08.204271+0000","last_became_active":"2024-09-20T22:48:08.203129+0000","last_became_peered":"2024-09-20T22:48:08.203129+0000","last_unstale":"2024-09-20T22:48:08.204271+0000","last_undegraded":"2024-09-20T22:48:08.204271+0000","last_fullsized":"2024-09-20T22:48:08.204271+0000","mapping_epoch":407,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":408,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'98","last_scrub_stamp":"2024-09-20T22:46:45.991157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:46:45.991157+0000","objects_scrubbed":0,"log_size":113,"log_dups_size":0,"ondisk_log_size":113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:14.982433+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00061180299999999996,"stat_sum":{"num_bytes":3755447,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":25,"num_read_kb":20437,"num_write":21,"num_write_kb":11116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":3755447,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"98","length":"1"},{"start":"a1","length":"2"}]},{"pgid":"4.b","version":"74'9","reported_seq":83,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.198670+0000","last_change":"2024-09-20T22:48:05.256987+0000","last_active":"2024-09-20T22:48:08.198670+0000","last_peered":"2024-09-20T22:48:08.198670+0000","last_clean":"2024-09-20T22:48:08.198670+0000","last_became_active":"2024-09-20T22:48:05.233267+0000","last_became_peered":"2024-09-20T22:48:05.233267+0000","last_unstale":"2024-09-20T22:48:08.198670+0000","last_undegraded":"2024-09-20T22:48:08.198670+0000","last_fullsized":"2024-09-20T22:48:08.198670+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-09-20T22:48:03.584242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:48:03.584242+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:49:11.956491+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":711,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.241941+0000","last_change":"2024-09-20T22:46:25.141094+0000","last_active":"2024-09-20T22:48:08.241941+0000","last_peered":"2024-09-20T22:48:08.241941+0000","last_clean":"2024-09-20T22:48:08.241941+0000","last_became_active":"2024-09-20T22:41:38.155953+0000","last_became_peered":"2024-09-20T22:41:38.155953+0000","last_unstale":"2024-09-20T22:48:08.241941+0000","last_undegraded":"2024-09-20T22:48:08.241941+0000","last_fullsized":"2024-09-20T22:48:08.241941+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:46:25.141028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:46:25.141028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:11.497642+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":734,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.199193+0000","last_change":"2024-09-20T22:48:05.241119+0000","last_active":"2024-09-20T22:48:08.199193+0000","last_peered":"2024-09-20T22:48:08.199193+0000","last_clean":"2024-09-20T22:48:08.199193+0000","last_became_active":"2024-09-20T22:48:05.240613+0000","last_became_peered":"2024-09-20T22:48:05.240613+0000","last_unstale":"2024-09-20T22:48:08.199193+0000","last_undegraded":"2024-09-20T22:48:08.199193+0000","last_fullsized":"2024-09-20T22:48:08.199193+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:46:49.061850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:46:49.061850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:15.112198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":49,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.196627+0000","last_change":"2024-09-20T22:47:49.219776+0000","last_active":"2024-09-20T22:48:08.196627+0000","last_peered":"2024-09-20T22:48:08.196627+0000","last_clean":"2024-09-20T22:48:08.196627+0000","last_became_active":"2024-09-20T22:47:49.219486+0000","last_became_peered":"2024-09-20T22:47:49.219486+0000","last_unstale":"2024-09-20T22:48:08.196627+0000","last_undegraded":"2024-09-20T22:48:08.196627+0000","last_fullsized":"2024-09-20T22:48:08.196627+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:46:51.469567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:46:51.469567+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:10.480416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"408'237","reported_seq":1884,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.210410+0000","last_change":"2024-09-20T22:48:08.209256+0000","last_active":"2024-09-20T22:48:08.210410+0000","last_peered":"2024-09-20T22:48:08.210410+0000","last_clean":"2024-09-20T22:48:08.210410+0000","last_became_active":"2024-09-20T22:48:08.206893+0000","last_became_peered":"2024-09-20T22:48:08.206893+0000","last_unstale":"2024-09-20T22:48:08.210410+0000","last_undegraded":"2024-09-20T22:48:08.210410+0000","last_fullsized":"2024-09-20T22:48:08.210410+0000","mapping_epoch":407,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":408,"parent":"0.0","parent_split_bits":0,"last_scrub":"257'132","last_scrub_stamp":"2024-09-20T22:45:36.793712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:45:36.793712+0000","objects_scrubbed":6,"log_size":237,"log_dups_size":0,"ondisk_log_size":237,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0016582560000000001,"stat_sum":{"num_bytes":14121291,"num_objects":7,"num_object_clones":5,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":87,"num_read_kb":99148,"num_write":69,"num_write_kb":26957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":65,"num_bytes_recovered":38710860,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"98","length":"1"},{"start":"a1","length":"2"}]},{"pgid":"4.a","version":"113'28","reported_seq":754,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.204570+0000","last_change":"2024-09-20T22:47:13.842452+0000","last_active":"2024-09-20T22:48:08.204570+0000","last_peered":"2024-09-20T22:48:08.204570+0000","last_clean":"2024-09-20T22:48:08.204570+0000","last_became_active":"2024-09-20T22:41:40.252011+0000","last_became_peered":"2024-09-20T22:41:40.252011+0000","last_unstale":"2024-09-20T22:48:08.204570+0000","last_undegraded":"2024-09-20T22:48:08.204570+0000","last_fullsized":"2024-09-20T22:48:08.204570+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'28","last_scrub_stamp":"2024-09-20T22:47:13.842316+0000","last_deep_scrub":"113'28","last_deep_scrub_stamp":"2024-09-20T22:43:12.725900+0000","last_clean_scrub_stamp":"2024-09-20T22:47:13.842316+0000","objects_scrubbed":9,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:31.564433+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":710,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.201804+0000","last_change":"2024-09-20T22:46:30.430426+0000","last_active":"2024-09-20T22:48:08.201804+0000","last_peered":"2024-09-20T22:48:08.201804+0000","last_clean":"2024-09-20T22:48:08.201804+0000","last_became_active":"2024-09-20T22:41:38.161524+0000","last_became_peered":"2024-09-20T22:41:38.161524+0000","last_unstale":"2024-09-20T22:48:08.201804+0000","last_undegraded":"2024-09-20T22:48:08.201804+0000","last_fullsized":"2024-09-20T22:48:08.201804+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:46:30.430353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:46:30.430353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:12.326207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":722,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.196954+0000","last_change":"2024-09-20T22:47:49.223627+0000","last_active":"2024-09-20T22:48:08.196954+0000","last_peered":"2024-09-20T22:48:08.196954+0000","last_clean":"2024-09-20T22:48:08.196954+0000","last_became_active":"2024-09-20T22:47:49.223220+0000","last_became_peered":"2024-09-20T22:47:49.223220+0000","last_unstale":"2024-09-20T22:48:08.196954+0000","last_undegraded":"2024-09-20T22:48:08.196954+0000","last_fullsized":"2024-09-20T22:48:08.196954+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:47:36.012147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:47:36.012147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:45.981117+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":706,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.199414+0000","last_change":"2024-09-20T22:47:18.329286+0000","last_active":"2024-09-20T22:48:08.199414+0000","last_peered":"2024-09-20T22:48:08.199414+0000","last_clean":"2024-09-20T22:48:08.199414+0000","last_became_active":"2024-09-20T22:41:44.199395+0000","last_became_peered":"2024-09-20T22:41:44.199395+0000","last_unstale":"2024-09-20T22:48:08.199414+0000","last_undegraded":"2024-09-20T22:48:08.199414+0000","last_fullsized":"2024-09-20T22:48:08.199414+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:47:18.329209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:47:18.329209+0000","last_clean_scrub_stamp":"2024-09-20T22:47:18.329209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:22.373197+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.a","version":"408'318","reported_seq":1895,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.373377+0000","last_change":"2024-09-20T22:48:04.211353+0000","last_active":"2024-09-20T22:48:08.373377+0000","last_peered":"2024-09-20T22:48:08.373377+0000","last_clean":"2024-09-20T22:48:08.373377+0000","last_became_active":"2024-09-20T22:42:52.405085+0000","last_became_peered":"2024-09-20T22:42:52.405085+0000","last_unstale":"2024-09-20T22:48:08.373377+0000","last_undegraded":"2024-09-20T22:48:08.373377+0000","last_fullsized":"2024-09-20T22:48:08.373377+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"301'219","last_scrub_stamp":"2024-09-20T22:46:23.260041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:46:23.260041+0000","objects_scrubbed":3,"log_size":318,"log_dups_size":0,"ondisk_log_size":318,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0018298100000000001,"stat_sum":{"num_bytes":23635726,"num_objects":10,"num_object_clones":7,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":1,"num_read":250,"num_read_kb":287297,"num_write":136,"num_write_kb":54968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"98","length":"1"},{"start":"a1","length":"2"}]},{"pgid":"4.9","version":"74'12","reported_seq":29,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.200073+0000","last_change":"2024-09-20T22:48:05.284523+0000","last_active":"2024-09-20T22:48:08.200073+0000","last_peered":"2024-09-20T22:48:08.200073+0000","last_clean":"2024-09-20T22:48:08.200073+0000","last_became_active":"2024-09-20T22:48:05.249438+0000","last_became_peered":"2024-09-20T22:48:05.249438+0000","last_unstale":"2024-09-20T22:48:08.200073+0000","last_undegraded":"2024-09-20T22:48:08.200073+0000","last_fullsized":"2024-09-20T22:48:08.200073+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-09-20T22:47:58.421473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:47:58.421473+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:49:19.311510+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":21,"num_read_kb":12,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":742,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.201973+0000","last_change":"2024-09-20T22:48:05.240596+0000","last_active":"2024-09-20T22:48:08.201973+0000","last_peered":"2024-09-20T22:48:08.201973+0000","last_clean":"2024-09-20T22:48:08.201973+0000","last_became_active":"2024-09-20T22:48:05.240225+0000","last_became_peered":"2024-09-20T22:48:05.240225+0000","last_unstale":"2024-09-20T22:48:08.201973+0000","last_undegraded":"2024-09-20T22:48:08.201973+0000","last_fullsized":"2024-09-20T22:48:08.201973+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:46:52.465940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:46:52.465940+0000","last_clean_scrub_stamp":"2024-09-20T22:46:52.465940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:12.326236+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":722,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.197486+0000","last_change":"2024-09-20T22:47:49.222432+0000","last_active":"2024-09-20T22:48:08.197486+0000","last_peered":"2024-09-20T22:48:08.197486+0000","last_clean":"2024-09-20T22:48:08.197486+0000","last_became_active":"2024-09-20T22:47:49.222088+0000","last_became_peered":"2024-09-20T22:47:49.222088+0000","last_unstale":"2024-09-20T22:48:08.197486+0000","last_undegraded":"2024-09-20T22:48:08.197486+0000","last_fullsized":"2024-09-20T22:48:08.197486+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:46:01.828080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:46:01.828080+0000","last_clean_scrub_stamp":"2024-09-20T22:46:01.828080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:13.135197+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":698,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.205557+0000","last_change":"2024-09-20T22:47:38.019602+0000","last_active":"2024-09-20T22:48:08.205557+0000","last_peered":"2024-09-20T22:48:08.205557+0000","last_clean":"2024-09-20T22:48:08.205557+0000","last_became_active":"2024-09-20T22:41:44.192609+0000","last_became_peered":"2024-09-20T22:41:44.192609+0000","last_unstale":"2024-09-20T22:48:08.205557+0000","last_undegraded":"2024-09-20T22:48:08.205557+0000","last_fullsized":"2024-09-20T22:48:08.205557+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:47:38.019533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:55.170379+0000","last_clean_scrub_stamp":"2024-09-20T22:47:38.019533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:49:06.773525+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"404'508","reported_seq":2084,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.204538+0000","last_change":"2024-09-20T22:48:06.237725+0000","last_active":"2024-09-20T22:48:08.204538+0000","last_peered":"2024-09-20T22:48:08.204538+0000","last_clean":"2024-09-20T22:48:08.204538+0000","last_became_active":"2024-09-20T22:42:52.409642+0000","last_became_peered":"2024-09-20T22:42:52.409642+0000","last_unstale":"2024-09-20T22:48:08.204538+0000","last_undegraded":"2024-09-20T22:48:08.204538+0000","last_fullsized":"2024-09-20T22:48:08.204538+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"194'116","last_scrub_stamp":"2024-09-20T22:44:37.097674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:44:37.097674+0000","objects_scrubbed":3,"log_size":508,"log_dups_size":0,"ondisk_log_size":508,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.026252276000000001,"stat_sum":{"num_bytes":32621266,"num_objects":20,"num_object_clones":16,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":1,"num_read":303,"num_read_kb":309599,"num_write":232,"num_write_kb":73603,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"98","length":"1"},{"start":"a1","length":"2"}]},{"pgid":"4.8","version":"74'15","reported_seq":733,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.241211+0000","last_change":"2024-09-20T22:47:35.313505+0000","last_active":"2024-09-20T22:48:08.241211+0000","last_peered":"2024-09-20T22:48:08.241211+0000","last_clean":"2024-09-20T22:48:08.241211+0000","last_became_active":"2024-09-20T22:41:40.252113+0000","last_became_peered":"2024-09-20T22:41:40.252113+0000","last_unstale":"2024-09-20T22:48:08.241211+0000","last_undegraded":"2024-09-20T22:48:08.241211+0000","last_fullsized":"2024-09-20T22:48:08.241211+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-09-20T22:47:35.313413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:47:35.313413+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:52.070278+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'2","reported_seq":770,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.202374+0000","last_change":"2024-09-20T22:48:05.269145+0000","last_active":"2024-09-20T22:48:08.202374+0000","last_peered":"2024-09-20T22:48:08.202374+0000","last_clean":"2024-09-20T22:48:08.202374+0000","last_became_active":"2024-09-20T22:48:05.238098+0000","last_became_peered":"2024-09-20T22:48:05.238098+0000","last_unstale":"2024-09-20T22:48:08.202374+0000","last_undegraded":"2024-09-20T22:48:08.202374+0000","last_fullsized":"2024-09-20T22:48:08.202374+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-09-20T22:46:48.473494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:46:48.473494+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:12.326222+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"408'10","reported_seq":817,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.199449+0000","last_change":"2024-09-20T22:48:05.265680+0000","last_active":"2024-09-20T22:48:08.199449+0000","last_peered":"2024-09-20T22:48:08.199449+0000","last_clean":"2024-09-20T22:48:08.199449+0000","last_became_active":"2024-09-20T22:48:05.239344+0000","last_became_peered":"2024-09-20T22:48:05.239344+0000","last_unstale":"2024-09-20T22:48:08.199449+0000","last_undegraded":"2024-09-20T22:48:08.199449+0000","last_fullsized":"2024-09-20T22:48:08.199449+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"159'9","last_scrub_stamp":"2024-09-20T22:47:37.382220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:47:37.382220+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:40.141204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":711,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.241553+0000","last_change":"2024-09-20T22:47:49.228372+0000","last_active":"2024-09-20T22:48:08.241553+0000","last_peered":"2024-09-20T22:48:08.241553+0000","last_clean":"2024-09-20T22:48:08.241553+0000","last_became_active":"2024-09-20T22:47:49.227569+0000","last_became_peered":"2024-09-20T22:47:49.227569+0000","last_unstale":"2024-09-20T22:48:08.241553+0000","last_undegraded":"2024-09-20T22:48:08.241553+0000","last_fullsized":"2024-09-20T22:48:08.241553+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:47:41.284992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:47:41.284992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:49:09.902863+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.14","version":"395'116","reported_seq":1433,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.208690+0000","last_change":"2024-09-20T22:48:04.217542+0000","last_active":"2024-09-20T22:48:08.208690+0000","last_peered":"2024-09-20T22:48:08.208690+0000","last_clean":"2024-09-20T22:48:08.208690+0000","last_became_active":"2024-09-20T22:43:39.132459+0000","last_became_peered":"2024-09-20T22:43:39.132459+0000","last_unstale":"2024-09-20T22:48:08.208690+0000","last_undegraded":"2024-09-20T22:48:08.208690+0000","last_fullsized":"2024-09-20T22:48:08.208690+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":5,"last_scrub":"395'116","last_scrub_stamp":"2024-09-20T22:48:03.152941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:48:03.152941+0000","objects_scrubbed":3,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:49:28.963523+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00076386800000000001,"stat_sum":{"num_bytes":5876815,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":44,"num_read_kb":36718,"num_write":37,"num_write_kb":10623,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"98","length":"1"},{"start":"a1","length":"2"}]},{"pgid":"3.10","version":"0'0","reported_seq":730,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.204196+0000","last_change":"2024-09-20T22:47:49.213597+0000","last_active":"2024-09-20T22:48:08.204196+0000","last_peered":"2024-09-20T22:48:08.204196+0000","last_clean":"2024-09-20T22:48:08.204196+0000","last_became_active":"2024-09-20T22:47:49.213292+0000","last_became_peered":"2024-09-20T22:47:49.213292+0000","last_unstale":"2024-09-20T22:48:08.204196+0000","last_undegraded":"2024-09-20T22:48:08.204196+0000","last_fullsized":"2024-09-20T22:48:08.204196+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:47:12.820304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:47:12.820304+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:28.113365+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"74'6","reported_seq":729,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.208420+0000","last_change":"2024-09-20T22:45:39.079425+0000","last_active":"2024-09-20T22:48:08.208420+0000","last_peered":"2024-09-20T22:48:08.208420+0000","last_clean":"2024-09-20T22:48:08.208420+0000","last_became_active":"2024-09-20T22:43:09.116887+0000","last_became_peered":"2024-09-20T22:43:09.116887+0000","last_unstale":"2024-09-20T22:48:08.208420+0000","last_undegraded":"2024-09-20T22:48:08.208420+0000","last_fullsized":"2024-09-20T22:48:08.208420+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'6","last_scrub_stamp":"2024-09-20T22:45:39.079346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:45:39.079346+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":703,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.242066+0000","last_change":"2024-09-20T22:45:54.949745+0000","last_active":"2024-09-20T22:48:08.242066+0000","last_peered":"2024-09-20T22:48:08.242066+0000","last_clean":"2024-09-20T22:48:08.242066+0000","last_became_active":"2024-09-20T22:41:42.184930+0000","last_became_peered":"2024-09-20T22:41:42.184930+0000","last_unstale":"2024-09-20T22:48:08.242066+0000","last_undegraded":"2024-09-20T22:48:08.242066+0000","last_fullsized":"2024-09-20T22:48:08.242066+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:45:54.949661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:09.705632+0000","last_clean_scrub_stamp":"2024-09-20T22:45:54.949661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:11.497534+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"74'1","reported_seq":719,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.201295+0000","last_change":"2024-09-20T22:48:05.262744+0000","last_active":"2024-09-20T22:48:08.201295+0000","last_peered":"2024-09-20T22:48:08.201295+0000","last_clean":"2024-09-20T22:48:08.201295+0000","last_became_active":"2024-09-20T22:48:05.237882+0000","last_became_peered":"2024-09-20T22:48:05.237882+0000","last_unstale":"2024-09-20T22:48:08.201295+0000","last_undegraded":"2024-09-20T22:48:08.201295+0000","last_fullsized":"2024-09-20T22:48:08.201295+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-09-20T22:45:59.486075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:45:59.486075+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:12.326149+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.15","version":"98'3","reported_seq":1547,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.200946+0000","last_change":"2024-09-20T22:48:04.223165+0000","last_active":"2024-09-20T22:48:08.200946+0000","last_peered":"2024-09-20T22:48:08.200946+0000","last_clean":"2024-09-20T22:48:08.200946+0000","last_became_active":"2024-09-20T22:44:09.174719+0000","last_became_peered":"2024-09-20T22:44:09.174719+0000","last_unstale":"2024-09-20T22:48:08.200946+0000","last_undegraded":"2024-09-20T22:48:08.200946+0000","last_fullsized":"2024-09-20T22:48:08.200946+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":5,"last_scrub":"98'3","last_scrub_stamp":"2024-09-20T22:47:35.357009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:47:35.357009+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:59.433553+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.010971771999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"98","length":"1"},{"start":"a1","length":"2"}]},{"pgid":"4.16","version":"74'9","reported_seq":59,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.199289+0000","last_change":"2024-09-20T22:47:49.262043+0000","last_active":"2024-09-20T22:48:08.199289+0000","last_peered":"2024-09-20T22:48:08.199289+0000","last_clean":"2024-09-20T22:48:08.199289+0000","last_became_active":"2024-09-20T22:47:49.231315+0000","last_became_peered":"2024-09-20T22:47:49.231315+0000","last_unstale":"2024-09-20T22:48:08.199289+0000","last_undegraded":"2024-09-20T22:48:08.199289+0000","last_fullsized":"2024-09-20T22:48:08.199289+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-09-20T22:46:21.269487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:46:21.269487+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:11.508611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":734,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.198872+0000","last_change":"2024-09-20T22:48:05.239747+0000","last_active":"2024-09-20T22:48:08.198872+0000","last_peered":"2024-09-20T22:48:08.198872+0000","last_clean":"2024-09-20T22:48:08.198872+0000","last_became_active":"2024-09-20T22:48:05.238801+0000","last_became_peered":"2024-09-20T22:48:05.238801+0000","last_unstale":"2024-09-20T22:48:08.198872+0000","last_undegraded":"2024-09-20T22:48:08.198872+0000","last_fullsized":"2024-09-20T22:48:08.198872+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:46:46.438556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:46:46.438556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:09.910531+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":702,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.208132+0000","last_change":"2024-09-20T22:46:48.012442+0000","last_active":"2024-09-20T22:48:08.208132+0000","last_peered":"2024-09-20T22:48:08.208132+0000","last_clean":"2024-09-20T22:48:08.208132+0000","last_became_active":"2024-09-20T22:41:42.187706+0000","last_became_peered":"2024-09-20T22:41:42.187706+0000","last_unstale":"2024-09-20T22:48:08.208132+0000","last_undegraded":"2024-09-20T22:48:08.208132+0000","last_fullsized":"2024-09-20T22:48:08.208132+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:46:48.012372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:46:48.012372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:11.078815+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":730,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.197648+0000","last_change":"2024-09-20T22:48:05.240005+0000","last_active":"2024-09-20T22:48:08.197648+0000","last_peered":"2024-09-20T22:48:08.197648+0000","last_clean":"2024-09-20T22:48:08.197648+0000","last_became_active":"2024-09-20T22:48:05.239710+0000","last_became_peered":"2024-09-20T22:48:05.239710+0000","last_unstale":"2024-09-20T22:48:08.197648+0000","last_undegraded":"2024-09-20T22:48:08.197648+0000","last_fullsized":"2024-09-20T22:48:08.197648+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:46:29.870228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:46:29.870228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:13.135213+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.16","version":"395'133","reported_seq":1587,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.208961+0000","last_change":"2024-09-20T22:48:08.207549+0000","last_active":"2024-09-20T22:48:08.208961+0000","last_peered":"2024-09-20T22:48:08.208961+0000","last_clean":"2024-09-20T22:48:08.208961+0000","last_became_active":"2024-09-20T22:48:08.206355+0000","last_became_peered":"2024-09-20T22:48:08.206355+0000","last_unstale":"2024-09-20T22:48:08.208961+0000","last_undegraded":"2024-09-20T22:48:08.208961+0000","last_fullsized":"2024-09-20T22:48:08.208961+0000","mapping_epoch":407,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":408,"parent":"0.0","parent_split_bits":5,"last_scrub":"282'91","last_scrub_stamp":"2024-09-20T22:46:05.393044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:46:05.393044+0000","objects_scrubbed":1,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00057551300000000004,"stat_sum":{"num_bytes":3647185,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":68,"num_read_kb":74958,"num_write":29,"num_write_kb":10963,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":10024944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"98","length":"1"},{"start":"a1","length":"2"}]},{"pgid":"4.15","version":"74'9","reported_seq":723,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.242293+0000","last_change":"2024-09-20T22:46:27.123331+0000","last_active":"2024-09-20T22:48:08.242293+0000","last_peered":"2024-09-20T22:48:08.242293+0000","last_clean":"2024-09-20T22:48:08.242293+0000","last_became_active":"2024-09-20T22:41:40.252486+0000","last_became_peered":"2024-09-20T22:41:40.252486+0000","last_unstale":"2024-09-20T22:48:08.242293+0000","last_undegraded":"2024-09-20T22:48:08.242293+0000","last_fullsized":"2024-09-20T22:48:08.242293+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-09-20T22:46:27.123248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:46:27.123248+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:11.497672+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":49,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.198056+0000","last_change":"2024-09-20T22:47:49.230303+0000","last_active":"2024-09-20T22:48:08.198056+0000","last_peered":"2024-09-20T22:48:08.198056+0000","last_clean":"2024-09-20T22:48:08.198056+0000","last_became_active":"2024-09-20T22:47:49.229936+0000","last_became_peered":"2024-09-20T22:47:49.229936+0000","last_unstale":"2024-09-20T22:48:08.198056+0000","last_undegraded":"2024-09-20T22:48:08.198056+0000","last_fullsized":"2024-09-20T22:48:08.198056+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:45:23.789053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:45:23.789053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:11.508471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"74'8","reported_seq":789,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.207656+0000","last_change":"2024-09-20T22:47:56.150641+0000","last_active":"2024-09-20T22:48:08.207656+0000","last_peered":"2024-09-20T22:48:08.207656+0000","last_clean":"2024-09-20T22:48:08.207656+0000","last_became_active":"2024-09-20T22:41:42.186352+0000","last_became_peered":"2024-09-20T22:41:42.186352+0000","last_unstale":"2024-09-20T22:48:08.207656+0000","last_undegraded":"2024-09-20T22:48:08.207656+0000","last_fullsized":"2024-09-20T22:48:08.207656+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-09-20T22:47:56.150562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:47:56.150562+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:49:03.909124+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":17,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.202277+0000","last_change":"2024-09-20T22:48:05.250192+0000","last_active":"2024-09-20T22:48:08.202277+0000","last_peered":"2024-09-20T22:48:08.202277+0000","last_clean":"2024-09-20T22:48:08.202277+0000","last_became_active":"2024-09-20T22:48:05.249656+0000","last_became_peered":"2024-09-20T22:48:05.249656+0000","last_unstale":"2024-09-20T22:48:08.202277+0000","last_undegraded":"2024-09-20T22:48:08.202277+0000","last_fullsized":"2024-09-20T22:48:08.202277+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:47:19.828541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:47:19.828541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:32.723932+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"408'360","reported_seq":1726,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.212570+0000","last_change":"2024-09-20T22:48:08.212973+0000","last_active":"2024-09-20T22:48:08.212570+0000","last_peered":"2024-09-20T22:48:08.212570+0000","last_clean":"2024-09-20T22:48:08.212570+0000","last_became_active":"2024-09-20T22:48:08.206790+0000","last_became_peered":"2024-09-20T22:48:08.206790+0000","last_unstale":"2024-09-20T22:48:08.212570+0000","last_undegraded":"2024-09-20T22:48:08.212570+0000","last_fullsized":"2024-09-20T22:48:08.212570+0000","mapping_epoch":407,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":408,"parent":"0.0","parent_split_bits":5,"last_scrub":"270'229","last_scrub_stamp":"2024-09-20T22:45:53.389694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:45:53.389694+0000","objects_scrubbed":5,"log_size":360,"log_dups_size":0,"ondisk_log_size":360,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0053840260000000001,"stat_sum":{"num_bytes":20020029,"num_objects":11,"num_object_clones":9,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":1,"num_read":135,"num_read_kb":145315,"num_write":87,"num_write_kb":30868,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":14257748,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"98","length":"1"},{"start":"a1","length":"2"}]},{"pgid":"4.14","version":"74'10","reported_seq":721,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.207293+0000","last_change":"2024-09-20T22:46:24.269736+0000","last_active":"2024-09-20T22:48:08.207293+0000","last_peered":"2024-09-20T22:48:08.207293+0000","last_clean":"2024-09-20T22:48:08.207293+0000","last_became_active":"2024-09-20T22:41:40.253614+0000","last_became_peered":"2024-09-20T22:41:40.253614+0000","last_unstale":"2024-09-20T22:48:08.207293+0000","last_undegraded":"2024-09-20T22:48:08.207293+0000","last_fullsized":"2024-09-20T22:48:08.207293+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-09-20T22:46:24.269655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:46:24.269655+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":730,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.200919+0000","last_change":"2024-09-20T22:48:05.240585+0000","last_active":"2024-09-20T22:48:08.200919+0000","last_peered":"2024-09-20T22:48:08.200919+0000","last_clean":"2024-09-20T22:48:08.200919+0000","last_became_active":"2024-09-20T22:48:05.240231+0000","last_became_peered":"2024-09-20T22:48:05.240231+0000","last_unstale":"2024-09-20T22:48:08.200919+0000","last_undegraded":"2024-09-20T22:48:08.200919+0000","last_fullsized":"2024-09-20T22:48:08.200919+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:48:01.305632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:48:01.305632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:49:16.408958+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"389'10","reported_seq":821,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.242173+0000","last_change":"2024-09-20T22:47:58.443470+0000","last_active":"2024-09-20T22:48:08.242173+0000","last_peered":"2024-09-20T22:48:08.242173+0000","last_clean":"2024-09-20T22:48:08.242173+0000","last_became_active":"2024-09-20T22:47:49.227113+0000","last_became_peered":"2024-09-20T22:47:49.227113+0000","last_unstale":"2024-09-20T22:48:08.242173+0000","last_undegraded":"2024-09-20T22:48:08.242173+0000","last_fullsized":"2024-09-20T22:48:08.242173+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'10","last_scrub_stamp":"2024-09-20T22:47:58.443393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:47:58.443393+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:49:10.836242+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":49,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.198842+0000","last_change":"2024-09-20T22:47:49.232560+0000","last_active":"2024-09-20T22:48:08.198842+0000","last_peered":"2024-09-20T22:48:08.198842+0000","last_clean":"2024-09-20T22:48:08.198842+0000","last_became_active":"2024-09-20T22:47:49.230674+0000","last_became_peered":"2024-09-20T22:47:49.230674+0000","last_unstale":"2024-09-20T22:48:08.198842+0000","last_undegraded":"2024-09-20T22:48:08.198842+0000","last_fullsized":"2024-09-20T22:48:08.198842+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:47:41.411716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:47:41.411716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:54.101804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.10","version":"395'120","reported_seq":1651,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.205863+0000","last_change":"2024-09-20T22:48:04.214307+0000","last_active":"2024-09-20T22:48:08.205863+0000","last_peered":"2024-09-20T22:48:08.205863+0000","last_clean":"2024-09-20T22:48:08.205863+0000","last_became_active":"2024-09-20T22:43:39.130391+0000","last_became_peered":"2024-09-20T22:43:39.130391+0000","last_unstale":"2024-09-20T22:48:08.205863+0000","last_undegraded":"2024-09-20T22:48:08.205863+0000","last_fullsized":"2024-09-20T22:48:08.205863+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":5,"last_scrub":"146'43","last_scrub_stamp":"2024-09-20T22:43:46.106059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:43:46.106059+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00087280599999999995,"stat_sum":{"num_bytes":7855858,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":48,"num_read_kb":65520,"num_write":31,"num_write_kb":15573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"98","length":"1"},{"start":"a1","length":"2"}]},{"pgid":"4.13","version":"74'11","reported_seq":29,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.199944+0000","last_change":"2024-09-20T22:48:05.307593+0000","last_active":"2024-09-20T22:48:08.199944+0000","last_peered":"2024-09-20T22:48:08.199944+0000","last_clean":"2024-09-20T22:48:08.199944+0000","last_became_active":"2024-09-20T22:48:05.249510+0000","last_became_peered":"2024-09-20T22:48:05.249510+0000","last_unstale":"2024-09-20T22:48:08.199944+0000","last_undegraded":"2024-09-20T22:48:08.199944+0000","last_fullsized":"2024-09-20T22:48:08.199944+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-09-20T22:45:37.940753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:45:37.940753+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:12.869691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":17,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.199023+0000","last_change":"2024-09-20T22:48:05.252646+0000","last_active":"2024-09-20T22:48:08.199023+0000","last_peered":"2024-09-20T22:48:08.199023+0000","last_clean":"2024-09-20T22:48:08.199023+0000","last_became_active":"2024-09-20T22:48:05.252026+0000","last_became_peered":"2024-09-20T22:48:05.252026+0000","last_unstale":"2024-09-20T22:48:08.199023+0000","last_undegraded":"2024-09-20T22:48:08.199023+0000","last_fullsized":"2024-09-20T22:48:08.199023+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:46:04.391306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:46:04.391306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:12.869705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":702,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.203918+0000","last_change":"2024-09-20T22:46:25.881993+0000","last_active":"2024-09-20T22:48:08.203918+0000","last_peered":"2024-09-20T22:48:08.203918+0000","last_clean":"2024-09-20T22:48:08.203918+0000","last_became_active":"2024-09-20T22:41:42.181736+0000","last_became_peered":"2024-09-20T22:41:42.181736+0000","last_unstale":"2024-09-20T22:48:08.203918+0000","last_undegraded":"2024-09-20T22:48:08.203918+0000","last_fullsized":"2024-09-20T22:48:08.203918+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:46:25.881921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:46:25.881921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":710,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.206246+0000","last_change":"2024-09-20T22:47:49.216479+0000","last_active":"2024-09-20T22:48:08.206246+0000","last_peered":"2024-09-20T22:48:08.206246+0000","last_clean":"2024-09-20T22:48:08.206246+0000","last_became_active":"2024-09-20T22:47:49.216141+0000","last_became_peered":"2024-09-20T22:47:49.216141+0000","last_unstale":"2024-09-20T22:48:08.206246+0000","last_undegraded":"2024-09-20T22:48:08.206246+0000","last_fullsized":"2024-09-20T22:48:08.206246+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:47:36.066480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:47:36.066480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:48.477867+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"135'39","reported_seq":1328,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.199753+0000","last_change":"2024-09-20T22:48:04.223979+0000","last_active":"2024-09-20T22:48:08.199753+0000","last_peered":"2024-09-20T22:48:08.199753+0000","last_clean":"2024-09-20T22:48:08.199753+0000","last_became_active":"2024-09-20T22:43:39.130210+0000","last_became_peered":"2024-09-20T22:43:39.130210+0000","last_unstale":"2024-09-20T22:48:08.199753+0000","last_undegraded":"2024-09-20T22:48:08.199753+0000","last_fullsized":"2024-09-20T22:48:08.199753+0000","mapping_epoch":139,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":140,"parent":"0.0","parent_split_bits":5,"last_scrub":"135'39","last_scrub_stamp":"2024-09-20T22:48:02.280014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:48:02.280014+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:49:20.583243+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.002786766,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"98","length":"1"},{"start":"a1","length":"2"}]},{"pgid":"4.12","version":"404'13","reported_seq":734,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.204087+0000","last_change":"2024-09-20T22:48:04.276356+0000","last_active":"2024-09-20T22:48:08.204087+0000","last_peered":"2024-09-20T22:48:08.204087+0000","last_clean":"2024-09-20T22:48:08.204087+0000","last_became_active":"2024-09-20T22:41:40.164796+0000","last_became_peered":"2024-09-20T22:41:40.164796+0000","last_unstale":"2024-09-20T22:48:08.204087+0000","last_undegraded":"2024-09-20T22:48:08.204087+0000","last_fullsized":"2024-09-20T22:48:08.204087+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"404'13","last_scrub_stamp":"2024-09-20T22:48:04.276282+0000","last_deep_scrub":"404'13","last_deep_scrub_stamp":"2024-09-20T22:48:04.276282+0000","last_clean_scrub_stamp":"2024-09-20T22:48:04.276282+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:49:13.107971+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":737,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.201659+0000","last_change":"2024-09-20T22:48:05.239255+0000","last_active":"2024-09-20T22:48:08.201659+0000","last_peered":"2024-09-20T22:48:08.201659+0000","last_clean":"2024-09-20T22:48:08.201659+0000","last_became_active":"2024-09-20T22:48:05.238812+0000","last_became_peered":"2024-09-20T22:48:05.238812+0000","last_unstale":"2024-09-20T22:48:08.201659+0000","last_undegraded":"2024-09-20T22:48:08.201659+0000","last_fullsized":"2024-09-20T22:48:08.201659+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:45:38.371521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:31.138067+0000","last_clean_scrub_stamp":"2024-09-20T22:45:38.371521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:12.326086+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":706,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.205633+0000","last_change":"2024-09-20T22:47:49.220020+0000","last_active":"2024-09-20T22:48:08.205633+0000","last_peered":"2024-09-20T22:48:08.205633+0000","last_clean":"2024-09-20T22:48:08.205633+0000","last_became_active":"2024-09-20T22:47:49.218708+0000","last_became_peered":"2024-09-20T22:47:49.218708+0000","last_unstale":"2024-09-20T22:48:08.205633+0000","last_undegraded":"2024-09-20T22:48:08.205633+0000","last_fullsized":"2024-09-20T22:48:08.205633+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:44:15.242881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:44:15.242881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":49,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.196579+0000","last_change":"2024-09-20T22:47:49.225362+0000","last_active":"2024-09-20T22:48:08.196579+0000","last_peered":"2024-09-20T22:48:08.196579+0000","last_clean":"2024-09-20T22:48:08.196579+0000","last_became_active":"2024-09-20T22:47:49.225027+0000","last_became_peered":"2024-09-20T22:47:49.225027+0000","last_unstale":"2024-09-20T22:48:08.196579+0000","last_undegraded":"2024-09-20T22:48:08.196579+0000","last_fullsized":"2024-09-20T22:48:08.196579+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:47:37.309842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:47:37.309842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:49:05.473839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.12","version":"400'375","reported_seq":1836,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.204167+0000","last_change":"2024-09-20T22:48:04.226775+0000","last_active":"2024-09-20T22:48:08.204167+0000","last_peered":"2024-09-20T22:48:08.204167+0000","last_clean":"2024-09-20T22:48:08.204167+0000","last_became_active":"2024-09-20T22:47:52.209332+0000","last_became_peered":"2024-09-20T22:47:52.209332+0000","last_unstale":"2024-09-20T22:48:08.204167+0000","last_undegraded":"2024-09-20T22:48:08.204167+0000","last_fullsized":"2024-09-20T22:48:08.204167+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":5,"last_scrub":"377'359","last_scrub_stamp":"2024-09-20T22:47:41.855224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:47:41.855224+0000","objects_scrubbed":9,"log_size":375,"log_dups_size":0,"ondisk_log_size":375,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:49:05.315401+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.0080037029999999992,"stat_sum":{"num_bytes":15739222,"num_objects":9,"num_object_clones":6,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":1,"num_read":147,"num_read_kb":137302,"num_write":110,"num_write_kb":37094,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":68,"num_bytes_recovered":27326001,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"98","length":"1"},{"start":"a1","length":"2"}]},{"pgid":"4.11","version":"74'11","reported_seq":516,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.241185+0000","last_change":"2024-09-20T22:46:31.254470+0000","last_active":"2024-09-20T22:48:08.241185+0000","last_peered":"2024-09-20T22:48:08.241185+0000","last_clean":"2024-09-20T22:48:08.241185+0000","last_became_active":"2024-09-20T22:44:09.170785+0000","last_became_peered":"2024-09-20T22:44:09.170785+0000","last_unstale":"2024-09-20T22:48:08.241185+0000","last_undegraded":"2024-09-20T22:48:08.241185+0000","last_fullsized":"2024-09-20T22:48:08.241185+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-09-20T22:46:31.254376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:46:31.254376+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:11.497688+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":719,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.242122+0000","last_change":"2024-09-20T22:48:02.512192+0000","last_active":"2024-09-20T22:48:08.242122+0000","last_peered":"2024-09-20T22:48:08.242122+0000","last_clean":"2024-09-20T22:48:08.242122+0000","last_became_active":"2024-09-20T22:41:38.158701+0000","last_became_peered":"2024-09-20T22:41:38.158701+0000","last_unstale":"2024-09-20T22:48:08.242122+0000","last_undegraded":"2024-09-20T22:48:08.242122+0000","last_fullsized":"2024-09-20T22:48:08.242122+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:48:02.512093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:45:24.765414+0000","last_clean_scrub_stamp":"2024-09-20T22:48:02.512093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:49:28.678353+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":726,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.199047+0000","last_change":"2024-09-20T22:48:05.238692+0000","last_active":"2024-09-20T22:48:08.199047+0000","last_peered":"2024-09-20T22:48:08.199047+0000","last_clean":"2024-09-20T22:48:08.199047+0000","last_became_active":"2024-09-20T22:48:05.238334+0000","last_became_peered":"2024-09-20T22:48:05.238334+0000","last_unstale":"2024-09-20T22:48:08.199047+0000","last_undegraded":"2024-09-20T22:48:08.199047+0000","last_fullsized":"2024-09-20T22:48:08.199047+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:46:29.466361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:50.174861+0000","last_clean_scrub_stamp":"2024-09-20T22:46:29.466361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:12.326193+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":702,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.206629+0000","last_change":"2024-09-20T22:47:49.215237+0000","last_active":"2024-09-20T22:48:08.206629+0000","last_peered":"2024-09-20T22:48:08.206629+0000","last_clean":"2024-09-20T22:48:08.206629+0000","last_became_active":"2024-09-20T22:47:49.214487+0000","last_became_peered":"2024-09-20T22:47:49.214487+0000","last_unstale":"2024-09-20T22:48:08.206629+0000","last_undegraded":"2024-09-20T22:48:08.206629+0000","last_fullsized":"2024-09-20T22:48:08.206629+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:45:00.006909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:45:00.006909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"408'91","reported_seq":127,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.239427+0000","last_change":"2024-09-20T22:48:04.216652+0000","last_active":"2024-09-20T22:48:08.239427+0000","last_peered":"2024-09-20T22:48:08.239427+0000","last_clean":"2024-09-20T22:48:08.239427+0000","last_became_active":"2024-09-20T22:47:49.226617+0000","last_became_peered":"2024-09-20T22:47:49.226617+0000","last_unstale":"2024-09-20T22:48:08.239427+0000","last_undegraded":"2024-09-20T22:48:08.239427+0000","last_fullsized":"2024-09-20T22:48:08.239427+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":5,"last_scrub":"255'21","last_scrub_stamp":"2024-09-20T22:45:35.767459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:45:35.767459+0000","objects_scrubbed":1,"log_size":91,"log_dups_size":0,"ondisk_log_size":91,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0040418980000000004,"stat_sum":{"num_bytes":10586804,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":54123,"num_write":25,"num_write_kb":11537,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":10261017,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"98","length":"1"},{"start":"a1","length":"2"}]},{"pgid":"4.10","version":"74'4","reported_seq":704,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.205482+0000","last_change":"2024-09-20T22:44:17.320732+0000","last_active":"2024-09-20T22:48:08.205482+0000","last_peered":"2024-09-20T22:48:08.205482+0000","last_clean":"2024-09-20T22:48:08.205482+0000","last_became_active":"2024-09-20T22:41:40.166123+0000","last_became_peered":"2024-09-20T22:41:40.166123+0000","last_unstale":"2024-09-20T22:48:08.205482+0000","last_undegraded":"2024-09-20T22:48:08.205482+0000","last_fullsized":"2024-09-20T22:48:08.205482+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'4","last_scrub_stamp":"2024-09-20T22:44:17.320608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:44:17.320608+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":49,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.198150+0000","last_change":"2024-09-20T22:47:49.232311+0000","last_active":"2024-09-20T22:48:08.198150+0000","last_peered":"2024-09-20T22:48:08.198150+0000","last_clean":"2024-09-20T22:48:08.198150+0000","last_became_active":"2024-09-20T22:47:49.231964+0000","last_became_peered":"2024-09-20T22:47:49.231964+0000","last_unstale":"2024-09-20T22:48:08.198150+0000","last_undegraded":"2024-09-20T22:48:08.198150+0000","last_fullsized":"2024-09-20T22:48:08.198150+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:44:58.632703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:44:58.632703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:11.508455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":734,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.204493+0000","last_change":"2024-09-20T22:48:05.245017+0000","last_active":"2024-09-20T22:48:08.204493+0000","last_peered":"2024-09-20T22:48:08.204493+0000","last_clean":"2024-09-20T22:48:08.204493+0000","last_became_active":"2024-09-20T22:48:05.244659+0000","last_became_peered":"2024-09-20T22:48:05.244659+0000","last_unstale":"2024-09-20T22:48:08.204493+0000","last_undegraded":"2024-09-20T22:48:08.204493+0000","last_fullsized":"2024-09-20T22:48:08.204493+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:47:37.826041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:47:37.826041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:53.997023+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":710,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.200372+0000","last_change":"2024-09-20T22:48:05.238107+0000","last_active":"2024-09-20T22:48:08.200372+0000","last_peered":"2024-09-20T22:48:08.200372+0000","last_clean":"2024-09-20T22:48:08.200372+0000","last_became_active":"2024-09-20T22:48:05.237765+0000","last_became_peered":"2024-09-20T22:48:05.237765+0000","last_unstale":"2024-09-20T22:48:08.200372+0000","last_undegraded":"2024-09-20T22:48:08.200372+0000","last_fullsized":"2024-09-20T22:48:08.200372+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:45:19.352162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:10.233487+0000","last_clean_scrub_stamp":"2024-09-20T22:45:19.352162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:12.325974+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":710,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.202847+0000","last_change":"2024-09-20T22:48:05.249807+0000","last_active":"2024-09-20T22:48:08.202847+0000","last_peered":"2024-09-20T22:48:08.202847+0000","last_clean":"2024-09-20T22:48:08.202847+0000","last_became_active":"2024-09-20T22:48:05.249492+0000","last_became_peered":"2024-09-20T22:48:05.249492+0000","last_unstale":"2024-09-20T22:48:08.202847+0000","last_undegraded":"2024-09-20T22:48:08.202847+0000","last_fullsized":"2024-09-20T22:48:08.202847+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:47:19.596232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:47:19.596232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:26.387650+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":728,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.205256+0000","last_change":"2024-09-20T22:47:49.218136+0000","last_active":"2024-09-20T22:48:08.205256+0000","last_peered":"2024-09-20T22:48:08.205256+0000","last_clean":"2024-09-20T22:48:08.205256+0000","last_became_active":"2024-09-20T22:47:49.217064+0000","last_became_peered":"2024-09-20T22:47:49.217064+0000","last_unstale":"2024-09-20T22:48:08.205256+0000","last_undegraded":"2024-09-20T22:48:08.205256+0000","last_fullsized":"2024-09-20T22:48:08.205256+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:44:10.267152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:44:10.267152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"74'11","reported_seq":737,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.204990+0000","last_change":"2024-09-20T22:47:38.862277+0000","last_active":"2024-09-20T22:48:08.204990+0000","last_peered":"2024-09-20T22:48:08.204990+0000","last_clean":"2024-09-20T22:48:08.204990+0000","last_became_active":"2024-09-20T22:41:40.171256+0000","last_became_peered":"2024-09-20T22:41:40.171256+0000","last_unstale":"2024-09-20T22:48:08.204990+0000","last_undegraded":"2024-09-20T22:48:08.204990+0000","last_fullsized":"2024-09-20T22:48:08.204990+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-09-20T22:47:38.862194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:47:38.862194+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:49:03.255502+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":49,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-09-20T22:48:08.199157+0000","last_change":"2024-09-20T22:47:49.221413+0000","last_active":"2024-09-20T22:48:08.199157+0000","last_peered":"2024-09-20T22:48:08.199157+0000","last_clean":"2024-09-20T22:48:08.199157+0000","last_became_active":"2024-09-20T22:47:49.220971+0000","last_became_peered":"2024-09-20T22:47:49.220971+0000","last_unstale":"2024-09-20T22:48:08.199157+0000","last_undegraded":"2024-09-20T22:48:08.199157+0000","last_fullsized":"2024-09-20T22:48:08.199157+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:46:20.485453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:51.667341+0000","last_clean_scrub_stamp":"2024-09-20T22:46:20.485453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:11.508624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":26,"stat_sum":{"num_bytes":336068697,"num_objects":170,"num_object_clones":122,"num_object_copies":510,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":170,"num_whiteouts":13,"num_read":2757,"num_read_kb":2871277,"num_write":1858,"num_write_kb":704508,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":482,"num_bytes_recovered":237661293,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":794284032,"data_stored":1113618378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5820,"ondisk_log_size":5820,"up":78,"acting":78,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":82,"ondisk_log_size":82,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":682,"num_read_kb":439,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":238,"num_bytes_recovered":5126,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":434,"ondisk_log_size":434,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":1034,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":381,"num_read_kb":376,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":384,"ondisk_log_size":384,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070546,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":124556,"kb_used_data":93364,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93620596,"statfs":{"total":95995035648,"available":95867490304,"internally_reserved":0,"allocated":95604736,"data_stored":141587429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430296,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":158528,"kb_used_data":127336,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93586624,"statfs":{"total":95995035648,"available":95832702976,"internally_reserved":0,"allocated":130392064,"data_stored":192688143,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691933,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":111228,"kb_used_data":84196,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93633924,"statfs":{"total":95995035648,"available":95881138176,"internally_reserved":0,"allocated":86216704,"data_stored":117521242,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986274,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":84316,"kb_used_data":48960,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93660836,"statfs":{"total":95995035648,"available":95908696064,"internally_reserved":0,"allocated":50135040,"data_stored":61846494,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182504,"num_pgs":74,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":138036,"kb_used_data":106844,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93607116,"statfs":{"total":95995035648,"available":95853686784,"internally_reserved":0,"allocated":109408256,"data_stored":155957413,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411437,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":147308,"kb_used_data":116112,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93597844,"statfs":{"total":95995035648,"available":95844192256,"internally_reserved":0,"allocated":118898688,"data_stored":161511319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574962,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":113736,"kb_used_data":86708,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93631416,"statfs":{"total":95995035648,"available":95878569984,"internally_reserved":0,"allocated":88788992,"data_stored":125576055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738487,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":164640,"kb_used_data":137604,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93580512,"statfs":{"total":95995035648,"available":95826444288,"internally_reserved":0,"allocated":140906496,"data_stored":179877515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":138125312,"data_stored":178205223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":85979136,"data_stored":123903428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":110452736,"data_stored":149908867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":109367296,"data_stored":155550120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":47362048,"data_stored":60181988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":82853888,"data_stored":115257914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":127643648,"data_stored":191023249,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":92499968,"data_stored":139587589,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-20T22:48:13.892 INFO:tasks.cephadm.ceph_manager.ceph:decrease pool size by 10 2024-09-20T22:48:13.892 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd pool set unique_pool_0 pg_num 16 2024-09-20T22:48:13.922 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:48:13.956 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:13.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:13.957 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:48:14.058 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:48:14.081 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:14.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:14.082 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:48:14.183 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:48:14.205 INFO:tasks.rados.rados.0.smithi084.stdout:1594: done (0 left) 2024-09-20T22:48:14.205 INFO:tasks.rados.rados.0.smithi084.stdout:1595: snap_remove snap 161 2024-09-20T22:48:14.208 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:14.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:14.209 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:48:14.259 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd unset noscrub 2024-09-20T22:48:14.309 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:48:14.331 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:14.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:14.332 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:48:14.339 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:48:14.339 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:48:14.372 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:48:14.373 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:48:14.374 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:48:14.406 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:48:14.406 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:48:14.432 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:48:14.438 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-20T22:48:14.438 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T22:48:14.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:14.454 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:14.454 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:48:14.471 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:48:14.471 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:48:14.496 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:48:14.496 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:48:14.515 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:48:14.515 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:48:14.534 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:48:14.534 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:48:14.556 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:48:14.557 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.4 dump_blocked_ops 2024-09-20T22:48:14.618 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:14.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:14.619 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:48:14.719 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:48:14.741 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:14.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:14.742 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:48:14.843 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:48:14.844 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:48:14.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:14.866 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:14.866 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:48:14.967 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:48:14.989 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:14.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:14.990 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:48:15.090 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:48:15.112 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:15.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:15.113 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:48:15.211 INFO:tasks.rados.rados.0.smithi084.stdout:1595: done (0 left) 2024-09-20T22:48:15.212 INFO:tasks.rados.rados.0.smithi084.stdout:1596: write oid 1 current snap is 170 2024-09-20T22:48:15.212 INFO:tasks.rados.rados.0.smithi084.stdout:1596: seq_num 563 ranges {662341=575309,1878774=498528,2894439=457555} 2024-09-20T22:48:15.214 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:48:15.218 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:15 smithi154 ceph-mon[33264]: pgmap v522: 168 pgs: 2 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 160 active+clean; 347 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 27 MiB/s rd, 6.2 MiB/s wr, 35 op/s 2024-09-20T22:48:15.218 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:15 smithi154 ceph-mon[33264]: osdmap e414: 8 total, 8 up, 8 in 2024-09-20T22:48:15.218 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:15 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/795350581' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:48:15.219 INFO:tasks.rados.rados.0.smithi084.stdout:1596: writing smithi08477203-1 from 662341 to 1237650 tid 1 2024-09-20T22:48:15.226 INFO:tasks.rados.rados.0.smithi084.stdout:1596: writing smithi08477203-1 from 1878774 to 2377302 tid 2 2024-09-20T22:48:15.233 INFO:tasks.rados.rados.0.smithi084.stdout:1596: writing smithi08477203-1 from 2894439 to 3351994 tid 3 2024-09-20T22:48:15.233 INFO:tasks.rados.rados.0.smithi084.stdout:1597: rollback oid 10 current snap is 170 2024-09-20T22:48:15.233 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 10 to 134 2024-09-20T22:48:15.233 INFO:tasks.rados.rados.0.smithi084.stdout:1596: finishing write tid 1 to smithi08477203-1 2024-09-20T22:48:15.233 INFO:tasks.rados.rados.0.smithi084.stdout:1596: finishing write tid 2 to smithi08477203-1 2024-09-20T22:48:15.233 INFO:tasks.rados.rados.0.smithi084.stdout:1598: read oid 46 snap -1 2024-09-20T22:48:15.233 INFO:tasks.rados.rados.0.smithi084.stdout:1598: expect (ObjNum 470 snap 130 seq_num 470) 2024-09-20T22:48:15.233 INFO:tasks.rados.rados.0.smithi084.stdout:1599: write oid 19 current snap is 170 2024-09-20T22:48:15.233 INFO:tasks.rados.rados.0.smithi084.stdout:1599: seq_num 564 ranges {679148=650558,1937256=642186,3321862=157286} 2024-09-20T22:48:15.239 INFO:tasks.rados.rados.0.smithi084.stdout:1599: writing smithi08477203-19 from 679148 to 1329706 tid 1 2024-09-20T22:48:15.243 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:15.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:15.244 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:48:15.247 INFO:tasks.rados.rados.0.smithi084.stdout:1599: writing smithi08477203-19 from 1937256 to 2579442 tid 2 2024-09-20T22:48:15.252 INFO:tasks.rados.rados.0.smithi084.stdout:1599: writing smithi08477203-19 from 3321862 to 3479148 tid 3 2024-09-20T22:48:15.252 INFO:tasks.rados.rados.0.smithi084.stdout:1600: read oid 32 snap -1 2024-09-20T22:48:15.252 INFO:tasks.rados.rados.0.smithi084.stdout:1600: expect (ObjNum 551 snap 168 seq_num 551) 2024-09-20T22:48:15.252 INFO:tasks.rados.rados.0.smithi084.stdout:1601: write oid 5 current snap is 170 2024-09-20T22:48:15.252 INFO:tasks.rados.rados.0.smithi084.stdout:1596: finishing write tid 3 to smithi08477203-1 2024-09-20T22:48:15.253 INFO:tasks.rados.rados.0.smithi084.stdout:1601: seq_num 565 ranges {695955=725807,1879385=502197,2949285=480099} 2024-09-20T22:48:15.260 INFO:tasks.rados.rados.0.smithi084.stdout:1601: writing smithi08477203-5 from 695955 to 1421762 tid 1 2024-09-20T22:48:15.265 INFO:tasks.rados.rados.0.smithi084.stdout:1601: writing smithi08477203-5 from 1879385 to 2381582 tid 2 2024-09-20T22:48:15.271 INFO:tasks.rados.rados.0.smithi084.stdout:1601: writing smithi08477203-5 from 2949285 to 3429384 tid 3 2024-09-20T22:48:15.272 INFO:tasks.rados.rados.0.smithi084.stdout:1602: read oid 23 snap -1 2024-09-20T22:48:15.272 INFO:tasks.rados.rados.0.smithi084.stdout:1602: expect (ObjNum 440 snap 115 seq_num 440) 2024-09-20T22:48:15.272 INFO:tasks.rados.rados.0.smithi084.stdout:1603: write oid 47 current snap is 170 2024-09-20T22:48:15.272 INFO:tasks.rados.rados.0.smithi084.stdout:1596: finishing write tid 4 to smithi08477203-1 2024-09-20T22:48:15.272 INFO:tasks.rados.rados.0.smithi084.stdout:1596: finishing write tid 5 to smithi08477203-1 2024-09-20T22:48:15.272 INFO:tasks.rados.rados.0.smithi084.stdout:1596: oid 1 updating version 0 to 549 2024-09-20T22:48:15.272 INFO:tasks.rados.rados.0.smithi084.stdout:1596: oid 1 version 549 is already newer than 546 2024-09-20T22:48:15.272 INFO:tasks.rados.rados.0.smithi084.stdout:1596: oid 1 version 549 is already newer than 548 2024-09-20T22:48:15.272 INFO:tasks.rados.rados.0.smithi084.stdout:1596: oid 1 version 549 is already newer than 547 2024-09-20T22:48:15.272 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 1 v 549 (ObjNum 563 snap 170 seq_num 563) dirty exists 2024-09-20T22:48:15.272 INFO:tasks.rados.rados.0.smithi084.stdout:1596: left oid 1 (ObjNum 563 snap 170 seq_num 563) 2024-09-20T22:48:15.272 INFO:tasks.rados.rados.0.smithi084.stdout:1597: finishing rollback tid 0 to smithi08477203-10 2024-09-20T22:48:15.272 INFO:tasks.rados.rados.0.smithi084.stdout:1603: seq_num 566 ranges {712762=401056,1537867=645855,2860355=652407} 2024-09-20T22:48:15.278 INFO:tasks.rados.rados.0.smithi084.stdout:1603: writing smithi08477203-47 from 712762 to 1113818 tid 1 2024-09-20T22:48:15.283 INFO:tasks.rados.rados.0.smithi084.stdout:1603: writing smithi08477203-47 from 1537867 to 2183722 tid 2 2024-09-20T22:48:15.291 INFO:tasks.rados.rados.0.smithi084.stdout:1603: writing smithi08477203-47 from 2860355 to 3512762 tid 3 2024-09-20T22:48:15.291 INFO:tasks.rados.rados.0.smithi084.stdout:1596: done (7 left) 2024-09-20T22:48:15.291 INFO:tasks.rados.rados.0.smithi084.stdout:1604: snap_create 2024-09-20T22:48:15.291 INFO:tasks.rados.rados.0.smithi084.stdout:1597: finishing rollback tid 1 to smithi08477203-10 2024-09-20T22:48:15.292 INFO:tasks.rados.rados.0.smithi084.stdout:1597: finishing rollback tid 2 to smithi08477203-10 2024-09-20T22:48:15.292 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 10 v 552 (ObjNum 419 snap 105 seq_num 419) dirty exists 2024-09-20T22:48:15.292 INFO:tasks.rados.rados.0.smithi084.stdout:1598: expect (ObjNum 470 snap 130 seq_num 470) 2024-09-20T22:48:15.331 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-20T22:48:15.331 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [], 2024-09-20T22:48:15.331 INFO:teuthology.orchestra.run.smithi154.stdout: "complaint_time": 30, 2024-09-20T22:48:15.331 INFO:teuthology.orchestra.run.smithi154.stdout: "num_blocked_ops": 0 2024-09-20T22:48:15.331 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-20T22:48:15.345 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:48:15.367 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:15.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:15.368 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:48:15.425 INFO:tasks.rados.rados.0.smithi084.stdout:1599: finishing write tid 1 to smithi08477203-19 2024-09-20T22:48:15.426 INFO:tasks.rados.rados.0.smithi084.stdout:1599: finishing write tid 2 to smithi08477203-19 2024-09-20T22:48:15.426 INFO:tasks.rados.rados.0.smithi084.stdout:1599: finishing write tid 3 to smithi08477203-19 2024-09-20T22:48:15.426 INFO:tasks.rados.rados.0.smithi084.stdout:1599: finishing write tid 4 to smithi08477203-19 2024-09-20T22:48:15.426 INFO:tasks.rados.rados.0.smithi084.stdout:1599: finishing write tid 5 to smithi08477203-19 2024-09-20T22:48:15.426 INFO:tasks.rados.rados.0.smithi084.stdout:1599: oid 19 updating version 0 to 888 2024-09-20T22:48:15.426 INFO:tasks.rados.rados.0.smithi084.stdout:1599: oid 19 updating version 888 to 890 2024-09-20T22:48:15.426 INFO:tasks.rados.rados.0.smithi084.stdout:1599: oid 19 version 890 is already newer than 887 2024-09-20T22:48:15.426 INFO:tasks.rados.rados.0.smithi084.stdout:1599: oid 19 version 890 is already newer than 889 2024-09-20T22:48:15.426 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 19 v 890 (ObjNum 564 snap 170 seq_num 564) dirty exists 2024-09-20T22:48:15.426 INFO:tasks.rados.rados.0.smithi084.stdout:1599: left oid 19 (ObjNum 564 snap 170 seq_num 564) 2024-09-20T22:48:15.426 INFO:tasks.rados.rados.0.smithi084.stdout:1600: expect (ObjNum 551 snap 168 seq_num 551) 2024-09-20T22:48:15.470 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:48:15.491 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:15.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:15.492 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:48:15.525 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:15 smithi084 ceph-mon[40024]: pgmap v522: 168 pgs: 2 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 160 active+clean; 347 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 27 MiB/s rd, 6.2 MiB/s wr, 35 op/s 2024-09-20T22:48:15.525 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:15 smithi084 ceph-mon[40024]: osdmap e414: 8 total, 8 up, 8 in 2024-09-20T22:48:15.526 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:15 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/795350581' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:48:15.526 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:15 smithi084 ceph-mon[29774]: pgmap v522: 168 pgs: 2 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 160 active+clean; 347 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 27 MiB/s rd, 6.2 MiB/s wr, 35 op/s 2024-09-20T22:48:15.526 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:15 smithi084 ceph-mon[29774]: osdmap e414: 8 total, 8 up, 8 in 2024-09-20T22:48:15.526 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:15 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/795350581' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:48:15.593 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:48:15.611 INFO:tasks.rados.rados.0.smithi084.stdout:1601: finishing write tid 1 to smithi08477203-5 2024-09-20T22:48:15.611 INFO:tasks.rados.rados.0.smithi084.stdout:1601: finishing write tid 2 to smithi08477203-5 2024-09-20T22:48:15.611 INFO:tasks.rados.rados.0.smithi084.stdout:1601: finishing write tid 3 to smithi08477203-5 2024-09-20T22:48:15.612 INFO:tasks.rados.rados.0.smithi084.stdout:1601: finishing write tid 4 to smithi08477203-5 2024-09-20T22:48:15.612 INFO:tasks.rados.rados.0.smithi084.stdout:1601: finishing write tid 5 to smithi08477203-5 2024-09-20T22:48:15.612 INFO:tasks.rados.rados.0.smithi084.stdout:1601: oid 5 updating version 0 to 407 2024-09-20T22:48:15.612 INFO:tasks.rados.rados.0.smithi084.stdout:1601: oid 5 updating version 407 to 410 2024-09-20T22:48:15.612 INFO:tasks.rados.rados.0.smithi084.stdout:1601: oid 5 version 410 is already newer than 409 2024-09-20T22:48:15.612 INFO:tasks.rados.rados.0.smithi084.stdout:1601: oid 5 version 410 is already newer than 408 2024-09-20T22:48:15.612 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 5 v 410 (ObjNum 565 snap 170 seq_num 565) dirty exists 2024-09-20T22:48:15.612 INFO:tasks.rados.rados.0.smithi084.stdout:1601: left oid 5 (ObjNum 565 snap 170 seq_num 565) 2024-09-20T22:48:15.612 INFO:tasks.rados.rados.0.smithi084.stdout:1602: expect (ObjNum 440 snap 115 seq_num 440) 2024-09-20T22:48:15.614 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:15.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:15.615 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:48:15.716 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:48:15.737 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:15.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:15.738 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:48:15.839 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:48:15.860 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:48:15.861 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:15.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:15.862 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:48:15.879 INFO:tasks.rados.rados.0.smithi084.stdout:1603: finishing write tid 1 to smithi08477203-47 2024-09-20T22:48:15.879 INFO:tasks.rados.rados.0.smithi084.stdout:1603: finishing write tid 2 to smithi08477203-47 2024-09-20T22:48:15.879 INFO:tasks.rados.rados.0.smithi084.stdout:1603: finishing write tid 3 to smithi08477203-47 2024-09-20T22:48:15.879 INFO:tasks.rados.rados.0.smithi084.stdout:1603: finishing write tid 4 to smithi08477203-47 2024-09-20T22:48:15.879 INFO:tasks.rados.rados.0.smithi084.stdout:1603: finishing write tid 5 to smithi08477203-47 2024-09-20T22:48:15.879 INFO:tasks.rados.rados.0.smithi084.stdout:1603: oid 47 updating version 0 to 419 2024-09-20T22:48:15.879 INFO:tasks.rados.rados.0.smithi084.stdout:1603: oid 47 version 419 is already newer than 418 2024-09-20T22:48:15.879 INFO:tasks.rados.rados.0.smithi084.stdout:1603: oid 47 updating version 419 to 420 2024-09-20T22:48:15.879 INFO:tasks.rados.rados.0.smithi084.stdout:1603: oid 47 updating version 420 to 421 2024-09-20T22:48:15.880 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 47 v 421 (ObjNum 566 snap 170 seq_num 566) dirty exists 2024-09-20T22:48:15.880 INFO:tasks.rados.rados.0.smithi084.stdout:1603: left oid 47 (ObjNum 566 snap 170 seq_num 566) 2024-09-20T22:48:15.962 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:48:15.984 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:15.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:15.985 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:48:16.008 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.4 dump_historic_ops 2024-09-20T22:48:16.086 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:48:16.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:16.107 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:16.107 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:48:16.208 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:48:16.217 INFO:tasks.rados.rados.0.smithi084.stdout:1597: done (7 left) 2024-09-20T22:48:16.217 INFO:tasks.rados.rados.0.smithi084.stdout:1598: done (6 left) 2024-09-20T22:48:16.217 INFO:tasks.rados.rados.0.smithi084.stdout:1599: done (5 left) 2024-09-20T22:48:16.217 INFO:tasks.rados.rados.0.smithi084.stdout:1600: done (4 left) 2024-09-20T22:48:16.217 INFO:tasks.rados.rados.0.smithi084.stdout:1601: done (3 left) 2024-09-20T22:48:16.217 INFO:tasks.rados.rados.0.smithi084.stdout:1602: done (2 left) 2024-09-20T22:48:16.218 INFO:tasks.rados.rados.0.smithi084.stdout:1603: done (1 left) 2024-09-20T22:48:16.218 INFO:tasks.rados.rados.0.smithi084.stdout:1604: done (0 left) 2024-09-20T22:48:16.218 INFO:tasks.rados.rados.0.smithi084.stdout:1605: write (excl) oid 19 current snap is 171 2024-09-20T22:48:16.218 INFO:tasks.rados.rados.0.smithi084.stdout:1605: seq_num 567 ranges {729569=476305,1996349=505866,3004131=525438} 2024-09-20T22:48:16.218 INFO:teuthology.orchestra.run.smithi084.stderr:set pool 7 pg_num to 16 2024-09-20T22:48:16.224 INFO:tasks.rados.rados.0.smithi084.stdout:1605: writing smithi08477203-19 from 729569 to 1205874 tid 1 2024-09-20T22:48:16.231 INFO:tasks.rados.rados.0.smithi084.stdout:1605: writing smithi08477203-19 from 1996349 to 2502215 tid 2 2024-09-20T22:48:16.233 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:16.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:16.234 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:48:16.237 INFO:tasks.rados.rados.0.smithi084.stdout:1605: writing smithi08477203-19 from 3004131 to 3529569 tid 3 2024-09-20T22:48:16.237 INFO:tasks.rados.rados.0.smithi084.stdout:1606: snap_remove snap 144 2024-09-20T22:48:16.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:16 smithi154 ceph-mon[33264]: osdmap e415: 8 total, 8 up, 8 in 2024-09-20T22:48:16.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:16 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/762648486' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "16"}]: dispatch 2024-09-20T22:48:16.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:16 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "16"}]: dispatch 2024-09-20T22:48:16.256 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:48:16.335 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:48:16.357 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:16.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:16.359 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:48:16.459 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:48:16.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:16.485 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:16.485 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:48:16.586 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:48:16.608 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:16.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:16.609 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:48:16.614 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:16 smithi084 ceph-mon[40024]: osdmap e415: 8 total, 8 up, 8 in 2024-09-20T22:48:16.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:16 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/762648486' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "16"}]: dispatch 2024-09-20T22:48:16.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:16 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "16"}]: dispatch 2024-09-20T22:48:16.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:16 smithi084 ceph-mon[29774]: osdmap e415: 8 total, 8 up, 8 in 2024-09-20T22:48:16.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:16 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/762648486' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "16"}]: dispatch 2024-09-20T22:48:16.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:16 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "16"}]: dispatch 2024-09-20T22:48:16.694 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-20T22:48:16.694 INFO:teuthology.orchestra.run.smithi154.stdout: "size": 20, 2024-09-20T22:48:16.694 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 600, 2024-09-20T22:48:16.694 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [ 2024-09-20T22:48:16.694 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.694 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(osd.0.0:116 7.9 e106/94 7:9eac4560:::smithi08477203-30:5 v 106'27, mlcod=105'24)", 2024-09-20T22:48:16.694 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:05.532754+0000", 2024-09-20T22:48:16.694 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 311.15841327800001, 2024-09-20T22:48:16.694 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0023975530000000002, 2024-09-20T22:48:16.695 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:48:16.695 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:48:16.695 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:48:16.695 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:48:16.695 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.695 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:48:16.695 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:05.532754+0000", 2024-09-20T22:48:16.695 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:48:16.695 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.695 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.695 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:48:16.695 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:05.532754+0000", 2024-09-20T22:48:16.695 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:48:16.696 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.696 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.696 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:48:16.696 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:05.532754+0000", 2024-09-20T22:48:16.696 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.4999999999999999e-07 2024-09-20T22:48:16.696 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.696 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.696 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:48:16.696 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:05.532758+0000", 2024-09-20T22:48:16.696 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.0389999999999998e-06 2024-09-20T22:48:16.696 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.696 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.696 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:48:16.696 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:05.532776+0000", 2024-09-20T22:48:16.696 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.8045000000000001e-05 2024-09-20T22:48:16.697 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.697 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.697 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:48:16.697 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:05.532783+0000", 2024-09-20T22:48:16.697 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.3300000000000001e-06 2024-09-20T22:48:16.697 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.697 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.697 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:48:16.697 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:05.532851+0000", 2024-09-20T22:48:16.697 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.7939000000000001e-05 2024-09-20T22:48:16.697 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.697 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.697 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:48:16.697 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:05.532865+0000", 2024-09-20T22:48:16.698 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3416e-05 2024-09-20T22:48:16.698 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.698 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.698 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:48:16.698 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:05.535106+0000", 2024-09-20T22:48:16.698 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0022412159999999999 2024-09-20T22:48:16.698 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.698 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.698 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:48:16.698 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:05.535151+0000", 2024-09-20T22:48:16.698 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.5318000000000002e-05 2024-09-20T22:48:16.698 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:48:16.698 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:48:16.698 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:48:16.699 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.699 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.699 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:324 7.8 e106/94 7:12e9c8db:::smithi08477203-11:head v 106'10, mlcod=98'8)", 2024-09-20T22:48:16.699 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:05.548487+0000", 2024-09-20T22:48:16.699 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 311.142680267, 2024-09-20T22:48:16.699 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.002619522, 2024-09-20T22:48:16.699 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:48:16.699 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:48:16.699 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:48:16.699 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:48:16.699 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.699 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:48:16.699 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:05.548487+0000", 2024-09-20T22:48:16.699 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:48:16.700 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.700 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.700 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:48:16.700 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:05.548487+0000", 2024-09-20T22:48:16.700 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:48:16.700 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.700 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.700 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:48:16.700 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:05.548487+0000", 2024-09-20T22:48:16.700 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.7300000000000001e-07 2024-09-20T22:48:16.700 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.700 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.700 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:48:16.700 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:05.550209+0000", 2024-09-20T22:48:16.701 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0017213370000000001 2024-09-20T22:48:16.701 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.701 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.701 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:48:16.701 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:05.550218+0000", 2024-09-20T22:48:16.701 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.9189999999999994e-06 2024-09-20T22:48:16.701 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.701 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.701 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:48:16.701 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:05.550224+0000", 2024-09-20T22:48:16.701 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.5260000000000001e-06 2024-09-20T22:48:16.701 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.701 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.701 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:48:16.701 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:05.550267+0000", 2024-09-20T22:48:16.702 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.3231e-05 2024-09-20T22:48:16.702 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.702 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.702 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:48:16.702 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:05.550283+0000", 2024-09-20T22:48:16.702 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.5421999999999999e-05 2024-09-20T22:48:16.702 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.702 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.702 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:48:16.702 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:05.551070+0000", 2024-09-20T22:48:16.702 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00078768700000000002 2024-09-20T22:48:16.702 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.702 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.702 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:48:16.703 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:05.551106+0000", 2024-09-20T22:48:16.703 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.5926999999999997e-05 2024-09-20T22:48:16.703 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:48:16.703 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:48:16.703 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:48:16.703 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.703 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.703 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:359 7.9 e108/94 7:91f2d47c:::smithi08477203-8:head v 108'28, mlcod=105'25)", 2024-09-20T22:48:16.703 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:07.834938+0000", 2024-09-20T22:48:16.703 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 308.85622862000002, 2024-09-20T22:48:16.703 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0054540810000000004, 2024-09-20T22:48:16.703 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:48:16.703 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:48:16.704 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:48:16.704 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:48:16.704 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.704 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:48:16.704 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:07.834938+0000", 2024-09-20T22:48:16.704 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:48:16.704 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.704 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.704 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:48:16.704 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:07.834938+0000", 2024-09-20T22:48:16.704 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:48:16.704 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.704 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.704 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:48:16.705 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:07.834939+0000", 2024-09-20T22:48:16.705 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.2399999999999998e-07 2024-09-20T22:48:16.705 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.705 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.705 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:48:16.705 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:07.837461+0000", 2024-09-20T22:48:16.705 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0025222769999999999 2024-09-20T22:48:16.705 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.705 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.705 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:48:16.705 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:07.837473+0000", 2024-09-20T22:48:16.705 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1422e-05 2024-09-20T22:48:16.705 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.705 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.706 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:48:16.706 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:07.837479+0000", 2024-09-20T22:48:16.706 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.083e-06 2024-09-20T22:48:16.706 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.706 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.706 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:48:16.706 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:07.837561+0000", 2024-09-20T22:48:16.706 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.2243999999999994e-05 2024-09-20T22:48:16.706 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.706 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.706 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:48:16.706 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:07.837586+0000", 2024-09-20T22:48:16.706 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.4638999999999999e-05 2024-09-20T22:48:16.707 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.707 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.707 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:48:16.707 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:07.840355+0000", 2024-09-20T22:48:16.707 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.002769681 2024-09-20T22:48:16.707 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.707 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.707 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:48:16.707 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:07.840392+0000", 2024-09-20T22:48:16.707 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.7211e-05 2024-09-20T22:48:16.707 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:48:16.707 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:48:16.707 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:48:16.707 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.708 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.708 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:379 7.1 7:8422f82d:::smithi08477203-22:head [write 742876~447756,stat] snapc 8=[7] ondisk+write+known_if_redirected+supports_pool_eio e111)", 2024-09-20T22:48:16.708 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:10.210918+0000", 2024-09-20T22:48:16.708 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 306.48024869900001, 2024-09-20T22:48:16.708 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0030494419999999999, 2024-09-20T22:48:16.708 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:48:16.709 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:48:16.709 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:48:16.709 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:48:16.709 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:48:16.709 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:48:16.709 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 379 2024-09-20T22:48:16.709 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.709 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:48:16.709 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.709 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:48:16.709 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:10.210918+0000", 2024-09-20T22:48:16.710 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:48:16.710 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.710 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.710 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:48:16.710 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:10.210918+0000", 2024-09-20T22:48:16.710 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:48:16.710 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.710 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.710 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:48:16.710 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:10.210920+0000", 2024-09-20T22:48:16.710 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.8419999999999999e-06 2024-09-20T22:48:16.710 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.710 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.710 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:48:16.711 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:10.211774+0000", 2024-09-20T22:48:16.711 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00085377000000000001 2024-09-20T22:48:16.711 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.711 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.711 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:48:16.711 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:10.211791+0000", 2024-09-20T22:48:16.711 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.7186000000000001e-05 2024-09-20T22:48:16.711 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.711 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.711 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:48:16.711 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:10.211799+0000", 2024-09-20T22:48:16.711 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.9179999999999994e-06 2024-09-20T22:48:16.711 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.711 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.712 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:48:16.712 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:10.211870+0000", 2024-09-20T22:48:16.712 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.1271000000000003e-05 2024-09-20T22:48:16.712 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.712 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.712 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:48:16.712 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:10.211939+0000", 2024-09-20T22:48:16.712 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.8245999999999997e-05 2024-09-20T22:48:16.712 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.712 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.712 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 3,6", 2024-09-20T22:48:16.712 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:10.212067+0000", 2024-09-20T22:48:16.712 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000127995 2024-09-20T22:48:16.712 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.713 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.713 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:48:16.713 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:10.213450+0000", 2024-09-20T22:48:16.713 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001383205 2024-09-20T22:48:16.713 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.713 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.713 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:48:16.713 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:10.213457+0000", 2024-09-20T22:48:16.713 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.3159999999999999e-06 2024-09-20T22:48:16.713 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.713 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.713 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:48:16.713 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:10.213884+0000", 2024-09-20T22:48:16.713 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000426444 2024-09-20T22:48:16.714 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.714 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.714 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:48:16.714 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:10.213920+0000", 2024-09-20T22:48:16.714 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.6665000000000002e-05 2024-09-20T22:48:16.714 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.714 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.714 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:48:16.714 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:10.213968+0000", 2024-09-20T22:48:16.714 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.7583999999999997e-05 2024-09-20T22:48:16.714 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:48:16.714 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:48:16.714 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:48:16.715 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.715 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.715 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:397 7.9 e112/94 7:91f2d47c:::smithi08477203-8:head v 112'29, mlcod=106'27)", 2024-09-20T22:48:16.715 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:11.366618+0000", 2024-09-20T22:48:16.715 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 305.32454867500002, 2024-09-20T22:48:16.715 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.005929686, 2024-09-20T22:48:16.715 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:48:16.715 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:48:16.715 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:48:16.715 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:48:16.715 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.715 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:48:16.715 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:11.366618+0000", 2024-09-20T22:48:16.715 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:48:16.715 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.716 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.716 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:48:16.716 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:11.366618+0000", 2024-09-20T22:48:16.716 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:48:16.716 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.716 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.716 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:48:16.716 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:11.366619+0000", 2024-09-20T22:48:16.716 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.67e-07 2024-09-20T22:48:16.716 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.716 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.716 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:48:16.716 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:11.370298+0000", 2024-09-20T22:48:16.716 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0036796559999999999 2024-09-20T22:48:16.717 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.717 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.717 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:48:16.717 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:11.370311+0000", 2024-09-20T22:48:16.717 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3230999999999999e-05 2024-09-20T22:48:16.717 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.717 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.717 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:48:16.717 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:11.370318+0000", 2024-09-20T22:48:16.717 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.3990000000000001e-06 2024-09-20T22:48:16.717 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.717 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.717 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:48:16.717 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:11.370380+0000", 2024-09-20T22:48:16.718 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.2385999999999995e-05 2024-09-20T22:48:16.718 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.718 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.718 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:48:16.718 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:11.370403+0000", 2024-09-20T22:48:16.718 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.2707000000000001e-05 2024-09-20T22:48:16.718 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.718 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.718 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:48:16.718 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:11.372510+0000", 2024-09-20T22:48:16.718 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0021069370000000001 2024-09-20T22:48:16.718 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.718 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.718 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:48:16.718 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:11.372548+0000", 2024-09-20T22:48:16.719 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.8203e-05 2024-09-20T22:48:16.719 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:48:16.719 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:48:16.719 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:48:16.719 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.719 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.719 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:522 7.9 e124/94 7:9970b578:::smithi08477203-15:head v 124'40, mlcod=118'36)", 2024-09-20T22:48:16.719 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:23.137032+0000", 2024-09-20T22:48:16.719 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 293.55413528999998, 2024-09-20T22:48:16.719 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0095727380000000008, 2024-09-20T22:48:16.719 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:48:16.719 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:48:16.719 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:48:16.719 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:48:16.719 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.720 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:48:16.720 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:23.137032+0000", 2024-09-20T22:48:16.720 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:48:16.720 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.720 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.720 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:48:16.720 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:23.137032+0000", 2024-09-20T22:48:16.720 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:48:16.720 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.720 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.720 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:48:16.720 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:23.137033+0000", 2024-09-20T22:48:16.720 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.034e-06 2024-09-20T22:48:16.720 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.721 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.721 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:48:16.721 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:23.140240+0000", 2024-09-20T22:48:16.721 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0032075419999999999 2024-09-20T22:48:16.721 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.721 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.721 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:48:16.721 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:23.140261+0000", 2024-09-20T22:48:16.721 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.1155999999999999e-05 2024-09-20T22:48:16.721 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.721 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.721 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:48:16.721 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:23.140271+0000", 2024-09-20T22:48:16.721 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.6970000000000005e-06 2024-09-20T22:48:16.721 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.722 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.722 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:48:16.722 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:23.140339+0000", 2024-09-20T22:48:16.722 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.7964000000000002e-05 2024-09-20T22:48:16.722 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.722 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.722 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:48:16.722 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:23.140361+0000", 2024-09-20T22:48:16.722 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.196e-05 2024-09-20T22:48:16.722 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.722 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.722 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:48:16.722 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:23.146563+0000", 2024-09-20T22:48:16.722 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0062014879999999998 2024-09-20T22:48:16.723 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.723 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.723 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:48:16.723 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:23.146604+0000", 2024-09-20T22:48:16.723 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.1897000000000002e-05 2024-09-20T22:48:16.723 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:48:16.723 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:48:16.723 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:48:16.723 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.723 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.723 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:745 7.1 7:8422f82d:::smithi08477203-22:head [append 0~0] snapc 18=[17] ondisk+write+known_if_redirected+supports_pool_eio e143)", 2024-09-20T22:48:16.723 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:42.616952+0000", 2024-09-20T22:48:16.723 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 274.07421489799998, 2024-09-20T22:48:16.723 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0025256609999999998, 2024-09-20T22:48:16.724 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:48:16.724 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:48:16.724 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:48:16.724 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:48:16.724 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:48:16.724 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:48:16.724 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 745 2024-09-20T22:48:16.724 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.724 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:48:16.724 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.724 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:48:16.724 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.616952+0000", 2024-09-20T22:48:16.724 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:48:16.724 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.724 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.725 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:48:16.725 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.616952+0000", 2024-09-20T22:48:16.725 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:48:16.725 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.725 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.725 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:48:16.725 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.616954+0000", 2024-09-20T22:48:16.725 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.449e-06 2024-09-20T22:48:16.725 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.725 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.725 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:48:16.725 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.616961+0000", 2024-09-20T22:48:16.725 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.1250000000000004e-06 2024-09-20T22:48:16.725 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.725 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.726 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:48:16.726 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.616975+0000", 2024-09-20T22:48:16.726 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4669e-05 2024-09-20T22:48:16.726 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.726 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.726 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:48:16.726 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.616983+0000", 2024-09-20T22:48:16.726 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.3769999999999998e-06 2024-09-20T22:48:16.726 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.726 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.726 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:48:16.726 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.617040+0000", 2024-09-20T22:48:16.726 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.7147e-05 2024-09-20T22:48:16.726 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.727 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.727 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:48:16.727 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.617152+0000", 2024-09-20T22:48:16.727 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000111809 2024-09-20T22:48:16.727 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.727 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.727 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 3,6", 2024-09-20T22:48:16.727 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.617272+0000", 2024-09-20T22:48:16.727 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000120677 2024-09-20T22:48:16.727 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.727 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.727 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:48:16.727 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.619251+0000", 2024-09-20T22:48:16.727 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001978914 2024-09-20T22:48:16.727 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.728 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.728 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:48:16.728 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.619298+0000", 2024-09-20T22:48:16.728 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.7067000000000002e-05 2024-09-20T22:48:16.728 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.728 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.728 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:48:16.728 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.619391+0000", 2024-09-20T22:48:16.728 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.2793000000000004e-05 2024-09-20T22:48:16.728 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.728 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.728 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:48:16.728 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.619425+0000", 2024-09-20T22:48:16.728 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.3626e-05 2024-09-20T22:48:16.729 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.729 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.729 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:48:16.729 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.619478+0000", 2024-09-20T22:48:16.729 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.3007999999999998e-05 2024-09-20T22:48:16.729 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:48:16.729 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:48:16.729 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:48:16.729 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.729 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.729 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:746 7.1 7:8422f82d:::smithi08477203-22:head [rollback 17] snapc 18=[17] ondisk+write+known_if_redirected+supports_pool_eio e143)", 2024-09-20T22:48:16.729 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:42.617006+0000", 2024-09-20T22:48:16.729 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 274.074161371, 2024-09-20T22:48:16.729 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0025901660000000001, 2024-09-20T22:48:16.730 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:48:16.730 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:48:16.730 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:48:16.730 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:48:16.730 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:48:16.730 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:48:16.730 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 746 2024-09-20T22:48:16.730 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.730 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:48:16.730 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.730 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:48:16.730 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.617006+0000", 2024-09-20T22:48:16.730 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:48:16.730 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.731 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.731 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:48:16.731 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.617006+0000", 2024-09-20T22:48:16.731 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:48:16.731 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.731 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.731 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:48:16.731 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.617006+0000", 2024-09-20T22:48:16.731 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.4499999999999997e-07 2024-09-20T22:48:16.731 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.731 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.731 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:48:16.731 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.617009+0000", 2024-09-20T22:48:16.731 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.0120000000000001e-06 2024-09-20T22:48:16.732 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.732 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.732 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:48:16.732 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.617022+0000", 2024-09-20T22:48:16.732 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2468e-05 2024-09-20T22:48:16.732 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.732 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.732 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:48:16.732 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.617028+0000", 2024-09-20T22:48:16.732 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.9190000000000004e-06 2024-09-20T22:48:16.732 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.732 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.732 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:48:16.732 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.617952+0000", 2024-09-20T22:48:16.733 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000924488 2024-09-20T22:48:16.733 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.733 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.733 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:48:16.733 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.618027+0000", 2024-09-20T22:48:16.733 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.5164000000000001e-05 2024-09-20T22:48:16.733 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.733 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.733 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 3,6", 2024-09-20T22:48:16.733 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.618216+0000", 2024-09-20T22:48:16.733 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00018830799999999999 2024-09-20T22:48:16.733 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.733 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.733 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:48:16.733 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.619377+0000", 2024-09-20T22:48:16.734 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0011610909999999999 2024-09-20T22:48:16.734 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.734 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.734 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:48:16.734 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.619527+0000", 2024-09-20T22:48:16.734 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00015048500000000001 2024-09-20T22:48:16.734 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.734 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.734 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:48:16.734 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.619544+0000", 2024-09-20T22:48:16.734 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.6691e-05 2024-09-20T22:48:16.734 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.734 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.734 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:48:16.735 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.619573+0000", 2024-09-20T22:48:16.735 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.8895e-05 2024-09-20T22:48:16.735 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.735 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.735 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:48:16.735 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.619596+0000", 2024-09-20T22:48:16.735 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.3099999999999999e-05 2024-09-20T22:48:16.735 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:48:16.735 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:48:16.735 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:48:16.735 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.735 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.735 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:747 7.1 7:8422f82d:::smithi08477203-22:head [append 0~0] snapc 18=[17] ondisk+write+known_if_redirected+supports_pool_eio e143)", 2024-09-20T22:48:16.735 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:42.617038+0000", 2024-09-20T22:48:16.735 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 274.07412916099997, 2024-09-20T22:48:16.736 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0026729190000000002, 2024-09-20T22:48:16.736 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:48:16.736 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:48:16.736 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:48:16.736 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:48:16.736 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:48:16.736 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:48:16.736 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 747 2024-09-20T22:48:16.736 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.736 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:48:16.736 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.736 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:48:16.736 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.617038+0000", 2024-09-20T22:48:16.736 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:48:16.737 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.737 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.737 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:48:16.737 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.617038+0000", 2024-09-20T22:48:16.737 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:48:16.737 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.737 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.737 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:48:16.737 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.617038+0000", 2024-09-20T22:48:16.737 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.8500000000000002e-07 2024-09-20T22:48:16.737 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.737 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.737 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:48:16.737 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.617041+0000", 2024-09-20T22:48:16.738 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.6800000000000002e-06 2024-09-20T22:48:16.738 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.738 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.738 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:48:16.738 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.617047+0000", 2024-09-20T22:48:16.738 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.7699999999999998e-06 2024-09-20T22:48:16.738 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.738 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.738 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:48:16.738 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.617052+0000", 2024-09-20T22:48:16.738 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.9069999999999998e-06 2024-09-20T22:48:16.738 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.738 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.738 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:48:16.738 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.618852+0000", 2024-09-20T22:48:16.739 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001800458 2024-09-20T22:48:16.739 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.739 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.739 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:48:16.739 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.618940+0000", 2024-09-20T22:48:16.739 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.8208000000000007e-05 2024-09-20T22:48:16.739 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.739 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.739 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 3,6", 2024-09-20T22:48:16.739 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.619042+0000", 2024-09-20T22:48:16.739 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000101364 2024-09-20T22:48:16.739 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.739 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.739 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:48:16.739 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.619601+0000", 2024-09-20T22:48:16.740 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00055948900000000004 2024-09-20T22:48:16.740 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.740 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.740 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:48:16.740 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.619634+0000", 2024-09-20T22:48:16.740 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.2325e-05 2024-09-20T22:48:16.740 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.740 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.740 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:48:16.740 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.619661+0000", 2024-09-20T22:48:16.740 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.7169999999999999e-05 2024-09-20T22:48:16.740 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.740 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.740 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:48:16.741 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.619690+0000", 2024-09-20T22:48:16.741 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.8903000000000001e-05 2024-09-20T22:48:16.741 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.741 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.741 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:48:16.741 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:42.619711+0000", 2024-09-20T22:48:16.741 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.1160000000000001e-05 2024-09-20T22:48:16.741 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:48:16.741 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:48:16.741 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:48:16.741 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.741 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.741 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(osd.2.18:3 7.1 7:84a866f5:::smithi08477203-27:head [assert-version v48,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e156)", 2024-09-20T22:48:16.741 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:55.627602+0000", 2024-09-20T22:48:16.741 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 261.06356463899999, 2024-09-20T22:48:16.742 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0032967339999999999, 2024-09-20T22:48:16.742 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:48:16.742 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:48:16.742 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:48:16.742 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:48:16.742 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.742 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:48:16.742 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:55.627602+0000", 2024-09-20T22:48:16.742 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:48:16.742 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.742 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.742 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:48:16.742 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:55.627602+0000", 2024-09-20T22:48:16.742 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:48:16.743 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.743 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.743 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:48:16.743 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:55.627604+0000", 2024-09-20T22:48:16.743 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.703e-06 2024-09-20T22:48:16.743 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.743 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.743 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:48:16.743 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:55.627622+0000", 2024-09-20T22:48:16.743 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.7683999999999999e-05 2024-09-20T22:48:16.743 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.743 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.743 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:48:16.743 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:55.627632+0000", 2024-09-20T22:48:16.744 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0719e-05 2024-09-20T22:48:16.744 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.744 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.744 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:48:16.744 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:55.627644+0000", 2024-09-20T22:48:16.744 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1097000000000001e-05 2024-09-20T22:48:16.744 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.744 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.744 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:48:16.744 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:55.627739+0000", 2024-09-20T22:48:16.744 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.5031000000000005e-05 2024-09-20T22:48:16.744 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.744 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.744 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:48:16.744 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:55.627869+0000", 2024-09-20T22:48:16.745 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00012995500000000001 2024-09-20T22:48:16.745 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.745 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.745 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:48:16.745 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:55.630899+0000", 2024-09-20T22:48:16.745 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0030305449999999999 2024-09-20T22:48:16.745 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:48:16.745 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:48:16.745 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:48:16.745 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.745 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.745 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "MOSDPGPush(7.4 405/404 [PushOp(7:24b1a612:::smithi08477203-43:90, version: 395'83, data_included: [625326~537144,1944179~511432], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:24b1a612:::smithi08477203-43:90@395'83, size: 3025326, copy_subset: [(0, 3025326)], clone_subset: {7:24b1a612:::smithi08477203-43:89: []}, snapset: 98=[]:{89: [88, 87, 86, 80, 7f, 73], 90: [8e, 8d], 98: [96, 95, 94, 93, 92]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2455611, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-20T22:48:16.745 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:48:05.275682+0000", 2024-09-20T22:48:16.745 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 11.41548474, 2024-09-20T22:48:16.746 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0036991799999999998, 2024-09-20T22:48:16.746 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:48:16.746 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:48:16.746 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:48:16.746 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:48:16.746 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.746 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:48:16.746 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:05.275682+0000", 2024-09-20T22:48:16.746 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:48:16.746 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.746 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.746 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:48:16.746 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:05.275682+0000", 2024-09-20T22:48:16.746 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:48:16.746 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.746 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.747 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:48:16.747 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:05.275683+0000", 2024-09-20T22:48:16.747 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.13e-07 2024-09-20T22:48:16.747 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.747 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.747 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:48:16.747 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:05.278175+0000", 2024-09-20T22:48:16.747 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0024916719999999999 2024-09-20T22:48:16.747 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.747 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.747 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:48:16.747 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:05.278199+0000", 2024-09-20T22:48:16.747 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.4539e-05 2024-09-20T22:48:16.748 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.748 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.748 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:48:16.748 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:05.278220+0000", 2024-09-20T22:48:16.748 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.1322e-05 2024-09-20T22:48:16.748 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.748 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.748 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:48:16.748 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:05.278706+0000", 2024-09-20T22:48:16.748 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00048536599999999999 2024-09-20T22:48:16.748 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.748 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.748 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:48:16.748 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:05.278722+0000", 2024-09-20T22:48:16.748 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.6473999999999998e-05 2024-09-20T22:48:16.749 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.749 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.749 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:48:16.749 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:05.279381+0000", 2024-09-20T22:48:16.749 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00065919399999999999 2024-09-20T22:48:16.749 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:48:16.749 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:48:16.749 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:48:16.749 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.749 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.749 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "MOSDPGPush(7.4 405/404 [PushOp(7:24b1a612:::smithi08477203-43:head, version: 400'86, data_included: [2695617~144865], data_size: 144865, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(7:24b1a612:::smithi08477203-43:head@400'86, size: 2840482, copy_subset: [(440482, 409331), (1380352, 405181), (2461553, 378929)], clone_subset: {7:24b1a612:::smithi08477203-43:98: [(0, 440482), (849813, 530539), (1785533, 676020)]}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2840482, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2695617, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-20T22:48:16.749 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:48:05.289695+0000", 2024-09-20T22:48:16.749 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 11.401471569, 2024-09-20T22:48:16.749 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.002447416, 2024-09-20T22:48:16.749 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:48:16.750 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:48:16.750 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:48:16.750 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:48:16.750 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.750 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:48:16.750 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:05.289695+0000", 2024-09-20T22:48:16.750 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:48:16.750 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.750 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.750 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:48:16.750 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:05.289695+0000", 2024-09-20T22:48:16.750 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:48:16.750 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.750 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.750 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:48:16.751 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:05.289696+0000", 2024-09-20T22:48:16.751 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.3799999999999997e-07 2024-09-20T22:48:16.751 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.751 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.751 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:48:16.751 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:05.289868+0000", 2024-09-20T22:48:16.751 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00017195399999999999 2024-09-20T22:48:16.751 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.751 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.751 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:48:16.751 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:05.289883+0000", 2024-09-20T22:48:16.751 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4642e-05 2024-09-20T22:48:16.751 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.751 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.752 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:48:16.752 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:05.289893+0000", 2024-09-20T22:48:16.752 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0332999999999999e-05 2024-09-20T22:48:16.752 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.752 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.752 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:48:16.752 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:05.289959+0000", 2024-09-20T22:48:16.752 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.6025000000000006e-05 2024-09-20T22:48:16.752 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.752 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.752 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:48:16.752 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:05.289968+0000", 2024-09-20T22:48:16.752 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.6500000000000002e-06 2024-09-20T22:48:16.752 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.753 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.753 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:48:16.753 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:05.292143+0000", 2024-09-20T22:48:16.753 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.002175274 2024-09-20T22:48:16.753 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:48:16.753 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:48:16.753 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:48:16.753 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.753 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.753 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "MOSDPGPush(7.8 406/405 [PushOp(7:12e9c8db:::smithi08477203-11:8c, version: 395'112, data_included: [2097152~810525], data_size: 810525, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(7:12e9c8db:::smithi08477203-11:8c@395'112, size: 2907677, copy_subset: [(0, 2907677)], clone_subset: {}, snapset: 8c=[]:{8c: [8a, 88, 87, 86, 80]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2907677, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-20T22:48:16.753 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:48:06.246221+0000", 2024-09-20T22:48:16.753 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 10.444945666000001, 2024-09-20T22:48:16.753 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0026978369999999998, 2024-09-20T22:48:16.753 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:48:16.753 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:48:16.754 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:48:16.754 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:48:16.754 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.754 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:48:16.754 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:06.246221+0000", 2024-09-20T22:48:16.754 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:48:16.754 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.754 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.754 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:48:16.754 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:06.246221+0000", 2024-09-20T22:48:16.754 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:48:16.754 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.754 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.754 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:48:16.755 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:06.246222+0000", 2024-09-20T22:48:16.755 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.8100000000000002e-07 2024-09-20T22:48:16.755 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.755 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.755 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:48:16.755 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:06.248132+0000", 2024-09-20T22:48:16.755 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0019102310000000001 2024-09-20T22:48:16.755 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.755 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.755 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:48:16.755 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:06.248150+0000", 2024-09-20T22:48:16.755 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.7883000000000001e-05 2024-09-20T22:48:16.755 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.755 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.756 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:48:16.756 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:06.248163+0000", 2024-09-20T22:48:16.756 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3010000000000001e-05 2024-09-20T22:48:16.756 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.756 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.756 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:48:16.756 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:06.248251+0000", 2024-09-20T22:48:16.756 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.7435999999999994e-05 2024-09-20T22:48:16.756 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.756 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.756 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:48:16.756 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:06.248263+0000", 2024-09-20T22:48:16.756 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2639999999999999e-05 2024-09-20T22:48:16.756 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.756 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.757 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:48:16.757 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:06.248919+0000", 2024-09-20T22:48:16.757 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00065585600000000002 2024-09-20T22:48:16.757 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:48:16.757 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:48:16.757 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:48:16.757 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.757 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.757 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "MOSDPGPush(7.16 406/405 [PushOp(7:6885a65f:::smithi08477203-14:94, version: 395'132, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(7:6885a65f:::smithi08477203-14:94@395'132, size: 3647185, copy_subset: [(0, 3647185)], clone_subset: {}, snapset: 94=[]:{94: [94, 93, 92, 8e, 8d, 8a, 88, 87, 86, 80, 7f]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-20T22:48:16.757 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:48:06.247167+0000", 2024-09-20T22:48:16.757 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 10.444000280000001, 2024-09-20T22:48:16.757 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0023922190000000001, 2024-09-20T22:48:16.757 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:48:16.757 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:48:16.758 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:48:16.758 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:48:16.758 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.758 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:48:16.758 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:06.247167+0000", 2024-09-20T22:48:16.758 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:48:16.758 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.758 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.758 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:48:16.758 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:06.247167+0000", 2024-09-20T22:48:16.758 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:48:16.758 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.758 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.758 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:48:16.759 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:06.247167+0000", 2024-09-20T22:48:16.759 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.5599999999999995e-07 2024-09-20T22:48:16.759 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.759 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.759 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:48:16.760 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:48:16.760 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:06.249123+0000", 2024-09-20T22:48:16.760 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0019561190000000001 2024-09-20T22:48:16.760 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.760 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.760 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:48:16.760 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:06.249143+0000", 2024-09-20T22:48:16.761 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9385000000000002e-05 2024-09-20T22:48:16.761 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.761 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.761 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:48:16.761 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:06.249156+0000", 2024-09-20T22:48:16.761 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3286e-05 2024-09-20T22:48:16.761 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.761 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.761 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:48:16.761 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:06.249236+0000", 2024-09-20T22:48:16.761 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.0036000000000004e-05 2024-09-20T22:48:16.761 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.761 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.761 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:48:16.762 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:06.249244+0000", 2024-09-20T22:48:16.762 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.0700000000000007e-06 2024-09-20T22:48:16.762 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.762 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.762 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:48:16.762 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:06.249559+0000", 2024-09-20T22:48:16.762 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00031476699999999998 2024-09-20T22:48:16.762 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:48:16.762 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:48:16.762 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:48:16.762 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.762 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.762 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "MOSDPGPush(7.9 406/405 [PushOp(7:907aec0e:::smithi08477203-3:83, version: 376'223, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(7:907aec0e:::smithi08477203-3:83@376'223, size: 3680766, copy_subset: [(0, 3680766)], clone_subset: {}, snapset: 88=[]:{83: [80, 7f, 73], 88: [88, 87, 86]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-20T22:48:16.762 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:48:06.275742+0000", 2024-09-20T22:48:16.763 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 10.415424550000001, 2024-09-20T22:48:16.763 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0023898359999999998, 2024-09-20T22:48:16.763 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:48:16.763 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:48:16.763 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:48:16.763 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:48:16.763 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.763 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:48:16.763 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:06.275742+0000", 2024-09-20T22:48:16.763 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:48:16.763 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.763 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.763 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:48:16.763 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:06.275742+0000", 2024-09-20T22:48:16.764 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:48:16.764 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.764 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.764 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:48:16.764 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:06.275743+0000", 2024-09-20T22:48:16.764 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.3799999999999996e-07 2024-09-20T22:48:16.764 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.764 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.764 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:48:16.764 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:06.277611+0000", 2024-09-20T22:48:16.764 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001867914 2024-09-20T22:48:16.764 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.764 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.764 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:48:16.765 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:06.277631+0000", 2024-09-20T22:48:16.765 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.0254e-05 2024-09-20T22:48:16.765 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.765 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.765 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:48:16.765 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:06.277643+0000", 2024-09-20T22:48:16.765 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2131e-05 2024-09-20T22:48:16.765 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.765 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.765 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:48:16.765 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:06.277738+0000", 2024-09-20T22:48:16.765 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.4758000000000003e-05 2024-09-20T22:48:16.765 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.765 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.765 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:48:16.766 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:06.277750+0000", 2024-09-20T22:48:16.766 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1807999999999999e-05 2024-09-20T22:48:16.766 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.766 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.766 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:48:16.766 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:06.278132+0000", 2024-09-20T22:48:16.766 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000382233 2024-09-20T22:48:16.766 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:48:16.766 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:48:16.766 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:48:16.766 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.766 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.766 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "MOSDPGPush(7.1 406/405 [PushOp(7:8422f82d:::smithi08477203-22:82, version: 380'196, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:8422f82d:::smithi08477203-22:82@380'196, size: 3109361, copy_subset: [(0, 3109361)], clone_subset: {}, snapset: 98=[]:{82: [80, 7f, 73], 88: [88, 87, 86], 98: [96, 95, 94, 93, 92, 8e, 8d, 8a]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-20T22:48:16.766 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:48:06.276288+0000", 2024-09-20T22:48:16.767 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 10.414878843, 2024-09-20T22:48:16.767 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0030636370000000001, 2024-09-20T22:48:16.767 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:48:16.767 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:48:16.767 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:48:16.767 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:48:16.767 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.767 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:48:16.767 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:06.276288+0000", 2024-09-20T22:48:16.767 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:48:16.767 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.767 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.767 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:48:16.767 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:06.276288+0000", 2024-09-20T22:48:16.768 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:48:16.768 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.768 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.768 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:48:16.768 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:06.276289+0000", 2024-09-20T22:48:16.768 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.6399999999999998e-07 2024-09-20T22:48:16.768 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.768 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.768 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:48:16.768 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:06.278939+0000", 2024-09-20T22:48:16.768 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0026504850000000002 2024-09-20T22:48:16.768 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.768 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.768 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:48:16.768 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:06.278954+0000", 2024-09-20T22:48:16.769 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.5064e-05 2024-09-20T22:48:16.769 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.769 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.769 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:48:16.769 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:06.278973+0000", 2024-09-20T22:48:16.769 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9360000000000001e-05 2024-09-20T22:48:16.769 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.769 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.769 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:48:16.769 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:06.279037+0000", 2024-09-20T22:48:16.769 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.3906999999999996e-05 2024-09-20T22:48:16.769 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.769 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.769 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:48:16.769 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:06.279048+0000", 2024-09-20T22:48:16.770 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0266e-05 2024-09-20T22:48:16.770 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.770 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.770 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:48:16.770 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:06.279352+0000", 2024-09-20T22:48:16.770 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00030419099999999999 2024-09-20T22:48:16.770 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:48:16.770 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:48:16.770 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:48:16.770 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.770 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.770 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "MOSDPGPush(7.1 406/405 [PushOp(7:8422f82d:::smithi08477203-22:88, version: 382'198, data_included: [665676~714065,1886514~334511], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:8422f82d:::smithi08477203-22:88@382'198, size: 3865676, copy_subset: [(665676, 714065), (1886514, 531486), (3109361, 756315)], clone_subset: {7:8422f82d:::smithi08477203-22:82: [(0, 665676), (1379741, 506773), (2418000, 691361)]}, snapset: 98=[]:{82: [80, 7f, 73], 88: [88, 87, 86], 98: [96, 95, 94, 93, 92, 8e, 8d, 8a]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2221025, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-20T22:48:16.770 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:48:06.296621+0000", 2024-09-20T22:48:16.771 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 10.394545608, 2024-09-20T22:48:16.771 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0027842140000000001, 2024-09-20T22:48:16.771 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:48:16.771 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:48:16.771 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:48:16.771 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:48:16.771 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.771 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:48:16.771 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:06.296621+0000", 2024-09-20T22:48:16.771 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:48:16.771 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.771 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.771 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:48:16.771 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:06.296621+0000", 2024-09-20T22:48:16.772 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:48:16.772 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.772 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.772 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:48:16.772 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:06.296622+0000", 2024-09-20T22:48:16.772 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.5899999999999996e-07 2024-09-20T22:48:16.772 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.772 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.772 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:48:16.772 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:06.298787+0000", 2024-09-20T22:48:16.772 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0021646590000000002 2024-09-20T22:48:16.772 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.772 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.772 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:48:16.772 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:06.298804+0000", 2024-09-20T22:48:16.773 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.6918999999999998e-05 2024-09-20T22:48:16.773 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.773 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.773 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:48:16.773 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:06.298827+0000", 2024-09-20T22:48:16.773 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.3507e-05 2024-09-20T22:48:16.773 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.773 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.773 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:48:16.773 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:06.298906+0000", 2024-09-20T22:48:16.773 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.9212e-05 2024-09-20T22:48:16.773 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.773 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.773 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:48:16.774 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:06.298916+0000", 2024-09-20T22:48:16.774 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.8279999999999998e-06 2024-09-20T22:48:16.774 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.774 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.774 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:48:16.774 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:06.299406+0000", 2024-09-20T22:48:16.774 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00048943000000000001 2024-09-20T22:48:16.774 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:48:16.774 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:48:16.774 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:48:16.774 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.774 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.774 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "MOSDPGPush(7.9 406/405 [PushOp(7:907aec0e:::smithi08477203-3:88, version: 382'227, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(7:907aec0e:::smithi08477203-3:88@382'227, size: 3042133, copy_subset: [(0, 3042133)], clone_subset: {7:907aec0e:::smithi08477203-3:83: []}, snapset: 88=[]:{83: [80, 7f, 73], 88: [88, 87, 86]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-20T22:48:16.774 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:48:06.296892+0000", 2024-09-20T22:48:16.775 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 10.394274641999999, 2024-09-20T22:48:16.775 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.002719553, 2024-09-20T22:48:16.775 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:48:16.775 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:48:16.775 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:48:16.775 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:48:16.775 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.775 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:48:16.775 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:06.296892+0000", 2024-09-20T22:48:16.775 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:48:16.775 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.775 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.775 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:48:16.776 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:06.296892+0000", 2024-09-20T22:48:16.776 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:48:16.776 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.776 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.776 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:48:16.776 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:06.296893+0000", 2024-09-20T22:48:16.776 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.3099999999999997e-07 2024-09-20T22:48:16.776 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.776 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.776 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:48:16.776 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:06.299196+0000", 2024-09-20T22:48:16.776 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.002302864 2024-09-20T22:48:16.776 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.776 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.777 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:48:16.777 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:06.299212+0000", 2024-09-20T22:48:16.777 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.5648999999999999e-05 2024-09-20T22:48:16.777 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.777 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.777 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:48:16.777 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:06.299223+0000", 2024-09-20T22:48:16.777 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.188e-05 2024-09-20T22:48:16.777 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.777 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.777 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:48:16.777 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:06.299290+0000", 2024-09-20T22:48:16.777 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.6934000000000007e-05 2024-09-20T22:48:16.777 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.777 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.778 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:48:16.778 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:06.299299+0000", 2024-09-20T22:48:16.778 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.7879999999999995e-06 2024-09-20T22:48:16.778 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.778 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.778 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:48:16.778 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:06.299612+0000", 2024-09-20T22:48:16.778 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00031270700000000001 2024-09-20T22:48:16.778 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:48:16.778 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:48:16.778 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:48:16.778 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.778 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.778 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "MOSDPGPush(7.17 406/405 [PushOp(7:ed54065f:::smithi08477203-12:86, version: 382'330, data_included: [2071626~116504,2784436~560223], data_size: 676727, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(7:ed54065f:::smithi08477203-12:86@382'330, size: 3344659, copy_subset: [(544659, 715934), (1738984, 449146), (2784436, 560223)], clone_subset: {7:ed54065f:::smithi08477203-12:74: [(0, 544659), (1260593, 478391), (2188130, 596306)], 7:ed54065f:::smithi08477203-12:88: [(0, 544659), (1260593, 478391), (2188130, 596306)]}, snapset: a5=[]:{74: [73], 86: [86, 80, 7f], 88: [88, 87], 90: [8e, 8d, 8a], 9b: [9b, 9a], a5: [a5, a4, a3, a1, a0, 9f, 9e, 9d, 9c]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3344659, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2071626, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-20T22:48:16.779 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:48:06.335790+0000", 2024-09-20T22:48:16.779 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 10.355377163, 2024-09-20T22:48:16.779 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0037307030000000001, 2024-09-20T22:48:16.779 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:48:16.779 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:48:16.779 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:48:16.779 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:48:16.779 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.779 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:48:16.779 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:06.335790+0000", 2024-09-20T22:48:16.779 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:48:16.779 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.779 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.779 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:48:16.779 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:06.335790+0000", 2024-09-20T22:48:16.780 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:48:16.780 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.780 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.780 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:48:16.780 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:06.335790+0000", 2024-09-20T22:48:16.780 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.06e-07 2024-09-20T22:48:16.780 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.780 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.780 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:48:16.780 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:06.336008+0000", 2024-09-20T22:48:16.780 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00021747800000000001 2024-09-20T22:48:16.780 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.780 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.780 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:48:16.780 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:06.336024+0000", 2024-09-20T22:48:16.781 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.6246e-05 2024-09-20T22:48:16.781 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.781 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.781 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:48:16.781 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:06.336039+0000", 2024-09-20T22:48:16.781 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.5187999999999999e-05 2024-09-20T22:48:16.781 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.781 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.781 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:48:16.781 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:06.336131+0000", 2024-09-20T22:48:16.781 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.1355999999999997e-05 2024-09-20T22:48:16.781 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.781 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.781 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:48:16.782 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:06.336150+0000", 2024-09-20T22:48:16.782 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9661999999999999e-05 2024-09-20T22:48:16.782 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.782 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.782 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:48:16.782 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:06.339521+0000", 2024-09-20T22:48:16.782 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0033701669999999999 2024-09-20T22:48:16.782 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:48:16.782 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:48:16.782 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:48:16.782 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.782 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.782 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "MOSDPGPush(7.17 406/405 [PushOp(7:ea720b85:::smithi08477203-26:9b, version: 400'353, data_included: [1920123~83226], data_size: 83226, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(7:ea720b85:::smithi08477203-26:9b@400'353, size: 2437114, copy_subset: [(0, 2437114)], clone_subset: {7:ea720b85:::smithi08477203-26:90: []}, snapset: 9b=[]:{8c: [8a, 88, 87, 86, 80, 7f, 73], 90: [8e, 8d], 9b: [9b, 9a, 96, 95, 94, 93, 92]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2437114, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1920123, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-20T22:48:16.782 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:48:06.363525+0000", 2024-09-20T22:48:16.783 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 10.327641953000001, 2024-09-20T22:48:16.783 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0034189939999999999, 2024-09-20T22:48:16.783 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:48:16.783 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:48:16.783 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:48:16.783 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:48:16.783 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.783 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:48:16.783 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:06.363525+0000", 2024-09-20T22:48:16.783 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:48:16.783 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.783 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.783 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:48:16.783 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:06.363525+0000", 2024-09-20T22:48:16.784 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:48:16.784 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.784 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.784 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:48:16.784 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:06.363526+0000", 2024-09-20T22:48:16.784 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.2399999999999997e-07 2024-09-20T22:48:16.784 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.784 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.784 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:48:16.784 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:06.363553+0000", 2024-09-20T22:48:16.784 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.7484999999999999e-05 2024-09-20T22:48:16.784 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.784 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.784 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:48:16.784 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:06.363569+0000", 2024-09-20T22:48:16.785 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.5710999999999999e-05 2024-09-20T22:48:16.785 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.785 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.785 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:48:16.785 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:06.363582+0000", 2024-09-20T22:48:16.785 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3129000000000001e-05 2024-09-20T22:48:16.785 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.785 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.785 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:48:16.785 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:06.363666+0000", 2024-09-20T22:48:16.785 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.3931000000000007e-05 2024-09-20T22:48:16.785 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.785 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.785 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:48:16.785 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:06.363682+0000", 2024-09-20T22:48:16.786 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.6237000000000001e-05 2024-09-20T22:48:16.786 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:16.786 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:16.786 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:48:16.786 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:06.366944+0000", 2024-09-20T22:48:16.786 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0032616770000000002 2024-09-20T22:48:16.786 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:48:16.786 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:48:16.786 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:48:16.786 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:48:16.786 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:48:16.786 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-20T22:48:16.807 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:16.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:16.808 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:48:16.908 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:48:16.930 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:16.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:16.931 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:48:17.032 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:48:17.053 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:17.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:17.054 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:48:17.154 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:48:17.175 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:17.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:17.176 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:48:17.223 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is unset 2024-09-20T22:48:17.227 INFO:tasks.rados.rados.0.smithi084.stdout:1606: done (1 left) 2024-09-20T22:48:17.228 INFO:tasks.rados.rados.0.smithi084.stdout:1607: rollback oid 8 current snap is 171 2024-09-20T22:48:17.228 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 8 to 145 2024-09-20T22:48:17.228 INFO:tasks.rados.rados.0.smithi084.stdout:1608: write (excl) oid 18 current snap is 171 2024-09-20T22:48:17.228 INFO:tasks.rados.rados.0.smithi084.stdout:1608: seq_num 568 ranges {746376=551554,2054831=649524,3315201=231175} 2024-09-20T22:48:17.235 INFO:tasks.rados.rados.0.smithi084.stdout:1608: writing smithi08477203-18 from 746376 to 1297930 tid 1 2024-09-20T22:48:17.243 INFO:tasks.rados.rados.0.smithi084.stdout:1608: writing smithi08477203-18 from 2054831 to 2704355 tid 2 2024-09-20T22:48:17.248 INFO:tasks.rados.rados.0.smithi084.stdout:1608: writing smithi08477203-18 from 3315201 to 3546376 tid 3 2024-09-20T22:48:17.249 INFO:tasks.rados.rados.0.smithi084.stdout:1609: delete oid 45 current snap is 171 2024-09-20T22:48:17.249 INFO:tasks.rados.rados.0.smithi084.stdout:1605: finishing write tid 1 to smithi08477203-19 2024-09-20T22:48:17.249 INFO:tasks.rados.rados.0.smithi084.stdout:1605: finishing write tid 2 to smithi08477203-19 2024-09-20T22:48:17.249 INFO:tasks.rados.rados.0.smithi084.stdout:1605: finishing write tid 3 to smithi08477203-19 2024-09-20T22:48:17.249 INFO:tasks.rados.rados.0.smithi084.stdout:1605: finishing write tid 4 to smithi08477203-19 2024-09-20T22:48:17.249 INFO:tasks.rados.rados.0.smithi084.stdout:1605: finishing write tid 5 to smithi08477203-19 2024-09-20T22:48:17.249 INFO:tasks.rados.rados.0.smithi084.stdout:1605: oid 19 updating version 0 to 892 2024-09-20T22:48:17.249 INFO:tasks.rados.rados.0.smithi084.stdout:1605: oid 19 updating version 892 to 893 2024-09-20T22:48:17.249 INFO:tasks.rados.rados.0.smithi084.stdout:1605: oid 19 updating version 893 to 894 2024-09-20T22:48:17.249 INFO:tasks.rados.rados.0.smithi084.stdout:1605: oid 19 updating version 894 to 895 2024-09-20T22:48:17.249 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 19 v 895 (ObjNum 567 snap 171 seq_num 567) dirty exists 2024-09-20T22:48:17.249 INFO:tasks.rados.rados.0.smithi084.stdout:1605: left oid 19 (ObjNum 567 snap 171 seq_num 567) 2024-09-20T22:48:17.249 INFO:tasks.rados.rados.0.smithi084.stdout:1607: finishing rollback tid 1 to smithi08477203-8 2024-09-20T22:48:17.249 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 8 v 404 (ObjNum 129 snap 0 seq_num 2091148064) dirty dne 2024-09-20T22:48:17.250 INFO:tasks.rados.rados.0.smithi084.stdout:1608: finishing write tid 1 to smithi08477203-18 2024-09-20T22:48:17.250 INFO:tasks.rados.rados.0.smithi084.stdout:1608: finishing write tid 2 to smithi08477203-18 2024-09-20T22:48:17.251 INFO:tasks.rados.rados.0.smithi084.stdout:1608: finishing write tid 3 to smithi08477203-18 2024-09-20T22:48:17.251 INFO:tasks.rados.rados.0.smithi084.stdout:1605: done (3 left) 2024-09-20T22:48:17.251 INFO:tasks.rados.rados.0.smithi084.stdout:1607: done (2 left) 2024-09-20T22:48:17.251 INFO:tasks.rados.rados.0.smithi084.stdout:1609: done (1 left) 2024-09-20T22:48:17.251 INFO:tasks.rados.rados.0.smithi084.stdout:1610: copy_from oid 28 from oid 44 current snap is 171 2024-09-20T22:48:17.251 INFO:tasks.rados.rados.0.smithi084.stdout:1611: delete oid 48 current snap is 171 2024-09-20T22:48:17.251 INFO:tasks.rados.rados.0.smithi084.stdout:1608: finishing write tid 4 to smithi08477203-18 2024-09-20T22:48:17.251 INFO:tasks.rados.rados.0.smithi084.stdout:1608: finishing write tid 5 to smithi08477203-18 2024-09-20T22:48:17.251 INFO:tasks.rados.rados.0.smithi084.stdout:1608: oid 18 updating version 0 to 556 2024-09-20T22:48:17.252 INFO:tasks.rados.rados.0.smithi084.stdout:1608: oid 18 version 556 is already newer than 554 2024-09-20T22:48:17.252 INFO:tasks.rados.rados.0.smithi084.stdout:1608: oid 18 version 556 is already newer than 553 2024-09-20T22:48:17.252 INFO:tasks.rados.rados.0.smithi084.stdout:1608: oid 18 version 556 is already newer than 555 2024-09-20T22:48:17.252 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 18 v 556 (ObjNum 568 snap 171 seq_num 568) dirty exists 2024-09-20T22:48:17.252 INFO:tasks.rados.rados.0.smithi084.stdout:1608: left oid 18 (ObjNum 568 snap 171 seq_num 568) 2024-09-20T22:48:17.252 INFO:tasks.rados.rados.0.smithi084.stdout:1608: done (2 left) 2024-09-20T22:48:17.252 INFO:tasks.rados.rados.0.smithi084.stdout:1611: done (1 left) 2024-09-20T22:48:17.252 INFO:tasks.rados.rados.0.smithi084.stdout:1612: delete oid 31 current snap is 171 2024-09-20T22:48:17.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:17 smithi154 ceph-mon[33264]: purged_snaps scrub starts 2024-09-20T22:48:17.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:17 smithi154 ceph-mon[33264]: purged_snaps scrub ok 2024-09-20T22:48:17.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:17 smithi154 ceph-mon[33264]: pgmap v525: 168 pgs: 2 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 160 active+clean; 347 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 27 MiB/s rd, 6.2 MiB/s wr, 35 op/s 2024-09-20T22:48:17.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:17 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "16"}]': finished 2024-09-20T22:48:17.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:17 smithi154 ceph-mon[33264]: osdmap e416: 8 total, 8 up, 8 in 2024-09-20T22:48:17.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:17 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/2691974748' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:48:17.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:17 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:48:17.256 INFO:tasks.rados.rados.0.smithi084.stdout:1612: done (1 left) 2024-09-20T22:48:17.256 INFO:tasks.rados.rados.0.smithi084.stdout:1613: read oid 39 snap -1 2024-09-20T22:48:17.256 INFO:tasks.rados.rados.0.smithi084.stdout:1613: expect deleted 2024-09-20T22:48:17.256 INFO:tasks.rados.rados.0.smithi084.stdout:1614: read oid 46 snap -1 2024-09-20T22:48:17.256 INFO:tasks.rados.rados.0.smithi084.stdout:1614: expect (ObjNum 470 snap 130 seq_num 470) 2024-09-20T22:48:17.256 INFO:tasks.rados.rados.0.smithi084.stdout:1615: write (excl) oid 27 current snap is 171 2024-09-20T22:48:17.257 INFO:tasks.rados.rados.0.smithi084.stdout:1615: seq_num 572 ranges {413604=568903,1488759=656862,2624893=591709} 2024-09-20T22:48:17.262 INFO:tasks.rados.rados.0.smithi084.stdout:1615: writing smithi08477203-27 from 413604 to 982507 tid 1 2024-09-20T22:48:17.268 INFO:tasks.rados.rados.0.smithi084.stdout:1615: writing smithi08477203-27 from 1488759 to 2145621 tid 2 2024-09-20T22:48:17.275 INFO:tasks.rados.rados.0.smithi084.stdout:1615: writing smithi08477203-27 from 2624893 to 3216602 tid 3 2024-09-20T22:48:17.275 INFO:tasks.rados.rados.0.smithi084.stdout:1616: snap_create 2024-09-20T22:48:17.275 INFO:tasks.rados.rados.0.smithi084.stdout:1615: finishing write tid 1 to smithi08477203-27 2024-09-20T22:48:17.275 INFO:tasks.rados.rados.0.smithi084.stdout:1614: expect (ObjNum 470 snap 130 seq_num 470) 2024-09-20T22:48:17.276 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:48:17.297 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:17.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:17.298 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:48:17.364 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:17 smithi084 ceph-mon[40024]: purged_snaps scrub starts 2024-09-20T22:48:17.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:17 smithi084 ceph-mon[40024]: purged_snaps scrub ok 2024-09-20T22:48:17.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:17 smithi084 ceph-mon[40024]: pgmap v525: 168 pgs: 2 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 160 active+clean; 347 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 27 MiB/s rd, 6.2 MiB/s wr, 35 op/s 2024-09-20T22:48:17.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:17 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "16"}]': finished 2024-09-20T22:48:17.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:17 smithi084 ceph-mon[40024]: osdmap e416: 8 total, 8 up, 8 in 2024-09-20T22:48:17.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:17 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/2691974748' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:48:17.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:17 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:48:17.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:17 smithi084 ceph-mon[29774]: purged_snaps scrub starts 2024-09-20T22:48:17.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:17 smithi084 ceph-mon[29774]: purged_snaps scrub ok 2024-09-20T22:48:17.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:17 smithi084 ceph-mon[29774]: pgmap v525: 168 pgs: 2 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 160 active+clean; 347 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 27 MiB/s rd, 6.2 MiB/s wr, 35 op/s 2024-09-20T22:48:17.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:17 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "16"}]': finished 2024-09-20T22:48:17.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:17 smithi084 ceph-mon[29774]: osdmap e416: 8 total, 8 up, 8 in 2024-09-20T22:48:17.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:17 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/2691974748' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:48:17.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:17 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:48:17.399 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:48:17.400 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-20T22:48:17.403 INFO:tasks.rados.rados.0.smithi084.stdout:1610: finishing copy_from racing read to smithi08477203-28 2024-09-20T22:48:17.403 INFO:tasks.rados.rados.0.smithi084.stdout:1610: finishing copy_from to smithi08477203-28 2024-09-20T22:48:17.403 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 28 v 722 (ObjNum 470 snap 130 seq_num 470) dirty exists 2024-09-20T22:48:17.403 INFO:tasks.rados.rados.0.smithi084.stdout:1615: finishing write tid 2 to smithi08477203-27 2024-09-20T22:48:17.403 INFO:tasks.rados.rados.0.smithi084.stdout:1615: finishing write tid 3 to smithi08477203-27 2024-09-20T22:48:17.404 INFO:tasks.rados.rados.0.smithi084.stdout:1615: finishing write tid 4 to smithi08477203-27 2024-09-20T22:48:17.404 INFO:tasks.rados.rados.0.smithi084.stdout:1615: finishing write tid 5 to smithi08477203-27 2024-09-20T22:48:17.404 INFO:tasks.rados.rados.0.smithi084.stdout:1615: oid 27 updating version 0 to 326 2024-09-20T22:48:17.404 INFO:tasks.rados.rados.0.smithi084.stdout:1615: oid 27 version 326 is already newer than 325 2024-09-20T22:48:17.404 INFO:tasks.rados.rados.0.smithi084.stdout:1615: oid 27 updating version 326 to 328 2024-09-20T22:48:17.404 INFO:tasks.rados.rados.0.smithi084.stdout:1615: oid 27 version 328 is already newer than 327 2024-09-20T22:48:17.404 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 27 v 328 (ObjNum 572 snap 171 seq_num 572) dirty exists 2024-09-20T22:48:17.404 INFO:tasks.rados.rados.0.smithi084.stdout:1615: left oid 27 (ObjNum 572 snap 171 seq_num 572) 2024-09-20T22:48:17.421 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:17.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:17.422 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:48:17.523 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:48:17.544 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:17.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:17.545 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:48:17.646 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:48:17.647 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:48:17.667 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:17.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:17.669 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:48:17.769 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:48:17.790 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:17.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:17.791 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:48:17.850 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-20T22:48:17.892 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:48:17.913 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:17.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:17.914 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:48:18.015 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:48:18.036 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:18.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:18.037 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:48:18.095 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:48:18.102 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-20T22:48:18.102 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [], 2024-09-20T22:48:18.102 INFO:teuthology.orchestra.run.smithi154.stdout: "num_ops": 0 2024-09-20T22:48:18.102 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-20T22:48:18.138 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:48:18.160 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:18.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:18.161 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:48:18.206 INFO:tasks.rados.rados.0.smithi084.stdout:1610: done (4 left) 2024-09-20T22:48:18.206 INFO:tasks.rados.rados.0.smithi084.stdout:1613: done (3 left) 2024-09-20T22:48:18.206 INFO:tasks.rados.rados.0.smithi084.stdout:1614: done (2 left) 2024-09-20T22:48:18.206 INFO:tasks.rados.rados.0.smithi084.stdout:1615: done (1 left) 2024-09-20T22:48:18.206 INFO:tasks.rados.rados.0.smithi084.stdout:1616: done (0 left) 2024-09-20T22:48:18.206 INFO:tasks.rados.rados.0.smithi084.stdout:1617: snap_remove snap 168 2024-09-20T22:48:18.262 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:48:18.283 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:18.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:18.284 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:48:18.385 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:48:18.406 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:18.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:18.407 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:48:18.430 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:18 smithi154 ceph-mon[33264]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:48:18.430 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:18 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T22:48:18.430 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:18 smithi154 ceph-mon[33264]: osdmap e417: 8 total, 8 up, 8 in 2024-09-20T22:48:18.430 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:18 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "16"}]: dispatch 2024-09-20T22:48:18.430 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:18 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "16"}]: dispatch 2024-09-20T22:48:18.430 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:18 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "16"}]': finished 2024-09-20T22:48:18.431 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:18 smithi154 ceph-mon[33264]: osdmap e418: 8 total, 8 up, 8 in 2024-09-20T22:48:18.508 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:48:18.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:18.530 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:18.530 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:48:18.614 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:48:18 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: ::ffff:172.21.15.154 - - [20/Sep/2024:22:48:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-20T22:48:18.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:18 smithi084 ceph-mon[29774]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:48:18.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:18 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T22:48:18.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:18 smithi084 ceph-mon[29774]: osdmap e417: 8 total, 8 up, 8 in 2024-09-20T22:48:18.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:18 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "16"}]: dispatch 2024-09-20T22:48:18.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:18 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "16"}]: dispatch 2024-09-20T22:48:18.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:18 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "16"}]': finished 2024-09-20T22:48:18.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:18 smithi084 ceph-mon[29774]: osdmap e418: 8 total, 8 up, 8 in 2024-09-20T22:48:18.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:18 smithi084 ceph-mon[40024]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:48:18.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:18 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T22:48:18.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:18 smithi084 ceph-mon[40024]: osdmap e417: 8 total, 8 up, 8 in 2024-09-20T22:48:18.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:18 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "16"}]: dispatch 2024-09-20T22:48:18.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:18 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "16"}]: dispatch 2024-09-20T22:48:18.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:18 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "16"}]': finished 2024-09-20T22:48:18.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:18 smithi084 ceph-mon[40024]: osdmap e418: 8 total, 8 up, 8 in 2024-09-20T22:48:18.631 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:48:18.653 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:18.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:18.654 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:48:18.754 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:48:18.771 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:48:18.772 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:48:18.778 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:18.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:18.779 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:48:18.788 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:48:18.789 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:48:18.801 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.5 dump_blocked_ops 2024-09-20T22:48:18.804 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:48:18.804 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:48:18.820 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-20T22:48:18.820 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T22:48:18.837 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:48:18.837 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:48:18.853 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:48:18.853 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:48:18.870 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:48:18.870 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:48:18.880 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:48:18.887 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:48:18.887 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:48:18.901 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:18.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:18.902 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:48:19.003 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:48:19.025 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:19.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:19.027 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:48:19.088 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:48:19.128 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:48:19.149 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:19.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:19.150 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:48:19.212 INFO:tasks.rados.rados.0.smithi084.stdout:1617: done (0 left) 2024-09-20T22:48:19.212 INFO:tasks.rados.rados.0.smithi084.stdout:1618: read oid 28 snap -1 2024-09-20T22:48:19.212 INFO:tasks.rados.rados.0.smithi084.stdout:1618: expect (ObjNum 470 snap 130 seq_num 470) 2024-09-20T22:48:19.212 INFO:tasks.rados.rados.0.smithi084.stdout:1619: write (excl) oid 13 current snap is 172 2024-09-20T22:48:19.212 INFO:tasks.rados.rados.0.smithi084.stdout:1619: seq_num 573 ranges {430411=644152,1830888=400520,2935963=402981} 2024-09-20T22:48:19.214 INFO:tasks.rados.rados.0.smithi084.stdout:1619: writing smithi08477203-13 from 430411 to 1074563 tid 1 2024-09-20T22:48:19.221 INFO:tasks.rados.rados.0.smithi084.stdout:1619: writing smithi08477203-13 from 1830888 to 2231408 tid 2 2024-09-20T22:48:19.228 INFO:tasks.rados.rados.0.smithi084.stdout:1619: writing smithi08477203-13 from 2935963 to 3338944 tid 3 2024-09-20T22:48:19.228 INFO:tasks.rados.rados.0.smithi084.stdout:1620: read oid 7 snap -1 2024-09-20T22:48:19.228 INFO:tasks.rados.rados.0.smithi084.stdout:1620: expect deleted 2024-09-20T22:48:19.228 INFO:tasks.rados.rados.0.smithi084.stdout:1621: read oid 31 snap -1 2024-09-20T22:48:19.228 INFO:tasks.rados.rados.0.smithi084.stdout:1621: expect deleted 2024-09-20T22:48:19.229 INFO:tasks.rados.rados.0.smithi084.stdout:1622: write (excl) oid 43 current snap is 172 2024-09-20T22:48:19.229 INFO:tasks.rados.rados.0.smithi084.stdout:1622: seq_num 574 ranges {447218=719401,1889370=660531,2963386=614253} 2024-09-20T22:48:19.234 INFO:tasks.rados.rados.0.smithi084.stdout:1622: writing smithi08477203-43 from 447218 to 1166619 tid 1 2024-09-20T22:48:19.242 INFO:tasks.rados.rados.0.smithi084.stdout:1622: writing smithi08477203-43 from 1889370 to 2549901 tid 2 2024-09-20T22:48:19.248 INFO:tasks.rados.rados.0.smithi084.stdout:1622: writing smithi08477203-43 from 2963386 to 3577639 tid 3 2024-09-20T22:48:19.248 INFO:tasks.rados.rados.0.smithi084.stdout:1623: snap_remove snap 145 2024-09-20T22:48:19.251 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:48:19.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:19.273 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:19.273 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:48:19.374 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:48:19.395 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:19.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:19.396 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:48:19.497 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:48:19.504 INFO:journalctl@ceph.iscsi.iscsi.a.smithi154.stdout:Sep 20 22:48:19 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-iscsi-iscsi-a[59730]: debug there is no tcmu-runner data available 2024-09-20T22:48:19.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:19 smithi154 ceph-mon[33264]: 7.17 scrub starts 2024-09-20T22:48:19.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:19 smithi154 ceph-mon[33264]: 7.17 scrub ok 2024-09-20T22:48:19.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:19 smithi154 ceph-mon[33264]: 1.8 scrub starts 2024-09-20T22:48:19.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:19 smithi154 ceph-mon[33264]: 1.8 scrub ok 2024-09-20T22:48:19.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:19 smithi154 ceph-mon[33264]: pgmap v528: 168 pgs: 2 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 160 active+clean; 347 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail 2024-09-20T22:48:19.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:19 smithi154 ceph-mon[33264]: 5.1f scrub starts 2024-09-20T22:48:19.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:19 smithi154 ceph-mon[33264]: 5.1f scrub ok 2024-09-20T22:48:19.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:19 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/3559299363' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:48:19.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:19 smithi154 ceph-mon[33264]: osdmap e419: 8 total, 8 up, 8 in 2024-09-20T22:48:19.518 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:19.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:19.519 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:48:19.609 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-20T22:48:19.609 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:48:19.614 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:19 smithi084 ceph-mon[40024]: 7.17 scrub starts 2024-09-20T22:48:19.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:19 smithi084 ceph-mon[40024]: 7.17 scrub ok 2024-09-20T22:48:19.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:19 smithi084 ceph-mon[40024]: 1.8 scrub starts 2024-09-20T22:48:19.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:19 smithi084 ceph-mon[40024]: 1.8 scrub ok 2024-09-20T22:48:19.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:19 smithi084 ceph-mon[40024]: pgmap v528: 168 pgs: 2 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 160 active+clean; 347 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail 2024-09-20T22:48:19.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:19 smithi084 ceph-mon[40024]: 5.1f scrub starts 2024-09-20T22:48:19.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:19 smithi084 ceph-mon[40024]: 5.1f scrub ok 2024-09-20T22:48:19.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:19 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/3559299363' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:48:19.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:19 smithi084 ceph-mon[40024]: osdmap e419: 8 total, 8 up, 8 in 2024-09-20T22:48:19.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:19 smithi084 ceph-mon[29774]: 7.17 scrub starts 2024-09-20T22:48:19.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:19 smithi084 ceph-mon[29774]: 7.17 scrub ok 2024-09-20T22:48:19.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:19 smithi084 ceph-mon[29774]: 1.8 scrub starts 2024-09-20T22:48:19.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:19 smithi084 ceph-mon[29774]: 1.8 scrub ok 2024-09-20T22:48:19.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:19 smithi084 ceph-mon[29774]: pgmap v528: 168 pgs: 2 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 160 active+clean; 347 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail 2024-09-20T22:48:19.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:19 smithi084 ceph-mon[29774]: 5.1f scrub starts 2024-09-20T22:48:19.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:19 smithi084 ceph-mon[29774]: 5.1f scrub ok 2024-09-20T22:48:19.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:19 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/3559299363' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:48:19.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:19 smithi084 ceph-mon[29774]: osdmap e419: 8 total, 8 up, 8 in 2024-09-20T22:48:19.620 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:48:19.643 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:19.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:19.644 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:48:19.745 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:48:19.766 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:19.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:19.767 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:48:19.831 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-20T22:48:19.868 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:48:19.889 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:19.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:19.890 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:48:19.991 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:48:20.012 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:20.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:20.013 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:48:20.075 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:48:20.114 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:48:20.135 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:20.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:20.136 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:48:20.213 INFO:tasks.rados.rados.0.smithi084.stdout:1623: done (5 left) 2024-09-20T22:48:20.213 INFO:tasks.rados.rados.0.smithi084.stdout:1624: copy_from oid 49 from oid 26 current snap is 172 2024-09-20T22:48:20.213 INFO:tasks.rados.rados.0.smithi084.stdout:1625: snap_remove snap 158 2024-09-20T22:48:20.237 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:48:20.261 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:20.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:20.262 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:48:20.319 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.5 dump_historic_ops 2024-09-20T22:48:20.363 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:48:20.385 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:20.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:20.386 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:48:20.467 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:20 smithi084 ceph-mon[29774]: 3.17 scrub starts 2024-09-20T22:48:20.467 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:20 smithi084 ceph-mon[29774]: 3.17 scrub ok 2024-09-20T22:48:20.467 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:20 smithi084 ceph-mon[29774]: 6.7 scrub starts 2024-09-20T22:48:20.467 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:20 smithi084 ceph-mon[29774]: 6.7 scrub ok 2024-09-20T22:48:20.467 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:20 smithi084 ceph-mon[29774]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:48:20.468 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:20 smithi084 ceph-mon[29774]: 6.12 scrub starts 2024-09-20T22:48:20.468 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:20 smithi084 ceph-mon[29774]: 6.12 scrub ok 2024-09-20T22:48:20.468 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:20 smithi084 ceph-mon[29774]: 7.11 scrub starts 2024-09-20T22:48:20.468 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:20 smithi084 ceph-mon[29774]: 7.11 scrub ok 2024-09-20T22:48:20.468 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:20 smithi084 ceph-mon[29774]: osdmap e420: 8 total, 8 up, 8 in 2024-09-20T22:48:20.468 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:20 smithi084 ceph-mon[40024]: 3.17 scrub starts 2024-09-20T22:48:20.468 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:20 smithi084 ceph-mon[40024]: 3.17 scrub ok 2024-09-20T22:48:20.468 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:20 smithi084 ceph-mon[40024]: 6.7 scrub starts 2024-09-20T22:48:20.468 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:20 smithi084 ceph-mon[40024]: 6.7 scrub ok 2024-09-20T22:48:20.468 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:20 smithi084 ceph-mon[40024]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:48:20.468 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:20 smithi084 ceph-mon[40024]: 6.12 scrub starts 2024-09-20T22:48:20.468 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:20 smithi084 ceph-mon[40024]: 6.12 scrub ok 2024-09-20T22:48:20.469 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:20 smithi084 ceph-mon[40024]: 7.11 scrub starts 2024-09-20T22:48:20.469 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:20 smithi084 ceph-mon[40024]: 7.11 scrub ok 2024-09-20T22:48:20.469 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:20 smithi084 ceph-mon[40024]: osdmap e420: 8 total, 8 up, 8 in 2024-09-20T22:48:20.487 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:48:20.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:20 smithi154 ceph-mon[33264]: 3.17 scrub starts 2024-09-20T22:48:20.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:20 smithi154 ceph-mon[33264]: 3.17 scrub ok 2024-09-20T22:48:20.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:20 smithi154 ceph-mon[33264]: 6.7 scrub starts 2024-09-20T22:48:20.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:20 smithi154 ceph-mon[33264]: 6.7 scrub ok 2024-09-20T22:48:20.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:20 smithi154 ceph-mon[33264]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:48:20.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:20 smithi154 ceph-mon[33264]: 6.12 scrub starts 2024-09-20T22:48:20.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:20 smithi154 ceph-mon[33264]: 6.12 scrub ok 2024-09-20T22:48:20.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:20 smithi154 ceph-mon[33264]: 7.11 scrub starts 2024-09-20T22:48:20.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:20 smithi154 ceph-mon[33264]: 7.11 scrub ok 2024-09-20T22:48:20.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:20 smithi154 ceph-mon[33264]: osdmap e420: 8 total, 8 up, 8 in 2024-09-20T22:48:20.508 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:20.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:20.509 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:48:20.577 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:48:20.610 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:48:20.632 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:20.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:20.633 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:48:20.734 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:48:20.755 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:20.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:20.756 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:48:20.857 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:48:20.879 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:20.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:20.880 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:48:20.980 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:48:21.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:21.003 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:21.003 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:48:21.010 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-20T22:48:21.010 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:48:21.104 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:48:21.126 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:21.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:21.127 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:48:21.206 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is unset 2024-09-20T22:48:21.208 INFO:tasks.rados.rados.0.smithi084.stdout:1625: done (6 left) 2024-09-20T22:48:21.208 INFO:tasks.rados.rados.0.smithi084.stdout:1626: write oid 10 current snap is 172 2024-09-20T22:48:21.209 INFO:tasks.rados.rados.0.smithi084.stdout:1626: seq_num 575 ranges {464025=794650,1947852=404189,2990809=425525} 2024-09-20T22:48:21.216 INFO:tasks.rados.rados.0.smithi084.stdout:1626: writing smithi08477203-10 from 464025 to 1258675 tid 1 2024-09-20T22:48:21.223 INFO:tasks.rados.rados.0.smithi084.stdout:1626: writing smithi08477203-10 from 1947852 to 2352041 tid 2 2024-09-20T22:48:21.228 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:48:21.230 INFO:tasks.rados.rados.0.smithi084.stdout:1626: writing smithi08477203-10 from 2990809 to 3416334 tid 3 2024-09-20T22:48:21.230 INFO:tasks.rados.rados.0.smithi084.stdout:1627: write (excl) oid 35 current snap is 172 2024-09-20T22:48:21.230 INFO:tasks.rados.rados.0.smithi084.stdout:1627: seq_num 576 ranges {480832=469899,1606334=664200,3018232=662600} 2024-09-20T22:48:21.236 INFO:tasks.rados.rados.0.smithi084.stdout:1627: writing smithi08477203-35 from 480832 to 950731 tid 1 2024-09-20T22:48:21.244 INFO:tasks.rados.rados.0.smithi084.stdout:1627: writing smithi08477203-35 from 1606334 to 2270534 tid 2 2024-09-20T22:48:21.253 INFO:tasks.rados.rados.0.smithi084.stdout:1627: writing smithi08477203-35 from 3018232 to 3680832 tid 3 2024-09-20T22:48:21.253 INFO:tasks.rados.rados.0.smithi084.stdout:1628: snap_create 2024-09-20T22:48:21.253 INFO:tasks.rados.rados.0.smithi084.stdout:1618: expect (ObjNum 470 snap 130 seq_num 470) 2024-09-20T22:48:21.256 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:21.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:21.257 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:48:21.332 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:21 smithi154 ceph-mon[33264]: 3.12 scrub starts 2024-09-20T22:48:21.332 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:21 smithi154 ceph-mon[33264]: 3.12 scrub ok 2024-09-20T22:48:21.332 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:21 smithi154 ceph-mon[33264]: pgmap v531: 168 pgs: 6 unknown, 3 peering, 5 active+clean+snaptrim, 154 active+clean; 293 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail 2024-09-20T22:48:21.332 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:21 smithi154 ceph-mon[33264]: 4.5 scrub starts 2024-09-20T22:48:21.333 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:21 smithi154 ceph-mon[33264]: 4.5 scrub ok 2024-09-20T22:48:21.333 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:21 smithi154 ceph-mon[33264]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-20T22:48:21.333 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:21 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/2882907803' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:48:21.333 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:21 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:48:21.333 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:21 smithi154 ceph-mon[33264]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T22:48:21.333 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:21 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T22:48:21.333 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:21 smithi154 ceph-mon[33264]: osdmap e421: 8 total, 8 up, 8 in 2024-09-20T22:48:21.359 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:48:21.379 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:21.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:21.380 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:48:21.381 INFO:tasks.rados.rados.0.smithi084.stdout:1619: finishing write tid 1 to smithi08477203-13 2024-09-20T22:48:21.381 INFO:tasks.rados.rados.0.smithi084.stdout:1619: finishing write tid 2 to smithi08477203-13 2024-09-20T22:48:21.381 INFO:tasks.rados.rados.0.smithi084.stdout:1619: finishing write tid 3 to smithi08477203-13 2024-09-20T22:48:21.381 INFO:tasks.rados.rados.0.smithi084.stdout:1619: finishing write tid 4 to smithi08477203-13 2024-09-20T22:48:21.381 INFO:tasks.rados.rados.0.smithi084.stdout:1619: finishing write tid 5 to smithi08477203-13 2024-09-20T22:48:21.381 INFO:tasks.rados.rados.0.smithi084.stdout:1619: oid 13 updating version 0 to 910 2024-09-20T22:48:21.381 INFO:tasks.rados.rados.0.smithi084.stdout:1619: oid 13 version 910 is already newer than 907 2024-09-20T22:48:21.381 INFO:tasks.rados.rados.0.smithi084.stdout:1619: oid 13 version 910 is already newer than 909 2024-09-20T22:48:21.381 INFO:tasks.rados.rados.0.smithi084.stdout:1619: oid 13 version 910 is already newer than 908 2024-09-20T22:48:21.381 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 13 v 910 (ObjNum 573 snap 172 seq_num 573) dirty exists 2024-09-20T22:48:21.381 INFO:tasks.rados.rados.0.smithi084.stdout:1619: left oid 13 (ObjNum 573 snap 172 seq_num 573) 2024-09-20T22:48:21.382 INFO:tasks.rados.rados.0.smithi084.stdout:1622: finishing write tid 1 to smithi08477203-43 2024-09-20T22:48:21.382 INFO:tasks.rados.rados.0.smithi084.stdout:1622: finishing write tid 2 to smithi08477203-43 2024-09-20T22:48:21.382 INFO:tasks.rados.rados.0.smithi084.stdout:1622: finishing write tid 3 to smithi08477203-43 2024-09-20T22:48:21.382 INFO:tasks.rados.rados.0.smithi084.stdout:1622: finishing write tid 4 to smithi08477203-43 2024-09-20T22:48:21.382 INFO:tasks.rados.rados.0.smithi084.stdout:1622: finishing write tid 5 to smithi08477203-43 2024-09-20T22:48:21.382 INFO:tasks.rados.rados.0.smithi084.stdout:1622: oid 43 updating version 0 to 99 2024-09-20T22:48:21.382 INFO:tasks.rados.rados.0.smithi084.stdout:1622: oid 43 updating version 99 to 101 2024-09-20T22:48:21.382 INFO:tasks.rados.rados.0.smithi084.stdout:1622: oid 43 updating version 101 to 102 2024-09-20T22:48:21.382 INFO:tasks.rados.rados.0.smithi084.stdout:1622: oid 43 version 102 is already newer than 100 2024-09-20T22:48:21.382 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 43 v 102 (ObjNum 574 snap 172 seq_num 574) dirty exists 2024-09-20T22:48:21.382 INFO:tasks.rados.rados.0.smithi084.stdout:1622: left oid 43 (ObjNum 574 snap 172 seq_num 574) 2024-09-20T22:48:21.382 INFO:tasks.rados.rados.0.smithi084.stdout:1624: finishing copy_from to smithi08477203-49 2024-09-20T22:48:21.382 INFO:tasks.rados.rados.0.smithi084.stdout:1624: got expected ENOENT (src dne) 2024-09-20T22:48:21.382 INFO:tasks.rados.rados.0.smithi084.stdout:1624: finishing copy_from racing read to smithi08477203-49 2024-09-20T22:48:21.382 INFO:tasks.rados.rados.0.smithi084.stdout:1626: finishing write tid 1 to smithi08477203-10 2024-09-20T22:48:21.383 INFO:tasks.rados.rados.0.smithi084.stdout:1626: finishing write tid 2 to smithi08477203-10 2024-09-20T22:48:21.383 INFO:tasks.rados.rados.0.smithi084.stdout:1626: finishing write tid 3 to smithi08477203-10 2024-09-20T22:48:21.383 INFO:tasks.rados.rados.0.smithi084.stdout:1626: finishing write tid 4 to smithi08477203-10 2024-09-20T22:48:21.383 INFO:tasks.rados.rados.0.smithi084.stdout:1626: finishing write tid 5 to smithi08477203-10 2024-09-20T22:48:21.383 INFO:tasks.rados.rados.0.smithi084.stdout:1626: oid 10 updating version 0 to 559 2024-09-20T22:48:21.383 INFO:tasks.rados.rados.0.smithi084.stdout:1626: oid 10 version 559 is already newer than 558 2024-09-20T22:48:21.383 INFO:tasks.rados.rados.0.smithi084.stdout:1626: oid 10 version 559 is already newer than 557 2024-09-20T22:48:21.383 INFO:tasks.rados.rados.0.smithi084.stdout:1626: oid 10 updating version 559 to 560 2024-09-20T22:48:21.383 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 10 v 560 (ObjNum 575 snap 172 seq_num 575) dirty exists 2024-09-20T22:48:21.383 INFO:tasks.rados.rados.0.smithi084.stdout:1626: left oid 10 (ObjNum 575 snap 172 seq_num 575) 2024-09-20T22:48:21.481 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:48:21.503 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:21.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:21.504 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:48:21.587 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:21 smithi084 ceph-mon[40024]: 3.12 scrub starts 2024-09-20T22:48:21.588 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:21 smithi084 ceph-mon[40024]: 3.12 scrub ok 2024-09-20T22:48:21.588 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:21 smithi084 ceph-mon[40024]: pgmap v531: 168 pgs: 6 unknown, 3 peering, 5 active+clean+snaptrim, 154 active+clean; 293 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail 2024-09-20T22:48:21.588 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:21 smithi084 ceph-mon[40024]: 4.5 scrub starts 2024-09-20T22:48:21.588 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:21 smithi084 ceph-mon[40024]: 4.5 scrub ok 2024-09-20T22:48:21.588 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:21 smithi084 ceph-mon[40024]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-20T22:48:21.588 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:21 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/2882907803' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:48:21.588 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:21 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:48:21.588 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:21 smithi084 ceph-mon[40024]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T22:48:21.588 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:21 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T22:48:21.588 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:21 smithi084 ceph-mon[40024]: osdmap e421: 8 total, 8 up, 8 in 2024-09-20T22:48:21.588 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:21 smithi084 ceph-mon[29774]: 3.12 scrub starts 2024-09-20T22:48:21.589 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:21 smithi084 ceph-mon[29774]: 3.12 scrub ok 2024-09-20T22:48:21.589 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:21 smithi084 ceph-mon[29774]: pgmap v531: 168 pgs: 6 unknown, 3 peering, 5 active+clean+snaptrim, 154 active+clean; 293 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail 2024-09-20T22:48:21.589 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:21 smithi084 ceph-mon[29774]: 4.5 scrub starts 2024-09-20T22:48:21.589 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:21 smithi084 ceph-mon[29774]: 4.5 scrub ok 2024-09-20T22:48:21.589 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:21 smithi084 ceph-mon[29774]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-20T22:48:21.589 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:21 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/2882907803' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:48:21.589 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:21 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:48:21.589 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:21 smithi084 ceph-mon[29774]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T22:48:21.590 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:21 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T22:48:21.590 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:21 smithi084 ceph-mon[29774]: osdmap e421: 8 total, 8 up, 8 in 2024-09-20T22:48:21.604 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:48:21.625 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:21.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:21.626 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:48:21.726 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:48:21.742 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-20T22:48:21.748 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:21.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:21.749 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:48:21.781 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-20T22:48:21.850 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:48:21.871 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:21.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:21.872 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:48:21.972 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:48:21.984 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:48:21.994 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:21.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:21.995 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:48:22.025 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 5, 6, 7, 0, 4] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-20T22:48:22.025 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-20T22:48:22.025 INFO:tasks.thrashosds.thrasher:inject_pause on osd.6 2024-09-20T22:48:22.025 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-20T22:48:22.025 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-20T22:48:22.026 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.6 config set filestore_inject_stall 3 2024-09-20T22:48:22.075 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:48:22.097 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:48:22.117 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:22.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:22.118 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:48:22.212 INFO:tasks.rados.rados.0.smithi084.stdout:1618: done (8 left) 2024-09-20T22:48:22.212 INFO:tasks.rados.rados.0.smithi084.stdout:1619: done (7 left) 2024-09-20T22:48:22.212 INFO:tasks.rados.rados.0.smithi084.stdout:1620: done (6 left) 2024-09-20T22:48:22.212 INFO:tasks.rados.rados.0.smithi084.stdout:1621: done (5 left) 2024-09-20T22:48:22.212 INFO:tasks.rados.rados.0.smithi084.stdout:1622: done (4 left) 2024-09-20T22:48:22.212 INFO:tasks.rados.rados.0.smithi084.stdout:1624: done (3 left) 2024-09-20T22:48:22.212 INFO:tasks.rados.rados.0.smithi084.stdout:1626: done (2 left) 2024-09-20T22:48:22.212 INFO:tasks.rados.rados.0.smithi084.stdout:1628: done (1 left) 2024-09-20T22:48:22.213 INFO:tasks.rados.rados.0.smithi084.stdout:1629: write (excl) oid 46 current snap is 173 2024-09-20T22:48:22.213 INFO:tasks.rados.rados.0.smithi084.stdout:1629: seq_num 577 ranges {497639=545148,1548463=407858,2529302=564422,3528143=169496} 2024-09-20T22:48:22.219 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:48:22.221 INFO:tasks.rados.rados.0.smithi084.stdout:1629: writing smithi08477203-46 from 497639 to 1042787 tid 1 2024-09-20T22:48:22.226 INFO:tasks.rados.rados.0.smithi084.stdout:1629: writing smithi08477203-46 from 1548463 to 1956321 tid 2 2024-09-20T22:48:22.233 INFO:tasks.rados.rados.0.smithi084.stdout:1629: writing smithi08477203-46 from 2529302 to 3093724 tid 3 2024-09-20T22:48:22.237 INFO:tasks.rados.rados.0.smithi084.stdout:1629: writing smithi08477203-46 from 3528143 to 3697639 tid 4 2024-09-20T22:48:22.237 INFO:tasks.rados.rados.0.smithi084.stdout:1630: write oid 43 current snap is 173 2024-09-20T22:48:22.237 INFO:tasks.rados.rados.0.smithi084.stdout:1627: finishing write tid 1 to smithi08477203-35 2024-09-20T22:48:22.237 INFO:tasks.rados.rados.0.smithi084.stdout:1630: seq_num 578 ranges {514446=736750,1723298=667869,3073078=641368} 2024-09-20T22:48:22.244 INFO:tasks.rados.rados.0.smithi084.stdout:1630: writing smithi08477203-43 from 514446 to 1251196 tid 1 2024-09-20T22:48:22.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:22.245 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:22.245 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:48:22.250 INFO:tasks.rados.rados.0.smithi084.stdout:1630: writing smithi08477203-43 from 1723298 to 2391167 tid 2 2024-09-20T22:48:22.258 INFO:tasks.rados.rados.0.smithi084.stdout:1630: writing smithi08477203-43 from 3073078 to 3714446 tid 3 2024-09-20T22:48:22.258 INFO:tasks.rados.rados.0.smithi084.stdout:1631: delete oid 4 current snap is 173 2024-09-20T22:48:22.258 INFO:tasks.rados.rados.0.smithi084.stdout:1627: finishing write tid 2 to smithi08477203-35 2024-09-20T22:48:22.258 INFO:tasks.rados.rados.0.smithi084.stdout:1627: finishing write tid 3 to smithi08477203-35 2024-09-20T22:48:22.259 INFO:tasks.rados.rados.0.smithi084.stdout:1627: finishing write tid 4 to smithi08477203-35 2024-09-20T22:48:22.259 INFO:tasks.rados.rados.0.smithi084.stdout:1627: finishing write tid 5 to smithi08477203-35 2024-09-20T22:48:22.259 INFO:tasks.rados.rados.0.smithi084.stdout:1627: oid 35 updating version 0 to 414 2024-09-20T22:48:22.259 INFO:tasks.rados.rados.0.smithi084.stdout:1627: oid 35 version 414 is already newer than 412 2024-09-20T22:48:22.259 INFO:tasks.rados.rados.0.smithi084.stdout:1627: oid 35 version 414 is already newer than 411 2024-09-20T22:48:22.259 INFO:tasks.rados.rados.0.smithi084.stdout:1627: oid 35 version 414 is already newer than 413 2024-09-20T22:48:22.259 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 35 v 414 (ObjNum 576 snap 172 seq_num 576) dirty exists 2024-09-20T22:48:22.259 INFO:tasks.rados.rados.0.smithi084.stdout:1627: left oid 35 (ObjNum 576 snap 172 seq_num 576) 2024-09-20T22:48:22.259 INFO:tasks.rados.rados.0.smithi084.stdout:1629: finishing write tid 1 to smithi08477203-46 2024-09-20T22:48:22.259 INFO:tasks.rados.rados.0.smithi084.stdout:1629: finishing write tid 2 to smithi08477203-46 2024-09-20T22:48:22.259 INFO:tasks.rados.rados.0.smithi084.stdout:1629: finishing write tid 3 to smithi08477203-46 2024-09-20T22:48:22.259 INFO:tasks.rados.rados.0.smithi084.stdout:1629: finishing write tid 4 to smithi08477203-46 2024-09-20T22:48:22.259 INFO:tasks.rados.rados.0.smithi084.stdout:1629: finishing write tid 5 to smithi08477203-46 2024-09-20T22:48:22.259 INFO:tasks.rados.rados.0.smithi084.stdout:1629: finishing write tid 6 to smithi08477203-46 2024-09-20T22:48:22.259 INFO:tasks.rados.rados.0.smithi084.stdout:1629: oid 46 updating version 0 to 726 2024-09-20T22:48:22.260 INFO:tasks.rados.rados.0.smithi084.stdout:1629: oid 46 version 726 is already newer than 724 2024-09-20T22:48:22.260 INFO:tasks.rados.rados.0.smithi084.stdout:1629: oid 46 updating version 726 to 727 2024-09-20T22:48:22.260 INFO:tasks.rados.rados.0.smithi084.stdout:1629: oid 46 version 727 is already newer than 725 2024-09-20T22:48:22.260 INFO:tasks.rados.rados.0.smithi084.stdout:1629: oid 46 version 727 is already newer than 723 2024-09-20T22:48:22.260 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 46 v 727 (ObjNum 577 snap 173 seq_num 577) dirty exists 2024-09-20T22:48:22.260 INFO:tasks.rados.rados.0.smithi084.stdout:1629: left oid 46 (ObjNum 577 snap 173 seq_num 577) 2024-09-20T22:48:22.260 INFO:tasks.rados.rados.0.smithi084.stdout:1630: finishing write tid 1 to smithi08477203-43 2024-09-20T22:48:22.260 INFO:tasks.rados.rados.0.smithi084.stdout:1630: finishing write tid 2 to smithi08477203-43 2024-09-20T22:48:22.261 INFO:tasks.rados.rados.0.smithi084.stdout:1630: finishing write tid 3 to smithi08477203-43 2024-09-20T22:48:22.261 INFO:tasks.rados.rados.0.smithi084.stdout:1630: finishing write tid 4 to smithi08477203-43 2024-09-20T22:48:22.261 INFO:tasks.rados.rados.0.smithi084.stdout:1627: done (3 left) 2024-09-20T22:48:22.261 INFO:tasks.rados.rados.0.smithi084.stdout:1629: done (2 left) 2024-09-20T22:48:22.261 INFO:tasks.rados.rados.0.smithi084.stdout:1631: done (1 left) 2024-09-20T22:48:22.261 INFO:tasks.rados.rados.0.smithi084.stdout:1632: rollback oid 39 current snap is 173 2024-09-20T22:48:22.261 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 39 to 171 2024-09-20T22:48:22.261 INFO:tasks.rados.rados.0.smithi084.stdout:1633: write (excl) oid 23 current snap is 173 2024-09-20T22:48:22.261 INFO:tasks.rados.rados.0.smithi084.stdout:1633: seq_num 580 ranges {548060=487248,1440262=671538,2844277=798238} 2024-09-20T22:48:22.265 INFO:tasks.rados.rados.0.smithi084.stdout:1633: writing smithi08477203-23 from 548060 to 1035308 tid 1 2024-09-20T22:48:22.271 INFO:tasks.rados.rados.0.smithi084.stdout:1633: writing smithi08477203-23 from 1440262 to 2111800 tid 2 2024-09-20T22:48:22.280 INFO:tasks.rados.rados.0.smithi084.stdout:1633: writing smithi08477203-23 from 2844277 to 3642515 tid 3 2024-09-20T22:48:22.280 INFO:tasks.rados.rados.0.smithi084.stdout:1634: write oid 8 current snap is 173 2024-09-20T22:48:22.280 INFO:tasks.rados.rados.0.smithi084.stdout:1634: seq_num 581 ranges {564867=562497,1898744=415196,2755347=725863} 2024-09-20T22:48:22.286 INFO:tasks.rados.rados.0.smithi084.stdout:1634: writing smithi08477203-8 from 564867 to 1127364 tid 1 2024-09-20T22:48:22.292 INFO:tasks.rados.rados.0.smithi084.stdout:1634: writing smithi08477203-8 from 1898744 to 2313940 tid 2 2024-09-20T22:48:22.299 INFO:tasks.rados.rados.0.smithi084.stdout:1634: writing smithi08477203-8 from 2755347 to 3481210 tid 3 2024-09-20T22:48:22.299 INFO:tasks.rados.rados.0.smithi084.stdout:1635: snap_remove snap 134 2024-09-20T22:48:22.316 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:22 smithi154 ceph-mon[33264]: 4.1c scrub starts 2024-09-20T22:48:22.316 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:22 smithi154 ceph-mon[33264]: 4.1c scrub ok 2024-09-20T22:48:22.316 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:22 smithi154 ceph-mon[33264]: 4.f scrub starts 2024-09-20T22:48:22.316 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:22 smithi154 ceph-mon[33264]: 4.f scrub ok 2024-09-20T22:48:22.316 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:22 smithi154 ceph-mon[33264]: osdmap e422: 8 total, 8 up, 8 in 2024-09-20T22:48:22.321 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:22 smithi084 ceph-mon[40024]: 4.1c scrub starts 2024-09-20T22:48:22.321 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:22 smithi084 ceph-mon[40024]: 4.1c scrub ok 2024-09-20T22:48:22.321 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:22 smithi084 ceph-mon[40024]: 4.f scrub starts 2024-09-20T22:48:22.321 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:22 smithi084 ceph-mon[40024]: 4.f scrub ok 2024-09-20T22:48:22.321 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:22 smithi084 ceph-mon[40024]: osdmap e422: 8 total, 8 up, 8 in 2024-09-20T22:48:22.321 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:22 smithi084 ceph-mon[29774]: 4.1c scrub starts 2024-09-20T22:48:22.322 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:22 smithi084 ceph-mon[29774]: 4.1c scrub ok 2024-09-20T22:48:22.322 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:22 smithi084 ceph-mon[29774]: 4.f scrub starts 2024-09-20T22:48:22.322 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:22 smithi084 ceph-mon[29774]: 4.f scrub ok 2024-09-20T22:48:22.322 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:22 smithi084 ceph-mon[29774]: osdmap e422: 8 total, 8 up, 8 in 2024-09-20T22:48:22.322 INFO:journalctl@ceph.osd.2.smithi084.stdout:Sep 20 22:48:22 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-2[55375]: 2024-09-20T22:48:22.220+0000 7fd475347640 -1 snap_mapper.get_next_objects_to_trim::New Clone-Objects were added to Snap 93 after trimming was started 2024-09-20T22:48:22.346 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:48:22.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:22.388 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:22.389 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:48:22.489 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:48:22.511 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:22.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:22.512 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:48:22.551 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:48:22.551 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:48:22.567 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:48:22.567 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:48:22.579 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-20T22:48:22.579 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:48:22.583 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:48:22.583 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:48:22.599 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-20T22:48:22.599 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T22:48:22.612 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:48:22.615 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:48:22.615 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:48:22.632 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:48:22.632 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:48:22.633 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:22.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:22.633 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:48:22.648 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:48:22.649 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:48:22.665 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:48:22.665 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:48:22.734 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:48:22.755 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:22.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:22.756 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:48:22.796 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:48:22.856 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:48:22.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:22.881 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:22.882 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:48:22.982 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:48:23.004 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:23.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:23.005 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:48:23.106 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:48:23.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:23.128 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:23.128 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:48:23.229 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:48:23.253 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:23.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:23.254 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:48:23.257 INFO:tasks.rados.rados.0.smithi084.stdout:1635: done (4 left) 2024-09-20T22:48:23.257 INFO:tasks.rados.rados.0.smithi084.stdout:1636: read oid 40 snap -1 2024-09-20T22:48:23.257 INFO:tasks.rados.rados.0.smithi084.stdout:1636: expect deleted 2024-09-20T22:48:23.258 INFO:tasks.rados.rados.0.smithi084.stdout:1637: delete oid 30 current snap is 173 2024-09-20T22:48:23.258 INFO:tasks.rados.rados.0.smithi084.stdout:1630: finishing write tid 5 to smithi08477203-43 2024-09-20T22:48:23.258 INFO:tasks.rados.rados.0.smithi084.stdout:1630: oid 43 updating version 0 to 108 2024-09-20T22:48:23.258 INFO:tasks.rados.rados.0.smithi084.stdout:1630: oid 43 updating version 108 to 111 2024-09-20T22:48:23.258 INFO:tasks.rados.rados.0.smithi084.stdout:1630: oid 43 version 111 is already newer than 110 2024-09-20T22:48:23.258 INFO:tasks.rados.rados.0.smithi084.stdout:1630: oid 43 version 111 is already newer than 109 2024-09-20T22:48:23.258 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 43 v 111 (ObjNum 578 snap 173 seq_num 578) dirty exists 2024-09-20T22:48:23.258 INFO:tasks.rados.rados.0.smithi084.stdout:1630: left oid 43 (ObjNum 578 snap 173 seq_num 578) 2024-09-20T22:48:23.258 INFO:tasks.rados.rados.0.smithi084.stdout:1632: finishing rollback tid 1 to smithi08477203-39 2024-09-20T22:48:23.258 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 39 v 533 (ObjNum 289 snap 0 seq_num 1) dirty dne 2024-09-20T22:48:23.258 INFO:tasks.rados.rados.0.smithi084.stdout:1633: finishing write tid 1 to smithi08477203-23 2024-09-20T22:48:23.258 INFO:tasks.rados.rados.0.smithi084.stdout:1633: finishing write tid 2 to smithi08477203-23 2024-09-20T22:48:23.258 INFO:tasks.rados.rados.0.smithi084.stdout:1633: finishing write tid 3 to smithi08477203-23 2024-09-20T22:48:23.258 INFO:tasks.rados.rados.0.smithi084.stdout:1633: finishing write tid 4 to smithi08477203-23 2024-09-20T22:48:23.259 INFO:tasks.rados.rados.0.smithi084.stdout:1633: finishing write tid 5 to smithi08477203-23 2024-09-20T22:48:23.259 INFO:tasks.rados.rados.0.smithi084.stdout:1633: oid 23 updating version 0 to 699 2024-09-20T22:48:23.259 INFO:tasks.rados.rados.0.smithi084.stdout:1633: oid 23 version 699 is already newer than 698 2024-09-20T22:48:23.259 INFO:tasks.rados.rados.0.smithi084.stdout:1633: oid 23 version 699 is already newer than 697 2024-09-20T22:48:23.259 INFO:tasks.rados.rados.0.smithi084.stdout:1633: oid 23 updating version 699 to 700 2024-09-20T22:48:23.259 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 23 v 700 (ObjNum 580 snap 173 seq_num 580) dirty exists 2024-09-20T22:48:23.259 INFO:tasks.rados.rados.0.smithi084.stdout:1633: left oid 23 (ObjNum 580 snap 173 seq_num 580) 2024-09-20T22:48:23.259 INFO:tasks.rados.rados.0.smithi084.stdout:1634: finishing write tid 1 to smithi08477203-8 2024-09-20T22:48:23.259 INFO:tasks.rados.rados.0.smithi084.stdout:1634: finishing write tid 2 to smithi08477203-8 2024-09-20T22:48:23.259 INFO:tasks.rados.rados.0.smithi084.stdout:1634: finishing write tid 3 to smithi08477203-8 2024-09-20T22:48:23.260 INFO:tasks.rados.rados.0.smithi084.stdout:1634: finishing write tid 4 to smithi08477203-8 2024-09-20T22:48:23.260 INFO:tasks.rados.rados.0.smithi084.stdout:1634: finishing write tid 5 to smithi08477203-8 2024-09-20T22:48:23.260 INFO:tasks.rados.rados.0.smithi084.stdout:1634: oid 8 updating version 0 to 523 2024-09-20T22:48:23.260 INFO:tasks.rados.rados.0.smithi084.stdout:1634: oid 8 version 523 is already newer than 522 2024-09-20T22:48:23.260 INFO:tasks.rados.rados.0.smithi084.stdout:1634: oid 8 version 523 is already newer than 521 2024-09-20T22:48:23.260 INFO:tasks.rados.rados.0.smithi084.stdout:1634: oid 8 version 523 is already newer than 520 2024-09-20T22:48:23.260 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 8 v 523 (ObjNum 581 snap 173 seq_num 581) dirty exists 2024-09-20T22:48:23.260 INFO:tasks.rados.rados.0.smithi084.stdout:1634: left oid 8 (ObjNum 581 snap 173 seq_num 581) 2024-09-20T22:48:23.262 INFO:tasks.rados.rados.0.smithi084.stdout:1630: done (5 left) 2024-09-20T22:48:23.263 INFO:tasks.rados.rados.0.smithi084.stdout:1632: done (4 left) 2024-09-20T22:48:23.263 INFO:tasks.rados.rados.0.smithi084.stdout:1633: done (3 left) 2024-09-20T22:48:23.263 INFO:tasks.rados.rados.0.smithi084.stdout:1634: done (2 left) 2024-09-20T22:48:23.263 INFO:tasks.rados.rados.0.smithi084.stdout:1636: done (1 left) 2024-09-20T22:48:23.263 INFO:tasks.rados.rados.0.smithi084.stdout:1637: done (0 left) 2024-09-20T22:48:23.263 INFO:tasks.rados.rados.0.smithi084.stdout:1638: rollback oid 5 current snap is 173 2024-09-20T22:48:23.263 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 5 to 146 2024-09-20T22:48:23.263 INFO:tasks.rados.rados.0.smithi084.stdout:1639: write oid 29 current snap is 173 2024-09-20T22:48:23.263 INFO:tasks.rados.rados.0.smithi084.stdout:1639: seq_num 583 ranges {598481=712995,1899355=418865,3093840=704641} 2024-09-20T22:48:23.270 INFO:tasks.rados.rados.0.smithi084.stdout:1639: writing smithi08477203-29 from 598481 to 1311476 tid 1 2024-09-20T22:48:23.273 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.6 dump_blocked_ops 2024-09-20T22:48:23.275 INFO:tasks.rados.rados.0.smithi084.stdout:1639: writing smithi08477203-29 from 1899355 to 2318220 tid 2 2024-09-20T22:48:23.284 INFO:tasks.rados.rados.0.smithi084.stdout:1639: writing smithi08477203-29 from 3093840 to 3798481 tid 3 2024-09-20T22:48:23.285 INFO:tasks.rados.rados.0.smithi084.stdout:1640: read oid 10 snap -1 2024-09-20T22:48:23.285 INFO:tasks.rados.rados.0.smithi084.stdout:1640: expect (ObjNum 575 snap 172 seq_num 575) 2024-09-20T22:48:23.285 INFO:tasks.rados.rados.0.smithi084.stdout:1641: read oid 2 snap -1 2024-09-20T22:48:23.285 INFO:tasks.rados.rados.0.smithi084.stdout:1641: expect (ObjNum 541 snap 164 seq_num 541) 2024-09-20T22:48:23.285 INFO:tasks.rados.rados.0.smithi084.stdout:1642: read oid 50 snap 147 2024-09-20T22:48:23.285 INFO:tasks.rados.rados.0.smithi084.stdout:1642: expect (ObjNum 505 snap 143 seq_num 505) 2024-09-20T22:48:23.285 INFO:tasks.rados.rados.0.smithi084.stdout:1643: delete oid 20 current snap is 173 2024-09-20T22:48:23.285 INFO:tasks.rados.rados.0.smithi084.stdout:1638: finishing rollback tid 0 to smithi08477203-5 2024-09-20T22:48:23.285 INFO:tasks.rados.rados.0.smithi084.stdout:1638: finishing rollback tid 1 to smithi08477203-5 2024-09-20T22:48:23.285 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 5 v 416 (ObjNum 33 snap 0 seq_num 2091917760) dirty dne 2024-09-20T22:48:23.285 INFO:tasks.rados.rados.0.smithi084.stdout:1639: finishing write tid 1 to smithi08477203-29 2024-09-20T22:48:23.286 INFO:tasks.rados.rados.0.smithi084.stdout:1639: finishing write tid 2 to smithi08477203-29 2024-09-20T22:48:23.287 INFO:tasks.rados.rados.0.smithi084.stdout:1638: done (5 left) 2024-09-20T22:48:23.287 INFO:tasks.rados.rados.0.smithi084.stdout:1643: done (4 left) 2024-09-20T22:48:23.287 INFO:tasks.rados.rados.0.smithi084.stdout:1644: delete oid 44 current snap is 173 2024-09-20T22:48:23.289 INFO:tasks.rados.rados.0.smithi084.stdout:1639: finishing write tid 3 to smithi08477203-29 2024-09-20T22:48:23.289 INFO:tasks.rados.rados.0.smithi084.stdout:1639: finishing write tid 4 to smithi08477203-29 2024-09-20T22:48:23.290 INFO:tasks.rados.rados.0.smithi084.stdout:1639: finishing write tid 5 to smithi08477203-29 2024-09-20T22:48:23.290 INFO:tasks.rados.rados.0.smithi084.stdout:1639: oid 29 updating version 0 to 937 2024-09-20T22:48:23.290 INFO:tasks.rados.rados.0.smithi084.stdout:1639: oid 29 version 937 is already newer than 935 2024-09-20T22:48:23.290 INFO:tasks.rados.rados.0.smithi084.stdout:1639: oid 29 version 937 is already newer than 936 2024-09-20T22:48:23.290 INFO:tasks.rados.rados.0.smithi084.stdout:1639: oid 29 version 937 is already newer than 934 2024-09-20T22:48:23.290 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 29 v 937 (ObjNum 583 snap 173 seq_num 583) dirty exists 2024-09-20T22:48:23.290 INFO:tasks.rados.rados.0.smithi084.stdout:1639: left oid 29 (ObjNum 583 snap 173 seq_num 583) 2024-09-20T22:48:23.292 INFO:tasks.rados.rados.0.smithi084.stdout:1639: done (4 left) 2024-09-20T22:48:23.292 INFO:tasks.rados.rados.0.smithi084.stdout:1644: done (3 left) 2024-09-20T22:48:23.292 INFO:tasks.rados.rados.0.smithi084.stdout:1645: read oid 20 snap 167 2024-09-20T22:48:23.292 INFO:tasks.rados.rados.0.smithi084.stdout:1645: expect deleted 2024-09-20T22:48:23.292 INFO:tasks.rados.rados.0.smithi084.stdout:1646: copy_from oid 41 from oid 35 current snap is 173 2024-09-20T22:48:23.292 INFO:tasks.rados.rados.0.smithi084.stdout:1647: snap_create 2024-09-20T22:48:23.292 INFO:tasks.rados.rados.0.smithi084.stdout:1640: expect (ObjNum 575 snap 172 seq_num 575) 2024-09-20T22:48:23.354 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:48:23.376 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:23.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:23.377 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:48:23.477 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:48:23.496 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:23 smithi154 ceph-mon[33264]: pgmap v534: 168 pgs: 6 unknown, 3 peering, 5 active+clean+snaptrim, 154 active+clean; 293 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail 2024-09-20T22:48:23.496 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:23 smithi154 ceph-mon[33264]: 5.8 scrub starts 2024-09-20T22:48:23.496 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:23 smithi154 ceph-mon[33264]: 5.8 scrub ok 2024-09-20T22:48:23.496 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:23 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/2373541626' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:48:23.496 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:23 smithi154 ceph-mon[33264]: 7.1 scrub starts 2024-09-20T22:48:23.496 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:23 smithi154 ceph-mon[33264]: 7.1 scrub ok 2024-09-20T22:48:23.496 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:23 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:48:23.496 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:23 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:48:23.498 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:23.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:23.499 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:48:23.541 INFO:tasks.rados.rados.0.smithi084.stdout:1641: expect (ObjNum 541 snap 164 seq_num 541) 2024-09-20T22:48:23.599 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:48:23.614 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:23 smithi084 ceph-mon[40024]: pgmap v534: 168 pgs: 6 unknown, 3 peering, 5 active+clean+snaptrim, 154 active+clean; 293 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail 2024-09-20T22:48:23.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:23 smithi084 ceph-mon[40024]: 5.8 scrub starts 2024-09-20T22:48:23.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:23 smithi084 ceph-mon[40024]: 5.8 scrub ok 2024-09-20T22:48:23.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:23 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/2373541626' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:48:23.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:23 smithi084 ceph-mon[40024]: 7.1 scrub starts 2024-09-20T22:48:23.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:23 smithi084 ceph-mon[40024]: 7.1 scrub ok 2024-09-20T22:48:23.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:23 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:48:23.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:23 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:48:23.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:23 smithi084 ceph-mon[29774]: pgmap v534: 168 pgs: 6 unknown, 3 peering, 5 active+clean+snaptrim, 154 active+clean; 293 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail 2024-09-20T22:48:23.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:23 smithi084 ceph-mon[29774]: 5.8 scrub starts 2024-09-20T22:48:23.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:23 smithi084 ceph-mon[29774]: 5.8 scrub ok 2024-09-20T22:48:23.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:23 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/2373541626' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:48:23.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:23 smithi084 ceph-mon[29774]: 7.1 scrub starts 2024-09-20T22:48:23.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:23 smithi084 ceph-mon[29774]: 7.1 scrub ok 2024-09-20T22:48:23.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:23 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:48:23.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:23 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:48:23.621 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:23.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:23.622 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:48:23.696 INFO:tasks.rados.rados.0.smithi084.stdout:1642: expect (ObjNum 505 snap 143 seq_num 505) 2024-09-20T22:48:23.723 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:48:23.744 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:23.745 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-20T22:48:23.745 INFO:teuthology.orchestra.run.smithi154.stdout: "success": "" 2024-09-20T22:48:23.745 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-20T22:48:23.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:23.746 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:48:23.847 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:48:23.850 INFO:tasks.rados.rados.0.smithi084.stdout:1646: finishing copy_from racing read to smithi08477203-41 2024-09-20T22:48:23.850 INFO:tasks.rados.rados.0.smithi084.stdout:1646: finishing copy_from to smithi08477203-41 2024-09-20T22:48:23.850 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 41 v 414 (ObjNum 576 snap 172 seq_num 576) dirty exists 2024-09-20T22:48:23.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:23.872 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:23.872 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:48:23.973 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:48:23.994 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:23.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:23.995 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:48:24.012 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:48:24.026 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd set noscrub 2024-09-20T22:48:24.096 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:48:24.117 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:24.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:24.118 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:48:24.219 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:48:24.240 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:24.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:24.241 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:48:24.254 INFO:tasks.rados.rados.0.smithi084.stdout:1640: done (5 left) 2024-09-20T22:48:24.255 INFO:tasks.rados.rados.0.smithi084.stdout:1641: done (4 left) 2024-09-20T22:48:24.255 INFO:tasks.rados.rados.0.smithi084.stdout:1642: done (3 left) 2024-09-20T22:48:24.255 INFO:tasks.rados.rados.0.smithi084.stdout:1645: done (2 left) 2024-09-20T22:48:24.255 INFO:tasks.rados.rados.0.smithi084.stdout:1646: done (1 left) 2024-09-20T22:48:24.255 INFO:tasks.rados.rados.0.smithi084.stdout:1647: done (0 left) 2024-09-20T22:48:24.255 INFO:tasks.rados.rados.0.smithi084.stdout:1648: delete oid 24 current snap is 174 2024-09-20T22:48:24.259 INFO:tasks.rados.rados.0.smithi084.stdout:1648: done (0 left) 2024-09-20T22:48:24.259 INFO:tasks.rados.rados.0.smithi084.stdout:1649: snap_remove snap 146 2024-09-20T22:48:24.281 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:48:24.342 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:48:24.363 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:24.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:24.364 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:48:24.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:24 smithi084 ceph-mon[40024]: 7.5 scrub starts 2024-09-20T22:48:24.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:24 smithi084 ceph-mon[40024]: 7.5 scrub ok 2024-09-20T22:48:24.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:24 smithi084 ceph-mon[40024]: 4.10 scrub starts 2024-09-20T22:48:24.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:24 smithi084 ceph-mon[40024]: 4.10 scrub ok 2024-09-20T22:48:24.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:24 smithi084 ceph-mon[40024]: osdmap e423: 8 total, 8 up, 8 in 2024-09-20T22:48:24.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:24 smithi084 ceph-mon[40024]: 3.4 deep-scrub starts 2024-09-20T22:48:24.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:24 smithi084 ceph-mon[40024]: 3.4 deep-scrub ok 2024-09-20T22:48:24.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:24 smithi084 ceph-mon[40024]: purged_snaps scrub starts 2024-09-20T22:48:24.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:24 smithi084 ceph-mon[40024]: purged_snaps scrub ok 2024-09-20T22:48:24.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:24 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "25"}]: dispatch 2024-09-20T22:48:24.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:24 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "25"}]: dispatch 2024-09-20T22:48:24.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:24 smithi084 ceph-mon[40024]: 5.1c scrub starts 2024-09-20T22:48:24.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:24 smithi084 ceph-mon[40024]: 5.1c scrub ok 2024-09-20T22:48:24.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:24 smithi084 ceph-mon[29774]: 7.5 scrub starts 2024-09-20T22:48:24.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:24 smithi084 ceph-mon[29774]: 7.5 scrub ok 2024-09-20T22:48:24.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:24 smithi084 ceph-mon[29774]: 4.10 scrub starts 2024-09-20T22:48:24.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:24 smithi084 ceph-mon[29774]: 4.10 scrub ok 2024-09-20T22:48:24.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:24 smithi084 ceph-mon[29774]: osdmap e423: 8 total, 8 up, 8 in 2024-09-20T22:48:24.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:24 smithi084 ceph-mon[29774]: 3.4 deep-scrub starts 2024-09-20T22:48:24.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:24 smithi084 ceph-mon[29774]: 3.4 deep-scrub ok 2024-09-20T22:48:24.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:24 smithi084 ceph-mon[29774]: purged_snaps scrub starts 2024-09-20T22:48:24.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:24 smithi084 ceph-mon[29774]: purged_snaps scrub ok 2024-09-20T22:48:24.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:24 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "25"}]: dispatch 2024-09-20T22:48:24.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:24 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "25"}]: dispatch 2024-09-20T22:48:24.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:24 smithi084 ceph-mon[29774]: 5.1c scrub starts 2024-09-20T22:48:24.367 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:24 smithi084 ceph-mon[29774]: 5.1c scrub ok 2024-09-20T22:48:24.417 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:24 smithi154 ceph-mon[33264]: 7.5 scrub starts 2024-09-20T22:48:24.417 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:24 smithi154 ceph-mon[33264]: 7.5 scrub ok 2024-09-20T22:48:24.417 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:24 smithi154 ceph-mon[33264]: 4.10 scrub starts 2024-09-20T22:48:24.417 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:24 smithi154 ceph-mon[33264]: 4.10 scrub ok 2024-09-20T22:48:24.417 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:24 smithi154 ceph-mon[33264]: osdmap e423: 8 total, 8 up, 8 in 2024-09-20T22:48:24.417 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:24 smithi154 ceph-mon[33264]: 3.4 deep-scrub starts 2024-09-20T22:48:24.417 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:24 smithi154 ceph-mon[33264]: 3.4 deep-scrub ok 2024-09-20T22:48:24.417 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:24 smithi154 ceph-mon[33264]: purged_snaps scrub starts 2024-09-20T22:48:24.417 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:24 smithi154 ceph-mon[33264]: purged_snaps scrub ok 2024-09-20T22:48:24.418 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:24 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "25"}]: dispatch 2024-09-20T22:48:24.418 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:24 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "25"}]: dispatch 2024-09-20T22:48:24.418 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:24 smithi154 ceph-mon[33264]: 5.1c scrub starts 2024-09-20T22:48:24.418 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:24 smithi154 ceph-mon[33264]: 5.1c scrub ok 2024-09-20T22:48:24.465 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:48:24.486 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:24.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:24.487 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:48:24.587 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:48:24.609 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:24.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:24.610 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:48:24.711 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:48:24.733 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:24.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:24.734 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:48:24.834 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:48:24.856 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:24.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:24.857 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:48:24.943 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-20T22:48:24.943 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [], 2024-09-20T22:48:24.943 INFO:teuthology.orchestra.run.smithi154.stdout: "complaint_time": 30, 2024-09-20T22:48:24.943 INFO:teuthology.orchestra.run.smithi154.stdout: "num_blocked_ops": 0 2024-09-20T22:48:24.943 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-20T22:48:24.958 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:48:24.979 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:24.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:24.980 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:48:25.081 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:48:25.102 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:25.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:25.103 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:48:25.204 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:48:25.225 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:25.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:25.226 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:48:25.252 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is set 2024-09-20T22:48:25.256 INFO:tasks.rados.rados.0.smithi084.stdout:1649: done (0 left) 2024-09-20T22:48:25.256 INFO:tasks.rados.rados.0.smithi084.stdout:1650: read oid 44 snap -1 2024-09-20T22:48:25.256 INFO:tasks.rados.rados.0.smithi084.stdout:1650: expect deleted 2024-09-20T22:48:25.256 INFO:tasks.rados.rados.0.smithi084.stdout:1651: read oid 8 snap 152 2024-09-20T22:48:25.256 INFO:tasks.rados.rados.0.smithi084.stdout:1651: expect deleted 2024-09-20T22:48:25.256 INFO:tasks.rados.rados.0.smithi084.stdout:1652: rollback oid 30 current snap is 174 2024-09-20T22:48:25.256 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 30 to 148 2024-09-20T22:48:25.256 INFO:tasks.rados.rados.0.smithi084.stdout:1653: snap_remove snap 170 2024-09-20T22:48:25.327 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:48:25.349 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:25.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:25.350 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:48:25.393 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-20T22:48:25.450 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:48:25.472 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:25.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:25.472 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:48:25.573 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:48:25.594 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:25.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:25.595 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:48:25.612 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:25 smithi154 ceph-mon[33264]: 7.9 scrub starts 2024-09-20T22:48:25.612 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:25 smithi154 ceph-mon[33264]: 7.9 scrub ok 2024-09-20T22:48:25.612 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:25 smithi154 ceph-mon[33264]: pgmap v537: 168 pgs: 3 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 159 active+clean; 391 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 141 MiB/s rd, 39 MiB/s wr, 242 op/s; 23 MiB/s, 50 objects/s recovering 2024-09-20T22:48:25.613 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:25 smithi154 ceph-mon[33264]: 7.18 scrub starts 2024-09-20T22:48:25.613 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:25 smithi154 ceph-mon[33264]: 7.18 scrub ok 2024-09-20T22:48:25.613 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:25 smithi154 ceph-mon[33264]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-20T22:48:25.613 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:25 smithi154 ceph-mon[33264]: Cluster is now healthy 2024-09-20T22:48:25.613 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:25 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "25"}]': finished 2024-09-20T22:48:25.613 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:25 smithi154 ceph-mon[33264]: osdmap e424: 8 total, 8 up, 8 in 2024-09-20T22:48:25.613 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:25 smithi154 ceph-mon[33264]: 4.2 scrub starts 2024-09-20T22:48:25.613 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:25 smithi154 ceph-mon[33264]: 4.2 scrub ok 2024-09-20T22:48:25.613 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:25 smithi154 ceph-mon[33264]: 1.a scrub starts 2024-09-20T22:48:25.613 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:25 smithi154 ceph-mon[33264]: 1.a scrub ok 2024-09-20T22:48:25.613 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:25 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/102672119' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-20T22:48:25.613 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:25 smithi154 ceph-mon[33264]: 3.1a deep-scrub starts 2024-09-20T22:48:25.613 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:25 smithi154 ceph-mon[33264]: 3.1a deep-scrub ok 2024-09-20T22:48:25.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:25 smithi084 ceph-mon[40024]: 7.9 scrub starts 2024-09-20T22:48:25.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:25 smithi084 ceph-mon[40024]: 7.9 scrub ok 2024-09-20T22:48:25.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:25 smithi084 ceph-mon[40024]: pgmap v537: 168 pgs: 3 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 159 active+clean; 391 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 141 MiB/s rd, 39 MiB/s wr, 242 op/s; 23 MiB/s, 50 objects/s recovering 2024-09-20T22:48:25.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:25 smithi084 ceph-mon[40024]: 7.18 scrub starts 2024-09-20T22:48:25.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:25 smithi084 ceph-mon[40024]: 7.18 scrub ok 2024-09-20T22:48:25.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:25 smithi084 ceph-mon[40024]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-20T22:48:25.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:25 smithi084 ceph-mon[40024]: Cluster is now healthy 2024-09-20T22:48:25.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:25 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "25"}]': finished 2024-09-20T22:48:25.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:25 smithi084 ceph-mon[40024]: osdmap e424: 8 total, 8 up, 8 in 2024-09-20T22:48:25.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:25 smithi084 ceph-mon[40024]: 4.2 scrub starts 2024-09-20T22:48:25.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:25 smithi084 ceph-mon[40024]: 4.2 scrub ok 2024-09-20T22:48:25.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:25 smithi084 ceph-mon[40024]: 1.a scrub starts 2024-09-20T22:48:25.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:25 smithi084 ceph-mon[40024]: 1.a scrub ok 2024-09-20T22:48:25.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:25 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/102672119' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-20T22:48:25.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:25 smithi084 ceph-mon[40024]: 3.1a deep-scrub starts 2024-09-20T22:48:25.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:25 smithi084 ceph-mon[40024]: 3.1a deep-scrub ok 2024-09-20T22:48:25.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:25 smithi084 ceph-mon[29774]: 7.9 scrub starts 2024-09-20T22:48:25.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:25 smithi084 ceph-mon[29774]: 7.9 scrub ok 2024-09-20T22:48:25.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:25 smithi084 ceph-mon[29774]: pgmap v537: 168 pgs: 3 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 159 active+clean; 391 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 141 MiB/s rd, 39 MiB/s wr, 242 op/s; 23 MiB/s, 50 objects/s recovering 2024-09-20T22:48:25.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:25 smithi084 ceph-mon[29774]: 7.18 scrub starts 2024-09-20T22:48:25.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:25 smithi084 ceph-mon[29774]: 7.18 scrub ok 2024-09-20T22:48:25.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:25 smithi084 ceph-mon[29774]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-20T22:48:25.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:25 smithi084 ceph-mon[29774]: Cluster is now healthy 2024-09-20T22:48:25.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:25 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "25"}]': finished 2024-09-20T22:48:25.617 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:25 smithi084 ceph-mon[29774]: osdmap e424: 8 total, 8 up, 8 in 2024-09-20T22:48:25.617 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:25 smithi084 ceph-mon[29774]: 4.2 scrub starts 2024-09-20T22:48:25.617 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:25 smithi084 ceph-mon[29774]: 4.2 scrub ok 2024-09-20T22:48:25.617 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:25 smithi084 ceph-mon[29774]: 1.a scrub starts 2024-09-20T22:48:25.617 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:25 smithi084 ceph-mon[29774]: 1.a scrub ok 2024-09-20T22:48:25.617 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:25 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/102672119' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-20T22:48:25.617 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:25 smithi084 ceph-mon[29774]: 3.1a deep-scrub starts 2024-09-20T22:48:25.617 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:25 smithi084 ceph-mon[29774]: 3.1a deep-scrub ok 2024-09-20T22:48:25.678 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.6 dump_historic_ops 2024-09-20T22:48:25.696 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:48:25.740 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:25.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:25.741 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:48:25.842 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:48:25.863 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:25.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:25.864 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:48:25.962 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:48:25.965 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:48:25.987 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:25.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:25.988 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:48:26.089 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:48:26.110 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:26.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:26.111 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:48:26.169 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:48:26.212 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:48:26.233 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:26.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:26.234 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:48:26.275 INFO:tasks.rados.rados.0.smithi084.stdout:1653: done (3 left) 2024-09-20T22:48:26.275 INFO:tasks.rados.rados.0.smithi084.stdout:1654: write (excl) oid 13 current snap is 174 2024-09-20T22:48:26.275 INFO:tasks.rados.rados.0.smithi084.stdout:1654: seq_num 587 ranges {665709=730344,2133283=426203,3203532=509848} 2024-09-20T22:48:26.282 INFO:tasks.rados.rados.0.smithi084.stdout:1654: writing smithi08477203-13 from 665709 to 1396053 tid 1 2024-09-20T22:48:26.289 INFO:tasks.rados.rados.0.smithi084.stdout:1654: writing smithi08477203-13 from 2133283 to 2559486 tid 2 2024-09-20T22:48:26.289 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:26 smithi084 ceph-mon[29774]: 3.15 scrub starts 2024-09-20T22:48:26.290 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:26 smithi084 ceph-mon[29774]: 3.15 scrub ok 2024-09-20T22:48:26.290 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:26 smithi084 ceph-mon[29774]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:48:26.290 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:26 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/102672119' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T22:48:26.290 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:26 smithi084 ceph-mon[29774]: osdmap e425: 8 total, 8 up, 8 in 2024-09-20T22:48:26.290 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:26 smithi084 ceph-mon[40024]: 3.15 scrub starts 2024-09-20T22:48:26.290 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:26 smithi084 ceph-mon[40024]: 3.15 scrub ok 2024-09-20T22:48:26.290 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:26 smithi084 ceph-mon[40024]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:48:26.290 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:26 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/102672119' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T22:48:26.290 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:26 smithi084 ceph-mon[40024]: osdmap e425: 8 total, 8 up, 8 in 2024-09-20T22:48:26.295 INFO:tasks.rados.rados.0.smithi084.stdout:1654: writing smithi08477203-13 from 3203532 to 3713380 tid 3 2024-09-20T22:48:26.295 INFO:tasks.rados.rados.0.smithi084.stdout:1655: delete oid 3 current snap is 174 2024-09-20T22:48:26.295 INFO:tasks.rados.rados.0.smithi084.stdout:1654: finishing write tid 1 to smithi08477203-13 2024-09-20T22:48:26.295 INFO:tasks.rados.rados.0.smithi084.stdout:1654: finishing write tid 2 to smithi08477203-13 2024-09-20T22:48:26.298 INFO:tasks.rados.rados.0.smithi084.stdout:1654: finishing write tid 3 to smithi08477203-13 2024-09-20T22:48:26.298 INFO:tasks.rados.rados.0.smithi084.stdout:1654: finishing write tid 4 to smithi08477203-13 2024-09-20T22:48:26.298 INFO:tasks.rados.rados.0.smithi084.stdout:1654: finishing write tid 5 to smithi08477203-13 2024-09-20T22:48:26.298 INFO:tasks.rados.rados.0.smithi084.stdout:1654: oid 13 updating version 0 to 949 2024-09-20T22:48:26.298 INFO:tasks.rados.rados.0.smithi084.stdout:1654: oid 13 updating version 949 to 964 2024-09-20T22:48:26.298 INFO:tasks.rados.rados.0.smithi084.stdout:1654: oid 13 version 964 is already newer than 963 2024-09-20T22:48:26.298 INFO:tasks.rados.rados.0.smithi084.stdout:1654: oid 13 version 964 is already newer than 954 2024-09-20T22:48:26.298 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 13 v 964 (ObjNum 587 snap 174 seq_num 587) dirty exists 2024-09-20T22:48:26.298 INFO:tasks.rados.rados.0.smithi084.stdout:1654: left oid 13 (ObjNum 587 snap 174 seq_num 587) 2024-09-20T22:48:26.335 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:48:26.357 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:26.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:26.358 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:48:26.388 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:26 smithi154 ceph-mon[33264]: 3.15 scrub starts 2024-09-20T22:48:26.388 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:26 smithi154 ceph-mon[33264]: 3.15 scrub ok 2024-09-20T22:48:26.388 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:26 smithi154 ceph-mon[33264]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:48:26.388 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:26 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/102672119' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T22:48:26.388 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:26 smithi154 ceph-mon[33264]: osdmap e425: 8 total, 8 up, 8 in 2024-09-20T22:48:26.459 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:48:26.480 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:26.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:26.481 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:48:26.504 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-20T22:48:26.505 INFO:teuthology.orchestra.run.smithi154.stdout: "size": 20, 2024-09-20T22:48:26.505 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 600, 2024-09-20T22:48:26.505 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [ 2024-09-20T22:48:26.505 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.505 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:336 7.7 7:e67788e2:::smithi08477203-31:head [copy-from ver 30] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-09-20T22:48:26.505 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:06.542956+0000", 2024-09-20T22:48:26.505 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 319.94318070499997, 2024-09-20T22:48:26.505 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.019630893999999999, 2024-09-20T22:48:26.505 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:48:26.506 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:48:26.506 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:48:26.506 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:48:26.506 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:48:26.506 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:48:26.506 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 336 2024-09-20T22:48:26.506 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.506 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:48:26.506 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.506 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:48:26.506 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:06.542956+0000", 2024-09-20T22:48:26.506 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:48:26.506 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.507 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.507 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:48:26.507 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:06.542956+0000", 2024-09-20T22:48:26.507 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:48:26.507 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.507 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.507 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:48:26.507 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:06.542957+0000", 2024-09-20T22:48:26.507 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2580000000000001e-06 2024-09-20T22:48:26.507 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.507 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.507 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:48:26.507 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:06.542967+0000", 2024-09-20T22:48:26.507 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0701e-05 2024-09-20T22:48:26.508 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.508 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.508 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:48:26.508 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:06.542987+0000", 2024-09-20T22:48:26.508 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9760000000000001e-05 2024-09-20T22:48:26.508 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.508 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.508 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:48:26.508 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:06.543002+0000", 2024-09-20T22:48:26.508 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4578e-05 2024-09-20T22:48:26.508 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.508 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.508 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:48:26.508 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:06.543070+0000", 2024-09-20T22:48:26.509 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.8263999999999996e-05 2024-09-20T22:48:26.509 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.509 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.509 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:48:26.509 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:06.543159+0000", 2024-09-20T22:48:26.509 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.9241999999999997e-05 2024-09-20T22:48:26.509 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.509 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.509 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 0,3", 2024-09-20T22:48:26.509 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:06.546618+0000", 2024-09-20T22:48:26.509 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0034581949999999998 2024-09-20T22:48:26.509 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.509 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.509 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:48:26.510 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:06.551199+0000", 2024-09-20T22:48:26.510 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0045811309999999996 2024-09-20T22:48:26.510 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.510 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.510 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:48:26.510 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:06.560992+0000", 2024-09-20T22:48:26.510 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0097937270000000003 2024-09-20T22:48:26.510 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.510 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.510 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:48:26.510 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:06.562419+0000", 2024-09-20T22:48:26.510 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0014269040000000001 2024-09-20T22:48:26.510 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.511 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.511 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:48:26.511 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:06.562515+0000", 2024-09-20T22:48:26.511 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.5544999999999994e-05 2024-09-20T22:48:26.511 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.511 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.511 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:48:26.511 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:06.562586+0000", 2024-09-20T22:48:26.511 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.1588999999999995e-05 2024-09-20T22:48:26.511 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:48:26.511 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:48:26.511 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:48:26.511 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.511 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.512 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:654 7.7 7:ea720b85:::smithi08477203-26:head [append 0~0] snapc 17=[17,16,15,13,11,d] ondisk+write+known_if_redirected+supports_pool_eio e136)", 2024-09-20T22:48:26.512 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:35.126065+0000", 2024-09-20T22:48:26.512 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 291.360070775, 2024-09-20T22:48:26.512 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.069757472000000001, 2024-09-20T22:48:26.512 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:48:26.512 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:48:26.512 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:48:26.512 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:48:26.512 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:48:26.512 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:48:26.512 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 654 2024-09-20T22:48:26.512 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.512 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:48:26.512 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.513 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:48:26.513 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.126065+0000", 2024-09-20T22:48:26.513 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:48:26.513 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.513 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.513 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:48:26.513 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.126065+0000", 2024-09-20T22:48:26.513 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:48:26.513 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.513 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.513 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:48:26.513 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.126067+0000", 2024-09-20T22:48:26.513 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.6139999999999999e-06 2024-09-20T22:48:26.513 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.513 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.514 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:48:26.514 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.126074+0000", 2024-09-20T22:48:26.514 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.1509999999999998e-06 2024-09-20T22:48:26.514 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.514 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.514 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:48:26.514 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.126090+0000", 2024-09-20T22:48:26.514 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.6001000000000001e-05 2024-09-20T22:48:26.514 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.514 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.514 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:48:26.514 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.126098+0000", 2024-09-20T22:48:26.514 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.2379999999999997e-06 2024-09-20T22:48:26.514 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.515 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.515 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:48:26.515 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.126155+0000", 2024-09-20T22:48:26.515 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.6279000000000001e-05 2024-09-20T22:48:26.515 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.515 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.515 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op must wait for map", 2024-09-20T22:48:26.515 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.126160+0000", 2024-09-20T22:48:26.515 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.8110000000000002e-06 2024-09-20T22:48:26.515 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.515 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.515 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:48:26.515 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.193660+0000", 2024-09-20T22:48:26.515 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.067499997000000006 2024-09-20T22:48:26.515 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.516 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.516 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:48:26.516 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.193751+0000", 2024-09-20T22:48:26.516 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.1571999999999997e-05 2024-09-20T22:48:26.516 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.516 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.516 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 0,3", 2024-09-20T22:48:26.516 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.193941+0000", 2024-09-20T22:48:26.516 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00018952699999999999 2024-09-20T22:48:26.516 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.516 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.516 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:48:26.516 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.195649+0000", 2024-09-20T22:48:26.516 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001707966 2024-09-20T22:48:26.517 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.517 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.517 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:48:26.517 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.195707+0000", 2024-09-20T22:48:26.517 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.8649e-05 2024-09-20T22:48:26.517 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.517 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.517 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:48:26.517 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.195722+0000", 2024-09-20T22:48:26.517 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4309e-05 2024-09-20T22:48:26.517 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.517 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.517 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:48:26.517 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.195773+0000", 2024-09-20T22:48:26.518 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.1368e-05 2024-09-20T22:48:26.518 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.518 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.518 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:48:26.518 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.195823+0000", 2024-09-20T22:48:26.518 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.9990000000000001e-05 2024-09-20T22:48:26.518 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:48:26.518 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:48:26.518 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:48:26.518 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.518 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.518 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:655 7.7 7:ea720b85:::smithi08477203-26:head [rollback 16] snapc 17=[17,16,15,13,11,d] ondisk+write+known_if_redirected+supports_pool_eio e136)", 2024-09-20T22:48:26.518 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:35.126118+0000", 2024-09-20T22:48:26.518 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 291.36001852200002, 2024-09-20T22:48:26.519 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.069926453, 2024-09-20T22:48:26.519 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:48:26.519 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:48:26.519 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:48:26.519 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:48:26.519 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:48:26.519 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:48:26.519 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 655 2024-09-20T22:48:26.519 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.519 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:48:26.519 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.519 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:48:26.519 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.126118+0000", 2024-09-20T22:48:26.519 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:48:26.520 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.520 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.520 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:48:26.520 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.126118+0000", 2024-09-20T22:48:26.520 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:48:26.520 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.520 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.520 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:48:26.520 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.126118+0000", 2024-09-20T22:48:26.520 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.3000000000000001e-07 2024-09-20T22:48:26.520 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.521 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.521 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:48:26.521 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.126123+0000", 2024-09-20T22:48:26.521 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.7650000000000001e-06 2024-09-20T22:48:26.521 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.521 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.521 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:48:26.521 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.126141+0000", 2024-09-20T22:48:26.521 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.7538e-05 2024-09-20T22:48:26.521 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.521 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.521 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:48:26.521 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.126150+0000", 2024-09-20T22:48:26.521 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.3549999999999994e-06 2024-09-20T22:48:26.522 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.522 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.522 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:48:26.522 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.126204+0000", 2024-09-20T22:48:26.522 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.4289999999999997e-05 2024-09-20T22:48:26.522 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.522 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.522 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting_for_map not empty", 2024-09-20T22:48:26.522 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.126210+0000", 2024-09-20T22:48:26.522 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.361e-06 2024-09-20T22:48:26.522 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.522 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.522 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:48:26.522 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.194528+0000", 2024-09-20T22:48:26.523 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.068318913999999994 2024-09-20T22:48:26.523 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.523 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.523 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:48:26.523 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.194614+0000", 2024-09-20T22:48:26.523 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.5280999999999999e-05 2024-09-20T22:48:26.523 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.523 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.523 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 0,3", 2024-09-20T22:48:26.523 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.194759+0000", 2024-09-20T22:48:26.523 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000145678 2024-09-20T22:48:26.523 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.523 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.523 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:48:26.524 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.195836+0000", 2024-09-20T22:48:26.524 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001076474 2024-09-20T22:48:26.524 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.524 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.524 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:48:26.524 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.195884+0000", 2024-09-20T22:48:26.524 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.8421000000000003e-05 2024-09-20T22:48:26.524 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.524 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.524 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:48:26.524 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.195947+0000", 2024-09-20T22:48:26.524 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.2522999999999998e-05 2024-09-20T22:48:26.524 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.524 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.525 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:48:26.525 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.196003+0000", 2024-09-20T22:48:26.525 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.5911000000000001e-05 2024-09-20T22:48:26.525 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.525 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.525 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:48:26.525 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.196044+0000", 2024-09-20T22:48:26.525 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.1412000000000003e-05 2024-09-20T22:48:26.525 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:48:26.525 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:48:26.525 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:48:26.525 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.525 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.525 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:656 7.7 7:ea720b85:::smithi08477203-26:head [append 0~0] snapc 17=[17,16,15,13,11,d] ondisk+write+known_if_redirected+supports_pool_eio e136)", 2024-09-20T22:48:26.526 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:35.126159+0000", 2024-09-20T22:48:26.526 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 291.35997768800001, 2024-09-20T22:48:26.526 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.070101607999999996, 2024-09-20T22:48:26.526 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:48:26.526 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:48:26.526 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:48:26.526 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:48:26.526 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:48:26.526 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:48:26.526 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 656 2024-09-20T22:48:26.526 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.526 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:48:26.526 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.527 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:48:26.527 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.126159+0000", 2024-09-20T22:48:26.527 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:48:26.527 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.527 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.527 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:48:26.527 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.126159+0000", 2024-09-20T22:48:26.527 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:48:26.527 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.527 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.527 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:48:26.527 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.126159+0000", 2024-09-20T22:48:26.527 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.1500000000000005e-07 2024-09-20T22:48:26.527 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.528 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.528 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:48:26.528 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.126168+0000", 2024-09-20T22:48:26.528 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.4600000000000003e-06 2024-09-20T22:48:26.528 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.528 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.528 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:48:26.528 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.126173+0000", 2024-09-20T22:48:26.528 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.7089999999999999e-06 2024-09-20T22:48:26.528 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.528 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.528 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:48:26.528 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.126177+0000", 2024-09-20T22:48:26.528 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.9040000000000002e-06 2024-09-20T22:48:26.528 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.529 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.529 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:48:26.529 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.126232+0000", 2024-09-20T22:48:26.529 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.5337999999999998e-05 2024-09-20T22:48:26.529 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.529 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.529 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting_for_map not empty", 2024-09-20T22:48:26.529 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.126238+0000", 2024-09-20T22:48:26.529 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.2390000000000002e-06 2024-09-20T22:48:26.529 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.529 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.529 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:48:26.529 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.195116+0000", 2024-09-20T22:48:26.529 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.068878012000000002 2024-09-20T22:48:26.529 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.530 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.530 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:48:26.530 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.195197+0000", 2024-09-20T22:48:26.530 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.0964999999999995e-05 2024-09-20T22:48:26.530 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.530 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.530 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 0,3", 2024-09-20T22:48:26.530 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.195338+0000", 2024-09-20T22:48:26.530 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000141006 2024-09-20T22:48:26.530 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.530 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.530 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:48:26.530 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.195897+0000", 2024-09-20T22:48:26.530 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00055979999999999995 2024-09-20T22:48:26.531 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.531 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.531 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:48:26.531 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.196102+0000", 2024-09-20T22:48:26.531 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00020405099999999999 2024-09-20T22:48:26.531 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.531 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.531 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:48:26.531 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.196160+0000", 2024-09-20T22:48:26.531 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.8677000000000001e-05 2024-09-20T22:48:26.531 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.531 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.531 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:48:26.531 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.196214+0000", 2024-09-20T22:48:26.531 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.414e-05 2024-09-20T22:48:26.532 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.532 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.532 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:48:26.532 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:35.196260+0000", 2024-09-20T22:48:26.532 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.5791999999999998e-05 2024-09-20T22:48:26.532 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:48:26.532 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:48:26.532 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:48:26.532 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.532 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.532 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:1017 7.19 7:9cc0b7e2:::smithi08477203-46:head [copy-from ver 99] snapc 25=[25,24,23,21] ondisk+write+known_if_redirected+supports_pool_eio e166)", 2024-09-20T22:48:26.532 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:44:05.130713+0000", 2024-09-20T22:48:26.532 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 261.35542314100002, 2024-09-20T22:48:26.532 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.021441722999999999, 2024-09-20T22:48:26.533 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:48:26.533 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:48:26.533 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:48:26.533 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:48:26.533 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:48:26.533 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:48:26.533 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 1017 2024-09-20T22:48:26.533 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.533 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:48:26.533 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.533 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:48:26.533 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:05.130713+0000", 2024-09-20T22:48:26.533 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:48:26.533 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.533 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.534 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:48:26.534 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:05.130713+0000", 2024-09-20T22:48:26.534 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:48:26.534 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.534 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.534 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:48:26.534 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:05.130715+0000", 2024-09-20T22:48:26.534 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.66e-06 2024-09-20T22:48:26.534 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.534 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.534 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:48:26.534 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:05.130721+0000", 2024-09-20T22:48:26.534 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.0430000000000004e-06 2024-09-20T22:48:26.535 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.535 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.535 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:48:26.535 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:05.130732+0000", 2024-09-20T22:48:26.535 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1494999999999999e-05 2024-09-20T22:48:26.535 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.535 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.535 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:48:26.535 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:05.130748+0000", 2024-09-20T22:48:26.535 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.5520999999999999e-05 2024-09-20T22:48:26.535 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.535 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.535 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:48:26.535 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:05.130827+0000", 2024-09-20T22:48:26.535 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.8990000000000001e-05 2024-09-20T22:48:26.536 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.536 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.536 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op must wait for map", 2024-09-20T22:48:26.536 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:05.130834+0000", 2024-09-20T22:48:26.536 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.097e-06 2024-09-20T22:48:26.536 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.536 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.536 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:48:26.536 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:05.133176+0000", 2024-09-20T22:48:26.536 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.002341704 2024-09-20T22:48:26.536 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.536 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.536 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:48:26.536 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:05.133285+0000", 2024-09-20T22:48:26.536 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000109473 2024-09-20T22:48:26.537 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.537 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.537 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 2,7", 2024-09-20T22:48:26.537 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:05.142516+0000", 2024-09-20T22:48:26.537 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0092311130000000009 2024-09-20T22:48:26.537 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.537 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.537 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:48:26.537 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:05.150724+0000", 2024-09-20T22:48:26.537 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0082083139999999995 2024-09-20T22:48:26.537 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.537 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.537 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:48:26.537 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:05.150825+0000", 2024-09-20T22:48:26.538 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000100431 2024-09-20T22:48:26.538 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.538 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.538 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:48:26.538 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:05.152061+0000", 2024-09-20T22:48:26.538 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001235912 2024-09-20T22:48:26.538 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.538 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.538 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:48:26.538 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:05.152108+0000", 2024-09-20T22:48:26.538 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.7068999999999999e-05 2024-09-20T22:48:26.538 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.538 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.538 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:48:26.539 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:05.152155+0000", 2024-09-20T22:48:26.539 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.6900999999999997e-05 2024-09-20T22:48:26.539 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:48:26.539 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:48:26.539 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:48:26.539 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.539 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.539 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:1054 7.12 7:4fa4b2b3:::smithi08477203-21:head [copy-from ver 67] snapc 25=[25,24,23,21] ondisk+write+known_if_redirected+supports_pool_eio e166)", 2024-09-20T22:48:26.539 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:44:05.634022+0000", 2024-09-20T22:48:26.539 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 260.852113863, 2024-09-20T22:48:26.539 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.019656218, 2024-09-20T22:48:26.539 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:48:26.539 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:48:26.539 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:48:26.539 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:48:26.540 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:48:26.540 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:48:26.540 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 1054 2024-09-20T22:48:26.540 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.540 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:48:26.540 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.540 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:48:26.540 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:05.634022+0000", 2024-09-20T22:48:26.540 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:48:26.540 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.540 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.540 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:48:26.540 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:05.634022+0000", 2024-09-20T22:48:26.540 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:48:26.541 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.541 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.541 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:48:26.541 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:05.634024+0000", 2024-09-20T22:48:26.541 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.013e-06 2024-09-20T22:48:26.541 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.541 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.541 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:48:26.541 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:05.634035+0000", 2024-09-20T22:48:26.541 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0644e-05 2024-09-20T22:48:26.541 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.541 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.541 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:48:26.541 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:05.634061+0000", 2024-09-20T22:48:26.541 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.6352000000000001e-05 2024-09-20T22:48:26.542 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.542 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.542 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:48:26.542 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:05.634081+0000", 2024-09-20T22:48:26.542 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9347e-05 2024-09-20T22:48:26.542 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.542 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.542 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:48:26.542 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:05.634257+0000", 2024-09-20T22:48:26.542 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000176365 2024-09-20T22:48:26.542 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.542 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.542 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:48:26.542 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:05.634379+0000", 2024-09-20T22:48:26.543 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000122173 2024-09-20T22:48:26.543 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.543 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.543 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 0,2", 2024-09-20T22:48:26.543 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:05.643905+0000", 2024-09-20T22:48:26.543 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0095258679999999998 2024-09-20T22:48:26.543 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.543 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.543 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:48:26.543 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:05.647750+0000", 2024-09-20T22:48:26.543 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.003844564 2024-09-20T22:48:26.543 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.543 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.543 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:48:26.543 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:05.650803+0000", 2024-09-20T22:48:26.544 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.003053114 2024-09-20T22:48:26.544 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.544 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.544 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:48:26.544 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:05.653610+0000", 2024-09-20T22:48:26.544 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0028071459999999999 2024-09-20T22:48:26.544 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.544 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.544 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:48:26.544 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:05.653645+0000", 2024-09-20T22:48:26.544 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.5231000000000002e-05 2024-09-20T22:48:26.544 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.544 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.544 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:48:26.545 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:05.653679+0000", 2024-09-20T22:48:26.545 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.3401000000000001e-05 2024-09-20T22:48:26.545 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:48:26.545 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:48:26.545 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:48:26.545 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.545 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.545 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:1420 7.19 7:9cc0b7e2:::smithi08477203-46:head [write 480733~537415,stat] snapc 2f=[2f] ondisk+write+known_if_redirected+supports_pool_eio e189)", 2024-09-20T22:48:26.545 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:44:28.143096+0000", 2024-09-20T22:48:26.545 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 238.343040344, 2024-09-20T22:48:26.545 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.021594787000000001, 2024-09-20T22:48:26.545 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:48:26.545 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:48:26.545 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:48:26.546 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:48:26.546 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:48:26.546 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:48:26.546 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 1420 2024-09-20T22:48:26.546 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.546 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:48:26.546 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.546 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:48:26.546 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:28.143096+0000", 2024-09-20T22:48:26.546 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:48:26.546 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.546 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.546 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:48:26.546 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:28.143096+0000", 2024-09-20T22:48:26.547 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:48:26.547 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.547 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.547 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:48:26.547 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:28.143098+0000", 2024-09-20T22:48:26.547 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.1840000000000002e-06 2024-09-20T22:48:26.547 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.547 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.547 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:48:26.547 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:28.143663+0000", 2024-09-20T22:48:26.547 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00056486000000000004 2024-09-20T22:48:26.547 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.547 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.547 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:48:26.548 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:28.143684+0000", 2024-09-20T22:48:26.548 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.0792e-05 2024-09-20T22:48:26.548 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.548 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.548 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:48:26.548 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:28.143697+0000", 2024-09-20T22:48:26.548 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.292e-05 2024-09-20T22:48:26.548 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.548 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.548 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:48:26.548 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:28.143788+0000", 2024-09-20T22:48:26.548 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.1218999999999995e-05 2024-09-20T22:48:26.548 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.548 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.548 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op must wait for map", 2024-09-20T22:48:26.549 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:28.143798+0000", 2024-09-20T22:48:26.549 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.7999999999999993e-06 2024-09-20T22:48:26.549 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.549 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.549 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:48:26.549 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:28.148428+0000", 2024-09-20T22:48:26.549 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0046303780000000001 2024-09-20T22:48:26.549 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.549 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.549 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:48:26.549 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:28.148611+0000", 2024-09-20T22:48:26.549 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00018319699999999999 2024-09-20T22:48:26.549 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.549 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.550 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 2,7", 2024-09-20T22:48:26.550 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:28.148807+0000", 2024-09-20T22:48:26.550 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00019560399999999999 2024-09-20T22:48:26.550 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.550 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.550 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:48:26.550 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:28.157814+0000", 2024-09-20T22:48:26.550 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0090068449999999994 2024-09-20T22:48:26.550 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.550 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.550 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:48:26.550 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:28.158746+0000", 2024-09-20T22:48:26.550 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000932076 2024-09-20T22:48:26.550 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.550 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.551 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:48:26.551 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:28.164580+0000", 2024-09-20T22:48:26.551 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0058337850000000002 2024-09-20T22:48:26.551 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.551 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.551 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:48:26.551 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:28.164638+0000", 2024-09-20T22:48:26.551 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.8100999999999998e-05 2024-09-20T22:48:26.551 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.551 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.551 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:48:26.551 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:28.164691+0000", 2024-09-20T22:48:26.551 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.3025999999999997e-05 2024-09-20T22:48:26.551 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:48:26.552 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:48:26.552 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:48:26.552 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.552 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.552 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:1525 7.c 7:3f865ef5:::smithi08477203-28:head [copy-from ver 127] snapc 33=[33] ondisk+write+known_if_redirected+supports_pool_eio e197)", 2024-09-20T22:48:26.552 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:44:36.198921+0000", 2024-09-20T22:48:26.552 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 230.28721544000001, 2024-09-20T22:48:26.552 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.028621220999999999, 2024-09-20T22:48:26.552 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:48:26.552 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:48:26.552 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:48:26.552 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:48:26.552 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:48:26.552 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:48:26.553 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 1525 2024-09-20T22:48:26.553 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.553 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:48:26.553 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.553 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:48:26.553 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.198921+0000", 2024-09-20T22:48:26.553 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:48:26.553 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.553 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.553 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:48:26.553 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.198921+0000", 2024-09-20T22:48:26.553 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:48:26.553 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.553 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.554 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:48:26.554 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.198923+0000", 2024-09-20T22:48:26.554 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.1090000000000001e-06 2024-09-20T22:48:26.554 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.554 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.554 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:48:26.554 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.198934+0000", 2024-09-20T22:48:26.554 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0837e-05 2024-09-20T22:48:26.554 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.554 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.554 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:48:26.554 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.198955+0000", 2024-09-20T22:48:26.554 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.0789e-05 2024-09-20T22:48:26.554 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.554 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.555 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:48:26.555 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.198966+0000", 2024-09-20T22:48:26.555 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1778e-05 2024-09-20T22:48:26.555 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.555 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.555 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:48:26.555 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.199046+0000", 2024-09-20T22:48:26.555 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.9567999999999994e-05 2024-09-20T22:48:26.555 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.555 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.555 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op must wait for map", 2024-09-20T22:48:26.555 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.199054+0000", 2024-09-20T22:48:26.555 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.9349999999999994e-06 2024-09-20T22:48:26.555 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.556 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.556 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:48:26.556 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.201428+0000", 2024-09-20T22:48:26.556 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0023744389999999999 2024-09-20T22:48:26.556 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.556 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.556 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:48:26.556 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.201527+0000", 2024-09-20T22:48:26.556 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.8554000000000005e-05 2024-09-20T22:48:26.556 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.556 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.556 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 3,7", 2024-09-20T22:48:26.556 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.215277+0000", 2024-09-20T22:48:26.556 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.013750326 2024-09-20T22:48:26.557 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.557 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.557 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:48:26.557 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.226629+0000", 2024-09-20T22:48:26.557 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.011351781999999999 2024-09-20T22:48:26.557 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.557 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.557 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:48:26.557 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.227366+0000", 2024-09-20T22:48:26.557 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00073709199999999996 2024-09-20T22:48:26.557 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.557 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.557 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:48:26.557 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.227450+0000", 2024-09-20T22:48:26.557 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.3902999999999999e-05 2024-09-20T22:48:26.558 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.558 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.558 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:48:26.558 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.227489+0000", 2024-09-20T22:48:26.558 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.9103000000000002e-05 2024-09-20T22:48:26.558 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.558 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.558 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:48:26.558 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.227542+0000", 2024-09-20T22:48:26.558 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.3006e-05 2024-09-20T22:48:26.558 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:48:26.558 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:48:26.558 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:48:26.558 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.558 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.559 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:1526 7.c 7:3f865ef5:::smithi08477203-28:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e197)", 2024-09-20T22:48:26.559 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:44:36.198993+0000", 2024-09-20T22:48:26.559 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 230.287143074, 2024-09-20T22:48:26.560 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.022582641000000001, 2024-09-20T22:48:26.560 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:48:26.560 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:48:26.560 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:48:26.560 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:48:26.560 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:48:26.560 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:48:26.560 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 1526 2024-09-20T22:48:26.560 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.560 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:48:26.561 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.561 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:48:26.561 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.198993+0000", 2024-09-20T22:48:26.561 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:48:26.561 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.561 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.561 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:48:26.561 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.198993+0000", 2024-09-20T22:48:26.561 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:48:26.561 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.561 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.561 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:48:26.561 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.198994+0000", 2024-09-20T22:48:26.561 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.2e-07 2024-09-20T22:48:26.562 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.562 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.562 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:48:26.562 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.198996+0000", 2024-09-20T22:48:26.562 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.6410000000000002e-06 2024-09-20T22:48:26.562 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.562 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.562 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:48:26.562 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.199005+0000", 2024-09-20T22:48:26.562 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.8079999999999994e-06 2024-09-20T22:48:26.562 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.562 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.562 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:48:26.562 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.199011+0000", 2024-09-20T22:48:26.563 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.7010000000000001e-06 2024-09-20T22:48:26.563 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.563 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.563 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:48:26.563 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.199112+0000", 2024-09-20T22:48:26.563 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000101335 2024-09-20T22:48:26.563 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.563 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.563 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting_for_map not empty", 2024-09-20T22:48:26.563 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.199117+0000", 2024-09-20T22:48:26.563 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.2979999999999999e-06 2024-09-20T22:48:26.563 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.563 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.563 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:48:26.564 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.201667+0000", 2024-09-20T22:48:26.564 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.002549922 2024-09-20T22:48:26.564 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.564 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.564 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for blocked object", 2024-09-20T22:48:26.564 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.201708+0000", 2024-09-20T22:48:26.564 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.0772999999999997e-05 2024-09-20T22:48:26.564 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.564 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.564 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:48:26.564 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.221443+0000", 2024-09-20T22:48:26.564 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.019735229 2024-09-20T22:48:26.564 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.564 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.564 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:48:26.565 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.221509+0000", 2024-09-20T22:48:26.565 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.5618999999999996e-05 2024-09-20T22:48:26.565 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.565 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.565 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:48:26.565 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.221576+0000", 2024-09-20T22:48:26.565 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.6795e-05 2024-09-20T22:48:26.565 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:48:26.565 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:48:26.565 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:48:26.565 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.565 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.565 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:1652 7.c 7:31d0211f:::smithi08477203-25:head [copy-from ver 161] snapc 38=[38,37] ondisk+write+known_if_redirected+supports_pool_eio e206)", 2024-09-20T22:48:26.565 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:44:45.159981+0000", 2024-09-20T22:48:26.566 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 221.32615504, 2024-09-20T22:48:26.566 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.029481621, 2024-09-20T22:48:26.566 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:48:26.566 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:48:26.566 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:48:26.566 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:48:26.566 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:48:26.566 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:48:26.566 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 1652 2024-09-20T22:48:26.566 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.566 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:48:26.566 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.566 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:48:26.566 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:45.159981+0000", 2024-09-20T22:48:26.567 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:48:26.567 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.567 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.567 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:48:26.567 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:45.159981+0000", 2024-09-20T22:48:26.567 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:48:26.567 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.567 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.567 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:48:26.567 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:45.159982+0000", 2024-09-20T22:48:26.567 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.8999999999999995e-07 2024-09-20T22:48:26.567 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.567 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.567 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:48:26.568 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:45.159986+0000", 2024-09-20T22:48:26.568 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.732e-06 2024-09-20T22:48:26.568 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.568 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.568 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:48:26.568 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:45.159996+0000", 2024-09-20T22:48:26.568 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.9639999999999998e-06 2024-09-20T22:48:26.568 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.568 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.568 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:48:26.568 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:45.160001+0000", 2024-09-20T22:48:26.568 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.8010000000000003e-06 2024-09-20T22:48:26.568 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.568 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.569 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:48:26.569 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:45.165664+0000", 2024-09-20T22:48:26.569 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0056630869999999998 2024-09-20T22:48:26.569 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.569 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.569 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:48:26.569 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:45.165710+0000", 2024-09-20T22:48:26.569 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.6371e-05 2024-09-20T22:48:26.569 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.569 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.569 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 3,7", 2024-09-20T22:48:26.569 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:45.175729+0000", 2024-09-20T22:48:26.569 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.010018750999999999 2024-09-20T22:48:26.569 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.570 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.570 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:48:26.570 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:45.186279+0000", 2024-09-20T22:48:26.570 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.010550241 2024-09-20T22:48:26.570 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.570 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.570 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:48:26.570 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:45.186520+0000", 2024-09-20T22:48:26.570 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000241394 2024-09-20T22:48:26.570 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.570 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.570 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:48:26.570 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:45.189408+0000", 2024-09-20T22:48:26.570 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0028872849999999999 2024-09-20T22:48:26.571 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.571 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.571 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:48:26.571 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:45.189440+0000", 2024-09-20T22:48:26.571 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.2341000000000001e-05 2024-09-20T22:48:26.571 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.571 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.571 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:48:26.571 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:45.189463+0000", 2024-09-20T22:48:26.571 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.2764000000000001e-05 2024-09-20T22:48:26.571 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:48:26.571 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:48:26.571 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:48:26.571 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.572 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.572 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:1653 7.c 7:31d0211f:::smithi08477203-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e206)", 2024-09-20T22:48:26.572 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:44:45.160009+0000", 2024-09-20T22:48:26.572 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 221.32612684200001, 2024-09-20T22:48:26.572 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.020255434999999999, 2024-09-20T22:48:26.572 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:48:26.572 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:48:26.572 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:48:26.572 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:48:26.572 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:48:26.572 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:48:26.572 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 1653 2024-09-20T22:48:26.572 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.572 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:48:26.572 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.573 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:48:26.573 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:45.160009+0000", 2024-09-20T22:48:26.573 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:48:26.573 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.573 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.573 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:48:26.573 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:45.160009+0000", 2024-09-20T22:48:26.573 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:48:26.573 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.573 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.573 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:48:26.573 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:45.160010+0000", 2024-09-20T22:48:26.573 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.5100000000000001e-07 2024-09-20T22:48:26.573 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.573 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.574 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:48:26.574 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:45.160011+0000", 2024-09-20T22:48:26.574 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.547e-06 2024-09-20T22:48:26.574 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.574 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.574 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:48:26.574 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:45.160015+0000", 2024-09-20T22:48:26.574 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.3749999999999999e-06 2024-09-20T22:48:26.574 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.574 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.574 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:48:26.574 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:45.160018+0000", 2024-09-20T22:48:26.574 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.1250000000000001e-06 2024-09-20T22:48:26.574 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.575 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.575 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:48:26.575 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:45.165846+0000", 2024-09-20T22:48:26.575 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0058279270000000001 2024-09-20T22:48:26.575 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.575 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.575 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for blocked object", 2024-09-20T22:48:26.575 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:45.165873+0000", 2024-09-20T22:48:26.575 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.7059999999999998e-05 2024-09-20T22:48:26.575 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.575 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.575 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:48:26.575 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:45.180130+0000", 2024-09-20T22:48:26.575 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.014256998999999999 2024-09-20T22:48:26.576 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.576 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.576 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:48:26.576 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:45.180199+0000", 2024-09-20T22:48:26.576 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.9550999999999999e-05 2024-09-20T22:48:26.576 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.576 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.576 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:48:26.576 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:45.180265+0000", 2024-09-20T22:48:26.576 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.5500000000000006e-05 2024-09-20T22:48:26.576 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:48:26.576 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:48:26.576 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:48:26.576 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.576 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.577 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:1654 7.19 7:9970b578:::smithi08477203-15:head [copy-from ver 110] snapc 38=[38,37] ondisk+write+known_if_redirected+supports_pool_eio e206)", 2024-09-20T22:48:26.577 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:44:45.160022+0000", 2024-09-20T22:48:26.577 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 221.32611443499999, 2024-09-20T22:48:26.577 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.021661590000000001, 2024-09-20T22:48:26.577 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:48:26.577 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:48:26.577 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:48:26.577 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:48:26.577 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:48:26.577 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:48:26.577 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 1654 2024-09-20T22:48:26.577 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.577 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:48:26.577 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.577 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:48:26.578 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:45.160022+0000", 2024-09-20T22:48:26.578 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:48:26.578 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.578 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.578 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:48:26.578 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:45.160022+0000", 2024-09-20T22:48:26.578 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:48:26.578 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.578 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.578 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:48:26.578 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:45.160022+0000", 2024-09-20T22:48:26.578 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.4299999999999999e-07 2024-09-20T22:48:26.578 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.578 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.579 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:48:26.579 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:45.160024+0000", 2024-09-20T22:48:26.579 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.018e-06 2024-09-20T22:48:26.579 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.579 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.579 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:48:26.579 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:45.160027+0000", 2024-09-20T22:48:26.579 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.9100000000000001e-06 2024-09-20T22:48:26.579 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.579 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.579 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:48:26.579 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:45.160030+0000", 2024-09-20T22:48:26.579 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.7140000000000002e-06 2024-09-20T22:48:26.579 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.579 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.580 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:48:26.580 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:45.160078+0000", 2024-09-20T22:48:26.580 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.7933000000000003e-05 2024-09-20T22:48:26.580 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.580 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.580 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:48:26.580 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:45.160133+0000", 2024-09-20T22:48:26.580 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.5823999999999999e-05 2024-09-20T22:48:26.580 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.580 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.580 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 2,7", 2024-09-20T22:48:26.580 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:45.167724+0000", 2024-09-20T22:48:26.580 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0075904060000000001 2024-09-20T22:48:26.580 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.581 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.581 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:48:26.581 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:45.177349+0000", 2024-09-20T22:48:26.581 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0096249809999999995 2024-09-20T22:48:26.581 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.581 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.581 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:48:26.581 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:45.178986+0000", 2024-09-20T22:48:26.581 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00163722 2024-09-20T22:48:26.581 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.581 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.581 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:48:26.581 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:45.181613+0000", 2024-09-20T22:48:26.581 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.002626888 2024-09-20T22:48:26.581 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.582 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.582 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:48:26.582 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:45.181650+0000", 2024-09-20T22:48:26.582 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.7249000000000002e-05 2024-09-20T22:48:26.582 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.582 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.582 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:48:26.582 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:45.181683+0000", 2024-09-20T22:48:26.582 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.3204000000000003e-05 2024-09-20T22:48:26.582 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:48:26.582 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:48:26.582 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:48:26.582 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.582 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.583 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:2179 7.12 7:4fa4b2b3:::smithi08477203-21:head [write 595047~641755,stat] snapc 53=[53,52,51,50,4b,47,45,3e] ondisk+write+known_if_redirected+supports_pool_eio e254)", 2024-09-20T22:48:26.583 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:45:33.166066+0000", 2024-09-20T22:48:26.583 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 173.32007044700001, 2024-09-20T22:48:26.583 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.020018068999999999, 2024-09-20T22:48:26.583 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:48:26.583 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:48:26.583 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:48:26.583 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:48:26.583 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:48:26.583 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:48:26.583 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 2179 2024-09-20T22:48:26.583 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.583 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:48:26.583 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.584 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:48:26.584 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:33.166066+0000", 2024-09-20T22:48:26.584 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:48:26.584 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.584 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.584 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:48:26.584 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:33.166066+0000", 2024-09-20T22:48:26.584 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:48:26.584 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.584 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.584 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:48:26.584 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:33.166067+0000", 2024-09-20T22:48:26.584 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.6500000000000001e-06 2024-09-20T22:48:26.584 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.585 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.585 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:48:26.585 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:33.167423+0000", 2024-09-20T22:48:26.585 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0013558999999999999 2024-09-20T22:48:26.585 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.585 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.585 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:48:26.585 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:33.167433+0000", 2024-09-20T22:48:26.585 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.4790000000000002e-06 2024-09-20T22:48:26.585 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.585 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.586 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:48:26.586 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:33.167442+0000", 2024-09-20T22:48:26.586 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.1439999999999992e-06 2024-09-20T22:48:26.586 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.586 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.586 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:48:26.586 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:33.167627+0000", 2024-09-20T22:48:26.586 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00018459100000000001 2024-09-20T22:48:26.586 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.586 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.586 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:48:26.586 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:33.167690+0000", 2024-09-20T22:48:26.586 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.3806000000000005e-05 2024-09-20T22:48:26.586 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.587 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.587 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 0,2", 2024-09-20T22:48:26.587 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:33.167834+0000", 2024-09-20T22:48:26.587 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00014365600000000001 2024-09-20T22:48:26.587 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.587 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.587 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:48:26.587 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:33.169264+0000", 2024-09-20T22:48:26.587 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0014299040000000001 2024-09-20T22:48:26.587 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.587 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.587 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:48:26.587 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:33.175836+0000", 2024-09-20T22:48:26.587 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0065721750000000004 2024-09-20T22:48:26.587 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.588 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.588 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:48:26.588 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:33.185986+0000", 2024-09-20T22:48:26.588 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.010149696 2024-09-20T22:48:26.588 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.588 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.588 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:48:26.588 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:33.186040+0000", 2024-09-20T22:48:26.588 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.4601000000000001e-05 2024-09-20T22:48:26.588 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.588 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.588 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:48:26.588 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:33.186084+0000", 2024-09-20T22:48:26.588 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.3467000000000002e-05 2024-09-20T22:48:26.589 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:48:26.589 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:48:26.589 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:48:26.589 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.589 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.589 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:3132 7.19 7:9eac4560:::smithi08477203-30:head [copy-from ver 389] snapc 77=[77,76,75,74,73,70,6b] ondisk+write+known_if_redirected+supports_pool_eio e328)", 2024-09-20T22:48:26.589 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:46:48.209876+0000", 2024-09-20T22:48:26.589 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 98.276260300000004, 2024-09-20T22:48:26.589 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.030505467000000001, 2024-09-20T22:48:26.589 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:48:26.589 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:48:26.589 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:48:26.589 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:48:26.589 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:48:26.590 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:48:26.590 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 3132 2024-09-20T22:48:26.590 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.590 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:48:26.590 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.590 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:48:26.590 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.209876+0000", 2024-09-20T22:48:26.590 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:48:26.590 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.590 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.590 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:48:26.590 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.209876+0000", 2024-09-20T22:48:26.590 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:48:26.590 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.591 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.591 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:48:26.591 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.209876+0000", 2024-09-20T22:48:26.591 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.7e-07 2024-09-20T22:48:26.591 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.591 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.591 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:48:26.591 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.209880+0000", 2024-09-20T22:48:26.591 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.4149999999999999e-06 2024-09-20T22:48:26.591 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.591 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.591 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:48:26.591 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.209886+0000", 2024-09-20T22:48:26.591 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.5819999999999998e-06 2024-09-20T22:48:26.592 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.592 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.592 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:48:26.592 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.209893+0000", 2024-09-20T22:48:26.592 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.1820000000000003e-06 2024-09-20T22:48:26.592 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.592 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.592 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:48:26.592 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.209945+0000", 2024-09-20T22:48:26.592 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.2528999999999998e-05 2024-09-20T22:48:26.592 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.592 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.592 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:48:26.592 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.210028+0000", 2024-09-20T22:48:26.592 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.3067999999999998e-05 2024-09-20T22:48:26.593 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.593 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.593 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 2,7", 2024-09-20T22:48:26.593 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.223189+0000", 2024-09-20T22:48:26.593 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.013160579 2024-09-20T22:48:26.593 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.593 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.593 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:48:26.593 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.232498+0000", 2024-09-20T22:48:26.593 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0093088760000000007 2024-09-20T22:48:26.593 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.593 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.593 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:48:26.593 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.240058+0000", 2024-09-20T22:48:26.594 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0075599930000000001 2024-09-20T22:48:26.594 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.594 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.594 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:48:26.594 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.240300+0000", 2024-09-20T22:48:26.594 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00024190999999999999 2024-09-20T22:48:26.594 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.594 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.594 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:48:26.594 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.240343+0000", 2024-09-20T22:48:26.594 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.2972000000000001e-05 2024-09-20T22:48:26.594 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.594 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.594 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:48:26.595 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.240381+0000", 2024-09-20T22:48:26.595 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.8890999999999997e-05 2024-09-20T22:48:26.595 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:48:26.595 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:48:26.595 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:48:26.595 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.595 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.595 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:3133 7.19 7:9eac4560:::smithi08477203-30:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e328)", 2024-09-20T22:48:26.595 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:46:48.209912+0000", 2024-09-20T22:48:26.595 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 98.276224534999997, 2024-09-20T22:48:26.595 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.020991129000000001, 2024-09-20T22:48:26.595 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:48:26.595 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:48:26.595 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:48:26.596 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:48:26.596 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:48:26.596 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:48:26.596 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 3133 2024-09-20T22:48:26.596 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.596 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:48:26.596 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.596 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:48:26.596 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.209912+0000", 2024-09-20T22:48:26.596 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:48:26.596 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.596 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.596 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:48:26.596 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.209912+0000", 2024-09-20T22:48:26.596 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:48:26.597 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.597 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.597 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:48:26.597 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.209912+0000", 2024-09-20T22:48:26.597 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.0399999999999996e-07 2024-09-20T22:48:26.597 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.597 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.597 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:48:26.597 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.209915+0000", 2024-09-20T22:48:26.597 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.8890000000000002e-06 2024-09-20T22:48:26.597 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.597 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.597 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:48:26.597 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.209922+0000", 2024-09-20T22:48:26.597 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.153e-06 2024-09-20T22:48:26.598 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.598 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.598 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:48:26.598 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.209929+0000", 2024-09-20T22:48:26.598 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.5400000000000001e-06 2024-09-20T22:48:26.598 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.598 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.598 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:48:26.598 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.210307+0000", 2024-09-20T22:48:26.598 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00037839900000000002 2024-09-20T22:48:26.598 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.598 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.598 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for blocked object", 2024-09-20T22:48:26.598 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.210386+0000", 2024-09-20T22:48:26.598 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.8536999999999997e-05 2024-09-20T22:48:26.599 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.599 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.599 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:48:26.599 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.230565+0000", 2024-09-20T22:48:26.599 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.020179389999999998 2024-09-20T22:48:26.599 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.599 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.599 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:48:26.599 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.230784+0000", 2024-09-20T22:48:26.599 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00021929 2024-09-20T22:48:26.599 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.599 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.599 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:48:26.599 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.230903+0000", 2024-09-20T22:48:26.600 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00011832699999999999 2024-09-20T22:48:26.600 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:48:26.600 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:48:26.600 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:48:26.600 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.600 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.600 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:3567 7.12 7:4d20232a:::smithi08477203-34:head [copy-from ver 321] snapc 8c=[8c,8b,8a,89,88,87,86,85,84,80,7f,7a,74,73,6b] ondisk+write+known_if_redirected+supports_pool_eio e362)", 2024-09-20T22:48:26.600 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:47:22.219645+0000", 2024-09-20T22:48:26.600 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 64.266491177000006, 2024-09-20T22:48:26.600 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.021532453, 2024-09-20T22:48:26.600 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:48:26.600 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:48:26.600 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:48:26.600 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:48:26.600 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:48:26.601 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:48:26.601 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 3567 2024-09-20T22:48:26.601 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.601 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:48:26.601 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.601 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:48:26.601 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:22.219645+0000", 2024-09-20T22:48:26.601 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:48:26.601 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.601 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.601 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:48:26.601 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:22.219645+0000", 2024-09-20T22:48:26.601 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:48:26.601 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.602 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.602 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:48:26.602 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:22.219647+0000", 2024-09-20T22:48:26.602 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.3649999999999998e-06 2024-09-20T22:48:26.602 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.602 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.602 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:48:26.602 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:22.219665+0000", 2024-09-20T22:48:26.602 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.7368e-05 2024-09-20T22:48:26.602 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.602 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.602 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:48:26.602 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:22.219685+0000", 2024-09-20T22:48:26.602 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9771000000000001e-05 2024-09-20T22:48:26.603 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.603 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.603 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:48:26.603 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:22.219695+0000", 2024-09-20T22:48:26.603 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0074e-05 2024-09-20T22:48:26.603 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.603 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.603 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:48:26.603 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:22.219752+0000", 2024-09-20T22:48:26.603 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.7633000000000002e-05 2024-09-20T22:48:26.603 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.603 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.603 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:48:26.603 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:22.219833+0000", 2024-09-20T22:48:26.604 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.0370999999999993e-05 2024-09-20T22:48:26.604 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.604 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.604 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 2,7", 2024-09-20T22:48:26.604 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:22.230376+0000", 2024-09-20T22:48:26.604 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.010543703 2024-09-20T22:48:26.604 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.604 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.604 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:48:26.604 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:22.240312+0000", 2024-09-20T22:48:26.604 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.009935213 2024-09-20T22:48:26.604 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.604 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.604 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:48:26.605 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:22.240836+0000", 2024-09-20T22:48:26.605 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00052436400000000004 2024-09-20T22:48:26.605 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.605 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.605 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:48:26.605 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:22.241104+0000", 2024-09-20T22:48:26.605 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00026767000000000002 2024-09-20T22:48:26.605 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.605 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.605 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:48:26.605 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:22.241142+0000", 2024-09-20T22:48:26.605 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.8071999999999997e-05 2024-09-20T22:48:26.605 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.605 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.605 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:48:26.606 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:22.241178+0000", 2024-09-20T22:48:26.606 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.5849000000000002e-05 2024-09-20T22:48:26.606 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:48:26.606 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:48:26.606 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:48:26.606 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.606 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.606 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:3580 7.d 7:b97fe42f:::smithi08477203-36:head [copy-from ver 100] snapc 8c=[8c,8b,8a,89,88,87,86,85,84,80,7f,7a,74,73,6b] ondisk+write+known_if_redirected+supports_pool_eio e362)", 2024-09-20T22:48:26.606 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:47:22.529195+0000", 2024-09-20T22:48:26.606 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 63.956941463, 2024-09-20T22:48:26.606 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.023634354, 2024-09-20T22:48:26.606 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:48:26.606 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:48:26.607 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:48:26.607 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:48:26.607 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:48:26.607 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:48:26.607 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 3580 2024-09-20T22:48:26.607 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.607 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:48:26.607 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.607 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:48:26.607 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:22.529195+0000", 2024-09-20T22:48:26.607 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:48:26.607 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.607 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.607 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:48:26.607 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:22.529195+0000", 2024-09-20T22:48:26.608 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:48:26.608 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.608 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.608 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:48:26.608 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:22.529195+0000", 2024-09-20T22:48:26.608 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.3000000000000001e-07 2024-09-20T22:48:26.608 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.608 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.608 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:48:26.608 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:22.529199+0000", 2024-09-20T22:48:26.608 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.416e-06 2024-09-20T22:48:26.608 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.608 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.608 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:48:26.609 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:22.529204+0000", 2024-09-20T22:48:26.609 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.6319999999999996e-06 2024-09-20T22:48:26.609 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.609 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.609 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:48:26.609 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:22.529209+0000", 2024-09-20T22:48:26.609 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.5850000000000001e-06 2024-09-20T22:48:26.609 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.609 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.609 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:48:26.609 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:22.529345+0000", 2024-09-20T22:48:26.609 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00013665899999999999 2024-09-20T22:48:26.609 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.609 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.609 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:48:26.610 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:22.529428+0000", 2024-09-20T22:48:26.610 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.2408000000000001e-05 2024-09-20T22:48:26.610 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.610 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.610 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 1,3", 2024-09-20T22:48:26.610 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:22.536949+0000", 2024-09-20T22:48:26.610 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0075212220000000002 2024-09-20T22:48:26.610 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.610 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.610 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:48:26.610 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:22.539734+0000", 2024-09-20T22:48:26.610 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0027850829999999998 2024-09-20T22:48:26.610 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.610 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.611 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:48:26.611 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:22.552673+0000", 2024-09-20T22:48:26.611 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.012938586 2024-09-20T22:48:26.611 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.611 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.611 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:48:26.611 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:22.552718+0000", 2024-09-20T22:48:26.611 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.5071999999999998e-05 2024-09-20T22:48:26.611 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.611 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.611 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:48:26.611 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:22.552764+0000", 2024-09-20T22:48:26.611 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.6570999999999998e-05 2024-09-20T22:48:26.611 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.611 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.612 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:48:26.612 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:22.552829+0000", 2024-09-20T22:48:26.612 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.4690000000000006e-05 2024-09-20T22:48:26.612 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:48:26.612 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:48:26.612 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:48:26.612 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.612 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.612 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:3747 7.c 7:3f865ef5:::smithi08477203-28:head [copy-from ver 108] snapc 98=[98,97,96,95,94,93,92,8e,8d,8c,8b,8a,88,87,86,80,7f,73] ondisk+write+known_if_redirected+supports_pool_eio e383)", 2024-09-20T22:48:26.612 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:47:43.253429+0000", 2024-09-20T22:48:26.612 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 43.232707148000003, 2024-09-20T22:48:26.612 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.028706249999999999, 2024-09-20T22:48:26.612 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:48:26.612 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:48:26.612 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:48:26.613 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:48:26.613 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:48:26.613 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:48:26.613 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 3747 2024-09-20T22:48:26.613 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.613 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:48:26.613 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.613 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:48:26.613 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.253429+0000", 2024-09-20T22:48:26.613 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:48:26.613 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.613 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.613 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:48:26.613 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.253429+0000", 2024-09-20T22:48:26.614 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:48:26.614 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.614 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.614 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:48:26.614 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.253431+0000", 2024-09-20T22:48:26.614 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.829e-06 2024-09-20T22:48:26.614 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.614 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.614 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:48:26.614 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.253438+0000", 2024-09-20T22:48:26.614 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.418e-06 2024-09-20T22:48:26.614 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.614 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.614 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:48:26.614 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.253451+0000", 2024-09-20T22:48:26.615 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2317000000000001e-05 2024-09-20T22:48:26.615 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.615 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.615 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:48:26.615 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.253457+0000", 2024-09-20T22:48:26.615 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.4710000000000004e-06 2024-09-20T22:48:26.615 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.615 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.615 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:48:26.615 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.253530+0000", 2024-09-20T22:48:26.615 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.3047999999999996e-05 2024-09-20T22:48:26.615 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.615 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.615 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op must wait for map", 2024-09-20T22:48:26.616 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.253536+0000", 2024-09-20T22:48:26.616 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.5140000000000001e-06 2024-09-20T22:48:26.616 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.616 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.616 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:48:26.616 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.257582+0000", 2024-09-20T22:48:26.616 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0040464849999999998 2024-09-20T22:48:26.616 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.616 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.616 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:48:26.616 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.257697+0000", 2024-09-20T22:48:26.616 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00011471 2024-09-20T22:48:26.616 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.616 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.616 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 3,7", 2024-09-20T22:48:26.617 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.267067+0000", 2024-09-20T22:48:26.617 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0093696560000000005 2024-09-20T22:48:26.617 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.617 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.617 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:48:26.617 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.276328+0000", 2024-09-20T22:48:26.617 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0092619669999999994 2024-09-20T22:48:26.617 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.617 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.617 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:48:26.617 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.278462+0000", 2024-09-20T22:48:26.617 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0021334430000000001 2024-09-20T22:48:26.617 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.617 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.617 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:48:26.618 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.282036+0000", 2024-09-20T22:48:26.618 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0035745249999999998 2024-09-20T22:48:26.618 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.618 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.618 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:48:26.618 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.282094+0000", 2024-09-20T22:48:26.618 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.7312999999999998e-05 2024-09-20T22:48:26.618 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.618 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.618 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:48:26.618 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.282135+0000", 2024-09-20T22:48:26.618 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.1554000000000002e-05 2024-09-20T22:48:26.618 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:48:26.619 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:48:26.619 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:48:26.619 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.619 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.619 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:3856 7.12 7:4fa4b2b3:::smithi08477203-21:head [copy-from ver 170] snapc 9e=[9e,9d,9c,9b,9a,98,96,95,94,93,92,8e,8d,8c,8b,8a,88,87,86,80,7f,73] ondisk+write+known_if_redirected+supports_pool_eio e391)", 2024-09-20T22:48:26.619 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:47:51.204384+0000", 2024-09-20T22:48:26.619 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 35.281751829999997, 2024-09-20T22:48:26.619 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0068049349999999, 2024-09-20T22:48:26.619 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:48:26.619 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:48:26.619 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:48:26.619 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:48:26.619 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:48:26.619 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:48:26.619 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 3856 2024-09-20T22:48:26.620 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.620 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:48:26.620 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.620 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:48:26.620 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:51.204384+0000", 2024-09-20T22:48:26.620 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:48:26.620 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.620 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.620 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:48:26.620 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:51.204384+0000", 2024-09-20T22:48:26.620 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:48:26.620 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.620 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.620 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:48:26.621 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:51.204386+0000", 2024-09-20T22:48:26.621 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.57e-06 2024-09-20T22:48:26.621 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.621 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.621 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:48:26.621 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:51.204391+0000", 2024-09-20T22:48:26.621 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.6009999999999997e-06 2024-09-20T22:48:26.621 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.621 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.621 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:48:26.621 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:51.204400+0000", 2024-09-20T22:48:26.621 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.0499999999999997e-06 2024-09-20T22:48:26.621 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.621 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.621 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:48:26.622 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:51.204410+0000", 2024-09-20T22:48:26.622 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.01e-05 2024-09-20T22:48:26.622 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.622 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.622 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:48:26.622 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:51.204944+0000", 2024-09-20T22:48:26.622 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00053440599999999997 2024-09-20T22:48:26.622 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.622 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.622 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-20T22:48:26.622 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:51.204954+0000", 2024-09-20T22:48:26.622 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0162e-05 2024-09-20T22:48:26.622 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.622 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.623 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:48:26.623 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:52.210122+0000", 2024-09-20T22:48:26.623 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0051679019999999 2024-09-20T22:48:26.623 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.623 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.623 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:48:26.623 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:52.210316+0000", 2024-09-20T22:48:26.623 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000193704 2024-09-20T22:48:26.623 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.623 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.623 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:48:26.623 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:52.211189+0000", 2024-09-20T22:48:26.623 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00087343999999999996 2024-09-20T22:48:26.623 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:48:26.623 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:48:26.624 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:48:26.624 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.624 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.624 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:3857 7.12 7:4fa4b2b3:::smithi08477203-21:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e391)", 2024-09-20T22:48:26.624 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:47:51.204423+0000", 2024-09-20T22:48:26.624 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 35.28171339, 2024-09-20T22:48:26.624 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0069932960000001, 2024-09-20T22:48:26.624 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:48:26.624 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:48:26.624 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:48:26.624 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:48:26.624 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:48:26.624 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:48:26.624 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 3857 2024-09-20T22:48:26.625 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.625 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:48:26.625 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.625 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:48:26.625 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:51.204423+0000", 2024-09-20T22:48:26.625 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:48:26.625 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.625 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.625 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:48:26.625 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:51.204423+0000", 2024-09-20T22:48:26.625 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:48:26.625 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.625 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.625 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:48:26.626 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:51.204423+0000", 2024-09-20T22:48:26.626 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.3099999999999998e-07 2024-09-20T22:48:26.626 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.626 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.626 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:48:26.626 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:51.204426+0000", 2024-09-20T22:48:26.626 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.4030000000000001e-06 2024-09-20T22:48:26.626 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.626 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.626 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:48:26.626 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:51.204432+0000", 2024-09-20T22:48:26.626 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.4099999999999996e-06 2024-09-20T22:48:26.626 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.626 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.627 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:48:26.627 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:51.204438+0000", 2024-09-20T22:48:26.627 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.4600000000000002e-06 2024-09-20T22:48:26.627 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.627 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.627 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:48:26.627 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:51.204980+0000", 2024-09-20T22:48:26.627 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00054220600000000005 2024-09-20T22:48:26.627 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.627 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.627 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-20T22:48:26.627 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:51.204987+0000", 2024-09-20T22:48:26.627 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.3320000000000003e-06 2024-09-20T22:48:26.627 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.627 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.628 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:48:26.628 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:52.211064+0000", 2024-09-20T22:48:26.628 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0060768899999999 2024-09-20T22:48:26.628 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.628 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.628 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for blocked object", 2024-09-20T22:48:26.628 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:52.211095+0000", 2024-09-20T22:48:26.628 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.1399999999999998e-05 2024-09-20T22:48:26.628 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.628 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.628 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:48:26.628 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:52.211260+0000", 2024-09-20T22:48:26.628 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00016436099999999999 2024-09-20T22:48:26.628 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.629 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.629 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:48:26.629 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:52.211310+0000", 2024-09-20T22:48:26.629 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.0417999999999997e-05 2024-09-20T22:48:26.629 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:26.629 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:26.629 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:48:26.629 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:52.211416+0000", 2024-09-20T22:48:26.629 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00010588499999999999 2024-09-20T22:48:26.629 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:48:26.629 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:48:26.629 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:48:26.629 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:48:26.629 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:48:26.629 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-20T22:48:26.630 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:48:26.633 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:48:26.633 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:48:26.649 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:48:26.649 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:48:26.651 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:26.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:26.652 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:48:26.664 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:48:26.665 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:48:26.680 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-20T22:48:26.680 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T22:48:26.696 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:48:26.697 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:48:26.713 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:48:26.714 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:48:26.730 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:48:26.731 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:48:26.747 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:48:26.747 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:48:26.753 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:48:26.774 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:26.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:26.775 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:48:26.875 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:48:26.898 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:26.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:26.899 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:48:26.999 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:48:27.021 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:27.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:27.022 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:48:27.122 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:48:27.143 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:27.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:27.144 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:48:27.245 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:48:27.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:27.267 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:27.267 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:48:27.275 INFO:tasks.rados.rados.0.smithi084.stdout:1652: finishing rollback tid 1 to smithi08477203-30 2024-09-20T22:48:27.275 INFO:tasks.rados.rados.0.smithi084.stdout:1652: finishing rollback tid 2 to smithi08477203-30 2024-09-20T22:48:27.275 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 30 v 730 (ObjNum 416 snap 105 seq_num 416) dirty exists 2024-09-20T22:48:27.277 INFO:tasks.rados.rados.0.smithi084.stdout:1650: done (4 left) 2024-09-20T22:48:27.277 INFO:tasks.rados.rados.0.smithi084.stdout:1651: done (3 left) 2024-09-20T22:48:27.277 INFO:tasks.rados.rados.0.smithi084.stdout:1652: done (2 left) 2024-09-20T22:48:27.277 INFO:tasks.rados.rados.0.smithi084.stdout:1654: done (1 left) 2024-09-20T22:48:27.277 INFO:tasks.rados.rados.0.smithi084.stdout:1655: done (0 left) 2024-09-20T22:48:27.277 INFO:tasks.rados.rados.0.smithi084.stdout:1656: rollback oid 13 current snap is 174 2024-09-20T22:48:27.278 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 13 to 139 2024-09-20T22:48:27.278 INFO:tasks.rados.rados.0.smithi084.stdout:1657: write oid 19 current snap is 174 2024-09-20T22:48:27.278 INFO:tasks.rados.rados.0.smithi084.stdout:1657: seq_num 589 ranges {699323=480842,1850247=429872,2858378=532392} 2024-09-20T22:48:27.283 INFO:tasks.rados.rados.0.smithi084.stdout:1657: writing smithi08477203-19 from 699323 to 1180165 tid 1 2024-09-20T22:48:27.289 INFO:tasks.rados.rados.0.smithi084.stdout:1657: writing smithi08477203-19 from 1850247 to 2280119 tid 2 2024-09-20T22:48:27.296 INFO:tasks.rados.rados.0.smithi084.stdout:1657: writing smithi08477203-19 from 2858378 to 3390770 tid 3 2024-09-20T22:48:27.296 INFO:tasks.rados.rados.0.smithi084.stdout:1658: rollback oid 22 current snap is 174 2024-09-20T22:48:27.296 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 22 to 152 2024-09-20T22:48:27.296 INFO:tasks.rados.rados.0.smithi084.stdout:1656: finishing rollback tid 0 to smithi08477203-13 2024-09-20T22:48:27.296 INFO:tasks.rados.rados.0.smithi084.stdout:1656: finishing rollback tid 1 to smithi08477203-13 2024-09-20T22:48:27.296 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 13 v 966 (ObjNum 65 snap 0 seq_num 1) dirty dne 2024-09-20T22:48:27.297 INFO:tasks.rados.rados.0.smithi084.stdout:1657: finishing write tid 1 to smithi08477203-19 2024-09-20T22:48:27.297 INFO:tasks.rados.rados.0.smithi084.stdout:1657: finishing write tid 2 to smithi08477203-19 2024-09-20T22:48:27.297 INFO:tasks.rados.rados.0.smithi084.stdout:1656: done (2 left) 2024-09-20T22:48:27.297 INFO:tasks.rados.rados.0.smithi084.stdout:1659: rollback oid 26 current snap is 174 2024-09-20T22:48:27.297 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 26 to 155 2024-09-20T22:48:27.297 INFO:tasks.rados.rados.0.smithi084.stdout:1660: delete oid 46 current snap is 174 2024-09-20T22:48:27.297 INFO:tasks.rados.rados.0.smithi084.stdout:1660: done (3 left) 2024-09-20T22:48:27.298 INFO:tasks.rados.rados.0.smithi084.stdout:1661: snap_remove snap 113 2024-09-20T22:48:27.334 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-20T22:48:27.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:27 smithi084 ceph-mon[29774]: pgmap v540: 168 pgs: 3 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 159 active+clean; 391 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 141 MiB/s rd, 39 MiB/s wr, 242 op/s; 23 MiB/s, 50 objects/s recovering 2024-09-20T22:48:27.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:27 smithi084 ceph-mon[29774]: osdmap e426: 8 total, 8 up, 8 in 2024-09-20T22:48:27.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:27 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/3254440883' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:48:27.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:27 smithi084 ceph-mon[40024]: pgmap v540: 168 pgs: 3 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 159 active+clean; 391 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 141 MiB/s rd, 39 MiB/s wr, 242 op/s; 23 MiB/s, 50 objects/s recovering 2024-09-20T22:48:27.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:27 smithi084 ceph-mon[40024]: osdmap e426: 8 total, 8 up, 8 in 2024-09-20T22:48:27.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:27 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/3254440883' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:48:27.369 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:48:27.397 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:27.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:27.398 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:48:27.499 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:48:27.522 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:27.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:27.523 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:48:27.582 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:48:27.623 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:48:27.645 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:27.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:27.645 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:48:27.703 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:27 smithi154 ceph-mon[33264]: pgmap v540: 168 pgs: 3 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 159 active+clean; 391 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 141 MiB/s rd, 39 MiB/s wr, 242 op/s; 23 MiB/s, 50 objects/s recovering 2024-09-20T22:48:27.703 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:27 smithi154 ceph-mon[33264]: osdmap e426: 8 total, 8 up, 8 in 2024-09-20T22:48:27.703 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:27 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/3254440883' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:48:27.746 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:48:27.767 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:27.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:27.768 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:48:27.869 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:48:27.891 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:27.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:27.892 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:48:27.993 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:48:28.014 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:28.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:28.016 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:48:28.049 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-20T22:48:28.049 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:48:28.050 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-20T22:48:28.117 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:48:28.138 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:28.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:28.139 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:48:28.205 INFO:tasks.rados.rados.0.smithi084.stdout:1661: done (3 left) 2024-09-20T22:48:28.205 INFO:tasks.rados.rados.0.smithi084.stdout:1662: rollback oid 37 current snap is 174 2024-09-20T22:48:28.206 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 37 to 132 2024-09-20T22:48:28.206 INFO:tasks.rados.rados.0.smithi084.stdout:1659: finishing rollback tid 1 to smithi08477203-26 2024-09-20T22:48:28.206 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 26 v 340 (ObjNum 49 snap 0 seq_num 1953066355) dirty dne 2024-09-20T22:48:28.206 INFO:tasks.rados.rados.0.smithi084.stdout:1658: finishing rollback tid 1 to smithi08477203-22 2024-09-20T22:48:28.206 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 22 v 324 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-09-20T22:48:28.206 INFO:tasks.rados.rados.0.smithi084.stdout:1657: finishing write tid 3 to smithi08477203-19 2024-09-20T22:48:28.206 INFO:tasks.rados.rados.0.smithi084.stdout:1657: finishing write tid 4 to smithi08477203-19 2024-09-20T22:48:28.206 INFO:tasks.rados.rados.0.smithi084.stdout:1657: finishing write tid 5 to smithi08477203-19 2024-09-20T22:48:28.206 INFO:tasks.rados.rados.0.smithi084.stdout:1657: oid 19 updating version 0 to 970 2024-09-20T22:48:28.206 INFO:tasks.rados.rados.0.smithi084.stdout:1657: oid 19 version 970 is already newer than 968 2024-09-20T22:48:28.206 INFO:tasks.rados.rados.0.smithi084.stdout:1657: oid 19 version 970 is already newer than 969 2024-09-20T22:48:28.206 INFO:tasks.rados.rados.0.smithi084.stdout:1657: oid 19 updating version 970 to 971 2024-09-20T22:48:28.206 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 19 v 971 (ObjNum 589 snap 174 seq_num 589) dirty exists 2024-09-20T22:48:28.207 INFO:tasks.rados.rados.0.smithi084.stdout:1657: left oid 19 (ObjNum 589 snap 174 seq_num 589) 2024-09-20T22:48:28.207 INFO:tasks.rados.rados.0.smithi084.stdout:1657: done (3 left) 2024-09-20T22:48:28.207 INFO:tasks.rados.rados.0.smithi084.stdout:1658: done (2 left) 2024-09-20T22:48:28.207 INFO:tasks.rados.rados.0.smithi084.stdout:1659: done (1 left) 2024-09-20T22:48:28.207 INFO:tasks.rados.rados.0.smithi084.stdout:1663: read oid 41 snap -1 2024-09-20T22:48:28.207 INFO:tasks.rados.rados.0.smithi084.stdout:1663: expect (ObjNum 576 snap 172 seq_num 576) 2024-09-20T22:48:28.207 INFO:tasks.rados.rados.0.smithi084.stdout:1664: write (excl) oid 13 current snap is 174 2024-09-20T22:48:28.207 INFO:tasks.rados.rados.0.smithi084.stdout:1664: seq_num 591 ranges {732937=631340,1850858=433541,2796871=671289,3888195=44742} 2024-09-20T22:48:28.213 INFO:tasks.rados.rados.0.smithi084.stdout:1664: writing smithi08477203-13 from 732937 to 1364277 tid 1 2024-09-20T22:48:28.219 INFO:tasks.rados.rados.0.smithi084.stdout:1664: writing smithi08477203-13 from 1850858 to 2284399 tid 2 2024-09-20T22:48:28.227 INFO:tasks.rados.rados.0.smithi084.stdout:1664: writing smithi08477203-13 from 2796871 to 3468160 tid 3 2024-09-20T22:48:28.230 INFO:tasks.rados.rados.0.smithi084.stdout:1664: writing smithi08477203-13 from 3888195 to 3932937 tid 4 2024-09-20T22:48:28.232 INFO:tasks.rados.rados.0.smithi084.stdout:1664: finishing write tid 1 to smithi08477203-13 2024-09-20T22:48:28.232 INFO:tasks.rados.rados.0.smithi084.stdout:1663: expect (ObjNum 576 snap 172 seq_num 576) 2024-09-20T22:48:28.241 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:48:28.262 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:28.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:28.263 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:48:28.326 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:48:28.361 INFO:tasks.rados.rados.0.smithi084.stdout:1664: finishing write tid 2 to smithi08477203-13 2024-09-20T22:48:28.361 INFO:tasks.rados.rados.0.smithi084.stdout:1662: finishing rollback tid 0 to smithi08477203-37 2024-09-20T22:48:28.362 INFO:tasks.rados.rados.0.smithi084.stdout:1662: finishing rollback tid 1 to smithi08477203-37 2024-09-20T22:48:28.362 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 37 v 842 (ObjNum 289 snap 0 seq_num 1) dirty dne 2024-09-20T22:48:28.362 INFO:tasks.rados.rados.0.smithi084.stdout:1662: done (2 left) 2024-09-20T22:48:28.362 INFO:tasks.rados.rados.0.smithi084.stdout:1663: done (1 left) 2024-09-20T22:48:28.362 INFO:tasks.rados.rados.0.smithi084.stdout:1665: write (excl) oid 2 current snap is 174 2024-09-20T22:48:28.362 INFO:tasks.rados.rados.0.smithi084.stdout:1664: finishing write tid 3 to smithi08477203-13 2024-09-20T22:48:28.362 INFO:tasks.rados.rados.0.smithi084.stdout:1664: finishing write tid 4 to smithi08477203-13 2024-09-20T22:48:28.362 INFO:tasks.rados.rados.0.smithi084.stdout:1664: finishing write tid 5 to smithi08477203-13 2024-09-20T22:48:28.362 INFO:tasks.rados.rados.0.smithi084.stdout:1664: finishing write tid 6 to smithi08477203-13 2024-09-20T22:48:28.362 INFO:tasks.rados.rados.0.smithi084.stdout:1664: oid 13 updating version 0 to 977 2024-09-20T22:48:28.362 INFO:tasks.rados.rados.0.smithi084.stdout:1664: oid 13 updating version 977 to 982 2024-09-20T22:48:28.362 INFO:tasks.rados.rados.0.smithi084.stdout:1664: oid 13 version 982 is already newer than 976 2024-09-20T22:48:28.362 INFO:tasks.rados.rados.0.smithi084.stdout:1664: oid 13 updating version 982 to 983 2024-09-20T22:48:28.363 INFO:tasks.rados.rados.0.smithi084.stdout:1664: oid 13 updating version 983 to 984 2024-09-20T22:48:28.363 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 13 v 984 (ObjNum 591 snap 174 seq_num 591) dirty exists 2024-09-20T22:48:28.363 INFO:tasks.rados.rados.0.smithi084.stdout:1664: left oid 13 (ObjNum 591 snap 174 seq_num 591) 2024-09-20T22:48:28.363 INFO:tasks.rados.rados.0.smithi084.stdout:1665: seq_num 592 ranges {749744=706589,1909340=577199,3224294=482561} 2024-09-20T22:48:28.364 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:48:28.365 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:48:28 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: ::ffff:172.21.15.154 - - [20/Sep/2024:22:48:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-20T22:48:28.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:28 smithi084 ceph-mon[40024]: osdmap e427: 8 total, 8 up, 8 in 2024-09-20T22:48:28.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:28 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "24"}]: dispatch 2024-09-20T22:48:28.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:28 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "24"}]: dispatch 2024-09-20T22:48:28.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:28 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "24"}]': finished 2024-09-20T22:48:28.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:28 smithi084 ceph-mon[40024]: osdmap e428: 8 total, 8 up, 8 in 2024-09-20T22:48:28.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:28 smithi084 ceph-mon[29774]: osdmap e427: 8 total, 8 up, 8 in 2024-09-20T22:48:28.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:28 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "24"}]: dispatch 2024-09-20T22:48:28.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:28 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "24"}]: dispatch 2024-09-20T22:48:28.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:28 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "24"}]': finished 2024-09-20T22:48:28.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:28 smithi084 ceph-mon[29774]: osdmap e428: 8 total, 8 up, 8 in 2024-09-20T22:48:28.369 INFO:tasks.rados.rados.0.smithi084.stdout:1665: writing smithi08477203-2 from 749744 to 1456333 tid 1 2024-09-20T22:48:28.375 INFO:tasks.rados.rados.0.smithi084.stdout:1665: writing smithi08477203-2 from 1909340 to 2486539 tid 2 2024-09-20T22:48:28.382 INFO:tasks.rados.rados.0.smithi084.stdout:1665: writing smithi08477203-2 from 3224294 to 3706855 tid 3 2024-09-20T22:48:28.382 INFO:tasks.rados.rados.0.smithi084.stdout:1664: done (1 left) 2024-09-20T22:48:28.382 INFO:tasks.rados.rados.0.smithi084.stdout:1666: write (excl) oid 35 current snap is 174 2024-09-20T22:48:28.382 INFO:tasks.rados.rados.0.smithi084.stdout:1665: finishing write tid 1 to smithi08477203-2 2024-09-20T22:48:28.382 INFO:tasks.rados.rados.0.smithi084.stdout:1665: finishing write tid 2 to smithi08477203-2 2024-09-20T22:48:28.382 INFO:tasks.rados.rados.0.smithi084.stdout:1666: seq_num 593 ranges {766551=498191,1684175=720857,2851717=693833} 2024-09-20T22:48:28.386 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:28.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:28.387 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:48:28.389 INFO:tasks.rados.rados.0.smithi084.stdout:1666: writing smithi08477203-35 from 766551 to 1264742 tid 1 2024-09-20T22:48:28.395 INFO:tasks.rados.rados.0.smithi084.stdout:1666: writing smithi08477203-35 from 1684175 to 2405032 tid 2 2024-09-20T22:48:28.401 INFO:tasks.rados.rados.0.smithi084.stdout:1666: writing smithi08477203-35 from 2851717 to 3545550 tid 3 2024-09-20T22:48:28.401 INFO:tasks.rados.rados.0.smithi084.stdout:1667: rollback oid 6 current snap is 174 2024-09-20T22:48:28.402 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 6 to 157 2024-09-20T22:48:28.402 INFO:tasks.rados.rados.0.smithi084.stdout:1665: finishing write tid 3 to smithi08477203-2 2024-09-20T22:48:28.402 INFO:tasks.rados.rados.0.smithi084.stdout:1665: finishing write tid 4 to smithi08477203-2 2024-09-20T22:48:28.402 INFO:tasks.rados.rados.0.smithi084.stdout:1665: finishing write tid 5 to smithi08477203-2 2024-09-20T22:48:28.402 INFO:tasks.rados.rados.0.smithi084.stdout:1665: oid 2 updating version 0 to 196 2024-09-20T22:48:28.402 INFO:tasks.rados.rados.0.smithi084.stdout:1665: oid 2 version 196 is already newer than 195 2024-09-20T22:48:28.402 INFO:tasks.rados.rados.0.smithi084.stdout:1665: oid 2 version 196 is already newer than 194 2024-09-20T22:48:28.402 INFO:tasks.rados.rados.0.smithi084.stdout:1665: oid 2 updating version 196 to 197 2024-09-20T22:48:28.402 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 2 v 197 (ObjNum 592 snap 174 seq_num 592) dirty exists 2024-09-20T22:48:28.402 INFO:tasks.rados.rados.0.smithi084.stdout:1665: left oid 2 (ObjNum 592 snap 174 seq_num 592) 2024-09-20T22:48:28.402 INFO:tasks.rados.rados.0.smithi084.stdout:1665: done (2 left) 2024-09-20T22:48:28.402 INFO:tasks.rados.rados.0.smithi084.stdout:1668: read oid 28 snap -1 2024-09-20T22:48:28.402 INFO:tasks.rados.rados.0.smithi084.stdout:1668: expect (ObjNum 470 snap 130 seq_num 470) 2024-09-20T22:48:28.403 INFO:tasks.rados.rados.0.smithi084.stdout:1669: read oid 26 snap -1 2024-09-20T22:48:28.403 INFO:tasks.rados.rados.0.smithi084.stdout:1669: expect deleted 2024-09-20T22:48:28.403 INFO:tasks.rados.rados.0.smithi084.stdout:1670: snap_remove snap 139 2024-09-20T22:48:28.488 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:48:28.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:28 smithi154 ceph-mon[33264]: osdmap e427: 8 total, 8 up, 8 in 2024-09-20T22:48:28.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:28 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "24"}]: dispatch 2024-09-20T22:48:28.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:28 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "24"}]: dispatch 2024-09-20T22:48:28.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:28 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "24"}]': finished 2024-09-20T22:48:28.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:28 smithi154 ceph-mon[33264]: osdmap e428: 8 total, 8 up, 8 in 2024-09-20T22:48:28.510 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:28.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:28.511 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:48:28.612 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:48:28.633 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:28.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:28.634 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:48:28.735 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:48:28.757 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:28.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:28.758 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:48:28.760 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.7 dump_blocked_ops 2024-09-20T22:48:28.859 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:48:28.880 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:28.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:28.881 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:48:28.982 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:48:29.004 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:29.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:29.005 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:48:29.050 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:48:29.106 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:48:29.127 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:29.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:29.128 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:48:29.213 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is set 2024-09-20T22:48:29.222 INFO:tasks.rados.rados.0.smithi084.stdout:1670: done (4 left) 2024-09-20T22:48:29.222 INFO:tasks.rados.rados.0.smithi084.stdout:1671: read oid 1 snap -1 2024-09-20T22:48:29.222 INFO:tasks.rados.rados.0.smithi084.stdout:1671: expect (ObjNum 563 snap 170 seq_num 563) 2024-09-20T22:48:29.222 INFO:tasks.rados.rados.0.smithi084.stdout:1672: snap_remove snap 126 2024-09-20T22:48:29.229 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:48:29.255 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:29.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:29.256 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:48:29.357 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:48:29.377 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:29.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:29.378 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:48:29.444 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 5, 6, 7, 0, 4] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-20T22:48:29.444 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-20T22:48:29.444 INFO:tasks.thrashosds.thrasher:inject_pause on osd.6 2024-09-20T22:48:29.444 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-20T22:48:29.445 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-20T22:48:29.445 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.6 config set filestore_inject_stall 3 2024-09-20T22:48:29.458 INFO:journalctl@ceph.iscsi.iscsi.a.smithi154.stdout:Sep 20 22:48:29 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-iscsi-iscsi-a[59730]: debug there is no tcmu-runner data available 2024-09-20T22:48:29.458 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:29 smithi154 ceph-mon[33264]: pgmap v543: 167 pgs: 3 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 158 active+clean; 369 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail 2024-09-20T22:48:29.458 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:29 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/2905972036' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:48:29.459 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:29 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/2905972036' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T22:48:29.459 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:29 smithi154 ceph-mon[33264]: osdmap e429: 8 total, 8 up, 8 in 2024-09-20T22:48:29.479 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:48:29.500 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:29.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:29.501 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:48:29.516 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:29 smithi084 ceph-mon[40024]: pgmap v543: 167 pgs: 3 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 158 active+clean; 369 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail 2024-09-20T22:48:29.516 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:29 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/2905972036' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:48:29.516 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:29 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/2905972036' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T22:48:29.516 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:29 smithi084 ceph-mon[40024]: osdmap e429: 8 total, 8 up, 8 in 2024-09-20T22:48:29.516 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:29 smithi084 ceph-mon[29774]: pgmap v543: 167 pgs: 3 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 158 active+clean; 369 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail 2024-09-20T22:48:29.517 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:29 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/2905972036' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:48:29.517 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:29 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/2905972036' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T22:48:29.517 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:29 smithi084 ceph-mon[29774]: osdmap e429: 8 total, 8 up, 8 in 2024-09-20T22:48:29.552 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-20T22:48:29.560 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-20T22:48:29.560 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:48:29.601 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:48:29.622 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:29.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:29.623 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:48:29.724 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:48:29.745 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:29.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:29.746 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:48:29.846 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:48:29.868 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:29.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:29.869 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:48:29.913 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:48:29.970 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:48:29.992 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:29.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:29.993 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:48:30.094 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:48:30.115 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:30.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:30.116 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:48:30.217 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:48:30.221 INFO:tasks.rados.rados.0.smithi084.stdout:1672: done (5 left) 2024-09-20T22:48:30.221 INFO:tasks.rados.rados.0.smithi084.stdout:1673: read oid 39 snap 163 2024-09-20T22:48:30.221 INFO:tasks.rados.rados.0.smithi084.stdout:1673: expect deleted 2024-09-20T22:48:30.222 INFO:tasks.rados.rados.0.smithi084.stdout:1674: rollback oid 17 current snap is 174 2024-09-20T22:48:30.222 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 17 to 147 2024-09-20T22:48:30.222 INFO:tasks.rados.rados.0.smithi084.stdout:1675: write oid 41 current snap is 174 2024-09-20T22:48:30.222 INFO:tasks.rados.rados.0.smithi084.stdout:1675: seq_num 594 ranges {783358=573440,2026304=580868,3279140=621458} 2024-09-20T22:48:30.229 INFO:tasks.rados.rados.0.smithi084.stdout:1675: writing smithi08477203-41 from 783358 to 1356798 tid 1 2024-09-20T22:48:30.237 INFO:tasks.rados.rados.0.smithi084.stdout:1675: writing smithi08477203-41 from 2026304 to 2607172 tid 2 2024-09-20T22:48:30.245 INFO:tasks.rados.rados.0.smithi084.stdout:1675: writing smithi08477203-41 from 3279140 to 3900598 tid 3 2024-09-20T22:48:30.245 INFO:tasks.rados.rados.0.smithi084.stdout:1676: snap_remove snap 153 2024-09-20T22:48:30.247 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:30.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:30.248 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:48:30.348 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:48:30.362 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:48:30.362 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:48:30.369 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:30.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:30.370 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:48:30.377 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:48:30.377 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:48:30.393 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:48:30.393 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:48:30.393 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:30 smithi084 ceph-mon[40024]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:48:30.393 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:30 smithi084 ceph-mon[40024]: osdmap e430: 8 total, 8 up, 8 in 2024-09-20T22:48:30.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:30 smithi084 ceph-mon[29774]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:48:30.394 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:30 smithi084 ceph-mon[29774]: osdmap e430: 8 total, 8 up, 8 in 2024-09-20T22:48:30.409 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-20T22:48:30.409 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T22:48:30.426 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:48:30.426 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:48:30.442 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:48:30.443 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:48:30.459 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:48:30.459 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:48:30.471 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:48:30.476 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:48:30.476 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:48:30.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:30.495 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:30.495 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:48:30.542 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:30 smithi154 ceph-mon[33264]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:48:30.543 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:30 smithi154 ceph-mon[33264]: osdmap e430: 8 total, 8 up, 8 in 2024-09-20T22:48:30.544 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.7 dump_historic_ops 2024-09-20T22:48:30.599 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:48:30.621 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:30.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:30.622 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:48:30.638 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:48:30.723 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:48:30.745 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:30.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:30.746 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:48:30.847 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:48:30.870 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:30.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:30.871 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:48:30.972 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:48:30.994 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:30.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:30.995 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:48:31.095 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:48:31.117 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:31.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:31.118 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:48:31.123 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-20T22:48:31.123 INFO:teuthology.orchestra.run.smithi154.stdout: "success": "" 2024-09-20T22:48:31.123 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-20T22:48:31.218 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:48:31.220 INFO:tasks.rados.rados.0.smithi084.stdout:1676: done (8 left) 2024-09-20T22:48:31.220 INFO:tasks.rados.rados.0.smithi084.stdout:1677: snap_create 2024-09-20T22:48:31.220 INFO:tasks.rados.rados.0.smithi084.stdout:1667: finishing rollback tid 0 to smithi08477203-6 2024-09-20T22:48:31.221 INFO:tasks.rados.rados.0.smithi084.stdout:1667: finishing rollback tid 1 to smithi08477203-6 2024-09-20T22:48:31.221 INFO:tasks.rados.rados.0.smithi084.stdout:1667: finishing rollback tid 2 to smithi08477203-6 2024-09-20T22:48:31.221 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 6 v 320 (ObjNum 502 snap 143 seq_num 502) dirty exists 2024-09-20T22:48:31.221 INFO:tasks.rados.rados.0.smithi084.stdout:1668: expect (ObjNum 470 snap 130 seq_num 470) 2024-09-20T22:48:31.245 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:31.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:31.246 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:48:31.347 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:48:31.354 INFO:tasks.rados.rados.0.smithi084.stdout:1671: expect (ObjNum 563 snap 170 seq_num 563) 2024-09-20T22:48:31.368 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:48:31.368 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:31.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:31.369 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:48:31.470 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:48:31.490 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:31.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:31.491 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:48:31.518 INFO:tasks.rados.rados.0.smithi084.stdout:1674: finishing rollback tid 1 to smithi08477203-17 2024-09-20T22:48:31.518 INFO:tasks.rados.rados.0.smithi084.stdout:1674: finishing rollback tid 2 to smithi08477203-17 2024-09-20T22:48:31.519 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 17 v 322 (ObjNum 426 snap 109 seq_num 426) dirty exists 2024-09-20T22:48:31.519 INFO:tasks.rados.rados.0.smithi084.stdout:1675: finishing write tid 1 to smithi08477203-41 2024-09-20T22:48:31.519 INFO:tasks.rados.rados.0.smithi084.stdout:1675: finishing write tid 2 to smithi08477203-41 2024-09-20T22:48:31.519 INFO:tasks.rados.rados.0.smithi084.stdout:1675: finishing write tid 3 to smithi08477203-41 2024-09-20T22:48:31.519 INFO:tasks.rados.rados.0.smithi084.stdout:1675: finishing write tid 4 to smithi08477203-41 2024-09-20T22:48:31.519 INFO:tasks.rados.rados.0.smithi084.stdout:1675: finishing write tid 5 to smithi08477203-41 2024-09-20T22:48:31.520 INFO:tasks.rados.rados.0.smithi084.stdout:1675: oid 41 updating version 0 to 417 2024-09-20T22:48:31.520 INFO:tasks.rados.rados.0.smithi084.stdout:1675: oid 41 version 417 is already newer than 415 2024-09-20T22:48:31.520 INFO:tasks.rados.rados.0.smithi084.stdout:1675: oid 41 updating version 417 to 418 2024-09-20T22:48:31.520 INFO:tasks.rados.rados.0.smithi084.stdout:1675: oid 41 version 418 is already newer than 416 2024-09-20T22:48:31.520 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 41 v 418 (ObjNum 594 snap 174 seq_num 594) dirty exists 2024-09-20T22:48:31.520 INFO:tasks.rados.rados.0.smithi084.stdout:1675: left oid 41 (ObjNum 594 snap 174 seq_num 594) 2024-09-20T22:48:31.520 INFO:tasks.rados.rados.0.smithi084.stdout:1666: finishing write tid 1 to smithi08477203-35 2024-09-20T22:48:31.521 INFO:tasks.rados.rados.0.smithi084.stdout:1666: finishing write tid 2 to smithi08477203-35 2024-09-20T22:48:31.521 INFO:tasks.rados.rados.0.smithi084.stdout:1666: finishing write tid 3 to smithi08477203-35 2024-09-20T22:48:31.521 INFO:tasks.rados.rados.0.smithi084.stdout:1666: finishing write tid 4 to smithi08477203-35 2024-09-20T22:48:31.521 INFO:tasks.rados.rados.0.smithi084.stdout:1666: finishing write tid 5 to smithi08477203-35 2024-09-20T22:48:31.521 INFO:tasks.rados.rados.0.smithi084.stdout:1666: oid 35 updating version 0 to 685 2024-09-20T22:48:31.521 INFO:tasks.rados.rados.0.smithi084.stdout:1666: oid 35 version 685 is already newer than 683 2024-09-20T22:48:31.522 INFO:tasks.rados.rados.0.smithi084.stdout:1666: oid 35 updating version 685 to 686 2024-09-20T22:48:31.522 INFO:tasks.rados.rados.0.smithi084.stdout:1666: oid 35 version 686 is already newer than 684 2024-09-20T22:48:31.522 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 35 v 686 (ObjNum 593 snap 174 seq_num 593) dirty exists 2024-09-20T22:48:31.523 INFO:tasks.rados.rados.0.smithi084.stdout:1666: left oid 35 (ObjNum 593 snap 174 seq_num 593) 2024-09-20T22:48:31.592 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:48:31.614 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:31 smithi084 ceph-mon[29774]: pgmap v546: 167 pgs: 2 peering, 3 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 156 active+clean; 376 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 8.1 MiB/s rd, 1.6 MiB/s wr, 15 op/s 2024-09-20T22:48:31.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:31 smithi084 ceph-mon[29774]: purged_snaps scrub starts 2024-09-20T22:48:31.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:31 smithi084 ceph-mon[29774]: purged_snaps scrub ok 2024-09-20T22:48:31.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:31 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/3050005222' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:48:31.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:31 smithi084 ceph-mon[29774]: osdmap e431: 8 total, 8 up, 8 in 2024-09-20T22:48:31.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:31 smithi084 ceph-mon[40024]: pgmap v546: 167 pgs: 2 peering, 3 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 156 active+clean; 376 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 8.1 MiB/s rd, 1.6 MiB/s wr, 15 op/s 2024-09-20T22:48:31.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:31 smithi084 ceph-mon[40024]: purged_snaps scrub starts 2024-09-20T22:48:31.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:31 smithi084 ceph-mon[40024]: purged_snaps scrub ok 2024-09-20T22:48:31.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:31 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/3050005222' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:48:31.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:31 smithi084 ceph-mon[40024]: osdmap e431: 8 total, 8 up, 8 in 2024-09-20T22:48:31.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:31.616 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:31.617 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:48:31.717 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:48:31.739 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:31.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:31.740 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:48:31.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:31 smithi154 ceph-mon[33264]: pgmap v546: 167 pgs: 2 peering, 3 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 156 active+clean; 376 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 8.1 MiB/s rd, 1.6 MiB/s wr, 15 op/s 2024-09-20T22:48:31.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:31 smithi154 ceph-mon[33264]: purged_snaps scrub starts 2024-09-20T22:48:31.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:31 smithi154 ceph-mon[33264]: purged_snaps scrub ok 2024-09-20T22:48:31.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:31 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/3050005222' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:48:31.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:31 smithi154 ceph-mon[33264]: osdmap e431: 8 total, 8 up, 8 in 2024-09-20T22:48:31.841 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:48:31.862 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:31.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:31.863 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:48:31.868 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd unset noscrub 2024-09-20T22:48:31.964 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:48:31.986 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:31.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:31.987 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:48:32.088 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:48:32.110 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:32.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:32.111 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:48:32.148 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:48:32.212 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:48:32.223 INFO:tasks.rados.rados.0.smithi084.stdout:1666: done (8 left) 2024-09-20T22:48:32.223 INFO:tasks.rados.rados.0.smithi084.stdout:1667: done (7 left) 2024-09-20T22:48:32.223 INFO:tasks.rados.rados.0.smithi084.stdout:1668: done (6 left) 2024-09-20T22:48:32.223 INFO:tasks.rados.rados.0.smithi084.stdout:1669: done (5 left) 2024-09-20T22:48:32.223 INFO:tasks.rados.rados.0.smithi084.stdout:1671: done (4 left) 2024-09-20T22:48:32.223 INFO:tasks.rados.rados.0.smithi084.stdout:1673: done (3 left) 2024-09-20T22:48:32.223 INFO:tasks.rados.rados.0.smithi084.stdout:1674: done (2 left) 2024-09-20T22:48:32.223 INFO:tasks.rados.rados.0.smithi084.stdout:1675: done (1 left) 2024-09-20T22:48:32.223 INFO:tasks.rados.rados.0.smithi084.stdout:1677: done (0 left) 2024-09-20T22:48:32.224 INFO:tasks.rados.rados.0.smithi084.stdout:1678: rollback oid 38 current snap is 175 2024-09-20T22:48:32.224 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 38 to 167 2024-09-20T22:48:32.224 INFO:tasks.rados.rados.0.smithi084.stdout:1679: snap_create 2024-09-20T22:48:32.236 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:32.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:32.237 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:48:32.239 INFO:tasks.rados.rados.0.smithi084.stdout:1678: finishing rollback tid 0 to smithi08477203-38 2024-09-20T22:48:32.239 INFO:tasks.rados.rados.0.smithi084.stdout:1678: finishing rollback tid 1 to smithi08477203-38 2024-09-20T22:48:32.239 INFO:tasks.rados.rados.0.smithi084.stdout:1678: finishing rollback tid 2 to smithi08477203-38 2024-09-20T22:48:32.240 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 38 v 442 (ObjNum 489 snap 137 seq_num 489) dirty exists 2024-09-20T22:48:32.338 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:48:32.360 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:32.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:32.361 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:48:32.395 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-20T22:48:32.395 INFO:teuthology.orchestra.run.smithi154.stdout: "size": 20, 2024-09-20T22:48:32.395 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 600, 2024-09-20T22:48:32.395 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [ 2024-09-20T22:48:32.395 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.395 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:786 7.17 7:ed54065f:::smithi08477203-12:head [copy-from ver 61] snapc 19=[17] ondisk+write+known_if_redirected+supports_pool_eio e145)", 2024-09-20T22:48:32.395 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:44.333196+0000", 2024-09-20T22:48:32.395 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 288.05869032800001, 2024-09-20T22:48:32.395 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.023328503, 2024-09-20T22:48:32.396 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:48:32.396 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:48:32.396 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:48:32.396 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:48:32.396 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:48:32.396 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:48:32.396 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 786 2024-09-20T22:48:32.396 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.396 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:48:32.396 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.396 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:48:32.396 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.333196+0000", 2024-09-20T22:48:32.396 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:48:32.396 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.396 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.397 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:48:32.397 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.333196+0000", 2024-09-20T22:48:32.397 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:48:32.397 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.397 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.397 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:48:32.397 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.333197+0000", 2024-09-20T22:48:32.397 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.486e-06 2024-09-20T22:48:32.397 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.397 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.397 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:48:32.397 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.333207+0000", 2024-09-20T22:48:32.397 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0307e-05 2024-09-20T22:48:32.397 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.398 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.398 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:48:32.398 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.333220+0000", 2024-09-20T22:48:32.398 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.306e-05 2024-09-20T22:48:32.398 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.398 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.398 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:48:32.398 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.333229+0000", 2024-09-20T22:48:32.398 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.3450000000000006e-06 2024-09-20T22:48:32.398 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.398 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.398 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:48:32.398 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.333292+0000", 2024-09-20T22:48:32.398 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.3137000000000002e-05 2024-09-20T22:48:32.399 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.399 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.399 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op must wait for map", 2024-09-20T22:48:32.399 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.333298+0000", 2024-09-20T22:48:32.399 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.7780000000000004e-06 2024-09-20T22:48:32.399 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.399 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.399 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:48:32.399 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.336639+0000", 2024-09-20T22:48:32.399 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0033414479999999999 2024-09-20T22:48:32.399 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.399 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.399 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:48:32.399 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.336729+0000", 2024-09-20T22:48:32.400 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.9888000000000004e-05 2024-09-20T22:48:32.400 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.400 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.400 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 2,5", 2024-09-20T22:48:32.400 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.344876+0000", 2024-09-20T22:48:32.400 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0081473989999999996 2024-09-20T22:48:32.400 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.400 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.400 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:48:32.400 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.355138+0000", 2024-09-20T22:48:32.400 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.010261462000000001 2024-09-20T22:48:32.400 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.400 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.400 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:48:32.401 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.355972+0000", 2024-09-20T22:48:32.401 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00083377899999999999 2024-09-20T22:48:32.401 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.401 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.401 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:48:32.401 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.356451+0000", 2024-09-20T22:48:32.401 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00047985200000000001 2024-09-20T22:48:32.401 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.401 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.401 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:48:32.401 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.356486+0000", 2024-09-20T22:48:32.401 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.4653000000000002e-05 2024-09-20T22:48:32.401 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.401 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.401 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:48:32.402 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.356524+0000", 2024-09-20T22:48:32.402 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.7908999999999999e-05 2024-09-20T22:48:32.402 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:48:32.402 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:48:32.402 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:48:32.402 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.402 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.402 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:787 7.17 7:ed54065f:::smithi08477203-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e145)", 2024-09-20T22:48:32.402 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:44.333248+0000", 2024-09-20T22:48:32.402 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 288.05863823300001, 2024-09-20T22:48:32.402 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.016881184, 2024-09-20T22:48:32.402 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:48:32.402 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:48:32.403 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:48:32.403 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:48:32.403 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:48:32.403 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:48:32.403 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 787 2024-09-20T22:48:32.403 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.403 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:48:32.403 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.403 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:48:32.403 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.333248+0000", 2024-09-20T22:48:32.403 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:48:32.403 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.403 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.403 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:48:32.404 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.333248+0000", 2024-09-20T22:48:32.404 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:48:32.404 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.404 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.404 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:48:32.404 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.333248+0000", 2024-09-20T22:48:32.404 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.1399999999999997e-07 2024-09-20T22:48:32.404 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.404 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.404 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:48:32.404 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.333250+0000", 2024-09-20T22:48:32.404 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.8199999999999999e-06 2024-09-20T22:48:32.404 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.404 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.404 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:48:32.405 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.333257+0000", 2024-09-20T22:48:32.405 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.43e-06 2024-09-20T22:48:32.405 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.405 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.405 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:48:32.405 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.333261+0000", 2024-09-20T22:48:32.405 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.3670000000000001e-06 2024-09-20T22:48:32.405 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.405 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.405 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:48:32.405 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.333351+0000", 2024-09-20T22:48:32.405 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.0486000000000001e-05 2024-09-20T22:48:32.405 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.405 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.405 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting_for_map not empty", 2024-09-20T22:48:32.406 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.333357+0000", 2024-09-20T22:48:32.406 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.3520000000000003e-06 2024-09-20T22:48:32.406 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.406 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.406 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:48:32.406 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.336912+0000", 2024-09-20T22:48:32.406 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.003554612 2024-09-20T22:48:32.406 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.406 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.406 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for blocked object", 2024-09-20T22:48:32.406 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.336957+0000", 2024-09-20T22:48:32.406 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.4919000000000001e-05 2024-09-20T22:48:32.406 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.406 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.406 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:48:32.407 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.349989+0000", 2024-09-20T22:48:32.407 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.01303238 2024-09-20T22:48:32.407 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.407 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.407 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:48:32.407 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.350058+0000", 2024-09-20T22:48:32.407 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.8418999999999997e-05 2024-09-20T22:48:32.407 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.407 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.407 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:48:32.407 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.350129+0000", 2024-09-20T22:48:32.407 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.0984999999999999e-05 2024-09-20T22:48:32.407 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:48:32.407 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:48:32.408 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:48:32.408 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.408 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.408 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:804 7.17 7:ea720b85:::smithi08477203-26:head [copy-from ver 61] snapc 19=[17] ondisk+write+known_if_redirected+supports_pool_eio e145)", 2024-09-20T22:48:32.408 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:44.451241+0000", 2024-09-20T22:48:32.408 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 287.94064439800002, 2024-09-20T22:48:32.408 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.012865619999999999, 2024-09-20T22:48:32.408 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:48:32.408 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:48:32.408 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:48:32.408 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:48:32.408 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:48:32.408 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:48:32.408 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 804 2024-09-20T22:48:32.408 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.409 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:48:32.409 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.409 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:48:32.409 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.451241+0000", 2024-09-20T22:48:32.409 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:48:32.409 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.409 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.409 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:48:32.409 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.451241+0000", 2024-09-20T22:48:32.409 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:48:32.409 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.409 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.409 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:48:32.409 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.451244+0000", 2024-09-20T22:48:32.409 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.2689999999999998e-06 2024-09-20T22:48:32.410 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.410 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.410 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:48:32.410 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.451255+0000", 2024-09-20T22:48:32.410 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1151e-05 2024-09-20T22:48:32.410 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.410 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.410 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:48:32.410 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.451274+0000", 2024-09-20T22:48:32.410 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9389999999999999e-05 2024-09-20T22:48:32.410 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.410 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.410 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:48:32.410 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.451286+0000", 2024-09-20T22:48:32.411 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1703e-05 2024-09-20T22:48:32.411 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.411 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.411 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:48:32.411 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.451361+0000", 2024-09-20T22:48:32.411 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.5310999999999998e-05 2024-09-20T22:48:32.411 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.411 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.411 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:48:32.411 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.451462+0000", 2024-09-20T22:48:32.411 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000100809 2024-09-20T22:48:32.411 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.411 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.411 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 2,5", 2024-09-20T22:48:32.412 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.456603+0000", 2024-09-20T22:48:32.412 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0051411169999999997 2024-09-20T22:48:32.412 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.412 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.412 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:48:32.412 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.463017+0000", 2024-09-20T22:48:32.412 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.006413307 2024-09-20T22:48:32.412 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.412 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.412 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:48:32.412 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.463754+0000", 2024-09-20T22:48:32.412 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00073758000000000005 2024-09-20T22:48:32.412 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.412 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.412 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:48:32.413 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.464036+0000", 2024-09-20T22:48:32.413 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00028171000000000001 2024-09-20T22:48:32.413 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.413 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.413 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:48:32.413 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.464076+0000", 2024-09-20T22:48:32.413 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.9736e-05 2024-09-20T22:48:32.413 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.413 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.413 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:48:32.413 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.464107+0000", 2024-09-20T22:48:32.413 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.1537e-05 2024-09-20T22:48:32.413 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:48:32.413 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:48:32.413 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:48:32.414 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.414 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.414 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:1527 7.17 7:ea720b85:::smithi08477203-26:head [copy-from ver 83] snapc 33=[33] ondisk+write+known_if_redirected+supports_pool_eio e197)", 2024-09-20T22:48:32.414 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:44:36.198921+0000", 2024-09-20T22:48:32.414 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 236.192965074, 2024-09-20T22:48:32.414 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.021621703999999999, 2024-09-20T22:48:32.414 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:48:32.414 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:48:32.414 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:48:32.414 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:48:32.414 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:48:32.414 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:48:32.414 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 1527 2024-09-20T22:48:32.414 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.415 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:48:32.415 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.415 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:48:32.415 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.198921+0000", 2024-09-20T22:48:32.415 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:48:32.415 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.415 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.415 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:48:32.415 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.198921+0000", 2024-09-20T22:48:32.415 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:48:32.415 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.416 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.416 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:48:32.416 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.198923+0000", 2024-09-20T22:48:32.416 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.0080000000000001e-06 2024-09-20T22:48:32.416 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.416 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.416 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:48:32.416 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.198934+0000", 2024-09-20T22:48:32.416 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0923e-05 2024-09-20T22:48:32.416 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.416 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.416 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:48:32.416 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.198955+0000", 2024-09-20T22:48:32.416 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.0806e-05 2024-09-20T22:48:32.417 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.417 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.417 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:48:32.417 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.198966+0000", 2024-09-20T22:48:32.417 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1779e-05 2024-09-20T22:48:32.417 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.417 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.417 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:48:32.417 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.199051+0000", 2024-09-20T22:48:32.417 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.4357999999999994e-05 2024-09-20T22:48:32.417 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.417 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.417 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op must wait for map", 2024-09-20T22:48:32.417 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.199058+0000", 2024-09-20T22:48:32.417 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.8530000000000003e-06 2024-09-20T22:48:32.417 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.418 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.418 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:48:32.418 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.202420+0000", 2024-09-20T22:48:32.418 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0033623350000000002 2024-09-20T22:48:32.418 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.418 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.418 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:48:32.418 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.202498+0000", 2024-09-20T22:48:32.418 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.8316e-05 2024-09-20T22:48:32.418 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.418 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.418 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 2,5", 2024-09-20T22:48:32.418 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.209140+0000", 2024-09-20T22:48:32.418 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0066415759999999997 2024-09-20T22:48:32.419 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.419 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.419 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:48:32.419 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.214931+0000", 2024-09-20T22:48:32.419 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0057916520000000004 2024-09-20T22:48:32.419 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.419 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.419 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:48:32.419 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.220009+0000", 2024-09-20T22:48:32.419 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0050773750000000003 2024-09-20T22:48:32.419 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.419 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.419 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:48:32.419 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.220417+0000", 2024-09-20T22:48:32.419 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00040778599999999999 2024-09-20T22:48:32.419 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.420 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.420 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:48:32.420 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.220479+0000", 2024-09-20T22:48:32.420 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.1977000000000006e-05 2024-09-20T22:48:32.420 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.420 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.420 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:48:32.420 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.220542+0000", 2024-09-20T22:48:32.420 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.3960000000000004e-05 2024-09-20T22:48:32.420 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:48:32.420 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:48:32.420 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:48:32.420 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.420 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.421 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:1528 7.17 7:ea720b85:::smithi08477203-26:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e197)", 2024-09-20T22:48:32.421 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:44:36.198994+0000", 2024-09-20T22:48:32.421 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 236.192892347, 2024-09-20T22:48:32.421 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.015013924, 2024-09-20T22:48:32.421 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:48:32.421 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:48:32.421 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:48:32.421 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:48:32.421 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:48:32.421 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:48:32.421 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 1528 2024-09-20T22:48:32.421 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.421 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:48:32.421 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.421 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:48:32.421 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.198994+0000", 2024-09-20T22:48:32.422 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:48:32.422 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.422 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.423 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:48:32.423 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.198994+0000", 2024-09-20T22:48:32.423 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:48:32.423 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.423 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.423 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:48:32.423 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.198994+0000", 2024-09-20T22:48:32.423 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.8100000000000003e-07 2024-09-20T22:48:32.423 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.423 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.423 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:48:32.423 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.198996+0000", 2024-09-20T22:48:32.423 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.3e-06 2024-09-20T22:48:32.424 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.424 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.424 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:48:32.424 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.199008+0000", 2024-09-20T22:48:32.424 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1603999999999999e-05 2024-09-20T22:48:32.424 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.424 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.424 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:48:32.424 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.199013+0000", 2024-09-20T22:48:32.424 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.4500000000000003e-06 2024-09-20T22:48:32.424 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.424 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.424 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:48:32.424 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.199120+0000", 2024-09-20T22:48:32.424 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000106772 2024-09-20T22:48:32.425 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.425 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.425 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting_for_map not empty", 2024-09-20T22:48:32.425 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.199126+0000", 2024-09-20T22:48:32.425 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.8320000000000002e-06 2024-09-20T22:48:32.425 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.425 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.425 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:48:32.425 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.202625+0000", 2024-09-20T22:48:32.425 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0034986309999999999 2024-09-20T22:48:32.425 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.425 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.425 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for blocked object", 2024-09-20T22:48:32.425 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.202667+0000", 2024-09-20T22:48:32.425 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.2759000000000002e-05 2024-09-20T22:48:32.426 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.426 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.426 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:48:32.426 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.213787+0000", 2024-09-20T22:48:32.426 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.011119377 2024-09-20T22:48:32.426 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.426 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.426 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:48:32.426 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.213902+0000", 2024-09-20T22:48:32.426 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000115369 2024-09-20T22:48:32.426 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.426 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.426 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:48:32.426 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.214007+0000", 2024-09-20T22:48:32.427 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000105349 2024-09-20T22:48:32.427 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:48:32.427 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:48:32.427 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:48:32.427 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.427 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.427 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:1550 7.17 7:ed54065f:::smithi08477203-12:head [copy-from ver 88] snapc 34=[34,33] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2024-09-20T22:48:32.427 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:44:37.238376+0000", 2024-09-20T22:48:32.427 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 235.15350943499999, 2024-09-20T22:48:32.427 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.016634917999999999, 2024-09-20T22:48:32.427 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:48:32.427 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:48:32.427 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:48:32.427 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:48:32.427 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:48:32.428 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:48:32.428 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 1550 2024-09-20T22:48:32.428 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.428 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:48:32.428 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.428 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:48:32.428 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:37.238376+0000", 2024-09-20T22:48:32.428 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:48:32.428 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.428 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.428 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:48:32.428 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:37.238376+0000", 2024-09-20T22:48:32.428 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:48:32.428 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.428 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.429 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:48:32.429 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:37.238379+0000", 2024-09-20T22:48:32.429 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.7989999999999998e-06 2024-09-20T22:48:32.429 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.429 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.429 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:48:32.429 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:37.238408+0000", 2024-09-20T22:48:32.429 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.8980999999999999e-05 2024-09-20T22:48:32.429 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.429 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.429 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:48:32.429 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:37.238433+0000", 2024-09-20T22:48:32.429 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.5185999999999999e-05 2024-09-20T22:48:32.429 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.429 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.430 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:48:32.430 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:37.238446+0000", 2024-09-20T22:48:32.430 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2883e-05 2024-09-20T22:48:32.430 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.430 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.430 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:48:32.430 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:37.238523+0000", 2024-09-20T22:48:32.430 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.6396999999999994e-05 2024-09-20T22:48:32.430 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.430 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.430 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:48:32.430 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:37.238618+0000", 2024-09-20T22:48:32.430 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.5049000000000004e-05 2024-09-20T22:48:32.431 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.431 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.431 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 2,5", 2024-09-20T22:48:32.431 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:37.246556+0000", 2024-09-20T22:48:32.431 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0079378090000000005 2024-09-20T22:48:32.431 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.431 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.431 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:48:32.431 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:37.254025+0000", 2024-09-20T22:48:32.431 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0074697139999999997 2024-09-20T22:48:32.431 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.431 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.431 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:48:32.431 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:37.254033+0000", 2024-09-20T22:48:32.431 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.6979999999999998e-06 2024-09-20T22:48:32.432 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.432 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.432 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:48:32.432 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:37.254919+0000", 2024-09-20T22:48:32.432 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00088564700000000004 2024-09-20T22:48:32.432 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.432 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.432 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:48:32.432 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:37.254958+0000", 2024-09-20T22:48:32.432 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.9515999999999999e-05 2024-09-20T22:48:32.432 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.432 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.432 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:48:32.432 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:37.255011+0000", 2024-09-20T22:48:32.432 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.3239000000000003e-05 2024-09-20T22:48:32.433 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:48:32.433 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:48:32.433 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:48:32.433 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.433 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.433 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:1799 7.17 7:ea720b85:::smithi08477203-26:head [write 521083~714336,stat] snapc 46=[45,44,43,42,41,40,3e] ondisk+write+known_if_redirected+supports_pool_eio e229)", 2024-09-20T22:48:32.433 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:45:08.170663+0000", 2024-09-20T22:48:32.433 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 204.22122279000001, 2024-09-20T22:48:32.433 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.016368226999999999, 2024-09-20T22:48:32.433 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:48:32.433 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:48:32.433 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:48:32.433 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:48:32.433 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:48:32.433 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:48:32.434 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 1799 2024-09-20T22:48:32.434 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.434 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:48:32.434 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.434 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:48:32.434 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.170663+0000", 2024-09-20T22:48:32.434 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:48:32.434 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.434 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.434 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:48:32.434 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.170663+0000", 2024-09-20T22:48:32.434 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:48:32.434 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.434 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.435 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:48:32.435 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.170664+0000", 2024-09-20T22:48:32.435 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.085e-06 2024-09-20T22:48:32.435 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.435 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.435 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:48:32.435 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.171481+0000", 2024-09-20T22:48:32.435 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00081653900000000002 2024-09-20T22:48:32.435 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.435 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.435 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:48:32.435 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.171492+0000", 2024-09-20T22:48:32.435 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1565e-05 2024-09-20T22:48:32.435 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.436 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.436 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:48:32.436 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.171498+0000", 2024-09-20T22:48:32.436 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.925e-06 2024-09-20T22:48:32.436 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.436 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.436 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:48:32.436 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.171547+0000", 2024-09-20T22:48:32.436 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.8916999999999999e-05 2024-09-20T22:48:32.436 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.436 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.436 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:48:32.436 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.171611+0000", 2024-09-20T22:48:32.436 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.3838999999999996e-05 2024-09-20T22:48:32.437 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.437 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.437 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 2,5", 2024-09-20T22:48:32.437 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.171740+0000", 2024-09-20T22:48:32.437 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00012881500000000001 2024-09-20T22:48:32.437 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.437 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.437 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:48:32.437 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.173862+0000", 2024-09-20T22:48:32.437 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0021225649999999999 2024-09-20T22:48:32.437 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.437 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.438 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:48:32.438 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.179252+0000", 2024-09-20T22:48:32.438 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0053895490000000004 2024-09-20T22:48:32.438 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.438 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.438 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:48:32.438 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.186956+0000", 2024-09-20T22:48:32.438 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0077038050000000002 2024-09-20T22:48:32.438 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.438 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.438 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:48:32.438 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.186993+0000", 2024-09-20T22:48:32.438 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.7141000000000002e-05 2024-09-20T22:48:32.439 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.439 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.439 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:48:32.439 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.187031+0000", 2024-09-20T22:48:32.439 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.8482000000000002e-05 2024-09-20T22:48:32.439 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:48:32.439 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:48:32.439 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:48:32.439 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.439 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.439 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:1800 7.17 7:ea720b85:::smithi08477203-26:head [write 1724715~555421] snapc 46=[45,44,43,42,41,40,3e] ondisk+write+known_if_redirected+supports_pool_eio e229)", 2024-09-20T22:48:32.439 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:45:08.177237+0000", 2024-09-20T22:48:32.439 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 204.214649165, 2024-09-20T22:48:32.439 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.012771694, 2024-09-20T22:48:32.440 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:48:32.440 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:48:32.440 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:48:32.440 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:48:32.440 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:48:32.440 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:48:32.440 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 1800 2024-09-20T22:48:32.440 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.440 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:48:32.440 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.440 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:48:32.440 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.177237+0000", 2024-09-20T22:48:32.440 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:48:32.440 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.440 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.441 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:48:32.441 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.177237+0000", 2024-09-20T22:48:32.441 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:48:32.441 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.441 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.441 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:48:32.441 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.177238+0000", 2024-09-20T22:48:32.441 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.6080000000000001e-06 2024-09-20T22:48:32.441 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.441 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.441 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:48:32.441 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.177844+0000", 2024-09-20T22:48:32.441 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00060579999999999998 2024-09-20T22:48:32.441 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.442 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.442 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:48:32.442 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.177857+0000", 2024-09-20T22:48:32.442 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2529e-05 2024-09-20T22:48:32.442 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.442 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.442 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:48:32.442 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.177864+0000", 2024-09-20T22:48:32.442 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.2470000000000002e-06 2024-09-20T22:48:32.442 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.442 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.442 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:48:32.442 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.177908+0000", 2024-09-20T22:48:32.442 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.4329999999999997e-05 2024-09-20T22:48:32.443 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.443 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.443 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:48:32.443 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.177977+0000", 2024-09-20T22:48:32.443 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.8598999999999998e-05 2024-09-20T22:48:32.443 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.443 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.443 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 2,5", 2024-09-20T22:48:32.443 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.178079+0000", 2024-09-20T22:48:32.443 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000102074 2024-09-20T22:48:32.443 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.443 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.443 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:48:32.443 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.181043+0000", 2024-09-20T22:48:32.444 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0029639810000000001 2024-09-20T22:48:32.444 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.444 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.444 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:48:32.444 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.181481+0000", 2024-09-20T22:48:32.444 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00043855899999999999 2024-09-20T22:48:32.444 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.444 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.444 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:48:32.444 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.189928+0000", 2024-09-20T22:48:32.444 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0084463149999999994 2024-09-20T22:48:32.444 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.444 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.444 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:48:32.444 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.189968+0000", 2024-09-20T22:48:32.445 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.0268999999999997e-05 2024-09-20T22:48:32.445 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.445 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.445 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:48:32.445 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.190008+0000", 2024-09-20T22:48:32.445 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.0383000000000003e-05 2024-09-20T22:48:32.445 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:48:32.445 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:48:32.445 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:48:32.445 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.445 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.445 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:1802 7.17 7:ea720b85:::smithi08477203-26:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e229)", 2024-09-20T22:48:32.445 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:45:08.177901+0000", 2024-09-20T22:48:32.445 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 204.213985221, 2024-09-20T22:48:32.445 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.012470591, 2024-09-20T22:48:32.446 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:48:32.446 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:48:32.446 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:48:32.446 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:48:32.446 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:48:32.446 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:48:32.446 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 1802 2024-09-20T22:48:32.446 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.446 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:48:32.446 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.446 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:48:32.446 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.177901+0000", 2024-09-20T22:48:32.446 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:48:32.446 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.447 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.447 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:48:32.447 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.177901+0000", 2024-09-20T22:48:32.447 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:48:32.447 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.447 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.447 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:48:32.447 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.177901+0000", 2024-09-20T22:48:32.447 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.2899999999999999e-07 2024-09-20T22:48:32.447 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.447 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.447 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:48:32.447 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.177902+0000", 2024-09-20T22:48:32.447 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.452e-06 2024-09-20T22:48:32.447 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.448 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.448 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:48:32.448 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.177906+0000", 2024-09-20T22:48:32.448 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.788e-06 2024-09-20T22:48:32.448 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.448 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.448 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:48:32.448 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.177909+0000", 2024-09-20T22:48:32.448 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.9840000000000001e-06 2024-09-20T22:48:32.448 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.448 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.448 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:48:32.448 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.179283+0000", 2024-09-20T22:48:32.448 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001373733 2024-09-20T22:48:32.449 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.449 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.449 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for rw locks", 2024-09-20T22:48:32.449 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.179364+0000", 2024-09-20T22:48:32.449 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.1209000000000001e-05 2024-09-20T22:48:32.449 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.449 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.449 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:48:32.449 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.190140+0000", 2024-09-20T22:48:32.449 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.010775527 2024-09-20T22:48:32.449 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.449 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.449 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:48:32.449 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.190186+0000", 2024-09-20T22:48:32.450 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.5853000000000003e-05 2024-09-20T22:48:32.450 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.450 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.450 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:48:32.450 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.190371+0000", 2024-09-20T22:48:32.450 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000185716 2024-09-20T22:48:32.450 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:48:32.450 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:48:32.450 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:48:32.450 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.450 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.450 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:2318 7.5 7:a160b5d4:::smithi08477203-16:head [copy-from ver 168] snapc 55=[55,53,52,51,50,4b,47,3e] ondisk+write+known_if_redirected+supports_pool_eio e258)", 2024-09-20T22:48:32.450 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:45:37.175863+0000", 2024-09-20T22:48:32.450 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 175.21602271500001, 2024-09-20T22:48:32.451 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.030719883999999999, 2024-09-20T22:48:32.451 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:48:32.451 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:48:32.451 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:48:32.451 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:48:32.451 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:48:32.451 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:48:32.451 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 2318 2024-09-20T22:48:32.451 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.451 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:48:32.451 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.451 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:48:32.451 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.175863+0000", 2024-09-20T22:48:32.451 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:48:32.451 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.452 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.452 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:48:32.452 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.175863+0000", 2024-09-20T22:48:32.452 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:48:32.452 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.452 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.452 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:48:32.452 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.175865+0000", 2024-09-20T22:48:32.452 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.488e-06 2024-09-20T22:48:32.452 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.452 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.452 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:48:32.452 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.175871+0000", 2024-09-20T22:48:32.452 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.438e-06 2024-09-20T22:48:32.453 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.453 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.453 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:48:32.453 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.175881+0000", 2024-09-20T22:48:32.453 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.6430000000000007e-06 2024-09-20T22:48:32.453 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.453 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.453 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:48:32.453 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.175886+0000", 2024-09-20T22:48:32.453 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.3059999999999997e-06 2024-09-20T22:48:32.453 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.453 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.453 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:48:32.453 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.175944+0000", 2024-09-20T22:48:32.454 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.7982999999999997e-05 2024-09-20T22:48:32.454 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.454 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.454 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op must wait for map", 2024-09-20T22:48:32.454 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.175951+0000", 2024-09-20T22:48:32.454 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.9280000000000004e-06 2024-09-20T22:48:32.454 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.454 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.454 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:48:32.454 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.179794+0000", 2024-09-20T22:48:32.454 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0038435209999999999 2024-09-20T22:48:32.454 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.454 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.454 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:48:32.454 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.179940+0000", 2024-09-20T22:48:32.454 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00014550000000000001 2024-09-20T22:48:32.455 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.455 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.455 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 0,2", 2024-09-20T22:48:32.455 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.190992+0000", 2024-09-20T22:48:32.455 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.011051587999999999 2024-09-20T22:48:32.455 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.455 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.455 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:48:32.455 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.195070+0000", 2024-09-20T22:48:32.455 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0040784929999999999 2024-09-20T22:48:32.455 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.455 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.455 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:48:32.455 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.202908+0000", 2024-09-20T22:48:32.456 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0078376690000000002 2024-09-20T22:48:32.456 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.456 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.456 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:48:32.456 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.206517+0000", 2024-09-20T22:48:32.456 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0036088880000000002 2024-09-20T22:48:32.456 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.456 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.456 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:48:32.456 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.206553+0000", 2024-09-20T22:48:32.456 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.6363999999999999e-05 2024-09-20T22:48:32.456 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.456 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.456 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:48:32.457 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.206583+0000", 2024-09-20T22:48:32.457 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.0074999999999999e-05 2024-09-20T22:48:32.457 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:48:32.457 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:48:32.457 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:48:32.457 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.457 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.457 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:2319 7.5 7:a160b5d4:::smithi08477203-16:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e258)", 2024-09-20T22:48:32.457 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:45:37.175983+0000", 2024-09-20T22:48:32.457 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 175.21590295999999, 2024-09-20T22:48:32.457 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.018323547999999999, 2024-09-20T22:48:32.457 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:48:32.457 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:48:32.457 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:48:32.457 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:48:32.458 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:48:32.458 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:48:32.458 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 2319 2024-09-20T22:48:32.458 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.458 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:48:32.458 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.458 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:48:32.458 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.175983+0000", 2024-09-20T22:48:32.458 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:48:32.458 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.458 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.458 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:48:32.458 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.175983+0000", 2024-09-20T22:48:32.458 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:48:32.459 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.459 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.459 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:48:32.459 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.175984+0000", 2024-09-20T22:48:32.459 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.1099999999999995e-07 2024-09-20T22:48:32.459 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.459 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.459 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:48:32.459 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.175996+0000", 2024-09-20T22:48:32.459 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2121e-05 2024-09-20T22:48:32.459 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.459 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.459 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:48:32.459 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.176019+0000", 2024-09-20T22:48:32.459 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.3465999999999999e-05 2024-09-20T22:48:32.460 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.460 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.460 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:48:32.460 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.176026+0000", 2024-09-20T22:48:32.460 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.8190000000000003e-06 2024-09-20T22:48:32.460 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.460 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.460 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:48:32.460 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.176110+0000", 2024-09-20T22:48:32.460 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.3967000000000004e-05 2024-09-20T22:48:32.460 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.460 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.460 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting_for_map not empty", 2024-09-20T22:48:32.460 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.176116+0000", 2024-09-20T22:48:32.460 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.0249999999999999e-06 2024-09-20T22:48:32.461 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.461 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.461 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:48:32.461 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.180106+0000", 2024-09-20T22:48:32.461 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0039898980000000004 2024-09-20T22:48:32.461 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.461 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.461 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for blocked object", 2024-09-20T22:48:32.461 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.180135+0000", 2024-09-20T22:48:32.461 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.9595999999999999e-05 2024-09-20T22:48:32.461 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.461 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.461 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:48:32.461 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.194106+0000", 2024-09-20T22:48:32.461 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.013970761999999999 2024-09-20T22:48:32.462 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.462 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.462 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:48:32.462 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.194203+0000", 2024-09-20T22:48:32.462 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.7211000000000002e-05 2024-09-20T22:48:32.462 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.462 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.462 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:48:32.462 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.194306+0000", 2024-09-20T22:48:32.462 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00010307200000000001 2024-09-20T22:48:32.462 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:48:32.462 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:48:32.462 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:48:32.462 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.462 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.463 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:32 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd rm-pg-upmap", "pgid": "7.17"}]: dispatch 2024-09-20T22:48:32.463 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:32 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "7.17"}]: dispatch 2024-09-20T22:48:32.463 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:32 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd rm-pg-upmap", "pgid": "7.17"}]: dispatch 2024-09-20T22:48:32.463 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:32 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "7.17"}]: dispatch 2024-09-20T22:48:32.463 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:32 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd rm-pg-upmap", "pgid": "7.17"}]': finished 2024-09-20T22:48:32.463 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:32 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd rm-pg-upmap-items", "pgid": "7.17"}]': finished 2024-09-20T22:48:32.463 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:32 smithi154 ceph-mon[33264]: osdmap e432: 8 total, 8 up, 8 in 2024-09-20T22:48:32.464 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24415.0:230 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch (1) in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e259)", 2024-09-20T22:48:32.464 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:45:39.207410+0000", 2024-09-20T22:48:32.464 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 173.18447588800001, 2024-09-20T22:48:32.464 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0072298289999999, 2024-09-20T22:48:32.464 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:48:32.464 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:48:32.464 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:48:32.464 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:48:32.465 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24415", 2024-09-20T22:48:32.465 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/2056244102", 2024-09-20T22:48:32.465 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 230 2024-09-20T22:48:32.465 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.465 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:48:32.465 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.465 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:48:32.465 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.207410+0000", 2024-09-20T22:48:32.465 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:48:32.465 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.465 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.465 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:48:32.465 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.207410+0000", 2024-09-20T22:48:32.465 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:48:32.466 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.466 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.466 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:48:32.466 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.207412+0000", 2024-09-20T22:48:32.466 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.7519999999999999e-06 2024-09-20T22:48:32.466 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.466 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.466 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:48:32.466 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.207419+0000", 2024-09-20T22:48:32.466 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.097e-06 2024-09-20T22:48:32.466 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.466 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.466 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:48:32.466 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.207432+0000", 2024-09-20T22:48:32.466 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2719e-05 2024-09-20T22:48:32.467 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.467 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.467 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:48:32.467 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.207464+0000", 2024-09-20T22:48:32.467 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.2820000000000001e-05 2024-09-20T22:48:32.467 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.467 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.467 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:48:32.467 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.207670+0000", 2024-09-20T22:48:32.467 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000205486 2024-09-20T22:48:32.467 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.467 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.467 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-20T22:48:32.467 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.207687+0000", 2024-09-20T22:48:32.468 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.7107000000000001e-05 2024-09-20T22:48:32.468 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.468 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.468 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:48:32.468 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:40.212672+0000", 2024-09-20T22:48:32.468 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.004984562 2024-09-20T22:48:32.468 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.468 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.468 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:48:32.468 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:40.213576+0000", 2024-09-20T22:48:32.468 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00090404799999999996 2024-09-20T22:48:32.468 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.468 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.468 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:48:32.468 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:40.214640+0000", 2024-09-20T22:48:32.469 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0010642379999999999 2024-09-20T22:48:32.469 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:48:32.469 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:48:32.469 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:48:32.469 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.469 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.469 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:2367 7.5 7:a160b5d4:::smithi08477203-16:head [stat,delete] snapc 56=[56,55,53,52,51,50,47,3e] ondisk+write+known_if_redirected+supports_pool_eio e260)", 2024-09-20T22:48:32.469 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:45:39.601299+0000", 2024-09-20T22:48:32.469 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 172.790586467, 2024-09-20T22:48:32.469 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.64714523300000004, 2024-09-20T22:48:32.469 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:48:32.469 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:48:32.469 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:48:32.469 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:48:32.469 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:48:32.470 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:48:32.470 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 2367 2024-09-20T22:48:32.470 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.470 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:48:32.470 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.470 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:48:32.470 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.601299+0000", 2024-09-20T22:48:32.470 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:48:32.470 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.470 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.470 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:48:32.470 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.601299+0000", 2024-09-20T22:48:32.470 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:48:32.470 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.470 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.471 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:48:32.471 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.601301+0000", 2024-09-20T22:48:32.471 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3850000000000001e-06 2024-09-20T22:48:32.471 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.471 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.471 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:48:32.471 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.601306+0000", 2024-09-20T22:48:32.471 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.2730000000000002e-06 2024-09-20T22:48:32.471 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.471 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.471 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:48:32.471 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.601318+0000", 2024-09-20T22:48:32.471 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2148e-05 2024-09-20T22:48:32.471 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.472 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.472 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:48:32.472 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.601324+0000", 2024-09-20T22:48:32.472 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.852e-06 2024-09-20T22:48:32.472 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.472 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.472 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:48:32.472 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.601383+0000", 2024-09-20T22:48:32.472 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.8570000000000003e-05 2024-09-20T22:48:32.472 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.472 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.472 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-20T22:48:32.472 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.601391+0000", 2024-09-20T22:48:32.472 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.5290000000000008e-06 2024-09-20T22:48:32.473 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.473 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.473 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:48:32.473 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:40.221227+0000", 2024-09-20T22:48:32.473 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.619835515 2024-09-20T22:48:32.473 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.473 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.473 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for degraded object", 2024-09-20T22:48:32.473 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:40.223919+0000", 2024-09-20T22:48:32.473 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.002692168 2024-09-20T22:48:32.473 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.473 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.473 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:48:32.473 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:40.244536+0000", 2024-09-20T22:48:32.473 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.020617613 2024-09-20T22:48:32.474 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.474 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.474 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:48:32.474 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:40.244653+0000", 2024-09-20T22:48:32.474 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000116796 2024-09-20T22:48:32.474 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.474 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.474 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 1,2", 2024-09-20T22:48:32.474 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:40.244832+0000", 2024-09-20T22:48:32.474 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00017833500000000001 2024-09-20T22:48:32.474 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.474 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.474 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:48:32.474 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:40.245494+0000", 2024-09-20T22:48:32.475 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00066241399999999995 2024-09-20T22:48:32.475 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.475 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.475 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:48:32.475 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:40.246501+0000", 2024-09-20T22:48:32.475 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0010066299999999999 2024-09-20T22:48:32.475 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.475 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.475 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:48:32.475 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:40.248326+0000", 2024-09-20T22:48:32.475 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001824895 2024-09-20T22:48:32.475 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.475 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.475 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:48:32.476 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:40.248370+0000", 2024-09-20T22:48:32.476 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.4298999999999997e-05 2024-09-20T22:48:32.476 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.476 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.476 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:48:32.476 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:40.248445+0000", 2024-09-20T22:48:32.476 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.4810999999999999e-05 2024-09-20T22:48:32.476 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:48:32.476 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:48:32.476 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:48:32.476 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.476 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.476 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:3132 7.19 e328/139 7:9eac4560:::smithi08477203-30:head v 328'198, mlcod=322'194)", 2024-09-20T22:48:32.476 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:46:48.223406+0000", 2024-09-20T22:48:32.476 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 104.168480031, 2024-09-20T22:48:32.477 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.016765688000000001, 2024-09-20T22:48:32.477 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:48:32.477 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:48:32.477 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:48:32.477 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:48:32.477 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.477 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:48:32.477 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.223406+0000", 2024-09-20T22:48:32.477 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:48:32.477 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.477 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.477 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:48:32.477 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.223406+0000", 2024-09-20T22:48:32.477 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:48:32.477 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.478 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.478 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:48:32.478 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.223406+0000", 2024-09-20T22:48:32.478 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.6699999999999999e-07 2024-09-20T22:48:32.478 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.478 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.478 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:48:32.478 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.226227+0000", 2024-09-20T22:48:32.478 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0028205690000000002 2024-09-20T22:48:32.478 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.478 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.478 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:48:32.478 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.226238+0000", 2024-09-20T22:48:32.478 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0879999999999999e-05 2024-09-20T22:48:32.479 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.479 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.479 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:48:32.479 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.226249+0000", 2024-09-20T22:48:32.479 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0912e-05 2024-09-20T22:48:32.479 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.479 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.479 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:48:32.479 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.227960+0000", 2024-09-20T22:48:32.479 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0017109619999999999 2024-09-20T22:48:32.479 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.479 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.479 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:48:32.479 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.227987+0000", 2024-09-20T22:48:32.479 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.7659999999999999e-05 2024-09-20T22:48:32.480 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.480 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.480 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:48:32.480 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.240119+0000", 2024-09-20T22:48:32.480 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.012131885 2024-09-20T22:48:32.480 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.480 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.480 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:48:32.480 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.240172+0000", 2024-09-20T22:48:32.480 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.2453000000000001e-05 2024-09-20T22:48:32.480 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:48:32.480 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:48:32.480 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:48:32.480 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.481 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.481 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:3232 7.16 7:6885a65f:::smithi08477203-14:head [copy-from ver 187] snapc 7e=[7e,7d,7c,7b,7a,79,78,75,74,73,70,6b] ondisk+write+known_if_redirected+supports_pool_eio e337)", 2024-09-20T22:48:32.481 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:46:57.213186+0000", 2024-09-20T22:48:32.481 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 95.178700359999993, 2024-09-20T22:48:32.481 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.028083105000000001, 2024-09-20T22:48:32.481 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:48:32.481 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:48:32.481 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:48:32.481 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:48:32.481 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:48:32.481 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:48:32.481 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 3232 2024-09-20T22:48:32.481 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.481 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:48:32.481 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.482 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:48:32.482 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.213186+0000", 2024-09-20T22:48:32.482 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:48:32.482 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.482 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.482 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:48:32.482 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.213186+0000", 2024-09-20T22:48:32.482 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:48:32.482 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.482 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.482 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:48:32.482 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.213188+0000", 2024-09-20T22:48:32.482 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.5129999999999999e-06 2024-09-20T22:48:32.482 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.483 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.483 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:48:32.483 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.213194+0000", 2024-09-20T22:48:32.483 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.4810000000000003e-06 2024-09-20T22:48:32.483 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.483 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.483 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:48:32.483 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.213215+0000", 2024-09-20T22:48:32.483 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.0455999999999999e-05 2024-09-20T22:48:32.483 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.483 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.483 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:48:32.483 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.213225+0000", 2024-09-20T22:48:32.483 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0212e-05 2024-09-20T22:48:32.483 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.484 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.484 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:48:32.484 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.213315+0000", 2024-09-20T22:48:32.484 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.9524000000000005e-05 2024-09-20T22:48:32.484 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.484 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.484 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op must wait for map", 2024-09-20T22:48:32.484 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.213324+0000", 2024-09-20T22:48:32.484 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.8890000000000008e-06 2024-09-20T22:48:32.484 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.484 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.484 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:48:32.484 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.220517+0000", 2024-09-20T22:48:32.484 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0071931219999999997 2024-09-20T22:48:32.485 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.485 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.485 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:48:32.485 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.220636+0000", 2024-09-20T22:48:32.485 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000118875 2024-09-20T22:48:32.485 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.485 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.485 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 2,5", 2024-09-20T22:48:32.485 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.230308+0000", 2024-09-20T22:48:32.485 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0096726299999999998 2024-09-20T22:48:32.485 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.485 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.485 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:48:32.485 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.238007+0000", 2024-09-20T22:48:32.485 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.007699287 2024-09-20T22:48:32.486 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.486 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.486 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:48:32.486 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.240863+0000", 2024-09-20T22:48:32.486 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0028554880000000002 2024-09-20T22:48:32.486 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.486 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.486 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:48:32.486 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.241194+0000", 2024-09-20T22:48:32.486 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00033087399999999999 2024-09-20T22:48:32.486 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.486 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.486 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:48:32.486 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.241233+0000", 2024-09-20T22:48:32.486 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.9338000000000002e-05 2024-09-20T22:48:32.486 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.487 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.487 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:48:32.487 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.241269+0000", 2024-09-20T22:48:32.487 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.5416000000000001e-05 2024-09-20T22:48:32.487 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:48:32.487 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:48:32.487 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:48:32.487 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.487 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.487 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:3233 7.16 7:6885a65f:::smithi08477203-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e337)", 2024-09-20T22:48:32.487 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:46:57.213253+0000", 2024-09-20T22:48:32.487 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 95.178632936, 2024-09-20T22:48:32.487 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.023594825, 2024-09-20T22:48:32.487 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:48:32.488 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:48:32.488 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:48:32.488 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:48:32.488 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:48:32.488 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:48:32.488 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 3233 2024-09-20T22:48:32.488 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.488 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:48:32.488 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.488 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:48:32.488 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.213253+0000", 2024-09-20T22:48:32.488 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:48:32.488 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.488 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.488 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:48:32.488 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.213253+0000", 2024-09-20T22:48:32.489 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:48:32.489 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.489 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.489 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:48:32.489 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.213254+0000", 2024-09-20T22:48:32.489 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.4799999999999998e-07 2024-09-20T22:48:32.489 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.489 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.489 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:48:32.489 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.213257+0000", 2024-09-20T22:48:32.489 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.3780000000000001e-06 2024-09-20T22:48:32.489 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.489 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.489 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:48:32.489 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.213267+0000", 2024-09-20T22:48:32.490 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.9480000000000003e-06 2024-09-20T22:48:32.490 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.490 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.490 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:48:32.490 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.213273+0000", 2024-09-20T22:48:32.490 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.1240000000000002e-06 2024-09-20T22:48:32.490 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.490 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.490 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:48:32.490 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.213394+0000", 2024-09-20T22:48:32.490 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000120717 2024-09-20T22:48:32.490 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.490 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.490 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting_for_map not empty", 2024-09-20T22:48:32.491 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.213400+0000", 2024-09-20T22:48:32.491 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.0769999999999996e-06 2024-09-20T22:48:32.491 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.491 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.491 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:48:32.491 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.220851+0000", 2024-09-20T22:48:32.491 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0074511309999999997 2024-09-20T22:48:32.491 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.491 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.491 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for blocked object", 2024-09-20T22:48:32.491 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.220903+0000", 2024-09-20T22:48:32.491 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.215e-05 2024-09-20T22:48:32.491 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.491 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.491 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:48:32.492 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.236698+0000", 2024-09-20T22:48:32.492 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.015794823 2024-09-20T22:48:32.492 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.492 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.492 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:48:32.492 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.236766+0000", 2024-09-20T22:48:32.492 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.8126999999999993e-05 2024-09-20T22:48:32.492 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.492 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.492 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:48:32.492 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.236848+0000", 2024-09-20T22:48:32.492 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.1701999999999998e-05 2024-09-20T22:48:32.492 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:48:32.492 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:48:32.493 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:48:32.493 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.493 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.493 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:3671 7.5 7:a160b5d4:::smithi08477203-16:head [copy-from ver 236] snapc 91=[91,8e,8d,8c,8b,8a,88,87,86,85,84,80,7f,7a,74,73,6b] ondisk+write+known_if_redirected+supports_pool_eio e370)", 2024-09-20T22:48:32.493 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:47:30.237235+0000", 2024-09-20T22:48:32.493 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 62.154651137000002, 2024-09-20T22:48:32.493 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.019719474000000001, 2024-09-20T22:48:32.493 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:48:32.493 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:48:32.493 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:48:32.493 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:48:32.493 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:48:32.493 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:48:32.493 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 3671 2024-09-20T22:48:32.494 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.494 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:48:32.494 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.494 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:48:32.494 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:30.237235+0000", 2024-09-20T22:48:32.494 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:48:32.494 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.494 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.494 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:48:32.494 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:30.237235+0000", 2024-09-20T22:48:32.494 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:48:32.494 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.494 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.494 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:48:32.495 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:30.237236+0000", 2024-09-20T22:48:32.495 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.5570000000000001e-06 2024-09-20T22:48:32.495 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.495 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.495 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:48:32.495 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:30.237241+0000", 2024-09-20T22:48:32.495 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.3279999999999999e-06 2024-09-20T22:48:32.495 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.495 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.495 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:48:32.495 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:30.237252+0000", 2024-09-20T22:48:32.495 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1004e-05 2024-09-20T22:48:32.495 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.495 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.496 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:48:32.496 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:30.237263+0000", 2024-09-20T22:48:32.496 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1341e-05 2024-09-20T22:48:32.496 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.496 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.496 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:48:32.496 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:30.237319+0000", 2024-09-20T22:48:32.496 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.5939000000000001e-05 2024-09-20T22:48:32.496 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.496 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.496 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:48:32.496 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:30.237436+0000", 2024-09-20T22:48:32.496 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000116866 2024-09-20T22:48:32.496 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.497 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.497 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 1,2", 2024-09-20T22:48:32.497 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:30.244330+0000", 2024-09-20T22:48:32.497 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0068945619999999999 2024-09-20T22:48:32.497 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.497 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.497 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:48:32.497 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:30.249103+0000", 2024-09-20T22:48:32.497 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0047726319999999997 2024-09-20T22:48:32.497 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.497 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.497 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:48:32.497 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:30.255771+0000", 2024-09-20T22:48:32.497 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.006667666 2024-09-20T22:48:32.497 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.498 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.498 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:48:32.498 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:30.256879+0000", 2024-09-20T22:48:32.498 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0011083110000000001 2024-09-20T22:48:32.498 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.498 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.498 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:48:32.498 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:30.256919+0000", 2024-09-20T22:48:32.498 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.9589000000000003e-05 2024-09-20T22:48:32.498 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.498 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.498 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:48:32.498 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:30.256954+0000", 2024-09-20T22:48:32.498 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.5679000000000002e-05 2024-09-20T22:48:32.499 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:48:32.499 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:48:32.499 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:48:32.499 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.499 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.499 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:3747 7.c e383/94 7:3f865ef5:::smithi08477203-28:head v 383'822, mlcod=382'818)", 2024-09-20T22:48:32.499 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:47:43.267267+0000", 2024-09-20T22:48:32.499 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 49.124618673000001, 2024-09-20T22:48:32.499 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.014405072, 2024-09-20T22:48:32.499 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:48:32.499 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:48:32.499 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:48:32.499 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:48:32.499 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.500 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:48:32.500 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.267267+0000", 2024-09-20T22:48:32.500 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:48:32.500 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.500 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.500 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:48:32.500 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.267267+0000", 2024-09-20T22:48:32.500 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:48:32.500 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.500 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.500 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:48:32.500 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.267267+0000", 2024-09-20T22:48:32.500 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.73e-07 2024-09-20T22:48:32.500 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.500 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.500 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:48:32.500 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.269537+0000", 2024-09-20T22:48:32.501 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0022696650000000001 2024-09-20T22:48:32.501 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.501 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.501 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:48:32.501 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.269564+0000", 2024-09-20T22:48:32.501 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.6653999999999999e-05 2024-09-20T22:48:32.501 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.501 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.501 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:48:32.501 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.269575+0000", 2024-09-20T22:48:32.501 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1661999999999999e-05 2024-09-20T22:48:32.501 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.501 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.502 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:48:32.502 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.269672+0000", 2024-09-20T22:48:32.502 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.6430999999999999e-05 2024-09-20T22:48:32.502 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.502 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.502 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:48:32.502 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.269716+0000", 2024-09-20T22:48:32.502 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.4518000000000003e-05 2024-09-20T22:48:32.502 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.502 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.502 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:48:32.502 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.281609+0000", 2024-09-20T22:48:32.502 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.011892982999999999 2024-09-20T22:48:32.502 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.503 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.503 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:48:32.503 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.281672+0000", 2024-09-20T22:48:32.503 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.2985999999999996e-05 2024-09-20T22:48:32.503 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:48:32.503 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:48:32.503 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:48:32.503 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.503 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.503 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:3762 7.17 7:ed54065f:::smithi08477203-12:head [write 739640~641484] snapc 98=[98,97,96,95,94,93,92,8e,8d,8c,8b,8a,88,87,86,80,7f,73] ondisk+write+known_if_redirected+supports_pool_eio e383)", 2024-09-20T22:48:32.503 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:47:43.291672+0000", 2024-09-20T22:48:32.503 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 49.100214321999999, 2024-09-20T22:48:32.503 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.012690432999999999, 2024-09-20T22:48:32.503 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:48:32.504 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:48:32.504 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:48:32.504 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:48:32.504 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:48:32.504 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:48:32.504 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 3762 2024-09-20T22:48:32.504 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.504 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:48:32.504 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.504 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:48:32.504 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.291672+0000", 2024-09-20T22:48:32.504 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:48:32.504 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.504 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.505 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:48:32.505 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.291672+0000", 2024-09-20T22:48:32.505 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:48:32.505 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.505 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.505 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:48:32.505 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.291673+0000", 2024-09-20T22:48:32.505 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.5680000000000001e-06 2024-09-20T22:48:32.505 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.505 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.505 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:48:32.505 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.292393+0000", 2024-09-20T22:48:32.505 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00072015399999999998 2024-09-20T22:48:32.505 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.505 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.506 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:48:32.506 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.292407+0000", 2024-09-20T22:48:32.506 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.365e-05 2024-09-20T22:48:32.506 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.506 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.506 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:48:32.506 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.292431+0000", 2024-09-20T22:48:32.506 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.3740999999999999e-05 2024-09-20T22:48:32.506 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.506 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.506 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:48:32.506 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.292482+0000", 2024-09-20T22:48:32.506 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.1459999999999999e-05 2024-09-20T22:48:32.506 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.506 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.507 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:48:32.507 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.292549+0000", 2024-09-20T22:48:32.507 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.7223000000000003e-05 2024-09-20T22:48:32.507 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.507 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.507 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 2,5", 2024-09-20T22:48:32.507 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.292684+0000", 2024-09-20T22:48:32.507 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000134251 2024-09-20T22:48:32.507 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.507 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.507 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:48:32.507 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.293845+0000", 2024-09-20T22:48:32.507 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0011617789999999999 2024-09-20T22:48:32.507 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.508 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.508 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:48:32.508 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.294392+0000", 2024-09-20T22:48:32.508 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00054659800000000003 2024-09-20T22:48:32.508 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.508 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.508 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:48:32.508 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.304282+0000", 2024-09-20T22:48:32.508 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0098903099999999994 2024-09-20T22:48:32.508 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.508 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.508 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:48:32.508 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.304322+0000", 2024-09-20T22:48:32.508 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.0059999999999999e-05 2024-09-20T22:48:32.508 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.509 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.509 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:48:32.509 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.304362+0000", 2024-09-20T22:48:32.509 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.9638999999999998e-05 2024-09-20T22:48:32.509 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:48:32.509 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:48:32.509 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:48:32.509 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.509 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.509 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:4129 7.c e417/94 7:3f865ef5:::smithi08477203-28:head v 417'903, mlcod=417'899)", 2024-09-20T22:48:32.509 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:48:17.254548+0000", 2024-09-20T22:48:32.509 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 15.137338158, 2024-09-20T22:48:32.509 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.012969022, 2024-09-20T22:48:32.509 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:48:32.509 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:48:32.509 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:48:32.510 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:48:32.510 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.510 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:48:32.510 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:17.254548+0000", 2024-09-20T22:48:32.510 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:48:32.510 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.510 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.510 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:48:32.510 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:17.254548+0000", 2024-09-20T22:48:32.510 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:48:32.510 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.510 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.510 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:48:32.510 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:17.254549+0000", 2024-09-20T22:48:32.510 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.446e-06 2024-09-20T22:48:32.511 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.511 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.511 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:48:32.511 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:17.257313+0000", 2024-09-20T22:48:32.511 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0027641319999999999 2024-09-20T22:48:32.511 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.511 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.511 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:48:32.511 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:17.257333+0000", 2024-09-20T22:48:32.511 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9950999999999999e-05 2024-09-20T22:48:32.511 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.511 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.511 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:48:32.511 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:17.257343+0000", 2024-09-20T22:48:32.512 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.2750000000000001e-06 2024-09-20T22:48:32.512 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.512 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.512 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:48:32.512 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:17.257416+0000", 2024-09-20T22:48:32.512 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.3041999999999996e-05 2024-09-20T22:48:32.512 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.512 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.512 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:48:32.512 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:17.257442+0000", 2024-09-20T22:48:32.512 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.6095999999999999e-05 2024-09-20T22:48:32.512 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.512 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.512 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:48:32.513 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:17.267454+0000", 2024-09-20T22:48:32.513 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.010012455999999999 2024-09-20T22:48:32.513 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:48:32.513 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:48:32.513 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:48:32.513 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:17.267517+0000", 2024-09-20T22:48:32.513 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.2624000000000002e-05 2024-09-20T22:48:32.513 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:48:32.513 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:48:32.513 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:48:32.513 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:48:32.513 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:48:32.513 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-20T22:48:32.514 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:48:32.535 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:32.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:32.536 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:48:32.554 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:32 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd rm-pg-upmap", "pgid": "7.17"}]: dispatch 2024-09-20T22:48:32.555 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:32 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "7.17"}]: dispatch 2024-09-20T22:48:32.555 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:32 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd rm-pg-upmap", "pgid": "7.17"}]: dispatch 2024-09-20T22:48:32.555 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:32 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "7.17"}]: dispatch 2024-09-20T22:48:32.555 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:32 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd rm-pg-upmap", "pgid": "7.17"}]': finished 2024-09-20T22:48:32.555 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:32 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd rm-pg-upmap-items", "pgid": "7.17"}]': finished 2024-09-20T22:48:32.555 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:32 smithi084 ceph-mon[40024]: osdmap e432: 8 total, 8 up, 8 in 2024-09-20T22:48:32.555 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:32 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd rm-pg-upmap", "pgid": "7.17"}]: dispatch 2024-09-20T22:48:32.555 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:32 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "7.17"}]: dispatch 2024-09-20T22:48:32.555 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:32 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd rm-pg-upmap", "pgid": "7.17"}]: dispatch 2024-09-20T22:48:32.555 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:32 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "7.17"}]: dispatch 2024-09-20T22:48:32.555 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:32 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd rm-pg-upmap", "pgid": "7.17"}]': finished 2024-09-20T22:48:32.555 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:32 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd rm-pg-upmap-items", "pgid": "7.17"}]': finished 2024-09-20T22:48:32.555 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:32 smithi084 ceph-mon[29774]: osdmap e432: 8 total, 8 up, 8 in 2024-09-20T22:48:32.637 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:48:32.659 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:32.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:32.660 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:48:32.761 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:48:32.782 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:32.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:32.783 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:48:32.884 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:48:32.906 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:32.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:32.907 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:48:33.008 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:48:33.029 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:33.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:33.030 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:48:33.131 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:48:33.152 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:33.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:33.153 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:48:33.185 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-20T22:48:33.204 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is unset 2024-09-20T22:48:33.213 INFO:tasks.rados.rados.0.smithi084.stdout:1678: done (1 left) 2024-09-20T22:48:33.213 INFO:tasks.rados.rados.0.smithi084.stdout:1679: done (0 left) 2024-09-20T22:48:33.213 INFO:tasks.rados.rados.0.smithi084.stdout:1680: rollback oid 12 current snap is 176 2024-09-20T22:48:33.213 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 12 to 148 2024-09-20T22:48:33.213 INFO:tasks.rados.rados.0.smithi084.stdout:1681: read oid 4 snap -1 2024-09-20T22:48:33.213 INFO:tasks.rados.rados.0.smithi084.stdout:1681: expect deleted 2024-09-20T22:48:33.213 INFO:tasks.rados.rados.0.smithi084.stdout:1682: rollback oid 46 current snap is 176 2024-09-20T22:48:33.213 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 46 to 172 2024-09-20T22:48:33.213 INFO:tasks.rados.rados.0.smithi084.stdout:1683: copy_from oid 7 from oid 18 current snap is 176 2024-09-20T22:48:33.213 INFO:tasks.rados.rados.0.smithi084.stdout:1684: write oid 14 current snap is 176 2024-09-20T22:48:33.213 INFO:tasks.rados.rados.0.smithi084.stdout:1684: seq_num 595 ranges {400165=648689,1684786=315379} 2024-09-20T22:48:33.219 INFO:tasks.rados.rados.0.smithi084.stdout:1684: writing smithi08477203-14 from 400165 to 1048854 tid 1 2024-09-20T22:48:33.225 INFO:tasks.rados.rados.0.smithi084.stdout:1684: writing smithi08477203-14 from 1684786 to 2000165 tid 2 2024-09-20T22:48:33.226 INFO:tasks.rados.rados.0.smithi084.stdout:1682: finishing rollback tid 1 to smithi08477203-46 2024-09-20T22:48:33.226 INFO:tasks.rados.rados.0.smithi084.stdout:1682: finishing rollback tid 2 to smithi08477203-46 2024-09-20T22:48:33.226 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 46 v 734 (ObjNum 470 snap 130 seq_num 470) dirty exists 2024-09-20T22:48:33.226 INFO:tasks.rados.rados.0.smithi084.stdout:1681: done (4 left) 2024-09-20T22:48:33.226 INFO:tasks.rados.rados.0.smithi084.stdout:1682: done (3 left) 2024-09-20T22:48:33.226 INFO:tasks.rados.rados.0.smithi084.stdout:1685: write oid 50 current snap is 176 2024-09-20T22:48:33.226 INFO:tasks.rados.rados.0.smithi084.stdout:1685: seq_num 596 ranges {416972=723938,1743268=273704} 2024-09-20T22:48:33.233 INFO:tasks.rados.rados.0.smithi084.stdout:1685: writing smithi08477203-50 from 416972 to 1140910 tid 1 2024-09-20T22:48:33.239 INFO:tasks.rados.rados.0.smithi084.stdout:1685: writing smithi08477203-50 from 1743268 to 2016972 tid 2 2024-09-20T22:48:33.240 INFO:tasks.rados.rados.0.smithi084.stdout:1684: finishing write tid 1 to smithi08477203-14 2024-09-20T22:48:33.240 INFO:tasks.rados.rados.0.smithi084.stdout:1683: finishing copy_from racing read to smithi08477203-7 2024-09-20T22:48:33.240 INFO:tasks.rados.rados.0.smithi084.stdout:1683: finishing copy_from to smithi08477203-7 2024-09-20T22:48:33.240 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 7 v 556 (ObjNum 568 snap 171 seq_num 568) dirty exists 2024-09-20T22:48:33.240 INFO:tasks.rados.rados.0.smithi084.stdout:1684: finishing write tid 2 to smithi08477203-14 2024-09-20T22:48:33.240 INFO:tasks.rados.rados.0.smithi084.stdout:1684: finishing write tid 3 to smithi08477203-14 2024-09-20T22:48:33.240 INFO:tasks.rados.rados.0.smithi084.stdout:1684: finishing write tid 4 to smithi08477203-14 2024-09-20T22:48:33.240 INFO:tasks.rados.rados.0.smithi084.stdout:1684: oid 14 updating version 0 to 195 2024-09-20T22:48:33.240 INFO:tasks.rados.rados.0.smithi084.stdout:1684: oid 14 version 195 is already newer than 193 2024-09-20T22:48:33.240 INFO:tasks.rados.rados.0.smithi084.stdout:1684: oid 14 version 195 is already newer than 194 2024-09-20T22:48:33.240 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 14 v 195 (ObjNum 595 snap 176 seq_num 595) dirty exists 2024-09-20T22:48:33.241 INFO:tasks.rados.rados.0.smithi084.stdout:1684: left oid 14 (ObjNum 595 snap 176 seq_num 595) 2024-09-20T22:48:33.241 INFO:tasks.rados.rados.0.smithi084.stdout:1683: done (3 left) 2024-09-20T22:48:33.241 INFO:tasks.rados.rados.0.smithi084.stdout:1684: done (2 left) 2024-09-20T22:48:33.241 INFO:tasks.rados.rados.0.smithi084.stdout:1686: snap_remove snap 155 2024-09-20T22:48:33.254 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:48:33.256 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-20T22:48:33.276 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:33.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:33.278 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:48:33.378 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:48:33.401 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:33.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:33.402 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:48:33.502 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:48:33.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:33.524 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:33.524 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:48:33.542 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:33 smithi084 ceph-mon[40024]: pgmap v549: 166 pgs: 1 peering, 3 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 156 active+clean; 363 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 4.3 MiB/s wr, 19 op/s 2024-09-20T22:48:33.542 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:33 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/3071043061' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:48:33.542 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:33 smithi084 ceph-mon[40024]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:48:33.542 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:33 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/3071043061' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T22:48:33.542 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:33 smithi084 ceph-mon[40024]: osdmap e433: 8 total, 8 up, 8 in 2024-09-20T22:48:33.543 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:33 smithi084 ceph-mon[29774]: pgmap v549: 166 pgs: 1 peering, 3 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 156 active+clean; 363 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 4.3 MiB/s wr, 19 op/s 2024-09-20T22:48:33.544 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:33 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/3071043061' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:48:33.544 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:33 smithi084 ceph-mon[29774]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:48:33.544 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:33 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/3071043061' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T22:48:33.544 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:33 smithi084 ceph-mon[29774]: osdmap e433: 8 total, 8 up, 8 in 2024-09-20T22:48:33.625 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:48:33.647 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:33.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:33.648 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:48:33.748 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:48:33.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:33 smithi154 ceph-mon[33264]: pgmap v549: 166 pgs: 1 peering, 3 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 156 active+clean; 363 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 4.3 MiB/s wr, 19 op/s 2024-09-20T22:48:33.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:33 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/3071043061' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:48:33.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:33 smithi154 ceph-mon[33264]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:48:33.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:33 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/3071043061' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T22:48:33.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:33 smithi154 ceph-mon[33264]: osdmap e433: 8 total, 8 up, 8 in 2024-09-20T22:48:33.770 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:33.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:33.771 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:48:33.872 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:48:33.894 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:33.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:33.895 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:48:33.996 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:48:34.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:34.021 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:34.021 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:48:34.025 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:48:34.045 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:48:34.122 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:48:34.143 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:34.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:34.144 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:48:34.212 INFO:tasks.rados.rados.0.smithi084.stdout:1686: done (2 left) 2024-09-20T22:48:34.212 INFO:tasks.rados.rados.0.smithi084.stdout:1687: delete oid 9 current snap is 176 2024-09-20T22:48:34.212 INFO:tasks.rados.rados.0.smithi084.stdout:1685: finishing write tid 1 to smithi08477203-50 2024-09-20T22:48:34.212 INFO:tasks.rados.rados.0.smithi084.stdout:1685: finishing write tid 2 to smithi08477203-50 2024-09-20T22:48:34.212 INFO:tasks.rados.rados.0.smithi084.stdout:1685: finishing write tid 3 to smithi08477203-50 2024-09-20T22:48:34.212 INFO:tasks.rados.rados.0.smithi084.stdout:1685: finishing write tid 4 to smithi08477203-50 2024-09-20T22:48:34.213 INFO:tasks.rados.rados.0.smithi084.stdout:1685: oid 50 updating version 0 to 253 2024-09-20T22:48:34.213 INFO:tasks.rados.rados.0.smithi084.stdout:1685: oid 50 updating version 253 to 255 2024-09-20T22:48:34.213 INFO:tasks.rados.rados.0.smithi084.stdout:1685: oid 50 version 255 is already newer than 254 2024-09-20T22:48:34.213 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 50 v 255 (ObjNum 596 snap 176 seq_num 596) dirty exists 2024-09-20T22:48:34.213 INFO:tasks.rados.rados.0.smithi084.stdout:1685: left oid 50 (ObjNum 596 snap 176 seq_num 596) 2024-09-20T22:48:34.217 INFO:tasks.rados.rados.0.smithi084.stdout:1685: done (2 left) 2024-09-20T22:48:34.217 INFO:tasks.rados.rados.0.smithi084.stdout:1687: done (1 left) 2024-09-20T22:48:34.217 INFO:tasks.rados.rados.0.smithi084.stdout:1688: write oid 24 current snap is 176 2024-09-20T22:48:34.217 INFO:tasks.rados.rados.0.smithi084.stdout:1688: seq_num 598 ranges {450586=474436,1343879=588206} 2024-09-20T22:48:34.223 INFO:tasks.rados.rados.0.smithi084.stdout:1688: writing smithi08477203-24 from 450586 to 925022 tid 1 2024-09-20T22:48:34.229 INFO:tasks.rados.rados.0.smithi084.stdout:1688: writing smithi08477203-24 from 1343879 to 1932085 tid 2 2024-09-20T22:48:34.229 INFO:tasks.rados.rados.0.smithi084.stdout:1689: delete oid 25 current snap is 176 2024-09-20T22:48:34.229 INFO:tasks.rados.rados.0.smithi084.stdout:1688: finishing write tid 1 to smithi08477203-24 2024-09-20T22:48:34.231 INFO:tasks.rados.rados.0.smithi084.stdout:1689: done (2 left) 2024-09-20T22:48:34.231 INFO:tasks.rados.rados.0.smithi084.stdout:1690: snap_remove snap 140 2024-09-20T22:48:34.245 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:48:34.267 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:34.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:34.268 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:48:34.369 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:48:34.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:34.391 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:34.391 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:48:34.492 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:48:34.497 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:48:34.497 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:48:34.511 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:48:34.511 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:48:34.513 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:34.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:34.514 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:48:34.526 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:48:34.527 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:48:34.543 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-20T22:48:34.544 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T22:48:34.560 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:48:34.561 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:48:34.577 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:48:34.577 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:48:34.593 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:48:34.593 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:48:34.610 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:48:34.610 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:48:34.615 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:48:34.638 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:34.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:34.639 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:48:34.740 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:48:34.761 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:34.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:34.763 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:48:34.863 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:48:34.884 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:34.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:34.885 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:48:34.986 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:48:35.007 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:35.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:35.008 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:48:35.109 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:48:35.130 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:35.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:35.131 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:48:35.201 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-20T22:48:35.202 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:48:35.216 INFO:tasks.rados.rados.0.smithi084.stdout:1690: done (2 left) 2024-09-20T22:48:35.216 INFO:tasks.rados.rados.0.smithi084.stdout:1691: read oid 6 snap -1 2024-09-20T22:48:35.216 INFO:tasks.rados.rados.0.smithi084.stdout:1691: expect (ObjNum 502 snap 143 seq_num 502) 2024-09-20T22:48:35.217 INFO:tasks.rados.rados.0.smithi084.stdout:1692: write oid 23 current snap is 176 2024-09-20T22:48:35.217 INFO:tasks.rados.rados.0.smithi084.stdout:1692: seq_num 600 ranges {484200=624934,1860843=223357} 2024-09-20T22:48:35.223 INFO:tasks.rados.rados.0.smithi084.stdout:1692: writing smithi08477203-23 from 484200 to 1109134 tid 1 2024-09-20T22:48:35.230 INFO:tasks.rados.rados.0.smithi084.stdout:1692: writing smithi08477203-23 from 1860843 to 2084200 tid 2 2024-09-20T22:48:35.230 INFO:tasks.rados.rados.0.smithi084.stdout:1688: finishing write tid 2 to smithi08477203-24 2024-09-20T22:48:35.230 INFO:tasks.rados.rados.0.smithi084.stdout:1688: finishing write tid 3 to smithi08477203-24 2024-09-20T22:48:35.230 INFO:tasks.rados.rados.0.smithi084.stdout:1688: finishing write tid 4 to smithi08477203-24 2024-09-20T22:48:35.230 INFO:tasks.rados.rados.0.smithi084.stdout:1688: oid 24 updating version 0 to 200 2024-09-20T22:48:35.230 INFO:tasks.rados.rados.0.smithi084.stdout:1688: oid 24 version 200 is already newer than 198 2024-09-20T22:48:35.231 INFO:tasks.rados.rados.0.smithi084.stdout:1688: oid 24 version 200 is already newer than 199 2024-09-20T22:48:35.231 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 24 v 200 (ObjNum 598 snap 176 seq_num 598) dirty exists 2024-09-20T22:48:35.231 INFO:tasks.rados.rados.0.smithi084.stdout:1688: left oid 24 (ObjNum 598 snap 176 seq_num 598) 2024-09-20T22:48:35.231 INFO:tasks.rados.rados.0.smithi084.stdout:1680: finishing rollback tid 0 to smithi08477203-12 2024-09-20T22:48:35.231 INFO:tasks.rados.rados.0.smithi084.stdout:1680: finishing rollback tid 1 to smithi08477203-12 2024-09-20T22:48:35.231 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 12 v 404 (ObjNum 129 snap 0 seq_num 2091838832) dirty dne 2024-09-20T22:48:35.231 INFO:tasks.rados.rados.0.smithi084.stdout:1691: expect (ObjNum 502 snap 143 seq_num 502) 2024-09-20T22:48:35.232 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:48:35.255 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:35.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:35.256 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:48:35.314 INFO:tasks.rados.rados.0.smithi084.stdout:1692: finishing write tid 1 to smithi08477203-23 2024-09-20T22:48:35.314 INFO:tasks.rados.rados.0.smithi084.stdout:1680: done (3 left) 2024-09-20T22:48:35.314 INFO:tasks.rados.rados.0.smithi084.stdout:1688: done (2 left) 2024-09-20T22:48:35.314 INFO:tasks.rados.rados.0.smithi084.stdout:1691: done (1 left) 2024-09-20T22:48:35.314 INFO:tasks.rados.rados.0.smithi084.stdout:1693: snap_remove snap 171 2024-09-20T22:48:35.358 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:48:35.364 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:35 smithi084 ceph-mon[40024]: pgmap v552: 166 pgs: 1 remapped+peering, 3 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 156 active+clean; 359 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 4.0 MiB/s rd, 2.2 MiB/s wr, 10 op/s 2024-09-20T22:48:35.364 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:35 smithi084 ceph-mon[40024]: osdmap e434: 8 total, 8 up, 8 in 2024-09-20T22:48:35.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:35 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/3221128270' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:48:35.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:35 smithi084 ceph-mon[40024]: 5.7 scrub starts 2024-09-20T22:48:35.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:35 smithi084 ceph-mon[40024]: 5.7 scrub ok 2024-09-20T22:48:35.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:35 smithi084 ceph-mon[29774]: pgmap v552: 166 pgs: 1 remapped+peering, 3 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 156 active+clean; 359 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 4.0 MiB/s rd, 2.2 MiB/s wr, 10 op/s 2024-09-20T22:48:35.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:35 smithi084 ceph-mon[29774]: osdmap e434: 8 total, 8 up, 8 in 2024-09-20T22:48:35.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:35 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/3221128270' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:48:35.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:35 smithi084 ceph-mon[29774]: 5.7 scrub starts 2024-09-20T22:48:35.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:35 smithi084 ceph-mon[29774]: 5.7 scrub ok 2024-09-20T22:48:35.379 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:35.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:35.380 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:48:35.481 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:48:35.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:35.504 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:35.504 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:48:35.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:35 smithi154 ceph-mon[33264]: pgmap v552: 166 pgs: 1 remapped+peering, 3 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 156 active+clean; 359 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 4.0 MiB/s rd, 2.2 MiB/s wr, 10 op/s 2024-09-20T22:48:35.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:35 smithi154 ceph-mon[33264]: osdmap e434: 8 total, 8 up, 8 in 2024-09-20T22:48:35.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:35 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/3221128270' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:48:35.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:35 smithi154 ceph-mon[33264]: 5.7 scrub starts 2024-09-20T22:48:35.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:35 smithi154 ceph-mon[33264]: 5.7 scrub ok 2024-09-20T22:48:35.605 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:48:35.627 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:35.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:35.628 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:48:35.730 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:48:35.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:35.753 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:35.753 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:48:35.854 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:48:35.875 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.0 dump_blocked_ops 2024-09-20T22:48:35.876 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:35.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:35.877 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:48:35.954 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-20T22:48:35.978 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:48:36.000 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:36.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:36.001 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:48:36.102 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:48:36.125 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:36.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:36.126 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:48:36.226 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:48:36.228 INFO:tasks.rados.rados.0.smithi084.stdout:1693: done (1 left) 2024-09-20T22:48:36.229 INFO:tasks.rados.rados.0.smithi084.stdout:1694: snap_create 2024-09-20T22:48:36.229 INFO:tasks.rados.rados.0.smithi084.stdout:1692: finishing write tid 2 to smithi08477203-23 2024-09-20T22:48:36.229 INFO:tasks.rados.rados.0.smithi084.stdout:1692: finishing write tid 3 to smithi08477203-23 2024-09-20T22:48:36.229 INFO:tasks.rados.rados.0.smithi084.stdout:1692: finishing write tid 4 to smithi08477203-23 2024-09-20T22:48:36.229 INFO:tasks.rados.rados.0.smithi084.stdout:1692: oid 23 updating version 0 to 703 2024-09-20T22:48:36.229 INFO:tasks.rados.rados.0.smithi084.stdout:1692: oid 23 version 703 is already newer than 702 2024-09-20T22:48:36.229 INFO:tasks.rados.rados.0.smithi084.stdout:1692: oid 23 version 703 is already newer than 701 2024-09-20T22:48:36.229 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 23 v 703 (ObjNum 600 snap 176 seq_num 600) dirty exists 2024-09-20T22:48:36.229 INFO:tasks.rados.rados.0.smithi084.stdout:1692: left oid 23 (ObjNum 600 snap 176 seq_num 600) 2024-09-20T22:48:36.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:36.253 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:36.254 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:48:36.354 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:48:36.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:36.378 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:36.378 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:48:36.479 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:48:36.501 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:36.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:36.502 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:48:36.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:36 smithi154 ceph-mon[33264]: osdmap e435: 8 total, 8 up, 8 in 2024-09-20T22:48:36.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:36 smithi154 ceph-mon[33264]: 5.4 scrub starts 2024-09-20T22:48:36.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:36 smithi154 ceph-mon[33264]: 5.4 scrub ok 2024-09-20T22:48:36.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:36 smithi154 ceph-mon[33264]: 5.16 scrub starts 2024-09-20T22:48:36.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:36 smithi154 ceph-mon[33264]: 5.16 scrub ok 2024-09-20T22:48:36.537 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:48:36.565 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:48:36.603 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:48:36.614 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:36 smithi084 ceph-mon[29774]: osdmap e435: 8 total, 8 up, 8 in 2024-09-20T22:48:36.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:36 smithi084 ceph-mon[29774]: 5.4 scrub starts 2024-09-20T22:48:36.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:36 smithi084 ceph-mon[29774]: 5.4 scrub ok 2024-09-20T22:48:36.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:36 smithi084 ceph-mon[29774]: 5.16 scrub starts 2024-09-20T22:48:36.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:36 smithi084 ceph-mon[29774]: 5.16 scrub ok 2024-09-20T22:48:36.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:36 smithi084 ceph-mon[40024]: osdmap e435: 8 total, 8 up, 8 in 2024-09-20T22:48:36.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:36 smithi084 ceph-mon[40024]: 5.4 scrub starts 2024-09-20T22:48:36.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:36 smithi084 ceph-mon[40024]: 5.4 scrub ok 2024-09-20T22:48:36.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:36 smithi084 ceph-mon[40024]: 5.16 scrub starts 2024-09-20T22:48:36.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:36 smithi084 ceph-mon[40024]: 5.16 scrub ok 2024-09-20T22:48:36.624 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:36.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:36.625 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:48:36.726 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:48:36.748 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:36.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:36.749 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:48:36.850 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:48:36.871 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:36.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:36.872 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:48:36.900 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 5, 6, 7, 0, 4] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-20T22:48:36.900 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-20T22:48:36.900 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-09-20T22:48:36.900 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd dump -f json-pretty 2024-09-20T22:48:36.973 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:48:36.995 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:36.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:36.996 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:48:37.097 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:48:37.120 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:37.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:37.121 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:48:37.221 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:48:37.233 INFO:tasks.rados.rados.0.smithi084.stdout:1692: done (1 left) 2024-09-20T22:48:37.233 INFO:tasks.rados.rados.0.smithi084.stdout:1694: done (0 left) 2024-09-20T22:48:37.233 INFO:tasks.rados.rados.0.smithi084.stdout:1695: delete oid 49 current snap is 177 2024-09-20T22:48:37.244 INFO:tasks.rados.rados.0.smithi084.stdout:1695: done (0 left) 2024-09-20T22:48:37.244 INFO:tasks.rados.rados.0.smithi084.stdout:1696: write (excl) oid 30 current snap is 177 2024-09-20T22:48:37.244 INFO:tasks.rados.rados.0.smithi084.stdout:1696: seq_num 602 ranges {517814=491785,1577807=540007} 2024-09-20T22:48:37.249 INFO:tasks.rados.rados.0.smithi084.stdout:1696: writing smithi08477203-30 from 517814 to 1009599 tid 1 2024-09-20T22:48:37.255 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:37.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:37.256 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:48:37.257 INFO:tasks.rados.rados.0.smithi084.stdout:1696: writing smithi08477203-30 from 1577807 to 2117814 tid 2 2024-09-20T22:48:37.257 INFO:tasks.rados.rados.0.smithi084.stdout:1697: rollback oid 23 current snap is 177 2024-09-20T22:48:37.257 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 23 to 152 2024-09-20T22:48:37.257 INFO:tasks.rados.rados.0.smithi084.stdout:1696: finishing write tid 1 to smithi08477203-30 2024-09-20T22:48:37.257 INFO:tasks.rados.rados.0.smithi084.stdout:1698: snap_create 2024-09-20T22:48:37.260 INFO:tasks.rados.rados.0.smithi084.stdout:1697: finishing rollback tid 0 to smithi08477203-23 2024-09-20T22:48:37.260 INFO:tasks.rados.rados.0.smithi084.stdout:1697: finishing rollback tid 1 to smithi08477203-23 2024-09-20T22:48:37.260 INFO:tasks.rados.rados.0.smithi084.stdout:1697: finishing rollback tid 2 to smithi08477203-23 2024-09-20T22:48:37.261 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 23 v 706 (ObjNum 428 snap 110 seq_num 428) dirty exists 2024-09-20T22:48:37.261 INFO:tasks.rados.rados.0.smithi084.stdout:1696: finishing write tid 2 to smithi08477203-30 2024-09-20T22:48:37.261 INFO:tasks.rados.rados.0.smithi084.stdout:1696: finishing write tid 3 to smithi08477203-30 2024-09-20T22:48:37.261 INFO:tasks.rados.rados.0.smithi084.stdout:1696: finishing write tid 4 to smithi08477203-30 2024-09-20T22:48:37.261 INFO:tasks.rados.rados.0.smithi084.stdout:1696: oid 30 updating version 0 to 736 2024-09-20T22:48:37.261 INFO:tasks.rados.rados.0.smithi084.stdout:1696: oid 30 version 736 is already newer than 735 2024-09-20T22:48:37.261 INFO:tasks.rados.rados.0.smithi084.stdout:1696: oid 30 updating version 736 to 737 2024-09-20T22:48:37.261 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 30 v 737 (ObjNum 602 snap 177 seq_num 602) dirty exists 2024-09-20T22:48:37.262 INFO:tasks.rados.rados.0.smithi084.stdout:1696: left oid 30 (ObjNum 602 snap 177 seq_num 602) 2024-09-20T22:48:37.329 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-20T22:48:37.330 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:48:37.356 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:48:37.378 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:37.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:37.379 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:48:37.480 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:48:37.501 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:37.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:37.502 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:48:37.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:37 smithi154 ceph-mon[33264]: pgmap v555: 166 pgs: 1 remapped+peering, 3 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 156 active+clean; 359 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 27 MiB/s rd, 9.2 MiB/s wr, 55 op/s; 5.3 MiB/s, 14 objects/s recovering 2024-09-20T22:48:37.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:37 smithi154 ceph-mon[33264]: osdmap e436: 8 total, 8 up, 8 in 2024-09-20T22:48:37.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:37 smithi154 ceph-mon[33264]: 3.18 scrub starts 2024-09-20T22:48:37.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:37 smithi154 ceph-mon[33264]: 3.18 scrub ok 2024-09-20T22:48:37.603 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:48:37.614 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:37 smithi084 ceph-mon[40024]: pgmap v555: 166 pgs: 1 remapped+peering, 3 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 156 active+clean; 359 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 27 MiB/s rd, 9.2 MiB/s wr, 55 op/s; 5.3 MiB/s, 14 objects/s recovering 2024-09-20T22:48:37.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:37 smithi084 ceph-mon[40024]: osdmap e436: 8 total, 8 up, 8 in 2024-09-20T22:48:37.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:37 smithi084 ceph-mon[40024]: 3.18 scrub starts 2024-09-20T22:48:37.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:37 smithi084 ceph-mon[40024]: 3.18 scrub ok 2024-09-20T22:48:37.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:37 smithi084 ceph-mon[29774]: pgmap v555: 166 pgs: 1 remapped+peering, 3 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 156 active+clean; 359 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 27 MiB/s rd, 9.2 MiB/s wr, 55 op/s; 5.3 MiB/s, 14 objects/s recovering 2024-09-20T22:48:37.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:37 smithi084 ceph-mon[29774]: osdmap e436: 8 total, 8 up, 8 in 2024-09-20T22:48:37.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:37 smithi084 ceph-mon[29774]: 3.18 scrub starts 2024-09-20T22:48:37.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:37 smithi084 ceph-mon[29774]: 3.18 scrub ok 2024-09-20T22:48:37.625 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:37.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:37.626 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:48:37.727 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:48:37.750 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:37.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:37.751 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:48:37.851 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:48:37.875 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:37.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:37.876 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:48:37.940 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.0 dump_historic_ops 2024-09-20T22:48:37.976 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:48:37.999 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:38.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:38.000 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:48:38.086 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:48:38.101 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:48:38.125 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:38.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:38.126 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:48:38.213 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is unset 2024-09-20T22:48:38.213 INFO:tasks.rados.rados.0.smithi084.stdout:1696: done (2 left) 2024-09-20T22:48:38.213 INFO:tasks.rados.rados.0.smithi084.stdout:1697: done (1 left) 2024-09-20T22:48:38.213 INFO:tasks.rados.rados.0.smithi084.stdout:1698: done (0 left) 2024-09-20T22:48:38.213 INFO:tasks.rados.rados.0.smithi084.stdout:1699: rollback oid 44 current snap is 178 2024-09-20T22:48:38.214 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 44 to 172 2024-09-20T22:48:38.214 INFO:tasks.rados.rados.0.smithi084.stdout:1700: snap_create 2024-09-20T22:48:38.218 INFO:tasks.rados.rados.0.smithi084.stdout:1699: finishing rollback tid 1 to smithi08477203-44 2024-09-20T22:48:38.218 INFO:tasks.rados.rados.0.smithi084.stdout:1699: finishing rollback tid 2 to smithi08477203-44 2024-09-20T22:48:38.218 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 44 v 1046 (ObjNum 470 snap 130 seq_num 470) dirty exists 2024-09-20T22:48:38.227 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:48:38.253 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:38.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:38.254 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:48:38.355 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:48:38.379 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:38.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:38.380 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:48:38.445 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-20T22:48:38.467 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:48:38 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: ::ffff:172.21.15.154 - - [20/Sep/2024:22:48:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-20T22:48:38.467 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:38 smithi084 ceph-mon[29774]: 3.6 scrub starts 2024-09-20T22:48:38.467 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:38 smithi084 ceph-mon[29774]: 3.6 scrub ok 2024-09-20T22:48:38.467 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:38 smithi084 ceph-mon[29774]: osdmap e437: 8 total, 8 up, 8 in 2024-09-20T22:48:38.468 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:38 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/3380721340' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:48:38.468 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:38 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:48:38.468 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:38 smithi084 ceph-mon[29774]: 3.1c scrub starts 2024-09-20T22:48:38.468 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:38 smithi084 ceph-mon[29774]: 3.1c scrub ok 2024-09-20T22:48:38.468 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:38 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:48:38.468 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:38 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:48:38.468 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:38 smithi084 ceph-mon[29774]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:48:38.468 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:38 smithi084 ceph-mon[29774]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T22:48:38.468 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:38 smithi084 ceph-mon[29774]: Cluster is now healthy 2024-09-20T22:48:38.468 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:38 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T22:48:38.468 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:38 smithi084 ceph-mon[29774]: osdmap e438: 8 total, 8 up, 8 in 2024-09-20T22:48:38.468 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:38 smithi084 ceph-mon[40024]: 3.6 scrub starts 2024-09-20T22:48:38.469 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:38 smithi084 ceph-mon[40024]: 3.6 scrub ok 2024-09-20T22:48:38.469 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:38 smithi084 ceph-mon[40024]: osdmap e437: 8 total, 8 up, 8 in 2024-09-20T22:48:38.469 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:38 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/3380721340' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:48:38.469 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:38 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:48:38.469 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:38 smithi084 ceph-mon[40024]: 3.1c scrub starts 2024-09-20T22:48:38.469 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:38 smithi084 ceph-mon[40024]: 3.1c scrub ok 2024-09-20T22:48:38.469 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:38 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:48:38.469 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:38 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:48:38.469 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:38 smithi084 ceph-mon[40024]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:48:38.469 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:38 smithi084 ceph-mon[40024]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T22:48:38.469 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:38 smithi084 ceph-mon[40024]: Cluster is now healthy 2024-09-20T22:48:38.469 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:38 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T22:48:38.469 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:38 smithi084 ceph-mon[40024]: osdmap e438: 8 total, 8 up, 8 in 2024-09-20T22:48:38.482 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:48:38.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:38 smithi154 ceph-mon[33264]: 3.6 scrub starts 2024-09-20T22:48:38.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:38 smithi154 ceph-mon[33264]: 3.6 scrub ok 2024-09-20T22:48:38.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:38 smithi154 ceph-mon[33264]: osdmap e437: 8 total, 8 up, 8 in 2024-09-20T22:48:38.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:38 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/3380721340' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:48:38.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:38 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:48:38.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:38 smithi154 ceph-mon[33264]: 3.1c scrub starts 2024-09-20T22:48:38.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:38 smithi154 ceph-mon[33264]: 3.1c scrub ok 2024-09-20T22:48:38.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:38 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:48:38.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:38 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:48:38.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:38 smithi154 ceph-mon[33264]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:48:38.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:38 smithi154 ceph-mon[33264]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T22:48:38.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:38 smithi154 ceph-mon[33264]: Cluster is now healthy 2024-09-20T22:48:38.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:38 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T22:48:38.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:38 smithi154 ceph-mon[33264]: osdmap e438: 8 total, 8 up, 8 in 2024-09-20T22:48:38.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:38.506 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:38.506 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:48:38.607 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:48:38.630 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:38.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:38.631 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:48:38.732 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:48:38.753 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:38.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:38.754 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:48:38.855 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:48:38.878 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:38.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:38.879 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:48:38.980 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:48:39.002 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:39.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:39.003 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:48:39.103 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:48:39.125 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:39.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:39.126 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:48:39.214 INFO:tasks.rados.rados.0.smithi084.stdout:1699: done (1 left) 2024-09-20T22:48:39.214 INFO:tasks.rados.rados.0.smithi084.stdout:1700: done (0 left) 2024-09-20T22:48:39.214 INFO:tasks.rados.rados.0.smithi084.stdout:1701: rollback oid 49 current snap is 179 2024-09-20T22:48:39.214 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 49 to 173 2024-09-20T22:48:39.214 INFO:tasks.rados.rados.0.smithi084.stdout:1702: read oid 21 snap 177 2024-09-20T22:48:39.214 INFO:tasks.rados.rados.0.smithi084.stdout:1702: expect (ObjNum 501 snap 143 seq_num 501) 2024-09-20T22:48:39.215 INFO:tasks.rados.rados.0.smithi084.stdout:1703: delete oid 6 current snap is 179 2024-09-20T22:48:39.220 INFO:tasks.rados.rados.0.smithi084.stdout:1703: done (2 left) 2024-09-20T22:48:39.220 INFO:tasks.rados.rados.0.smithi084.stdout:1704: copy_from oid 16 from oid 3 current snap is 179 2024-09-20T22:48:39.220 INFO:tasks.rados.rados.0.smithi084.stdout:1705: read oid 24 snap -1 2024-09-20T22:48:39.221 INFO:tasks.rados.rados.0.smithi084.stdout:1705: expect (ObjNum 598 snap 176 seq_num 598) 2024-09-20T22:48:39.221 INFO:tasks.rados.rados.0.smithi084.stdout:1706: copy_from oid 7 from oid 37 current snap is 179 2024-09-20T22:48:39.221 INFO:tasks.rados.rados.0.smithi084.stdout:1707: copy_from oid 39 from oid 9 current snap is 179 2024-09-20T22:48:39.221 INFO:tasks.rados.rados.0.smithi084.stdout:1708: snap_create 2024-09-20T22:48:39.221 INFO:tasks.rados.rados.0.smithi084.stdout:1707: finishing copy_from to smithi08477203-39 2024-09-20T22:48:39.221 INFO:tasks.rados.rados.0.smithi084.stdout:1707: got expected ENOENT (src dne) 2024-09-20T22:48:39.222 INFO:tasks.rados.rados.0.smithi084.stdout:1707: finishing copy_from racing read to smithi08477203-39 2024-09-20T22:48:39.222 INFO:tasks.rados.rados.0.smithi084.stdout:1706: finishing copy_from to smithi08477203-7 2024-09-20T22:48:39.222 INFO:tasks.rados.rados.0.smithi084.stdout:1706: got expected ENOENT (src dne) 2024-09-20T22:48:39.222 INFO:tasks.rados.rados.0.smithi084.stdout:1706: finishing copy_from racing read to smithi08477203-7 2024-09-20T22:48:39.223 INFO:tasks.rados.rados.0.smithi084.stdout:1701: finishing rollback tid 1 to smithi08477203-49 2024-09-20T22:48:39.224 INFO:tasks.rados.rados.0.smithi084.stdout:1701: finishing rollback tid 2 to smithi08477203-49 2024-09-20T22:48:39.224 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 49 v 534 (ObjNum 557 snap 169 seq_num 557) dirty exists 2024-09-20T22:48:39.224 INFO:tasks.rados.rados.0.smithi084.stdout:1704: finishing copy_from to smithi08477203-16 2024-09-20T22:48:39.224 INFO:tasks.rados.rados.0.smithi084.stdout:1704: got expected ENOENT (src dne) 2024-09-20T22:48:39.224 INFO:tasks.rados.rados.0.smithi084.stdout:1704: finishing copy_from racing read to smithi08477203-16 2024-09-20T22:48:39.224 INFO:tasks.rados.rados.0.smithi084.stdout:1702: expect (ObjNum 501 snap 143 seq_num 501) 2024-09-20T22:48:39.227 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:48:39.247 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:48:39.247 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:48:39.252 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:39.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:39.253 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:48:39.353 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:48:39.376 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:39.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:39.377 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:48:39.402 INFO:tasks.rados.rados.0.smithi084.stdout:1705: expect (ObjNum 598 snap 176 seq_num 598) 2024-09-20T22:48:39.436 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-20T22:48:39.436 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-20T22:48:39.436 INFO:teuthology.orchestra.run.smithi084.stdout: "epoch": 439, 2024-09-20T22:48:39.436 INFO:teuthology.orchestra.run.smithi084.stdout: "fsid": "a9c0bc70-77a0-11ef-baf5-efdc52797490", 2024-09-20T22:48:39.436 INFO:teuthology.orchestra.run.smithi084.stdout: "created": "2024-09-20T22:36:30.240626+0000", 2024-09-20T22:48:39.436 INFO:teuthology.orchestra.run.smithi084.stdout: "modified": "2024-09-20T22:48:39.204612+0000", 2024-09-20T22:48:39.436 INFO:teuthology.orchestra.run.smithi084.stdout: "last_up_change": "2024-09-20T22:41:27.098565+0000", 2024-09-20T22:48:39.436 INFO:teuthology.orchestra.run.smithi084.stdout: "last_in_change": "2024-09-20T22:48:04.201694+0000", 2024-09-20T22:48:39.436 INFO:teuthology.orchestra.run.smithi084.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-20T22:48:39.436 INFO:teuthology.orchestra.run.smithi084.stdout: "flags_num": 5799936, 2024-09-20T22:48:39.436 INFO:teuthology.orchestra.run.smithi084.stdout: "flags_set": [ 2024-09-20T22:48:39.437 INFO:teuthology.orchestra.run.smithi084.stdout: "pglog_hardlimit", 2024-09-20T22:48:39.437 INFO:teuthology.orchestra.run.smithi084.stdout: "purged_snapdirs", 2024-09-20T22:48:39.437 INFO:teuthology.orchestra.run.smithi084.stdout: "recovery_deletes", 2024-09-20T22:48:39.437 INFO:teuthology.orchestra.run.smithi084.stdout: "sortbitwise" 2024-09-20T22:48:39.437 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2024-09-20T22:48:39.437 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_version": 18, 2024-09-20T22:48:39.437 INFO:teuthology.orchestra.run.smithi084.stdout: "full_ratio": 0.94999998807907104, 2024-09-20T22:48:39.437 INFO:teuthology.orchestra.run.smithi084.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-20T22:48:39.437 INFO:teuthology.orchestra.run.smithi084.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-20T22:48:39.437 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_snapshot": "", 2024-09-20T22:48:39.437 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_max": 7, 2024-09-20T22:48:39.437 INFO:teuthology.orchestra.run.smithi084.stdout: "max_osd": 8, 2024-09-20T22:48:39.437 INFO:teuthology.orchestra.run.smithi084.stdout: "require_min_compat_client": "luminous", 2024-09-20T22:48:39.437 INFO:teuthology.orchestra.run.smithi084.stdout: "min_compat_client": "luminous", 2024-09-20T22:48:39.438 INFO:teuthology.orchestra.run.smithi084.stdout: "require_osd_release": "squid", 2024-09-20T22:48:39.438 INFO:teuthology.orchestra.run.smithi084.stdout: "allow_crimson": false, 2024-09-20T22:48:39.438 INFO:teuthology.orchestra.run.smithi084.stdout: "pools": [ 2024-09-20T22:48:39.438 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:39.438 INFO:teuthology.orchestra.run.smithi084.stdout: "pool": 1, 2024-09-20T22:48:39.438 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_name": ".mgr", 2024-09-20T22:48:39.438 INFO:teuthology.orchestra.run.smithi084.stdout: "create_time": "2024-09-20T22:39:21.984381+0000", 2024-09-20T22:48:39.438 INFO:teuthology.orchestra.run.smithi084.stdout: "flags": 1, 2024-09-20T22:48:39.438 INFO:teuthology.orchestra.run.smithi084.stdout: "flags_names": "hashpspool", 2024-09-20T22:48:39.438 INFO:teuthology.orchestra.run.smithi084.stdout: "type": 1, 2024-09-20T22:48:39.438 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 3, 2024-09-20T22:48:39.438 INFO:teuthology.orchestra.run.smithi084.stdout: "min_size": 2, 2024-09-20T22:48:39.438 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_rule": 0, 2024-09-20T22:48:39.438 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_count": 0, 2024-09-20T22:48:39.439 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_target": 0, 2024-09-20T22:48:39.439 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_barrier": 0, 2024-09-20T22:48:39.439 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-20T22:48:39.439 INFO:teuthology.orchestra.run.smithi084.stdout: "is_stretch_pool": false, 2024-09-20T22:48:39.439 INFO:teuthology.orchestra.run.smithi084.stdout: "object_hash": 2, 2024-09-20T22:48:39.439 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_autoscale_mode": "off", 2024-09-20T22:48:39.439 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num": 11, 2024-09-20T22:48:39.439 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num": 11, 2024-09-20T22:48:39.439 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num_target": 11, 2024-09-20T22:48:39.439 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_target": 11, 2024-09-20T22:48:39.439 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_pending": 11, 2024-09-20T22:48:39.439 INFO:teuthology.orchestra.run.smithi084.stdout: "last_pg_merge_meta": { 2024-09-20T22:48:39.439 INFO:teuthology.orchestra.run.smithi084.stdout: "source_pgid": "1.b", 2024-09-20T22:48:39.439 INFO:teuthology.orchestra.run.smithi084.stdout: "ready_epoch": 299, 2024-09-20T22:48:39.440 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_started": 298, 2024-09-20T22:48:39.440 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_clean": 298, 2024-09-20T22:48:39.440 INFO:teuthology.orchestra.run.smithi084.stdout: "source_version": "77'128", 2024-09-20T22:48:39.440 INFO:teuthology.orchestra.run.smithi084.stdout: "target_version": "77'128" 2024-09-20T22:48:39.440 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:39.440 INFO:teuthology.orchestra.run.smithi084.stdout: "last_change": "299", 2024-09-20T22:48:39.440 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend": "0", 2024-09-20T22:48:39.440 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_prenautilus": "299", 2024-09-20T22:48:39.440 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_preluminous": "297", 2024-09-20T22:48:39.440 INFO:teuthology.orchestra.run.smithi084.stdout: "auid": 0, 2024-09-20T22:48:39.440 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_mode": "selfmanaged", 2024-09-20T22:48:39.440 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_seq": 0, 2024-09-20T22:48:39.440 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_epoch": 0, 2024-09-20T22:48:39.440 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_snaps": [], 2024-09-20T22:48:39.441 INFO:teuthology.orchestra.run.smithi084.stdout: "removed_snaps": "[]", 2024-09-20T22:48:39.441 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_bytes": 0, 2024-09-20T22:48:39.441 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_objects": 0, 2024-09-20T22:48:39.441 INFO:teuthology.orchestra.run.smithi084.stdout: "tiers": [], 2024-09-20T22:48:39.441 INFO:teuthology.orchestra.run.smithi084.stdout: "tier_of": -1, 2024-09-20T22:48:39.441 INFO:teuthology.orchestra.run.smithi084.stdout: "read_tier": -1, 2024-09-20T22:48:39.441 INFO:teuthology.orchestra.run.smithi084.stdout: "write_tier": -1, 2024-09-20T22:48:39.441 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_mode": "none", 2024-09-20T22:48:39.441 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_bytes": 0, 2024-09-20T22:48:39.441 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_objects": 0, 2024-09-20T22:48:39.441 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-20T22:48:39.441 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-20T22:48:39.441 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-20T22:48:39.441 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_flush_age": 0, 2024-09-20T22:48:39.442 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_evict_age": 0, 2024-09-20T22:48:39.442 INFO:teuthology.orchestra.run.smithi084.stdout: "erasure_code_profile": "", 2024-09-20T22:48:39.442 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_params": { 2024-09-20T22:48:39.442 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "none" 2024-09-20T22:48:39.442 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:39.442 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_period": 0, 2024-09-20T22:48:39.442 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_count": 0, 2024-09-20T22:48:39.442 INFO:teuthology.orchestra.run.smithi084.stdout: "use_gmt_hitset": true, 2024-09-20T22:48:39.442 INFO:teuthology.orchestra.run.smithi084.stdout: "min_read_recency_for_promote": 0, 2024-09-20T22:48:39.442 INFO:teuthology.orchestra.run.smithi084.stdout: "min_write_recency_for_promote": 0, 2024-09-20T22:48:39.442 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_grade_decay_rate": 0, 2024-09-20T22:48:39.442 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_search_last_n": 0, 2024-09-20T22:48:39.442 INFO:teuthology.orchestra.run.smithi084.stdout: "grade_table": [], 2024-09-20T22:48:39.442 INFO:teuthology.orchestra.run.smithi084.stdout: "stripe_width": 0, 2024-09-20T22:48:39.443 INFO:teuthology.orchestra.run.smithi084.stdout: "expected_num_objects": 0, 2024-09-20T22:48:39.443 INFO:teuthology.orchestra.run.smithi084.stdout: "fast_read": false, 2024-09-20T22:48:39.443 INFO:teuthology.orchestra.run.smithi084.stdout: "options": { 2024-09-20T22:48:39.443 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_max": 32, 2024-09-20T22:48:39.443 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_min": 1 2024-09-20T22:48:39.443 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:39.443 INFO:teuthology.orchestra.run.smithi084.stdout: "application_metadata": { 2024-09-20T22:48:39.443 INFO:teuthology.orchestra.run.smithi084.stdout: "mgr": {} 2024-09-20T22:48:39.443 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:39.443 INFO:teuthology.orchestra.run.smithi084.stdout: "read_balance": { 2024-09-20T22:48:39.443 INFO:teuthology.orchestra.run.smithi084.stdout: "score_type": "Fair distribution", 2024-09-20T22:48:39.443 INFO:teuthology.orchestra.run.smithi084.stdout: "score_acting": 2.1800000667572021, 2024-09-20T22:48:39.443 INFO:teuthology.orchestra.run.smithi084.stdout: "score_stable": 2.1800000667572021, 2024-09-20T22:48:39.444 INFO:teuthology.orchestra.run.smithi084.stdout: "optimal_score": 1, 2024-09-20T22:48:39.444 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_acting": 2.1800000667572021, 2024-09-20T22:48:39.444 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_stable": 2.1800000667572021, 2024-09-20T22:48:39.444 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity_weighted": 1, 2024-09-20T22:48:39.444 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity": 1, 2024-09-20T22:48:39.444 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity_weighted": 1 2024-09-20T22:48:39.444 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:39.444 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:39.444 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:39.444 INFO:teuthology.orchestra.run.smithi084.stdout: "pool": 2, 2024-09-20T22:48:39.444 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_name": "datapool", 2024-09-20T22:48:39.444 INFO:teuthology.orchestra.run.smithi084.stdout: "create_time": "2024-09-20T22:41:35.820081+0000", 2024-09-20T22:48:39.444 INFO:teuthology.orchestra.run.smithi084.stdout: "flags": 8193, 2024-09-20T22:48:39.444 INFO:teuthology.orchestra.run.smithi084.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-20T22:48:39.444 INFO:teuthology.orchestra.run.smithi084.stdout: "type": 1, 2024-09-20T22:48:39.445 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 3, 2024-09-20T22:48:39.445 INFO:teuthology.orchestra.run.smithi084.stdout: "min_size": 2, 2024-09-20T22:48:39.445 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_rule": 0, 2024-09-20T22:48:39.445 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_count": 0, 2024-09-20T22:48:39.445 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_target": 0, 2024-09-20T22:48:39.445 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_barrier": 0, 2024-09-20T22:48:39.445 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-20T22:48:39.445 INFO:teuthology.orchestra.run.smithi084.stdout: "is_stretch_pool": false, 2024-09-20T22:48:39.445 INFO:teuthology.orchestra.run.smithi084.stdout: "object_hash": 2, 2024-09-20T22:48:39.445 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_autoscale_mode": "off", 2024-09-20T22:48:39.445 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num": 3, 2024-09-20T22:48:39.445 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num": 3, 2024-09-20T22:48:39.445 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num_target": 3, 2024-09-20T22:48:39.445 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_target": 3, 2024-09-20T22:48:39.446 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_pending": 3, 2024-09-20T22:48:39.446 INFO:teuthology.orchestra.run.smithi084.stdout: "last_pg_merge_meta": { 2024-09-20T22:48:39.446 INFO:teuthology.orchestra.run.smithi084.stdout: "source_pgid": "0.0", 2024-09-20T22:48:39.446 INFO:teuthology.orchestra.run.smithi084.stdout: "ready_epoch": 0, 2024-09-20T22:48:39.446 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_started": 0, 2024-09-20T22:48:39.446 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_clean": 0, 2024-09-20T22:48:39.446 INFO:teuthology.orchestra.run.smithi084.stdout: "source_version": "0'0", 2024-09-20T22:48:39.446 INFO:teuthology.orchestra.run.smithi084.stdout: "target_version": "0'0" 2024-09-20T22:48:39.446 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:39.446 INFO:teuthology.orchestra.run.smithi084.stdout: "last_change": "69", 2024-09-20T22:48:39.446 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend": "0", 2024-09-20T22:48:39.446 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-20T22:48:39.446 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_preluminous": "0", 2024-09-20T22:48:39.446 INFO:teuthology.orchestra.run.smithi084.stdout: "auid": 0, 2024-09-20T22:48:39.447 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_mode": "selfmanaged", 2024-09-20T22:48:39.447 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_seq": 2, 2024-09-20T22:48:39.447 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_epoch": 69, 2024-09-20T22:48:39.447 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_snaps": [], 2024-09-20T22:48:39.447 INFO:teuthology.orchestra.run.smithi084.stdout: "removed_snaps": "[]", 2024-09-20T22:48:39.447 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_bytes": 0, 2024-09-20T22:48:39.447 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_objects": 0, 2024-09-20T22:48:39.447 INFO:teuthology.orchestra.run.smithi084.stdout: "tiers": [], 2024-09-20T22:48:39.447 INFO:teuthology.orchestra.run.smithi084.stdout: "tier_of": -1, 2024-09-20T22:48:39.447 INFO:teuthology.orchestra.run.smithi084.stdout: "read_tier": -1, 2024-09-20T22:48:39.447 INFO:teuthology.orchestra.run.smithi084.stdout: "write_tier": -1, 2024-09-20T22:48:39.447 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_mode": "none", 2024-09-20T22:48:39.447 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_bytes": 0, 2024-09-20T22:48:39.447 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_objects": 0, 2024-09-20T22:48:39.448 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-20T22:48:39.448 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-20T22:48:39.448 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-20T22:48:39.448 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_flush_age": 0, 2024-09-20T22:48:39.448 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_evict_age": 0, 2024-09-20T22:48:39.448 INFO:teuthology.orchestra.run.smithi084.stdout: "erasure_code_profile": "", 2024-09-20T22:48:39.448 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_params": { 2024-09-20T22:48:39.448 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "none" 2024-09-20T22:48:39.448 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:39.448 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_period": 0, 2024-09-20T22:48:39.448 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_count": 0, 2024-09-20T22:48:39.448 INFO:teuthology.orchestra.run.smithi084.stdout: "use_gmt_hitset": true, 2024-09-20T22:48:39.448 INFO:teuthology.orchestra.run.smithi084.stdout: "min_read_recency_for_promote": 0, 2024-09-20T22:48:39.449 INFO:teuthology.orchestra.run.smithi084.stdout: "min_write_recency_for_promote": 0, 2024-09-20T22:48:39.449 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_grade_decay_rate": 0, 2024-09-20T22:48:39.449 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_search_last_n": 0, 2024-09-20T22:48:39.449 INFO:teuthology.orchestra.run.smithi084.stdout: "grade_table": [], 2024-09-20T22:48:39.449 INFO:teuthology.orchestra.run.smithi084.stdout: "stripe_width": 0, 2024-09-20T22:48:39.449 INFO:teuthology.orchestra.run.smithi084.stdout: "expected_num_objects": 0, 2024-09-20T22:48:39.449 INFO:teuthology.orchestra.run.smithi084.stdout: "fast_read": false, 2024-09-20T22:48:39.449 INFO:teuthology.orchestra.run.smithi084.stdout: "options": {}, 2024-09-20T22:48:39.449 INFO:teuthology.orchestra.run.smithi084.stdout: "application_metadata": { 2024-09-20T22:48:39.449 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd": {} 2024-09-20T22:48:39.449 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:39.449 INFO:teuthology.orchestra.run.smithi084.stdout: "read_balance": { 2024-09-20T22:48:39.449 INFO:teuthology.orchestra.run.smithi084.stdout: "score_type": "Fair distribution", 2024-09-20T22:48:39.450 INFO:teuthology.orchestra.run.smithi084.stdout: "score_acting": 2.6500000953674316, 2024-09-20T22:48:39.450 INFO:teuthology.orchestra.run.smithi084.stdout: "score_stable": 2.6500000953674316, 2024-09-20T22:48:39.450 INFO:teuthology.orchestra.run.smithi084.stdout: "optimal_score": 0.87999999523162842, 2024-09-20T22:48:39.450 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_acting": 2.3299999237060547, 2024-09-20T22:48:39.450 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_stable": 2.3299999237060547, 2024-09-20T22:48:39.450 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity_weighted": 1, 2024-09-20T22:48:39.450 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity": 1, 2024-09-20T22:48:39.450 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity_weighted": 1 2024-09-20T22:48:39.450 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:39.450 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:39.450 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:39.450 INFO:teuthology.orchestra.run.smithi084.stdout: "pool": 3, 2024-09-20T22:48:39.450 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_name": ".rgw.root", 2024-09-20T22:48:39.450 INFO:teuthology.orchestra.run.smithi084.stdout: "create_time": "2024-09-20T22:41:36.546389+0000", 2024-09-20T22:48:39.451 INFO:teuthology.orchestra.run.smithi084.stdout: "flags": 1, 2024-09-20T22:48:39.451 INFO:teuthology.orchestra.run.smithi084.stdout: "flags_names": "hashpspool", 2024-09-20T22:48:39.451 INFO:teuthology.orchestra.run.smithi084.stdout: "type": 1, 2024-09-20T22:48:39.451 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 3, 2024-09-20T22:48:39.451 INFO:teuthology.orchestra.run.smithi084.stdout: "min_size": 2, 2024-09-20T22:48:39.451 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_rule": 0, 2024-09-20T22:48:39.451 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_count": 0, 2024-09-20T22:48:39.451 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_target": 0, 2024-09-20T22:48:39.451 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_barrier": 0, 2024-09-20T22:48:39.451 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-20T22:48:39.451 INFO:teuthology.orchestra.run.smithi084.stdout: "is_stretch_pool": false, 2024-09-20T22:48:39.451 INFO:teuthology.orchestra.run.smithi084.stdout: "object_hash": 2, 2024-09-20T22:48:39.451 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_autoscale_mode": "off", 2024-09-20T22:48:39.452 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num": 32, 2024-09-20T22:48:39.452 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num": 32, 2024-09-20T22:48:39.452 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num_target": 32, 2024-09-20T22:48:39.452 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_target": 32, 2024-09-20T22:48:39.452 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_pending": 32, 2024-09-20T22:48:39.452 INFO:teuthology.orchestra.run.smithi084.stdout: "last_pg_merge_meta": { 2024-09-20T22:48:39.452 INFO:teuthology.orchestra.run.smithi084.stdout: "source_pgid": "0.0", 2024-09-20T22:48:39.452 INFO:teuthology.orchestra.run.smithi084.stdout: "ready_epoch": 0, 2024-09-20T22:48:39.452 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_started": 0, 2024-09-20T22:48:39.453 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_clean": 0, 2024-09-20T22:48:39.453 INFO:teuthology.orchestra.run.smithi084.stdout: "source_version": "0'0", 2024-09-20T22:48:39.453 INFO:teuthology.orchestra.run.smithi084.stdout: "target_version": "0'0" 2024-09-20T22:48:39.453 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:39.453 INFO:teuthology.orchestra.run.smithi084.stdout: "last_change": "68", 2024-09-20T22:48:39.453 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend": "0", 2024-09-20T22:48:39.453 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-20T22:48:39.454 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_preluminous": "0", 2024-09-20T22:48:39.454 INFO:teuthology.orchestra.run.smithi084.stdout: "auid": 0, 2024-09-20T22:48:39.454 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_mode": "selfmanaged", 2024-09-20T22:48:39.454 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_seq": 0, 2024-09-20T22:48:39.454 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_epoch": 0, 2024-09-20T22:48:39.454 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_snaps": [], 2024-09-20T22:48:39.454 INFO:teuthology.orchestra.run.smithi084.stdout: "removed_snaps": "[]", 2024-09-20T22:48:39.454 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_bytes": 0, 2024-09-20T22:48:39.454 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_objects": 0, 2024-09-20T22:48:39.454 INFO:teuthology.orchestra.run.smithi084.stdout: "tiers": [], 2024-09-20T22:48:39.454 INFO:teuthology.orchestra.run.smithi084.stdout: "tier_of": -1, 2024-09-20T22:48:39.454 INFO:teuthology.orchestra.run.smithi084.stdout: "read_tier": -1, 2024-09-20T22:48:39.454 INFO:teuthology.orchestra.run.smithi084.stdout: "write_tier": -1, 2024-09-20T22:48:39.455 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_mode": "none", 2024-09-20T22:48:39.455 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_bytes": 0, 2024-09-20T22:48:39.455 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_objects": 0, 2024-09-20T22:48:39.455 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-20T22:48:39.455 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-20T22:48:39.455 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-20T22:48:39.455 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_flush_age": 0, 2024-09-20T22:48:39.455 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_evict_age": 0, 2024-09-20T22:48:39.455 INFO:teuthology.orchestra.run.smithi084.stdout: "erasure_code_profile": "", 2024-09-20T22:48:39.455 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_params": { 2024-09-20T22:48:39.455 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "none" 2024-09-20T22:48:39.455 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:39.455 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_period": 0, 2024-09-20T22:48:39.455 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_count": 0, 2024-09-20T22:48:39.455 INFO:teuthology.orchestra.run.smithi084.stdout: "use_gmt_hitset": true, 2024-09-20T22:48:39.456 INFO:teuthology.orchestra.run.smithi084.stdout: "min_read_recency_for_promote": 0, 2024-09-20T22:48:39.456 INFO:teuthology.orchestra.run.smithi084.stdout: "min_write_recency_for_promote": 0, 2024-09-20T22:48:39.456 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_grade_decay_rate": 0, 2024-09-20T22:48:39.456 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_search_last_n": 0, 2024-09-20T22:48:39.456 INFO:teuthology.orchestra.run.smithi084.stdout: "grade_table": [], 2024-09-20T22:48:39.456 INFO:teuthology.orchestra.run.smithi084.stdout: "stripe_width": 0, 2024-09-20T22:48:39.456 INFO:teuthology.orchestra.run.smithi084.stdout: "expected_num_objects": 0, 2024-09-20T22:48:39.456 INFO:teuthology.orchestra.run.smithi084.stdout: "fast_read": false, 2024-09-20T22:48:39.456 INFO:teuthology.orchestra.run.smithi084.stdout: "options": {}, 2024-09-20T22:48:39.456 INFO:teuthology.orchestra.run.smithi084.stdout: "application_metadata": { 2024-09-20T22:48:39.456 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw": {} 2024-09-20T22:48:39.456 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:39.456 INFO:teuthology.orchestra.run.smithi084.stdout: "read_balance": { 2024-09-20T22:48:39.456 INFO:teuthology.orchestra.run.smithi084.stdout: "score_type": "Fair distribution", 2024-09-20T22:48:39.457 INFO:teuthology.orchestra.run.smithi084.stdout: "score_acting": 1.5, 2024-09-20T22:48:39.457 INFO:teuthology.orchestra.run.smithi084.stdout: "score_stable": 1.5, 2024-09-20T22:48:39.457 INFO:teuthology.orchestra.run.smithi084.stdout: "optimal_score": 1, 2024-09-20T22:48:39.457 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_acting": 1.5, 2024-09-20T22:48:39.457 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_stable": 1.5, 2024-09-20T22:48:39.457 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity_weighted": 1, 2024-09-20T22:48:39.457 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity": 1, 2024-09-20T22:48:39.457 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity_weighted": 1 2024-09-20T22:48:39.457 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:39.457 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:39.457 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:39.457 INFO:teuthology.orchestra.run.smithi084.stdout: "pool": 4, 2024-09-20T22:48:39.457 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_name": "default.rgw.log", 2024-09-20T22:48:39.457 INFO:teuthology.orchestra.run.smithi084.stdout: "create_time": "2024-09-20T22:41:38.454640+0000", 2024-09-20T22:48:39.458 INFO:teuthology.orchestra.run.smithi084.stdout: "flags": 1, 2024-09-20T22:48:39.458 INFO:teuthology.orchestra.run.smithi084.stdout: "flags_names": "hashpspool", 2024-09-20T22:48:39.458 INFO:teuthology.orchestra.run.smithi084.stdout: "type": 1, 2024-09-20T22:48:39.458 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 3, 2024-09-20T22:48:39.458 INFO:teuthology.orchestra.run.smithi084.stdout: "min_size": 2, 2024-09-20T22:48:39.458 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_rule": 0, 2024-09-20T22:48:39.458 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_count": 0, 2024-09-20T22:48:39.458 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_target": 0, 2024-09-20T22:48:39.458 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_barrier": 0, 2024-09-20T22:48:39.458 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-20T22:48:39.458 INFO:teuthology.orchestra.run.smithi084.stdout: "is_stretch_pool": false, 2024-09-20T22:48:39.458 INFO:teuthology.orchestra.run.smithi084.stdout: "object_hash": 2, 2024-09-20T22:48:39.458 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_autoscale_mode": "off", 2024-09-20T22:48:39.458 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num": 32, 2024-09-20T22:48:39.459 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num": 32, 2024-09-20T22:48:39.459 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num_target": 32, 2024-09-20T22:48:39.459 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_target": 32, 2024-09-20T22:48:39.459 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_pending": 32, 2024-09-20T22:48:39.459 INFO:teuthology.orchestra.run.smithi084.stdout: "last_pg_merge_meta": { 2024-09-20T22:48:39.459 INFO:teuthology.orchestra.run.smithi084.stdout: "source_pgid": "0.0", 2024-09-20T22:48:39.459 INFO:teuthology.orchestra.run.smithi084.stdout: "ready_epoch": 0, 2024-09-20T22:48:39.459 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_started": 0, 2024-09-20T22:48:39.459 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_clean": 0, 2024-09-20T22:48:39.459 INFO:teuthology.orchestra.run.smithi084.stdout: "source_version": "0'0", 2024-09-20T22:48:39.459 INFO:teuthology.orchestra.run.smithi084.stdout: "target_version": "0'0" 2024-09-20T22:48:39.459 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:39.459 INFO:teuthology.orchestra.run.smithi084.stdout: "last_change": "70", 2024-09-20T22:48:39.460 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend": "0", 2024-09-20T22:48:39.460 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-20T22:48:39.460 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_preluminous": "0", 2024-09-20T22:48:39.460 INFO:teuthology.orchestra.run.smithi084.stdout: "auid": 0, 2024-09-20T22:48:39.460 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_mode": "selfmanaged", 2024-09-20T22:48:39.460 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_seq": 0, 2024-09-20T22:48:39.460 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_epoch": 0, 2024-09-20T22:48:39.460 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_snaps": [], 2024-09-20T22:48:39.460 INFO:teuthology.orchestra.run.smithi084.stdout: "removed_snaps": "[]", 2024-09-20T22:48:39.460 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_bytes": 0, 2024-09-20T22:48:39.460 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_objects": 0, 2024-09-20T22:48:39.460 INFO:teuthology.orchestra.run.smithi084.stdout: "tiers": [], 2024-09-20T22:48:39.460 INFO:teuthology.orchestra.run.smithi084.stdout: "tier_of": -1, 2024-09-20T22:48:39.460 INFO:teuthology.orchestra.run.smithi084.stdout: "read_tier": -1, 2024-09-20T22:48:39.461 INFO:teuthology.orchestra.run.smithi084.stdout: "write_tier": -1, 2024-09-20T22:48:39.461 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_mode": "none", 2024-09-20T22:48:39.461 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_bytes": 0, 2024-09-20T22:48:39.461 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_objects": 0, 2024-09-20T22:48:39.461 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-20T22:48:39.461 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-20T22:48:39.461 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-20T22:48:39.461 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_flush_age": 0, 2024-09-20T22:48:39.461 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_evict_age": 0, 2024-09-20T22:48:39.461 INFO:teuthology.orchestra.run.smithi084.stdout: "erasure_code_profile": "", 2024-09-20T22:48:39.461 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_params": { 2024-09-20T22:48:39.461 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "none" 2024-09-20T22:48:39.461 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:39.461 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_period": 0, 2024-09-20T22:48:39.462 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_count": 0, 2024-09-20T22:48:39.462 INFO:teuthology.orchestra.run.smithi084.stdout: "use_gmt_hitset": true, 2024-09-20T22:48:39.462 INFO:teuthology.orchestra.run.smithi084.stdout: "min_read_recency_for_promote": 0, 2024-09-20T22:48:39.462 INFO:teuthology.orchestra.run.smithi084.stdout: "min_write_recency_for_promote": 0, 2024-09-20T22:48:39.462 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_grade_decay_rate": 0, 2024-09-20T22:48:39.462 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_search_last_n": 0, 2024-09-20T22:48:39.462 INFO:teuthology.orchestra.run.smithi084.stdout: "grade_table": [], 2024-09-20T22:48:39.462 INFO:teuthology.orchestra.run.smithi084.stdout: "stripe_width": 0, 2024-09-20T22:48:39.462 INFO:teuthology.orchestra.run.smithi084.stdout: "expected_num_objects": 0, 2024-09-20T22:48:39.462 INFO:teuthology.orchestra.run.smithi084.stdout: "fast_read": false, 2024-09-20T22:48:39.462 INFO:teuthology.orchestra.run.smithi084.stdout: "options": {}, 2024-09-20T22:48:39.462 INFO:teuthology.orchestra.run.smithi084.stdout: "application_metadata": { 2024-09-20T22:48:39.462 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw": {} 2024-09-20T22:48:39.462 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:39.463 INFO:teuthology.orchestra.run.smithi084.stdout: "read_balance": { 2024-09-20T22:48:39.463 INFO:teuthology.orchestra.run.smithi084.stdout: "score_type": "Fair distribution", 2024-09-20T22:48:39.463 INFO:teuthology.orchestra.run.smithi084.stdout: "score_acting": 2.25, 2024-09-20T22:48:39.463 INFO:teuthology.orchestra.run.smithi084.stdout: "score_stable": 2.25, 2024-09-20T22:48:39.463 INFO:teuthology.orchestra.run.smithi084.stdout: "optimal_score": 1, 2024-09-20T22:48:39.463 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_acting": 2.25, 2024-09-20T22:48:39.463 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_stable": 2.25, 2024-09-20T22:48:39.463 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity_weighted": 1, 2024-09-20T22:48:39.463 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity": 1, 2024-09-20T22:48:39.463 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity_weighted": 1 2024-09-20T22:48:39.463 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:39.463 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:39.463 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:39.464 INFO:teuthology.orchestra.run.smithi084.stdout: "pool": 5, 2024-09-20T22:48:39.464 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_name": "default.rgw.control", 2024-09-20T22:48:39.464 INFO:teuthology.orchestra.run.smithi084.stdout: "create_time": "2024-09-20T22:41:40.407129+0000", 2024-09-20T22:48:39.464 INFO:teuthology.orchestra.run.smithi084.stdout: "flags": 1, 2024-09-20T22:48:39.464 INFO:teuthology.orchestra.run.smithi084.stdout: "flags_names": "hashpspool", 2024-09-20T22:48:39.464 INFO:teuthology.orchestra.run.smithi084.stdout: "type": 1, 2024-09-20T22:48:39.464 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 3, 2024-09-20T22:48:39.464 INFO:teuthology.orchestra.run.smithi084.stdout: "min_size": 2, 2024-09-20T22:48:39.464 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_rule": 0, 2024-09-20T22:48:39.464 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_count": 0, 2024-09-20T22:48:39.464 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_target": 0, 2024-09-20T22:48:39.464 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_barrier": 0, 2024-09-20T22:48:39.464 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-20T22:48:39.464 INFO:teuthology.orchestra.run.smithi084.stdout: "is_stretch_pool": false, 2024-09-20T22:48:39.465 INFO:teuthology.orchestra.run.smithi084.stdout: "object_hash": 2, 2024-09-20T22:48:39.465 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_autoscale_mode": "off", 2024-09-20T22:48:39.465 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num": 32, 2024-09-20T22:48:39.465 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num": 32, 2024-09-20T22:48:39.465 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num_target": 32, 2024-09-20T22:48:39.465 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_target": 32, 2024-09-20T22:48:39.465 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_pending": 32, 2024-09-20T22:48:39.465 INFO:teuthology.orchestra.run.smithi084.stdout: "last_pg_merge_meta": { 2024-09-20T22:48:39.465 INFO:teuthology.orchestra.run.smithi084.stdout: "source_pgid": "0.0", 2024-09-20T22:48:39.465 INFO:teuthology.orchestra.run.smithi084.stdout: "ready_epoch": 0, 2024-09-20T22:48:39.465 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_started": 0, 2024-09-20T22:48:39.465 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_clean": 0, 2024-09-20T22:48:39.465 INFO:teuthology.orchestra.run.smithi084.stdout: "source_version": "0'0", 2024-09-20T22:48:39.465 INFO:teuthology.orchestra.run.smithi084.stdout: "target_version": "0'0" 2024-09-20T22:48:39.466 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:39.466 INFO:teuthology.orchestra.run.smithi084.stdout: "last_change": "72", 2024-09-20T22:48:39.466 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend": "0", 2024-09-20T22:48:39.466 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-20T22:48:39.466 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_preluminous": "0", 2024-09-20T22:48:39.466 INFO:teuthology.orchestra.run.smithi084.stdout: "auid": 0, 2024-09-20T22:48:39.466 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_mode": "selfmanaged", 2024-09-20T22:48:39.466 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_seq": 0, 2024-09-20T22:48:39.466 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_epoch": 0, 2024-09-20T22:48:39.466 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_snaps": [], 2024-09-20T22:48:39.466 INFO:teuthology.orchestra.run.smithi084.stdout: "removed_snaps": "[]", 2024-09-20T22:48:39.466 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_bytes": 0, 2024-09-20T22:48:39.466 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_objects": 0, 2024-09-20T22:48:39.466 INFO:teuthology.orchestra.run.smithi084.stdout: "tiers": [], 2024-09-20T22:48:39.467 INFO:teuthology.orchestra.run.smithi084.stdout: "tier_of": -1, 2024-09-20T22:48:39.467 INFO:teuthology.orchestra.run.smithi084.stdout: "read_tier": -1, 2024-09-20T22:48:39.467 INFO:teuthology.orchestra.run.smithi084.stdout: "write_tier": -1, 2024-09-20T22:48:39.467 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_mode": "none", 2024-09-20T22:48:39.467 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_bytes": 0, 2024-09-20T22:48:39.467 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_objects": 0, 2024-09-20T22:48:39.467 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-20T22:48:39.467 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-20T22:48:39.467 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-20T22:48:39.467 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_flush_age": 0, 2024-09-20T22:48:39.467 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_evict_age": 0, 2024-09-20T22:48:39.467 INFO:teuthology.orchestra.run.smithi084.stdout: "erasure_code_profile": "", 2024-09-20T22:48:39.467 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_params": { 2024-09-20T22:48:39.467 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "none" 2024-09-20T22:48:39.468 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:39.468 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_period": 0, 2024-09-20T22:48:39.468 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_count": 0, 2024-09-20T22:48:39.468 INFO:teuthology.orchestra.run.smithi084.stdout: "use_gmt_hitset": true, 2024-09-20T22:48:39.468 INFO:teuthology.orchestra.run.smithi084.stdout: "min_read_recency_for_promote": 0, 2024-09-20T22:48:39.468 INFO:teuthology.orchestra.run.smithi084.stdout: "min_write_recency_for_promote": 0, 2024-09-20T22:48:39.468 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_grade_decay_rate": 0, 2024-09-20T22:48:39.468 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_search_last_n": 0, 2024-09-20T22:48:39.468 INFO:teuthology.orchestra.run.smithi084.stdout: "grade_table": [], 2024-09-20T22:48:39.468 INFO:teuthology.orchestra.run.smithi084.stdout: "stripe_width": 0, 2024-09-20T22:48:39.468 INFO:teuthology.orchestra.run.smithi084.stdout: "expected_num_objects": 0, 2024-09-20T22:48:39.468 INFO:teuthology.orchestra.run.smithi084.stdout: "fast_read": false, 2024-09-20T22:48:39.468 INFO:teuthology.orchestra.run.smithi084.stdout: "options": {}, 2024-09-20T22:48:39.468 INFO:teuthology.orchestra.run.smithi084.stdout: "application_metadata": { 2024-09-20T22:48:39.469 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw": {} 2024-09-20T22:48:39.469 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:39.469 INFO:teuthology.orchestra.run.smithi084.stdout: "read_balance": { 2024-09-20T22:48:39.469 INFO:teuthology.orchestra.run.smithi084.stdout: "score_type": "Fair distribution", 2024-09-20T22:48:39.469 INFO:teuthology.orchestra.run.smithi084.stdout: "score_acting": 1.25, 2024-09-20T22:48:39.469 INFO:teuthology.orchestra.run.smithi084.stdout: "score_stable": 1.25, 2024-09-20T22:48:39.469 INFO:teuthology.orchestra.run.smithi084.stdout: "optimal_score": 1, 2024-09-20T22:48:39.469 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_acting": 1.25, 2024-09-20T22:48:39.469 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_stable": 1.25, 2024-09-20T22:48:39.469 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity_weighted": 1, 2024-09-20T22:48:39.469 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity": 1, 2024-09-20T22:48:39.469 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity_weighted": 1 2024-09-20T22:48:39.469 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:39.469 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:39.470 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:39.470 INFO:teuthology.orchestra.run.smithi084.stdout: "pool": 6, 2024-09-20T22:48:39.470 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_name": "default.rgw.meta", 2024-09-20T22:48:39.470 INFO:teuthology.orchestra.run.smithi084.stdout: "create_time": "2024-09-20T22:41:42.476716+0000", 2024-09-20T22:48:39.470 INFO:teuthology.orchestra.run.smithi084.stdout: "flags": 1, 2024-09-20T22:48:39.470 INFO:teuthology.orchestra.run.smithi084.stdout: "flags_names": "hashpspool", 2024-09-20T22:48:39.470 INFO:teuthology.orchestra.run.smithi084.stdout: "type": 1, 2024-09-20T22:48:39.470 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 3, 2024-09-20T22:48:39.470 INFO:teuthology.orchestra.run.smithi084.stdout: "min_size": 2, 2024-09-20T22:48:39.470 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_rule": 0, 2024-09-20T22:48:39.470 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_count": 0, 2024-09-20T22:48:39.471 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_target": 0, 2024-09-20T22:48:39.471 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_barrier": 0, 2024-09-20T22:48:39.471 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-20T22:48:39.471 INFO:teuthology.orchestra.run.smithi084.stdout: "is_stretch_pool": false, 2024-09-20T22:48:39.471 INFO:teuthology.orchestra.run.smithi084.stdout: "object_hash": 2, 2024-09-20T22:48:39.471 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_autoscale_mode": "off", 2024-09-20T22:48:39.471 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num": 32, 2024-09-20T22:48:39.471 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num": 32, 2024-09-20T22:48:39.471 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num_target": 32, 2024-09-20T22:48:39.471 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_target": 32, 2024-09-20T22:48:39.471 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_pending": 32, 2024-09-20T22:48:39.471 INFO:teuthology.orchestra.run.smithi084.stdout: "last_pg_merge_meta": { 2024-09-20T22:48:39.471 INFO:teuthology.orchestra.run.smithi084.stdout: "source_pgid": "0.0", 2024-09-20T22:48:39.471 INFO:teuthology.orchestra.run.smithi084.stdout: "ready_epoch": 0, 2024-09-20T22:48:39.472 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_started": 0, 2024-09-20T22:48:39.472 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_clean": 0, 2024-09-20T22:48:39.472 INFO:teuthology.orchestra.run.smithi084.stdout: "source_version": "0'0", 2024-09-20T22:48:39.472 INFO:teuthology.orchestra.run.smithi084.stdout: "target_version": "0'0" 2024-09-20T22:48:39.472 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:39.472 INFO:teuthology.orchestra.run.smithi084.stdout: "last_change": "74", 2024-09-20T22:48:39.472 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend": "0", 2024-09-20T22:48:39.472 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-20T22:48:39.472 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_preluminous": "0", 2024-09-20T22:48:39.472 INFO:teuthology.orchestra.run.smithi084.stdout: "auid": 0, 2024-09-20T22:48:39.472 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_mode": "selfmanaged", 2024-09-20T22:48:39.472 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_seq": 0, 2024-09-20T22:48:39.472 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_epoch": 0, 2024-09-20T22:48:39.472 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_snaps": [], 2024-09-20T22:48:39.473 INFO:teuthology.orchestra.run.smithi084.stdout: "removed_snaps": "[]", 2024-09-20T22:48:39.473 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_bytes": 0, 2024-09-20T22:48:39.473 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_objects": 0, 2024-09-20T22:48:39.473 INFO:teuthology.orchestra.run.smithi084.stdout: "tiers": [], 2024-09-20T22:48:39.473 INFO:teuthology.orchestra.run.smithi084.stdout: "tier_of": -1, 2024-09-20T22:48:39.473 INFO:teuthology.orchestra.run.smithi084.stdout: "read_tier": -1, 2024-09-20T22:48:39.473 INFO:teuthology.orchestra.run.smithi084.stdout: "write_tier": -1, 2024-09-20T22:48:39.473 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_mode": "none", 2024-09-20T22:48:39.473 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_bytes": 0, 2024-09-20T22:48:39.473 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_objects": 0, 2024-09-20T22:48:39.473 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-20T22:48:39.473 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-20T22:48:39.473 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-20T22:48:39.473 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_flush_age": 0, 2024-09-20T22:48:39.474 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_evict_age": 0, 2024-09-20T22:48:39.474 INFO:teuthology.orchestra.run.smithi084.stdout: "erasure_code_profile": "", 2024-09-20T22:48:39.474 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_params": { 2024-09-20T22:48:39.474 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "none" 2024-09-20T22:48:39.474 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:39.474 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_period": 0, 2024-09-20T22:48:39.474 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_count": 0, 2024-09-20T22:48:39.474 INFO:teuthology.orchestra.run.smithi084.stdout: "use_gmt_hitset": true, 2024-09-20T22:48:39.474 INFO:teuthology.orchestra.run.smithi084.stdout: "min_read_recency_for_promote": 0, 2024-09-20T22:48:39.474 INFO:teuthology.orchestra.run.smithi084.stdout: "min_write_recency_for_promote": 0, 2024-09-20T22:48:39.474 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_grade_decay_rate": 0, 2024-09-20T22:48:39.474 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_search_last_n": 0, 2024-09-20T22:48:39.474 INFO:teuthology.orchestra.run.smithi084.stdout: "grade_table": [], 2024-09-20T22:48:39.474 INFO:teuthology.orchestra.run.smithi084.stdout: "stripe_width": 0, 2024-09-20T22:48:39.475 INFO:teuthology.orchestra.run.smithi084.stdout: "expected_num_objects": 0, 2024-09-20T22:48:39.475 INFO:teuthology.orchestra.run.smithi084.stdout: "fast_read": false, 2024-09-20T22:48:39.475 INFO:teuthology.orchestra.run.smithi084.stdout: "options": { 2024-09-20T22:48:39.475 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_autoscale_bias": 4 2024-09-20T22:48:39.475 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:39.475 INFO:teuthology.orchestra.run.smithi084.stdout: "application_metadata": { 2024-09-20T22:48:39.475 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw": {} 2024-09-20T22:48:39.475 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:39.475 INFO:teuthology.orchestra.run.smithi084.stdout: "read_balance": { 2024-09-20T22:48:39.475 INFO:teuthology.orchestra.run.smithi084.stdout: "score_type": "Fair distribution", 2024-09-20T22:48:39.475 INFO:teuthology.orchestra.run.smithi084.stdout: "score_acting": 1.75, 2024-09-20T22:48:39.475 INFO:teuthology.orchestra.run.smithi084.stdout: "score_stable": 1.75, 2024-09-20T22:48:39.475 INFO:teuthology.orchestra.run.smithi084.stdout: "optimal_score": 1, 2024-09-20T22:48:39.475 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_acting": 1.75, 2024-09-20T22:48:39.475 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_stable": 1.75, 2024-09-20T22:48:39.476 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity_weighted": 1, 2024-09-20T22:48:39.476 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity": 1, 2024-09-20T22:48:39.476 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity_weighted": 1 2024-09-20T22:48:39.476 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:39.476 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:39.476 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:39.476 INFO:teuthology.orchestra.run.smithi084.stdout: "pool": 7, 2024-09-20T22:48:39.476 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_name": "unique_pool_0", 2024-09-20T22:48:39.476 INFO:teuthology.orchestra.run.smithi084.stdout: "create_time": "2024-09-20T22:42:51.354358+0000", 2024-09-20T22:48:39.476 INFO:teuthology.orchestra.run.smithi084.stdout: "flags": 8193, 2024-09-20T22:48:39.476 INFO:teuthology.orchestra.run.smithi084.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-20T22:48:39.476 INFO:teuthology.orchestra.run.smithi084.stdout: "type": 1, 2024-09-20T22:48:39.476 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 3, 2024-09-20T22:48:39.477 INFO:teuthology.orchestra.run.smithi084.stdout: "min_size": 2, 2024-09-20T22:48:39.477 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_rule": 0, 2024-09-20T22:48:39.477 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_count": 0, 2024-09-20T22:48:39.477 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_target": 0, 2024-09-20T22:48:39.477 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_barrier": 0, 2024-09-20T22:48:39.477 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-20T22:48:39.477 INFO:teuthology.orchestra.run.smithi084.stdout: "is_stretch_pool": false, 2024-09-20T22:48:39.477 INFO:teuthology.orchestra.run.smithi084.stdout: "object_hash": 2, 2024-09-20T22:48:39.477 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_autoscale_mode": "off", 2024-09-20T22:48:39.477 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num": 24, 2024-09-20T22:48:39.477 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num": 16, 2024-09-20T22:48:39.477 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num_target": 16, 2024-09-20T22:48:39.477 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_target": 16, 2024-09-20T22:48:39.477 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_pending": 24, 2024-09-20T22:48:39.478 INFO:teuthology.orchestra.run.smithi084.stdout: "last_pg_merge_meta": { 2024-09-20T22:48:39.478 INFO:teuthology.orchestra.run.smithi084.stdout: "source_pgid": "7.18", 2024-09-20T22:48:39.478 INFO:teuthology.orchestra.run.smithi084.stdout: "ready_epoch": 430, 2024-09-20T22:48:39.478 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_started": 429, 2024-09-20T22:48:39.478 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_clean": 429, 2024-09-20T22:48:39.478 INFO:teuthology.orchestra.run.smithi084.stdout: "source_version": "426'268", 2024-09-20T22:48:39.478 INFO:teuthology.orchestra.run.smithi084.stdout: "target_version": "425'136" 2024-09-20T22:48:39.478 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:39.478 INFO:teuthology.orchestra.run.smithi084.stdout: "last_change": "439", 2024-09-20T22:48:39.478 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend": "0", 2024-09-20T22:48:39.478 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_prenautilus": "430", 2024-09-20T22:48:39.478 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_preluminous": "428", 2024-09-20T22:48:39.478 INFO:teuthology.orchestra.run.smithi084.stdout: "auid": 0, 2024-09-20T22:48:39.479 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_mode": "selfmanaged", 2024-09-20T22:48:39.479 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_seq": 180, 2024-09-20T22:48:39.479 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_epoch": 439, 2024-09-20T22:48:39.479 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_snaps": [], 2024-09-20T22:48:39.479 INFO:teuthology.orchestra.run.smithi084.stdout: "removed_snaps": "[]", 2024-09-20T22:48:39.479 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_bytes": 0, 2024-09-20T22:48:39.479 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_objects": 0, 2024-09-20T22:48:39.479 INFO:teuthology.orchestra.run.smithi084.stdout: "tiers": [], 2024-09-20T22:48:39.479 INFO:teuthology.orchestra.run.smithi084.stdout: "tier_of": -1, 2024-09-20T22:48:39.479 INFO:teuthology.orchestra.run.smithi084.stdout: "read_tier": -1, 2024-09-20T22:48:39.479 INFO:teuthology.orchestra.run.smithi084.stdout: "write_tier": -1, 2024-09-20T22:48:39.479 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_mode": "none", 2024-09-20T22:48:39.480 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_bytes": 0, 2024-09-20T22:48:39.480 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_objects": 0, 2024-09-20T22:48:39.480 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-20T22:48:39.480 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-20T22:48:39.480 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-20T22:48:39.480 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_flush_age": 0, 2024-09-20T22:48:39.480 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_evict_age": 0, 2024-09-20T22:48:39.480 INFO:teuthology.orchestra.run.smithi084.stdout: "erasure_code_profile": "", 2024-09-20T22:48:39.480 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_params": { 2024-09-20T22:48:39.480 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "none" 2024-09-20T22:48:39.480 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:39.480 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_period": 0, 2024-09-20T22:48:39.480 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_count": 0, 2024-09-20T22:48:39.481 INFO:teuthology.orchestra.run.smithi084.stdout: "use_gmt_hitset": true, 2024-09-20T22:48:39.481 INFO:teuthology.orchestra.run.smithi084.stdout: "min_read_recency_for_promote": 0, 2024-09-20T22:48:39.481 INFO:teuthology.orchestra.run.smithi084.stdout: "min_write_recency_for_promote": 0, 2024-09-20T22:48:39.481 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_grade_decay_rate": 0, 2024-09-20T22:48:39.481 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_search_last_n": 0, 2024-09-20T22:48:39.481 INFO:teuthology.orchestra.run.smithi084.stdout: "grade_table": [], 2024-09-20T22:48:39.481 INFO:teuthology.orchestra.run.smithi084.stdout: "stripe_width": 0, 2024-09-20T22:48:39.481 INFO:teuthology.orchestra.run.smithi084.stdout: "expected_num_objects": 0, 2024-09-20T22:48:39.481 INFO:teuthology.orchestra.run.smithi084.stdout: "fast_read": false, 2024-09-20T22:48:39.481 INFO:teuthology.orchestra.run.smithi084.stdout: "options": {}, 2024-09-20T22:48:39.481 INFO:teuthology.orchestra.run.smithi084.stdout: "application_metadata": { 2024-09-20T22:48:39.481 INFO:teuthology.orchestra.run.smithi084.stdout: "rados": {} 2024-09-20T22:48:39.481 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:39.481 INFO:teuthology.orchestra.run.smithi084.stdout: "read_balance": { 2024-09-20T22:48:39.482 INFO:teuthology.orchestra.run.smithi084.stdout: "score_type": "Fair distribution", 2024-09-20T22:48:39.482 INFO:teuthology.orchestra.run.smithi084.stdout: "score_acting": 2, 2024-09-20T22:48:39.482 INFO:teuthology.orchestra.run.smithi084.stdout: "score_stable": 2, 2024-09-20T22:48:39.482 INFO:teuthology.orchestra.run.smithi084.stdout: "optimal_score": 1, 2024-09-20T22:48:39.482 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_acting": 2, 2024-09-20T22:48:39.482 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_stable": 2, 2024-09-20T22:48:39.482 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity_weighted": 1, 2024-09-20T22:48:39.482 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity": 1, 2024-09-20T22:48:39.482 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity_weighted": 1 2024-09-20T22:48:39.482 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:39.482 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:39.482 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2024-09-20T22:48:39.482 INFO:teuthology.orchestra.run.smithi084.stdout: "osds": [ 2024-09-20T22:48:39.482 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:39.483 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 0, 2024-09-20T22:48:39.483 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "ea36a489-6f91-4b2f-b784-b90506b9805d", 2024-09-20T22:48:39.483 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-09-20T22:48:39.483 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-09-20T22:48:39.483 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-09-20T22:48:39.483 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-09-20T22:48:39.483 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-09-20T22:48:39.483 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-09-20T22:48:39.483 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 8, 2024-09-20T22:48:39.483 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 434, 2024-09-20T22:48:39.483 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-09-20T22:48:39.483 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-09-20T22:48:39.483 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-09-20T22:48:39.483 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:48:39.484 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:39.484 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:48:39.484 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6801", 2024-09-20T22:48:39.484 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 268084626 2024-09-20T22:48:39.484 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:39.484 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:48:39.484 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:39.484 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-09-20T22:48:39.484 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:48:39.484 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:39.484 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:48:39.484 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6802", 2024-09-20T22:48:39.484 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 268084626 2024-09-20T22:48:39.484 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:39.485 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:48:39.485 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:39.485 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-09-20T22:48:39.485 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:48:39.485 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:39.485 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:48:39.485 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6804", 2024-09-20T22:48:39.485 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 268084626 2024-09-20T22:48:39.485 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:39.485 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:48:39.485 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:39.485 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-09-20T22:48:39.485 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:48:39.485 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:39.486 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:48:39.486 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6803", 2024-09-20T22:48:39.486 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 268084626 2024-09-20T22:48:39.486 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:39.486 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:48:39.486 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:39.486 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.84:6801/268084626", 2024-09-20T22:48:39.486 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.84:6802/268084626", 2024-09-20T22:48:39.486 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.84:6804/268084626", 2024-09-20T22:48:39.486 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.84:6803/268084626", 2024-09-20T22:48:39.486 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-09-20T22:48:39.486 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-09-20T22:48:39.486 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-09-20T22:48:39.486 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:48:39.487 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:39.487 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:39.487 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 1, 2024-09-20T22:48:39.487 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "c8649d2f-27ef-47f8-bc94-f1cd9e62bdb8", 2024-09-20T22:48:39.487 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-09-20T22:48:39.487 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-09-20T22:48:39.487 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-09-20T22:48:39.487 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-09-20T22:48:39.487 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-09-20T22:48:39.487 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-09-20T22:48:39.487 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 13, 2024-09-20T22:48:39.488 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 420, 2024-09-20T22:48:39.488 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-09-20T22:48:39.488 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-09-20T22:48:39.488 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-09-20T22:48:39.488 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:48:39.488 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:39.488 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:48:39.488 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6805", 2024-09-20T22:48:39.488 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 418374136 2024-09-20T22:48:39.488 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:39.488 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:48:39.488 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:39.488 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-09-20T22:48:39.488 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:48:39.488 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:39.489 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:48:39.489 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6806", 2024-09-20T22:48:39.489 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 418374136 2024-09-20T22:48:39.489 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:39.489 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:48:39.489 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:39.489 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-09-20T22:48:39.489 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:48:39.489 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:39.489 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:48:39.489 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6808", 2024-09-20T22:48:39.489 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 418374136 2024-09-20T22:48:39.489 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:39.490 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:48:39.490 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:39.490 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-09-20T22:48:39.490 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:48:39.490 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:39.490 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:48:39.490 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6807", 2024-09-20T22:48:39.490 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 418374136 2024-09-20T22:48:39.490 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:39.490 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:48:39.490 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:39.490 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.84:6805/418374136", 2024-09-20T22:48:39.490 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.84:6806/418374136", 2024-09-20T22:48:39.490 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.84:6808/418374136", 2024-09-20T22:48:39.491 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.84:6807/418374136", 2024-09-20T22:48:39.491 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-09-20T22:48:39.491 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-09-20T22:48:39.491 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-09-20T22:48:39.491 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:48:39.491 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:39.491 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:39.491 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 2, 2024-09-20T22:48:39.491 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "c728eb48-a0f0-4de4-a651-7f3bec867280", 2024-09-20T22:48:39.491 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-09-20T22:48:39.491 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-09-20T22:48:39.491 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-09-20T22:48:39.491 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-09-20T22:48:39.492 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-09-20T22:48:39.492 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-09-20T22:48:39.492 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 18, 2024-09-20T22:48:39.492 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 430, 2024-09-20T22:48:39.492 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-09-20T22:48:39.492 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-09-20T22:48:39.492 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-09-20T22:48:39.492 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:48:39.492 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:39.492 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:48:39.492 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6809", 2024-09-20T22:48:39.492 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3977168611 2024-09-20T22:48:39.492 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:39.492 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:48:39.493 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:39.493 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-09-20T22:48:39.493 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:48:39.493 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:39.493 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:48:39.493 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6810", 2024-09-20T22:48:39.493 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3977168611 2024-09-20T22:48:39.493 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:39.493 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:48:39.493 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:39.493 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-09-20T22:48:39.493 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:48:39.493 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:39.494 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:48:39.494 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6812", 2024-09-20T22:48:39.494 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3977168611 2024-09-20T22:48:39.494 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:39.494 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:48:39.494 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:39.494 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-09-20T22:48:39.494 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:48:39.494 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:39.494 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:48:39.494 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6811", 2024-09-20T22:48:39.494 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3977168611 2024-09-20T22:48:39.494 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:39.495 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:48:39.495 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:39.495 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.84:6809/3977168611", 2024-09-20T22:48:39.495 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.84:6810/3977168611", 2024-09-20T22:48:39.495 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.84:6812/3977168611", 2024-09-20T22:48:39.495 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.84:6811/3977168611", 2024-09-20T22:48:39.495 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-09-20T22:48:39.495 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-09-20T22:48:39.495 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-09-20T22:48:39.495 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:48:39.495 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:39.495 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:39.495 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 3, 2024-09-20T22:48:39.495 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "573f44eb-0853-4630-a7d8-9caaeb256f9b", 2024-09-20T22:48:39.495 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-09-20T22:48:39.496 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-09-20T22:48:39.496 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-09-20T22:48:39.496 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-09-20T22:48:39.496 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-09-20T22:48:39.496 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-09-20T22:48:39.496 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 25, 2024-09-20T22:48:39.496 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 421, 2024-09-20T22:48:39.496 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-09-20T22:48:39.496 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-09-20T22:48:39.496 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-09-20T22:48:39.496 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:48:39.496 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:39.496 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:48:39.496 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6813", 2024-09-20T22:48:39.496 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2192239945 2024-09-20T22:48:39.496 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:39.497 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:48:39.497 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:39.497 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-09-20T22:48:39.497 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:48:39.497 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:39.497 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:48:39.497 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6814", 2024-09-20T22:48:39.497 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2192239945 2024-09-20T22:48:39.497 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:39.497 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:48:39.497 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:39.497 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-09-20T22:48:39.497 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:48:39.497 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:39.498 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:48:39.498 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6816", 2024-09-20T22:48:39.498 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2192239945 2024-09-20T22:48:39.498 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:39.498 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:48:39.498 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:39.498 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-09-20T22:48:39.498 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:48:39.498 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:39.498 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:48:39.498 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6815", 2024-09-20T22:48:39.498 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2192239945 2024-09-20T22:48:39.498 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:39.498 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:48:39.499 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:39.499 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.84:6813/2192239945", 2024-09-20T22:48:39.499 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.84:6814/2192239945", 2024-09-20T22:48:39.499 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.84:6816/2192239945", 2024-09-20T22:48:39.499 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.84:6815/2192239945", 2024-09-20T22:48:39.499 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-09-20T22:48:39.499 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-09-20T22:48:39.499 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-09-20T22:48:39.499 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:48:39.499 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:39.499 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:39.499 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 4, 2024-09-20T22:48:39.499 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "a52c3b94-c5f3-490f-8ac8-09061b9b2abb", 2024-09-20T22:48:39.500 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-09-20T22:48:39.500 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-09-20T22:48:39.500 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-09-20T22:48:39.500 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-09-20T22:48:39.500 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-09-20T22:48:39.500 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-09-20T22:48:39.500 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 31, 2024-09-20T22:48:39.500 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 418, 2024-09-20T22:48:39.500 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-09-20T22:48:39.500 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-09-20T22:48:39.500 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-09-20T22:48:39.500 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:48:39.500 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:39.500 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:48:39.501 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.154:6800", 2024-09-20T22:48:39.501 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1793136002 2024-09-20T22:48:39.501 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:39.501 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:48:39.501 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:39.501 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-09-20T22:48:39.501 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:48:39.501 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:39.501 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:48:39.501 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.154:6801", 2024-09-20T22:48:39.501 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1793136002 2024-09-20T22:48:39.501 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:39.501 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:48:39.501 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:39.502 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-09-20T22:48:39.502 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:48:39.502 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:39.502 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:48:39.502 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.154:6803", 2024-09-20T22:48:39.502 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1793136002 2024-09-20T22:48:39.502 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:39.502 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:48:39.502 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:39.502 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-09-20T22:48:39.502 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:48:39.502 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:39.502 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:48:39.503 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.154:6802", 2024-09-20T22:48:39.503 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1793136002 2024-09-20T22:48:39.503 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:39.503 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:48:39.503 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:39.503 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.154:6800/1793136002", 2024-09-20T22:48:39.503 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.154:6801/1793136002", 2024-09-20T22:48:39.503 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.154:6803/1793136002", 2024-09-20T22:48:39.503 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.154:6802/1793136002", 2024-09-20T22:48:39.503 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-09-20T22:48:39.503 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-09-20T22:48:39.503 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-09-20T22:48:39.503 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:48:39.503 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:39.504 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:39.504 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 5, 2024-09-20T22:48:39.504 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "4fa0dc8e-25cc-46cd-96ff-33f20b6faec3", 2024-09-20T22:48:39.504 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-09-20T22:48:39.504 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-09-20T22:48:39.504 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-09-20T22:48:39.504 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-09-20T22:48:39.504 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-09-20T22:48:39.504 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-09-20T22:48:39.504 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 40, 2024-09-20T22:48:39.504 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 421, 2024-09-20T22:48:39.504 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-09-20T22:48:39.504 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-09-20T22:48:39.505 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-09-20T22:48:39.505 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:48:39.505 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:39.505 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:48:39.505 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.154:6804", 2024-09-20T22:48:39.505 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 493899927 2024-09-20T22:48:39.505 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:39.505 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:48:39.505 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:39.505 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-09-20T22:48:39.505 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:48:39.505 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:39.505 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:48:39.505 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.154:6805", 2024-09-20T22:48:39.506 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 493899927 2024-09-20T22:48:39.506 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:39.506 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:48:39.506 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:39.506 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-09-20T22:48:39.506 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:48:39.506 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:39.506 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:48:39.506 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:48:39.507 INFO:journalctl@ceph.iscsi.iscsi.a.smithi154.stdout:Sep 20 22:48:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-iscsi-iscsi-a[59730]: debug there is no tcmu-runner data available 2024-09-20T22:48:39.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:39 smithi154 ceph-mon[33264]: 1.3 scrub starts 2024-09-20T22:48:39.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:39 smithi154 ceph-mon[33264]: 1.3 scrub ok 2024-09-20T22:48:39.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:39 smithi154 ceph-mon[33264]: 4.1e scrub starts 2024-09-20T22:48:39.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:39 smithi154 ceph-mon[33264]: 4.1e scrub ok 2024-09-20T22:48:39.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:39 smithi154 ceph-mon[33264]: pgmap v558: 166 pgs: 1 remapped+peering, 3 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 156 active+clean; 359 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail 2024-09-20T22:48:39.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:39 smithi154 ceph-mon[33264]: 5.13 scrub starts 2024-09-20T22:48:39.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:39 smithi154 ceph-mon[33264]: 5.13 scrub ok 2024-09-20T22:48:39.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:39 smithi154 ceph-mon[33264]: 7.c scrub starts 2024-09-20T22:48:39.508 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:39 smithi154 ceph-mon[33264]: osdmap e439: 8 total, 8 up, 8 in 2024-09-20T22:48:39.509 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.154:6807", 2024-09-20T22:48:39.509 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 493899927 2024-09-20T22:48:39.509 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:39.509 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:48:39.509 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:39.509 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-09-20T22:48:39.509 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:48:39.509 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:39.509 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:48:39.509 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.154:6806", 2024-09-20T22:48:39.509 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 493899927 2024-09-20T22:48:39.510 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:39.510 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:48:39.510 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:39.510 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.154:6804/493899927", 2024-09-20T22:48:39.510 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.154:6805/493899927", 2024-09-20T22:48:39.510 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.154:6807/493899927", 2024-09-20T22:48:39.510 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.154:6806/493899927", 2024-09-20T22:48:39.510 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-09-20T22:48:39.510 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-09-20T22:48:39.510 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-09-20T22:48:39.510 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:48:39.510 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:39.510 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:39.510 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 6, 2024-09-20T22:48:39.511 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "56788d74-810e-4827-a615-958c12f8a368", 2024-09-20T22:48:39.511 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-09-20T22:48:39.511 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-09-20T22:48:39.511 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-09-20T22:48:39.511 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-09-20T22:48:39.511 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-09-20T22:48:39.511 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-09-20T22:48:39.511 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 48, 2024-09-20T22:48:39.511 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 435, 2024-09-20T22:48:39.511 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-09-20T22:48:39.511 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-09-20T22:48:39.511 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-09-20T22:48:39.511 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:48:39.511 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:39.512 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:48:39.512 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.154:6808", 2024-09-20T22:48:39.512 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2375091033 2024-09-20T22:48:39.512 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:39.512 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:48:39.512 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:39.512 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-09-20T22:48:39.512 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:48:39.512 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:39.512 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:48:39.512 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.154:6809", 2024-09-20T22:48:39.512 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2375091033 2024-09-20T22:48:39.512 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:39.512 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:48:39.513 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:39.513 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-09-20T22:48:39.513 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:48:39.513 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:39.513 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:48:39.513 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.154:6811", 2024-09-20T22:48:39.513 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2375091033 2024-09-20T22:48:39.513 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:39.513 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:48:39.513 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:39.513 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-09-20T22:48:39.513 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:48:39.514 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:39.514 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:48:39.514 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.154:6810", 2024-09-20T22:48:39.514 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2375091033 2024-09-20T22:48:39.514 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:39.514 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:48:39.514 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:39.514 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.154:6808/2375091033", 2024-09-20T22:48:39.514 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.154:6809/2375091033", 2024-09-20T22:48:39.514 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.154:6811/2375091033", 2024-09-20T22:48:39.514 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.154:6810/2375091033", 2024-09-20T22:48:39.514 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-09-20T22:48:39.514 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-09-20T22:48:39.514 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-09-20T22:48:39.515 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:48:39.515 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:39.515 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:39.515 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 7, 2024-09-20T22:48:39.515 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "83503727-afc1-4b95-8b96-ecb63f3338c2", 2024-09-20T22:48:39.515 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-09-20T22:48:39.515 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-09-20T22:48:39.515 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-09-20T22:48:39.515 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-09-20T22:48:39.515 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-09-20T22:48:39.515 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-09-20T22:48:39.515 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 59, 2024-09-20T22:48:39.515 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 418, 2024-09-20T22:48:39.515 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-09-20T22:48:39.516 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-09-20T22:48:39.516 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-09-20T22:48:39.516 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:48:39.516 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:39.516 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:48:39.516 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.154:6812", 2024-09-20T22:48:39.516 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3717704491 2024-09-20T22:48:39.516 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:39.516 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:48:39.516 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:39.516 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-09-20T22:48:39.516 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:48:39.516 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:39.517 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:48:39.517 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.154:6813", 2024-09-20T22:48:39.517 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3717704491 2024-09-20T22:48:39.517 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:39.517 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:48:39.517 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:39.517 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-09-20T22:48:39.517 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:48:39.517 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:39.517 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:48:39.517 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.154:6815", 2024-09-20T22:48:39.517 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3717704491 2024-09-20T22:48:39.517 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:39.518 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:48:39.518 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:39.518 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-09-20T22:48:39.518 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:48:39.518 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:39.518 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:48:39.518 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.154:6814", 2024-09-20T22:48:39.518 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3717704491 2024-09-20T22:48:39.518 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:39.518 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:48:39.518 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:39.518 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.154:6812/3717704491", 2024-09-20T22:48:39.518 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.154:6813/3717704491", 2024-09-20T22:48:39.518 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.154:6815/3717704491", 2024-09-20T22:48:39.519 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.154:6814/3717704491", 2024-09-20T22:48:39.519 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-09-20T22:48:39.519 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-09-20T22:48:39.519 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-09-20T22:48:39.519 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:48:39.519 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:39.519 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2024-09-20T22:48:39.519 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_xinfo": [ 2024-09-20T22:48:39.519 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:39.519 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 0, 2024-09-20T22:48:39.519 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-09-20T22:48:39.519 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-09-20T22:48:39.519 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-09-20T22:48:39.520 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540701547738038271, 2024-09-20T22:48:39.520 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-09-20T22:48:39.520 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-09-20T22:47:46.150000+0000", 2024-09-20T22:48:39.520 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-09-20T22:48:39.520 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:39.520 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:39.520 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 1, 2024-09-20T22:48:39.520 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-09-20T22:48:39.520 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-09-20T22:48:39.520 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-09-20T22:48:39.520 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540701547738038271, 2024-09-20T22:48:39.520 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-09-20T22:48:39.520 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-09-20T22:48:06.402076+0000", 2024-09-20T22:48:39.520 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-09-20T22:48:39.521 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:39.521 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:39.521 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 2, 2024-09-20T22:48:39.521 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-09-20T22:48:39.521 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-09-20T22:48:39.521 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-09-20T22:48:39.521 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540701547738038271, 2024-09-20T22:48:39.521 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-09-20T22:48:39.521 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-09-20T22:48:23.610047+0000", 2024-09-20T22:48:39.521 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-09-20T22:48:39.521 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:39.521 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:39.521 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 3, 2024-09-20T22:48:39.522 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-09-20T22:48:39.522 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-09-20T22:48:39.522 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-09-20T22:48:39.522 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540701547738038271, 2024-09-20T22:48:39.522 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-09-20T22:48:39.522 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-09-20T22:47:38.488604+0000", 2024-09-20T22:48:39.522 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-09-20T22:48:39.522 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:39.522 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:39.522 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 4, 2024-09-20T22:48:39.522 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-09-20T22:48:39.522 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-09-20T22:48:39.522 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-09-20T22:48:39.523 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540701547738038271, 2024-09-20T22:48:39.523 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-09-20T22:48:39.523 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-09-20T22:47:43.022898+0000", 2024-09-20T22:48:39.523 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-09-20T22:48:39.523 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:39.523 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:39.523 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 5, 2024-09-20T22:48:39.523 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-09-20T22:48:39.523 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-09-20T22:48:39.523 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-09-20T22:48:39.523 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540701547738038271, 2024-09-20T22:48:39.523 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-09-20T22:48:39.523 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-09-20T22:48:30.072642+0000", 2024-09-20T22:48:39.523 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-09-20T22:48:39.524 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:39.524 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:39.524 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 6, 2024-09-20T22:48:39.524 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-09-20T22:48:39.524 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-09-20T22:48:39.524 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-09-20T22:48:39.524 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540701547738038271, 2024-09-20T22:48:39.524 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-09-20T22:48:39.524 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-09-20T22:48:15.304260+0000", 2024-09-20T22:48:39.524 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-09-20T22:48:39.524 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:39.524 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:39.524 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 7, 2024-09-20T22:48:39.524 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-09-20T22:48:39.525 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-09-20T22:48:39.525 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-09-20T22:48:39.525 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540701547738038271, 2024-09-20T22:48:39.525 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-09-20T22:48:39.525 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-09-20T22:47:39.257733+0000", 2024-09-20T22:48:39.525 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-09-20T22:48:39.525 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:39.525 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2024-09-20T22:48:39.525 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_upmap": [], 2024-09-20T22:48:39.525 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_upmap_items": [ 2024-09-20T22:48:39.525 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:39.525 INFO:teuthology.orchestra.run.smithi084.stdout: "pgid": "3.a", 2024-09-20T22:48:39.525 INFO:teuthology.orchestra.run.smithi084.stdout: "mappings": [ 2024-09-20T22:48:39.525 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:39.526 INFO:teuthology.orchestra.run.smithi084.stdout: "from": 6, 2024-09-20T22:48:39.526 INFO:teuthology.orchestra.run.smithi084.stdout: "to": 2 2024-09-20T22:48:39.526 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:39.526 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:48:39.526 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:39.526 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:39.526 INFO:teuthology.orchestra.run.smithi084.stdout: "pgid": "4.6", 2024-09-20T22:48:39.526 INFO:teuthology.orchestra.run.smithi084.stdout: "mappings": [ 2024-09-20T22:48:39.526 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:39.526 INFO:teuthology.orchestra.run.smithi084.stdout: "from": 1, 2024-09-20T22:48:39.526 INFO:teuthology.orchestra.run.smithi084.stdout: "to": 5 2024-09-20T22:48:39.526 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:39.526 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:48:39.526 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:39.527 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:39.527 INFO:teuthology.orchestra.run.smithi084.stdout: "pgid": "4.11", 2024-09-20T22:48:39.527 INFO:teuthology.orchestra.run.smithi084.stdout: "mappings": [ 2024-09-20T22:48:39.527 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:39.527 INFO:teuthology.orchestra.run.smithi084.stdout: "from": 3, 2024-09-20T22:48:39.527 INFO:teuthology.orchestra.run.smithi084.stdout: "to": 5 2024-09-20T22:48:39.527 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:39.527 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:48:39.527 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:39.527 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:39.527 INFO:teuthology.orchestra.run.smithi084.stdout: "pgid": "4.17", 2024-09-20T22:48:39.527 INFO:teuthology.orchestra.run.smithi084.stdout: "mappings": [ 2024-09-20T22:48:39.527 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:39.527 INFO:teuthology.orchestra.run.smithi084.stdout: "from": 1, 2024-09-20T22:48:39.527 INFO:teuthology.orchestra.run.smithi084.stdout: "to": 2 2024-09-20T22:48:39.528 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:39.528 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:48:39.528 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:39.528 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:39.528 INFO:teuthology.orchestra.run.smithi084.stdout: "pgid": "4.1d", 2024-09-20T22:48:39.528 INFO:teuthology.orchestra.run.smithi084.stdout: "mappings": [ 2024-09-20T22:48:39.528 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:39.528 INFO:teuthology.orchestra.run.smithi084.stdout: "from": 1, 2024-09-20T22:48:39.528 INFO:teuthology.orchestra.run.smithi084.stdout: "to": 2 2024-09-20T22:48:39.528 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:39.528 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:48:39.528 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:39.528 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:39.528 INFO:teuthology.orchestra.run.smithi084.stdout: "pgid": "7.2", 2024-09-20T22:48:39.529 INFO:teuthology.orchestra.run.smithi084.stdout: "mappings": [ 2024-09-20T22:48:39.529 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:39.529 INFO:teuthology.orchestra.run.smithi084.stdout: "from": 3, 2024-09-20T22:48:39.529 INFO:teuthology.orchestra.run.smithi084.stdout: "to": 5 2024-09-20T22:48:39.529 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:39.529 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:48:39.529 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:39.529 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:39.529 INFO:teuthology.orchestra.run.smithi084.stdout: "pgid": "7.15", 2024-09-20T22:48:39.529 INFO:teuthology.orchestra.run.smithi084.stdout: "mappings": [ 2024-09-20T22:48:39.529 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:39.529 INFO:teuthology.orchestra.run.smithi084.stdout: "from": 2, 2024-09-20T22:48:39.529 INFO:teuthology.orchestra.run.smithi084.stdout: "to": 6 2024-09-20T22:48:39.529 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:39.530 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:48:39.530 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:39.530 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2024-09-20T22:48:39.530 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_upmap_primaries": [], 2024-09-20T22:48:39.530 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_temp": [], 2024-09-20T22:48:39.530 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_temp": [], 2024-09-20T22:48:39.530 INFO:teuthology.orchestra.run.smithi084.stdout: "blocklist": { 2024-09-20T22:48:39.530 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/542020545": "2024-09-21T22:42:07.596045+0000", 2024-09-20T22:48:39.530 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/25873236": "2024-09-21T22:36:54.917812+0000", 2024-09-20T22:48:39.530 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:6800/820908470": "2024-09-21T22:36:54.917812+0000", 2024-09-20T22:48:39.530 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/567614908": "2024-09-21T22:36:54.917812+0000", 2024-09-20T22:48:39.530 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/2963622459": "2024-09-21T22:42:07.596045+0000", 2024-09-20T22:48:39.530 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/1176849276": "2024-09-21T22:37:17.690104+0000", 2024-09-20T22:48:39.530 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:6800/2932420050": "2024-09-21T22:37:17.690104+0000", 2024-09-20T22:48:39.530 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/3879263635": "2024-09-21T22:37:17.690104+0000", 2024-09-20T22:48:39.531 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/969472279": "2024-09-21T22:36:54.917812+0000", 2024-09-20T22:48:39.531 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/1361143214": "2024-09-21T22:37:17.690104+0000", 2024-09-20T22:48:39.531 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/3353126955": "2024-09-21T22:42:07.596045+0000", 2024-09-20T22:48:39.531 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:6800/4024278029": "2024-09-21T22:42:07.596045+0000", 2024-09-20T22:48:39.531 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/547902262": "2024-09-21T22:42:07.596045+0000", 2024-09-20T22:48:39.531 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/1242929317": "2024-09-21T22:42:07.596045+0000" 2024-09-20T22:48:39.531 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:39.531 INFO:teuthology.orchestra.run.smithi084.stdout: "range_blocklist": {}, 2024-09-20T22:48:39.531 INFO:teuthology.orchestra.run.smithi084.stdout: "erasure_code_profiles": { 2024-09-20T22:48:39.531 INFO:teuthology.orchestra.run.smithi084.stdout: "default": { 2024-09-20T22:48:39.531 INFO:teuthology.orchestra.run.smithi084.stdout: "crush-failure-domain": "osd", 2024-09-20T22:48:39.531 INFO:teuthology.orchestra.run.smithi084.stdout: "k": "2", 2024-09-20T22:48:39.531 INFO:teuthology.orchestra.run.smithi084.stdout: "m": "1", 2024-09-20T22:48:39.532 INFO:teuthology.orchestra.run.smithi084.stdout: "plugin": "jerasure", 2024-09-20T22:48:39.532 INFO:teuthology.orchestra.run.smithi084.stdout: "technique": "reed_sol_van" 2024-09-20T22:48:39.532 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:39.532 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:39.532 INFO:teuthology.orchestra.run.smithi084.stdout: "removed_snaps_queue": [ 2024-09-20T22:48:39.532 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:39.532 INFO:teuthology.orchestra.run.smithi084.stdout: "pool": 7, 2024-09-20T22:48:39.532 INFO:teuthology.orchestra.run.smithi084.stdout: "snaps": [ 2024-09-20T22:48:39.532 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:39.532 INFO:teuthology.orchestra.run.smithi084.stdout: "begin": 128, 2024-09-20T22:48:39.532 INFO:teuthology.orchestra.run.smithi084.stdout: "length": 1 2024-09-20T22:48:39.532 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:39.532 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:39.533 INFO:teuthology.orchestra.run.smithi084.stdout: "begin": 141, 2024-09-20T22:48:39.533 INFO:teuthology.orchestra.run.smithi084.stdout: "length": 2 2024-09-20T22:48:39.533 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:39.533 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:39.533 INFO:teuthology.orchestra.run.smithi084.stdout: "begin": 147, 2024-09-20T22:48:39.533 INFO:teuthology.orchestra.run.smithi084.stdout: "length": 2 2024-09-20T22:48:39.533 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:39.533 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:39.533 INFO:teuthology.orchestra.run.smithi084.stdout: "begin": 155, 2024-09-20T22:48:39.533 INFO:teuthology.orchestra.run.smithi084.stdout: "length": 1 2024-09-20T22:48:39.533 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:39.533 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:39.533 INFO:teuthology.orchestra.run.smithi084.stdout: "begin": 157, 2024-09-20T22:48:39.533 INFO:teuthology.orchestra.run.smithi084.stdout: "length": 1 2024-09-20T22:48:39.533 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:39.534 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:39.534 INFO:teuthology.orchestra.run.smithi084.stdout: "begin": 160, 2024-09-20T22:48:39.534 INFO:teuthology.orchestra.run.smithi084.stdout: "length": 1 2024-09-20T22:48:39.534 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:39.534 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:39.534 INFO:teuthology.orchestra.run.smithi084.stdout: "begin": 163, 2024-09-20T22:48:39.534 INFO:teuthology.orchestra.run.smithi084.stdout: "length": 2 2024-09-20T22:48:39.534 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:39.534 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:39.534 INFO:teuthology.orchestra.run.smithi084.stdout: "begin": 168, 2024-09-20T22:48:39.534 INFO:teuthology.orchestra.run.smithi084.stdout: "length": 1 2024-09-20T22:48:39.534 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:39.534 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:39.534 INFO:teuthology.orchestra.run.smithi084.stdout: "begin": 170, 2024-09-20T22:48:39.534 INFO:teuthology.orchestra.run.smithi084.stdout: "length": 1 2024-09-20T22:48:39.535 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:39.535 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:39.535 INFO:teuthology.orchestra.run.smithi084.stdout: "begin": 172, 2024-09-20T22:48:39.535 INFO:teuthology.orchestra.run.smithi084.stdout: "length": 2 2024-09-20T22:48:39.535 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:39.535 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:48:39.535 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:39.535 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2024-09-20T22:48:39.535 INFO:teuthology.orchestra.run.smithi084.stdout: "new_removed_snaps": [], 2024-09-20T22:48:39.535 INFO:teuthology.orchestra.run.smithi084.stdout: "new_purged_snaps": [], 2024-09-20T22:48:39.535 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_node_flags": {}, 2024-09-20T22:48:39.535 INFO:teuthology.orchestra.run.smithi084.stdout: "device_class_flags": {}, 2024-09-20T22:48:39.535 INFO:teuthology.orchestra.run.smithi084.stdout: "stretch_mode": { 2024-09-20T22:48:39.535 INFO:teuthology.orchestra.run.smithi084.stdout: "stretch_mode_enabled": false, 2024-09-20T22:48:39.536 INFO:teuthology.orchestra.run.smithi084.stdout: "stretch_bucket_count": 0, 2024-09-20T22:48:39.536 INFO:teuthology.orchestra.run.smithi084.stdout: "degraded_stretch_mode": 0, 2024-09-20T22:48:39.536 INFO:teuthology.orchestra.run.smithi084.stdout: "recovering_stretch_mode": 0, 2024-09-20T22:48:39.536 INFO:teuthology.orchestra.run.smithi084.stdout: "stretch_mode_bucket": 0 2024-09-20T22:48:39.536 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:39.536 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-20T22:48:39.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:39.537 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:39.537 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:48:39.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:39 smithi084 ceph-mon[29774]: 1.3 scrub starts 2024-09-20T22:48:39.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:39 smithi084 ceph-mon[29774]: 1.3 scrub ok 2024-09-20T22:48:39.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:39 smithi084 ceph-mon[29774]: 4.1e scrub starts 2024-09-20T22:48:39.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:39 smithi084 ceph-mon[29774]: 4.1e scrub ok 2024-09-20T22:48:39.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:39 smithi084 ceph-mon[29774]: pgmap v558: 166 pgs: 1 remapped+peering, 3 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 156 active+clean; 359 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail 2024-09-20T22:48:39.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:39 smithi084 ceph-mon[29774]: 5.13 scrub starts 2024-09-20T22:48:39.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:39 smithi084 ceph-mon[29774]: 5.13 scrub ok 2024-09-20T22:48:39.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:39 smithi084 ceph-mon[29774]: 7.c scrub starts 2024-09-20T22:48:39.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:39 smithi084 ceph-mon[29774]: osdmap e439: 8 total, 8 up, 8 in 2024-09-20T22:48:39.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:39 smithi084 ceph-mon[40024]: 1.3 scrub starts 2024-09-20T22:48:39.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:39 smithi084 ceph-mon[40024]: 1.3 scrub ok 2024-09-20T22:48:39.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:39 smithi084 ceph-mon[40024]: 4.1e scrub starts 2024-09-20T22:48:39.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:39 smithi084 ceph-mon[40024]: 4.1e scrub ok 2024-09-20T22:48:39.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:39 smithi084 ceph-mon[40024]: pgmap v558: 166 pgs: 1 remapped+peering, 3 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 156 active+clean; 359 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail 2024-09-20T22:48:39.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:39 smithi084 ceph-mon[40024]: 5.13 scrub starts 2024-09-20T22:48:39.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:39 smithi084 ceph-mon[40024]: 5.13 scrub ok 2024-09-20T22:48:39.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:39 smithi084 ceph-mon[40024]: 7.c scrub starts 2024-09-20T22:48:39.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:39 smithi084 ceph-mon[40024]: osdmap e439: 8 total, 8 up, 8 in 2024-09-20T22:48:39.638 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:48:39.660 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:39.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:39.661 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:48:39.761 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:48:39.784 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:39.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:39.785 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:48:39.886 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:48:39.909 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:39.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:39.910 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:48:40.011 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:48:40.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:40.034 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:40.034 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:48:40.135 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:48:40.157 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:40.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:40.158 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:48:40.213 INFO:tasks.rados.rados.0.smithi084.stdout:1701: done (6 left) 2024-09-20T22:48:40.213 INFO:tasks.rados.rados.0.smithi084.stdout:1702: done (5 left) 2024-09-20T22:48:40.213 INFO:tasks.rados.rados.0.smithi084.stdout:1704: done (4 left) 2024-09-20T22:48:40.213 INFO:tasks.rados.rados.0.smithi084.stdout:1705: done (3 left) 2024-09-20T22:48:40.214 INFO:tasks.rados.rados.0.smithi084.stdout:1706: done (2 left) 2024-09-20T22:48:40.214 INFO:tasks.rados.rados.0.smithi084.stdout:1707: done (1 left) 2024-09-20T22:48:40.214 INFO:tasks.rados.rados.0.smithi084.stdout:1708: done (0 left) 2024-09-20T22:48:40.214 INFO:tasks.rados.rados.0.smithi084.stdout:1709: rollback oid 26 current snap is 180 2024-09-20T22:48:40.214 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 26 to 173 2024-09-20T22:48:40.214 INFO:tasks.rados.rados.0.smithi084.stdout:1710: delete oid 2 current snap is 180 2024-09-20T22:48:40.215 INFO:tasks.rados.rados.0.smithi084.stdout:1710: done (1 left) 2024-09-20T22:48:40.215 INFO:tasks.rados.rados.0.smithi084.stdout:1711: snap_create 2024-09-20T22:48:40.259 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:48:40.282 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:40.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:40.283 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:48:40.384 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:48:40.407 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:40.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:40.408 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:48:40.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:40 smithi154 ceph-mon[33264]: 7.c scrub ok 2024-09-20T22:48:40.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:40 smithi154 ceph-mon[33264]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:48:40.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:40 smithi154 ceph-mon[33264]: 6.1c scrub starts 2024-09-20T22:48:40.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:40 smithi154 ceph-mon[33264]: 6.1c scrub ok 2024-09-20T22:48:40.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:40 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/2492529169' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-20T22:48:40.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:40 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "23"}]: dispatch 2024-09-20T22:48:40.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:40 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "23"}]: dispatch 2024-09-20T22:48:40.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:40 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "23"}]': finished 2024-09-20T22:48:40.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:40 smithi154 ceph-mon[33264]: osdmap e440: 8 total, 8 up, 8 in 2024-09-20T22:48:40.509 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:48:40.531 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:40.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:40.531 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:48:40.548 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:40 smithi084 ceph-mon[40024]: 7.c scrub ok 2024-09-20T22:48:40.548 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:40 smithi084 ceph-mon[40024]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:48:40.548 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:40 smithi084 ceph-mon[40024]: 6.1c scrub starts 2024-09-20T22:48:40.548 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:40 smithi084 ceph-mon[40024]: 6.1c scrub ok 2024-09-20T22:48:40.549 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:40 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/2492529169' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-20T22:48:40.549 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:40 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "23"}]: dispatch 2024-09-20T22:48:40.549 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:40 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "23"}]: dispatch 2024-09-20T22:48:40.549 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:40 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "23"}]': finished 2024-09-20T22:48:40.549 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:40 smithi084 ceph-mon[40024]: osdmap e440: 8 total, 8 up, 8 in 2024-09-20T22:48:40.549 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:40 smithi084 ceph-mon[29774]: 7.c scrub ok 2024-09-20T22:48:40.549 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:40 smithi084 ceph-mon[29774]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:48:40.549 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:40 smithi084 ceph-mon[29774]: 6.1c scrub starts 2024-09-20T22:48:40.549 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:40 smithi084 ceph-mon[29774]: 6.1c scrub ok 2024-09-20T22:48:40.549 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:40 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/2492529169' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-20T22:48:40.549 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:40 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "23"}]: dispatch 2024-09-20T22:48:40.549 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:40 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "23"}]: dispatch 2024-09-20T22:48:40.550 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:40 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "23"}]': finished 2024-09-20T22:48:40.550 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:40 smithi084 ceph-mon[29774]: osdmap e440: 8 total, 8 up, 8 in 2024-09-20T22:48:40.574 INFO:tasks.thrashosds.thrasher:j is {'epoch': 439, 'fsid': 'a9c0bc70-77a0-11ef-baf5-efdc52797490', 'created': '2024-09-20T22:36:30.240626+0000', 'modified': '2024-09-20T22:48:39.204612+0000', 'last_up_change': '2024-09-20T22:41:27.098565+0000', 'last_in_change': '2024-09-20T22:48:04.201694+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 18, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 7, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-20T22:39:21.984381+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 11, 'pg_placement_num': 11, 'pg_placement_num_target': 11, 'pg_num_target': 11, 'pg_num_pending': 11, 'last_pg_merge_meta': {'source_pgid': '1.b', 'ready_epoch': 299, 'last_epoch_started': 298, 'last_epoch_clean': 298, 'source_version': "77'128", 'target_version': "77'128"}, 'last_change': '299', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '299', 'last_force_op_resend_preluminous': '297', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.180000066757202, 'score_stable': 2.180000066757202, 'optimal_score': 1, 'raw_score_acting': 2.180000066757202, 'raw_score_stable': 2.180000066757202, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-09-20T22:41:35.820081+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '69', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 69, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.6500000953674316, 'score_stable': 2.6500000953674316, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 2.3299999237060547, 'raw_score_stable': 2.3299999237060547, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-09-20T22:41:36.546389+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-09-20T22:41:38.454640+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '70', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-09-20T22:41:40.407129+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '72', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-09-20T22:41:42.476716+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '74', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'unique_pool_0', 'create_time': '2024-09-20T22:42:51.354358+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 24, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 24, 'last_pg_merge_meta': {'source_pgid': '7.18', 'ready_epoch': 430, 'last_epoch_started': 429, 'last_epoch_clean': 429, 'source_version': "426'268", 'target_version': "425'136"}, 'last_change': '439', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '430', 'last_force_op_resend_preluminous': '428', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 180, 'snap_epoch': 439, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': 'ea36a489-6f91-4b2f-b784-b90506b9805d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 434, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6801', 'nonce': 268084626}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6802', 'nonce': 268084626}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6804', 'nonce': 268084626}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6803', 'nonce': 268084626}]}, 'public_addr': '172.21.15.84:6801/268084626', 'cluster_addr': '172.21.15.84:6802/268084626', 'heartbeat_back_addr': '172.21.15.84:6804/268084626', 'heartbeat_front_addr': '172.21.15.84:6803/268084626', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'c8649d2f-27ef-47f8-bc94-f1cd9e62bdb8', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 420, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6805', 'nonce': 418374136}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6806', 'nonce': 418374136}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6808', 'nonce': 418374136}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6807', 'nonce': 418374136}]}, 'public_addr': '172.21.15.84:6805/418374136', 'cluster_addr': '172.21.15.84:6806/418374136', 'heartbeat_back_addr': '172.21.15.84:6808/418374136', 'heartbeat_front_addr': '172.21.15.84:6807/418374136', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'c728eb48-a0f0-4de4-a651-7f3bec867280', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 430, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6809', 'nonce': 3977168611}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6810', 'nonce': 3977168611}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6812', 'nonce': 3977168611}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6811', 'nonce': 3977168611}]}, 'public_addr': '172.21.15.84:6809/3977168611', 'cluster_addr': '172.21.15.84:6810/3977168611', 'heartbeat_back_addr': '172.21.15.84:6812/3977168611', 'heartbeat_front_addr': '172.21.15.84:6811/3977168611', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '573f44eb-0853-4630-a7d8-9caaeb256f9b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 25, 'up_thru': 421, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6813', 'nonce': 2192239945}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6814', 'nonce': 2192239945}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6816', 'nonce': 2192239945}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6815', 'nonce': 2192239945}]}, 'public_addr': '172.21.15.84:6813/2192239945', 'cluster_addr': '172.21.15.84:6814/2192239945', 'heartbeat_back_addr': '172.21.15.84:6816/2192239945', 'heartbeat_front_addr': '172.21.15.84:6815/2192239945', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'a52c3b94-c5f3-490f-8ac8-09061b9b2abb', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 418, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6800', 'nonce': 1793136002}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6801', 'nonce': 1793136002}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6803', 'nonce': 1793136002}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6802', 'nonce': 1793136002}]}, 'public_addr': '172.21.15.154:6800/1793136002', 'cluster_addr': '172.21.15.154:6801/1793136002', 'heartbeat_back_addr': '172.21.15.154:6803/1793136002', 'heartbeat_front_addr': '172.21.15.154:6802/1793136002', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '4fa0dc8e-25cc-46cd-96ff-33f20b6faec3', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 40, 'up_thru': 421, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6804', 'nonce': 493899927}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6805', 'nonce': 493899927}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6807', 'nonce': 493899927}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6806', 'nonce': 493899927}]}, 'public_addr': '172.21.15.154:6804/493899927', 'cluster_addr': '172.21.15.154:6805/493899927', 'heartbeat_back_addr': '172.21.15.154:6807/493899927', 'heartbeat_front_addr': '172.21.15.154:6806/493899927', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '56788d74-810e-4827-a615-958c12f8a368', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 48, 'up_thru': 435, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6808', 'nonce': 2375091033}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6809', 'nonce': 2375091033}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6811', 'nonce': 2375091033}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6810', 'nonce': 2375091033}]}, 'public_addr': '172.21.15.154:6808/2375091033', 'cluster_addr': '172.21.15.154:6809/2375091033', 'heartbeat_back_addr': '172.21.15.154:6811/2375091033', 'heartbeat_front_addr': '172.21.15.154:6810/2375091033', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '83503727-afc1-4b95-8b96-ecb63f3338c2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 59, 'up_thru': 418, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6812', 'nonce': 3717704491}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6813', 'nonce': 3717704491}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6815', 'nonce': 3717704491}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6814', 'nonce': 3717704491}]}, 'public_addr': '172.21.15.154:6812/3717704491', 'cluster_addr': '172.21.15.154:6813/3717704491', 'heartbeat_back_addr': '172.21.15.154:6815/3717704491', 'heartbeat_front_addr': '172.21.15.154:6814/3717704491', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-20T22:47:46.150000+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-20T22:48:06.402076+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-20T22:48:23.610047+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-20T22:47:38.488604+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-20T22:47:43.022898+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-20T22:48:30.072642+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-20T22:48:15.304260+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-20T22:47:39.257733+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '3.a', 'mappings': [{'from': 6, 'to': 2}]}, {'pgid': '4.6', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.11', 'mappings': [{'from': 3, 'to': 5}]}, {'pgid': '4.17', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.1d', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '7.2', 'mappings': [{'from': 3, 'to': 5}]}, {'pgid': '7.15', 'mappings': [{'from': 2, 'to': 6}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.84:0/542020545': '2024-09-21T22:42:07.596045+0000', '172.21.15.84:0/25873236': '2024-09-21T22:36:54.917812+0000', '172.21.15.84:6800/820908470': '2024-09-21T22:36:54.917812+0000', '172.21.15.84:0/567614908': '2024-09-21T22:36:54.917812+0000', '172.21.15.84:0/2963622459': '2024-09-21T22:42:07.596045+0000', '172.21.15.84:0/1176849276': '2024-09-21T22:37:17.690104+0000', '172.21.15.84:6800/2932420050': '2024-09-21T22:37:17.690104+0000', '172.21.15.84:0/3879263635': '2024-09-21T22:37:17.690104+0000', '172.21.15.84:0/969472279': '2024-09-21T22:36:54.917812+0000', '172.21.15.84:0/1361143214': '2024-09-21T22:37:17.690104+0000', '172.21.15.84:0/3353126955': '2024-09-21T22:42:07.596045+0000', '172.21.15.84:6800/4024278029': '2024-09-21T22:42:07.596045+0000', '172.21.15.84:0/547902262': '2024-09-21T22:42:07.596045+0000', '172.21.15.84:0/1242929317': '2024-09-21T22:42:07.596045+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [{'pool': 7, 'snaps': [{'begin': 128, 'length': 1}, {'begin': 141, 'length': 2}, {'begin': 147, 'length': 2}, {'begin': 155, 'length': 1}, {'begin': 157, 'length': 1}, {'begin': 160, 'length': 1}, {'begin': 163, 'length': 2}, {'begin': 168, 'length': 1}, {'begin': 170, 'length': 1}, {'begin': 172, 'length': 2}]}], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-20T22:48:40.574 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph pg dump --format=json 2024-09-20T22:48:40.632 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:48:40.678 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:40.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:40.679 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:48:40.780 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:48:40.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:40.803 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:40.803 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:48:40.904 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:48:40.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:40.927 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:40.927 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:48:40.978 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd set noscrub 2024-09-20T22:48:41.028 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:48:41.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:41.050 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:41.050 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:48:41.151 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:48:41.173 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:41.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:41.174 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:48:41.215 INFO:tasks.rados.rados.0.smithi084.stdout:1711: done (1 left) 2024-09-20T22:48:41.215 INFO:tasks.rados.rados.0.smithi084.stdout:1712: write (excl) oid 15 current snap is 181 2024-09-20T22:48:41.215 INFO:tasks.rados.rados.0.smithi084.stdout:1712: seq_num 605 ranges {568235=717532,1753253=414982} 2024-09-20T22:48:41.222 INFO:tasks.rados.rados.0.smithi084.stdout:1712: writing smithi08477203-15 from 568235 to 1285767 tid 1 2024-09-20T22:48:41.227 INFO:tasks.rados.rados.0.smithi084.stdout:1712: writing smithi08477203-15 from 1753253 to 2168235 tid 2 2024-09-20T22:48:41.227 INFO:tasks.rados.rados.0.smithi084.stdout:1713: rollback oid 50 current snap is 181 2024-09-20T22:48:41.227 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 50 to 156 2024-09-20T22:48:41.227 INFO:tasks.rados.rados.0.smithi084.stdout:1712: finishing write tid 1 to smithi08477203-15 2024-09-20T22:48:41.227 INFO:tasks.rados.rados.0.smithi084.stdout:1714: write oid 36 current snap is 181 2024-09-20T22:48:41.227 INFO:tasks.rados.rados.0.smithi084.stdout:1714: seq_num 606 ranges {585042=792781,2095382=89660} 2024-09-20T22:48:41.235 INFO:tasks.rados.rados.0.smithi084.stdout:1714: writing smithi08477203-36 from 585042 to 1377823 tid 1 2024-09-20T22:48:41.239 INFO:tasks.rados.rados.0.smithi084.stdout:1714: writing smithi08477203-36 from 2095382 to 2185042 tid 2 2024-09-20T22:48:41.240 INFO:tasks.rados.rados.0.smithi084.stdout:1715: snap_remove snap 169 2024-09-20T22:48:41.275 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:48:41.284 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:48:41.285 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:48:41.298 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:41.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:41.299 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:48:41.301 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:48:41.301 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:48:41.317 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:48:41.317 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:48:41.333 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-20T22:48:41.333 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T22:48:41.350 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:48:41.350 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:48:41.366 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:48:41.366 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:48:41.383 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:48:41.384 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:48:41.400 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:48:41.401 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:48:41.401 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:48:41.423 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:41.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:41.424 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:48:41.479 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:41 smithi084 ceph-mon[40024]: pgmap v561: 166 pgs: 3 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 158 active+clean; 359 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.7 MiB/s rd, 719 KiB/s wr, 8 op/s; 3.1 MiB/s, 7 objects/s recovering 2024-09-20T22:48:41.479 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:41 smithi084 ceph-mon[40024]: 6.c deep-scrub starts 2024-09-20T22:48:41.479 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:41 smithi084 ceph-mon[40024]: 6.c deep-scrub ok 2024-09-20T22:48:41.479 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:41 smithi084 ceph-mon[40024]: 6.14 scrub starts 2024-09-20T22:48:41.479 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:41 smithi084 ceph-mon[40024]: 6.14 scrub ok 2024-09-20T22:48:41.479 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:41 smithi084 ceph-mon[40024]: 6.15 scrub starts 2024-09-20T22:48:41.479 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:41 smithi084 ceph-mon[40024]: 6.15 scrub ok 2024-09-20T22:48:41.479 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:41 smithi084 ceph-mon[40024]: 4.1b deep-scrub starts 2024-09-20T22:48:41.479 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:41 smithi084 ceph-mon[40024]: 4.1b deep-scrub ok 2024-09-20T22:48:41.480 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:41 smithi084 ceph-mon[40024]: 6.1 deep-scrub starts 2024-09-20T22:48:41.480 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:41 smithi084 ceph-mon[40024]: 6.1 deep-scrub ok 2024-09-20T22:48:41.480 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:41 smithi084 ceph-mon[40024]: osdmap e441: 8 total, 8 up, 8 in 2024-09-20T22:48:41.480 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:41 smithi084 ceph-mon[29774]: pgmap v561: 166 pgs: 3 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 158 active+clean; 359 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.7 MiB/s rd, 719 KiB/s wr, 8 op/s; 3.1 MiB/s, 7 objects/s recovering 2024-09-20T22:48:41.480 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:41 smithi084 ceph-mon[29774]: 6.c deep-scrub starts 2024-09-20T22:48:41.480 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:41 smithi084 ceph-mon[29774]: 6.c deep-scrub ok 2024-09-20T22:48:41.480 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:41 smithi084 ceph-mon[29774]: 6.14 scrub starts 2024-09-20T22:48:41.480 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:41 smithi084 ceph-mon[29774]: 6.14 scrub ok 2024-09-20T22:48:41.480 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:41 smithi084 ceph-mon[29774]: 6.15 scrub starts 2024-09-20T22:48:41.480 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:41 smithi084 ceph-mon[29774]: 6.15 scrub ok 2024-09-20T22:48:41.480 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:41 smithi084 ceph-mon[29774]: 4.1b deep-scrub starts 2024-09-20T22:48:41.481 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:41 smithi084 ceph-mon[29774]: 4.1b deep-scrub ok 2024-09-20T22:48:41.481 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:41 smithi084 ceph-mon[29774]: 6.1 deep-scrub starts 2024-09-20T22:48:41.481 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:41 smithi084 ceph-mon[29774]: 6.1 deep-scrub ok 2024-09-20T22:48:41.481 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:41 smithi084 ceph-mon[29774]: osdmap e441: 8 total, 8 up, 8 in 2024-09-20T22:48:41.482 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-20T22:48:41.482 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2024-09-20T22:48:41.482 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2024-09-20T22:48:41.482 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [ 2024-09-20T22:48:41.482 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.482 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.14460.0:1 3.12 3:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-20T22:48:41.482 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:41:37.966675+0000", 2024-09-20T22:48:41.482 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 423.51151101400001, 2024-09-20T22:48:41.482 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.19206436900000001, 2024-09-20T22:48:41.482 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:48:41.482 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:48:41.482 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "reached pg", 2024-09-20T22:48:41.483 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:48:41.483 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.14460", 2024-09-20T22:48:41.483 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/1273482728", 2024-09-20T22:48:41.483 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1 2024-09-20T22:48:41.483 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.483 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:48:41.483 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.483 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:48:41.483 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:41:37.966675+0000", 2024-09-20T22:48:41.483 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:48:41.483 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.483 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.483 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:48:41.483 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:41:37.966675+0000", 2024-09-20T22:48:41.484 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:48:41.484 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.484 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.484 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:48:41.484 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:41:37.966682+0000", 2024-09-20T22:48:41.484 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.3100000000000003e-06 2024-09-20T22:48:41.484 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.484 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.484 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:48:41.484 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:41:37.966688+0000", 2024-09-20T22:48:41.484 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.1209999999999998e-06 2024-09-20T22:48:41.484 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.484 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.484 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:48:41.485 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:41:37.966698+0000", 2024-09-20T22:48:41.485 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0220999999999999e-05 2024-09-20T22:48:41.485 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.485 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.485 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:48:41.485 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:41:37.966706+0000", 2024-09-20T22:48:41.485 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.4460000000000001e-06 2024-09-20T22:48:41.485 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.485 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.485 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:48:41.485 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:41:37.966763+0000", 2024-09-20T22:48:41.485 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.7025999999999999e-05 2024-09-20T22:48:41.486 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.486 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.486 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-09-20T22:48:41.486 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:41:37.966771+0000", 2024-09-20T22:48:41.486 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.9540000000000004e-06 2024-09-20T22:48:41.486 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.486 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.486 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:48:41.486 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:41:38.158654+0000", 2024-09-20T22:48:41.486 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.19188287000000001 2024-09-20T22:48:41.486 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.486 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.486 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:48:41.486 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:41:38.158740+0000", 2024-09-20T22:48:41.486 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.5420999999999994e-05 2024-09-20T22:48:41.487 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:41.487 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:48:41.487 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:41.487 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.487 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.487 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24652.0:336 7.7 e107/94 7:e67788e2:::smithi08477203-31:head v 107'45, mlcod=105'41)", 2024-09-20T22:48:41.487 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:43:06.546789+0000", 2024-09-20T22:48:41.487 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 334.93139724000002, 2024-09-20T22:48:41.487 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013900732000000001, 2024-09-20T22:48:41.487 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:48:41.487 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:48:41.487 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:48:41.487 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:48:41.487 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.488 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:48:41.488 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:06.546789+0000", 2024-09-20T22:48:41.488 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:48:41.488 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.488 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.488 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:48:41.488 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:06.546789+0000", 2024-09-20T22:48:41.488 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:48:41.488 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.488 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.488 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:48:41.488 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:06.546789+0000", 2024-09-20T22:48:41.488 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.1500000000000001e-07 2024-09-20T22:48:41.488 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.489 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.489 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:48:41.489 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:06.554560+0000", 2024-09-20T22:48:41.489 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0077711159999999998 2024-09-20T22:48:41.489 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.489 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.489 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:48:41.489 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:06.554577+0000", 2024-09-20T22:48:41.489 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.6509000000000001e-05 2024-09-20T22:48:41.489 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.489 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.489 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:48:41.489 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:06.554586+0000", 2024-09-20T22:48:41.489 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.7089999999999997e-06 2024-09-20T22:48:41.490 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.490 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.490 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:48:41.490 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:06.554645+0000", 2024-09-20T22:48:41.490 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.8202999999999998e-05 2024-09-20T22:48:41.490 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.490 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.490 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:48:41.490 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:06.554661+0000", 2024-09-20T22:48:41.490 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.6099e-05 2024-09-20T22:48:41.490 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.490 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.490 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:48:41.490 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:06.560643+0000", 2024-09-20T22:48:41.490 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0059824750000000001 2024-09-20T22:48:41.491 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.491 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.491 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:48:41.491 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:06.560690+0000", 2024-09-20T22:48:41.491 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.6406000000000003e-05 2024-09-20T22:48:41.491 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:41.491 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:48:41.491 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:41.491 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.491 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.491 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:353 7.f 7:f93102e8:::smithi08477203-37:head [copy-from ver 3] snapc 6=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-09-20T22:48:41.492 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:43:07.813602+0000", 2024-09-20T22:48:41.492 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 333.66458460600001, 2024-09-20T22:48:41.492 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014089683, 2024-09-20T22:48:41.492 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:48:41.492 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:48:41.492 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:48:41.492 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:48:41.492 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:48:41.492 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:48:41.492 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 353 2024-09-20T22:48:41.492 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.492 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:48:41.492 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.492 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:48:41.493 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.813602+0000", 2024-09-20T22:48:41.493 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:48:41.493 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.493 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.493 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:48:41.493 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.813602+0000", 2024-09-20T22:48:41.493 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:48:41.493 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.493 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.493 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:48:41.493 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.813604+0000", 2024-09-20T22:48:41.493 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.9939999999999999e-06 2024-09-20T22:48:41.493 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.493 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.493 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:48:41.494 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.813612+0000", 2024-09-20T22:48:41.494 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.3690000000000007e-06 2024-09-20T22:48:41.494 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.494 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.494 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:48:41.494 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.813631+0000", 2024-09-20T22:48:41.494 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.8589000000000001e-05 2024-09-20T22:48:41.494 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.494 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.494 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:48:41.494 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.813642+0000", 2024-09-20T22:48:41.494 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1766999999999999e-05 2024-09-20T22:48:41.494 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.494 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.495 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:48:41.495 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.813711+0000", 2024-09-20T22:48:41.495 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.9175000000000001e-05 2024-09-20T22:48:41.495 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.495 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.495 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:48:41.495 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.813811+0000", 2024-09-20T22:48:41.495 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.9672000000000004e-05 2024-09-20T22:48:41.495 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.495 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.495 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 3,7", 2024-09-20T22:48:41.495 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.817022+0000", 2024-09-20T22:48:41.495 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00321135 2024-09-20T22:48:41.495 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.495 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.496 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:48:41.496 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.820378+0000", 2024-09-20T22:48:41.496 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.003355224 2024-09-20T22:48:41.496 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.496 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.496 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:48:41.496 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.820397+0000", 2024-09-20T22:48:41.496 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.8899999999999999e-05 2024-09-20T22:48:41.496 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.496 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.496 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:48:41.496 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.827619+0000", 2024-09-20T22:48:41.496 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.007222816 2024-09-20T22:48:41.496 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.497 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.497 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:48:41.497 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.827653+0000", 2024-09-20T22:48:41.497 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.392e-05 2024-09-20T22:48:41.497 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.497 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.497 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:48:41.497 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.827691+0000", 2024-09-20T22:48:41.497 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.7907000000000001e-05 2024-09-20T22:48:41.497 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:41.497 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:48:41.497 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:41.497 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.498 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.498 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:359 7.9 7:91f2d47c:::smithi08477203-8:head [copy-from ver 6] snapc 6=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-09-20T22:48:41.498 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:43:07.829163+0000", 2024-09-20T22:48:41.498 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 333.64902358699999, 2024-09-20T22:48:41.498 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011899852000000001, 2024-09-20T22:48:41.498 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:48:41.498 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:48:41.498 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:48:41.498 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:48:41.498 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:48:41.498 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:48:41.498 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 359 2024-09-20T22:48:41.498 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.498 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:48:41.499 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.499 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:48:41.499 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.829163+0000", 2024-09-20T22:48:41.499 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:48:41.499 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.499 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.499 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:48:41.499 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.829163+0000", 2024-09-20T22:48:41.499 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:48:41.499 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.499 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.499 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:48:41.499 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.829163+0000", 2024-09-20T22:48:41.499 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.6199999999999999e-07 2024-09-20T22:48:41.500 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.500 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.500 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:48:41.500 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.829165+0000", 2024-09-20T22:48:41.500 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.7209999999999999e-06 2024-09-20T22:48:41.500 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.500 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.500 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:48:41.500 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.829169+0000", 2024-09-20T22:48:41.500 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.121e-06 2024-09-20T22:48:41.500 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.500 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.500 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:48:41.501 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.829172+0000", 2024-09-20T22:48:41.501 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.8839999999999998e-06 2024-09-20T22:48:41.501 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.501 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.501 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:48:41.501 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.829211+0000", 2024-09-20T22:48:41.501 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.9551000000000001e-05 2024-09-20T22:48:41.501 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.501 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.501 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:48:41.501 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.829294+0000", 2024-09-20T22:48:41.502 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.2548999999999999e-05 2024-09-20T22:48:41.502 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.502 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.502 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 1,4", 2024-09-20T22:48:41.502 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.834578+0000", 2024-09-20T22:48:41.502 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.005284179 2024-09-20T22:48:41.502 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.502 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.502 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:48:41.502 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.840537+0000", 2024-09-20T22:48:41.502 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0059587670000000002 2024-09-20T22:48:41.502 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.502 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.502 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:48:41.503 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.840753+0000", 2024-09-20T22:48:41.503 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00021661799999999999 2024-09-20T22:48:41.503 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.503 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.503 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:48:41.503 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.840999+0000", 2024-09-20T22:48:41.503 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000246009 2024-09-20T22:48:41.503 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.503 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.503 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:48:41.503 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.841032+0000", 2024-09-20T22:48:41.503 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.2567999999999997e-05 2024-09-20T22:48:41.503 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.503 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.504 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:48:41.504 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:07.841062+0000", 2024-09-20T22:48:41.504 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.0623000000000001e-05 2024-09-20T22:48:41.504 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:41.504 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:48:41.504 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:41.504 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.504 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.504 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:522 7.9 7:9970b578:::smithi08477203-15:head [copy-from ver 28] snapc f=[f,d] ondisk+write+known_if_redirected+supports_pool_eio e124)", 2024-09-20T22:48:41.504 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:43:23.126433+0000", 2024-09-20T22:48:41.504 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 318.35175345800002, 2024-09-20T22:48:41.504 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.027726022999999999, 2024-09-20T22:48:41.504 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:48:41.504 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:48:41.504 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:48:41.505 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:48:41.505 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:48:41.505 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:48:41.505 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 522 2024-09-20T22:48:41.505 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.505 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:48:41.505 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.505 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:48:41.505 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:23.126433+0000", 2024-09-20T22:48:41.505 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:48:41.505 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.505 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.505 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:48:41.505 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:23.126433+0000", 2024-09-20T22:48:41.506 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:48:41.506 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.506 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.506 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:48:41.506 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:23.126434+0000", 2024-09-20T22:48:41.506 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3200000000000001e-06 2024-09-20T22:48:41.506 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.506 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.506 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:48:41.506 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:23.126438+0000", 2024-09-20T22:48:41.506 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.6959999999999998e-06 2024-09-20T22:48:41.506 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.506 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.506 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:48:41.506 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:23.126448+0000", 2024-09-20T22:48:41.507 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0436e-05 2024-09-20T22:48:41.507 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.507 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.507 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:48:41.507 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:23.126455+0000", 2024-09-20T22:48:41.507 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.7399999999999998e-06 2024-09-20T22:48:41.507 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.507 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.507 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:48:41.507 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:23.126505+0000", 2024-09-20T22:48:41.507 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.9622e-05 2024-09-20T22:48:41.507 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.507 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.507 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:48:41.508 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:23.126598+0000", 2024-09-20T22:48:41.508 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.3905000000000003e-05 2024-09-20T22:48:41.508 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.508 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.508 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 1,4", 2024-09-20T22:48:41.508 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:23.136852+0000", 2024-09-20T22:48:41.508 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010253372 2024-09-20T22:48:41.508 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.508 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.508 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:48:41.508 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:23.146781+0000", 2024-09-20T22:48:41.508 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0099292019999999998 2024-09-20T22:48:41.508 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.508 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.509 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:48:41.509 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:23.146838+0000", 2024-09-20T22:48:41.509 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.7046999999999998e-05 2024-09-20T22:48:41.509 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.509 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.509 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:48:41.509 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:23.154054+0000", 2024-09-20T22:48:41.509 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0072155910000000004 2024-09-20T22:48:41.509 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.509 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.509 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:48:41.509 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:23.154100+0000", 2024-09-20T22:48:41.509 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.6613999999999998e-05 2024-09-20T22:48:41.509 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:41 smithi154 ceph-mon[33264]: pgmap v561: 166 pgs: 3 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 158 active+clean; 359 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.7 MiB/s rd, 719 KiB/s wr, 8 op/s; 3.1 MiB/s, 7 objects/s recovering 2024-09-20T22:48:41.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:41 smithi154 ceph-mon[33264]: 6.c deep-scrub starts 2024-09-20T22:48:41.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:41 smithi154 ceph-mon[33264]: 6.c deep-scrub ok 2024-09-20T22:48:41.510 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:41 smithi154 ceph-mon[33264]: 6.14 scrub starts 2024-09-20T22:48:41.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:41 smithi154 ceph-mon[33264]: 6.14 scrub ok 2024-09-20T22:48:41.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:41 smithi154 ceph-mon[33264]: 6.15 scrub starts 2024-09-20T22:48:41.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:41 smithi154 ceph-mon[33264]: 6.15 scrub ok 2024-09-20T22:48:41.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:41 smithi154 ceph-mon[33264]: 4.1b deep-scrub starts 2024-09-20T22:48:41.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:41 smithi154 ceph-mon[33264]: 4.1b deep-scrub ok 2024-09-20T22:48:41.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:41 smithi154 ceph-mon[33264]: 6.1 deep-scrub starts 2024-09-20T22:48:41.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:41 smithi154 ceph-mon[33264]: 6.1 deep-scrub ok 2024-09-20T22:48:41.511 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:41 smithi154 ceph-mon[33264]: osdmap e441: 8 total, 8 up, 8 in 2024-09-20T22:48:41.512 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.512 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:48:41.512 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:23.154159+0000", 2024-09-20T22:48:41.512 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.8477999999999998e-05 2024-09-20T22:48:41.512 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:41.512 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:48:41.512 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:41.512 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.512 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.512 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:523 7.9 7:9970b578:::smithi08477203-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e124)", 2024-09-20T22:48:41.513 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:43:23.126473+0000", 2024-09-20T22:48:41.513 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 318.35171334299997, 2024-09-20T22:48:41.513 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.015897446999999999, 2024-09-20T22:48:41.513 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:48:41.513 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:48:41.513 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-20T22:48:41.513 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:48:41.513 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:48:41.513 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:48:41.513 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 523 2024-09-20T22:48:41.513 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.513 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:48:41.513 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.514 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:48:41.514 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:23.126473+0000", 2024-09-20T22:48:41.514 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:48:41.514 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.514 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.514 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:48:41.514 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:23.126473+0000", 2024-09-20T22:48:41.514 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:48:41.514 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.514 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.514 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:48:41.514 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:23.126473+0000", 2024-09-20T22:48:41.514 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.5599999999999995e-07 2024-09-20T22:48:41.514 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.514 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.515 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:48:41.515 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:23.126476+0000", 2024-09-20T22:48:41.515 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.357e-06 2024-09-20T22:48:41.515 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.515 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.515 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:48:41.515 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:23.126482+0000", 2024-09-20T22:48:41.515 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.5899999999999996e-06 2024-09-20T22:48:41.515 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.515 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.515 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:48:41.515 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:23.126488+0000", 2024-09-20T22:48:41.515 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.6219999999999997e-06 2024-09-20T22:48:41.516 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.516 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.516 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:48:41.516 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:23.126755+0000", 2024-09-20T22:48:41.516 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026716100000000003 2024-09-20T22:48:41.516 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.516 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.516 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for blocked object", 2024-09-20T22:48:41.516 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:23.126808+0000", 2024-09-20T22:48:41.516 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.2859999999999999e-05 2024-09-20T22:48:41.516 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.516 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.516 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:48:41.516 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:23.142223+0000", 2024-09-20T22:48:41.517 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.015414773999999999 2024-09-20T22:48:41.517 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.517 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.517 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:48:41.517 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:23.142296+0000", 2024-09-20T22:48:41.517 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.3394999999999998e-05 2024-09-20T22:48:41.517 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.517 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.517 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:48:41.517 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:23.142370+0000", 2024-09-20T22:48:41.517 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.4132000000000001e-05 2024-09-20T22:48:41.517 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:41.517 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:48:41.518 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:41.518 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.518 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.518 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:928 7.9 7:91f2d47c:::smithi08477203-8:head [stat,delete] snapc 1f=[] ondisk+write+known_if_redirected+supports_pool_eio e158)", 2024-09-20T22:48:41.518 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:43:57.433849+0000", 2024-09-20T22:48:41.518 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 284.044337343, 2024-09-20T22:48:41.518 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.012508692, 2024-09-20T22:48:41.518 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:48:41.518 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:48:41.518 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-20T22:48:41.518 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:48:41.518 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:48:41.518 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:48:41.518 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 928 2024-09-20T22:48:41.519 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.519 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:48:41.519 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.519 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:48:41.519 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:57.433849+0000", 2024-09-20T22:48:41.519 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:48:41.519 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.519 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.519 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:48:41.519 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:57.433849+0000", 2024-09-20T22:48:41.519 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:48:41.519 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.520 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.520 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:48:41.520 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:57.433850+0000", 2024-09-20T22:48:41.520 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.4670000000000001e-06 2024-09-20T22:48:41.520 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.520 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.520 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:48:41.520 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:57.433855+0000", 2024-09-20T22:48:41.520 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.4399999999999998e-06 2024-09-20T22:48:41.520 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.520 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.520 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:48:41.521 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:57.433865+0000", 2024-09-20T22:48:41.521 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.8779999999999993e-06 2024-09-20T22:48:41.521 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.521 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.521 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:48:41.521 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:57.433872+0000", 2024-09-20T22:48:41.521 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.2350000000000002e-06 2024-09-20T22:48:41.521 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.521 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.521 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:48:41.521 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:57.434024+0000", 2024-09-20T22:48:41.521 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000152091 2024-09-20T22:48:41.521 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.521 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.521 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for missing object", 2024-09-20T22:48:41.522 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:57.434140+0000", 2024-09-20T22:48:41.522 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000116242 2024-09-20T22:48:41.522 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.522 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.522 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:48:41.522 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:57.438648+0000", 2024-09-20T22:48:41.522 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0045080349999999996 2024-09-20T22:48:41.522 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.522 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.522 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:48:41.522 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:57.438752+0000", 2024-09-20T22:48:41.522 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00010382699999999999 2024-09-20T22:48:41.522 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.523 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.523 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:48:41.523 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:57.446358+0000", 2024-09-20T22:48:41.523 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0076054770000000002 2024-09-20T22:48:41.523 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:41.523 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:48:41.523 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:41.523 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.523 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.523 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:950 7.18 7:1cd232b2:::smithi08477203-35:head [copy-from ver 6] snapc 20=[20] ondisk+write+known_if_redirected+supports_pool_eio e159)", 2024-09-20T22:48:41.523 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:43:58.151870+0000", 2024-09-20T22:48:41.523 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 283.32631598400002, 2024-09-20T22:48:41.523 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.01686789, 2024-09-20T22:48:41.523 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:48:41.523 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:48:41.524 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:48:41.524 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:48:41.524 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:48:41.524 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:48:41.524 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 950 2024-09-20T22:48:41.524 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.524 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:48:41.524 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.524 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:48:41.524 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:58.151870+0000", 2024-09-20T22:48:41.524 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:48:41.524 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.524 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.524 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:48:41.525 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:58.151870+0000", 2024-09-20T22:48:41.525 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:48:41.525 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.525 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.525 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:48:41.525 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:58.151873+0000", 2024-09-20T22:48:41.525 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.3140000000000002e-06 2024-09-20T22:48:41.525 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.525 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.525 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:48:41.525 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:58.151896+0000", 2024-09-20T22:48:41.525 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.3788e-05 2024-09-20T22:48:41.525 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.525 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.525 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:48:41.526 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:58.151934+0000", 2024-09-20T22:48:41.526 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.7873999999999997e-05 2024-09-20T22:48:41.526 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.526 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.526 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:48:41.526 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:58.151956+0000", 2024-09-20T22:48:41.526 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.1523e-05 2024-09-20T22:48:41.526 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.526 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.526 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:48:41.526 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:58.152044+0000", 2024-09-20T22:48:41.526 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.8732000000000004e-05 2024-09-20T22:48:41.526 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.526 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.527 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:48:41.527 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:58.152152+0000", 2024-09-20T22:48:41.527 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000107769 2024-09-20T22:48:41.527 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.527 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.527 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 6,7", 2024-09-20T22:48:41.527 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:58.158276+0000", 2024-09-20T22:48:41.527 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0061237970000000003 2024-09-20T22:48:41.527 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.527 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.527 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:48:41.527 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:58.166555+0000", 2024-09-20T22:48:41.527 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0082786119999999994 2024-09-20T22:48:41.527 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.527 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.528 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:48:41.528 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:58.168271+0000", 2024-09-20T22:48:41.528 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001716062 2024-09-20T22:48:41.528 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.528 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.528 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:48:41.528 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:58.168658+0000", 2024-09-20T22:48:41.528 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000387563 2024-09-20T22:48:41.528 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.528 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.528 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:48:41.528 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:58.168693+0000", 2024-09-20T22:48:41.528 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.4870999999999999e-05 2024-09-20T22:48:41.528 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.528 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.529 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:48:41.529 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:58.168738+0000", 2024-09-20T22:48:41.529 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.4984999999999997e-05 2024-09-20T22:48:41.529 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:41.529 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:48:41.529 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:41.529 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.529 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.529 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:1079 7.f 7:fa55e5ae:::smithi08477203-7:head [copy-from ver 87] snapc 26=[26,24] ondisk+write+known_if_redirected+supports_pool_eio e170)", 2024-09-20T22:48:41.529 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:44:09.184851+0000", 2024-09-20T22:48:41.529 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 272.29333491900002, 2024-09-20T22:48:41.529 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.028346902, 2024-09-20T22:48:41.529 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:48:41.529 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:48:41.530 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:48:41.530 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:48:41.530 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:48:41.530 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:48:41.530 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1079 2024-09-20T22:48:41.530 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.530 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:48:41.530 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.530 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:48:41.530 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:09.184851+0000", 2024-09-20T22:48:41.530 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:48:41.530 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.530 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.530 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:48:41.530 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:09.184851+0000", 2024-09-20T22:48:41.531 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:48:41.531 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.531 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.531 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:48:41.531 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:09.184852+0000", 2024-09-20T22:48:41.531 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.3399999999999999e-07 2024-09-20T22:48:41.531 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.531 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.531 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:48:41.531 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:09.184888+0000", 2024-09-20T22:48:41.531 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.6112000000000003e-05 2024-09-20T22:48:41.531 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.531 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.531 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:48:41.531 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:09.184895+0000", 2024-09-20T22:48:41.532 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.8539999999999999e-06 2024-09-20T22:48:41.532 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.532 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.532 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:48:41.532 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:09.184926+0000", 2024-09-20T22:48:41.532 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.1293e-05 2024-09-20T22:48:41.532 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.532 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.532 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:48:41.532 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:09.188919+0000", 2024-09-20T22:48:41.533 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0039926830000000003 2024-09-20T22:48:41.533 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.533 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.533 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:48:41.533 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:09.189012+0000", 2024-09-20T22:48:41.533 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.3690999999999995e-05 2024-09-20T22:48:41.533 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.533 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.533 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 3,7", 2024-09-20T22:48:41.533 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:09.205339+0000", 2024-09-20T22:48:41.533 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.016326092 2024-09-20T22:48:41.533 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.533 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.534 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:48:41.534 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:09.209753+0000", 2024-09-20T22:48:41.534 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0044144789999999998 2024-09-20T22:48:41.534 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.534 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.534 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:48:41.534 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:09.212376+0000", 2024-09-20T22:48:41.534 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026232180000000001 2024-09-20T22:48:41.534 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.534 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.534 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:48:41.534 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:09.213093+0000", 2024-09-20T22:48:41.534 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00071704400000000001 2024-09-20T22:48:41.534 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.535 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.535 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:48:41.535 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:09.213141+0000", 2024-09-20T22:48:41.535 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.7991999999999997e-05 2024-09-20T22:48:41.535 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.535 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.535 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:48:41.535 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:09.213198+0000", 2024-09-20T22:48:41.535 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.6910000000000002e-05 2024-09-20T22:48:41.535 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:41.535 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:48:41.535 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:41.535 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.535 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.535 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:1080 7.f 7:fa55e5ae:::smithi08477203-7:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e170)", 2024-09-20T22:48:41.536 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:44:09.184935+0000", 2024-09-20T22:48:41.536 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 272.29325122699998, 2024-09-20T22:48:41.536 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.024005572999999999, 2024-09-20T22:48:41.536 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:48:41.536 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:48:41.536 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-20T22:48:41.536 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:48:41.536 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:48:41.536 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:48:41.536 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1080 2024-09-20T22:48:41.536 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.536 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:48:41.536 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.536 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:48:41.537 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:09.184935+0000", 2024-09-20T22:48:41.537 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:48:41.537 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.537 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.537 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:48:41.537 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:09.184935+0000", 2024-09-20T22:48:41.537 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:48:41.537 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.537 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.537 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:48:41.537 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:09.184936+0000", 2024-09-20T22:48:41.537 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.6599999999999996e-07 2024-09-20T22:48:41.537 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.537 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.537 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:48:41.538 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:09.184938+0000", 2024-09-20T22:48:41.538 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.5359999999999999e-06 2024-09-20T22:48:41.538 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.538 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.538 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:48:41.538 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:09.184971+0000", 2024-09-20T22:48:41.538 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.2848000000000001e-05 2024-09-20T22:48:41.538 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.538 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.538 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:48:41.538 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:09.184977+0000", 2024-09-20T22:48:41.538 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.0859999999999998e-06 2024-09-20T22:48:41.538 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.538 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.538 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:48:41.539 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:09.189687+0000", 2024-09-20T22:48:41.539 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0047097700000000003 2024-09-20T22:48:41.539 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.539 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.539 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for blocked object", 2024-09-20T22:48:41.539 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:09.189852+0000", 2024-09-20T22:48:41.539 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000165243 2024-09-20T22:48:41.539 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.539 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.539 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:48:41.539 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:09.208617+0000", 2024-09-20T22:48:41.539 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.018765190000000001 2024-09-20T22:48:41.539 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.539 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.539 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:48:41.540 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:09.208772+0000", 2024-09-20T22:48:41.540 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000154318 2024-09-20T22:48:41.540 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.540 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.540 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:48:41.540 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:09.208941+0000", 2024-09-20T22:48:41.540 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000169016 2024-09-20T22:48:41.540 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:41.540 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:48:41.540 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:41.540 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.541 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.541 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:1583 7.18 7:1e9e2524:::smithi08477203-5:head [copy-from ver 154] snapc 35=[35] ondisk+write+known_if_redirected+supports_pool_eio e201)", 2024-09-20T22:48:41.541 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:44:40.229595+0000", 2024-09-20T22:48:41.541 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 241.248590984, 2024-09-20T22:48:41.541 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.026717541000000001, 2024-09-20T22:48:41.541 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:48:41.541 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:48:41.541 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:48:41.541 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:48:41.541 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:48:41.541 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:48:41.541 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1583 2024-09-20T22:48:41.541 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.541 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:48:41.542 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.542 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:48:41.542 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:40.229595+0000", 2024-09-20T22:48:41.542 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:48:41.542 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.542 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.542 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:48:41.542 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:40.229595+0000", 2024-09-20T22:48:41.542 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:48:41.542 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.542 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.542 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:48:41.542 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:40.229596+0000", 2024-09-20T22:48:41.542 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.68e-07 2024-09-20T22:48:41.543 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.543 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.543 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:48:41.543 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:40.229600+0000", 2024-09-20T22:48:41.543 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.805e-06 2024-09-20T22:48:41.543 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.543 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.543 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:48:41.543 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:40.229606+0000", 2024-09-20T22:48:41.543 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.4300000000000003e-06 2024-09-20T22:48:41.543 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.543 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.543 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:48:41.543 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:40.229611+0000", 2024-09-20T22:48:41.544 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.0250000000000002e-06 2024-09-20T22:48:41.544 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.544 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.544 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:48:41.544 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:40.229661+0000", 2024-09-20T22:48:41.544 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.0343999999999998e-05 2024-09-20T22:48:41.544 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.544 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.544 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:48:41.544 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:40.229737+0000", 2024-09-20T22:48:41.544 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.5444000000000005e-05 2024-09-20T22:48:41.544 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.544 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.544 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 6,7", 2024-09-20T22:48:41.544 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:40.244365+0000", 2024-09-20T22:48:41.545 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014628238999999999 2024-09-20T22:48:41.545 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.545 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.545 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:48:41.545 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:40.249487+0000", 2024-09-20T22:48:41.545 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0051221799999999996 2024-09-20T22:48:41.545 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.545 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.545 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:48:41.545 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:40.252864+0000", 2024-09-20T22:48:41.545 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0033763830000000002 2024-09-20T22:48:41.545 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.545 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.545 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:48:41.546 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:40.256237+0000", 2024-09-20T22:48:41.546 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0033731550000000001 2024-09-20T22:48:41.546 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.546 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.546 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:48:41.546 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:40.256277+0000", 2024-09-20T22:48:41.546 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.0092999999999997e-05 2024-09-20T22:48:41.546 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.546 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.546 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:48:41.546 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:40.256313+0000", 2024-09-20T22:48:41.546 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.5874999999999998e-05 2024-09-20T22:48:41.546 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:41.547 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:48:41.547 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:41.547 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.547 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.547 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:1584 7.18 7:1e9e2524:::smithi08477203-5:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e201)", 2024-09-20T22:48:41.547 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:44:40.229623+0000", 2024-09-20T22:48:41.547 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 241.24856328999999, 2024-09-20T22:48:41.547 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.019097652, 2024-09-20T22:48:41.547 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:48:41.547 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:48:41.547 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-20T22:48:41.547 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:48:41.547 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:48:41.547 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:48:41.548 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1584 2024-09-20T22:48:41.548 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.548 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:48:41.548 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.548 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:48:41.548 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:40.229623+0000", 2024-09-20T22:48:41.548 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:48:41.548 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.548 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.548 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:48:41.548 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:40.229623+0000", 2024-09-20T22:48:41.548 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:48:41.548 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.548 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.548 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:48:41.548 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:40.229623+0000", 2024-09-20T22:48:41.549 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.7300000000000002e-07 2024-09-20T22:48:41.549 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.549 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.549 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:48:41.549 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:40.229625+0000", 2024-09-20T22:48:41.549 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.199e-06 2024-09-20T22:48:41.549 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.549 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.549 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:48:41.549 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:40.229631+0000", 2024-09-20T22:48:41.549 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.1599999999999997e-06 2024-09-20T22:48:41.549 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.549 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.549 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:48:41.549 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:40.229636+0000", 2024-09-20T22:48:41.550 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.7780000000000004e-06 2024-09-20T22:48:41.550 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.550 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.550 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:48:41.550 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:40.229830+0000", 2024-09-20T22:48:41.550 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00019321 2024-09-20T22:48:41.550 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.550 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.550 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for blocked object", 2024-09-20T22:48:41.550 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:40.229857+0000", 2024-09-20T22:48:41.550 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.7030999999999999e-05 2024-09-20T22:48:41.550 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.550 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.550 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:48:41.551 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:40.248577+0000", 2024-09-20T22:48:41.551 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.018720554 2024-09-20T22:48:41.551 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.551 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.551 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:48:41.551 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:40.248643+0000", 2024-09-20T22:48:41.551 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.5361e-05 2024-09-20T22:48:41.551 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.551 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.551 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:48:41.551 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:40.248721+0000", 2024-09-20T22:48:41.551 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.7985999999999995e-05 2024-09-20T22:48:41.551 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:41.551 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:48:41.552 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:41.552 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.552 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.552 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:1809 7.16 7:6885a65f:::smithi08477203-14:head [delete] snapc 46=[45,44,43,42,41,40,3e] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e229)", 2024-09-20T22:48:41.552 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:45:08.177381+0000", 2024-09-20T22:48:41.552 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 213.30080543599999, 2024-09-20T22:48:41.552 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.01285646, 2024-09-20T22:48:41.552 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:48:41.552 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:48:41.552 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:48:41.552 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:48:41.552 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:48:41.552 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:48:41.553 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1809 2024-09-20T22:48:41.553 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.553 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:48:41.553 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.553 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:48:41.553 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.177381+0000", 2024-09-20T22:48:41.553 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:48:41.553 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.553 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.553 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:48:41.553 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.177381+0000", 2024-09-20T22:48:41.553 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:48:41.553 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.554 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.554 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:48:41.554 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.177383+0000", 2024-09-20T22:48:41.554 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.1399999999999998e-06 2024-09-20T22:48:41.554 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.554 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.554 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:48:41.554 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.177391+0000", 2024-09-20T22:48:41.554 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.7319999999999998e-06 2024-09-20T22:48:41.554 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.554 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.554 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:48:41.554 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.177415+0000", 2024-09-20T22:48:41.554 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.3879999999999998e-05 2024-09-20T22:48:41.555 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.555 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.555 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:48:41.555 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.177425+0000", 2024-09-20T22:48:41.555 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0509999999999999e-05 2024-09-20T22:48:41.555 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.555 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.555 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:48:41.555 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.177494+0000", 2024-09-20T22:48:41.555 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.8544e-05 2024-09-20T22:48:41.555 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.555 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.555 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:48:41.555 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.177590+0000", 2024-09-20T22:48:41.555 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.6729000000000002e-05 2024-09-20T22:48:41.556 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.556 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.556 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 2,7", 2024-09-20T22:48:41.556 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.177798+0000", 2024-09-20T22:48:41.556 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000207398 2024-09-20T22:48:41.556 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.556 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.556 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:48:41.556 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.179598+0000", 2024-09-20T22:48:41.556 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0018002160000000001 2024-09-20T22:48:41.556 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.556 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.556 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:48:41.556 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.186392+0000", 2024-09-20T22:48:41.557 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.006794423 2024-09-20T22:48:41.557 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.557 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.557 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:48:41.557 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.190106+0000", 2024-09-20T22:48:41.557 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0037136109999999999 2024-09-20T22:48:41.557 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.557 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.557 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:48:41.557 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.190180+0000", 2024-09-20T22:48:41.557 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.3831000000000005e-05 2024-09-20T22:48:41.557 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.557 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.557 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:48:41.557 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.190237+0000", 2024-09-20T22:48:41.558 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.7445999999999999e-05 2024-09-20T22:48:41.558 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:41.558 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:48:41.558 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:41.558 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.558 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.558 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:3965 7.f 7:fa55e5ae:::smithi08477203-7:head [copy-from ver 838] snapc a8=[a8,a7,a6,a5,a4,a3,a0,9f,9e,9d,9c,9b,9a,96,95,94,93,92,8e,8d,8a,88,87,86,80,7f,73] ondisk+write+known_if_redirected+supports_pool_eio e406)", 2024-09-20T22:48:41.558 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:48:06.215099+0000", 2024-09-20T22:48:41.558 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 35.263087665, 2024-09-20T22:48:41.558 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.021929697000000001, 2024-09-20T22:48:41.558 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:48:41.558 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:48:41.558 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:48:41.558 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:48:41.558 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:48:41.559 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:48:41.559 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 3965 2024-09-20T22:48:41.559 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.559 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:48:41.559 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.559 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:48:41.559 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:06.215099+0000", 2024-09-20T22:48:41.559 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:48:41.559 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.559 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.559 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:48:41.559 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:06.215099+0000", 2024-09-20T22:48:41.559 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:48:41.560 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.560 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.560 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:48:41.560 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:06.215100+0000", 2024-09-20T22:48:41.560 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.6980000000000001e-06 2024-09-20T22:48:41.560 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.560 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.560 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:48:41.560 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:06.215107+0000", 2024-09-20T22:48:41.560 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.2990000000000002e-06 2024-09-20T22:48:41.560 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.560 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.560 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:48:41.561 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:06.215118+0000", 2024-09-20T22:48:41.561 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.173e-05 2024-09-20T22:48:41.561 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.561 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.561 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:48:41.561 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:06.215128+0000", 2024-09-20T22:48:41.561 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.5920000000000007e-06 2024-09-20T22:48:41.561 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.561 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.561 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:48:41.561 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:06.215190+0000", 2024-09-20T22:48:41.561 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.1970000000000005e-05 2024-09-20T22:48:41.561 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.561 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.562 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:48:41.562 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:06.215295+0000", 2024-09-20T22:48:41.562 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000105027 2024-09-20T22:48:41.562 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.562 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.562 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 3,7", 2024-09-20T22:48:41.562 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:06.226459+0000", 2024-09-20T22:48:41.562 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011164431000000001 2024-09-20T22:48:41.562 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.562 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.562 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:48:41.562 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:06.231688+0000", 2024-09-20T22:48:41.562 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0052288739999999997 2024-09-20T22:48:41.562 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.563 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.563 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:48:41.563 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:06.235132+0000", 2024-09-20T22:48:41.563 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0034440859999999998 2024-09-20T22:48:41.563 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.563 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.563 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:48:41.563 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:06.236877+0000", 2024-09-20T22:48:41.563 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00174515 2024-09-20T22:48:41.563 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.563 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.563 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:48:41.563 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:06.236974+0000", 2024-09-20T22:48:41.563 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.6951000000000001e-05 2024-09-20T22:48:41.563 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.564 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.564 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:48:41.564 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:06.237028+0000", 2024-09-20T22:48:41.564 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.3888999999999999e-05 2024-09-20T22:48:41.564 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:41.564 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:48:41.564 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:41.564 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.564 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.564 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:3966 7.f 7:fa55e5ae:::smithi08477203-7:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e406)", 2024-09-20T22:48:41.564 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:48:06.215181+0000", 2024-09-20T22:48:41.564 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 35.263004916, 2024-09-20T22:48:41.564 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.015666863, 2024-09-20T22:48:41.564 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:48:41.565 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:48:41.565 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-20T22:48:41.565 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:48:41.565 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:48:41.565 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:48:41.565 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 3966 2024-09-20T22:48:41.565 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.565 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:48:41.565 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.565 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:48:41.565 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:06.215181+0000", 2024-09-20T22:48:41.565 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:48:41.565 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.565 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.565 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:48:41.566 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:06.215181+0000", 2024-09-20T22:48:41.566 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:48:41.566 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.566 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.566 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:48:41.566 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:06.215182+0000", 2024-09-20T22:48:41.566 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.0299999999999998e-07 2024-09-20T22:48:41.566 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.566 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.566 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:48:41.566 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:06.215186+0000", 2024-09-20T22:48:41.566 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.9659999999999998e-06 2024-09-20T22:48:41.566 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.566 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.567 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:48:41.567 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:06.215194+0000", 2024-09-20T22:48:41.567 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.2779999999999993e-06 2024-09-20T22:48:41.567 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.567 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.567 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:48:41.567 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:06.215200+0000", 2024-09-20T22:48:41.567 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.9220000000000002e-06 2024-09-20T22:48:41.567 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.567 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.567 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:48:41.567 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:06.215511+0000", 2024-09-20T22:48:41.568 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00031064500000000002 2024-09-20T22:48:41.568 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.568 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.568 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for blocked object", 2024-09-20T22:48:41.568 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:06.215557+0000", 2024-09-20T22:48:41.568 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.5668999999999999e-05 2024-09-20T22:48:41.568 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.568 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.568 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:48:41.568 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:06.230683+0000", 2024-09-20T22:48:41.568 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.015126094 2024-09-20T22:48:41.568 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.568 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.568 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:48:41.568 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:06.230778+0000", 2024-09-20T22:48:41.569 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.5699000000000006e-05 2024-09-20T22:48:41.569 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.569 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.569 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:48:41.569 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:06.230848+0000", 2024-09-20T22:48:41.569 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.9787000000000001e-05 2024-09-20T22:48:41.569 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:41.569 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:48:41.569 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:41.569 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.569 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.569 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:4229 7.14 7:2f8647a2:::smithi08477203-41:head [copy-from ver 414] snapc ae=[ae,ad,ac,ab,a9,a7,a6,a5,9f,9e,9d,9c,9b,9a,96,95,94,8e,8d,87,86,80,73] ondisk+write+known_if_redirected+supports_pool_eio e423)", 2024-09-20T22:48:41.569 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:48:23.291424+0000", 2024-09-20T22:48:41.569 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 18.18676206, 2024-09-20T22:48:41.569 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.017026243999999999, 2024-09-20T22:48:41.570 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:48:41.570 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:48:41.570 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:48:41.570 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:48:41.570 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:48:41.570 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:48:41.570 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 4229 2024-09-20T22:48:41.570 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.570 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:48:41.570 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.570 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:48:41.570 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:23.291424+0000", 2024-09-20T22:48:41.570 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:48:41.570 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.571 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.571 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:48:41.571 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:23.291424+0000", 2024-09-20T22:48:41.571 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:48:41.571 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.571 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.571 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:48:41.571 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:23.291425+0000", 2024-09-20T22:48:41.571 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.8500000000000002e-07 2024-09-20T22:48:41.571 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.571 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.571 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:48:41.571 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:23.291429+0000", 2024-09-20T22:48:41.571 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.924e-06 2024-09-20T22:48:41.571 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.572 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.572 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:48:41.572 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:23.291452+0000", 2024-09-20T22:48:41.572 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.3728999999999999e-05 2024-09-20T22:48:41.572 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.572 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.572 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:48:41.572 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:23.291458+0000", 2024-09-20T22:48:41.572 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.8390000000000003e-06 2024-09-20T22:48:41.572 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.572 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.572 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:48:41.572 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:23.291501+0000", 2024-09-20T22:48:41.572 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.2577999999999998e-05 2024-09-20T22:48:41.572 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.573 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.573 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:48:41.573 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:23.291610+0000", 2024-09-20T22:48:41.573 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000108848 2024-09-20T22:48:41.573 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.573 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.573 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 1,4", 2024-09-20T22:48:41.573 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:23.296527+0000", 2024-09-20T22:48:41.573 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0049178429999999999 2024-09-20T22:48:41.573 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.573 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.573 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:48:41.573 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:23.306032+0000", 2024-09-20T22:48:41.573 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0095044080000000006 2024-09-20T22:48:41.574 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.574 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.574 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:48:41.574 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:23.306306+0000", 2024-09-20T22:48:41.574 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00027432999999999998 2024-09-20T22:48:41.574 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.574 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.574 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:48:41.574 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:23.308371+0000", 2024-09-20T22:48:41.574 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0020648379999999998 2024-09-20T22:48:41.574 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.574 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.574 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:48:41.574 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:23.308412+0000", 2024-09-20T22:48:41.574 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.0867e-05 2024-09-20T22:48:41.575 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.575 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.575 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:48:41.575 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:23.308450+0000", 2024-09-20T22:48:41.575 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.8554999999999999e-05 2024-09-20T22:48:41.575 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:41.575 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:48:41.575 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:41.575 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.575 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.575 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:4311 7.17 7:ed54065f:::smithi08477203-12:head [append 0~0] snapc b1=[b1,b0,af,ae,ad,ab,a9,a7,a6,a5,9f,9e,9d,9c,9a,96,95,8e,87,86] ondisk+write+known_if_redirected+supports_pool_eio e433)", 2024-09-20T22:48:41.575 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:48:33.213556+0000", 2024-09-20T22:48:41.575 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 8.2646298139999992, 2024-09-20T22:48:41.575 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0280906110000001, 2024-09-20T22:48:41.576 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:48:41.576 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:48:41.576 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:48:41.576 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:48:41.576 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:48:41.576 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:48:41.576 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 4311 2024-09-20T22:48:41.576 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.576 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:48:41.576 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.576 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:48:41.576 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:33.213556+0000", 2024-09-20T22:48:41.576 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:48:41.576 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.576 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.577 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:48:41.577 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:33.213556+0000", 2024-09-20T22:48:41.577 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:48:41.577 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.577 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.577 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:48:41.577 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:33.213558+0000", 2024-09-20T22:48:41.577 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.4470000000000001e-06 2024-09-20T22:48:41.577 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.577 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.577 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:48:41.577 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:33.213563+0000", 2024-09-20T22:48:41.577 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.6840000000000004e-06 2024-09-20T22:48:41.577 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.577 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.578 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:48:41.578 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:33.213573+0000", 2024-09-20T22:48:41.578 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0873000000000001e-05 2024-09-20T22:48:41.578 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.578 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.578 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:48:41.578 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:33.213584+0000", 2024-09-20T22:48:41.578 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0852e-05 2024-09-20T22:48:41.578 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.578 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.578 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:48:41.578 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:33.213633+0000", 2024-09-20T22:48:41.578 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.8683999999999997e-05 2024-09-20T22:48:41.578 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.579 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.579 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-09-20T22:48:41.579 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:33.213646+0000", 2024-09-20T22:48:41.579 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3205e-05 2024-09-20T22:48:41.579 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.579 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.579 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:48:41.579 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:34.226325+0000", 2024-09-20T22:48:41.579 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0126793300000001 2024-09-20T22:48:41.579 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.579 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.579 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for degraded object", 2024-09-20T22:48:41.579 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:34.228377+0000", 2024-09-20T22:48:41.579 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0020512540000000002 2024-09-20T22:48:41.579 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.580 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.580 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:48:41.580 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:34.239166+0000", 2024-09-20T22:48:41.580 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010789029 2024-09-20T22:48:41.580 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.580 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:48:41.581 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.581 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:48:41.581 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:34.239275+0000", 2024-09-20T22:48:41.581 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00010938900000000001 2024-09-20T22:48:41.581 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.581 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.581 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 3,6", 2024-09-20T22:48:41.582 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:34.239569+0000", 2024-09-20T22:48:41.582 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00029410699999999999 2024-09-20T22:48:41.582 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.582 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.582 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:48:41.582 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:34.241371+0000", 2024-09-20T22:48:41.582 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00180171 2024-09-20T22:48:41.582 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.582 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.582 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:48:41.582 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:34.241393+0000", 2024-09-20T22:48:41.582 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.1928000000000001e-05 2024-09-20T22:48:41.582 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.582 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.582 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:48:41.583 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:34.241465+0000", 2024-09-20T22:48:41.583 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.1799999999999997e-05 2024-09-20T22:48:41.583 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.583 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.583 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:48:41.583 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:34.241528+0000", 2024-09-20T22:48:41.583 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.2946000000000003e-05 2024-09-20T22:48:41.583 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.583 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.583 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:48:41.583 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:34.241647+0000", 2024-09-20T22:48:41.583 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000119373 2024-09-20T22:48:41.583 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:41.583 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:48:41.583 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:41.584 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.584 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.584 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:4312 7.17 7:ed54065f:::smithi08477203-12:head [rollback 96] snapc b1=[b1,b0,af,ae,ad,ab,a9,a7,a6,a5,9f,9e,9d,9c,9a,96,95,8e,87,86] ondisk+write+known_if_redirected+supports_pool_eio e433)", 2024-09-20T22:48:41.584 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:48:33.213603+0000", 2024-09-20T22:48:41.584 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 8.2645833920000005, 2024-09-20T22:48:41.584 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0286283940000001, 2024-09-20T22:48:41.584 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:48:41.584 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:48:41.584 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:48:41.584 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:48:41.584 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:48:41.584 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:48:41.584 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 4312 2024-09-20T22:48:41.585 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.585 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:48:41.585 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.585 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:48:41.585 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:33.213603+0000", 2024-09-20T22:48:41.585 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:48:41.585 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.585 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.585 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:48:41.585 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:33.213603+0000", 2024-09-20T22:48:41.585 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:48:41.585 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.585 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.585 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:48:41.585 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:33.213603+0000", 2024-09-20T22:48:41.585 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.9200000000000001e-07 2024-09-20T22:48:41.586 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.586 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.586 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:48:41.586 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:33.213606+0000", 2024-09-20T22:48:41.586 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.6390000000000001e-06 2024-09-20T22:48:41.586 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.586 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.586 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:48:41.586 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:33.213612+0000", 2024-09-20T22:48:41.586 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.5429999999999999e-06 2024-09-20T22:48:41.586 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.586 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.586 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:48:41.586 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:33.213618+0000", 2024-09-20T22:48:41.586 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.4249999999999997e-06 2024-09-20T22:48:41.586 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.587 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.587 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:48:41.587 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:33.213674+0000", 2024-09-20T22:48:41.587 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.6341000000000001e-05 2024-09-20T22:48:41.587 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.587 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.587 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-09-20T22:48:41.587 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:33.214118+0000", 2024-09-20T22:48:41.587 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00044406600000000001 2024-09-20T22:48:41.587 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.587 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.587 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:48:41.587 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:34.228938+0000", 2024-09-20T22:48:41.587 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0148198470000001 2024-09-20T22:48:41.587 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.588 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.588 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for degraded object", 2024-09-20T22:48:41.588 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:34.229016+0000", 2024-09-20T22:48:41.588 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.7383000000000001e-05 2024-09-20T22:48:41.588 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.588 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.588 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:48:41.588 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:34.240634+0000", 2024-09-20T22:48:41.588 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011618277999999999 2024-09-20T22:48:41.588 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.588 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.588 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:48:41.588 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:34.240749+0000", 2024-09-20T22:48:41.588 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00011473500000000001 2024-09-20T22:48:41.589 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.589 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.589 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 3,6", 2024-09-20T22:48:41.589 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:34.241041+0000", 2024-09-20T22:48:41.589 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00029246699999999998 2024-09-20T22:48:41.589 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.589 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.589 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:48:41.589 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:34.242022+0000", 2024-09-20T22:48:41.589 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00098068999999999999 2024-09-20T22:48:41.589 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.589 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.589 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:48:41.590 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:34.242068+0000", 2024-09-20T22:48:41.590 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.6581e-05 2024-09-20T22:48:41.590 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.590 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.590 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:48:41.590 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:34.242081+0000", 2024-09-20T22:48:41.590 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2709e-05 2024-09-20T22:48:41.590 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.590 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.590 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:48:41.590 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:34.242116+0000", 2024-09-20T22:48:41.590 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.4579000000000002e-05 2024-09-20T22:48:41.590 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.591 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.591 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:48:41.591 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:34.242231+0000", 2024-09-20T22:48:41.591 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00011561900000000001 2024-09-20T22:48:41.591 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:41.591 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:48:41.591 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:41.591 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.591 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.591 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:4318 7.f 7:fa55e5ae:::smithi08477203-7:head [copy-from ver 556] snapc b1=[b1,b0,af,ae,ad,ab,a9,a7,a6,a5,9f,9e,9d,9c,9a,96,95,8e,87,86] ondisk+write+known_if_redirected+supports_pool_eio e433)", 2024-09-20T22:48:41.591 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:48:33.213673+0000", 2024-09-20T22:48:41.591 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 8.2645131319999994, 2024-09-20T22:48:41.591 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.019496726999999998, 2024-09-20T22:48:41.591 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:48:41.592 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:48:41.592 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:48:41.592 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:48:41.592 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:48:41.592 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:48:41.592 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 4318 2024-09-20T22:48:41.592 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.592 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:48:41.592 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.592 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:48:41.592 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:33.213673+0000", 2024-09-20T22:48:41.592 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:48:41.592 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.592 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.593 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:48:41.593 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:33.213673+0000", 2024-09-20T22:48:41.593 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:48:41.593 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.593 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.593 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:48:41.593 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:33.213674+0000", 2024-09-20T22:48:41.593 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.6600000000000002e-07 2024-09-20T22:48:41.593 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.593 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.593 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:48:41.593 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:33.213678+0000", 2024-09-20T22:48:41.593 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.3229999999999999e-06 2024-09-20T22:48:41.593 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.593 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.593 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:48:41.593 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:33.213685+0000", 2024-09-20T22:48:41.594 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.1400000000000002e-06 2024-09-20T22:48:41.594 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.594 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.594 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:48:41.594 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:33.213691+0000", 2024-09-20T22:48:41.594 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.7819999999999999e-06 2024-09-20T22:48:41.594 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.594 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.594 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:48:41.594 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:33.214153+0000", 2024-09-20T22:48:41.594 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00046270099999999999 2024-09-20T22:48:41.594 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.594 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.594 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:48:41.595 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:33.214276+0000", 2024-09-20T22:48:41.595 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00012271999999999999 2024-09-20T22:48:41.595 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.595 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.595 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 3,7", 2024-09-20T22:48:41.595 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:33.220974+0000", 2024-09-20T22:48:41.595 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0066979199999999996 2024-09-20T22:48:41.595 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.595 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.595 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:48:41.595 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:33.230425+0000", 2024-09-20T22:48:41.595 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0094512309999999992 2024-09-20T22:48:41.595 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.595 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.595 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:48:41.596 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:33.232841+0000", 2024-09-20T22:48:41.596 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0024154340000000002 2024-09-20T22:48:41.596 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.596 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.596 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:48:41.596 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:33.233039+0000", 2024-09-20T22:48:41.596 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000197939 2024-09-20T22:48:41.596 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.596 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.596 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:48:41.596 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:33.233108+0000", 2024-09-20T22:48:41.596 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.8956999999999997e-05 2024-09-20T22:48:41.596 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.596 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.596 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:48:41.597 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:33.233170+0000", 2024-09-20T22:48:41.597 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.2113999999999995e-05 2024-09-20T22:48:41.597 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:41.597 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:48:41.597 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:41.597 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.597 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.597 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:4319 7.f 7:fa55e5ae:::smithi08477203-7:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e433)", 2024-09-20T22:48:41.597 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:48:33.213698+0000", 2024-09-20T22:48:41.597 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 8.2644883319999991, 2024-09-20T22:48:41.597 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013433402000000001, 2024-09-20T22:48:41.597 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:48:41.597 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:48:41.597 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-20T22:48:41.597 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:48:41.598 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:48:41.598 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:48:41.598 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 4319 2024-09-20T22:48:41.598 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.598 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:48:41.598 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.598 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:48:41.598 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:33.213698+0000", 2024-09-20T22:48:41.598 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:48:41.598 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.598 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.598 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:48:41.598 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:33.213698+0000", 2024-09-20T22:48:41.599 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:48:41.599 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.599 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.599 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:48:41.599 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:33.213698+0000", 2024-09-20T22:48:41.599 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.9700000000000002e-07 2024-09-20T22:48:41.599 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.599 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.599 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:48:41.599 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:33.213700+0000", 2024-09-20T22:48:41.599 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.075e-06 2024-09-20T22:48:41.599 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.599 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.599 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:48:41.599 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:33.213710+0000", 2024-09-20T22:48:41.600 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0045e-05 2024-09-20T22:48:41.600 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.600 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.600 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:48:41.600 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:33.213716+0000", 2024-09-20T22:48:41.600 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.0129999999999999e-06 2024-09-20T22:48:41.600 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.600 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.600 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:48:41.600 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:33.214450+0000", 2024-09-20T22:48:41.600 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00073312400000000004 2024-09-20T22:48:41.600 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.600 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.600 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for blocked object", 2024-09-20T22:48:41.600 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:33.214505+0000", 2024-09-20T22:48:41.601 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.5983999999999998e-05 2024-09-20T22:48:41.601 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.601 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.601 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:48:41.601 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:33.226868+0000", 2024-09-20T22:48:41.601 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.012362102 2024-09-20T22:48:41.601 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.601 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.601 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:48:41.601 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:33.226995+0000", 2024-09-20T22:48:41.601 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00012702500000000001 2024-09-20T22:48:41.601 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:41.601 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:41.601 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:48:41.601 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:33.227131+0000", 2024-09-20T22:48:41.602 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000136637 2024-09-20T22:48:41.602 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:41.602 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:48:41.602 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:41.602 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:41.602 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:48:41.602 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-20T22:48:41.626 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:41.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:41.627 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:48:41.728 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:48:41.752 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:41.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:41.753 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:48:41.854 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:48:41.878 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:41.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:41.879 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:48:41.980 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:48:42.002 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:42.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:42.003 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:48:42.104 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:48:42.130 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:42.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:42.131 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:48:42.215 INFO:tasks.rados.rados.0.smithi084.stdout:1715: done (4 left) 2024-09-20T22:48:42.216 INFO:tasks.rados.rados.0.smithi084.stdout:1716: snap_remove snap 157 2024-09-20T22:48:42.231 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:48:42.255 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:42.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:42.256 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:48:42.356 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:48:42.380 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:42.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:42.381 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:48:42.482 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:48:42.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:42 smithi154 ceph-mon[33264]: 7.6 deep-scrub starts 2024-09-20T22:48:42.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:42 smithi154 ceph-mon[33264]: 7.6 deep-scrub ok 2024-09-20T22:48:42.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:42 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/578766514' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:48:42.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:42 smithi154 ceph-mon[33264]: 3.19 scrub starts 2024-09-20T22:48:42.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:42 smithi154 ceph-mon[33264]: 3.19 scrub ok 2024-09-20T22:48:42.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:42 smithi154 ceph-mon[33264]: osdmap e442: 8 total, 8 up, 8 in 2024-09-20T22:48:42.505 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:42.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:42.505 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:48:42.571 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:42 smithi084 ceph-mon[40024]: 7.6 deep-scrub starts 2024-09-20T22:48:42.571 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:42 smithi084 ceph-mon[40024]: 7.6 deep-scrub ok 2024-09-20T22:48:42.571 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:42 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/578766514' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:48:42.571 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:42 smithi084 ceph-mon[40024]: 3.19 scrub starts 2024-09-20T22:48:42.571 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:42 smithi084 ceph-mon[40024]: 3.19 scrub ok 2024-09-20T22:48:42.571 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:42 smithi084 ceph-mon[40024]: osdmap e442: 8 total, 8 up, 8 in 2024-09-20T22:48:42.572 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:42 smithi084 ceph-mon[29774]: 7.6 deep-scrub starts 2024-09-20T22:48:42.572 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:42 smithi084 ceph-mon[29774]: 7.6 deep-scrub ok 2024-09-20T22:48:42.572 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:42 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/578766514' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:48:42.572 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:42 smithi084 ceph-mon[29774]: 3.19 scrub starts 2024-09-20T22:48:42.572 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:42 smithi084 ceph-mon[29774]: 3.19 scrub ok 2024-09-20T22:48:42.572 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:42 smithi084 ceph-mon[29774]: osdmap e442: 8 total, 8 up, 8 in 2024-09-20T22:48:42.606 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:48:42.628 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:42.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:42.629 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:48:42.730 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:48:42.756 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:42.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:42.757 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:48:42.857 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:48:42.884 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:42.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:42.885 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:48:42.986 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:48:43.009 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:43.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:43.010 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:48:43.110 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:48:43.136 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:43.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:43.137 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:48:43.213 INFO:tasks.rados.rados.0.smithi084.stdout:1716: done (4 left) 2024-09-20T22:48:43.213 INFO:tasks.rados.rados.0.smithi084.stdout:1717: write (excl) oid 29 current snap is 181 2024-09-20T22:48:43.213 INFO:tasks.rados.rados.0.smithi084.stdout:1717: seq_num 607 ranges {601849=468030,1753864=447985} 2024-09-20T22:48:43.219 INFO:tasks.rados.rados.0.smithi084.stdout:1717: writing smithi08477203-29 from 601849 to 1069879 tid 1 2024-09-20T22:48:43.227 INFO:tasks.rados.rados.0.smithi084.stdout:1717: writing smithi08477203-29 from 1753864 to 2201849 tid 2 2024-09-20T22:48:43.227 INFO:tasks.rados.rados.0.smithi084.stdout:1718: read oid 4 snap -1 2024-09-20T22:48:43.228 INFO:tasks.rados.rados.0.smithi084.stdout:1718: expect deleted 2024-09-20T22:48:43.228 INFO:tasks.rados.rados.0.smithi084.stdout:1719: snap_create 2024-09-20T22:48:43.228 INFO:tasks.rados.rados.0.smithi084.stdout:1712: finishing write tid 2 to smithi08477203-15 2024-09-20T22:48:43.228 INFO:tasks.rados.rados.0.smithi084.stdout:1712: finishing write tid 3 to smithi08477203-15 2024-09-20T22:48:43.228 INFO:tasks.rados.rados.0.smithi084.stdout:1712: finishing write tid 4 to smithi08477203-15 2024-09-20T22:48:43.228 INFO:tasks.rados.rados.0.smithi084.stdout:1712: oid 15 updating version 0 to 738 2024-09-20T22:48:43.228 INFO:tasks.rados.rados.0.smithi084.stdout:1712: oid 15 updating version 738 to 739 2024-09-20T22:48:43.228 INFO:tasks.rados.rados.0.smithi084.stdout:1712: oid 15 updating version 739 to 740 2024-09-20T22:48:43.228 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 15 v 740 (ObjNum 605 snap 181 seq_num 605) dirty exists 2024-09-20T22:48:43.228 INFO:tasks.rados.rados.0.smithi084.stdout:1712: left oid 15 (ObjNum 605 snap 181 seq_num 605) 2024-09-20T22:48:43.228 INFO:tasks.rados.rados.0.smithi084.stdout:1713: finishing rollback tid 0 to smithi08477203-50 2024-09-20T22:48:43.228 INFO:tasks.rados.rados.0.smithi084.stdout:1713: finishing rollback tid 1 to smithi08477203-50 2024-09-20T22:48:43.229 INFO:tasks.rados.rados.0.smithi084.stdout:1713: finishing rollback tid 2 to smithi08477203-50 2024-09-20T22:48:43.229 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 50 v 258 (ObjNum 515 snap 148 seq_num 515) dirty exists 2024-09-20T22:48:43.229 INFO:tasks.rados.rados.0.smithi084.stdout:1714: finishing write tid 1 to smithi08477203-36 2024-09-20T22:48:43.229 INFO:tasks.rados.rados.0.smithi084.stdout:1714: finishing write tid 2 to smithi08477203-36 2024-09-20T22:48:43.229 INFO:tasks.rados.rados.0.smithi084.stdout:1714: finishing write tid 3 to smithi08477203-36 2024-09-20T22:48:43.229 INFO:tasks.rados.rados.0.smithi084.stdout:1714: finishing write tid 4 to smithi08477203-36 2024-09-20T22:48:43.229 INFO:tasks.rados.rados.0.smithi084.stdout:1714: oid 36 updating version 0 to 394 2024-09-20T22:48:43.229 INFO:tasks.rados.rados.0.smithi084.stdout:1714: oid 36 version 394 is already newer than 393 2024-09-20T22:48:43.229 INFO:tasks.rados.rados.0.smithi084.stdout:1714: oid 36 updating version 394 to 395 2024-09-20T22:48:43.229 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 36 v 395 (ObjNum 606 snap 181 seq_num 606) dirty exists 2024-09-20T22:48:43.229 INFO:tasks.rados.rados.0.smithi084.stdout:1714: left oid 36 (ObjNum 606 snap 181 seq_num 606) 2024-09-20T22:48:43.231 INFO:tasks.rados.rados.0.smithi084.stdout:1709: finishing rollback tid 1 to smithi08477203-26 2024-09-20T22:48:43.231 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 26 v 340 (ObjNum 129 snap 0 seq_num 2091167792) dirty dne 2024-09-20T22:48:43.233 INFO:tasks.rados.rados.0.smithi084.stdout:1717: finishing write tid 1 to smithi08477203-29 2024-09-20T22:48:43.236 INFO:tasks.rados.rados.0.smithi084.stdout:1717: finishing write tid 2 to smithi08477203-29 2024-09-20T22:48:43.236 INFO:tasks.rados.rados.0.smithi084.stdout:1717: finishing write tid 3 to smithi08477203-29 2024-09-20T22:48:43.236 INFO:tasks.rados.rados.0.smithi084.stdout:1717: finishing write tid 4 to smithi08477203-29 2024-09-20T22:48:43.237 INFO:tasks.rados.rados.0.smithi084.stdout:1717: oid 29 updating version 0 to 1069 2024-09-20T22:48:43.237 INFO:tasks.rados.rados.0.smithi084.stdout:1717: oid 29 updating version 1069 to 1070 2024-09-20T22:48:43.237 INFO:tasks.rados.rados.0.smithi084.stdout:1717: oid 29 version 1070 is already newer than 1064 2024-09-20T22:48:43.237 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 29 v 1070 (ObjNum 607 snap 181 seq_num 607) dirty exists 2024-09-20T22:48:43.237 INFO:tasks.rados.rados.0.smithi084.stdout:1717: left oid 29 (ObjNum 607 snap 181 seq_num 607) 2024-09-20T22:48:43.237 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:48:43.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:43.261 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:43.261 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:48:43.362 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:48:43.384 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:43.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:43.385 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:48:43.405 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-20T22:48:43.434 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:48:43.434 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:48:43.486 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:48:43.507 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:43.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:43.508 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:48:43.556 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:43 smithi084 ceph-mon[29774]: pgmap v564: 166 pgs: 3 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 158 active+clean; 359 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.7 MiB/s rd, 719 KiB/s wr, 8 op/s; 3.1 MiB/s, 7 objects/s recovering 2024-09-20T22:48:43.556 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:43 smithi084 ceph-mon[29774]: 5.19 scrub starts 2024-09-20T22:48:43.556 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:43 smithi084 ceph-mon[29774]: 5.19 scrub ok 2024-09-20T22:48:43.556 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:43 smithi084 ceph-mon[29774]: osdmap e443: 8 total, 8 up, 8 in 2024-09-20T22:48:43.557 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:43 smithi084 ceph-mon[40024]: pgmap v564: 166 pgs: 3 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 158 active+clean; 359 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.7 MiB/s rd, 719 KiB/s wr, 8 op/s; 3.1 MiB/s, 7 objects/s recovering 2024-09-20T22:48:43.557 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:43 smithi084 ceph-mon[40024]: 5.19 scrub starts 2024-09-20T22:48:43.557 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:43 smithi084 ceph-mon[40024]: 5.19 scrub ok 2024-09-20T22:48:43.557 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:43 smithi084 ceph-mon[40024]: osdmap e443: 8 total, 8 up, 8 in 2024-09-20T22:48:43.609 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:48:43.631 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:43.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:43.632 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:48:43.733 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:48:43.753 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:43.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:43 smithi154 ceph-mon[33264]: pgmap v564: 166 pgs: 3 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 158 active+clean; 359 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.7 MiB/s rd, 719 KiB/s wr, 8 op/s; 3.1 MiB/s, 7 objects/s recovering 2024-09-20T22:48:43.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:43 smithi154 ceph-mon[33264]: 5.19 scrub starts 2024-09-20T22:48:43.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:43 smithi154 ceph-mon[33264]: 5.19 scrub ok 2024-09-20T22:48:43.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:43 smithi154 ceph-mon[33264]: osdmap e443: 8 total, 8 up, 8 in 2024-09-20T22:48:43.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:43.755 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:48:43.856 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:48:43.879 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:43.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:43.880 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:48:43.980 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:48:44.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:44.004 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:44.004 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:48:44.105 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:48:44.128 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:44.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:44.129 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:48:44.225 INFO:tasks.rados.rados.0.smithi084.stdout:1709: done (6 left) 2024-09-20T22:48:44.225 INFO:tasks.rados.rados.0.smithi084.stdout:1712: done (5 left) 2024-09-20T22:48:44.225 INFO:tasks.rados.rados.0.smithi084.stdout:1713: done (4 left) 2024-09-20T22:48:44.225 INFO:tasks.rados.rados.0.smithi084.stdout:1714: done (3 left) 2024-09-20T22:48:44.225 INFO:tasks.rados.rados.0.smithi084.stdout:1717: done (2 left) 2024-09-20T22:48:44.225 INFO:tasks.rados.rados.0.smithi084.stdout:1718: done (1 left) 2024-09-20T22:48:44.226 INFO:tasks.rados.rados.0.smithi084.stdout:1719: done (0 left) 2024-09-20T22:48:44.226 INFO:tasks.rados.rados.0.smithi084.stdout:1720: rollback oid 32 current snap is 182 2024-09-20T22:48:44.226 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 32 to 133 2024-09-20T22:48:44.226 INFO:tasks.rados.rados.0.smithi084.stdout:1721: delete oid 36 current snap is 182 2024-09-20T22:48:44.228 INFO:tasks.rados.rados.0.smithi084.stdout:1720: finishing rollback tid 0 to smithi08477203-32 2024-09-20T22:48:44.229 INFO:tasks.rados.rados.0.smithi084.stdout:1720: finishing rollback tid 1 to smithi08477203-32 2024-09-20T22:48:44.229 INFO:tasks.rados.rados.0.smithi084.stdout:1720: finishing rollback tid 2 to smithi08477203-32 2024-09-20T22:48:44.229 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 32 v 399 (ObjNum 457 snap 126 seq_num 457) dirty exists 2024-09-20T22:48:44.230 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:48:44.231 INFO:tasks.rados.rados.0.smithi084.stdout:1720: done (1 left) 2024-09-20T22:48:44.231 INFO:tasks.rados.rados.0.smithi084.stdout:1721: done (0 left) 2024-09-20T22:48:44.231 INFO:tasks.rados.rados.0.smithi084.stdout:1722: read oid 37 snap 174 2024-09-20T22:48:44.231 INFO:tasks.rados.rados.0.smithi084.stdout:1722: expect deleted 2024-09-20T22:48:44.231 INFO:tasks.rados.rados.0.smithi084.stdout:1723: write (excl) oid 2 current snap is 182 2024-09-20T22:48:44.231 INFO:tasks.rados.rados.0.smithi084.stdout:1723: seq_num 609 ranges {635463=734881,1987181=248282} 2024-09-20T22:48:44.237 INFO:tasks.rados.rados.0.smithi084.stdout:1723: writing smithi08477203-2 from 635463 to 1370344 tid 1 2024-09-20T22:48:44.243 INFO:tasks.rados.rados.0.smithi084.stdout:1723: writing smithi08477203-2 from 1987181 to 2235463 tid 2 2024-09-20T22:48:44.243 INFO:tasks.rados.rados.0.smithi084.stdout:1722: done (1 left) 2024-09-20T22:48:44.243 INFO:tasks.rados.rados.0.smithi084.stdout:1724: delete oid 25 current snap is 182 2024-09-20T22:48:44.245 INFO:tasks.rados.rados.0.smithi084.stdout:1723: finishing write tid 1 to smithi08477203-2 2024-09-20T22:48:44.245 INFO:tasks.rados.rados.0.smithi084.stdout:1723: finishing write tid 2 to smithi08477203-2 2024-09-20T22:48:44.245 INFO:tasks.rados.rados.0.smithi084.stdout:1724: done (1 left) 2024-09-20T22:48:44.245 INFO:tasks.rados.rados.0.smithi084.stdout:1725: write (excl) oid 35 current snap is 182 2024-09-20T22:48:44.245 INFO:tasks.rados.rados.0.smithi084.stdout:1725: seq_num 611 ranges {669077=485379,1587792=681285} 2024-09-20T22:48:44.251 INFO:tasks.rados.rados.0.smithi084.stdout:1725: writing smithi08477203-35 from 669077 to 1154456 tid 1 2024-09-20T22:48:44.257 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:44.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:44.258 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:48:44.259 INFO:tasks.rados.rados.0.smithi084.stdout:1725: writing smithi08477203-35 from 1587792 to 2269077 tid 2 2024-09-20T22:48:44.259 INFO:tasks.rados.rados.0.smithi084.stdout:1726: copy_from oid 43 from oid 23 current snap is 182 2024-09-20T22:48:44.259 INFO:tasks.rados.rados.0.smithi084.stdout:1723: finishing write tid 3 to smithi08477203-2 2024-09-20T22:48:44.259 INFO:tasks.rados.rados.0.smithi084.stdout:1723: finishing write tid 4 to smithi08477203-2 2024-09-20T22:48:44.260 INFO:tasks.rados.rados.0.smithi084.stdout:1723: oid 2 updating version 0 to 210 2024-09-20T22:48:44.260 INFO:tasks.rados.rados.0.smithi084.stdout:1723: oid 2 updating version 210 to 211 2024-09-20T22:48:44.260 INFO:tasks.rados.rados.0.smithi084.stdout:1723: oid 2 updating version 211 to 212 2024-09-20T22:48:44.260 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 2 v 212 (ObjNum 609 snap 182 seq_num 609) dirty exists 2024-09-20T22:48:44.260 INFO:tasks.rados.rados.0.smithi084.stdout:1723: left oid 2 (ObjNum 609 snap 182 seq_num 609) 2024-09-20T22:48:44.260 INFO:tasks.rados.rados.0.smithi084.stdout:1725: finishing write tid 1 to smithi08477203-35 2024-09-20T22:48:44.260 INFO:tasks.rados.rados.0.smithi084.stdout:1723: done (2 left) 2024-09-20T22:48:44.260 INFO:tasks.rados.rados.0.smithi084.stdout:1727: write oid 14 current snap is 182 2024-09-20T22:48:44.260 INFO:tasks.rados.rados.0.smithi084.stdout:1727: seq_num 612 ranges {685884=560628,2046274=239610} 2024-09-20T22:48:44.266 INFO:tasks.rados.rados.0.smithi084.stdout:1727: writing smithi08477203-14 from 685884 to 1246512 tid 1 2024-09-20T22:48:44.274 INFO:tasks.rados.rados.0.smithi084.stdout:1727: writing smithi08477203-14 from 2046274 to 2285884 tid 2 2024-09-20T22:48:44.274 INFO:tasks.rados.rados.0.smithi084.stdout:1728: delete oid 1 current snap is 182 2024-09-20T22:48:44.274 INFO:tasks.rados.rados.0.smithi084.stdout:1725: finishing write tid 2 to smithi08477203-35 2024-09-20T22:48:44.275 INFO:tasks.rados.rados.0.smithi084.stdout:1725: finishing write tid 3 to smithi08477203-35 2024-09-20T22:48:44.275 INFO:tasks.rados.rados.0.smithi084.stdout:1725: finishing write tid 4 to smithi08477203-35 2024-09-20T22:48:44.275 INFO:tasks.rados.rados.0.smithi084.stdout:1725: oid 35 updating version 0 to 689 2024-09-20T22:48:44.275 INFO:tasks.rados.rados.0.smithi084.stdout:1725: oid 35 version 689 is already newer than 687 2024-09-20T22:48:44.275 INFO:tasks.rados.rados.0.smithi084.stdout:1725: oid 35 version 689 is already newer than 688 2024-09-20T22:48:44.275 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 35 v 689 (ObjNum 611 snap 182 seq_num 611) dirty exists 2024-09-20T22:48:44.275 INFO:tasks.rados.rados.0.smithi084.stdout:1725: left oid 35 (ObjNum 611 snap 182 seq_num 611) 2024-09-20T22:48:44.276 INFO:tasks.rados.rados.0.smithi084.stdout:1725: done (3 left) 2024-09-20T22:48:44.277 INFO:tasks.rados.rados.0.smithi084.stdout:1728: done (2 left) 2024-09-20T22:48:44.277 INFO:tasks.rados.rados.0.smithi084.stdout:1729: write oid 39 current snap is 182 2024-09-20T22:48:44.277 INFO:tasks.rados.rados.0.smithi084.stdout:1726: finishing copy_from racing read to smithi08477203-43 2024-09-20T22:48:44.277 INFO:tasks.rados.rados.0.smithi084.stdout:1729: seq_num 614 ranges {719498=711126,2046885=272613} 2024-09-20T22:48:44.284 INFO:tasks.rados.rados.0.smithi084.stdout:1729: writing smithi08477203-39 from 719498 to 1430624 tid 1 2024-09-20T22:48:44.290 INFO:tasks.rados.rados.0.smithi084.stdout:1729: writing smithi08477203-39 from 2046885 to 2319498 tid 2 2024-09-20T22:48:44.290 INFO:tasks.rados.rados.0.smithi084.stdout:1730: read oid 17 snap 180 2024-09-20T22:48:44.290 INFO:tasks.rados.rados.0.smithi084.stdout:1730: expect (ObjNum 426 snap 109 seq_num 426) 2024-09-20T22:48:44.290 INFO:tasks.rados.rados.0.smithi084.stdout:1731: snap_remove snap 173 2024-09-20T22:48:44.358 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:48:44.381 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:44.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:44.382 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:48:44.482 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:48:44.503 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:44.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:44.505 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:48:44.543 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:44 smithi084 ceph-mon[40024]: 4.0 scrub starts 2024-09-20T22:48:44.544 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:44 smithi084 ceph-mon[40024]: 4.0 scrub ok 2024-09-20T22:48:44.544 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:44 smithi084 ceph-mon[40024]: 5.d scrub starts 2024-09-20T22:48:44.544 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:44 smithi084 ceph-mon[40024]: 5.d scrub ok 2024-09-20T22:48:44.544 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:44 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "22"}]: dispatch 2024-09-20T22:48:44.544 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:44 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "22"}]: dispatch 2024-09-20T22:48:44.544 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:44 smithi084 ceph-mon[40024]: 4.13 scrub starts 2024-09-20T22:48:44.544 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:44 smithi084 ceph-mon[40024]: 4.13 scrub ok 2024-09-20T22:48:44.544 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:44 smithi084 ceph-mon[40024]: 4.7 deep-scrub starts 2024-09-20T22:48:44.544 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:44 smithi084 ceph-mon[40024]: 4.7 deep-scrub ok 2024-09-20T22:48:44.544 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:44 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "22"}]': finished 2024-09-20T22:48:44.544 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:44 smithi084 ceph-mon[40024]: osdmap e444: 8 total, 8 up, 8 in 2024-09-20T22:48:44.545 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:44 smithi084 ceph-mon[29774]: 4.0 scrub starts 2024-09-20T22:48:44.545 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:44 smithi084 ceph-mon[29774]: 4.0 scrub ok 2024-09-20T22:48:44.545 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:44 smithi084 ceph-mon[29774]: 5.d scrub starts 2024-09-20T22:48:44.545 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:44 smithi084 ceph-mon[29774]: 5.d scrub ok 2024-09-20T22:48:44.545 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:44 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "22"}]: dispatch 2024-09-20T22:48:44.545 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:44 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "22"}]: dispatch 2024-09-20T22:48:44.545 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:44 smithi084 ceph-mon[29774]: 4.13 scrub starts 2024-09-20T22:48:44.545 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:44 smithi084 ceph-mon[29774]: 4.13 scrub ok 2024-09-20T22:48:44.545 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:44 smithi084 ceph-mon[29774]: 4.7 deep-scrub starts 2024-09-20T22:48:44.545 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:44 smithi084 ceph-mon[29774]: 4.7 deep-scrub ok 2024-09-20T22:48:44.545 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:44 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "22"}]': finished 2024-09-20T22:48:44.546 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:44 smithi084 ceph-mon[29774]: osdmap e444: 8 total, 8 up, 8 in 2024-09-20T22:48:44.579 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:48:44.603 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-20T22:48:44.605 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:48:44.628 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:44.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:44.629 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:48:44.730 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:48:44.740 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-20T22:48:44.743 INFO:teuthology.orchestra.run.smithi084.stderr:dumped all 2024-09-20T22:48:44.751 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:44.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:44.752 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:48:44.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:44 smithi154 ceph-mon[33264]: 4.0 scrub starts 2024-09-20T22:48:44.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:44 smithi154 ceph-mon[33264]: 4.0 scrub ok 2024-09-20T22:48:44.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:44 smithi154 ceph-mon[33264]: 5.d scrub starts 2024-09-20T22:48:44.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:44 smithi154 ceph-mon[33264]: 5.d scrub ok 2024-09-20T22:48:44.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:44 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "22"}]: dispatch 2024-09-20T22:48:44.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:44 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "22"}]: dispatch 2024-09-20T22:48:44.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:44 smithi154 ceph-mon[33264]: 4.13 scrub starts 2024-09-20T22:48:44.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:44 smithi154 ceph-mon[33264]: 4.13 scrub ok 2024-09-20T22:48:44.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:44 smithi154 ceph-mon[33264]: 4.7 deep-scrub starts 2024-09-20T22:48:44.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:44 smithi154 ceph-mon[33264]: 4.7 deep-scrub ok 2024-09-20T22:48:44.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:44 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "22"}]': finished 2024-09-20T22:48:44.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:44 smithi154 ceph-mon[33264]: osdmap e444: 8 total, 8 up, 8 in 2024-09-20T22:48:44.852 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:48:44.874 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:44.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:44.875 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:48:44.975 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:48:44.997 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:44.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:44.998 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:48:45.099 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:48:45.120 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:45.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:45.121 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:48:45.219 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is set 2024-09-20T22:48:45.222 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:48:45.226 INFO:tasks.rados.rados.0.smithi084.stdout:1731: done (4 left) 2024-09-20T22:48:45.227 INFO:tasks.rados.rados.0.smithi084.stdout:1732: snap_create 2024-09-20T22:48:45.227 INFO:tasks.rados.rados.0.smithi084.stdout:1726: finishing copy_from to smithi08477203-43 2024-09-20T22:48:45.227 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 43 v 706 (ObjNum 428 snap 110 seq_num 428) dirty exists 2024-09-20T22:48:45.227 INFO:tasks.rados.rados.0.smithi084.stdout:1729: finishing write tid 1 to smithi08477203-39 2024-09-20T22:48:45.227 INFO:tasks.rados.rados.0.smithi084.stdout:1729: finishing write tid 2 to smithi08477203-39 2024-09-20T22:48:45.227 INFO:tasks.rados.rados.0.smithi084.stdout:1729: finishing write tid 3 to smithi08477203-39 2024-09-20T22:48:45.227 INFO:tasks.rados.rados.0.smithi084.stdout:1729: finishing write tid 4 to smithi08477203-39 2024-09-20T22:48:45.227 INFO:tasks.rados.rados.0.smithi084.stdout:1729: oid 39 updating version 0 to 602 2024-09-20T22:48:45.227 INFO:tasks.rados.rados.0.smithi084.stdout:1729: oid 39 updating version 602 to 603 2024-09-20T22:48:45.227 INFO:tasks.rados.rados.0.smithi084.stdout:1729: oid 39 updating version 603 to 604 2024-09-20T22:48:45.227 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 39 v 604 (ObjNum 614 snap 182 seq_num 614) dirty exists 2024-09-20T22:48:45.227 INFO:tasks.rados.rados.0.smithi084.stdout:1729: left oid 39 (ObjNum 614 snap 182 seq_num 614) 2024-09-20T22:48:45.227 INFO:tasks.rados.rados.0.smithi084.stdout:1730: expect (ObjNum 426 snap 109 seq_num 426) 2024-09-20T22:48:45.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:45.254 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:45.254 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:48:45.355 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:48:45.364 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:45 smithi084 ceph-mon[40024]: pgmap v567: 165 pgs: 3 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 157 active+clean; 359 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 4.4 MiB/s rd, 716 KiB/s wr, 11 op/s 2024-09-20T22:48:45.364 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:45 smithi084 ceph-mon[40024]: 2.0 scrub starts 2024-09-20T22:48:45.364 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:45 smithi084 ceph-mon[40024]: 2.0 scrub ok 2024-09-20T22:48:45.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:45 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/391701904' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-20T22:48:45.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:45 smithi084 ceph-mon[40024]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:48:45.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:45 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/391701904' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T22:48:45.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:45 smithi084 ceph-mon[40024]: osdmap e445: 8 total, 8 up, 8 in 2024-09-20T22:48:45.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:45 smithi084 ceph-mon[29774]: pgmap v567: 165 pgs: 3 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 157 active+clean; 359 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 4.4 MiB/s rd, 716 KiB/s wr, 11 op/s 2024-09-20T22:48:45.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:45 smithi084 ceph-mon[29774]: 2.0 scrub starts 2024-09-20T22:48:45.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:45 smithi084 ceph-mon[29774]: 2.0 scrub ok 2024-09-20T22:48:45.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:45 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/391701904' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-20T22:48:45.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:45 smithi084 ceph-mon[29774]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:48:45.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:45 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/391701904' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T22:48:45.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:45 smithi084 ceph-mon[29774]: osdmap e445: 8 total, 8 up, 8 in 2024-09-20T22:48:45.378 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:45.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:45.379 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:48:45.479 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:48:45.502 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:45.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:45.503 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:48:45.526 INFO:teuthology.orchestra.run.smithi084.stdout:{"pg_ready":true,"pg_map":{"version":568,"stamp":"2024-09-20T22:48:44.217063+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":376316555,"num_objects":375,"num_object_clones":127,"num_object_copies":1125,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":375,"num_whiteouts":16,"num_read":4135,"num_read_kb":3160940,"num_write":2525,"num_write_kb":785309,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":874,"num_bytes_recovered":297926788,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6247,"ondisk_log_size":6247,"up":495,"acting":495,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":496,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":1187072,"kb_used_data":848716,"kb_used_omap":12,"kb_used_meta":338163,"kb_avail":748774144,"statfs":{"total":767960285184,"available":766744723456,"internally_reserved":0,"allocated":869085184,"data_stored":1188083643,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":346279472},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":50558,"num_objects":2,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":30,"num_read_kb":17932,"num_write":17,"num_write_kb":2866,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.008816"},"pg_stats":[{"pgid":"3.1f","version":"86'2","reported_seq":123,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-09-20T22:48:43.212243+0000","last_change":"2024-09-20T22:47:49.237762+0000","last_active":"2024-09-20T22:48:43.212243+0000","last_peered":"2024-09-20T22:48:43.212243+0000","last_clean":"2024-09-20T22:48:43.212243+0000","last_became_active":"2024-09-20T22:47:49.222114+0000","last_became_peered":"2024-09-20T22:47:49.222114+0000","last_unstale":"2024-09-20T22:48:43.212243+0000","last_undegraded":"2024-09-20T22:48:43.212243+0000","last_fullsized":"2024-09-20T22:48:43.212243+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'2","last_scrub_stamp":"2024-09-20T22:46:42.206600+0000","last_deep_scrub":"86'2","last_deep_scrub_stamp":"2024-09-20T22:42:39.664421+0000","last_clean_scrub_stamp":"2024-09-20T22:46:42.206600+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":436,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"74'9","reported_seq":97,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-09-20T22:48:43.206905+0000","last_change":"2024-09-20T22:48:05.283966+0000","last_active":"2024-09-20T22:48:43.206905+0000","last_peered":"2024-09-20T22:48:43.206905+0000","last_clean":"2024-09-20T22:48:43.206905+0000","last_became_active":"2024-09-20T22:48:05.248980+0000","last_became_peered":"2024-09-20T22:48:05.248980+0000","last_unstale":"2024-09-20T22:48:43.206905+0000","last_undegraded":"2024-09-20T22:48:43.206905+0000","last_fullsized":"2024-09-20T22:48:43.206905+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-09-20T22:46:20.889510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:46:20.889510+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":780,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-09-20T22:48:43.209406+0000","last_change":"2024-09-20T22:48:43.157340+0000","last_active":"2024-09-20T22:48:43.209406+0000","last_peered":"2024-09-20T22:48:43.209406+0000","last_clean":"2024-09-20T22:48:43.209406+0000","last_became_active":"2024-09-20T22:41:42.182024+0000","last_became_peered":"2024-09-20T22:41:42.182024+0000","last_unstale":"2024-09-20T22:48:43.209406+0000","last_undegraded":"2024-09-20T22:48:43.209406+0000","last_fullsized":"2024-09-20T22:48:43.209406+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:48:43.157266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:48:43.157266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:01.873970+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":87,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-09-20T22:48:43.207213+0000","last_change":"2024-09-20T22:48:05.248713+0000","last_active":"2024-09-20T22:48:43.207213+0000","last_peered":"2024-09-20T22:48:43.207213+0000","last_clean":"2024-09-20T22:48:43.207213+0000","last_became_active":"2024-09-20T22:48:05.247569+0000","last_became_peered":"2024-09-20T22:48:05.247569+0000","last_unstale":"2024-09-20T22:48:43.207213+0000","last_undegraded":"2024-09-20T22:48:43.207213+0000","last_fullsized":"2024-09-20T22:48:43.207213+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:46:45.177105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:46:45.177105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":768,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-09-20T22:48:43.221447+0000","last_change":"2024-09-20T22:45:40.053329+0000","last_active":"2024-09-20T22:48:43.221447+0000","last_peered":"2024-09-20T22:48:43.221447+0000","last_clean":"2024-09-20T22:48:43.221447+0000","last_became_active":"2024-09-20T22:41:44.832099+0000","last_became_peered":"2024-09-20T22:41:44.832099+0000","last_unstale":"2024-09-20T22:48:43.221447+0000","last_undegraded":"2024-09-20T22:48:43.221447+0000","last_fullsized":"2024-09-20T22:48:43.221447+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:45:40.053263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:45:40.053263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":773,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-09-20T22:48:43.220108+0000","last_change":"2024-09-20T22:44:58.994410+0000","last_active":"2024-09-20T22:48:43.220108+0000","last_peered":"2024-09-20T22:48:43.220108+0000","last_clean":"2024-09-20T22:48:43.220108+0000","last_became_active":"2024-09-20T22:41:38.157298+0000","last_became_peered":"2024-09-20T22:41:38.157298+0000","last_unstale":"2024-09-20T22:48:43.220108+0000","last_undegraded":"2024-09-20T22:48:43.220108+0000","last_fullsized":"2024-09-20T22:48:43.220108+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:44:58.994340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:44:58.994340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"74'15","reported_seq":837,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-09-20T22:48:43.219235+0000","last_change":"2024-09-20T22:47:49.246946+0000","last_active":"2024-09-20T22:48:43.219235+0000","last_peered":"2024-09-20T22:48:43.219235+0000","last_clean":"2024-09-20T22:48:43.219235+0000","last_became_active":"2024-09-20T22:47:49.218798+0000","last_became_peered":"2024-09-20T22:47:49.218798+0000","last_unstale":"2024-09-20T22:48:43.219235+0000","last_undegraded":"2024-09-20T22:48:43.219235+0000","last_fullsized":"2024-09-20T22:48:43.219235+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-09-20T22:44:41.108205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:44:41.108205+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":87,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-09-20T22:48:43.206781+0000","last_change":"2024-09-20T22:48:05.247938+0000","last_active":"2024-09-20T22:48:43.206781+0000","last_peered":"2024-09-20T22:48:43.206781+0000","last_clean":"2024-09-20T22:48:43.206781+0000","last_became_active":"2024-09-20T22:48:05.247636+0000","last_became_peered":"2024-09-20T22:48:05.247636+0000","last_unstale":"2024-09-20T22:48:43.206781+0000","last_undegraded":"2024-09-20T22:48:43.206781+0000","last_fullsized":"2024-09-20T22:48:43.206781+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:47:19.805701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:47:19.805701+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":793,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-09-20T22:48:43.229229+0000","last_change":"2024-09-20T22:48:05.242386+0000","last_active":"2024-09-20T22:48:43.229229+0000","last_peered":"2024-09-20T22:48:43.229229+0000","last_clean":"2024-09-20T22:48:43.229229+0000","last_became_active":"2024-09-20T22:48:05.242059+0000","last_became_peered":"2024-09-20T22:48:05.242059+0000","last_unstale":"2024-09-20T22:48:43.229229+0000","last_undegraded":"2024-09-20T22:48:43.229229+0000","last_fullsized":"2024-09-20T22:48:43.229229+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:46:00.999939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:46:00.999939+0000","last_clean_scrub_stamp":"2024-09-20T22:46:00.999939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"74'9","reported_seq":97,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-09-20T22:48:43.207224+0000","last_change":"2024-09-20T22:48:05.259778+0000","last_active":"2024-09-20T22:48:43.207224+0000","last_peered":"2024-09-20T22:48:43.207224+0000","last_clean":"2024-09-20T22:48:43.207224+0000","last_became_active":"2024-09-20T22:48:05.233429+0000","last_became_peered":"2024-09-20T22:48:05.233429+0000","last_unstale":"2024-09-20T22:48:43.207224+0000","last_undegraded":"2024-09-20T22:48:43.207224+0000","last_fullsized":"2024-09-20T22:48:43.207224+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-09-20T22:46:45.095957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:46:45.095957+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":785,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-09-20T22:48:43.228849+0000","last_change":"2024-09-20T22:47:49.228490+0000","last_active":"2024-09-20T22:48:43.228849+0000","last_peered":"2024-09-20T22:48:43.228849+0000","last_clean":"2024-09-20T22:48:43.228849+0000","last_became_active":"2024-09-20T22:47:49.227379+0000","last_became_peered":"2024-09-20T22:47:49.227379+0000","last_unstale":"2024-09-20T22:48:43.228849+0000","last_undegraded":"2024-09-20T22:48:43.228849+0000","last_fullsized":"2024-09-20T22:48:43.228849+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:46:52.239508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:46:52.239508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":119,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-09-20T22:48:43.210372+0000","last_change":"2024-09-20T22:47:49.225833+0000","last_active":"2024-09-20T22:48:43.210372+0000","last_peered":"2024-09-20T22:48:43.210372+0000","last_clean":"2024-09-20T22:48:43.210372+0000","last_became_active":"2024-09-20T22:47:49.225515+0000","last_became_peered":"2024-09-20T22:47:49.225515+0000","last_unstale":"2024-09-20T22:48:43.210372+0000","last_undegraded":"2024-09-20T22:48:43.210372+0000","last_fullsized":"2024-09-20T22:48:43.210372+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:46:29.801854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:46:29.801854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":801,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-09-20T22:48:43.228615+0000","last_change":"2024-09-20T22:48:37.559081+0000","last_active":"2024-09-20T22:48:43.228615+0000","last_peered":"2024-09-20T22:48:43.228615+0000","last_clean":"2024-09-20T22:48:43.228615+0000","last_became_active":"2024-09-20T22:48:05.242084+0000","last_became_peered":"2024-09-20T22:48:05.242084+0000","last_unstale":"2024-09-20T22:48:43.228615+0000","last_undegraded":"2024-09-20T22:48:43.228615+0000","last_fullsized":"2024-09-20T22:48:43.228615+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:48:37.559007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:48:37.559007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:49:43.690136+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"181'7","reported_seq":101,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-09-20T22:48:43.207686+0000","last_change":"2024-09-20T22:48:41.049491+0000","last_active":"2024-09-20T22:48:43.207686+0000","last_peered":"2024-09-20T22:48:43.207686+0000","last_clean":"2024-09-20T22:48:43.207686+0000","last_became_active":"2024-09-20T22:48:05.249763+0000","last_became_peered":"2024-09-20T22:48:05.249763+0000","last_unstale":"2024-09-20T22:48:43.207686+0000","last_undegraded":"2024-09-20T22:48:43.207686+0000","last_fullsized":"2024-09-20T22:48:43.207686+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"181'7","last_scrub_stamp":"2024-09-20T22:48:41.049381+0000","last_deep_scrub":"181'7","last_deep_scrub_stamp":"2024-09-20T22:48:41.049381+0000","last_clean_scrub_stamp":"2024-09-20T22:48:41.049381+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:03.477658+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":6,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":806,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-09-20T22:48:43.216216+0000","last_change":"2024-09-20T22:48:05.238275+0000","last_active":"2024-09-20T22:48:43.216216+0000","last_peered":"2024-09-20T22:48:43.216216+0000","last_clean":"2024-09-20T22:48:43.216216+0000","last_became_active":"2024-09-20T22:48:05.237791+0000","last_became_peered":"2024-09-20T22:48:05.237791+0000","last_unstale":"2024-09-20T22:48:43.216216+0000","last_undegraded":"2024-09-20T22:48:43.216216+0000","last_fullsized":"2024-09-20T22:48:43.216216+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:47:40.361958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:47:40.361958+0000","last_clean_scrub_stamp":"2024-09-20T22:47:40.361958+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:56.944095+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":769,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-09-20T22:48:43.228175+0000","last_change":"2024-09-20T22:46:06.068842+0000","last_active":"2024-09-20T22:48:43.228175+0000","last_peered":"2024-09-20T22:48:43.228175+0000","last_clean":"2024-09-20T22:48:43.228175+0000","last_became_active":"2024-09-20T22:41:44.197927+0000","last_became_peered":"2024-09-20T22:41:44.197927+0000","last_unstale":"2024-09-20T22:48:43.228175+0000","last_undegraded":"2024-09-20T22:48:43.228175+0000","last_fullsized":"2024-09-20T22:48:43.228175+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:46:06.068745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:46:06.068745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":87,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-09-20T22:48:43.208706+0000","last_change":"2024-09-20T22:48:05.244188+0000","last_active":"2024-09-20T22:48:43.208706+0000","last_peered":"2024-09-20T22:48:43.208706+0000","last_clean":"2024-09-20T22:48:43.208706+0000","last_became_active":"2024-09-20T22:48:05.243877+0000","last_became_peered":"2024-09-20T22:48:05.243877+0000","last_unstale":"2024-09-20T22:48:43.208706+0000","last_undegraded":"2024-09-20T22:48:43.208706+0000","last_fullsized":"2024-09-20T22:48:43.208706+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:47:21.819572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:47:21.819572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:48.818606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":125,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-09-20T22:48:43.209842+0000","last_change":"2024-09-20T22:48:05.234091+0000","last_active":"2024-09-20T22:48:43.209842+0000","last_peered":"2024-09-20T22:48:43.209842+0000","last_clean":"2024-09-20T22:48:43.209842+0000","last_became_active":"2024-09-20T22:48:05.233687+0000","last_became_peered":"2024-09-20T22:48:05.233687+0000","last_unstale":"2024-09-20T22:48:43.209842+0000","last_undegraded":"2024-09-20T22:48:43.209842+0000","last_fullsized":"2024-09-20T22:48:43.209842+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:47:13.413912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:47:13.413912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"113'22","reported_seq":823,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-09-20T22:48:43.225057+0000","last_change":"2024-09-20T22:48:21.243488+0000","last_active":"2024-09-20T22:48:43.225057+0000","last_peered":"2024-09-20T22:48:43.225057+0000","last_clean":"2024-09-20T22:48:43.225057+0000","last_became_active":"2024-09-20T22:41:40.170131+0000","last_became_peered":"2024-09-20T22:41:40.170131+0000","last_unstale":"2024-09-20T22:48:43.225057+0000","last_undegraded":"2024-09-20T22:48:43.225057+0000","last_fullsized":"2024-09-20T22:48:43.225057+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'22","last_scrub_stamp":"2024-09-20T22:48:21.243355+0000","last_deep_scrub":"113'22","last_deep_scrub_stamp":"2024-09-20T22:43:12.191878+0000","last_clean_scrub_stamp":"2024-09-20T22:48:21.243355+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:49:37.343367+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":796,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-09-20T22:48:43.210958+0000","last_change":"2024-09-20T22:48:05.249765+0000","last_active":"2024-09-20T22:48:43.210958+0000","last_peered":"2024-09-20T22:48:43.210958+0000","last_clean":"2024-09-20T22:48:43.210958+0000","last_became_active":"2024-09-20T22:48:05.249417+0000","last_became_peered":"2024-09-20T22:48:05.249417+0000","last_unstale":"2024-09-20T22:48:43.210958+0000","last_undegraded":"2024-09-20T22:48:43.210958+0000","last_fullsized":"2024-09-20T22:48:43.210958+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:46:49.872479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:46:49.872479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":761,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-09-20T22:48:43.220268+0000","last_change":"2024-09-20T22:45:25.002270+0000","last_active":"2024-09-20T22:48:43.220268+0000","last_peered":"2024-09-20T22:48:43.220268+0000","last_clean":"2024-09-20T22:48:43.220268+0000","last_became_active":"2024-09-20T22:41:44.832131+0000","last_became_peered":"2024-09-20T22:41:44.832131+0000","last_unstale":"2024-09-20T22:48:43.220268+0000","last_undegraded":"2024-09-20T22:48:43.220268+0000","last_fullsized":"2024-09-20T22:48:43.220268+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:45:25.002199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:45:25.002199+0000","last_clean_scrub_stamp":"2024-09-20T22:45:25.002199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":95,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-09-20T22:48:43.207357+0000","last_change":"2024-09-20T22:48:24.986582+0000","last_active":"2024-09-20T22:48:43.207357+0000","last_peered":"2024-09-20T22:48:43.207357+0000","last_clean":"2024-09-20T22:48:43.207357+0000","last_became_active":"2024-09-20T22:48:05.250581+0000","last_became_peered":"2024-09-20T22:48:05.250581+0000","last_unstale":"2024-09-20T22:48:43.207357+0000","last_undegraded":"2024-09-20T22:48:43.207357+0000","last_fullsized":"2024-09-20T22:48:43.207357+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:48:24.986484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:48:24.986484+0000","last_clean_scrub_stamp":"2024-09-20T22:48:24.986484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:49:46.510379+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"74'12","reported_seq":845,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-09-20T22:48:43.221216+0000","last_change":"2024-09-20T22:48:05.258994+0000","last_active":"2024-09-20T22:48:43.221216+0000","last_peered":"2024-09-20T22:48:43.221216+0000","last_clean":"2024-09-20T22:48:43.221216+0000","last_became_active":"2024-09-20T22:48:05.233983+0000","last_became_peered":"2024-09-20T22:48:05.233983+0000","last_unstale":"2024-09-20T22:48:43.221216+0000","last_undegraded":"2024-09-20T22:48:43.221216+0000","last_fullsized":"2024-09-20T22:48:43.221216+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-09-20T22:44:42.140915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:44:42.140915+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":95,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-09-20T22:48:43.208000+0000","last_change":"2024-09-20T22:48:23.940233+0000","last_active":"2024-09-20T22:48:43.208000+0000","last_peered":"2024-09-20T22:48:43.208000+0000","last_clean":"2024-09-20T22:48:43.208000+0000","last_became_active":"2024-09-20T22:48:05.250319+0000","last_became_peered":"2024-09-20T22:48:05.250319+0000","last_unstale":"2024-09-20T22:48:43.208000+0000","last_undegraded":"2024-09-20T22:48:43.208000+0000","last_fullsized":"2024-09-20T22:48:43.208000+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:48:23.940157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:48:23.940157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:49:46.541532+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"154'2","reported_seq":777,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-09-20T22:48:43.216929+0000","last_change":"2024-09-20T22:48:39.357067+0000","last_active":"2024-09-20T22:48:43.216929+0000","last_peered":"2024-09-20T22:48:43.216929+0000","last_clean":"2024-09-20T22:48:43.216929+0000","last_became_active":"2024-09-20T22:41:44.199822+0000","last_became_peered":"2024-09-20T22:41:44.199822+0000","last_unstale":"2024-09-20T22:48:43.216929+0000","last_undegraded":"2024-09-20T22:48:43.216929+0000","last_fullsized":"2024-09-20T22:48:43.216929+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"154'2","last_scrub_stamp":"2024-09-20T22:48:39.356937+0000","last_deep_scrub":"154'2","last_deep_scrub_stamp":"2024-09-20T22:43:54.134066+0000","last_clean_scrub_stamp":"2024-09-20T22:48:39.356937+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:49:48.454806+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"67'1","reported_seq":809,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-09-20T22:48:43.209546+0000","last_change":"2024-09-20T22:48:42.188853+0000","last_active":"2024-09-20T22:48:43.209546+0000","last_peered":"2024-09-20T22:48:43.209546+0000","last_clean":"2024-09-20T22:48:43.209546+0000","last_became_active":"2024-09-20T22:48:05.249578+0000","last_became_peered":"2024-09-20T22:48:05.249578+0000","last_unstale":"2024-09-20T22:48:43.209546+0000","last_undegraded":"2024-09-20T22:48:43.209546+0000","last_fullsized":"2024-09-20T22:48:43.209546+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-09-20T22:48:42.188772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:48:42.188772+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:09.372929+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"151'15","reported_seq":139,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-09-20T22:48:43.212818+0000","last_change":"2024-09-20T22:48:37.528315+0000","last_active":"2024-09-20T22:48:43.212818+0000","last_peered":"2024-09-20T22:48:43.212818+0000","last_clean":"2024-09-20T22:48:43.212818+0000","last_became_active":"2024-09-20T22:47:49.232181+0000","last_became_peered":"2024-09-20T22:47:49.232181+0000","last_unstale":"2024-09-20T22:48:43.212818+0000","last_undegraded":"2024-09-20T22:48:43.212818+0000","last_fullsized":"2024-09-20T22:48:43.212818+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"151'15","last_scrub_stamp":"2024-09-20T22:48:37.528207+0000","last_deep_scrub":"151'15","last_deep_scrub_stamp":"2024-09-20T22:43:50.715761+0000","last_clean_scrub_stamp":"2024-09-20T22:48:37.528207+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:49:47.363120+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"408'10","reported_seq":901,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-09-20T22:48:43.214367+0000","last_change":"2024-09-20T22:48:18.083640+0000","last_active":"2024-09-20T22:48:43.214367+0000","last_peered":"2024-09-20T22:48:43.214367+0000","last_clean":"2024-09-20T22:48:43.214367+0000","last_became_active":"2024-09-20T22:48:05.244270+0000","last_became_peered":"2024-09-20T22:48:05.244270+0000","last_unstale":"2024-09-20T22:48:43.214367+0000","last_undegraded":"2024-09-20T22:48:43.214367+0000","last_fullsized":"2024-09-20T22:48:43.214367+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"408'10","last_scrub_stamp":"2024-09-20T22:48:18.083560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:48:18.083560+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:49:38.952569+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.c","version":"443'1074","reported_seq":2832,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-09-20T22:48:43.236382+0000","last_change":"2024-09-20T22:48:43.236561+0000","last_active":"2024-09-20T22:48:43.236382+0000","last_peered":"2024-09-20T22:48:43.236382+0000","last_clean":"2024-09-20T22:48:43.236382+0000","last_became_active":"2024-09-20T22:42:52.404027+0000","last_became_peered":"2024-09-20T22:42:52.404027+0000","last_unstale":"2024-09-20T22:48:43.236382+0000","last_undegraded":"2024-09-20T22:48:43.236382+0000","last_fullsized":"2024-09-20T22:48:43.236382+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"438'1046","last_scrub_stamp":"2024-09-20T22:48:38.253809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:48:38.253809+0000","objects_scrubbed":22,"log_size":1074,"log_dups_size":0,"ondisk_log_size":1074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:49:45.390373+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0.022860579999999998,"stat_sum":{"num_bytes":51915798,"num_objects":22,"num_object_clones":15,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":2,"num_read":542,"num_read_kb":532921,"num_write":382,"num_write_kb":149980,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"8d","length":"2"},{"start":"93","length":"2"},{"start":"9b","length":"1"},{"start":"9d","length":"1"},{"start":"9f","length":"2"},{"start":"a3","length":"2"},{"start":"a8","length":"1"},{"start":"aa","length":"4"}]},{"pgid":"1.a","version":"77'128","reported_seq":1048,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-09-20T22:48:43.228269+0000","last_change":"2024-09-20T22:48:24.566023+0000","last_active":"2024-09-20T22:48:43.228269+0000","last_peered":"2024-09-20T22:48:43.228269+0000","last_clean":"2024-09-20T22:48:43.228269+0000","last_became_active":"2024-09-20T22:42:52.407791+0000","last_became_peered":"2024-09-20T22:42:52.407791+0000","last_unstale":"2024-09-20T22:48:43.228269+0000","last_undegraded":"2024-09-20T22:48:43.228269+0000","last_fullsized":"2024-09-20T22:48:43.228269+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:48:24.565920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:48:24.565920+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:49:47.507188+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"74'15","reported_seq":854,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-09-20T22:48:43.210510+0000","last_change":"2024-09-20T22:48:21.288601+0000","last_active":"2024-09-20T22:48:43.210510+0000","last_peered":"2024-09-20T22:48:43.210510+0000","last_clean":"2024-09-20T22:48:43.210510+0000","last_became_active":"2024-09-20T22:48:05.250308+0000","last_became_peered":"2024-09-20T22:48:05.250308+0000","last_unstale":"2024-09-20T22:48:43.210510+0000","last_undegraded":"2024-09-20T22:48:43.210510+0000","last_fullsized":"2024-09-20T22:48:43.210510+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-09-20T22:48:21.288511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:48:21.288511+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:49:49.906057+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":773,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-09-20T22:48:43.218619+0000","last_change":"2024-09-20T22:45:19.027462+0000","last_active":"2024-09-20T22:48:43.218619+0000","last_peered":"2024-09-20T22:48:43.218619+0000","last_clean":"2024-09-20T22:48:43.218619+0000","last_became_active":"2024-09-20T22:41:38.156749+0000","last_became_peered":"2024-09-20T22:41:38.156749+0000","last_unstale":"2024-09-20T22:48:43.218619+0000","last_undegraded":"2024-09-20T22:48:43.218619+0000","last_fullsized":"2024-09-20T22:48:43.218619+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:45:19.027393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:45:19.027393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"408'12","reported_seq":99,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-09-20T22:48:43.208633+0000","last_change":"2024-09-20T22:48:05.251167+0000","last_active":"2024-09-20T22:48:43.208633+0000","last_peered":"2024-09-20T22:48:43.208633+0000","last_clean":"2024-09-20T22:48:43.208633+0000","last_became_active":"2024-09-20T22:48:05.242914+0000","last_became_peered":"2024-09-20T22:48:05.242914+0000","last_unstale":"2024-09-20T22:48:43.208633+0000","last_undegraded":"2024-09-20T22:48:43.208633+0000","last_fullsized":"2024-09-20T22:48:43.208633+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"264'10","last_scrub_stamp":"2024-09-20T22:47:20.300787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:47:20.300787+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":782,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-09-20T22:48:43.228348+0000","last_change":"2024-09-20T22:47:49.227726+0000","last_active":"2024-09-20T22:48:43.228348+0000","last_peered":"2024-09-20T22:48:43.228348+0000","last_clean":"2024-09-20T22:48:43.228348+0000","last_became_active":"2024-09-20T22:47:49.227039+0000","last_became_peered":"2024-09-20T22:47:49.227039+0000","last_unstale":"2024-09-20T22:48:43.228348+0000","last_undegraded":"2024-09-20T22:48:43.228348+0000","last_fullsized":"2024-09-20T22:48:43.228348+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:46:26.147167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:46:26.147167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3","version":"135'8","reported_seq":1727,"reported_epoch":443,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-20T22:48:43.220588+0000","last_change":"2024-09-20T22:48:42.255131+0000","last_active":"2024-09-20T22:48:43.220588+0000","last_peered":"2024-09-20T22:48:43.220588+0000","last_clean":"2024-09-20T22:48:43.220588+0000","last_became_active":"2024-09-20T22:48:05.234893+0000","last_became_peered":"2024-09-20T22:48:05.234893+0000","last_unstale":"2024-09-20T22:48:43.220588+0000","last_undegraded":"2024-09-20T22:48:43.220588+0000","last_fullsized":"2024-09-20T22:48:43.220588+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'8","last_scrub_stamp":"2024-09-20T22:47:23.137750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:47:23.137750+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.033415331,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"8d","length":"2"},{"start":"93","length":"2"},{"start":"9b","length":"1"},{"start":"9d","length":"1"},{"start":"a0","length":"1"},{"start":"a3","length":"2"},{"start":"a8","length":"1"},{"start":"aa","length":"2"}]},{"pgid":"1.5","version":"77'128","reported_seq":945,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-09-20T22:48:43.208664+0000","last_change":"2024-09-20T22:48:08.207463+0000","last_active":"2024-09-20T22:48:43.208664+0000","last_peered":"2024-09-20T22:48:43.208664+0000","last_clean":"2024-09-20T22:48:43.208664+0000","last_became_active":"2024-09-20T22:48:08.207018+0000","last_became_peered":"2024-09-20T22:48:08.207018+0000","last_unstale":"2024-09-20T22:48:43.208664+0000","last_undegraded":"2024-09-20T22:48:43.208664+0000","last_fullsized":"2024-09-20T22:48:43.208664+0000","mapping_epoch":407,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":408,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:48:02.056211+0000","last_deep_scrub":"77'128","last_deep_scrub_stamp":"2024-09-20T22:45:41.608075+0000","last_clean_scrub_stamp":"2024-09-20T22:48:02.056211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:49:15.339453+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"74'18","reported_seq":857,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-09-20T22:48:43.218858+0000","last_change":"2024-09-20T22:48:43.146697+0000","last_active":"2024-09-20T22:48:43.218858+0000","last_peered":"2024-09-20T22:48:43.218858+0000","last_clean":"2024-09-20T22:48:43.218858+0000","last_became_active":"2024-09-20T22:47:49.217605+0000","last_became_peered":"2024-09-20T22:47:49.217605+0000","last_unstale":"2024-09-20T22:48:43.218858+0000","last_undegraded":"2024-09-20T22:48:43.218858+0000","last_fullsized":"2024-09-20T22:48:43.218858+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'18","last_scrub_stamp":"2024-09-20T22:48:43.146602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:48:43.146602+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:00.379899+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":793,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-09-20T22:48:43.220768+0000","last_change":"2024-09-20T22:47:49.227431+0000","last_active":"2024-09-20T22:48:43.220768+0000","last_peered":"2024-09-20T22:48:43.220768+0000","last_clean":"2024-09-20T22:48:43.220768+0000","last_became_active":"2024-09-20T22:47:49.227069+0000","last_became_peered":"2024-09-20T22:47:49.227069+0000","last_unstale":"2024-09-20T22:48:43.220768+0000","last_undegraded":"2024-09-20T22:48:43.220768+0000","last_fullsized":"2024-09-20T22:48:43.220768+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:47:39.982616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:47:39.982616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:59.960690+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":87,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-09-20T22:48:43.207711+0000","last_change":"2024-09-20T22:48:05.231356+0000","last_active":"2024-09-20T22:48:43.207711+0000","last_peered":"2024-09-20T22:48:43.207711+0000","last_clean":"2024-09-20T22:48:43.207711+0000","last_became_active":"2024-09-20T22:48:05.230003+0000","last_became_peered":"2024-09-20T22:48:05.230003+0000","last_unstale":"2024-09-20T22:48:43.207711+0000","last_undegraded":"2024-09-20T22:48:43.207711+0000","last_fullsized":"2024-09-20T22:48:43.207711+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:45:56.199432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:07.947945+0000","last_clean_scrub_stamp":"2024-09-20T22:45:56.199432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":87,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-09-20T22:48:43.207443+0000","last_change":"2024-09-20T22:48:05.240266+0000","last_active":"2024-09-20T22:48:43.207443+0000","last_peered":"2024-09-20T22:48:43.207443+0000","last_clean":"2024-09-20T22:48:43.207443+0000","last_became_active":"2024-09-20T22:48:05.239917+0000","last_became_peered":"2024-09-20T22:48:05.239917+0000","last_unstale":"2024-09-20T22:48:43.207443+0000","last_undegraded":"2024-09-20T22:48:43.207443+0000","last_fullsized":"2024-09-20T22:48:43.207443+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:47:18.179575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:53.036601+0000","last_clean_scrub_stamp":"2024-09-20T22:47:18.179575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2","version":"442'209","reported_seq":1511,"reported_epoch":443,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T22:48:43.228082+0000","last_change":"2024-09-20T22:48:42.250708+0000","last_active":"2024-09-20T22:48:43.228082+0000","last_peered":"2024-09-20T22:48:43.228082+0000","last_clean":"2024-09-20T22:48:43.228082+0000","last_became_active":"2024-09-20T22:44:09.177625+0000","last_became_peered":"2024-09-20T22:44:09.177625+0000","last_unstale":"2024-09-20T22:48:43.228082+0000","last_undegraded":"2024-09-20T22:48:43.228082+0000","last_fullsized":"2024-09-20T22:48:43.228082+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"282'120","last_scrub_stamp":"2024-09-20T22:46:22.099642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:46:22.099642+0000","objects_scrubbed":1,"log_size":209,"log_dups_size":0,"ondisk_log_size":209,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.032154877999999998,"stat_sum":{"num_bytes":6300577,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":91,"num_read_kb":103745,"num_write":41,"num_write_kb":12558,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"8d","length":"2"},{"start":"93","length":"2"},{"start":"9b","length":"1"},{"start":"9d","length":"1"},{"start":"a0","length":"1"},{"start":"a3","length":"2"},{"start":"a8","length":"1"},{"start":"aa","length":"1"},{"start":"ac","length":"1"}]},{"pgid":"1.4","version":"77'128","reported_seq":1072,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-09-20T22:48:43.220998+0000","last_change":"2024-09-20T22:46:15.181632+0000","last_active":"2024-09-20T22:48:43.220998+0000","last_peered":"2024-09-20T22:48:43.220998+0000","last_clean":"2024-09-20T22:48:43.220998+0000","last_became_active":"2024-09-20T22:46:15.181305+0000","last_became_peered":"2024-09-20T22:46:15.181305+0000","last_unstale":"2024-09-20T22:48:43.220998+0000","last_undegraded":"2024-09-20T22:48:43.220998+0000","last_fullsized":"2024-09-20T22:48:43.220998+0000","mapping_epoch":94,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:45:54.131305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:45:54.131305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":59,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"74'14","reported_seq":103,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-09-20T22:48:43.207603+0000","last_change":"2024-09-20T22:48:05.262315+0000","last_active":"2024-09-20T22:48:43.207603+0000","last_peered":"2024-09-20T22:48:43.207603+0000","last_clean":"2024-09-20T22:48:43.207603+0000","last_became_active":"2024-09-20T22:48:05.235812+0000","last_became_peered":"2024-09-20T22:48:05.235812+0000","last_unstale":"2024-09-20T22:48:43.207603+0000","last_undegraded":"2024-09-20T22:48:43.207603+0000","last_fullsized":"2024-09-20T22:48:43.207603+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'14","last_scrub_stamp":"2024-09-20T22:46:48.202311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:46:48.202311+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"67'1","reported_seq":141,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-09-20T22:48:43.212609+0000","last_change":"2024-09-20T22:48:36.570753+0000","last_active":"2024-09-20T22:48:43.212609+0000","last_peered":"2024-09-20T22:48:43.212609+0000","last_clean":"2024-09-20T22:48:43.212609+0000","last_became_active":"2024-09-20T22:48:05.237425+0000","last_became_peered":"2024-09-20T22:48:05.237425+0000","last_unstale":"2024-09-20T22:48:43.212609+0000","last_undegraded":"2024-09-20T22:48:43.212609+0000","last_fullsized":"2024-09-20T22:48:43.212609+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-09-20T22:48:36.570672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:48:36.570672+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:49:55.065551+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"408'12","reported_seq":899,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-09-20T22:48:43.219189+0000","last_change":"2024-09-20T22:48:05.247422+0000","last_active":"2024-09-20T22:48:43.219189+0000","last_peered":"2024-09-20T22:48:43.219189+0000","last_clean":"2024-09-20T22:48:43.219189+0000","last_became_active":"2024-09-20T22:48:05.234607+0000","last_became_peered":"2024-09-20T22:48:05.234607+0000","last_unstale":"2024-09-20T22:48:43.219189+0000","last_undegraded":"2024-09-20T22:48:43.219189+0000","last_fullsized":"2024-09-20T22:48:43.219189+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"159'9","last_scrub_stamp":"2024-09-20T22:45:23.017605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:45:23.017605+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":768,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-09-20T22:48:43.216180+0000","last_change":"2024-09-20T22:47:16.381511+0000","last_active":"2024-09-20T22:48:43.216180+0000","last_peered":"2024-09-20T22:48:43.216180+0000","last_clean":"2024-09-20T22:48:43.216180+0000","last_became_active":"2024-09-20T22:41:44.831901+0000","last_became_peered":"2024-09-20T22:41:44.831901+0000","last_unstale":"2024-09-20T22:48:43.216180+0000","last_undegraded":"2024-09-20T22:48:43.216180+0000","last_fullsized":"2024-09-20T22:48:43.216180+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:47:16.381390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:47:16.381390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1","version":"435'248","reported_seq":1931,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-09-20T22:48:43.208020+0000","last_change":"2024-09-20T22:48:43.208364+0000","last_active":"2024-09-20T22:48:43.208020+0000","last_peered":"2024-09-20T22:48:43.208020+0000","last_clean":"2024-09-20T22:48:43.208020+0000","last_became_active":"2024-09-20T22:48:08.208698+0000","last_became_peered":"2024-09-20T22:48:08.208698+0000","last_unstale":"2024-09-20T22:48:43.208020+0000","last_undegraded":"2024-09-20T22:48:43.208020+0000","last_fullsized":"2024-09-20T22:48:43.208020+0000","mapping_epoch":407,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":408,"parent":"0.0","parent_split_bits":0,"last_scrub":"420'231","last_scrub_stamp":"2024-09-20T22:48:22.915993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:48:22.915993+0000","objects_scrubbed":6,"log_size":248,"log_dups_size":0,"ondisk_log_size":248,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:49:28.113459+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00060902299999999999,"stat_sum":{"num_bytes":11160046,"num_objects":5,"num_object_clones":3,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":62,"num_read_kb":78459,"num_write":51,"num_write_kb":36252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":8732217,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"8d","length":"2"},{"start":"93","length":"2"},{"start":"9b","length":"1"},{"start":"9d","length":"1"},{"start":"9f","length":"2"},{"start":"a3","length":"2"},{"start":"a8","length":"1"},{"start":"aa","length":"4"}]},{"pgid":"1.7","version":"77'128","reported_seq":1077,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-09-20T22:48:43.214189+0000","last_change":"2024-09-20T22:48:03.022941+0000","last_active":"2024-09-20T22:48:43.214189+0000","last_peered":"2024-09-20T22:48:43.214189+0000","last_clean":"2024-09-20T22:48:43.214189+0000","last_became_active":"2024-09-20T22:46:03.169596+0000","last_became_peered":"2024-09-20T22:46:03.169596+0000","last_unstale":"2024-09-20T22:48:43.214189+0000","last_undegraded":"2024-09-20T22:48:43.214189+0000","last_fullsized":"2024-09-20T22:48:43.214189+0000","mapping_epoch":94,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:48:03.022838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:48:03.022838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:49:29.905235+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"74'10","reported_seq":872,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-09-20T22:48:43.209999+0000","last_change":"2024-09-20T22:48:24.335766+0000","last_active":"2024-09-20T22:48:43.209999+0000","last_peered":"2024-09-20T22:48:43.209999+0000","last_clean":"2024-09-20T22:48:43.209999+0000","last_became_active":"2024-09-20T22:48:05.251200+0000","last_became_peered":"2024-09-20T22:48:05.251200+0000","last_unstale":"2024-09-20T22:48:43.209999+0000","last_undegraded":"2024-09-20T22:48:43.209999+0000","last_fullsized":"2024-09-20T22:48:43.209999+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-09-20T22:48:24.335674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:48:24.335674+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:49:30.306070+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":790,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-09-20T22:48:43.229266+0000","last_change":"2024-09-20T22:48:04.520131+0000","last_active":"2024-09-20T22:48:43.229266+0000","last_peered":"2024-09-20T22:48:43.229266+0000","last_clean":"2024-09-20T22:48:43.229266+0000","last_became_active":"2024-09-20T22:41:38.155629+0000","last_became_peered":"2024-09-20T22:41:38.155629+0000","last_unstale":"2024-09-20T22:48:43.229266+0000","last_undegraded":"2024-09-20T22:48:43.229266+0000","last_fullsized":"2024-09-20T22:48:43.229266+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:48:04.520053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:48:04.520053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:49:32.650911+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"389'11","reported_seq":135,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-09-20T22:48:43.209788+0000","last_change":"2024-09-20T22:47:49.246973+0000","last_active":"2024-09-20T22:48:43.209788+0000","last_peered":"2024-09-20T22:48:43.209788+0000","last_clean":"2024-09-20T22:48:43.209788+0000","last_became_active":"2024-09-20T22:47:49.227264+0000","last_became_peered":"2024-09-20T22:47:49.227264+0000","last_unstale":"2024-09-20T22:48:43.209788+0000","last_undegraded":"2024-09-20T22:48:43.209788+0000","last_fullsized":"2024-09-20T22:48:43.209788+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"264'10","last_scrub_stamp":"2024-09-20T22:45:57.889636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:45:57.889636+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":119,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-09-20T22:48:43.210224+0000","last_change":"2024-09-20T22:47:49.232739+0000","last_active":"2024-09-20T22:48:43.210224+0000","last_peered":"2024-09-20T22:48:43.210224+0000","last_clean":"2024-09-20T22:48:43.210224+0000","last_became_active":"2024-09-20T22:47:49.232380+0000","last_became_peered":"2024-09-20T22:47:49.232380+0000","last_unstale":"2024-09-20T22:48:43.210224+0000","last_undegraded":"2024-09-20T22:48:43.210224+0000","last_fullsized":"2024-09-20T22:48:43.210224+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:46:44.117761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:46:44.117761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.0","version":"441'208","reported_seq":1962,"reported_epoch":443,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-20T22:48:43.262788+0000","last_change":"2024-09-20T22:48:43.262830+0000","last_active":"2024-09-20T22:48:43.262788+0000","last_peered":"2024-09-20T22:48:43.262788+0000","last_clean":"2024-09-20T22:48:43.262788+0000","last_became_active":"2024-09-20T22:43:37.136845+0000","last_became_peered":"2024-09-20T22:43:37.136845+0000","last_unstale":"2024-09-20T22:48:43.262788+0000","last_undegraded":"2024-09-20T22:48:43.262788+0000","last_fullsized":"2024-09-20T22:48:43.262788+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"280'122","last_scrub_stamp":"2024-09-20T22:46:03.006423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:46:03.006423+0000","objects_scrubbed":1,"log_size":208,"log_dups_size":0,"ondisk_log_size":208,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.034016946999999999,"stat_sum":{"num_bytes":10347665,"num_objects":6,"num_object_clones":4,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":81,"num_read_kb":83295,"num_write":50,"num_write_kb":11886,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"8d","length":"2"},{"start":"93","length":"2"},{"start":"9b","length":"1"},{"start":"9d","length":"1"},{"start":"a0","length":"1"},{"start":"a3","length":"2"},{"start":"a8","length":"1"},{"start":"aa","length":"1"},{"start":"ac","length":"2"}]},{"pgid":"1.6","version":"77'128","reported_seq":1099,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-09-20T22:48:43.213694+0000","last_change":"2024-09-20T22:48:02.056439+0000","last_active":"2024-09-20T22:48:43.213694+0000","last_peered":"2024-09-20T22:48:43.213694+0000","last_clean":"2024-09-20T22:48:43.213694+0000","last_became_active":"2024-09-20T22:47:52.212190+0000","last_became_peered":"2024-09-20T22:47:52.212190+0000","last_unstale":"2024-09-20T22:48:43.213694+0000","last_undegraded":"2024-09-20T22:48:43.213694+0000","last_fullsized":"2024-09-20T22:48:43.213694+0000","mapping_epoch":391,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:48:02.056333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:48:02.056333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:49:14.033978+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"74'19","reported_seq":137,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-09-20T22:48:43.209511+0000","last_change":"2024-09-20T22:47:49.249445+0000","last_active":"2024-09-20T22:48:43.209511+0000","last_peered":"2024-09-20T22:48:43.209511+0000","last_clean":"2024-09-20T22:48:43.209511+0000","last_became_active":"2024-09-20T22:47:49.222009+0000","last_became_peered":"2024-09-20T22:47:49.222009+0000","last_unstale":"2024-09-20T22:48:43.209511+0000","last_undegraded":"2024-09-20T22:48:43.209511+0000","last_fullsized":"2024-09-20T22:48:43.209511+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'19","last_scrub_stamp":"2024-09-20T22:47:13.403224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:47:13.403224+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":38,"num_read_kb":24,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":789,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-09-20T22:48:43.210926+0000","last_change":"2024-09-20T22:48:23.320100+0000","last_active":"2024-09-20T22:48:43.210926+0000","last_peered":"2024-09-20T22:48:43.210926+0000","last_clean":"2024-09-20T22:48:43.210926+0000","last_became_active":"2024-09-20T22:41:38.160832+0000","last_became_peered":"2024-09-20T22:41:38.160832+0000","last_unstale":"2024-09-20T22:48:43.210926+0000","last_undegraded":"2024-09-20T22:48:43.210926+0000","last_fullsized":"2024-09-20T22:48:43.210926+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:48:23.320031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:48:23.320031+0000","last_clean_scrub_stamp":"2024-09-20T22:48:23.320031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:49:24.057655+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":785,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-09-20T22:48:43.213158+0000","last_change":"2024-09-20T22:47:49.214380+0000","last_active":"2024-09-20T22:48:43.213158+0000","last_peered":"2024-09-20T22:48:43.213158+0000","last_clean":"2024-09-20T22:48:43.213158+0000","last_became_active":"2024-09-20T22:47:49.214055+0000","last_became_peered":"2024-09-20T22:47:49.214055+0000","last_unstale":"2024-09-20T22:48:43.213158+0000","last_undegraded":"2024-09-20T22:48:43.213158+0000","last_fullsized":"2024-09-20T22:48:43.213158+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:46:43.827680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:46:43.827680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":777,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-09-20T22:48:43.209314+0000","last_change":"2024-09-20T22:48:41.208933+0000","last_active":"2024-09-20T22:48:43.209314+0000","last_peered":"2024-09-20T22:48:43.209314+0000","last_clean":"2024-09-20T22:48:43.209314+0000","last_became_active":"2024-09-20T22:41:44.830538+0000","last_became_peered":"2024-09-20T22:41:44.830538+0000","last_unstale":"2024-09-20T22:48:43.209314+0000","last_undegraded":"2024-09-20T22:48:43.209314+0000","last_fullsized":"2024-09-20T22:48:43.209314+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:48:41.208853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:48:41.208853+0000","last_clean_scrub_stamp":"2024-09-20T22:48:41.208853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:02.984046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.7","version":"443'469","reported_seq":2341,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-09-20T22:48:43.236308+0000","last_change":"2024-09-20T22:48:43.236489+0000","last_active":"2024-09-20T22:48:43.236308+0000","last_peered":"2024-09-20T22:48:43.236308+0000","last_clean":"2024-09-20T22:48:43.236308+0000","last_became_active":"2024-09-20T22:48:43.224386+0000","last_became_peered":"2024-09-20T22:48:43.224386+0000","last_unstale":"2024-09-20T22:48:43.236308+0000","last_undegraded":"2024-09-20T22:48:43.236308+0000","last_fullsized":"2024-09-20T22:48:43.236308+0000","mapping_epoch":391,"log_start":"436'456","ondisk_log_start":"436'456","created":94,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"372'324","last_scrub_stamp":"2024-09-20T22:47:33.829464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:47:33.829464+0000","objects_scrubbed":7,"log_size":13,"log_dups_size":886,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.010672777,"stat_sum":{"num_bytes":39485691,"num_objects":21,"num_object_clones":16,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":3,"num_read":290,"num_read_kb":315441,"num_write":202,"num_write_kb":70935,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":115,"num_bytes_recovered":51615167,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"8d","length":"2"},{"start":"93","length":"2"},{"start":"9b","length":"1"},{"start":"9d","length":"1"},{"start":"9f","length":"2"},{"start":"a3","length":"2"},{"start":"a8","length":"1"},{"start":"aa","length":"4"}]},{"pgid":"1.1","version":"77'128","reported_seq":1088,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-09-20T22:48:43.227956+0000","last_change":"2024-09-20T22:47:52.208511+0000","last_active":"2024-09-20T22:48:43.227956+0000","last_peered":"2024-09-20T22:48:43.227956+0000","last_clean":"2024-09-20T22:48:43.227956+0000","last_became_active":"2024-09-20T22:47:52.208102+0000","last_became_peered":"2024-09-20T22:47:52.208102+0000","last_unstale":"2024-09-20T22:48:43.227956+0000","last_undegraded":"2024-09-20T22:48:43.227956+0000","last_fullsized":"2024-09-20T22:48:43.227956+0000","mapping_epoch":391,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:45:58.908433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:45:58.908433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"86'38","reported_seq":929,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-09-20T22:48:43.210365+0000","last_change":"2024-09-20T22:48:00.380100+0000","last_active":"2024-09-20T22:48:43.210365+0000","last_peered":"2024-09-20T22:48:43.210365+0000","last_clean":"2024-09-20T22:48:43.210365+0000","last_became_active":"2024-09-20T22:41:40.166958+0000","last_became_peered":"2024-09-20T22:41:40.166958+0000","last_unstale":"2024-09-20T22:48:43.210365+0000","last_undegraded":"2024-09-20T22:48:43.210365+0000","last_fullsized":"2024-09-20T22:48:43.210365+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'38","last_scrub_stamp":"2024-09-20T22:48:00.379987+0000","last_deep_scrub":"86'38","last_deep_scrub_stamp":"2024-09-20T22:42:43.935677+0000","last_clean_scrub_stamp":"2024-09-20T22:48:00.379987+0000","objects_scrubbed":10,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:49:30.129855+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":87,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-09-20T22:48:43.207571+0000","last_change":"2024-09-20T22:48:05.238748+0000","last_active":"2024-09-20T22:48:43.207571+0000","last_peered":"2024-09-20T22:48:43.207571+0000","last_clean":"2024-09-20T22:48:43.207571+0000","last_became_active":"2024-09-20T22:48:05.237756+0000","last_became_peered":"2024-09-20T22:48:05.237756+0000","last_unstale":"2024-09-20T22:48:43.207571+0000","last_undegraded":"2024-09-20T22:48:43.207571+0000","last_fullsized":"2024-09-20T22:48:43.207571+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:47:16.817468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:47:16.817468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"69'2","reported_seq":800,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-09-20T22:48:43.228872+0000","last_change":"2024-09-20T22:46:44.210668+0000","last_active":"2024-09-20T22:48:43.228872+0000","last_peered":"2024-09-20T22:48:43.228872+0000","last_clean":"2024-09-20T22:48:43.228872+0000","last_became_active":"2024-09-20T22:41:37.148479+0000","last_became_peered":"2024-09-20T22:41:37.148479+0000","last_unstale":"2024-09-20T22:48:43.228872+0000","last_undegraded":"2024-09-20T22:48:43.228872+0000","last_fullsized":"2024-09-20T22:48:43.228872+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2024-09-20T22:46:44.210559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:36.130684+0000","last_clean_scrub_stamp":"2024-09-20T22:46:44.210559+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00087280599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":125,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-09-20T22:48:43.211817+0000","last_change":"2024-09-20T22:48:05.236164+0000","last_active":"2024-09-20T22:48:43.211817+0000","last_peered":"2024-09-20T22:48:43.211817+0000","last_clean":"2024-09-20T22:48:43.211817+0000","last_became_active":"2024-09-20T22:48:05.233519+0000","last_became_peered":"2024-09-20T22:48:05.233519+0000","last_unstale":"2024-09-20T22:48:43.211817+0000","last_undegraded":"2024-09-20T22:48:43.211817+0000","last_fullsized":"2024-09-20T22:48:43.211817+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:46:06.104786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:46:06.104786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":781,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-09-20T22:48:43.219861+0000","last_change":"2024-09-20T22:48:05.234499+0000","last_active":"2024-09-20T22:48:43.219861+0000","last_peered":"2024-09-20T22:48:43.219861+0000","last_clean":"2024-09-20T22:48:43.219861+0000","last_became_active":"2024-09-20T22:48:05.234193+0000","last_became_peered":"2024-09-20T22:48:05.234193+0000","last_unstale":"2024-09-20T22:48:43.219861+0000","last_undegraded":"2024-09-20T22:48:43.219861+0000","last_fullsized":"2024-09-20T22:48:43.219861+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:45:58.212988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:45:58.212988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"443'129","reported_seq":372,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-09-20T22:48:43.217890+0000","last_change":"2024-09-20T22:48:43.218626+0000","last_active":"2024-09-20T22:48:43.217890+0000","last_peered":"2024-09-20T22:48:43.217890+0000","last_clean":"2024-09-20T22:48:43.217890+0000","last_became_active":"2024-09-20T22:48:05.242705+0000","last_became_peered":"2024-09-20T22:48:05.242705+0000","last_unstale":"2024-09-20T22:48:43.217890+0000","last_undegraded":"2024-09-20T22:48:43.217890+0000","last_fullsized":"2024-09-20T22:48:43.217890+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"305'57","last_scrub_stamp":"2024-09-20T22:47:13.636468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:47:13.636468+0000","objects_scrubbed":1,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0078693579999999999,"stat_sum":{"num_bytes":12416314,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":44975,"num_write":32,"num_write_kb":8665,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":10294275,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"8d","length":"2"},{"start":"93","length":"2"},{"start":"9b","length":"1"},{"start":"9d","length":"1"},{"start":"9f","length":"2"},{"start":"a3","length":"2"},{"start":"a8","length":"1"},{"start":"aa","length":"4"}]},{"pgid":"1.2","version":"77'128","reported_seq":1156,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-09-20T22:48:43.215910+0000","last_change":"2024-09-20T22:48:08.203458+0000","last_active":"2024-09-20T22:48:43.215910+0000","last_peered":"2024-09-20T22:48:43.215910+0000","last_clean":"2024-09-20T22:48:43.215910+0000","last_became_active":"2024-09-20T22:48:08.203028+0000","last_became_peered":"2024-09-20T22:48:08.203028+0000","last_unstale":"2024-09-20T22:48:43.215910+0000","last_undegraded":"2024-09-20T22:48:43.215910+0000","last_fullsized":"2024-09-20T22:48:43.215910+0000","mapping_epoch":407,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":408,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:47:56.410772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:47:56.410772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:49:17.378865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"74'13","reported_seq":840,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-09-20T22:48:43.210643+0000","last_change":"2024-09-20T22:47:49.267177+0000","last_active":"2024-09-20T22:48:43.210643+0000","last_peered":"2024-09-20T22:48:43.210643+0000","last_clean":"2024-09-20T22:48:43.210643+0000","last_became_active":"2024-09-20T22:47:49.228702+0000","last_became_peered":"2024-09-20T22:47:49.228702+0000","last_unstale":"2024-09-20T22:48:43.210643+0000","last_undegraded":"2024-09-20T22:48:43.210643+0000","last_fullsized":"2024-09-20T22:48:43.210643+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'13","last_scrub_stamp":"2024-09-20T22:46:04.145892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:46:04.145892+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":789,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-09-20T22:48:43.208714+0000","last_change":"2024-09-20T22:47:55.478609+0000","last_active":"2024-09-20T22:48:43.208714+0000","last_peered":"2024-09-20T22:48:43.208714+0000","last_clean":"2024-09-20T22:48:43.208714+0000","last_became_active":"2024-09-20T22:41:38.156837+0000","last_became_peered":"2024-09-20T22:41:38.156837+0000","last_unstale":"2024-09-20T22:48:43.208714+0000","last_undegraded":"2024-09-20T22:48:43.208714+0000","last_fullsized":"2024-09-20T22:48:43.208714+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:47:55.478501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:47:55.478501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:49:23.720355+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"380'2","reported_seq":826,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-09-20T22:48:43.224807+0000","last_change":"2024-09-20T22:47:49.233750+0000","last_active":"2024-09-20T22:48:43.224807+0000","last_peered":"2024-09-20T22:48:43.224807+0000","last_clean":"2024-09-20T22:48:43.224807+0000","last_became_active":"2024-09-20T22:47:49.222812+0000","last_became_peered":"2024-09-20T22:47:49.222812+0000","last_unstale":"2024-09-20T22:48:43.224807+0000","last_undegraded":"2024-09-20T22:48:43.224807+0000","last_fullsized":"2024-09-20T22:48:43.224807+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"380'2","last_scrub_stamp":"2024-09-20T22:47:40.983039+0000","last_deep_scrub":"380'2","last_deep_scrub_stamp":"2024-09-20T22:47:40.983039+0000","last_clean_scrub_stamp":"2024-09-20T22:47:40.983039+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:59.344462+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.001301459,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":780,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-09-20T22:48:43.225356+0000","last_change":"2024-09-20T22:47:22.881180+0000","last_active":"2024-09-20T22:48:43.225356+0000","last_peered":"2024-09-20T22:48:43.225356+0000","last_clean":"2024-09-20T22:48:43.225356+0000","last_became_active":"2024-09-20T22:41:42.185112+0000","last_became_peered":"2024-09-20T22:41:42.185112+0000","last_unstale":"2024-09-20T22:48:43.225356+0000","last_undegraded":"2024-09-20T22:48:43.225356+0000","last_fullsized":"2024-09-20T22:48:43.225356+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:47:22.881109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:47:22.881109+0000","last_clean_scrub_stamp":"2024-09-20T22:47:22.881109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:46.218094+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":768,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-09-20T22:48:43.217381+0000","last_change":"2024-09-20T22:46:27.448645+0000","last_active":"2024-09-20T22:48:43.217381+0000","last_peered":"2024-09-20T22:48:43.217381+0000","last_clean":"2024-09-20T22:48:43.217381+0000","last_became_active":"2024-09-20T22:41:44.830515+0000","last_became_peered":"2024-09-20T22:41:44.830515+0000","last_unstale":"2024-09-20T22:48:43.217381+0000","last_undegraded":"2024-09-20T22:48:43.217381+0000","last_fullsized":"2024-09-20T22:48:43.217381+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:46:27.448572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:46:27.448572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"443'186","reported_seq":1980,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-09-20T22:48:43.223878+0000","last_change":"2024-09-20T22:48:43.224339+0000","last_active":"2024-09-20T22:48:43.223878+0000","last_peered":"2024-09-20T22:48:43.223878+0000","last_clean":"2024-09-20T22:48:43.223878+0000","last_became_active":"2024-09-20T22:47:52.206033+0000","last_became_peered":"2024-09-20T22:47:52.206033+0000","last_unstale":"2024-09-20T22:48:43.223878+0000","last_undegraded":"2024-09-20T22:48:43.223878+0000","last_fullsized":"2024-09-20T22:48:43.223878+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'168","last_scrub_stamp":"2024-09-20T22:48:22.483112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:48:22.483112+0000","objects_scrubbed":7,"log_size":186,"log_dups_size":0,"ondisk_log_size":186,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:49:30.735972+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0072663659999999998,"stat_sum":{"num_bytes":11338963,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":38714,"num_write":68,"num_write_kb":31788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":19673726,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"8d","length":"2"},{"start":"93","length":"2"},{"start":"9b","length":"1"},{"start":"9d","length":"1"},{"start":"9f","length":"2"},{"start":"a3","length":"2"},{"start":"a8","length":"1"},{"start":"aa","length":"4"}]},{"pgid":"1.3","version":"77'128","reported_seq":1104,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-09-20T22:48:43.209681+0000","last_change":"2024-09-20T22:48:37.305505+0000","last_active":"2024-09-20T22:48:43.209681+0000","last_peered":"2024-09-20T22:48:43.209681+0000","last_clean":"2024-09-20T22:48:43.209681+0000","last_became_active":"2024-09-20T22:48:08.208402+0000","last_became_peered":"2024-09-20T22:48:08.208402+0000","last_unstale":"2024-09-20T22:48:43.209681+0000","last_undegraded":"2024-09-20T22:48:43.209681+0000","last_fullsized":"2024-09-20T22:48:43.209681+0000","mapping_epoch":407,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":408,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:48:37.305426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:48:37.305426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:49:46.648477+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"74'12","reported_seq":133,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-09-20T22:48:43.212923+0000","last_change":"2024-09-20T22:47:49.246080+0000","last_active":"2024-09-20T22:48:43.212923+0000","last_peered":"2024-09-20T22:48:43.212923+0000","last_clean":"2024-09-20T22:48:43.212923+0000","last_became_active":"2024-09-20T22:47:49.224223+0000","last_became_peered":"2024-09-20T22:47:49.224223+0000","last_unstale":"2024-09-20T22:48:43.212923+0000","last_undegraded":"2024-09-20T22:48:43.212923+0000","last_fullsized":"2024-09-20T22:48:43.212923+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-09-20T22:47:39.293455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:47:39.293455+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":133,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-09-20T22:48:43.210930+0000","last_change":"2024-09-20T22:48:05.233412+0000","last_active":"2024-09-20T22:48:43.210930+0000","last_peered":"2024-09-20T22:48:43.210930+0000","last_clean":"2024-09-20T22:48:43.210930+0000","last_became_active":"2024-09-20T22:48:05.233093+0000","last_became_peered":"2024-09-20T22:48:05.233093+0000","last_unstale":"2024-09-20T22:48:43.210930+0000","last_undegraded":"2024-09-20T22:48:43.210930+0000","last_fullsized":"2024-09-20T22:48:43.210930+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:47:55.576301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:44:19.834641+0000","last_clean_scrub_stamp":"2024-09-20T22:47:55.576301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:49:20.279866+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"75'5","reported_seq":1229,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-09-20T22:48:43.217638+0000","last_change":"2024-09-20T22:47:49.235292+0000","last_active":"2024-09-20T22:48:43.217638+0000","last_peered":"2024-09-20T22:48:43.217638+0000","last_clean":"2024-09-20T22:48:43.217638+0000","last_became_active":"2024-09-20T22:47:49.214414+0000","last_became_peered":"2024-09-20T22:47:49.214414+0000","last_unstale":"2024-09-20T22:48:43.217638+0000","last_undegraded":"2024-09-20T22:48:43.217638+0000","last_fullsized":"2024-09-20T22:48:43.217638+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-09-20T22:46:22.436424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:36.130684+0000","last_clean_scrub_stamp":"2024-09-20T22:46:22.436424+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00078631700000000003,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":416,"num_read_kb":411,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":794,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-09-20T22:48:43.229696+0000","last_change":"2024-09-20T22:48:34.535169+0000","last_active":"2024-09-20T22:48:43.229696+0000","last_peered":"2024-09-20T22:48:43.229696+0000","last_clean":"2024-09-20T22:48:43.229696+0000","last_became_active":"2024-09-20T22:47:49.227371+0000","last_became_peered":"2024-09-20T22:47:49.227371+0000","last_unstale":"2024-09-20T22:48:43.229696+0000","last_undegraded":"2024-09-20T22:48:43.229696+0000","last_fullsized":"2024-09-20T22:48:43.229696+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:48:34.535100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:48:34.535100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:49:46.167401+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":777,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-09-20T22:48:43.209077+0000","last_change":"2024-09-20T22:47:59.388634+0000","last_active":"2024-09-20T22:48:43.209077+0000","last_peered":"2024-09-20T22:48:43.209077+0000","last_clean":"2024-09-20T22:48:43.209077+0000","last_became_active":"2024-09-20T22:41:44.199046+0000","last_became_peered":"2024-09-20T22:41:44.199046+0000","last_unstale":"2024-09-20T22:48:43.209077+0000","last_undegraded":"2024-09-20T22:48:43.209077+0000","last_fullsized":"2024-09-20T22:48:43.209077+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:47:59.388571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:47:59.388571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:49:09.987768+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.6","version":"443'181","reported_seq":1908,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-09-20T22:48:43.234208+0000","last_change":"2024-09-20T22:48:43.234888+0000","last_active":"2024-09-20T22:48:43.234208+0000","last_peered":"2024-09-20T22:48:43.234208+0000","last_clean":"2024-09-20T22:48:43.234208+0000","last_became_active":"2024-09-20T22:47:52.216302+0000","last_became_peered":"2024-09-20T22:47:52.216302+0000","last_unstale":"2024-09-20T22:48:43.234208+0000","last_undegraded":"2024-09-20T22:48:43.234208+0000","last_fullsized":"2024-09-20T22:48:43.234208+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"440'177","last_scrub_stamp":"2024-09-20T22:48:40.602342+0000","last_deep_scrub":"440'177","last_deep_scrub_stamp":"2024-09-20T22:48:40.602342+0000","last_clean_scrub_stamp":"2024-09-20T22:48:40.602342+0000","objects_scrubbed":4,"log_size":181,"log_dups_size":0,"ondisk_log_size":181,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:09.840044+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0.021794540000000001,"stat_sum":{"num_bytes":6672511,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":31,"num_read_kb":32795,"num_write":27,"num_write_kb":8197,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":5260887,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"8d","length":"2"},{"start":"93","length":"2"},{"start":"9b","length":"1"},{"start":"9d","length":"1"},{"start":"9f","length":"2"},{"start":"a3","length":"2"},{"start":"a8","length":"1"},{"start":"aa","length":"4"}]},{"pgid":"4.5","version":"408'20","reported_seq":989,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-09-20T22:48:43.214224+0000","last_change":"2024-09-20T22:48:20.112710+0000","last_active":"2024-09-20T22:48:43.214224+0000","last_peered":"2024-09-20T22:48:43.214224+0000","last_clean":"2024-09-20T22:48:43.214224+0000","last_became_active":"2024-09-20T22:48:05.243523+0000","last_became_peered":"2024-09-20T22:48:05.243523+0000","last_unstale":"2024-09-20T22:48:43.214224+0000","last_undegraded":"2024-09-20T22:48:43.214224+0000","last_fullsized":"2024-09-20T22:48:43.214224+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"408'20","last_scrub_stamp":"2024-09-20T22:48:20.112621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:48:20.112621+0000","objects_scrubbed":8,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:49:46.242509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"67'1","reported_seq":779,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-09-20T22:48:43.222480+0000","last_change":"2024-09-20T22:45:22.053343+0000","last_active":"2024-09-20T22:48:43.222480+0000","last_peered":"2024-09-20T22:48:43.222480+0000","last_clean":"2024-09-20T22:48:43.222480+0000","last_became_active":"2024-09-20T22:41:38.158917+0000","last_became_peered":"2024-09-20T22:41:38.158917+0000","last_unstale":"2024-09-20T22:48:43.222480+0000","last_undegraded":"2024-09-20T22:48:43.222480+0000","last_fullsized":"2024-09-20T22:48:43.222480+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-09-20T22:45:22.053262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:45:22.053262+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"77'128","reported_seq":1128,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-09-20T22:48:43.217831+0000","last_change":"2024-09-20T22:47:52.206552+0000","last_active":"2024-09-20T22:48:43.217831+0000","last_peered":"2024-09-20T22:48:43.217831+0000","last_clean":"2024-09-20T22:48:43.217831+0000","last_became_active":"2024-09-20T22:47:52.206169+0000","last_became_peered":"2024-09-20T22:47:52.206169+0000","last_unstale":"2024-09-20T22:48:43.217831+0000","last_undegraded":"2024-09-20T22:48:43.217831+0000","last_fullsized":"2024-09-20T22:48:43.217831+0000","mapping_epoch":391,"log_start":"77'128","ondisk_log_start":"77'128","created":19,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:47:22.417099+0000","last_deep_scrub":"77'128","last_deep_scrub_stamp":"2024-09-20T22:47:22.417099+0000","last_clean_scrub_stamp":"2024-09-20T22:47:22.417099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":780,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-09-20T22:48:43.217174+0000","last_change":"2024-09-20T22:48:35.359093+0000","last_active":"2024-09-20T22:48:43.217174+0000","last_peered":"2024-09-20T22:48:43.217174+0000","last_clean":"2024-09-20T22:48:43.217174+0000","last_became_active":"2024-09-20T22:41:42.184387+0000","last_became_peered":"2024-09-20T22:41:42.184387+0000","last_unstale":"2024-09-20T22:48:43.217174+0000","last_undegraded":"2024-09-20T22:48:43.217174+0000","last_fullsized":"2024-09-20T22:48:43.217174+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:48:35.359016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:48:35.359016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:00.848745+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":790,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-09-20T22:48:43.229531+0000","last_change":"2024-09-20T22:48:18.622779+0000","last_active":"2024-09-20T22:48:43.229531+0000","last_peered":"2024-09-20T22:48:43.229531+0000","last_clean":"2024-09-20T22:48:43.229531+0000","last_became_active":"2024-09-20T22:48:05.241602+0000","last_became_peered":"2024-09-20T22:48:05.241602+0000","last_unstale":"2024-09-20T22:48:43.229531+0000","last_undegraded":"2024-09-20T22:48:43.229531+0000","last_fullsized":"2024-09-20T22:48:43.229531+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:48:18.622674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:48:18.622674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:49:26.359118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"443'220","reported_seq":2012,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-09-20T22:48:43.222387+0000","last_change":"2024-09-20T22:48:43.222878+0000","last_active":"2024-09-20T22:48:43.222387+0000","last_peered":"2024-09-20T22:48:43.222387+0000","last_clean":"2024-09-20T22:48:43.222387+0000","last_became_active":"2024-09-20T22:47:52.208200+0000","last_became_peered":"2024-09-20T22:47:52.208200+0000","last_unstale":"2024-09-20T22:48:43.222387+0000","last_undegraded":"2024-09-20T22:48:43.222387+0000","last_fullsized":"2024-09-20T22:48:43.222387+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"327'109","last_scrub_stamp":"2024-09-20T22:46:51.808234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:46:51.808234+0000","objects_scrubbed":2,"log_size":220,"log_dups_size":0,"ondisk_log_size":220,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0089026850000000005,"stat_sum":{"num_bytes":10866030,"num_objects":5,"num_object_clones":3,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":95,"num_read_kb":93499,"num_write":53,"num_write_kb":21551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":13529610,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"8d","length":"2"},{"start":"93","length":"2"},{"start":"9b","length":"1"},{"start":"9d","length":"1"},{"start":"9f","length":"2"},{"start":"a3","length":"2"},{"start":"a8","length":"1"},{"start":"aa","length":"4"}]},{"pgid":"4.e","version":"74'11","reported_seq":99,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-09-20T22:48:43.208729+0000","last_change":"2024-09-20T22:48:05.271944+0000","last_active":"2024-09-20T22:48:43.208729+0000","last_peered":"2024-09-20T22:48:43.208729+0000","last_clean":"2024-09-20T22:48:43.208729+0000","last_became_active":"2024-09-20T22:48:05.248271+0000","last_became_peered":"2024-09-20T22:48:05.248271+0000","last_unstale":"2024-09-20T22:48:43.208729+0000","last_undegraded":"2024-09-20T22:48:43.208729+0000","last_fullsized":"2024-09-20T22:48:43.208729+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-09-20T22:47:59.092235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:47:59.092235+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:49:11.673300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":87,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-09-20T22:48:43.207164+0000","last_change":"2024-09-20T22:48:05.255396+0000","last_active":"2024-09-20T22:48:43.207164+0000","last_peered":"2024-09-20T22:48:43.207164+0000","last_clean":"2024-09-20T22:48:43.207164+0000","last_became_active":"2024-09-20T22:48:05.254688+0000","last_became_peered":"2024-09-20T22:48:05.254688+0000","last_unstale":"2024-09-20T22:48:43.207164+0000","last_undegraded":"2024-09-20T22:48:43.207164+0000","last_fullsized":"2024-09-20T22:48:43.207164+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:46:30.871771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:46:30.871771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":786,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-09-20T22:48:43.229511+0000","last_change":"2024-09-20T22:48:05.234120+0000","last_active":"2024-09-20T22:48:43.229511+0000","last_peered":"2024-09-20T22:48:43.229511+0000","last_clean":"2024-09-20T22:48:43.229511+0000","last_became_active":"2024-09-20T22:48:05.233778+0000","last_became_peered":"2024-09-20T22:48:05.233778+0000","last_unstale":"2024-09-20T22:48:43.229511+0000","last_undegraded":"2024-09-20T22:48:43.229511+0000","last_fullsized":"2024-09-20T22:48:43.229511+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:47:38.288926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:47:38.288926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:49:08.085240+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":769,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-09-20T22:48:43.220870+0000","last_change":"2024-09-20T22:48:40.127195+0000","last_active":"2024-09-20T22:48:43.220870+0000","last_peered":"2024-09-20T22:48:43.220870+0000","last_clean":"2024-09-20T22:48:43.220870+0000","last_became_active":"2024-09-20T22:41:44.832009+0000","last_became_peered":"2024-09-20T22:41:44.832009+0000","last_unstale":"2024-09-20T22:48:43.220870+0000","last_undegraded":"2024-09-20T22:48:43.220870+0000","last_fullsized":"2024-09-20T22:48:43.220870+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:48:40.127123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:48:40.127123+0000","last_clean_scrub_stamp":"2024-09-20T22:48:40.127123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:04.008351+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.e","version":"443'314","reported_seq":2132,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-09-20T22:48:43.229754+0000","last_change":"2024-09-20T22:48:43.230058+0000","last_active":"2024-09-20T22:48:43.229754+0000","last_peered":"2024-09-20T22:48:43.229754+0000","last_clean":"2024-09-20T22:48:43.229754+0000","last_became_active":"2024-09-20T22:47:52.209452+0000","last_became_peered":"2024-09-20T22:47:52.209452+0000","last_unstale":"2024-09-20T22:48:43.229754+0000","last_undegraded":"2024-09-20T22:48:43.229754+0000","last_fullsized":"2024-09-20T22:48:43.229754+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"377'202","last_scrub_stamp":"2024-09-20T22:47:42.000554+0000","last_deep_scrub":"174'60","last_deep_scrub_stamp":"2024-09-20T22:44:18.301776+0000","last_clean_scrub_stamp":"2024-09-20T22:47:42.000554+0000","objects_scrubbed":5,"log_size":314,"log_dups_size":0,"ondisk_log_size":314,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0048970120000000001,"stat_sum":{"num_bytes":20681829,"num_objects":10,"num_object_clones":8,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":1,"num_read":146,"num_read_kb":152340,"num_write":96,"num_write_kb":37433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":23110741,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"8d","length":"2"},{"start":"93","length":"2"},{"start":"9b","length":"1"},{"start":"9d","length":"1"},{"start":"9f","length":"2"},{"start":"a3","length":"2"},{"start":"a8","length":"1"},{"start":"aa","length":"4"}]},{"pgid":"1.8","version":"77'128","reported_seq":1048,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-09-20T22:48:43.228678+0000","last_change":"2024-09-20T22:48:17.613298+0000","last_active":"2024-09-20T22:48:43.228678+0000","last_peered":"2024-09-20T22:48:43.228678+0000","last_clean":"2024-09-20T22:48:43.228678+0000","last_became_active":"2024-09-20T22:42:52.405594+0000","last_became_peered":"2024-09-20T22:42:52.405594+0000","last_unstale":"2024-09-20T22:48:43.228678+0000","last_undegraded":"2024-09-20T22:48:43.228678+0000","last_fullsized":"2024-09-20T22:48:43.228678+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:48:17.613185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:48:17.613185+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:49:18.820494+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"74'17","reported_seq":105,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-09-20T22:48:43.208527+0000","last_change":"2024-09-20T22:48:05.273509+0000","last_active":"2024-09-20T22:48:43.208527+0000","last_peered":"2024-09-20T22:48:43.208527+0000","last_clean":"2024-09-20T22:48:43.208527+0000","last_became_active":"2024-09-20T22:48:05.247971+0000","last_became_peered":"2024-09-20T22:48:05.247971+0000","last_unstale":"2024-09-20T22:48:43.208527+0000","last_undegraded":"2024-09-20T22:48:43.208527+0000","last_fullsized":"2024-09-20T22:48:43.208527+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'17","last_scrub_stamp":"2024-09-20T22:47:38.052166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:47:38.052166+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:59.439253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":28,"num_read_kb":18,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":329,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-09-20T22:48:43.223954+0000","last_change":"2024-09-20T22:48:06.135135+0000","last_active":"2024-09-20T22:48:43.223954+0000","last_peered":"2024-09-20T22:48:43.223954+0000","last_clean":"2024-09-20T22:48:43.223954+0000","last_became_active":"2024-09-20T22:48:05.240481+0000","last_became_peered":"2024-09-20T22:48:05.240481+0000","last_unstale":"2024-09-20T22:48:43.223954+0000","last_undegraded":"2024-09-20T22:48:43.223954+0000","last_fullsized":"2024-09-20T22:48:43.223954+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:48:06.135068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:35.492562+0000","last_clean_scrub_stamp":"2024-09-20T22:48:06.135068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:49:31.308860+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":797,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-09-20T22:48:43.209995+0000","last_change":"2024-09-20T22:48:05.252451+0000","last_active":"2024-09-20T22:48:43.209995+0000","last_peered":"2024-09-20T22:48:43.209995+0000","last_clean":"2024-09-20T22:48:43.209995+0000","last_became_active":"2024-09-20T22:48:05.250175+0000","last_became_peered":"2024-09-20T22:48:05.250175+0000","last_unstale":"2024-09-20T22:48:43.209995+0000","last_undegraded":"2024-09-20T22:48:43.209995+0000","last_fullsized":"2024-09-20T22:48:43.209995+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:47:36.658662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:47:36.658662+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:50.405263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":796,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-09-20T22:48:43.224198+0000","last_change":"2024-09-20T22:48:05.239302+0000","last_active":"2024-09-20T22:48:43.224198+0000","last_peered":"2024-09-20T22:48:43.224198+0000","last_clean":"2024-09-20T22:48:43.224198+0000","last_became_active":"2024-09-20T22:48:05.239006+0000","last_became_peered":"2024-09-20T22:48:05.239006+0000","last_unstale":"2024-09-20T22:48:43.224198+0000","last_undegraded":"2024-09-20T22:48:43.224198+0000","last_fullsized":"2024-09-20T22:48:43.224198+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:47:58.989773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:47:58.989773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:49:24.841257+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"443'419","reported_seq":2198,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-09-20T22:48:43.234237+0000","last_change":"2024-09-20T22:48:43.234672+0000","last_active":"2024-09-20T22:48:43.234237+0000","last_peered":"2024-09-20T22:48:43.234237+0000","last_clean":"2024-09-20T22:48:43.234237+0000","last_became_active":"2024-09-20T22:47:52.216198+0000","last_became_peered":"2024-09-20T22:47:52.216198+0000","last_unstale":"2024-09-20T22:48:43.234237+0000","last_undegraded":"2024-09-20T22:48:43.234237+0000","last_fullsized":"2024-09-20T22:48:43.234237+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"395'325","last_scrub_stamp":"2024-09-20T22:47:57.577936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:47:57.577936+0000","objects_scrubbed":7,"log_size":419,"log_dups_size":0,"ondisk_log_size":419,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:59.764289+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.022196486000000001,"stat_sum":{"num_bytes":16908933,"num_objects":9,"num_object_clones":6,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":2,"num_read":214,"num_read_kb":214524,"num_write":133,"num_write_kb":43892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":53,"num_bytes_recovered":14983556,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"8d","length":"2"},{"start":"93","length":"2"},{"start":"9b","length":"1"},{"start":"9d","length":"1"},{"start":"9f","length":"2"},{"start":"a3","length":"2"},{"start":"a8","length":"1"},{"start":"aa","length":"4"}]},{"pgid":"1.9","version":"77'128","reported_seq":1087,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-09-20T22:48:43.216430+0000","last_change":"2024-09-20T22:48:03.252010+0000","last_active":"2024-09-20T22:48:43.216430+0000","last_peered":"2024-09-20T22:48:43.216430+0000","last_clean":"2024-09-20T22:48:43.216430+0000","last_became_active":"2024-09-20T22:42:52.400353+0000","last_became_peered":"2024-09-20T22:42:52.400353+0000","last_unstale":"2024-09-20T22:48:43.216430+0000","last_undegraded":"2024-09-20T22:48:43.216430+0000","last_fullsized":"2024-09-20T22:48:43.216430+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:48:03.251924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:48:03.251924+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:49:08.807948+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"74'10","reported_seq":99,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-09-20T22:48:43.208009+0000","last_change":"2024-09-20T22:48:05.262827+0000","last_active":"2024-09-20T22:48:43.208009+0000","last_peered":"2024-09-20T22:48:43.208009+0000","last_clean":"2024-09-20T22:48:43.208009+0000","last_became_active":"2024-09-20T22:48:05.242811+0000","last_became_peered":"2024-09-20T22:48:05.242811+0000","last_unstale":"2024-09-20T22:48:43.208009+0000","last_undegraded":"2024-09-20T22:48:43.208009+0000","last_fullsized":"2024-09-20T22:48:43.208009+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-09-20T22:47:21.092309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:47:21.092309+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":805,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-09-20T22:48:43.221857+0000","last_change":"2024-09-20T22:48:05.234296+0000","last_active":"2024-09-20T22:48:43.221857+0000","last_peered":"2024-09-20T22:48:43.221857+0000","last_clean":"2024-09-20T22:48:43.221857+0000","last_became_active":"2024-09-20T22:48:05.233999+0000","last_became_peered":"2024-09-20T22:48:05.233999+0000","last_unstale":"2024-09-20T22:48:43.221857+0000","last_undegraded":"2024-09-20T22:48:43.221857+0000","last_fullsized":"2024-09-20T22:48:43.221857+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:45:57.194715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:45:57.194715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"261'9","reported_seq":883,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-09-20T22:48:43.282679+0000","last_change":"2024-09-20T22:48:43.282707+0000","last_active":"2024-09-20T22:48:43.282679+0000","last_peered":"2024-09-20T22:48:43.282679+0000","last_clean":"2024-09-20T22:48:43.282679+0000","last_became_active":"2024-09-20T22:41:42.185863+0000","last_became_peered":"2024-09-20T22:41:42.185863+0000","last_unstale":"2024-09-20T22:48:43.282679+0000","last_undegraded":"2024-09-20T22:48:43.282679+0000","last_fullsized":"2024-09-20T22:48:43.282679+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'9","last_scrub_stamp":"2024-09-20T22:48:43.282627+0000","last_deep_scrub":"261'9","last_deep_scrub_stamp":"2024-09-20T22:46:50.974735+0000","last_clean_scrub_stamp":"2024-09-20T22:48:43.282627+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:49:57.177125+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":87,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-09-20T22:48:43.208534+0000","last_change":"2024-09-20T22:48:05.250642+0000","last_active":"2024-09-20T22:48:43.208534+0000","last_peered":"2024-09-20T22:48:43.208534+0000","last_clean":"2024-09-20T22:48:43.208534+0000","last_became_active":"2024-09-20T22:48:05.250284+0000","last_became_peered":"2024-09-20T22:48:05.250284+0000","last_unstale":"2024-09-20T22:48:43.208534+0000","last_undegraded":"2024-09-20T22:48:43.208534+0000","last_fullsized":"2024-09-20T22:48:43.208534+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:46:24.906516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:46:24.906516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.8","version":"443'293","reported_seq":1940,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-09-20T22:48:43.231293+0000","last_change":"2024-09-20T22:48:43.231587+0000","last_active":"2024-09-20T22:48:43.231293+0000","last_peered":"2024-09-20T22:48:43.231293+0000","last_clean":"2024-09-20T22:48:43.231293+0000","last_became_active":"2024-09-20T22:48:31.224359+0000","last_became_peered":"2024-09-20T22:48:31.224359+0000","last_unstale":"2024-09-20T22:48:43.231293+0000","last_undegraded":"2024-09-20T22:48:43.231293+0000","last_fullsized":"2024-09-20T22:48:43.231293+0000","mapping_epoch":407,"log_start":"426'268","ondisk_log_start":"426'268","created":94,"last_epoch_clean":431,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'98","last_scrub_stamp":"2024-09-20T22:46:45.991157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:46:45.991157+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":445,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0068006550000000001,"stat_sum":{"num_bytes":18519177,"num_objects":9,"num_object_clones":6,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":1,"num_read":126,"num_read_kb":115266,"num_write":113,"num_write_kb":43138,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":26027409,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"8d","length":"2"},{"start":"93","length":"2"},{"start":"9b","length":"1"},{"start":"9d","length":"1"},{"start":"9f","length":"2"},{"start":"a3","length":"2"},{"start":"a8","length":"1"},{"start":"aa","length":"4"}]},{"pgid":"4.b","version":"74'9","reported_seq":153,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-09-20T22:48:43.209594+0000","last_change":"2024-09-20T22:48:05.256987+0000","last_active":"2024-09-20T22:48:43.209594+0000","last_peered":"2024-09-20T22:48:43.209594+0000","last_clean":"2024-09-20T22:48:43.209594+0000","last_became_active":"2024-09-20T22:48:05.233267+0000","last_became_peered":"2024-09-20T22:48:05.233267+0000","last_unstale":"2024-09-20T22:48:43.209594+0000","last_undegraded":"2024-09-20T22:48:43.209594+0000","last_fullsized":"2024-09-20T22:48:43.209594+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-09-20T22:48:03.584242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:48:03.584242+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:49:11.956491+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":782,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-09-20T22:48:43.228764+0000","last_change":"2024-09-20T22:46:25.141094+0000","last_active":"2024-09-20T22:48:43.228764+0000","last_peered":"2024-09-20T22:48:43.228764+0000","last_clean":"2024-09-20T22:48:43.228764+0000","last_became_active":"2024-09-20T22:41:38.155953+0000","last_became_peered":"2024-09-20T22:41:38.155953+0000","last_unstale":"2024-09-20T22:48:43.228764+0000","last_undegraded":"2024-09-20T22:48:43.228764+0000","last_fullsized":"2024-09-20T22:48:43.228764+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:46:25.141028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:46:25.141028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":804,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-09-20T22:48:43.224189+0000","last_change":"2024-09-20T22:48:05.241119+0000","last_active":"2024-09-20T22:48:43.224189+0000","last_peered":"2024-09-20T22:48:43.224189+0000","last_clean":"2024-09-20T22:48:43.224189+0000","last_became_active":"2024-09-20T22:48:05.240613+0000","last_became_peered":"2024-09-20T22:48:05.240613+0000","last_unstale":"2024-09-20T22:48:43.224189+0000","last_undegraded":"2024-09-20T22:48:43.224189+0000","last_fullsized":"2024-09-20T22:48:43.224189+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:46:49.061850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:46:49.061850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":119,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-09-20T22:48:43.211015+0000","last_change":"2024-09-20T22:47:49.219776+0000","last_active":"2024-09-20T22:48:43.211015+0000","last_peered":"2024-09-20T22:48:43.211015+0000","last_clean":"2024-09-20T22:48:43.211015+0000","last_became_active":"2024-09-20T22:47:49.219486+0000","last_became_peered":"2024-09-20T22:47:49.219486+0000","last_unstale":"2024-09-20T22:48:43.211015+0000","last_undegraded":"2024-09-20T22:48:43.211015+0000","last_fullsized":"2024-09-20T22:48:43.211015+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:46:51.469567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:46:51.469567+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"443'354","reported_seq":2181,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-09-20T22:48:43.231518+0000","last_change":"2024-09-20T22:48:43.231764+0000","last_active":"2024-09-20T22:48:43.231518+0000","last_peered":"2024-09-20T22:48:43.231518+0000","last_clean":"2024-09-20T22:48:43.231518+0000","last_became_active":"2024-09-20T22:48:27.272202+0000","last_became_peered":"2024-09-20T22:48:27.272202+0000","last_unstale":"2024-09-20T22:48:43.231518+0000","last_undegraded":"2024-09-20T22:48:43.231518+0000","last_fullsized":"2024-09-20T22:48:43.231518+0000","mapping_epoch":407,"log_start":"423'280","ondisk_log_start":"423'280","created":94,"last_epoch_clean":427,"parent":"0.0","parent_split_bits":0,"last_scrub":"423'271","last_scrub_stamp":"2024-09-20T22:48:23.382092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:48:23.382092+0000","objects_scrubbed":8,"log_size":74,"log_dups_size":649,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:49:47.478415+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.020491059999999998,"stat_sum":{"num_bytes":42078859,"num_objects":17,"num_object_clones":12,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":1,"num_read":203,"num_read_kb":210482,"num_write":177,"num_write_kb":73049,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":65,"num_bytes_recovered":38710860,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"8d","length":"2"},{"start":"93","length":"2"},{"start":"9b","length":"1"},{"start":"9d","length":"1"},{"start":"9f","length":"2"},{"start":"a3","length":"2"},{"start":"a8","length":"1"},{"start":"aa","length":"4"}]},{"pgid":"4.a","version":"113'28","reported_seq":825,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-09-20T22:48:43.213263+0000","last_change":"2024-09-20T22:47:13.842452+0000","last_active":"2024-09-20T22:48:43.213263+0000","last_peered":"2024-09-20T22:48:43.213263+0000","last_clean":"2024-09-20T22:48:43.213263+0000","last_became_active":"2024-09-20T22:41:40.252011+0000","last_became_peered":"2024-09-20T22:41:40.252011+0000","last_unstale":"2024-09-20T22:48:43.213263+0000","last_undegraded":"2024-09-20T22:48:43.213263+0000","last_fullsized":"2024-09-20T22:48:43.213263+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'28","last_scrub_stamp":"2024-09-20T22:47:13.842316+0000","last_deep_scrub":"113'28","last_deep_scrub_stamp":"2024-09-20T22:43:12.725900+0000","last_clean_scrub_stamp":"2024-09-20T22:47:13.842316+0000","objects_scrubbed":9,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":780,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-09-20T22:48:43.217767+0000","last_change":"2024-09-20T22:46:30.430426+0000","last_active":"2024-09-20T22:48:43.217767+0000","last_peered":"2024-09-20T22:48:43.217767+0000","last_clean":"2024-09-20T22:48:43.217767+0000","last_became_active":"2024-09-20T22:41:38.161524+0000","last_became_peered":"2024-09-20T22:41:38.161524+0000","last_unstale":"2024-09-20T22:48:43.217767+0000","last_undegraded":"2024-09-20T22:48:43.217767+0000","last_fullsized":"2024-09-20T22:48:43.217767+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:46:30.430353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:46:30.430353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":792,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-09-20T22:48:43.224348+0000","last_change":"2024-09-20T22:47:49.223627+0000","last_active":"2024-09-20T22:48:43.224348+0000","last_peered":"2024-09-20T22:48:43.224348+0000","last_clean":"2024-09-20T22:48:43.224348+0000","last_became_active":"2024-09-20T22:47:49.223220+0000","last_became_peered":"2024-09-20T22:47:49.223220+0000","last_unstale":"2024-09-20T22:48:43.224348+0000","last_undegraded":"2024-09-20T22:48:43.224348+0000","last_fullsized":"2024-09-20T22:48:43.224348+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:47:36.012147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:47:36.012147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:45.981117+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":776,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-09-20T22:48:43.217714+0000","last_change":"2024-09-20T22:47:18.329286+0000","last_active":"2024-09-20T22:48:43.217714+0000","last_peered":"2024-09-20T22:48:43.217714+0000","last_clean":"2024-09-20T22:48:43.217714+0000","last_became_active":"2024-09-20T22:41:44.199395+0000","last_became_peered":"2024-09-20T22:41:44.199395+0000","last_unstale":"2024-09-20T22:48:43.217714+0000","last_undegraded":"2024-09-20T22:48:43.217714+0000","last_fullsized":"2024-09-20T22:48:43.217714+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:47:18.329209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:47:18.329209+0000","last_clean_scrub_stamp":"2024-09-20T22:47:18.329209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.a","version":"440'372","reported_seq":2085,"reported_epoch":443,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-20T22:48:43.220712+0000","last_change":"2024-09-20T22:48:42.254708+0000","last_active":"2024-09-20T22:48:43.220712+0000","last_peered":"2024-09-20T22:48:43.220712+0000","last_clean":"2024-09-20T22:48:43.220712+0000","last_became_active":"2024-09-20T22:42:52.405085+0000","last_became_peered":"2024-09-20T22:42:52.405085+0000","last_unstale":"2024-09-20T22:48:43.220712+0000","last_undegraded":"2024-09-20T22:48:43.220712+0000","last_fullsized":"2024-09-20T22:48:43.220712+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"301'219","last_scrub_stamp":"2024-09-20T22:46:23.260041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:46:23.260041+0000","objects_scrubbed":3,"log_size":372,"log_dups_size":0,"ondisk_log_size":372,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":13,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.032908554999999999,"stat_sum":{"num_bytes":24092698,"num_objects":11,"num_object_clones":8,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":1,"num_read":271,"num_read_kb":320467,"num_write":147,"num_write_kb":57710,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"8d","length":"2"}]},{"pgid":"4.9","version":"74'12","reported_seq":99,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-09-20T22:48:43.207413+0000","last_change":"2024-09-20T22:48:05.284523+0000","last_active":"2024-09-20T22:48:43.207413+0000","last_peered":"2024-09-20T22:48:43.207413+0000","last_clean":"2024-09-20T22:48:43.207413+0000","last_became_active":"2024-09-20T22:48:05.249438+0000","last_became_peered":"2024-09-20T22:48:05.249438+0000","last_unstale":"2024-09-20T22:48:43.207413+0000","last_undegraded":"2024-09-20T22:48:43.207413+0000","last_fullsized":"2024-09-20T22:48:43.207413+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-09-20T22:47:58.421473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:47:58.421473+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:49:19.311510+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":21,"num_read_kb":12,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":812,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-09-20T22:48:43.216775+0000","last_change":"2024-09-20T22:48:05.240596+0000","last_active":"2024-09-20T22:48:43.216775+0000","last_peered":"2024-09-20T22:48:43.216775+0000","last_clean":"2024-09-20T22:48:43.216775+0000","last_became_active":"2024-09-20T22:48:05.240225+0000","last_became_peered":"2024-09-20T22:48:05.240225+0000","last_unstale":"2024-09-20T22:48:43.216775+0000","last_undegraded":"2024-09-20T22:48:43.216775+0000","last_fullsized":"2024-09-20T22:48:43.216775+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:46:52.465940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:46:52.465940+0000","last_clean_scrub_stamp":"2024-09-20T22:46:52.465940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":800,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-09-20T22:48:43.225099+0000","last_change":"2024-09-20T22:48:22.197881+0000","last_active":"2024-09-20T22:48:43.225099+0000","last_peered":"2024-09-20T22:48:43.225099+0000","last_clean":"2024-09-20T22:48:43.225099+0000","last_became_active":"2024-09-20T22:47:49.222088+0000","last_became_peered":"2024-09-20T22:47:49.222088+0000","last_unstale":"2024-09-20T22:48:43.225099+0000","last_undegraded":"2024-09-20T22:48:43.225099+0000","last_fullsized":"2024-09-20T22:48:43.225099+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:48:22.197808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:46:01.828080+0000","last_clean_scrub_stamp":"2024-09-20T22:48:22.197808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:49:43.419742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":769,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-09-20T22:48:43.221556+0000","last_change":"2024-09-20T22:47:38.019602+0000","last_active":"2024-09-20T22:48:43.221556+0000","last_peered":"2024-09-20T22:48:43.221556+0000","last_clean":"2024-09-20T22:48:43.221556+0000","last_became_active":"2024-09-20T22:41:44.192609+0000","last_became_peered":"2024-09-20T22:41:44.192609+0000","last_unstale":"2024-09-20T22:48:43.221556+0000","last_undegraded":"2024-09-20T22:48:43.221556+0000","last_fullsized":"2024-09-20T22:48:43.221556+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:47:38.019533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:55.170379+0000","last_clean_scrub_stamp":"2024-09-20T22:47:38.019533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:49:06.773525+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"443'599","reported_seq":2317,"reported_epoch":443,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T22:48:43.336643+0000","last_change":"2024-09-20T22:48:43.336643+0000","last_active":"2024-09-20T22:48:43.336643+0000","last_peered":"2024-09-20T22:48:43.336643+0000","last_clean":"2024-09-20T22:48:43.336643+0000","last_became_active":"2024-09-20T22:42:52.409642+0000","last_became_peered":"2024-09-20T22:42:52.409642+0000","last_unstale":"2024-09-20T22:48:43.336643+0000","last_undegraded":"2024-09-20T22:48:43.336643+0000","last_fullsized":"2024-09-20T22:48:43.336643+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"194'116","last_scrub_stamp":"2024-09-20T22:44:37.097674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:44:37.097674+0000","objects_scrubbed":3,"log_size":599,"log_dups_size":0,"ondisk_log_size":599,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":12,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.126311855,"stat_sum":{"num_bytes":36654195,"num_objects":21,"num_object_clones":17,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":1,"num_read":353,"num_read_kb":383655,"num_write":252,"num_write_kb":80681,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"8d","length":"2"},{"start":"93","length":"1"}]},{"pgid":"4.8","version":"74'15","reported_seq":803,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-09-20T22:48:43.228848+0000","last_change":"2024-09-20T22:47:35.313505+0000","last_active":"2024-09-20T22:48:43.228848+0000","last_peered":"2024-09-20T22:48:43.228848+0000","last_clean":"2024-09-20T22:48:43.228848+0000","last_became_active":"2024-09-20T22:41:40.252113+0000","last_became_peered":"2024-09-20T22:41:40.252113+0000","last_unstale":"2024-09-20T22:48:43.228848+0000","last_undegraded":"2024-09-20T22:48:43.228848+0000","last_fullsized":"2024-09-20T22:48:43.228848+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-09-20T22:47:35.313413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:47:35.313413+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:52.070278+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'2","reported_seq":840,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-09-20T22:48:43.217785+0000","last_change":"2024-09-20T22:48:05.269145+0000","last_active":"2024-09-20T22:48:43.217785+0000","last_peered":"2024-09-20T22:48:43.217785+0000","last_clean":"2024-09-20T22:48:43.217785+0000","last_became_active":"2024-09-20T22:48:05.238098+0000","last_became_peered":"2024-09-20T22:48:05.238098+0000","last_unstale":"2024-09-20T22:48:43.217785+0000","last_undegraded":"2024-09-20T22:48:43.217785+0000","last_fullsized":"2024-09-20T22:48:43.217785+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-09-20T22:46:48.473494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:46:48.473494+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"408'10","reported_seq":894,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-09-20T22:48:43.216251+0000","last_change":"2024-09-20T22:48:05.265680+0000","last_active":"2024-09-20T22:48:43.216251+0000","last_peered":"2024-09-20T22:48:43.216251+0000","last_clean":"2024-09-20T22:48:43.216251+0000","last_became_active":"2024-09-20T22:48:05.239344+0000","last_became_peered":"2024-09-20T22:48:05.239344+0000","last_unstale":"2024-09-20T22:48:43.216251+0000","last_undegraded":"2024-09-20T22:48:43.216251+0000","last_fullsized":"2024-09-20T22:48:43.216251+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"159'9","last_scrub_stamp":"2024-09-20T22:47:37.382220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:47:37.382220+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":782,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-09-20T22:48:43.228456+0000","last_change":"2024-09-20T22:47:49.228372+0000","last_active":"2024-09-20T22:48:43.228456+0000","last_peered":"2024-09-20T22:48:43.228456+0000","last_clean":"2024-09-20T22:48:43.228456+0000","last_became_active":"2024-09-20T22:47:49.227569+0000","last_became_peered":"2024-09-20T22:47:49.227569+0000","last_unstale":"2024-09-20T22:48:43.228456+0000","last_undegraded":"2024-09-20T22:48:43.228456+0000","last_fullsized":"2024-09-20T22:48:43.228456+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:47:41.284992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:47:41.284992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:49:09.902863+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.14","version":"443'160","reported_seq":1666,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-09-20T22:48:43.215677+0000","last_change":"2024-09-20T22:48:43.218399+0000","last_active":"2024-09-20T22:48:43.215677+0000","last_peered":"2024-09-20T22:48:43.215677+0000","last_clean":"2024-09-20T22:48:43.215677+0000","last_became_active":"2024-09-20T22:48:22.211157+0000","last_became_peered":"2024-09-20T22:48:22.211157+0000","last_unstale":"2024-09-20T22:48:43.215677+0000","last_undegraded":"2024-09-20T22:48:43.215677+0000","last_fullsized":"2024-09-20T22:48:43.215677+0000","mapping_epoch":421,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":422,"parent":"0.0","parent_split_bits":5,"last_scrub":"395'116","last_scrub_stamp":"2024-09-20T22:48:03.152941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:48:03.152941+0000","objects_scrubbed":3,"log_size":160,"log_dups_size":0,"ondisk_log_size":160,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:49:15.660878+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0092583300000000004,"stat_sum":{"num_bytes":12074054,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":57,"num_read_kb":54699,"num_write":48,"num_write_kb":17575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"8d","length":"2"},{"start":"93","length":"2"},{"start":"9b","length":"1"},{"start":"9d","length":"1"},{"start":"9f","length":"2"},{"start":"a3","length":"2"},{"start":"a8","length":"1"},{"start":"aa","length":"4"}]},{"pgid":"3.10","version":"0'0","reported_seq":801,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-09-20T22:48:43.212963+0000","last_change":"2024-09-20T22:47:49.213597+0000","last_active":"2024-09-20T22:48:43.212963+0000","last_peered":"2024-09-20T22:48:43.212963+0000","last_clean":"2024-09-20T22:48:43.212963+0000","last_became_active":"2024-09-20T22:47:49.213292+0000","last_became_peered":"2024-09-20T22:47:49.213292+0000","last_unstale":"2024-09-20T22:48:43.212963+0000","last_undegraded":"2024-09-20T22:48:43.212963+0000","last_fullsized":"2024-09-20T22:48:43.212963+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:47:12.820304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:47:12.820304+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"74'6","reported_seq":799,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-09-20T22:48:43.220611+0000","last_change":"2024-09-20T22:45:39.079425+0000","last_active":"2024-09-20T22:48:43.220611+0000","last_peered":"2024-09-20T22:48:43.220611+0000","last_clean":"2024-09-20T22:48:43.220611+0000","last_became_active":"2024-09-20T22:43:09.116887+0000","last_became_peered":"2024-09-20T22:43:09.116887+0000","last_unstale":"2024-09-20T22:48:43.220611+0000","last_undegraded":"2024-09-20T22:48:43.220611+0000","last_fullsized":"2024-09-20T22:48:43.220611+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'6","last_scrub_stamp":"2024-09-20T22:45:39.079346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:45:39.079346+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":781,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-09-20T22:48:43.229456+0000","last_change":"2024-09-20T22:48:35.550867+0000","last_active":"2024-09-20T22:48:43.229456+0000","last_peered":"2024-09-20T22:48:43.229456+0000","last_clean":"2024-09-20T22:48:43.229456+0000","last_became_active":"2024-09-20T22:41:42.184930+0000","last_became_peered":"2024-09-20T22:41:42.184930+0000","last_unstale":"2024-09-20T22:48:43.229456+0000","last_undegraded":"2024-09-20T22:48:43.229456+0000","last_fullsized":"2024-09-20T22:48:43.229456+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:48:35.550781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:09.705632+0000","last_clean_scrub_stamp":"2024-09-20T22:48:35.550781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:05.315763+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"74'1","reported_seq":796,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-09-20T22:48:43.217999+0000","last_change":"2024-09-20T22:48:40.309005+0000","last_active":"2024-09-20T22:48:43.217999+0000","last_peered":"2024-09-20T22:48:43.217999+0000","last_clean":"2024-09-20T22:48:43.217999+0000","last_became_active":"2024-09-20T22:48:05.237882+0000","last_became_peered":"2024-09-20T22:48:05.237882+0000","last_unstale":"2024-09-20T22:48:43.217999+0000","last_undegraded":"2024-09-20T22:48:43.217999+0000","last_fullsized":"2024-09-20T22:48:43.217999+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-09-20T22:48:40.308879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:48:40.308879+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:07.308273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.15","version":"98'3","reported_seq":1738,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-09-20T22:48:43.216164+0000","last_change":"2024-09-20T22:48:43.216752+0000","last_active":"2024-09-20T22:48:43.216164+0000","last_peered":"2024-09-20T22:48:43.216164+0000","last_clean":"2024-09-20T22:48:43.216164+0000","last_became_active":"2024-09-20T22:48:19.210470+0000","last_became_peered":"2024-09-20T22:48:19.210470+0000","last_unstale":"2024-09-20T22:48:43.216164+0000","last_undegraded":"2024-09-20T22:48:43.216164+0000","last_fullsized":"2024-09-20T22:48:43.216164+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":5,"last_scrub":"98'3","last_scrub_stamp":"2024-09-20T22:47:35.357009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:47:35.357009+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:51.429591+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00087314700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"8d","length":"2"},{"start":"93","length":"2"},{"start":"9b","length":"1"},{"start":"9d","length":"1"},{"start":"9f","length":"2"},{"start":"a3","length":"2"},{"start":"a8","length":"1"},{"start":"aa","length":"4"}]},{"pgid":"4.16","version":"74'9","reported_seq":129,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-09-20T22:48:43.212720+0000","last_change":"2024-09-20T22:47:49.262043+0000","last_active":"2024-09-20T22:48:43.212720+0000","last_peered":"2024-09-20T22:48:43.212720+0000","last_clean":"2024-09-20T22:48:43.212720+0000","last_became_active":"2024-09-20T22:47:49.231315+0000","last_became_peered":"2024-09-20T22:47:49.231315+0000","last_unstale":"2024-09-20T22:48:43.212720+0000","last_undegraded":"2024-09-20T22:48:43.212720+0000","last_fullsized":"2024-09-20T22:48:43.212720+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-09-20T22:46:21.269487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:46:21.269487+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":804,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-09-20T22:48:43.215908+0000","last_change":"2024-09-20T22:48:05.239747+0000","last_active":"2024-09-20T22:48:43.215908+0000","last_peered":"2024-09-20T22:48:43.215908+0000","last_clean":"2024-09-20T22:48:43.215908+0000","last_became_active":"2024-09-20T22:48:05.238801+0000","last_became_peered":"2024-09-20T22:48:05.238801+0000","last_unstale":"2024-09-20T22:48:43.215908+0000","last_undegraded":"2024-09-20T22:48:43.215908+0000","last_fullsized":"2024-09-20T22:48:43.215908+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:46:46.438556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:46:46.438556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":772,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-09-20T22:48:43.220368+0000","last_change":"2024-09-20T22:46:48.012442+0000","last_active":"2024-09-20T22:48:43.220368+0000","last_peered":"2024-09-20T22:48:43.220368+0000","last_clean":"2024-09-20T22:48:43.220368+0000","last_became_active":"2024-09-20T22:41:42.187706+0000","last_became_peered":"2024-09-20T22:41:42.187706+0000","last_unstale":"2024-09-20T22:48:43.220368+0000","last_undegraded":"2024-09-20T22:48:43.220368+0000","last_fullsized":"2024-09-20T22:48:43.220368+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:46:48.012372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:46:48.012372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":807,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-09-20T22:48:43.224564+0000","last_change":"2024-09-20T22:48:40.247992+0000","last_active":"2024-09-20T22:48:43.224564+0000","last_peered":"2024-09-20T22:48:43.224564+0000","last_clean":"2024-09-20T22:48:43.224564+0000","last_became_active":"2024-09-20T22:48:05.239710+0000","last_became_peered":"2024-09-20T22:48:05.239710+0000","last_unstale":"2024-09-20T22:48:43.224564+0000","last_undegraded":"2024-09-20T22:48:43.224564+0000","last_fullsized":"2024-09-20T22:48:43.224564+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:48:40.247889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:48:40.247889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:09.303289+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.16","version":"435'154","reported_seq":1820,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-09-20T22:48:43.219337+0000","last_change":"2024-09-20T22:48:43.230227+0000","last_active":"2024-09-20T22:48:43.219337+0000","last_peered":"2024-09-20T22:48:43.219337+0000","last_clean":"2024-09-20T22:48:43.219337+0000","last_became_active":"2024-09-20T22:48:22.214743+0000","last_became_peered":"2024-09-20T22:48:22.214743+0000","last_unstale":"2024-09-20T22:48:43.219337+0000","last_undegraded":"2024-09-20T22:48:43.219337+0000","last_fullsized":"2024-09-20T22:48:43.219337+0000","mapping_epoch":421,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":422,"parent":"0.0","parent_split_bits":5,"last_scrub":"282'91","last_scrub_stamp":"2024-09-20T22:46:05.393044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:46:05.393044+0000","objects_scrubbed":1,"log_size":154,"log_dups_size":0,"ondisk_log_size":154,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.017916616999999999,"stat_sum":{"num_bytes":5647350,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":69,"num_read_kb":74959,"num_write":33,"num_write_kb":11905,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":13401642,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"8d","length":"2"},{"start":"93","length":"2"},{"start":"9b","length":"1"},{"start":"9d","length":"1"},{"start":"9f","length":"2"},{"start":"a3","length":"2"},{"start":"a8","length":"1"},{"start":"aa","length":"4"}]},{"pgid":"4.15","version":"74'9","reported_seq":793,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-09-20T22:48:43.229173+0000","last_change":"2024-09-20T22:46:27.123331+0000","last_active":"2024-09-20T22:48:43.229173+0000","last_peered":"2024-09-20T22:48:43.229173+0000","last_clean":"2024-09-20T22:48:43.229173+0000","last_became_active":"2024-09-20T22:41:40.252486+0000","last_became_peered":"2024-09-20T22:41:40.252486+0000","last_unstale":"2024-09-20T22:48:43.229173+0000","last_undegraded":"2024-09-20T22:48:43.229173+0000","last_fullsized":"2024-09-20T22:48:43.229173+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-09-20T22:46:27.123248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:46:27.123248+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":127,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-09-20T22:48:43.209159+0000","last_change":"2024-09-20T22:48:19.379265+0000","last_active":"2024-09-20T22:48:43.209159+0000","last_peered":"2024-09-20T22:48:43.209159+0000","last_clean":"2024-09-20T22:48:43.209159+0000","last_became_active":"2024-09-20T22:47:49.229936+0000","last_became_peered":"2024-09-20T22:47:49.229936+0000","last_unstale":"2024-09-20T22:48:43.209159+0000","last_undegraded":"2024-09-20T22:48:43.209159+0000","last_fullsized":"2024-09-20T22:48:43.209159+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:48:19.379196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:48:19.379196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:49:35.159658+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"74'8","reported_seq":867,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-09-20T22:48:43.280443+0000","last_change":"2024-09-20T22:47:56.150641+0000","last_active":"2024-09-20T22:48:43.280443+0000","last_peered":"2024-09-20T22:48:43.280443+0000","last_clean":"2024-09-20T22:48:43.280443+0000","last_became_active":"2024-09-20T22:41:42.186352+0000","last_became_peered":"2024-09-20T22:41:42.186352+0000","last_unstale":"2024-09-20T22:48:43.280443+0000","last_undegraded":"2024-09-20T22:48:43.280443+0000","last_fullsized":"2024-09-20T22:48:43.280443+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-09-20T22:47:56.150562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:47:56.150562+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:49:03.909124+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":87,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-09-20T22:48:43.208636+0000","last_change":"2024-09-20T22:48:05.250192+0000","last_active":"2024-09-20T22:48:43.208636+0000","last_peered":"2024-09-20T22:48:43.208636+0000","last_clean":"2024-09-20T22:48:43.208636+0000","last_became_active":"2024-09-20T22:48:05.249656+0000","last_became_peered":"2024-09-20T22:48:05.249656+0000","last_unstale":"2024-09-20T22:48:43.208636+0000","last_undegraded":"2024-09-20T22:48:43.208636+0000","last_fullsized":"2024-09-20T22:48:43.208636+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:47:19.828541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:47:19.828541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"74'10","reported_seq":792,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-09-20T22:48:43.219425+0000","last_change":"2024-09-20T22:46:24.269736+0000","last_active":"2024-09-20T22:48:43.219425+0000","last_peered":"2024-09-20T22:48:43.219425+0000","last_clean":"2024-09-20T22:48:43.219425+0000","last_became_active":"2024-09-20T22:41:40.253614+0000","last_became_peered":"2024-09-20T22:41:40.253614+0000","last_unstale":"2024-09-20T22:48:43.219425+0000","last_undegraded":"2024-09-20T22:48:43.219425+0000","last_fullsized":"2024-09-20T22:48:43.219425+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-09-20T22:46:24.269655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:46:24.269655+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":800,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-09-20T22:48:43.216719+0000","last_change":"2024-09-20T22:48:05.240585+0000","last_active":"2024-09-20T22:48:43.216719+0000","last_peered":"2024-09-20T22:48:43.216719+0000","last_clean":"2024-09-20T22:48:43.216719+0000","last_became_active":"2024-09-20T22:48:05.240231+0000","last_became_peered":"2024-09-20T22:48:05.240231+0000","last_unstale":"2024-09-20T22:48:43.216719+0000","last_undegraded":"2024-09-20T22:48:43.216719+0000","last_fullsized":"2024-09-20T22:48:43.216719+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:48:01.305632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:48:01.305632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:49:16.408958+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"389'10","reported_seq":898,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-09-20T22:48:43.229091+0000","last_change":"2024-09-20T22:47:58.443470+0000","last_active":"2024-09-20T22:48:43.229091+0000","last_peered":"2024-09-20T22:48:43.229091+0000","last_clean":"2024-09-20T22:48:43.229091+0000","last_became_active":"2024-09-20T22:47:49.227113+0000","last_became_peered":"2024-09-20T22:47:49.227113+0000","last_unstale":"2024-09-20T22:48:43.229091+0000","last_undegraded":"2024-09-20T22:48:43.229091+0000","last_fullsized":"2024-09-20T22:48:43.229091+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'10","last_scrub_stamp":"2024-09-20T22:47:58.443393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:47:58.443393+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:49:10.836242+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":119,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-09-20T22:48:43.212410+0000","last_change":"2024-09-20T22:47:49.232560+0000","last_active":"2024-09-20T22:48:43.212410+0000","last_peered":"2024-09-20T22:48:43.212410+0000","last_clean":"2024-09-20T22:48:43.212410+0000","last_became_active":"2024-09-20T22:47:49.230674+0000","last_became_peered":"2024-09-20T22:47:49.230674+0000","last_unstale":"2024-09-20T22:48:43.212410+0000","last_undegraded":"2024-09-20T22:48:43.212410+0000","last_fullsized":"2024-09-20T22:48:43.212410+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:47:41.411716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:47:41.411716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:54.101804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.10","version":"443'150","reported_seq":1837,"reported_epoch":443,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T22:48:43.262855+0000","last_change":"2024-09-20T22:48:43.262855+0000","last_active":"2024-09-20T22:48:43.262855+0000","last_peered":"2024-09-20T22:48:43.262855+0000","last_clean":"2024-09-20T22:48:43.262855+0000","last_became_active":"2024-09-20T22:48:22.213631+0000","last_became_peered":"2024-09-20T22:48:22.213631+0000","last_unstale":"2024-09-20T22:48:43.262855+0000","last_undegraded":"2024-09-20T22:48:43.262855+0000","last_fullsized":"2024-09-20T22:48:43.262855+0000","mapping_epoch":421,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":422,"parent":"0.0","parent_split_bits":5,"last_scrub":"146'43","last_scrub_stamp":"2024-09-20T22:43:46.106059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:43:46.106059+0000","objects_scrubbed":0,"log_size":150,"log_dups_size":0,"ondisk_log_size":150,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":9,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.032513029999999998,"stat_sum":{"num_bytes":8863761,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":58,"num_read_kb":77675,"num_write":37,"num_write_kb":16548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":7603720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"8d","length":"2"},{"start":"93","length":"2"},{"start":"9b","length":"1"},{"start":"9d","length":"1"}]},{"pgid":"4.13","version":"74'11","reported_seq":99,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-09-20T22:48:43.207610+0000","last_change":"2024-09-20T22:48:05.307593+0000","last_active":"2024-09-20T22:48:43.207610+0000","last_peered":"2024-09-20T22:48:43.207610+0000","last_clean":"2024-09-20T22:48:43.207610+0000","last_became_active":"2024-09-20T22:48:05.249510+0000","last_became_peered":"2024-09-20T22:48:05.249510+0000","last_unstale":"2024-09-20T22:48:43.207610+0000","last_undegraded":"2024-09-20T22:48:43.207610+0000","last_fullsized":"2024-09-20T22:48:43.207610+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-09-20T22:45:37.940753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:45:37.940753+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":87,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-09-20T22:48:43.207848+0000","last_change":"2024-09-20T22:48:05.252646+0000","last_active":"2024-09-20T22:48:43.207848+0000","last_peered":"2024-09-20T22:48:43.207848+0000","last_clean":"2024-09-20T22:48:43.207848+0000","last_became_active":"2024-09-20T22:48:05.252026+0000","last_became_peered":"2024-09-20T22:48:05.252026+0000","last_unstale":"2024-09-20T22:48:43.207848+0000","last_undegraded":"2024-09-20T22:48:43.207848+0000","last_fullsized":"2024-09-20T22:48:43.207848+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:46:04.391306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:46:04.391306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":772,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-09-20T22:48:43.209815+0000","last_change":"2024-09-20T22:46:25.881993+0000","last_active":"2024-09-20T22:48:43.209815+0000","last_peered":"2024-09-20T22:48:43.209815+0000","last_clean":"2024-09-20T22:48:43.209815+0000","last_became_active":"2024-09-20T22:41:42.181736+0000","last_became_peered":"2024-09-20T22:41:42.181736+0000","last_unstale":"2024-09-20T22:48:43.209815+0000","last_undegraded":"2024-09-20T22:48:43.209815+0000","last_fullsized":"2024-09-20T22:48:43.209815+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:46:25.881921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:46:25.881921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":780,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-09-20T22:48:43.218862+0000","last_change":"2024-09-20T22:47:49.216479+0000","last_active":"2024-09-20T22:48:43.218862+0000","last_peered":"2024-09-20T22:48:43.218862+0000","last_clean":"2024-09-20T22:48:43.218862+0000","last_became_active":"2024-09-20T22:47:49.216141+0000","last_became_peered":"2024-09-20T22:47:49.216141+0000","last_unstale":"2024-09-20T22:48:43.218862+0000","last_undegraded":"2024-09-20T22:48:43.218862+0000","last_fullsized":"2024-09-20T22:48:43.218862+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:47:36.066480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:47:36.066480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:48.477867+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"135'39","reported_seq":150,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-09-20T22:48:43.207370+0000","last_change":"2024-09-20T22:48:43.208177+0000","last_active":"2024-09-20T22:48:43.207370+0000","last_peered":"2024-09-20T22:48:43.207370+0000","last_clean":"2024-09-20T22:48:43.207370+0000","last_became_active":"2024-09-20T22:48:19.212541+0000","last_became_peered":"2024-09-20T22:48:19.212541+0000","last_unstale":"2024-09-20T22:48:43.207370+0000","last_undegraded":"2024-09-20T22:48:43.207370+0000","last_fullsized":"2024-09-20T22:48:43.207370+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":5,"last_scrub":"135'39","last_scrub_stamp":"2024-09-20T22:48:19.969753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:48:19.969753+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:49:49.199414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011956670000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"8d","length":"2"},{"start":"93","length":"2"},{"start":"9b","length":"1"},{"start":"9d","length":"1"},{"start":"9f","length":"2"},{"start":"a3","length":"2"},{"start":"a8","length":"1"},{"start":"aa","length":"4"}]},{"pgid":"4.12","version":"404'13","reported_seq":805,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-09-20T22:48:43.210160+0000","last_change":"2024-09-20T22:48:04.276356+0000","last_active":"2024-09-20T22:48:43.210160+0000","last_peered":"2024-09-20T22:48:43.210160+0000","last_clean":"2024-09-20T22:48:43.210160+0000","last_became_active":"2024-09-20T22:41:40.164796+0000","last_became_peered":"2024-09-20T22:41:40.164796+0000","last_unstale":"2024-09-20T22:48:43.210160+0000","last_undegraded":"2024-09-20T22:48:43.210160+0000","last_fullsized":"2024-09-20T22:48:43.210160+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"404'13","last_scrub_stamp":"2024-09-20T22:48:04.276282+0000","last_deep_scrub":"404'13","last_deep_scrub_stamp":"2024-09-20T22:48:04.276282+0000","last_clean_scrub_stamp":"2024-09-20T22:48:04.276282+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:49:13.107971+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":814,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-09-20T22:48:43.217605+0000","last_change":"2024-09-20T22:48:24.422893+0000","last_active":"2024-09-20T22:48:43.217605+0000","last_peered":"2024-09-20T22:48:43.217605+0000","last_clean":"2024-09-20T22:48:43.217605+0000","last_became_active":"2024-09-20T22:48:05.238812+0000","last_became_peered":"2024-09-20T22:48:05.238812+0000","last_unstale":"2024-09-20T22:48:43.217605+0000","last_undegraded":"2024-09-20T22:48:43.217605+0000","last_fullsized":"2024-09-20T22:48:43.217605+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:48:24.422757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:31.138067+0000","last_clean_scrub_stamp":"2024-09-20T22:48:24.422757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:49:32.075381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":785,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-09-20T22:48:43.221624+0000","last_change":"2024-09-20T22:48:38.200896+0000","last_active":"2024-09-20T22:48:43.221624+0000","last_peered":"2024-09-20T22:48:43.221624+0000","last_clean":"2024-09-20T22:48:43.221624+0000","last_became_active":"2024-09-20T22:47:49.218708+0000","last_became_peered":"2024-09-20T22:47:49.218708+0000","last_unstale":"2024-09-20T22:48:43.221624+0000","last_undegraded":"2024-09-20T22:48:43.221624+0000","last_fullsized":"2024-09-20T22:48:43.221624+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:48:38.200797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:48:38.200797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:49:51.213367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":119,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-09-20T22:48:43.210460+0000","last_change":"2024-09-20T22:47:49.225362+0000","last_active":"2024-09-20T22:48:43.210460+0000","last_peered":"2024-09-20T22:48:43.210460+0000","last_clean":"2024-09-20T22:48:43.210460+0000","last_became_active":"2024-09-20T22:47:49.225027+0000","last_became_peered":"2024-09-20T22:47:49.225027+0000","last_unstale":"2024-09-20T22:48:43.210460+0000","last_undegraded":"2024-09-20T22:48:43.210460+0000","last_fullsized":"2024-09-20T22:48:43.210460+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:47:37.309842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:47:37.309842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:49:05.473839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.12","version":"442'428","reported_seq":2034,"reported_epoch":443,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T22:48:43.220380+0000","last_change":"2024-09-20T22:48:42.255189+0000","last_active":"2024-09-20T22:48:43.220380+0000","last_peered":"2024-09-20T22:48:43.220380+0000","last_clean":"2024-09-20T22:48:43.220380+0000","last_became_active":"2024-09-20T22:48:22.216595+0000","last_became_peered":"2024-09-20T22:48:22.216595+0000","last_unstale":"2024-09-20T22:48:43.220380+0000","last_undegraded":"2024-09-20T22:48:43.220380+0000","last_fullsized":"2024-09-20T22:48:43.220380+0000","mapping_epoch":421,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":422,"parent":"0.0","parent_split_bits":5,"last_scrub":"377'359","last_scrub_stamp":"2024-09-20T22:47:41.855224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:47:41.855224+0000","objects_scrubbed":9,"log_size":428,"log_dups_size":0,"ondisk_log_size":428,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":9,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:56.865108+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.027864885999999998,"stat_sum":{"num_bytes":20487666,"num_objects":8,"num_object_clones":5,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":160,"num_read_kb":146174,"num_write":122,"num_write_kb":39984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":107,"num_bytes_recovered":44451977,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"8d","length":"2"},{"start":"93","length":"1"},{"start":"a3","length":"2"},{"start":"a8","length":"1"}]},{"pgid":"4.11","version":"74'11","reported_seq":586,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-09-20T22:48:43.227845+0000","last_change":"2024-09-20T22:46:31.254470+0000","last_active":"2024-09-20T22:48:43.227845+0000","last_peered":"2024-09-20T22:48:43.227845+0000","last_clean":"2024-09-20T22:48:43.227845+0000","last_became_active":"2024-09-20T22:44:09.170785+0000","last_became_peered":"2024-09-20T22:44:09.170785+0000","last_unstale":"2024-09-20T22:48:43.227845+0000","last_undegraded":"2024-09-20T22:48:43.227845+0000","last_fullsized":"2024-09-20T22:48:43.227845+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-09-20T22:46:31.254376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:46:31.254376+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":789,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-09-20T22:48:43.229550+0000","last_change":"2024-09-20T22:48:02.512192+0000","last_active":"2024-09-20T22:48:43.229550+0000","last_peered":"2024-09-20T22:48:43.229550+0000","last_clean":"2024-09-20T22:48:43.229550+0000","last_became_active":"2024-09-20T22:41:38.158701+0000","last_became_peered":"2024-09-20T22:41:38.158701+0000","last_unstale":"2024-09-20T22:48:43.229550+0000","last_undegraded":"2024-09-20T22:48:43.229550+0000","last_fullsized":"2024-09-20T22:48:43.229550+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:48:02.512093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:45:24.765414+0000","last_clean_scrub_stamp":"2024-09-20T22:48:02.512093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:49:28.678353+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":796,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-09-20T22:48:43.216987+0000","last_change":"2024-09-20T22:48:05.238692+0000","last_active":"2024-09-20T22:48:43.216987+0000","last_peered":"2024-09-20T22:48:43.216987+0000","last_clean":"2024-09-20T22:48:43.216987+0000","last_became_active":"2024-09-20T22:48:05.238334+0000","last_became_peered":"2024-09-20T22:48:05.238334+0000","last_unstale":"2024-09-20T22:48:43.216987+0000","last_undegraded":"2024-09-20T22:48:43.216987+0000","last_fullsized":"2024-09-20T22:48:43.216987+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:46:29.466361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:50.174861+0000","last_clean_scrub_stamp":"2024-09-20T22:46:29.466361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":772,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-09-20T22:48:43.221741+0000","last_change":"2024-09-20T22:47:49.215237+0000","last_active":"2024-09-20T22:48:43.221741+0000","last_peered":"2024-09-20T22:48:43.221741+0000","last_clean":"2024-09-20T22:48:43.221741+0000","last_became_active":"2024-09-20T22:47:49.214487+0000","last_became_peered":"2024-09-20T22:47:49.214487+0000","last_unstale":"2024-09-20T22:48:43.221741+0000","last_undegraded":"2024-09-20T22:48:43.221741+0000","last_fullsized":"2024-09-20T22:48:43.221741+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:45:00.006909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:45:00.006909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"442'115","reported_seq":313,"reported_epoch":443,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T22:48:43.220145+0000","last_change":"2024-09-20T22:48:42.249421+0000","last_active":"2024-09-20T22:48:43.220145+0000","last_peered":"2024-09-20T22:48:43.220145+0000","last_clean":"2024-09-20T22:48:43.220145+0000","last_became_active":"2024-09-20T22:48:22.213639+0000","last_became_peered":"2024-09-20T22:48:22.213639+0000","last_unstale":"2024-09-20T22:48:43.220145+0000","last_undegraded":"2024-09-20T22:48:43.220145+0000","last_fullsized":"2024-09-20T22:48:43.220145+0000","mapping_epoch":421,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":422,"parent":"0.0","parent_split_bits":5,"last_scrub":"255'21","last_scrub_stamp":"2024-09-20T22:45:35.767459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:45:35.767459+0000","objects_scrubbed":1,"log_size":115,"log_dups_size":0,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":8,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.035917817999999997,"stat_sum":{"num_bytes":9207931,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":63,"num_read_kb":85981,"num_write":25,"num_write_kb":11537,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":20196357,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"8d","length":"2"},{"start":"93","length":"2"},{"start":"a3","length":"2"},{"start":"a8","length":"1"}]},{"pgid":"4.10","version":"74'4","reported_seq":783,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-09-20T22:48:43.218962+0000","last_change":"2024-09-20T22:48:23.236306+0000","last_active":"2024-09-20T22:48:43.218962+0000","last_peered":"2024-09-20T22:48:43.218962+0000","last_clean":"2024-09-20T22:48:43.218962+0000","last_became_active":"2024-09-20T22:41:40.166123+0000","last_became_peered":"2024-09-20T22:41:40.166123+0000","last_unstale":"2024-09-20T22:48:43.218962+0000","last_undegraded":"2024-09-20T22:48:43.218962+0000","last_fullsized":"2024-09-20T22:48:43.218962+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'4","last_scrub_stamp":"2024-09-20T22:48:23.236227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:48:23.236227+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:49:28.615652+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":127,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-09-20T22:48:43.212134+0000","last_change":"2024-09-20T22:48:18.432371+0000","last_active":"2024-09-20T22:48:43.212134+0000","last_peered":"2024-09-20T22:48:43.212134+0000","last_clean":"2024-09-20T22:48:43.212134+0000","last_became_active":"2024-09-20T22:47:49.231964+0000","last_became_peered":"2024-09-20T22:47:49.231964+0000","last_unstale":"2024-09-20T22:48:43.212134+0000","last_undegraded":"2024-09-20T22:48:43.212134+0000","last_fullsized":"2024-09-20T22:48:43.212134+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:48:18.432271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:48:18.432271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:49:21.454351+0000","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":805,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-09-20T22:48:43.212967+0000","last_change":"2024-09-20T22:48:05.245017+0000","last_active":"2024-09-20T22:48:43.212967+0000","last_peered":"2024-09-20T22:48:43.212967+0000","last_clean":"2024-09-20T22:48:43.212967+0000","last_became_active":"2024-09-20T22:48:05.244659+0000","last_became_peered":"2024-09-20T22:48:05.244659+0000","last_unstale":"2024-09-20T22:48:43.212967+0000","last_undegraded":"2024-09-20T22:48:43.212967+0000","last_fullsized":"2024-09-20T22:48:43.212967+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:47:37.826041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:47:37.826041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:53.997023+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":787,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-09-20T22:48:43.216107+0000","last_change":"2024-09-20T22:48:19.379992+0000","last_active":"2024-09-20T22:48:43.216107+0000","last_peered":"2024-09-20T22:48:43.216107+0000","last_clean":"2024-09-20T22:48:43.216107+0000","last_became_active":"2024-09-20T22:48:05.237765+0000","last_became_peered":"2024-09-20T22:48:05.237765+0000","last_unstale":"2024-09-20T22:48:43.216107+0000","last_undegraded":"2024-09-20T22:48:43.216107+0000","last_fullsized":"2024-09-20T22:48:43.216107+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:48:19.379926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:10.233487+0000","last_clean_scrub_stamp":"2024-09-20T22:48:19.379926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:49:32.710554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":780,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-09-20T22:48:43.210624+0000","last_change":"2024-09-20T22:48:05.249807+0000","last_active":"2024-09-20T22:48:43.210624+0000","last_peered":"2024-09-20T22:48:43.210624+0000","last_clean":"2024-09-20T22:48:43.210624+0000","last_became_active":"2024-09-20T22:48:05.249492+0000","last_became_peered":"2024-09-20T22:48:05.249492+0000","last_unstale":"2024-09-20T22:48:43.210624+0000","last_undegraded":"2024-09-20T22:48:43.210624+0000","last_fullsized":"2024-09-20T22:48:43.210624+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:47:19.596232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:47:19.596232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":806,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-09-20T22:48:43.218721+0000","last_change":"2024-09-20T22:48:36.246135+0000","last_active":"2024-09-20T22:48:43.218721+0000","last_peered":"2024-09-20T22:48:43.218721+0000","last_clean":"2024-09-20T22:48:43.218721+0000","last_became_active":"2024-09-20T22:47:49.217064+0000","last_became_peered":"2024-09-20T22:47:49.217064+0000","last_unstale":"2024-09-20T22:48:43.218721+0000","last_undegraded":"2024-09-20T22:48:43.218721+0000","last_fullsized":"2024-09-20T22:48:43.218721+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:48:36.246069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:48:36.246069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:49:36.464648+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"74'11","reported_seq":807,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-09-20T22:48:43.214378+0000","last_change":"2024-09-20T22:47:38.862277+0000","last_active":"2024-09-20T22:48:43.214378+0000","last_peered":"2024-09-20T22:48:43.214378+0000","last_clean":"2024-09-20T22:48:43.214378+0000","last_became_active":"2024-09-20T22:41:40.171256+0000","last_became_peered":"2024-09-20T22:41:40.171256+0000","last_unstale":"2024-09-20T22:48:43.214378+0000","last_undegraded":"2024-09-20T22:48:43.214378+0000","last_fullsized":"2024-09-20T22:48:43.214378+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-09-20T22:47:38.862194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:47:38.862194+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:49:03.255502+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":119,"reported_epoch":443,"state":"active+clean","last_fresh":"2024-09-20T22:48:43.212517+0000","last_change":"2024-09-20T22:47:49.221413+0000","last_active":"2024-09-20T22:48:43.212517+0000","last_peered":"2024-09-20T22:48:43.212517+0000","last_clean":"2024-09-20T22:48:43.212517+0000","last_became_active":"2024-09-20T22:47:49.220971+0000","last_became_peered":"2024-09-20T22:47:49.220971+0000","last_unstale":"2024-09-20T22:48:43.212517+0000","last_undegraded":"2024-09-20T22:48:43.212517+0000","last_fullsized":"2024-09-20T22:48:43.212517+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:46:20.485453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:51.667341+0000","last_clean_scrub_stamp":"2024-09-20T22:46:20.485453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":23,"stat_sum":{"num_bytes":375720048,"num_objects":176,"num_object_clones":127,"num_object_copies":528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":176,"num_whiteouts":16,"num_read":3013,"num_read_kb":3160066,"num_write":2089,"num_write_kb":785264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":600,"num_bytes_recovered":297592144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":842592256,"data_stored":1173045999,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5328,"ondisk_log_size":5328,"up":69,"acting":69,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":82,"ondisk_log_size":82,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":682,"num_read_kb":439,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":238,"num_bytes_recovered":5126,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":434,"ondisk_log_size":434,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":1034,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":416,"num_read_kb":411,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":384,"ondisk_log_size":384,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070553,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":136216,"kb_used_data":82820,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93608936,"statfs":{"total":95995035648,"available":95855550464,"internally_reserved":0,"allocated":84807680,"data_stored":108839862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":54655434},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430303,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":166420,"kb_used_data":113020,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93578732,"statfs":{"total":95995035648,"available":95824621568,"internally_reserved":0,"allocated":115732480,"data_stored":167970411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691940,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":113160,"kb_used_data":81972,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93631992,"statfs":{"total":95995035648,"available":95879159808,"internally_reserved":0,"allocated":83939328,"data_stored":132003253,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986281,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":117932,"kb_used_data":82580,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93627220,"statfs":{"total":95995035648,"available":95874273280,"internally_reserved":0,"allocated":84561920,"data_stored":107819829,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182511,"num_pgs":74,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":203160,"kb_used_data":149764,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93541992,"statfs":{"total":95995035648,"available":95786999808,"internally_reserved":0,"allocated":153358336,"data_stored":202064067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411444,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":118232,"kb_used_data":87040,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93626920,"statfs":{"total":95995035648,"available":95873966080,"internally_reserved":0,"allocated":89128960,"data_stored":116689638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574969,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":148216,"kb_used_data":117024,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93596936,"statfs":{"total":95995035648,"available":95843262464,"internally_reserved":0,"allocated":119832576,"data_stored":174039372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738494,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":183736,"kb_used_data":134496,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":93561416,"statfs":{"total":95995035648,"available":95806889984,"internally_reserved":0,"allocated":137723904,"data_stored":178657211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":134512640,"data_stored":176720842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":116604928,"data_stored":172110240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":85671936,"data_stored":114491007,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":150482944,"data_stored":201746207,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":81358848,"data_stored":105891644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":80146432,"data_stored":129475848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":112541696,"data_stored":166034266,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":81272832,"data_stored":106575945,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-20T22:48:45.533 INFO:tasks.thrashosds.thrasher:Setting 6.1e to [0, 1, 3] 2024-09-20T22:48:45.533 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '6.1e', '0', '1', '3'] 2024-09-20T22:48:45.533 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd pg-upmap 6.1e 0 1 3 2024-09-20T22:48:45.605 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:48:45.628 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:45.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:45.629 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:48:45.729 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:48:45.751 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:45.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:45.752 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:48:45.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:45 smithi154 ceph-mon[33264]: pgmap v567: 165 pgs: 3 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 157 active+clean; 359 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 4.4 MiB/s rd, 716 KiB/s wr, 11 op/s 2024-09-20T22:48:45.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:45 smithi154 ceph-mon[33264]: 2.0 scrub starts 2024-09-20T22:48:45.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:45 smithi154 ceph-mon[33264]: 2.0 scrub ok 2024-09-20T22:48:45.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:45 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/391701904' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-20T22:48:45.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:45 smithi154 ceph-mon[33264]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:48:45.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:45 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/391701904' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T22:48:45.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:45 smithi154 ceph-mon[33264]: osdmap e445: 8 total, 8 up, 8 in 2024-09-20T22:48:45.853 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:48:45.875 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:45.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:45.876 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:48:45.977 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:48:45.998 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:45.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:45.999 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:48:46.100 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:48:46.122 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:46.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:46.123 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:48:46.223 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:48:46.224 INFO:tasks.rados.rados.0.smithi084.stdout:1726: done (4 left) 2024-09-20T22:48:46.224 INFO:tasks.rados.rados.0.smithi084.stdout:1729: done (3 left) 2024-09-20T22:48:46.224 INFO:tasks.rados.rados.0.smithi084.stdout:1730: done (2 left) 2024-09-20T22:48:46.224 INFO:tasks.rados.rados.0.smithi084.stdout:1732: done (1 left) 2024-09-20T22:48:46.224 INFO:tasks.rados.rados.0.smithi084.stdout:1733: snap_remove snap 152 2024-09-20T22:48:46.246 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:46.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:46.247 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:48:46.348 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:48:46.369 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:46.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:46.370 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:48:46.430 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-20T22:48:46.430 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2024-09-20T22:48:46.430 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2024-09-20T22:48:46.431 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-20T22:48:46.470 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:48:46.491 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:46.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:46.492 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:48:46.593 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:48:46.614 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:46.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:46 smithi084 ceph-mon[40024]: 5.3 scrub starts 2024-09-20T22:48:46.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:46 smithi084 ceph-mon[40024]: 5.3 scrub ok 2024-09-20T22:48:46.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:46 smithi084 ceph-mon[40024]: 1.1 scrub starts 2024-09-20T22:48:46.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:46 smithi084 ceph-mon[40024]: 1.1 scrub ok 2024-09-20T22:48:46.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:46 smithi084 ceph-mon[40024]: from='client.15933 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T22:48:46.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:46 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:48:46.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:46 smithi084 ceph-mon[40024]: osdmap e446: 8 total, 8 up, 8 in 2024-09-20T22:48:46.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:46 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:48:46.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:46 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:48:46.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:46 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:48:46.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:46 smithi084 ceph-mon[29774]: 5.3 scrub starts 2024-09-20T22:48:46.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:46 smithi084 ceph-mon[29774]: 5.3 scrub ok 2024-09-20T22:48:46.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:46 smithi084 ceph-mon[29774]: 1.1 scrub starts 2024-09-20T22:48:46.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:46 smithi084 ceph-mon[29774]: 1.1 scrub ok 2024-09-20T22:48:46.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:46 smithi084 ceph-mon[29774]: from='client.15933 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T22:48:46.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:46 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:48:46.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:46 smithi084 ceph-mon[29774]: osdmap e446: 8 total, 8 up, 8 in 2024-09-20T22:48:46.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:46 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:48:46.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:46 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:48:46.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:46 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:48:46.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:46.617 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:48:46.677 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:48:46.718 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:48:46.740 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:46.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:46.741 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:48:46.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:46 smithi154 ceph-mon[33264]: 5.3 scrub starts 2024-09-20T22:48:46.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:46 smithi154 ceph-mon[33264]: 5.3 scrub ok 2024-09-20T22:48:46.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:46 smithi154 ceph-mon[33264]: 1.1 scrub starts 2024-09-20T22:48:46.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:46 smithi154 ceph-mon[33264]: 1.1 scrub ok 2024-09-20T22:48:46.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:46 smithi154 ceph-mon[33264]: from='client.15933 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T22:48:46.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:46 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:48:46.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:46 smithi154 ceph-mon[33264]: osdmap e446: 8 total, 8 up, 8 in 2024-09-20T22:48:46.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:46 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:48:46.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:46 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:48:46.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:46 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:48:46.841 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:48:46.863 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:46.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:46.864 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:48:46.965 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:48:46.986 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:46.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:46.987 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:48:47.088 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:48:47.110 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:47.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:47.111 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:48:47.115 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.1 dump_blocked_ops 2024-09-20T22:48:47.211 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:48:47.228 INFO:tasks.rados.rados.0.smithi084.stdout:1733: done (1 left) 2024-09-20T22:48:47.228 INFO:tasks.rados.rados.0.smithi084.stdout:1734: read oid 42 snap -1 2024-09-20T22:48:47.228 INFO:tasks.rados.rados.0.smithi084.stdout:1734: expect (ObjNum 554 snap 169 seq_num 554) 2024-09-20T22:48:47.228 INFO:tasks.rados.rados.0.smithi084.stdout:1735: rollback oid 36 current snap is 183 2024-09-20T22:48:47.228 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 36 to 176 2024-09-20T22:48:47.228 INFO:tasks.rados.rados.0.smithi084.stdout:1736: rollback oid 38 current snap is 183 2024-09-20T22:48:47.228 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 38 to 174 2024-09-20T22:48:47.228 INFO:tasks.rados.rados.0.smithi084.stdout:1737: read oid 29 snap -1 2024-09-20T22:48:47.228 INFO:tasks.rados.rados.0.smithi084.stdout:1737: expect (ObjNum 607 snap 181 seq_num 607) 2024-09-20T22:48:47.228 INFO:tasks.rados.rados.0.smithi084.stdout:1738: rollback oid 49 current snap is 183 2024-09-20T22:48:47.229 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 49 to 182 2024-09-20T22:48:47.229 INFO:tasks.rados.rados.0.smithi084.stdout:1739: copy_from oid 24 from oid 47 current snap is 183 2024-09-20T22:48:47.229 INFO:tasks.rados.rados.0.smithi084.stdout:1740: write (excl) oid 15 current snap is 183 2024-09-20T22:48:47.229 INFO:tasks.rados.rados.0.smithi084.stdout:1740: seq_num 615 ranges {736305=786375,2105367=230938} 2024-09-20T22:48:47.237 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:47.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:47.238 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:48:47.239 INFO:tasks.rados.rados.0.smithi084.stdout:1740: writing smithi08477203-15 from 736305 to 1522680 tid 1 2024-09-20T22:48:47.245 INFO:tasks.rados.rados.0.smithi084.stdout:1740: writing smithi08477203-15 from 2105367 to 2336305 tid 2 2024-09-20T22:48:47.245 INFO:tasks.rados.rados.0.smithi084.stdout:1741: read oid 7 snap 164 2024-09-20T22:48:47.245 INFO:tasks.rados.rados.0.smithi084.stdout:1741: expect (ObjNum 528 snap 152 seq_num 528) 2024-09-20T22:48:47.245 INFO:tasks.rados.rados.0.smithi084.stdout:1742: snap_remove snap 172 2024-09-20T22:48:47.245 INFO:tasks.rados.rados.0.smithi084.stdout:1734: expect (ObjNum 554 snap 169 seq_num 554) 2024-09-20T22:48:47.339 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:48:47.342 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:47 smithi084 ceph-mon[29774]: pgmap v570: 165 pgs: 3 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 157 active+clean; 359 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 4.4 MiB/s rd, 716 KiB/s wr, 11 op/s 2024-09-20T22:48:47.342 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:47 smithi084 ceph-mon[29774]: osdmap e447: 8 total, 8 up, 8 in 2024-09-20T22:48:47.343 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:47 smithi084 ceph-mon[40024]: pgmap v570: 165 pgs: 3 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 157 active+clean; 359 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 4.4 MiB/s rd, 716 KiB/s wr, 11 op/s 2024-09-20T22:48:47.343 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:47 smithi084 ceph-mon[40024]: osdmap e447: 8 total, 8 up, 8 in 2024-09-20T22:48:47.366 INFO:tasks.rados.rados.0.smithi084.stdout:1727: finishing write tid 1 to smithi08477203-14 2024-09-20T22:48:47.367 INFO:tasks.rados.rados.0.smithi084.stdout:1727: finishing write tid 2 to smithi08477203-14 2024-09-20T22:48:47.367 INFO:tasks.rados.rados.0.smithi084.stdout:1738: finishing rollback tid 0 to smithi08477203-49 2024-09-20T22:48:47.373 INFO:tasks.rados.rados.0.smithi084.stdout:1738: finishing rollback tid 1 to smithi08477203-49 2024-09-20T22:48:47.373 INFO:tasks.rados.rados.0.smithi084.stdout:1738: finishing rollback tid 2 to smithi08477203-49 2024-09-20T22:48:47.373 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 49 v 537 (ObjNum 557 snap 169 seq_num 557) dirty exists 2024-09-20T22:48:47.373 INFO:tasks.rados.rados.0.smithi084.stdout:1740: finishing write tid 1 to smithi08477203-15 2024-09-20T22:48:47.373 INFO:tasks.rados.rados.0.smithi084.stdout:1739: finishing copy_from racing read to smithi08477203-24 2024-09-20T22:48:47.374 INFO:tasks.rados.rados.0.smithi084.stdout:1727: finishing write tid 3 to smithi08477203-14 2024-09-20T22:48:47.374 INFO:tasks.rados.rados.0.smithi084.stdout:1727: finishing write tid 4 to smithi08477203-14 2024-09-20T22:48:47.374 INFO:tasks.rados.rados.0.smithi084.stdout:1727: oid 14 updating version 0 to 303 2024-09-20T22:48:47.374 INFO:tasks.rados.rados.0.smithi084.stdout:1727: oid 14 updating version 303 to 304 2024-09-20T22:48:47.374 INFO:tasks.rados.rados.0.smithi084.stdout:1727: oid 14 version 304 is already newer than 302 2024-09-20T22:48:47.374 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 14 v 304 (ObjNum 612 snap 182 seq_num 612) dirty exists 2024-09-20T22:48:47.374 INFO:tasks.rados.rados.0.smithi084.stdout:1727: left oid 14 (ObjNum 612 snap 182 seq_num 612) 2024-09-20T22:48:47.374 INFO:tasks.rados.rados.0.smithi084.stdout:1737: expect (ObjNum 607 snap 181 seq_num 607) 2024-09-20T22:48:47.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:47.384 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:47.384 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:48:47.485 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:48:47.506 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:47.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:47.507 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:48:47.574 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:48:47.575 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:48:47.591 INFO:tasks.rados.rados.0.smithi084.stdout:1736: finishing rollback tid 0 to smithi08477203-38 2024-09-20T22:48:47.591 INFO:tasks.rados.rados.0.smithi084.stdout:1736: finishing rollback tid 1 to smithi08477203-38 2024-09-20T22:48:47.592 INFO:tasks.rados.rados.0.smithi084.stdout:1736: finishing rollback tid 2 to smithi08477203-38 2024-09-20T22:48:47.592 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 38 v 477 (ObjNum 489 snap 137 seq_num 489) dirty exists 2024-09-20T22:48:47.592 INFO:tasks.rados.rados.0.smithi084.stdout:1735: finishing rollback tid 1 to smithi08477203-36 2024-09-20T22:48:47.592 INFO:tasks.rados.rados.0.smithi084.stdout:1735: finishing rollback tid 2 to smithi08477203-36 2024-09-20T22:48:47.592 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 36 v 398 (ObjNum 440 snap 115 seq_num 440) dirty exists 2024-09-20T22:48:47.592 INFO:tasks.rados.rados.0.smithi084.stdout:1740: finishing write tid 2 to smithi08477203-15 2024-09-20T22:48:47.592 INFO:tasks.rados.rados.0.smithi084.stdout:1740: finishing write tid 3 to smithi08477203-15 2024-09-20T22:48:47.592 INFO:tasks.rados.rados.0.smithi084.stdout:1740: finishing write tid 4 to smithi08477203-15 2024-09-20T22:48:47.592 INFO:tasks.rados.rados.0.smithi084.stdout:1740: oid 15 updating version 0 to 741 2024-09-20T22:48:47.592 INFO:tasks.rados.rados.0.smithi084.stdout:1740: oid 15 updating version 741 to 743 2024-09-20T22:48:47.592 INFO:tasks.rados.rados.0.smithi084.stdout:1740: oid 15 version 743 is already newer than 742 2024-09-20T22:48:47.592 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 15 v 743 (ObjNum 615 snap 183 seq_num 615) dirty exists 2024-09-20T22:48:47.592 INFO:tasks.rados.rados.0.smithi084.stdout:1740: left oid 15 (ObjNum 615 snap 183 seq_num 615) 2024-09-20T22:48:47.594 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:48:47.594 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:48:47.608 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:48:47.612 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:48:47.613 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:48:47.613 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:48:47.630 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:48:47.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:47.631 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:47.631 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:48:47.631 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-20T22:48:47.632 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T22:48:47.649 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:48:47.650 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:48:47.666 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:48:47.666 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:48:47.683 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:48:47.683 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:48:47.700 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:48:47.700 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:48:47.731 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:48:47.752 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:47.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:47.753 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:48:47.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:47 smithi154 ceph-mon[33264]: pgmap v570: 165 pgs: 3 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 157 active+clean; 359 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 4.4 MiB/s rd, 716 KiB/s wr, 11 op/s 2024-09-20T22:48:47.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:47 smithi154 ceph-mon[33264]: osdmap e447: 8 total, 8 up, 8 in 2024-09-20T22:48:47.853 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:48:47.875 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:47.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:47.876 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:48:47.977 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:48:47.998 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:47.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:47.999 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:48:48.100 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:48:48.116 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-20T22:48:48.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:48.123 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:48.123 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:48:48.214 INFO:tasks.rados.rados.0.smithi084.stdout:1727: done (9 left) 2024-09-20T22:48:48.214 INFO:tasks.rados.rados.0.smithi084.stdout:1734: done (8 left) 2024-09-20T22:48:48.214 INFO:tasks.rados.rados.0.smithi084.stdout:1735: done (7 left) 2024-09-20T22:48:48.214 INFO:tasks.rados.rados.0.smithi084.stdout:1736: done (6 left) 2024-09-20T22:48:48.215 INFO:tasks.rados.rados.0.smithi084.stdout:1737: done (5 left) 2024-09-20T22:48:48.215 INFO:tasks.rados.rados.0.smithi084.stdout:1738: done (4 left) 2024-09-20T22:48:48.215 INFO:tasks.rados.rados.0.smithi084.stdout:1740: done (3 left) 2024-09-20T22:48:48.215 INFO:tasks.rados.rados.0.smithi084.stdout:1742: done (2 left) 2024-09-20T22:48:48.215 INFO:tasks.rados.rados.0.smithi084.stdout:1743: snap_remove snap 167 2024-09-20T22:48:48.224 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:48:48.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:48.252 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:48.252 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:48:48.353 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:48:48.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:48.376 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:48.376 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:48:48.477 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:48:48.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:48.499 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:48.499 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:48:48.540 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:48:48 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: ::ffff:172.21.15.154 - - [20/Sep/2024:22:48:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-20T22:48:48.541 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:48 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/289074490' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:48:48.541 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:48 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd rm-pg-upmap", "pgid": "7.15"}]: dispatch 2024-09-20T22:48:48.541 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:48 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "7.15"}]: dispatch 2024-09-20T22:48:48.541 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:48 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd rm-pg-upmap", "pgid": "7.15"}]: dispatch 2024-09-20T22:48:48.541 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:48 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "7.15"}]: dispatch 2024-09-20T22:48:48.541 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:48 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd rm-pg-upmap", "pgid": "7.15"}]': finished 2024-09-20T22:48:48.541 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:48 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd rm-pg-upmap-items", "pgid": "7.15"}]': finished 2024-09-20T22:48:48.541 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:48 smithi084 ceph-mon[40024]: osdmap e448: 8 total, 8 up, 8 in 2024-09-20T22:48:48.542 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:48 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/289074490' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:48:48.542 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:48 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd rm-pg-upmap", "pgid": "7.15"}]: dispatch 2024-09-20T22:48:48.542 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:48 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "7.15"}]: dispatch 2024-09-20T22:48:48.542 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:48 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd rm-pg-upmap", "pgid": "7.15"}]: dispatch 2024-09-20T22:48:48.542 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:48 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "7.15"}]: dispatch 2024-09-20T22:48:48.542 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:48 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd rm-pg-upmap", "pgid": "7.15"}]': finished 2024-09-20T22:48:48.542 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:48 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd rm-pg-upmap-items", "pgid": "7.15"}]': finished 2024-09-20T22:48:48.542 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:48 smithi084 ceph-mon[29774]: osdmap e448: 8 total, 8 up, 8 in 2024-09-20T22:48:48.600 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:48:48.621 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:48.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:48.622 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:48:48.722 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:48:48.743 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:48.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:48.744 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:48:48.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:48 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/289074490' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:48:48.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:48 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd rm-pg-upmap", "pgid": "7.15"}]: dispatch 2024-09-20T22:48:48.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:48 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "7.15"}]: dispatch 2024-09-20T22:48:48.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:48 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd rm-pg-upmap", "pgid": "7.15"}]: dispatch 2024-09-20T22:48:48.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:48 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "7.15"}]: dispatch 2024-09-20T22:48:48.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:48 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd rm-pg-upmap", "pgid": "7.15"}]': finished 2024-09-20T22:48:48.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:48 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd rm-pg-upmap-items", "pgid": "7.15"}]': finished 2024-09-20T22:48:48.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:48 smithi154 ceph-mon[33264]: osdmap e448: 8 total, 8 up, 8 in 2024-09-20T22:48:48.845 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:48:48.866 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:48.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:48.867 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:48:48.968 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:48:48.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:48.991 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:48.992 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:48:49.092 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:48:49.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:49.116 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:49.116 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:48:49.213 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-20T22:48:49.213 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:48:49.214 INFO:teuthology.orchestra.run.smithi084.stderr:set 6.1e pg_upmap mapping to [0,1,3] 2024-09-20T22:48:49.214 INFO:tasks.rados.rados.0.smithi084.stdout:1743: done (2 left) 2024-09-20T22:48:49.214 INFO:tasks.rados.rados.0.smithi084.stdout:1744: rollback oid 13 current snap is 183 2024-09-20T22:48:49.214 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 13 to 178 2024-09-20T22:48:49.214 INFO:tasks.rados.rados.0.smithi084.stdout:1745: snap_remove snap 148 2024-09-20T22:48:49.217 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:48:49.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:49.243 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:49.243 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:48:49.344 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:48:49.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:49 smithi084 ceph-mon[40024]: pgmap v573: 164 pgs: 3 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 156 active+clean; 353 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail 2024-09-20T22:48:49.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:49 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/2694735304' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "6.1e", "id": [0, 1, 3]}]: dispatch 2024-09-20T22:48:49.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:49 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "6.1e", "id": [0, 1, 3]}]: dispatch 2024-09-20T22:48:49.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:49 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "6.1e", "id": [0, 1, 3]}]': finished 2024-09-20T22:48:49.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:49 smithi084 ceph-mon[40024]: osdmap e449: 8 total, 8 up, 8 in 2024-09-20T22:48:49.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:49 smithi084 ceph-mon[29774]: pgmap v573: 164 pgs: 3 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 156 active+clean; 353 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail 2024-09-20T22:48:49.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:49 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/2694735304' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "6.1e", "id": [0, 1, 3]}]: dispatch 2024-09-20T22:48:49.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:49 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "6.1e", "id": [0, 1, 3]}]: dispatch 2024-09-20T22:48:49.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:49 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "6.1e", "id": [0, 1, 3]}]': finished 2024-09-20T22:48:49.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:49 smithi084 ceph-mon[29774]: osdmap e449: 8 total, 8 up, 8 in 2024-09-20T22:48:49.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:49.367 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:49.367 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:48:49.468 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:48:49.489 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:49.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:49.490 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:48:49.591 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:48:49.613 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:49.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:49.614 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:48:49.715 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:48:49.736 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:49.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:49.737 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:48:49.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:49 smithi154 ceph-mon[33264]: pgmap v573: 164 pgs: 3 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 156 active+clean; 353 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail 2024-09-20T22:48:49.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:49 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/2694735304' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "6.1e", "id": [0, 1, 3]}]: dispatch 2024-09-20T22:48:49.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:49 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "6.1e", "id": [0, 1, 3]}]: dispatch 2024-09-20T22:48:49.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:49 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "6.1e", "id": [0, 1, 3]}]': finished 2024-09-20T22:48:49.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:49 smithi154 ceph-mon[33264]: osdmap e449: 8 total, 8 up, 8 in 2024-09-20T22:48:49.755 INFO:journalctl@ceph.iscsi.iscsi.a.smithi154.stdout:Sep 20 22:48:49 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-iscsi-iscsi-a[59730]: debug there is no tcmu-runner data available 2024-09-20T22:48:49.838 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:48:49.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:49.861 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:49.861 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:48:49.927 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.1 dump_historic_ops 2024-09-20T22:48:49.962 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:48:49.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:49.992 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:49.993 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:48:50.094 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:48:50.115 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:50.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:50.116 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:48:50.182 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:48:50.218 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:48:50.219 INFO:tasks.rados.rados.0.smithi084.stdout:1745: done (3 left) 2024-09-20T22:48:50.219 INFO:tasks.rados.rados.0.smithi084.stdout:1746: write (excl) oid 8 current snap is 183 2024-09-20T22:48:50.219 INFO:tasks.rados.rados.0.smithi084.stdout:1746: seq_num 616 ranges {753112=577977,1880202=472910} 2024-09-20T22:48:50.228 INFO:tasks.rados.rados.0.smithi084.stdout:1746: writing smithi08477203-8 from 753112 to 1331089 tid 1 2024-09-20T22:48:50.234 INFO:tasks.rados.rados.0.smithi084.stdout:1746: writing smithi08477203-8 from 1880202 to 2353112 tid 2 2024-09-20T22:48:50.235 INFO:tasks.rados.rados.0.smithi084.stdout:1747: snap_create 2024-09-20T22:48:50.235 INFO:tasks.rados.rados.0.smithi084.stdout:1741: expect (ObjNum 528 snap 152 seq_num 528) 2024-09-20T22:48:50.242 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:50.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:50.243 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:48:50.344 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:48:50.360 INFO:tasks.rados.rados.0.smithi084.stdout:1739: finishing copy_from to smithi08477203-24 2024-09-20T22:48:50.360 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 24 v 421 (ObjNum 566 snap 170 seq_num 566) dirty exists 2024-09-20T22:48:50.360 INFO:tasks.rados.rados.0.smithi084.stdout:1744: finishing rollback tid 0 to smithi08477203-13 2024-09-20T22:48:50.360 INFO:tasks.rados.rados.0.smithi084.stdout:1744: finishing rollback tid 1 to smithi08477203-13 2024-09-20T22:48:50.360 INFO:tasks.rados.rados.0.smithi084.stdout:1744: finishing rollback tid 2 to smithi08477203-13 2024-09-20T22:48:50.360 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 13 v 1115 (ObjNum 591 snap 174 seq_num 591) dirty exists 2024-09-20T22:48:50.360 INFO:tasks.rados.rados.0.smithi084.stdout:1746: finishing write tid 1 to smithi08477203-8 2024-09-20T22:48:50.360 INFO:tasks.rados.rados.0.smithi084.stdout:1746: finishing write tid 2 to smithi08477203-8 2024-09-20T22:48:50.361 INFO:tasks.rados.rados.0.smithi084.stdout:1746: finishing write tid 3 to smithi08477203-8 2024-09-20T22:48:50.361 INFO:tasks.rados.rados.0.smithi084.stdout:1746: finishing write tid 4 to smithi08477203-8 2024-09-20T22:48:50.361 INFO:tasks.rados.rados.0.smithi084.stdout:1746: oid 8 updating version 0 to 746 2024-09-20T22:48:50.361 INFO:tasks.rados.rados.0.smithi084.stdout:1746: oid 8 version 746 is already newer than 745 2024-09-20T22:48:50.361 INFO:tasks.rados.rados.0.smithi084.stdout:1746: oid 8 version 746 is already newer than 744 2024-09-20T22:48:50.361 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 8 v 746 (ObjNum 616 snap 183 seq_num 616) dirty exists 2024-09-20T22:48:50.361 INFO:tasks.rados.rados.0.smithi084.stdout:1746: left oid 8 (ObjNum 616 snap 183 seq_num 616) 2024-09-20T22:48:50.365 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:50.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:50.366 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:48:50.467 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:48:50.488 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:50.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:50.489 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:48:50.569 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-20T22:48:50.589 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:48:50.592 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:50 smithi084 ceph-mon[40024]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:48:50.592 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:50 smithi084 ceph-mon[40024]: osdmap e450: 8 total, 8 up, 8 in 2024-09-20T22:48:50.592 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:50 smithi084 ceph-mon[29774]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:48:50.592 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:50 smithi084 ceph-mon[29774]: osdmap e450: 8 total, 8 up, 8 in 2024-09-20T22:48:50.610 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:50.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:50.611 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:48:50.712 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:48:50.733 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:50.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:50.734 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:48:50.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:50 smithi154 ceph-mon[33264]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:48:50.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:50 smithi154 ceph-mon[33264]: osdmap e450: 8 total, 8 up, 8 in 2024-09-20T22:48:50.835 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:48:50.856 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:50.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:50.857 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:48:50.957 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:48:50.978 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:50.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:50.979 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:48:51.080 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:48:51.081 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:48:51.107 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:51.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:51.108 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:48:51.114 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:48:51.209 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:48:51.217 INFO:tasks.rados.rados.0.smithi084.stdout:1739: done (4 left) 2024-09-20T22:48:51.217 INFO:tasks.rados.rados.0.smithi084.stdout:1741: done (3 left) 2024-09-20T22:48:51.217 INFO:tasks.rados.rados.0.smithi084.stdout:1744: done (2 left) 2024-09-20T22:48:51.217 INFO:tasks.rados.rados.0.smithi084.stdout:1746: done (1 left) 2024-09-20T22:48:51.217 INFO:tasks.rados.rados.0.smithi084.stdout:1747: done (0 left) 2024-09-20T22:48:51.217 INFO:tasks.rados.rados.0.smithi084.stdout:1748: write oid 19 current snap is 184 2024-09-20T22:48:51.217 INFO:tasks.rados.rados.0.smithi084.stdout:1748: seq_num 617 ranges {769919=653226,1938684=431235} 2024-09-20T22:48:51.224 INFO:tasks.rados.rados.0.smithi084.stdout:1748: writing smithi08477203-19 from 769919 to 1423145 tid 1 2024-09-20T22:48:51.230 INFO:tasks.rados.rados.0.smithi084.stdout:1748: writing smithi08477203-19 from 1938684 to 2369919 tid 2 2024-09-20T22:48:51.230 INFO:tasks.rados.rados.0.smithi084.stdout:1749: write oid 14 current snap is 184 2024-09-20T22:48:51.230 INFO:tasks.rados.rados.0.smithi084.stdout:1749: seq_num 618 ranges {786726=728475,2280813=105913} 2024-09-20T22:48:51.233 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:51.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:51.234 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:48:51.238 INFO:tasks.rados.rados.0.smithi084.stdout:1749: writing smithi08477203-14 from 786726 to 1515201 tid 1 2024-09-20T22:48:51.243 INFO:tasks.rados.rados.0.smithi084.stdout:1749: writing smithi08477203-14 from 2280813 to 2386726 tid 2 2024-09-20T22:48:51.243 INFO:tasks.rados.rados.0.smithi084.stdout:1750: write oid 26 current snap is 184 2024-09-20T22:48:51.244 INFO:tasks.rados.rados.0.smithi084.stdout:1748: finishing write tid 1 to smithi08477203-19 2024-09-20T22:48:51.244 INFO:tasks.rados.rados.0.smithi084.stdout:1750: seq_num 619 ranges {403533=403724,1539295=652201} 2024-09-20T22:48:51.248 INFO:tasks.rados.rados.0.smithi084.stdout:1750: writing smithi08477203-26 from 403533 to 807257 tid 1 2024-09-20T22:48:51.257 INFO:tasks.rados.rados.0.smithi084.stdout:1750: writing smithi08477203-26 from 1539295 to 2191496 tid 2 2024-09-20T22:48:51.257 INFO:tasks.rados.rados.0.smithi084.stdout:1751: copy_from oid 17 from oid 46 current snap is 184 2024-09-20T22:48:51.257 INFO:tasks.rados.rados.0.smithi084.stdout:1748: finishing write tid 2 to smithi08477203-19 2024-09-20T22:48:51.258 INFO:tasks.rados.rados.0.smithi084.stdout:1748: finishing write tid 3 to smithi08477203-19 2024-09-20T22:48:51.258 INFO:tasks.rados.rados.0.smithi084.stdout:1748: finishing write tid 4 to smithi08477203-19 2024-09-20T22:48:51.258 INFO:tasks.rados.rados.0.smithi084.stdout:1748: oid 19 updating version 0 to 1139 2024-09-20T22:48:51.258 INFO:tasks.rados.rados.0.smithi084.stdout:1748: oid 19 version 1139 is already newer than 1138 2024-09-20T22:48:51.258 INFO:tasks.rados.rados.0.smithi084.stdout:1748: oid 19 version 1139 is already newer than 1137 2024-09-20T22:48:51.258 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 19 v 1139 (ObjNum 617 snap 184 seq_num 617) dirty exists 2024-09-20T22:48:51.258 INFO:tasks.rados.rados.0.smithi084.stdout:1748: left oid 19 (ObjNum 617 snap 184 seq_num 617) 2024-09-20T22:48:51.258 INFO:tasks.rados.rados.0.smithi084.stdout:1749: finishing write tid 1 to smithi08477203-14 2024-09-20T22:48:51.258 INFO:tasks.rados.rados.0.smithi084.stdout:1748: done (3 left) 2024-09-20T22:48:51.258 INFO:tasks.rados.rados.0.smithi084.stdout:1752: read oid 8 snap -1 2024-09-20T22:48:51.258 INFO:tasks.rados.rados.0.smithi084.stdout:1752: expect (ObjNum 616 snap 183 seq_num 616) 2024-09-20T22:48:51.258 INFO:tasks.rados.rados.0.smithi084.stdout:1753: read oid 9 snap -1 2024-09-20T22:48:51.259 INFO:tasks.rados.rados.0.smithi084.stdout:1753: expect deleted 2024-09-20T22:48:51.259 INFO:tasks.rados.rados.0.smithi084.stdout:1754: snap_create 2024-09-20T22:48:51.259 INFO:tasks.rados.rados.0.smithi084.stdout:1749: finishing write tid 2 to smithi08477203-14 2024-09-20T22:48:51.259 INFO:tasks.rados.rados.0.smithi084.stdout:1749: finishing write tid 3 to smithi08477203-14 2024-09-20T22:48:51.259 INFO:tasks.rados.rados.0.smithi084.stdout:1749: finishing write tid 4 to smithi08477203-14 2024-09-20T22:48:51.259 INFO:tasks.rados.rados.0.smithi084.stdout:1749: oid 14 updating version 0 to 305 2024-09-20T22:48:51.259 INFO:tasks.rados.rados.0.smithi084.stdout:1749: oid 14 updating version 305 to 306 2024-09-20T22:48:51.259 INFO:tasks.rados.rados.0.smithi084.stdout:1749: oid 14 updating version 306 to 307 2024-09-20T22:48:51.259 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 14 v 307 (ObjNum 618 snap 184 seq_num 618) dirty exists 2024-09-20T22:48:51.259 INFO:tasks.rados.rados.0.smithi084.stdout:1749: left oid 14 (ObjNum 618 snap 184 seq_num 618) 2024-09-20T22:48:51.259 INFO:tasks.rados.rados.0.smithi084.stdout:1750: finishing write tid 1 to smithi08477203-26 2024-09-20T22:48:51.261 INFO:tasks.rados.rados.0.smithi084.stdout:1750: finishing write tid 2 to smithi08477203-26 2024-09-20T22:48:51.261 INFO:tasks.rados.rados.0.smithi084.stdout:1750: finishing write tid 3 to smithi08477203-26 2024-09-20T22:48:51.261 INFO:tasks.rados.rados.0.smithi084.stdout:1750: finishing write tid 4 to smithi08477203-26 2024-09-20T22:48:51.261 INFO:tasks.rados.rados.0.smithi084.stdout:1750: oid 26 updating version 0 to 509 2024-09-20T22:48:51.261 INFO:tasks.rados.rados.0.smithi084.stdout:1750: oid 26 updating version 509 to 510 2024-09-20T22:48:51.262 INFO:tasks.rados.rados.0.smithi084.stdout:1750: oid 26 version 510 is already newer than 508 2024-09-20T22:48:51.262 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 26 v 510 (ObjNum 619 snap 184 seq_num 619) dirty exists 2024-09-20T22:48:51.262 INFO:tasks.rados.rados.0.smithi084.stdout:1750: left oid 26 (ObjNum 619 snap 184 seq_num 619) 2024-09-20T22:48:51.262 INFO:tasks.rados.rados.0.smithi084.stdout:1752: expect (ObjNum 616 snap 183 seq_num 616) 2024-09-20T22:48:51.335 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:48:51.348 INFO:tasks.rados.rados.0.smithi084.stdout:1751: finishing copy_from racing read to smithi08477203-17 2024-09-20T22:48:51.349 INFO:tasks.rados.rados.0.smithi084.stdout:1751: finishing copy_from to smithi08477203-17 2024-09-20T22:48:51.349 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 17 v 734 (ObjNum 470 snap 130 seq_num 470) dirty exists 2024-09-20T22:48:51.357 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:51.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:51.358 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:48:51.458 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:48:51.478 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:51 smithi084 ceph-mon[40024]: pgmap v576: 164 pgs: 1 peering, 3 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 155 active+clean; 352 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 7.5 MiB/s rd, 2.8 MiB/s wr, 19 op/s 2024-09-20T22:48:51.478 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:51 smithi084 ceph-mon[40024]: purged_snaps scrub starts 2024-09-20T22:48:51.478 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:51 smithi084 ceph-mon[40024]: purged_snaps scrub ok 2024-09-20T22:48:51.478 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:51 smithi084 ceph-mon[40024]: osdmap e451: 8 total, 8 up, 8 in 2024-09-20T22:48:51.478 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:51 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/3845076316' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:48:51.478 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:51 smithi084 ceph-mon[29774]: pgmap v576: 164 pgs: 1 peering, 3 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 155 active+clean; 352 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 7.5 MiB/s rd, 2.8 MiB/s wr, 19 op/s 2024-09-20T22:48:51.478 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:51 smithi084 ceph-mon[29774]: purged_snaps scrub starts 2024-09-20T22:48:51.478 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:51 smithi084 ceph-mon[29774]: purged_snaps scrub ok 2024-09-20T22:48:51.479 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:51 smithi084 ceph-mon[29774]: osdmap e451: 8 total, 8 up, 8 in 2024-09-20T22:48:51.479 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:51 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/3845076316' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:48:51.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:51.481 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:51.481 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:48:51.578 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-20T22:48:51.578 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:48:51.582 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:48:51.604 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:51.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:51.606 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:48:51.706 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:48:51.728 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:51.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:51.729 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:48:51.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:51 smithi154 ceph-mon[33264]: pgmap v576: 164 pgs: 1 peering, 3 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 155 active+clean; 352 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 7.5 MiB/s rd, 2.8 MiB/s wr, 19 op/s 2024-09-20T22:48:51.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:51 smithi154 ceph-mon[33264]: purged_snaps scrub starts 2024-09-20T22:48:51.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:51 smithi154 ceph-mon[33264]: purged_snaps scrub ok 2024-09-20T22:48:51.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:51 smithi154 ceph-mon[33264]: osdmap e451: 8 total, 8 up, 8 in 2024-09-20T22:48:51.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:51 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/3845076316' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:48:51.830 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:48:51.852 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:51.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:51.853 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:48:51.953 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:48:51.975 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:51.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:51.976 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:48:52.077 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:48:52.099 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:52.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:52.099 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:48:52.200 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:48:52.213 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is set 2024-09-20T22:48:52.217 INFO:tasks.rados.rados.0.smithi084.stdout:1749: done (5 left) 2024-09-20T22:48:52.218 INFO:tasks.rados.rados.0.smithi084.stdout:1750: done (4 left) 2024-09-20T22:48:52.218 INFO:tasks.rados.rados.0.smithi084.stdout:1751: done (3 left) 2024-09-20T22:48:52.218 INFO:tasks.rados.rados.0.smithi084.stdout:1752: done (2 left) 2024-09-20T22:48:52.218 INFO:tasks.rados.rados.0.smithi084.stdout:1753: done (1 left) 2024-09-20T22:48:52.218 INFO:tasks.rados.rados.0.smithi084.stdout:1754: done (0 left) 2024-09-20T22:48:52.218 INFO:tasks.rados.rados.0.smithi084.stdout:1755: snap_remove snap 177 2024-09-20T22:48:52.223 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:52.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:52.223 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:48:52.324 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:48:52.346 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:52.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:52.347 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:48:52.364 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:52 smithi084 ceph-mon[40024]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:48:52.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:52 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/3845076316' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T22:48:52.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:52 smithi084 ceph-mon[40024]: osdmap e452: 8 total, 8 up, 8 in 2024-09-20T22:48:52.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:52 smithi084 ceph-mon[29774]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:48:52.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:52 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/3845076316' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T22:48:52.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:52 smithi084 ceph-mon[29774]: osdmap e452: 8 total, 8 up, 8 in 2024-09-20T22:48:52.448 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:48:52.469 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:52.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:52.470 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:48:52.484 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-20T22:48:52.571 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:48:52.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:52.593 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:52.593 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:48:52.695 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:48:52.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:52.718 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:52.719 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:48:52.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:52 smithi154 ceph-mon[33264]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:48:52.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:52 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/3845076316' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T22:48:52.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:52 smithi154 ceph-mon[33264]: osdmap e452: 8 total, 8 up, 8 in 2024-09-20T22:48:52.820 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:48:52.841 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:52.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:52.842 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:48:52.942 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:48:52.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:52.965 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:52.965 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:48:53.065 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:48:53.087 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:53.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:53.088 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:48:53.189 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:48:53.210 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:53.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:53.211 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:48:53.213 INFO:tasks.rados.rados.0.smithi084.stdout:1755: done (0 left) 2024-09-20T22:48:53.213 INFO:tasks.rados.rados.0.smithi084.stdout:1756: read oid 20 snap -1 2024-09-20T22:48:53.213 INFO:tasks.rados.rados.0.smithi084.stdout:1756: expect deleted 2024-09-20T22:48:53.213 INFO:tasks.rados.rados.0.smithi084.stdout:1757: write oid 17 current snap is 185 2024-09-20T22:48:53.213 INFO:tasks.rados.rados.0.smithi084.stdout:1757: seq_num 620 ranges {420340=478973,1597777=512212} 2024-09-20T22:48:53.218 INFO:tasks.rados.rados.0.smithi084.stdout:1757: writing smithi08477203-17 from 420340 to 899313 tid 1 2024-09-20T22:48:53.226 INFO:tasks.rados.rados.0.smithi084.stdout:1757: writing smithi08477203-17 from 1597777 to 2109989 tid 2 2024-09-20T22:48:53.226 INFO:tasks.rados.rados.0.smithi084.stdout:1756: done (1 left) 2024-09-20T22:48:53.226 INFO:tasks.rados.rados.0.smithi084.stdout:1758: read oid 2 snap -1 2024-09-20T22:48:53.226 INFO:tasks.rados.rados.0.smithi084.stdout:1758: expect (ObjNum 609 snap 182 seq_num 609) 2024-09-20T22:48:53.226 INFO:tasks.rados.rados.0.smithi084.stdout:1759: snap_create 2024-09-20T22:48:53.233 INFO:tasks.rados.rados.0.smithi084.stdout:1757: finishing write tid 1 to smithi08477203-17 2024-09-20T22:48:53.233 INFO:tasks.rados.rados.0.smithi084.stdout:1757: finishing write tid 2 to smithi08477203-17 2024-09-20T22:48:53.234 INFO:tasks.rados.rados.0.smithi084.stdout:1757: finishing write tid 3 to smithi08477203-17 2024-09-20T22:48:53.234 INFO:tasks.rados.rados.0.smithi084.stdout:1758: expect (ObjNum 609 snap 182 seq_num 609) 2024-09-20T22:48:53.297 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:53 smithi084 ceph-mon[40024]: pgmap v579: 164 pgs: 1 peering, 3 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 155 active+clean; 352 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 26 MiB/s rd, 5.7 MiB/s wr, 44 op/s; 3.2 MiB/s, 6 objects/s recovering 2024-09-20T22:48:53.298 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:53 smithi084 ceph-mon[29774]: pgmap v579: 164 pgs: 1 peering, 3 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 155 active+clean; 352 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 26 MiB/s rd, 5.7 MiB/s wr, 44 op/s; 3.2 MiB/s, 6 objects/s recovering 2024-09-20T22:48:53.298 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:53 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:48:53.298 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:53 smithi084 ceph-mon[29774]: osdmap e453: 8 total, 8 up, 8 in 2024-09-20T22:48:53.312 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:48:53.313 INFO:tasks.rados.rados.0.smithi084.stdout:1757: finishing write tid 4 to smithi08477203-17 2024-09-20T22:48:53.313 INFO:tasks.rados.rados.0.smithi084.stdout:1757: oid 17 updating version 0 to 735 2024-09-20T22:48:53.313 INFO:tasks.rados.rados.0.smithi084.stdout:1757: oid 17 updating version 735 to 736 2024-09-20T22:48:53.313 INFO:tasks.rados.rados.0.smithi084.stdout:1757: oid 17 updating version 736 to 737 2024-09-20T22:48:53.313 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 17 v 737 (ObjNum 620 snap 185 seq_num 620) dirty exists 2024-09-20T22:48:53.313 INFO:tasks.rados.rados.0.smithi084.stdout:1757: left oid 17 (ObjNum 620 snap 185 seq_num 620) 2024-09-20T22:48:53.335 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:53.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:53.336 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:48:53.436 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:48:53.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:53.458 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:53.459 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:48:53.559 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:48:53.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:53.582 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:53.583 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:48:53.614 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:53 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:48:53.614 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:53 smithi084 ceph-mon[40024]: osdmap e453: 8 total, 8 up, 8 in 2024-09-20T22:48:53.683 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:48:53.705 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:53.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:53.706 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:48:53.717 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:48:53.717 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:48:53.734 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:48:53.735 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:48:53.752 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:48:53.752 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:48:53.752 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:48:53.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:53 smithi154 ceph-mon[33264]: pgmap v579: 164 pgs: 1 peering, 3 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 155 active+clean; 352 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 26 MiB/s rd, 5.7 MiB/s wr, 44 op/s; 3.2 MiB/s, 6 objects/s recovering 2024-09-20T22:48:53.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:53 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:48:53.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:53 smithi154 ceph-mon[33264]: osdmap e453: 8 total, 8 up, 8 in 2024-09-20T22:48:53.770 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-20T22:48:53.770 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T22:48:53.788 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:48:53.788 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:48:53.804 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:48:53.804 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:48:53.808 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:48:53.821 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:48:53.821 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:48:53.829 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:53.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:53.830 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:48:53.837 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:48:53.837 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:48:53.931 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:48:53.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:53.954 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:53.955 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:48:54.055 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:48:54.077 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:54.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:54.078 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:48:54.179 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:48:54.200 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:54.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:54.201 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:48:54.224 INFO:tasks.rados.rados.0.smithi084.stdout:1757: done (2 left) 2024-09-20T22:48:54.224 INFO:tasks.rados.rados.0.smithi084.stdout:1758: done (1 left) 2024-09-20T22:48:54.224 INFO:tasks.rados.rados.0.smithi084.stdout:1759: done (0 left) 2024-09-20T22:48:54.224 INFO:tasks.rados.rados.0.smithi084.stdout:1760: read oid 30 snap -1 2024-09-20T22:48:54.224 INFO:tasks.rados.rados.0.smithi084.stdout:1760: expect (ObjNum 602 snap 177 seq_num 602) 2024-09-20T22:48:54.224 INFO:tasks.rados.rados.0.smithi084.stdout:1761: rollback oid 12 current snap is 186 2024-09-20T22:48:54.224 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 12 to 181 2024-09-20T22:48:54.224 INFO:tasks.rados.rados.0.smithi084.stdout:1762: read oid 45 snap -1 2024-09-20T22:48:54.224 INFO:tasks.rados.rados.0.smithi084.stdout:1762: expect deleted 2024-09-20T22:48:54.224 INFO:tasks.rados.rados.0.smithi084.stdout:1763: delete oid 41 current snap is 186 2024-09-20T22:48:54.227 INFO:tasks.rados.rados.0.smithi084.stdout:1760: expect (ObjNum 602 snap 177 seq_num 602) 2024-09-20T22:48:54.302 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:48:54.323 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:54.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:54.324 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:48:54.325 INFO:tasks.rados.rados.0.smithi084.stdout:1761: finishing rollback tid 1 to smithi08477203-12 2024-09-20T22:48:54.325 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 12 v 404 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-09-20T22:48:54.326 INFO:tasks.rados.rados.0.smithi084.stdout:1760: done (3 left) 2024-09-20T22:48:54.326 INFO:tasks.rados.rados.0.smithi084.stdout:1761: done (2 left) 2024-09-20T22:48:54.326 INFO:tasks.rados.rados.0.smithi084.stdout:1762: done (1 left) 2024-09-20T22:48:54.326 INFO:tasks.rados.rados.0.smithi084.stdout:1763: done (0 left) 2024-09-20T22:48:54.326 INFO:tasks.rados.rados.0.smithi084.stdout:1764: write (excl) oid 34 current snap is 186 2024-09-20T22:48:54.326 INFO:tasks.rados.rados.0.smithi084.stdout:1764: seq_num 622 ranges {453954=629471,1598388=515881} 2024-09-20T22:48:54.331 INFO:tasks.rados.rados.0.smithi084.stdout:1764: writing smithi08477203-34 from 453954 to 1083425 tid 1 2024-09-20T22:48:54.337 INFO:tasks.rados.rados.0.smithi084.stdout:1764: writing smithi08477203-34 from 1598388 to 2114269 tid 2 2024-09-20T22:48:54.337 INFO:tasks.rados.rados.0.smithi084.stdout:1765: snap_remove snap 174 2024-09-20T22:48:54.425 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:48:54.447 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:54.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:54.448 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:48:54.549 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:48:54.572 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:54.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:54.573 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:48:54.614 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:54 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/1039753886' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:48:54.614 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:54 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-09-20T22:48:54.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:54 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-09-20T22:48:54.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:54 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "21"}]': finished 2024-09-20T22:48:54.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:54 smithi084 ceph-mon[40024]: osdmap e454: 8 total, 8 up, 8 in 2024-09-20T22:48:54.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:54 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/1039753886' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:48:54.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:54 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-09-20T22:48:54.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:54 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-09-20T22:48:54.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:54 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "21"}]': finished 2024-09-20T22:48:54.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:54 smithi084 ceph-mon[29774]: osdmap e454: 8 total, 8 up, 8 in 2024-09-20T22:48:54.673 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:48:54.696 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:54.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:54.697 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:48:54.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:54 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/1039753886' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:48:54.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:54 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-09-20T22:48:54.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:54 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "21"}]: dispatch 2024-09-20T22:48:54.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:54 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "21"}]': finished 2024-09-20T22:48:54.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:54 smithi154 ceph-mon[33264]: osdmap e454: 8 total, 8 up, 8 in 2024-09-20T22:48:54.757 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-20T22:48:54.758 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2024-09-20T22:48:54.758 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2024-09-20T22:48:54.758 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-20T22:48:54.798 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:48:54.822 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:54.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:54.823 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:48:54.923 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:48:54.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:54.945 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:54.945 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:48:55.046 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:48:55.068 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:55.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:55.069 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:48:55.169 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:48:55.196 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:55.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:55.197 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:48:55.227 INFO:tasks.rados.rados.0.smithi084.stdout:1765: done (1 left) 2024-09-20T22:48:55.227 INFO:tasks.rados.rados.0.smithi084.stdout:1766: read oid 3 snap 156 2024-09-20T22:48:55.227 INFO:tasks.rados.rados.0.smithi084.stdout:1766: expect (ObjNum 418 snap 105 seq_num 418) 2024-09-20T22:48:55.227 INFO:tasks.rados.rados.0.smithi084.stdout:1767: read oid 29 snap -1 2024-09-20T22:48:55.228 INFO:tasks.rados.rados.0.smithi084.stdout:1767: expect (ObjNum 607 snap 181 seq_num 607) 2024-09-20T22:48:55.228 INFO:tasks.rados.rados.0.smithi084.stdout:1768: snap_remove snap 180 2024-09-20T22:48:55.298 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:48:55.320 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:55.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:55.321 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:48:55.338 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.2 dump_blocked_ops 2024-09-20T22:48:55.422 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:48:55.447 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:55.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:55.448 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:48:55.468 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd unset noscrub 2024-09-20T22:48:55.549 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:48:55.570 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:55.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:55.571 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:48:55.614 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:55 smithi084 ceph-mon[40024]: pgmap v582: 164 pgs: 3 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 156 active+clean; 356 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 4.9 MiB/s rd, 2.1 MiB/s wr, 13 op/s 2024-09-20T22:48:55.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:55 smithi084 ceph-mon[40024]: osdmap e455: 8 total, 8 up, 8 in 2024-09-20T22:48:55.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:55 smithi084 ceph-mon[29774]: pgmap v582: 164 pgs: 3 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 156 active+clean; 356 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 4.9 MiB/s rd, 2.1 MiB/s wr, 13 op/s 2024-09-20T22:48:55.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:55 smithi084 ceph-mon[29774]: osdmap e455: 8 total, 8 up, 8 in 2024-09-20T22:48:55.631 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:48:55.634 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 5, 6, 7, 0, 4] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-20T22:48:55.634 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-20T22:48:55.634 INFO:tasks.thrashosds.thrasher:fixing pg num pool unique_pool_0 2024-09-20T22:48:55.634 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph pg dump --format=json 2024-09-20T22:48:55.672 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:48:55.698 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:55.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:55.699 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:48:55.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:55 smithi154 ceph-mon[33264]: pgmap v582: 164 pgs: 3 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 156 active+clean; 356 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 4.9 MiB/s rd, 2.1 MiB/s wr, 13 op/s 2024-09-20T22:48:55.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:55 smithi154 ceph-mon[33264]: osdmap e455: 8 total, 8 up, 8 in 2024-09-20T22:48:55.799 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:48:55.821 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:55.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:55.822 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:48:55.923 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:48:55.943 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:55.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:55.944 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:48:56.045 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:48:56.071 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:56.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:56.072 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:48:56.117 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-20T22:48:56.117 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2024-09-20T22:48:56.117 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2024-09-20T22:48:56.117 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2024-09-20T22:48:56.117 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-20T22:48:56.173 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:48:56.198 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:56.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:56.199 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:48:56.231 INFO:tasks.rados.rados.0.smithi084.stdout:1768: done (3 left) 2024-09-20T22:48:56.231 INFO:tasks.rados.rados.0.smithi084.stdout:1769: rollback oid 6 current snap is 186 2024-09-20T22:48:56.231 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 6 to 156 2024-09-20T22:48:56.231 INFO:tasks.rados.rados.0.smithi084.stdout:1770: read oid 37 snap -1 2024-09-20T22:48:56.231 INFO:tasks.rados.rados.0.smithi084.stdout:1770: expect deleted 2024-09-20T22:48:56.231 INFO:tasks.rados.rados.0.smithi084.stdout:1771: rollback oid 11 current snap is 186 2024-09-20T22:48:56.231 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 11 to 133 2024-09-20T22:48:56.232 INFO:tasks.rados.rados.0.smithi084.stdout:1772: read oid 30 snap -1 2024-09-20T22:48:56.232 INFO:tasks.rados.rados.0.smithi084.stdout:1772: expect (ObjNum 602 snap 177 seq_num 602) 2024-09-20T22:48:56.232 INFO:tasks.rados.rados.0.smithi084.stdout:1773: read oid 31 snap 178 2024-09-20T22:48:56.232 INFO:tasks.rados.rados.0.smithi084.stdout:1773: expect deleted 2024-09-20T22:48:56.232 INFO:tasks.rados.rados.0.smithi084.stdout:1774: read oid 18 snap 164 2024-09-20T22:48:56.232 INFO:tasks.rados.rados.0.smithi084.stdout:1774: expect (ObjNum 535 snap 159 seq_num 535) 2024-09-20T22:48:56.232 INFO:tasks.rados.rados.0.smithi084.stdout:1775: copy_from oid 49 from oid 28 current snap is 186 2024-09-20T22:48:56.232 INFO:tasks.rados.rados.0.smithi084.stdout:1764: finishing write tid 1 to smithi08477203-34 2024-09-20T22:48:56.232 INFO:tasks.rados.rados.0.smithi084.stdout:1764: finishing write tid 2 to smithi08477203-34 2024-09-20T22:48:56.232 INFO:tasks.rados.rados.0.smithi084.stdout:1764: finishing write tid 3 to smithi08477203-34 2024-09-20T22:48:56.232 INFO:tasks.rados.rados.0.smithi084.stdout:1764: finishing write tid 4 to smithi08477203-34 2024-09-20T22:48:56.232 INFO:tasks.rados.rados.0.smithi084.stdout:1764: oid 34 updating version 0 to 538 2024-09-20T22:48:56.232 INFO:tasks.rados.rados.0.smithi084.stdout:1764: oid 34 updating version 538 to 540 2024-09-20T22:48:56.232 INFO:tasks.rados.rados.0.smithi084.stdout:1764: oid 34 version 540 is already newer than 539 2024-09-20T22:48:56.232 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 34 v 540 (ObjNum 622 snap 186 seq_num 622) dirty exists 2024-09-20T22:48:56.233 INFO:tasks.rados.rados.0.smithi084.stdout:1764: left oid 34 (ObjNum 622 snap 186 seq_num 622) 2024-09-20T22:48:56.233 INFO:tasks.rados.rados.0.smithi084.stdout:1766: expect (ObjNum 418 snap 105 seq_num 418) 2024-09-20T22:48:56.300 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:48:56.323 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:56.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:56.324 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:48:56.339 INFO:tasks.rados.rados.0.smithi084.stdout:1767: expect (ObjNum 607 snap 181 seq_num 607) 2024-09-20T22:48:56.379 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:48:56.425 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:48:56.447 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:56.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:56.448 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:48:56.548 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:48:56.556 INFO:tasks.rados.rados.0.smithi084.stdout:1772: expect (ObjNum 602 snap 177 seq_num 602) 2024-09-20T22:48:56.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:56.571 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:56.571 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:48:56.649 INFO:tasks.rados.rados.0.smithi084.stdout:1771: finishing rollback tid 0 to smithi08477203-11 2024-09-20T22:48:56.649 INFO:tasks.rados.rados.0.smithi084.stdout:1774: expect (ObjNum 535 snap 159 seq_num 535) 2024-09-20T22:48:56.671 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-20T22:48:56.672 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:48:56.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:56.695 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:56.695 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:48:56.796 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:48:56.818 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:56.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:56.819 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:48:56.826 INFO:tasks.rados.rados.0.smithi084.stdout:1769: finishing rollback tid 1 to smithi08477203-6 2024-09-20T22:48:56.826 INFO:tasks.rados.rados.0.smithi084.stdout:1769: finishing rollback tid 2 to smithi08477203-6 2024-09-20T22:48:56.826 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 6 v 739 (ObjNum 502 snap 143 seq_num 502) dirty exists 2024-09-20T22:48:56.827 INFO:tasks.rados.rados.0.smithi084.stdout:1771: finishing rollback tid 1 to smithi08477203-11 2024-09-20T22:48:56.827 INFO:tasks.rados.rados.0.smithi084.stdout:1771: finishing rollback tid 2 to smithi08477203-11 2024-09-20T22:48:56.827 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 11 v 692 (ObjNum 411 snap 105 seq_num 411) dirty exists 2024-09-20T22:48:56.827 INFO:tasks.rados.rados.0.smithi084.stdout:1775: finishing copy_from racing read to smithi08477203-49 2024-09-20T22:48:56.827 INFO:tasks.rados.rados.0.smithi084.stdout:1775: finishing copy_from to smithi08477203-49 2024-09-20T22:48:56.827 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 49 v 722 (ObjNum 470 snap 130 seq_num 470) dirty exists 2024-09-20T22:48:56.827 INFO:tasks.rados.rados.0.smithi084.stdout:1764: done (9 left) 2024-09-20T22:48:56.827 INFO:tasks.rados.rados.0.smithi084.stdout:1766: done (8 left) 2024-09-20T22:48:56.827 INFO:tasks.rados.rados.0.smithi084.stdout:1767: done (7 left) 2024-09-20T22:48:56.827 INFO:tasks.rados.rados.0.smithi084.stdout:1769: done (6 left) 2024-09-20T22:48:56.827 INFO:tasks.rados.rados.0.smithi084.stdout:1770: done (5 left) 2024-09-20T22:48:56.827 INFO:tasks.rados.rados.0.smithi084.stdout:1771: done (4 left) 2024-09-20T22:48:56.827 INFO:tasks.rados.rados.0.smithi084.stdout:1772: done (3 left) 2024-09-20T22:48:56.827 INFO:tasks.rados.rados.0.smithi084.stdout:1773: done (2 left) 2024-09-20T22:48:56.828 INFO:tasks.rados.rados.0.smithi084.stdout:1774: done (1 left) 2024-09-20T22:48:56.828 INFO:tasks.rados.rados.0.smithi084.stdout:1775: done (0 left) 2024-09-20T22:48:56.828 INFO:tasks.rados.rados.0.smithi084.stdout:1776: write oid 1 current snap is 186 2024-09-20T22:48:56.828 INFO:tasks.rados.rados.0.smithi084.stdout:1776: seq_num 623 ranges {470761=704720,1656870=659539} 2024-09-20T22:48:56.833 INFO:tasks.rados.rados.0.smithi084.stdout:1776: writing smithi08477203-1 from 470761 to 1175481 tid 1 2024-09-20T22:48:56.839 INFO:tasks.rados.rados.0.smithi084.stdout:1776: writing smithi08477203-1 from 1656870 to 2316409 tid 2 2024-09-20T22:48:56.839 INFO:tasks.rados.rados.0.smithi084.stdout:1777: delete oid 42 current snap is 186 2024-09-20T22:48:56.839 INFO:tasks.rados.rados.0.smithi084.stdout:1776: finishing write tid 1 to smithi08477203-1 2024-09-20T22:48:56.841 INFO:tasks.rados.rados.0.smithi084.stdout:1777: done (1 left) 2024-09-20T22:48:56.841 INFO:tasks.rados.rados.0.smithi084.stdout:1778: write oid 15 current snap is 186 2024-09-20T22:48:56.841 INFO:tasks.rados.rados.0.smithi084.stdout:1778: seq_num 625 ranges {504375=571571,1490187=663208} 2024-09-20T22:48:56.847 INFO:tasks.rados.rados.0.smithi084.stdout:1778: writing smithi08477203-15 from 504375 to 1075946 tid 1 2024-09-20T22:48:56.853 INFO:tasks.rados.rados.0.smithi084.stdout:1778: writing smithi08477203-15 from 1490187 to 2153395 tid 2 2024-09-20T22:48:56.854 INFO:tasks.rados.rados.0.smithi084.stdout:1779: snap_create 2024-09-20T22:48:56.854 INFO:tasks.rados.rados.0.smithi084.stdout:1776: finishing write tid 2 to smithi08477203-1 2024-09-20T22:48:56.854 INFO:tasks.rados.rados.0.smithi084.stdout:1776: finishing write tid 3 to smithi08477203-1 2024-09-20T22:48:56.854 INFO:tasks.rados.rados.0.smithi084.stdout:1776: finishing write tid 4 to smithi08477203-1 2024-09-20T22:48:56.854 INFO:tasks.rados.rados.0.smithi084.stdout:1776: oid 1 updating version 0 to 631 2024-09-20T22:48:56.854 INFO:tasks.rados.rados.0.smithi084.stdout:1776: oid 1 version 631 is already newer than 629 2024-09-20T22:48:56.854 INFO:tasks.rados.rados.0.smithi084.stdout:1776: oid 1 version 631 is already newer than 630 2024-09-20T22:48:56.854 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 1 v 631 (ObjNum 623 snap 186 seq_num 623) dirty exists 2024-09-20T22:48:56.854 INFO:tasks.rados.rados.0.smithi084.stdout:1776: left oid 1 (ObjNum 623 snap 186 seq_num 623) 2024-09-20T22:48:56.854 INFO:tasks.rados.rados.0.smithi084.stdout:1778: finishing write tid 1 to smithi08477203-15 2024-09-20T22:48:56.857 INFO:tasks.rados.rados.0.smithi084.stdout:1778: finishing write tid 2 to smithi08477203-15 2024-09-20T22:48:56.857 INFO:tasks.rados.rados.0.smithi084.stdout:1778: finishing write tid 3 to smithi08477203-15 2024-09-20T22:48:56.858 INFO:tasks.rados.rados.0.smithi084.stdout:1778: finishing write tid 4 to smithi08477203-15 2024-09-20T22:48:56.858 INFO:tasks.rados.rados.0.smithi084.stdout:1778: oid 15 updating version 0 to 747 2024-09-20T22:48:56.858 INFO:tasks.rados.rados.0.smithi084.stdout:1778: oid 15 updating version 747 to 749 2024-09-20T22:48:56.858 INFO:tasks.rados.rados.0.smithi084.stdout:1778: oid 15 version 749 is already newer than 748 2024-09-20T22:48:56.858 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 15 v 749 (ObjNum 625 snap 186 seq_num 625) dirty exists 2024-09-20T22:48:56.858 INFO:tasks.rados.rados.0.smithi084.stdout:1778: left oid 15 (ObjNum 625 snap 186 seq_num 625) 2024-09-20T22:48:56.890 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.2 dump_historic_ops 2024-09-20T22:48:56.919 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:48:56.951 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:56.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:56.952 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:48:57.053 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:48:57.073 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:57.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:57.074 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:48:57.110 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:48:57.175 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:48:57.199 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:57.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:57.200 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:48:57.237 INFO:tasks.rados.rados.0.smithi084.stdout:1776: done (2 left) 2024-09-20T22:48:57.237 INFO:tasks.rados.rados.0.smithi084.stdout:1778: done (1 left) 2024-09-20T22:48:57.237 INFO:tasks.rados.rados.0.smithi084.stdout:1779: done (0 left) 2024-09-20T22:48:57.237 INFO:tasks.rados.rados.0.smithi084.stdout:1780: snap_create 2024-09-20T22:48:57.300 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:48:57.323 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:57.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:57.323 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:48:57.364 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:57 smithi084 ceph-mon[40024]: pgmap v585: 164 pgs: 3 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 156 active+clean; 356 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 4.9 MiB/s rd, 2.1 MiB/s wr, 13 op/s 2024-09-20T22:48:57.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:57 smithi084 ceph-mon[40024]: osdmap e456: 8 total, 8 up, 8 in 2024-09-20T22:48:57.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:57 smithi084 ceph-mon[29774]: pgmap v585: 164 pgs: 3 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 156 active+clean; 356 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 4.9 MiB/s rd, 2.1 MiB/s wr, 13 op/s 2024-09-20T22:48:57.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:57 smithi084 ceph-mon[29774]: osdmap e456: 8 total, 8 up, 8 in 2024-09-20T22:48:57.425 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:48:57.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:57.447 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:57.447 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:48:57.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:57 smithi154 ceph-mon[33264]: pgmap v585: 164 pgs: 3 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 156 active+clean; 356 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 4.9 MiB/s rd, 2.1 MiB/s wr, 13 op/s 2024-09-20T22:48:57.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:57 smithi154 ceph-mon[33264]: osdmap e456: 8 total, 8 up, 8 in 2024-09-20T22:48:57.548 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:48:57.571 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:57.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:57.572 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:48:57.673 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:48:57.694 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:57.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:57.695 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:48:57.796 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:48:57.817 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:57.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:57.818 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:48:57.833 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:48:57.834 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:48:57.919 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:48:57.943 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:57.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:57.944 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:48:58.042 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-20T22:48:58.044 INFO:teuthology.orchestra.run.smithi084.stderr:dumped all 2024-09-20T22:48:58.045 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:48:58.071 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:58.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:58.072 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:48:58.172 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:48:58.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:58.194 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:58.194 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:48:58.210 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is unset 2024-09-20T22:48:58.216 INFO:tasks.rados.rados.0.smithi084.stdout:1780: done (0 left) 2024-09-20T22:48:58.216 INFO:tasks.rados.rados.0.smithi084.stdout:1781: snap_create 2024-09-20T22:48:58.295 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:48:58.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:58.317 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:58.317 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:48:58.365 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:48:58 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: ::ffff:172.21.15.154 - - [20/Sep/2024:22:48:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-20T22:48:58.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:58 smithi084 ceph-mon[40024]: osdmap e457: 8 total, 8 up, 8 in 2024-09-20T22:48:58.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:58 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/2877928590' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:48:58.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:58 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "20"}]: dispatch 2024-09-20T22:48:58.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:58 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "20"}]: dispatch 2024-09-20T22:48:58.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:58 smithi084 ceph-mon[40024]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:48:58.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:58 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/2877928590' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T22:48:58.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:58 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "20"}]': finished 2024-09-20T22:48:58.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:58 smithi084 ceph-mon[40024]: osdmap e458: 8 total, 8 up, 8 in 2024-09-20T22:48:58.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:58 smithi084 ceph-mon[29774]: osdmap e457: 8 total, 8 up, 8 in 2024-09-20T22:48:58.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:58 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/2877928590' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:48:58.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:58 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "20"}]: dispatch 2024-09-20T22:48:58.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:58 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "20"}]: dispatch 2024-09-20T22:48:58.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:58 smithi084 ceph-mon[29774]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:48:58.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:58 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/2877928590' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T22:48:58.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:58 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "20"}]': finished 2024-09-20T22:48:58.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:58 smithi084 ceph-mon[29774]: osdmap e458: 8 total, 8 up, 8 in 2024-09-20T22:48:58.418 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:48:58.443 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:58.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:58.444 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:48:58.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:58 smithi154 ceph-mon[33264]: osdmap e457: 8 total, 8 up, 8 in 2024-09-20T22:48:58.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:58 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/2877928590' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:48:58.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:58 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "20"}]: dispatch 2024-09-20T22:48:58.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:58 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "20"}]: dispatch 2024-09-20T22:48:58.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:58 smithi154 ceph-mon[33264]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:48:58.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:58 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/2877928590' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T22:48:58.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:58 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "20"}]': finished 2024-09-20T22:48:58.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:58 smithi154 ceph-mon[33264]: osdmap e458: 8 total, 8 up, 8 in 2024-09-20T22:48:58.545 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:48:58.570 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:58.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:58.571 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:48:58.672 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:48:58.696 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:58.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:58.697 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:48:58.714 INFO:teuthology.orchestra.run.smithi084.stdout:{"pg_ready":true,"pg_map":{"version":588,"stamp":"2024-09-20T22:48:57.720418+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":373322857,"num_objects":374,"num_object_clones":127,"num_object_copies":1122,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":374,"num_whiteouts":13,"num_read":4211,"num_read_kb":3211640,"num_write":2579,"num_write_kb":805411,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":874,"num_bytes_recovered":297926788,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6288,"ondisk_log_size":6288,"up":489,"acting":489,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":492,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":1174128,"kb_used_data":835744,"kb_used_omap":12,"kb_used_meta":338163,"kb_avail":748787088,"statfs":{"total":767960285184,"available":766757978112,"internally_reserved":0,"allocated":855801856,"data_stored":1171503416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":346279472},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001163"},"pg_stats":[{"pgid":"3.1f","version":"86'2","reported_seq":143,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-20T22:48:53.211620+0000","last_change":"2024-09-20T22:47:49.237762+0000","last_active":"2024-09-20T22:48:53.211620+0000","last_peered":"2024-09-20T22:48:53.211620+0000","last_clean":"2024-09-20T22:48:53.211620+0000","last_became_active":"2024-09-20T22:47:49.222114+0000","last_became_peered":"2024-09-20T22:47:49.222114+0000","last_unstale":"2024-09-20T22:48:53.211620+0000","last_undegraded":"2024-09-20T22:48:53.211620+0000","last_fullsized":"2024-09-20T22:48:53.211620+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'2","last_scrub_stamp":"2024-09-20T22:46:42.206600+0000","last_deep_scrub":"86'2","last_deep_scrub_stamp":"2024-09-20T22:42:39.664421+0000","last_clean_scrub_stamp":"2024-09-20T22:46:42.206600+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:56.612228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":436,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"74'9","reported_seq":117,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-20T22:48:53.214164+0000","last_change":"2024-09-20T22:48:05.283966+0000","last_active":"2024-09-20T22:48:53.214164+0000","last_peered":"2024-09-20T22:48:53.214164+0000","last_clean":"2024-09-20T22:48:53.214164+0000","last_became_active":"2024-09-20T22:48:05.248980+0000","last_became_peered":"2024-09-20T22:48:05.248980+0000","last_unstale":"2024-09-20T22:48:53.214164+0000","last_undegraded":"2024-09-20T22:48:53.214164+0000","last_fullsized":"2024-09-20T22:48:53.214164+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-09-20T22:46:20.889510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:46:20.889510+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:58.056027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":800,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-20T22:48:53.209580+0000","last_change":"2024-09-20T22:48:43.157340+0000","last_active":"2024-09-20T22:48:53.209580+0000","last_peered":"2024-09-20T22:48:53.209580+0000","last_clean":"2024-09-20T22:48:53.209580+0000","last_became_active":"2024-09-20T22:41:42.182024+0000","last_became_peered":"2024-09-20T22:41:42.182024+0000","last_unstale":"2024-09-20T22:48:53.209580+0000","last_undegraded":"2024-09-20T22:48:53.209580+0000","last_fullsized":"2024-09-20T22:48:53.209580+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:48:43.157266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:48:43.157266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:01.873970+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":107,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-20T22:48:53.214632+0000","last_change":"2024-09-20T22:48:05.248713+0000","last_active":"2024-09-20T22:48:53.214632+0000","last_peered":"2024-09-20T22:48:53.214632+0000","last_clean":"2024-09-20T22:48:53.214632+0000","last_became_active":"2024-09-20T22:48:05.247569+0000","last_became_peered":"2024-09-20T22:48:05.247569+0000","last_unstale":"2024-09-20T22:48:53.214632+0000","last_undegraded":"2024-09-20T22:48:53.214632+0000","last_fullsized":"2024-09-20T22:48:53.214632+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:46:45.177105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:46:45.177105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:58.056010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":788,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-20T22:48:53.216404+0000","last_change":"2024-09-20T22:45:40.053329+0000","last_active":"2024-09-20T22:48:53.216404+0000","last_peered":"2024-09-20T22:48:53.216404+0000","last_clean":"2024-09-20T22:48:53.216404+0000","last_became_active":"2024-09-20T22:41:44.832099+0000","last_became_peered":"2024-09-20T22:41:44.832099+0000","last_unstale":"2024-09-20T22:48:53.216404+0000","last_undegraded":"2024-09-20T22:48:53.216404+0000","last_fullsized":"2024-09-20T22:48:53.216404+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:45:40.053263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:45:40.053263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:55.172327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":793,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-20T22:48:53.215543+0000","last_change":"2024-09-20T22:44:58.994410+0000","last_active":"2024-09-20T22:48:53.215543+0000","last_peered":"2024-09-20T22:48:53.215543+0000","last_clean":"2024-09-20T22:48:53.215543+0000","last_became_active":"2024-09-20T22:41:38.157298+0000","last_became_peered":"2024-09-20T22:41:38.157298+0000","last_unstale":"2024-09-20T22:48:53.215543+0000","last_undegraded":"2024-09-20T22:48:53.215543+0000","last_fullsized":"2024-09-20T22:48:53.215543+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:44:58.994340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:44:58.994340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:55.172183+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"74'15","reported_seq":857,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-20T22:48:53.213568+0000","last_change":"2024-09-20T22:47:49.246946+0000","last_active":"2024-09-20T22:48:53.213568+0000","last_peered":"2024-09-20T22:48:53.213568+0000","last_clean":"2024-09-20T22:48:53.213568+0000","last_became_active":"2024-09-20T22:47:49.218798+0000","last_became_peered":"2024-09-20T22:47:49.218798+0000","last_unstale":"2024-09-20T22:48:53.213568+0000","last_undegraded":"2024-09-20T22:48:53.213568+0000","last_fullsized":"2024-09-20T22:48:53.213568+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-09-20T22:44:41.108205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:44:41.108205+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:55.172171+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":107,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-20T22:48:53.214071+0000","last_change":"2024-09-20T22:48:05.247938+0000","last_active":"2024-09-20T22:48:53.214071+0000","last_peered":"2024-09-20T22:48:53.214071+0000","last_clean":"2024-09-20T22:48:53.214071+0000","last_became_active":"2024-09-20T22:48:05.247636+0000","last_became_peered":"2024-09-20T22:48:05.247636+0000","last_unstale":"2024-09-20T22:48:53.214071+0000","last_undegraded":"2024-09-20T22:48:53.214071+0000","last_fullsized":"2024-09-20T22:48:53.214071+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:47:19.805701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:47:19.805701+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:58.056089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":813,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-20T22:48:53.227626+0000","last_change":"2024-09-20T22:48:05.242386+0000","last_active":"2024-09-20T22:48:53.227626+0000","last_peered":"2024-09-20T22:48:53.227626+0000","last_clean":"2024-09-20T22:48:53.227626+0000","last_became_active":"2024-09-20T22:48:05.242059+0000","last_became_peered":"2024-09-20T22:48:05.242059+0000","last_unstale":"2024-09-20T22:48:53.227626+0000","last_undegraded":"2024-09-20T22:48:53.227626+0000","last_fullsized":"2024-09-20T22:48:53.227626+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:46:00.999939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:46:00.999939+0000","last_clean_scrub_stamp":"2024-09-20T22:46:00.999939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"74'9","reported_seq":117,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-20T22:48:53.214644+0000","last_change":"2024-09-20T22:48:05.259778+0000","last_active":"2024-09-20T22:48:53.214644+0000","last_peered":"2024-09-20T22:48:53.214644+0000","last_clean":"2024-09-20T22:48:53.214644+0000","last_became_active":"2024-09-20T22:48:05.233429+0000","last_became_peered":"2024-09-20T22:48:05.233429+0000","last_unstale":"2024-09-20T22:48:53.214644+0000","last_undegraded":"2024-09-20T22:48:53.214644+0000","last_fullsized":"2024-09-20T22:48:53.214644+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-09-20T22:46:45.095957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:46:45.095957+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:58.056058+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":805,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-20T22:48:53.227263+0000","last_change":"2024-09-20T22:47:49.228490+0000","last_active":"2024-09-20T22:48:53.227263+0000","last_peered":"2024-09-20T22:48:53.227263+0000","last_clean":"2024-09-20T22:48:53.227263+0000","last_became_active":"2024-09-20T22:47:49.227379+0000","last_became_peered":"2024-09-20T22:47:49.227379+0000","last_unstale":"2024-09-20T22:48:53.227263+0000","last_undegraded":"2024-09-20T22:48:53.227263+0000","last_fullsized":"2024-09-20T22:48:53.227263+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:46:52.239508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:46:52.239508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":139,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-20T22:48:53.208780+0000","last_change":"2024-09-20T22:47:49.225833+0000","last_active":"2024-09-20T22:48:53.208780+0000","last_peered":"2024-09-20T22:48:53.208780+0000","last_clean":"2024-09-20T22:48:53.208780+0000","last_became_active":"2024-09-20T22:47:49.225515+0000","last_became_peered":"2024-09-20T22:47:49.225515+0000","last_unstale":"2024-09-20T22:48:53.208780+0000","last_undegraded":"2024-09-20T22:48:53.208780+0000","last_fullsized":"2024-09-20T22:48:53.208780+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:46:29.801854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:46:29.801854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:56.612181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":821,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-20T22:48:53.227388+0000","last_change":"2024-09-20T22:48:37.559081+0000","last_active":"2024-09-20T22:48:53.227388+0000","last_peered":"2024-09-20T22:48:53.227388+0000","last_clean":"2024-09-20T22:48:53.227388+0000","last_became_active":"2024-09-20T22:48:05.242084+0000","last_became_peered":"2024-09-20T22:48:05.242084+0000","last_unstale":"2024-09-20T22:48:53.227388+0000","last_undegraded":"2024-09-20T22:48:53.227388+0000","last_fullsized":"2024-09-20T22:48:53.227388+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:48:37.559007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:48:37.559007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:49:43.690136+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"181'7","reported_seq":121,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-20T22:48:53.215127+0000","last_change":"2024-09-20T22:48:41.049491+0000","last_active":"2024-09-20T22:48:53.215127+0000","last_peered":"2024-09-20T22:48:53.215127+0000","last_clean":"2024-09-20T22:48:53.215127+0000","last_became_active":"2024-09-20T22:48:05.249763+0000","last_became_peered":"2024-09-20T22:48:05.249763+0000","last_unstale":"2024-09-20T22:48:53.215127+0000","last_undegraded":"2024-09-20T22:48:53.215127+0000","last_fullsized":"2024-09-20T22:48:53.215127+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"181'7","last_scrub_stamp":"2024-09-20T22:48:41.049381+0000","last_deep_scrub":"181'7","last_deep_scrub_stamp":"2024-09-20T22:48:41.049381+0000","last_clean_scrub_stamp":"2024-09-20T22:48:41.049381+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:03.477658+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":6,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":826,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-20T22:48:53.212444+0000","last_change":"2024-09-20T22:48:05.238275+0000","last_active":"2024-09-20T22:48:53.212444+0000","last_peered":"2024-09-20T22:48:53.212444+0000","last_clean":"2024-09-20T22:48:53.212444+0000","last_became_active":"2024-09-20T22:48:05.237791+0000","last_became_peered":"2024-09-20T22:48:05.237791+0000","last_unstale":"2024-09-20T22:48:53.212444+0000","last_undegraded":"2024-09-20T22:48:53.212444+0000","last_fullsized":"2024-09-20T22:48:53.212444+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:47:40.361958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:47:40.361958+0000","last_clean_scrub_stamp":"2024-09-20T22:47:40.361958+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:56.944095+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":789,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-20T22:48:53.227254+0000","last_change":"2024-09-20T22:46:06.068842+0000","last_active":"2024-09-20T22:48:53.227254+0000","last_peered":"2024-09-20T22:48:53.227254+0000","last_clean":"2024-09-20T22:48:53.227254+0000","last_became_active":"2024-09-20T22:41:44.197927+0000","last_became_peered":"2024-09-20T22:41:44.197927+0000","last_unstale":"2024-09-20T22:48:53.227254+0000","last_undegraded":"2024-09-20T22:48:53.227254+0000","last_fullsized":"2024-09-20T22:48:53.227254+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:46:06.068745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:46:06.068745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":17,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-20T22:48:53.210774+0000","last_change":"2024-09-20T22:48:50.224321+0000","last_active":"2024-09-20T22:48:53.210774+0000","last_peered":"2024-09-20T22:48:53.210774+0000","last_clean":"2024-09-20T22:48:53.210774+0000","last_became_active":"2024-09-20T22:48:50.223895+0000","last_became_peered":"2024-09-20T22:48:50.223895+0000","last_unstale":"2024-09-20T22:48:53.210774+0000","last_undegraded":"2024-09-20T22:48:53.210774+0000","last_fullsized":"2024-09-20T22:48:53.210774+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:47:21.819572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:47:21.819572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:56.612345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":145,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-20T22:48:53.210445+0000","last_change":"2024-09-20T22:48:05.234091+0000","last_active":"2024-09-20T22:48:53.210445+0000","last_peered":"2024-09-20T22:48:53.210445+0000","last_clean":"2024-09-20T22:48:53.210445+0000","last_became_active":"2024-09-20T22:48:05.233687+0000","last_became_peered":"2024-09-20T22:48:05.233687+0000","last_unstale":"2024-09-20T22:48:53.210445+0000","last_undegraded":"2024-09-20T22:48:53.210445+0000","last_fullsized":"2024-09-20T22:48:53.210445+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:47:13.413912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:47:13.413912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:56.612273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"113'22","reported_seq":843,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-20T22:48:53.223123+0000","last_change":"2024-09-20T22:48:21.243488+0000","last_active":"2024-09-20T22:48:53.223123+0000","last_peered":"2024-09-20T22:48:53.223123+0000","last_clean":"2024-09-20T22:48:53.223123+0000","last_became_active":"2024-09-20T22:41:40.170131+0000","last_became_peered":"2024-09-20T22:41:40.170131+0000","last_unstale":"2024-09-20T22:48:53.223123+0000","last_undegraded":"2024-09-20T22:48:53.223123+0000","last_fullsized":"2024-09-20T22:48:53.223123+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'22","last_scrub_stamp":"2024-09-20T22:48:21.243355+0000","last_deep_scrub":"113'22","last_deep_scrub_stamp":"2024-09-20T22:43:12.191878+0000","last_clean_scrub_stamp":"2024-09-20T22:48:21.243355+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:49:37.343367+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":816,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-20T22:48:53.212746+0000","last_change":"2024-09-20T22:48:05.249765+0000","last_active":"2024-09-20T22:48:53.212746+0000","last_peered":"2024-09-20T22:48:53.212746+0000","last_clean":"2024-09-20T22:48:53.212746+0000","last_became_active":"2024-09-20T22:48:05.249417+0000","last_became_peered":"2024-09-20T22:48:05.249417+0000","last_unstale":"2024-09-20T22:48:53.212746+0000","last_undegraded":"2024-09-20T22:48:53.212746+0000","last_fullsized":"2024-09-20T22:48:53.212746+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:46:49.872479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:46:49.872479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:57.165199+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":781,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-20T22:48:53.215460+0000","last_change":"2024-09-20T22:45:25.002270+0000","last_active":"2024-09-20T22:48:53.215460+0000","last_peered":"2024-09-20T22:48:53.215460+0000","last_clean":"2024-09-20T22:48:53.215460+0000","last_became_active":"2024-09-20T22:41:44.832131+0000","last_became_peered":"2024-09-20T22:41:44.832131+0000","last_unstale":"2024-09-20T22:48:53.215460+0000","last_undegraded":"2024-09-20T22:48:53.215460+0000","last_fullsized":"2024-09-20T22:48:53.215460+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:45:25.002199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:45:25.002199+0000","last_clean_scrub_stamp":"2024-09-20T22:45:25.002199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:55.172286+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":115,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-20T22:48:53.214739+0000","last_change":"2024-09-20T22:48:24.986582+0000","last_active":"2024-09-20T22:48:53.214739+0000","last_peered":"2024-09-20T22:48:53.214739+0000","last_clean":"2024-09-20T22:48:53.214739+0000","last_became_active":"2024-09-20T22:48:05.250581+0000","last_became_peered":"2024-09-20T22:48:05.250581+0000","last_unstale":"2024-09-20T22:48:53.214739+0000","last_undegraded":"2024-09-20T22:48:53.214739+0000","last_fullsized":"2024-09-20T22:48:53.214739+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:48:24.986484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:48:24.986484+0000","last_clean_scrub_stamp":"2024-09-20T22:48:24.986484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:49:46.510379+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"74'12","reported_seq":865,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-20T22:48:53.213847+0000","last_change":"2024-09-20T22:48:05.258994+0000","last_active":"2024-09-20T22:48:53.213847+0000","last_peered":"2024-09-20T22:48:53.213847+0000","last_clean":"2024-09-20T22:48:53.213847+0000","last_became_active":"2024-09-20T22:48:05.233983+0000","last_became_peered":"2024-09-20T22:48:05.233983+0000","last_unstale":"2024-09-20T22:48:53.213847+0000","last_undegraded":"2024-09-20T22:48:53.213847+0000","last_fullsized":"2024-09-20T22:48:53.213847+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-09-20T22:44:42.140915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:44:42.140915+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:55.172140+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":115,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-20T22:48:53.215459+0000","last_change":"2024-09-20T22:48:23.940233+0000","last_active":"2024-09-20T22:48:53.215459+0000","last_peered":"2024-09-20T22:48:53.215459+0000","last_clean":"2024-09-20T22:48:53.215459+0000","last_became_active":"2024-09-20T22:48:05.250319+0000","last_became_peered":"2024-09-20T22:48:05.250319+0000","last_unstale":"2024-09-20T22:48:53.215459+0000","last_undegraded":"2024-09-20T22:48:53.215459+0000","last_fullsized":"2024-09-20T22:48:53.215459+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:48:23.940157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:48:23.940157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:49:46.541532+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"154'2","reported_seq":797,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-20T22:48:53.212955+0000","last_change":"2024-09-20T22:48:39.357067+0000","last_active":"2024-09-20T22:48:53.212955+0000","last_peered":"2024-09-20T22:48:53.212955+0000","last_clean":"2024-09-20T22:48:53.212955+0000","last_became_active":"2024-09-20T22:41:44.199822+0000","last_became_peered":"2024-09-20T22:41:44.199822+0000","last_unstale":"2024-09-20T22:48:53.212955+0000","last_undegraded":"2024-09-20T22:48:53.212955+0000","last_fullsized":"2024-09-20T22:48:53.212955+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"154'2","last_scrub_stamp":"2024-09-20T22:48:39.356937+0000","last_deep_scrub":"154'2","last_deep_scrub_stamp":"2024-09-20T22:43:54.134066+0000","last_clean_scrub_stamp":"2024-09-20T22:48:39.356937+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:49:48.454806+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"67'1","reported_seq":829,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-20T22:48:53.209716+0000","last_change":"2024-09-20T22:48:42.188853+0000","last_active":"2024-09-20T22:48:53.209716+0000","last_peered":"2024-09-20T22:48:53.209716+0000","last_clean":"2024-09-20T22:48:53.209716+0000","last_became_active":"2024-09-20T22:48:05.249578+0000","last_became_peered":"2024-09-20T22:48:05.249578+0000","last_unstale":"2024-09-20T22:48:53.209716+0000","last_undegraded":"2024-09-20T22:48:53.209716+0000","last_fullsized":"2024-09-20T22:48:53.209716+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-09-20T22:48:42.188772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:48:42.188772+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:09.372929+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"151'15","reported_seq":159,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-20T22:48:53.212233+0000","last_change":"2024-09-20T22:48:37.528315+0000","last_active":"2024-09-20T22:48:53.212233+0000","last_peered":"2024-09-20T22:48:53.212233+0000","last_clean":"2024-09-20T22:48:53.212233+0000","last_became_active":"2024-09-20T22:47:49.232181+0000","last_became_peered":"2024-09-20T22:47:49.232181+0000","last_unstale":"2024-09-20T22:48:53.212233+0000","last_undegraded":"2024-09-20T22:48:53.212233+0000","last_fullsized":"2024-09-20T22:48:53.212233+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"151'15","last_scrub_stamp":"2024-09-20T22:48:37.528207+0000","last_deep_scrub":"151'15","last_deep_scrub_stamp":"2024-09-20T22:43:50.715761+0000","last_clean_scrub_stamp":"2024-09-20T22:48:37.528207+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:49:47.363120+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"408'10","reported_seq":923,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-20T22:48:53.210107+0000","last_change":"2024-09-20T22:48:18.083640+0000","last_active":"2024-09-20T22:48:53.210107+0000","last_peered":"2024-09-20T22:48:53.210107+0000","last_clean":"2024-09-20T22:48:53.210107+0000","last_became_active":"2024-09-20T22:48:05.244270+0000","last_became_peered":"2024-09-20T22:48:05.244270+0000","last_unstale":"2024-09-20T22:48:53.210107+0000","last_undegraded":"2024-09-20T22:48:53.210107+0000","last_fullsized":"2024-09-20T22:48:53.210107+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"408'10","last_scrub_stamp":"2024-09-20T22:48:18.083560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:48:18.083560+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:49:38.952569+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.c","version":"453'1145","reported_seq":2944,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-20T22:48:53.221684+0000","last_change":"2024-09-20T22:48:53.221900+0000","last_active":"2024-09-20T22:48:53.221684+0000","last_peered":"2024-09-20T22:48:53.221684+0000","last_clean":"2024-09-20T22:48:53.221684+0000","last_became_active":"2024-09-20T22:42:52.404027+0000","last_became_peered":"2024-09-20T22:42:52.404027+0000","last_unstale":"2024-09-20T22:48:53.221684+0000","last_undegraded":"2024-09-20T22:48:53.221684+0000","last_fullsized":"2024-09-20T22:48:53.221684+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"438'1046","last_scrub_stamp":"2024-09-20T22:48:38.253809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:48:38.253809+0000","objects_scrubbed":22,"log_size":1145,"log_dups_size":0,"ondisk_log_size":1145,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:49:43.681512+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0.012053837,"stat_sum":{"num_bytes":49302377,"num_objects":22,"num_object_clones":15,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":2,"num_read":550,"num_read_kb":539379,"num_write":388,"num_write_kb":151040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"96","length":"1"},{"start":"9a","length":"1"},{"start":"9d","length":"1"},{"start":"9f","length":"2"},{"start":"a3","length":"2"},{"start":"a8","length":"8"},{"start":"b3","length":"1"}]},{"pgid":"1.a","version":"77'128","reported_seq":1068,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-20T22:48:53.227500+0000","last_change":"2024-09-20T22:48:24.566023+0000","last_active":"2024-09-20T22:48:53.227500+0000","last_peered":"2024-09-20T22:48:53.227500+0000","last_clean":"2024-09-20T22:48:53.227500+0000","last_became_active":"2024-09-20T22:42:52.407791+0000","last_became_peered":"2024-09-20T22:42:52.407791+0000","last_unstale":"2024-09-20T22:48:53.227500+0000","last_undegraded":"2024-09-20T22:48:53.227500+0000","last_fullsized":"2024-09-20T22:48:53.227500+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:48:24.565920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:48:24.565920+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:49:47.507188+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"74'15","reported_seq":874,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-20T22:48:53.213083+0000","last_change":"2024-09-20T22:48:21.288601+0000","last_active":"2024-09-20T22:48:53.213083+0000","last_peered":"2024-09-20T22:48:53.213083+0000","last_clean":"2024-09-20T22:48:53.213083+0000","last_became_active":"2024-09-20T22:48:05.250308+0000","last_became_peered":"2024-09-20T22:48:05.250308+0000","last_unstale":"2024-09-20T22:48:53.213083+0000","last_undegraded":"2024-09-20T22:48:53.213083+0000","last_fullsized":"2024-09-20T22:48:53.213083+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-09-20T22:48:21.288511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:48:21.288511+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:49:49.906057+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":793,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-20T22:48:53.214073+0000","last_change":"2024-09-20T22:45:19.027462+0000","last_active":"2024-09-20T22:48:53.214073+0000","last_peered":"2024-09-20T22:48:53.214073+0000","last_clean":"2024-09-20T22:48:53.214073+0000","last_became_active":"2024-09-20T22:41:38.156749+0000","last_became_peered":"2024-09-20T22:41:38.156749+0000","last_unstale":"2024-09-20T22:48:53.214073+0000","last_undegraded":"2024-09-20T22:48:53.214073+0000","last_fullsized":"2024-09-20T22:48:53.214073+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:45:19.027393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:45:19.027393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:55.172201+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"408'12","reported_seq":121,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-20T22:48:53.216082+0000","last_change":"2024-09-20T22:48:05.251167+0000","last_active":"2024-09-20T22:48:53.216082+0000","last_peered":"2024-09-20T22:48:53.216082+0000","last_clean":"2024-09-20T22:48:53.216082+0000","last_became_active":"2024-09-20T22:48:05.242914+0000","last_became_peered":"2024-09-20T22:48:05.242914+0000","last_unstale":"2024-09-20T22:48:53.216082+0000","last_undegraded":"2024-09-20T22:48:53.216082+0000","last_fullsized":"2024-09-20T22:48:53.216082+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"264'10","last_scrub_stamp":"2024-09-20T22:47:20.300787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:47:20.300787+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:58.056148+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":802,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-20T22:48:53.227534+0000","last_change":"2024-09-20T22:47:49.227726+0000","last_active":"2024-09-20T22:48:53.227534+0000","last_peered":"2024-09-20T22:48:53.227534+0000","last_clean":"2024-09-20T22:48:53.227534+0000","last_became_active":"2024-09-20T22:47:49.227039+0000","last_became_peered":"2024-09-20T22:47:49.227039+0000","last_unstale":"2024-09-20T22:48:53.227534+0000","last_undegraded":"2024-09-20T22:48:53.227534+0000","last_fullsized":"2024-09-20T22:48:53.227534+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:46:26.147167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:46:26.147167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3","version":"135'8","reported_seq":1781,"reported_epoch":453,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-20T22:48:53.216238+0000","last_change":"2024-09-20T22:48:50.276978+0000","last_active":"2024-09-20T22:48:53.216238+0000","last_peered":"2024-09-20T22:48:53.216238+0000","last_clean":"2024-09-20T22:48:53.216238+0000","last_became_active":"2024-09-20T22:48:05.234893+0000","last_became_peered":"2024-09-20T22:48:05.234893+0000","last_unstale":"2024-09-20T22:48:53.216238+0000","last_undegraded":"2024-09-20T22:48:53.216238+0000","last_fullsized":"2024-09-20T22:48:53.216238+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'8","last_scrub_stamp":"2024-09-20T22:47:23.137750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:47:23.137750+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.060754071,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"96","length":"1"},{"start":"9a","length":"1"},{"start":"9d","length":"1"},{"start":"9f","length":"2"},{"start":"a3","length":"2"},{"start":"a8","length":"1"},{"start":"aa","length":"4"},{"start":"af","length":"1"}]},{"pgid":"1.5","version":"77'128","reported_seq":965,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-20T22:48:53.215932+0000","last_change":"2024-09-20T22:48:08.207463+0000","last_active":"2024-09-20T22:48:53.215932+0000","last_peered":"2024-09-20T22:48:53.215932+0000","last_clean":"2024-09-20T22:48:53.215932+0000","last_became_active":"2024-09-20T22:48:08.207018+0000","last_became_peered":"2024-09-20T22:48:08.207018+0000","last_unstale":"2024-09-20T22:48:53.215932+0000","last_undegraded":"2024-09-20T22:48:53.215932+0000","last_fullsized":"2024-09-20T22:48:53.215932+0000","mapping_epoch":407,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":408,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:48:02.056211+0000","last_deep_scrub":"77'128","last_deep_scrub_stamp":"2024-09-20T22:45:41.608075+0000","last_clean_scrub_stamp":"2024-09-20T22:48:02.056211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:49:15.339453+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"74'18","reported_seq":877,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-20T22:48:53.214309+0000","last_change":"2024-09-20T22:48:43.146697+0000","last_active":"2024-09-20T22:48:53.214309+0000","last_peered":"2024-09-20T22:48:53.214309+0000","last_clean":"2024-09-20T22:48:53.214309+0000","last_became_active":"2024-09-20T22:47:49.217605+0000","last_became_peered":"2024-09-20T22:47:49.217605+0000","last_unstale":"2024-09-20T22:48:53.214309+0000","last_undegraded":"2024-09-20T22:48:53.214309+0000","last_fullsized":"2024-09-20T22:48:53.214309+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'18","last_scrub_stamp":"2024-09-20T22:48:43.146602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:48:43.146602+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:00.379899+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":813,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-20T22:48:53.215845+0000","last_change":"2024-09-20T22:47:49.227431+0000","last_active":"2024-09-20T22:48:53.215845+0000","last_peered":"2024-09-20T22:48:53.215845+0000","last_clean":"2024-09-20T22:48:53.215845+0000","last_became_active":"2024-09-20T22:47:49.227069+0000","last_became_peered":"2024-09-20T22:47:49.227069+0000","last_unstale":"2024-09-20T22:48:53.215845+0000","last_undegraded":"2024-09-20T22:48:53.215845+0000","last_fullsized":"2024-09-20T22:48:53.215845+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:47:39.982616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:47:39.982616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:59.960690+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":107,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-20T22:48:53.214919+0000","last_change":"2024-09-20T22:48:05.231356+0000","last_active":"2024-09-20T22:48:53.214919+0000","last_peered":"2024-09-20T22:48:53.214919+0000","last_clean":"2024-09-20T22:48:53.214919+0000","last_became_active":"2024-09-20T22:48:05.230003+0000","last_became_peered":"2024-09-20T22:48:05.230003+0000","last_unstale":"2024-09-20T22:48:53.214919+0000","last_undegraded":"2024-09-20T22:48:53.214919+0000","last_fullsized":"2024-09-20T22:48:53.214919+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:45:56.199432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:07.947945+0000","last_clean_scrub_stamp":"2024-09-20T22:45:56.199432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:58.055982+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":107,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-20T22:48:53.214746+0000","last_change":"2024-09-20T22:48:05.240266+0000","last_active":"2024-09-20T22:48:53.214746+0000","last_peered":"2024-09-20T22:48:53.214746+0000","last_clean":"2024-09-20T22:48:53.214746+0000","last_became_active":"2024-09-20T22:48:05.239917+0000","last_became_peered":"2024-09-20T22:48:05.239917+0000","last_unstale":"2024-09-20T22:48:53.214746+0000","last_undegraded":"2024-09-20T22:48:53.214746+0000","last_fullsized":"2024-09-20T22:48:53.214746+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:47:18.179575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:53.036601+0000","last_clean_scrub_stamp":"2024-09-20T22:47:18.179575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:58.056103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2","version":"453'220","reported_seq":1572,"reported_epoch":453,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T22:48:53.278057+0000","last_change":"2024-09-20T22:48:53.278057+0000","last_active":"2024-09-20T22:48:53.278057+0000","last_peered":"2024-09-20T22:48:53.278057+0000","last_clean":"2024-09-20T22:48:53.278057+0000","last_became_active":"2024-09-20T22:44:09.177625+0000","last_became_peered":"2024-09-20T22:44:09.177625+0000","last_unstale":"2024-09-20T22:48:53.278057+0000","last_undegraded":"2024-09-20T22:48:53.278057+0000","last_fullsized":"2024-09-20T22:48:53.278057+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"282'120","last_scrub_stamp":"2024-09-20T22:46:22.099642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:46:22.099642+0000","objects_scrubbed":1,"log_size":220,"log_dups_size":0,"ondisk_log_size":220,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":5,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.049806796,"stat_sum":{"num_bytes":8536040,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":101,"num_read_kb":110040,"num_write":45,"num_write_kb":13519,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"96","length":"1"},{"start":"9a","length":"1"},{"start":"9d","length":"1"},{"start":"9f","length":"2"},{"start":"a3","length":"2"},{"start":"a8","length":"1"},{"start":"aa","length":"3"}]},{"pgid":"1.4","version":"77'128","reported_seq":1092,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-20T22:48:53.214840+0000","last_change":"2024-09-20T22:46:15.181632+0000","last_active":"2024-09-20T22:48:53.214840+0000","last_peered":"2024-09-20T22:48:53.214840+0000","last_clean":"2024-09-20T22:48:53.214840+0000","last_became_active":"2024-09-20T22:46:15.181305+0000","last_became_peered":"2024-09-20T22:46:15.181305+0000","last_unstale":"2024-09-20T22:48:53.214840+0000","last_undegraded":"2024-09-20T22:48:53.214840+0000","last_fullsized":"2024-09-20T22:48:53.214840+0000","mapping_epoch":94,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:45:54.131305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:45:54.131305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":59,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:55.172356+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"74'14","reported_seq":123,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-20T22:48:53.214825+0000","last_change":"2024-09-20T22:48:05.262315+0000","last_active":"2024-09-20T22:48:53.214825+0000","last_peered":"2024-09-20T22:48:53.214825+0000","last_clean":"2024-09-20T22:48:53.214825+0000","last_became_active":"2024-09-20T22:48:05.235812+0000","last_became_peered":"2024-09-20T22:48:05.235812+0000","last_unstale":"2024-09-20T22:48:53.214825+0000","last_undegraded":"2024-09-20T22:48:53.214825+0000","last_fullsized":"2024-09-20T22:48:53.214825+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'14","last_scrub_stamp":"2024-09-20T22:46:48.202311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:46:48.202311+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:58.056074+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"67'1","reported_seq":161,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-20T22:48:53.211067+0000","last_change":"2024-09-20T22:48:36.570753+0000","last_active":"2024-09-20T22:48:53.211067+0000","last_peered":"2024-09-20T22:48:53.211067+0000","last_clean":"2024-09-20T22:48:53.211067+0000","last_became_active":"2024-09-20T22:48:05.237425+0000","last_became_peered":"2024-09-20T22:48:05.237425+0000","last_unstale":"2024-09-20T22:48:53.211067+0000","last_undegraded":"2024-09-20T22:48:53.211067+0000","last_fullsized":"2024-09-20T22:48:53.211067+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-09-20T22:48:36.570672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:48:36.570672+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:49:55.065551+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"408'12","reported_seq":921,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-20T22:48:53.214648+0000","last_change":"2024-09-20T22:48:05.247422+0000","last_active":"2024-09-20T22:48:53.214648+0000","last_peered":"2024-09-20T22:48:53.214648+0000","last_clean":"2024-09-20T22:48:53.214648+0000","last_became_active":"2024-09-20T22:48:05.234607+0000","last_became_peered":"2024-09-20T22:48:05.234607+0000","last_unstale":"2024-09-20T22:48:53.214648+0000","last_undegraded":"2024-09-20T22:48:53.214648+0000","last_fullsized":"2024-09-20T22:48:53.214648+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"159'9","last_scrub_stamp":"2024-09-20T22:45:23.017605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:45:23.017605+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:55.172231+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":788,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-20T22:48:53.212440+0000","last_change":"2024-09-20T22:47:16.381511+0000","last_active":"2024-09-20T22:48:53.212440+0000","last_peered":"2024-09-20T22:48:53.212440+0000","last_clean":"2024-09-20T22:48:53.212440+0000","last_became_active":"2024-09-20T22:41:44.831901+0000","last_became_peered":"2024-09-20T22:41:44.831901+0000","last_unstale":"2024-09-20T22:48:53.212440+0000","last_undegraded":"2024-09-20T22:48:53.212440+0000","last_fullsized":"2024-09-20T22:48:53.212440+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:47:16.381390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:47:16.381390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:54.196311+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1","version":"450'250","reported_seq":1987,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-20T22:48:53.215214+0000","last_change":"2024-09-20T22:48:53.215517+0000","last_active":"2024-09-20T22:48:53.215214+0000","last_peered":"2024-09-20T22:48:53.215214+0000","last_clean":"2024-09-20T22:48:53.215214+0000","last_became_active":"2024-09-20T22:48:08.208698+0000","last_became_peered":"2024-09-20T22:48:08.208698+0000","last_unstale":"2024-09-20T22:48:53.215214+0000","last_undegraded":"2024-09-20T22:48:53.215214+0000","last_fullsized":"2024-09-20T22:48:53.215214+0000","mapping_epoch":407,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":408,"parent":"0.0","parent_split_bits":0,"last_scrub":"420'231","last_scrub_stamp":"2024-09-20T22:48:22.915993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:48:22.915993+0000","objects_scrubbed":6,"log_size":250,"log_dups_size":0,"ondisk_log_size":250,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:49:46.612473+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00057991900000000003,"stat_sum":{"num_bytes":11160046,"num_objects":5,"num_object_clones":3,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":62,"num_read_kb":78459,"num_write":51,"num_write_kb":36252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":8732217,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"96","length":"1"},{"start":"9a","length":"1"},{"start":"9d","length":"1"},{"start":"9f","length":"2"},{"start":"a3","length":"2"},{"start":"a8","length":"8"},{"start":"b3","length":"1"}]},{"pgid":"1.7","version":"77'128","reported_seq":1097,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-20T22:48:53.209562+0000","last_change":"2024-09-20T22:48:03.022941+0000","last_active":"2024-09-20T22:48:53.209562+0000","last_peered":"2024-09-20T22:48:53.209562+0000","last_clean":"2024-09-20T22:48:53.209562+0000","last_became_active":"2024-09-20T22:46:03.169596+0000","last_became_peered":"2024-09-20T22:46:03.169596+0000","last_unstale":"2024-09-20T22:48:53.209562+0000","last_undegraded":"2024-09-20T22:48:53.209562+0000","last_fullsized":"2024-09-20T22:48:53.209562+0000","mapping_epoch":94,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:48:03.022838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:48:03.022838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:49:29.905235+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"74'10","reported_seq":892,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-20T22:48:53.209625+0000","last_change":"2024-09-20T22:48:24.335766+0000","last_active":"2024-09-20T22:48:53.209625+0000","last_peered":"2024-09-20T22:48:53.209625+0000","last_clean":"2024-09-20T22:48:53.209625+0000","last_became_active":"2024-09-20T22:48:05.251200+0000","last_became_peered":"2024-09-20T22:48:05.251200+0000","last_unstale":"2024-09-20T22:48:53.209625+0000","last_undegraded":"2024-09-20T22:48:53.209625+0000","last_fullsized":"2024-09-20T22:48:53.209625+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-09-20T22:48:24.335674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:48:24.335674+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:49:30.306070+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":810,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-20T22:48:53.227678+0000","last_change":"2024-09-20T22:48:04.520131+0000","last_active":"2024-09-20T22:48:53.227678+0000","last_peered":"2024-09-20T22:48:53.227678+0000","last_clean":"2024-09-20T22:48:53.227678+0000","last_became_active":"2024-09-20T22:41:38.155629+0000","last_became_peered":"2024-09-20T22:41:38.155629+0000","last_unstale":"2024-09-20T22:48:53.227678+0000","last_undegraded":"2024-09-20T22:48:53.227678+0000","last_fullsized":"2024-09-20T22:48:53.227678+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:48:04.520053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:48:04.520053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:49:32.650911+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"389'11","reported_seq":165,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-20T22:48:53.210392+0000","last_change":"2024-09-20T22:48:44.601907+0000","last_active":"2024-09-20T22:48:53.210392+0000","last_peered":"2024-09-20T22:48:53.210392+0000","last_clean":"2024-09-20T22:48:53.210392+0000","last_became_active":"2024-09-20T22:47:49.227264+0000","last_became_peered":"2024-09-20T22:47:49.227264+0000","last_unstale":"2024-09-20T22:48:53.210392+0000","last_undegraded":"2024-09-20T22:48:53.210392+0000","last_fullsized":"2024-09-20T22:48:53.210392+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'11","last_scrub_stamp":"2024-09-20T22:48:44.601816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:48:44.601816+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:10.964457+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":139,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-20T22:48:53.208700+0000","last_change":"2024-09-20T22:47:49.232739+0000","last_active":"2024-09-20T22:48:53.208700+0000","last_peered":"2024-09-20T22:48:53.208700+0000","last_clean":"2024-09-20T22:48:53.208700+0000","last_became_active":"2024-09-20T22:47:49.232380+0000","last_became_peered":"2024-09-20T22:47:49.232380+0000","last_unstale":"2024-09-20T22:48:53.208700+0000","last_undegraded":"2024-09-20T22:48:53.208700+0000","last_fullsized":"2024-09-20T22:48:53.208700+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:46:44.117761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:46:44.117761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:56.612244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.0","version":"451'224","reported_seq":2023,"reported_epoch":453,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T22:48:53.228446+0000","last_change":"2024-09-20T22:48:52.264797+0000","last_active":"2024-09-20T22:48:53.228446+0000","last_peered":"2024-09-20T22:48:53.228446+0000","last_clean":"2024-09-20T22:48:53.228446+0000","last_became_active":"2024-09-20T22:43:37.136845+0000","last_became_peered":"2024-09-20T22:43:37.136845+0000","last_unstale":"2024-09-20T22:48:53.228446+0000","last_undegraded":"2024-09-20T22:48:53.228446+0000","last_fullsized":"2024-09-20T22:48:53.228446+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"280'122","last_scrub_stamp":"2024-09-20T22:46:03.006423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:46:03.006423+0000","objects_scrubbed":1,"log_size":224,"log_dups_size":0,"ondisk_log_size":224,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.056735831,"stat_sum":{"num_bytes":13860427,"num_objects":7,"num_object_clones":5,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":82,"num_read_kb":83295,"num_write":51,"num_write_kb":15317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"9a","length":"1"},{"start":"9d","length":"1"},{"start":"9f","length":"2"},{"start":"a3","length":"2"},{"start":"a8","length":"1"},{"start":"aa","length":"4"},{"start":"af","length":"1"}]},{"pgid":"1.6","version":"77'128","reported_seq":1119,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-20T22:48:53.209533+0000","last_change":"2024-09-20T22:48:02.056439+0000","last_active":"2024-09-20T22:48:53.209533+0000","last_peered":"2024-09-20T22:48:53.209533+0000","last_clean":"2024-09-20T22:48:53.209533+0000","last_became_active":"2024-09-20T22:47:52.212190+0000","last_became_peered":"2024-09-20T22:47:52.212190+0000","last_unstale":"2024-09-20T22:48:53.209533+0000","last_undegraded":"2024-09-20T22:48:53.209533+0000","last_fullsized":"2024-09-20T22:48:53.209533+0000","mapping_epoch":391,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:48:02.056333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:48:02.056333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:49:14.033978+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"74'19","reported_seq":157,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-20T22:48:53.210201+0000","last_change":"2024-09-20T22:47:49.249445+0000","last_active":"2024-09-20T22:48:53.210201+0000","last_peered":"2024-09-20T22:48:53.210201+0000","last_clean":"2024-09-20T22:48:53.210201+0000","last_became_active":"2024-09-20T22:47:49.222009+0000","last_became_peered":"2024-09-20T22:47:49.222009+0000","last_unstale":"2024-09-20T22:48:53.210201+0000","last_undegraded":"2024-09-20T22:48:53.210201+0000","last_fullsized":"2024-09-20T22:48:53.210201+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'19","last_scrub_stamp":"2024-09-20T22:47:13.403224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:47:13.403224+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:56.612314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":38,"num_read_kb":24,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":809,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-20T22:48:53.212084+0000","last_change":"2024-09-20T22:48:23.320100+0000","last_active":"2024-09-20T22:48:53.212084+0000","last_peered":"2024-09-20T22:48:53.212084+0000","last_clean":"2024-09-20T22:48:53.212084+0000","last_became_active":"2024-09-20T22:41:38.160832+0000","last_became_peered":"2024-09-20T22:41:38.160832+0000","last_unstale":"2024-09-20T22:48:53.212084+0000","last_undegraded":"2024-09-20T22:48:53.212084+0000","last_fullsized":"2024-09-20T22:48:53.212084+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:48:23.320031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:48:23.320031+0000","last_clean_scrub_stamp":"2024-09-20T22:48:23.320031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:49:24.057655+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":805,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-20T22:48:53.209124+0000","last_change":"2024-09-20T22:47:49.214380+0000","last_active":"2024-09-20T22:48:53.209124+0000","last_peered":"2024-09-20T22:48:53.209124+0000","last_clean":"2024-09-20T22:48:53.209124+0000","last_became_active":"2024-09-20T22:47:49.214055+0000","last_became_peered":"2024-09-20T22:47:49.214055+0000","last_unstale":"2024-09-20T22:48:53.209124+0000","last_undegraded":"2024-09-20T22:48:53.209124+0000","last_fullsized":"2024-09-20T22:48:53.209124+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:46:43.827680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:46:43.827680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":797,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-20T22:48:53.209500+0000","last_change":"2024-09-20T22:48:41.208933+0000","last_active":"2024-09-20T22:48:53.209500+0000","last_peered":"2024-09-20T22:48:53.209500+0000","last_clean":"2024-09-20T22:48:53.209500+0000","last_became_active":"2024-09-20T22:41:44.830538+0000","last_became_peered":"2024-09-20T22:41:44.830538+0000","last_unstale":"2024-09-20T22:48:53.209500+0000","last_undegraded":"2024-09-20T22:48:53.209500+0000","last_fullsized":"2024-09-20T22:48:53.209500+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:48:41.208853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:48:41.208853+0000","last_clean_scrub_stamp":"2024-09-20T22:48:41.208853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:02.984046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.7","version":"453'512","reported_seq":2431,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-20T22:48:53.218643+0000","last_change":"2024-09-20T22:48:53.218864+0000","last_active":"2024-09-20T22:48:53.218643+0000","last_peered":"2024-09-20T22:48:53.218643+0000","last_clean":"2024-09-20T22:48:53.218643+0000","last_became_active":"2024-09-20T22:48:43.224386+0000","last_became_peered":"2024-09-20T22:48:43.224386+0000","last_unstale":"2024-09-20T22:48:53.218643+0000","last_undegraded":"2024-09-20T22:48:53.218643+0000","last_fullsized":"2024-09-20T22:48:53.218643+0000","mapping_epoch":391,"log_start":"436'456","ondisk_log_start":"436'456","created":94,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"372'324","last_scrub_stamp":"2024-09-20T22:47:33.829464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:47:33.829464+0000","objects_scrubbed":7,"log_size":56,"log_dups_size":886,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0086589370000000002,"stat_sum":{"num_bytes":40547471,"num_objects":21,"num_object_clones":16,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":2,"num_read":291,"num_read_kb":315442,"num_write":208,"num_write_kb":71967,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":115,"num_bytes_recovered":51615167,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"96","length":"1"},{"start":"9a","length":"1"},{"start":"9d","length":"1"},{"start":"9f","length":"2"},{"start":"a3","length":"2"},{"start":"a8","length":"8"},{"start":"b3","length":"1"}]},{"pgid":"1.1","version":"77'128","reported_seq":1116,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-20T22:48:53.227087+0000","last_change":"2024-09-20T22:48:44.631654+0000","last_active":"2024-09-20T22:48:53.227087+0000","last_peered":"2024-09-20T22:48:53.227087+0000","last_clean":"2024-09-20T22:48:53.227087+0000","last_became_active":"2024-09-20T22:47:52.208102+0000","last_became_peered":"2024-09-20T22:47:52.208102+0000","last_unstale":"2024-09-20T22:48:53.227087+0000","last_undegraded":"2024-09-20T22:48:53.227087+0000","last_fullsized":"2024-09-20T22:48:53.227087+0000","mapping_epoch":391,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:48:44.631571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:48:44.631571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:02.077211+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"86'38","reported_seq":951,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-20T22:48:53.211977+0000","last_change":"2024-09-20T22:48:00.380100+0000","last_active":"2024-09-20T22:48:53.211977+0000","last_peered":"2024-09-20T22:48:53.211977+0000","last_clean":"2024-09-20T22:48:53.211977+0000","last_became_active":"2024-09-20T22:41:40.166958+0000","last_became_peered":"2024-09-20T22:41:40.166958+0000","last_unstale":"2024-09-20T22:48:53.211977+0000","last_undegraded":"2024-09-20T22:48:53.211977+0000","last_fullsized":"2024-09-20T22:48:53.211977+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'38","last_scrub_stamp":"2024-09-20T22:48:00.379987+0000","last_deep_scrub":"86'38","last_deep_scrub_stamp":"2024-09-20T22:42:43.935677+0000","last_clean_scrub_stamp":"2024-09-20T22:48:00.379987+0000","objects_scrubbed":10,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:49:30.129855+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":107,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-20T22:48:53.215034+0000","last_change":"2024-09-20T22:48:05.238748+0000","last_active":"2024-09-20T22:48:53.215034+0000","last_peered":"2024-09-20T22:48:53.215034+0000","last_clean":"2024-09-20T22:48:53.215034+0000","last_became_active":"2024-09-20T22:48:05.237756+0000","last_became_peered":"2024-09-20T22:48:05.237756+0000","last_unstale":"2024-09-20T22:48:53.215034+0000","last_undegraded":"2024-09-20T22:48:53.215034+0000","last_fullsized":"2024-09-20T22:48:53.215034+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:47:16.817468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:47:16.817468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:58.056177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"69'2","reported_seq":820,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-20T22:48:53.228229+0000","last_change":"2024-09-20T22:46:44.210668+0000","last_active":"2024-09-20T22:48:53.228229+0000","last_peered":"2024-09-20T22:48:53.228229+0000","last_clean":"2024-09-20T22:48:53.228229+0000","last_became_active":"2024-09-20T22:41:37.148479+0000","last_became_peered":"2024-09-20T22:41:37.148479+0000","last_unstale":"2024-09-20T22:48:53.228229+0000","last_undegraded":"2024-09-20T22:48:53.228229+0000","last_fullsized":"2024-09-20T22:48:53.228229+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2024-09-20T22:46:44.210559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:36.130684+0000","last_clean_scrub_stamp":"2024-09-20T22:46:44.210559+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00087280599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":145,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-20T22:48:53.209914+0000","last_change":"2024-09-20T22:48:05.236164+0000","last_active":"2024-09-20T22:48:53.209914+0000","last_peered":"2024-09-20T22:48:53.209914+0000","last_clean":"2024-09-20T22:48:53.209914+0000","last_became_active":"2024-09-20T22:48:05.233519+0000","last_became_peered":"2024-09-20T22:48:05.233519+0000","last_unstale":"2024-09-20T22:48:53.209914+0000","last_undegraded":"2024-09-20T22:48:53.209914+0000","last_fullsized":"2024-09-20T22:48:53.209914+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:46:06.104786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:46:06.104786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:56.612164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":801,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-20T22:48:53.215291+0000","last_change":"2024-09-20T22:48:05.234499+0000","last_active":"2024-09-20T22:48:53.215291+0000","last_peered":"2024-09-20T22:48:53.215291+0000","last_clean":"2024-09-20T22:48:53.215291+0000","last_became_active":"2024-09-20T22:48:05.234193+0000","last_became_peered":"2024-09-20T22:48:05.234193+0000","last_unstale":"2024-09-20T22:48:53.215291+0000","last_undegraded":"2024-09-20T22:48:53.215291+0000","last_fullsized":"2024-09-20T22:48:53.215291+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:45:58.212988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:45:58.212988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:55.172341+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"453'143","reported_seq":440,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-20T22:48:53.214171+0000","last_change":"2024-09-20T22:48:53.214780+0000","last_active":"2024-09-20T22:48:53.214171+0000","last_peered":"2024-09-20T22:48:53.214171+0000","last_clean":"2024-09-20T22:48:53.214171+0000","last_became_active":"2024-09-20T22:48:05.242705+0000","last_became_peered":"2024-09-20T22:48:05.242705+0000","last_unstale":"2024-09-20T22:48:53.214171+0000","last_undegraded":"2024-09-20T22:48:53.214171+0000","last_fullsized":"2024-09-20T22:48:53.214171+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"305'57","last_scrub_stamp":"2024-09-20T22:47:13.636468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:47:13.636468+0000","objects_scrubbed":1,"log_size":143,"log_dups_size":0,"ondisk_log_size":143,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0040323049999999999,"stat_sum":{"num_bytes":13630951,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":46,"num_read_kb":44975,"num_write":33,"num_write_kb":11784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":10294275,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"96","length":"1"},{"start":"9a","length":"1"},{"start":"9d","length":"1"},{"start":"9f","length":"2"},{"start":"a3","length":"2"},{"start":"a8","length":"8"},{"start":"b3","length":"1"}]},{"pgid":"1.2","version":"77'128","reported_seq":1176,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-20T22:48:53.212291+0000","last_change":"2024-09-20T22:48:08.203458+0000","last_active":"2024-09-20T22:48:53.212291+0000","last_peered":"2024-09-20T22:48:53.212291+0000","last_clean":"2024-09-20T22:48:53.212291+0000","last_became_active":"2024-09-20T22:48:08.203028+0000","last_became_peered":"2024-09-20T22:48:08.203028+0000","last_unstale":"2024-09-20T22:48:53.212291+0000","last_undegraded":"2024-09-20T22:48:53.212291+0000","last_fullsized":"2024-09-20T22:48:53.212291+0000","mapping_epoch":407,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":408,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:47:56.410772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:47:56.410772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:49:17.378865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"443'18","reported_seq":873,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-20T22:48:53.213178+0000","last_change":"2024-09-20T22:48:44.182598+0000","last_active":"2024-09-20T22:48:53.213178+0000","last_peered":"2024-09-20T22:48:53.213178+0000","last_clean":"2024-09-20T22:48:53.213178+0000","last_became_active":"2024-09-20T22:47:49.228702+0000","last_became_peered":"2024-09-20T22:47:49.228702+0000","last_unstale":"2024-09-20T22:48:53.213178+0000","last_undegraded":"2024-09-20T22:48:53.213178+0000","last_fullsized":"2024-09-20T22:48:53.213178+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"443'18","last_scrub_stamp":"2024-09-20T22:48:44.182517+0000","last_deep_scrub":"443'18","last_deep_scrub_stamp":"2024-09-20T22:48:44.182517+0000","last_clean_scrub_stamp":"2024-09-20T22:48:44.182517+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:49:53.005519+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":809,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-20T22:48:53.208807+0000","last_change":"2024-09-20T22:47:55.478609+0000","last_active":"2024-09-20T22:48:53.208807+0000","last_peered":"2024-09-20T22:48:53.208807+0000","last_clean":"2024-09-20T22:48:53.208807+0000","last_became_active":"2024-09-20T22:41:38.156837+0000","last_became_peered":"2024-09-20T22:41:38.156837+0000","last_unstale":"2024-09-20T22:48:53.208807+0000","last_undegraded":"2024-09-20T22:48:53.208807+0000","last_fullsized":"2024-09-20T22:48:53.208807+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:47:55.478501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:47:55.478501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:49:23.720355+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"380'2","reported_seq":846,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-20T22:48:53.222594+0000","last_change":"2024-09-20T22:47:49.233750+0000","last_active":"2024-09-20T22:48:53.222594+0000","last_peered":"2024-09-20T22:48:53.222594+0000","last_clean":"2024-09-20T22:48:53.222594+0000","last_became_active":"2024-09-20T22:47:49.222812+0000","last_became_peered":"2024-09-20T22:47:49.222812+0000","last_unstale":"2024-09-20T22:48:53.222594+0000","last_undegraded":"2024-09-20T22:48:53.222594+0000","last_fullsized":"2024-09-20T22:48:53.222594+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"380'2","last_scrub_stamp":"2024-09-20T22:47:40.983039+0000","last_deep_scrub":"380'2","last_deep_scrub_stamp":"2024-09-20T22:47:40.983039+0000","last_clean_scrub_stamp":"2024-09-20T22:47:40.983039+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:59.344462+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.001301459,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":800,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-20T22:48:53.224484+0000","last_change":"2024-09-20T22:47:22.881180+0000","last_active":"2024-09-20T22:48:53.224484+0000","last_peered":"2024-09-20T22:48:53.224484+0000","last_clean":"2024-09-20T22:48:53.224484+0000","last_became_active":"2024-09-20T22:41:42.185112+0000","last_became_peered":"2024-09-20T22:41:42.185112+0000","last_unstale":"2024-09-20T22:48:53.224484+0000","last_undegraded":"2024-09-20T22:48:53.224484+0000","last_fullsized":"2024-09-20T22:48:53.224484+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:47:22.881109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:47:22.881109+0000","last_clean_scrub_stamp":"2024-09-20T22:47:22.881109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:53.304589+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":788,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-20T22:48:53.213079+0000","last_change":"2024-09-20T22:46:27.448645+0000","last_active":"2024-09-20T22:48:53.213079+0000","last_peered":"2024-09-20T22:48:53.213079+0000","last_clean":"2024-09-20T22:48:53.213079+0000","last_became_active":"2024-09-20T22:41:44.830515+0000","last_became_peered":"2024-09-20T22:41:44.830515+0000","last_unstale":"2024-09-20T22:48:53.213079+0000","last_undegraded":"2024-09-20T22:48:53.213079+0000","last_fullsized":"2024-09-20T22:48:53.213079+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:46:27.448572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:46:27.448572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:54.196199+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"450'190","reported_seq":2038,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-20T22:48:53.213345+0000","last_change":"2024-09-20T22:48:53.214495+0000","last_active":"2024-09-20T22:48:53.213345+0000","last_peered":"2024-09-20T22:48:53.213345+0000","last_clean":"2024-09-20T22:48:53.213345+0000","last_became_active":"2024-09-20T22:47:52.206033+0000","last_became_peered":"2024-09-20T22:47:52.206033+0000","last_unstale":"2024-09-20T22:48:53.213345+0000","last_undegraded":"2024-09-20T22:48:53.213345+0000","last_fullsized":"2024-09-20T22:48:53.213345+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'168","last_scrub_stamp":"2024-09-20T22:48:22.483112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:48:22.483112+0000","objects_scrubbed":7,"log_size":190,"log_dups_size":0,"ondisk_log_size":190,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:49:41.739019+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0013677400000000001,"stat_sum":{"num_bytes":11338963,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":56,"num_read_kb":38714,"num_write":68,"num_write_kb":31788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":19673726,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"96","length":"1"},{"start":"9a","length":"1"},{"start":"9d","length":"1"},{"start":"9f","length":"2"},{"start":"a3","length":"2"},{"start":"a8","length":"8"},{"start":"b3","length":"1"}]},{"pgid":"1.3","version":"77'128","reported_seq":1124,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-20T22:48:53.209178+0000","last_change":"2024-09-20T22:48:37.305505+0000","last_active":"2024-09-20T22:48:53.209178+0000","last_peered":"2024-09-20T22:48:53.209178+0000","last_clean":"2024-09-20T22:48:53.209178+0000","last_became_active":"2024-09-20T22:48:08.208402+0000","last_became_peered":"2024-09-20T22:48:08.208402+0000","last_unstale":"2024-09-20T22:48:53.209178+0000","last_undegraded":"2024-09-20T22:48:53.209178+0000","last_fullsized":"2024-09-20T22:48:53.209178+0000","mapping_epoch":407,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":408,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:48:37.305426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:48:37.305426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:49:46.648477+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"74'12","reported_seq":153,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-20T22:48:53.212323+0000","last_change":"2024-09-20T22:47:49.246080+0000","last_active":"2024-09-20T22:48:53.212323+0000","last_peered":"2024-09-20T22:48:53.212323+0000","last_clean":"2024-09-20T22:48:53.212323+0000","last_became_active":"2024-09-20T22:47:49.224223+0000","last_became_peered":"2024-09-20T22:47:49.224223+0000","last_unstale":"2024-09-20T22:48:53.212323+0000","last_undegraded":"2024-09-20T22:48:53.212323+0000","last_fullsized":"2024-09-20T22:48:53.212323+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-09-20T22:47:39.293455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:47:39.293455+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:56.612329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":153,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-20T22:48:53.209490+0000","last_change":"2024-09-20T22:48:05.233412+0000","last_active":"2024-09-20T22:48:53.209490+0000","last_peered":"2024-09-20T22:48:53.209490+0000","last_clean":"2024-09-20T22:48:53.209490+0000","last_became_active":"2024-09-20T22:48:05.233093+0000","last_became_peered":"2024-09-20T22:48:05.233093+0000","last_unstale":"2024-09-20T22:48:53.209490+0000","last_undegraded":"2024-09-20T22:48:53.209490+0000","last_fullsized":"2024-09-20T22:48:53.209490+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:47:55.576301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:44:19.834641+0000","last_clean_scrub_stamp":"2024-09-20T22:47:55.576301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:49:20.279866+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"75'5","reported_seq":1267,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-20T22:48:53.212274+0000","last_change":"2024-09-20T22:48:44.254029+0000","last_active":"2024-09-20T22:48:53.212274+0000","last_peered":"2024-09-20T22:48:53.212274+0000","last_clean":"2024-09-20T22:48:53.212274+0000","last_became_active":"2024-09-20T22:47:49.214414+0000","last_became_peered":"2024-09-20T22:47:49.214414+0000","last_unstale":"2024-09-20T22:48:53.212274+0000","last_undegraded":"2024-09-20T22:48:53.212274+0000","last_fullsized":"2024-09-20T22:48:53.212274+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-09-20T22:48:44.253907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:36.130684+0000","last_clean_scrub_stamp":"2024-09-20T22:48:44.253907+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:49:45.116608+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00078631700000000003,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":426,"num_read_kb":421,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":814,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-20T22:48:53.227854+0000","last_change":"2024-09-20T22:48:34.535169+0000","last_active":"2024-09-20T22:48:53.227854+0000","last_peered":"2024-09-20T22:48:53.227854+0000","last_clean":"2024-09-20T22:48:53.227854+0000","last_became_active":"2024-09-20T22:47:49.227371+0000","last_became_peered":"2024-09-20T22:47:49.227371+0000","last_unstale":"2024-09-20T22:48:53.227854+0000","last_undegraded":"2024-09-20T22:48:53.227854+0000","last_fullsized":"2024-09-20T22:48:53.227854+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:48:34.535100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:48:34.535100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:49:46.167401+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":797,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-20T22:48:53.209137+0000","last_change":"2024-09-20T22:47:59.388634+0000","last_active":"2024-09-20T22:48:53.209137+0000","last_peered":"2024-09-20T22:48:53.209137+0000","last_clean":"2024-09-20T22:48:53.209137+0000","last_became_active":"2024-09-20T22:41:44.199046+0000","last_became_peered":"2024-09-20T22:41:44.199046+0000","last_unstale":"2024-09-20T22:48:53.209137+0000","last_undegraded":"2024-09-20T22:48:53.209137+0000","last_fullsized":"2024-09-20T22:48:53.209137+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:47:59.388571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:47:59.388571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:49:09.987768+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.6","version":"453'197","reported_seq":1986,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-20T22:48:53.221352+0000","last_change":"2024-09-20T22:48:53.221615+0000","last_active":"2024-09-20T22:48:53.221352+0000","last_peered":"2024-09-20T22:48:53.221352+0000","last_clean":"2024-09-20T22:48:53.221352+0000","last_became_active":"2024-09-20T22:48:47.234531+0000","last_became_peered":"2024-09-20T22:48:47.234531+0000","last_unstale":"2024-09-20T22:48:53.221352+0000","last_undegraded":"2024-09-20T22:48:53.221352+0000","last_fullsized":"2024-09-20T22:48:53.221352+0000","mapping_epoch":391,"log_start":"443'181","ondisk_log_start":"443'181","created":94,"last_epoch_clean":447,"parent":"0.0","parent_split_bits":0,"last_scrub":"440'177","last_scrub_stamp":"2024-09-20T22:48:40.602342+0000","last_deep_scrub":"440'177","last_deep_scrub_stamp":"2024-09-20T22:48:40.602342+0000","last_clean_scrub_stamp":"2024-09-20T22:48:40.602342+0000","objects_scrubbed":4,"log_size":16,"log_dups_size":314,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:49:54.268810+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0.010220480000000001,"stat_sum":{"num_bytes":10874428,"num_objects":7,"num_object_clones":5,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":102,"num_read_kb":107756,"num_write":68,"num_write_kb":21700,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":18662529,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"96","length":"1"},{"start":"9a","length":"1"},{"start":"9d","length":"1"},{"start":"9f","length":"2"},{"start":"a3","length":"2"},{"start":"a8","length":"8"},{"start":"b3","length":"1"}]},{"pgid":"4.5","version":"408'20","reported_seq":1011,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-20T22:48:53.209686+0000","last_change":"2024-09-20T22:48:20.112710+0000","last_active":"2024-09-20T22:48:53.209686+0000","last_peered":"2024-09-20T22:48:53.209686+0000","last_clean":"2024-09-20T22:48:53.209686+0000","last_became_active":"2024-09-20T22:48:05.243523+0000","last_became_peered":"2024-09-20T22:48:05.243523+0000","last_unstale":"2024-09-20T22:48:53.209686+0000","last_undegraded":"2024-09-20T22:48:53.209686+0000","last_fullsized":"2024-09-20T22:48:53.209686+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"408'20","last_scrub_stamp":"2024-09-20T22:48:20.112621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:48:20.112621+0000","objects_scrubbed":8,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:49:46.242509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"67'1","reported_seq":799,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-20T22:48:53.215960+0000","last_change":"2024-09-20T22:45:22.053343+0000","last_active":"2024-09-20T22:48:53.215960+0000","last_peered":"2024-09-20T22:48:53.215960+0000","last_clean":"2024-09-20T22:48:53.215960+0000","last_became_active":"2024-09-20T22:41:38.158917+0000","last_became_peered":"2024-09-20T22:41:38.158917+0000","last_unstale":"2024-09-20T22:48:53.215960+0000","last_undegraded":"2024-09-20T22:48:53.215960+0000","last_fullsized":"2024-09-20T22:48:53.215960+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-09-20T22:45:22.053262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:45:22.053262+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:55.172271+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"77'128","reported_seq":1148,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-20T22:48:53.212343+0000","last_change":"2024-09-20T22:47:52.206552+0000","last_active":"2024-09-20T22:48:53.212343+0000","last_peered":"2024-09-20T22:48:53.212343+0000","last_clean":"2024-09-20T22:48:53.212343+0000","last_became_active":"2024-09-20T22:47:52.206169+0000","last_became_peered":"2024-09-20T22:47:52.206169+0000","last_unstale":"2024-09-20T22:48:53.212343+0000","last_undegraded":"2024-09-20T22:48:53.212343+0000","last_fullsized":"2024-09-20T22:48:53.212343+0000","mapping_epoch":391,"log_start":"77'128","ondisk_log_start":"77'128","created":19,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:47:22.417099+0000","last_deep_scrub":"77'128","last_deep_scrub_stamp":"2024-09-20T22:47:22.417099+0000","last_clean_scrub_stamp":"2024-09-20T22:47:22.417099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:54.196327+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":800,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-20T22:48:53.213190+0000","last_change":"2024-09-20T22:48:35.359093+0000","last_active":"2024-09-20T22:48:53.213190+0000","last_peered":"2024-09-20T22:48:53.213190+0000","last_clean":"2024-09-20T22:48:53.213190+0000","last_became_active":"2024-09-20T22:41:42.184387+0000","last_became_peered":"2024-09-20T22:41:42.184387+0000","last_unstale":"2024-09-20T22:48:53.213190+0000","last_undegraded":"2024-09-20T22:48:53.213190+0000","last_fullsized":"2024-09-20T22:48:53.213190+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:48:35.359016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:48:35.359016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:00.848745+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":810,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-20T22:48:53.227687+0000","last_change":"2024-09-20T22:48:18.622779+0000","last_active":"2024-09-20T22:48:53.227687+0000","last_peered":"2024-09-20T22:48:53.227687+0000","last_clean":"2024-09-20T22:48:53.227687+0000","last_became_active":"2024-09-20T22:48:05.241602+0000","last_became_peered":"2024-09-20T22:48:05.241602+0000","last_unstale":"2024-09-20T22:48:53.227687+0000","last_undegraded":"2024-09-20T22:48:53.227687+0000","last_fullsized":"2024-09-20T22:48:53.227687+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:48:18.622674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:48:18.622674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:49:26.359118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"453'238","reported_seq":2085,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-20T22:48:53.215486+0000","last_change":"2024-09-20T22:48:53.215765+0000","last_active":"2024-09-20T22:48:53.215486+0000","last_peered":"2024-09-20T22:48:53.215486+0000","last_clean":"2024-09-20T22:48:53.215486+0000","last_became_active":"2024-09-20T22:47:52.208200+0000","last_became_peered":"2024-09-20T22:47:52.208200+0000","last_unstale":"2024-09-20T22:48:53.215486+0000","last_undegraded":"2024-09-20T22:48:53.215486+0000","last_fullsized":"2024-09-20T22:48:53.215486+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"327'109","last_scrub_stamp":"2024-09-20T22:46:51.808234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:46:51.808234+0000","objects_scrubbed":2,"log_size":238,"log_dups_size":0,"ondisk_log_size":238,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0061489329999999997,"stat_sum":{"num_bytes":14261110,"num_objects":6,"num_object_clones":4,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":102,"num_read_kb":97466,"num_write":55,"num_write_kb":21551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":13529610,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"96","length":"1"},{"start":"9a","length":"1"},{"start":"9d","length":"1"},{"start":"9f","length":"2"},{"start":"a3","length":"2"},{"start":"a8","length":"8"},{"start":"b3","length":"1"}]},{"pgid":"4.e","version":"74'11","reported_seq":119,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-20T22:48:53.216096+0000","last_change":"2024-09-20T22:48:05.271944+0000","last_active":"2024-09-20T22:48:53.216096+0000","last_peered":"2024-09-20T22:48:53.216096+0000","last_clean":"2024-09-20T22:48:53.216096+0000","last_became_active":"2024-09-20T22:48:05.248271+0000","last_became_peered":"2024-09-20T22:48:05.248271+0000","last_unstale":"2024-09-20T22:48:53.216096+0000","last_undegraded":"2024-09-20T22:48:53.216096+0000","last_fullsized":"2024-09-20T22:48:53.216096+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-09-20T22:47:59.092235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:47:59.092235+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:49:11.673300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":107,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-20T22:48:53.214564+0000","last_change":"2024-09-20T22:48:05.255396+0000","last_active":"2024-09-20T22:48:53.214564+0000","last_peered":"2024-09-20T22:48:53.214564+0000","last_clean":"2024-09-20T22:48:53.214564+0000","last_became_active":"2024-09-20T22:48:05.254688+0000","last_became_peered":"2024-09-20T22:48:05.254688+0000","last_unstale":"2024-09-20T22:48:53.214564+0000","last_undegraded":"2024-09-20T22:48:53.214564+0000","last_fullsized":"2024-09-20T22:48:53.214564+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:46:30.871771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:46:30.871771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:58.055996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":806,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-20T22:48:53.227607+0000","last_change":"2024-09-20T22:48:05.234120+0000","last_active":"2024-09-20T22:48:53.227607+0000","last_peered":"2024-09-20T22:48:53.227607+0000","last_clean":"2024-09-20T22:48:53.227607+0000","last_became_active":"2024-09-20T22:48:05.233778+0000","last_became_peered":"2024-09-20T22:48:05.233778+0000","last_unstale":"2024-09-20T22:48:53.227607+0000","last_undegraded":"2024-09-20T22:48:53.227607+0000","last_fullsized":"2024-09-20T22:48:53.227607+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:47:38.288926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:47:38.288926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:49:08.085240+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":789,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-20T22:48:53.214791+0000","last_change":"2024-09-20T22:48:40.127195+0000","last_active":"2024-09-20T22:48:53.214791+0000","last_peered":"2024-09-20T22:48:53.214791+0000","last_clean":"2024-09-20T22:48:53.214791+0000","last_became_active":"2024-09-20T22:41:44.832009+0000","last_became_peered":"2024-09-20T22:41:44.832009+0000","last_unstale":"2024-09-20T22:48:53.214791+0000","last_undegraded":"2024-09-20T22:48:53.214791+0000","last_fullsized":"2024-09-20T22:48:53.214791+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:48:40.127123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:48:40.127123+0000","last_clean_scrub_stamp":"2024-09-20T22:48:40.127123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:04.008351+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.e","version":"453'338","reported_seq":2210,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-20T22:48:53.233771+0000","last_change":"2024-09-20T22:48:53.234094+0000","last_active":"2024-09-20T22:48:53.233771+0000","last_peered":"2024-09-20T22:48:53.233771+0000","last_clean":"2024-09-20T22:48:53.233771+0000","last_became_active":"2024-09-20T22:47:52.209452+0000","last_became_peered":"2024-09-20T22:47:52.209452+0000","last_unstale":"2024-09-20T22:48:53.233771+0000","last_undegraded":"2024-09-20T22:48:53.233771+0000","last_fullsized":"2024-09-20T22:48:53.233771+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"377'202","last_scrub_stamp":"2024-09-20T22:47:42.000554+0000","last_deep_scrub":"174'60","last_deep_scrub_stamp":"2024-09-20T22:44:18.301776+0000","last_clean_scrub_stamp":"2024-09-20T22:47:42.000554+0000","objects_scrubbed":5,"log_size":338,"log_dups_size":0,"ondisk_log_size":338,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0094115989999999997,"stat_sum":{"num_bytes":22293074,"num_objects":10,"num_object_clones":8,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":1,"num_read":156,"num_read_kb":164329,"num_write":101,"num_write_kb":42210,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":23110741,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"96","length":"1"},{"start":"9a","length":"1"},{"start":"9d","length":"1"},{"start":"9f","length":"2"},{"start":"a3","length":"2"},{"start":"a8","length":"8"},{"start":"b3","length":"1"}]},{"pgid":"1.8","version":"77'128","reported_seq":1068,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-20T22:48:53.228501+0000","last_change":"2024-09-20T22:48:17.613298+0000","last_active":"2024-09-20T22:48:53.228501+0000","last_peered":"2024-09-20T22:48:53.228501+0000","last_clean":"2024-09-20T22:48:53.228501+0000","last_became_active":"2024-09-20T22:42:52.405594+0000","last_became_peered":"2024-09-20T22:42:52.405594+0000","last_unstale":"2024-09-20T22:48:53.228501+0000","last_undegraded":"2024-09-20T22:48:53.228501+0000","last_fullsized":"2024-09-20T22:48:53.228501+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:48:17.613185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:48:17.613185+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:49:18.820494+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"74'17","reported_seq":125,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-20T22:48:53.215731+0000","last_change":"2024-09-20T22:48:05.273509+0000","last_active":"2024-09-20T22:48:53.215731+0000","last_peered":"2024-09-20T22:48:53.215731+0000","last_clean":"2024-09-20T22:48:53.215731+0000","last_became_active":"2024-09-20T22:48:05.247971+0000","last_became_peered":"2024-09-20T22:48:05.247971+0000","last_unstale":"2024-09-20T22:48:53.215731+0000","last_undegraded":"2024-09-20T22:48:53.215731+0000","last_fullsized":"2024-09-20T22:48:53.215731+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'17","last_scrub_stamp":"2024-09-20T22:47:38.052166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:47:38.052166+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:59.439253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":28,"num_read_kb":18,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":349,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-20T22:48:53.223287+0000","last_change":"2024-09-20T22:48:06.135135+0000","last_active":"2024-09-20T22:48:53.223287+0000","last_peered":"2024-09-20T22:48:53.223287+0000","last_clean":"2024-09-20T22:48:53.223287+0000","last_became_active":"2024-09-20T22:48:05.240481+0000","last_became_peered":"2024-09-20T22:48:05.240481+0000","last_unstale":"2024-09-20T22:48:53.223287+0000","last_undegraded":"2024-09-20T22:48:53.223287+0000","last_fullsized":"2024-09-20T22:48:53.223287+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:48:06.135068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:35.492562+0000","last_clean_scrub_stamp":"2024-09-20T22:48:06.135068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:49:31.308860+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":817,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-20T22:48:53.211612+0000","last_change":"2024-09-20T22:48:05.252451+0000","last_active":"2024-09-20T22:48:53.211612+0000","last_peered":"2024-09-20T22:48:53.211612+0000","last_clean":"2024-09-20T22:48:53.211612+0000","last_became_active":"2024-09-20T22:48:05.250175+0000","last_became_peered":"2024-09-20T22:48:05.250175+0000","last_unstale":"2024-09-20T22:48:53.211612+0000","last_undegraded":"2024-09-20T22:48:53.211612+0000","last_fullsized":"2024-09-20T22:48:53.211612+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:47:36.658662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:47:36.658662+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:57.165274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":816,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-20T22:48:53.224035+0000","last_change":"2024-09-20T22:48:05.239302+0000","last_active":"2024-09-20T22:48:53.224035+0000","last_peered":"2024-09-20T22:48:53.224035+0000","last_clean":"2024-09-20T22:48:53.224035+0000","last_became_active":"2024-09-20T22:48:05.239006+0000","last_became_peered":"2024-09-20T22:48:05.239006+0000","last_unstale":"2024-09-20T22:48:53.224035+0000","last_undegraded":"2024-09-20T22:48:53.224035+0000","last_fullsized":"2024-09-20T22:48:53.224035+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:47:58.989773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:47:58.989773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:49:24.841257+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"450'437","reported_seq":2270,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-20T22:48:53.215336+0000","last_change":"2024-09-20T22:48:53.215649+0000","last_active":"2024-09-20T22:48:53.215336+0000","last_peered":"2024-09-20T22:48:53.215336+0000","last_clean":"2024-09-20T22:48:53.215336+0000","last_became_active":"2024-09-20T22:47:52.216198+0000","last_became_peered":"2024-09-20T22:47:52.216198+0000","last_unstale":"2024-09-20T22:48:53.215336+0000","last_undegraded":"2024-09-20T22:48:53.215336+0000","last_fullsized":"2024-09-20T22:48:53.215336+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"395'325","last_scrub_stamp":"2024-09-20T22:47:57.577936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:47:57.577936+0000","objects_scrubbed":7,"log_size":437,"log_dups_size":0,"ondisk_log_size":437,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:49:21.584843+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0037847990000000001,"stat_sum":{"num_bytes":14186133,"num_objects":8,"num_object_clones":5,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":2,"num_read":222,"num_read_kb":222706,"num_write":133,"num_write_kb":43892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":53,"num_bytes_recovered":14983556,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"96","length":"1"},{"start":"9a","length":"1"},{"start":"9d","length":"1"},{"start":"9f","length":"2"},{"start":"a3","length":"2"},{"start":"a8","length":"8"},{"start":"b3","length":"1"}]},{"pgid":"1.9","version":"77'128","reported_seq":1107,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-20T22:48:53.212335+0000","last_change":"2024-09-20T22:48:03.252010+0000","last_active":"2024-09-20T22:48:53.212335+0000","last_peered":"2024-09-20T22:48:53.212335+0000","last_clean":"2024-09-20T22:48:53.212335+0000","last_became_active":"2024-09-20T22:42:52.400353+0000","last_became_peered":"2024-09-20T22:42:52.400353+0000","last_unstale":"2024-09-20T22:48:53.212335+0000","last_undegraded":"2024-09-20T22:48:53.212335+0000","last_fullsized":"2024-09-20T22:48:53.212335+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:48:03.251924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:48:03.251924+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:49:08.807948+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"74'10","reported_seq":119,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-20T22:48:53.215504+0000","last_change":"2024-09-20T22:48:05.262827+0000","last_active":"2024-09-20T22:48:53.215504+0000","last_peered":"2024-09-20T22:48:53.215504+0000","last_clean":"2024-09-20T22:48:53.215504+0000","last_became_active":"2024-09-20T22:48:05.242811+0000","last_became_peered":"2024-09-20T22:48:05.242811+0000","last_unstale":"2024-09-20T22:48:53.215504+0000","last_undegraded":"2024-09-20T22:48:53.215504+0000","last_fullsized":"2024-09-20T22:48:53.215504+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-09-20T22:47:21.092309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:47:21.092309+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:58.056163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":825,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-20T22:48:53.216739+0000","last_change":"2024-09-20T22:48:05.234296+0000","last_active":"2024-09-20T22:48:53.216739+0000","last_peered":"2024-09-20T22:48:53.216739+0000","last_clean":"2024-09-20T22:48:53.216739+0000","last_became_active":"2024-09-20T22:48:05.233999+0000","last_became_peered":"2024-09-20T22:48:05.233999+0000","last_unstale":"2024-09-20T22:48:53.216739+0000","last_undegraded":"2024-09-20T22:48:53.216739+0000","last_fullsized":"2024-09-20T22:48:53.216739+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:45:57.194715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:45:57.194715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:55.172370+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"261'9","reported_seq":905,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-20T22:48:53.223619+0000","last_change":"2024-09-20T22:48:43.282707+0000","last_active":"2024-09-20T22:48:53.223619+0000","last_peered":"2024-09-20T22:48:53.223619+0000","last_clean":"2024-09-20T22:48:53.223619+0000","last_became_active":"2024-09-20T22:41:42.185863+0000","last_became_peered":"2024-09-20T22:41:42.185863+0000","last_unstale":"2024-09-20T22:48:53.223619+0000","last_undegraded":"2024-09-20T22:48:53.223619+0000","last_fullsized":"2024-09-20T22:48:53.223619+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'9","last_scrub_stamp":"2024-09-20T22:48:43.282627+0000","last_deep_scrub":"261'9","last_deep_scrub_stamp":"2024-09-20T22:46:50.974735+0000","last_clean_scrub_stamp":"2024-09-20T22:48:43.282627+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:49:57.177125+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":107,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-20T22:48:53.216010+0000","last_change":"2024-09-20T22:48:05.250642+0000","last_active":"2024-09-20T22:48:53.216010+0000","last_peered":"2024-09-20T22:48:53.216010+0000","last_clean":"2024-09-20T22:48:53.216010+0000","last_became_active":"2024-09-20T22:48:05.250284+0000","last_became_peered":"2024-09-20T22:48:05.250284+0000","last_unstale":"2024-09-20T22:48:53.216010+0000","last_undegraded":"2024-09-20T22:48:53.216010+0000","last_fullsized":"2024-09-20T22:48:53.216010+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:46:24.906516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:46:24.906516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:58.056042+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.8","version":"453'311","reported_seq":2011,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-20T22:48:53.225074+0000","last_change":"2024-09-20T22:48:53.225265+0000","last_active":"2024-09-20T22:48:53.225074+0000","last_peered":"2024-09-20T22:48:53.225074+0000","last_clean":"2024-09-20T22:48:53.225074+0000","last_became_active":"2024-09-20T22:48:31.224359+0000","last_became_peered":"2024-09-20T22:48:31.224359+0000","last_unstale":"2024-09-20T22:48:53.225074+0000","last_undegraded":"2024-09-20T22:48:53.225074+0000","last_fullsized":"2024-09-20T22:48:53.225074+0000","mapping_epoch":407,"log_start":"426'268","ondisk_log_start":"426'268","created":94,"last_epoch_clean":431,"parent":"0.0","parent_split_bits":0,"last_scrub":"295'98","last_scrub_stamp":"2024-09-20T22:46:45.991157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:46:45.991157+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":445,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0026658710000000002,"stat_sum":{"num_bytes":14277005,"num_objects":7,"num_object_clones":5,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":128,"num_read_kb":115267,"num_write":117,"num_write_kb":44279,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":26027409,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"96","length":"1"},{"start":"9a","length":"1"},{"start":"9d","length":"1"},{"start":"9f","length":"2"},{"start":"a3","length":"2"},{"start":"a8","length":"8"},{"start":"b3","length":"1"}]},{"pgid":"4.b","version":"74'9","reported_seq":173,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-20T22:48:53.210231+0000","last_change":"2024-09-20T22:48:05.256987+0000","last_active":"2024-09-20T22:48:53.210231+0000","last_peered":"2024-09-20T22:48:53.210231+0000","last_clean":"2024-09-20T22:48:53.210231+0000","last_became_active":"2024-09-20T22:48:05.233267+0000","last_became_peered":"2024-09-20T22:48:05.233267+0000","last_unstale":"2024-09-20T22:48:53.210231+0000","last_undegraded":"2024-09-20T22:48:53.210231+0000","last_fullsized":"2024-09-20T22:48:53.210231+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-09-20T22:48:03.584242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:48:03.584242+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:49:11.956491+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":802,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-20T22:48:53.227480+0000","last_change":"2024-09-20T22:46:25.141094+0000","last_active":"2024-09-20T22:48:53.227480+0000","last_peered":"2024-09-20T22:48:53.227480+0000","last_clean":"2024-09-20T22:48:53.227480+0000","last_became_active":"2024-09-20T22:41:38.155953+0000","last_became_peered":"2024-09-20T22:41:38.155953+0000","last_unstale":"2024-09-20T22:48:53.227480+0000","last_undegraded":"2024-09-20T22:48:53.227480+0000","last_fullsized":"2024-09-20T22:48:53.227480+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:46:25.141028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:46:25.141028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":824,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-20T22:48:53.223488+0000","last_change":"2024-09-20T22:48:05.241119+0000","last_active":"2024-09-20T22:48:53.223488+0000","last_peered":"2024-09-20T22:48:53.223488+0000","last_clean":"2024-09-20T22:48:53.223488+0000","last_became_active":"2024-09-20T22:48:05.240613+0000","last_became_peered":"2024-09-20T22:48:05.240613+0000","last_unstale":"2024-09-20T22:48:53.223488+0000","last_undegraded":"2024-09-20T22:48:53.223488+0000","last_fullsized":"2024-09-20T22:48:53.223488+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:46:49.061850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:46:49.061850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:53.304557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":139,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-20T22:48:53.209571+0000","last_change":"2024-09-20T22:47:49.219776+0000","last_active":"2024-09-20T22:48:53.209571+0000","last_peered":"2024-09-20T22:48:53.209571+0000","last_clean":"2024-09-20T22:48:53.209571+0000","last_became_active":"2024-09-20T22:47:49.219486+0000","last_became_peered":"2024-09-20T22:47:49.219486+0000","last_unstale":"2024-09-20T22:48:53.209571+0000","last_undegraded":"2024-09-20T22:48:53.209571+0000","last_fullsized":"2024-09-20T22:48:53.209571+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:46:51.469567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:46:51.469567+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:56.612259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"453'398","reported_seq":2276,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-20T22:48:53.214447+0000","last_change":"2024-09-20T22:48:53.214950+0000","last_active":"2024-09-20T22:48:53.214447+0000","last_peered":"2024-09-20T22:48:53.214447+0000","last_clean":"2024-09-20T22:48:53.214447+0000","last_became_active":"2024-09-20T22:48:27.272202+0000","last_became_peered":"2024-09-20T22:48:27.272202+0000","last_unstale":"2024-09-20T22:48:53.214447+0000","last_undegraded":"2024-09-20T22:48:53.214447+0000","last_fullsized":"2024-09-20T22:48:53.214447+0000","mapping_epoch":407,"log_start":"423'280","ondisk_log_start":"423'280","created":94,"last_epoch_clean":427,"parent":"0.0","parent_split_bits":0,"last_scrub":"423'271","last_scrub_stamp":"2024-09-20T22:48:23.382092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:48:23.382092+0000","objects_scrubbed":8,"log_size":118,"log_dups_size":649,"ondisk_log_size":118,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:49:23.636137+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.005134759,"stat_sum":{"num_bytes":34489609,"num_objects":15,"num_object_clones":10,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":1,"num_read":217,"num_read_kb":224276,"num_write":185,"num_write_kb":75070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":65,"num_bytes_recovered":38710860,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"96","length":"1"},{"start":"9a","length":"1"},{"start":"9d","length":"1"},{"start":"9f","length":"2"},{"start":"a3","length":"2"},{"start":"a8","length":"8"},{"start":"b3","length":"1"}]},{"pgid":"4.a","version":"113'28","reported_seq":845,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-20T22:48:53.209165+0000","last_change":"2024-09-20T22:47:13.842452+0000","last_active":"2024-09-20T22:48:53.209165+0000","last_peered":"2024-09-20T22:48:53.209165+0000","last_clean":"2024-09-20T22:48:53.209165+0000","last_became_active":"2024-09-20T22:41:40.252011+0000","last_became_peered":"2024-09-20T22:41:40.252011+0000","last_unstale":"2024-09-20T22:48:53.209165+0000","last_undegraded":"2024-09-20T22:48:53.209165+0000","last_fullsized":"2024-09-20T22:48:53.209165+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'28","last_scrub_stamp":"2024-09-20T22:47:13.842316+0000","last_deep_scrub":"113'28","last_deep_scrub_stamp":"2024-09-20T22:43:12.725900+0000","last_clean_scrub_stamp":"2024-09-20T22:47:13.842316+0000","objects_scrubbed":9,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":800,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-20T22:48:53.213945+0000","last_change":"2024-09-20T22:46:30.430426+0000","last_active":"2024-09-20T22:48:53.213945+0000","last_peered":"2024-09-20T22:48:53.213945+0000","last_clean":"2024-09-20T22:48:53.213945+0000","last_became_active":"2024-09-20T22:41:38.161524+0000","last_became_peered":"2024-09-20T22:41:38.161524+0000","last_unstale":"2024-09-20T22:48:53.213945+0000","last_undegraded":"2024-09-20T22:48:53.213945+0000","last_fullsized":"2024-09-20T22:48:53.213945+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:46:30.430353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:46:30.430353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:54.196233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":812,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-20T22:48:53.222800+0000","last_change":"2024-09-20T22:47:49.223627+0000","last_active":"2024-09-20T22:48:53.222800+0000","last_peered":"2024-09-20T22:48:53.222800+0000","last_clean":"2024-09-20T22:48:53.222800+0000","last_became_active":"2024-09-20T22:47:49.223220+0000","last_became_peered":"2024-09-20T22:47:49.223220+0000","last_unstale":"2024-09-20T22:48:53.222800+0000","last_undegraded":"2024-09-20T22:48:53.222800+0000","last_fullsized":"2024-09-20T22:48:53.222800+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:47:36.012147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:47:36.012147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:53.304573+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":796,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-20T22:48:53.212301+0000","last_change":"2024-09-20T22:47:18.329286+0000","last_active":"2024-09-20T22:48:53.212301+0000","last_peered":"2024-09-20T22:48:53.212301+0000","last_clean":"2024-09-20T22:48:53.212301+0000","last_became_active":"2024-09-20T22:41:44.199395+0000","last_became_peered":"2024-09-20T22:41:44.199395+0000","last_unstale":"2024-09-20T22:48:53.212301+0000","last_undegraded":"2024-09-20T22:48:53.212301+0000","last_fullsized":"2024-09-20T22:48:53.212301+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:47:18.329209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:47:18.329209+0000","last_clean_scrub_stamp":"2024-09-20T22:47:18.329209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:54.196297+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.a","version":"452'386","reported_seq":2172,"reported_epoch":453,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T22:48:53.215563+0000","last_change":"2024-09-20T22:48:50.277001+0000","last_active":"2024-09-20T22:48:53.215563+0000","last_peered":"2024-09-20T22:48:53.215563+0000","last_clean":"2024-09-20T22:48:53.215563+0000","last_became_active":"2024-09-20T22:42:52.405085+0000","last_became_peered":"2024-09-20T22:42:52.405085+0000","last_unstale":"2024-09-20T22:48:53.215563+0000","last_undegraded":"2024-09-20T22:48:53.215563+0000","last_fullsized":"2024-09-20T22:48:53.215563+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"301'219","last_scrub_stamp":"2024-09-20T22:46:23.260041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:46:23.260041+0000","objects_scrubbed":3,"log_size":386,"log_dups_size":0,"ondisk_log_size":386,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":8,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.047471777999999999,"stat_sum":{"num_bytes":24092698,"num_objects":11,"num_object_clones":8,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":1,"num_read":271,"num_read_kb":320467,"num_write":147,"num_write_kb":57710,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"9a","length":"1"},{"start":"9d","length":"1"},{"start":"9f","length":"2"},{"start":"a3","length":"2"},{"start":"a8","length":"1"},{"start":"aa","length":"1"}]},{"pgid":"4.9","version":"74'12","reported_seq":119,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-20T22:48:53.214657+0000","last_change":"2024-09-20T22:48:05.284523+0000","last_active":"2024-09-20T22:48:53.214657+0000","last_peered":"2024-09-20T22:48:53.214657+0000","last_clean":"2024-09-20T22:48:53.214657+0000","last_became_active":"2024-09-20T22:48:05.249438+0000","last_became_peered":"2024-09-20T22:48:05.249438+0000","last_unstale":"2024-09-20T22:48:53.214657+0000","last_undegraded":"2024-09-20T22:48:53.214657+0000","last_fullsized":"2024-09-20T22:48:53.214657+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-09-20T22:47:58.421473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:47:58.421473+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:49:19.311510+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":21,"num_read_kb":12,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":832,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-20T22:48:53.212716+0000","last_change":"2024-09-20T22:48:05.240596+0000","last_active":"2024-09-20T22:48:53.212716+0000","last_peered":"2024-09-20T22:48:53.212716+0000","last_clean":"2024-09-20T22:48:53.212716+0000","last_became_active":"2024-09-20T22:48:05.240225+0000","last_became_peered":"2024-09-20T22:48:05.240225+0000","last_unstale":"2024-09-20T22:48:53.212716+0000","last_undegraded":"2024-09-20T22:48:53.212716+0000","last_fullsized":"2024-09-20T22:48:53.212716+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:46:52.465940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:46:52.465940+0000","last_clean_scrub_stamp":"2024-09-20T22:46:52.465940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:54.196267+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":820,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-20T22:48:53.223072+0000","last_change":"2024-09-20T22:48:22.197881+0000","last_active":"2024-09-20T22:48:53.223072+0000","last_peered":"2024-09-20T22:48:53.223072+0000","last_clean":"2024-09-20T22:48:53.223072+0000","last_became_active":"2024-09-20T22:47:49.222088+0000","last_became_peered":"2024-09-20T22:47:49.222088+0000","last_unstale":"2024-09-20T22:48:53.223072+0000","last_undegraded":"2024-09-20T22:48:53.223072+0000","last_fullsized":"2024-09-20T22:48:53.223072+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:48:22.197808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:46:01.828080+0000","last_clean_scrub_stamp":"2024-09-20T22:48:22.197808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:49:43.419742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":789,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-20T22:48:53.216501+0000","last_change":"2024-09-20T22:47:38.019602+0000","last_active":"2024-09-20T22:48:53.216501+0000","last_peered":"2024-09-20T22:48:53.216501+0000","last_clean":"2024-09-20T22:48:53.216501+0000","last_became_active":"2024-09-20T22:41:44.192609+0000","last_became_peered":"2024-09-20T22:41:44.192609+0000","last_unstale":"2024-09-20T22:48:53.216501+0000","last_undegraded":"2024-09-20T22:48:53.216501+0000","last_fullsized":"2024-09-20T22:48:53.216501+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:47:38.019533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:55.170379+0000","last_clean_scrub_stamp":"2024-09-20T22:47:38.019533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:49:06.773525+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"453'624","reported_seq":2379,"reported_epoch":453,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T22:48:53.349796+0000","last_change":"2024-09-20T22:48:51.346393+0000","last_active":"2024-09-20T22:48:53.349796+0000","last_peered":"2024-09-20T22:48:53.349796+0000","last_clean":"2024-09-20T22:48:53.349796+0000","last_became_active":"2024-09-20T22:42:52.409642+0000","last_became_peered":"2024-09-20T22:42:52.409642+0000","last_unstale":"2024-09-20T22:48:53.349796+0000","last_undegraded":"2024-09-20T22:48:53.349796+0000","last_fullsized":"2024-09-20T22:48:53.349796+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"194'116","last_scrub_stamp":"2024-09-20T22:44:37.097674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:44:37.097674+0000","objects_scrubbed":3,"log_size":624,"log_dups_size":0,"ondisk_log_size":624,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":15,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.13223131900000001,"stat_sum":{"num_bytes":38973693,"num_objects":22,"num_object_clones":18,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":1,"num_read":355,"num_read_kb":383656,"num_write":257,"num_write_kb":81643,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"9a","length":"1"}]},{"pgid":"4.8","version":"74'15","reported_seq":823,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-20T22:48:53.228560+0000","last_change":"2024-09-20T22:47:35.313505+0000","last_active":"2024-09-20T22:48:53.228560+0000","last_peered":"2024-09-20T22:48:53.228560+0000","last_clean":"2024-09-20T22:48:53.228560+0000","last_became_active":"2024-09-20T22:41:40.252113+0000","last_became_peered":"2024-09-20T22:41:40.252113+0000","last_unstale":"2024-09-20T22:48:53.228560+0000","last_undegraded":"2024-09-20T22:48:53.228560+0000","last_fullsized":"2024-09-20T22:48:53.228560+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-09-20T22:47:35.313413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:47:35.313413+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'2","reported_seq":860,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-20T22:48:53.213518+0000","last_change":"2024-09-20T22:48:05.269145+0000","last_active":"2024-09-20T22:48:53.213518+0000","last_peered":"2024-09-20T22:48:53.213518+0000","last_clean":"2024-09-20T22:48:53.213518+0000","last_became_active":"2024-09-20T22:48:05.238098+0000","last_became_peered":"2024-09-20T22:48:05.238098+0000","last_unstale":"2024-09-20T22:48:53.213518+0000","last_undegraded":"2024-09-20T22:48:53.213518+0000","last_fullsized":"2024-09-20T22:48:53.213518+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-09-20T22:46:48.473494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:46:48.473494+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:54.196252+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"408'10","reported_seq":916,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-20T22:48:53.212704+0000","last_change":"2024-09-20T22:48:05.265680+0000","last_active":"2024-09-20T22:48:53.212704+0000","last_peered":"2024-09-20T22:48:53.212704+0000","last_clean":"2024-09-20T22:48:53.212704+0000","last_became_active":"2024-09-20T22:48:05.239344+0000","last_became_peered":"2024-09-20T22:48:05.239344+0000","last_unstale":"2024-09-20T22:48:53.212704+0000","last_undegraded":"2024-09-20T22:48:53.212704+0000","last_fullsized":"2024-09-20T22:48:53.212704+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"159'9","last_scrub_stamp":"2024-09-20T22:47:37.382220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:47:37.382220+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:54.196344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":802,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-20T22:48:53.227661+0000","last_change":"2024-09-20T22:47:49.228372+0000","last_active":"2024-09-20T22:48:53.227661+0000","last_peered":"2024-09-20T22:48:53.227661+0000","last_clean":"2024-09-20T22:48:53.227661+0000","last_became_active":"2024-09-20T22:47:49.227569+0000","last_became_peered":"2024-09-20T22:47:49.227569+0000","last_unstale":"2024-09-20T22:48:53.227661+0000","last_undegraded":"2024-09-20T22:48:53.227661+0000","last_fullsized":"2024-09-20T22:48:53.227661+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:47:41.284992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:47:41.284992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:49:09.902863+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.14","version":"450'170","reported_seq":1730,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-20T22:48:53.210415+0000","last_change":"2024-09-20T22:48:53.211308+0000","last_active":"2024-09-20T22:48:53.210415+0000","last_peered":"2024-09-20T22:48:53.210415+0000","last_clean":"2024-09-20T22:48:53.210415+0000","last_became_active":"2024-09-20T22:48:22.211157+0000","last_became_peered":"2024-09-20T22:48:22.211157+0000","last_unstale":"2024-09-20T22:48:53.210415+0000","last_undegraded":"2024-09-20T22:48:53.210415+0000","last_fullsized":"2024-09-20T22:48:53.210415+0000","mapping_epoch":421,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":422,"parent":"0.0","parent_split_bits":5,"last_scrub":"395'116","last_scrub_stamp":"2024-09-20T22:48:03.152941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:48:03.152941+0000","objects_scrubbed":3,"log_size":170,"log_dups_size":0,"ondisk_log_size":170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:49:07.290387+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001136108,"stat_sum":{"num_bytes":8662168,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":57,"num_read_kb":54699,"num_write":48,"num_write_kb":17575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"96","length":"1"},{"start":"9a","length":"1"},{"start":"9d","length":"1"},{"start":"9f","length":"2"},{"start":"a3","length":"2"},{"start":"a8","length":"8"},{"start":"b3","length":"1"}]},{"pgid":"3.10","version":"0'0","reported_seq":821,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-20T22:48:53.209037+0000","last_change":"2024-09-20T22:47:49.213597+0000","last_active":"2024-09-20T22:48:53.209037+0000","last_peered":"2024-09-20T22:48:53.209037+0000","last_clean":"2024-09-20T22:48:53.209037+0000","last_became_active":"2024-09-20T22:47:49.213292+0000","last_became_peered":"2024-09-20T22:47:49.213292+0000","last_unstale":"2024-09-20T22:48:53.209037+0000","last_undegraded":"2024-09-20T22:48:53.209037+0000","last_fullsized":"2024-09-20T22:48:53.209037+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:47:12.820304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:47:12.820304+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"74'6","reported_seq":819,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-20T22:48:53.215709+0000","last_change":"2024-09-20T22:45:39.079425+0000","last_active":"2024-09-20T22:48:53.215709+0000","last_peered":"2024-09-20T22:48:53.215709+0000","last_clean":"2024-09-20T22:48:53.215709+0000","last_became_active":"2024-09-20T22:43:09.116887+0000","last_became_peered":"2024-09-20T22:43:09.116887+0000","last_unstale":"2024-09-20T22:48:53.215709+0000","last_undegraded":"2024-09-20T22:48:53.215709+0000","last_fullsized":"2024-09-20T22:48:53.215709+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'6","last_scrub_stamp":"2024-09-20T22:45:39.079346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:45:39.079346+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:55.172314+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":801,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-20T22:48:53.228090+0000","last_change":"2024-09-20T22:48:35.550867+0000","last_active":"2024-09-20T22:48:53.228090+0000","last_peered":"2024-09-20T22:48:53.228090+0000","last_clean":"2024-09-20T22:48:53.228090+0000","last_became_active":"2024-09-20T22:41:42.184930+0000","last_became_peered":"2024-09-20T22:41:42.184930+0000","last_unstale":"2024-09-20T22:48:53.228090+0000","last_undegraded":"2024-09-20T22:48:53.228090+0000","last_fullsized":"2024-09-20T22:48:53.228090+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:48:35.550781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:09.705632+0000","last_clean_scrub_stamp":"2024-09-20T22:48:35.550781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:05.315763+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"74'1","reported_seq":816,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-20T22:48:53.212979+0000","last_change":"2024-09-20T22:48:40.309005+0000","last_active":"2024-09-20T22:48:53.212979+0000","last_peered":"2024-09-20T22:48:53.212979+0000","last_clean":"2024-09-20T22:48:53.212979+0000","last_became_active":"2024-09-20T22:48:05.237882+0000","last_became_peered":"2024-09-20T22:48:05.237882+0000","last_unstale":"2024-09-20T22:48:53.212979+0000","last_undegraded":"2024-09-20T22:48:53.212979+0000","last_fullsized":"2024-09-20T22:48:53.212979+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-09-20T22:48:40.308879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:48:40.308879+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:07.308273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"74'9","reported_seq":149,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-20T22:48:53.212112+0000","last_change":"2024-09-20T22:47:49.262043+0000","last_active":"2024-09-20T22:48:53.212112+0000","last_peered":"2024-09-20T22:48:53.212112+0000","last_clean":"2024-09-20T22:48:53.212112+0000","last_became_active":"2024-09-20T22:47:49.231315+0000","last_became_peered":"2024-09-20T22:47:49.231315+0000","last_unstale":"2024-09-20T22:48:53.212112+0000","last_undegraded":"2024-09-20T22:48:53.212112+0000","last_fullsized":"2024-09-20T22:48:53.212112+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-09-20T22:46:21.269487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:46:21.269487+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:56.612198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":824,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-20T22:48:53.212841+0000","last_change":"2024-09-20T22:48:05.239747+0000","last_active":"2024-09-20T22:48:53.212841+0000","last_peered":"2024-09-20T22:48:53.212841+0000","last_clean":"2024-09-20T22:48:53.212841+0000","last_became_active":"2024-09-20T22:48:05.238801+0000","last_became_peered":"2024-09-20T22:48:05.238801+0000","last_unstale":"2024-09-20T22:48:53.212841+0000","last_undegraded":"2024-09-20T22:48:53.212841+0000","last_fullsized":"2024-09-20T22:48:53.212841+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:46:46.438556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:46:46.438556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:54.196281+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":792,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-20T22:48:53.215513+0000","last_change":"2024-09-20T22:46:48.012442+0000","last_active":"2024-09-20T22:48:53.215513+0000","last_peered":"2024-09-20T22:48:53.215513+0000","last_clean":"2024-09-20T22:48:53.215513+0000","last_became_active":"2024-09-20T22:41:42.187706+0000","last_became_peered":"2024-09-20T22:41:42.187706+0000","last_unstale":"2024-09-20T22:48:53.215513+0000","last_undegraded":"2024-09-20T22:48:53.215513+0000","last_fullsized":"2024-09-20T22:48:53.215513+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:46:48.012372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:46:48.012372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:55.172411+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":828,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-20T22:48:53.222689+0000","last_change":"2024-09-20T22:48:40.247992+0000","last_active":"2024-09-20T22:48:53.222689+0000","last_peered":"2024-09-20T22:48:53.222689+0000","last_clean":"2024-09-20T22:48:53.222689+0000","last_became_active":"2024-09-20T22:48:05.239710+0000","last_became_peered":"2024-09-20T22:48:05.239710+0000","last_unstale":"2024-09-20T22:48:53.222689+0000","last_undegraded":"2024-09-20T22:48:53.222689+0000","last_fullsized":"2024-09-20T22:48:53.222689+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:48:40.247889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:48:40.247889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:09.303289+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"74'9","reported_seq":814,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-20T22:48:53.227628+0000","last_change":"2024-09-20T22:46:27.123331+0000","last_active":"2024-09-20T22:48:53.227628+0000","last_peered":"2024-09-20T22:48:53.227628+0000","last_clean":"2024-09-20T22:48:53.227628+0000","last_became_active":"2024-09-20T22:41:40.252486+0000","last_became_peered":"2024-09-20T22:41:40.252486+0000","last_unstale":"2024-09-20T22:48:53.227628+0000","last_undegraded":"2024-09-20T22:48:53.227628+0000","last_fullsized":"2024-09-20T22:48:53.227628+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-09-20T22:46:27.123248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:46:27.123248+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":147,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-20T22:48:53.209392+0000","last_change":"2024-09-20T22:48:19.379265+0000","last_active":"2024-09-20T22:48:53.209392+0000","last_peered":"2024-09-20T22:48:53.209392+0000","last_clean":"2024-09-20T22:48:53.209392+0000","last_became_active":"2024-09-20T22:47:49.229936+0000","last_became_peered":"2024-09-20T22:47:49.229936+0000","last_unstale":"2024-09-20T22:48:53.209392+0000","last_undegraded":"2024-09-20T22:48:53.209392+0000","last_fullsized":"2024-09-20T22:48:53.209392+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:48:19.379196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:48:19.379196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:49:35.159658+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"74'8","reported_seq":889,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-20T22:48:53.214741+0000","last_change":"2024-09-20T22:47:56.150641+0000","last_active":"2024-09-20T22:48:53.214741+0000","last_peered":"2024-09-20T22:48:53.214741+0000","last_clean":"2024-09-20T22:48:53.214741+0000","last_became_active":"2024-09-20T22:41:42.186352+0000","last_became_peered":"2024-09-20T22:41:42.186352+0000","last_unstale":"2024-09-20T22:48:53.214741+0000","last_undegraded":"2024-09-20T22:48:53.214741+0000","last_fullsized":"2024-09-20T22:48:53.214741+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-09-20T22:47:56.150562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:47:56.150562+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:49:03.909124+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":107,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-20T22:48:53.216229+0000","last_change":"2024-09-20T22:48:05.250192+0000","last_active":"2024-09-20T22:48:53.216229+0000","last_peered":"2024-09-20T22:48:53.216229+0000","last_clean":"2024-09-20T22:48:53.216229+0000","last_became_active":"2024-09-20T22:48:05.249656+0000","last_became_peered":"2024-09-20T22:48:05.249656+0000","last_unstale":"2024-09-20T22:48:53.216229+0000","last_undegraded":"2024-09-20T22:48:53.216229+0000","last_fullsized":"2024-09-20T22:48:53.216229+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:47:19.828541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:47:19.828541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:58.056132+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"74'10","reported_seq":812,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-20T22:48:53.214489+0000","last_change":"2024-09-20T22:46:24.269736+0000","last_active":"2024-09-20T22:48:53.214489+0000","last_peered":"2024-09-20T22:48:53.214489+0000","last_clean":"2024-09-20T22:48:53.214489+0000","last_became_active":"2024-09-20T22:41:40.253614+0000","last_became_peered":"2024-09-20T22:41:40.253614+0000","last_unstale":"2024-09-20T22:48:53.214489+0000","last_undegraded":"2024-09-20T22:48:53.214489+0000","last_fullsized":"2024-09-20T22:48:53.214489+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-09-20T22:46:24.269655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:46:24.269655+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:55.172386+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":820,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-20T22:48:53.212741+0000","last_change":"2024-09-20T22:48:05.240585+0000","last_active":"2024-09-20T22:48:53.212741+0000","last_peered":"2024-09-20T22:48:53.212741+0000","last_clean":"2024-09-20T22:48:53.212741+0000","last_became_active":"2024-09-20T22:48:05.240231+0000","last_became_peered":"2024-09-20T22:48:05.240231+0000","last_unstale":"2024-09-20T22:48:53.212741+0000","last_undegraded":"2024-09-20T22:48:53.212741+0000","last_fullsized":"2024-09-20T22:48:53.212741+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:48:01.305632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:48:01.305632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:49:16.408958+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"389'10","reported_seq":920,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-20T22:48:53.227423+0000","last_change":"2024-09-20T22:47:58.443470+0000","last_active":"2024-09-20T22:48:53.227423+0000","last_peered":"2024-09-20T22:48:53.227423+0000","last_clean":"2024-09-20T22:48:53.227423+0000","last_became_active":"2024-09-20T22:47:49.227113+0000","last_became_peered":"2024-09-20T22:47:49.227113+0000","last_unstale":"2024-09-20T22:48:53.227423+0000","last_undegraded":"2024-09-20T22:48:53.227423+0000","last_fullsized":"2024-09-20T22:48:53.227423+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'10","last_scrub_stamp":"2024-09-20T22:47:58.443393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:47:58.443393+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:49:10.836242+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":139,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-20T22:48:53.210878+0000","last_change":"2024-09-20T22:47:49.232560+0000","last_active":"2024-09-20T22:48:53.210878+0000","last_peered":"2024-09-20T22:48:53.210878+0000","last_clean":"2024-09-20T22:48:53.210878+0000","last_became_active":"2024-09-20T22:47:49.230674+0000","last_became_peered":"2024-09-20T22:47:49.230674+0000","last_unstale":"2024-09-20T22:48:53.210878+0000","last_undegraded":"2024-09-20T22:48:53.210878+0000","last_fullsized":"2024-09-20T22:48:53.210878+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:47:41.411716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:47:41.411716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:54.101804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.10","version":"450'154","reported_seq":1894,"reported_epoch":453,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-20T22:48:53.278012+0000","last_change":"2024-09-20T22:48:53.278050+0000","last_active":"2024-09-20T22:48:53.278012+0000","last_peered":"2024-09-20T22:48:53.278012+0000","last_clean":"2024-09-20T22:48:53.278012+0000","last_became_active":"2024-09-20T22:48:22.213631+0000","last_became_peered":"2024-09-20T22:48:22.213631+0000","last_unstale":"2024-09-20T22:48:53.278012+0000","last_undegraded":"2024-09-20T22:48:53.278012+0000","last_fullsized":"2024-09-20T22:48:53.278012+0000","mapping_epoch":421,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":422,"parent":"0.0","parent_split_bits":5,"last_scrub":"146'43","last_scrub_stamp":"2024-09-20T22:43:46.106059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:43:46.106059+0000","objects_scrubbed":0,"log_size":154,"log_dups_size":0,"ondisk_log_size":154,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":9,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.049871116,"stat_sum":{"num_bytes":8863761,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":58,"num_read_kb":77675,"num_write":37,"num_write_kb":16548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":7603720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"96","length":"1"},{"start":"9a","length":"1"},{"start":"9d","length":"1"},{"start":"9f","length":"2"},{"start":"a3","length":"2"}]},{"pgid":"4.13","version":"74'11","reported_seq":127,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-20T22:48:53.215098+0000","last_change":"2024-09-20T22:48:44.023878+0000","last_active":"2024-09-20T22:48:53.215098+0000","last_peered":"2024-09-20T22:48:53.215098+0000","last_clean":"2024-09-20T22:48:53.215098+0000","last_became_active":"2024-09-20T22:48:05.249510+0000","last_became_peered":"2024-09-20T22:48:05.249510+0000","last_unstale":"2024-09-20T22:48:53.215098+0000","last_undegraded":"2024-09-20T22:48:53.215098+0000","last_fullsized":"2024-09-20T22:48:53.215098+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-09-20T22:48:44.023781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:48:44.023781+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:07.884280+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":107,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-20T22:48:53.215380+0000","last_change":"2024-09-20T22:48:05.252646+0000","last_active":"2024-09-20T22:48:53.215380+0000","last_peered":"2024-09-20T22:48:53.215380+0000","last_clean":"2024-09-20T22:48:53.215380+0000","last_became_active":"2024-09-20T22:48:05.252026+0000","last_became_peered":"2024-09-20T22:48:05.252026+0000","last_unstale":"2024-09-20T22:48:53.215380+0000","last_undegraded":"2024-09-20T22:48:53.215380+0000","last_fullsized":"2024-09-20T22:48:53.215380+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:46:04.391306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:46:04.391306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:58.055966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":793,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-20T22:48:53.209486+0000","last_change":"2024-09-20T22:46:25.881993+0000","last_active":"2024-09-20T22:48:53.209486+0000","last_peered":"2024-09-20T22:48:53.209486+0000","last_clean":"2024-09-20T22:48:53.209486+0000","last_became_active":"2024-09-20T22:41:42.181736+0000","last_became_peered":"2024-09-20T22:41:42.181736+0000","last_unstale":"2024-09-20T22:48:53.209486+0000","last_undegraded":"2024-09-20T22:48:53.209486+0000","last_fullsized":"2024-09-20T22:48:53.209486+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:46:25.881921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:46:25.881921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:57.165157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":801,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-20T22:48:53.213202+0000","last_change":"2024-09-20T22:47:49.216479+0000","last_active":"2024-09-20T22:48:53.213202+0000","last_peered":"2024-09-20T22:48:53.213202+0000","last_clean":"2024-09-20T22:48:53.213202+0000","last_became_active":"2024-09-20T22:47:49.216141+0000","last_became_peered":"2024-09-20T22:47:49.216141+0000","last_unstale":"2024-09-20T22:48:53.213202+0000","last_undegraded":"2024-09-20T22:48:53.213202+0000","last_fullsized":"2024-09-20T22:48:53.213202+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:47:36.066480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:47:36.066480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:55.172425+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"135'39","reported_seq":204,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-20T22:48:53.214512+0000","last_change":"2024-09-20T22:48:53.215631+0000","last_active":"2024-09-20T22:48:53.214512+0000","last_peered":"2024-09-20T22:48:53.214512+0000","last_clean":"2024-09-20T22:48:53.214512+0000","last_became_active":"2024-09-20T22:48:19.212541+0000","last_became_peered":"2024-09-20T22:48:19.212541+0000","last_unstale":"2024-09-20T22:48:53.214512+0000","last_undegraded":"2024-09-20T22:48:53.214512+0000","last_fullsized":"2024-09-20T22:48:53.214512+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":5,"last_scrub":"135'39","last_scrub_stamp":"2024-09-20T22:48:19.969753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:48:19.969753+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:49:44.277096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013712539999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"96","length":"1"},{"start":"9a","length":"1"},{"start":"9d","length":"1"},{"start":"9f","length":"2"},{"start":"a3","length":"2"},{"start":"a8","length":"8"},{"start":"b3","length":"1"}]},{"pgid":"4.12","version":"404'13","reported_seq":825,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-20T22:48:53.209715+0000","last_change":"2024-09-20T22:48:04.276356+0000","last_active":"2024-09-20T22:48:53.209715+0000","last_peered":"2024-09-20T22:48:53.209715+0000","last_clean":"2024-09-20T22:48:53.209715+0000","last_became_active":"2024-09-20T22:41:40.164796+0000","last_became_peered":"2024-09-20T22:41:40.164796+0000","last_unstale":"2024-09-20T22:48:53.209715+0000","last_undegraded":"2024-09-20T22:48:53.209715+0000","last_fullsized":"2024-09-20T22:48:53.209715+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"404'13","last_scrub_stamp":"2024-09-20T22:48:04.276282+0000","last_deep_scrub":"404'13","last_deep_scrub_stamp":"2024-09-20T22:48:04.276282+0000","last_clean_scrub_stamp":"2024-09-20T22:48:04.276282+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:49:13.107971+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":835,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-20T22:48:53.213330+0000","last_change":"2024-09-20T22:48:24.422893+0000","last_active":"2024-09-20T22:48:53.213330+0000","last_peered":"2024-09-20T22:48:53.213330+0000","last_clean":"2024-09-20T22:48:53.213330+0000","last_became_active":"2024-09-20T22:48:05.238812+0000","last_became_peered":"2024-09-20T22:48:05.238812+0000","last_unstale":"2024-09-20T22:48:53.213330+0000","last_undegraded":"2024-09-20T22:48:53.213330+0000","last_fullsized":"2024-09-20T22:48:53.213330+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:48:24.422757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:31.138067+0000","last_clean_scrub_stamp":"2024-09-20T22:48:24.422757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:49:32.075381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":805,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-20T22:48:53.216590+0000","last_change":"2024-09-20T22:48:38.200896+0000","last_active":"2024-09-20T22:48:53.216590+0000","last_peered":"2024-09-20T22:48:53.216590+0000","last_clean":"2024-09-20T22:48:53.216590+0000","last_became_active":"2024-09-20T22:47:49.218708+0000","last_became_peered":"2024-09-20T22:47:49.218708+0000","last_unstale":"2024-09-20T22:48:53.216590+0000","last_undegraded":"2024-09-20T22:48:53.216590+0000","last_fullsized":"2024-09-20T22:48:53.216590+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:48:38.200797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:48:38.200797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:49:51.213367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":139,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-20T22:48:53.208789+0000","last_change":"2024-09-20T22:47:49.225362+0000","last_active":"2024-09-20T22:48:53.208789+0000","last_peered":"2024-09-20T22:48:53.208789+0000","last_clean":"2024-09-20T22:48:53.208789+0000","last_became_active":"2024-09-20T22:47:49.225027+0000","last_became_peered":"2024-09-20T22:47:49.225027+0000","last_unstale":"2024-09-20T22:48:53.208789+0000","last_undegraded":"2024-09-20T22:48:53.208789+0000","last_fullsized":"2024-09-20T22:48:53.208789+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:47:37.309842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:47:37.309842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:49:05.473839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.12","version":"451'440","reported_seq":2094,"reported_epoch":453,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T22:48:53.215242+0000","last_change":"2024-09-20T22:48:52.265582+0000","last_active":"2024-09-20T22:48:53.215242+0000","last_peered":"2024-09-20T22:48:53.215242+0000","last_clean":"2024-09-20T22:48:53.215242+0000","last_became_active":"2024-09-20T22:48:22.216595+0000","last_became_peered":"2024-09-20T22:48:22.216595+0000","last_unstale":"2024-09-20T22:48:53.215242+0000","last_undegraded":"2024-09-20T22:48:53.215242+0000","last_fullsized":"2024-09-20T22:48:53.215242+0000","mapping_epoch":421,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":422,"parent":"0.0","parent_split_bits":5,"last_scrub":"377'359","last_scrub_stamp":"2024-09-20T22:47:41.855224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:47:41.855224+0000","objects_scrubbed":9,"log_size":440,"log_dups_size":0,"ondisk_log_size":440,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":10,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.060756681999999999,"stat_sum":{"num_bytes":20487666,"num_objects":9,"num_object_clones":6,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":160,"num_read_kb":146174,"num_write":124,"num_write_kb":39984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":107,"num_bytes_recovered":44451977,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"9a","length":"1"},{"start":"9d","length":"1"},{"start":"9f","length":"1"},{"start":"a3","length":"2"},{"start":"a8","length":"1"}]},{"pgid":"4.11","version":"74'11","reported_seq":607,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-20T22:48:53.227050+0000","last_change":"2024-09-20T22:46:31.254470+0000","last_active":"2024-09-20T22:48:53.227050+0000","last_peered":"2024-09-20T22:48:53.227050+0000","last_clean":"2024-09-20T22:48:53.227050+0000","last_became_active":"2024-09-20T22:44:09.170785+0000","last_became_peered":"2024-09-20T22:44:09.170785+0000","last_unstale":"2024-09-20T22:48:53.227050+0000","last_undegraded":"2024-09-20T22:48:53.227050+0000","last_fullsized":"2024-09-20T22:48:53.227050+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-09-20T22:46:31.254376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:46:31.254376+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":810,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-20T22:48:53.228160+0000","last_change":"2024-09-20T22:48:02.512192+0000","last_active":"2024-09-20T22:48:53.228160+0000","last_peered":"2024-09-20T22:48:53.228160+0000","last_clean":"2024-09-20T22:48:53.228160+0000","last_became_active":"2024-09-20T22:41:38.158701+0000","last_became_peered":"2024-09-20T22:41:38.158701+0000","last_unstale":"2024-09-20T22:48:53.228160+0000","last_undegraded":"2024-09-20T22:48:53.228160+0000","last_fullsized":"2024-09-20T22:48:53.228160+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:48:02.512093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:45:24.765414+0000","last_clean_scrub_stamp":"2024-09-20T22:48:02.512093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:49:28.678353+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":816,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-20T22:48:53.212034+0000","last_change":"2024-09-20T22:48:05.238692+0000","last_active":"2024-09-20T22:48:53.212034+0000","last_peered":"2024-09-20T22:48:53.212034+0000","last_clean":"2024-09-20T22:48:53.212034+0000","last_became_active":"2024-09-20T22:48:05.238334+0000","last_became_peered":"2024-09-20T22:48:05.238334+0000","last_unstale":"2024-09-20T22:48:53.212034+0000","last_undegraded":"2024-09-20T22:48:53.212034+0000","last_fullsized":"2024-09-20T22:48:53.212034+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:46:29.466361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:50.174861+0000","last_clean_scrub_stamp":"2024-09-20T22:46:29.466361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:54.196218+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":793,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-20T22:48:53.216707+0000","last_change":"2024-09-20T22:47:49.215237+0000","last_active":"2024-09-20T22:48:53.216707+0000","last_peered":"2024-09-20T22:48:53.216707+0000","last_clean":"2024-09-20T22:48:53.216707+0000","last_became_active":"2024-09-20T22:47:49.214487+0000","last_became_peered":"2024-09-20T22:47:49.214487+0000","last_unstale":"2024-09-20T22:48:53.216707+0000","last_undegraded":"2024-09-20T22:48:53.216707+0000","last_fullsized":"2024-09-20T22:48:53.216707+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:45:00.006909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:45:00.006909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:55.172155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"450'125","reported_seq":391,"reported_epoch":453,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-20T22:48:53.215868+0000","last_change":"2024-09-20T22:48:52.265551+0000","last_active":"2024-09-20T22:48:53.215868+0000","last_peered":"2024-09-20T22:48:53.215868+0000","last_clean":"2024-09-20T22:48:53.215868+0000","last_became_active":"2024-09-20T22:48:22.213639+0000","last_became_peered":"2024-09-20T22:48:22.213639+0000","last_unstale":"2024-09-20T22:48:53.215868+0000","last_undegraded":"2024-09-20T22:48:53.215868+0000","last_fullsized":"2024-09-20T22:48:53.215868+0000","mapping_epoch":421,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":422,"parent":"0.0","parent_split_bits":5,"last_scrub":"255'21","last_scrub_stamp":"2024-09-20T22:45:35.767459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:45:35.767459+0000","objects_scrubbed":1,"log_size":125,"log_dups_size":0,"ondisk_log_size":125,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":1.0359132289999999,"stat_sum":{"num_bytes":12888730,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":63,"num_read_kb":85981,"num_write":27,"num_write_kb":11537,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":20196357,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"96","length":"1"},{"start":"9d","length":"1"},{"start":"9f","length":"2"},{"start":"a3","length":"2"},{"start":"a8","length":"1"},{"start":"aa","length":"4"},{"start":"af","length":"1"}]},{"pgid":"4.10","version":"74'4","reported_seq":803,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-20T22:48:53.214403+0000","last_change":"2024-09-20T22:48:23.236306+0000","last_active":"2024-09-20T22:48:53.214403+0000","last_peered":"2024-09-20T22:48:53.214403+0000","last_clean":"2024-09-20T22:48:53.214403+0000","last_became_active":"2024-09-20T22:41:40.166123+0000","last_became_peered":"2024-09-20T22:41:40.166123+0000","last_unstale":"2024-09-20T22:48:53.214403+0000","last_undegraded":"2024-09-20T22:48:53.214403+0000","last_fullsized":"2024-09-20T22:48:53.214403+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'4","last_scrub_stamp":"2024-09-20T22:48:23.236227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:48:23.236227+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:49:28.615652+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":147,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-20T22:48:53.211420+0000","last_change":"2024-09-20T22:48:18.432371+0000","last_active":"2024-09-20T22:48:53.211420+0000","last_peered":"2024-09-20T22:48:53.211420+0000","last_clean":"2024-09-20T22:48:53.211420+0000","last_became_active":"2024-09-20T22:47:49.231964+0000","last_became_peered":"2024-09-20T22:47:49.231964+0000","last_unstale":"2024-09-20T22:48:53.211420+0000","last_undegraded":"2024-09-20T22:48:53.211420+0000","last_fullsized":"2024-09-20T22:48:53.211420+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:48:18.432271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:48:18.432271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:49:21.454351+0000","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":825,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-20T22:48:53.209143+0000","last_change":"2024-09-20T22:48:05.245017+0000","last_active":"2024-09-20T22:48:53.209143+0000","last_peered":"2024-09-20T22:48:53.209143+0000","last_clean":"2024-09-20T22:48:53.209143+0000","last_became_active":"2024-09-20T22:48:05.244659+0000","last_became_peered":"2024-09-20T22:48:05.244659+0000","last_unstale":"2024-09-20T22:48:53.209143+0000","last_undegraded":"2024-09-20T22:48:53.209143+0000","last_fullsized":"2024-09-20T22:48:53.209143+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:47:37.826041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:47:37.826041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:53.997023+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":808,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-20T22:48:53.212350+0000","last_change":"2024-09-20T22:48:19.379992+0000","last_active":"2024-09-20T22:48:53.212350+0000","last_peered":"2024-09-20T22:48:53.212350+0000","last_clean":"2024-09-20T22:48:53.212350+0000","last_became_active":"2024-09-20T22:48:05.237765+0000","last_became_peered":"2024-09-20T22:48:05.237765+0000","last_unstale":"2024-09-20T22:48:53.212350+0000","last_undegraded":"2024-09-20T22:48:53.212350+0000","last_fullsized":"2024-09-20T22:48:53.212350+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:48:19.379926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:10.233487+0000","last_clean_scrub_stamp":"2024-09-20T22:48:19.379926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:49:32.710554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":800,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-20T22:48:53.212413+0000","last_change":"2024-09-20T22:48:05.249807+0000","last_active":"2024-09-20T22:48:53.212413+0000","last_peered":"2024-09-20T22:48:53.212413+0000","last_clean":"2024-09-20T22:48:53.212413+0000","last_became_active":"2024-09-20T22:48:05.249492+0000","last_became_peered":"2024-09-20T22:48:05.249492+0000","last_unstale":"2024-09-20T22:48:53.212413+0000","last_undegraded":"2024-09-20T22:48:53.212413+0000","last_fullsized":"2024-09-20T22:48:53.212413+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:47:19.596232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:47:19.596232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:57.165241+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":826,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-20T22:48:53.214153+0000","last_change":"2024-09-20T22:48:36.246135+0000","last_active":"2024-09-20T22:48:53.214153+0000","last_peered":"2024-09-20T22:48:53.214153+0000","last_clean":"2024-09-20T22:48:53.214153+0000","last_became_active":"2024-09-20T22:47:49.217064+0000","last_became_peered":"2024-09-20T22:47:49.217064+0000","last_unstale":"2024-09-20T22:48:53.214153+0000","last_undegraded":"2024-09-20T22:48:53.214153+0000","last_fullsized":"2024-09-20T22:48:53.214153+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:48:36.246069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:48:36.246069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:49:36.464648+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"74'11","reported_seq":827,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-20T22:48:53.210224+0000","last_change":"2024-09-20T22:47:38.862277+0000","last_active":"2024-09-20T22:48:53.210224+0000","last_peered":"2024-09-20T22:48:53.210224+0000","last_clean":"2024-09-20T22:48:53.210224+0000","last_became_active":"2024-09-20T22:41:40.171256+0000","last_became_peered":"2024-09-20T22:41:40.171256+0000","last_unstale":"2024-09-20T22:48:53.210224+0000","last_undegraded":"2024-09-20T22:48:53.210224+0000","last_fullsized":"2024-09-20T22:48:53.210224+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-09-20T22:47:38.862194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:47:38.862194+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:49:03.255502+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":139,"reported_epoch":453,"state":"active+clean","last_fresh":"2024-09-20T22:48:53.210979+0000","last_change":"2024-09-20T22:47:49.221413+0000","last_active":"2024-09-20T22:48:53.210979+0000","last_peered":"2024-09-20T22:48:53.210979+0000","last_clean":"2024-09-20T22:48:53.210979+0000","last_became_active":"2024-09-20T22:47:49.220971+0000","last_became_peered":"2024-09-20T22:47:49.220971+0000","last_unstale":"2024-09-20T22:48:53.210979+0000","last_undegraded":"2024-09-20T22:48:53.210979+0000","last_fullsized":"2024-09-20T22:48:53.210979+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:46:20.485453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:51.667341+0000","last_clean_scrub_stamp":"2024-09-20T22:46:20.485453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:48:56.612212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":21,"stat_sum":{"num_bytes":372726350,"num_objects":175,"num_object_clones":127,"num_object_copies":525,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":175,"num_whiteouts":13,"num_read":3079,"num_read_kb":3210756,"num_write":2143,"num_write_kb":805366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":600,"num_bytes_recovered":297592144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":828387328,"data_stored":1156076043,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5364,"ondisk_log_size":5364,"up":63,"acting":63,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":82,"ondisk_log_size":82,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":682,"num_read_kb":439,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":238,"num_bytes_recovered":5126,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":439,"ondisk_log_size":439,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":1034,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":426,"num_read_kb":421,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":384,"ondisk_log_size":384,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070555,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":128480,"kb_used_data":75080,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93616672,"statfs":{"total":95995035648,"available":95863472128,"internally_reserved":0,"allocated":76881920,"data_stored":102204478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":54655434},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430305,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":165904,"kb_used_data":112500,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93579248,"statfs":{"total":95995035648,"available":95825149952,"internally_reserved":0,"allocated":115200000,"data_stored":171627738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691942,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":120212,"kb_used_data":89020,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93624940,"statfs":{"total":95995035648,"available":95871938560,"internally_reserved":0,"allocated":91156480,"data_stored":137205054,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986283,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":112216,"kb_used_data":76860,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93632936,"statfs":{"total":95995035648,"available":95880126464,"internally_reserved":0,"allocated":78704640,"data_stored":95172292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182513,"num_pgs":73,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":201904,"kb_used_data":148504,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93543248,"statfs":{"total":95995035648,"available":95788285952,"internally_reserved":0,"allocated":152068096,"data_stored":209466662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411446,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":117700,"kb_used_data":86500,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93627452,"statfs":{"total":95995035648,"available":95874510848,"internally_reserved":0,"allocated":88576000,"data_stored":117624037,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574971,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":150080,"kb_used_data":118892,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93595072,"statfs":{"total":95995035648,"available":95841353728,"internally_reserved":0,"allocated":121745408,"data_stored":171659726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738496,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":177632,"kb_used_data":128388,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":93567520,"statfs":{"total":95995035648,"available":95813140480,"internally_reserved":0,"allocated":131469312,"data_stored":166543429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":128135168,"data_stored":164531976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":118382592,"data_stored":169647938,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":85995520,"data_stored":117770662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":148144128,"data_stored":206871881,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":75378688,"data_stored":93169035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":87240704,"data_stored":134602565,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":111886336,"data_stored":169616509,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":73224192,"data_stored":99865477,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-20T22:48:58.762 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd pool set unique_pool_0 pgp_num 16 2024-09-20T22:48:58.798 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:48:58.821 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:58.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:58.822 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:48:58.923 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:48:58.946 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:58.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:58.947 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:48:59.048 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:48:59.072 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:59.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:59.073 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:48:59.087 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-20T22:48:59.087 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2024-09-20T22:48:59.087 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2024-09-20T22:48:59.087 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [ 2024-09-20T22:48:59.087 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.088 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:901 7.e 7:7c62ced4:::smithi08477203-17:head [copy-from ver 48] snapc 1e=[] ondisk+write+known_if_redirected+supports_pool_eio e156)", 2024-09-20T22:48:59.088 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:43:55.624762+0000", 2024-09-20T22:48:59.088 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 303.22885199299998, 2024-09-20T22:48:59.088 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.021835856000000001, 2024-09-20T22:48:59.088 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:48:59.088 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:48:59.088 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:48:59.088 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:48:59.088 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:48:59.088 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:48:59.088 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 901 2024-09-20T22:48:59.088 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.088 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:48:59.088 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.089 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:48:59.089 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.624762+0000", 2024-09-20T22:48:59.089 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:48:59.089 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.089 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.089 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:48:59.089 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.624762+0000", 2024-09-20T22:48:59.089 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:48:59.089 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.089 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.089 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:48:59.089 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.624765+0000", 2024-09-20T22:48:59.089 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.57e-06 2024-09-20T22:48:59.090 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.090 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.090 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:48:59.090 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.624776+0000", 2024-09-20T22:48:59.090 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.078e-05 2024-09-20T22:48:59.090 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.090 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.090 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:48:59.090 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.624803+0000", 2024-09-20T22:48:59.090 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.6769999999999999e-05 2024-09-20T22:48:59.090 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.090 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.090 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:48:59.091 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.624815+0000", 2024-09-20T22:48:59.091 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2459e-05 2024-09-20T22:48:59.091 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.091 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.091 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:48:59.091 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.624936+0000", 2024-09-20T22:48:59.091 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000120467 2024-09-20T22:48:59.091 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.091 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.091 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:48:59.091 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.625046+0000", 2024-09-20T22:48:59.091 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000110222 2024-09-20T22:48:59.091 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.091 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.092 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 0,5", 2024-09-20T22:48:59.092 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.635547+0000", 2024-09-20T22:48:59.092 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010501765999999999 2024-09-20T22:48:59.092 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.092 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.092 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:48:59.092 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.644332+0000", 2024-09-20T22:48:59.092 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0087847729999999992 2024-09-20T22:48:59.092 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.092 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.092 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:48:59.092 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.646168+0000", 2024-09-20T22:48:59.092 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0018360010000000001 2024-09-20T22:48:59.093 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.093 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.093 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:48:59.093 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.646512+0000", 2024-09-20T22:48:59.093 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000343269 2024-09-20T22:48:59.093 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.093 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.093 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:48:59.093 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.646550+0000", 2024-09-20T22:48:59.093 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.8383000000000002e-05 2024-09-20T22:48:59.093 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.093 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.093 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:48:59.094 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.646598+0000", 2024-09-20T22:48:59.094 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.8396000000000002e-05 2024-09-20T22:48:59.094 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:59.094 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:48:59.094 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:59.094 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.094 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.094 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:902 7.e 7:7c62ced4:::smithi08477203-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e156)", 2024-09-20T22:48:59.094 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:43:55.624853+0000", 2024-09-20T22:48:59.094 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 303.22876191500001, 2024-09-20T22:48:59.094 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.015323603999999999, 2024-09-20T22:48:59.094 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:48:59.094 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:48:59.095 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-20T22:48:59.095 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:48:59.095 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:48:59.095 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:48:59.095 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 902 2024-09-20T22:48:59.095 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.095 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:48:59.095 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.095 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:48:59.095 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.624853+0000", 2024-09-20T22:48:59.095 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:48:59.095 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.095 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.095 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:48:59.095 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.624853+0000", 2024-09-20T22:48:59.096 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:48:59.096 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.096 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.096 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:48:59.096 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.624853+0000", 2024-09-20T22:48:59.096 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.1399999999999997e-07 2024-09-20T22:48:59.096 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.096 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.096 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:48:59.096 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.624859+0000", 2024-09-20T22:48:59.096 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.9610000000000002e-06 2024-09-20T22:48:59.096 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.096 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.097 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:48:59.097 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.624887+0000", 2024-09-20T22:48:59.097 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.7597e-05 2024-09-20T22:48:59.097 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.097 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.097 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:48:59.097 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.624894+0000", 2024-09-20T22:48:59.097 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.469e-06 2024-09-20T22:48:59.097 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.097 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.097 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:48:59.097 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.625219+0000", 2024-09-20T22:48:59.098 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00032538499999999998 2024-09-20T22:48:59.098 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.098 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.098 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for blocked object", 2024-09-20T22:48:59.098 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.625258+0000", 2024-09-20T22:48:59.098 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.8467000000000003e-05 2024-09-20T22:48:59.098 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.098 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.098 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:48:59.098 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.640050+0000", 2024-09-20T22:48:59.098 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0147923 2024-09-20T22:48:59.098 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.098 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.098 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:48:59.099 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.640112+0000", 2024-09-20T22:48:59.099 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.1723000000000005e-05 2024-09-20T22:48:59.099 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.099 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.099 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:48:59.099 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.640176+0000", 2024-09-20T22:48:59.099 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.4188000000000003e-05 2024-09-20T22:48:59.099 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:59.099 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:48:59.099 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:59.099 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.099 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.099 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24652.0:1420 7.19 e189/139 7:9cc0b7e2:::smithi08477203-46:head v 189'99, mlcod=174'95)", 2024-09-20T22:48:59.099 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:44:28.150258+0000", 2024-09-20T22:48:59.099 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 270.703356219, 2024-09-20T22:48:59.100 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014180626, 2024-09-20T22:48:59.100 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:48:59.100 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:48:59.100 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:48:59.100 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:48:59.100 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.100 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:48:59.100 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.150258+0000", 2024-09-20T22:48:59.100 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:48:59.100 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.100 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.100 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:48:59.100 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.150258+0000", 2024-09-20T22:48:59.100 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:48:59.101 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.101 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.101 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:48:59.101 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.150259+0000", 2024-09-20T22:48:59.101 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.99e-07 2024-09-20T22:48:59.101 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.101 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.101 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:48:59.101 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.150674+0000", 2024-09-20T22:48:59.101 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00041528000000000002 2024-09-20T22:48:59.101 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.101 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.101 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:48:59.101 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.150688+0000", 2024-09-20T22:48:59.101 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3399e-05 2024-09-20T22:48:59.102 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.102 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.102 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:48:59.102 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.150699+0000", 2024-09-20T22:48:59.102 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1022999999999999e-05 2024-09-20T22:48:59.102 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.102 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.102 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:48:59.102 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.150772+0000", 2024-09-20T22:48:59.102 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.3455999999999996e-05 2024-09-20T22:48:59.102 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.102 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.102 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:48:59.102 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.150798+0000", 2024-09-20T22:48:59.103 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.5777000000000001e-05 2024-09-20T22:48:59.103 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.103 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.103 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:48:59.103 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.164390+0000", 2024-09-20T22:48:59.103 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013591789999999999 2024-09-20T22:48:59.103 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.103 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.103 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:48:59.103 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.164439+0000", 2024-09-20T22:48:59.103 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.9202000000000001e-05 2024-09-20T22:48:59.104 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:59.104 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:48:59.104 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:59.104 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.104 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.104 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24652.0:1421 7.19 e189/139 7:9cc0b7e2:::smithi08477203-46:head v 189'100, mlcod=174'95)", 2024-09-20T22:48:59.104 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:44:28.156805+0000", 2024-09-20T22:48:59.104 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 270.69680936100002, 2024-09-20T22:48:59.104 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013180661999999999, 2024-09-20T22:48:59.104 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:48:59.104 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:48:59.104 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:48:59.104 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:48:59.104 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.105 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:48:59.105 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.156805+0000", 2024-09-20T22:48:59.105 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:48:59.105 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.105 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.105 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:48:59.105 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.156805+0000", 2024-09-20T22:48:59.105 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:48:59.105 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.105 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.105 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:48:59.105 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.156806+0000", 2024-09-20T22:48:59.105 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.074e-06 2024-09-20T22:48:59.105 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.106 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.106 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:48:59.106 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.157126+0000", 2024-09-20T22:48:59.106 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00031991399999999997 2024-09-20T22:48:59.106 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.106 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.106 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:48:59.106 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.157136+0000", 2024-09-20T22:48:59.106 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.4499999999999993e-06 2024-09-20T22:48:59.106 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.106 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.106 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:48:59.106 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.157143+0000", 2024-09-20T22:48:59.106 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.7649999999999993e-06 2024-09-20T22:48:59.107 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.107 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.107 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:48:59.107 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.157188+0000", 2024-09-20T22:48:59.107 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.4885000000000001e-05 2024-09-20T22:48:59.107 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.107 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.107 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:48:59.107 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.157208+0000", 2024-09-20T22:48:59.107 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.0015000000000001e-05 2024-09-20T22:48:59.107 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.107 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.107 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:48:59.107 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.169936+0000", 2024-09-20T22:48:59.108 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.012728204 2024-09-20T22:48:59.108 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.108 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.108 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:48:59.108 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.169986+0000", 2024-09-20T22:48:59.108 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.9354999999999998e-05 2024-09-20T22:48:59.108 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:59.108 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:48:59.108 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:59.108 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.108 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.108 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24652.0:1654 7.19 e206/139 7:9970b578:::smithi08477203-15:head v 206'114, mlcod=205'111)", 2024-09-20T22:48:59.108 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:44:45.167994+0000", 2024-09-20T22:48:59.108 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 253.68562081100001, 2024-09-20T22:48:59.109 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.01345262, 2024-09-20T22:48:59.109 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:48:59.109 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:48:59.109 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:48:59.109 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:48:59.109 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.109 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:48:59.109 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:45.167994+0000", 2024-09-20T22:48:59.109 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:48:59.109 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.109 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.109 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:48:59.109 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:45.167994+0000", 2024-09-20T22:48:59.109 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:48:59.109 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.110 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.110 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:48:59.110 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:45.167996+0000", 2024-09-20T22:48:59.110 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.5100000000000001e-06 2024-09-20T22:48:59.110 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.110 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.110 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:48:59.110 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:45.175169+0000", 2024-09-20T22:48:59.110 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0071732740000000003 2024-09-20T22:48:59.110 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.110 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.110 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:48:59.110 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:45.175208+0000", 2024-09-20T22:48:59.110 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.8597000000000003e-05 2024-09-20T22:48:59.111 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.111 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.111 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:48:59.111 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:45.175223+0000", 2024-09-20T22:48:59.111 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.4627000000000001e-05 2024-09-20T22:48:59.111 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.111 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.111 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:48:59.111 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:45.175304+0000", 2024-09-20T22:48:59.111 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.1111000000000003e-05 2024-09-20T22:48:59.111 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.111 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.111 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:48:59.111 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:45.175336+0000", 2024-09-20T22:48:59.111 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.1945e-05 2024-09-20T22:48:59.112 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.112 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.112 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:48:59.112 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:45.181392+0000", 2024-09-20T22:48:59.112 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0060566630000000003 2024-09-20T22:48:59.112 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.112 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.112 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:48:59.112 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:45.181446+0000", 2024-09-20T22:48:59.112 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.3893000000000001e-05 2024-09-20T22:48:59.112 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:59.112 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:48:59.112 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:59.112 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.112 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.113 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24652.0:1799 7.17 e229/139 7:ea720b85:::smithi08477203-26:head v 229'168, mlcod=222'164)", 2024-09-20T22:48:59.113 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:45:08.171867+0000", 2024-09-20T22:48:59.113 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 230.68174759300001, 2024-09-20T22:48:59.113 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014913454, 2024-09-20T22:48:59.113 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:48:59.113 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:48:59.113 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:48:59.113 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:48:59.113 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.113 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:48:59.113 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.171867+0000", 2024-09-20T22:48:59.113 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:48:59.113 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.113 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.114 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:48:59.114 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.171867+0000", 2024-09-20T22:48:59.114 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:48:59.114 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.114 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.114 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:48:59.114 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.171867+0000", 2024-09-20T22:48:59.114 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.3799999999999999e-07 2024-09-20T22:48:59.114 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.114 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.114 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:48:59.114 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.173016+0000", 2024-09-20T22:48:59.114 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0011485740000000001 2024-09-20T22:48:59.114 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.115 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.115 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:48:59.115 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.173024+0000", 2024-09-20T22:48:59.115 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.4780000000000008e-06 2024-09-20T22:48:59.115 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.115 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.115 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:48:59.115 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.173034+0000", 2024-09-20T22:48:59.115 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.7860000000000008e-06 2024-09-20T22:48:59.115 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.115 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.115 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:48:59.115 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.173098+0000", 2024-09-20T22:48:59.116 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.4405000000000004e-05 2024-09-20T22:48:59.116 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.116 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.116 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:48:59.116 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.173125+0000", 2024-09-20T22:48:59.116 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.6950000000000001e-05 2024-09-20T22:48:59.116 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.116 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.116 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:48:59.116 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.186711+0000", 2024-09-20T22:48:59.116 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.01358608 2024-09-20T22:48:59.116 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.116 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.116 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:48:59.116 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.186780+0000", 2024-09-20T22:48:59.117 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.8942999999999994e-05 2024-09-20T22:48:59.117 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:59.117 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:48:59.117 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:59.117 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.117 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.117 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:2322 7.8 7:12e9c8db:::smithi08477203-11:head [append 0~0] snapc 55=[55,53,52,51,50,4b,47,3e] ondisk+write+known_if_redirected+supports_pool_eio e258)", 2024-09-20T22:48:59.117 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:45:37.176152+0000", 2024-09-20T22:48:59.117 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 201.677462803, 2024-09-20T22:48:59.117 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013033472000000001, 2024-09-20T22:48:59.117 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:48:59.117 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:48:59.117 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:48:59.117 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:48:59.118 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:48:59.118 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:48:59.118 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 2322 2024-09-20T22:48:59.118 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.118 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:48:59.118 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.118 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:48:59.118 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.176152+0000", 2024-09-20T22:48:59.118 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:48:59.118 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.118 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.118 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:48:59.118 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.176152+0000", 2024-09-20T22:48:59.118 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:48:59.119 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.119 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.119 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:48:59.119 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.176152+0000", 2024-09-20T22:48:59.119 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.8899999999999999e-07 2024-09-20T22:48:59.119 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.119 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.119 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:48:59.119 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.176156+0000", 2024-09-20T22:48:59.119 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.2669999999999998e-06 2024-09-20T22:48:59.119 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.119 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.119 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:48:59.119 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.176162+0000", 2024-09-20T22:48:59.120 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.1589999999999998e-06 2024-09-20T22:48:59.120 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.120 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.120 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:48:59.120 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.176167+0000", 2024-09-20T22:48:59.120 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.3040000000000004e-06 2024-09-20T22:48:59.120 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.120 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.120 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:48:59.120 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.176212+0000", 2024-09-20T22:48:59.120 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.5448999999999998e-05 2024-09-20T22:48:59.120 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.120 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.120 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting_for_map not empty", 2024-09-20T22:48:59.121 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.176218+0000", 2024-09-20T22:48:59.121 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.5520000000000004e-06 2024-09-20T22:48:59.121 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.121 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.121 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:48:59.121 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.182721+0000", 2024-09-20T22:48:59.121 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0065028550000000001 2024-09-20T22:48:59.121 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.121 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.121 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:48:59.121 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.182787+0000", 2024-09-20T22:48:59.121 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.5995999999999996e-05 2024-09-20T22:48:59.121 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.121 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.122 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 3,7", 2024-09-20T22:48:59.122 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.183002+0000", 2024-09-20T22:48:59.122 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00021516699999999999 2024-09-20T22:48:59.122 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.122 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.122 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:48:59.122 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.185137+0000", 2024-09-20T22:48:59.122 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0021350570000000001 2024-09-20T22:48:59.122 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.122 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.122 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:48:59.122 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.185875+0000", 2024-09-20T22:48:59.122 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00073794799999999999 2024-09-20T22:48:59.122 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.123 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.123 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:48:59.123 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.188620+0000", 2024-09-20T22:48:59.123 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0027446229999999999 2024-09-20T22:48:59.123 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.123 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.123 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:48:59.123 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.189088+0000", 2024-09-20T22:48:59.123 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00046794499999999999 2024-09-20T22:48:59.123 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.123 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.123 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:48:59.123 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.189185+0000", 2024-09-20T22:48:59.123 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.7560999999999996e-05 2024-09-20T22:48:59.124 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:59.124 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:48:59.124 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:59.124 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.124 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.124 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24652.0:2318 7.5 e258/137 7:a160b5d4:::smithi08477203-16:head v 258'43, mlcod=239'39)", 2024-09-20T22:48:59.124 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:45:37.192419+0000", 2024-09-20T22:48:59.124 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 201.66119550600001, 2024-09-20T22:48:59.124 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013974723, 2024-09-20T22:48:59.124 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:48:59.124 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:48:59.124 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:48:59.124 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:48:59.124 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.124 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:48:59.125 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.192419+0000", 2024-09-20T22:48:59.125 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:48:59.125 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.125 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.125 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:48:59.125 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.192419+0000", 2024-09-20T22:48:59.125 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:48:59.125 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.125 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.125 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:48:59.125 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.192420+0000", 2024-09-20T22:48:59.125 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.3799999999999997e-07 2024-09-20T22:48:59.125 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.125 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.126 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:48:59.126 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.204088+0000", 2024-09-20T22:48:59.126 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011668883 2024-09-20T22:48:59.126 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.126 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.126 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:48:59.126 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.204104+0000", 2024-09-20T22:48:59.126 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.5437000000000001e-05 2024-09-20T22:48:59.126 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.126 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.126 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:48:59.126 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.204112+0000", 2024-09-20T22:48:59.126 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.2519999999999999e-06 2024-09-20T22:48:59.126 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.126 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.127 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:48:59.127 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.204168+0000", 2024-09-20T22:48:59.127 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.5977999999999998e-05 2024-09-20T22:48:59.127 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.127 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.127 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:48:59.127 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.204188+0000", 2024-09-20T22:48:59.127 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.9400999999999999e-05 2024-09-20T22:48:59.127 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.127 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.127 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:48:59.127 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.206340+0000", 2024-09-20T22:48:59.127 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0021527159999999998 2024-09-20T22:48:59.127 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.128 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.128 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:48:59.128 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.206394+0000", 2024-09-20T22:48:59.128 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.3418000000000003e-05 2024-09-20T22:48:59.128 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:59.128 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:48:59.128 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:59.128 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.128 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.128 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(osd.6.0:726 7.19 e260/139 7:9eac4560:::smithi08477203-30:4c v 260'159, mlcod=259'156)", 2024-09-20T22:48:59.128 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:45:39.197025+0000", 2024-09-20T22:48:59.128 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 199.656589596, 2024-09-20T22:48:59.128 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.018150454, 2024-09-20T22:48:59.128 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:48:59.129 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:48:59.129 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:48:59.129 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:48:59.129 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.129 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:48:59.129 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.197025+0000", 2024-09-20T22:48:59.129 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:48:59.129 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.129 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.129 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:48:59.129 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.197025+0000", 2024-09-20T22:48:59.129 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:48:59.129 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.129 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.130 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:48:59.130 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.197025+0000", 2024-09-20T22:48:59.130 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.7599999999999998e-07 2024-09-20T22:48:59.130 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.130 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.130 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:48:59.130 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.197033+0000", 2024-09-20T22:48:59.130 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.9119999999999998e-06 2024-09-20T22:48:59.130 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.130 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.130 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:48:59.130 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.197042+0000", 2024-09-20T22:48:59.130 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.3349999999999996e-06 2024-09-20T22:48:59.130 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.131 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.131 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:48:59.131 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.197050+0000", 2024-09-20T22:48:59.131 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.7230000000000004e-06 2024-09-20T22:48:59.131 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.131 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.131 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:48:59.131 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.200756+0000", 2024-09-20T22:48:59.131 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0037059749999999998 2024-09-20T22:48:59.131 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.131 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.131 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:48:59.131 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.200778+0000", 2024-09-20T22:48:59.131 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.2140000000000001e-05 2024-09-20T22:48:59.131 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.132 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.132 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:48:59.132 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.215131+0000", 2024-09-20T22:48:59.132 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014353054000000001 2024-09-20T22:48:59.132 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.132 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.132 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:48:59.132 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.215175+0000", 2024-09-20T22:48:59.132 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.4039000000000003e-05 2024-09-20T22:48:59.132 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:59.132 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:48:59.132 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:59.132 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.132 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.133 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(osd.7.0:152 7.5 e269/260 7:a160b5d4:::smithi08477203-16:55 v 269'55, mlcod=263'51)", 2024-09-20T22:48:59.133 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:45:48.173893+0000", 2024-09-20T22:48:59.133 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 190.679721855, 2024-09-20T22:48:59.133 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014936935, 2024-09-20T22:48:59.133 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:48:59.133 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:48:59.133 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:48:59.133 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:48:59.133 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.133 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:48:59.133 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.173893+0000", 2024-09-20T22:48:59.133 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:48:59.133 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.133 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.134 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:48:59.134 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.173893+0000", 2024-09-20T22:48:59.134 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:48:59.134 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.134 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.134 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:48:59.134 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.173893+0000", 2024-09-20T22:48:59.134 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.8800000000000002e-07 2024-09-20T22:48:59.134 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.134 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.134 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:48:59.134 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.173916+0000", 2024-09-20T22:48:59.134 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.2889999999999999e-05 2024-09-20T22:48:59.134 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.135 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.135 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:48:59.135 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.173924+0000", 2024-09-20T22:48:59.135 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.4420000000000001e-06 2024-09-20T22:48:59.135 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.135 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.135 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:48:59.135 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.173932+0000", 2024-09-20T22:48:59.135 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.5299999999999996e-06 2024-09-20T22:48:59.135 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.135 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.135 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:48:59.135 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.174011+0000", 2024-09-20T22:48:59.136 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.9342e-05 2024-09-20T22:48:59.136 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.136 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.136 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:48:59.136 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.174047+0000", 2024-09-20T22:48:59.136 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.5237999999999997e-05 2024-09-20T22:48:59.136 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.136 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.136 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:48:59.136 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.188782+0000", 2024-09-20T22:48:59.136 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014735599 2024-09-20T22:48:59.136 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.136 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.137 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:48:59.137 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.188830+0000", 2024-09-20T22:48:59.137 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.7206000000000002e-05 2024-09-20T22:48:59.137 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:59.137 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:48:59.137 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:59.137 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.137 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.137 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(osd.7.0:153 7.17 e269/139 7:ea720b85:::smithi08477203-26:55 v 269'225, mlcod=263'221)", 2024-09-20T22:48:59.137 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:45:48.173991+0000", 2024-09-20T22:48:59.137 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 190.67962337399999, 2024-09-20T22:48:59.137 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013064868, 2024-09-20T22:48:59.137 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:48:59.138 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:48:59.138 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:48:59.138 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:48:59.138 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.138 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:48:59.138 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.173991+0000", 2024-09-20T22:48:59.138 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:48:59.138 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.138 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.138 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:48:59.138 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.173991+0000", 2024-09-20T22:48:59.138 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:48:59.138 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.138 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.139 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:48:59.139 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.173991+0000", 2024-09-20T22:48:59.139 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.9999999999999999e-07 2024-09-20T22:48:59.139 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.139 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.139 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:48:59.139 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.173996+0000", 2024-09-20T22:48:59.139 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.3080000000000001e-06 2024-09-20T22:48:59.139 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.139 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.139 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:48:59.139 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.174001+0000", 2024-09-20T22:48:59.139 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.203e-06 2024-09-20T22:48:59.139 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.140 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.140 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:48:59.140 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.174005+0000", 2024-09-20T22:48:59.140 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.4880000000000001e-06 2024-09-20T22:48:59.140 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.140 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.140 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:48:59.140 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.175600+0000", 2024-09-20T22:48:59.140 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015942459999999999 2024-09-20T22:48:59.140 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.140 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.140 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:48:59.140 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.175631+0000", 2024-09-20T22:48:59.141 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.1816000000000002e-05 2024-09-20T22:48:59.141 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.141 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.141 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:48:59.141 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.187003+0000", 2024-09-20T22:48:59.141 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011372006 2024-09-20T22:48:59.141 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.141 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.141 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:48:59.141 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.187056+0000", 2024-09-20T22:48:59.141 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.2500999999999998e-05 2024-09-20T22:48:59.141 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:59.141 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:48:59.141 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:59.142 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.142 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.142 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(osd.6.0:767 7.12 e269/263 7:4fa4b2b3:::smithi08477203-21:53 v 269'262, mlcod=265'259)", 2024-09-20T22:48:59.142 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:45:48.175130+0000", 2024-09-20T22:48:59.142 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 190.678484185, 2024-09-20T22:48:59.142 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013842727000000001, 2024-09-20T22:48:59.142 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:48:59.142 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:48:59.142 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:48:59.142 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:48:59.142 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.142 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:48:59.142 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.175130+0000", 2024-09-20T22:48:59.142 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:48:59.143 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.143 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.143 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:48:59.143 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.175130+0000", 2024-09-20T22:48:59.143 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:48:59.143 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.143 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.143 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:48:59.143 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.175130+0000", 2024-09-20T22:48:59.143 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.98e-07 2024-09-20T22:48:59.143 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.143 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.143 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:48:59.144 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.175139+0000", 2024-09-20T22:48:59.144 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.5909999999999996e-06 2024-09-20T22:48:59.144 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.144 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.144 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:48:59.144 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.175144+0000", 2024-09-20T22:48:59.144 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.361e-06 2024-09-20T22:48:59.144 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.144 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.144 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:48:59.144 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.175156+0000", 2024-09-20T22:48:59.144 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1236999999999999e-05 2024-09-20T22:48:59.144 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.145 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.145 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:48:59.145 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.176359+0000", 2024-09-20T22:48:59.145 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0012036060000000001 2024-09-20T22:48:59.145 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.145 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.145 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:48:59.145 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.176380+0000", 2024-09-20T22:48:59.145 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.0497999999999999e-05 2024-09-20T22:48:59.145 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.145 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.145 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:48:59.145 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.188936+0000", 2024-09-20T22:48:59.145 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.012556286999999999 2024-09-20T22:48:59.146 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.146 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.146 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:48:59.146 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.188973+0000", 2024-09-20T22:48:59.146 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.6949000000000001e-05 2024-09-20T22:48:59.146 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:59.146 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:48:59.146 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:59.146 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.146 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.146 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24652.0:2853 7.10 e303/139 7:0f524219:::smithi08477203-50:head v 303'85, mlcod=301'82)", 2024-09-20T22:48:59.146 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:46:23.293837+0000", 2024-09-20T22:48:59.146 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 155.55977747099999, 2024-09-20T22:48:59.146 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013368555000000001, 2024-09-20T22:48:59.147 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:48:59.147 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:48:59.147 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:48:59.147 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:48:59.147 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.147 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:48:59.147 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.293837+0000", 2024-09-20T22:48:59.147 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:48:59.147 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.147 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.147 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:48:59.147 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.293837+0000", 2024-09-20T22:48:59.147 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:48:59.147 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.148 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.148 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:48:59.148 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.293839+0000", 2024-09-20T22:48:59.148 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.7239999999999999e-06 2024-09-20T22:48:59.148 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.148 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.148 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:48:59.148 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.297226+0000", 2024-09-20T22:48:59.148 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.003387583 2024-09-20T22:48:59.148 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.148 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.148 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:48:59.148 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.297253+0000", 2024-09-20T22:48:59.148 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.6849e-05 2024-09-20T22:48:59.149 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.149 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.149 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:48:59.149 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.297266+0000", 2024-09-20T22:48:59.149 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2481e-05 2024-09-20T22:48:59.149 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.149 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.149 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:48:59.149 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.297362+0000", 2024-09-20T22:48:59.149 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.6347000000000004e-05 2024-09-20T22:48:59.149 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.149 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.149 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:48:59.149 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.297394+0000", 2024-09-20T22:48:59.150 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.1888999999999999e-05 2024-09-20T22:48:59.150 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.150 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.150 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:48:59.150 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.307147+0000", 2024-09-20T22:48:59.150 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0097533719999999997 2024-09-20T22:48:59.150 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.150 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.150 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:48:59.150 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.307206+0000", 2024-09-20T22:48:59.150 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.8310000000000002e-05 2024-09-20T22:48:59.150 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:59.150 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:48:59.150 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:59.151 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.151 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.151 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:3030 7.e 7:71b5c915:::smithi08477203-6:head [copy-from ver 216] snapc 72=[72,71,70,6e,6c,6b,6a,68] ondisk+write+known_if_redirected+supports_pool_eio e317)", 2024-09-20T22:48:59.151 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:46:37.293029+0000", 2024-09-20T22:48:59.151 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 141.560585113, 2024-09-20T22:48:59.151 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.019929916999999998, 2024-09-20T22:48:59.151 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:48:59.151 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:48:59.151 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:48:59.151 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:48:59.151 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:48:59.151 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:48:59.151 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 3030 2024-09-20T22:48:59.151 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.151 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:48:59.152 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.152 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:48:59.152 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.293029+0000", 2024-09-20T22:48:59.152 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:48:59.152 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.152 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.152 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:48:59.152 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.293029+0000", 2024-09-20T22:48:59.152 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:48:59.152 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.152 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.152 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:48:59.152 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.293031+0000", 2024-09-20T22:48:59.152 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.4190000000000001e-06 2024-09-20T22:48:59.153 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.153 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.153 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:48:59.153 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.293040+0000", 2024-09-20T22:48:59.153 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.6039999999999999e-06 2024-09-20T22:48:59.153 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.153 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.153 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:48:59.153 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.293054+0000", 2024-09-20T22:48:59.153 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.4014e-05 2024-09-20T22:48:59.153 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.153 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.153 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:48:59.153 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.293062+0000", 2024-09-20T22:48:59.154 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.8820000000000001e-06 2024-09-20T22:48:59.154 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.154 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.154 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:48:59.154 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.293121+0000", 2024-09-20T22:48:59.154 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.8977999999999997e-05 2024-09-20T22:48:59.154 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.154 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.154 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op must wait for map", 2024-09-20T22:48:59.154 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.293125+0000", 2024-09-20T22:48:59.154 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.2089999999999999e-06 2024-09-20T22:48:59.154 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.154 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.155 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:48:59.155 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.298805+0000", 2024-09-20T22:48:59.155 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0056790870000000002 2024-09-20T22:48:59.155 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.155 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.155 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:48:59.155 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.298886+0000", 2024-09-20T22:48:59.155 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.1363e-05 2024-09-20T22:48:59.155 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.155 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.155 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 3,5", 2024-09-20T22:48:59.155 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.305883+0000", 2024-09-20T22:48:59.155 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0069966159999999998 2024-09-20T22:48:59.155 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.156 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.156 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:48:59.156 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.312556+0000", 2024-09-20T22:48:59.156 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0066737009999999998 2024-09-20T22:48:59.156 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.156 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.156 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:48:59.156 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.312587+0000", 2024-09-20T22:48:59.156 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.0286000000000001e-05 2024-09-20T22:48:59.156 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.156 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.156 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:48:59.156 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.312890+0000", 2024-09-20T22:48:59.156 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00030374800000000001 2024-09-20T22:48:59.156 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.157 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.157 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:48:59.157 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.312932+0000", 2024-09-20T22:48:59.157 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.1621e-05 2024-09-20T22:48:59.157 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.157 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.157 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:48:59.157 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.312959+0000", 2024-09-20T22:48:59.157 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.7389000000000002e-05 2024-09-20T22:48:59.157 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:59.157 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:48:59.157 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:59.157 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.158 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.158 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:3031 7.e 7:71b5c915:::smithi08477203-6:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e317)", 2024-09-20T22:48:59.158 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:46:37.293082+0000", 2024-09-20T22:48:59.158 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 141.56053273500001, 2024-09-20T22:48:59.158 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.016134757, 2024-09-20T22:48:59.158 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:48:59.158 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:48:59.158 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-20T22:48:59.158 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:48:59.158 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:48:59.158 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:48:59.158 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 3031 2024-09-20T22:48:59.158 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.158 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:48:59.159 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.159 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:48:59.159 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.293082+0000", 2024-09-20T22:48:59.159 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:48:59.159 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.159 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.159 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:48:59.159 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.293082+0000", 2024-09-20T22:48:59.159 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:48:59.159 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.159 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.159 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:48:59.159 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.293082+0000", 2024-09-20T22:48:59.160 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.1799999999999995e-07 2024-09-20T22:48:59.160 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.160 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.160 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:48:59.160 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.293085+0000", 2024-09-20T22:48:59.160 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.937e-06 2024-09-20T22:48:59.160 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.160 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.160 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:48:59.160 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.293093+0000", 2024-09-20T22:48:59.160 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.554e-06 2024-09-20T22:48:59.160 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.160 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.161 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:48:59.161 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.293098+0000", 2024-09-20T22:48:59.161 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.643e-06 2024-09-20T22:48:59.161 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.161 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.161 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:48:59.161 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.293164+0000", 2024-09-20T22:48:59.161 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.5806e-05 2024-09-20T22:48:59.161 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.161 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.161 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting_for_map not empty", 2024-09-20T22:48:59.161 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.293168+0000", 2024-09-20T22:48:59.161 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.2459999999999999e-06 2024-09-20T22:48:59.161 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.162 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.162 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:48:59.162 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.299033+0000", 2024-09-20T22:48:59.162 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0058656699999999999 2024-09-20T22:48:59.162 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.162 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.162 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for blocked object", 2024-09-20T22:48:59.162 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.299069+0000", 2024-09-20T22:48:59.162 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.5392999999999998e-05 2024-09-20T22:48:59.162 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.162 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.162 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:48:59.162 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.309088+0000", 2024-09-20T22:48:59.163 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010019287999999999 2024-09-20T22:48:59.163 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.163 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.163 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:48:59.163 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.309150+0000", 2024-09-20T22:48:59.163 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.2008000000000007e-05 2024-09-20T22:48:59.163 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.163 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.163 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:48:59.163 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.309216+0000", 2024-09-20T22:48:59.163 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.6594000000000006e-05 2024-09-20T22:48:59.163 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:59.163 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:48:59.163 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:59.164 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.164 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.164 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:3290 7.8 7:12e9c8db:::smithi08477203-11:head [copy-from ver 675] snapc 7f=[7f,7e,7d,7c,7b,7a,79,78,74,73,6b] ondisk+write+known_if_redirected+supports_pool_eio e340)", 2024-09-20T22:48:59.164 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:47:00.706846+0000", 2024-09-20T22:48:59.164 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 118.146768876, 2024-09-20T22:48:59.164 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.018408746, 2024-09-20T22:48:59.164 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:48:59.164 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:48:59.164 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:48:59.164 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:48:59.164 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:48:59.164 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:48:59.164 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 3290 2024-09-20T22:48:59.165 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.165 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:48:59.165 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.165 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:48:59.165 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.706846+0000", 2024-09-20T22:48:59.165 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:48:59.165 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.165 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.165 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:48:59.165 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.706846+0000", 2024-09-20T22:48:59.165 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:48:59.165 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.165 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.165 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:48:59.166 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.706848+0000", 2024-09-20T22:48:59.166 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.2340000000000001e-06 2024-09-20T22:48:59.166 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.166 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.166 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:48:59.166 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.706859+0000", 2024-09-20T22:48:59.166 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1001e-05 2024-09-20T22:48:59.166 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.166 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.166 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:48:59.166 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.706879+0000", 2024-09-20T22:48:59.166 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.0556000000000001e-05 2024-09-20T22:48:59.166 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.166 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.167 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:48:59.167 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.706890+0000", 2024-09-20T22:48:59.167 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0328e-05 2024-09-20T22:48:59.167 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.167 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.167 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:48:59.167 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.706973+0000", 2024-09-20T22:48:59.167 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.3574999999999999e-05 2024-09-20T22:48:59.167 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.167 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.167 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:48:59.167 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.707042+0000", 2024-09-20T22:48:59.167 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.8987999999999997e-05 2024-09-20T22:48:59.168 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.168 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.168 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 3,7", 2024-09-20T22:48:59.168 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.717491+0000", 2024-09-20T22:48:59.168 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010448618 2024-09-20T22:48:59.168 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.168 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.168 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:48:59.168 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.722875+0000", 2024-09-20T22:48:59.168 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0053846190000000002 2024-09-20T22:48:59.168 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.168 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.168 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:48:59.168 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.724841+0000", 2024-09-20T22:48:59.169 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0019651579999999998 2024-09-20T22:48:59.169 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.169 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.169 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:48:59.169 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.725168+0000", 2024-09-20T22:48:59.169 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00032734499999999999 2024-09-20T22:48:59.169 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.169 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.169 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:48:59.169 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.725218+0000", 2024-09-20T22:48:59.169 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.0123000000000001e-05 2024-09-20T22:48:59.169 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.169 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.169 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:48:59.169 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.725254+0000", 2024-09-20T22:48:59.170 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.6201000000000001e-05 2024-09-20T22:48:59.170 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:59.170 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:48:59.170 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:59.170 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.170 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.170 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:3291 7.8 7:12e9c8db:::smithi08477203-11:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e340)", 2024-09-20T22:48:59.170 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:47:00.706927+0000", 2024-09-20T22:48:59.170 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 118.14668723699999, 2024-09-20T22:48:59.170 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014159255000000001, 2024-09-20T22:48:59.170 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:48:59.170 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:48:59.170 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-20T22:48:59.170 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:48:59.171 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:48:59.171 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:48:59.171 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 3291 2024-09-20T22:48:59.171 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.171 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:48:59.171 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.171 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:48:59.171 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.706927+0000", 2024-09-20T22:48:59.171 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:48:59.171 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.171 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.171 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:48:59.171 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.706927+0000", 2024-09-20T22:48:59.172 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:48:59.172 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.172 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.172 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:48:59.172 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.706928+0000", 2024-09-20T22:48:59.172 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.4099999999999999e-07 2024-09-20T22:48:59.172 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.172 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.172 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:48:59.172 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.706941+0000", 2024-09-20T22:48:59.172 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3335e-05 2024-09-20T22:48:59.172 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.172 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.172 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:48:59.173 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.706948+0000", 2024-09-20T22:48:59.173 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.3350000000000001e-06 2024-09-20T22:48:59.173 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.173 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.173 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:48:59.173 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.706954+0000", 2024-09-20T22:48:59.173 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.9190000000000004e-06 2024-09-20T22:48:59.173 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.173 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.173 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:48:59.173 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.707166+0000", 2024-09-20T22:48:59.173 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00021159 2024-09-20T22:48:59.173 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.173 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.173 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for blocked object", 2024-09-20T22:48:59.174 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.707217+0000", 2024-09-20T22:48:59.174 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.1167e-05 2024-09-20T22:48:59.174 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.174 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.174 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:48:59.174 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.720939+0000", 2024-09-20T22:48:59.174 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013721458000000001 2024-09-20T22:48:59.174 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.174 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.174 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:48:59.174 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.721010+0000", 2024-09-20T22:48:59.174 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.1570999999999996e-05 2024-09-20T22:48:59.174 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.174 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.174 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:48:59.175 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.721086+0000", 2024-09-20T22:48:59.175 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.6339000000000002e-05 2024-09-20T22:48:59.175 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:59.175 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:48:59.175 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:59.175 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.175 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.175 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(osd.5.0:167 7.2 e347/169 7:45b54b6b:::smithi08477203-2:81 v 347'140, mlcod=345'136)", 2024-09-20T22:48:59.175 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:47:08.204691+0000", 2024-09-20T22:48:59.175 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 110.64892313599999, 2024-09-20T22:48:59.175 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.01335918, 2024-09-20T22:48:59.175 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:48:59.175 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:48:59.175 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:48:59.175 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:48:59.176 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.176 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:48:59.176 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:08.204691+0000", 2024-09-20T22:48:59.176 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:48:59.176 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.176 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.176 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:48:59.176 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:08.204691+0000", 2024-09-20T22:48:59.176 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:48:59.176 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.176 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.176 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:48:59.176 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:08.204692+0000", 2024-09-20T22:48:59.176 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.5400000000000001e-07 2024-09-20T22:48:59.177 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.177 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.177 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:48:59.177 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:08.204697+0000", 2024-09-20T22:48:59.177 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.614e-06 2024-09-20T22:48:59.177 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.177 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.177 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:48:59.177 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:08.204705+0000", 2024-09-20T22:48:59.177 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.8259999999999993e-06 2024-09-20T22:48:59.177 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.177 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.177 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:48:59.178 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:08.204715+0000", 2024-09-20T22:48:59.178 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.8279999999999998e-06 2024-09-20T22:48:59.178 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.178 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.178 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:48:59.178 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:08.204801+0000", 2024-09-20T22:48:59.178 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.6263999999999999e-05 2024-09-20T22:48:59.178 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.178 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.178 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:48:59.178 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:08.204823+0000", 2024-09-20T22:48:59.178 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.1971999999999999e-05 2024-09-20T22:48:59.179 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.179 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.179 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:48:59.179 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:08.217961+0000", 2024-09-20T22:48:59.179 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013137618 2024-09-20T22:48:59.179 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.179 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.179 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:48:59.179 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:08.218050+0000", 2024-09-20T22:48:59.179 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.9504000000000002e-05 2024-09-20T22:48:59.179 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:59.179 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:48:59.179 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:59.179 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.180 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.180 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24652.0:3760 7.13 e383/260 7:cae5792a:::smithi08477203-32:head v 383'78, mlcod=377'74)", 2024-09-20T22:48:59.180 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:47:43.291645+0000", 2024-09-20T22:48:59.180 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 75.561969263999998, 2024-09-20T22:48:59.180 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.012696648, 2024-09-20T22:48:59.180 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:48:59.180 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:48:59.180 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:48:59.180 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:48:59.180 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.180 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:48:59.180 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:43.291645+0000", 2024-09-20T22:48:59.180 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:48:59.180 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.181 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.181 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:48:59.181 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:43.291645+0000", 2024-09-20T22:48:59.181 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:48:59.181 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.181 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.181 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:48:59.181 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:43.291645+0000", 2024-09-20T22:48:59.181 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.0800000000000001e-07 2024-09-20T22:48:59.181 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.181 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.181 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:48:59.181 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:43.294764+0000", 2024-09-20T22:48:59.181 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0031183420000000001 2024-09-20T22:48:59.182 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.182 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.182 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:48:59.182 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:43.294777+0000", 2024-09-20T22:48:59.182 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3465999999999999e-05 2024-09-20T22:48:59.182 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.182 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.182 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:48:59.182 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:43.294788+0000", 2024-09-20T22:48:59.182 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0662999999999999e-05 2024-09-20T22:48:59.182 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.182 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.182 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:48:59.182 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:43.294952+0000", 2024-09-20T22:48:59.183 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00016369799999999999 2024-09-20T22:48:59.183 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.183 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.183 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:48:59.183 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:43.294978+0000", 2024-09-20T22:48:59.183 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.6414000000000002e-05 2024-09-20T22:48:59.183 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.183 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.183 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:48:59.183 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:43.304313+0000", 2024-09-20T22:48:59.183 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0093352039999999997 2024-09-20T22:48:59.183 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.183 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.184 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:48:59.184 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:43.304342+0000", 2024-09-20T22:48:59.184 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.8552999999999999e-05 2024-09-20T22:48:59.184 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:59.184 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:48:59.184 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:59.184 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.184 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.184 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24652.0:4534 7.12 e456/421 7:4b06cbed:::smithi08477203-49:head v 456'446, mlcod=454'443)", 2024-09-20T22:48:59.184 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:48:56.245037+0000", 2024-09-20T22:48:59.184 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 2.608577243, 2024-09-20T22:48:59.184 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013221933999999999, 2024-09-20T22:48:59.184 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:48:59.184 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:48:59.185 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:48:59.185 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:48:59.185 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.185 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:48:59.185 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:56.245037+0000", 2024-09-20T22:48:59.185 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:48:59.185 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.185 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.185 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:48:59.185 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:56.245037+0000", 2024-09-20T22:48:59.185 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:48:59.185 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.185 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.186 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:48:59.186 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:56.245037+0000", 2024-09-20T22:48:59.186 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.4600000000000001e-07 2024-09-20T22:48:59.186 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.186 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.186 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:48:59.186 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:56.248142+0000", 2024-09-20T22:48:59.186 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0031044839999999998 2024-09-20T22:48:59.186 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.186 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.186 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:48:59.186 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:56.248174+0000", 2024-09-20T22:48:59.186 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.2555000000000003e-05 2024-09-20T22:48:59.187 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.187 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.187 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:48:59.187 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:56.248188+0000", 2024-09-20T22:48:59.187 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.308e-05 2024-09-20T22:48:59.187 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.187 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.187 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:48:59.187 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:56.248292+0000", 2024-09-20T22:48:59.187 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000104182 2024-09-20T22:48:59.187 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.187 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.187 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:48:59.188 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:56.248318+0000", 2024-09-20T22:48:59.188 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.652e-05 2024-09-20T22:48:59.188 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.188 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.188 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:48:59.188 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:56.258207+0000", 2024-09-20T22:48:59.188 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0098883909999999998 2024-09-20T22:48:59.188 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:48:59.188 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:48:59.188 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:48:59.188 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:56.258259+0000", 2024-09-20T22:48:59.188 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.2475999999999997e-05 2024-09-20T22:48:59.188 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:59.188 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:48:59.188 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:59.189 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:48:59.189 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:48:59.189 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-20T22:48:59.189 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:48:59.210 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:59.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:59.211 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:48:59.216 INFO:tasks.rados.rados.0.smithi084.stdout:1781: done (0 left) 2024-09-20T22:48:59.216 INFO:tasks.rados.rados.0.smithi084.stdout:1782: write (excl) oid 4 current snap is 189 2024-09-20T22:48:59.217 INFO:tasks.rados.rados.0.smithi084.stdout:1782: seq_num 626 ranges {521182=646820,1948669=523219} 2024-09-20T22:48:59.223 INFO:tasks.rados.rados.0.smithi084.stdout:1782: writing smithi08477203-4 from 521182 to 1168002 tid 1 2024-09-20T22:48:59.231 INFO:tasks.rados.rados.0.smithi084.stdout:1782: writing smithi08477203-4 from 1948669 to 2471888 tid 2 2024-09-20T22:48:59.231 INFO:tasks.rados.rados.0.smithi084.stdout:1783: delete oid 25 current snap is 189 2024-09-20T22:48:59.231 INFO:tasks.rados.rados.0.smithi084.stdout:1782: finishing write tid 1 to smithi08477203-4 2024-09-20T22:48:59.235 INFO:tasks.rados.rados.0.smithi084.stdout:1782: finishing write tid 2 to smithi08477203-4 2024-09-20T22:48:59.235 INFO:tasks.rados.rados.0.smithi084.stdout:1782: finishing write tid 3 to smithi08477203-4 2024-09-20T22:48:59.235 INFO:tasks.rados.rados.0.smithi084.stdout:1782: finishing write tid 4 to smithi08477203-4 2024-09-20T22:48:59.235 INFO:tasks.rados.rados.0.smithi084.stdout:1782: oid 4 updating version 0 to 1161 2024-09-20T22:48:59.235 INFO:tasks.rados.rados.0.smithi084.stdout:1782: oid 4 version 1161 is already newer than 1160 2024-09-20T22:48:59.235 INFO:tasks.rados.rados.0.smithi084.stdout:1782: oid 4 updating version 1161 to 1162 2024-09-20T22:48:59.235 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 4 v 1162 (ObjNum 626 snap 189 seq_num 626) dirty exists 2024-09-20T22:48:59.235 INFO:tasks.rados.rados.0.smithi084.stdout:1782: left oid 4 (ObjNum 626 snap 189 seq_num 626) 2024-09-20T22:48:59.236 INFO:tasks.rados.rados.0.smithi084.stdout:1782: done (1 left) 2024-09-20T22:48:59.236 INFO:tasks.rados.rados.0.smithi084.stdout:1783: done (0 left) 2024-09-20T22:48:59.236 INFO:tasks.rados.rados.0.smithi084.stdout:1784: read oid 49 snap -1 2024-09-20T22:48:59.236 INFO:tasks.rados.rados.0.smithi084.stdout:1784: expect (ObjNum 470 snap 130 seq_num 470) 2024-09-20T22:48:59.236 INFO:tasks.rados.rados.0.smithi084.stdout:1785: snap_create 2024-09-20T22:48:59.245 INFO:tasks.rados.rados.0.smithi084.stdout:1784: expect (ObjNum 470 snap 130 seq_num 470) 2024-09-20T22:48:59.313 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:48:59.333 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:59.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:59.334 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:48:59.364 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:59 smithi084 ceph-mon[40024]: pgmap v588: 163 pgs: 3 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 155 active+clean; 356 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail 2024-09-20T22:48:59.364 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:59 smithi084 ceph-mon[40024]: from='client.15969 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T22:48:59.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:48:59 smithi084 ceph-mon[40024]: osdmap e459: 8 total, 8 up, 8 in 2024-09-20T22:48:59.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:59 smithi084 ceph-mon[29774]: pgmap v588: 163 pgs: 3 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 155 active+clean; 356 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail 2024-09-20T22:48:59.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:59 smithi084 ceph-mon[29774]: from='client.15969 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T22:48:59.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:48:59 smithi084 ceph-mon[29774]: osdmap e459: 8 total, 8 up, 8 in 2024-09-20T22:48:59.435 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:48:59.460 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:59.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:59.461 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:48:59.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:59 smithi154 ceph-mon[33264]: pgmap v588: 163 pgs: 3 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 155 active+clean; 356 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail 2024-09-20T22:48:59.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:59 smithi154 ceph-mon[33264]: from='client.15969 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T22:48:59.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:48:59 smithi154 ceph-mon[33264]: osdmap e459: 8 total, 8 up, 8 in 2024-09-20T22:48:59.505 INFO:journalctl@ceph.iscsi.iscsi.a.smithi154.stdout:Sep 20 22:48:59 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-iscsi-iscsi-a[59730]: debug there is no tcmu-runner data available 2024-09-20T22:48:59.562 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:48:59.586 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:59.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:59.587 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:48:59.688 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:48:59.713 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:59.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:59.714 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:48:59.815 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:48:59.838 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:48:59.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:59.839 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:48:59.940 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:48:59.962 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:48:59.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:48:59.963 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:49:00.064 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:49:00.090 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:00.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:00.091 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:49:00.191 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:49:00.217 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:00.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:00.219 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:49:00.219 INFO:tasks.rados.rados.0.smithi084.stdout:1784: done (1 left) 2024-09-20T22:49:00.220 INFO:tasks.rados.rados.0.smithi084.stdout:1785: done (0 left) 2024-09-20T22:49:00.220 INFO:tasks.rados.rados.0.smithi084.stdout:1786: rollback oid 38 current snap is 190 2024-09-20T22:49:00.220 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 38 to 185 2024-09-20T22:49:00.220 INFO:tasks.rados.rados.0.smithi084.stdout:1787: delete oid 36 current snap is 190 2024-09-20T22:49:00.224 INFO:tasks.rados.rados.0.smithi084.stdout:1786: finishing rollback tid 0 to smithi08477203-38 2024-09-20T22:49:00.224 INFO:tasks.rados.rados.0.smithi084.stdout:1787: done (1 left) 2024-09-20T22:49:00.224 INFO:tasks.rados.rados.0.smithi084.stdout:1788: delete oid 27 current snap is 190 2024-09-20T22:49:00.224 INFO:tasks.rados.rados.0.smithi084.stdout:1786: finishing rollback tid 1 to smithi08477203-38 2024-09-20T22:49:00.224 INFO:tasks.rados.rados.0.smithi084.stdout:1786: finishing rollback tid 2 to smithi08477203-38 2024-09-20T22:49:00.224 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 38 v 523 (ObjNum 489 snap 137 seq_num 489) dirty exists 2024-09-20T22:49:00.226 INFO:tasks.rados.rados.0.smithi084.stdout:1786: done (1 left) 2024-09-20T22:49:00.226 INFO:tasks.rados.rados.0.smithi084.stdout:1788: done (0 left) 2024-09-20T22:49:00.226 INFO:tasks.rados.rados.0.smithi084.stdout:1789: rollback oid 19 current snap is 190 2024-09-20T22:49:00.226 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 19 to 154 2024-09-20T22:49:00.227 INFO:tasks.rados.rados.0.smithi084.stdout:1790: read oid 9 snap -1 2024-09-20T22:49:00.227 INFO:tasks.rados.rados.0.smithi084.stdout:1790: expect deleted 2024-09-20T22:49:00.227 INFO:tasks.rados.rados.0.smithi084.stdout:1791: snap_create 2024-09-20T22:49:00.229 INFO:tasks.rados.rados.0.smithi084.stdout:1789: finishing rollback tid 0 to smithi08477203-19 2024-09-20T22:49:00.230 INFO:tasks.rados.rados.0.smithi084.stdout:1789: finishing rollback tid 1 to smithi08477203-19 2024-09-20T22:49:00.230 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 19 v 1166 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-09-20T22:49:00.258 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-20T22:49:00.319 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:49:00.343 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:00.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:00.344 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:49:00.445 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:49:00.468 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:00.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:00.469 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:49:00.474 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:00 smithi084 ceph-mon[40024]: 4.1d scrub starts 2024-09-20T22:49:00.474 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:00 smithi084 ceph-mon[40024]: 4.1d scrub ok 2024-09-20T22:49:00.474 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:00 smithi084 ceph-mon[40024]: 7.8 scrub starts 2024-09-20T22:49:00.474 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:00 smithi084 ceph-mon[40024]: 7.8 scrub ok 2024-09-20T22:49:00.474 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:00 smithi084 ceph-mon[40024]: 5.12 scrub starts 2024-09-20T22:49:00.474 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:00 smithi084 ceph-mon[40024]: 5.12 scrub ok 2024-09-20T22:49:00.475 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:00 smithi084 ceph-mon[40024]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:49:00.475 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:00 smithi084 ceph-mon[40024]: osdmap e460: 8 total, 8 up, 8 in 2024-09-20T22:49:00.475 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:00 smithi084 ceph-mon[29774]: 4.1d scrub starts 2024-09-20T22:49:00.475 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:00 smithi084 ceph-mon[29774]: 4.1d scrub ok 2024-09-20T22:49:00.475 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:00 smithi084 ceph-mon[29774]: 7.8 scrub starts 2024-09-20T22:49:00.475 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:00 smithi084 ceph-mon[29774]: 7.8 scrub ok 2024-09-20T22:49:00.475 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:00 smithi084 ceph-mon[29774]: 5.12 scrub starts 2024-09-20T22:49:00.475 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:00 smithi084 ceph-mon[29774]: 5.12 scrub ok 2024-09-20T22:49:00.475 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:00 smithi084 ceph-mon[29774]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:49:00.476 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:00 smithi084 ceph-mon[29774]: osdmap e460: 8 total, 8 up, 8 in 2024-09-20T22:49:00.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:00 smithi154 ceph-mon[33264]: 4.1d scrub starts 2024-09-20T22:49:00.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:00 smithi154 ceph-mon[33264]: 4.1d scrub ok 2024-09-20T22:49:00.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:00 smithi154 ceph-mon[33264]: 7.8 scrub starts 2024-09-20T22:49:00.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:00 smithi154 ceph-mon[33264]: 7.8 scrub ok 2024-09-20T22:49:00.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:00 smithi154 ceph-mon[33264]: 5.12 scrub starts 2024-09-20T22:49:00.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:00 smithi154 ceph-mon[33264]: 5.12 scrub ok 2024-09-20T22:49:00.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:00 smithi154 ceph-mon[33264]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:49:00.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:00 smithi154 ceph-mon[33264]: osdmap e460: 8 total, 8 up, 8 in 2024-09-20T22:49:00.570 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:49:00.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:00.594 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:00.595 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:49:00.695 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:49:00.697 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:49:00.697 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:49:00.713 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:49:00.714 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:49:00.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:00.718 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:00.719 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:49:00.730 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:49:00.730 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:49:00.746 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-20T22:49:00.746 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T22:49:00.763 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:49:00.763 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:49:00.779 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:49:00.780 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:49:00.797 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:49:00.797 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:49:00.813 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:49:00.813 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:49:00.819 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:49:00.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:00.843 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:00.844 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:49:00.944 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:49:00.967 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:00.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:00.968 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:49:01.069 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:49:01.092 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:01.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:01.093 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:49:01.193 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:49:01.216 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:01.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:01.217 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:49:01.219 INFO:tasks.rados.rados.0.smithi084.stdout:1789: done (2 left) 2024-09-20T22:49:01.219 INFO:tasks.rados.rados.0.smithi084.stdout:1790: done (1 left) 2024-09-20T22:49:01.219 INFO:tasks.rados.rados.0.smithi084.stdout:1791: done (0 left) 2024-09-20T22:49:01.219 INFO:tasks.rados.rados.0.smithi084.stdout:1792: delete oid 43 current snap is 191 2024-09-20T22:49:01.226 INFO:tasks.rados.rados.0.smithi084.stdout:1792: done (0 left) 2024-09-20T22:49:01.226 INFO:tasks.rados.rados.0.smithi084.stdout:1793: read oid 5 snap 163 2024-09-20T22:49:01.226 INFO:tasks.rados.rados.0.smithi084.stdout:1793: expect deleted 2024-09-20T22:49:01.226 INFO:tasks.rados.rados.0.smithi084.stdout:1794: copy_from oid 33 from oid 31 current snap is 191 2024-09-20T22:49:01.226 INFO:tasks.rados.rados.0.smithi084.stdout:1795: rollback oid 36 current snap is 191 2024-09-20T22:49:01.227 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 36 to 179 2024-09-20T22:49:01.227 INFO:tasks.rados.rados.0.smithi084.stdout:1796: write oid 35 current snap is 191 2024-09-20T22:49:01.227 INFO:tasks.rados.rados.0.smithi084.stdout:1796: seq_num 631 ranges {605217=739418,2124726=480491} 2024-09-20T22:49:01.234 INFO:tasks.rados.rados.0.smithi084.stdout:1796: writing smithi08477203-35 from 605217 to 1344635 tid 1 2024-09-20T22:49:01.241 INFO:tasks.rados.rados.0.smithi084.stdout:1796: writing smithi08477203-35 from 2124726 to 2605217 tid 2 2024-09-20T22:49:01.241 INFO:tasks.rados.rados.0.smithi084.stdout:1797: snap_create 2024-09-20T22:49:01.241 INFO:tasks.rados.rados.0.smithi084.stdout:1794: finishing copy_from to smithi08477203-33 2024-09-20T22:49:01.241 INFO:tasks.rados.rados.0.smithi084.stdout:1794: got expected ENOENT (src dne) 2024-09-20T22:49:01.242 INFO:tasks.rados.rados.0.smithi084.stdout:1794: finishing copy_from racing read to smithi08477203-33 2024-09-20T22:49:01.242 INFO:tasks.rados.rados.0.smithi084.stdout:1795: finishing rollback tid 1 to smithi08477203-36 2024-09-20T22:49:01.242 INFO:tasks.rados.rados.0.smithi084.stdout:1795: finishing rollback tid 2 to smithi08477203-36 2024-09-20T22:49:01.242 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 36 v 402 (ObjNum 440 snap 115 seq_num 440) dirty exists 2024-09-20T22:49:01.242 INFO:tasks.rados.rados.0.smithi084.stdout:1796: finishing write tid 1 to smithi08477203-35 2024-09-20T22:49:01.244 INFO:tasks.rados.rados.0.smithi084.stdout:1796: finishing write tid 2 to smithi08477203-35 2024-09-20T22:49:01.244 INFO:tasks.rados.rados.0.smithi084.stdout:1796: finishing write tid 3 to smithi08477203-35 2024-09-20T22:49:01.245 INFO:tasks.rados.rados.0.smithi084.stdout:1796: finishing write tid 4 to smithi08477203-35 2024-09-20T22:49:01.245 INFO:tasks.rados.rados.0.smithi084.stdout:1796: oid 35 updating version 0 to 693 2024-09-20T22:49:01.245 INFO:tasks.rados.rados.0.smithi084.stdout:1796: oid 35 updating version 693 to 694 2024-09-20T22:49:01.245 INFO:tasks.rados.rados.0.smithi084.stdout:1796: oid 35 updating version 694 to 695 2024-09-20T22:49:01.245 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 35 v 695 (ObjNum 631 snap 191 seq_num 631) dirty exists 2024-09-20T22:49:01.245 INFO:tasks.rados.rados.0.smithi084.stdout:1796: left oid 35 (ObjNum 631 snap 191 seq_num 631) 2024-09-20T22:49:01.319 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:49:01.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:01.342 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:01.342 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:49:01.443 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:49:01.464 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:01.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:01.465 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:49:01.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:01 smithi154 ceph-mon[33264]: 7.0 scrub starts 2024-09-20T22:49:01.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:01 smithi154 ceph-mon[33264]: 7.0 scrub ok 2024-09-20T22:49:01.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:01 smithi154 ceph-mon[33264]: pgmap v591: 163 pgs: 2 peering, 2 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 154 active+clean; 367 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 13 MiB/s rd, 1.8 MiB/s wr, 17 op/s 2024-09-20T22:49:01.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:01 smithi154 ceph-mon[33264]: 5.2 scrub starts 2024-09-20T22:49:01.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:01 smithi154 ceph-mon[33264]: 5.2 scrub ok 2024-09-20T22:49:01.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:01 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/236552952' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:49:01.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:01 smithi154 ceph-mon[33264]: 5.1d scrub starts 2024-09-20T22:49:01.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:01 smithi154 ceph-mon[33264]: 5.1d scrub ok 2024-09-20T22:49:01.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:01 smithi154 ceph-mon[33264]: osdmap e461: 8 total, 8 up, 8 in 2024-09-20T22:49:01.566 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:49:01.588 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:01.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:01.589 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:49:01.614 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:01 smithi084 ceph-mon[29774]: 7.0 scrub starts 2024-09-20T22:49:01.614 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:01 smithi084 ceph-mon[29774]: 7.0 scrub ok 2024-09-20T22:49:01.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:01 smithi084 ceph-mon[29774]: pgmap v591: 163 pgs: 2 peering, 2 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 154 active+clean; 367 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 13 MiB/s rd, 1.8 MiB/s wr, 17 op/s 2024-09-20T22:49:01.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:01 smithi084 ceph-mon[29774]: 5.2 scrub starts 2024-09-20T22:49:01.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:01 smithi084 ceph-mon[29774]: 5.2 scrub ok 2024-09-20T22:49:01.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:01 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/236552952' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:49:01.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:01 smithi084 ceph-mon[29774]: 5.1d scrub starts 2024-09-20T22:49:01.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:01 smithi084 ceph-mon[29774]: 5.1d scrub ok 2024-09-20T22:49:01.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:01 smithi084 ceph-mon[29774]: osdmap e461: 8 total, 8 up, 8 in 2024-09-20T22:49:01.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:01 smithi084 ceph-mon[40024]: 7.0 scrub starts 2024-09-20T22:49:01.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:01 smithi084 ceph-mon[40024]: 7.0 scrub ok 2024-09-20T22:49:01.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:01 smithi084 ceph-mon[40024]: pgmap v591: 163 pgs: 2 peering, 2 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 154 active+clean; 367 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 13 MiB/s rd, 1.8 MiB/s wr, 17 op/s 2024-09-20T22:49:01.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:01 smithi084 ceph-mon[40024]: 5.2 scrub starts 2024-09-20T22:49:01.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:01 smithi084 ceph-mon[40024]: 5.2 scrub ok 2024-09-20T22:49:01.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:01 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/236552952' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:49:01.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:01 smithi084 ceph-mon[40024]: 5.1d scrub starts 2024-09-20T22:49:01.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:01 smithi084 ceph-mon[40024]: 5.1d scrub ok 2024-09-20T22:49:01.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:01 smithi084 ceph-mon[40024]: osdmap e461: 8 total, 8 up, 8 in 2024-09-20T22:49:01.690 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:49:01.712 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:01.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:01.713 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:49:01.814 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:49:01.838 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:01.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:01.839 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:49:01.939 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:49:01.962 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:01.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:01.963 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:49:02.022 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:49:02.022 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:49:02.064 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:49:02.085 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:02.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:02.086 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:49:02.187 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:49:02.208 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:02.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:02.209 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:49:02.220 INFO:tasks.rados.rados.0.smithi084.stdout:1793: done (4 left) 2024-09-20T22:49:02.220 INFO:tasks.rados.rados.0.smithi084.stdout:1794: done (3 left) 2024-09-20T22:49:02.220 INFO:tasks.rados.rados.0.smithi084.stdout:1795: done (2 left) 2024-09-20T22:49:02.220 INFO:tasks.rados.rados.0.smithi084.stdout:1796: done (1 left) 2024-09-20T22:49:02.220 INFO:tasks.rados.rados.0.smithi084.stdout:1797: done (0 left) 2024-09-20T22:49:02.220 INFO:tasks.rados.rados.0.smithi084.stdout:1798: read oid 17 snap 175 2024-09-20T22:49:02.220 INFO:tasks.rados.rados.0.smithi084.stdout:1798: expect (ObjNum 426 snap 109 seq_num 426) 2024-09-20T22:49:02.220 INFO:tasks.rados.rados.0.smithi084.stdout:1799: rollback oid 39 current snap is 192 2024-09-20T22:49:02.220 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 39 to 156 2024-09-20T22:49:02.220 INFO:tasks.rados.rados.0.smithi084.stdout:1800: delete oid 20 current snap is 192 2024-09-20T22:49:02.224 INFO:tasks.rados.rados.0.smithi084.stdout:1800: done (2 left) 2024-09-20T22:49:02.224 INFO:tasks.rados.rados.0.smithi084.stdout:1801: delete oid 38 current snap is 192 2024-09-20T22:49:02.225 INFO:tasks.rados.rados.0.smithi084.stdout:1798: expect (ObjNum 426 snap 109 seq_num 426) 2024-09-20T22:49:02.310 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:49:02.333 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:02.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:02.334 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:49:02.434 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:49:02.458 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:02.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:02.459 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:49:02.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:02 smithi154 ceph-mon[33264]: 7.10 scrub starts 2024-09-20T22:49:02.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:02 smithi154 ceph-mon[33264]: 7.10 scrub ok 2024-09-20T22:49:02.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:02 smithi154 ceph-mon[33264]: 7.d scrub starts 2024-09-20T22:49:02.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:02 smithi154 ceph-mon[33264]: 7.d scrub ok 2024-09-20T22:49:02.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:02 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "19"}]: dispatch 2024-09-20T22:49:02.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:02 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "19"}]: dispatch 2024-09-20T22:49:02.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:02 smithi154 ceph-mon[33264]: 4.d scrub starts 2024-09-20T22:49:02.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:02 smithi154 ceph-mon[33264]: 4.d scrub ok 2024-09-20T22:49:02.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:02 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "19"}]': finished 2024-09-20T22:49:02.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:02 smithi154 ceph-mon[33264]: osdmap e462: 8 total, 8 up, 8 in 2024-09-20T22:49:02.556 INFO:tasks.rados.rados.0.smithi084.stdout:1799: finishing rollback tid 0 to smithi08477203-39 2024-09-20T22:49:02.556 INFO:tasks.rados.rados.0.smithi084.stdout:1799: finishing rollback tid 1 to smithi08477203-39 2024-09-20T22:49:02.556 INFO:tasks.rados.rados.0.smithi084.stdout:1799: finishing rollback tid 2 to smithi08477203-39 2024-09-20T22:49:02.556 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 39 v 641 (ObjNum 525 snap 151 seq_num 525) dirty exists 2024-09-20T22:49:02.556 INFO:tasks.rados.rados.0.smithi084.stdout:1798: done (2 left) 2024-09-20T22:49:02.556 INFO:tasks.rados.rados.0.smithi084.stdout:1799: done (1 left) 2024-09-20T22:49:02.556 INFO:tasks.rados.rados.0.smithi084.stdout:1801: done (0 left) 2024-09-20T22:49:02.556 INFO:tasks.rados.rados.0.smithi084.stdout:1802: rollback oid 8 current snap is 192 2024-09-20T22:49:02.556 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 8 to 188 2024-09-20T22:49:02.556 INFO:tasks.rados.rados.0.smithi084.stdout:1803: snap_remove snap 187 2024-09-20T22:49:02.561 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:49:02.561 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-20T22:49:02.561 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:49:02.562 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:02 smithi084 ceph-mon[40024]: 7.10 scrub starts 2024-09-20T22:49:02.562 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:02 smithi084 ceph-mon[40024]: 7.10 scrub ok 2024-09-20T22:49:02.562 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:02 smithi084 ceph-mon[40024]: 7.d scrub starts 2024-09-20T22:49:02.562 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:02 smithi084 ceph-mon[40024]: 7.d scrub ok 2024-09-20T22:49:02.562 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:02 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "19"}]: dispatch 2024-09-20T22:49:02.562 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:02 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "19"}]: dispatch 2024-09-20T22:49:02.562 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:02 smithi084 ceph-mon[40024]: 4.d scrub starts 2024-09-20T22:49:02.562 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:02 smithi084 ceph-mon[40024]: 4.d scrub ok 2024-09-20T22:49:02.562 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:02 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "19"}]': finished 2024-09-20T22:49:02.562 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:02 smithi084 ceph-mon[40024]: osdmap e462: 8 total, 8 up, 8 in 2024-09-20T22:49:02.563 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:02 smithi084 ceph-mon[29774]: 7.10 scrub starts 2024-09-20T22:49:02.563 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:02 smithi084 ceph-mon[29774]: 7.10 scrub ok 2024-09-20T22:49:02.563 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:02 smithi084 ceph-mon[29774]: 7.d scrub starts 2024-09-20T22:49:02.563 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:02 smithi084 ceph-mon[29774]: 7.d scrub ok 2024-09-20T22:49:02.563 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:02 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "19"}]: dispatch 2024-09-20T22:49:02.563 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:02 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "19"}]: dispatch 2024-09-20T22:49:02.563 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:02 smithi084 ceph-mon[29774]: 4.d scrub starts 2024-09-20T22:49:02.563 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:02 smithi084 ceph-mon[29774]: 4.d scrub ok 2024-09-20T22:49:02.563 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:02 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "19"}]': finished 2024-09-20T22:49:02.563 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:02 smithi084 ceph-mon[29774]: osdmap e462: 8 total, 8 up, 8 in 2024-09-20T22:49:02.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:02.587 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:02.587 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:49:02.687 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:49:02.709 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:02.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:02.710 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:49:02.811 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:49:02.832 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:02.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:02.833 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:49:02.934 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:49:02.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:02.959 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:02.960 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:49:03.061 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:49:03.085 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:03.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:03.086 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:49:03.187 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:49:03.208 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:03.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:03.209 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:49:03.215 INFO:tasks.rados.rados.0.smithi084.stdout:1803: done (1 left) 2024-09-20T22:49:03.215 INFO:tasks.rados.rados.0.smithi084.stdout:1804: snap_create 2024-09-20T22:49:03.215 INFO:tasks.rados.rados.0.smithi084.stdout:1802: finishing rollback tid 0 to smithi08477203-8 2024-09-20T22:49:03.215 INFO:tasks.rados.rados.0.smithi084.stdout:1802: finishing rollback tid 1 to smithi08477203-8 2024-09-20T22:49:03.215 INFO:tasks.rados.rados.0.smithi084.stdout:1802: finishing rollback tid 2 to smithi08477203-8 2024-09-20T22:49:03.215 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 8 v 752 (ObjNum 616 snap 183 seq_num 616) dirty exists 2024-09-20T22:49:03.215 INFO:teuthology.orchestra.run.smithi084.stderr:set pool 7 pgp_num to 16 2024-09-20T22:49:03.310 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:49:03.332 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:03.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:03.333 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:49:03.346 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-20T22:49:03.434 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:49:03.455 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:03.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:03.456 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:49:03.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:03 smithi154 ceph-mon[33264]: 7.4 scrub starts 2024-09-20T22:49:03.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:03 smithi154 ceph-mon[33264]: 7.4 scrub ok 2024-09-20T22:49:03.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:03 smithi154 ceph-mon[33264]: pgmap v594: 162 pgs: 1 peering, 2 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 154 active+clean; 359 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail 2024-09-20T22:49:03.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:03 smithi154 ceph-mon[33264]: 3.10 scrub starts 2024-09-20T22:49:03.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:03 smithi154 ceph-mon[33264]: 3.10 scrub ok 2024-09-20T22:49:03.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:03 smithi154 ceph-mon[33264]: 5.a scrub starts 2024-09-20T22:49:03.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:03 smithi154 ceph-mon[33264]: 5.a scrub ok 2024-09-20T22:49:03.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:03 smithi154 ceph-mon[33264]: purged_snaps scrub starts 2024-09-20T22:49:03.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:03 smithi154 ceph-mon[33264]: purged_snaps scrub ok 2024-09-20T22:49:03.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:03 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/319969754' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num", "val": "16"}]: dispatch 2024-09-20T22:49:03.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:03 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num", "val": "16"}]: dispatch 2024-09-20T22:49:03.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:03 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num", "val": "16"}]': finished 2024-09-20T22:49:03.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:03 smithi154 ceph-mon[33264]: osdmap e463: 8 total, 8 up, 8 in 2024-09-20T22:49:03.556 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:49:03.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:03.581 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:03.581 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:49:03.587 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:03 smithi084 ceph-mon[40024]: 7.4 scrub starts 2024-09-20T22:49:03.587 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:03 smithi084 ceph-mon[40024]: 7.4 scrub ok 2024-09-20T22:49:03.587 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:03 smithi084 ceph-mon[40024]: pgmap v594: 162 pgs: 1 peering, 2 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 154 active+clean; 359 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail 2024-09-20T22:49:03.587 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:03 smithi084 ceph-mon[40024]: 3.10 scrub starts 2024-09-20T22:49:03.587 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:03 smithi084 ceph-mon[40024]: 3.10 scrub ok 2024-09-20T22:49:03.587 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:03 smithi084 ceph-mon[40024]: 5.a scrub starts 2024-09-20T22:49:03.587 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:03 smithi084 ceph-mon[40024]: 5.a scrub ok 2024-09-20T22:49:03.587 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:03 smithi084 ceph-mon[40024]: purged_snaps scrub starts 2024-09-20T22:49:03.588 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:03 smithi084 ceph-mon[40024]: purged_snaps scrub ok 2024-09-20T22:49:03.588 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:03 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/319969754' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num", "val": "16"}]: dispatch 2024-09-20T22:49:03.588 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:03 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num", "val": "16"}]: dispatch 2024-09-20T22:49:03.588 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:03 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num", "val": "16"}]': finished 2024-09-20T22:49:03.588 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:03 smithi084 ceph-mon[40024]: osdmap e463: 8 total, 8 up, 8 in 2024-09-20T22:49:03.590 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:03 smithi084 ceph-mon[29774]: 7.4 scrub starts 2024-09-20T22:49:03.590 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:03 smithi084 ceph-mon[29774]: 7.4 scrub ok 2024-09-20T22:49:03.590 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:03 smithi084 ceph-mon[29774]: pgmap v594: 162 pgs: 1 peering, 2 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 154 active+clean; 359 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail 2024-09-20T22:49:03.590 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:03 smithi084 ceph-mon[29774]: 3.10 scrub starts 2024-09-20T22:49:03.590 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:03 smithi084 ceph-mon[29774]: 3.10 scrub ok 2024-09-20T22:49:03.590 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:03 smithi084 ceph-mon[29774]: 5.a scrub starts 2024-09-20T22:49:03.590 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:03 smithi084 ceph-mon[29774]: 5.a scrub ok 2024-09-20T22:49:03.590 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:03 smithi084 ceph-mon[29774]: purged_snaps scrub starts 2024-09-20T22:49:03.590 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:03 smithi084 ceph-mon[29774]: purged_snaps scrub ok 2024-09-20T22:49:03.590 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:03 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/319969754' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num", "val": "16"}]: dispatch 2024-09-20T22:49:03.590 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:03 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num", "val": "16"}]: dispatch 2024-09-20T22:49:03.590 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:03 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num", "val": "16"}]': finished 2024-09-20T22:49:03.591 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:03 smithi084 ceph-mon[29774]: osdmap e463: 8 total, 8 up, 8 in 2024-09-20T22:49:03.682 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:49:03.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:03.704 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:03.705 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:49:03.806 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:49:03.830 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:03.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:03.831 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:49:03.932 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:49:03.955 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:03.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:03.956 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:49:04.010 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-20T22:49:04.056 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:49:04.079 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:04.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:04.080 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:49:04.180 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:49:04.202 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:04.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:04.206 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:49:04.219 INFO:tasks.rados.rados.0.smithi084.stdout:1802: done (1 left) 2024-09-20T22:49:04.219 INFO:tasks.rados.rados.0.smithi084.stdout:1804: done (0 left) 2024-09-20T22:49:04.219 INFO:tasks.rados.rados.0.smithi084.stdout:1805: rollback oid 23 current snap is 193 2024-09-20T22:49:04.219 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 23 to 175 2024-09-20T22:49:04.219 INFO:tasks.rados.rados.0.smithi084.stdout:1806: write (excl) oid 3 current snap is 193 2024-09-20T22:49:04.219 INFO:tasks.rados.rados.0.smithi084.stdout:1806: seq_num 634 ranges {655638=565165,1900172=421542} 2024-09-20T22:49:04.225 INFO:tasks.rados.rados.0.smithi084.stdout:1806: writing smithi08477203-3 from 655638 to 1220803 tid 1 2024-09-20T22:49:04.232 INFO:tasks.rados.rados.0.smithi084.stdout:1806: writing smithi08477203-3 from 1900172 to 2321714 tid 2 2024-09-20T22:49:04.233 INFO:tasks.rados.rados.0.smithi084.stdout:1807: rollback oid 45 current snap is 193 2024-09-20T22:49:04.233 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 45 to 183 2024-09-20T22:49:04.233 INFO:tasks.rados.rados.0.smithi084.stdout:1805: finishing rollback tid 0 to smithi08477203-23 2024-09-20T22:49:04.233 INFO:tasks.rados.rados.0.smithi084.stdout:1805: finishing rollback tid 1 to smithi08477203-23 2024-09-20T22:49:04.233 INFO:tasks.rados.rados.0.smithi084.stdout:1805: finishing rollback tid 2 to smithi08477203-23 2024-09-20T22:49:04.233 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 23 v 709 (ObjNum 580 snap 173 seq_num 580) dirty exists 2024-09-20T22:49:04.233 INFO:tasks.rados.rados.0.smithi084.stdout:1806: finishing write tid 1 to smithi08477203-3 2024-09-20T22:49:04.233 INFO:tasks.rados.rados.0.smithi084.stdout:1805: done (2 left) 2024-09-20T22:49:04.233 INFO:tasks.rados.rados.0.smithi084.stdout:1808: read oid 35 snap -1 2024-09-20T22:49:04.233 INFO:tasks.rados.rados.0.smithi084.stdout:1808: expect (ObjNum 631 snap 191 seq_num 631) 2024-09-20T22:49:04.233 INFO:tasks.rados.rados.0.smithi084.stdout:1809: rollback oid 22 current snap is 193 2024-09-20T22:49:04.233 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 22 to 184 2024-09-20T22:49:04.233 INFO:tasks.rados.rados.0.smithi084.stdout:1810: copy_from oid 19 from oid 42 current snap is 193 2024-09-20T22:49:04.233 INFO:tasks.rados.rados.0.smithi084.stdout:1811: delete oid 33 current snap is 193 2024-09-20T22:49:04.234 INFO:tasks.rados.rados.0.smithi084.stdout:1807: finishing rollback tid 1 to smithi08477203-45 2024-09-20T22:49:04.234 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 45 v 179 (ObjNum 33 snap 0 seq_num 2085656408) dirty dne 2024-09-20T22:49:04.234 INFO:tasks.rados.rados.0.smithi084.stdout:1809: finishing rollback tid 1 to smithi08477203-22 2024-09-20T22:49:04.234 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 22 v 324 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-09-20T22:49:04.235 INFO:tasks.rados.rados.0.smithi084.stdout:1807: done (5 left) 2024-09-20T22:49:04.235 INFO:tasks.rados.rados.0.smithi084.stdout:1809: done (4 left) 2024-09-20T22:49:04.235 INFO:tasks.rados.rados.0.smithi084.stdout:1811: done (3 left) 2024-09-20T22:49:04.235 INFO:tasks.rados.rados.0.smithi084.stdout:1812: write (excl) oid 21 current snap is 193 2024-09-20T22:49:04.235 INFO:tasks.rados.rados.0.smithi084.stdout:1812: seq_num 636 ranges {689252=715663,1900783=425211} 2024-09-20T22:49:04.243 INFO:tasks.rados.rados.0.smithi084.stdout:1812: writing smithi08477203-21 from 689252 to 1404915 tid 1 2024-09-20T22:49:04.248 INFO:tasks.rados.rados.0.smithi084.stdout:1812: writing smithi08477203-21 from 1900783 to 2325994 tid 2 2024-09-20T22:49:04.248 INFO:tasks.rados.rados.0.smithi084.stdout:1813: read oid 12 snap -1 2024-09-20T22:49:04.248 INFO:tasks.rados.rados.0.smithi084.stdout:1813: expect deleted 2024-09-20T22:49:04.249 INFO:tasks.rados.rados.0.smithi084.stdout:1814: read oid 43 snap -1 2024-09-20T22:49:04.249 INFO:tasks.rados.rados.0.smithi084.stdout:1814: expect deleted 2024-09-20T22:49:04.249 INFO:tasks.rados.rados.0.smithi084.stdout:1815: delete oid 32 current snap is 193 2024-09-20T22:49:04.249 INFO:tasks.rados.rados.0.smithi084.stdout:1806: finishing write tid 2 to smithi08477203-3 2024-09-20T22:49:04.249 INFO:tasks.rados.rados.0.smithi084.stdout:1806: finishing write tid 3 to smithi08477203-3 2024-09-20T22:49:04.249 INFO:tasks.rados.rados.0.smithi084.stdout:1810: finishing copy_from to smithi08477203-19 2024-09-20T22:49:04.249 INFO:tasks.rados.rados.0.smithi084.stdout:1810: got expected ENOENT (src dne) 2024-09-20T22:49:04.249 INFO:tasks.rados.rados.0.smithi084.stdout:1810: finishing copy_from racing read to smithi08477203-19 2024-09-20T22:49:04.249 INFO:tasks.rados.rados.0.smithi084.stdout:1806: finishing write tid 4 to smithi08477203-3 2024-09-20T22:49:04.249 INFO:tasks.rados.rados.0.smithi084.stdout:1806: oid 3 updating version 0 to 753 2024-09-20T22:49:04.249 INFO:tasks.rados.rados.0.smithi084.stdout:1806: oid 3 updating version 753 to 755 2024-09-20T22:49:04.249 INFO:tasks.rados.rados.0.smithi084.stdout:1806: oid 3 version 755 is already newer than 754 2024-09-20T22:49:04.249 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 3 v 755 (ObjNum 634 snap 193 seq_num 634) dirty exists 2024-09-20T22:49:04.250 INFO:tasks.rados.rados.0.smithi084.stdout:1806: left oid 3 (ObjNum 634 snap 193 seq_num 634) 2024-09-20T22:49:04.250 INFO:tasks.rados.rados.0.smithi084.stdout:1808: expect (ObjNum 631 snap 191 seq_num 631) 2024-09-20T22:49:04.306 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:49:04.327 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:04.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:04.328 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:49:04.369 INFO:tasks.rados.rados.0.smithi084.stdout:1812: finishing write tid 1 to smithi08477203-21 2024-09-20T22:49:04.369 INFO:tasks.rados.rados.0.smithi084.stdout:1812: finishing write tid 2 to smithi08477203-21 2024-09-20T22:49:04.369 INFO:tasks.rados.rados.0.smithi084.stdout:1812: finishing write tid 3 to smithi08477203-21 2024-09-20T22:49:04.369 INFO:tasks.rados.rados.0.smithi084.stdout:1812: finishing write tid 4 to smithi08477203-21 2024-09-20T22:49:04.369 INFO:tasks.rados.rados.0.smithi084.stdout:1812: oid 21 updating version 0 to 725 2024-09-20T22:49:04.369 INFO:tasks.rados.rados.0.smithi084.stdout:1812: oid 21 version 725 is already newer than 723 2024-09-20T22:49:04.369 INFO:tasks.rados.rados.0.smithi084.stdout:1812: oid 21 version 725 is already newer than 724 2024-09-20T22:49:04.369 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 21 v 725 (ObjNum 636 snap 193 seq_num 636) dirty exists 2024-09-20T22:49:04.370 INFO:tasks.rados.rados.0.smithi084.stdout:1812: left oid 21 (ObjNum 636 snap 193 seq_num 636) 2024-09-20T22:49:04.429 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:49:04.450 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:04.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:04.451 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:49:04.552 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:49:04.558 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:04 smithi084 ceph-mon[40024]: 3.1d scrub starts 2024-09-20T22:49:04.558 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:04 smithi084 ceph-mon[40024]: 3.1d scrub ok 2024-09-20T22:49:04.558 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:04 smithi084 ceph-mon[40024]: 5.5 scrub starts 2024-09-20T22:49:04.558 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:04 smithi084 ceph-mon[40024]: 5.5 scrub ok 2024-09-20T22:49:04.559 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:04 smithi084 ceph-mon[40024]: purged_snaps scrub starts 2024-09-20T22:49:04.559 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:04 smithi084 ceph-mon[40024]: purged_snaps scrub ok 2024-09-20T22:49:04.559 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:04 smithi084 ceph-mon[40024]: 4.a scrub starts 2024-09-20T22:49:04.559 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:04 smithi084 ceph-mon[40024]: 4.a scrub ok 2024-09-20T22:49:04.559 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:04 smithi084 ceph-mon[40024]: osdmap e464: 8 total, 8 up, 8 in 2024-09-20T22:49:04.559 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:04 smithi084 ceph-mon[29774]: 3.1d scrub starts 2024-09-20T22:49:04.559 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:04 smithi084 ceph-mon[29774]: 3.1d scrub ok 2024-09-20T22:49:04.559 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:04 smithi084 ceph-mon[29774]: 5.5 scrub starts 2024-09-20T22:49:04.559 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:04 smithi084 ceph-mon[29774]: 5.5 scrub ok 2024-09-20T22:49:04.559 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:04 smithi084 ceph-mon[29774]: purged_snaps scrub starts 2024-09-20T22:49:04.559 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:04 smithi084 ceph-mon[29774]: purged_snaps scrub ok 2024-09-20T22:49:04.559 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:04 smithi084 ceph-mon[29774]: 4.a scrub starts 2024-09-20T22:49:04.560 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:04 smithi084 ceph-mon[29774]: 4.a scrub ok 2024-09-20T22:49:04.560 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:04 smithi084 ceph-mon[29774]: osdmap e464: 8 total, 8 up, 8 in 2024-09-20T22:49:04.574 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:04.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:04.575 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:49:04.584 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.3 dump_blocked_ops 2024-09-20T22:49:04.589 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:49:04.676 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:49:04.698 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:04.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:04.699 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:49:04.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:04 smithi154 ceph-mon[33264]: 3.1d scrub starts 2024-09-20T22:49:04.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:04 smithi154 ceph-mon[33264]: 3.1d scrub ok 2024-09-20T22:49:04.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:04 smithi154 ceph-mon[33264]: 5.5 scrub starts 2024-09-20T22:49:04.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:04 smithi154 ceph-mon[33264]: 5.5 scrub ok 2024-09-20T22:49:04.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:04 smithi154 ceph-mon[33264]: purged_snaps scrub starts 2024-09-20T22:49:04.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:04 smithi154 ceph-mon[33264]: purged_snaps scrub ok 2024-09-20T22:49:04.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:04 smithi154 ceph-mon[33264]: 4.a scrub starts 2024-09-20T22:49:04.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:04 smithi154 ceph-mon[33264]: 4.a scrub ok 2024-09-20T22:49:04.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:04 smithi154 ceph-mon[33264]: osdmap e464: 8 total, 8 up, 8 in 2024-09-20T22:49:04.799 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:49:04.824 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:04.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:04.825 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:49:04.926 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:49:04.947 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:04.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:04.948 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:49:05.049 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:49:05.071 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:05.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:05.072 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:49:05.130 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:49:05.130 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:49:05.173 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:49:05.194 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:05.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:05.195 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:49:05.224 INFO:tasks.rados.rados.0.smithi084.stdout:1806: done (6 left) 2024-09-20T22:49:05.224 INFO:tasks.rados.rados.0.smithi084.stdout:1808: done (5 left) 2024-09-20T22:49:05.224 INFO:tasks.rados.rados.0.smithi084.stdout:1810: done (4 left) 2024-09-20T22:49:05.224 INFO:tasks.rados.rados.0.smithi084.stdout:1812: done (3 left) 2024-09-20T22:49:05.224 INFO:tasks.rados.rados.0.smithi084.stdout:1813: done (2 left) 2024-09-20T22:49:05.224 INFO:tasks.rados.rados.0.smithi084.stdout:1814: done (1 left) 2024-09-20T22:49:05.225 INFO:tasks.rados.rados.0.smithi084.stdout:1815: done (0 left) 2024-09-20T22:49:05.225 INFO:tasks.rados.rados.0.smithi084.stdout:1816: snap_remove snap 179 2024-09-20T22:49:05.296 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:49:05.318 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:05.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:05.319 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:49:05.364 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:05 smithi084 ceph-mon[29774]: pgmap v597: 162 pgs: 2 clean+premerge+peered, 2 active+clean+snaptrim, 158 active+clean; 375 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 7.7 MiB/s rd, 584 KiB/s wr, 12 op/s 2024-09-20T22:49:05.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:05 smithi084 ceph-mon[29774]: 6.13 scrub starts 2024-09-20T22:49:05.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:05 smithi084 ceph-mon[29774]: 6.13 scrub ok 2024-09-20T22:49:05.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:05 smithi084 ceph-mon[29774]: 7.7 scrub starts 2024-09-20T22:49:05.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:05 smithi084 ceph-mon[29774]: 7.7 scrub ok 2024-09-20T22:49:05.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:05 smithi084 ceph-mon[29774]: 5.b scrub starts 2024-09-20T22:49:05.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:05 smithi084 ceph-mon[29774]: 5.b scrub ok 2024-09-20T22:49:05.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:05 smithi084 ceph-mon[29774]: 6.5 scrub starts 2024-09-20T22:49:05.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:05 smithi084 ceph-mon[29774]: 6.5 scrub ok 2024-09-20T22:49:05.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:05 smithi084 ceph-mon[29774]: 6.1d scrub starts 2024-09-20T22:49:05.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:05 smithi084 ceph-mon[29774]: 6.1d scrub ok 2024-09-20T22:49:05.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:05 smithi084 ceph-mon[29774]: osdmap e465: 8 total, 8 up, 8 in 2024-09-20T22:49:05.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:05 smithi084 ceph-mon[40024]: pgmap v597: 162 pgs: 2 clean+premerge+peered, 2 active+clean+snaptrim, 158 active+clean; 375 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 7.7 MiB/s rd, 584 KiB/s wr, 12 op/s 2024-09-20T22:49:05.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:05 smithi084 ceph-mon[40024]: 6.13 scrub starts 2024-09-20T22:49:05.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:05 smithi084 ceph-mon[40024]: 6.13 scrub ok 2024-09-20T22:49:05.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:05 smithi084 ceph-mon[40024]: 7.7 scrub starts 2024-09-20T22:49:05.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:05 smithi084 ceph-mon[40024]: 7.7 scrub ok 2024-09-20T22:49:05.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:05 smithi084 ceph-mon[40024]: 5.b scrub starts 2024-09-20T22:49:05.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:05 smithi084 ceph-mon[40024]: 5.b scrub ok 2024-09-20T22:49:05.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:05 smithi084 ceph-mon[40024]: 6.5 scrub starts 2024-09-20T22:49:05.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:05 smithi084 ceph-mon[40024]: 6.5 scrub ok 2024-09-20T22:49:05.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:05 smithi084 ceph-mon[40024]: 6.1d scrub starts 2024-09-20T22:49:05.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:05 smithi084 ceph-mon[40024]: 6.1d scrub ok 2024-09-20T22:49:05.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:05 smithi084 ceph-mon[40024]: osdmap e465: 8 total, 8 up, 8 in 2024-09-20T22:49:05.420 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:49:05.472 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:05.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:05.473 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:49:05.574 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:49:05.595 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:05.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:05.596 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:49:05.697 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:49:05.719 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:05.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:05.720 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:49:05.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:05 smithi154 ceph-mon[33264]: pgmap v597: 162 pgs: 2 clean+premerge+peered, 2 active+clean+snaptrim, 158 active+clean; 375 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 7.7 MiB/s rd, 584 KiB/s wr, 12 op/s 2024-09-20T22:49:05.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:05 smithi154 ceph-mon[33264]: 6.13 scrub starts 2024-09-20T22:49:05.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:05 smithi154 ceph-mon[33264]: 6.13 scrub ok 2024-09-20T22:49:05.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:05 smithi154 ceph-mon[33264]: 7.7 scrub starts 2024-09-20T22:49:05.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:05 smithi154 ceph-mon[33264]: 7.7 scrub ok 2024-09-20T22:49:05.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:05 smithi154 ceph-mon[33264]: 5.b scrub starts 2024-09-20T22:49:05.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:05 smithi154 ceph-mon[33264]: 5.b scrub ok 2024-09-20T22:49:05.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:05 smithi154 ceph-mon[33264]: 6.5 scrub starts 2024-09-20T22:49:05.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:05 smithi154 ceph-mon[33264]: 6.5 scrub ok 2024-09-20T22:49:05.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:05 smithi154 ceph-mon[33264]: 6.1d scrub starts 2024-09-20T22:49:05.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:05 smithi154 ceph-mon[33264]: 6.1d scrub ok 2024-09-20T22:49:05.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:05 smithi154 ceph-mon[33264]: osdmap e465: 8 total, 8 up, 8 in 2024-09-20T22:49:05.821 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:49:05.842 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:05.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:05.843 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:49:05.947 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:49:05.970 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:05.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:05.971 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:49:05.993 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-20T22:49:05.994 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2024-09-20T22:49:05.994 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2024-09-20T22:49:05.994 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2024-09-20T22:49:05.994 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-20T22:49:06.071 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:49:06.092 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:06.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:06.093 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:49:06.194 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:49:06.215 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is unset 2024-09-20T22:49:06.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:06.216 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:06.216 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:49:06.223 INFO:tasks.rados.rados.0.smithi084.stdout:1816: done (0 left) 2024-09-20T22:49:06.223 INFO:tasks.rados.rados.0.smithi084.stdout:1817: read oid 48 snap 189 2024-09-20T22:49:06.223 INFO:tasks.rados.rados.0.smithi084.stdout:1817: expect deleted 2024-09-20T22:49:06.223 INFO:tasks.rados.rados.0.smithi084.stdout:1818: snap_remove snap 191 2024-09-20T22:49:06.317 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:49:06.338 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:06.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:06.339 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:49:06.439 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:49:06.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:06.463 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:06.463 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:49:06.564 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:49:06.585 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:06.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:06.586 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:49:06.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:06 smithi084 ceph-mon[29774]: 6.19 scrub starts 2024-09-20T22:49:06.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:06 smithi084 ceph-mon[29774]: 6.19 scrub ok 2024-09-20T22:49:06.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:06 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/3721802497' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:49:06.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:06 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd rm-pg-upmap", "pgid": "7.2"}]: dispatch 2024-09-20T22:49:06.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:06 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "7.2"}]: dispatch 2024-09-20T22:49:06.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:06 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd rm-pg-upmap", "pgid": "7.2"}]: dispatch 2024-09-20T22:49:06.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:06 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "7.2"}]: dispatch 2024-09-20T22:49:06.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:06 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/2543197646' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:49:06.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:06 smithi084 ceph-mon[29774]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T22:49:06.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:06 smithi084 ceph-mon[29774]: Cluster is now healthy 2024-09-20T22:49:06.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:06 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/3721802497' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T22:49:06.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:06 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd rm-pg-upmap", "pgid": "7.2"}]': finished 2024-09-20T22:49:06.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:06 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd rm-pg-upmap-items", "pgid": "7.2"}]': finished 2024-09-20T22:49:06.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:06 smithi084 ceph-mon[29774]: osdmap e466: 8 total, 8 up, 8 in 2024-09-20T22:49:06.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:06 smithi084 ceph-mon[40024]: 6.19 scrub starts 2024-09-20T22:49:06.617 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:06 smithi084 ceph-mon[40024]: 6.19 scrub ok 2024-09-20T22:49:06.617 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:06 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/3721802497' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:49:06.617 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:06 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd rm-pg-upmap", "pgid": "7.2"}]: dispatch 2024-09-20T22:49:06.617 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:06 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "7.2"}]: dispatch 2024-09-20T22:49:06.617 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:06 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd rm-pg-upmap", "pgid": "7.2"}]: dispatch 2024-09-20T22:49:06.617 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:06 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "7.2"}]: dispatch 2024-09-20T22:49:06.617 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:06 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/2543197646' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:49:06.617 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:06 smithi084 ceph-mon[40024]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T22:49:06.617 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:06 smithi084 ceph-mon[40024]: Cluster is now healthy 2024-09-20T22:49:06.617 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:06 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/3721802497' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T22:49:06.618 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:06 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd rm-pg-upmap", "pgid": "7.2"}]': finished 2024-09-20T22:49:06.618 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:06 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd rm-pg-upmap-items", "pgid": "7.2"}]': finished 2024-09-20T22:49:06.618 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:06 smithi084 ceph-mon[40024]: osdmap e466: 8 total, 8 up, 8 in 2024-09-20T22:49:06.686 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:49:06.708 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:06.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:06.709 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:49:06.743 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.3 dump_historic_ops 2024-09-20T22:49:06.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:06 smithi154 ceph-mon[33264]: 6.19 scrub starts 2024-09-20T22:49:06.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:06 smithi154 ceph-mon[33264]: 6.19 scrub ok 2024-09-20T22:49:06.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:06 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/3721802497' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:49:06.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:06 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd rm-pg-upmap", "pgid": "7.2"}]: dispatch 2024-09-20T22:49:06.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:06 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "7.2"}]: dispatch 2024-09-20T22:49:06.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:06 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd rm-pg-upmap", "pgid": "7.2"}]: dispatch 2024-09-20T22:49:06.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:06 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "7.2"}]: dispatch 2024-09-20T22:49:06.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:06 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/2543197646' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:49:06.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:06 smithi154 ceph-mon[33264]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T22:49:06.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:06 smithi154 ceph-mon[33264]: Cluster is now healthy 2024-09-20T22:49:06.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:06 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/3721802497' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T22:49:06.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:06 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd rm-pg-upmap", "pgid": "7.2"}]': finished 2024-09-20T22:49:06.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:06 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd rm-pg-upmap-items", "pgid": "7.2"}]': finished 2024-09-20T22:49:06.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:06 smithi154 ceph-mon[33264]: osdmap e466: 8 total, 8 up, 8 in 2024-09-20T22:49:06.799 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:49:06.799 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:49:06.799 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:49:06.799 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-20T22:49:06.876 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:49:06.876 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:49:06.876 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:49:06.876 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:49:06.876 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:49:06.876 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T22:49:06.876 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:49:06.877 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:49:06.877 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:49:06.877 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:49:06.877 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:49:06.877 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:49:06.878 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:49:06.899 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:06.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:06.900 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:49:07.001 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:49:07.023 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:07.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:07.024 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:49:07.125 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:49:07.146 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:07.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:07.147 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:49:07.223 INFO:tasks.rados.rados.0.smithi084.stdout:1818: done (1 left) 2024-09-20T22:49:07.224 INFO:tasks.rados.rados.0.smithi084.stdout:1819: rollback oid 4 current snap is 193 2024-09-20T22:49:07.224 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 4 to 188 2024-09-20T22:49:07.224 INFO:tasks.rados.rados.0.smithi084.stdout:1820: write oid 20 current snap is 193 2024-09-20T22:49:07.224 INFO:tasks.rados.rados.0.smithi084.stdout:1820: seq_num 638 ranges {722866=466161,1617747=428880,2587634=135232} 2024-09-20T22:49:07.229 INFO:tasks.rados.rados.0.smithi084.stdout:1820: writing smithi08477203-20 from 722866 to 1189027 tid 1 2024-09-20T22:49:07.234 INFO:tasks.rados.rados.0.smithi084.stdout:1820: writing smithi08477203-20 from 1617747 to 2046627 tid 2 2024-09-20T22:49:07.238 INFO:tasks.rados.rados.0.smithi084.stdout:1820: writing smithi08477203-20 from 2587634 to 2722866 tid 3 2024-09-20T22:49:07.238 INFO:tasks.rados.rados.0.smithi084.stdout:1821: delete oid 16 current snap is 193 2024-09-20T22:49:07.238 INFO:tasks.rados.rados.0.smithi084.stdout:1819: finishing rollback tid 0 to smithi08477203-4 2024-09-20T22:49:07.238 INFO:tasks.rados.rados.0.smithi084.stdout:1819: finishing rollback tid 1 to smithi08477203-4 2024-09-20T22:49:07.238 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 4 v 1177 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-09-20T22:49:07.238 INFO:tasks.rados.rados.0.smithi084.stdout:1820: finishing write tid 1 to smithi08477203-20 2024-09-20T22:49:07.238 INFO:tasks.rados.rados.0.smithi084.stdout:1820: finishing write tid 2 to smithi08477203-20 2024-09-20T22:49:07.239 INFO:tasks.rados.rados.0.smithi084.stdout:1820: finishing write tid 3 to smithi08477203-20 2024-09-20T22:49:07.240 INFO:tasks.rados.rados.0.smithi084.stdout:1820: finishing write tid 4 to smithi08477203-20 2024-09-20T22:49:07.240 INFO:tasks.rados.rados.0.smithi084.stdout:1817: done (3 left) 2024-09-20T22:49:07.240 INFO:tasks.rados.rados.0.smithi084.stdout:1819: done (2 left) 2024-09-20T22:49:07.240 INFO:tasks.rados.rados.0.smithi084.stdout:1821: done (1 left) 2024-09-20T22:49:07.240 INFO:tasks.rados.rados.0.smithi084.stdout:1822: snap_remove snap 164 2024-09-20T22:49:07.248 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:49:07.270 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:07.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:07.271 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:49:07.362 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:07 smithi084 ceph-mon[29774]: 3.c scrub starts 2024-09-20T22:49:07.362 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:07 smithi084 ceph-mon[29774]: 3.c scrub ok 2024-09-20T22:49:07.362 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:07 smithi084 ceph-mon[29774]: pgmap v600: 161 pgs: 1 clean+premerge+peered, 2 active+clean+snaptrim, 158 active+clean; 362 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 75 KiB/s rd, 2.0 MiB/s wr, 16 op/s 2024-09-20T22:49:07.362 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:07 smithi084 ceph-mon[29774]: 5.11 scrub starts 2024-09-20T22:49:07.363 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:07 smithi084 ceph-mon[29774]: 5.11 scrub ok 2024-09-20T22:49:07.363 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:07 smithi084 ceph-mon[29774]: osdmap e467: 8 total, 8 up, 8 in 2024-09-20T22:49:07.363 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:07 smithi084 ceph-mon[40024]: 3.c scrub starts 2024-09-20T22:49:07.363 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:07 smithi084 ceph-mon[40024]: 3.c scrub ok 2024-09-20T22:49:07.363 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:07 smithi084 ceph-mon[40024]: pgmap v600: 161 pgs: 1 clean+premerge+peered, 2 active+clean+snaptrim, 158 active+clean; 362 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 75 KiB/s rd, 2.0 MiB/s wr, 16 op/s 2024-09-20T22:49:07.363 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:07 smithi084 ceph-mon[40024]: 5.11 scrub starts 2024-09-20T22:49:07.363 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:07 smithi084 ceph-mon[40024]: 5.11 scrub ok 2024-09-20T22:49:07.363 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:07 smithi084 ceph-mon[40024]: osdmap e467: 8 total, 8 up, 8 in 2024-09-20T22:49:07.372 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:49:07.393 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:07.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:07.394 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:49:07.495 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:49:07.516 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:07.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:07.517 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:49:07.618 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:49:07.640 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:07.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:07.641 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:49:07.742 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:49:07.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:07 smithi154 ceph-mon[33264]: 3.c scrub starts 2024-09-20T22:49:07.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:07 smithi154 ceph-mon[33264]: 3.c scrub ok 2024-09-20T22:49:07.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:07 smithi154 ceph-mon[33264]: pgmap v600: 161 pgs: 1 clean+premerge+peered, 2 active+clean+snaptrim, 158 active+clean; 362 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 75 KiB/s rd, 2.0 MiB/s wr, 16 op/s 2024-09-20T22:49:07.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:07 smithi154 ceph-mon[33264]: 5.11 scrub starts 2024-09-20T22:49:07.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:07 smithi154 ceph-mon[33264]: 5.11 scrub ok 2024-09-20T22:49:07.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:07 smithi154 ceph-mon[33264]: osdmap e467: 8 total, 8 up, 8 in 2024-09-20T22:49:07.763 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:07.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:07.764 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:49:07.865 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:49:07.886 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:07.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:07.887 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:49:07.987 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:49:08.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:08.009 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:08.009 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:49:08.110 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:49:08.118 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:49:08.132 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:08.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:08.133 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:49:08.219 INFO:tasks.rados.rados.0.smithi084.stdout:1822: done (1 left) 2024-09-20T22:49:08.219 INFO:tasks.rados.rados.0.smithi084.stdout:1823: write (excl) oid 24 current snap is 193 2024-09-20T22:49:08.220 INFO:tasks.rados.rados.0.smithi084.stdout:1823: seq_num 640 ranges {756480=733012,2134711=432549} 2024-09-20T22:49:08.228 INFO:tasks.rados.rados.0.smithi084.stdout:1823: writing smithi08477203-24 from 756480 to 1489492 tid 1 2024-09-20T22:49:08.234 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:49:08.238 INFO:tasks.rados.rados.0.smithi084.stdout:1823: writing smithi08477203-24 from 2134711 to 2567260 tid 2 2024-09-20T22:49:08.238 INFO:tasks.rados.rados.0.smithi084.stdout:1824: copy_from oid 42 from oid 10 current snap is 193 2024-09-20T22:49:08.238 INFO:tasks.rados.rados.0.smithi084.stdout:1825: write oid 3 current snap is 193 2024-09-20T22:49:08.238 INFO:tasks.rados.rados.0.smithi084.stdout:1820: finishing write tid 5 to smithi08477203-20 2024-09-20T22:49:08.238 INFO:tasks.rados.rados.0.smithi084.stdout:1820: oid 20 updating version 0 to 311 2024-09-20T22:49:08.238 INFO:tasks.rados.rados.0.smithi084.stdout:1820: oid 20 version 311 is already newer than 310 2024-09-20T22:49:08.239 INFO:tasks.rados.rados.0.smithi084.stdout:1820: oid 20 version 311 is already newer than 308 2024-09-20T22:49:08.239 INFO:tasks.rados.rados.0.smithi084.stdout:1820: oid 20 version 311 is already newer than 309 2024-09-20T22:49:08.239 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 20 v 311 (ObjNum 638 snap 193 seq_num 638) dirty exists 2024-09-20T22:49:08.239 INFO:tasks.rados.rados.0.smithi084.stdout:1820: left oid 20 (ObjNum 638 snap 193 seq_num 638) 2024-09-20T22:49:08.239 INFO:tasks.rados.rados.0.smithi084.stdout:1825: seq_num 641 ranges {773287=408261,1793193=576207} 2024-09-20T22:49:08.245 INFO:tasks.rados.rados.0.smithi084.stdout:1825: writing smithi08477203-3 from 773287 to 1181548 tid 1 2024-09-20T22:49:08.252 INFO:tasks.rados.rados.0.smithi084.stdout:1825: writing smithi08477203-3 from 1793193 to 2369400 tid 2 2024-09-20T22:49:08.252 INFO:tasks.rados.rados.0.smithi084.stdout:1823: finishing write tid 1 to smithi08477203-24 2024-09-20T22:49:08.252 INFO:tasks.rados.rados.0.smithi084.stdout:1823: finishing write tid 2 to smithi08477203-24 2024-09-20T22:49:08.253 INFO:tasks.rados.rados.0.smithi084.stdout:1823: finishing write tid 3 to smithi08477203-24 2024-09-20T22:49:08.253 INFO:tasks.rados.rados.0.smithi084.stdout:1823: finishing write tid 4 to smithi08477203-24 2024-09-20T22:49:08.253 INFO:tasks.rados.rados.0.smithi084.stdout:1823: oid 24 updating version 0 to 422 2024-09-20T22:49:08.253 INFO:tasks.rados.rados.0.smithi084.stdout:1823: oid 24 updating version 422 to 423 2024-09-20T22:49:08.253 INFO:tasks.rados.rados.0.smithi084.stdout:1823: oid 24 updating version 423 to 424 2024-09-20T22:49:08.253 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 24 v 424 (ObjNum 640 snap 193 seq_num 640) dirty exists 2024-09-20T22:49:08.253 INFO:tasks.rados.rados.0.smithi084.stdout:1823: left oid 24 (ObjNum 640 snap 193 seq_num 640) 2024-09-20T22:49:08.253 INFO:tasks.rados.rados.0.smithi084.stdout:1820: done (3 left) 2024-09-20T22:49:08.253 INFO:tasks.rados.rados.0.smithi084.stdout:1823: done (2 left) 2024-09-20T22:49:08.253 INFO:tasks.rados.rados.0.smithi084.stdout:1826: write oid 26 current snap is 193 2024-09-20T22:49:08.253 INFO:tasks.rados.rados.0.smithi084.stdout:1826: seq_num 642 ranges {790094=483510,1851675=436218} 2024-09-20T22:49:08.260 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:08.260 INFO:tasks.rados.rados.0.smithi084.stdout:1826: writing smithi08477203-26 from 790094 to 1273604 tid 1 2024-09-20T22:49:08.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:08.261 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:49:08.267 INFO:tasks.rados.rados.0.smithi084.stdout:1826: writing smithi08477203-26 from 1851675 to 2287893 tid 2 2024-09-20T22:49:08.268 INFO:tasks.rados.rados.0.smithi084.stdout:1827: copy_from oid 36 from oid 32 current snap is 193 2024-09-20T22:49:08.268 INFO:tasks.rados.rados.0.smithi084.stdout:1824: finishing copy_from racing read to smithi08477203-42 2024-09-20T22:49:08.268 INFO:tasks.rados.rados.0.smithi084.stdout:1828: write oid 5 current snap is 193 2024-09-20T22:49:08.268 INFO:tasks.rados.rados.0.smithi084.stdout:1825: finishing write tid 1 to smithi08477203-3 2024-09-20T22:49:08.268 INFO:tasks.rados.rados.0.smithi084.stdout:1825: finishing write tid 2 to smithi08477203-3 2024-09-20T22:49:08.268 INFO:tasks.rados.rados.0.smithi084.stdout:1825: finishing write tid 3 to smithi08477203-3 2024-09-20T22:49:08.268 INFO:tasks.rados.rados.0.smithi084.stdout:1825: finishing write tid 4 to smithi08477203-3 2024-09-20T22:49:08.268 INFO:tasks.rados.rados.0.smithi084.stdout:1825: oid 3 updating version 0 to 758 2024-09-20T22:49:08.268 INFO:tasks.rados.rados.0.smithi084.stdout:1825: oid 3 version 758 is already newer than 756 2024-09-20T22:49:08.268 INFO:tasks.rados.rados.0.smithi084.stdout:1825: oid 3 version 758 is already newer than 757 2024-09-20T22:49:08.268 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 3 v 758 (ObjNum 641 snap 193 seq_num 641) dirty exists 2024-09-20T22:49:08.268 INFO:tasks.rados.rados.0.smithi084.stdout:1825: left oid 3 (ObjNum 641 snap 193 seq_num 641) 2024-09-20T22:49:08.268 INFO:tasks.rados.rados.0.smithi084.stdout:1828: seq_num 643 ranges {406901=558759,1393804=579876,2608396=198505} 2024-09-20T22:49:08.272 INFO:tasks.rados.rados.0.smithi084.stdout:1828: writing smithi08477203-5 from 406901 to 965660 tid 1 2024-09-20T22:49:08.278 INFO:tasks.rados.rados.0.smithi084.stdout:1828: writing smithi08477203-5 from 1393804 to 1973680 tid 2 2024-09-20T22:49:08.284 INFO:tasks.rados.rados.0.smithi084.stdout:1828: writing smithi08477203-5 from 2608396 to 2806901 tid 3 2024-09-20T22:49:08.284 INFO:tasks.rados.rados.0.smithi084.stdout:1825: done (4 left) 2024-09-20T22:49:08.284 INFO:tasks.rados.rados.0.smithi084.stdout:1829: delete oid 41 current snap is 193 2024-09-20T22:49:08.284 INFO:tasks.rados.rados.0.smithi084.stdout:1826: finishing write tid 1 to smithi08477203-26 2024-09-20T22:49:08.284 INFO:tasks.rados.rados.0.smithi084.stdout:1827: finishing copy_from to smithi08477203-36 2024-09-20T22:49:08.284 INFO:tasks.rados.rados.0.smithi084.stdout:1827: got expected ENOENT (src dne) 2024-09-20T22:49:08.284 INFO:tasks.rados.rados.0.smithi084.stdout:1827: finishing copy_from racing read to smithi08477203-36 2024-09-20T22:49:08.285 INFO:tasks.rados.rados.0.smithi084.stdout:1824: finishing copy_from to smithi08477203-42 2024-09-20T22:49:08.285 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 42 v 560 (ObjNum 575 snap 172 seq_num 575) dirty exists 2024-09-20T22:49:08.285 INFO:tasks.rados.rados.0.smithi084.stdout:1826: finishing write tid 2 to smithi08477203-26 2024-09-20T22:49:08.285 INFO:tasks.rados.rados.0.smithi084.stdout:1828: finishing write tid 1 to smithi08477203-5 2024-09-20T22:49:08.285 INFO:tasks.rados.rados.0.smithi084.stdout:1826: finishing write tid 3 to smithi08477203-26 2024-09-20T22:49:08.285 INFO:tasks.rados.rados.0.smithi084.stdout:1826: finishing write tid 4 to smithi08477203-26 2024-09-20T22:49:08.285 INFO:tasks.rados.rados.0.smithi084.stdout:1826: oid 26 updating version 0 to 543 2024-09-20T22:49:08.285 INFO:tasks.rados.rados.0.smithi084.stdout:1826: oid 26 version 543 is already newer than 542 2024-09-20T22:49:08.285 INFO:tasks.rados.rados.0.smithi084.stdout:1826: oid 26 version 543 is already newer than 537 2024-09-20T22:49:08.285 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 26 v 543 (ObjNum 642 snap 193 seq_num 642) dirty exists 2024-09-20T22:49:08.285 INFO:tasks.rados.rados.0.smithi084.stdout:1826: left oid 26 (ObjNum 642 snap 193 seq_num 642) 2024-09-20T22:49:08.285 INFO:tasks.rados.rados.0.smithi084.stdout:1828: finishing write tid 2 to smithi08477203-5 2024-09-20T22:49:08.286 INFO:tasks.rados.rados.0.smithi084.stdout:1824: done (4 left) 2024-09-20T22:49:08.286 INFO:tasks.rados.rados.0.smithi084.stdout:1826: done (3 left) 2024-09-20T22:49:08.286 INFO:tasks.rados.rados.0.smithi084.stdout:1827: done (2 left) 2024-09-20T22:49:08.286 INFO:tasks.rados.rados.0.smithi084.stdout:1829: done (1 left) 2024-09-20T22:49:08.286 INFO:tasks.rados.rados.0.smithi084.stdout:1830: delete oid 1 current snap is 193 2024-09-20T22:49:08.286 INFO:tasks.rados.rados.0.smithi084.stdout:1828: finishing write tid 3 to smithi08477203-5 2024-09-20T22:49:08.287 INFO:tasks.rados.rados.0.smithi084.stdout:1828: finishing write tid 4 to smithi08477203-5 2024-09-20T22:49:08.287 INFO:tasks.rados.rados.0.smithi084.stdout:1828: finishing write tid 5 to smithi08477203-5 2024-09-20T22:49:08.287 INFO:tasks.rados.rados.0.smithi084.stdout:1828: oid 5 updating version 0 to 697 2024-09-20T22:49:08.287 INFO:tasks.rados.rados.0.smithi084.stdout:1828: oid 5 version 697 is already newer than 696 2024-09-20T22:49:08.287 INFO:tasks.rados.rados.0.smithi084.stdout:1828: oid 5 updating version 697 to 698 2024-09-20T22:49:08.287 INFO:tasks.rados.rados.0.smithi084.stdout:1828: oid 5 updating version 698 to 699 2024-09-20T22:49:08.287 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 5 v 699 (ObjNum 643 snap 193 seq_num 643) dirty exists 2024-09-20T22:49:08.287 INFO:tasks.rados.rados.0.smithi084.stdout:1828: left oid 5 (ObjNum 643 snap 193 seq_num 643) 2024-09-20T22:49:08.288 INFO:tasks.rados.rados.0.smithi084.stdout:1828: done (1 left) 2024-09-20T22:49:08.288 INFO:tasks.rados.rados.0.smithi084.stdout:1830: done (0 left) 2024-09-20T22:49:08.288 INFO:tasks.rados.rados.0.smithi084.stdout:1831: rollback oid 38 current snap is 193 2024-09-20T22:49:08.288 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 38 to 186 2024-09-20T22:49:08.288 INFO:tasks.rados.rados.0.smithi084.stdout:1832: read oid 43 snap -1 2024-09-20T22:49:08.288 INFO:tasks.rados.rados.0.smithi084.stdout:1832: expect deleted 2024-09-20T22:49:08.288 INFO:tasks.rados.rados.0.smithi084.stdout:1833: snap_create 2024-09-20T22:49:08.289 INFO:tasks.rados.rados.0.smithi084.stdout:1831: finishing rollback tid 1 to smithi08477203-38 2024-09-20T22:49:08.289 INFO:tasks.rados.rados.0.smithi084.stdout:1831: finishing rollback tid 2 to smithi08477203-38 2024-09-20T22:49:08.289 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 38 v 545 (ObjNum 489 snap 137 seq_num 489) dirty exists 2024-09-20T22:49:08.353 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:49:08 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: ::ffff:172.21.15.154 - - [20/Sep/2024:22:49:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-20T22:49:08.353 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:08 smithi084 ceph-mon[40024]: 4.19 scrub starts 2024-09-20T22:49:08.353 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:08 smithi084 ceph-mon[40024]: 4.19 scrub ok 2024-09-20T22:49:08.353 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:08 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:49:08.353 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:08 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:49:08.353 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:08 smithi084 ceph-mon[40024]: osdmap e468: 8 total, 8 up, 8 in 2024-09-20T22:49:08.354 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:08 smithi084 ceph-mon[29774]: 4.19 scrub starts 2024-09-20T22:49:08.354 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:08 smithi084 ceph-mon[29774]: 4.19 scrub ok 2024-09-20T22:49:08.354 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:08 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:49:08.354 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:08 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:49:08.354 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:08 smithi084 ceph-mon[29774]: osdmap e468: 8 total, 8 up, 8 in 2024-09-20T22:49:08.361 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:49:08.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:08.383 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:08.383 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:49:08.484 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:49:08.506 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:08.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:08.507 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:49:08.607 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:49:08.620 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-20T22:49:08.620 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2024-09-20T22:49:08.620 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2024-09-20T22:49:08.620 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [ 2024-09-20T22:49:08.620 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.621 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:554 7.a 7:5122292a:::smithi08477203-33:head [copy-from ver 6] snapc 13=[13,12,11,10,f,d] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-09-20T22:49:08.621 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:43:27.336824+0000", 2024-09-20T22:49:08.621 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 341.280445009, 2024-09-20T22:49:08.621 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.056403397000000001, 2024-09-20T22:49:08.621 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:49:08.621 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:49:08.621 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:49:08.621 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:49:08.621 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:49:08.621 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:49:08.621 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 554 2024-09-20T22:49:08.621 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.621 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:49:08.621 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.622 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:49:08.622 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:27.336824+0000", 2024-09-20T22:49:08.622 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:49:08.622 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.622 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.622 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:49:08.622 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:27.336824+0000", 2024-09-20T22:49:08.622 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:49:08.622 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.622 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.622 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:49:08.622 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:27.336826+0000", 2024-09-20T22:49:08.622 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.0339999999999999e-06 2024-09-20T22:49:08.622 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.623 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.623 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:49:08.623 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:27.336837+0000", 2024-09-20T22:49:08.623 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1243000000000001e-05 2024-09-20T22:49:08.623 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.623 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.623 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:49:08.623 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:27.336857+0000", 2024-09-20T22:49:08.623 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.9868999999999999e-05 2024-09-20T22:49:08.623 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.623 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.623 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:49:08.623 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:27.336869+0000", 2024-09-20T22:49:08.623 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1348999999999999e-05 2024-09-20T22:49:08.624 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.624 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.624 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:49:08.624 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:27.336946+0000", 2024-09-20T22:49:08.624 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.7683999999999997e-05 2024-09-20T22:49:08.624 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.624 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.624 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:49:08.624 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:27.337045+0000", 2024-09-20T22:49:08.624 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.9265999999999995e-05 2024-09-20T22:49:08.624 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.624 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.624 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 2,5", 2024-09-20T22:49:08.624 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:27.340603+0000", 2024-09-20T22:49:08.625 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.003557892 2024-09-20T22:49:08.625 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.625 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.625 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:49:08.625 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:27.343571+0000", 2024-09-20T22:49:08.625 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0029676609999999999 2024-09-20T22:49:08.625 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.625 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.625 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:49:08.625 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:27.346457+0000", 2024-09-20T22:49:08.625 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0028860040000000002 2024-09-20T22:49:08.625 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.625 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.625 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:49:08.625 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:27.393131+0000", 2024-09-20T22:49:08.626 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.046673813000000001 2024-09-20T22:49:08.626 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.626 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.626 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:49:08.626 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:27.393173+0000", 2024-09-20T22:49:08.626 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.2076999999999997e-05 2024-09-20T22:49:08.626 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.626 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.626 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:49:08.626 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:27.393227+0000", 2024-09-20T22:49:08.626 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.4505000000000001e-05 2024-09-20T22:49:08.626 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:49:08.626 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:49:08.626 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:49:08.627 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.627 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.627 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:663 7.12 7:4fa4b2b3:::smithi08477203-21:head [write 665577~781581,stat] snapc 17=[17,16,15,11,d] ondisk+write+known_if_redirected+supports_pool_eio e137)", 2024-09-20T22:49:08.627 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:43:36.161568+0000", 2024-09-20T22:49:08.627 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 332.45570083199999, 2024-09-20T22:49:08.627 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.99286350899999998, 2024-09-20T22:49:08.627 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:49:08.627 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:49:08.627 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:49:08.627 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:49:08.627 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:49:08.627 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:49:08.627 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 663 2024-09-20T22:49:08.627 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.628 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:49:08.628 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.628 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:49:08.628 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.161568+0000", 2024-09-20T22:49:08.628 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:49:08.628 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.628 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.628 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:49:08.628 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.161568+0000", 2024-09-20T22:49:08.628 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:49:08.628 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.628 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.628 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:49:08.628 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.161570+0000", 2024-09-20T22:49:08.628 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.0679999999999999e-06 2024-09-20T22:49:08.629 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.629 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.629 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:49:08.629 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.162310+0000", 2024-09-20T22:49:08.629 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00074005999999999996 2024-09-20T22:49:08.629 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.629 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.629 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:49:08.629 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.162349+0000", 2024-09-20T22:49:08.629 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.8778e-05 2024-09-20T22:49:08.629 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.629 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.629 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:49:08.629 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.162375+0000", 2024-09-20T22:49:08.629 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.6089e-05 2024-09-20T22:49:08.630 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.630 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.630 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:49:08.630 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.162434+0000", 2024-09-20T22:49:08.630 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.8764999999999997e-05 2024-09-20T22:49:08.630 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.630 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.630 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-09-20T22:49:08.630 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.162443+0000", 2024-09-20T22:49:08.630 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.054e-06 2024-09-20T22:49:08.630 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.630 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.630 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:49:08.630 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.143416+0000", 2024-09-20T22:49:08.631 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.98097291600000003 2024-09-20T22:49:08.631 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.631 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.631 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:49:08.631 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.143522+0000", 2024-09-20T22:49:08.631 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00010626 2024-09-20T22:49:08.631 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.631 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.631 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 1,2", 2024-09-20T22:49:08.631 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.143654+0000", 2024-09-20T22:49:08.631 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013218799999999999 2024-09-20T22:49:08.631 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.631 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.631 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:49:08.631 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.148086+0000", 2024-09-20T22:49:08.632 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0044314189999999998 2024-09-20T22:49:08.632 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.632 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.632 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:49:08.632 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.153546+0000", 2024-09-20T22:49:08.632 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0054599920000000003 2024-09-20T22:49:08.632 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.632 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.632 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:49:08.632 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.154317+0000", 2024-09-20T22:49:08.632 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00077125999999999996 2024-09-20T22:49:08.632 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.632 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.632 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:49:08.632 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.154380+0000", 2024-09-20T22:49:08.633 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.2526000000000004e-05 2024-09-20T22:49:08.633 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.633 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.633 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:49:08.633 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.154432+0000", 2024-09-20T22:49:08.633 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.2133999999999999e-05 2024-09-20T22:49:08.633 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:49:08.633 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:49:08.633 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:49:08.633 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.633 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.633 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:664 7.12 7:4fa4b2b3:::smithi08477203-21:head [write 2062560~563688] snapc 17=[17,16,15,11,d] ondisk+write+known_if_redirected+supports_pool_eio e137)", 2024-09-20T22:49:08.634 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:43:36.167947+0000", 2024-09-20T22:49:08.634 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 332.44932236199998, 2024-09-20T22:49:08.634 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.98665172599999995, 2024-09-20T22:49:08.634 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:49:08.634 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:49:08.635 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:49:08.635 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:49:08.635 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:49:08.635 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:49:08.635 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 664 2024-09-20T22:49:08.635 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.635 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:49:08.635 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.635 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:49:08.635 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.167947+0000", 2024-09-20T22:49:08.635 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:49:08.635 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.635 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.635 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:49:08.635 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.167947+0000", 2024-09-20T22:49:08.636 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:49:08.636 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.636 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.636 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:49:08.636 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.167948+0000", 2024-09-20T22:49:08.636 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.187e-06 2024-09-20T22:49:08.636 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.636 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.636 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:49:08.636 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.168193+0000", 2024-09-20T22:49:08.636 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00024496199999999999 2024-09-20T22:49:08.636 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.636 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.636 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:49:08.636 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.168203+0000", 2024-09-20T22:49:08.637 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0543999999999999e-05 2024-09-20T22:49:08.637 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.637 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.637 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:49:08.637 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.168209+0000", 2024-09-20T22:49:08.637 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.2839999999999997e-06 2024-09-20T22:49:08.637 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.637 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.637 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:49:08.637 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.168252+0000", 2024-09-20T22:49:08.637 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.3115000000000003e-05 2024-09-20T22:49:08.637 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.637 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.637 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-09-20T22:49:08.638 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.168257+0000", 2024-09-20T22:49:08.638 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.6860000000000003e-06 2024-09-20T22:49:08.638 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.638 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.638 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:49:08.638 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.144795+0000", 2024-09-20T22:49:08.638 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.97653758099999999 2024-09-20T22:49:08.638 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.638 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.638 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:49:08.638 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.144865+0000", 2024-09-20T22:49:08.638 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.9641000000000006e-05 2024-09-20T22:49:08.638 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.638 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.639 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 1,2", 2024-09-20T22:49:08.639 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.144986+0000", 2024-09-20T22:49:08.639 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000121248 2024-09-20T22:49:08.639 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.639 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.639 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:49:08.639 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.148091+0000", 2024-09-20T22:49:08.639 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0031050399999999999 2024-09-20T22:49:08.639 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.639 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.639 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:49:08.639 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.153603+0000", 2024-09-20T22:49:08.639 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0055118479999999997 2024-09-20T22:49:08.639 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.639 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.640 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:49:08.640 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.154508+0000", 2024-09-20T22:49:08.640 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00090530600000000004 2024-09-20T22:49:08.640 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.640 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.640 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:49:08.640 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.154560+0000", 2024-09-20T22:49:08.640 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.1474000000000002e-05 2024-09-20T22:49:08.640 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.640 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.640 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:49:08.640 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.154598+0000", 2024-09-20T22:49:08.640 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.8810000000000003e-05 2024-09-20T22:49:08.640 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:49:08.641 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:49:08.641 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:49:08.641 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.641 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.641 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:665 7.12 7:4fa4b2b3:::smithi08477203-21:head [write 3182305~643664] snapc 17=[17,16,15,11,d] ondisk+write+known_if_redirected+supports_pool_eio e137)", 2024-09-20T22:49:08.641 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:43:36.174520+0000", 2024-09-20T22:49:08.641 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 332.44274927200001, 2024-09-20T22:49:08.641 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.98020083400000002, 2024-09-20T22:49:08.641 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:49:08.641 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:49:08.641 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:49:08.641 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:49:08.641 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:49:08.641 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:49:08.642 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 665 2024-09-20T22:49:08.642 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.642 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:49:08.642 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.642 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:49:08.642 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174520+0000", 2024-09-20T22:49:08.642 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:49:08.642 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.642 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.642 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:49:08.642 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174520+0000", 2024-09-20T22:49:08.642 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:49:08.642 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.643 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.643 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:49:08.643 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174521+0000", 2024-09-20T22:49:08.643 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.234e-06 2024-09-20T22:49:08.643 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.643 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.643 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:49:08.643 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174816+0000", 2024-09-20T22:49:08.643 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000294682 2024-09-20T22:49:08.643 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.643 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.643 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:49:08.643 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174827+0000", 2024-09-20T22:49:08.643 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1025e-05 2024-09-20T22:49:08.643 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.644 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.644 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:49:08.644 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174832+0000", 2024-09-20T22:49:08.644 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.3990000000000003e-06 2024-09-20T22:49:08.644 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.644 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.644 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:49:08.644 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174868+0000", 2024-09-20T22:49:08.644 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.6225e-05 2024-09-20T22:49:08.644 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.644 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.644 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-09-20T22:49:08.644 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174873+0000", 2024-09-20T22:49:08.644 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.848e-06 2024-09-20T22:49:08.645 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.645 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.645 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:49:08.645 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.145844+0000", 2024-09-20T22:49:08.645 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.97097048900000005 2024-09-20T22:49:08.645 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.645 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.645 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:49:08.645 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.145917+0000", 2024-09-20T22:49:08.645 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.3248000000000001e-05 2024-09-20T22:49:08.645 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.645 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.645 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 1,2", 2024-09-20T22:49:08.645 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.146028+0000", 2024-09-20T22:49:08.646 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000110773 2024-09-20T22:49:08.646 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.646 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.646 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:49:08.646 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.148095+0000", 2024-09-20T22:49:08.646 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0020678179999999999 2024-09-20T22:49:08.646 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.646 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.646 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:49:08.646 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.153967+0000", 2024-09-20T22:49:08.646 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0058717350000000003 2024-09-20T22:49:08.646 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.646 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.646 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:49:08.647 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.154633+0000", 2024-09-20T22:49:08.647 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00066600600000000002 2024-09-20T22:49:08.647 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.647 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.647 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:49:08.647 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.154683+0000", 2024-09-20T22:49:08.647 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.9872e-05 2024-09-20T22:49:08.647 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.647 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.647 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:49:08.647 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.154721+0000", 2024-09-20T22:49:08.647 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.748e-05 2024-09-20T22:49:08.647 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:49:08.647 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:49:08.648 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:49:08.648 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.648 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.648 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:666 7.12 7:4fa4b2b3:::smithi08477203-21:head [setxattr _header (57) in=7b,truncate 3865577] snapc 17=[17,16,15,11,d] ondisk+write+known_if_redirected+supports_pool_eio e137)", 2024-09-20T22:49:08.648 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:43:36.174844+0000", 2024-09-20T22:49:08.648 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 332.44242507299998, 2024-09-20T22:49:08.648 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.98002428600000002, 2024-09-20T22:49:08.648 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:49:08.648 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:49:08.648 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:49:08.648 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:49:08.648 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:49:08.648 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:49:08.649 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 666 2024-09-20T22:49:08.649 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.649 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:49:08.649 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.649 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:49:08.649 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174844+0000", 2024-09-20T22:49:08.649 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:49:08.649 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.649 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.649 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:49:08.649 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174844+0000", 2024-09-20T22:49:08.649 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:49:08.649 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.649 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.649 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:49:08.650 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174845+0000", 2024-09-20T22:49:08.650 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.1999999999999999e-07 2024-09-20T22:49:08.650 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.650 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.650 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:49:08.650 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174853+0000", 2024-09-20T22:49:08.650 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.1950000000000003e-06 2024-09-20T22:49:08.650 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.650 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.650 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:49:08.650 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174858+0000", 2024-09-20T22:49:08.650 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.5650000000000001e-06 2024-09-20T22:49:08.650 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.650 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.651 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:49:08.651 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174863+0000", 2024-09-20T22:49:08.651 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.1649999999999997e-06 2024-09-20T22:49:08.651 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.651 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.651 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:49:08.651 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174915+0000", 2024-09-20T22:49:08.651 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.1045999999999999e-05 2024-09-20T22:49:08.651 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.651 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.651 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-09-20T22:49:08.651 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174921+0000", 2024-09-20T22:49:08.651 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.404e-06 2024-09-20T22:49:08.651 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.652 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.652 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:49:08.652 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.146528+0000", 2024-09-20T22:49:08.652 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.97160681199999999 2024-09-20T22:49:08.652 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.652 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.652 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:49:08.652 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.146881+0000", 2024-09-20T22:49:08.652 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00035348200000000001 2024-09-20T22:49:08.652 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.652 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.652 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 1,2", 2024-09-20T22:49:08.652 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.147011+0000", 2024-09-20T22:49:08.653 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013018900000000001 2024-09-20T22:49:08.653 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.653 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.653 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:49:08.653 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.148100+0000", 2024-09-20T22:49:08.653 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010881580000000001 2024-09-20T22:49:08.653 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.653 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.653 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:49:08.653 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.154278+0000", 2024-09-20T22:49:08.653 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0061781730000000003 2024-09-20T22:49:08.653 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.653 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.653 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:49:08.654 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.154764+0000", 2024-09-20T22:49:08.654 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00048589199999999998 2024-09-20T22:49:08.654 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.654 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.654 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:49:08.654 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.154815+0000", 2024-09-20T22:49:08.654 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.1428000000000003e-05 2024-09-20T22:49:08.654 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.654 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.654 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:49:08.654 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.154868+0000", 2024-09-20T22:49:08.654 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.3156999999999999e-05 2024-09-20T22:49:08.654 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:49:08.655 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:49:08.655 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:49:08.655 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.655 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.655 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:667 7.12 7:4fa4b2b3:::smithi08477203-21:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e137)", 2024-09-20T22:49:08.655 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:43:36.174872+0000", 2024-09-20T22:49:08.655 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 332.44239737999999, 2024-09-20T22:49:08.655 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.98024790299999998, 2024-09-20T22:49:08.655 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:49:08.655 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:49:08.655 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-20T22:49:08.655 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:49:08.655 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:49:08.655 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:49:08.655 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 667 2024-09-20T22:49:08.656 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.656 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:49:08.656 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.656 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:49:08.656 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174872+0000", 2024-09-20T22:49:08.656 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:49:08.656 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.656 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.656 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:49:08.656 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174872+0000", 2024-09-20T22:49:08.656 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:49:08.656 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.656 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.656 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:49:08.656 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174872+0000", 2024-09-20T22:49:08.657 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.2199999999999999e-07 2024-09-20T22:49:08.657 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.657 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.657 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:49:08.657 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174874+0000", 2024-09-20T22:49:08.657 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.187e-06 2024-09-20T22:49:08.657 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.657 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.657 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:49:08.657 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174881+0000", 2024-09-20T22:49:08.657 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.674e-06 2024-09-20T22:49:08.657 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.657 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.657 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:49:08.657 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174886+0000", 2024-09-20T22:49:08.658 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.2240000000000003e-06 2024-09-20T22:49:08.658 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.658 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.658 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:49:08.658 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174936+0000", 2024-09-20T22:49:08.658 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.9824999999999998e-05 2024-09-20T22:49:08.658 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.658 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.658 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-09-20T22:49:08.658 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174940+0000", 2024-09-20T22:49:08.658 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.5700000000000001e-06 2024-09-20T22:49:08.658 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.658 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.658 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:49:08.659 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.147523+0000", 2024-09-20T22:49:08.659 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.97258315399999995 2024-09-20T22:49:08.659 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.659 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.659 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for rw locks", 2024-09-20T22:49:08.659 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.147591+0000", 2024-09-20T22:49:08.659 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.7878000000000003e-05 2024-09-20T22:49:08.659 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.659 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.659 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:49:08.659 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.154925+0000", 2024-09-20T22:49:08.659 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0073344430000000004 2024-09-20T22:49:08.659 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.659 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.659 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:49:08.660 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.155020+0000", 2024-09-20T22:49:08.660 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.4767999999999998e-05 2024-09-20T22:49:08.660 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.660 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.660 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:49:08.660 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.155120+0000", 2024-09-20T22:49:08.660 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.9758000000000003e-05 2024-09-20T22:49:08.660 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:49:08.660 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:49:08.660 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:49:08.660 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.660 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.660 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.14478.0:1259 5.3 5:c609908c:::notify.5:head [watch reconnect cookie 94843898679808 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e169)", 2024-09-20T22:49:08.660 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:44:08.230723+0000", 2024-09-20T22:49:08.660 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 300.38654607500001, 2024-09-20T22:49:08.661 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.94983577399999997, 2024-09-20T22:49:08.661 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:49:08.661 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:49:08.661 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-20T22:49:08.661 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:49:08.661 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.14478", 2024-09-20T22:49:08.661 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/2994273271", 2024-09-20T22:49:08.661 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1259 2024-09-20T22:49:08.661 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.661 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:49:08.661 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.661 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:49:08.661 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:08.230723+0000", 2024-09-20T22:49:08.661 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:49:08.662 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.662 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.662 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:49:08.662 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:08.230723+0000", 2024-09-20T22:49:08.662 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:49:08.662 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.662 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.662 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:49:08.662 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:08.230724+0000", 2024-09-20T22:49:08.662 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.077e-06 2024-09-20T22:49:08.662 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.662 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.662 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:49:08.662 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:08.230729+0000", 2024-09-20T22:49:08.662 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.9930000000000003e-06 2024-09-20T22:49:08.663 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.663 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.663 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:49:08.663 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:08.230741+0000", 2024-09-20T22:49:08.663 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1897e-05 2024-09-20T22:49:08.663 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.663 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.663 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:49:08.663 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:08.230747+0000", 2024-09-20T22:49:08.663 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.9830000000000001e-06 2024-09-20T22:49:08.663 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.663 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.663 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:49:08.664 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:08.230802+0000", 2024-09-20T22:49:08.664 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.4657000000000002e-05 2024-09-20T22:49:08.664 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.664 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.664 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-09-20T22:49:08.664 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:08.230811+0000", 2024-09-20T22:49:08.664 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.1160000000000005e-06 2024-09-20T22:49:08.664 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.664 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.664 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:49:08.664 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:09.171968+0000", 2024-09-20T22:49:08.664 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.94115747299999997 2024-09-20T22:49:08.664 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.664 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.664 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for missing object", 2024-09-20T22:49:08.665 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:09.172098+0000", 2024-09-20T22:49:08.665 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00012937499999999999 2024-09-20T22:49:08.665 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.665 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.665 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:49:08.665 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:09.174248+0000", 2024-09-20T22:49:08.665 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0021508959999999998 2024-09-20T22:49:08.665 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.665 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.665 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for rw locks", 2024-09-20T22:49:08.665 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:09.174341+0000", 2024-09-20T22:49:08.665 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.2906999999999997e-05 2024-09-20T22:49:08.665 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.665 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.665 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:49:08.666 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:09.175083+0000", 2024-09-20T22:49:08.666 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00074120100000000003 2024-09-20T22:49:08.666 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.666 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.666 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:49:08.666 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:09.175149+0000", 2024-09-20T22:49:08.666 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.6877000000000004e-05 2024-09-20T22:49:08.666 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.666 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.666 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:49:08.666 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:09.180559+0000", 2024-09-20T22:49:08.666 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0054093220000000003 2024-09-20T22:49:08.666 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:49:08.666 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:49:08.667 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:49:08.667 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.667 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.667 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:1418 7.1 7:84a866f5:::smithi08477203-27:head [rollback 2f] snapc 2f=[2f] ondisk+write+known_if_redirected+supports_pool_eio e189)", 2024-09-20T22:49:08.667 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:44:28.137587+0000", 2024-09-20T22:49:08.667 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 280.47968187800001, 2024-09-20T22:49:08.667 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.020550452, 2024-09-20T22:49:08.667 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:49:08.667 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:49:08.667 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:49:08.667 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:49:08.667 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:49:08.667 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:49:08.667 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1418 2024-09-20T22:49:08.668 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.668 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:49:08.668 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.668 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:49:08.668 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.137587+0000", 2024-09-20T22:49:08.668 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:49:08.668 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.668 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.668 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:49:08.668 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.137587+0000", 2024-09-20T22:49:08.668 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:49:08.668 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.668 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.668 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:49:08.668 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.137588+0000", 2024-09-20T22:49:08.669 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.4700000000000001e-07 2024-09-20T22:49:08.669 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.669 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.669 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:49:08.669 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.137591+0000", 2024-09-20T22:49:08.669 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.0000000000000001e-06 2024-09-20T22:49:08.669 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.669 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.669 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:49:08.669 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.137681+0000", 2024-09-20T22:49:08.669 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.0486000000000001e-05 2024-09-20T22:49:08.669 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.669 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.669 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:49:08.670 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.138018+0000", 2024-09-20T22:49:08.670 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00033709899999999999 2024-09-20T22:49:08.670 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.670 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.670 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:49:08.670 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.138109+0000", 2024-09-20T22:49:08.670 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.0538999999999995e-05 2024-09-20T22:49:08.670 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.670 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.670 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting_for_map not empty", 2024-09-20T22:49:08.670 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.138114+0000", 2024-09-20T22:49:08.670 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.6590000000000004e-06 2024-09-20T22:49:08.670 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.670 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.671 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:49:08.671 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.144398+0000", 2024-09-20T22:49:08.671 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0062832599999999997 2024-09-20T22:49:08.671 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.671 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.671 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:49:08.671 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.144483+0000", 2024-09-20T22:49:08.671 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.4772999999999996e-05 2024-09-20T22:49:08.671 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.671 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.671 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 1,6", 2024-09-20T22:49:08.671 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.144661+0000", 2024-09-20T22:49:08.671 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000178736 2024-09-20T22:49:08.671 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.672 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.672 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:49:08.672 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.155602+0000", 2024-09-20T22:49:08.672 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010940302000000001 2024-09-20T22:49:08.672 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.672 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.672 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:49:08.672 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.156162+0000", 2024-09-20T22:49:08.672 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00056014599999999997 2024-09-20T22:49:08.672 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.672 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.672 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:49:08.672 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.158054+0000", 2024-09-20T22:49:08.672 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00189223 2024-09-20T22:49:08.672 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.673 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.673 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:49:08.673 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.158100+0000", 2024-09-20T22:49:08.673 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.6279999999999997e-05 2024-09-20T22:49:08.673 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.673 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.673 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:49:08.673 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.158138+0000", 2024-09-20T22:49:08.673 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.7394999999999997e-05 2024-09-20T22:49:08.673 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:49:08.673 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:49:08.673 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:49:08.673 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.673 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.673 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:1419 7.1 7:84a866f5:::smithi08477203-27:head [append 0~0] snapc 2f=[2f] ondisk+write+known_if_redirected+supports_pool_eio e189)", 2024-09-20T22:49:08.674 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:44:28.138055+0000", 2024-09-20T22:49:08.674 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 280.47921443600001, 2024-09-20T22:49:08.674 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.030290338999999999, 2024-09-20T22:49:08.674 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:49:08.674 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:49:08.674 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:49:08.674 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:49:08.674 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:49:08.674 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:49:08.674 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1419 2024-09-20T22:49:08.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:08.675 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:08.676 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:49:08.676 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.676 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:49:08.677 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.677 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:49:08.677 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.138055+0000", 2024-09-20T22:49:08.677 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:49:08.677 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.677 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.677 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:49:08.677 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.138055+0000", 2024-09-20T22:49:08.677 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:49:08.677 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.677 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.677 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:49:08.677 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.138056+0000", 2024-09-20T22:49:08.678 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.13e-06 2024-09-20T22:49:08.678 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.678 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.678 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:49:08.678 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.138060+0000", 2024-09-20T22:49:08.678 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.0269999999999997e-06 2024-09-20T22:49:08.678 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.678 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.678 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:49:08.678 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.138070+0000", 2024-09-20T22:49:08.678 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.961e-06 2024-09-20T22:49:08.678 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.678 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.678 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:49:08.678 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.138076+0000", 2024-09-20T22:49:08.679 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.1800000000000001e-06 2024-09-20T22:49:08.679 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.679 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.679 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:49:08.679 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.138137+0000", 2024-09-20T22:49:08.679 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.1159000000000002e-05 2024-09-20T22:49:08.679 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.679 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.679 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting_for_map not empty", 2024-09-20T22:49:08.679 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.138141+0000", 2024-09-20T22:49:08.679 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.189e-06 2024-09-20T22:49:08.679 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.679 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.679 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:49:08.680 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.154982+0000", 2024-09-20T22:49:08.680 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.016840738000000001 2024-09-20T22:49:08.680 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.680 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.680 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:49:08.680 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.155089+0000", 2024-09-20T22:49:08.680 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00010653399999999999 2024-09-20T22:49:08.680 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.680 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.680 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 1,6", 2024-09-20T22:49:08.680 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.155320+0000", 2024-09-20T22:49:08.680 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00023196299999999999 2024-09-20T22:49:08.681 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.681 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.681 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:49:08.681 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.158224+0000", 2024-09-20T22:49:08.681 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00290361 2024-09-20T22:49:08.681 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.681 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.681 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:49:08.681 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.167686+0000", 2024-09-20T22:49:08.681 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0094622360000000006 2024-09-20T22:49:08.681 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.681 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.681 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:49:08.681 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.168183+0000", 2024-09-20T22:49:08.682 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00049655999999999995 2024-09-20T22:49:08.682 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.682 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.682 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:49:08.682 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.168273+0000", 2024-09-20T22:49:08.682 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.9758000000000004e-05 2024-09-20T22:49:08.682 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.682 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.682 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:49:08.682 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.168345+0000", 2024-09-20T22:49:08.682 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.2293999999999996e-05 2024-09-20T22:49:08.682 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:49:08.682 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:49:08.683 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:49:08.683 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.683 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.683 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:2347 7.a 7:5122292a:::smithi08477203-33:head [sparse-read 0~2590699,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e260)", 2024-09-20T22:49:08.683 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:45:39.196440+0000", 2024-09-20T22:49:08.683 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 209.42082930800001, 2024-09-20T22:49:08.683 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.026701163, 2024-09-20T22:49:08.683 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:49:08.683 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:49:08.683 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-20T22:49:08.683 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:49:08.683 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:49:08.683 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:49:08.683 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 2347 2024-09-20T22:49:08.684 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.684 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:49:08.684 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.684 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:49:08.684 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.196440+0000", 2024-09-20T22:49:08.684 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:49:08.684 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.684 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.684 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:49:08.684 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.196440+0000", 2024-09-20T22:49:08.684 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:49:08.684 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.684 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.684 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:49:08.684 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.196441+0000", 2024-09-20T22:49:08.685 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.714e-06 2024-09-20T22:49:08.685 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.685 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.685 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:49:08.685 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.196449+0000", 2024-09-20T22:49:08.685 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.8730000000000007e-06 2024-09-20T22:49:08.685 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.685 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.685 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:49:08.685 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.196462+0000", 2024-09-20T22:49:08.685 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2649e-05 2024-09-20T22:49:08.685 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.685 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.686 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:49:08.686 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.196470+0000", 2024-09-20T22:49:08.686 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.4989999999999994e-06 2024-09-20T22:49:08.686 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.686 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.686 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:49:08.686 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.212172+0000", 2024-09-20T22:49:08.686 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.015701335 2024-09-20T22:49:08.686 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.686 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.686 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:49:08.686 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.212270+0000", 2024-09-20T22:49:08.686 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.8257999999999993e-05 2024-09-20T22:49:08.687 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.687 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.687 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:49:08.687 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.223141+0000", 2024-09-20T22:49:08.687 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010870835000000001 2024-09-20T22:49:08.687 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:49:08.687 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:49:08.687 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:49:08.687 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.687 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.687 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:2348 7.a 7:5122292a:::smithi08477203-33:head [read 0~3848803,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e260)", 2024-09-20T22:49:08.687 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:45:39.196485+0000", 2024-09-20T22:49:08.687 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 209.42078383500001, 2024-09-20T22:49:08.687 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0284703, 2024-09-20T22:49:08.688 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:49:08.688 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:49:08.688 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-20T22:49:08.688 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:49:08.688 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:49:08.688 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:49:08.688 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 2348 2024-09-20T22:49:08.688 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.688 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:49:08.688 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.688 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:49:08.688 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.196485+0000", 2024-09-20T22:49:08.688 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:49:08.688 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.688 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.689 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:49:08.689 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.196485+0000", 2024-09-20T22:49:08.689 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:49:08.689 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.689 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.689 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:49:08.689 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.196486+0000", 2024-09-20T22:49:08.689 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.9599999999999999e-07 2024-09-20T22:49:08.689 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.689 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.689 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:49:08.689 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.196489+0000", 2024-09-20T22:49:08.689 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.5470000000000001e-06 2024-09-20T22:49:08.690 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.690 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.690 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:49:08.690 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.196496+0000", 2024-09-20T22:49:08.690 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.3570000000000003e-06 2024-09-20T22:49:08.690 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.690 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.690 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:49:08.690 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.196501+0000", 2024-09-20T22:49:08.690 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.6239999999999999e-06 2024-09-20T22:49:08.690 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.690 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.690 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:49:08.690 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.223176+0000", 2024-09-20T22:49:08.691 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.026675285999999999 2024-09-20T22:49:08.691 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.691 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.691 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:49:08.691 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.223274+0000", 2024-09-20T22:49:08.691 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.7009999999999994e-05 2024-09-20T22:49:08.691 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.691 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.691 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:49:08.691 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.224955+0000", 2024-09-20T22:49:08.691 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00168198 2024-09-20T22:49:08.691 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:49:08.691 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:49:08.692 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:49:08.692 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.692 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.692 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:2349 7.a 7:5122292a:::smithi08477203-33:head [sparse-read 0~2590699] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e260)", 2024-09-20T22:49:08.692 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:45:39.196508+0000", 2024-09-20T22:49:08.692 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 209.42076054899999, 2024-09-20T22:49:08.692 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.028731293000000001, 2024-09-20T22:49:08.692 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:49:08.692 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:49:08.692 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-20T22:49:08.692 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:49:08.692 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:49:08.692 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:49:08.692 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 2349 2024-09-20T22:49:08.692 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.693 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:49:08.693 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.693 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:49:08.693 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.196508+0000", 2024-09-20T22:49:08.693 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:49:08.693 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.693 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.693 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:49:08.693 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.196508+0000", 2024-09-20T22:49:08.693 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:49:08.693 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.693 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.693 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:49:08.693 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.196509+0000", 2024-09-20T22:49:08.694 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.08e-07 2024-09-20T22:49:08.694 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.694 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.694 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:49:08.694 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.196511+0000", 2024-09-20T22:49:08.694 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.142e-06 2024-09-20T22:49:08.694 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.694 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.694 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:49:08.694 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.196517+0000", 2024-09-20T22:49:08.694 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.6330000000000001e-06 2024-09-20T22:49:08.694 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.694 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.694 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:49:08.695 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.196522+0000", 2024-09-20T22:49:08.695 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.3530000000000002e-06 2024-09-20T22:49:08.695 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.695 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.695 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:49:08.695 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.224990+0000", 2024-09-20T22:49:08.695 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.028467864999999998 2024-09-20T22:49:08.695 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.695 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.695 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:49:08.695 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.225088+0000", 2024-09-20T22:49:08.695 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.8281000000000003e-05 2024-09-20T22:49:08.695 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.695 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.696 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:49:08.696 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.225240+0000", 2024-09-20T22:49:08.696 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00015161100000000001 2024-09-20T22:49:08.696 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:49:08.696 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:49:08.696 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:49:08.696 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.696 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.696 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(osd.6.0:727 7.c e260/94 7:318c7512:::smithi08477203-44:4c v 260'428, mlcod=258'424)", 2024-09-20T22:49:08.696 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:45:39.202413+0000", 2024-09-20T22:49:08.696 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 209.41485600999999, 2024-09-20T22:49:08.696 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.022852897, 2024-09-20T22:49:08.696 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:49:08.696 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:49:08.697 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:49:08.697 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:49:08.697 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.697 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:49:08.697 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.202413+0000", 2024-09-20T22:49:08.697 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:49:08.697 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.697 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.697 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:49:08.697 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.202413+0000", 2024-09-20T22:49:08.697 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:49:08.697 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.697 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.697 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:49:08.697 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.202414+0000", 2024-09-20T22:49:08.698 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.9299999999999997e-07 2024-09-20T22:49:08.698 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.698 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.698 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:49:08.698 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.202421+0000", 2024-09-20T22:49:08.698 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.8109999999999997e-06 2024-09-20T22:49:08.698 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.698 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.698 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:49:08.698 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.202433+0000", 2024-09-20T22:49:08.698 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2411e-05 2024-09-20T22:49:08.698 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.698 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.699 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:49:08.699 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.202442+0000", 2024-09-20T22:49:08.699 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.7269999999999996e-06 2024-09-20T22:49:08.699 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.699 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.699 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:49:08.699 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.221764+0000", 2024-09-20T22:49:08.699 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.019322068000000001 2024-09-20T22:49:08.699 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.699 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.699 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:49:08.699 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.221786+0000", 2024-09-20T22:49:08.699 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.2167e-05 2024-09-20T22:49:08.699 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.700 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.700 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:49:08.700 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.225217+0000", 2024-09-20T22:49:08.700 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0034313709999999999 2024-09-20T22:49:08.700 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.700 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.700 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:49:08.700 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.225266+0000", 2024-09-20T22:49:08.700 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.8649000000000001e-05 2024-09-20T22:49:08.700 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:49:08.700 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:49:08.700 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:49:08.700 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.700 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.700 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(osd.6.0:728 7.c e260/94 7:31d0211f:::smithi08477203-25:51 v 260'430, mlcod=258'424)", 2024-09-20T22:49:08.701 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:45:39.202454+0000", 2024-09-20T22:49:08.701 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 209.41481514700001, 2024-09-20T22:49:08.701 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.022837705, 2024-09-20T22:49:08.701 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:49:08.701 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:49:08.701 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:49:08.701 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:49:08.701 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.701 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:49:08.701 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.202454+0000", 2024-09-20T22:49:08.701 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:49:08.701 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.701 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.701 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:49:08.701 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.202454+0000", 2024-09-20T22:49:08.702 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:49:08.702 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.702 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.702 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:49:08.702 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.202454+0000", 2024-09-20T22:49:08.702 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.9999999999999999e-07 2024-09-20T22:49:08.702 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.702 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.702 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:49:08.702 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.202464+0000", 2024-09-20T22:49:08.702 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.8179999999999999e-06 2024-09-20T22:49:08.703 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.703 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.703 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:49:08.703 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.202469+0000", 2024-09-20T22:49:08.703 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.1089999999999997e-06 2024-09-20T22:49:08.703 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.703 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.703 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:49:08.703 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.202473+0000", 2024-09-20T22:49:08.703 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.0940000000000001e-06 2024-09-20T22:49:08.703 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.703 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.703 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:49:08.703 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.222250+0000", 2024-09-20T22:49:08.703 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.019776353999999999 2024-09-20T22:49:08.704 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.704 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.704 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:49:08.704 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.222268+0000", 2024-09-20T22:49:08.704 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.8342000000000002e-05 2024-09-20T22:49:08.704 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.704 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.704 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:49:08.704 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.225268+0000", 2024-09-20T22:49:08.704 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00300038 2024-09-20T22:49:08.704 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.704 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.704 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:49:08.704 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.225292+0000", 2024-09-20T22:49:08.705 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.3308000000000001e-05 2024-09-20T22:49:08.705 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:49:08.705 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:49:08.705 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:49:08.705 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.705 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.705 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:2525 7.1 7:8422f82d:::smithi08477203-22:head [copy-from ver 174] snapc 5d=[5d,5c,5a,50,47] ondisk+write+known_if_redirected+supports_pool_eio e278)", 2024-09-20T22:49:08.705 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:45:57.170348+0000", 2024-09-20T22:49:08.705 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 191.44692103599999, 2024-09-20T22:49:08.705 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.023333486000000001, 2024-09-20T22:49:08.705 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:49:08.705 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:49:08.705 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:49:08.705 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:49:08.705 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:49:08.706 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:49:08.706 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 2525 2024-09-20T22:49:08.706 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.706 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:49:08.706 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.706 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:49:08.706 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:57.170348+0000", 2024-09-20T22:49:08.706 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:49:08.706 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.706 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.706 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:49:08.706 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:57.170348+0000", 2024-09-20T22:49:08.706 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:49:08.706 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.707 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.707 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:49:08.707 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:57.170350+0000", 2024-09-20T22:49:08.707 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.1509999999999998e-06 2024-09-20T22:49:08.707 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.707 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.707 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:49:08.707 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:57.170361+0000", 2024-09-20T22:49:08.707 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0492999999999999e-05 2024-09-20T22:49:08.707 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.707 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.707 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:49:08.707 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:57.170381+0000", 2024-09-20T22:49:08.707 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.0509999999999998e-05 2024-09-20T22:49:08.708 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.708 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.708 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:49:08.708 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:57.170393+0000", 2024-09-20T22:49:08.708 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2009e-05 2024-09-20T22:49:08.708 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.708 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.708 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:49:08.708 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:57.170465+0000", 2024-09-20T22:49:08.708 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.1927000000000004e-05 2024-09-20T22:49:08.708 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.708 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.708 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op must wait for map", 2024-09-20T22:49:08.708 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:57.170473+0000", 2024-09-20T22:49:08.708 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.1890000000000007e-06 2024-09-20T22:49:08.709 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.709 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.709 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:49:08.709 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:57.172855+0000", 2024-09-20T22:49:08.709 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002382168 2024-09-20T22:49:08.709 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.709 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.709 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:49:08.709 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:57.172967+0000", 2024-09-20T22:49:08.709 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00011108299999999999 2024-09-20T22:49:08.709 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.709 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.709 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 1,6", 2024-09-20T22:49:08.709 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:57.181533+0000", 2024-09-20T22:49:08.710 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0085667499999999997 2024-09-20T22:49:08.710 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.710 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.710 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:49:08.710 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:57.191599+0000", 2024-09-20T22:49:08.710 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010065795000000001 2024-09-20T22:49:08.710 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.710 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.710 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:49:08.710 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:57.193469+0000", 2024-09-20T22:49:08.710 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0018695199999999999 2024-09-20T22:49:08.710 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.710 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.710 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:49:08.710 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:57.193550+0000", 2024-09-20T22:49:08.711 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.1891000000000006e-05 2024-09-20T22:49:08.711 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.711 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.711 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:49:08.711 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:57.193625+0000", 2024-09-20T22:49:08.711 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.4679999999999996e-05 2024-09-20T22:49:08.711 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.711 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.711 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:49:08.711 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:57.193681+0000", 2024-09-20T22:49:08.711 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.6320000000000003e-05 2024-09-20T22:49:08.711 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:49:08.711 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:49:08.711 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:49:08.712 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.712 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.712 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:2853 7.10 7:0f524219:::smithi08477203-50:head [copy-from ver 114] snapc 68=[68,67] ondisk+write+known_if_redirected+supports_pool_eio e303)", 2024-09-20T22:49:08.712 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:46:23.285507+0000", 2024-09-20T22:49:08.712 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 165.33176230699999, 2024-09-20T22:49:08.712 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.023440631, 2024-09-20T22:49:08.712 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:49:08.712 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:49:08.712 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:49:08.712 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:49:08.712 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:49:08.712 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:49:08.712 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 2853 2024-09-20T22:49:08.712 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.713 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:49:08.713 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.713 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:49:08.713 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.285507+0000", 2024-09-20T22:49:08.713 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:49:08.713 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.713 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.713 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:49:08.713 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.285507+0000", 2024-09-20T22:49:08.713 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:49:08.713 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.713 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.713 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:49:08.713 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.285508+0000", 2024-09-20T22:49:08.713 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.491e-06 2024-09-20T22:49:08.714 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.714 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.714 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:49:08.714 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.285514+0000", 2024-09-20T22:49:08.714 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.502e-06 2024-09-20T22:49:08.714 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.714 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.714 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:49:08.714 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.285526+0000", 2024-09-20T22:49:08.714 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2639e-05 2024-09-20T22:49:08.714 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.714 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.714 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:49:08.714 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.285535+0000", 2024-09-20T22:49:08.715 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.4689999999999997e-06 2024-09-20T22:49:08.715 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.715 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.715 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:49:08.715 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.285591+0000", 2024-09-20T22:49:08.715 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.6407999999999999e-05 2024-09-20T22:49:08.715 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.715 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.715 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:49:08.715 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.285681+0000", 2024-09-20T22:49:08.715 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.9845000000000005e-05 2024-09-20T22:49:08.715 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.715 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.715 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 1,2", 2024-09-20T22:49:08.716 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.293719+0000", 2024-09-20T22:49:08.716 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0080383950000000003 2024-09-20T22:49:08.716 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.716 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.716 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:49:08.716 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.307398+0000", 2024-09-20T22:49:08.716 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013678074 2024-09-20T22:49:08.716 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.716 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.716 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:49:08.716 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.308451+0000", 2024-09-20T22:49:08.716 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001052975 2024-09-20T22:49:08.716 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.716 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.716 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:49:08.717 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.308811+0000", 2024-09-20T22:49:08.717 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00036041399999999998 2024-09-20T22:49:08.717 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.717 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.717 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:49:08.717 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.308878+0000", 2024-09-20T22:49:08.717 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.7114999999999997e-05 2024-09-20T22:49:08.717 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.717 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.717 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:49:08.717 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.308947+0000", 2024-09-20T22:49:08.717 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.9303999999999999e-05 2024-09-20T22:49:08.717 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:49:08.717 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:49:08.717 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:49:08.718 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.718 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.718 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:3136 7.a 7:5122292a:::smithi08477203-33:head [copy-from ver 385] snapc 77=[77,76,75,74,73,70,6b] ondisk+write+known_if_redirected+supports_pool_eio e328)", 2024-09-20T22:49:08.718 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:46:48.210350+0000", 2024-09-20T22:49:08.718 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 140.406919176, 2024-09-20T22:49:08.718 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.030150626999999999, 2024-09-20T22:49:08.718 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:49:08.718 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:49:08.718 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:49:08.718 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:49:08.718 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:49:08.718 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:49:08.718 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 3136 2024-09-20T22:49:08.718 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.719 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:49:08.719 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.719 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:49:08.719 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.210350+0000", 2024-09-20T22:49:08.719 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:49:08.719 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.719 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.719 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:49:08.719 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.210350+0000", 2024-09-20T22:49:08.719 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:49:08.719 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.719 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.719 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:49:08.719 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.210352+0000", 2024-09-20T22:49:08.720 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.889e-06 2024-09-20T22:49:08.720 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.720 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.720 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:49:08.720 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.210360+0000", 2024-09-20T22:49:08.720 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.8890000000000002e-06 2024-09-20T22:49:08.720 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.720 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.720 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:49:08.720 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.210372+0000", 2024-09-20T22:49:08.720 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2662e-05 2024-09-20T22:49:08.720 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.720 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.720 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:49:08.721 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.210381+0000", 2024-09-20T22:49:08.721 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.8589999999999994e-06 2024-09-20T22:49:08.721 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.721 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.721 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:49:08.721 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.210470+0000", 2024-09-20T22:49:08.721 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.9034000000000002e-05 2024-09-20T22:49:08.721 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.721 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.721 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:49:08.721 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.210568+0000", 2024-09-20T22:49:08.721 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.7919999999999998e-05 2024-09-20T22:49:08.721 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.721 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.721 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 2,5", 2024-09-20T22:49:08.722 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.218491+0000", 2024-09-20T22:49:08.722 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0079229490000000003 2024-09-20T22:49:08.722 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.722 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.722 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:49:08.722 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.224053+0000", 2024-09-20T22:49:08.722 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0055624330000000003 2024-09-20T22:49:08.722 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.722 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.722 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:49:08.722 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.227143+0000", 2024-09-20T22:49:08.722 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0030896360000000002 2024-09-20T22:49:08.722 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.722 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.723 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:49:08.723 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.240355+0000", 2024-09-20T22:49:08.723 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013211624999999999 2024-09-20T22:49:08.723 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.723 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.723 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:49:08.723 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.240439+0000", 2024-09-20T22:49:08.723 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.4009000000000002e-05 2024-09-20T22:49:08.723 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.725 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.725 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:49:08.725 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.240500+0000", 2024-09-20T22:49:08.725 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.1722000000000003e-05 2024-09-20T22:49:08.725 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:49:08.725 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:49:08.726 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:49:08.726 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.726 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.726 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:3235 7.1 7:84a866f5:::smithi08477203-27:head [copy-from ver 114] snapc 7e=[7e,7d,7c,7b,7a,79,78,75,74,73,6b] ondisk+write+known_if_redirected+supports_pool_eio e338)", 2024-09-20T22:49:08.726 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:46:58.180892+0000", 2024-09-20T22:49:08.726 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 130.43637695300001, 2024-09-20T22:49:08.726 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.044574093000000002, 2024-09-20T22:49:08.726 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:49:08.726 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:49:08.726 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:49:08.726 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:49:08.726 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:49:08.726 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:49:08.727 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 3235 2024-09-20T22:49:08.727 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.727 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:49:08.727 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.727 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:49:08.727 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.180892+0000", 2024-09-20T22:49:08.727 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:49:08.727 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.727 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.727 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:49:08.727 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.180892+0000", 2024-09-20T22:49:08.727 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:49:08.727 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.727 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.727 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:49:08.728 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.180894+0000", 2024-09-20T22:49:08.728 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.1789999999999998e-06 2024-09-20T22:49:08.728 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.728 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.728 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:49:08.728 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.180915+0000", 2024-09-20T22:49:08.728 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.0859000000000001e-05 2024-09-20T22:49:08.728 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.728 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.728 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:49:08.728 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.180937+0000", 2024-09-20T22:49:08.728 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.2357999999999998e-05 2024-09-20T22:49:08.728 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.728 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.729 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:49:08.729 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.180947+0000", 2024-09-20T22:49:08.729 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.8090000000000004e-06 2024-09-20T22:49:08.729 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.729 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.729 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:49:08.729 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.181042+0000", 2024-09-20T22:49:08.729 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.4327999999999995e-05 2024-09-20T22:49:08.729 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.729 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.729 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op must wait for map", 2024-09-20T22:49:08.729 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.181049+0000", 2024-09-20T22:49:08.729 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.1640000000000004e-06 2024-09-20T22:49:08.729 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.729 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.730 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:49:08.730 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.185933+0000", 2024-09-20T22:49:08.730 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.004884645 2024-09-20T22:49:08.730 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.730 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.730 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:49:08.730 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.186177+0000", 2024-09-20T22:49:08.730 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000243107 2024-09-20T22:49:08.730 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.730 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.730 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 1,6", 2024-09-20T22:49:08.730 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.204218+0000", 2024-09-20T22:49:08.730 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.018041805000000001 2024-09-20T22:49:08.730 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.731 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.731 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:49:08.731 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.218275+0000", 2024-09-20T22:49:08.731 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014056528 2024-09-20T22:49:08.731 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.731 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.731 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:49:08.731 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.224283+0000", 2024-09-20T22:49:08.731 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0060084860000000004 2024-09-20T22:49:08.731 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.731 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.731 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:49:08.731 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.225379+0000", 2024-09-20T22:49:08.731 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010960830000000001 2024-09-20T22:49:08.731 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.732 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.732 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:49:08.732 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.225429+0000", 2024-09-20T22:49:08.732 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.9440000000000001e-05 2024-09-20T22:49:08.732 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.732 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.732 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:49:08.732 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.225466+0000", 2024-09-20T22:49:08.732 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.7302000000000003e-05 2024-09-20T22:49:08.732 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:49:08.732 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:49:08.732 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:49:08.732 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.732 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.732 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:3236 7.1 7:84a866f5:::smithi08477203-27:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e338)", 2024-09-20T22:49:08.733 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:46:58.180973+0000", 2024-09-20T22:49:08.733 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 130.43629633099999, 2024-09-20T22:49:08.733 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.028358540000000002, 2024-09-20T22:49:08.733 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:49:08.733 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:49:08.733 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-20T22:49:08.733 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:49:08.733 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:49:08.733 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:49:08.733 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 3236 2024-09-20T22:49:08.733 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.733 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:49:08.733 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.733 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:49:08.734 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.180973+0000", 2024-09-20T22:49:08.734 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:49:08.734 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.734 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.734 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:49:08.734 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.180973+0000", 2024-09-20T22:49:08.734 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:49:08.734 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.734 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.734 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:49:08.734 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.180973+0000", 2024-09-20T22:49:08.734 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.9100000000000004e-07 2024-09-20T22:49:08.734 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.734 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.735 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:49:08.735 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.180976+0000", 2024-09-20T22:49:08.735 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.118e-06 2024-09-20T22:49:08.735 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.735 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.735 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:49:08.735 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.180984+0000", 2024-09-20T22:49:08.735 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.43e-06 2024-09-20T22:49:08.735 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.735 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.735 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:49:08.735 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.180990+0000", 2024-09-20T22:49:08.735 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.1109999999999996e-06 2024-09-20T22:49:08.735 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.736 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.736 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:49:08.736 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.181102+0000", 2024-09-20T22:49:08.736 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000112193 2024-09-20T22:49:08.736 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.736 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.736 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting_for_map not empty", 2024-09-20T22:49:08.736 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.181107+0000", 2024-09-20T22:49:08.736 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.0490000000000003e-06 2024-09-20T22:49:08.736 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.736 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.736 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:49:08.736 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.186302+0000", 2024-09-20T22:49:08.736 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0051950240000000003 2024-09-20T22:49:08.737 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.737 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.737 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for blocked object", 2024-09-20T22:49:08.737 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.186344+0000", 2024-09-20T22:49:08.737 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.2168e-05 2024-09-20T22:49:08.737 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.737 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.737 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:49:08.737 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.209115+0000", 2024-09-20T22:49:08.737 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.022770291000000002 2024-09-20T22:49:08.737 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.737 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.737 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:49:08.737 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.209217+0000", 2024-09-20T22:49:08.738 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000102192 2024-09-20T22:49:08.738 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.738 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.738 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:49:08.738 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.209331+0000", 2024-09-20T22:49:08.738 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000114373 2024-09-20T22:49:08.738 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:49:08.738 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:49:08.738 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:49:08.738 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.738 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.738 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:4534 7.12 7:4b06cbed:::smithi08477203-49:head [copy-from ver 722] snapc bb=[bb,ba,b9,b8,b7,b5,b4,b2,b1,a7,a6,a5,9e,9c,95,87,86] ondisk+write+known_if_redirected+supports_pool_eio e456)", 2024-09-20T22:49:08.738 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:48:56.230481+0000", 2024-09-20T22:49:08.738 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 12.386788285, 2024-09-20T22:49:08.739 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.028199295999999999, 2024-09-20T22:49:08.739 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:49:08.739 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:49:08.739 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:49:08.739 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:49:08.739 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:49:08.739 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:49:08.739 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 4534 2024-09-20T22:49:08.739 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.739 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:49:08.739 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.739 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:49:08.739 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:56.230481+0000", 2024-09-20T22:49:08.739 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:49:08.739 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.740 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.740 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:49:08.740 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:56.230481+0000", 2024-09-20T22:49:08.740 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:49:08.740 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.740 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.740 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:49:08.740 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:56.230482+0000", 2024-09-20T22:49:08.740 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.674e-06 2024-09-20T22:49:08.740 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.740 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.740 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:49:08.740 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:56.230497+0000", 2024-09-20T22:49:08.740 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.4178e-05 2024-09-20T22:49:08.741 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.741 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.741 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:49:08.741 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:56.230517+0000", 2024-09-20T22:49:08.741 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.0336e-05 2024-09-20T22:49:08.741 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.741 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.741 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:49:08.741 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:56.230527+0000", 2024-09-20T22:49:08.741 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.6039999999999999e-06 2024-09-20T22:49:08.741 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.741 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.741 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:49:08.741 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:56.230631+0000", 2024-09-20T22:49:08.741 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00010445 2024-09-20T22:49:08.742 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.742 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.742 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op must wait for map", 2024-09-20T22:49:08.742 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:56.230639+0000", 2024-09-20T22:49:08.742 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.7759999999999997e-06 2024-09-20T22:49:08.742 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.742 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.742 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:49:08.742 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:56.237931+0000", 2024-09-20T22:49:08.742 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0072925630000000002 2024-09-20T22:49:08.742 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.742 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.742 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:49:08.742 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:56.238030+0000", 2024-09-20T22:49:08.742 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.8592999999999996e-05 2024-09-20T22:49:08.743 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.743 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.743 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 1,2", 2024-09-20T22:49:08.743 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:56.244691+0000", 2024-09-20T22:49:08.743 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0066614680000000003 2024-09-20T22:49:08.743 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.743 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.743 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:49:08.743 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:56.258211+0000", 2024-09-20T22:49:08.743 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013519721 2024-09-20T22:49:08.743 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.743 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.743 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:49:08.743 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:56.258404+0000", 2024-09-20T22:49:08.744 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00019257000000000001 2024-09-20T22:49:08.744 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.744 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.744 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:49:08.744 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:56.258595+0000", 2024-09-20T22:49:08.744 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000191387 2024-09-20T22:49:08.744 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.744 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.744 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:49:08.744 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:56.258635+0000", 2024-09-20T22:49:08.744 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.9918e-05 2024-09-20T22:49:08.744 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:08.744 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:08.744 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:49:08.744 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:56.258680+0000", 2024-09-20T22:49:08.745 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.5058000000000001e-05 2024-09-20T22:49:08.745 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:49:08.745 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:49:08.745 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:49:08.745 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:49:08.745 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:49:08.745 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-20T22:49:08.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:08 smithi154 ceph-mon[33264]: 4.19 scrub starts 2024-09-20T22:49:08.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:08 smithi154 ceph-mon[33264]: 4.19 scrub ok 2024-09-20T22:49:08.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:08 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:49:08.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:08 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:49:08.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:08 smithi154 ceph-mon[33264]: osdmap e468: 8 total, 8 up, 8 in 2024-09-20T22:49:08.777 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:49:08.803 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:08.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:08.804 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:49:08.905 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:49:08.927 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:08.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:08.930 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:49:09.009 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 5, 6, 7, 0, 4] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-20T22:49:09.009 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-20T22:49:09.009 INFO:tasks.thrashosds.thrasher:inject_pause on osd.0 2024-09-20T22:49:09.009 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-20T22:49:09.009 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-20T22:49:09.010 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.0 config set filestore_inject_stall 3 2024-09-20T22:49:09.031 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:49:09.073 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:09.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:09.074 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:49:09.175 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:49:09.201 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:09.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:09.202 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:49:09.289 INFO:tasks.rados.rados.0.smithi084.stdout:1831: done (2 left) 2024-09-20T22:49:09.289 INFO:tasks.rados.rados.0.smithi084.stdout:1832: done (1 left) 2024-09-20T22:49:09.289 INFO:tasks.rados.rados.0.smithi084.stdout:1833: done (0 left) 2024-09-20T22:49:09.290 INFO:tasks.rados.rados.0.smithi084.stdout:1834: read oid 49 snap 186 2024-09-20T22:49:09.290 INFO:tasks.rados.rados.0.smithi084.stdout:1834: expect (ObjNum 470 snap 130 seq_num 470) 2024-09-20T22:49:09.290 INFO:tasks.rados.rados.0.smithi084.stdout:1835: delete oid 6 current snap is 194 2024-09-20T22:49:09.296 INFO:tasks.rados.rados.0.smithi084.stdout:1834: expect (ObjNum 470 snap 130 seq_num 470) 2024-09-20T22:49:09.303 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:49:09.324 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:09.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:09.325 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:49:09.424 INFO:tasks.rados.rados.0.smithi084.stdout:1834: done (1 left) 2024-09-20T22:49:09.425 INFO:tasks.rados.rados.0.smithi084.stdout:1835: done (0 left) 2024-09-20T22:49:09.425 INFO:tasks.rados.rados.0.smithi084.stdout:1836: delete oid 24 current snap is 194 2024-09-20T22:49:09.426 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:49:09.427 INFO:tasks.rados.rados.0.smithi084.stdout:1836: done (0 left) 2024-09-20T22:49:09.427 INFO:tasks.rados.rados.0.smithi084.stdout:1837: rollback oid 5 current snap is 194 2024-09-20T22:49:09.427 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 5 to 189 2024-09-20T22:49:09.427 INFO:tasks.rados.rados.0.smithi084.stdout:1838: write oid 23 current snap is 194 2024-09-20T22:49:09.427 INFO:tasks.rados.rados.0.smithi084.stdout:1838: seq_num 648 ranges {490936=651357,1686214=447225,2861864=29072} 2024-09-20T22:49:09.433 INFO:tasks.rados.rados.0.smithi084.stdout:1838: writing smithi08477203-23 from 490936 to 1142293 tid 1 2024-09-20T22:49:09.439 INFO:tasks.rados.rados.0.smithi084.stdout:1838: writing smithi08477203-23 from 1686214 to 2133439 tid 2 2024-09-20T22:49:09.443 INFO:tasks.rados.rados.0.smithi084.stdout:1838: writing smithi08477203-23 from 2861864 to 2890936 tid 3 2024-09-20T22:49:09.443 INFO:tasks.rados.rados.0.smithi084.stdout:1839: read oid 45 snap -1 2024-09-20T22:49:09.443 INFO:tasks.rados.rados.0.smithi084.stdout:1839: expect deleted 2024-09-20T22:49:09.443 INFO:tasks.rados.rados.0.smithi084.stdout:1840: write oid 3 current snap is 194 2024-09-20T22:49:09.443 INFO:tasks.rados.rados.0.smithi084.stdout:1837: finishing rollback tid 0 to smithi08477203-5 2024-09-20T22:49:09.444 INFO:tasks.rados.rados.0.smithi084.stdout:1837: finishing rollback tid 1 to smithi08477203-5 2024-09-20T22:49:09.444 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 5 v 701 (ObjNum 129 snap 0 seq_num 2091849920) dirty dne 2024-09-20T22:49:09.444 INFO:tasks.rados.rados.0.smithi084.stdout:1838: finishing write tid 1 to smithi08477203-23 2024-09-20T22:49:09.444 INFO:tasks.rados.rados.0.smithi084.stdout:1838: finishing write tid 2 to smithi08477203-23 2024-09-20T22:49:09.444 INFO:tasks.rados.rados.0.smithi084.stdout:1840: seq_num 649 ranges {507743=726606,1744696=590883,2772934=134809} 2024-09-20T22:49:09.449 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:09.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:09.450 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:49:09.450 INFO:tasks.rados.rados.0.smithi084.stdout:1840: writing smithi08477203-3 from 507743 to 1234349 tid 1 2024-09-20T22:49:09.456 INFO:tasks.rados.rados.0.smithi084.stdout:1840: writing smithi08477203-3 from 1744696 to 2335579 tid 2 2024-09-20T22:49:09.459 INFO:tasks.rados.rados.0.smithi084.stdout:1840: writing smithi08477203-3 from 2772934 to 2907743 tid 3 2024-09-20T22:49:09.459 INFO:tasks.rados.rados.0.smithi084.stdout:1837: done (3 left) 2024-09-20T22:49:09.460 INFO:tasks.rados.rados.0.smithi084.stdout:1841: delete oid 37 current snap is 194 2024-09-20T22:49:09.460 INFO:tasks.rados.rados.0.smithi084.stdout:1838: finishing write tid 3 to smithi08477203-23 2024-09-20T22:49:09.460 INFO:tasks.rados.rados.0.smithi084.stdout:1838: finishing write tid 4 to smithi08477203-23 2024-09-20T22:49:09.460 INFO:tasks.rados.rados.0.smithi084.stdout:1838: finishing write tid 5 to smithi08477203-23 2024-09-20T22:49:09.460 INFO:tasks.rados.rados.0.smithi084.stdout:1838: oid 23 updating version 0 to 712 2024-09-20T22:49:09.460 INFO:tasks.rados.rados.0.smithi084.stdout:1838: oid 23 updating version 712 to 713 2024-09-20T22:49:09.460 INFO:tasks.rados.rados.0.smithi084.stdout:1838: oid 23 version 713 is already newer than 711 2024-09-20T22:49:09.460 INFO:tasks.rados.rados.0.smithi084.stdout:1838: oid 23 updating version 713 to 714 2024-09-20T22:49:09.460 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 23 v 714 (ObjNum 648 snap 194 seq_num 648) dirty exists 2024-09-20T22:49:09.460 INFO:tasks.rados.rados.0.smithi084.stdout:1838: left oid 23 (ObjNum 648 snap 194 seq_num 648) 2024-09-20T22:49:09.460 INFO:tasks.rados.rados.0.smithi084.stdout:1840: finishing write tid 1 to smithi08477203-3 2024-09-20T22:49:09.460 INFO:tasks.rados.rados.0.smithi084.stdout:1840: finishing write tid 2 to smithi08477203-3 2024-09-20T22:49:09.461 INFO:tasks.rados.rados.0.smithi084.stdout:1840: finishing write tid 3 to smithi08477203-3 2024-09-20T22:49:09.461 INFO:tasks.rados.rados.0.smithi084.stdout:1838: done (3 left) 2024-09-20T22:49:09.461 INFO:tasks.rados.rados.0.smithi084.stdout:1839: done (2 left) 2024-09-20T22:49:09.461 INFO:tasks.rados.rados.0.smithi084.stdout:1841: done (1 left) 2024-09-20T22:49:09.461 INFO:tasks.rados.rados.0.smithi084.stdout:1842: write oid 13 current snap is 194 2024-09-20T22:49:09.461 INFO:tasks.rados.rados.0.smithi084.stdout:1842: seq_num 651 ranges {541357=477104,1745307=594552} 2024-09-20T22:49:09.466 INFO:tasks.rados.rados.0.smithi084.stdout:1842: writing smithi08477203-13 from 541357 to 1018461 tid 1 2024-09-20T22:49:09.474 INFO:tasks.rados.rados.0.smithi084.stdout:1842: writing smithi08477203-13 from 1745307 to 2339859 tid 2 2024-09-20T22:49:09.474 INFO:tasks.rados.rados.0.smithi084.stdout:1843: delete oid 19 current snap is 194 2024-09-20T22:49:09.474 INFO:tasks.rados.rados.0.smithi084.stdout:1840: finishing write tid 4 to smithi08477203-3 2024-09-20T22:49:09.474 INFO:tasks.rados.rados.0.smithi084.stdout:1840: finishing write tid 5 to smithi08477203-3 2024-09-20T22:49:09.474 INFO:tasks.rados.rados.0.smithi084.stdout:1840: oid 3 updating version 0 to 762 2024-09-20T22:49:09.474 INFO:tasks.rados.rados.0.smithi084.stdout:1840: oid 3 version 762 is already newer than 760 2024-09-20T22:49:09.474 INFO:tasks.rados.rados.0.smithi084.stdout:1840: oid 3 version 762 is already newer than 759 2024-09-20T22:49:09.474 INFO:tasks.rados.rados.0.smithi084.stdout:1840: oid 3 version 762 is already newer than 761 2024-09-20T22:49:09.475 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 3 v 762 (ObjNum 649 snap 194 seq_num 649) dirty exists 2024-09-20T22:49:09.475 INFO:tasks.rados.rados.0.smithi084.stdout:1840: left oid 3 (ObjNum 649 snap 194 seq_num 649) 2024-09-20T22:49:09.475 INFO:tasks.rados.rados.0.smithi084.stdout:1842: finishing write tid 1 to smithi08477203-13 2024-09-20T22:49:09.478 INFO:tasks.rados.rados.0.smithi084.stdout:1842: finishing write tid 2 to smithi08477203-13 2024-09-20T22:49:09.478 INFO:tasks.rados.rados.0.smithi084.stdout:1842: finishing write tid 3 to smithi08477203-13 2024-09-20T22:49:09.478 INFO:tasks.rados.rados.0.smithi084.stdout:1842: finishing write tid 4 to smithi08477203-13 2024-09-20T22:49:09.478 INFO:tasks.rados.rados.0.smithi084.stdout:1842: oid 13 updating version 0 to 1192 2024-09-20T22:49:09.478 INFO:tasks.rados.rados.0.smithi084.stdout:1842: oid 13 updating version 1192 to 1193 2024-09-20T22:49:09.478 INFO:tasks.rados.rados.0.smithi084.stdout:1842: oid 13 version 1193 is already newer than 1191 2024-09-20T22:49:09.478 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 13 v 1193 (ObjNum 651 snap 194 seq_num 651) dirty exists 2024-09-20T22:49:09.478 INFO:tasks.rados.rados.0.smithi084.stdout:1842: left oid 13 (ObjNum 651 snap 194 seq_num 651) 2024-09-20T22:49:09.479 INFO:tasks.rados.rados.0.smithi084.stdout:1840: done (2 left) 2024-09-20T22:49:09.479 INFO:tasks.rados.rados.0.smithi084.stdout:1842: done (1 left) 2024-09-20T22:49:09.479 INFO:tasks.rados.rados.0.smithi084.stdout:1843: done (0 left) 2024-09-20T22:49:09.479 INFO:tasks.rados.rados.0.smithi084.stdout:1844: snap_create 2024-09-20T22:49:09.551 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:49:09.563 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:09 smithi084 ceph-mon[40024]: 6.d scrub starts 2024-09-20T22:49:09.563 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:09 smithi084 ceph-mon[40024]: 6.d scrub ok 2024-09-20T22:49:09.563 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:09 smithi084 ceph-mon[40024]: pgmap v603: 161 pgs: 1 clean+premerge+peered, 2 active+clean+snaptrim, 158 active+clean; 362 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail 2024-09-20T22:49:09.563 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:09 smithi084 ceph-mon[40024]: 5.c scrub starts 2024-09-20T22:49:09.563 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:09 smithi084 ceph-mon[40024]: 5.c scrub ok 2024-09-20T22:49:09.563 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:09 smithi084 ceph-mon[29774]: 6.d scrub starts 2024-09-20T22:49:09.563 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:09 smithi084 ceph-mon[29774]: 6.d scrub ok 2024-09-20T22:49:09.563 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:09 smithi084 ceph-mon[29774]: pgmap v603: 161 pgs: 1 clean+premerge+peered, 2 active+clean+snaptrim, 158 active+clean; 362 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail 2024-09-20T22:49:09.564 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:09 smithi084 ceph-mon[29774]: 5.c scrub starts 2024-09-20T22:49:09.564 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:09 smithi084 ceph-mon[29774]: 5.c scrub ok 2024-09-20T22:49:09.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:09.575 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:09.575 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:49:09.591 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-20T22:49:09.638 INFO:journalctl@ceph.iscsi.iscsi.a.smithi154.stdout:Sep 20 22:49:09 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-iscsi-iscsi-a[59730]: debug there is no tcmu-runner data available 2024-09-20T22:49:09.638 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:09 smithi154 ceph-mon[33264]: 6.d scrub starts 2024-09-20T22:49:09.639 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:09 smithi154 ceph-mon[33264]: 6.d scrub ok 2024-09-20T22:49:09.639 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:09 smithi154 ceph-mon[33264]: pgmap v603: 161 pgs: 1 clean+premerge+peered, 2 active+clean+snaptrim, 158 active+clean; 362 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail 2024-09-20T22:49:09.639 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:09 smithi154 ceph-mon[33264]: 5.c scrub starts 2024-09-20T22:49:09.639 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:09 smithi154 ceph-mon[33264]: 5.c scrub ok 2024-09-20T22:49:09.674 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd set noscrub 2024-09-20T22:49:09.675 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:49:09.688 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:49:09.697 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:09.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:09.698 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:49:09.799 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:49:09.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:09.820 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:09.820 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:49:09.864 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:49:09.921 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:49:09.943 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:09.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:09.944 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:49:10.045 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:49:10.067 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:10.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:10.068 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:49:10.114 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-20T22:49:10.169 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:49:10.193 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:10.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:10.194 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:49:10.238 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-20T22:49:10.239 INFO:teuthology.orchestra.run.smithi084.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-20T22:49:10.239 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-20T22:49:10.294 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:49:10.299 INFO:tasks.rados.rados.0.smithi084.stdout:1844: done (0 left) 2024-09-20T22:49:10.299 INFO:tasks.rados.rados.0.smithi084.stdout:1845: write (excl) oid 5 current snap is 195 2024-09-20T22:49:10.299 INFO:tasks.rados.rados.0.smithi084.stdout:1845: seq_num 653 ranges {574971=627602,1862271=598221} 2024-09-20T22:49:10.306 INFO:tasks.rados.rados.0.smithi084.stdout:1845: writing smithi08477203-5 from 574971 to 1202573 tid 1 2024-09-20T22:49:10.313 INFO:tasks.rados.rados.0.smithi084.stdout:1845: writing smithi08477203-5 from 1862271 to 2460492 tid 2 2024-09-20T22:49:10.314 INFO:tasks.rados.rados.0.smithi084.stdout:1846: read oid 43 snap -1 2024-09-20T22:49:10.314 INFO:tasks.rados.rados.0.smithi084.stdout:1846: expect deleted 2024-09-20T22:49:10.314 INFO:tasks.rados.rados.0.smithi084.stdout:1847: copy_from oid 21 from oid 24 current snap is 195 2024-09-20T22:49:10.314 INFO:tasks.rados.rados.0.smithi084.stdout:1848: snap_create 2024-09-20T22:49:10.314 INFO:tasks.rados.rados.0.smithi084.stdout:1845: finishing write tid 1 to smithi08477203-5 2024-09-20T22:49:10.317 INFO:tasks.rados.rados.0.smithi084.stdout:1847: finishing copy_from to smithi08477203-21 2024-09-20T22:49:10.317 INFO:tasks.rados.rados.0.smithi084.stdout:1847: got expected ENOENT (src dne) 2024-09-20T22:49:10.317 INFO:tasks.rados.rados.0.smithi084.stdout:1845: finishing write tid 2 to smithi08477203-5 2024-09-20T22:49:10.317 INFO:tasks.rados.rados.0.smithi084.stdout:1847: finishing copy_from racing read to smithi08477203-21 2024-09-20T22:49:10.317 INFO:tasks.rados.rados.0.smithi084.stdout:1845: finishing write tid 3 to smithi08477203-5 2024-09-20T22:49:10.318 INFO:tasks.rados.rados.0.smithi084.stdout:1845: finishing write tid 4 to smithi08477203-5 2024-09-20T22:49:10.318 INFO:tasks.rados.rados.0.smithi084.stdout:1845: oid 5 updating version 0 to 703 2024-09-20T22:49:10.318 INFO:tasks.rados.rados.0.smithi084.stdout:1845: oid 5 updating version 703 to 704 2024-09-20T22:49:10.318 INFO:tasks.rados.rados.0.smithi084.stdout:1845: oid 5 version 704 is already newer than 702 2024-09-20T22:49:10.318 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 5 v 704 (ObjNum 653 snap 195 seq_num 653) dirty exists 2024-09-20T22:49:10.318 INFO:tasks.rados.rados.0.smithi084.stdout:1845: left oid 5 (ObjNum 653 snap 195 seq_num 653) 2024-09-20T22:49:10.322 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:10.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:10.322 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:49:10.420 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-20T22:49:10.420 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [], 2024-09-20T22:49:10.420 INFO:teuthology.orchestra.run.smithi154.stdout: "num_ops": 0 2024-09-20T22:49:10.420 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-20T22:49:10.423 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:49:10.447 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:10.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:10.448 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:49:10.549 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:49:10.570 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:10.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:10.571 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:49:10.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:10 smithi084 ceph-mon[29774]: 6.18 deep-scrub starts 2024-09-20T22:49:10.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:10 smithi084 ceph-mon[29774]: 6.18 deep-scrub ok 2024-09-20T22:49:10.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:10 smithi084 ceph-mon[29774]: osdmap e469: 8 total, 8 up, 8 in 2024-09-20T22:49:10.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:10 smithi084 ceph-mon[29774]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:49:10.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:10 smithi084 ceph-mon[29774]: 4.1f scrub starts 2024-09-20T22:49:10.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:10 smithi084 ceph-mon[29774]: 4.1f scrub ok 2024-09-20T22:49:10.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:10 smithi084 ceph-mon[29774]: 6.4 scrub starts 2024-09-20T22:49:10.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:10 smithi084 ceph-mon[29774]: 6.4 scrub ok 2024-09-20T22:49:10.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:10 smithi084 ceph-mon[40024]: 6.18 deep-scrub starts 2024-09-20T22:49:10.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:10 smithi084 ceph-mon[40024]: 6.18 deep-scrub ok 2024-09-20T22:49:10.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:10 smithi084 ceph-mon[40024]: osdmap e469: 8 total, 8 up, 8 in 2024-09-20T22:49:10.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:10 smithi084 ceph-mon[40024]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:49:10.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:10 smithi084 ceph-mon[40024]: 4.1f scrub starts 2024-09-20T22:49:10.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:10 smithi084 ceph-mon[40024]: 4.1f scrub ok 2024-09-20T22:49:10.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:10 smithi084 ceph-mon[40024]: 6.4 scrub starts 2024-09-20T22:49:10.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:10 smithi084 ceph-mon[40024]: 6.4 scrub ok 2024-09-20T22:49:10.671 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:49:10.692 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:10.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:10.693 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:49:10.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:10 smithi154 ceph-mon[33264]: 6.18 deep-scrub starts 2024-09-20T22:49:10.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:10 smithi154 ceph-mon[33264]: 6.18 deep-scrub ok 2024-09-20T22:49:10.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:10 smithi154 ceph-mon[33264]: osdmap e469: 8 total, 8 up, 8 in 2024-09-20T22:49:10.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:10 smithi154 ceph-mon[33264]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:49:10.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:10 smithi154 ceph-mon[33264]: 4.1f scrub starts 2024-09-20T22:49:10.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:10 smithi154 ceph-mon[33264]: 4.1f scrub ok 2024-09-20T22:49:10.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:10 smithi154 ceph-mon[33264]: 6.4 scrub starts 2024-09-20T22:49:10.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:10 smithi154 ceph-mon[33264]: 6.4 scrub ok 2024-09-20T22:49:10.794 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:49:10.815 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:10.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:10.816 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:49:10.916 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:49:10.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:10.938 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:10.939 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:49:11.039 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:49:11.043 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:49:11.060 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:11.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:11.061 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:49:11.079 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:49:11.162 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:49:11.183 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:11.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:11.184 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:49:11.285 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:49:11.301 INFO:tasks.rados.rados.0.smithi084.stdout:1845: done (3 left) 2024-09-20T22:49:11.301 INFO:tasks.rados.rados.0.smithi084.stdout:1846: done (2 left) 2024-09-20T22:49:11.301 INFO:tasks.rados.rados.0.smithi084.stdout:1847: done (1 left) 2024-09-20T22:49:11.302 INFO:tasks.rados.rados.0.smithi084.stdout:1848: done (0 left) 2024-09-20T22:49:11.302 INFO:tasks.rados.rados.0.smithi084.stdout:1849: snap_create 2024-09-20T22:49:11.307 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:11.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:11.308 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:49:11.367 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:11 smithi154 ceph-mon[33264]: 4.15 scrub starts 2024-09-20T22:49:11.367 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:11 smithi154 ceph-mon[33264]: 4.15 scrub ok 2024-09-20T22:49:11.367 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:11 smithi154 ceph-mon[33264]: pgmap v606: 161 pgs: 1 active+clean+snaptrim_wait, 1 active+undersized+remapped, 5 active+clean+snaptrim, 154 active+clean; 394 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 24 MiB/s rd, 5.5 MiB/s wr, 39 op/s; 6.1 MiB/s, 7 objects/s recovering 2024-09-20T22:49:11.367 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:11 smithi154 ceph-mon[33264]: 3.1e scrub starts 2024-09-20T22:49:11.367 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:11 smithi154 ceph-mon[33264]: 3.1e scrub ok 2024-09-20T22:49:11.367 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:11 smithi154 ceph-mon[33264]: osdmap e470: 8 total, 8 up, 8 in 2024-09-20T22:49:11.367 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:11 smithi154 ceph-mon[33264]: 3.14 scrub starts 2024-09-20T22:49:11.367 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:11 smithi154 ceph-mon[33264]: 3.14 scrub ok 2024-09-20T22:49:11.395 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.4 dump_blocked_ops 2024-09-20T22:49:11.409 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:49:11.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:11.431 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:11.431 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:49:11.532 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:49:11.542 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:11 smithi084 ceph-mon[40024]: 4.15 scrub starts 2024-09-20T22:49:11.542 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:11 smithi084 ceph-mon[40024]: 4.15 scrub ok 2024-09-20T22:49:11.543 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:11 smithi084 ceph-mon[40024]: pgmap v606: 161 pgs: 1 active+clean+snaptrim_wait, 1 active+undersized+remapped, 5 active+clean+snaptrim, 154 active+clean; 394 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 24 MiB/s rd, 5.5 MiB/s wr, 39 op/s; 6.1 MiB/s, 7 objects/s recovering 2024-09-20T22:49:11.543 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:11 smithi084 ceph-mon[40024]: 3.1e scrub starts 2024-09-20T22:49:11.543 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:11 smithi084 ceph-mon[40024]: 3.1e scrub ok 2024-09-20T22:49:11.543 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:11 smithi084 ceph-mon[40024]: osdmap e470: 8 total, 8 up, 8 in 2024-09-20T22:49:11.543 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:11 smithi084 ceph-mon[40024]: 3.14 scrub starts 2024-09-20T22:49:11.543 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:11 smithi084 ceph-mon[40024]: 3.14 scrub ok 2024-09-20T22:49:11.553 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:11.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:11.554 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:49:11.561 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:11 smithi084 ceph-mon[29774]: 4.15 scrub starts 2024-09-20T22:49:11.561 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:11 smithi084 ceph-mon[29774]: 4.15 scrub ok 2024-09-20T22:49:11.561 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:11 smithi084 ceph-mon[29774]: pgmap v606: 161 pgs: 1 active+clean+snaptrim_wait, 1 active+undersized+remapped, 5 active+clean+snaptrim, 154 active+clean; 394 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 24 MiB/s rd, 5.5 MiB/s wr, 39 op/s; 6.1 MiB/s, 7 objects/s recovering 2024-09-20T22:49:11.561 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:11 smithi084 ceph-mon[29774]: 3.1e scrub starts 2024-09-20T22:49:11.561 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:11 smithi084 ceph-mon[29774]: 3.1e scrub ok 2024-09-20T22:49:11.562 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:11 smithi084 ceph-mon[29774]: osdmap e470: 8 total, 8 up, 8 in 2024-09-20T22:49:11.562 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:11 smithi084 ceph-mon[29774]: 3.14 scrub starts 2024-09-20T22:49:11.562 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:11 smithi084 ceph-mon[29774]: 3.14 scrub ok 2024-09-20T22:49:11.643 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:49:11.655 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:49:11.677 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:11.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:11.678 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:49:11.733 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:49:11.734 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:49:11.751 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:49:11.751 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:49:11.767 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:49:11.767 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:49:11.779 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:49:11.783 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-20T22:49:11.783 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T22:49:11.799 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:49:11.800 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:49:11.801 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:11.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:11.802 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:49:11.816 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:49:11.817 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:49:11.833 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:49:11.833 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:49:11.850 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:49:11.850 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:49:11.902 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:49:11.923 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:11.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:11.924 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:49:12.025 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:49:12.047 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:12.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:12.048 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:49:12.145 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-20T22:49:12.145 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:49:12.149 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:49:12.170 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:12.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:12.171 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:49:12.272 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:49:12.293 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:12.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:12.294 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:49:12.300 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is set 2024-09-20T22:49:12.305 INFO:tasks.rados.rados.0.smithi084.stdout:1849: done (0 left) 2024-09-20T22:49:12.305 INFO:tasks.rados.rados.0.smithi084.stdout:1850: snap_remove snap 190 2024-09-20T22:49:12.395 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:49:12.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:12.416 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:12.417 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:49:12.517 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:49:12.539 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:12.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:12.540 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:49:12.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:12 smithi084 ceph-mon[29774]: purged_snaps scrub starts 2024-09-20T22:49:12.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:12 smithi084 ceph-mon[29774]: purged_snaps scrub ok 2024-09-20T22:49:12.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:12 smithi084 ceph-mon[29774]: 4.11 deep-scrub starts 2024-09-20T22:49:12.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:12 smithi084 ceph-mon[29774]: 4.11 deep-scrub ok 2024-09-20T22:49:12.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:12 smithi084 ceph-mon[29774]: osdmap e471: 8 total, 8 up, 8 in 2024-09-20T22:49:12.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:12 smithi084 ceph-mon[29774]: 5.10 scrub starts 2024-09-20T22:49:12.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:12 smithi084 ceph-mon[29774]: 5.10 scrub ok 2024-09-20T22:49:12.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:12 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/2402560729' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:49:12.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:12 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/2184737694' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-20T22:49:12.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:12 smithi084 ceph-mon[40024]: purged_snaps scrub starts 2024-09-20T22:49:12.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:12 smithi084 ceph-mon[40024]: purged_snaps scrub ok 2024-09-20T22:49:12.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:12 smithi084 ceph-mon[40024]: 4.11 deep-scrub starts 2024-09-20T22:49:12.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:12 smithi084 ceph-mon[40024]: 4.11 deep-scrub ok 2024-09-20T22:49:12.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:12 smithi084 ceph-mon[40024]: osdmap e471: 8 total, 8 up, 8 in 2024-09-20T22:49:12.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:12 smithi084 ceph-mon[40024]: 5.10 scrub starts 2024-09-20T22:49:12.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:12 smithi084 ceph-mon[40024]: 5.10 scrub ok 2024-09-20T22:49:12.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:12 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/2402560729' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:49:12.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:12 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/2184737694' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-20T22:49:12.640 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:49:12.662 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:12.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:12.663 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:49:12.683 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:12 smithi154 ceph-mon[33264]: purged_snaps scrub starts 2024-09-20T22:49:12.683 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:12 smithi154 ceph-mon[33264]: purged_snaps scrub ok 2024-09-20T22:49:12.684 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:12 smithi154 ceph-mon[33264]: 4.11 deep-scrub starts 2024-09-20T22:49:12.684 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:12 smithi154 ceph-mon[33264]: 4.11 deep-scrub ok 2024-09-20T22:49:12.684 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:12 smithi154 ceph-mon[33264]: osdmap e471: 8 total, 8 up, 8 in 2024-09-20T22:49:12.684 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:12 smithi154 ceph-mon[33264]: 5.10 scrub starts 2024-09-20T22:49:12.684 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:12 smithi154 ceph-mon[33264]: 5.10 scrub ok 2024-09-20T22:49:12.684 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:12 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/2402560729' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:49:12.684 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:12 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/2184737694' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-20T22:49:12.711 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.4 dump_historic_ops 2024-09-20T22:49:12.764 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:49:12.785 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:12.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:12.786 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:49:12.887 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:49:12.908 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:12.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:12.909 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:49:12.960 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:49:13.009 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:49:13.031 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:13.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:13.032 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:49:13.133 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:49:13.154 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:13.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:13.155 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:49:13.220 INFO:tasks.rados.rados.0.smithi084.stdout:1850: done (0 left) 2024-09-20T22:49:13.220 INFO:tasks.rados.rados.0.smithi084.stdout:1851: snap_remove snap 186 2024-09-20T22:49:13.256 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:49:13.277 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:13.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:13.278 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:49:13.354 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:13 smithi154 ceph-mon[33264]: pgmap v609: 161 pgs: 1 active+clean+snaptrim_wait, 1 active+undersized+remapped, 5 active+clean+snaptrim, 154 active+clean; 394 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 24 MiB/s rd, 5.5 MiB/s wr, 39 op/s; 6.1 MiB/s, 7 objects/s recovering 2024-09-20T22:49:13.354 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:13 smithi154 ceph-mon[33264]: 4.16 scrub starts 2024-09-20T22:49:13.354 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:13 smithi154 ceph-mon[33264]: 4.16 scrub ok 2024-09-20T22:49:13.354 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:13 smithi154 ceph-mon[33264]: 3.8 deep-scrub starts 2024-09-20T22:49:13.354 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:13 smithi154 ceph-mon[33264]: 3.8 deep-scrub ok 2024-09-20T22:49:13.354 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:13 smithi154 ceph-mon[33264]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:49:13.354 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:13 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/2184737694' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T22:49:13.354 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:13 smithi154 ceph-mon[33264]: osdmap e472: 8 total, 8 up, 8 in 2024-09-20T22:49:13.355 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:13 smithi154 ceph-mon[33264]: 3.d scrub starts 2024-09-20T22:49:13.355 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:13 smithi154 ceph-mon[33264]: 3.d scrub ok 2024-09-20T22:49:13.355 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:13 smithi154 ceph-mon[33264]: osdmap e473: 8 total, 8 up, 8 in 2024-09-20T22:49:13.379 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:49:13.400 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:13.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:13.401 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:49:13.462 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-20T22:49:13.462 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:49:13.502 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:49:13.523 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:13.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:13.524 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:49:13.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:13 smithi084 ceph-mon[40024]: pgmap v609: 161 pgs: 1 active+clean+snaptrim_wait, 1 active+undersized+remapped, 5 active+clean+snaptrim, 154 active+clean; 394 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 24 MiB/s rd, 5.5 MiB/s wr, 39 op/s; 6.1 MiB/s, 7 objects/s recovering 2024-09-20T22:49:13.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:13 smithi084 ceph-mon[40024]: 4.16 scrub starts 2024-09-20T22:49:13.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:13 smithi084 ceph-mon[40024]: 4.16 scrub ok 2024-09-20T22:49:13.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:13 smithi084 ceph-mon[40024]: 3.8 deep-scrub starts 2024-09-20T22:49:13.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:13 smithi084 ceph-mon[40024]: 3.8 deep-scrub ok 2024-09-20T22:49:13.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:13 smithi084 ceph-mon[40024]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:49:13.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:13 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/2184737694' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T22:49:13.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:13 smithi084 ceph-mon[40024]: osdmap e472: 8 total, 8 up, 8 in 2024-09-20T22:49:13.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:13 smithi084 ceph-mon[40024]: 3.d scrub starts 2024-09-20T22:49:13.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:13 smithi084 ceph-mon[40024]: 3.d scrub ok 2024-09-20T22:49:13.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:13 smithi084 ceph-mon[40024]: osdmap e473: 8 total, 8 up, 8 in 2024-09-20T22:49:13.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:13 smithi084 ceph-mon[29774]: pgmap v609: 161 pgs: 1 active+clean+snaptrim_wait, 1 active+undersized+remapped, 5 active+clean+snaptrim, 154 active+clean; 394 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 24 MiB/s rd, 5.5 MiB/s wr, 39 op/s; 6.1 MiB/s, 7 objects/s recovering 2024-09-20T22:49:13.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:13 smithi084 ceph-mon[29774]: 4.16 scrub starts 2024-09-20T22:49:13.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:13 smithi084 ceph-mon[29774]: 4.16 scrub ok 2024-09-20T22:49:13.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:13 smithi084 ceph-mon[29774]: 3.8 deep-scrub starts 2024-09-20T22:49:13.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:13 smithi084 ceph-mon[29774]: 3.8 deep-scrub ok 2024-09-20T22:49:13.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:13 smithi084 ceph-mon[29774]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:49:13.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:13 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/2184737694' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T22:49:13.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:13 smithi084 ceph-mon[29774]: osdmap e472: 8 total, 8 up, 8 in 2024-09-20T22:49:13.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:13 smithi084 ceph-mon[29774]: 3.d scrub starts 2024-09-20T22:49:13.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:13 smithi084 ceph-mon[29774]: 3.d scrub ok 2024-09-20T22:49:13.617 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:13 smithi084 ceph-mon[29774]: osdmap e473: 8 total, 8 up, 8 in 2024-09-20T22:49:13.625 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:49:13.646 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:13.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:13.647 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:49:13.747 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:49:13.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:13.771 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:13.771 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:49:13.872 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:49:13.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:13.894 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:13.894 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:49:13.994 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:49:14.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:14.017 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:14.017 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:49:14.059 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-20T22:49:14.118 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:49:14.140 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:14.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:14.141 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:49:14.228 INFO:tasks.rados.rados.0.smithi084.stdout:1851: done (0 left) 2024-09-20T22:49:14.228 INFO:tasks.rados.rados.0.smithi084.stdout:1852: read oid 18 snap -1 2024-09-20T22:49:14.228 INFO:tasks.rados.rados.0.smithi084.stdout:1852: expect (ObjNum 568 snap 171 seq_num 568) 2024-09-20T22:49:14.228 INFO:tasks.rados.rados.0.smithi084.stdout:1853: write oid 40 current snap is 197 2024-09-20T22:49:14.229 INFO:tasks.rados.rados.0.smithi084.stdout:1853: seq_num 654 ranges {591778=419204,1637106=741879,2910049=81729} 2024-09-20T22:49:14.229 INFO:tasks.rados.rados.0.smithi084.stdout:1853: writing smithi08477203-40 from 591778 to 1010982 tid 1 2024-09-20T22:49:14.237 INFO:tasks.rados.rados.0.smithi084.stdout:1853: writing smithi08477203-40 from 1637106 to 2378985 tid 2 2024-09-20T22:49:14.241 INFO:tasks.rados.rados.0.smithi084.stdout:1853: writing smithi08477203-40 from 2910049 to 2991778 tid 3 2024-09-20T22:49:14.241 INFO:tasks.rados.rados.0.smithi084.stdout:1854: write oid 38 current snap is 197 2024-09-20T22:49:14.241 INFO:tasks.rados.rados.0.smithi084.stdout:1853: finishing write tid 1 to smithi08477203-40 2024-09-20T22:49:14.241 INFO:tasks.rados.rados.0.smithi084.stdout:1854: seq_num 655 ranges {608585=494453,1579235=601890,2821119=187466} 2024-09-20T22:49:14.241 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:49:14.247 INFO:tasks.rados.rados.0.smithi084.stdout:1854: writing smithi08477203-38 from 608585 to 1103038 tid 1 2024-09-20T22:49:14.253 INFO:tasks.rados.rados.0.smithi084.stdout:1854: writing smithi08477203-38 from 1579235 to 2181125 tid 2 2024-09-20T22:49:14.257 INFO:tasks.rados.rados.0.smithi084.stdout:1854: writing smithi08477203-38 from 2821119 to 3008585 tid 3 2024-09-20T22:49:14.258 INFO:tasks.rados.rados.0.smithi084.stdout:1855: snap_create 2024-09-20T22:49:14.258 INFO:tasks.rados.rados.0.smithi084.stdout:1852: expect (ObjNum 568 snap 171 seq_num 568) 2024-09-20T22:49:14.263 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:14.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:14.264 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:49:14.310 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:49:14.365 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:49:14.387 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:14.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:14.388 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:49:14.463 INFO:tasks.rados.rados.0.smithi084.stdout:1853: finishing write tid 2 to smithi08477203-40 2024-09-20T22:49:14.463 INFO:tasks.rados.rados.0.smithi084.stdout:1853: finishing write tid 3 to smithi08477203-40 2024-09-20T22:49:14.463 INFO:tasks.rados.rados.0.smithi084.stdout:1853: finishing write tid 4 to smithi08477203-40 2024-09-20T22:49:14.463 INFO:tasks.rados.rados.0.smithi084.stdout:1853: finishing write tid 5 to smithi08477203-40 2024-09-20T22:49:14.463 INFO:tasks.rados.rados.0.smithi084.stdout:1853: oid 40 updating version 0 to 718 2024-09-20T22:49:14.463 INFO:tasks.rados.rados.0.smithi084.stdout:1853: oid 40 version 718 is already newer than 715 2024-09-20T22:49:14.464 INFO:tasks.rados.rados.0.smithi084.stdout:1853: oid 40 version 718 is already newer than 717 2024-09-20T22:49:14.464 INFO:tasks.rados.rados.0.smithi084.stdout:1853: oid 40 version 718 is already newer than 716 2024-09-20T22:49:14.464 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 40 v 718 (ObjNum 654 snap 197 seq_num 654) dirty exists 2024-09-20T22:49:14.464 INFO:tasks.rados.rados.0.smithi084.stdout:1853: left oid 40 (ObjNum 654 snap 197 seq_num 654) 2024-09-20T22:49:14.464 INFO:tasks.rados.rados.0.smithi084.stdout:1854: finishing write tid 1 to smithi08477203-38 2024-09-20T22:49:14.464 INFO:tasks.rados.rados.0.smithi084.stdout:1854: finishing write tid 2 to smithi08477203-38 2024-09-20T22:49:14.464 INFO:tasks.rados.rados.0.smithi084.stdout:1854: finishing write tid 3 to smithi08477203-38 2024-09-20T22:49:14.464 INFO:tasks.rados.rados.0.smithi084.stdout:1854: finishing write tid 4 to smithi08477203-38 2024-09-20T22:49:14.464 INFO:tasks.rados.rados.0.smithi084.stdout:1854: finishing write tid 5 to smithi08477203-38 2024-09-20T22:49:14.464 INFO:tasks.rados.rados.0.smithi084.stdout:1854: oid 38 updating version 0 to 555 2024-09-20T22:49:14.464 INFO:tasks.rados.rados.0.smithi084.stdout:1854: oid 38 updating version 555 to 558 2024-09-20T22:49:14.464 INFO:tasks.rados.rados.0.smithi084.stdout:1854: oid 38 version 558 is already newer than 557 2024-09-20T22:49:14.464 INFO:tasks.rados.rados.0.smithi084.stdout:1854: oid 38 version 558 is already newer than 556 2024-09-20T22:49:14.464 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 38 v 558 (ObjNum 655 snap 197 seq_num 655) dirty exists 2024-09-20T22:49:14.464 INFO:tasks.rados.rados.0.smithi084.stdout:1854: left oid 38 (ObjNum 655 snap 197 seq_num 655) 2024-09-20T22:49:14.489 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:49:14.510 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:14.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:14.511 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:49:14.612 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:49:14.614 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:14 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "18"}]: dispatch 2024-09-20T22:49:14.614 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:14 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "18"}]: dispatch 2024-09-20T22:49:14.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:14 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "18"}]': finished 2024-09-20T22:49:14.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:14 smithi084 ceph-mon[29774]: osdmap e474: 8 total, 8 up, 8 in 2024-09-20T22:49:14.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:14 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "18"}]: dispatch 2024-09-20T22:49:14.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:14 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "18"}]: dispatch 2024-09-20T22:49:14.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:14 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "18"}]': finished 2024-09-20T22:49:14.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:14 smithi084 ceph-mon[40024]: osdmap e474: 8 total, 8 up, 8 in 2024-09-20T22:49:14.633 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:14 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "18"}]: dispatch 2024-09-20T22:49:14.633 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:14 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "18"}]: dispatch 2024-09-20T22:49:14.633 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:14 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "18"}]': finished 2024-09-20T22:49:14.633 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:14 smithi154 ceph-mon[33264]: osdmap e474: 8 total, 8 up, 8 in 2024-09-20T22:49:14.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:14.634 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:14.634 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:49:14.669 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-20T22:49:14.741 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:49:14.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:14.764 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:14.764 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:49:14.866 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:49:14.890 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:14.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:14.891 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:49:14.894 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-20T22:49:14.894 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:49:14.923 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:49:14.992 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:49:15.015 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:15.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:15.016 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:49:15.117 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:49:15.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:15.139 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:15.140 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:49:15.229 INFO:tasks.rados.rados.0.smithi084.stdout:1852: done (3 left) 2024-09-20T22:49:15.229 INFO:tasks.rados.rados.0.smithi084.stdout:1853: done (2 left) 2024-09-20T22:49:15.229 INFO:tasks.rados.rados.0.smithi084.stdout:1854: done (1 left) 2024-09-20T22:49:15.229 INFO:tasks.rados.rados.0.smithi084.stdout:1855: done (0 left) 2024-09-20T22:49:15.229 INFO:tasks.rados.rados.0.smithi084.stdout:1856: copy_from oid 46 from oid 43 current snap is 198 2024-09-20T22:49:15.229 INFO:tasks.rados.rados.0.smithi084.stdout:1857: write (excl) oid 28 current snap is 198 2024-09-20T22:49:15.229 INFO:tasks.rados.rados.0.smithi084.stdout:1857: seq_num 656 ranges {625392=569702,1637717=745548,2848542=176850} 2024-09-20T22:49:15.235 INFO:tasks.rados.rados.0.smithi084.stdout:1857: writing smithi08477203-28 from 625392 to 1195094 tid 1 2024-09-20T22:49:15.241 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:49:15.242 INFO:tasks.rados.rados.0.smithi084.stdout:1857: writing smithi08477203-28 from 1637717 to 2383265 tid 2 2024-09-20T22:49:15.246 INFO:tasks.rados.rados.0.smithi084.stdout:1857: writing smithi08477203-28 from 2848542 to 3025392 tid 3 2024-09-20T22:49:15.247 INFO:tasks.rados.rados.0.smithi084.stdout:1858: snap_create 2024-09-20T22:49:15.247 INFO:tasks.rados.rados.0.smithi084.stdout:1856: finishing copy_from to smithi08477203-46 2024-09-20T22:49:15.247 INFO:tasks.rados.rados.0.smithi084.stdout:1856: got expected ENOENT (src dne) 2024-09-20T22:49:15.247 INFO:tasks.rados.rados.0.smithi084.stdout:1856: finishing copy_from racing read to smithi08477203-46 2024-09-20T22:49:15.247 INFO:tasks.rados.rados.0.smithi084.stdout:1857: finishing write tid 1 to smithi08477203-28 2024-09-20T22:49:15.247 INFO:tasks.rados.rados.0.smithi084.stdout:1857: finishing write tid 2 to smithi08477203-28 2024-09-20T22:49:15.248 INFO:tasks.rados.rados.0.smithi084.stdout:1857: finishing write tid 3 to smithi08477203-28 2024-09-20T22:49:15.249 INFO:tasks.rados.rados.0.smithi084.stdout:1857: finishing write tid 4 to smithi08477203-28 2024-09-20T22:49:15.249 INFO:tasks.rados.rados.0.smithi084.stdout:1857: finishing write tid 5 to smithi08477203-28 2024-09-20T22:49:15.249 INFO:tasks.rados.rados.0.smithi084.stdout:1857: oid 28 updating version 0 to 1204 2024-09-20T22:49:15.249 INFO:tasks.rados.rados.0.smithi084.stdout:1857: oid 28 updating version 1204 to 1206 2024-09-20T22:49:15.249 INFO:tasks.rados.rados.0.smithi084.stdout:1857: oid 28 updating version 1206 to 1207 2024-09-20T22:49:15.249 INFO:tasks.rados.rados.0.smithi084.stdout:1857: oid 28 version 1207 is already newer than 1205 2024-09-20T22:49:15.249 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 28 v 1207 (ObjNum 656 snap 198 seq_num 656) dirty exists 2024-09-20T22:49:15.249 INFO:tasks.rados.rados.0.smithi084.stdout:1857: left oid 28 (ObjNum 656 snap 198 seq_num 656) 2024-09-20T22:49:15.262 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:15.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:15.263 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:49:15.364 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:49:15.371 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-20T22:49:15.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:15.386 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:15.386 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:49:15.460 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.5 dump_blocked_ops 2024-09-20T22:49:15.487 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:49:15.508 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:15.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:15.509 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:49:15.572 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:15 smithi084 ceph-mon[40024]: pgmap v612: 161 pgs: 2 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 155 active+clean; 401 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 4.7 MiB/s rd, 1.2 MiB/s wr, 10 op/s 2024-09-20T22:49:15.572 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:15 smithi084 ceph-mon[40024]: osdmap e475: 8 total, 8 up, 8 in 2024-09-20T22:49:15.572 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:15 smithi084 ceph-mon[29774]: pgmap v612: 161 pgs: 2 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 155 active+clean; 401 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 4.7 MiB/s rd, 1.2 MiB/s wr, 10 op/s 2024-09-20T22:49:15.572 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:15 smithi084 ceph-mon[29774]: osdmap e475: 8 total, 8 up, 8 in 2024-09-20T22:49:15.585 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:49:15.586 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:49:15.610 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:49:15.613 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:49:15.614 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:49:15.631 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:49:15.632 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:49:15.634 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:15.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:15.635 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:49:15.654 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-20T22:49:15.655 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T22:49:15.655 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:49:15.672 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:49:15.672 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:49:15.689 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:49:15.690 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:49:15.706 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:49:15.708 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:49:15.708 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:49:15.725 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:49:15.725 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:49:15.735 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:49:15.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:15 smithi154 ceph-mon[33264]: pgmap v612: 161 pgs: 2 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 155 active+clean; 401 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 4.7 MiB/s rd, 1.2 MiB/s wr, 10 op/s 2024-09-20T22:49:15.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:15 smithi154 ceph-mon[33264]: osdmap e475: 8 total, 8 up, 8 in 2024-09-20T22:49:15.757 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:15.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:15.758 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:49:15.859 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:49:15.880 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:15.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:15.881 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:49:15.982 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:49:16.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:16.004 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:16.004 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:49:16.076 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 5, 6, 7, 0, 4] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-20T22:49:16.076 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-20T22:49:16.076 INFO:tasks.thrashosds.thrasher:inject_pause on osd.0 2024-09-20T22:49:16.076 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-20T22:49:16.076 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-20T22:49:16.077 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.0 config set filestore_inject_stall 3 2024-09-20T22:49:16.105 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:49:16.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:16.142 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:16.142 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:49:16.227 INFO:tasks.rados.rados.0.smithi084.stdout:1856: done (2 left) 2024-09-20T22:49:16.227 INFO:tasks.rados.rados.0.smithi084.stdout:1857: done (1 left) 2024-09-20T22:49:16.227 INFO:tasks.rados.rados.0.smithi084.stdout:1858: done (0 left) 2024-09-20T22:49:16.228 INFO:tasks.rados.rados.0.smithi084.stdout:1859: write oid 18 current snap is 199 2024-09-20T22:49:16.228 INFO:tasks.rados.rados.0.smithi084.stdout:1859: seq_num 657 ranges {642199=644951,1696199=605559,2992318=49881} 2024-09-20T22:49:16.234 INFO:tasks.rados.rados.0.smithi084.stdout:1859: writing smithi08477203-18 from 642199 to 1287150 tid 1 2024-09-20T22:49:16.241 INFO:tasks.rados.rados.0.smithi084.stdout:1859: writing smithi08477203-18 from 1696199 to 2301758 tid 2 2024-09-20T22:49:16.242 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:49:16.245 INFO:tasks.rados.rados.0.smithi084.stdout:1859: writing smithi08477203-18 from 2992318 to 3042199 tid 3 2024-09-20T22:49:16.245 INFO:tasks.rados.rados.0.smithi084.stdout:1860: delete oid 10 current snap is 199 2024-09-20T22:49:16.246 INFO:tasks.rados.rados.0.smithi084.stdout:1859: finishing write tid 1 to smithi08477203-18 2024-09-20T22:49:16.246 INFO:tasks.rados.rados.0.smithi084.stdout:1859: finishing write tid 2 to smithi08477203-18 2024-09-20T22:49:16.247 INFO:tasks.rados.rados.0.smithi084.stdout:1859: finishing write tid 3 to smithi08477203-18 2024-09-20T22:49:16.248 INFO:tasks.rados.rados.0.smithi084.stdout:1859: finishing write tid 4 to smithi08477203-18 2024-09-20T22:49:16.248 INFO:tasks.rados.rados.0.smithi084.stdout:1859: finishing write tid 5 to smithi08477203-18 2024-09-20T22:49:16.249 INFO:tasks.rados.rados.0.smithi084.stdout:1859: oid 18 updating version 0 to 666 2024-09-20T22:49:16.249 INFO:tasks.rados.rados.0.smithi084.stdout:1859: oid 18 version 666 is already newer than 663 2024-09-20T22:49:16.249 INFO:tasks.rados.rados.0.smithi084.stdout:1859: oid 18 version 666 is already newer than 664 2024-09-20T22:49:16.249 INFO:tasks.rados.rados.0.smithi084.stdout:1859: oid 18 version 666 is already newer than 665 2024-09-20T22:49:16.249 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 18 v 666 (ObjNum 657 snap 199 seq_num 657) dirty exists 2024-09-20T22:49:16.249 INFO:tasks.rados.rados.0.smithi084.stdout:1859: left oid 18 (ObjNum 657 snap 199 seq_num 657) 2024-09-20T22:49:16.249 INFO:tasks.rados.rados.0.smithi084.stdout:1859: done (1 left) 2024-09-20T22:49:16.249 INFO:tasks.rados.rados.0.smithi084.stdout:1860: done (0 left) 2024-09-20T22:49:16.249 INFO:tasks.rados.rados.0.smithi084.stdout:1861: delete oid 11 current snap is 199 2024-09-20T22:49:16.251 INFO:tasks.rados.rados.0.smithi084.stdout:1861: done (0 left) 2024-09-20T22:49:16.251 INFO:tasks.rados.rados.0.smithi084.stdout:1862: read oid 33 snap 165 2024-09-20T22:49:16.251 INFO:tasks.rados.rados.0.smithi084.stdout:1862: expect (ObjNum 395 snap 101 seq_num 395) 2024-09-20T22:49:16.251 INFO:tasks.rados.rados.0.smithi084.stdout:1863: write (excl) oid 50 current snap is 199 2024-09-20T22:49:16.251 INFO:tasks.rados.rados.0.smithi084.stdout:1863: seq_num 660 ranges {692620=470698,1755292=752886} 2024-09-20T22:49:16.258 INFO:tasks.rados.rados.0.smithi084.stdout:1863: writing smithi08477203-50 from 692620 to 1163318 tid 1 2024-09-20T22:49:16.265 INFO:tasks.rados.rados.0.smithi084.stdout:1863: writing smithi08477203-50 from 1755292 to 2508178 tid 2 2024-09-20T22:49:16.265 INFO:tasks.rados.rados.0.smithi084.stdout:1864: delete oid 20 current snap is 199 2024-09-20T22:49:16.265 INFO:tasks.rados.rados.0.smithi084.stdout:1862: expect (ObjNum 395 snap 101 seq_num 395) 2024-09-20T22:49:16.267 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:16.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:16.268 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:49:16.321 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-20T22:49:16.321 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [], 2024-09-20T22:49:16.321 INFO:teuthology.orchestra.run.smithi154.stdout: "complaint_time": 30, 2024-09-20T22:49:16.321 INFO:teuthology.orchestra.run.smithi154.stdout: "num_blocked_ops": 0 2024-09-20T22:49:16.321 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-20T22:49:16.369 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:49:16.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:16.391 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:16.392 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:49:16.395 INFO:tasks.rados.rados.0.smithi084.stdout:1863: finishing write tid 1 to smithi08477203-50 2024-09-20T22:49:16.395 INFO:tasks.rados.rados.0.smithi084.stdout:1862: done (2 left) 2024-09-20T22:49:16.395 INFO:tasks.rados.rados.0.smithi084.stdout:1864: done (1 left) 2024-09-20T22:49:16.396 INFO:tasks.rados.rados.0.smithi084.stdout:1865: copy_from oid 40 from oid 11 current snap is 199 2024-09-20T22:49:16.396 INFO:tasks.rados.rados.0.smithi084.stdout:1863: finishing write tid 2 to smithi08477203-50 2024-09-20T22:49:16.396 INFO:tasks.rados.rados.0.smithi084.stdout:1863: finishing write tid 3 to smithi08477203-50 2024-09-20T22:49:16.396 INFO:tasks.rados.rados.0.smithi084.stdout:1863: finishing write tid 4 to smithi08477203-50 2024-09-20T22:49:16.396 INFO:tasks.rados.rados.0.smithi084.stdout:1863: oid 50 updating version 0 to 260 2024-09-20T22:49:16.396 INFO:tasks.rados.rados.0.smithi084.stdout:1863: oid 50 version 260 is already newer than 259 2024-09-20T22:49:16.397 INFO:tasks.rados.rados.0.smithi084.stdout:1863: oid 50 updating version 260 to 261 2024-09-20T22:49:16.397 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 50 v 261 (ObjNum 660 snap 199 seq_num 660) dirty exists 2024-09-20T22:49:16.397 INFO:tasks.rados.rados.0.smithi084.stdout:1863: left oid 50 (ObjNum 660 snap 199 seq_num 660) 2024-09-20T22:49:16.397 INFO:tasks.rados.rados.0.smithi084.stdout:1863: done (1 left) 2024-09-20T22:49:16.397 INFO:tasks.rados.rados.0.smithi084.stdout:1866: read oid 9 snap -1 2024-09-20T22:49:16.397 INFO:tasks.rados.rados.0.smithi084.stdout:1866: expect deleted 2024-09-20T22:49:16.397 INFO:tasks.rados.rados.0.smithi084.stdout:1867: copy_from oid 44 from oid 15 current snap is 199 2024-09-20T22:49:16.397 INFO:tasks.rados.rados.0.smithi084.stdout:1868: copy_from oid 12 from oid 26 current snap is 199 2024-09-20T22:49:16.397 INFO:tasks.rados.rados.0.smithi084.stdout:1869: snap_remove snap 163 2024-09-20T22:49:16.493 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:49:16.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:16 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/3130824515' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:49:16.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:16 smithi154 ceph-mon[33264]: osdmap e476: 8 total, 8 up, 8 in 2024-09-20T22:49:16.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:16.515 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:16.515 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:49:16.614 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:16 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/3130824515' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:49:16.614 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:16 smithi084 ceph-mon[29774]: osdmap e476: 8 total, 8 up, 8 in 2024-09-20T22:49:16.614 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:16 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/3130824515' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:49:16.614 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:16 smithi084 ceph-mon[40024]: osdmap e476: 8 total, 8 up, 8 in 2024-09-20T22:49:16.615 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:49:16.637 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:16.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:16.638 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:49:16.739 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:49:16.761 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:16.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:16.762 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:49:16.862 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:49:16.884 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:16.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:16.885 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:49:16.974 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:49:16.986 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:49:17.004 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.5 dump_historic_ops 2024-09-20T22:49:17.008 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:17.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:17.008 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:49:17.110 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:49:17.131 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:17.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:17.132 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:49:17.225 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is set 2024-09-20T22:49:17.232 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:49:17.233 INFO:tasks.rados.rados.0.smithi084.stdout:1869: done (4 left) 2024-09-20T22:49:17.233 INFO:tasks.rados.rados.0.smithi084.stdout:1870: snap_create 2024-09-20T22:49:17.233 INFO:tasks.rados.rados.0.smithi084.stdout:1865: finishing copy_from to smithi08477203-40 2024-09-20T22:49:17.233 INFO:tasks.rados.rados.0.smithi084.stdout:1865: got expected ENOENT (src dne) 2024-09-20T22:49:17.234 INFO:tasks.rados.rados.0.smithi084.stdout:1865: finishing copy_from racing read to smithi08477203-40 2024-09-20T22:49:17.234 INFO:tasks.rados.rados.0.smithi084.stdout:1868: finishing copy_from racing read to smithi08477203-12 2024-09-20T22:49:17.234 INFO:tasks.rados.rados.0.smithi084.stdout:1867: finishing copy_from racing read to smithi08477203-44 2024-09-20T22:49:17.234 INFO:tasks.rados.rados.0.smithi084.stdout:1868: finishing copy_from to smithi08477203-12 2024-09-20T22:49:17.234 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 12 v 543 (ObjNum 642 snap 193 seq_num 642) dirty exists 2024-09-20T22:49:17.234 INFO:tasks.rados.rados.0.smithi084.stdout:1867: finishing copy_from to smithi08477203-44 2024-09-20T22:49:17.234 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 44 v 749 (ObjNum 625 snap 186 seq_num 625) dirty exists 2024-09-20T22:49:17.259 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:17.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:17.260 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:49:17.261 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:49:17.362 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:49:17.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:17 smithi084 ceph-mon[29774]: pgmap v615: 161 pgs: 2 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 155 active+clean; 401 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 4.7 MiB/s rd, 1.2 MiB/s wr, 10 op/s 2024-09-20T22:49:17.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:17 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/1956175249' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:49:17.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:17 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/1956175249' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T22:49:17.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:17 smithi084 ceph-mon[29774]: osdmap e477: 8 total, 8 up, 8 in 2024-09-20T22:49:17.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:17 smithi084 ceph-mon[40024]: pgmap v615: 161 pgs: 2 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 155 active+clean; 401 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 4.7 MiB/s rd, 1.2 MiB/s wr, 10 op/s 2024-09-20T22:49:17.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:17 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/1956175249' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:49:17.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:17 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/1956175249' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T22:49:17.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:17 smithi084 ceph-mon[40024]: osdmap e477: 8 total, 8 up, 8 in 2024-09-20T22:49:17.386 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:17.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:17.387 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:49:17.488 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:49:17.509 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:17.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:17.510 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:49:17.583 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:17 smithi154 ceph-mon[33264]: pgmap v615: 161 pgs: 2 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 155 active+clean; 401 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 4.7 MiB/s rd, 1.2 MiB/s wr, 10 op/s 2024-09-20T22:49:17.583 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:17 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/1956175249' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:49:17.583 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:17 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/1956175249' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T22:49:17.583 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:17 smithi154 ceph-mon[33264]: osdmap e477: 8 total, 8 up, 8 in 2024-09-20T22:49:17.611 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:49:17.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:17.635 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:17.635 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:49:17.736 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:49:17.759 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:17.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:17.760 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:49:17.861 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:49:17.884 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:17.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:17.885 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:49:17.895 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-20T22:49:17.895 INFO:teuthology.orchestra.run.smithi154.stdout: "size": 20, 2024-09-20T22:49:17.895 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 600, 2024-09-20T22:49:17.896 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [ 2024-09-20T22:49:17.896 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.896 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24334.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-09-20T22:49:17.896 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:41:40.156321+0000", 2024-09-20T22:49:17.896 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 457.73678822400001, 2024-09-20T22:49:17.896 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.012506929, 2024-09-20T22:49:17.896 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:49:17.896 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:49:17.896 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:49:17.896 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:49:17.896 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24334", 2024-09-20T22:49:17.896 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/343304630", 2024-09-20T22:49:17.896 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 7 2024-09-20T22:49:17.896 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.897 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:49:17.897 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.897 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:49:17.897 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:40.156321+0000", 2024-09-20T22:49:17.897 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:49:17.897 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.897 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.897 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:49:17.897 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:40.156321+0000", 2024-09-20T22:49:17.897 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:49:17.897 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.897 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.897 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:49:17.897 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:40.156323+0000", 2024-09-20T22:49:17.898 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9700000000000002e-06 2024-09-20T22:49:17.898 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.898 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.898 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:49:17.898 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:40.156330+0000", 2024-09-20T22:49:17.898 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.9809999999999997e-06 2024-09-20T22:49:17.898 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.898 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.898 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:49:17.898 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:40.156348+0000", 2024-09-20T22:49:17.898 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.7938e-05 2024-09-20T22:49:17.898 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.898 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.898 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:49:17.899 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:40.156362+0000", 2024-09-20T22:49:17.899 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3641e-05 2024-09-20T22:49:17.899 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.899 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.899 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:49:17.899 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:40.156430+0000", 2024-09-20T22:49:17.899 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.7543e-05 2024-09-20T22:49:17.899 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.899 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.899 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:49:17.899 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:40.156543+0000", 2024-09-20T22:49:17.900 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00011390100000000001 2024-09-20T22:49:17.900 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.900 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.900 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 1,6", 2024-09-20T22:49:17.900 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:40.156712+0000", 2024-09-20T22:49:17.900 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00016825499999999999 2024-09-20T22:49:17.900 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.900 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.900 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:49:17.900 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:40.157640+0000", 2024-09-20T22:49:17.900 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00092863699999999995 2024-09-20T22:49:17.900 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.900 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.900 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:49:17.901 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:40.159647+0000", 2024-09-20T22:49:17.901 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0020068320000000001 2024-09-20T22:49:17.901 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.901 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.901 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:49:17.901 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:40.168687+0000", 2024-09-20T22:49:17.901 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0090397749999999999 2024-09-20T22:49:17.901 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.901 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.901 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:49:17.901 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:40.168768+0000", 2024-09-20T22:49:17.901 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.0819e-05 2024-09-20T22:49:17.901 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.902 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.902 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:49:17.902 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:40.168828+0000", 2024-09-20T22:49:17.902 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.0637000000000002e-05 2024-09-20T22:49:17.902 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:49:17.902 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:49:17.902 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:49:17.902 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.902 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.902 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "MOSDPGPush(1.4 93/92 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 77'128, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@77'128, size: 327680, copy_subset: [(0, 327680)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 327680, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-20T22:49:17.902 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:42:50.646393+0000", 2024-09-20T22:49:17.902 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 387.24671678499999, 2024-09-20T22:49:17.902 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.033640915, 2024-09-20T22:49:17.903 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:49:17.903 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:49:17.903 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:49:17.903 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:49:17.903 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.903 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:49:17.903 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:42:50.646393+0000", 2024-09-20T22:49:17.903 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:49:17.903 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.903 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.903 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:49:17.903 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:42:50.646393+0000", 2024-09-20T22:49:17.903 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:49:17.903 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.904 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.904 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:49:17.904 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:42:50.646393+0000", 2024-09-20T22:49:17.904 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.23e-07 2024-09-20T22:49:17.904 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.904 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.904 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:49:17.904 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:42:50.679431+0000", 2024-09-20T22:49:17.904 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.033038349000000002 2024-09-20T22:49:17.904 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.904 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.904 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:49:17.904 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:42:50.679478+0000", 2024-09-20T22:49:17.904 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.6828e-05 2024-09-20T22:49:17.905 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.905 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.905 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:49:17.905 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:42:50.679492+0000", 2024-09-20T22:49:17.905 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4062e-05 2024-09-20T22:49:17.905 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.905 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.905 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:49:17.905 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:42:50.679593+0000", 2024-09-20T22:49:17.905 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000100865 2024-09-20T22:49:17.905 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.905 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.905 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:49:17.905 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:42:50.679604+0000", 2024-09-20T22:49:17.906 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0348e-05 2024-09-20T22:49:17.906 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.906 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.906 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:49:17.906 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:42:50.680034+0000", 2024-09-20T22:49:17.906 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00043023999999999997 2024-09-20T22:49:17.906 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:49:17.906 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:49:17.906 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:49:17.906 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.906 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.906 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:554 7.a e128/94 7:5122292a:::smithi08477203-33:head v 128'32, mlcod=119'28)", 2024-09-20T22:49:17.906 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:27.341250+0000", 2024-09-20T22:49:17.906 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 350.55185920100001, 2024-09-20T22:49:17.907 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.051661319999999997, 2024-09-20T22:49:17.907 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:49:17.907 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:49:17.907 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:49:17.907 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:49:17.907 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.907 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:49:17.907 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.341250+0000", 2024-09-20T22:49:17.907 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:49:17.907 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.907 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.907 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:49:17.907 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.341250+0000", 2024-09-20T22:49:17.907 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:49:17.908 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.908 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.908 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:49:17.908 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.341251+0000", 2024-09-20T22:49:17.908 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.7299999999999996e-07 2024-09-20T22:49:17.908 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.908 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.908 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:49:17.908 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.390395+0000", 2024-09-20T22:49:17.908 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.049143539999999999 2024-09-20T22:49:17.908 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.908 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.908 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:49:17.908 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.390413+0000", 2024-09-20T22:49:17.909 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.8709999999999999e-05 2024-09-20T22:49:17.909 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.909 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.910 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:49:17.910 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.390423+0000", 2024-09-20T22:49:17.910 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.4660000000000005e-06 2024-09-20T22:49:17.910 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.910 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.910 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:49:17.910 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.390484+0000", 2024-09-20T22:49:17.910 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.0851999999999999e-05 2024-09-20T22:49:17.910 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.910 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.910 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:49:17.911 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.390504+0000", 2024-09-20T22:49:17.911 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9976999999999999e-05 2024-09-20T22:49:17.911 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.911 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.911 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:49:17.911 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.392875+0000", 2024-09-20T22:49:17.911 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0023713710000000002 2024-09-20T22:49:17.911 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.911 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.911 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:49:17.911 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.392912+0000", 2024-09-20T22:49:17.911 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.6831e-05 2024-09-20T22:49:17.911 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:49:17.912 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:49:17.912 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:49:17.912 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.912 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.912 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:786 7.17 e145/139 7:ed54065f:::smithi08477203-12:head v 145'119, mlcod=143'117)", 2024-09-20T22:49:17.912 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:44.345115+0000", 2024-09-20T22:49:17.912 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 333.54799471899997, 2024-09-20T22:49:17.912 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0098654899999999993, 2024-09-20T22:49:17.912 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:49:17.912 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:49:17.912 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:49:17.912 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:49:17.912 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.913 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:49:17.913 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.345115+0000", 2024-09-20T22:49:17.913 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:49:17.913 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.913 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.913 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:49:17.913 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.345115+0000", 2024-09-20T22:49:17.913 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:49:17.913 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.913 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.913 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:49:17.913 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.345116+0000", 2024-09-20T22:49:17.913 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.8299999999999995e-07 2024-09-20T22:49:17.913 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.913 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.914 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:49:17.914 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.347955+0000", 2024-09-20T22:49:17.914 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.002839028 2024-09-20T22:49:17.914 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.914 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.914 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:49:17.914 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.347977+0000", 2024-09-20T22:49:17.914 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.1897999999999999e-05 2024-09-20T22:49:17.914 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.914 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.914 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:49:17.914 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.347992+0000", 2024-09-20T22:49:17.914 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4761e-05 2024-09-20T22:49:17.914 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.915 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.915 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:49:17.915 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.348072+0000", 2024-09-20T22:49:17.915 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.0586000000000004e-05 2024-09-20T22:49:17.915 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.915 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.915 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:49:17.915 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.348097+0000", 2024-09-20T22:49:17.915 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.5026000000000001e-05 2024-09-20T22:49:17.915 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.915 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.915 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:49:17.915 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.354924+0000", 2024-09-20T22:49:17.915 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0068271240000000004 2024-09-20T22:49:17.915 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.916 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.916 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:49:17.916 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.354980+0000", 2024-09-20T22:49:17.916 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.6184000000000003e-05 2024-09-20T22:49:17.916 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:49:17.916 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:49:17.916 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:49:17.916 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.916 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.916 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:894 7.a e155/94 7:50097c2d:::smithi08477203-23:head v 155'54, mlcod=143'50)", 2024-09-20T22:49:17.916 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:54.450745+0000", 2024-09-20T22:49:17.916 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 323.44236463099998, 2024-09-20T22:49:17.916 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0091901029999999998, 2024-09-20T22:49:17.916 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:49:17.917 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:49:17.917 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:49:17.917 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:49:17.917 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.917 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:49:17.917 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:54.450745+0000", 2024-09-20T22:49:17.917 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:49:17.917 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.917 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.917 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:49:17.917 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:54.450745+0000", 2024-09-20T22:49:17.917 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:49:17.917 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.917 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.918 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:49:17.918 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:54.450745+0000", 2024-09-20T22:49:17.918 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.4200000000000002e-07 2024-09-20T22:49:17.918 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.918 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.918 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:49:17.918 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:54.456336+0000", 2024-09-20T22:49:17.918 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.005590589 2024-09-20T22:49:17.918 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.918 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.918 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:49:17.918 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:54.456353+0000", 2024-09-20T22:49:17.918 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.7187e-05 2024-09-20T22:49:17.918 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.919 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.919 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:49:17.919 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:54.456362+0000", 2024-09-20T22:49:17.919 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.7409999999999998e-06 2024-09-20T22:49:17.919 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.919 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.919 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:49:17.919 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:54.456418+0000", 2024-09-20T22:49:17.919 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.6431000000000002e-05 2024-09-20T22:49:17.919 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.919 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.919 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:49:17.919 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:54.456438+0000", 2024-09-20T22:49:17.919 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9718999999999999e-05 2024-09-20T22:49:17.920 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.920 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.920 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:49:17.920 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:54.459899+0000", 2024-09-20T22:49:17.920 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.003461254 2024-09-20T22:49:17.920 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.920 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.920 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:49:17.920 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:54.459935+0000", 2024-09-20T22:49:17.920 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.5939999999999998e-05 2024-09-20T22:49:17.920 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:49:17.920 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:49:17.920 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:49:17.920 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.920 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.921 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:901 7.e e156/94 7:7c62ced4:::smithi08477203-17:head v 156'42, mlcod=154'40)", 2024-09-20T22:49:17.921 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:55.635897+0000", 2024-09-20T22:49:17.921 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 322.25721223800002, 2024-09-20T22:49:17.921 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0082336760000000005, 2024-09-20T22:49:17.921 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:49:17.921 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:49:17.921 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:49:17.921 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:49:17.921 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.921 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:49:17.921 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:55.635897+0000", 2024-09-20T22:49:17.921 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:49:17.921 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.921 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.922 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:49:17.922 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:55.635897+0000", 2024-09-20T22:49:17.922 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:49:17.922 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.922 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.922 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:49:17.922 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:55.635898+0000", 2024-09-20T22:49:17.922 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.23e-07 2024-09-20T22:49:17.922 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.922 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.922 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:49:17.922 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:55.640084+0000", 2024-09-20T22:49:17.922 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0041865499999999998 2024-09-20T22:49:17.922 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.922 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.923 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:49:17.923 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:55.640118+0000", 2024-09-20T22:49:17.923 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.4093e-05 2024-09-20T22:49:17.923 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.923 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.923 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:49:17.923 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:55.640129+0000", 2024-09-20T22:49:17.923 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0281e-05 2024-09-20T22:49:17.923 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.923 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.923 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:49:17.923 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:55.640230+0000", 2024-09-20T22:49:17.923 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000101538 2024-09-20T22:49:17.923 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.924 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.924 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:49:17.924 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:55.640251+0000", 2024-09-20T22:49:17.924 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.1293000000000001e-05 2024-09-20T22:49:17.924 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.924 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.924 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:49:17.924 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:55.644088+0000", 2024-09-20T22:49:17.924 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.003836178 2024-09-20T22:49:17.924 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.924 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.924 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:49:17.924 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:55.644131+0000", 2024-09-20T22:49:17.925 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.3519999999999997e-05 2024-09-20T22:49:17.925 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:49:17.925 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:49:17.925 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:49:17.925 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.925 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.925 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:1316 7.0 7:05bb442d:::smithi08477203-24:head [stat,delete] snapc 2b=[2b,2a] ondisk+write+known_if_redirected+supports_pool_eio e180)", 2024-09-20T22:49:17.925 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:44:19.225492+0000", 2024-09-20T22:49:17.925 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 298.667617667, 2024-09-20T22:49:17.925 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0083403250000000009, 2024-09-20T22:49:17.925 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:49:17.925 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:49:17.925 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:49:17.925 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:49:17.926 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:49:17.926 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:49:17.926 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 1316 2024-09-20T22:49:17.926 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.926 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:49:17.926 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.926 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:49:17.926 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:19.225492+0000", 2024-09-20T22:49:17.926 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:49:17.926 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.926 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.926 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:49:17.926 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:19.225492+0000", 2024-09-20T22:49:17.926 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:49:17.927 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.927 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.927 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:49:17.927 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:19.225495+0000", 2024-09-20T22:49:17.927 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.96e-06 2024-09-20T22:49:17.927 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.927 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.927 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:49:17.927 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:19.225508+0000", 2024-09-20T22:49:17.927 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3343000000000001e-05 2024-09-20T22:49:17.927 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.927 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.927 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:49:17.927 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:19.225542+0000", 2024-09-20T22:49:17.928 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.3973000000000002e-05 2024-09-20T22:49:17.928 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.928 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.928 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:49:17.928 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:19.225570+0000", 2024-09-20T22:49:17.928 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.7713999999999999e-05 2024-09-20T22:49:17.928 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.928 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.928 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:49:17.928 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:19.225655+0000", 2024-09-20T22:49:17.928 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.4944999999999994e-05 2024-09-20T22:49:17.928 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.928 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.928 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op must wait for map", 2024-09-20T22:49:17.928 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:19.225660+0000", 2024-09-20T22:49:17.929 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.5470000000000004e-06 2024-09-20T22:49:17.929 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.929 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.929 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:49:17.929 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:19.232618+0000", 2024-09-20T22:49:17.929 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0069571069999999997 2024-09-20T22:49:17.929 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.929 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.929 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:49:17.929 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:19.232717+0000", 2024-09-20T22:49:17.929 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.9306000000000001e-05 2024-09-20T22:49:17.929 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.930 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.930 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:49:17.930 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:19.233832+0000", 2024-09-20T22:49:17.930 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0011154299999999999 2024-09-20T22:49:17.930 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:49:17.930 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:49:17.930 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:49:17.930 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.930 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.930 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:1527 7.17 e197/139 7:ea720b85:::smithi08477203-26:head v 197'159, mlcod=187'156)", 2024-09-20T22:49:17.930 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:44:36.209560+0000", 2024-09-20T22:49:17.931 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 281.68355004, 2024-09-20T22:49:17.931 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.010263069999999999, 2024-09-20T22:49:17.931 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:49:17.931 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:49:17.931 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:49:17.931 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:49:17.931 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.931 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:49:17.931 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.209560+0000", 2024-09-20T22:49:17.931 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:49:17.932 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.932 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.932 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:49:17.932 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.209560+0000", 2024-09-20T22:49:17.932 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:49:17.932 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.932 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.932 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:49:17.932 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.209560+0000", 2024-09-20T22:49:17.933 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.0800000000000001e-07 2024-09-20T22:49:17.933 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.933 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.933 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:49:17.933 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.212844+0000", 2024-09-20T22:49:17.933 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.003283907 2024-09-20T22:49:17.933 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.933 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.933 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:49:17.933 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.212867+0000", 2024-09-20T22:49:17.933 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.3229000000000001e-05 2024-09-20T22:49:17.933 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.933 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.933 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:49:17.933 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.212878+0000", 2024-09-20T22:49:17.934 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0998e-05 2024-09-20T22:49:17.934 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.934 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.934 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:49:17.934 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.212971+0000", 2024-09-20T22:49:17.934 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.2872000000000001e-05 2024-09-20T22:49:17.934 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.934 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.934 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:49:17.934 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.212996+0000", 2024-09-20T22:49:17.934 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.4921000000000001e-05 2024-09-20T22:49:17.934 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.934 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.935 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:49:17.935 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.219758+0000", 2024-09-20T22:49:17.935 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0067625139999999999 2024-09-20T22:49:17.935 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.935 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.935 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:49:17.935 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.219823+0000", 2024-09-20T22:49:17.935 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.4420999999999999e-05 2024-09-20T22:49:17.935 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:49:17.935 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:49:17.935 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:49:17.935 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.935 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.935 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:1825 7.b e230/94 7:d960cad4:::smithi08477203-18:head v 230'138, mlcod=229'135)", 2024-09-20T22:49:17.935 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:45:09.177560+0000", 2024-09-20T22:49:17.936 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 248.71555017599999, 2024-09-20T22:49:17.936 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0086763780000000002, 2024-09-20T22:49:17.936 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:49:17.936 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:49:17.936 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:49:17.936 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:49:17.936 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.936 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:49:17.936 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:09.177560+0000", 2024-09-20T22:49:17.936 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:49:17.936 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.936 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.936 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:49:17.936 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:09.177560+0000", 2024-09-20T22:49:17.937 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:49:17.937 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.937 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.937 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:49:17.937 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:09.177560+0000", 2024-09-20T22:49:17.937 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.0599999999999999e-07 2024-09-20T22:49:17.937 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.937 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.937 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:49:17.937 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:09.178522+0000", 2024-09-20T22:49:17.937 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00096277000000000005 2024-09-20T22:49:17.937 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.937 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.937 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:49:17.938 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:09.178536+0000", 2024-09-20T22:49:17.938 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3886999999999999e-05 2024-09-20T22:49:17.938 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.938 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.938 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:49:17.938 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:09.178547+0000", 2024-09-20T22:49:17.938 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0526999999999999e-05 2024-09-20T22:49:17.938 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.938 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.938 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:49:17.938 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:09.178614+0000", 2024-09-20T22:49:17.938 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.7547999999999998e-05 2024-09-20T22:49:17.938 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.938 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.939 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:49:17.939 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:09.178638+0000", 2024-09-20T22:49:17.939 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.3190999999999999e-05 2024-09-20T22:49:17.939 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.939 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.939 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:49:17.939 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:09.186193+0000", 2024-09-20T22:49:17.939 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0075549559999999998 2024-09-20T22:49:17.939 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.939 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.939 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:49:17.939 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:09.186236+0000", 2024-09-20T22:49:17.939 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.3293e-05 2024-09-20T22:49:17.940 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:49:17.940 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:49:17.940 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:49:17.940 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.940 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.940 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:2350 7.0 7:064f66ab:::smithi08477203-45:head [stat,delete] snapc 56=[56,55,53,52,51,50,47,3e] ondisk+write+known_if_redirected+supports_pool_eio e260)", 2024-09-20T22:49:17.940 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:45:39.199566+0000", 2024-09-20T22:49:17.940 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 218.693543611, 2024-09-20T22:49:17.940 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.027897148, 2024-09-20T22:49:17.940 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:49:17.940 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:49:17.940 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:49:17.940 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:49:17.940 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:49:17.941 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:49:17.941 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 2350 2024-09-20T22:49:17.941 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.941 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:49:17.941 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.941 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:49:17.941 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.199566+0000", 2024-09-20T22:49:17.941 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:49:17.941 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.941 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.941 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:49:17.941 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.199566+0000", 2024-09-20T22:49:17.941 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:49:17.941 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.941 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.942 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:49:17.942 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.199568+0000", 2024-09-20T22:49:17.942 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9290000000000001e-06 2024-09-20T22:49:17.942 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.942 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.942 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:49:17.942 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.199574+0000", 2024-09-20T22:49:17.942 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.3409999999999999e-06 2024-09-20T22:49:17.942 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.942 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.942 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:49:17.942 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.199584+0000", 2024-09-20T22:49:17.942 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0091e-05 2024-09-20T22:49:17.943 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.943 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.943 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:49:17.943 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.199593+0000", 2024-09-20T22:49:17.943 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.1079999999999993e-06 2024-09-20T22:49:17.943 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.943 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.943 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:49:17.943 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.199661+0000", 2024-09-20T22:49:17.943 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.8767000000000001e-05 2024-09-20T22:49:17.943 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.943 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.944 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:49:17.945 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.199765+0000", 2024-09-20T22:49:17.945 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000103886 2024-09-20T22:49:17.945 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.945 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.945 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:49:17.945 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.227463+0000", 2024-09-20T22:49:17.945 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.027698026000000001 2024-09-20T22:49:17.945 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:49:17.945 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:49:17.945 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:49:17.945 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.945 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.945 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:3134 7.2 7:45b54b6b:::smithi08477203-2:head [copy-from ver 151] snapc 77=[77,76,75,74,73,70,6b] ondisk+write+known_if_redirected+supports_pool_eio e328)", 2024-09-20T22:49:17.945 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:46:48.211103+0000", 2024-09-20T22:49:17.946 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 149.68200710400001, 2024-09-20T22:49:17.946 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.030812519, 2024-09-20T22:49:17.946 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:49:17.946 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:49:17.946 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:49:17.946 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:49:17.946 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:49:17.946 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:49:17.946 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 3134 2024-09-20T22:49:17.946 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.946 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:49:17.946 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.946 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:49:17.946 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.211103+0000", 2024-09-20T22:49:17.947 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:49:17.947 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.947 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.947 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:49:17.947 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.211103+0000", 2024-09-20T22:49:17.947 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:49:17.947 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.947 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.947 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:49:17.947 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.211105+0000", 2024-09-20T22:49:17.947 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.3369999999999998e-06 2024-09-20T22:49:17.947 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.947 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.948 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:49:17.948 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.211111+0000", 2024-09-20T22:49:17.948 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.2759999999999997e-06 2024-09-20T22:49:17.948 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.948 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.948 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:49:17.948 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.211123+0000", 2024-09-20T22:49:17.948 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2119999999999999e-05 2024-09-20T22:49:17.948 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.948 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.948 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:49:17.948 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.211131+0000", 2024-09-20T22:49:17.948 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.2899999999999997e-06 2024-09-20T22:49:17.948 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.948 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.949 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:49:17.949 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.211213+0000", 2024-09-20T22:49:17.949 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.2756999999999994e-05 2024-09-20T22:49:17.949 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.949 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.949 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:49:17.949 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.211310+0000", 2024-09-20T22:49:17.949 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.6644999999999994e-05 2024-09-20T22:49:17.949 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.949 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.949 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 1,2", 2024-09-20T22:49:17.949 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.229109+0000", 2024-09-20T22:49:17.949 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.017798587000000001 2024-09-20T22:49:17.949 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.950 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.950 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:49:17.950 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.240265+0000", 2024-09-20T22:49:17.950 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.011156716000000001 2024-09-20T22:49:17.950 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.950 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.950 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:49:17.950 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.241608+0000", 2024-09-20T22:49:17.950 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001343033 2024-09-20T22:49:17.950 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.950 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.950 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:49:17.950 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.241831+0000", 2024-09-20T22:49:17.950 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000223148 2024-09-20T22:49:17.950 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.951 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.951 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:49:17.951 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.241885+0000", 2024-09-20T22:49:17.951 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.3161000000000001e-05 2024-09-20T22:49:17.951 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.951 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.951 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:49:17.951 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.241915+0000", 2024-09-20T22:49:17.951 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.0448999999999999e-05 2024-09-20T22:49:17.951 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:49:17.951 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:49:17.951 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:49:17.951 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.951 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.952 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:3135 7.2 7:45b54b6b:::smithi08477203-2:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e328)", 2024-09-20T22:49:17.952 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:46:48.211153+0000", 2024-09-20T22:49:17.952 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 149.68195713200001, 2024-09-20T22:49:17.952 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.022731534000000001, 2024-09-20T22:49:17.952 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:49:17.952 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:49:17.952 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:49:17.952 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:49:17.952 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:49:17.952 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:49:17.952 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 3135 2024-09-20T22:49:17.952 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.952 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:49:17.952 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.952 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:49:17.953 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.211153+0000", 2024-09-20T22:49:17.953 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:49:17.953 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.953 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.953 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:49:17.953 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.211153+0000", 2024-09-20T22:49:17.953 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:49:17.953 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.953 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.953 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:49:17.953 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.211153+0000", 2024-09-20T22:49:17.953 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.1600000000000001e-07 2024-09-20T22:49:17.953 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.953 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.954 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:49:17.954 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.211157+0000", 2024-09-20T22:49:17.954 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.5470000000000001e-06 2024-09-20T22:49:17.954 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.954 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.954 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:49:17.954 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.211164+0000", 2024-09-20T22:49:17.954 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.2799999999999998e-06 2024-09-20T22:49:17.954 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.954 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.954 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:49:17.954 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.211170+0000", 2024-09-20T22:49:17.954 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.0730000000000001e-06 2024-09-20T22:49:17.954 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.955 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.955 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:49:17.955 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.211634+0000", 2024-09-20T22:49:17.955 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00046401800000000002 2024-09-20T22:49:17.955 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.955 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.955 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for blocked object", 2024-09-20T22:49:17.955 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.211690+0000", 2024-09-20T22:49:17.955 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.5587999999999997e-05 2024-09-20T22:49:17.955 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.955 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.955 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:49:17.955 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.233669+0000", 2024-09-20T22:49:17.955 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.021979240000000001 2024-09-20T22:49:17.956 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.956 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.956 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:49:17.956 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.233762+0000", 2024-09-20T22:49:17.956 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.2658999999999995e-05 2024-09-20T22:49:17.956 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.956 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.956 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:49:17.956 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.233884+0000", 2024-09-20T22:49:17.956 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00012241300000000001 2024-09-20T22:49:17.956 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:49:17.956 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:49:17.956 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:49:17.956 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.957 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.957 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:3136 7.a e328/94 7:5122292a:::smithi08477203-33:head v 328'252, mlcod=322'248)", 2024-09-20T22:49:17.957 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:46:48.223492+0000", 2024-09-20T22:49:17.957 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 149.669617421, 2024-09-20T22:49:17.957 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.016662746999999999, 2024-09-20T22:49:17.957 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:49:17.957 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:49:17.957 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:49:17.957 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:49:17.957 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.957 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:49:17.957 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.223492+0000", 2024-09-20T22:49:17.957 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:49:17.958 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.958 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.958 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:49:17.958 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.223492+0000", 2024-09-20T22:49:17.958 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:49:17.958 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.958 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.958 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:49:17.958 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.223493+0000", 2024-09-20T22:49:17.958 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.4299999999999998e-07 2024-09-20T22:49:17.958 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.958 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.958 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:49:17.958 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.228060+0000", 2024-09-20T22:49:17.958 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0045669350000000003 2024-09-20T22:49:17.959 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.959 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.959 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:49:17.959 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.228107+0000", 2024-09-20T22:49:17.959 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.6932999999999999e-05 2024-09-20T22:49:17.959 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.959 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.959 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:49:17.959 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.228127+0000", 2024-09-20T22:49:17.959 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.0777000000000001e-05 2024-09-20T22:49:17.959 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.959 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.959 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:49:17.960 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.228206+0000", 2024-09-20T22:49:17.960 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.8660999999999998e-05 2024-09-20T22:49:17.960 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.960 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.960 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:49:17.960 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.228232+0000", 2024-09-20T22:49:17.960 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.6199000000000001e-05 2024-09-20T22:49:17.960 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.960 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.960 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:49:17.960 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.240103+0000", 2024-09-20T22:49:17.960 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.011871008000000001 2024-09-20T22:49:17.960 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.960 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.960 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:49:17.961 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.240155+0000", 2024-09-20T22:49:17.961 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.1790999999999999e-05 2024-09-20T22:49:17.961 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:49:17.961 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:49:17.961 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:49:17.961 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.961 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.961 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:3232 7.16 e337/260 7:6885a65f:::smithi08477203-14:head v 337'107, mlcod=331'104)", 2024-09-20T22:49:17.961 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:46:57.231995+0000", 2024-09-20T22:49:17.961 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 140.661114467, 2024-09-20T22:49:17.961 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0090885140000000007, 2024-09-20T22:49:17.961 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:49:17.961 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:49:17.961 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:49:17.962 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:49:17.962 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.962 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:49:17.962 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.231995+0000", 2024-09-20T22:49:17.962 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:49:17.962 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.962 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.962 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:49:17.962 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.231995+0000", 2024-09-20T22:49:17.962 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:49:17.962 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.962 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.962 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:49:17.962 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.231996+0000", 2024-09-20T22:49:17.963 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.5799999999999998e-07 2024-09-20T22:49:17.963 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.963 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.963 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:49:17.963 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.234590+0000", 2024-09-20T22:49:17.963 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.002594482 2024-09-20T22:49:17.963 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.963 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.963 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:49:17.963 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.234617+0000", 2024-09-20T22:49:17.963 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.6855e-05 2024-09-20T22:49:17.963 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.963 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.963 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:49:17.964 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.234636+0000", 2024-09-20T22:49:17.964 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.8289000000000001e-05 2024-09-20T22:49:17.964 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.964 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.964 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:49:17.964 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.234729+0000", 2024-09-20T22:49:17.964 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.3207000000000004e-05 2024-09-20T22:49:17.964 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.964 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.964 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:49:17.964 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.234758+0000", 2024-09-20T22:49:17.964 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.8994999999999999e-05 2024-09-20T22:49:17.964 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.964 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.964 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:49:17.965 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.241031+0000", 2024-09-20T22:49:17.965 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0062733279999999999 2024-09-20T22:49:17.965 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.965 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.965 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:49:17.965 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.241084+0000", 2024-09-20T22:49:17.965 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.2599999999999998e-05 2024-09-20T22:49:17.965 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:49:17.965 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:49:17.965 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:49:17.965 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.965 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.965 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:3589 7.b e363/94 7:dc95255e:::smithi08477203-1:head v 363'419, mlcod=363'417)", 2024-09-20T22:49:17.965 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:47:23.205271+0000", 2024-09-20T22:49:17.966 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 114.68783843999999, 2024-09-20T22:49:17.966 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.014301911, 2024-09-20T22:49:17.966 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:49:17.966 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:49:17.966 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:49:17.966 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:49:17.966 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.966 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:49:17.966 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:23.205271+0000", 2024-09-20T22:49:17.966 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:49:17.966 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.966 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.966 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:49:17.966 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:23.205271+0000", 2024-09-20T22:49:17.966 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:49:17.967 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.967 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.967 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:49:17.967 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:23.205273+0000", 2024-09-20T22:49:17.967 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.409e-06 2024-09-20T22:49:17.967 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.967 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.967 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:49:17.968 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-20T22:49:17.968 INFO:teuthology.orchestra.run.smithi084.stdout: "success": "" 2024-09-20T22:49:17.968 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-20T22:49:17.968 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:23.210044+0000", 2024-09-20T22:49:17.968 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0047712609999999997 2024-09-20T22:49:17.968 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.969 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.969 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:49:17.969 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:23.210064+0000", 2024-09-20T22:49:17.969 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.0421e-05 2024-09-20T22:49:17.969 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.969 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.969 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:49:17.969 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:23.210074+0000", 2024-09-20T22:49:17.969 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.4329999999999993e-06 2024-09-20T22:49:17.969 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.969 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.969 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:49:17.969 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:23.210130+0000", 2024-09-20T22:49:17.969 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.6286000000000003e-05 2024-09-20T22:49:17.970 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.970 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.970 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:49:17.970 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:23.210148+0000", 2024-09-20T22:49:17.970 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.7895999999999999e-05 2024-09-20T22:49:17.970 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.970 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.970 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:49:17.970 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:23.219532+0000", 2024-09-20T22:49:17.970 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.009383677 2024-09-20T22:49:17.970 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.970 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.970 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:49:17.970 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:23.219573+0000", 2024-09-20T22:49:17.971 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.1527999999999999e-05 2024-09-20T22:49:17.971 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:49:17.971 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:49:17.971 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:49:17.971 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.971 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.971 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:3760 7.13 e383/260 7:cae5792a:::smithi08477203-32:head v 383'78, mlcod=377'74)", 2024-09-20T22:49:17.971 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:47:43.291657+0000", 2024-09-20T22:49:17.971 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 94.601452562000006, 2024-09-20T22:49:17.971 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.011377195, 2024-09-20T22:49:17.971 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:49:17.971 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:49:17.971 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:49:17.971 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:49:17.972 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.972 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:49:17.972 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.291657+0000", 2024-09-20T22:49:17.972 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:49:17.972 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.972 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.972 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:49:17.972 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.291657+0000", 2024-09-20T22:49:17.972 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:49:17.972 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.972 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.972 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:49:17.972 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.291658+0000", 2024-09-20T22:49:17.972 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1009999999999999e-06 2024-09-20T22:49:17.972 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.973 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.973 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:49:17.973 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.296704+0000", 2024-09-20T22:49:17.973 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0050458070000000002 2024-09-20T22:49:17.973 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.973 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.973 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:49:17.973 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.296716+0000", 2024-09-20T22:49:17.973 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1525000000000001e-05 2024-09-20T22:49:17.973 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.973 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.973 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:49:17.974 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.296721+0000", 2024-09-20T22:49:17.974 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.8379999999999998e-06 2024-09-20T22:49:17.974 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.974 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.974 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:49:17.974 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.296769+0000", 2024-09-20T22:49:17.974 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.7120000000000003e-05 2024-09-20T22:49:17.974 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.974 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.974 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:49:17.974 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.296786+0000", 2024-09-20T22:49:17.974 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.7374e-05 2024-09-20T22:49:17.974 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.974 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.975 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:49:17.975 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.302998+0000", 2024-09-20T22:49:17.975 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0062118219999999997 2024-09-20T22:49:17.975 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.975 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.975 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:49:17.975 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.303034+0000", 2024-09-20T22:49:17.975 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.6607999999999999e-05 2024-09-20T22:49:17.975 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:49:17.975 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:49:17.975 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:49:17.975 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.975 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.975 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:4444 7.0 7:05bb442d:::smithi08477203-24:head [copy-from ver 421] snapc b8=[b8,b7,b6,b5,b4,b3,b2,b1,b0,ae,a9,a7,a6,a5,9e,9c,96,95,87,86] ondisk+write+known_if_redirected+supports_pool_eio e447)", 2024-09-20T22:49:17.976 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:48:47.227411+0000", 2024-09-20T22:49:17.976 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 30.665698996, 2024-09-20T22:49:17.976 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.033566952999999997, 2024-09-20T22:49:17.976 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:49:17.976 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:49:17.976 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:49:17.976 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:49:17.976 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:49:17.976 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:49:17.976 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 4444 2024-09-20T22:49:17.976 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.976 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:49:17.976 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.976 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:49:17.976 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.227411+0000", 2024-09-20T22:49:17.977 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:49:17.977 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.977 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.977 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:49:17.977 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.227411+0000", 2024-09-20T22:49:17.977 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:49:17.977 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.977 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.977 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:49:17.977 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.227413+0000", 2024-09-20T22:49:17.977 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.057e-06 2024-09-20T22:49:17.977 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.977 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.977 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:49:17.977 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.227422+0000", 2024-09-20T22:49:17.978 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.3219999999999999e-06 2024-09-20T22:49:17.978 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.978 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.978 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:49:17.978 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.227442+0000", 2024-09-20T22:49:17.978 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9449999999999998e-05 2024-09-20T22:49:17.978 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.978 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.978 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:49:17.978 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.227452+0000", 2024-09-20T22:49:17.978 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0699999999999999e-05 2024-09-20T22:49:17.978 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.978 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.978 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:49:17.979 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.227531+0000", 2024-09-20T22:49:17.979 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.8993999999999996e-05 2024-09-20T22:49:17.979 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.979 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.979 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op must wait for map", 2024-09-20T22:49:17.979 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.227542+0000", 2024-09-20T22:49:17.979 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0797e-05 2024-09-20T22:49:17.979 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.979 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.979 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:49:17.979 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.229501+0000", 2024-09-20T22:49:17.979 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0019594080000000002 2024-09-20T22:49:17.979 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.979 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.979 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:49:17.980 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.229616+0000", 2024-09-20T22:49:17.980 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000114577 2024-09-20T22:49:17.980 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.980 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.980 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 1,3", 2024-09-20T22:49:17.980 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.241752+0000", 2024-09-20T22:49:17.980 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.012135920999999999 2024-09-20T22:49:17.980 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.980 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.980 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:49:17.980 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.249601+0000", 2024-09-20T22:49:17.980 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0078493120000000006 2024-09-20T22:49:17.980 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.980 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.981 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:49:17.981 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.260731+0000", 2024-09-20T22:49:17.981 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.011129781 2024-09-20T22:49:17.981 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.981 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.981 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:49:17.981 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.260861+0000", 2024-09-20T22:49:17.981 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00013027899999999999 2024-09-20T22:49:17.981 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.981 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.981 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:49:17.981 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.260913+0000", 2024-09-20T22:49:17.981 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.2172999999999997e-05 2024-09-20T22:49:17.981 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.981 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.982 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:49:17.982 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.260978+0000", 2024-09-20T22:49:17.982 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.4182000000000003e-05 2024-09-20T22:49:17.982 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:49:17.982 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:49:17.982 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:49:17.982 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.982 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.982 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:4445 7.0 7:05bb442d:::smithi08477203-24:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e447)", 2024-09-20T22:49:17.982 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:48:47.227478+0000", 2024-09-20T22:49:17.982 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 30.665632059, 2024-09-20T22:49:17.982 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.019465758999999999, 2024-09-20T22:49:17.982 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:49:17.982 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:49:17.983 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:49:17.983 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:49:17.983 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:49:17.983 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:49:17.983 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 4445 2024-09-20T22:49:17.983 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.983 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:49:17.983 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.983 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:49:17.983 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.227478+0000", 2024-09-20T22:49:17.983 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:49:17.983 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.983 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.983 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:49:17.984 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.227478+0000", 2024-09-20T22:49:17.984 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:49:17.984 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.984 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.984 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:49:17.984 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.227478+0000", 2024-09-20T22:49:17.984 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.2200000000000004e-07 2024-09-20T22:49:17.984 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.984 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.984 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:49:17.984 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.227481+0000", 2024-09-20T22:49:17.984 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.0060000000000001e-06 2024-09-20T22:49:17.984 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.984 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.984 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:49:17.985 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.227500+0000", 2024-09-20T22:49:17.985 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9071000000000001e-05 2024-09-20T22:49:17.985 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.985 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.985 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:49:17.985 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.227507+0000", 2024-09-20T22:49:17.985 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.1779999999999997e-06 2024-09-20T22:49:17.985 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.985 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.985 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:49:17.985 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.227628+0000", 2024-09-20T22:49:17.985 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000120627 2024-09-20T22:49:17.985 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.985 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.985 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting_for_map not empty", 2024-09-20T22:49:17.986 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.227633+0000", 2024-09-20T22:49:17.986 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.4730000000000002e-06 2024-09-20T22:49:17.986 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.986 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.986 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:49:17.986 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.229766+0000", 2024-09-20T22:49:17.986 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0021332339999999999 2024-09-20T22:49:17.986 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.986 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.986 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for blocked object", 2024-09-20T22:49:17.986 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.229814+0000", 2024-09-20T22:49:17.986 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.7746e-05 2024-09-20T22:49:17.986 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.986 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.987 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:49:17.987 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.246707+0000", 2024-09-20T22:49:17.987 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.016893542000000001 2024-09-20T22:49:17.987 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.987 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.987 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:49:17.987 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.246863+0000", 2024-09-20T22:49:17.987 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000156105 2024-09-20T22:49:17.987 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.987 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.987 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:49:17.987 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.246943+0000", 2024-09-20T22:49:17.987 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.0154999999999994e-05 2024-09-20T22:49:17.987 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:49:17.988 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:49:17.988 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:49:17.988 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.988 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.988 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:4634 7.0 7:05bb442d:::smithi08477203-24:head [write 756480~733012,stat] snapc c2=[c2,c0,bf,be,bc,bb,ba,b9,b8,b7,b4,b2,b1,a7,a5,9e,9c,95,87,86] ondisk+write+known_if_redirected+supports_pool_eio e468)", 2024-09-20T22:49:17.988 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:49:08.228503+0000", 2024-09-20T22:49:17.988 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 9.6646068229999997, 2024-09-20T22:49:17.988 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0080592119999999996, 2024-09-20T22:49:17.988 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:49:17.988 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:49:17.988 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:49:17.988 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:49:17.988 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:49:17.988 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:49:17.988 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 4634 2024-09-20T22:49:17.989 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.989 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:49:17.989 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.989 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:49:17.989 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:08.228503+0000", 2024-09-20T22:49:17.989 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:49:17.989 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.989 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.989 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:49:17.989 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:08.228503+0000", 2024-09-20T22:49:17.989 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:49:17.989 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.989 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.989 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:49:17.990 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:08.228504+0000", 2024-09-20T22:49:17.990 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.5820000000000001e-06 2024-09-20T22:49:17.990 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.990 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.990 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:49:17.990 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:08.229652+0000", 2024-09-20T22:49:17.990 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0011480189999999999 2024-09-20T22:49:17.990 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.990 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.990 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:49:17.990 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:08.229667+0000", 2024-09-20T22:49:17.990 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4327e-05 2024-09-20T22:49:17.990 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.990 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.990 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:49:17.991 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:08.229674+0000", 2024-09-20T22:49:17.991 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.6660000000000008e-06 2024-09-20T22:49:17.991 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.991 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.991 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:49:17.991 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:08.229727+0000", 2024-09-20T22:49:17.991 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.2967000000000003e-05 2024-09-20T22:49:17.991 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.991 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.991 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:49:17.991 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:08.229808+0000", 2024-09-20T22:49:17.991 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.0370000000000005e-05 2024-09-20T22:49:17.991 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.991 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.992 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 1,3", 2024-09-20T22:49:17.992 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:08.229976+0000", 2024-09-20T22:49:17.992 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000168219 2024-09-20T22:49:17.992 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.992 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.992 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:49:17.992 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:08.232075+0000", 2024-09-20T22:49:17.992 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.002099343 2024-09-20T22:49:17.992 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.992 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.992 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:49:17.992 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:08.235985+0000", 2024-09-20T22:49:17.992 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0039099490000000002 2024-09-20T22:49:17.992 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.993 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.993 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:49:17.993 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:08.236474+0000", 2024-09-20T22:49:17.993 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00048882000000000005 2024-09-20T22:49:17.993 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.993 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.993 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:49:17.993 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:08.236528+0000", 2024-09-20T22:49:17.993 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.4141000000000002e-05 2024-09-20T22:49:17.993 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.993 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.993 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:49:17.993 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:08.236562+0000", 2024-09-20T22:49:17.993 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.3809000000000001e-05 2024-09-20T22:49:17.993 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:49:17.994 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:49:17.994 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:49:17.994 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.994 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.994 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:4637 7.0 7:05bb442d:::smithi08477203-24:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e468)", 2024-09-20T22:49:17.994 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:49:08.238143+0000", 2024-09-20T22:49:17.994 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 9.6549667929999998, 2024-09-20T22:49:17.994 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0078299709999999998, 2024-09-20T22:49:17.994 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:49:17.994 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:49:17.994 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:49:17.994 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:49:17.994 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:49:17.994 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:49:17.995 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 4637 2024-09-20T22:49:17.995 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.995 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:49:17.995 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.995 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:49:17.995 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:08.238143+0000", 2024-09-20T22:49:17.995 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:49:17.995 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.995 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.995 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:49:17.995 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:08.238143+0000", 2024-09-20T22:49:17.995 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:49:17.995 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.995 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.996 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:49:17.996 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:08.238144+0000", 2024-09-20T22:49:17.996 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.4000000000000001e-07 2024-09-20T22:49:17.996 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.996 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.996 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:49:17.996 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:08.238147+0000", 2024-09-20T22:49:17.996 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.179e-06 2024-09-20T22:49:17.996 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.996 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.996 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:49:17.996 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:08.238155+0000", 2024-09-20T22:49:17.996 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.7710000000000006e-06 2024-09-20T22:49:17.996 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.996 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.997 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:49:17.997 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:08.238162+0000", 2024-09-20T22:49:17.997 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.9369999999999998e-06 2024-09-20T22:49:17.997 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.997 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.997 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:49:17.997 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:08.239897+0000", 2024-09-20T22:49:17.997 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0017351350000000001 2024-09-20T22:49:17.997 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.997 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.997 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for rw locks", 2024-09-20T22:49:17.997 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:08.239993+0000", 2024-09-20T22:49:17.997 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.6043000000000002e-05 2024-09-20T22:49:17.997 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.997 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.998 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:49:17.998 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:08.243952+0000", 2024-09-20T22:49:17.998 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0039592810000000003 2024-09-20T22:49:17.998 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.998 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.998 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:49:17.998 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:08.244036+0000", 2024-09-20T22:49:17.998 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.3968000000000007e-05 2024-09-20T22:49:17.998 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:17.998 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:17.998 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:49:17.998 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:08.245973+0000", 2024-09-20T22:49:17.998 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001936917 2024-09-20T22:49:17.998 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:49:17.999 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:49:17.999 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:49:17.999 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:49:17.999 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:49:17.999 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-20T22:49:17.999 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:49:18.021 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:18.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:18.022 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:49:18.122 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:49:18.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:18.145 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:18.146 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:49:18.220 INFO:tasks.rados.rados.0.smithi084.stdout:1865: done (4 left) 2024-09-20T22:49:18.220 INFO:tasks.rados.rados.0.smithi084.stdout:1866: done (3 left) 2024-09-20T22:49:18.220 INFO:tasks.rados.rados.0.smithi084.stdout:1867: done (2 left) 2024-09-20T22:49:18.220 INFO:tasks.rados.rados.0.smithi084.stdout:1868: done (1 left) 2024-09-20T22:49:18.220 INFO:tasks.rados.rados.0.smithi084.stdout:1870: done (0 left) 2024-09-20T22:49:18.220 INFO:tasks.rados.rados.0.smithi084.stdout:1871: read oid 38 snap 176 2024-09-20T22:49:18.221 INFO:tasks.rados.rados.0.smithi084.stdout:1871: expect (ObjNum 489 snap 137 seq_num 489) 2024-09-20T22:49:18.221 INFO:tasks.rados.rados.0.smithi084.stdout:1872: snap_remove snap 181 2024-09-20T22:49:18.246 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:49:18.267 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:18.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:18.268 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:49:18.369 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:49:18.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:18.391 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:18.391 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:49:18.492 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:49:18.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:18 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "17"}]: dispatch 2024-09-20T22:49:18.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:18 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "17"}]: dispatch 2024-09-20T22:49:18.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:18 smithi154 ceph-mon[33264]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:49:18.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:18 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "17"}]': finished 2024-09-20T22:49:18.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:18 smithi154 ceph-mon[33264]: osdmap e478: 8 total, 8 up, 8 in 2024-09-20T22:49:18.513 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:18.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:18.514 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:49:18.615 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:49:18 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: ::ffff:172.21.15.154 - - [20/Sep/2024:22:49:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-20T22:49:18.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:18 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "17"}]: dispatch 2024-09-20T22:49:18.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:18 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "17"}]: dispatch 2024-09-20T22:49:18.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:18 smithi084 ceph-mon[29774]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:49:18.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:18 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "17"}]': finished 2024-09-20T22:49:18.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:18 smithi084 ceph-mon[29774]: osdmap e478: 8 total, 8 up, 8 in 2024-09-20T22:49:18.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:18 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "17"}]: dispatch 2024-09-20T22:49:18.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:18 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "17"}]: dispatch 2024-09-20T22:49:18.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:18 smithi084 ceph-mon[40024]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:49:18.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:18 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "17"}]': finished 2024-09-20T22:49:18.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:18 smithi084 ceph-mon[40024]: osdmap e478: 8 total, 8 up, 8 in 2024-09-20T22:49:18.616 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:49:18.638 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:18.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:18.639 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:49:18.640 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-20T22:49:18.645 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-20T22:49:18.740 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:49:18.761 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:18.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:18.762 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:49:18.863 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:49:18.886 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:18.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:18.887 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:49:18.930 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:49:18.987 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:49:19.008 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:19.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:19.009 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:49:19.110 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:49:19.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:19.132 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:19.132 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:49:19.222 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:49:19.230 INFO:tasks.rados.rados.0.smithi084.stdout:1872: done (1 left) 2024-09-20T22:49:19.230 INFO:tasks.rados.rados.0.smithi084.stdout:1873: delete oid 14 current snap is 200 2024-09-20T22:49:19.230 INFO:tasks.rados.rados.0.smithi084.stdout:1871: expect (ObjNum 489 snap 137 seq_num 489) 2024-09-20T22:49:19.233 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:49:19.258 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:19.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:19.259 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:49:19.308 INFO:tasks.rados.rados.0.smithi084.stdout:1871: done (1 left) 2024-09-20T22:49:19.309 INFO:tasks.rados.rados.0.smithi084.stdout:1873: done (0 left) 2024-09-20T22:49:19.309 INFO:tasks.rados.rados.0.smithi084.stdout:1874: rollback oid 46 current snap is 200 2024-09-20T22:49:19.309 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 46 to 133 2024-09-20T22:49:19.309 INFO:tasks.rados.rados.0.smithi084.stdout:1875: snap_remove snap 185 2024-09-20T22:49:19.361 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:49:19.381 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-20T22:49:19.381 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [], 2024-09-20T22:49:19.381 INFO:teuthology.orchestra.run.smithi154.stdout: "num_ops": 0 2024-09-20T22:49:19.381 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-20T22:49:19.383 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:19.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:19.384 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:49:19.454 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:19 smithi084 ceph-mon[29774]: pgmap v618: 160 pgs: 2 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 155 active+clean; 375 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail 2024-09-20T22:49:19.454 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:19 smithi084 ceph-mon[29774]: osdmap e479: 8 total, 8 up, 8 in 2024-09-20T22:49:19.454 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:19 smithi084 ceph-mon[40024]: pgmap v618: 160 pgs: 2 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 155 active+clean; 375 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail 2024-09-20T22:49:19.454 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:19 smithi084 ceph-mon[40024]: osdmap e479: 8 total, 8 up, 8 in 2024-09-20T22:49:19.486 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:49:19.509 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:19.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:19.510 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:49:19.610 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:49:19.631 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:19.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:19.632 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:49:19.695 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:49:19.695 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:49:19.711 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:49:19.712 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:49:19.728 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:49:19.728 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:49:19.732 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:49:19.744 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-20T22:49:19.744 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T22:49:19.754 INFO:journalctl@ceph.iscsi.iscsi.a.smithi154.stdout:Sep 20 22:49:19 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-iscsi-iscsi-a[59730]: debug there is no tcmu-runner data available 2024-09-20T22:49:19.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:19 smithi154 ceph-mon[33264]: pgmap v618: 160 pgs: 2 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 155 active+clean; 375 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail 2024-09-20T22:49:19.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:19 smithi154 ceph-mon[33264]: osdmap e479: 8 total, 8 up, 8 in 2024-09-20T22:49:19.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:19.755 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:19.755 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:49:19.761 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:49:19.761 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:49:19.777 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:49:19.777 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:49:19.794 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:49:19.794 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:49:19.811 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:49:19.811 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:49:19.856 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:49:19.878 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:19.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:19.879 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:49:19.980 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:49:20.001 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:20.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:20.002 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:49:20.029 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.6 dump_blocked_ops 2024-09-20T22:49:20.103 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:49:20.125 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:20.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:20.125 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:49:20.226 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:49:20.232 INFO:tasks.rados.rados.0.smithi084.stdout:1875: done (1 left) 2024-09-20T22:49:20.232 INFO:tasks.rados.rados.0.smithi084.stdout:1876: read oid 6 snap -1 2024-09-20T22:49:20.232 INFO:tasks.rados.rados.0.smithi084.stdout:1876: expect deleted 2024-09-20T22:49:20.233 INFO:tasks.rados.rados.0.smithi084.stdout:1877: read oid 5 snap -1 2024-09-20T22:49:20.233 INFO:tasks.rados.rados.0.smithi084.stdout:1877: expect (ObjNum 653 snap 195 seq_num 653) 2024-09-20T22:49:20.233 INFO:tasks.rados.rados.0.smithi084.stdout:1878: snap_create 2024-09-20T22:49:20.233 INFO:tasks.rados.rados.0.smithi084.stdout:1874: finishing rollback tid 0 to smithi08477203-46 2024-09-20T22:49:20.233 INFO:tasks.rados.rados.0.smithi084.stdout:1874: finishing rollback tid 1 to smithi08477203-46 2024-09-20T22:49:20.233 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 46 v 764 (ObjNum 129 snap 0 seq_num 2091187616) dirty dne 2024-09-20T22:49:20.242 INFO:tasks.rados.rados.0.smithi084.stdout:1877: expect (ObjNum 653 snap 195 seq_num 653) 2024-09-20T22:49:20.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:20.254 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:20.255 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:49:20.321 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:49:20.355 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:49:20.376 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:20.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:20.377 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:49:20.478 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:49:20.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:20.500 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:20.500 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:49:20.590 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:20 smithi154 ceph-mon[33264]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:49:20.591 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:20 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/2992715439' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:49:20.591 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:20 smithi154 ceph-mon[33264]: osdmap e480: 8 total, 8 up, 8 in 2024-09-20T22:49:20.601 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:49:20.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:20 smithi084 ceph-mon[29774]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:49:20.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:20 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/2992715439' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:49:20.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:20 smithi084 ceph-mon[29774]: osdmap e480: 8 total, 8 up, 8 in 2024-09-20T22:49:20.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:20 smithi084 ceph-mon[40024]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:49:20.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:20 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/2992715439' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:49:20.617 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:20 smithi084 ceph-mon[40024]: osdmap e480: 8 total, 8 up, 8 in 2024-09-20T22:49:20.622 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:20.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:20.623 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:49:20.673 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd unset noscrub 2024-09-20T22:49:20.723 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:49:20.744 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:20.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:20.745 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:49:20.799 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-20T22:49:20.799 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:49:20.845 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:49:20.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:20.867 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:20.867 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:49:20.920 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:49:20.968 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:49:20.990 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:20.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:20.991 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:49:21.092 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:49:21.113 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:21.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:21.114 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:49:21.214 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:49:21.231 INFO:tasks.rados.rados.0.smithi084.stdout:1874: done (3 left) 2024-09-20T22:49:21.231 INFO:tasks.rados.rados.0.smithi084.stdout:1876: done (2 left) 2024-09-20T22:49:21.231 INFO:tasks.rados.rados.0.smithi084.stdout:1877: done (1 left) 2024-09-20T22:49:21.231 INFO:tasks.rados.rados.0.smithi084.stdout:1878: done (0 left) 2024-09-20T22:49:21.231 INFO:tasks.rados.rados.0.smithi084.stdout:1879: delete oid 7 current snap is 201 2024-09-20T22:49:21.235 INFO:tasks.rados.rados.0.smithi084.stdout:1879: done (0 left) 2024-09-20T22:49:21.236 INFO:tasks.rados.rados.0.smithi084.stdout:1880: write oid 31 current snap is 201 2024-09-20T22:49:21.236 INFO:tasks.rados.rados.0.smithi084.stdout:1880: seq_num 664 ranges {759848=488047,1989220=760224} 2024-09-20T22:49:21.238 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:21.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:21.239 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:49:21.243 INFO:tasks.rados.rados.0.smithi084.stdout:1880: writing smithi08477203-31 from 759848 to 1247895 tid 1 2024-09-20T22:49:21.251 INFO:tasks.rados.rados.0.smithi084.stdout:1880: writing smithi08477203-31 from 1989220 to 2749444 tid 2 2024-09-20T22:49:21.252 INFO:tasks.rados.rados.0.smithi084.stdout:1881: write oid 7 current snap is 201 2024-09-20T22:49:21.252 INFO:tasks.rados.rados.0.smithi084.stdout:1880: finishing write tid 1 to smithi08477203-31 2024-09-20T22:49:21.252 INFO:tasks.rados.rados.0.smithi084.stdout:1881: seq_num 665 ranges {776655=563296,2047702=620235,3095349=81306} 2024-09-20T22:49:21.259 INFO:tasks.rados.rados.0.smithi084.stdout:1881: writing smithi08477203-7 from 776655 to 1339951 tid 1 2024-09-20T22:49:21.266 INFO:tasks.rados.rados.0.smithi084.stdout:1881: writing smithi08477203-7 from 2047702 to 2667937 tid 2 2024-09-20T22:49:21.269 INFO:tasks.rados.rados.0.smithi084.stdout:1881: writing smithi08477203-7 from 3095349 to 3176655 tid 3 2024-09-20T22:49:21.270 INFO:tasks.rados.rados.0.smithi084.stdout:1882: snap_create 2024-09-20T22:49:21.270 INFO:tasks.rados.rados.0.smithi084.stdout:1880: finishing write tid 2 to smithi08477203-31 2024-09-20T22:49:21.270 INFO:tasks.rados.rados.0.smithi084.stdout:1880: finishing write tid 3 to smithi08477203-31 2024-09-20T22:49:21.270 INFO:tasks.rados.rados.0.smithi084.stdout:1880: finishing write tid 4 to smithi08477203-31 2024-09-20T22:49:21.270 INFO:tasks.rados.rados.0.smithi084.stdout:1880: oid 31 updating version 0 to 574 2024-09-20T22:49:21.270 INFO:tasks.rados.rados.0.smithi084.stdout:1880: oid 31 updating version 574 to 576 2024-09-20T22:49:21.270 INFO:tasks.rados.rados.0.smithi084.stdout:1880: oid 31 version 576 is already newer than 575 2024-09-20T22:49:21.270 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 31 v 576 (ObjNum 664 snap 201 seq_num 664) dirty exists 2024-09-20T22:49:21.270 INFO:tasks.rados.rados.0.smithi084.stdout:1880: left oid 31 (ObjNum 664 snap 201 seq_num 664) 2024-09-20T22:49:21.270 INFO:tasks.rados.rados.0.smithi084.stdout:1881: finishing write tid 1 to smithi08477203-7 2024-09-20T22:49:21.270 INFO:tasks.rados.rados.0.smithi084.stdout:1881: finishing write tid 2 to smithi08477203-7 2024-09-20T22:49:21.271 INFO:tasks.rados.rados.0.smithi084.stdout:1881: finishing write tid 3 to smithi08477203-7 2024-09-20T22:49:21.271 INFO:tasks.rados.rados.0.smithi084.stdout:1881: finishing write tid 4 to smithi08477203-7 2024-09-20T22:49:21.271 INFO:tasks.rados.rados.0.smithi084.stdout:1881: finishing write tid 5 to smithi08477203-7 2024-09-20T22:49:21.271 INFO:tasks.rados.rados.0.smithi084.stdout:1881: oid 7 updating version 0 to 847 2024-09-20T22:49:21.271 INFO:tasks.rados.rados.0.smithi084.stdout:1881: oid 7 version 847 is already newer than 845 2024-09-20T22:49:21.272 INFO:tasks.rados.rados.0.smithi084.stdout:1881: oid 7 version 847 is already newer than 844 2024-09-20T22:49:21.272 INFO:tasks.rados.rados.0.smithi084.stdout:1881: oid 7 version 847 is already newer than 846 2024-09-20T22:49:21.272 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 7 v 847 (ObjNum 665 snap 201 seq_num 665) dirty exists 2024-09-20T22:49:21.272 INFO:tasks.rados.rados.0.smithi084.stdout:1881: left oid 7 (ObjNum 665 snap 201 seq_num 665) 2024-09-20T22:49:21.340 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:49:21.361 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:21.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:21.362 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:49:21.463 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:49:21.484 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:21.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:21.485 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:49:21.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:21 smithi154 ceph-mon[33264]: pgmap v621: 160 pgs: 2 peering, 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 415 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 9.5 MiB/s rd, 2.8 MiB/s wr, 18 op/s 2024-09-20T22:49:21.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:21 smithi154 ceph-mon[33264]: osdmap e481: 8 total, 8 up, 8 in 2024-09-20T22:49:21.537 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.6 dump_historic_ops 2024-09-20T22:49:21.586 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:49:21.608 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:21.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:21.609 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:49:21.614 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:21 smithi084 ceph-mon[29774]: pgmap v621: 160 pgs: 2 peering, 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 415 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 9.5 MiB/s rd, 2.8 MiB/s wr, 18 op/s 2024-09-20T22:49:21.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:21 smithi084 ceph-mon[29774]: osdmap e481: 8 total, 8 up, 8 in 2024-09-20T22:49:21.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:21 smithi084 ceph-mon[40024]: pgmap v621: 160 pgs: 2 peering, 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 415 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 9.5 MiB/s rd, 2.8 MiB/s wr, 18 op/s 2024-09-20T22:49:21.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:21 smithi084 ceph-mon[40024]: osdmap e481: 8 total, 8 up, 8 in 2024-09-20T22:49:21.711 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:49:21.732 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:21.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:21.733 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:49:21.820 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:49:21.834 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:49:21.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:21.857 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:21.857 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:49:21.958 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:49:21.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:21.981 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:21.982 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:49:22.083 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:49:22.106 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:22.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:22.107 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:49:22.208 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:49:22.227 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is unset 2024-09-20T22:49:22.230 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:22.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:22.231 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:49:22.231 INFO:tasks.rados.rados.0.smithi084.stdout:1880: done (2 left) 2024-09-20T22:49:22.231 INFO:tasks.rados.rados.0.smithi084.stdout:1881: done (1 left) 2024-09-20T22:49:22.232 INFO:tasks.rados.rados.0.smithi084.stdout:1882: done (0 left) 2024-09-20T22:49:22.232 INFO:tasks.rados.rados.0.smithi084.stdout:1883: read oid 10 snap 193 2024-09-20T22:49:22.232 INFO:tasks.rados.rados.0.smithi084.stdout:1883: expect (ObjNum 575 snap 172 seq_num 575) 2024-09-20T22:49:22.232 INFO:tasks.rados.rados.0.smithi084.stdout:1884: copy_from oid 11 from oid 1 current snap is 202 2024-09-20T22:49:22.232 INFO:tasks.rados.rados.0.smithi084.stdout:1885: read oid 48 snap -1 2024-09-20T22:49:22.232 INFO:tasks.rados.rados.0.smithi084.stdout:1885: expect deleted 2024-09-20T22:49:22.232 INFO:tasks.rados.rados.0.smithi084.stdout:1886: snap_remove snap 199 2024-09-20T22:49:22.298 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-20T22:49:22.298 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:49:22.332 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:49:22.354 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:22.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:22.355 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:49:22.456 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:49:22.477 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:22.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:22.478 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:49:22.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:22 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/2449390404' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:49:22.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:22 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "16"}]: dispatch 2024-09-20T22:49:22.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:22 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "16"}]: dispatch 2024-09-20T22:49:22.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:22 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/2449390404' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T22:49:22.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:22 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "16"}]': finished 2024-09-20T22:49:22.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:22 smithi154 ceph-mon[33264]: osdmap e482: 8 total, 8 up, 8 in 2024-09-20T22:49:22.538 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-20T22:49:22.560 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:22 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/2449390404' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:49:22.560 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:22 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "16"}]: dispatch 2024-09-20T22:49:22.560 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:22 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "16"}]: dispatch 2024-09-20T22:49:22.560 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:22 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/2449390404' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T22:49:22.560 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:22 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "16"}]': finished 2024-09-20T22:49:22.560 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:22 smithi084 ceph-mon[40024]: osdmap e482: 8 total, 8 up, 8 in 2024-09-20T22:49:22.560 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:22 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/2449390404' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:49:22.560 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:22 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "16"}]: dispatch 2024-09-20T22:49:22.560 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:22 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "16"}]: dispatch 2024-09-20T22:49:22.560 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:22 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/2449390404' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T22:49:22.561 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:22 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "16"}]': finished 2024-09-20T22:49:22.561 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:22 smithi084 ceph-mon[29774]: osdmap e482: 8 total, 8 up, 8 in 2024-09-20T22:49:22.579 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:49:22.601 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:22.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:22.602 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:49:22.703 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:49:22.725 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:22.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:22.726 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:49:22.826 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:49:22.847 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:22.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:22.848 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:49:22.904 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-20T22:49:22.949 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:49:22.970 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:22.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:22.971 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:49:23.071 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:49:23.093 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:23.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:23.094 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:49:23.116 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:49:23.151 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:49:23.195 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:49:23.216 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:23.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:23.217 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:49:23.222 INFO:tasks.rados.rados.0.smithi084.stdout:1886: done (3 left) 2024-09-20T22:49:23.223 INFO:tasks.rados.rados.0.smithi084.stdout:1887: read oid 26 snap -1 2024-09-20T22:49:23.223 INFO:tasks.rados.rados.0.smithi084.stdout:1887: expect (ObjNum 642 snap 193 seq_num 642) 2024-09-20T22:49:23.223 INFO:tasks.rados.rados.0.smithi084.stdout:1888: read oid 29 snap -1 2024-09-20T22:49:23.223 INFO:tasks.rados.rados.0.smithi084.stdout:1888: expect (ObjNum 607 snap 181 seq_num 607) 2024-09-20T22:49:23.223 INFO:tasks.rados.rados.0.smithi084.stdout:1889: write oid 39 current snap is 202 2024-09-20T22:49:23.223 INFO:tasks.rados.rados.0.smithi084.stdout:1889: seq_num 666 ranges {793462=638545,2106184=763893} 2024-09-20T22:49:23.231 INFO:tasks.rados.rados.0.smithi084.stdout:1889: writing smithi08477203-39 from 793462 to 1432007 tid 1 2024-09-20T22:49:23.239 INFO:tasks.rados.rados.0.smithi084.stdout:1889: writing smithi08477203-39 from 2106184 to 2870077 tid 2 2024-09-20T22:49:23.239 INFO:tasks.rados.rados.0.smithi084.stdout:1890: snap_remove snap 201 2024-09-20T22:49:23.318 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:49:23.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:23.340 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:23.340 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:49:23.441 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:49:23.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:23.463 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:23.463 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:49:23.564 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:49:23.579 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:23 smithi154 ceph-mon[33264]: pgmap v624: 159 pgs: 1 peering, 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 415 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 55 MiB/s rd, 14 MiB/s wr, 98 op/s; 11 MiB/s, 26 objects/s recovering 2024-09-20T22:49:23.579 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:23 smithi154 ceph-mon[33264]: 3.f scrub starts 2024-09-20T22:49:23.579 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:23 smithi154 ceph-mon[33264]: 3.f scrub ok 2024-09-20T22:49:23.579 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:23 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:49:23.579 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:23 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:49:23.579 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:23 smithi154 ceph-mon[33264]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:49:23.580 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:23 smithi154 ceph-mon[33264]: osdmap e483: 8 total, 8 up, 8 in 2024-09-20T22:49:23.588 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:23.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:23.589 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:49:23.603 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:49:23.604 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:49:23.604 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:23 smithi084 ceph-mon[40024]: pgmap v624: 159 pgs: 1 peering, 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 415 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 55 MiB/s rd, 14 MiB/s wr, 98 op/s; 11 MiB/s, 26 objects/s recovering 2024-09-20T22:49:23.604 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:23 smithi084 ceph-mon[40024]: 3.f scrub starts 2024-09-20T22:49:23.604 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:23 smithi084 ceph-mon[40024]: 3.f scrub ok 2024-09-20T22:49:23.604 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:23 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:49:23.604 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:23 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:49:23.605 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:23 smithi084 ceph-mon[40024]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:49:23.605 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:23 smithi084 ceph-mon[40024]: osdmap e483: 8 total, 8 up, 8 in 2024-09-20T22:49:23.605 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:23 smithi084 ceph-mon[29774]: pgmap v624: 159 pgs: 1 peering, 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 415 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 55 MiB/s rd, 14 MiB/s wr, 98 op/s; 11 MiB/s, 26 objects/s recovering 2024-09-20T22:49:23.605 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:23 smithi084 ceph-mon[29774]: 3.f scrub starts 2024-09-20T22:49:23.605 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:23 smithi084 ceph-mon[29774]: 3.f scrub ok 2024-09-20T22:49:23.605 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:23 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:49:23.605 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:23 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:49:23.605 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:23 smithi084 ceph-mon[29774]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:49:23.605 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:23 smithi084 ceph-mon[29774]: osdmap e483: 8 total, 8 up, 8 in 2024-09-20T22:49:23.620 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:49:23.621 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:49:23.636 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:49:23.636 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:49:23.652 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-20T22:49:23.652 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T22:49:23.670 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:49:23.671 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:49:23.686 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:49:23.687 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:49:23.690 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:49:23.690 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-20T22:49:23.690 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:49:23.703 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:49:23.703 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:49:23.711 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:23.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:23.712 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:49:23.720 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:49:23.720 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:49:23.813 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:49:23.835 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:23.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:23.836 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:49:23.936 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:49:23.957 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:23.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:23.958 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:49:24.059 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:49:24.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:24.081 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:24.081 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:49:24.172 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 5, 6, 7, 0, 4] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-20T22:49:24.172 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-20T22:49:24.172 INFO:tasks.thrashosds.thrasher:inject_pause on osd.0 2024-09-20T22:49:24.172 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-20T22:49:24.172 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-20T22:49:24.173 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.0 config set filestore_inject_stall 3 2024-09-20T22:49:24.182 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:49:24.227 INFO:tasks.rados.rados.0.smithi084.stdout:1890: done (6 left) 2024-09-20T22:49:24.227 INFO:tasks.rados.rados.0.smithi084.stdout:1891: snap_remove snap 175 2024-09-20T22:49:24.240 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:24.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:24.241 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:49:24.341 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:49:24.363 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:24.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:24.364 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:49:24.400 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:24 smithi154 ceph-mon[33264]: 7.f scrub starts 2024-09-20T22:49:24.400 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:24 smithi154 ceph-mon[33264]: 7.f scrub ok 2024-09-20T22:49:24.400 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:24 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/1982320506' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:49:24.400 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:24 smithi154 ceph-mon[33264]: osdmap e484: 8 total, 8 up, 8 in 2024-09-20T22:49:24.427 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.7 dump_blocked_ops 2024-09-20T22:49:24.449 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:49:24.466 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:49:24.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:24.487 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:24.488 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:49:24.588 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:49:24.609 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:24.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:24.611 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:49:24.612 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:24 smithi084 ceph-mon[40024]: 7.f scrub starts 2024-09-20T22:49:24.612 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:24 smithi084 ceph-mon[40024]: 7.f scrub ok 2024-09-20T22:49:24.612 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:24 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/1982320506' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:49:24.612 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:24 smithi084 ceph-mon[40024]: osdmap e484: 8 total, 8 up, 8 in 2024-09-20T22:49:24.613 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:24 smithi084 ceph-mon[29774]: 7.f scrub starts 2024-09-20T22:49:24.613 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:24 smithi084 ceph-mon[29774]: 7.f scrub ok 2024-09-20T22:49:24.613 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:24 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/1982320506' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:49:24.613 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:24 smithi084 ceph-mon[29774]: osdmap e484: 8 total, 8 up, 8 in 2024-09-20T22:49:24.671 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:49:24.712 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:49:24.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:24.734 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:24.735 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:49:24.836 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:49:24.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:24.857 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:24.857 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:49:24.930 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-20T22:49:24.930 INFO:teuthology.orchestra.run.smithi084.stdout: "success": "" 2024-09-20T22:49:24.930 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-20T22:49:24.958 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:49:24.980 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:24.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:24.981 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:49:25.048 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-20T22:49:25.082 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:49:25.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:25.112 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:25.112 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:49:25.137 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-20T22:49:25.137 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [], 2024-09-20T22:49:25.137 INFO:teuthology.orchestra.run.smithi154.stdout: "complaint_time": 30, 2024-09-20T22:49:25.137 INFO:teuthology.orchestra.run.smithi154.stdout: "num_blocked_ops": 0 2024-09-20T22:49:25.137 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-20T22:49:25.213 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:49:25.227 INFO:tasks.rados.rados.0.smithi084.stdout:1891: done (6 left) 2024-09-20T22:49:25.227 INFO:tasks.rados.rados.0.smithi084.stdout:1892: snap_create 2024-09-20T22:49:25.227 INFO:tasks.rados.rados.0.smithi084.stdout:1884: finishing copy_from to smithi08477203-11 2024-09-20T22:49:25.227 INFO:tasks.rados.rados.0.smithi084.stdout:1884: got expected ENOENT (src dne) 2024-09-20T22:49:25.228 INFO:tasks.rados.rados.0.smithi084.stdout:1884: finishing copy_from racing read to smithi08477203-11 2024-09-20T22:49:25.228 INFO:tasks.rados.rados.0.smithi084.stdout:1883: expect (ObjNum 575 snap 172 seq_num 575) 2024-09-20T22:49:25.243 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:25.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:25.244 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:49:25.345 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:49:25.367 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:25.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:25.368 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:49:25.468 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:49:25.475 INFO:tasks.rados.rados.0.smithi084.stdout:1888: expect (ObjNum 607 snap 181 seq_num 607) 2024-09-20T22:49:25.482 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:25 smithi154 ceph-mon[33264]: 5.1e scrub starts 2024-09-20T22:49:25.482 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:25 smithi154 ceph-mon[33264]: 5.1e scrub ok 2024-09-20T22:49:25.483 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:25 smithi154 ceph-mon[33264]: pgmap v627: 159 pgs: 1 active+clean+snaptrim_wait, 2 clean+premerge+peered, 3 active+clean+snaptrim, 153 active+clean; 422 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 957 KiB/s wr, 14 op/s 2024-09-20T22:49:25.483 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:25 smithi154 ceph-mon[33264]: 3.e scrub starts 2024-09-20T22:49:25.483 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:25 smithi154 ceph-mon[33264]: 3.e scrub ok 2024-09-20T22:49:25.483 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:25 smithi154 ceph-mon[33264]: purged_snaps scrub starts 2024-09-20T22:49:25.483 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:25 smithi154 ceph-mon[33264]: purged_snaps scrub ok 2024-09-20T22:49:25.483 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:25 smithi154 ceph-mon[33264]: 4.12 scrub starts 2024-09-20T22:49:25.483 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:25 smithi154 ceph-mon[33264]: 4.12 scrub ok 2024-09-20T22:49:25.483 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:25 smithi154 ceph-mon[33264]: osdmap e485: 8 total, 8 up, 8 in 2024-09-20T22:49:25.489 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:25.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:25.490 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:49:25.591 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:49:25.611 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:25 smithi084 ceph-mon[29774]: 5.1e scrub starts 2024-09-20T22:49:25.611 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:25 smithi084 ceph-mon[29774]: 5.1e scrub ok 2024-09-20T22:49:25.611 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:25 smithi084 ceph-mon[29774]: pgmap v627: 159 pgs: 1 active+clean+snaptrim_wait, 2 clean+premerge+peered, 3 active+clean+snaptrim, 153 active+clean; 422 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 957 KiB/s wr, 14 op/s 2024-09-20T22:49:25.611 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:25 smithi084 ceph-mon[29774]: 3.e scrub starts 2024-09-20T22:49:25.611 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:25 smithi084 ceph-mon[29774]: 3.e scrub ok 2024-09-20T22:49:25.611 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:25 smithi084 ceph-mon[29774]: purged_snaps scrub starts 2024-09-20T22:49:25.611 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:25 smithi084 ceph-mon[29774]: purged_snaps scrub ok 2024-09-20T22:49:25.611 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:25 smithi084 ceph-mon[29774]: 4.12 scrub starts 2024-09-20T22:49:25.611 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:25 smithi084 ceph-mon[29774]: 4.12 scrub ok 2024-09-20T22:49:25.611 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:25 smithi084 ceph-mon[29774]: osdmap e485: 8 total, 8 up, 8 in 2024-09-20T22:49:25.612 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:25 smithi084 ceph-mon[40024]: 5.1e scrub starts 2024-09-20T22:49:25.612 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:25 smithi084 ceph-mon[40024]: 5.1e scrub ok 2024-09-20T22:49:25.612 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:25 smithi084 ceph-mon[40024]: pgmap v627: 159 pgs: 1 active+clean+snaptrim_wait, 2 clean+premerge+peered, 3 active+clean+snaptrim, 153 active+clean; 422 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 957 KiB/s wr, 14 op/s 2024-09-20T22:49:25.612 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:25 smithi084 ceph-mon[40024]: 3.e scrub starts 2024-09-20T22:49:25.612 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:25 smithi084 ceph-mon[40024]: 3.e scrub ok 2024-09-20T22:49:25.612 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:25 smithi084 ceph-mon[40024]: purged_snaps scrub starts 2024-09-20T22:49:25.612 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:25 smithi084 ceph-mon[40024]: purged_snaps scrub ok 2024-09-20T22:49:25.612 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:25 smithi084 ceph-mon[40024]: 4.12 scrub starts 2024-09-20T22:49:25.612 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:25 smithi084 ceph-mon[40024]: 4.12 scrub ok 2024-09-20T22:49:25.612 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:25 smithi084 ceph-mon[40024]: osdmap e485: 8 total, 8 up, 8 in 2024-09-20T22:49:25.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:25.613 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:25.613 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:49:25.692 INFO:tasks.rados.rados.0.smithi084.stdout:1889: finishing write tid 1 to smithi08477203-39 2024-09-20T22:49:25.692 INFO:tasks.rados.rados.0.smithi084.stdout:1887: expect (ObjNum 642 snap 193 seq_num 642) 2024-09-20T22:49:25.715 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:49:25.737 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:25.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:25.738 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:49:25.755 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:49:25.779 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.7 dump_historic_ops 2024-09-20T22:49:25.830 INFO:tasks.rados.rados.0.smithi084.stdout:1889: finishing write tid 2 to smithi08477203-39 2024-09-20T22:49:25.830 INFO:tasks.rados.rados.0.smithi084.stdout:1889: finishing write tid 3 to smithi08477203-39 2024-09-20T22:49:25.830 INFO:tasks.rados.rados.0.smithi084.stdout:1889: finishing write tid 4 to smithi08477203-39 2024-09-20T22:49:25.830 INFO:tasks.rados.rados.0.smithi084.stdout:1889: oid 39 updating version 0 to 680 2024-09-20T22:49:25.830 INFO:tasks.rados.rados.0.smithi084.stdout:1889: oid 39 updating version 680 to 682 2024-09-20T22:49:25.830 INFO:tasks.rados.rados.0.smithi084.stdout:1889: oid 39 version 682 is already newer than 681 2024-09-20T22:49:25.830 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 39 v 682 (ObjNum 666 snap 202 seq_num 666) dirty exists 2024-09-20T22:49:25.830 INFO:tasks.rados.rados.0.smithi084.stdout:1889: left oid 39 (ObjNum 666 snap 202 seq_num 666) 2024-09-20T22:49:25.839 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:49:25.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:25.860 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:25.861 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:49:25.961 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:49:25.984 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:25.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:25.985 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:49:26.024 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:49:26.086 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:49:26.108 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:26.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:26.109 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:49:26.210 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:49:26.224 INFO:tasks.rados.rados.0.smithi084.stdout:1883: done (6 left) 2024-09-20T22:49:26.224 INFO:tasks.rados.rados.0.smithi084.stdout:1884: done (5 left) 2024-09-20T22:49:26.224 INFO:tasks.rados.rados.0.smithi084.stdout:1885: done (4 left) 2024-09-20T22:49:26.224 INFO:tasks.rados.rados.0.smithi084.stdout:1887: done (3 left) 2024-09-20T22:49:26.225 INFO:tasks.rados.rados.0.smithi084.stdout:1888: done (2 left) 2024-09-20T22:49:26.225 INFO:tasks.rados.rados.0.smithi084.stdout:1889: done (1 left) 2024-09-20T22:49:26.225 INFO:tasks.rados.rados.0.smithi084.stdout:1892: done (0 left) 2024-09-20T22:49:26.225 INFO:tasks.rados.rados.0.smithi084.stdout:1893: snap_create 2024-09-20T22:49:26.232 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:26.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:26.233 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:49:26.334 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:49:26.371 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:26.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:26.372 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:49:26.411 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-20T22:49:26.473 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:49:26.495 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:26.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:26.496 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:49:26.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:26 smithi154 ceph-mon[33264]: 7.e scrub starts 2024-09-20T22:49:26.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:26 smithi154 ceph-mon[33264]: 7.e scrub ok 2024-09-20T22:49:26.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:26 smithi154 ceph-mon[33264]: 5.0 scrub starts 2024-09-20T22:49:26.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:26 smithi154 ceph-mon[33264]: 5.0 scrub ok 2024-09-20T22:49:26.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:26 smithi154 ceph-mon[33264]: 3.11 scrub starts 2024-09-20T22:49:26.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:26 smithi154 ceph-mon[33264]: 3.11 scrub ok 2024-09-20T22:49:26.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:26 smithi154 ceph-mon[33264]: 1.5 scrub starts 2024-09-20T22:49:26.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:26 smithi154 ceph-mon[33264]: 1.5 scrub ok 2024-09-20T22:49:26.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:26 smithi154 ceph-mon[33264]: 3.0 scrub starts 2024-09-20T22:49:26.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:26 smithi154 ceph-mon[33264]: 3.0 scrub ok 2024-09-20T22:49:26.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:26 smithi154 ceph-mon[33264]: osdmap e486: 8 total, 8 up, 8 in 2024-09-20T22:49:26.596 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:49:26.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:26 smithi084 ceph-mon[40024]: 7.e scrub starts 2024-09-20T22:49:26.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:26 smithi084 ceph-mon[40024]: 7.e scrub ok 2024-09-20T22:49:26.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:26 smithi084 ceph-mon[40024]: 5.0 scrub starts 2024-09-20T22:49:26.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:26 smithi084 ceph-mon[40024]: 5.0 scrub ok 2024-09-20T22:49:26.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:26 smithi084 ceph-mon[40024]: 3.11 scrub starts 2024-09-20T22:49:26.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:26 smithi084 ceph-mon[40024]: 3.11 scrub ok 2024-09-20T22:49:26.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:26 smithi084 ceph-mon[40024]: 1.5 scrub starts 2024-09-20T22:49:26.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:26 smithi084 ceph-mon[40024]: 1.5 scrub ok 2024-09-20T22:49:26.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:26 smithi084 ceph-mon[40024]: 3.0 scrub starts 2024-09-20T22:49:26.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:26 smithi084 ceph-mon[40024]: 3.0 scrub ok 2024-09-20T22:49:26.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:26 smithi084 ceph-mon[40024]: osdmap e486: 8 total, 8 up, 8 in 2024-09-20T22:49:26.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:26 smithi084 ceph-mon[29774]: 7.e scrub starts 2024-09-20T22:49:26.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:26 smithi084 ceph-mon[29774]: 7.e scrub ok 2024-09-20T22:49:26.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:26 smithi084 ceph-mon[29774]: 5.0 scrub starts 2024-09-20T22:49:26.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:26 smithi084 ceph-mon[29774]: 5.0 scrub ok 2024-09-20T22:49:26.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:26 smithi084 ceph-mon[29774]: 3.11 scrub starts 2024-09-20T22:49:26.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:26 smithi084 ceph-mon[29774]: 3.11 scrub ok 2024-09-20T22:49:26.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:26 smithi084 ceph-mon[29774]: 1.5 scrub starts 2024-09-20T22:49:26.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:26 smithi084 ceph-mon[29774]: 1.5 scrub ok 2024-09-20T22:49:26.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:26 smithi084 ceph-mon[29774]: 3.0 scrub starts 2024-09-20T22:49:26.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:26 smithi084 ceph-mon[29774]: 3.0 scrub ok 2024-09-20T22:49:26.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:26 smithi084 ceph-mon[29774]: osdmap e486: 8 total, 8 up, 8 in 2024-09-20T22:49:26.619 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:26.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:26.620 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:49:26.621 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-20T22:49:26.621 INFO:teuthology.orchestra.run.smithi154.stdout: "size": 20, 2024-09-20T22:49:26.621 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 600, 2024-09-20T22:49:26.621 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [ 2024-09-20T22:49:26.621 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.621 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:786 7.17 7:ed54065f:::smithi08477203-12:head [copy-from ver 61] snapc 19=[17] ondisk+write+known_if_redirected+supports_pool_eio e145)", 2024-09-20T22:49:26.622 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:44.333196+0000", 2024-09-20T22:49:26.622 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 342.28550558000001, 2024-09-20T22:49:26.622 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.023328503, 2024-09-20T22:49:26.622 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:49:26.622 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:49:26.622 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:49:26.622 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:49:26.622 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:49:26.622 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:49:26.622 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 786 2024-09-20T22:49:26.622 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.622 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:49:26.622 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.622 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:49:26.623 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.333196+0000", 2024-09-20T22:49:26.623 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:49:26.623 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.623 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.623 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:49:26.623 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.333196+0000", 2024-09-20T22:49:26.623 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:49:26.623 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.623 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.623 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:49:26.623 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.333197+0000", 2024-09-20T22:49:26.623 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.486e-06 2024-09-20T22:49:26.623 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.623 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.624 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:49:26.624 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.333207+0000", 2024-09-20T22:49:26.624 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0307e-05 2024-09-20T22:49:26.624 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.624 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.624 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:49:26.624 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.333220+0000", 2024-09-20T22:49:26.624 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.306e-05 2024-09-20T22:49:26.624 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.624 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.624 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:49:26.624 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.333229+0000", 2024-09-20T22:49:26.624 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.3450000000000006e-06 2024-09-20T22:49:26.625 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.625 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.625 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:49:26.625 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.333292+0000", 2024-09-20T22:49:26.625 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.3137000000000002e-05 2024-09-20T22:49:26.625 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.625 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.625 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op must wait for map", 2024-09-20T22:49:26.625 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.333298+0000", 2024-09-20T22:49:26.625 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.7780000000000004e-06 2024-09-20T22:49:26.625 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.625 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.625 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:49:26.625 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.336639+0000", 2024-09-20T22:49:26.626 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0033414479999999999 2024-09-20T22:49:26.626 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.626 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.626 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:49:26.626 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.336729+0000", 2024-09-20T22:49:26.626 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.9888000000000004e-05 2024-09-20T22:49:26.626 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.626 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.626 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 2,5", 2024-09-20T22:49:26.626 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.344876+0000", 2024-09-20T22:49:26.626 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0081473989999999996 2024-09-20T22:49:26.626 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.626 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.626 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:49:26.627 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.355138+0000", 2024-09-20T22:49:26.627 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.010261462000000001 2024-09-20T22:49:26.627 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.627 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.627 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:49:26.627 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.355972+0000", 2024-09-20T22:49:26.627 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00083377899999999999 2024-09-20T22:49:26.627 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.627 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.627 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:49:26.627 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.356451+0000", 2024-09-20T22:49:26.627 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00047985200000000001 2024-09-20T22:49:26.627 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.627 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.628 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:49:26.628 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.356486+0000", 2024-09-20T22:49:26.628 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.4653000000000002e-05 2024-09-20T22:49:26.628 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.628 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.628 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:49:26.628 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.356524+0000", 2024-09-20T22:49:26.628 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.7908999999999999e-05 2024-09-20T22:49:26.628 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:49:26.628 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:49:26.628 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:49:26.628 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.628 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.628 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:787 7.17 7:ed54065f:::smithi08477203-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e145)", 2024-09-20T22:49:26.628 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:44.333248+0000", 2024-09-20T22:49:26.629 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 342.28545348500001, 2024-09-20T22:49:26.629 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.016881184, 2024-09-20T22:49:26.629 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:49:26.629 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:49:26.629 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:49:26.629 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:49:26.629 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:49:26.629 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:49:26.629 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 787 2024-09-20T22:49:26.629 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.629 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:49:26.629 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.629 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:49:26.629 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.333248+0000", 2024-09-20T22:49:26.630 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:49:26.630 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.630 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.630 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:49:26.630 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.333248+0000", 2024-09-20T22:49:26.630 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:49:26.630 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.630 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.630 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:49:26.630 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.333248+0000", 2024-09-20T22:49:26.630 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.1399999999999997e-07 2024-09-20T22:49:26.630 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.630 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.630 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:49:26.631 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.333250+0000", 2024-09-20T22:49:26.631 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.8199999999999999e-06 2024-09-20T22:49:26.631 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.631 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.631 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:49:26.631 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.333257+0000", 2024-09-20T22:49:26.631 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.43e-06 2024-09-20T22:49:26.631 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.631 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.631 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:49:26.631 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.333261+0000", 2024-09-20T22:49:26.631 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.3670000000000001e-06 2024-09-20T22:49:26.631 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.631 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.632 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:49:26.632 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.333351+0000", 2024-09-20T22:49:26.632 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.0486000000000001e-05 2024-09-20T22:49:26.632 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.632 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.632 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting_for_map not empty", 2024-09-20T22:49:26.632 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.333357+0000", 2024-09-20T22:49:26.632 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.3520000000000003e-06 2024-09-20T22:49:26.632 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.632 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.632 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:49:26.632 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.336912+0000", 2024-09-20T22:49:26.632 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.003554612 2024-09-20T22:49:26.632 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.632 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.633 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for blocked object", 2024-09-20T22:49:26.633 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.336957+0000", 2024-09-20T22:49:26.633 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.4919000000000001e-05 2024-09-20T22:49:26.633 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.633 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.633 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:49:26.633 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.349989+0000", 2024-09-20T22:49:26.633 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.01303238 2024-09-20T22:49:26.633 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.633 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.633 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:49:26.633 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.350058+0000", 2024-09-20T22:49:26.633 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.8418999999999997e-05 2024-09-20T22:49:26.633 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.634 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.634 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:49:26.634 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.350129+0000", 2024-09-20T22:49:26.634 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.0984999999999999e-05 2024-09-20T22:49:26.634 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:49:26.634 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:49:26.634 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:49:26.634 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.634 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.634 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:804 7.17 7:ea720b85:::smithi08477203-26:head [copy-from ver 61] snapc 19=[17] ondisk+write+known_if_redirected+supports_pool_eio e145)", 2024-09-20T22:49:26.634 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:44.451241+0000", 2024-09-20T22:49:26.634 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 342.16745965000001, 2024-09-20T22:49:26.634 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.012865619999999999, 2024-09-20T22:49:26.634 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:49:26.634 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:49:26.635 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:49:26.635 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:49:26.635 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:49:26.635 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:49:26.636 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 804 2024-09-20T22:49:26.636 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.636 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:49:26.636 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.636 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:49:26.636 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.451241+0000", 2024-09-20T22:49:26.636 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:49:26.636 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.636 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.636 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:49:26.636 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.451241+0000", 2024-09-20T22:49:26.636 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:49:26.636 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.636 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.637 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:49:26.637 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.451244+0000", 2024-09-20T22:49:26.637 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.2689999999999998e-06 2024-09-20T22:49:26.637 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.637 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.637 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:49:26.637 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.451255+0000", 2024-09-20T22:49:26.637 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1151e-05 2024-09-20T22:49:26.637 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.637 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.637 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:49:26.637 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.451274+0000", 2024-09-20T22:49:26.637 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9389999999999999e-05 2024-09-20T22:49:26.637 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.638 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.638 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:49:26.638 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.451286+0000", 2024-09-20T22:49:26.638 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1703e-05 2024-09-20T22:49:26.638 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.638 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.638 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:49:26.638 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.451361+0000", 2024-09-20T22:49:26.638 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.5310999999999998e-05 2024-09-20T22:49:26.638 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.638 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.638 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:49:26.638 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.451462+0000", 2024-09-20T22:49:26.639 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000100809 2024-09-20T22:49:26.639 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.639 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.639 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 2,5", 2024-09-20T22:49:26.639 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.456603+0000", 2024-09-20T22:49:26.639 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0051411169999999997 2024-09-20T22:49:26.639 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.639 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.639 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:49:26.639 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.463017+0000", 2024-09-20T22:49:26.639 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.006413307 2024-09-20T22:49:26.639 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.639 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.639 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:49:26.640 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.463754+0000", 2024-09-20T22:49:26.640 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00073758000000000005 2024-09-20T22:49:26.640 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.640 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.640 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:49:26.640 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.464036+0000", 2024-09-20T22:49:26.640 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00028171000000000001 2024-09-20T22:49:26.640 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.640 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.640 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:49:26.640 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.464076+0000", 2024-09-20T22:49:26.640 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.9736e-05 2024-09-20T22:49:26.640 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.641 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.641 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:49:26.641 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.464107+0000", 2024-09-20T22:49:26.641 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.1537e-05 2024-09-20T22:49:26.641 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:49:26.641 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:49:26.641 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:49:26.641 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.641 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.641 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:1527 7.17 7:ea720b85:::smithi08477203-26:head [copy-from ver 83] snapc 33=[33] ondisk+write+known_if_redirected+supports_pool_eio e197)", 2024-09-20T22:49:26.641 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:44:36.198921+0000", 2024-09-20T22:49:26.641 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 290.41978032600002, 2024-09-20T22:49:26.641 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.021621703999999999, 2024-09-20T22:49:26.641 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:49:26.642 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:49:26.642 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:49:26.642 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:49:26.642 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:49:26.642 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:49:26.642 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 1527 2024-09-20T22:49:26.642 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.642 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:49:26.642 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.642 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:49:26.642 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.198921+0000", 2024-09-20T22:49:26.642 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:49:26.642 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.642 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.643 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:49:26.643 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.198921+0000", 2024-09-20T22:49:26.643 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:49:26.643 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.643 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.643 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:49:26.643 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.198923+0000", 2024-09-20T22:49:26.643 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.0080000000000001e-06 2024-09-20T22:49:26.643 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.643 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.643 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:49:26.643 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.198934+0000", 2024-09-20T22:49:26.643 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0923e-05 2024-09-20T22:49:26.643 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.644 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.644 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:49:26.644 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.198955+0000", 2024-09-20T22:49:26.644 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.0806e-05 2024-09-20T22:49:26.644 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.644 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.644 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:49:26.644 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.198966+0000", 2024-09-20T22:49:26.644 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1779e-05 2024-09-20T22:49:26.644 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.644 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.644 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:49:26.644 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.199051+0000", 2024-09-20T22:49:26.644 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.4357999999999994e-05 2024-09-20T22:49:26.644 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.645 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.645 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op must wait for map", 2024-09-20T22:49:26.645 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.199058+0000", 2024-09-20T22:49:26.645 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.8530000000000003e-06 2024-09-20T22:49:26.645 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.645 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.645 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:49:26.645 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.202420+0000", 2024-09-20T22:49:26.645 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0033623350000000002 2024-09-20T22:49:26.645 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.645 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.645 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:49:26.645 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.202498+0000", 2024-09-20T22:49:26.645 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.8316e-05 2024-09-20T22:49:26.645 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.646 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.646 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 2,5", 2024-09-20T22:49:26.646 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.209140+0000", 2024-09-20T22:49:26.646 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0066415759999999997 2024-09-20T22:49:26.646 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.646 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.646 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:49:26.646 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.214931+0000", 2024-09-20T22:49:26.646 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0057916520000000004 2024-09-20T22:49:26.646 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.646 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.646 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:49:26.646 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.220009+0000", 2024-09-20T22:49:26.646 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0050773750000000003 2024-09-20T22:49:26.647 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.647 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.647 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:49:26.647 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.220417+0000", 2024-09-20T22:49:26.647 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00040778599999999999 2024-09-20T22:49:26.647 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.647 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.647 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:49:26.647 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.220479+0000", 2024-09-20T22:49:26.647 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.1977000000000006e-05 2024-09-20T22:49:26.647 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.647 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.647 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:49:26.647 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.220542+0000", 2024-09-20T22:49:26.648 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.3960000000000004e-05 2024-09-20T22:49:26.648 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:49:26.648 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:49:26.648 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:49:26.648 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.648 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.648 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:1528 7.17 7:ea720b85:::smithi08477203-26:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e197)", 2024-09-20T22:49:26.648 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:44:36.198994+0000", 2024-09-20T22:49:26.648 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 290.41970759899999, 2024-09-20T22:49:26.648 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.015013924, 2024-09-20T22:49:26.648 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:49:26.648 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:49:26.648 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:49:26.648 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:49:26.649 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:49:26.649 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:49:26.649 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 1528 2024-09-20T22:49:26.649 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.649 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:49:26.649 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.649 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:49:26.649 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.198994+0000", 2024-09-20T22:49:26.649 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:49:26.649 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.649 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.649 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:49:26.649 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.198994+0000", 2024-09-20T22:49:26.649 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:49:26.649 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.650 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.650 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:49:26.650 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.198994+0000", 2024-09-20T22:49:26.650 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.8100000000000003e-07 2024-09-20T22:49:26.650 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.650 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.650 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:49:26.650 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.198996+0000", 2024-09-20T22:49:26.650 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.3e-06 2024-09-20T22:49:26.650 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.650 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.650 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:49:26.650 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.199008+0000", 2024-09-20T22:49:26.651 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1603999999999999e-05 2024-09-20T22:49:26.651 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.651 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.651 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:49:26.651 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.199013+0000", 2024-09-20T22:49:26.651 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.4500000000000003e-06 2024-09-20T22:49:26.651 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.651 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.651 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:49:26.651 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.199120+0000", 2024-09-20T22:49:26.651 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000106772 2024-09-20T22:49:26.651 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.651 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.651 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting_for_map not empty", 2024-09-20T22:49:26.652 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.199126+0000", 2024-09-20T22:49:26.652 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.8320000000000002e-06 2024-09-20T22:49:26.652 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.652 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.652 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:49:26.652 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.202625+0000", 2024-09-20T22:49:26.652 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0034986309999999999 2024-09-20T22:49:26.652 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.652 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.652 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for blocked object", 2024-09-20T22:49:26.652 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.202667+0000", 2024-09-20T22:49:26.652 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.2759000000000002e-05 2024-09-20T22:49:26.652 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.653 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.653 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:49:26.653 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.213787+0000", 2024-09-20T22:49:26.653 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.011119377 2024-09-20T22:49:26.653 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.653 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.653 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:49:26.653 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.213902+0000", 2024-09-20T22:49:26.653 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000115369 2024-09-20T22:49:26.653 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.653 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.653 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:49:26.653 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.214007+0000", 2024-09-20T22:49:26.653 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000105349 2024-09-20T22:49:26.654 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:49:26.654 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:49:26.654 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:49:26.654 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.654 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.654 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:1550 7.17 7:ed54065f:::smithi08477203-12:head [copy-from ver 88] snapc 34=[34,33] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2024-09-20T22:49:26.654 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:44:37.238376+0000", 2024-09-20T22:49:26.654 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 289.38032468699998, 2024-09-20T22:49:26.654 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.016634917999999999, 2024-09-20T22:49:26.654 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:49:26.654 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:49:26.654 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:49:26.654 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:49:26.654 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:49:26.655 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:49:26.655 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 1550 2024-09-20T22:49:26.655 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.655 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:49:26.655 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.655 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:49:26.655 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:37.238376+0000", 2024-09-20T22:49:26.655 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:49:26.655 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.655 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.655 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:49:26.655 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:37.238376+0000", 2024-09-20T22:49:26.655 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:49:26.655 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.656 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.656 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:49:26.656 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:37.238379+0000", 2024-09-20T22:49:26.656 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.7989999999999998e-06 2024-09-20T22:49:26.656 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.656 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.656 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:49:26.656 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:37.238408+0000", 2024-09-20T22:49:26.656 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.8980999999999999e-05 2024-09-20T22:49:26.656 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.656 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.656 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:49:26.656 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:37.238433+0000", 2024-09-20T22:49:26.656 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.5185999999999999e-05 2024-09-20T22:49:26.657 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.657 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.657 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:49:26.657 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:37.238446+0000", 2024-09-20T22:49:26.657 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2883e-05 2024-09-20T22:49:26.657 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.657 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.657 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:49:26.657 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:37.238523+0000", 2024-09-20T22:49:26.657 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.6396999999999994e-05 2024-09-20T22:49:26.657 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.657 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.657 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:49:26.657 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:37.238618+0000", 2024-09-20T22:49:26.657 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.5049000000000004e-05 2024-09-20T22:49:26.658 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.658 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.658 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 2,5", 2024-09-20T22:49:26.658 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:37.246556+0000", 2024-09-20T22:49:26.658 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0079378090000000005 2024-09-20T22:49:26.658 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.658 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.658 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:49:26.658 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:37.254025+0000", 2024-09-20T22:49:26.658 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0074697139999999997 2024-09-20T22:49:26.658 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.658 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.658 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:49:26.658 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:37.254033+0000", 2024-09-20T22:49:26.659 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.6979999999999998e-06 2024-09-20T22:49:26.659 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.659 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.659 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:49:26.659 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:37.254919+0000", 2024-09-20T22:49:26.659 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00088564700000000004 2024-09-20T22:49:26.659 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.659 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.659 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:49:26.659 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:37.254958+0000", 2024-09-20T22:49:26.659 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.9515999999999999e-05 2024-09-20T22:49:26.659 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.659 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.659 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:49:26.660 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:37.255011+0000", 2024-09-20T22:49:26.660 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.3239000000000003e-05 2024-09-20T22:49:26.660 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:49:26.660 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:49:26.660 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:49:26.660 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.660 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.660 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:1799 7.17 7:ea720b85:::smithi08477203-26:head [write 521083~714336,stat] snapc 46=[45,44,43,42,41,40,3e] ondisk+write+known_if_redirected+supports_pool_eio e229)", 2024-09-20T22:49:26.660 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:45:08.170663+0000", 2024-09-20T22:49:26.660 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 258.44803804200001, 2024-09-20T22:49:26.660 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.016368226999999999, 2024-09-20T22:49:26.660 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:49:26.660 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:49:26.660 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:49:26.661 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:49:26.661 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:49:26.661 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:49:26.661 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 1799 2024-09-20T22:49:26.661 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.661 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:49:26.661 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.661 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:49:26.661 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.170663+0000", 2024-09-20T22:49:26.661 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:49:26.661 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.661 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.661 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:49:26.661 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.170663+0000", 2024-09-20T22:49:26.662 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:49:26.662 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.662 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.662 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:49:26.662 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.170664+0000", 2024-09-20T22:49:26.662 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.085e-06 2024-09-20T22:49:26.662 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.662 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.662 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:49:26.662 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.171481+0000", 2024-09-20T22:49:26.662 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00081653900000000002 2024-09-20T22:49:26.662 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.662 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.662 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:49:26.663 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.171492+0000", 2024-09-20T22:49:26.663 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1565e-05 2024-09-20T22:49:26.663 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.663 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.663 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:49:26.663 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.171498+0000", 2024-09-20T22:49:26.663 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.925e-06 2024-09-20T22:49:26.663 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.663 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.663 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:49:26.663 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.171547+0000", 2024-09-20T22:49:26.663 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.8916999999999999e-05 2024-09-20T22:49:26.663 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.664 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.664 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:49:26.664 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.171611+0000", 2024-09-20T22:49:26.664 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.3838999999999996e-05 2024-09-20T22:49:26.664 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.664 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.664 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 2,5", 2024-09-20T22:49:26.664 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.171740+0000", 2024-09-20T22:49:26.664 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00012881500000000001 2024-09-20T22:49:26.664 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.664 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.664 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:49:26.664 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.173862+0000", 2024-09-20T22:49:26.664 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0021225649999999999 2024-09-20T22:49:26.664 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.665 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.665 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:49:26.665 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.179252+0000", 2024-09-20T22:49:26.665 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0053895490000000004 2024-09-20T22:49:26.665 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.665 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.665 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:49:26.665 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.186956+0000", 2024-09-20T22:49:26.665 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0077038050000000002 2024-09-20T22:49:26.665 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.665 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.665 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:49:26.665 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.186993+0000", 2024-09-20T22:49:26.665 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.7141000000000002e-05 2024-09-20T22:49:26.666 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.666 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.666 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:49:26.666 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.187031+0000", 2024-09-20T22:49:26.666 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.8482000000000002e-05 2024-09-20T22:49:26.666 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:49:26.666 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:49:26.666 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:49:26.666 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.666 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.666 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:1800 7.17 7:ea720b85:::smithi08477203-26:head [write 1724715~555421] snapc 46=[45,44,43,42,41,40,3e] ondisk+write+known_if_redirected+supports_pool_eio e229)", 2024-09-20T22:49:26.666 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:45:08.177237+0000", 2024-09-20T22:49:26.666 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 258.44146441700002, 2024-09-20T22:49:26.666 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.012771694, 2024-09-20T22:49:26.667 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:49:26.667 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:49:26.667 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:49:26.667 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:49:26.667 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:49:26.667 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:49:26.667 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 1800 2024-09-20T22:49:26.667 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.667 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:49:26.667 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.667 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:49:26.667 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.177237+0000", 2024-09-20T22:49:26.667 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:49:26.667 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.667 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.668 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:49:26.668 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.177237+0000", 2024-09-20T22:49:26.668 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:49:26.668 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.668 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.668 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:49:26.668 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.177238+0000", 2024-09-20T22:49:26.668 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.6080000000000001e-06 2024-09-20T22:49:26.668 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.668 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.668 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:49:26.668 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.177844+0000", 2024-09-20T22:49:26.668 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00060579999999999998 2024-09-20T22:49:26.668 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.669 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.669 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:49:26.669 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.177857+0000", 2024-09-20T22:49:26.669 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2529e-05 2024-09-20T22:49:26.669 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.669 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.669 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:49:26.669 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.177864+0000", 2024-09-20T22:49:26.669 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.2470000000000002e-06 2024-09-20T22:49:26.669 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.669 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.669 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:49:26.669 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.177908+0000", 2024-09-20T22:49:26.669 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.4329999999999997e-05 2024-09-20T22:49:26.669 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.670 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.670 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:49:26.670 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.177977+0000", 2024-09-20T22:49:26.670 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.8598999999999998e-05 2024-09-20T22:49:26.670 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.670 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.670 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 2,5", 2024-09-20T22:49:26.670 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.178079+0000", 2024-09-20T22:49:26.670 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000102074 2024-09-20T22:49:26.670 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.670 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.670 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:49:26.670 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.181043+0000", 2024-09-20T22:49:26.670 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0029639810000000001 2024-09-20T22:49:26.671 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.671 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.671 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:49:26.671 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.181481+0000", 2024-09-20T22:49:26.671 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00043855899999999999 2024-09-20T22:49:26.671 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.671 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.671 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:49:26.671 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.189928+0000", 2024-09-20T22:49:26.671 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0084463149999999994 2024-09-20T22:49:26.671 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.671 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.671 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:49:26.671 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.189968+0000", 2024-09-20T22:49:26.671 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.0268999999999997e-05 2024-09-20T22:49:26.672 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.672 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.672 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:49:26.672 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.190008+0000", 2024-09-20T22:49:26.672 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.0383000000000003e-05 2024-09-20T22:49:26.672 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:49:26.672 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:49:26.672 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:49:26.672 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.672 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.672 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:1802 7.17 7:ea720b85:::smithi08477203-26:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e229)", 2024-09-20T22:49:26.672 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:45:08.177901+0000", 2024-09-20T22:49:26.672 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 258.44080047300002, 2024-09-20T22:49:26.672 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.012470591, 2024-09-20T22:49:26.673 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:49:26.673 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:49:26.673 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:49:26.673 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:49:26.673 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:49:26.673 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:49:26.673 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 1802 2024-09-20T22:49:26.673 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.673 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:49:26.673 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.673 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:49:26.673 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.177901+0000", 2024-09-20T22:49:26.673 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:49:26.673 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.674 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.674 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:49:26.674 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.177901+0000", 2024-09-20T22:49:26.674 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:49:26.674 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.674 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.674 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:49:26.674 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.177901+0000", 2024-09-20T22:49:26.674 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.2899999999999999e-07 2024-09-20T22:49:26.674 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.674 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.674 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:49:26.674 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.177902+0000", 2024-09-20T22:49:26.674 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.452e-06 2024-09-20T22:49:26.675 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.675 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.675 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:49:26.675 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.177906+0000", 2024-09-20T22:49:26.675 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.788e-06 2024-09-20T22:49:26.675 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.675 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.675 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:49:26.675 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.177909+0000", 2024-09-20T22:49:26.675 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.9840000000000001e-06 2024-09-20T22:49:26.675 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.675 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.675 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:49:26.675 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.179283+0000", 2024-09-20T22:49:26.675 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001373733 2024-09-20T22:49:26.676 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.676 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.676 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for rw locks", 2024-09-20T22:49:26.676 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.179364+0000", 2024-09-20T22:49:26.676 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.1209000000000001e-05 2024-09-20T22:49:26.676 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.676 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.676 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:49:26.676 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.190140+0000", 2024-09-20T22:49:26.676 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.010775527 2024-09-20T22:49:26.677 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:49:26.678 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.678 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.678 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:49:26.678 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.190186+0000", 2024-09-20T22:49:26.678 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.5853000000000003e-05 2024-09-20T22:49:26.678 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.678 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.678 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:49:26.678 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.190371+0000", 2024-09-20T22:49:26.678 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000185716 2024-09-20T22:49:26.678 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:49:26.678 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:49:26.678 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:49:26.679 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.679 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.679 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:2318 7.5 7:a160b5d4:::smithi08477203-16:head [copy-from ver 168] snapc 55=[55,53,52,51,50,4b,47,3e] ondisk+write+known_if_redirected+supports_pool_eio e258)", 2024-09-20T22:49:26.679 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:45:37.175863+0000", 2024-09-20T22:49:26.679 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 229.442837967, 2024-09-20T22:49:26.679 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.030719883999999999, 2024-09-20T22:49:26.679 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:49:26.679 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:49:26.679 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:49:26.679 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:49:26.679 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:49:26.679 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:49:26.679 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 2318 2024-09-20T22:49:26.679 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.680 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:49:26.680 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.680 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:49:26.680 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.175863+0000", 2024-09-20T22:49:26.680 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:49:26.680 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.680 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.680 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:49:26.680 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.175863+0000", 2024-09-20T22:49:26.680 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:49:26.680 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.680 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.680 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:49:26.680 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.175865+0000", 2024-09-20T22:49:26.680 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.488e-06 2024-09-20T22:49:26.681 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.681 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.681 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:49:26.681 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.175871+0000", 2024-09-20T22:49:26.681 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.438e-06 2024-09-20T22:49:26.681 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.681 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.681 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:49:26.681 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.175881+0000", 2024-09-20T22:49:26.681 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.6430000000000007e-06 2024-09-20T22:49:26.681 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.681 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.681 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:49:26.681 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.175886+0000", 2024-09-20T22:49:26.682 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.3059999999999997e-06 2024-09-20T22:49:26.682 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.682 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.682 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:49:26.682 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.175944+0000", 2024-09-20T22:49:26.682 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.7982999999999997e-05 2024-09-20T22:49:26.682 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.682 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.682 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op must wait for map", 2024-09-20T22:49:26.682 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.175951+0000", 2024-09-20T22:49:26.682 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.9280000000000004e-06 2024-09-20T22:49:26.682 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.683 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.683 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:49:26.683 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.179794+0000", 2024-09-20T22:49:26.683 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0038435209999999999 2024-09-20T22:49:26.683 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.683 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.683 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:49:26.683 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.179940+0000", 2024-09-20T22:49:26.683 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00014550000000000001 2024-09-20T22:49:26.683 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.683 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.683 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 0,2", 2024-09-20T22:49:26.683 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.190992+0000", 2024-09-20T22:49:26.683 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.011051587999999999 2024-09-20T22:49:26.683 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.684 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.684 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:49:26.684 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.195070+0000", 2024-09-20T22:49:26.684 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0040784929999999999 2024-09-20T22:49:26.684 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.684 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.684 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:49:26.684 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.202908+0000", 2024-09-20T22:49:26.684 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0078376690000000002 2024-09-20T22:49:26.684 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.684 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.684 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:49:26.684 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.206517+0000", 2024-09-20T22:49:26.684 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0036088880000000002 2024-09-20T22:49:26.684 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.685 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.685 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:49:26.685 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.206553+0000", 2024-09-20T22:49:26.685 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.6363999999999999e-05 2024-09-20T22:49:26.685 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.685 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.685 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:49:26.685 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.206583+0000", 2024-09-20T22:49:26.685 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.0074999999999999e-05 2024-09-20T22:49:26.685 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:49:26.685 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:49:26.685 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:49:26.685 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.685 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.686 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:2319 7.5 7:a160b5d4:::smithi08477203-16:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e258)", 2024-09-20T22:49:26.686 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:45:37.175983+0000", 2024-09-20T22:49:26.686 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 229.44271821199999, 2024-09-20T22:49:26.686 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.018323547999999999, 2024-09-20T22:49:26.686 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:49:26.686 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:49:26.686 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:49:26.686 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:49:26.686 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:49:26.686 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:49:26.686 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 2319 2024-09-20T22:49:26.686 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.686 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:49:26.686 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.687 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:49:26.687 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.175983+0000", 2024-09-20T22:49:26.687 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:49:26.687 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.687 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.687 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:49:26.687 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.175983+0000", 2024-09-20T22:49:26.687 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:49:26.687 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.687 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.687 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:49:26.687 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.175984+0000", 2024-09-20T22:49:26.687 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.1099999999999995e-07 2024-09-20T22:49:26.687 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.687 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.688 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:49:26.688 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.175996+0000", 2024-09-20T22:49:26.688 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2121e-05 2024-09-20T22:49:26.688 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.688 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.688 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:49:26.688 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.176019+0000", 2024-09-20T22:49:26.688 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.3465999999999999e-05 2024-09-20T22:49:26.688 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.688 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.688 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:49:26.688 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.176026+0000", 2024-09-20T22:49:26.688 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.8190000000000003e-06 2024-09-20T22:49:26.688 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.689 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.689 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:49:26.689 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.176110+0000", 2024-09-20T22:49:26.689 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.3967000000000004e-05 2024-09-20T22:49:26.689 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.689 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.689 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting_for_map not empty", 2024-09-20T22:49:26.689 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.176116+0000", 2024-09-20T22:49:26.689 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.0249999999999999e-06 2024-09-20T22:49:26.689 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.689 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.689 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:49:26.689 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.180106+0000", 2024-09-20T22:49:26.689 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0039898980000000004 2024-09-20T22:49:26.689 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.690 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.690 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for blocked object", 2024-09-20T22:49:26.690 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.180135+0000", 2024-09-20T22:49:26.690 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.9595999999999999e-05 2024-09-20T22:49:26.690 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.690 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.690 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:49:26.690 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.194106+0000", 2024-09-20T22:49:26.690 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.013970761999999999 2024-09-20T22:49:26.690 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.690 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.690 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:49:26.690 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.194203+0000", 2024-09-20T22:49:26.690 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.7211000000000002e-05 2024-09-20T22:49:26.691 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.691 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.691 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:49:26.691 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.194306+0000", 2024-09-20T22:49:26.691 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00010307200000000001 2024-09-20T22:49:26.691 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:49:26.691 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:49:26.691 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:49:26.691 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.691 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.691 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24415.0:230 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch (1) in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e259)", 2024-09-20T22:49:26.691 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:45:39.207410+0000", 2024-09-20T22:49:26.691 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 227.41129114, 2024-09-20T22:49:26.691 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0072298289999999, 2024-09-20T22:49:26.691 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:49:26.692 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:49:26.692 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:49:26.692 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:49:26.692 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24415", 2024-09-20T22:49:26.692 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/2056244102", 2024-09-20T22:49:26.692 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 230 2024-09-20T22:49:26.692 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.692 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:49:26.692 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.692 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:49:26.692 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.207410+0000", 2024-09-20T22:49:26.692 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:49:26.692 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.692 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.693 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:49:26.693 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.207410+0000", 2024-09-20T22:49:26.693 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:49:26.693 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.693 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.693 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:49:26.693 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.207412+0000", 2024-09-20T22:49:26.693 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.7519999999999999e-06 2024-09-20T22:49:26.693 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.693 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.693 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:49:26.693 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.207419+0000", 2024-09-20T22:49:26.693 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.097e-06 2024-09-20T22:49:26.693 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.694 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.694 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:49:26.694 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.207432+0000", 2024-09-20T22:49:26.694 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2719e-05 2024-09-20T22:49:26.694 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.694 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.694 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:49:26.694 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.207464+0000", 2024-09-20T22:49:26.694 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.2820000000000001e-05 2024-09-20T22:49:26.694 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.694 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.694 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:49:26.694 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.207670+0000", 2024-09-20T22:49:26.694 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000205486 2024-09-20T22:49:26.695 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.695 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.695 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-20T22:49:26.695 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.207687+0000", 2024-09-20T22:49:26.695 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.7107000000000001e-05 2024-09-20T22:49:26.695 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.695 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.695 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:49:26.695 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:40.212672+0000", 2024-09-20T22:49:26.695 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.004984562 2024-09-20T22:49:26.695 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.695 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.695 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:49:26.695 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:40.213576+0000", 2024-09-20T22:49:26.696 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00090404799999999996 2024-09-20T22:49:26.696 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.696 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.696 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:49:26.696 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:40.214640+0000", 2024-09-20T22:49:26.696 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0010642379999999999 2024-09-20T22:49:26.696 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:49:26.696 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:49:26.696 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:49:26.696 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.696 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.696 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:2367 7.5 7:a160b5d4:::smithi08477203-16:head [stat,delete] snapc 56=[56,55,53,52,51,50,47,3e] ondisk+write+known_if_redirected+supports_pool_eio e260)", 2024-09-20T22:49:26.696 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:45:39.601299+0000", 2024-09-20T22:49:26.696 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 227.01740171899999, 2024-09-20T22:49:26.697 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.64714523300000004, 2024-09-20T22:49:26.697 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:49:26.697 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:49:26.697 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:49:26.697 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:49:26.697 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:49:26.697 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:49:26.697 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 2367 2024-09-20T22:49:26.697 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.697 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:49:26.697 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.697 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:49:26.697 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.601299+0000", 2024-09-20T22:49:26.697 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:49:26.698 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.698 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.698 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:49:26.698 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.601299+0000", 2024-09-20T22:49:26.698 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:49:26.698 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.698 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.698 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:49:26.698 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.601301+0000", 2024-09-20T22:49:26.698 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3850000000000001e-06 2024-09-20T22:49:26.698 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.698 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.698 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:49:26.698 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.601306+0000", 2024-09-20T22:49:26.699 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.2730000000000002e-06 2024-09-20T22:49:26.699 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.699 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.699 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:49:26.699 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.601318+0000", 2024-09-20T22:49:26.699 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2148e-05 2024-09-20T22:49:26.699 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.699 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.699 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:49:26.699 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.601324+0000", 2024-09-20T22:49:26.699 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.852e-06 2024-09-20T22:49:26.699 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.699 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.699 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:49:26.700 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.601383+0000", 2024-09-20T22:49:26.700 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.8570000000000003e-05 2024-09-20T22:49:26.700 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.700 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.700 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-20T22:49:26.700 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.601391+0000", 2024-09-20T22:49:26.700 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.5290000000000008e-06 2024-09-20T22:49:26.700 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.700 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.700 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:49:26.700 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:40.221227+0000", 2024-09-20T22:49:26.700 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.619835515 2024-09-20T22:49:26.700 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.700 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.701 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for degraded object", 2024-09-20T22:49:26.701 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:40.223919+0000", 2024-09-20T22:49:26.701 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.002692168 2024-09-20T22:49:26.701 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.701 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.701 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:49:26.701 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:40.244536+0000", 2024-09-20T22:49:26.701 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.020617613 2024-09-20T22:49:26.701 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.701 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.701 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:49:26.701 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:40.244653+0000", 2024-09-20T22:49:26.701 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000116796 2024-09-20T22:49:26.701 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.702 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.702 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 1,2", 2024-09-20T22:49:26.702 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:40.244832+0000", 2024-09-20T22:49:26.702 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00017833500000000001 2024-09-20T22:49:26.702 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.702 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.702 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:49:26.702 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:40.245494+0000", 2024-09-20T22:49:26.702 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00066241399999999995 2024-09-20T22:49:26.702 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.702 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.702 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:49:26.702 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:40.246501+0000", 2024-09-20T22:49:26.702 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0010066299999999999 2024-09-20T22:49:26.702 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.703 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.703 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:49:26.703 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:40.248326+0000", 2024-09-20T22:49:26.703 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001824895 2024-09-20T22:49:26.703 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.703 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.703 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:49:26.703 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:40.248370+0000", 2024-09-20T22:49:26.703 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.4298999999999997e-05 2024-09-20T22:49:26.703 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.703 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.703 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:49:26.703 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:40.248445+0000", 2024-09-20T22:49:26.703 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.4810999999999999e-05 2024-09-20T22:49:26.704 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:49:26.704 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:49:26.704 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:49:26.704 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.704 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.704 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:3132 7.19 e328/139 7:9eac4560:::smithi08477203-30:head v 328'198, mlcod=322'194)", 2024-09-20T22:49:26.704 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:46:48.223406+0000", 2024-09-20T22:49:26.704 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 158.395295283, 2024-09-20T22:49:26.704 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.016765688000000001, 2024-09-20T22:49:26.704 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:49:26.704 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:49:26.704 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:49:26.704 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:49:26.705 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.705 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:49:26.705 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.223406+0000", 2024-09-20T22:49:26.705 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:49:26.705 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.705 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.705 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:49:26.705 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.223406+0000", 2024-09-20T22:49:26.705 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:49:26.705 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.705 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.705 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:49:26.705 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.223406+0000", 2024-09-20T22:49:26.705 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.6699999999999999e-07 2024-09-20T22:49:26.705 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.706 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.706 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:49:26.706 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.226227+0000", 2024-09-20T22:49:26.706 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0028205690000000002 2024-09-20T22:49:26.706 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.706 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.706 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:49:26.706 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.226238+0000", 2024-09-20T22:49:26.706 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0879999999999999e-05 2024-09-20T22:49:26.706 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.706 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.706 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:49:26.706 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.226249+0000", 2024-09-20T22:49:26.706 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0912e-05 2024-09-20T22:49:26.707 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.707 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.707 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:49:26.707 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.227960+0000", 2024-09-20T22:49:26.707 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0017109619999999999 2024-09-20T22:49:26.707 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.707 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.707 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:49:26.707 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.227987+0000", 2024-09-20T22:49:26.707 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.7659999999999999e-05 2024-09-20T22:49:26.707 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.707 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.707 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:49:26.707 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.240119+0000", 2024-09-20T22:49:26.708 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.012131885 2024-09-20T22:49:26.708 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.708 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.708 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:49:26.708 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.240172+0000", 2024-09-20T22:49:26.708 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.2453000000000001e-05 2024-09-20T22:49:26.708 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:49:26.708 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:49:26.708 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:49:26.708 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.708 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.708 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:3232 7.16 7:6885a65f:::smithi08477203-14:head [copy-from ver 187] snapc 7e=[7e,7d,7c,7b,7a,79,78,75,74,73,70,6b] ondisk+write+known_if_redirected+supports_pool_eio e337)", 2024-09-20T22:49:26.708 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:46:57.213186+0000", 2024-09-20T22:49:26.708 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 149.40551561199999, 2024-09-20T22:49:26.708 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.028083105000000001, 2024-09-20T22:49:26.709 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:49:26.709 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:49:26.709 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:49:26.709 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:49:26.709 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:49:26.709 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:49:26.709 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 3232 2024-09-20T22:49:26.709 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.709 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:49:26.709 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.709 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:49:26.709 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.213186+0000", 2024-09-20T22:49:26.709 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:49:26.709 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.710 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.710 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:49:26.710 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.213186+0000", 2024-09-20T22:49:26.710 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:49:26.710 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.710 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.710 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:49:26.710 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.213188+0000", 2024-09-20T22:49:26.710 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.5129999999999999e-06 2024-09-20T22:49:26.710 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.710 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.710 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:49:26.710 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.213194+0000", 2024-09-20T22:49:26.710 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.4810000000000003e-06 2024-09-20T22:49:26.710 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.711 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.711 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:49:26.711 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.213215+0000", 2024-09-20T22:49:26.711 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.0455999999999999e-05 2024-09-20T22:49:26.711 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.711 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.711 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:49:26.711 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.213225+0000", 2024-09-20T22:49:26.711 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0212e-05 2024-09-20T22:49:26.711 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.711 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.711 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:49:26.711 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.213315+0000", 2024-09-20T22:49:26.711 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.9524000000000005e-05 2024-09-20T22:49:26.712 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.712 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.712 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op must wait for map", 2024-09-20T22:49:26.712 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.213324+0000", 2024-09-20T22:49:26.712 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.8890000000000008e-06 2024-09-20T22:49:26.712 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.712 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.712 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:49:26.712 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.220517+0000", 2024-09-20T22:49:26.712 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0071931219999999997 2024-09-20T22:49:26.712 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.712 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.712 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:49:26.712 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.220636+0000", 2024-09-20T22:49:26.712 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000118875 2024-09-20T22:49:26.713 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.713 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.713 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 2,5", 2024-09-20T22:49:26.713 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.230308+0000", 2024-09-20T22:49:26.713 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0096726299999999998 2024-09-20T22:49:26.713 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.713 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.713 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:49:26.713 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.238007+0000", 2024-09-20T22:49:26.713 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.007699287 2024-09-20T22:49:26.713 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.713 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.713 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:49:26.713 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.240863+0000", 2024-09-20T22:49:26.714 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0028554880000000002 2024-09-20T22:49:26.714 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.714 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.714 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:49:26.714 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.241194+0000", 2024-09-20T22:49:26.714 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00033087399999999999 2024-09-20T22:49:26.714 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.714 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.714 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:49:26.714 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.241233+0000", 2024-09-20T22:49:26.714 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.9338000000000002e-05 2024-09-20T22:49:26.714 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.714 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.714 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:49:26.714 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.241269+0000", 2024-09-20T22:49:26.715 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.5416000000000001e-05 2024-09-20T22:49:26.715 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:49:26.715 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:49:26.715 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:49:26.715 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.715 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.715 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:3233 7.16 7:6885a65f:::smithi08477203-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e337)", 2024-09-20T22:49:26.715 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:46:57.213253+0000", 2024-09-20T22:49:26.715 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 149.40544818800001, 2024-09-20T22:49:26.715 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.023594825, 2024-09-20T22:49:26.715 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:49:26.715 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:49:26.715 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:49:26.715 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:49:26.716 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:49:26.716 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:49:26.716 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 3233 2024-09-20T22:49:26.716 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.716 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:49:26.716 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.716 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:49:26.716 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.213253+0000", 2024-09-20T22:49:26.716 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:49:26.716 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.716 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.716 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:49:26.716 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.213253+0000", 2024-09-20T22:49:26.716 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:49:26.716 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.717 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.717 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:49:26.717 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.213254+0000", 2024-09-20T22:49:26.717 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.4799999999999998e-07 2024-09-20T22:49:26.717 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.717 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.717 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:49:26.717 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.213257+0000", 2024-09-20T22:49:26.717 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.3780000000000001e-06 2024-09-20T22:49:26.717 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.717 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.717 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:49:26.717 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.213267+0000", 2024-09-20T22:49:26.717 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.9480000000000003e-06 2024-09-20T22:49:26.717 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.718 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.718 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:49:26.718 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.213273+0000", 2024-09-20T22:49:26.718 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.1240000000000002e-06 2024-09-20T22:49:26.718 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.718 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.718 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:49:26.718 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.213394+0000", 2024-09-20T22:49:26.718 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000120717 2024-09-20T22:49:26.718 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.718 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.718 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting_for_map not empty", 2024-09-20T22:49:26.718 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.213400+0000", 2024-09-20T22:49:26.718 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.0769999999999996e-06 2024-09-20T22:49:26.719 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.719 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.719 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:49:26.719 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.220851+0000", 2024-09-20T22:49:26.719 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0074511309999999997 2024-09-20T22:49:26.719 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.719 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.719 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for blocked object", 2024-09-20T22:49:26.719 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.220903+0000", 2024-09-20T22:49:26.719 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.215e-05 2024-09-20T22:49:26.719 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.719 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.719 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:49:26.719 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.236698+0000", 2024-09-20T22:49:26.720 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.015794823 2024-09-20T22:49:26.720 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.720 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.720 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:49:26.720 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.236766+0000", 2024-09-20T22:49:26.720 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.8126999999999993e-05 2024-09-20T22:49:26.720 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.720 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.720 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:49:26.720 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.236848+0000", 2024-09-20T22:49:26.720 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.1701999999999998e-05 2024-09-20T22:49:26.720 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:49:26.720 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:49:26.720 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:49:26.720 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.721 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.721 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:3671 7.5 7:a160b5d4:::smithi08477203-16:head [copy-from ver 236] snapc 91=[91,8e,8d,8c,8b,8a,88,87,86,85,84,80,7f,7a,74,73,6b] ondisk+write+known_if_redirected+supports_pool_eio e370)", 2024-09-20T22:49:26.721 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:47:30.237235+0000", 2024-09-20T22:49:26.721 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 116.381466389, 2024-09-20T22:49:26.721 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.019719474000000001, 2024-09-20T22:49:26.721 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:49:26.721 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:49:26.721 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:49:26.721 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:49:26.721 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:49:26.721 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:49:26.721 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 3671 2024-09-20T22:49:26.721 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.721 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:49:26.722 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.722 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:49:26.722 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:30.237235+0000", 2024-09-20T22:49:26.722 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:49:26.722 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.722 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.722 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:49:26.722 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:30.237235+0000", 2024-09-20T22:49:26.722 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:49:26.722 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.722 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.722 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:49:26.722 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:30.237236+0000", 2024-09-20T22:49:26.722 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.5570000000000001e-06 2024-09-20T22:49:26.722 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.723 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.723 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:49:26.723 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:30.237241+0000", 2024-09-20T22:49:26.723 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.3279999999999999e-06 2024-09-20T22:49:26.723 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.723 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.723 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:49:26.723 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:30.237252+0000", 2024-09-20T22:49:26.723 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1004e-05 2024-09-20T22:49:26.723 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.723 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.723 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:49:26.723 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:30.237263+0000", 2024-09-20T22:49:26.723 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1341e-05 2024-09-20T22:49:26.724 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.724 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.724 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:49:26.724 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:30.237319+0000", 2024-09-20T22:49:26.724 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.5939000000000001e-05 2024-09-20T22:49:26.724 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.724 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.724 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:49:26.724 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:30.237436+0000", 2024-09-20T22:49:26.724 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000116866 2024-09-20T22:49:26.724 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.724 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.724 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 1,2", 2024-09-20T22:49:26.724 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:30.244330+0000", 2024-09-20T22:49:26.724 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0068945619999999999 2024-09-20T22:49:26.725 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.725 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.725 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:49:26.725 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:30.249103+0000", 2024-09-20T22:49:26.725 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0047726319999999997 2024-09-20T22:49:26.725 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.725 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.725 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:49:26.725 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:30.255771+0000", 2024-09-20T22:49:26.725 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.006667666 2024-09-20T22:49:26.725 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.725 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.725 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:49:26.725 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:30.256879+0000", 2024-09-20T22:49:26.725 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0011083110000000001 2024-09-20T22:49:26.726 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.726 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.726 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:49:26.726 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:30.256919+0000", 2024-09-20T22:49:26.726 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.9589000000000003e-05 2024-09-20T22:49:26.726 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.726 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.726 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:49:26.726 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:30.256954+0000", 2024-09-20T22:49:26.726 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.5679000000000002e-05 2024-09-20T22:49:26.726 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:49:26.726 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:49:26.726 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:49:26.726 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.727 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.727 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:3747 7.c e383/94 7:3f865ef5:::smithi08477203-28:head v 383'822, mlcod=382'818)", 2024-09-20T22:49:26.727 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:47:43.267267+0000", 2024-09-20T22:49:26.727 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 103.35143392499999, 2024-09-20T22:49:26.727 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.014405072, 2024-09-20T22:49:26.727 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:49:26.727 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:49:26.727 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:49:26.727 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:49:26.727 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.727 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:49:26.727 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.267267+0000", 2024-09-20T22:49:26.727 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:49:26.727 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.728 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.728 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:49:26.728 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.267267+0000", 2024-09-20T22:49:26.728 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:49:26.728 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.728 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.728 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:49:26.728 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.267267+0000", 2024-09-20T22:49:26.728 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.73e-07 2024-09-20T22:49:26.728 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.728 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.728 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:49:26.728 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.269537+0000", 2024-09-20T22:49:26.728 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0022696650000000001 2024-09-20T22:49:26.728 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.729 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.729 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:49:26.729 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.269564+0000", 2024-09-20T22:49:26.729 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.6653999999999999e-05 2024-09-20T22:49:26.729 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.729 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.729 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:49:26.729 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.269575+0000", 2024-09-20T22:49:26.729 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1661999999999999e-05 2024-09-20T22:49:26.729 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.729 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.729 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:49:26.729 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.269672+0000", 2024-09-20T22:49:26.729 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.6430999999999999e-05 2024-09-20T22:49:26.729 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.730 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.730 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:49:26.730 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.269716+0000", 2024-09-20T22:49:26.730 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.4518000000000003e-05 2024-09-20T22:49:26.730 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.730 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.730 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:49:26.730 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.281609+0000", 2024-09-20T22:49:26.730 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.011892982999999999 2024-09-20T22:49:26.730 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.730 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.730 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:49:26.730 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.281672+0000", 2024-09-20T22:49:26.730 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.2985999999999996e-05 2024-09-20T22:49:26.730 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:49:26.731 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:49:26.731 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:49:26.731 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.731 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.731 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:3762 7.17 7:ed54065f:::smithi08477203-12:head [write 739640~641484] snapc 98=[98,97,96,95,94,93,92,8e,8d,8c,8b,8a,88,87,86,80,7f,73] ondisk+write+known_if_redirected+supports_pool_eio e383)", 2024-09-20T22:49:26.731 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:47:43.291672+0000", 2024-09-20T22:49:26.731 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 103.32702957399999, 2024-09-20T22:49:26.731 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.012690432999999999, 2024-09-20T22:49:26.731 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:49:26.731 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:49:26.731 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:49:26.731 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:49:26.731 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:49:26.731 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:49:26.731 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 3762 2024-09-20T22:49:26.732 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.732 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:49:26.732 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.732 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:49:26.732 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.291672+0000", 2024-09-20T22:49:26.732 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:49:26.732 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.732 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.732 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:49:26.732 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.291672+0000", 2024-09-20T22:49:26.732 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:49:26.732 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.732 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.732 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:49:26.733 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.291673+0000", 2024-09-20T22:49:26.733 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.5680000000000001e-06 2024-09-20T22:49:26.733 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.733 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.733 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:49:26.733 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.292393+0000", 2024-09-20T22:49:26.733 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00072015399999999998 2024-09-20T22:49:26.733 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.733 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.733 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:49:26.733 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.292407+0000", 2024-09-20T22:49:26.733 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.365e-05 2024-09-20T22:49:26.733 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.733 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.733 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:49:26.733 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.292431+0000", 2024-09-20T22:49:26.734 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.3740999999999999e-05 2024-09-20T22:49:26.734 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.734 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.734 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:49:26.734 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.292482+0000", 2024-09-20T22:49:26.734 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.1459999999999999e-05 2024-09-20T22:49:26.734 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.734 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.734 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:49:26.734 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.292549+0000", 2024-09-20T22:49:26.734 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.7223000000000003e-05 2024-09-20T22:49:26.734 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.734 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.734 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 2,5", 2024-09-20T22:49:26.735 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.292684+0000", 2024-09-20T22:49:26.735 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000134251 2024-09-20T22:49:26.735 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.735 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.735 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:49:26.735 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.293845+0000", 2024-09-20T22:49:26.735 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0011617789999999999 2024-09-20T22:49:26.735 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.735 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.735 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:49:26.735 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.294392+0000", 2024-09-20T22:49:26.735 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00054659800000000003 2024-09-20T22:49:26.735 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.735 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.735 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:49:26.736 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.304282+0000", 2024-09-20T22:49:26.736 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0098903099999999994 2024-09-20T22:49:26.736 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.736 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.736 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:49:26.736 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.304322+0000", 2024-09-20T22:49:26.736 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.0059999999999999e-05 2024-09-20T22:49:26.736 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.736 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.736 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:49:26.736 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.304362+0000", 2024-09-20T22:49:26.736 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.9638999999999998e-05 2024-09-20T22:49:26.736 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:49:26.736 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:49:26.737 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:49:26.737 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.737 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.737 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:4129 7.c e417/94 7:3f865ef5:::smithi08477203-28:head v 417'903, mlcod=417'899)", 2024-09-20T22:49:26.737 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:48:17.254548+0000", 2024-09-20T22:49:26.737 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 69.36415341, 2024-09-20T22:49:26.737 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.012969022, 2024-09-20T22:49:26.737 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:49:26.737 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:49:26.737 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:49:26.737 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:49:26.737 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.737 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:49:26.737 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:17.254548+0000", 2024-09-20T22:49:26.737 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:49:26.738 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.738 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.738 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:49:26.738 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:17.254548+0000", 2024-09-20T22:49:26.738 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:49:26.738 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.738 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.738 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:49:26.738 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:17.254549+0000", 2024-09-20T22:49:26.738 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.446e-06 2024-09-20T22:49:26.738 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.738 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.738 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:49:26.738 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:17.257313+0000", 2024-09-20T22:49:26.738 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0027641319999999999 2024-09-20T22:49:26.739 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.739 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.739 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:49:26.739 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:17.257333+0000", 2024-09-20T22:49:26.739 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9950999999999999e-05 2024-09-20T22:49:26.739 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.739 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.739 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:49:26.739 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:17.257343+0000", 2024-09-20T22:49:26.739 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.2750000000000001e-06 2024-09-20T22:49:26.739 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.739 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.739 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:49:26.739 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:17.257416+0000", 2024-09-20T22:49:26.740 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.3041999999999996e-05 2024-09-20T22:49:26.740 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.740 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.740 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:49:26.740 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:17.257442+0000", 2024-09-20T22:49:26.740 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.6095999999999999e-05 2024-09-20T22:49:26.740 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.740 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.740 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:49:26.740 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:17.267454+0000", 2024-09-20T22:49:26.740 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.010012455999999999 2024-09-20T22:49:26.740 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:49:26.740 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:49:26.740 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:49:26.740 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:17.267517+0000", 2024-09-20T22:49:26.741 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.2624000000000002e-05 2024-09-20T22:49:26.741 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:49:26.741 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:49:26.741 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:49:26.741 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:49:26.741 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:49:26.741 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-20T22:49:26.741 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:49:26.763 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:26.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:26.764 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:49:26.865 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:49:26.887 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:26.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:26.888 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:49:26.988 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:49:27.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:27.010 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:27.011 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:49:27.111 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:49:27.117 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:49:27.117 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:49:27.129 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:49:27.130 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:49:27.132 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:27.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:27.133 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:49:27.145 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:49:27.145 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:49:27.161 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-20T22:49:27.161 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T22:49:27.178 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:49:27.178 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:49:27.195 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:49:27.195 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:49:27.212 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:49:27.212 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:49:27.222 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is unset 2024-09-20T22:49:27.226 INFO:tasks.rados.rados.0.smithi084.stdout:1893: done (0 left) 2024-09-20T22:49:27.226 INFO:tasks.rados.rados.0.smithi084.stdout:1894: write (excl) oid 42 current snap is 204 2024-09-20T22:49:27.226 INFO:tasks.rados.rados.0.smithi084.stdout:1894: seq_num 667 ranges {410269=713794,1648313=507551,2917489=292780} 2024-09-20T22:49:27.231 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:49:27.231 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:49:27.233 INFO:tasks.rados.rados.0.smithi084.stdout:1894: writing smithi08477203-42 from 410269 to 1124063 tid 1 2024-09-20T22:49:27.234 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:49:27.240 INFO:tasks.rados.rados.0.smithi084.stdout:1894: writing smithi08477203-42 from 1648313 to 2155864 tid 2 2024-09-20T22:49:27.247 INFO:tasks.rados.rados.0.smithi084.stdout:1894: writing smithi08477203-42 from 2917489 to 3210269 tid 3 2024-09-20T22:49:27.247 INFO:tasks.rados.rados.0.smithi084.stdout:1895: read oid 50 snap 197 2024-09-20T22:49:27.247 INFO:tasks.rados.rados.0.smithi084.stdout:1895: expect (ObjNum 515 snap 148 seq_num 515) 2024-09-20T22:49:27.247 INFO:tasks.rados.rados.0.smithi084.stdout:1896: snap_remove snap 193 2024-09-20T22:49:27.257 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:27.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:27.258 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:49:27.359 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:49:27.405 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:27 smithi084 ceph-mon[40024]: pgmap v630: 158 pgs: 1 active+clean+snaptrim_wait, 1 clean+premerge+peered, 3 active+clean+snaptrim, 153 active+clean; 411 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-20T22:49:27.406 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:27 smithi084 ceph-mon[40024]: 6.f scrub starts 2024-09-20T22:49:27.406 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:27 smithi084 ceph-mon[40024]: 6.f scrub ok 2024-09-20T22:49:27.406 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:27 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/1392076497' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:49:27.406 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:27 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/3668029767' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:49:27.406 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:27 smithi084 ceph-mon[40024]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T22:49:27.406 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:27 smithi084 ceph-mon[40024]: Cluster is now healthy 2024-09-20T22:49:27.406 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:27 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/1392076497' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T22:49:27.407 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:27 smithi084 ceph-mon[40024]: osdmap e487: 8 total, 8 up, 8 in 2024-09-20T22:49:27.407 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:27 smithi084 ceph-mon[29774]: pgmap v630: 158 pgs: 1 active+clean+snaptrim_wait, 1 clean+premerge+peered, 3 active+clean+snaptrim, 153 active+clean; 411 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-20T22:49:27.407 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:27 smithi084 ceph-mon[29774]: 6.f scrub starts 2024-09-20T22:49:27.407 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:27 smithi084 ceph-mon[29774]: 6.f scrub ok 2024-09-20T22:49:27.408 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:27 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/1392076497' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:49:27.408 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:27 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/3668029767' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:49:27.408 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:27 smithi084 ceph-mon[29774]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T22:49:27.408 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:27 smithi084 ceph-mon[29774]: Cluster is now healthy 2024-09-20T22:49:27.408 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:27 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/1392076497' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T22:49:27.408 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:27 smithi084 ceph-mon[29774]: osdmap e487: 8 total, 8 up, 8 in 2024-09-20T22:49:27.430 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:27.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:27.431 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:49:27.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:27 smithi154 ceph-mon[33264]: pgmap v630: 158 pgs: 1 active+clean+snaptrim_wait, 1 clean+premerge+peered, 3 active+clean+snaptrim, 153 active+clean; 411 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-20T22:49:27.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:27 smithi154 ceph-mon[33264]: 6.f scrub starts 2024-09-20T22:49:27.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:27 smithi154 ceph-mon[33264]: 6.f scrub ok 2024-09-20T22:49:27.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:27 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/1392076497' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:49:27.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:27 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/3668029767' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:49:27.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:27 smithi154 ceph-mon[33264]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T22:49:27.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:27 smithi154 ceph-mon[33264]: Cluster is now healthy 2024-09-20T22:49:27.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:27 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/1392076497' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T22:49:27.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:27 smithi154 ceph-mon[33264]: osdmap e487: 8 total, 8 up, 8 in 2024-09-20T22:49:27.532 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:49:27.553 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:27.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:27.554 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:49:27.645 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-20T22:49:27.655 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:49:27.676 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:27.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:27.677 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:49:27.778 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:49:27.799 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:27.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:27.800 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:49:27.900 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:49:27.922 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:27.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:27.923 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:49:28.024 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:49:28.045 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:28.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:28.046 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:49:28.147 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:49:28.168 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:28.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:28.169 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:49:28.222 INFO:tasks.rados.rados.0.smithi084.stdout:1896: done (2 left) 2024-09-20T22:49:28.223 INFO:tasks.rados.rados.0.smithi084.stdout:1897: snap_remove snap 183 2024-09-20T22:49:28.270 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:49:28.291 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:28.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:28.293 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:49:28.393 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:49:28.415 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:28.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:28.416 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:49:28.516 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:49:28.537 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:28.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:28.539 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:49:28.615 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:49:28 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: ::ffff:172.21.15.154 - - [20/Sep/2024:22:49:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-20T22:49:28.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:28 smithi084 ceph-mon[40024]: 3.1f scrub starts 2024-09-20T22:49:28.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:28 smithi084 ceph-mon[40024]: 3.1f scrub ok 2024-09-20T22:49:28.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:28 smithi084 ceph-mon[40024]: 5.6 scrub starts 2024-09-20T22:49:28.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:28 smithi084 ceph-mon[40024]: 5.6 scrub ok 2024-09-20T22:49:28.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:28 smithi084 ceph-mon[40024]: 2.2 scrub starts 2024-09-20T22:49:28.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:28 smithi084 ceph-mon[40024]: 2.2 scrub ok 2024-09-20T22:49:28.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:28 smithi084 ceph-mon[40024]: 5.1 scrub starts 2024-09-20T22:49:28.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:28 smithi084 ceph-mon[40024]: 5.1 scrub ok 2024-09-20T22:49:28.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:28 smithi084 ceph-mon[40024]: 3.4 scrub starts 2024-09-20T22:49:28.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:28 smithi084 ceph-mon[40024]: 3.4 scrub ok 2024-09-20T22:49:28.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:28 smithi084 ceph-mon[40024]: 1.6 scrub starts 2024-09-20T22:49:28.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:28 smithi084 ceph-mon[40024]: 1.6 scrub ok 2024-09-20T22:49:28.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:28 smithi084 ceph-mon[40024]: osdmap e488: 8 total, 8 up, 8 in 2024-09-20T22:49:28.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:28 smithi084 ceph-mon[29774]: 3.1f scrub starts 2024-09-20T22:49:28.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:28 smithi084 ceph-mon[29774]: 3.1f scrub ok 2024-09-20T22:49:28.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:28 smithi084 ceph-mon[29774]: 5.6 scrub starts 2024-09-20T22:49:28.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:28 smithi084 ceph-mon[29774]: 5.6 scrub ok 2024-09-20T22:49:28.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:28 smithi084 ceph-mon[29774]: 2.2 scrub starts 2024-09-20T22:49:28.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:28 smithi084 ceph-mon[29774]: 2.2 scrub ok 2024-09-20T22:49:28.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:28 smithi084 ceph-mon[29774]: 5.1 scrub starts 2024-09-20T22:49:28.617 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:28 smithi084 ceph-mon[29774]: 5.1 scrub ok 2024-09-20T22:49:28.617 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:28 smithi084 ceph-mon[29774]: 3.4 scrub starts 2024-09-20T22:49:28.617 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:28 smithi084 ceph-mon[29774]: 3.4 scrub ok 2024-09-20T22:49:28.617 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:28 smithi084 ceph-mon[29774]: 1.6 scrub starts 2024-09-20T22:49:28.617 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:28 smithi084 ceph-mon[29774]: 1.6 scrub ok 2024-09-20T22:49:28.617 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:28 smithi084 ceph-mon[29774]: osdmap e488: 8 total, 8 up, 8 in 2024-09-20T22:49:28.639 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:49:28.661 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:28.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:28.662 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:49:28.724 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:49:28.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:28 smithi154 ceph-mon[33264]: 3.1f scrub starts 2024-09-20T22:49:28.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:28 smithi154 ceph-mon[33264]: 3.1f scrub ok 2024-09-20T22:49:28.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:28 smithi154 ceph-mon[33264]: 5.6 scrub starts 2024-09-20T22:49:28.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:28 smithi154 ceph-mon[33264]: 5.6 scrub ok 2024-09-20T22:49:28.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:28 smithi154 ceph-mon[33264]: 2.2 scrub starts 2024-09-20T22:49:28.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:28 smithi154 ceph-mon[33264]: 2.2 scrub ok 2024-09-20T22:49:28.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:28 smithi154 ceph-mon[33264]: 5.1 scrub starts 2024-09-20T22:49:28.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:28 smithi154 ceph-mon[33264]: 5.1 scrub ok 2024-09-20T22:49:28.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:28 smithi154 ceph-mon[33264]: 3.4 scrub starts 2024-09-20T22:49:28.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:28 smithi154 ceph-mon[33264]: 3.4 scrub ok 2024-09-20T22:49:28.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:28 smithi154 ceph-mon[33264]: 1.6 scrub starts 2024-09-20T22:49:28.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:28 smithi154 ceph-mon[33264]: 1.6 scrub ok 2024-09-20T22:49:28.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:28 smithi154 ceph-mon[33264]: osdmap e488: 8 total, 8 up, 8 in 2024-09-20T22:49:28.762 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:49:28.783 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:28.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:28.784 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:49:28.885 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:49:28.906 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:28.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:28.907 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:49:29.008 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:49:29.030 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:29.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:29.031 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:49:29.132 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:49:29.154 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:29.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:29.155 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:49:29.198 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-20T22:49:29.198 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:49:29.228 INFO:tasks.rados.rados.0.smithi084.stdout:1897: done (2 left) 2024-09-20T22:49:29.229 INFO:tasks.rados.rados.0.smithi084.stdout:1898: write oid 40 current snap is 204 2024-09-20T22:49:29.229 INFO:tasks.rados.rados.0.smithi084.stdout:1898: seq_num 668 ranges {427076=789043,1706795=767562,2944912=282164} 2024-09-20T22:49:29.236 INFO:tasks.rados.rados.0.smithi084.stdout:1898: writing smithi08477203-40 from 427076 to 1216119 tid 1 2024-09-20T22:49:29.244 INFO:tasks.rados.rados.0.smithi084.stdout:1898: writing smithi08477203-40 from 1706795 to 2474357 tid 2 2024-09-20T22:49:29.248 INFO:tasks.rados.rados.0.smithi084.stdout:1898: writing smithi08477203-40 from 2944912 to 3227076 tid 3 2024-09-20T22:49:29.248 INFO:tasks.rados.rados.0.smithi084.stdout:1899: write oid 26 current snap is 204 2024-09-20T22:49:29.249 INFO:tasks.rados.rados.0.smithi084.stdout:1899: seq_num 669 ranges {443883=464292,1365277=511220,2572335=671548} 2024-09-20T22:49:29.253 INFO:tasks.rados.rados.0.smithi084.stdout:1899: writing smithi08477203-26 from 443883 to 908175 tid 1 2024-09-20T22:49:29.256 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:49:29.258 INFO:tasks.rados.rados.0.smithi084.stdout:1899: writing smithi08477203-26 from 1365277 to 1876497 tid 2 2024-09-20T22:49:29.266 INFO:tasks.rados.rados.0.smithi084.stdout:1899: writing smithi08477203-26 from 2572335 to 3243883 tid 3 2024-09-20T22:49:29.266 INFO:tasks.rados.rados.0.smithi084.stdout:1900: copy_from oid 8 from oid 15 current snap is 204 2024-09-20T22:49:29.266 INFO:tasks.rados.rados.0.smithi084.stdout:1894: finishing write tid 1 to smithi08477203-42 2024-09-20T22:49:29.266 INFO:tasks.rados.rados.0.smithi084.stdout:1894: finishing write tid 2 to smithi08477203-42 2024-09-20T22:49:29.267 INFO:tasks.rados.rados.0.smithi084.stdout:1894: finishing write tid 3 to smithi08477203-42 2024-09-20T22:49:29.267 INFO:tasks.rados.rados.0.smithi084.stdout:1894: finishing write tid 4 to smithi08477203-42 2024-09-20T22:49:29.267 INFO:tasks.rados.rados.0.smithi084.stdout:1894: finishing write tid 5 to smithi08477203-42 2024-09-20T22:49:29.267 INFO:tasks.rados.rados.0.smithi084.stdout:1894: oid 42 updating version 0 to 563 2024-09-20T22:49:29.267 INFO:tasks.rados.rados.0.smithi084.stdout:1894: oid 42 updating version 563 to 564 2024-09-20T22:49:29.267 INFO:tasks.rados.rados.0.smithi084.stdout:1894: oid 42 version 564 is already newer than 562 2024-09-20T22:49:29.267 INFO:tasks.rados.rados.0.smithi084.stdout:1894: oid 42 version 564 is already newer than 561 2024-09-20T22:49:29.267 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 42 v 564 (ObjNum 667 snap 204 seq_num 667) dirty exists 2024-09-20T22:49:29.267 INFO:tasks.rados.rados.0.smithi084.stdout:1894: left oid 42 (ObjNum 667 snap 204 seq_num 667) 2024-09-20T22:49:29.267 INFO:tasks.rados.rados.0.smithi084.stdout:1895: expect (ObjNum 515 snap 148 seq_num 515) 2024-09-20T22:49:29.280 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:29.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:29.281 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:49:29.382 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:49:29.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:29.405 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:29.405 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:49:29.423 INFO:tasks.rados.rados.0.smithi084.stdout:1898: finishing write tid 1 to smithi08477203-40 2024-09-20T22:49:29.423 INFO:tasks.rados.rados.0.smithi084.stdout:1898: finishing write tid 2 to smithi08477203-40 2024-09-20T22:49:29.424 INFO:tasks.rados.rados.0.smithi084.stdout:1898: finishing write tid 3 to smithi08477203-40 2024-09-20T22:49:29.424 INFO:tasks.rados.rados.0.smithi084.stdout:1898: finishing write tid 4 to smithi08477203-40 2024-09-20T22:49:29.424 INFO:tasks.rados.rados.0.smithi084.stdout:1898: finishing write tid 5 to smithi08477203-40 2024-09-20T22:49:29.424 INFO:tasks.rados.rados.0.smithi084.stdout:1898: oid 40 updating version 0 to 721 2024-09-20T22:49:29.424 INFO:tasks.rados.rados.0.smithi084.stdout:1898: oid 40 version 721 is already newer than 720 2024-09-20T22:49:29.424 INFO:tasks.rados.rados.0.smithi084.stdout:1898: oid 40 version 721 is already newer than 719 2024-09-20T22:49:29.424 INFO:tasks.rados.rados.0.smithi084.stdout:1898: oid 40 updating version 721 to 722 2024-09-20T22:49:29.424 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 40 v 722 (ObjNum 668 snap 204 seq_num 668) dirty exists 2024-09-20T22:49:29.424 INFO:tasks.rados.rados.0.smithi084.stdout:1898: left oid 40 (ObjNum 668 snap 204 seq_num 668) 2024-09-20T22:49:29.424 INFO:tasks.rados.rados.0.smithi084.stdout:1899: finishing write tid 1 to smithi08477203-26 2024-09-20T22:49:29.424 INFO:tasks.rados.rados.0.smithi084.stdout:1894: done (4 left) 2024-09-20T22:49:29.424 INFO:tasks.rados.rados.0.smithi084.stdout:1895: done (3 left) 2024-09-20T22:49:29.424 INFO:tasks.rados.rados.0.smithi084.stdout:1898: done (2 left) 2024-09-20T22:49:29.425 INFO:tasks.rados.rados.0.smithi084.stdout:1901: snap_create 2024-09-20T22:49:29.425 INFO:tasks.rados.rados.0.smithi084.stdout:1899: finishing write tid 2 to smithi08477203-26 2024-09-20T22:49:29.425 INFO:tasks.rados.rados.0.smithi084.stdout:1899: finishing write tid 3 to smithi08477203-26 2024-09-20T22:49:29.425 INFO:tasks.rados.rados.0.smithi084.stdout:1899: finishing write tid 4 to smithi08477203-26 2024-09-20T22:49:29.425 INFO:tasks.rados.rados.0.smithi084.stdout:1899: finishing write tid 5 to smithi08477203-26 2024-09-20T22:49:29.425 INFO:tasks.rados.rados.0.smithi084.stdout:1899: oid 26 updating version 0 to 586 2024-09-20T22:49:29.425 INFO:tasks.rados.rados.0.smithi084.stdout:1899: oid 26 updating version 586 to 588 2024-09-20T22:49:29.425 INFO:tasks.rados.rados.0.smithi084.stdout:1899: oid 26 updating version 588 to 589 2024-09-20T22:49:29.425 INFO:tasks.rados.rados.0.smithi084.stdout:1899: oid 26 version 589 is already newer than 587 2024-09-20T22:49:29.425 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 26 v 589 (ObjNum 669 snap 204 seq_num 669) dirty exists 2024-09-20T22:49:29.425 INFO:tasks.rados.rados.0.smithi084.stdout:1899: left oid 26 (ObjNum 669 snap 204 seq_num 669) 2024-09-20T22:49:29.426 INFO:tasks.rados.rados.0.smithi084.stdout:1900: finishing copy_from racing read to smithi08477203-8 2024-09-20T22:49:29.426 INFO:tasks.rados.rados.0.smithi084.stdout:1900: finishing copy_from to smithi08477203-8 2024-09-20T22:49:29.426 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 8 v 749 (ObjNum 625 snap 186 seq_num 625) dirty exists 2024-09-20T22:49:29.506 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:49:29.528 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:29.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:29.529 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:49:29.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:29 smithi084 ceph-mon[40024]: 6.0 scrub starts 2024-09-20T22:49:29.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:29 smithi084 ceph-mon[40024]: 6.0 scrub ok 2024-09-20T22:49:29.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:29 smithi084 ceph-mon[40024]: pgmap v633: 158 pgs: 1 active+clean+snaptrim_wait, 1 clean+premerge+peered, 3 active+clean+snaptrim, 153 active+clean; 411 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-20T22:49:29.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:29 smithi084 ceph-mon[40024]: osdmap e489: 8 total, 8 up, 8 in 2024-09-20T22:49:29.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:29 smithi084 ceph-mon[29774]: 6.0 scrub starts 2024-09-20T22:49:29.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:29 smithi084 ceph-mon[29774]: 6.0 scrub ok 2024-09-20T22:49:29.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:29 smithi084 ceph-mon[29774]: pgmap v633: 158 pgs: 1 active+clean+snaptrim_wait, 1 clean+premerge+peered, 3 active+clean+snaptrim, 153 active+clean; 411 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-20T22:49:29.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:29 smithi084 ceph-mon[29774]: osdmap e489: 8 total, 8 up, 8 in 2024-09-20T22:49:29.629 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:49:29.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:29.651 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:29.652 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:49:29.752 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:49:29.754 INFO:journalctl@ceph.iscsi.iscsi.a.smithi154.stdout:Sep 20 22:49:29 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-iscsi-iscsi-a[59730]: debug there is no tcmu-runner data available 2024-09-20T22:49:29.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:29 smithi154 ceph-mon[33264]: 6.0 scrub starts 2024-09-20T22:49:29.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:29 smithi154 ceph-mon[33264]: 6.0 scrub ok 2024-09-20T22:49:29.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:29 smithi154 ceph-mon[33264]: pgmap v633: 158 pgs: 1 active+clean+snaptrim_wait, 1 clean+premerge+peered, 3 active+clean+snaptrim, 153 active+clean; 411 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-20T22:49:29.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:29 smithi154 ceph-mon[33264]: osdmap e489: 8 total, 8 up, 8 in 2024-09-20T22:49:29.774 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:29.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:29.775 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:49:29.854 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.0 dump_blocked_ops 2024-09-20T22:49:29.876 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:49:29.897 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:29.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:29.898 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:49:29.999 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:49:30.020 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:30.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:30.021 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:49:30.100 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:49:30.121 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:49:30.143 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:30.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:30.144 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:49:30.218 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-20T22:49:30.245 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:49:30.267 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:30.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:30.268 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:49:30.369 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:49:30.370 INFO:tasks.rados.rados.0.smithi084.stdout:1899: done (2 left) 2024-09-20T22:49:30.370 INFO:tasks.rados.rados.0.smithi084.stdout:1900: done (1 left) 2024-09-20T22:49:30.371 INFO:tasks.rados.rados.0.smithi084.stdout:1901: done (0 left) 2024-09-20T22:49:30.371 INFO:tasks.rados.rados.0.smithi084.stdout:1902: write (excl) oid 5 current snap is 205 2024-09-20T22:49:30.371 INFO:tasks.rados.rados.0.smithi084.stdout:1902: seq_num 670 ranges {460690=655894,1540112=771231,2716111=539778} 2024-09-20T22:49:30.376 INFO:tasks.rados.rados.0.smithi084.stdout:1902: writing smithi08477203-5 from 460690 to 1116584 tid 1 2024-09-20T22:49:30.383 INFO:tasks.rados.rados.0.smithi084.stdout:1902: writing smithi08477203-5 from 1540112 to 2311343 tid 2 2024-09-20T22:49:30.389 INFO:tasks.rados.rados.0.smithi084.stdout:1902: writing smithi08477203-5 from 2716111 to 3255889 tid 3 2024-09-20T22:49:30.389 INFO:tasks.rados.rados.0.smithi084.stdout:1903: rollback oid 10 current snap is 205 2024-09-20T22:49:30.389 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 10 to 182 2024-09-20T22:49:30.389 INFO:tasks.rados.rados.0.smithi084.stdout:1904: write (excl) oid 47 current snap is 205 2024-09-20T22:49:30.389 INFO:tasks.rados.rados.0.smithi084.stdout:1904: seq_num 671 ranges {477497=731143,1998594=514889,3143534=133963} 2024-09-20T22:49:30.392 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:30.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:30.393 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:49:30.395 INFO:tasks.rados.rados.0.smithi084.stdout:1904: writing smithi08477203-47 from 477497 to 1208640 tid 1 2024-09-20T22:49:30.403 INFO:tasks.rados.rados.0.smithi084.stdout:1904: writing smithi08477203-47 from 1998594 to 2513483 tid 2 2024-09-20T22:49:30.407 INFO:tasks.rados.rados.0.smithi084.stdout:1904: writing smithi08477203-47 from 3143534 to 3277497 tid 3 2024-09-20T22:49:30.407 INFO:tasks.rados.rados.0.smithi084.stdout:1905: write oid 44 current snap is 205 2024-09-20T22:49:30.407 INFO:tasks.rados.rados.0.smithi084.stdout:1903: finishing rollback tid 1 to smithi08477203-10 2024-09-20T22:49:30.407 INFO:tasks.rados.rados.0.smithi084.stdout:1905: seq_num 672 ranges {494304=406392,1540723=774900,2770957=523347} 2024-09-20T22:49:30.412 INFO:tasks.rados.rados.0.smithi084.stdout:1905: writing smithi08477203-44 from 494304 to 900696 tid 1 2024-09-20T22:49:30.420 INFO:tasks.rados.rados.0.smithi084.stdout:1905: writing smithi08477203-44 from 1540723 to 2315623 tid 2 2024-09-20T22:49:30.426 INFO:tasks.rados.rados.0.smithi084.stdout:1905: writing smithi08477203-44 from 2770957 to 3294304 tid 3 2024-09-20T22:49:30.426 INFO:tasks.rados.rados.0.smithi084.stdout:1906: read oid 3 snap -1 2024-09-20T22:49:30.426 INFO:tasks.rados.rados.0.smithi084.stdout:1906: expect (ObjNum 649 snap 194 seq_num 649) 2024-09-20T22:49:30.426 INFO:tasks.rados.rados.0.smithi084.stdout:1907: read oid 34 snap -1 2024-09-20T22:49:30.426 INFO:tasks.rados.rados.0.smithi084.stdout:1907: expect (ObjNum 622 snap 186 seq_num 622) 2024-09-20T22:49:30.427 INFO:tasks.rados.rados.0.smithi084.stdout:1908: snap_create 2024-09-20T22:49:30.427 INFO:tasks.rados.rados.0.smithi084.stdout:1903: finishing rollback tid 2 to smithi08477203-10 2024-09-20T22:49:30.427 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 10 v 696 (ObjNum 575 snap 172 seq_num 575) dirty exists 2024-09-20T22:49:30.427 INFO:tasks.rados.rados.0.smithi084.stdout:1902: finishing write tid 1 to smithi08477203-5 2024-09-20T22:49:30.427 INFO:tasks.rados.rados.0.smithi084.stdout:1905: finishing write tid 1 to smithi08477203-44 2024-09-20T22:49:30.427 INFO:tasks.rados.rados.0.smithi084.stdout:1905: finishing write tid 2 to smithi08477203-44 2024-09-20T22:49:30.429 INFO:tasks.rados.rados.0.smithi084.stdout:1905: finishing write tid 3 to smithi08477203-44 2024-09-20T22:49:30.429 INFO:tasks.rados.rados.0.smithi084.stdout:1905: finishing write tid 4 to smithi08477203-44 2024-09-20T22:49:30.430 INFO:tasks.rados.rados.0.smithi084.stdout:1902: finishing write tid 2 to smithi08477203-5 2024-09-20T22:49:30.431 INFO:tasks.rados.rados.0.smithi084.stdout:1902: finishing write tid 3 to smithi08477203-5 2024-09-20T22:49:30.431 INFO:tasks.rados.rados.0.smithi084.stdout:1905: finishing write tid 5 to smithi08477203-44 2024-09-20T22:49:30.431 INFO:tasks.rados.rados.0.smithi084.stdout:1905: oid 44 updating version 0 to 1265 2024-09-20T22:49:30.431 INFO:tasks.rados.rados.0.smithi084.stdout:1905: oid 44 updating version 1265 to 1266 2024-09-20T22:49:30.431 INFO:tasks.rados.rados.0.smithi084.stdout:1905: oid 44 version 1266 is already newer than 1263 2024-09-20T22:49:30.431 INFO:tasks.rados.rados.0.smithi084.stdout:1905: oid 44 version 1266 is already newer than 1264 2024-09-20T22:49:30.431 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 44 v 1266 (ObjNum 672 snap 205 seq_num 672) dirty exists 2024-09-20T22:49:30.431 INFO:tasks.rados.rados.0.smithi084.stdout:1905: left oid 44 (ObjNum 672 snap 205 seq_num 672) 2024-09-20T22:49:30.431 INFO:tasks.rados.rados.0.smithi084.stdout:1902: finishing write tid 4 to smithi08477203-5 2024-09-20T22:49:30.431 INFO:tasks.rados.rados.0.smithi084.stdout:1902: finishing write tid 5 to smithi08477203-5 2024-09-20T22:49:30.432 INFO:tasks.rados.rados.0.smithi084.stdout:1902: oid 5 updating version 0 to 709 2024-09-20T22:49:30.432 INFO:tasks.rados.rados.0.smithi084.stdout:1902: oid 5 version 709 is already newer than 706 2024-09-20T22:49:30.432 INFO:tasks.rados.rados.0.smithi084.stdout:1902: oid 5 version 709 is already newer than 707 2024-09-20T22:49:30.433 INFO:tasks.rados.rados.0.smithi084.stdout:1902: oid 5 version 709 is already newer than 708 2024-09-20T22:49:30.433 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 5 v 709 (ObjNum 670 snap 205 seq_num 670) dirty exists 2024-09-20T22:49:30.433 INFO:tasks.rados.rados.0.smithi084.stdout:1902: left oid 5 (ObjNum 670 snap 205 seq_num 670) 2024-09-20T22:49:30.434 INFO:tasks.rados.rados.0.smithi084.stdout:1906: expect (ObjNum 649 snap 194 seq_num 649) 2024-09-20T22:49:30.494 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:49:30.514 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:30 smithi084 ceph-mon[40024]: 3.2 scrub starts 2024-09-20T22:49:30.515 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:30 smithi084 ceph-mon[40024]: 3.2 scrub ok 2024-09-20T22:49:30.515 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:30 smithi084 ceph-mon[40024]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:49:30.515 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:30 smithi084 ceph-mon[29774]: 3.2 scrub starts 2024-09-20T22:49:30.515 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:30 smithi084 ceph-mon[29774]: 3.2 scrub ok 2024-09-20T22:49:30.515 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:30 smithi084 ceph-mon[29774]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:49:30.517 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:30.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:30.518 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:49:30.609 INFO:tasks.rados.rados.0.smithi084.stdout:1904: finishing write tid 1 to smithi08477203-47 2024-09-20T22:49:30.609 INFO:tasks.rados.rados.0.smithi084.stdout:1904: finishing write tid 2 to smithi08477203-47 2024-09-20T22:49:30.609 INFO:tasks.rados.rados.0.smithi084.stdout:1904: finishing write tid 3 to smithi08477203-47 2024-09-20T22:49:30.609 INFO:tasks.rados.rados.0.smithi084.stdout:1904: finishing write tid 4 to smithi08477203-47 2024-09-20T22:49:30.609 INFO:tasks.rados.rados.0.smithi084.stdout:1904: finishing write tid 5 to smithi08477203-47 2024-09-20T22:49:30.609 INFO:tasks.rados.rados.0.smithi084.stdout:1904: oid 47 updating version 0 to 592 2024-09-20T22:49:30.609 INFO:tasks.rados.rados.0.smithi084.stdout:1904: oid 47 updating version 592 to 593 2024-09-20T22:49:30.609 INFO:tasks.rados.rados.0.smithi084.stdout:1904: oid 47 version 593 is already newer than 591 2024-09-20T22:49:30.609 INFO:tasks.rados.rados.0.smithi084.stdout:1904: oid 47 updating version 593 to 594 2024-09-20T22:49:30.609 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 47 v 594 (ObjNum 671 snap 205 seq_num 671) dirty exists 2024-09-20T22:49:30.609 INFO:tasks.rados.rados.0.smithi084.stdout:1904: left oid 47 (ObjNum 671 snap 205 seq_num 671) 2024-09-20T22:49:30.609 INFO:tasks.rados.rados.0.smithi084.stdout:1907: expect (ObjNum 622 snap 186 seq_num 622) 2024-09-20T22:49:30.614 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-20T22:49:30.614 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:49:30.618 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:49:30.638 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 5, 6, 7, 0, 4] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-20T22:49:30.638 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-20T22:49:30.639 INFO:tasks.thrashosds.thrasher:inject_pause on osd.4 2024-09-20T22:49:30.639 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-20T22:49:30.639 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-20T22:49:30.639 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.4 config set filestore_inject_stall 3 2024-09-20T22:49:30.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:30.642 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:30.642 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:49:30.664 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd set noscrub 2024-09-20T22:49:30.703 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:30 smithi154 ceph-mon[33264]: 3.2 scrub starts 2024-09-20T22:49:30.703 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:30 smithi154 ceph-mon[33264]: 3.2 scrub ok 2024-09-20T22:49:30.703 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:30 smithi154 ceph-mon[33264]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:49:30.743 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:49:30.763 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:30.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:30.764 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:49:30.865 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:49:30.888 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:30.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:30.889 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:49:30.927 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:49:30.989 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:49:31.010 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:31.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:31.011 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:49:31.112 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:49:31.134 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:31.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:31.135 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:49:31.235 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:49:31.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:31.258 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:31.258 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:49:31.359 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:49:31.364 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.0 dump_historic_ops 2024-09-20T22:49:31.372 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:49:31.376 INFO:tasks.rados.rados.0.smithi084.stdout:1902: done (6 left) 2024-09-20T22:49:31.376 INFO:tasks.rados.rados.0.smithi084.stdout:1903: done (5 left) 2024-09-20T22:49:31.376 INFO:tasks.rados.rados.0.smithi084.stdout:1904: done (4 left) 2024-09-20T22:49:31.376 INFO:tasks.rados.rados.0.smithi084.stdout:1905: done (3 left) 2024-09-20T22:49:31.376 INFO:tasks.rados.rados.0.smithi084.stdout:1906: done (2 left) 2024-09-20T22:49:31.376 INFO:tasks.rados.rados.0.smithi084.stdout:1907: done (1 left) 2024-09-20T22:49:31.376 INFO:tasks.rados.rados.0.smithi084.stdout:1908: done (0 left) 2024-09-20T22:49:31.376 INFO:tasks.rados.rados.0.smithi084.stdout:1909: write (excl) oid 12 current snap is 206 2024-09-20T22:49:31.376 INFO:tasks.rados.rados.0.smithi084.stdout:1909: seq_num 673 ranges {511111=481641,1599205=518558,2682027=489947} 2024-09-20T22:49:31.379 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:49:31.381 INFO:tasks.rados.rados.0.smithi084.stdout:1909: writing smithi08477203-12 from 511111 to 992752 tid 1 2024-09-20T22:49:31.386 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:31.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:31.387 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:49:31.389 INFO:tasks.rados.rados.0.smithi084.stdout:1909: writing smithi08477203-12 from 1599205 to 2117763 tid 2 2024-09-20T22:49:31.395 INFO:tasks.rados.rados.0.smithi084.stdout:1909: writing smithi08477203-12 from 2682027 to 3171974 tid 3 2024-09-20T22:49:31.395 INFO:tasks.rados.rados.0.smithi084.stdout:1910: delete oid 30 current snap is 206 2024-09-20T22:49:31.395 INFO:tasks.rados.rados.0.smithi084.stdout:1909: finishing write tid 1 to smithi08477203-12 2024-09-20T22:49:31.396 INFO:tasks.rados.rados.0.smithi084.stdout:1909: finishing write tid 2 to smithi08477203-12 2024-09-20T22:49:31.399 INFO:tasks.rados.rados.0.smithi084.stdout:1910: done (1 left) 2024-09-20T22:49:31.399 INFO:tasks.rados.rados.0.smithi084.stdout:1911: delete oid 3 current snap is 206 2024-09-20T22:49:31.399 INFO:tasks.rados.rados.0.smithi084.stdout:1909: finishing write tid 3 to smithi08477203-12 2024-09-20T22:49:31.399 INFO:tasks.rados.rados.0.smithi084.stdout:1909: finishing write tid 4 to smithi08477203-12 2024-09-20T22:49:31.400 INFO:tasks.rados.rados.0.smithi084.stdout:1909: finishing write tid 5 to smithi08477203-12 2024-09-20T22:49:31.400 INFO:tasks.rados.rados.0.smithi084.stdout:1909: oid 12 updating version 0 to 599 2024-09-20T22:49:31.400 INFO:tasks.rados.rados.0.smithi084.stdout:1909: oid 12 version 599 is already newer than 597 2024-09-20T22:49:31.400 INFO:tasks.rados.rados.0.smithi084.stdout:1909: oid 12 version 599 is already newer than 598 2024-09-20T22:49:31.400 INFO:tasks.rados.rados.0.smithi084.stdout:1909: oid 12 version 599 is already newer than 596 2024-09-20T22:49:31.400 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 12 v 599 (ObjNum 673 snap 206 seq_num 673) dirty exists 2024-09-20T22:49:31.400 INFO:tasks.rados.rados.0.smithi084.stdout:1909: left oid 12 (ObjNum 673 snap 206 seq_num 673) 2024-09-20T22:49:31.400 INFO:tasks.rados.rados.0.smithi084.stdout:1909: done (1 left) 2024-09-20T22:49:31.401 INFO:tasks.rados.rados.0.smithi084.stdout:1911: done (0 left) 2024-09-20T22:49:31.401 INFO:tasks.rados.rados.0.smithi084.stdout:1912: read oid 20 snap -1 2024-09-20T22:49:31.401 INFO:tasks.rados.rados.0.smithi084.stdout:1912: expect deleted 2024-09-20T22:49:31.401 INFO:tasks.rados.rados.0.smithi084.stdout:1913: snap_create 2024-09-20T22:49:31.487 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:49:31.502 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:31 smithi084 ceph-mon[40024]: pgmap v636: 158 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 153 active+clean; 414 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 21 MiB/s rd, 4.7 MiB/s wr, 29 op/s; 1.8 MiB/s, 2 objects/s recovering 2024-09-20T22:49:31.503 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:31 smithi084 ceph-mon[40024]: 6.1f scrub starts 2024-09-20T22:49:31.503 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:31 smithi084 ceph-mon[40024]: 6.1f scrub ok 2024-09-20T22:49:31.503 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:31 smithi084 ceph-mon[40024]: 2.1 scrub starts 2024-09-20T22:49:31.503 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:31 smithi084 ceph-mon[40024]: 2.1 scrub ok 2024-09-20T22:49:31.503 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:31 smithi084 ceph-mon[40024]: 6.8 scrub starts 2024-09-20T22:49:31.503 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:31 smithi084 ceph-mon[40024]: 6.8 scrub ok 2024-09-20T22:49:31.503 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:31 smithi084 ceph-mon[40024]: osdmap e490: 8 total, 8 up, 8 in 2024-09-20T22:49:31.503 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:31 smithi084 ceph-mon[40024]: 3.9 scrub starts 2024-09-20T22:49:31.503 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:31 smithi084 ceph-mon[40024]: 3.9 scrub ok 2024-09-20T22:49:31.503 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:31 smithi084 ceph-mon[40024]: 4.4 scrub starts 2024-09-20T22:49:31.503 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:31 smithi084 ceph-mon[40024]: 4.4 scrub ok 2024-09-20T22:49:31.504 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:31 smithi084 ceph-mon[29774]: pgmap v636: 158 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 153 active+clean; 414 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 21 MiB/s rd, 4.7 MiB/s wr, 29 op/s; 1.8 MiB/s, 2 objects/s recovering 2024-09-20T22:49:31.504 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:31 smithi084 ceph-mon[29774]: 6.1f scrub starts 2024-09-20T22:49:31.504 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:31 smithi084 ceph-mon[29774]: 6.1f scrub ok 2024-09-20T22:49:31.504 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:31 smithi084 ceph-mon[29774]: 2.1 scrub starts 2024-09-20T22:49:31.504 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:31 smithi084 ceph-mon[29774]: 2.1 scrub ok 2024-09-20T22:49:31.504 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:31 smithi084 ceph-mon[29774]: 6.8 scrub starts 2024-09-20T22:49:31.504 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:31 smithi084 ceph-mon[29774]: 6.8 scrub ok 2024-09-20T22:49:31.504 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:31 smithi084 ceph-mon[29774]: osdmap e490: 8 total, 8 up, 8 in 2024-09-20T22:49:31.504 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:31 smithi084 ceph-mon[29774]: 3.9 scrub starts 2024-09-20T22:49:31.504 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:31 smithi084 ceph-mon[29774]: 3.9 scrub ok 2024-09-20T22:49:31.504 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:31 smithi084 ceph-mon[29774]: 4.4 scrub starts 2024-09-20T22:49:31.504 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:31 smithi084 ceph-mon[29774]: 4.4 scrub ok 2024-09-20T22:49:31.508 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:31.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:31.509 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:49:31.571 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-20T22:49:31.571 INFO:teuthology.orchestra.run.smithi154.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-20T22:49:31.572 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-20T22:49:31.610 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:49:31.631 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:31.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:31.632 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:49:31.733 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:49:31.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:31.755 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:31.755 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:49:31.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:31 smithi154 ceph-mon[33264]: pgmap v636: 158 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 153 active+clean; 414 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 21 MiB/s rd, 4.7 MiB/s wr, 29 op/s; 1.8 MiB/s, 2 objects/s recovering 2024-09-20T22:49:31.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:31 smithi154 ceph-mon[33264]: 6.1f scrub starts 2024-09-20T22:49:31.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:31 smithi154 ceph-mon[33264]: 6.1f scrub ok 2024-09-20T22:49:31.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:31 smithi154 ceph-mon[33264]: 2.1 scrub starts 2024-09-20T22:49:31.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:31 smithi154 ceph-mon[33264]: 2.1 scrub ok 2024-09-20T22:49:31.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:31 smithi154 ceph-mon[33264]: 6.8 scrub starts 2024-09-20T22:49:31.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:31 smithi154 ceph-mon[33264]: 6.8 scrub ok 2024-09-20T22:49:31.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:31 smithi154 ceph-mon[33264]: osdmap e490: 8 total, 8 up, 8 in 2024-09-20T22:49:31.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:31 smithi154 ceph-mon[33264]: 3.9 scrub starts 2024-09-20T22:49:31.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:31 smithi154 ceph-mon[33264]: 3.9 scrub ok 2024-09-20T22:49:31.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:31 smithi154 ceph-mon[33264]: 4.4 scrub starts 2024-09-20T22:49:31.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:31 smithi154 ceph-mon[33264]: 4.4 scrub ok 2024-09-20T22:49:31.856 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:49:31.877 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:31.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:31.878 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:49:31.979 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:49:32.000 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:32.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:32.001 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:49:32.101 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:49:32.123 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:32.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:32.124 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:49:32.225 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:49:32.246 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:32.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:32.247 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:49:32.348 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:49:32.351 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:49:32.351 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:49:32.351 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:49:32.351 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:49:32.352 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:49:32.352 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:49:32.358 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-20T22:49:32.358 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T22:49:32.369 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:32.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:32.371 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:49:32.376 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:49:32.376 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:49:32.390 INFO:tasks.rados.rados.0.smithi084.stdout:1912: done (1 left) 2024-09-20T22:49:32.390 INFO:tasks.rados.rados.0.smithi084.stdout:1913: done (0 left) 2024-09-20T22:49:32.390 INFO:tasks.rados.rados.0.smithi084.stdout:1914: rollback oid 7 current snap is 207 2024-09-20T22:49:32.390 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 7 to 184 2024-09-20T22:49:32.390 INFO:tasks.rados.rados.0.smithi084.stdout:1915: read oid 15 snap -1 2024-09-20T22:49:32.390 INFO:tasks.rados.rados.0.smithi084.stdout:1915: expect (ObjNum 625 snap 186 seq_num 625) 2024-09-20T22:49:32.391 INFO:tasks.rados.rados.0.smithi084.stdout:1916: snap_remove snap 182 2024-09-20T22:49:32.391 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is set 2024-09-20T22:49:32.395 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:49:32.402 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:49:32.402 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:49:32.421 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:49:32.421 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:49:32.438 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:49:32.439 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:49:32.472 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:49:32.494 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:32.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:32.495 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:49:32.595 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:49:32.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:32 smithi084 ceph-mon[29774]: 6.3 scrub starts 2024-09-20T22:49:32.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:32 smithi084 ceph-mon[29774]: 6.3 scrub ok 2024-09-20T22:49:32.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:32 smithi084 ceph-mon[29774]: osdmap e491: 8 total, 8 up, 8 in 2024-09-20T22:49:32.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:32 smithi084 ceph-mon[29774]: 6.9 scrub starts 2024-09-20T22:49:32.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:32 smithi084 ceph-mon[29774]: 6.9 scrub ok 2024-09-20T22:49:32.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:32 smithi084 ceph-mon[29774]: 1.7 scrub starts 2024-09-20T22:49:32.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:32 smithi084 ceph-mon[29774]: 1.7 scrub ok 2024-09-20T22:49:32.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:32 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/1272313064' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-20T22:49:32.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:32 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-20T22:49:32.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:32 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/594531401' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:49:32.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:32 smithi084 ceph-mon[40024]: 6.3 scrub starts 2024-09-20T22:49:32.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:32 smithi084 ceph-mon[40024]: 6.3 scrub ok 2024-09-20T22:49:32.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:32 smithi084 ceph-mon[40024]: osdmap e491: 8 total, 8 up, 8 in 2024-09-20T22:49:32.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:32 smithi084 ceph-mon[40024]: 6.9 scrub starts 2024-09-20T22:49:32.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:32 smithi084 ceph-mon[40024]: 6.9 scrub ok 2024-09-20T22:49:32.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:32 smithi084 ceph-mon[40024]: 1.7 scrub starts 2024-09-20T22:49:32.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:32 smithi084 ceph-mon[40024]: 1.7 scrub ok 2024-09-20T22:49:32.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:32 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/1272313064' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-20T22:49:32.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:32 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-20T22:49:32.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:32 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/594531401' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:49:32.617 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:32.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:32.618 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:49:32.719 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:49:32.740 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:32.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:32.741 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:49:32.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:32 smithi154 ceph-mon[33264]: 6.3 scrub starts 2024-09-20T22:49:32.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:32 smithi154 ceph-mon[33264]: 6.3 scrub ok 2024-09-20T22:49:32.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:32 smithi154 ceph-mon[33264]: osdmap e491: 8 total, 8 up, 8 in 2024-09-20T22:49:32.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:32 smithi154 ceph-mon[33264]: 6.9 scrub starts 2024-09-20T22:49:32.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:32 smithi154 ceph-mon[33264]: 6.9 scrub ok 2024-09-20T22:49:32.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:32 smithi154 ceph-mon[33264]: 1.7 scrub starts 2024-09-20T22:49:32.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:32 smithi154 ceph-mon[33264]: 1.7 scrub ok 2024-09-20T22:49:32.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:32 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/1272313064' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-20T22:49:32.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:32 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-20T22:49:32.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:32 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/594531401' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:49:32.842 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:49:32.864 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:32.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:32.865 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:49:32.966 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:49:32.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:32.987 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:32.987 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:49:33.088 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:49:33.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:33.110 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:33.110 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:49:33.211 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:49:33.233 INFO:tasks.rados.rados.0.smithi084.stdout:1916: done (2 left) 2024-09-20T22:49:33.233 INFO:tasks.rados.rados.0.smithi084.stdout:1917: rollback oid 42 current snap is 207 2024-09-20T22:49:33.233 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 42 to 197 2024-09-20T22:49:33.233 INFO:tasks.rados.rados.0.smithi084.stdout:1918: snap_remove snap 132 2024-09-20T22:49:33.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:33.240 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:33.240 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:49:33.341 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:49:33.362 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:33.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:33.363 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:49:33.464 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:49:33.487 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:33.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:33.488 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:49:33.588 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:49:33.610 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:33.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:33.611 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:49:33.614 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:33 smithi084 ceph-mon[29774]: 5.1b scrub starts 2024-09-20T22:49:33.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:33 smithi084 ceph-mon[29774]: 5.1b scrub ok 2024-09-20T22:49:33.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:33 smithi084 ceph-mon[29774]: pgmap v639: 158 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 153 active+clean; 414 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 21 MiB/s rd, 4.7 MiB/s wr, 29 op/s; 1.8 MiB/s, 2 objects/s recovering 2024-09-20T22:49:33.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:33 smithi084 ceph-mon[29774]: 3.a scrub starts 2024-09-20T22:49:33.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:33 smithi084 ceph-mon[29774]: 3.a scrub ok 2024-09-20T22:49:33.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:33 smithi084 ceph-mon[29774]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:49:33.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:33 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T22:49:33.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:33 smithi084 ceph-mon[29774]: osdmap e492: 8 total, 8 up, 8 in 2024-09-20T22:49:33.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:33 smithi084 ceph-mon[29774]: osdmap e493: 8 total, 8 up, 8 in 2024-09-20T22:49:33.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:33 smithi084 ceph-mon[40024]: 5.1b scrub starts 2024-09-20T22:49:33.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:33 smithi084 ceph-mon[40024]: 5.1b scrub ok 2024-09-20T22:49:33.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:33 smithi084 ceph-mon[40024]: pgmap v639: 158 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 153 active+clean; 414 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 21 MiB/s rd, 4.7 MiB/s wr, 29 op/s; 1.8 MiB/s, 2 objects/s recovering 2024-09-20T22:49:33.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:33 smithi084 ceph-mon[40024]: 3.a scrub starts 2024-09-20T22:49:33.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:33 smithi084 ceph-mon[40024]: 3.a scrub ok 2024-09-20T22:49:33.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:33 smithi084 ceph-mon[40024]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:49:33.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:33 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T22:49:33.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:33 smithi084 ceph-mon[40024]: osdmap e492: 8 total, 8 up, 8 in 2024-09-20T22:49:33.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:33 smithi084 ceph-mon[40024]: osdmap e493: 8 total, 8 up, 8 in 2024-09-20T22:49:33.712 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:49:33.734 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:33.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:33.735 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:49:33.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:33 smithi154 ceph-mon[33264]: 5.1b scrub starts 2024-09-20T22:49:33.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:33 smithi154 ceph-mon[33264]: 5.1b scrub ok 2024-09-20T22:49:33.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:33 smithi154 ceph-mon[33264]: pgmap v639: 158 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 153 active+clean; 414 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 21 MiB/s rd, 4.7 MiB/s wr, 29 op/s; 1.8 MiB/s, 2 objects/s recovering 2024-09-20T22:49:33.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:33 smithi154 ceph-mon[33264]: 3.a scrub starts 2024-09-20T22:49:33.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:33 smithi154 ceph-mon[33264]: 3.a scrub ok 2024-09-20T22:49:33.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:33 smithi154 ceph-mon[33264]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:49:33.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:33 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T22:49:33.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:33 smithi154 ceph-mon[33264]: osdmap e492: 8 total, 8 up, 8 in 2024-09-20T22:49:33.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:33 smithi154 ceph-mon[33264]: osdmap e493: 8 total, 8 up, 8 in 2024-09-20T22:49:33.770 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-20T22:49:33.770 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2024-09-20T22:49:33.770 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2024-09-20T22:49:33.771 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [ 2024-09-20T22:49:33.771 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.771 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.14460.0:1 3.12 3:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-20T22:49:33.771 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:41:37.966675+0000", 2024-09-20T22:49:33.771 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 475.77595072499997, 2024-09-20T22:49:33.771 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.19206436900000001, 2024-09-20T22:49:33.771 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:49:33.771 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:49:33.771 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "reached pg", 2024-09-20T22:49:33.771 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:49:33.771 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.14460", 2024-09-20T22:49:33.771 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/1273482728", 2024-09-20T22:49:33.771 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1 2024-09-20T22:49:33.772 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.772 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:49:33.772 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.772 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:49:33.772 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:41:37.966675+0000", 2024-09-20T22:49:33.772 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:49:33.772 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.772 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.772 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:49:33.772 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:41:37.966675+0000", 2024-09-20T22:49:33.772 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:49:33.772 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.772 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.772 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:49:33.773 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:41:37.966682+0000", 2024-09-20T22:49:33.773 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.3100000000000003e-06 2024-09-20T22:49:33.773 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.773 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.773 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:49:33.773 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:41:37.966688+0000", 2024-09-20T22:49:33.773 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.1209999999999998e-06 2024-09-20T22:49:33.773 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.773 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.773 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:49:33.773 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:41:37.966698+0000", 2024-09-20T22:49:33.773 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0220999999999999e-05 2024-09-20T22:49:33.773 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.773 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.774 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:49:33.774 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:41:37.966706+0000", 2024-09-20T22:49:33.774 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.4460000000000001e-06 2024-09-20T22:49:33.774 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.774 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.774 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:49:33.774 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:41:37.966763+0000", 2024-09-20T22:49:33.774 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.7025999999999999e-05 2024-09-20T22:49:33.774 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.774 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.774 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-09-20T22:49:33.774 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:41:37.966771+0000", 2024-09-20T22:49:33.774 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.9540000000000004e-06 2024-09-20T22:49:33.774 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.775 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.775 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:49:33.775 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:41:38.158654+0000", 2024-09-20T22:49:33.775 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.19188287000000001 2024-09-20T22:49:33.775 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.775 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.775 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:49:33.775 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:41:38.158740+0000", 2024-09-20T22:49:33.775 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.5420999999999994e-05 2024-09-20T22:49:33.775 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:49:33.775 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:49:33.775 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:49:33.775 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.775 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.776 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:522 7.9 7:9970b578:::smithi08477203-15:head [copy-from ver 28] snapc f=[f,d] ondisk+write+known_if_redirected+supports_pool_eio e124)", 2024-09-20T22:49:33.776 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:43:23.126433+0000", 2024-09-20T22:49:33.776 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 370.61619316899998, 2024-09-20T22:49:33.776 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.027726022999999999, 2024-09-20T22:49:33.776 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:49:33.776 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:49:33.776 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:49:33.776 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:49:33.776 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:49:33.776 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:49:33.776 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 522 2024-09-20T22:49:33.776 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.776 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:49:33.776 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.777 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:49:33.777 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:23.126433+0000", 2024-09-20T22:49:33.777 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:49:33.777 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.777 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.777 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:49:33.777 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:23.126433+0000", 2024-09-20T22:49:33.777 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:49:33.777 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.777 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.777 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:49:33.777 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:23.126434+0000", 2024-09-20T22:49:33.777 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3200000000000001e-06 2024-09-20T22:49:33.777 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.778 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.778 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:49:33.778 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:23.126438+0000", 2024-09-20T22:49:33.778 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.6959999999999998e-06 2024-09-20T22:49:33.778 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.778 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.778 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:49:33.778 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:23.126448+0000", 2024-09-20T22:49:33.778 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0436e-05 2024-09-20T22:49:33.778 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.778 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.778 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:49:33.778 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:23.126455+0000", 2024-09-20T22:49:33.778 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.7399999999999998e-06 2024-09-20T22:49:33.779 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.779 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.779 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:49:33.779 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:23.126505+0000", 2024-09-20T22:49:33.779 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.9622e-05 2024-09-20T22:49:33.779 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.779 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.779 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:49:33.779 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:23.126598+0000", 2024-09-20T22:49:33.779 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.3905000000000003e-05 2024-09-20T22:49:33.779 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.779 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.779 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 1,4", 2024-09-20T22:49:33.779 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:23.136852+0000", 2024-09-20T22:49:33.780 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010253372 2024-09-20T22:49:33.780 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.780 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.780 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:49:33.780 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:23.146781+0000", 2024-09-20T22:49:33.780 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0099292019999999998 2024-09-20T22:49:33.780 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.780 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.780 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:49:33.780 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:23.146838+0000", 2024-09-20T22:49:33.780 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.7046999999999998e-05 2024-09-20T22:49:33.780 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.780 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.780 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:49:33.781 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:23.154054+0000", 2024-09-20T22:49:33.781 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0072155910000000004 2024-09-20T22:49:33.781 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.781 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.781 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:49:33.781 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:23.154100+0000", 2024-09-20T22:49:33.781 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.6613999999999998e-05 2024-09-20T22:49:33.781 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.781 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.781 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:49:33.781 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:23.154159+0000", 2024-09-20T22:49:33.781 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.8477999999999998e-05 2024-09-20T22:49:33.781 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:49:33.781 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:49:33.781 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:49:33.782 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.782 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.782 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:1079 7.f 7:fa55e5ae:::smithi08477203-7:head [copy-from ver 87] snapc 26=[26,24] ondisk+write+known_if_redirected+supports_pool_eio e170)", 2024-09-20T22:49:33.782 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:44:09.184851+0000", 2024-09-20T22:49:33.782 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 324.55777462999998, 2024-09-20T22:49:33.782 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.028346902, 2024-09-20T22:49:33.782 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:49:33.782 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:49:33.782 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:49:33.782 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:49:33.782 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:49:33.782 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:49:33.782 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1079 2024-09-20T22:49:33.782 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.783 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:49:33.783 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.783 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:49:33.783 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:09.184851+0000", 2024-09-20T22:49:33.783 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:49:33.783 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.783 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.783 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:49:33.783 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:09.184851+0000", 2024-09-20T22:49:33.783 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:49:33.783 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.783 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.783 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:49:33.783 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:09.184852+0000", 2024-09-20T22:49:33.783 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.3399999999999999e-07 2024-09-20T22:49:33.784 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.784 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.784 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:49:33.784 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:09.184888+0000", 2024-09-20T22:49:33.784 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.6112000000000003e-05 2024-09-20T22:49:33.784 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.784 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.784 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:49:33.784 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:09.184895+0000", 2024-09-20T22:49:33.784 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.8539999999999999e-06 2024-09-20T22:49:33.784 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.785 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.785 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:49:33.785 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:09.184926+0000", 2024-09-20T22:49:33.785 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.1293e-05 2024-09-20T22:49:33.785 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.785 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.785 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:49:33.785 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:09.188919+0000", 2024-09-20T22:49:33.785 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0039926830000000003 2024-09-20T22:49:33.785 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.785 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.785 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:49:33.785 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:09.189012+0000", 2024-09-20T22:49:33.785 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.3690999999999995e-05 2024-09-20T22:49:33.785 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.786 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.786 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 3,7", 2024-09-20T22:49:33.786 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:09.205339+0000", 2024-09-20T22:49:33.786 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.016326092 2024-09-20T22:49:33.786 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.786 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.786 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:49:33.786 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:09.209753+0000", 2024-09-20T22:49:33.786 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0044144789999999998 2024-09-20T22:49:33.786 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.786 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.786 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:49:33.786 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:09.212376+0000", 2024-09-20T22:49:33.786 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026232180000000001 2024-09-20T22:49:33.787 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.787 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.787 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:49:33.787 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:09.213093+0000", 2024-09-20T22:49:33.787 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00071704400000000001 2024-09-20T22:49:33.787 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.787 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.787 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:49:33.787 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:09.213141+0000", 2024-09-20T22:49:33.787 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.7991999999999997e-05 2024-09-20T22:49:33.787 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.787 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.787 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:49:33.787 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:09.213198+0000", 2024-09-20T22:49:33.788 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.6910000000000002e-05 2024-09-20T22:49:33.788 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:49:33.788 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:49:33.788 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:49:33.788 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.788 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.788 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:1080 7.f 7:fa55e5ae:::smithi08477203-7:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e170)", 2024-09-20T22:49:33.788 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:44:09.184935+0000", 2024-09-20T22:49:33.788 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 324.55769093800001, 2024-09-20T22:49:33.788 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.024005572999999999, 2024-09-20T22:49:33.788 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:49:33.788 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:49:33.788 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-20T22:49:33.788 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:49:33.789 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:49:33.789 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:49:33.789 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1080 2024-09-20T22:49:33.789 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.789 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:49:33.789 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.789 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:49:33.789 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:09.184935+0000", 2024-09-20T22:49:33.789 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:49:33.789 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.789 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.789 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:49:33.789 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:09.184935+0000", 2024-09-20T22:49:33.789 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:49:33.790 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.790 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.790 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:49:33.790 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:09.184936+0000", 2024-09-20T22:49:33.790 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.6599999999999996e-07 2024-09-20T22:49:33.790 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.790 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.790 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:49:33.790 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:09.184938+0000", 2024-09-20T22:49:33.790 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.5359999999999999e-06 2024-09-20T22:49:33.790 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.790 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.790 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:49:33.790 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:09.184971+0000", 2024-09-20T22:49:33.791 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.2848000000000001e-05 2024-09-20T22:49:33.791 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.791 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.791 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:49:33.791 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:09.184977+0000", 2024-09-20T22:49:33.791 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.0859999999999998e-06 2024-09-20T22:49:33.791 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.791 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.791 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:49:33.791 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:09.189687+0000", 2024-09-20T22:49:33.791 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0047097700000000003 2024-09-20T22:49:33.791 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.791 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.791 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for blocked object", 2024-09-20T22:49:33.792 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:09.189852+0000", 2024-09-20T22:49:33.792 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000165243 2024-09-20T22:49:33.792 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.792 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.792 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:49:33.792 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:09.208617+0000", 2024-09-20T22:49:33.792 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.018765190000000001 2024-09-20T22:49:33.792 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.792 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.792 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:49:33.792 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:09.208772+0000", 2024-09-20T22:49:33.792 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000154318 2024-09-20T22:49:33.792 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.792 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.792 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:49:33.793 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:09.208941+0000", 2024-09-20T22:49:33.793 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000169016 2024-09-20T22:49:33.793 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:49:33.793 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:49:33.793 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:49:33.793 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.793 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.793 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:1583 7.18 7:1e9e2524:::smithi08477203-5:head [copy-from ver 154] snapc 35=[35] ondisk+write+known_if_redirected+supports_pool_eio e201)", 2024-09-20T22:49:33.793 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:44:40.229595+0000", 2024-09-20T22:49:33.793 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 293.513030695, 2024-09-20T22:49:33.793 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.026717541000000001, 2024-09-20T22:49:33.793 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:49:33.793 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:49:33.793 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:49:33.794 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:49:33.794 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:49:33.794 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:49:33.794 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1583 2024-09-20T22:49:33.794 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.794 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:49:33.794 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.794 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:49:33.794 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:40.229595+0000", 2024-09-20T22:49:33.794 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:49:33.794 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.794 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.794 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:49:33.794 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:40.229595+0000", 2024-09-20T22:49:33.795 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:49:33.795 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.795 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.795 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:49:33.795 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:40.229596+0000", 2024-09-20T22:49:33.795 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.68e-07 2024-09-20T22:49:33.795 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.795 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.795 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:49:33.795 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:40.229600+0000", 2024-09-20T22:49:33.795 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.805e-06 2024-09-20T22:49:33.795 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.795 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.795 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:49:33.796 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:40.229606+0000", 2024-09-20T22:49:33.796 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.4300000000000003e-06 2024-09-20T22:49:33.796 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.796 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.796 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:49:33.796 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:40.229611+0000", 2024-09-20T22:49:33.796 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.0250000000000002e-06 2024-09-20T22:49:33.796 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.796 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.796 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:49:33.797 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:40.229661+0000", 2024-09-20T22:49:33.797 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.0343999999999998e-05 2024-09-20T22:49:33.797 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.797 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.797 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:49:33.797 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:40.229737+0000", 2024-09-20T22:49:33.797 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.5444000000000005e-05 2024-09-20T22:49:33.797 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.797 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.797 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 6,7", 2024-09-20T22:49:33.798 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:40.244365+0000", 2024-09-20T22:49:33.798 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014628238999999999 2024-09-20T22:49:33.798 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.798 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.798 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:49:33.798 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:40.249487+0000", 2024-09-20T22:49:33.798 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0051221799999999996 2024-09-20T22:49:33.798 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.798 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.798 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:49:33.798 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:40.252864+0000", 2024-09-20T22:49:33.798 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0033763830000000002 2024-09-20T22:49:33.798 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.799 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.799 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:49:33.799 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:40.256237+0000", 2024-09-20T22:49:33.799 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0033731550000000001 2024-09-20T22:49:33.799 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.799 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.799 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:49:33.799 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:40.256277+0000", 2024-09-20T22:49:33.799 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.0092999999999997e-05 2024-09-20T22:49:33.799 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.799 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.799 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:49:33.799 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:40.256313+0000", 2024-09-20T22:49:33.799 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.5874999999999998e-05 2024-09-20T22:49:33.800 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:49:33.800 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:49:33.800 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:49:33.800 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.800 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.800 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:1584 7.18 7:1e9e2524:::smithi08477203-5:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e201)", 2024-09-20T22:49:33.800 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:44:40.229623+0000", 2024-09-20T22:49:33.800 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 293.51300300100002, 2024-09-20T22:49:33.800 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.019097652, 2024-09-20T22:49:33.801 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:49:33.801 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:49:33.801 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-20T22:49:33.801 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:49:33.801 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:49:33.801 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:49:33.801 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1584 2024-09-20T22:49:33.801 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.801 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:49:33.801 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.802 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:49:33.802 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:40.229623+0000", 2024-09-20T22:49:33.802 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:49:33.802 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.802 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.802 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:49:33.802 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:40.229623+0000", 2024-09-20T22:49:33.802 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:49:33.802 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.802 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.802 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:49:33.802 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:40.229623+0000", 2024-09-20T22:49:33.802 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.7300000000000002e-07 2024-09-20T22:49:33.802 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.803 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.803 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:49:33.803 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:40.229625+0000", 2024-09-20T22:49:33.803 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.199e-06 2024-09-20T22:49:33.803 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.803 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.803 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:49:33.803 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:40.229631+0000", 2024-09-20T22:49:33.803 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.1599999999999997e-06 2024-09-20T22:49:33.803 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.804 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.804 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:49:33.804 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:40.229636+0000", 2024-09-20T22:49:33.804 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.7780000000000004e-06 2024-09-20T22:49:33.804 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.804 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.804 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:49:33.804 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:40.229830+0000", 2024-09-20T22:49:33.804 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00019321 2024-09-20T22:49:33.804 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.804 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.804 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for blocked object", 2024-09-20T22:49:33.804 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:40.229857+0000", 2024-09-20T22:49:33.805 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.7030999999999999e-05 2024-09-20T22:49:33.805 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.805 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.805 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:49:33.805 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:40.248577+0000", 2024-09-20T22:49:33.805 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.018720554 2024-09-20T22:49:33.805 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.805 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.805 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:49:33.805 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:40.248643+0000", 2024-09-20T22:49:33.805 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.5361e-05 2024-09-20T22:49:33.805 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.805 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.805 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:49:33.805 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:40.248721+0000", 2024-09-20T22:49:33.806 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.7985999999999995e-05 2024-09-20T22:49:33.806 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:49:33.806 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:49:33.806 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:49:33.806 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.806 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.806 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:3965 7.f 7:fa55e5ae:::smithi08477203-7:head [copy-from ver 838] snapc a8=[a8,a7,a6,a5,a4,a3,a0,9f,9e,9d,9c,9b,9a,96,95,94,93,92,8e,8d,8a,88,87,86,80,7f,73] ondisk+write+known_if_redirected+supports_pool_eio e406)", 2024-09-20T22:49:33.806 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:48:06.215099+0000", 2024-09-20T22:49:33.806 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 87.527527375999995, 2024-09-20T22:49:33.806 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.021929697000000001, 2024-09-20T22:49:33.806 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:49:33.806 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:49:33.806 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:49:33.807 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:49:33.807 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:49:33.807 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:49:33.807 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 3965 2024-09-20T22:49:33.807 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.807 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:49:33.807 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.807 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:49:33.807 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:06.215099+0000", 2024-09-20T22:49:33.807 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:49:33.807 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.807 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.807 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:49:33.807 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:06.215099+0000", 2024-09-20T22:49:33.807 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:49:33.808 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.808 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.808 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:49:33.808 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:06.215100+0000", 2024-09-20T22:49:33.808 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.6980000000000001e-06 2024-09-20T22:49:33.808 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.808 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.808 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:49:33.808 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:06.215107+0000", 2024-09-20T22:49:33.808 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.2990000000000002e-06 2024-09-20T22:49:33.808 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.808 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.808 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:49:33.808 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:06.215118+0000", 2024-09-20T22:49:33.808 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.173e-05 2024-09-20T22:49:33.809 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.809 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.809 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:49:33.809 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:06.215128+0000", 2024-09-20T22:49:33.809 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.5920000000000007e-06 2024-09-20T22:49:33.809 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.809 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.809 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:49:33.809 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:06.215190+0000", 2024-09-20T22:49:33.809 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.1970000000000005e-05 2024-09-20T22:49:33.809 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.809 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.809 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:49:33.809 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:06.215295+0000", 2024-09-20T22:49:33.809 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000105027 2024-09-20T22:49:33.810 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.810 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.810 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 3,7", 2024-09-20T22:49:33.810 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:06.226459+0000", 2024-09-20T22:49:33.810 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011164431000000001 2024-09-20T22:49:33.810 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.810 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.810 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:49:33.810 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:06.231688+0000", 2024-09-20T22:49:33.810 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0052288739999999997 2024-09-20T22:49:33.810 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.810 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.810 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:49:33.810 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:06.235132+0000", 2024-09-20T22:49:33.810 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0034440859999999998 2024-09-20T22:49:33.811 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.811 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.811 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:49:33.811 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:06.236877+0000", 2024-09-20T22:49:33.811 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00174515 2024-09-20T22:49:33.811 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.811 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.811 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:49:33.811 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:06.236974+0000", 2024-09-20T22:49:33.811 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.6951000000000001e-05 2024-09-20T22:49:33.811 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.811 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.811 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:49:33.811 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:06.237028+0000", 2024-09-20T22:49:33.811 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.3888999999999999e-05 2024-09-20T22:49:33.812 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:49:33.812 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:49:33.812 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:49:33.812 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.812 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.812 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:4311 7.17 7:ed54065f:::smithi08477203-12:head [append 0~0] snapc b1=[b1,b0,af,ae,ad,ab,a9,a7,a6,a5,9f,9e,9d,9c,9a,96,95,8e,87,86] ondisk+write+known_if_redirected+supports_pool_eio e433)", 2024-09-20T22:49:33.812 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:48:33.213556+0000", 2024-09-20T22:49:33.812 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 60.529069524999997, 2024-09-20T22:49:33.812 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0280906110000001, 2024-09-20T22:49:33.812 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:49:33.812 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:49:33.812 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:49:33.812 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:49:33.812 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:49:33.812 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:49:33.813 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 4311 2024-09-20T22:49:33.813 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.813 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:49:33.813 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.813 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:49:33.813 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:33.213556+0000", 2024-09-20T22:49:33.813 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:49:33.813 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.813 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.813 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:49:33.813 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:33.213556+0000", 2024-09-20T22:49:33.813 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:49:33.813 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.813 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.813 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:49:33.814 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:33.213558+0000", 2024-09-20T22:49:33.814 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.4470000000000001e-06 2024-09-20T22:49:33.814 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.814 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.814 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:49:33.814 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:33.213563+0000", 2024-09-20T22:49:33.814 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.6840000000000004e-06 2024-09-20T22:49:33.814 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.814 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.814 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:49:33.814 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:33.213573+0000", 2024-09-20T22:49:33.814 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0873000000000001e-05 2024-09-20T22:49:33.814 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.814 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.815 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:49:33.815 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:33.213584+0000", 2024-09-20T22:49:33.815 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0852e-05 2024-09-20T22:49:33.815 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.815 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.815 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:49:33.815 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:33.213633+0000", 2024-09-20T22:49:33.815 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.8683999999999997e-05 2024-09-20T22:49:33.815 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.815 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.815 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-09-20T22:49:33.815 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:33.213646+0000", 2024-09-20T22:49:33.815 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3205e-05 2024-09-20T22:49:33.815 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.816 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.816 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:49:33.816 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:34.226325+0000", 2024-09-20T22:49:33.816 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0126793300000001 2024-09-20T22:49:33.816 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.816 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.816 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for degraded object", 2024-09-20T22:49:33.816 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:34.228377+0000", 2024-09-20T22:49:33.816 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0020512540000000002 2024-09-20T22:49:33.816 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.816 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.816 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:49:33.816 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:34.239166+0000", 2024-09-20T22:49:33.816 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010789029 2024-09-20T22:49:33.817 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.817 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.817 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:49:33.817 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:34.239275+0000", 2024-09-20T22:49:33.817 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00010938900000000001 2024-09-20T22:49:33.817 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.817 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.817 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 3,6", 2024-09-20T22:49:33.817 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:34.239569+0000", 2024-09-20T22:49:33.817 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00029410699999999999 2024-09-20T22:49:33.817 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.817 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.817 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:49:33.817 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:34.241371+0000", 2024-09-20T22:49:33.817 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00180171 2024-09-20T22:49:33.818 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.818 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.818 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:49:33.818 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:34.241393+0000", 2024-09-20T22:49:33.818 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.1928000000000001e-05 2024-09-20T22:49:33.818 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.818 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.818 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:49:33.818 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:34.241465+0000", 2024-09-20T22:49:33.818 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.1799999999999997e-05 2024-09-20T22:49:33.818 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.818 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.818 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:49:33.818 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:34.241528+0000", 2024-09-20T22:49:33.818 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.2946000000000003e-05 2024-09-20T22:49:33.819 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.819 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.819 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:49:33.819 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:34.241647+0000", 2024-09-20T22:49:33.819 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000119373 2024-09-20T22:49:33.819 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:49:33.819 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:49:33.819 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:49:33.819 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.819 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.819 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:4312 7.17 7:ed54065f:::smithi08477203-12:head [rollback 96] snapc b1=[b1,b0,af,ae,ad,ab,a9,a7,a6,a5,9f,9e,9d,9c,9a,96,95,8e,87,86] ondisk+write+known_if_redirected+supports_pool_eio e433)", 2024-09-20T22:49:33.819 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:48:33.213603+0000", 2024-09-20T22:49:33.819 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 60.529023103, 2024-09-20T22:49:33.819 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0286283940000001, 2024-09-20T22:49:33.819 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:49:33.820 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:49:33.820 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:49:33.820 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:49:33.820 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:49:33.820 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:49:33.820 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 4312 2024-09-20T22:49:33.820 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.820 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:49:33.820 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.820 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:49:33.820 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:33.213603+0000", 2024-09-20T22:49:33.820 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:49:33.820 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.820 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.820 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:49:33.821 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:33.213603+0000", 2024-09-20T22:49:33.821 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:49:33.821 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.821 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.821 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:49:33.821 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:33.213603+0000", 2024-09-20T22:49:33.821 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.9200000000000001e-07 2024-09-20T22:49:33.821 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.821 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.821 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:49:33.821 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:33.213606+0000", 2024-09-20T22:49:33.821 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.6390000000000001e-06 2024-09-20T22:49:33.821 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.821 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.821 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:49:33.822 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:33.213612+0000", 2024-09-20T22:49:33.822 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.5429999999999999e-06 2024-09-20T22:49:33.822 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.822 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.822 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:49:33.822 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:33.213618+0000", 2024-09-20T22:49:33.822 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.4249999999999997e-06 2024-09-20T22:49:33.822 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.822 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.822 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:49:33.822 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:33.213674+0000", 2024-09-20T22:49:33.822 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.6341000000000001e-05 2024-09-20T22:49:33.822 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.822 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.822 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-09-20T22:49:33.823 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:33.214118+0000", 2024-09-20T22:49:33.823 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00044406600000000001 2024-09-20T22:49:33.823 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.823 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.823 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:49:33.823 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:34.228938+0000", 2024-09-20T22:49:33.823 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0148198470000001 2024-09-20T22:49:33.823 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.823 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.823 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for degraded object", 2024-09-20T22:49:33.823 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:34.229016+0000", 2024-09-20T22:49:33.823 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.7383000000000001e-05 2024-09-20T22:49:33.823 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.823 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.824 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:49:33.824 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:34.240634+0000", 2024-09-20T22:49:33.824 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011618277999999999 2024-09-20T22:49:33.824 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.824 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.824 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:49:33.824 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:34.240749+0000", 2024-09-20T22:49:33.824 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00011473500000000001 2024-09-20T22:49:33.824 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.824 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.824 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 3,6", 2024-09-20T22:49:33.824 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:34.241041+0000", 2024-09-20T22:49:33.824 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00029246699999999998 2024-09-20T22:49:33.824 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.824 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.825 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:49:33.825 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:34.242022+0000", 2024-09-20T22:49:33.825 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00098068999999999999 2024-09-20T22:49:33.825 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.825 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.825 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:49:33.825 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:34.242068+0000", 2024-09-20T22:49:33.825 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.6581e-05 2024-09-20T22:49:33.825 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.825 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.825 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:49:33.825 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:34.242081+0000", 2024-09-20T22:49:33.825 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2709e-05 2024-09-20T22:49:33.825 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.826 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.826 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:49:33.826 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:34.242116+0000", 2024-09-20T22:49:33.826 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.4579000000000002e-05 2024-09-20T22:49:33.826 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.826 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.826 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:49:33.826 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:34.242231+0000", 2024-09-20T22:49:33.826 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00011561900000000001 2024-09-20T22:49:33.826 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:49:33.826 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:49:33.826 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:49:33.826 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.826 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.826 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:4318 7.f 7:fa55e5ae:::smithi08477203-7:head [copy-from ver 556] snapc b1=[b1,b0,af,ae,ad,ab,a9,a7,a6,a5,9f,9e,9d,9c,9a,96,95,8e,87,86] ondisk+write+known_if_redirected+supports_pool_eio e433)", 2024-09-20T22:49:33.827 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:48:33.213673+0000", 2024-09-20T22:49:33.827 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 60.528952842999999, 2024-09-20T22:49:33.827 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.019496726999999998, 2024-09-20T22:49:33.827 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:49:33.827 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:49:33.827 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:49:33.827 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:49:33.827 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:49:33.827 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:49:33.827 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 4318 2024-09-20T22:49:33.827 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.827 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:49:33.827 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.827 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:49:33.828 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:33.213673+0000", 2024-09-20T22:49:33.828 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:49:33.828 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.828 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.828 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:49:33.828 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:33.213673+0000", 2024-09-20T22:49:33.828 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:49:33.828 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.828 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.828 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:49:33.828 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:33.213674+0000", 2024-09-20T22:49:33.828 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.6600000000000002e-07 2024-09-20T22:49:33.828 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.829 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.829 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:49:33.829 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:33.213678+0000", 2024-09-20T22:49:33.829 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.3229999999999999e-06 2024-09-20T22:49:33.829 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.829 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.829 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:49:33.829 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:33.213685+0000", 2024-09-20T22:49:33.829 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.1400000000000002e-06 2024-09-20T22:49:33.829 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.829 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.829 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:49:33.829 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:33.213691+0000", 2024-09-20T22:49:33.829 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.7819999999999999e-06 2024-09-20T22:49:33.830 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.830 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.830 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:49:33.830 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:33.214153+0000", 2024-09-20T22:49:33.830 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00046270099999999999 2024-09-20T22:49:33.830 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.830 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.830 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:49:33.830 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:33.214276+0000", 2024-09-20T22:49:33.830 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00012271999999999999 2024-09-20T22:49:33.830 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.830 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.830 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 3,7", 2024-09-20T22:49:33.830 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:33.220974+0000", 2024-09-20T22:49:33.830 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0066979199999999996 2024-09-20T22:49:33.831 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.831 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.831 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:49:33.831 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:33.230425+0000", 2024-09-20T22:49:33.831 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0094512309999999992 2024-09-20T22:49:33.831 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.831 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.831 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:49:33.831 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:33.232841+0000", 2024-09-20T22:49:33.831 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0024154340000000002 2024-09-20T22:49:33.831 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.831 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.831 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:49:33.831 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:33.233039+0000", 2024-09-20T22:49:33.832 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000197939 2024-09-20T22:49:33.832 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.832 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.832 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:49:33.832 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:33.233108+0000", 2024-09-20T22:49:33.832 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.8956999999999997e-05 2024-09-20T22:49:33.832 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.832 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.832 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:49:33.832 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:33.233170+0000", 2024-09-20T22:49:33.832 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.2113999999999995e-05 2024-09-20T22:49:33.832 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:49:33.832 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:49:33.832 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:49:33.833 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.833 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.833 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:4413 7.4 7:24b1a612:::smithi08477203-43:head [copy-from ver 706] snapc b7=[b7,b6,b5,b4,b3,b2,b1,b0,af,ae,a9,a7,a6,a5,9e,9c,9a,96,95,87,86] ondisk+write+known_if_redirected+supports_pool_eio e444)", 2024-09-20T22:49:33.833 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:48:44.258741+0000", 2024-09-20T22:49:33.833 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 49.48388456, 2024-09-20T22:49:33.833 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.021489458999999999, 2024-09-20T22:49:33.833 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:49:33.833 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:49:33.833 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:49:33.833 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:49:33.833 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:49:33.833 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:49:33.833 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 4413 2024-09-20T22:49:33.833 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.833 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:49:33.834 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.834 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:49:33.834 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:44.258741+0000", 2024-09-20T22:49:33.834 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:49:33.834 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.834 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.834 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:49:33.834 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:44.258741+0000", 2024-09-20T22:49:33.834 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:49:33.834 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.834 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.834 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:49:33.834 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:44.258746+0000", 2024-09-20T22:49:33.834 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.6450000000000004e-06 2024-09-20T22:49:33.835 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.835 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.835 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:49:33.835 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:44.258753+0000", 2024-09-20T22:49:33.835 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.4089999999999997e-06 2024-09-20T22:49:33.835 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.835 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.835 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:49:33.835 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:44.258769+0000", 2024-09-20T22:49:33.835 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.5546999999999999e-05 2024-09-20T22:49:33.835 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.835 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.835 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:49:33.835 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:44.258776+0000", 2024-09-20T22:49:33.836 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.1069999999999999e-06 2024-09-20T22:49:33.836 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.836 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.836 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:49:33.836 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:44.258832+0000", 2024-09-20T22:49:33.836 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.6011000000000003e-05 2024-09-20T22:49:33.836 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.836 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.836 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:49:33.836 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:44.258945+0000", 2024-09-20T22:49:33.836 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00011262 2024-09-20T22:49:33.836 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.836 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.836 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 1,4", 2024-09-20T22:49:33.836 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:44.270466+0000", 2024-09-20T22:49:33.836 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011521747000000001 2024-09-20T22:49:33.837 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.837 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.837 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:49:33.837 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:44.277780+0000", 2024-09-20T22:49:33.837 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0073139600000000004 2024-09-20T22:49:33.837 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.837 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.837 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:49:33.837 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:44.280147+0000", 2024-09-20T22:49:33.837 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0023665940000000001 2024-09-20T22:49:33.837 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.837 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.837 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:49:33.837 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:44.280157+0000", 2024-09-20T22:49:33.838 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0349e-05 2024-09-20T22:49:33.838 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.838 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.838 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:49:33.838 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:44.280194+0000", 2024-09-20T22:49:33.838 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.6300000000000001e-05 2024-09-20T22:49:33.838 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.838 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.838 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:49:33.838 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:44.280231+0000", 2024-09-20T22:49:33.838 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.7169999999999998e-05 2024-09-20T22:49:33.838 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:49:33.838 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:49:33.838 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:49:33.838 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.839 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.839 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:4414 7.4 7:24b1a612:::smithi08477203-43:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e444)", 2024-09-20T22:49:33.839 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:48:44.258795+0000", 2024-09-20T22:49:33.839 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 49.483830437999998, 2024-09-20T22:49:33.839 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.017457381000000001, 2024-09-20T22:49:33.839 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:49:33.839 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:49:33.839 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-20T22:49:33.839 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:49:33.839 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:49:33.839 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:49:33.839 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 4414 2024-09-20T22:49:33.839 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.840 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:49:33.840 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.840 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:49:33.840 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:44.258795+0000", 2024-09-20T22:49:33.840 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:49:33.840 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.840 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.840 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:49:33.840 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:44.258795+0000", 2024-09-20T22:49:33.840 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:49:33.840 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.840 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.840 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:49:33.840 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:44.258796+0000", 2024-09-20T22:49:33.841 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.7899999999999998e-07 2024-09-20T22:49:33.841 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.841 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.841 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:49:33.841 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:44.258799+0000", 2024-09-20T22:49:33.841 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.0019999999999998e-06 2024-09-20T22:49:33.841 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.841 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.841 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:49:33.841 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:44.258805+0000", 2024-09-20T22:49:33.841 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.3849999999999999e-06 2024-09-20T22:49:33.841 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.841 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.841 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:49:33.842 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:44.258812+0000", 2024-09-20T22:49:33.842 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.5039999999999999e-06 2024-09-20T22:49:33.842 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.842 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.842 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:49:33.842 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:44.259378+0000", 2024-09-20T22:49:33.842 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00056618800000000002 2024-09-20T22:49:33.842 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.842 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.842 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for blocked object", 2024-09-20T22:49:33.842 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:44.259406+0000", 2024-09-20T22:49:33.842 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.7962000000000001e-05 2024-09-20T22:49:33.842 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.843 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.843 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:49:33.843 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:44.276053+0000", 2024-09-20T22:49:33.843 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.016647114000000001 2024-09-20T22:49:33.843 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.843 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.843 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:49:33.843 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:44.276146+0000", 2024-09-20T22:49:33.843 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.2613000000000003e-05 2024-09-20T22:49:33.843 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.843 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.843 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:49:33.843 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:44.276253+0000", 2024-09-20T22:49:33.843 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00010703400000000001 2024-09-20T22:49:33.843 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:49:33.844 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:49:33.844 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:49:33.844 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.844 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.844 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:4415 7.16 7.fa65a116 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e444)", 2024-09-20T22:49:33.844 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:48:44.266352+0000", 2024-09-20T22:49:33.844 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 49.476273786999997, 2024-09-20T22:49:33.844 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.96293146200000002, 2024-09-20T22:49:33.844 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:49:33.844 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:49:33.844 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "reached pg", 2024-09-20T22:49:33.844 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:49:33.844 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:49:33.844 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:49:33.845 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 4415 2024-09-20T22:49:33.845 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.845 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:49:33.845 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.845 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:49:33.845 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:44.266352+0000", 2024-09-20T22:49:33.845 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:49:33.845 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.845 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.845 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:49:33.845 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:44.266352+0000", 2024-09-20T22:49:33.845 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:49:33.845 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.845 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.845 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:49:33.845 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:44.266354+0000", 2024-09-20T22:49:33.846 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.733e-06 2024-09-20T22:49:33.846 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.846 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.846 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:49:33.846 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:44.266610+0000", 2024-09-20T22:49:33.846 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000256352 2024-09-20T22:49:33.846 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.846 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.846 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:49:33.846 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:44.266626+0000", 2024-09-20T22:49:33.846 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.5347000000000001e-05 2024-09-20T22:49:33.846 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.846 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.846 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:49:33.847 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:44.266634+0000", 2024-09-20T22:49:33.847 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.5390000000000007e-06 2024-09-20T22:49:33.847 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.847 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.847 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:49:33.847 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:44.266701+0000", 2024-09-20T22:49:33.847 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.6908000000000004e-05 2024-09-20T22:49:33.847 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.847 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.847 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-09-20T22:49:33.847 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:44.266710+0000", 2024-09-20T22:49:33.847 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.2920000000000001e-06 2024-09-20T22:49:33.847 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.847 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.847 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:49:33.848 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:45.229222+0000", 2024-09-20T22:49:33.848 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.96251129899999999 2024-09-20T22:49:33.848 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.848 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.848 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:49:33.848 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:45.229284+0000", 2024-09-20T22:49:33.848 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.1991999999999999e-05 2024-09-20T22:49:33.848 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:49:33.848 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:49:33.848 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:49:33.848 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.848 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.848 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:4416 7.16 7.fa65a116 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e444)", 2024-09-20T22:49:33.848 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:48:44.273462+0000", 2024-09-20T22:49:33.848 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 49.469164356999997, 2024-09-20T22:49:33.849 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.95612304000000004, 2024-09-20T22:49:33.849 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:49:33.849 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:49:33.849 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "reached pg", 2024-09-20T22:49:33.849 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:49:33.849 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:49:33.849 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:49:33.849 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 4416 2024-09-20T22:49:33.849 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.849 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:49:33.849 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.849 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:49:33.849 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:44.273462+0000", 2024-09-20T22:49:33.849 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:49:33.849 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.850 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.850 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:49:33.850 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:44.273462+0000", 2024-09-20T22:49:33.850 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:49:33.850 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.850 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.850 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:49:33.850 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:44.273464+0000", 2024-09-20T22:49:33.850 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.3609999999999999e-06 2024-09-20T22:49:33.850 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.850 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.850 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:49:33.850 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:44.273675+0000", 2024-09-20T22:49:33.850 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00021111400000000001 2024-09-20T22:49:33.851 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.851 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.851 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:49:33.851 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:44.273697+0000", 2024-09-20T22:49:33.851 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.2305000000000001e-05 2024-09-20T22:49:33.851 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.851 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.851 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:49:33.851 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:44.273708+0000", 2024-09-20T22:49:33.851 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0669999999999999e-05 2024-09-20T22:49:33.851 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.851 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.851 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:49:33.851 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:44.273865+0000", 2024-09-20T22:49:33.851 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000156766 2024-09-20T22:49:33.852 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.852 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.852 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-09-20T22:49:33.852 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:44.273878+0000", 2024-09-20T22:49:33.852 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3178e-05 2024-09-20T22:49:33.852 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.852 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.852 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:49:33.852 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:45.229577+0000", 2024-09-20T22:49:33.852 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.95569872600000005 2024-09-20T22:49:33.852 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.852 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.852 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:49:33.852 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:45.229585+0000", 2024-09-20T22:49:33.853 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.9200000000000004e-06 2024-09-20T22:49:33.853 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:49:33.853 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:49:33.853 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:49:33.853 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.853 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.853 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:4417 7.16 7.fa65a116 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e444)", 2024-09-20T22:49:33.853 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:48:44.273740+0000", 2024-09-20T22:49:33.853 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 49.468885677000003, 2024-09-20T22:49:33.853 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.95591940399999997, 2024-09-20T22:49:33.853 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:49:33.853 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:49:33.853 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "reached pg", 2024-09-20T22:49:33.853 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:49:33.854 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:49:33.854 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:49:33.854 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 4417 2024-09-20T22:49:33.854 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.854 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:49:33.854 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.854 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:49:33.854 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:44.273740+0000", 2024-09-20T22:49:33.854 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:49:33.854 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.854 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.854 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:49:33.854 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:44.273740+0000", 2024-09-20T22:49:33.854 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:49:33.855 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.855 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.855 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:49:33.855 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:44.273741+0000", 2024-09-20T22:49:33.855 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.06e-07 2024-09-20T22:49:33.855 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.855 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.855 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:49:33.855 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:44.273746+0000", 2024-09-20T22:49:33.855 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.7129999999999996e-06 2024-09-20T22:49:33.855 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.855 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.855 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:49:33.855 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:44.273751+0000", 2024-09-20T22:49:33.855 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.8309999999999997e-06 2024-09-20T22:49:33.855 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.856 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.856 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:49:33.856 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:44.273756+0000", 2024-09-20T22:49:33.856 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.6859999999999997e-06 2024-09-20T22:49:33.856 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.856 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.856 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:49:33.856 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:44.274433+0000", 2024-09-20T22:49:33.856 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00067743000000000002 2024-09-20T22:49:33.856 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.856 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.856 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-09-20T22:49:33.856 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:44.274446+0000", 2024-09-20T22:49:33.856 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2737e-05 2024-09-20T22:49:33.856 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.857 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.857 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:49:33.857 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:45.229650+0000", 2024-09-20T22:49:33.857 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.95520395499999999 2024-09-20T22:49:33.857 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.857 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.857 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:49:33.857 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:45.229660+0000", 2024-09-20T22:49:33.857 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.4460000000000007e-06 2024-09-20T22:49:33.857 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:49:33.857 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:49:33.857 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:49:33.857 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.857 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.858 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:4418 7.16 7.fa65a116 (undecoded) ondisk+read+rwordered+known_if_redirected+supports_pool_eio e444)", 2024-09-20T22:49:33.858 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:48:44.273764+0000", 2024-09-20T22:49:33.858 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 49.468862023, 2024-09-20T22:49:33.858 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.95592935400000001, 2024-09-20T22:49:33.858 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:49:33.858 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:49:33.858 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "reached pg", 2024-09-20T22:49:33.858 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:49:33.858 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:49:33.858 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:49:33.858 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 4418 2024-09-20T22:49:33.858 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.858 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:49:33.858 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.858 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:49:33.859 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:44.273764+0000", 2024-09-20T22:49:33.859 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:49:33.859 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.859 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.859 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:49:33.859 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:44.273764+0000", 2024-09-20T22:49:33.859 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:49:33.859 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.859 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.859 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:49:33.859 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:44.273764+0000", 2024-09-20T22:49:33.859 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.2e-07 2024-09-20T22:49:33.859 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.859 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.859 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:49:33.860 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:44.273773+0000", 2024-09-20T22:49:33.860 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.0489999999999993e-06 2024-09-20T22:49:33.860 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.860 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.860 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:49:33.860 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:44.273780+0000", 2024-09-20T22:49:33.860 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.195e-06 2024-09-20T22:49:33.860 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.860 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.860 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:49:33.860 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:44.273784+0000", 2024-09-20T22:49:33.860 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.245e-06 2024-09-20T22:49:33.860 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.860 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.861 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:49:33.861 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:44.274475+0000", 2024-09-20T22:49:33.861 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00069082200000000001 2024-09-20T22:49:33.861 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.861 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.861 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-09-20T22:49:33.861 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:44.274484+0000", 2024-09-20T22:49:33.861 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.6940000000000007e-06 2024-09-20T22:49:33.861 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.861 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.861 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:49:33.861 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:45.229685+0000", 2024-09-20T22:49:33.861 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.95520094 2024-09-20T22:49:33.861 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.862 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.862 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:49:33.862 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:45.229693+0000", 2024-09-20T22:49:33.862 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.9889999999999992e-06 2024-09-20T22:49:33.862 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:49:33.862 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:49:33.862 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:49:33.862 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.862 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.862 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:4415 7.6 7:6885a65f:::smithi08477203-14:head [write 685884~560628] snapc b7=[b7,b6,b5,b4,b3,b2,b1,b0,af,ae,a9,a7,a6,a5,9e,9c,9a,96,95,87,86] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e446)", 2024-09-20T22:49:33.862 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:48:46.223519+0000", 2024-09-20T22:49:33.862 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 47.519106968000003, 2024-09-20T22:49:33.862 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0194105410000001, 2024-09-20T22:49:33.862 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:49:33.862 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:49:33.863 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:49:33.863 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:49:33.863 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:49:33.863 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:49:33.863 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 4415 2024-09-20T22:49:33.863 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.863 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:49:33.863 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.863 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:49:33.863 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:46.223519+0000", 2024-09-20T22:49:33.863 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:49:33.863 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.863 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.863 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:49:33.864 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:46.223519+0000", 2024-09-20T22:49:33.864 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:49:33.864 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.864 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.864 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:49:33.864 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:46.223521+0000", 2024-09-20T22:49:33.864 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.7260000000000001e-06 2024-09-20T22:49:33.864 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.864 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.864 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:49:33.864 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:46.223861+0000", 2024-09-20T22:49:33.864 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00034058200000000003 2024-09-20T22:49:33.864 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.864 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.865 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:49:33.865 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:46.223874+0000", 2024-09-20T22:49:33.865 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2777e-05 2024-09-20T22:49:33.865 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.865 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.865 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:49:33.865 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:46.223884+0000", 2024-09-20T22:49:33.865 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0326000000000001e-05 2024-09-20T22:49:33.865 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.865 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.865 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:49:33.865 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:46.228721+0000", 2024-09-20T22:49:33.865 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0048363470000000004 2024-09-20T22:49:33.865 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.865 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.866 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-09-20T22:49:33.866 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:49:33.867 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:46.228734+0000", 2024-09-20T22:49:33.867 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3219e-05 2024-09-20T22:49:33.867 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.867 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.867 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:49:33.867 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:47.235776+0000", 2024-09-20T22:49:33.867 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0070420579999999 2024-09-20T22:49:33.867 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.867 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.868 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:49:33.868 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:47.235914+0000", 2024-09-20T22:49:33.868 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013761599999999999 2024-09-20T22:49:33.868 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.868 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.868 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 3,7", 2024-09-20T22:49:33.868 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:47.236144+0000", 2024-09-20T22:49:33.868 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000230165 2024-09-20T22:49:33.868 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.868 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.868 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:49:33.868 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:47.242485+0000", 2024-09-20T22:49:33.868 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0063409929999999996 2024-09-20T22:49:33.868 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.869 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.869 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:49:33.869 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:47.242500+0000", 2024-09-20T22:49:33.869 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.5676e-05 2024-09-20T22:49:33.869 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.869 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.869 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:49:33.869 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:47.242790+0000", 2024-09-20T22:49:33.869 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00028984699999999998 2024-09-20T22:49:33.869 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.869 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.869 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:49:33.869 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:47.242860+0000", 2024-09-20T22:49:33.869 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.9765999999999996e-05 2024-09-20T22:49:33.870 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.870 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.870 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:49:33.870 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:47.242929+0000", 2024-09-20T22:49:33.870 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.9443000000000006e-05 2024-09-20T22:49:33.870 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:49:33.870 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:49:33.870 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:49:33.870 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.870 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.870 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:4416 7.6 7:6885a65f:::smithi08477203-14:head [write 2046274~239610] snapc b7=[b7,b6,b5,b4,b3,b2,b1,b0,af,ae,a9,a7,a6,a5,9e,9c,9a,96,95,87,86] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e446)", 2024-09-20T22:49:33.870 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:48:46.225256+0000", 2024-09-20T22:49:33.870 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 47.517369674999998, 2024-09-20T22:49:33.870 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0189633840000001, 2024-09-20T22:49:33.871 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:49:33.871 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:49:33.871 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:49:33.871 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:49:33.871 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:49:33.871 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:49:33.871 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 4416 2024-09-20T22:49:33.871 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.871 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:49:33.871 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.871 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:49:33.871 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:46.225256+0000", 2024-09-20T22:49:33.871 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:49:33.871 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.872 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.872 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:49:33.872 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:46.225256+0000", 2024-09-20T22:49:33.872 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:49:33.872 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.872 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.872 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:49:33.872 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:46.225258+0000", 2024-09-20T22:49:33.872 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.279e-06 2024-09-20T22:49:33.872 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.872 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.872 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:49:33.872 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:46.225882+0000", 2024-09-20T22:49:33.872 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00062460200000000003 2024-09-20T22:49:33.873 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.873 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.873 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:49:33.873 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:46.225894+0000", 2024-09-20T22:49:33.873 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2306999999999999e-05 2024-09-20T22:49:33.873 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.873 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.873 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:49:33.873 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:46.225908+0000", 2024-09-20T22:49:33.873 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3643e-05 2024-09-20T22:49:33.873 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.873 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.873 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:49:33.873 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:46.228769+0000", 2024-09-20T22:49:33.873 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002860656 2024-09-20T22:49:33.874 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.874 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.874 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-09-20T22:49:33.874 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:46.228778+0000", 2024-09-20T22:49:33.874 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.8370000000000003e-06 2024-09-20T22:49:33.874 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.874 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.874 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:49:33.874 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:47.237586+0000", 2024-09-20T22:49:33.874 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.008808207 2024-09-20T22:49:33.874 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.874 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.874 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:49:33.874 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:47.237766+0000", 2024-09-20T22:49:33.875 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00018071600000000001 2024-09-20T22:49:33.875 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.875 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.875 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 3,7", 2024-09-20T22:49:33.875 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:47.238007+0000", 2024-09-20T22:49:33.875 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00024016400000000001 2024-09-20T22:49:33.875 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.875 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.875 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:49:33.875 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:47.242507+0000", 2024-09-20T22:49:33.875 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0045000539999999999 2024-09-20T22:49:33.875 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.875 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.875 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:49:33.876 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:47.243911+0000", 2024-09-20T22:49:33.876 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001403856 2024-09-20T22:49:33.876 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.876 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.876 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:49:33.876 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:47.244094+0000", 2024-09-20T22:49:33.876 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000183239 2024-09-20T22:49:33.876 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.876 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.876 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:49:33.876 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:47.244161+0000", 2024-09-20T22:49:33.876 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.7688000000000007e-05 2024-09-20T22:49:33.876 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.876 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.877 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:49:33.877 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:47.244220+0000", 2024-09-20T22:49:33.877 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.8136e-05 2024-09-20T22:49:33.877 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:49:33.877 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:49:33.877 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:49:33.877 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.877 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.877 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:4417 7.6 7:6885a65f:::smithi08477203-14:head [setxattr _header (58) in=7b,truncate 2285884] snapc b7=[b7,b6,b5,b4,b3,b2,b1,b0,af,ae,a9,a7,a6,a5,9e,9c,9a,96,95,87,86] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e446)", 2024-09-20T22:49:33.877 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:48:46.225923+0000", 2024-09-20T22:49:33.877 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 47.516703063000001, 2024-09-20T22:49:33.877 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.021878477, 2024-09-20T22:49:33.877 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:49:33.877 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:49:33.877 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:49:33.878 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:49:33.878 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:49:33.878 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:49:33.878 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 4417 2024-09-20T22:49:33.878 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.878 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:49:33.878 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.878 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:49:33.878 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:46.225923+0000", 2024-09-20T22:49:33.878 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:49:33.878 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.878 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.878 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:49:33.878 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:46.225923+0000", 2024-09-20T22:49:33.879 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:49:33.879 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.879 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.879 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:49:33.879 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:46.225924+0000", 2024-09-20T22:49:33.879 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.8199999999999999e-07 2024-09-20T22:49:33.879 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.879 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.879 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:49:33.879 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:46.225928+0000", 2024-09-20T22:49:33.879 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.2409999999999997e-06 2024-09-20T22:49:33.879 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.879 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.879 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:49:33.879 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:46.225934+0000", 2024-09-20T22:49:33.880 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.353e-06 2024-09-20T22:49:33.880 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.880 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.880 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:49:33.880 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:46.225939+0000", 2024-09-20T22:49:33.880 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.3299999999999998e-06 2024-09-20T22:49:33.880 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.880 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.880 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:49:33.880 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:46.228811+0000", 2024-09-20T22:49:33.880 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0028720199999999999 2024-09-20T22:49:33.880 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.880 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.880 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-09-20T22:49:33.881 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:46.228819+0000", 2024-09-20T22:49:33.881 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.875e-06 2024-09-20T22:49:33.881 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.881 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.881 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:49:33.881 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:47.241348+0000", 2024-09-20T22:49:33.881 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0125285070000001 2024-09-20T22:49:33.881 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.881 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.881 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:49:33.881 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:47.241459+0000", 2024-09-20T22:49:33.881 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000111622 2024-09-20T22:49:33.881 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.881 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.881 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 3,7", 2024-09-20T22:49:33.882 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:47.241661+0000", 2024-09-20T22:49:33.882 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00020146699999999999 2024-09-20T22:49:33.882 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.882 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.882 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:49:33.882 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:47.242513+0000", 2024-09-20T22:49:33.882 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00085188600000000005 2024-09-20T22:49:33.882 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.882 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.882 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:49:33.882 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:47.243955+0000", 2024-09-20T22:49:33.882 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001442313 2024-09-20T22:49:33.882 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.882 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.882 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:49:33.883 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:47.247555+0000", 2024-09-20T22:49:33.883 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0036002529999999999 2024-09-20T22:49:33.883 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.883 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.883 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:49:33.883 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:47.247628+0000", 2024-09-20T22:49:33.883 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.2878000000000003e-05 2024-09-20T22:49:33.883 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.883 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.883 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:49:33.883 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:47.247801+0000", 2024-09-20T22:49:33.883 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00017305000000000001 2024-09-20T22:49:33.883 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:49:33.883 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:49:33.884 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:49:33.884 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.884 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.884 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:4418 7.6 7:6885a65f:::smithi08477203-14:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e446)", 2024-09-20T22:49:33.884 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:48:46.225948+0000", 2024-09-20T22:49:33.884 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 47.516678184, 2024-09-20T22:49:33.884 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.022323377, 2024-09-20T22:49:33.884 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:49:33.884 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:49:33.884 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-20T22:49:33.884 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:49:33.884 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:49:33.884 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:49:33.884 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 4418 2024-09-20T22:49:33.884 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.885 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:49:33.885 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.885 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:49:33.885 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:46.225948+0000", 2024-09-20T22:49:33.885 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:49:33.885 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.885 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.885 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:49:33.885 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:46.225948+0000", 2024-09-20T22:49:33.885 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:49:33.885 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.885 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.885 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:49:33.885 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:46.225948+0000", 2024-09-20T22:49:33.886 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.9999999999999998e-07 2024-09-20T22:49:33.886 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.886 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.886 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:49:33.886 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:46.225951+0000", 2024-09-20T22:49:33.886 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.4499999999999998e-06 2024-09-20T22:49:33.886 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.886 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.886 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:49:33.886 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:46.225956+0000", 2024-09-20T22:49:33.886 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.0309999999999998e-06 2024-09-20T22:49:33.886 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.886 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.886 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:49:33.887 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:46.225961+0000", 2024-09-20T22:49:33.887 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.8359999999999999e-06 2024-09-20T22:49:33.887 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.887 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.887 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:49:33.887 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:46.228848+0000", 2024-09-20T22:49:33.887 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0028873670000000001 2024-09-20T22:49:33.887 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.887 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.887 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-09-20T22:49:33.887 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:46.228857+0000", 2024-09-20T22:49:33.887 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.6999999999999997e-06 2024-09-20T22:49:33.887 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.887 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.887 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:49:33.888 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:47.242965+0000", 2024-09-20T22:49:33.888 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0141083310000001 2024-09-20T22:49:33.888 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.888 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.888 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for rw locks", 2024-09-20T22:49:33.888 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:47.243076+0000", 2024-09-20T22:49:33.888 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000110945 2024-09-20T22:49:33.888 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.888 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.888 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:49:33.888 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:47.247935+0000", 2024-09-20T22:49:33.888 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0048591049999999998 2024-09-20T22:49:33.888 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.888 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.889 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:49:33.889 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:47.248116+0000", 2024-09-20T22:49:33.889 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000180936 2024-09-20T22:49:33.889 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:33.889 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:33.889 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:49:33.889 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:47.248271+0000", 2024-09-20T22:49:33.889 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00015517600000000001 2024-09-20T22:49:33.889 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:49:33.889 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:49:33.889 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:49:33.889 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:49:33.889 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:49:33.889 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-20T22:49:33.911 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:33.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:33.912 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:49:34.013 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:49:34.039 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:34.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:34.040 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:49:34.141 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:49:34.162 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:34.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:34.163 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:49:34.227 INFO:tasks.rados.rados.0.smithi084.stdout:1918: done (3 left) 2024-09-20T22:49:34.227 INFO:tasks.rados.rados.0.smithi084.stdout:1919: read oid 8 snap 165 2024-09-20T22:49:34.227 INFO:tasks.rados.rados.0.smithi084.stdout:1919: expect deleted 2024-09-20T22:49:34.227 INFO:tasks.rados.rados.0.smithi084.stdout:1920: write (excl) oid 20 current snap is 207 2024-09-20T22:49:34.227 INFO:tasks.rados.rados.0.smithi084.stdout:1920: seq_num 676 ranges {561532=707388,2058298=782238} 2024-09-20T22:49:34.235 INFO:tasks.rados.rados.0.smithi084.stdout:1920: writing smithi08477203-20 from 561532 to 1268920 tid 1 2024-09-20T22:49:34.244 INFO:tasks.rados.rados.0.smithi084.stdout:1920: writing smithi08477203-20 from 2058298 to 2840536 tid 2 2024-09-20T22:49:34.244 INFO:tasks.rados.rados.0.smithi084.stdout:1921: read oid 3 snap -1 2024-09-20T22:49:34.244 INFO:tasks.rados.rados.0.smithi084.stdout:1921: expect deleted 2024-09-20T22:49:34.244 INFO:tasks.rados.rados.0.smithi084.stdout:1922: write oid 2 current snap is 207 2024-09-20T22:49:34.244 INFO:tasks.rados.rados.0.smithi084.stdout:1922: seq_num 677 ranges {578339=498990,1833133=525896,2791719=586620} 2024-09-20T22:49:34.249 INFO:tasks.rados.rados.0.smithi084.stdout:1922: writing smithi08477203-2 from 578339 to 1077329 tid 1 2024-09-20T22:49:34.256 INFO:tasks.rados.rados.0.smithi084.stdout:1922: writing smithi08477203-2 from 1833133 to 2359029 tid 2 2024-09-20T22:49:34.262 INFO:tasks.rados.rados.0.smithi084.stdout:1922: writing smithi08477203-2 from 2791719 to 3378339 tid 3 2024-09-20T22:49:34.262 INFO:tasks.rados.rados.0.smithi084.stdout:1923: read oid 44 snap 165 2024-09-20T22:49:34.263 INFO:tasks.rados.rados.0.smithi084.stdout:1923: expect (ObjNum 470 snap 130 seq_num 470) 2024-09-20T22:49:34.263 INFO:tasks.rados.rados.0.smithi084.stdout:1924: snap_remove snap 194 2024-09-20T22:49:34.264 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:49:34.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:34.287 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:34.288 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:49:34.388 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:49:34.410 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:34.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:34.411 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:49:34.511 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:49:34.536 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:34.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:34.537 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:49:34.638 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:49:34.662 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:34.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:34.663 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:49:34.764 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:49:34.789 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:34.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:34.790 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:49:34.890 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:49:34.912 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:34.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:34.913 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:49:35.014 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:49:35.040 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:35.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:35.041 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:49:35.142 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:49:35.167 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:35.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:35.168 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:49:35.240 INFO:tasks.rados.rados.0.smithi084.stdout:1924: done (8 left) 2024-09-20T22:49:35.240 INFO:tasks.rados.rados.0.smithi084.stdout:1925: write oid 11 current snap is 207 2024-09-20T22:49:35.240 INFO:tasks.rados.rados.0.smithi084.stdout:1925: seq_num 678 ranges {595146=574239,1891615=669554,3219142=176004} 2024-09-20T22:49:35.246 INFO:tasks.rados.rados.0.smithi084.stdout:1925: writing smithi08477203-11 from 595146 to 1169385 tid 1 2024-09-20T22:49:35.254 INFO:tasks.rados.rados.0.smithi084.stdout:1925: writing smithi08477203-11 from 1891615 to 2561169 tid 2 2024-09-20T22:49:35.259 INFO:tasks.rados.rados.0.smithi084.stdout:1925: writing smithi08477203-11 from 3219142 to 3395146 tid 3 2024-09-20T22:49:35.259 INFO:tasks.rados.rados.0.smithi084.stdout:1926: read oid 31 snap -1 2024-09-20T22:49:35.259 INFO:tasks.rados.rados.0.smithi084.stdout:1926: expect (ObjNum 664 snap 201 seq_num 664) 2024-09-20T22:49:35.259 INFO:tasks.rados.rados.0.smithi084.stdout:1915: expect (ObjNum 625 snap 186 seq_num 625) 2024-09-20T22:49:35.268 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:49:35.290 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:35.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:35.291 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:49:35.318 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-20T22:49:35.381 INFO:tasks.rados.rados.0.smithi084.stdout:1914: finishing rollback tid 0 to smithi08477203-7 2024-09-20T22:49:35.381 INFO:tasks.rados.rados.0.smithi084.stdout:1914: finishing rollback tid 1 to smithi08477203-7 2024-09-20T22:49:35.381 INFO:tasks.rados.rados.0.smithi084.stdout:1914: finishing rollback tid 2 to smithi08477203-7 2024-09-20T22:49:35.381 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 7 v 850 (ObjNum 568 snap 171 seq_num 568) dirty exists 2024-09-20T22:49:35.382 INFO:tasks.rados.rados.0.smithi084.stdout:1917: finishing rollback tid 0 to smithi08477203-42 2024-09-20T22:49:35.382 INFO:tasks.rados.rados.0.smithi084.stdout:1917: finishing rollback tid 1 to smithi08477203-42 2024-09-20T22:49:35.382 INFO:tasks.rados.rados.0.smithi084.stdout:1917: finishing rollback tid 2 to smithi08477203-42 2024-09-20T22:49:35.382 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 42 v 567 (ObjNum 575 snap 172 seq_num 575) dirty exists 2024-09-20T22:49:35.382 INFO:tasks.rados.rados.0.smithi084.stdout:1914: done (9 left) 2024-09-20T22:49:35.382 INFO:tasks.rados.rados.0.smithi084.stdout:1915: done (8 left) 2024-09-20T22:49:35.382 INFO:tasks.rados.rados.0.smithi084.stdout:1917: done (7 left) 2024-09-20T22:49:35.382 INFO:tasks.rados.rados.0.smithi084.stdout:1927: read oid 16 snap -1 2024-09-20T22:49:35.382 INFO:tasks.rados.rados.0.smithi084.stdout:1927: expect deleted 2024-09-20T22:49:35.382 INFO:tasks.rados.rados.0.smithi084.stdout:1928: read oid 48 snap 154 2024-09-20T22:49:35.382 INFO:tasks.rados.rados.0.smithi084.stdout:1928: expect (ObjNum 424 snap 109 seq_num 424) 2024-09-20T22:49:35.382 INFO:tasks.rados.rados.0.smithi084.stdout:1929: delete oid 26 current snap is 207 2024-09-20T22:49:35.383 INFO:tasks.rados.rados.0.smithi084.stdout:1920: finishing write tid 1 to smithi08477203-20 2024-09-20T22:49:35.383 INFO:tasks.rados.rados.0.smithi084.stdout:1920: finishing write tid 2 to smithi08477203-20 2024-09-20T22:49:35.383 INFO:tasks.rados.rados.0.smithi084.stdout:1920: finishing write tid 3 to smithi08477203-20 2024-09-20T22:49:35.383 INFO:tasks.rados.rados.0.smithi084.stdout:1920: finishing write tid 4 to smithi08477203-20 2024-09-20T22:49:35.383 INFO:tasks.rados.rados.0.smithi084.stdout:1920: oid 20 updating version 0 to 316 2024-09-20T22:49:35.383 INFO:tasks.rados.rados.0.smithi084.stdout:1920: oid 20 version 316 is already newer than 315 2024-09-20T22:49:35.383 INFO:tasks.rados.rados.0.smithi084.stdout:1920: oid 20 version 316 is already newer than 314 2024-09-20T22:49:35.383 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 20 v 316 (ObjNum 676 snap 207 seq_num 676) dirty exists 2024-09-20T22:49:35.383 INFO:tasks.rados.rados.0.smithi084.stdout:1920: left oid 20 (ObjNum 676 snap 207 seq_num 676) 2024-09-20T22:49:35.383 INFO:tasks.rados.rados.0.smithi084.stdout:1922: finishing write tid 1 to smithi08477203-2 2024-09-20T22:49:35.383 INFO:tasks.rados.rados.0.smithi084.stdout:1922: finishing write tid 2 to smithi08477203-2 2024-09-20T22:49:35.383 INFO:tasks.rados.rados.0.smithi084.stdout:1922: finishing write tid 3 to smithi08477203-2 2024-09-20T22:49:35.383 INFO:tasks.rados.rados.0.smithi084.stdout:1922: finishing write tid 4 to smithi08477203-2 2024-09-20T22:49:35.383 INFO:tasks.rados.rados.0.smithi084.stdout:1922: finishing write tid 5 to smithi08477203-2 2024-09-20T22:49:35.384 INFO:tasks.rados.rados.0.smithi084.stdout:1922: oid 2 updating version 0 to 485 2024-09-20T22:49:35.384 INFO:tasks.rados.rados.0.smithi084.stdout:1922: oid 2 updating version 485 to 487 2024-09-20T22:49:35.384 INFO:tasks.rados.rados.0.smithi084.stdout:1922: oid 2 version 487 is already newer than 484 2024-09-20T22:49:35.384 INFO:tasks.rados.rados.0.smithi084.stdout:1922: oid 2 version 487 is already newer than 486 2024-09-20T22:49:35.384 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 2 v 487 (ObjNum 677 snap 207 seq_num 677) dirty exists 2024-09-20T22:49:35.384 INFO:tasks.rados.rados.0.smithi084.stdout:1922: left oid 2 (ObjNum 677 snap 207 seq_num 677) 2024-09-20T22:49:35.384 INFO:tasks.rados.rados.0.smithi084.stdout:1923: expect (ObjNum 470 snap 130 seq_num 470) 2024-09-20T22:49:35.391 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:49:35.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:35.415 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:35.415 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:49:35.487 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-20T22:49:35.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:35 smithi154 ceph-mon[33264]: pgmap v642: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 434 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 8.4 MiB/s rd, 3.0 MiB/s wr, 19 op/s 2024-09-20T22:49:35.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:35 smithi154 ceph-mon[33264]: osdmap e494: 8 total, 8 up, 8 in 2024-09-20T22:49:35.508 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:35 smithi084 ceph-mon[40024]: pgmap v642: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 434 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 8.4 MiB/s rd, 3.0 MiB/s wr, 19 op/s 2024-09-20T22:49:35.508 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:35 smithi084 ceph-mon[40024]: osdmap e494: 8 total, 8 up, 8 in 2024-09-20T22:49:35.508 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:35 smithi084 ceph-mon[29774]: pgmap v642: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 434 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 8.4 MiB/s rd, 3.0 MiB/s wr, 19 op/s 2024-09-20T22:49:35.508 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:35 smithi084 ceph-mon[29774]: osdmap e494: 8 total, 8 up, 8 in 2024-09-20T22:49:35.511 INFO:tasks.rados.rados.0.smithi084.stdout:1925: finishing write tid 1 to smithi08477203-11 2024-09-20T22:49:35.511 INFO:tasks.rados.rados.0.smithi084.stdout:1925: finishing write tid 2 to smithi08477203-11 2024-09-20T22:49:35.511 INFO:tasks.rados.rados.0.smithi084.stdout:1925: finishing write tid 3 to smithi08477203-11 2024-09-20T22:49:35.511 INFO:tasks.rados.rados.0.smithi084.stdout:1925: finishing write tid 4 to smithi08477203-11 2024-09-20T22:49:35.511 INFO:tasks.rados.rados.0.smithi084.stdout:1925: finishing write tid 5 to smithi08477203-11 2024-09-20T22:49:35.511 INFO:tasks.rados.rados.0.smithi084.stdout:1925: oid 11 updating version 0 to 710 2024-09-20T22:49:35.511 INFO:tasks.rados.rados.0.smithi084.stdout:1925: oid 11 updating version 710 to 711 2024-09-20T22:49:35.511 INFO:tasks.rados.rados.0.smithi084.stdout:1925: oid 11 updating version 711 to 713 2024-09-20T22:49:35.511 INFO:tasks.rados.rados.0.smithi084.stdout:1925: oid 11 version 713 is already newer than 712 2024-09-20T22:49:35.511 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 11 v 713 (ObjNum 678 snap 207 seq_num 678) dirty exists 2024-09-20T22:49:35.512 INFO:tasks.rados.rados.0.smithi084.stdout:1925: left oid 11 (ObjNum 678 snap 207 seq_num 678) 2024-09-20T22:49:35.512 INFO:tasks.rados.rados.0.smithi084.stdout:1919: done (9 left) 2024-09-20T22:49:35.512 INFO:tasks.rados.rados.0.smithi084.stdout:1920: done (8 left) 2024-09-20T22:49:35.512 INFO:tasks.rados.rados.0.smithi084.stdout:1921: done (7 left) 2024-09-20T22:49:35.512 INFO:tasks.rados.rados.0.smithi084.stdout:1922: done (6 left) 2024-09-20T22:49:35.512 INFO:tasks.rados.rados.0.smithi084.stdout:1923: done (5 left) 2024-09-20T22:49:35.512 INFO:tasks.rados.rados.0.smithi084.stdout:1925: done (4 left) 2024-09-20T22:49:35.512 INFO:tasks.rados.rados.0.smithi084.stdout:1929: done (3 left) 2024-09-20T22:49:35.512 INFO:tasks.rados.rados.0.smithi084.stdout:1930: rollback oid 37 current snap is 207 2024-09-20T22:49:35.512 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 37 to 189 2024-09-20T22:49:35.512 INFO:tasks.rados.rados.0.smithi084.stdout:1926: expect (ObjNum 664 snap 201 seq_num 664) 2024-09-20T22:49:35.516 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:49:35.537 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:35.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:35.538 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:49:35.630 INFO:tasks.rados.rados.0.smithi084.stdout:1926: done (3 left) 2024-09-20T22:49:35.630 INFO:tasks.rados.rados.0.smithi084.stdout:1931: copy_from oid 19 from oid 28 current snap is 207 2024-09-20T22:49:35.630 INFO:tasks.rados.rados.0.smithi084.stdout:1928: expect (ObjNum 424 snap 109 seq_num 424) 2024-09-20T22:49:35.635 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:49:35.639 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:49:35.661 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:35.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:35.662 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:49:35.754 INFO:tasks.rados.rados.0.smithi084.stdout:1930: finishing rollback tid 1 to smithi08477203-37 2024-09-20T22:49:35.754 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 37 v 842 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-09-20T22:49:35.754 INFO:tasks.rados.rados.0.smithi084.stdout:1931: finishing copy_from racing read to smithi08477203-19 2024-09-20T22:49:35.754 INFO:tasks.rados.rados.0.smithi084.stdout:1931: finishing copy_from to smithi08477203-19 2024-09-20T22:49:35.755 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 19 v 1207 (ObjNum 656 snap 198 seq_num 656) dirty exists 2024-09-20T22:49:35.755 INFO:tasks.rados.rados.0.smithi084.stdout:1927: done (3 left) 2024-09-20T22:49:35.755 INFO:tasks.rados.rados.0.smithi084.stdout:1928: done (2 left) 2024-09-20T22:49:35.755 INFO:tasks.rados.rados.0.smithi084.stdout:1930: done (1 left) 2024-09-20T22:49:35.755 INFO:tasks.rados.rados.0.smithi084.stdout:1931: done (0 left) 2024-09-20T22:49:35.755 INFO:tasks.rados.rados.0.smithi084.stdout:1932: snap_create 2024-09-20T22:49:35.763 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:49:35.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:35.789 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:35.790 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:49:35.890 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:49:35.913 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:35.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:35.914 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:49:36.015 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:49:36.036 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:36.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:36.037 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:49:36.138 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:49:36.139 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-20T22:49:36.139 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2024-09-20T22:49:36.139 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2024-09-20T22:49:36.139 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-20T22:49:36.162 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:36.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:36.163 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:49:36.235 INFO:tasks.rados.rados.0.smithi084.stdout:1932: done (0 left) 2024-09-20T22:49:36.235 INFO:tasks.rados.rados.0.smithi084.stdout:1933: snap_create 2024-09-20T22:49:36.264 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:49:36.286 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:36.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:36.287 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:49:36.364 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:36 smithi084 ceph-mon[40024]: osdmap e495: 8 total, 8 up, 8 in 2024-09-20T22:49:36.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:36 smithi084 ceph-mon[29774]: osdmap e495: 8 total, 8 up, 8 in 2024-09-20T22:49:36.388 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:49:36.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:36.415 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:36.415 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:49:36.503 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-20T22:49:36.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:36 smithi154 ceph-mon[33264]: osdmap e495: 8 total, 8 up, 8 in 2024-09-20T22:49:36.516 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:49:36.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:36.568 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:36.569 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:49:36.669 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:49:36.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:36.692 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:36.693 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:49:36.792 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:49:36.793 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:49:36.801 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.1 dump_blocked_ops 2024-09-20T22:49:36.819 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:36.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:36.820 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:49:36.921 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:49:36.943 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:36.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:36.944 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:49:37.045 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:49:37.071 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:37.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:37.072 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:49:37.172 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:49:37.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:37.194 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:37.194 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:49:37.252 INFO:tasks.rados.rados.0.smithi084.stdout:1933: done (0 left) 2024-09-20T22:49:37.252 INFO:tasks.rados.rados.0.smithi084.stdout:1934: delete oid 42 current snap is 209 2024-09-20T22:49:37.256 INFO:tasks.rados.rados.0.smithi084.stdout:1934: done (0 left) 2024-09-20T22:49:37.256 INFO:tasks.rados.rados.0.smithi084.stdout:1935: write (excl) oid 46 current snap is 209 2024-09-20T22:49:37.256 INFO:tasks.rados.rados.0.smithi084.stdout:1935: seq_num 681 ranges {645567=799986,1950708=533234,3185058=260509} 2024-09-20T22:49:37.263 INFO:tasks.rados.rados.0.smithi084.stdout:1935: writing smithi08477203-46 from 645567 to 1445553 tid 1 2024-09-20T22:49:37.270 INFO:tasks.rados.rados.0.smithi084.stdout:1935: writing smithi08477203-46 from 1950708 to 2483942 tid 2 2024-09-20T22:49:37.273 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:49:37.273 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:49:37.276 INFO:tasks.rados.rados.0.smithi084.stdout:1935: writing smithi08477203-46 from 3185058 to 3445567 tid 3 2024-09-20T22:49:37.276 INFO:tasks.rados.rados.0.smithi084.stdout:1936: read oid 21 snap 207 2024-09-20T22:49:37.276 INFO:tasks.rados.rados.0.smithi084.stdout:1936: expect (ObjNum 636 snap 193 seq_num 636) 2024-09-20T22:49:37.277 INFO:tasks.rados.rados.0.smithi084.stdout:1937: rollback oid 30 current snap is 209 2024-09-20T22:49:37.277 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 30 to 198 2024-09-20T22:49:37.277 INFO:tasks.rados.rados.0.smithi084.stdout:1935: finishing write tid 1 to smithi08477203-46 2024-09-20T22:49:37.277 INFO:tasks.rados.rados.0.smithi084.stdout:1935: finishing write tid 2 to smithi08477203-46 2024-09-20T22:49:37.277 INFO:tasks.rados.rados.0.smithi084.stdout:1938: read oid 15 snap -1 2024-09-20T22:49:37.277 INFO:tasks.rados.rados.0.smithi084.stdout:1938: expect (ObjNum 625 snap 186 seq_num 625) 2024-09-20T22:49:37.277 INFO:tasks.rados.rados.0.smithi084.stdout:1939: snap_create 2024-09-20T22:49:37.280 INFO:tasks.rados.rados.0.smithi084.stdout:1935: finishing write tid 3 to smithi08477203-46 2024-09-20T22:49:37.280 INFO:tasks.rados.rados.0.smithi084.stdout:1935: finishing write tid 4 to smithi08477203-46 2024-09-20T22:49:37.280 INFO:tasks.rados.rados.0.smithi084.stdout:1935: finishing write tid 5 to smithi08477203-46 2024-09-20T22:49:37.280 INFO:tasks.rados.rados.0.smithi084.stdout:1935: oid 46 updating version 0 to 770 2024-09-20T22:49:37.281 INFO:tasks.rados.rados.0.smithi084.stdout:1935: oid 46 version 770 is already newer than 767 2024-09-20T22:49:37.281 INFO:tasks.rados.rados.0.smithi084.stdout:1935: oid 46 version 770 is already newer than 769 2024-09-20T22:49:37.281 INFO:tasks.rados.rados.0.smithi084.stdout:1935: oid 46 version 770 is already newer than 768 2024-09-20T22:49:37.281 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 46 v 770 (ObjNum 681 snap 209 seq_num 681) dirty exists 2024-09-20T22:49:37.281 INFO:tasks.rados.rados.0.smithi084.stdout:1935: left oid 46 (ObjNum 681 snap 209 seq_num 681) 2024-09-20T22:49:37.281 INFO:tasks.rados.rados.0.smithi084.stdout:1937: finishing rollback tid 1 to smithi08477203-30 2024-09-20T22:49:37.281 INFO:tasks.rados.rados.0.smithi084.stdout:1937: finishing rollback tid 2 to smithi08477203-30 2024-09-20T22:49:37.281 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 30 v 772 (ObjNum 602 snap 177 seq_num 602) dirty exists 2024-09-20T22:49:37.282 INFO:tasks.rados.rados.0.smithi084.stdout:1938: expect (ObjNum 625 snap 186 seq_num 625) 2024-09-20T22:49:37.293 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:49:37.294 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:49:37.295 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:49:37.311 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:49:37.312 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:49:37.312 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:37 smithi084 ceph-mon[40024]: pgmap v645: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 434 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 8.4 MiB/s rd, 3.0 MiB/s wr, 19 op/s 2024-09-20T22:49:37.313 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:37 smithi084 ceph-mon[40024]: purged_snaps scrub starts 2024-09-20T22:49:37.313 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:37 smithi084 ceph-mon[40024]: purged_snaps scrub ok 2024-09-20T22:49:37.313 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:37 smithi084 ceph-mon[40024]: osdmap e496: 8 total, 8 up, 8 in 2024-09-20T22:49:37.313 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:37 smithi084 ceph-mon[29774]: pgmap v645: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 434 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 8.4 MiB/s rd, 3.0 MiB/s wr, 19 op/s 2024-09-20T22:49:37.314 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:37 smithi084 ceph-mon[29774]: purged_snaps scrub starts 2024-09-20T22:49:37.314 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:37 smithi084 ceph-mon[29774]: purged_snaps scrub ok 2024-09-20T22:49:37.314 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:37 smithi084 ceph-mon[29774]: osdmap e496: 8 total, 8 up, 8 in 2024-09-20T22:49:37.317 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:37.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:37.319 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:49:37.321 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:49:37.333 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-20T22:49:37.333 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T22:49:37.359 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:49:37.360 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:49:37.364 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:49:37.377 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:49:37.377 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:49:37.380 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 5, 6, 7, 0, 4] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-20T22:49:37.381 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd reweight 1 1 2024-09-20T22:49:37.398 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:49:37.398 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:49:37.417 INFO:tasks.rados.rados.0.smithi084.stdout:1936: expect (ObjNum 636 snap 193 seq_num 636) 2024-09-20T22:49:37.417 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:49:37.417 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:49:37.420 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:49:37.445 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:37.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:37.446 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:49:37.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:37 smithi154 ceph-mon[33264]: pgmap v645: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 434 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 8.4 MiB/s rd, 3.0 MiB/s wr, 19 op/s 2024-09-20T22:49:37.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:37 smithi154 ceph-mon[33264]: purged_snaps scrub starts 2024-09-20T22:49:37.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:37 smithi154 ceph-mon[33264]: purged_snaps scrub ok 2024-09-20T22:49:37.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:37 smithi154 ceph-mon[33264]: osdmap e496: 8 total, 8 up, 8 in 2024-09-20T22:49:37.547 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:49:37.573 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:37.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:37.574 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:49:37.675 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:49:37.696 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:37.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:37.697 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:49:37.798 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:49:37.819 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:37.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:37.820 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:49:37.920 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:49:37.946 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:37.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:37.947 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:49:38.048 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:49:38.073 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:38.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:38.074 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:49:38.175 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:49:38.196 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:38.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:38.197 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:49:38.226 INFO:tasks.rados.rados.0.smithi084.stdout:1935: done (4 left) 2024-09-20T22:49:38.226 INFO:tasks.rados.rados.0.smithi084.stdout:1936: done (3 left) 2024-09-20T22:49:38.226 INFO:tasks.rados.rados.0.smithi084.stdout:1937: done (2 left) 2024-09-20T22:49:38.226 INFO:tasks.rados.rados.0.smithi084.stdout:1938: done (1 left) 2024-09-20T22:49:38.226 INFO:tasks.rados.rados.0.smithi084.stdout:1939: done (0 left) 2024-09-20T22:49:38.227 INFO:tasks.rados.rados.0.smithi084.stdout:1940: snap_create 2024-09-20T22:49:38.298 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:49:38.323 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:38.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:38.325 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:49:38.426 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:49:38.447 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:38.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:38.448 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:49:38.475 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:38 smithi084 ceph-mon[40024]: osdmap e497: 8 total, 8 up, 8 in 2024-09-20T22:49:38.475 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:38 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/158151661' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:49:38.476 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:38 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:49:38.476 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:38 smithi084 ceph-mon[40024]: osdmap e498: 8 total, 8 up, 8 in 2024-09-20T22:49:38.476 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:38 smithi084 ceph-mon[29774]: osdmap e497: 8 total, 8 up, 8 in 2024-09-20T22:49:38.476 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:38 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/158151661' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:49:38.476 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:38 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:49:38.476 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:38 smithi084 ceph-mon[29774]: osdmap e498: 8 total, 8 up, 8 in 2024-09-20T22:49:38.477 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:49:38 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: ::ffff:172.21.15.154 - - [20/Sep/2024:22:49:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-20T22:49:38.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:38 smithi154 ceph-mon[33264]: osdmap e497: 8 total, 8 up, 8 in 2024-09-20T22:49:38.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:38 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/158151661' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:49:38.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:38 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:49:38.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:38 smithi154 ceph-mon[33264]: osdmap e498: 8 total, 8 up, 8 in 2024-09-20T22:49:38.550 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:49:38.572 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:38.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:38.573 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:49:38.673 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:49:38.700 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:38.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:38.701 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:49:38.802 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:49:38.827 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:38.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:38.828 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:49:38.886 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-20T22:49:38.886 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:49:38.929 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:49:38.951 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:38.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:38.952 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:49:39.053 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:49:39.079 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:39.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:39.080 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:49:39.136 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:49:39.180 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:49:39.203 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:39.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:39.204 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:49:39.223 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is set 2024-09-20T22:49:39.228 INFO:tasks.rados.rados.0.smithi084.stdout:1940: done (0 left) 2024-09-20T22:49:39.229 INFO:tasks.rados.rados.0.smithi084.stdout:1941: snap_remove snap 198 2024-09-20T22:49:39.304 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:49:39.330 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:39.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:39.331 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:49:39.432 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:49:39.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:39.455 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:39.455 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:49:39.504 INFO:journalctl@ceph.iscsi.iscsi.a.smithi154.stdout:Sep 20 22:49:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-iscsi-iscsi-a[59730]: debug there is no tcmu-runner data available 2024-09-20T22:49:39.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:39 smithi154 ceph-mon[33264]: pgmap v648: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 434 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-20T22:49:39.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:39 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/3569491108' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:49:39.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:39 smithi154 ceph-mon[33264]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:49:39.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:39 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/3569491108' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T22:49:39.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:39 smithi154 ceph-mon[33264]: osdmap e499: 8 total, 8 up, 8 in 2024-09-20T22:49:39.555 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:49:39.581 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:39.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:39.582 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:49:39.614 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:39 smithi084 ceph-mon[40024]: pgmap v648: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 434 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-20T22:49:39.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:39 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/3569491108' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:49:39.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:39 smithi084 ceph-mon[40024]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:49:39.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:39 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/3569491108' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T22:49:39.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:39 smithi084 ceph-mon[40024]: osdmap e499: 8 total, 8 up, 8 in 2024-09-20T22:49:39.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:39 smithi084 ceph-mon[29774]: pgmap v648: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 434 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-20T22:49:39.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:39 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/3569491108' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:49:39.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:39 smithi084 ceph-mon[29774]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:49:39.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:39 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/3569491108' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T22:49:39.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:39 smithi084 ceph-mon[29774]: osdmap e499: 8 total, 8 up, 8 in 2024-09-20T22:49:39.680 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.1 dump_historic_ops 2024-09-20T22:49:39.683 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:49:39.709 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:39.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:39.710 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:49:39.810 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:49:39.837 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:39.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:39.838 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:49:39.939 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:49:39.961 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:39.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:39.962 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:49:40.063 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:49:40.084 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:40.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:40.085 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:49:40.186 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:49:40.212 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:40.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:40.213 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:49:40.231 INFO:tasks.rados.rados.0.smithi084.stdout:1941: done (0 left) 2024-09-20T22:49:40.231 INFO:tasks.rados.rados.0.smithi084.stdout:1942: read oid 19 snap -1 2024-09-20T22:49:40.231 INFO:tasks.rados.rados.0.smithi084.stdout:1942: expect (ObjNum 656 snap 198 seq_num 656) 2024-09-20T22:49:40.231 INFO:tasks.rados.rados.0.smithi084.stdout:1943: copy_from oid 42 from oid 28 current snap is 211 2024-09-20T22:49:40.232 INFO:tasks.rados.rados.0.smithi084.stdout:1944: write oid 30 current snap is 211 2024-09-20T22:49:40.232 INFO:tasks.rados.rados.0.smithi084.stdout:1944: seq_num 682 ranges {662374=475235,1609190=676892,2812481=624101} 2024-09-20T22:49:40.238 INFO:tasks.rados.rados.0.smithi084.stdout:1944: writing smithi08477203-30 from 662374 to 1137609 tid 1 2024-09-20T22:49:40.241 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-20T22:49:40.245 INFO:tasks.rados.rados.0.smithi084.stdout:1944: writing smithi08477203-30 from 1609190 to 2286082 tid 2 2024-09-20T22:49:40.252 INFO:tasks.rados.rados.0.smithi084.stdout:1944: writing smithi08477203-30 from 2812481 to 3436582 tid 3 2024-09-20T22:49:40.252 INFO:tasks.rados.rados.0.smithi084.stdout:1945: write (excl) oid 40 current snap is 211 2024-09-20T22:49:40.252 INFO:tasks.rados.rados.0.smithi084.stdout:1944: finishing write tid 1 to smithi08477203-30 2024-09-20T22:49:40.252 INFO:tasks.rados.rados.0.smithi084.stdout:1945: seq_num 683 ranges {679181=550484,1667672=536903,2839904=435373} 2024-09-20T22:49:40.259 INFO:tasks.rados.rados.0.smithi084.stdout:1945: writing smithi08477203-40 from 679181 to 1229665 tid 1 2024-09-20T22:49:40.265 INFO:tasks.rados.rados.0.smithi084.stdout:1945: writing smithi08477203-40 from 1667672 to 2204575 tid 2 2024-09-20T22:49:40.271 INFO:tasks.rados.rados.0.smithi084.stdout:1945: writing smithi08477203-40 from 2839904 to 3275277 tid 3 2024-09-20T22:49:40.271 INFO:tasks.rados.rados.0.smithi084.stdout:1946: copy_from oid 7 from oid 18 current snap is 211 2024-09-20T22:49:40.271 INFO:tasks.rados.rados.0.smithi084.stdout:1944: finishing write tid 2 to smithi08477203-30 2024-09-20T22:49:40.271 INFO:tasks.rados.rados.0.smithi084.stdout:1942: expect (ObjNum 656 snap 198 seq_num 656) 2024-09-20T22:49:40.314 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:49:40.337 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:40.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:40.338 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:49:40.423 INFO:tasks.rados.rados.0.smithi084.stdout:1943: finishing copy_from racing read to smithi08477203-42 2024-09-20T22:49:40.424 INFO:tasks.rados.rados.0.smithi084.stdout:1944: finishing write tid 3 to smithi08477203-30 2024-09-20T22:49:40.424 INFO:tasks.rados.rados.0.smithi084.stdout:1944: finishing write tid 4 to smithi08477203-30 2024-09-20T22:49:40.424 INFO:tasks.rados.rados.0.smithi084.stdout:1942: done (4 left) 2024-09-20T22:49:40.424 INFO:tasks.rados.rados.0.smithi084.stdout:1947: write (excl) oid 32 current snap is 211 2024-09-20T22:49:40.424 INFO:tasks.rados.rados.0.smithi084.stdout:1944: finishing write tid 5 to smithi08477203-30 2024-09-20T22:49:40.424 INFO:tasks.rados.rados.0.smithi084.stdout:1944: oid 30 updating version 0 to 775 2024-09-20T22:49:40.424 INFO:tasks.rados.rados.0.smithi084.stdout:1944: oid 30 updating version 775 to 776 2024-09-20T22:49:40.424 INFO:tasks.rados.rados.0.smithi084.stdout:1944: oid 30 version 776 is already newer than 774 2024-09-20T22:49:40.424 INFO:tasks.rados.rados.0.smithi084.stdout:1944: oid 30 version 776 is already newer than 773 2024-09-20T22:49:40.424 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 30 v 776 (ObjNum 682 snap 211 seq_num 682) dirty exists 2024-09-20T22:49:40.424 INFO:tasks.rados.rados.0.smithi084.stdout:1944: left oid 30 (ObjNum 682 snap 211 seq_num 682) 2024-09-20T22:49:40.424 INFO:tasks.rados.rados.0.smithi084.stdout:1943: finishing copy_from to smithi08477203-42 2024-09-20T22:49:40.424 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 42 v 1207 (ObjNum 656 snap 198 seq_num 656) dirty exists 2024-09-20T22:49:40.424 INFO:tasks.rados.rados.0.smithi084.stdout:1945: finishing write tid 1 to smithi08477203-40 2024-09-20T22:49:40.425 INFO:tasks.rados.rados.0.smithi084.stdout:1945: finishing write tid 2 to smithi08477203-40 2024-09-20T22:49:40.425 INFO:tasks.rados.rados.0.smithi084.stdout:1947: seq_num 684 ranges {695988=625733,2009801=680561,3150974=345014} 2024-09-20T22:49:40.430 INFO:tasks.rados.rados.0.smithi084.stdout:1947: writing smithi08477203-32 from 695988 to 1321721 tid 1 2024-09-20T22:49:40.438 INFO:tasks.rados.rados.0.smithi084.stdout:1947: writing smithi08477203-32 from 2009801 to 2690362 tid 2 2024-09-20T22:49:40.439 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:49:40.443 INFO:tasks.rados.rados.0.smithi084.stdout:1947: writing smithi08477203-32 from 3150974 to 3495988 tid 3 2024-09-20T22:49:40.443 INFO:tasks.rados.rados.0.smithi084.stdout:1943: done (4 left) 2024-09-20T22:49:40.443 INFO:tasks.rados.rados.0.smithi084.stdout:1944: done (3 left) 2024-09-20T22:49:40.443 INFO:tasks.rados.rados.0.smithi084.stdout:1948: delete oid 3 current snap is 211 2024-09-20T22:49:40.443 INFO:tasks.rados.rados.0.smithi084.stdout:1945: finishing write tid 3 to smithi08477203-40 2024-09-20T22:49:40.443 INFO:tasks.rados.rados.0.smithi084.stdout:1945: finishing write tid 4 to smithi08477203-40 2024-09-20T22:49:40.443 INFO:tasks.rados.rados.0.smithi084.stdout:1945: finishing write tid 5 to smithi08477203-40 2024-09-20T22:49:40.443 INFO:tasks.rados.rados.0.smithi084.stdout:1945: oid 40 updating version 0 to 726 2024-09-20T22:49:40.443 INFO:tasks.rados.rados.0.smithi084.stdout:1945: oid 40 version 726 is already newer than 723 2024-09-20T22:49:40.443 INFO:tasks.rados.rados.0.smithi084.stdout:1945: oid 40 version 726 is already newer than 725 2024-09-20T22:49:40.443 INFO:tasks.rados.rados.0.smithi084.stdout:1945: oid 40 version 726 is already newer than 724 2024-09-20T22:49:40.443 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 40 v 726 (ObjNum 683 snap 211 seq_num 683) dirty exists 2024-09-20T22:49:40.444 INFO:tasks.rados.rados.0.smithi084.stdout:1945: left oid 40 (ObjNum 683 snap 211 seq_num 683) 2024-09-20T22:49:40.444 INFO:tasks.rados.rados.0.smithi084.stdout:1946: finishing copy_from racing read to smithi08477203-7 2024-09-20T22:49:40.444 INFO:tasks.rados.rados.0.smithi084.stdout:1946: finishing copy_from to smithi08477203-7 2024-09-20T22:49:40.444 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 7 v 666 (ObjNum 657 snap 199 seq_num 657) dirty exists 2024-09-20T22:49:40.444 INFO:tasks.rados.rados.0.smithi084.stdout:1947: finishing write tid 1 to smithi08477203-32 2024-09-20T22:49:40.444 INFO:tasks.rados.rados.0.smithi084.stdout:1947: finishing write tid 2 to smithi08477203-32 2024-09-20T22:49:40.445 INFO:tasks.rados.rados.0.smithi084.stdout:1945: done (3 left) 2024-09-20T22:49:40.445 INFO:tasks.rados.rados.0.smithi084.stdout:1946: done (2 left) 2024-09-20T22:49:40.445 INFO:tasks.rados.rados.0.smithi084.stdout:1948: done (1 left) 2024-09-20T22:49:40.445 INFO:tasks.rados.rados.0.smithi084.stdout:1949: snap_remove snap 188 2024-09-20T22:49:40.462 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:40.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:40.463 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:49:40.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:40 smithi154 ceph-mon[33264]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:49:40.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:40 smithi154 ceph-mon[33264]: osdmap e500: 8 total, 8 up, 8 in 2024-09-20T22:49:40.505 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:40 smithi084 ceph-mon[40024]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:49:40.505 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:40 smithi084 ceph-mon[40024]: osdmap e500: 8 total, 8 up, 8 in 2024-09-20T22:49:40.506 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:40 smithi084 ceph-mon[29774]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:49:40.506 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:40 smithi084 ceph-mon[29774]: osdmap e500: 8 total, 8 up, 8 in 2024-09-20T22:49:40.564 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:49:40.585 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:40.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:40.586 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:49:40.687 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:49:40.708 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:40.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:40.709 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:49:40.810 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:49:40.831 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:40.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:40.832 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:49:40.902 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:49:40.933 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:49:40.936 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:49:40.955 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:40.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:40.956 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:49:41.057 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:49:41.078 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:41.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:41.079 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:49:41.180 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:49:41.206 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:41.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:41.207 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:49:41.276 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.1 to 1 (10000) 2024-09-20T22:49:41.280 INFO:tasks.rados.rados.0.smithi084.stdout:1949: done (1 left) 2024-09-20T22:49:41.280 INFO:tasks.rados.rados.0.smithi084.stdout:1950: snap_create 2024-09-20T22:49:41.280 INFO:tasks.rados.rados.0.smithi084.stdout:1947: finishing write tid 3 to smithi08477203-32 2024-09-20T22:49:41.281 INFO:tasks.rados.rados.0.smithi084.stdout:1947: finishing write tid 4 to smithi08477203-32 2024-09-20T22:49:41.281 INFO:tasks.rados.rados.0.smithi084.stdout:1947: finishing write tid 5 to smithi08477203-32 2024-09-20T22:49:41.281 INFO:tasks.rados.rados.0.smithi084.stdout:1947: oid 32 updating version 0 to 401 2024-09-20T22:49:41.281 INFO:tasks.rados.rados.0.smithi084.stdout:1947: oid 32 updating version 401 to 404 2024-09-20T22:49:41.281 INFO:tasks.rados.rados.0.smithi084.stdout:1947: oid 32 version 404 is already newer than 403 2024-09-20T22:49:41.281 INFO:tasks.rados.rados.0.smithi084.stdout:1947: oid 32 version 404 is already newer than 402 2024-09-20T22:49:41.281 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 32 v 404 (ObjNum 684 snap 211 seq_num 684) dirty exists 2024-09-20T22:49:41.281 INFO:tasks.rados.rados.0.smithi084.stdout:1947: left oid 32 (ObjNum 684 snap 211 seq_num 684) 2024-09-20T22:49:41.307 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:49:41.329 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:41.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:41.330 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:49:41.431 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:49:41.452 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:41.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:41.453 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:49:41.482 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:41 smithi084 ceph-mon[29774]: pgmap v651: 158 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 153 active+clean; 443 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 16 MiB/s rd, 2.2 MiB/s wr, 19 op/s 2024-09-20T22:49:41.482 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:41 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/2601766574' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-09-20T22:49:41.482 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:41 smithi084 ceph-mon[40024]: pgmap v651: 158 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 153 active+clean; 443 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 16 MiB/s rd, 2.2 MiB/s wr, 19 op/s 2024-09-20T22:49:41.482 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:41 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/2601766574' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-09-20T22:49:41.554 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:49:41.576 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:41.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:41.577 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:49:41.675 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:41 smithi154 ceph-mon[33264]: pgmap v651: 158 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 153 active+clean; 443 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 16 MiB/s rd, 2.2 MiB/s wr, 19 op/s 2024-09-20T22:49:41.675 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:41 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/2601766574' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-09-20T22:49:41.678 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:49:41.699 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:41.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:41.700 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:49:41.800 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:49:41.800 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:49:41.801 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:49:41.817 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:49:41.817 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:49:41.822 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:41.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:41.822 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:49:41.833 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:49:41.834 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:49:41.850 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-20T22:49:41.850 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T22:49:41.867 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:49:41.867 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:49:41.883 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:49:41.883 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:49:41.899 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-20T22:49:41.899 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:49:41.900 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:49:41.900 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:49:41.918 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:49:41.918 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:49:41.924 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:49:41.947 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:41.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:41.948 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:49:42.048 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:49:42.071 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:42.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:42.072 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:49:42.173 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:49:42.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:42.195 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:42.196 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:49:42.291 INFO:tasks.rados.rados.0.smithi084.stdout:1947: done (1 left) 2024-09-20T22:49:42.291 INFO:tasks.rados.rados.0.smithi084.stdout:1950: done (0 left) 2024-09-20T22:49:42.291 INFO:tasks.rados.rados.0.smithi084.stdout:1951: delete oid 12 current snap is 212 2024-09-20T22:49:42.295 INFO:tasks.rados.rados.0.smithi084.stdout:1951: done (0 left) 2024-09-20T22:49:42.295 INFO:tasks.rados.rados.0.smithi084.stdout:1952: read oid 5 snap -1 2024-09-20T22:49:42.296 INFO:tasks.rados.rados.0.smithi084.stdout:1952: expect (ObjNum 670 snap 205 seq_num 670) 2024-09-20T22:49:42.296 INFO:tasks.rados.rados.0.smithi084.stdout:1953: delete oid 27 current snap is 212 2024-09-20T22:49:42.296 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:49:42.297 INFO:tasks.rados.rados.0.smithi084.stdout:1953: done (1 left) 2024-09-20T22:49:42.297 INFO:tasks.rados.rados.0.smithi084.stdout:1954: copy_from oid 31 from oid 14 current snap is 212 2024-09-20T22:49:42.298 INFO:tasks.rados.rados.0.smithi084.stdout:1955: write oid 25 current snap is 212 2024-09-20T22:49:42.298 INFO:tasks.rados.rados.0.smithi084.stdout:1955: seq_num 688 ranges {763216=643082,1843729=687899,3260666=302550} 2024-09-20T22:49:42.305 INFO:tasks.rados.rados.0.smithi084.stdout:1955: writing smithi08477203-25 from 763216 to 1406298 tid 1 2024-09-20T22:49:42.311 INFO:tasks.rados.rados.0.smithi084.stdout:1955: writing smithi08477203-25 from 1843729 to 2531628 tid 2 2024-09-20T22:49:42.317 INFO:tasks.rados.rados.0.smithi084.stdout:1955: writing smithi08477203-25 from 3260666 to 3563216 tid 3 2024-09-20T22:49:42.317 INFO:tasks.rados.rados.0.smithi084.stdout:1956: write (excl) oid 13 current snap is 212 2024-09-20T22:49:42.317 INFO:tasks.rados.rados.0.smithi084.stdout:1954: finishing copy_from to smithi08477203-31 2024-09-20T22:49:42.317 INFO:tasks.rados.rados.0.smithi084.stdout:1954: got expected ENOENT (src dne) 2024-09-20T22:49:42.317 INFO:tasks.rados.rados.0.smithi084.stdout:1956: seq_num 689 ranges {780023=718331,1902211=547910,3004442=575581} 2024-09-20T22:49:42.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:42.322 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:42.322 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:49:42.325 INFO:tasks.rados.rados.0.smithi084.stdout:1956: writing smithi08477203-13 from 780023 to 1498354 tid 1 2024-09-20T22:49:42.330 INFO:tasks.rados.rados.0.smithi084.stdout:1956: writing smithi08477203-13 from 1902211 to 2450121 tid 2 2024-09-20T22:49:42.337 INFO:tasks.rados.rados.0.smithi084.stdout:1956: writing smithi08477203-13 from 3004442 to 3580023 tid 3 2024-09-20T22:49:42.337 INFO:tasks.rados.rados.0.smithi084.stdout:1957: write (excl) oid 17 current snap is 212 2024-09-20T22:49:42.337 INFO:tasks.rados.rados.0.smithi084.stdout:1957: seq_num 690 ranges {796830=793580,2360693=691568} 2024-09-20T22:49:42.346 INFO:tasks.rados.rados.0.smithi084.stdout:1957: writing smithi08477203-17 from 796830 to 1590410 tid 1 2024-09-20T22:49:42.354 INFO:tasks.rados.rados.0.smithi084.stdout:1957: writing smithi08477203-17 from 2360693 to 3052261 tid 2 2024-09-20T22:49:42.354 INFO:tasks.rados.rados.0.smithi084.stdout:1958: snap_create 2024-09-20T22:49:42.354 INFO:tasks.rados.rados.0.smithi084.stdout:1954: finishing copy_from racing read to smithi08477203-31 2024-09-20T22:49:42.354 INFO:tasks.rados.rados.0.smithi084.stdout:1952: expect (ObjNum 670 snap 205 seq_num 670) 2024-09-20T22:49:42.384 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd unset noscrub 2024-09-20T22:49:42.424 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:49:42.445 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:42.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:42.445 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:49:42.506 INFO:tasks.rados.rados.0.smithi084.stdout:1955: finishing write tid 1 to smithi08477203-25 2024-09-20T22:49:42.506 INFO:tasks.rados.rados.0.smithi084.stdout:1955: finishing write tid 2 to smithi08477203-25 2024-09-20T22:49:42.506 INFO:tasks.rados.rados.0.smithi084.stdout:1955: finishing write tid 3 to smithi08477203-25 2024-09-20T22:49:42.506 INFO:tasks.rados.rados.0.smithi084.stdout:1955: finishing write tid 4 to smithi08477203-25 2024-09-20T22:49:42.506 INFO:tasks.rados.rados.0.smithi084.stdout:1955: finishing write tid 5 to smithi08477203-25 2024-09-20T22:49:42.506 INFO:tasks.rados.rados.0.smithi084.stdout:1955: oid 25 updating version 0 to 1303 2024-09-20T22:49:42.507 INFO:tasks.rados.rados.0.smithi084.stdout:1955: oid 25 updating version 1303 to 1305 2024-09-20T22:49:42.507 INFO:tasks.rados.rados.0.smithi084.stdout:1955: oid 25 version 1305 is already newer than 1304 2024-09-20T22:49:42.507 INFO:tasks.rados.rados.0.smithi084.stdout:1955: oid 25 version 1305 is already newer than 1302 2024-09-20T22:49:42.507 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 25 v 1305 (ObjNum 688 snap 212 seq_num 688) dirty exists 2024-09-20T22:49:42.507 INFO:tasks.rados.rados.0.smithi084.stdout:1955: left oid 25 (ObjNum 688 snap 212 seq_num 688) 2024-09-20T22:49:42.507 INFO:tasks.rados.rados.0.smithi084.stdout:1956: finishing write tid 1 to smithi08477203-13 2024-09-20T22:49:42.507 INFO:tasks.rados.rados.0.smithi084.stdout:1956: finishing write tid 2 to smithi08477203-13 2024-09-20T22:49:42.507 INFO:tasks.rados.rados.0.smithi084.stdout:1956: finishing write tid 3 to smithi08477203-13 2024-09-20T22:49:42.507 INFO:tasks.rados.rados.0.smithi084.stdout:1956: finishing write tid 4 to smithi08477203-13 2024-09-20T22:49:42.507 INFO:tasks.rados.rados.0.smithi084.stdout:1956: finishing write tid 5 to smithi08477203-13 2024-09-20T22:49:42.507 INFO:tasks.rados.rados.0.smithi084.stdout:1956: oid 13 updating version 0 to 1307 2024-09-20T22:49:42.507 INFO:tasks.rados.rados.0.smithi084.stdout:1956: oid 13 updating version 1307 to 1310 2024-09-20T22:49:42.507 INFO:tasks.rados.rados.0.smithi084.stdout:1956: oid 13 version 1310 is already newer than 1308 2024-09-20T22:49:42.508 INFO:tasks.rados.rados.0.smithi084.stdout:1956: oid 13 version 1310 is already newer than 1309 2024-09-20T22:49:42.508 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 13 v 1310 (ObjNum 689 snap 212 seq_num 689) dirty exists 2024-09-20T22:49:42.508 INFO:tasks.rados.rados.0.smithi084.stdout:1956: left oid 13 (ObjNum 689 snap 212 seq_num 689) 2024-09-20T22:49:42.508 INFO:tasks.rados.rados.0.smithi084.stdout:1957: finishing write tid 1 to smithi08477203-17 2024-09-20T22:49:42.508 INFO:tasks.rados.rados.0.smithi084.stdout:1957: finishing write tid 2 to smithi08477203-17 2024-09-20T22:49:42.508 INFO:tasks.rados.rados.0.smithi084.stdout:1957: finishing write tid 3 to smithi08477203-17 2024-09-20T22:49:42.508 INFO:tasks.rados.rados.0.smithi084.stdout:1957: finishing write tid 4 to smithi08477203-17 2024-09-20T22:49:42.508 INFO:tasks.rados.rados.0.smithi084.stdout:1957: oid 17 updating version 0 to 742 2024-09-20T22:49:42.508 INFO:tasks.rados.rados.0.smithi084.stdout:1957: oid 17 version 742 is already newer than 741 2024-09-20T22:49:42.508 INFO:tasks.rados.rados.0.smithi084.stdout:1957: oid 17 updating version 742 to 743 2024-09-20T22:49:42.508 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 17 v 743 (ObjNum 690 snap 212 seq_num 690) dirty exists 2024-09-20T22:49:42.508 INFO:tasks.rados.rados.0.smithi084.stdout:1957: left oid 17 (ObjNum 690 snap 212 seq_num 690) 2024-09-20T22:49:42.546 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:49:42.567 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:42.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:42.568 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:49:42.614 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:42 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/2601766574' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-20T22:49:42.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:42 smithi084 ceph-mon[40024]: osdmap e501: 8 total, 8 up, 8 in 2024-09-20T22:49:42.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:42 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/2466795964' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:49:42.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:42 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/2601766574' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-20T22:49:42.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:42 smithi084 ceph-mon[29774]: osdmap e501: 8 total, 8 up, 8 in 2024-09-20T22:49:42.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:42 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/2466795964' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:49:42.658 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:42 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/2601766574' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-20T22:49:42.658 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:42 smithi154 ceph-mon[33264]: osdmap e501: 8 total, 8 up, 8 in 2024-09-20T22:49:42.658 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:42 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/2466795964' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:49:42.668 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:49:42.691 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:42.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:42.692 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:49:42.693 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-20T22:49:42.793 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:49:42.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:42.815 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:42.815 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:49:42.916 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:49:42.937 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:42.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:42.938 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:49:43.039 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:49:43.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:43.061 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:43.061 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:49:43.162 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:49:43.184 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:43.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:43.186 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:49:43.230 INFO:tasks.rados.rados.0.smithi084.stdout:1952: done (5 left) 2024-09-20T22:49:43.230 INFO:tasks.rados.rados.0.smithi084.stdout:1954: done (4 left) 2024-09-20T22:49:43.230 INFO:tasks.rados.rados.0.smithi084.stdout:1955: done (3 left) 2024-09-20T22:49:43.230 INFO:tasks.rados.rados.0.smithi084.stdout:1956: done (2 left) 2024-09-20T22:49:43.230 INFO:tasks.rados.rados.0.smithi084.stdout:1957: done (1 left) 2024-09-20T22:49:43.231 INFO:tasks.rados.rados.0.smithi084.stdout:1958: done (0 left) 2024-09-20T22:49:43.231 INFO:tasks.rados.rados.0.smithi084.stdout:1959: read oid 21 snap 189 2024-09-20T22:49:43.231 INFO:tasks.rados.rados.0.smithi084.stdout:1959: expect (ObjNum 501 snap 143 seq_num 501) 2024-09-20T22:49:43.231 INFO:tasks.rados.rados.0.smithi084.stdout:1960: read oid 45 snap -1 2024-09-20T22:49:43.231 INFO:tasks.rados.rados.0.smithi084.stdout:1960: expect deleted 2024-09-20T22:49:43.231 INFO:tasks.rados.rados.0.smithi084.stdout:1961: rollback oid 16 current snap is 213 2024-09-20T22:49:43.231 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 16 to 192 2024-09-20T22:49:43.232 INFO:tasks.rados.rados.0.smithi084.stdout:1962: read oid 5 snap -1 2024-09-20T22:49:43.232 INFO:tasks.rados.rados.0.smithi084.stdout:1962: expect (ObjNum 670 snap 205 seq_num 670) 2024-09-20T22:49:43.232 INFO:tasks.rados.rados.0.smithi084.stdout:1963: write oid 41 current snap is 213 2024-09-20T22:49:43.232 INFO:tasks.rados.rados.0.smithi084.stdout:1963: seq_num 691 ranges {413637=468829,1619175=435226,2542935=758255} 2024-09-20T22:49:43.234 INFO:tasks.rados.rados.0.smithi084.stdout:1963: writing smithi08477203-41 from 413637 to 882466 tid 1 2024-09-20T22:49:43.241 INFO:tasks.rados.rados.0.smithi084.stdout:1963: writing smithi08477203-41 from 1619175 to 2054401 tid 2 2024-09-20T22:49:43.250 INFO:tasks.rados.rados.0.smithi084.stdout:1963: writing smithi08477203-41 from 2542935 to 3301190 tid 3 2024-09-20T22:49:43.250 INFO:tasks.rados.rados.0.smithi084.stdout:1964: read oid 19 snap -1 2024-09-20T22:49:43.251 INFO:tasks.rados.rados.0.smithi084.stdout:1964: expect (ObjNum 656 snap 198 seq_num 656) 2024-09-20T22:49:43.251 INFO:tasks.rados.rados.0.smithi084.stdout:1965: snap_create 2024-09-20T22:49:43.251 INFO:tasks.rados.rados.0.smithi084.stdout:1962: expect (ObjNum 670 snap 205 seq_num 670) 2024-09-20T22:49:43.286 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:49:43.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:43.309 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:43.310 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:49:43.333 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:43 smithi084 ceph-mon[40024]: pgmap v654: 158 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 153 active+clean; 443 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 16 MiB/s rd, 2.2 MiB/s wr, 19 op/s 2024-09-20T22:49:43.333 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:43 smithi084 ceph-mon[40024]: osdmap e502: 8 total, 8 up, 8 in 2024-09-20T22:49:43.333 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:43 smithi084 ceph-mon[40024]: osdmap e503: 8 total, 8 up, 8 in 2024-09-20T22:49:43.333 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:43 smithi084 ceph-mon[29774]: pgmap v654: 158 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 153 active+clean; 443 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 16 MiB/s rd, 2.2 MiB/s wr, 19 op/s 2024-09-20T22:49:43.333 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:43 smithi084 ceph-mon[29774]: osdmap e502: 8 total, 8 up, 8 in 2024-09-20T22:49:43.333 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:43 smithi084 ceph-mon[29774]: osdmap e503: 8 total, 8 up, 8 in 2024-09-20T22:49:43.334 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd reweight 2 1 2024-09-20T22:49:43.405 INFO:tasks.rados.rados.0.smithi084.stdout:1961: finishing rollback tid 1 to smithi08477203-16 2024-09-20T22:49:43.405 INFO:tasks.rados.rados.0.smithi084.stdout:1961: finishing rollback tid 2 to smithi08477203-16 2024-09-20T22:49:43.405 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 16 v 425 (ObjNum 419 snap 105 seq_num 419) dirty exists 2024-09-20T22:49:43.406 INFO:tasks.rados.rados.0.smithi084.stdout:1963: finishing write tid 1 to smithi08477203-41 2024-09-20T22:49:43.406 INFO:tasks.rados.rados.0.smithi084.stdout:1963: finishing write tid 2 to smithi08477203-41 2024-09-20T22:49:43.406 INFO:tasks.rados.rados.0.smithi084.stdout:1959: expect (ObjNum 501 snap 143 seq_num 501) 2024-09-20T22:49:43.411 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:49:43.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:43.433 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:43.433 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:49:43.534 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:49:43.554 INFO:tasks.rados.rados.0.smithi084.stdout:1963: finishing write tid 3 to smithi08477203-41 2024-09-20T22:49:43.554 INFO:tasks.rados.rados.0.smithi084.stdout:1963: finishing write tid 4 to smithi08477203-41 2024-09-20T22:49:43.554 INFO:tasks.rados.rados.0.smithi084.stdout:1963: finishing write tid 5 to smithi08477203-41 2024-09-20T22:49:43.555 INFO:tasks.rados.rados.0.smithi084.stdout:1963: oid 41 updating version 0 to 710 2024-09-20T22:49:43.555 INFO:tasks.rados.rados.0.smithi084.stdout:1963: oid 41 version 710 is already newer than 709 2024-09-20T22:49:43.555 INFO:tasks.rados.rados.0.smithi084.stdout:1963: oid 41 updating version 710 to 711 2024-09-20T22:49:43.555 INFO:tasks.rados.rados.0.smithi084.stdout:1963: oid 41 version 711 is already newer than 708 2024-09-20T22:49:43.555 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 41 v 711 (ObjNum 691 snap 213 seq_num 691) dirty exists 2024-09-20T22:49:43.555 INFO:tasks.rados.rados.0.smithi084.stdout:1963: left oid 41 (ObjNum 691 snap 213 seq_num 691) 2024-09-20T22:49:43.555 INFO:tasks.rados.rados.0.smithi084.stdout:1964: expect (ObjNum 656 snap 198 seq_num 656) 2024-09-20T22:49:43.556 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:43.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:43.559 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:49:43.659 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:49:43.680 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:43.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:43.681 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:49:43.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:43 smithi154 ceph-mon[33264]: pgmap v654: 158 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 153 active+clean; 443 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 16 MiB/s rd, 2.2 MiB/s wr, 19 op/s 2024-09-20T22:49:43.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:43 smithi154 ceph-mon[33264]: osdmap e502: 8 total, 8 up, 8 in 2024-09-20T22:49:43.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:43 smithi154 ceph-mon[33264]: osdmap e503: 8 total, 8 up, 8 in 2024-09-20T22:49:43.783 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:49:43.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:43.805 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:43.805 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:49:43.906 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:49:43.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:43.929 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:43.929 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:49:44.021 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:49:44.028 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:49:44.030 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:49:44.035 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:49:44.056 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:44.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:44.057 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:49:44.158 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:49:44.181 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:44.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:44.182 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:49:44.235 INFO:tasks.rados.rados.0.smithi084.stdout:1959: done (6 left) 2024-09-20T22:49:44.235 INFO:tasks.rados.rados.0.smithi084.stdout:1960: done (5 left) 2024-09-20T22:49:44.236 INFO:tasks.rados.rados.0.smithi084.stdout:1961: done (4 left) 2024-09-20T22:49:44.236 INFO:tasks.rados.rados.0.smithi084.stdout:1962: done (3 left) 2024-09-20T22:49:44.236 INFO:tasks.rados.rados.0.smithi084.stdout:1963: done (2 left) 2024-09-20T22:49:44.236 INFO:tasks.rados.rados.0.smithi084.stdout:1964: done (1 left) 2024-09-20T22:49:44.236 INFO:tasks.rados.rados.0.smithi084.stdout:1965: done (0 left) 2024-09-20T22:49:44.236 INFO:tasks.rados.rados.0.smithi084.stdout:1966: rollback oid 31 current snap is 214 2024-09-20T22:49:44.236 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 31 to 195 2024-09-20T22:49:44.236 INFO:tasks.rados.rados.0.smithi084.stdout:1967: snap_create 2024-09-20T22:49:44.239 INFO:tasks.rados.rados.0.smithi084.stdout:1966: finishing rollback tid 0 to smithi08477203-31 2024-09-20T22:49:44.240 INFO:tasks.rados.rados.0.smithi084.stdout:1966: finishing rollback tid 1 to smithi08477203-31 2024-09-20T22:49:44.240 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 31 v 636 (ObjNum 97 snap 0 seq_num 709426828) dirty dne 2024-09-20T22:49:44.283 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:49:44.306 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:44.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:44.307 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:49:44.408 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:49:44.429 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:44.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:44.430 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:49:44.531 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:49:44.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:44.556 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:44.556 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:49:44.657 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:49:44.679 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:44.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:44.680 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:49:44.781 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:49:44.803 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:44.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:44.804 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:49:44.904 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:49:44.926 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:44.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:44.927 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:49:45.028 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:49:45.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:45.051 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:45.051 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:49:45.152 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:49:45.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:45.175 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:45.176 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:49:45.233 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.2 to 1 (10000) 2024-09-20T22:49:45.238 INFO:tasks.rados.rados.0.smithi084.stdout:1966: done (1 left) 2024-09-20T22:49:45.238 INFO:tasks.rados.rados.0.smithi084.stdout:1967: done (0 left) 2024-09-20T22:49:45.238 INFO:tasks.rados.rados.0.smithi084.stdout:1968: write oid 29 current snap is 215 2024-09-20T22:49:45.238 INFO:tasks.rados.rados.0.smithi084.stdout:1968: seq_num 692 ranges {430444=660431,1677657=695237,2970358=569527} 2024-09-20T22:49:45.244 INFO:tasks.rados.rados.0.smithi084.stdout:1968: writing smithi08477203-29 from 430444 to 1090875 tid 1 2024-09-20T22:49:45.251 INFO:tasks.rados.rados.0.smithi084.stdout:1968: writing smithi08477203-29 from 1677657 to 2372894 tid 2 2024-09-20T22:49:45.258 INFO:tasks.rados.rados.0.smithi084.stdout:1968: writing smithi08477203-29 from 2970358 to 3539885 tid 3 2024-09-20T22:49:45.258 INFO:tasks.rados.rados.0.smithi084.stdout:1969: write oid 32 current snap is 215 2024-09-20T22:49:45.259 INFO:tasks.rados.rados.0.smithi084.stdout:1968: finishing write tid 1 to smithi08477203-29 2024-09-20T22:49:45.259 INFO:tasks.rados.rados.0.smithi084.stdout:1969: seq_num 693 ranges {447251=735680,1736139=438895,2597781=780799} 2024-09-20T22:49:45.265 INFO:tasks.rados.rados.0.smithi084.stdout:1969: writing smithi08477203-32 from 447251 to 1182931 tid 1 2024-09-20T22:49:45.271 INFO:tasks.rados.rados.0.smithi084.stdout:1969: writing smithi08477203-32 from 1736139 to 2175034 tid 2 2024-09-20T22:49:45.277 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:49:45.278 INFO:tasks.rados.rados.0.smithi084.stdout:1969: writing smithi08477203-32 from 2597781 to 3378580 tid 3 2024-09-20T22:49:45.278 INFO:tasks.rados.rados.0.smithi084.stdout:1970: write (excl) oid 8 current snap is 215 2024-09-20T22:49:45.278 INFO:tasks.rados.rados.0.smithi084.stdout:1968: finishing write tid 2 to smithi08477203-29 2024-09-20T22:49:45.278 INFO:tasks.rados.rados.0.smithi084.stdout:1970: seq_num 694 ranges {464058=410929,1394621=698906,2625204=592071,3656107=7951} 2024-09-20T22:49:45.283 INFO:tasks.rados.rados.0.smithi084.stdout:1970: writing smithi08477203-8 from 464058 to 874987 tid 1 2024-09-20T22:49:45.289 INFO:tasks.rados.rados.0.smithi084.stdout:1970: writing smithi08477203-8 from 1394621 to 2093527 tid 2 2024-09-20T22:49:45.295 INFO:tasks.rados.rados.0.smithi084.stdout:1970: writing smithi08477203-8 from 2625204 to 3217275 tid 3 2024-09-20T22:49:45.300 INFO:tasks.rados.rados.0.smithi084.stdout:1970: writing smithi08477203-8 from 3656107 to 3664058 tid 4 2024-09-20T22:49:45.300 INFO:tasks.rados.rados.0.smithi084.stdout:1971: copy_from oid 10 from oid 42 current snap is 215 2024-09-20T22:49:45.300 INFO:tasks.rados.rados.0.smithi084.stdout:1968: finishing write tid 3 to smithi08477203-29 2024-09-20T22:49:45.300 INFO:tasks.rados.rados.0.smithi084.stdout:1968: finishing write tid 4 to smithi08477203-29 2024-09-20T22:49:45.300 INFO:tasks.rados.rados.0.smithi084.stdout:1968: finishing write tid 5 to smithi08477203-29 2024-09-20T22:49:45.300 INFO:tasks.rados.rados.0.smithi084.stdout:1968: oid 29 updating version 0 to 1315 2024-09-20T22:49:45.300 INFO:tasks.rados.rados.0.smithi084.stdout:1968: oid 29 version 1315 is already newer than 1313 2024-09-20T22:49:45.300 INFO:tasks.rados.rados.0.smithi084.stdout:1968: oid 29 version 1315 is already newer than 1314 2024-09-20T22:49:45.300 INFO:tasks.rados.rados.0.smithi084.stdout:1968: oid 29 version 1315 is already newer than 1312 2024-09-20T22:49:45.300 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 29 v 1315 (ObjNum 692 snap 215 seq_num 692) dirty exists 2024-09-20T22:49:45.301 INFO:tasks.rados.rados.0.smithi084.stdout:1968: left oid 29 (ObjNum 692 snap 215 seq_num 692) 2024-09-20T22:49:45.301 INFO:tasks.rados.rados.0.smithi084.stdout:1969: finishing write tid 1 to smithi08477203-32 2024-09-20T22:49:45.301 INFO:tasks.rados.rados.0.smithi084.stdout:1969: finishing write tid 2 to smithi08477203-32 2024-09-20T22:49:45.301 INFO:tasks.rados.rados.0.smithi084.stdout:1968: done (3 left) 2024-09-20T22:49:45.301 INFO:tasks.rados.rados.0.smithi084.stdout:1972: rollback oid 15 current snap is 215 2024-09-20T22:49:45.301 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 15 to 207 2024-09-20T22:49:45.301 INFO:tasks.rados.rados.0.smithi084.stdout:1969: finishing write tid 3 to smithi08477203-32 2024-09-20T22:49:45.301 INFO:tasks.rados.rados.0.smithi084.stdout:1969: finishing write tid 4 to smithi08477203-32 2024-09-20T22:49:45.301 INFO:tasks.rados.rados.0.smithi084.stdout:1969: finishing write tid 5 to smithi08477203-32 2024-09-20T22:49:45.301 INFO:tasks.rados.rados.0.smithi084.stdout:1969: oid 32 updating version 0 to 408 2024-09-20T22:49:45.301 INFO:tasks.rados.rados.0.smithi084.stdout:1969: oid 32 version 408 is already newer than 406 2024-09-20T22:49:45.301 INFO:tasks.rados.rados.0.smithi084.stdout:1969: oid 32 version 408 is already newer than 405 2024-09-20T22:49:45.301 INFO:tasks.rados.rados.0.smithi084.stdout:1969: oid 32 version 408 is already newer than 407 2024-09-20T22:49:45.301 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 32 v 408 (ObjNum 693 snap 215 seq_num 693) dirty exists 2024-09-20T22:49:45.302 INFO:tasks.rados.rados.0.smithi084.stdout:1969: left oid 32 (ObjNum 693 snap 215 seq_num 693) 2024-09-20T22:49:45.302 INFO:tasks.rados.rados.0.smithi084.stdout:1970: finishing write tid 1 to smithi08477203-8 2024-09-20T22:49:45.302 INFO:tasks.rados.rados.0.smithi084.stdout:1970: finishing write tid 2 to smithi08477203-8 2024-09-20T22:49:45.302 INFO:tasks.rados.rados.0.smithi084.stdout:1970: finishing write tid 3 to smithi08477203-8 2024-09-20T22:49:45.302 INFO:tasks.rados.rados.0.smithi084.stdout:1969: done (3 left) 2024-09-20T22:49:45.302 INFO:tasks.rados.rados.0.smithi084.stdout:1973: copy_from oid 40 from oid 31 current snap is 215 2024-09-20T22:49:45.302 INFO:tasks.rados.rados.0.smithi084.stdout:1974: rollback oid 50 current snap is 215 2024-09-20T22:49:45.302 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 50 to 214 2024-09-20T22:49:45.302 INFO:tasks.rados.rados.0.smithi084.stdout:1975: read oid 11 snap -1 2024-09-20T22:49:45.302 INFO:tasks.rados.rados.0.smithi084.stdout:1975: expect (ObjNum 678 snap 207 seq_num 678) 2024-09-20T22:49:45.302 INFO:tasks.rados.rados.0.smithi084.stdout:1976: write oid 22 current snap is 215 2024-09-20T22:49:45.302 INFO:tasks.rados.rados.0.smithi084.stdout:1976: seq_num 695 ranges {480865=486178,1453103=442564,2652627=519696} 2024-09-20T22:49:45.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:45.303 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:45.304 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:49:45.304 INFO:tasks.rados.rados.0.smithi084.stdout:1976: writing smithi08477203-22 from 480865 to 967043 tid 1 2024-09-20T22:49:45.310 INFO:tasks.rados.rados.0.smithi084.stdout:1976: writing smithi08477203-22 from 1453103 to 1895667 tid 2 2024-09-20T22:49:45.318 INFO:tasks.rados.rados.0.smithi084.stdout:1976: writing smithi08477203-22 from 2652627 to 3172323 tid 3 2024-09-20T22:49:45.318 INFO:tasks.rados.rados.0.smithi084.stdout:1977: write (excl) oid 23 current snap is 215 2024-09-20T22:49:45.318 INFO:tasks.rados.rados.0.smithi084.stdout:1977: seq_num 696 ranges {497672=561427,1795232=702575,2963697=730968} 2024-09-20T22:49:45.324 INFO:tasks.rados.rados.0.smithi084.stdout:1977: writing smithi08477203-23 from 497672 to 1059099 tid 1 2024-09-20T22:49:45.332 INFO:tasks.rados.rados.0.smithi084.stdout:1977: writing smithi08477203-23 from 1795232 to 2497807 tid 2 2024-09-20T22:49:45.339 INFO:tasks.rados.rados.0.smithi084.stdout:1977: writing smithi08477203-23 from 2963697 to 3694665 tid 3 2024-09-20T22:49:45.339 INFO:tasks.rados.rados.0.smithi084.stdout:1978: delete oid 5 current snap is 215 2024-09-20T22:49:45.339 INFO:tasks.rados.rados.0.smithi084.stdout:1973: finishing copy_from to smithi08477203-40 2024-09-20T22:49:45.339 INFO:tasks.rados.rados.0.smithi084.stdout:1973: got expected ENOENT (src dne) 2024-09-20T22:49:45.339 INFO:tasks.rados.rados.0.smithi084.stdout:1973: finishing copy_from racing read to smithi08477203-40 2024-09-20T22:49:45.339 INFO:tasks.rados.rados.0.smithi084.stdout:1974: finishing rollback tid 0 to smithi08477203-50 2024-09-20T22:49:45.340 INFO:tasks.rados.rados.0.smithi084.stdout:1974: finishing rollback tid 1 to smithi08477203-50 2024-09-20T22:49:45.340 INFO:tasks.rados.rados.0.smithi084.stdout:1970: finishing write tid 4 to smithi08477203-8 2024-09-20T22:49:45.340 INFO:tasks.rados.rados.0.smithi084.stdout:1970: finishing write tid 5 to smithi08477203-8 2024-09-20T22:49:45.340 INFO:tasks.rados.rados.0.smithi084.stdout:1974: finishing rollback tid 2 to smithi08477203-50 2024-09-20T22:49:45.340 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 50 v 428 (ObjNum 660 snap 199 seq_num 660) dirty exists 2024-09-20T22:49:45.340 INFO:tasks.rados.rados.0.smithi084.stdout:1972: finishing rollback tid 0 to smithi08477203-15 2024-09-20T22:49:45.340 INFO:tasks.rados.rados.0.smithi084.stdout:1970: finishing write tid 6 to smithi08477203-8 2024-09-20T22:49:45.340 INFO:tasks.rados.rados.0.smithi084.stdout:1970: oid 8 updating version 0 to 777 2024-09-20T22:49:45.340 INFO:tasks.rados.rados.0.smithi084.stdout:1970: oid 8 updating version 777 to 780 2024-09-20T22:49:45.340 INFO:tasks.rados.rados.0.smithi084.stdout:1970: oid 8 updating version 780 to 781 2024-09-20T22:49:45.340 INFO:tasks.rados.rados.0.smithi084.stdout:1970: oid 8 version 781 is already newer than 778 2024-09-20T22:49:45.340 INFO:tasks.rados.rados.0.smithi084.stdout:1970: oid 8 version 781 is already newer than 779 2024-09-20T22:49:45.340 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 8 v 781 (ObjNum 694 snap 215 seq_num 694) dirty exists 2024-09-20T22:49:45.341 INFO:tasks.rados.rados.0.smithi084.stdout:1970: left oid 8 (ObjNum 694 snap 215 seq_num 694) 2024-09-20T22:49:45.341 INFO:tasks.rados.rados.0.smithi084.stdout:1972: finishing rollback tid 1 to smithi08477203-15 2024-09-20T22:49:45.341 INFO:tasks.rados.rados.0.smithi084.stdout:1972: finishing rollback tid 2 to smithi08477203-15 2024-09-20T22:49:45.341 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 15 v 784 (ObjNum 625 snap 186 seq_num 625) dirty exists 2024-09-20T22:49:45.341 INFO:tasks.rados.rados.0.smithi084.stdout:1975: expect (ObjNum 678 snap 207 seq_num 678) 2024-09-20T22:49:45.364 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:45 smithi084 ceph-mon[40024]: pgmap v657: 158 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 153 active+clean; 468 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 15 MiB/s rd, 4.2 MiB/s wr, 24 op/s 2024-09-20T22:49:45.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:45 smithi084 ceph-mon[40024]: osdmap e504: 8 total, 8 up, 8 in 2024-09-20T22:49:45.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:45 smithi084 ceph-mon[40024]: purged_snaps scrub starts 2024-09-20T22:49:45.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:45 smithi084 ceph-mon[40024]: purged_snaps scrub ok 2024-09-20T22:49:45.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:45 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/2754206928' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-09-20T22:49:45.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:45 smithi084 ceph-mon[29774]: pgmap v657: 158 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 153 active+clean; 468 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 15 MiB/s rd, 4.2 MiB/s wr, 24 op/s 2024-09-20T22:49:45.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:45 smithi084 ceph-mon[29774]: osdmap e504: 8 total, 8 up, 8 in 2024-09-20T22:49:45.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:45 smithi084 ceph-mon[29774]: purged_snaps scrub starts 2024-09-20T22:49:45.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:45 smithi084 ceph-mon[29774]: purged_snaps scrub ok 2024-09-20T22:49:45.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:45 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/2754206928' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-09-20T22:49:45.404 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:49:45.421 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-20T22:49:45.422 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2024-09-20T22:49:45.422 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2024-09-20T22:49:45.422 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-20T22:49:45.427 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:45.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:45.428 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:49:45.469 INFO:tasks.rados.rados.0.smithi084.stdout:1976: finishing write tid 1 to smithi08477203-22 2024-09-20T22:49:45.469 INFO:tasks.rados.rados.0.smithi084.stdout:1971: finishing copy_from racing read to smithi08477203-10 2024-09-20T22:49:45.469 INFO:tasks.rados.rados.0.smithi084.stdout:1976: finishing write tid 2 to smithi08477203-22 2024-09-20T22:49:45.469 INFO:tasks.rados.rados.0.smithi084.stdout:1971: finishing copy_from to smithi08477203-10 2024-09-20T22:49:45.469 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 10 v 1207 (ObjNum 656 snap 198 seq_num 656) dirty exists 2024-09-20T22:49:45.469 INFO:tasks.rados.rados.0.smithi084.stdout:1976: finishing write tid 3 to smithi08477203-22 2024-09-20T22:49:45.469 INFO:tasks.rados.rados.0.smithi084.stdout:1976: finishing write tid 4 to smithi08477203-22 2024-09-20T22:49:45.470 INFO:tasks.rados.rados.0.smithi084.stdout:1976: finishing write tid 5 to smithi08477203-22 2024-09-20T22:49:45.470 INFO:tasks.rados.rados.0.smithi084.stdout:1976: oid 22 updating version 0 to 332 2024-09-20T22:49:45.470 INFO:tasks.rados.rados.0.smithi084.stdout:1976: oid 22 version 332 is already newer than 330 2024-09-20T22:49:45.470 INFO:tasks.rados.rados.0.smithi084.stdout:1976: oid 22 version 332 is already newer than 331 2024-09-20T22:49:45.470 INFO:tasks.rados.rados.0.smithi084.stdout:1976: oid 22 updating version 332 to 333 2024-09-20T22:49:45.470 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 22 v 333 (ObjNum 695 snap 215 seq_num 695) dirty exists 2024-09-20T22:49:45.470 INFO:tasks.rados.rados.0.smithi084.stdout:1976: left oid 22 (ObjNum 695 snap 215 seq_num 695) 2024-09-20T22:49:45.470 INFO:tasks.rados.rados.0.smithi084.stdout:1977: finishing write tid 1 to smithi08477203-23 2024-09-20T22:49:45.470 INFO:tasks.rados.rados.0.smithi084.stdout:1970: done (8 left) 2024-09-20T22:49:45.470 INFO:tasks.rados.rados.0.smithi084.stdout:1971: done (7 left) 2024-09-20T22:49:45.470 INFO:tasks.rados.rados.0.smithi084.stdout:1972: done (6 left) 2024-09-20T22:49:45.470 INFO:tasks.rados.rados.0.smithi084.stdout:1973: done (5 left) 2024-09-20T22:49:45.470 INFO:tasks.rados.rados.0.smithi084.stdout:1974: done (4 left) 2024-09-20T22:49:45.470 INFO:tasks.rados.rados.0.smithi084.stdout:1975: done (3 left) 2024-09-20T22:49:45.471 INFO:tasks.rados.rados.0.smithi084.stdout:1976: done (2 left) 2024-09-20T22:49:45.471 INFO:tasks.rados.rados.0.smithi084.stdout:1978: done (1 left) 2024-09-20T22:49:45.471 INFO:tasks.rados.rados.0.smithi084.stdout:1979: rollback oid 22 current snap is 215 2024-09-20T22:49:45.471 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 22 to 207 2024-09-20T22:49:45.471 INFO:tasks.rados.rados.0.smithi084.stdout:1977: finishing write tid 2 to smithi08477203-23 2024-09-20T22:49:45.471 INFO:tasks.rados.rados.0.smithi084.stdout:1977: finishing write tid 3 to smithi08477203-23 2024-09-20T22:49:45.471 INFO:tasks.rados.rados.0.smithi084.stdout:1977: finishing write tid 4 to smithi08477203-23 2024-09-20T22:49:45.471 INFO:tasks.rados.rados.0.smithi084.stdout:1977: finishing write tid 5 to smithi08477203-23 2024-09-20T22:49:45.471 INFO:tasks.rados.rados.0.smithi084.stdout:1977: oid 23 updating version 0 to 727 2024-09-20T22:49:45.471 INFO:tasks.rados.rados.0.smithi084.stdout:1977: oid 23 updating version 727 to 729 2024-09-20T22:49:45.471 INFO:tasks.rados.rados.0.smithi084.stdout:1977: oid 23 version 729 is already newer than 728 2024-09-20T22:49:45.471 INFO:tasks.rados.rados.0.smithi084.stdout:1977: oid 23 updating version 729 to 730 2024-09-20T22:49:45.471 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 23 v 730 (ObjNum 696 snap 215 seq_num 696) dirty exists 2024-09-20T22:49:45.471 INFO:tasks.rados.rados.0.smithi084.stdout:1977: left oid 23 (ObjNum 696 snap 215 seq_num 696) 2024-09-20T22:49:45.471 INFO:tasks.rados.rados.0.smithi084.stdout:1977: done (1 left) 2024-09-20T22:49:45.472 INFO:tasks.rados.rados.0.smithi084.stdout:1980: write oid 10 current snap is 215 2024-09-20T22:49:45.472 INFO:tasks.rados.rados.0.smithi084.stdout:1980: seq_num 698 ranges {531286=711925,1912196=706244,3018543=712743} 2024-09-20T22:49:45.475 INFO:tasks.rados.rados.0.smithi084.stdout:1980: writing smithi08477203-10 from 531286 to 1243211 tid 1 2024-09-20T22:49:45.483 INFO:tasks.rados.rados.0.smithi084.stdout:1980: writing smithi08477203-10 from 1912196 to 2618440 tid 2 2024-09-20T22:49:45.489 INFO:tasks.rados.rados.0.smithi084.stdout:1980: writing smithi08477203-10 from 3018543 to 3731286 tid 3 2024-09-20T22:49:45.489 INFO:tasks.rados.rados.0.smithi084.stdout:1981: rollback oid 34 current snap is 215 2024-09-20T22:49:45.489 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 34 to 213 2024-09-20T22:49:45.489 INFO:tasks.rados.rados.0.smithi084.stdout:1979: finishing rollback tid 0 to smithi08477203-22 2024-09-20T22:49:45.490 INFO:tasks.rados.rados.0.smithi084.stdout:1979: finishing rollback tid 1 to smithi08477203-22 2024-09-20T22:49:45.490 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 22 v 335 (ObjNum 65 snap 0 seq_num 711107362) dirty dne 2024-09-20T22:49:45.490 INFO:tasks.rados.rados.0.smithi084.stdout:1980: finishing write tid 1 to smithi08477203-10 2024-09-20T22:49:45.490 INFO:tasks.rados.rados.0.smithi084.stdout:1980: finishing write tid 2 to smithi08477203-10 2024-09-20T22:49:45.490 INFO:tasks.rados.rados.0.smithi084.stdout:1979: done (2 left) 2024-09-20T22:49:45.490 INFO:tasks.rados.rados.0.smithi084.stdout:1982: snap_create 2024-09-20T22:49:45.492 INFO:tasks.rados.rados.0.smithi084.stdout:1981: finishing rollback tid 0 to smithi08477203-34 2024-09-20T22:49:45.492 INFO:tasks.rados.rados.0.smithi084.stdout:1981: finishing rollback tid 1 to smithi08477203-34 2024-09-20T22:49:45.493 INFO:tasks.rados.rados.0.smithi084.stdout:1981: finishing rollback tid 2 to smithi08477203-34 2024-09-20T22:49:45.493 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 34 v 543 (ObjNum 622 snap 186 seq_num 622) dirty exists 2024-09-20T22:49:45.493 INFO:tasks.rados.rados.0.smithi084.stdout:1980: finishing write tid 3 to smithi08477203-10 2024-09-20T22:49:45.493 INFO:tasks.rados.rados.0.smithi084.stdout:1980: finishing write tid 4 to smithi08477203-10 2024-09-20T22:49:45.493 INFO:tasks.rados.rados.0.smithi084.stdout:1980: finishing write tid 5 to smithi08477203-10 2024-09-20T22:49:45.493 INFO:tasks.rados.rados.0.smithi084.stdout:1980: oid 10 updating version 0 to 1208 2024-09-20T22:49:45.493 INFO:tasks.rados.rados.0.smithi084.stdout:1980: oid 10 updating version 1208 to 1209 2024-09-20T22:49:45.493 INFO:tasks.rados.rados.0.smithi084.stdout:1980: oid 10 updating version 1209 to 1211 2024-09-20T22:49:45.493 INFO:tasks.rados.rados.0.smithi084.stdout:1980: oid 10 version 1211 is already newer than 1210 2024-09-20T22:49:45.493 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 10 v 1211 (ObjNum 698 snap 215 seq_num 698) dirty exists 2024-09-20T22:49:45.494 INFO:tasks.rados.rados.0.smithi084.stdout:1980: left oid 10 (ObjNum 698 snap 215 seq_num 698) 2024-09-20T22:49:45.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:45 smithi154 ceph-mon[33264]: pgmap v657: 158 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 153 active+clean; 468 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 15 MiB/s rd, 4.2 MiB/s wr, 24 op/s 2024-09-20T22:49:45.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:45 smithi154 ceph-mon[33264]: osdmap e504: 8 total, 8 up, 8 in 2024-09-20T22:49:45.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:45 smithi154 ceph-mon[33264]: purged_snaps scrub starts 2024-09-20T22:49:45.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:45 smithi154 ceph-mon[33264]: purged_snaps scrub ok 2024-09-20T22:49:45.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:45 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/2754206928' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-09-20T22:49:45.529 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:49:45.550 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:45.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:45.551 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:49:45.652 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:49:45.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:45.673 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:45.674 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:49:45.774 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:49:45.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:45.797 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:45.797 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:49:45.898 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:49:45.920 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:45.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:45.921 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:49:45.983 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-20T22:49:46.022 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:49:46.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:46.048 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:46.049 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:49:46.149 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:49:46.155 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd reweight 3 1 2024-09-20T22:49:46.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:46.173 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:46.173 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:49:46.243 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is unset 2024-09-20T22:49:46.248 INFO:tasks.rados.rados.0.smithi084.stdout:1980: done (2 left) 2024-09-20T22:49:46.249 INFO:tasks.rados.rados.0.smithi084.stdout:1981: done (1 left) 2024-09-20T22:49:46.249 INFO:tasks.rados.rados.0.smithi084.stdout:1982: done (0 left) 2024-09-20T22:49:46.249 INFO:tasks.rados.rados.0.smithi084.stdout:1983: read oid 26 snap -1 2024-09-20T22:49:46.249 INFO:tasks.rados.rados.0.smithi084.stdout:1983: expect deleted 2024-09-20T22:49:46.249 INFO:tasks.rados.rados.0.smithi084.stdout:1984: read oid 15 snap 200 2024-09-20T22:49:46.249 INFO:tasks.rados.rados.0.smithi084.stdout:1984: expect (ObjNum 625 snap 186 seq_num 625) 2024-09-20T22:49:46.249 INFO:tasks.rados.rados.0.smithi084.stdout:1985: snap_create 2024-09-20T22:49:46.254 INFO:tasks.rados.rados.0.smithi084.stdout:1984: expect (ObjNum 625 snap 186 seq_num 625) 2024-09-20T22:49:46.261 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:46 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/2754206928' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-20T22:49:46.261 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:46 smithi084 ceph-mon[40024]: osdmap e505: 8 total, 8 up, 8 in 2024-09-20T22:49:46.261 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:46 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/3221793875' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:49:46.261 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:46 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/2754206928' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-20T22:49:46.261 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:46 smithi084 ceph-mon[29774]: osdmap e505: 8 total, 8 up, 8 in 2024-09-20T22:49:46.261 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:46 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/3221793875' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:49:46.274 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:49:46.296 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:46.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:46.297 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:49:46.398 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:49:46.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:46.422 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:46.423 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:49:46.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:46 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/2754206928' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-20T22:49:46.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:46 smithi154 ceph-mon[33264]: osdmap e505: 8 total, 8 up, 8 in 2024-09-20T22:49:46.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:46 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/3221793875' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:49:46.523 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:49:46.544 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:46.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:46.545 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:49:46.646 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:49:46.668 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:46.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:46.669 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:49:46.769 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:49:46.791 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:46.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:46.792 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:49:46.893 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:49:46.913 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:46.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:46.914 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:49:47.015 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:49:47.037 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:47.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:47.038 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:49:47.138 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:49:47.159 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:47.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:47.160 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:49:47.258 INFO:tasks.rados.rados.0.smithi084.stdout:1983: done (2 left) 2024-09-20T22:49:47.258 INFO:tasks.rados.rados.0.smithi084.stdout:1984: done (1 left) 2024-09-20T22:49:47.258 INFO:tasks.rados.rados.0.smithi084.stdout:1985: done (0 left) 2024-09-20T22:49:47.259 INFO:tasks.rados.rados.0.smithi084.stdout:1986: snap_remove snap 208 2024-09-20T22:49:47.261 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:49:47.283 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:47.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:47.284 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:49:47.364 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:47 smithi084 ceph-mon[29774]: pgmap v660: 158 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 153 active+clean; 468 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 15 MiB/s rd, 4.2 MiB/s wr, 24 op/s 2024-09-20T22:49:47.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:47 smithi084 ceph-mon[29774]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:49:47.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:47 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/3221793875' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T22:49:47.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:47 smithi084 ceph-mon[29774]: osdmap e506: 8 total, 8 up, 8 in 2024-09-20T22:49:47.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:47 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:49:47.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:47 smithi084 ceph-mon[29774]: 1.0 scrub starts 2024-09-20T22:49:47.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:47 smithi084 ceph-mon[29774]: 1.0 scrub ok 2024-09-20T22:49:47.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:47 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:49:47.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:47 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:49:47.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:47 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:49:47.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:47 smithi084 ceph-mon[29774]: 4.2 scrub starts 2024-09-20T22:49:47.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:47 smithi084 ceph-mon[29774]: 4.2 scrub ok 2024-09-20T22:49:47.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:47 smithi084 ceph-mon[40024]: pgmap v660: 158 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 153 active+clean; 468 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 15 MiB/s rd, 4.2 MiB/s wr, 24 op/s 2024-09-20T22:49:47.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:47 smithi084 ceph-mon[40024]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:49:47.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:47 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/3221793875' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T22:49:47.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:47 smithi084 ceph-mon[40024]: osdmap e506: 8 total, 8 up, 8 in 2024-09-20T22:49:47.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:47 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:49:47.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:47 smithi084 ceph-mon[40024]: 1.0 scrub starts 2024-09-20T22:49:47.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:47 smithi084 ceph-mon[40024]: 1.0 scrub ok 2024-09-20T22:49:47.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:47 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:49:47.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:47 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:49:47.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:47 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:49:47.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:47 smithi084 ceph-mon[40024]: 4.2 scrub starts 2024-09-20T22:49:47.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:47 smithi084 ceph-mon[40024]: 4.2 scrub ok 2024-09-20T22:49:47.385 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:49:47.407 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:47.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:47.408 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:49:47.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:47 smithi154 ceph-mon[33264]: pgmap v660: 158 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 153 active+clean; 468 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 15 MiB/s rd, 4.2 MiB/s wr, 24 op/s 2024-09-20T22:49:47.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:47 smithi154 ceph-mon[33264]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:49:47.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:47 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/3221793875' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T22:49:47.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:47 smithi154 ceph-mon[33264]: osdmap e506: 8 total, 8 up, 8 in 2024-09-20T22:49:47.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:47 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:49:47.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:47 smithi154 ceph-mon[33264]: 1.0 scrub starts 2024-09-20T22:49:47.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:47 smithi154 ceph-mon[33264]: 1.0 scrub ok 2024-09-20T22:49:47.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:47 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:49:47.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:47 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:49:47.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:47 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:49:47.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:47 smithi154 ceph-mon[33264]: 4.2 scrub starts 2024-09-20T22:49:47.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:47 smithi154 ceph-mon[33264]: 4.2 scrub ok 2024-09-20T22:49:47.508 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:49:47.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:47.532 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:47.533 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:49:47.634 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:49:47.656 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.2 dump_blocked_ops 2024-09-20T22:49:47.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:47.657 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:47.657 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:49:47.758 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:49:47.769 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:49:47.770 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:49:47.782 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:47.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:47.783 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:49:47.884 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:49:47.906 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:47.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:47.907 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:49:48.008 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:49:48.030 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:48.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:48.031 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:49:48.132 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:49:48.153 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:48.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:48.154 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:49:48.236 INFO:tasks.rados.rados.0.smithi084.stdout:1986: done (0 left) 2024-09-20T22:49:48.237 INFO:tasks.rados.rados.0.smithi084.stdout:1987: rollback oid 47 current snap is 217 2024-09-20T22:49:48.237 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 47 to 215 2024-09-20T22:49:48.237 INFO:tasks.rados.rados.0.smithi084.stdout:1988: write (excl) oid 30 current snap is 217 2024-09-20T22:49:48.237 INFO:tasks.rados.rados.0.smithi084.stdout:1988: seq_num 699 ranges {548093=787174,1970678=449902,3045966=681137} 2024-09-20T22:49:48.244 INFO:tasks.rados.rados.0.smithi084.stdout:1988: writing smithi08477203-30 from 548093 to 1335267 tid 1 2024-09-20T22:49:48.251 INFO:tasks.rados.rados.0.smithi084.stdout:1988: writing smithi08477203-30 from 1970678 to 2420580 tid 2 2024-09-20T22:49:48.255 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:49:48.260 INFO:tasks.rados.rados.0.smithi084.stdout:1988: writing smithi08477203-30 from 3045966 to 3727103 tid 3 2024-09-20T22:49:48.260 INFO:tasks.rados.rados.0.smithi084.stdout:1989: delete oid 20 current snap is 217 2024-09-20T22:49:48.260 INFO:tasks.rados.rados.0.smithi084.stdout:1987: finishing rollback tid 0 to smithi08477203-47 2024-09-20T22:49:48.261 INFO:tasks.rados.rados.0.smithi084.stdout:1987: finishing rollback tid 1 to smithi08477203-47 2024-09-20T22:49:48.261 INFO:tasks.rados.rados.0.smithi084.stdout:1987: finishing rollback tid 2 to smithi08477203-47 2024-09-20T22:49:48.261 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 47 v 640 (ObjNum 671 snap 205 seq_num 671) dirty exists 2024-09-20T22:49:48.268 INFO:tasks.rados.rados.0.smithi084.stdout:1987: done (2 left) 2024-09-20T22:49:48.268 INFO:tasks.rados.rados.0.smithi084.stdout:1989: done (1 left) 2024-09-20T22:49:48.268 INFO:tasks.rados.rados.0.smithi084.stdout:1990: rollback oid 44 current snap is 217 2024-09-20T22:49:48.268 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 44 to 184 2024-09-20T22:49:48.268 INFO:tasks.rados.rados.0.smithi084.stdout:1991: snap_create 2024-09-20T22:49:48.276 INFO:tasks.rados.rados.0.smithi084.stdout:1990: finishing rollback tid 0 to smithi08477203-44 2024-09-20T22:49:48.276 INFO:tasks.rados.rados.0.smithi084.stdout:1990: finishing rollback tid 1 to smithi08477203-44 2024-09-20T22:49:48.276 INFO:tasks.rados.rados.0.smithi084.stdout:1990: finishing rollback tid 2 to smithi08477203-44 2024-09-20T22:49:48.276 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 44 v 1323 (ObjNum 470 snap 130 seq_num 470) dirty exists 2024-09-20T22:49:48.277 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:48.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:48.278 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:49:48.283 INFO:tasks.rados.rados.0.smithi084.stdout:1988: finishing write tid 1 to smithi08477203-30 2024-09-20T22:49:48.283 INFO:tasks.rados.rados.0.smithi084.stdout:1988: finishing write tid 2 to smithi08477203-30 2024-09-20T22:49:48.283 INFO:tasks.rados.rados.0.smithi084.stdout:1988: finishing write tid 3 to smithi08477203-30 2024-09-20T22:49:48.283 INFO:tasks.rados.rados.0.smithi084.stdout:1988: finishing write tid 4 to smithi08477203-30 2024-09-20T22:49:48.283 INFO:tasks.rados.rados.0.smithi084.stdout:1988: finishing write tid 5 to smithi08477203-30 2024-09-20T22:49:48.283 INFO:tasks.rados.rados.0.smithi084.stdout:1988: oid 30 updating version 0 to 786 2024-09-20T22:49:48.284 INFO:tasks.rados.rados.0.smithi084.stdout:1988: oid 30 updating version 786 to 788 2024-09-20T22:49:48.284 INFO:tasks.rados.rados.0.smithi084.stdout:1988: oid 30 version 788 is already newer than 785 2024-09-20T22:49:48.284 INFO:tasks.rados.rados.0.smithi084.stdout:1988: oid 30 version 788 is already newer than 787 2024-09-20T22:49:48.284 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 30 v 788 (ObjNum 699 snap 217 seq_num 699) dirty exists 2024-09-20T22:49:48.284 INFO:tasks.rados.rados.0.smithi084.stdout:1988: left oid 30 (ObjNum 699 snap 217 seq_num 699) 2024-09-20T22:49:48.378 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:49:48.400 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:48.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:48.401 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:49:48.502 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:49:48.519 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:49:48 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: ::ffff:172.21.15.154 - - [20/Sep/2024:22:49:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-20T22:49:48.519 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:48 smithi084 ceph-mon[40024]: 5.1f scrub starts 2024-09-20T22:49:48.520 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:48 smithi084 ceph-mon[40024]: 5.1f scrub ok 2024-09-20T22:49:48.520 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:48 smithi084 ceph-mon[40024]: 7.a scrub starts 2024-09-20T22:49:48.520 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:48 smithi084 ceph-mon[40024]: 7.a scrub ok 2024-09-20T22:49:48.520 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:48 smithi084 ceph-mon[40024]: osdmap e507: 8 total, 8 up, 8 in 2024-09-20T22:49:48.520 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:48 smithi084 ceph-mon[40024]: 5.9 scrub starts 2024-09-20T22:49:48.520 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:48 smithi084 ceph-mon[40024]: 5.9 scrub ok 2024-09-20T22:49:48.520 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:48 smithi084 ceph-mon[40024]: osdmap e508: 8 total, 8 up, 8 in 2024-09-20T22:49:48.520 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:48 smithi084 ceph-mon[29774]: 5.1f scrub starts 2024-09-20T22:49:48.520 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:48 smithi084 ceph-mon[29774]: 5.1f scrub ok 2024-09-20T22:49:48.520 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:48 smithi084 ceph-mon[29774]: 7.a scrub starts 2024-09-20T22:49:48.521 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:48 smithi084 ceph-mon[29774]: 7.a scrub ok 2024-09-20T22:49:48.521 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:48 smithi084 ceph-mon[29774]: osdmap e507: 8 total, 8 up, 8 in 2024-09-20T22:49:48.521 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:48 smithi084 ceph-mon[29774]: 5.9 scrub starts 2024-09-20T22:49:48.521 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:48 smithi084 ceph-mon[29774]: 5.9 scrub ok 2024-09-20T22:49:48.521 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:48 smithi084 ceph-mon[29774]: osdmap e508: 8 total, 8 up, 8 in 2024-09-20T22:49:48.524 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:48.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:48.525 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:49:48.626 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:49:48.647 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:48.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:48.648 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:49:48.749 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:49:48.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:48 smithi154 ceph-mon[33264]: 5.1f scrub starts 2024-09-20T22:49:48.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:48 smithi154 ceph-mon[33264]: 5.1f scrub ok 2024-09-20T22:49:48.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:48 smithi154 ceph-mon[33264]: 7.a scrub starts 2024-09-20T22:49:48.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:48 smithi154 ceph-mon[33264]: 7.a scrub ok 2024-09-20T22:49:48.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:48 smithi154 ceph-mon[33264]: osdmap e507: 8 total, 8 up, 8 in 2024-09-20T22:49:48.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:48 smithi154 ceph-mon[33264]: 5.9 scrub starts 2024-09-20T22:49:48.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:48 smithi154 ceph-mon[33264]: 5.9 scrub ok 2024-09-20T22:49:48.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:48 smithi154 ceph-mon[33264]: osdmap e508: 8 total, 8 up, 8 in 2024-09-20T22:49:48.770 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:49:48.770 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:49:48.770 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:49:48.770 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-20T22:49:48.770 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:49:48.770 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:49:48.770 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:49:48.770 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:49:48.771 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T22:49:48.771 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:49:48.771 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:49:48.771 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:49:48.771 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:49:48.771 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:49:48.772 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:49:48.772 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:49:48.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:48.773 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:48.773 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:49:48.873 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:49:48.895 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:48.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:48.896 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:49:48.997 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:49:49.019 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:49.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:49.020 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:49:49.069 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-20T22:49:49.121 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:49:49.141 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:49.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:49.142 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:49:49.243 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:49:49.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:49.268 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:49.268 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:49:49.270 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.3 to 1 (10000) 2024-09-20T22:49:49.275 INFO:tasks.rados.rados.0.smithi084.stdout:1988: done (2 left) 2024-09-20T22:49:49.275 INFO:tasks.rados.rados.0.smithi084.stdout:1990: done (1 left) 2024-09-20T22:49:49.275 INFO:tasks.rados.rados.0.smithi084.stdout:1991: done (0 left) 2024-09-20T22:49:49.275 INFO:tasks.rados.rados.0.smithi084.stdout:1992: copy_from oid 27 from oid 4 current snap is 218 2024-09-20T22:49:49.275 INFO:tasks.rados.rados.0.smithi084.stdout:1993: write oid 16 current snap is 218 2024-09-20T22:49:49.275 INFO:tasks.rados.rados.0.smithi084.stdout:1993: seq_num 701 ranges {581707=654025,1687642=453571,2700812=703681} 2024-09-20T22:49:49.282 INFO:tasks.rados.rados.0.smithi084.stdout:1993: writing smithi08477203-16 from 581707 to 1235732 tid 1 2024-09-20T22:49:49.289 INFO:tasks.rados.rados.0.smithi084.stdout:1993: writing smithi08477203-16 from 1687642 to 2141213 tid 2 2024-09-20T22:49:49.297 INFO:tasks.rados.rados.0.smithi084.stdout:1993: writing smithi08477203-16 from 2700812 to 3404493 tid 3 2024-09-20T22:49:49.297 INFO:tasks.rados.rados.0.smithi084.stdout:1994: read oid 18 snap -1 2024-09-20T22:49:49.297 INFO:tasks.rados.rados.0.smithi084.stdout:1994: expect (ObjNum 657 snap 199 seq_num 657) 2024-09-20T22:49:49.297 INFO:tasks.rados.rados.0.smithi084.stdout:1995: read oid 43 snap -1 2024-09-20T22:49:49.297 INFO:tasks.rados.rados.0.smithi084.stdout:1995: expect deleted 2024-09-20T22:49:49.297 INFO:tasks.rados.rados.0.smithi084.stdout:1996: read oid 29 snap 178 2024-09-20T22:49:49.297 INFO:tasks.rados.rados.0.smithi084.stdout:1996: expect (ObjNum 583 snap 173 seq_num 583) 2024-09-20T22:49:49.298 INFO:tasks.rados.rados.0.smithi084.stdout:1997: read oid 15 snap -1 2024-09-20T22:49:49.298 INFO:tasks.rados.rados.0.smithi084.stdout:1997: expect (ObjNum 625 snap 186 seq_num 625) 2024-09-20T22:49:49.298 INFO:tasks.rados.rados.0.smithi084.stdout:1998: snap_create 2024-09-20T22:49:49.298 INFO:tasks.rados.rados.0.smithi084.stdout:1992: finishing copy_from to smithi08477203-27 2024-09-20T22:49:49.298 INFO:tasks.rados.rados.0.smithi084.stdout:1992: got expected ENOENT (src dne) 2024-09-20T22:49:49.298 INFO:tasks.rados.rados.0.smithi084.stdout:1992: finishing copy_from racing read to smithi08477203-27 2024-09-20T22:49:49.298 INFO:tasks.rados.rados.0.smithi084.stdout:1993: finishing write tid 1 to smithi08477203-16 2024-09-20T22:49:49.298 INFO:tasks.rados.rados.0.smithi084.stdout:1993: finishing write tid 2 to smithi08477203-16 2024-09-20T22:49:49.302 INFO:tasks.rados.rados.0.smithi084.stdout:1994: expect (ObjNum 657 snap 199 seq_num 657) 2024-09-20T22:49:49.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:49 smithi084 ceph-mon[40024]: purged_snaps scrub starts 2024-09-20T22:49:49.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:49 smithi084 ceph-mon[40024]: purged_snaps scrub ok 2024-09-20T22:49:49.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:49 smithi084 ceph-mon[40024]: 4.8 scrub starts 2024-09-20T22:49:49.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:49 smithi084 ceph-mon[40024]: 4.8 scrub ok 2024-09-20T22:49:49.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:49 smithi084 ceph-mon[40024]: pgmap v663: 158 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 153 active+clean; 468 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-20T22:49:49.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:49 smithi084 ceph-mon[40024]: 4.17 scrub starts 2024-09-20T22:49:49.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:49 smithi084 ceph-mon[40024]: 4.17 scrub ok 2024-09-20T22:49:49.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:49 smithi084 ceph-mon[40024]: 5.1a scrub starts 2024-09-20T22:49:49.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:49 smithi084 ceph-mon[40024]: 5.1a scrub ok 2024-09-20T22:49:49.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:49 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/3216014955' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:49:49.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:49 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/733301550' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-09-20T22:49:49.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:49 smithi084 ceph-mon[40024]: 7.c scrub starts 2024-09-20T22:49:49.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:49 smithi084 ceph-mon[40024]: 7.c scrub ok 2024-09-20T22:49:49.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:49 smithi084 ceph-mon[40024]: 6.1a scrub starts 2024-09-20T22:49:49.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:49 smithi084 ceph-mon[40024]: 6.1a scrub ok 2024-09-20T22:49:49.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:49 smithi084 ceph-mon[29774]: purged_snaps scrub starts 2024-09-20T22:49:49.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:49 smithi084 ceph-mon[29774]: purged_snaps scrub ok 2024-09-20T22:49:49.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:49 smithi084 ceph-mon[29774]: 4.8 scrub starts 2024-09-20T22:49:49.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:49 smithi084 ceph-mon[29774]: 4.8 scrub ok 2024-09-20T22:49:49.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:49 smithi084 ceph-mon[29774]: pgmap v663: 158 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 153 active+clean; 468 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-20T22:49:49.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:49 smithi084 ceph-mon[29774]: 4.17 scrub starts 2024-09-20T22:49:49.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:49 smithi084 ceph-mon[29774]: 4.17 scrub ok 2024-09-20T22:49:49.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:49 smithi084 ceph-mon[29774]: 5.1a scrub starts 2024-09-20T22:49:49.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:49 smithi084 ceph-mon[29774]: 5.1a scrub ok 2024-09-20T22:49:49.367 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:49 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/3216014955' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:49:49.367 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:49 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/733301550' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-09-20T22:49:49.367 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:49 smithi084 ceph-mon[29774]: 7.c scrub starts 2024-09-20T22:49:49.367 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:49 smithi084 ceph-mon[29774]: 7.c scrub ok 2024-09-20T22:49:49.367 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:49 smithi084 ceph-mon[29774]: 6.1a scrub starts 2024-09-20T22:49:49.367 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:49 smithi084 ceph-mon[29774]: 6.1a scrub ok 2024-09-20T22:49:49.370 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:49:49.391 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:49.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:49.392 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:49:49.493 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:49:49.514 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:49.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:49.515 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:49:49.539 INFO:tasks.rados.rados.0.smithi084.stdout:1997: expect (ObjNum 625 snap 186 seq_num 625) 2024-09-20T22:49:49.616 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:49:49.636 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:49.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:49.637 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:49:49.655 INFO:tasks.rados.rados.0.smithi084.stdout:1993: finishing write tid 3 to smithi08477203-16 2024-09-20T22:49:49.656 INFO:tasks.rados.rados.0.smithi084.stdout:1993: finishing write tid 4 to smithi08477203-16 2024-09-20T22:49:49.656 INFO:tasks.rados.rados.0.smithi084.stdout:1993: finishing write tid 5 to smithi08477203-16 2024-09-20T22:49:49.656 INFO:tasks.rados.rados.0.smithi084.stdout:1993: oid 16 updating version 0 to 427 2024-09-20T22:49:49.656 INFO:tasks.rados.rados.0.smithi084.stdout:1993: oid 16 updating version 427 to 428 2024-09-20T22:49:49.656 INFO:tasks.rados.rados.0.smithi084.stdout:1993: oid 16 version 428 is already newer than 426 2024-09-20T22:49:49.656 INFO:tasks.rados.rados.0.smithi084.stdout:1993: oid 16 updating version 428 to 429 2024-09-20T22:49:49.656 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 16 v 429 (ObjNum 701 snap 218 seq_num 701) dirty exists 2024-09-20T22:49:49.656 INFO:tasks.rados.rados.0.smithi084.stdout:1993: left oid 16 (ObjNum 701 snap 218 seq_num 701) 2024-09-20T22:49:49.656 INFO:tasks.rados.rados.0.smithi084.stdout:1996: expect (ObjNum 583 snap 173 seq_num 583) 2024-09-20T22:49:49.738 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:49:49.754 INFO:journalctl@ceph.iscsi.iscsi.a.smithi154.stdout:Sep 20 22:49:49 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-iscsi-iscsi-a[59730]: debug there is no tcmu-runner data available 2024-09-20T22:49:49.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:49 smithi154 ceph-mon[33264]: purged_snaps scrub starts 2024-09-20T22:49:49.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:49 smithi154 ceph-mon[33264]: purged_snaps scrub ok 2024-09-20T22:49:49.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:49 smithi154 ceph-mon[33264]: 4.8 scrub starts 2024-09-20T22:49:49.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:49 smithi154 ceph-mon[33264]: 4.8 scrub ok 2024-09-20T22:49:49.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:49 smithi154 ceph-mon[33264]: pgmap v663: 158 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 153 active+clean; 468 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail 2024-09-20T22:49:49.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:49 smithi154 ceph-mon[33264]: 4.17 scrub starts 2024-09-20T22:49:49.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:49 smithi154 ceph-mon[33264]: 4.17 scrub ok 2024-09-20T22:49:49.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:49 smithi154 ceph-mon[33264]: 5.1a scrub starts 2024-09-20T22:49:49.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:49 smithi154 ceph-mon[33264]: 5.1a scrub ok 2024-09-20T22:49:49.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:49 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/3216014955' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:49:49.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:49 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/733301550' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-09-20T22:49:49.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:49 smithi154 ceph-mon[33264]: 7.c scrub starts 2024-09-20T22:49:49.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:49 smithi154 ceph-mon[33264]: 7.c scrub ok 2024-09-20T22:49:49.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:49 smithi154 ceph-mon[33264]: 6.1a scrub starts 2024-09-20T22:49:49.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:49 smithi154 ceph-mon[33264]: 6.1a scrub ok 2024-09-20T22:49:49.759 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:49.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:49.760 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:49:49.861 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:49:49.882 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:49.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:49.883 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:49:49.983 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:49:50.010 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:50.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:50.011 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:49:50.112 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:49:50.137 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:50.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:50.138 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:49:50.173 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd reweight 5 1 2024-09-20T22:49:50.223 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:49:50.238 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:49:50.263 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:50.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:50.264 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:49:50.286 INFO:tasks.rados.rados.0.smithi084.stdout:1992: done (6 left) 2024-09-20T22:49:50.286 INFO:tasks.rados.rados.0.smithi084.stdout:1993: done (5 left) 2024-09-20T22:49:50.286 INFO:tasks.rados.rados.0.smithi084.stdout:1994: done (4 left) 2024-09-20T22:49:50.286 INFO:tasks.rados.rados.0.smithi084.stdout:1995: done (3 left) 2024-09-20T22:49:50.286 INFO:tasks.rados.rados.0.smithi084.stdout:1996: done (2 left) 2024-09-20T22:49:50.286 INFO:tasks.rados.rados.0.smithi084.stdout:1997: done (1 left) 2024-09-20T22:49:50.286 INFO:tasks.rados.rados.0.smithi084.stdout:1998: done (0 left) 2024-09-20T22:49:50.286 INFO:tasks.rados.rados.0.smithi084.stdout:1999: delete oid 8 current snap is 219 2024-09-20T22:49:50.287 INFO:tasks.rados.rados.0.smithi084.stdout:1999: done (0 left) 2024-09-20T22:49:50.288 INFO:tasks.rados.rados.0.smithi084.stdout:2000: rollback oid 41 current snap is 219 2024-09-20T22:49:50.288 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 41 to 216 2024-09-20T22:49:50.288 INFO:tasks.rados.rados.0.smithi084.stdout:2001: delete oid 46 current snap is 219 2024-09-20T22:49:50.290 INFO:tasks.rados.rados.0.smithi084.stdout:2001: done (1 left) 2024-09-20T22:49:50.290 INFO:tasks.rados.rados.0.smithi084.stdout:2002: read oid 6 snap -1 2024-09-20T22:49:50.290 INFO:tasks.rados.rados.0.smithi084.stdout:2002: expect deleted 2024-09-20T22:49:50.290 INFO:tasks.rados.rados.0.smithi084.stdout:2003: write oid 11 current snap is 219 2024-09-20T22:49:50.290 INFO:tasks.rados.rados.0.smithi084.stdout:2003: seq_num 704 ranges {632128=479772,1746735=600898,3066728=653850} 2024-09-20T22:49:50.295 INFO:tasks.rados.rados.0.smithi084.stdout:2003: writing smithi08477203-11 from 632128 to 1111900 tid 1 2024-09-20T22:49:50.303 INFO:tasks.rados.rados.0.smithi084.stdout:2003: writing smithi08477203-11 from 1746735 to 2347633 tid 2 2024-09-20T22:49:50.310 INFO:tasks.rados.rados.0.smithi084.stdout:2003: writing smithi08477203-11 from 3066728 to 3720578 tid 3 2024-09-20T22:49:50.310 INFO:tasks.rados.rados.0.smithi084.stdout:2004: snap_remove snap 154 2024-09-20T22:49:50.365 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:49:50.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:50.389 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:50.390 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:49:50.489 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:50 smithi084 ceph-mon[29774]: 5.f scrub starts 2024-09-20T22:49:50.489 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:50 smithi084 ceph-mon[29774]: 5.f scrub ok 2024-09-20T22:49:50.489 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:50 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/733301550' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-20T22:49:50.489 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:50 smithi084 ceph-mon[29774]: osdmap e509: 8 total, 8 up, 8 in 2024-09-20T22:49:50.489 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:50 smithi084 ceph-mon[29774]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:49:50.489 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:50 smithi084 ceph-mon[29774]: 1.3 scrub starts 2024-09-20T22:49:50.489 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:50 smithi084 ceph-mon[29774]: 1.3 scrub ok 2024-09-20T22:49:50.490 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:50 smithi084 ceph-mon[40024]: 5.f scrub starts 2024-09-20T22:49:50.490 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:50 smithi084 ceph-mon[40024]: 5.f scrub ok 2024-09-20T22:49:50.490 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:50 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/733301550' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-20T22:49:50.490 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:50 smithi084 ceph-mon[40024]: osdmap e509: 8 total, 8 up, 8 in 2024-09-20T22:49:50.490 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:50 smithi084 ceph-mon[40024]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:49:50.490 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:50 smithi084 ceph-mon[40024]: 1.3 scrub starts 2024-09-20T22:49:50.490 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:50 smithi084 ceph-mon[40024]: 1.3 scrub ok 2024-09-20T22:49:50.490 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:49:50.502 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:49:50.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:50.517 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:50.517 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:49:50.617 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:49:50.644 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:50.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:50.645 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:49:50.724 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-20T22:49:50.724 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:49:50.745 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:49:50.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:50 smithi154 ceph-mon[33264]: 5.f scrub starts 2024-09-20T22:49:50.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:50 smithi154 ceph-mon[33264]: 5.f scrub ok 2024-09-20T22:49:50.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:50 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/733301550' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-20T22:49:50.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:50 smithi154 ceph-mon[33264]: osdmap e509: 8 total, 8 up, 8 in 2024-09-20T22:49:50.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:50 smithi154 ceph-mon[33264]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:49:50.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:50 smithi154 ceph-mon[33264]: 1.3 scrub starts 2024-09-20T22:49:50.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:50 smithi154 ceph-mon[33264]: 1.3 scrub ok 2024-09-20T22:49:50.768 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:50.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:50.769 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:49:50.870 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:49:50.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:50.891 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:50.891 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:49:50.992 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:49:51.018 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:51.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:51.019 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:49:51.120 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:49:51.141 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:51.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:51.142 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:49:51.243 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:49:51.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:51.264 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:51.265 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:49:51.292 INFO:tasks.rados.rados.0.smithi084.stdout:2004: done (3 left) 2024-09-20T22:49:51.292 INFO:tasks.rados.rados.0.smithi084.stdout:2005: snap_create 2024-09-20T22:49:51.292 INFO:tasks.rados.rados.0.smithi084.stdout:2000: finishing rollback tid 0 to smithi08477203-41 2024-09-20T22:49:51.292 INFO:tasks.rados.rados.0.smithi084.stdout:2000: finishing rollback tid 1 to smithi08477203-41 2024-09-20T22:49:51.292 INFO:tasks.rados.rados.0.smithi084.stdout:2000: finishing rollback tid 2 to smithi08477203-41 2024-09-20T22:49:51.292 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 41 v 714 (ObjNum 691 snap 213 seq_num 691) dirty exists 2024-09-20T22:49:51.292 INFO:tasks.rados.rados.0.smithi084.stdout:2003: finishing write tid 1 to smithi08477203-11 2024-09-20T22:49:51.292 INFO:tasks.rados.rados.0.smithi084.stdout:2003: finishing write tid 2 to smithi08477203-11 2024-09-20T22:49:51.292 INFO:tasks.rados.rados.0.smithi084.stdout:2003: finishing write tid 3 to smithi08477203-11 2024-09-20T22:49:51.292 INFO:tasks.rados.rados.0.smithi084.stdout:2003: finishing write tid 4 to smithi08477203-11 2024-09-20T22:49:51.292 INFO:tasks.rados.rados.0.smithi084.stdout:2003: finishing write tid 5 to smithi08477203-11 2024-09-20T22:49:51.293 INFO:tasks.rados.rados.0.smithi084.stdout:2003: oid 11 updating version 0 to 718 2024-09-20T22:49:51.293 INFO:tasks.rados.rados.0.smithi084.stdout:2003: oid 11 version 718 is already newer than 715 2024-09-20T22:49:51.293 INFO:tasks.rados.rados.0.smithi084.stdout:2003: oid 11 version 718 is already newer than 716 2024-09-20T22:49:51.293 INFO:tasks.rados.rados.0.smithi084.stdout:2003: oid 11 version 718 is already newer than 717 2024-09-20T22:49:51.293 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 11 v 718 (ObjNum 704 snap 219 seq_num 704) dirty exists 2024-09-20T22:49:51.293 INFO:tasks.rados.rados.0.smithi084.stdout:2003: left oid 11 (ObjNum 704 snap 219 seq_num 704) 2024-09-20T22:49:51.312 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:51 smithi084 ceph-mon[29774]: pgmap v666: 158 pgs: 4 active+clean+snaptrim, 154 active+clean; 486 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 5.2 MiB/s rd, 3.8 MiB/s wr, 22 op/s 2024-09-20T22:49:51.312 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:51 smithi084 ceph-mon[29774]: 6.1b scrub starts 2024-09-20T22:49:51.313 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:51 smithi084 ceph-mon[29774]: 6.1b scrub ok 2024-09-20T22:49:51.313 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:51 smithi084 ceph-mon[29774]: osdmap e510: 8 total, 8 up, 8 in 2024-09-20T22:49:51.313 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:51 smithi084 ceph-mon[29774]: 4.18 scrub starts 2024-09-20T22:49:51.313 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:51 smithi084 ceph-mon[29774]: 4.18 scrub ok 2024-09-20T22:49:51.313 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:51 smithi084 ceph-mon[40024]: pgmap v666: 158 pgs: 4 active+clean+snaptrim, 154 active+clean; 486 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 5.2 MiB/s rd, 3.8 MiB/s wr, 22 op/s 2024-09-20T22:49:51.313 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:51 smithi084 ceph-mon[40024]: 6.1b scrub starts 2024-09-20T22:49:51.313 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:51 smithi084 ceph-mon[40024]: 6.1b scrub ok 2024-09-20T22:49:51.313 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:51 smithi084 ceph-mon[40024]: osdmap e510: 8 total, 8 up, 8 in 2024-09-20T22:49:51.313 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:51 smithi084 ceph-mon[40024]: 4.18 scrub starts 2024-09-20T22:49:51.313 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:51 smithi084 ceph-mon[40024]: 4.18 scrub ok 2024-09-20T22:49:51.365 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:49:51.386 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:51.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:51.387 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:49:51.488 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:49:51.514 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:51.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:51.515 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:49:51.535 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-20T22:49:51.616 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:49:51.637 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:51.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:51.638 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:49:51.739 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:49:51.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:51 smithi154 ceph-mon[33264]: pgmap v666: 158 pgs: 4 active+clean+snaptrim, 154 active+clean; 486 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 5.2 MiB/s rd, 3.8 MiB/s wr, 22 op/s 2024-09-20T22:49:51.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:51 smithi154 ceph-mon[33264]: 6.1b scrub starts 2024-09-20T22:49:51.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:51 smithi154 ceph-mon[33264]: 6.1b scrub ok 2024-09-20T22:49:51.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:51 smithi154 ceph-mon[33264]: osdmap e510: 8 total, 8 up, 8 in 2024-09-20T22:49:51.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:51 smithi154 ceph-mon[33264]: 4.18 scrub starts 2024-09-20T22:49:51.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:51 smithi154 ceph-mon[33264]: 4.18 scrub ok 2024-09-20T22:49:51.760 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:51.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:51.761 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:49:51.862 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:49:51.864 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:49:51.886 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:51.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:51.887 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:49:51.893 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.2 dump_historic_ops 2024-09-20T22:49:51.988 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:49:52.014 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:52.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:52.015 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:49:52.116 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:49:52.141 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:52.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:52.142 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:49:52.243 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:49:52.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:52.267 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:52.267 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:49:52.301 INFO:tasks.rados.rados.0.smithi084.stdout:2000: done (3 left) 2024-09-20T22:49:52.301 INFO:tasks.rados.rados.0.smithi084.stdout:2002: done (2 left) 2024-09-20T22:49:52.301 INFO:tasks.rados.rados.0.smithi084.stdout:2003: done (1 left) 2024-09-20T22:49:52.301 INFO:tasks.rados.rados.0.smithi084.stdout:2005: done (0 left) 2024-09-20T22:49:52.301 INFO:tasks.rados.rados.0.smithi084.stdout:2006: read oid 39 snap 212 2024-09-20T22:49:52.301 INFO:tasks.rados.rados.0.smithi084.stdout:2006: expect (ObjNum 666 snap 202 seq_num 666) 2024-09-20T22:49:52.301 INFO:tasks.rados.rados.0.smithi084.stdout:2007: read oid 44 snap 212 2024-09-20T22:49:52.301 INFO:tasks.rados.rados.0.smithi084.stdout:2007: expect (ObjNum 672 snap 205 seq_num 672) 2024-09-20T22:49:52.302 INFO:tasks.rados.rados.0.smithi084.stdout:2008: rollback oid 6 current snap is 220 2024-09-20T22:49:52.302 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 6 to 204 2024-09-20T22:49:52.302 INFO:tasks.rados.rados.0.smithi084.stdout:2009: delete oid 28 current snap is 220 2024-09-20T22:49:52.304 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is unset 2024-09-20T22:49:52.308 INFO:tasks.rados.rados.0.smithi084.stdout:2008: finishing rollback tid 1 to smithi08477203-6 2024-09-20T22:49:52.308 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 6 v 740 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-09-20T22:49:52.309 INFO:tasks.rados.rados.0.smithi084.stdout:2006: expect (ObjNum 666 snap 202 seq_num 666) 2024-09-20T22:49:52.368 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:49:52.391 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:52.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:52.392 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:49:52.492 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:49:52.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:52 smithi154 ceph-mon[33264]: 6.a scrub starts 2024-09-20T22:49:52.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:52 smithi154 ceph-mon[33264]: 6.a scrub ok 2024-09-20T22:49:52.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:52 smithi154 ceph-mon[33264]: osdmap e511: 8 total, 8 up, 8 in 2024-09-20T22:49:52.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:52 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/2602761679' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:49:52.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:52 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:49:52.515 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:52.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:52.516 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:49:52.529 INFO:tasks.rados.rados.0.smithi084.stdout:2007: expect (ObjNum 672 snap 205 seq_num 672) 2024-09-20T22:49:52.614 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:52 smithi084 ceph-mon[29774]: 6.a scrub starts 2024-09-20T22:49:52.614 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:52 smithi084 ceph-mon[29774]: 6.a scrub ok 2024-09-20T22:49:52.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:52 smithi084 ceph-mon[29774]: osdmap e511: 8 total, 8 up, 8 in 2024-09-20T22:49:52.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:52 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/2602761679' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:49:52.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:52 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:49:52.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:52 smithi084 ceph-mon[40024]: 6.a scrub starts 2024-09-20T22:49:52.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:52 smithi084 ceph-mon[40024]: 6.a scrub ok 2024-09-20T22:49:52.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:52 smithi084 ceph-mon[40024]: osdmap e511: 8 total, 8 up, 8 in 2024-09-20T22:49:52.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:52 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/2602761679' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:49:52.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:52 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:49:52.617 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:49:52.638 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:52.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:52.639 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:49:52.711 INFO:tasks.rados.rados.0.smithi084.stdout:2006: done (3 left) 2024-09-20T22:49:52.711 INFO:tasks.rados.rados.0.smithi084.stdout:2007: done (2 left) 2024-09-20T22:49:52.711 INFO:tasks.rados.rados.0.smithi084.stdout:2008: done (1 left) 2024-09-20T22:49:52.711 INFO:tasks.rados.rados.0.smithi084.stdout:2009: done (0 left) 2024-09-20T22:49:52.711 INFO:tasks.rados.rados.0.smithi084.stdout:2010: rollback oid 9 current snap is 220 2024-09-20T22:49:52.712 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 9 to 216 2024-09-20T22:49:52.712 INFO:tasks.rados.rados.0.smithi084.stdout:2011: read oid 11 snap -1 2024-09-20T22:49:52.712 INFO:tasks.rados.rados.0.smithi084.stdout:2011: expect (ObjNum 704 snap 219 seq_num 704) 2024-09-20T22:49:52.712 INFO:tasks.rados.rados.0.smithi084.stdout:2012: copy_from oid 32 from oid 21 current snap is 220 2024-09-20T22:49:52.712 INFO:tasks.rados.rados.0.smithi084.stdout:2013: rollback oid 5 current snap is 220 2024-09-20T22:49:52.712 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 5 to 210 2024-09-20T22:49:52.712 INFO:tasks.rados.rados.0.smithi084.stdout:2014: copy_from oid 1 from oid 26 current snap is 220 2024-09-20T22:49:52.712 INFO:tasks.rados.rados.0.smithi084.stdout:2015: copy_from oid 40 from oid 36 current snap is 220 2024-09-20T22:49:52.712 INFO:tasks.rados.rados.0.smithi084.stdout:2016: snap_remove snap 202 2024-09-20T22:49:52.740 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:49:52.761 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:52.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:52.762 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:49:52.864 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:49:52.885 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:52.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:52.886 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:49:52.986 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:49:53.009 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:53.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:53.010 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:49:53.110 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:49:53.129 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:49:53.130 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:49:53.134 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:53.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:53.135 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:49:53.229 INFO:tasks.rados.rados.0.smithi084.stdout:2016: done (6 left) 2024-09-20T22:49:53.229 INFO:tasks.rados.rados.0.smithi084.stdout:2017: delete oid 39 current snap is 220 2024-09-20T22:49:53.229 INFO:tasks.rados.rados.0.smithi084.stdout:2014: finishing copy_from to smithi08477203-1 2024-09-20T22:49:53.229 INFO:tasks.rados.rados.0.smithi084.stdout:2014: got expected ENOENT (src dne) 2024-09-20T22:49:53.229 INFO:tasks.rados.rados.0.smithi084.stdout:2014: finishing copy_from racing read to smithi08477203-1 2024-09-20T22:49:53.229 INFO:tasks.rados.rados.0.smithi084.stdout:2010: finishing rollback tid 1 to smithi08477203-9 2024-09-20T22:49:53.229 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 9 v 0 (ObjNum 129 snap 0 seq_num 2091964000) dirty dne 2024-09-20T22:49:53.229 INFO:tasks.rados.rados.0.smithi084.stdout:2011: expect (ObjNum 704 snap 219 seq_num 704) 2024-09-20T22:49:53.230 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.5 to 1 (10000) 2024-09-20T22:49:53.236 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:49:53.259 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:53.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:53.260 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:49:53.361 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:49:53.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:53 smithi084 ceph-mon[29774]: 5.15 scrub starts 2024-09-20T22:49:53.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:53 smithi084 ceph-mon[29774]: 5.15 scrub ok 2024-09-20T22:49:53.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:53 smithi084 ceph-mon[29774]: pgmap v669: 158 pgs: 4 active+clean+snaptrim, 154 active+clean; 486 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 5.2 MiB/s rd, 3.8 MiB/s wr, 22 op/s 2024-09-20T22:49:53.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:53 smithi084 ceph-mon[29774]: 4.1c scrub starts 2024-09-20T22:49:53.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:53 smithi084 ceph-mon[29774]: 4.1c scrub ok 2024-09-20T22:49:53.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:53 smithi084 ceph-mon[29774]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T22:49:53.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:53 smithi084 ceph-mon[29774]: Cluster is now healthy 2024-09-20T22:49:53.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:53 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T22:49:53.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:53 smithi084 ceph-mon[29774]: osdmap e512: 8 total, 8 up, 8 in 2024-09-20T22:49:53.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:53 smithi084 ceph-mon[29774]: 1.9 scrub starts 2024-09-20T22:49:53.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:53 smithi084 ceph-mon[29774]: 1.9 scrub ok 2024-09-20T22:49:53.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:53 smithi084 ceph-mon[29774]: purged_snaps scrub starts 2024-09-20T22:49:53.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:53 smithi084 ceph-mon[29774]: purged_snaps scrub ok 2024-09-20T22:49:53.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:53 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/2286143407' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-20T22:49:53.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:53 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-20T22:49:53.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:53 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:49:53.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:53 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:49:53.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:53 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-20T22:49:53.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:53 smithi084 ceph-mon[29774]: osdmap e513: 8 total, 8 up, 8 in 2024-09-20T22:49:53.367 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:53 smithi084 ceph-mon[40024]: 5.15 scrub starts 2024-09-20T22:49:53.367 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:53 smithi084 ceph-mon[40024]: 5.15 scrub ok 2024-09-20T22:49:53.367 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:53 smithi084 ceph-mon[40024]: pgmap v669: 158 pgs: 4 active+clean+snaptrim, 154 active+clean; 486 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 5.2 MiB/s rd, 3.8 MiB/s wr, 22 op/s 2024-09-20T22:49:53.367 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:53 smithi084 ceph-mon[40024]: 4.1c scrub starts 2024-09-20T22:49:53.367 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:53 smithi084 ceph-mon[40024]: 4.1c scrub ok 2024-09-20T22:49:53.367 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:53 smithi084 ceph-mon[40024]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T22:49:53.367 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:53 smithi084 ceph-mon[40024]: Cluster is now healthy 2024-09-20T22:49:53.367 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:53 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T22:49:53.367 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:53 smithi084 ceph-mon[40024]: osdmap e512: 8 total, 8 up, 8 in 2024-09-20T22:49:53.367 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:53 smithi084 ceph-mon[40024]: 1.9 scrub starts 2024-09-20T22:49:53.368 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:53 smithi084 ceph-mon[40024]: 1.9 scrub ok 2024-09-20T22:49:53.368 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:53 smithi084 ceph-mon[40024]: purged_snaps scrub starts 2024-09-20T22:49:53.368 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:53 smithi084 ceph-mon[40024]: purged_snaps scrub ok 2024-09-20T22:49:53.368 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:53 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/2286143407' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-20T22:49:53.368 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:53 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-20T22:49:53.368 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:53 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:49:53.368 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:53 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:49:53.368 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:53 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-20T22:49:53.368 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:53 smithi084 ceph-mon[40024]: osdmap e513: 8 total, 8 up, 8 in 2024-09-20T22:49:53.383 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:53.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:53.385 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:49:53.415 INFO:tasks.rados.rados.0.smithi084.stdout:2013: finishing rollback tid 1 to smithi08477203-5 2024-09-20T22:49:53.415 INFO:tasks.rados.rados.0.smithi084.stdout:2013: finishing rollback tid 2 to smithi08477203-5 2024-09-20T22:49:53.415 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 5 v 720 (ObjNum 670 snap 205 seq_num 670) dirty exists 2024-09-20T22:49:53.415 INFO:tasks.rados.rados.0.smithi084.stdout:2012: finishing copy_from racing read to smithi08477203-32 2024-09-20T22:49:53.415 INFO:tasks.rados.rados.0.smithi084.stdout:2012: finishing copy_from to smithi08477203-32 2024-09-20T22:49:53.415 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 32 v 725 (ObjNum 636 snap 193 seq_num 636) dirty exists 2024-09-20T22:49:53.415 INFO:tasks.rados.rados.0.smithi084.stdout:2015: finishing copy_from racing read to smithi08477203-40 2024-09-20T22:49:53.415 INFO:tasks.rados.rados.0.smithi084.stdout:2015: finishing copy_from to smithi08477203-40 2024-09-20T22:49:53.416 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 40 v 402 (ObjNum 440 snap 115 seq_num 440) dirty exists 2024-09-20T22:49:53.416 INFO:tasks.rados.rados.0.smithi084.stdout:2010: done (6 left) 2024-09-20T22:49:53.416 INFO:tasks.rados.rados.0.smithi084.stdout:2011: done (5 left) 2024-09-20T22:49:53.416 INFO:tasks.rados.rados.0.smithi084.stdout:2012: done (4 left) 2024-09-20T22:49:53.416 INFO:tasks.rados.rados.0.smithi084.stdout:2013: done (3 left) 2024-09-20T22:49:53.416 INFO:tasks.rados.rados.0.smithi084.stdout:2014: done (2 left) 2024-09-20T22:49:53.416 INFO:tasks.rados.rados.0.smithi084.stdout:2015: done (1 left) 2024-09-20T22:49:53.416 INFO:tasks.rados.rados.0.smithi084.stdout:2017: done (0 left) 2024-09-20T22:49:53.416 INFO:tasks.rados.rados.0.smithi084.stdout:2018: snap_remove snap 207 2024-09-20T22:49:53.486 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:49:53.509 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:53.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:53.510 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:49:53.611 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:49:53.633 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:53.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:53.634 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:49:53.736 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:49:53.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:53 smithi154 ceph-mon[33264]: 5.15 scrub starts 2024-09-20T22:49:53.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:53 smithi154 ceph-mon[33264]: 5.15 scrub ok 2024-09-20T22:49:53.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:53 smithi154 ceph-mon[33264]: pgmap v669: 158 pgs: 4 active+clean+snaptrim, 154 active+clean; 486 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 5.2 MiB/s rd, 3.8 MiB/s wr, 22 op/s 2024-09-20T22:49:53.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:53 smithi154 ceph-mon[33264]: 4.1c scrub starts 2024-09-20T22:49:53.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:53 smithi154 ceph-mon[33264]: 4.1c scrub ok 2024-09-20T22:49:53.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:53 smithi154 ceph-mon[33264]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T22:49:53.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:53 smithi154 ceph-mon[33264]: Cluster is now healthy 2024-09-20T22:49:53.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:53 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T22:49:53.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:53 smithi154 ceph-mon[33264]: osdmap e512: 8 total, 8 up, 8 in 2024-09-20T22:49:53.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:53 smithi154 ceph-mon[33264]: 1.9 scrub starts 2024-09-20T22:49:53.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:53 smithi154 ceph-mon[33264]: 1.9 scrub ok 2024-09-20T22:49:53.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:53 smithi154 ceph-mon[33264]: purged_snaps scrub starts 2024-09-20T22:49:53.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:53 smithi154 ceph-mon[33264]: purged_snaps scrub ok 2024-09-20T22:49:53.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:53 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/2286143407' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-20T22:49:53.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:53 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-20T22:49:53.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:53 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:49:53.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:53 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:49:53.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:53 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-20T22:49:53.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:53 smithi154 ceph-mon[33264]: osdmap e513: 8 total, 8 up, 8 in 2024-09-20T22:49:53.756 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:53.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:53.757 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:49:53.858 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:49:53.882 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:53.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:53.883 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:49:53.886 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:49:53.887 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:49:53.903 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:49:53.904 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:49:53.919 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:49:53.920 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:49:53.936 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-20T22:49:53.936 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T22:49:53.952 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:49:53.953 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:49:53.970 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:49:53.970 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:49:53.984 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:49:53.987 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:49:53.987 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:49:54.004 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:49:54.004 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:49:54.007 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:54.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:54.008 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:49:54.109 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:49:54.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:54.133 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:54.133 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:49:54.236 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:49:54.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:54.259 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:54.259 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:49:54.306 INFO:tasks.rados.rados.0.smithi084.stdout:2018: done (0 left) 2024-09-20T22:49:54.307 INFO:tasks.rados.rados.0.smithi084.stdout:2019: delete oid 19 current snap is 220 2024-09-20T22:49:54.313 INFO:tasks.rados.rados.0.smithi084.stdout:2019: done (0 left) 2024-09-20T22:49:54.313 INFO:tasks.rados.rados.0.smithi084.stdout:2020: write oid 20 current snap is 220 2024-09-20T22:49:54.313 INFO:tasks.rados.rados.0.smithi084.stdout:2020: seq_num 708 ranges {699356=497121,1980663=608236,3176420=415291} 2024-09-20T22:49:54.319 INFO:tasks.rados.rados.0.smithi084.stdout:2020: writing smithi08477203-20 from 699356 to 1196477 tid 1 2024-09-20T22:49:54.327 INFO:tasks.rados.rados.0.smithi084.stdout:2020: writing smithi08477203-20 from 1980663 to 2588899 tid 2 2024-09-20T22:49:54.333 INFO:tasks.rados.rados.0.smithi084.stdout:2020: writing smithi08477203-20 from 3176420 to 3591711 tid 3 2024-09-20T22:49:54.333 INFO:tasks.rados.rados.0.smithi084.stdout:2021: read oid 42 snap -1 2024-09-20T22:49:54.333 INFO:tasks.rados.rados.0.smithi084.stdout:2021: expect (ObjNum 656 snap 198 seq_num 656) 2024-09-20T22:49:54.333 INFO:tasks.rados.rados.0.smithi084.stdout:2022: rollback oid 4 current snap is 220 2024-09-20T22:49:54.333 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 4 to 217 2024-09-20T22:49:54.333 INFO:tasks.rados.rados.0.smithi084.stdout:2020: finishing write tid 1 to smithi08477203-20 2024-09-20T22:49:54.334 INFO:tasks.rados.rados.0.smithi084.stdout:2023: rollback oid 10 current snap is 220 2024-09-20T22:49:54.334 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 10 to 210 2024-09-20T22:49:54.334 INFO:tasks.rados.rados.0.smithi084.stdout:2020: finishing write tid 2 to smithi08477203-20 2024-09-20T22:49:54.334 INFO:tasks.rados.rados.0.smithi084.stdout:2024: read oid 49 snap -1 2024-09-20T22:49:54.334 INFO:tasks.rados.rados.0.smithi084.stdout:2024: expect (ObjNum 470 snap 130 seq_num 470) 2024-09-20T22:49:54.334 INFO:tasks.rados.rados.0.smithi084.stdout:2025: copy_from oid 30 from oid 9 current snap is 220 2024-09-20T22:49:54.334 INFO:tasks.rados.rados.0.smithi084.stdout:2026: delete oid 19 current snap is 220 2024-09-20T22:49:54.336 INFO:tasks.rados.rados.0.smithi084.stdout:2025: finishing copy_from to smithi08477203-30 2024-09-20T22:49:54.336 INFO:tasks.rados.rados.0.smithi084.stdout:2025: got expected ENOENT (src dne) 2024-09-20T22:49:54.337 INFO:tasks.rados.rados.0.smithi084.stdout:2025: finishing copy_from racing read to smithi08477203-30 2024-09-20T22:49:54.338 INFO:tasks.rados.rados.0.smithi084.stdout:2020: finishing write tid 3 to smithi08477203-20 2024-09-20T22:49:54.338 INFO:tasks.rados.rados.0.smithi084.stdout:2020: finishing write tid 4 to smithi08477203-20 2024-09-20T22:49:54.338 INFO:tasks.rados.rados.0.smithi084.stdout:2020: finishing write tid 5 to smithi08477203-20 2024-09-20T22:49:54.338 INFO:tasks.rados.rados.0.smithi084.stdout:2020: oid 20 updating version 0 to 319 2024-09-20T22:49:54.338 INFO:tasks.rados.rados.0.smithi084.stdout:2020: oid 20 updating version 319 to 321 2024-09-20T22:49:54.339 INFO:tasks.rados.rados.0.smithi084.stdout:2020: oid 20 version 321 is already newer than 320 2024-09-20T22:49:54.339 INFO:tasks.rados.rados.0.smithi084.stdout:2020: oid 20 version 321 is already newer than 318 2024-09-20T22:49:54.339 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 20 v 321 (ObjNum 708 snap 220 seq_num 708) dirty exists 2024-09-20T22:49:54.339 INFO:tasks.rados.rados.0.smithi084.stdout:2020: left oid 20 (ObjNum 708 snap 220 seq_num 708) 2024-09-20T22:49:54.339 INFO:tasks.rados.rados.0.smithi084.stdout:2023: finishing rollback tid 0 to smithi08477203-10 2024-09-20T22:49:54.339 INFO:tasks.rados.rados.0.smithi084.stdout:2023: finishing rollback tid 1 to smithi08477203-10 2024-09-20T22:49:54.339 INFO:tasks.rados.rados.0.smithi084.stdout:2023: finishing rollback tid 2 to smithi08477203-10 2024-09-20T22:49:54.340 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 10 v 1215 (ObjNum 575 snap 172 seq_num 575) dirty exists 2024-09-20T22:49:54.343 INFO:tasks.rados.rados.0.smithi084.stdout:2021: expect (ObjNum 656 snap 198 seq_num 656) 2024-09-20T22:49:54.360 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:49:54.382 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:54.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:54.383 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:49:54.484 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:49:54.486 INFO:tasks.rados.rados.0.smithi084.stdout:2022: finishing rollback tid 1 to smithi08477203-4 2024-09-20T22:49:54.486 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 4 v 1177 (ObjNum 289 snap 0 seq_num 0) dirty dne 2024-09-20T22:49:54.486 INFO:tasks.rados.rados.0.smithi084.stdout:2024: expect (ObjNum 470 snap 130 seq_num 470) 2024-09-20T22:49:54.508 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:54.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:54.509 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:49:54.610 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:49:54.611 INFO:tasks.rados.rados.0.smithi084.stdout:2020: done (6 left) 2024-09-20T22:49:54.613 INFO:tasks.rados.rados.0.smithi084.stdout:2021: done (5 left) 2024-09-20T22:49:54.613 INFO:tasks.rados.rados.0.smithi084.stdout:2022: done (4 left) 2024-09-20T22:49:54.613 INFO:tasks.rados.rados.0.smithi084.stdout:2023: done (3 left) 2024-09-20T22:49:54.614 INFO:tasks.rados.rados.0.smithi084.stdout:2024: done (2 left) 2024-09-20T22:49:54.614 INFO:tasks.rados.rados.0.smithi084.stdout:2025: done (1 left) 2024-09-20T22:49:54.614 INFO:tasks.rados.rados.0.smithi084.stdout:2026: done (0 left) 2024-09-20T22:49:54.614 INFO:tasks.rados.rados.0.smithi084.stdout:2027: write oid 44 current snap is 220 2024-09-20T22:49:54.614 INFO:tasks.rados.rados.0.smithi084.stdout:2027: seq_num 710 ranges {732970=647619,2097627=611905,3231266=437835} 2024-09-20T22:49:54.614 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:54 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/3325022668' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:49:54.614 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:54 smithi084 ceph-mon[40024]: 6.e scrub starts 2024-09-20T22:49:54.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:54 smithi084 ceph-mon[40024]: 6.e scrub ok 2024-09-20T22:49:54.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:54 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/3325022668' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:49:54.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:54 smithi084 ceph-mon[29774]: 6.e scrub starts 2024-09-20T22:49:54.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:54 smithi084 ceph-mon[29774]: 6.e scrub ok 2024-09-20T22:49:54.617 INFO:tasks.rados.rados.0.smithi084.stdout:2027: writing smithi08477203-44 from 732970 to 1380589 tid 1 2024-09-20T22:49:54.625 INFO:tasks.rados.rados.0.smithi084.stdout:2027: writing smithi08477203-44 from 2097627 to 2709532 tid 2 2024-09-20T22:49:54.630 INFO:tasks.rados.rados.0.smithi084.stdout:2027: writing smithi08477203-44 from 3231266 to 3669101 tid 3 2024-09-20T22:49:54.631 INFO:tasks.rados.rados.0.smithi084.stdout:2028: copy_from oid 20 from oid 10 current snap is 220 2024-09-20T22:49:54.631 INFO:tasks.rados.rados.0.smithi084.stdout:2027: finishing write tid 1 to smithi08477203-44 2024-09-20T22:49:54.631 INFO:tasks.rados.rados.0.smithi084.stdout:2027: finishing write tid 2 to smithi08477203-44 2024-09-20T22:49:54.631 INFO:tasks.rados.rados.0.smithi084.stdout:2029: write oid 24 current snap is 220 2024-09-20T22:49:54.631 INFO:tasks.rados.rados.0.smithi084.stdout:2029: seq_num 711 ranges {749777=722868,2156109=471916,3258689=649107} 2024-09-20T22:49:54.636 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:54.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:54.637 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:49:54.639 INFO:tasks.rados.rados.0.smithi084.stdout:2029: writing smithi08477203-24 from 749777 to 1472645 tid 1 2024-09-20T22:49:54.645 INFO:tasks.rados.rados.0.smithi084.stdout:2029: writing smithi08477203-24 from 2156109 to 2628025 tid 2 2024-09-20T22:49:54.653 INFO:tasks.rados.rados.0.smithi084.stdout:2029: writing smithi08477203-24 from 3258689 to 3907796 tid 3 2024-09-20T22:49:54.653 INFO:tasks.rados.rados.0.smithi084.stdout:2030: snap_create 2024-09-20T22:49:54.653 INFO:tasks.rados.rados.0.smithi084.stdout:2027: finishing write tid 3 to smithi08477203-44 2024-09-20T22:49:54.653 INFO:tasks.rados.rados.0.smithi084.stdout:2027: finishing write tid 4 to smithi08477203-44 2024-09-20T22:49:54.653 INFO:tasks.rados.rados.0.smithi084.stdout:2027: finishing write tid 5 to smithi08477203-44 2024-09-20T22:49:54.653 INFO:tasks.rados.rados.0.smithi084.stdout:2027: oid 44 updating version 0 to 1359 2024-09-20T22:49:54.653 INFO:tasks.rados.rados.0.smithi084.stdout:2027: oid 44 updating version 1359 to 1360 2024-09-20T22:49:54.653 INFO:tasks.rados.rados.0.smithi084.stdout:2027: oid 44 updating version 1360 to 1362 2024-09-20T22:49:54.653 INFO:tasks.rados.rados.0.smithi084.stdout:2027: oid 44 version 1362 is already newer than 1361 2024-09-20T22:49:54.654 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 44 v 1362 (ObjNum 710 snap 220 seq_num 710) dirty exists 2024-09-20T22:49:54.654 INFO:tasks.rados.rados.0.smithi084.stdout:2027: left oid 44 (ObjNum 710 snap 220 seq_num 710) 2024-09-20T22:49:54.654 INFO:tasks.rados.rados.0.smithi084.stdout:2028: finishing copy_from racing read to smithi08477203-20 2024-09-20T22:49:54.654 INFO:tasks.rados.rados.0.smithi084.stdout:2028: finishing copy_from to smithi08477203-20 2024-09-20T22:49:54.654 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 20 v 1215 (ObjNum 575 snap 172 seq_num 575) dirty exists 2024-09-20T22:49:54.654 INFO:tasks.rados.rados.0.smithi084.stdout:2029: finishing write tid 1 to smithi08477203-24 2024-09-20T22:49:54.654 INFO:tasks.rados.rados.0.smithi084.stdout:2029: finishing write tid 2 to smithi08477203-24 2024-09-20T22:49:54.656 INFO:tasks.rados.rados.0.smithi084.stdout:2029: finishing write tid 3 to smithi08477203-24 2024-09-20T22:49:54.656 INFO:tasks.rados.rados.0.smithi084.stdout:2029: finishing write tid 4 to smithi08477203-24 2024-09-20T22:49:54.656 INFO:tasks.rados.rados.0.smithi084.stdout:2029: finishing write tid 5 to smithi08477203-24 2024-09-20T22:49:54.656 INFO:tasks.rados.rados.0.smithi084.stdout:2029: oid 24 updating version 0 to 431 2024-09-20T22:49:54.656 INFO:tasks.rados.rados.0.smithi084.stdout:2029: oid 24 version 431 is already newer than 430 2024-09-20T22:49:54.657 INFO:tasks.rados.rados.0.smithi084.stdout:2029: oid 24 version 431 is already newer than 429 2024-09-20T22:49:54.657 INFO:tasks.rados.rados.0.smithi084.stdout:2029: oid 24 updating version 431 to 432 2024-09-20T22:49:54.657 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 24 v 432 (ObjNum 711 snap 220 seq_num 711) dirty exists 2024-09-20T22:49:54.657 INFO:tasks.rados.rados.0.smithi084.stdout:2029: left oid 24 (ObjNum 711 snap 220 seq_num 711) 2024-09-20T22:49:54.738 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:49:54.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:54 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/3325022668' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:49:54.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:54 smithi154 ceph-mon[33264]: 6.e scrub starts 2024-09-20T22:49:54.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:54 smithi154 ceph-mon[33264]: 6.e scrub ok 2024-09-20T22:49:54.760 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:54.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:54.761 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:49:54.862 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:49:54.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:54.883 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:54.883 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:49:54.984 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:49:55.005 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:55.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:55.006 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:49:55.087 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd set noscrub 2024-09-20T22:49:55.107 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:49:55.150 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:55.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:55.151 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:49:55.252 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:49:55.278 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:55.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:55.279 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:49:55.310 INFO:tasks.rados.rados.0.smithi084.stdout:2027: done (3 left) 2024-09-20T22:49:55.310 INFO:tasks.rados.rados.0.smithi084.stdout:2028: done (2 left) 2024-09-20T22:49:55.310 INFO:tasks.rados.rados.0.smithi084.stdout:2029: done (1 left) 2024-09-20T22:49:55.310 INFO:tasks.rados.rados.0.smithi084.stdout:2030: done (0 left) 2024-09-20T22:49:55.310 INFO:tasks.rados.rados.0.smithi084.stdout:2031: rollback oid 17 current snap is 221 2024-09-20T22:49:55.310 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 17 to 205 2024-09-20T22:49:55.310 INFO:tasks.rados.rados.0.smithi084.stdout:2032: rollback oid 32 current snap is 221 2024-09-20T22:49:55.310 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 32 to 212 2024-09-20T22:49:55.311 INFO:tasks.rados.rados.0.smithi084.stdout:2033: snap_remove snap 197 2024-09-20T22:49:55.364 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:55 smithi084 ceph-mon[29774]: pgmap v672: 158 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 153 active+clean; 497 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 17 MiB/s rd, 2.3 MiB/s wr, 19 op/s 2024-09-20T22:49:55.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:55 smithi084 ceph-mon[29774]: 3.1b deep-scrub starts 2024-09-20T22:49:55.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:55 smithi084 ceph-mon[29774]: 3.1b deep-scrub ok 2024-09-20T22:49:55.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:55 smithi084 ceph-mon[29774]: 5.8 scrub starts 2024-09-20T22:49:55.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:55 smithi084 ceph-mon[29774]: 5.8 scrub ok 2024-09-20T22:49:55.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:55 smithi084 ceph-mon[29774]: osdmap e514: 8 total, 8 up, 8 in 2024-09-20T22:49:55.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:55 smithi084 ceph-mon[29774]: 3.13 scrub starts 2024-09-20T22:49:55.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:55 smithi084 ceph-mon[29774]: 3.13 scrub ok 2024-09-20T22:49:55.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:55 smithi084 ceph-mon[40024]: pgmap v672: 158 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 153 active+clean; 497 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 17 MiB/s rd, 2.3 MiB/s wr, 19 op/s 2024-09-20T22:49:55.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:55 smithi084 ceph-mon[40024]: 3.1b deep-scrub starts 2024-09-20T22:49:55.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:55 smithi084 ceph-mon[40024]: 3.1b deep-scrub ok 2024-09-20T22:49:55.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:55 smithi084 ceph-mon[40024]: 5.8 scrub starts 2024-09-20T22:49:55.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:55 smithi084 ceph-mon[40024]: 5.8 scrub ok 2024-09-20T22:49:55.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:55 smithi084 ceph-mon[40024]: osdmap e514: 8 total, 8 up, 8 in 2024-09-20T22:49:55.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:55 smithi084 ceph-mon[40024]: 3.13 scrub starts 2024-09-20T22:49:55.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:55 smithi084 ceph-mon[40024]: 3.13 scrub ok 2024-09-20T22:49:55.375 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd reweight 6 1 2024-09-20T22:49:55.380 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:49:55.439 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:55.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:55.440 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:49:55.541 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:49:55.567 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:55.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:55.568 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:49:55.667 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-20T22:49:55.667 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2024-09-20T22:49:55.667 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2024-09-20T22:49:55.667 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [ 2024-09-20T22:49:55.667 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.667 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:901 7.e 7:7c62ced4:::smithi08477203-17:head [copy-from ver 48] snapc 1e=[] ondisk+write+known_if_redirected+supports_pool_eio e156)", 2024-09-20T22:49:55.667 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:43:55.624762+0000", 2024-09-20T22:49:55.667 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 360.03969879099998, 2024-09-20T22:49:55.667 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.021835856000000001, 2024-09-20T22:49:55.668 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:49:55.668 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:49:55.668 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:49:55.668 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:49:55.668 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:49:55.668 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:49:55.668 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 901 2024-09-20T22:49:55.668 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.668 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:49:55.668 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.668 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:49:55.668 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.624762+0000", 2024-09-20T22:49:55.668 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:49:55.669 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.669 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.669 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:49:55.669 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.624762+0000", 2024-09-20T22:49:55.669 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:49:55.669 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.669 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.669 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:49:55.669 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.624765+0000", 2024-09-20T22:49:55.669 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.57e-06 2024-09-20T22:49:55.669 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.669 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.669 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:49:55.669 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.624776+0000", 2024-09-20T22:49:55.669 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.078e-05 2024-09-20T22:49:55.670 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.670 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.670 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:49:55.670 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.624803+0000", 2024-09-20T22:49:55.670 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.6769999999999999e-05 2024-09-20T22:49:55.670 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.670 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.670 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:49:55.670 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.624815+0000", 2024-09-20T22:49:55.670 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2459e-05 2024-09-20T22:49:55.670 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.670 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.670 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:49:55.670 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.624936+0000", 2024-09-20T22:49:55.671 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000120467 2024-09-20T22:49:55.671 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.671 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.671 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:49:55.671 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.625046+0000", 2024-09-20T22:49:55.671 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000110222 2024-09-20T22:49:55.671 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.671 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.671 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 0,5", 2024-09-20T22:49:55.671 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.635547+0000", 2024-09-20T22:49:55.671 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010501765999999999 2024-09-20T22:49:55.671 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.671 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.671 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:49:55.672 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.644332+0000", 2024-09-20T22:49:55.672 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0087847729999999992 2024-09-20T22:49:55.672 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.672 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.672 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:49:55.672 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.646168+0000", 2024-09-20T22:49:55.672 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0018360010000000001 2024-09-20T22:49:55.672 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.672 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.672 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:49:55.672 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.646512+0000", 2024-09-20T22:49:55.672 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000343269 2024-09-20T22:49:55.672 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.673 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.673 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:49:55.673 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.646550+0000", 2024-09-20T22:49:55.673 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.8383000000000002e-05 2024-09-20T22:49:55.673 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.673 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.673 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:49:55.673 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.646598+0000", 2024-09-20T22:49:55.673 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.8396000000000002e-05 2024-09-20T22:49:55.673 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:49:55.673 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:49:55.673 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:49:55.674 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.674 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.674 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:902 7.e 7:7c62ced4:::smithi08477203-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e156)", 2024-09-20T22:49:55.674 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:43:55.624853+0000", 2024-09-20T22:49:55.674 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 360.03960871300001, 2024-09-20T22:49:55.674 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.015323603999999999, 2024-09-20T22:49:55.674 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:49:55.674 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:49:55.674 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-20T22:49:55.674 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:49:55.674 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:49:55.674 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:49:55.674 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 902 2024-09-20T22:49:55.674 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.675 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:49:55.675 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.675 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:49:55.675 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.624853+0000", 2024-09-20T22:49:55.675 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:49:55.675 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.675 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.675 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:49:55.675 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.624853+0000", 2024-09-20T22:49:55.675 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:49:55.675 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.675 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.675 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:49:55.675 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.624853+0000", 2024-09-20T22:49:55.676 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.1399999999999997e-07 2024-09-20T22:49:55.676 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.676 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.676 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:49:55.676 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.624859+0000", 2024-09-20T22:49:55.676 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.9610000000000002e-06 2024-09-20T22:49:55.676 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.676 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.676 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:49:55.676 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.624887+0000", 2024-09-20T22:49:55.676 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.7597e-05 2024-09-20T22:49:55.676 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.676 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.676 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:49:55.677 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.624894+0000", 2024-09-20T22:49:55.677 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.469e-06 2024-09-20T22:49:55.677 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.677 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.677 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:49:55.677 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.625219+0000", 2024-09-20T22:49:55.677 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00032538499999999998 2024-09-20T22:49:55.677 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.677 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.677 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for blocked object", 2024-09-20T22:49:55.677 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.625258+0000", 2024-09-20T22:49:55.677 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.8467000000000003e-05 2024-09-20T22:49:55.677 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.677 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.677 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:49:55.678 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.640050+0000", 2024-09-20T22:49:55.678 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0147923 2024-09-20T22:49:55.678 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.678 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.678 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:49:55.678 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.640112+0000", 2024-09-20T22:49:55.678 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.1723000000000005e-05 2024-09-20T22:49:55.678 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.678 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.678 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:49:55.678 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.640176+0000", 2024-09-20T22:49:55.678 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.4188000000000003e-05 2024-09-20T22:49:55.678 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:49:55.678 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:49:55.679 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:49:55.679 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.679 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.679 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24652.0:1420 7.19 e189/139 7:9cc0b7e2:::smithi08477203-46:head v 189'99, mlcod=174'95)", 2024-09-20T22:49:55.679 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:44:28.150258+0000", 2024-09-20T22:49:55.679 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 327.514203017, 2024-09-20T22:49:55.679 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014180626, 2024-09-20T22:49:55.679 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:49:55.679 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:49:55.679 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:49:55.679 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:49:55.679 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.679 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:49:55.679 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.150258+0000", 2024-09-20T22:49:55.680 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:49:55.680 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.680 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.680 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:49:55.680 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.150258+0000", 2024-09-20T22:49:55.680 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:49:55.680 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.680 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.680 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:49:55.680 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.150259+0000", 2024-09-20T22:49:55.680 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.99e-07 2024-09-20T22:49:55.680 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.680 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.680 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:49:55.681 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.150674+0000", 2024-09-20T22:49:55.681 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00041528000000000002 2024-09-20T22:49:55.681 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.681 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.681 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:49:55.681 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.150688+0000", 2024-09-20T22:49:55.681 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3399e-05 2024-09-20T22:49:55.681 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:49:55.683 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.683 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.683 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:49:55.683 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.150699+0000", 2024-09-20T22:49:55.683 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1022999999999999e-05 2024-09-20T22:49:55.683 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.683 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.683 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:49:55.683 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.150772+0000", 2024-09-20T22:49:55.683 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.3455999999999996e-05 2024-09-20T22:49:55.683 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.684 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.684 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:49:55.684 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.150798+0000", 2024-09-20T22:49:55.684 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.5777000000000001e-05 2024-09-20T22:49:55.684 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.684 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.684 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:49:55.684 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.164390+0000", 2024-09-20T22:49:55.684 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013591789999999999 2024-09-20T22:49:55.684 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.684 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.684 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:49:55.684 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.164439+0000", 2024-09-20T22:49:55.684 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.9202000000000001e-05 2024-09-20T22:49:55.685 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:49:55.685 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:49:55.685 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:49:55.685 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.685 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.685 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24652.0:1421 7.19 e189/139 7:9cc0b7e2:::smithi08477203-46:head v 189'100, mlcod=174'95)", 2024-09-20T22:49:55.685 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:44:28.156805+0000", 2024-09-20T22:49:55.685 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 327.50765615900002, 2024-09-20T22:49:55.685 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013180661999999999, 2024-09-20T22:49:55.685 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:49:55.685 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:49:55.685 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:49:55.685 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:49:55.685 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.686 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:49:55.686 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.156805+0000", 2024-09-20T22:49:55.686 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:49:55.686 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.686 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.686 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:49:55.686 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.156805+0000", 2024-09-20T22:49:55.686 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:49:55.686 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.686 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.686 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:49:55.686 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.156806+0000", 2024-09-20T22:49:55.686 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.074e-06 2024-09-20T22:49:55.686 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.687 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.687 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:49:55.687 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.157126+0000", 2024-09-20T22:49:55.687 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00031991399999999997 2024-09-20T22:49:55.687 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.687 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.687 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:49:55.687 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.157136+0000", 2024-09-20T22:49:55.687 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.4499999999999993e-06 2024-09-20T22:49:55.687 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.687 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.687 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:49:55.687 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.157143+0000", 2024-09-20T22:49:55.687 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.7649999999999993e-06 2024-09-20T22:49:55.688 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.688 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.688 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:49:55.688 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.157188+0000", 2024-09-20T22:49:55.688 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.4885000000000001e-05 2024-09-20T22:49:55.688 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.688 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.688 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:49:55.688 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.157208+0000", 2024-09-20T22:49:55.688 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.0015000000000001e-05 2024-09-20T22:49:55.688 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.688 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.688 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:49:55.688 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.169936+0000", 2024-09-20T22:49:55.689 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.012728204 2024-09-20T22:49:55.689 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.689 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.689 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:49:55.689 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.169986+0000", 2024-09-20T22:49:55.689 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.9354999999999998e-05 2024-09-20T22:49:55.689 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:49:55.689 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:49:55.689 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:49:55.689 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.689 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.689 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24652.0:1654 7.19 e206/139 7:9970b578:::smithi08477203-15:head v 206'114, mlcod=205'111)", 2024-09-20T22:49:55.689 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:44:45.167994+0000", 2024-09-20T22:49:55.689 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 310.49646760899998, 2024-09-20T22:49:55.690 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.01345262, 2024-09-20T22:49:55.690 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:49:55.690 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:49:55.690 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:49:55.690 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:49:55.690 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.690 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:49:55.690 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:45.167994+0000", 2024-09-20T22:49:55.690 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:49:55.690 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.690 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.691 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:49:55.691 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:45.167994+0000", 2024-09-20T22:49:55.691 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:49:55.691 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.691 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.692 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:49:55.692 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:45.167996+0000", 2024-09-20T22:49:55.692 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.5100000000000001e-06 2024-09-20T22:49:55.692 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.692 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.692 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:49:55.692 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:45.175169+0000", 2024-09-20T22:49:55.692 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0071732740000000003 2024-09-20T22:49:55.692 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.692 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.692 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:49:55.692 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:45.175208+0000", 2024-09-20T22:49:55.692 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.8597000000000003e-05 2024-09-20T22:49:55.692 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.692 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.693 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:49:55.693 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:45.175223+0000", 2024-09-20T22:49:55.693 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.4627000000000001e-05 2024-09-20T22:49:55.693 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.693 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.693 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:49:55.693 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:45.175304+0000", 2024-09-20T22:49:55.693 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.1111000000000003e-05 2024-09-20T22:49:55.693 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.693 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.693 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:49:55.693 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:45.175336+0000", 2024-09-20T22:49:55.693 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.1945e-05 2024-09-20T22:49:55.693 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.694 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.694 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:49:55.694 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:45.181392+0000", 2024-09-20T22:49:55.694 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0060566630000000003 2024-09-20T22:49:55.694 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.694 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.694 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:49:55.694 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:45.181446+0000", 2024-09-20T22:49:55.694 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.3893000000000001e-05 2024-09-20T22:49:55.694 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:49:55.694 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:49:55.694 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:49:55.694 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.695 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.695 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24652.0:1799 7.17 e229/139 7:ea720b85:::smithi08477203-26:head v 229'168, mlcod=222'164)", 2024-09-20T22:49:55.695 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:45:08.171867+0000", 2024-09-20T22:49:55.695 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 287.49259439100001, 2024-09-20T22:49:55.695 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014913454, 2024-09-20T22:49:55.695 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:49:55.695 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:49:55.695 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:49:55.695 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:49:55.695 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.695 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:49:55.695 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.171867+0000", 2024-09-20T22:49:55.695 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:49:55.695 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.696 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.696 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:49:55.696 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.171867+0000", 2024-09-20T22:49:55.696 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:49:55.696 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.696 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.696 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:49:55.696 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.171867+0000", 2024-09-20T22:49:55.696 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.3799999999999999e-07 2024-09-20T22:49:55.696 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.696 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.696 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:49:55.697 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.173016+0000", 2024-09-20T22:49:55.697 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0011485740000000001 2024-09-20T22:49:55.697 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.697 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.697 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:49:55.697 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.173024+0000", 2024-09-20T22:49:55.697 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.4780000000000008e-06 2024-09-20T22:49:55.697 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.697 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.697 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:49:55.697 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.173034+0000", 2024-09-20T22:49:55.697 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.7860000000000008e-06 2024-09-20T22:49:55.697 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.697 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.697 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:49:55.698 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.173098+0000", 2024-09-20T22:49:55.698 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.4405000000000004e-05 2024-09-20T22:49:55.698 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.698 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.698 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:49:55.698 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.173125+0000", 2024-09-20T22:49:55.698 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.6950000000000001e-05 2024-09-20T22:49:55.698 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.698 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.698 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:49:55.698 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.186711+0000", 2024-09-20T22:49:55.698 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.01358608 2024-09-20T22:49:55.698 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.698 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.699 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:49:55.699 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.186780+0000", 2024-09-20T22:49:55.699 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.8942999999999994e-05 2024-09-20T22:49:55.699 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:49:55.699 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:49:55.699 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:49:55.699 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.699 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.699 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:2322 7.8 7:12e9c8db:::smithi08477203-11:head [append 0~0] snapc 55=[55,53,52,51,50,4b,47,3e] ondisk+write+known_if_redirected+supports_pool_eio e258)", 2024-09-20T22:49:55.699 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:45:37.176152+0000", 2024-09-20T22:49:55.699 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 258.48830960100003, 2024-09-20T22:49:55.699 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013033472000000001, 2024-09-20T22:49:55.699 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:49:55.700 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:49:55.700 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:49:55.700 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:49:55.700 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:49:55.700 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:49:55.700 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 2322 2024-09-20T22:49:55.700 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.700 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:49:55.700 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.700 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:49:55.700 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.176152+0000", 2024-09-20T22:49:55.700 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:49:55.700 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.700 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.701 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:49:55.701 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.176152+0000", 2024-09-20T22:49:55.701 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:49:55.701 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.701 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.701 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:49:55.701 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.176152+0000", 2024-09-20T22:49:55.701 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.8899999999999999e-07 2024-09-20T22:49:55.701 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.701 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.701 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:49:55.701 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.176156+0000", 2024-09-20T22:49:55.701 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.2669999999999998e-06 2024-09-20T22:49:55.702 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.702 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.702 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:49:55.702 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.176162+0000", 2024-09-20T22:49:55.702 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.1589999999999998e-06 2024-09-20T22:49:55.702 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.702 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.702 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:49:55.702 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.176167+0000", 2024-09-20T22:49:55.702 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.3040000000000004e-06 2024-09-20T22:49:55.702 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.702 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.702 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:49:55.702 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.176212+0000", 2024-09-20T22:49:55.703 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.5448999999999998e-05 2024-09-20T22:49:55.703 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.703 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.703 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting_for_map not empty", 2024-09-20T22:49:55.703 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.176218+0000", 2024-09-20T22:49:55.703 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.5520000000000004e-06 2024-09-20T22:49:55.703 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.703 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.703 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:49:55.703 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.182721+0000", 2024-09-20T22:49:55.703 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0065028550000000001 2024-09-20T22:49:55.703 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.703 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.703 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:49:55.704 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.182787+0000", 2024-09-20T22:49:55.704 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.5995999999999996e-05 2024-09-20T22:49:55.704 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.704 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.704 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 3,7", 2024-09-20T22:49:55.704 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.183002+0000", 2024-09-20T22:49:55.704 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00021516699999999999 2024-09-20T22:49:55.704 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.704 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.704 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:49:55.704 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.185137+0000", 2024-09-20T22:49:55.704 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0021350570000000001 2024-09-20T22:49:55.704 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.704 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.705 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:49:55.705 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.185875+0000", 2024-09-20T22:49:55.705 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00073794799999999999 2024-09-20T22:49:55.705 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.705 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.705 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:49:55.705 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.188620+0000", 2024-09-20T22:49:55.705 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0027446229999999999 2024-09-20T22:49:55.705 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.705 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.705 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:49:55.705 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.189088+0000", 2024-09-20T22:49:55.705 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00046794499999999999 2024-09-20T22:49:55.705 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.706 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.706 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:49:55.706 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.189185+0000", 2024-09-20T22:49:55.706 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.7560999999999996e-05 2024-09-20T22:49:55.706 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:49:55.706 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:49:55.706 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:49:55.706 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.706 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.706 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24652.0:2318 7.5 e258/137 7:a160b5d4:::smithi08477203-16:head v 258'43, mlcod=239'39)", 2024-09-20T22:49:55.706 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:45:37.192419+0000", 2024-09-20T22:49:55.706 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 258.47204230400001, 2024-09-20T22:49:55.706 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013974723, 2024-09-20T22:49:55.706 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:49:55.707 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:49:55.707 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:49:55.707 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:49:55.707 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.707 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:49:55.707 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.192419+0000", 2024-09-20T22:49:55.707 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:49:55.707 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.707 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.707 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:49:55.707 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.192419+0000", 2024-09-20T22:49:55.707 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:49:55.707 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.707 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.708 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:49:55.708 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.192420+0000", 2024-09-20T22:49:55.708 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.3799999999999997e-07 2024-09-20T22:49:55.708 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.708 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.708 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:49:55.708 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.204088+0000", 2024-09-20T22:49:55.708 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011668883 2024-09-20T22:49:55.708 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.708 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.708 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:49:55.708 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.204104+0000", 2024-09-20T22:49:55.708 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.5437000000000001e-05 2024-09-20T22:49:55.708 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.709 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.709 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:49:55.709 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.204112+0000", 2024-09-20T22:49:55.709 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.2519999999999999e-06 2024-09-20T22:49:55.709 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.709 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.709 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:49:55.709 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.204168+0000", 2024-09-20T22:49:55.709 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.5977999999999998e-05 2024-09-20T22:49:55.709 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.709 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.709 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:49:55.709 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.204188+0000", 2024-09-20T22:49:55.709 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.9400999999999999e-05 2024-09-20T22:49:55.710 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.710 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.710 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:49:55.710 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.206340+0000", 2024-09-20T22:49:55.710 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0021527159999999998 2024-09-20T22:49:55.710 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.710 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.710 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:49:55.710 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.206394+0000", 2024-09-20T22:49:55.710 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.3418000000000003e-05 2024-09-20T22:49:55.710 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:49:55.710 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:49:55.710 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:49:55.711 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.711 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.711 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(osd.6.0:726 7.19 e260/139 7:9eac4560:::smithi08477203-30:4c v 260'159, mlcod=259'156)", 2024-09-20T22:49:55.711 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:45:39.197025+0000", 2024-09-20T22:49:55.711 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 256.467436394, 2024-09-20T22:49:55.711 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.018150454, 2024-09-20T22:49:55.711 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:49:55.711 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:49:55.711 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:49:55.711 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:49:55.711 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.711 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:49:55.711 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.197025+0000", 2024-09-20T22:49:55.711 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:49:55.712 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.712 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.712 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:49:55.712 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.197025+0000", 2024-09-20T22:49:55.712 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:49:55.712 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.712 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.712 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:49:55.712 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.197025+0000", 2024-09-20T22:49:55.712 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.7599999999999998e-07 2024-09-20T22:49:55.712 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.712 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.712 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:49:55.713 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.197033+0000", 2024-09-20T22:49:55.713 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.9119999999999998e-06 2024-09-20T22:49:55.713 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.713 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.713 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:49:55.713 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.197042+0000", 2024-09-20T22:49:55.713 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.3349999999999996e-06 2024-09-20T22:49:55.713 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.713 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.713 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:49:55.713 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.197050+0000", 2024-09-20T22:49:55.713 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.7230000000000004e-06 2024-09-20T22:49:55.713 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.713 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.713 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:49:55.714 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.200756+0000", 2024-09-20T22:49:55.714 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0037059749999999998 2024-09-20T22:49:55.714 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.714 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.714 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:49:55.714 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.200778+0000", 2024-09-20T22:49:55.714 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.2140000000000001e-05 2024-09-20T22:49:55.714 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.714 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.714 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:49:55.714 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.215131+0000", 2024-09-20T22:49:55.714 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014353054000000001 2024-09-20T22:49:55.714 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.715 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.715 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:49:55.715 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.215175+0000", 2024-09-20T22:49:55.715 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.4039000000000003e-05 2024-09-20T22:49:55.715 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:49:55.715 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:49:55.715 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:49:55.715 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.715 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.715 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(osd.7.0:152 7.5 e269/260 7:a160b5d4:::smithi08477203-16:55 v 269'55, mlcod=263'51)", 2024-09-20T22:49:55.715 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:45:48.173893+0000", 2024-09-20T22:49:55.715 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 247.490568653, 2024-09-20T22:49:55.715 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014936935, 2024-09-20T22:49:55.715 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:49:55.715 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:49:55.716 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:49:55.716 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:49:55.716 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.716 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:49:55.716 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.173893+0000", 2024-09-20T22:49:55.716 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:49:55.716 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.716 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.716 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:49:55.716 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.173893+0000", 2024-09-20T22:49:55.716 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:49:55.716 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.716 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.717 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:49:55.717 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.173893+0000", 2024-09-20T22:49:55.717 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.8800000000000002e-07 2024-09-20T22:49:55.717 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.717 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.717 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:49:55.717 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.173916+0000", 2024-09-20T22:49:55.717 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.2889999999999999e-05 2024-09-20T22:49:55.717 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.717 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.717 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:49:55.717 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.173924+0000", 2024-09-20T22:49:55.717 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.4420000000000001e-06 2024-09-20T22:49:55.717 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.717 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.718 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:49:55.718 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.173932+0000", 2024-09-20T22:49:55.718 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.5299999999999996e-06 2024-09-20T22:49:55.718 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.718 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.718 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:49:55.718 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.174011+0000", 2024-09-20T22:49:55.718 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.9342e-05 2024-09-20T22:49:55.718 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.718 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.718 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:49:55.718 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.174047+0000", 2024-09-20T22:49:55.718 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.5237999999999997e-05 2024-09-20T22:49:55.719 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.719 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.719 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:49:55.719 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.188782+0000", 2024-09-20T22:49:55.719 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014735599 2024-09-20T22:49:55.719 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.719 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.719 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:49:55.719 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.188830+0000", 2024-09-20T22:49:55.719 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.7206000000000002e-05 2024-09-20T22:49:55.719 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:49:55.719 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:49:55.719 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:49:55.719 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.720 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.720 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(osd.7.0:153 7.17 e269/139 7:ea720b85:::smithi08477203-26:55 v 269'225, mlcod=263'221)", 2024-09-20T22:49:55.720 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:45:48.173991+0000", 2024-09-20T22:49:55.720 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 247.49047017199999, 2024-09-20T22:49:55.720 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013064868, 2024-09-20T22:49:55.720 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:49:55.720 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:49:55.720 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:49:55.720 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:49:55.720 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.720 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:49:55.720 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.173991+0000", 2024-09-20T22:49:55.720 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:49:55.720 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.720 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.721 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:49:55.721 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.173991+0000", 2024-09-20T22:49:55.722 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:49:55.722 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.722 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.723 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:49:55.723 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.173991+0000", 2024-09-20T22:49:55.723 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.9999999999999999e-07 2024-09-20T22:49:55.723 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.723 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.723 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:49:55.723 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.173996+0000", 2024-09-20T22:49:55.723 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.3080000000000001e-06 2024-09-20T22:49:55.723 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.723 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.723 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:49:55.723 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.174001+0000", 2024-09-20T22:49:55.723 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.203e-06 2024-09-20T22:49:55.724 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.724 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.724 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:49:55.724 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.174005+0000", 2024-09-20T22:49:55.724 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.4880000000000001e-06 2024-09-20T22:49:55.724 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.724 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.724 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:49:55.724 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.175600+0000", 2024-09-20T22:49:55.724 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015942459999999999 2024-09-20T22:49:55.724 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.724 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.724 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:49:55.724 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.175631+0000", 2024-09-20T22:49:55.725 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.1816000000000002e-05 2024-09-20T22:49:55.725 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.725 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.725 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:49:55.725 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.187003+0000", 2024-09-20T22:49:55.725 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011372006 2024-09-20T22:49:55.725 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.725 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.725 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:49:55.725 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.187056+0000", 2024-09-20T22:49:55.725 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.2500999999999998e-05 2024-09-20T22:49:55.725 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:49:55.725 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:49:55.725 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:49:55.726 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.726 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.726 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(osd.6.0:767 7.12 e269/263 7:4fa4b2b3:::smithi08477203-21:53 v 269'262, mlcod=265'259)", 2024-09-20T22:49:55.726 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:45:48.175130+0000", 2024-09-20T22:49:55.726 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 247.489330983, 2024-09-20T22:49:55.726 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013842727000000001, 2024-09-20T22:49:55.726 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:49:55.726 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:49:55.726 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:49:55.726 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:49:55.726 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.726 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:49:55.726 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.175130+0000", 2024-09-20T22:49:55.726 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:49:55.727 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.727 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.727 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:49:55.727 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.175130+0000", 2024-09-20T22:49:55.727 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:49:55.727 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.727 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.727 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:49:55.727 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.175130+0000", 2024-09-20T22:49:55.727 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.98e-07 2024-09-20T22:49:55.727 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.727 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.727 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:49:55.727 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.175139+0000", 2024-09-20T22:49:55.728 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.5909999999999996e-06 2024-09-20T22:49:55.728 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.728 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.728 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:49:55.728 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.175144+0000", 2024-09-20T22:49:55.728 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.361e-06 2024-09-20T22:49:55.728 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.728 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.728 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:49:55.728 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.175156+0000", 2024-09-20T22:49:55.728 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1236999999999999e-05 2024-09-20T22:49:55.728 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.728 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.729 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:49:55.729 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.176359+0000", 2024-09-20T22:49:55.729 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0012036060000000001 2024-09-20T22:49:55.729 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.729 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.729 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:49:55.729 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.176380+0000", 2024-09-20T22:49:55.729 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.0497999999999999e-05 2024-09-20T22:49:55.729 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.729 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.729 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:49:55.729 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.188936+0000", 2024-09-20T22:49:55.729 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.012556286999999999 2024-09-20T22:49:55.729 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.730 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.730 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:49:55.730 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.188973+0000", 2024-09-20T22:49:55.730 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.6949000000000001e-05 2024-09-20T22:49:55.730 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:49:55.730 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:49:55.730 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:49:55.730 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.730 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.730 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24652.0:2853 7.10 e303/139 7:0f524219:::smithi08477203-50:head v 303'85, mlcod=301'82)", 2024-09-20T22:49:55.730 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:46:23.293837+0000", 2024-09-20T22:49:55.730 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 212.37062426899999, 2024-09-20T22:49:55.730 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013368555000000001, 2024-09-20T22:49:55.731 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:49:55.731 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:49:55.731 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:49:55.731 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:49:55.731 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.731 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:49:55.732 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.293837+0000", 2024-09-20T22:49:55.732 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:49:55.732 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.732 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.732 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:49:55.732 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.293837+0000", 2024-09-20T22:49:55.732 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:49:55.732 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.733 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.733 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:49:55.733 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.293839+0000", 2024-09-20T22:49:55.733 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.7239999999999999e-06 2024-09-20T22:49:55.733 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.733 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.733 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:49:55.733 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.297226+0000", 2024-09-20T22:49:55.733 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.003387583 2024-09-20T22:49:55.733 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.733 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.733 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:49:55.733 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.297253+0000", 2024-09-20T22:49:55.733 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.6849e-05 2024-09-20T22:49:55.734 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.734 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.734 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:49:55.734 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.297266+0000", 2024-09-20T22:49:55.734 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2481e-05 2024-09-20T22:49:55.734 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.734 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.734 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:49:55.734 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.297362+0000", 2024-09-20T22:49:55.734 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.6347000000000004e-05 2024-09-20T22:49:55.734 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.734 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.734 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:49:55.734 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.297394+0000", 2024-09-20T22:49:55.735 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.1888999999999999e-05 2024-09-20T22:49:55.735 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.735 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.735 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:49:55.735 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.307147+0000", 2024-09-20T22:49:55.735 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0097533719999999997 2024-09-20T22:49:55.735 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.735 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.735 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:49:55.735 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.307206+0000", 2024-09-20T22:49:55.735 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.8310000000000002e-05 2024-09-20T22:49:55.735 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:49:55.735 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:49:55.735 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:49:55.736 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.736 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.736 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:3030 7.e 7:71b5c915:::smithi08477203-6:head [copy-from ver 216] snapc 72=[72,71,70,6e,6c,6b,6a,68] ondisk+write+known_if_redirected+supports_pool_eio e317)", 2024-09-20T22:49:55.736 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:46:37.293029+0000", 2024-09-20T22:49:55.736 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 198.371431911, 2024-09-20T22:49:55.736 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.019929916999999998, 2024-09-20T22:49:55.736 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:49:55.736 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:49:55.736 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:49:55.736 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:49:55.736 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:49:55.736 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:49:55.737 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 3030 2024-09-20T22:49:55.737 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.737 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:49:55.737 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.737 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:49:55.737 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.293029+0000", 2024-09-20T22:49:55.737 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:49:55.737 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.738 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.738 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:49:55.738 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.293029+0000", 2024-09-20T22:49:55.738 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:49:55.738 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.738 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.738 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:49:55.738 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.293031+0000", 2024-09-20T22:49:55.738 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.4190000000000001e-06 2024-09-20T22:49:55.738 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.738 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.738 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:49:55.738 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.293040+0000", 2024-09-20T22:49:55.738 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.6039999999999999e-06 2024-09-20T22:49:55.739 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.739 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.739 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:49:55.739 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.293054+0000", 2024-09-20T22:49:55.739 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.4014e-05 2024-09-20T22:49:55.739 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.739 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.739 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:49:55.739 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.293062+0000", 2024-09-20T22:49:55.739 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.8820000000000001e-06 2024-09-20T22:49:55.739 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.739 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.739 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:49:55.739 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.293121+0000", 2024-09-20T22:49:55.740 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.8977999999999997e-05 2024-09-20T22:49:55.740 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.740 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.740 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op must wait for map", 2024-09-20T22:49:55.740 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.293125+0000", 2024-09-20T22:49:55.740 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.2089999999999999e-06 2024-09-20T22:49:55.740 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.740 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.740 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:49:55.740 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.298805+0000", 2024-09-20T22:49:55.740 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0056790870000000002 2024-09-20T22:49:55.740 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.740 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.740 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:49:55.741 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.298886+0000", 2024-09-20T22:49:55.741 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.1363e-05 2024-09-20T22:49:55.741 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.741 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.741 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 3,5", 2024-09-20T22:49:55.741 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.305883+0000", 2024-09-20T22:49:55.741 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0069966159999999998 2024-09-20T22:49:55.741 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.741 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.741 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:49:55.741 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.312556+0000", 2024-09-20T22:49:55.741 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0066737009999999998 2024-09-20T22:49:55.742 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.742 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.742 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:49:55.742 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.312587+0000", 2024-09-20T22:49:55.742 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.0286000000000001e-05 2024-09-20T22:49:55.742 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.742 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.742 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:49:55.742 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.312890+0000", 2024-09-20T22:49:55.743 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00030374800000000001 2024-09-20T22:49:55.743 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.743 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.743 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:49:55.743 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.312932+0000", 2024-09-20T22:49:55.743 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.1621e-05 2024-09-20T22:49:55.743 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.743 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.743 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:49:55.743 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.312959+0000", 2024-09-20T22:49:55.743 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.7389000000000002e-05 2024-09-20T22:49:55.745 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:49:55.745 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:49:55.745 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:49:55.745 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.745 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.745 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:3031 7.e 7:71b5c915:::smithi08477203-6:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e317)", 2024-09-20T22:49:55.745 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:46:37.293082+0000", 2024-09-20T22:49:55.745 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 198.37137953300001, 2024-09-20T22:49:55.745 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.016134757, 2024-09-20T22:49:55.746 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:49:55.746 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:49:55.746 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-20T22:49:55.746 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:49:55.746 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:49:55.746 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:49:55.746 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 3031 2024-09-20T22:49:55.746 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.746 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:49:55.746 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.746 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:49:55.746 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.293082+0000", 2024-09-20T22:49:55.746 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:49:55.746 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.747 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.747 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:49:55.747 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.293082+0000", 2024-09-20T22:49:55.747 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:49:55.747 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.747 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.747 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:49:55.747 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.293082+0000", 2024-09-20T22:49:55.747 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.1799999999999995e-07 2024-09-20T22:49:55.748 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.748 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.748 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:49:55.748 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.293085+0000", 2024-09-20T22:49:55.748 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.937e-06 2024-09-20T22:49:55.748 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.748 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.748 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:49:55.748 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.293093+0000", 2024-09-20T22:49:55.748 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.554e-06 2024-09-20T22:49:55.748 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.748 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.748 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:49:55.749 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.293098+0000", 2024-09-20T22:49:55.749 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.643e-06 2024-09-20T22:49:55.749 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.749 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.749 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:49:55.749 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.293164+0000", 2024-09-20T22:49:55.749 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.5806e-05 2024-09-20T22:49:55.749 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.749 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.749 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting_for_map not empty", 2024-09-20T22:49:55.749 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.293168+0000", 2024-09-20T22:49:55.749 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.2459999999999999e-06 2024-09-20T22:49:55.749 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.749 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.750 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:49:55.750 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.299033+0000", 2024-09-20T22:49:55.750 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0058656699999999999 2024-09-20T22:49:55.750 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.750 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.750 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for blocked object", 2024-09-20T22:49:55.750 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.299069+0000", 2024-09-20T22:49:55.750 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.5392999999999998e-05 2024-09-20T22:49:55.750 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.750 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.750 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:49:55.750 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.309088+0000", 2024-09-20T22:49:55.750 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010019287999999999 2024-09-20T22:49:55.750 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.751 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.751 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:49:55.751 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.309150+0000", 2024-09-20T22:49:55.751 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.2008000000000007e-05 2024-09-20T22:49:55.751 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.751 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.751 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:49:55.751 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.309216+0000", 2024-09-20T22:49:55.751 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.6594000000000006e-05 2024-09-20T22:49:55.751 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:49:55.751 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:49:55.751 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:49:55.751 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.751 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.752 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:3290 7.8 7:12e9c8db:::smithi08477203-11:head [copy-from ver 675] snapc 7f=[7f,7e,7d,7c,7b,7a,79,78,74,73,6b] ondisk+write+known_if_redirected+supports_pool_eio e340)", 2024-09-20T22:49:55.752 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:47:00.706846+0000", 2024-09-20T22:49:55.752 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 174.95761567400001, 2024-09-20T22:49:55.752 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.018408746, 2024-09-20T22:49:55.752 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:49:55.752 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:49:55.752 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:49:55.752 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:49:55.752 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:49:55.752 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:49:55.752 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 3290 2024-09-20T22:49:55.752 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.752 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:49:55.752 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.753 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:49:55.753 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.706846+0000", 2024-09-20T22:49:55.753 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:49:55.753 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.753 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.753 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:49:55.753 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.706846+0000", 2024-09-20T22:49:55.753 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:49:55.753 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.753 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.753 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:49:55.753 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.706848+0000", 2024-09-20T22:49:55.753 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.2340000000000001e-06 2024-09-20T22:49:55.753 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.753 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.753 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:49:55.754 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.706859+0000", 2024-09-20T22:49:55.754 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1001e-05 2024-09-20T22:49:55.754 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.754 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.754 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:49:55.754 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.706879+0000", 2024-09-20T22:49:55.754 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.0556000000000001e-05 2024-09-20T22:49:55.754 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.754 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.754 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:49:55.754 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.706890+0000", 2024-09-20T22:49:55.754 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0328e-05 2024-09-20T22:49:55.754 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.755 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.755 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:49:55.755 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.706973+0000", 2024-09-20T22:49:55.755 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.3574999999999999e-05 2024-09-20T22:49:55.755 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.755 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.755 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:49:55.755 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.707042+0000", 2024-09-20T22:49:55.755 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.8987999999999997e-05 2024-09-20T22:49:55.755 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.755 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.755 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 3,7", 2024-09-20T22:49:55.755 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.717491+0000", 2024-09-20T22:49:55.755 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010448618 2024-09-20T22:49:55.756 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.756 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.756 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:49:55.756 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.722875+0000", 2024-09-20T22:49:55.756 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0053846190000000002 2024-09-20T22:49:55.756 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.756 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.756 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:49:55.756 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.724841+0000", 2024-09-20T22:49:55.756 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0019651579999999998 2024-09-20T22:49:55.756 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.756 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.756 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:49:55.757 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.725168+0000", 2024-09-20T22:49:55.757 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00032734499999999999 2024-09-20T22:49:55.757 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.757 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.757 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:49:55.757 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.725218+0000", 2024-09-20T22:49:55.757 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.0123000000000001e-05 2024-09-20T22:49:55.757 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.757 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.757 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:49:55.757 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.725254+0000", 2024-09-20T22:49:55.757 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.6201000000000001e-05 2024-09-20T22:49:55.757 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:49:55.758 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:49:55.758 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:49:55.758 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.758 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.758 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:3291 7.8 7:12e9c8db:::smithi08477203-11:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e340)", 2024-09-20T22:49:55.758 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:47:00.706927+0000", 2024-09-20T22:49:55.758 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 174.95753403500001, 2024-09-20T22:49:55.758 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014159255000000001, 2024-09-20T22:49:55.758 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:49:55.758 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:49:55.758 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-20T22:49:55.758 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:49:55.758 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:49:55.758 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:49:55.759 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 3291 2024-09-20T22:49:55.759 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.759 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:49:55.759 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.759 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:49:55.759 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.706927+0000", 2024-09-20T22:49:55.759 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:49:55.759 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.759 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.759 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:49:55.759 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.706927+0000", 2024-09-20T22:49:55.759 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:49:55.759 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.759 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.760 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:49:55.760 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.706928+0000", 2024-09-20T22:49:55.760 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.4099999999999999e-07 2024-09-20T22:49:55.760 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.760 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.760 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:49:55.760 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.706941+0000", 2024-09-20T22:49:55.760 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3335e-05 2024-09-20T22:49:55.760 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.760 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.760 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:49:55.760 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.706948+0000", 2024-09-20T22:49:55.760 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.3350000000000001e-06 2024-09-20T22:49:55.760 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.760 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.761 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:49:55.761 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.706954+0000", 2024-09-20T22:49:55.761 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.9190000000000004e-06 2024-09-20T22:49:55.761 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.761 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.761 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:49:55.761 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.707166+0000", 2024-09-20T22:49:55.761 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00021159 2024-09-20T22:49:55.761 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.761 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.761 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for blocked object", 2024-09-20T22:49:55.761 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.707217+0000", 2024-09-20T22:49:55.761 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.1167e-05 2024-09-20T22:49:55.761 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.761 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.762 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:49:55.762 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.720939+0000", 2024-09-20T22:49:55.762 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013721458000000001 2024-09-20T22:49:55.762 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.762 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.762 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:49:55.762 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.721010+0000", 2024-09-20T22:49:55.762 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.1570999999999996e-05 2024-09-20T22:49:55.762 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.762 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.762 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:49:55.762 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.721086+0000", 2024-09-20T22:49:55.762 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.6339000000000002e-05 2024-09-20T22:49:55.762 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:49:55.763 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:49:55.763 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:49:55.763 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.763 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.763 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(osd.5.0:167 7.2 e347/169 7:45b54b6b:::smithi08477203-2:81 v 347'140, mlcod=345'136)", 2024-09-20T22:49:55.763 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:47:08.204691+0000", 2024-09-20T22:49:55.763 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 167.45976993400001, 2024-09-20T22:49:55.763 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.01335918, 2024-09-20T22:49:55.763 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:49:55.763 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:49:55.763 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:49:55.763 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:49:55.763 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.763 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:49:55.764 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:08.204691+0000", 2024-09-20T22:49:55.764 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:49:55.764 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.764 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.764 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:49:55.764 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:08.204691+0000", 2024-09-20T22:49:55.764 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:49:55.764 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.764 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.764 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:49:55.764 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:08.204692+0000", 2024-09-20T22:49:55.764 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.5400000000000001e-07 2024-09-20T22:49:55.764 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.764 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.764 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:49:55.765 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:08.204697+0000", 2024-09-20T22:49:55.765 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.614e-06 2024-09-20T22:49:55.765 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.765 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.765 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:49:55.765 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:08.204705+0000", 2024-09-20T22:49:55.765 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.8259999999999993e-06 2024-09-20T22:49:55.765 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.765 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.765 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:49:55.765 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:08.204715+0000", 2024-09-20T22:49:55.765 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.8279999999999998e-06 2024-09-20T22:49:55.765 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.765 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.766 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:49:55.766 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:08.204801+0000", 2024-09-20T22:49:55.766 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.6263999999999999e-05 2024-09-20T22:49:55.766 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.766 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.766 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:49:55.766 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:08.204823+0000", 2024-09-20T22:49:55.766 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.1971999999999999e-05 2024-09-20T22:49:55.766 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.766 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.766 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:49:55.766 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:08.217961+0000", 2024-09-20T22:49:55.766 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013137618 2024-09-20T22:49:55.766 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.767 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.767 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:49:55.767 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:08.218050+0000", 2024-09-20T22:49:55.767 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.9504000000000002e-05 2024-09-20T22:49:55.767 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:49:55.767 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:49:55.767 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:49:55.767 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.767 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.767 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24652.0:3760 7.13 e383/260 7:cae5792a:::smithi08477203-32:head v 383'78, mlcod=377'74)", 2024-09-20T22:49:55.767 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:47:43.291645+0000", 2024-09-20T22:49:55.767 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 132.372816062, 2024-09-20T22:49:55.767 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.012696648, 2024-09-20T22:49:55.767 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:49:55.767 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:49:55.768 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:49:55.768 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:49:55.768 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.768 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:49:55.768 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:43.291645+0000", 2024-09-20T22:49:55.768 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:49:55.768 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.768 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.768 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:49:55.768 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:43.291645+0000", 2024-09-20T22:49:55.768 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:49:55.768 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.768 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.768 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:49:55.769 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:43.291645+0000", 2024-09-20T22:49:55.769 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.0800000000000001e-07 2024-09-20T22:49:55.769 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.769 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.769 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:49:55.769 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:43.294764+0000", 2024-09-20T22:49:55.769 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0031183420000000001 2024-09-20T22:49:55.769 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.769 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.769 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:49:55.769 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:43.294777+0000", 2024-09-20T22:49:55.769 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3465999999999999e-05 2024-09-20T22:49:55.769 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.769 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.770 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:49:55.770 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:43.294788+0000", 2024-09-20T22:49:55.770 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0662999999999999e-05 2024-09-20T22:49:55.770 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.770 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.770 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:49:55.770 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:43.294952+0000", 2024-09-20T22:49:55.770 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00016369799999999999 2024-09-20T22:49:55.770 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.770 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.770 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:49:55.770 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:43.294978+0000", 2024-09-20T22:49:55.770 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.6414000000000002e-05 2024-09-20T22:49:55.770 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.770 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.771 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:49:55.771 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:43.304313+0000", 2024-09-20T22:49:55.771 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0093352039999999997 2024-09-20T22:49:55.771 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.771 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.771 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:49:55.771 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:43.304342+0000", 2024-09-20T22:49:55.771 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.8552999999999999e-05 2024-09-20T22:49:55.771 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:49:55.771 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:49:55.771 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:49:55.771 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.771 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.772 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24652.0:4534 7.12 e456/421 7:4b06cbed:::smithi08477203-49:head v 456'446, mlcod=454'443)", 2024-09-20T22:49:55.772 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:48:56.245037+0000", 2024-09-20T22:49:55.772 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 59.419424040999999, 2024-09-20T22:49:55.772 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013221933999999999, 2024-09-20T22:49:55.772 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:49:55.772 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:49:55.772 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:49:55.772 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:49:55.772 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.772 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:49:55.772 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:56.245037+0000", 2024-09-20T22:49:55.772 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:49:55.772 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.772 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.772 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:49:55.773 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:56.245037+0000", 2024-09-20T22:49:55.773 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:49:55.773 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.773 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.773 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:49:55.773 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:56.245037+0000", 2024-09-20T22:49:55.773 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.4600000000000001e-07 2024-09-20T22:49:55.773 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.773 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.773 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:49:55.773 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:56.248142+0000", 2024-09-20T22:49:55.773 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0031044839999999998 2024-09-20T22:49:55.773 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.773 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.774 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:49:55.774 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:56.248174+0000", 2024-09-20T22:49:55.774 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.2555000000000003e-05 2024-09-20T22:49:55.774 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.774 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.774 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:49:55.774 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:56.248188+0000", 2024-09-20T22:49:55.774 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.308e-05 2024-09-20T22:49:55.774 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.774 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.774 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:49:55.774 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:56.248292+0000", 2024-09-20T22:49:55.774 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000104182 2024-09-20T22:49:55.774 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.774 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.775 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:49:55.775 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:56.248318+0000", 2024-09-20T22:49:55.775 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.652e-05 2024-09-20T22:49:55.775 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.775 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.775 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:49:55.775 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:56.258207+0000", 2024-09-20T22:49:55.775 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0098883909999999998 2024-09-20T22:49:55.775 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:49:55.775 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:49:55.775 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:49:55.775 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:56.258259+0000", 2024-09-20T22:49:55.775 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.2475999999999997e-05 2024-09-20T22:49:55.775 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:49:55.775 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:49:55.776 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:49:55.776 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:49:55.776 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:49:55.776 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-20T22:49:55.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:55.776 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:55.777 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:49:55.777 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:55 smithi154 ceph-mon[33264]: pgmap v672: 158 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 153 active+clean; 497 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 17 MiB/s rd, 2.3 MiB/s wr, 19 op/s 2024-09-20T22:49:55.777 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:55 smithi154 ceph-mon[33264]: 3.1b deep-scrub starts 2024-09-20T22:49:55.777 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:55 smithi154 ceph-mon[33264]: 3.1b deep-scrub ok 2024-09-20T22:49:55.777 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:55 smithi154 ceph-mon[33264]: 5.8 scrub starts 2024-09-20T22:49:55.777 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:55 smithi154 ceph-mon[33264]: 5.8 scrub ok 2024-09-20T22:49:55.777 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:55 smithi154 ceph-mon[33264]: osdmap e514: 8 total, 8 up, 8 in 2024-09-20T22:49:55.777 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:55 smithi154 ceph-mon[33264]: 3.13 scrub starts 2024-09-20T22:49:55.777 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:55 smithi154 ceph-mon[33264]: 3.13 scrub ok 2024-09-20T22:49:55.877 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:49:55.899 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:55.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:55.900 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:49:56.001 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:49:56.023 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:56.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:56.024 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:49:56.124 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:49:56.145 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:56.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:56.146 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:49:56.247 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:49:56.273 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:56.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:56.275 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:49:56.320 INFO:tasks.rados.rados.0.smithi084.stdout:2033: done (2 left) 2024-09-20T22:49:56.321 INFO:tasks.rados.rados.0.smithi084.stdout:2034: delete oid 26 current snap is 221 2024-09-20T22:49:56.321 INFO:tasks.rados.rados.0.smithi084.stdout:2032: finishing rollback tid 0 to smithi08477203-32 2024-09-20T22:49:56.321 INFO:tasks.rados.rados.0.smithi084.stdout:2032: finishing rollback tid 1 to smithi08477203-32 2024-09-20T22:49:56.321 INFO:tasks.rados.rados.0.smithi084.stdout:2032: finishing rollback tid 2 to smithi08477203-32 2024-09-20T22:49:56.321 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 32 v 728 (ObjNum 684 snap 211 seq_num 684) dirty exists 2024-09-20T22:49:56.321 INFO:tasks.rados.rados.0.smithi084.stdout:2031: finishing rollback tid 0 to smithi08477203-17 2024-09-20T22:49:56.321 INFO:tasks.rados.rados.0.smithi084.stdout:2031: finishing rollback tid 1 to smithi08477203-17 2024-09-20T22:49:56.321 INFO:tasks.rados.rados.0.smithi084.stdout:2031: finishing rollback tid 2 to smithi08477203-17 2024-09-20T22:49:56.321 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 17 v 746 (ObjNum 620 snap 185 seq_num 620) dirty exists 2024-09-20T22:49:56.326 INFO:tasks.rados.rados.0.smithi084.stdout:2031: done (2 left) 2024-09-20T22:49:56.327 INFO:tasks.rados.rados.0.smithi084.stdout:2032: done (1 left) 2024-09-20T22:49:56.327 INFO:tasks.rados.rados.0.smithi084.stdout:2034: done (0 left) 2024-09-20T22:49:56.327 INFO:tasks.rados.rados.0.smithi084.stdout:2035: snap_create 2024-09-20T22:49:56.375 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:49:56.400 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:56.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:56.401 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:49:56.501 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:49:56.508 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-20T22:49:56.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:56.523 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:56.523 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:49:56.546 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:49:56.587 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:49:56.614 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:56 smithi084 ceph-mon[40024]: osdmap e515: 8 total, 8 up, 8 in 2024-09-20T22:49:56.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:56 smithi084 ceph-mon[40024]: 1.2 scrub starts 2024-09-20T22:49:56.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:56 smithi084 ceph-mon[40024]: 1.2 scrub ok 2024-09-20T22:49:56.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:56 smithi084 ceph-mon[40024]: 4.f scrub starts 2024-09-20T22:49:56.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:56 smithi084 ceph-mon[40024]: 4.f scrub ok 2024-09-20T22:49:56.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:56 smithi084 ceph-mon[40024]: 4.5 scrub starts 2024-09-20T22:49:56.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:56 smithi084 ceph-mon[40024]: 4.5 scrub ok 2024-09-20T22:49:56.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:56 smithi084 ceph-mon[40024]: 4.1a scrub starts 2024-09-20T22:49:56.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:56 smithi084 ceph-mon[40024]: 4.1a scrub ok 2024-09-20T22:49:56.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:56 smithi084 ceph-mon[29774]: osdmap e515: 8 total, 8 up, 8 in 2024-09-20T22:49:56.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:56 smithi084 ceph-mon[29774]: 1.2 scrub starts 2024-09-20T22:49:56.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:56 smithi084 ceph-mon[29774]: 1.2 scrub ok 2024-09-20T22:49:56.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:56 smithi084 ceph-mon[29774]: 4.f scrub starts 2024-09-20T22:49:56.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:56 smithi084 ceph-mon[29774]: 4.f scrub ok 2024-09-20T22:49:56.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:56 smithi084 ceph-mon[29774]: 4.5 scrub starts 2024-09-20T22:49:56.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:56 smithi084 ceph-mon[29774]: 4.5 scrub ok 2024-09-20T22:49:56.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:56 smithi084 ceph-mon[29774]: 4.1a scrub starts 2024-09-20T22:49:56.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:56 smithi084 ceph-mon[29774]: 4.1a scrub ok 2024-09-20T22:49:56.624 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:49:56.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:56.646 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:56.646 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:49:56.747 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:49:56.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:56 smithi154 ceph-mon[33264]: osdmap e515: 8 total, 8 up, 8 in 2024-09-20T22:49:56.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:56 smithi154 ceph-mon[33264]: 1.2 scrub starts 2024-09-20T22:49:56.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:56 smithi154 ceph-mon[33264]: 1.2 scrub ok 2024-09-20T22:49:56.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:56 smithi154 ceph-mon[33264]: 4.f scrub starts 2024-09-20T22:49:56.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:56 smithi154 ceph-mon[33264]: 4.f scrub ok 2024-09-20T22:49:56.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:56 smithi154 ceph-mon[33264]: 4.5 scrub starts 2024-09-20T22:49:56.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:56 smithi154 ceph-mon[33264]: 4.5 scrub ok 2024-09-20T22:49:56.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:56 smithi154 ceph-mon[33264]: 4.1a scrub starts 2024-09-20T22:49:56.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:56 smithi154 ceph-mon[33264]: 4.1a scrub ok 2024-09-20T22:49:56.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:56.770 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:56.770 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:49:56.805 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-20T22:49:56.870 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:49:56.891 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:56.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:56.892 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:49:56.993 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:49:57.019 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:57.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:57.020 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:49:57.121 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:49:57.147 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:57.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:57.148 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:49:57.151 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:49:57.181 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:49:57.249 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:49:57.273 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:57.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:57.274 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:49:57.323 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.6 to 1 (10000) 2024-09-20T22:49:57.330 INFO:tasks.rados.rados.0.smithi084.stdout:2035: done (0 left) 2024-09-20T22:49:57.330 INFO:tasks.rados.rados.0.smithi084.stdout:2036: read oid 41 snap 213 2024-09-20T22:49:57.330 INFO:tasks.rados.rados.0.smithi084.stdout:2036: expect (ObjNum 691 snap 213 seq_num 691) 2024-09-20T22:49:57.330 INFO:tasks.rados.rados.0.smithi084.stdout:2037: snap_remove snap 203 2024-09-20T22:49:57.374 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:49:57.398 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:57.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:57.399 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:49:57.500 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:49:57.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:57.524 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:57.524 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:49:57.531 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:57 smithi084 ceph-mon[40024]: pgmap v675: 158 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 153 active+clean; 497 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 17 MiB/s rd, 2.3 MiB/s wr, 19 op/s 2024-09-20T22:49:57.532 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:57 smithi084 ceph-mon[40024]: 4.3 deep-scrub starts 2024-09-20T22:49:57.532 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:57 smithi084 ceph-mon[40024]: 4.3 deep-scrub ok 2024-09-20T22:49:57.532 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:57 smithi084 ceph-mon[40024]: osdmap e516: 8 total, 8 up, 8 in 2024-09-20T22:49:57.532 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:57 smithi084 ceph-mon[40024]: 3.15 scrub starts 2024-09-20T22:49:57.532 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:57 smithi084 ceph-mon[40024]: 3.15 scrub ok 2024-09-20T22:49:57.532 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:57 smithi084 ceph-mon[40024]: 4.1 scrub starts 2024-09-20T22:49:57.532 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:57 smithi084 ceph-mon[40024]: 4.1 scrub ok 2024-09-20T22:49:57.532 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:57 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/3848035442' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-09-20T22:49:57.550 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:57 smithi084 ceph-mon[29774]: pgmap v675: 158 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 153 active+clean; 497 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 17 MiB/s rd, 2.3 MiB/s wr, 19 op/s 2024-09-20T22:49:57.550 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:57 smithi084 ceph-mon[29774]: 4.3 deep-scrub starts 2024-09-20T22:49:57.550 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:57 smithi084 ceph-mon[29774]: 4.3 deep-scrub ok 2024-09-20T22:49:57.550 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:57 smithi084 ceph-mon[29774]: osdmap e516: 8 total, 8 up, 8 in 2024-09-20T22:49:57.551 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:57 smithi084 ceph-mon[29774]: 3.15 scrub starts 2024-09-20T22:49:57.551 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:57 smithi084 ceph-mon[29774]: 3.15 scrub ok 2024-09-20T22:49:57.551 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:57 smithi084 ceph-mon[29774]: 4.1 scrub starts 2024-09-20T22:49:57.551 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:57 smithi084 ceph-mon[29774]: 4.1 scrub ok 2024-09-20T22:49:57.551 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:57 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/3848035442' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-09-20T22:49:57.625 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:49:57.648 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:57.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:57.649 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:49:57.750 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:49:57.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:57 smithi154 ceph-mon[33264]: pgmap v675: 158 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 153 active+clean; 497 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 17 MiB/s rd, 2.3 MiB/s wr, 19 op/s 2024-09-20T22:49:57.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:57 smithi154 ceph-mon[33264]: 4.3 deep-scrub starts 2024-09-20T22:49:57.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:57 smithi154 ceph-mon[33264]: 4.3 deep-scrub ok 2024-09-20T22:49:57.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:57 smithi154 ceph-mon[33264]: osdmap e516: 8 total, 8 up, 8 in 2024-09-20T22:49:57.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:57 smithi154 ceph-mon[33264]: 3.15 scrub starts 2024-09-20T22:49:57.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:57 smithi154 ceph-mon[33264]: 3.15 scrub ok 2024-09-20T22:49:57.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:57 smithi154 ceph-mon[33264]: 4.1 scrub starts 2024-09-20T22:49:57.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:57 smithi154 ceph-mon[33264]: 4.1 scrub ok 2024-09-20T22:49:57.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:57 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/3848035442' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-09-20T22:49:57.772 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:57.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:57.773 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:49:57.874 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:49:57.900 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:57.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:57.902 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:49:58.002 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:49:58.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:58.083 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:58.083 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:49:58.184 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:49:58.207 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:58.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:58.208 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:49:58.237 INFO:tasks.rados.rados.0.smithi084.stdout:2037: done (1 left) 2024-09-20T22:49:58.237 INFO:tasks.rados.rados.0.smithi084.stdout:2038: read oid 47 snap -1 2024-09-20T22:49:58.237 INFO:tasks.rados.rados.0.smithi084.stdout:2038: expect (ObjNum 671 snap 205 seq_num 671) 2024-09-20T22:49:58.237 INFO:tasks.rados.rados.0.smithi084.stdout:2039: write oid 38 current snap is 222 2024-09-20T22:49:58.237 INFO:tasks.rados.rados.0.smithi084.stdout:2039: seq_num 713 ranges {783391=473366,1756720=475585,2797182=788004} 2024-09-20T22:49:58.241 INFO:tasks.rados.rados.0.smithi084.stdout:2039: writing smithi08477203-38 from 783391 to 1256757 tid 1 2024-09-20T22:49:58.247 INFO:tasks.rados.rados.0.smithi084.stdout:2039: writing smithi08477203-38 from 1756720 to 2232305 tid 2 2024-09-20T22:49:58.256 INFO:tasks.rados.rados.0.smithi084.stdout:2039: writing smithi08477203-38 from 2797182 to 3585186 tid 3 2024-09-20T22:49:58.256 INFO:tasks.rados.rados.0.smithi084.stdout:2040: snap_create 2024-09-20T22:49:58.256 INFO:tasks.rados.rados.0.smithi084.stdout:2036: expect (ObjNum 691 snap 213 seq_num 691) 2024-09-20T22:49:58.308 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:49:58.336 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:58.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:58.337 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:49:58.398 INFO:tasks.rados.rados.0.smithi084.stdout:2038: expect (ObjNum 671 snap 205 seq_num 671) 2024-09-20T22:49:58.413 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:49:58.413 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:49:58.436 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:49:58.436 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:49:58.438 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:49:58.454 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:49:58.455 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:49:58.455 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:49:58 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: ::ffff:172.21.15.154 - - [20/Sep/2024:22:49:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-20T22:49:58.455 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:58 smithi084 ceph-mon[40024]: 1.8 scrub starts 2024-09-20T22:49:58.455 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:58 smithi084 ceph-mon[40024]: 1.8 scrub ok 2024-09-20T22:49:58.455 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:58 smithi084 ceph-mon[40024]: 6.6 scrub starts 2024-09-20T22:49:58.455 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:58 smithi084 ceph-mon[40024]: 6.6 scrub ok 2024-09-20T22:49:58.456 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:58 smithi084 ceph-mon[40024]: 5.d scrub starts 2024-09-20T22:49:58.456 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:58 smithi084 ceph-mon[40024]: 5.d scrub ok 2024-09-20T22:49:58.456 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:58 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/3848035442' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-20T22:49:58.456 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:58 smithi084 ceph-mon[40024]: osdmap e517: 8 total, 8 up, 8 in 2024-09-20T22:49:58.456 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:58 smithi084 ceph-mon[40024]: 6.12 scrub starts 2024-09-20T22:49:58.456 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:58 smithi084 ceph-mon[40024]: 4.7 scrub starts 2024-09-20T22:49:58.456 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:58 smithi084 ceph-mon[40024]: 4.7 scrub ok 2024-09-20T22:49:58.456 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:58 smithi084 ceph-mon[40024]: osdmap e518: 8 total, 8 up, 8 in 2024-09-20T22:49:58.456 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:58 smithi084 ceph-mon[29774]: 1.8 scrub starts 2024-09-20T22:49:58.456 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:58 smithi084 ceph-mon[29774]: 1.8 scrub ok 2024-09-20T22:49:58.456 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:58 smithi084 ceph-mon[29774]: 6.6 scrub starts 2024-09-20T22:49:58.456 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:58 smithi084 ceph-mon[29774]: 6.6 scrub ok 2024-09-20T22:49:58.456 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:58 smithi084 ceph-mon[29774]: 5.d scrub starts 2024-09-20T22:49:58.457 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:58 smithi084 ceph-mon[29774]: 5.d scrub ok 2024-09-20T22:49:58.457 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:58 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/3848035442' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-20T22:49:58.457 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:58 smithi084 ceph-mon[29774]: osdmap e517: 8 total, 8 up, 8 in 2024-09-20T22:49:58.457 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:58 smithi084 ceph-mon[29774]: 6.12 scrub starts 2024-09-20T22:49:58.457 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:58 smithi084 ceph-mon[29774]: 4.7 scrub starts 2024-09-20T22:49:58.457 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:58 smithi084 ceph-mon[29774]: 4.7 scrub ok 2024-09-20T22:49:58.457 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:58 smithi084 ceph-mon[29774]: osdmap e518: 8 total, 8 up, 8 in 2024-09-20T22:49:58.459 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:58.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:58.460 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:49:58.471 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-20T22:49:58.472 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T22:49:58.488 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:49:58.488 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:49:58.505 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:49:58.505 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:49:58.522 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:49:58.522 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:49:58.539 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:49:58.539 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:49:58.560 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:49:58.584 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:58.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:58.585 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:49:58.686 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:49:58.709 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:58.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:58.710 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:49:58.726 INFO:tasks.rados.rados.0.smithi084.stdout:2039: finishing write tid 1 to smithi08477203-38 2024-09-20T22:49:58.726 INFO:tasks.rados.rados.0.smithi084.stdout:2039: finishing write tid 2 to smithi08477203-38 2024-09-20T22:49:58.726 INFO:tasks.rados.rados.0.smithi084.stdout:2039: finishing write tid 3 to smithi08477203-38 2024-09-20T22:49:58.727 INFO:tasks.rados.rados.0.smithi084.stdout:2039: finishing write tid 4 to smithi08477203-38 2024-09-20T22:49:58.727 INFO:tasks.rados.rados.0.smithi084.stdout:2039: finishing write tid 5 to smithi08477203-38 2024-09-20T22:49:58.727 INFO:tasks.rados.rados.0.smithi084.stdout:2039: oid 38 updating version 0 to 678 2024-09-20T22:49:58.727 INFO:tasks.rados.rados.0.smithi084.stdout:2039: oid 38 version 678 is already newer than 673 2024-09-20T22:49:58.727 INFO:tasks.rados.rados.0.smithi084.stdout:2039: oid 38 updating version 678 to 683 2024-09-20T22:49:58.727 INFO:tasks.rados.rados.0.smithi084.stdout:2039: oid 38 updating version 683 to 684 2024-09-20T22:49:58.727 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 38 v 684 (ObjNum 713 snap 222 seq_num 713) dirty exists 2024-09-20T22:49:58.727 INFO:tasks.rados.rados.0.smithi084.stdout:2039: left oid 38 (ObjNum 713 snap 222 seq_num 713) 2024-09-20T22:49:58.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:58 smithi154 ceph-mon[33264]: 1.8 scrub starts 2024-09-20T22:49:58.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:58 smithi154 ceph-mon[33264]: 1.8 scrub ok 2024-09-20T22:49:58.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:58 smithi154 ceph-mon[33264]: 6.6 scrub starts 2024-09-20T22:49:58.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:58 smithi154 ceph-mon[33264]: 6.6 scrub ok 2024-09-20T22:49:58.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:58 smithi154 ceph-mon[33264]: 5.d scrub starts 2024-09-20T22:49:58.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:58 smithi154 ceph-mon[33264]: 5.d scrub ok 2024-09-20T22:49:58.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:58 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/3848035442' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-20T22:49:58.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:58 smithi154 ceph-mon[33264]: osdmap e517: 8 total, 8 up, 8 in 2024-09-20T22:49:58.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:58 smithi154 ceph-mon[33264]: 6.12 scrub starts 2024-09-20T22:49:58.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:58 smithi154 ceph-mon[33264]: 4.7 scrub starts 2024-09-20T22:49:58.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:58 smithi154 ceph-mon[33264]: 4.7 scrub ok 2024-09-20T22:49:58.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:58 smithi154 ceph-mon[33264]: osdmap e518: 8 total, 8 up, 8 in 2024-09-20T22:49:58.812 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:49:58.833 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:58.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:58.834 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:49:58.935 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:49:58.957 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:58.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:58.958 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:49:58.958 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-20T22:49:58.959 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:49:59.059 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:49:59.085 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:59.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:59.086 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:49:59.187 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:49:59.209 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:59.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:59.210 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:49:59.236 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is set 2024-09-20T22:49:59.240 INFO:tasks.rados.rados.0.smithi084.stdout:2036: done (3 left) 2024-09-20T22:49:59.241 INFO:tasks.rados.rados.0.smithi084.stdout:2038: done (2 left) 2024-09-20T22:49:59.241 INFO:tasks.rados.rados.0.smithi084.stdout:2039: done (1 left) 2024-09-20T22:49:59.241 INFO:tasks.rados.rados.0.smithi084.stdout:2040: done (0 left) 2024-09-20T22:49:59.241 INFO:tasks.rados.rados.0.smithi084.stdout:2041: rollback oid 8 current snap is 223 2024-09-20T22:49:59.241 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 8 to 206 2024-09-20T22:49:59.241 INFO:tasks.rados.rados.0.smithi084.stdout:2042: snap_create 2024-09-20T22:49:59.243 INFO:tasks.rados.rados.0.smithi084.stdout:2041: finishing rollback tid 1 to smithi08477203-8 2024-09-20T22:49:59.243 INFO:tasks.rados.rados.0.smithi084.stdout:2041: finishing rollback tid 2 to smithi08477203-8 2024-09-20T22:49:59.243 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 8 v 792 (ObjNum 625 snap 186 seq_num 625) dirty exists 2024-09-20T22:49:59.311 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:49:59.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:59.338 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:59.338 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:49:59.439 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:49:59.461 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:49:59.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:59.462 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:49:59.563 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:49:59.580 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:59 smithi084 ceph-mon[40024]: 6.12 scrub ok 2024-09-20T22:49:59.580 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:59 smithi084 ceph-mon[40024]: 6.7 scrub starts 2024-09-20T22:49:59.580 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:59 smithi084 ceph-mon[40024]: 6.7 scrub ok 2024-09-20T22:49:59.580 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:59 smithi084 ceph-mon[40024]: pgmap v678: 158 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 153 active+clean; 497 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-20T22:49:59.580 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:59 smithi084 ceph-mon[40024]: 1.4 scrub starts 2024-09-20T22:49:59.580 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:59 smithi084 ceph-mon[40024]: 1.4 scrub ok 2024-09-20T22:49:59.580 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:59 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/1373478469' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:49:59.580 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:59 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/2507292134' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-20T22:49:59.580 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:59 smithi084 ceph-mon[40024]: 5.18 scrub starts 2024-09-20T22:49:59.580 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:59 smithi084 ceph-mon[40024]: 5.18 scrub ok 2024-09-20T22:49:59.580 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:59 smithi084 ceph-mon[40024]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:49:59.580 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:59 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/2507292134' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T22:49:59.580 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:49:59 smithi084 ceph-mon[40024]: osdmap e519: 8 total, 8 up, 8 in 2024-09-20T22:49:59.581 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:59 smithi084 ceph-mon[29774]: 6.12 scrub ok 2024-09-20T22:49:59.581 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:59 smithi084 ceph-mon[29774]: 6.7 scrub starts 2024-09-20T22:49:59.581 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:59 smithi084 ceph-mon[29774]: 6.7 scrub ok 2024-09-20T22:49:59.581 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:59 smithi084 ceph-mon[29774]: pgmap v678: 158 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 153 active+clean; 497 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-20T22:49:59.581 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:59 smithi084 ceph-mon[29774]: 1.4 scrub starts 2024-09-20T22:49:59.581 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:59 smithi084 ceph-mon[29774]: 1.4 scrub ok 2024-09-20T22:49:59.581 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:59 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/1373478469' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:49:59.581 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:59 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/2507292134' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-20T22:49:59.581 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:59 smithi084 ceph-mon[29774]: 5.18 scrub starts 2024-09-20T22:49:59.581 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:59 smithi084 ceph-mon[29774]: 5.18 scrub ok 2024-09-20T22:49:59.581 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:59 smithi084 ceph-mon[29774]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:49:59.581 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:59 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/2507292134' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T22:49:59.582 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:49:59 smithi084 ceph-mon[29774]: osdmap e519: 8 total, 8 up, 8 in 2024-09-20T22:49:59.588 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:59.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:59.589 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:49:59.690 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:49:59.714 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:59.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:59.715 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:49:59.754 INFO:journalctl@ceph.iscsi.iscsi.a.smithi154.stdout:Sep 20 22:49:59 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-iscsi-iscsi-a[59730]: debug there is no tcmu-runner data available 2024-09-20T22:49:59.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:59 smithi154 ceph-mon[33264]: 6.12 scrub ok 2024-09-20T22:49:59.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:59 smithi154 ceph-mon[33264]: 6.7 scrub starts 2024-09-20T22:49:59.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:59 smithi154 ceph-mon[33264]: 6.7 scrub ok 2024-09-20T22:49:59.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:59 smithi154 ceph-mon[33264]: pgmap v678: 158 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 153 active+clean; 497 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-20T22:49:59.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:59 smithi154 ceph-mon[33264]: 1.4 scrub starts 2024-09-20T22:49:59.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:59 smithi154 ceph-mon[33264]: 1.4 scrub ok 2024-09-20T22:49:59.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:59 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/1373478469' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:49:59.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:59 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/2507292134' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-20T22:49:59.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:59 smithi154 ceph-mon[33264]: 5.18 scrub starts 2024-09-20T22:49:59.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:59 smithi154 ceph-mon[33264]: 5.18 scrub ok 2024-09-20T22:49:59.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:59 smithi154 ceph-mon[33264]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:49:59.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:59 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/2507292134' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T22:49:59.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:49:59 smithi154 ceph-mon[33264]: osdmap e519: 8 total, 8 up, 8 in 2024-09-20T22:49:59.816 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:49:59.841 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:59.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:59.842 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:49:59.943 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:49:59.968 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:49:59.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:49:59.969 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:50:00.070 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:50:00.091 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:00.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:00.092 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:50:00.193 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:50:00.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:00.216 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:00.216 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:50:00.246 INFO:tasks.rados.rados.0.smithi084.stdout:2041: done (1 left) 2024-09-20T22:50:00.246 INFO:tasks.rados.rados.0.smithi084.stdout:2042: done (0 left) 2024-09-20T22:50:00.246 INFO:tasks.rados.rados.0.smithi084.stdout:2043: write oid 27 current snap is 224 2024-09-20T22:50:00.246 INFO:tasks.rados.rados.0.smithi084.stdout:2043: seq_num 714 ranges {400198=548615,1415202=584996} 2024-09-20T22:50:00.252 INFO:tasks.rados.rados.0.smithi084.stdout:2043: writing smithi08477203-27 from 400198 to 948813 tid 1 2024-09-20T22:50:00.258 INFO:tasks.rados.rados.0.smithi084.stdout:2043: writing smithi08477203-27 from 1415202 to 2000198 tid 2 2024-09-20T22:50:00.258 INFO:tasks.rados.rados.0.smithi084.stdout:2044: read oid 15 snap -1 2024-09-20T22:50:00.258 INFO:tasks.rados.rados.0.smithi084.stdout:2044: expect (ObjNum 625 snap 186 seq_num 625) 2024-09-20T22:50:00.258 INFO:tasks.rados.rados.0.smithi084.stdout:2045: write oid 44 current snap is 224 2024-09-20T22:50:00.258 INFO:tasks.rados.rados.0.smithi084.stdout:2043: finishing write tid 1 to smithi08477203-27 2024-09-20T22:50:00.258 INFO:tasks.rados.rados.0.smithi084.stdout:2045: seq_num 715 ranges {417005=740217,1590037=426968} 2024-09-20T22:50:00.264 INFO:tasks.rados.rados.0.smithi084.stdout:2045: writing smithi08477203-44 from 417005 to 1157222 tid 1 2024-09-20T22:50:00.269 INFO:tasks.rados.rados.0.smithi084.stdout:2045: writing smithi08477203-44 from 1590037 to 2017005 tid 2 2024-09-20T22:50:00.269 INFO:tasks.rados.rados.0.smithi084.stdout:2046: rollback oid 6 current snap is 224 2024-09-20T22:50:00.269 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 6 to 222 2024-09-20T22:50:00.269 INFO:tasks.rados.rados.0.smithi084.stdout:2043: finishing write tid 2 to smithi08477203-27 2024-09-20T22:50:00.269 INFO:tasks.rados.rados.0.smithi084.stdout:2043: finishing write tid 3 to smithi08477203-27 2024-09-20T22:50:00.269 INFO:tasks.rados.rados.0.smithi084.stdout:2043: finishing write tid 4 to smithi08477203-27 2024-09-20T22:50:00.270 INFO:tasks.rados.rados.0.smithi084.stdout:2043: oid 27 updating version 0 to 337 2024-09-20T22:50:00.270 INFO:tasks.rados.rados.0.smithi084.stdout:2043: oid 27 version 337 is already newer than 336 2024-09-20T22:50:00.270 INFO:tasks.rados.rados.0.smithi084.stdout:2043: oid 27 updating version 337 to 338 2024-09-20T22:50:00.270 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 27 v 338 (ObjNum 714 snap 224 seq_num 714) dirty exists 2024-09-20T22:50:00.270 INFO:tasks.rados.rados.0.smithi084.stdout:2043: left oid 27 (ObjNum 714 snap 224 seq_num 714) 2024-09-20T22:50:00.270 INFO:tasks.rados.rados.0.smithi084.stdout:2044: expect (ObjNum 625 snap 186 seq_num 625) 2024-09-20T22:50:00.317 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:50:00.339 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:00.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:00.341 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:50:00.388 INFO:tasks.rados.rados.0.smithi084.stdout:2045: finishing write tid 1 to smithi08477203-44 2024-09-20T22:50:00.388 INFO:tasks.rados.rados.0.smithi084.stdout:2043: done (3 left) 2024-09-20T22:50:00.388 INFO:tasks.rados.rados.0.smithi084.stdout:2044: done (2 left) 2024-09-20T22:50:00.388 INFO:tasks.rados.rados.0.smithi084.stdout:2047: read oid 24 snap -1 2024-09-20T22:50:00.389 INFO:tasks.rados.rados.0.smithi084.stdout:2047: expect (ObjNum 711 snap 220 seq_num 711) 2024-09-20T22:50:00.389 INFO:tasks.rados.rados.0.smithi084.stdout:2048: read oid 45 snap 220 2024-09-20T22:50:00.389 INFO:tasks.rados.rados.0.smithi084.stdout:2048: expect deleted 2024-09-20T22:50:00.389 INFO:tasks.rados.rados.0.smithi084.stdout:2049: snap_remove snap 218 2024-09-20T22:50:00.441 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:50:00.462 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd reweight 7 1 2024-09-20T22:50:00.465 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:00.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:00.465 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:50:00.566 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:50:00.589 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:00.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:00.590 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:50:00.614 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:00 smithi084 ceph-mon[29774]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:50:00.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:00 smithi084 ceph-mon[29774]: Health detail: HEALTH_WARN noscrub flag(s) set 2024-09-20T22:50:00.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:00 smithi084 ceph-mon[29774]: [WRN] OSDMAP_FLAGS: noscrub flag(s) set 2024-09-20T22:50:00.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:00 smithi084 ceph-mon[29774]: osdmap e520: 8 total, 8 up, 8 in 2024-09-20T22:50:00.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:00 smithi084 ceph-mon[40024]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:50:00.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:00 smithi084 ceph-mon[40024]: Health detail: HEALTH_WARN noscrub flag(s) set 2024-09-20T22:50:00.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:00 smithi084 ceph-mon[40024]: [WRN] OSDMAP_FLAGS: noscrub flag(s) set 2024-09-20T22:50:00.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:00 smithi084 ceph-mon[40024]: osdmap e520: 8 total, 8 up, 8 in 2024-09-20T22:50:00.691 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:50:00.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:00.717 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:00.717 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:50:00.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:00 smithi154 ceph-mon[33264]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:50:00.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:00 smithi154 ceph-mon[33264]: Health detail: HEALTH_WARN noscrub flag(s) set 2024-09-20T22:50:00.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:00 smithi154 ceph-mon[33264]: [WRN] OSDMAP_FLAGS: noscrub flag(s) set 2024-09-20T22:50:00.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:00 smithi154 ceph-mon[33264]: osdmap e520: 8 total, 8 up, 8 in 2024-09-20T22:50:00.818 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:50:00.842 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:00.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:00.843 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:50:00.885 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.3 dump_blocked_ops 2024-09-20T22:50:00.943 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:50:00.988 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:00.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:00.989 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:50:01.090 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:50:01.116 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:01.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:01.117 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:50:01.218 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:50:01.243 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:01.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:01.244 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:50:01.344 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:50:01.345 INFO:tasks.rados.rados.0.smithi084.stdout:2049: done (4 left) 2024-09-20T22:50:01.346 INFO:tasks.rados.rados.0.smithi084.stdout:2050: write (excl) oid 33 current snap is 224 2024-09-20T22:50:01.346 INFO:tasks.rados.rados.0.smithi084.stdout:2050: seq_num 716 ranges {433812=415466,1648519=385293} 2024-09-20T22:50:01.350 INFO:tasks.rados.rados.0.smithi084.stdout:2050: writing smithi08477203-33 from 433812 to 849278 tid 1 2024-09-20T22:50:01.359 INFO:tasks.rados.rados.0.smithi084.stdout:2050: writing smithi08477203-33 from 1648519 to 2033812 tid 2 2024-09-20T22:50:01.359 INFO:tasks.rados.rados.0.smithi084.stdout:2051: write (excl) oid 38 current snap is 224 2024-09-20T22:50:01.359 INFO:tasks.rados.rados.0.smithi084.stdout:2046: finishing rollback tid 1 to smithi08477203-6 2024-09-20T22:50:01.359 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 6 v 740 (ObjNum 225 snap 0 seq_num 803184048) dirty dne 2024-09-20T22:50:01.359 INFO:tasks.rados.rados.0.smithi084.stdout:2045: finishing write tid 2 to smithi08477203-44 2024-09-20T22:50:01.359 INFO:tasks.rados.rados.0.smithi084.stdout:2051: seq_num 717 ranges {450619=490715,1590648=459971} 2024-09-20T22:50:01.364 INFO:tasks.rados.rados.0.smithi084.stdout:2051: writing smithi08477203-38 from 450619 to 941334 tid 1 2024-09-20T22:50:01.371 INFO:tasks.rados.rados.0.smithi084.stdout:2051: writing smithi08477203-38 from 1590648 to 2050619 tid 2 2024-09-20T22:50:01.372 INFO:tasks.rados.rados.0.smithi084.stdout:2045: finishing write tid 3 to smithi08477203-44 2024-09-20T22:50:01.373 INFO:tasks.rados.rados.0.smithi084.stdout:2045: finishing write tid 4 to smithi08477203-44 2024-09-20T22:50:01.373 INFO:tasks.rados.rados.0.smithi084.stdout:2045: oid 44 updating version 0 to 1381 2024-09-20T22:50:01.373 INFO:tasks.rados.rados.0.smithi084.stdout:2045: oid 44 version 1381 is already newer than 1380 2024-09-20T22:50:01.373 INFO:tasks.rados.rados.0.smithi084.stdout:2045: oid 44 updating version 1381 to 1382 2024-09-20T22:50:01.373 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 44 v 1382 (ObjNum 715 snap 224 seq_num 715) dirty exists 2024-09-20T22:50:01.373 INFO:tasks.rados.rados.0.smithi084.stdout:2045: left oid 44 (ObjNum 715 snap 224 seq_num 715) 2024-09-20T22:50:01.373 INFO:tasks.rados.rados.0.smithi084.stdout:2047: expect (ObjNum 711 snap 220 seq_num 711) 2024-09-20T22:50:01.378 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:01.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:01.379 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:50:01.394 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:50:01.422 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:50:01.480 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:50:01.502 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:01.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:01.503 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:50:01.512 INFO:tasks.rados.rados.0.smithi084.stdout:2050: finishing write tid 1 to smithi08477203-33 2024-09-20T22:50:01.512 INFO:tasks.rados.rados.0.smithi084.stdout:2050: finishing write tid 2 to smithi08477203-33 2024-09-20T22:50:01.512 INFO:tasks.rados.rados.0.smithi084.stdout:2050: finishing write tid 3 to smithi08477203-33 2024-09-20T22:50:01.512 INFO:tasks.rados.rados.0.smithi084.stdout:2050: finishing write tid 4 to smithi08477203-33 2024-09-20T22:50:01.512 INFO:tasks.rados.rados.0.smithi084.stdout:2050: oid 33 updating version 0 to 732 2024-09-20T22:50:01.512 INFO:tasks.rados.rados.0.smithi084.stdout:2050: oid 33 updating version 732 to 733 2024-09-20T22:50:01.512 INFO:tasks.rados.rados.0.smithi084.stdout:2050: oid 33 version 733 is already newer than 731 2024-09-20T22:50:01.512 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 33 v 733 (ObjNum 716 snap 224 seq_num 716) dirty exists 2024-09-20T22:50:01.513 INFO:tasks.rados.rados.0.smithi084.stdout:2050: left oid 33 (ObjNum 716 snap 224 seq_num 716) 2024-09-20T22:50:01.513 INFO:tasks.rados.rados.0.smithi084.stdout:2051: finishing write tid 1 to smithi08477203-38 2024-09-20T22:50:01.513 INFO:tasks.rados.rados.0.smithi084.stdout:2045: done (5 left) 2024-09-20T22:50:01.513 INFO:tasks.rados.rados.0.smithi084.stdout:2046: done (4 left) 2024-09-20T22:50:01.513 INFO:tasks.rados.rados.0.smithi084.stdout:2047: done (3 left) 2024-09-20T22:50:01.513 INFO:tasks.rados.rados.0.smithi084.stdout:2048: done (2 left) 2024-09-20T22:50:01.513 INFO:tasks.rados.rados.0.smithi084.stdout:2050: done (1 left) 2024-09-20T22:50:01.513 INFO:tasks.rados.rados.0.smithi084.stdout:2052: read oid 49 snap -1 2024-09-20T22:50:01.513 INFO:tasks.rados.rados.0.smithi084.stdout:2052: expect (ObjNum 470 snap 130 seq_num 470) 2024-09-20T22:50:01.513 INFO:tasks.rados.rados.0.smithi084.stdout:2053: copy_from oid 20 from oid 9 current snap is 224 2024-09-20T22:50:01.513 INFO:tasks.rados.rados.0.smithi084.stdout:2051: finishing write tid 2 to smithi08477203-38 2024-09-20T22:50:01.513 INFO:tasks.rados.rados.0.smithi084.stdout:2051: finishing write tid 3 to smithi08477203-38 2024-09-20T22:50:01.513 INFO:tasks.rados.rados.0.smithi084.stdout:2051: finishing write tid 4 to smithi08477203-38 2024-09-20T22:50:01.513 INFO:tasks.rados.rados.0.smithi084.stdout:2051: oid 38 updating version 0 to 690 2024-09-20T22:50:01.514 INFO:tasks.rados.rados.0.smithi084.stdout:2051: oid 38 version 690 is already newer than 689 2024-09-20T22:50:01.514 INFO:tasks.rados.rados.0.smithi084.stdout:2051: oid 38 version 690 is already newer than 688 2024-09-20T22:50:01.514 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 38 v 690 (ObjNum 717 snap 224 seq_num 717) dirty exists 2024-09-20T22:50:01.514 INFO:tasks.rados.rados.0.smithi084.stdout:2051: left oid 38 (ObjNum 717 snap 224 seq_num 717) 2024-09-20T22:50:01.514 INFO:tasks.rados.rados.0.smithi084.stdout:2051: done (2 left) 2024-09-20T22:50:01.514 INFO:tasks.rados.rados.0.smithi084.stdout:2054: copy_from oid 43 from oid 44 current snap is 224 2024-09-20T22:50:01.514 INFO:tasks.rados.rados.0.smithi084.stdout:2055: read oid 46 snap -1 2024-09-20T22:50:01.514 INFO:tasks.rados.rados.0.smithi084.stdout:2055: expect deleted 2024-09-20T22:50:01.514 INFO:tasks.rados.rados.0.smithi084.stdout:2056: read oid 35 snap -1 2024-09-20T22:50:01.514 INFO:tasks.rados.rados.0.smithi084.stdout:2056: expect (ObjNum 631 snap 191 seq_num 631) 2024-09-20T22:50:01.514 INFO:tasks.rados.rados.0.smithi084.stdout:2057: write (excl) oid 32 current snap is 224 2024-09-20T22:50:01.514 INFO:tasks.rados.rados.0.smithi084.stdout:2057: seq_num 718 ranges {467426=565964,1649130=418296} 2024-09-20T22:50:01.517 INFO:tasks.rados.rados.0.smithi084.stdout:2057: writing smithi08477203-32 from 467426 to 1033390 tid 1 2024-09-20T22:50:01.524 INFO:tasks.rados.rados.0.smithi084.stdout:2057: writing smithi08477203-32 from 1649130 to 2067426 tid 2 2024-09-20T22:50:01.524 INFO:tasks.rados.rados.0.smithi084.stdout:2058: snap_create 2024-09-20T22:50:01.524 INFO:tasks.rados.rados.0.smithi084.stdout:2053: finishing copy_from to smithi08477203-20 2024-09-20T22:50:01.524 INFO:tasks.rados.rados.0.smithi084.stdout:2053: got expected ENOENT (src dne) 2024-09-20T22:50:01.524 INFO:tasks.rados.rados.0.smithi084.stdout:2053: finishing copy_from racing read to smithi08477203-20 2024-09-20T22:50:01.524 INFO:tasks.rados.rados.0.smithi084.stdout:2052: expect (ObjNum 470 snap 130 seq_num 470) 2024-09-20T22:50:01.604 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:50:01.614 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:01 smithi084 ceph-mon[29774]: pgmap v681: 158 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 153 active+clean; 514 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 15 MiB/s rd, 2.5 MiB/s wr, 18 op/s 2024-09-20T22:50:01.614 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:01 smithi084 ceph-mon[40024]: pgmap v681: 158 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 153 active+clean; 514 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 15 MiB/s rd, 2.5 MiB/s wr, 18 op/s 2024-09-20T22:50:01.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:01.630 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:01.630 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:50:01.652 INFO:tasks.rados.rados.0.smithi084.stdout:2056: expect (ObjNum 631 snap 191 seq_num 631) 2024-09-20T22:50:01.731 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:50:01.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:01 smithi154 ceph-mon[33264]: pgmap v681: 158 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 153 active+clean; 514 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 15 MiB/s rd, 2.5 MiB/s wr, 18 op/s 2024-09-20T22:50:01.757 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:01.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:01.758 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:50:01.774 INFO:tasks.rados.rados.0.smithi084.stdout:2054: finishing copy_from racing read to smithi08477203-43 2024-09-20T22:50:01.774 INFO:tasks.rados.rados.0.smithi084.stdout:2057: finishing write tid 1 to smithi08477203-32 2024-09-20T22:50:01.774 INFO:tasks.rados.rados.0.smithi084.stdout:2054: finishing copy_from to smithi08477203-43 2024-09-20T22:50:01.774 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 43 v 1382 (ObjNum 715 snap 224 seq_num 715) dirty exists 2024-09-20T22:50:01.774 INFO:tasks.rados.rados.0.smithi084.stdout:2057: finishing write tid 2 to smithi08477203-32 2024-09-20T22:50:01.774 INFO:tasks.rados.rados.0.smithi084.stdout:2057: finishing write tid 3 to smithi08477203-32 2024-09-20T22:50:01.774 INFO:tasks.rados.rados.0.smithi084.stdout:2057: finishing write tid 4 to smithi08477203-32 2024-09-20T22:50:01.774 INFO:tasks.rados.rados.0.smithi084.stdout:2057: oid 32 updating version 0 to 731 2024-09-20T22:50:01.774 INFO:tasks.rados.rados.0.smithi084.stdout:2057: oid 32 version 731 is already newer than 730 2024-09-20T22:50:01.774 INFO:tasks.rados.rados.0.smithi084.stdout:2057: oid 32 version 731 is already newer than 729 2024-09-20T22:50:01.774 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 32 v 731 (ObjNum 718 snap 224 seq_num 718) dirty exists 2024-09-20T22:50:01.774 INFO:tasks.rados.rados.0.smithi084.stdout:2057: left oid 32 (ObjNum 718 snap 224 seq_num 718) 2024-09-20T22:50:01.858 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:50:01.883 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:01.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:01.885 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:50:01.926 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-20T22:50:01.985 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:50:02.006 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:02.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:02.007 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:50:02.108 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:50:02.134 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:02.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:02.135 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:50:02.236 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:50:02.257 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:02.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:02.258 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:50:02.322 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-20T22:50:02.322 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:50:02.353 INFO:tasks.rados.rados.0.smithi084.stdout:2052: done (6 left) 2024-09-20T22:50:02.353 INFO:tasks.rados.rados.0.smithi084.stdout:2053: done (5 left) 2024-09-20T22:50:02.353 INFO:tasks.rados.rados.0.smithi084.stdout:2054: done (4 left) 2024-09-20T22:50:02.353 INFO:tasks.rados.rados.0.smithi084.stdout:2055: done (3 left) 2024-09-20T22:50:02.353 INFO:tasks.rados.rados.0.smithi084.stdout:2056: done (2 left) 2024-09-20T22:50:02.354 INFO:tasks.rados.rados.0.smithi084.stdout:2057: done (1 left) 2024-09-20T22:50:02.354 INFO:tasks.rados.rados.0.smithi084.stdout:2058: done (0 left) 2024-09-20T22:50:02.354 INFO:tasks.rados.rados.0.smithi084.stdout:2059: copy_from oid 24 from oid 5 current snap is 225 2024-09-20T22:50:02.354 INFO:tasks.rados.rados.0.smithi084.stdout:2060: snap_create 2024-09-20T22:50:02.354 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.7 to 1 (10000) 2024-09-20T22:50:02.359 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:50:02.378 INFO:tasks.rados.rados.0.smithi084.stdout:2059: finishing copy_from racing read to smithi08477203-24 2024-09-20T22:50:02.384 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:02.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:02.385 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:50:02.387 INFO:tasks.rados.rados.0.smithi084.stdout:2059: finishing copy_from to smithi08477203-24 2024-09-20T22:50:02.388 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 24 v 720 (ObjNum 670 snap 205 seq_num 670) dirty exists 2024-09-20T22:50:02.486 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:50:02.507 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:02.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:02.508 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:50:02.609 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:50:02.614 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:02 smithi084 ceph-mon[29774]: osdmap e521: 8 total, 8 up, 8 in 2024-09-20T22:50:02.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:02 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/4276345870' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-09-20T22:50:02.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:02 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-09-20T22:50:02.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:02 smithi084 ceph-mon[40024]: osdmap e521: 8 total, 8 up, 8 in 2024-09-20T22:50:02.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:02 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/4276345870' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-09-20T22:50:02.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:02 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-09-20T22:50:02.630 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:02.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:02.631 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:50:02.732 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:50:02.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:02 smithi154 ceph-mon[33264]: osdmap e521: 8 total, 8 up, 8 in 2024-09-20T22:50:02.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:02 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/4276345870' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-09-20T22:50:02.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:02 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-09-20T22:50:02.756 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:02.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:02.757 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:50:02.858 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:50:02.882 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:02.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:02.883 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:50:02.984 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:50:03.008 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:03.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:03.009 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:50:03.110 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:50:03.133 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:03.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:03.134 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:50:03.140 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.3 dump_historic_ops 2024-09-20T22:50:03.235 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:50:03.236 INFO:tasks.rados.rados.0.smithi084.stdout:2059: done (1 left) 2024-09-20T22:50:03.236 INFO:tasks.rados.rados.0.smithi084.stdout:2060: done (0 left) 2024-09-20T22:50:03.237 INFO:tasks.rados.rados.0.smithi084.stdout:2061: write oid 38 current snap is 226 2024-09-20T22:50:03.237 INFO:tasks.rados.rados.0.smithi084.stdout:2061: seq_num 719 ranges {484233=641213,1707612=376621} 2024-09-20T22:50:03.241 INFO:tasks.rados.rados.0.smithi084.stdout:2061: writing smithi08477203-38 from 484233 to 1125446 tid 1 2024-09-20T22:50:03.247 INFO:tasks.rados.rados.0.smithi084.stdout:2061: writing smithi08477203-38 from 1707612 to 2084233 tid 2 2024-09-20T22:50:03.247 INFO:tasks.rados.rados.0.smithi084.stdout:2062: snap_remove snap 212 2024-09-20T22:50:03.260 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:03.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:03.261 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:50:03.328 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-20T22:50:03.361 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:50:03.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:03 smithi084 ceph-mon[29774]: pgmap v684: 158 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 153 active+clean; 514 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 15 MiB/s rd, 2.5 MiB/s wr, 18 op/s 2024-09-20T22:50:03.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:03 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-20T22:50:03.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:03 smithi084 ceph-mon[29774]: osdmap e522: 8 total, 8 up, 8 in 2024-09-20T22:50:03.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:03 smithi084 ceph-mon[29774]: osdmap e523: 8 total, 8 up, 8 in 2024-09-20T22:50:03.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:03 smithi084 ceph-mon[40024]: pgmap v684: 158 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 153 active+clean; 514 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 15 MiB/s rd, 2.5 MiB/s wr, 18 op/s 2024-09-20T22:50:03.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:03 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-20T22:50:03.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:03 smithi084 ceph-mon[40024]: osdmap e522: 8 total, 8 up, 8 in 2024-09-20T22:50:03.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:03 smithi084 ceph-mon[40024]: osdmap e523: 8 total, 8 up, 8 in 2024-09-20T22:50:03.384 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:03.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:03.385 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:50:03.486 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:50:03.509 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:03.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:03.510 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:50:03.611 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:50:03.632 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:03.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:03.633 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:50:03.734 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:50:03.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:03 smithi154 ceph-mon[33264]: pgmap v684: 158 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 153 active+clean; 514 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 15 MiB/s rd, 2.5 MiB/s wr, 18 op/s 2024-09-20T22:50:03.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:03 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-20T22:50:03.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:03 smithi154 ceph-mon[33264]: osdmap e522: 8 total, 8 up, 8 in 2024-09-20T22:50:03.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:03 smithi154 ceph-mon[33264]: osdmap e523: 8 total, 8 up, 8 in 2024-09-20T22:50:03.755 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:03.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:03.756 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:50:03.850 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd reweight 0 1 2024-09-20T22:50:03.857 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:50:03.864 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:50:03.881 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:03.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:03.882 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:50:03.983 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:50:04.007 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:04.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:04.008 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:50:04.109 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:50:04.131 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:04.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:04.133 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:50:04.149 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:50:04.233 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:50:04.239 INFO:tasks.rados.rados.0.smithi084.stdout:2062: done (1 left) 2024-09-20T22:50:04.239 INFO:tasks.rados.rados.0.smithi084.stdout:2063: delete oid 44 current snap is 226 2024-09-20T22:50:04.239 INFO:tasks.rados.rados.0.smithi084.stdout:2061: finishing write tid 1 to smithi08477203-38 2024-09-20T22:50:04.239 INFO:tasks.rados.rados.0.smithi084.stdout:2061: finishing write tid 2 to smithi08477203-38 2024-09-20T22:50:04.239 INFO:tasks.rados.rados.0.smithi084.stdout:2061: finishing write tid 3 to smithi08477203-38 2024-09-20T22:50:04.239 INFO:tasks.rados.rados.0.smithi084.stdout:2061: finishing write tid 4 to smithi08477203-38 2024-09-20T22:50:04.239 INFO:tasks.rados.rados.0.smithi084.stdout:2061: oid 38 updating version 0 to 692 2024-09-20T22:50:04.239 INFO:tasks.rados.rados.0.smithi084.stdout:2061: oid 38 updating version 692 to 693 2024-09-20T22:50:04.239 INFO:tasks.rados.rados.0.smithi084.stdout:2061: oid 38 updating version 693 to 694 2024-09-20T22:50:04.239 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 38 v 694 (ObjNum 719 snap 226 seq_num 719) dirty exists 2024-09-20T22:50:04.240 INFO:tasks.rados.rados.0.smithi084.stdout:2061: left oid 38 (ObjNum 719 snap 226 seq_num 719) 2024-09-20T22:50:04.247 INFO:tasks.rados.rados.0.smithi084.stdout:2061: done (1 left) 2024-09-20T22:50:04.247 INFO:tasks.rados.rados.0.smithi084.stdout:2063: done (0 left) 2024-09-20T22:50:04.247 INFO:tasks.rados.rados.0.smithi084.stdout:2064: delete oid 12 current snap is 226 2024-09-20T22:50:04.248 INFO:tasks.rados.rados.0.smithi084.stdout:2064: done (0 left) 2024-09-20T22:50:04.248 INFO:tasks.rados.rados.0.smithi084.stdout:2065: delete oid 38 current snap is 226 2024-09-20T22:50:04.250 INFO:tasks.rados.rados.0.smithi084.stdout:2065: done (0 left) 2024-09-20T22:50:04.250 INFO:tasks.rados.rados.0.smithi084.stdout:2066: snap_create 2024-09-20T22:50:04.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:04.262 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:04.262 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:50:04.363 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:50:04.385 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:50:04.386 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:50:04.386 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:50:04.386 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:50:04.387 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:04.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:04.388 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:50:04.407 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:50:04.408 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:50:04.408 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:50:04.427 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-20T22:50:04.428 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T22:50:04.444 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:50:04.445 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:50:04.461 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:50:04.461 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:50:04.478 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:50:04.478 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:50:04.489 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:50:04.494 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:50:04.494 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:50:04.512 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:04.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:04.513 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:50:04.614 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:50:04.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:04.637 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:04.637 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:50:04.738 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:50:04.760 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:50:04.762 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:04.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:04.763 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:50:04.864 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:50:04.888 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:04.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:04.889 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:50:04.990 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:50:05.013 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:05.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:05.014 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:50:05.115 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:50:05.138 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:05.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:05.139 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:50:05.236 INFO:tasks.rados.rados.0.smithi084.stdout:2066: done (0 left) 2024-09-20T22:50:05.236 INFO:tasks.rados.rados.0.smithi084.stdout:2067: rollback oid 46 current snap is 227 2024-09-20T22:50:05.236 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 46 to 196 2024-09-20T22:50:05.236 INFO:tasks.rados.rados.0.smithi084.stdout:2068: read oid 32 snap 196 2024-09-20T22:50:05.236 INFO:tasks.rados.rados.0.smithi084.stdout:2068: expect deleted 2024-09-20T22:50:05.236 INFO:tasks.rados.rados.0.smithi084.stdout:2069: rollback oid 5 current snap is 227 2024-09-20T22:50:05.236 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 5 to 205 2024-09-20T22:50:05.236 INFO:tasks.rados.rados.0.smithi084.stdout:2070: rollback oid 23 current snap is 227 2024-09-20T22:50:05.237 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 23 to 219 2024-09-20T22:50:05.237 INFO:tasks.rados.rados.0.smithi084.stdout:2071: snap_create 2024-09-20T22:50:05.239 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:50:05.240 INFO:tasks.rados.rados.0.smithi084.stdout:2069: finishing rollback tid 0 to smithi08477203-5 2024-09-20T22:50:05.249 INFO:tasks.rados.rados.0.smithi084.stdout:2069: finishing rollback tid 1 to smithi08477203-5 2024-09-20T22:50:05.249 INFO:tasks.rados.rados.0.smithi084.stdout:2069: finishing rollback tid 2 to smithi08477203-5 2024-09-20T22:50:05.250 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 5 v 723 (ObjNum 670 snap 205 seq_num 670) dirty exists 2024-09-20T22:50:05.250 INFO:tasks.rados.rados.0.smithi084.stdout:2067: finishing rollback tid 1 to smithi08477203-46 2024-09-20T22:50:05.250 INFO:tasks.rados.rados.0.smithi084.stdout:2067: finishing rollback tid 2 to smithi08477203-46 2024-09-20T22:50:05.250 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 46 v 794 (ObjNum 470 snap 130 seq_num 470) dirty exists 2024-09-20T22:50:05.250 INFO:tasks.rados.rados.0.smithi084.stdout:2070: finishing rollback tid 0 to smithi08477203-23 2024-09-20T22:50:05.250 INFO:tasks.rados.rados.0.smithi084.stdout:2070: finishing rollback tid 1 to smithi08477203-23 2024-09-20T22:50:05.251 INFO:tasks.rados.rados.0.smithi084.stdout:2070: finishing rollback tid 2 to smithi08477203-23 2024-09-20T22:50:05.251 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 23 v 736 (ObjNum 696 snap 215 seq_num 696) dirty exists 2024-09-20T22:50:05.264 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:05.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:05.266 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:50:05.367 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:50:05.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:05.390 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:05.390 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:50:05.490 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:50:05.495 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:05 smithi084 ceph-mon[40024]: pgmap v687: 158 pgs: 2 active+clean+snaptrim, 156 active+clean; 529 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 15 MiB/s rd, 2.7 MiB/s wr, 19 op/s 2024-09-20T22:50:05.495 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:05 smithi084 ceph-mon[40024]: osdmap e524: 8 total, 8 up, 8 in 2024-09-20T22:50:05.495 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:05 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/3292672010' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:50:05.495 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:05 smithi084 ceph-mon[29774]: pgmap v687: 158 pgs: 2 active+clean+snaptrim, 156 active+clean; 529 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 15 MiB/s rd, 2.7 MiB/s wr, 19 op/s 2024-09-20T22:50:05.495 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:05 smithi084 ceph-mon[29774]: osdmap e524: 8 total, 8 up, 8 in 2024-09-20T22:50:05.495 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:05 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/3292672010' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:50:05.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:05 smithi154 ceph-mon[33264]: pgmap v687: 158 pgs: 2 active+clean+snaptrim, 156 active+clean; 529 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 15 MiB/s rd, 2.7 MiB/s wr, 19 op/s 2024-09-20T22:50:05.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:05 smithi154 ceph-mon[33264]: osdmap e524: 8 total, 8 up, 8 in 2024-09-20T22:50:05.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:05 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/3292672010' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:50:05.512 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:05.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:05.513 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:50:05.614 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:50:05.635 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:05.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:05.636 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:50:05.737 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:50:05.758 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:05.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:05.759 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:50:05.860 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:50:05.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:05.882 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:05.882 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:50:05.983 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:50:06.005 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:06.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:06.006 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:50:06.106 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:50:06.127 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:06.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:06.128 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:50:06.132 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-20T22:50:06.132 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2024-09-20T22:50:06.132 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2024-09-20T22:50:06.132 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [ 2024-09-20T22:50:06.132 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.132 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:554 7.a 7:5122292a:::smithi08477203-33:head [copy-from ver 6] snapc 13=[13,12,11,10,f,d] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-09-20T22:50:06.132 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:43:27.336824+0000", 2024-09-20T22:50:06.132 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 398.792307489, 2024-09-20T22:50:06.132 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.056403397000000001, 2024-09-20T22:50:06.133 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:50:06.133 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:50:06.133 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:50:06.133 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:50:06.133 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:50:06.133 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:50:06.133 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 554 2024-09-20T22:50:06.133 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.133 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:50:06.133 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.133 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:50:06.133 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:27.336824+0000", 2024-09-20T22:50:06.133 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:50:06.134 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.134 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.134 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:50:06.134 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:27.336824+0000", 2024-09-20T22:50:06.134 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:50:06.134 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.134 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.134 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:50:06.134 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:27.336826+0000", 2024-09-20T22:50:06.134 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.0339999999999999e-06 2024-09-20T22:50:06.134 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.134 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.134 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:50:06.134 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:27.336837+0000", 2024-09-20T22:50:06.135 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1243000000000001e-05 2024-09-20T22:50:06.135 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.135 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.135 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:50:06.135 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:27.336857+0000", 2024-09-20T22:50:06.135 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.9868999999999999e-05 2024-09-20T22:50:06.135 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.135 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.135 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:50:06.135 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:27.336869+0000", 2024-09-20T22:50:06.135 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1348999999999999e-05 2024-09-20T22:50:06.135 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.136 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.136 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:50:06.136 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:27.336946+0000", 2024-09-20T22:50:06.136 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.7683999999999997e-05 2024-09-20T22:50:06.136 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.136 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.136 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:50:06.136 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:27.337045+0000", 2024-09-20T22:50:06.136 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.9265999999999995e-05 2024-09-20T22:50:06.136 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.136 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.136 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 2,5", 2024-09-20T22:50:06.136 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:27.340603+0000", 2024-09-20T22:50:06.136 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.003557892 2024-09-20T22:50:06.137 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.137 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.137 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:50:06.137 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:27.343571+0000", 2024-09-20T22:50:06.137 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0029676609999999999 2024-09-20T22:50:06.137 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.137 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.137 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:50:06.137 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:27.346457+0000", 2024-09-20T22:50:06.137 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0028860040000000002 2024-09-20T22:50:06.137 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.137 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.137 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:50:06.138 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:27.393131+0000", 2024-09-20T22:50:06.138 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.046673813000000001 2024-09-20T22:50:06.138 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.138 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.138 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:50:06.138 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:27.393173+0000", 2024-09-20T22:50:06.138 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.2076999999999997e-05 2024-09-20T22:50:06.138 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.138 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.138 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:50:06.138 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:27.393227+0000", 2024-09-20T22:50:06.138 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.4505000000000001e-05 2024-09-20T22:50:06.138 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:50:06.139 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:50:06.139 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:50:06.139 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.139 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.139 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:663 7.12 7:4fa4b2b3:::smithi08477203-21:head [write 665577~781581,stat] snapc 17=[17,16,15,11,d] ondisk+write+known_if_redirected+supports_pool_eio e137)", 2024-09-20T22:50:06.139 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:43:36.161568+0000", 2024-09-20T22:50:06.139 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 389.96756331199998, 2024-09-20T22:50:06.139 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.99286350899999998, 2024-09-20T22:50:06.139 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:50:06.139 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:50:06.139 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:50:06.139 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:50:06.139 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:50:06.140 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:50:06.140 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 663 2024-09-20T22:50:06.140 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.140 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:50:06.140 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.141 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:50:06.141 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.161568+0000", 2024-09-20T22:50:06.141 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:50:06.141 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.141 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.141 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:50:06.141 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.161568+0000", 2024-09-20T22:50:06.141 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:50:06.141 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.141 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.141 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:50:06.141 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.161570+0000", 2024-09-20T22:50:06.142 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.0679999999999999e-06 2024-09-20T22:50:06.142 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.142 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.142 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:50:06.142 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.162310+0000", 2024-09-20T22:50:06.142 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00074005999999999996 2024-09-20T22:50:06.142 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.142 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.142 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:50:06.142 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.162349+0000", 2024-09-20T22:50:06.142 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.8778e-05 2024-09-20T22:50:06.142 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.142 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.142 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:50:06.142 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.162375+0000", 2024-09-20T22:50:06.143 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.6089e-05 2024-09-20T22:50:06.143 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.143 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.143 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:50:06.143 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.162434+0000", 2024-09-20T22:50:06.143 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.8764999999999997e-05 2024-09-20T22:50:06.143 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.143 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.143 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-09-20T22:50:06.143 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.162443+0000", 2024-09-20T22:50:06.143 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.054e-06 2024-09-20T22:50:06.143 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.143 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.143 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:50:06.144 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.143416+0000", 2024-09-20T22:50:06.144 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.98097291600000003 2024-09-20T22:50:06.144 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.144 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.144 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:50:06.144 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.143522+0000", 2024-09-20T22:50:06.144 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00010626 2024-09-20T22:50:06.144 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.144 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.144 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 1,2", 2024-09-20T22:50:06.144 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.143654+0000", 2024-09-20T22:50:06.144 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013218799999999999 2024-09-20T22:50:06.144 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.144 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.145 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:50:06.145 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.148086+0000", 2024-09-20T22:50:06.145 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0044314189999999998 2024-09-20T22:50:06.145 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.145 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.145 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:50:06.145 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.153546+0000", 2024-09-20T22:50:06.145 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0054599920000000003 2024-09-20T22:50:06.145 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.145 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.145 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:50:06.145 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.154317+0000", 2024-09-20T22:50:06.145 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00077125999999999996 2024-09-20T22:50:06.145 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.145 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.146 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:50:06.146 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.154380+0000", 2024-09-20T22:50:06.146 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.2526000000000004e-05 2024-09-20T22:50:06.146 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.146 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.146 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:50:06.146 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.154432+0000", 2024-09-20T22:50:06.146 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.2133999999999999e-05 2024-09-20T22:50:06.146 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:50:06.146 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:50:06.146 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:50:06.146 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.146 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.146 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:664 7.12 7:4fa4b2b3:::smithi08477203-21:head [write 2062560~563688] snapc 17=[17,16,15,11,d] ondisk+write+known_if_redirected+supports_pool_eio e137)", 2024-09-20T22:50:06.147 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:43:36.167947+0000", 2024-09-20T22:50:06.147 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 389.96118484200002, 2024-09-20T22:50:06.147 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.98665172599999995, 2024-09-20T22:50:06.147 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:50:06.148 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:50:06.148 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:50:06.148 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:50:06.148 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:50:06.148 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:50:06.148 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 664 2024-09-20T22:50:06.148 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.148 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:50:06.148 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.148 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:50:06.148 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.167947+0000", 2024-09-20T22:50:06.148 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:50:06.148 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.148 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.149 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:50:06.149 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.167947+0000", 2024-09-20T22:50:06.149 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:50:06.149 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.149 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.149 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:50:06.149 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.167948+0000", 2024-09-20T22:50:06.149 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.187e-06 2024-09-20T22:50:06.149 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.149 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.149 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:50:06.149 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.168193+0000", 2024-09-20T22:50:06.149 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00024496199999999999 2024-09-20T22:50:06.149 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.150 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.150 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:50:06.150 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.168203+0000", 2024-09-20T22:50:06.150 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0543999999999999e-05 2024-09-20T22:50:06.150 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.150 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.150 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:50:06.150 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.168209+0000", 2024-09-20T22:50:06.150 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.2839999999999997e-06 2024-09-20T22:50:06.150 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.150 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.150 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:50:06.150 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.168252+0000", 2024-09-20T22:50:06.151 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.3115000000000003e-05 2024-09-20T22:50:06.151 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.151 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.151 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-09-20T22:50:06.151 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.168257+0000", 2024-09-20T22:50:06.151 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.6860000000000003e-06 2024-09-20T22:50:06.151 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.151 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.151 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:50:06.151 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.144795+0000", 2024-09-20T22:50:06.151 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.97653758099999999 2024-09-20T22:50:06.151 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.151 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.151 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:50:06.152 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.144865+0000", 2024-09-20T22:50:06.152 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.9641000000000006e-05 2024-09-20T22:50:06.152 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.152 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.152 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 1,2", 2024-09-20T22:50:06.152 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.144986+0000", 2024-09-20T22:50:06.152 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000121248 2024-09-20T22:50:06.152 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.152 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.152 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:50:06.152 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.148091+0000", 2024-09-20T22:50:06.152 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0031050399999999999 2024-09-20T22:50:06.152 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.152 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.153 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:50:06.153 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.153603+0000", 2024-09-20T22:50:06.153 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0055118479999999997 2024-09-20T22:50:06.153 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.153 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.153 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:50:06.153 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.154508+0000", 2024-09-20T22:50:06.153 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00090530600000000004 2024-09-20T22:50:06.153 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.153 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.153 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:50:06.153 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.154560+0000", 2024-09-20T22:50:06.153 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.1474000000000002e-05 2024-09-20T22:50:06.153 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.154 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.154 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:50:06.154 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.154598+0000", 2024-09-20T22:50:06.154 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.8810000000000003e-05 2024-09-20T22:50:06.154 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:50:06.154 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:50:06.154 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:50:06.154 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.154 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.154 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:665 7.12 7:4fa4b2b3:::smithi08477203-21:head [write 3182305~643664] snapc 17=[17,16,15,11,d] ondisk+write+known_if_redirected+supports_pool_eio e137)", 2024-09-20T22:50:06.154 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:43:36.174520+0000", 2024-09-20T22:50:06.154 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 389.95461175200001, 2024-09-20T22:50:06.154 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.98020083400000002, 2024-09-20T22:50:06.154 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:50:06.155 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:50:06.155 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:50:06.155 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:50:06.155 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:50:06.155 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:50:06.155 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 665 2024-09-20T22:50:06.155 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.155 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:50:06.155 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.155 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:50:06.155 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174520+0000", 2024-09-20T22:50:06.155 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:50:06.155 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.155 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.156 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:50:06.156 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174520+0000", 2024-09-20T22:50:06.156 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:50:06.156 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.156 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.156 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:50:06.156 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174521+0000", 2024-09-20T22:50:06.156 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.234e-06 2024-09-20T22:50:06.156 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.156 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.156 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:50:06.156 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174816+0000", 2024-09-20T22:50:06.156 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000294682 2024-09-20T22:50:06.156 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.157 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.157 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:50:06.157 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174827+0000", 2024-09-20T22:50:06.157 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1025e-05 2024-09-20T22:50:06.157 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.157 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.157 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:50:06.157 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174832+0000", 2024-09-20T22:50:06.157 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.3990000000000003e-06 2024-09-20T22:50:06.157 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.157 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.157 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:50:06.157 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174868+0000", 2024-09-20T22:50:06.157 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.6225e-05 2024-09-20T22:50:06.158 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.158 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.158 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-09-20T22:50:06.158 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174873+0000", 2024-09-20T22:50:06.158 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.848e-06 2024-09-20T22:50:06.158 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.158 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.158 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:50:06.158 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.145844+0000", 2024-09-20T22:50:06.158 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.97097048900000005 2024-09-20T22:50:06.158 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.158 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.158 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:50:06.158 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.145917+0000", 2024-09-20T22:50:06.159 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.3248000000000001e-05 2024-09-20T22:50:06.159 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.159 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.159 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 1,2", 2024-09-20T22:50:06.159 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.146028+0000", 2024-09-20T22:50:06.159 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000110773 2024-09-20T22:50:06.159 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.159 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.159 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:50:06.159 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.148095+0000", 2024-09-20T22:50:06.159 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0020678179999999999 2024-09-20T22:50:06.159 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.159 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.159 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:50:06.159 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.153967+0000", 2024-09-20T22:50:06.160 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0058717350000000003 2024-09-20T22:50:06.160 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.160 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.160 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:50:06.160 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.154633+0000", 2024-09-20T22:50:06.160 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00066600600000000002 2024-09-20T22:50:06.160 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.160 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.160 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:50:06.160 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.154683+0000", 2024-09-20T22:50:06.160 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.9872e-05 2024-09-20T22:50:06.160 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.160 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.160 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:50:06.160 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.154721+0000", 2024-09-20T22:50:06.161 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.748e-05 2024-09-20T22:50:06.161 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:50:06.161 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:50:06.161 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:50:06.161 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.161 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.161 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:666 7.12 7:4fa4b2b3:::smithi08477203-21:head [setxattr _header (57) in=7b,truncate 3865577] snapc 17=[17,16,15,11,d] ondisk+write+known_if_redirected+supports_pool_eio e137)", 2024-09-20T22:50:06.161 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:43:36.174844+0000", 2024-09-20T22:50:06.161 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 389.95428755299997, 2024-09-20T22:50:06.161 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.98002428600000002, 2024-09-20T22:50:06.161 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:50:06.161 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:50:06.161 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:50:06.162 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:50:06.162 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:50:06.162 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:50:06.162 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 666 2024-09-20T22:50:06.162 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.162 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:50:06.162 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.162 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:50:06.162 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174844+0000", 2024-09-20T22:50:06.162 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:50:06.162 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.162 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.163 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:50:06.163 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174844+0000", 2024-09-20T22:50:06.163 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:50:06.163 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.163 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.163 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:50:06.163 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174845+0000", 2024-09-20T22:50:06.163 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.1999999999999999e-07 2024-09-20T22:50:06.163 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.163 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.163 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:50:06.163 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174853+0000", 2024-09-20T22:50:06.163 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.1950000000000003e-06 2024-09-20T22:50:06.163 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.163 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.164 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:50:06.164 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174858+0000", 2024-09-20T22:50:06.164 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.5650000000000001e-06 2024-09-20T22:50:06.164 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.164 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.164 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:50:06.164 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174863+0000", 2024-09-20T22:50:06.164 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.1649999999999997e-06 2024-09-20T22:50:06.164 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.164 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.164 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:50:06.164 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174915+0000", 2024-09-20T22:50:06.164 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.1045999999999999e-05 2024-09-20T22:50:06.164 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.165 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.165 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-09-20T22:50:06.165 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174921+0000", 2024-09-20T22:50:06.165 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.404e-06 2024-09-20T22:50:06.165 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.165 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.165 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:50:06.165 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.146528+0000", 2024-09-20T22:50:06.165 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.97160681199999999 2024-09-20T22:50:06.165 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.165 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.165 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:50:06.165 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.146881+0000", 2024-09-20T22:50:06.165 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00035348200000000001 2024-09-20T22:50:06.166 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.166 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.166 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 1,2", 2024-09-20T22:50:06.166 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.147011+0000", 2024-09-20T22:50:06.166 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013018900000000001 2024-09-20T22:50:06.166 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.166 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.166 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:50:06.166 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.148100+0000", 2024-09-20T22:50:06.166 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010881580000000001 2024-09-20T22:50:06.166 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.166 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.166 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:50:06.167 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.154278+0000", 2024-09-20T22:50:06.167 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0061781730000000003 2024-09-20T22:50:06.167 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.167 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.167 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:50:06.167 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.154764+0000", 2024-09-20T22:50:06.167 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00048589199999999998 2024-09-20T22:50:06.167 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.167 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.167 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:50:06.167 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.154815+0000", 2024-09-20T22:50:06.167 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.1428000000000003e-05 2024-09-20T22:50:06.167 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.167 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.167 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:50:06.167 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.154868+0000", 2024-09-20T22:50:06.168 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.3156999999999999e-05 2024-09-20T22:50:06.168 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:50:06.168 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:50:06.168 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:50:06.168 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.168 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.168 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:667 7.12 7:4fa4b2b3:::smithi08477203-21:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e137)", 2024-09-20T22:50:06.168 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:43:36.174872+0000", 2024-09-20T22:50:06.168 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 389.95425985999998, 2024-09-20T22:50:06.168 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.98024790299999998, 2024-09-20T22:50:06.168 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:50:06.168 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:50:06.168 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-20T22:50:06.168 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:50:06.168 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:50:06.168 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:50:06.169 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 667 2024-09-20T22:50:06.169 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.169 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:50:06.169 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.169 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:50:06.169 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174872+0000", 2024-09-20T22:50:06.169 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:50:06.169 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.169 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.169 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:50:06.169 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174872+0000", 2024-09-20T22:50:06.169 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:50:06.169 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.169 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.169 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:50:06.170 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174872+0000", 2024-09-20T22:50:06.170 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.2199999999999999e-07 2024-09-20T22:50:06.170 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.170 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.170 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:50:06.170 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174874+0000", 2024-09-20T22:50:06.170 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.187e-06 2024-09-20T22:50:06.170 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.170 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.170 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:50:06.170 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174881+0000", 2024-09-20T22:50:06.170 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.674e-06 2024-09-20T22:50:06.170 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.170 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.171 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:50:06.171 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174886+0000", 2024-09-20T22:50:06.171 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.2240000000000003e-06 2024-09-20T22:50:06.171 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.171 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.171 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:50:06.171 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174936+0000", 2024-09-20T22:50:06.171 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.9824999999999998e-05 2024-09-20T22:50:06.171 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.171 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.171 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-09-20T22:50:06.171 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174940+0000", 2024-09-20T22:50:06.171 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.5700000000000001e-06 2024-09-20T22:50:06.171 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.172 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.172 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:50:06.172 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.147523+0000", 2024-09-20T22:50:06.172 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.97258315399999995 2024-09-20T22:50:06.172 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.172 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.172 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for rw locks", 2024-09-20T22:50:06.172 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.147591+0000", 2024-09-20T22:50:06.172 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.7878000000000003e-05 2024-09-20T22:50:06.172 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.172 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.172 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:50:06.172 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.154925+0000", 2024-09-20T22:50:06.172 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0073344430000000004 2024-09-20T22:50:06.173 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.173 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.173 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:50:06.173 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.155020+0000", 2024-09-20T22:50:06.173 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.4767999999999998e-05 2024-09-20T22:50:06.173 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.173 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.173 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:50:06.173 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.155120+0000", 2024-09-20T22:50:06.173 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.9758000000000003e-05 2024-09-20T22:50:06.173 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:50:06.173 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:50:06.173 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:50:06.173 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.174 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.174 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.14478.0:1259 5.3 5:c609908c:::notify.5:head [watch reconnect cookie 94843898679808 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e169)", 2024-09-20T22:50:06.174 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:44:08.230723+0000", 2024-09-20T22:50:06.174 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 357.898408555, 2024-09-20T22:50:06.174 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.94983577399999997, 2024-09-20T22:50:06.174 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:50:06.174 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:50:06.174 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-20T22:50:06.174 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:50:06.174 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.14478", 2024-09-20T22:50:06.174 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/2994273271", 2024-09-20T22:50:06.174 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1259 2024-09-20T22:50:06.174 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.174 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:50:06.175 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.175 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:50:06.175 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:08.230723+0000", 2024-09-20T22:50:06.175 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:50:06.175 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.175 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.175 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:50:06.175 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:08.230723+0000", 2024-09-20T22:50:06.175 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:50:06.175 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.175 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.175 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:50:06.175 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:08.230724+0000", 2024-09-20T22:50:06.175 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.077e-06 2024-09-20T22:50:06.176 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.176 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.176 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:50:06.176 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:08.230729+0000", 2024-09-20T22:50:06.176 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.9930000000000003e-06 2024-09-20T22:50:06.176 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.176 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.176 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:50:06.176 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:08.230741+0000", 2024-09-20T22:50:06.176 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1897e-05 2024-09-20T22:50:06.176 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.177 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.177 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:50:06.177 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:08.230747+0000", 2024-09-20T22:50:06.177 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.9830000000000001e-06 2024-09-20T22:50:06.177 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.177 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.177 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:50:06.177 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:08.230802+0000", 2024-09-20T22:50:06.177 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.4657000000000002e-05 2024-09-20T22:50:06.177 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.177 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.177 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-09-20T22:50:06.177 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:08.230811+0000", 2024-09-20T22:50:06.178 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.1160000000000005e-06 2024-09-20T22:50:06.178 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.178 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.178 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:50:06.178 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:09.171968+0000", 2024-09-20T22:50:06.178 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.94115747299999997 2024-09-20T22:50:06.178 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.178 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.178 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for missing object", 2024-09-20T22:50:06.178 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:09.172098+0000", 2024-09-20T22:50:06.178 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00012937499999999999 2024-09-20T22:50:06.178 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.178 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.178 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:50:06.179 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:09.174248+0000", 2024-09-20T22:50:06.179 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0021508959999999998 2024-09-20T22:50:06.179 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.179 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.179 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for rw locks", 2024-09-20T22:50:06.179 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:09.174341+0000", 2024-09-20T22:50:06.179 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.2906999999999997e-05 2024-09-20T22:50:06.179 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.179 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.179 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:50:06.179 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:09.175083+0000", 2024-09-20T22:50:06.179 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00074120100000000003 2024-09-20T22:50:06.179 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.179 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.180 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:50:06.180 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:09.175149+0000", 2024-09-20T22:50:06.180 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.6877000000000004e-05 2024-09-20T22:50:06.180 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.180 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.180 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:50:06.180 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:09.180559+0000", 2024-09-20T22:50:06.180 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0054093220000000003 2024-09-20T22:50:06.180 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:50:06.180 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:50:06.180 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:50:06.180 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.180 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.180 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:1418 7.1 7:84a866f5:::smithi08477203-27:head [rollback 2f] snapc 2f=[2f] ondisk+write+known_if_redirected+supports_pool_eio e189)", 2024-09-20T22:50:06.181 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:44:28.137587+0000", 2024-09-20T22:50:06.181 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 337.991544358, 2024-09-20T22:50:06.181 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.020550452, 2024-09-20T22:50:06.181 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:50:06.181 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:50:06.181 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:50:06.181 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:50:06.181 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:50:06.181 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:50:06.181 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1418 2024-09-20T22:50:06.181 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.181 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:50:06.182 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.182 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:50:06.182 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.137587+0000", 2024-09-20T22:50:06.182 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:50:06.182 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.182 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.182 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:50:06.182 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.137587+0000", 2024-09-20T22:50:06.182 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:50:06.182 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.182 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.182 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:50:06.182 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.137588+0000", 2024-09-20T22:50:06.182 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.4700000000000001e-07 2024-09-20T22:50:06.183 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.183 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.183 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:50:06.183 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.137591+0000", 2024-09-20T22:50:06.183 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.0000000000000001e-06 2024-09-20T22:50:06.183 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.183 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.183 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:50:06.183 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.137681+0000", 2024-09-20T22:50:06.183 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.0486000000000001e-05 2024-09-20T22:50:06.183 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.183 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.183 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:50:06.183 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.138018+0000", 2024-09-20T22:50:06.184 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00033709899999999999 2024-09-20T22:50:06.184 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.184 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.184 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:50:06.184 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.138109+0000", 2024-09-20T22:50:06.184 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.0538999999999995e-05 2024-09-20T22:50:06.184 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.184 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.184 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting_for_map not empty", 2024-09-20T22:50:06.184 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.138114+0000", 2024-09-20T22:50:06.184 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.6590000000000004e-06 2024-09-20T22:50:06.184 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.184 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.184 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:50:06.184 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.144398+0000", 2024-09-20T22:50:06.185 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0062832599999999997 2024-09-20T22:50:06.185 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.185 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.185 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:50:06.185 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.144483+0000", 2024-09-20T22:50:06.185 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.4772999999999996e-05 2024-09-20T22:50:06.185 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.185 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.185 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 1,6", 2024-09-20T22:50:06.185 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.144661+0000", 2024-09-20T22:50:06.185 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000178736 2024-09-20T22:50:06.185 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.185 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.185 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:50:06.186 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.155602+0000", 2024-09-20T22:50:06.186 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010940302000000001 2024-09-20T22:50:06.186 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.186 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.186 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:50:06.186 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.156162+0000", 2024-09-20T22:50:06.186 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00056014599999999997 2024-09-20T22:50:06.186 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.186 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.186 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:50:06.186 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.158054+0000", 2024-09-20T22:50:06.186 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00189223 2024-09-20T22:50:06.186 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.186 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.187 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:50:06.187 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.158100+0000", 2024-09-20T22:50:06.187 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.6279999999999997e-05 2024-09-20T22:50:06.187 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.187 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.187 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:50:06.187 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.158138+0000", 2024-09-20T22:50:06.187 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.7394999999999997e-05 2024-09-20T22:50:06.187 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:50:06.187 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:50:06.187 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:50:06.187 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.187 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.187 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:1419 7.1 7:84a866f5:::smithi08477203-27:head [append 0~0] snapc 2f=[2f] ondisk+write+known_if_redirected+supports_pool_eio e189)", 2024-09-20T22:50:06.188 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:44:28.138055+0000", 2024-09-20T22:50:06.188 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 337.991076916, 2024-09-20T22:50:06.188 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.030290338999999999, 2024-09-20T22:50:06.188 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:50:06.188 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:50:06.188 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:50:06.188 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:50:06.188 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:50:06.188 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:50:06.188 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1419 2024-09-20T22:50:06.188 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.188 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:50:06.188 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.188 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:50:06.189 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.138055+0000", 2024-09-20T22:50:06.189 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:50:06.189 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.189 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.189 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:50:06.189 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.138055+0000", 2024-09-20T22:50:06.189 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:50:06.189 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.189 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.189 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:50:06.189 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.138056+0000", 2024-09-20T22:50:06.189 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.13e-06 2024-09-20T22:50:06.189 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.189 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.189 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:50:06.190 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.138060+0000", 2024-09-20T22:50:06.190 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.0269999999999997e-06 2024-09-20T22:50:06.190 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.190 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.190 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:50:06.190 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.138070+0000", 2024-09-20T22:50:06.190 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.961e-06 2024-09-20T22:50:06.190 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.190 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.190 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:50:06.190 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.138076+0000", 2024-09-20T22:50:06.190 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.1800000000000001e-06 2024-09-20T22:50:06.190 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.190 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.190 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:50:06.191 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.138137+0000", 2024-09-20T22:50:06.191 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.1159000000000002e-05 2024-09-20T22:50:06.191 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.191 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.191 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting_for_map not empty", 2024-09-20T22:50:06.191 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.138141+0000", 2024-09-20T22:50:06.191 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.189e-06 2024-09-20T22:50:06.191 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.191 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.191 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:50:06.191 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.154982+0000", 2024-09-20T22:50:06.191 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.016840738000000001 2024-09-20T22:50:06.191 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.191 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.191 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:50:06.192 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.155089+0000", 2024-09-20T22:50:06.192 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00010653399999999999 2024-09-20T22:50:06.192 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.192 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.192 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 1,6", 2024-09-20T22:50:06.192 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.155320+0000", 2024-09-20T22:50:06.192 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00023196299999999999 2024-09-20T22:50:06.192 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.192 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.192 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:50:06.192 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.158224+0000", 2024-09-20T22:50:06.192 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00290361 2024-09-20T22:50:06.192 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.192 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.193 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:50:06.193 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.167686+0000", 2024-09-20T22:50:06.193 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0094622360000000006 2024-09-20T22:50:06.193 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.193 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.193 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:50:06.193 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.168183+0000", 2024-09-20T22:50:06.193 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00049655999999999995 2024-09-20T22:50:06.193 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.193 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.193 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:50:06.193 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.168273+0000", 2024-09-20T22:50:06.193 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.9758000000000004e-05 2024-09-20T22:50:06.193 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.193 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.194 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:50:06.194 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.168345+0000", 2024-09-20T22:50:06.194 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.2293999999999996e-05 2024-09-20T22:50:06.194 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:50:06.194 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:50:06.194 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:50:06.194 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.194 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.194 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:2347 7.a 7:5122292a:::smithi08477203-33:head [sparse-read 0~2590699,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e260)", 2024-09-20T22:50:06.194 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:45:39.196440+0000", 2024-09-20T22:50:06.194 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 266.932691788, 2024-09-20T22:50:06.194 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.026701163, 2024-09-20T22:50:06.194 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:50:06.195 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:50:06.195 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-20T22:50:06.195 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:50:06.195 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:50:06.195 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:50:06.195 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 2347 2024-09-20T22:50:06.195 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.195 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:50:06.195 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.195 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:50:06.195 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.196440+0000", 2024-09-20T22:50:06.195 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:50:06.195 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.195 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.196 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:50:06.196 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.196440+0000", 2024-09-20T22:50:06.196 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:50:06.196 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.196 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.196 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:50:06.196 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.196441+0000", 2024-09-20T22:50:06.196 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.714e-06 2024-09-20T22:50:06.196 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.196 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.196 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:50:06.196 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.196449+0000", 2024-09-20T22:50:06.196 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.8730000000000007e-06 2024-09-20T22:50:06.196 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.196 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.197 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:50:06.197 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.196462+0000", 2024-09-20T22:50:06.197 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2649e-05 2024-09-20T22:50:06.197 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.197 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.197 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:50:06.197 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.196470+0000", 2024-09-20T22:50:06.197 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.4989999999999994e-06 2024-09-20T22:50:06.197 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.197 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.197 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:50:06.197 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.212172+0000", 2024-09-20T22:50:06.197 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.015701335 2024-09-20T22:50:06.197 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.197 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.198 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:50:06.198 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.212270+0000", 2024-09-20T22:50:06.198 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.8257999999999993e-05 2024-09-20T22:50:06.198 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.198 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.198 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:50:06.198 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.223141+0000", 2024-09-20T22:50:06.198 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010870835000000001 2024-09-20T22:50:06.198 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:50:06.198 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:50:06.198 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:50:06.198 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.198 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.198 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:2348 7.a 7:5122292a:::smithi08477203-33:head [read 0~3848803,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e260)", 2024-09-20T22:50:06.199 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:45:39.196485+0000", 2024-09-20T22:50:06.199 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 266.932646315, 2024-09-20T22:50:06.199 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0284703, 2024-09-20T22:50:06.199 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:50:06.199 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:50:06.199 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-20T22:50:06.199 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:50:06.199 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:50:06.199 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:50:06.199 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 2348 2024-09-20T22:50:06.199 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.199 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:50:06.199 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.199 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:50:06.199 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.196485+0000", 2024-09-20T22:50:06.199 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:50:06.200 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.200 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.200 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:50:06.200 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.196485+0000", 2024-09-20T22:50:06.200 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:50:06.200 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.200 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.200 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:50:06.200 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.196486+0000", 2024-09-20T22:50:06.200 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.9599999999999999e-07 2024-09-20T22:50:06.200 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.200 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.200 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:50:06.200 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.196489+0000", 2024-09-20T22:50:06.200 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.5470000000000001e-06 2024-09-20T22:50:06.200 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.201 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.201 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:50:06.201 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.196496+0000", 2024-09-20T22:50:06.201 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.3570000000000003e-06 2024-09-20T22:50:06.201 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.201 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.201 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:50:06.201 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.196501+0000", 2024-09-20T22:50:06.201 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.6239999999999999e-06 2024-09-20T22:50:06.201 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.201 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.201 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:50:06.201 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.223176+0000", 2024-09-20T22:50:06.202 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.026675285999999999 2024-09-20T22:50:06.202 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.202 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.202 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:50:06.202 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.223274+0000", 2024-09-20T22:50:06.203 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.7009999999999994e-05 2024-09-20T22:50:06.203 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.203 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.203 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:50:06.203 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.224955+0000", 2024-09-20T22:50:06.203 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00168198 2024-09-20T22:50:06.203 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:50:06.203 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:50:06.203 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:50:06.203 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.203 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.203 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:2349 7.a 7:5122292a:::smithi08477203-33:head [sparse-read 0~2590699] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e260)", 2024-09-20T22:50:06.203 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:45:39.196508+0000", 2024-09-20T22:50:06.203 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 266.93262302900001, 2024-09-20T22:50:06.204 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.028731293000000001, 2024-09-20T22:50:06.204 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:50:06.204 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:50:06.204 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-20T22:50:06.204 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:50:06.204 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:50:06.204 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:50:06.204 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 2349 2024-09-20T22:50:06.204 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.204 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:50:06.204 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.204 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:50:06.204 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.196508+0000", 2024-09-20T22:50:06.204 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:50:06.205 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.205 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.205 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:50:06.205 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.196508+0000", 2024-09-20T22:50:06.205 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:50:06.205 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.205 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.205 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:50:06.205 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.196509+0000", 2024-09-20T22:50:06.205 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.08e-07 2024-09-20T22:50:06.205 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.205 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.205 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:50:06.205 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.196511+0000", 2024-09-20T22:50:06.205 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.142e-06 2024-09-20T22:50:06.205 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.206 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.206 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:50:06.206 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.196517+0000", 2024-09-20T22:50:06.206 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.6330000000000001e-06 2024-09-20T22:50:06.206 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.206 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.206 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:50:06.206 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.196522+0000", 2024-09-20T22:50:06.206 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.3530000000000002e-06 2024-09-20T22:50:06.206 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.206 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.206 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:50:06.206 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.224990+0000", 2024-09-20T22:50:06.206 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.028467864999999998 2024-09-20T22:50:06.207 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.207 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.207 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:50:06.207 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.225088+0000", 2024-09-20T22:50:06.207 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.8281000000000003e-05 2024-09-20T22:50:06.207 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.207 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.207 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:50:06.207 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.225240+0000", 2024-09-20T22:50:06.207 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00015161100000000001 2024-09-20T22:50:06.207 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:50:06.207 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:50:06.207 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:50:06.207 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.208 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.208 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(osd.6.0:727 7.c e260/94 7:318c7512:::smithi08477203-44:4c v 260'428, mlcod=258'424)", 2024-09-20T22:50:06.208 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:45:39.202413+0000", 2024-09-20T22:50:06.208 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 266.92671848999998, 2024-09-20T22:50:06.208 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.022852897, 2024-09-20T22:50:06.208 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:50:06.208 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:50:06.208 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:50:06.208 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:50:06.208 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.208 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:50:06.208 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.202413+0000", 2024-09-20T22:50:06.208 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:50:06.208 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.209 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.209 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:50:06.209 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.202413+0000", 2024-09-20T22:50:06.209 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:50:06.209 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.209 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.209 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:50:06.209 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.202414+0000", 2024-09-20T22:50:06.209 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.9299999999999997e-07 2024-09-20T22:50:06.209 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.209 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.209 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:50:06.209 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.202421+0000", 2024-09-20T22:50:06.210 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.8109999999999997e-06 2024-09-20T22:50:06.210 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.210 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.210 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:50:06.210 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.202433+0000", 2024-09-20T22:50:06.210 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2411e-05 2024-09-20T22:50:06.210 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.210 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.210 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:50:06.210 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.202442+0000", 2024-09-20T22:50:06.210 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.7269999999999996e-06 2024-09-20T22:50:06.211 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.211 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.211 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:50:06.211 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.221764+0000", 2024-09-20T22:50:06.211 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.019322068000000001 2024-09-20T22:50:06.211 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.211 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.211 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:50:06.211 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.221786+0000", 2024-09-20T22:50:06.211 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.2167e-05 2024-09-20T22:50:06.211 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.211 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.211 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:50:06.212 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.225217+0000", 2024-09-20T22:50:06.212 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0034313709999999999 2024-09-20T22:50:06.212 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.212 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.212 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:50:06.212 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.225266+0000", 2024-09-20T22:50:06.212 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.8649000000000001e-05 2024-09-20T22:50:06.212 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:50:06.212 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:50:06.212 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:50:06.212 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.212 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.212 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(osd.6.0:728 7.c e260/94 7:31d0211f:::smithi08477203-25:51 v 260'430, mlcod=258'424)", 2024-09-20T22:50:06.213 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:45:39.202454+0000", 2024-09-20T22:50:06.213 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 266.926677627, 2024-09-20T22:50:06.213 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.022837705, 2024-09-20T22:50:06.213 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:50:06.213 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:50:06.213 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:50:06.213 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:50:06.213 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.213 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:50:06.213 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.202454+0000", 2024-09-20T22:50:06.213 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:50:06.213 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.213 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.213 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:50:06.214 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.202454+0000", 2024-09-20T22:50:06.214 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:50:06.214 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.214 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.214 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:50:06.214 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.202454+0000", 2024-09-20T22:50:06.214 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.9999999999999999e-07 2024-09-20T22:50:06.214 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.214 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.214 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:50:06.214 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.202464+0000", 2024-09-20T22:50:06.214 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.8179999999999999e-06 2024-09-20T22:50:06.214 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.214 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.215 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:50:06.215 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.202469+0000", 2024-09-20T22:50:06.215 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.1089999999999997e-06 2024-09-20T22:50:06.215 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.215 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.215 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:50:06.215 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.202473+0000", 2024-09-20T22:50:06.215 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.0940000000000001e-06 2024-09-20T22:50:06.215 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.215 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.215 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:50:06.215 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.222250+0000", 2024-09-20T22:50:06.215 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.019776353999999999 2024-09-20T22:50:06.216 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.216 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.216 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:50:06.216 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.222268+0000", 2024-09-20T22:50:06.216 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.8342000000000002e-05 2024-09-20T22:50:06.216 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.216 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.216 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:50:06.216 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.225268+0000", 2024-09-20T22:50:06.216 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00300038 2024-09-20T22:50:06.216 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.216 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.216 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:50:06.216 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.225292+0000", 2024-09-20T22:50:06.217 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.3308000000000001e-05 2024-09-20T22:50:06.217 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:50:06.217 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:50:06.217 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:50:06.217 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.217 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.217 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:2525 7.1 7:8422f82d:::smithi08477203-22:head [copy-from ver 174] snapc 5d=[5d,5c,5a,50,47] ondisk+write+known_if_redirected+supports_pool_eio e278)", 2024-09-20T22:50:06.217 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:45:57.170348+0000", 2024-09-20T22:50:06.217 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 248.95878351600001, 2024-09-20T22:50:06.217 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.023333486000000001, 2024-09-20T22:50:06.217 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:50:06.217 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:50:06.217 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:50:06.217 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:50:06.217 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:50:06.218 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:50:06.218 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 2525 2024-09-20T22:50:06.218 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.218 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:50:06.218 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.218 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:50:06.218 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:57.170348+0000", 2024-09-20T22:50:06.218 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:50:06.218 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.218 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.218 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:50:06.218 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:57.170348+0000", 2024-09-20T22:50:06.218 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:50:06.218 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.219 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.219 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:50:06.219 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:57.170350+0000", 2024-09-20T22:50:06.219 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.1509999999999998e-06 2024-09-20T22:50:06.219 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.219 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.219 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:50:06.219 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:57.170361+0000", 2024-09-20T22:50:06.219 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0492999999999999e-05 2024-09-20T22:50:06.219 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.219 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.219 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:50:06.219 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:57.170381+0000", 2024-09-20T22:50:06.219 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.0509999999999998e-05 2024-09-20T22:50:06.219 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.220 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.220 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:50:06.220 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:57.170393+0000", 2024-09-20T22:50:06.220 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2009e-05 2024-09-20T22:50:06.220 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.220 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.220 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:50:06.220 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:57.170465+0000", 2024-09-20T22:50:06.220 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.1927000000000004e-05 2024-09-20T22:50:06.220 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.220 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.220 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op must wait for map", 2024-09-20T22:50:06.220 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:57.170473+0000", 2024-09-20T22:50:06.220 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.1890000000000007e-06 2024-09-20T22:50:06.220 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.220 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.221 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:50:06.221 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:57.172855+0000", 2024-09-20T22:50:06.221 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002382168 2024-09-20T22:50:06.221 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.221 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.221 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:50:06.221 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:57.172967+0000", 2024-09-20T22:50:06.221 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00011108299999999999 2024-09-20T22:50:06.221 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.221 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.221 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 1,6", 2024-09-20T22:50:06.221 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:57.181533+0000", 2024-09-20T22:50:06.221 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0085667499999999997 2024-09-20T22:50:06.221 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.221 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.222 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:50:06.222 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:57.191599+0000", 2024-09-20T22:50:06.222 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010065795000000001 2024-09-20T22:50:06.222 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.222 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.222 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:50:06.222 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:57.193469+0000", 2024-09-20T22:50:06.222 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0018695199999999999 2024-09-20T22:50:06.222 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.222 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.222 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:50:06.222 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:57.193550+0000", 2024-09-20T22:50:06.222 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.1891000000000006e-05 2024-09-20T22:50:06.222 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.223 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.223 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:50:06.223 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:57.193625+0000", 2024-09-20T22:50:06.223 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.4679999999999996e-05 2024-09-20T22:50:06.223 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.223 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.223 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:50:06.223 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:57.193681+0000", 2024-09-20T22:50:06.223 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.6320000000000003e-05 2024-09-20T22:50:06.223 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:50:06.223 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:50:06.223 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:50:06.223 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.223 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.223 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:2853 7.10 7:0f524219:::smithi08477203-50:head [copy-from ver 114] snapc 68=[68,67] ondisk+write+known_if_redirected+supports_pool_eio e303)", 2024-09-20T22:50:06.224 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:46:23.285507+0000", 2024-09-20T22:50:06.224 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 222.84362478700001, 2024-09-20T22:50:06.224 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.023440631, 2024-09-20T22:50:06.224 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:50:06.224 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:50:06.224 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:50:06.224 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:50:06.224 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:50:06.224 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:50:06.224 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 2853 2024-09-20T22:50:06.224 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.224 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:50:06.224 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.224 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:50:06.224 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.285507+0000", 2024-09-20T22:50:06.225 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:50:06.225 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.225 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.225 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:50:06.225 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.285507+0000", 2024-09-20T22:50:06.225 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:50:06.225 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.225 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.225 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:50:06.225 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.285508+0000", 2024-09-20T22:50:06.225 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.491e-06 2024-09-20T22:50:06.225 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.225 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.225 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:50:06.225 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.285514+0000", 2024-09-20T22:50:06.225 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.502e-06 2024-09-20T22:50:06.226 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.226 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.226 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:50:06.226 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.285526+0000", 2024-09-20T22:50:06.226 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2639e-05 2024-09-20T22:50:06.226 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.226 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.226 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:50:06.226 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.285535+0000", 2024-09-20T22:50:06.226 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.4689999999999997e-06 2024-09-20T22:50:06.226 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.226 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.226 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:50:06.226 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.285591+0000", 2024-09-20T22:50:06.226 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.6407999999999999e-05 2024-09-20T22:50:06.227 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.227 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.227 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:50:06.227 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.285681+0000", 2024-09-20T22:50:06.227 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.9845000000000005e-05 2024-09-20T22:50:06.227 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.227 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.227 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 1,2", 2024-09-20T22:50:06.227 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.293719+0000", 2024-09-20T22:50:06.227 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0080383950000000003 2024-09-20T22:50:06.227 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.227 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.227 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:50:06.227 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.307398+0000", 2024-09-20T22:50:06.227 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013678074 2024-09-20T22:50:06.227 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.228 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.228 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:50:06.228 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.308451+0000", 2024-09-20T22:50:06.228 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001052975 2024-09-20T22:50:06.228 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.228 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.228 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:50:06.228 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.308811+0000", 2024-09-20T22:50:06.228 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00036041399999999998 2024-09-20T22:50:06.228 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.228 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.228 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:50:06.228 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.308878+0000", 2024-09-20T22:50:06.228 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.7114999999999997e-05 2024-09-20T22:50:06.228 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.228 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.229 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:50:06.229 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.308947+0000", 2024-09-20T22:50:06.229 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.9303999999999999e-05 2024-09-20T22:50:06.229 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:50:06.229 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:50:06.229 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:50:06.229 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.229 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.229 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:3136 7.a 7:5122292a:::smithi08477203-33:head [copy-from ver 385] snapc 77=[77,76,75,74,73,70,6b] ondisk+write+known_if_redirected+supports_pool_eio e328)", 2024-09-20T22:50:06.229 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:46:48.210350+0000", 2024-09-20T22:50:06.229 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 197.91878165599999, 2024-09-20T22:50:06.229 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.030150626999999999, 2024-09-20T22:50:06.229 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:50:06.229 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:50:06.229 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:50:06.230 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:50:06.230 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:50:06.230 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:50:06.230 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 3136 2024-09-20T22:50:06.230 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.230 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:50:06.230 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.230 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:50:06.230 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.210350+0000", 2024-09-20T22:50:06.230 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:50:06.230 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.230 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.230 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:50:06.230 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.210350+0000", 2024-09-20T22:50:06.230 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:50:06.231 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.231 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.231 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:50:06.231 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.210352+0000", 2024-09-20T22:50:06.231 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.889e-06 2024-09-20T22:50:06.231 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.231 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.231 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:50:06.231 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.210360+0000", 2024-09-20T22:50:06.231 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.8890000000000002e-06 2024-09-20T22:50:06.231 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.231 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.231 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:50:06.231 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.210372+0000", 2024-09-20T22:50:06.231 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2662e-05 2024-09-20T22:50:06.231 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.232 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.232 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:50:06.232 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.210381+0000", 2024-09-20T22:50:06.232 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.8589999999999994e-06 2024-09-20T22:50:06.232 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.232 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.232 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:50:06.232 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.210470+0000", 2024-09-20T22:50:06.232 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.9034000000000002e-05 2024-09-20T22:50:06.232 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.232 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.232 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:50:06.232 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.210568+0000", 2024-09-20T22:50:06.232 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.7919999999999998e-05 2024-09-20T22:50:06.233 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.233 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.233 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 2,5", 2024-09-20T22:50:06.233 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.218491+0000", 2024-09-20T22:50:06.233 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0079229490000000003 2024-09-20T22:50:06.233 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.233 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.233 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:50:06.233 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.224053+0000", 2024-09-20T22:50:06.233 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0055624330000000003 2024-09-20T22:50:06.233 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.233 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.233 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:50:06.233 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.227143+0000", 2024-09-20T22:50:06.233 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0030896360000000002 2024-09-20T22:50:06.234 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.234 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.234 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:50:06.234 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.240355+0000", 2024-09-20T22:50:06.234 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013211624999999999 2024-09-20T22:50:06.234 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.234 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.234 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:50:06.234 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.240439+0000", 2024-09-20T22:50:06.234 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.4009000000000002e-05 2024-09-20T22:50:06.234 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.234 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.234 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:50:06.234 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.240500+0000", 2024-09-20T22:50:06.234 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.1722000000000003e-05 2024-09-20T22:50:06.235 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:50:06.235 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:50:06.235 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:50:06.235 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.235 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.235 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:3235 7.1 7:84a866f5:::smithi08477203-27:head [copy-from ver 114] snapc 7e=[7e,7d,7c,7b,7a,79,78,75,74,73,6b] ondisk+write+known_if_redirected+supports_pool_eio e338)", 2024-09-20T22:50:06.235 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:46:58.180892+0000", 2024-09-20T22:50:06.235 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 187.948239433, 2024-09-20T22:50:06.235 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.044574093000000002, 2024-09-20T22:50:06.235 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:50:06.235 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:50:06.235 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:50:06.235 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:50:06.235 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:50:06.235 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:50:06.236 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 3235 2024-09-20T22:50:06.236 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.236 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:50:06.236 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.236 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:50:06.236 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.180892+0000", 2024-09-20T22:50:06.236 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:50:06.236 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.236 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.236 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:50:06.236 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.180892+0000", 2024-09-20T22:50:06.236 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:50:06.236 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.236 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.236 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:50:06.237 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.180894+0000", 2024-09-20T22:50:06.237 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.1789999999999998e-06 2024-09-20T22:50:06.237 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.237 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.237 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:50:06.237 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.180915+0000", 2024-09-20T22:50:06.237 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.0859000000000001e-05 2024-09-20T22:50:06.237 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.237 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.237 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:50:06.237 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.180937+0000", 2024-09-20T22:50:06.237 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.2357999999999998e-05 2024-09-20T22:50:06.237 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.237 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.237 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:50:06.237 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.180947+0000", 2024-09-20T22:50:06.238 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.8090000000000004e-06 2024-09-20T22:50:06.238 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.238 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.238 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:50:06.238 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.181042+0000", 2024-09-20T22:50:06.238 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.4327999999999995e-05 2024-09-20T22:50:06.238 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.238 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.238 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op must wait for map", 2024-09-20T22:50:06.238 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.181049+0000", 2024-09-20T22:50:06.238 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.1640000000000004e-06 2024-09-20T22:50:06.238 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.238 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.239 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:50:06.239 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.185933+0000", 2024-09-20T22:50:06.239 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.004884645 2024-09-20T22:50:06.239 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.239 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.239 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:50:06.239 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.186177+0000", 2024-09-20T22:50:06.239 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000243107 2024-09-20T22:50:06.239 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.239 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.239 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 1,6", 2024-09-20T22:50:06.239 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.204218+0000", 2024-09-20T22:50:06.239 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.018041805000000001 2024-09-20T22:50:06.239 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.239 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.240 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:50:06.240 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.218275+0000", 2024-09-20T22:50:06.240 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014056528 2024-09-20T22:50:06.240 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.240 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.240 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:50:06.240 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.224283+0000", 2024-09-20T22:50:06.240 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0060084860000000004 2024-09-20T22:50:06.240 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.240 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.240 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:50:06.240 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.225379+0000", 2024-09-20T22:50:06.240 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010960830000000001 2024-09-20T22:50:06.240 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.241 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.241 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:50:06.241 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.225429+0000", 2024-09-20T22:50:06.241 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.9440000000000001e-05 2024-09-20T22:50:06.241 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.241 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.241 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:50:06.241 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.225466+0000", 2024-09-20T22:50:06.241 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.7302000000000003e-05 2024-09-20T22:50:06.241 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:50:06.241 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:50:06.241 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:50:06.241 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.242 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.242 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:3236 7.1 7:84a866f5:::smithi08477203-27:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e338)", 2024-09-20T22:50:06.242 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:46:58.180973+0000", 2024-09-20T22:50:06.242 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 187.94815881100001, 2024-09-20T22:50:06.242 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.028358540000000002, 2024-09-20T22:50:06.242 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:50:06.242 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:50:06.242 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-20T22:50:06.242 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:50:06.242 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:50:06.242 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:50:06.242 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 3236 2024-09-20T22:50:06.242 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.242 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:50:06.243 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.243 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:50:06.243 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.180973+0000", 2024-09-20T22:50:06.243 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:50:06.243 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.243 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.243 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:50:06.243 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.180973+0000", 2024-09-20T22:50:06.243 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:50:06.243 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.243 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.243 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:50:06.243 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.180973+0000", 2024-09-20T22:50:06.243 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.9100000000000004e-07 2024-09-20T22:50:06.243 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.244 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.244 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:50:06.244 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.180976+0000", 2024-09-20T22:50:06.244 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.118e-06 2024-09-20T22:50:06.244 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.244 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.244 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:50:06.244 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.180984+0000", 2024-09-20T22:50:06.244 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.43e-06 2024-09-20T22:50:06.244 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.244 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.244 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:50:06.244 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.180990+0000", 2024-09-20T22:50:06.244 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.1109999999999996e-06 2024-09-20T22:50:06.244 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.244 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.245 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:50:06.245 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.181102+0000", 2024-09-20T22:50:06.245 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000112193 2024-09-20T22:50:06.245 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.245 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.245 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting_for_map not empty", 2024-09-20T22:50:06.245 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.181107+0000", 2024-09-20T22:50:06.245 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.0490000000000003e-06 2024-09-20T22:50:06.245 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.245 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.245 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:50:06.245 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.186302+0000", 2024-09-20T22:50:06.245 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0051950240000000003 2024-09-20T22:50:06.245 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.245 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.245 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for blocked object", 2024-09-20T22:50:06.246 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.186344+0000", 2024-09-20T22:50:06.246 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.2168e-05 2024-09-20T22:50:06.246 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.246 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.246 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:50:06.246 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.209115+0000", 2024-09-20T22:50:06.246 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.022770291000000002 2024-09-20T22:50:06.246 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.246 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.246 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:50:06.246 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.209217+0000", 2024-09-20T22:50:06.246 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000102192 2024-09-20T22:50:06.246 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.246 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.247 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:50:06.247 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.209331+0000", 2024-09-20T22:50:06.247 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000114373 2024-09-20T22:50:06.247 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:50:06.247 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:50:06.247 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:50:06.247 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.247 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.247 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:4534 7.12 7:4b06cbed:::smithi08477203-49:head [copy-from ver 722] snapc bb=[bb,ba,b9,b8,b7,b5,b4,b2,b1,a7,a6,a5,9e,9c,95,87,86] ondisk+write+known_if_redirected+supports_pool_eio e456)", 2024-09-20T22:50:06.247 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:48:56.230481+0000", 2024-09-20T22:50:06.247 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 69.898650764999999, 2024-09-20T22:50:06.247 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.028199295999999999, 2024-09-20T22:50:06.247 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:50:06.247 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:50:06.247 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:50:06.248 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:50:06.248 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:50:06.248 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:50:06.248 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 4534 2024-09-20T22:50:06.248 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.248 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:50:06.248 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.248 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:50:06.248 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:56.230481+0000", 2024-09-20T22:50:06.248 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:50:06.248 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.248 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.248 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:50:06.248 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:56.230481+0000", 2024-09-20T22:50:06.248 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:50:06.248 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.248 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.249 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:50:06.249 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:56.230482+0000", 2024-09-20T22:50:06.249 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.674e-06 2024-09-20T22:50:06.249 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.249 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.249 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:50:06.249 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:56.230497+0000", 2024-09-20T22:50:06.249 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.4178e-05 2024-09-20T22:50:06.249 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.249 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.249 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:50:06.249 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:56.230517+0000", 2024-09-20T22:50:06.249 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.0336e-05 2024-09-20T22:50:06.249 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.249 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.249 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:50:06.250 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:56.230527+0000", 2024-09-20T22:50:06.250 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.6039999999999999e-06 2024-09-20T22:50:06.250 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.250 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.250 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:50:06.250 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:56.230631+0000", 2024-09-20T22:50:06.250 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00010445 2024-09-20T22:50:06.250 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.250 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.250 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op must wait for map", 2024-09-20T22:50:06.250 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:56.230639+0000", 2024-09-20T22:50:06.250 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.7759999999999997e-06 2024-09-20T22:50:06.250 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.251 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.251 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:50:06.251 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:56.237931+0000", 2024-09-20T22:50:06.251 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0072925630000000002 2024-09-20T22:50:06.251 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.251 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.251 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:50:06.251 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:56.238030+0000", 2024-09-20T22:50:06.251 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.8592999999999996e-05 2024-09-20T22:50:06.251 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.251 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.251 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 1,2", 2024-09-20T22:50:06.251 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:56.244691+0000", 2024-09-20T22:50:06.251 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0066614680000000003 2024-09-20T22:50:06.251 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.252 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.252 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:50:06.252 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:56.258211+0000", 2024-09-20T22:50:06.252 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013519721 2024-09-20T22:50:06.252 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.252 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.252 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:50:06.252 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:56.258404+0000", 2024-09-20T22:50:06.252 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00019257000000000001 2024-09-20T22:50:06.252 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.252 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.252 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:50:06.252 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:56.258595+0000", 2024-09-20T22:50:06.252 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000191387 2024-09-20T22:50:06.252 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.253 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.253 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:50:06.253 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:56.258635+0000", 2024-09-20T22:50:06.253 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.9918e-05 2024-09-20T22:50:06.253 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:06.253 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:06.253 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:50:06.253 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:56.258680+0000", 2024-09-20T22:50:06.253 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.5058000000000001e-05 2024-09-20T22:50:06.253 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:50:06.253 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:50:06.253 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:50:06.253 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:50:06.253 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:50:06.253 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-20T22:50:06.254 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:50:06.255 INFO:tasks.rados.rados.0.smithi084.stdout:2067: done (4 left) 2024-09-20T22:50:06.255 INFO:tasks.rados.rados.0.smithi084.stdout:2068: done (3 left) 2024-09-20T22:50:06.255 INFO:tasks.rados.rados.0.smithi084.stdout:2069: done (2 left) 2024-09-20T22:50:06.255 INFO:tasks.rados.rados.0.smithi084.stdout:2070: done (1 left) 2024-09-20T22:50:06.255 INFO:tasks.rados.rados.0.smithi084.stdout:2071: done (0 left) 2024-09-20T22:50:06.255 INFO:tasks.rados.rados.0.smithi084.stdout:2072: read oid 48 snap -1 2024-09-20T22:50:06.255 INFO:tasks.rados.rados.0.smithi084.stdout:2072: expect deleted 2024-09-20T22:50:06.255 INFO:tasks.rados.rados.0.smithi084.stdout:2073: write (excl) oid 5 current snap is 228 2024-09-20T22:50:06.255 INFO:tasks.rados.rados.0.smithi084.stdout:2073: seq_num 723 ranges {551461=658562,1941540=209921} 2024-09-20T22:50:06.255 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is set 2024-09-20T22:50:06.256 INFO:tasks.rados.rados.0.smithi084.stdout:2073: writing smithi08477203-5 from 551461 to 1210023 tid 1 2024-09-20T22:50:06.256 INFO:tasks.rados.rados.0.smithi084.stdout:2073: writing smithi08477203-5 from 1941540 to 2151461 tid 2 2024-09-20T22:50:06.257 INFO:tasks.rados.rados.0.smithi084.stdout:2074: delete oid 41 current snap is 228 2024-09-20T22:50:06.257 INFO:tasks.rados.rados.0.smithi084.stdout:2073: finishing write tid 1 to smithi08477203-5 2024-09-20T22:50:06.257 INFO:tasks.rados.rados.0.smithi084.stdout:2072: done (2 left) 2024-09-20T22:50:06.257 INFO:tasks.rados.rados.0.smithi084.stdout:2074: done (1 left) 2024-09-20T22:50:06.257 INFO:tasks.rados.rados.0.smithi084.stdout:2075: read oid 2 snap 184 2024-09-20T22:50:06.257 INFO:tasks.rados.rados.0.smithi084.stdout:2075: expect (ObjNum 609 snap 182 seq_num 609) 2024-09-20T22:50:06.257 INFO:tasks.rados.rados.0.smithi084.stdout:2076: copy_from oid 4 from oid 14 current snap is 228 2024-09-20T22:50:06.257 INFO:tasks.rados.rados.0.smithi084.stdout:2077: write (excl) oid 20 current snap is 228 2024-09-20T22:50:06.257 INFO:tasks.rados.rados.0.smithi084.stdout:2073: finishing write tid 2 to smithi08477203-5 2024-09-20T22:50:06.257 INFO:tasks.rados.rados.0.smithi084.stdout:2077: seq_num 725 ranges {585075=409060,1542151=642924} 2024-09-20T22:50:06.259 INFO:tasks.rados.rados.0.smithi084.stdout:2077: writing smithi08477203-20 from 585075 to 994135 tid 1 2024-09-20T22:50:06.266 INFO:tasks.rados.rados.0.smithi084.stdout:2077: writing smithi08477203-20 from 1542151 to 2185075 tid 2 2024-09-20T22:50:06.266 INFO:tasks.rados.rados.0.smithi084.stdout:2078: snap_remove snap 220 2024-09-20T22:50:06.282 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:06.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:06.283 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:50:06.384 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:50:06.404 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:06.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:06.405 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:50:06.504 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:06 smithi084 ceph-mon[40024]: osdmap e525: 8 total, 8 up, 8 in 2024-09-20T22:50:06.504 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:06 smithi084 ceph-mon[40024]: purged_snaps scrub starts 2024-09-20T22:50:06.504 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:06 smithi084 ceph-mon[40024]: purged_snaps scrub ok 2024-09-20T22:50:06.504 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:06 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/618493652' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:50:06.504 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:06 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:50:06.504 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:06 smithi084 ceph-mon[40024]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:50:06.504 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:06 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T22:50:06.504 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:06 smithi084 ceph-mon[29774]: osdmap e525: 8 total, 8 up, 8 in 2024-09-20T22:50:06.504 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:06 smithi084 ceph-mon[29774]: purged_snaps scrub starts 2024-09-20T22:50:06.505 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:06 smithi084 ceph-mon[29774]: purged_snaps scrub ok 2024-09-20T22:50:06.505 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:06 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/618493652' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:50:06.505 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:06 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:50:06.505 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:06 smithi084 ceph-mon[29774]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:50:06.505 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:06 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T22:50:06.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:06 smithi154 ceph-mon[33264]: osdmap e525: 8 total, 8 up, 8 in 2024-09-20T22:50:06.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:06 smithi154 ceph-mon[33264]: purged_snaps scrub starts 2024-09-20T22:50:06.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:06 smithi154 ceph-mon[33264]: purged_snaps scrub ok 2024-09-20T22:50:06.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:06 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/618493652' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:50:06.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:06 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:50:06.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:06 smithi154 ceph-mon[33264]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:50:06.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:06 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T22:50:06.506 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:50:06.527 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:06.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:06.528 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:50:06.628 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:50:06.649 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:06.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:06.650 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:50:06.751 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:50:06.771 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:06.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:06.772 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:50:06.872 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:50:06.893 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:06.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:06.894 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:50:06.995 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:50:07.017 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:07.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:07.018 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:50:07.119 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:50:07.140 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:07.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:07.141 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:50:07.242 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:50:07.247 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.0 to 1 (10000) 2024-09-20T22:50:07.252 INFO:tasks.rados.rados.0.smithi084.stdout:2078: done (4 left) 2024-09-20T22:50:07.252 INFO:tasks.rados.rados.0.smithi084.stdout:2079: rollback oid 32 current snap is 228 2024-09-20T22:50:07.252 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 32 to 156 2024-09-20T22:50:07.252 INFO:tasks.rados.rados.0.smithi084.stdout:2073: finishing write tid 3 to smithi08477203-5 2024-09-20T22:50:07.252 INFO:tasks.rados.rados.0.smithi084.stdout:2073: finishing write tid 4 to smithi08477203-5 2024-09-20T22:50:07.253 INFO:tasks.rados.rados.0.smithi084.stdout:2073: oid 5 updating version 0 to 726 2024-09-20T22:50:07.253 INFO:tasks.rados.rados.0.smithi084.stdout:2073: oid 5 version 726 is already newer than 724 2024-09-20T22:50:07.253 INFO:tasks.rados.rados.0.smithi084.stdout:2073: oid 5 version 726 is already newer than 725 2024-09-20T22:50:07.253 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 5 v 726 (ObjNum 723 snap 228 seq_num 723) dirty exists 2024-09-20T22:50:07.253 INFO:tasks.rados.rados.0.smithi084.stdout:2073: left oid 5 (ObjNum 723 snap 228 seq_num 723) 2024-09-20T22:50:07.253 INFO:tasks.rados.rados.0.smithi084.stdout:2076: finishing copy_from to smithi08477203-4 2024-09-20T22:50:07.253 INFO:tasks.rados.rados.0.smithi084.stdout:2076: got expected ENOENT (src dne) 2024-09-20T22:50:07.253 INFO:tasks.rados.rados.0.smithi084.stdout:2076: finishing copy_from racing read to smithi08477203-4 2024-09-20T22:50:07.253 INFO:tasks.rados.rados.0.smithi084.stdout:2075: expect (ObjNum 609 snap 182 seq_num 609) 2024-09-20T22:50:07.270 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:07.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:07.271 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:50:07.277 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-20T22:50:07.332 INFO:tasks.rados.rados.0.smithi084.stdout:2077: finishing write tid 1 to smithi08477203-20 2024-09-20T22:50:07.332 INFO:tasks.rados.rados.0.smithi084.stdout:2077: finishing write tid 2 to smithi08477203-20 2024-09-20T22:50:07.332 INFO:tasks.rados.rados.0.smithi084.stdout:2073: done (4 left) 2024-09-20T22:50:07.333 INFO:tasks.rados.rados.0.smithi084.stdout:2075: done (3 left) 2024-09-20T22:50:07.333 INFO:tasks.rados.rados.0.smithi084.stdout:2076: done (2 left) 2024-09-20T22:50:07.333 INFO:tasks.rados.rados.0.smithi084.stdout:2080: snap_remove snap 219 2024-09-20T22:50:07.371 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:50:07.392 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:07.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:07.393 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:50:07.493 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:50:07.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:07 smithi154 ceph-mon[33264]: pgmap v690: 158 pgs: 2 active+clean+snaptrim, 156 active+clean; 529 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 15 MiB/s rd, 2.7 MiB/s wr, 19 op/s 2024-09-20T22:50:07.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:07 smithi154 ceph-mon[33264]: osdmap e526: 8 total, 8 up, 8 in 2024-09-20T22:50:07.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:07 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/4176294002' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-09-20T22:50:07.514 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:07.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:07.515 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:50:07.614 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:07 smithi084 ceph-mon[29774]: pgmap v690: 158 pgs: 2 active+clean+snaptrim, 156 active+clean; 529 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 15 MiB/s rd, 2.7 MiB/s wr, 19 op/s 2024-09-20T22:50:07.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:07 smithi084 ceph-mon[29774]: osdmap e526: 8 total, 8 up, 8 in 2024-09-20T22:50:07.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:07 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/4176294002' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-09-20T22:50:07.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:07 smithi084 ceph-mon[40024]: pgmap v690: 158 pgs: 2 active+clean+snaptrim, 156 active+clean; 529 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 15 MiB/s rd, 2.7 MiB/s wr, 19 op/s 2024-09-20T22:50:07.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:07 smithi084 ceph-mon[40024]: osdmap e526: 8 total, 8 up, 8 in 2024-09-20T22:50:07.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:07 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/4176294002' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-09-20T22:50:07.616 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:50:07.637 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:07.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:07.638 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:50:07.739 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:50:07.765 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:07.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:07.766 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:50:07.867 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:50:07.889 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:07.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:07.890 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:50:07.990 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:50:08.011 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:08.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:08.012 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:50:08.113 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:50:08.134 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:08.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:08.135 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:50:08.191 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-20T22:50:08.235 INFO:tasks.rados.rados.0.smithi084.stdout:2080: done (2 left) 2024-09-20T22:50:08.235 INFO:tasks.rados.rados.0.smithi084.stdout:2081: snap_remove snap 223 2024-09-20T22:50:08.236 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:50:08.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:08.267 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:08.267 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:50:08.368 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:50:08.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:08.389 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:08.390 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:50:08.464 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:50:08.469 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:50:08 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: ::ffff:172.21.15.154 - - [20/Sep/2024:22:50:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-20T22:50:08.470 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:08 smithi084 ceph-mon[40024]: purged_snaps scrub starts 2024-09-20T22:50:08.470 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:08 smithi084 ceph-mon[40024]: purged_snaps scrub ok 2024-09-20T22:50:08.470 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:08 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/4176294002' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-20T22:50:08.470 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:08 smithi084 ceph-mon[40024]: osdmap e527: 8 total, 8 up, 8 in 2024-09-20T22:50:08.470 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:08 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:50:08.470 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:08 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:50:08.470 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:08 smithi084 ceph-mon[40024]: osdmap e528: 8 total, 8 up, 8 in 2024-09-20T22:50:08.470 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:08 smithi084 ceph-mon[29774]: purged_snaps scrub starts 2024-09-20T22:50:08.470 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:08 smithi084 ceph-mon[29774]: purged_snaps scrub ok 2024-09-20T22:50:08.470 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:08 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/4176294002' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-20T22:50:08.471 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:08 smithi084 ceph-mon[29774]: osdmap e527: 8 total, 8 up, 8 in 2024-09-20T22:50:08.471 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:08 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:50:08.471 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:08 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:50:08.471 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:08 smithi084 ceph-mon[29774]: osdmap e528: 8 total, 8 up, 8 in 2024-09-20T22:50:08.490 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:50:08.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:08 smithi154 ceph-mon[33264]: purged_snaps scrub starts 2024-09-20T22:50:08.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:08 smithi154 ceph-mon[33264]: purged_snaps scrub ok 2024-09-20T22:50:08.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:08 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/4176294002' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-20T22:50:08.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:08 smithi154 ceph-mon[33264]: osdmap e527: 8 total, 8 up, 8 in 2024-09-20T22:50:08.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:08 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:50:08.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:08 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:50:08.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:08 smithi154 ceph-mon[33264]: osdmap e528: 8 total, 8 up, 8 in 2024-09-20T22:50:08.511 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:08.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:08.512 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:50:08.613 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:50:08.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:08.635 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:08.635 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:50:08.736 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:50:08.757 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:08.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:08.758 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:50:08.859 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:50:08.880 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:08.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:08.882 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:50:08.966 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-20T22:50:08.966 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [], 2024-09-20T22:50:08.967 INFO:teuthology.orchestra.run.smithi154.stdout: "num_ops": 0 2024-09-20T22:50:08.967 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-20T22:50:08.982 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:50:09.003 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:09.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:09.003 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:50:09.030 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd reweight 4 1 2024-09-20T22:50:09.104 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:50:09.127 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:09.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:09.128 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:50:09.162 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:50:09.229 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:50:09.245 INFO:tasks.rados.rados.0.smithi084.stdout:2081: done (2 left) 2024-09-20T22:50:09.245 INFO:tasks.rados.rados.0.smithi084.stdout:2082: copy_from oid 50 from oid 37 current snap is 228 2024-09-20T22:50:09.245 INFO:tasks.rados.rados.0.smithi084.stdout:2083: write oid 24 current snap is 228 2024-09-20T22:50:09.246 INFO:tasks.rados.rados.0.smithi084.stdout:2083: seq_num 726 ranges {601882=484309,1600633=524904} 2024-09-20T22:50:09.252 INFO:tasks.rados.rados.0.smithi084.stdout:2083: writing smithi08477203-24 from 601882 to 1086191 tid 1 2024-09-20T22:50:09.254 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:09.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:09.255 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:50:09.259 INFO:tasks.rados.rados.0.smithi084.stdout:2083: writing smithi08477203-24 from 1600633 to 2125537 tid 2 2024-09-20T22:50:09.259 INFO:tasks.rados.rados.0.smithi084.stdout:2084: read oid 39 snap -1 2024-09-20T22:50:09.259 INFO:tasks.rados.rados.0.smithi084.stdout:2084: expect deleted 2024-09-20T22:50:09.259 INFO:tasks.rados.rados.0.smithi084.stdout:2077: finishing write tid 3 to smithi08477203-20 2024-09-20T22:50:09.259 INFO:tasks.rados.rados.0.smithi084.stdout:2077: finishing write tid 4 to smithi08477203-20 2024-09-20T22:50:09.259 INFO:tasks.rados.rados.0.smithi084.stdout:2077: oid 20 updating version 0 to 1218 2024-09-20T22:50:09.259 INFO:tasks.rados.rados.0.smithi084.stdout:2077: oid 20 version 1218 is already newer than 1216 2024-09-20T22:50:09.260 INFO:tasks.rados.rados.0.smithi084.stdout:2077: oid 20 version 1218 is already newer than 1217 2024-09-20T22:50:09.260 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 20 v 1218 (ObjNum 725 snap 228 seq_num 725) dirty exists 2024-09-20T22:50:09.260 INFO:tasks.rados.rados.0.smithi084.stdout:2077: left oid 20 (ObjNum 725 snap 228 seq_num 725) 2024-09-20T22:50:09.260 INFO:tasks.rados.rados.0.smithi084.stdout:2079: finishing rollback tid 0 to smithi08477203-32 2024-09-20T22:50:09.260 INFO:tasks.rados.rados.0.smithi084.stdout:2079: finishing rollback tid 1 to smithi08477203-32 2024-09-20T22:50:09.260 INFO:tasks.rados.rados.0.smithi084.stdout:2079: finishing rollback tid 2 to smithi08477203-32 2024-09-20T22:50:09.260 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 32 v 734 (ObjNum 466 snap 128 seq_num 466) dirty exists 2024-09-20T22:50:09.260 INFO:tasks.rados.rados.0.smithi084.stdout:2082: finishing copy_from to smithi08477203-50 2024-09-20T22:50:09.260 INFO:tasks.rados.rados.0.smithi084.stdout:2082: got expected ENOENT (src dne) 2024-09-20T22:50:09.260 INFO:tasks.rados.rados.0.smithi084.stdout:2082: finishing copy_from racing read to smithi08477203-50 2024-09-20T22:50:09.260 INFO:tasks.rados.rados.0.smithi084.stdout:2083: finishing write tid 1 to smithi08477203-24 2024-09-20T22:50:09.260 INFO:tasks.rados.rados.0.smithi084.stdout:2077: done (4 left) 2024-09-20T22:50:09.260 INFO:tasks.rados.rados.0.smithi084.stdout:2079: done (3 left) 2024-09-20T22:50:09.260 INFO:tasks.rados.rados.0.smithi084.stdout:2082: done (2 left) 2024-09-20T22:50:09.261 INFO:tasks.rados.rados.0.smithi084.stdout:2085: delete oid 33 current snap is 228 2024-09-20T22:50:09.264 INFO:tasks.rados.rados.0.smithi084.stdout:2083: finishing write tid 2 to smithi08477203-24 2024-09-20T22:50:09.264 INFO:tasks.rados.rados.0.smithi084.stdout:2083: finishing write tid 3 to smithi08477203-24 2024-09-20T22:50:09.264 INFO:tasks.rados.rados.0.smithi084.stdout:2084: done (2 left) 2024-09-20T22:50:09.264 INFO:tasks.rados.rados.0.smithi084.stdout:2085: done (1 left) 2024-09-20T22:50:09.264 INFO:tasks.rados.rados.0.smithi084.stdout:2086: snap_create 2024-09-20T22:50:09.264 INFO:tasks.rados.rados.0.smithi084.stdout:2083: finishing write tid 4 to smithi08477203-24 2024-09-20T22:50:09.264 INFO:tasks.rados.rados.0.smithi084.stdout:2083: oid 24 updating version 0 to 721 2024-09-20T22:50:09.264 INFO:tasks.rados.rados.0.smithi084.stdout:2083: oid 24 updating version 721 to 722 2024-09-20T22:50:09.264 INFO:tasks.rados.rados.0.smithi084.stdout:2083: oid 24 updating version 722 to 723 2024-09-20T22:50:09.264 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 24 v 723 (ObjNum 726 snap 228 seq_num 726) dirty exists 2024-09-20T22:50:09.265 INFO:tasks.rados.rados.0.smithi084.stdout:2083: left oid 24 (ObjNum 726 snap 228 seq_num 726) 2024-09-20T22:50:09.356 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:50:09.378 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:09.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:09.379 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:50:09.480 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:50:09.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:09.501 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:09.502 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:50:09.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:09 smithi154 ceph-mon[33264]: pgmap v693: 158 pgs: 2 active+clean+snaptrim, 156 active+clean; 529 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-20T22:50:09.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:09 smithi154 ceph-mon[33264]: osdmap e529: 8 total, 8 up, 8 in 2024-09-20T22:50:09.504 INFO:journalctl@ceph.iscsi.iscsi.a.smithi154.stdout:Sep 20 22:50:09 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-iscsi-iscsi-a[59730]: debug there is no tcmu-runner data available 2024-09-20T22:50:09.575 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:09 smithi084 ceph-mon[40024]: pgmap v693: 158 pgs: 2 active+clean+snaptrim, 156 active+clean; 529 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-20T22:50:09.575 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:09 smithi084 ceph-mon[40024]: osdmap e529: 8 total, 8 up, 8 in 2024-09-20T22:50:09.576 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:09 smithi084 ceph-mon[29774]: pgmap v693: 158 pgs: 2 active+clean+snaptrim, 156 active+clean; 529 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-20T22:50:09.576 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:09 smithi084 ceph-mon[29774]: osdmap e529: 8 total, 8 up, 8 in 2024-09-20T22:50:09.602 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:50:09.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:09.625 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:09.625 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:50:09.675 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.4 dump_blocked_ops 2024-09-20T22:50:09.691 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:50:09.691 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:50:09.692 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:50:09.693 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:50:09.710 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:50:09.710 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:50:09.726 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:50:09.727 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-20T22:50:09.727 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T22:50:09.727 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:50:09.741 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd unset noscrub 2024-09-20T22:50:09.746 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:50:09.746 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:50:09.747 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:09.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:09.748 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:50:09.766 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:50:09.766 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:50:09.783 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:50:09.783 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:50:09.799 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:50:09.800 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:50:09.849 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:50:09.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:09.872 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:09.872 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:50:09.936 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:50:09.973 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:50:09.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:09.998 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:09.999 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:50:10.100 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:50:10.121 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:10.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:10.123 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:50:10.224 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:50:10.246 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:10.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:10.247 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:50:10.281 INFO:tasks.rados.rados.0.smithi084.stdout:2083: done (1 left) 2024-09-20T22:50:10.281 INFO:tasks.rados.rados.0.smithi084.stdout:2086: done (0 left) 2024-09-20T22:50:10.282 INFO:tasks.rados.rados.0.smithi084.stdout:2087: snap_remove snap 215 2024-09-20T22:50:10.348 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:50:10.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:10.373 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:10.373 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:50:10.474 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:50:10.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:10.496 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:10.496 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:50:10.540 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-20T22:50:10.540 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:50:10.541 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:10 smithi154 ceph-mon[33264]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:50:10.541 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:10 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/2339905456' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:50:10.597 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:50:10.614 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:10 smithi084 ceph-mon[29774]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:50:10.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:10 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/2339905456' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:50:10.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:10 smithi084 ceph-mon[40024]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:50:10.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:10 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/2339905456' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:50:10.619 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:10.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:10.620 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:50:10.721 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:50:10.742 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:10.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:10.743 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:50:10.844 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:50:10.866 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:10.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:10.867 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:50:10.968 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:50:10.990 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:10.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:10.991 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:50:11.013 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:50:11.092 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:50:11.115 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:11.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:11.116 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:50:11.131 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.4 dump_historic_ops 2024-09-20T22:50:11.217 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:50:11.238 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:11.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:11.239 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:50:11.278 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.4 to 1 (10000) 2024-09-20T22:50:11.282 INFO:tasks.rados.rados.0.smithi084.stdout:2087: done (0 left) 2024-09-20T22:50:11.283 INFO:tasks.rados.rados.0.smithi084.stdout:2088: snap_create 2024-09-20T22:50:11.340 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:50:11.363 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:11.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:11.364 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:50:11.431 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:50:11.465 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:50:11.477 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:11 smithi084 ceph-mon[40024]: pgmap v696: 158 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 153 active+clean; 538 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 1.5 MiB/s rd, 469 KiB/s wr, 9 op/s 2024-09-20T22:50:11.478 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:11 smithi084 ceph-mon[40024]: osdmap e530: 8 total, 8 up, 8 in 2024-09-20T22:50:11.478 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:11 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/823109332' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-09-20T22:50:11.478 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:11 smithi084 ceph-mon[29774]: pgmap v696: 158 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 153 active+clean; 538 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 1.5 MiB/s rd, 469 KiB/s wr, 9 op/s 2024-09-20T22:50:11.478 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:11 smithi084 ceph-mon[29774]: osdmap e530: 8 total, 8 up, 8 in 2024-09-20T22:50:11.478 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:11 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/823109332' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-09-20T22:50:11.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:11.488 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:11.489 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:50:11.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:11 smithi154 ceph-mon[33264]: pgmap v696: 158 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 153 active+clean; 538 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 1.5 MiB/s rd, 469 KiB/s wr, 9 op/s 2024-09-20T22:50:11.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:11 smithi154 ceph-mon[33264]: osdmap e530: 8 total, 8 up, 8 in 2024-09-20T22:50:11.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:11 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/823109332' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-09-20T22:50:11.590 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:50:11.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:11.613 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:11.613 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:50:11.714 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:50:11.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:11.735 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:11.735 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:50:11.836 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:50:11.857 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:11.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:11.858 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:50:11.933 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-20T22:50:11.933 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:50:11.959 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:50:11.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:11.981 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:11.981 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:50:12.082 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:50:12.104 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:12.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:12.105 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:50:12.206 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:50:12.227 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:12.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:12.228 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:50:12.253 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-09-20T22:50:12.253 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph pg dump --format=json 2024-09-20T22:50:12.301 INFO:tasks.rados.rados.0.smithi084.stdout:2088: done (0 left) 2024-09-20T22:50:12.301 INFO:tasks.rados.rados.0.smithi084.stdout:2089: write oid 7 current snap is 230 2024-09-20T22:50:12.301 INFO:tasks.rados.rados.0.smithi084.stdout:2089: seq_num 728 ranges {635496=634807,1717597=517899} 2024-09-20T22:50:12.301 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is unset 2024-09-20T22:50:12.308 INFO:tasks.rados.rados.0.smithi084.stdout:2089: writing smithi08477203-7 from 635496 to 1270303 tid 1 2024-09-20T22:50:12.314 INFO:tasks.rados.rados.0.smithi084.stdout:2089: writing smithi08477203-7 from 1717597 to 2235496 tid 2 2024-09-20T22:50:12.314 INFO:tasks.rados.rados.0.smithi084.stdout:2090: read oid 26 snap -1 2024-09-20T22:50:12.314 INFO:tasks.rados.rados.0.smithi084.stdout:2090: expect deleted 2024-09-20T22:50:12.314 INFO:tasks.rados.rados.0.smithi084.stdout:2091: snap_create 2024-09-20T22:50:12.317 INFO:tasks.rados.rados.0.smithi084.stdout:2089: finishing write tid 1 to smithi08477203-7 2024-09-20T22:50:12.318 INFO:tasks.rados.rados.0.smithi084.stdout:2089: finishing write tid 2 to smithi08477203-7 2024-09-20T22:50:12.318 INFO:tasks.rados.rados.0.smithi084.stdout:2089: finishing write tid 3 to smithi08477203-7 2024-09-20T22:50:12.320 INFO:tasks.rados.rados.0.smithi084.stdout:2089: finishing write tid 4 to smithi08477203-7 2024-09-20T22:50:12.320 INFO:tasks.rados.rados.0.smithi084.stdout:2089: oid 7 updating version 0 to 851 2024-09-20T22:50:12.320 INFO:tasks.rados.rados.0.smithi084.stdout:2089: oid 7 updating version 851 to 852 2024-09-20T22:50:12.320 INFO:tasks.rados.rados.0.smithi084.stdout:2089: oid 7 updating version 852 to 853 2024-09-20T22:50:12.320 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 7 v 853 (ObjNum 728 snap 230 seq_num 728) dirty exists 2024-09-20T22:50:12.320 INFO:tasks.rados.rados.0.smithi084.stdout:2089: left oid 7 (ObjNum 728 snap 230 seq_num 728) 2024-09-20T22:50:12.329 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:50:12.351 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:12.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:12.352 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:50:12.452 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:50:12.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:12.476 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:12.476 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:50:12.502 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:12 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/823109332' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-20T22:50:12.503 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:12 smithi084 ceph-mon[40024]: osdmap e531: 8 total, 8 up, 8 in 2024-09-20T22:50:12.503 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:12 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/4183702341' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:50:12.503 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:12 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:50:12.503 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:12 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/823109332' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-20T22:50:12.503 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:12 smithi084 ceph-mon[29774]: osdmap e531: 8 total, 8 up, 8 in 2024-09-20T22:50:12.503 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:12 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/4183702341' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:50:12.503 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:12 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:50:12.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:12 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/823109332' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-20T22:50:12.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:12 smithi154 ceph-mon[33264]: osdmap e531: 8 total, 8 up, 8 in 2024-09-20T22:50:12.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:12 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/4183702341' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:50:12.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:12 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:50:12.537 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-20T22:50:12.578 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:50:12.599 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:12.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:12.600 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:50:12.701 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:50:12.723 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:12.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:12.724 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:50:12.735 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-20T22:50:12.824 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:50:12.825 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:50:12.847 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:12.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:12.848 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:50:12.949 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:50:12.970 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:12.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:12.971 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:50:13.072 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:50:13.094 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:13.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:13.095 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:50:13.095 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:50:13.196 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:50:13.217 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:13.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:13.218 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:50:13.237 INFO:tasks.rados.rados.0.smithi084.stdout:2089: done (2 left) 2024-09-20T22:50:13.237 INFO:tasks.rados.rados.0.smithi084.stdout:2090: done (1 left) 2024-09-20T22:50:13.238 INFO:tasks.rados.rados.0.smithi084.stdout:2091: done (0 left) 2024-09-20T22:50:13.238 INFO:tasks.rados.rados.0.smithi084.stdout:2092: write oid 4 current snap is 231 2024-09-20T22:50:13.238 INFO:tasks.rados.rados.0.smithi084.stdout:2092: seq_num 729 ranges {652303=710056,2059726=192577} 2024-09-20T22:50:13.246 INFO:tasks.rados.rados.0.smithi084.stdout:2092: writing smithi08477203-4 from 652303 to 1362359 tid 1 2024-09-20T22:50:13.251 INFO:tasks.rados.rados.0.smithi084.stdout:2092: writing smithi08477203-4 from 2059726 to 2252303 tid 2 2024-09-20T22:50:13.251 INFO:tasks.rados.rados.0.smithi084.stdout:2093: delete oid 5 current snap is 231 2024-09-20T22:50:13.255 INFO:tasks.rados.rados.0.smithi084.stdout:2093: done (1 left) 2024-09-20T22:50:13.256 INFO:tasks.rados.rados.0.smithi084.stdout:2094: snap_remove snap 133 2024-09-20T22:50:13.290 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-20T22:50:13.290 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:50:13.319 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:50:13.341 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:13.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:13.342 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:50:13.344 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:13 smithi084 ceph-mon[40024]: pgmap v699: 158 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 153 active+clean; 538 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 1.5 MiB/s rd, 469 KiB/s wr, 9 op/s 2024-09-20T22:50:13.344 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:13 smithi084 ceph-mon[40024]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:50:13.344 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:13 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T22:50:13.344 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:13 smithi084 ceph-mon[40024]: osdmap e532: 8 total, 8 up, 8 in 2024-09-20T22:50:13.344 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:13 smithi084 ceph-mon[40024]: 5.19 scrub starts 2024-09-20T22:50:13.344 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:13 smithi084 ceph-mon[40024]: 5.19 scrub ok 2024-09-20T22:50:13.344 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:13 smithi084 ceph-mon[40024]: osdmap e533: 8 total, 8 up, 8 in 2024-09-20T22:50:13.344 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:13 smithi084 ceph-mon[29774]: pgmap v699: 158 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 153 active+clean; 538 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 1.5 MiB/s rd, 469 KiB/s wr, 9 op/s 2024-09-20T22:50:13.345 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:13 smithi084 ceph-mon[29774]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:50:13.345 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:13 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T22:50:13.345 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:13 smithi084 ceph-mon[29774]: osdmap e532: 8 total, 8 up, 8 in 2024-09-20T22:50:13.345 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:13 smithi084 ceph-mon[29774]: 5.19 scrub starts 2024-09-20T22:50:13.345 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:13 smithi084 ceph-mon[29774]: 5.19 scrub ok 2024-09-20T22:50:13.345 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:13 smithi084 ceph-mon[29774]: osdmap e533: 8 total, 8 up, 8 in 2024-09-20T22:50:13.443 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:50:13.464 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:13.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:13.465 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:50:13.566 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:50:13.572 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:50:13.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:13.588 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:13.588 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:50:13.689 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:50:13.711 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:13.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:13.712 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:50:13.746 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-20T22:50:13.748 INFO:teuthology.orchestra.run.smithi084.stderr:dumped all 2024-09-20T22:50:13.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:13 smithi154 ceph-mon[33264]: pgmap v699: 158 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 153 active+clean; 538 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 1.5 MiB/s rd, 469 KiB/s wr, 9 op/s 2024-09-20T22:50:13.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:13 smithi154 ceph-mon[33264]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:50:13.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:13 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T22:50:13.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:13 smithi154 ceph-mon[33264]: osdmap e532: 8 total, 8 up, 8 in 2024-09-20T22:50:13.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:13 smithi154 ceph-mon[33264]: 5.19 scrub starts 2024-09-20T22:50:13.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:13 smithi154 ceph-mon[33264]: 5.19 scrub ok 2024-09-20T22:50:13.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:13 smithi154 ceph-mon[33264]: osdmap e533: 8 total, 8 up, 8 in 2024-09-20T22:50:13.813 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:50:13.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:13.837 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:13.837 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:50:13.937 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:50:13.959 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:13.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:13.960 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:50:14.060 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:50:14.081 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:14.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:14.082 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:50:14.183 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:50:14.204 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:14.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:14.205 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:50:14.253 INFO:tasks.rados.rados.0.smithi084.stdout:2094: done (1 left) 2024-09-20T22:50:14.253 INFO:tasks.rados.rados.0.smithi084.stdout:2095: write (excl) oid 8 current snap is 231 2024-09-20T22:50:14.253 INFO:tasks.rados.rados.0.smithi084.stdout:2095: seq_num 731 ranges {685917=576907,1893043=392874} 2024-09-20T22:50:14.260 INFO:tasks.rados.rados.0.smithi084.stdout:2095: writing smithi08477203-8 from 685917 to 1262824 tid 1 2024-09-20T22:50:14.266 INFO:tasks.rados.rados.0.smithi084.stdout:2095: writing smithi08477203-8 from 1893043 to 2285917 tid 2 2024-09-20T22:50:14.266 INFO:tasks.rados.rados.0.smithi084.stdout:2096: delete oid 20 current snap is 231 2024-09-20T22:50:14.266 INFO:tasks.rados.rados.0.smithi084.stdout:2092: finishing write tid 1 to smithi08477203-4 2024-09-20T22:50:14.266 INFO:tasks.rados.rados.0.smithi084.stdout:2092: finishing write tid 2 to smithi08477203-4 2024-09-20T22:50:14.266 INFO:tasks.rados.rados.0.smithi084.stdout:2092: finishing write tid 3 to smithi08477203-4 2024-09-20T22:50:14.266 INFO:tasks.rados.rados.0.smithi084.stdout:2092: finishing write tid 4 to smithi08477203-4 2024-09-20T22:50:14.266 INFO:tasks.rados.rados.0.smithi084.stdout:2092: oid 4 updating version 0 to 1417 2024-09-20T22:50:14.266 INFO:tasks.rados.rados.0.smithi084.stdout:2092: oid 4 version 1417 is already newer than 1416 2024-09-20T22:50:14.266 INFO:tasks.rados.rados.0.smithi084.stdout:2092: oid 4 version 1417 is already newer than 1415 2024-09-20T22:50:14.266 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 4 v 1417 (ObjNum 729 snap 231 seq_num 729) dirty exists 2024-09-20T22:50:14.267 INFO:tasks.rados.rados.0.smithi084.stdout:2092: left oid 4 (ObjNum 729 snap 231 seq_num 729) 2024-09-20T22:50:14.267 INFO:tasks.rados.rados.0.smithi084.stdout:2095: finishing write tid 1 to smithi08477203-8 2024-09-20T22:50:14.267 INFO:tasks.rados.rados.0.smithi084.stdout:2092: done (2 left) 2024-09-20T22:50:14.268 INFO:tasks.rados.rados.0.smithi084.stdout:2096: done (1 left) 2024-09-20T22:50:14.268 INFO:tasks.rados.rados.0.smithi084.stdout:2097: snap_create 2024-09-20T22:50:14.268 INFO:tasks.rados.rados.0.smithi084.stdout:2095: finishing write tid 2 to smithi08477203-8 2024-09-20T22:50:14.268 INFO:tasks.rados.rados.0.smithi084.stdout:2095: finishing write tid 3 to smithi08477203-8 2024-09-20T22:50:14.268 INFO:tasks.rados.rados.0.smithi084.stdout:2095: finishing write tid 4 to smithi08477203-8 2024-09-20T22:50:14.268 INFO:tasks.rados.rados.0.smithi084.stdout:2095: oid 8 updating version 0 to 796 2024-09-20T22:50:14.269 INFO:tasks.rados.rados.0.smithi084.stdout:2095: oid 8 updating version 796 to 797 2024-09-20T22:50:14.269 INFO:tasks.rados.rados.0.smithi084.stdout:2095: oid 8 version 797 is already newer than 795 2024-09-20T22:50:14.269 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 8 v 797 (ObjNum 731 snap 231 seq_num 731) dirty exists 2024-09-20T22:50:14.269 INFO:tasks.rados.rados.0.smithi084.stdout:2095: left oid 8 (ObjNum 731 snap 231 seq_num 731) 2024-09-20T22:50:14.283 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.5 dump_blocked_ops 2024-09-20T22:50:14.306 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:50:14.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:14.348 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:14.348 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:50:14.448 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:50:14.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:14.471 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:14.471 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:50:14.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:14 smithi154 ceph-mon[33264]: 3.b scrub starts 2024-09-20T22:50:14.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:14 smithi154 ceph-mon[33264]: 3.b scrub ok 2024-09-20T22:50:14.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:14 smithi154 ceph-mon[33264]: osdmap e534: 8 total, 8 up, 8 in 2024-09-20T22:50:14.531 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:50:14.532 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:14 smithi084 ceph-mon[40024]: 3.b scrub starts 2024-09-20T22:50:14.532 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:14 smithi084 ceph-mon[40024]: 3.b scrub ok 2024-09-20T22:50:14.532 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:14 smithi084 ceph-mon[40024]: osdmap e534: 8 total, 8 up, 8 in 2024-09-20T22:50:14.533 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:14 smithi084 ceph-mon[29774]: 3.b scrub starts 2024-09-20T22:50:14.533 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:14 smithi084 ceph-mon[29774]: 3.b scrub ok 2024-09-20T22:50:14.533 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:14 smithi084 ceph-mon[29774]: osdmap e534: 8 total, 8 up, 8 in 2024-09-20T22:50:14.572 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:50:14.584 INFO:teuthology.orchestra.run.smithi084.stdout:{"pg_ready":true,"pg_map":{"version":701,"stamp":"2024-09-20T22:50:13.232005+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":564054638,"num_objects":456,"num_object_clones":208,"num_object_copies":1368,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":456,"num_whiteouts":18,"num_read":4819,"num_read_kb":3795417,"num_write":2975,"num_write_kb":934573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":887,"num_bytes_recovered":303303598,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5973,"ondisk_log_size":5973,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":474,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":1555676,"kb_used_data":1152092,"kb_used_omap":12,"kb_used_meta":403379,"kb_avail":748405540,"statfs":{"total":767960285184,"available":766367272960,"internally_reserved":0,"allocated":1179742208,"data_stored":1709759537,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":413060656},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.993929"},"pg_stats":[{"pgid":"3.1f","version":"86'2","reported_seq":302,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-20T22:50:08.234456+0000","last_change":"2024-09-20T22:49:26.647633+0000","last_active":"2024-09-20T22:50:08.234456+0000","last_peered":"2024-09-20T22:50:08.234456+0000","last_clean":"2024-09-20T22:50:08.234456+0000","last_became_active":"2024-09-20T22:47:49.222114+0000","last_became_peered":"2024-09-20T22:47:49.222114+0000","last_unstale":"2024-09-20T22:50:08.234456+0000","last_undegraded":"2024-09-20T22:50:08.234456+0000","last_fullsized":"2024-09-20T22:50:08.234456+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'2","last_scrub_stamp":"2024-09-20T22:49:26.647557+0000","last_deep_scrub":"86'2","last_deep_scrub_stamp":"2024-09-20T22:42:39.664421+0000","last_clean_scrub_stamp":"2024-09-20T22:49:26.647557+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:40.650510+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":436,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"74'9","reported_seq":275,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-20T22:50:08.236066+0000","last_change":"2024-09-20T22:49:51.091564+0000","last_active":"2024-09-20T22:50:08.236066+0000","last_peered":"2024-09-20T22:50:08.236066+0000","last_clean":"2024-09-20T22:50:08.236066+0000","last_became_active":"2024-09-20T22:48:05.248980+0000","last_became_peered":"2024-09-20T22:48:05.248980+0000","last_unstale":"2024-09-20T22:50:08.236066+0000","last_undegraded":"2024-09-20T22:50:08.236066+0000","last_fullsized":"2024-09-20T22:50:08.236066+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-09-20T22:49:51.091416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:49:51.091416+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:55.306855+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":951,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-20T22:50:08.235003+0000","last_change":"2024-09-20T22:48:43.157340+0000","last_active":"2024-09-20T22:50:08.235003+0000","last_peered":"2024-09-20T22:50:08.235003+0000","last_clean":"2024-09-20T22:50:08.235003+0000","last_became_active":"2024-09-20T22:41:42.182024+0000","last_became_peered":"2024-09-20T22:41:42.182024+0000","last_unstale":"2024-09-20T22:50:08.235003+0000","last_undegraded":"2024-09-20T22:50:08.235003+0000","last_fullsized":"2024-09-20T22:50:08.235003+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:48:43.157266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:48:43.157266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:10.647434+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":265,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-20T22:50:08.236438+0000","last_change":"2024-09-20T22:49:49.117854+0000","last_active":"2024-09-20T22:50:08.236438+0000","last_peered":"2024-09-20T22:50:08.236438+0000","last_clean":"2024-09-20T22:50:08.236438+0000","last_became_active":"2024-09-20T22:48:05.247569+0000","last_became_peered":"2024-09-20T22:48:05.247569+0000","last_unstale":"2024-09-20T22:50:08.236438+0000","last_undegraded":"2024-09-20T22:50:08.236438+0000","last_fullsized":"2024-09-20T22:50:08.236438+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:49.117772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:49:49.117772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:51.805116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":947,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-20T22:50:08.245371+0000","last_change":"2024-09-20T22:49:50.043884+0000","last_active":"2024-09-20T22:50:08.245371+0000","last_peered":"2024-09-20T22:50:08.245371+0000","last_clean":"2024-09-20T22:50:08.245371+0000","last_became_active":"2024-09-20T22:41:44.832099+0000","last_became_peered":"2024-09-20T22:41:44.832099+0000","last_unstale":"2024-09-20T22:50:08.245371+0000","last_undegraded":"2024-09-20T22:50:08.245371+0000","last_fullsized":"2024-09-20T22:50:08.245371+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:50.043776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:49:50.043776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:51:00.726179+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":952,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-20T22:50:08.246604+0000","last_change":"2024-09-20T22:49:10.077843+0000","last_active":"2024-09-20T22:50:08.246604+0000","last_peered":"2024-09-20T22:50:08.246604+0000","last_clean":"2024-09-20T22:50:08.246604+0000","last_became_active":"2024-09-20T22:41:38.157298+0000","last_became_peered":"2024-09-20T22:41:38.157298+0000","last_unstale":"2024-09-20T22:50:08.246604+0000","last_undegraded":"2024-09-20T22:50:08.246604+0000","last_fullsized":"2024-09-20T22:50:08.246604+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:10.077774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:49:10.077774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:21.915867+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"74'15","reported_seq":1016,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-20T22:50:08.245376+0000","last_change":"2024-09-20T22:49:07.042205+0000","last_active":"2024-09-20T22:50:08.245376+0000","last_peered":"2024-09-20T22:50:08.245376+0000","last_clean":"2024-09-20T22:50:08.245376+0000","last_became_active":"2024-09-20T22:47:49.218798+0000","last_became_peered":"2024-09-20T22:47:49.218798+0000","last_unstale":"2024-09-20T22:50:08.245376+0000","last_undegraded":"2024-09-20T22:50:08.245376+0000","last_fullsized":"2024-09-20T22:50:08.245376+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-09-20T22:49:07.042130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:49:07.042130+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:20.414124+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":265,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-20T22:50:08.236016+0000","last_change":"2024-09-20T22:49:59.011922+0000","last_active":"2024-09-20T22:50:08.236016+0000","last_peered":"2024-09-20T22:50:08.236016+0000","last_clean":"2024-09-20T22:50:08.236016+0000","last_became_active":"2024-09-20T22:48:05.247636+0000","last_became_peered":"2024-09-20T22:48:05.247636+0000","last_unstale":"2024-09-20T22:50:08.236016+0000","last_undegraded":"2024-09-20T22:50:08.236016+0000","last_fullsized":"2024-09-20T22:50:08.236016+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:59.011829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:49:59.011829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:51:07.355546+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":971,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-20T22:50:08.233864+0000","last_change":"2024-09-20T22:49:02.680764+0000","last_active":"2024-09-20T22:50:08.233864+0000","last_peered":"2024-09-20T22:50:08.233864+0000","last_clean":"2024-09-20T22:50:08.233864+0000","last_became_active":"2024-09-20T22:48:05.242059+0000","last_became_peered":"2024-09-20T22:48:05.242059+0000","last_unstale":"2024-09-20T22:50:08.233864+0000","last_undegraded":"2024-09-20T22:50:08.233864+0000","last_fullsized":"2024-09-20T22:50:08.233864+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:02.680682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:46:00.999939+0000","last_clean_scrub_stamp":"2024-09-20T22:49:02.680682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:22.162227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"74'9","reported_seq":275,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-20T22:50:08.236485+0000","last_change":"2024-09-20T22:49:56.048604+0000","last_active":"2024-09-20T22:50:08.236485+0000","last_peered":"2024-09-20T22:50:08.236485+0000","last_clean":"2024-09-20T22:50:08.236485+0000","last_became_active":"2024-09-20T22:48:05.233429+0000","last_became_peered":"2024-09-20T22:48:05.233429+0000","last_unstale":"2024-09-20T22:50:08.236485+0000","last_undegraded":"2024-09-20T22:50:08.236485+0000","last_fullsized":"2024-09-20T22:50:08.236485+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-09-20T22:49:56.048504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:49:56.048504+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:58.603199+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":963,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-20T22:50:08.231737+0000","last_change":"2024-09-20T22:49:31.649058+0000","last_active":"2024-09-20T22:50:08.231737+0000","last_peered":"2024-09-20T22:50:08.231737+0000","last_clean":"2024-09-20T22:50:08.231737+0000","last_became_active":"2024-09-20T22:47:49.227379+0000","last_became_peered":"2024-09-20T22:47:49.227379+0000","last_unstale":"2024-09-20T22:50:08.231737+0000","last_undegraded":"2024-09-20T22:50:08.231737+0000","last_fullsized":"2024-09-20T22:50:08.231737+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:31.648986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:49:31.648986+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:47.218836+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":297,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-20T22:50:08.231040+0000","last_change":"2024-09-20T22:49:08.664519+0000","last_active":"2024-09-20T22:50:08.231040+0000","last_peered":"2024-09-20T22:50:08.231040+0000","last_clean":"2024-09-20T22:50:08.231040+0000","last_became_active":"2024-09-20T22:47:49.225515+0000","last_became_peered":"2024-09-20T22:47:49.225515+0000","last_unstale":"2024-09-20T22:50:08.231040+0000","last_undegraded":"2024-09-20T22:50:08.231040+0000","last_fullsized":"2024-09-20T22:50:08.231040+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:08.664407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:49:08.664407+0000","last_clean_scrub_stamp":"2024-09-20T22:49:08.664407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:33.922538+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":971,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-20T22:50:08.233232+0000","last_change":"2024-09-20T22:48:37.559081+0000","last_active":"2024-09-20T22:50:08.233232+0000","last_peered":"2024-09-20T22:50:08.233232+0000","last_clean":"2024-09-20T22:50:08.233232+0000","last_became_active":"2024-09-20T22:48:05.242084+0000","last_became_peered":"2024-09-20T22:48:05.242084+0000","last_unstale":"2024-09-20T22:50:08.233232+0000","last_undegraded":"2024-09-20T22:50:08.233232+0000","last_fullsized":"2024-09-20T22:50:08.233232+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:48:37.559007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:48:37.559007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:12.708677+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"181'7","reported_seq":271,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-20T22:50:08.237076+0000","last_change":"2024-09-20T22:48:41.049491+0000","last_active":"2024-09-20T22:50:08.237076+0000","last_peered":"2024-09-20T22:50:08.237076+0000","last_clean":"2024-09-20T22:50:08.237076+0000","last_became_active":"2024-09-20T22:48:05.249763+0000","last_became_peered":"2024-09-20T22:48:05.249763+0000","last_unstale":"2024-09-20T22:50:08.237076+0000","last_undegraded":"2024-09-20T22:50:08.237076+0000","last_fullsized":"2024-09-20T22:50:08.237076+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"181'7","last_scrub_stamp":"2024-09-20T22:48:41.049381+0000","last_deep_scrub":"181'7","last_deep_scrub_stamp":"2024-09-20T22:48:41.049381+0000","last_clean_scrub_stamp":"2024-09-20T22:48:41.049381+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:11.149981+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":6,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":985,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-20T22:50:08.246631+0000","last_change":"2024-09-20T22:49:48.313924+0000","last_active":"2024-09-20T22:50:08.246631+0000","last_peered":"2024-09-20T22:50:08.246631+0000","last_clean":"2024-09-20T22:50:08.246631+0000","last_became_active":"2024-09-20T22:48:05.237791+0000","last_became_peered":"2024-09-20T22:48:05.237791+0000","last_unstale":"2024-09-20T22:50:08.246631+0000","last_undegraded":"2024-09-20T22:50:08.246631+0000","last_fullsized":"2024-09-20T22:50:08.246631+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:48.313860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:47:40.361958+0000","last_clean_scrub_stamp":"2024-09-20T22:49:48.313860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:49.887950+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":947,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-20T22:50:08.231729+0000","last_change":"2024-09-20T22:49:04.679724+0000","last_active":"2024-09-20T22:50:08.231729+0000","last_peered":"2024-09-20T22:50:08.231729+0000","last_clean":"2024-09-20T22:50:08.231729+0000","last_became_active":"2024-09-20T22:41:44.197927+0000","last_became_peered":"2024-09-20T22:41:44.197927+0000","last_unstale":"2024-09-20T22:50:08.231729+0000","last_undegraded":"2024-09-20T22:50:08.231729+0000","last_fullsized":"2024-09-20T22:50:08.231729+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:04.679617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:49:04.679617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:11.087913+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":167,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-20T22:50:08.232558+0000","last_change":"2024-09-20T22:48:50.224321+0000","last_active":"2024-09-20T22:50:08.232558+0000","last_peered":"2024-09-20T22:50:08.232558+0000","last_clean":"2024-09-20T22:50:08.232558+0000","last_became_active":"2024-09-20T22:48:50.223895+0000","last_became_peered":"2024-09-20T22:48:50.223895+0000","last_unstale":"2024-09-20T22:50:08.232558+0000","last_undegraded":"2024-09-20T22:50:08.232558+0000","last_fullsized":"2024-09-20T22:50:08.232558+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:47:21.819572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:47:21.819572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":303,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-20T22:50:08.232526+0000","last_change":"2024-09-20T22:49:53.933673+0000","last_active":"2024-09-20T22:50:08.232526+0000","last_peered":"2024-09-20T22:50:08.232526+0000","last_clean":"2024-09-20T22:50:08.232526+0000","last_became_active":"2024-09-20T22:48:05.233687+0000","last_became_peered":"2024-09-20T22:48:05.233687+0000","last_unstale":"2024-09-20T22:50:08.232526+0000","last_undegraded":"2024-09-20T22:50:08.232526+0000","last_fullsized":"2024-09-20T22:50:08.232526+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:53.933601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:49:53.933601+0000","last_clean_scrub_stamp":"2024-09-20T22:49:53.933601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:51:03.708862+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"113'22","reported_seq":1002,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-20T22:50:08.231499+0000","last_change":"2024-09-20T22:49:52.154967+0000","last_active":"2024-09-20T22:50:08.231499+0000","last_peered":"2024-09-20T22:50:08.231499+0000","last_clean":"2024-09-20T22:50:08.231499+0000","last_became_active":"2024-09-20T22:41:40.170131+0000","last_became_peered":"2024-09-20T22:41:40.170131+0000","last_unstale":"2024-09-20T22:50:08.231499+0000","last_undegraded":"2024-09-20T22:50:08.231499+0000","last_fullsized":"2024-09-20T22:50:08.231499+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'22","last_scrub_stamp":"2024-09-20T22:49:52.154856+0000","last_deep_scrub":"113'22","last_deep_scrub_stamp":"2024-09-20T22:43:12.191878+0000","last_clean_scrub_stamp":"2024-09-20T22:49:52.154856+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:51:04.660560+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":975,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-20T22:50:08.235895+0000","last_change":"2024-09-20T22:49:01.165679+0000","last_active":"2024-09-20T22:50:08.235895+0000","last_peered":"2024-09-20T22:50:08.235895+0000","last_clean":"2024-09-20T22:50:08.235895+0000","last_became_active":"2024-09-20T22:48:05.249417+0000","last_became_peered":"2024-09-20T22:48:05.249417+0000","last_unstale":"2024-09-20T22:50:08.235895+0000","last_undegraded":"2024-09-20T22:50:08.235895+0000","last_fullsized":"2024-09-20T22:50:08.235895+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:01.165605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:49:01.165605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:10.647465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":940,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-20T22:50:08.246784+0000","last_change":"2024-09-20T22:49:30.132559+0000","last_active":"2024-09-20T22:50:08.246784+0000","last_peered":"2024-09-20T22:50:08.246784+0000","last_clean":"2024-09-20T22:50:08.246784+0000","last_became_active":"2024-09-20T22:41:44.832131+0000","last_became_peered":"2024-09-20T22:41:44.832131+0000","last_unstale":"2024-09-20T22:50:08.246784+0000","last_undegraded":"2024-09-20T22:50:08.246784+0000","last_fullsized":"2024-09-20T22:50:08.246784+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:30.132466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:45:25.002199+0000","last_clean_scrub_stamp":"2024-09-20T22:49:30.132466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:30.604086+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":265,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-20T22:50:08.236632+0000","last_change":"2024-09-20T22:48:24.986582+0000","last_active":"2024-09-20T22:50:08.236632+0000","last_peered":"2024-09-20T22:50:08.236632+0000","last_clean":"2024-09-20T22:50:08.236632+0000","last_became_active":"2024-09-20T22:48:05.250581+0000","last_became_peered":"2024-09-20T22:48:05.250581+0000","last_unstale":"2024-09-20T22:50:08.236632+0000","last_undegraded":"2024-09-20T22:50:08.236632+0000","last_fullsized":"2024-09-20T22:50:08.236632+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:48:24.986484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:48:24.986484+0000","last_clean_scrub_stamp":"2024-09-20T22:48:24.986484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:11.149951+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"74'12","reported_seq":1024,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-20T22:50:08.245915+0000","last_change":"2024-09-20T22:48:59.088132+0000","last_active":"2024-09-20T22:50:08.245915+0000","last_peered":"2024-09-20T22:50:08.245915+0000","last_clean":"2024-09-20T22:50:08.245915+0000","last_became_active":"2024-09-20T22:48:05.233983+0000","last_became_peered":"2024-09-20T22:48:05.233983+0000","last_unstale":"2024-09-20T22:50:08.245915+0000","last_undegraded":"2024-09-20T22:50:08.245915+0000","last_fullsized":"2024-09-20T22:50:08.245915+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-09-20T22:48:59.088030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:48:59.088030+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:08.919507+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":265,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-20T22:50:08.236655+0000","last_change":"2024-09-20T22:48:23.940233+0000","last_active":"2024-09-20T22:50:08.236655+0000","last_peered":"2024-09-20T22:50:08.236655+0000","last_clean":"2024-09-20T22:50:08.236655+0000","last_became_active":"2024-09-20T22:48:05.250319+0000","last_became_peered":"2024-09-20T22:48:05.250319+0000","last_unstale":"2024-09-20T22:50:08.236655+0000","last_undegraded":"2024-09-20T22:50:08.236655+0000","last_fullsized":"2024-09-20T22:50:08.236655+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:48:23.940157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:48:23.940157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:11.149966+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"154'2","reported_seq":948,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-20T22:50:08.244744+0000","last_change":"2024-09-20T22:48:39.357067+0000","last_active":"2024-09-20T22:50:08.244744+0000","last_peered":"2024-09-20T22:50:08.244744+0000","last_clean":"2024-09-20T22:50:08.244744+0000","last_became_active":"2024-09-20T22:41:44.199822+0000","last_became_peered":"2024-09-20T22:41:44.199822+0000","last_unstale":"2024-09-20T22:50:08.244744+0000","last_undegraded":"2024-09-20T22:50:08.244744+0000","last_fullsized":"2024-09-20T22:50:08.244744+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"154'2","last_scrub_stamp":"2024-09-20T22:48:39.356937+0000","last_deep_scrub":"154'2","last_deep_scrub_stamp":"2024-09-20T22:43:54.134066+0000","last_clean_scrub_stamp":"2024-09-20T22:48:39.356937+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:09.481691+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"67'1","reported_seq":980,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-20T22:50:08.235680+0000","last_change":"2024-09-20T22:48:42.188853+0000","last_active":"2024-09-20T22:50:08.235680+0000","last_peered":"2024-09-20T22:50:08.235680+0000","last_clean":"2024-09-20T22:50:08.235680+0000","last_became_active":"2024-09-20T22:48:05.249578+0000","last_became_peered":"2024-09-20T22:48:05.249578+0000","last_unstale":"2024-09-20T22:50:08.235680+0000","last_undegraded":"2024-09-20T22:50:08.235680+0000","last_fullsized":"2024-09-20T22:50:08.235680+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-09-20T22:48:42.188772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:48:42.188772+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:09.372929+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"151'15","reported_seq":310,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-20T22:50:08.234294+0000","last_change":"2024-09-20T22:48:37.528315+0000","last_active":"2024-09-20T22:50:08.234294+0000","last_peered":"2024-09-20T22:50:08.234294+0000","last_clean":"2024-09-20T22:50:08.234294+0000","last_became_active":"2024-09-20T22:47:49.232181+0000","last_became_peered":"2024-09-20T22:47:49.232181+0000","last_unstale":"2024-09-20T22:50:08.234294+0000","last_undegraded":"2024-09-20T22:50:08.234294+0000","last_fullsized":"2024-09-20T22:50:08.234294+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"151'15","last_scrub_stamp":"2024-09-20T22:48:37.528207+0000","last_deep_scrub":"151'15","last_deep_scrub_stamp":"2024-09-20T22:43:50.715761+0000","last_clean_scrub_stamp":"2024-09-20T22:48:37.528207+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"408'10","reported_seq":1097,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-20T22:50:08.242686+0000","last_change":"2024-09-20T22:49:47.030294+0000","last_active":"2024-09-20T22:50:08.242686+0000","last_peered":"2024-09-20T22:50:08.242686+0000","last_clean":"2024-09-20T22:50:08.242686+0000","last_became_active":"2024-09-20T22:48:05.244270+0000","last_became_peered":"2024-09-20T22:48:05.244270+0000","last_unstale":"2024-09-20T22:50:08.242686+0000","last_undegraded":"2024-09-20T22:50:08.242686+0000","last_fullsized":"2024-09-20T22:50:08.242686+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"408'10","last_scrub_stamp":"2024-09-20T22:49:47.030217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:49:47.030217+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:56.479090+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.c","version":"528'1406","reported_seq":3539,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-20T22:50:08.252807+0000","last_change":"2024-09-20T22:50:08.253128+0000","last_active":"2024-09-20T22:50:08.252807+0000","last_peered":"2024-09-20T22:50:08.252807+0000","last_clean":"2024-09-20T22:50:08.252807+0000","last_became_active":"2024-09-20T22:42:52.404027+0000","last_became_peered":"2024-09-20T22:42:52.404027+0000","last_unstale":"2024-09-20T22:50:08.252807+0000","last_undegraded":"2024-09-20T22:50:08.252807+0000","last_fullsized":"2024-09-20T22:50:08.252807+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"508'1323","last_scrub_stamp":"2024-09-20T22:49:49.001444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:49:49.001444+0000","objects_scrubbed":31,"log_size":1406,"log_dups_size":0,"ondisk_log_size":1406,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:51:13.893170+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0.010916566000000001,"stat_sum":{"num_bytes":75474586,"num_objects":35,"num_object_clones":28,"num_object_copies":105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":4,"num_read":628,"num_read_kb":641118,"num_write":441,"num_write_kb":169834,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"d6","length":"1"},{"start":"dc","length":"3"}]},{"pgid":"1.a","version":"77'128","reported_seq":1218,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-20T22:50:08.231603+0000","last_change":"2024-09-20T22:48:24.566023+0000","last_active":"2024-09-20T22:50:08.231603+0000","last_peered":"2024-09-20T22:50:08.231603+0000","last_clean":"2024-09-20T22:50:08.231603+0000","last_became_active":"2024-09-20T22:42:52.407791+0000","last_became_peered":"2024-09-20T22:42:52.407791+0000","last_unstale":"2024-09-20T22:50:08.231603+0000","last_undegraded":"2024-09-20T22:50:08.231603+0000","last_fullsized":"2024-09-20T22:50:08.231603+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:48:24.565920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:48:24.565920+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:12.708713+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"74'15","reported_seq":1032,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-20T22:50:08.235966+0000","last_change":"2024-09-20T22:49:55.709162+0000","last_active":"2024-09-20T22:50:08.235966+0000","last_peered":"2024-09-20T22:50:08.235966+0000","last_clean":"2024-09-20T22:50:08.235966+0000","last_became_active":"2024-09-20T22:48:05.250308+0000","last_became_peered":"2024-09-20T22:48:05.250308+0000","last_unstale":"2024-09-20T22:50:08.235966+0000","last_undegraded":"2024-09-20T22:50:08.235966+0000","last_fullsized":"2024-09-20T22:50:08.235966+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-09-20T22:49:55.709043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:49:55.709043+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:51:25.064971+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":951,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-20T22:50:08.244572+0000","last_change":"2024-09-20T22:49:12.070747+0000","last_active":"2024-09-20T22:50:08.244572+0000","last_peered":"2024-09-20T22:50:08.244572+0000","last_clean":"2024-09-20T22:50:08.244572+0000","last_became_active":"2024-09-20T22:41:38.156749+0000","last_became_peered":"2024-09-20T22:41:38.156749+0000","last_unstale":"2024-09-20T22:50:08.244572+0000","last_undegraded":"2024-09-20T22:50:08.244572+0000","last_fullsized":"2024-09-20T22:50:08.244572+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:12.070675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:49:12.070675+0000","last_clean_scrub_stamp":"2024-09-20T22:49:12.070675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:34.126084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"408'12","reported_seq":286,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-20T22:50:08.238563+0000","last_change":"2024-09-20T22:48:05.251167+0000","last_active":"2024-09-20T22:50:08.238563+0000","last_peered":"2024-09-20T22:50:08.238563+0000","last_clean":"2024-09-20T22:50:08.238563+0000","last_became_active":"2024-09-20T22:48:05.242914+0000","last_became_peered":"2024-09-20T22:48:05.242914+0000","last_unstale":"2024-09-20T22:50:08.238563+0000","last_undegraded":"2024-09-20T22:50:08.238563+0000","last_fullsized":"2024-09-20T22:50:08.238563+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"264'10","last_scrub_stamp":"2024-09-20T22:47:20.300787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:47:20.300787+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:11.149775+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":959,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-20T22:50:08.233328+0000","last_change":"2024-09-20T22:49:07.658309+0000","last_active":"2024-09-20T22:50:08.233328+0000","last_peered":"2024-09-20T22:50:08.233328+0000","last_clean":"2024-09-20T22:50:08.233328+0000","last_became_active":"2024-09-20T22:47:49.227039+0000","last_became_peered":"2024-09-20T22:47:49.227039+0000","last_unstale":"2024-09-20T22:50:08.233328+0000","last_undegraded":"2024-09-20T22:50:08.233328+0000","last_fullsized":"2024-09-20T22:50:08.233328+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:07.658234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:49:07.658234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:26.817461+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3","version":"527'184","reported_seq":2197,"reported_epoch":528,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T22:50:08.245239+0000","last_change":"2024-09-20T22:50:07.320249+0000","last_active":"2024-09-20T22:50:08.245239+0000","last_peered":"2024-09-20T22:50:08.245239+0000","last_clean":"2024-09-20T22:50:08.245239+0000","last_became_active":"2024-09-20T22:49:05.221402+0000","last_became_peered":"2024-09-20T22:49:05.221402+0000","last_unstale":"2024-09-20T22:50:08.245239+0000","last_undegraded":"2024-09-20T22:50:08.245239+0000","last_fullsized":"2024-09-20T22:50:08.245239+0000","mapping_epoch":404,"log_start":"456'129","ondisk_log_start":"456'129","created":94,"last_epoch_clean":465,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'8","last_scrub_stamp":"2024-09-20T22:47:23.137750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:47:23.137750+0000","objects_scrubbed":0,"log_size":55,"log_dups_size":149,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":1.005271037,"stat_sum":{"num_bytes":26803665,"num_objects":9,"num_object_clones":8,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":69,"num_read_kb":85984,"num_write":47,"num_write_kb":18651,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":20196357,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"d6","length":"1"},{"start":"dc","length":"1"}]},{"pgid":"1.5","version":"77'128","reported_seq":1123,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-20T22:50:08.239239+0000","last_change":"2024-09-20T22:49:26.023144+0000","last_active":"2024-09-20T22:50:08.239239+0000","last_peered":"2024-09-20T22:50:08.239239+0000","last_clean":"2024-09-20T22:50:08.239239+0000","last_became_active":"2024-09-20T22:48:08.207018+0000","last_became_peered":"2024-09-20T22:48:08.207018+0000","last_unstale":"2024-09-20T22:50:08.239239+0000","last_undegraded":"2024-09-20T22:50:08.239239+0000","last_fullsized":"2024-09-20T22:50:08.239239+0000","mapping_epoch":407,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":408,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:49:26.023044+0000","last_deep_scrub":"77'128","last_deep_scrub_stamp":"2024-09-20T22:45:41.608075+0000","last_clean_scrub_stamp":"2024-09-20T22:49:26.023044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:29.392816+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"74'18","reported_seq":1028,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-20T22:50:08.244757+0000","last_change":"2024-09-20T22:48:43.146697+0000","last_active":"2024-09-20T22:50:08.244757+0000","last_peered":"2024-09-20T22:50:08.244757+0000","last_clean":"2024-09-20T22:50:08.244757+0000","last_became_active":"2024-09-20T22:47:49.217605+0000","last_became_peered":"2024-09-20T22:47:49.217605+0000","last_unstale":"2024-09-20T22:50:08.244757+0000","last_undegraded":"2024-09-20T22:50:08.244757+0000","last_fullsized":"2024-09-20T22:50:08.244757+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'18","last_scrub_stamp":"2024-09-20T22:48:43.146602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:48:43.146602+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:12.131576+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":963,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-20T22:50:08.247065+0000","last_change":"2024-09-20T22:47:49.227431+0000","last_active":"2024-09-20T22:50:08.247065+0000","last_peered":"2024-09-20T22:50:08.247065+0000","last_clean":"2024-09-20T22:50:08.247065+0000","last_became_active":"2024-09-20T22:47:49.227069+0000","last_became_peered":"2024-09-20T22:47:49.227069+0000","last_unstale":"2024-09-20T22:50:08.247065+0000","last_undegraded":"2024-09-20T22:50:08.247065+0000","last_fullsized":"2024-09-20T22:50:08.247065+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:47:39.982616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:47:39.982616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:12.131484+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":265,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-20T22:50:08.237057+0000","last_change":"2024-09-20T22:49:27.968569+0000","last_active":"2024-09-20T22:50:08.237057+0000","last_peered":"2024-09-20T22:50:08.237057+0000","last_clean":"2024-09-20T22:50:08.237057+0000","last_became_active":"2024-09-20T22:48:05.230003+0000","last_became_peered":"2024-09-20T22:48:05.230003+0000","last_unstale":"2024-09-20T22:50:08.237057+0000","last_undegraded":"2024-09-20T22:50:08.237057+0000","last_fullsized":"2024-09-20T22:50:08.237057+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:27.968498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:07.947945+0000","last_clean_scrub_stamp":"2024-09-20T22:49:27.968498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:39.692203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":257,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-20T22:50:08.236577+0000","last_change":"2024-09-20T22:48:05.240266+0000","last_active":"2024-09-20T22:50:08.236577+0000","last_peered":"2024-09-20T22:50:08.236577+0000","last_clean":"2024-09-20T22:50:08.236577+0000","last_became_active":"2024-09-20T22:48:05.239917+0000","last_became_peered":"2024-09-20T22:48:05.239917+0000","last_unstale":"2024-09-20T22:50:08.236577+0000","last_undegraded":"2024-09-20T22:50:08.236577+0000","last_fullsized":"2024-09-20T22:50:08.236577+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:47:18.179575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:53.036601+0000","last_clean_scrub_stamp":"2024-09-20T22:47:18.179575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:11.149714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2","version":"525'529","reported_seq":1978,"reported_epoch":528,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-20T22:50:08.247978+0000","last_change":"2024-09-20T22:50:08.248020+0000","last_active":"2024-09-20T22:50:08.247978+0000","last_peered":"2024-09-20T22:50:08.247978+0000","last_clean":"2024-09-20T22:50:08.247978+0000","last_became_active":"2024-09-20T22:49:17.242481+0000","last_became_peered":"2024-09-20T22:49:17.242481+0000","last_unstale":"2024-09-20T22:50:08.247978+0000","last_undegraded":"2024-09-20T22:50:08.247978+0000","last_fullsized":"2024-09-20T22:50:08.247978+0000","mapping_epoch":469,"log_start":"471'464","ondisk_log_start":"471'464","created":94,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"282'120","last_scrub_stamp":"2024-09-20T22:46:22.099642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:46:22.099642+0000","objects_scrubbed":1,"log_size":65,"log_dups_size":687,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0024688800000000001,"stat_sum":{"num_bytes":37522363,"num_objects":17,"num_object_clones":13,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":339,"num_read_kb":374975,"num_write":185,"num_write_kb":61120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":120,"num_bytes_recovered":49828787,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"d6","length":"1"},{"start":"dc","length":"2"}]},{"pgid":"1.4","version":"77'128","reported_seq":1251,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-20T22:50:08.246578+0000","last_change":"2024-09-20T22:49:58.063457+0000","last_active":"2024-09-20T22:50:08.246578+0000","last_peered":"2024-09-20T22:50:08.246578+0000","last_clean":"2024-09-20T22:50:08.246578+0000","last_became_active":"2024-09-20T22:46:15.181305+0000","last_became_peered":"2024-09-20T22:46:15.181305+0000","last_unstale":"2024-09-20T22:50:08.246578+0000","last_undegraded":"2024-09-20T22:50:08.246578+0000","last_fullsized":"2024-09-20T22:50:08.246578+0000","mapping_epoch":94,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:49:58.063382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:49:58.063382+0000","objects_scrubbed":1,"log_size":0,"log_dups_size":59,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:51:06.139917+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"74'14","reported_seq":281,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-20T22:50:08.236964+0000","last_change":"2024-09-20T22:49:57.026346+0000","last_active":"2024-09-20T22:50:08.236964+0000","last_peered":"2024-09-20T22:50:08.236964+0000","last_clean":"2024-09-20T22:50:08.236964+0000","last_became_active":"2024-09-20T22:48:05.235812+0000","last_became_peered":"2024-09-20T22:48:05.235812+0000","last_unstale":"2024-09-20T22:50:08.236964+0000","last_undegraded":"2024-09-20T22:50:08.236964+0000","last_fullsized":"2024-09-20T22:50:08.236964+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'14","last_scrub_stamp":"2024-09-20T22:49:57.026245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:49:57.026245+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:51:23.191208+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"67'1","reported_seq":312,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-20T22:50:08.233955+0000","last_change":"2024-09-20T22:48:36.570753+0000","last_active":"2024-09-20T22:50:08.233955+0000","last_peered":"2024-09-20T22:50:08.233955+0000","last_clean":"2024-09-20T22:50:08.233955+0000","last_became_active":"2024-09-20T22:48:05.237425+0000","last_became_peered":"2024-09-20T22:48:05.237425+0000","last_unstale":"2024-09-20T22:50:08.233955+0000","last_undegraded":"2024-09-20T22:50:08.233955+0000","last_fullsized":"2024-09-20T22:50:08.233955+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-09-20T22:48:36.570672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:48:36.570672+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"408'12","reported_seq":1095,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-20T22:50:08.244954+0000","last_change":"2024-09-20T22:49:25.151447+0000","last_active":"2024-09-20T22:50:08.244954+0000","last_peered":"2024-09-20T22:50:08.244954+0000","last_clean":"2024-09-20T22:50:08.244954+0000","last_became_active":"2024-09-20T22:48:05.234607+0000","last_became_peered":"2024-09-20T22:48:05.234607+0000","last_unstale":"2024-09-20T22:50:08.244954+0000","last_undegraded":"2024-09-20T22:50:08.244954+0000","last_fullsized":"2024-09-20T22:50:08.244954+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"408'12","last_scrub_stamp":"2024-09-20T22:49:25.151326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:49:25.151326+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:44.177686+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":947,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-20T22:50:08.247291+0000","last_change":"2024-09-20T22:49:31.310399+0000","last_active":"2024-09-20T22:50:08.247291+0000","last_peered":"2024-09-20T22:50:08.247291+0000","last_clean":"2024-09-20T22:50:08.247291+0000","last_became_active":"2024-09-20T22:41:44.831901+0000","last_became_peered":"2024-09-20T22:41:44.831901+0000","last_unstale":"2024-09-20T22:50:08.247291+0000","last_undegraded":"2024-09-20T22:50:08.247291+0000","last_fullsized":"2024-09-20T22:50:08.247291+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:31.310330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:49:31.310330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:34.862425+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1","version":"520'283","reported_seq":2376,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-20T22:50:08.237365+0000","last_change":"2024-09-20T22:50:08.237622+0000","last_active":"2024-09-20T22:50:08.237365+0000","last_peered":"2024-09-20T22:50:08.237365+0000","last_clean":"2024-09-20T22:50:08.237365+0000","last_became_active":"2024-09-20T22:49:21.232750+0000","last_became_peered":"2024-09-20T22:49:21.232750+0000","last_unstale":"2024-09-20T22:50:08.237365+0000","last_undegraded":"2024-09-20T22:50:08.237365+0000","last_fullsized":"2024-09-20T22:50:08.237365+0000","mapping_epoch":407,"log_start":"474'259","ondisk_log_start":"474'259","created":94,"last_epoch_clean":481,"parent":"0.0","parent_split_bits":0,"last_scrub":"420'231","last_scrub_stamp":"2024-09-20T22:48:22.915993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:48:22.915993+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":338,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00066265900000000001,"stat_sum":{"num_bytes":13160244,"num_objects":6,"num_object_clones":4,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":65,"num_read_kb":78461,"num_write":63,"num_write_kb":38776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":8732217,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"d6","length":"1"},{"start":"dc","length":"3"}]},{"pgid":"1.7","version":"77'128","reported_seq":1256,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-20T22:50:08.242457+0000","last_change":"2024-09-20T22:49:31.994386+0000","last_active":"2024-09-20T22:50:08.242457+0000","last_peered":"2024-09-20T22:50:08.242457+0000","last_clean":"2024-09-20T22:50:08.242457+0000","last_became_active":"2024-09-20T22:46:03.169596+0000","last_became_peered":"2024-09-20T22:46:03.169596+0000","last_unstale":"2024-09-20T22:50:08.242457+0000","last_undegraded":"2024-09-20T22:50:08.242457+0000","last_fullsized":"2024-09-20T22:50:08.242457+0000","mapping_epoch":94,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:49:31.994311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:49:31.994311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:35.643288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"74'10","reported_seq":1051,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-20T22:50:08.235169+0000","last_change":"2024-09-20T22:49:46.855799+0000","last_active":"2024-09-20T22:50:08.235169+0000","last_peered":"2024-09-20T22:50:08.235169+0000","last_clean":"2024-09-20T22:50:08.235169+0000","last_became_active":"2024-09-20T22:48:05.251200+0000","last_became_peered":"2024-09-20T22:48:05.251200+0000","last_unstale":"2024-09-20T22:50:08.235169+0000","last_undegraded":"2024-09-20T22:50:08.235169+0000","last_fullsized":"2024-09-20T22:50:08.235169+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-09-20T22:49:46.855673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:49:46.855673+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:55.301300+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":960,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-20T22:50:08.233935+0000","last_change":"2024-09-20T22:48:04.520131+0000","last_active":"2024-09-20T22:50:08.233935+0000","last_peered":"2024-09-20T22:50:08.233935+0000","last_clean":"2024-09-20T22:50:08.233935+0000","last_became_active":"2024-09-20T22:41:38.155629+0000","last_became_peered":"2024-09-20T22:41:38.155629+0000","last_unstale":"2024-09-20T22:50:08.233935+0000","last_undegraded":"2024-09-20T22:50:08.233935+0000","last_fullsized":"2024-09-20T22:50:08.233935+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:48:04.520053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:48:04.520053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:12.708660+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"389'11","reported_seq":331,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-20T22:50:08.232544+0000","last_change":"2024-09-20T22:48:44.601907+0000","last_active":"2024-09-20T22:50:08.232544+0000","last_peered":"2024-09-20T22:50:08.232544+0000","last_clean":"2024-09-20T22:50:08.232544+0000","last_became_active":"2024-09-20T22:47:49.227264+0000","last_became_peered":"2024-09-20T22:47:49.227264+0000","last_unstale":"2024-09-20T22:50:08.232544+0000","last_undegraded":"2024-09-20T22:50:08.232544+0000","last_fullsized":"2024-09-20T22:50:08.232544+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'11","last_scrub_stamp":"2024-09-20T22:48:44.601816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:48:44.601816+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:10.964457+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":298,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-20T22:50:08.230826+0000","last_change":"2024-09-20T22:49:27.627363+0000","last_active":"2024-09-20T22:50:08.230826+0000","last_peered":"2024-09-20T22:50:08.230826+0000","last_clean":"2024-09-20T22:50:08.230826+0000","last_became_active":"2024-09-20T22:47:49.232380+0000","last_became_peered":"2024-09-20T22:47:49.232380+0000","last_unstale":"2024-09-20T22:50:08.230826+0000","last_undegraded":"2024-09-20T22:50:08.230826+0000","last_fullsized":"2024-09-20T22:50:08.230826+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:27.627295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:49:27.627295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:29.554480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.0","version":"527'309","reported_seq":2478,"reported_epoch":528,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T22:50:08.231543+0000","last_change":"2024-09-20T22:50:07.265007+0000","last_active":"2024-09-20T22:50:08.231543+0000","last_peered":"2024-09-20T22:50:08.231543+0000","last_clean":"2024-09-20T22:50:08.231543+0000","last_became_active":"2024-09-20T22:49:25.236419+0000","last_became_peered":"2024-09-20T22:49:25.236419+0000","last_unstale":"2024-09-20T22:50:08.231543+0000","last_undegraded":"2024-09-20T22:50:08.231543+0000","last_fullsized":"2024-09-20T22:50:08.231543+0000","mapping_epoch":94,"log_start":"480'253","ondisk_log_start":"480'253","created":94,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'228","last_scrub_stamp":"2024-09-20T22:48:59.633944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:48:59.633944+0000","objects_scrubbed":7,"log_size":56,"log_dups_size":416,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.99560686899999995,"stat_sum":{"num_bytes":31595254,"num_objects":16,"num_object_clones":13,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":1,"num_read":163,"num_read_kb":185504,"num_write":105,"num_write_kb":39186,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":7603720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"d6","length":"1"},{"start":"dc","length":"1"}]},{"pgid":"1.6","version":"77'128","reported_seq":1278,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-20T22:50:08.241164+0000","last_change":"2024-09-20T22:49:28.091254+0000","last_active":"2024-09-20T22:50:08.241164+0000","last_peered":"2024-09-20T22:50:08.241164+0000","last_clean":"2024-09-20T22:50:08.241164+0000","last_became_active":"2024-09-20T22:47:52.212190+0000","last_became_peered":"2024-09-20T22:47:52.212190+0000","last_unstale":"2024-09-20T22:50:08.241164+0000","last_undegraded":"2024-09-20T22:50:08.241164+0000","last_fullsized":"2024-09-20T22:50:08.241164+0000","mapping_epoch":391,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:49:28.091184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:49:28.091184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:40.740952+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"515'27","reported_seq":324,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-20T22:50:08.232256+0000","last_change":"2024-09-20T22:49:55.964247+0000","last_active":"2024-09-20T22:50:08.232256+0000","last_peered":"2024-09-20T22:50:08.232256+0000","last_clean":"2024-09-20T22:50:08.232256+0000","last_became_active":"2024-09-20T22:47:49.222009+0000","last_became_peered":"2024-09-20T22:47:49.222009+0000","last_unstale":"2024-09-20T22:50:08.232256+0000","last_undegraded":"2024-09-20T22:50:08.232256+0000","last_fullsized":"2024-09-20T22:50:08.232256+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"515'27","last_scrub_stamp":"2024-09-20T22:49:55.964153+0000","last_deep_scrub":"515'27","last_deep_scrub_stamp":"2024-09-20T22:49:55.964153+0000","last_clean_scrub_stamp":"2024-09-20T22:49:55.964153+0000","objects_scrubbed":8,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:57.993517+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":38,"num_read_kb":24,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":968,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-20T22:50:08.236446+0000","last_change":"2024-09-20T22:49:28.084441+0000","last_active":"2024-09-20T22:50:08.236446+0000","last_peered":"2024-09-20T22:50:08.236446+0000","last_clean":"2024-09-20T22:50:08.236446+0000","last_became_active":"2024-09-20T22:41:38.160832+0000","last_became_peered":"2024-09-20T22:41:38.160832+0000","last_unstale":"2024-09-20T22:50:08.236446+0000","last_undegraded":"2024-09-20T22:50:08.236446+0000","last_fullsized":"2024-09-20T22:50:08.236446+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:28.084371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:48:23.320031+0000","last_clean_scrub_stamp":"2024-09-20T22:49:28.084371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:30.046341+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":964,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-20T22:50:08.241282+0000","last_change":"2024-09-20T22:49:00.208581+0000","last_active":"2024-09-20T22:50:08.241282+0000","last_peered":"2024-09-20T22:50:08.241282+0000","last_clean":"2024-09-20T22:50:08.241282+0000","last_became_active":"2024-09-20T22:47:49.214055+0000","last_became_peered":"2024-09-20T22:47:49.214055+0000","last_unstale":"2024-09-20T22:50:08.241282+0000","last_undegraded":"2024-09-20T22:50:08.241282+0000","last_fullsized":"2024-09-20T22:50:08.241282+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:00.208454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:49:00.208454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:10.920020+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":948,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-20T22:50:08.234914+0000","last_change":"2024-09-20T22:48:41.208933+0000","last_active":"2024-09-20T22:50:08.234914+0000","last_peered":"2024-09-20T22:50:08.234914+0000","last_clean":"2024-09-20T22:50:08.234914+0000","last_became_active":"2024-09-20T22:41:44.830538+0000","last_became_peered":"2024-09-20T22:41:44.830538+0000","last_unstale":"2024-09-20T22:50:08.234914+0000","last_undegraded":"2024-09-20T22:50:08.234914+0000","last_fullsized":"2024-09-20T22:50:08.234914+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:48:41.208853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:48:41.208853+0000","last_clean_scrub_stamp":"2024-09-20T22:48:41.208853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:10.647450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.7","version":"528'706","reported_seq":2967,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-20T22:50:08.250011+0000","last_change":"2024-09-20T22:50:08.250300+0000","last_active":"2024-09-20T22:50:08.250011+0000","last_peered":"2024-09-20T22:50:08.250011+0000","last_clean":"2024-09-20T22:50:08.250011+0000","last_became_active":"2024-09-20T22:48:43.224386+0000","last_became_peered":"2024-09-20T22:48:43.224386+0000","last_unstale":"2024-09-20T22:50:08.250011+0000","last_undegraded":"2024-09-20T22:50:08.250011+0000","last_fullsized":"2024-09-20T22:50:08.250011+0000","mapping_epoch":391,"log_start":"436'456","ondisk_log_start":"436'456","created":94,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"463'527","last_scrub_stamp":"2024-09-20T22:49:04.175508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:49:04.175508+0000","objects_scrubbed":23,"log_size":250,"log_dups_size":886,"ondisk_log_size":250,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:20.507328+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.0071963569999999996,"stat_sum":{"num_bytes":53169886,"num_objects":31,"num_object_clones":26,"num_object_copies":93,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":4,"num_read":341,"num_read_kb":364157,"num_write":261,"num_write_kb":86105,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":115,"num_bytes_recovered":51615167,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"d6","length":"1"},{"start":"dc","length":"3"}]},{"pgid":"1.1","version":"77'128","reported_seq":1266,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-20T22:50:08.231476+0000","last_change":"2024-09-20T22:48:44.631654+0000","last_active":"2024-09-20T22:50:08.231476+0000","last_peered":"2024-09-20T22:50:08.231476+0000","last_clean":"2024-09-20T22:50:08.231476+0000","last_became_active":"2024-09-20T22:47:52.208102+0000","last_became_peered":"2024-09-20T22:47:52.208102+0000","last_unstale":"2024-09-20T22:50:08.231476+0000","last_undegraded":"2024-09-20T22:50:08.231476+0000","last_fullsized":"2024-09-20T22:50:08.231476+0000","mapping_epoch":391,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:48:44.631571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:48:44.631571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:09.749886+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"86'38","reported_seq":1125,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-20T22:50:08.236394+0000","last_change":"2024-09-20T22:49:31.040446+0000","last_active":"2024-09-20T22:50:08.236394+0000","last_peered":"2024-09-20T22:50:08.236394+0000","last_clean":"2024-09-20T22:50:08.236394+0000","last_became_active":"2024-09-20T22:41:40.166958+0000","last_became_peered":"2024-09-20T22:41:40.166958+0000","last_unstale":"2024-09-20T22:50:08.236394+0000","last_undegraded":"2024-09-20T22:50:08.236394+0000","last_fullsized":"2024-09-20T22:50:08.236394+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'38","last_scrub_stamp":"2024-09-20T22:49:31.040362+0000","last_deep_scrub":"86'38","last_deep_scrub_stamp":"2024-09-20T22:42:43.935677+0000","last_clean_scrub_stamp":"2024-09-20T22:49:31.040362+0000","objects_scrubbed":10,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:55.004800+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":257,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-20T22:50:08.236843+0000","last_change":"2024-09-20T22:48:05.238748+0000","last_active":"2024-09-20T22:50:08.236843+0000","last_peered":"2024-09-20T22:50:08.236843+0000","last_clean":"2024-09-20T22:50:08.236843+0000","last_became_active":"2024-09-20T22:48:05.237756+0000","last_became_peered":"2024-09-20T22:48:05.237756+0000","last_unstale":"2024-09-20T22:50:08.236843+0000","last_undegraded":"2024-09-20T22:50:08.236843+0000","last_fullsized":"2024-09-20T22:50:08.236843+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:47:16.817468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:47:16.817468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:11.149836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"69'2","reported_seq":978,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-20T22:50:08.232078+0000","last_change":"2024-09-20T22:49:27.604083+0000","last_active":"2024-09-20T22:50:08.232078+0000","last_peered":"2024-09-20T22:50:08.232078+0000","last_clean":"2024-09-20T22:50:08.232078+0000","last_became_active":"2024-09-20T22:41:37.148479+0000","last_became_peered":"2024-09-20T22:41:37.148479+0000","last_unstale":"2024-09-20T22:50:08.232078+0000","last_undegraded":"2024-09-20T22:50:08.232078+0000","last_fullsized":"2024-09-20T22:50:08.232078+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2024-09-20T22:49:27.603975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:36.130684+0000","last_clean_scrub_stamp":"2024-09-20T22:49:27.603975+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:37.292200+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00087280599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":304,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-20T22:50:08.231345+0000","last_change":"2024-09-20T22:49:02.723760+0000","last_active":"2024-09-20T22:50:08.231345+0000","last_peered":"2024-09-20T22:50:08.231345+0000","last_clean":"2024-09-20T22:50:08.231345+0000","last_became_active":"2024-09-20T22:48:05.233519+0000","last_became_peered":"2024-09-20T22:48:05.233519+0000","last_unstale":"2024-09-20T22:50:08.231345+0000","last_undegraded":"2024-09-20T22:50:08.231345+0000","last_fullsized":"2024-09-20T22:50:08.231345+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:02.723689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:49:02.723689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:30.036737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":959,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-20T22:50:08.246328+0000","last_change":"2024-09-20T22:49:57.022315+0000","last_active":"2024-09-20T22:50:08.246328+0000","last_peered":"2024-09-20T22:50:08.246328+0000","last_clean":"2024-09-20T22:50:08.246328+0000","last_became_active":"2024-09-20T22:48:05.234193+0000","last_became_peered":"2024-09-20T22:48:05.234193+0000","last_unstale":"2024-09-20T22:50:08.246328+0000","last_undegraded":"2024-09-20T22:50:08.246328+0000","last_fullsized":"2024-09-20T22:50:08.246328+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:57.022244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:49:57.022244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:51:26.737278+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"528'244","reported_seq":869,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-20T22:50:08.235452+0000","last_change":"2024-09-20T22:50:08.235834+0000","last_active":"2024-09-20T22:50:08.235452+0000","last_peered":"2024-09-20T22:50:08.235452+0000","last_clean":"2024-09-20T22:50:08.235452+0000","last_became_active":"2024-09-20T22:49:01.223578+0000","last_became_peered":"2024-09-20T22:49:01.223578+0000","last_unstale":"2024-09-20T22:50:08.235452+0000","last_undegraded":"2024-09-20T22:50:08.235452+0000","last_fullsized":"2024-09-20T22:50:08.235452+0000","mapping_epoch":404,"log_start":"456'176","ondisk_log_start":"456'176","created":94,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"461'178","last_scrub_stamp":"2024-09-20T22:49:01.696205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:49:01.696205+0000","objects_scrubbed":10,"log_size":68,"log_dups_size":318,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.004745778,"stat_sum":{"num_bytes":27923761,"num_objects":12,"num_object_clones":10,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":1,"num_read":117,"num_read_kb":115419,"num_write":92,"num_write_kb":32954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":10294275,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"d6","length":"1"},{"start":"dc","length":"3"}]},{"pgid":"1.2","version":"77'128","reported_seq":1335,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-20T22:50:08.246345+0000","last_change":"2024-09-20T22:49:55.364495+0000","last_active":"2024-09-20T22:50:08.246345+0000","last_peered":"2024-09-20T22:50:08.246345+0000","last_clean":"2024-09-20T22:50:08.246345+0000","last_became_active":"2024-09-20T22:48:08.203028+0000","last_became_peered":"2024-09-20T22:48:08.203028+0000","last_unstale":"2024-09-20T22:50:08.246345+0000","last_undegraded":"2024-09-20T22:50:08.246345+0000","last_fullsized":"2024-09-20T22:50:08.246345+0000","mapping_epoch":407,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":408,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:49:55.364421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:49:55.364421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:59.878790+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"443'18","reported_seq":1031,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-20T22:50:08.236062+0000","last_change":"2024-09-20T22:49:57.739451+0000","last_active":"2024-09-20T22:50:08.236062+0000","last_peered":"2024-09-20T22:50:08.236062+0000","last_clean":"2024-09-20T22:50:08.236062+0000","last_became_active":"2024-09-20T22:47:49.228702+0000","last_became_peered":"2024-09-20T22:47:49.228702+0000","last_unstale":"2024-09-20T22:50:08.236062+0000","last_undegraded":"2024-09-20T22:50:08.236062+0000","last_fullsized":"2024-09-20T22:50:08.236062+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"443'18","last_scrub_stamp":"2024-09-20T22:49:57.739362+0000","last_deep_scrub":"443'18","last_deep_scrub_stamp":"2024-09-20T22:48:44.182517+0000","last_clean_scrub_stamp":"2024-09-20T22:49:57.739362+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:51:00.790911+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":968,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-20T22:50:08.234684+0000","last_change":"2024-09-20T22:49:26.102066+0000","last_active":"2024-09-20T22:50:08.234684+0000","last_peered":"2024-09-20T22:50:08.234684+0000","last_clean":"2024-09-20T22:50:08.234684+0000","last_became_active":"2024-09-20T22:41:38.156837+0000","last_became_peered":"2024-09-20T22:41:38.156837+0000","last_unstale":"2024-09-20T22:50:08.234684+0000","last_undegraded":"2024-09-20T22:50:08.234684+0000","last_fullsized":"2024-09-20T22:50:08.234684+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:26.101994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:49:26.101994+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:29.729787+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"380'2","reported_seq":1005,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-20T22:50:08.231217+0000","last_change":"2024-09-20T22:49:30.148041+0000","last_active":"2024-09-20T22:50:08.231217+0000","last_peered":"2024-09-20T22:50:08.231217+0000","last_clean":"2024-09-20T22:50:08.231217+0000","last_became_active":"2024-09-20T22:47:49.222812+0000","last_became_peered":"2024-09-20T22:47:49.222812+0000","last_unstale":"2024-09-20T22:50:08.231217+0000","last_undegraded":"2024-09-20T22:50:08.231217+0000","last_fullsized":"2024-09-20T22:50:08.231217+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"380'2","last_scrub_stamp":"2024-09-20T22:49:30.147961+0000","last_deep_scrub":"380'2","last_deep_scrub_stamp":"2024-09-20T22:47:40.983039+0000","last_clean_scrub_stamp":"2024-09-20T22:49:30.147961+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:32.024321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001301459,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":958,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-20T22:50:08.232719+0000","last_change":"2024-09-20T22:49:27.195816+0000","last_active":"2024-09-20T22:50:08.232719+0000","last_peered":"2024-09-20T22:50:08.232719+0000","last_clean":"2024-09-20T22:50:08.232719+0000","last_became_active":"2024-09-20T22:41:42.185112+0000","last_became_peered":"2024-09-20T22:41:42.185112+0000","last_unstale":"2024-09-20T22:50:08.232719+0000","last_undegraded":"2024-09-20T22:50:08.232719+0000","last_fullsized":"2024-09-20T22:50:08.232719+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:27.195708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:47:22.881109+0000","last_clean_scrub_stamp":"2024-09-20T22:49:27.195708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:46.992983+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":947,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-20T22:50:08.248405+0000","last_change":"2024-09-20T22:49:04.332292+0000","last_active":"2024-09-20T22:50:08.248405+0000","last_peered":"2024-09-20T22:50:08.248405+0000","last_clean":"2024-09-20T22:50:08.248405+0000","last_became_active":"2024-09-20T22:41:44.830515+0000","last_became_peered":"2024-09-20T22:41:44.830515+0000","last_unstale":"2024-09-20T22:50:08.248405+0000","last_undegraded":"2024-09-20T22:50:08.248405+0000","last_fullsized":"2024-09-20T22:50:08.248405+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:04.332219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:49:04.332219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:26.530832+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"511'223","reported_seq":2427,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-20T22:50:08.248381+0000","last_change":"2024-09-20T22:50:08.248908+0000","last_active":"2024-09-20T22:50:08.248381+0000","last_peered":"2024-09-20T22:50:08.248381+0000","last_clean":"2024-09-20T22:50:08.248381+0000","last_became_active":"2024-09-20T22:48:57.236417+0000","last_became_peered":"2024-09-20T22:48:57.236417+0000","last_unstale":"2024-09-20T22:50:08.248381+0000","last_undegraded":"2024-09-20T22:50:08.248381+0000","last_fullsized":"2024-09-20T22:50:08.248381+0000","mapping_epoch":391,"log_start":"450'190","ondisk_log_start":"450'190","created":94,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'168","last_scrub_stamp":"2024-09-20T22:48:22.483112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:48:22.483112+0000","objects_scrubbed":7,"log_size":33,"log_dups_size":240,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00076496699999999997,"stat_sum":{"num_bytes":16569587,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":57,"num_read_kb":38715,"num_write":75,"num_write_kb":33558,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":19673726,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"d6","length":"1"},{"start":"dc","length":"3"}]},{"pgid":"1.3","version":"77'128","reported_seq":1283,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-20T22:50:08.235411+0000","last_change":"2024-09-20T22:49:49.789854+0000","last_active":"2024-09-20T22:50:08.235411+0000","last_peered":"2024-09-20T22:50:08.235411+0000","last_clean":"2024-09-20T22:50:08.235411+0000","last_became_active":"2024-09-20T22:48:08.208402+0000","last_became_peered":"2024-09-20T22:48:08.208402+0000","last_unstale":"2024-09-20T22:50:08.235411+0000","last_undegraded":"2024-09-20T22:50:08.235411+0000","last_fullsized":"2024-09-20T22:50:08.235411+0000","mapping_epoch":407,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":408,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:49:49.789778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:49:49.789778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:51:06.258069+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"74'12","reported_seq":304,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-20T22:50:08.234305+0000","last_change":"2024-09-20T22:47:49.246080+0000","last_active":"2024-09-20T22:50:08.234305+0000","last_peered":"2024-09-20T22:50:08.234305+0000","last_clean":"2024-09-20T22:50:08.234305+0000","last_became_active":"2024-09-20T22:47:49.224223+0000","last_became_peered":"2024-09-20T22:47:49.224223+0000","last_unstale":"2024-09-20T22:50:08.234305+0000","last_undegraded":"2024-09-20T22:50:08.234305+0000","last_fullsized":"2024-09-20T22:50:08.234305+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-09-20T22:47:39.293455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:47:39.293455+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":304,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-20T22:50:08.232224+0000","last_change":"2024-09-20T22:48:05.233412+0000","last_active":"2024-09-20T22:50:08.232224+0000","last_peered":"2024-09-20T22:50:08.232224+0000","last_clean":"2024-09-20T22:50:08.232224+0000","last_became_active":"2024-09-20T22:48:05.233093+0000","last_became_peered":"2024-09-20T22:48:05.233093+0000","last_unstale":"2024-09-20T22:50:08.232224+0000","last_undegraded":"2024-09-20T22:50:08.232224+0000","last_fullsized":"2024-09-20T22:50:08.232224+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:47:55.576301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:44:19.834641+0000","last_clean_scrub_stamp":"2024-09-20T22:47:55.576301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"75'5","reported_seq":1493,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-20T22:50:08.246229+0000","last_change":"2024-09-20T22:48:44.254029+0000","last_active":"2024-09-20T22:50:08.246229+0000","last_peered":"2024-09-20T22:50:08.246229+0000","last_clean":"2024-09-20T22:50:08.246229+0000","last_became_active":"2024-09-20T22:47:49.214414+0000","last_became_peered":"2024-09-20T22:47:49.214414+0000","last_unstale":"2024-09-20T22:50:08.246229+0000","last_undegraded":"2024-09-20T22:50:08.246229+0000","last_fullsized":"2024-09-20T22:50:08.246229+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-09-20T22:48:44.253907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:36.130684+0000","last_clean_scrub_stamp":"2024-09-20T22:48:44.253907+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:09.481674+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00078631700000000003,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":501,"num_read_kb":496,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":963,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-20T22:50:08.234330+0000","last_change":"2024-09-20T22:48:34.535169+0000","last_active":"2024-09-20T22:50:08.234330+0000","last_peered":"2024-09-20T22:50:08.234330+0000","last_clean":"2024-09-20T22:50:08.234330+0000","last_became_active":"2024-09-20T22:47:49.227371+0000","last_became_peered":"2024-09-20T22:47:49.227371+0000","last_unstale":"2024-09-20T22:50:08.234330+0000","last_undegraded":"2024-09-20T22:50:08.234330+0000","last_fullsized":"2024-09-20T22:50:08.234330+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:48:34.535100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:48:34.535100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:12.708693+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":956,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-20T22:50:08.235407+0000","last_change":"2024-09-20T22:49:10.150277+0000","last_active":"2024-09-20T22:50:08.235407+0000","last_peered":"2024-09-20T22:50:08.235407+0000","last_clean":"2024-09-20T22:50:08.235407+0000","last_became_active":"2024-09-20T22:41:44.199046+0000","last_became_peered":"2024-09-20T22:41:44.199046+0000","last_unstale":"2024-09-20T22:50:08.235407+0000","last_undegraded":"2024-09-20T22:50:08.235407+0000","last_fullsized":"2024-09-20T22:50:08.235407+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:10.150205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:49:10.150205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:12.971375+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.6","version":"527'268","reported_seq":2403,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-20T22:50:08.233835+0000","last_change":"2024-09-20T22:50:08.235053+0000","last_active":"2024-09-20T22:50:08.233835+0000","last_peered":"2024-09-20T22:50:08.233835+0000","last_clean":"2024-09-20T22:50:08.233835+0000","last_became_active":"2024-09-20T22:48:47.234531+0000","last_became_peered":"2024-09-20T22:48:47.234531+0000","last_unstale":"2024-09-20T22:50:08.233835+0000","last_undegraded":"2024-09-20T22:50:08.233835+0000","last_fullsized":"2024-09-20T22:50:08.233835+0000","mapping_epoch":391,"log_start":"443'181","ondisk_log_start":"443'181","created":94,"last_epoch_clean":447,"parent":"0.0","parent_split_bits":0,"last_scrub":"440'177","last_scrub_stamp":"2024-09-20T22:48:40.602342+0000","last_deep_scrub":"440'177","last_deep_scrub_stamp":"2024-09-20T22:48:40.602342+0000","last_clean_scrub_stamp":"2024-09-20T22:48:40.602342+0000","objects_scrubbed":4,"log_size":87,"log_dups_size":314,"ondisk_log_size":87,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0.0017159759999999999,"stat_sum":{"num_bytes":21209978,"num_objects":10,"num_object_clones":8,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":1,"num_read":113,"num_read_kb":107760,"num_write":90,"num_write_kb":30256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":18662529,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"d6","length":"1"},{"start":"dc","length":"3"}]},{"pgid":"4.5","version":"408'20","reported_seq":1185,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-20T22:50:08.242455+0000","last_change":"2024-09-20T22:49:56.004694+0000","last_active":"2024-09-20T22:50:08.242455+0000","last_peered":"2024-09-20T22:50:08.242455+0000","last_clean":"2024-09-20T22:50:08.242455+0000","last_became_active":"2024-09-20T22:48:05.243523+0000","last_became_peered":"2024-09-20T22:48:05.243523+0000","last_unstale":"2024-09-20T22:50:08.242455+0000","last_undegraded":"2024-09-20T22:50:08.242455+0000","last_fullsized":"2024-09-20T22:50:08.242455+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"408'20","last_scrub_stamp":"2024-09-20T22:49:56.004590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:49:56.004590+0000","objects_scrubbed":8,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:51:25.151724+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"67'1","reported_seq":958,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-20T22:50:08.246102+0000","last_change":"2024-09-20T22:49:29.161456+0000","last_active":"2024-09-20T22:50:08.246102+0000","last_peered":"2024-09-20T22:50:08.246102+0000","last_clean":"2024-09-20T22:50:08.246102+0000","last_became_active":"2024-09-20T22:41:38.158917+0000","last_became_peered":"2024-09-20T22:41:38.158917+0000","last_unstale":"2024-09-20T22:50:08.246102+0000","last_undegraded":"2024-09-20T22:50:08.246102+0000","last_fullsized":"2024-09-20T22:50:08.246102+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-09-20T22:49:29.161334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:49:29.161334+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:43.763458+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"77'128","reported_seq":1307,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-20T22:50:08.246293+0000","last_change":"2024-09-20T22:49:46.391695+0000","last_active":"2024-09-20T22:50:08.246293+0000","last_peered":"2024-09-20T22:50:08.246293+0000","last_clean":"2024-09-20T22:50:08.246293+0000","last_became_active":"2024-09-20T22:47:52.206169+0000","last_became_peered":"2024-09-20T22:47:52.206169+0000","last_unstale":"2024-09-20T22:50:08.246293+0000","last_undegraded":"2024-09-20T22:50:08.246293+0000","last_fullsized":"2024-09-20T22:50:08.246293+0000","mapping_epoch":391,"log_start":"77'128","ondisk_log_start":"77'128","created":19,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:49:46.391614+0000","last_deep_scrub":"77'128","last_deep_scrub_stamp":"2024-09-20T22:47:22.417099+0000","last_clean_scrub_stamp":"2024-09-20T22:49:46.391614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:51.817750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":951,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-20T22:50:08.248160+0000","last_change":"2024-09-20T22:48:35.359093+0000","last_active":"2024-09-20T22:50:08.248160+0000","last_peered":"2024-09-20T22:50:08.248160+0000","last_clean":"2024-09-20T22:50:08.248160+0000","last_became_active":"2024-09-20T22:41:42.184387+0000","last_became_peered":"2024-09-20T22:41:42.184387+0000","last_unstale":"2024-09-20T22:50:08.248160+0000","last_undegraded":"2024-09-20T22:50:08.248160+0000","last_fullsized":"2024-09-20T22:50:08.248160+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:48:35.359016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:48:35.359016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:09.481707+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":967,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-20T22:50:08.234162+0000","last_change":"2024-09-20T22:49:57.715494+0000","last_active":"2024-09-20T22:50:08.234162+0000","last_peered":"2024-09-20T22:50:08.234162+0000","last_clean":"2024-09-20T22:50:08.234162+0000","last_became_active":"2024-09-20T22:48:05.241602+0000","last_became_peered":"2024-09-20T22:48:05.241602+0000","last_unstale":"2024-09-20T22:50:08.234162+0000","last_undegraded":"2024-09-20T22:50:08.234162+0000","last_fullsized":"2024-09-20T22:50:08.234162+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:57.715416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:49:57.715416+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:51:26.966224+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"518'313","reported_seq":2516,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-20T22:50:08.242666+0000","last_change":"2024-09-20T22:50:08.243915+0000","last_active":"2024-09-20T22:50:08.242666+0000","last_peered":"2024-09-20T22:50:08.242666+0000","last_clean":"2024-09-20T22:50:08.242666+0000","last_became_active":"2024-09-20T22:47:52.208200+0000","last_became_peered":"2024-09-20T22:47:52.208200+0000","last_unstale":"2024-09-20T22:50:08.242666+0000","last_undegraded":"2024-09-20T22:50:08.242666+0000","last_fullsized":"2024-09-20T22:50:08.242666+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"460'246","last_scrub_stamp":"2024-09-20T22:49:01.214729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:49:01.214729+0000","objects_scrubbed":8,"log_size":313,"log_dups_size":0,"ondisk_log_size":313,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:14.487340+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0015358839999999999,"stat_sum":{"num_bytes":24977291,"num_objects":9,"num_object_clones":7,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":114,"num_read_kb":106336,"num_write":68,"num_write_kb":29565,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":13529610,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"d6","length":"1"},{"start":"dc","length":"3"}]},{"pgid":"4.e","version":"74'11","reported_seq":269,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-20T22:50:08.238617+0000","last_change":"2024-09-20T22:48:05.271944+0000","last_active":"2024-09-20T22:50:08.238617+0000","last_peered":"2024-09-20T22:50:08.238617+0000","last_clean":"2024-09-20T22:50:08.238617+0000","last_became_active":"2024-09-20T22:48:05.248271+0000","last_became_peered":"2024-09-20T22:48:05.248271+0000","last_unstale":"2024-09-20T22:50:08.238617+0000","last_undegraded":"2024-09-20T22:50:08.238617+0000","last_fullsized":"2024-09-20T22:50:08.238617+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-09-20T22:47:59.092235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:47:59.092235+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:11.149889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":265,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-20T22:50:08.236229+0000","last_change":"2024-09-20T22:49:30.998080+0000","last_active":"2024-09-20T22:50:08.236229+0000","last_peered":"2024-09-20T22:50:08.236229+0000","last_clean":"2024-09-20T22:50:08.236229+0000","last_became_active":"2024-09-20T22:48:05.254688+0000","last_became_peered":"2024-09-20T22:48:05.254688+0000","last_unstale":"2024-09-20T22:50:08.236229+0000","last_undegraded":"2024-09-20T22:50:08.236229+0000","last_fullsized":"2024-09-20T22:50:08.236229+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:30.998001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:49:30.998001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:47.839033+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":963,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-20T22:50:08.234072+0000","last_change":"2024-09-20T22:49:48.693437+0000","last_active":"2024-09-20T22:50:08.234072+0000","last_peered":"2024-09-20T22:50:08.234072+0000","last_clean":"2024-09-20T22:50:08.234072+0000","last_became_active":"2024-09-20T22:48:05.233778+0000","last_became_peered":"2024-09-20T22:48:05.233778+0000","last_unstale":"2024-09-20T22:50:08.234072+0000","last_undegraded":"2024-09-20T22:50:08.234072+0000","last_fullsized":"2024-09-20T22:50:08.234072+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:48.693333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:49:48.693333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:56.935700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":939,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-20T22:50:08.246464+0000","last_change":"2024-09-20T22:48:40.127195+0000","last_active":"2024-09-20T22:50:08.246464+0000","last_peered":"2024-09-20T22:50:08.246464+0000","last_clean":"2024-09-20T22:50:08.246464+0000","last_became_active":"2024-09-20T22:41:44.832009+0000","last_became_peered":"2024-09-20T22:41:44.832009+0000","last_unstale":"2024-09-20T22:50:08.246464+0000","last_undegraded":"2024-09-20T22:50:08.246464+0000","last_fullsized":"2024-09-20T22:50:08.246464+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:48:40.127123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:48:40.127123+0000","last_clean_scrub_stamp":"2024-09-20T22:48:40.127123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:12.131590+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.e","version":"528'406","reported_seq":2633,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-20T22:50:08.236954+0000","last_change":"2024-09-20T22:50:08.237204+0000","last_active":"2024-09-20T22:50:08.236954+0000","last_peered":"2024-09-20T22:50:08.236954+0000","last_clean":"2024-09-20T22:50:08.236954+0000","last_became_active":"2024-09-20T22:47:52.209452+0000","last_became_peered":"2024-09-20T22:47:52.209452+0000","last_unstale":"2024-09-20T22:50:08.236954+0000","last_undegraded":"2024-09-20T22:50:08.236954+0000","last_fullsized":"2024-09-20T22:50:08.236954+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'360","last_scrub_stamp":"2024-09-20T22:49:25.149145+0000","last_deep_scrub":"174'60","last_deep_scrub_stamp":"2024-09-20T22:44:18.301776+0000","last_clean_scrub_stamp":"2024-09-20T22:49:25.149145+0000","objects_scrubbed":11,"log_size":406,"log_dups_size":0,"ondisk_log_size":406,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:45.785660+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.0048260949999999999,"stat_sum":{"num_bytes":27613132,"num_objects":13,"num_object_clones":11,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":1,"num_read":166,"num_read_kb":173052,"num_write":109,"num_write_kb":43661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":23110741,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"d6","length":"1"},{"start":"dc","length":"3"}]},{"pgid":"1.8","version":"77'128","reported_seq":1226,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-20T22:50:08.231397+0000","last_change":"2024-09-20T22:49:56.715035+0000","last_active":"2024-09-20T22:50:08.231397+0000","last_peered":"2024-09-20T22:50:08.231397+0000","last_clean":"2024-09-20T22:50:08.231397+0000","last_became_active":"2024-09-20T22:42:52.405594+0000","last_became_peered":"2024-09-20T22:42:52.405594+0000","last_unstale":"2024-09-20T22:50:08.231397+0000","last_undegraded":"2024-09-20T22:50:08.231397+0000","last_fullsized":"2024-09-20T22:50:08.231397+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:49:56.714919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:49:56.714919+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:51:24.489539+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"74'17","reported_seq":283,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-20T22:50:08.239085+0000","last_change":"2024-09-20T22:49:01.949542+0000","last_active":"2024-09-20T22:50:08.239085+0000","last_peered":"2024-09-20T22:50:08.239085+0000","last_clean":"2024-09-20T22:50:08.239085+0000","last_became_active":"2024-09-20T22:48:05.247971+0000","last_became_peered":"2024-09-20T22:48:05.247971+0000","last_unstale":"2024-09-20T22:50:08.239085+0000","last_undegraded":"2024-09-20T22:50:08.239085+0000","last_fullsized":"2024-09-20T22:50:08.239085+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'17","last_scrub_stamp":"2024-09-20T22:49:01.949451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:49:01.949451+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:11.149998+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":28,"num_read_kb":18,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":507,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-20T22:50:08.233037+0000","last_change":"2024-09-20T22:49:32.138822+0000","last_active":"2024-09-20T22:50:08.233037+0000","last_peered":"2024-09-20T22:50:08.233037+0000","last_clean":"2024-09-20T22:50:08.233037+0000","last_became_active":"2024-09-20T22:48:05.240481+0000","last_became_peered":"2024-09-20T22:48:05.240481+0000","last_unstale":"2024-09-20T22:50:08.233037+0000","last_undegraded":"2024-09-20T22:50:08.233037+0000","last_fullsized":"2024-09-20T22:50:08.233037+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:32.138751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:35.492562+0000","last_clean_scrub_stamp":"2024-09-20T22:49:32.138751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:44.032030+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":975,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-20T22:50:08.236090+0000","last_change":"2024-09-20T22:49:09.188448+0000","last_active":"2024-09-20T22:50:08.236090+0000","last_peered":"2024-09-20T22:50:08.236090+0000","last_clean":"2024-09-20T22:50:08.236090+0000","last_became_active":"2024-09-20T22:48:05.250175+0000","last_became_peered":"2024-09-20T22:48:05.250175+0000","last_unstale":"2024-09-20T22:50:08.236090+0000","last_undegraded":"2024-09-20T22:50:08.236090+0000","last_fullsized":"2024-09-20T22:50:08.236090+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:09.188362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:49:09.188362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:36.362440+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":974,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-20T22:50:08.231500+0000","last_change":"2024-09-20T22:49:26.152417+0000","last_active":"2024-09-20T22:50:08.231500+0000","last_peered":"2024-09-20T22:50:08.231500+0000","last_clean":"2024-09-20T22:50:08.231500+0000","last_became_active":"2024-09-20T22:48:05.239006+0000","last_became_peered":"2024-09-20T22:48:05.239006+0000","last_unstale":"2024-09-20T22:50:08.231500+0000","last_undegraded":"2024-09-20T22:50:08.231500+0000","last_fullsized":"2024-09-20T22:50:08.231500+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:26.152316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:49:26.152316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:40.036383+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"518'495","reported_seq":2683,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-20T22:50:08.234974+0000","last_change":"2024-09-20T22:50:08.235441+0000","last_active":"2024-09-20T22:50:08.234974+0000","last_peered":"2024-09-20T22:50:08.234974+0000","last_clean":"2024-09-20T22:50:08.234974+0000","last_became_active":"2024-09-20T22:47:52.216198+0000","last_became_peered":"2024-09-20T22:47:52.216198+0000","last_unstale":"2024-09-20T22:50:08.234974+0000","last_undegraded":"2024-09-20T22:50:08.234974+0000","last_fullsized":"2024-09-20T22:50:08.234974+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"481'456","last_scrub_stamp":"2024-09-20T22:49:22.637524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:49:22.637524+0000","objects_scrubbed":9,"log_size":495,"log_dups_size":0,"ondisk_log_size":495,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:39.039699+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.00081402700000000004,"stat_sum":{"num_bytes":23951363,"num_objects":11,"num_object_clones":8,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":2,"num_read":232,"num_read_kb":231557,"num_write":142,"num_write_kb":48100,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":53,"num_bytes_recovered":14983556,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"d6","length":"1"},{"start":"dc","length":"3"}]},{"pgid":"1.9","version":"77'128","reported_seq":1266,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-20T22:50:08.245619+0000","last_change":"2024-09-20T22:49:52.301456+0000","last_active":"2024-09-20T22:50:08.245619+0000","last_peered":"2024-09-20T22:50:08.245619+0000","last_clean":"2024-09-20T22:50:08.245619+0000","last_became_active":"2024-09-20T22:42:52.400353+0000","last_became_peered":"2024-09-20T22:42:52.400353+0000","last_unstale":"2024-09-20T22:50:08.245619+0000","last_undegraded":"2024-09-20T22:50:08.245619+0000","last_fullsized":"2024-09-20T22:50:08.245619+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:49:52.301348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:49:52.301348+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:51:11.793390+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"74'10","reported_seq":269,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-20T22:50:08.236745+0000","last_change":"2024-09-20T22:48:05.262827+0000","last_active":"2024-09-20T22:50:08.236745+0000","last_peered":"2024-09-20T22:50:08.236745+0000","last_clean":"2024-09-20T22:50:08.236745+0000","last_became_active":"2024-09-20T22:48:05.242811+0000","last_became_peered":"2024-09-20T22:48:05.242811+0000","last_unstale":"2024-09-20T22:50:08.236745+0000","last_undegraded":"2024-09-20T22:50:08.236745+0000","last_fullsized":"2024-09-20T22:50:08.236745+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-09-20T22:47:21.092309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:47:21.092309+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:11.149805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":975,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-20T22:50:08.245580+0000","last_change":"2024-09-20T22:48:05.234296+0000","last_active":"2024-09-20T22:50:08.245580+0000","last_peered":"2024-09-20T22:50:08.245580+0000","last_clean":"2024-09-20T22:50:08.245580+0000","last_became_active":"2024-09-20T22:48:05.233999+0000","last_became_peered":"2024-09-20T22:48:05.233999+0000","last_unstale":"2024-09-20T22:50:08.245580+0000","last_undegraded":"2024-09-20T22:50:08.245580+0000","last_fullsized":"2024-09-20T22:50:08.245580+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:45:57.194715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:45:57.194715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:12.131401+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"261'9","reported_seq":1078,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-20T22:50:08.231940+0000","last_change":"2024-09-20T22:49:57.205107+0000","last_active":"2024-09-20T22:50:08.231940+0000","last_peered":"2024-09-20T22:50:08.231940+0000","last_clean":"2024-09-20T22:50:08.231940+0000","last_became_active":"2024-09-20T22:41:42.185863+0000","last_became_peered":"2024-09-20T22:41:42.185863+0000","last_unstale":"2024-09-20T22:50:08.231940+0000","last_undegraded":"2024-09-20T22:50:08.231940+0000","last_fullsized":"2024-09-20T22:50:08.231940+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'9","last_scrub_stamp":"2024-09-20T22:49:57.204987+0000","last_deep_scrub":"261'9","last_deep_scrub_stamp":"2024-09-20T22:46:50.974735+0000","last_clean_scrub_stamp":"2024-09-20T22:49:57.204987+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:51:09.810048+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":265,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-20T22:50:08.238420+0000","last_change":"2024-09-20T22:49:54.058299+0000","last_active":"2024-09-20T22:50:08.238420+0000","last_peered":"2024-09-20T22:50:08.238420+0000","last_clean":"2024-09-20T22:50:08.238420+0000","last_became_active":"2024-09-20T22:48:05.250284+0000","last_became_peered":"2024-09-20T22:48:05.250284+0000","last_unstale":"2024-09-20T22:50:08.238420+0000","last_undegraded":"2024-09-20T22:50:08.238420+0000","last_fullsized":"2024-09-20T22:50:08.238420+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:54.058184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:49:54.058184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:51:00.325103+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.8","version":"527'437","reported_seq":2501,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-20T22:50:08.232965+0000","last_change":"2024-09-20T22:50:08.233265+0000","last_active":"2024-09-20T22:50:08.232965+0000","last_peered":"2024-09-20T22:50:08.232965+0000","last_clean":"2024-09-20T22:50:08.232965+0000","last_became_active":"2024-09-20T22:48:31.224359+0000","last_became_peered":"2024-09-20T22:48:31.224359+0000","last_unstale":"2024-09-20T22:50:08.232965+0000","last_undegraded":"2024-09-20T22:50:08.232965+0000","last_fullsized":"2024-09-20T22:50:08.232965+0000","mapping_epoch":407,"log_start":"426'268","ondisk_log_start":"426'268","created":94,"last_epoch_clean":431,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'319","last_scrub_stamp":"2024-09-20T22:48:59.219393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:48:59.219393+0000","objects_scrubbed":8,"log_size":169,"log_dups_size":445,"ondisk_log_size":169,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:23.194482+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0018288880000000001,"stat_sum":{"num_bytes":34739607,"num_objects":16,"num_object_clones":13,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":198,"num_read_kb":189772,"num_write":158,"num_write_kb":53832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":26027409,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"d6","length":"1"},{"start":"dc","length":"3"}]},{"pgid":"4.b","version":"74'9","reported_seq":324,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-20T22:50:08.232352+0000","last_change":"2024-09-20T22:48:05.256987+0000","last_active":"2024-09-20T22:50:08.232352+0000","last_peered":"2024-09-20T22:50:08.232352+0000","last_clean":"2024-09-20T22:50:08.232352+0000","last_became_active":"2024-09-20T22:48:05.233267+0000","last_became_peered":"2024-09-20T22:48:05.233267+0000","last_unstale":"2024-09-20T22:50:08.232352+0000","last_undegraded":"2024-09-20T22:50:08.232352+0000","last_fullsized":"2024-09-20T22:50:08.232352+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-09-20T22:48:03.584242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:48:03.584242+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":959,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-20T22:50:08.233140+0000","last_change":"2024-09-20T22:49:05.638528+0000","last_active":"2024-09-20T22:50:08.233140+0000","last_peered":"2024-09-20T22:50:08.233140+0000","last_clean":"2024-09-20T22:50:08.233140+0000","last_became_active":"2024-09-20T22:41:38.155953+0000","last_became_peered":"2024-09-20T22:41:38.155953+0000","last_unstale":"2024-09-20T22:50:08.233140+0000","last_undegraded":"2024-09-20T22:50:08.233140+0000","last_fullsized":"2024-09-20T22:50:08.233140+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:05.638457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:49:05.638457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:25.666047+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":982,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-20T22:50:08.232042+0000","last_change":"2024-09-20T22:49:02.242848+0000","last_active":"2024-09-20T22:50:08.232042+0000","last_peered":"2024-09-20T22:50:08.232042+0000","last_clean":"2024-09-20T22:50:08.232042+0000","last_became_active":"2024-09-20T22:48:05.240613+0000","last_became_peered":"2024-09-20T22:48:05.240613+0000","last_unstale":"2024-09-20T22:50:08.232042+0000","last_undegraded":"2024-09-20T22:50:08.232042+0000","last_fullsized":"2024-09-20T22:50:08.232042+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:02.242757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:49:02.242757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:24.494551+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":298,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-20T22:50:08.232350+0000","last_change":"2024-09-20T22:49:31.666061+0000","last_active":"2024-09-20T22:50:08.232350+0000","last_peered":"2024-09-20T22:50:08.232350+0000","last_clean":"2024-09-20T22:50:08.232350+0000","last_became_active":"2024-09-20T22:47:49.219486+0000","last_became_peered":"2024-09-20T22:47:49.219486+0000","last_unstale":"2024-09-20T22:50:08.232350+0000","last_undegraded":"2024-09-20T22:50:08.232350+0000","last_fullsized":"2024-09-20T22:50:08.232350+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:31.665954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:49:31.665954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:38.433613+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"525'613","reported_seq":2836,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-20T22:50:08.233326+0000","last_change":"2024-09-20T22:50:08.233577+0000","last_active":"2024-09-20T22:50:08.233326+0000","last_peered":"2024-09-20T22:50:08.233326+0000","last_clean":"2024-09-20T22:50:08.233326+0000","last_became_active":"2024-09-20T22:48:27.272202+0000","last_became_peered":"2024-09-20T22:48:27.272202+0000","last_unstale":"2024-09-20T22:50:08.233326+0000","last_undegraded":"2024-09-20T22:50:08.233326+0000","last_fullsized":"2024-09-20T22:50:08.233326+0000","mapping_epoch":407,"log_start":"423'280","ondisk_log_start":"423'280","created":94,"last_epoch_clean":427,"parent":"0.0","parent_split_bits":0,"last_scrub":"423'271","last_scrub_stamp":"2024-09-20T22:48:23.382092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:48:23.382092+0000","objects_scrubbed":8,"log_size":333,"log_dups_size":649,"ondisk_log_size":333,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00081258399999999998,"stat_sum":{"num_bytes":59196432,"num_objects":26,"num_object_clones":21,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":1,"num_read":311,"num_read_kb":319052,"num_write":237,"num_write_kb":88911,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":65,"num_bytes_recovered":38710860,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"d6","length":"1"},{"start":"dc","length":"3"}]},{"pgid":"4.a","version":"113'28","reported_seq":1003,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-20T22:50:08.241377+0000","last_change":"2024-09-20T22:49:03.201199+0000","last_active":"2024-09-20T22:50:08.241377+0000","last_peered":"2024-09-20T22:50:08.241377+0000","last_clean":"2024-09-20T22:50:08.241377+0000","last_became_active":"2024-09-20T22:41:40.252011+0000","last_became_peered":"2024-09-20T22:41:40.252011+0000","last_unstale":"2024-09-20T22:50:08.241377+0000","last_undegraded":"2024-09-20T22:50:08.241377+0000","last_fullsized":"2024-09-20T22:50:08.241377+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'28","last_scrub_stamp":"2024-09-20T22:49:03.201108+0000","last_deep_scrub":"113'28","last_deep_scrub_stamp":"2024-09-20T22:43:12.725900+0000","last_clean_scrub_stamp":"2024-09-20T22:49:03.201108+0000","objects_scrubbed":9,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:31.362019+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":958,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-20T22:50:08.248574+0000","last_change":"2024-09-20T22:49:12.324218+0000","last_active":"2024-09-20T22:50:08.248574+0000","last_peered":"2024-09-20T22:50:08.248574+0000","last_clean":"2024-09-20T22:50:08.248574+0000","last_became_active":"2024-09-20T22:41:38.161524+0000","last_became_peered":"2024-09-20T22:41:38.161524+0000","last_unstale":"2024-09-20T22:50:08.248574+0000","last_undegraded":"2024-09-20T22:50:08.248574+0000","last_fullsized":"2024-09-20T22:50:08.248574+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:12.324148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:49:12.324148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:36.961267+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":970,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-20T22:50:08.231878+0000","last_change":"2024-09-20T22:49:04.244778+0000","last_active":"2024-09-20T22:50:08.231878+0000","last_peered":"2024-09-20T22:50:08.231878+0000","last_clean":"2024-09-20T22:50:08.231878+0000","last_became_active":"2024-09-20T22:47:49.223220+0000","last_became_peered":"2024-09-20T22:47:49.223220+0000","last_unstale":"2024-09-20T22:50:08.231878+0000","last_undegraded":"2024-09-20T22:50:08.231878+0000","last_fullsized":"2024-09-20T22:50:08.231878+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:04.244659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:49:04.244659+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:08.596273+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":954,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-20T22:50:08.245493+0000","last_change":"2024-09-20T22:49:30.310175+0000","last_active":"2024-09-20T22:50:08.245493+0000","last_peered":"2024-09-20T22:50:08.245493+0000","last_clean":"2024-09-20T22:50:08.245493+0000","last_became_active":"2024-09-20T22:41:44.199395+0000","last_became_peered":"2024-09-20T22:41:44.199395+0000","last_unstale":"2024-09-20T22:50:08.245493+0000","last_undegraded":"2024-09-20T22:50:08.245493+0000","last_fullsized":"2024-09-20T22:50:08.245493+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:30.310069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:47:18.329209+0000","last_clean_scrub_stamp":"2024-09-20T22:49:30.310069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:44.675926+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.a","version":"528'507","reported_seq":2662,"reported_epoch":528,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T22:50:08.251216+0000","last_change":"2024-09-20T22:50:08.248035+0000","last_active":"2024-09-20T22:50:08.251216+0000","last_peered":"2024-09-20T22:50:08.251216+0000","last_clean":"2024-09-20T22:50:08.251216+0000","last_became_active":"2024-09-20T22:42:52.405085+0000","last_became_peered":"2024-09-20T22:42:52.405085+0000","last_unstale":"2024-09-20T22:50:08.251216+0000","last_undegraded":"2024-09-20T22:50:08.251216+0000","last_fullsized":"2024-09-20T22:50:08.251216+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"505'466","last_scrub_stamp":"2024-09-20T22:49:47.088165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:49:47.088165+0000","objects_scrubbed":13,"log_size":507,"log_dups_size":0,"ondisk_log_size":507,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:49.184716+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.0013055740000000001,"stat_sum":{"num_bytes":35889860,"num_objects":15,"num_object_clones":12,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":294,"num_read_kb":332020,"num_write":182,"num_write_kb":69363,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"d6","length":"1"},{"start":"dc","length":"1"}]},{"pgid":"4.9","version":"74'12","reported_seq":269,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-20T22:50:08.236645+0000","last_change":"2024-09-20T22:48:05.284523+0000","last_active":"2024-09-20T22:50:08.236645+0000","last_peered":"2024-09-20T22:50:08.236645+0000","last_clean":"2024-09-20T22:50:08.236645+0000","last_became_active":"2024-09-20T22:48:05.249438+0000","last_became_peered":"2024-09-20T22:48:05.249438+0000","last_unstale":"2024-09-20T22:50:08.236645+0000","last_undegraded":"2024-09-20T22:50:08.236645+0000","last_fullsized":"2024-09-20T22:50:08.236645+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-09-20T22:47:58.421473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:47:58.421473+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:11.149917+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":21,"num_read_kb":12,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":990,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-20T22:50:08.248028+0000","last_change":"2024-09-20T22:49:24.300511+0000","last_active":"2024-09-20T22:50:08.248028+0000","last_peered":"2024-09-20T22:50:08.248028+0000","last_clean":"2024-09-20T22:50:08.248028+0000","last_became_active":"2024-09-20T22:48:05.240225+0000","last_became_peered":"2024-09-20T22:48:05.240225+0000","last_unstale":"2024-09-20T22:50:08.248028+0000","last_undegraded":"2024-09-20T22:50:08.248028+0000","last_fullsized":"2024-09-20T22:50:08.248028+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:24.300441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:46:52.465940+0000","last_clean_scrub_stamp":"2024-09-20T22:49:24.300441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:46.323016+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":978,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-20T22:50:08.231794+0000","last_change":"2024-09-20T22:49:54.216094+0000","last_active":"2024-09-20T22:50:08.231794+0000","last_peered":"2024-09-20T22:50:08.231794+0000","last_clean":"2024-09-20T22:50:08.231794+0000","last_became_active":"2024-09-20T22:47:49.222088+0000","last_became_peered":"2024-09-20T22:47:49.222088+0000","last_unstale":"2024-09-20T22:50:08.231794+0000","last_undegraded":"2024-09-20T22:50:08.231794+0000","last_fullsized":"2024-09-20T22:50:08.231794+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:54.216022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:46:01.828080+0000","last_clean_scrub_stamp":"2024-09-20T22:49:54.216022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:56.397469+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":939,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-20T22:50:08.245400+0000","last_change":"2024-09-20T22:47:38.019602+0000","last_active":"2024-09-20T22:50:08.245400+0000","last_peered":"2024-09-20T22:50:08.245400+0000","last_clean":"2024-09-20T22:50:08.245400+0000","last_became_active":"2024-09-20T22:41:44.192609+0000","last_became_peered":"2024-09-20T22:41:44.192609+0000","last_unstale":"2024-09-20T22:50:08.245400+0000","last_undegraded":"2024-09-20T22:50:08.245400+0000","last_fullsized":"2024-09-20T22:50:08.245400+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:47:38.019533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:55.170379+0000","last_clean_scrub_stamp":"2024-09-20T22:47:38.019533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:12.131514+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"528'769","reported_seq":2925,"reported_epoch":528,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T22:50:08.244931+0000","last_change":"2024-09-20T22:50:08.236430+0000","last_active":"2024-09-20T22:50:08.244931+0000","last_peered":"2024-09-20T22:50:08.244931+0000","last_clean":"2024-09-20T22:50:08.244931+0000","last_became_active":"2024-09-20T22:42:52.409642+0000","last_became_peered":"2024-09-20T22:42:52.409642+0000","last_unstale":"2024-09-20T22:50:08.244931+0000","last_undegraded":"2024-09-20T22:50:08.244931+0000","last_fullsized":"2024-09-20T22:50:08.244931+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"194'116","last_scrub_stamp":"2024-09-20T22:44:37.097674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:44:37.097674+0000","objects_scrubbed":3,"log_size":769,"log_dups_size":0,"ondisk_log_size":769,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.063923616000000003,"stat_sum":{"num_bytes":53661122,"num_objects":25,"num_object_clones":21,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":2,"num_read":405,"num_read_kb":450576,"num_write":284,"num_write_kb":90656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"d6","length":"1"},{"start":"dc","length":"1"},{"start":"de","length":"1"}]},{"pgid":"4.8","version":"74'15","reported_seq":980,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-20T22:50:08.231786+0000","last_change":"2024-09-20T22:49:47.687063+0000","last_active":"2024-09-20T22:50:08.231786+0000","last_peered":"2024-09-20T22:50:08.231786+0000","last_clean":"2024-09-20T22:50:08.231786+0000","last_became_active":"2024-09-20T22:41:40.252113+0000","last_became_peered":"2024-09-20T22:41:40.252113+0000","last_unstale":"2024-09-20T22:50:08.231786+0000","last_undegraded":"2024-09-20T22:50:08.231786+0000","last_fullsized":"2024-09-20T22:50:08.231786+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-09-20T22:49:47.686984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:49:47.686984+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:51:06.196255+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'2","reported_seq":1018,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-20T22:50:08.249529+0000","last_change":"2024-09-20T22:49:22.379093+0000","last_active":"2024-09-20T22:50:08.249529+0000","last_peered":"2024-09-20T22:50:08.249529+0000","last_clean":"2024-09-20T22:50:08.249529+0000","last_became_active":"2024-09-20T22:48:05.238098+0000","last_became_peered":"2024-09-20T22:48:05.238098+0000","last_unstale":"2024-09-20T22:50:08.249529+0000","last_undegraded":"2024-09-20T22:50:08.249529+0000","last_fullsized":"2024-09-20T22:50:08.249529+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-09-20T22:49:22.379012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:49:22.379012+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:31.674675+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"408'10","reported_seq":1089,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-20T22:50:08.246077+0000","last_change":"2024-09-20T22:49:47.363588+0000","last_active":"2024-09-20T22:50:08.246077+0000","last_peered":"2024-09-20T22:50:08.246077+0000","last_clean":"2024-09-20T22:50:08.246077+0000","last_became_active":"2024-09-20T22:48:05.239344+0000","last_became_peered":"2024-09-20T22:48:05.239344+0000","last_unstale":"2024-09-20T22:50:08.246077+0000","last_undegraded":"2024-09-20T22:50:08.246077+0000","last_fullsized":"2024-09-20T22:50:08.246077+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"408'10","last_scrub_stamp":"2024-09-20T22:49:47.363515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:49:47.363515+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:51:01.473177+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":959,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-20T22:50:08.231788+0000","last_change":"2024-09-20T22:49:50.693099+0000","last_active":"2024-09-20T22:50:08.231788+0000","last_peered":"2024-09-20T22:50:08.231788+0000","last_clean":"2024-09-20T22:50:08.231788+0000","last_became_active":"2024-09-20T22:47:49.227569+0000","last_became_peered":"2024-09-20T22:47:49.227569+0000","last_unstale":"2024-09-20T22:50:08.231788+0000","last_undegraded":"2024-09-20T22:50:08.231788+0000","last_fullsized":"2024-09-20T22:50:08.231788+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:50.693003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:49:50.693003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:51:08.138542+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":979,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-20T22:50:08.242005+0000","last_change":"2024-09-20T22:49:02.203750+0000","last_active":"2024-09-20T22:50:08.242005+0000","last_peered":"2024-09-20T22:50:08.242005+0000","last_clean":"2024-09-20T22:50:08.242005+0000","last_became_active":"2024-09-20T22:47:49.213292+0000","last_became_peered":"2024-09-20T22:47:49.213292+0000","last_unstale":"2024-09-20T22:50:08.242005+0000","last_undegraded":"2024-09-20T22:50:08.242005+0000","last_fullsized":"2024-09-20T22:50:08.242005+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:02.203661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:49:02.203661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:30.443605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"74'6","reported_seq":978,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-20T22:50:08.246906+0000","last_change":"2024-09-20T22:49:48.113810+0000","last_active":"2024-09-20T22:50:08.246906+0000","last_peered":"2024-09-20T22:50:08.246906+0000","last_clean":"2024-09-20T22:50:08.246906+0000","last_became_active":"2024-09-20T22:43:09.116887+0000","last_became_peered":"2024-09-20T22:43:09.116887+0000","last_unstale":"2024-09-20T22:50:08.246906+0000","last_undegraded":"2024-09-20T22:50:08.246906+0000","last_fullsized":"2024-09-20T22:50:08.246906+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'6","last_scrub_stamp":"2024-09-20T22:49:48.113740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:49:48.113740+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:51:07.964989+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":951,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-20T22:50:08.234464+0000","last_change":"2024-09-20T22:48:35.550867+0000","last_active":"2024-09-20T22:50:08.234464+0000","last_peered":"2024-09-20T22:50:08.234464+0000","last_clean":"2024-09-20T22:50:08.234464+0000","last_became_active":"2024-09-20T22:41:42.184930+0000","last_became_peered":"2024-09-20T22:41:42.184930+0000","last_unstale":"2024-09-20T22:50:08.234464+0000","last_undegraded":"2024-09-20T22:50:08.234464+0000","last_fullsized":"2024-09-20T22:50:08.234464+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:48:35.550781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:09.705632+0000","last_clean_scrub_stamp":"2024-09-20T22:48:35.550781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:12.708731+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"74'1","reported_seq":967,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-20T22:50:08.248071+0000","last_change":"2024-09-20T22:48:40.309005+0000","last_active":"2024-09-20T22:50:08.248071+0000","last_peered":"2024-09-20T22:50:08.248071+0000","last_clean":"2024-09-20T22:50:08.248071+0000","last_became_active":"2024-09-20T22:48:05.237882+0000","last_became_peered":"2024-09-20T22:48:05.237882+0000","last_unstale":"2024-09-20T22:50:08.248071+0000","last_undegraded":"2024-09-20T22:50:08.248071+0000","last_fullsized":"2024-09-20T22:50:08.248071+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-09-20T22:48:40.308879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:48:40.308879+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:12.494711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"74'9","reported_seq":307,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-20T22:50:08.234174+0000","last_change":"2024-09-20T22:49:11.739149+0000","last_active":"2024-09-20T22:50:08.234174+0000","last_peered":"2024-09-20T22:50:08.234174+0000","last_clean":"2024-09-20T22:50:08.234174+0000","last_became_active":"2024-09-20T22:47:49.231315+0000","last_became_peered":"2024-09-20T22:47:49.231315+0000","last_unstale":"2024-09-20T22:50:08.234174+0000","last_undegraded":"2024-09-20T22:50:08.234174+0000","last_fullsized":"2024-09-20T22:50:08.234174+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-09-20T22:49:11.739076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:49:11.739076+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:17.900700+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":982,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-20T22:50:08.245192+0000","last_change":"2024-09-20T22:49:25.302834+0000","last_active":"2024-09-20T22:50:08.245192+0000","last_peered":"2024-09-20T22:50:08.245192+0000","last_clean":"2024-09-20T22:50:08.245192+0000","last_became_active":"2024-09-20T22:48:05.238801+0000","last_became_peered":"2024-09-20T22:48:05.238801+0000","last_unstale":"2024-09-20T22:50:08.245192+0000","last_undegraded":"2024-09-20T22:50:08.245192+0000","last_fullsized":"2024-09-20T22:50:08.245192+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:25.302756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:49:25.302756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:30.192311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":943,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-20T22:50:08.246762+0000","last_change":"2024-09-20T22:46:48.012442+0000","last_active":"2024-09-20T22:50:08.246762+0000","last_peered":"2024-09-20T22:50:08.246762+0000","last_clean":"2024-09-20T22:50:08.246762+0000","last_became_active":"2024-09-20T22:41:42.187706+0000","last_became_peered":"2024-09-20T22:41:42.187706+0000","last_unstale":"2024-09-20T22:50:08.246762+0000","last_undegraded":"2024-09-20T22:50:08.246762+0000","last_fullsized":"2024-09-20T22:50:08.246762+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:46:48.012372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:46:48.012372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:12.131433+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":978,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-20T22:50:08.231102+0000","last_change":"2024-09-20T22:48:40.247992+0000","last_active":"2024-09-20T22:50:08.231102+0000","last_peered":"2024-09-20T22:50:08.231102+0000","last_clean":"2024-09-20T22:50:08.231102+0000","last_became_active":"2024-09-20T22:48:05.239710+0000","last_became_peered":"2024-09-20T22:48:05.239710+0000","last_unstale":"2024-09-20T22:50:08.231102+0000","last_undegraded":"2024-09-20T22:50:08.231102+0000","last_fullsized":"2024-09-20T22:50:08.231102+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:48:40.247889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:48:40.247889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:09.303289+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"74'9","reported_seq":971,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-20T22:50:08.233833+0000","last_change":"2024-09-20T22:49:09.704604+0000","last_active":"2024-09-20T22:50:08.233833+0000","last_peered":"2024-09-20T22:50:08.233833+0000","last_clean":"2024-09-20T22:50:08.233833+0000","last_became_active":"2024-09-20T22:41:40.252486+0000","last_became_peered":"2024-09-20T22:41:40.252486+0000","last_unstale":"2024-09-20T22:50:08.233833+0000","last_undegraded":"2024-09-20T22:50:08.233833+0000","last_fullsized":"2024-09-20T22:50:08.233833+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-09-20T22:49:09.704521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:49:09.704521+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:15.287583+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":298,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-20T22:50:08.231709+0000","last_change":"2024-09-20T22:48:19.379265+0000","last_active":"2024-09-20T22:50:08.231709+0000","last_peered":"2024-09-20T22:50:08.231709+0000","last_clean":"2024-09-20T22:50:08.231709+0000","last_became_active":"2024-09-20T22:47:49.229936+0000","last_became_peered":"2024-09-20T22:47:49.229936+0000","last_unstale":"2024-09-20T22:50:08.231709+0000","last_undegraded":"2024-09-20T22:50:08.231709+0000","last_fullsized":"2024-09-20T22:50:08.231709+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:48:19.379196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:48:19.379196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"74'8","reported_seq":1054,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-20T22:50:08.246414+0000","last_change":"2024-09-20T22:47:56.150641+0000","last_active":"2024-09-20T22:50:08.246414+0000","last_peered":"2024-09-20T22:50:08.246414+0000","last_clean":"2024-09-20T22:50:08.246414+0000","last_became_active":"2024-09-20T22:41:42.186352+0000","last_became_peered":"2024-09-20T22:41:42.186352+0000","last_unstale":"2024-09-20T22:50:08.246414+0000","last_undegraded":"2024-09-20T22:50:08.246414+0000","last_fullsized":"2024-09-20T22:50:08.246414+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-09-20T22:47:56.150562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:47:56.150562+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:12.131500+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":257,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-20T22:50:08.240480+0000","last_change":"2024-09-20T22:48:05.250192+0000","last_active":"2024-09-20T22:50:08.240480+0000","last_peered":"2024-09-20T22:50:08.240480+0000","last_clean":"2024-09-20T22:50:08.240480+0000","last_became_active":"2024-09-20T22:48:05.249656+0000","last_became_peered":"2024-09-20T22:48:05.249656+0000","last_unstale":"2024-09-20T22:50:08.240480+0000","last_undegraded":"2024-09-20T22:50:08.240480+0000","last_fullsized":"2024-09-20T22:50:08.240480+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:47:19.828541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:47:19.828541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:11.149743+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"74'10","reported_seq":962,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-20T22:50:08.245555+0000","last_change":"2024-09-20T22:46:24.269736+0000","last_active":"2024-09-20T22:50:08.245555+0000","last_peered":"2024-09-20T22:50:08.245555+0000","last_clean":"2024-09-20T22:50:08.245555+0000","last_became_active":"2024-09-20T22:41:40.253614+0000","last_became_peered":"2024-09-20T22:41:40.253614+0000","last_unstale":"2024-09-20T22:50:08.245555+0000","last_undegraded":"2024-09-20T22:50:08.245555+0000","last_fullsized":"2024-09-20T22:50:08.245555+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-09-20T22:46:24.269655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:46:24.269655+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:12.131419+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":978,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-20T22:50:08.247738+0000","last_change":"2024-09-20T22:49:54.381836+0000","last_active":"2024-09-20T22:50:08.247738+0000","last_peered":"2024-09-20T22:50:08.247738+0000","last_clean":"2024-09-20T22:50:08.247738+0000","last_became_active":"2024-09-20T22:48:05.240231+0000","last_became_peered":"2024-09-20T22:48:05.240231+0000","last_unstale":"2024-09-20T22:50:08.247738+0000","last_undegraded":"2024-09-20T22:50:08.247738+0000","last_fullsized":"2024-09-20T22:50:08.247738+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:54.381721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:49:54.381721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:51:04.253551+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"389'10","reported_seq":1093,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-20T22:50:08.233654+0000","last_change":"2024-09-20T22:49:51.715574+0000","last_active":"2024-09-20T22:50:08.233654+0000","last_peered":"2024-09-20T22:50:08.233654+0000","last_clean":"2024-09-20T22:50:08.233654+0000","last_became_active":"2024-09-20T22:47:49.227113+0000","last_became_peered":"2024-09-20T22:47:49.227113+0000","last_unstale":"2024-09-20T22:50:08.233654+0000","last_undegraded":"2024-09-20T22:50:08.233654+0000","last_fullsized":"2024-09-20T22:50:08.233654+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'10","last_scrub_stamp":"2024-09-20T22:49:51.715471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:49:51.715471+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:51:04.561910+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":289,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-20T22:50:08.232716+0000","last_change":"2024-09-20T22:47:49.232560+0000","last_active":"2024-09-20T22:50:08.232716+0000","last_peered":"2024-09-20T22:50:08.232716+0000","last_clean":"2024-09-20T22:50:08.232716+0000","last_became_active":"2024-09-20T22:47:49.230674+0000","last_became_peered":"2024-09-20T22:47:49.230674+0000","last_unstale":"2024-09-20T22:50:08.232716+0000","last_undegraded":"2024-09-20T22:50:08.232716+0000","last_fullsized":"2024-09-20T22:50:08.232716+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:47:41.411716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:47:41.411716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"74'11","reported_seq":277,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-20T22:50:08.236969+0000","last_change":"2024-09-20T22:48:44.023878+0000","last_active":"2024-09-20T22:50:08.236969+0000","last_peered":"2024-09-20T22:50:08.236969+0000","last_clean":"2024-09-20T22:50:08.236969+0000","last_became_active":"2024-09-20T22:48:05.249510+0000","last_became_peered":"2024-09-20T22:48:05.249510+0000","last_unstale":"2024-09-20T22:50:08.236969+0000","last_undegraded":"2024-09-20T22:50:08.236969+0000","last_fullsized":"2024-09-20T22:50:08.236969+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-09-20T22:48:44.023781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:48:44.023781+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:13.089164+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":265,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-20T22:50:08.236573+0000","last_change":"2024-09-20T22:49:11.015881+0000","last_active":"2024-09-20T22:50:08.236573+0000","last_peered":"2024-09-20T22:50:08.236573+0000","last_clean":"2024-09-20T22:50:08.236573+0000","last_became_active":"2024-09-20T22:48:05.252026+0000","last_became_peered":"2024-09-20T22:48:05.252026+0000","last_unstale":"2024-09-20T22:50:08.236573+0000","last_undegraded":"2024-09-20T22:50:08.236573+0000","last_fullsized":"2024-09-20T22:50:08.236573+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:11.015797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:49:11.015797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:39.578934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":951,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-20T22:50:08.235050+0000","last_change":"2024-09-20T22:48:59.241704+0000","last_active":"2024-09-20T22:50:08.235050+0000","last_peered":"2024-09-20T22:50:08.235050+0000","last_clean":"2024-09-20T22:50:08.235050+0000","last_became_active":"2024-09-20T22:41:42.181736+0000","last_became_peered":"2024-09-20T22:41:42.181736+0000","last_unstale":"2024-09-20T22:50:08.235050+0000","last_undegraded":"2024-09-20T22:50:08.235050+0000","last_fullsized":"2024-09-20T22:50:08.235050+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:48:59.241606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:48:59.241606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:13.923480+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":951,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-20T22:50:08.245112+0000","last_change":"2024-09-20T22:47:49.216479+0000","last_active":"2024-09-20T22:50:08.245112+0000","last_peered":"2024-09-20T22:50:08.245112+0000","last_clean":"2024-09-20T22:50:08.245112+0000","last_became_active":"2024-09-20T22:47:49.216141+0000","last_became_peered":"2024-09-20T22:47:49.216141+0000","last_unstale":"2024-09-20T22:50:08.245112+0000","last_undegraded":"2024-09-20T22:50:08.245112+0000","last_fullsized":"2024-09-20T22:50:08.245112+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:47:36.066480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:47:36.066480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:12.131469+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"404'13","reported_seq":983,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-20T22:50:08.235265+0000","last_change":"2024-09-20T22:49:25.147397+0000","last_active":"2024-09-20T22:50:08.235265+0000","last_peered":"2024-09-20T22:50:08.235265+0000","last_clean":"2024-09-20T22:50:08.235265+0000","last_became_active":"2024-09-20T22:41:40.164796+0000","last_became_peered":"2024-09-20T22:41:40.164796+0000","last_unstale":"2024-09-20T22:50:08.235265+0000","last_undegraded":"2024-09-20T22:50:08.235265+0000","last_fullsized":"2024-09-20T22:50:08.235265+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"404'13","last_scrub_stamp":"2024-09-20T22:49:25.147286+0000","last_deep_scrub":"404'13","last_deep_scrub_stamp":"2024-09-20T22:48:04.276282+0000","last_clean_scrub_stamp":"2024-09-20T22:49:25.147286+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:52.567714+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":993,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-20T22:50:08.248479+0000","last_change":"2024-09-20T22:49:56.396435+0000","last_active":"2024-09-20T22:50:08.248479+0000","last_peered":"2024-09-20T22:50:08.248479+0000","last_clean":"2024-09-20T22:50:08.248479+0000","last_became_active":"2024-09-20T22:48:05.238812+0000","last_became_peered":"2024-09-20T22:48:05.238812+0000","last_unstale":"2024-09-20T22:50:08.248479+0000","last_undegraded":"2024-09-20T22:50:08.248479+0000","last_fullsized":"2024-09-20T22:50:08.248479+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:56.396347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:31.138067+0000","last_clean_scrub_stamp":"2024-09-20T22:49:56.396347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:51:17.230220+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":955,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-20T22:50:08.245473+0000","last_change":"2024-09-20T22:48:38.200896+0000","last_active":"2024-09-20T22:50:08.245473+0000","last_peered":"2024-09-20T22:50:08.245473+0000","last_clean":"2024-09-20T22:50:08.245473+0000","last_became_active":"2024-09-20T22:47:49.218708+0000","last_became_peered":"2024-09-20T22:47:49.218708+0000","last_unstale":"2024-09-20T22:50:08.245473+0000","last_undegraded":"2024-09-20T22:50:08.245473+0000","last_fullsized":"2024-09-20T22:50:08.245473+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:48:38.200797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:48:38.200797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:12.131560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":290,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-20T22:50:08.231190+0000","last_change":"2024-09-20T22:47:49.225362+0000","last_active":"2024-09-20T22:50:08.231190+0000","last_peered":"2024-09-20T22:50:08.231190+0000","last_clean":"2024-09-20T22:50:08.231190+0000","last_became_active":"2024-09-20T22:47:49.225027+0000","last_became_peered":"2024-09-20T22:47:49.225027+0000","last_unstale":"2024-09-20T22:50:08.231190+0000","last_undegraded":"2024-09-20T22:50:08.231190+0000","last_fullsized":"2024-09-20T22:50:08.231190+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:47:37.309842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:47:37.309842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"470'16","reported_seq":769,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-20T22:50:08.231509+0000","last_change":"2024-09-20T22:49:10.659945+0000","last_active":"2024-09-20T22:50:08.231509+0000","last_peered":"2024-09-20T22:50:08.231509+0000","last_clean":"2024-09-20T22:50:08.231509+0000","last_became_active":"2024-09-20T22:44:09.170785+0000","last_became_peered":"2024-09-20T22:44:09.170785+0000","last_unstale":"2024-09-20T22:50:08.231509+0000","last_undegraded":"2024-09-20T22:50:08.231509+0000","last_fullsized":"2024-09-20T22:50:08.231509+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"470'16","last_scrub_stamp":"2024-09-20T22:49:10.659868+0000","last_deep_scrub":"470'16","last_deep_scrub_stamp":"2024-09-20T22:49:10.659868+0000","last_clean_scrub_stamp":"2024-09-20T22:49:10.659868+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:10.701010+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":959,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-20T22:50:08.234514+0000","last_change":"2024-09-20T22:48:02.512192+0000","last_active":"2024-09-20T22:50:08.234514+0000","last_peered":"2024-09-20T22:50:08.234514+0000","last_clean":"2024-09-20T22:50:08.234514+0000","last_became_active":"2024-09-20T22:41:38.158701+0000","last_became_peered":"2024-09-20T22:41:38.158701+0000","last_unstale":"2024-09-20T22:50:08.234514+0000","last_undegraded":"2024-09-20T22:50:08.234514+0000","last_fullsized":"2024-09-20T22:50:08.234514+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:48:02.512093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:45:24.765414+0000","last_clean_scrub_stamp":"2024-09-20T22:48:02.512093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:12.708641+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":974,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-20T22:50:08.245811+0000","last_change":"2024-09-20T22:49:11.373441+0000","last_active":"2024-09-20T22:50:08.245811+0000","last_peered":"2024-09-20T22:50:08.245811+0000","last_clean":"2024-09-20T22:50:08.245811+0000","last_became_active":"2024-09-20T22:48:05.238334+0000","last_became_peered":"2024-09-20T22:48:05.238334+0000","last_unstale":"2024-09-20T22:50:08.245811+0000","last_undegraded":"2024-09-20T22:50:08.245811+0000","last_fullsized":"2024-09-20T22:50:08.245811+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:11.373370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:50.174861+0000","last_clean_scrub_stamp":"2024-09-20T22:49:11.373370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:14.536205+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":951,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-20T22:50:08.245570+0000","last_change":"2024-09-20T22:49:04.034724+0000","last_active":"2024-09-20T22:50:08.245570+0000","last_peered":"2024-09-20T22:50:08.245570+0000","last_clean":"2024-09-20T22:50:08.245570+0000","last_became_active":"2024-09-20T22:47:49.214487+0000","last_became_peered":"2024-09-20T22:47:49.214487+0000","last_unstale":"2024-09-20T22:50:08.245570+0000","last_undegraded":"2024-09-20T22:50:08.245570+0000","last_fullsized":"2024-09-20T22:50:08.245570+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:04.034601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:49:04.034601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:18.245352+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"74'4","reported_seq":953,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-20T22:50:08.244849+0000","last_change":"2024-09-20T22:48:23.236306+0000","last_active":"2024-09-20T22:50:08.244849+0000","last_peered":"2024-09-20T22:50:08.244849+0000","last_clean":"2024-09-20T22:50:08.244849+0000","last_became_active":"2024-09-20T22:41:40.166123+0000","last_became_peered":"2024-09-20T22:41:40.166123+0000","last_unstale":"2024-09-20T22:50:08.244849+0000","last_undegraded":"2024-09-20T22:50:08.244849+0000","last_fullsized":"2024-09-20T22:50:08.244849+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'4","last_scrub_stamp":"2024-09-20T22:48:23.236227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:48:23.236227+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:12.131530+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":297,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-20T22:50:08.234220+0000","last_change":"2024-09-20T22:48:18.432371+0000","last_active":"2024-09-20T22:50:08.234220+0000","last_peered":"2024-09-20T22:50:08.234220+0000","last_clean":"2024-09-20T22:50:08.234220+0000","last_became_active":"2024-09-20T22:47:49.231964+0000","last_became_peered":"2024-09-20T22:47:49.231964+0000","last_unstale":"2024-09-20T22:50:08.234220+0000","last_undegraded":"2024-09-20T22:50:08.234220+0000","last_fullsized":"2024-09-20T22:50:08.234220+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:48:18.432271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:48:18.432271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":983,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-20T22:50:08.241430+0000","last_change":"2024-09-20T22:49:07.125306+0000","last_active":"2024-09-20T22:50:08.241430+0000","last_peered":"2024-09-20T22:50:08.241430+0000","last_clean":"2024-09-20T22:50:08.241430+0000","last_became_active":"2024-09-20T22:48:05.244659+0000","last_became_peered":"2024-09-20T22:48:05.244659+0000","last_unstale":"2024-09-20T22:50:08.241430+0000","last_undegraded":"2024-09-20T22:50:08.241430+0000","last_fullsized":"2024-09-20T22:50:08.241430+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:07.125231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:49:07.125231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:23.332189+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":966,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-20T22:50:08.246484+0000","last_change":"2024-09-20T22:49:57.405922+0000","last_active":"2024-09-20T22:50:08.246484+0000","last_peered":"2024-09-20T22:50:08.246484+0000","last_clean":"2024-09-20T22:50:08.246484+0000","last_became_active":"2024-09-20T22:48:05.237765+0000","last_became_peered":"2024-09-20T22:48:05.237765+0000","last_unstale":"2024-09-20T22:50:08.246484+0000","last_undegraded":"2024-09-20T22:50:08.246484+0000","last_fullsized":"2024-09-20T22:50:08.246484+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:57.405785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:10.233487+0000","last_clean_scrub_stamp":"2024-09-20T22:49:57.405785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:51:00.718232+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":959,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-20T22:50:08.235602+0000","last_change":"2024-09-20T22:49:05.205734+0000","last_active":"2024-09-20T22:50:08.235602+0000","last_peered":"2024-09-20T22:50:08.235602+0000","last_clean":"2024-09-20T22:50:08.235602+0000","last_became_active":"2024-09-20T22:48:05.249492+0000","last_became_peered":"2024-09-20T22:48:05.249492+0000","last_unstale":"2024-09-20T22:50:08.235602+0000","last_undegraded":"2024-09-20T22:50:08.235602+0000","last_fullsized":"2024-09-20T22:50:08.235602+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:05.205665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:49:05.205665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:32.772495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":977,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-20T22:50:08.244600+0000","last_change":"2024-09-20T22:48:36.246135+0000","last_active":"2024-09-20T22:50:08.244600+0000","last_peered":"2024-09-20T22:50:08.244600+0000","last_clean":"2024-09-20T22:50:08.244600+0000","last_became_active":"2024-09-20T22:47:49.217064+0000","last_became_peered":"2024-09-20T22:47:49.217064+0000","last_unstale":"2024-09-20T22:50:08.244600+0000","last_undegraded":"2024-09-20T22:50:08.244600+0000","last_fullsized":"2024-09-20T22:50:08.244600+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:48:36.246069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:48:36.246069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:12.131546+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"74'11","reported_seq":986,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-20T22:50:08.242744+0000","last_change":"2024-09-20T22:49:10.137213+0000","last_active":"2024-09-20T22:50:08.242744+0000","last_peered":"2024-09-20T22:50:08.242744+0000","last_clean":"2024-09-20T22:50:08.242744+0000","last_became_active":"2024-09-20T22:41:40.171256+0000","last_became_peered":"2024-09-20T22:41:40.171256+0000","last_unstale":"2024-09-20T22:50:08.242744+0000","last_undegraded":"2024-09-20T22:50:08.242744+0000","last_fullsized":"2024-09-20T22:50:08.242744+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-09-20T22:49:10.137128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:49:10.137128+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:16.411613+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":298,"reported_epoch":528,"state":"active+clean","last_fresh":"2024-09-20T22:50:08.232841+0000","last_change":"2024-09-20T22:49:23.660691+0000","last_active":"2024-09-20T22:50:08.232841+0000","last_peered":"2024-09-20T22:50:08.232841+0000","last_clean":"2024-09-20T22:50:08.232841+0000","last_became_active":"2024-09-20T22:47:49.220971+0000","last_became_peered":"2024-09-20T22:47:49.220971+0000","last_unstale":"2024-09-20T22:50:08.232841+0000","last_undegraded":"2024-09-20T22:50:08.232841+0000","last_fullsized":"2024-09-20T22:50:08.232841+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:23.660625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:51.667341+0000","last_clean_scrub_stamp":"2024-09-20T22:49:23.660625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:38.664121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":563458131,"num_objects":257,"num_object_clones":208,"num_object_copies":771,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":257,"num_whiteouts":18,"num_read":3612,"num_read_kb":3794458,"num_write":2539,"num_write_kb":934528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":613,"num_bytes_recovered":302968954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1144029184,"data_stored":1687484562,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5036,"ondisk_log_size":5036,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":82,"ondisk_log_size":82,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":682,"num_read_kb":439,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":238,"num_bytes_recovered":5126,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":452,"ondisk_log_size":452,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":1034,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":501,"num_read_kb":496,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":384,"ondisk_log_size":384,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070570,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":172156,"kb_used_data":114596,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":93572996,"statfs":{"total":95995035648,"available":95818747904,"internally_reserved":0,"allocated":117346304,"data_stored":165813546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":58915274},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430320,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":210388,"kb_used_data":152832,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":93534764,"statfs":{"total":95995035648,"available":95779598336,"internally_reserved":0,"allocated":156499968,"data_stored":231307554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691957,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":136832,"kb_used_data":105644,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93608320,"statfs":{"total":95995035648,"available":95854919680,"internally_reserved":0,"allocated":108179456,"data_stored":165967904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986298,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":146236,"kb_used_data":106720,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93598916,"statfs":{"total":95995035648,"available":95845289984,"internally_reserved":0,"allocated":109281280,"data_stored":155830203,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182528,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":275432,"kb_used_data":217872,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":93469720,"statfs":{"total":95995035648,"available":95712993280,"internally_reserved":0,"allocated":223100928,"data_stored":314028183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":58915271},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411461,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":172964,"kb_used_data":119560,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93572188,"statfs":{"total":95995035648,"available":95817920512,"internally_reserved":0,"allocated":122429440,"data_stored":183174474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574986,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":215048,"kb_used_data":161644,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93530104,"statfs":{"total":95995035648,"available":95774826496,"internally_reserved":0,"allocated":165523456,"data_stored":239497323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738511,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":226620,"kb_used_data":173224,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93518532,"statfs":{"total":95995035648,"available":95762976768,"internally_reserved":0,"allocated":177381376,"data_stored":254140350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":173137920,"data_stored":251577385,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":161251328,"data_stored":236934023,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":117940224,"data_stored":180349247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":218255360,"data_stored":310874306,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":105033728,"data_stored":153267850,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":103354368,"data_stored":162813903,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":152289280,"data_stored":228752322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":112766976,"data_stored":162915526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-20T22:50:14.591 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph pg dump --format=json 2024-09-20T22:50:14.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:14.594 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:14.594 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:50:14.696 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:50:14.718 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:14.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:14.719 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:50:14.820 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:50:14.841 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:14.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:14.842 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:50:14.842 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:50:14.843 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:50:14.860 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:50:14.860 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:50:14.885 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:50:14.885 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:50:14.905 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-20T22:50:14.906 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T22:50:14.929 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:50:14.930 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:50:14.932 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:50:14.942 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:50:14.946 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:50:14.947 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:50:14.965 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:14.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:14.966 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:50:14.966 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:50:14.967 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:50:14.983 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:50:14.983 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:50:15.014 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-20T22:50:15.014 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [], 2024-09-20T22:50:15.015 INFO:teuthology.orchestra.run.smithi154.stdout: "complaint_time": 30, 2024-09-20T22:50:15.015 INFO:teuthology.orchestra.run.smithi154.stdout: "num_blocked_ops": 0 2024-09-20T22:50:15.015 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-20T22:50:15.068 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:50:15.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:15.090 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:15.090 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:50:15.092 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-20T22:50:15.191 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:50:15.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:15.215 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:15.215 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:50:15.256 INFO:tasks.rados.rados.0.smithi084.stdout:2095: done (1 left) 2024-09-20T22:50:15.256 INFO:tasks.rados.rados.0.smithi084.stdout:2097: done (0 left) 2024-09-20T22:50:15.256 INFO:tasks.rados.rados.0.smithi084.stdout:2098: delete oid 33 current snap is 232 2024-09-20T22:50:15.260 INFO:tasks.rados.rados.0.smithi084.stdout:2098: done (0 left) 2024-09-20T22:50:15.260 INFO:tasks.rados.rados.0.smithi084.stdout:2099: write (excl) oid 38 current snap is 232 2024-09-20T22:50:15.260 INFO:tasks.rados.rados.0.smithi084.stdout:2099: seq_num 734 ranges {736338=402654,1552136=539580} 2024-09-20T22:50:15.266 INFO:tasks.rados.rados.0.smithi084.stdout:2099: writing smithi08477203-38 from 736338 to 1138992 tid 1 2024-09-20T22:50:15.271 INFO:tasks.rados.rados.0.smithi084.stdout:2099: writing smithi08477203-38 from 1552136 to 2091716 tid 2 2024-09-20T22:50:15.272 INFO:tasks.rados.rados.0.smithi084.stdout:2100: snap_remove snap 225 2024-09-20T22:50:15.315 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:50:15.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:15.340 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:15.340 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:50:15.441 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:50:15.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:15.463 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:15.463 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:50:15.533 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:15 smithi084 ceph-mon[40024]: 3.16 scrub starts 2024-09-20T22:50:15.534 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:15 smithi084 ceph-mon[40024]: 3.16 scrub ok 2024-09-20T22:50:15.534 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:15 smithi084 ceph-mon[40024]: from='client.16191 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T22:50:15.534 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:15 smithi084 ceph-mon[40024]: pgmap v702: 158 pgs: 2 active+clean+snaptrim, 156 active+clean; 542 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 2.0 KiB/s rd, 748 KiB/s wr, 5 op/s 2024-09-20T22:50:15.534 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:15 smithi084 ceph-mon[40024]: purged_snaps scrub starts 2024-09-20T22:50:15.534 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:15 smithi084 ceph-mon[40024]: purged_snaps scrub ok 2024-09-20T22:50:15.534 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:15 smithi084 ceph-mon[40024]: 7.8 scrub starts 2024-09-20T22:50:15.534 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:15 smithi084 ceph-mon[40024]: 7.8 scrub ok 2024-09-20T22:50:15.534 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:15 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/2244573132' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:50:15.534 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:15 smithi084 ceph-mon[40024]: osdmap e535: 8 total, 8 up, 8 in 2024-09-20T22:50:15.534 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:15 smithi084 ceph-mon[29774]: 3.16 scrub starts 2024-09-20T22:50:15.534 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:15 smithi084 ceph-mon[29774]: 3.16 scrub ok 2024-09-20T22:50:15.535 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:15 smithi084 ceph-mon[29774]: from='client.16191 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T22:50:15.535 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:15 smithi084 ceph-mon[29774]: pgmap v702: 158 pgs: 2 active+clean+snaptrim, 156 active+clean; 542 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 2.0 KiB/s rd, 748 KiB/s wr, 5 op/s 2024-09-20T22:50:15.535 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:15 smithi084 ceph-mon[29774]: purged_snaps scrub starts 2024-09-20T22:50:15.535 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:15 smithi084 ceph-mon[29774]: purged_snaps scrub ok 2024-09-20T22:50:15.535 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:15 smithi084 ceph-mon[29774]: 7.8 scrub starts 2024-09-20T22:50:15.535 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:15 smithi084 ceph-mon[29774]: 7.8 scrub ok 2024-09-20T22:50:15.535 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:15 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/2244573132' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:50:15.535 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:15 smithi084 ceph-mon[29774]: osdmap e535: 8 total, 8 up, 8 in 2024-09-20T22:50:15.565 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:50:15.586 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:15.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:15.587 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:50:15.594 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:15 smithi154 ceph-mon[33264]: 3.16 scrub starts 2024-09-20T22:50:15.594 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:15 smithi154 ceph-mon[33264]: 3.16 scrub ok 2024-09-20T22:50:15.594 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:15 smithi154 ceph-mon[33264]: from='client.16191 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T22:50:15.595 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:15 smithi154 ceph-mon[33264]: pgmap v702: 158 pgs: 2 active+clean+snaptrim, 156 active+clean; 542 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 2.0 KiB/s rd, 748 KiB/s wr, 5 op/s 2024-09-20T22:50:15.595 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:15 smithi154 ceph-mon[33264]: purged_snaps scrub starts 2024-09-20T22:50:15.595 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:15 smithi154 ceph-mon[33264]: purged_snaps scrub ok 2024-09-20T22:50:15.595 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:15 smithi154 ceph-mon[33264]: 7.8 scrub starts 2024-09-20T22:50:15.595 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:15 smithi154 ceph-mon[33264]: 7.8 scrub ok 2024-09-20T22:50:15.595 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:15 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/2244573132' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:50:15.595 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:15 smithi154 ceph-mon[33264]: osdmap e535: 8 total, 8 up, 8 in 2024-09-20T22:50:15.621 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.5 dump_historic_ops 2024-09-20T22:50:15.687 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:50:15.709 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:15.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:15.710 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:50:15.810 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:50:15.832 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:15.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:15.833 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:50:15.865 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:50:15.934 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:50:15.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:15.956 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:15.957 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:50:16.057 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:50:16.079 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:16.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:16.080 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:50:16.181 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:50:16.202 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:16.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:16.203 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:50:16.254 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:50:16.258 INFO:tasks.rados.rados.0.smithi084.stdout:2100: done (1 left) 2024-09-20T22:50:16.259 INFO:tasks.rados.rados.0.smithi084.stdout:2101: write oid 49 current snap is 232 2024-09-20T22:50:16.259 INFO:tasks.rados.rados.0.smithi084.stdout:2101: seq_num 735 ranges {753145=477903,2010618=342527} 2024-09-20T22:50:16.266 INFO:tasks.rados.rados.0.smithi084.stdout:2101: writing smithi08477203-49 from 753145 to 1231048 tid 1 2024-09-20T22:50:16.273 INFO:tasks.rados.rados.0.smithi084.stdout:2101: writing smithi08477203-49 from 2010618 to 2353145 tid 2 2024-09-20T22:50:16.274 INFO:tasks.rados.rados.0.smithi084.stdout:2102: delete oid 44 current snap is 232 2024-09-20T22:50:16.274 INFO:tasks.rados.rados.0.smithi084.stdout:2099: finishing write tid 1 to smithi08477203-38 2024-09-20T22:50:16.274 INFO:tasks.rados.rados.0.smithi084.stdout:2099: finishing write tid 2 to smithi08477203-38 2024-09-20T22:50:16.274 INFO:tasks.rados.rados.0.smithi084.stdout:2099: finishing write tid 3 to smithi08477203-38 2024-09-20T22:50:16.274 INFO:tasks.rados.rados.0.smithi084.stdout:2099: finishing write tid 4 to smithi08477203-38 2024-09-20T22:50:16.274 INFO:tasks.rados.rados.0.smithi084.stdout:2099: oid 38 updating version 0 to 721 2024-09-20T22:50:16.274 INFO:tasks.rados.rados.0.smithi084.stdout:2099: oid 38 updating version 721 to 723 2024-09-20T22:50:16.274 INFO:tasks.rados.rados.0.smithi084.stdout:2099: oid 38 version 723 is already newer than 722 2024-09-20T22:50:16.274 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 38 v 723 (ObjNum 734 snap 232 seq_num 734) dirty exists 2024-09-20T22:50:16.274 INFO:tasks.rados.rados.0.smithi084.stdout:2099: left oid 38 (ObjNum 734 snap 232 seq_num 734) 2024-09-20T22:50:16.274 INFO:tasks.rados.rados.0.smithi084.stdout:2101: finishing write tid 1 to smithi08477203-49 2024-09-20T22:50:16.276 INFO:tasks.rados.rados.0.smithi084.stdout:2101: finishing write tid 2 to smithi08477203-49 2024-09-20T22:50:16.276 INFO:tasks.rados.rados.0.smithi084.stdout:2099: done (2 left) 2024-09-20T22:50:16.276 INFO:tasks.rados.rados.0.smithi084.stdout:2102: done (1 left) 2024-09-20T22:50:16.277 INFO:tasks.rados.rados.0.smithi084.stdout:2103: read oid 1 snap -1 2024-09-20T22:50:16.277 INFO:tasks.rados.rados.0.smithi084.stdout:2103: expect deleted 2024-09-20T22:50:16.277 INFO:tasks.rados.rados.0.smithi084.stdout:2104: read oid 22 snap -1 2024-09-20T22:50:16.277 INFO:tasks.rados.rados.0.smithi084.stdout:2104: expect deleted 2024-09-20T22:50:16.277 INFO:tasks.rados.rados.0.smithi084.stdout:2105: read oid 44 snap -1 2024-09-20T22:50:16.277 INFO:tasks.rados.rados.0.smithi084.stdout:2105: expect deleted 2024-09-20T22:50:16.277 INFO:tasks.rados.rados.0.smithi084.stdout:2106: write (excl) oid 42 current snap is 232 2024-09-20T22:50:16.277 INFO:tasks.rados.rados.0.smithi084.stdout:2101: finishing write tid 3 to smithi08477203-49 2024-09-20T22:50:16.277 INFO:tasks.rados.rados.0.smithi084.stdout:2101: finishing write tid 4 to smithi08477203-49 2024-09-20T22:50:16.277 INFO:tasks.rados.rados.0.smithi084.stdout:2101: oid 49 updating version 0 to 723 2024-09-20T22:50:16.277 INFO:tasks.rados.rados.0.smithi084.stdout:2101: oid 49 updating version 723 to 725 2024-09-20T22:50:16.277 INFO:tasks.rados.rados.0.smithi084.stdout:2101: oid 49 version 725 is already newer than 724 2024-09-20T22:50:16.277 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 49 v 725 (ObjNum 735 snap 232 seq_num 735) dirty exists 2024-09-20T22:50:16.277 INFO:tasks.rados.rados.0.smithi084.stdout:2101: left oid 49 (ObjNum 735 snap 232 seq_num 735) 2024-09-20T22:50:16.278 INFO:tasks.rados.rados.0.smithi084.stdout:2106: seq_num 737 ranges {786759=628401,2011229=375530} 2024-09-20T22:50:16.284 INFO:tasks.rados.rados.0.smithi084.stdout:2106: writing smithi08477203-42 from 786759 to 1415160 tid 1 2024-09-20T22:50:16.291 INFO:tasks.rados.rados.0.smithi084.stdout:2106: writing smithi08477203-42 from 2011229 to 2386759 tid 2 2024-09-20T22:50:16.291 INFO:tasks.rados.rados.0.smithi084.stdout:2101: done (4 left) 2024-09-20T22:50:16.291 INFO:tasks.rados.rados.0.smithi084.stdout:2107: snap_remove snap 196 2024-09-20T22:50:16.304 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:50:16.326 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:16.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:16.327 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:50:16.356 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-20T22:50:16.356 INFO:teuthology.orchestra.run.smithi154.stdout: "size": 20, 2024-09-20T22:50:16.356 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 600, 2024-09-20T22:50:16.356 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [ 2024-09-20T22:50:16.356 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.356 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24334.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-09-20T22:50:16.356 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:41:40.156321+0000", 2024-09-20T22:50:16.356 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 516.19693481000002, 2024-09-20T22:50:16.356 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.012506929, 2024-09-20T22:50:16.356 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:50:16.356 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:50:16.357 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:50:16.357 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:50:16.357 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24334", 2024-09-20T22:50:16.357 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/343304630", 2024-09-20T22:50:16.357 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 7 2024-09-20T22:50:16.357 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.357 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:50:16.357 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.357 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:50:16.357 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:40.156321+0000", 2024-09-20T22:50:16.357 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:50:16.357 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.357 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.358 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:50:16.358 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:40.156321+0000", 2024-09-20T22:50:16.358 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:50:16.358 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.358 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.358 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:50:16.358 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:40.156323+0000", 2024-09-20T22:50:16.358 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9700000000000002e-06 2024-09-20T22:50:16.358 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.358 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.358 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:50:16.358 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:40.156330+0000", 2024-09-20T22:50:16.358 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.9809999999999997e-06 2024-09-20T22:50:16.358 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.359 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.359 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:50:16.359 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:40.156348+0000", 2024-09-20T22:50:16.359 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.7938e-05 2024-09-20T22:50:16.359 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.359 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.359 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:50:16.359 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:40.156362+0000", 2024-09-20T22:50:16.359 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3641e-05 2024-09-20T22:50:16.359 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.359 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.359 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:50:16.359 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:40.156430+0000", 2024-09-20T22:50:16.359 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.7543e-05 2024-09-20T22:50:16.360 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.360 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.360 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:50:16.360 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:40.156543+0000", 2024-09-20T22:50:16.360 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00011390100000000001 2024-09-20T22:50:16.360 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.360 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.360 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 1,6", 2024-09-20T22:50:16.360 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:40.156712+0000", 2024-09-20T22:50:16.360 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00016825499999999999 2024-09-20T22:50:16.360 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.360 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.360 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:50:16.360 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:40.157640+0000", 2024-09-20T22:50:16.360 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00092863699999999995 2024-09-20T22:50:16.360 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.361 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.361 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:50:16.361 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:40.159647+0000", 2024-09-20T22:50:16.361 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0020068320000000001 2024-09-20T22:50:16.361 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.361 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.361 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:50:16.361 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:40.168687+0000", 2024-09-20T22:50:16.361 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0090397749999999999 2024-09-20T22:50:16.361 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.361 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.361 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:50:16.361 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:40.168768+0000", 2024-09-20T22:50:16.361 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.0819e-05 2024-09-20T22:50:16.362 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.362 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.362 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:50:16.362 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:40.168828+0000", 2024-09-20T22:50:16.362 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.0637000000000002e-05 2024-09-20T22:50:16.362 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:50:16.362 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:50:16.362 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:50:16.362 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.362 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.362 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "MOSDPGPush(1.4 93/92 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 77'128, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@77'128, size: 327680, copy_subset: [(0, 327680)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 327680, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-20T22:50:16.362 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:42:50.646393+0000", 2024-09-20T22:50:16.362 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 445.706863371, 2024-09-20T22:50:16.362 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.033640915, 2024-09-20T22:50:16.363 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:50:16.363 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:50:16.363 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:50:16.363 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:50:16.363 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.363 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:50:16.363 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:42:50.646393+0000", 2024-09-20T22:50:16.363 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:50:16.363 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.363 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.363 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:50:16.363 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:42:50.646393+0000", 2024-09-20T22:50:16.363 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:50:16.364 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.364 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.364 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:50:16.364 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:42:50.646393+0000", 2024-09-20T22:50:16.364 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.23e-07 2024-09-20T22:50:16.364 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.364 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.364 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:50:16.364 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:42:50.679431+0000", 2024-09-20T22:50:16.364 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.033038349000000002 2024-09-20T22:50:16.364 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.364 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.364 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:50:16.364 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:42:50.679478+0000", 2024-09-20T22:50:16.364 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.6828e-05 2024-09-20T22:50:16.365 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.365 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.365 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:50:16.365 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:42:50.679492+0000", 2024-09-20T22:50:16.365 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4062e-05 2024-09-20T22:50:16.365 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.365 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.365 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:50:16.365 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:42:50.679593+0000", 2024-09-20T22:50:16.365 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000100865 2024-09-20T22:50:16.365 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.365 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.365 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:50:16.366 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:42:50.679604+0000", 2024-09-20T22:50:16.366 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0348e-05 2024-09-20T22:50:16.366 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.366 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.366 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:50:16.366 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:42:50.680034+0000", 2024-09-20T22:50:16.366 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00043023999999999997 2024-09-20T22:50:16.366 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:50:16.366 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:50:16.366 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:50:16.366 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.366 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.366 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:554 7.a e128/94 7:5122292a:::smithi08477203-33:head v 128'32, mlcod=119'28)", 2024-09-20T22:50:16.366 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:27.341250+0000", 2024-09-20T22:50:16.367 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 409.01200578700002, 2024-09-20T22:50:16.367 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.051661319999999997, 2024-09-20T22:50:16.367 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:50:16.367 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:50:16.367 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:50:16.367 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:50:16.367 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.367 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:50:16.367 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.341250+0000", 2024-09-20T22:50:16.367 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:50:16.367 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.367 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.367 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:50:16.368 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.341250+0000", 2024-09-20T22:50:16.368 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:50:16.368 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.368 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.368 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:50:16.368 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.341251+0000", 2024-09-20T22:50:16.368 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.7299999999999996e-07 2024-09-20T22:50:16.368 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.368 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.368 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:50:16.368 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.390395+0000", 2024-09-20T22:50:16.368 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.049143539999999999 2024-09-20T22:50:16.368 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.368 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.368 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:50:16.369 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.390413+0000", 2024-09-20T22:50:16.369 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.8709999999999999e-05 2024-09-20T22:50:16.369 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.369 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.369 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:50:16.369 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.390423+0000", 2024-09-20T22:50:16.369 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.4660000000000005e-06 2024-09-20T22:50:16.369 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.369 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.369 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:50:16.369 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.390484+0000", 2024-09-20T22:50:16.369 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.0851999999999999e-05 2024-09-20T22:50:16.369 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.370 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.370 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:50:16.370 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.390504+0000", 2024-09-20T22:50:16.370 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9976999999999999e-05 2024-09-20T22:50:16.370 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.370 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.370 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:50:16.370 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.392875+0000", 2024-09-20T22:50:16.370 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0023713710000000002 2024-09-20T22:50:16.370 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.370 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.370 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:50:16.370 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.392912+0000", 2024-09-20T22:50:16.370 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.6831e-05 2024-09-20T22:50:16.371 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:50:16.371 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:50:16.371 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:50:16.371 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.371 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.371 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:786 7.17 e145/139 7:ed54065f:::smithi08477203-12:head v 145'119, mlcod=143'117)", 2024-09-20T22:50:16.371 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:44.345115+0000", 2024-09-20T22:50:16.371 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 392.00814130499998, 2024-09-20T22:50:16.371 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0098654899999999993, 2024-09-20T22:50:16.371 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:50:16.371 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:50:16.371 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:50:16.371 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:50:16.372 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.372 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:50:16.372 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.345115+0000", 2024-09-20T22:50:16.372 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:50:16.372 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.372 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.372 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:50:16.372 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.345115+0000", 2024-09-20T22:50:16.372 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:50:16.372 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.372 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.372 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:50:16.372 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.345116+0000", 2024-09-20T22:50:16.372 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.8299999999999995e-07 2024-09-20T22:50:16.373 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.373 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.373 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:50:16.373 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.347955+0000", 2024-09-20T22:50:16.373 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.002839028 2024-09-20T22:50:16.373 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.373 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.373 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:50:16.373 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.347977+0000", 2024-09-20T22:50:16.373 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.1897999999999999e-05 2024-09-20T22:50:16.373 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.373 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.373 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:50:16.374 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.347992+0000", 2024-09-20T22:50:16.374 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4761e-05 2024-09-20T22:50:16.374 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.374 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.374 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:50:16.374 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.348072+0000", 2024-09-20T22:50:16.374 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.0586000000000004e-05 2024-09-20T22:50:16.374 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.374 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.374 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:50:16.374 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.348097+0000", 2024-09-20T22:50:16.374 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.5026000000000001e-05 2024-09-20T22:50:16.374 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.374 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.375 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:50:16.375 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.354924+0000", 2024-09-20T22:50:16.375 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0068271240000000004 2024-09-20T22:50:16.375 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.375 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.375 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:50:16.375 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.354980+0000", 2024-09-20T22:50:16.375 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.6184000000000003e-05 2024-09-20T22:50:16.375 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:50:16.375 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:50:16.375 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:50:16.375 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.375 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.375 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:894 7.a e155/94 7:50097c2d:::smithi08477203-23:head v 155'54, mlcod=143'50)", 2024-09-20T22:50:16.376 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:54.450745+0000", 2024-09-20T22:50:16.376 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 381.90251121699998, 2024-09-20T22:50:16.376 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0091901029999999998, 2024-09-20T22:50:16.376 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:50:16.376 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:50:16.376 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:50:16.376 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:50:16.376 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.376 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:50:16.376 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:54.450745+0000", 2024-09-20T22:50:16.376 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:50:16.376 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.377 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.377 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:50:16.377 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:54.450745+0000", 2024-09-20T22:50:16.377 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:50:16.377 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.377 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.377 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:50:16.377 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:54.450745+0000", 2024-09-20T22:50:16.377 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.4200000000000002e-07 2024-09-20T22:50:16.377 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.377 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.377 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:50:16.377 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:54.456336+0000", 2024-09-20T22:50:16.377 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.005590589 2024-09-20T22:50:16.378 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.378 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.378 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:50:16.378 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:54.456353+0000", 2024-09-20T22:50:16.378 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.7187e-05 2024-09-20T22:50:16.378 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.378 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.378 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:50:16.378 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:54.456362+0000", 2024-09-20T22:50:16.378 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.7409999999999998e-06 2024-09-20T22:50:16.378 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.378 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.378 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:50:16.378 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:54.456418+0000", 2024-09-20T22:50:16.378 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.6431000000000002e-05 2024-09-20T22:50:16.379 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.379 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.379 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:50:16.379 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:54.456438+0000", 2024-09-20T22:50:16.379 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9718999999999999e-05 2024-09-20T22:50:16.379 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.379 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.379 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:50:16.379 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:54.459899+0000", 2024-09-20T22:50:16.379 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.003461254 2024-09-20T22:50:16.379 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.379 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.379 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:50:16.379 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:54.459935+0000", 2024-09-20T22:50:16.380 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.5939999999999998e-05 2024-09-20T22:50:16.380 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:50:16.380 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:50:16.380 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:50:16.380 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.380 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.380 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:901 7.e e156/94 7:7c62ced4:::smithi08477203-17:head v 156'42, mlcod=154'40)", 2024-09-20T22:50:16.380 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:55.635897+0000", 2024-09-20T22:50:16.380 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 380.71735882399997, 2024-09-20T22:50:16.380 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0082336760000000005, 2024-09-20T22:50:16.380 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:50:16.380 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:50:16.380 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:50:16.380 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:50:16.381 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.381 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:50:16.381 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:55.635897+0000", 2024-09-20T22:50:16.381 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:50:16.381 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.381 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.381 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:50:16.381 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:55.635897+0000", 2024-09-20T22:50:16.381 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:50:16.381 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.381 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.381 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:50:16.381 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:55.635898+0000", 2024-09-20T22:50:16.381 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.23e-07 2024-09-20T22:50:16.382 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.382 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.382 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:50:16.382 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:55.640084+0000", 2024-09-20T22:50:16.382 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0041865499999999998 2024-09-20T22:50:16.382 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.382 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.382 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:50:16.382 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:55.640118+0000", 2024-09-20T22:50:16.382 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.4093e-05 2024-09-20T22:50:16.382 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.382 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.382 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:50:16.382 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:55.640129+0000", 2024-09-20T22:50:16.383 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0281e-05 2024-09-20T22:50:16.383 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.383 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.383 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:50:16.383 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:55.640230+0000", 2024-09-20T22:50:16.383 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000101538 2024-09-20T22:50:16.383 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.383 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.383 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:50:16.385 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:55.640251+0000", 2024-09-20T22:50:16.385 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.1293000000000001e-05 2024-09-20T22:50:16.385 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.385 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.385 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:50:16.385 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:55.644088+0000", 2024-09-20T22:50:16.385 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.003836178 2024-09-20T22:50:16.386 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.386 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.386 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:50:16.386 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:55.644131+0000", 2024-09-20T22:50:16.386 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.3519999999999997e-05 2024-09-20T22:50:16.386 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:50:16.386 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:50:16.386 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:50:16.386 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.386 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.386 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:1316 7.0 7:05bb442d:::smithi08477203-24:head [stat,delete] snapc 2b=[2b,2a] ondisk+write+known_if_redirected+supports_pool_eio e180)", 2024-09-20T22:50:16.386 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:44:19.225492+0000", 2024-09-20T22:50:16.386 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 357.12776425300001, 2024-09-20T22:50:16.387 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0083403250000000009, 2024-09-20T22:50:16.387 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:50:16.387 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:50:16.387 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:50:16.387 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:50:16.387 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:50:16.387 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:50:16.387 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 1316 2024-09-20T22:50:16.387 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.387 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:50:16.387 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.387 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:50:16.387 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:19.225492+0000", 2024-09-20T22:50:16.387 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:50:16.388 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.388 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.388 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:50:16.388 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:19.225492+0000", 2024-09-20T22:50:16.388 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:50:16.388 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.388 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.388 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:50:16.388 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:19.225495+0000", 2024-09-20T22:50:16.388 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.96e-06 2024-09-20T22:50:16.388 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.388 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.388 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:50:16.389 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:19.225508+0000", 2024-09-20T22:50:16.389 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3343000000000001e-05 2024-09-20T22:50:16.389 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.389 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.389 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:50:16.389 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:19.225542+0000", 2024-09-20T22:50:16.389 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.3973000000000002e-05 2024-09-20T22:50:16.389 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.389 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.389 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:50:16.389 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:19.225570+0000", 2024-09-20T22:50:16.389 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.7713999999999999e-05 2024-09-20T22:50:16.389 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.390 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.390 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:50:16.390 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:19.225655+0000", 2024-09-20T22:50:16.390 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.4944999999999994e-05 2024-09-20T22:50:16.390 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.390 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.390 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op must wait for map", 2024-09-20T22:50:16.390 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:19.225660+0000", 2024-09-20T22:50:16.390 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.5470000000000004e-06 2024-09-20T22:50:16.390 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.390 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.390 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:50:16.390 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:19.232618+0000", 2024-09-20T22:50:16.390 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0069571069999999997 2024-09-20T22:50:16.391 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.391 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.391 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:50:16.391 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:19.232717+0000", 2024-09-20T22:50:16.391 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.9306000000000001e-05 2024-09-20T22:50:16.391 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.391 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.391 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:50:16.391 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:19.233832+0000", 2024-09-20T22:50:16.391 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0011154299999999999 2024-09-20T22:50:16.391 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:50:16.391 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:50:16.391 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:50:16.391 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.392 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.392 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:1527 7.17 e197/139 7:ea720b85:::smithi08477203-26:head v 197'159, mlcod=187'156)", 2024-09-20T22:50:16.392 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:44:36.209560+0000", 2024-09-20T22:50:16.392 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 340.14369662600001, 2024-09-20T22:50:16.392 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.010263069999999999, 2024-09-20T22:50:16.392 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:50:16.392 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:50:16.392 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:50:16.392 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:50:16.392 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.392 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:50:16.392 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.209560+0000", 2024-09-20T22:50:16.392 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:50:16.392 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.393 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.393 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:50:16.393 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.209560+0000", 2024-09-20T22:50:16.393 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:50:16.393 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.393 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.393 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:50:16.393 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.209560+0000", 2024-09-20T22:50:16.393 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.0800000000000001e-07 2024-09-20T22:50:16.393 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.393 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.393 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:50:16.393 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.212844+0000", 2024-09-20T22:50:16.393 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.003283907 2024-09-20T22:50:16.393 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.394 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.394 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:50:16.394 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.212867+0000", 2024-09-20T22:50:16.394 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.3229000000000001e-05 2024-09-20T22:50:16.394 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.394 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.394 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:50:16.394 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.212878+0000", 2024-09-20T22:50:16.394 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0998e-05 2024-09-20T22:50:16.394 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.394 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.394 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:50:16.394 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.212971+0000", 2024-09-20T22:50:16.394 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.2872000000000001e-05 2024-09-20T22:50:16.395 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.395 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.395 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:50:16.395 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.212996+0000", 2024-09-20T22:50:16.395 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.4921000000000001e-05 2024-09-20T22:50:16.395 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.395 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.395 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:50:16.395 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.219758+0000", 2024-09-20T22:50:16.395 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0067625139999999999 2024-09-20T22:50:16.395 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.395 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.395 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:50:16.395 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.219823+0000", 2024-09-20T22:50:16.396 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.4420999999999999e-05 2024-09-20T22:50:16.396 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:50:16.396 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:50:16.396 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:50:16.396 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.396 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.396 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:1825 7.b e230/94 7:d960cad4:::smithi08477203-18:head v 230'138, mlcod=229'135)", 2024-09-20T22:50:16.396 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:45:09.177560+0000", 2024-09-20T22:50:16.396 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 307.17569676199997, 2024-09-20T22:50:16.396 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0086763780000000002, 2024-09-20T22:50:16.396 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:50:16.396 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:50:16.396 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:50:16.397 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:50:16.397 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.397 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:50:16.397 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:09.177560+0000", 2024-09-20T22:50:16.397 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:50:16.397 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.397 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.397 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:50:16.397 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:09.177560+0000", 2024-09-20T22:50:16.397 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:50:16.397 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.397 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.397 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:50:16.397 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:09.177560+0000", 2024-09-20T22:50:16.398 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.0599999999999999e-07 2024-09-20T22:50:16.398 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.398 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.398 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:50:16.398 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:09.178522+0000", 2024-09-20T22:50:16.398 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00096277000000000005 2024-09-20T22:50:16.398 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.398 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.398 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:50:16.398 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:09.178536+0000", 2024-09-20T22:50:16.398 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3886999999999999e-05 2024-09-20T22:50:16.398 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.398 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.399 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:50:16.399 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:09.178547+0000", 2024-09-20T22:50:16.399 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0526999999999999e-05 2024-09-20T22:50:16.399 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.399 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.399 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:50:16.399 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:09.178614+0000", 2024-09-20T22:50:16.399 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.7547999999999998e-05 2024-09-20T22:50:16.399 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.399 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.399 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:50:16.399 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:09.178638+0000", 2024-09-20T22:50:16.399 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.3190999999999999e-05 2024-09-20T22:50:16.399 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.399 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.400 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:50:16.400 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:09.186193+0000", 2024-09-20T22:50:16.400 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0075549559999999998 2024-09-20T22:50:16.400 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.400 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.400 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:50:16.400 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:09.186236+0000", 2024-09-20T22:50:16.400 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.3293e-05 2024-09-20T22:50:16.400 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:50:16.400 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:50:16.400 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:50:16.400 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.400 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.401 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:2350 7.0 7:064f66ab:::smithi08477203-45:head [stat,delete] snapc 56=[56,55,53,52,51,50,47,3e] ondisk+write+known_if_redirected+supports_pool_eio e260)", 2024-09-20T22:50:16.401 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:45:39.199566+0000", 2024-09-20T22:50:16.401 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 277.153690197, 2024-09-20T22:50:16.401 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.027897148, 2024-09-20T22:50:16.401 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:50:16.401 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:50:16.401 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:50:16.401 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:50:16.401 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:50:16.401 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:50:16.401 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 2350 2024-09-20T22:50:16.401 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.401 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:50:16.402 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.402 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:50:16.402 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.199566+0000", 2024-09-20T22:50:16.402 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:50:16.402 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.402 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.402 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:50:16.402 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.199566+0000", 2024-09-20T22:50:16.402 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:50:16.402 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.402 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.402 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:50:16.402 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.199568+0000", 2024-09-20T22:50:16.402 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9290000000000001e-06 2024-09-20T22:50:16.403 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.403 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.403 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:50:16.403 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.199574+0000", 2024-09-20T22:50:16.403 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.3409999999999999e-06 2024-09-20T22:50:16.403 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.403 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.403 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:50:16.403 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.199584+0000", 2024-09-20T22:50:16.403 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0091e-05 2024-09-20T22:50:16.403 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.403 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.403 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:50:16.403 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.199593+0000", 2024-09-20T22:50:16.404 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.1079999999999993e-06 2024-09-20T22:50:16.404 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.404 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.404 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:50:16.404 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.199661+0000", 2024-09-20T22:50:16.404 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.8767000000000001e-05 2024-09-20T22:50:16.404 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.404 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.404 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:50:16.404 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.199765+0000", 2024-09-20T22:50:16.404 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000103886 2024-09-20T22:50:16.404 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.404 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.404 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:50:16.405 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.227463+0000", 2024-09-20T22:50:16.405 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.027698026000000001 2024-09-20T22:50:16.405 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:50:16.405 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:50:16.405 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:50:16.405 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.405 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.405 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:3134 7.2 7:45b54b6b:::smithi08477203-2:head [copy-from ver 151] snapc 77=[77,76,75,74,73,70,6b] ondisk+write+known_if_redirected+supports_pool_eio e328)", 2024-09-20T22:50:16.405 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:46:48.211103+0000", 2024-09-20T22:50:16.405 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 208.14215368999999, 2024-09-20T22:50:16.405 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.030812519, 2024-09-20T22:50:16.405 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:50:16.405 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:50:16.405 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:50:16.406 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:50:16.406 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:50:16.406 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:50:16.406 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 3134 2024-09-20T22:50:16.406 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.406 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:50:16.406 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.406 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:50:16.406 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.211103+0000", 2024-09-20T22:50:16.406 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:50:16.406 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.406 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.406 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:50:16.406 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.211103+0000", 2024-09-20T22:50:16.407 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:50:16.407 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.407 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.407 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:50:16.407 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.211105+0000", 2024-09-20T22:50:16.407 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.3369999999999998e-06 2024-09-20T22:50:16.407 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.407 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.407 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:50:16.407 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.211111+0000", 2024-09-20T22:50:16.407 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.2759999999999997e-06 2024-09-20T22:50:16.407 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.407 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.407 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:50:16.408 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.211123+0000", 2024-09-20T22:50:16.408 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2119999999999999e-05 2024-09-20T22:50:16.408 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.408 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.408 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:50:16.408 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.211131+0000", 2024-09-20T22:50:16.408 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.2899999999999997e-06 2024-09-20T22:50:16.408 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.408 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.408 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:50:16.408 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.211213+0000", 2024-09-20T22:50:16.408 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.2756999999999994e-05 2024-09-20T22:50:16.408 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.408 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.408 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:50:16.409 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.211310+0000", 2024-09-20T22:50:16.409 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.6644999999999994e-05 2024-09-20T22:50:16.409 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.409 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.409 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 1,2", 2024-09-20T22:50:16.409 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.229109+0000", 2024-09-20T22:50:16.409 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.017798587000000001 2024-09-20T22:50:16.409 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.409 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.409 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:50:16.409 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.240265+0000", 2024-09-20T22:50:16.409 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.011156716000000001 2024-09-20T22:50:16.409 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.409 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.410 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:50:16.410 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.241608+0000", 2024-09-20T22:50:16.410 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001343033 2024-09-20T22:50:16.410 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.410 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.410 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:50:16.410 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.241831+0000", 2024-09-20T22:50:16.410 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000223148 2024-09-20T22:50:16.410 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.410 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.410 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:50:16.410 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.241885+0000", 2024-09-20T22:50:16.410 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.3161000000000001e-05 2024-09-20T22:50:16.410 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.411 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.411 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:50:16.411 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.241915+0000", 2024-09-20T22:50:16.411 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.0448999999999999e-05 2024-09-20T22:50:16.411 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:50:16.411 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:50:16.411 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:50:16.411 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.411 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.411 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:3135 7.2 7:45b54b6b:::smithi08477203-2:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e328)", 2024-09-20T22:50:16.411 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:46:48.211153+0000", 2024-09-20T22:50:16.411 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 208.14210371799999, 2024-09-20T22:50:16.411 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.022731534000000001, 2024-09-20T22:50:16.411 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:50:16.412 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:50:16.412 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:50:16.412 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:50:16.412 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:50:16.412 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:50:16.412 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 3135 2024-09-20T22:50:16.412 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.412 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:50:16.412 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.412 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:50:16.412 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.211153+0000", 2024-09-20T22:50:16.412 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:50:16.412 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.412 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.412 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:50:16.413 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.211153+0000", 2024-09-20T22:50:16.413 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:50:16.413 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.413 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.413 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:50:16.413 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.211153+0000", 2024-09-20T22:50:16.413 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.1600000000000001e-07 2024-09-20T22:50:16.413 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.413 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.413 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:50:16.413 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.211157+0000", 2024-09-20T22:50:16.413 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.5470000000000001e-06 2024-09-20T22:50:16.413 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.413 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.413 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:50:16.414 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.211164+0000", 2024-09-20T22:50:16.414 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.2799999999999998e-06 2024-09-20T22:50:16.414 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.414 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.414 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:50:16.414 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.211170+0000", 2024-09-20T22:50:16.414 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.0730000000000001e-06 2024-09-20T22:50:16.414 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.414 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.414 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:50:16.414 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.211634+0000", 2024-09-20T22:50:16.414 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00046401800000000002 2024-09-20T22:50:16.414 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.414 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.415 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for blocked object", 2024-09-20T22:50:16.415 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.211690+0000", 2024-09-20T22:50:16.415 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.5587999999999997e-05 2024-09-20T22:50:16.415 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.415 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.415 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:50:16.415 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.233669+0000", 2024-09-20T22:50:16.415 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.021979240000000001 2024-09-20T22:50:16.415 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.415 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.415 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:50:16.415 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.233762+0000", 2024-09-20T22:50:16.415 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.2658999999999995e-05 2024-09-20T22:50:16.416 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.416 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.416 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:50:16.416 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.233884+0000", 2024-09-20T22:50:16.416 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00012241300000000001 2024-09-20T22:50:16.416 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:50:16.416 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:50:16.416 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:50:16.416 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.416 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.416 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:3136 7.a e328/94 7:5122292a:::smithi08477203-33:head v 328'252, mlcod=322'248)", 2024-09-20T22:50:16.416 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:46:48.223492+0000", 2024-09-20T22:50:16.416 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 208.12976400700001, 2024-09-20T22:50:16.416 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.016662746999999999, 2024-09-20T22:50:16.416 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:50:16.417 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:50:16.417 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:50:16.417 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:50:16.417 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.417 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:50:16.417 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.223492+0000", 2024-09-20T22:50:16.417 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:50:16.417 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.417 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.417 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:50:16.417 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.223492+0000", 2024-09-20T22:50:16.417 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:50:16.417 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.417 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.417 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:50:16.417 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.223493+0000", 2024-09-20T22:50:16.418 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.4299999999999998e-07 2024-09-20T22:50:16.418 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.418 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.418 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:50:16.418 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.228060+0000", 2024-09-20T22:50:16.418 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0045669350000000003 2024-09-20T22:50:16.418 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.418 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.418 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:50:16.418 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.228107+0000", 2024-09-20T22:50:16.418 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.6932999999999999e-05 2024-09-20T22:50:16.418 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.418 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.418 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:50:16.419 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.228127+0000", 2024-09-20T22:50:16.419 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.0777000000000001e-05 2024-09-20T22:50:16.419 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.419 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.419 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:50:16.419 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.228206+0000", 2024-09-20T22:50:16.419 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.8660999999999998e-05 2024-09-20T22:50:16.419 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.419 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.419 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:50:16.419 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.228232+0000", 2024-09-20T22:50:16.419 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.6199000000000001e-05 2024-09-20T22:50:16.419 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.419 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.419 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:50:16.420 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.240103+0000", 2024-09-20T22:50:16.420 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.011871008000000001 2024-09-20T22:50:16.420 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.420 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.420 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:50:16.420 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.240155+0000", 2024-09-20T22:50:16.420 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.1790999999999999e-05 2024-09-20T22:50:16.420 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:50:16.420 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:50:16.420 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:50:16.420 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.420 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.420 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:3232 7.16 e337/260 7:6885a65f:::smithi08477203-14:head v 337'107, mlcod=331'104)", 2024-09-20T22:50:16.420 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:46:57.231995+0000", 2024-09-20T22:50:16.421 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 199.12126105300001, 2024-09-20T22:50:16.421 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0090885140000000007, 2024-09-20T22:50:16.421 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:50:16.421 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:50:16.421 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:50:16.421 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:50:16.421 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.421 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:50:16.421 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.231995+0000", 2024-09-20T22:50:16.421 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:50:16.421 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.421 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.421 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:50:16.421 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.231995+0000", 2024-09-20T22:50:16.421 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:50:16.422 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.422 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.422 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:50:16.422 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.231996+0000", 2024-09-20T22:50:16.422 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.5799999999999998e-07 2024-09-20T22:50:16.422 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.422 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.422 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:50:16.422 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.234590+0000", 2024-09-20T22:50:16.422 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.002594482 2024-09-20T22:50:16.422 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.422 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.422 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:50:16.422 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.234617+0000", 2024-09-20T22:50:16.422 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.6855e-05 2024-09-20T22:50:16.423 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.423 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.423 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:50:16.423 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.234636+0000", 2024-09-20T22:50:16.423 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.8289000000000001e-05 2024-09-20T22:50:16.423 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.423 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.423 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:50:16.423 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.234729+0000", 2024-09-20T22:50:16.423 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.3207000000000004e-05 2024-09-20T22:50:16.423 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.423 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.423 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:50:16.424 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.234758+0000", 2024-09-20T22:50:16.424 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.8994999999999999e-05 2024-09-20T22:50:16.424 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.424 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.424 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:50:16.424 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.241031+0000", 2024-09-20T22:50:16.424 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0062733279999999999 2024-09-20T22:50:16.424 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.424 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.424 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:50:16.424 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.241084+0000", 2024-09-20T22:50:16.424 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.2599999999999998e-05 2024-09-20T22:50:16.424 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:50:16.424 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:50:16.424 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:50:16.425 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.425 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.425 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:3589 7.b e363/94 7:dc95255e:::smithi08477203-1:head v 363'419, mlcod=363'417)", 2024-09-20T22:50:16.425 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:47:23.205271+0000", 2024-09-20T22:50:16.425 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 173.14798502599999, 2024-09-20T22:50:16.425 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.014301911, 2024-09-20T22:50:16.425 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:50:16.425 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:50:16.425 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:50:16.425 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:50:16.425 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.425 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:50:16.425 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:23.205271+0000", 2024-09-20T22:50:16.425 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:50:16.425 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.426 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.426 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:50:16.426 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:23.205271+0000", 2024-09-20T22:50:16.426 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:50:16.426 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.426 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.426 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:50:16.426 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:23.205273+0000", 2024-09-20T22:50:16.426 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.409e-06 2024-09-20T22:50:16.426 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.426 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.426 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:50:16.427 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:16 smithi154 ceph-mon[33264]: 7.5 scrub starts 2024-09-20T22:50:16.427 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:16 smithi154 ceph-mon[33264]: 7.5 scrub ok 2024-09-20T22:50:16.427 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:16 smithi154 ceph-mon[33264]: 3.5 scrub starts 2024-09-20T22:50:16.427 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:16 smithi154 ceph-mon[33264]: 3.5 scrub ok 2024-09-20T22:50:16.427 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:16 smithi154 ceph-mon[33264]: 4.6 scrub starts 2024-09-20T22:50:16.427 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:16 smithi154 ceph-mon[33264]: 4.6 scrub ok 2024-09-20T22:50:16.427 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:16 smithi154 ceph-mon[33264]: 4.14 scrub starts 2024-09-20T22:50:16.427 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:16 smithi154 ceph-mon[33264]: 4.14 scrub ok 2024-09-20T22:50:16.428 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:16 smithi154 ceph-mon[33264]: 5.b scrub starts 2024-09-20T22:50:16.428 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:16 smithi154 ceph-mon[33264]: 5.b scrub ok 2024-09-20T22:50:16.428 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:16 smithi154 ceph-mon[33264]: 5.2 scrub starts 2024-09-20T22:50:16.428 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:16 smithi154 ceph-mon[33264]: 5.2 scrub ok 2024-09-20T22:50:16.428 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:16 smithi154 ceph-mon[33264]: 6.2 scrub starts 2024-09-20T22:50:16.428 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:16 smithi154 ceph-mon[33264]: 6.2 scrub ok 2024-09-20T22:50:16.428 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:16 smithi154 ceph-mon[33264]: osdmap e536: 8 total, 8 up, 8 in 2024-09-20T22:50:16.429 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:23.210044+0000", 2024-09-20T22:50:16.430 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0047712609999999997 2024-09-20T22:50:16.430 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.430 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.430 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:50:16.430 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:23.210064+0000", 2024-09-20T22:50:16.430 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.0421e-05 2024-09-20T22:50:16.430 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.430 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.430 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:50:16.430 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:23.210074+0000", 2024-09-20T22:50:16.430 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.4329999999999993e-06 2024-09-20T22:50:16.431 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.431 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.431 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:50:16.431 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:23.210130+0000", 2024-09-20T22:50:16.431 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.6286000000000003e-05 2024-09-20T22:50:16.431 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.431 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.431 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:50:16.431 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:23.210148+0000", 2024-09-20T22:50:16.431 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.7895999999999999e-05 2024-09-20T22:50:16.431 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.431 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.431 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:50:16.432 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:23.219532+0000", 2024-09-20T22:50:16.432 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.009383677 2024-09-20T22:50:16.432 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.432 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.432 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:50:16.432 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:23.219573+0000", 2024-09-20T22:50:16.432 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.1527999999999999e-05 2024-09-20T22:50:16.432 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:50:16.432 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:50:16.432 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:50:16.432 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.432 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.432 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:3760 7.13 e383/260 7:cae5792a:::smithi08477203-32:head v 383'78, mlcod=377'74)", 2024-09-20T22:50:16.432 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:47:43.291657+0000", 2024-09-20T22:50:16.432 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 153.061599148, 2024-09-20T22:50:16.433 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.011377195, 2024-09-20T22:50:16.433 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:50:16.433 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:50:16.433 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:50:16.433 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:50:16.433 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.433 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:50:16.433 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.291657+0000", 2024-09-20T22:50:16.433 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:50:16.433 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.433 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.433 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:50:16.433 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.291657+0000", 2024-09-20T22:50:16.434 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:50:16.434 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.434 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.434 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:50:16.434 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.291658+0000", 2024-09-20T22:50:16.434 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1009999999999999e-06 2024-09-20T22:50:16.434 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.434 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.434 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:50:16.434 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.296704+0000", 2024-09-20T22:50:16.434 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0050458070000000002 2024-09-20T22:50:16.434 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.434 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.434 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:50:16.435 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.296716+0000", 2024-09-20T22:50:16.435 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1525000000000001e-05 2024-09-20T22:50:16.435 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.435 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.435 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:50:16.435 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.296721+0000", 2024-09-20T22:50:16.435 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.8379999999999998e-06 2024-09-20T22:50:16.435 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.435 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.435 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:50:16.435 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.296769+0000", 2024-09-20T22:50:16.435 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.7120000000000003e-05 2024-09-20T22:50:16.435 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.436 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.436 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:50:16.436 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.296786+0000", 2024-09-20T22:50:16.436 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.7374e-05 2024-09-20T22:50:16.436 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.436 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.436 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:50:16.436 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.302998+0000", 2024-09-20T22:50:16.436 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0062118219999999997 2024-09-20T22:50:16.436 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.436 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.436 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:50:16.436 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.303034+0000", 2024-09-20T22:50:16.436 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.6607999999999999e-05 2024-09-20T22:50:16.437 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:50:16.437 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:50:16.437 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:50:16.437 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.437 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.437 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:4444 7.0 7:05bb442d:::smithi08477203-24:head [copy-from ver 421] snapc b8=[b8,b7,b6,b5,b4,b3,b2,b1,b0,ae,a9,a7,a6,a5,9e,9c,96,95,87,86] ondisk+write+known_if_redirected+supports_pool_eio e447)", 2024-09-20T22:50:16.437 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:48:47.227411+0000", 2024-09-20T22:50:16.437 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 89.125845581999997, 2024-09-20T22:50:16.437 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.033566952999999997, 2024-09-20T22:50:16.437 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:50:16.437 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:50:16.437 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:50:16.437 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:50:16.437 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:50:16.438 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:50:16.438 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 4444 2024-09-20T22:50:16.438 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.438 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:50:16.438 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.438 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:50:16.438 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.227411+0000", 2024-09-20T22:50:16.438 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:50:16.438 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.438 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.438 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:50:16.438 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.227411+0000", 2024-09-20T22:50:16.439 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:50:16.439 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.439 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.439 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:50:16.439 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.227413+0000", 2024-09-20T22:50:16.439 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.057e-06 2024-09-20T22:50:16.439 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.439 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.439 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:50:16.439 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.227422+0000", 2024-09-20T22:50:16.439 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.3219999999999999e-06 2024-09-20T22:50:16.439 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.439 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.439 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:50:16.440 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.227442+0000", 2024-09-20T22:50:16.440 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9449999999999998e-05 2024-09-20T22:50:16.440 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.440 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.440 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:50:16.440 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.227452+0000", 2024-09-20T22:50:16.440 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0699999999999999e-05 2024-09-20T22:50:16.440 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.440 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.440 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:50:16.440 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.227531+0000", 2024-09-20T22:50:16.440 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.8993999999999996e-05 2024-09-20T22:50:16.440 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.441 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.441 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op must wait for map", 2024-09-20T22:50:16.441 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.227542+0000", 2024-09-20T22:50:16.441 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0797e-05 2024-09-20T22:50:16.441 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.441 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.441 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:50:16.441 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.229501+0000", 2024-09-20T22:50:16.441 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0019594080000000002 2024-09-20T22:50:16.441 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.441 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.441 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:50:16.441 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.229616+0000", 2024-09-20T22:50:16.441 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000114577 2024-09-20T22:50:16.442 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.442 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.442 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 1,3", 2024-09-20T22:50:16.442 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.241752+0000", 2024-09-20T22:50:16.442 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.012135920999999999 2024-09-20T22:50:16.442 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.442 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.442 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:50:16.442 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.249601+0000", 2024-09-20T22:50:16.442 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0078493120000000006 2024-09-20T22:50:16.442 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.442 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.442 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:50:16.442 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.260731+0000", 2024-09-20T22:50:16.443 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.011129781 2024-09-20T22:50:16.443 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.443 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.443 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:50:16.443 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.260861+0000", 2024-09-20T22:50:16.443 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00013027899999999999 2024-09-20T22:50:16.443 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.443 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.443 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:50:16.443 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.260913+0000", 2024-09-20T22:50:16.443 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.2172999999999997e-05 2024-09-20T22:50:16.443 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.443 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.443 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:50:16.444 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.260978+0000", 2024-09-20T22:50:16.444 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.4182000000000003e-05 2024-09-20T22:50:16.444 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:50:16.444 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:50:16.444 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:50:16.444 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.444 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.444 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:4445 7.0 7:05bb442d:::smithi08477203-24:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e447)", 2024-09-20T22:50:16.444 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:48:47.227478+0000", 2024-09-20T22:50:16.444 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 89.125778644999997, 2024-09-20T22:50:16.444 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.019465758999999999, 2024-09-20T22:50:16.444 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:50:16.444 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:50:16.445 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:50:16.445 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:50:16.445 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:50:16.445 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:50:16.445 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 4445 2024-09-20T22:50:16.445 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.445 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:50:16.445 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.445 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:50:16.445 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.227478+0000", 2024-09-20T22:50:16.445 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:50:16.445 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.445 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.446 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:50:16.446 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.227478+0000", 2024-09-20T22:50:16.446 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:50:16.446 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.446 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.446 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:50:16.446 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.227478+0000", 2024-09-20T22:50:16.446 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.2200000000000004e-07 2024-09-20T22:50:16.446 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.446 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.446 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:50:16.446 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.227481+0000", 2024-09-20T22:50:16.446 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.0060000000000001e-06 2024-09-20T22:50:16.446 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.447 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.447 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:50:16.447 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.227500+0000", 2024-09-20T22:50:16.447 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9071000000000001e-05 2024-09-20T22:50:16.447 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.447 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.447 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:50:16.447 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.227507+0000", 2024-09-20T22:50:16.447 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.1779999999999997e-06 2024-09-20T22:50:16.447 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.447 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.447 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:50:16.447 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.227628+0000", 2024-09-20T22:50:16.448 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000120627 2024-09-20T22:50:16.448 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.448 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.448 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting_for_map not empty", 2024-09-20T22:50:16.448 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.227633+0000", 2024-09-20T22:50:16.448 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.4730000000000002e-06 2024-09-20T22:50:16.448 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.448 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.448 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:50:16.448 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.229766+0000", 2024-09-20T22:50:16.448 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0021332339999999999 2024-09-20T22:50:16.448 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.448 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.449 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for blocked object", 2024-09-20T22:50:16.449 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.229814+0000", 2024-09-20T22:50:16.449 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.7746e-05 2024-09-20T22:50:16.449 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.449 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.449 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:50:16.449 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.246707+0000", 2024-09-20T22:50:16.449 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.016893542000000001 2024-09-20T22:50:16.449 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.449 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.449 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:50:16.449 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.246863+0000", 2024-09-20T22:50:16.449 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000156105 2024-09-20T22:50:16.449 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.450 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.450 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:50:16.450 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.246943+0000", 2024-09-20T22:50:16.450 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.0154999999999994e-05 2024-09-20T22:50:16.450 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:50:16.450 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:50:16.450 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:50:16.450 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.450 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.450 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:5011 7.b e505/94 7:df7b2b73:::smithi08477203-10:head v 505'722, mlcod=503'718)", 2024-09-20T22:50:16.450 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:49:45.305974+0000", 2024-09-20T22:50:16.450 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 31.047282548999998, 2024-09-20T22:50:16.450 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.014321993, 2024-09-20T22:50:16.451 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:50:16.451 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:50:16.451 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:50:16.451 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:50:16.451 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.451 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:50:16.451 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:45.305974+0000", 2024-09-20T22:50:16.451 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:50:16.451 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.451 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.451 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:50:16.451 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:45.305974+0000", 2024-09-20T22:50:16.451 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:50:16.451 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.452 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.452 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:50:16.452 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:45.305974+0000", 2024-09-20T22:50:16.452 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.6199999999999997e-07 2024-09-20T22:50:16.452 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.452 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.452 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:50:16.452 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:45.311756+0000", 2024-09-20T22:50:16.452 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0057811019999999998 2024-09-20T22:50:16.452 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.452 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.452 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:50:16.452 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:45.311773+0000", 2024-09-20T22:50:16.452 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.7757999999999998e-05 2024-09-20T22:50:16.453 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.453 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.453 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:50:16.453 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:45.311783+0000", 2024-09-20T22:50:16.453 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.6639999999999993e-06 2024-09-20T22:50:16.453 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.453 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.453 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:50:16.453 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:45.311873+0000", 2024-09-20T22:50:16.453 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.0062000000000006e-05 2024-09-20T22:50:16.453 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.453 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.453 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:50:16.453 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:45.311900+0000", 2024-09-20T22:50:16.454 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.6475999999999999e-05 2024-09-20T22:50:16.454 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.454 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.454 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:50:16.454 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:45.320251+0000", 2024-09-20T22:50:16.454 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0083511560000000002 2024-09-20T22:50:16.454 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.454 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.454 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:50:16.454 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:45.320296+0000", 2024-09-20T22:50:16.454 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.5012999999999997e-05 2024-09-20T22:50:16.454 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:50:16.454 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:50:16.454 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:50:16.455 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.455 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.455 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:5151 7.0 7:05bb442d:::smithi08477203-24:head [write 749777~722868] snapc dd=[dd,dc,db,da,d9,d8,d7,d6,d5,d4,d3,d0,cf,ce,cd,ca,c7,c6,c5,c2,bf,ba,b4,b2,a7,9e,95,87] ondisk+write+known_if_redirected+supports_pool_eio e514)", 2024-09-20T22:50:16.455 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:49:54.638549+0000", 2024-09-20T22:50:16.455 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 21.714707406999999, 2024-09-20T22:50:16.455 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0092067450000000006, 2024-09-20T22:50:16.455 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:50:16.455 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:50:16.455 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:50:16.455 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:50:16.455 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:50:16.455 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:50:16.455 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 5151 2024-09-20T22:50:16.455 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.456 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:50:16.456 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.456 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:50:16.456 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:54.638549+0000", 2024-09-20T22:50:16.456 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:50:16.456 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.456 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.456 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:50:16.456 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:54.638549+0000", 2024-09-20T22:50:16.456 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:50:16.456 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.456 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.456 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:50:16.457 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:54.638551+0000", 2024-09-20T22:50:16.457 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9709999999999998e-06 2024-09-20T22:50:16.457 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.457 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.457 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:50:16.457 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:54.639792+0000", 2024-09-20T22:50:16.457 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0012410139999999999 2024-09-20T22:50:16.457 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.457 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.457 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:50:16.457 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:54.639813+0000", 2024-09-20T22:50:16.457 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.1148000000000001e-05 2024-09-20T22:50:16.457 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.457 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.458 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:50:16.458 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:54.639835+0000", 2024-09-20T22:50:16.458 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.1905000000000001e-05 2024-09-20T22:50:16.458 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.458 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.458 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:50:16.458 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:54.639901+0000", 2024-09-20T22:50:16.458 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.6548e-05 2024-09-20T22:50:16.458 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.458 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.458 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:50:16.458 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:54.639981+0000", 2024-09-20T22:50:16.458 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.9148999999999997e-05 2024-09-20T22:50:16.459 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.459 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.459 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 1,3", 2024-09-20T22:50:16.459 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:54.640098+0000", 2024-09-20T22:50:16.459 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000117337 2024-09-20T22:50:16.459 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.459 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.459 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:50:16.459 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:54.641401+0000", 2024-09-20T22:50:16.459 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0013027099999999999 2024-09-20T22:50:16.459 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.459 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.459 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:50:16.459 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:54.647561+0000", 2024-09-20T22:50:16.460 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.006160502 2024-09-20T22:50:16.460 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.460 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.460 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:50:16.460 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:54.647676+0000", 2024-09-20T22:50:16.460 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000114671 2024-09-20T22:50:16.460 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.460 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.460 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:50:16.460 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:54.647717+0000", 2024-09-20T22:50:16.460 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.1446000000000003e-05 2024-09-20T22:50:16.460 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:16.460 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:16.461 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:50:16.461 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:54.647756+0000", 2024-09-20T22:50:16.461 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.8343999999999997e-05 2024-09-20T22:50:16.461 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:50:16.461 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:50:16.461 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:50:16.461 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:50:16.461 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:50:16.461 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-20T22:50:16.463 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:50:16.464 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-20T22:50:16.466 INFO:teuthology.orchestra.run.smithi084.stderr:dumped all 2024-09-20T22:50:16.468 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:16 smithi084 ceph-mon[40024]: 7.5 scrub starts 2024-09-20T22:50:16.468 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:16 smithi084 ceph-mon[40024]: 7.5 scrub ok 2024-09-20T22:50:16.468 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:16 smithi084 ceph-mon[40024]: 3.5 scrub starts 2024-09-20T22:50:16.468 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:16 smithi084 ceph-mon[40024]: 3.5 scrub ok 2024-09-20T22:50:16.468 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:16 smithi084 ceph-mon[40024]: 4.6 scrub starts 2024-09-20T22:50:16.468 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:16 smithi084 ceph-mon[40024]: 4.6 scrub ok 2024-09-20T22:50:16.468 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:16 smithi084 ceph-mon[40024]: 4.14 scrub starts 2024-09-20T22:50:16.468 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:16 smithi084 ceph-mon[40024]: 4.14 scrub ok 2024-09-20T22:50:16.468 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:16 smithi084 ceph-mon[40024]: 5.b scrub starts 2024-09-20T22:50:16.468 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:16 smithi084 ceph-mon[40024]: 5.b scrub ok 2024-09-20T22:50:16.469 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:16 smithi084 ceph-mon[40024]: 5.2 scrub starts 2024-09-20T22:50:16.469 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:16 smithi084 ceph-mon[40024]: 5.2 scrub ok 2024-09-20T22:50:16.469 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:16 smithi084 ceph-mon[40024]: 6.2 scrub starts 2024-09-20T22:50:16.469 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:16 smithi084 ceph-mon[40024]: 6.2 scrub ok 2024-09-20T22:50:16.469 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:16 smithi084 ceph-mon[40024]: osdmap e536: 8 total, 8 up, 8 in 2024-09-20T22:50:16.469 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:16 smithi084 ceph-mon[29774]: 7.5 scrub starts 2024-09-20T22:50:16.469 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:16 smithi084 ceph-mon[29774]: 7.5 scrub ok 2024-09-20T22:50:16.469 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:16 smithi084 ceph-mon[29774]: 3.5 scrub starts 2024-09-20T22:50:16.469 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:16 smithi084 ceph-mon[29774]: 3.5 scrub ok 2024-09-20T22:50:16.469 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:16 smithi084 ceph-mon[29774]: 4.6 scrub starts 2024-09-20T22:50:16.470 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:16 smithi084 ceph-mon[29774]: 4.6 scrub ok 2024-09-20T22:50:16.470 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:16 smithi084 ceph-mon[29774]: 4.14 scrub starts 2024-09-20T22:50:16.470 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:16 smithi084 ceph-mon[29774]: 4.14 scrub ok 2024-09-20T22:50:16.470 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:16 smithi084 ceph-mon[29774]: 5.b scrub starts 2024-09-20T22:50:16.470 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:16 smithi084 ceph-mon[29774]: 5.b scrub ok 2024-09-20T22:50:16.470 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:16 smithi084 ceph-mon[29774]: 5.2 scrub starts 2024-09-20T22:50:16.470 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:16 smithi084 ceph-mon[29774]: 5.2 scrub ok 2024-09-20T22:50:16.470 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:16 smithi084 ceph-mon[29774]: 6.2 scrub starts 2024-09-20T22:50:16.470 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:16 smithi084 ceph-mon[29774]: 6.2 scrub ok 2024-09-20T22:50:16.470 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:16 smithi084 ceph-mon[29774]: osdmap e536: 8 total, 8 up, 8 in 2024-09-20T22:50:16.526 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:16.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:16.527 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:50:16.628 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:50:16.649 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:16.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:16.650 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:50:16.751 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:50:16.771 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:16.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:16.772 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:50:16.873 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:50:16.895 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:16.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:16.896 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:50:16.997 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:50:17.018 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:17.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:17.019 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:50:17.120 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:50:17.142 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:17.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:17.143 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:50:17.243 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:50:17.261 INFO:tasks.rados.rados.0.smithi084.stdout:2107: done (4 left) 2024-09-20T22:50:17.261 INFO:tasks.rados.rados.0.smithi084.stdout:2108: read oid 7 snap 230 2024-09-20T22:50:17.261 INFO:tasks.rados.rados.0.smithi084.stdout:2108: expect (ObjNum 728 snap 230 seq_num 728) 2024-09-20T22:50:17.261 INFO:tasks.rados.rados.0.smithi084.stdout:2109: snap_create 2024-09-20T22:50:17.262 INFO:tasks.rados.rados.0.smithi084.stdout:2106: finishing write tid 1 to smithi08477203-42 2024-09-20T22:50:17.262 INFO:tasks.rados.rados.0.smithi084.stdout:2106: finishing write tid 2 to smithi08477203-42 2024-09-20T22:50:17.262 INFO:tasks.rados.rados.0.smithi084.stdout:2106: finishing write tid 3 to smithi08477203-42 2024-09-20T22:50:17.262 INFO:tasks.rados.rados.0.smithi084.stdout:2106: finishing write tid 4 to smithi08477203-42 2024-09-20T22:50:17.262 INFO:tasks.rados.rados.0.smithi084.stdout:2106: oid 42 updating version 0 to 1209 2024-09-20T22:50:17.262 INFO:tasks.rados.rados.0.smithi084.stdout:2106: oid 42 updating version 1209 to 1210 2024-09-20T22:50:17.262 INFO:tasks.rados.rados.0.smithi084.stdout:2106: oid 42 version 1210 is already newer than 1208 2024-09-20T22:50:17.262 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 42 v 1210 (ObjNum 737 snap 232 seq_num 737) dirty exists 2024-09-20T22:50:17.262 INFO:tasks.rados.rados.0.smithi084.stdout:2106: left oid 42 (ObjNum 737 snap 232 seq_num 737) 2024-09-20T22:50:17.269 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-20T22:50:17.270 INFO:tasks.rados.rados.0.smithi084.stdout:2108: expect (ObjNum 728 snap 230 seq_num 728) 2024-09-20T22:50:17.271 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:17.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:17.272 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:50:17.373 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:50:17.395 INFO:teuthology.orchestra.run.smithi084.stdout:{"pg_ready":true,"pg_map":{"version":706,"stamp":"2024-09-20T22:50:16.254371+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":568468860,"num_objects":460,"num_object_clones":212,"num_object_copies":1380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":460,"num_whiteouts":19,"num_read":4829,"num_read_kb":3795425,"num_write":2989,"num_write_kb":937568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":887,"num_bytes_recovered":303303598,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6042,"ondisk_log_size":6042,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":474,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":1563424,"kb_used_data":1159856,"kb_used_omap":12,"kb_used_meta":403379,"kb_avail":748397792,"statfs":{"total":767960285184,"available":766359339008,"internally_reserved":0,"allocated":1187692544,"data_stored":1724278189,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":413060656},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":4414222,"num_objects":4,"num_object_clones":4,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":10,"num_read_kb":8,"num_write":14,"num_write_kb":2995,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.961586"},"pg_stats":[{"pgid":"3.1f","version":"86'2","reported_seq":312,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.233777+0000","last_change":"2024-09-20T22:49:26.647633+0000","last_active":"2024-09-20T22:50:13.233777+0000","last_peered":"2024-09-20T22:50:13.233777+0000","last_clean":"2024-09-20T22:50:13.233777+0000","last_became_active":"2024-09-20T22:47:49.222114+0000","last_became_peered":"2024-09-20T22:47:49.222114+0000","last_unstale":"2024-09-20T22:50:13.233777+0000","last_undegraded":"2024-09-20T22:50:13.233777+0000","last_fullsized":"2024-09-20T22:50:13.233777+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'2","last_scrub_stamp":"2024-09-20T22:49:26.647557+0000","last_deep_scrub":"86'2","last_deep_scrub_stamp":"2024-09-20T22:42:39.664421+0000","last_clean_scrub_stamp":"2024-09-20T22:49:26.647557+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:40.650510+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":436,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"74'9","reported_seq":285,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.232611+0000","last_change":"2024-09-20T22:49:51.091564+0000","last_active":"2024-09-20T22:50:13.232611+0000","last_peered":"2024-09-20T22:50:13.232611+0000","last_clean":"2024-09-20T22:50:13.232611+0000","last_became_active":"2024-09-20T22:48:05.248980+0000","last_became_peered":"2024-09-20T22:48:05.248980+0000","last_unstale":"2024-09-20T22:50:13.232611+0000","last_undegraded":"2024-09-20T22:50:13.232611+0000","last_fullsized":"2024-09-20T22:50:13.232611+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-09-20T22:49:51.091416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:49:51.091416+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:55.306855+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":969,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.528842+0000","last_change":"2024-09-20T22:50:12.718744+0000","last_active":"2024-09-20T22:50:13.528842+0000","last_peered":"2024-09-20T22:50:13.528842+0000","last_clean":"2024-09-20T22:50:13.528842+0000","last_became_active":"2024-09-20T22:41:42.182024+0000","last_became_peered":"2024-09-20T22:41:42.182024+0000","last_unstale":"2024-09-20T22:50:13.528842+0000","last_undegraded":"2024-09-20T22:50:13.528842+0000","last_fullsized":"2024-09-20T22:50:13.528842+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:50:12.718677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:50:12.718677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:51:36.198787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":275,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.233010+0000","last_change":"2024-09-20T22:49:49.117854+0000","last_active":"2024-09-20T22:50:13.233010+0000","last_peered":"2024-09-20T22:50:13.233010+0000","last_clean":"2024-09-20T22:50:13.233010+0000","last_became_active":"2024-09-20T22:48:05.247569+0000","last_became_peered":"2024-09-20T22:48:05.247569+0000","last_unstale":"2024-09-20T22:50:13.233010+0000","last_undegraded":"2024-09-20T22:50:13.233010+0000","last_fullsized":"2024-09-20T22:50:13.233010+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:49.117772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:49:49.117772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:51.805116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":956,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.252805+0000","last_change":"2024-09-20T22:49:50.043884+0000","last_active":"2024-09-20T22:50:13.252805+0000","last_peered":"2024-09-20T22:50:13.252805+0000","last_clean":"2024-09-20T22:50:13.252805+0000","last_became_active":"2024-09-20T22:41:44.832099+0000","last_became_peered":"2024-09-20T22:41:44.832099+0000","last_unstale":"2024-09-20T22:50:13.252805+0000","last_undegraded":"2024-09-20T22:50:13.252805+0000","last_fullsized":"2024-09-20T22:50:13.252805+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:50.043776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:49:50.043776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:51:00.726179+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":961,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.253673+0000","last_change":"2024-09-20T22:49:10.077843+0000","last_active":"2024-09-20T22:50:13.253673+0000","last_peered":"2024-09-20T22:50:13.253673+0000","last_clean":"2024-09-20T22:50:13.253673+0000","last_became_active":"2024-09-20T22:41:38.157298+0000","last_became_peered":"2024-09-20T22:41:38.157298+0000","last_unstale":"2024-09-20T22:50:13.253673+0000","last_undegraded":"2024-09-20T22:50:13.253673+0000","last_fullsized":"2024-09-20T22:50:13.253673+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:10.077774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:49:10.077774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:21.915867+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"74'15","reported_seq":1025,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.254951+0000","last_change":"2024-09-20T22:49:07.042205+0000","last_active":"2024-09-20T22:50:13.254951+0000","last_peered":"2024-09-20T22:50:13.254951+0000","last_clean":"2024-09-20T22:50:13.254951+0000","last_became_active":"2024-09-20T22:47:49.218798+0000","last_became_peered":"2024-09-20T22:47:49.218798+0000","last_unstale":"2024-09-20T22:50:13.254951+0000","last_undegraded":"2024-09-20T22:50:13.254951+0000","last_fullsized":"2024-09-20T22:50:13.254951+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-09-20T22:49:07.042130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:49:07.042130+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:20.414124+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":275,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.232481+0000","last_change":"2024-09-20T22:49:59.011922+0000","last_active":"2024-09-20T22:50:13.232481+0000","last_peered":"2024-09-20T22:50:13.232481+0000","last_clean":"2024-09-20T22:50:13.232481+0000","last_became_active":"2024-09-20T22:48:05.247636+0000","last_became_peered":"2024-09-20T22:48:05.247636+0000","last_unstale":"2024-09-20T22:50:13.232481+0000","last_undegraded":"2024-09-20T22:50:13.232481+0000","last_fullsized":"2024-09-20T22:50:13.232481+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:59.011829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:49:59.011829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:51:07.355546+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":981,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.527710+0000","last_change":"2024-09-20T22:49:02.680764+0000","last_active":"2024-09-20T22:50:13.527710+0000","last_peered":"2024-09-20T22:50:13.527710+0000","last_clean":"2024-09-20T22:50:13.527710+0000","last_became_active":"2024-09-20T22:48:05.242059+0000","last_became_peered":"2024-09-20T22:48:05.242059+0000","last_unstale":"2024-09-20T22:50:13.527710+0000","last_undegraded":"2024-09-20T22:50:13.527710+0000","last_fullsized":"2024-09-20T22:50:13.527710+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:02.680682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:46:00.999939+0000","last_clean_scrub_stamp":"2024-09-20T22:49:02.680682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:22.162227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"74'9","reported_seq":285,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.233024+0000","last_change":"2024-09-20T22:49:56.048604+0000","last_active":"2024-09-20T22:50:13.233024+0000","last_peered":"2024-09-20T22:50:13.233024+0000","last_clean":"2024-09-20T22:50:13.233024+0000","last_became_active":"2024-09-20T22:48:05.233429+0000","last_became_peered":"2024-09-20T22:48:05.233429+0000","last_unstale":"2024-09-20T22:50:13.233024+0000","last_undegraded":"2024-09-20T22:50:13.233024+0000","last_fullsized":"2024-09-20T22:50:13.233024+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-09-20T22:49:56.048504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:49:56.048504+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:58.603199+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":973,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.528151+0000","last_change":"2024-09-20T22:49:31.649058+0000","last_active":"2024-09-20T22:50:13.528151+0000","last_peered":"2024-09-20T22:50:13.528151+0000","last_clean":"2024-09-20T22:50:13.528151+0000","last_became_active":"2024-09-20T22:47:49.227379+0000","last_became_peered":"2024-09-20T22:47:49.227379+0000","last_unstale":"2024-09-20T22:50:13.528151+0000","last_undegraded":"2024-09-20T22:50:13.528151+0000","last_fullsized":"2024-09-20T22:50:13.528151+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:31.648986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:49:31.648986+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:47.218836+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":308,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.232758+0000","last_change":"2024-09-20T22:49:08.664519+0000","last_active":"2024-09-20T22:50:13.232758+0000","last_peered":"2024-09-20T22:50:13.232758+0000","last_clean":"2024-09-20T22:50:13.232758+0000","last_became_active":"2024-09-20T22:47:49.225515+0000","last_became_peered":"2024-09-20T22:47:49.225515+0000","last_unstale":"2024-09-20T22:50:13.232758+0000","last_undegraded":"2024-09-20T22:50:13.232758+0000","last_fullsized":"2024-09-20T22:50:13.232758+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:08.664407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:49:08.664407+0000","last_clean_scrub_stamp":"2024-09-20T22:49:08.664407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:33.922538+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":981,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.527685+0000","last_change":"2024-09-20T22:48:37.559081+0000","last_active":"2024-09-20T22:50:13.527685+0000","last_peered":"2024-09-20T22:50:13.527685+0000","last_clean":"2024-09-20T22:50:13.527685+0000","last_became_active":"2024-09-20T22:48:05.242084+0000","last_became_peered":"2024-09-20T22:48:05.242084+0000","last_unstale":"2024-09-20T22:50:13.527685+0000","last_undegraded":"2024-09-20T22:50:13.527685+0000","last_fullsized":"2024-09-20T22:50:13.527685+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:48:37.559007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:48:37.559007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"181'7","reported_seq":281,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.233513+0000","last_change":"2024-09-20T22:48:41.049491+0000","last_active":"2024-09-20T22:50:13.233513+0000","last_peered":"2024-09-20T22:50:13.233513+0000","last_clean":"2024-09-20T22:50:13.233513+0000","last_became_active":"2024-09-20T22:48:05.249763+0000","last_became_peered":"2024-09-20T22:48:05.249763+0000","last_unstale":"2024-09-20T22:50:13.233513+0000","last_undegraded":"2024-09-20T22:50:13.233513+0000","last_fullsized":"2024-09-20T22:50:13.233513+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"181'7","last_scrub_stamp":"2024-09-20T22:48:41.049381+0000","last_deep_scrub":"181'7","last_deep_scrub_stamp":"2024-09-20T22:48:41.049381+0000","last_clean_scrub_stamp":"2024-09-20T22:48:41.049381+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":6,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":995,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.251509+0000","last_change":"2024-09-20T22:49:48.313924+0000","last_active":"2024-09-20T22:50:13.251509+0000","last_peered":"2024-09-20T22:50:13.251509+0000","last_clean":"2024-09-20T22:50:13.251509+0000","last_became_active":"2024-09-20T22:48:05.237791+0000","last_became_peered":"2024-09-20T22:48:05.237791+0000","last_unstale":"2024-09-20T22:50:13.251509+0000","last_undegraded":"2024-09-20T22:50:13.251509+0000","last_fullsized":"2024-09-20T22:50:13.251509+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:48.313860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:47:40.361958+0000","last_clean_scrub_stamp":"2024-09-20T22:49:48.313860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:49.887950+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":957,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.528200+0000","last_change":"2024-09-20T22:49:04.679724+0000","last_active":"2024-09-20T22:50:13.528200+0000","last_peered":"2024-09-20T22:50:13.528200+0000","last_clean":"2024-09-20T22:50:13.528200+0000","last_became_active":"2024-09-20T22:41:44.197927+0000","last_became_peered":"2024-09-20T22:41:44.197927+0000","last_unstale":"2024-09-20T22:50:13.528200+0000","last_undegraded":"2024-09-20T22:50:13.528200+0000","last_fullsized":"2024-09-20T22:50:13.528200+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:04.679617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:49:04.679617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":177,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.234229+0000","last_change":"2024-09-20T22:48:50.224321+0000","last_active":"2024-09-20T22:50:13.234229+0000","last_peered":"2024-09-20T22:50:13.234229+0000","last_clean":"2024-09-20T22:50:13.234229+0000","last_became_active":"2024-09-20T22:48:50.223895+0000","last_became_peered":"2024-09-20T22:48:50.223895+0000","last_unstale":"2024-09-20T22:50:13.234229+0000","last_undegraded":"2024-09-20T22:50:13.234229+0000","last_fullsized":"2024-09-20T22:50:13.234229+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:47:21.819572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:47:21.819572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:13.855674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":313,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.233557+0000","last_change":"2024-09-20T22:49:53.933673+0000","last_active":"2024-09-20T22:50:13.233557+0000","last_peered":"2024-09-20T22:50:13.233557+0000","last_clean":"2024-09-20T22:50:13.233557+0000","last_became_active":"2024-09-20T22:48:05.233687+0000","last_became_peered":"2024-09-20T22:48:05.233687+0000","last_unstale":"2024-09-20T22:50:13.233557+0000","last_undegraded":"2024-09-20T22:50:13.233557+0000","last_fullsized":"2024-09-20T22:50:13.233557+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:53.933601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:49:53.933601+0000","last_clean_scrub_stamp":"2024-09-20T22:49:53.933601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:51:03.708862+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"113'22","reported_seq":1011,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.237379+0000","last_change":"2024-09-20T22:49:52.154967+0000","last_active":"2024-09-20T22:50:13.237379+0000","last_peered":"2024-09-20T22:50:13.237379+0000","last_clean":"2024-09-20T22:50:13.237379+0000","last_became_active":"2024-09-20T22:41:40.170131+0000","last_became_peered":"2024-09-20T22:41:40.170131+0000","last_unstale":"2024-09-20T22:50:13.237379+0000","last_undegraded":"2024-09-20T22:50:13.237379+0000","last_fullsized":"2024-09-20T22:50:13.237379+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'22","last_scrub_stamp":"2024-09-20T22:49:52.154856+0000","last_deep_scrub":"113'22","last_deep_scrub_stamp":"2024-09-20T22:43:12.191878+0000","last_clean_scrub_stamp":"2024-09-20T22:49:52.154856+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:51:04.660560+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":985,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.527968+0000","last_change":"2024-09-20T22:49:01.165679+0000","last_active":"2024-09-20T22:50:13.527968+0000","last_peered":"2024-09-20T22:50:13.527968+0000","last_clean":"2024-09-20T22:50:13.527968+0000","last_became_active":"2024-09-20T22:48:05.249417+0000","last_became_peered":"2024-09-20T22:48:05.249417+0000","last_unstale":"2024-09-20T22:50:13.527968+0000","last_undegraded":"2024-09-20T22:50:13.527968+0000","last_fullsized":"2024-09-20T22:50:13.527968+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:01.165605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:49:01.165605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":949,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.252755+0000","last_change":"2024-09-20T22:49:30.132559+0000","last_active":"2024-09-20T22:50:13.252755+0000","last_peered":"2024-09-20T22:50:13.252755+0000","last_clean":"2024-09-20T22:50:13.252755+0000","last_became_active":"2024-09-20T22:41:44.832131+0000","last_became_peered":"2024-09-20T22:41:44.832131+0000","last_unstale":"2024-09-20T22:50:13.252755+0000","last_undegraded":"2024-09-20T22:50:13.252755+0000","last_fullsized":"2024-09-20T22:50:13.252755+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:30.132466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:45:25.002199+0000","last_clean_scrub_stamp":"2024-09-20T22:49:30.132466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:30.604086+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":275,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.233226+0000","last_change":"2024-09-20T22:48:24.986582+0000","last_active":"2024-09-20T22:50:13.233226+0000","last_peered":"2024-09-20T22:50:13.233226+0000","last_clean":"2024-09-20T22:50:13.233226+0000","last_became_active":"2024-09-20T22:48:05.250581+0000","last_became_peered":"2024-09-20T22:48:05.250581+0000","last_unstale":"2024-09-20T22:50:13.233226+0000","last_undegraded":"2024-09-20T22:50:13.233226+0000","last_fullsized":"2024-09-20T22:50:13.233226+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:48:24.986484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:48:24.986484+0000","last_clean_scrub_stamp":"2024-09-20T22:48:24.986484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"74'12","reported_seq":1033,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.251153+0000","last_change":"2024-09-20T22:48:59.088132+0000","last_active":"2024-09-20T22:50:13.251153+0000","last_peered":"2024-09-20T22:50:13.251153+0000","last_clean":"2024-09-20T22:50:13.251153+0000","last_became_active":"2024-09-20T22:48:05.233983+0000","last_became_peered":"2024-09-20T22:48:05.233983+0000","last_unstale":"2024-09-20T22:50:13.251153+0000","last_undegraded":"2024-09-20T22:50:13.251153+0000","last_fullsized":"2024-09-20T22:50:13.251153+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-09-20T22:48:59.088030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:48:59.088030+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:14.192165+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":275,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.233609+0000","last_change":"2024-09-20T22:48:23.940233+0000","last_active":"2024-09-20T22:50:13.233609+0000","last_peered":"2024-09-20T22:50:13.233609+0000","last_clean":"2024-09-20T22:50:13.233609+0000","last_became_active":"2024-09-20T22:48:05.250319+0000","last_became_peered":"2024-09-20T22:48:05.250319+0000","last_unstale":"2024-09-20T22:50:13.233609+0000","last_undegraded":"2024-09-20T22:50:13.233609+0000","last_fullsized":"2024-09-20T22:50:13.233609+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:48:23.940157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:48:23.940157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"154'2","reported_seq":958,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.254969+0000","last_change":"2024-09-20T22:48:39.357067+0000","last_active":"2024-09-20T22:50:13.254969+0000","last_peered":"2024-09-20T22:50:13.254969+0000","last_clean":"2024-09-20T22:50:13.254969+0000","last_became_active":"2024-09-20T22:41:44.199822+0000","last_became_peered":"2024-09-20T22:41:44.199822+0000","last_unstale":"2024-09-20T22:50:13.254969+0000","last_undegraded":"2024-09-20T22:50:13.254969+0000","last_fullsized":"2024-09-20T22:50:13.254969+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"154'2","last_scrub_stamp":"2024-09-20T22:48:39.356937+0000","last_deep_scrub":"154'2","last_deep_scrub_stamp":"2024-09-20T22:43:54.134066+0000","last_clean_scrub_stamp":"2024-09-20T22:48:39.356937+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"67'1","reported_seq":990,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.527647+0000","last_change":"2024-09-20T22:48:42.188853+0000","last_active":"2024-09-20T22:50:13.527647+0000","last_peered":"2024-09-20T22:50:13.527647+0000","last_clean":"2024-09-20T22:50:13.527647+0000","last_became_active":"2024-09-20T22:48:05.249578+0000","last_became_peered":"2024-09-20T22:48:05.249578+0000","last_unstale":"2024-09-20T22:50:13.527647+0000","last_undegraded":"2024-09-20T22:50:13.527647+0000","last_fullsized":"2024-09-20T22:50:13.527647+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-09-20T22:48:42.188772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:48:42.188772+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"151'15","reported_seq":320,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.235300+0000","last_change":"2024-09-20T22:48:37.528315+0000","last_active":"2024-09-20T22:50:13.235300+0000","last_peered":"2024-09-20T22:50:13.235300+0000","last_clean":"2024-09-20T22:50:13.235300+0000","last_became_active":"2024-09-20T22:47:49.232181+0000","last_became_peered":"2024-09-20T22:47:49.232181+0000","last_unstale":"2024-09-20T22:50:13.235300+0000","last_undegraded":"2024-09-20T22:50:13.235300+0000","last_fullsized":"2024-09-20T22:50:13.235300+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"151'15","last_scrub_stamp":"2024-09-20T22:48:37.528207+0000","last_deep_scrub":"151'15","last_deep_scrub_stamp":"2024-09-20T22:43:50.715761+0000","last_clean_scrub_stamp":"2024-09-20T22:48:37.528207+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:13.855820+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"408'10","reported_seq":1107,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.248187+0000","last_change":"2024-09-20T22:49:47.030294+0000","last_active":"2024-09-20T22:50:13.248187+0000","last_peered":"2024-09-20T22:50:13.248187+0000","last_clean":"2024-09-20T22:50:13.248187+0000","last_became_active":"2024-09-20T22:48:05.244270+0000","last_became_peered":"2024-09-20T22:48:05.244270+0000","last_unstale":"2024-09-20T22:50:13.248187+0000","last_undegraded":"2024-09-20T22:50:13.248187+0000","last_fullsized":"2024-09-20T22:50:13.248187+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"408'10","last_scrub_stamp":"2024-09-20T22:49:47.030217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:49:47.030217+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:56.479090+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.c","version":"533'1417","reported_seq":3571,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.271092+0000","last_change":"2024-09-20T22:50:11.300284+0000","last_active":"2024-09-20T22:50:13.271092+0000","last_peered":"2024-09-20T22:50:13.271092+0000","last_clean":"2024-09-20T22:50:13.271092+0000","last_became_active":"2024-09-20T22:42:52.404027+0000","last_became_peered":"2024-09-20T22:42:52.404027+0000","last_unstale":"2024-09-20T22:50:13.271092+0000","last_undegraded":"2024-09-20T22:50:13.271092+0000","last_fullsized":"2024-09-20T22:50:13.271092+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"508'1323","last_scrub_stamp":"2024-09-20T22:49:49.001444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:49:49.001444+0000","objects_scrubbed":31,"log_size":1417,"log_dups_size":0,"ondisk_log_size":1417,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:51:17.495494+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0.0046908890000000002,"stat_sum":{"num_bytes":77726889,"num_objects":35,"num_object_clones":28,"num_object_copies":105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":3,"num_read":629,"num_read_kb":641119,"num_write":445,"num_write_kb":170717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"d9","length":"1"},{"start":"dd","length":"2"},{"start":"e1","length":"1"}]},{"pgid":"1.a","version":"77'128","reported_seq":1228,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.528019+0000","last_change":"2024-09-20T22:48:24.566023+0000","last_active":"2024-09-20T22:50:13.528019+0000","last_peered":"2024-09-20T22:50:13.528019+0000","last_clean":"2024-09-20T22:50:13.528019+0000","last_became_active":"2024-09-20T22:42:52.407791+0000","last_became_peered":"2024-09-20T22:42:52.407791+0000","last_unstale":"2024-09-20T22:50:13.528019+0000","last_undegraded":"2024-09-20T22:50:13.528019+0000","last_fullsized":"2024-09-20T22:50:13.528019+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:48:24.565920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:48:24.565920+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"74'15","reported_seq":1042,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.528280+0000","last_change":"2024-09-20T22:49:55.709162+0000","last_active":"2024-09-20T22:50:13.528280+0000","last_peered":"2024-09-20T22:50:13.528280+0000","last_clean":"2024-09-20T22:50:13.528280+0000","last_became_active":"2024-09-20T22:48:05.250308+0000","last_became_peered":"2024-09-20T22:48:05.250308+0000","last_unstale":"2024-09-20T22:50:13.528280+0000","last_undegraded":"2024-09-20T22:50:13.528280+0000","last_fullsized":"2024-09-20T22:50:13.528280+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-09-20T22:49:55.709043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:49:55.709043+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:51:25.064971+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":960,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.251635+0000","last_change":"2024-09-20T22:49:12.070747+0000","last_active":"2024-09-20T22:50:13.251635+0000","last_peered":"2024-09-20T22:50:13.251635+0000","last_clean":"2024-09-20T22:50:13.251635+0000","last_became_active":"2024-09-20T22:41:38.156749+0000","last_became_peered":"2024-09-20T22:41:38.156749+0000","last_unstale":"2024-09-20T22:50:13.251635+0000","last_undegraded":"2024-09-20T22:50:13.251635+0000","last_fullsized":"2024-09-20T22:50:13.251635+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:12.070675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:49:12.070675+0000","last_clean_scrub_stamp":"2024-09-20T22:49:12.070675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:34.126084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"408'12","reported_seq":297,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.234172+0000","last_change":"2024-09-20T22:48:05.251167+0000","last_active":"2024-09-20T22:50:13.234172+0000","last_peered":"2024-09-20T22:50:13.234172+0000","last_clean":"2024-09-20T22:50:13.234172+0000","last_became_active":"2024-09-20T22:48:05.242914+0000","last_became_peered":"2024-09-20T22:48:05.242914+0000","last_unstale":"2024-09-20T22:50:13.234172+0000","last_undegraded":"2024-09-20T22:50:13.234172+0000","last_fullsized":"2024-09-20T22:50:13.234172+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"264'10","last_scrub_stamp":"2024-09-20T22:47:20.300787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:47:20.300787+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":969,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.528405+0000","last_change":"2024-09-20T22:49:07.658309+0000","last_active":"2024-09-20T22:50:13.528405+0000","last_peered":"2024-09-20T22:50:13.528405+0000","last_clean":"2024-09-20T22:50:13.528405+0000","last_became_active":"2024-09-20T22:47:49.227039+0000","last_became_peered":"2024-09-20T22:47:49.227039+0000","last_unstale":"2024-09-20T22:50:13.528405+0000","last_undegraded":"2024-09-20T22:50:13.528405+0000","last_fullsized":"2024-09-20T22:50:13.528405+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:07.658234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:49:07.658234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:26.817461+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3","version":"533'190","reported_seq":2228,"reported_epoch":533,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T22:50:13.325928+0000","last_change":"2024-09-20T22:50:13.325928+0000","last_active":"2024-09-20T22:50:13.325928+0000","last_peered":"2024-09-20T22:50:13.325928+0000","last_clean":"2024-09-20T22:50:13.325928+0000","last_became_active":"2024-09-20T22:49:05.221402+0000","last_became_peered":"2024-09-20T22:49:05.221402+0000","last_unstale":"2024-09-20T22:50:13.325928+0000","last_undegraded":"2024-09-20T22:50:13.325928+0000","last_fullsized":"2024-09-20T22:50:13.325928+0000","mapping_epoch":404,"log_start":"456'129","ondisk_log_start":"456'129","created":94,"last_epoch_clean":465,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'8","last_scrub_stamp":"2024-09-20T22:47:23.137750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:47:23.137750+0000","objects_scrubbed":0,"log_size":61,"log_dups_size":149,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.073199573000000004,"stat_sum":{"num_bytes":26803665,"num_objects":9,"num_object_clones":8,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":69,"num_read_kb":85984,"num_write":47,"num_write_kb":18651,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":20196357,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"d9","length":"1"},{"start":"dd","length":"2"}]},{"pgid":"1.5","version":"77'128","reported_seq":1134,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.233946+0000","last_change":"2024-09-20T22:49:26.023144+0000","last_active":"2024-09-20T22:50:13.233946+0000","last_peered":"2024-09-20T22:50:13.233946+0000","last_clean":"2024-09-20T22:50:13.233946+0000","last_became_active":"2024-09-20T22:48:08.207018+0000","last_became_peered":"2024-09-20T22:48:08.207018+0000","last_unstale":"2024-09-20T22:50:13.233946+0000","last_undegraded":"2024-09-20T22:50:13.233946+0000","last_fullsized":"2024-09-20T22:50:13.233946+0000","mapping_epoch":407,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":408,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:49:26.023044+0000","last_deep_scrub":"77'128","last_deep_scrub_stamp":"2024-09-20T22:45:41.608075+0000","last_clean_scrub_stamp":"2024-09-20T22:49:26.023044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:29.392816+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"74'18","reported_seq":1037,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.251878+0000","last_change":"2024-09-20T22:48:43.146697+0000","last_active":"2024-09-20T22:50:13.251878+0000","last_peered":"2024-09-20T22:50:13.251878+0000","last_clean":"2024-09-20T22:50:13.251878+0000","last_became_active":"2024-09-20T22:47:49.217605+0000","last_became_peered":"2024-09-20T22:47:49.217605+0000","last_unstale":"2024-09-20T22:50:13.251878+0000","last_undegraded":"2024-09-20T22:50:13.251878+0000","last_fullsized":"2024-09-20T22:50:13.251878+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'18","last_scrub_stamp":"2024-09-20T22:48:43.146602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:48:43.146602+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":972,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.251243+0000","last_change":"2024-09-20T22:47:49.227431+0000","last_active":"2024-09-20T22:50:13.251243+0000","last_peered":"2024-09-20T22:50:13.251243+0000","last_clean":"2024-09-20T22:50:13.251243+0000","last_became_active":"2024-09-20T22:47:49.227069+0000","last_became_peered":"2024-09-20T22:47:49.227069+0000","last_unstale":"2024-09-20T22:50:13.251243+0000","last_undegraded":"2024-09-20T22:50:13.251243+0000","last_fullsized":"2024-09-20T22:50:13.251243+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:47:39.982616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:47:39.982616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":276,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.233049+0000","last_change":"2024-09-20T22:49:27.968569+0000","last_active":"2024-09-20T22:50:13.233049+0000","last_peered":"2024-09-20T22:50:13.233049+0000","last_clean":"2024-09-20T22:50:13.233049+0000","last_became_active":"2024-09-20T22:48:05.230003+0000","last_became_peered":"2024-09-20T22:48:05.230003+0000","last_unstale":"2024-09-20T22:50:13.233049+0000","last_undegraded":"2024-09-20T22:50:13.233049+0000","last_fullsized":"2024-09-20T22:50:13.233049+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:27.968498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:07.947945+0000","last_clean_scrub_stamp":"2024-09-20T22:49:27.968498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:39.692203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":268,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.233161+0000","last_change":"2024-09-20T22:48:05.240266+0000","last_active":"2024-09-20T22:50:13.233161+0000","last_peered":"2024-09-20T22:50:13.233161+0000","last_clean":"2024-09-20T22:50:13.233161+0000","last_became_active":"2024-09-20T22:48:05.239917+0000","last_became_peered":"2024-09-20T22:48:05.239917+0000","last_unstale":"2024-09-20T22:50:13.233161+0000","last_undegraded":"2024-09-20T22:50:13.233161+0000","last_fullsized":"2024-09-20T22:50:13.233161+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:47:18.179575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:53.036601+0000","last_clean_scrub_stamp":"2024-09-20T22:47:18.179575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2","version":"525'529","reported_seq":2002,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.253953+0000","last_change":"2024-09-20T22:50:11.324189+0000","last_active":"2024-09-20T22:50:13.253953+0000","last_peered":"2024-09-20T22:50:13.253953+0000","last_clean":"2024-09-20T22:50:13.253953+0000","last_became_active":"2024-09-20T22:49:17.242481+0000","last_became_peered":"2024-09-20T22:49:17.242481+0000","last_unstale":"2024-09-20T22:50:13.253953+0000","last_undegraded":"2024-09-20T22:50:13.253953+0000","last_fullsized":"2024-09-20T22:50:13.253953+0000","mapping_epoch":469,"log_start":"471'464","ondisk_log_start":"471'464","created":94,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"282'120","last_scrub_stamp":"2024-09-20T22:46:22.099642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:46:22.099642+0000","objects_scrubbed":1,"log_size":65,"log_dups_size":687,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.030344758999999999,"stat_sum":{"num_bytes":37522363,"num_objects":17,"num_object_clones":13,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":339,"num_read_kb":374975,"num_write":185,"num_write_kb":61120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":120,"num_bytes_recovered":49828787,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"d9","length":"1"},{"start":"dd","length":"2"},{"start":"e1","length":"1"}]},{"pgid":"1.4","version":"77'128","reported_seq":1260,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.255548+0000","last_change":"2024-09-20T22:49:58.063457+0000","last_active":"2024-09-20T22:50:13.255548+0000","last_peered":"2024-09-20T22:50:13.255548+0000","last_clean":"2024-09-20T22:50:13.255548+0000","last_became_active":"2024-09-20T22:46:15.181305+0000","last_became_peered":"2024-09-20T22:46:15.181305+0000","last_unstale":"2024-09-20T22:50:13.255548+0000","last_undegraded":"2024-09-20T22:50:13.255548+0000","last_fullsized":"2024-09-20T22:50:13.255548+0000","mapping_epoch":94,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:49:58.063382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:49:58.063382+0000","objects_scrubbed":1,"log_size":0,"log_dups_size":59,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:51:06.139917+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"74'14","reported_seq":292,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.232961+0000","last_change":"2024-09-20T22:49:57.026346+0000","last_active":"2024-09-20T22:50:13.232961+0000","last_peered":"2024-09-20T22:50:13.232961+0000","last_clean":"2024-09-20T22:50:13.232961+0000","last_became_active":"2024-09-20T22:48:05.235812+0000","last_became_peered":"2024-09-20T22:48:05.235812+0000","last_unstale":"2024-09-20T22:50:13.232961+0000","last_undegraded":"2024-09-20T22:50:13.232961+0000","last_fullsized":"2024-09-20T22:50:13.232961+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'14","last_scrub_stamp":"2024-09-20T22:49:57.026245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:49:57.026245+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:51:23.191208+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"67'1","reported_seq":322,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.234831+0000","last_change":"2024-09-20T22:48:36.570753+0000","last_active":"2024-09-20T22:50:13.234831+0000","last_peered":"2024-09-20T22:50:13.234831+0000","last_clean":"2024-09-20T22:50:13.234831+0000","last_became_active":"2024-09-20T22:48:05.237425+0000","last_became_peered":"2024-09-20T22:48:05.237425+0000","last_unstale":"2024-09-20T22:50:13.234831+0000","last_undegraded":"2024-09-20T22:50:13.234831+0000","last_fullsized":"2024-09-20T22:50:13.234831+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-09-20T22:48:36.570672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:48:36.570672+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:13.855836+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"408'12","reported_seq":1105,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.250982+0000","last_change":"2024-09-20T22:49:25.151447+0000","last_active":"2024-09-20T22:50:13.250982+0000","last_peered":"2024-09-20T22:50:13.250982+0000","last_clean":"2024-09-20T22:50:13.250982+0000","last_became_active":"2024-09-20T22:48:05.234607+0000","last_became_peered":"2024-09-20T22:48:05.234607+0000","last_unstale":"2024-09-20T22:50:13.250982+0000","last_undegraded":"2024-09-20T22:50:13.250982+0000","last_fullsized":"2024-09-20T22:50:13.250982+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"408'12","last_scrub_stamp":"2024-09-20T22:49:25.151326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:49:25.151326+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:44.177686+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":957,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.253670+0000","last_change":"2024-09-20T22:49:31.310399+0000","last_active":"2024-09-20T22:50:13.253670+0000","last_peered":"2024-09-20T22:50:13.253670+0000","last_clean":"2024-09-20T22:50:13.253670+0000","last_became_active":"2024-09-20T22:41:44.831901+0000","last_became_peered":"2024-09-20T22:41:44.831901+0000","last_unstale":"2024-09-20T22:50:13.253670+0000","last_undegraded":"2024-09-20T22:50:13.253670+0000","last_fullsized":"2024-09-20T22:50:13.253670+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:31.310330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:49:31.310330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:34.862425+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1","version":"520'283","reported_seq":2400,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.233148+0000","last_change":"2024-09-20T22:50:11.283938+0000","last_active":"2024-09-20T22:50:13.233148+0000","last_peered":"2024-09-20T22:50:13.233148+0000","last_clean":"2024-09-20T22:50:13.233148+0000","last_became_active":"2024-09-20T22:49:21.232750+0000","last_became_peered":"2024-09-20T22:49:21.232750+0000","last_unstale":"2024-09-20T22:50:13.233148+0000","last_undegraded":"2024-09-20T22:50:13.233148+0000","last_fullsized":"2024-09-20T22:50:13.233148+0000","mapping_epoch":407,"log_start":"474'259","ondisk_log_start":"474'259","created":94,"last_epoch_clean":481,"parent":"0.0","parent_split_bits":0,"last_scrub":"420'231","last_scrub_stamp":"2024-09-20T22:48:22.915993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:48:22.915993+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":338,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00047044300000000002,"stat_sum":{"num_bytes":13160244,"num_objects":6,"num_object_clones":4,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":65,"num_read_kb":78461,"num_write":63,"num_write_kb":38776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":8732217,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"d9","length":"1"},{"start":"dd","length":"2"},{"start":"e1","length":"1"}]},{"pgid":"1.7","version":"77'128","reported_seq":1265,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.249274+0000","last_change":"2024-09-20T22:49:31.994386+0000","last_active":"2024-09-20T22:50:13.249274+0000","last_peered":"2024-09-20T22:50:13.249274+0000","last_clean":"2024-09-20T22:50:13.249274+0000","last_became_active":"2024-09-20T22:46:03.169596+0000","last_became_peered":"2024-09-20T22:46:03.169596+0000","last_unstale":"2024-09-20T22:50:13.249274+0000","last_undegraded":"2024-09-20T22:50:13.249274+0000","last_fullsized":"2024-09-20T22:50:13.249274+0000","mapping_epoch":94,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:49:31.994311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:49:31.994311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:35.643288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"74'10","reported_seq":1061,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.527823+0000","last_change":"2024-09-20T22:49:46.855799+0000","last_active":"2024-09-20T22:50:13.527823+0000","last_peered":"2024-09-20T22:50:13.527823+0000","last_clean":"2024-09-20T22:50:13.527823+0000","last_became_active":"2024-09-20T22:48:05.251200+0000","last_became_peered":"2024-09-20T22:48:05.251200+0000","last_unstale":"2024-09-20T22:50:13.527823+0000","last_undegraded":"2024-09-20T22:50:13.527823+0000","last_fullsized":"2024-09-20T22:50:13.527823+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-09-20T22:49:46.855673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:49:46.855673+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:55.301300+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":970,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.528756+0000","last_change":"2024-09-20T22:48:04.520131+0000","last_active":"2024-09-20T22:50:13.528756+0000","last_peered":"2024-09-20T22:50:13.528756+0000","last_clean":"2024-09-20T22:50:13.528756+0000","last_became_active":"2024-09-20T22:41:38.155629+0000","last_became_peered":"2024-09-20T22:41:38.155629+0000","last_unstale":"2024-09-20T22:50:13.528756+0000","last_undegraded":"2024-09-20T22:50:13.528756+0000","last_fullsized":"2024-09-20T22:50:13.528756+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:48:04.520053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:48:04.520053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"389'11","reported_seq":342,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.233508+0000","last_change":"2024-09-20T22:48:44.601907+0000","last_active":"2024-09-20T22:50:13.233508+0000","last_peered":"2024-09-20T22:50:13.233508+0000","last_clean":"2024-09-20T22:50:13.233508+0000","last_became_active":"2024-09-20T22:47:49.227264+0000","last_became_peered":"2024-09-20T22:47:49.227264+0000","last_unstale":"2024-09-20T22:50:13.233508+0000","last_undegraded":"2024-09-20T22:50:13.233508+0000","last_fullsized":"2024-09-20T22:50:13.233508+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'11","last_scrub_stamp":"2024-09-20T22:48:44.601816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:48:44.601816+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:16.790892+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":308,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.232580+0000","last_change":"2024-09-20T22:49:27.627363+0000","last_active":"2024-09-20T22:50:13.232580+0000","last_peered":"2024-09-20T22:50:13.232580+0000","last_clean":"2024-09-20T22:50:13.232580+0000","last_became_active":"2024-09-20T22:47:49.232380+0000","last_became_peered":"2024-09-20T22:47:49.232380+0000","last_unstale":"2024-09-20T22:50:13.232580+0000","last_undegraded":"2024-09-20T22:50:13.232580+0000","last_fullsized":"2024-09-20T22:50:13.232580+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:27.627295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:49:27.627295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:29.554480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.0","version":"529'315","reported_seq":2514,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.528056+0000","last_change":"2024-09-20T22:50:11.279156+0000","last_active":"2024-09-20T22:50:13.528056+0000","last_peered":"2024-09-20T22:50:13.528056+0000","last_clean":"2024-09-20T22:50:13.528056+0000","last_became_active":"2024-09-20T22:49:25.236419+0000","last_became_peered":"2024-09-20T22:49:25.236419+0000","last_unstale":"2024-09-20T22:50:13.528056+0000","last_undegraded":"2024-09-20T22:50:13.528056+0000","last_fullsized":"2024-09-20T22:50:13.528056+0000","mapping_epoch":94,"log_start":"480'253","ondisk_log_start":"480'253","created":94,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'228","last_scrub_stamp":"2024-09-20T22:48:59.633944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:48:59.633944+0000","objects_scrubbed":7,"log_size":62,"log_dups_size":416,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:23.086443+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.000544186,"stat_sum":{"num_bytes":32604467,"num_objects":17,"num_object_clones":14,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":1,"num_read":165,"num_read_kb":185505,"num_write":109,"num_write_kb":40172,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":7603720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"d9","length":"1"},{"start":"dd","length":"2"},{"start":"e1","length":"1"}]},{"pgid":"1.6","version":"77'128","reported_seq":1287,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.249034+0000","last_change":"2024-09-20T22:49:28.091254+0000","last_active":"2024-09-20T22:50:13.249034+0000","last_peered":"2024-09-20T22:50:13.249034+0000","last_clean":"2024-09-20T22:50:13.249034+0000","last_became_active":"2024-09-20T22:47:52.212190+0000","last_became_peered":"2024-09-20T22:47:52.212190+0000","last_unstale":"2024-09-20T22:50:13.249034+0000","last_undegraded":"2024-09-20T22:50:13.249034+0000","last_fullsized":"2024-09-20T22:50:13.249034+0000","mapping_epoch":391,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:49:28.091184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:49:28.091184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:40.740952+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"515'27","reported_seq":334,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.232193+0000","last_change":"2024-09-20T22:49:55.964247+0000","last_active":"2024-09-20T22:50:13.232193+0000","last_peered":"2024-09-20T22:50:13.232193+0000","last_clean":"2024-09-20T22:50:13.232193+0000","last_became_active":"2024-09-20T22:47:49.222009+0000","last_became_peered":"2024-09-20T22:47:49.222009+0000","last_unstale":"2024-09-20T22:50:13.232193+0000","last_undegraded":"2024-09-20T22:50:13.232193+0000","last_fullsized":"2024-09-20T22:50:13.232193+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"515'27","last_scrub_stamp":"2024-09-20T22:49:55.964153+0000","last_deep_scrub":"515'27","last_deep_scrub_stamp":"2024-09-20T22:49:55.964153+0000","last_clean_scrub_stamp":"2024-09-20T22:49:55.964153+0000","objects_scrubbed":8,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:57.993517+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":38,"num_read_kb":24,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":978,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.529214+0000","last_change":"2024-09-20T22:49:28.084441+0000","last_active":"2024-09-20T22:50:13.529214+0000","last_peered":"2024-09-20T22:50:13.529214+0000","last_clean":"2024-09-20T22:50:13.529214+0000","last_became_active":"2024-09-20T22:41:38.160832+0000","last_became_peered":"2024-09-20T22:41:38.160832+0000","last_unstale":"2024-09-20T22:50:13.529214+0000","last_undegraded":"2024-09-20T22:50:13.529214+0000","last_fullsized":"2024-09-20T22:50:13.529214+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:28.084371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:48:23.320031+0000","last_clean_scrub_stamp":"2024-09-20T22:49:28.084371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:30.046341+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":973,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.248487+0000","last_change":"2024-09-20T22:49:00.208581+0000","last_active":"2024-09-20T22:50:13.248487+0000","last_peered":"2024-09-20T22:50:13.248487+0000","last_clean":"2024-09-20T22:50:13.248487+0000","last_became_active":"2024-09-20T22:47:49.214055+0000","last_became_peered":"2024-09-20T22:47:49.214055+0000","last_unstale":"2024-09-20T22:50:13.248487+0000","last_undegraded":"2024-09-20T22:50:13.248487+0000","last_fullsized":"2024-09-20T22:50:13.248487+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:00.208454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:49:00.208454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":958,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.528741+0000","last_change":"2024-09-20T22:48:41.208933+0000","last_active":"2024-09-20T22:50:13.528741+0000","last_peered":"2024-09-20T22:50:13.528741+0000","last_clean":"2024-09-20T22:50:13.528741+0000","last_became_active":"2024-09-20T22:41:44.830538+0000","last_became_peered":"2024-09-20T22:41:44.830538+0000","last_unstale":"2024-09-20T22:50:13.528741+0000","last_undegraded":"2024-09-20T22:50:13.528741+0000","last_fullsized":"2024-09-20T22:50:13.528741+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:48:41.208853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:48:41.208853+0000","last_clean_scrub_stamp":"2024-09-20T22:48:41.208853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.7","version":"531'712","reported_seq":2993,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.249528+0000","last_change":"2024-09-20T22:50:11.297831+0000","last_active":"2024-09-20T22:50:13.249528+0000","last_peered":"2024-09-20T22:50:13.249528+0000","last_clean":"2024-09-20T22:50:13.249528+0000","last_became_active":"2024-09-20T22:48:43.224386+0000","last_became_peered":"2024-09-20T22:48:43.224386+0000","last_unstale":"2024-09-20T22:50:13.249528+0000","last_undegraded":"2024-09-20T22:50:13.249528+0000","last_fullsized":"2024-09-20T22:50:13.249528+0000","mapping_epoch":391,"log_start":"436'456","ondisk_log_start":"436'456","created":94,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"463'527","last_scrub_stamp":"2024-09-20T22:49:04.175508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:49:04.175508+0000","objects_scrubbed":23,"log_size":256,"log_dups_size":886,"ondisk_log_size":256,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.0034558119999999999,"stat_sum":{"num_bytes":53169886,"num_objects":31,"num_object_clones":26,"num_object_copies":93,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":4,"num_read":341,"num_read_kb":364157,"num_write":261,"num_write_kb":86105,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":115,"num_bytes_recovered":51615167,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"d9","length":"1"},{"start":"dd","length":"2"},{"start":"e1","length":"1"}]},{"pgid":"1.1","version":"77'128","reported_seq":1276,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.528166+0000","last_change":"2024-09-20T22:48:44.631654+0000","last_active":"2024-09-20T22:50:13.528166+0000","last_peered":"2024-09-20T22:50:13.528166+0000","last_clean":"2024-09-20T22:50:13.528166+0000","last_became_active":"2024-09-20T22:47:52.208102+0000","last_became_peered":"2024-09-20T22:47:52.208102+0000","last_unstale":"2024-09-20T22:50:13.528166+0000","last_undegraded":"2024-09-20T22:50:13.528166+0000","last_fullsized":"2024-09-20T22:50:13.528166+0000","mapping_epoch":391,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:48:44.631571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:48:44.631571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:15.693452+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"86'38","reported_seq":1136,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.529109+0000","last_change":"2024-09-20T22:49:31.040446+0000","last_active":"2024-09-20T22:50:13.529109+0000","last_peered":"2024-09-20T22:50:13.529109+0000","last_clean":"2024-09-20T22:50:13.529109+0000","last_became_active":"2024-09-20T22:41:40.166958+0000","last_became_peered":"2024-09-20T22:41:40.166958+0000","last_unstale":"2024-09-20T22:50:13.529109+0000","last_undegraded":"2024-09-20T22:50:13.529109+0000","last_fullsized":"2024-09-20T22:50:13.529109+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'38","last_scrub_stamp":"2024-09-20T22:49:31.040362+0000","last_deep_scrub":"86'38","last_deep_scrub_stamp":"2024-09-20T22:42:43.935677+0000","last_clean_scrub_stamp":"2024-09-20T22:49:31.040362+0000","objects_scrubbed":10,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:55.004800+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":268,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.233463+0000","last_change":"2024-09-20T22:48:05.238748+0000","last_active":"2024-09-20T22:50:13.233463+0000","last_peered":"2024-09-20T22:50:13.233463+0000","last_clean":"2024-09-20T22:50:13.233463+0000","last_became_active":"2024-09-20T22:48:05.237756+0000","last_became_peered":"2024-09-20T22:48:05.237756+0000","last_unstale":"2024-09-20T22:50:13.233463+0000","last_undegraded":"2024-09-20T22:50:13.233463+0000","last_fullsized":"2024-09-20T22:50:13.233463+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:47:16.817468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:47:16.817468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"69'2","reported_seq":988,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.528741+0000","last_change":"2024-09-20T22:49:27.604083+0000","last_active":"2024-09-20T22:50:13.528741+0000","last_peered":"2024-09-20T22:50:13.528741+0000","last_clean":"2024-09-20T22:50:13.528741+0000","last_became_active":"2024-09-20T22:41:37.148479+0000","last_became_peered":"2024-09-20T22:41:37.148479+0000","last_unstale":"2024-09-20T22:50:13.528741+0000","last_undegraded":"2024-09-20T22:50:13.528741+0000","last_fullsized":"2024-09-20T22:50:13.528741+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2024-09-20T22:49:27.603975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:36.130684+0000","last_clean_scrub_stamp":"2024-09-20T22:49:27.603975+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:37.292200+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00087280599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":314,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.234017+0000","last_change":"2024-09-20T22:49:02.723760+0000","last_active":"2024-09-20T22:50:13.234017+0000","last_peered":"2024-09-20T22:50:13.234017+0000","last_clean":"2024-09-20T22:50:13.234017+0000","last_became_active":"2024-09-20T22:48:05.233519+0000","last_became_peered":"2024-09-20T22:48:05.233519+0000","last_unstale":"2024-09-20T22:50:13.234017+0000","last_undegraded":"2024-09-20T22:50:13.234017+0000","last_fullsized":"2024-09-20T22:50:13.234017+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:02.723689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:49:02.723689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:30.036737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":968,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.251368+0000","last_change":"2024-09-20T22:49:57.022315+0000","last_active":"2024-09-20T22:50:13.251368+0000","last_peered":"2024-09-20T22:50:13.251368+0000","last_clean":"2024-09-20T22:50:13.251368+0000","last_became_active":"2024-09-20T22:48:05.234193+0000","last_became_peered":"2024-09-20T22:48:05.234193+0000","last_unstale":"2024-09-20T22:50:13.251368+0000","last_undegraded":"2024-09-20T22:50:13.251368+0000","last_fullsized":"2024-09-20T22:50:13.251368+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:57.022244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:49:57.022244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:51:26.737278+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"531'248","reported_seq":896,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.232773+0000","last_change":"2024-09-20T22:50:11.291631+0000","last_active":"2024-09-20T22:50:13.232773+0000","last_peered":"2024-09-20T22:50:13.232773+0000","last_clean":"2024-09-20T22:50:13.232773+0000","last_became_active":"2024-09-20T22:49:01.223578+0000","last_became_peered":"2024-09-20T22:49:01.223578+0000","last_unstale":"2024-09-20T22:50:13.232773+0000","last_undegraded":"2024-09-20T22:50:13.232773+0000","last_fullsized":"2024-09-20T22:50:13.232773+0000","mapping_epoch":404,"log_start":"456'176","ondisk_log_start":"456'176","created":94,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"461'178","last_scrub_stamp":"2024-09-20T22:49:01.696205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:49:01.696205+0000","objects_scrubbed":10,"log_size":72,"log_dups_size":318,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:18.018181+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0090190549999999998,"stat_sum":{"num_bytes":27923761,"num_objects":12,"num_object_clones":10,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":1,"num_read":117,"num_read_kb":115419,"num_write":92,"num_write_kb":32954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":10294275,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"d9","length":"1"},{"start":"dd","length":"2"},{"start":"e1","length":"1"}]},{"pgid":"1.2","version":"77'128","reported_seq":1345,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.253221+0000","last_change":"2024-09-20T22:49:55.364495+0000","last_active":"2024-09-20T22:50:13.253221+0000","last_peered":"2024-09-20T22:50:13.253221+0000","last_clean":"2024-09-20T22:50:13.253221+0000","last_became_active":"2024-09-20T22:48:08.203028+0000","last_became_peered":"2024-09-20T22:48:08.203028+0000","last_unstale":"2024-09-20T22:50:13.253221+0000","last_undegraded":"2024-09-20T22:50:13.253221+0000","last_fullsized":"2024-09-20T22:50:13.253221+0000","mapping_epoch":407,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":408,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:49:55.364421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:49:55.364421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:59.878790+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"443'18","reported_seq":1041,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.528196+0000","last_change":"2024-09-20T22:49:57.739451+0000","last_active":"2024-09-20T22:50:13.528196+0000","last_peered":"2024-09-20T22:50:13.528196+0000","last_clean":"2024-09-20T22:50:13.528196+0000","last_became_active":"2024-09-20T22:47:49.228702+0000","last_became_peered":"2024-09-20T22:47:49.228702+0000","last_unstale":"2024-09-20T22:50:13.528196+0000","last_undegraded":"2024-09-20T22:50:13.528196+0000","last_fullsized":"2024-09-20T22:50:13.528196+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"443'18","last_scrub_stamp":"2024-09-20T22:49:57.739362+0000","last_deep_scrub":"443'18","last_deep_scrub_stamp":"2024-09-20T22:48:44.182517+0000","last_clean_scrub_stamp":"2024-09-20T22:49:57.739362+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:51:00.790911+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":978,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.528581+0000","last_change":"2024-09-20T22:49:26.102066+0000","last_active":"2024-09-20T22:50:13.528581+0000","last_peered":"2024-09-20T22:50:13.528581+0000","last_clean":"2024-09-20T22:50:13.528581+0000","last_became_active":"2024-09-20T22:41:38.156837+0000","last_became_peered":"2024-09-20T22:41:38.156837+0000","last_unstale":"2024-09-20T22:50:13.528581+0000","last_undegraded":"2024-09-20T22:50:13.528581+0000","last_fullsized":"2024-09-20T22:50:13.528581+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:26.101994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:49:26.101994+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:29.729787+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"380'2","reported_seq":1014,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.237229+0000","last_change":"2024-09-20T22:49:30.148041+0000","last_active":"2024-09-20T22:50:13.237229+0000","last_peered":"2024-09-20T22:50:13.237229+0000","last_clean":"2024-09-20T22:50:13.237229+0000","last_became_active":"2024-09-20T22:47:49.222812+0000","last_became_peered":"2024-09-20T22:47:49.222812+0000","last_unstale":"2024-09-20T22:50:13.237229+0000","last_undegraded":"2024-09-20T22:50:13.237229+0000","last_fullsized":"2024-09-20T22:50:13.237229+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"380'2","last_scrub_stamp":"2024-09-20T22:49:30.147961+0000","last_deep_scrub":"380'2","last_deep_scrub_stamp":"2024-09-20T22:47:40.983039+0000","last_clean_scrub_stamp":"2024-09-20T22:49:30.147961+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:32.024321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001301459,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":967,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.237528+0000","last_change":"2024-09-20T22:49:27.195816+0000","last_active":"2024-09-20T22:50:13.237528+0000","last_peered":"2024-09-20T22:50:13.237528+0000","last_clean":"2024-09-20T22:50:13.237528+0000","last_became_active":"2024-09-20T22:41:42.185112+0000","last_became_peered":"2024-09-20T22:41:42.185112+0000","last_unstale":"2024-09-20T22:50:13.237528+0000","last_undegraded":"2024-09-20T22:50:13.237528+0000","last_fullsized":"2024-09-20T22:50:13.237528+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:27.195708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:47:22.881109+0000","last_clean_scrub_stamp":"2024-09-20T22:49:27.195708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:46.992983+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":957,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.255259+0000","last_change":"2024-09-20T22:49:04.332292+0000","last_active":"2024-09-20T22:50:13.255259+0000","last_peered":"2024-09-20T22:50:13.255259+0000","last_clean":"2024-09-20T22:50:13.255259+0000","last_became_active":"2024-09-20T22:41:44.830515+0000","last_became_peered":"2024-09-20T22:41:44.830515+0000","last_unstale":"2024-09-20T22:50:13.255259+0000","last_undegraded":"2024-09-20T22:50:13.255259+0000","last_fullsized":"2024-09-20T22:50:13.255259+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:04.332219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:49:04.332219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:26.530832+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"531'225","reported_seq":2452,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.255058+0000","last_change":"2024-09-20T22:50:11.288571+0000","last_active":"2024-09-20T22:50:13.255058+0000","last_peered":"2024-09-20T22:50:13.255058+0000","last_clean":"2024-09-20T22:50:13.255058+0000","last_became_active":"2024-09-20T22:48:57.236417+0000","last_became_peered":"2024-09-20T22:48:57.236417+0000","last_unstale":"2024-09-20T22:50:13.255058+0000","last_undegraded":"2024-09-20T22:50:13.255058+0000","last_fullsized":"2024-09-20T22:50:13.255058+0000","mapping_epoch":391,"log_start":"450'190","ondisk_log_start":"450'190","created":94,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'168","last_scrub_stamp":"2024-09-20T22:48:22.483112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:48:22.483112+0000","objects_scrubbed":7,"log_size":35,"log_dups_size":240,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0040204330000000003,"stat_sum":{"num_bytes":16569587,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":57,"num_read_kb":38715,"num_write":75,"num_write_kb":33558,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":19673726,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"d9","length":"1"},{"start":"dd","length":"2"},{"start":"e1","length":"1"}]},{"pgid":"1.3","version":"77'128","reported_seq":1293,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.528587+0000","last_change":"2024-09-20T22:49:49.789854+0000","last_active":"2024-09-20T22:50:13.528587+0000","last_peered":"2024-09-20T22:50:13.528587+0000","last_clean":"2024-09-20T22:50:13.528587+0000","last_became_active":"2024-09-20T22:48:08.208402+0000","last_became_peered":"2024-09-20T22:48:08.208402+0000","last_unstale":"2024-09-20T22:50:13.528587+0000","last_undegraded":"2024-09-20T22:50:13.528587+0000","last_fullsized":"2024-09-20T22:50:13.528587+0000","mapping_epoch":407,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":408,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:49:49.789778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:49:49.789778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:51:06.258069+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"74'12","reported_seq":314,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.232423+0000","last_change":"2024-09-20T22:47:49.246080+0000","last_active":"2024-09-20T22:50:13.232423+0000","last_peered":"2024-09-20T22:50:13.232423+0000","last_clean":"2024-09-20T22:50:13.232423+0000","last_became_active":"2024-09-20T22:47:49.224223+0000","last_became_peered":"2024-09-20T22:47:49.224223+0000","last_unstale":"2024-09-20T22:50:13.232423+0000","last_undegraded":"2024-09-20T22:50:13.232423+0000","last_fullsized":"2024-09-20T22:50:13.232423+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-09-20T22:47:39.293455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:47:39.293455+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:13.855657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":314,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.233366+0000","last_change":"2024-09-20T22:48:05.233412+0000","last_active":"2024-09-20T22:50:13.233366+0000","last_peered":"2024-09-20T22:50:13.233366+0000","last_clean":"2024-09-20T22:50:13.233366+0000","last_became_active":"2024-09-20T22:48:05.233093+0000","last_became_peered":"2024-09-20T22:48:05.233093+0000","last_unstale":"2024-09-20T22:50:13.233366+0000","last_undegraded":"2024-09-20T22:50:13.233366+0000","last_fullsized":"2024-09-20T22:50:13.233366+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:47:55.576301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:44:19.834641+0000","last_clean_scrub_stamp":"2024-09-20T22:47:55.576301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:13.855736+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"75'5","reported_seq":1508,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.253391+0000","last_change":"2024-09-20T22:48:44.254029+0000","last_active":"2024-09-20T22:50:13.253391+0000","last_peered":"2024-09-20T22:50:13.253391+0000","last_clean":"2024-09-20T22:50:13.253391+0000","last_became_active":"2024-09-20T22:47:49.214414+0000","last_became_peered":"2024-09-20T22:47:49.214414+0000","last_unstale":"2024-09-20T22:50:13.253391+0000","last_undegraded":"2024-09-20T22:50:13.253391+0000","last_fullsized":"2024-09-20T22:50:13.253391+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-09-20T22:48:44.253907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:36.130684+0000","last_clean_scrub_stamp":"2024-09-20T22:48:44.253907+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00078631700000000003,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":506,"num_read_kb":501,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":973,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.528773+0000","last_change":"2024-09-20T22:48:34.535169+0000","last_active":"2024-09-20T22:50:13.528773+0000","last_peered":"2024-09-20T22:50:13.528773+0000","last_clean":"2024-09-20T22:50:13.528773+0000","last_became_active":"2024-09-20T22:47:49.227371+0000","last_became_peered":"2024-09-20T22:47:49.227371+0000","last_unstale":"2024-09-20T22:50:13.528773+0000","last_undegraded":"2024-09-20T22:50:13.528773+0000","last_fullsized":"2024-09-20T22:50:13.528773+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:48:34.535100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:48:34.535100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":966,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.528887+0000","last_change":"2024-09-20T22:49:10.150277+0000","last_active":"2024-09-20T22:50:13.528887+0000","last_peered":"2024-09-20T22:50:13.528887+0000","last_clean":"2024-09-20T22:50:13.528887+0000","last_became_active":"2024-09-20T22:41:44.199046+0000","last_became_peered":"2024-09-20T22:41:44.199046+0000","last_unstale":"2024-09-20T22:50:13.528887+0000","last_undegraded":"2024-09-20T22:50:13.528887+0000","last_fullsized":"2024-09-20T22:50:13.528887+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:10.150205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:49:10.150205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.6","version":"531'272","reported_seq":2430,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.235053+0000","last_change":"2024-09-20T22:50:11.291261+0000","last_active":"2024-09-20T22:50:13.235053+0000","last_peered":"2024-09-20T22:50:13.235053+0000","last_clean":"2024-09-20T22:50:13.235053+0000","last_became_active":"2024-09-20T22:48:47.234531+0000","last_became_peered":"2024-09-20T22:48:47.234531+0000","last_unstale":"2024-09-20T22:50:13.235053+0000","last_undegraded":"2024-09-20T22:50:13.235053+0000","last_fullsized":"2024-09-20T22:50:13.235053+0000","mapping_epoch":391,"log_start":"443'181","ondisk_log_start":"443'181","created":94,"last_epoch_clean":447,"parent":"0.0","parent_split_bits":0,"last_scrub":"440'177","last_scrub_stamp":"2024-09-20T22:48:40.602342+0000","last_deep_scrub":"440'177","last_deep_scrub_stamp":"2024-09-20T22:48:40.602342+0000","last_clean_scrub_stamp":"2024-09-20T22:48:40.602342+0000","objects_scrubbed":4,"log_size":91,"log_dups_size":314,"ondisk_log_size":91,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0.0079483039999999998,"stat_sum":{"num_bytes":21209978,"num_objects":10,"num_object_clones":8,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":1,"num_read":113,"num_read_kb":107760,"num_write":90,"num_write_kb":30256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":18662529,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"d9","length":"1"},{"start":"dd","length":"2"},{"start":"e1","length":"1"}]},{"pgid":"4.5","version":"408'20","reported_seq":1195,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.248150+0000","last_change":"2024-09-20T22:49:56.004694+0000","last_active":"2024-09-20T22:50:13.248150+0000","last_peered":"2024-09-20T22:50:13.248150+0000","last_clean":"2024-09-20T22:50:13.248150+0000","last_became_active":"2024-09-20T22:48:05.243523+0000","last_became_peered":"2024-09-20T22:48:05.243523+0000","last_unstale":"2024-09-20T22:50:13.248150+0000","last_undegraded":"2024-09-20T22:50:13.248150+0000","last_fullsized":"2024-09-20T22:50:13.248150+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"408'20","last_scrub_stamp":"2024-09-20T22:49:56.004590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:49:56.004590+0000","objects_scrubbed":8,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:51:25.151724+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"67'1","reported_seq":967,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.254451+0000","last_change":"2024-09-20T22:49:29.161456+0000","last_active":"2024-09-20T22:50:13.254451+0000","last_peered":"2024-09-20T22:50:13.254451+0000","last_clean":"2024-09-20T22:50:13.254451+0000","last_became_active":"2024-09-20T22:41:38.158917+0000","last_became_peered":"2024-09-20T22:41:38.158917+0000","last_unstale":"2024-09-20T22:50:13.254451+0000","last_undegraded":"2024-09-20T22:50:13.254451+0000","last_fullsized":"2024-09-20T22:50:13.254451+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-09-20T22:49:29.161334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:49:29.161334+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:43.763458+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"77'128","reported_seq":1317,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.253542+0000","last_change":"2024-09-20T22:49:46.391695+0000","last_active":"2024-09-20T22:50:13.253542+0000","last_peered":"2024-09-20T22:50:13.253542+0000","last_clean":"2024-09-20T22:50:13.253542+0000","last_became_active":"2024-09-20T22:47:52.206169+0000","last_became_peered":"2024-09-20T22:47:52.206169+0000","last_unstale":"2024-09-20T22:50:13.253542+0000","last_undegraded":"2024-09-20T22:50:13.253542+0000","last_fullsized":"2024-09-20T22:50:13.253542+0000","mapping_epoch":391,"log_start":"77'128","ondisk_log_start":"77'128","created":19,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:49:46.391614+0000","last_deep_scrub":"77'128","last_deep_scrub_stamp":"2024-09-20T22:47:22.417099+0000","last_clean_scrub_stamp":"2024-09-20T22:49:46.391614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:51.817750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":961,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.255073+0000","last_change":"2024-09-20T22:48:35.359093+0000","last_active":"2024-09-20T22:50:13.255073+0000","last_peered":"2024-09-20T22:50:13.255073+0000","last_clean":"2024-09-20T22:50:13.255073+0000","last_became_active":"2024-09-20T22:41:42.184387+0000","last_became_peered":"2024-09-20T22:41:42.184387+0000","last_unstale":"2024-09-20T22:50:13.255073+0000","last_undegraded":"2024-09-20T22:50:13.255073+0000","last_fullsized":"2024-09-20T22:50:13.255073+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:48:35.359016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:48:35.359016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":977,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.527884+0000","last_change":"2024-09-20T22:49:57.715494+0000","last_active":"2024-09-20T22:50:13.527884+0000","last_peered":"2024-09-20T22:50:13.527884+0000","last_clean":"2024-09-20T22:50:13.527884+0000","last_became_active":"2024-09-20T22:48:05.241602+0000","last_became_peered":"2024-09-20T22:48:05.241602+0000","last_unstale":"2024-09-20T22:50:13.527884+0000","last_undegraded":"2024-09-20T22:50:13.527884+0000","last_fullsized":"2024-09-20T22:50:13.527884+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:57.715416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:49:57.715416+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:51:26.966224+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"518'313","reported_seq":2537,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.249052+0000","last_change":"2024-09-20T22:50:11.296028+0000","last_active":"2024-09-20T22:50:13.249052+0000","last_peered":"2024-09-20T22:50:13.249052+0000","last_clean":"2024-09-20T22:50:13.249052+0000","last_became_active":"2024-09-20T22:47:52.208200+0000","last_became_peered":"2024-09-20T22:47:52.208200+0000","last_unstale":"2024-09-20T22:50:13.249052+0000","last_undegraded":"2024-09-20T22:50:13.249052+0000","last_fullsized":"2024-09-20T22:50:13.249052+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"460'246","last_scrub_stamp":"2024-09-20T22:49:01.214729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:49:01.214729+0000","objects_scrubbed":8,"log_size":313,"log_dups_size":0,"ondisk_log_size":313,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00079812500000000001,"stat_sum":{"num_bytes":24977291,"num_objects":9,"num_object_clones":7,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":114,"num_read_kb":106336,"num_write":68,"num_write_kb":29565,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":13529610,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"d9","length":"1"},{"start":"dd","length":"2"},{"start":"e1","length":"1"}]},{"pgid":"4.e","version":"74'11","reported_seq":279,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.234166+0000","last_change":"2024-09-20T22:48:05.271944+0000","last_active":"2024-09-20T22:50:13.234166+0000","last_peered":"2024-09-20T22:50:13.234166+0000","last_clean":"2024-09-20T22:50:13.234166+0000","last_became_active":"2024-09-20T22:48:05.248271+0000","last_became_peered":"2024-09-20T22:48:05.248271+0000","last_unstale":"2024-09-20T22:50:13.234166+0000","last_undegraded":"2024-09-20T22:50:13.234166+0000","last_fullsized":"2024-09-20T22:50:13.234166+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-09-20T22:47:59.092235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:47:59.092235+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":275,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.232617+0000","last_change":"2024-09-20T22:49:30.998080+0000","last_active":"2024-09-20T22:50:13.232617+0000","last_peered":"2024-09-20T22:50:13.232617+0000","last_clean":"2024-09-20T22:50:13.232617+0000","last_became_active":"2024-09-20T22:48:05.254688+0000","last_became_peered":"2024-09-20T22:48:05.254688+0000","last_unstale":"2024-09-20T22:50:13.232617+0000","last_undegraded":"2024-09-20T22:50:13.232617+0000","last_fullsized":"2024-09-20T22:50:13.232617+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:30.998001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:49:30.998001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:47.839033+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":973,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.528008+0000","last_change":"2024-09-20T22:49:48.693437+0000","last_active":"2024-09-20T22:50:13.528008+0000","last_peered":"2024-09-20T22:50:13.528008+0000","last_clean":"2024-09-20T22:50:13.528008+0000","last_became_active":"2024-09-20T22:48:05.233778+0000","last_became_peered":"2024-09-20T22:48:05.233778+0000","last_unstale":"2024-09-20T22:50:13.528008+0000","last_undegraded":"2024-09-20T22:50:13.528008+0000","last_fullsized":"2024-09-20T22:50:13.528008+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:48.693333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:49:48.693333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:56.935700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":948,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.255484+0000","last_change":"2024-09-20T22:48:40.127195+0000","last_active":"2024-09-20T22:50:13.255484+0000","last_peered":"2024-09-20T22:50:13.255484+0000","last_clean":"2024-09-20T22:50:13.255484+0000","last_became_active":"2024-09-20T22:41:44.832009+0000","last_became_peered":"2024-09-20T22:41:44.832009+0000","last_unstale":"2024-09-20T22:50:13.255484+0000","last_undegraded":"2024-09-20T22:50:13.255484+0000","last_fullsized":"2024-09-20T22:50:13.255484+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:48:40.127123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:48:40.127123+0000","last_clean_scrub_stamp":"2024-09-20T22:48:40.127123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.e","version":"531'408","reported_seq":2656,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.237538+0000","last_change":"2024-09-20T22:50:11.287467+0000","last_active":"2024-09-20T22:50:13.237538+0000","last_peered":"2024-09-20T22:50:13.237538+0000","last_clean":"2024-09-20T22:50:13.237538+0000","last_became_active":"2024-09-20T22:47:52.209452+0000","last_became_peered":"2024-09-20T22:47:52.209452+0000","last_unstale":"2024-09-20T22:50:13.237538+0000","last_undegraded":"2024-09-20T22:50:13.237538+0000","last_fullsized":"2024-09-20T22:50:13.237538+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'360","last_scrub_stamp":"2024-09-20T22:49:25.149145+0000","last_deep_scrub":"174'60","last_deep_scrub_stamp":"2024-09-20T22:44:18.301776+0000","last_clean_scrub_stamp":"2024-09-20T22:49:25.149145+0000","objects_scrubbed":11,"log_size":408,"log_dups_size":0,"ondisk_log_size":408,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:26.290244+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.0063337599999999999,"stat_sum":{"num_bytes":27613132,"num_objects":13,"num_object_clones":11,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":1,"num_read":166,"num_read_kb":173052,"num_write":109,"num_write_kb":43661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":23110741,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"d9","length":"1"},{"start":"dd","length":"2"},{"start":"e1","length":"1"}]},{"pgid":"1.8","version":"77'128","reported_seq":1236,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.527975+0000","last_change":"2024-09-20T22:49:56.715035+0000","last_active":"2024-09-20T22:50:13.527975+0000","last_peered":"2024-09-20T22:50:13.527975+0000","last_clean":"2024-09-20T22:50:13.527975+0000","last_became_active":"2024-09-20T22:42:52.405594+0000","last_became_peered":"2024-09-20T22:42:52.405594+0000","last_unstale":"2024-09-20T22:50:13.527975+0000","last_undegraded":"2024-09-20T22:50:13.527975+0000","last_fullsized":"2024-09-20T22:50:13.527975+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:49:56.714919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:49:56.714919+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:51:24.489539+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"74'17","reported_seq":293,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.233851+0000","last_change":"2024-09-20T22:49:01.949542+0000","last_active":"2024-09-20T22:50:13.233851+0000","last_peered":"2024-09-20T22:50:13.233851+0000","last_clean":"2024-09-20T22:50:13.233851+0000","last_became_active":"2024-09-20T22:48:05.247971+0000","last_became_peered":"2024-09-20T22:48:05.247971+0000","last_unstale":"2024-09-20T22:50:13.233851+0000","last_undegraded":"2024-09-20T22:50:13.233851+0000","last_fullsized":"2024-09-20T22:50:13.233851+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'17","last_scrub_stamp":"2024-09-20T22:49:01.949451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:49:01.949451+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":28,"num_read_kb":18,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":516,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.237207+0000","last_change":"2024-09-20T22:49:32.138822+0000","last_active":"2024-09-20T22:50:13.237207+0000","last_peered":"2024-09-20T22:50:13.237207+0000","last_clean":"2024-09-20T22:50:13.237207+0000","last_became_active":"2024-09-20T22:48:05.240481+0000","last_became_peered":"2024-09-20T22:48:05.240481+0000","last_unstale":"2024-09-20T22:50:13.237207+0000","last_undegraded":"2024-09-20T22:50:13.237207+0000","last_fullsized":"2024-09-20T22:50:13.237207+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:32.138751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:35.492562+0000","last_clean_scrub_stamp":"2024-09-20T22:49:32.138751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:44.032030+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":985,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.528076+0000","last_change":"2024-09-20T22:49:09.188448+0000","last_active":"2024-09-20T22:50:13.528076+0000","last_peered":"2024-09-20T22:50:13.528076+0000","last_clean":"2024-09-20T22:50:13.528076+0000","last_became_active":"2024-09-20T22:48:05.250175+0000","last_became_peered":"2024-09-20T22:48:05.250175+0000","last_unstale":"2024-09-20T22:50:13.528076+0000","last_undegraded":"2024-09-20T22:50:13.528076+0000","last_fullsized":"2024-09-20T22:50:13.528076+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:09.188362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:49:09.188362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:36.362440+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":983,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.236528+0000","last_change":"2024-09-20T22:49:26.152417+0000","last_active":"2024-09-20T22:50:13.236528+0000","last_peered":"2024-09-20T22:50:13.236528+0000","last_clean":"2024-09-20T22:50:13.236528+0000","last_became_active":"2024-09-20T22:48:05.239006+0000","last_became_peered":"2024-09-20T22:48:05.239006+0000","last_unstale":"2024-09-20T22:50:13.236528+0000","last_undegraded":"2024-09-20T22:50:13.236528+0000","last_fullsized":"2024-09-20T22:50:13.236528+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:26.152316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:49:26.152316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:40.036383+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"532'499","reported_seq":2710,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.235215+0000","last_change":"2024-09-20T22:50:11.292082+0000","last_active":"2024-09-20T22:50:13.235215+0000","last_peered":"2024-09-20T22:50:13.235215+0000","last_clean":"2024-09-20T22:50:13.235215+0000","last_became_active":"2024-09-20T22:47:52.216198+0000","last_became_peered":"2024-09-20T22:47:52.216198+0000","last_unstale":"2024-09-20T22:50:13.235215+0000","last_undegraded":"2024-09-20T22:50:13.235215+0000","last_fullsized":"2024-09-20T22:50:13.235215+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"481'456","last_scrub_stamp":"2024-09-20T22:49:22.637524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:49:22.637524+0000","objects_scrubbed":9,"log_size":499,"log_dups_size":0,"ondisk_log_size":499,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:38.238133+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.007154082,"stat_sum":{"num_bytes":25104069,"num_objects":12,"num_object_clones":9,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":2,"num_read":233,"num_read_kb":231558,"num_write":146,"num_write_kb":49226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":53,"num_bytes_recovered":14983556,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"d9","length":"1"},{"start":"dd","length":"2"},{"start":"e1","length":"1"}]},{"pgid":"1.9","version":"77'128","reported_seq":1276,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.252945+0000","last_change":"2024-09-20T22:49:52.301456+0000","last_active":"2024-09-20T22:50:13.252945+0000","last_peered":"2024-09-20T22:50:13.252945+0000","last_clean":"2024-09-20T22:50:13.252945+0000","last_became_active":"2024-09-20T22:42:52.400353+0000","last_became_peered":"2024-09-20T22:42:52.400353+0000","last_unstale":"2024-09-20T22:50:13.252945+0000","last_undegraded":"2024-09-20T22:50:13.252945+0000","last_fullsized":"2024-09-20T22:50:13.252945+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:49:52.301348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:49:52.301348+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:51:11.793390+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"74'10","reported_seq":279,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.233659+0000","last_change":"2024-09-20T22:48:05.262827+0000","last_active":"2024-09-20T22:50:13.233659+0000","last_peered":"2024-09-20T22:50:13.233659+0000","last_clean":"2024-09-20T22:50:13.233659+0000","last_became_active":"2024-09-20T22:48:05.242811+0000","last_became_peered":"2024-09-20T22:48:05.242811+0000","last_unstale":"2024-09-20T22:50:13.233659+0000","last_undegraded":"2024-09-20T22:50:13.233659+0000","last_fullsized":"2024-09-20T22:50:13.233659+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-09-20T22:47:21.092309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:47:21.092309+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":992,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.251127+0000","last_change":"2024-09-20T22:50:13.205821+0000","last_active":"2024-09-20T22:50:13.251127+0000","last_peered":"2024-09-20T22:50:13.251127+0000","last_clean":"2024-09-20T22:50:13.251127+0000","last_became_active":"2024-09-20T22:48:05.233999+0000","last_became_peered":"2024-09-20T22:48:05.233999+0000","last_unstale":"2024-09-20T22:50:13.251127+0000","last_undegraded":"2024-09-20T22:50:13.251127+0000","last_fullsized":"2024-09-20T22:50:13.251127+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:50:13.205743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:50:13.205743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:51:41.953774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"261'9","reported_seq":1088,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.236239+0000","last_change":"2024-09-20T22:49:57.205107+0000","last_active":"2024-09-20T22:50:13.236239+0000","last_peered":"2024-09-20T22:50:13.236239+0000","last_clean":"2024-09-20T22:50:13.236239+0000","last_became_active":"2024-09-20T22:41:42.185863+0000","last_became_peered":"2024-09-20T22:41:42.185863+0000","last_unstale":"2024-09-20T22:50:13.236239+0000","last_undegraded":"2024-09-20T22:50:13.236239+0000","last_fullsized":"2024-09-20T22:50:13.236239+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'9","last_scrub_stamp":"2024-09-20T22:49:57.204987+0000","last_deep_scrub":"261'9","last_deep_scrub_stamp":"2024-09-20T22:46:50.974735+0000","last_clean_scrub_stamp":"2024-09-20T22:49:57.204987+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:51:09.810048+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":275,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.234114+0000","last_change":"2024-09-20T22:49:54.058299+0000","last_active":"2024-09-20T22:50:13.234114+0000","last_peered":"2024-09-20T22:50:13.234114+0000","last_clean":"2024-09-20T22:50:13.234114+0000","last_became_active":"2024-09-20T22:48:05.250284+0000","last_became_peered":"2024-09-20T22:48:05.250284+0000","last_unstale":"2024-09-20T22:50:13.234114+0000","last_undegraded":"2024-09-20T22:50:13.234114+0000","last_fullsized":"2024-09-20T22:50:13.234114+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:54.058184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:49:54.058184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:51:00.325103+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.8","version":"533'443","reported_seq":2527,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.236907+0000","last_change":"2024-09-20T22:50:11.286068+0000","last_active":"2024-09-20T22:50:13.236907+0000","last_peered":"2024-09-20T22:50:13.236907+0000","last_clean":"2024-09-20T22:50:13.236907+0000","last_became_active":"2024-09-20T22:48:31.224359+0000","last_became_peered":"2024-09-20T22:48:31.224359+0000","last_unstale":"2024-09-20T22:50:13.236907+0000","last_undegraded":"2024-09-20T22:50:13.236907+0000","last_fullsized":"2024-09-20T22:50:13.236907+0000","mapping_epoch":407,"log_start":"426'268","ondisk_log_start":"426'268","created":94,"last_epoch_clean":431,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'319","last_scrub_stamp":"2024-09-20T22:48:59.219393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:48:59.219393+0000","objects_scrubbed":8,"log_size":175,"log_dups_size":445,"ondisk_log_size":175,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:23.085464+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0072628140000000002,"stat_sum":{"num_bytes":34739607,"num_objects":17,"num_object_clones":14,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":1,"num_read":198,"num_read_kb":189772,"num_write":159,"num_write_kb":53832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":26027409,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"d9","length":"1"},{"start":"dd","length":"2"},{"start":"e1","length":"1"}]},{"pgid":"4.b","version":"74'9","reported_seq":334,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.233318+0000","last_change":"2024-09-20T22:48:05.256987+0000","last_active":"2024-09-20T22:50:13.233318+0000","last_peered":"2024-09-20T22:50:13.233318+0000","last_clean":"2024-09-20T22:50:13.233318+0000","last_became_active":"2024-09-20T22:48:05.233267+0000","last_became_peered":"2024-09-20T22:48:05.233267+0000","last_unstale":"2024-09-20T22:50:13.233318+0000","last_undegraded":"2024-09-20T22:50:13.233318+0000","last_fullsized":"2024-09-20T22:50:13.233318+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-09-20T22:48:03.584242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:48:03.584242+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:13.855721+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":969,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.528549+0000","last_change":"2024-09-20T22:49:05.638528+0000","last_active":"2024-09-20T22:50:13.528549+0000","last_peered":"2024-09-20T22:50:13.528549+0000","last_clean":"2024-09-20T22:50:13.528549+0000","last_became_active":"2024-09-20T22:41:38.155953+0000","last_became_peered":"2024-09-20T22:41:38.155953+0000","last_unstale":"2024-09-20T22:50:13.528549+0000","last_undegraded":"2024-09-20T22:50:13.528549+0000","last_fullsized":"2024-09-20T22:50:13.528549+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:05.638457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:49:05.638457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:25.666047+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":991,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.237397+0000","last_change":"2024-09-20T22:49:02.242848+0000","last_active":"2024-09-20T22:50:13.237397+0000","last_peered":"2024-09-20T22:50:13.237397+0000","last_clean":"2024-09-20T22:50:13.237397+0000","last_became_active":"2024-09-20T22:48:05.240613+0000","last_became_peered":"2024-09-20T22:48:05.240613+0000","last_unstale":"2024-09-20T22:50:13.237397+0000","last_undegraded":"2024-09-20T22:50:13.237397+0000","last_fullsized":"2024-09-20T22:50:13.237397+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:02.242757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:49:02.242757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:24.494551+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":308,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.233455+0000","last_change":"2024-09-20T22:49:31.666061+0000","last_active":"2024-09-20T22:50:13.233455+0000","last_peered":"2024-09-20T22:50:13.233455+0000","last_clean":"2024-09-20T22:50:13.233455+0000","last_became_active":"2024-09-20T22:47:49.219486+0000","last_became_peered":"2024-09-20T22:47:49.219486+0000","last_unstale":"2024-09-20T22:50:13.233455+0000","last_undegraded":"2024-09-20T22:50:13.233455+0000","last_fullsized":"2024-09-20T22:50:13.233455+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:31.665954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:49:31.665954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:38.433613+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"531'619","reported_seq":2864,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.234114+0000","last_change":"2024-09-20T22:50:11.296392+0000","last_active":"2024-09-20T22:50:13.234114+0000","last_peered":"2024-09-20T22:50:13.234114+0000","last_clean":"2024-09-20T22:50:13.234114+0000","last_became_active":"2024-09-20T22:48:27.272202+0000","last_became_peered":"2024-09-20T22:48:27.272202+0000","last_unstale":"2024-09-20T22:50:13.234114+0000","last_undegraded":"2024-09-20T22:50:13.234114+0000","last_fullsized":"2024-09-20T22:50:13.234114+0000","mapping_epoch":407,"log_start":"423'280","ondisk_log_start":"423'280","created":94,"last_epoch_clean":427,"parent":"0.0","parent_split_bits":0,"last_scrub":"423'271","last_scrub_stamp":"2024-09-20T22:48:23.382092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:48:23.382092+0000","objects_scrubbed":8,"log_size":339,"log_dups_size":649,"ondisk_log_size":339,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.011940723,"stat_sum":{"num_bytes":59196432,"num_objects":26,"num_object_clones":21,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":1,"num_read":311,"num_read_kb":319052,"num_write":237,"num_write_kb":88911,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":65,"num_bytes_recovered":38710860,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"d9","length":"1"},{"start":"dd","length":"2"},{"start":"e1","length":"1"}]},{"pgid":"4.a","version":"113'28","reported_seq":1012,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.248545+0000","last_change":"2024-09-20T22:49:03.201199+0000","last_active":"2024-09-20T22:50:13.248545+0000","last_peered":"2024-09-20T22:50:13.248545+0000","last_clean":"2024-09-20T22:50:13.248545+0000","last_became_active":"2024-09-20T22:41:40.252011+0000","last_became_peered":"2024-09-20T22:41:40.252011+0000","last_unstale":"2024-09-20T22:50:13.248545+0000","last_undegraded":"2024-09-20T22:50:13.248545+0000","last_fullsized":"2024-09-20T22:50:13.248545+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'28","last_scrub_stamp":"2024-09-20T22:49:03.201108+0000","last_deep_scrub":"113'28","last_deep_scrub_stamp":"2024-09-20T22:43:12.725900+0000","last_clean_scrub_stamp":"2024-09-20T22:49:03.201108+0000","objects_scrubbed":9,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:31.362019+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":968,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.255537+0000","last_change":"2024-09-20T22:49:12.324218+0000","last_active":"2024-09-20T22:50:13.255537+0000","last_peered":"2024-09-20T22:50:13.255537+0000","last_clean":"2024-09-20T22:50:13.255537+0000","last_became_active":"2024-09-20T22:41:38.161524+0000","last_became_peered":"2024-09-20T22:41:38.161524+0000","last_unstale":"2024-09-20T22:50:13.255537+0000","last_undegraded":"2024-09-20T22:50:13.255537+0000","last_fullsized":"2024-09-20T22:50:13.255537+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:12.324148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:49:12.324148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:36.961267+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":979,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.236895+0000","last_change":"2024-09-20T22:49:04.244778+0000","last_active":"2024-09-20T22:50:13.236895+0000","last_peered":"2024-09-20T22:50:13.236895+0000","last_clean":"2024-09-20T22:50:13.236895+0000","last_became_active":"2024-09-20T22:47:49.223220+0000","last_became_peered":"2024-09-20T22:47:49.223220+0000","last_unstale":"2024-09-20T22:50:13.236895+0000","last_undegraded":"2024-09-20T22:50:13.236895+0000","last_fullsized":"2024-09-20T22:50:13.236895+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:04.244659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:49:04.244659+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":964,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.253445+0000","last_change":"2024-09-20T22:49:30.310175+0000","last_active":"2024-09-20T22:50:13.253445+0000","last_peered":"2024-09-20T22:50:13.253445+0000","last_clean":"2024-09-20T22:50:13.253445+0000","last_became_active":"2024-09-20T22:41:44.199395+0000","last_became_peered":"2024-09-20T22:41:44.199395+0000","last_unstale":"2024-09-20T22:50:13.253445+0000","last_undegraded":"2024-09-20T22:50:13.253445+0000","last_fullsized":"2024-09-20T22:50:13.253445+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:30.310069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:47:18.329209+0000","last_clean_scrub_stamp":"2024-09-20T22:49:30.310069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:44.675926+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.a","version":"531'515","reported_seq":2693,"reported_epoch":533,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T22:50:13.254914+0000","last_change":"2024-09-20T22:50:12.254068+0000","last_active":"2024-09-20T22:50:13.254914+0000","last_peered":"2024-09-20T22:50:13.254914+0000","last_clean":"2024-09-20T22:50:13.254914+0000","last_became_active":"2024-09-20T22:42:52.405085+0000","last_became_peered":"2024-09-20T22:42:52.405085+0000","last_unstale":"2024-09-20T22:50:13.254914+0000","last_undegraded":"2024-09-20T22:50:13.254914+0000","last_fullsized":"2024-09-20T22:50:13.254914+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"505'466","last_scrub_stamp":"2024-09-20T22:49:47.088165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:49:47.088165+0000","objects_scrubbed":13,"log_size":515,"log_dups_size":0,"ondisk_log_size":515,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:52.309508+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.95982113800000002,"stat_sum":{"num_bytes":35889860,"num_objects":16,"num_object_clones":13,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":1,"num_read":295,"num_read_kb":332020,"num_write":183,"num_write_kb":69363,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"dd","length":"2"},{"start":"e1","length":"1"}]},{"pgid":"4.9","version":"74'12","reported_seq":279,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.232760+0000","last_change":"2024-09-20T22:48:05.284523+0000","last_active":"2024-09-20T22:50:13.232760+0000","last_peered":"2024-09-20T22:50:13.232760+0000","last_clean":"2024-09-20T22:50:13.232760+0000","last_became_active":"2024-09-20T22:48:05.249438+0000","last_became_peered":"2024-09-20T22:48:05.249438+0000","last_unstale":"2024-09-20T22:50:13.232760+0000","last_undegraded":"2024-09-20T22:50:13.232760+0000","last_fullsized":"2024-09-20T22:50:13.232760+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-09-20T22:47:58.421473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:47:58.421473+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":21,"num_read_kb":12,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":1000,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.252060+0000","last_change":"2024-09-20T22:49:24.300511+0000","last_active":"2024-09-20T22:50:13.252060+0000","last_peered":"2024-09-20T22:50:13.252060+0000","last_clean":"2024-09-20T22:50:13.252060+0000","last_became_active":"2024-09-20T22:48:05.240225+0000","last_became_peered":"2024-09-20T22:48:05.240225+0000","last_unstale":"2024-09-20T22:50:13.252060+0000","last_undegraded":"2024-09-20T22:50:13.252060+0000","last_fullsized":"2024-09-20T22:50:13.252060+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:24.300441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:46:52.465940+0000","last_clean_scrub_stamp":"2024-09-20T22:49:24.300441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:46.323016+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":987,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.236954+0000","last_change":"2024-09-20T22:49:54.216094+0000","last_active":"2024-09-20T22:50:13.236954+0000","last_peered":"2024-09-20T22:50:13.236954+0000","last_clean":"2024-09-20T22:50:13.236954+0000","last_became_active":"2024-09-20T22:47:49.222088+0000","last_became_peered":"2024-09-20T22:47:49.222088+0000","last_unstale":"2024-09-20T22:50:13.236954+0000","last_undegraded":"2024-09-20T22:50:13.236954+0000","last_fullsized":"2024-09-20T22:50:13.236954+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:54.216022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:46:01.828080+0000","last_clean_scrub_stamp":"2024-09-20T22:49:54.216022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:56.397469+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":948,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.252869+0000","last_change":"2024-09-20T22:47:38.019602+0000","last_active":"2024-09-20T22:50:13.252869+0000","last_peered":"2024-09-20T22:50:13.252869+0000","last_clean":"2024-09-20T22:50:13.252869+0000","last_became_active":"2024-09-20T22:41:44.192609+0000","last_became_peered":"2024-09-20T22:41:44.192609+0000","last_unstale":"2024-09-20T22:50:13.252869+0000","last_undegraded":"2024-09-20T22:50:13.252869+0000","last_fullsized":"2024-09-20T22:50:13.252869+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:47:38.019533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:55.170379+0000","last_clean_scrub_stamp":"2024-09-20T22:47:38.019533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"531'773","reported_seq":2953,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.529112+0000","last_change":"2024-09-20T22:50:13.296304+0000","last_active":"2024-09-20T22:50:13.529112+0000","last_peered":"2024-09-20T22:50:13.529112+0000","last_clean":"2024-09-20T22:50:13.529112+0000","last_became_active":"2024-09-20T22:42:52.409642+0000","last_became_peered":"2024-09-20T22:42:52.409642+0000","last_unstale":"2024-09-20T22:50:13.529112+0000","last_undegraded":"2024-09-20T22:50:13.529112+0000","last_fullsized":"2024-09-20T22:50:13.529112+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"194'116","last_scrub_stamp":"2024-09-20T22:44:37.097674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:44:37.097674+0000","objects_scrubbed":3,"log_size":773,"log_dups_size":0,"ondisk_log_size":773,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.99785855400000001,"stat_sum":{"num_bytes":53661122,"num_objects":25,"num_object_clones":21,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":2,"num_read":405,"num_read_kb":450576,"num_write":284,"num_write_kb":90656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"d9","length":"1"},{"start":"dd","length":"2"},{"start":"e1","length":"1"}]},{"pgid":"4.8","version":"74'15","reported_seq":990,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.528214+0000","last_change":"2024-09-20T22:49:47.687063+0000","last_active":"2024-09-20T22:50:13.528214+0000","last_peered":"2024-09-20T22:50:13.528214+0000","last_clean":"2024-09-20T22:50:13.528214+0000","last_became_active":"2024-09-20T22:41:40.252113+0000","last_became_peered":"2024-09-20T22:41:40.252113+0000","last_unstale":"2024-09-20T22:50:13.528214+0000","last_undegraded":"2024-09-20T22:50:13.528214+0000","last_fullsized":"2024-09-20T22:50:13.528214+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-09-20T22:49:47.686984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:49:47.686984+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:51:06.196255+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'2","reported_seq":1028,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.252226+0000","last_change":"2024-09-20T22:49:22.379093+0000","last_active":"2024-09-20T22:50:13.252226+0000","last_peered":"2024-09-20T22:50:13.252226+0000","last_clean":"2024-09-20T22:50:13.252226+0000","last_became_active":"2024-09-20T22:48:05.238098+0000","last_became_peered":"2024-09-20T22:48:05.238098+0000","last_unstale":"2024-09-20T22:50:13.252226+0000","last_undegraded":"2024-09-20T22:50:13.252226+0000","last_fullsized":"2024-09-20T22:50:13.252226+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-09-20T22:49:22.379012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:49:22.379012+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:31.674675+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"408'10","reported_seq":1100,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.251127+0000","last_change":"2024-09-20T22:49:47.363588+0000","last_active":"2024-09-20T22:50:13.251127+0000","last_peered":"2024-09-20T22:50:13.251127+0000","last_clean":"2024-09-20T22:50:13.251127+0000","last_became_active":"2024-09-20T22:48:05.239344+0000","last_became_peered":"2024-09-20T22:48:05.239344+0000","last_unstale":"2024-09-20T22:50:13.251127+0000","last_undegraded":"2024-09-20T22:50:13.251127+0000","last_fullsized":"2024-09-20T22:50:13.251127+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"408'10","last_scrub_stamp":"2024-09-20T22:49:47.363515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:49:47.363515+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:51:01.473177+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":969,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.527691+0000","last_change":"2024-09-20T22:49:50.693099+0000","last_active":"2024-09-20T22:50:13.527691+0000","last_peered":"2024-09-20T22:50:13.527691+0000","last_clean":"2024-09-20T22:50:13.527691+0000","last_became_active":"2024-09-20T22:47:49.227569+0000","last_became_peered":"2024-09-20T22:47:49.227569+0000","last_unstale":"2024-09-20T22:50:13.527691+0000","last_undegraded":"2024-09-20T22:50:13.527691+0000","last_fullsized":"2024-09-20T22:50:13.527691+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:50.693003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:49:50.693003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:51:08.138542+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":988,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.248284+0000","last_change":"2024-09-20T22:49:02.203750+0000","last_active":"2024-09-20T22:50:13.248284+0000","last_peered":"2024-09-20T22:50:13.248284+0000","last_clean":"2024-09-20T22:50:13.248284+0000","last_became_active":"2024-09-20T22:47:49.213292+0000","last_became_peered":"2024-09-20T22:47:49.213292+0000","last_unstale":"2024-09-20T22:50:13.248284+0000","last_undegraded":"2024-09-20T22:50:13.248284+0000","last_fullsized":"2024-09-20T22:50:13.248284+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:02.203661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:49:02.203661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:30.443605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"74'6","reported_seq":987,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.253085+0000","last_change":"2024-09-20T22:49:48.113810+0000","last_active":"2024-09-20T22:50:13.253085+0000","last_peered":"2024-09-20T22:50:13.253085+0000","last_clean":"2024-09-20T22:50:13.253085+0000","last_became_active":"2024-09-20T22:43:09.116887+0000","last_became_peered":"2024-09-20T22:43:09.116887+0000","last_unstale":"2024-09-20T22:50:13.253085+0000","last_undegraded":"2024-09-20T22:50:13.253085+0000","last_fullsized":"2024-09-20T22:50:13.253085+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'6","last_scrub_stamp":"2024-09-20T22:49:48.113740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:49:48.113740+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:51:07.964989+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":961,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.528605+0000","last_change":"2024-09-20T22:48:35.550867+0000","last_active":"2024-09-20T22:50:13.528605+0000","last_peered":"2024-09-20T22:50:13.528605+0000","last_clean":"2024-09-20T22:50:13.528605+0000","last_became_active":"2024-09-20T22:41:42.184930+0000","last_became_peered":"2024-09-20T22:41:42.184930+0000","last_unstale":"2024-09-20T22:50:13.528605+0000","last_undegraded":"2024-09-20T22:50:13.528605+0000","last_fullsized":"2024-09-20T22:50:13.528605+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:48:35.550781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:09.705632+0000","last_clean_scrub_stamp":"2024-09-20T22:48:35.550781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"74'1","reported_seq":977,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.252078+0000","last_change":"2024-09-20T22:48:40.309005+0000","last_active":"2024-09-20T22:50:13.252078+0000","last_peered":"2024-09-20T22:50:13.252078+0000","last_clean":"2024-09-20T22:50:13.252078+0000","last_became_active":"2024-09-20T22:48:05.237882+0000","last_became_peered":"2024-09-20T22:48:05.237882+0000","last_unstale":"2024-09-20T22:50:13.252078+0000","last_undegraded":"2024-09-20T22:50:13.252078+0000","last_fullsized":"2024-09-20T22:50:13.252078+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-09-20T22:48:40.308879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:48:40.308879+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"74'9","reported_seq":318,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.235248+0000","last_change":"2024-09-20T22:49:11.739149+0000","last_active":"2024-09-20T22:50:13.235248+0000","last_peered":"2024-09-20T22:50:13.235248+0000","last_clean":"2024-09-20T22:50:13.235248+0000","last_became_active":"2024-09-20T22:47:49.231315+0000","last_became_peered":"2024-09-20T22:47:49.231315+0000","last_unstale":"2024-09-20T22:50:13.235248+0000","last_undegraded":"2024-09-20T22:50:13.235248+0000","last_fullsized":"2024-09-20T22:50:13.235248+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-09-20T22:49:11.739076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:49:11.739076+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:17.900700+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":992,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.251315+0000","last_change":"2024-09-20T22:49:25.302834+0000","last_active":"2024-09-20T22:50:13.251315+0000","last_peered":"2024-09-20T22:50:13.251315+0000","last_clean":"2024-09-20T22:50:13.251315+0000","last_became_active":"2024-09-20T22:48:05.238801+0000","last_became_peered":"2024-09-20T22:48:05.238801+0000","last_unstale":"2024-09-20T22:50:13.251315+0000","last_undegraded":"2024-09-20T22:50:13.251315+0000","last_fullsized":"2024-09-20T22:50:13.251315+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:25.302756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:49:25.302756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:30.192311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":952,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.252849+0000","last_change":"2024-09-20T22:46:48.012442+0000","last_active":"2024-09-20T22:50:13.252849+0000","last_peered":"2024-09-20T22:50:13.252849+0000","last_clean":"2024-09-20T22:50:13.252849+0000","last_became_active":"2024-09-20T22:41:42.187706+0000","last_became_peered":"2024-09-20T22:41:42.187706+0000","last_unstale":"2024-09-20T22:50:13.252849+0000","last_undegraded":"2024-09-20T22:50:13.252849+0000","last_fullsized":"2024-09-20T22:50:13.252849+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:46:48.012372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:46:48.012372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":987,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.237059+0000","last_change":"2024-09-20T22:48:40.247992+0000","last_active":"2024-09-20T22:50:13.237059+0000","last_peered":"2024-09-20T22:50:13.237059+0000","last_clean":"2024-09-20T22:50:13.237059+0000","last_became_active":"2024-09-20T22:48:05.239710+0000","last_became_peered":"2024-09-20T22:48:05.239710+0000","last_unstale":"2024-09-20T22:50:13.237059+0000","last_undegraded":"2024-09-20T22:50:13.237059+0000","last_fullsized":"2024-09-20T22:50:13.237059+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:48:40.247889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:48:40.247889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"74'9","reported_seq":981,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.528659+0000","last_change":"2024-09-20T22:49:09.704604+0000","last_active":"2024-09-20T22:50:13.528659+0000","last_peered":"2024-09-20T22:50:13.528659+0000","last_clean":"2024-09-20T22:50:13.528659+0000","last_became_active":"2024-09-20T22:41:40.252486+0000","last_became_peered":"2024-09-20T22:41:40.252486+0000","last_unstale":"2024-09-20T22:50:13.528659+0000","last_undegraded":"2024-09-20T22:50:13.528659+0000","last_fullsized":"2024-09-20T22:50:13.528659+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-09-20T22:49:09.704521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:49:09.704521+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:15.287583+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":308,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.232420+0000","last_change":"2024-09-20T22:48:19.379265+0000","last_active":"2024-09-20T22:50:13.232420+0000","last_peered":"2024-09-20T22:50:13.232420+0000","last_clean":"2024-09-20T22:50:13.232420+0000","last_became_active":"2024-09-20T22:47:49.229936+0000","last_became_peered":"2024-09-20T22:47:49.229936+0000","last_unstale":"2024-09-20T22:50:13.232420+0000","last_undegraded":"2024-09-20T22:50:13.232420+0000","last_fullsized":"2024-09-20T22:50:13.232420+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:48:19.379196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:48:19.379196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:13.855785+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"74'8","reported_seq":1064,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.255293+0000","last_change":"2024-09-20T22:47:56.150641+0000","last_active":"2024-09-20T22:50:13.255293+0000","last_peered":"2024-09-20T22:50:13.255293+0000","last_clean":"2024-09-20T22:50:13.255293+0000","last_became_active":"2024-09-20T22:41:42.186352+0000","last_became_peered":"2024-09-20T22:41:42.186352+0000","last_unstale":"2024-09-20T22:50:13.255293+0000","last_undegraded":"2024-09-20T22:50:13.255293+0000","last_fullsized":"2024-09-20T22:50:13.255293+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-09-20T22:47:56.150562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:47:56.150562+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":267,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.234059+0000","last_change":"2024-09-20T22:48:05.250192+0000","last_active":"2024-09-20T22:50:13.234059+0000","last_peered":"2024-09-20T22:50:13.234059+0000","last_clean":"2024-09-20T22:50:13.234059+0000","last_became_active":"2024-09-20T22:48:05.249656+0000","last_became_peered":"2024-09-20T22:48:05.249656+0000","last_unstale":"2024-09-20T22:50:13.234059+0000","last_undegraded":"2024-09-20T22:50:13.234059+0000","last_fullsized":"2024-09-20T22:50:13.234059+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:47:19.828541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:47:19.828541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"74'10","reported_seq":971,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.254977+0000","last_change":"2024-09-20T22:46:24.269736+0000","last_active":"2024-09-20T22:50:13.254977+0000","last_peered":"2024-09-20T22:50:13.254977+0000","last_clean":"2024-09-20T22:50:13.254977+0000","last_became_active":"2024-09-20T22:41:40.253614+0000","last_became_peered":"2024-09-20T22:41:40.253614+0000","last_unstale":"2024-09-20T22:50:13.254977+0000","last_undegraded":"2024-09-20T22:50:13.254977+0000","last_fullsized":"2024-09-20T22:50:13.254977+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-09-20T22:46:24.269655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:46:24.269655+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":988,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.251667+0000","last_change":"2024-09-20T22:49:54.381836+0000","last_active":"2024-09-20T22:50:13.251667+0000","last_peered":"2024-09-20T22:50:13.251667+0000","last_clean":"2024-09-20T22:50:13.251667+0000","last_became_active":"2024-09-20T22:48:05.240231+0000","last_became_peered":"2024-09-20T22:48:05.240231+0000","last_unstale":"2024-09-20T22:50:13.251667+0000","last_undegraded":"2024-09-20T22:50:13.251667+0000","last_fullsized":"2024-09-20T22:50:13.251667+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:54.381721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:49:54.381721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:51:04.253551+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"389'10","reported_seq":1104,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.528521+0000","last_change":"2024-09-20T22:49:51.715574+0000","last_active":"2024-09-20T22:50:13.528521+0000","last_peered":"2024-09-20T22:50:13.528521+0000","last_clean":"2024-09-20T22:50:13.528521+0000","last_became_active":"2024-09-20T22:47:49.227113+0000","last_became_peered":"2024-09-20T22:47:49.227113+0000","last_unstale":"2024-09-20T22:50:13.528521+0000","last_undegraded":"2024-09-20T22:50:13.528521+0000","last_fullsized":"2024-09-20T22:50:13.528521+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'10","last_scrub_stamp":"2024-09-20T22:49:51.715471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:49:51.715471+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:51:04.561910+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":300,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.234462+0000","last_change":"2024-09-20T22:47:49.232560+0000","last_active":"2024-09-20T22:50:13.234462+0000","last_peered":"2024-09-20T22:50:13.234462+0000","last_clean":"2024-09-20T22:50:13.234462+0000","last_became_active":"2024-09-20T22:47:49.230674+0000","last_became_peered":"2024-09-20T22:47:49.230674+0000","last_unstale":"2024-09-20T22:50:13.234462+0000","last_undegraded":"2024-09-20T22:50:13.234462+0000","last_fullsized":"2024-09-20T22:50:13.234462+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:47:41.411716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:47:41.411716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:13.855690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"74'11","reported_seq":287,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.233465+0000","last_change":"2024-09-20T22:48:44.023878+0000","last_active":"2024-09-20T22:50:13.233465+0000","last_peered":"2024-09-20T22:50:13.233465+0000","last_clean":"2024-09-20T22:50:13.233465+0000","last_became_active":"2024-09-20T22:48:05.249510+0000","last_became_peered":"2024-09-20T22:48:05.249510+0000","last_unstale":"2024-09-20T22:50:13.233465+0000","last_undegraded":"2024-09-20T22:50:13.233465+0000","last_fullsized":"2024-09-20T22:50:13.233465+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-09-20T22:48:44.023781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:48:44.023781+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":275,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.233560+0000","last_change":"2024-09-20T22:49:11.015881+0000","last_active":"2024-09-20T22:50:13.233560+0000","last_peered":"2024-09-20T22:50:13.233560+0000","last_clean":"2024-09-20T22:50:13.233560+0000","last_became_active":"2024-09-20T22:48:05.252026+0000","last_became_peered":"2024-09-20T22:48:05.252026+0000","last_unstale":"2024-09-20T22:50:13.233560+0000","last_undegraded":"2024-09-20T22:50:13.233560+0000","last_fullsized":"2024-09-20T22:50:13.233560+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:11.015797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:49:11.015797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:39.578934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":961,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.529045+0000","last_change":"2024-09-20T22:48:59.241704+0000","last_active":"2024-09-20T22:50:13.529045+0000","last_peered":"2024-09-20T22:50:13.529045+0000","last_clean":"2024-09-20T22:50:13.529045+0000","last_became_active":"2024-09-20T22:41:42.181736+0000","last_became_peered":"2024-09-20T22:41:42.181736+0000","last_unstale":"2024-09-20T22:50:13.529045+0000","last_undegraded":"2024-09-20T22:50:13.529045+0000","last_fullsized":"2024-09-20T22:50:13.529045+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:48:59.241606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:48:59.241606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:13.923480+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":960,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.253084+0000","last_change":"2024-09-20T22:47:49.216479+0000","last_active":"2024-09-20T22:50:13.253084+0000","last_peered":"2024-09-20T22:50:13.253084+0000","last_clean":"2024-09-20T22:50:13.253084+0000","last_became_active":"2024-09-20T22:47:49.216141+0000","last_became_peered":"2024-09-20T22:47:49.216141+0000","last_unstale":"2024-09-20T22:50:13.253084+0000","last_undegraded":"2024-09-20T22:50:13.253084+0000","last_fullsized":"2024-09-20T22:50:13.253084+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:47:36.066480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:47:36.066480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"404'13","reported_seq":993,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.529204+0000","last_change":"2024-09-20T22:49:25.147397+0000","last_active":"2024-09-20T22:50:13.529204+0000","last_peered":"2024-09-20T22:50:13.529204+0000","last_clean":"2024-09-20T22:50:13.529204+0000","last_became_active":"2024-09-20T22:41:40.164796+0000","last_became_peered":"2024-09-20T22:41:40.164796+0000","last_unstale":"2024-09-20T22:50:13.529204+0000","last_undegraded":"2024-09-20T22:50:13.529204+0000","last_fullsized":"2024-09-20T22:50:13.529204+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"404'13","last_scrub_stamp":"2024-09-20T22:49:25.147286+0000","last_deep_scrub":"404'13","last_deep_scrub_stamp":"2024-09-20T22:48:04.276282+0000","last_clean_scrub_stamp":"2024-09-20T22:49:25.147286+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:52.567714+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":1003,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.251966+0000","last_change":"2024-09-20T22:49:56.396435+0000","last_active":"2024-09-20T22:50:13.251966+0000","last_peered":"2024-09-20T22:50:13.251966+0000","last_clean":"2024-09-20T22:50:13.251966+0000","last_became_active":"2024-09-20T22:48:05.238812+0000","last_became_peered":"2024-09-20T22:48:05.238812+0000","last_unstale":"2024-09-20T22:50:13.251966+0000","last_undegraded":"2024-09-20T22:50:13.251966+0000","last_fullsized":"2024-09-20T22:50:13.251966+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:56.396347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:31.138067+0000","last_clean_scrub_stamp":"2024-09-20T22:49:56.396347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:51:17.230220+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":964,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.252946+0000","last_change":"2024-09-20T22:48:38.200896+0000","last_active":"2024-09-20T22:50:13.252946+0000","last_peered":"2024-09-20T22:50:13.252946+0000","last_clean":"2024-09-20T22:50:13.252946+0000","last_became_active":"2024-09-20T22:47:49.218708+0000","last_became_peered":"2024-09-20T22:47:49.218708+0000","last_unstale":"2024-09-20T22:50:13.252946+0000","last_undegraded":"2024-09-20T22:50:13.252946+0000","last_fullsized":"2024-09-20T22:50:13.252946+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:48:38.200797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:48:38.200797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":300,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.232800+0000","last_change":"2024-09-20T22:47:49.225362+0000","last_active":"2024-09-20T22:50:13.232800+0000","last_peered":"2024-09-20T22:50:13.232800+0000","last_clean":"2024-09-20T22:50:13.232800+0000","last_became_active":"2024-09-20T22:47:49.225027+0000","last_became_peered":"2024-09-20T22:47:49.225027+0000","last_unstale":"2024-09-20T22:50:13.232800+0000","last_undegraded":"2024-09-20T22:50:13.232800+0000","last_fullsized":"2024-09-20T22:50:13.232800+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:47:37.309842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:47:37.309842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:13.855704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"470'16","reported_seq":779,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.527857+0000","last_change":"2024-09-20T22:49:10.659945+0000","last_active":"2024-09-20T22:50:13.527857+0000","last_peered":"2024-09-20T22:50:13.527857+0000","last_clean":"2024-09-20T22:50:13.527857+0000","last_became_active":"2024-09-20T22:44:09.170785+0000","last_became_peered":"2024-09-20T22:44:09.170785+0000","last_unstale":"2024-09-20T22:50:13.527857+0000","last_undegraded":"2024-09-20T22:50:13.527857+0000","last_fullsized":"2024-09-20T22:50:13.527857+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"470'16","last_scrub_stamp":"2024-09-20T22:49:10.659868+0000","last_deep_scrub":"470'16","last_deep_scrub_stamp":"2024-09-20T22:49:10.659868+0000","last_clean_scrub_stamp":"2024-09-20T22:49:10.659868+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":969,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.528652+0000","last_change":"2024-09-20T22:48:02.512192+0000","last_active":"2024-09-20T22:50:13.528652+0000","last_peered":"2024-09-20T22:50:13.528652+0000","last_clean":"2024-09-20T22:50:13.528652+0000","last_became_active":"2024-09-20T22:41:38.158701+0000","last_became_peered":"2024-09-20T22:41:38.158701+0000","last_unstale":"2024-09-20T22:50:13.528652+0000","last_undegraded":"2024-09-20T22:50:13.528652+0000","last_fullsized":"2024-09-20T22:50:13.528652+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:48:02.512093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:45:24.765414+0000","last_clean_scrub_stamp":"2024-09-20T22:48:02.512093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":984,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.250998+0000","last_change":"2024-09-20T22:49:11.373441+0000","last_active":"2024-09-20T22:50:13.250998+0000","last_peered":"2024-09-20T22:50:13.250998+0000","last_clean":"2024-09-20T22:50:13.250998+0000","last_became_active":"2024-09-20T22:48:05.238334+0000","last_became_peered":"2024-09-20T22:48:05.238334+0000","last_unstale":"2024-09-20T22:50:13.250998+0000","last_undegraded":"2024-09-20T22:50:13.250998+0000","last_fullsized":"2024-09-20T22:50:13.250998+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:11.373370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:50.174861+0000","last_clean_scrub_stamp":"2024-09-20T22:49:11.373370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:14.536205+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":960,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.253019+0000","last_change":"2024-09-20T22:49:04.034724+0000","last_active":"2024-09-20T22:50:13.253019+0000","last_peered":"2024-09-20T22:50:13.253019+0000","last_clean":"2024-09-20T22:50:13.253019+0000","last_became_active":"2024-09-20T22:47:49.214487+0000","last_became_peered":"2024-09-20T22:47:49.214487+0000","last_unstale":"2024-09-20T22:50:13.253019+0000","last_undegraded":"2024-09-20T22:50:13.253019+0000","last_fullsized":"2024-09-20T22:50:13.253019+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:04.034601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:49:04.034601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:18.245352+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"74'4","reported_seq":962,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.251980+0000","last_change":"2024-09-20T22:48:23.236306+0000","last_active":"2024-09-20T22:50:13.251980+0000","last_peered":"2024-09-20T22:50:13.251980+0000","last_clean":"2024-09-20T22:50:13.251980+0000","last_became_active":"2024-09-20T22:41:40.166123+0000","last_became_peered":"2024-09-20T22:41:40.166123+0000","last_unstale":"2024-09-20T22:50:13.251980+0000","last_undegraded":"2024-09-20T22:50:13.251980+0000","last_fullsized":"2024-09-20T22:50:13.251980+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'4","last_scrub_stamp":"2024-09-20T22:48:23.236227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:48:23.236227+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":308,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.235053+0000","last_change":"2024-09-20T22:48:18.432371+0000","last_active":"2024-09-20T22:50:13.235053+0000","last_peered":"2024-09-20T22:50:13.235053+0000","last_clean":"2024-09-20T22:50:13.235053+0000","last_became_active":"2024-09-20T22:47:49.231964+0000","last_became_peered":"2024-09-20T22:47:49.231964+0000","last_unstale":"2024-09-20T22:50:13.235053+0000","last_undegraded":"2024-09-20T22:50:13.235053+0000","last_fullsized":"2024-09-20T22:50:13.235053+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:48:18.432271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:48:18.432271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:13.855750+0000","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":992,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.247961+0000","last_change":"2024-09-20T22:49:07.125306+0000","last_active":"2024-09-20T22:50:13.247961+0000","last_peered":"2024-09-20T22:50:13.247961+0000","last_clean":"2024-09-20T22:50:13.247961+0000","last_became_active":"2024-09-20T22:48:05.244659+0000","last_became_peered":"2024-09-20T22:48:05.244659+0000","last_unstale":"2024-09-20T22:50:13.247961+0000","last_undegraded":"2024-09-20T22:50:13.247961+0000","last_fullsized":"2024-09-20T22:50:13.247961+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:07.125231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:49:07.125231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:23.332189+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":976,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.251595+0000","last_change":"2024-09-20T22:49:57.405922+0000","last_active":"2024-09-20T22:50:13.251595+0000","last_peered":"2024-09-20T22:50:13.251595+0000","last_clean":"2024-09-20T22:50:13.251595+0000","last_became_active":"2024-09-20T22:48:05.237765+0000","last_became_peered":"2024-09-20T22:48:05.237765+0000","last_unstale":"2024-09-20T22:50:13.251595+0000","last_undegraded":"2024-09-20T22:50:13.251595+0000","last_fullsized":"2024-09-20T22:50:13.251595+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:57.405785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:10.233487+0000","last_clean_scrub_stamp":"2024-09-20T22:49:57.405785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:51:00.718232+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":969,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.528188+0000","last_change":"2024-09-20T22:49:05.205734+0000","last_active":"2024-09-20T22:50:13.528188+0000","last_peered":"2024-09-20T22:50:13.528188+0000","last_clean":"2024-09-20T22:50:13.528188+0000","last_became_active":"2024-09-20T22:48:05.249492+0000","last_became_peered":"2024-09-20T22:48:05.249492+0000","last_unstale":"2024-09-20T22:50:13.528188+0000","last_undegraded":"2024-09-20T22:50:13.528188+0000","last_fullsized":"2024-09-20T22:50:13.528188+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:05.205665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:49:05.205665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:32.772495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":986,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.251711+0000","last_change":"2024-09-20T22:48:36.246135+0000","last_active":"2024-09-20T22:50:13.251711+0000","last_peered":"2024-09-20T22:50:13.251711+0000","last_clean":"2024-09-20T22:50:13.251711+0000","last_became_active":"2024-09-20T22:47:49.217064+0000","last_became_peered":"2024-09-20T22:47:49.217064+0000","last_unstale":"2024-09-20T22:50:13.251711+0000","last_undegraded":"2024-09-20T22:50:13.251711+0000","last_fullsized":"2024-09-20T22:50:13.251711+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:48:36.246069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:48:36.246069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"74'11","reported_seq":995,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.249449+0000","last_change":"2024-09-20T22:49:10.137213+0000","last_active":"2024-09-20T22:50:13.249449+0000","last_peered":"2024-09-20T22:50:13.249449+0000","last_clean":"2024-09-20T22:50:13.249449+0000","last_became_active":"2024-09-20T22:41:40.171256+0000","last_became_peered":"2024-09-20T22:41:40.171256+0000","last_unstale":"2024-09-20T22:50:13.249449+0000","last_undegraded":"2024-09-20T22:50:13.249449+0000","last_fullsized":"2024-09-20T22:50:13.249449+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-09-20T22:49:10.137128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:49:10.137128+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:16.411613+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":308,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.234567+0000","last_change":"2024-09-20T22:49:23.660691+0000","last_active":"2024-09-20T22:50:13.234567+0000","last_peered":"2024-09-20T22:50:13.234567+0000","last_clean":"2024-09-20T22:50:13.234567+0000","last_became_active":"2024-09-20T22:47:49.220971+0000","last_became_peered":"2024-09-20T22:47:49.220971+0000","last_unstale":"2024-09-20T22:50:13.234567+0000","last_undegraded":"2024-09-20T22:50:13.234567+0000","last_fullsized":"2024-09-20T22:50:13.234567+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:23.660625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:51.667341+0000","last_clean_scrub_stamp":"2024-09-20T22:49:23.660625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:38.664121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":567872353,"num_objects":261,"num_object_clones":212,"num_object_copies":783,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":261,"num_whiteouts":19,"num_read":3617,"num_read_kb":3794461,"num_write":2553,"num_write_kb":937523,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":613,"num_bytes_recovered":302968954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1153294336,"data_stored":1703504595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5105,"ondisk_log_size":5105,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":82,"ondisk_log_size":82,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":682,"num_read_kb":439,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":238,"num_bytes_recovered":5126,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":452,"ondisk_log_size":452,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":1034,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":506,"num_read_kb":501,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":384,"ondisk_log_size":384,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070571,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":174232,"kb_used_data":116676,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":93570920,"statfs":{"total":95995035648,"available":95816622080,"internally_reserved":0,"allocated":119476224,"data_stored":168988650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":58915274},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430321,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":210460,"kb_used_data":152904,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":93534692,"statfs":{"total":95995035648,"available":95779524608,"internally_reserved":0,"allocated":156573696,"data_stored":231352110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691958,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":137896,"kb_used_data":106700,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93607256,"statfs":{"total":95995035648,"available":95853830144,"internally_reserved":0,"allocated":109260800,"data_stored":168206833,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986299,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":146276,"kb_used_data":106768,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93598876,"statfs":{"total":95995035648,"available":95845249024,"internally_reserved":0,"allocated":109330432,"data_stored":155859887,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182529,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":277616,"kb_used_data":220060,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":93467536,"statfs":{"total":95995035648,"available":95710756864,"internally_reserved":0,"allocated":225341440,"data_stored":318502608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":58915271},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411462,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":173036,"kb_used_data":119632,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93572116,"statfs":{"total":95995035648,"available":95817846784,"internally_reserved":0,"allocated":122503168,"data_stored":183218956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574987,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":216100,"kb_used_data":162700,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93529052,"statfs":{"total":95995035648,"available":95773749248,"internally_reserved":0,"allocated":166604800,"data_stored":241736252,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738512,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":227808,"kb_used_data":174416,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93517344,"statfs":{"total":95995035648,"available":95761760256,"internally_reserved":0,"allocated":178601984,"data_stored":256412893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":174297088,"data_stored":253812881,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":162271232,"data_stored":239135905,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":117940224,"data_stored":180349247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":221343744,"data_stored":316214317,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":105033728,"data_stored":153267850,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":104374272,"data_stored":165015785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":153198592,"data_stored":229654955,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":114835456,"data_stored":166053655,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-20T22:50:17.401 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-09-20T22:50:17.402 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-20T22:50:17.402 INFO:tasks.thrashosds.thrasher:fixing pg num pool .mgr 2024-09-20T22:50:17.402 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph pg dump --format=json 2024-09-20T22:50:17.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:17.403 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:17.403 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:50:17.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:17 smithi154 ceph-mon[33264]: pgmap v705: 158 pgs: 2 active+clean+snaptrim, 156 active+clean; 542 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 2.0 KiB/s rd, 748 KiB/s wr, 5 op/s 2024-09-20T22:50:17.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:17 smithi154 ceph-mon[33264]: 6.14 scrub starts 2024-09-20T22:50:17.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:17 smithi154 ceph-mon[33264]: 6.14 scrub ok 2024-09-20T22:50:17.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:17 smithi154 ceph-mon[33264]: from='client.16203 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T22:50:17.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:17 smithi154 ceph-mon[33264]: osdmap e537: 8 total, 8 up, 8 in 2024-09-20T22:50:17.504 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:50:17.514 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:50:17.525 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:17.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:17.526 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:50:17.558 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:17 smithi084 ceph-mon[40024]: pgmap v705: 158 pgs: 2 active+clean+snaptrim, 156 active+clean; 542 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 2.0 KiB/s rd, 748 KiB/s wr, 5 op/s 2024-09-20T22:50:17.558 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:17 smithi084 ceph-mon[40024]: 6.14 scrub starts 2024-09-20T22:50:17.558 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:17 smithi084 ceph-mon[40024]: 6.14 scrub ok 2024-09-20T22:50:17.558 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:17 smithi084 ceph-mon[40024]: from='client.16203 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T22:50:17.558 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:17 smithi084 ceph-mon[40024]: osdmap e537: 8 total, 8 up, 8 in 2024-09-20T22:50:17.558 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:17 smithi084 ceph-mon[29774]: pgmap v705: 158 pgs: 2 active+clean+snaptrim, 156 active+clean; 542 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 2.0 KiB/s rd, 748 KiB/s wr, 5 op/s 2024-09-20T22:50:17.558 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:17 smithi084 ceph-mon[29774]: 6.14 scrub starts 2024-09-20T22:50:17.558 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:17 smithi084 ceph-mon[29774]: 6.14 scrub ok 2024-09-20T22:50:17.559 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:17 smithi084 ceph-mon[29774]: from='client.16203 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T22:50:17.559 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:17 smithi084 ceph-mon[29774]: osdmap e537: 8 total, 8 up, 8 in 2024-09-20T22:50:17.627 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:50:17.649 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:17.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:17.650 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:50:17.750 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:50:17.764 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:50:17.771 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:17.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:17.774 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:50:17.875 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:50:17.896 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:17.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:17.897 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:50:17.902 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-20T22:50:17.991 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-20T22:50:17.991 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [], 2024-09-20T22:50:17.991 INFO:teuthology.orchestra.run.smithi154.stdout: "num_ops": 0 2024-09-20T22:50:17.991 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-20T22:50:17.997 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:50:18.020 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:18.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:18.021 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:50:18.122 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:50:18.144 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:18.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:18.145 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:50:18.233 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is unset 2024-09-20T22:50:18.238 INFO:tasks.rados.rados.0.smithi084.stdout:2103: done (5 left) 2024-09-20T22:50:18.238 INFO:tasks.rados.rados.0.smithi084.stdout:2104: done (4 left) 2024-09-20T22:50:18.238 INFO:tasks.rados.rados.0.smithi084.stdout:2105: done (3 left) 2024-09-20T22:50:18.238 INFO:tasks.rados.rados.0.smithi084.stdout:2106: done (2 left) 2024-09-20T22:50:18.238 INFO:tasks.rados.rados.0.smithi084.stdout:2108: done (1 left) 2024-09-20T22:50:18.238 INFO:tasks.rados.rados.0.smithi084.stdout:2109: done (0 left) 2024-09-20T22:50:18.239 INFO:tasks.rados.rados.0.smithi084.stdout:2110: snap_create 2024-09-20T22:50:18.246 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:50:18.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:18.271 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:18.271 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:50:18.372 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:50:18.393 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:18.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:18.394 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:50:18.495 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:50:18.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:18.517 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:18.517 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:50:18.529 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-20T22:50:18.530 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:50:18 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: ::ffff:172.21.15.154 - - [20/Sep/2024:22:50:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-20T22:50:18.531 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:18 smithi084 ceph-mon[40024]: 6.1e scrub starts 2024-09-20T22:50:18.531 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:18 smithi084 ceph-mon[40024]: 6.1e scrub ok 2024-09-20T22:50:18.531 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:18 smithi084 ceph-mon[40024]: 5.17 scrub starts 2024-09-20T22:50:18.531 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:18 smithi084 ceph-mon[40024]: 5.17 scrub ok 2024-09-20T22:50:18.531 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:18 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/4175246851' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:50:18.531 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:18 smithi084 ceph-mon[40024]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T22:50:18.531 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:18 smithi084 ceph-mon[40024]: Cluster is now healthy 2024-09-20T22:50:18.531 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:18 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/4175246851' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T22:50:18.531 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:18 smithi084 ceph-mon[40024]: osdmap e538: 8 total, 8 up, 8 in 2024-09-20T22:50:18.532 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:18 smithi084 ceph-mon[29774]: 6.1e scrub starts 2024-09-20T22:50:18.532 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:18 smithi084 ceph-mon[29774]: 6.1e scrub ok 2024-09-20T22:50:18.532 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:18 smithi084 ceph-mon[29774]: 5.17 scrub starts 2024-09-20T22:50:18.532 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:18 smithi084 ceph-mon[29774]: 5.17 scrub ok 2024-09-20T22:50:18.532 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:18 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/4175246851' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:50:18.532 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:18 smithi084 ceph-mon[29774]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T22:50:18.532 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:18 smithi084 ceph-mon[29774]: Cluster is now healthy 2024-09-20T22:50:18.532 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:18 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/4175246851' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T22:50:18.533 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:18 smithi084 ceph-mon[29774]: osdmap e538: 8 total, 8 up, 8 in 2024-09-20T22:50:18.537 INFO:teuthology.orchestra.run.smithi084.stderr:dumped all 2024-09-20T22:50:18.617 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:50:18.639 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:18.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:18.640 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:50:18.740 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:50:18.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:18 smithi154 ceph-mon[33264]: 6.1e scrub starts 2024-09-20T22:50:18.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:18 smithi154 ceph-mon[33264]: 6.1e scrub ok 2024-09-20T22:50:18.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:18 smithi154 ceph-mon[33264]: 5.17 scrub starts 2024-09-20T22:50:18.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:18 smithi154 ceph-mon[33264]: 5.17 scrub ok 2024-09-20T22:50:18.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:18 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/4175246851' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:50:18.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:18 smithi154 ceph-mon[33264]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T22:50:18.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:18 smithi154 ceph-mon[33264]: Cluster is now healthy 2024-09-20T22:50:18.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:18 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/4175246851' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T22:50:18.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:18 smithi154 ceph-mon[33264]: osdmap e538: 8 total, 8 up, 8 in 2024-09-20T22:50:18.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:18.762 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:18.763 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:50:18.863 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:50:18.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:18.889 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:18.889 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:50:18.908 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.6 dump_blocked_ops 2024-09-20T22:50:18.989 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:50:19.011 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:19.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:19.012 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:50:19.113 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:50:19.136 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:19.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:19.138 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:50:19.195 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:50:19.239 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:50:19.240 INFO:tasks.rados.rados.0.smithi084.stdout:2110: done (0 left) 2024-09-20T22:50:19.240 INFO:tasks.rados.rados.0.smithi084.stdout:2111: read oid 14 snap -1 2024-09-20T22:50:19.240 INFO:tasks.rados.rados.0.smithi084.stdout:2111: expect deleted 2024-09-20T22:50:19.240 INFO:tasks.rados.rados.0.smithi084.stdout:2112: read oid 32 snap -1 2024-09-20T22:50:19.240 INFO:tasks.rados.rados.0.smithi084.stdout:2112: expect (ObjNum 466 snap 128 seq_num 466) 2024-09-20T22:50:19.241 INFO:tasks.rados.rados.0.smithi084.stdout:2113: read oid 24 snap 205 2024-09-20T22:50:19.241 INFO:tasks.rados.rados.0.smithi084.stdout:2113: expect deleted 2024-09-20T22:50:19.241 INFO:tasks.rados.rados.0.smithi084.stdout:2114: copy_from oid 29 from oid 36 current snap is 234 2024-09-20T22:50:19.241 INFO:tasks.rados.rados.0.smithi084.stdout:2115: read oid 15 snap -1 2024-09-20T22:50:19.241 INFO:tasks.rados.rados.0.smithi084.stdout:2115: expect (ObjNum 625 snap 186 seq_num 625) 2024-09-20T22:50:19.241 INFO:tasks.rados.rados.0.smithi084.stdout:2116: snap_create 2024-09-20T22:50:19.242 INFO:tasks.rados.rados.0.smithi084.stdout:2115: expect (ObjNum 625 snap 186 seq_num 625) 2024-09-20T22:50:19.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:19.265 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:19.265 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:50:19.363 INFO:tasks.rados.rados.0.smithi084.stdout:2112: expect (ObjNum 466 snap 128 seq_num 466) 2024-09-20T22:50:19.366 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:50:19.387 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:19.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:19.388 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:50:19.489 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:50:19.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:19.511 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:19.511 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:50:19.551 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:50:19.600 INFO:tasks.rados.rados.0.smithi084.stdout:2114: finishing copy_from racing read to smithi08477203-29 2024-09-20T22:50:19.600 INFO:tasks.rados.rados.0.smithi084.stdout:2114: finishing copy_from to smithi08477203-29 2024-09-20T22:50:19.600 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 29 v 402 (ObjNum 440 snap 115 seq_num 440) dirty exists 2024-09-20T22:50:19.600 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:19 smithi154 ceph-mon[33264]: 3.1c scrub starts 2024-09-20T22:50:19.600 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:19 smithi154 ceph-mon[33264]: 3.1c scrub ok 2024-09-20T22:50:19.600 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:19 smithi154 ceph-mon[33264]: pgmap v708: 158 pgs: 2 active+clean+snaptrim, 156 active+clean; 542 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-20T22:50:19.600 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:19 smithi154 ceph-mon[33264]: from='client.26114 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T22:50:19.601 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:19 smithi154 ceph-mon[33264]: osdmap e539: 8 total, 8 up, 8 in 2024-09-20T22:50:19.601 INFO:journalctl@ceph.iscsi.iscsi.a.smithi154.stdout:Sep 20 22:50:19 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-iscsi-iscsi-a[59730]: debug there is no tcmu-runner data available 2024-09-20T22:50:19.614 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:50:19.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:19 smithi084 ceph-mon[29774]: 3.1c scrub starts 2024-09-20T22:50:19.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:19 smithi084 ceph-mon[29774]: 3.1c scrub ok 2024-09-20T22:50:19.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:19 smithi084 ceph-mon[29774]: pgmap v708: 158 pgs: 2 active+clean+snaptrim, 156 active+clean; 542 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-20T22:50:19.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:19 smithi084 ceph-mon[29774]: from='client.26114 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T22:50:19.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:19 smithi084 ceph-mon[29774]: osdmap e539: 8 total, 8 up, 8 in 2024-09-20T22:50:19.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:19 smithi084 ceph-mon[40024]: 3.1c scrub starts 2024-09-20T22:50:19.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:19 smithi084 ceph-mon[40024]: 3.1c scrub ok 2024-09-20T22:50:19.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:19 smithi084 ceph-mon[40024]: pgmap v708: 158 pgs: 2 active+clean+snaptrim, 156 active+clean; 542 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-20T22:50:19.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:19 smithi084 ceph-mon[40024]: from='client.26114 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T22:50:19.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:19 smithi084 ceph-mon[40024]: osdmap e539: 8 total, 8 up, 8 in 2024-09-20T22:50:19.637 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:19.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:19.638 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:50:19.700 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-20T22:50:19.700 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [], 2024-09-20T22:50:19.700 INFO:teuthology.orchestra.run.smithi154.stdout: "complaint_time": 30, 2024-09-20T22:50:19.700 INFO:teuthology.orchestra.run.smithi154.stdout: "num_blocked_ops": 0 2024-09-20T22:50:19.701 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-20T22:50:19.739 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:50:19.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:19.762 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:19.762 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:50:19.863 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:50:19.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:19.886 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:19.886 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:50:19.964 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:50:19.964 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:50:19.980 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:50:19.981 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:50:19.988 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:50:19.996 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:50:19.997 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:50:20.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:20.010 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:20.010 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:50:20.013 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-20T22:50:20.013 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T22:50:20.030 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:50:20.030 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:50:20.046 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:50:20.047 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:50:20.064 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:50:20.065 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:50:20.081 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:50:20.081 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:50:20.111 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:50:20.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:20.133 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:20.133 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:50:20.234 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:50:20.242 INFO:tasks.rados.rados.0.smithi084.stdout:2111: done (5 left) 2024-09-20T22:50:20.242 INFO:tasks.rados.rados.0.smithi084.stdout:2112: done (4 left) 2024-09-20T22:50:20.242 INFO:tasks.rados.rados.0.smithi084.stdout:2113: done (3 left) 2024-09-20T22:50:20.242 INFO:tasks.rados.rados.0.smithi084.stdout:2114: done (2 left) 2024-09-20T22:50:20.242 INFO:tasks.rados.rados.0.smithi084.stdout:2115: done (1 left) 2024-09-20T22:50:20.242 INFO:tasks.rados.rados.0.smithi084.stdout:2116: done (0 left) 2024-09-20T22:50:20.242 INFO:tasks.rados.rados.0.smithi084.stdout:2117: write oid 33 current snap is 235 2024-09-20T22:50:20.242 INFO:tasks.rados.rados.0.smithi084.stdout:2117: seq_num 738 ranges {403566=420003,1386064=546918} 2024-09-20T22:50:20.246 INFO:tasks.rados.rados.0.smithi084.stdout:2117: writing smithi08477203-33 from 403566 to 823569 tid 1 2024-09-20T22:50:20.253 INFO:tasks.rados.rados.0.smithi084.stdout:2117: writing smithi08477203-33 from 1386064 to 1932982 tid 2 2024-09-20T22:50:20.253 INFO:tasks.rados.rados.0.smithi084.stdout:2118: rollback oid 4 current snap is 235 2024-09-20T22:50:20.253 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 4 to 211 2024-09-20T22:50:20.253 INFO:tasks.rados.rados.0.smithi084.stdout:2117: finishing write tid 1 to smithi08477203-33 2024-09-20T22:50:20.254 INFO:tasks.rados.rados.0.smithi084.stdout:2119: snap_remove snap 156 2024-09-20T22:50:20.258 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:20.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:20.258 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:50:20.322 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.6 dump_historic_ops 2024-09-20T22:50:20.362 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:50:20.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:20.385 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:20.385 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:50:20.486 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:50:20.508 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:20.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:20.509 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:50:20.510 INFO:teuthology.orchestra.run.smithi084.stdout:{"pg_ready":true,"pg_map":{"version":709,"stamp":"2024-09-20T22:50:18.233674+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":568468860,"num_objects":460,"num_object_clones":212,"num_object_copies":1380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":460,"num_whiteouts":19,"num_read":4829,"num_read_kb":3795425,"num_write":2989,"num_write_kb":937568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":887,"num_bytes_recovered":303303598,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6042,"ondisk_log_size":6042,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":474,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":1563424,"kb_used_data":1159856,"kb_used_omap":12,"kb_used_meta":403379,"kb_avail":748397792,"statfs":{"total":767960285184,"available":766359339008,"internally_reserved":0,"allocated":1187692544,"data_stored":1724278189,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":413060656},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.993106"},"pg_stats":[{"pgid":"3.1f","version":"86'2","reported_seq":312,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.233777+0000","last_change":"2024-09-20T22:49:26.647633+0000","last_active":"2024-09-20T22:50:13.233777+0000","last_peered":"2024-09-20T22:50:13.233777+0000","last_clean":"2024-09-20T22:50:13.233777+0000","last_became_active":"2024-09-20T22:47:49.222114+0000","last_became_peered":"2024-09-20T22:47:49.222114+0000","last_unstale":"2024-09-20T22:50:13.233777+0000","last_undegraded":"2024-09-20T22:50:13.233777+0000","last_fullsized":"2024-09-20T22:50:13.233777+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'2","last_scrub_stamp":"2024-09-20T22:49:26.647557+0000","last_deep_scrub":"86'2","last_deep_scrub_stamp":"2024-09-20T22:42:39.664421+0000","last_clean_scrub_stamp":"2024-09-20T22:49:26.647557+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:40.650510+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":436,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"74'9","reported_seq":285,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.232611+0000","last_change":"2024-09-20T22:49:51.091564+0000","last_active":"2024-09-20T22:50:13.232611+0000","last_peered":"2024-09-20T22:50:13.232611+0000","last_clean":"2024-09-20T22:50:13.232611+0000","last_became_active":"2024-09-20T22:48:05.248980+0000","last_became_peered":"2024-09-20T22:48:05.248980+0000","last_unstale":"2024-09-20T22:50:13.232611+0000","last_undegraded":"2024-09-20T22:50:13.232611+0000","last_fullsized":"2024-09-20T22:50:13.232611+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-09-20T22:49:51.091416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:49:51.091416+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:55.306855+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":969,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.528842+0000","last_change":"2024-09-20T22:50:12.718744+0000","last_active":"2024-09-20T22:50:13.528842+0000","last_peered":"2024-09-20T22:50:13.528842+0000","last_clean":"2024-09-20T22:50:13.528842+0000","last_became_active":"2024-09-20T22:41:42.182024+0000","last_became_peered":"2024-09-20T22:41:42.182024+0000","last_unstale":"2024-09-20T22:50:13.528842+0000","last_undegraded":"2024-09-20T22:50:13.528842+0000","last_fullsized":"2024-09-20T22:50:13.528842+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:50:12.718677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:50:12.718677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:51:36.198787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":275,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.233010+0000","last_change":"2024-09-20T22:49:49.117854+0000","last_active":"2024-09-20T22:50:13.233010+0000","last_peered":"2024-09-20T22:50:13.233010+0000","last_clean":"2024-09-20T22:50:13.233010+0000","last_became_active":"2024-09-20T22:48:05.247569+0000","last_became_peered":"2024-09-20T22:48:05.247569+0000","last_unstale":"2024-09-20T22:50:13.233010+0000","last_undegraded":"2024-09-20T22:50:13.233010+0000","last_fullsized":"2024-09-20T22:50:13.233010+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:49.117772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:49:49.117772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:51.805116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":956,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.252805+0000","last_change":"2024-09-20T22:49:50.043884+0000","last_active":"2024-09-20T22:50:13.252805+0000","last_peered":"2024-09-20T22:50:13.252805+0000","last_clean":"2024-09-20T22:50:13.252805+0000","last_became_active":"2024-09-20T22:41:44.832099+0000","last_became_peered":"2024-09-20T22:41:44.832099+0000","last_unstale":"2024-09-20T22:50:13.252805+0000","last_undegraded":"2024-09-20T22:50:13.252805+0000","last_fullsized":"2024-09-20T22:50:13.252805+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:50.043776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:49:50.043776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:51:00.726179+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":961,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.253673+0000","last_change":"2024-09-20T22:49:10.077843+0000","last_active":"2024-09-20T22:50:13.253673+0000","last_peered":"2024-09-20T22:50:13.253673+0000","last_clean":"2024-09-20T22:50:13.253673+0000","last_became_active":"2024-09-20T22:41:38.157298+0000","last_became_peered":"2024-09-20T22:41:38.157298+0000","last_unstale":"2024-09-20T22:50:13.253673+0000","last_undegraded":"2024-09-20T22:50:13.253673+0000","last_fullsized":"2024-09-20T22:50:13.253673+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:10.077774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:49:10.077774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:21.915867+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"74'15","reported_seq":1025,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.254951+0000","last_change":"2024-09-20T22:49:07.042205+0000","last_active":"2024-09-20T22:50:13.254951+0000","last_peered":"2024-09-20T22:50:13.254951+0000","last_clean":"2024-09-20T22:50:13.254951+0000","last_became_active":"2024-09-20T22:47:49.218798+0000","last_became_peered":"2024-09-20T22:47:49.218798+0000","last_unstale":"2024-09-20T22:50:13.254951+0000","last_undegraded":"2024-09-20T22:50:13.254951+0000","last_fullsized":"2024-09-20T22:50:13.254951+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-09-20T22:49:07.042130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:49:07.042130+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:20.414124+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":275,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.232481+0000","last_change":"2024-09-20T22:49:59.011922+0000","last_active":"2024-09-20T22:50:13.232481+0000","last_peered":"2024-09-20T22:50:13.232481+0000","last_clean":"2024-09-20T22:50:13.232481+0000","last_became_active":"2024-09-20T22:48:05.247636+0000","last_became_peered":"2024-09-20T22:48:05.247636+0000","last_unstale":"2024-09-20T22:50:13.232481+0000","last_undegraded":"2024-09-20T22:50:13.232481+0000","last_fullsized":"2024-09-20T22:50:13.232481+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:59.011829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:49:59.011829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:51:07.355546+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":981,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.527710+0000","last_change":"2024-09-20T22:49:02.680764+0000","last_active":"2024-09-20T22:50:13.527710+0000","last_peered":"2024-09-20T22:50:13.527710+0000","last_clean":"2024-09-20T22:50:13.527710+0000","last_became_active":"2024-09-20T22:48:05.242059+0000","last_became_peered":"2024-09-20T22:48:05.242059+0000","last_unstale":"2024-09-20T22:50:13.527710+0000","last_undegraded":"2024-09-20T22:50:13.527710+0000","last_fullsized":"2024-09-20T22:50:13.527710+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:02.680682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:46:00.999939+0000","last_clean_scrub_stamp":"2024-09-20T22:49:02.680682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:22.162227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"74'9","reported_seq":285,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.233024+0000","last_change":"2024-09-20T22:49:56.048604+0000","last_active":"2024-09-20T22:50:13.233024+0000","last_peered":"2024-09-20T22:50:13.233024+0000","last_clean":"2024-09-20T22:50:13.233024+0000","last_became_active":"2024-09-20T22:48:05.233429+0000","last_became_peered":"2024-09-20T22:48:05.233429+0000","last_unstale":"2024-09-20T22:50:13.233024+0000","last_undegraded":"2024-09-20T22:50:13.233024+0000","last_fullsized":"2024-09-20T22:50:13.233024+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-09-20T22:49:56.048504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:49:56.048504+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:58.603199+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":973,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.528151+0000","last_change":"2024-09-20T22:49:31.649058+0000","last_active":"2024-09-20T22:50:13.528151+0000","last_peered":"2024-09-20T22:50:13.528151+0000","last_clean":"2024-09-20T22:50:13.528151+0000","last_became_active":"2024-09-20T22:47:49.227379+0000","last_became_peered":"2024-09-20T22:47:49.227379+0000","last_unstale":"2024-09-20T22:50:13.528151+0000","last_undegraded":"2024-09-20T22:50:13.528151+0000","last_fullsized":"2024-09-20T22:50:13.528151+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:31.648986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:49:31.648986+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:47.218836+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":308,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.232758+0000","last_change":"2024-09-20T22:49:08.664519+0000","last_active":"2024-09-20T22:50:13.232758+0000","last_peered":"2024-09-20T22:50:13.232758+0000","last_clean":"2024-09-20T22:50:13.232758+0000","last_became_active":"2024-09-20T22:47:49.225515+0000","last_became_peered":"2024-09-20T22:47:49.225515+0000","last_unstale":"2024-09-20T22:50:13.232758+0000","last_undegraded":"2024-09-20T22:50:13.232758+0000","last_fullsized":"2024-09-20T22:50:13.232758+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:08.664407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:49:08.664407+0000","last_clean_scrub_stamp":"2024-09-20T22:49:08.664407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:33.922538+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":981,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.527685+0000","last_change":"2024-09-20T22:48:37.559081+0000","last_active":"2024-09-20T22:50:13.527685+0000","last_peered":"2024-09-20T22:50:13.527685+0000","last_clean":"2024-09-20T22:50:13.527685+0000","last_became_active":"2024-09-20T22:48:05.242084+0000","last_became_peered":"2024-09-20T22:48:05.242084+0000","last_unstale":"2024-09-20T22:50:13.527685+0000","last_undegraded":"2024-09-20T22:50:13.527685+0000","last_fullsized":"2024-09-20T22:50:13.527685+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:48:37.559007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:48:37.559007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"181'7","reported_seq":281,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.233513+0000","last_change":"2024-09-20T22:48:41.049491+0000","last_active":"2024-09-20T22:50:13.233513+0000","last_peered":"2024-09-20T22:50:13.233513+0000","last_clean":"2024-09-20T22:50:13.233513+0000","last_became_active":"2024-09-20T22:48:05.249763+0000","last_became_peered":"2024-09-20T22:48:05.249763+0000","last_unstale":"2024-09-20T22:50:13.233513+0000","last_undegraded":"2024-09-20T22:50:13.233513+0000","last_fullsized":"2024-09-20T22:50:13.233513+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"181'7","last_scrub_stamp":"2024-09-20T22:48:41.049381+0000","last_deep_scrub":"181'7","last_deep_scrub_stamp":"2024-09-20T22:48:41.049381+0000","last_clean_scrub_stamp":"2024-09-20T22:48:41.049381+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":6,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":995,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.251509+0000","last_change":"2024-09-20T22:49:48.313924+0000","last_active":"2024-09-20T22:50:13.251509+0000","last_peered":"2024-09-20T22:50:13.251509+0000","last_clean":"2024-09-20T22:50:13.251509+0000","last_became_active":"2024-09-20T22:48:05.237791+0000","last_became_peered":"2024-09-20T22:48:05.237791+0000","last_unstale":"2024-09-20T22:50:13.251509+0000","last_undegraded":"2024-09-20T22:50:13.251509+0000","last_fullsized":"2024-09-20T22:50:13.251509+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:48.313860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:47:40.361958+0000","last_clean_scrub_stamp":"2024-09-20T22:49:48.313860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:49.887950+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":957,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.528200+0000","last_change":"2024-09-20T22:49:04.679724+0000","last_active":"2024-09-20T22:50:13.528200+0000","last_peered":"2024-09-20T22:50:13.528200+0000","last_clean":"2024-09-20T22:50:13.528200+0000","last_became_active":"2024-09-20T22:41:44.197927+0000","last_became_peered":"2024-09-20T22:41:44.197927+0000","last_unstale":"2024-09-20T22:50:13.528200+0000","last_undegraded":"2024-09-20T22:50:13.528200+0000","last_fullsized":"2024-09-20T22:50:13.528200+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:04.679617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:49:04.679617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":177,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.234229+0000","last_change":"2024-09-20T22:48:50.224321+0000","last_active":"2024-09-20T22:50:13.234229+0000","last_peered":"2024-09-20T22:50:13.234229+0000","last_clean":"2024-09-20T22:50:13.234229+0000","last_became_active":"2024-09-20T22:48:50.223895+0000","last_became_peered":"2024-09-20T22:48:50.223895+0000","last_unstale":"2024-09-20T22:50:13.234229+0000","last_undegraded":"2024-09-20T22:50:13.234229+0000","last_fullsized":"2024-09-20T22:50:13.234229+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:47:21.819572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:47:21.819572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:13.855674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":313,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.233557+0000","last_change":"2024-09-20T22:49:53.933673+0000","last_active":"2024-09-20T22:50:13.233557+0000","last_peered":"2024-09-20T22:50:13.233557+0000","last_clean":"2024-09-20T22:50:13.233557+0000","last_became_active":"2024-09-20T22:48:05.233687+0000","last_became_peered":"2024-09-20T22:48:05.233687+0000","last_unstale":"2024-09-20T22:50:13.233557+0000","last_undegraded":"2024-09-20T22:50:13.233557+0000","last_fullsized":"2024-09-20T22:50:13.233557+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:53.933601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:49:53.933601+0000","last_clean_scrub_stamp":"2024-09-20T22:49:53.933601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:51:03.708862+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"113'22","reported_seq":1011,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.237379+0000","last_change":"2024-09-20T22:49:52.154967+0000","last_active":"2024-09-20T22:50:13.237379+0000","last_peered":"2024-09-20T22:50:13.237379+0000","last_clean":"2024-09-20T22:50:13.237379+0000","last_became_active":"2024-09-20T22:41:40.170131+0000","last_became_peered":"2024-09-20T22:41:40.170131+0000","last_unstale":"2024-09-20T22:50:13.237379+0000","last_undegraded":"2024-09-20T22:50:13.237379+0000","last_fullsized":"2024-09-20T22:50:13.237379+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'22","last_scrub_stamp":"2024-09-20T22:49:52.154856+0000","last_deep_scrub":"113'22","last_deep_scrub_stamp":"2024-09-20T22:43:12.191878+0000","last_clean_scrub_stamp":"2024-09-20T22:49:52.154856+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:51:04.660560+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":985,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.527968+0000","last_change":"2024-09-20T22:49:01.165679+0000","last_active":"2024-09-20T22:50:13.527968+0000","last_peered":"2024-09-20T22:50:13.527968+0000","last_clean":"2024-09-20T22:50:13.527968+0000","last_became_active":"2024-09-20T22:48:05.249417+0000","last_became_peered":"2024-09-20T22:48:05.249417+0000","last_unstale":"2024-09-20T22:50:13.527968+0000","last_undegraded":"2024-09-20T22:50:13.527968+0000","last_fullsized":"2024-09-20T22:50:13.527968+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:01.165605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:49:01.165605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":949,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.252755+0000","last_change":"2024-09-20T22:49:30.132559+0000","last_active":"2024-09-20T22:50:13.252755+0000","last_peered":"2024-09-20T22:50:13.252755+0000","last_clean":"2024-09-20T22:50:13.252755+0000","last_became_active":"2024-09-20T22:41:44.832131+0000","last_became_peered":"2024-09-20T22:41:44.832131+0000","last_unstale":"2024-09-20T22:50:13.252755+0000","last_undegraded":"2024-09-20T22:50:13.252755+0000","last_fullsized":"2024-09-20T22:50:13.252755+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:30.132466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:45:25.002199+0000","last_clean_scrub_stamp":"2024-09-20T22:49:30.132466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:30.604086+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":275,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.233226+0000","last_change":"2024-09-20T22:48:24.986582+0000","last_active":"2024-09-20T22:50:13.233226+0000","last_peered":"2024-09-20T22:50:13.233226+0000","last_clean":"2024-09-20T22:50:13.233226+0000","last_became_active":"2024-09-20T22:48:05.250581+0000","last_became_peered":"2024-09-20T22:48:05.250581+0000","last_unstale":"2024-09-20T22:50:13.233226+0000","last_undegraded":"2024-09-20T22:50:13.233226+0000","last_fullsized":"2024-09-20T22:50:13.233226+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:48:24.986484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:48:24.986484+0000","last_clean_scrub_stamp":"2024-09-20T22:48:24.986484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"74'12","reported_seq":1033,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.251153+0000","last_change":"2024-09-20T22:48:59.088132+0000","last_active":"2024-09-20T22:50:13.251153+0000","last_peered":"2024-09-20T22:50:13.251153+0000","last_clean":"2024-09-20T22:50:13.251153+0000","last_became_active":"2024-09-20T22:48:05.233983+0000","last_became_peered":"2024-09-20T22:48:05.233983+0000","last_unstale":"2024-09-20T22:50:13.251153+0000","last_undegraded":"2024-09-20T22:50:13.251153+0000","last_fullsized":"2024-09-20T22:50:13.251153+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-09-20T22:48:59.088030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:48:59.088030+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:14.192165+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":275,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.233609+0000","last_change":"2024-09-20T22:48:23.940233+0000","last_active":"2024-09-20T22:50:13.233609+0000","last_peered":"2024-09-20T22:50:13.233609+0000","last_clean":"2024-09-20T22:50:13.233609+0000","last_became_active":"2024-09-20T22:48:05.250319+0000","last_became_peered":"2024-09-20T22:48:05.250319+0000","last_unstale":"2024-09-20T22:50:13.233609+0000","last_undegraded":"2024-09-20T22:50:13.233609+0000","last_fullsized":"2024-09-20T22:50:13.233609+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:48:23.940157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:48:23.940157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"154'2","reported_seq":958,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.254969+0000","last_change":"2024-09-20T22:48:39.357067+0000","last_active":"2024-09-20T22:50:13.254969+0000","last_peered":"2024-09-20T22:50:13.254969+0000","last_clean":"2024-09-20T22:50:13.254969+0000","last_became_active":"2024-09-20T22:41:44.199822+0000","last_became_peered":"2024-09-20T22:41:44.199822+0000","last_unstale":"2024-09-20T22:50:13.254969+0000","last_undegraded":"2024-09-20T22:50:13.254969+0000","last_fullsized":"2024-09-20T22:50:13.254969+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"154'2","last_scrub_stamp":"2024-09-20T22:48:39.356937+0000","last_deep_scrub":"154'2","last_deep_scrub_stamp":"2024-09-20T22:43:54.134066+0000","last_clean_scrub_stamp":"2024-09-20T22:48:39.356937+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"67'1","reported_seq":990,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.527647+0000","last_change":"2024-09-20T22:48:42.188853+0000","last_active":"2024-09-20T22:50:13.527647+0000","last_peered":"2024-09-20T22:50:13.527647+0000","last_clean":"2024-09-20T22:50:13.527647+0000","last_became_active":"2024-09-20T22:48:05.249578+0000","last_became_peered":"2024-09-20T22:48:05.249578+0000","last_unstale":"2024-09-20T22:50:13.527647+0000","last_undegraded":"2024-09-20T22:50:13.527647+0000","last_fullsized":"2024-09-20T22:50:13.527647+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-09-20T22:48:42.188772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:48:42.188772+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"151'15","reported_seq":320,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.235300+0000","last_change":"2024-09-20T22:48:37.528315+0000","last_active":"2024-09-20T22:50:13.235300+0000","last_peered":"2024-09-20T22:50:13.235300+0000","last_clean":"2024-09-20T22:50:13.235300+0000","last_became_active":"2024-09-20T22:47:49.232181+0000","last_became_peered":"2024-09-20T22:47:49.232181+0000","last_unstale":"2024-09-20T22:50:13.235300+0000","last_undegraded":"2024-09-20T22:50:13.235300+0000","last_fullsized":"2024-09-20T22:50:13.235300+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"151'15","last_scrub_stamp":"2024-09-20T22:48:37.528207+0000","last_deep_scrub":"151'15","last_deep_scrub_stamp":"2024-09-20T22:43:50.715761+0000","last_clean_scrub_stamp":"2024-09-20T22:48:37.528207+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:13.855820+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"408'10","reported_seq":1107,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.248187+0000","last_change":"2024-09-20T22:49:47.030294+0000","last_active":"2024-09-20T22:50:13.248187+0000","last_peered":"2024-09-20T22:50:13.248187+0000","last_clean":"2024-09-20T22:50:13.248187+0000","last_became_active":"2024-09-20T22:48:05.244270+0000","last_became_peered":"2024-09-20T22:48:05.244270+0000","last_unstale":"2024-09-20T22:50:13.248187+0000","last_undegraded":"2024-09-20T22:50:13.248187+0000","last_fullsized":"2024-09-20T22:50:13.248187+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"408'10","last_scrub_stamp":"2024-09-20T22:49:47.030217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:49:47.030217+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:56.479090+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.c","version":"533'1417","reported_seq":3571,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.271092+0000","last_change":"2024-09-20T22:50:11.300284+0000","last_active":"2024-09-20T22:50:13.271092+0000","last_peered":"2024-09-20T22:50:13.271092+0000","last_clean":"2024-09-20T22:50:13.271092+0000","last_became_active":"2024-09-20T22:42:52.404027+0000","last_became_peered":"2024-09-20T22:42:52.404027+0000","last_unstale":"2024-09-20T22:50:13.271092+0000","last_undegraded":"2024-09-20T22:50:13.271092+0000","last_fullsized":"2024-09-20T22:50:13.271092+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"508'1323","last_scrub_stamp":"2024-09-20T22:49:49.001444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:49:49.001444+0000","objects_scrubbed":31,"log_size":1417,"log_dups_size":0,"ondisk_log_size":1417,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:51:17.495494+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0.0046908890000000002,"stat_sum":{"num_bytes":77726889,"num_objects":35,"num_object_clones":28,"num_object_copies":105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":3,"num_read":629,"num_read_kb":641119,"num_write":445,"num_write_kb":170717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"d9","length":"1"},{"start":"dd","length":"2"},{"start":"e1","length":"1"}]},{"pgid":"1.a","version":"77'128","reported_seq":1228,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.528019+0000","last_change":"2024-09-20T22:48:24.566023+0000","last_active":"2024-09-20T22:50:13.528019+0000","last_peered":"2024-09-20T22:50:13.528019+0000","last_clean":"2024-09-20T22:50:13.528019+0000","last_became_active":"2024-09-20T22:42:52.407791+0000","last_became_peered":"2024-09-20T22:42:52.407791+0000","last_unstale":"2024-09-20T22:50:13.528019+0000","last_undegraded":"2024-09-20T22:50:13.528019+0000","last_fullsized":"2024-09-20T22:50:13.528019+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:48:24.565920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:48:24.565920+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"74'15","reported_seq":1042,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.528280+0000","last_change":"2024-09-20T22:49:55.709162+0000","last_active":"2024-09-20T22:50:13.528280+0000","last_peered":"2024-09-20T22:50:13.528280+0000","last_clean":"2024-09-20T22:50:13.528280+0000","last_became_active":"2024-09-20T22:48:05.250308+0000","last_became_peered":"2024-09-20T22:48:05.250308+0000","last_unstale":"2024-09-20T22:50:13.528280+0000","last_undegraded":"2024-09-20T22:50:13.528280+0000","last_fullsized":"2024-09-20T22:50:13.528280+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-09-20T22:49:55.709043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:49:55.709043+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:51:25.064971+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":960,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.251635+0000","last_change":"2024-09-20T22:49:12.070747+0000","last_active":"2024-09-20T22:50:13.251635+0000","last_peered":"2024-09-20T22:50:13.251635+0000","last_clean":"2024-09-20T22:50:13.251635+0000","last_became_active":"2024-09-20T22:41:38.156749+0000","last_became_peered":"2024-09-20T22:41:38.156749+0000","last_unstale":"2024-09-20T22:50:13.251635+0000","last_undegraded":"2024-09-20T22:50:13.251635+0000","last_fullsized":"2024-09-20T22:50:13.251635+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:12.070675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:49:12.070675+0000","last_clean_scrub_stamp":"2024-09-20T22:49:12.070675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:34.126084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"408'12","reported_seq":297,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.234172+0000","last_change":"2024-09-20T22:48:05.251167+0000","last_active":"2024-09-20T22:50:13.234172+0000","last_peered":"2024-09-20T22:50:13.234172+0000","last_clean":"2024-09-20T22:50:13.234172+0000","last_became_active":"2024-09-20T22:48:05.242914+0000","last_became_peered":"2024-09-20T22:48:05.242914+0000","last_unstale":"2024-09-20T22:50:13.234172+0000","last_undegraded":"2024-09-20T22:50:13.234172+0000","last_fullsized":"2024-09-20T22:50:13.234172+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"264'10","last_scrub_stamp":"2024-09-20T22:47:20.300787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:47:20.300787+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":969,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.528405+0000","last_change":"2024-09-20T22:49:07.658309+0000","last_active":"2024-09-20T22:50:13.528405+0000","last_peered":"2024-09-20T22:50:13.528405+0000","last_clean":"2024-09-20T22:50:13.528405+0000","last_became_active":"2024-09-20T22:47:49.227039+0000","last_became_peered":"2024-09-20T22:47:49.227039+0000","last_unstale":"2024-09-20T22:50:13.528405+0000","last_undegraded":"2024-09-20T22:50:13.528405+0000","last_fullsized":"2024-09-20T22:50:13.528405+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:07.658234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:49:07.658234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:26.817461+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3","version":"533'190","reported_seq":2228,"reported_epoch":533,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T22:50:13.325928+0000","last_change":"2024-09-20T22:50:13.325928+0000","last_active":"2024-09-20T22:50:13.325928+0000","last_peered":"2024-09-20T22:50:13.325928+0000","last_clean":"2024-09-20T22:50:13.325928+0000","last_became_active":"2024-09-20T22:49:05.221402+0000","last_became_peered":"2024-09-20T22:49:05.221402+0000","last_unstale":"2024-09-20T22:50:13.325928+0000","last_undegraded":"2024-09-20T22:50:13.325928+0000","last_fullsized":"2024-09-20T22:50:13.325928+0000","mapping_epoch":404,"log_start":"456'129","ondisk_log_start":"456'129","created":94,"last_epoch_clean":465,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'8","last_scrub_stamp":"2024-09-20T22:47:23.137750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:47:23.137750+0000","objects_scrubbed":0,"log_size":61,"log_dups_size":149,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.073199573000000004,"stat_sum":{"num_bytes":26803665,"num_objects":9,"num_object_clones":8,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":69,"num_read_kb":85984,"num_write":47,"num_write_kb":18651,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":20196357,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"d9","length":"1"},{"start":"dd","length":"2"}]},{"pgid":"1.5","version":"77'128","reported_seq":1134,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.233946+0000","last_change":"2024-09-20T22:49:26.023144+0000","last_active":"2024-09-20T22:50:13.233946+0000","last_peered":"2024-09-20T22:50:13.233946+0000","last_clean":"2024-09-20T22:50:13.233946+0000","last_became_active":"2024-09-20T22:48:08.207018+0000","last_became_peered":"2024-09-20T22:48:08.207018+0000","last_unstale":"2024-09-20T22:50:13.233946+0000","last_undegraded":"2024-09-20T22:50:13.233946+0000","last_fullsized":"2024-09-20T22:50:13.233946+0000","mapping_epoch":407,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":408,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:49:26.023044+0000","last_deep_scrub":"77'128","last_deep_scrub_stamp":"2024-09-20T22:45:41.608075+0000","last_clean_scrub_stamp":"2024-09-20T22:49:26.023044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:29.392816+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"74'18","reported_seq":1037,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.251878+0000","last_change":"2024-09-20T22:48:43.146697+0000","last_active":"2024-09-20T22:50:13.251878+0000","last_peered":"2024-09-20T22:50:13.251878+0000","last_clean":"2024-09-20T22:50:13.251878+0000","last_became_active":"2024-09-20T22:47:49.217605+0000","last_became_peered":"2024-09-20T22:47:49.217605+0000","last_unstale":"2024-09-20T22:50:13.251878+0000","last_undegraded":"2024-09-20T22:50:13.251878+0000","last_fullsized":"2024-09-20T22:50:13.251878+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'18","last_scrub_stamp":"2024-09-20T22:48:43.146602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:48:43.146602+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":972,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.251243+0000","last_change":"2024-09-20T22:47:49.227431+0000","last_active":"2024-09-20T22:50:13.251243+0000","last_peered":"2024-09-20T22:50:13.251243+0000","last_clean":"2024-09-20T22:50:13.251243+0000","last_became_active":"2024-09-20T22:47:49.227069+0000","last_became_peered":"2024-09-20T22:47:49.227069+0000","last_unstale":"2024-09-20T22:50:13.251243+0000","last_undegraded":"2024-09-20T22:50:13.251243+0000","last_fullsized":"2024-09-20T22:50:13.251243+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:47:39.982616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:47:39.982616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":276,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.233049+0000","last_change":"2024-09-20T22:49:27.968569+0000","last_active":"2024-09-20T22:50:13.233049+0000","last_peered":"2024-09-20T22:50:13.233049+0000","last_clean":"2024-09-20T22:50:13.233049+0000","last_became_active":"2024-09-20T22:48:05.230003+0000","last_became_peered":"2024-09-20T22:48:05.230003+0000","last_unstale":"2024-09-20T22:50:13.233049+0000","last_undegraded":"2024-09-20T22:50:13.233049+0000","last_fullsized":"2024-09-20T22:50:13.233049+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:27.968498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:07.947945+0000","last_clean_scrub_stamp":"2024-09-20T22:49:27.968498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:39.692203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":268,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.233161+0000","last_change":"2024-09-20T22:48:05.240266+0000","last_active":"2024-09-20T22:50:13.233161+0000","last_peered":"2024-09-20T22:50:13.233161+0000","last_clean":"2024-09-20T22:50:13.233161+0000","last_became_active":"2024-09-20T22:48:05.239917+0000","last_became_peered":"2024-09-20T22:48:05.239917+0000","last_unstale":"2024-09-20T22:50:13.233161+0000","last_undegraded":"2024-09-20T22:50:13.233161+0000","last_fullsized":"2024-09-20T22:50:13.233161+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:47:18.179575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:53.036601+0000","last_clean_scrub_stamp":"2024-09-20T22:47:18.179575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2","version":"525'529","reported_seq":2002,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.253953+0000","last_change":"2024-09-20T22:50:11.324189+0000","last_active":"2024-09-20T22:50:13.253953+0000","last_peered":"2024-09-20T22:50:13.253953+0000","last_clean":"2024-09-20T22:50:13.253953+0000","last_became_active":"2024-09-20T22:49:17.242481+0000","last_became_peered":"2024-09-20T22:49:17.242481+0000","last_unstale":"2024-09-20T22:50:13.253953+0000","last_undegraded":"2024-09-20T22:50:13.253953+0000","last_fullsized":"2024-09-20T22:50:13.253953+0000","mapping_epoch":469,"log_start":"471'464","ondisk_log_start":"471'464","created":94,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"282'120","last_scrub_stamp":"2024-09-20T22:46:22.099642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:46:22.099642+0000","objects_scrubbed":1,"log_size":65,"log_dups_size":687,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.030344758999999999,"stat_sum":{"num_bytes":37522363,"num_objects":17,"num_object_clones":13,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":339,"num_read_kb":374975,"num_write":185,"num_write_kb":61120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":120,"num_bytes_recovered":49828787,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"d9","length":"1"},{"start":"dd","length":"2"},{"start":"e1","length":"1"}]},{"pgid":"1.4","version":"77'128","reported_seq":1260,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.255548+0000","last_change":"2024-09-20T22:49:58.063457+0000","last_active":"2024-09-20T22:50:13.255548+0000","last_peered":"2024-09-20T22:50:13.255548+0000","last_clean":"2024-09-20T22:50:13.255548+0000","last_became_active":"2024-09-20T22:46:15.181305+0000","last_became_peered":"2024-09-20T22:46:15.181305+0000","last_unstale":"2024-09-20T22:50:13.255548+0000","last_undegraded":"2024-09-20T22:50:13.255548+0000","last_fullsized":"2024-09-20T22:50:13.255548+0000","mapping_epoch":94,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:49:58.063382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:49:58.063382+0000","objects_scrubbed":1,"log_size":0,"log_dups_size":59,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:51:06.139917+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"74'14","reported_seq":292,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.232961+0000","last_change":"2024-09-20T22:49:57.026346+0000","last_active":"2024-09-20T22:50:13.232961+0000","last_peered":"2024-09-20T22:50:13.232961+0000","last_clean":"2024-09-20T22:50:13.232961+0000","last_became_active":"2024-09-20T22:48:05.235812+0000","last_became_peered":"2024-09-20T22:48:05.235812+0000","last_unstale":"2024-09-20T22:50:13.232961+0000","last_undegraded":"2024-09-20T22:50:13.232961+0000","last_fullsized":"2024-09-20T22:50:13.232961+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'14","last_scrub_stamp":"2024-09-20T22:49:57.026245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:49:57.026245+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:51:23.191208+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"67'1","reported_seq":322,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.234831+0000","last_change":"2024-09-20T22:48:36.570753+0000","last_active":"2024-09-20T22:50:13.234831+0000","last_peered":"2024-09-20T22:50:13.234831+0000","last_clean":"2024-09-20T22:50:13.234831+0000","last_became_active":"2024-09-20T22:48:05.237425+0000","last_became_peered":"2024-09-20T22:48:05.237425+0000","last_unstale":"2024-09-20T22:50:13.234831+0000","last_undegraded":"2024-09-20T22:50:13.234831+0000","last_fullsized":"2024-09-20T22:50:13.234831+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-09-20T22:48:36.570672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:48:36.570672+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:13.855836+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"408'12","reported_seq":1105,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.250982+0000","last_change":"2024-09-20T22:49:25.151447+0000","last_active":"2024-09-20T22:50:13.250982+0000","last_peered":"2024-09-20T22:50:13.250982+0000","last_clean":"2024-09-20T22:50:13.250982+0000","last_became_active":"2024-09-20T22:48:05.234607+0000","last_became_peered":"2024-09-20T22:48:05.234607+0000","last_unstale":"2024-09-20T22:50:13.250982+0000","last_undegraded":"2024-09-20T22:50:13.250982+0000","last_fullsized":"2024-09-20T22:50:13.250982+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"408'12","last_scrub_stamp":"2024-09-20T22:49:25.151326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:49:25.151326+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:44.177686+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":957,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.253670+0000","last_change":"2024-09-20T22:49:31.310399+0000","last_active":"2024-09-20T22:50:13.253670+0000","last_peered":"2024-09-20T22:50:13.253670+0000","last_clean":"2024-09-20T22:50:13.253670+0000","last_became_active":"2024-09-20T22:41:44.831901+0000","last_became_peered":"2024-09-20T22:41:44.831901+0000","last_unstale":"2024-09-20T22:50:13.253670+0000","last_undegraded":"2024-09-20T22:50:13.253670+0000","last_fullsized":"2024-09-20T22:50:13.253670+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:31.310330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:49:31.310330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:34.862425+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1","version":"520'283","reported_seq":2400,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.233148+0000","last_change":"2024-09-20T22:50:11.283938+0000","last_active":"2024-09-20T22:50:13.233148+0000","last_peered":"2024-09-20T22:50:13.233148+0000","last_clean":"2024-09-20T22:50:13.233148+0000","last_became_active":"2024-09-20T22:49:21.232750+0000","last_became_peered":"2024-09-20T22:49:21.232750+0000","last_unstale":"2024-09-20T22:50:13.233148+0000","last_undegraded":"2024-09-20T22:50:13.233148+0000","last_fullsized":"2024-09-20T22:50:13.233148+0000","mapping_epoch":407,"log_start":"474'259","ondisk_log_start":"474'259","created":94,"last_epoch_clean":481,"parent":"0.0","parent_split_bits":0,"last_scrub":"420'231","last_scrub_stamp":"2024-09-20T22:48:22.915993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:48:22.915993+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":338,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00047044300000000002,"stat_sum":{"num_bytes":13160244,"num_objects":6,"num_object_clones":4,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":65,"num_read_kb":78461,"num_write":63,"num_write_kb":38776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":8732217,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"d9","length":"1"},{"start":"dd","length":"2"},{"start":"e1","length":"1"}]},{"pgid":"1.7","version":"77'128","reported_seq":1265,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.249274+0000","last_change":"2024-09-20T22:49:31.994386+0000","last_active":"2024-09-20T22:50:13.249274+0000","last_peered":"2024-09-20T22:50:13.249274+0000","last_clean":"2024-09-20T22:50:13.249274+0000","last_became_active":"2024-09-20T22:46:03.169596+0000","last_became_peered":"2024-09-20T22:46:03.169596+0000","last_unstale":"2024-09-20T22:50:13.249274+0000","last_undegraded":"2024-09-20T22:50:13.249274+0000","last_fullsized":"2024-09-20T22:50:13.249274+0000","mapping_epoch":94,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:49:31.994311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:49:31.994311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:35.643288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"74'10","reported_seq":1061,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.527823+0000","last_change":"2024-09-20T22:49:46.855799+0000","last_active":"2024-09-20T22:50:13.527823+0000","last_peered":"2024-09-20T22:50:13.527823+0000","last_clean":"2024-09-20T22:50:13.527823+0000","last_became_active":"2024-09-20T22:48:05.251200+0000","last_became_peered":"2024-09-20T22:48:05.251200+0000","last_unstale":"2024-09-20T22:50:13.527823+0000","last_undegraded":"2024-09-20T22:50:13.527823+0000","last_fullsized":"2024-09-20T22:50:13.527823+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-09-20T22:49:46.855673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:49:46.855673+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:55.301300+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":970,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.528756+0000","last_change":"2024-09-20T22:48:04.520131+0000","last_active":"2024-09-20T22:50:13.528756+0000","last_peered":"2024-09-20T22:50:13.528756+0000","last_clean":"2024-09-20T22:50:13.528756+0000","last_became_active":"2024-09-20T22:41:38.155629+0000","last_became_peered":"2024-09-20T22:41:38.155629+0000","last_unstale":"2024-09-20T22:50:13.528756+0000","last_undegraded":"2024-09-20T22:50:13.528756+0000","last_fullsized":"2024-09-20T22:50:13.528756+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:48:04.520053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:48:04.520053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"389'11","reported_seq":342,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.233508+0000","last_change":"2024-09-20T22:48:44.601907+0000","last_active":"2024-09-20T22:50:13.233508+0000","last_peered":"2024-09-20T22:50:13.233508+0000","last_clean":"2024-09-20T22:50:13.233508+0000","last_became_active":"2024-09-20T22:47:49.227264+0000","last_became_peered":"2024-09-20T22:47:49.227264+0000","last_unstale":"2024-09-20T22:50:13.233508+0000","last_undegraded":"2024-09-20T22:50:13.233508+0000","last_fullsized":"2024-09-20T22:50:13.233508+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'11","last_scrub_stamp":"2024-09-20T22:48:44.601816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:48:44.601816+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:16.790892+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":308,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.232580+0000","last_change":"2024-09-20T22:49:27.627363+0000","last_active":"2024-09-20T22:50:13.232580+0000","last_peered":"2024-09-20T22:50:13.232580+0000","last_clean":"2024-09-20T22:50:13.232580+0000","last_became_active":"2024-09-20T22:47:49.232380+0000","last_became_peered":"2024-09-20T22:47:49.232380+0000","last_unstale":"2024-09-20T22:50:13.232580+0000","last_undegraded":"2024-09-20T22:50:13.232580+0000","last_fullsized":"2024-09-20T22:50:13.232580+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:27.627295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:49:27.627295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:29.554480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.0","version":"529'315","reported_seq":2514,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.528056+0000","last_change":"2024-09-20T22:50:11.279156+0000","last_active":"2024-09-20T22:50:13.528056+0000","last_peered":"2024-09-20T22:50:13.528056+0000","last_clean":"2024-09-20T22:50:13.528056+0000","last_became_active":"2024-09-20T22:49:25.236419+0000","last_became_peered":"2024-09-20T22:49:25.236419+0000","last_unstale":"2024-09-20T22:50:13.528056+0000","last_undegraded":"2024-09-20T22:50:13.528056+0000","last_fullsized":"2024-09-20T22:50:13.528056+0000","mapping_epoch":94,"log_start":"480'253","ondisk_log_start":"480'253","created":94,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'228","last_scrub_stamp":"2024-09-20T22:48:59.633944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:48:59.633944+0000","objects_scrubbed":7,"log_size":62,"log_dups_size":416,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:23.086443+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.000544186,"stat_sum":{"num_bytes":32604467,"num_objects":17,"num_object_clones":14,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":1,"num_read":165,"num_read_kb":185505,"num_write":109,"num_write_kb":40172,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":7603720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"d9","length":"1"},{"start":"dd","length":"2"},{"start":"e1","length":"1"}]},{"pgid":"1.6","version":"77'128","reported_seq":1287,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.249034+0000","last_change":"2024-09-20T22:49:28.091254+0000","last_active":"2024-09-20T22:50:13.249034+0000","last_peered":"2024-09-20T22:50:13.249034+0000","last_clean":"2024-09-20T22:50:13.249034+0000","last_became_active":"2024-09-20T22:47:52.212190+0000","last_became_peered":"2024-09-20T22:47:52.212190+0000","last_unstale":"2024-09-20T22:50:13.249034+0000","last_undegraded":"2024-09-20T22:50:13.249034+0000","last_fullsized":"2024-09-20T22:50:13.249034+0000","mapping_epoch":391,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:49:28.091184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:49:28.091184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:40.740952+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"515'27","reported_seq":334,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.232193+0000","last_change":"2024-09-20T22:49:55.964247+0000","last_active":"2024-09-20T22:50:13.232193+0000","last_peered":"2024-09-20T22:50:13.232193+0000","last_clean":"2024-09-20T22:50:13.232193+0000","last_became_active":"2024-09-20T22:47:49.222009+0000","last_became_peered":"2024-09-20T22:47:49.222009+0000","last_unstale":"2024-09-20T22:50:13.232193+0000","last_undegraded":"2024-09-20T22:50:13.232193+0000","last_fullsized":"2024-09-20T22:50:13.232193+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"515'27","last_scrub_stamp":"2024-09-20T22:49:55.964153+0000","last_deep_scrub":"515'27","last_deep_scrub_stamp":"2024-09-20T22:49:55.964153+0000","last_clean_scrub_stamp":"2024-09-20T22:49:55.964153+0000","objects_scrubbed":8,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:57.993517+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":38,"num_read_kb":24,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":978,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.529214+0000","last_change":"2024-09-20T22:49:28.084441+0000","last_active":"2024-09-20T22:50:13.529214+0000","last_peered":"2024-09-20T22:50:13.529214+0000","last_clean":"2024-09-20T22:50:13.529214+0000","last_became_active":"2024-09-20T22:41:38.160832+0000","last_became_peered":"2024-09-20T22:41:38.160832+0000","last_unstale":"2024-09-20T22:50:13.529214+0000","last_undegraded":"2024-09-20T22:50:13.529214+0000","last_fullsized":"2024-09-20T22:50:13.529214+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:28.084371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:48:23.320031+0000","last_clean_scrub_stamp":"2024-09-20T22:49:28.084371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:30.046341+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":973,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.248487+0000","last_change":"2024-09-20T22:49:00.208581+0000","last_active":"2024-09-20T22:50:13.248487+0000","last_peered":"2024-09-20T22:50:13.248487+0000","last_clean":"2024-09-20T22:50:13.248487+0000","last_became_active":"2024-09-20T22:47:49.214055+0000","last_became_peered":"2024-09-20T22:47:49.214055+0000","last_unstale":"2024-09-20T22:50:13.248487+0000","last_undegraded":"2024-09-20T22:50:13.248487+0000","last_fullsized":"2024-09-20T22:50:13.248487+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:00.208454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:49:00.208454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":958,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.528741+0000","last_change":"2024-09-20T22:48:41.208933+0000","last_active":"2024-09-20T22:50:13.528741+0000","last_peered":"2024-09-20T22:50:13.528741+0000","last_clean":"2024-09-20T22:50:13.528741+0000","last_became_active":"2024-09-20T22:41:44.830538+0000","last_became_peered":"2024-09-20T22:41:44.830538+0000","last_unstale":"2024-09-20T22:50:13.528741+0000","last_undegraded":"2024-09-20T22:50:13.528741+0000","last_fullsized":"2024-09-20T22:50:13.528741+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:48:41.208853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:48:41.208853+0000","last_clean_scrub_stamp":"2024-09-20T22:48:41.208853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.7","version":"531'712","reported_seq":2993,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.249528+0000","last_change":"2024-09-20T22:50:11.297831+0000","last_active":"2024-09-20T22:50:13.249528+0000","last_peered":"2024-09-20T22:50:13.249528+0000","last_clean":"2024-09-20T22:50:13.249528+0000","last_became_active":"2024-09-20T22:48:43.224386+0000","last_became_peered":"2024-09-20T22:48:43.224386+0000","last_unstale":"2024-09-20T22:50:13.249528+0000","last_undegraded":"2024-09-20T22:50:13.249528+0000","last_fullsized":"2024-09-20T22:50:13.249528+0000","mapping_epoch":391,"log_start":"436'456","ondisk_log_start":"436'456","created":94,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"463'527","last_scrub_stamp":"2024-09-20T22:49:04.175508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:49:04.175508+0000","objects_scrubbed":23,"log_size":256,"log_dups_size":886,"ondisk_log_size":256,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.0034558119999999999,"stat_sum":{"num_bytes":53169886,"num_objects":31,"num_object_clones":26,"num_object_copies":93,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":4,"num_read":341,"num_read_kb":364157,"num_write":261,"num_write_kb":86105,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":115,"num_bytes_recovered":51615167,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"d9","length":"1"},{"start":"dd","length":"2"},{"start":"e1","length":"1"}]},{"pgid":"1.1","version":"77'128","reported_seq":1276,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.528166+0000","last_change":"2024-09-20T22:48:44.631654+0000","last_active":"2024-09-20T22:50:13.528166+0000","last_peered":"2024-09-20T22:50:13.528166+0000","last_clean":"2024-09-20T22:50:13.528166+0000","last_became_active":"2024-09-20T22:47:52.208102+0000","last_became_peered":"2024-09-20T22:47:52.208102+0000","last_unstale":"2024-09-20T22:50:13.528166+0000","last_undegraded":"2024-09-20T22:50:13.528166+0000","last_fullsized":"2024-09-20T22:50:13.528166+0000","mapping_epoch":391,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:48:44.631571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:48:44.631571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:15.693452+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"86'38","reported_seq":1136,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.529109+0000","last_change":"2024-09-20T22:49:31.040446+0000","last_active":"2024-09-20T22:50:13.529109+0000","last_peered":"2024-09-20T22:50:13.529109+0000","last_clean":"2024-09-20T22:50:13.529109+0000","last_became_active":"2024-09-20T22:41:40.166958+0000","last_became_peered":"2024-09-20T22:41:40.166958+0000","last_unstale":"2024-09-20T22:50:13.529109+0000","last_undegraded":"2024-09-20T22:50:13.529109+0000","last_fullsized":"2024-09-20T22:50:13.529109+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'38","last_scrub_stamp":"2024-09-20T22:49:31.040362+0000","last_deep_scrub":"86'38","last_deep_scrub_stamp":"2024-09-20T22:42:43.935677+0000","last_clean_scrub_stamp":"2024-09-20T22:49:31.040362+0000","objects_scrubbed":10,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:55.004800+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":268,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.233463+0000","last_change":"2024-09-20T22:48:05.238748+0000","last_active":"2024-09-20T22:50:13.233463+0000","last_peered":"2024-09-20T22:50:13.233463+0000","last_clean":"2024-09-20T22:50:13.233463+0000","last_became_active":"2024-09-20T22:48:05.237756+0000","last_became_peered":"2024-09-20T22:48:05.237756+0000","last_unstale":"2024-09-20T22:50:13.233463+0000","last_undegraded":"2024-09-20T22:50:13.233463+0000","last_fullsized":"2024-09-20T22:50:13.233463+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:47:16.817468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:47:16.817468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"69'2","reported_seq":988,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.528741+0000","last_change":"2024-09-20T22:49:27.604083+0000","last_active":"2024-09-20T22:50:13.528741+0000","last_peered":"2024-09-20T22:50:13.528741+0000","last_clean":"2024-09-20T22:50:13.528741+0000","last_became_active":"2024-09-20T22:41:37.148479+0000","last_became_peered":"2024-09-20T22:41:37.148479+0000","last_unstale":"2024-09-20T22:50:13.528741+0000","last_undegraded":"2024-09-20T22:50:13.528741+0000","last_fullsized":"2024-09-20T22:50:13.528741+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2024-09-20T22:49:27.603975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:36.130684+0000","last_clean_scrub_stamp":"2024-09-20T22:49:27.603975+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:37.292200+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00087280599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":314,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.234017+0000","last_change":"2024-09-20T22:49:02.723760+0000","last_active":"2024-09-20T22:50:13.234017+0000","last_peered":"2024-09-20T22:50:13.234017+0000","last_clean":"2024-09-20T22:50:13.234017+0000","last_became_active":"2024-09-20T22:48:05.233519+0000","last_became_peered":"2024-09-20T22:48:05.233519+0000","last_unstale":"2024-09-20T22:50:13.234017+0000","last_undegraded":"2024-09-20T22:50:13.234017+0000","last_fullsized":"2024-09-20T22:50:13.234017+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:02.723689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:49:02.723689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:30.036737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":968,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.251368+0000","last_change":"2024-09-20T22:49:57.022315+0000","last_active":"2024-09-20T22:50:13.251368+0000","last_peered":"2024-09-20T22:50:13.251368+0000","last_clean":"2024-09-20T22:50:13.251368+0000","last_became_active":"2024-09-20T22:48:05.234193+0000","last_became_peered":"2024-09-20T22:48:05.234193+0000","last_unstale":"2024-09-20T22:50:13.251368+0000","last_undegraded":"2024-09-20T22:50:13.251368+0000","last_fullsized":"2024-09-20T22:50:13.251368+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:57.022244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:49:57.022244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:51:26.737278+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"531'248","reported_seq":896,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.232773+0000","last_change":"2024-09-20T22:50:11.291631+0000","last_active":"2024-09-20T22:50:13.232773+0000","last_peered":"2024-09-20T22:50:13.232773+0000","last_clean":"2024-09-20T22:50:13.232773+0000","last_became_active":"2024-09-20T22:49:01.223578+0000","last_became_peered":"2024-09-20T22:49:01.223578+0000","last_unstale":"2024-09-20T22:50:13.232773+0000","last_undegraded":"2024-09-20T22:50:13.232773+0000","last_fullsized":"2024-09-20T22:50:13.232773+0000","mapping_epoch":404,"log_start":"456'176","ondisk_log_start":"456'176","created":94,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"461'178","last_scrub_stamp":"2024-09-20T22:49:01.696205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:49:01.696205+0000","objects_scrubbed":10,"log_size":72,"log_dups_size":318,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:18.018181+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0090190549999999998,"stat_sum":{"num_bytes":27923761,"num_objects":12,"num_object_clones":10,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":1,"num_read":117,"num_read_kb":115419,"num_write":92,"num_write_kb":32954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":10294275,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"d9","length":"1"},{"start":"dd","length":"2"},{"start":"e1","length":"1"}]},{"pgid":"1.2","version":"77'128","reported_seq":1345,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.253221+0000","last_change":"2024-09-20T22:49:55.364495+0000","last_active":"2024-09-20T22:50:13.253221+0000","last_peered":"2024-09-20T22:50:13.253221+0000","last_clean":"2024-09-20T22:50:13.253221+0000","last_became_active":"2024-09-20T22:48:08.203028+0000","last_became_peered":"2024-09-20T22:48:08.203028+0000","last_unstale":"2024-09-20T22:50:13.253221+0000","last_undegraded":"2024-09-20T22:50:13.253221+0000","last_fullsized":"2024-09-20T22:50:13.253221+0000","mapping_epoch":407,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":408,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:49:55.364421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:49:55.364421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:59.878790+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"443'18","reported_seq":1041,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.528196+0000","last_change":"2024-09-20T22:49:57.739451+0000","last_active":"2024-09-20T22:50:13.528196+0000","last_peered":"2024-09-20T22:50:13.528196+0000","last_clean":"2024-09-20T22:50:13.528196+0000","last_became_active":"2024-09-20T22:47:49.228702+0000","last_became_peered":"2024-09-20T22:47:49.228702+0000","last_unstale":"2024-09-20T22:50:13.528196+0000","last_undegraded":"2024-09-20T22:50:13.528196+0000","last_fullsized":"2024-09-20T22:50:13.528196+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"443'18","last_scrub_stamp":"2024-09-20T22:49:57.739362+0000","last_deep_scrub":"443'18","last_deep_scrub_stamp":"2024-09-20T22:48:44.182517+0000","last_clean_scrub_stamp":"2024-09-20T22:49:57.739362+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:51:00.790911+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":978,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.528581+0000","last_change":"2024-09-20T22:49:26.102066+0000","last_active":"2024-09-20T22:50:13.528581+0000","last_peered":"2024-09-20T22:50:13.528581+0000","last_clean":"2024-09-20T22:50:13.528581+0000","last_became_active":"2024-09-20T22:41:38.156837+0000","last_became_peered":"2024-09-20T22:41:38.156837+0000","last_unstale":"2024-09-20T22:50:13.528581+0000","last_undegraded":"2024-09-20T22:50:13.528581+0000","last_fullsized":"2024-09-20T22:50:13.528581+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:26.101994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:49:26.101994+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:29.729787+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"380'2","reported_seq":1014,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.237229+0000","last_change":"2024-09-20T22:49:30.148041+0000","last_active":"2024-09-20T22:50:13.237229+0000","last_peered":"2024-09-20T22:50:13.237229+0000","last_clean":"2024-09-20T22:50:13.237229+0000","last_became_active":"2024-09-20T22:47:49.222812+0000","last_became_peered":"2024-09-20T22:47:49.222812+0000","last_unstale":"2024-09-20T22:50:13.237229+0000","last_undegraded":"2024-09-20T22:50:13.237229+0000","last_fullsized":"2024-09-20T22:50:13.237229+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"380'2","last_scrub_stamp":"2024-09-20T22:49:30.147961+0000","last_deep_scrub":"380'2","last_deep_scrub_stamp":"2024-09-20T22:47:40.983039+0000","last_clean_scrub_stamp":"2024-09-20T22:49:30.147961+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:32.024321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001301459,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":967,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.237528+0000","last_change":"2024-09-20T22:49:27.195816+0000","last_active":"2024-09-20T22:50:13.237528+0000","last_peered":"2024-09-20T22:50:13.237528+0000","last_clean":"2024-09-20T22:50:13.237528+0000","last_became_active":"2024-09-20T22:41:42.185112+0000","last_became_peered":"2024-09-20T22:41:42.185112+0000","last_unstale":"2024-09-20T22:50:13.237528+0000","last_undegraded":"2024-09-20T22:50:13.237528+0000","last_fullsized":"2024-09-20T22:50:13.237528+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:27.195708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:47:22.881109+0000","last_clean_scrub_stamp":"2024-09-20T22:49:27.195708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:46.992983+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":957,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.255259+0000","last_change":"2024-09-20T22:49:04.332292+0000","last_active":"2024-09-20T22:50:13.255259+0000","last_peered":"2024-09-20T22:50:13.255259+0000","last_clean":"2024-09-20T22:50:13.255259+0000","last_became_active":"2024-09-20T22:41:44.830515+0000","last_became_peered":"2024-09-20T22:41:44.830515+0000","last_unstale":"2024-09-20T22:50:13.255259+0000","last_undegraded":"2024-09-20T22:50:13.255259+0000","last_fullsized":"2024-09-20T22:50:13.255259+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:04.332219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:49:04.332219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:26.530832+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"531'225","reported_seq":2452,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.255058+0000","last_change":"2024-09-20T22:50:11.288571+0000","last_active":"2024-09-20T22:50:13.255058+0000","last_peered":"2024-09-20T22:50:13.255058+0000","last_clean":"2024-09-20T22:50:13.255058+0000","last_became_active":"2024-09-20T22:48:57.236417+0000","last_became_peered":"2024-09-20T22:48:57.236417+0000","last_unstale":"2024-09-20T22:50:13.255058+0000","last_undegraded":"2024-09-20T22:50:13.255058+0000","last_fullsized":"2024-09-20T22:50:13.255058+0000","mapping_epoch":391,"log_start":"450'190","ondisk_log_start":"450'190","created":94,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'168","last_scrub_stamp":"2024-09-20T22:48:22.483112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:48:22.483112+0000","objects_scrubbed":7,"log_size":35,"log_dups_size":240,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0040204330000000003,"stat_sum":{"num_bytes":16569587,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":57,"num_read_kb":38715,"num_write":75,"num_write_kb":33558,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":19673726,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"d9","length":"1"},{"start":"dd","length":"2"},{"start":"e1","length":"1"}]},{"pgid":"1.3","version":"77'128","reported_seq":1293,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.528587+0000","last_change":"2024-09-20T22:49:49.789854+0000","last_active":"2024-09-20T22:50:13.528587+0000","last_peered":"2024-09-20T22:50:13.528587+0000","last_clean":"2024-09-20T22:50:13.528587+0000","last_became_active":"2024-09-20T22:48:08.208402+0000","last_became_peered":"2024-09-20T22:48:08.208402+0000","last_unstale":"2024-09-20T22:50:13.528587+0000","last_undegraded":"2024-09-20T22:50:13.528587+0000","last_fullsized":"2024-09-20T22:50:13.528587+0000","mapping_epoch":407,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":408,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:49:49.789778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:49:49.789778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:51:06.258069+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"74'12","reported_seq":314,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.232423+0000","last_change":"2024-09-20T22:47:49.246080+0000","last_active":"2024-09-20T22:50:13.232423+0000","last_peered":"2024-09-20T22:50:13.232423+0000","last_clean":"2024-09-20T22:50:13.232423+0000","last_became_active":"2024-09-20T22:47:49.224223+0000","last_became_peered":"2024-09-20T22:47:49.224223+0000","last_unstale":"2024-09-20T22:50:13.232423+0000","last_undegraded":"2024-09-20T22:50:13.232423+0000","last_fullsized":"2024-09-20T22:50:13.232423+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-09-20T22:47:39.293455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:47:39.293455+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:13.855657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":314,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.233366+0000","last_change":"2024-09-20T22:48:05.233412+0000","last_active":"2024-09-20T22:50:13.233366+0000","last_peered":"2024-09-20T22:50:13.233366+0000","last_clean":"2024-09-20T22:50:13.233366+0000","last_became_active":"2024-09-20T22:48:05.233093+0000","last_became_peered":"2024-09-20T22:48:05.233093+0000","last_unstale":"2024-09-20T22:50:13.233366+0000","last_undegraded":"2024-09-20T22:50:13.233366+0000","last_fullsized":"2024-09-20T22:50:13.233366+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:47:55.576301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:44:19.834641+0000","last_clean_scrub_stamp":"2024-09-20T22:47:55.576301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:13.855736+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"75'5","reported_seq":1508,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.253391+0000","last_change":"2024-09-20T22:48:44.254029+0000","last_active":"2024-09-20T22:50:13.253391+0000","last_peered":"2024-09-20T22:50:13.253391+0000","last_clean":"2024-09-20T22:50:13.253391+0000","last_became_active":"2024-09-20T22:47:49.214414+0000","last_became_peered":"2024-09-20T22:47:49.214414+0000","last_unstale":"2024-09-20T22:50:13.253391+0000","last_undegraded":"2024-09-20T22:50:13.253391+0000","last_fullsized":"2024-09-20T22:50:13.253391+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-09-20T22:48:44.253907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:36.130684+0000","last_clean_scrub_stamp":"2024-09-20T22:48:44.253907+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00078631700000000003,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":506,"num_read_kb":501,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":973,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.528773+0000","last_change":"2024-09-20T22:48:34.535169+0000","last_active":"2024-09-20T22:50:13.528773+0000","last_peered":"2024-09-20T22:50:13.528773+0000","last_clean":"2024-09-20T22:50:13.528773+0000","last_became_active":"2024-09-20T22:47:49.227371+0000","last_became_peered":"2024-09-20T22:47:49.227371+0000","last_unstale":"2024-09-20T22:50:13.528773+0000","last_undegraded":"2024-09-20T22:50:13.528773+0000","last_fullsized":"2024-09-20T22:50:13.528773+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:48:34.535100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:48:34.535100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":966,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.528887+0000","last_change":"2024-09-20T22:49:10.150277+0000","last_active":"2024-09-20T22:50:13.528887+0000","last_peered":"2024-09-20T22:50:13.528887+0000","last_clean":"2024-09-20T22:50:13.528887+0000","last_became_active":"2024-09-20T22:41:44.199046+0000","last_became_peered":"2024-09-20T22:41:44.199046+0000","last_unstale":"2024-09-20T22:50:13.528887+0000","last_undegraded":"2024-09-20T22:50:13.528887+0000","last_fullsized":"2024-09-20T22:50:13.528887+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:10.150205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:49:10.150205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.6","version":"531'272","reported_seq":2430,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.235053+0000","last_change":"2024-09-20T22:50:11.291261+0000","last_active":"2024-09-20T22:50:13.235053+0000","last_peered":"2024-09-20T22:50:13.235053+0000","last_clean":"2024-09-20T22:50:13.235053+0000","last_became_active":"2024-09-20T22:48:47.234531+0000","last_became_peered":"2024-09-20T22:48:47.234531+0000","last_unstale":"2024-09-20T22:50:13.235053+0000","last_undegraded":"2024-09-20T22:50:13.235053+0000","last_fullsized":"2024-09-20T22:50:13.235053+0000","mapping_epoch":391,"log_start":"443'181","ondisk_log_start":"443'181","created":94,"last_epoch_clean":447,"parent":"0.0","parent_split_bits":0,"last_scrub":"440'177","last_scrub_stamp":"2024-09-20T22:48:40.602342+0000","last_deep_scrub":"440'177","last_deep_scrub_stamp":"2024-09-20T22:48:40.602342+0000","last_clean_scrub_stamp":"2024-09-20T22:48:40.602342+0000","objects_scrubbed":4,"log_size":91,"log_dups_size":314,"ondisk_log_size":91,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0.0079483039999999998,"stat_sum":{"num_bytes":21209978,"num_objects":10,"num_object_clones":8,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":1,"num_read":113,"num_read_kb":107760,"num_write":90,"num_write_kb":30256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":18662529,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"d9","length":"1"},{"start":"dd","length":"2"},{"start":"e1","length":"1"}]},{"pgid":"4.5","version":"408'20","reported_seq":1195,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.248150+0000","last_change":"2024-09-20T22:49:56.004694+0000","last_active":"2024-09-20T22:50:13.248150+0000","last_peered":"2024-09-20T22:50:13.248150+0000","last_clean":"2024-09-20T22:50:13.248150+0000","last_became_active":"2024-09-20T22:48:05.243523+0000","last_became_peered":"2024-09-20T22:48:05.243523+0000","last_unstale":"2024-09-20T22:50:13.248150+0000","last_undegraded":"2024-09-20T22:50:13.248150+0000","last_fullsized":"2024-09-20T22:50:13.248150+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"408'20","last_scrub_stamp":"2024-09-20T22:49:56.004590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:49:56.004590+0000","objects_scrubbed":8,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:51:25.151724+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"67'1","reported_seq":967,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.254451+0000","last_change":"2024-09-20T22:49:29.161456+0000","last_active":"2024-09-20T22:50:13.254451+0000","last_peered":"2024-09-20T22:50:13.254451+0000","last_clean":"2024-09-20T22:50:13.254451+0000","last_became_active":"2024-09-20T22:41:38.158917+0000","last_became_peered":"2024-09-20T22:41:38.158917+0000","last_unstale":"2024-09-20T22:50:13.254451+0000","last_undegraded":"2024-09-20T22:50:13.254451+0000","last_fullsized":"2024-09-20T22:50:13.254451+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-09-20T22:49:29.161334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:49:29.161334+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:43.763458+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"77'128","reported_seq":1317,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.253542+0000","last_change":"2024-09-20T22:49:46.391695+0000","last_active":"2024-09-20T22:50:13.253542+0000","last_peered":"2024-09-20T22:50:13.253542+0000","last_clean":"2024-09-20T22:50:13.253542+0000","last_became_active":"2024-09-20T22:47:52.206169+0000","last_became_peered":"2024-09-20T22:47:52.206169+0000","last_unstale":"2024-09-20T22:50:13.253542+0000","last_undegraded":"2024-09-20T22:50:13.253542+0000","last_fullsized":"2024-09-20T22:50:13.253542+0000","mapping_epoch":391,"log_start":"77'128","ondisk_log_start":"77'128","created":19,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:49:46.391614+0000","last_deep_scrub":"77'128","last_deep_scrub_stamp":"2024-09-20T22:47:22.417099+0000","last_clean_scrub_stamp":"2024-09-20T22:49:46.391614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:51.817750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":961,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.255073+0000","last_change":"2024-09-20T22:48:35.359093+0000","last_active":"2024-09-20T22:50:13.255073+0000","last_peered":"2024-09-20T22:50:13.255073+0000","last_clean":"2024-09-20T22:50:13.255073+0000","last_became_active":"2024-09-20T22:41:42.184387+0000","last_became_peered":"2024-09-20T22:41:42.184387+0000","last_unstale":"2024-09-20T22:50:13.255073+0000","last_undegraded":"2024-09-20T22:50:13.255073+0000","last_fullsized":"2024-09-20T22:50:13.255073+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:48:35.359016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:48:35.359016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":977,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.527884+0000","last_change":"2024-09-20T22:49:57.715494+0000","last_active":"2024-09-20T22:50:13.527884+0000","last_peered":"2024-09-20T22:50:13.527884+0000","last_clean":"2024-09-20T22:50:13.527884+0000","last_became_active":"2024-09-20T22:48:05.241602+0000","last_became_peered":"2024-09-20T22:48:05.241602+0000","last_unstale":"2024-09-20T22:50:13.527884+0000","last_undegraded":"2024-09-20T22:50:13.527884+0000","last_fullsized":"2024-09-20T22:50:13.527884+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:57.715416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:49:57.715416+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:51:26.966224+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"518'313","reported_seq":2537,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.249052+0000","last_change":"2024-09-20T22:50:11.296028+0000","last_active":"2024-09-20T22:50:13.249052+0000","last_peered":"2024-09-20T22:50:13.249052+0000","last_clean":"2024-09-20T22:50:13.249052+0000","last_became_active":"2024-09-20T22:47:52.208200+0000","last_became_peered":"2024-09-20T22:47:52.208200+0000","last_unstale":"2024-09-20T22:50:13.249052+0000","last_undegraded":"2024-09-20T22:50:13.249052+0000","last_fullsized":"2024-09-20T22:50:13.249052+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"460'246","last_scrub_stamp":"2024-09-20T22:49:01.214729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:49:01.214729+0000","objects_scrubbed":8,"log_size":313,"log_dups_size":0,"ondisk_log_size":313,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00079812500000000001,"stat_sum":{"num_bytes":24977291,"num_objects":9,"num_object_clones":7,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":114,"num_read_kb":106336,"num_write":68,"num_write_kb":29565,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":13529610,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"d9","length":"1"},{"start":"dd","length":"2"},{"start":"e1","length":"1"}]},{"pgid":"4.e","version":"74'11","reported_seq":279,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.234166+0000","last_change":"2024-09-20T22:48:05.271944+0000","last_active":"2024-09-20T22:50:13.234166+0000","last_peered":"2024-09-20T22:50:13.234166+0000","last_clean":"2024-09-20T22:50:13.234166+0000","last_became_active":"2024-09-20T22:48:05.248271+0000","last_became_peered":"2024-09-20T22:48:05.248271+0000","last_unstale":"2024-09-20T22:50:13.234166+0000","last_undegraded":"2024-09-20T22:50:13.234166+0000","last_fullsized":"2024-09-20T22:50:13.234166+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-09-20T22:47:59.092235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:47:59.092235+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":275,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.232617+0000","last_change":"2024-09-20T22:49:30.998080+0000","last_active":"2024-09-20T22:50:13.232617+0000","last_peered":"2024-09-20T22:50:13.232617+0000","last_clean":"2024-09-20T22:50:13.232617+0000","last_became_active":"2024-09-20T22:48:05.254688+0000","last_became_peered":"2024-09-20T22:48:05.254688+0000","last_unstale":"2024-09-20T22:50:13.232617+0000","last_undegraded":"2024-09-20T22:50:13.232617+0000","last_fullsized":"2024-09-20T22:50:13.232617+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:30.998001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:49:30.998001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:47.839033+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":973,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.528008+0000","last_change":"2024-09-20T22:49:48.693437+0000","last_active":"2024-09-20T22:50:13.528008+0000","last_peered":"2024-09-20T22:50:13.528008+0000","last_clean":"2024-09-20T22:50:13.528008+0000","last_became_active":"2024-09-20T22:48:05.233778+0000","last_became_peered":"2024-09-20T22:48:05.233778+0000","last_unstale":"2024-09-20T22:50:13.528008+0000","last_undegraded":"2024-09-20T22:50:13.528008+0000","last_fullsized":"2024-09-20T22:50:13.528008+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:48.693333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:49:48.693333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:56.935700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":948,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.255484+0000","last_change":"2024-09-20T22:48:40.127195+0000","last_active":"2024-09-20T22:50:13.255484+0000","last_peered":"2024-09-20T22:50:13.255484+0000","last_clean":"2024-09-20T22:50:13.255484+0000","last_became_active":"2024-09-20T22:41:44.832009+0000","last_became_peered":"2024-09-20T22:41:44.832009+0000","last_unstale":"2024-09-20T22:50:13.255484+0000","last_undegraded":"2024-09-20T22:50:13.255484+0000","last_fullsized":"2024-09-20T22:50:13.255484+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:48:40.127123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:48:40.127123+0000","last_clean_scrub_stamp":"2024-09-20T22:48:40.127123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.e","version":"531'408","reported_seq":2656,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.237538+0000","last_change":"2024-09-20T22:50:11.287467+0000","last_active":"2024-09-20T22:50:13.237538+0000","last_peered":"2024-09-20T22:50:13.237538+0000","last_clean":"2024-09-20T22:50:13.237538+0000","last_became_active":"2024-09-20T22:47:52.209452+0000","last_became_peered":"2024-09-20T22:47:52.209452+0000","last_unstale":"2024-09-20T22:50:13.237538+0000","last_undegraded":"2024-09-20T22:50:13.237538+0000","last_fullsized":"2024-09-20T22:50:13.237538+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'360","last_scrub_stamp":"2024-09-20T22:49:25.149145+0000","last_deep_scrub":"174'60","last_deep_scrub_stamp":"2024-09-20T22:44:18.301776+0000","last_clean_scrub_stamp":"2024-09-20T22:49:25.149145+0000","objects_scrubbed":11,"log_size":408,"log_dups_size":0,"ondisk_log_size":408,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:26.290244+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.0063337599999999999,"stat_sum":{"num_bytes":27613132,"num_objects":13,"num_object_clones":11,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":1,"num_read":166,"num_read_kb":173052,"num_write":109,"num_write_kb":43661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":23110741,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"d9","length":"1"},{"start":"dd","length":"2"},{"start":"e1","length":"1"}]},{"pgid":"1.8","version":"77'128","reported_seq":1236,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.527975+0000","last_change":"2024-09-20T22:49:56.715035+0000","last_active":"2024-09-20T22:50:13.527975+0000","last_peered":"2024-09-20T22:50:13.527975+0000","last_clean":"2024-09-20T22:50:13.527975+0000","last_became_active":"2024-09-20T22:42:52.405594+0000","last_became_peered":"2024-09-20T22:42:52.405594+0000","last_unstale":"2024-09-20T22:50:13.527975+0000","last_undegraded":"2024-09-20T22:50:13.527975+0000","last_fullsized":"2024-09-20T22:50:13.527975+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:49:56.714919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:49:56.714919+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:51:24.489539+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"74'17","reported_seq":293,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.233851+0000","last_change":"2024-09-20T22:49:01.949542+0000","last_active":"2024-09-20T22:50:13.233851+0000","last_peered":"2024-09-20T22:50:13.233851+0000","last_clean":"2024-09-20T22:50:13.233851+0000","last_became_active":"2024-09-20T22:48:05.247971+0000","last_became_peered":"2024-09-20T22:48:05.247971+0000","last_unstale":"2024-09-20T22:50:13.233851+0000","last_undegraded":"2024-09-20T22:50:13.233851+0000","last_fullsized":"2024-09-20T22:50:13.233851+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'17","last_scrub_stamp":"2024-09-20T22:49:01.949451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:49:01.949451+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":28,"num_read_kb":18,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":516,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.237207+0000","last_change":"2024-09-20T22:49:32.138822+0000","last_active":"2024-09-20T22:50:13.237207+0000","last_peered":"2024-09-20T22:50:13.237207+0000","last_clean":"2024-09-20T22:50:13.237207+0000","last_became_active":"2024-09-20T22:48:05.240481+0000","last_became_peered":"2024-09-20T22:48:05.240481+0000","last_unstale":"2024-09-20T22:50:13.237207+0000","last_undegraded":"2024-09-20T22:50:13.237207+0000","last_fullsized":"2024-09-20T22:50:13.237207+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:32.138751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:35.492562+0000","last_clean_scrub_stamp":"2024-09-20T22:49:32.138751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:44.032030+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":985,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.528076+0000","last_change":"2024-09-20T22:49:09.188448+0000","last_active":"2024-09-20T22:50:13.528076+0000","last_peered":"2024-09-20T22:50:13.528076+0000","last_clean":"2024-09-20T22:50:13.528076+0000","last_became_active":"2024-09-20T22:48:05.250175+0000","last_became_peered":"2024-09-20T22:48:05.250175+0000","last_unstale":"2024-09-20T22:50:13.528076+0000","last_undegraded":"2024-09-20T22:50:13.528076+0000","last_fullsized":"2024-09-20T22:50:13.528076+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:09.188362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:49:09.188362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:36.362440+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":983,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.236528+0000","last_change":"2024-09-20T22:49:26.152417+0000","last_active":"2024-09-20T22:50:13.236528+0000","last_peered":"2024-09-20T22:50:13.236528+0000","last_clean":"2024-09-20T22:50:13.236528+0000","last_became_active":"2024-09-20T22:48:05.239006+0000","last_became_peered":"2024-09-20T22:48:05.239006+0000","last_unstale":"2024-09-20T22:50:13.236528+0000","last_undegraded":"2024-09-20T22:50:13.236528+0000","last_fullsized":"2024-09-20T22:50:13.236528+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:26.152316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:49:26.152316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:40.036383+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"532'499","reported_seq":2710,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.235215+0000","last_change":"2024-09-20T22:50:11.292082+0000","last_active":"2024-09-20T22:50:13.235215+0000","last_peered":"2024-09-20T22:50:13.235215+0000","last_clean":"2024-09-20T22:50:13.235215+0000","last_became_active":"2024-09-20T22:47:52.216198+0000","last_became_peered":"2024-09-20T22:47:52.216198+0000","last_unstale":"2024-09-20T22:50:13.235215+0000","last_undegraded":"2024-09-20T22:50:13.235215+0000","last_fullsized":"2024-09-20T22:50:13.235215+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"481'456","last_scrub_stamp":"2024-09-20T22:49:22.637524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:49:22.637524+0000","objects_scrubbed":9,"log_size":499,"log_dups_size":0,"ondisk_log_size":499,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:38.238133+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.007154082,"stat_sum":{"num_bytes":25104069,"num_objects":12,"num_object_clones":9,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":2,"num_read":233,"num_read_kb":231558,"num_write":146,"num_write_kb":49226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":53,"num_bytes_recovered":14983556,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"d9","length":"1"},{"start":"dd","length":"2"},{"start":"e1","length":"1"}]},{"pgid":"1.9","version":"77'128","reported_seq":1276,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.252945+0000","last_change":"2024-09-20T22:49:52.301456+0000","last_active":"2024-09-20T22:50:13.252945+0000","last_peered":"2024-09-20T22:50:13.252945+0000","last_clean":"2024-09-20T22:50:13.252945+0000","last_became_active":"2024-09-20T22:42:52.400353+0000","last_became_peered":"2024-09-20T22:42:52.400353+0000","last_unstale":"2024-09-20T22:50:13.252945+0000","last_undegraded":"2024-09-20T22:50:13.252945+0000","last_fullsized":"2024-09-20T22:50:13.252945+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:49:52.301348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:49:52.301348+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:51:11.793390+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"74'10","reported_seq":279,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.233659+0000","last_change":"2024-09-20T22:48:05.262827+0000","last_active":"2024-09-20T22:50:13.233659+0000","last_peered":"2024-09-20T22:50:13.233659+0000","last_clean":"2024-09-20T22:50:13.233659+0000","last_became_active":"2024-09-20T22:48:05.242811+0000","last_became_peered":"2024-09-20T22:48:05.242811+0000","last_unstale":"2024-09-20T22:50:13.233659+0000","last_undegraded":"2024-09-20T22:50:13.233659+0000","last_fullsized":"2024-09-20T22:50:13.233659+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-09-20T22:47:21.092309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:47:21.092309+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":992,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.251127+0000","last_change":"2024-09-20T22:50:13.205821+0000","last_active":"2024-09-20T22:50:13.251127+0000","last_peered":"2024-09-20T22:50:13.251127+0000","last_clean":"2024-09-20T22:50:13.251127+0000","last_became_active":"2024-09-20T22:48:05.233999+0000","last_became_peered":"2024-09-20T22:48:05.233999+0000","last_unstale":"2024-09-20T22:50:13.251127+0000","last_undegraded":"2024-09-20T22:50:13.251127+0000","last_fullsized":"2024-09-20T22:50:13.251127+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:50:13.205743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:50:13.205743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:51:41.953774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"261'9","reported_seq":1088,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.236239+0000","last_change":"2024-09-20T22:49:57.205107+0000","last_active":"2024-09-20T22:50:13.236239+0000","last_peered":"2024-09-20T22:50:13.236239+0000","last_clean":"2024-09-20T22:50:13.236239+0000","last_became_active":"2024-09-20T22:41:42.185863+0000","last_became_peered":"2024-09-20T22:41:42.185863+0000","last_unstale":"2024-09-20T22:50:13.236239+0000","last_undegraded":"2024-09-20T22:50:13.236239+0000","last_fullsized":"2024-09-20T22:50:13.236239+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'9","last_scrub_stamp":"2024-09-20T22:49:57.204987+0000","last_deep_scrub":"261'9","last_deep_scrub_stamp":"2024-09-20T22:46:50.974735+0000","last_clean_scrub_stamp":"2024-09-20T22:49:57.204987+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:51:09.810048+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":275,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.234114+0000","last_change":"2024-09-20T22:49:54.058299+0000","last_active":"2024-09-20T22:50:13.234114+0000","last_peered":"2024-09-20T22:50:13.234114+0000","last_clean":"2024-09-20T22:50:13.234114+0000","last_became_active":"2024-09-20T22:48:05.250284+0000","last_became_peered":"2024-09-20T22:48:05.250284+0000","last_unstale":"2024-09-20T22:50:13.234114+0000","last_undegraded":"2024-09-20T22:50:13.234114+0000","last_fullsized":"2024-09-20T22:50:13.234114+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:54.058184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:49:54.058184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:51:00.325103+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.8","version":"533'443","reported_seq":2527,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.236907+0000","last_change":"2024-09-20T22:50:11.286068+0000","last_active":"2024-09-20T22:50:13.236907+0000","last_peered":"2024-09-20T22:50:13.236907+0000","last_clean":"2024-09-20T22:50:13.236907+0000","last_became_active":"2024-09-20T22:48:31.224359+0000","last_became_peered":"2024-09-20T22:48:31.224359+0000","last_unstale":"2024-09-20T22:50:13.236907+0000","last_undegraded":"2024-09-20T22:50:13.236907+0000","last_fullsized":"2024-09-20T22:50:13.236907+0000","mapping_epoch":407,"log_start":"426'268","ondisk_log_start":"426'268","created":94,"last_epoch_clean":431,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'319","last_scrub_stamp":"2024-09-20T22:48:59.219393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:48:59.219393+0000","objects_scrubbed":8,"log_size":175,"log_dups_size":445,"ondisk_log_size":175,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:23.085464+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0072628140000000002,"stat_sum":{"num_bytes":34739607,"num_objects":17,"num_object_clones":14,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":1,"num_read":198,"num_read_kb":189772,"num_write":159,"num_write_kb":53832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":26027409,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"d9","length":"1"},{"start":"dd","length":"2"},{"start":"e1","length":"1"}]},{"pgid":"4.b","version":"74'9","reported_seq":334,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.233318+0000","last_change":"2024-09-20T22:48:05.256987+0000","last_active":"2024-09-20T22:50:13.233318+0000","last_peered":"2024-09-20T22:50:13.233318+0000","last_clean":"2024-09-20T22:50:13.233318+0000","last_became_active":"2024-09-20T22:48:05.233267+0000","last_became_peered":"2024-09-20T22:48:05.233267+0000","last_unstale":"2024-09-20T22:50:13.233318+0000","last_undegraded":"2024-09-20T22:50:13.233318+0000","last_fullsized":"2024-09-20T22:50:13.233318+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-09-20T22:48:03.584242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:48:03.584242+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:13.855721+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":969,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.528549+0000","last_change":"2024-09-20T22:49:05.638528+0000","last_active":"2024-09-20T22:50:13.528549+0000","last_peered":"2024-09-20T22:50:13.528549+0000","last_clean":"2024-09-20T22:50:13.528549+0000","last_became_active":"2024-09-20T22:41:38.155953+0000","last_became_peered":"2024-09-20T22:41:38.155953+0000","last_unstale":"2024-09-20T22:50:13.528549+0000","last_undegraded":"2024-09-20T22:50:13.528549+0000","last_fullsized":"2024-09-20T22:50:13.528549+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:05.638457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:49:05.638457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:25.666047+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":991,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.237397+0000","last_change":"2024-09-20T22:49:02.242848+0000","last_active":"2024-09-20T22:50:13.237397+0000","last_peered":"2024-09-20T22:50:13.237397+0000","last_clean":"2024-09-20T22:50:13.237397+0000","last_became_active":"2024-09-20T22:48:05.240613+0000","last_became_peered":"2024-09-20T22:48:05.240613+0000","last_unstale":"2024-09-20T22:50:13.237397+0000","last_undegraded":"2024-09-20T22:50:13.237397+0000","last_fullsized":"2024-09-20T22:50:13.237397+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:02.242757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:49:02.242757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:24.494551+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":308,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.233455+0000","last_change":"2024-09-20T22:49:31.666061+0000","last_active":"2024-09-20T22:50:13.233455+0000","last_peered":"2024-09-20T22:50:13.233455+0000","last_clean":"2024-09-20T22:50:13.233455+0000","last_became_active":"2024-09-20T22:47:49.219486+0000","last_became_peered":"2024-09-20T22:47:49.219486+0000","last_unstale":"2024-09-20T22:50:13.233455+0000","last_undegraded":"2024-09-20T22:50:13.233455+0000","last_fullsized":"2024-09-20T22:50:13.233455+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:31.665954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:49:31.665954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:38.433613+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"531'619","reported_seq":2864,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.234114+0000","last_change":"2024-09-20T22:50:11.296392+0000","last_active":"2024-09-20T22:50:13.234114+0000","last_peered":"2024-09-20T22:50:13.234114+0000","last_clean":"2024-09-20T22:50:13.234114+0000","last_became_active":"2024-09-20T22:48:27.272202+0000","last_became_peered":"2024-09-20T22:48:27.272202+0000","last_unstale":"2024-09-20T22:50:13.234114+0000","last_undegraded":"2024-09-20T22:50:13.234114+0000","last_fullsized":"2024-09-20T22:50:13.234114+0000","mapping_epoch":407,"log_start":"423'280","ondisk_log_start":"423'280","created":94,"last_epoch_clean":427,"parent":"0.0","parent_split_bits":0,"last_scrub":"423'271","last_scrub_stamp":"2024-09-20T22:48:23.382092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:48:23.382092+0000","objects_scrubbed":8,"log_size":339,"log_dups_size":649,"ondisk_log_size":339,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.011940723,"stat_sum":{"num_bytes":59196432,"num_objects":26,"num_object_clones":21,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":1,"num_read":311,"num_read_kb":319052,"num_write":237,"num_write_kb":88911,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":65,"num_bytes_recovered":38710860,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"d9","length":"1"},{"start":"dd","length":"2"},{"start":"e1","length":"1"}]},{"pgid":"4.a","version":"113'28","reported_seq":1012,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.248545+0000","last_change":"2024-09-20T22:49:03.201199+0000","last_active":"2024-09-20T22:50:13.248545+0000","last_peered":"2024-09-20T22:50:13.248545+0000","last_clean":"2024-09-20T22:50:13.248545+0000","last_became_active":"2024-09-20T22:41:40.252011+0000","last_became_peered":"2024-09-20T22:41:40.252011+0000","last_unstale":"2024-09-20T22:50:13.248545+0000","last_undegraded":"2024-09-20T22:50:13.248545+0000","last_fullsized":"2024-09-20T22:50:13.248545+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'28","last_scrub_stamp":"2024-09-20T22:49:03.201108+0000","last_deep_scrub":"113'28","last_deep_scrub_stamp":"2024-09-20T22:43:12.725900+0000","last_clean_scrub_stamp":"2024-09-20T22:49:03.201108+0000","objects_scrubbed":9,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:31.362019+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":968,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.255537+0000","last_change":"2024-09-20T22:49:12.324218+0000","last_active":"2024-09-20T22:50:13.255537+0000","last_peered":"2024-09-20T22:50:13.255537+0000","last_clean":"2024-09-20T22:50:13.255537+0000","last_became_active":"2024-09-20T22:41:38.161524+0000","last_became_peered":"2024-09-20T22:41:38.161524+0000","last_unstale":"2024-09-20T22:50:13.255537+0000","last_undegraded":"2024-09-20T22:50:13.255537+0000","last_fullsized":"2024-09-20T22:50:13.255537+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:12.324148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:49:12.324148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:36.961267+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":979,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.236895+0000","last_change":"2024-09-20T22:49:04.244778+0000","last_active":"2024-09-20T22:50:13.236895+0000","last_peered":"2024-09-20T22:50:13.236895+0000","last_clean":"2024-09-20T22:50:13.236895+0000","last_became_active":"2024-09-20T22:47:49.223220+0000","last_became_peered":"2024-09-20T22:47:49.223220+0000","last_unstale":"2024-09-20T22:50:13.236895+0000","last_undegraded":"2024-09-20T22:50:13.236895+0000","last_fullsized":"2024-09-20T22:50:13.236895+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:04.244659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:49:04.244659+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":964,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.253445+0000","last_change":"2024-09-20T22:49:30.310175+0000","last_active":"2024-09-20T22:50:13.253445+0000","last_peered":"2024-09-20T22:50:13.253445+0000","last_clean":"2024-09-20T22:50:13.253445+0000","last_became_active":"2024-09-20T22:41:44.199395+0000","last_became_peered":"2024-09-20T22:41:44.199395+0000","last_unstale":"2024-09-20T22:50:13.253445+0000","last_undegraded":"2024-09-20T22:50:13.253445+0000","last_fullsized":"2024-09-20T22:50:13.253445+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:30.310069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:47:18.329209+0000","last_clean_scrub_stamp":"2024-09-20T22:49:30.310069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:44.675926+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.a","version":"531'515","reported_seq":2693,"reported_epoch":533,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T22:50:13.254914+0000","last_change":"2024-09-20T22:50:12.254068+0000","last_active":"2024-09-20T22:50:13.254914+0000","last_peered":"2024-09-20T22:50:13.254914+0000","last_clean":"2024-09-20T22:50:13.254914+0000","last_became_active":"2024-09-20T22:42:52.405085+0000","last_became_peered":"2024-09-20T22:42:52.405085+0000","last_unstale":"2024-09-20T22:50:13.254914+0000","last_undegraded":"2024-09-20T22:50:13.254914+0000","last_fullsized":"2024-09-20T22:50:13.254914+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"505'466","last_scrub_stamp":"2024-09-20T22:49:47.088165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:49:47.088165+0000","objects_scrubbed":13,"log_size":515,"log_dups_size":0,"ondisk_log_size":515,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:52.309508+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.95982113800000002,"stat_sum":{"num_bytes":35889860,"num_objects":16,"num_object_clones":13,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":1,"num_read":295,"num_read_kb":332020,"num_write":183,"num_write_kb":69363,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"dd","length":"2"},{"start":"e1","length":"1"}]},{"pgid":"4.9","version":"74'12","reported_seq":279,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.232760+0000","last_change":"2024-09-20T22:48:05.284523+0000","last_active":"2024-09-20T22:50:13.232760+0000","last_peered":"2024-09-20T22:50:13.232760+0000","last_clean":"2024-09-20T22:50:13.232760+0000","last_became_active":"2024-09-20T22:48:05.249438+0000","last_became_peered":"2024-09-20T22:48:05.249438+0000","last_unstale":"2024-09-20T22:50:13.232760+0000","last_undegraded":"2024-09-20T22:50:13.232760+0000","last_fullsized":"2024-09-20T22:50:13.232760+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-09-20T22:47:58.421473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:47:58.421473+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":21,"num_read_kb":12,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":1000,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.252060+0000","last_change":"2024-09-20T22:49:24.300511+0000","last_active":"2024-09-20T22:50:13.252060+0000","last_peered":"2024-09-20T22:50:13.252060+0000","last_clean":"2024-09-20T22:50:13.252060+0000","last_became_active":"2024-09-20T22:48:05.240225+0000","last_became_peered":"2024-09-20T22:48:05.240225+0000","last_unstale":"2024-09-20T22:50:13.252060+0000","last_undegraded":"2024-09-20T22:50:13.252060+0000","last_fullsized":"2024-09-20T22:50:13.252060+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:24.300441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:46:52.465940+0000","last_clean_scrub_stamp":"2024-09-20T22:49:24.300441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:46.323016+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":987,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.236954+0000","last_change":"2024-09-20T22:49:54.216094+0000","last_active":"2024-09-20T22:50:13.236954+0000","last_peered":"2024-09-20T22:50:13.236954+0000","last_clean":"2024-09-20T22:50:13.236954+0000","last_became_active":"2024-09-20T22:47:49.222088+0000","last_became_peered":"2024-09-20T22:47:49.222088+0000","last_unstale":"2024-09-20T22:50:13.236954+0000","last_undegraded":"2024-09-20T22:50:13.236954+0000","last_fullsized":"2024-09-20T22:50:13.236954+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:54.216022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:46:01.828080+0000","last_clean_scrub_stamp":"2024-09-20T22:49:54.216022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:56.397469+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":948,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.252869+0000","last_change":"2024-09-20T22:47:38.019602+0000","last_active":"2024-09-20T22:50:13.252869+0000","last_peered":"2024-09-20T22:50:13.252869+0000","last_clean":"2024-09-20T22:50:13.252869+0000","last_became_active":"2024-09-20T22:41:44.192609+0000","last_became_peered":"2024-09-20T22:41:44.192609+0000","last_unstale":"2024-09-20T22:50:13.252869+0000","last_undegraded":"2024-09-20T22:50:13.252869+0000","last_fullsized":"2024-09-20T22:50:13.252869+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:47:38.019533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:55.170379+0000","last_clean_scrub_stamp":"2024-09-20T22:47:38.019533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"531'773","reported_seq":2953,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.529112+0000","last_change":"2024-09-20T22:50:13.296304+0000","last_active":"2024-09-20T22:50:13.529112+0000","last_peered":"2024-09-20T22:50:13.529112+0000","last_clean":"2024-09-20T22:50:13.529112+0000","last_became_active":"2024-09-20T22:42:52.409642+0000","last_became_peered":"2024-09-20T22:42:52.409642+0000","last_unstale":"2024-09-20T22:50:13.529112+0000","last_undegraded":"2024-09-20T22:50:13.529112+0000","last_fullsized":"2024-09-20T22:50:13.529112+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"194'116","last_scrub_stamp":"2024-09-20T22:44:37.097674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:44:37.097674+0000","objects_scrubbed":3,"log_size":773,"log_dups_size":0,"ondisk_log_size":773,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.99785855400000001,"stat_sum":{"num_bytes":53661122,"num_objects":25,"num_object_clones":21,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":2,"num_read":405,"num_read_kb":450576,"num_write":284,"num_write_kb":90656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"d9","length":"1"},{"start":"dd","length":"2"},{"start":"e1","length":"1"}]},{"pgid":"4.8","version":"74'15","reported_seq":990,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.528214+0000","last_change":"2024-09-20T22:49:47.687063+0000","last_active":"2024-09-20T22:50:13.528214+0000","last_peered":"2024-09-20T22:50:13.528214+0000","last_clean":"2024-09-20T22:50:13.528214+0000","last_became_active":"2024-09-20T22:41:40.252113+0000","last_became_peered":"2024-09-20T22:41:40.252113+0000","last_unstale":"2024-09-20T22:50:13.528214+0000","last_undegraded":"2024-09-20T22:50:13.528214+0000","last_fullsized":"2024-09-20T22:50:13.528214+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-09-20T22:49:47.686984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:49:47.686984+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:51:06.196255+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'2","reported_seq":1028,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.252226+0000","last_change":"2024-09-20T22:49:22.379093+0000","last_active":"2024-09-20T22:50:13.252226+0000","last_peered":"2024-09-20T22:50:13.252226+0000","last_clean":"2024-09-20T22:50:13.252226+0000","last_became_active":"2024-09-20T22:48:05.238098+0000","last_became_peered":"2024-09-20T22:48:05.238098+0000","last_unstale":"2024-09-20T22:50:13.252226+0000","last_undegraded":"2024-09-20T22:50:13.252226+0000","last_fullsized":"2024-09-20T22:50:13.252226+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-09-20T22:49:22.379012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:49:22.379012+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:31.674675+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"408'10","reported_seq":1100,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.251127+0000","last_change":"2024-09-20T22:49:47.363588+0000","last_active":"2024-09-20T22:50:13.251127+0000","last_peered":"2024-09-20T22:50:13.251127+0000","last_clean":"2024-09-20T22:50:13.251127+0000","last_became_active":"2024-09-20T22:48:05.239344+0000","last_became_peered":"2024-09-20T22:48:05.239344+0000","last_unstale":"2024-09-20T22:50:13.251127+0000","last_undegraded":"2024-09-20T22:50:13.251127+0000","last_fullsized":"2024-09-20T22:50:13.251127+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"408'10","last_scrub_stamp":"2024-09-20T22:49:47.363515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:49:47.363515+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:51:01.473177+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":969,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.527691+0000","last_change":"2024-09-20T22:49:50.693099+0000","last_active":"2024-09-20T22:50:13.527691+0000","last_peered":"2024-09-20T22:50:13.527691+0000","last_clean":"2024-09-20T22:50:13.527691+0000","last_became_active":"2024-09-20T22:47:49.227569+0000","last_became_peered":"2024-09-20T22:47:49.227569+0000","last_unstale":"2024-09-20T22:50:13.527691+0000","last_undegraded":"2024-09-20T22:50:13.527691+0000","last_fullsized":"2024-09-20T22:50:13.527691+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:50.693003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:49:50.693003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:51:08.138542+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":988,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.248284+0000","last_change":"2024-09-20T22:49:02.203750+0000","last_active":"2024-09-20T22:50:13.248284+0000","last_peered":"2024-09-20T22:50:13.248284+0000","last_clean":"2024-09-20T22:50:13.248284+0000","last_became_active":"2024-09-20T22:47:49.213292+0000","last_became_peered":"2024-09-20T22:47:49.213292+0000","last_unstale":"2024-09-20T22:50:13.248284+0000","last_undegraded":"2024-09-20T22:50:13.248284+0000","last_fullsized":"2024-09-20T22:50:13.248284+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:02.203661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:49:02.203661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:30.443605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"74'6","reported_seq":987,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.253085+0000","last_change":"2024-09-20T22:49:48.113810+0000","last_active":"2024-09-20T22:50:13.253085+0000","last_peered":"2024-09-20T22:50:13.253085+0000","last_clean":"2024-09-20T22:50:13.253085+0000","last_became_active":"2024-09-20T22:43:09.116887+0000","last_became_peered":"2024-09-20T22:43:09.116887+0000","last_unstale":"2024-09-20T22:50:13.253085+0000","last_undegraded":"2024-09-20T22:50:13.253085+0000","last_fullsized":"2024-09-20T22:50:13.253085+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'6","last_scrub_stamp":"2024-09-20T22:49:48.113740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:49:48.113740+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:51:07.964989+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":961,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.528605+0000","last_change":"2024-09-20T22:48:35.550867+0000","last_active":"2024-09-20T22:50:13.528605+0000","last_peered":"2024-09-20T22:50:13.528605+0000","last_clean":"2024-09-20T22:50:13.528605+0000","last_became_active":"2024-09-20T22:41:42.184930+0000","last_became_peered":"2024-09-20T22:41:42.184930+0000","last_unstale":"2024-09-20T22:50:13.528605+0000","last_undegraded":"2024-09-20T22:50:13.528605+0000","last_fullsized":"2024-09-20T22:50:13.528605+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:48:35.550781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:09.705632+0000","last_clean_scrub_stamp":"2024-09-20T22:48:35.550781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"74'1","reported_seq":977,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.252078+0000","last_change":"2024-09-20T22:48:40.309005+0000","last_active":"2024-09-20T22:50:13.252078+0000","last_peered":"2024-09-20T22:50:13.252078+0000","last_clean":"2024-09-20T22:50:13.252078+0000","last_became_active":"2024-09-20T22:48:05.237882+0000","last_became_peered":"2024-09-20T22:48:05.237882+0000","last_unstale":"2024-09-20T22:50:13.252078+0000","last_undegraded":"2024-09-20T22:50:13.252078+0000","last_fullsized":"2024-09-20T22:50:13.252078+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-09-20T22:48:40.308879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:48:40.308879+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"74'9","reported_seq":318,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.235248+0000","last_change":"2024-09-20T22:49:11.739149+0000","last_active":"2024-09-20T22:50:13.235248+0000","last_peered":"2024-09-20T22:50:13.235248+0000","last_clean":"2024-09-20T22:50:13.235248+0000","last_became_active":"2024-09-20T22:47:49.231315+0000","last_became_peered":"2024-09-20T22:47:49.231315+0000","last_unstale":"2024-09-20T22:50:13.235248+0000","last_undegraded":"2024-09-20T22:50:13.235248+0000","last_fullsized":"2024-09-20T22:50:13.235248+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-09-20T22:49:11.739076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:49:11.739076+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:17.900700+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":992,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.251315+0000","last_change":"2024-09-20T22:49:25.302834+0000","last_active":"2024-09-20T22:50:13.251315+0000","last_peered":"2024-09-20T22:50:13.251315+0000","last_clean":"2024-09-20T22:50:13.251315+0000","last_became_active":"2024-09-20T22:48:05.238801+0000","last_became_peered":"2024-09-20T22:48:05.238801+0000","last_unstale":"2024-09-20T22:50:13.251315+0000","last_undegraded":"2024-09-20T22:50:13.251315+0000","last_fullsized":"2024-09-20T22:50:13.251315+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:25.302756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:49:25.302756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:30.192311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":952,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.252849+0000","last_change":"2024-09-20T22:46:48.012442+0000","last_active":"2024-09-20T22:50:13.252849+0000","last_peered":"2024-09-20T22:50:13.252849+0000","last_clean":"2024-09-20T22:50:13.252849+0000","last_became_active":"2024-09-20T22:41:42.187706+0000","last_became_peered":"2024-09-20T22:41:42.187706+0000","last_unstale":"2024-09-20T22:50:13.252849+0000","last_undegraded":"2024-09-20T22:50:13.252849+0000","last_fullsized":"2024-09-20T22:50:13.252849+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:46:48.012372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:46:48.012372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":987,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.237059+0000","last_change":"2024-09-20T22:48:40.247992+0000","last_active":"2024-09-20T22:50:13.237059+0000","last_peered":"2024-09-20T22:50:13.237059+0000","last_clean":"2024-09-20T22:50:13.237059+0000","last_became_active":"2024-09-20T22:48:05.239710+0000","last_became_peered":"2024-09-20T22:48:05.239710+0000","last_unstale":"2024-09-20T22:50:13.237059+0000","last_undegraded":"2024-09-20T22:50:13.237059+0000","last_fullsized":"2024-09-20T22:50:13.237059+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:48:40.247889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:48:40.247889+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"74'9","reported_seq":981,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.528659+0000","last_change":"2024-09-20T22:49:09.704604+0000","last_active":"2024-09-20T22:50:13.528659+0000","last_peered":"2024-09-20T22:50:13.528659+0000","last_clean":"2024-09-20T22:50:13.528659+0000","last_became_active":"2024-09-20T22:41:40.252486+0000","last_became_peered":"2024-09-20T22:41:40.252486+0000","last_unstale":"2024-09-20T22:50:13.528659+0000","last_undegraded":"2024-09-20T22:50:13.528659+0000","last_fullsized":"2024-09-20T22:50:13.528659+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-09-20T22:49:09.704521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:49:09.704521+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:15.287583+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":308,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.232420+0000","last_change":"2024-09-20T22:48:19.379265+0000","last_active":"2024-09-20T22:50:13.232420+0000","last_peered":"2024-09-20T22:50:13.232420+0000","last_clean":"2024-09-20T22:50:13.232420+0000","last_became_active":"2024-09-20T22:47:49.229936+0000","last_became_peered":"2024-09-20T22:47:49.229936+0000","last_unstale":"2024-09-20T22:50:13.232420+0000","last_undegraded":"2024-09-20T22:50:13.232420+0000","last_fullsized":"2024-09-20T22:50:13.232420+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:48:19.379196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:48:19.379196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:13.855785+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"74'8","reported_seq":1064,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.255293+0000","last_change":"2024-09-20T22:47:56.150641+0000","last_active":"2024-09-20T22:50:13.255293+0000","last_peered":"2024-09-20T22:50:13.255293+0000","last_clean":"2024-09-20T22:50:13.255293+0000","last_became_active":"2024-09-20T22:41:42.186352+0000","last_became_peered":"2024-09-20T22:41:42.186352+0000","last_unstale":"2024-09-20T22:50:13.255293+0000","last_undegraded":"2024-09-20T22:50:13.255293+0000","last_fullsized":"2024-09-20T22:50:13.255293+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-09-20T22:47:56.150562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:47:56.150562+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":267,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.234059+0000","last_change":"2024-09-20T22:48:05.250192+0000","last_active":"2024-09-20T22:50:13.234059+0000","last_peered":"2024-09-20T22:50:13.234059+0000","last_clean":"2024-09-20T22:50:13.234059+0000","last_became_active":"2024-09-20T22:48:05.249656+0000","last_became_peered":"2024-09-20T22:48:05.249656+0000","last_unstale":"2024-09-20T22:50:13.234059+0000","last_undegraded":"2024-09-20T22:50:13.234059+0000","last_fullsized":"2024-09-20T22:50:13.234059+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:47:19.828541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:47:19.828541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"74'10","reported_seq":971,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.254977+0000","last_change":"2024-09-20T22:46:24.269736+0000","last_active":"2024-09-20T22:50:13.254977+0000","last_peered":"2024-09-20T22:50:13.254977+0000","last_clean":"2024-09-20T22:50:13.254977+0000","last_became_active":"2024-09-20T22:41:40.253614+0000","last_became_peered":"2024-09-20T22:41:40.253614+0000","last_unstale":"2024-09-20T22:50:13.254977+0000","last_undegraded":"2024-09-20T22:50:13.254977+0000","last_fullsized":"2024-09-20T22:50:13.254977+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-09-20T22:46:24.269655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:46:24.269655+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":988,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.251667+0000","last_change":"2024-09-20T22:49:54.381836+0000","last_active":"2024-09-20T22:50:13.251667+0000","last_peered":"2024-09-20T22:50:13.251667+0000","last_clean":"2024-09-20T22:50:13.251667+0000","last_became_active":"2024-09-20T22:48:05.240231+0000","last_became_peered":"2024-09-20T22:48:05.240231+0000","last_unstale":"2024-09-20T22:50:13.251667+0000","last_undegraded":"2024-09-20T22:50:13.251667+0000","last_fullsized":"2024-09-20T22:50:13.251667+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:54.381721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:49:54.381721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:51:04.253551+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"389'10","reported_seq":1104,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.528521+0000","last_change":"2024-09-20T22:49:51.715574+0000","last_active":"2024-09-20T22:50:13.528521+0000","last_peered":"2024-09-20T22:50:13.528521+0000","last_clean":"2024-09-20T22:50:13.528521+0000","last_became_active":"2024-09-20T22:47:49.227113+0000","last_became_peered":"2024-09-20T22:47:49.227113+0000","last_unstale":"2024-09-20T22:50:13.528521+0000","last_undegraded":"2024-09-20T22:50:13.528521+0000","last_fullsized":"2024-09-20T22:50:13.528521+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'10","last_scrub_stamp":"2024-09-20T22:49:51.715471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:49:51.715471+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:51:04.561910+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":300,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.234462+0000","last_change":"2024-09-20T22:47:49.232560+0000","last_active":"2024-09-20T22:50:13.234462+0000","last_peered":"2024-09-20T22:50:13.234462+0000","last_clean":"2024-09-20T22:50:13.234462+0000","last_became_active":"2024-09-20T22:47:49.230674+0000","last_became_peered":"2024-09-20T22:47:49.230674+0000","last_unstale":"2024-09-20T22:50:13.234462+0000","last_undegraded":"2024-09-20T22:50:13.234462+0000","last_fullsized":"2024-09-20T22:50:13.234462+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:47:41.411716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:47:41.411716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:13.855690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"74'11","reported_seq":287,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.233465+0000","last_change":"2024-09-20T22:48:44.023878+0000","last_active":"2024-09-20T22:50:13.233465+0000","last_peered":"2024-09-20T22:50:13.233465+0000","last_clean":"2024-09-20T22:50:13.233465+0000","last_became_active":"2024-09-20T22:48:05.249510+0000","last_became_peered":"2024-09-20T22:48:05.249510+0000","last_unstale":"2024-09-20T22:50:13.233465+0000","last_undegraded":"2024-09-20T22:50:13.233465+0000","last_fullsized":"2024-09-20T22:50:13.233465+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-09-20T22:48:44.023781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:48:44.023781+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":275,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.233560+0000","last_change":"2024-09-20T22:49:11.015881+0000","last_active":"2024-09-20T22:50:13.233560+0000","last_peered":"2024-09-20T22:50:13.233560+0000","last_clean":"2024-09-20T22:50:13.233560+0000","last_became_active":"2024-09-20T22:48:05.252026+0000","last_became_peered":"2024-09-20T22:48:05.252026+0000","last_unstale":"2024-09-20T22:50:13.233560+0000","last_undegraded":"2024-09-20T22:50:13.233560+0000","last_fullsized":"2024-09-20T22:50:13.233560+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:11.015797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:49:11.015797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:39.578934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":961,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.529045+0000","last_change":"2024-09-20T22:48:59.241704+0000","last_active":"2024-09-20T22:50:13.529045+0000","last_peered":"2024-09-20T22:50:13.529045+0000","last_clean":"2024-09-20T22:50:13.529045+0000","last_became_active":"2024-09-20T22:41:42.181736+0000","last_became_peered":"2024-09-20T22:41:42.181736+0000","last_unstale":"2024-09-20T22:50:13.529045+0000","last_undegraded":"2024-09-20T22:50:13.529045+0000","last_fullsized":"2024-09-20T22:50:13.529045+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:48:59.241606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:48:59.241606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:13.923480+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":960,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.253084+0000","last_change":"2024-09-20T22:47:49.216479+0000","last_active":"2024-09-20T22:50:13.253084+0000","last_peered":"2024-09-20T22:50:13.253084+0000","last_clean":"2024-09-20T22:50:13.253084+0000","last_became_active":"2024-09-20T22:47:49.216141+0000","last_became_peered":"2024-09-20T22:47:49.216141+0000","last_unstale":"2024-09-20T22:50:13.253084+0000","last_undegraded":"2024-09-20T22:50:13.253084+0000","last_fullsized":"2024-09-20T22:50:13.253084+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:47:36.066480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:47:36.066480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"404'13","reported_seq":993,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.529204+0000","last_change":"2024-09-20T22:49:25.147397+0000","last_active":"2024-09-20T22:50:13.529204+0000","last_peered":"2024-09-20T22:50:13.529204+0000","last_clean":"2024-09-20T22:50:13.529204+0000","last_became_active":"2024-09-20T22:41:40.164796+0000","last_became_peered":"2024-09-20T22:41:40.164796+0000","last_unstale":"2024-09-20T22:50:13.529204+0000","last_undegraded":"2024-09-20T22:50:13.529204+0000","last_fullsized":"2024-09-20T22:50:13.529204+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"404'13","last_scrub_stamp":"2024-09-20T22:49:25.147286+0000","last_deep_scrub":"404'13","last_deep_scrub_stamp":"2024-09-20T22:48:04.276282+0000","last_clean_scrub_stamp":"2024-09-20T22:49:25.147286+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:52.567714+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":1003,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.251966+0000","last_change":"2024-09-20T22:49:56.396435+0000","last_active":"2024-09-20T22:50:13.251966+0000","last_peered":"2024-09-20T22:50:13.251966+0000","last_clean":"2024-09-20T22:50:13.251966+0000","last_became_active":"2024-09-20T22:48:05.238812+0000","last_became_peered":"2024-09-20T22:48:05.238812+0000","last_unstale":"2024-09-20T22:50:13.251966+0000","last_undegraded":"2024-09-20T22:50:13.251966+0000","last_fullsized":"2024-09-20T22:50:13.251966+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:56.396347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:31.138067+0000","last_clean_scrub_stamp":"2024-09-20T22:49:56.396347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:51:17.230220+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":964,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.252946+0000","last_change":"2024-09-20T22:48:38.200896+0000","last_active":"2024-09-20T22:50:13.252946+0000","last_peered":"2024-09-20T22:50:13.252946+0000","last_clean":"2024-09-20T22:50:13.252946+0000","last_became_active":"2024-09-20T22:47:49.218708+0000","last_became_peered":"2024-09-20T22:47:49.218708+0000","last_unstale":"2024-09-20T22:50:13.252946+0000","last_undegraded":"2024-09-20T22:50:13.252946+0000","last_fullsized":"2024-09-20T22:50:13.252946+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:48:38.200797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:48:38.200797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":300,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.232800+0000","last_change":"2024-09-20T22:47:49.225362+0000","last_active":"2024-09-20T22:50:13.232800+0000","last_peered":"2024-09-20T22:50:13.232800+0000","last_clean":"2024-09-20T22:50:13.232800+0000","last_became_active":"2024-09-20T22:47:49.225027+0000","last_became_peered":"2024-09-20T22:47:49.225027+0000","last_unstale":"2024-09-20T22:50:13.232800+0000","last_undegraded":"2024-09-20T22:50:13.232800+0000","last_fullsized":"2024-09-20T22:50:13.232800+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:47:37.309842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:47:37.309842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:13.855704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"470'16","reported_seq":779,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.527857+0000","last_change":"2024-09-20T22:49:10.659945+0000","last_active":"2024-09-20T22:50:13.527857+0000","last_peered":"2024-09-20T22:50:13.527857+0000","last_clean":"2024-09-20T22:50:13.527857+0000","last_became_active":"2024-09-20T22:44:09.170785+0000","last_became_peered":"2024-09-20T22:44:09.170785+0000","last_unstale":"2024-09-20T22:50:13.527857+0000","last_undegraded":"2024-09-20T22:50:13.527857+0000","last_fullsized":"2024-09-20T22:50:13.527857+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"470'16","last_scrub_stamp":"2024-09-20T22:49:10.659868+0000","last_deep_scrub":"470'16","last_deep_scrub_stamp":"2024-09-20T22:49:10.659868+0000","last_clean_scrub_stamp":"2024-09-20T22:49:10.659868+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":969,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.528652+0000","last_change":"2024-09-20T22:48:02.512192+0000","last_active":"2024-09-20T22:50:13.528652+0000","last_peered":"2024-09-20T22:50:13.528652+0000","last_clean":"2024-09-20T22:50:13.528652+0000","last_became_active":"2024-09-20T22:41:38.158701+0000","last_became_peered":"2024-09-20T22:41:38.158701+0000","last_unstale":"2024-09-20T22:50:13.528652+0000","last_undegraded":"2024-09-20T22:50:13.528652+0000","last_fullsized":"2024-09-20T22:50:13.528652+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:48:02.512093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:45:24.765414+0000","last_clean_scrub_stamp":"2024-09-20T22:48:02.512093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":984,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.250998+0000","last_change":"2024-09-20T22:49:11.373441+0000","last_active":"2024-09-20T22:50:13.250998+0000","last_peered":"2024-09-20T22:50:13.250998+0000","last_clean":"2024-09-20T22:50:13.250998+0000","last_became_active":"2024-09-20T22:48:05.238334+0000","last_became_peered":"2024-09-20T22:48:05.238334+0000","last_unstale":"2024-09-20T22:50:13.250998+0000","last_undegraded":"2024-09-20T22:50:13.250998+0000","last_fullsized":"2024-09-20T22:50:13.250998+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:11.373370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:50.174861+0000","last_clean_scrub_stamp":"2024-09-20T22:49:11.373370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:14.536205+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":960,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.253019+0000","last_change":"2024-09-20T22:49:04.034724+0000","last_active":"2024-09-20T22:50:13.253019+0000","last_peered":"2024-09-20T22:50:13.253019+0000","last_clean":"2024-09-20T22:50:13.253019+0000","last_became_active":"2024-09-20T22:47:49.214487+0000","last_became_peered":"2024-09-20T22:47:49.214487+0000","last_unstale":"2024-09-20T22:50:13.253019+0000","last_undegraded":"2024-09-20T22:50:13.253019+0000","last_fullsized":"2024-09-20T22:50:13.253019+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:04.034601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:49:04.034601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:18.245352+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"74'4","reported_seq":962,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.251980+0000","last_change":"2024-09-20T22:48:23.236306+0000","last_active":"2024-09-20T22:50:13.251980+0000","last_peered":"2024-09-20T22:50:13.251980+0000","last_clean":"2024-09-20T22:50:13.251980+0000","last_became_active":"2024-09-20T22:41:40.166123+0000","last_became_peered":"2024-09-20T22:41:40.166123+0000","last_unstale":"2024-09-20T22:50:13.251980+0000","last_undegraded":"2024-09-20T22:50:13.251980+0000","last_fullsized":"2024-09-20T22:50:13.251980+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'4","last_scrub_stamp":"2024-09-20T22:48:23.236227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:48:23.236227+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":308,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.235053+0000","last_change":"2024-09-20T22:48:18.432371+0000","last_active":"2024-09-20T22:50:13.235053+0000","last_peered":"2024-09-20T22:50:13.235053+0000","last_clean":"2024-09-20T22:50:13.235053+0000","last_became_active":"2024-09-20T22:47:49.231964+0000","last_became_peered":"2024-09-20T22:47:49.231964+0000","last_unstale":"2024-09-20T22:50:13.235053+0000","last_undegraded":"2024-09-20T22:50:13.235053+0000","last_fullsized":"2024-09-20T22:50:13.235053+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:48:18.432271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:48:18.432271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:13.855750+0000","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":992,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.247961+0000","last_change":"2024-09-20T22:49:07.125306+0000","last_active":"2024-09-20T22:50:13.247961+0000","last_peered":"2024-09-20T22:50:13.247961+0000","last_clean":"2024-09-20T22:50:13.247961+0000","last_became_active":"2024-09-20T22:48:05.244659+0000","last_became_peered":"2024-09-20T22:48:05.244659+0000","last_unstale":"2024-09-20T22:50:13.247961+0000","last_undegraded":"2024-09-20T22:50:13.247961+0000","last_fullsized":"2024-09-20T22:50:13.247961+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:07.125231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:49:07.125231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:23.332189+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":976,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.251595+0000","last_change":"2024-09-20T22:49:57.405922+0000","last_active":"2024-09-20T22:50:13.251595+0000","last_peered":"2024-09-20T22:50:13.251595+0000","last_clean":"2024-09-20T22:50:13.251595+0000","last_became_active":"2024-09-20T22:48:05.237765+0000","last_became_peered":"2024-09-20T22:48:05.237765+0000","last_unstale":"2024-09-20T22:50:13.251595+0000","last_undegraded":"2024-09-20T22:50:13.251595+0000","last_fullsized":"2024-09-20T22:50:13.251595+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:57.405785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:10.233487+0000","last_clean_scrub_stamp":"2024-09-20T22:49:57.405785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:51:00.718232+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":969,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.528188+0000","last_change":"2024-09-20T22:49:05.205734+0000","last_active":"2024-09-20T22:50:13.528188+0000","last_peered":"2024-09-20T22:50:13.528188+0000","last_clean":"2024-09-20T22:50:13.528188+0000","last_became_active":"2024-09-20T22:48:05.249492+0000","last_became_peered":"2024-09-20T22:48:05.249492+0000","last_unstale":"2024-09-20T22:50:13.528188+0000","last_undegraded":"2024-09-20T22:50:13.528188+0000","last_fullsized":"2024-09-20T22:50:13.528188+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:05.205665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:49:05.205665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:32.772495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":986,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.251711+0000","last_change":"2024-09-20T22:48:36.246135+0000","last_active":"2024-09-20T22:50:13.251711+0000","last_peered":"2024-09-20T22:50:13.251711+0000","last_clean":"2024-09-20T22:50:13.251711+0000","last_became_active":"2024-09-20T22:47:49.217064+0000","last_became_peered":"2024-09-20T22:47:49.217064+0000","last_unstale":"2024-09-20T22:50:13.251711+0000","last_undegraded":"2024-09-20T22:50:13.251711+0000","last_fullsized":"2024-09-20T22:50:13.251711+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:48:36.246069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:48:36.246069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"74'11","reported_seq":995,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.249449+0000","last_change":"2024-09-20T22:49:10.137213+0000","last_active":"2024-09-20T22:50:13.249449+0000","last_peered":"2024-09-20T22:50:13.249449+0000","last_clean":"2024-09-20T22:50:13.249449+0000","last_became_active":"2024-09-20T22:41:40.171256+0000","last_became_peered":"2024-09-20T22:41:40.171256+0000","last_unstale":"2024-09-20T22:50:13.249449+0000","last_undegraded":"2024-09-20T22:50:13.249449+0000","last_fullsized":"2024-09-20T22:50:13.249449+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-09-20T22:49:10.137128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:49:10.137128+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:16.411613+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":308,"reported_epoch":533,"state":"active+clean","last_fresh":"2024-09-20T22:50:13.234567+0000","last_change":"2024-09-20T22:49:23.660691+0000","last_active":"2024-09-20T22:50:13.234567+0000","last_peered":"2024-09-20T22:50:13.234567+0000","last_clean":"2024-09-20T22:50:13.234567+0000","last_became_active":"2024-09-20T22:47:49.220971+0000","last_became_peered":"2024-09-20T22:47:49.220971+0000","last_unstale":"2024-09-20T22:50:13.234567+0000","last_undegraded":"2024-09-20T22:50:13.234567+0000","last_fullsized":"2024-09-20T22:50:13.234567+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:23.660625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:51.667341+0000","last_clean_scrub_stamp":"2024-09-20T22:49:23.660625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:38.664121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":567872353,"num_objects":261,"num_object_clones":212,"num_object_copies":783,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":261,"num_whiteouts":19,"num_read":3617,"num_read_kb":3794461,"num_write":2553,"num_write_kb":937523,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":613,"num_bytes_recovered":302968954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1153294336,"data_stored":1703504595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5105,"ondisk_log_size":5105,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":82,"ondisk_log_size":82,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":682,"num_read_kb":439,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":238,"num_bytes_recovered":5126,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":452,"ondisk_log_size":452,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":1034,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":506,"num_read_kb":501,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":384,"ondisk_log_size":384,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070571,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":174232,"kb_used_data":116676,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":93570920,"statfs":{"total":95995035648,"available":95816622080,"internally_reserved":0,"allocated":119476224,"data_stored":168988650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":58915274},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430321,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":210460,"kb_used_data":152904,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":93534692,"statfs":{"total":95995035648,"available":95779524608,"internally_reserved":0,"allocated":156573696,"data_stored":231352110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691958,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":137896,"kb_used_data":106700,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93607256,"statfs":{"total":95995035648,"available":95853830144,"internally_reserved":0,"allocated":109260800,"data_stored":168206833,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986299,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":146276,"kb_used_data":106768,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93598876,"statfs":{"total":95995035648,"available":95845249024,"internally_reserved":0,"allocated":109330432,"data_stored":155859887,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182529,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":277616,"kb_used_data":220060,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":93467536,"statfs":{"total":95995035648,"available":95710756864,"internally_reserved":0,"allocated":225341440,"data_stored":318502608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":58915271},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411462,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":173036,"kb_used_data":119632,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93572116,"statfs":{"total":95995035648,"available":95817846784,"internally_reserved":0,"allocated":122503168,"data_stored":183218956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574987,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":216100,"kb_used_data":162700,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93529052,"statfs":{"total":95995035648,"available":95773749248,"internally_reserved":0,"allocated":166604800,"data_stored":241736252,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738512,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":227808,"kb_used_data":174416,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93517344,"statfs":{"total":95995035648,"available":95761760256,"internally_reserved":0,"allocated":178601984,"data_stored":256412893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":174297088,"data_stored":253812881,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":162271232,"data_stored":239135905,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":117940224,"data_stored":180349247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":221343744,"data_stored":316214317,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":105033728,"data_stored":153267850,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":104374272,"data_stored":165015785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":153198592,"data_stored":229654955,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":114835456,"data_stored":166053655,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-20T22:50:20.516 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd pool set .mgr pgp_num 11 2024-09-20T22:50:20.569 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:50:20.571 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:20 smithi084 ceph-mon[40024]: 5.7 scrub starts 2024-09-20T22:50:20.571 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:20 smithi084 ceph-mon[40024]: 5.7 scrub ok 2024-09-20T22:50:20.571 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:20 smithi084 ceph-mon[40024]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:50:20.571 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:20 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/1680964518' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:50:20.571 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:20 smithi084 ceph-mon[40024]: osdmap e540: 8 total, 8 up, 8 in 2024-09-20T22:50:20.571 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:20 smithi084 ceph-mon[29774]: 5.7 scrub starts 2024-09-20T22:50:20.572 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:20 smithi084 ceph-mon[29774]: 5.7 scrub ok 2024-09-20T22:50:20.572 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:20 smithi084 ceph-mon[29774]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:50:20.572 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:20 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/1680964518' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:50:20.572 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:20 smithi084 ceph-mon[29774]: osdmap e540: 8 total, 8 up, 8 in 2024-09-20T22:50:20.610 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:50:20.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:20.633 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:20.634 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:50:20.690 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:20 smithi154 ceph-mon[33264]: 5.7 scrub starts 2024-09-20T22:50:20.690 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:20 smithi154 ceph-mon[33264]: 5.7 scrub ok 2024-09-20T22:50:20.690 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:20 smithi154 ceph-mon[33264]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:50:20.690 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:20 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/1680964518' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:50:20.690 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:20 smithi154 ceph-mon[33264]: osdmap e540: 8 total, 8 up, 8 in 2024-09-20T22:50:20.734 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:50:20.761 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:20.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:20.762 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:50:20.862 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:50:20.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:20.885 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:20.885 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:50:20.986 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:50:21.008 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:21.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:21.009 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:50:21.084 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-20T22:50:21.084 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:50:21.109 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:50:21.131 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:21.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:21.132 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:50:21.144 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd set noscrub 2024-09-20T22:50:21.232 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:50:21.238 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:50:21.243 INFO:tasks.rados.rados.0.smithi084.stdout:2119: done (2 left) 2024-09-20T22:50:21.243 INFO:tasks.rados.rados.0.smithi084.stdout:2120: read oid 5 snap -1 2024-09-20T22:50:21.244 INFO:tasks.rados.rados.0.smithi084.stdout:2120: expect deleted 2024-09-20T22:50:21.244 INFO:tasks.rados.rados.0.smithi084.stdout:2121: snap_create 2024-09-20T22:50:21.244 INFO:tasks.rados.rados.0.smithi084.stdout:2117: finishing write tid 2 to smithi08477203-33 2024-09-20T22:50:21.244 INFO:tasks.rados.rados.0.smithi084.stdout:2117: finishing write tid 3 to smithi08477203-33 2024-09-20T22:50:21.244 INFO:tasks.rados.rados.0.smithi084.stdout:2117: finishing write tid 4 to smithi08477203-33 2024-09-20T22:50:21.244 INFO:tasks.rados.rados.0.smithi084.stdout:2117: oid 33 updating version 0 to 738 2024-09-20T22:50:21.244 INFO:tasks.rados.rados.0.smithi084.stdout:2117: oid 33 updating version 738 to 740 2024-09-20T22:50:21.244 INFO:tasks.rados.rados.0.smithi084.stdout:2117: oid 33 version 740 is already newer than 739 2024-09-20T22:50:21.244 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 33 v 740 (ObjNum 738 snap 235 seq_num 738) dirty exists 2024-09-20T22:50:21.244 INFO:tasks.rados.rados.0.smithi084.stdout:2117: left oid 33 (ObjNum 738 snap 235 seq_num 738) 2024-09-20T22:50:21.244 INFO:tasks.rados.rados.0.smithi084.stdout:2118: finishing rollback tid 0 to smithi08477203-4 2024-09-20T22:50:21.245 INFO:tasks.rados.rados.0.smithi084.stdout:2118: finishing rollback tid 1 to smithi08477203-4 2024-09-20T22:50:21.245 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 4 v 1445 (ObjNum 49 snap 0 seq_num 2085656144) dirty dne 2024-09-20T22:50:21.266 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:21.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:21.267 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:50:21.368 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:50:21.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:21.390 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:21.390 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:50:21.491 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:50:21.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:21.513 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:21.513 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:50:21.515 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:21 smithi084 ceph-mon[40024]: 1.a scrub starts 2024-09-20T22:50:21.515 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:21 smithi084 ceph-mon[40024]: 1.a scrub ok 2024-09-20T22:50:21.515 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:21 smithi084 ceph-mon[40024]: pgmap v711: 158 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 153 active+clean; 493 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 2.7 MiB/s rd, 913 KiB/s wr, 9 op/s 2024-09-20T22:50:21.515 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:21 smithi084 ceph-mon[40024]: 6.11 deep-scrub starts 2024-09-20T22:50:21.515 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:21 smithi084 ceph-mon[40024]: 6.11 deep-scrub ok 2024-09-20T22:50:21.515 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:21 smithi084 ceph-mon[40024]: osdmap e541: 8 total, 8 up, 8 in 2024-09-20T22:50:21.515 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:21 smithi084 ceph-mon[29774]: 1.a scrub starts 2024-09-20T22:50:21.515 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:21 smithi084 ceph-mon[29774]: 1.a scrub ok 2024-09-20T22:50:21.515 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:21 smithi084 ceph-mon[29774]: pgmap v711: 158 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 153 active+clean; 493 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 2.7 MiB/s rd, 913 KiB/s wr, 9 op/s 2024-09-20T22:50:21.515 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:21 smithi084 ceph-mon[29774]: 6.11 deep-scrub starts 2024-09-20T22:50:21.515 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:21 smithi084 ceph-mon[29774]: 6.11 deep-scrub ok 2024-09-20T22:50:21.515 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:21 smithi084 ceph-mon[29774]: osdmap e541: 8 total, 8 up, 8 in 2024-09-20T22:50:21.614 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:50:21.635 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:21.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:21.636 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:50:21.680 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:21 smithi154 ceph-mon[33264]: 1.a scrub starts 2024-09-20T22:50:21.680 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:21 smithi154 ceph-mon[33264]: 1.a scrub ok 2024-09-20T22:50:21.680 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:21 smithi154 ceph-mon[33264]: pgmap v711: 158 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 153 active+clean; 493 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 2.7 MiB/s rd, 913 KiB/s wr, 9 op/s 2024-09-20T22:50:21.680 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:21 smithi154 ceph-mon[33264]: 6.11 deep-scrub starts 2024-09-20T22:50:21.680 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:21 smithi154 ceph-mon[33264]: 6.11 deep-scrub ok 2024-09-20T22:50:21.680 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:21 smithi154 ceph-mon[33264]: osdmap e541: 8 total, 8 up, 8 in 2024-09-20T22:50:21.707 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-20T22:50:21.737 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:50:21.759 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:21.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:21.760 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:50:21.831 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:50:21.861 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:50:21.883 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:21.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:21.884 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:50:21.952 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:50:21.985 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:50:22.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:22.007 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:22.007 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:50:22.108 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:50:22.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:22.129 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:22.130 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:50:22.230 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:50:22.243 INFO:teuthology.orchestra.run.smithi084.stderr:set pool 1 pgp_num to 11 2024-09-20T22:50:22.244 INFO:tasks.rados.rados.0.smithi084.stdout:2117: done (3 left) 2024-09-20T22:50:22.244 INFO:tasks.rados.rados.0.smithi084.stdout:2118: done (2 left) 2024-09-20T22:50:22.244 INFO:tasks.rados.rados.0.smithi084.stdout:2120: done (1 left) 2024-09-20T22:50:22.244 INFO:tasks.rados.rados.0.smithi084.stdout:2121: done (0 left) 2024-09-20T22:50:22.244 INFO:tasks.rados.rados.0.smithi084.stdout:2122: read oid 49 snap -1 2024-09-20T22:50:22.244 INFO:tasks.rados.rados.0.smithi084.stdout:2122: expect (ObjNum 735 snap 232 seq_num 735) 2024-09-20T22:50:22.244 INFO:tasks.rados.rados.0.smithi084.stdout:2123: write oid 40 current snap is 236 2024-09-20T22:50:22.245 INFO:tasks.rados.rados.0.smithi084.stdout:2123: seq_num 739 ranges {420373=495252,1444546=690576} 2024-09-20T22:50:22.251 INFO:tasks.rados.rados.0.smithi084.stdout:2123: writing smithi08477203-40 from 420373 to 915625 tid 1 2024-09-20T22:50:22.255 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:22.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:22.256 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:50:22.256 INFO:tasks.rados.rados.0.smithi084.stdout:2123: writing smithi08477203-40 from 1444546 to 2135122 tid 2 2024-09-20T22:50:22.256 INFO:tasks.rados.rados.0.smithi084.stdout:2124: write (excl) oid 45 current snap is 236 2024-09-20T22:50:22.257 INFO:tasks.rados.rados.0.smithi084.stdout:2124: seq_num 740 ranges {437180=570501,1503028=550587} 2024-09-20T22:50:22.262 INFO:tasks.rados.rados.0.smithi084.stdout:2124: writing smithi08477203-45 from 437180 to 1007681 tid 1 2024-09-20T22:50:22.269 INFO:tasks.rados.rados.0.smithi084.stdout:2124: writing smithi08477203-45 from 1503028 to 2053615 tid 2 2024-09-20T22:50:22.269 INFO:tasks.rados.rados.0.smithi084.stdout:2125: snap_create 2024-09-20T22:50:22.269 INFO:tasks.rados.rados.0.smithi084.stdout:2122: expect (ObjNum 735 snap 232 seq_num 735) 2024-09-20T22:50:22.357 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:50:22.379 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:22.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:22.380 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:50:22.381 INFO:tasks.rados.rados.0.smithi084.stdout:2123: finishing write tid 1 to smithi08477203-40 2024-09-20T22:50:22.381 INFO:tasks.rados.rados.0.smithi084.stdout:2123: finishing write tid 2 to smithi08477203-40 2024-09-20T22:50:22.381 INFO:tasks.rados.rados.0.smithi084.stdout:2123: finishing write tid 3 to smithi08477203-40 2024-09-20T22:50:22.381 INFO:tasks.rados.rados.0.smithi084.stdout:2123: finishing write tid 4 to smithi08477203-40 2024-09-20T22:50:22.382 INFO:tasks.rados.rados.0.smithi084.stdout:2123: oid 40 updating version 0 to 741 2024-09-20T22:50:22.382 INFO:tasks.rados.rados.0.smithi084.stdout:2123: oid 40 updating version 741 to 742 2024-09-20T22:50:22.382 INFO:tasks.rados.rados.0.smithi084.stdout:2123: oid 40 updating version 742 to 743 2024-09-20T22:50:22.382 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 40 v 743 (ObjNum 739 snap 236 seq_num 739) dirty exists 2024-09-20T22:50:22.382 INFO:tasks.rados.rados.0.smithi084.stdout:2123: left oid 40 (ObjNum 739 snap 236 seq_num 739) 2024-09-20T22:50:22.382 INFO:tasks.rados.rados.0.smithi084.stdout:2124: finishing write tid 1 to smithi08477203-45 2024-09-20T22:50:22.382 INFO:tasks.rados.rados.0.smithi084.stdout:2124: finishing write tid 2 to smithi08477203-45 2024-09-20T22:50:22.382 INFO:tasks.rados.rados.0.smithi084.stdout:2124: finishing write tid 3 to smithi08477203-45 2024-09-20T22:50:22.382 INFO:tasks.rados.rados.0.smithi084.stdout:2124: finishing write tid 4 to smithi08477203-45 2024-09-20T22:50:22.382 INFO:tasks.rados.rados.0.smithi084.stdout:2124: oid 45 updating version 0 to 726 2024-09-20T22:50:22.382 INFO:tasks.rados.rados.0.smithi084.stdout:2124: oid 45 version 726 is already newer than 725 2024-09-20T22:50:22.382 INFO:tasks.rados.rados.0.smithi084.stdout:2124: oid 45 version 726 is already newer than 724 2024-09-20T22:50:22.382 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 45 v 726 (ObjNum 740 snap 236 seq_num 740) dirty exists 2024-09-20T22:50:22.383 INFO:tasks.rados.rados.0.smithi084.stdout:2124: left oid 45 (ObjNum 740 snap 236 seq_num 740) 2024-09-20T22:50:22.409 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:22 smithi154 ceph-mon[33264]: 1.1 scrub starts 2024-09-20T22:50:22.409 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:22 smithi154 ceph-mon[33264]: 1.1 scrub ok 2024-09-20T22:50:22.409 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:22 smithi154 ceph-mon[33264]: 2.0 scrub starts 2024-09-20T22:50:22.409 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:22 smithi154 ceph-mon[33264]: 2.0 scrub ok 2024-09-20T22:50:22.410 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:22 smithi154 ceph-mon[33264]: 6.1 scrub starts 2024-09-20T22:50:22.410 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:22 smithi154 ceph-mon[33264]: 6.1 scrub ok 2024-09-20T22:50:22.410 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:22 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/4164710010' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]: dispatch 2024-09-20T22:50:22.410 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:22 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]: dispatch 2024-09-20T22:50:22.410 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:22 smithi154 ceph-mon[33264]: 6.17 scrub starts 2024-09-20T22:50:22.410 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:22 smithi154 ceph-mon[33264]: 6.17 scrub ok 2024-09-20T22:50:22.410 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:22 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]': finished 2024-09-20T22:50:22.410 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:22 smithi154 ceph-mon[33264]: osdmap e542: 8 total, 8 up, 8 in 2024-09-20T22:50:22.480 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:50:22.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:22.504 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:22.504 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:50:22.509 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-20T22:50:22.509 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:50:22.568 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:22 smithi084 ceph-mon[40024]: 1.1 scrub starts 2024-09-20T22:50:22.568 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:22 smithi084 ceph-mon[40024]: 1.1 scrub ok 2024-09-20T22:50:22.568 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:22 smithi084 ceph-mon[40024]: 2.0 scrub starts 2024-09-20T22:50:22.568 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:22 smithi084 ceph-mon[40024]: 2.0 scrub ok 2024-09-20T22:50:22.568 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:22 smithi084 ceph-mon[40024]: 6.1 scrub starts 2024-09-20T22:50:22.568 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:22 smithi084 ceph-mon[40024]: 6.1 scrub ok 2024-09-20T22:50:22.568 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:22 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/4164710010' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]: dispatch 2024-09-20T22:50:22.568 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:22 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]: dispatch 2024-09-20T22:50:22.568 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:22 smithi084 ceph-mon[40024]: 6.17 scrub starts 2024-09-20T22:50:22.568 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:22 smithi084 ceph-mon[40024]: 6.17 scrub ok 2024-09-20T22:50:22.568 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:22 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]': finished 2024-09-20T22:50:22.568 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:22 smithi084 ceph-mon[40024]: osdmap e542: 8 total, 8 up, 8 in 2024-09-20T22:50:22.569 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:22 smithi084 ceph-mon[29774]: 1.1 scrub starts 2024-09-20T22:50:22.569 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:22 smithi084 ceph-mon[29774]: 1.1 scrub ok 2024-09-20T22:50:22.569 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:22 smithi084 ceph-mon[29774]: 2.0 scrub starts 2024-09-20T22:50:22.569 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:22 smithi084 ceph-mon[29774]: 2.0 scrub ok 2024-09-20T22:50:22.569 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:22 smithi084 ceph-mon[29774]: 6.1 scrub starts 2024-09-20T22:50:22.569 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:22 smithi084 ceph-mon[29774]: 6.1 scrub ok 2024-09-20T22:50:22.569 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:22 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/4164710010' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]: dispatch 2024-09-20T22:50:22.569 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:22 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]: dispatch 2024-09-20T22:50:22.569 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:22 smithi084 ceph-mon[29774]: 6.17 scrub starts 2024-09-20T22:50:22.569 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:22 smithi084 ceph-mon[29774]: 6.17 scrub ok 2024-09-20T22:50:22.569 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:22 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]': finished 2024-09-20T22:50:22.569 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:22 smithi084 ceph-mon[29774]: osdmap e542: 8 total, 8 up, 8 in 2024-09-20T22:50:22.605 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:50:22.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:22.627 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:22.627 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:50:22.728 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:50:22.749 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:22.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:22.750 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:50:22.851 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:50:22.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:22.873 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:22.873 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:50:22.974 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:50:22.996 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:22.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:22.997 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:50:23.098 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:50:23.099 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.7 dump_blocked_ops 2024-09-20T22:50:23.120 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:23.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:23.121 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:50:23.207 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-20T22:50:23.222 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:50:23.235 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is set 2024-09-20T22:50:23.241 INFO:tasks.rados.rados.0.smithi084.stdout:2122: done (3 left) 2024-09-20T22:50:23.242 INFO:tasks.rados.rados.0.smithi084.stdout:2123: done (2 left) 2024-09-20T22:50:23.242 INFO:tasks.rados.rados.0.smithi084.stdout:2124: done (1 left) 2024-09-20T22:50:23.242 INFO:tasks.rados.rados.0.smithi084.stdout:2125: done (0 left) 2024-09-20T22:50:23.242 INFO:tasks.rados.rados.0.smithi084.stdout:2126: snap_remove snap 231 2024-09-20T22:50:23.262 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:23.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:23.263 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:50:23.352 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:50:23.364 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:50:23.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:23.386 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:23.386 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:50:23.487 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:50:23.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:23.510 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:23.510 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:50:23.592 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:23 smithi154 ceph-mon[33264]: pgmap v714: 158 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 153 active+clean; 493 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 2.7 MiB/s rd, 913 KiB/s wr, 9 op/s 2024-09-20T22:50:23.592 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:23 smithi154 ceph-mon[33264]: 6.16 deep-scrub starts 2024-09-20T22:50:23.592 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:23 smithi154 ceph-mon[33264]: 6.16 deep-scrub ok 2024-09-20T22:50:23.592 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:23 smithi154 ceph-mon[33264]: 3.7 scrub starts 2024-09-20T22:50:23.592 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:23 smithi154 ceph-mon[33264]: 3.7 scrub ok 2024-09-20T22:50:23.592 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:23 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/3941313540' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-20T22:50:23.592 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:23 smithi154 ceph-mon[33264]: 5.16 scrub starts 2024-09-20T22:50:23.592 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:23 smithi154 ceph-mon[33264]: 5.16 scrub ok 2024-09-20T22:50:23.592 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:23 smithi154 ceph-mon[33264]: 5.e scrub starts 2024-09-20T22:50:23.593 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:23 smithi154 ceph-mon[33264]: 5.e scrub ok 2024-09-20T22:50:23.593 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:23 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:50:23.593 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:23 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:50:23.593 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:23 smithi154 ceph-mon[33264]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:50:23.593 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:23 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/3941313540' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T22:50:23.593 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:23 smithi154 ceph-mon[33264]: osdmap e543: 8 total, 8 up, 8 in 2024-09-20T22:50:23.611 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:50:23.614 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:23 smithi084 ceph-mon[40024]: pgmap v714: 158 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 153 active+clean; 493 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 2.7 MiB/s rd, 913 KiB/s wr, 9 op/s 2024-09-20T22:50:23.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:23 smithi084 ceph-mon[40024]: 6.16 deep-scrub starts 2024-09-20T22:50:23.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:23 smithi084 ceph-mon[40024]: 6.16 deep-scrub ok 2024-09-20T22:50:23.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:23 smithi084 ceph-mon[40024]: 3.7 scrub starts 2024-09-20T22:50:23.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:23 smithi084 ceph-mon[40024]: 3.7 scrub ok 2024-09-20T22:50:23.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:23 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/3941313540' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-20T22:50:23.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:23 smithi084 ceph-mon[40024]: 5.16 scrub starts 2024-09-20T22:50:23.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:23 smithi084 ceph-mon[40024]: 5.16 scrub ok 2024-09-20T22:50:23.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:23 smithi084 ceph-mon[40024]: 5.e scrub starts 2024-09-20T22:50:23.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:23 smithi084 ceph-mon[40024]: 5.e scrub ok 2024-09-20T22:50:23.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:23 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:50:23.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:23 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:50:23.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:23 smithi084 ceph-mon[40024]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:50:23.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:23 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/3941313540' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T22:50:23.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:23 smithi084 ceph-mon[40024]: osdmap e543: 8 total, 8 up, 8 in 2024-09-20T22:50:23.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:23 smithi084 ceph-mon[29774]: pgmap v714: 158 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 153 active+clean; 493 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 2.7 MiB/s rd, 913 KiB/s wr, 9 op/s 2024-09-20T22:50:23.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:23 smithi084 ceph-mon[29774]: 6.16 deep-scrub starts 2024-09-20T22:50:23.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:23 smithi084 ceph-mon[29774]: 6.16 deep-scrub ok 2024-09-20T22:50:23.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:23 smithi084 ceph-mon[29774]: 3.7 scrub starts 2024-09-20T22:50:23.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:23 smithi084 ceph-mon[29774]: 3.7 scrub ok 2024-09-20T22:50:23.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:23 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/3941313540' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-20T22:50:23.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:23 smithi084 ceph-mon[29774]: 5.16 scrub starts 2024-09-20T22:50:23.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:23 smithi084 ceph-mon[29774]: 5.16 scrub ok 2024-09-20T22:50:23.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:23 smithi084 ceph-mon[29774]: 5.e scrub starts 2024-09-20T22:50:23.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:23 smithi084 ceph-mon[29774]: 5.e scrub ok 2024-09-20T22:50:23.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:23 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:50:23.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:23 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:50:23.617 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:23 smithi084 ceph-mon[29774]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:50:23.617 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:23 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/3941313540' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T22:50:23.617 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:23 smithi084 ceph-mon[29774]: osdmap e543: 8 total, 8 up, 8 in 2024-09-20T22:50:23.633 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:23.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:23.633 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:50:23.734 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:50:23.756 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:23.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:23.757 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:50:23.857 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:50:23.879 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:23.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:23.880 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:50:23.949 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-20T22:50:23.949 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:50:23.981 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:50:24.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:24.004 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:24.004 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:50:24.081 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:50:24.104 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:50:24.125 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:24.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:24.126 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:50:24.227 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:50:24.249 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:24.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:24.250 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:50:24.252 INFO:tasks.rados.rados.0.smithi084.stdout:2126: done (0 left) 2024-09-20T22:50:24.252 INFO:tasks.rados.rados.0.smithi084.stdout:2127: write oid 12 current snap is 237 2024-09-20T22:50:24.252 INFO:tasks.rados.rados.0.smithi084.stdout:2127: seq_num 741 ranges {453987=645750,1845157=608830} 2024-09-20T22:50:24.258 INFO:tasks.rados.rados.0.smithi084.stdout:2127: writing smithi08477203-12 from 453987 to 1099737 tid 1 2024-09-20T22:50:24.267 INFO:tasks.rados.rados.0.smithi084.stdout:2127: writing smithi08477203-12 from 1845157 to 2453987 tid 2 2024-09-20T22:50:24.267 INFO:tasks.rados.rados.0.smithi084.stdout:2128: write oid 40 current snap is 237 2024-09-20T22:50:24.267 INFO:tasks.rados.rados.0.smithi084.stdout:2127: finishing write tid 1 to smithi08477203-12 2024-09-20T22:50:24.267 INFO:tasks.rados.rados.0.smithi084.stdout:2128: seq_num 742 ranges {470794=720999,1903639=554256} 2024-09-20T22:50:24.273 INFO:tasks.rados.rados.0.smithi084.stdout:2128: writing smithi08477203-40 from 470794 to 1191793 tid 1 2024-09-20T22:50:24.281 INFO:tasks.rados.rados.0.smithi084.stdout:2128: writing smithi08477203-40 from 1903639 to 2457895 tid 2 2024-09-20T22:50:24.281 INFO:tasks.rados.rados.0.smithi084.stdout:2129: snap_remove snap 236 2024-09-20T22:50:24.350 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:50:24.371 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:24.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:24.372 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:50:24.473 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:50:24.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:24.495 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:24.495 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:50:24.505 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:24 smithi084 ceph-mon[40024]: 5.14 scrub starts 2024-09-20T22:50:24.506 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:24 smithi084 ceph-mon[40024]: osdmap e544: 8 total, 8 up, 8 in 2024-09-20T22:50:24.506 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:24 smithi084 ceph-mon[29774]: 5.14 scrub starts 2024-09-20T22:50:24.506 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:24 smithi084 ceph-mon[29774]: osdmap e544: 8 total, 8 up, 8 in 2024-09-20T22:50:24.579 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:24 smithi154 ceph-mon[33264]: 5.14 scrub starts 2024-09-20T22:50:24.580 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:24 smithi154 ceph-mon[33264]: osdmap e544: 8 total, 8 up, 8 in 2024-09-20T22:50:24.596 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:50:24.602 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:50:24.602 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:50:24.607 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.7 dump_historic_ops 2024-09-20T22:50:24.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:24.618 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:24.618 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:50:24.619 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:50:24.619 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:50:24.634 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:50:24.635 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:50:24.650 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-20T22:50:24.651 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T22:50:24.667 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:50:24.667 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:50:24.684 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:50:24.684 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:50:24.700 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:50:24.701 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:50:24.717 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:50:24.717 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:50:24.719 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:50:24.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:24.741 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:24.741 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:50:24.842 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:50:24.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:24.866 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:24.866 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:50:24.866 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:50:24.967 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:50:24.988 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:24.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:24.989 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:50:25.090 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:50:25.112 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:25.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:25.113 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:50:25.214 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:50:25.235 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:25.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:25.236 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:50:25.254 INFO:tasks.rados.rados.0.smithi084.stdout:2129: done (2 left) 2024-09-20T22:50:25.254 INFO:tasks.rados.rados.0.smithi084.stdout:2130: snap_remove snap 222 2024-09-20T22:50:25.337 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:50:25.344 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-20T22:50:25.345 INFO:teuthology.orchestra.run.smithi154.stdout: "size": 20, 2024-09-20T22:50:25.345 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 600, 2024-09-20T22:50:25.345 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [ 2024-09-20T22:50:25.345 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.345 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:786 7.17 7:ed54065f:::smithi08477203-12:head [copy-from ver 61] snapc 19=[17] ondisk+write+known_if_redirected+supports_pool_eio e145)", 2024-09-20T22:50:25.345 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:44.333196+0000", 2024-09-20T22:50:25.345 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 401.00904885300002, 2024-09-20T22:50:25.345 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.023328503, 2024-09-20T22:50:25.345 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:50:25.345 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:50:25.345 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:50:25.345 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:50:25.346 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:50:25.346 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:50:25.346 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 786 2024-09-20T22:50:25.346 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.346 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:50:25.346 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.346 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:50:25.346 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.333196+0000", 2024-09-20T22:50:25.346 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:50:25.346 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.346 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.346 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:50:25.346 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.333196+0000", 2024-09-20T22:50:25.346 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:50:25.347 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.347 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.347 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:50:25.347 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.333197+0000", 2024-09-20T22:50:25.347 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.486e-06 2024-09-20T22:50:25.347 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.347 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.347 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:50:25.347 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.333207+0000", 2024-09-20T22:50:25.347 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0307e-05 2024-09-20T22:50:25.347 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.347 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.347 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:50:25.348 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.333220+0000", 2024-09-20T22:50:25.348 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.306e-05 2024-09-20T22:50:25.348 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.348 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.348 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:50:25.348 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.333229+0000", 2024-09-20T22:50:25.348 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.3450000000000006e-06 2024-09-20T22:50:25.348 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.348 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.348 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:50:25.348 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.333292+0000", 2024-09-20T22:50:25.348 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.3137000000000002e-05 2024-09-20T22:50:25.349 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.349 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.350 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op must wait for map", 2024-09-20T22:50:25.350 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.333298+0000", 2024-09-20T22:50:25.350 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.7780000000000004e-06 2024-09-20T22:50:25.350 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.350 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.350 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:50:25.350 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.336639+0000", 2024-09-20T22:50:25.350 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0033414479999999999 2024-09-20T22:50:25.350 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.350 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.350 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:50:25.350 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.336729+0000", 2024-09-20T22:50:25.350 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.9888000000000004e-05 2024-09-20T22:50:25.350 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.351 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.351 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 2,5", 2024-09-20T22:50:25.351 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.344876+0000", 2024-09-20T22:50:25.351 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0081473989999999996 2024-09-20T22:50:25.351 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.351 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.351 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:50:25.351 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.355138+0000", 2024-09-20T22:50:25.351 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.010261462000000001 2024-09-20T22:50:25.351 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.351 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.351 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:50:25.351 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.355972+0000", 2024-09-20T22:50:25.351 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00083377899999999999 2024-09-20T22:50:25.352 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.352 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.352 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:50:25.352 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.356451+0000", 2024-09-20T22:50:25.352 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00047985200000000001 2024-09-20T22:50:25.352 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.352 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.352 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:50:25.352 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.356486+0000", 2024-09-20T22:50:25.352 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.4653000000000002e-05 2024-09-20T22:50:25.352 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.352 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.352 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:50:25.353 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.356524+0000", 2024-09-20T22:50:25.353 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.7908999999999999e-05 2024-09-20T22:50:25.353 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:50:25.353 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:50:25.353 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:50:25.353 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.353 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.353 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:787 7.17 7:ed54065f:::smithi08477203-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e145)", 2024-09-20T22:50:25.353 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:44.333248+0000", 2024-09-20T22:50:25.353 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 401.00899675800002, 2024-09-20T22:50:25.353 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.016881184, 2024-09-20T22:50:25.353 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:50:25.353 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:50:25.353 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:50:25.354 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:50:25.354 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:50:25.354 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:50:25.354 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 787 2024-09-20T22:50:25.354 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.354 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:50:25.354 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.354 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:50:25.354 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.333248+0000", 2024-09-20T22:50:25.354 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:50:25.354 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.354 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.354 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:50:25.354 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.333248+0000", 2024-09-20T22:50:25.355 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:50:25.355 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.355 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.355 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:50:25.355 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.333248+0000", 2024-09-20T22:50:25.355 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.1399999999999997e-07 2024-09-20T22:50:25.355 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.355 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.355 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:50:25.355 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.333250+0000", 2024-09-20T22:50:25.355 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.8199999999999999e-06 2024-09-20T22:50:25.355 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.355 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.355 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:50:25.356 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.333257+0000", 2024-09-20T22:50:25.356 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.43e-06 2024-09-20T22:50:25.356 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.356 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.356 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:50:25.356 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.333261+0000", 2024-09-20T22:50:25.356 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.3670000000000001e-06 2024-09-20T22:50:25.356 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.356 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.356 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:50:25.356 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.333351+0000", 2024-09-20T22:50:25.356 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.0486000000000001e-05 2024-09-20T22:50:25.356 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.356 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.357 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting_for_map not empty", 2024-09-20T22:50:25.357 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.333357+0000", 2024-09-20T22:50:25.357 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.3520000000000003e-06 2024-09-20T22:50:25.357 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.357 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.357 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:50:25.357 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.336912+0000", 2024-09-20T22:50:25.357 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.003554612 2024-09-20T22:50:25.357 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.357 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.357 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for blocked object", 2024-09-20T22:50:25.357 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.336957+0000", 2024-09-20T22:50:25.357 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.4919000000000001e-05 2024-09-20T22:50:25.357 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.358 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.358 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:50:25.358 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.349989+0000", 2024-09-20T22:50:25.358 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.01303238 2024-09-20T22:50:25.358 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.358 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.358 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:50:25.358 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.350058+0000", 2024-09-20T22:50:25.358 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.8418999999999997e-05 2024-09-20T22:50:25.358 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.358 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.358 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:50:25.358 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.350129+0000", 2024-09-20T22:50:25.358 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.0984999999999999e-05 2024-09-20T22:50:25.359 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:50:25.359 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:50:25.359 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:50:25.359 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.359 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.359 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:804 7.17 7:ea720b85:::smithi08477203-26:head [copy-from ver 61] snapc 19=[17] ondisk+write+known_if_redirected+supports_pool_eio e145)", 2024-09-20T22:50:25.359 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:44.451241+0000", 2024-09-20T22:50:25.359 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 400.89100292299997, 2024-09-20T22:50:25.359 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.012865619999999999, 2024-09-20T22:50:25.359 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:50:25.359 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:50:25.359 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:50:25.359 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:50:25.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:25.361 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:50:25.361 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:50:25.361 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 804 2024-09-20T22:50:25.361 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.361 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:50:25.361 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.361 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:50:25.361 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.451241+0000", 2024-09-20T22:50:25.362 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:50:25.362 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.362 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.362 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:50:25.362 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.451241+0000", 2024-09-20T22:50:25.362 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:50:25.362 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.362 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.362 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:50:25.362 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.451244+0000", 2024-09-20T22:50:25.362 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.2689999999999998e-06 2024-09-20T22:50:25.362 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.362 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.362 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:50:25.363 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.451255+0000", 2024-09-20T22:50:25.363 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1151e-05 2024-09-20T22:50:25.363 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.363 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.363 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:50:25.363 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.451274+0000", 2024-09-20T22:50:25.363 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9389999999999999e-05 2024-09-20T22:50:25.363 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.363 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.363 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:50:25.363 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.451286+0000", 2024-09-20T22:50:25.363 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1703e-05 2024-09-20T22:50:25.363 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.364 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.364 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:50:25.364 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.451361+0000", 2024-09-20T22:50:25.364 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.5310999999999998e-05 2024-09-20T22:50:25.364 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.364 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.364 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:50:25.364 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.451462+0000", 2024-09-20T22:50:25.364 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000100809 2024-09-20T22:50:25.364 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.364 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.364 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 2,5", 2024-09-20T22:50:25.364 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.456603+0000", 2024-09-20T22:50:25.364 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0051411169999999997 2024-09-20T22:50:25.364 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.365 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.365 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:50:25.365 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.463017+0000", 2024-09-20T22:50:25.365 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.006413307 2024-09-20T22:50:25.365 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.365 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.365 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:50:25.365 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.463754+0000", 2024-09-20T22:50:25.365 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00073758000000000005 2024-09-20T22:50:25.365 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.366 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.366 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:50:25.366 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.464036+0000", 2024-09-20T22:50:25.366 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00028171000000000001 2024-09-20T22:50:25.366 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.366 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.366 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:50:25.366 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.464076+0000", 2024-09-20T22:50:25.366 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.9736e-05 2024-09-20T22:50:25.366 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.367 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.367 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:50:25.367 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.464107+0000", 2024-09-20T22:50:25.367 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.1537e-05 2024-09-20T22:50:25.367 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:50:25.367 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:50:25.367 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:50:25.367 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.367 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.367 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:1527 7.17 7:ea720b85:::smithi08477203-26:head [copy-from ver 83] snapc 33=[33] ondisk+write+known_if_redirected+supports_pool_eio e197)", 2024-09-20T22:50:25.367 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:44:36.198921+0000", 2024-09-20T22:50:25.367 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 349.14332359899998, 2024-09-20T22:50:25.367 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.021621703999999999, 2024-09-20T22:50:25.367 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:50:25.368 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:50:25.368 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:50:25.368 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:50:25.368 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:50:25.368 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:50:25.368 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 1527 2024-09-20T22:50:25.368 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.368 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:50:25.368 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.368 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:50:25.368 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.198921+0000", 2024-09-20T22:50:25.368 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:50:25.368 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.368 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.368 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:50:25.369 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.198921+0000", 2024-09-20T22:50:25.369 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:50:25.369 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.369 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.369 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:50:25.369 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.198923+0000", 2024-09-20T22:50:25.369 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.0080000000000001e-06 2024-09-20T22:50:25.369 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.369 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.369 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:50:25.369 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.198934+0000", 2024-09-20T22:50:25.369 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0923e-05 2024-09-20T22:50:25.369 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.369 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.369 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:50:25.370 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.198955+0000", 2024-09-20T22:50:25.370 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.0806e-05 2024-09-20T22:50:25.370 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.370 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.370 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:50:25.370 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.198966+0000", 2024-09-20T22:50:25.370 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1779e-05 2024-09-20T22:50:25.370 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.370 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.370 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:50:25.371 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.199051+0000", 2024-09-20T22:50:25.371 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.4357999999999994e-05 2024-09-20T22:50:25.371 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.371 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.371 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op must wait for map", 2024-09-20T22:50:25.371 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.199058+0000", 2024-09-20T22:50:25.371 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.8530000000000003e-06 2024-09-20T22:50:25.371 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.371 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.371 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:50:25.371 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.202420+0000", 2024-09-20T22:50:25.371 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0033623350000000002 2024-09-20T22:50:25.371 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.371 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.372 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:50:25.372 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.202498+0000", 2024-09-20T22:50:25.372 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.8316e-05 2024-09-20T22:50:25.372 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.372 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.372 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 2,5", 2024-09-20T22:50:25.372 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.209140+0000", 2024-09-20T22:50:25.372 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0066415759999999997 2024-09-20T22:50:25.372 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.372 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.372 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:50:25.372 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.214931+0000", 2024-09-20T22:50:25.372 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0057916520000000004 2024-09-20T22:50:25.372 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.373 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.373 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:50:25.373 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.220009+0000", 2024-09-20T22:50:25.373 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0050773750000000003 2024-09-20T22:50:25.373 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.373 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.373 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:50:25.373 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.220417+0000", 2024-09-20T22:50:25.373 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00040778599999999999 2024-09-20T22:50:25.373 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.373 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.373 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:50:25.373 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.220479+0000", 2024-09-20T22:50:25.373 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.1977000000000006e-05 2024-09-20T22:50:25.373 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.374 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.374 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:50:25.374 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.220542+0000", 2024-09-20T22:50:25.374 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.3960000000000004e-05 2024-09-20T22:50:25.374 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:50:25.374 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:50:25.374 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:50:25.374 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.374 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.374 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:1528 7.17 7:ea720b85:::smithi08477203-26:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e197)", 2024-09-20T22:50:25.374 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:44:36.198994+0000", 2024-09-20T22:50:25.374 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 349.14325087200001, 2024-09-20T22:50:25.374 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.015013924, 2024-09-20T22:50:25.374 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:50:25.375 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:50:25.375 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:50:25.375 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:50:25.375 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:50:25.375 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:50:25.375 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 1528 2024-09-20T22:50:25.375 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.375 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:50:25.375 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.375 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:50:25.375 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.198994+0000", 2024-09-20T22:50:25.375 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:50:25.375 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.375 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.376 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:50:25.376 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.198994+0000", 2024-09-20T22:50:25.376 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:50:25.376 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.376 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.376 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:50:25.376 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.198994+0000", 2024-09-20T22:50:25.376 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.8100000000000003e-07 2024-09-20T22:50:25.376 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.376 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.376 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:50:25.376 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.198996+0000", 2024-09-20T22:50:25.376 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.3e-06 2024-09-20T22:50:25.377 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.377 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.377 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:50:25.377 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.199008+0000", 2024-09-20T22:50:25.377 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1603999999999999e-05 2024-09-20T22:50:25.377 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.377 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.377 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:50:25.377 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.199013+0000", 2024-09-20T22:50:25.377 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.4500000000000003e-06 2024-09-20T22:50:25.377 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.377 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.377 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:50:25.377 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.199120+0000", 2024-09-20T22:50:25.377 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000106772 2024-09-20T22:50:25.378 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.378 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.378 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting_for_map not empty", 2024-09-20T22:50:25.378 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.199126+0000", 2024-09-20T22:50:25.378 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.8320000000000002e-06 2024-09-20T22:50:25.378 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.378 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.378 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:50:25.378 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.202625+0000", 2024-09-20T22:50:25.378 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0034986309999999999 2024-09-20T22:50:25.378 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.378 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.378 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for blocked object", 2024-09-20T22:50:25.378 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.202667+0000", 2024-09-20T22:50:25.379 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.2759000000000002e-05 2024-09-20T22:50:25.379 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.379 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.379 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:50:25.379 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.213787+0000", 2024-09-20T22:50:25.379 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.011119377 2024-09-20T22:50:25.379 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.379 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.379 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:50:25.379 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.213902+0000", 2024-09-20T22:50:25.379 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000115369 2024-09-20T22:50:25.379 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.379 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.379 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:50:25.380 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.214007+0000", 2024-09-20T22:50:25.380 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000105349 2024-09-20T22:50:25.380 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:50:25.380 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:50:25.380 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:50:25.380 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.380 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.380 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:1550 7.17 7:ed54065f:::smithi08477203-12:head [copy-from ver 88] snapc 34=[34,33] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2024-09-20T22:50:25.380 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:44:37.238376+0000", 2024-09-20T22:50:25.380 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 348.10386796, 2024-09-20T22:50:25.380 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.016634917999999999, 2024-09-20T22:50:25.380 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:50:25.381 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:50:25.381 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:50:25.381 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:50:25.381 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:50:25.381 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:50:25.381 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 1550 2024-09-20T22:50:25.381 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.381 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:50:25.381 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.381 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:50:25.381 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:37.238376+0000", 2024-09-20T22:50:25.381 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:50:25.382 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.382 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.382 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:50:25.382 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:37.238376+0000", 2024-09-20T22:50:25.382 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:50:25.382 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.382 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.382 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:50:25.382 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:37.238379+0000", 2024-09-20T22:50:25.382 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.7989999999999998e-06 2024-09-20T22:50:25.382 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.382 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.382 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:50:25.382 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:37.238408+0000", 2024-09-20T22:50:25.382 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.8980999999999999e-05 2024-09-20T22:50:25.383 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.383 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.383 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:50:25.383 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:37.238433+0000", 2024-09-20T22:50:25.383 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.5185999999999999e-05 2024-09-20T22:50:25.383 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.383 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.383 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:50:25.383 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:37.238446+0000", 2024-09-20T22:50:25.383 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2883e-05 2024-09-20T22:50:25.383 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.383 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.383 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:50:25.383 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:37.238523+0000", 2024-09-20T22:50:25.383 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.6396999999999994e-05 2024-09-20T22:50:25.384 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.384 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.384 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:50:25.384 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:37.238618+0000", 2024-09-20T22:50:25.384 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.5049000000000004e-05 2024-09-20T22:50:25.384 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.384 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.384 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 2,5", 2024-09-20T22:50:25.384 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:37.246556+0000", 2024-09-20T22:50:25.384 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0079378090000000005 2024-09-20T22:50:25.384 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.384 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.384 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:50:25.384 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:37.254025+0000", 2024-09-20T22:50:25.385 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0074697139999999997 2024-09-20T22:50:25.385 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.385 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.385 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:50:25.385 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:37.254033+0000", 2024-09-20T22:50:25.385 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.6979999999999998e-06 2024-09-20T22:50:25.385 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.385 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.385 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:50:25.385 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:37.254919+0000", 2024-09-20T22:50:25.385 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00088564700000000004 2024-09-20T22:50:25.385 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.385 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.385 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:50:25.385 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:37.254958+0000", 2024-09-20T22:50:25.386 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.9515999999999999e-05 2024-09-20T22:50:25.386 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.386 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.386 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:50:25.386 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:37.255011+0000", 2024-09-20T22:50:25.386 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.3239000000000003e-05 2024-09-20T22:50:25.386 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:50:25.386 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:50:25.386 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:50:25.386 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.386 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.386 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:1799 7.17 7:ea720b85:::smithi08477203-26:head [write 521083~714336,stat] snapc 46=[45,44,43,42,41,40,3e] ondisk+write+known_if_redirected+supports_pool_eio e229)", 2024-09-20T22:50:25.386 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:45:08.170663+0000", 2024-09-20T22:50:25.386 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 317.17158131500003, 2024-09-20T22:50:25.386 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.016368226999999999, 2024-09-20T22:50:25.387 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:50:25.387 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:50:25.387 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:50:25.387 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:50:25.387 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:50:25.387 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:50:25.387 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 1799 2024-09-20T22:50:25.387 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.387 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:50:25.387 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.387 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:50:25.387 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.170663+0000", 2024-09-20T22:50:25.387 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:50:25.387 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.388 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.388 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:50:25.388 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.170663+0000", 2024-09-20T22:50:25.388 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:50:25.388 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.388 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.388 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:50:25.388 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.170664+0000", 2024-09-20T22:50:25.388 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.085e-06 2024-09-20T22:50:25.388 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.388 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.388 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:50:25.388 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.171481+0000", 2024-09-20T22:50:25.388 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00081653900000000002 2024-09-20T22:50:25.388 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.389 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.389 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:50:25.389 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.171492+0000", 2024-09-20T22:50:25.389 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1565e-05 2024-09-20T22:50:25.389 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.389 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.389 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:50:25.389 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.171498+0000", 2024-09-20T22:50:25.389 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.925e-06 2024-09-20T22:50:25.389 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.389 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.389 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:50:25.389 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.171547+0000", 2024-09-20T22:50:25.389 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.8916999999999999e-05 2024-09-20T22:50:25.390 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.390 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.390 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:50:25.390 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.171611+0000", 2024-09-20T22:50:25.390 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.3838999999999996e-05 2024-09-20T22:50:25.390 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.390 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.390 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 2,5", 2024-09-20T22:50:25.390 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.171740+0000", 2024-09-20T22:50:25.390 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00012881500000000001 2024-09-20T22:50:25.390 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.390 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.390 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:50:25.390 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.173862+0000", 2024-09-20T22:50:25.390 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0021225649999999999 2024-09-20T22:50:25.391 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.391 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.391 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:50:25.391 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.179252+0000", 2024-09-20T22:50:25.391 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0053895490000000004 2024-09-20T22:50:25.391 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.391 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.391 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:50:25.391 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.186956+0000", 2024-09-20T22:50:25.391 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0077038050000000002 2024-09-20T22:50:25.391 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.391 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.391 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:50:25.392 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.186993+0000", 2024-09-20T22:50:25.392 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.7141000000000002e-05 2024-09-20T22:50:25.392 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.392 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.392 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:50:25.392 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.187031+0000", 2024-09-20T22:50:25.392 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.8482000000000002e-05 2024-09-20T22:50:25.392 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:50:25.392 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:50:25.392 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:50:25.392 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.392 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.392 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:1800 7.17 7:ea720b85:::smithi08477203-26:head [write 1724715~555421] snapc 46=[45,44,43,42,41,40,3e] ondisk+write+known_if_redirected+supports_pool_eio e229)", 2024-09-20T22:50:25.393 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:45:08.177237+0000", 2024-09-20T22:50:25.393 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 317.16500768999998, 2024-09-20T22:50:25.393 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.012771694, 2024-09-20T22:50:25.393 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:50:25.393 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:50:25.393 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:50:25.393 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:50:25.393 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:50:25.393 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:50:25.393 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 1800 2024-09-20T22:50:25.393 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.393 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:50:25.393 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.393 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:50:25.393 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.177237+0000", 2024-09-20T22:50:25.394 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:50:25.394 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.394 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.394 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:50:25.394 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.177237+0000", 2024-09-20T22:50:25.394 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:50:25.394 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.394 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.394 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:50:25.394 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.177238+0000", 2024-09-20T22:50:25.394 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.6080000000000001e-06 2024-09-20T22:50:25.394 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.394 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.394 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:50:25.395 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.177844+0000", 2024-09-20T22:50:25.395 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00060579999999999998 2024-09-20T22:50:25.395 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.395 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.395 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:50:25.395 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.177857+0000", 2024-09-20T22:50:25.395 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2529e-05 2024-09-20T22:50:25.395 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.395 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.395 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:50:25.395 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.177864+0000", 2024-09-20T22:50:25.395 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.2470000000000002e-06 2024-09-20T22:50:25.395 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.396 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.396 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:50:25.396 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.177908+0000", 2024-09-20T22:50:25.396 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.4329999999999997e-05 2024-09-20T22:50:25.396 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.396 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.396 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:50:25.396 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.177977+0000", 2024-09-20T22:50:25.396 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.8598999999999998e-05 2024-09-20T22:50:25.396 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.396 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.396 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 2,5", 2024-09-20T22:50:25.396 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.178079+0000", 2024-09-20T22:50:25.396 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000102074 2024-09-20T22:50:25.396 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.397 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.397 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:50:25.397 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.181043+0000", 2024-09-20T22:50:25.397 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0029639810000000001 2024-09-20T22:50:25.397 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.397 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.397 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:50:25.397 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.181481+0000", 2024-09-20T22:50:25.397 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00043855899999999999 2024-09-20T22:50:25.397 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.397 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.397 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:50:25.397 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.189928+0000", 2024-09-20T22:50:25.397 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0084463149999999994 2024-09-20T22:50:25.398 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.398 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.398 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:50:25.398 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.189968+0000", 2024-09-20T22:50:25.398 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.0268999999999997e-05 2024-09-20T22:50:25.398 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.398 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.398 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:50:25.398 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.190008+0000", 2024-09-20T22:50:25.398 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.0383000000000003e-05 2024-09-20T22:50:25.398 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:50:25.398 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:50:25.398 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:50:25.398 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.399 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.399 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:1802 7.17 7:ea720b85:::smithi08477203-26:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e229)", 2024-09-20T22:50:25.399 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:45:08.177901+0000", 2024-09-20T22:50:25.399 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 317.16434374599999, 2024-09-20T22:50:25.399 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.012470591, 2024-09-20T22:50:25.399 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:50:25.399 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:50:25.399 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:50:25.399 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:50:25.399 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:50:25.399 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:50:25.399 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 1802 2024-09-20T22:50:25.399 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.399 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:50:25.400 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.400 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:50:25.400 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.177901+0000", 2024-09-20T22:50:25.400 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:50:25.400 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.400 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.400 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:50:25.400 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.177901+0000", 2024-09-20T22:50:25.400 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:50:25.400 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.400 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.400 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:50:25.400 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.177901+0000", 2024-09-20T22:50:25.400 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.2899999999999999e-07 2024-09-20T22:50:25.401 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.401 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.401 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:50:25.401 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.177902+0000", 2024-09-20T22:50:25.401 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.452e-06 2024-09-20T22:50:25.401 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.401 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.401 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:50:25.401 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.177906+0000", 2024-09-20T22:50:25.401 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.788e-06 2024-09-20T22:50:25.401 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.401 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.401 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:50:25.401 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.177909+0000", 2024-09-20T22:50:25.402 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.9840000000000001e-06 2024-09-20T22:50:25.402 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.402 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.402 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:50:25.402 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.179283+0000", 2024-09-20T22:50:25.402 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001373733 2024-09-20T22:50:25.402 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.402 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.402 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for rw locks", 2024-09-20T22:50:25.402 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.179364+0000", 2024-09-20T22:50:25.402 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.1209000000000001e-05 2024-09-20T22:50:25.402 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.402 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.402 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:50:25.403 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.190140+0000", 2024-09-20T22:50:25.403 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.010775527 2024-09-20T22:50:25.403 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.403 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.403 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:50:25.403 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.190186+0000", 2024-09-20T22:50:25.403 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.5853000000000003e-05 2024-09-20T22:50:25.403 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.403 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.403 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:50:25.403 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.190371+0000", 2024-09-20T22:50:25.403 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000185716 2024-09-20T22:50:25.403 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:50:25.404 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:50:25.404 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:50:25.404 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.404 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.404 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:2318 7.5 7:a160b5d4:::smithi08477203-16:head [copy-from ver 168] snapc 55=[55,53,52,51,50,4b,47,3e] ondisk+write+known_if_redirected+supports_pool_eio e258)", 2024-09-20T22:50:25.404 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:45:37.175863+0000", 2024-09-20T22:50:25.404 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 288.16638124000002, 2024-09-20T22:50:25.404 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.030719883999999999, 2024-09-20T22:50:25.404 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:50:25.404 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:50:25.404 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:50:25.404 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:50:25.404 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:50:25.404 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:50:25.405 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 2318 2024-09-20T22:50:25.405 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.405 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:50:25.405 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.405 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:50:25.405 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.175863+0000", 2024-09-20T22:50:25.405 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:50:25.405 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.405 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.405 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:50:25.405 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.175863+0000", 2024-09-20T22:50:25.405 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:50:25.405 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.405 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.405 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:50:25.406 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.175865+0000", 2024-09-20T22:50:25.406 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.488e-06 2024-09-20T22:50:25.406 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.406 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.406 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:50:25.406 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.175871+0000", 2024-09-20T22:50:25.406 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.438e-06 2024-09-20T22:50:25.406 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.406 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.406 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:50:25.406 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.175881+0000", 2024-09-20T22:50:25.406 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.6430000000000007e-06 2024-09-20T22:50:25.406 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.406 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.407 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:50:25.407 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.175886+0000", 2024-09-20T22:50:25.407 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.3059999999999997e-06 2024-09-20T22:50:25.407 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.407 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.407 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:50:25.407 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.175944+0000", 2024-09-20T22:50:25.407 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.7982999999999997e-05 2024-09-20T22:50:25.407 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.407 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.407 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op must wait for map", 2024-09-20T22:50:25.407 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.175951+0000", 2024-09-20T22:50:25.407 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.9280000000000004e-06 2024-09-20T22:50:25.407 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.408 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.408 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:50:25.408 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.179794+0000", 2024-09-20T22:50:25.408 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0038435209999999999 2024-09-20T22:50:25.408 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.408 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.408 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:50:25.408 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.179940+0000", 2024-09-20T22:50:25.408 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00014550000000000001 2024-09-20T22:50:25.408 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.408 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.409 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 0,2", 2024-09-20T22:50:25.409 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.190992+0000", 2024-09-20T22:50:25.409 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.011051587999999999 2024-09-20T22:50:25.409 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.409 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.409 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:50:25.409 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.195070+0000", 2024-09-20T22:50:25.409 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0040784929999999999 2024-09-20T22:50:25.409 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.409 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.409 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:50:25.409 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.202908+0000", 2024-09-20T22:50:25.409 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0078376690000000002 2024-09-20T22:50:25.409 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.410 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.410 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:50:25.410 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.206517+0000", 2024-09-20T22:50:25.410 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0036088880000000002 2024-09-20T22:50:25.410 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.410 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.410 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:50:25.410 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.206553+0000", 2024-09-20T22:50:25.410 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.6363999999999999e-05 2024-09-20T22:50:25.410 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.410 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.410 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:50:25.410 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.206583+0000", 2024-09-20T22:50:25.410 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.0074999999999999e-05 2024-09-20T22:50:25.411 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:50:25.411 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:50:25.411 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:50:25.411 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.411 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.411 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:2319 7.5 7:a160b5d4:::smithi08477203-16:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e258)", 2024-09-20T22:50:25.411 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:45:37.175983+0000", 2024-09-20T22:50:25.411 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 288.16626148500001, 2024-09-20T22:50:25.411 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.018323547999999999, 2024-09-20T22:50:25.411 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:50:25.411 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:50:25.411 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:50:25.411 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:50:25.411 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:50:25.412 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:50:25.412 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 2319 2024-09-20T22:50:25.412 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.412 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:50:25.412 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.412 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:50:25.412 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.175983+0000", 2024-09-20T22:50:25.412 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:50:25.412 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.412 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.412 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:50:25.412 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.175983+0000", 2024-09-20T22:50:25.412 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:50:25.412 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.412 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.412 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:50:25.413 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.175984+0000", 2024-09-20T22:50:25.413 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.1099999999999995e-07 2024-09-20T22:50:25.413 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.413 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.413 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:50:25.413 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.175996+0000", 2024-09-20T22:50:25.413 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2121e-05 2024-09-20T22:50:25.413 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.413 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.413 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:50:25.413 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.176019+0000", 2024-09-20T22:50:25.413 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.3465999999999999e-05 2024-09-20T22:50:25.413 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.413 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.414 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:50:25.414 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.176026+0000", 2024-09-20T22:50:25.414 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.8190000000000003e-06 2024-09-20T22:50:25.414 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.414 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.414 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:50:25.414 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.176110+0000", 2024-09-20T22:50:25.414 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.3967000000000004e-05 2024-09-20T22:50:25.414 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.414 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.414 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting_for_map not empty", 2024-09-20T22:50:25.414 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.176116+0000", 2024-09-20T22:50:25.414 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.0249999999999999e-06 2024-09-20T22:50:25.415 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.415 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.415 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:50:25.415 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.180106+0000", 2024-09-20T22:50:25.415 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0039898980000000004 2024-09-20T22:50:25.415 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.415 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.415 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for blocked object", 2024-09-20T22:50:25.415 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.180135+0000", 2024-09-20T22:50:25.415 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.9595999999999999e-05 2024-09-20T22:50:25.415 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.415 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.415 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:50:25.415 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.194106+0000", 2024-09-20T22:50:25.416 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.013970761999999999 2024-09-20T22:50:25.416 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.416 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.416 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:50:25.416 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.194203+0000", 2024-09-20T22:50:25.416 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.7211000000000002e-05 2024-09-20T22:50:25.416 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.416 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.416 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:50:25.416 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.194306+0000", 2024-09-20T22:50:25.416 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00010307200000000001 2024-09-20T22:50:25.416 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:50:25.416 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:50:25.416 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:50:25.417 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.417 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.417 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:25.417 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:50:25.418 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24415.0:230 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch (1) in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e259)", 2024-09-20T22:50:25.418 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:45:39.207410+0000", 2024-09-20T22:50:25.418 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 286.13483441300002, 2024-09-20T22:50:25.418 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0072298289999999, 2024-09-20T22:50:25.418 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:50:25.418 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:50:25.419 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:50:25.419 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:50:25.419 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24415", 2024-09-20T22:50:25.419 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/2056244102", 2024-09-20T22:50:25.419 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 230 2024-09-20T22:50:25.419 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.419 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:50:25.419 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.419 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:50:25.419 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.207410+0000", 2024-09-20T22:50:25.419 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:50:25.419 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.419 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.419 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:50:25.419 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.207410+0000", 2024-09-20T22:50:25.420 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:50:25.420 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.420 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.420 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:50:25.420 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.207412+0000", 2024-09-20T22:50:25.420 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.7519999999999999e-06 2024-09-20T22:50:25.420 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.420 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.420 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:50:25.420 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.207419+0000", 2024-09-20T22:50:25.420 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.097e-06 2024-09-20T22:50:25.420 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.420 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.420 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:50:25.420 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.207432+0000", 2024-09-20T22:50:25.421 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2719e-05 2024-09-20T22:50:25.421 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.421 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.421 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:50:25.421 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.207464+0000", 2024-09-20T22:50:25.421 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.2820000000000001e-05 2024-09-20T22:50:25.421 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.421 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.421 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:50:25.421 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.207670+0000", 2024-09-20T22:50:25.421 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000205486 2024-09-20T22:50:25.421 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.421 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.421 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-20T22:50:25.422 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.207687+0000", 2024-09-20T22:50:25.422 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.7107000000000001e-05 2024-09-20T22:50:25.422 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.422 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.422 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:50:25.422 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:40.212672+0000", 2024-09-20T22:50:25.422 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.004984562 2024-09-20T22:50:25.422 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.422 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.422 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:50:25.422 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:40.213576+0000", 2024-09-20T22:50:25.422 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00090404799999999996 2024-09-20T22:50:25.422 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.423 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.423 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:50:25.423 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:40.214640+0000", 2024-09-20T22:50:25.423 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0010642379999999999 2024-09-20T22:50:25.423 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:50:25.423 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:50:25.423 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:50:25.423 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.423 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.423 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:2367 7.5 7:a160b5d4:::smithi08477203-16:head [stat,delete] snapc 56=[56,55,53,52,51,50,47,3e] ondisk+write+known_if_redirected+supports_pool_eio e260)", 2024-09-20T22:50:25.423 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:45:39.601299+0000", 2024-09-20T22:50:25.423 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 285.74094499199998, 2024-09-20T22:50:25.423 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.64714523300000004, 2024-09-20T22:50:25.423 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:50:25.423 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:50:25.424 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:50:25.424 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:50:25.424 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:50:25.424 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:50:25.424 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 2367 2024-09-20T22:50:25.424 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.424 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:50:25.424 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.424 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:50:25.424 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.601299+0000", 2024-09-20T22:50:25.424 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:50:25.424 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.424 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.424 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:50:25.424 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.601299+0000", 2024-09-20T22:50:25.425 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:50:25.425 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.425 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.425 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:50:25.425 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.601301+0000", 2024-09-20T22:50:25.425 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3850000000000001e-06 2024-09-20T22:50:25.425 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.425 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.425 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:50:25.425 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.601306+0000", 2024-09-20T22:50:25.425 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.2730000000000002e-06 2024-09-20T22:50:25.425 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.425 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.425 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:50:25.425 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.601318+0000", 2024-09-20T22:50:25.426 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2148e-05 2024-09-20T22:50:25.426 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.426 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.426 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:50:25.426 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.601324+0000", 2024-09-20T22:50:25.426 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.852e-06 2024-09-20T22:50:25.426 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.426 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.426 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:50:25.426 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.601383+0000", 2024-09-20T22:50:25.426 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.8570000000000003e-05 2024-09-20T22:50:25.426 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.426 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.426 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-20T22:50:25.427 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.601391+0000", 2024-09-20T22:50:25.427 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.5290000000000008e-06 2024-09-20T22:50:25.427 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.427 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.427 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:50:25.427 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:40.221227+0000", 2024-09-20T22:50:25.427 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.619835515 2024-09-20T22:50:25.427 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.427 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.427 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for degraded object", 2024-09-20T22:50:25.427 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:40.223919+0000", 2024-09-20T22:50:25.427 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.002692168 2024-09-20T22:50:25.427 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.427 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.428 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:50:25.428 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:40.244536+0000", 2024-09-20T22:50:25.428 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.020617613 2024-09-20T22:50:25.428 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.428 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.428 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:50:25.428 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:40.244653+0000", 2024-09-20T22:50:25.428 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000116796 2024-09-20T22:50:25.428 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.428 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.428 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 1,2", 2024-09-20T22:50:25.428 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:40.244832+0000", 2024-09-20T22:50:25.428 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00017833500000000001 2024-09-20T22:50:25.428 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.428 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.429 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:50:25.429 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:40.245494+0000", 2024-09-20T22:50:25.429 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00066241399999999995 2024-09-20T22:50:25.429 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.429 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.429 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:50:25.429 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:40.246501+0000", 2024-09-20T22:50:25.429 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0010066299999999999 2024-09-20T22:50:25.429 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.429 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.429 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:50:25.429 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:40.248326+0000", 2024-09-20T22:50:25.429 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001824895 2024-09-20T22:50:25.430 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.430 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.430 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:50:25.430 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:40.248370+0000", 2024-09-20T22:50:25.430 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.4298999999999997e-05 2024-09-20T22:50:25.430 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.430 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.430 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:50:25.430 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:40.248445+0000", 2024-09-20T22:50:25.430 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.4810999999999999e-05 2024-09-20T22:50:25.430 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:50:25.430 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:50:25.430 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:50:25.430 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.430 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.431 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:3132 7.19 e328/139 7:9eac4560:::smithi08477203-30:head v 328'198, mlcod=322'194)", 2024-09-20T22:50:25.431 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:46:48.223406+0000", 2024-09-20T22:50:25.431 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 217.11883855599999, 2024-09-20T22:50:25.431 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.016765688000000001, 2024-09-20T22:50:25.431 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:50:25.431 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:50:25.431 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:50:25.431 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:50:25.431 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.431 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:50:25.431 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.223406+0000", 2024-09-20T22:50:25.431 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:50:25.431 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.432 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.432 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:50:25.432 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.223406+0000", 2024-09-20T22:50:25.432 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:50:25.432 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.432 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.432 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:50:25.432 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.223406+0000", 2024-09-20T22:50:25.432 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.6699999999999999e-07 2024-09-20T22:50:25.432 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.432 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.432 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:50:25.432 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.226227+0000", 2024-09-20T22:50:25.432 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0028205690000000002 2024-09-20T22:50:25.433 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.433 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.433 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:50:25.433 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.226238+0000", 2024-09-20T22:50:25.433 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0879999999999999e-05 2024-09-20T22:50:25.433 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.433 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.433 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:50:25.433 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.226249+0000", 2024-09-20T22:50:25.433 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0912e-05 2024-09-20T22:50:25.433 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.433 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.433 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:50:25.433 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.227960+0000", 2024-09-20T22:50:25.433 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0017109619999999999 2024-09-20T22:50:25.434 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.434 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.434 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:50:25.434 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.227987+0000", 2024-09-20T22:50:25.434 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.7659999999999999e-05 2024-09-20T22:50:25.434 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.434 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.434 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:50:25.434 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.240119+0000", 2024-09-20T22:50:25.434 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.012131885 2024-09-20T22:50:25.434 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.434 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.434 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:50:25.434 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.240172+0000", 2024-09-20T22:50:25.434 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.2453000000000001e-05 2024-09-20T22:50:25.435 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:50:25.435 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:50:25.435 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:50:25.435 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.435 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.435 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:3232 7.16 7:6885a65f:::smithi08477203-14:head [copy-from ver 187] snapc 7e=[7e,7d,7c,7b,7a,79,78,75,74,73,70,6b] ondisk+write+known_if_redirected+supports_pool_eio e337)", 2024-09-20T22:50:25.435 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:46:57.213186+0000", 2024-09-20T22:50:25.435 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 208.12905888500001, 2024-09-20T22:50:25.435 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.028083105000000001, 2024-09-20T22:50:25.435 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:50:25.435 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:50:25.435 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:50:25.435 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:50:25.435 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:50:25.435 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:50:25.436 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 3232 2024-09-20T22:50:25.436 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.436 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:50:25.436 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.436 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:50:25.436 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.213186+0000", 2024-09-20T22:50:25.436 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:50:25.436 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.436 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.436 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:50:25.436 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.213186+0000", 2024-09-20T22:50:25.436 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:50:25.436 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.436 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.436 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:50:25.437 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.213188+0000", 2024-09-20T22:50:25.437 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.5129999999999999e-06 2024-09-20T22:50:25.437 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.437 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.437 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:50:25.437 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.213194+0000", 2024-09-20T22:50:25.437 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.4810000000000003e-06 2024-09-20T22:50:25.437 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.437 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.437 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:50:25.437 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.213215+0000", 2024-09-20T22:50:25.437 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.0455999999999999e-05 2024-09-20T22:50:25.437 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.438 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.438 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:50:25.438 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.213225+0000", 2024-09-20T22:50:25.438 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0212e-05 2024-09-20T22:50:25.438 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.438 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.438 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:50:25.438 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.213315+0000", 2024-09-20T22:50:25.438 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.9524000000000005e-05 2024-09-20T22:50:25.438 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.438 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.438 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op must wait for map", 2024-09-20T22:50:25.438 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.213324+0000", 2024-09-20T22:50:25.438 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.8890000000000008e-06 2024-09-20T22:50:25.439 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.439 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.439 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:50:25.439 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.220517+0000", 2024-09-20T22:50:25.439 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0071931219999999997 2024-09-20T22:50:25.439 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.439 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.439 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:50:25.439 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.220636+0000", 2024-09-20T22:50:25.439 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000118875 2024-09-20T22:50:25.439 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.439 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.439 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 2,5", 2024-09-20T22:50:25.439 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.230308+0000", 2024-09-20T22:50:25.440 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0096726299999999998 2024-09-20T22:50:25.440 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.440 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.440 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:50:25.440 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.238007+0000", 2024-09-20T22:50:25.440 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.007699287 2024-09-20T22:50:25.440 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.440 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.440 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:50:25.440 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.240863+0000", 2024-09-20T22:50:25.440 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0028554880000000002 2024-09-20T22:50:25.440 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.440 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.440 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:50:25.441 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.241194+0000", 2024-09-20T22:50:25.441 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00033087399999999999 2024-09-20T22:50:25.441 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.441 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.441 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:50:25.441 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.241233+0000", 2024-09-20T22:50:25.441 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.9338000000000002e-05 2024-09-20T22:50:25.441 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.441 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.441 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:50:25.441 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.241269+0000", 2024-09-20T22:50:25.441 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.5416000000000001e-05 2024-09-20T22:50:25.441 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:50:25.441 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:50:25.442 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:50:25.442 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.442 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.442 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:3233 7.16 7:6885a65f:::smithi08477203-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e337)", 2024-09-20T22:50:25.442 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:46:57.213253+0000", 2024-09-20T22:50:25.442 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 208.128991461, 2024-09-20T22:50:25.442 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.023594825, 2024-09-20T22:50:25.442 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:50:25.442 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:50:25.442 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:50:25.442 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:50:25.442 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:50:25.442 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:50:25.442 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 3233 2024-09-20T22:50:25.442 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.443 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:50:25.443 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.443 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:50:25.443 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.213253+0000", 2024-09-20T22:50:25.443 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:50:25.443 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.443 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.443 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:50:25.443 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.213253+0000", 2024-09-20T22:50:25.443 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:50:25.443 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.443 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.443 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:50:25.443 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.213254+0000", 2024-09-20T22:50:25.443 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.4799999999999998e-07 2024-09-20T22:50:25.444 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.444 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.444 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:50:25.444 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.213257+0000", 2024-09-20T22:50:25.444 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.3780000000000001e-06 2024-09-20T22:50:25.444 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.444 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.444 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:50:25.444 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.213267+0000", 2024-09-20T22:50:25.444 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.9480000000000003e-06 2024-09-20T22:50:25.444 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.444 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.444 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:50:25.444 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.213273+0000", 2024-09-20T22:50:25.444 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.1240000000000002e-06 2024-09-20T22:50:25.445 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.445 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.445 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:50:25.445 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.213394+0000", 2024-09-20T22:50:25.445 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000120717 2024-09-20T22:50:25.445 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.445 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.445 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting_for_map not empty", 2024-09-20T22:50:25.445 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.213400+0000", 2024-09-20T22:50:25.445 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.0769999999999996e-06 2024-09-20T22:50:25.445 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.445 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.445 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:50:25.445 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.220851+0000", 2024-09-20T22:50:25.446 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0074511309999999997 2024-09-20T22:50:25.446 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.446 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.446 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for blocked object", 2024-09-20T22:50:25.446 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.220903+0000", 2024-09-20T22:50:25.446 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.215e-05 2024-09-20T22:50:25.446 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.446 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.446 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:50:25.446 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.236698+0000", 2024-09-20T22:50:25.446 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.015794823 2024-09-20T22:50:25.446 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.446 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.446 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:50:25.447 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.236766+0000", 2024-09-20T22:50:25.447 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.8126999999999993e-05 2024-09-20T22:50:25.447 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.447 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.447 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:50:25.447 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.236848+0000", 2024-09-20T22:50:25.447 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.1701999999999998e-05 2024-09-20T22:50:25.447 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:50:25.447 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:50:25.447 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:50:25.447 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.447 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.447 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:3671 7.5 7:a160b5d4:::smithi08477203-16:head [copy-from ver 236] snapc 91=[91,8e,8d,8c,8b,8a,88,87,86,85,84,80,7f,7a,74,73,6b] ondisk+write+known_if_redirected+supports_pool_eio e370)", 2024-09-20T22:50:25.447 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:47:30.237235+0000", 2024-09-20T22:50:25.447 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 175.10500966199999, 2024-09-20T22:50:25.448 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.019719474000000001, 2024-09-20T22:50:25.448 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:50:25.448 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:50:25.448 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:50:25.448 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:50:25.448 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:50:25.448 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:50:25.448 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 3671 2024-09-20T22:50:25.448 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.448 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:50:25.448 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.448 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:50:25.448 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:30.237235+0000", 2024-09-20T22:50:25.448 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:50:25.448 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.449 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.449 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:50:25.449 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:30.237235+0000", 2024-09-20T22:50:25.449 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:50:25.449 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.449 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.449 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:50:25.449 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:30.237236+0000", 2024-09-20T22:50:25.449 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.5570000000000001e-06 2024-09-20T22:50:25.449 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.449 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.449 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:50:25.449 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:30.237241+0000", 2024-09-20T22:50:25.449 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.3279999999999999e-06 2024-09-20T22:50:25.450 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.450 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.450 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:50:25.450 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:30.237252+0000", 2024-09-20T22:50:25.450 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1004e-05 2024-09-20T22:50:25.450 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.450 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.450 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:50:25.450 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:30.237263+0000", 2024-09-20T22:50:25.450 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1341e-05 2024-09-20T22:50:25.450 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.450 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.450 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:50:25.450 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:30.237319+0000", 2024-09-20T22:50:25.451 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.5939000000000001e-05 2024-09-20T22:50:25.451 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.451 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.451 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:50:25.451 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:30.237436+0000", 2024-09-20T22:50:25.451 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000116866 2024-09-20T22:50:25.451 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.451 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.451 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 1,2", 2024-09-20T22:50:25.451 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:30.244330+0000", 2024-09-20T22:50:25.451 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0068945619999999999 2024-09-20T22:50:25.451 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.451 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.451 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:50:25.452 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:30.249103+0000", 2024-09-20T22:50:25.452 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0047726319999999997 2024-09-20T22:50:25.452 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.452 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.452 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:50:25.452 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:30.255771+0000", 2024-09-20T22:50:25.452 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.006667666 2024-09-20T22:50:25.452 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.452 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.452 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:50:25.452 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:30.256879+0000", 2024-09-20T22:50:25.452 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0011083110000000001 2024-09-20T22:50:25.452 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.452 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.452 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:50:25.453 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:30.256919+0000", 2024-09-20T22:50:25.453 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.9589000000000003e-05 2024-09-20T22:50:25.453 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.453 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.453 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:50:25.453 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:30.256954+0000", 2024-09-20T22:50:25.453 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.5679000000000002e-05 2024-09-20T22:50:25.453 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:50:25.453 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:50:25.453 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:50:25.453 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.453 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.453 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:3747 7.c e383/94 7:3f865ef5:::smithi08477203-28:head v 383'822, mlcod=382'818)", 2024-09-20T22:50:25.453 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:47:43.267267+0000", 2024-09-20T22:50:25.453 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 162.074977198, 2024-09-20T22:50:25.454 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.014405072, 2024-09-20T22:50:25.454 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:50:25.454 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:50:25.454 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:50:25.454 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:50:25.454 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.454 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:50:25.454 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.267267+0000", 2024-09-20T22:50:25.454 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:50:25.454 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.454 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.454 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:50:25.454 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.267267+0000", 2024-09-20T22:50:25.454 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:50:25.455 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.455 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.455 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:50:25.455 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.267267+0000", 2024-09-20T22:50:25.455 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.73e-07 2024-09-20T22:50:25.455 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.455 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.455 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:50:25.455 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.269537+0000", 2024-09-20T22:50:25.455 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0022696650000000001 2024-09-20T22:50:25.455 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.455 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.455 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:50:25.455 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.269564+0000", 2024-09-20T22:50:25.455 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.6653999999999999e-05 2024-09-20T22:50:25.456 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.456 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.456 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:50:25.456 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.269575+0000", 2024-09-20T22:50:25.456 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1661999999999999e-05 2024-09-20T22:50:25.456 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.456 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.456 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:50:25.456 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.269672+0000", 2024-09-20T22:50:25.456 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.6430999999999999e-05 2024-09-20T22:50:25.456 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.456 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.456 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:50:25.456 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.269716+0000", 2024-09-20T22:50:25.457 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.4518000000000003e-05 2024-09-20T22:50:25.457 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.457 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.457 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:50:25.457 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.281609+0000", 2024-09-20T22:50:25.457 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.011892982999999999 2024-09-20T22:50:25.457 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.457 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.457 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:50:25.457 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.281672+0000", 2024-09-20T22:50:25.457 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.2985999999999996e-05 2024-09-20T22:50:25.457 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:50:25.457 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:50:25.457 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:50:25.458 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.458 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.458 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:3762 7.17 7:ed54065f:::smithi08477203-12:head [write 739640~641484] snapc 98=[98,97,96,95,94,93,92,8e,8d,8c,8b,8a,88,87,86,80,7f,73] ondisk+write+known_if_redirected+supports_pool_eio e383)", 2024-09-20T22:50:25.458 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:47:43.291672+0000", 2024-09-20T22:50:25.458 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 162.05057284700001, 2024-09-20T22:50:25.458 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.012690432999999999, 2024-09-20T22:50:25.458 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:50:25.458 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:50:25.458 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:50:25.458 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:50:25.458 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:50:25.458 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:50:25.458 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 3762 2024-09-20T22:50:25.459 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.459 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:50:25.459 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.459 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:50:25.459 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.291672+0000", 2024-09-20T22:50:25.459 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:50:25.459 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.459 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.459 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:50:25.459 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.291672+0000", 2024-09-20T22:50:25.459 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:50:25.459 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.459 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.459 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:50:25.460 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.291673+0000", 2024-09-20T22:50:25.460 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.5680000000000001e-06 2024-09-20T22:50:25.460 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.460 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.460 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:50:25.460 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.292393+0000", 2024-09-20T22:50:25.460 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00072015399999999998 2024-09-20T22:50:25.460 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.460 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.460 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:50:25.460 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.292407+0000", 2024-09-20T22:50:25.460 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.365e-05 2024-09-20T22:50:25.460 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.460 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.460 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:50:25.460 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.292431+0000", 2024-09-20T22:50:25.461 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.3740999999999999e-05 2024-09-20T22:50:25.461 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.461 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.461 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:50:25.461 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.292482+0000", 2024-09-20T22:50:25.461 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.1459999999999999e-05 2024-09-20T22:50:25.461 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.461 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.461 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:50:25.461 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.292549+0000", 2024-09-20T22:50:25.461 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.7223000000000003e-05 2024-09-20T22:50:25.461 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.461 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.461 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 2,5", 2024-09-20T22:50:25.462 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.292684+0000", 2024-09-20T22:50:25.462 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000134251 2024-09-20T22:50:25.462 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.462 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.462 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:50:25.462 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.293845+0000", 2024-09-20T22:50:25.462 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0011617789999999999 2024-09-20T22:50:25.462 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.462 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.462 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:50:25.462 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.294392+0000", 2024-09-20T22:50:25.462 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00054659800000000003 2024-09-20T22:50:25.462 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.462 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.463 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:50:25.463 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.304282+0000", 2024-09-20T22:50:25.463 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0098903099999999994 2024-09-20T22:50:25.463 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.463 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.463 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:50:25.463 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.304322+0000", 2024-09-20T22:50:25.463 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.0059999999999999e-05 2024-09-20T22:50:25.463 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.463 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.463 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:50:25.463 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.304362+0000", 2024-09-20T22:50:25.463 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.9638999999999998e-05 2024-09-20T22:50:25.463 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:50:25.463 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:50:25.464 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:50:25.464 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.464 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.464 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:4129 7.c e417/94 7:3f865ef5:::smithi08477203-28:head v 417'903, mlcod=417'899)", 2024-09-20T22:50:25.464 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:48:17.254548+0000", 2024-09-20T22:50:25.464 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 128.08769668299999, 2024-09-20T22:50:25.464 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.012969022, 2024-09-20T22:50:25.464 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:50:25.464 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:50:25.464 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:50:25.464 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:50:25.464 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.464 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:50:25.464 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:17.254548+0000", 2024-09-20T22:50:25.465 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:50:25.465 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.465 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.465 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:50:25.465 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:17.254548+0000", 2024-09-20T22:50:25.465 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:50:25.465 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.465 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.465 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:50:25.465 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:17.254549+0000", 2024-09-20T22:50:25.465 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.446e-06 2024-09-20T22:50:25.465 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.465 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.465 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:50:25.466 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:17.257313+0000", 2024-09-20T22:50:25.466 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0027641319999999999 2024-09-20T22:50:25.466 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.466 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.466 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:50:25.466 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:17.257333+0000", 2024-09-20T22:50:25.466 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9950999999999999e-05 2024-09-20T22:50:25.466 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.466 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.466 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:50:25.466 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:17.257343+0000", 2024-09-20T22:50:25.466 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.2750000000000001e-06 2024-09-20T22:50:25.466 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.466 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.466 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:50:25.467 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:17.257416+0000", 2024-09-20T22:50:25.467 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.3041999999999996e-05 2024-09-20T22:50:25.467 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.467 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.467 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:50:25.467 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:17.257442+0000", 2024-09-20T22:50:25.467 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.6095999999999999e-05 2024-09-20T22:50:25.467 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.467 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.467 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:50:25.467 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:17.267454+0000", 2024-09-20T22:50:25.467 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.010012455999999999 2024-09-20T22:50:25.467 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:50:25.468 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:50:25.468 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:50:25.468 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:17.267517+0000", 2024-09-20T22:50:25.468 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.2624000000000002e-05 2024-09-20T22:50:25.468 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:50:25.468 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:50:25.468 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:50:25.468 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:50:25.468 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:50:25.468 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-20T22:50:25.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:25 smithi154 ceph-mon[33264]: pgmap v717: 158 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 153 active+clean; 497 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 9.7 MiB/s rd, 1.6 MiB/s wr, 12 op/s 2024-09-20T22:50:25.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:25 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/318887551' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:50:25.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:25 smithi154 ceph-mon[33264]: osdmap e545: 8 total, 8 up, 8 in 2024-09-20T22:50:25.518 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:50:25.539 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:25.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:25.540 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:50:25.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:25 smithi084 ceph-mon[40024]: pgmap v717: 158 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 153 active+clean; 497 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 9.7 MiB/s rd, 1.6 MiB/s wr, 12 op/s 2024-09-20T22:50:25.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:25 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/318887551' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:50:25.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:25 smithi084 ceph-mon[40024]: osdmap e545: 8 total, 8 up, 8 in 2024-09-20T22:50:25.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:25 smithi084 ceph-mon[29774]: pgmap v717: 158 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 153 active+clean; 497 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 9.7 MiB/s rd, 1.6 MiB/s wr, 12 op/s 2024-09-20T22:50:25.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:25 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/318887551' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:50:25.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:25 smithi084 ceph-mon[29774]: osdmap e545: 8 total, 8 up, 8 in 2024-09-20T22:50:25.641 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:50:25.662 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:25.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:25.663 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:50:25.763 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:50:25.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:25.785 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:25.785 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:50:25.886 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:50:25.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:25.907 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:25.908 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:50:25.988 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-20T22:50:26.008 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:50:26.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:26.054 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:26.054 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:50:26.155 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:50:26.178 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:26.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:26.179 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:50:26.189 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-20T22:50:26.246 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:50:26.261 INFO:tasks.rados.rados.0.smithi084.stdout:2130: done (2 left) 2024-09-20T22:50:26.261 INFO:tasks.rados.rados.0.smithi084.stdout:2131: read oid 20 snap -1 2024-09-20T22:50:26.261 INFO:tasks.rados.rados.0.smithi084.stdout:2131: expect deleted 2024-09-20T22:50:26.261 INFO:tasks.rados.rados.0.smithi084.stdout:2132: write oid 24 current snap is 237 2024-09-20T22:50:26.261 INFO:tasks.rados.rados.0.smithi084.stdout:2127: finishing write tid 2 to smithi08477203-12 2024-09-20T22:50:26.261 INFO:tasks.rados.rados.0.smithi084.stdout:2127: finishing write tid 3 to smithi08477203-12 2024-09-20T22:50:26.261 INFO:tasks.rados.rados.0.smithi084.stdout:2127: finishing write tid 4 to smithi08477203-12 2024-09-20T22:50:26.261 INFO:tasks.rados.rados.0.smithi084.stdout:2127: oid 12 updating version 0 to 740 2024-09-20T22:50:26.262 INFO:tasks.rados.rados.0.smithi084.stdout:2127: oid 12 updating version 740 to 742 2024-09-20T22:50:26.262 INFO:tasks.rados.rados.0.smithi084.stdout:2127: oid 12 version 742 is already newer than 741 2024-09-20T22:50:26.262 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 12 v 742 (ObjNum 741 snap 237 seq_num 741) dirty exists 2024-09-20T22:50:26.262 INFO:tasks.rados.rados.0.smithi084.stdout:2127: left oid 12 (ObjNum 741 snap 237 seq_num 741) 2024-09-20T22:50:26.262 INFO:tasks.rados.rados.0.smithi084.stdout:2128: finishing write tid 1 to smithi08477203-40 2024-09-20T22:50:26.262 INFO:tasks.rados.rados.0.smithi084.stdout:2128: finishing write tid 2 to smithi08477203-40 2024-09-20T22:50:26.262 INFO:tasks.rados.rados.0.smithi084.stdout:2128: finishing write tid 3 to smithi08477203-40 2024-09-20T22:50:26.262 INFO:tasks.rados.rados.0.smithi084.stdout:2128: finishing write tid 4 to smithi08477203-40 2024-09-20T22:50:26.262 INFO:tasks.rados.rados.0.smithi084.stdout:2128: oid 40 updating version 0 to 744 2024-09-20T22:50:26.262 INFO:tasks.rados.rados.0.smithi084.stdout:2128: oid 40 updating version 744 to 746 2024-09-20T22:50:26.262 INFO:tasks.rados.rados.0.smithi084.stdout:2128: oid 40 version 746 is already newer than 745 2024-09-20T22:50:26.262 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 40 v 746 (ObjNum 742 snap 237 seq_num 742) dirty exists 2024-09-20T22:50:26.262 INFO:tasks.rados.rados.0.smithi084.stdout:2128: left oid 40 (ObjNum 742 snap 237 seq_num 742) 2024-09-20T22:50:26.262 INFO:tasks.rados.rados.0.smithi084.stdout:2132: seq_num 743 ranges {487601=796248,1962121=525480} 2024-09-20T22:50:26.269 INFO:tasks.rados.rados.0.smithi084.stdout:2132: writing smithi08477203-24 from 487601 to 1283849 tid 1 2024-09-20T22:50:26.278 INFO:tasks.rados.rados.0.smithi084.stdout:2132: writing smithi08477203-24 from 1962121 to 2487601 tid 2 2024-09-20T22:50:26.278 INFO:tasks.rados.rados.0.smithi084.stdout:2127: done (3 left) 2024-09-20T22:50:26.278 INFO:tasks.rados.rados.0.smithi084.stdout:2128: done (2 left) 2024-09-20T22:50:26.278 INFO:tasks.rados.rados.0.smithi084.stdout:2133: copy_from oid 36 from oid 19 current snap is 237 2024-09-20T22:50:26.278 INFO:tasks.rados.rados.0.smithi084.stdout:2132: finishing write tid 1 to smithi08477203-24 2024-09-20T22:50:26.278 INFO:tasks.rados.rados.0.smithi084.stdout:2131: done (2 left) 2024-09-20T22:50:26.278 INFO:tasks.rados.rados.0.smithi084.stdout:2134: write oid 4 current snap is 237 2024-09-20T22:50:26.279 INFO:tasks.rados.rados.0.smithi084.stdout:2134: seq_num 744 ranges {504408=471497,1620603=557925} 2024-09-20T22:50:26.280 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:50:26.284 INFO:tasks.rados.rados.0.smithi084.stdout:2134: writing smithi08477203-4 from 504408 to 975905 tid 1 2024-09-20T22:50:26.291 INFO:tasks.rados.rados.0.smithi084.stdout:2134: writing smithi08477203-4 from 1620603 to 2178528 tid 2 2024-09-20T22:50:26.291 INFO:tasks.rados.rados.0.smithi084.stdout:2135: rollback oid 50 current snap is 237 2024-09-20T22:50:26.292 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 50 to 165 2024-09-20T22:50:26.292 INFO:tasks.rados.rados.0.smithi084.stdout:2133: finishing copy_from to smithi08477203-36 2024-09-20T22:50:26.292 INFO:tasks.rados.rados.0.smithi084.stdout:2133: got expected ENOENT (src dne) 2024-09-20T22:50:26.292 INFO:tasks.rados.rados.0.smithi084.stdout:2133: finishing copy_from racing read to smithi08477203-36 2024-09-20T22:50:26.292 INFO:tasks.rados.rados.0.smithi084.stdout:2132: finishing write tid 2 to smithi08477203-24 2024-09-20T22:50:26.292 INFO:tasks.rados.rados.0.smithi084.stdout:2132: finishing write tid 3 to smithi08477203-24 2024-09-20T22:50:26.292 INFO:tasks.rados.rados.0.smithi084.stdout:2132: finishing write tid 4 to smithi08477203-24 2024-09-20T22:50:26.292 INFO:tasks.rados.rados.0.smithi084.stdout:2132: oid 24 updating version 0 to 728 2024-09-20T22:50:26.292 INFO:tasks.rados.rados.0.smithi084.stdout:2132: oid 24 version 728 is already newer than 727 2024-09-20T22:50:26.292 INFO:tasks.rados.rados.0.smithi084.stdout:2132: oid 24 updating version 728 to 729 2024-09-20T22:50:26.292 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 24 v 729 (ObjNum 743 snap 237 seq_num 743) dirty exists 2024-09-20T22:50:26.292 INFO:tasks.rados.rados.0.smithi084.stdout:2132: left oid 24 (ObjNum 743 snap 237 seq_num 743) 2024-09-20T22:50:26.292 INFO:tasks.rados.rados.0.smithi084.stdout:2134: finishing write tid 1 to smithi08477203-4 2024-09-20T22:50:26.293 INFO:tasks.rados.rados.0.smithi084.stdout:2132: done (3 left) 2024-09-20T22:50:26.293 INFO:tasks.rados.rados.0.smithi084.stdout:2133: done (2 left) 2024-09-20T22:50:26.293 INFO:tasks.rados.rados.0.smithi084.stdout:2136: write oid 31 current snap is 237 2024-09-20T22:50:26.293 INFO:tasks.rados.rados.0.smithi084.stdout:2136: seq_num 745 ranges {521215=546746,1562732=701583} 2024-09-20T22:50:26.297 INFO:tasks.rados.rados.0.smithi084.stdout:2136: writing smithi08477203-31 from 521215 to 1067961 tid 1 2024-09-20T22:50:26.304 INFO:tasks.rados.rados.0.smithi084.stdout:2136: writing smithi08477203-31 from 1562732 to 2264315 tid 2 2024-09-20T22:50:26.304 INFO:tasks.rados.rados.0.smithi084.stdout:2137: copy_from oid 24 from oid 6 current snap is 237 2024-09-20T22:50:26.304 INFO:tasks.rados.rados.0.smithi084.stdout:2135: finishing rollback tid 0 to smithi08477203-50 2024-09-20T22:50:26.305 INFO:tasks.rados.rados.0.smithi084.stdout:2135: finishing rollback tid 1 to smithi08477203-50 2024-09-20T22:50:26.305 INFO:tasks.rados.rados.0.smithi084.stdout:2135: finishing rollback tid 2 to smithi08477203-50 2024-09-20T22:50:26.305 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 50 v 732 (ObjNum 515 snap 148 seq_num 515) dirty exists 2024-09-20T22:50:26.305 INFO:tasks.rados.rados.0.smithi084.stdout:2134: finishing write tid 2 to smithi08477203-4 2024-09-20T22:50:26.305 INFO:tasks.rados.rados.0.smithi084.stdout:2135: done (3 left) 2024-09-20T22:50:26.305 INFO:tasks.rados.rados.0.smithi084.stdout:2138: snap_create 2024-09-20T22:50:26.305 INFO:tasks.rados.rados.0.smithi084.stdout:2134: finishing write tid 3 to smithi08477203-4 2024-09-20T22:50:26.305 INFO:tasks.rados.rados.0.smithi084.stdout:2134: finishing write tid 4 to smithi08477203-4 2024-09-20T22:50:26.305 INFO:tasks.rados.rados.0.smithi084.stdout:2134: oid 4 updating version 0 to 1466 2024-09-20T22:50:26.305 INFO:tasks.rados.rados.0.smithi084.stdout:2134: oid 4 updating version 1466 to 1468 2024-09-20T22:50:26.305 INFO:tasks.rados.rados.0.smithi084.stdout:2134: oid 4 version 1468 is already newer than 1467 2024-09-20T22:50:26.305 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 4 v 1468 (ObjNum 744 snap 237 seq_num 744) dirty exists 2024-09-20T22:50:26.305 INFO:tasks.rados.rados.0.smithi084.stdout:2134: left oid 4 (ObjNum 744 snap 237 seq_num 744) 2024-09-20T22:50:26.306 INFO:tasks.rados.rados.0.smithi084.stdout:2136: finishing write tid 1 to smithi08477203-31 2024-09-20T22:50:26.307 INFO:tasks.rados.rados.0.smithi084.stdout:2137: finishing copy_from to smithi08477203-24 2024-09-20T22:50:26.307 INFO:tasks.rados.rados.0.smithi084.stdout:2137: got expected ENOENT (src dne) 2024-09-20T22:50:26.307 INFO:tasks.rados.rados.0.smithi084.stdout:2137: finishing copy_from racing read to smithi08477203-24 2024-09-20T22:50:26.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:26.307 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:26.308 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:50:26.309 INFO:tasks.rados.rados.0.smithi084.stdout:2136: finishing write tid 2 to smithi08477203-31 2024-09-20T22:50:26.309 INFO:tasks.rados.rados.0.smithi084.stdout:2136: finishing write tid 3 to smithi08477203-31 2024-09-20T22:50:26.309 INFO:tasks.rados.rados.0.smithi084.stdout:2136: finishing write tid 4 to smithi08477203-31 2024-09-20T22:50:26.309 INFO:tasks.rados.rados.0.smithi084.stdout:2136: oid 31 updating version 0 to 747 2024-09-20T22:50:26.309 INFO:tasks.rados.rados.0.smithi084.stdout:2136: oid 31 version 747 is already newer than 745 2024-09-20T22:50:26.309 INFO:tasks.rados.rados.0.smithi084.stdout:2136: oid 31 version 747 is already newer than 746 2024-09-20T22:50:26.309 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 31 v 747 (ObjNum 745 snap 237 seq_num 745) dirty exists 2024-09-20T22:50:26.309 INFO:tasks.rados.rados.0.smithi084.stdout:2136: left oid 31 (ObjNum 745 snap 237 seq_num 745) 2024-09-20T22:50:26.409 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:50:26.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:26.431 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:26.431 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:50:26.532 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:50:26.553 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:26.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:26.555 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:50:26.655 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:50:26.676 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:26.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:26.677 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:50:26.778 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:50:26.800 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:26.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:26.801 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:50:26.801 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:50:26.902 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:50:26.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:26.924 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:26.924 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:50:27.025 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:50:27.047 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:27.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:27.048 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:50:27.149 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:50:27.170 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:27.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:27.171 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:50:27.258 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is set 2024-09-20T22:50:27.266 INFO:tasks.rados.rados.0.smithi084.stdout:2134: done (3 left) 2024-09-20T22:50:27.266 INFO:tasks.rados.rados.0.smithi084.stdout:2136: done (2 left) 2024-09-20T22:50:27.266 INFO:tasks.rados.rados.0.smithi084.stdout:2137: done (1 left) 2024-09-20T22:50:27.266 INFO:tasks.rados.rados.0.smithi084.stdout:2138: done (0 left) 2024-09-20T22:50:27.266 INFO:tasks.rados.rados.0.smithi084.stdout:2139: delete oid 15 current snap is 238 2024-09-20T22:50:27.267 INFO:tasks.rados.rados.0.smithi084.stdout:2139: done (0 left) 2024-09-20T22:50:27.268 INFO:tasks.rados.rados.0.smithi084.stdout:2140: snap_remove snap 184 2024-09-20T22:50:27.272 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:50:27.293 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:27.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:27.294 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:50:27.341 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-20T22:50:27.342 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2024-09-20T22:50:27.342 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2024-09-20T22:50:27.342 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-20T22:50:27.395 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:50:27.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:27.417 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:27.418 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:50:27.480 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-20T22:50:27.502 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:27 smithi084 ceph-mon[40024]: pgmap v720: 158 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 153 active+clean; 497 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 9.7 MiB/s rd, 1.6 MiB/s wr, 12 op/s 2024-09-20T22:50:27.502 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:27 smithi084 ceph-mon[40024]: osdmap e546: 8 total, 8 up, 8 in 2024-09-20T22:50:27.502 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:27 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/4195890921' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:50:27.502 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:27 smithi084 ceph-mon[29774]: pgmap v720: 158 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 153 active+clean; 497 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 9.7 MiB/s rd, 1.6 MiB/s wr, 12 op/s 2024-09-20T22:50:27.503 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:27 smithi084 ceph-mon[29774]: osdmap e546: 8 total, 8 up, 8 in 2024-09-20T22:50:27.503 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:27 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/4195890921' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:50:27.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:27 smithi154 ceph-mon[33264]: pgmap v720: 158 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 153 active+clean; 497 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 9.7 MiB/s rd, 1.6 MiB/s wr, 12 op/s 2024-09-20T22:50:27.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:27 smithi154 ceph-mon[33264]: osdmap e546: 8 total, 8 up, 8 in 2024-09-20T22:50:27.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:27 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/4195890921' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:50:27.518 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:50:27.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:27.541 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:27.541 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:50:27.641 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:50:27.663 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:27.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:27.664 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:50:27.764 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:50:27.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:27.788 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:27.788 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:50:27.889 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:50:27.911 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:27.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:27.912 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:50:28.013 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:50:28.035 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:28.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:28.036 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:50:28.075 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 5, 6, 7, 0, 4] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-20T22:50:28.076 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-20T22:50:28.076 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-09-20T22:50:28.076 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd dump -f json-pretty 2024-09-20T22:50:28.137 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:50:28.161 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:28.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:28.162 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:50:28.241 INFO:tasks.rados.rados.0.smithi084.stdout:2140: done (0 left) 2024-09-20T22:50:28.241 INFO:tasks.rados.rados.0.smithi084.stdout:2141: snap_remove snap 214 2024-09-20T22:50:28.262 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:50:28.288 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:28.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:28.289 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:50:28.390 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:50:28.410 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:28.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:28.411 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:50:28.512 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:50:28.533 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:28.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:28.534 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:50:28.614 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:50:28 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: ::ffff:172.21.15.154 - - [20/Sep/2024:22:50:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-20T22:50:28.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:28 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/4195890921' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T22:50:28.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:28 smithi084 ceph-mon[40024]: osdmap e547: 8 total, 8 up, 8 in 2024-09-20T22:50:28.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:28 smithi084 ceph-mon[40024]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:50:28.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:28 smithi084 ceph-mon[40024]: osdmap e548: 8 total, 8 up, 8 in 2024-09-20T22:50:28.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:28 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/4195890921' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T22:50:28.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:28 smithi084 ceph-mon[29774]: osdmap e547: 8 total, 8 up, 8 in 2024-09-20T22:50:28.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:28 smithi084 ceph-mon[29774]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:50:28.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:28 smithi084 ceph-mon[29774]: osdmap e548: 8 total, 8 up, 8 in 2024-09-20T22:50:28.635 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:50:28.658 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:28.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:28.659 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:50:28.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:28 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/4195890921' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T22:50:28.785 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:28 smithi154 ceph-mon[33264]: osdmap e547: 8 total, 8 up, 8 in 2024-09-20T22:50:28.785 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:28 smithi154 ceph-mon[33264]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:50:28.785 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:28 smithi154 ceph-mon[33264]: osdmap e548: 8 total, 8 up, 8 in 2024-09-20T22:50:28.785 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:50:28.811 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:28.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:28.812 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:50:28.871 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.0 dump_blocked_ops 2024-09-20T22:50:28.906 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:50:28.913 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:50:28.936 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:28.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:28.937 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:50:29.038 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:50:29.064 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:29.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:29.065 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:50:29.165 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:50:29.191 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:29.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:29.192 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:50:29.254 INFO:tasks.rados.rados.0.smithi084.stdout:2141: done (0 left) 2024-09-20T22:50:29.254 INFO:tasks.rados.rados.0.smithi084.stdout:2142: copy_from oid 20 from oid 46 current snap is 238 2024-09-20T22:50:29.254 INFO:tasks.rados.rados.0.smithi084.stdout:2143: write (excl) oid 35 current snap is 238 2024-09-20T22:50:29.254 INFO:tasks.rados.rados.0.smithi084.stdout:2143: seq_num 747 ranges {554829=413597,1396049=705252} 2024-09-20T22:50:29.260 INFO:tasks.rados.rados.0.smithi084.stdout:2143: writing smithi08477203-35 from 554829 to 968426 tid 1 2024-09-20T22:50:29.267 INFO:tasks.rados.rados.0.smithi084.stdout:2143: writing smithi08477203-35 from 1396049 to 2101301 tid 2 2024-09-20T22:50:29.267 INFO:tasks.rados.rados.0.smithi084.stdout:2144: copy_from oid 22 from oid 45 current snap is 238 2024-09-20T22:50:29.267 INFO:tasks.rados.rados.0.smithi084.stdout:2145: write oid 1 current snap is 238 2024-09-20T22:50:29.267 INFO:tasks.rados.rados.0.smithi084.stdout:2145: seq_num 748 ranges {571636=488846,1854531=565263} 2024-09-20T22:50:29.273 INFO:tasks.rados.rados.0.smithi084.stdout:2145: writing smithi08477203-1 from 571636 to 1060482 tid 1 2024-09-20T22:50:29.281 INFO:tasks.rados.rados.0.smithi084.stdout:2145: writing smithi08477203-1 from 1854531 to 2419794 tid 2 2024-09-20T22:50:29.281 INFO:tasks.rados.rados.0.smithi084.stdout:2146: write oid 32 current snap is 238 2024-09-20T22:50:29.282 INFO:tasks.rados.rados.0.smithi084.stdout:2143: finishing write tid 1 to smithi08477203-35 2024-09-20T22:50:29.282 INFO:tasks.rados.rados.0.smithi084.stdout:2143: finishing write tid 2 to smithi08477203-35 2024-09-20T22:50:29.282 INFO:tasks.rados.rados.0.smithi084.stdout:2143: finishing write tid 3 to smithi08477203-35 2024-09-20T22:50:29.282 INFO:tasks.rados.rados.0.smithi084.stdout:2143: finishing write tid 4 to smithi08477203-35 2024-09-20T22:50:29.282 INFO:tasks.rados.rados.0.smithi084.stdout:2143: oid 35 updating version 0 to 730 2024-09-20T22:50:29.282 INFO:tasks.rados.rados.0.smithi084.stdout:2143: oid 35 version 730 is already newer than 728 2024-09-20T22:50:29.282 INFO:tasks.rados.rados.0.smithi084.stdout:2143: oid 35 version 730 is already newer than 729 2024-09-20T22:50:29.282 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 35 v 730 (ObjNum 747 snap 238 seq_num 747) dirty exists 2024-09-20T22:50:29.282 INFO:tasks.rados.rados.0.smithi084.stdout:2143: left oid 35 (ObjNum 747 snap 238 seq_num 747) 2024-09-20T22:50:29.282 INFO:tasks.rados.rados.0.smithi084.stdout:2145: finishing write tid 1 to smithi08477203-1 2024-09-20T22:50:29.282 INFO:tasks.rados.rados.0.smithi084.stdout:2144: finishing copy_from racing read to smithi08477203-22 2024-09-20T22:50:29.282 INFO:tasks.rados.rados.0.smithi084.stdout:2142: finishing copy_from racing read to smithi08477203-20 2024-09-20T22:50:29.282 INFO:tasks.rados.rados.0.smithi084.stdout:2146: seq_num 749 ranges {588443=564095,1796660=708921} 2024-09-20T22:50:29.287 INFO:tasks.rados.rados.0.smithi084.stdout:2146: writing smithi08477203-32 from 588443 to 1152538 tid 1 2024-09-20T22:50:29.293 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:50:29.295 INFO:tasks.rados.rados.0.smithi084.stdout:2146: writing smithi08477203-32 from 1796660 to 2505581 tid 2 2024-09-20T22:50:29.295 INFO:tasks.rados.rados.0.smithi084.stdout:2143: done (4 left) 2024-09-20T22:50:29.296 INFO:tasks.rados.rados.0.smithi084.stdout:2147: snap_remove snap 233 2024-09-20T22:50:29.315 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:29.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:29.316 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:50:29.416 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:50:29.439 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:29.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:29.439 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:50:29.469 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:50:29.502 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:50:29.540 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:50:29.563 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:29.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:29.564 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:50:29.583 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:29 smithi084 ceph-mon[40024]: pgmap v723: 158 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 153 active+clean; 497 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-20T22:50:29.583 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:29 smithi084 ceph-mon[40024]: osdmap e549: 8 total, 8 up, 8 in 2024-09-20T22:50:29.583 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:29 smithi084 ceph-mon[29774]: pgmap v723: 158 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 153 active+clean; 497 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-20T22:50:29.583 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:29 smithi084 ceph-mon[29774]: osdmap e549: 8 total, 8 up, 8 in 2024-09-20T22:50:29.612 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-20T22:50:29.612 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-20T22:50:29.612 INFO:teuthology.orchestra.run.smithi084.stdout: "epoch": 549, 2024-09-20T22:50:29.612 INFO:teuthology.orchestra.run.smithi084.stdout: "fsid": "a9c0bc70-77a0-11ef-baf5-efdc52797490", 2024-09-20T22:50:29.612 INFO:teuthology.orchestra.run.smithi084.stdout: "created": "2024-09-20T22:36:30.240626+0000", 2024-09-20T22:50:29.612 INFO:teuthology.orchestra.run.smithi084.stdout: "modified": "2024-09-20T22:50:29.243921+0000", 2024-09-20T22:50:29.613 INFO:teuthology.orchestra.run.smithi084.stdout: "last_up_change": "2024-09-20T22:41:27.098565+0000", 2024-09-20T22:50:29.613 INFO:teuthology.orchestra.run.smithi084.stdout: "last_in_change": "2024-09-20T22:48:04.201694+0000", 2024-09-20T22:50:29.613 INFO:teuthology.orchestra.run.smithi084.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-20T22:50:29.613 INFO:teuthology.orchestra.run.smithi084.stdout: "flags_num": 5806080, 2024-09-20T22:50:29.613 INFO:teuthology.orchestra.run.smithi084.stdout: "flags_set": [ 2024-09-20T22:50:29.613 INFO:teuthology.orchestra.run.smithi084.stdout: "nodeep-scrub", 2024-09-20T22:50:29.613 INFO:teuthology.orchestra.run.smithi084.stdout: "noscrub", 2024-09-20T22:50:29.613 INFO:teuthology.orchestra.run.smithi084.stdout: "pglog_hardlimit", 2024-09-20T22:50:29.613 INFO:teuthology.orchestra.run.smithi084.stdout: "purged_snapdirs", 2024-09-20T22:50:29.613 INFO:teuthology.orchestra.run.smithi084.stdout: "recovery_deletes", 2024-09-20T22:50:29.613 INFO:teuthology.orchestra.run.smithi084.stdout: "sortbitwise" 2024-09-20T22:50:29.613 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2024-09-20T22:50:29.613 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_version": 18, 2024-09-20T22:50:29.613 INFO:teuthology.orchestra.run.smithi084.stdout: "full_ratio": 0.94999998807907104, 2024-09-20T22:50:29.613 INFO:teuthology.orchestra.run.smithi084.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-20T22:50:29.614 INFO:teuthology.orchestra.run.smithi084.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-20T22:50:29.614 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_snapshot": "", 2024-09-20T22:50:29.614 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_max": 7, 2024-09-20T22:50:29.614 INFO:teuthology.orchestra.run.smithi084.stdout: "max_osd": 8, 2024-09-20T22:50:29.614 INFO:teuthology.orchestra.run.smithi084.stdout: "require_min_compat_client": "luminous", 2024-09-20T22:50:29.614 INFO:teuthology.orchestra.run.smithi084.stdout: "min_compat_client": "luminous", 2024-09-20T22:50:29.614 INFO:teuthology.orchestra.run.smithi084.stdout: "require_osd_release": "squid", 2024-09-20T22:50:29.614 INFO:teuthology.orchestra.run.smithi084.stdout: "allow_crimson": false, 2024-09-20T22:50:29.614 INFO:teuthology.orchestra.run.smithi084.stdout: "pools": [ 2024-09-20T22:50:29.614 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:29.614 INFO:teuthology.orchestra.run.smithi084.stdout: "pool": 1, 2024-09-20T22:50:29.614 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_name": ".mgr", 2024-09-20T22:50:29.614 INFO:teuthology.orchestra.run.smithi084.stdout: "create_time": "2024-09-20T22:39:21.984381+0000", 2024-09-20T22:50:29.615 INFO:teuthology.orchestra.run.smithi084.stdout: "flags": 1, 2024-09-20T22:50:29.615 INFO:teuthology.orchestra.run.smithi084.stdout: "flags_names": "hashpspool", 2024-09-20T22:50:29.615 INFO:teuthology.orchestra.run.smithi084.stdout: "type": 1, 2024-09-20T22:50:29.615 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 3, 2024-09-20T22:50:29.615 INFO:teuthology.orchestra.run.smithi084.stdout: "min_size": 2, 2024-09-20T22:50:29.615 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_rule": 0, 2024-09-20T22:50:29.615 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_count": 0, 2024-09-20T22:50:29.615 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_target": 0, 2024-09-20T22:50:29.615 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_barrier": 0, 2024-09-20T22:50:29.615 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-20T22:50:29.615 INFO:teuthology.orchestra.run.smithi084.stdout: "is_stretch_pool": false, 2024-09-20T22:50:29.615 INFO:teuthology.orchestra.run.smithi084.stdout: "object_hash": 2, 2024-09-20T22:50:29.615 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_autoscale_mode": "off", 2024-09-20T22:50:29.615 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num": 11, 2024-09-20T22:50:29.616 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num": 11, 2024-09-20T22:50:29.616 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num_target": 11, 2024-09-20T22:50:29.616 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_target": 11, 2024-09-20T22:50:29.616 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_pending": 11, 2024-09-20T22:50:29.616 INFO:teuthology.orchestra.run.smithi084.stdout: "last_pg_merge_meta": { 2024-09-20T22:50:29.616 INFO:teuthology.orchestra.run.smithi084.stdout: "source_pgid": "1.b", 2024-09-20T22:50:29.616 INFO:teuthology.orchestra.run.smithi084.stdout: "ready_epoch": 299, 2024-09-20T22:50:29.616 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_started": 298, 2024-09-20T22:50:29.616 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_clean": 298, 2024-09-20T22:50:29.616 INFO:teuthology.orchestra.run.smithi084.stdout: "source_version": "77'128", 2024-09-20T22:50:29.616 INFO:teuthology.orchestra.run.smithi084.stdout: "target_version": "77'128" 2024-09-20T22:50:29.616 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:29.616 INFO:teuthology.orchestra.run.smithi084.stdout: "last_change": "542", 2024-09-20T22:50:29.616 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend": "0", 2024-09-20T22:50:29.617 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_prenautilus": "299", 2024-09-20T22:50:29.617 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_preluminous": "297", 2024-09-20T22:50:29.617 INFO:teuthology.orchestra.run.smithi084.stdout: "auid": 0, 2024-09-20T22:50:29.617 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_mode": "selfmanaged", 2024-09-20T22:50:29.617 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_seq": 0, 2024-09-20T22:50:29.617 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_epoch": 0, 2024-09-20T22:50:29.617 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_snaps": [], 2024-09-20T22:50:29.617 INFO:teuthology.orchestra.run.smithi084.stdout: "removed_snaps": "[]", 2024-09-20T22:50:29.617 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_bytes": 0, 2024-09-20T22:50:29.617 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_objects": 0, 2024-09-20T22:50:29.617 INFO:teuthology.orchestra.run.smithi084.stdout: "tiers": [], 2024-09-20T22:50:29.617 INFO:teuthology.orchestra.run.smithi084.stdout: "tier_of": -1, 2024-09-20T22:50:29.617 INFO:teuthology.orchestra.run.smithi084.stdout: "read_tier": -1, 2024-09-20T22:50:29.617 INFO:teuthology.orchestra.run.smithi084.stdout: "write_tier": -1, 2024-09-20T22:50:29.617 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_mode": "none", 2024-09-20T22:50:29.618 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_bytes": 0, 2024-09-20T22:50:29.618 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_objects": 0, 2024-09-20T22:50:29.618 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-20T22:50:29.618 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-20T22:50:29.618 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-20T22:50:29.618 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_flush_age": 0, 2024-09-20T22:50:29.618 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_evict_age": 0, 2024-09-20T22:50:29.618 INFO:teuthology.orchestra.run.smithi084.stdout: "erasure_code_profile": "", 2024-09-20T22:50:29.618 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_params": { 2024-09-20T22:50:29.618 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "none" 2024-09-20T22:50:29.618 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:29.618 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_period": 0, 2024-09-20T22:50:29.618 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_count": 0, 2024-09-20T22:50:29.618 INFO:teuthology.orchestra.run.smithi084.stdout: "use_gmt_hitset": true, 2024-09-20T22:50:29.619 INFO:teuthology.orchestra.run.smithi084.stdout: "min_read_recency_for_promote": 0, 2024-09-20T22:50:29.619 INFO:teuthology.orchestra.run.smithi084.stdout: "min_write_recency_for_promote": 0, 2024-09-20T22:50:29.619 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_grade_decay_rate": 0, 2024-09-20T22:50:29.619 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_search_last_n": 0, 2024-09-20T22:50:29.619 INFO:teuthology.orchestra.run.smithi084.stdout: "grade_table": [], 2024-09-20T22:50:29.619 INFO:teuthology.orchestra.run.smithi084.stdout: "stripe_width": 0, 2024-09-20T22:50:29.619 INFO:teuthology.orchestra.run.smithi084.stdout: "expected_num_objects": 0, 2024-09-20T22:50:29.619 INFO:teuthology.orchestra.run.smithi084.stdout: "fast_read": false, 2024-09-20T22:50:29.619 INFO:teuthology.orchestra.run.smithi084.stdout: "options": { 2024-09-20T22:50:29.619 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_max": 32, 2024-09-20T22:50:29.619 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_min": 1 2024-09-20T22:50:29.620 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:29.620 INFO:teuthology.orchestra.run.smithi084.stdout: "application_metadata": { 2024-09-20T22:50:29.620 INFO:teuthology.orchestra.run.smithi084.stdout: "mgr": {} 2024-09-20T22:50:29.620 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:29.620 INFO:teuthology.orchestra.run.smithi084.stdout: "read_balance": { 2024-09-20T22:50:29.620 INFO:teuthology.orchestra.run.smithi084.stdout: "score_type": "Fair distribution", 2024-09-20T22:50:29.620 INFO:teuthology.orchestra.run.smithi084.stdout: "score_acting": 2.1800000667572021, 2024-09-20T22:50:29.620 INFO:teuthology.orchestra.run.smithi084.stdout: "score_stable": 2.1800000667572021, 2024-09-20T22:50:29.620 INFO:teuthology.orchestra.run.smithi084.stdout: "optimal_score": 1, 2024-09-20T22:50:29.620 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_acting": 2.1800000667572021, 2024-09-20T22:50:29.620 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_stable": 2.1800000667572021, 2024-09-20T22:50:29.620 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity_weighted": 1, 2024-09-20T22:50:29.620 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity": 1, 2024-09-20T22:50:29.620 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity_weighted": 1 2024-09-20T22:50:29.620 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:50:29.621 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:29.621 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:29.621 INFO:teuthology.orchestra.run.smithi084.stdout: "pool": 2, 2024-09-20T22:50:29.621 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_name": "datapool", 2024-09-20T22:50:29.621 INFO:teuthology.orchestra.run.smithi084.stdout: "create_time": "2024-09-20T22:41:35.820081+0000", 2024-09-20T22:50:29.621 INFO:teuthology.orchestra.run.smithi084.stdout: "flags": 8193, 2024-09-20T22:50:29.621 INFO:teuthology.orchestra.run.smithi084.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-20T22:50:29.621 INFO:teuthology.orchestra.run.smithi084.stdout: "type": 1, 2024-09-20T22:50:29.621 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 3, 2024-09-20T22:50:29.621 INFO:teuthology.orchestra.run.smithi084.stdout: "min_size": 2, 2024-09-20T22:50:29.621 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_rule": 0, 2024-09-20T22:50:29.621 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_count": 0, 2024-09-20T22:50:29.621 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_target": 0, 2024-09-20T22:50:29.621 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_barrier": 0, 2024-09-20T22:50:29.621 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-20T22:50:29.622 INFO:teuthology.orchestra.run.smithi084.stdout: "is_stretch_pool": false, 2024-09-20T22:50:29.622 INFO:teuthology.orchestra.run.smithi084.stdout: "object_hash": 2, 2024-09-20T22:50:29.622 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_autoscale_mode": "off", 2024-09-20T22:50:29.622 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num": 3, 2024-09-20T22:50:29.622 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num": 3, 2024-09-20T22:50:29.622 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num_target": 3, 2024-09-20T22:50:29.622 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_target": 3, 2024-09-20T22:50:29.622 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_pending": 3, 2024-09-20T22:50:29.622 INFO:teuthology.orchestra.run.smithi084.stdout: "last_pg_merge_meta": { 2024-09-20T22:50:29.622 INFO:teuthology.orchestra.run.smithi084.stdout: "source_pgid": "0.0", 2024-09-20T22:50:29.622 INFO:teuthology.orchestra.run.smithi084.stdout: "ready_epoch": 0, 2024-09-20T22:50:29.622 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_started": 0, 2024-09-20T22:50:29.622 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_clean": 0, 2024-09-20T22:50:29.622 INFO:teuthology.orchestra.run.smithi084.stdout: "source_version": "0'0", 2024-09-20T22:50:29.623 INFO:teuthology.orchestra.run.smithi084.stdout: "target_version": "0'0" 2024-09-20T22:50:29.623 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:29.623 INFO:teuthology.orchestra.run.smithi084.stdout: "last_change": "69", 2024-09-20T22:50:29.623 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend": "0", 2024-09-20T22:50:29.623 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-20T22:50:29.623 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_preluminous": "0", 2024-09-20T22:50:29.623 INFO:teuthology.orchestra.run.smithi084.stdout: "auid": 0, 2024-09-20T22:50:29.623 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_mode": "selfmanaged", 2024-09-20T22:50:29.623 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_seq": 2, 2024-09-20T22:50:29.623 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_epoch": 69, 2024-09-20T22:50:29.623 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_snaps": [], 2024-09-20T22:50:29.623 INFO:teuthology.orchestra.run.smithi084.stdout: "removed_snaps": "[]", 2024-09-20T22:50:29.623 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_bytes": 0, 2024-09-20T22:50:29.623 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_objects": 0, 2024-09-20T22:50:29.624 INFO:teuthology.orchestra.run.smithi084.stdout: "tiers": [], 2024-09-20T22:50:29.624 INFO:teuthology.orchestra.run.smithi084.stdout: "tier_of": -1, 2024-09-20T22:50:29.624 INFO:teuthology.orchestra.run.smithi084.stdout: "read_tier": -1, 2024-09-20T22:50:29.624 INFO:teuthology.orchestra.run.smithi084.stdout: "write_tier": -1, 2024-09-20T22:50:29.624 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_mode": "none", 2024-09-20T22:50:29.624 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_bytes": 0, 2024-09-20T22:50:29.624 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_objects": 0, 2024-09-20T22:50:29.624 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-20T22:50:29.624 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-20T22:50:29.624 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-20T22:50:29.624 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_flush_age": 0, 2024-09-20T22:50:29.624 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_evict_age": 0, 2024-09-20T22:50:29.624 INFO:teuthology.orchestra.run.smithi084.stdout: "erasure_code_profile": "", 2024-09-20T22:50:29.624 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_params": { 2024-09-20T22:50:29.624 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "none" 2024-09-20T22:50:29.625 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:29.625 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_period": 0, 2024-09-20T22:50:29.625 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_count": 0, 2024-09-20T22:50:29.625 INFO:teuthology.orchestra.run.smithi084.stdout: "use_gmt_hitset": true, 2024-09-20T22:50:29.625 INFO:teuthology.orchestra.run.smithi084.stdout: "min_read_recency_for_promote": 0, 2024-09-20T22:50:29.625 INFO:teuthology.orchestra.run.smithi084.stdout: "min_write_recency_for_promote": 0, 2024-09-20T22:50:29.625 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_grade_decay_rate": 0, 2024-09-20T22:50:29.625 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_search_last_n": 0, 2024-09-20T22:50:29.625 INFO:teuthology.orchestra.run.smithi084.stdout: "grade_table": [], 2024-09-20T22:50:29.625 INFO:teuthology.orchestra.run.smithi084.stdout: "stripe_width": 0, 2024-09-20T22:50:29.625 INFO:teuthology.orchestra.run.smithi084.stdout: "expected_num_objects": 0, 2024-09-20T22:50:29.625 INFO:teuthology.orchestra.run.smithi084.stdout: "fast_read": false, 2024-09-20T22:50:29.625 INFO:teuthology.orchestra.run.smithi084.stdout: "options": {}, 2024-09-20T22:50:29.625 INFO:teuthology.orchestra.run.smithi084.stdout: "application_metadata": { 2024-09-20T22:50:29.625 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd": {} 2024-09-20T22:50:29.626 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:29.626 INFO:teuthology.orchestra.run.smithi084.stdout: "read_balance": { 2024-09-20T22:50:29.626 INFO:teuthology.orchestra.run.smithi084.stdout: "score_type": "Fair distribution", 2024-09-20T22:50:29.626 INFO:teuthology.orchestra.run.smithi084.stdout: "score_acting": 2.6500000953674316, 2024-09-20T22:50:29.626 INFO:teuthology.orchestra.run.smithi084.stdout: "score_stable": 2.6500000953674316, 2024-09-20T22:50:29.626 INFO:teuthology.orchestra.run.smithi084.stdout: "optimal_score": 0.87999999523162842, 2024-09-20T22:50:29.626 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_acting": 2.3299999237060547, 2024-09-20T22:50:29.626 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_stable": 2.3299999237060547, 2024-09-20T22:50:29.626 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity_weighted": 1, 2024-09-20T22:50:29.626 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity": 1, 2024-09-20T22:50:29.626 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity_weighted": 1 2024-09-20T22:50:29.626 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:50:29.627 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:29.627 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:29.627 INFO:teuthology.orchestra.run.smithi084.stdout: "pool": 3, 2024-09-20T22:50:29.627 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_name": ".rgw.root", 2024-09-20T22:50:29.627 INFO:teuthology.orchestra.run.smithi084.stdout: "create_time": "2024-09-20T22:41:36.546389+0000", 2024-09-20T22:50:29.627 INFO:teuthology.orchestra.run.smithi084.stdout: "flags": 1, 2024-09-20T22:50:29.627 INFO:teuthology.orchestra.run.smithi084.stdout: "flags_names": "hashpspool", 2024-09-20T22:50:29.627 INFO:teuthology.orchestra.run.smithi084.stdout: "type": 1, 2024-09-20T22:50:29.627 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 3, 2024-09-20T22:50:29.627 INFO:teuthology.orchestra.run.smithi084.stdout: "min_size": 2, 2024-09-20T22:50:29.627 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_rule": 0, 2024-09-20T22:50:29.627 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_count": 0, 2024-09-20T22:50:29.627 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_target": 0, 2024-09-20T22:50:29.627 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_barrier": 0, 2024-09-20T22:50:29.627 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-20T22:50:29.628 INFO:teuthology.orchestra.run.smithi084.stdout: "is_stretch_pool": false, 2024-09-20T22:50:29.628 INFO:teuthology.orchestra.run.smithi084.stdout: "object_hash": 2, 2024-09-20T22:50:29.628 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_autoscale_mode": "off", 2024-09-20T22:50:29.628 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num": 32, 2024-09-20T22:50:29.628 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num": 32, 2024-09-20T22:50:29.628 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num_target": 32, 2024-09-20T22:50:29.628 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_target": 32, 2024-09-20T22:50:29.628 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_pending": 32, 2024-09-20T22:50:29.628 INFO:teuthology.orchestra.run.smithi084.stdout: "last_pg_merge_meta": { 2024-09-20T22:50:29.628 INFO:teuthology.orchestra.run.smithi084.stdout: "source_pgid": "0.0", 2024-09-20T22:50:29.628 INFO:teuthology.orchestra.run.smithi084.stdout: "ready_epoch": 0, 2024-09-20T22:50:29.629 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_started": 0, 2024-09-20T22:50:29.629 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_clean": 0, 2024-09-20T22:50:29.629 INFO:teuthology.orchestra.run.smithi084.stdout: "source_version": "0'0", 2024-09-20T22:50:29.629 INFO:teuthology.orchestra.run.smithi084.stdout: "target_version": "0'0" 2024-09-20T22:50:29.629 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:29.629 INFO:teuthology.orchestra.run.smithi084.stdout: "last_change": "68", 2024-09-20T22:50:29.629 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend": "0", 2024-09-20T22:50:29.629 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-20T22:50:29.629 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_preluminous": "0", 2024-09-20T22:50:29.630 INFO:teuthology.orchestra.run.smithi084.stdout: "auid": 0, 2024-09-20T22:50:29.630 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_mode": "selfmanaged", 2024-09-20T22:50:29.630 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_seq": 0, 2024-09-20T22:50:29.630 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_epoch": 0, 2024-09-20T22:50:29.630 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_snaps": [], 2024-09-20T22:50:29.630 INFO:teuthology.orchestra.run.smithi084.stdout: "removed_snaps": "[]", 2024-09-20T22:50:29.630 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_bytes": 0, 2024-09-20T22:50:29.630 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_objects": 0, 2024-09-20T22:50:29.630 INFO:teuthology.orchestra.run.smithi084.stdout: "tiers": [], 2024-09-20T22:50:29.630 INFO:teuthology.orchestra.run.smithi084.stdout: "tier_of": -1, 2024-09-20T22:50:29.630 INFO:teuthology.orchestra.run.smithi084.stdout: "read_tier": -1, 2024-09-20T22:50:29.630 INFO:teuthology.orchestra.run.smithi084.stdout: "write_tier": -1, 2024-09-20T22:50:29.630 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_mode": "none", 2024-09-20T22:50:29.630 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_bytes": 0, 2024-09-20T22:50:29.631 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_objects": 0, 2024-09-20T22:50:29.631 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-20T22:50:29.631 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-20T22:50:29.631 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-20T22:50:29.631 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_flush_age": 0, 2024-09-20T22:50:29.631 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_evict_age": 0, 2024-09-20T22:50:29.631 INFO:teuthology.orchestra.run.smithi084.stdout: "erasure_code_profile": "", 2024-09-20T22:50:29.631 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_params": { 2024-09-20T22:50:29.631 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "none" 2024-09-20T22:50:29.631 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:29.631 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_period": 0, 2024-09-20T22:50:29.631 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_count": 0, 2024-09-20T22:50:29.631 INFO:teuthology.orchestra.run.smithi084.stdout: "use_gmt_hitset": true, 2024-09-20T22:50:29.631 INFO:teuthology.orchestra.run.smithi084.stdout: "min_read_recency_for_promote": 0, 2024-09-20T22:50:29.632 INFO:teuthology.orchestra.run.smithi084.stdout: "min_write_recency_for_promote": 0, 2024-09-20T22:50:29.632 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_grade_decay_rate": 0, 2024-09-20T22:50:29.632 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_search_last_n": 0, 2024-09-20T22:50:29.632 INFO:teuthology.orchestra.run.smithi084.stdout: "grade_table": [], 2024-09-20T22:50:29.632 INFO:teuthology.orchestra.run.smithi084.stdout: "stripe_width": 0, 2024-09-20T22:50:29.632 INFO:teuthology.orchestra.run.smithi084.stdout: "expected_num_objects": 0, 2024-09-20T22:50:29.632 INFO:teuthology.orchestra.run.smithi084.stdout: "fast_read": false, 2024-09-20T22:50:29.632 INFO:teuthology.orchestra.run.smithi084.stdout: "options": {}, 2024-09-20T22:50:29.632 INFO:teuthology.orchestra.run.smithi084.stdout: "application_metadata": { 2024-09-20T22:50:29.632 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw": {} 2024-09-20T22:50:29.632 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:29.632 INFO:teuthology.orchestra.run.smithi084.stdout: "read_balance": { 2024-09-20T22:50:29.632 INFO:teuthology.orchestra.run.smithi084.stdout: "score_type": "Fair distribution", 2024-09-20T22:50:29.632 INFO:teuthology.orchestra.run.smithi084.stdout: "score_acting": 1.5, 2024-09-20T22:50:29.633 INFO:teuthology.orchestra.run.smithi084.stdout: "score_stable": 1.5, 2024-09-20T22:50:29.633 INFO:teuthology.orchestra.run.smithi084.stdout: "optimal_score": 1, 2024-09-20T22:50:29.633 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_acting": 1.5, 2024-09-20T22:50:29.633 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_stable": 1.5, 2024-09-20T22:50:29.633 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity_weighted": 1, 2024-09-20T22:50:29.633 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity": 1, 2024-09-20T22:50:29.633 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity_weighted": 1 2024-09-20T22:50:29.633 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:50:29.633 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:29.633 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:29.633 INFO:teuthology.orchestra.run.smithi084.stdout: "pool": 4, 2024-09-20T22:50:29.633 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_name": "default.rgw.log", 2024-09-20T22:50:29.634 INFO:teuthology.orchestra.run.smithi084.stdout: "create_time": "2024-09-20T22:41:38.454640+0000", 2024-09-20T22:50:29.634 INFO:teuthology.orchestra.run.smithi084.stdout: "flags": 1, 2024-09-20T22:50:29.634 INFO:teuthology.orchestra.run.smithi084.stdout: "flags_names": "hashpspool", 2024-09-20T22:50:29.634 INFO:teuthology.orchestra.run.smithi084.stdout: "type": 1, 2024-09-20T22:50:29.634 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 3, 2024-09-20T22:50:29.634 INFO:teuthology.orchestra.run.smithi084.stdout: "min_size": 2, 2024-09-20T22:50:29.634 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_rule": 0, 2024-09-20T22:50:29.634 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_count": 0, 2024-09-20T22:50:29.634 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_target": 0, 2024-09-20T22:50:29.634 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_barrier": 0, 2024-09-20T22:50:29.634 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-20T22:50:29.634 INFO:teuthology.orchestra.run.smithi084.stdout: "is_stretch_pool": false, 2024-09-20T22:50:29.635 INFO:teuthology.orchestra.run.smithi084.stdout: "object_hash": 2, 2024-09-20T22:50:29.635 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_autoscale_mode": "off", 2024-09-20T22:50:29.635 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num": 32, 2024-09-20T22:50:29.635 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num": 32, 2024-09-20T22:50:29.635 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num_target": 32, 2024-09-20T22:50:29.635 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_target": 32, 2024-09-20T22:50:29.635 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_pending": 32, 2024-09-20T22:50:29.635 INFO:teuthology.orchestra.run.smithi084.stdout: "last_pg_merge_meta": { 2024-09-20T22:50:29.635 INFO:teuthology.orchestra.run.smithi084.stdout: "source_pgid": "0.0", 2024-09-20T22:50:29.635 INFO:teuthology.orchestra.run.smithi084.stdout: "ready_epoch": 0, 2024-09-20T22:50:29.635 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_started": 0, 2024-09-20T22:50:29.635 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_clean": 0, 2024-09-20T22:50:29.635 INFO:teuthology.orchestra.run.smithi084.stdout: "source_version": "0'0", 2024-09-20T22:50:29.635 INFO:teuthology.orchestra.run.smithi084.stdout: "target_version": "0'0" 2024-09-20T22:50:29.635 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:29.636 INFO:teuthology.orchestra.run.smithi084.stdout: "last_change": "70", 2024-09-20T22:50:29.636 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend": "0", 2024-09-20T22:50:29.636 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-20T22:50:29.636 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_preluminous": "0", 2024-09-20T22:50:29.636 INFO:teuthology.orchestra.run.smithi084.stdout: "auid": 0, 2024-09-20T22:50:29.636 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_mode": "selfmanaged", 2024-09-20T22:50:29.636 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_seq": 0, 2024-09-20T22:50:29.636 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_epoch": 0, 2024-09-20T22:50:29.636 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_snaps": [], 2024-09-20T22:50:29.636 INFO:teuthology.orchestra.run.smithi084.stdout: "removed_snaps": "[]", 2024-09-20T22:50:29.636 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_bytes": 0, 2024-09-20T22:50:29.636 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_objects": 0, 2024-09-20T22:50:29.636 INFO:teuthology.orchestra.run.smithi084.stdout: "tiers": [], 2024-09-20T22:50:29.636 INFO:teuthology.orchestra.run.smithi084.stdout: "tier_of": -1, 2024-09-20T22:50:29.637 INFO:teuthology.orchestra.run.smithi084.stdout: "read_tier": -1, 2024-09-20T22:50:29.637 INFO:teuthology.orchestra.run.smithi084.stdout: "write_tier": -1, 2024-09-20T22:50:29.637 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_mode": "none", 2024-09-20T22:50:29.637 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_bytes": 0, 2024-09-20T22:50:29.637 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_objects": 0, 2024-09-20T22:50:29.637 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-20T22:50:29.637 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-20T22:50:29.637 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-20T22:50:29.637 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_flush_age": 0, 2024-09-20T22:50:29.637 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_evict_age": 0, 2024-09-20T22:50:29.637 INFO:teuthology.orchestra.run.smithi084.stdout: "erasure_code_profile": "", 2024-09-20T22:50:29.637 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_params": { 2024-09-20T22:50:29.637 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "none" 2024-09-20T22:50:29.637 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:29.638 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_period": 0, 2024-09-20T22:50:29.638 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_count": 0, 2024-09-20T22:50:29.638 INFO:teuthology.orchestra.run.smithi084.stdout: "use_gmt_hitset": true, 2024-09-20T22:50:29.638 INFO:teuthology.orchestra.run.smithi084.stdout: "min_read_recency_for_promote": 0, 2024-09-20T22:50:29.638 INFO:teuthology.orchestra.run.smithi084.stdout: "min_write_recency_for_promote": 0, 2024-09-20T22:50:29.638 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_grade_decay_rate": 0, 2024-09-20T22:50:29.638 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_search_last_n": 0, 2024-09-20T22:50:29.638 INFO:teuthology.orchestra.run.smithi084.stdout: "grade_table": [], 2024-09-20T22:50:29.638 INFO:teuthology.orchestra.run.smithi084.stdout: "stripe_width": 0, 2024-09-20T22:50:29.638 INFO:teuthology.orchestra.run.smithi084.stdout: "expected_num_objects": 0, 2024-09-20T22:50:29.638 INFO:teuthology.orchestra.run.smithi084.stdout: "fast_read": false, 2024-09-20T22:50:29.638 INFO:teuthology.orchestra.run.smithi084.stdout: "options": {}, 2024-09-20T22:50:29.638 INFO:teuthology.orchestra.run.smithi084.stdout: "application_metadata": { 2024-09-20T22:50:29.638 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw": {} 2024-09-20T22:50:29.639 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:29.639 INFO:teuthology.orchestra.run.smithi084.stdout: "read_balance": { 2024-09-20T22:50:29.639 INFO:teuthology.orchestra.run.smithi084.stdout: "score_type": "Fair distribution", 2024-09-20T22:50:29.639 INFO:teuthology.orchestra.run.smithi084.stdout: "score_acting": 2.25, 2024-09-20T22:50:29.639 INFO:teuthology.orchestra.run.smithi084.stdout: "score_stable": 2.25, 2024-09-20T22:50:29.639 INFO:teuthology.orchestra.run.smithi084.stdout: "optimal_score": 1, 2024-09-20T22:50:29.639 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_acting": 2.25, 2024-09-20T22:50:29.639 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_stable": 2.25, 2024-09-20T22:50:29.639 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity_weighted": 1, 2024-09-20T22:50:29.639 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity": 1, 2024-09-20T22:50:29.639 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity_weighted": 1 2024-09-20T22:50:29.639 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:50:29.639 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:29.639 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:29.640 INFO:teuthology.orchestra.run.smithi084.stdout: "pool": 5, 2024-09-20T22:50:29.640 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_name": "default.rgw.control", 2024-09-20T22:50:29.640 INFO:teuthology.orchestra.run.smithi084.stdout: "create_time": "2024-09-20T22:41:40.407129+0000", 2024-09-20T22:50:29.640 INFO:teuthology.orchestra.run.smithi084.stdout: "flags": 1, 2024-09-20T22:50:29.640 INFO:teuthology.orchestra.run.smithi084.stdout: "flags_names": "hashpspool", 2024-09-20T22:50:29.640 INFO:teuthology.orchestra.run.smithi084.stdout: "type": 1, 2024-09-20T22:50:29.640 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 3, 2024-09-20T22:50:29.640 INFO:teuthology.orchestra.run.smithi084.stdout: "min_size": 2, 2024-09-20T22:50:29.640 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_rule": 0, 2024-09-20T22:50:29.640 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_count": 0, 2024-09-20T22:50:29.640 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_target": 0, 2024-09-20T22:50:29.640 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_barrier": 0, 2024-09-20T22:50:29.640 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-20T22:50:29.641 INFO:teuthology.orchestra.run.smithi084.stdout: "is_stretch_pool": false, 2024-09-20T22:50:29.641 INFO:teuthology.orchestra.run.smithi084.stdout: "object_hash": 2, 2024-09-20T22:50:29.641 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_autoscale_mode": "off", 2024-09-20T22:50:29.641 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num": 32, 2024-09-20T22:50:29.641 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num": 32, 2024-09-20T22:50:29.641 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num_target": 32, 2024-09-20T22:50:29.641 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_target": 32, 2024-09-20T22:50:29.641 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_pending": 32, 2024-09-20T22:50:29.641 INFO:teuthology.orchestra.run.smithi084.stdout: "last_pg_merge_meta": { 2024-09-20T22:50:29.641 INFO:teuthology.orchestra.run.smithi084.stdout: "source_pgid": "0.0", 2024-09-20T22:50:29.641 INFO:teuthology.orchestra.run.smithi084.stdout: "ready_epoch": 0, 2024-09-20T22:50:29.641 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_started": 0, 2024-09-20T22:50:29.641 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_clean": 0, 2024-09-20T22:50:29.641 INFO:teuthology.orchestra.run.smithi084.stdout: "source_version": "0'0", 2024-09-20T22:50:29.641 INFO:teuthology.orchestra.run.smithi084.stdout: "target_version": "0'0" 2024-09-20T22:50:29.642 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:29.642 INFO:teuthology.orchestra.run.smithi084.stdout: "last_change": "72", 2024-09-20T22:50:29.642 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend": "0", 2024-09-20T22:50:29.642 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-20T22:50:29.642 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_preluminous": "0", 2024-09-20T22:50:29.642 INFO:teuthology.orchestra.run.smithi084.stdout: "auid": 0, 2024-09-20T22:50:29.642 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_mode": "selfmanaged", 2024-09-20T22:50:29.642 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_seq": 0, 2024-09-20T22:50:29.642 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_epoch": 0, 2024-09-20T22:50:29.642 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_snaps": [], 2024-09-20T22:50:29.642 INFO:teuthology.orchestra.run.smithi084.stdout: "removed_snaps": "[]", 2024-09-20T22:50:29.642 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_bytes": 0, 2024-09-20T22:50:29.642 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_objects": 0, 2024-09-20T22:50:29.642 INFO:teuthology.orchestra.run.smithi084.stdout: "tiers": [], 2024-09-20T22:50:29.643 INFO:teuthology.orchestra.run.smithi084.stdout: "tier_of": -1, 2024-09-20T22:50:29.643 INFO:teuthology.orchestra.run.smithi084.stdout: "read_tier": -1, 2024-09-20T22:50:29.643 INFO:teuthology.orchestra.run.smithi084.stdout: "write_tier": -1, 2024-09-20T22:50:29.643 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_mode": "none", 2024-09-20T22:50:29.643 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_bytes": 0, 2024-09-20T22:50:29.643 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_objects": 0, 2024-09-20T22:50:29.643 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-20T22:50:29.643 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-20T22:50:29.643 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-20T22:50:29.643 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_flush_age": 0, 2024-09-20T22:50:29.643 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_evict_age": 0, 2024-09-20T22:50:29.643 INFO:teuthology.orchestra.run.smithi084.stdout: "erasure_code_profile": "", 2024-09-20T22:50:29.643 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_params": { 2024-09-20T22:50:29.643 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "none" 2024-09-20T22:50:29.644 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:29.644 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_period": 0, 2024-09-20T22:50:29.644 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_count": 0, 2024-09-20T22:50:29.644 INFO:teuthology.orchestra.run.smithi084.stdout: "use_gmt_hitset": true, 2024-09-20T22:50:29.644 INFO:teuthology.orchestra.run.smithi084.stdout: "min_read_recency_for_promote": 0, 2024-09-20T22:50:29.644 INFO:teuthology.orchestra.run.smithi084.stdout: "min_write_recency_for_promote": 0, 2024-09-20T22:50:29.644 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_grade_decay_rate": 0, 2024-09-20T22:50:29.644 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_search_last_n": 0, 2024-09-20T22:50:29.644 INFO:teuthology.orchestra.run.smithi084.stdout: "grade_table": [], 2024-09-20T22:50:29.644 INFO:teuthology.orchestra.run.smithi084.stdout: "stripe_width": 0, 2024-09-20T22:50:29.644 INFO:teuthology.orchestra.run.smithi084.stdout: "expected_num_objects": 0, 2024-09-20T22:50:29.644 INFO:teuthology.orchestra.run.smithi084.stdout: "fast_read": false, 2024-09-20T22:50:29.644 INFO:teuthology.orchestra.run.smithi084.stdout: "options": {}, 2024-09-20T22:50:29.644 INFO:teuthology.orchestra.run.smithi084.stdout: "application_metadata": { 2024-09-20T22:50:29.645 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw": {} 2024-09-20T22:50:29.645 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:29.645 INFO:teuthology.orchestra.run.smithi084.stdout: "read_balance": { 2024-09-20T22:50:29.645 INFO:teuthology.orchestra.run.smithi084.stdout: "score_type": "Fair distribution", 2024-09-20T22:50:29.645 INFO:teuthology.orchestra.run.smithi084.stdout: "score_acting": 1.25, 2024-09-20T22:50:29.645 INFO:teuthology.orchestra.run.smithi084.stdout: "score_stable": 1.25, 2024-09-20T22:50:29.645 INFO:teuthology.orchestra.run.smithi084.stdout: "optimal_score": 1, 2024-09-20T22:50:29.645 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_acting": 1.25, 2024-09-20T22:50:29.645 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_stable": 1.25, 2024-09-20T22:50:29.645 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity_weighted": 1, 2024-09-20T22:50:29.645 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity": 1, 2024-09-20T22:50:29.645 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity_weighted": 1 2024-09-20T22:50:29.645 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:50:29.645 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:29.646 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:29.646 INFO:teuthology.orchestra.run.smithi084.stdout: "pool": 6, 2024-09-20T22:50:29.646 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_name": "default.rgw.meta", 2024-09-20T22:50:29.646 INFO:teuthology.orchestra.run.smithi084.stdout: "create_time": "2024-09-20T22:41:42.476716+0000", 2024-09-20T22:50:29.646 INFO:teuthology.orchestra.run.smithi084.stdout: "flags": 1, 2024-09-20T22:50:29.646 INFO:teuthology.orchestra.run.smithi084.stdout: "flags_names": "hashpspool", 2024-09-20T22:50:29.646 INFO:teuthology.orchestra.run.smithi084.stdout: "type": 1, 2024-09-20T22:50:29.646 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 3, 2024-09-20T22:50:29.646 INFO:teuthology.orchestra.run.smithi084.stdout: "min_size": 2, 2024-09-20T22:50:29.646 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_rule": 0, 2024-09-20T22:50:29.646 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_count": 0, 2024-09-20T22:50:29.646 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_target": 0, 2024-09-20T22:50:29.647 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_barrier": 0, 2024-09-20T22:50:29.647 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-20T22:50:29.647 INFO:teuthology.orchestra.run.smithi084.stdout: "is_stretch_pool": false, 2024-09-20T22:50:29.647 INFO:teuthology.orchestra.run.smithi084.stdout: "object_hash": 2, 2024-09-20T22:50:29.647 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_autoscale_mode": "off", 2024-09-20T22:50:29.647 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num": 32, 2024-09-20T22:50:29.647 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num": 32, 2024-09-20T22:50:29.647 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num_target": 32, 2024-09-20T22:50:29.647 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_target": 32, 2024-09-20T22:50:29.647 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_pending": 32, 2024-09-20T22:50:29.647 INFO:teuthology.orchestra.run.smithi084.stdout: "last_pg_merge_meta": { 2024-09-20T22:50:29.647 INFO:teuthology.orchestra.run.smithi084.stdout: "source_pgid": "0.0", 2024-09-20T22:50:29.647 INFO:teuthology.orchestra.run.smithi084.stdout: "ready_epoch": 0, 2024-09-20T22:50:29.647 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_started": 0, 2024-09-20T22:50:29.647 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_clean": 0, 2024-09-20T22:50:29.648 INFO:teuthology.orchestra.run.smithi084.stdout: "source_version": "0'0", 2024-09-20T22:50:29.648 INFO:teuthology.orchestra.run.smithi084.stdout: "target_version": "0'0" 2024-09-20T22:50:29.648 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:29.648 INFO:teuthology.orchestra.run.smithi084.stdout: "last_change": "74", 2024-09-20T22:50:29.648 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend": "0", 2024-09-20T22:50:29.648 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-20T22:50:29.648 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_preluminous": "0", 2024-09-20T22:50:29.648 INFO:teuthology.orchestra.run.smithi084.stdout: "auid": 0, 2024-09-20T22:50:29.648 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_mode": "selfmanaged", 2024-09-20T22:50:29.648 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_seq": 0, 2024-09-20T22:50:29.648 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_epoch": 0, 2024-09-20T22:50:29.648 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_snaps": [], 2024-09-20T22:50:29.648 INFO:teuthology.orchestra.run.smithi084.stdout: "removed_snaps": "[]", 2024-09-20T22:50:29.648 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_bytes": 0, 2024-09-20T22:50:29.649 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_objects": 0, 2024-09-20T22:50:29.649 INFO:teuthology.orchestra.run.smithi084.stdout: "tiers": [], 2024-09-20T22:50:29.649 INFO:teuthology.orchestra.run.smithi084.stdout: "tier_of": -1, 2024-09-20T22:50:29.649 INFO:teuthology.orchestra.run.smithi084.stdout: "read_tier": -1, 2024-09-20T22:50:29.649 INFO:teuthology.orchestra.run.smithi084.stdout: "write_tier": -1, 2024-09-20T22:50:29.649 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_mode": "none", 2024-09-20T22:50:29.649 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_bytes": 0, 2024-09-20T22:50:29.649 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_objects": 0, 2024-09-20T22:50:29.649 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-20T22:50:29.649 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-20T22:50:29.649 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-20T22:50:29.649 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_flush_age": 0, 2024-09-20T22:50:29.649 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_evict_age": 0, 2024-09-20T22:50:29.649 INFO:teuthology.orchestra.run.smithi084.stdout: "erasure_code_profile": "", 2024-09-20T22:50:29.650 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_params": { 2024-09-20T22:50:29.650 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "none" 2024-09-20T22:50:29.650 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:29.650 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_period": 0, 2024-09-20T22:50:29.650 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_count": 0, 2024-09-20T22:50:29.650 INFO:teuthology.orchestra.run.smithi084.stdout: "use_gmt_hitset": true, 2024-09-20T22:50:29.650 INFO:teuthology.orchestra.run.smithi084.stdout: "min_read_recency_for_promote": 0, 2024-09-20T22:50:29.650 INFO:teuthology.orchestra.run.smithi084.stdout: "min_write_recency_for_promote": 0, 2024-09-20T22:50:29.650 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_grade_decay_rate": 0, 2024-09-20T22:50:29.650 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_search_last_n": 0, 2024-09-20T22:50:29.650 INFO:teuthology.orchestra.run.smithi084.stdout: "grade_table": [], 2024-09-20T22:50:29.650 INFO:teuthology.orchestra.run.smithi084.stdout: "stripe_width": 0, 2024-09-20T22:50:29.650 INFO:teuthology.orchestra.run.smithi084.stdout: "expected_num_objects": 0, 2024-09-20T22:50:29.650 INFO:teuthology.orchestra.run.smithi084.stdout: "fast_read": false, 2024-09-20T22:50:29.651 INFO:teuthology.orchestra.run.smithi084.stdout: "options": { 2024-09-20T22:50:29.651 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_autoscale_bias": 4 2024-09-20T22:50:29.651 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:29.651 INFO:teuthology.orchestra.run.smithi084.stdout: "application_metadata": { 2024-09-20T22:50:29.651 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw": {} 2024-09-20T22:50:29.651 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:29.651 INFO:teuthology.orchestra.run.smithi084.stdout: "read_balance": { 2024-09-20T22:50:29.651 INFO:teuthology.orchestra.run.smithi084.stdout: "score_type": "Fair distribution", 2024-09-20T22:50:29.651 INFO:teuthology.orchestra.run.smithi084.stdout: "score_acting": 1.75, 2024-09-20T22:50:29.651 INFO:teuthology.orchestra.run.smithi084.stdout: "score_stable": 1.75, 2024-09-20T22:50:29.651 INFO:teuthology.orchestra.run.smithi084.stdout: "optimal_score": 1, 2024-09-20T22:50:29.651 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_acting": 1.75, 2024-09-20T22:50:29.651 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_stable": 1.75, 2024-09-20T22:50:29.651 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity_weighted": 1, 2024-09-20T22:50:29.651 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity": 1, 2024-09-20T22:50:29.652 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity_weighted": 1 2024-09-20T22:50:29.652 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:50:29.652 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:29.652 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:29.652 INFO:teuthology.orchestra.run.smithi084.stdout: "pool": 7, 2024-09-20T22:50:29.652 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_name": "unique_pool_0", 2024-09-20T22:50:29.652 INFO:teuthology.orchestra.run.smithi084.stdout: "create_time": "2024-09-20T22:42:51.354358+0000", 2024-09-20T22:50:29.652 INFO:teuthology.orchestra.run.smithi084.stdout: "flags": 8193, 2024-09-20T22:50:29.652 INFO:teuthology.orchestra.run.smithi084.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-20T22:50:29.652 INFO:teuthology.orchestra.run.smithi084.stdout: "type": 1, 2024-09-20T22:50:29.652 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 3, 2024-09-20T22:50:29.652 INFO:teuthology.orchestra.run.smithi084.stdout: "min_size": 2, 2024-09-20T22:50:29.652 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_rule": 0, 2024-09-20T22:50:29.652 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_count": 0, 2024-09-20T22:50:29.653 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_target": 0, 2024-09-20T22:50:29.653 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_barrier": 0, 2024-09-20T22:50:29.653 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-20T22:50:29.653 INFO:teuthology.orchestra.run.smithi084.stdout: "is_stretch_pool": false, 2024-09-20T22:50:29.653 INFO:teuthology.orchestra.run.smithi084.stdout: "object_hash": 2, 2024-09-20T22:50:29.653 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_autoscale_mode": "off", 2024-09-20T22:50:29.653 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num": 16, 2024-09-20T22:50:29.653 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num": 16, 2024-09-20T22:50:29.653 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num_target": 16, 2024-09-20T22:50:29.653 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_target": 16, 2024-09-20T22:50:29.653 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_pending": 16, 2024-09-20T22:50:29.653 INFO:teuthology.orchestra.run.smithi084.stdout: "last_pg_merge_meta": { 2024-09-20T22:50:29.653 INFO:teuthology.orchestra.run.smithi084.stdout: "source_pgid": "7.10", 2024-09-20T22:50:29.654 INFO:teuthology.orchestra.run.smithi084.stdout: "ready_epoch": 484, 2024-09-20T22:50:29.654 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_started": 483, 2024-09-20T22:50:29.654 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_clean": 483, 2024-09-20T22:50:29.654 INFO:teuthology.orchestra.run.smithi084.stdout: "source_version": "480'172", 2024-09-20T22:50:29.654 INFO:teuthology.orchestra.run.smithi084.stdout: "target_version": "480'253" 2024-09-20T22:50:29.654 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:29.654 INFO:teuthology.orchestra.run.smithi084.stdout: "last_change": "549", 2024-09-20T22:50:29.654 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend": "0", 2024-09-20T22:50:29.654 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_prenautilus": "484", 2024-09-20T22:50:29.654 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_preluminous": "482", 2024-09-20T22:50:29.654 INFO:teuthology.orchestra.run.smithi084.stdout: "auid": 0, 2024-09-20T22:50:29.654 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_mode": "selfmanaged", 2024-09-20T22:50:29.654 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_seq": 239, 2024-09-20T22:50:29.654 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_epoch": 549, 2024-09-20T22:50:29.655 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_snaps": [], 2024-09-20T22:50:29.655 INFO:teuthology.orchestra.run.smithi084.stdout: "removed_snaps": "[]", 2024-09-20T22:50:29.655 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_bytes": 0, 2024-09-20T22:50:29.655 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_objects": 0, 2024-09-20T22:50:29.655 INFO:teuthology.orchestra.run.smithi084.stdout: "tiers": [], 2024-09-20T22:50:29.655 INFO:teuthology.orchestra.run.smithi084.stdout: "tier_of": -1, 2024-09-20T22:50:29.655 INFO:teuthology.orchestra.run.smithi084.stdout: "read_tier": -1, 2024-09-20T22:50:29.655 INFO:teuthology.orchestra.run.smithi084.stdout: "write_tier": -1, 2024-09-20T22:50:29.655 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_mode": "none", 2024-09-20T22:50:29.655 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_bytes": 0, 2024-09-20T22:50:29.655 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_objects": 0, 2024-09-20T22:50:29.655 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-20T22:50:29.655 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-20T22:50:29.655 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-20T22:50:29.655 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_flush_age": 0, 2024-09-20T22:50:29.656 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_evict_age": 0, 2024-09-20T22:50:29.656 INFO:teuthology.orchestra.run.smithi084.stdout: "erasure_code_profile": "", 2024-09-20T22:50:29.656 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_params": { 2024-09-20T22:50:29.656 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "none" 2024-09-20T22:50:29.656 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:29.656 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_period": 0, 2024-09-20T22:50:29.656 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_count": 0, 2024-09-20T22:50:29.656 INFO:teuthology.orchestra.run.smithi084.stdout: "use_gmt_hitset": true, 2024-09-20T22:50:29.656 INFO:teuthology.orchestra.run.smithi084.stdout: "min_read_recency_for_promote": 0, 2024-09-20T22:50:29.656 INFO:teuthology.orchestra.run.smithi084.stdout: "min_write_recency_for_promote": 0, 2024-09-20T22:50:29.656 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_grade_decay_rate": 0, 2024-09-20T22:50:29.656 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_search_last_n": 0, 2024-09-20T22:50:29.656 INFO:teuthology.orchestra.run.smithi084.stdout: "grade_table": [], 2024-09-20T22:50:29.656 INFO:teuthology.orchestra.run.smithi084.stdout: "stripe_width": 0, 2024-09-20T22:50:29.657 INFO:teuthology.orchestra.run.smithi084.stdout: "expected_num_objects": 0, 2024-09-20T22:50:29.657 INFO:teuthology.orchestra.run.smithi084.stdout: "fast_read": false, 2024-09-20T22:50:29.657 INFO:teuthology.orchestra.run.smithi084.stdout: "options": {}, 2024-09-20T22:50:29.657 INFO:teuthology.orchestra.run.smithi084.stdout: "application_metadata": { 2024-09-20T22:50:29.657 INFO:teuthology.orchestra.run.smithi084.stdout: "rados": {} 2024-09-20T22:50:29.657 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:29.657 INFO:teuthology.orchestra.run.smithi084.stdout: "read_balance": { 2024-09-20T22:50:29.657 INFO:teuthology.orchestra.run.smithi084.stdout: "score_type": "Fair distribution", 2024-09-20T22:50:29.657 INFO:teuthology.orchestra.run.smithi084.stdout: "score_acting": 2, 2024-09-20T22:50:29.657 INFO:teuthology.orchestra.run.smithi084.stdout: "score_stable": 2, 2024-09-20T22:50:29.657 INFO:teuthology.orchestra.run.smithi084.stdout: "optimal_score": 1, 2024-09-20T22:50:29.657 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_acting": 2, 2024-09-20T22:50:29.657 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_stable": 2, 2024-09-20T22:50:29.657 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity_weighted": 1, 2024-09-20T22:50:29.658 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity": 1, 2024-09-20T22:50:29.658 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity_weighted": 1 2024-09-20T22:50:29.658 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:50:29.658 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:50:29.658 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2024-09-20T22:50:29.658 INFO:teuthology.orchestra.run.smithi084.stdout: "osds": [ 2024-09-20T22:50:29.658 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:29.658 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 0, 2024-09-20T22:50:29.658 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "ea36a489-6f91-4b2f-b784-b90506b9805d", 2024-09-20T22:50:29.658 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-09-20T22:50:29.658 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-09-20T22:50:29.658 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-09-20T22:50:29.658 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-09-20T22:50:29.658 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-09-20T22:50:29.658 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-09-20T22:50:29.659 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 8, 2024-09-20T22:50:29.659 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 460, 2024-09-20T22:50:29.659 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-09-20T22:50:29.659 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-09-20T22:50:29.659 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-09-20T22:50:29.659 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:50:29.659 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:29.659 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:50:29.659 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6801", 2024-09-20T22:50:29.659 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 268084626 2024-09-20T22:50:29.659 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:50:29.659 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:50:29.659 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:29.659 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-09-20T22:50:29.660 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:50:29.660 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:29.660 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:50:29.660 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6802", 2024-09-20T22:50:29.660 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 268084626 2024-09-20T22:50:29.660 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:50:29.660 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:50:29.660 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:29.660 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-09-20T22:50:29.662 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:50:29.662 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:29.662 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:50:29.662 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6804", 2024-09-20T22:50:29.662 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 268084626 2024-09-20T22:50:29.662 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:50:29.662 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:50:29.663 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:29.663 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-09-20T22:50:29.663 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:50:29.663 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:29.663 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:50:29.663 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6803", 2024-09-20T22:50:29.663 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 268084626 2024-09-20T22:50:29.663 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:50:29.663 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:50:29.663 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:29.663 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.84:6801/268084626", 2024-09-20T22:50:29.663 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.84:6802/268084626", 2024-09-20T22:50:29.663 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.84:6804/268084626", 2024-09-20T22:50:29.663 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.84:6803/268084626", 2024-09-20T22:50:29.663 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-09-20T22:50:29.664 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-09-20T22:50:29.664 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-09-20T22:50:29.664 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:50:29.664 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:29.664 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:29.664 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 1, 2024-09-20T22:50:29.664 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "c8649d2f-27ef-47f8-bc94-f1cd9e62bdb8", 2024-09-20T22:50:29.665 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-09-20T22:50:29.665 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-09-20T22:50:29.665 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-09-20T22:50:29.665 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-09-20T22:50:29.665 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-09-20T22:50:29.665 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-09-20T22:50:29.665 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 13, 2024-09-20T22:50:29.665 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 468, 2024-09-20T22:50:29.665 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-09-20T22:50:29.665 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-09-20T22:50:29.665 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-09-20T22:50:29.666 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:50:29.666 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:29.666 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:50:29.666 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6805", 2024-09-20T22:50:29.666 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 418374136 2024-09-20T22:50:29.666 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:50:29.666 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:50:29.666 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:29.666 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-09-20T22:50:29.666 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:50:29.666 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:29.666 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:50:29.667 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6806", 2024-09-20T22:50:29.667 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 418374136 2024-09-20T22:50:29.667 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:50:29.667 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:50:29.667 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:29.667 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-09-20T22:50:29.667 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:50:29.667 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:29.667 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:50:29.667 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6808", 2024-09-20T22:50:29.667 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 418374136 2024-09-20T22:50:29.667 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:50:29.667 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:50:29.667 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:29.668 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-09-20T22:50:29.668 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:50:29.668 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:29.668 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:50:29.668 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6807", 2024-09-20T22:50:29.668 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 418374136 2024-09-20T22:50:29.668 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:50:29.668 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:50:29.668 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:29.668 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.84:6805/418374136", 2024-09-20T22:50:29.668 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.84:6806/418374136", 2024-09-20T22:50:29.668 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.84:6808/418374136", 2024-09-20T22:50:29.668 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.84:6807/418374136", 2024-09-20T22:50:29.668 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-09-20T22:50:29.669 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-09-20T22:50:29.669 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-09-20T22:50:29.669 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:50:29.669 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:29.669 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:29.669 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 2, 2024-09-20T22:50:29.669 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "c728eb48-a0f0-4de4-a651-7f3bec867280", 2024-09-20T22:50:29.669 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-09-20T22:50:29.669 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-09-20T22:50:29.669 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-09-20T22:50:29.669 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-09-20T22:50:29.669 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-09-20T22:50:29.669 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-09-20T22:50:29.669 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 18, 2024-09-20T22:50:29.670 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 430, 2024-09-20T22:50:29.670 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-09-20T22:50:29.670 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-09-20T22:50:29.670 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-09-20T22:50:29.670 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:50:29.670 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:29.670 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:50:29.670 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6809", 2024-09-20T22:50:29.670 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3977168611 2024-09-20T22:50:29.670 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:50:29.670 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:50:29.670 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:29.670 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-09-20T22:50:29.670 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:50:29.671 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:29.671 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:50:29.671 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6810", 2024-09-20T22:50:29.671 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3977168611 2024-09-20T22:50:29.671 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:50:29.671 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:50:29.671 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:29.671 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-09-20T22:50:29.671 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:50:29.671 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:29.671 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:50:29.671 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6812", 2024-09-20T22:50:29.671 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3977168611 2024-09-20T22:50:29.671 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:50:29.671 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:50:29.672 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:29.672 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-09-20T22:50:29.672 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:50:29.672 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:29.672 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:50:29.672 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6811", 2024-09-20T22:50:29.672 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3977168611 2024-09-20T22:50:29.672 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:50:29.672 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:50:29.672 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:29.672 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.84:6809/3977168611", 2024-09-20T22:50:29.672 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.84:6810/3977168611", 2024-09-20T22:50:29.672 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.84:6812/3977168611", 2024-09-20T22:50:29.672 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.84:6811/3977168611", 2024-09-20T22:50:29.673 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-09-20T22:50:29.673 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-09-20T22:50:29.673 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-09-20T22:50:29.673 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:50:29.673 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:29.673 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:29.673 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 3, 2024-09-20T22:50:29.673 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "573f44eb-0853-4630-a7d8-9caaeb256f9b", 2024-09-20T22:50:29.673 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-09-20T22:50:29.673 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-09-20T22:50:29.673 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-09-20T22:50:29.673 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-09-20T22:50:29.673 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-09-20T22:50:29.674 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-09-20T22:50:29.674 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 25, 2024-09-20T22:50:29.674 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 476, 2024-09-20T22:50:29.674 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-09-20T22:50:29.674 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-09-20T22:50:29.674 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-09-20T22:50:29.674 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:50:29.674 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:29.674 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:50:29.674 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6813", 2024-09-20T22:50:29.674 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2192239945 2024-09-20T22:50:29.674 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:50:29.674 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:50:29.674 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:29.675 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-09-20T22:50:29.675 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:50:29.675 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:29.675 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:50:29.675 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6814", 2024-09-20T22:50:29.675 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2192239945 2024-09-20T22:50:29.675 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:50:29.675 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:50:29.675 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:29.675 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-09-20T22:50:29.675 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:50:29.675 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:29.675 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:50:29.675 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6816", 2024-09-20T22:50:29.675 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2192239945 2024-09-20T22:50:29.676 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:50:29.676 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:50:29.676 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:29.676 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-09-20T22:50:29.676 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:50:29.676 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:29.676 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:50:29.676 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6815", 2024-09-20T22:50:29.676 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2192239945 2024-09-20T22:50:29.676 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:50:29.676 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:50:29.676 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:29.676 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.84:6813/2192239945", 2024-09-20T22:50:29.676 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.84:6814/2192239945", 2024-09-20T22:50:29.676 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.84:6816/2192239945", 2024-09-20T22:50:29.677 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.84:6815/2192239945", 2024-09-20T22:50:29.677 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-09-20T22:50:29.677 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-09-20T22:50:29.677 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-09-20T22:50:29.677 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:50:29.677 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:29.677 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:29.677 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 4, 2024-09-20T22:50:29.677 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "a52c3b94-c5f3-490f-8ac8-09061b9b2abb", 2024-09-20T22:50:29.677 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-09-20T22:50:29.677 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-09-20T22:50:29.677 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-09-20T22:50:29.677 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-09-20T22:50:29.677 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-09-20T22:50:29.677 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-09-20T22:50:29.678 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 31, 2024-09-20T22:50:29.678 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 480, 2024-09-20T22:50:29.678 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-09-20T22:50:29.678 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-09-20T22:50:29.678 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-09-20T22:50:29.678 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:50:29.678 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:29.678 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:50:29.678 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.154:6800", 2024-09-20T22:50:29.678 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1793136002 2024-09-20T22:50:29.678 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:50:29.678 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:50:29.678 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:29.678 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-09-20T22:50:29.679 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:50:29.679 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:29.679 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:50:29.679 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.154:6801", 2024-09-20T22:50:29.679 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1793136002 2024-09-20T22:50:29.679 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:50:29.679 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:50:29.679 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:29.679 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-09-20T22:50:29.679 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:50:29.679 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:29.679 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:50:29.679 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.154:6803", 2024-09-20T22:50:29.679 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1793136002 2024-09-20T22:50:29.680 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:50:29.680 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:50:29.680 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:29.680 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-09-20T22:50:29.680 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:50:29.680 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:29.680 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:50:29.680 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.154:6802", 2024-09-20T22:50:29.680 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1793136002 2024-09-20T22:50:29.680 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:50:29.680 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:50:29.680 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:29.681 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.154:6800/1793136002", 2024-09-20T22:50:29.681 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.154:6801/1793136002", 2024-09-20T22:50:29.681 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.154:6803/1793136002", 2024-09-20T22:50:29.681 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.154:6802/1793136002", 2024-09-20T22:50:29.681 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-09-20T22:50:29.681 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-09-20T22:50:29.681 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-09-20T22:50:29.681 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:50:29.681 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:29.681 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:29.681 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 5, 2024-09-20T22:50:29.681 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "4fa0dc8e-25cc-46cd-96ff-33f20b6faec3", 2024-09-20T22:50:29.681 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-09-20T22:50:29.681 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-09-20T22:50:29.681 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-09-20T22:50:29.682 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-09-20T22:50:29.682 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-09-20T22:50:29.682 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-09-20T22:50:29.682 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 40, 2024-09-20T22:50:29.682 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 484, 2024-09-20T22:50:29.682 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-09-20T22:50:29.682 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-09-20T22:50:29.682 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-09-20T22:50:29.682 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:50:29.682 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:29.682 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:50:29.682 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.154:6804", 2024-09-20T22:50:29.682 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 493899927 2024-09-20T22:50:29.682 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:50:29.682 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:50:29.683 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:29.683 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-09-20T22:50:29.683 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:50:29.683 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:29.683 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:50:29.683 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.154:6805", 2024-09-20T22:50:29.683 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 493899927 2024-09-20T22:50:29.683 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:50:29.683 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:50:29.683 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:29.683 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-09-20T22:50:29.683 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:50:29.683 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:29.683 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:50:29.684 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.154:6807", 2024-09-20T22:50:29.684 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 493899927 2024-09-20T22:50:29.684 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:50:29.684 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:50:29.684 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:29.684 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-09-20T22:50:29.684 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:50:29.684 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:29.684 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:50:29.684 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.154:6806", 2024-09-20T22:50:29.684 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 493899927 2024-09-20T22:50:29.684 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:50:29.684 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:50:29.684 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:29.684 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.154:6804/493899927", 2024-09-20T22:50:29.685 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.154:6805/493899927", 2024-09-20T22:50:29.685 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.154:6807/493899927", 2024-09-20T22:50:29.685 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.154:6806/493899927", 2024-09-20T22:50:29.685 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-09-20T22:50:29.685 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-09-20T22:50:29.685 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-09-20T22:50:29.685 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:50:29.685 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:29.685 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:29.685 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 6, 2024-09-20T22:50:29.685 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "56788d74-810e-4827-a615-958c12f8a368", 2024-09-20T22:50:29.685 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-09-20T22:50:29.685 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-09-20T22:50:29.685 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-09-20T22:50:29.686 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-09-20T22:50:29.686 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-09-20T22:50:29.686 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-09-20T22:50:29.686 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 48, 2024-09-20T22:50:29.686 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 442, 2024-09-20T22:50:29.686 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-09-20T22:50:29.686 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-09-20T22:50:29.686 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-09-20T22:50:29.686 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:50:29.686 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:29.686 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:50:29.686 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.154:6808", 2024-09-20T22:50:29.686 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2375091033 2024-09-20T22:50:29.686 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:50:29.687 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:50:29.687 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:29.687 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-09-20T22:50:29.687 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:50:29.687 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:29.687 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:50:29.687 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.154:6809", 2024-09-20T22:50:29.687 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2375091033 2024-09-20T22:50:29.687 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:50:29.687 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:50:29.687 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:29.687 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-09-20T22:50:29.687 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:50:29.688 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:29.688 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:50:29.688 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.154:6811", 2024-09-20T22:50:29.688 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2375091033 2024-09-20T22:50:29.688 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:50:29.688 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:50:29.688 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:29.688 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-09-20T22:50:29.688 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:50:29.688 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:29.688 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:50:29.688 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.154:6810", 2024-09-20T22:50:29.688 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2375091033 2024-09-20T22:50:29.688 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:50:29.688 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:50:29.689 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:29.689 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.154:6808/2375091033", 2024-09-20T22:50:29.689 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.154:6809/2375091033", 2024-09-20T22:50:29.689 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.154:6811/2375091033", 2024-09-20T22:50:29.689 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.154:6810/2375091033", 2024-09-20T22:50:29.689 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-09-20T22:50:29.689 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-09-20T22:50:29.689 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-09-20T22:50:29.689 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:50:29.689 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:29.689 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:29.689 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 7, 2024-09-20T22:50:29.689 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "83503727-afc1-4b95-8b96-ecb63f3338c2", 2024-09-20T22:50:29.689 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-09-20T22:50:29.690 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-09-20T22:50:29.690 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-09-20T22:50:29.690 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-09-20T22:50:29.690 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-09-20T22:50:29.690 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-09-20T22:50:29.690 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 59, 2024-09-20T22:50:29.690 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 456, 2024-09-20T22:50:29.690 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-09-20T22:50:29.690 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-09-20T22:50:29.690 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-09-20T22:50:29.690 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:50:29.690 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:29.690 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:50:29.690 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.154:6812", 2024-09-20T22:50:29.691 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3717704491 2024-09-20T22:50:29.691 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:50:29.691 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:50:29.691 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:29.691 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-09-20T22:50:29.691 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:50:29.691 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:29.691 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:50:29.691 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.154:6813", 2024-09-20T22:50:29.691 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3717704491 2024-09-20T22:50:29.691 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:50:29.691 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:50:29.691 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:29.691 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-09-20T22:50:29.691 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:50:29.692 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:29.692 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:50:29.692 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.154:6815", 2024-09-20T22:50:29.692 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3717704491 2024-09-20T22:50:29.692 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:50:29.692 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:50:29.692 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:29.692 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-09-20T22:50:29.692 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:50:29.692 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:29.692 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:50:29.692 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.154:6814", 2024-09-20T22:50:29.692 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3717704491 2024-09-20T22:50:29.692 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:50:29.693 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:50:29.693 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:29.693 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.154:6812/3717704491", 2024-09-20T22:50:29.693 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.154:6813/3717704491", 2024-09-20T22:50:29.693 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.154:6815/3717704491", 2024-09-20T22:50:29.693 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.154:6814/3717704491", 2024-09-20T22:50:29.693 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-09-20T22:50:29.693 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-09-20T22:50:29.693 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-09-20T22:50:29.693 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:50:29.693 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:50:29.693 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2024-09-20T22:50:29.693 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_xinfo": [ 2024-09-20T22:50:29.693 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:29.693 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 0, 2024-09-20T22:50:29.694 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-09-20T22:50:29.694 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-09-20T22:50:29.694 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-09-20T22:50:29.694 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540701547738038271, 2024-09-20T22:50:29.694 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-09-20T22:50:29.694 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-09-20T22:50:14.131015+0000", 2024-09-20T22:50:29.694 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-09-20T22:50:29.694 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:29.694 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:29.694 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 1, 2024-09-20T22:50:29.694 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-09-20T22:50:29.694 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-09-20T22:50:29.695 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-09-20T22:50:29.695 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540701547738038271, 2024-09-20T22:50:29.695 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-09-20T22:50:29.695 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-09-20T22:49:25.080173+0000", 2024-09-20T22:50:29.695 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-09-20T22:50:29.695 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:29.695 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:29.695 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 2, 2024-09-20T22:50:29.695 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-09-20T22:50:29.695 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-09-20T22:50:29.695 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-09-20T22:50:29.695 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540701547738038271, 2024-09-20T22:50:29.695 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-09-20T22:50:29.695 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-09-20T22:49:44.500257+0000", 2024-09-20T22:50:29.695 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-09-20T22:50:29.696 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:29.696 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:29.696 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 3, 2024-09-20T22:50:29.696 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-09-20T22:50:29.696 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-09-20T22:50:29.696 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-09-20T22:50:29.696 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540701547738038271, 2024-09-20T22:50:29.696 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-09-20T22:50:29.696 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-09-20T22:49:52.402042+0000", 2024-09-20T22:50:29.696 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-09-20T22:50:29.696 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:29.696 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:29.696 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 4, 2024-09-20T22:50:29.696 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-09-20T22:50:29.696 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-09-20T22:50:29.697 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-09-20T22:50:29.697 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540701547738038271, 2024-09-20T22:50:29.697 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-09-20T22:50:29.697 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-09-20T22:50:05.515503+0000", 2024-09-20T22:50:29.697 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-09-20T22:50:29.697 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:29.697 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:29.697 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 5, 2024-09-20T22:50:29.697 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-09-20T22:50:29.697 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-09-20T22:50:29.697 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-09-20T22:50:29.697 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540701547738038271, 2024-09-20T22:50:29.697 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-09-20T22:50:29.697 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-09-20T22:49:47.639223+0000", 2024-09-20T22:50:29.697 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-09-20T22:50:29.698 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:29.698 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:29.698 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 6, 2024-09-20T22:50:29.698 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-09-20T22:50:29.698 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-09-20T22:50:29.698 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-09-20T22:50:29.698 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540701547738038271, 2024-09-20T22:50:29.698 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-09-20T22:50:29.698 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-09-20T22:49:35.895711+0000", 2024-09-20T22:50:29.698 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-09-20T22:50:29.698 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:29.698 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:29.698 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 7, 2024-09-20T22:50:29.698 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-09-20T22:50:29.698 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-09-20T22:50:29.699 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-09-20T22:50:29.699 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540701547738038271, 2024-09-20T22:50:29.699 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-09-20T22:50:29.699 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-09-20T22:50:07.018067+0000", 2024-09-20T22:50:29.699 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-09-20T22:50:29.699 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:50:29.699 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2024-09-20T22:50:29.699 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_upmap": [ 2024-09-20T22:50:29.699 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:29.699 INFO:teuthology.orchestra.run.smithi084.stdout: "pgid": "6.1e", 2024-09-20T22:50:29.699 INFO:teuthology.orchestra.run.smithi084.stdout: "osds": [ 2024-09-20T22:50:29.699 INFO:teuthology.orchestra.run.smithi084.stdout: 0, 2024-09-20T22:50:29.699 INFO:teuthology.orchestra.run.smithi084.stdout: 1, 2024-09-20T22:50:29.699 INFO:teuthology.orchestra.run.smithi084.stdout: 3 2024-09-20T22:50:29.699 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:50:29.700 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:50:29.700 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2024-09-20T22:50:29.700 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_upmap_items": [ 2024-09-20T22:50:29.700 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:29.700 INFO:teuthology.orchestra.run.smithi084.stdout: "pgid": "3.a", 2024-09-20T22:50:29.700 INFO:teuthology.orchestra.run.smithi084.stdout: "mappings": [ 2024-09-20T22:50:29.700 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:29.700 INFO:teuthology.orchestra.run.smithi084.stdout: "from": 6, 2024-09-20T22:50:29.700 INFO:teuthology.orchestra.run.smithi084.stdout: "to": 2 2024-09-20T22:50:29.700 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:50:29.700 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:50:29.700 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:29.700 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:29.700 INFO:teuthology.orchestra.run.smithi084.stdout: "pgid": "4.6", 2024-09-20T22:50:29.701 INFO:teuthology.orchestra.run.smithi084.stdout: "mappings": [ 2024-09-20T22:50:29.701 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:29.701 INFO:teuthology.orchestra.run.smithi084.stdout: "from": 1, 2024-09-20T22:50:29.701 INFO:teuthology.orchestra.run.smithi084.stdout: "to": 5 2024-09-20T22:50:29.701 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:50:29.701 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:50:29.701 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:29.701 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:29.701 INFO:teuthology.orchestra.run.smithi084.stdout: "pgid": "4.11", 2024-09-20T22:50:29.701 INFO:teuthology.orchestra.run.smithi084.stdout: "mappings": [ 2024-09-20T22:50:29.702 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:29.702 INFO:teuthology.orchestra.run.smithi084.stdout: "from": 3, 2024-09-20T22:50:29.702 INFO:teuthology.orchestra.run.smithi084.stdout: "to": 5 2024-09-20T22:50:29.702 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:50:29.702 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:50:29.702 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:29.702 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:29.702 INFO:teuthology.orchestra.run.smithi084.stdout: "pgid": "4.17", 2024-09-20T22:50:29.702 INFO:teuthology.orchestra.run.smithi084.stdout: "mappings": [ 2024-09-20T22:50:29.702 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:29.702 INFO:teuthology.orchestra.run.smithi084.stdout: "from": 1, 2024-09-20T22:50:29.702 INFO:teuthology.orchestra.run.smithi084.stdout: "to": 2 2024-09-20T22:50:29.702 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:50:29.702 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:50:29.703 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:29.703 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:29.703 INFO:teuthology.orchestra.run.smithi084.stdout: "pgid": "4.1d", 2024-09-20T22:50:29.703 INFO:teuthology.orchestra.run.smithi084.stdout: "mappings": [ 2024-09-20T22:50:29.703 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:29.703 INFO:teuthology.orchestra.run.smithi084.stdout: "from": 1, 2024-09-20T22:50:29.703 INFO:teuthology.orchestra.run.smithi084.stdout: "to": 2 2024-09-20T22:50:29.703 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:50:29.703 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:50:29.703 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:50:29.703 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2024-09-20T22:50:29.703 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_upmap_primaries": [], 2024-09-20T22:50:29.703 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_temp": [], 2024-09-20T22:50:29.703 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_temp": [], 2024-09-20T22:50:29.703 INFO:teuthology.orchestra.run.smithi084.stdout: "blocklist": { 2024-09-20T22:50:29.704 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/542020545": "2024-09-21T22:42:07.596045+0000", 2024-09-20T22:50:29.704 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/25873236": "2024-09-21T22:36:54.917812+0000", 2024-09-20T22:50:29.704 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:6800/820908470": "2024-09-21T22:36:54.917812+0000", 2024-09-20T22:50:29.704 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/567614908": "2024-09-21T22:36:54.917812+0000", 2024-09-20T22:50:29.704 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/2963622459": "2024-09-21T22:42:07.596045+0000", 2024-09-20T22:50:29.704 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/1176849276": "2024-09-21T22:37:17.690104+0000", 2024-09-20T22:50:29.704 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:6800/2932420050": "2024-09-21T22:37:17.690104+0000", 2024-09-20T22:50:29.704 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/3879263635": "2024-09-21T22:37:17.690104+0000", 2024-09-20T22:50:29.704 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/969472279": "2024-09-21T22:36:54.917812+0000", 2024-09-20T22:50:29.704 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/1361143214": "2024-09-21T22:37:17.690104+0000", 2024-09-20T22:50:29.704 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/3353126955": "2024-09-21T22:42:07.596045+0000", 2024-09-20T22:50:29.704 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:6800/4024278029": "2024-09-21T22:42:07.596045+0000", 2024-09-20T22:50:29.704 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/547902262": "2024-09-21T22:42:07.596045+0000", 2024-09-20T22:50:29.704 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/1242929317": "2024-09-21T22:42:07.596045+0000" 2024-09-20T22:50:29.704 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:29.705 INFO:teuthology.orchestra.run.smithi084.stdout: "range_blocklist": {}, 2024-09-20T22:50:29.705 INFO:teuthology.orchestra.run.smithi084.stdout: "erasure_code_profiles": { 2024-09-20T22:50:29.705 INFO:teuthology.orchestra.run.smithi084.stdout: "default": { 2024-09-20T22:50:29.705 INFO:teuthology.orchestra.run.smithi084.stdout: "crush-failure-domain": "osd", 2024-09-20T22:50:29.705 INFO:teuthology.orchestra.run.smithi084.stdout: "k": "2", 2024-09-20T22:50:29.705 INFO:teuthology.orchestra.run.smithi084.stdout: "m": "1", 2024-09-20T22:50:29.705 INFO:teuthology.orchestra.run.smithi084.stdout: "plugin": "jerasure", 2024-09-20T22:50:29.705 INFO:teuthology.orchestra.run.smithi084.stdout: "technique": "reed_sol_van" 2024-09-20T22:50:29.705 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:50:29.705 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:29.705 INFO:teuthology.orchestra.run.smithi084.stdout: "removed_snaps_queue": [ 2024-09-20T22:50:29.705 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:29.705 INFO:teuthology.orchestra.run.smithi084.stdout: "pool": 7, 2024-09-20T22:50:29.705 INFO:teuthology.orchestra.run.smithi084.stdout: "snaps": [ 2024-09-20T22:50:29.706 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:29.706 INFO:teuthology.orchestra.run.smithi084.stdout: "begin": 158, 2024-09-20T22:50:29.706 INFO:teuthology.orchestra.run.smithi084.stdout: "length": 1 2024-09-20T22:50:29.706 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:29.706 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:29.706 INFO:teuthology.orchestra.run.smithi084.stdout: "begin": 186, 2024-09-20T22:50:29.706 INFO:teuthology.orchestra.run.smithi084.stdout: "length": 1 2024-09-20T22:50:29.706 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:29.706 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:29.706 INFO:teuthology.orchestra.run.smithi084.stdout: "begin": 216, 2024-09-20T22:50:29.706 INFO:teuthology.orchestra.run.smithi084.stdout: "length": 1 2024-09-20T22:50:29.706 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:29.706 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:29.706 INFO:teuthology.orchestra.run.smithi084.stdout: "begin": 224, 2024-09-20T22:50:29.706 INFO:teuthology.orchestra.run.smithi084.stdout: "length": 1 2024-09-20T22:50:29.707 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:29.707 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:29.707 INFO:teuthology.orchestra.run.smithi084.stdout: "begin": 227, 2024-09-20T22:50:29.707 INFO:teuthology.orchestra.run.smithi084.stdout: "length": 1 2024-09-20T22:50:29.707 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:29.707 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:29.707 INFO:teuthology.orchestra.run.smithi084.stdout: "begin": 233, 2024-09-20T22:50:29.707 INFO:teuthology.orchestra.run.smithi084.stdout: "length": 1 2024-09-20T22:50:29.707 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:29.707 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:29.707 INFO:teuthology.orchestra.run.smithi084.stdout: "begin": 238, 2024-09-20T22:50:29.707 INFO:teuthology.orchestra.run.smithi084.stdout: "length": 1 2024-09-20T22:50:29.707 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:50:29.708 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:50:29.708 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:50:29.708 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2024-09-20T22:50:29.708 INFO:teuthology.orchestra.run.smithi084.stdout: "new_removed_snaps": [ 2024-09-20T22:50:29.708 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:29.708 INFO:teuthology.orchestra.run.smithi084.stdout: "pool": 7, 2024-09-20T22:50:29.708 INFO:teuthology.orchestra.run.smithi084.stdout: "snaps": [ 2024-09-20T22:50:29.708 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:29.708 INFO:teuthology.orchestra.run.smithi084.stdout: "begin": 216, 2024-09-20T22:50:29.708 INFO:teuthology.orchestra.run.smithi084.stdout: "length": 1 2024-09-20T22:50:29.708 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:50:29.708 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:50:29.708 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:50:29.709 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2024-09-20T22:50:29.709 INFO:teuthology.orchestra.run.smithi084.stdout: "new_purged_snaps": [], 2024-09-20T22:50:29.709 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_node_flags": {}, 2024-09-20T22:50:29.709 INFO:teuthology.orchestra.run.smithi084.stdout: "device_class_flags": {}, 2024-09-20T22:50:29.709 INFO:teuthology.orchestra.run.smithi084.stdout: "stretch_mode": { 2024-09-20T22:50:29.709 INFO:teuthology.orchestra.run.smithi084.stdout: "stretch_mode_enabled": false, 2024-09-20T22:50:29.709 INFO:teuthology.orchestra.run.smithi084.stdout: "stretch_bucket_count": 0, 2024-09-20T22:50:29.709 INFO:teuthology.orchestra.run.smithi084.stdout: "degraded_stretch_mode": 0, 2024-09-20T22:50:29.709 INFO:teuthology.orchestra.run.smithi084.stdout: "recovering_stretch_mode": 0, 2024-09-20T22:50:29.709 INFO:teuthology.orchestra.run.smithi084.stdout: "stretch_mode_bucket": 0 2024-09-20T22:50:29.709 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:50:29.709 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-20T22:50:29.710 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:50:29.733 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:29.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:29.734 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:50:29.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:29 smithi154 ceph-mon[33264]: pgmap v723: 158 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 153 active+clean; 497 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-20T22:50:29.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:29 smithi154 ceph-mon[33264]: osdmap e549: 8 total, 8 up, 8 in 2024-09-20T22:50:29.754 INFO:journalctl@ceph.iscsi.iscsi.a.smithi154.stdout:Sep 20 22:50:29 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-iscsi-iscsi-a[59730]: debug there is no tcmu-runner data available 2024-09-20T22:50:29.835 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:50:29.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:29.859 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:29.860 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:50:29.960 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:50:29.982 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:29.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:29.984 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:50:30.084 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:50:30.110 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:30.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:30.111 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:50:30.212 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:50:30.233 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:30.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:30.234 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:50:30.276 INFO:tasks.rados.rados.0.smithi084.stdout:2147: done (4 left) 2024-09-20T22:50:30.276 INFO:tasks.rados.rados.0.smithi084.stdout:2148: write oid 30 current snap is 238 2024-09-20T22:50:30.277 INFO:tasks.rados.rados.0.smithi084.stdout:2142: finishing copy_from to smithi08477203-20 2024-09-20T22:50:30.277 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 20 v 794 (ObjNum 470 snap 130 seq_num 470) dirty exists 2024-09-20T22:50:30.277 INFO:tasks.rados.rados.0.smithi084.stdout:2148: seq_num 750 ranges {605250=639344,1855142=452579} 2024-09-20T22:50:30.283 INFO:tasks.rados.rados.0.smithi084.stdout:2148: writing smithi08477203-30 from 605250 to 1244594 tid 1 2024-09-20T22:50:30.290 INFO:tasks.rados.rados.0.smithi084.stdout:2148: writing smithi08477203-30 from 1855142 to 2307721 tid 2 2024-09-20T22:50:30.290 INFO:tasks.rados.rados.0.smithi084.stdout:2142: done (4 left) 2024-09-20T22:50:30.290 INFO:tasks.rados.rados.0.smithi084.stdout:2149: write oid 34 current snap is 238 2024-09-20T22:50:30.290 INFO:tasks.rados.rados.0.smithi084.stdout:2145: finishing write tid 2 to smithi08477203-1 2024-09-20T22:50:30.290 INFO:tasks.rados.rados.0.smithi084.stdout:2145: finishing write tid 3 to smithi08477203-1 2024-09-20T22:50:30.290 INFO:tasks.rados.rados.0.smithi084.stdout:2144: finishing copy_from to smithi08477203-22 2024-09-20T22:50:30.290 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 22 v 726 (ObjNum 740 snap 236 seq_num 740) dirty exists 2024-09-20T22:50:30.290 INFO:tasks.rados.rados.0.smithi084.stdout:2145: finishing write tid 4 to smithi08477203-1 2024-09-20T22:50:30.290 INFO:tasks.rados.rados.0.smithi084.stdout:2145: oid 1 updating version 0 to 1218 2024-09-20T22:50:30.291 INFO:tasks.rados.rados.0.smithi084.stdout:2145: oid 1 version 1218 is already newer than 1216 2024-09-20T22:50:30.291 INFO:tasks.rados.rados.0.smithi084.stdout:2145: oid 1 version 1218 is already newer than 1217 2024-09-20T22:50:30.291 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 1 v 1218 (ObjNum 748 snap 238 seq_num 748) dirty exists 2024-09-20T22:50:30.291 INFO:tasks.rados.rados.0.smithi084.stdout:2145: left oid 1 (ObjNum 748 snap 238 seq_num 748) 2024-09-20T22:50:30.291 INFO:tasks.rados.rados.0.smithi084.stdout:2146: finishing write tid 1 to smithi08477203-32 2024-09-20T22:50:30.291 INFO:tasks.rados.rados.0.smithi084.stdout:2146: finishing write tid 2 to smithi08477203-32 2024-09-20T22:50:30.291 INFO:tasks.rados.rados.0.smithi084.stdout:2149: seq_num 751 ranges {622057=714593,1913624=708433} 2024-09-20T22:50:30.297 INFO:tasks.rados.rados.0.smithi084.stdout:2149: writing smithi08477203-34 from 622057 to 1336650 tid 1 2024-09-20T22:50:30.305 INFO:tasks.rados.rados.0.smithi084.stdout:2149: writing smithi08477203-34 from 1913624 to 2622057 tid 2 2024-09-20T22:50:30.305 INFO:tasks.rados.rados.0.smithi084.stdout:2144: done (4 left) 2024-09-20T22:50:30.305 INFO:tasks.rados.rados.0.smithi084.stdout:2145: done (3 left) 2024-09-20T22:50:30.305 INFO:tasks.rados.rados.0.smithi084.stdout:2150: read oid 29 snap -1 2024-09-20T22:50:30.305 INFO:tasks.rados.rados.0.smithi084.stdout:2150: expect (ObjNum 440 snap 115 seq_num 440) 2024-09-20T22:50:30.305 INFO:tasks.rados.rados.0.smithi084.stdout:2151: read oid 19 snap -1 2024-09-20T22:50:30.306 INFO:tasks.rados.rados.0.smithi084.stdout:2151: expect deleted 2024-09-20T22:50:30.306 INFO:tasks.rados.rados.0.smithi084.stdout:2152: write (excl) oid 7 current snap is 238 2024-09-20T22:50:30.306 INFO:tasks.rados.rados.0.smithi084.stdout:2146: finishing write tid 3 to smithi08477203-32 2024-09-20T22:50:30.306 INFO:tasks.rados.rados.0.smithi084.stdout:2146: finishing write tid 4 to smithi08477203-32 2024-09-20T22:50:30.306 INFO:tasks.rados.rados.0.smithi084.stdout:2146: oid 32 updating version 0 to 737 2024-09-20T22:50:30.306 INFO:tasks.rados.rados.0.smithi084.stdout:2146: oid 32 version 737 is already newer than 735 2024-09-20T22:50:30.306 INFO:tasks.rados.rados.0.smithi084.stdout:2146: oid 32 version 737 is already newer than 736 2024-09-20T22:50:30.306 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 32 v 737 (ObjNum 749 snap 238 seq_num 749) dirty exists 2024-09-20T22:50:30.306 INFO:tasks.rados.rados.0.smithi084.stdout:2146: left oid 32 (ObjNum 749 snap 238 seq_num 749) 2024-09-20T22:50:30.306 INFO:tasks.rados.rados.0.smithi084.stdout:2152: seq_num 752 ranges {638864=789842,1972106=456248} 2024-09-20T22:50:30.312 INFO:tasks.rados.rados.0.smithi084.stdout:2152: writing smithi08477203-7 from 638864 to 1428706 tid 1 2024-09-20T22:50:30.318 INFO:tasks.rados.rados.0.smithi084.stdout:2152: writing smithi08477203-7 from 1972106 to 2428354 tid 2 2024-09-20T22:50:30.319 INFO:tasks.rados.rados.0.smithi084.stdout:2146: done (5 left) 2024-09-20T22:50:30.319 INFO:tasks.rados.rados.0.smithi084.stdout:2153: snap_remove snap 200 2024-09-20T22:50:30.335 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:50:30.343 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd unset noscrub 2024-09-20T22:50:30.358 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:30.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:30.359 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:50:30.460 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:50:30.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:30.482 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:30.482 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:50:30.530 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:30 smithi084 ceph-mon[29774]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:50:30.530 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:30 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/4166828720' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-20T22:50:30.531 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:30 smithi084 ceph-mon[40024]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:50:30.531 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:30 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/4166828720' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-20T22:50:30.564 INFO:tasks.thrashosds.thrasher:j is {'epoch': 549, 'fsid': 'a9c0bc70-77a0-11ef-baf5-efdc52797490', 'created': '2024-09-20T22:36:30.240626+0000', 'modified': '2024-09-20T22:50:29.243921+0000', 'last_up_change': '2024-09-20T22:41:27.098565+0000', 'last_in_change': '2024-09-20T22:48:04.201694+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 18, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 7, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-20T22:39:21.984381+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 11, 'pg_placement_num': 11, 'pg_placement_num_target': 11, 'pg_num_target': 11, 'pg_num_pending': 11, 'last_pg_merge_meta': {'source_pgid': '1.b', 'ready_epoch': 299, 'last_epoch_started': 298, 'last_epoch_clean': 298, 'source_version': "77'128", 'target_version': "77'128"}, 'last_change': '542', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '299', 'last_force_op_resend_preluminous': '297', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.180000066757202, 'score_stable': 2.180000066757202, 'optimal_score': 1, 'raw_score_acting': 2.180000066757202, 'raw_score_stable': 2.180000066757202, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-09-20T22:41:35.820081+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '69', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 69, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.6500000953674316, 'score_stable': 2.6500000953674316, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 2.3299999237060547, 'raw_score_stable': 2.3299999237060547, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-09-20T22:41:36.546389+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-09-20T22:41:38.454640+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '70', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-09-20T22:41:40.407129+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '72', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-09-20T22:41:42.476716+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '74', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'unique_pool_0', 'create_time': '2024-09-20T22:42:51.354358+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '7.10', 'ready_epoch': 484, 'last_epoch_started': 483, 'last_epoch_clean': 483, 'source_version': "480'172", 'target_version': "480'253"}, 'last_change': '549', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '484', 'last_force_op_resend_preluminous': '482', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 239, 'snap_epoch': 549, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': 'ea36a489-6f91-4b2f-b784-b90506b9805d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 460, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6801', 'nonce': 268084626}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6802', 'nonce': 268084626}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6804', 'nonce': 268084626}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6803', 'nonce': 268084626}]}, 'public_addr': '172.21.15.84:6801/268084626', 'cluster_addr': '172.21.15.84:6802/268084626', 'heartbeat_back_addr': '172.21.15.84:6804/268084626', 'heartbeat_front_addr': '172.21.15.84:6803/268084626', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'c8649d2f-27ef-47f8-bc94-f1cd9e62bdb8', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 468, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6805', 'nonce': 418374136}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6806', 'nonce': 418374136}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6808', 'nonce': 418374136}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6807', 'nonce': 418374136}]}, 'public_addr': '172.21.15.84:6805/418374136', 'cluster_addr': '172.21.15.84:6806/418374136', 'heartbeat_back_addr': '172.21.15.84:6808/418374136', 'heartbeat_front_addr': '172.21.15.84:6807/418374136', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'c728eb48-a0f0-4de4-a651-7f3bec867280', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 430, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6809', 'nonce': 3977168611}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6810', 'nonce': 3977168611}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6812', 'nonce': 3977168611}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6811', 'nonce': 3977168611}]}, 'public_addr': '172.21.15.84:6809/3977168611', 'cluster_addr': '172.21.15.84:6810/3977168611', 'heartbeat_back_addr': '172.21.15.84:6812/3977168611', 'heartbeat_front_addr': '172.21.15.84:6811/3977168611', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '573f44eb-0853-4630-a7d8-9caaeb256f9b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 25, 'up_thru': 476, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6813', 'nonce': 2192239945}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6814', 'nonce': 2192239945}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6816', 'nonce': 2192239945}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6815', 'nonce': 2192239945}]}, 'public_addr': '172.21.15.84:6813/2192239945', 'cluster_addr': '172.21.15.84:6814/2192239945', 'heartbeat_back_addr': '172.21.15.84:6816/2192239945', 'heartbeat_front_addr': '172.21.15.84:6815/2192239945', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'a52c3b94-c5f3-490f-8ac8-09061b9b2abb', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 480, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6800', 'nonce': 1793136002}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6801', 'nonce': 1793136002}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6803', 'nonce': 1793136002}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6802', 'nonce': 1793136002}]}, 'public_addr': '172.21.15.154:6800/1793136002', 'cluster_addr': '172.21.15.154:6801/1793136002', 'heartbeat_back_addr': '172.21.15.154:6803/1793136002', 'heartbeat_front_addr': '172.21.15.154:6802/1793136002', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '4fa0dc8e-25cc-46cd-96ff-33f20b6faec3', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 40, 'up_thru': 484, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6804', 'nonce': 493899927}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6805', 'nonce': 493899927}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6807', 'nonce': 493899927}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6806', 'nonce': 493899927}]}, 'public_addr': '172.21.15.154:6804/493899927', 'cluster_addr': '172.21.15.154:6805/493899927', 'heartbeat_back_addr': '172.21.15.154:6807/493899927', 'heartbeat_front_addr': '172.21.15.154:6806/493899927', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '56788d74-810e-4827-a615-958c12f8a368', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 48, 'up_thru': 442, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6808', 'nonce': 2375091033}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6809', 'nonce': 2375091033}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6811', 'nonce': 2375091033}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6810', 'nonce': 2375091033}]}, 'public_addr': '172.21.15.154:6808/2375091033', 'cluster_addr': '172.21.15.154:6809/2375091033', 'heartbeat_back_addr': '172.21.15.154:6811/2375091033', 'heartbeat_front_addr': '172.21.15.154:6810/2375091033', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '83503727-afc1-4b95-8b96-ecb63f3338c2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 59, 'up_thru': 456, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6812', 'nonce': 3717704491}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6813', 'nonce': 3717704491}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6815', 'nonce': 3717704491}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6814', 'nonce': 3717704491}]}, 'public_addr': '172.21.15.154:6812/3717704491', 'cluster_addr': '172.21.15.154:6813/3717704491', 'heartbeat_back_addr': '172.21.15.154:6815/3717704491', 'heartbeat_front_addr': '172.21.15.154:6814/3717704491', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-20T22:50:14.131015+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-20T22:49:25.080173+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-20T22:49:44.500257+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-20T22:49:52.402042+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-20T22:50:05.515503+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-20T22:49:47.639223+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-20T22:49:35.895711+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-20T22:50:07.018067+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '6.1e', 'osds': [0, 1, 3]}], 'pg_upmap_items': [{'pgid': '3.a', 'mappings': [{'from': 6, 'to': 2}]}, {'pgid': '4.6', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.11', 'mappings': [{'from': 3, 'to': 5}]}, {'pgid': '4.17', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.1d', 'mappings': [{'from': 1, 'to': 2}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.84:0/542020545': '2024-09-21T22:42:07.596045+0000', '172.21.15.84:0/25873236': '2024-09-21T22:36:54.917812+0000', '172.21.15.84:6800/820908470': '2024-09-21T22:36:54.917812+0000', '172.21.15.84:0/567614908': '2024-09-21T22:36:54.917812+0000', '172.21.15.84:0/2963622459': '2024-09-21T22:42:07.596045+0000', '172.21.15.84:0/1176849276': '2024-09-21T22:37:17.690104+0000', '172.21.15.84:6800/2932420050': '2024-09-21T22:37:17.690104+0000', '172.21.15.84:0/3879263635': '2024-09-21T22:37:17.690104+0000', '172.21.15.84:0/969472279': '2024-09-21T22:36:54.917812+0000', '172.21.15.84:0/1361143214': '2024-09-21T22:37:17.690104+0000', '172.21.15.84:0/3353126955': '2024-09-21T22:42:07.596045+0000', '172.21.15.84:6800/4024278029': '2024-09-21T22:42:07.596045+0000', '172.21.15.84:0/547902262': '2024-09-21T22:42:07.596045+0000', '172.21.15.84:0/1242929317': '2024-09-21T22:42:07.596045+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [{'pool': 7, 'snaps': [{'begin': 158, 'length': 1}, {'begin': 186, 'length': 1}, {'begin': 216, 'length': 1}, {'begin': 224, 'length': 1}, {'begin': 227, 'length': 1}, {'begin': 233, 'length': 1}, {'begin': 238, 'length': 1}]}], 'new_removed_snaps': [{'pool': 7, 'snaps': [{'begin': 216, 'length': 1}]}], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-20T22:50:30.565 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph pg dump --format=json 2024-09-20T22:50:30.583 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:50:30.606 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:30.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:30.607 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:50:30.708 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:50:30.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:30.730 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:30.730 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:50:30.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:30 smithi154 ceph-mon[33264]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:50:30.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:30 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/4166828720' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-20T22:50:30.831 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:50:30.856 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:30.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:30.857 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:50:30.958 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:50:30.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:30.984 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:30.984 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:50:30.998 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:50:30.998 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:50:31.015 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:50:31.016 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:50:31.031 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:50:31.032 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:50:31.048 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-20T22:50:31.048 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T22:50:31.065 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:50:31.065 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:50:31.081 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:50:31.082 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:50:31.084 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:50:31.099 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:50:31.099 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:50:31.105 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:31.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:31.106 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:50:31.117 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:50:31.118 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:50:31.207 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:50:31.213 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-20T22:50:31.213 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:50:31.229 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:31.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:31.230 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:50:31.288 INFO:tasks.rados.rados.0.smithi084.stdout:2153: done (5 left) 2024-09-20T22:50:31.288 INFO:tasks.rados.rados.0.smithi084.stdout:2154: write oid 8 current snap is 238 2024-09-20T22:50:31.288 INFO:tasks.rados.rados.0.smithi084.stdout:2154: seq_num 753 ranges {655671=581444,2030588=625083} 2024-09-20T22:50:31.294 INFO:tasks.rados.rados.0.smithi084.stdout:2154: writing smithi08477203-8 from 655671 to 1237115 tid 1 2024-09-20T22:50:31.303 INFO:tasks.rados.rados.0.smithi084.stdout:2154: writing smithi08477203-8 from 2030588 to 2655671 tid 2 2024-09-20T22:50:31.303 INFO:tasks.rados.rados.0.smithi084.stdout:2155: rollback oid 31 current snap is 238 2024-09-20T22:50:31.303 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 31 to 234 2024-09-20T22:50:31.303 INFO:tasks.rados.rados.0.smithi084.stdout:2148: finishing write tid 1 to smithi08477203-30 2024-09-20T22:50:31.303 INFO:tasks.rados.rados.0.smithi084.stdout:2148: finishing write tid 2 to smithi08477203-30 2024-09-20T22:50:31.303 INFO:tasks.rados.rados.0.smithi084.stdout:2148: finishing write tid 3 to smithi08477203-30 2024-09-20T22:50:31.303 INFO:tasks.rados.rados.0.smithi084.stdout:2148: finishing write tid 4 to smithi08477203-30 2024-09-20T22:50:31.304 INFO:tasks.rados.rados.0.smithi084.stdout:2148: oid 30 updating version 0 to 799 2024-09-20T22:50:31.304 INFO:tasks.rados.rados.0.smithi084.stdout:2148: oid 30 updating version 799 to 800 2024-09-20T22:50:31.304 INFO:tasks.rados.rados.0.smithi084.stdout:2148: oid 30 updating version 800 to 801 2024-09-20T22:50:31.304 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 30 v 801 (ObjNum 750 snap 238 seq_num 750) dirty exists 2024-09-20T22:50:31.304 INFO:tasks.rados.rados.0.smithi084.stdout:2148: left oid 30 (ObjNum 750 snap 238 seq_num 750) 2024-09-20T22:50:31.304 INFO:tasks.rados.rados.0.smithi084.stdout:2149: finishing write tid 1 to smithi08477203-34 2024-09-20T22:50:31.304 INFO:tasks.rados.rados.0.smithi084.stdout:2149: finishing write tid 2 to smithi08477203-34 2024-09-20T22:50:31.304 INFO:tasks.rados.rados.0.smithi084.stdout:2149: finishing write tid 3 to smithi08477203-34 2024-09-20T22:50:31.304 INFO:tasks.rados.rados.0.smithi084.stdout:2149: finishing write tid 4 to smithi08477203-34 2024-09-20T22:50:31.304 INFO:tasks.rados.rados.0.smithi084.stdout:2149: oid 34 updating version 0 to 727 2024-09-20T22:50:31.304 INFO:tasks.rados.rados.0.smithi084.stdout:2149: oid 34 version 727 is already newer than 726 2024-09-20T22:50:31.304 INFO:tasks.rados.rados.0.smithi084.stdout:2149: oid 34 updating version 727 to 728 2024-09-20T22:50:31.304 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 34 v 728 (ObjNum 751 snap 238 seq_num 751) dirty exists 2024-09-20T22:50:31.304 INFO:tasks.rados.rados.0.smithi084.stdout:2149: left oid 34 (ObjNum 751 snap 238 seq_num 751) 2024-09-20T22:50:31.304 INFO:tasks.rados.rados.0.smithi084.stdout:2152: finishing write tid 1 to smithi08477203-7 2024-09-20T22:50:31.305 INFO:tasks.rados.rados.0.smithi084.stdout:2148: done (6 left) 2024-09-20T22:50:31.305 INFO:tasks.rados.rados.0.smithi084.stdout:2149: done (5 left) 2024-09-20T22:50:31.305 INFO:tasks.rados.rados.0.smithi084.stdout:2156: write oid 6 current snap is 238 2024-09-20T22:50:31.305 INFO:tasks.rados.rados.0.smithi084.stdout:2152: finishing write tid 2 to smithi08477203-7 2024-09-20T22:50:31.305 INFO:tasks.rados.rados.0.smithi084.stdout:2152: finishing write tid 3 to smithi08477203-7 2024-09-20T22:50:31.305 INFO:tasks.rados.rados.0.smithi084.stdout:2152: finishing write tid 4 to smithi08477203-7 2024-09-20T22:50:31.305 INFO:tasks.rados.rados.0.smithi084.stdout:2152: oid 7 updating version 0 to 855 2024-09-20T22:50:31.305 INFO:tasks.rados.rados.0.smithi084.stdout:2152: oid 7 version 855 is already newer than 854 2024-09-20T22:50:31.305 INFO:tasks.rados.rados.0.smithi084.stdout:2152: oid 7 updating version 855 to 856 2024-09-20T22:50:31.305 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 7 v 856 (ObjNum 752 snap 238 seq_num 752) dirty exists 2024-09-20T22:50:31.305 INFO:tasks.rados.rados.0.smithi084.stdout:2152: left oid 7 (ObjNum 752 snap 238 seq_num 752) 2024-09-20T22:50:31.305 INFO:tasks.rados.rados.0.smithi084.stdout:2150: expect (ObjNum 440 snap 115 seq_num 440) 2024-09-20T22:50:31.306 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:31 smithi084 ceph-mon[29774]: pgmap v726: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 500 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 2.5 KiB/s rd, 1.5 MiB/s wr, 8 op/s 2024-09-20T22:50:31.306 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:31 smithi084 ceph-mon[29774]: osdmap e550: 8 total, 8 up, 8 in 2024-09-20T22:50:31.306 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:31 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/1006265998' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:50:31.306 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:31 smithi084 ceph-mon[40024]: pgmap v726: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 500 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 2.5 KiB/s rd, 1.5 MiB/s wr, 8 op/s 2024-09-20T22:50:31.306 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:31 smithi084 ceph-mon[40024]: osdmap e550: 8 total, 8 up, 8 in 2024-09-20T22:50:31.306 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:31 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/1006265998' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:50:31.331 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:50:31.353 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:31.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:31.354 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:50:31.455 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:50:31.479 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:31.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:31.480 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:50:31.572 INFO:tasks.rados.rados.0.smithi084.stdout:2156: seq_num 754 ranges {672478=656693,2089070=459917} 2024-09-20T22:50:31.579 INFO:tasks.rados.rados.0.smithi084.stdout:2156: writing smithi08477203-6 from 672478 to 1329171 tid 1 2024-09-20T22:50:31.581 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:50:31.587 INFO:tasks.rados.rados.0.smithi084.stdout:2156: writing smithi08477203-6 from 2089070 to 2548987 tid 2 2024-09-20T22:50:31.587 INFO:tasks.rados.rados.0.smithi084.stdout:2150: done (5 left) 2024-09-20T22:50:31.587 INFO:tasks.rados.rados.0.smithi084.stdout:2151: done (4 left) 2024-09-20T22:50:31.587 INFO:tasks.rados.rados.0.smithi084.stdout:2152: done (3 left) 2024-09-20T22:50:31.587 INFO:tasks.rados.rados.0.smithi084.stdout:2157: copy_from oid 42 from oid 27 current snap is 238 2024-09-20T22:50:31.588 INFO:tasks.rados.rados.0.smithi084.stdout:2154: finishing write tid 1 to smithi08477203-8 2024-09-20T22:50:31.588 INFO:tasks.rados.rados.0.smithi084.stdout:2154: finishing write tid 2 to smithi08477203-8 2024-09-20T22:50:31.588 INFO:tasks.rados.rados.0.smithi084.stdout:2154: finishing write tid 3 to smithi08477203-8 2024-09-20T22:50:31.588 INFO:tasks.rados.rados.0.smithi084.stdout:2154: finishing write tid 4 to smithi08477203-8 2024-09-20T22:50:31.588 INFO:tasks.rados.rados.0.smithi084.stdout:2154: oid 8 updating version 0 to 802 2024-09-20T22:50:31.588 INFO:tasks.rados.rados.0.smithi084.stdout:2154: oid 8 updating version 802 to 803 2024-09-20T22:50:31.589 INFO:tasks.rados.rados.0.smithi084.stdout:2154: oid 8 updating version 803 to 804 2024-09-20T22:50:31.589 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 8 v 804 (ObjNum 753 snap 238 seq_num 753) dirty exists 2024-09-20T22:50:31.589 INFO:tasks.rados.rados.0.smithi084.stdout:2154: left oid 8 (ObjNum 753 snap 238 seq_num 753) 2024-09-20T22:50:31.589 INFO:tasks.rados.rados.0.smithi084.stdout:2155: finishing rollback tid 0 to smithi08477203-31 2024-09-20T22:50:31.589 INFO:tasks.rados.rados.0.smithi084.stdout:2154: done (3 left) 2024-09-20T22:50:31.589 INFO:tasks.rados.rados.0.smithi084.stdout:2158: snap_create 2024-09-20T22:50:31.589 INFO:tasks.rados.rados.0.smithi084.stdout:2155: finishing rollback tid 1 to smithi08477203-31 2024-09-20T22:50:31.589 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 31 v 760 (ObjNum 65 snap 0 seq_num 1953066355) dirty dne 2024-09-20T22:50:31.589 INFO:tasks.rados.rados.0.smithi084.stdout:2156: finishing write tid 1 to smithi08477203-6 2024-09-20T22:50:31.590 INFO:tasks.rados.rados.0.smithi084.stdout:2156: finishing write tid 2 to smithi08477203-6 2024-09-20T22:50:31.590 INFO:tasks.rados.rados.0.smithi084.stdout:2156: finishing write tid 3 to smithi08477203-6 2024-09-20T22:50:31.590 INFO:tasks.rados.rados.0.smithi084.stdout:2156: finishing write tid 4 to smithi08477203-6 2024-09-20T22:50:31.590 INFO:tasks.rados.rados.0.smithi084.stdout:2156: oid 6 updating version 0 to 748 2024-09-20T22:50:31.590 INFO:tasks.rados.rados.0.smithi084.stdout:2156: oid 6 updating version 748 to 749 2024-09-20T22:50:31.590 INFO:tasks.rados.rados.0.smithi084.stdout:2156: oid 6 version 749 is already newer than 747 2024-09-20T22:50:31.590 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 6 v 749 (ObjNum 754 snap 238 seq_num 754) dirty exists 2024-09-20T22:50:31.590 INFO:tasks.rados.rados.0.smithi084.stdout:2156: left oid 6 (ObjNum 754 snap 238 seq_num 754) 2024-09-20T22:50:31.595 INFO:tasks.rados.rados.0.smithi084.stdout:2157: finishing copy_from racing read to smithi08477203-42 2024-09-20T22:50:31.601 INFO:tasks.rados.rados.0.smithi084.stdout:2157: finishing copy_from to smithi08477203-42 2024-09-20T22:50:31.601 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 42 v 338 (ObjNum 714 snap 224 seq_num 714) dirty exists 2024-09-20T22:50:31.606 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:31.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:31.607 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:50:31.709 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:50:31.734 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:31.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:31.735 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:50:31.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:31 smithi154 ceph-mon[33264]: pgmap v726: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 500 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 2.5 KiB/s rd, 1.5 MiB/s wr, 8 op/s 2024-09-20T22:50:31.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:31 smithi154 ceph-mon[33264]: osdmap e550: 8 total, 8 up, 8 in 2024-09-20T22:50:31.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:31 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/1006265998' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:50:31.836 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:50:31.857 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:31.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:31.858 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:50:31.895 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.0 dump_historic_ops 2024-09-20T22:50:31.960 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:50:31.985 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:31.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:31.986 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:50:32.087 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:50:32.113 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:32.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:32.114 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:50:32.215 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:50:32.240 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:32.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:32.241 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:50:32.290 INFO:tasks.rados.rados.0.smithi084.stdout:2155: done (3 left) 2024-09-20T22:50:32.290 INFO:tasks.rados.rados.0.smithi084.stdout:2156: done (2 left) 2024-09-20T22:50:32.290 INFO:tasks.rados.rados.0.smithi084.stdout:2157: done (1 left) 2024-09-20T22:50:32.291 INFO:tasks.rados.rados.0.smithi084.stdout:2158: done (0 left) 2024-09-20T22:50:32.291 INFO:tasks.rados.rados.0.smithi084.stdout:2159: rollback oid 5 current snap is 239 2024-09-20T22:50:32.291 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 5 to 213 2024-09-20T22:50:32.291 INFO:tasks.rados.rados.0.smithi084.stdout:2160: snap_remove snap 227 2024-09-20T22:50:32.342 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:50:32.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:32.364 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:32.364 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:50:32.465 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:50:32.490 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:32.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:32.491 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:50:32.530 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:32 smithi084 ceph-mon[40024]: osdmap e551: 8 total, 8 up, 8 in 2024-09-20T22:50:32.531 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:32 smithi084 ceph-mon[29774]: osdmap e551: 8 total, 8 up, 8 in 2024-09-20T22:50:32.547 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:50:32.592 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:50:32.617 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:32.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:32.618 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:50:32.718 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:50:32.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:32.744 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:32.744 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:50:32.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:32 smithi154 ceph-mon[33264]: osdmap e551: 8 total, 8 up, 8 in 2024-09-20T22:50:32.777 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:50:32.845 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:50:32.872 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:32.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:32.873 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:50:32.973 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:50:32.996 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:32.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:32.997 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:50:33.098 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:50:33.119 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:33.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:33.120 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:50:33.221 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:50:33.237 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is unset 2024-09-20T22:50:33.244 INFO:tasks.rados.rados.0.smithi084.stdout:2160: done (1 left) 2024-09-20T22:50:33.244 INFO:tasks.rados.rados.0.smithi084.stdout:2161: snap_remove snap 206 2024-09-20T22:50:33.250 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:33.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:33.251 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:50:33.352 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:50:33.375 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:33.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:33.377 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:50:33.477 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:50:33.499 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:33.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:33.500 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:50:33.564 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:33 smithi084 ceph-mon[29774]: pgmap v729: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 500 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 2.5 KiB/s rd, 1.5 MiB/s wr, 8 op/s 2024-09-20T22:50:33.564 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:33 smithi084 ceph-mon[29774]: osdmap e552: 8 total, 8 up, 8 in 2024-09-20T22:50:33.564 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:33 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/1788291594' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:50:33.564 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:33 smithi084 ceph-mon[29774]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:50:33.564 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:33 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/1788291594' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T22:50:33.564 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:33 smithi084 ceph-mon[29774]: osdmap e553: 8 total, 8 up, 8 in 2024-09-20T22:50:33.565 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:33 smithi084 ceph-mon[40024]: pgmap v729: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 500 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 2.5 KiB/s rd, 1.5 MiB/s wr, 8 op/s 2024-09-20T22:50:33.565 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:33 smithi084 ceph-mon[40024]: osdmap e552: 8 total, 8 up, 8 in 2024-09-20T22:50:33.565 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:33 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/1788291594' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:50:33.565 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:33 smithi084 ceph-mon[40024]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:50:33.565 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:33 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/1788291594' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T22:50:33.565 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:33 smithi084 ceph-mon[40024]: osdmap e553: 8 total, 8 up, 8 in 2024-09-20T22:50:33.601 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:50:33.622 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:33.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:33.623 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:50:33.724 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:50:33.748 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:33.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:33.749 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:50:33.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:33 smithi154 ceph-mon[33264]: pgmap v729: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 500 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 2.5 KiB/s rd, 1.5 MiB/s wr, 8 op/s 2024-09-20T22:50:33.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:33 smithi154 ceph-mon[33264]: osdmap e552: 8 total, 8 up, 8 in 2024-09-20T22:50:33.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:33 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/1788291594' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:50:33.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:33 smithi154 ceph-mon[33264]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:50:33.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:33 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/1788291594' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T22:50:33.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:33 smithi154 ceph-mon[33264]: osdmap e553: 8 total, 8 up, 8 in 2024-09-20T22:50:33.834 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-20T22:50:33.836 INFO:teuthology.orchestra.run.smithi084.stderr:dumped all 2024-09-20T22:50:33.849 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:50:33.871 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:33.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:33.872 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:50:33.974 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:50:33.999 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:34.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:34.000 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:50:34.101 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:50:34.127 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:34.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:34.128 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:50:34.228 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:50:34.244 INFO:tasks.rados.rados.0.smithi084.stdout:2161: done (1 left) 2024-09-20T22:50:34.244 INFO:tasks.rados.rados.0.smithi084.stdout:2162: snap_create 2024-09-20T22:50:34.247 INFO:tasks.rados.rados.0.smithi084.stdout:2159: finishing rollback tid 1 to smithi08477203-5 2024-09-20T22:50:34.247 INFO:tasks.rados.rados.0.smithi084.stdout:2159: finishing rollback tid 2 to smithi08477203-5 2024-09-20T22:50:34.247 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 5 v 732 (ObjNum 670 snap 205 seq_num 670) dirty exists 2024-09-20T22:50:34.255 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:34.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:34.256 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:50:34.356 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:50:34.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:34.379 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:34.379 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:50:34.480 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:50:34.505 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:34.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:34.507 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:50:34.567 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-20T22:50:34.607 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:50:34.632 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:34.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:34.633 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:50:34.734 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:50:34.756 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:34.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:34.757 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:50:34.827 INFO:teuthology.orchestra.run.smithi084.stdout:{"pg_ready":true,"pg_map":{"version":732,"stamp":"2024-09-20T22:50:33.744668+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":541888329,"num_objects":441,"num_object_clones":193,"num_object_copies":1323,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":441,"num_whiteouts":12,"num_read":4924,"num_read_kb":3856014,"num_write":3082,"num_write_kb":971185,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":887,"num_bytes_recovered":303303598,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6574,"ondisk_log_size":6574,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":474,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":1515152,"kb_used_data":1107404,"kb_used_omap":12,"kb_used_meta":407539,"kb_avail":748446064,"statfs":{"total":767960285184,"available":766408769536,"internally_reserved":0,"allocated":1133981696,"data_stored":1609845642,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":417320496},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":17468731,"num_objects":5,"num_object_clones":5,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":-4,"num_read":25,"num_read_kb":9965,"num_write":38,"num_write_kb":17457,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000852"},"pg_stats":[{"pgid":"3.1f","version":"86'2","reported_seq":352,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-20T22:50:33.242310+0000","last_change":"2024-09-20T22:49:26.647633+0000","last_active":"2024-09-20T22:50:33.242310+0000","last_peered":"2024-09-20T22:50:33.242310+0000","last_clean":"2024-09-20T22:50:33.242310+0000","last_became_active":"2024-09-20T22:47:49.222114+0000","last_became_peered":"2024-09-20T22:47:49.222114+0000","last_unstale":"2024-09-20T22:50:33.242310+0000","last_undegraded":"2024-09-20T22:50:33.242310+0000","last_fullsized":"2024-09-20T22:50:33.242310+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'2","last_scrub_stamp":"2024-09-20T22:49:26.647557+0000","last_deep_scrub":"86'2","last_deep_scrub_stamp":"2024-09-20T22:42:39.664421+0000","last_clean_scrub_stamp":"2024-09-20T22:49:26.647557+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:40.650510+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":436,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"74'9","reported_seq":326,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-20T22:50:33.239091+0000","last_change":"2024-09-20T22:49:51.091564+0000","last_active":"2024-09-20T22:50:33.239091+0000","last_peered":"2024-09-20T22:50:33.239091+0000","last_clean":"2024-09-20T22:50:33.239091+0000","last_became_active":"2024-09-20T22:48:05.248980+0000","last_became_peered":"2024-09-20T22:48:05.248980+0000","last_unstale":"2024-09-20T22:50:33.239091+0000","last_undegraded":"2024-09-20T22:50:33.239091+0000","last_fullsized":"2024-09-20T22:50:33.239091+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-09-20T22:49:51.091416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:49:51.091416+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:55.306855+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":1009,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-20T22:50:33.244907+0000","last_change":"2024-09-20T22:50:12.718744+0000","last_active":"2024-09-20T22:50:33.244907+0000","last_peered":"2024-09-20T22:50:33.244907+0000","last_clean":"2024-09-20T22:50:33.244907+0000","last_became_active":"2024-09-20T22:41:42.182024+0000","last_became_peered":"2024-09-20T22:41:42.182024+0000","last_unstale":"2024-09-20T22:50:33.244907+0000","last_undegraded":"2024-09-20T22:50:33.244907+0000","last_fullsized":"2024-09-20T22:50:33.244907+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:50:12.718677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:50:12.718677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:51:36.198787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":315,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-20T22:50:33.239442+0000","last_change":"2024-09-20T22:49:49.117854+0000","last_active":"2024-09-20T22:50:33.239442+0000","last_peered":"2024-09-20T22:50:33.239442+0000","last_clean":"2024-09-20T22:50:33.239442+0000","last_became_active":"2024-09-20T22:48:05.247569+0000","last_became_peered":"2024-09-20T22:48:05.247569+0000","last_unstale":"2024-09-20T22:50:33.239442+0000","last_undegraded":"2024-09-20T22:50:33.239442+0000","last_fullsized":"2024-09-20T22:50:33.239442+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:49.117772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:49:49.117772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:51.805116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":996,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-20T22:50:33.249412+0000","last_change":"2024-09-20T22:49:50.043884+0000","last_active":"2024-09-20T22:50:33.249412+0000","last_peered":"2024-09-20T22:50:33.249412+0000","last_clean":"2024-09-20T22:50:33.249412+0000","last_became_active":"2024-09-20T22:41:44.832099+0000","last_became_peered":"2024-09-20T22:41:44.832099+0000","last_unstale":"2024-09-20T22:50:33.249412+0000","last_undegraded":"2024-09-20T22:50:33.249412+0000","last_fullsized":"2024-09-20T22:50:33.249412+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:50.043776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:49:50.043776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:51:00.726179+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":1001,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-20T22:50:33.249722+0000","last_change":"2024-09-20T22:49:10.077843+0000","last_active":"2024-09-20T22:50:33.249722+0000","last_peered":"2024-09-20T22:50:33.249722+0000","last_clean":"2024-09-20T22:50:33.249722+0000","last_became_active":"2024-09-20T22:41:38.157298+0000","last_became_peered":"2024-09-20T22:41:38.157298+0000","last_unstale":"2024-09-20T22:50:33.249722+0000","last_undegraded":"2024-09-20T22:50:33.249722+0000","last_fullsized":"2024-09-20T22:50:33.249722+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:10.077774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:49:10.077774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:38.225830+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"74'15","reported_seq":1065,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-20T22:50:33.248959+0000","last_change":"2024-09-20T22:49:07.042205+0000","last_active":"2024-09-20T22:50:33.248959+0000","last_peered":"2024-09-20T22:50:33.248959+0000","last_clean":"2024-09-20T22:50:33.248959+0000","last_became_active":"2024-09-20T22:47:49.218798+0000","last_became_peered":"2024-09-20T22:47:49.218798+0000","last_unstale":"2024-09-20T22:50:33.248959+0000","last_undegraded":"2024-09-20T22:50:33.248959+0000","last_fullsized":"2024-09-20T22:50:33.248959+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-09-20T22:49:07.042130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:49:07.042130+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:38.225815+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":316,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-20T22:50:33.238983+0000","last_change":"2024-09-20T22:49:59.011922+0000","last_active":"2024-09-20T22:50:33.238983+0000","last_peered":"2024-09-20T22:50:33.238983+0000","last_clean":"2024-09-20T22:50:33.238983+0000","last_became_active":"2024-09-20T22:48:05.247636+0000","last_became_peered":"2024-09-20T22:48:05.247636+0000","last_unstale":"2024-09-20T22:50:33.238983+0000","last_undegraded":"2024-09-20T22:50:33.238983+0000","last_fullsized":"2024-09-20T22:50:33.238983+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:59.011829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:49:59.011829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:51:07.355546+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":1020,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-20T22:50:33.432295+0000","last_change":"2024-09-20T22:49:02.680764+0000","last_active":"2024-09-20T22:50:33.432295+0000","last_peered":"2024-09-20T22:50:33.432295+0000","last_clean":"2024-09-20T22:50:33.432295+0000","last_became_active":"2024-09-20T22:48:05.242059+0000","last_became_peered":"2024-09-20T22:48:05.242059+0000","last_unstale":"2024-09-20T22:50:33.432295+0000","last_undegraded":"2024-09-20T22:50:33.432295+0000","last_fullsized":"2024-09-20T22:50:33.432295+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:02.680682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:46:00.999939+0000","last_clean_scrub_stamp":"2024-09-20T22:49:02.680682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:37.616624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"74'9","reported_seq":325,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-20T22:50:33.239520+0000","last_change":"2024-09-20T22:49:56.048604+0000","last_active":"2024-09-20T22:50:33.239520+0000","last_peered":"2024-09-20T22:50:33.239520+0000","last_clean":"2024-09-20T22:50:33.239520+0000","last_became_active":"2024-09-20T22:48:05.233429+0000","last_became_peered":"2024-09-20T22:48:05.233429+0000","last_unstale":"2024-09-20T22:50:33.239520+0000","last_undegraded":"2024-09-20T22:50:33.239520+0000","last_fullsized":"2024-09-20T22:50:33.239520+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-09-20T22:49:56.048504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:49:56.048504+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:58.603199+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":1012,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-20T22:50:33.431196+0000","last_change":"2024-09-20T22:49:31.649058+0000","last_active":"2024-09-20T22:50:33.431196+0000","last_peered":"2024-09-20T22:50:33.431196+0000","last_clean":"2024-09-20T22:50:33.431196+0000","last_became_active":"2024-09-20T22:47:49.227379+0000","last_became_peered":"2024-09-20T22:47:49.227379+0000","last_unstale":"2024-09-20T22:50:33.431196+0000","last_undegraded":"2024-09-20T22:50:33.431196+0000","last_fullsized":"2024-09-20T22:50:33.431196+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:31.648986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:49:31.648986+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:47.218836+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":348,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-20T22:50:33.241024+0000","last_change":"2024-09-20T22:49:08.664519+0000","last_active":"2024-09-20T22:50:33.241024+0000","last_peered":"2024-09-20T22:50:33.241024+0000","last_clean":"2024-09-20T22:50:33.241024+0000","last_became_active":"2024-09-20T22:47:49.225515+0000","last_became_peered":"2024-09-20T22:47:49.225515+0000","last_unstale":"2024-09-20T22:50:33.241024+0000","last_undegraded":"2024-09-20T22:50:33.241024+0000","last_fullsized":"2024-09-20T22:50:33.241024+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:08.664407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:49:08.664407+0000","last_clean_scrub_stamp":"2024-09-20T22:49:08.664407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:33.922538+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":1028,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-20T22:50:33.431769+0000","last_change":"2024-09-20T22:50:17.735425+0000","last_active":"2024-09-20T22:50:33.431769+0000","last_peered":"2024-09-20T22:50:33.431769+0000","last_clean":"2024-09-20T22:50:33.431769+0000","last_became_active":"2024-09-20T22:48:05.242084+0000","last_became_peered":"2024-09-20T22:48:05.242084+0000","last_unstale":"2024-09-20T22:50:33.431769+0000","last_undegraded":"2024-09-20T22:50:33.431769+0000","last_fullsized":"2024-09-20T22:50:33.431769+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:50:17.735354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:50:17.735354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:51:25.918783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"181'7","reported_seq":321,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-20T22:50:33.239642+0000","last_change":"2024-09-20T22:48:41.049491+0000","last_active":"2024-09-20T22:50:33.239642+0000","last_peered":"2024-09-20T22:50:33.239642+0000","last_clean":"2024-09-20T22:50:33.239642+0000","last_became_active":"2024-09-20T22:48:05.249763+0000","last_became_peered":"2024-09-20T22:48:05.249763+0000","last_unstale":"2024-09-20T22:50:33.239642+0000","last_undegraded":"2024-09-20T22:50:33.239642+0000","last_fullsized":"2024-09-20T22:50:33.239642+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"181'7","last_scrub_stamp":"2024-09-20T22:48:41.049381+0000","last_deep_scrub":"181'7","last_deep_scrub_stamp":"2024-09-20T22:48:41.049381+0000","last_clean_scrub_stamp":"2024-09-20T22:48:41.049381+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:33.960932+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":6,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":1034,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-20T22:50:33.242905+0000","last_change":"2024-09-20T22:49:48.313924+0000","last_active":"2024-09-20T22:50:33.242905+0000","last_peered":"2024-09-20T22:50:33.242905+0000","last_clean":"2024-09-20T22:50:33.242905+0000","last_became_active":"2024-09-20T22:48:05.237791+0000","last_became_peered":"2024-09-20T22:48:05.237791+0000","last_unstale":"2024-09-20T22:50:33.242905+0000","last_undegraded":"2024-09-20T22:50:33.242905+0000","last_fullsized":"2024-09-20T22:50:33.242905+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:48.313860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:47:40.361958+0000","last_clean_scrub_stamp":"2024-09-20T22:49:48.313860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:49.887950+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":996,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-20T22:50:33.430731+0000","last_change":"2024-09-20T22:49:04.679724+0000","last_active":"2024-09-20T22:50:33.430731+0000","last_peered":"2024-09-20T22:50:33.430731+0000","last_clean":"2024-09-20T22:50:33.430731+0000","last_became_active":"2024-09-20T22:41:44.197927+0000","last_became_peered":"2024-09-20T22:41:44.197927+0000","last_unstale":"2024-09-20T22:50:33.430731+0000","last_undegraded":"2024-09-20T22:50:33.430731+0000","last_fullsized":"2024-09-20T22:50:33.430731+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:04.679617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:49:04.679617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:37.616590+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":225,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-20T22:50:33.241978+0000","last_change":"2024-09-20T22:50:16.816915+0000","last_active":"2024-09-20T22:50:33.241978+0000","last_peered":"2024-09-20T22:50:33.241978+0000","last_clean":"2024-09-20T22:50:33.241978+0000","last_became_active":"2024-09-20T22:48:50.223895+0000","last_became_peered":"2024-09-20T22:48:50.223895+0000","last_unstale":"2024-09-20T22:50:33.241978+0000","last_undegraded":"2024-09-20T22:50:33.241978+0000","last_fullsized":"2024-09-20T22:50:33.241978+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:50:16.816835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:50:16.816835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:51:28.244375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":354,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-20T22:50:33.241425+0000","last_change":"2024-09-20T22:49:53.933673+0000","last_active":"2024-09-20T22:50:33.241425+0000","last_peered":"2024-09-20T22:50:33.241425+0000","last_clean":"2024-09-20T22:50:33.241425+0000","last_became_active":"2024-09-20T22:48:05.233687+0000","last_became_peered":"2024-09-20T22:48:05.233687+0000","last_unstale":"2024-09-20T22:50:33.241425+0000","last_undegraded":"2024-09-20T22:50:33.241425+0000","last_fullsized":"2024-09-20T22:50:33.241425+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:53.933601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:49:53.933601+0000","last_clean_scrub_stamp":"2024-09-20T22:49:53.933601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:51:03.708862+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"113'22","reported_seq":1051,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-20T22:50:33.239136+0000","last_change":"2024-09-20T22:49:52.154967+0000","last_active":"2024-09-20T22:50:33.239136+0000","last_peered":"2024-09-20T22:50:33.239136+0000","last_clean":"2024-09-20T22:50:33.239136+0000","last_became_active":"2024-09-20T22:41:40.170131+0000","last_became_peered":"2024-09-20T22:41:40.170131+0000","last_unstale":"2024-09-20T22:50:33.239136+0000","last_undegraded":"2024-09-20T22:50:33.239136+0000","last_fullsized":"2024-09-20T22:50:33.239136+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'22","last_scrub_stamp":"2024-09-20T22:49:52.154856+0000","last_deep_scrub":"113'22","last_deep_scrub_stamp":"2024-09-20T22:43:12.191878+0000","last_clean_scrub_stamp":"2024-09-20T22:49:52.154856+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:51:04.660560+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":1025,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-20T22:50:33.245950+0000","last_change":"2024-09-20T22:49:01.165679+0000","last_active":"2024-09-20T22:50:33.245950+0000","last_peered":"2024-09-20T22:50:33.245950+0000","last_clean":"2024-09-20T22:50:33.245950+0000","last_became_active":"2024-09-20T22:48:05.249417+0000","last_became_peered":"2024-09-20T22:48:05.249417+0000","last_unstale":"2024-09-20T22:50:33.245950+0000","last_undegraded":"2024-09-20T22:50:33.245950+0000","last_fullsized":"2024-09-20T22:50:33.245950+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:01.165605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:49:01.165605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":989,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-20T22:50:33.249432+0000","last_change":"2024-09-20T22:49:30.132559+0000","last_active":"2024-09-20T22:50:33.249432+0000","last_peered":"2024-09-20T22:50:33.249432+0000","last_clean":"2024-09-20T22:50:33.249432+0000","last_became_active":"2024-09-20T22:41:44.832131+0000","last_became_peered":"2024-09-20T22:41:44.832131+0000","last_unstale":"2024-09-20T22:50:33.249432+0000","last_undegraded":"2024-09-20T22:50:33.249432+0000","last_fullsized":"2024-09-20T22:50:33.249432+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:30.132466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:45:25.002199+0000","last_clean_scrub_stamp":"2024-09-20T22:49:30.132466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:38.225844+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":315,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-20T22:50:33.239666+0000","last_change":"2024-09-20T22:48:24.986582+0000","last_active":"2024-09-20T22:50:33.239666+0000","last_peered":"2024-09-20T22:50:33.239666+0000","last_clean":"2024-09-20T22:50:33.239666+0000","last_became_active":"2024-09-20T22:48:05.250581+0000","last_became_peered":"2024-09-20T22:48:05.250581+0000","last_unstale":"2024-09-20T22:50:33.239666+0000","last_undegraded":"2024-09-20T22:50:33.239666+0000","last_fullsized":"2024-09-20T22:50:33.239666+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:48:24.986484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:48:24.986484+0000","last_clean_scrub_stamp":"2024-09-20T22:48:24.986484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:33.960878+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"74'12","reported_seq":1073,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-20T22:50:33.248854+0000","last_change":"2024-09-20T22:48:59.088132+0000","last_active":"2024-09-20T22:50:33.248854+0000","last_peered":"2024-09-20T22:50:33.248854+0000","last_clean":"2024-09-20T22:50:33.248854+0000","last_became_active":"2024-09-20T22:48:05.233983+0000","last_became_peered":"2024-09-20T22:48:05.233983+0000","last_unstale":"2024-09-20T22:50:33.248854+0000","last_undegraded":"2024-09-20T22:50:33.248854+0000","last_fullsized":"2024-09-20T22:50:33.248854+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-09-20T22:48:59.088030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:48:59.088030+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:38.225785+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":315,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-20T22:50:33.240048+0000","last_change":"2024-09-20T22:48:23.940233+0000","last_active":"2024-09-20T22:50:33.240048+0000","last_peered":"2024-09-20T22:50:33.240048+0000","last_clean":"2024-09-20T22:50:33.240048+0000","last_became_active":"2024-09-20T22:48:05.250319+0000","last_became_peered":"2024-09-20T22:48:05.250319+0000","last_unstale":"2024-09-20T22:50:33.240048+0000","last_undegraded":"2024-09-20T22:50:33.240048+0000","last_fullsized":"2024-09-20T22:50:33.240048+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:48:23.940157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:48:23.940157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:33.960896+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"154'2","reported_seq":997,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-20T22:50:33.243547+0000","last_change":"2024-09-20T22:48:39.357067+0000","last_active":"2024-09-20T22:50:33.243547+0000","last_peered":"2024-09-20T22:50:33.243547+0000","last_clean":"2024-09-20T22:50:33.243547+0000","last_became_active":"2024-09-20T22:41:44.199822+0000","last_became_peered":"2024-09-20T22:41:44.199822+0000","last_unstale":"2024-09-20T22:50:33.243547+0000","last_undegraded":"2024-09-20T22:50:33.243547+0000","last_fullsized":"2024-09-20T22:50:33.243547+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"154'2","last_scrub_stamp":"2024-09-20T22:48:39.356937+0000","last_deep_scrub":"154'2","last_deep_scrub_stamp":"2024-09-20T22:43:54.134066+0000","last_clean_scrub_stamp":"2024-09-20T22:48:39.356937+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"67'1","reported_seq":1030,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-20T22:50:33.245054+0000","last_change":"2024-09-20T22:48:42.188853+0000","last_active":"2024-09-20T22:50:33.245054+0000","last_peered":"2024-09-20T22:50:33.245054+0000","last_clean":"2024-09-20T22:50:33.245054+0000","last_became_active":"2024-09-20T22:48:05.249578+0000","last_became_peered":"2024-09-20T22:48:05.249578+0000","last_unstale":"2024-09-20T22:50:33.245054+0000","last_undegraded":"2024-09-20T22:50:33.245054+0000","last_fullsized":"2024-09-20T22:50:33.245054+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-09-20T22:48:42.188772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:48:42.188772+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"151'15","reported_seq":360,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-20T22:50:33.242734+0000","last_change":"2024-09-20T22:48:37.528315+0000","last_active":"2024-09-20T22:50:33.242734+0000","last_peered":"2024-09-20T22:50:33.242734+0000","last_clean":"2024-09-20T22:50:33.242734+0000","last_became_active":"2024-09-20T22:47:49.232181+0000","last_became_peered":"2024-09-20T22:47:49.232181+0000","last_unstale":"2024-09-20T22:50:33.242734+0000","last_undegraded":"2024-09-20T22:50:33.242734+0000","last_fullsized":"2024-09-20T22:50:33.242734+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"151'15","last_scrub_stamp":"2024-09-20T22:48:37.528207+0000","last_deep_scrub":"151'15","last_deep_scrub_stamp":"2024-09-20T22:43:50.715761+0000","last_clean_scrub_stamp":"2024-09-20T22:48:37.528207+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"408'10","reported_seq":1150,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-20T22:50:33.432022+0000","last_change":"2024-09-20T22:49:47.030294+0000","last_active":"2024-09-20T22:50:33.432022+0000","last_peered":"2024-09-20T22:50:33.432022+0000","last_clean":"2024-09-20T22:50:33.432022+0000","last_became_active":"2024-09-20T22:48:05.244270+0000","last_became_peered":"2024-09-20T22:48:05.244270+0000","last_unstale":"2024-09-20T22:50:33.432022+0000","last_undegraded":"2024-09-20T22:50:33.432022+0000","last_fullsized":"2024-09-20T22:50:33.432022+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"408'10","last_scrub_stamp":"2024-09-20T22:49:47.030217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:49:47.030217+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:56.479090+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.c","version":"553'1500","reported_seq":3746,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-20T22:50:33.433711+0000","last_change":"2024-09-20T22:50:33.433917+0000","last_active":"2024-09-20T22:50:33.433711+0000","last_peered":"2024-09-20T22:50:33.433711+0000","last_clean":"2024-09-20T22:50:33.433711+0000","last_became_active":"2024-09-20T22:42:52.404027+0000","last_became_peered":"2024-09-20T22:42:52.404027+0000","last_unstale":"2024-09-20T22:50:33.433711+0000","last_undegraded":"2024-09-20T22:50:33.433711+0000","last_fullsized":"2024-09-20T22:50:33.433711+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"508'1323","last_scrub_stamp":"2024-09-20T22:49:49.001444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:49:49.001444+0000","objects_scrubbed":31,"log_size":1500,"log_dups_size":0,"ondisk_log_size":1500,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:51:07.482753+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0.0026356019999999999,"stat_sum":{"num_bytes":80718700,"num_objects":35,"num_object_clones":28,"num_object_copies":105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":3,"num_read":638,"num_read_kb":651072,"num_write":452,"num_write_kb":175039,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"ba","length":"1"},{"start":"ca","length":"1"},{"start":"d8","length":"1"},{"start":"e0","length":"1"},{"start":"e3","length":"1"},{"start":"e5","length":"1"},{"start":"e9","length":"1"},{"start":"eb","length":"1"},{"start":"ee","length":"1"}]},{"pgid":"1.a","version":"77'128","reported_seq":1276,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-20T22:50:33.431684+0000","last_change":"2024-09-20T22:50:19.676094+0000","last_active":"2024-09-20T22:50:33.431684+0000","last_peered":"2024-09-20T22:50:33.431684+0000","last_clean":"2024-09-20T22:50:33.431684+0000","last_became_active":"2024-09-20T22:42:52.407791+0000","last_became_peered":"2024-09-20T22:42:52.407791+0000","last_unstale":"2024-09-20T22:50:33.431684+0000","last_undegraded":"2024-09-20T22:50:33.431684+0000","last_fullsized":"2024-09-20T22:50:33.431684+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:50:19.675992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:50:19.675992+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:51:39.072482+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"74'15","reported_seq":1082,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-20T22:50:33.246383+0000","last_change":"2024-09-20T22:49:55.709162+0000","last_active":"2024-09-20T22:50:33.246383+0000","last_peered":"2024-09-20T22:50:33.246383+0000","last_clean":"2024-09-20T22:50:33.246383+0000","last_became_active":"2024-09-20T22:48:05.250308+0000","last_became_peered":"2024-09-20T22:48:05.250308+0000","last_unstale":"2024-09-20T22:50:33.246383+0000","last_undegraded":"2024-09-20T22:50:33.246383+0000","last_fullsized":"2024-09-20T22:50:33.246383+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-09-20T22:49:55.709043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:49:55.709043+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:51:25.064971+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":1001,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-20T22:50:33.248363+0000","last_change":"2024-09-20T22:49:12.070747+0000","last_active":"2024-09-20T22:50:33.248363+0000","last_peered":"2024-09-20T22:50:33.248363+0000","last_clean":"2024-09-20T22:50:33.248363+0000","last_became_active":"2024-09-20T22:41:38.156749+0000","last_became_peered":"2024-09-20T22:41:38.156749+0000","last_unstale":"2024-09-20T22:50:33.248363+0000","last_undegraded":"2024-09-20T22:50:33.248363+0000","last_fullsized":"2024-09-20T22:50:33.248363+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:12.070675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:49:12.070675+0000","last_clean_scrub_stamp":"2024-09-20T22:49:12.070675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:34.126084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"408'12","reported_seq":350,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-20T22:50:33.241057+0000","last_change":"2024-09-20T22:50:22.967349+0000","last_active":"2024-09-20T22:50:33.241057+0000","last_peered":"2024-09-20T22:50:33.241057+0000","last_clean":"2024-09-20T22:50:33.241057+0000","last_became_active":"2024-09-20T22:48:05.242914+0000","last_became_peered":"2024-09-20T22:48:05.242914+0000","last_unstale":"2024-09-20T22:50:33.241057+0000","last_undegraded":"2024-09-20T22:50:33.241057+0000","last_fullsized":"2024-09-20T22:50:33.241057+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"408'12","last_scrub_stamp":"2024-09-20T22:50:22.967270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:50:22.967270+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:51:47.860151+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":1008,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-20T22:50:33.431833+0000","last_change":"2024-09-20T22:49:07.658309+0000","last_active":"2024-09-20T22:50:33.431833+0000","last_peered":"2024-09-20T22:50:33.431833+0000","last_clean":"2024-09-20T22:50:33.431833+0000","last_became_active":"2024-09-20T22:47:49.227039+0000","last_became_peered":"2024-09-20T22:47:49.227039+0000","last_unstale":"2024-09-20T22:50:33.431833+0000","last_undegraded":"2024-09-20T22:50:33.431833+0000","last_fullsized":"2024-09-20T22:50:33.431833+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:07.658234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:49:07.658234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:37.616656+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3","version":"551'204","reported_seq":2332,"reported_epoch":553,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-20T22:50:33.344426+0000","last_change":"2024-09-20T22:50:33.344489+0000","last_active":"2024-09-20T22:50:33.344426+0000","last_peered":"2024-09-20T22:50:33.344426+0000","last_clean":"2024-09-20T22:50:33.344426+0000","last_became_active":"2024-09-20T22:49:05.221402+0000","last_became_peered":"2024-09-20T22:49:05.221402+0000","last_unstale":"2024-09-20T22:50:33.344426+0000","last_undegraded":"2024-09-20T22:50:33.344426+0000","last_fullsized":"2024-09-20T22:50:33.344426+0000","mapping_epoch":404,"log_start":"456'129","ondisk_log_start":"456'129","created":94,"last_epoch_clean":465,"parent":"0.0","parent_split_bits":0,"last_scrub":"135'8","last_scrub_stamp":"2024-09-20T22:47:23.137750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:47:23.137750+0000","objects_scrubbed":0,"log_size":75,"log_dups_size":149,"ondisk_log_size":75,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":6,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.095166609999999999,"stat_sum":{"num_bytes":28076681,"num_objects":10,"num_object_clones":9,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":79,"num_read_kb":104704,"num_write":51,"num_write_kb":19895,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":20196357,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"ba","length":"1"},{"start":"e0","length":"1"},{"start":"e3","length":"1"}]},{"pgid":"1.5","version":"77'128","reported_seq":1175,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-20T22:50:33.240673+0000","last_change":"2024-09-20T22:49:26.023144+0000","last_active":"2024-09-20T22:50:33.240673+0000","last_peered":"2024-09-20T22:50:33.240673+0000","last_clean":"2024-09-20T22:50:33.240673+0000","last_became_active":"2024-09-20T22:48:08.207018+0000","last_became_peered":"2024-09-20T22:48:08.207018+0000","last_unstale":"2024-09-20T22:50:33.240673+0000","last_undegraded":"2024-09-20T22:50:33.240673+0000","last_fullsized":"2024-09-20T22:50:33.240673+0000","mapping_epoch":407,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":408,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:49:26.023044+0000","last_deep_scrub":"77'128","last_deep_scrub_stamp":"2024-09-20T22:45:41.608075+0000","last_clean_scrub_stamp":"2024-09-20T22:49:26.023044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:43.738215+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"74'18","reported_seq":1077,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-20T22:50:33.248649+0000","last_change":"2024-09-20T22:48:43.146697+0000","last_active":"2024-09-20T22:50:33.248649+0000","last_peered":"2024-09-20T22:50:33.248649+0000","last_clean":"2024-09-20T22:50:33.248649+0000","last_became_active":"2024-09-20T22:47:49.217605+0000","last_became_peered":"2024-09-20T22:47:49.217605+0000","last_unstale":"2024-09-20T22:50:33.248649+0000","last_undegraded":"2024-09-20T22:50:33.248649+0000","last_fullsized":"2024-09-20T22:50:33.248649+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'18","last_scrub_stamp":"2024-09-20T22:48:43.146602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:48:43.146602+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:38.225755+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":1021,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-20T22:50:33.249878+0000","last_change":"2024-09-20T22:50:22.220104+0000","last_active":"2024-09-20T22:50:33.249878+0000","last_peered":"2024-09-20T22:50:33.249878+0000","last_clean":"2024-09-20T22:50:33.249878+0000","last_became_active":"2024-09-20T22:47:49.227069+0000","last_became_peered":"2024-09-20T22:47:49.227069+0000","last_unstale":"2024-09-20T22:50:33.249878+0000","last_undegraded":"2024-09-20T22:50:33.249878+0000","last_fullsized":"2024-09-20T22:50:33.249878+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:50:22.219983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:50:22.219983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:51:42.667115+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":316,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-20T22:50:33.240621+0000","last_change":"2024-09-20T22:49:27.968569+0000","last_active":"2024-09-20T22:50:33.240621+0000","last_peered":"2024-09-20T22:50:33.240621+0000","last_clean":"2024-09-20T22:50:33.240621+0000","last_became_active":"2024-09-20T22:48:05.230003+0000","last_became_peered":"2024-09-20T22:48:05.230003+0000","last_unstale":"2024-09-20T22:50:33.240621+0000","last_undegraded":"2024-09-20T22:50:33.240621+0000","last_fullsized":"2024-09-20T22:50:33.240621+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:27.968498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:07.947945+0000","last_clean_scrub_stamp":"2024-09-20T22:49:27.968498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:39.692203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":316,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-20T22:50:33.239568+0000","last_change":"2024-09-20T22:50:16.143396+0000","last_active":"2024-09-20T22:50:33.239568+0000","last_peered":"2024-09-20T22:50:33.239568+0000","last_clean":"2024-09-20T22:50:33.239568+0000","last_became_active":"2024-09-20T22:48:05.239917+0000","last_became_peered":"2024-09-20T22:48:05.239917+0000","last_unstale":"2024-09-20T22:50:33.239568+0000","last_undegraded":"2024-09-20T22:50:33.239568+0000","last_fullsized":"2024-09-20T22:50:33.239568+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:50:16.143329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:53.036601+0000","last_clean_scrub_stamp":"2024-09-20T22:50:16.143329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:51:19.006414+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2","version":"553'565","reported_seq":2113,"reported_epoch":553,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T22:50:33.345811+0000","last_change":"2024-09-20T22:50:33.344478+0000","last_active":"2024-09-20T22:50:33.345811+0000","last_peered":"2024-09-20T22:50:33.345811+0000","last_clean":"2024-09-20T22:50:33.345811+0000","last_became_active":"2024-09-20T22:49:17.242481+0000","last_became_peered":"2024-09-20T22:49:17.242481+0000","last_unstale":"2024-09-20T22:50:33.345811+0000","last_undegraded":"2024-09-20T22:50:33.345811+0000","last_fullsized":"2024-09-20T22:50:33.345811+0000","mapping_epoch":469,"log_start":"471'464","ondisk_log_start":"471'464","created":94,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"282'120","last_scrub_stamp":"2024-09-20T22:46:22.099642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:46:22.099642+0000","objects_scrubbed":1,"log_size":101,"log_dups_size":687,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":8,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":1.0037772869999999,"stat_sum":{"num_bytes":31703134,"num_objects":16,"num_object_clones":12,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":350,"num_read_kb":386471,"num_write":193,"num_write_kb":63312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":120,"num_bytes_recovered":49828787,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"ba","length":"1"}]},{"pgid":"1.4","version":"77'128","reported_seq":1301,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-20T22:50:33.250418+0000","last_change":"2024-09-20T22:49:58.063457+0000","last_active":"2024-09-20T22:50:33.250418+0000","last_peered":"2024-09-20T22:50:33.250418+0000","last_clean":"2024-09-20T22:50:33.250418+0000","last_became_active":"2024-09-20T22:46:15.181305+0000","last_became_peered":"2024-09-20T22:46:15.181305+0000","last_unstale":"2024-09-20T22:50:33.250418+0000","last_undegraded":"2024-09-20T22:50:33.250418+0000","last_fullsized":"2024-09-20T22:50:33.250418+0000","mapping_epoch":94,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:49:58.063382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:49:58.063382+0000","objects_scrubbed":1,"log_size":0,"log_dups_size":59,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:58.916760+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"74'14","reported_seq":332,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-20T22:50:33.240528+0000","last_change":"2024-09-20T22:49:57.026346+0000","last_active":"2024-09-20T22:50:33.240528+0000","last_peered":"2024-09-20T22:50:33.240528+0000","last_clean":"2024-09-20T22:50:33.240528+0000","last_became_active":"2024-09-20T22:48:05.235812+0000","last_became_peered":"2024-09-20T22:48:05.235812+0000","last_unstale":"2024-09-20T22:50:33.240528+0000","last_undegraded":"2024-09-20T22:50:33.240528+0000","last_fullsized":"2024-09-20T22:50:33.240528+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'14","last_scrub_stamp":"2024-09-20T22:49:57.026245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:49:57.026245+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:51:23.191208+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"67'1","reported_seq":362,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-20T22:50:33.242637+0000","last_change":"2024-09-20T22:48:36.570753+0000","last_active":"2024-09-20T22:50:33.242637+0000","last_peered":"2024-09-20T22:50:33.242637+0000","last_clean":"2024-09-20T22:50:33.242637+0000","last_became_active":"2024-09-20T22:48:05.237425+0000","last_became_peered":"2024-09-20T22:48:05.237425+0000","last_unstale":"2024-09-20T22:50:33.242637+0000","last_undegraded":"2024-09-20T22:50:33.242637+0000","last_fullsized":"2024-09-20T22:50:33.242637+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-09-20T22:48:36.570672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:48:36.570672+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"408'12","reported_seq":1149,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-20T22:50:33.249000+0000","last_change":"2024-09-20T22:49:25.151447+0000","last_active":"2024-09-20T22:50:33.249000+0000","last_peered":"2024-09-20T22:50:33.249000+0000","last_clean":"2024-09-20T22:50:33.249000+0000","last_became_active":"2024-09-20T22:48:05.234607+0000","last_became_peered":"2024-09-20T22:48:05.234607+0000","last_unstale":"2024-09-20T22:50:33.249000+0000","last_undegraded":"2024-09-20T22:50:33.249000+0000","last_fullsized":"2024-09-20T22:50:33.249000+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"408'12","last_scrub_stamp":"2024-09-20T22:49:25.151326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:49:25.151326+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:44.177686+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":996,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-20T22:50:33.243142+0000","last_change":"2024-09-20T22:49:31.310399+0000","last_active":"2024-09-20T22:50:33.243142+0000","last_peered":"2024-09-20T22:50:33.243142+0000","last_clean":"2024-09-20T22:50:33.243142+0000","last_became_active":"2024-09-20T22:41:44.831901+0000","last_became_peered":"2024-09-20T22:41:44.831901+0000","last_unstale":"2024-09-20T22:50:33.243142+0000","last_undegraded":"2024-09-20T22:50:33.243142+0000","last_fullsized":"2024-09-20T22:50:33.243142+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:31.310330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:49:31.310330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:34.862425+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1","version":"549'290","reported_seq":2516,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-20T22:50:33.240310+0000","last_change":"2024-09-20T22:50:33.241158+0000","last_active":"2024-09-20T22:50:33.240310+0000","last_peered":"2024-09-20T22:50:33.240310+0000","last_clean":"2024-09-20T22:50:33.240310+0000","last_became_active":"2024-09-20T22:49:21.232750+0000","last_became_peered":"2024-09-20T22:49:21.232750+0000","last_unstale":"2024-09-20T22:50:33.240310+0000","last_undegraded":"2024-09-20T22:50:33.240310+0000","last_fullsized":"2024-09-20T22:50:33.240310+0000","mapping_epoch":407,"log_start":"474'259","ondisk_log_start":"474'259","created":94,"last_epoch_clean":481,"parent":"0.0","parent_split_bits":0,"last_scrub":"420'231","last_scrub_stamp":"2024-09-20T22:48:22.915993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:48:22.915993+0000","objects_scrubbed":6,"log_size":31,"log_dups_size":338,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0011104789999999999,"stat_sum":{"num_bytes":11916658,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":78461,"num_write":64,"num_write_kb":41157,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":8732217,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"ba","length":"1"},{"start":"ca","length":"1"},{"start":"d8","length":"1"},{"start":"e0","length":"1"},{"start":"e3","length":"1"},{"start":"e5","length":"1"},{"start":"e9","length":"1"},{"start":"eb","length":"1"},{"start":"ee","length":"1"}]},{"pgid":"1.7","version":"77'128","reported_seq":1305,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-20T22:50:33.431429+0000","last_change":"2024-09-20T22:49:31.994386+0000","last_active":"2024-09-20T22:50:33.431429+0000","last_peered":"2024-09-20T22:50:33.431429+0000","last_clean":"2024-09-20T22:50:33.431429+0000","last_became_active":"2024-09-20T22:46:03.169596+0000","last_became_peered":"2024-09-20T22:46:03.169596+0000","last_unstale":"2024-09-20T22:50:33.431429+0000","last_undegraded":"2024-09-20T22:50:33.431429+0000","last_fullsized":"2024-09-20T22:50:33.431429+0000","mapping_epoch":94,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:49:31.994311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:49:31.994311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:48.278248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"74'10","reported_seq":1101,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-20T22:50:33.245154+0000","last_change":"2024-09-20T22:49:46.855799+0000","last_active":"2024-09-20T22:50:33.245154+0000","last_peered":"2024-09-20T22:50:33.245154+0000","last_clean":"2024-09-20T22:50:33.245154+0000","last_became_active":"2024-09-20T22:48:05.251200+0000","last_became_peered":"2024-09-20T22:48:05.251200+0000","last_unstale":"2024-09-20T22:50:33.245154+0000","last_undegraded":"2024-09-20T22:50:33.245154+0000","last_fullsized":"2024-09-20T22:50:33.245154+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-09-20T22:49:46.855673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:49:46.855673+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:55.301300+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":1017,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-20T22:50:33.432301+0000","last_change":"2024-09-20T22:50:14.699223+0000","last_active":"2024-09-20T22:50:33.432301+0000","last_peered":"2024-09-20T22:50:33.432301+0000","last_clean":"2024-09-20T22:50:33.432301+0000","last_became_active":"2024-09-20T22:41:38.155629+0000","last_became_peered":"2024-09-20T22:41:38.155629+0000","last_unstale":"2024-09-20T22:50:33.432301+0000","last_undegraded":"2024-09-20T22:50:33.432301+0000","last_fullsized":"2024-09-20T22:50:33.432301+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:50:14.699152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:50:14.699152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:51:22.569678+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"389'11","reported_seq":386,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-20T22:50:33.241445+0000","last_change":"2024-09-20T22:48:44.601907+0000","last_active":"2024-09-20T22:50:33.241445+0000","last_peered":"2024-09-20T22:50:33.241445+0000","last_clean":"2024-09-20T22:50:33.241445+0000","last_became_active":"2024-09-20T22:47:49.227264+0000","last_became_peered":"2024-09-20T22:47:49.227264+0000","last_unstale":"2024-09-20T22:50:33.241445+0000","last_undegraded":"2024-09-20T22:50:33.241445+0000","last_fullsized":"2024-09-20T22:50:33.241445+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'11","last_scrub_stamp":"2024-09-20T22:48:44.601816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:48:44.601816+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":348,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-20T22:50:33.240849+0000","last_change":"2024-09-20T22:49:27.627363+0000","last_active":"2024-09-20T22:50:33.240849+0000","last_peered":"2024-09-20T22:50:33.240849+0000","last_clean":"2024-09-20T22:50:33.240849+0000","last_became_active":"2024-09-20T22:47:49.232380+0000","last_became_peered":"2024-09-20T22:47:49.232380+0000","last_unstale":"2024-09-20T22:50:33.240849+0000","last_undegraded":"2024-09-20T22:50:33.240849+0000","last_fullsized":"2024-09-20T22:50:33.240849+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:27.627295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:49:27.627295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:34.785981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.0","version":"551'352","reported_seq":2638,"reported_epoch":553,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T22:50:33.430517+0000","last_change":"2024-09-20T22:50:32.287237+0000","last_active":"2024-09-20T22:50:33.430517+0000","last_peered":"2024-09-20T22:50:33.430517+0000","last_clean":"2024-09-20T22:50:33.430517+0000","last_became_active":"2024-09-20T22:49:25.236419+0000","last_became_peered":"2024-09-20T22:49:25.236419+0000","last_unstale":"2024-09-20T22:50:33.430517+0000","last_undegraded":"2024-09-20T22:50:33.430517+0000","last_fullsized":"2024-09-20T22:50:33.430517+0000","mapping_epoch":94,"log_start":"480'253","ondisk_log_start":"480'253","created":94,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'228","last_scrub_stamp":"2024-09-20T22:48:59.633944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:48:59.633944+0000","objects_scrubbed":7,"log_size":99,"log_dups_size":416,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":5,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.94002013299999998,"stat_sum":{"num_bytes":35639883,"num_objects":17,"num_object_clones":14,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":169,"num_read_kb":185507,"num_write":119,"num_write_kb":42560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":7603720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"ba","length":"1"},{"start":"d8","length":"1"},{"start":"e0","length":"1"},{"start":"e3","length":"1"}]},{"pgid":"1.6","version":"77'128","reported_seq":1327,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-20T22:50:33.431067+0000","last_change":"2024-09-20T22:49:28.091254+0000","last_active":"2024-09-20T22:50:33.431067+0000","last_peered":"2024-09-20T22:50:33.431067+0000","last_clean":"2024-09-20T22:50:33.431067+0000","last_became_active":"2024-09-20T22:47:52.212190+0000","last_became_peered":"2024-09-20T22:47:52.212190+0000","last_unstale":"2024-09-20T22:50:33.431067+0000","last_undegraded":"2024-09-20T22:50:33.431067+0000","last_fullsized":"2024-09-20T22:50:33.431067+0000","mapping_epoch":391,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:49:28.091184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:49:28.091184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:45.801242+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"515'27","reported_seq":374,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-20T22:50:33.241272+0000","last_change":"2024-09-20T22:49:55.964247+0000","last_active":"2024-09-20T22:50:33.241272+0000","last_peered":"2024-09-20T22:50:33.241272+0000","last_clean":"2024-09-20T22:50:33.241272+0000","last_became_active":"2024-09-20T22:47:49.222009+0000","last_became_peered":"2024-09-20T22:47:49.222009+0000","last_unstale":"2024-09-20T22:50:33.241272+0000","last_undegraded":"2024-09-20T22:50:33.241272+0000","last_fullsized":"2024-09-20T22:50:33.241272+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"515'27","last_scrub_stamp":"2024-09-20T22:49:55.964153+0000","last_deep_scrub":"515'27","last_deep_scrub_stamp":"2024-09-20T22:49:55.964153+0000","last_clean_scrub_stamp":"2024-09-20T22:49:55.964153+0000","objects_scrubbed":8,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:57.993517+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":38,"num_read_kb":24,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":1018,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-20T22:50:33.245630+0000","last_change":"2024-09-20T22:49:28.084441+0000","last_active":"2024-09-20T22:50:33.245630+0000","last_peered":"2024-09-20T22:50:33.245630+0000","last_clean":"2024-09-20T22:50:33.245630+0000","last_became_active":"2024-09-20T22:41:38.160832+0000","last_became_peered":"2024-09-20T22:41:38.160832+0000","last_unstale":"2024-09-20T22:50:33.245630+0000","last_undegraded":"2024-09-20T22:50:33.245630+0000","last_fullsized":"2024-09-20T22:50:33.245630+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:28.084371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:48:23.320031+0000","last_clean_scrub_stamp":"2024-09-20T22:49:28.084371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":1020,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-20T22:50:33.430766+0000","last_change":"2024-09-20T22:50:15.964151+0000","last_active":"2024-09-20T22:50:33.430766+0000","last_peered":"2024-09-20T22:50:33.430766+0000","last_clean":"2024-09-20T22:50:33.430766+0000","last_became_active":"2024-09-20T22:47:49.214055+0000","last_became_peered":"2024-09-20T22:47:49.214055+0000","last_unstale":"2024-09-20T22:50:33.430766+0000","last_undegraded":"2024-09-20T22:50:33.430766+0000","last_fullsized":"2024-09-20T22:50:33.430766+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:50:15.964053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:50:15.964053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:51:24.250449+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":1006,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-20T22:50:33.244807+0000","last_change":"2024-09-20T22:50:21.729843+0000","last_active":"2024-09-20T22:50:33.244807+0000","last_peered":"2024-09-20T22:50:33.244807+0000","last_clean":"2024-09-20T22:50:33.244807+0000","last_became_active":"2024-09-20T22:41:44.830538+0000","last_became_peered":"2024-09-20T22:41:44.830538+0000","last_unstale":"2024-09-20T22:50:33.244807+0000","last_undegraded":"2024-09-20T22:50:33.244807+0000","last_fullsized":"2024-09-20T22:50:33.244807+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:50:21.729772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:48:41.208853+0000","last_clean_scrub_stamp":"2024-09-20T22:50:21.729772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:51:24.056210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.7","version":"551'768","reported_seq":3148,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-20T22:50:33.433419+0000","last_change":"2024-09-20T22:50:33.433589+0000","last_active":"2024-09-20T22:50:33.433419+0000","last_peered":"2024-09-20T22:50:33.433419+0000","last_clean":"2024-09-20T22:50:33.433419+0000","last_became_active":"2024-09-20T22:48:43.224386+0000","last_became_peered":"2024-09-20T22:48:43.224386+0000","last_unstale":"2024-09-20T22:50:33.433419+0000","last_undegraded":"2024-09-20T22:50:33.433419+0000","last_fullsized":"2024-09-20T22:50:33.433419+0000","mapping_epoch":391,"log_start":"436'456","ondisk_log_start":"436'456","created":94,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"463'527","last_scrub_stamp":"2024-09-20T22:49:04.175508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:49:04.175508+0000","objects_scrubbed":23,"log_size":312,"log_dups_size":886,"ondisk_log_size":312,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.0018082700000000001,"stat_sum":{"num_bytes":45921686,"num_objects":27,"num_object_clones":22,"num_object_copies":81,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":2,"num_read":345,"num_read_kb":364160,"num_write":275,"num_write_kb":89472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":115,"num_bytes_recovered":51615167,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"ba","length":"1"},{"start":"ca","length":"1"},{"start":"d8","length":"1"},{"start":"e0","length":"1"},{"start":"e3","length":"1"},{"start":"e5","length":"1"},{"start":"e9","length":"1"},{"start":"eb","length":"1"},{"start":"ee","length":"1"}]},{"pgid":"1.1","version":"77'128","reported_seq":1324,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-20T22:50:33.430447+0000","last_change":"2024-09-20T22:50:20.639645+0000","last_active":"2024-09-20T22:50:33.430447+0000","last_peered":"2024-09-20T22:50:33.430447+0000","last_clean":"2024-09-20T22:50:33.430447+0000","last_became_active":"2024-09-20T22:47:52.208102+0000","last_became_peered":"2024-09-20T22:47:52.208102+0000","last_unstale":"2024-09-20T22:50:33.430447+0000","last_undegraded":"2024-09-20T22:50:33.430447+0000","last_fullsized":"2024-09-20T22:50:33.430447+0000","mapping_epoch":391,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:50:20.639561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:50:20.639561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:51:21.950928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"86'38","reported_seq":1180,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-20T22:50:33.245617+0000","last_change":"2024-09-20T22:49:31.040446+0000","last_active":"2024-09-20T22:50:33.245617+0000","last_peered":"2024-09-20T22:50:33.245617+0000","last_clean":"2024-09-20T22:50:33.245617+0000","last_became_active":"2024-09-20T22:41:40.166958+0000","last_became_peered":"2024-09-20T22:41:40.166958+0000","last_unstale":"2024-09-20T22:50:33.245617+0000","last_undegraded":"2024-09-20T22:50:33.245617+0000","last_fullsized":"2024-09-20T22:50:33.245617+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'38","last_scrub_stamp":"2024-09-20T22:49:31.040362+0000","last_deep_scrub":"86'38","last_deep_scrub_stamp":"2024-09-20T22:42:43.935677+0000","last_clean_scrub_stamp":"2024-09-20T22:49:31.040362+0000","objects_scrubbed":10,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:55.004800+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":308,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-20T22:50:33.239542+0000","last_change":"2024-09-20T22:48:05.238748+0000","last_active":"2024-09-20T22:50:33.239542+0000","last_peered":"2024-09-20T22:50:33.239542+0000","last_clean":"2024-09-20T22:50:33.239542+0000","last_became_active":"2024-09-20T22:48:05.237756+0000","last_became_peered":"2024-09-20T22:48:05.237756+0000","last_unstale":"2024-09-20T22:50:33.239542+0000","last_undegraded":"2024-09-20T22:50:33.239542+0000","last_fullsized":"2024-09-20T22:50:33.239542+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:47:16.817468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:47:16.817468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:33.960789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"69'2","reported_seq":1027,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-20T22:50:33.431580+0000","last_change":"2024-09-20T22:49:27.604083+0000","last_active":"2024-09-20T22:50:33.431580+0000","last_peered":"2024-09-20T22:50:33.431580+0000","last_clean":"2024-09-20T22:50:33.431580+0000","last_became_active":"2024-09-20T22:41:37.148479+0000","last_became_peered":"2024-09-20T22:41:37.148479+0000","last_unstale":"2024-09-20T22:50:33.431580+0000","last_undegraded":"2024-09-20T22:50:33.431580+0000","last_fullsized":"2024-09-20T22:50:33.431580+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2024-09-20T22:49:27.603975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:36.130684+0000","last_clean_scrub_stamp":"2024-09-20T22:49:27.603975+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:37.292200+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00087280599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":354,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-20T22:50:33.241476+0000","last_change":"2024-09-20T22:49:02.723760+0000","last_active":"2024-09-20T22:50:33.241476+0000","last_peered":"2024-09-20T22:50:33.241476+0000","last_clean":"2024-09-20T22:50:33.241476+0000","last_became_active":"2024-09-20T22:48:05.233519+0000","last_became_peered":"2024-09-20T22:48:05.233519+0000","last_unstale":"2024-09-20T22:50:33.241476+0000","last_undegraded":"2024-09-20T22:50:33.241476+0000","last_fullsized":"2024-09-20T22:50:33.241476+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:02.723689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:49:02.723689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":1009,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-20T22:50:33.249425+0000","last_change":"2024-09-20T22:49:57.022315+0000","last_active":"2024-09-20T22:50:33.249425+0000","last_peered":"2024-09-20T22:50:33.249425+0000","last_clean":"2024-09-20T22:50:33.249425+0000","last_became_active":"2024-09-20T22:48:05.234193+0000","last_became_peered":"2024-09-20T22:48:05.234193+0000","last_unstale":"2024-09-20T22:50:33.249425+0000","last_undegraded":"2024-09-20T22:50:33.249425+0000","last_fullsized":"2024-09-20T22:50:33.249425+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:57.022244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:49:57.022244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:51:26.737278+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"553'268","reported_seq":1021,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-20T22:50:33.245983+0000","last_change":"2024-09-20T22:50:33.246258+0000","last_active":"2024-09-20T22:50:33.245983+0000","last_peered":"2024-09-20T22:50:33.245983+0000","last_clean":"2024-09-20T22:50:33.245983+0000","last_became_active":"2024-09-20T22:49:01.223578+0000","last_became_peered":"2024-09-20T22:49:01.223578+0000","last_unstale":"2024-09-20T22:50:33.245983+0000","last_undegraded":"2024-09-20T22:50:33.245983+0000","last_fullsized":"2024-09-20T22:50:33.245983+0000","mapping_epoch":404,"log_start":"456'176","ondisk_log_start":"456'176","created":94,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":0,"last_scrub":"461'178","last_scrub_stamp":"2024-09-20T22:49:01.696205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:49:01.696205+0000","objects_scrubbed":10,"log_size":92,"log_dups_size":318,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0045193289999999999,"stat_sum":{"num_bytes":22808002,"num_objects":10,"num_object_clones":8,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":1,"num_read":117,"num_read_kb":115419,"num_write":92,"num_write_kb":32954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":10294275,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"ba","length":"1"},{"start":"ca","length":"1"},{"start":"d8","length":"1"},{"start":"e0","length":"1"},{"start":"e3","length":"1"},{"start":"e5","length":"1"},{"start":"e9","length":"1"},{"start":"eb","length":"1"},{"start":"ee","length":"1"}]},{"pgid":"1.2","version":"77'128","reported_seq":1385,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-20T22:50:33.242616+0000","last_change":"2024-09-20T22:49:55.364495+0000","last_active":"2024-09-20T22:50:33.242616+0000","last_peered":"2024-09-20T22:50:33.242616+0000","last_clean":"2024-09-20T22:50:33.242616+0000","last_became_active":"2024-09-20T22:48:08.203028+0000","last_became_peered":"2024-09-20T22:48:08.203028+0000","last_unstale":"2024-09-20T22:50:33.242616+0000","last_undegraded":"2024-09-20T22:50:33.242616+0000","last_fullsized":"2024-09-20T22:50:33.242616+0000","mapping_epoch":407,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":408,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:49:55.364421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:49:55.364421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:51:00.893241+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"443'18","reported_seq":1082,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-20T22:50:33.246423+0000","last_change":"2024-09-20T22:49:57.739451+0000","last_active":"2024-09-20T22:50:33.246423+0000","last_peered":"2024-09-20T22:50:33.246423+0000","last_clean":"2024-09-20T22:50:33.246423+0000","last_became_active":"2024-09-20T22:47:49.228702+0000","last_became_peered":"2024-09-20T22:47:49.228702+0000","last_unstale":"2024-09-20T22:50:33.246423+0000","last_undegraded":"2024-09-20T22:50:33.246423+0000","last_fullsized":"2024-09-20T22:50:33.246423+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"443'18","last_scrub_stamp":"2024-09-20T22:49:57.739362+0000","last_deep_scrub":"443'18","last_deep_scrub_stamp":"2024-09-20T22:48:44.182517+0000","last_clean_scrub_stamp":"2024-09-20T22:49:57.739362+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:51:00.790911+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":1018,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-20T22:50:33.244504+0000","last_change":"2024-09-20T22:49:26.102066+0000","last_active":"2024-09-20T22:50:33.244504+0000","last_peered":"2024-09-20T22:50:33.244504+0000","last_clean":"2024-09-20T22:50:33.244504+0000","last_became_active":"2024-09-20T22:41:38.156837+0000","last_became_peered":"2024-09-20T22:41:38.156837+0000","last_unstale":"2024-09-20T22:50:33.244504+0000","last_undegraded":"2024-09-20T22:50:33.244504+0000","last_fullsized":"2024-09-20T22:50:33.244504+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:26.101994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:49:26.101994+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"380'2","reported_seq":1054,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-20T22:50:33.239409+0000","last_change":"2024-09-20T22:49:30.148041+0000","last_active":"2024-09-20T22:50:33.239409+0000","last_peered":"2024-09-20T22:50:33.239409+0000","last_clean":"2024-09-20T22:50:33.239409+0000","last_became_active":"2024-09-20T22:47:49.222812+0000","last_became_peered":"2024-09-20T22:47:49.222812+0000","last_unstale":"2024-09-20T22:50:33.239409+0000","last_undegraded":"2024-09-20T22:50:33.239409+0000","last_fullsized":"2024-09-20T22:50:33.239409+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"380'2","last_scrub_stamp":"2024-09-20T22:49:30.147961+0000","last_deep_scrub":"380'2","last_deep_scrub_stamp":"2024-09-20T22:47:40.983039+0000","last_clean_scrub_stamp":"2024-09-20T22:49:30.147961+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001301459,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":1008,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-20T22:50:33.240203+0000","last_change":"2024-09-20T22:49:27.195816+0000","last_active":"2024-09-20T22:50:33.240203+0000","last_peered":"2024-09-20T22:50:33.240203+0000","last_clean":"2024-09-20T22:50:33.240203+0000","last_became_active":"2024-09-20T22:41:42.185112+0000","last_became_peered":"2024-09-20T22:41:42.185112+0000","last_unstale":"2024-09-20T22:50:33.240203+0000","last_undegraded":"2024-09-20T22:50:33.240203+0000","last_fullsized":"2024-09-20T22:50:33.240203+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:27.195708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:47:22.881109+0000","last_clean_scrub_stamp":"2024-09-20T22:49:27.195708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:46.992983+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":996,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-20T22:50:33.243630+0000","last_change":"2024-09-20T22:49:04.332292+0000","last_active":"2024-09-20T22:50:33.243630+0000","last_peered":"2024-09-20T22:50:33.243630+0000","last_clean":"2024-09-20T22:50:33.243630+0000","last_became_active":"2024-09-20T22:41:44.830515+0000","last_became_peered":"2024-09-20T22:41:44.830515+0000","last_unstale":"2024-09-20T22:50:33.243630+0000","last_undegraded":"2024-09-20T22:50:33.243630+0000","last_fullsized":"2024-09-20T22:50:33.243630+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:04.332219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:49:04.332219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"549'233","reported_seq":2574,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-20T22:50:33.242729+0000","last_change":"2024-09-20T22:50:33.243031+0000","last_active":"2024-09-20T22:50:33.242729+0000","last_peered":"2024-09-20T22:50:33.242729+0000","last_clean":"2024-09-20T22:50:33.242729+0000","last_became_active":"2024-09-20T22:48:57.236417+0000","last_became_peered":"2024-09-20T22:48:57.236417+0000","last_unstale":"2024-09-20T22:50:33.242729+0000","last_undegraded":"2024-09-20T22:50:33.242729+0000","last_fullsized":"2024-09-20T22:50:33.242729+0000","mapping_epoch":391,"log_start":"450'190","ondisk_log_start":"450'190","created":94,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"534'227","last_scrub_stamp":"2024-09-20T22:50:14.435645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:50:14.435645+0000","objects_scrubbed":5,"log_size":43,"log_dups_size":240,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:51:31.210209+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00090684999999999995,"stat_sum":{"num_bytes":11466153,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":57,"num_read_kb":38715,"num_write":75,"num_write_kb":33558,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":19673726,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"ba","length":"1"},{"start":"ca","length":"1"},{"start":"d8","length":"1"},{"start":"e0","length":"1"},{"start":"e3","length":"1"},{"start":"e5","length":"1"},{"start":"e9","length":"1"},{"start":"eb","length":"1"},{"start":"ee","length":"1"}]},{"pgid":"1.3","version":"77'128","reported_seq":1334,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-20T22:50:33.245429+0000","last_change":"2024-09-20T22:49:49.789854+0000","last_active":"2024-09-20T22:50:33.245429+0000","last_peered":"2024-09-20T22:50:33.245429+0000","last_clean":"2024-09-20T22:50:33.245429+0000","last_became_active":"2024-09-20T22:48:08.208402+0000","last_became_peered":"2024-09-20T22:48:08.208402+0000","last_unstale":"2024-09-20T22:50:33.245429+0000","last_undegraded":"2024-09-20T22:50:33.245429+0000","last_fullsized":"2024-09-20T22:50:33.245429+0000","mapping_epoch":407,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":408,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:49:49.789778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:49:49.789778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:51:15.613397+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"74'12","reported_seq":362,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-20T22:50:33.242777+0000","last_change":"2024-09-20T22:50:14.746048+0000","last_active":"2024-09-20T22:50:33.242777+0000","last_peered":"2024-09-20T22:50:33.242777+0000","last_clean":"2024-09-20T22:50:33.242777+0000","last_became_active":"2024-09-20T22:47:49.224223+0000","last_became_peered":"2024-09-20T22:47:49.224223+0000","last_unstale":"2024-09-20T22:50:33.242777+0000","last_undegraded":"2024-09-20T22:50:33.242777+0000","last_fullsized":"2024-09-20T22:50:33.242777+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-09-20T22:50:14.745918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:50:14.745918+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:51:42.186355+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":354,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-20T22:50:33.240766+0000","last_change":"2024-09-20T22:48:05.233412+0000","last_active":"2024-09-20T22:50:33.240766+0000","last_peered":"2024-09-20T22:50:33.240766+0000","last_clean":"2024-09-20T22:50:33.240766+0000","last_became_active":"2024-09-20T22:48:05.233093+0000","last_became_peered":"2024-09-20T22:48:05.233093+0000","last_unstale":"2024-09-20T22:50:33.240766+0000","last_undegraded":"2024-09-20T22:50:33.240766+0000","last_fullsized":"2024-09-20T22:50:33.240766+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:47:55.576301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:44:19.834641+0000","last_clean_scrub_stamp":"2024-09-20T22:47:55.576301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"75'5","reported_seq":1575,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-20T22:50:33.243227+0000","last_change":"2024-09-20T22:50:21.363580+0000","last_active":"2024-09-20T22:50:33.243227+0000","last_peered":"2024-09-20T22:50:33.243227+0000","last_clean":"2024-09-20T22:50:33.243227+0000","last_became_active":"2024-09-20T22:47:49.214414+0000","last_became_peered":"2024-09-20T22:47:49.214414+0000","last_unstale":"2024-09-20T22:50:33.243227+0000","last_undegraded":"2024-09-20T22:50:33.243227+0000","last_fullsized":"2024-09-20T22:50:33.243227+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-09-20T22:50:21.363477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:36.130684+0000","last_clean_scrub_stamp":"2024-09-20T22:50:21.363477+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:51:32.143185+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00078631700000000003,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":526,"num_read_kb":521,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":1021,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-20T22:50:33.432705+0000","last_change":"2024-09-20T22:50:18.695099+0000","last_active":"2024-09-20T22:50:33.432705+0000","last_peered":"2024-09-20T22:50:33.432705+0000","last_clean":"2024-09-20T22:50:33.432705+0000","last_became_active":"2024-09-20T22:47:49.227371+0000","last_became_peered":"2024-09-20T22:47:49.227371+0000","last_unstale":"2024-09-20T22:50:33.432705+0000","last_undegraded":"2024-09-20T22:50:33.432705+0000","last_fullsized":"2024-09-20T22:50:33.432705+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:50:18.695027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:50:18.695027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:51:44.689516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":1006,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-20T22:50:33.245282+0000","last_change":"2024-09-20T22:49:10.150277+0000","last_active":"2024-09-20T22:50:33.245282+0000","last_peered":"2024-09-20T22:50:33.245282+0000","last_clean":"2024-09-20T22:50:33.245282+0000","last_became_active":"2024-09-20T22:41:44.199046+0000","last_became_peered":"2024-09-20T22:41:44.199046+0000","last_unstale":"2024-09-20T22:50:33.245282+0000","last_undegraded":"2024-09-20T22:50:33.245282+0000","last_fullsized":"2024-09-20T22:50:33.245282+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:10.150205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:49:10.150205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.6","version":"553'295","reported_seq":2559,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-20T22:50:33.247946+0000","last_change":"2024-09-20T22:50:33.248174+0000","last_active":"2024-09-20T22:50:33.247946+0000","last_peered":"2024-09-20T22:50:33.247946+0000","last_clean":"2024-09-20T22:50:33.247946+0000","last_became_active":"2024-09-20T22:48:47.234531+0000","last_became_peered":"2024-09-20T22:48:47.234531+0000","last_unstale":"2024-09-20T22:50:33.247946+0000","last_undegraded":"2024-09-20T22:50:33.247946+0000","last_fullsized":"2024-09-20T22:50:33.247946+0000","mapping_epoch":391,"log_start":"443'181","ondisk_log_start":"443'181","created":94,"last_epoch_clean":447,"parent":"0.0","parent_split_bits":0,"last_scrub":"440'177","last_scrub_stamp":"2024-09-20T22:48:40.602342+0000","last_deep_scrub":"440'177","last_deep_scrub_stamp":"2024-09-20T22:48:40.602342+0000","last_clean_scrub_stamp":"2024-09-20T22:48:40.602342+0000","objects_scrubbed":4,"log_size":114,"log_dups_size":314,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0.0057790840000000003,"stat_sum":{"num_bytes":21462083,"num_objects":9,"num_object_clones":7,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":1,"num_read":114,"num_read_kb":107760,"num_write":92,"num_write_kb":34064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":18662529,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"ba","length":"1"},{"start":"ca","length":"1"},{"start":"d8","length":"1"},{"start":"e0","length":"1"},{"start":"e3","length":"1"},{"start":"e5","length":"1"},{"start":"e9","length":"1"},{"start":"eb","length":"1"},{"start":"ee","length":"1"}]},{"pgid":"4.5","version":"408'20","reported_seq":1238,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-20T22:50:33.431374+0000","last_change":"2024-09-20T22:49:56.004694+0000","last_active":"2024-09-20T22:50:33.431374+0000","last_peered":"2024-09-20T22:50:33.431374+0000","last_clean":"2024-09-20T22:50:33.431374+0000","last_became_active":"2024-09-20T22:48:05.243523+0000","last_became_peered":"2024-09-20T22:48:05.243523+0000","last_unstale":"2024-09-20T22:50:33.431374+0000","last_undegraded":"2024-09-20T22:50:33.431374+0000","last_fullsized":"2024-09-20T22:50:33.431374+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"408'20","last_scrub_stamp":"2024-09-20T22:49:56.004590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:49:56.004590+0000","objects_scrubbed":8,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:51:25.151724+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"67'1","reported_seq":1007,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-20T22:50:33.249840+0000","last_change":"2024-09-20T22:49:29.161456+0000","last_active":"2024-09-20T22:50:33.249840+0000","last_peered":"2024-09-20T22:50:33.249840+0000","last_clean":"2024-09-20T22:50:33.249840+0000","last_became_active":"2024-09-20T22:41:38.158917+0000","last_became_peered":"2024-09-20T22:41:38.158917+0000","last_unstale":"2024-09-20T22:50:33.249840+0000","last_undegraded":"2024-09-20T22:50:33.249840+0000","last_fullsized":"2024-09-20T22:50:33.249840+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-09-20T22:49:29.161334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:49:29.161334+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:43.763458+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"77'128","reported_seq":1357,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-20T22:50:33.243439+0000","last_change":"2024-09-20T22:49:46.391695+0000","last_active":"2024-09-20T22:50:33.243439+0000","last_peered":"2024-09-20T22:50:33.243439+0000","last_clean":"2024-09-20T22:50:33.243439+0000","last_became_active":"2024-09-20T22:47:52.206169+0000","last_became_peered":"2024-09-20T22:47:52.206169+0000","last_unstale":"2024-09-20T22:50:33.243439+0000","last_undegraded":"2024-09-20T22:50:33.243439+0000","last_fullsized":"2024-09-20T22:50:33.243439+0000","mapping_epoch":391,"log_start":"77'128","ondisk_log_start":"77'128","created":19,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:49:46.391614+0000","last_deep_scrub":"77'128","last_deep_scrub_stamp":"2024-09-20T22:47:22.417099+0000","last_clean_scrub_stamp":"2024-09-20T22:49:46.391614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:51:03.708086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":1000,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-20T22:50:33.243846+0000","last_change":"2024-09-20T22:48:35.359093+0000","last_active":"2024-09-20T22:50:33.243846+0000","last_peered":"2024-09-20T22:50:33.243846+0000","last_clean":"2024-09-20T22:50:33.243846+0000","last_became_active":"2024-09-20T22:41:42.184387+0000","last_became_peered":"2024-09-20T22:41:42.184387+0000","last_unstale":"2024-09-20T22:50:33.243846+0000","last_undegraded":"2024-09-20T22:50:33.243846+0000","last_fullsized":"2024-09-20T22:50:33.243846+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:48:35.359016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:48:35.359016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":1017,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-20T22:50:33.432591+0000","last_change":"2024-09-20T22:49:57.715494+0000","last_active":"2024-09-20T22:50:33.432591+0000","last_peered":"2024-09-20T22:50:33.432591+0000","last_clean":"2024-09-20T22:50:33.432591+0000","last_became_active":"2024-09-20T22:48:05.241602+0000","last_became_peered":"2024-09-20T22:48:05.241602+0000","last_unstale":"2024-09-20T22:50:33.432591+0000","last_undegraded":"2024-09-20T22:50:33.432591+0000","last_fullsized":"2024-09-20T22:50:33.432591+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:57.715416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:49:57.715416+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:51:26.966224+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"553'339","reported_seq":2670,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-20T22:50:33.433153+0000","last_change":"2024-09-20T22:50:33.433362+0000","last_active":"2024-09-20T22:50:33.433153+0000","last_peered":"2024-09-20T22:50:33.433153+0000","last_clean":"2024-09-20T22:50:33.433153+0000","last_became_active":"2024-09-20T22:47:52.208200+0000","last_became_peered":"2024-09-20T22:47:52.208200+0000","last_unstale":"2024-09-20T22:50:33.433153+0000","last_undegraded":"2024-09-20T22:50:33.433153+0000","last_fullsized":"2024-09-20T22:50:33.433153+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"460'246","last_scrub_stamp":"2024-09-20T22:49:01.214729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:49:01.214729+0000","objects_scrubbed":8,"log_size":339,"log_dups_size":0,"ondisk_log_size":339,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0022119650000000002,"stat_sum":{"num_bytes":27981420,"num_objects":11,"num_object_clones":9,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":118,"num_read_kb":106337,"num_write":73,"num_write_kb":32500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":13529610,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"ba","length":"1"},{"start":"ca","length":"1"},{"start":"d8","length":"1"},{"start":"e0","length":"1"},{"start":"e3","length":"1"},{"start":"e5","length":"1"},{"start":"e9","length":"1"},{"start":"eb","length":"1"},{"start":"ee","length":"1"}]},{"pgid":"4.e","version":"74'11","reported_seq":320,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-20T22:50:33.241121+0000","last_change":"2024-09-20T22:48:05.271944+0000","last_active":"2024-09-20T22:50:33.241121+0000","last_peered":"2024-09-20T22:50:33.241121+0000","last_clean":"2024-09-20T22:50:33.241121+0000","last_became_active":"2024-09-20T22:48:05.248271+0000","last_became_peered":"2024-09-20T22:48:05.248271+0000","last_unstale":"2024-09-20T22:50:33.241121+0000","last_undegraded":"2024-09-20T22:50:33.241121+0000","last_fullsized":"2024-09-20T22:50:33.241121+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-09-20T22:47:59.092235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:47:59.092235+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:33.960808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":316,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-20T22:50:33.239483+0000","last_change":"2024-09-20T22:49:30.998080+0000","last_active":"2024-09-20T22:50:33.239483+0000","last_peered":"2024-09-20T22:50:33.239483+0000","last_clean":"2024-09-20T22:50:33.239483+0000","last_became_active":"2024-09-20T22:48:05.254688+0000","last_became_peered":"2024-09-20T22:48:05.254688+0000","last_unstale":"2024-09-20T22:50:33.239483+0000","last_undegraded":"2024-09-20T22:50:33.239483+0000","last_fullsized":"2024-09-20T22:50:33.239483+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:30.998001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:49:30.998001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:47.839033+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":1012,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-20T22:50:33.432564+0000","last_change":"2024-09-20T22:49:48.693437+0000","last_active":"2024-09-20T22:50:33.432564+0000","last_peered":"2024-09-20T22:50:33.432564+0000","last_clean":"2024-09-20T22:50:33.432564+0000","last_became_active":"2024-09-20T22:48:05.233778+0000","last_became_peered":"2024-09-20T22:48:05.233778+0000","last_unstale":"2024-09-20T22:50:33.432564+0000","last_undegraded":"2024-09-20T22:50:33.432564+0000","last_fullsized":"2024-09-20T22:50:33.432564+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:48.693333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:49:48.693333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:56.935700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":988,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-20T22:50:33.250389+0000","last_change":"2024-09-20T22:48:40.127195+0000","last_active":"2024-09-20T22:50:33.250389+0000","last_peered":"2024-09-20T22:50:33.250389+0000","last_clean":"2024-09-20T22:50:33.250389+0000","last_became_active":"2024-09-20T22:41:44.832009+0000","last_became_peered":"2024-09-20T22:41:44.832009+0000","last_unstale":"2024-09-20T22:50:33.250389+0000","last_undegraded":"2024-09-20T22:50:33.250389+0000","last_fullsized":"2024-09-20T22:50:33.250389+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:48:40.127123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:48:40.127123+0000","last_clean_scrub_stamp":"2024-09-20T22:48:40.127123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:38.225769+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.e","version":"551'425","reported_seq":2781,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-20T22:50:33.240070+0000","last_change":"2024-09-20T22:50:33.240500+0000","last_active":"2024-09-20T22:50:33.240070+0000","last_peered":"2024-09-20T22:50:33.240070+0000","last_clean":"2024-09-20T22:50:33.240070+0000","last_became_active":"2024-09-20T22:47:52.209452+0000","last_became_peered":"2024-09-20T22:47:52.209452+0000","last_unstale":"2024-09-20T22:50:33.240070+0000","last_undegraded":"2024-09-20T22:50:33.240070+0000","last_fullsized":"2024-09-20T22:50:33.240070+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'360","last_scrub_stamp":"2024-09-20T22:49:25.149145+0000","last_deep_scrub":"174'60","last_deep_scrub_stamp":"2024-09-20T22:44:18.301776+0000","last_clean_scrub_stamp":"2024-09-20T22:49:25.149145+0000","objects_scrubbed":11,"log_size":425,"log_dups_size":0,"ondisk_log_size":425,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.00067177299999999997,"stat_sum":{"num_bytes":23941620,"num_objects":10,"num_object_clones":8,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":167,"num_read_kb":173053,"num_write":113,"num_write_kb":44753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":23110741,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"ba","length":"1"},{"start":"ca","length":"1"},{"start":"d8","length":"1"},{"start":"e0","length":"1"},{"start":"e3","length":"1"},{"start":"e5","length":"1"},{"start":"e9","length":"1"},{"start":"eb","length":"1"},{"start":"ee","length":"1"}]},{"pgid":"1.8","version":"77'128","reported_seq":1276,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-20T22:50:33.430328+0000","last_change":"2024-09-20T22:49:56.715035+0000","last_active":"2024-09-20T22:50:33.430328+0000","last_peered":"2024-09-20T22:50:33.430328+0000","last_clean":"2024-09-20T22:50:33.430328+0000","last_became_active":"2024-09-20T22:42:52.405594+0000","last_became_peered":"2024-09-20T22:42:52.405594+0000","last_unstale":"2024-09-20T22:50:33.430328+0000","last_undegraded":"2024-09-20T22:50:33.430328+0000","last_fullsized":"2024-09-20T22:50:33.430328+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:49:56.714919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:49:56.714919+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:51:10.398817+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"74'17","reported_seq":334,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-20T22:50:33.240517+0000","last_change":"2024-09-20T22:49:01.949542+0000","last_active":"2024-09-20T22:50:33.240517+0000","last_peered":"2024-09-20T22:50:33.240517+0000","last_clean":"2024-09-20T22:50:33.240517+0000","last_became_active":"2024-09-20T22:48:05.247971+0000","last_became_peered":"2024-09-20T22:48:05.247971+0000","last_unstale":"2024-09-20T22:50:33.240517+0000","last_undegraded":"2024-09-20T22:50:33.240517+0000","last_fullsized":"2024-09-20T22:50:33.240517+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'17","last_scrub_stamp":"2024-09-20T22:49:01.949451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:49:01.949451+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:33.960950+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":28,"num_read_kb":18,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":557,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-20T22:50:33.239562+0000","last_change":"2024-09-20T22:49:32.138822+0000","last_active":"2024-09-20T22:50:33.239562+0000","last_peered":"2024-09-20T22:50:33.239562+0000","last_clean":"2024-09-20T22:50:33.239562+0000","last_became_active":"2024-09-20T22:48:05.240481+0000","last_became_peered":"2024-09-20T22:48:05.240481+0000","last_unstale":"2024-09-20T22:50:33.239562+0000","last_undegraded":"2024-09-20T22:50:33.239562+0000","last_fullsized":"2024-09-20T22:50:33.239562+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:32.138751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:35.492562+0000","last_clean_scrub_stamp":"2024-09-20T22:49:32.138751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:44.032030+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":1025,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-20T22:50:33.245286+0000","last_change":"2024-09-20T22:49:09.188448+0000","last_active":"2024-09-20T22:50:33.245286+0000","last_peered":"2024-09-20T22:50:33.245286+0000","last_clean":"2024-09-20T22:50:33.245286+0000","last_became_active":"2024-09-20T22:48:05.250175+0000","last_became_peered":"2024-09-20T22:48:05.250175+0000","last_unstale":"2024-09-20T22:50:33.245286+0000","last_undegraded":"2024-09-20T22:50:33.245286+0000","last_fullsized":"2024-09-20T22:50:33.245286+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:09.188362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:49:09.188362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:36.362440+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":1023,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-20T22:50:33.239287+0000","last_change":"2024-09-20T22:49:26.152417+0000","last_active":"2024-09-20T22:50:33.239287+0000","last_peered":"2024-09-20T22:50:33.239287+0000","last_clean":"2024-09-20T22:50:33.239287+0000","last_became_active":"2024-09-20T22:48:05.239006+0000","last_became_peered":"2024-09-20T22:48:05.239006+0000","last_unstale":"2024-09-20T22:50:33.239287+0000","last_undegraded":"2024-09-20T22:50:33.239287+0000","last_fullsized":"2024-09-20T22:50:33.239287+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:26.152316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:49:26.152316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:40.036383+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"553'525","reported_seq":2845,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-20T22:50:33.251163+0000","last_change":"2024-09-20T22:50:33.251369+0000","last_active":"2024-09-20T22:50:33.251163+0000","last_peered":"2024-09-20T22:50:33.251163+0000","last_clean":"2024-09-20T22:50:33.251163+0000","last_became_active":"2024-09-20T22:47:52.216198+0000","last_became_peered":"2024-09-20T22:47:52.216198+0000","last_unstale":"2024-09-20T22:50:33.251163+0000","last_undegraded":"2024-09-20T22:50:33.251163+0000","last_fullsized":"2024-09-20T22:50:33.251163+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"481'456","last_scrub_stamp":"2024-09-20T22:49:22.637524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:49:22.637524+0000","objects_scrubbed":9,"log_size":525,"log_dups_size":0,"ondisk_log_size":525,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.0088089090000000002,"stat_sum":{"num_bytes":26560669,"num_objects":12,"num_object_clones":9,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":2,"num_read":244,"num_read_kb":242483,"num_write":150,"num_write_kb":50444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":53,"num_bytes_recovered":14983556,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"ba","length":"1"},{"start":"ca","length":"1"},{"start":"d8","length":"1"},{"start":"e0","length":"1"},{"start":"e3","length":"1"},{"start":"e5","length":"1"},{"start":"e9","length":"1"},{"start":"eb","length":"1"},{"start":"ee","length":"1"}]},{"pgid":"1.9","version":"77'128","reported_seq":1316,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-20T22:50:33.242206+0000","last_change":"2024-09-20T22:49:52.301456+0000","last_active":"2024-09-20T22:50:33.242206+0000","last_peered":"2024-09-20T22:50:33.242206+0000","last_clean":"2024-09-20T22:50:33.242206+0000","last_became_active":"2024-09-20T22:42:52.400353+0000","last_became_peered":"2024-09-20T22:42:52.400353+0000","last_unstale":"2024-09-20T22:50:33.242206+0000","last_undegraded":"2024-09-20T22:50:33.242206+0000","last_fullsized":"2024-09-20T22:50:33.242206+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:49:52.301348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:49:52.301348+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:51:13.649456+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"74'10","reported_seq":320,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-20T22:50:33.240141+0000","last_change":"2024-09-20T22:48:05.262827+0000","last_active":"2024-09-20T22:50:33.240141+0000","last_peered":"2024-09-20T22:50:33.240141+0000","last_clean":"2024-09-20T22:50:33.240141+0000","last_became_active":"2024-09-20T22:48:05.242811+0000","last_became_peered":"2024-09-20T22:48:05.242811+0000","last_unstale":"2024-09-20T22:50:33.240141+0000","last_undegraded":"2024-09-20T22:50:33.240141+0000","last_fullsized":"2024-09-20T22:50:33.240141+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-09-20T22:47:21.092309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:47:21.092309+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:33.960771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":1032,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-20T22:50:33.249830+0000","last_change":"2024-09-20T22:50:13.205821+0000","last_active":"2024-09-20T22:50:33.249830+0000","last_peered":"2024-09-20T22:50:33.249830+0000","last_clean":"2024-09-20T22:50:33.249830+0000","last_became_active":"2024-09-20T22:48:05.233999+0000","last_became_peered":"2024-09-20T22:48:05.233999+0000","last_unstale":"2024-09-20T22:50:33.249830+0000","last_undegraded":"2024-09-20T22:50:33.249830+0000","last_fullsized":"2024-09-20T22:50:33.249830+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:50:13.205743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:50:13.205743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:51:41.953774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"261'9","reported_seq":1132,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-20T22:50:33.239054+0000","last_change":"2024-09-20T22:49:57.205107+0000","last_active":"2024-09-20T22:50:33.239054+0000","last_peered":"2024-09-20T22:50:33.239054+0000","last_clean":"2024-09-20T22:50:33.239054+0000","last_became_active":"2024-09-20T22:41:42.185863+0000","last_became_peered":"2024-09-20T22:41:42.185863+0000","last_unstale":"2024-09-20T22:50:33.239054+0000","last_undegraded":"2024-09-20T22:50:33.239054+0000","last_fullsized":"2024-09-20T22:50:33.239054+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'9","last_scrub_stamp":"2024-09-20T22:49:57.204987+0000","last_deep_scrub":"261'9","last_deep_scrub_stamp":"2024-09-20T22:46:50.974735+0000","last_clean_scrub_stamp":"2024-09-20T22:49:57.204987+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:51:09.810048+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":316,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-20T22:50:33.240909+0000","last_change":"2024-09-20T22:49:54.058299+0000","last_active":"2024-09-20T22:50:33.240909+0000","last_peered":"2024-09-20T22:50:33.240909+0000","last_clean":"2024-09-20T22:50:33.240909+0000","last_became_active":"2024-09-20T22:48:05.250284+0000","last_became_peered":"2024-09-20T22:48:05.250284+0000","last_unstale":"2024-09-20T22:50:33.240909+0000","last_undegraded":"2024-09-20T22:50:33.240909+0000","last_fullsized":"2024-09-20T22:50:33.240909+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:54.058184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:49:54.058184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:51:00.325103+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.8","version":"553'481","reported_seq":2675,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-20T22:50:33.244949+0000","last_change":"2024-09-20T22:50:33.245214+0000","last_active":"2024-09-20T22:50:33.244949+0000","last_peered":"2024-09-20T22:50:33.244949+0000","last_clean":"2024-09-20T22:50:33.244949+0000","last_became_active":"2024-09-20T22:48:31.224359+0000","last_became_peered":"2024-09-20T22:48:31.224359+0000","last_unstale":"2024-09-20T22:50:33.244949+0000","last_undegraded":"2024-09-20T22:50:33.244949+0000","last_fullsized":"2024-09-20T22:50:33.244949+0000","mapping_epoch":407,"log_start":"426'268","ondisk_log_start":"426'268","created":94,"last_epoch_clean":431,"parent":"0.0","parent_split_bits":0,"last_scrub":"534'447","last_scrub_stamp":"2024-09-20T22:50:14.324477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:50:14.324477+0000","objects_scrubbed":15,"log_size":213,"log_dups_size":445,"ondisk_log_size":213,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:51:18.505864+0000","scrub_duration":4,"objects_trimmed":1,"snaptrim_duration":0.005922035,"stat_sum":{"num_bytes":34542783,"num_objects":15,"num_object_clones":12,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":200,"num_read_kb":189773,"num_write":164,"num_write_kb":54925,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":26027409,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"ba","length":"1"},{"start":"ca","length":"1"},{"start":"d8","length":"1"},{"start":"e0","length":"1"},{"start":"e3","length":"1"},{"start":"e5","length":"1"},{"start":"e9","length":"1"},{"start":"eb","length":"1"},{"start":"ee","length":"1"}]},{"pgid":"4.b","version":"74'9","reported_seq":374,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-20T22:50:33.241300+0000","last_change":"2024-09-20T22:48:05.256987+0000","last_active":"2024-09-20T22:50:33.241300+0000","last_peered":"2024-09-20T22:50:33.241300+0000","last_clean":"2024-09-20T22:50:33.241300+0000","last_became_active":"2024-09-20T22:48:05.233267+0000","last_became_peered":"2024-09-20T22:48:05.233267+0000","last_unstale":"2024-09-20T22:50:33.241300+0000","last_undegraded":"2024-09-20T22:50:33.241300+0000","last_fullsized":"2024-09-20T22:50:33.241300+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-09-20T22:48:03.584242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:48:03.584242+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":1008,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-20T22:50:33.431915+0000","last_change":"2024-09-20T22:49:05.638528+0000","last_active":"2024-09-20T22:50:33.431915+0000","last_peered":"2024-09-20T22:50:33.431915+0000","last_clean":"2024-09-20T22:50:33.431915+0000","last_became_active":"2024-09-20T22:41:38.155953+0000","last_became_peered":"2024-09-20T22:41:38.155953+0000","last_unstale":"2024-09-20T22:50:33.431915+0000","last_undegraded":"2024-09-20T22:50:33.431915+0000","last_fullsized":"2024-09-20T22:50:33.431915+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:05.638457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:49:05.638457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:37.616640+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":1032,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-20T22:50:33.239701+0000","last_change":"2024-09-20T22:49:02.242848+0000","last_active":"2024-09-20T22:50:33.239701+0000","last_peered":"2024-09-20T22:50:33.239701+0000","last_clean":"2024-09-20T22:50:33.239701+0000","last_became_active":"2024-09-20T22:48:05.240613+0000","last_became_peered":"2024-09-20T22:48:05.240613+0000","last_unstale":"2024-09-20T22:50:33.239701+0000","last_undegraded":"2024-09-20T22:50:33.239701+0000","last_fullsized":"2024-09-20T22:50:33.239701+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:02.242757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:49:02.242757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":348,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-20T22:50:33.240856+0000","last_change":"2024-09-20T22:49:31.666061+0000","last_active":"2024-09-20T22:50:33.240856+0000","last_peered":"2024-09-20T22:50:33.240856+0000","last_clean":"2024-09-20T22:50:33.240856+0000","last_became_active":"2024-09-20T22:47:49.219486+0000","last_became_peered":"2024-09-20T22:47:49.219486+0000","last_unstale":"2024-09-20T22:50:33.240856+0000","last_undegraded":"2024-09-20T22:50:33.240856+0000","last_fullsized":"2024-09-20T22:50:33.240856+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:31.665954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:49:31.665954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:38.433613+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"553'691","reported_seq":3035,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-20T22:50:33.247979+0000","last_change":"2024-09-20T22:50:33.248177+0000","last_active":"2024-09-20T22:50:33.247979+0000","last_peered":"2024-09-20T22:50:33.247979+0000","last_clean":"2024-09-20T22:50:33.247979+0000","last_became_active":"2024-09-20T22:48:27.272202+0000","last_became_peered":"2024-09-20T22:48:27.272202+0000","last_unstale":"2024-09-20T22:50:33.247979+0000","last_undegraded":"2024-09-20T22:50:33.247979+0000","last_fullsized":"2024-09-20T22:50:33.247979+0000","mapping_epoch":407,"log_start":"423'280","ondisk_log_start":"423'280","created":94,"last_epoch_clean":427,"parent":"0.0","parent_split_bits":0,"last_scrub":"423'271","last_scrub_stamp":"2024-09-20T22:48:23.382092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:48:23.382092+0000","objects_scrubbed":8,"log_size":411,"log_dups_size":649,"ondisk_log_size":411,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0069375119999999998,"stat_sum":{"num_bytes":53469789,"num_objects":26,"num_object_clones":21,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":2,"num_read":324,"num_read_kb":328515,"num_write":250,"num_write_kb":92105,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":65,"num_bytes_recovered":38710860,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"ba","length":"1"},{"start":"ca","length":"1"},{"start":"d8","length":"1"},{"start":"e0","length":"1"},{"start":"e3","length":"1"},{"start":"e5","length":"1"},{"start":"e9","length":"1"},{"start":"eb","length":"1"},{"start":"ee","length":"1"}]},{"pgid":"4.a","version":"113'28","reported_seq":1052,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-20T22:50:33.430881+0000","last_change":"2024-09-20T22:49:03.201199+0000","last_active":"2024-09-20T22:50:33.430881+0000","last_peered":"2024-09-20T22:50:33.430881+0000","last_clean":"2024-09-20T22:50:33.430881+0000","last_became_active":"2024-09-20T22:41:40.252011+0000","last_became_peered":"2024-09-20T22:41:40.252011+0000","last_unstale":"2024-09-20T22:50:33.430881+0000","last_undegraded":"2024-09-20T22:50:33.430881+0000","last_fullsized":"2024-09-20T22:50:33.430881+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'28","last_scrub_stamp":"2024-09-20T22:49:03.201108+0000","last_deep_scrub":"113'28","last_deep_scrub_stamp":"2024-09-20T22:43:12.725900+0000","last_clean_scrub_stamp":"2024-09-20T22:49:03.201108+0000","objects_scrubbed":9,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":1007,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-20T22:50:33.244021+0000","last_change":"2024-09-20T22:49:12.324218+0000","last_active":"2024-09-20T22:50:33.244021+0000","last_peered":"2024-09-20T22:50:33.244021+0000","last_clean":"2024-09-20T22:50:33.244021+0000","last_became_active":"2024-09-20T22:41:38.161524+0000","last_became_peered":"2024-09-20T22:41:38.161524+0000","last_unstale":"2024-09-20T22:50:33.244021+0000","last_undegraded":"2024-09-20T22:50:33.244021+0000","last_fullsized":"2024-09-20T22:50:33.244021+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:12.324148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:49:12.324148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:36.961267+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":1027,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-20T22:50:33.239507+0000","last_change":"2024-09-20T22:50:15.301586+0000","last_active":"2024-09-20T22:50:33.239507+0000","last_peered":"2024-09-20T22:50:33.239507+0000","last_clean":"2024-09-20T22:50:33.239507+0000","last_became_active":"2024-09-20T22:47:49.223220+0000","last_became_peered":"2024-09-20T22:47:49.223220+0000","last_unstale":"2024-09-20T22:50:33.239507+0000","last_undegraded":"2024-09-20T22:50:33.239507+0000","last_fullsized":"2024-09-20T22:50:33.239507+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:50:15.301506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:50:15.301506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:51:44.987795+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":1004,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-20T22:50:33.243291+0000","last_change":"2024-09-20T22:49:30.310175+0000","last_active":"2024-09-20T22:50:33.243291+0000","last_peered":"2024-09-20T22:50:33.243291+0000","last_clean":"2024-09-20T22:50:33.243291+0000","last_became_active":"2024-09-20T22:41:44.199395+0000","last_became_peered":"2024-09-20T22:41:44.199395+0000","last_unstale":"2024-09-20T22:50:33.243291+0000","last_undegraded":"2024-09-20T22:50:33.243291+0000","last_fullsized":"2024-09-20T22:50:33.243291+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:30.310069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:47:18.329209+0000","last_clean_scrub_stamp":"2024-09-20T22:49:30.310069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:44.675926+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.a","version":"551'551","reported_seq":2809,"reported_epoch":553,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T22:50:33.249803+0000","last_change":"2024-09-20T22:50:32.288711+0000","last_active":"2024-09-20T22:50:33.249803+0000","last_peered":"2024-09-20T22:50:33.249803+0000","last_clean":"2024-09-20T22:50:33.249803+0000","last_became_active":"2024-09-20T22:42:52.405085+0000","last_became_peered":"2024-09-20T22:42:52.405085+0000","last_unstale":"2024-09-20T22:50:33.249803+0000","last_undegraded":"2024-09-20T22:50:33.249803+0000","last_fullsized":"2024-09-20T22:50:33.249803+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"505'466","last_scrub_stamp":"2024-09-20T22:49:47.088165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:49:47.088165+0000","objects_scrubbed":13,"log_size":551,"log_dups_size":0,"ondisk_log_size":551,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":7,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:50.870666+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.057859183000000002,"stat_sum":{"num_bytes":37146998,"num_objects":16,"num_object_clones":13,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":298,"num_read_kb":332023,"num_write":195,"num_write_kb":72715,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"ba","length":"1"},{"start":"e0","length":"1"}]},{"pgid":"4.9","version":"74'12","reported_seq":320,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-20T22:50:33.239349+0000","last_change":"2024-09-20T22:48:05.284523+0000","last_active":"2024-09-20T22:50:33.239349+0000","last_peered":"2024-09-20T22:50:33.239349+0000","last_clean":"2024-09-20T22:50:33.239349+0000","last_became_active":"2024-09-20T22:48:05.249438+0000","last_became_peered":"2024-09-20T22:48:05.249438+0000","last_unstale":"2024-09-20T22:50:33.239349+0000","last_undegraded":"2024-09-20T22:50:33.239349+0000","last_fullsized":"2024-09-20T22:50:33.239349+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-09-20T22:47:58.421473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:47:58.421473+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:33.960860+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":21,"num_read_kb":12,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":1039,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-20T22:50:33.243501+0000","last_change":"2024-09-20T22:49:24.300511+0000","last_active":"2024-09-20T22:50:33.243501+0000","last_peered":"2024-09-20T22:50:33.243501+0000","last_clean":"2024-09-20T22:50:33.243501+0000","last_became_active":"2024-09-20T22:48:05.240225+0000","last_became_peered":"2024-09-20T22:48:05.240225+0000","last_unstale":"2024-09-20T22:50:33.243501+0000","last_undegraded":"2024-09-20T22:50:33.243501+0000","last_fullsized":"2024-09-20T22:50:33.243501+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:24.300441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:46:52.465940+0000","last_clean_scrub_stamp":"2024-09-20T22:49:24.300441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:46.323016+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":1028,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-20T22:50:33.238682+0000","last_change":"2024-09-20T22:49:54.216094+0000","last_active":"2024-09-20T22:50:33.238682+0000","last_peered":"2024-09-20T22:50:33.238682+0000","last_clean":"2024-09-20T22:50:33.238682+0000","last_became_active":"2024-09-20T22:47:49.222088+0000","last_became_peered":"2024-09-20T22:47:49.222088+0000","last_unstale":"2024-09-20T22:50:33.238682+0000","last_undegraded":"2024-09-20T22:50:33.238682+0000","last_fullsized":"2024-09-20T22:50:33.238682+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:54.216022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:46:01.828080+0000","last_clean_scrub_stamp":"2024-09-20T22:49:54.216022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:56.397469+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":988,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-20T22:50:33.249502+0000","last_change":"2024-09-20T22:47:38.019602+0000","last_active":"2024-09-20T22:50:33.249502+0000","last_peered":"2024-09-20T22:50:33.249502+0000","last_clean":"2024-09-20T22:50:33.249502+0000","last_became_active":"2024-09-20T22:41:44.192609+0000","last_became_peered":"2024-09-20T22:41:44.192609+0000","last_unstale":"2024-09-20T22:50:33.249502+0000","last_undegraded":"2024-09-20T22:50:33.249502+0000","last_fullsized":"2024-09-20T22:50:33.249502+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:47:38.019533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:55.170379+0000","last_clean_scrub_stamp":"2024-09-20T22:47:38.019533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:38.225694+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"551'806","reported_seq":3075,"reported_epoch":553,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T22:50:33.246043+0000","last_change":"2024-09-20T22:50:32.255134+0000","last_active":"2024-09-20T22:50:33.246043+0000","last_peered":"2024-09-20T22:50:33.246043+0000","last_clean":"2024-09-20T22:50:33.246043+0000","last_became_active":"2024-09-20T22:42:52.409642+0000","last_became_peered":"2024-09-20T22:42:52.409642+0000","last_unstale":"2024-09-20T22:50:33.246043+0000","last_undegraded":"2024-09-20T22:50:33.246043+0000","last_fullsized":"2024-09-20T22:50:33.246043+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"194'116","last_scrub_stamp":"2024-09-20T22:44:37.097674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:44:37.097674+0000","objects_scrubbed":3,"log_size":806,"log_dups_size":0,"ondisk_log_size":806,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.96545217900000002,"stat_sum":{"num_bytes":47935563,"num_objects":20,"num_object_clones":16,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":1,"num_read":406,"num_read_kb":450577,"num_write":288,"num_write_kb":91687,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"ba","length":"1"},{"start":"e0","length":"1"},{"start":"e3","length":"1"},{"start":"e9","length":"1"},{"start":"ee","length":"1"}]},{"pgid":"4.8","version":"74'15","reported_seq":1030,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-20T22:50:33.430558+0000","last_change":"2024-09-20T22:49:47.687063+0000","last_active":"2024-09-20T22:50:33.430558+0000","last_peered":"2024-09-20T22:50:33.430558+0000","last_clean":"2024-09-20T22:50:33.430558+0000","last_became_active":"2024-09-20T22:41:40.252113+0000","last_became_peered":"2024-09-20T22:41:40.252113+0000","last_unstale":"2024-09-20T22:50:33.430558+0000","last_undegraded":"2024-09-20T22:50:33.430558+0000","last_fullsized":"2024-09-20T22:50:33.430558+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-09-20T22:49:47.686984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:49:47.686984+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:51:06.196255+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'2","reported_seq":1067,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-20T22:50:33.244430+0000","last_change":"2024-09-20T22:49:22.379093+0000","last_active":"2024-09-20T22:50:33.244430+0000","last_peered":"2024-09-20T22:50:33.244430+0000","last_clean":"2024-09-20T22:50:33.244430+0000","last_became_active":"2024-09-20T22:48:05.238098+0000","last_became_peered":"2024-09-20T22:48:05.238098+0000","last_unstale":"2024-09-20T22:50:33.244430+0000","last_undegraded":"2024-09-20T22:50:33.244430+0000","last_fullsized":"2024-09-20T22:50:33.244430+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-09-20T22:49:22.379012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:49:22.379012+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"408'10","reported_seq":1144,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-20T22:50:33.242581+0000","last_change":"2024-09-20T22:49:47.363588+0000","last_active":"2024-09-20T22:50:33.242581+0000","last_peered":"2024-09-20T22:50:33.242581+0000","last_clean":"2024-09-20T22:50:33.242581+0000","last_became_active":"2024-09-20T22:48:05.239344+0000","last_became_peered":"2024-09-20T22:48:05.239344+0000","last_unstale":"2024-09-20T22:50:33.242581+0000","last_undegraded":"2024-09-20T22:50:33.242581+0000","last_fullsized":"2024-09-20T22:50:33.242581+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"408'10","last_scrub_stamp":"2024-09-20T22:49:47.363515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:49:47.363515+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:51:01.473177+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":1009,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-20T22:50:33.431270+0000","last_change":"2024-09-20T22:49:50.693099+0000","last_active":"2024-09-20T22:50:33.431270+0000","last_peered":"2024-09-20T22:50:33.431270+0000","last_clean":"2024-09-20T22:50:33.431270+0000","last_became_active":"2024-09-20T22:47:49.227569+0000","last_became_peered":"2024-09-20T22:47:49.227569+0000","last_unstale":"2024-09-20T22:50:33.431270+0000","last_undegraded":"2024-09-20T22:50:33.431270+0000","last_fullsized":"2024-09-20T22:50:33.431270+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:50.693003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:49:50.693003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:51:08.138542+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":1027,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-20T22:50:33.430521+0000","last_change":"2024-09-20T22:49:02.203750+0000","last_active":"2024-09-20T22:50:33.430521+0000","last_peered":"2024-09-20T22:50:33.430521+0000","last_clean":"2024-09-20T22:50:33.430521+0000","last_became_active":"2024-09-20T22:47:49.213292+0000","last_became_peered":"2024-09-20T22:47:49.213292+0000","last_unstale":"2024-09-20T22:50:33.430521+0000","last_undegraded":"2024-09-20T22:50:33.430521+0000","last_fullsized":"2024-09-20T22:50:33.430521+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:02.203661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:49:02.203661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:35.887813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"74'6","reported_seq":1027,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-20T22:50:33.249756+0000","last_change":"2024-09-20T22:49:48.113810+0000","last_active":"2024-09-20T22:50:33.249756+0000","last_peered":"2024-09-20T22:50:33.249756+0000","last_clean":"2024-09-20T22:50:33.249756+0000","last_became_active":"2024-09-20T22:43:09.116887+0000","last_became_peered":"2024-09-20T22:43:09.116887+0000","last_unstale":"2024-09-20T22:50:33.249756+0000","last_undegraded":"2024-09-20T22:50:33.249756+0000","last_fullsized":"2024-09-20T22:50:33.249756+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'6","last_scrub_stamp":"2024-09-20T22:49:48.113740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:49:48.113740+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:51:07.964989+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":1008,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-20T22:50:33.432323+0000","last_change":"2024-09-20T22:50:22.597874+0000","last_active":"2024-09-20T22:50:33.432323+0000","last_peered":"2024-09-20T22:50:33.432323+0000","last_clean":"2024-09-20T22:50:33.432323+0000","last_became_active":"2024-09-20T22:41:42.184930+0000","last_became_peered":"2024-09-20T22:41:42.184930+0000","last_unstale":"2024-09-20T22:50:33.432323+0000","last_undegraded":"2024-09-20T22:50:33.432323+0000","last_fullsized":"2024-09-20T22:50:33.432323+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:50:22.597796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:09.705632+0000","last_clean_scrub_stamp":"2024-09-20T22:50:22.597796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:51:31.691020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"74'1","reported_seq":1016,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-20T22:50:33.243516+0000","last_change":"2024-09-20T22:48:40.309005+0000","last_active":"2024-09-20T22:50:33.243516+0000","last_peered":"2024-09-20T22:50:33.243516+0000","last_clean":"2024-09-20T22:50:33.243516+0000","last_became_active":"2024-09-20T22:48:05.237882+0000","last_became_peered":"2024-09-20T22:48:05.237882+0000","last_unstale":"2024-09-20T22:50:33.243516+0000","last_undegraded":"2024-09-20T22:50:33.243516+0000","last_fullsized":"2024-09-20T22:50:33.243516+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-09-20T22:48:40.308879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:48:40.308879+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"74'9","reported_seq":358,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-20T22:50:33.242648+0000","last_change":"2024-09-20T22:49:11.739149+0000","last_active":"2024-09-20T22:50:33.242648+0000","last_peered":"2024-09-20T22:50:33.242648+0000","last_clean":"2024-09-20T22:50:33.242648+0000","last_became_active":"2024-09-20T22:47:49.231315+0000","last_became_peered":"2024-09-20T22:47:49.231315+0000","last_unstale":"2024-09-20T22:50:33.242648+0000","last_undegraded":"2024-09-20T22:50:33.242648+0000","last_fullsized":"2024-09-20T22:50:33.242648+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-09-20T22:49:11.739076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:49:11.739076+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":1031,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-20T22:50:33.241794+0000","last_change":"2024-09-20T22:49:25.302834+0000","last_active":"2024-09-20T22:50:33.241794+0000","last_peered":"2024-09-20T22:50:33.241794+0000","last_clean":"2024-09-20T22:50:33.241794+0000","last_became_active":"2024-09-20T22:48:05.238801+0000","last_became_peered":"2024-09-20T22:48:05.238801+0000","last_unstale":"2024-09-20T22:50:33.241794+0000","last_undegraded":"2024-09-20T22:50:33.241794+0000","last_fullsized":"2024-09-20T22:50:33.241794+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:25.302756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:49:25.302756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:36.382428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":1000,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-20T22:50:33.249555+0000","last_change":"2024-09-20T22:50:17.192882+0000","last_active":"2024-09-20T22:50:33.249555+0000","last_peered":"2024-09-20T22:50:33.249555+0000","last_clean":"2024-09-20T22:50:33.249555+0000","last_became_active":"2024-09-20T22:41:42.187706+0000","last_became_peered":"2024-09-20T22:41:42.187706+0000","last_unstale":"2024-09-20T22:50:33.249555+0000","last_undegraded":"2024-09-20T22:50:33.249555+0000","last_fullsized":"2024-09-20T22:50:33.249555+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:50:17.192813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:50:17.192813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:51:29.793692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":1035,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-20T22:50:33.238725+0000","last_change":"2024-09-20T22:50:16.268746+0000","last_active":"2024-09-20T22:50:33.238725+0000","last_peered":"2024-09-20T22:50:33.238725+0000","last_clean":"2024-09-20T22:50:33.238725+0000","last_became_active":"2024-09-20T22:48:05.239710+0000","last_became_peered":"2024-09-20T22:48:05.239710+0000","last_unstale":"2024-09-20T22:50:33.238725+0000","last_undegraded":"2024-09-20T22:50:33.238725+0000","last_fullsized":"2024-09-20T22:50:33.238725+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:50:16.268655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:50:16.268655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:51:30.755872+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"74'9","reported_seq":1020,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-20T22:50:33.432202+0000","last_change":"2024-09-20T22:49:09.704604+0000","last_active":"2024-09-20T22:50:33.432202+0000","last_peered":"2024-09-20T22:50:33.432202+0000","last_clean":"2024-09-20T22:50:33.432202+0000","last_became_active":"2024-09-20T22:41:40.252486+0000","last_became_peered":"2024-09-20T22:41:40.252486+0000","last_unstale":"2024-09-20T22:50:33.432202+0000","last_undegraded":"2024-09-20T22:50:33.432202+0000","last_fullsized":"2024-09-20T22:50:33.432202+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-09-20T22:49:09.704521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:49:09.704521+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:37.616608+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":348,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-20T22:50:33.240942+0000","last_change":"2024-09-20T22:48:19.379265+0000","last_active":"2024-09-20T22:50:33.240942+0000","last_peered":"2024-09-20T22:50:33.240942+0000","last_clean":"2024-09-20T22:50:33.240942+0000","last_became_active":"2024-09-20T22:47:49.229936+0000","last_became_peered":"2024-09-20T22:47:49.229936+0000","last_unstale":"2024-09-20T22:50:33.240942+0000","last_undegraded":"2024-09-20T22:50:33.240942+0000","last_fullsized":"2024-09-20T22:50:33.240942+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:48:19.379196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:48:19.379196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"74'8","reported_seq":1111,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-20T22:50:33.250341+0000","last_change":"2024-09-20T22:50:23.299894+0000","last_active":"2024-09-20T22:50:33.250341+0000","last_peered":"2024-09-20T22:50:33.250341+0000","last_clean":"2024-09-20T22:50:33.250341+0000","last_became_active":"2024-09-20T22:41:42.186352+0000","last_became_peered":"2024-09-20T22:41:42.186352+0000","last_unstale":"2024-09-20T22:50:33.250341+0000","last_undegraded":"2024-09-20T22:50:33.250341+0000","last_fullsized":"2024-09-20T22:50:33.250341+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-09-20T22:47:56.150562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:47:56.150562+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:38.225679+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":316,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-20T22:50:33.242442+0000","last_change":"2024-09-20T22:50:21.999590+0000","last_active":"2024-09-20T22:50:33.242442+0000","last_peered":"2024-09-20T22:50:33.242442+0000","last_clean":"2024-09-20T22:50:33.242442+0000","last_became_active":"2024-09-20T22:48:05.249656+0000","last_became_peered":"2024-09-20T22:48:05.249656+0000","last_unstale":"2024-09-20T22:50:33.242442+0000","last_undegraded":"2024-09-20T22:50:33.242442+0000","last_fullsized":"2024-09-20T22:50:33.242442+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:50:21.999521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:50:21.999521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:51:28.981855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"74'10","reported_seq":1019,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-20T22:50:33.250008+0000","last_change":"2024-09-20T22:50:15.241059+0000","last_active":"2024-09-20T22:50:33.250008+0000","last_peered":"2024-09-20T22:50:33.250008+0000","last_clean":"2024-09-20T22:50:33.250008+0000","last_became_active":"2024-09-20T22:41:40.253614+0000","last_became_peered":"2024-09-20T22:41:40.253614+0000","last_unstale":"2024-09-20T22:50:33.250008+0000","last_undegraded":"2024-09-20T22:50:33.250008+0000","last_fullsized":"2024-09-20T22:50:33.250008+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-09-20T22:50:15.240945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:50:15.240945+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:51:43.975969+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":1027,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-20T22:50:33.243292+0000","last_change":"2024-09-20T22:49:54.381836+0000","last_active":"2024-09-20T22:50:33.243292+0000","last_peered":"2024-09-20T22:50:33.243292+0000","last_clean":"2024-09-20T22:50:33.243292+0000","last_became_active":"2024-09-20T22:48:05.240231+0000","last_became_peered":"2024-09-20T22:48:05.240231+0000","last_unstale":"2024-09-20T22:50:33.243292+0000","last_undegraded":"2024-09-20T22:50:33.243292+0000","last_fullsized":"2024-09-20T22:50:33.243292+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:54.381721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:49:54.381721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:51:04.253551+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"389'10","reported_seq":1147,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-20T22:50:33.432105+0000","last_change":"2024-09-20T22:49:51.715574+0000","last_active":"2024-09-20T22:50:33.432105+0000","last_peered":"2024-09-20T22:50:33.432105+0000","last_clean":"2024-09-20T22:50:33.432105+0000","last_became_active":"2024-09-20T22:47:49.227113+0000","last_became_peered":"2024-09-20T22:47:49.227113+0000","last_unstale":"2024-09-20T22:50:33.432105+0000","last_undegraded":"2024-09-20T22:50:33.432105+0000","last_fullsized":"2024-09-20T22:50:33.432105+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'10","last_scrub_stamp":"2024-09-20T22:49:51.715471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:49:51.715471+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:51:04.561910+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":348,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-20T22:50:33.242066+0000","last_change":"2024-09-20T22:50:21.853169+0000","last_active":"2024-09-20T22:50:33.242066+0000","last_peered":"2024-09-20T22:50:33.242066+0000","last_clean":"2024-09-20T22:50:33.242066+0000","last_became_active":"2024-09-20T22:47:49.230674+0000","last_became_peered":"2024-09-20T22:47:49.230674+0000","last_unstale":"2024-09-20T22:50:33.242066+0000","last_undegraded":"2024-09-20T22:50:33.242066+0000","last_fullsized":"2024-09-20T22:50:33.242066+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:50:21.853070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:50:21.853070+0000","last_clean_scrub_stamp":"2024-09-20T22:50:21.853070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:51:26.882477+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"74'11","reported_seq":328,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-20T22:50:33.239591+0000","last_change":"2024-09-20T22:48:44.023878+0000","last_active":"2024-09-20T22:50:33.239591+0000","last_peered":"2024-09-20T22:50:33.239591+0000","last_clean":"2024-09-20T22:50:33.239591+0000","last_became_active":"2024-09-20T22:48:05.249510+0000","last_became_peered":"2024-09-20T22:48:05.249510+0000","last_unstale":"2024-09-20T22:50:33.239591+0000","last_undegraded":"2024-09-20T22:50:33.239591+0000","last_fullsized":"2024-09-20T22:50:33.239591+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-09-20T22:48:44.023781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:48:44.023781+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:33.960967+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":316,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-20T22:50:33.239939+0000","last_change":"2024-09-20T22:49:11.015881+0000","last_active":"2024-09-20T22:50:33.239939+0000","last_peered":"2024-09-20T22:50:33.239939+0000","last_clean":"2024-09-20T22:50:33.239939+0000","last_became_active":"2024-09-20T22:48:05.252026+0000","last_became_peered":"2024-09-20T22:48:05.252026+0000","last_unstale":"2024-09-20T22:50:33.239939+0000","last_undegraded":"2024-09-20T22:50:33.239939+0000","last_fullsized":"2024-09-20T22:50:33.239939+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:11.015797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:49:11.015797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:39.578934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":1001,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-20T22:50:33.245012+0000","last_change":"2024-09-20T22:48:59.241704+0000","last_active":"2024-09-20T22:50:33.245012+0000","last_peered":"2024-09-20T22:50:33.245012+0000","last_clean":"2024-09-20T22:50:33.245012+0000","last_became_active":"2024-09-20T22:41:42.181736+0000","last_became_peered":"2024-09-20T22:41:42.181736+0000","last_unstale":"2024-09-20T22:50:33.245012+0000","last_undegraded":"2024-09-20T22:50:33.245012+0000","last_fullsized":"2024-09-20T22:50:33.245012+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:48:59.241606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:48:59.241606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":1008,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-20T22:50:33.248645+0000","last_change":"2024-09-20T22:50:20.207052+0000","last_active":"2024-09-20T22:50:33.248645+0000","last_peered":"2024-09-20T22:50:33.248645+0000","last_clean":"2024-09-20T22:50:33.248645+0000","last_became_active":"2024-09-20T22:47:49.216141+0000","last_became_peered":"2024-09-20T22:47:49.216141+0000","last_unstale":"2024-09-20T22:50:33.248645+0000","last_undegraded":"2024-09-20T22:50:33.248645+0000","last_fullsized":"2024-09-20T22:50:33.248645+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:50:20.206984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:50:20.206984+0000","last_clean_scrub_stamp":"2024-09-20T22:50:20.206984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:51:34.290144+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"404'13","reported_seq":1033,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-20T22:50:33.245254+0000","last_change":"2024-09-20T22:49:25.147397+0000","last_active":"2024-09-20T22:50:33.245254+0000","last_peered":"2024-09-20T22:50:33.245254+0000","last_clean":"2024-09-20T22:50:33.245254+0000","last_became_active":"2024-09-20T22:41:40.164796+0000","last_became_peered":"2024-09-20T22:41:40.164796+0000","last_unstale":"2024-09-20T22:50:33.245254+0000","last_undegraded":"2024-09-20T22:50:33.245254+0000","last_fullsized":"2024-09-20T22:50:33.245254+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"404'13","last_scrub_stamp":"2024-09-20T22:49:25.147286+0000","last_deep_scrub":"404'13","last_deep_scrub_stamp":"2024-09-20T22:48:04.276282+0000","last_clean_scrub_stamp":"2024-09-20T22:49:25.147286+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:52.567714+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":1042,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-20T22:50:33.243860+0000","last_change":"2024-09-20T22:49:56.396435+0000","last_active":"2024-09-20T22:50:33.243860+0000","last_peered":"2024-09-20T22:50:33.243860+0000","last_clean":"2024-09-20T22:50:33.243860+0000","last_became_active":"2024-09-20T22:48:05.238812+0000","last_became_peered":"2024-09-20T22:48:05.238812+0000","last_unstale":"2024-09-20T22:50:33.243860+0000","last_undegraded":"2024-09-20T22:50:33.243860+0000","last_fullsized":"2024-09-20T22:50:33.243860+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:56.396347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:31.138067+0000","last_clean_scrub_stamp":"2024-09-20T22:49:56.396347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:51:17.230220+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":1004,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-20T22:50:33.249587+0000","last_change":"2024-09-20T22:48:38.200896+0000","last_active":"2024-09-20T22:50:33.249587+0000","last_peered":"2024-09-20T22:50:33.249587+0000","last_clean":"2024-09-20T22:50:33.249587+0000","last_became_active":"2024-09-20T22:47:49.218708+0000","last_became_peered":"2024-09-20T22:47:49.218708+0000","last_unstale":"2024-09-20T22:50:33.249587+0000","last_undegraded":"2024-09-20T22:50:33.249587+0000","last_fullsized":"2024-09-20T22:50:33.249587+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:48:38.200797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:48:38.200797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:38.225738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":340,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-20T22:50:33.241107+0000","last_change":"2024-09-20T22:47:49.225362+0000","last_active":"2024-09-20T22:50:33.241107+0000","last_peered":"2024-09-20T22:50:33.241107+0000","last_clean":"2024-09-20T22:50:33.241107+0000","last_became_active":"2024-09-20T22:47:49.225027+0000","last_became_peered":"2024-09-20T22:47:49.225027+0000","last_unstale":"2024-09-20T22:50:33.241107+0000","last_undegraded":"2024-09-20T22:50:33.241107+0000","last_fullsized":"2024-09-20T22:50:33.241107+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:47:37.309842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:47:37.309842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"470'16","reported_seq":818,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-20T22:50:33.430384+0000","last_change":"2024-09-20T22:49:10.659945+0000","last_active":"2024-09-20T22:50:33.430384+0000","last_peered":"2024-09-20T22:50:33.430384+0000","last_clean":"2024-09-20T22:50:33.430384+0000","last_became_active":"2024-09-20T22:44:09.170785+0000","last_became_peered":"2024-09-20T22:44:09.170785+0000","last_unstale":"2024-09-20T22:50:33.430384+0000","last_undegraded":"2024-09-20T22:50:33.430384+0000","last_fullsized":"2024-09-20T22:50:33.430384+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"470'16","last_scrub_stamp":"2024-09-20T22:49:10.659868+0000","last_deep_scrub":"470'16","last_deep_scrub_stamp":"2024-09-20T22:49:10.659868+0000","last_clean_scrub_stamp":"2024-09-20T22:49:10.659868+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:37.616571+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":1016,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-20T22:50:33.432382+0000","last_change":"2024-09-20T22:50:13.688267+0000","last_active":"2024-09-20T22:50:33.432382+0000","last_peered":"2024-09-20T22:50:33.432382+0000","last_clean":"2024-09-20T22:50:33.432382+0000","last_became_active":"2024-09-20T22:41:38.158701+0000","last_became_peered":"2024-09-20T22:41:38.158701+0000","last_unstale":"2024-09-20T22:50:33.432382+0000","last_undegraded":"2024-09-20T22:50:33.432382+0000","last_fullsized":"2024-09-20T22:50:33.432382+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:50:13.688169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:45:24.765414+0000","last_clean_scrub_stamp":"2024-09-20T22:50:13.688169+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:51:24.846169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":1023,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-20T22:50:33.242575+0000","last_change":"2024-09-20T22:49:11.373441+0000","last_active":"2024-09-20T22:50:33.242575+0000","last_peered":"2024-09-20T22:50:33.242575+0000","last_clean":"2024-09-20T22:50:33.242575+0000","last_became_active":"2024-09-20T22:48:05.238334+0000","last_became_peered":"2024-09-20T22:48:05.238334+0000","last_unstale":"2024-09-20T22:50:33.242575+0000","last_undegraded":"2024-09-20T22:50:33.242575+0000","last_fullsized":"2024-09-20T22:50:33.242575+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:11.373370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:50.174861+0000","last_clean_scrub_stamp":"2024-09-20T22:49:11.373370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":1000,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-20T22:50:33.249741+0000","last_change":"2024-09-20T22:49:04.034724+0000","last_active":"2024-09-20T22:50:33.249741+0000","last_peered":"2024-09-20T22:50:33.249741+0000","last_clean":"2024-09-20T22:50:33.249741+0000","last_became_active":"2024-09-20T22:47:49.214487+0000","last_became_peered":"2024-09-20T22:47:49.214487+0000","last_unstale":"2024-09-20T22:50:33.249741+0000","last_undegraded":"2024-09-20T22:50:33.249741+0000","last_fullsized":"2024-09-20T22:50:33.249741+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:04.034601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:49:04.034601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:38.225799+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"74'4","reported_seq":1002,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-20T22:50:33.248747+0000","last_change":"2024-09-20T22:48:23.236306+0000","last_active":"2024-09-20T22:50:33.248747+0000","last_peered":"2024-09-20T22:50:33.248747+0000","last_clean":"2024-09-20T22:50:33.248747+0000","last_became_active":"2024-09-20T22:41:40.166123+0000","last_became_peered":"2024-09-20T22:41:40.166123+0000","last_unstale":"2024-09-20T22:50:33.248747+0000","last_undegraded":"2024-09-20T22:50:33.248747+0000","last_fullsized":"2024-09-20T22:50:33.248747+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'4","last_scrub_stamp":"2024-09-20T22:48:23.236227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:48:23.236227+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:38.225710+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":348,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-20T22:50:33.242173+0000","last_change":"2024-09-20T22:48:18.432371+0000","last_active":"2024-09-20T22:50:33.242173+0000","last_peered":"2024-09-20T22:50:33.242173+0000","last_clean":"2024-09-20T22:50:33.242173+0000","last_became_active":"2024-09-20T22:47:49.231964+0000","last_became_peered":"2024-09-20T22:47:49.231964+0000","last_unstale":"2024-09-20T22:50:33.242173+0000","last_undegraded":"2024-09-20T22:50:33.242173+0000","last_fullsized":"2024-09-20T22:50:33.242173+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:48:18.432271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:48:18.432271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":1031,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-20T22:50:33.430584+0000","last_change":"2024-09-20T22:49:07.125306+0000","last_active":"2024-09-20T22:50:33.430584+0000","last_peered":"2024-09-20T22:50:33.430584+0000","last_clean":"2024-09-20T22:50:33.430584+0000","last_became_active":"2024-09-20T22:48:05.244659+0000","last_became_peered":"2024-09-20T22:48:05.244659+0000","last_unstale":"2024-09-20T22:50:33.430584+0000","last_undegraded":"2024-09-20T22:50:33.430584+0000","last_fullsized":"2024-09-20T22:50:33.430584+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:07.125231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:49:07.125231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:34.936309+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":1015,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-20T22:50:33.242782+0000","last_change":"2024-09-20T22:49:57.405922+0000","last_active":"2024-09-20T22:50:33.242782+0000","last_peered":"2024-09-20T22:50:33.242782+0000","last_clean":"2024-09-20T22:50:33.242782+0000","last_became_active":"2024-09-20T22:48:05.237765+0000","last_became_peered":"2024-09-20T22:48:05.237765+0000","last_unstale":"2024-09-20T22:50:33.242782+0000","last_undegraded":"2024-09-20T22:50:33.242782+0000","last_fullsized":"2024-09-20T22:50:33.242782+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:57.405785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:10.233487+0000","last_clean_scrub_stamp":"2024-09-20T22:49:57.405785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:51:00.718232+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":1009,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-20T22:50:33.245708+0000","last_change":"2024-09-20T22:49:05.205734+0000","last_active":"2024-09-20T22:50:33.245708+0000","last_peered":"2024-09-20T22:50:33.245708+0000","last_clean":"2024-09-20T22:50:33.245708+0000","last_became_active":"2024-09-20T22:48:05.249492+0000","last_became_peered":"2024-09-20T22:48:05.249492+0000","last_unstale":"2024-09-20T22:50:33.245708+0000","last_undegraded":"2024-09-20T22:50:33.245708+0000","last_fullsized":"2024-09-20T22:50:33.245708+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:05.205665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:49:05.205665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":1026,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-20T22:50:33.248501+0000","last_change":"2024-09-20T22:48:36.246135+0000","last_active":"2024-09-20T22:50:33.248501+0000","last_peered":"2024-09-20T22:50:33.248501+0000","last_clean":"2024-09-20T22:50:33.248501+0000","last_became_active":"2024-09-20T22:47:49.217064+0000","last_became_peered":"2024-09-20T22:47:49.217064+0000","last_unstale":"2024-09-20T22:50:33.248501+0000","last_undegraded":"2024-09-20T22:50:33.248501+0000","last_fullsized":"2024-09-20T22:50:33.248501+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:48:36.246069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:48:36.246069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:38.225724+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"74'11","reported_seq":1034,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-20T22:50:33.432602+0000","last_change":"2024-09-20T22:49:10.137213+0000","last_active":"2024-09-20T22:50:33.432602+0000","last_peered":"2024-09-20T22:50:33.432602+0000","last_clean":"2024-09-20T22:50:33.432602+0000","last_became_active":"2024-09-20T22:41:40.171256+0000","last_became_peered":"2024-09-20T22:41:40.171256+0000","last_unstale":"2024-09-20T22:50:33.432602+0000","last_undegraded":"2024-09-20T22:50:33.432602+0000","last_fullsized":"2024-09-20T22:50:33.432602+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-09-20T22:49:10.137128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:49:10.137128+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:34.936293+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":348,"reported_epoch":553,"state":"active+clean","last_fresh":"2024-09-20T22:50:33.242154+0000","last_change":"2024-09-20T22:49:23.660691+0000","last_active":"2024-09-20T22:50:33.242154+0000","last_peered":"2024-09-20T22:50:33.242154+0000","last_clean":"2024-09-20T22:50:33.242154+0000","last_became_active":"2024-09-20T22:47:49.220971+0000","last_became_peered":"2024-09-20T22:47:49.220971+0000","last_unstale":"2024-09-20T22:50:33.242154+0000","last_undegraded":"2024-09-20T22:50:33.242154+0000","last_fullsized":"2024-09-20T22:50:33.242154+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:23.660625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:51.667341+0000","last_clean_scrub_stamp":"2024-09-20T22:49:23.660625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:50:38.664121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":541291822,"num_objects":242,"num_object_clones":193,"num_object_copies":726,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":242,"num_whiteouts":12,"num_read":3692,"num_read_kb":3855030,"num_write":2646,"num_write_kb":971140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":613,"num_bytes_recovered":302968954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1095774208,"data_stored":1583855919,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5637,"ondisk_log_size":5637,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":82,"ondisk_log_size":82,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":682,"num_read_kb":439,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":238,"num_bytes_recovered":5126,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":452,"ondisk_log_size":452,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":1034,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":526,"num_read_kb":521,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":384,"ondisk_log_size":384,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070575,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":175196,"kb_used_data":117632,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":93569956,"statfs":{"total":95995035648,"available":95815634944,"internally_reserved":0,"allocated":120455168,"data_stored":159688671,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":58915274},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430325,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":205400,"kb_used_data":147836,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":93539752,"statfs":{"total":95995035648,"available":95784706048,"internally_reserved":0,"allocated":151384064,"data_stored":213633560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691962,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":135048,"kb_used_data":99692,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93610104,"statfs":{"total":95995035648,"available":95856746496,"internally_reserved":0,"allocated":102084608,"data_stored":151559392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986303,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":139996,"kb_used_data":100488,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93605156,"statfs":{"total":95995035648,"available":95851679744,"internally_reserved":0,"allocated":102899712,"data_stored":146834550,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182533,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":274888,"kb_used_data":217328,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":93470264,"statfs":{"total":95995035648,"available":95713550336,"internally_reserved":0,"allocated":222543872,"data_stored":309174691,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":58915271},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411466,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":164840,"kb_used_data":111440,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93580312,"statfs":{"total":95995035648,"available":95826239488,"internally_reserved":0,"allocated":114114560,"data_stored":168346537,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574991,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":207064,"kb_used_data":153668,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93538088,"statfs":{"total":95995035648,"available":95783002112,"internally_reserved":0,"allocated":157356032,"data_stored":226400506,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738516,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":212720,"kb_used_data":159320,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93532432,"statfs":{"total":95995035648,"available":95777210368,"internally_reserved":0,"allocated":163143680,"data_stored":234207735,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":158593024,"data_stored":231458515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":152776704,"data_stored":223650951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":109305856,"data_stored":165327500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":217391104,"data_stored":305834559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":98344960,"data_stored":141883901,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":96940032,"data_stored":148211581,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":146853888,"data_stored":210884564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":115568640,"data_stored":156604348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-20T22:50:34.833 INFO:tasks.thrashosds.thrasher:Setting 1.9 to [7, 3, 5] 2024-09-20T22:50:34.834 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '1.9', '7', '3', '5'] 2024-09-20T22:50:34.834 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd pg-upmap 1.9 7 3 5 2024-09-20T22:50:34.834 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:50:34.858 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:50:34.897 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:34.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:34.898 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:50:34.998 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:50:35.025 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:35.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:35.026 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:50:35.058 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:50:35.126 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:50:35.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:35.148 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:35.148 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:50:35.248 INFO:tasks.rados.rados.0.smithi084.stdout:2159: done (1 left) 2024-09-20T22:50:35.248 INFO:tasks.rados.rados.0.smithi084.stdout:2162: done (0 left) 2024-09-20T22:50:35.248 INFO:tasks.rados.rados.0.smithi084.stdout:2163: write oid 22 current snap is 240 2024-09-20T22:50:35.248 INFO:tasks.rados.rados.0.smithi084.stdout:2163: seq_num 755 ranges {689285=731942,2147552=541733} 2024-09-20T22:50:35.249 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:50:35.256 INFO:tasks.rados.rados.0.smithi084.stdout:2163: writing smithi08477203-22 from 689285 to 1421227 tid 1 2024-09-20T22:50:35.263 INFO:tasks.rados.rados.0.smithi084.stdout:2163: writing smithi08477203-22 from 2147552 to 2689285 tid 2 2024-09-20T22:50:35.264 INFO:tasks.rados.rados.0.smithi084.stdout:2164: copy_from oid 28 from oid 34 current snap is 240 2024-09-20T22:50:35.264 INFO:tasks.rados.rados.0.smithi084.stdout:2163: finishing write tid 1 to smithi08477203-22 2024-09-20T22:50:35.264 INFO:tasks.rados.rados.0.smithi084.stdout:2165: read oid 20 snap 238 2024-09-20T22:50:35.264 INFO:tasks.rados.rados.0.smithi084.stdout:2165: expect (ObjNum 470 snap 130 seq_num 470) 2024-09-20T22:50:35.264 INFO:tasks.rados.rados.0.smithi084.stdout:2166: write oid 43 current snap is 240 2024-09-20T22:50:35.264 INFO:tasks.rados.rados.0.smithi084.stdout:2166: seq_num 756 ranges {706092=407191,1806034=463586} 2024-09-20T22:50:35.269 INFO:tasks.rados.rados.0.smithi084.stdout:2166: writing smithi08477203-43 from 706092 to 1113283 tid 1 2024-09-20T22:50:35.276 INFO:tasks.rados.rados.0.smithi084.stdout:2166: writing smithi08477203-43 from 1806034 to 2269620 tid 2 2024-09-20T22:50:35.277 INFO:tasks.rados.rados.0.smithi084.stdout:2167: snap_create 2024-09-20T22:50:35.277 INFO:tasks.rados.rados.0.smithi084.stdout:2163: finishing write tid 2 to smithi08477203-22 2024-09-20T22:50:35.277 INFO:tasks.rados.rados.0.smithi084.stdout:2163: finishing write tid 3 to smithi08477203-22 2024-09-20T22:50:35.277 INFO:tasks.rados.rados.0.smithi084.stdout:2163: finishing write tid 4 to smithi08477203-22 2024-09-20T22:50:35.278 INFO:tasks.rados.rados.0.smithi084.stdout:2163: oid 22 updating version 0 to 728 2024-09-20T22:50:35.278 INFO:tasks.rados.rados.0.smithi084.stdout:2163: oid 22 version 728 is already newer than 727 2024-09-20T22:50:35.278 INFO:tasks.rados.rados.0.smithi084.stdout:2163: oid 22 updating version 728 to 729 2024-09-20T22:50:35.278 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 22 v 729 (ObjNum 755 snap 240 seq_num 755) dirty exists 2024-09-20T22:50:35.278 INFO:tasks.rados.rados.0.smithi084.stdout:2163: left oid 22 (ObjNum 755 snap 240 seq_num 755) 2024-09-20T22:50:35.278 INFO:tasks.rados.rados.0.smithi084.stdout:2165: expect (ObjNum 470 snap 130 seq_num 470) 2024-09-20T22:50:35.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:35.280 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:35.280 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:50:35.301 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:35 smithi084 ceph-mon[40024]: pgmap v732: 158 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 153 active+clean; 517 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 2.4 MiB/s rd, 4.3 MiB/s wr, 15 op/s 2024-09-20T22:50:35.301 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:35 smithi084 ceph-mon[40024]: from='client.26168 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T22:50:35.301 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:35 smithi084 ceph-mon[40024]: 7.7 scrub starts 2024-09-20T22:50:35.301 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:35 smithi084 ceph-mon[40024]: 7.7 scrub ok 2024-09-20T22:50:35.301 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:35 smithi084 ceph-mon[40024]: osdmap e554: 8 total, 8 up, 8 in 2024-09-20T22:50:35.301 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:35 smithi084 ceph-mon[40024]: 6.1c scrub starts 2024-09-20T22:50:35.301 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:35 smithi084 ceph-mon[40024]: 6.1c scrub ok 2024-09-20T22:50:35.301 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:35 smithi084 ceph-mon[40024]: 4.c scrub starts 2024-09-20T22:50:35.301 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:35 smithi084 ceph-mon[40024]: 4.c scrub ok 2024-09-20T22:50:35.302 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:35 smithi084 ceph-mon[29774]: pgmap v732: 158 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 153 active+clean; 517 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 2.4 MiB/s rd, 4.3 MiB/s wr, 15 op/s 2024-09-20T22:50:35.302 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:35 smithi084 ceph-mon[29774]: from='client.26168 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T22:50:35.302 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:35 smithi084 ceph-mon[29774]: 7.7 scrub starts 2024-09-20T22:50:35.302 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:35 smithi084 ceph-mon[29774]: 7.7 scrub ok 2024-09-20T22:50:35.302 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:35 smithi084 ceph-mon[29774]: osdmap e554: 8 total, 8 up, 8 in 2024-09-20T22:50:35.302 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:35 smithi084 ceph-mon[29774]: 6.1c scrub starts 2024-09-20T22:50:35.302 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:35 smithi084 ceph-mon[29774]: 6.1c scrub ok 2024-09-20T22:50:35.302 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:35 smithi084 ceph-mon[29774]: 4.c scrub starts 2024-09-20T22:50:35.302 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:35 smithi084 ceph-mon[29774]: 4.c scrub ok 2024-09-20T22:50:35.306 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-20T22:50:35.306 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:50:35.380 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:50:35.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:35.402 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:35.402 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:50:35.412 INFO:tasks.rados.rados.0.smithi084.stdout:2164: finishing copy_from racing read to smithi08477203-28 2024-09-20T22:50:35.412 INFO:tasks.rados.rados.0.smithi084.stdout:2166: finishing write tid 1 to smithi08477203-43 2024-09-20T22:50:35.413 INFO:tasks.rados.rados.0.smithi084.stdout:2166: finishing write tid 2 to smithi08477203-43 2024-09-20T22:50:35.413 INFO:tasks.rados.rados.0.smithi084.stdout:2166: finishing write tid 3 to smithi08477203-43 2024-09-20T22:50:35.413 INFO:tasks.rados.rados.0.smithi084.stdout:2166: finishing write tid 4 to smithi08477203-43 2024-09-20T22:50:35.413 INFO:tasks.rados.rados.0.smithi084.stdout:2166: oid 43 updating version 0 to 1384 2024-09-20T22:50:35.413 INFO:tasks.rados.rados.0.smithi084.stdout:2166: oid 43 updating version 1384 to 1385 2024-09-20T22:50:35.413 INFO:tasks.rados.rados.0.smithi084.stdout:2166: oid 43 updating version 1385 to 1386 2024-09-20T22:50:35.413 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 43 v 1386 (ObjNum 756 snap 240 seq_num 756) dirty exists 2024-09-20T22:50:35.413 INFO:tasks.rados.rados.0.smithi084.stdout:2166: left oid 43 (ObjNum 756 snap 240 seq_num 756) 2024-09-20T22:50:35.413 INFO:tasks.rados.rados.0.smithi084.stdout:2164: finishing copy_from to smithi08477203-28 2024-09-20T22:50:35.413 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 28 v 728 (ObjNum 751 snap 238 seq_num 751) dirty exists 2024-09-20T22:50:35.503 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:50:35.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:35 smithi154 ceph-mon[33264]: pgmap v732: 158 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 153 active+clean; 517 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 2.4 MiB/s rd, 4.3 MiB/s wr, 15 op/s 2024-09-20T22:50:35.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:35 smithi154 ceph-mon[33264]: from='client.26168 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T22:50:35.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:35 smithi154 ceph-mon[33264]: 7.7 scrub starts 2024-09-20T22:50:35.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:35 smithi154 ceph-mon[33264]: 7.7 scrub ok 2024-09-20T22:50:35.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:35 smithi154 ceph-mon[33264]: osdmap e554: 8 total, 8 up, 8 in 2024-09-20T22:50:35.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:35 smithi154 ceph-mon[33264]: 6.1c scrub starts 2024-09-20T22:50:35.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:35 smithi154 ceph-mon[33264]: 6.1c scrub ok 2024-09-20T22:50:35.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:35 smithi154 ceph-mon[33264]: 4.c scrub starts 2024-09-20T22:50:35.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:35 smithi154 ceph-mon[33264]: 4.c scrub ok 2024-09-20T22:50:35.526 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:35.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:35.527 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:50:35.628 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:50:35.651 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:35.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:35.652 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:50:35.752 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:50:35.775 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:35.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:35.776 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:50:35.877 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:50:35.899 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:35.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:35.901 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:50:35.921 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:50:35.922 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:50:35.938 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:50:35.938 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:50:35.954 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:50:35.954 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:50:35.971 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:50:35.971 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-20T22:50:35.972 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T22:50:35.991 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:50:35.991 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:50:36.002 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:50:36.008 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:50:36.008 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:50:36.024 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:50:36.024 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:50:36.026 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:36.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:36.027 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:50:36.041 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:50:36.041 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:50:36.128 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:50:36.149 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:36.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:36.150 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:50:36.170 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-20T22:50:36.251 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:50:36.265 INFO:tasks.rados.rados.0.smithi084.stdout:2163: done (4 left) 2024-09-20T22:50:36.265 INFO:tasks.rados.rados.0.smithi084.stdout:2164: done (3 left) 2024-09-20T22:50:36.265 INFO:tasks.rados.rados.0.smithi084.stdout:2165: done (2 left) 2024-09-20T22:50:36.265 INFO:tasks.rados.rados.0.smithi084.stdout:2166: done (1 left) 2024-09-20T22:50:36.265 INFO:tasks.rados.rados.0.smithi084.stdout:2167: done (0 left) 2024-09-20T22:50:36.266 INFO:tasks.rados.rados.0.smithi084.stdout:2168: rollback oid 16 current snap is 241 2024-09-20T22:50:36.266 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 16 to 230 2024-09-20T22:50:36.266 INFO:tasks.rados.rados.0.smithi084.stdout:2169: read oid 29 snap -1 2024-09-20T22:50:36.266 INFO:tasks.rados.rados.0.smithi084.stdout:2169: expect (ObjNum 440 snap 115 seq_num 440) 2024-09-20T22:50:36.266 INFO:tasks.rados.rados.0.smithi084.stdout:2170: write oid 26 current snap is 241 2024-09-20T22:50:36.266 INFO:tasks.rados.rados.0.smithi084.stdout:2170: seq_num 757 ranges {722899=482440,1748163=723597} 2024-09-20T22:50:36.273 INFO:tasks.rados.rados.0.smithi084.stdout:2170: writing smithi08477203-26 from 722899 to 1205339 tid 1 2024-09-20T22:50:36.280 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:36.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:36.281 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:50:36.283 INFO:tasks.rados.rados.0.smithi084.stdout:2170: writing smithi08477203-26 from 1748163 to 2471760 tid 2 2024-09-20T22:50:36.283 INFO:tasks.rados.rados.0.smithi084.stdout:2171: read oid 27 snap -1 2024-09-20T22:50:36.283 INFO:tasks.rados.rados.0.smithi084.stdout:2171: expect (ObjNum 714 snap 224 seq_num 714) 2024-09-20T22:50:36.283 INFO:tasks.rados.rados.0.smithi084.stdout:2172: snap_remove snap 229 2024-09-20T22:50:36.382 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:50:36.406 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:36.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:36.407 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:50:36.480 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:36 smithi084 ceph-mon[40024]: 7.d scrub starts 2024-09-20T22:50:36.480 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:36 smithi084 ceph-mon[40024]: 7.d scrub ok 2024-09-20T22:50:36.480 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:36 smithi084 ceph-mon[40024]: osdmap e555: 8 total, 8 up, 8 in 2024-09-20T22:50:36.481 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:36 smithi084 ceph-mon[40024]: 3.3 scrub starts 2024-09-20T22:50:36.481 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:36 smithi084 ceph-mon[40024]: 3.3 scrub ok 2024-09-20T22:50:36.481 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:36 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/1102868920' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:50:36.481 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:36 smithi084 ceph-mon[29774]: 7.d scrub starts 2024-09-20T22:50:36.481 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:36 smithi084 ceph-mon[29774]: 7.d scrub ok 2024-09-20T22:50:36.481 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:36 smithi084 ceph-mon[29774]: osdmap e555: 8 total, 8 up, 8 in 2024-09-20T22:50:36.481 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:36 smithi084 ceph-mon[29774]: 3.3 scrub starts 2024-09-20T22:50:36.481 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:36 smithi084 ceph-mon[29774]: 3.3 scrub ok 2024-09-20T22:50:36.481 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:36 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/1102868920' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:50:36.489 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-20T22:50:36.508 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:50:36.552 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:36.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:36.553 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:50:36.654 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:50:36.675 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:36.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:36.676 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:50:36.719 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:50:36.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:36 smithi154 ceph-mon[33264]: 7.d scrub starts 2024-09-20T22:50:36.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:36 smithi154 ceph-mon[33264]: 7.d scrub ok 2024-09-20T22:50:36.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:36 smithi154 ceph-mon[33264]: osdmap e555: 8 total, 8 up, 8 in 2024-09-20T22:50:36.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:36 smithi154 ceph-mon[33264]: 3.3 scrub starts 2024-09-20T22:50:36.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:36 smithi154 ceph-mon[33264]: 3.3 scrub ok 2024-09-20T22:50:36.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:36 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/1102868920' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:50:36.777 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:50:36.803 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:36.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:36.804 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:50:36.904 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:50:36.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:36.927 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:36.927 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:50:37.028 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:50:37.056 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:37.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:37.057 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:50:37.158 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:50:37.179 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:50:37.183 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:37.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:37.184 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:50:37.270 INFO:tasks.rados.rados.0.smithi084.stdout:2172: done (4 left) 2024-09-20T22:50:37.270 INFO:tasks.rados.rados.0.smithi084.stdout:2173: read oid 6 snap -1 2024-09-20T22:50:37.270 INFO:tasks.rados.rados.0.smithi084.stdout:2173: expect (ObjNum 754 snap 238 seq_num 754) 2024-09-20T22:50:37.270 INFO:tasks.rados.rados.0.smithi084.stdout:2174: write (excl) oid 44 current snap is 241 2024-09-20T22:50:37.270 INFO:tasks.rados.rados.0.smithi084.stdout:2174: seq_num 758 ranges {739706=557689,1806645=467255} 2024-09-20T22:50:37.278 INFO:tasks.rados.rados.0.smithi084.stdout:2174: writing smithi08477203-44 from 739706 to 1297395 tid 1 2024-09-20T22:50:37.284 INFO:tasks.rados.rados.0.smithi084.stdout:2174: writing smithi08477203-44 from 1806645 to 2273900 tid 2 2024-09-20T22:50:37.285 INFO:tasks.rados.rados.0.smithi084.stdout:2175: snap_create 2024-09-20T22:50:37.285 INFO:tasks.rados.rados.0.smithi084.stdout:2168: finishing rollback tid 0 to smithi08477203-16 2024-09-20T22:50:37.285 INFO:tasks.rados.rados.0.smithi084.stdout:2168: finishing rollback tid 1 to smithi08477203-16 2024-09-20T22:50:37.285 INFO:tasks.rados.rados.0.smithi084.stdout:2168: finishing rollback tid 2 to smithi08477203-16 2024-09-20T22:50:37.285 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 16 v 432 (ObjNum 701 snap 218 seq_num 701) dirty exists 2024-09-20T22:50:37.285 INFO:tasks.rados.rados.0.smithi084.stdout:2169: expect (ObjNum 440 snap 115 seq_num 440) 2024-09-20T22:50:37.286 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:50:37.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:37.308 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:37.308 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:50:37.357 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:37 smithi084 ceph-mon[40024]: pgmap v735: 158 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 153 active+clean; 517 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 2.4 MiB/s rd, 4.3 MiB/s wr, 15 op/s 2024-09-20T22:50:37.358 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:37 smithi084 ceph-mon[40024]: 6.10 scrub starts 2024-09-20T22:50:37.358 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:37 smithi084 ceph-mon[40024]: 6.10 scrub ok 2024-09-20T22:50:37.358 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:37 smithi084 ceph-mon[40024]: 4.1f scrub starts 2024-09-20T22:50:37.358 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:37 smithi084 ceph-mon[40024]: 4.1f scrub ok 2024-09-20T22:50:37.358 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:37 smithi084 ceph-mon[40024]: osdmap e556: 8 total, 8 up, 8 in 2024-09-20T22:50:37.358 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:37 smithi084 ceph-mon[29774]: pgmap v735: 158 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 153 active+clean; 517 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 2.4 MiB/s rd, 4.3 MiB/s wr, 15 op/s 2024-09-20T22:50:37.358 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:37 smithi084 ceph-mon[29774]: 6.10 scrub starts 2024-09-20T22:50:37.358 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:37 smithi084 ceph-mon[29774]: 6.10 scrub ok 2024-09-20T22:50:37.358 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:37 smithi084 ceph-mon[29774]: 4.1f scrub starts 2024-09-20T22:50:37.358 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:37 smithi084 ceph-mon[29774]: 4.1f scrub ok 2024-09-20T22:50:37.359 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:37 smithi084 ceph-mon[29774]: osdmap e556: 8 total, 8 up, 8 in 2024-09-20T22:50:37.409 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:50:37.434 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:37.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:37.435 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:50:37.536 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:50:37.554 INFO:tasks.rados.rados.0.smithi084.stdout:2170: finishing write tid 1 to smithi08477203-26 2024-09-20T22:50:37.554 INFO:tasks.rados.rados.0.smithi084.stdout:2170: finishing write tid 2 to smithi08477203-26 2024-09-20T22:50:37.554 INFO:tasks.rados.rados.0.smithi084.stdout:2170: finishing write tid 3 to smithi08477203-26 2024-09-20T22:50:37.554 INFO:tasks.rados.rados.0.smithi084.stdout:2170: finishing write tid 4 to smithi08477203-26 2024-09-20T22:50:37.554 INFO:tasks.rados.rados.0.smithi084.stdout:2170: oid 26 updating version 0 to 781 2024-09-20T22:50:37.554 INFO:tasks.rados.rados.0.smithi084.stdout:2170: oid 26 version 781 is already newer than 780 2024-09-20T22:50:37.554 INFO:tasks.rados.rados.0.smithi084.stdout:2170: oid 26 version 781 is already newer than 779 2024-09-20T22:50:37.554 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 26 v 781 (ObjNum 757 snap 241 seq_num 757) dirty exists 2024-09-20T22:50:37.554 INFO:tasks.rados.rados.0.smithi084.stdout:2170: left oid 26 (ObjNum 757 snap 241 seq_num 757) 2024-09-20T22:50:37.555 INFO:tasks.rados.rados.0.smithi084.stdout:2171: expect (ObjNum 714 snap 224 seq_num 714) 2024-09-20T22:50:37.561 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:37.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:37.562 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:50:37.615 INFO:tasks.rados.rados.0.smithi084.stdout:2173: expect (ObjNum 754 snap 238 seq_num 754) 2024-09-20T22:50:37.662 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:50:37.685 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-20T22:50:37.685 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2024-09-20T22:50:37.685 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2024-09-20T22:50:37.685 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-20T22:50:37.687 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:37.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:37.688 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:50:37.705 INFO:tasks.rados.rados.0.smithi084.stdout:2174: finishing write tid 1 to smithi08477203-44 2024-09-20T22:50:37.705 INFO:tasks.rados.rados.0.smithi084.stdout:2174: finishing write tid 2 to smithi08477203-44 2024-09-20T22:50:37.705 INFO:tasks.rados.rados.0.smithi084.stdout:2174: finishing write tid 3 to smithi08477203-44 2024-09-20T22:50:37.705 INFO:tasks.rados.rados.0.smithi084.stdout:2174: finishing write tid 4 to smithi08477203-44 2024-09-20T22:50:37.705 INFO:tasks.rados.rados.0.smithi084.stdout:2174: oid 44 updating version 0 to 1514 2024-09-20T22:50:37.705 INFO:tasks.rados.rados.0.smithi084.stdout:2174: oid 44 version 1514 is already newer than 1510 2024-09-20T22:50:37.705 INFO:tasks.rados.rados.0.smithi084.stdout:2174: oid 44 version 1514 is already newer than 1513 2024-09-20T22:50:37.706 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 44 v 1514 (ObjNum 758 snap 241 seq_num 758) dirty exists 2024-09-20T22:50:37.706 INFO:tasks.rados.rados.0.smithi084.stdout:2174: left oid 44 (ObjNum 758 snap 241 seq_num 758) 2024-09-20T22:50:37.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:37 smithi154 ceph-mon[33264]: pgmap v735: 158 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 153 active+clean; 517 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 2.4 MiB/s rd, 4.3 MiB/s wr, 15 op/s 2024-09-20T22:50:37.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:37 smithi154 ceph-mon[33264]: 6.10 scrub starts 2024-09-20T22:50:37.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:37 smithi154 ceph-mon[33264]: 6.10 scrub ok 2024-09-20T22:50:37.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:37 smithi154 ceph-mon[33264]: 4.1f scrub starts 2024-09-20T22:50:37.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:37 smithi154 ceph-mon[33264]: 4.1f scrub ok 2024-09-20T22:50:37.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:37 smithi154 ceph-mon[33264]: osdmap e556: 8 total, 8 up, 8 in 2024-09-20T22:50:37.789 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:50:37.810 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:37.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:37.811 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:50:37.912 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:50:37.933 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:37.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:37.934 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:50:38.035 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:50:38.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:38.057 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:38.057 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:50:38.158 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:50:38.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:38.181 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:38.181 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:50:38.241 INFO:teuthology.orchestra.run.smithi084.stderr:set 1.9 pg_upmap mapping to [7,3,5] 2024-09-20T22:50:38.247 INFO:tasks.rados.rados.0.smithi084.stdout:2168: done (6 left) 2024-09-20T22:50:38.247 INFO:tasks.rados.rados.0.smithi084.stdout:2169: done (5 left) 2024-09-20T22:50:38.247 INFO:tasks.rados.rados.0.smithi084.stdout:2170: done (4 left) 2024-09-20T22:50:38.247 INFO:tasks.rados.rados.0.smithi084.stdout:2171: done (3 left) 2024-09-20T22:50:38.247 INFO:tasks.rados.rados.0.smithi084.stdout:2173: done (2 left) 2024-09-20T22:50:38.247 INFO:tasks.rados.rados.0.smithi084.stdout:2174: done (1 left) 2024-09-20T22:50:38.247 INFO:tasks.rados.rados.0.smithi084.stdout:2175: done (0 left) 2024-09-20T22:50:38.249 INFO:tasks.rados.rados.0.smithi084.stdout:2176: snap_remove snap 147 2024-09-20T22:50:38.282 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:50:38.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:38.307 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:38.307 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:50:38.408 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:50:38.432 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:38.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:38.433 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:50:38.457 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:50:38 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: ::ffff:172.21.15.154 - - [20/Sep/2024:22:50:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-20T22:50:38.457 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:38 smithi084 ceph-mon[29774]: 4.b scrub starts 2024-09-20T22:50:38.457 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:38 smithi084 ceph-mon[29774]: 4.b scrub ok 2024-09-20T22:50:38.457 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:38 smithi084 ceph-mon[29774]: 5.11 scrub starts 2024-09-20T22:50:38.457 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:38 smithi084 ceph-mon[29774]: 5.11 scrub ok 2024-09-20T22:50:38.457 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:38 smithi084 ceph-mon[29774]: osdmap e557: 8 total, 8 up, 8 in 2024-09-20T22:50:38.457 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:38 smithi084 ceph-mon[29774]: 5.a scrub starts 2024-09-20T22:50:38.457 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:38 smithi084 ceph-mon[29774]: 5.a scrub ok 2024-09-20T22:50:38.458 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:38 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/1462956727' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "1.9", "id": [7, 3, 5]}]: dispatch 2024-09-20T22:50:38.458 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:38 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:50:38.458 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:38 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:50:38.458 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:38 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/1462956727' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "1.9", "id": [7, 3, 5]}]': finished 2024-09-20T22:50:38.458 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:38 smithi084 ceph-mon[29774]: osdmap e558: 8 total, 8 up, 8 in 2024-09-20T22:50:38.458 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:38 smithi084 ceph-mon[40024]: 4.b scrub starts 2024-09-20T22:50:38.458 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:38 smithi084 ceph-mon[40024]: 4.b scrub ok 2024-09-20T22:50:38.458 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:38 smithi084 ceph-mon[40024]: 5.11 scrub starts 2024-09-20T22:50:38.458 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:38 smithi084 ceph-mon[40024]: 5.11 scrub ok 2024-09-20T22:50:38.458 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:38 smithi084 ceph-mon[40024]: osdmap e557: 8 total, 8 up, 8 in 2024-09-20T22:50:38.458 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:38 smithi084 ceph-mon[40024]: 5.a scrub starts 2024-09-20T22:50:38.458 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:38 smithi084 ceph-mon[40024]: 5.a scrub ok 2024-09-20T22:50:38.459 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:38 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/1462956727' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "1.9", "id": [7, 3, 5]}]: dispatch 2024-09-20T22:50:38.459 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:38 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:50:38.459 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:38 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:50:38.459 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:38 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/1462956727' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "1.9", "id": [7, 3, 5]}]': finished 2024-09-20T22:50:38.459 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:38 smithi084 ceph-mon[40024]: osdmap e558: 8 total, 8 up, 8 in 2024-09-20T22:50:38.534 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:50:38.555 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:38.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:38.557 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:50:38.659 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:50:38.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:38.681 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:38.681 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:50:38.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:38 smithi154 ceph-mon[33264]: 4.b scrub starts 2024-09-20T22:50:38.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:38 smithi154 ceph-mon[33264]: 4.b scrub ok 2024-09-20T22:50:38.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:38 smithi154 ceph-mon[33264]: 5.11 scrub starts 2024-09-20T22:50:38.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:38 smithi154 ceph-mon[33264]: 5.11 scrub ok 2024-09-20T22:50:38.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:38 smithi154 ceph-mon[33264]: osdmap e557: 8 total, 8 up, 8 in 2024-09-20T22:50:38.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:38 smithi154 ceph-mon[33264]: 5.a scrub starts 2024-09-20T22:50:38.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:38 smithi154 ceph-mon[33264]: 5.a scrub ok 2024-09-20T22:50:38.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:38 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/1462956727' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "1.9", "id": [7, 3, 5]}]: dispatch 2024-09-20T22:50:38.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:38 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:50:38.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:38 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:50:38.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:38 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/1462956727' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "1.9", "id": [7, 3, 5]}]': finished 2024-09-20T22:50:38.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:38 smithi154 ceph-mon[33264]: osdmap e558: 8 total, 8 up, 8 in 2024-09-20T22:50:38.781 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:50:38.808 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:38.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:38.809 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:50:38.909 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:50:38.931 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:38.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:38.932 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:50:39.033 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:50:39.054 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:39.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:39.055 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:50:39.156 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:50:39.177 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:39.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:39.178 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:50:39.258 INFO:tasks.rados.rados.0.smithi084.stdout:2176: done (0 left) 2024-09-20T22:50:39.258 INFO:tasks.rados.rados.0.smithi084.stdout:2177: read oid 12 snap -1 2024-09-20T22:50:39.258 INFO:tasks.rados.rados.0.smithi084.stdout:2177: expect (ObjNum 741 snap 237 seq_num 741) 2024-09-20T22:50:39.259 INFO:tasks.rados.rados.0.smithi084.stdout:2178: snap_create 2024-09-20T22:50:39.274 INFO:tasks.rados.rados.0.smithi084.stdout:2177: expect (ObjNum 741 snap 237 seq_num 741) 2024-09-20T22:50:39.279 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:50:39.301 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:39.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:39.302 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:50:39.403 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:50:39.414 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:39 smithi084 ceph-mon[29774]: pgmap v738: 158 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 153 active+clean; 517 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-20T22:50:39.414 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:39 smithi084 ceph-mon[29774]: 5.14 scrub starts 2024-09-20T22:50:39.414 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:39 smithi084 ceph-mon[29774]: 5.14 scrub ok 2024-09-20T22:50:39.414 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:39 smithi084 ceph-mon[29774]: 4.e scrub starts 2024-09-20T22:50:39.414 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:39 smithi084 ceph-mon[29774]: 4.e scrub ok 2024-09-20T22:50:39.414 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:39 smithi084 ceph-mon[29774]: osdmap e559: 8 total, 8 up, 8 in 2024-09-20T22:50:39.415 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:39 smithi084 ceph-mon[40024]: pgmap v738: 158 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 153 active+clean; 517 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-20T22:50:39.415 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:39 smithi084 ceph-mon[40024]: 5.14 scrub starts 2024-09-20T22:50:39.415 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:39 smithi084 ceph-mon[40024]: 5.14 scrub ok 2024-09-20T22:50:39.415 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:39 smithi084 ceph-mon[40024]: 4.e scrub starts 2024-09-20T22:50:39.415 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:39 smithi084 ceph-mon[40024]: 4.e scrub ok 2024-09-20T22:50:39.415 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:39 smithi084 ceph-mon[40024]: osdmap e559: 8 total, 8 up, 8 in 2024-09-20T22:50:39.425 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:39.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:39.426 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:50:39.526 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:50:39.547 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:39.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:39.548 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:50:39.649 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:50:39.670 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:39.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:39.671 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:50:39.754 INFO:journalctl@ceph.iscsi.iscsi.a.smithi154.stdout:Sep 20 22:50:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-iscsi-iscsi-a[59730]: debug there is no tcmu-runner data available 2024-09-20T22:50:39.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:39 smithi154 ceph-mon[33264]: pgmap v738: 158 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 153 active+clean; 517 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-20T22:50:39.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:39 smithi154 ceph-mon[33264]: 5.14 scrub starts 2024-09-20T22:50:39.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:39 smithi154 ceph-mon[33264]: 5.14 scrub ok 2024-09-20T22:50:39.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:39 smithi154 ceph-mon[33264]: 4.e scrub starts 2024-09-20T22:50:39.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:39 smithi154 ceph-mon[33264]: 4.e scrub ok 2024-09-20T22:50:39.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:39 smithi154 ceph-mon[33264]: osdmap e559: 8 total, 8 up, 8 in 2024-09-20T22:50:39.771 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:50:39.795 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:39.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:39.796 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:50:39.897 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:50:39.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:39.922 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:39.923 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:50:40.023 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:50:40.047 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:40.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:40.048 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:50:40.148 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:50:40.170 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:40.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:40.171 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:50:40.256 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is unset 2024-09-20T22:50:40.257 INFO:tasks.rados.rados.0.smithi084.stdout:2177: done (1 left) 2024-09-20T22:50:40.258 INFO:tasks.rados.rados.0.smithi084.stdout:2178: done (0 left) 2024-09-20T22:50:40.258 INFO:tasks.rados.rados.0.smithi084.stdout:2179: write (excl) oid 48 current snap is 243 2024-09-20T22:50:40.258 INFO:tasks.rados.rados.0.smithi084.stdout:2179: seq_num 759 ranges {756513=632938,1865127=727266} 2024-09-20T22:50:40.265 INFO:tasks.rados.rados.0.smithi084.stdout:2179: writing smithi08477203-48 from 756513 to 1389451 tid 1 2024-09-20T22:50:40.272 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:50:40.273 INFO:tasks.rados.rados.0.smithi084.stdout:2179: writing smithi08477203-48 from 1865127 to 2592393 tid 2 2024-09-20T22:50:40.273 INFO:tasks.rados.rados.0.smithi084.stdout:2180: read oid 32 snap -1 2024-09-20T22:50:40.273 INFO:tasks.rados.rados.0.smithi084.stdout:2180: expect (ObjNum 749 snap 238 seq_num 749) 2024-09-20T22:50:40.273 INFO:tasks.rados.rados.0.smithi084.stdout:2181: copy_from oid 39 from oid 24 current snap is 243 2024-09-20T22:50:40.273 INFO:tasks.rados.rados.0.smithi084.stdout:2179: finishing write tid 1 to smithi08477203-48 2024-09-20T22:50:40.273 INFO:tasks.rados.rados.0.smithi084.stdout:2182: snap_remove snap 241 2024-09-20T22:50:40.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:40.302 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:40.302 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:50:40.373 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:40 smithi084 ceph-mon[40024]: 6.b scrub starts 2024-09-20T22:50:40.373 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:40 smithi084 ceph-mon[40024]: 6.b scrub ok 2024-09-20T22:50:40.373 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:40 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/3680783883' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:50:40.373 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:40 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:50:40.373 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:40 smithi084 ceph-mon[40024]: 2.1 scrub starts 2024-09-20T22:50:40.373 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:40 smithi084 ceph-mon[40024]: 2.1 scrub ok 2024-09-20T22:50:40.373 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:40 smithi084 ceph-mon[40024]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:50:40.373 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:40 smithi084 ceph-mon[40024]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T22:50:40.373 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:40 smithi084 ceph-mon[40024]: Cluster is now healthy 2024-09-20T22:50:40.373 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:40 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T22:50:40.373 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:40 smithi084 ceph-mon[40024]: osdmap e560: 8 total, 8 up, 8 in 2024-09-20T22:50:40.374 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:40 smithi084 ceph-mon[29774]: 6.b scrub starts 2024-09-20T22:50:40.374 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:40 smithi084 ceph-mon[29774]: 6.b scrub ok 2024-09-20T22:50:40.374 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:40 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/3680783883' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:50:40.374 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:40 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:50:40.374 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:40 smithi084 ceph-mon[29774]: 2.1 scrub starts 2024-09-20T22:50:40.374 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:40 smithi084 ceph-mon[29774]: 2.1 scrub ok 2024-09-20T22:50:40.374 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:40 smithi084 ceph-mon[29774]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:50:40.374 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:40 smithi084 ceph-mon[29774]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T22:50:40.374 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:40 smithi084 ceph-mon[29774]: Cluster is now healthy 2024-09-20T22:50:40.374 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:40 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T22:50:40.374 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:40 smithi084 ceph-mon[29774]: osdmap e560: 8 total, 8 up, 8 in 2024-09-20T22:50:40.375 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.1 dump_blocked_ops 2024-09-20T22:50:40.404 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:50:40.427 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:40.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:40.428 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:50:40.486 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-20T22:50:40.528 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:50:40.550 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:40.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:40.551 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:50:40.652 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:50:40.655 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:40 smithi154 ceph-mon[33264]: 6.b scrub starts 2024-09-20T22:50:40.655 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:40 smithi154 ceph-mon[33264]: 6.b scrub ok 2024-09-20T22:50:40.656 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:40 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/3680783883' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:50:40.656 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:40 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:50:40.656 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:40 smithi154 ceph-mon[33264]: 2.1 scrub starts 2024-09-20T22:50:40.656 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:40 smithi154 ceph-mon[33264]: 2.1 scrub ok 2024-09-20T22:50:40.656 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:40 smithi154 ceph-mon[33264]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:50:40.656 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:40 smithi154 ceph-mon[33264]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T22:50:40.656 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:40 smithi154 ceph-mon[33264]: Cluster is now healthy 2024-09-20T22:50:40.656 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:40 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T22:50:40.656 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:40 smithi154 ceph-mon[33264]: osdmap e560: 8 total, 8 up, 8 in 2024-09-20T22:50:40.675 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:40.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:40.676 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:50:40.777 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:50:40.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:40.805 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:40.805 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:50:40.905 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:50:40.927 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:40.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:40.928 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:50:41.015 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:50:41.029 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:50:41.050 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:41.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:41.051 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:50:41.152 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:50:41.173 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:41.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:41.175 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:50:41.258 INFO:tasks.rados.rados.0.smithi084.stdout:2182: done (3 left) 2024-09-20T22:50:41.259 INFO:tasks.rados.rados.0.smithi084.stdout:2183: rollback oid 37 current snap is 243 2024-09-20T22:50:41.259 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 37 to 240 2024-09-20T22:50:41.259 INFO:tasks.rados.rados.0.smithi084.stdout:2184: write oid 43 current snap is 243 2024-09-20T22:50:41.259 INFO:tasks.rados.rados.0.smithi084.stdout:2184: seq_num 760 ranges {773320=708187,1923609=470924} 2024-09-20T22:50:41.267 INFO:tasks.rados.rados.0.smithi084.stdout:2184: writing smithi08477203-43 from 773320 to 1481507 tid 1 2024-09-20T22:50:41.272 INFO:tasks.rados.rados.0.smithi084.stdout:2184: writing smithi08477203-43 from 1923609 to 2394533 tid 2 2024-09-20T22:50:41.272 INFO:tasks.rados.rados.0.smithi084.stdout:2185: write oid 46 current snap is 243 2024-09-20T22:50:41.272 INFO:tasks.rados.rados.0.smithi084.stdout:2185: seq_num 761 ranges {790127=499789,1698444=730935} 2024-09-20T22:50:41.275 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:50:41.279 INFO:tasks.rados.rados.0.smithi084.stdout:2185: writing smithi08477203-46 from 790127 to 1289916 tid 1 2024-09-20T22:50:41.286 INFO:tasks.rados.rados.0.smithi084.stdout:2185: writing smithi08477203-46 from 1698444 to 2429379 tid 2 2024-09-20T22:50:41.286 INFO:tasks.rados.rados.0.smithi084.stdout:2186: read oid 8 snap -1 2024-09-20T22:50:41.286 INFO:tasks.rados.rados.0.smithi084.stdout:2186: expect (ObjNum 753 snap 238 seq_num 753) 2024-09-20T22:50:41.286 INFO:tasks.rados.rados.0.smithi084.stdout:2187: rollback oid 49 current snap is 243 2024-09-20T22:50:41.286 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 49 to 209 2024-09-20T22:50:41.286 INFO:tasks.rados.rados.0.smithi084.stdout:2180: expect (ObjNum 749 snap 238 seq_num 749) 2024-09-20T22:50:41.297 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:41.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:41.298 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:50:41.399 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:50:41.420 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:41.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:41.421 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:50:41.452 INFO:tasks.rados.rados.0.smithi084.stdout:2179: finishing write tid 2 to smithi08477203-48 2024-09-20T22:50:41.452 INFO:tasks.rados.rados.0.smithi084.stdout:2179: finishing write tid 3 to smithi08477203-48 2024-09-20T22:50:41.452 INFO:tasks.rados.rados.0.smithi084.stdout:2179: finishing write tid 4 to smithi08477203-48 2024-09-20T22:50:41.452 INFO:tasks.rados.rados.0.smithi084.stdout:2179: oid 48 updating version 0 to 858 2024-09-20T22:50:41.452 INFO:tasks.rados.rados.0.smithi084.stdout:2179: oid 48 version 858 is already newer than 857 2024-09-20T22:50:41.452 INFO:tasks.rados.rados.0.smithi084.stdout:2179: oid 48 updating version 858 to 859 2024-09-20T22:50:41.453 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 48 v 859 (ObjNum 759 snap 243 seq_num 759) dirty exists 2024-09-20T22:50:41.453 INFO:tasks.rados.rados.0.smithi084.stdout:2179: left oid 48 (ObjNum 759 snap 243 seq_num 759) 2024-09-20T22:50:41.453 INFO:tasks.rados.rados.0.smithi084.stdout:2181: finishing copy_from racing read to smithi08477203-39 2024-09-20T22:50:41.453 INFO:tasks.rados.rados.0.smithi084.stdout:2181: finishing copy_from to smithi08477203-39 2024-09-20T22:50:41.453 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 39 v 729 (ObjNum 743 snap 237 seq_num 743) dirty exists 2024-09-20T22:50:41.453 INFO:tasks.rados.rados.0.smithi084.stdout:2183: finishing rollback tid 1 to smithi08477203-37 2024-09-20T22:50:41.453 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 37 v 842 (ObjNum 129 snap 0 seq_num 2091730624) dirty dne 2024-09-20T22:50:41.453 INFO:tasks.rados.rados.0.smithi084.stdout:2179: done (7 left) 2024-09-20T22:50:41.453 INFO:tasks.rados.rados.0.smithi084.stdout:2180: done (6 left) 2024-09-20T22:50:41.453 INFO:tasks.rados.rados.0.smithi084.stdout:2181: done (5 left) 2024-09-20T22:50:41.453 INFO:tasks.rados.rados.0.smithi084.stdout:2183: done (4 left) 2024-09-20T22:50:41.453 INFO:tasks.rados.rados.0.smithi084.stdout:2188: copy_from oid 40 from oid 11 current snap is 243 2024-09-20T22:50:41.453 INFO:tasks.rados.rados.0.smithi084.stdout:2184: finishing write tid 1 to smithi08477203-43 2024-09-20T22:50:41.454 INFO:tasks.rados.rados.0.smithi084.stdout:2184: finishing write tid 2 to smithi08477203-43 2024-09-20T22:50:41.454 INFO:tasks.rados.rados.0.smithi084.stdout:2184: finishing write tid 3 to smithi08477203-43 2024-09-20T22:50:41.454 INFO:tasks.rados.rados.0.smithi084.stdout:2184: finishing write tid 4 to smithi08477203-43 2024-09-20T22:50:41.454 INFO:tasks.rados.rados.0.smithi084.stdout:2184: oid 43 updating version 0 to 1387 2024-09-20T22:50:41.454 INFO:tasks.rados.rados.0.smithi084.stdout:2184: oid 43 updating version 1387 to 1389 2024-09-20T22:50:41.454 INFO:tasks.rados.rados.0.smithi084.stdout:2184: oid 43 version 1389 is already newer than 1388 2024-09-20T22:50:41.454 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 43 v 1389 (ObjNum 760 snap 243 seq_num 760) dirty exists 2024-09-20T22:50:41.454 INFO:tasks.rados.rados.0.smithi084.stdout:2184: left oid 43 (ObjNum 760 snap 243 seq_num 760) 2024-09-20T22:50:41.454 INFO:tasks.rados.rados.0.smithi084.stdout:2185: finishing write tid 1 to smithi08477203-46 2024-09-20T22:50:41.454 INFO:tasks.rados.rados.0.smithi084.stdout:2187: finishing rollback tid 0 to smithi08477203-49 2024-09-20T22:50:41.454 INFO:tasks.rados.rados.0.smithi084.stdout:2187: finishing rollback tid 1 to smithi08477203-49 2024-09-20T22:50:41.454 INFO:tasks.rados.rados.0.smithi084.stdout:2187: finishing rollback tid 2 to smithi08477203-49 2024-09-20T22:50:41.454 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 49 v 731 (ObjNum 470 snap 130 seq_num 470) dirty exists 2024-09-20T22:50:41.454 INFO:tasks.rados.rados.0.smithi084.stdout:2185: finishing write tid 2 to smithi08477203-46 2024-09-20T22:50:41.454 INFO:tasks.rados.rados.0.smithi084.stdout:2185: finishing write tid 3 to smithi08477203-46 2024-09-20T22:50:41.455 INFO:tasks.rados.rados.0.smithi084.stdout:2185: finishing write tid 4 to smithi08477203-46 2024-09-20T22:50:41.455 INFO:tasks.rados.rados.0.smithi084.stdout:2185: oid 46 updating version 0 to 806 2024-09-20T22:50:41.455 INFO:tasks.rados.rados.0.smithi084.stdout:2185: oid 46 updating version 806 to 807 2024-09-20T22:50:41.455 INFO:tasks.rados.rados.0.smithi084.stdout:2185: oid 46 version 807 is already newer than 805 2024-09-20T22:50:41.455 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 46 v 807 (ObjNum 761 snap 243 seq_num 761) dirty exists 2024-09-20T22:50:41.455 INFO:tasks.rados.rados.0.smithi084.stdout:2185: left oid 46 (ObjNum 761 snap 243 seq_num 761) 2024-09-20T22:50:41.455 INFO:tasks.rados.rados.0.smithi084.stdout:2186: expect (ObjNum 753 snap 238 seq_num 753) 2024-09-20T22:50:41.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:41 smithi154 ceph-mon[33264]: 4.11 scrub starts 2024-09-20T22:50:41.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:41 smithi154 ceph-mon[33264]: 4.11 scrub ok 2024-09-20T22:50:41.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:41 smithi154 ceph-mon[33264]: pgmap v741: 158 pgs: 1 active+clean+snaptrim_wait, 1 peering, 3 active+clean+snaptrim, 153 active+clean; 527 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 8.7 MiB/s rd, 1.7 MiB/s wr, 14 op/s 2024-09-20T22:50:41.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:41 smithi154 ceph-mon[33264]: 3.1 scrub starts 2024-09-20T22:50:41.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:41 smithi154 ceph-mon[33264]: 3.1 scrub ok 2024-09-20T22:50:41.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:41 smithi154 ceph-mon[33264]: 4.10 scrub starts 2024-09-20T22:50:41.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:41 smithi154 ceph-mon[33264]: 4.10 scrub ok 2024-09-20T22:50:41.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:41 smithi154 ceph-mon[33264]: 5.1d deep-scrub starts 2024-09-20T22:50:41.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:41 smithi154 ceph-mon[33264]: 5.1d deep-scrub ok 2024-09-20T22:50:41.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:41 smithi154 ceph-mon[33264]: 4.9 scrub starts 2024-09-20T22:50:41.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:41 smithi154 ceph-mon[33264]: 4.9 scrub ok 2024-09-20T22:50:41.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:41 smithi154 ceph-mon[33264]: osdmap e561: 8 total, 8 up, 8 in 2024-09-20T22:50:41.522 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:50:41.545 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:41.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:41.546 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:50:41.569 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:41 smithi084 ceph-mon[40024]: 4.11 scrub starts 2024-09-20T22:50:41.569 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:41 smithi084 ceph-mon[40024]: 4.11 scrub ok 2024-09-20T22:50:41.569 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:41 smithi084 ceph-mon[40024]: pgmap v741: 158 pgs: 1 active+clean+snaptrim_wait, 1 peering, 3 active+clean+snaptrim, 153 active+clean; 527 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 8.7 MiB/s rd, 1.7 MiB/s wr, 14 op/s 2024-09-20T22:50:41.569 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:41 smithi084 ceph-mon[40024]: 3.1 scrub starts 2024-09-20T22:50:41.569 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:41 smithi084 ceph-mon[40024]: 3.1 scrub ok 2024-09-20T22:50:41.569 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:41 smithi084 ceph-mon[40024]: 4.10 scrub starts 2024-09-20T22:50:41.569 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:41 smithi084 ceph-mon[40024]: 4.10 scrub ok 2024-09-20T22:50:41.569 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:41 smithi084 ceph-mon[40024]: 5.1d deep-scrub starts 2024-09-20T22:50:41.569 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:41 smithi084 ceph-mon[40024]: 5.1d deep-scrub ok 2024-09-20T22:50:41.570 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:41 smithi084 ceph-mon[40024]: 4.9 scrub starts 2024-09-20T22:50:41.570 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:41 smithi084 ceph-mon[40024]: 4.9 scrub ok 2024-09-20T22:50:41.570 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:41 smithi084 ceph-mon[40024]: osdmap e561: 8 total, 8 up, 8 in 2024-09-20T22:50:41.570 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:41 smithi084 ceph-mon[29774]: 4.11 scrub starts 2024-09-20T22:50:41.570 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:41 smithi084 ceph-mon[29774]: 4.11 scrub ok 2024-09-20T22:50:41.570 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:41 smithi084 ceph-mon[29774]: pgmap v741: 158 pgs: 1 active+clean+snaptrim_wait, 1 peering, 3 active+clean+snaptrim, 153 active+clean; 527 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 8.7 MiB/s rd, 1.7 MiB/s wr, 14 op/s 2024-09-20T22:50:41.570 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:41 smithi084 ceph-mon[29774]: 3.1 scrub starts 2024-09-20T22:50:41.570 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:41 smithi084 ceph-mon[29774]: 3.1 scrub ok 2024-09-20T22:50:41.570 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:41 smithi084 ceph-mon[29774]: 4.10 scrub starts 2024-09-20T22:50:41.570 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:41 smithi084 ceph-mon[29774]: 4.10 scrub ok 2024-09-20T22:50:41.570 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:41 smithi084 ceph-mon[29774]: 5.1d deep-scrub starts 2024-09-20T22:50:41.570 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:41 smithi084 ceph-mon[29774]: 5.1d deep-scrub ok 2024-09-20T22:50:41.570 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:41 smithi084 ceph-mon[29774]: 4.9 scrub starts 2024-09-20T22:50:41.570 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:41 smithi084 ceph-mon[29774]: 4.9 scrub ok 2024-09-20T22:50:41.571 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:41 smithi084 ceph-mon[29774]: osdmap e561: 8 total, 8 up, 8 in 2024-09-20T22:50:41.572 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-20T22:50:41.572 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2024-09-20T22:50:41.572 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2024-09-20T22:50:41.573 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2024-09-20T22:50:41.573 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-20T22:50:41.575 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:50:41.627 INFO:tasks.rados.rados.0.smithi084.stdout:2188: finishing copy_from racing read to smithi08477203-40 2024-09-20T22:50:41.627 INFO:tasks.rados.rados.0.smithi084.stdout:2188: finishing copy_from to smithi08477203-40 2024-09-20T22:50:41.627 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 40 v 718 (ObjNum 704 snap 219 seq_num 704) dirty exists 2024-09-20T22:50:41.627 INFO:tasks.rados.rados.0.smithi084.stdout:2184: done (4 left) 2024-09-20T22:50:41.627 INFO:tasks.rados.rados.0.smithi084.stdout:2185: done (3 left) 2024-09-20T22:50:41.627 INFO:tasks.rados.rados.0.smithi084.stdout:2186: done (2 left) 2024-09-20T22:50:41.627 INFO:tasks.rados.rados.0.smithi084.stdout:2187: done (1 left) 2024-09-20T22:50:41.627 INFO:tasks.rados.rados.0.smithi084.stdout:2188: done (0 left) 2024-09-20T22:50:41.628 INFO:tasks.rados.rados.0.smithi084.stdout:2189: snap_remove snap 238 2024-09-20T22:50:41.647 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:50:41.668 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:41.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:41.669 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:50:41.770 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:50:41.791 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:41.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:41.792 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:50:41.893 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:50:41.914 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:41.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:41.915 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:50:42.016 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:50:42.036 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:42.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:42.037 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:50:42.138 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:50:42.159 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:42.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:42.160 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:50:42.259 INFO:tasks.rados.rados.0.smithi084.stdout:2189: done (0 left) 2024-09-20T22:50:42.260 INFO:tasks.rados.rados.0.smithi084.stdout:2190: write (excl) oid 26 current snap is 243 2024-09-20T22:50:42.260 INFO:tasks.rados.rados.0.smithi084.stdout:2190: seq_num 762 ranges {406934=575038,1640573=474593,2642791=164143} 2024-09-20T22:50:42.261 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:50:42.265 INFO:tasks.rados.rados.0.smithi084.stdout:2190: writing smithi08477203-26 from 406934 to 981972 tid 1 2024-09-20T22:50:42.271 INFO:tasks.rados.rados.0.smithi084.stdout:2190: writing smithi08477203-26 from 1640573 to 2115166 tid 2 2024-09-20T22:50:42.275 INFO:tasks.rados.rados.0.smithi084.stdout:2190: writing smithi08477203-26 from 2642791 to 2806934 tid 3 2024-09-20T22:50:42.276 INFO:tasks.rados.rados.0.smithi084.stdout:2191: snap_create 2024-09-20T22:50:42.276 INFO:tasks.rados.rados.0.smithi084.stdout:2190: finishing write tid 1 to smithi08477203-26 2024-09-20T22:50:42.276 INFO:tasks.rados.rados.0.smithi084.stdout:2190: finishing write tid 2 to smithi08477203-26 2024-09-20T22:50:42.278 INFO:tasks.rados.rados.0.smithi084.stdout:2190: finishing write tid 3 to smithi08477203-26 2024-09-20T22:50:42.278 INFO:tasks.rados.rados.0.smithi084.stdout:2190: finishing write tid 4 to smithi08477203-26 2024-09-20T22:50:42.278 INFO:tasks.rados.rados.0.smithi084.stdout:2190: finishing write tid 5 to smithi08477203-26 2024-09-20T22:50:42.278 INFO:tasks.rados.rados.0.smithi084.stdout:2190: oid 26 updating version 0 to 792 2024-09-20T22:50:42.278 INFO:tasks.rados.rados.0.smithi084.stdout:2190: oid 26 updating version 792 to 794 2024-09-20T22:50:42.278 INFO:tasks.rados.rados.0.smithi084.stdout:2190: oid 26 version 794 is already newer than 791 2024-09-20T22:50:42.278 INFO:tasks.rados.rados.0.smithi084.stdout:2190: oid 26 version 794 is already newer than 793 2024-09-20T22:50:42.278 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 26 v 794 (ObjNum 762 snap 243 seq_num 762) dirty exists 2024-09-20T22:50:42.278 INFO:tasks.rados.rados.0.smithi084.stdout:2190: left oid 26 (ObjNum 762 snap 243 seq_num 762) 2024-09-20T22:50:42.284 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:42.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:42.285 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:50:42.385 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:50:42.406 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:42.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:42.407 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:50:42.508 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:50:42.529 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:42.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:42.530 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:50:42.552 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:42 smithi084 ceph-mon[40024]: 3.18 scrub starts 2024-09-20T22:50:42.553 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:42 smithi084 ceph-mon[40024]: 3.18 scrub ok 2024-09-20T22:50:42.553 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:42 smithi084 ceph-mon[40024]: 6.f scrub starts 2024-09-20T22:50:42.553 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:42 smithi084 ceph-mon[40024]: 6.f scrub ok 2024-09-20T22:50:42.553 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:42 smithi084 ceph-mon[40024]: 5.4 scrub starts 2024-09-20T22:50:42.553 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:42 smithi084 ceph-mon[40024]: 5.4 scrub ok 2024-09-20T22:50:42.553 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:42 smithi084 ceph-mon[40024]: 3.1a deep-scrub starts 2024-09-20T22:50:42.553 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:42 smithi084 ceph-mon[40024]: 3.1a deep-scrub ok 2024-09-20T22:50:42.553 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:42 smithi084 ceph-mon[40024]: osdmap e562: 8 total, 8 up, 8 in 2024-09-20T22:50:42.553 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:42 smithi084 ceph-mon[29774]: 3.18 scrub starts 2024-09-20T22:50:42.553 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:42 smithi084 ceph-mon[29774]: 3.18 scrub ok 2024-09-20T22:50:42.553 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:42 smithi084 ceph-mon[29774]: 6.f scrub starts 2024-09-20T22:50:42.553 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:42 smithi084 ceph-mon[29774]: 6.f scrub ok 2024-09-20T22:50:42.553 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:42 smithi084 ceph-mon[29774]: 5.4 scrub starts 2024-09-20T22:50:42.554 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:42 smithi084 ceph-mon[29774]: 5.4 scrub ok 2024-09-20T22:50:42.554 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:42 smithi084 ceph-mon[29774]: 3.1a deep-scrub starts 2024-09-20T22:50:42.554 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:42 smithi084 ceph-mon[29774]: 3.1a deep-scrub ok 2024-09-20T22:50:42.554 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:42 smithi084 ceph-mon[29774]: osdmap e562: 8 total, 8 up, 8 in 2024-09-20T22:50:42.578 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.1 dump_historic_ops 2024-09-20T22:50:42.631 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:50:42.652 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:42.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:42.653 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:50:42.753 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:50:42.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:42 smithi154 ceph-mon[33264]: 3.18 scrub starts 2024-09-20T22:50:42.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:42 smithi154 ceph-mon[33264]: 3.18 scrub ok 2024-09-20T22:50:42.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:42 smithi154 ceph-mon[33264]: 6.f scrub starts 2024-09-20T22:50:42.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:42 smithi154 ceph-mon[33264]: 6.f scrub ok 2024-09-20T22:50:42.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:42 smithi154 ceph-mon[33264]: 5.4 scrub starts 2024-09-20T22:50:42.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:42 smithi154 ceph-mon[33264]: 5.4 scrub ok 2024-09-20T22:50:42.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:42 smithi154 ceph-mon[33264]: 3.1a deep-scrub starts 2024-09-20T22:50:42.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:42 smithi154 ceph-mon[33264]: 3.1a deep-scrub ok 2024-09-20T22:50:42.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:42 smithi154 ceph-mon[33264]: osdmap e562: 8 total, 8 up, 8 in 2024-09-20T22:50:42.776 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:42.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:42.777 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:50:42.811 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:50:42.811 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:50:42.822 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:50:42.822 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:50:42.840 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:50:42.840 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:50:42.858 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-20T22:50:42.858 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T22:50:42.876 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:50:42.876 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:50:42.878 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:50:42.890 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:50:42.894 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:50:42.894 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:50:42.899 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:42.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:42.900 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:50:42.911 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:50:42.912 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:50:42.929 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:50:42.929 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:50:43.001 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:50:43.022 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:43.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:43.023 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:50:43.032 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd set noscrub 2024-09-20T22:50:43.124 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:50:43.145 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:43.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:43.146 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:50:43.247 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:50:43.249 INFO:tasks.rados.rados.0.smithi084.stdout:2190: done (1 left) 2024-09-20T22:50:43.249 INFO:tasks.rados.rados.0.smithi084.stdout:2191: done (0 left) 2024-09-20T22:50:43.249 INFO:tasks.rados.rados.0.smithi084.stdout:2192: delete oid 20 current snap is 244 2024-09-20T22:50:43.251 INFO:tasks.rados.rados.0.smithi084.stdout:2192: done (0 left) 2024-09-20T22:50:43.252 INFO:tasks.rados.rados.0.smithi084.stdout:2193: snap_create 2024-09-20T22:50:43.270 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:43.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:43.271 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:50:43.336 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:43 smithi084 ceph-mon[40024]: pgmap v744: 158 pgs: 1 active+clean+snaptrim_wait, 1 peering, 3 active+clean+snaptrim, 153 active+clean; 527 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 8.7 MiB/s rd, 1.7 MiB/s wr, 14 op/s 2024-09-20T22:50:43.337 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:43 smithi084 ceph-mon[40024]: 3.19 scrub starts 2024-09-20T22:50:43.337 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:43 smithi084 ceph-mon[40024]: 3.19 scrub ok 2024-09-20T22:50:43.337 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:43 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/1222679400' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:50:43.337 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:43 smithi084 ceph-mon[40024]: 5.1c scrub starts 2024-09-20T22:50:43.337 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:43 smithi084 ceph-mon[40024]: 5.1c scrub ok 2024-09-20T22:50:43.337 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:43 smithi084 ceph-mon[40024]: 3.10 scrub starts 2024-09-20T22:50:43.337 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:43 smithi084 ceph-mon[40024]: 3.10 scrub ok 2024-09-20T22:50:43.337 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:43 smithi084 ceph-mon[40024]: osdmap e563: 8 total, 8 up, 8 in 2024-09-20T22:50:43.337 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:43 smithi084 ceph-mon[29774]: pgmap v744: 158 pgs: 1 active+clean+snaptrim_wait, 1 peering, 3 active+clean+snaptrim, 153 active+clean; 527 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 8.7 MiB/s rd, 1.7 MiB/s wr, 14 op/s 2024-09-20T22:50:43.337 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:43 smithi084 ceph-mon[29774]: 3.19 scrub starts 2024-09-20T22:50:43.337 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:43 smithi084 ceph-mon[29774]: 3.19 scrub ok 2024-09-20T22:50:43.337 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:43 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/1222679400' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:50:43.338 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:43 smithi084 ceph-mon[29774]: 5.1c scrub starts 2024-09-20T22:50:43.338 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:43 smithi084 ceph-mon[29774]: 5.1c scrub ok 2024-09-20T22:50:43.338 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:43 smithi084 ceph-mon[29774]: 3.10 scrub starts 2024-09-20T22:50:43.338 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:43 smithi084 ceph-mon[29774]: 3.10 scrub ok 2024-09-20T22:50:43.338 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:43 smithi084 ceph-mon[29774]: osdmap e563: 8 total, 8 up, 8 in 2024-09-20T22:50:43.372 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:50:43.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:43.395 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:43.395 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:50:43.449 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-20T22:50:43.450 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:50:43.496 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:50:43.518 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:43.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:43.519 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:50:43.619 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:50:43.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:43.642 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:43.642 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:50:43.743 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:50:43.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:43 smithi154 ceph-mon[33264]: pgmap v744: 158 pgs: 1 active+clean+snaptrim_wait, 1 peering, 3 active+clean+snaptrim, 153 active+clean; 527 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 8.7 MiB/s rd, 1.7 MiB/s wr, 14 op/s 2024-09-20T22:50:43.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:43 smithi154 ceph-mon[33264]: 3.19 scrub starts 2024-09-20T22:50:43.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:43 smithi154 ceph-mon[33264]: 3.19 scrub ok 2024-09-20T22:50:43.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:43 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/1222679400' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:50:43.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:43 smithi154 ceph-mon[33264]: 5.1c scrub starts 2024-09-20T22:50:43.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:43 smithi154 ceph-mon[33264]: 5.1c scrub ok 2024-09-20T22:50:43.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:43 smithi154 ceph-mon[33264]: 3.10 scrub starts 2024-09-20T22:50:43.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:43 smithi154 ceph-mon[33264]: 3.10 scrub ok 2024-09-20T22:50:43.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:43 smithi154 ceph-mon[33264]: osdmap e563: 8 total, 8 up, 8 in 2024-09-20T22:50:43.764 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:43.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:43.766 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:50:43.866 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:50:43.888 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:43.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:43.889 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:50:43.990 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:50:44.012 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:44.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:44.012 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:50:44.113 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:50:44.135 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-20T22:50:44.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:44.136 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:44.136 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:50:44.237 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:50:44.255 INFO:tasks.rados.rados.0.smithi084.stdout:2193: done (0 left) 2024-09-20T22:50:44.256 INFO:tasks.rados.rados.0.smithi084.stdout:2194: snap_remove snap 205 2024-09-20T22:50:44.260 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:44.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:44.261 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:50:44.361 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:50:44.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:44.386 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:44.387 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:50:44.487 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:50:44.509 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:44.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:44.510 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:50:44.611 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:50:44.633 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:44.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:44.633 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:50:44.734 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:50:44.757 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:44.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:44.759 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:50:44.859 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:50:44.860 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 5, 6, 7, 0, 4] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-20T22:50:44.860 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-20T22:50:44.860 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-09-20T22:50:44.860 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph tell osd.1 'trim stale osdmaps' 2024-09-20T22:50:44.883 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:44.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:44.884 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:50:44.910 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:50:44.984 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:50:45.006 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:45.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:45.007 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:50:45.108 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:50:45.129 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:45.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:45.130 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:50:45.231 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:50:45.255 INFO:tasks.rados.rados.0.smithi084.stdout:2194: done (0 left) 2024-09-20T22:50:45.256 INFO:tasks.rados.rados.0.smithi084.stdout:2195: rollback oid 11 current snap is 245 2024-09-20T22:50:45.256 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 11 to 176 2024-09-20T22:50:45.256 INFO:tasks.rados.rados.0.smithi084.stdout:2196: read oid 8 snap -1 2024-09-20T22:50:45.256 INFO:tasks.rados.rados.0.smithi084.stdout:2196: expect (ObjNum 753 snap 238 seq_num 753) 2024-09-20T22:50:45.256 INFO:tasks.rados.rados.0.smithi084.stdout:2197: snap_create 2024-09-20T22:50:45.260 INFO:tasks.rados.rados.0.smithi084.stdout:2196: expect (ObjNum 753 snap 238 seq_num 753) 2024-09-20T22:50:45.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:45.260 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:45.261 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:50:45.362 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:50:45.383 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:45.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:45.384 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:50:45.433 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:45 smithi084 ceph-mon[40024]: pgmap v747: 158 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 153 active+clean; 488 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 8.3 MiB/s rd, 2.7 MiB/s wr, 15 op/s 2024-09-20T22:50:45.433 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:45 smithi084 ceph-mon[40024]: 3.17 scrub starts 2024-09-20T22:50:45.433 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:45 smithi084 ceph-mon[40024]: 3.17 scrub ok 2024-09-20T22:50:45.434 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:45 smithi084 ceph-mon[40024]: 4.a scrub starts 2024-09-20T22:50:45.434 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:45 smithi084 ceph-mon[40024]: 4.a scrub ok 2024-09-20T22:50:45.434 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:45 smithi084 ceph-mon[40024]: osdmap e564: 8 total, 8 up, 8 in 2024-09-20T22:50:45.434 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:45 smithi084 ceph-mon[40024]: 5.13 scrub starts 2024-09-20T22:50:45.434 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:45 smithi084 ceph-mon[40024]: 5.13 scrub ok 2024-09-20T22:50:45.435 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:45 smithi084 ceph-mon[29774]: pgmap v747: 158 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 153 active+clean; 488 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 8.3 MiB/s rd, 2.7 MiB/s wr, 15 op/s 2024-09-20T22:50:45.435 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:45 smithi084 ceph-mon[29774]: 3.17 scrub starts 2024-09-20T22:50:45.435 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:45 smithi084 ceph-mon[29774]: 3.17 scrub ok 2024-09-20T22:50:45.435 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:45 smithi084 ceph-mon[29774]: 4.a scrub starts 2024-09-20T22:50:45.435 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:45 smithi084 ceph-mon[29774]: 4.a scrub ok 2024-09-20T22:50:45.435 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:45 smithi084 ceph-mon[29774]: osdmap e564: 8 total, 8 up, 8 in 2024-09-20T22:50:45.435 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:45 smithi084 ceph-mon[29774]: 5.13 scrub starts 2024-09-20T22:50:45.435 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:45 smithi084 ceph-mon[29774]: 5.13 scrub ok 2024-09-20T22:50:45.440 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-20T22:50:45.440 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:50:45.440 INFO:tasks.rados.rados.0.smithi084.stdout:2195: finishing rollback tid 0 to smithi08477203-11 2024-09-20T22:50:45.440 INFO:tasks.rados.rados.0.smithi084.stdout:2195: finishing rollback tid 1 to smithi08477203-11 2024-09-20T22:50:45.440 INFO:tasks.rados.rados.0.smithi084.stdout:2195: finishing rollback tid 2 to smithi08477203-11 2024-09-20T22:50:45.440 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 11 v 735 (ObjNum 556 snap 169 seq_num 556) dirty exists 2024-09-20T22:50:45.485 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:50:45.508 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:45.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:45.509 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:50:45.609 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:50:45.632 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:45.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:45.634 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:50:45.735 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:50:45.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:45 smithi154 ceph-mon[33264]: pgmap v747: 158 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 153 active+clean; 488 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 8.3 MiB/s rd, 2.7 MiB/s wr, 15 op/s 2024-09-20T22:50:45.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:45 smithi154 ceph-mon[33264]: 3.17 scrub starts 2024-09-20T22:50:45.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:45 smithi154 ceph-mon[33264]: 3.17 scrub ok 2024-09-20T22:50:45.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:45 smithi154 ceph-mon[33264]: 4.a scrub starts 2024-09-20T22:50:45.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:45 smithi154 ceph-mon[33264]: 4.a scrub ok 2024-09-20T22:50:45.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:45 smithi154 ceph-mon[33264]: osdmap e564: 8 total, 8 up, 8 in 2024-09-20T22:50:45.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:45 smithi154 ceph-mon[33264]: 5.13 scrub starts 2024-09-20T22:50:45.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:45 smithi154 ceph-mon[33264]: 5.13 scrub ok 2024-09-20T22:50:45.757 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:45.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:45.758 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:50:45.858 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:50:45.881 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:45.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:45.882 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:50:45.982 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:50:46.004 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:46.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:46.005 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:50:46.078 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:50:46.092 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.2 dump_blocked_ops 2024-09-20T22:50:46.105 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:50:46.111 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:50:46.131 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:46.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:46.132 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:50:46.233 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:50:46.255 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:46.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:46.256 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:50:46.265 INFO:tasks.rados.rados.0.smithi084.stdout:2195: done (2 left) 2024-09-20T22:50:46.265 INFO:tasks.rados.rados.0.smithi084.stdout:2196: done (1 left) 2024-09-20T22:50:46.265 INFO:tasks.rados.rados.0.smithi084.stdout:2197: done (0 left) 2024-09-20T22:50:46.266 INFO:tasks.rados.rados.0.smithi084.stdout:2198: read oid 39 snap -1 2024-09-20T22:50:46.266 INFO:tasks.rados.rados.0.smithi084.stdout:2198: expect (ObjNum 743 snap 237 seq_num 743) 2024-09-20T22:50:46.266 INFO:tasks.rados.rados.0.smithi084.stdout:2199: write (excl) oid 13 current snap is 246 2024-09-20T22:50:46.266 INFO:tasks.rados.rados.0.smithi084.stdout:2199: seq_num 764 ranges {440548=725536,1757537=478262,2813990=26558} 2024-09-20T22:50:46.272 INFO:tasks.rados.rados.0.smithi084.stdout:2199: writing smithi08477203-13 from 440548 to 1166084 tid 1 2024-09-20T22:50:46.278 INFO:tasks.rados.rados.0.smithi084.stdout:2199: writing smithi08477203-13 from 1757537 to 2235799 tid 2 2024-09-20T22:50:46.282 INFO:tasks.rados.rados.0.smithi084.stdout:2199: writing smithi08477203-13 from 2813990 to 2840548 tid 3 2024-09-20T22:50:46.282 INFO:tasks.rados.rados.0.smithi084.stdout:2200: rollback oid 28 current snap is 246 2024-09-20T22:50:46.282 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 28 to 243 2024-09-20T22:50:46.282 INFO:tasks.rados.rados.0.smithi084.stdout:2198: expect (ObjNum 743 snap 237 seq_num 743) 2024-09-20T22:50:46.336 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:46 smithi084 ceph-mon[40024]: purged_snaps scrub starts 2024-09-20T22:50:46.336 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:46 smithi084 ceph-mon[40024]: purged_snaps scrub ok 2024-09-20T22:50:46.336 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:46 smithi084 ceph-mon[40024]: 7.0 scrub starts 2024-09-20T22:50:46.336 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:46 smithi084 ceph-mon[40024]: 7.0 scrub ok 2024-09-20T22:50:46.336 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:46 smithi084 ceph-mon[40024]: 3.12 scrub starts 2024-09-20T22:50:46.336 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:46 smithi084 ceph-mon[40024]: 3.12 scrub ok 2024-09-20T22:50:46.336 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:46 smithi084 ceph-mon[40024]: osdmap e565: 8 total, 8 up, 8 in 2024-09-20T22:50:46.336 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:46 smithi084 ceph-mon[40024]: 4.0 scrub starts 2024-09-20T22:50:46.337 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:46 smithi084 ceph-mon[40024]: 7.e scrub starts 2024-09-20T22:50:46.337 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:46 smithi084 ceph-mon[40024]: 4.0 scrub ok 2024-09-20T22:50:46.337 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:46 smithi084 ceph-mon[40024]: 7.e scrub ok 2024-09-20T22:50:46.337 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:46 smithi084 ceph-mon[40024]: 7.1 scrub starts 2024-09-20T22:50:46.337 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:46 smithi084 ceph-mon[40024]: 7.1 scrub ok 2024-09-20T22:50:46.337 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:46 smithi084 ceph-mon[29774]: purged_snaps scrub starts 2024-09-20T22:50:46.337 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:46 smithi084 ceph-mon[29774]: purged_snaps scrub ok 2024-09-20T22:50:46.337 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:46 smithi084 ceph-mon[29774]: 7.0 scrub starts 2024-09-20T22:50:46.337 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:46 smithi084 ceph-mon[29774]: 7.0 scrub ok 2024-09-20T22:50:46.337 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:46 smithi084 ceph-mon[29774]: 3.12 scrub starts 2024-09-20T22:50:46.337 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:46 smithi084 ceph-mon[29774]: 3.12 scrub ok 2024-09-20T22:50:46.337 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:46 smithi084 ceph-mon[29774]: osdmap e565: 8 total, 8 up, 8 in 2024-09-20T22:50:46.337 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:46 smithi084 ceph-mon[29774]: 4.0 scrub starts 2024-09-20T22:50:46.337 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:46 smithi084 ceph-mon[29774]: 7.e scrub starts 2024-09-20T22:50:46.338 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:46 smithi084 ceph-mon[29774]: 4.0 scrub ok 2024-09-20T22:50:46.338 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:46 smithi084 ceph-mon[29774]: 7.e scrub ok 2024-09-20T22:50:46.338 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:46 smithi084 ceph-mon[29774]: 7.1 scrub starts 2024-09-20T22:50:46.338 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:46 smithi084 ceph-mon[29774]: 7.1 scrub ok 2024-09-20T22:50:46.357 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:50:46.381 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:46.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:46.382 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:50:46.428 INFO:tasks.rados.rados.0.smithi084.stdout:2199: finishing write tid 1 to smithi08477203-13 2024-09-20T22:50:46.428 INFO:tasks.rados.rados.0.smithi084.stdout:2199: finishing write tid 2 to smithi08477203-13 2024-09-20T22:50:46.429 INFO:tasks.rados.rados.0.smithi084.stdout:2199: finishing write tid 3 to smithi08477203-13 2024-09-20T22:50:46.429 INFO:tasks.rados.rados.0.smithi084.stdout:2199: finishing write tid 4 to smithi08477203-13 2024-09-20T22:50:46.430 INFO:tasks.rados.rados.0.smithi084.stdout:2199: finishing write tid 5 to smithi08477203-13 2024-09-20T22:50:46.430 INFO:tasks.rados.rados.0.smithi084.stdout:2199: oid 13 updating version 0 to 1538 2024-09-20T22:50:46.430 INFO:tasks.rados.rados.0.smithi084.stdout:2199: oid 13 version 1538 is already newer than 1537 2024-09-20T22:50:46.430 INFO:tasks.rados.rados.0.smithi084.stdout:2199: oid 13 version 1538 is already newer than 1536 2024-09-20T22:50:46.430 INFO:tasks.rados.rados.0.smithi084.stdout:2199: oid 13 updating version 1538 to 1539 2024-09-20T22:50:46.430 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 13 v 1539 (ObjNum 764 snap 246 seq_num 764) dirty exists 2024-09-20T22:50:46.430 INFO:tasks.rados.rados.0.smithi084.stdout:2199: left oid 13 (ObjNum 764 snap 246 seq_num 764) 2024-09-20T22:50:46.430 INFO:tasks.rados.rados.0.smithi084.stdout:2200: finishing rollback tid 0 to smithi08477203-28 2024-09-20T22:50:46.431 INFO:tasks.rados.rados.0.smithi084.stdout:2200: finishing rollback tid 1 to smithi08477203-28 2024-09-20T22:50:46.431 INFO:tasks.rados.rados.0.smithi084.stdout:2200: finishing rollback tid 2 to smithi08477203-28 2024-09-20T22:50:46.431 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 28 v 1543 (ObjNum 751 snap 238 seq_num 751) dirty exists 2024-09-20T22:50:46.431 INFO:tasks.rados.rados.0.smithi084.stdout:2198: done (2 left) 2024-09-20T22:50:46.431 INFO:tasks.rados.rados.0.smithi084.stdout:2199: done (1 left) 2024-09-20T22:50:46.431 INFO:tasks.rados.rados.0.smithi084.stdout:2200: done (0 left) 2024-09-20T22:50:46.431 INFO:tasks.rados.rados.0.smithi084.stdout:2201: write oid 6 current snap is 246 2024-09-20T22:50:46.431 INFO:tasks.rados.rados.0.smithi084.stdout:2201: seq_num 765 ranges {457355=400785,1416019=621920,2725060=132295} 2024-09-20T22:50:46.432 INFO:tasks.rados.rados.0.smithi084.stdout:2201: writing smithi08477203-6 from 457355 to 858140 tid 1 2024-09-20T22:50:46.438 INFO:tasks.rados.rados.0.smithi084.stdout:2201: writing smithi08477203-6 from 1416019 to 2037939 tid 2 2024-09-20T22:50:46.443 INFO:tasks.rados.rados.0.smithi084.stdout:2201: writing smithi08477203-6 from 2725060 to 2857355 tid 3 2024-09-20T22:50:46.443 INFO:tasks.rados.rados.0.smithi084.stdout:2202: read oid 13 snap -1 2024-09-20T22:50:46.443 INFO:tasks.rados.rados.0.smithi084.stdout:2202: expect (ObjNum 764 snap 246 seq_num 764) 2024-09-20T22:50:46.443 INFO:tasks.rados.rados.0.smithi084.stdout:2203: copy_from oid 34 from oid 7 current snap is 246 2024-09-20T22:50:46.444 INFO:tasks.rados.rados.0.smithi084.stdout:2201: finishing write tid 1 to smithi08477203-6 2024-09-20T22:50:46.444 INFO:tasks.rados.rados.0.smithi084.stdout:2201: finishing write tid 2 to smithi08477203-6 2024-09-20T22:50:46.444 INFO:tasks.rados.rados.0.smithi084.stdout:2204: snap_create 2024-09-20T22:50:46.445 INFO:tasks.rados.rados.0.smithi084.stdout:2201: finishing write tid 3 to smithi08477203-6 2024-09-20T22:50:46.445 INFO:tasks.rados.rados.0.smithi084.stdout:2201: finishing write tid 4 to smithi08477203-6 2024-09-20T22:50:46.446 INFO:tasks.rados.rados.0.smithi084.stdout:2201: finishing write tid 5 to smithi08477203-6 2024-09-20T22:50:46.446 INFO:tasks.rados.rados.0.smithi084.stdout:2201: oid 6 updating version 0 to 752 2024-09-20T22:50:46.446 INFO:tasks.rados.rados.0.smithi084.stdout:2201: oid 6 version 752 is already newer than 751 2024-09-20T22:50:46.446 INFO:tasks.rados.rados.0.smithi084.stdout:2201: oid 6 version 752 is already newer than 750 2024-09-20T22:50:46.446 INFO:tasks.rados.rados.0.smithi084.stdout:2201: oid 6 updating version 752 to 753 2024-09-20T22:50:46.446 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 6 v 753 (ObjNum 765 snap 246 seq_num 765) dirty exists 2024-09-20T22:50:46.446 INFO:tasks.rados.rados.0.smithi084.stdout:2201: left oid 6 (ObjNum 765 snap 246 seq_num 765) 2024-09-20T22:50:46.451 INFO:tasks.rados.rados.0.smithi084.stdout:2203: finishing copy_from racing read to smithi08477203-34 2024-09-20T22:50:46.453 INFO:tasks.rados.rados.0.smithi084.stdout:2202: expect (ObjNum 764 snap 246 seq_num 764) 2024-09-20T22:50:46.483 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:50:46.507 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:46.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:46.508 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:50:46.609 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:50:46.631 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:46.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:46.632 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:50:46.654 INFO:tasks.rados.rados.0.smithi084.stdout:2203: finishing copy_from to smithi08477203-34 2024-09-20T22:50:46.654 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 34 v 856 (ObjNum 752 snap 238 seq_num 752) dirty exists 2024-09-20T22:50:46.733 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:50:46.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:46 smithi154 ceph-mon[33264]: purged_snaps scrub starts 2024-09-20T22:50:46.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:46 smithi154 ceph-mon[33264]: purged_snaps scrub ok 2024-09-20T22:50:46.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:46 smithi154 ceph-mon[33264]: 7.0 scrub starts 2024-09-20T22:50:46.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:46 smithi154 ceph-mon[33264]: 7.0 scrub ok 2024-09-20T22:50:46.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:46 smithi154 ceph-mon[33264]: 3.12 scrub starts 2024-09-20T22:50:46.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:46 smithi154 ceph-mon[33264]: 3.12 scrub ok 2024-09-20T22:50:46.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:46 smithi154 ceph-mon[33264]: osdmap e565: 8 total, 8 up, 8 in 2024-09-20T22:50:46.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:46 smithi154 ceph-mon[33264]: 4.0 scrub starts 2024-09-20T22:50:46.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:46 smithi154 ceph-mon[33264]: 7.e scrub starts 2024-09-20T22:50:46.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:46 smithi154 ceph-mon[33264]: 4.0 scrub ok 2024-09-20T22:50:46.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:46 smithi154 ceph-mon[33264]: 7.e scrub ok 2024-09-20T22:50:46.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:46 smithi154 ceph-mon[33264]: 7.1 scrub starts 2024-09-20T22:50:46.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:46 smithi154 ceph-mon[33264]: 7.1 scrub ok 2024-09-20T22:50:46.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:46.756 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:46.756 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:50:46.857 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:50:46.877 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-20T22:50:46.878 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:46.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:46.880 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:50:46.980 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:50:47.004 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:47.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:47.005 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:50:47.073 INFO:teuthology.orchestra.run.smithi084.stderr:no valid command found; 10 closest matches: 2024-09-20T22:50:47.074 INFO:teuthology.orchestra.run.smithi084.stderr:0 2024-09-20T22:50:47.074 INFO:teuthology.orchestra.run.smithi084.stderr:1 2024-09-20T22:50:47.074 INFO:teuthology.orchestra.run.smithi084.stderr:2 2024-09-20T22:50:47.074 INFO:teuthology.orchestra.run.smithi084.stderr:abort 2024-09-20T22:50:47.074 INFO:teuthology.orchestra.run.smithi084.stderr:assert 2024-09-20T22:50:47.074 INFO:teuthology.orchestra.run.smithi084.stderr:bench [] [] [] [] 2024-09-20T22:50:47.074 INFO:teuthology.orchestra.run.smithi084.stderr:bluefs debug_inject_read_zeros 2024-09-20T22:50:47.074 INFO:teuthology.orchestra.run.smithi084.stderr:bluefs files list 2024-09-20T22:50:47.074 INFO:teuthology.orchestra.run.smithi084.stderr:bluefs stats 2024-09-20T22:50:47.074 INFO:teuthology.orchestra.run.smithi084.stderr:bluestore allocator dump block 2024-09-20T22:50:47.074 INFO:teuthology.orchestra.run.smithi084.stderr:Error EINVAL: invalid command 2024-09-20T22:50:47.106 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:50:47.126 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:47.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:47.127 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:50:47.228 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:50:47.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:47.251 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:47.251 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:50:47.273 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is set 2024-09-20T22:50:47.278 INFO:tasks.rados.rados.0.smithi084.stdout:2201: done (3 left) 2024-09-20T22:50:47.278 INFO:tasks.rados.rados.0.smithi084.stdout:2202: done (2 left) 2024-09-20T22:50:47.278 INFO:tasks.rados.rados.0.smithi084.stdout:2203: done (1 left) 2024-09-20T22:50:47.279 INFO:tasks.rados.rados.0.smithi084.stdout:2204: done (0 left) 2024-09-20T22:50:47.279 INFO:tasks.rados.rados.0.smithi084.stdout:2205: copy_from oid 5 from oid 11 current snap is 247 2024-09-20T22:50:47.279 INFO:tasks.rados.rados.0.smithi084.stdout:2206: write (excl) oid 34 current snap is 247 2024-09-20T22:50:47.279 INFO:tasks.rados.rados.0.smithi084.stdout:2206: seq_num 766 ranges {474162=476034,1358148=481931,2352483=465303} 2024-09-20T22:50:47.285 INFO:tasks.rados.rados.0.smithi084.stdout:2206: writing smithi08477203-34 from 474162 to 950196 tid 1 2024-09-20T22:50:47.290 INFO:tasks.rados.rados.0.smithi084.stdout:2206: writing smithi08477203-34 from 1358148 to 1840079 tid 2 2024-09-20T22:50:47.300 INFO:tasks.rados.rados.0.smithi084.stdout:2206: writing smithi08477203-34 from 2352483 to 2817786 tid 3 2024-09-20T22:50:47.300 INFO:tasks.rados.rados.0.smithi084.stdout:2207: read oid 10 snap -1 2024-09-20T22:50:47.300 INFO:tasks.rados.rados.0.smithi084.stdout:2207: expect (ObjNum 575 snap 172 seq_num 575) 2024-09-20T22:50:47.300 INFO:tasks.rados.rados.0.smithi084.stdout:2208: snap_create 2024-09-20T22:50:47.300 INFO:tasks.rados.rados.0.smithi084.stdout:2205: finishing copy_from racing read to smithi08477203-5 2024-09-20T22:50:47.300 INFO:tasks.rados.rados.0.smithi084.stdout:2206: finishing write tid 1 to smithi08477203-34 2024-09-20T22:50:47.301 INFO:tasks.rados.rados.0.smithi084.stdout:2206: finishing write tid 2 to smithi08477203-34 2024-09-20T22:50:47.302 INFO:tasks.rados.rados.0.smithi084.stdout:2206: finishing write tid 3 to smithi08477203-34 2024-09-20T22:50:47.303 INFO:tasks.rados.rados.0.smithi084.stdout:2206: finishing write tid 4 to smithi08477203-34 2024-09-20T22:50:47.303 INFO:tasks.rados.rados.0.smithi084.stdout:2206: finishing write tid 5 to smithi08477203-34 2024-09-20T22:50:47.303 INFO:tasks.rados.rados.0.smithi084.stdout:2206: oid 34 updating version 0 to 858 2024-09-20T22:50:47.303 INFO:tasks.rados.rados.0.smithi084.stdout:2206: oid 34 updating version 858 to 860 2024-09-20T22:50:47.303 INFO:tasks.rados.rados.0.smithi084.stdout:2206: oid 34 version 860 is already newer than 859 2024-09-20T22:50:47.303 INFO:tasks.rados.rados.0.smithi084.stdout:2206: oid 34 version 860 is already newer than 857 2024-09-20T22:50:47.303 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 34 v 860 (ObjNum 766 snap 247 seq_num 766) dirty exists 2024-09-20T22:50:47.304 INFO:tasks.rados.rados.0.smithi084.stdout:2206: left oid 34 (ObjNum 766 snap 247 seq_num 766) 2024-09-20T22:50:47.306 INFO:tasks.rados.rados.0.smithi084.stdout:2205: finishing copy_from to smithi08477203-5 2024-09-20T22:50:47.306 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 5 v 735 (ObjNum 556 snap 169 seq_num 556) dirty exists 2024-09-20T22:50:47.309 INFO:tasks.rados.rados.0.smithi084.stdout:2207: expect (ObjNum 575 snap 172 seq_num 575) 2024-09-20T22:50:47.352 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:50:47.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:47.373 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:47.373 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:50:47.474 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:50:47.495 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:47.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:47.496 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:50:47.525 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:50:47.597 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:50:47.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:47 smithi084 ceph-mon[40024]: pgmap v750: 158 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 153 active+clean; 488 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 8.3 MiB/s rd, 2.7 MiB/s wr, 15 op/s 2024-09-20T22:50:47.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:47 smithi084 ceph-mon[40024]: osdmap e566: 8 total, 8 up, 8 in 2024-09-20T22:50:47.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:47 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/1702583913' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-20T22:50:47.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:47 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:50:47.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:47 smithi084 ceph-mon[29774]: pgmap v750: 158 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 153 active+clean; 488 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 8.3 MiB/s rd, 2.7 MiB/s wr, 15 op/s 2024-09-20T22:50:47.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:47 smithi084 ceph-mon[29774]: osdmap e566: 8 total, 8 up, 8 in 2024-09-20T22:50:47.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:47 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/1702583913' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-20T22:50:47.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:47 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:50:47.619 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:47.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:47.620 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:50:47.721 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:50:47.742 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:47.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:47.743 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:50:47.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:47 smithi154 ceph-mon[33264]: pgmap v750: 158 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 153 active+clean; 488 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 8.3 MiB/s rd, 2.7 MiB/s wr, 15 op/s 2024-09-20T22:50:47.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:47 smithi154 ceph-mon[33264]: osdmap e566: 8 total, 8 up, 8 in 2024-09-20T22:50:47.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:47 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/1702583913' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-20T22:50:47.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:47 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:50:47.844 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:50:47.866 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:47.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:47.867 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:50:47.967 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:50:47.988 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:47.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:47.989 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:50:48.026 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-20T22:50:48.027 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-20T22:50:48.027 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph tell osd.2 'trim stale osdmaps' 2024-09-20T22:50:48.091 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:50:48.112 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:48.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:48.113 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:50:48.213 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:50:48.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:48.236 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:48.236 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:50:48.253 INFO:tasks.rados.rados.0.smithi084.stdout:2205: done (3 left) 2024-09-20T22:50:48.253 INFO:tasks.rados.rados.0.smithi084.stdout:2206: done (2 left) 2024-09-20T22:50:48.253 INFO:tasks.rados.rados.0.smithi084.stdout:2207: done (1 left) 2024-09-20T22:50:48.253 INFO:tasks.rados.rados.0.smithi084.stdout:2208: done (0 left) 2024-09-20T22:50:48.253 INFO:tasks.rados.rados.0.smithi084.stdout:2209: snap_remove snap 230 2024-09-20T22:50:48.337 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:50:48.358 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:48.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:48.359 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:50:48.460 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:50:48.481 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:48.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:48.482 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:50:48.572 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:50:48 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: ::ffff:172.21.15.154 - - [20/Sep/2024:22:50:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-20T22:50:48.573 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:48 smithi084 ceph-mon[29774]: 6.c scrub starts 2024-09-20T22:50:48.573 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:48 smithi084 ceph-mon[29774]: 6.c scrub ok 2024-09-20T22:50:48.573 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:48 smithi084 ceph-mon[29774]: 3.a scrub starts 2024-09-20T22:50:48.573 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:48 smithi084 ceph-mon[29774]: 3.a scrub ok 2024-09-20T22:50:48.573 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:48 smithi084 ceph-mon[29774]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:50:48.573 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:48 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/1702583913' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T22:50:48.573 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:48 smithi084 ceph-mon[29774]: osdmap e567: 8 total, 8 up, 8 in 2024-09-20T22:50:48.573 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:48 smithi084 ceph-mon[29774]: 6.19 scrub starts 2024-09-20T22:50:48.573 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:48 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:50:48.573 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:48 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:50:48.573 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:48 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:50:48.573 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:48 smithi084 ceph-mon[29774]: osdmap e568: 8 total, 8 up, 8 in 2024-09-20T22:50:48.574 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:48 smithi084 ceph-mon[40024]: 6.c scrub starts 2024-09-20T22:50:48.574 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:48 smithi084 ceph-mon[40024]: 6.c scrub ok 2024-09-20T22:50:48.574 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:48 smithi084 ceph-mon[40024]: 3.a scrub starts 2024-09-20T22:50:48.574 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:48 smithi084 ceph-mon[40024]: 3.a scrub ok 2024-09-20T22:50:48.574 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:48 smithi084 ceph-mon[40024]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:50:48.574 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:48 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/1702583913' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T22:50:48.574 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:48 smithi084 ceph-mon[40024]: osdmap e567: 8 total, 8 up, 8 in 2024-09-20T22:50:48.574 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:48 smithi084 ceph-mon[40024]: 6.19 scrub starts 2024-09-20T22:50:48.574 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:48 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:50:48.574 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:48 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:50:48.574 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:48 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:50:48.574 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:48 smithi084 ceph-mon[40024]: osdmap e568: 8 total, 8 up, 8 in 2024-09-20T22:50:48.577 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:50:48.583 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:50:48.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:48.609 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:48.609 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:50:48.710 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:50:48.731 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:48.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:48.732 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:50:48.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:48 smithi154 ceph-mon[33264]: 6.c scrub starts 2024-09-20T22:50:48.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:48 smithi154 ceph-mon[33264]: 6.c scrub ok 2024-09-20T22:50:48.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:48 smithi154 ceph-mon[33264]: 3.a scrub starts 2024-09-20T22:50:48.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:48 smithi154 ceph-mon[33264]: 3.a scrub ok 2024-09-20T22:50:48.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:48 smithi154 ceph-mon[33264]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:50:48.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:48 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/1702583913' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T22:50:48.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:48 smithi154 ceph-mon[33264]: osdmap e567: 8 total, 8 up, 8 in 2024-09-20T22:50:48.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:48 smithi154 ceph-mon[33264]: 6.19 scrub starts 2024-09-20T22:50:48.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:48 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:50:48.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:48 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:50:48.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:48 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:50:48.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:48 smithi154 ceph-mon[33264]: osdmap e568: 8 total, 8 up, 8 in 2024-09-20T22:50:48.832 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:50:48.854 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:48.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:48.855 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:50:48.955 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:50:48.978 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:48.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:48.979 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:50:49.080 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:50:49.101 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:49.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:49.102 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:50:49.203 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:50:49.224 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:49.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:49.225 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:50:49.273 INFO:tasks.rados.rados.0.smithi084.stdout:2209: done (0 left) 2024-09-20T22:50:49.273 INFO:tasks.rados.rados.0.smithi084.stdout:2210: read oid 10 snap 237 2024-09-20T22:50:49.273 INFO:tasks.rados.rados.0.smithi084.stdout:2210: expect (ObjNum 575 snap 172 seq_num 575) 2024-09-20T22:50:49.273 INFO:tasks.rados.rados.0.smithi084.stdout:2211: copy_from oid 23 from oid 38 current snap is 248 2024-09-20T22:50:49.273 INFO:tasks.rados.rados.0.smithi084.stdout:2212: read oid 50 snap -1 2024-09-20T22:50:49.273 INFO:tasks.rados.rados.0.smithi084.stdout:2212: expect (ObjNum 515 snap 148 seq_num 515) 2024-09-20T22:50:49.274 INFO:tasks.rados.rados.0.smithi084.stdout:2213: read oid 28 snap -1 2024-09-20T22:50:49.274 INFO:tasks.rados.rados.0.smithi084.stdout:2213: expect (ObjNum 751 snap 238 seq_num 751) 2024-09-20T22:50:49.274 INFO:tasks.rados.rados.0.smithi084.stdout:2214: copy_from oid 13 from oid 36 current snap is 248 2024-09-20T22:50:49.274 INFO:tasks.rados.rados.0.smithi084.stdout:2215: write (excl) oid 29 current snap is 248 2024-09-20T22:50:49.274 INFO:tasks.rados.rados.0.smithi084.stdout:2215: seq_num 767 ranges {490969=551283,1816630=625589} 2024-09-20T22:50:49.278 INFO:tasks.rados.rados.0.smithi084.stdout:2215: writing smithi08477203-29 from 490969 to 1042252 tid 1 2024-09-20T22:50:49.289 INFO:tasks.rados.rados.0.smithi084.stdout:2215: writing smithi08477203-29 from 1816630 to 2442219 tid 2 2024-09-20T22:50:49.289 INFO:tasks.rados.rados.0.smithi084.stdout:2216: snap_remove snap 211 2024-09-20T22:50:49.325 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:50:49.346 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:49.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:49.347 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:50:49.448 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:50:49.459 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:49 smithi084 ceph-mon[29774]: pgmap v753: 158 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 153 active+clean; 488 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-20T22:50:49.459 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:49 smithi084 ceph-mon[40024]: pgmap v753: 158 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 153 active+clean; 488 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-20T22:50:49.470 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:49.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:49.471 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:50:49.554 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-20T22:50:49.554 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2024-09-20T22:50:49.554 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2024-09-20T22:50:49.554 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2024-09-20T22:50:49.554 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-20T22:50:49.572 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:50:49.593 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:49.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:49.594 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:50:49.695 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:50:49.716 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:49.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:49.717 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:50:49.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:49 smithi154 ceph-mon[33264]: pgmap v753: 158 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 153 active+clean; 488 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail 2024-09-20T22:50:49.754 INFO:journalctl@ceph.iscsi.iscsi.a.smithi154.stdout:Sep 20 22:50:49 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-iscsi-iscsi-a[59730]: debug there is no tcmu-runner data available 2024-09-20T22:50:49.818 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:50:49.839 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:49.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:49.840 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:50:49.941 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:50:49.963 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:49.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:49.964 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:50:50.064 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:50:50.086 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:50.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:50.087 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:50:50.187 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:50:50.209 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:50.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:50.210 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:50:50.269 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.2 dump_historic_ops 2024-09-20T22:50:50.288 INFO:tasks.rados.rados.0.smithi084.stdout:2216: done (6 left) 2024-09-20T22:50:50.288 INFO:tasks.rados.rados.0.smithi084.stdout:2217: write oid 40 current snap is 248 2024-09-20T22:50:50.288 INFO:tasks.rados.rados.0.smithi084.stdout:2217: seq_num 768 ranges {507776=742885,1991465=485600} 2024-09-20T22:50:50.294 INFO:tasks.rados.rados.0.smithi084.stdout:2217: writing smithi08477203-40 from 507776 to 1250661 tid 1 2024-09-20T22:50:50.302 INFO:tasks.rados.rados.0.smithi084.stdout:2217: writing smithi08477203-40 from 1991465 to 2477065 tid 2 2024-09-20T22:50:50.302 INFO:tasks.rados.rados.0.smithi084.stdout:2218: read oid 18 snap -1 2024-09-20T22:50:50.302 INFO:tasks.rados.rados.0.smithi084.stdout:2218: expect (ObjNum 657 snap 199 seq_num 657) 2024-09-20T22:50:50.302 INFO:tasks.rados.rados.0.smithi084.stdout:2219: write oid 32 current snap is 248 2024-09-20T22:50:50.302 INFO:tasks.rados.rados.0.smithi084.stdout:2219: seq_num 769 ranges {524583=418134,1649947=629258,2834752=89831} 2024-09-20T22:50:50.307 INFO:tasks.rados.rados.0.smithi084.stdout:2219: writing smithi08477203-32 from 524583 to 942717 tid 1 2024-09-20T22:50:50.311 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:50:50.314 INFO:tasks.rados.rados.0.smithi084.stdout:2219: writing smithi08477203-32 from 1649947 to 2279205 tid 2 2024-09-20T22:50:50.318 INFO:tasks.rados.rados.0.smithi084.stdout:2219: writing smithi08477203-32 from 2834752 to 2924583 tid 3 2024-09-20T22:50:50.318 INFO:tasks.rados.rados.0.smithi084.stdout:2220: rollback oid 6 current snap is 248 2024-09-20T22:50:50.318 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 6 to 245 2024-09-20T22:50:50.318 INFO:tasks.rados.rados.0.smithi084.stdout:2210: expect (ObjNum 575 snap 172 seq_num 575) 2024-09-20T22:50:50.334 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:50.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:50.335 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:50:50.359 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:50:50.359 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:50:50.359 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:50:50.360 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-20T22:50:50.360 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:50:50.360 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:50:50.360 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:50:50.360 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:50:50.360 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T22:50:50.360 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:50:50.360 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:50:50.361 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:50:50.361 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:50:50.361 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:50:50.361 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:50:50.361 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:50:50.361 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:50 smithi084 ceph-mon[40024]: osdmap e569: 8 total, 8 up, 8 in 2024-09-20T22:50:50.361 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:50 smithi084 ceph-mon[40024]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:50:50.361 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:50 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/2801743501' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:50:50.362 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:50 smithi084 ceph-mon[29774]: osdmap e569: 8 total, 8 up, 8 in 2024-09-20T22:50:50.362 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:50 smithi084 ceph-mon[29774]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:50:50.362 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:50 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/2801743501' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:50:50.436 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:50:50.459 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:50.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:50.460 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:50:50.559 INFO:tasks.rados.rados.0.smithi084.stdout:2212: expect (ObjNum 515 snap 148 seq_num 515) 2024-09-20T22:50:50.560 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:50:50.581 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:50.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:50.582 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:50:50.609 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-20T22:50:50.683 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:50:50.708 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:50.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:50.710 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:50:50.719 INFO:tasks.rados.rados.0.smithi084.stdout:2211: finishing copy_from racing read to smithi08477203-23 2024-09-20T22:50:50.719 INFO:tasks.rados.rados.0.smithi084.stdout:2213: expect (ObjNum 751 snap 238 seq_num 751) 2024-09-20T22:50:50.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:50 smithi154 ceph-mon[33264]: osdmap e569: 8 total, 8 up, 8 in 2024-09-20T22:50:50.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:50 smithi154 ceph-mon[33264]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:50:50.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:50 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/2801743501' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:50:50.810 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:50:50.831 INFO:tasks.rados.rados.0.smithi084.stdout:2214: finishing copy_from racing read to smithi08477203-13 2024-09-20T22:50:50.831 INFO:tasks.rados.rados.0.smithi084.stdout:2215: finishing write tid 1 to smithi08477203-29 2024-09-20T22:50:50.831 INFO:tasks.rados.rados.0.smithi084.stdout:2211: finishing copy_from to smithi08477203-23 2024-09-20T22:50:50.831 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 23 v 723 (ObjNum 734 snap 232 seq_num 734) dirty exists 2024-09-20T22:50:50.831 INFO:tasks.rados.rados.0.smithi084.stdout:2214: finishing copy_from to smithi08477203-13 2024-09-20T22:50:50.831 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 13 v 402 (ObjNum 440 snap 115 seq_num 440) dirty exists 2024-09-20T22:50:50.831 INFO:tasks.rados.rados.0.smithi084.stdout:2215: finishing write tid 2 to smithi08477203-29 2024-09-20T22:50:50.831 INFO:tasks.rados.rados.0.smithi084.stdout:2215: finishing write tid 3 to smithi08477203-29 2024-09-20T22:50:50.831 INFO:tasks.rados.rados.0.smithi084.stdout:2215: finishing write tid 4 to smithi08477203-29 2024-09-20T22:50:50.831 INFO:tasks.rados.rados.0.smithi084.stdout:2215: oid 29 updating version 0 to 1552 2024-09-20T22:50:50.831 INFO:tasks.rados.rados.0.smithi084.stdout:2215: oid 29 version 1552 is already newer than 1545 2024-09-20T22:50:50.831 INFO:tasks.rados.rados.0.smithi084.stdout:2215: oid 29 updating version 1552 to 1553 2024-09-20T22:50:50.831 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 29 v 1553 (ObjNum 767 snap 248 seq_num 767) dirty exists 2024-09-20T22:50:50.832 INFO:tasks.rados.rados.0.smithi084.stdout:2215: left oid 29 (ObjNum 767 snap 248 seq_num 767) 2024-09-20T22:50:50.832 INFO:tasks.rados.rados.0.smithi084.stdout:2217: finishing write tid 1 to smithi08477203-40 2024-09-20T22:50:50.832 INFO:tasks.rados.rados.0.smithi084.stdout:2210: done (9 left) 2024-09-20T22:50:50.832 INFO:tasks.rados.rados.0.smithi084.stdout:2211: done (8 left) 2024-09-20T22:50:50.832 INFO:tasks.rados.rados.0.smithi084.stdout:2212: done (7 left) 2024-09-20T22:50:50.832 INFO:tasks.rados.rados.0.smithi084.stdout:2213: done (6 left) 2024-09-20T22:50:50.832 INFO:tasks.rados.rados.0.smithi084.stdout:2214: done (5 left) 2024-09-20T22:50:50.832 INFO:tasks.rados.rados.0.smithi084.stdout:2215: done (4 left) 2024-09-20T22:50:50.832 INFO:tasks.rados.rados.0.smithi084.stdout:2221: delete oid 28 current snap is 248 2024-09-20T22:50:50.832 INFO:tasks.rados.rados.0.smithi084.stdout:2217: finishing write tid 2 to smithi08477203-40 2024-09-20T22:50:50.832 INFO:tasks.rados.rados.0.smithi084.stdout:2217: finishing write tid 3 to smithi08477203-40 2024-09-20T22:50:50.832 INFO:tasks.rados.rados.0.smithi084.stdout:2217: finishing write tid 4 to smithi08477203-40 2024-09-20T22:50:50.832 INFO:tasks.rados.rados.0.smithi084.stdout:2217: oid 40 updating version 0 to 747 2024-09-20T22:50:50.832 INFO:tasks.rados.rados.0.smithi084.stdout:2217: oid 40 updating version 747 to 748 2024-09-20T22:50:50.832 INFO:tasks.rados.rados.0.smithi084.stdout:2217: oid 40 updating version 748 to 749 2024-09-20T22:50:50.832 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 40 v 749 (ObjNum 768 snap 248 seq_num 768) dirty exists 2024-09-20T22:50:50.832 INFO:tasks.rados.rados.0.smithi084.stdout:2217: left oid 40 (ObjNum 768 snap 248 seq_num 768) 2024-09-20T22:50:50.833 INFO:tasks.rados.rados.0.smithi084.stdout:2218: expect (ObjNum 657 snap 199 seq_num 657) 2024-09-20T22:50:50.836 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:50.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:50.837 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:50:50.938 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:50:50.960 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:50.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:50.961 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:50:51.061 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:50:51.065 INFO:tasks.rados.rados.0.smithi084.stdout:2219: finishing write tid 1 to smithi08477203-32 2024-09-20T22:50:51.065 INFO:tasks.rados.rados.0.smithi084.stdout:2219: finishing write tid 2 to smithi08477203-32 2024-09-20T22:50:51.065 INFO:tasks.rados.rados.0.smithi084.stdout:2219: finishing write tid 3 to smithi08477203-32 2024-09-20T22:50:51.065 INFO:tasks.rados.rados.0.smithi084.stdout:2219: finishing write tid 4 to smithi08477203-32 2024-09-20T22:50:51.066 INFO:tasks.rados.rados.0.smithi084.stdout:2219: finishing write tid 5 to smithi08477203-32 2024-09-20T22:50:51.066 INFO:tasks.rados.rados.0.smithi084.stdout:2219: oid 32 updating version 0 to 739 2024-09-20T22:50:51.066 INFO:tasks.rados.rados.0.smithi084.stdout:2219: oid 32 updating version 739 to 741 2024-09-20T22:50:51.066 INFO:tasks.rados.rados.0.smithi084.stdout:2219: oid 32 version 741 is already newer than 740 2024-09-20T22:50:51.066 INFO:tasks.rados.rados.0.smithi084.stdout:2219: oid 32 version 741 is already newer than 738 2024-09-20T22:50:51.066 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 32 v 741 (ObjNum 769 snap 248 seq_num 769) dirty exists 2024-09-20T22:50:51.066 INFO:tasks.rados.rados.0.smithi084.stdout:2219: left oid 32 (ObjNum 769 snap 248 seq_num 769) 2024-09-20T22:50:51.066 INFO:tasks.rados.rados.0.smithi084.stdout:2220: finishing rollback tid 0 to smithi08477203-6 2024-09-20T22:50:51.066 INFO:tasks.rados.rados.0.smithi084.stdout:2220: finishing rollback tid 1 to smithi08477203-6 2024-09-20T22:50:51.066 INFO:tasks.rados.rados.0.smithi084.stdout:2220: finishing rollback tid 2 to smithi08477203-6 2024-09-20T22:50:51.066 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 6 v 756 (ObjNum 754 snap 238 seq_num 754) dirty exists 2024-09-20T22:50:51.066 INFO:tasks.rados.rados.0.smithi084.stdout:2217: done (4 left) 2024-09-20T22:50:51.066 INFO:tasks.rados.rados.0.smithi084.stdout:2218: done (3 left) 2024-09-20T22:50:51.066 INFO:tasks.rados.rados.0.smithi084.stdout:2219: done (2 left) 2024-09-20T22:50:51.066 INFO:tasks.rados.rados.0.smithi084.stdout:2220: done (1 left) 2024-09-20T22:50:51.067 INFO:tasks.rados.rados.0.smithi084.stdout:2221: done (0 left) 2024-09-20T22:50:51.067 INFO:tasks.rados.rados.0.smithi084.stdout:2222: read oid 13 snap 165 2024-09-20T22:50:51.067 INFO:tasks.rados.rados.0.smithi084.stdout:2222: expect (ObjNum 501 snap 143 seq_num 501) 2024-09-20T22:50:51.067 INFO:tasks.rados.rados.0.smithi084.stdout:2223: read oid 26 snap 209 2024-09-20T22:50:51.067 INFO:tasks.rados.rados.0.smithi084.stdout:2223: expect deleted 2024-09-20T22:50:51.067 INFO:tasks.rados.rados.0.smithi084.stdout:2224: write (excl) oid 38 current snap is 248 2024-09-20T22:50:51.067 INFO:tasks.rados.rados.0.smithi084.stdout:2224: seq_num 771 ranges {558197=568632,1650558=632927,2773245=184952} 2024-09-20T22:50:51.071 INFO:tasks.rados.rados.0.smithi084.stdout:2224: writing smithi08477203-38 from 558197 to 1126829 tid 1 2024-09-20T22:50:51.077 INFO:tasks.rados.rados.0.smithi084.stdout:2224: writing smithi08477203-38 from 1650558 to 2283485 tid 2 2024-09-20T22:50:51.081 INFO:tasks.rados.rados.0.smithi084.stdout:2224: writing smithi08477203-38 from 2773245 to 2958197 tid 3 2024-09-20T22:50:51.081 INFO:tasks.rados.rados.0.smithi084.stdout:2225: write oid 12 current snap is 248 2024-09-20T22:50:51.081 INFO:tasks.rados.rados.0.smithi084.stdout:2225: seq_num 772 ranges {575004=643881,1709040=492938,2917021=57983} 2024-09-20T22:50:51.088 INFO:tasks.rados.rados.0.smithi084.stdout:2225: writing smithi08477203-12 from 575004 to 1218885 tid 1 2024-09-20T22:50:51.089 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:51.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:51.090 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:50:51.093 INFO:tasks.rados.rados.0.smithi084.stdout:2225: writing smithi08477203-12 from 1709040 to 2201978 tid 2 2024-09-20T22:50:51.098 INFO:tasks.rados.rados.0.smithi084.stdout:2225: writing smithi08477203-12 from 2917021 to 2975004 tid 3 2024-09-20T22:50:51.098 INFO:tasks.rados.rados.0.smithi084.stdout:2226: copy_from oid 20 from oid 49 current snap is 248 2024-09-20T22:50:51.098 INFO:tasks.rados.rados.0.smithi084.stdout:2222: expect (ObjNum 501 snap 143 seq_num 501) 2024-09-20T22:50:51.192 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:50:51.217 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:51.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:51.218 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:50:51.243 INFO:tasks.rados.rados.0.smithi084.stdout:2224: finishing write tid 1 to smithi08477203-38 2024-09-20T22:50:51.243 INFO:tasks.rados.rados.0.smithi084.stdout:2222: done (4 left) 2024-09-20T22:50:51.243 INFO:tasks.rados.rados.0.smithi084.stdout:2223: done (3 left) 2024-09-20T22:50:51.243 INFO:tasks.rados.rados.0.smithi084.stdout:2227: delete oid 39 current snap is 248 2024-09-20T22:50:51.243 INFO:tasks.rados.rados.0.smithi084.stdout:2224: finishing write tid 2 to smithi08477203-38 2024-09-20T22:50:51.243 INFO:tasks.rados.rados.0.smithi084.stdout:2224: finishing write tid 3 to smithi08477203-38 2024-09-20T22:50:51.243 INFO:tasks.rados.rados.0.smithi084.stdout:2224: finishing write tid 4 to smithi08477203-38 2024-09-20T22:50:51.243 INFO:tasks.rados.rados.0.smithi084.stdout:2224: finishing write tid 5 to smithi08477203-38 2024-09-20T22:50:51.243 INFO:tasks.rados.rados.0.smithi084.stdout:2224: oid 38 updating version 0 to 817 2024-09-20T22:50:51.243 INFO:tasks.rados.rados.0.smithi084.stdout:2224: oid 38 version 817 is already newer than 814 2024-09-20T22:50:51.243 INFO:tasks.rados.rados.0.smithi084.stdout:2224: oid 38 version 817 is already newer than 815 2024-09-20T22:50:51.243 INFO:tasks.rados.rados.0.smithi084.stdout:2224: oid 38 version 817 is already newer than 816 2024-09-20T22:50:51.244 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 38 v 817 (ObjNum 771 snap 248 seq_num 771) dirty exists 2024-09-20T22:50:51.244 INFO:tasks.rados.rados.0.smithi084.stdout:2224: left oid 38 (ObjNum 771 snap 248 seq_num 771) 2024-09-20T22:50:51.244 INFO:tasks.rados.rados.0.smithi084.stdout:2225: finishing write tid 1 to smithi08477203-12 2024-09-20T22:50:51.244 INFO:tasks.rados.rados.0.smithi084.stdout:2225: finishing write tid 2 to smithi08477203-12 2024-09-20T22:50:51.244 INFO:tasks.rados.rados.0.smithi084.stdout:2225: finishing write tid 3 to smithi08477203-12 2024-09-20T22:50:51.244 INFO:tasks.rados.rados.0.smithi084.stdout:2225: finishing write tid 4 to smithi08477203-12 2024-09-20T22:50:51.244 INFO:tasks.rados.rados.0.smithi084.stdout:2225: finishing write tid 5 to smithi08477203-12 2024-09-20T22:50:51.244 INFO:tasks.rados.rados.0.smithi084.stdout:2225: oid 12 updating version 0 to 820 2024-09-20T22:50:51.244 INFO:tasks.rados.rados.0.smithi084.stdout:2225: oid 12 updating version 820 to 822 2024-09-20T22:50:51.244 INFO:tasks.rados.rados.0.smithi084.stdout:2225: oid 12 version 822 is already newer than 821 2024-09-20T22:50:51.244 INFO:tasks.rados.rados.0.smithi084.stdout:2225: oid 12 version 822 is already newer than 819 2024-09-20T22:50:51.244 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 12 v 822 (ObjNum 772 snap 248 seq_num 772) dirty exists 2024-09-20T22:50:51.244 INFO:tasks.rados.rados.0.smithi084.stdout:2225: left oid 12 (ObjNum 772 snap 248 seq_num 772) 2024-09-20T22:50:51.245 INFO:tasks.rados.rados.0.smithi084.stdout:2226: finishing copy_from racing read to smithi08477203-20 2024-09-20T22:50:51.245 INFO:tasks.rados.rados.0.smithi084.stdout:2226: finishing copy_from to smithi08477203-20 2024-09-20T22:50:51.245 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 20 v 731 (ObjNum 470 snap 130 seq_num 470) dirty exists 2024-09-20T22:50:51.246 INFO:tasks.rados.rados.0.smithi084.stdout:2224: done (3 left) 2024-09-20T22:50:51.246 INFO:tasks.rados.rados.0.smithi084.stdout:2225: done (2 left) 2024-09-20T22:50:51.246 INFO:tasks.rados.rados.0.smithi084.stdout:2226: done (1 left) 2024-09-20T22:50:51.246 INFO:tasks.rados.rados.0.smithi084.stdout:2227: done (0 left) 2024-09-20T22:50:51.246 INFO:tasks.rados.rados.0.smithi084.stdout:2228: read oid 5 snap 165 2024-09-20T22:50:51.246 INFO:tasks.rados.rados.0.smithi084.stdout:2228: expect deleted 2024-09-20T22:50:51.246 INFO:tasks.rados.rados.0.smithi084.stdout:2229: rollback oid 31 current snap is 248 2024-09-20T22:50:51.246 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 31 to 239 2024-09-20T22:50:51.246 INFO:tasks.rados.rados.0.smithi084.stdout:2230: rollback oid 22 current snap is 248 2024-09-20T22:50:51.246 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 22 to 235 2024-09-20T22:50:51.246 INFO:tasks.rados.rados.0.smithi084.stdout:2231: rollback oid 24 current snap is 248 2024-09-20T22:50:51.247 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 24 to 221 2024-09-20T22:50:51.247 INFO:tasks.rados.rados.0.smithi084.stdout:2232: write (excl) oid 17 current snap is 248 2024-09-20T22:50:51.247 INFO:tasks.rados.rados.0.smithi084.stdout:2232: seq_num 774 ranges {608618=794379,2109651=780254} 2024-09-20T22:50:51.252 INFO:tasks.rados.rados.0.smithi084.stdout:2232: writing smithi08477203-17 from 608618 to 1402997 tid 1 2024-09-20T22:50:51.263 INFO:tasks.rados.rados.0.smithi084.stdout:2232: writing smithi08477203-17 from 2109651 to 2889905 tid 2 2024-09-20T22:50:51.263 INFO:tasks.rados.rados.0.smithi084.stdout:2233: rollback oid 37 current snap is 248 2024-09-20T22:50:51.263 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 37 to 244 2024-09-20T22:50:51.263 INFO:tasks.rados.rados.0.smithi084.stdout:2234: write oid 42 current snap is 248 2024-09-20T22:50:51.263 INFO:tasks.rados.rados.0.smithi084.stdout:2234: seq_num 775 ranges {625425=469628,1768133=640265} 2024-09-20T22:50:51.269 INFO:tasks.rados.rados.0.smithi084.stdout:2234: writing smithi08477203-42 from 625425 to 1095053 tid 1 2024-09-20T22:50:51.275 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:50:51.279 INFO:tasks.rados.rados.0.smithi084.stdout:2234: writing smithi08477203-42 from 1768133 to 2408398 tid 2 2024-09-20T22:50:51.279 INFO:tasks.rados.rados.0.smithi084.stdout:2235: read oid 9 snap -1 2024-09-20T22:50:51.279 INFO:tasks.rados.rados.0.smithi084.stdout:2235: expect deleted 2024-09-20T22:50:51.279 INFO:tasks.rados.rados.0.smithi084.stdout:2236: copy_from oid 1 from oid 49 current snap is 248 2024-09-20T22:50:51.279 INFO:tasks.rados.rados.0.smithi084.stdout:2230: finishing rollback tid 0 to smithi08477203-22 2024-09-20T22:50:51.279 INFO:tasks.rados.rados.0.smithi084.stdout:2230: finishing rollback tid 1 to smithi08477203-22 2024-09-20T22:50:51.279 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 22 v 731 (ObjNum 33 snap 0 seq_num 2091711168) dirty dne 2024-09-20T22:50:51.279 INFO:tasks.rados.rados.0.smithi084.stdout:2229: finishing rollback tid 1 to smithi08477203-31 2024-09-20T22:50:51.279 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 31 v 760 (ObjNum 0 snap 0 seq_num 304) dirty dne 2024-09-20T22:50:51.280 INFO:tasks.rados.rados.0.smithi084.stdout:2231: finishing rollback tid 0 to smithi08477203-24 2024-09-20T22:50:51.280 INFO:tasks.rados.rados.0.smithi084.stdout:2228: done (8 left) 2024-09-20T22:50:51.282 INFO:tasks.rados.rados.0.smithi084.stdout:2229: done (7 left) 2024-09-20T22:50:51.282 INFO:tasks.rados.rados.0.smithi084.stdout:2230: done (6 left) 2024-09-20T22:50:51.282 INFO:tasks.rados.rados.0.smithi084.stdout:2231: finishing rollback tid 1 to smithi08477203-24 2024-09-20T22:50:51.282 INFO:tasks.rados.rados.0.smithi084.stdout:2231: finishing rollback tid 2 to smithi08477203-24 2024-09-20T22:50:51.282 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 24 v 735 (ObjNum 711 snap 220 seq_num 711) dirty exists 2024-09-20T22:50:51.282 INFO:tasks.rados.rados.0.smithi084.stdout:2232: finishing write tid 1 to smithi08477203-17 2024-09-20T22:50:51.282 INFO:tasks.rados.rados.0.smithi084.stdout:2233: finishing rollback tid 1 to smithi08477203-37 2024-09-20T22:50:51.282 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 37 v 842 (ObjNum 33 snap 0 seq_num 2085656408) dirty dne 2024-09-20T22:50:51.282 INFO:tasks.rados.rados.0.smithi084.stdout:2232: finishing write tid 2 to smithi08477203-17 2024-09-20T22:50:51.282 INFO:tasks.rados.rados.0.smithi084.stdout:2232: finishing write tid 3 to smithi08477203-17 2024-09-20T22:50:51.282 INFO:tasks.rados.rados.0.smithi084.stdout:2232: finishing write tid 4 to smithi08477203-17 2024-09-20T22:50:51.283 INFO:tasks.rados.rados.0.smithi084.stdout:2232: oid 17 updating version 0 to 758 2024-09-20T22:50:51.283 INFO:tasks.rados.rados.0.smithi084.stdout:2232: oid 17 version 758 is already newer than 757 2024-09-20T22:50:51.283 INFO:tasks.rados.rados.0.smithi084.stdout:2232: oid 17 updating version 758 to 759 2024-09-20T22:50:51.283 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 17 v 759 (ObjNum 774 snap 248 seq_num 774) dirty exists 2024-09-20T22:50:51.283 INFO:tasks.rados.rados.0.smithi084.stdout:2232: left oid 17 (ObjNum 774 snap 248 seq_num 774) 2024-09-20T22:50:51.283 INFO:tasks.rados.rados.0.smithi084.stdout:2234: finishing write tid 1 to smithi08477203-42 2024-09-20T22:50:51.283 INFO:tasks.rados.rados.0.smithi084.stdout:2237: read oid 11 snap -1 2024-09-20T22:50:51.283 INFO:tasks.rados.rados.0.smithi084.stdout:2237: expect (ObjNum 556 snap 169 seq_num 556) 2024-09-20T22:50:51.283 INFO:tasks.rados.rados.0.smithi084.stdout:2231: done (6 left) 2024-09-20T22:50:51.283 INFO:tasks.rados.rados.0.smithi084.stdout:2232: done (5 left) 2024-09-20T22:50:51.283 INFO:tasks.rados.rados.0.smithi084.stdout:2233: done (4 left) 2024-09-20T22:50:51.283 INFO:tasks.rados.rados.0.smithi084.stdout:2238: snap_create 2024-09-20T22:50:51.284 INFO:tasks.rados.rados.0.smithi084.stdout:2237: expect (ObjNum 556 snap 169 seq_num 556) 2024-09-20T22:50:51.318 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:50:51.342 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:51.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:51.343 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:50:51.444 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:50:51.467 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:51.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:51.468 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:50:51.521 INFO:tasks.rados.rados.0.smithi084.stdout:2235: done (4 left) 2024-09-20T22:50:51.521 INFO:tasks.rados.rados.0.smithi084.stdout:2237: done (3 left) 2024-09-20T22:50:51.521 INFO:tasks.rados.rados.0.smithi084.stdout:2238: done (2 left) 2024-09-20T22:50:51.521 INFO:tasks.rados.rados.0.smithi084.stdout:2239: snap_remove snap 244 2024-09-20T22:50:51.531 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:51 smithi084 ceph-mon[40024]: pgmap v756: 158 pgs: 1 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 155 active+clean; 501 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 13 MiB/s rd, 2.3 MiB/s wr, 16 op/s 2024-09-20T22:50:51.531 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:51 smithi084 ceph-mon[40024]: osdmap e570: 8 total, 8 up, 8 in 2024-09-20T22:50:51.531 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:51 smithi084 ceph-mon[29774]: pgmap v756: 158 pgs: 1 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 155 active+clean; 501 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 13 MiB/s rd, 2.3 MiB/s wr, 16 op/s 2024-09-20T22:50:51.531 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:51 smithi084 ceph-mon[29774]: osdmap e570: 8 total, 8 up, 8 in 2024-09-20T22:50:51.555 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:50:51.568 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:50:51.589 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:51.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:51.590 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:50:51.691 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:50:51.713 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:51.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:51.714 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:50:51.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:51 smithi154 ceph-mon[33264]: pgmap v756: 158 pgs: 1 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 155 active+clean; 501 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 13 MiB/s rd, 2.3 MiB/s wr, 16 op/s 2024-09-20T22:50:51.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:51 smithi154 ceph-mon[33264]: osdmap e570: 8 total, 8 up, 8 in 2024-09-20T22:50:51.757 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:50:51.780 INFO:teuthology.orchestra.run.smithi084.stderr:no valid command found; 10 closest matches: 2024-09-20T22:50:51.780 INFO:teuthology.orchestra.run.smithi084.stderr:0 2024-09-20T22:50:51.780 INFO:teuthology.orchestra.run.smithi084.stderr:1 2024-09-20T22:50:51.780 INFO:teuthology.orchestra.run.smithi084.stderr:2 2024-09-20T22:50:51.780 INFO:teuthology.orchestra.run.smithi084.stderr:abort 2024-09-20T22:50:51.780 INFO:teuthology.orchestra.run.smithi084.stderr:assert 2024-09-20T22:50:51.780 INFO:teuthology.orchestra.run.smithi084.stderr:bench [] [] [] [] 2024-09-20T22:50:51.780 INFO:teuthology.orchestra.run.smithi084.stderr:bluefs debug_inject_read_zeros 2024-09-20T22:50:51.781 INFO:teuthology.orchestra.run.smithi084.stderr:bluefs files list 2024-09-20T22:50:51.781 INFO:teuthology.orchestra.run.smithi084.stderr:bluefs stats 2024-09-20T22:50:51.781 INFO:teuthology.orchestra.run.smithi084.stderr:bluestore allocator dump block 2024-09-20T22:50:51.781 INFO:teuthology.orchestra.run.smithi084.stderr:Error EINVAL: invalid command 2024-09-20T22:50:51.814 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:50:51.836 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:51.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:51.837 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:50:51.937 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:50:51.959 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:51.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:51.960 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:50:52.061 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:50:52.083 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:52.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:52.084 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:50:52.184 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:50:52.206 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:52.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:52.207 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:50:52.307 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:50:52.329 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:52.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:52.330 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:50:52.355 INFO:tasks.rados.rados.0.smithi084.stdout:2239: done (2 left) 2024-09-20T22:50:52.355 INFO:tasks.rados.rados.0.smithi084.stdout:2240: write (excl) oid 4 current snap is 249 2024-09-20T22:50:52.355 INFO:tasks.rados.rados.0.smithi084.stdout:2240: seq_num 776 ranges {642232=661230,1942968=783923} 2024-09-20T22:50:52.362 INFO:tasks.rados.rados.0.smithi084.stdout:2240: writing smithi08477203-4 from 642232 to 1303462 tid 1 2024-09-20T22:50:52.364 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:52 smithi084 ceph-mon[40024]: osdmap e571: 8 total, 8 up, 8 in 2024-09-20T22:50:52.364 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:52 smithi084 ceph-mon[29774]: osdmap e571: 8 total, 8 up, 8 in 2024-09-20T22:50:52.371 INFO:tasks.rados.rados.0.smithi084.stdout:2240: writing smithi08477203-4 from 1942968 to 2726891 tid 2 2024-09-20T22:50:52.371 INFO:tasks.rados.rados.0.smithi084.stdout:2241: rollback oid 50 current snap is 249 2024-09-20T22:50:52.371 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 50 to 216 2024-09-20T22:50:52.371 INFO:tasks.rados.rados.0.smithi084.stdout:2234: finishing write tid 2 to smithi08477203-42 2024-09-20T22:50:52.371 INFO:tasks.rados.rados.0.smithi084.stdout:2234: finishing write tid 3 to smithi08477203-42 2024-09-20T22:50:52.371 INFO:tasks.rados.rados.0.smithi084.stdout:2234: finishing write tid 4 to smithi08477203-42 2024-09-20T22:50:52.371 INFO:tasks.rados.rados.0.smithi084.stdout:2234: oid 42 updating version 0 to 1212 2024-09-20T22:50:52.371 INFO:tasks.rados.rados.0.smithi084.stdout:2234: oid 42 version 1212 is already newer than 1211 2024-09-20T22:50:52.371 INFO:tasks.rados.rados.0.smithi084.stdout:2234: oid 42 updating version 1212 to 1213 2024-09-20T22:50:52.371 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 42 v 1213 (ObjNum 775 snap 248 seq_num 775) dirty exists 2024-09-20T22:50:52.371 INFO:tasks.rados.rados.0.smithi084.stdout:2234: left oid 42 (ObjNum 775 snap 248 seq_num 775) 2024-09-20T22:50:52.371 INFO:tasks.rados.rados.0.smithi084.stdout:2236: finishing copy_from racing read to smithi08477203-1 2024-09-20T22:50:52.371 INFO:tasks.rados.rados.0.smithi084.stdout:2234: done (3 left) 2024-09-20T22:50:52.371 INFO:tasks.rados.rados.0.smithi084.stdout:2242: write oid 35 current snap is 249 2024-09-20T22:50:52.372 INFO:tasks.rados.rados.0.smithi084.stdout:2236: finishing copy_from to smithi08477203-1 2024-09-20T22:50:52.372 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 1 v 731 (ObjNum 470 snap 130 seq_num 470) dirty exists 2024-09-20T22:50:52.372 INFO:tasks.rados.rados.0.smithi084.stdout:2242: seq_num 777 ranges {659039=736479,2001450=643934} 2024-09-20T22:50:52.377 INFO:tasks.rados.rados.0.smithi084.stdout:2242: writing smithi08477203-35 from 659039 to 1395518 tid 1 2024-09-20T22:50:52.385 INFO:tasks.rados.rados.0.smithi084.stdout:2242: writing smithi08477203-35 from 2001450 to 2645384 tid 2 2024-09-20T22:50:52.385 INFO:tasks.rados.rados.0.smithi084.stdout:2236: done (3 left) 2024-09-20T22:50:52.385 INFO:tasks.rados.rados.0.smithi084.stdout:2243: copy_from oid 21 from oid 42 current snap is 249 2024-09-20T22:50:52.385 INFO:tasks.rados.rados.0.smithi084.stdout:2240: finishing write tid 1 to smithi08477203-4 2024-09-20T22:50:52.386 INFO:tasks.rados.rados.0.smithi084.stdout:2241: finishing rollback tid 0 to smithi08477203-50 2024-09-20T22:50:52.386 INFO:tasks.rados.rados.0.smithi084.stdout:2241: finishing rollback tid 1 to smithi08477203-50 2024-09-20T22:50:52.386 INFO:tasks.rados.rados.0.smithi084.stdout:2241: finishing rollback tid 2 to smithi08477203-50 2024-09-20T22:50:52.386 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 50 v 738 (ObjNum 660 snap 199 seq_num 660) dirty exists 2024-09-20T22:50:52.386 INFO:tasks.rados.rados.0.smithi084.stdout:2241: done (3 left) 2024-09-20T22:50:52.386 INFO:tasks.rados.rados.0.smithi084.stdout:2244: copy_from oid 46 from oid 30 current snap is 249 2024-09-20T22:50:52.386 INFO:tasks.rados.rados.0.smithi084.stdout:2240: finishing write tid 2 to smithi08477203-4 2024-09-20T22:50:52.386 INFO:tasks.rados.rados.0.smithi084.stdout:2240: finishing write tid 3 to smithi08477203-4 2024-09-20T22:50:52.386 INFO:tasks.rados.rados.0.smithi084.stdout:2240: finishing write tid 4 to smithi08477203-4 2024-09-20T22:50:52.386 INFO:tasks.rados.rados.0.smithi084.stdout:2240: oid 4 updating version 0 to 1574 2024-09-20T22:50:52.386 INFO:tasks.rados.rados.0.smithi084.stdout:2240: oid 4 updating version 1574 to 1575 2024-09-20T22:50:52.386 INFO:tasks.rados.rados.0.smithi084.stdout:2240: oid 4 version 1575 is already newer than 1573 2024-09-20T22:50:52.386 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 4 v 1575 (ObjNum 776 snap 249 seq_num 776) dirty exists 2024-09-20T22:50:52.386 INFO:tasks.rados.rados.0.smithi084.stdout:2240: left oid 4 (ObjNum 776 snap 249 seq_num 776) 2024-09-20T22:50:52.386 INFO:tasks.rados.rados.0.smithi084.stdout:2240: done (3 left) 2024-09-20T22:50:52.386 INFO:tasks.rados.rados.0.smithi084.stdout:2245: read oid 4 snap -1 2024-09-20T22:50:52.387 INFO:tasks.rados.rados.0.smithi084.stdout:2245: expect (ObjNum 776 snap 249 seq_num 776) 2024-09-20T22:50:52.387 INFO:tasks.rados.rados.0.smithi084.stdout:2246: rollback oid 44 current snap is 249 2024-09-20T22:50:52.387 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 44 to 248 2024-09-20T22:50:52.387 INFO:tasks.rados.rados.0.smithi084.stdout:2242: finishing write tid 1 to smithi08477203-35 2024-09-20T22:50:52.387 INFO:tasks.rados.rados.0.smithi084.stdout:2247: rollback oid 36 current snap is 249 2024-09-20T22:50:52.387 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 36 to 204 2024-09-20T22:50:52.387 INFO:tasks.rados.rados.0.smithi084.stdout:2248: snap_create 2024-09-20T22:50:52.388 INFO:tasks.rados.rados.0.smithi084.stdout:2247: finishing rollback tid 0 to smithi08477203-36 2024-09-20T22:50:52.388 INFO:tasks.rados.rados.0.smithi084.stdout:2242: finishing write tid 2 to smithi08477203-35 2024-09-20T22:50:52.388 INFO:tasks.rados.rados.0.smithi084.stdout:2242: finishing write tid 3 to smithi08477203-35 2024-09-20T22:50:52.389 INFO:tasks.rados.rados.0.smithi084.stdout:2242: finishing write tid 4 to smithi08477203-35 2024-09-20T22:50:52.389 INFO:tasks.rados.rados.0.smithi084.stdout:2242: oid 35 updating version 0 to 736 2024-09-20T22:50:52.389 INFO:tasks.rados.rados.0.smithi084.stdout:2242: oid 35 updating version 736 to 738 2024-09-20T22:50:52.389 INFO:tasks.rados.rados.0.smithi084.stdout:2242: oid 35 version 738 is already newer than 737 2024-09-20T22:50:52.389 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 35 v 738 (ObjNum 777 snap 249 seq_num 777) dirty exists 2024-09-20T22:50:52.389 INFO:tasks.rados.rados.0.smithi084.stdout:2242: left oid 35 (ObjNum 777 snap 249 seq_num 777) 2024-09-20T22:50:52.392 INFO:tasks.rados.rados.0.smithi084.stdout:2247: finishing rollback tid 1 to smithi08477203-36 2024-09-20T22:50:52.394 INFO:tasks.rados.rados.0.smithi084.stdout:2244: finishing copy_from racing read to smithi08477203-46 2024-09-20T22:50:52.397 INFO:tasks.rados.rados.0.smithi084.stdout:2245: expect (ObjNum 776 snap 249 seq_num 776) 2024-09-20T22:50:52.431 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:50:52.453 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:52.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:52.454 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:50:52.540 INFO:tasks.rados.rados.0.smithi084.stdout:2247: finishing rollback tid 2 to smithi08477203-36 2024-09-20T22:50:52.540 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 36 v 1216 (ObjNum 440 snap 115 seq_num 440) dirty exists 2024-09-20T22:50:52.540 INFO:tasks.rados.rados.0.smithi084.stdout:2246: finishing rollback tid 0 to smithi08477203-44 2024-09-20T22:50:52.540 INFO:tasks.rados.rados.0.smithi084.stdout:2246: finishing rollback tid 1 to smithi08477203-44 2024-09-20T22:50:52.540 INFO:tasks.rados.rados.0.smithi084.stdout:2244: finishing copy_from to smithi08477203-46 2024-09-20T22:50:52.541 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 46 v 801 (ObjNum 750 snap 238 seq_num 750) dirty exists 2024-09-20T22:50:52.541 INFO:tasks.rados.rados.0.smithi084.stdout:2246: finishing rollback tid 2 to smithi08477203-44 2024-09-20T22:50:52.541 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 44 v 1579 (ObjNum 758 snap 241 seq_num 758) dirty exists 2024-09-20T22:50:52.541 INFO:tasks.rados.rados.0.smithi084.stdout:2243: finishing copy_from racing read to smithi08477203-21 2024-09-20T22:50:52.541 INFO:tasks.rados.rados.0.smithi084.stdout:2243: finishing copy_from to smithi08477203-21 2024-09-20T22:50:52.541 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 21 v 1213 (ObjNum 775 snap 248 seq_num 775) dirty exists 2024-09-20T22:50:52.555 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:50:52.576 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:52.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:52.577 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:50:52.643 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-20T22:50:52.643 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2024-09-20T22:50:52.643 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2024-09-20T22:50:52.643 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [ 2024-09-20T22:50:52.643 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.643 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:901 7.e 7:7c62ced4:::smithi08477203-17:head [copy-from ver 48] snapc 1e=[] ondisk+write+known_if_redirected+supports_pool_eio e156)", 2024-09-20T22:50:52.644 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:43:55.624762+0000", 2024-09-20T22:50:52.644 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 417.01026222500002, 2024-09-20T22:50:52.644 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.021835856000000001, 2024-09-20T22:50:52.644 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:50:52.644 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:50:52.644 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:50:52.644 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:50:52.644 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:50:52.644 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:50:52.644 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 901 2024-09-20T22:50:52.644 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.644 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:50:52.644 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.644 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:50:52.644 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.624762+0000", 2024-09-20T22:50:52.645 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:50:52.645 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.645 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.645 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:50:52.645 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.624762+0000", 2024-09-20T22:50:52.645 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:50:52.645 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.645 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.645 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:50:52.645 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.624765+0000", 2024-09-20T22:50:52.645 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.57e-06 2024-09-20T22:50:52.645 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.645 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.645 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:50:52.645 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.624776+0000", 2024-09-20T22:50:52.646 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.078e-05 2024-09-20T22:50:52.646 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.646 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.646 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:50:52.646 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.624803+0000", 2024-09-20T22:50:52.646 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.6769999999999999e-05 2024-09-20T22:50:52.646 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.646 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.646 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:50:52.646 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.624815+0000", 2024-09-20T22:50:52.646 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2459e-05 2024-09-20T22:50:52.646 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.646 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.646 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:50:52.647 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.624936+0000", 2024-09-20T22:50:52.647 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000120467 2024-09-20T22:50:52.647 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.647 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.647 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:50:52.647 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.625046+0000", 2024-09-20T22:50:52.647 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000110222 2024-09-20T22:50:52.647 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.647 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.647 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 0,5", 2024-09-20T22:50:52.647 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.635547+0000", 2024-09-20T22:50:52.647 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010501765999999999 2024-09-20T22:50:52.647 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.647 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.647 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:50:52.647 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.644332+0000", 2024-09-20T22:50:52.648 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0087847729999999992 2024-09-20T22:50:52.648 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.648 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.648 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:50:52.648 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.646168+0000", 2024-09-20T22:50:52.648 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0018360010000000001 2024-09-20T22:50:52.648 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.648 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.648 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:50:52.648 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.646512+0000", 2024-09-20T22:50:52.648 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000343269 2024-09-20T22:50:52.648 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.648 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.649 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:50:52.649 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.646550+0000", 2024-09-20T22:50:52.649 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.8383000000000002e-05 2024-09-20T22:50:52.649 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.649 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.649 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:50:52.649 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.646598+0000", 2024-09-20T22:50:52.649 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.8396000000000002e-05 2024-09-20T22:50:52.649 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:50:52.649 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:50:52.649 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:50:52.649 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.649 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.649 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:902 7.e 7:7c62ced4:::smithi08477203-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e156)", 2024-09-20T22:50:52.650 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:43:55.624853+0000", 2024-09-20T22:50:52.650 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 417.01017214699999, 2024-09-20T22:50:52.650 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.015323603999999999, 2024-09-20T22:50:52.650 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:50:52.650 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:50:52.650 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-20T22:50:52.650 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:50:52.650 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:50:52.650 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:50:52.650 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 902 2024-09-20T22:50:52.650 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.650 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:50:52.651 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.651 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:50:52.651 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.624853+0000", 2024-09-20T22:50:52.651 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:50:52.651 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.651 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.651 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:50:52.651 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.624853+0000", 2024-09-20T22:50:52.651 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:50:52.651 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.651 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.651 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:50:52.651 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.624853+0000", 2024-09-20T22:50:52.652 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.1399999999999997e-07 2024-09-20T22:50:52.652 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.652 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.652 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:50:52.652 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.624859+0000", 2024-09-20T22:50:52.652 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.9610000000000002e-06 2024-09-20T22:50:52.652 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.652 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.652 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:50:52.652 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.624887+0000", 2024-09-20T22:50:52.652 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.7597e-05 2024-09-20T22:50:52.652 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.652 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.652 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:50:52.652 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.624894+0000", 2024-09-20T22:50:52.653 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.469e-06 2024-09-20T22:50:52.653 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.653 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.653 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:50:52.653 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.625219+0000", 2024-09-20T22:50:52.653 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00032538499999999998 2024-09-20T22:50:52.653 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.653 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.653 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for blocked object", 2024-09-20T22:50:52.653 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.625258+0000", 2024-09-20T22:50:52.653 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.8467000000000003e-05 2024-09-20T22:50:52.653 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.653 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.653 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:50:52.653 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.640050+0000", 2024-09-20T22:50:52.653 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0147923 2024-09-20T22:50:52.654 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.654 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.654 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:50:52.654 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.640112+0000", 2024-09-20T22:50:52.654 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.1723000000000005e-05 2024-09-20T22:50:52.654 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.654 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.654 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:50:52.654 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.640176+0000", 2024-09-20T22:50:52.654 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.4188000000000003e-05 2024-09-20T22:50:52.654 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:50:52.654 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:50:52.654 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:50:52.654 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.655 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.655 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24652.0:1420 7.19 e189/139 7:9cc0b7e2:::smithi08477203-46:head v 189'99, mlcod=174'95)", 2024-09-20T22:50:52.655 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:44:28.150258+0000", 2024-09-20T22:50:52.655 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 384.48476645099998, 2024-09-20T22:50:52.655 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014180626, 2024-09-20T22:50:52.655 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:50:52.655 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:50:52.655 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:50:52.655 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:50:52.655 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.655 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:50:52.655 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.150258+0000", 2024-09-20T22:50:52.655 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:50:52.656 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.656 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.656 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:50:52.656 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.150258+0000", 2024-09-20T22:50:52.656 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:50:52.656 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.656 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.656 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:50:52.656 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.150259+0000", 2024-09-20T22:50:52.656 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.99e-07 2024-09-20T22:50:52.656 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.656 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.656 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:50:52.656 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.150674+0000", 2024-09-20T22:50:52.657 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00041528000000000002 2024-09-20T22:50:52.657 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.657 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.657 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:50:52.657 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.150688+0000", 2024-09-20T22:50:52.657 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3399e-05 2024-09-20T22:50:52.657 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.657 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.657 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:50:52.657 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.150699+0000", 2024-09-20T22:50:52.657 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1022999999999999e-05 2024-09-20T22:50:52.657 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.657 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.658 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:50:52.658 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.150772+0000", 2024-09-20T22:50:52.658 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.3455999999999996e-05 2024-09-20T22:50:52.658 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.658 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.658 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:50:52.658 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.150798+0000", 2024-09-20T22:50:52.658 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.5777000000000001e-05 2024-09-20T22:50:52.658 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.658 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.658 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:50:52.658 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.164390+0000", 2024-09-20T22:50:52.658 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013591789999999999 2024-09-20T22:50:52.658 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.659 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.659 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:50:52.659 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.164439+0000", 2024-09-20T22:50:52.659 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.9202000000000001e-05 2024-09-20T22:50:52.659 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:50:52.659 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:50:52.659 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:50:52.659 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.659 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.659 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24652.0:1421 7.19 e189/139 7:9cc0b7e2:::smithi08477203-46:head v 189'100, mlcod=174'95)", 2024-09-20T22:50:52.659 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:44:28.156805+0000", 2024-09-20T22:50:52.659 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 384.47821959300001, 2024-09-20T22:50:52.659 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013180661999999999, 2024-09-20T22:50:52.660 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:50:52.660 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:50:52.660 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:50:52.660 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:50:52.660 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.660 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:50:52.660 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.156805+0000", 2024-09-20T22:50:52.660 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:50:52.660 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.660 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.660 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:50:52.660 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.156805+0000", 2024-09-20T22:50:52.660 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:50:52.660 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.660 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.661 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:50:52.661 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.156806+0000", 2024-09-20T22:50:52.661 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.074e-06 2024-09-20T22:50:52.661 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.661 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.661 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:50:52.661 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.157126+0000", 2024-09-20T22:50:52.661 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00031991399999999997 2024-09-20T22:50:52.661 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.661 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.661 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:50:52.661 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.157136+0000", 2024-09-20T22:50:52.661 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.4499999999999993e-06 2024-09-20T22:50:52.661 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.662 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.662 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:50:52.662 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.157143+0000", 2024-09-20T22:50:52.662 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.7649999999999993e-06 2024-09-20T22:50:52.662 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.662 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.662 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:50:52.662 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.157188+0000", 2024-09-20T22:50:52.662 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.4885000000000001e-05 2024-09-20T22:50:52.662 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.662 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.662 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:50:52.662 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.157208+0000", 2024-09-20T22:50:52.662 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.0015000000000001e-05 2024-09-20T22:50:52.663 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.663 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.663 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:50:52.663 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.169936+0000", 2024-09-20T22:50:52.663 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.012728204 2024-09-20T22:50:52.663 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.663 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.663 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:50:52.663 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.169986+0000", 2024-09-20T22:50:52.663 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.9354999999999998e-05 2024-09-20T22:50:52.663 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:50:52.663 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:50:52.663 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:50:52.663 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.663 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.664 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24652.0:1654 7.19 e206/139 7:9970b578:::smithi08477203-15:head v 206'114, mlcod=205'111)", 2024-09-20T22:50:52.664 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:44:45.167994+0000", 2024-09-20T22:50:52.664 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 367.46703104300002, 2024-09-20T22:50:52.664 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.01345262, 2024-09-20T22:50:52.664 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:50:52.664 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:50:52.664 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:50:52.664 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:50:52.664 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.664 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:50:52.664 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:45.167994+0000", 2024-09-20T22:50:52.664 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:50:52.664 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.664 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.665 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:50:52.665 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:45.167994+0000", 2024-09-20T22:50:52.665 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:50:52.665 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.665 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.665 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:50:52.665 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:45.167996+0000", 2024-09-20T22:50:52.665 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.5100000000000001e-06 2024-09-20T22:50:52.665 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.665 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.665 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:50:52.665 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:45.175169+0000", 2024-09-20T22:50:52.665 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0071732740000000003 2024-09-20T22:50:52.665 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.665 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.666 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:50:52.666 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:45.175208+0000", 2024-09-20T22:50:52.666 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.8597000000000003e-05 2024-09-20T22:50:52.666 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.666 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.666 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:50:52.666 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:45.175223+0000", 2024-09-20T22:50:52.666 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.4627000000000001e-05 2024-09-20T22:50:52.666 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.666 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.666 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:50:52.666 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:45.175304+0000", 2024-09-20T22:50:52.666 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.1111000000000003e-05 2024-09-20T22:50:52.666 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.666 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.667 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:50:52.667 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:45.175336+0000", 2024-09-20T22:50:52.667 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.1945e-05 2024-09-20T22:50:52.667 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.667 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.667 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:50:52.667 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:45.181392+0000", 2024-09-20T22:50:52.667 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0060566630000000003 2024-09-20T22:50:52.667 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.667 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.667 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:50:52.667 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:45.181446+0000", 2024-09-20T22:50:52.667 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.3893000000000001e-05 2024-09-20T22:50:52.668 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:50:52.668 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:50:52.668 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:50:52.668 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.668 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.668 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24652.0:1799 7.17 e229/139 7:ea720b85:::smithi08477203-26:head v 229'168, mlcod=222'164)", 2024-09-20T22:50:52.668 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:45:08.171867+0000", 2024-09-20T22:50:52.668 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 344.463157825, 2024-09-20T22:50:52.668 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014913454, 2024-09-20T22:50:52.668 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:50:52.668 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:50:52.668 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:50:52.668 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:50:52.668 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.668 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:50:52.669 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.171867+0000", 2024-09-20T22:50:52.669 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:50:52.669 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.669 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.669 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:50:52.669 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.171867+0000", 2024-09-20T22:50:52.669 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:50:52.669 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.669 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.669 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:50:52.669 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.171867+0000", 2024-09-20T22:50:52.669 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.3799999999999999e-07 2024-09-20T22:50:52.669 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.669 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.669 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:50:52.670 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.173016+0000", 2024-09-20T22:50:52.670 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0011485740000000001 2024-09-20T22:50:52.670 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.670 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.670 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:50:52.670 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.173024+0000", 2024-09-20T22:50:52.670 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.4780000000000008e-06 2024-09-20T22:50:52.670 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.670 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.670 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:50:52.670 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.173034+0000", 2024-09-20T22:50:52.670 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.7860000000000008e-06 2024-09-20T22:50:52.670 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.670 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.671 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:50:52.671 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.173098+0000", 2024-09-20T22:50:52.671 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.4405000000000004e-05 2024-09-20T22:50:52.671 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.671 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.671 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:50:52.671 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.173125+0000", 2024-09-20T22:50:52.671 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.6950000000000001e-05 2024-09-20T22:50:52.671 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.671 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.671 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:50:52.671 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.186711+0000", 2024-09-20T22:50:52.671 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.01358608 2024-09-20T22:50:52.672 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.672 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.672 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:50:52.672 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.186780+0000", 2024-09-20T22:50:52.672 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.8942999999999994e-05 2024-09-20T22:50:52.672 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:50:52.672 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:50:52.672 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:50:52.672 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.672 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.672 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:2322 7.8 7:12e9c8db:::smithi08477203-11:head [append 0~0] snapc 55=[55,53,52,51,50,4b,47,3e] ondisk+write+known_if_redirected+supports_pool_eio e258)", 2024-09-20T22:50:52.672 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:45:37.176152+0000", 2024-09-20T22:50:52.672 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 315.45887303500001, 2024-09-20T22:50:52.673 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013033472000000001, 2024-09-20T22:50:52.673 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:50:52.673 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:50:52.673 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:50:52.673 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:50:52.673 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:50:52.673 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:50:52.673 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 2322 2024-09-20T22:50:52.673 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.673 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:50:52.673 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.673 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:50:52.673 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.176152+0000", 2024-09-20T22:50:52.673 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:50:52.674 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.674 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.674 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:50:52.674 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.176152+0000", 2024-09-20T22:50:52.674 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:50:52.674 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.674 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.674 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:50:52.674 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.176152+0000", 2024-09-20T22:50:52.674 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.8899999999999999e-07 2024-09-20T22:50:52.674 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.674 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.674 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:50:52.674 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.176156+0000", 2024-09-20T22:50:52.675 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.2669999999999998e-06 2024-09-20T22:50:52.675 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.675 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.675 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:50:52.675 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.176162+0000", 2024-09-20T22:50:52.675 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.1589999999999998e-06 2024-09-20T22:50:52.675 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.675 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.675 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:50:52.675 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.176167+0000", 2024-09-20T22:50:52.675 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.3040000000000004e-06 2024-09-20T22:50:52.675 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.675 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.675 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:50:52.676 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.176212+0000", 2024-09-20T22:50:52.676 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.5448999999999998e-05 2024-09-20T22:50:52.676 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.676 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.676 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting_for_map not empty", 2024-09-20T22:50:52.676 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.176218+0000", 2024-09-20T22:50:52.676 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.5520000000000004e-06 2024-09-20T22:50:52.676 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.676 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.676 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:50:52.676 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.182721+0000", 2024-09-20T22:50:52.676 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0065028550000000001 2024-09-20T22:50:52.676 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.676 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.676 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:50:52.677 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.182787+0000", 2024-09-20T22:50:52.677 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.5995999999999996e-05 2024-09-20T22:50:52.677 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.677 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.677 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 3,7", 2024-09-20T22:50:52.677 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.183002+0000", 2024-09-20T22:50:52.677 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00021516699999999999 2024-09-20T22:50:52.677 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.677 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.677 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:50:52.677 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.185137+0000", 2024-09-20T22:50:52.677 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0021350570000000001 2024-09-20T22:50:52.677 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.677 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.678 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:50:52.678 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.185875+0000", 2024-09-20T22:50:52.678 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00073794799999999999 2024-09-20T22:50:52.678 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.678 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.678 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:50:52.678 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.188620+0000", 2024-09-20T22:50:52.678 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0027446229999999999 2024-09-20T22:50:52.678 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.678 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.678 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:50:52.678 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.189088+0000", 2024-09-20T22:50:52.678 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00046794499999999999 2024-09-20T22:50:52.678 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.679 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.679 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:50:52.679 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.189185+0000", 2024-09-20T22:50:52.679 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.7560999999999996e-05 2024-09-20T22:50:52.679 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:50:52.679 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:50:52.679 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:50:52.679 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.679 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.679 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24652.0:2318 7.5 e258/137 7:a160b5d4:::smithi08477203-16:head v 258'43, mlcod=239'39)", 2024-09-20T22:50:52.679 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:45:37.192419+0000", 2024-09-20T22:50:52.679 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 315.442605738, 2024-09-20T22:50:52.679 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013974723, 2024-09-20T22:50:52.679 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:50:52.680 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:50:52.680 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:50:52.680 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:50:52.680 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.680 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:50:52.680 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.192419+0000", 2024-09-20T22:50:52.680 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:50:52.680 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.680 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.680 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:50:52.680 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.192419+0000", 2024-09-20T22:50:52.680 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:50:52.680 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.680 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.680 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:50:52.681 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.192420+0000", 2024-09-20T22:50:52.681 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.3799999999999997e-07 2024-09-20T22:50:52.681 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.681 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.681 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:50:52.681 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.204088+0000", 2024-09-20T22:50:52.681 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011668883 2024-09-20T22:50:52.681 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.681 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.681 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:50:52.681 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.204104+0000", 2024-09-20T22:50:52.681 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.5437000000000001e-05 2024-09-20T22:50:52.681 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.681 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.681 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:50:52.682 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.204112+0000", 2024-09-20T22:50:52.682 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.2519999999999999e-06 2024-09-20T22:50:52.682 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.682 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.682 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:50:52.682 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.204168+0000", 2024-09-20T22:50:52.682 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.5977999999999998e-05 2024-09-20T22:50:52.682 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.682 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.682 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:50:52.682 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.204188+0000", 2024-09-20T22:50:52.682 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.9400999999999999e-05 2024-09-20T22:50:52.682 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.683 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.683 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:50:52.683 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.206340+0000", 2024-09-20T22:50:52.683 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0021527159999999998 2024-09-20T22:50:52.683 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.683 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.683 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:50:52.683 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.206394+0000", 2024-09-20T22:50:52.683 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.3418000000000003e-05 2024-09-20T22:50:52.683 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:50:52.683 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:50:52.683 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:50:52.683 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.683 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.684 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(osd.6.0:726 7.19 e260/139 7:9eac4560:::smithi08477203-30:4c v 260'159, mlcod=259'156)", 2024-09-20T22:50:52.684 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:45:39.197025+0000", 2024-09-20T22:50:52.684 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 313.43799982799999, 2024-09-20T22:50:52.684 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.018150454, 2024-09-20T22:50:52.684 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:50:52.684 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:50:52.684 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:50:52.684 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:50:52.684 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.684 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:50:52.684 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.197025+0000", 2024-09-20T22:50:52.684 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:50:52.684 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.684 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.685 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:50:52.686 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:50:52.686 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.197025+0000", 2024-09-20T22:50:52.686 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:50:52.686 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.686 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.686 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:50:52.687 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.197025+0000", 2024-09-20T22:50:52.687 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.7599999999999998e-07 2024-09-20T22:50:52.687 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.687 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.687 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:50:52.687 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.197033+0000", 2024-09-20T22:50:52.687 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.9119999999999998e-06 2024-09-20T22:50:52.687 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.687 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.687 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:50:52.687 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.197042+0000", 2024-09-20T22:50:52.687 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.3349999999999996e-06 2024-09-20T22:50:52.687 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.687 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.688 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:50:52.688 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.197050+0000", 2024-09-20T22:50:52.688 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.7230000000000004e-06 2024-09-20T22:50:52.688 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.688 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.688 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:50:52.688 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.200756+0000", 2024-09-20T22:50:52.688 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0037059749999999998 2024-09-20T22:50:52.688 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.688 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.688 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:50:52.688 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.200778+0000", 2024-09-20T22:50:52.688 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.2140000000000001e-05 2024-09-20T22:50:52.688 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.689 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.689 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:50:52.689 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.215131+0000", 2024-09-20T22:50:52.689 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014353054000000001 2024-09-20T22:50:52.689 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.689 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.689 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:50:52.689 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.215175+0000", 2024-09-20T22:50:52.689 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.4039000000000003e-05 2024-09-20T22:50:52.689 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:50:52.689 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:50:52.689 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:50:52.689 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.689 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.690 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(osd.7.0:152 7.5 e269/260 7:a160b5d4:::smithi08477203-16:55 v 269'55, mlcod=263'51)", 2024-09-20T22:50:52.690 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:45:48.173893+0000", 2024-09-20T22:50:52.690 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 304.46113208700001, 2024-09-20T22:50:52.690 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014936935, 2024-09-20T22:50:52.690 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:50:52.690 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:50:52.690 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:50:52.690 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:50:52.690 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.690 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:50:52.690 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.173893+0000", 2024-09-20T22:50:52.690 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:50:52.690 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.690 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.691 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:50:52.691 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.173893+0000", 2024-09-20T22:50:52.691 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:50:52.691 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.691 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.691 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:50:52.691 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.173893+0000", 2024-09-20T22:50:52.691 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.8800000000000002e-07 2024-09-20T22:50:52.691 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.691 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.691 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:50:52.691 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.173916+0000", 2024-09-20T22:50:52.691 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.2889999999999999e-05 2024-09-20T22:50:52.692 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.692 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.692 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:50:52.692 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.173924+0000", 2024-09-20T22:50:52.692 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.4420000000000001e-06 2024-09-20T22:50:52.692 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.692 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.692 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:50:52.692 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.173932+0000", 2024-09-20T22:50:52.692 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.5299999999999996e-06 2024-09-20T22:50:52.692 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.692 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.692 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:50:52.692 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.174011+0000", 2024-09-20T22:50:52.692 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.9342e-05 2024-09-20T22:50:52.693 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.693 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.693 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:50:52.693 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.174047+0000", 2024-09-20T22:50:52.693 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.5237999999999997e-05 2024-09-20T22:50:52.693 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.693 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.693 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:50:52.693 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.188782+0000", 2024-09-20T22:50:52.693 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014735599 2024-09-20T22:50:52.693 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.693 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.693 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:50:52.693 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.188830+0000", 2024-09-20T22:50:52.694 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.7206000000000002e-05 2024-09-20T22:50:52.694 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:50:52.694 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:50:52.694 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:50:52.694 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.694 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.694 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(osd.7.0:153 7.17 e269/139 7:ea720b85:::smithi08477203-26:55 v 269'225, mlcod=263'221)", 2024-09-20T22:50:52.694 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:45:48.173991+0000", 2024-09-20T22:50:52.694 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 304.461033606, 2024-09-20T22:50:52.694 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013064868, 2024-09-20T22:50:52.694 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:50:52.694 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:50:52.694 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:50:52.694 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:50:52.695 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.695 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:50:52.695 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.173991+0000", 2024-09-20T22:50:52.695 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:50:52.695 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.695 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.695 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:50:52.695 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.173991+0000", 2024-09-20T22:50:52.695 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:50:52.695 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.695 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.695 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:50:52.695 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.173991+0000", 2024-09-20T22:50:52.695 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.9999999999999999e-07 2024-09-20T22:50:52.696 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.696 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.696 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:50:52.696 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.173996+0000", 2024-09-20T22:50:52.696 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.3080000000000001e-06 2024-09-20T22:50:52.696 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.696 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.696 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:50:52.696 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.174001+0000", 2024-09-20T22:50:52.696 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.203e-06 2024-09-20T22:50:52.696 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.696 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.696 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:50:52.696 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.174005+0000", 2024-09-20T22:50:52.696 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.4880000000000001e-06 2024-09-20T22:50:52.697 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.697 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.697 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:50:52.697 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.175600+0000", 2024-09-20T22:50:52.697 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015942459999999999 2024-09-20T22:50:52.697 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.697 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.697 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:50:52.697 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.175631+0000", 2024-09-20T22:50:52.697 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.1816000000000002e-05 2024-09-20T22:50:52.697 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.697 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.697 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:50:52.697 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.187003+0000", 2024-09-20T22:50:52.697 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011372006 2024-09-20T22:50:52.698 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.698 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.698 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:50:52.698 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.187056+0000", 2024-09-20T22:50:52.698 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.2500999999999998e-05 2024-09-20T22:50:52.698 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:50:52.698 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:50:52.698 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:50:52.698 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.698 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.698 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(osd.6.0:767 7.12 e269/263 7:4fa4b2b3:::smithi08477203-21:53 v 269'262, mlcod=265'259)", 2024-09-20T22:50:52.698 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:45:48.175130+0000", 2024-09-20T22:50:52.698 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 304.45989441699999, 2024-09-20T22:50:52.698 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013842727000000001, 2024-09-20T22:50:52.699 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:50:52.699 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:50:52.699 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:50:52.699 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:50:52.699 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.699 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:50:52.699 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.175130+0000", 2024-09-20T22:50:52.699 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:50:52.699 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.699 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.699 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:50:52.699 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.175130+0000", 2024-09-20T22:50:52.699 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:50:52.699 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.700 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.700 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:50:52.700 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.175130+0000", 2024-09-20T22:50:52.700 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.98e-07 2024-09-20T22:50:52.700 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.700 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.700 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:50:52.700 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.175139+0000", 2024-09-20T22:50:52.700 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.5909999999999996e-06 2024-09-20T22:50:52.700 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.700 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.700 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:50:52.700 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.175144+0000", 2024-09-20T22:50:52.700 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.361e-06 2024-09-20T22:50:52.701 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.701 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.701 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:50:52.701 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.175156+0000", 2024-09-20T22:50:52.701 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1236999999999999e-05 2024-09-20T22:50:52.701 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.701 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.701 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:50:52.701 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.176359+0000", 2024-09-20T22:50:52.701 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0012036060000000001 2024-09-20T22:50:52.701 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.701 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.702 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:50:52.702 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.176380+0000", 2024-09-20T22:50:52.702 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.0497999999999999e-05 2024-09-20T22:50:52.702 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.702 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.702 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:50:52.702 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.188936+0000", 2024-09-20T22:50:52.702 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.012556286999999999 2024-09-20T22:50:52.702 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.702 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.702 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:50:52.702 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.188973+0000", 2024-09-20T22:50:52.702 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.6949000000000001e-05 2024-09-20T22:50:52.702 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:50:52.703 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:50:52.703 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:50:52.703 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.703 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.703 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24652.0:2853 7.10 e303/139 7:0f524219:::smithi08477203-50:head v 303'85, mlcod=301'82)", 2024-09-20T22:50:52.703 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:46:23.293837+0000", 2024-09-20T22:50:52.703 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 269.341187703, 2024-09-20T22:50:52.703 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013368555000000001, 2024-09-20T22:50:52.703 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:50:52.703 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:50:52.703 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:50:52.703 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:50:52.703 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.704 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:50:52.704 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.293837+0000", 2024-09-20T22:50:52.704 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:50:52.704 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.704 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.704 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:50:52.704 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.293837+0000", 2024-09-20T22:50:52.704 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:50:52.704 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.704 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.704 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:50:52.704 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.293839+0000", 2024-09-20T22:50:52.704 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.7239999999999999e-06 2024-09-20T22:50:52.704 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.705 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.705 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:50:52.705 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.297226+0000", 2024-09-20T22:50:52.705 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.003387583 2024-09-20T22:50:52.705 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.705 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.705 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:50:52.705 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.297253+0000", 2024-09-20T22:50:52.705 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.6849e-05 2024-09-20T22:50:52.705 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.705 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.705 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:50:52.705 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.297266+0000", 2024-09-20T22:50:52.706 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2481e-05 2024-09-20T22:50:52.706 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.706 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.706 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:50:52.706 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.297362+0000", 2024-09-20T22:50:52.706 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.6347000000000004e-05 2024-09-20T22:50:52.706 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.706 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.706 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:50:52.706 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.297394+0000", 2024-09-20T22:50:52.706 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.1888999999999999e-05 2024-09-20T22:50:52.706 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.706 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.707 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:50:52.707 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.307147+0000", 2024-09-20T22:50:52.707 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0097533719999999997 2024-09-20T22:50:52.707 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.707 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.707 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:50:52.707 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.307206+0000", 2024-09-20T22:50:52.707 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.8310000000000002e-05 2024-09-20T22:50:52.707 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:50:52.707 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:50:52.707 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:50:52.707 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.707 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.707 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:3030 7.e 7:71b5c915:::smithi08477203-6:head [copy-from ver 216] snapc 72=[72,71,70,6e,6c,6b,6a,68] ondisk+write+known_if_redirected+supports_pool_eio e317)", 2024-09-20T22:50:52.708 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:46:37.293029+0000", 2024-09-20T22:50:52.708 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 255.34199534499999, 2024-09-20T22:50:52.708 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.019929916999999998, 2024-09-20T22:50:52.708 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:50:52.708 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:50:52.708 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:50:52.708 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:50:52.708 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:50:52.708 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:50:52.708 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 3030 2024-09-20T22:50:52.708 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.708 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:50:52.708 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.708 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:50:52.708 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.293029+0000", 2024-09-20T22:50:52.709 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:50:52.709 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.709 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.709 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:50:52.709 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.293029+0000", 2024-09-20T22:50:52.709 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:50:52.709 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.709 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.709 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:50:52.709 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.293031+0000", 2024-09-20T22:50:52.709 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.4190000000000001e-06 2024-09-20T22:50:52.709 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.709 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.709 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:50:52.709 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.293040+0000", 2024-09-20T22:50:52.709 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.6039999999999999e-06 2024-09-20T22:50:52.710 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.710 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.710 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:50:52.710 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.293054+0000", 2024-09-20T22:50:52.710 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.4014e-05 2024-09-20T22:50:52.710 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.710 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.710 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:50:52.710 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.293062+0000", 2024-09-20T22:50:52.710 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.8820000000000001e-06 2024-09-20T22:50:52.710 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.710 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.710 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:50:52.711 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.293121+0000", 2024-09-20T22:50:52.711 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.8977999999999997e-05 2024-09-20T22:50:52.711 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.711 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.711 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op must wait for map", 2024-09-20T22:50:52.711 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.293125+0000", 2024-09-20T22:50:52.711 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.2089999999999999e-06 2024-09-20T22:50:52.711 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.711 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.711 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:50:52.711 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.298805+0000", 2024-09-20T22:50:52.711 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0056790870000000002 2024-09-20T22:50:52.711 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.711 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.711 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:50:52.712 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.298886+0000", 2024-09-20T22:50:52.712 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.1363e-05 2024-09-20T22:50:52.712 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.712 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.712 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 3,5", 2024-09-20T22:50:52.712 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.305883+0000", 2024-09-20T22:50:52.712 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0069966159999999998 2024-09-20T22:50:52.712 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.712 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.712 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:50:52.712 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.312556+0000", 2024-09-20T22:50:52.712 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0066737009999999998 2024-09-20T22:50:52.712 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.712 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.712 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:50:52.713 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.312587+0000", 2024-09-20T22:50:52.713 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.0286000000000001e-05 2024-09-20T22:50:52.713 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.713 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.713 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:50:52.713 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.312890+0000", 2024-09-20T22:50:52.713 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00030374800000000001 2024-09-20T22:50:52.713 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.713 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.713 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:50:52.713 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.312932+0000", 2024-09-20T22:50:52.713 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.1621e-05 2024-09-20T22:50:52.713 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.713 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.714 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:50:52.714 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.312959+0000", 2024-09-20T22:50:52.714 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.7389000000000002e-05 2024-09-20T22:50:52.714 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:50:52.714 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:50:52.714 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:50:52.714 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.714 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.714 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:3031 7.e 7:71b5c915:::smithi08477203-6:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e317)", 2024-09-20T22:50:52.714 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:46:37.293082+0000", 2024-09-20T22:50:52.714 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 255.34194296699999, 2024-09-20T22:50:52.714 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.016134757, 2024-09-20T22:50:52.714 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:50:52.715 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:50:52.715 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-20T22:50:52.715 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:50:52.715 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:50:52.715 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:50:52.715 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 3031 2024-09-20T22:50:52.715 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.715 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:50:52.715 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.715 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:50:52.715 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.293082+0000", 2024-09-20T22:50:52.715 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:50:52.715 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.715 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.716 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:50:52.716 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.293082+0000", 2024-09-20T22:50:52.716 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:50:52.716 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.716 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.716 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:50:52.716 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.293082+0000", 2024-09-20T22:50:52.716 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.1799999999999995e-07 2024-09-20T22:50:52.716 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.716 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.716 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:50:52.716 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.293085+0000", 2024-09-20T22:50:52.716 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.937e-06 2024-09-20T22:50:52.716 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.717 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.717 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:50:52.717 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.293093+0000", 2024-09-20T22:50:52.717 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.554e-06 2024-09-20T22:50:52.717 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.717 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.717 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:50:52.717 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.293098+0000", 2024-09-20T22:50:52.717 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.643e-06 2024-09-20T22:50:52.717 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.717 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.717 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:50:52.717 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.293164+0000", 2024-09-20T22:50:52.717 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.5806e-05 2024-09-20T22:50:52.718 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.718 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.718 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting_for_map not empty", 2024-09-20T22:50:52.718 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.293168+0000", 2024-09-20T22:50:52.718 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.2459999999999999e-06 2024-09-20T22:50:52.718 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.718 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.718 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:50:52.718 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.299033+0000", 2024-09-20T22:50:52.718 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0058656699999999999 2024-09-20T22:50:52.718 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.718 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.718 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for blocked object", 2024-09-20T22:50:52.718 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.299069+0000", 2024-09-20T22:50:52.719 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.5392999999999998e-05 2024-09-20T22:50:52.719 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.719 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.719 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:50:52.719 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.309088+0000", 2024-09-20T22:50:52.719 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010019287999999999 2024-09-20T22:50:52.719 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.719 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.719 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:50:52.719 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.309150+0000", 2024-09-20T22:50:52.719 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.2008000000000007e-05 2024-09-20T22:50:52.719 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.719 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.719 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:50:52.720 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.309216+0000", 2024-09-20T22:50:52.720 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.6594000000000006e-05 2024-09-20T22:50:52.720 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:50:52.720 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:50:52.720 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:50:52.720 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.720 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.720 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:3290 7.8 7:12e9c8db:::smithi08477203-11:head [copy-from ver 675] snapc 7f=[7f,7e,7d,7c,7b,7a,79,78,74,73,6b] ondisk+write+known_if_redirected+supports_pool_eio e340)", 2024-09-20T22:50:52.720 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:47:00.706846+0000", 2024-09-20T22:50:52.720 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 231.92817910799999, 2024-09-20T22:50:52.720 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.018408746, 2024-09-20T22:50:52.720 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:50:52.720 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:50:52.720 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:50:52.720 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:50:52.721 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:50:52.721 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:50:52.721 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 3290 2024-09-20T22:50:52.721 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.721 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:50:52.721 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.721 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:50:52.721 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.706846+0000", 2024-09-20T22:50:52.721 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:50:52.721 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.721 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.721 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:50:52.721 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.706846+0000", 2024-09-20T22:50:52.721 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:50:52.721 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.722 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.722 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:50:52.722 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.706848+0000", 2024-09-20T22:50:52.722 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.2340000000000001e-06 2024-09-20T22:50:52.722 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.722 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.722 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:50:52.722 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.706859+0000", 2024-09-20T22:50:52.722 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1001e-05 2024-09-20T22:50:52.722 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.722 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.722 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:50:52.722 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.706879+0000", 2024-09-20T22:50:52.722 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.0556000000000001e-05 2024-09-20T22:50:52.723 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.723 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.723 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:50:52.723 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.706890+0000", 2024-09-20T22:50:52.723 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0328e-05 2024-09-20T22:50:52.723 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.723 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.723 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:50:52.723 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.706973+0000", 2024-09-20T22:50:52.723 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.3574999999999999e-05 2024-09-20T22:50:52.723 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.723 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.723 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:50:52.723 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.707042+0000", 2024-09-20T22:50:52.724 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.8987999999999997e-05 2024-09-20T22:50:52.724 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.724 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.724 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 3,7", 2024-09-20T22:50:52.724 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.717491+0000", 2024-09-20T22:50:52.724 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010448618 2024-09-20T22:50:52.724 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.724 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.724 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:50:52.724 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.722875+0000", 2024-09-20T22:50:52.724 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0053846190000000002 2024-09-20T22:50:52.724 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.724 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.724 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:50:52.725 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.724841+0000", 2024-09-20T22:50:52.725 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0019651579999999998 2024-09-20T22:50:52.725 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.725 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.725 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:50:52.725 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.725168+0000", 2024-09-20T22:50:52.725 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00032734499999999999 2024-09-20T22:50:52.725 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.725 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.725 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:50:52.725 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.725218+0000", 2024-09-20T22:50:52.725 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.0123000000000001e-05 2024-09-20T22:50:52.725 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.725 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.726 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:50:52.726 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.725254+0000", 2024-09-20T22:50:52.726 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.6201000000000001e-05 2024-09-20T22:50:52.726 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:50:52.726 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:50:52.726 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:50:52.726 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.726 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.726 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:3291 7.8 7:12e9c8db:::smithi08477203-11:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e340)", 2024-09-20T22:50:52.726 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:47:00.706927+0000", 2024-09-20T22:50:52.726 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 231.92809746899999, 2024-09-20T22:50:52.726 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014159255000000001, 2024-09-20T22:50:52.726 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:50:52.726 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:50:52.726 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-20T22:50:52.727 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:50:52.727 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:50:52.727 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:50:52.727 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 3291 2024-09-20T22:50:52.727 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.727 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:50:52.727 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.727 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:50:52.727 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.706927+0000", 2024-09-20T22:50:52.727 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:50:52.727 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.727 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.727 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:50:52.727 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.706927+0000", 2024-09-20T22:50:52.728 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:50:52.728 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.728 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.728 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:50:52.728 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.706928+0000", 2024-09-20T22:50:52.728 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.4099999999999999e-07 2024-09-20T22:50:52.728 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.728 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.728 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:50:52.728 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.706941+0000", 2024-09-20T22:50:52.728 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3335e-05 2024-09-20T22:50:52.728 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.728 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.729 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:50:52.729 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.706948+0000", 2024-09-20T22:50:52.729 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.3350000000000001e-06 2024-09-20T22:50:52.729 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.729 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.729 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:50:52.729 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.706954+0000", 2024-09-20T22:50:52.729 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.9190000000000004e-06 2024-09-20T22:50:52.729 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.729 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.729 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:50:52.729 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.707166+0000", 2024-09-20T22:50:52.729 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00021159 2024-09-20T22:50:52.730 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.730 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.730 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for blocked object", 2024-09-20T22:50:52.730 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.707217+0000", 2024-09-20T22:50:52.730 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.1167e-05 2024-09-20T22:50:52.730 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.730 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.730 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:50:52.730 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.720939+0000", 2024-09-20T22:50:52.730 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013721458000000001 2024-09-20T22:50:52.730 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.730 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.730 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:50:52.730 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.721010+0000", 2024-09-20T22:50:52.730 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.1570999999999996e-05 2024-09-20T22:50:52.731 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.731 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.731 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:50:52.731 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.721086+0000", 2024-09-20T22:50:52.731 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.6339000000000002e-05 2024-09-20T22:50:52.731 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:50:52.731 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:50:52.731 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:50:52.731 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.731 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.731 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(osd.5.0:167 7.2 e347/169 7:45b54b6b:::smithi08477203-2:81 v 347'140, mlcod=345'136)", 2024-09-20T22:50:52.731 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:47:08.204691+0000", 2024-09-20T22:50:52.731 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 224.43033336799999, 2024-09-20T22:50:52.731 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.01335918, 2024-09-20T22:50:52.732 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:50:52.732 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:50:52.732 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:50:52.732 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:50:52.732 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.732 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:50:52.732 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:08.204691+0000", 2024-09-20T22:50:52.732 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:50:52.732 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.732 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.732 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:50:52.732 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:08.204691+0000", 2024-09-20T22:50:52.732 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:50:52.732 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.733 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.733 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:50:52.733 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:08.204692+0000", 2024-09-20T22:50:52.733 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.5400000000000001e-07 2024-09-20T22:50:52.733 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.733 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.733 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:50:52.733 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:08.204697+0000", 2024-09-20T22:50:52.733 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.614e-06 2024-09-20T22:50:52.733 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.733 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.733 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:50:52.733 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:08.204705+0000", 2024-09-20T22:50:52.733 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.8259999999999993e-06 2024-09-20T22:50:52.734 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.734 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.734 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:50:52.734 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:08.204715+0000", 2024-09-20T22:50:52.734 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.8279999999999998e-06 2024-09-20T22:50:52.734 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.734 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.734 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:50:52.734 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:08.204801+0000", 2024-09-20T22:50:52.734 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.6263999999999999e-05 2024-09-20T22:50:52.734 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.734 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.734 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:50:52.734 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:08.204823+0000", 2024-09-20T22:50:52.734 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.1971999999999999e-05 2024-09-20T22:50:52.735 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.735 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.735 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:50:52.735 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:08.217961+0000", 2024-09-20T22:50:52.735 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013137618 2024-09-20T22:50:52.735 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.735 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.735 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:50:52.735 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:08.218050+0000", 2024-09-20T22:50:52.735 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.9504000000000002e-05 2024-09-20T22:50:52.735 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:50:52.735 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:50:52.735 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:50:52.736 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.736 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.736 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24652.0:3760 7.13 e383/260 7:cae5792a:::smithi08477203-32:head v 383'78, mlcod=377'74)", 2024-09-20T22:50:52.736 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:47:43.291645+0000", 2024-09-20T22:50:52.736 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 189.34337949600001, 2024-09-20T22:50:52.736 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.012696648, 2024-09-20T22:50:52.736 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:50:52.736 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:50:52.736 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:50:52.736 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:50:52.736 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.736 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:50:52.736 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:43.291645+0000", 2024-09-20T22:50:52.736 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:50:52.737 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.737 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.737 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:50:52.737 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:43.291645+0000", 2024-09-20T22:50:52.737 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:50:52.737 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.737 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.737 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:50:52.737 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:43.291645+0000", 2024-09-20T22:50:52.737 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.0800000000000001e-07 2024-09-20T22:50:52.737 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.737 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.737 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:50:52.737 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:43.294764+0000", 2024-09-20T22:50:52.738 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0031183420000000001 2024-09-20T22:50:52.738 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.738 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.738 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:50:52.738 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:43.294777+0000", 2024-09-20T22:50:52.738 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3465999999999999e-05 2024-09-20T22:50:52.738 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.738 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.738 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:50:52.738 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:43.294788+0000", 2024-09-20T22:50:52.738 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0662999999999999e-05 2024-09-20T22:50:52.738 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.738 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.738 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:50:52.738 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:43.294952+0000", 2024-09-20T22:50:52.739 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00016369799999999999 2024-09-20T22:50:52.739 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.739 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.739 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:50:52.739 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:43.294978+0000", 2024-09-20T22:50:52.739 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.6414000000000002e-05 2024-09-20T22:50:52.739 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.739 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.739 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:50:52.739 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:43.304313+0000", 2024-09-20T22:50:52.739 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0093352039999999997 2024-09-20T22:50:52.739 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.739 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.739 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:50:52.739 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:43.304342+0000", 2024-09-20T22:50:52.740 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.8552999999999999e-05 2024-09-20T22:50:52.740 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:50:52.740 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:50:52.740 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:50:52.740 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.740 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.740 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24652.0:4534 7.12 e456/421 7:4b06cbed:::smithi08477203-49:head v 456'446, mlcod=454'443)", 2024-09-20T22:50:52.740 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:48:56.245037+0000", 2024-09-20T22:50:52.740 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 116.389987475, 2024-09-20T22:50:52.740 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013221933999999999, 2024-09-20T22:50:52.740 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:50:52.740 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:50:52.740 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:50:52.740 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:50:52.740 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.740 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:50:52.741 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:56.245037+0000", 2024-09-20T22:50:52.741 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:50:52.741 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.741 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.741 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:50:52.741 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:56.245037+0000", 2024-09-20T22:50:52.741 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:50:52.741 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.741 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.741 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:50:52.741 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:56.245037+0000", 2024-09-20T22:50:52.741 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.4600000000000001e-07 2024-09-20T22:50:52.741 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.741 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.741 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:50:52.741 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:56.248142+0000", 2024-09-20T22:50:52.742 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0031044839999999998 2024-09-20T22:50:52.742 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.742 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.742 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:50:52.742 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:56.248174+0000", 2024-09-20T22:50:52.742 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.2555000000000003e-05 2024-09-20T22:50:52.742 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.742 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.742 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:50:52.742 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:56.248188+0000", 2024-09-20T22:50:52.742 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.308e-05 2024-09-20T22:50:52.742 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.742 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.742 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:50:52.743 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:56.248292+0000", 2024-09-20T22:50:52.743 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000104182 2024-09-20T22:50:52.743 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.743 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.743 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:50:52.743 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:56.248318+0000", 2024-09-20T22:50:52.743 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.652e-05 2024-09-20T22:50:52.743 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.743 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.743 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:50:52.743 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:56.258207+0000", 2024-09-20T22:50:52.743 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0098883909999999998 2024-09-20T22:50:52.743 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:50:52.743 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:50:52.743 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:50:52.744 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:56.258259+0000", 2024-09-20T22:50:52.744 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.2475999999999997e-05 2024-09-20T22:50:52.744 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:50:52.744 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:50:52.744 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:50:52.744 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:50:52.744 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:50:52.744 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-20T22:50:52.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:52 smithi154 ceph-mon[33264]: osdmap e571: 8 total, 8 up, 8 in 2024-09-20T22:50:52.765 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:52.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:52.766 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:50:52.867 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:50:52.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:52.889 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:52.890 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:50:52.990 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:50:53.012 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:53.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:53.013 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:50:53.114 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:50:53.136 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:53.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:53.137 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:50:53.223 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-20T22:50:53.237 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:50:53.248 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is set 2024-09-20T22:50:53.256 INFO:tasks.rados.rados.0.smithi084.stdout:2242: done (6 left) 2024-09-20T22:50:53.256 INFO:tasks.rados.rados.0.smithi084.stdout:2243: done (5 left) 2024-09-20T22:50:53.256 INFO:tasks.rados.rados.0.smithi084.stdout:2244: done (4 left) 2024-09-20T22:50:53.256 INFO:tasks.rados.rados.0.smithi084.stdout:2245: done (3 left) 2024-09-20T22:50:53.256 INFO:tasks.rados.rados.0.smithi084.stdout:2246: done (2 left) 2024-09-20T22:50:53.256 INFO:tasks.rados.rados.0.smithi084.stdout:2247: done (1 left) 2024-09-20T22:50:53.256 INFO:tasks.rados.rados.0.smithi084.stdout:2248: done (0 left) 2024-09-20T22:50:53.256 INFO:tasks.rados.rados.0.smithi084.stdout:2249: snap_create 2024-09-20T22:50:53.262 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:53.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:53.262 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:50:53.363 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:50:53.385 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:53.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:53.386 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:50:53.487 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:50:53.508 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:53.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:53.509 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:50:53.610 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:50:53.614 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:53 smithi084 ceph-mon[40024]: pgmap v759: 158 pgs: 1 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 155 active+clean; 501 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 13 MiB/s rd, 2.3 MiB/s wr, 16 op/s 2024-09-20T22:50:53.614 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:53 smithi084 ceph-mon[40024]: osdmap e572: 8 total, 8 up, 8 in 2024-09-20T22:50:53.614 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:53 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/2302708955' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:50:53.614 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:53 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:50:53.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:53 smithi084 ceph-mon[40024]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:50:53.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:53 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/2302708955' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T22:50:53.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:53 smithi084 ceph-mon[40024]: osdmap e573: 8 total, 8 up, 8 in 2024-09-20T22:50:53.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:53 smithi084 ceph-mon[29774]: pgmap v759: 158 pgs: 1 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 155 active+clean; 501 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 13 MiB/s rd, 2.3 MiB/s wr, 16 op/s 2024-09-20T22:50:53.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:53 smithi084 ceph-mon[29774]: osdmap e572: 8 total, 8 up, 8 in 2024-09-20T22:50:53.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:53 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/2302708955' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:50:53.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:53 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:50:53.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:53 smithi084 ceph-mon[29774]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:50:53.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:53 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/2302708955' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T22:50:53.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:53 smithi084 ceph-mon[29774]: osdmap e573: 8 total, 8 up, 8 in 2024-09-20T22:50:53.631 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:53.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:53.632 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:50:53.703 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-20T22:50:53.703 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-20T22:50:53.704 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph tell osd.3 'trim stale osdmaps' 2024-09-20T22:50:53.732 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:50:53.754 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:53.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:53 smithi154 ceph-mon[33264]: pgmap v759: 158 pgs: 1 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 155 active+clean; 501 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 13 MiB/s rd, 2.3 MiB/s wr, 16 op/s 2024-09-20T22:50:53.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:53 smithi154 ceph-mon[33264]: osdmap e572: 8 total, 8 up, 8 in 2024-09-20T22:50:53.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:53 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/2302708955' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:50:53.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:53 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:50:53.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:53 smithi154 ceph-mon[33264]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:50:53.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:53 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/2302708955' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T22:50:53.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:53 smithi154 ceph-mon[33264]: osdmap e573: 8 total, 8 up, 8 in 2024-09-20T22:50:53.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:53.755 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:50:53.856 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:50:53.877 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:53.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:53.878 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:50:53.979 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:50:54.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:54.002 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:54.002 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:50:54.103 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:50:54.125 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:54.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:54.126 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:50:54.226 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:50:54.248 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:54.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:54.249 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:50:54.263 INFO:tasks.rados.rados.0.smithi084.stdout:2249: done (0 left) 2024-09-20T22:50:54.263 INFO:tasks.rados.rados.0.smithi084.stdout:2250: read oid 39 snap 209 2024-09-20T22:50:54.263 INFO:tasks.rados.rados.0.smithi084.stdout:2250: expect (ObjNum 666 snap 202 seq_num 666) 2024-09-20T22:50:54.263 INFO:tasks.rados.rados.0.smithi084.stdout:2251: copy_from oid 40 from oid 13 current snap is 251 2024-09-20T22:50:54.263 INFO:tasks.rados.rados.0.smithi084.stdout:2252: snap_remove snap 224 2024-09-20T22:50:54.350 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:50:54.371 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:54.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:54.372 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:50:54.472 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:50:54.494 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:54.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:54.494 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:50:54.595 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:50:54.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:54.619 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:54.619 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:50:54.696 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-20T22:50:54.720 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:50:54.762 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:54.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:54.763 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:50:54.863 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:50:54.886 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:54.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:54.887 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:50:54.987 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:50:55.009 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:55.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:55.010 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:50:55.111 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:50:55.132 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:55.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:55.133 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:50:55.173 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:50:55.233 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:50:55.255 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:55.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:55.256 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:50:55.263 INFO:tasks.rados.rados.0.smithi084.stdout:2252: done (2 left) 2024-09-20T22:50:55.264 INFO:tasks.rados.rados.0.smithi084.stdout:2253: read oid 6 snap -1 2024-09-20T22:50:55.264 INFO:tasks.rados.rados.0.smithi084.stdout:2253: expect (ObjNum 754 snap 238 seq_num 754) 2024-09-20T22:50:55.264 INFO:tasks.rados.rados.0.smithi084.stdout:2254: write oid 29 current snap is 251 2024-09-20T22:50:55.264 INFO:tasks.rados.rados.0.smithi084.stdout:2254: seq_num 778 ranges {675846=411728,1543579=787592,2848853=226993} 2024-09-20T22:50:55.270 INFO:tasks.rados.rados.0.smithi084.stdout:2254: writing smithi08477203-29 from 675846 to 1087574 tid 1 2024-09-20T22:50:55.278 INFO:tasks.rados.rados.0.smithi084.stdout:2254: writing smithi08477203-29 from 1543579 to 2331171 tid 2 2024-09-20T22:50:55.283 INFO:tasks.rados.rados.0.smithi084.stdout:2254: writing smithi08477203-29 from 2848853 to 3075846 tid 3 2024-09-20T22:50:55.283 INFO:tasks.rados.rados.0.smithi084.stdout:2255: read oid 17 snap -1 2024-09-20T22:50:55.283 INFO:tasks.rados.rados.0.smithi084.stdout:2255: expect (ObjNum 774 snap 248 seq_num 774) 2024-09-20T22:50:55.283 INFO:tasks.rados.rados.0.smithi084.stdout:2256: read oid 11 snap -1 2024-09-20T22:50:55.283 INFO:tasks.rados.rados.0.smithi084.stdout:2256: expect (ObjNum 556 snap 169 seq_num 556) 2024-09-20T22:50:55.283 INFO:tasks.rados.rados.0.smithi084.stdout:2257: read oid 22 snap -1 2024-09-20T22:50:55.283 INFO:tasks.rados.rados.0.smithi084.stdout:2257: expect deleted 2024-09-20T22:50:55.284 INFO:tasks.rados.rados.0.smithi084.stdout:2258: rollback oid 3 current snap is 251 2024-09-20T22:50:55.284 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 3 to 228 2024-09-20T22:50:55.284 INFO:tasks.rados.rados.0.smithi084.stdout:2250: expect (ObjNum 666 snap 202 seq_num 666) 2024-09-20T22:50:55.357 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:50:55.379 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:55.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:55.380 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:50:55.430 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:55 smithi084 ceph-mon[40024]: purged_snaps scrub starts 2024-09-20T22:50:55.430 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:55 smithi084 ceph-mon[40024]: purged_snaps scrub ok 2024-09-20T22:50:55.430 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:55 smithi084 ceph-mon[40024]: pgmap v762: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 540 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 19 MiB/s rd, 7.3 MiB/s wr, 34 op/s 2024-09-20T22:50:55.430 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:55 smithi084 ceph-mon[40024]: osdmap e574: 8 total, 8 up, 8 in 2024-09-20T22:50:55.430 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:55 smithi084 ceph-mon[29774]: purged_snaps scrub starts 2024-09-20T22:50:55.430 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:55 smithi084 ceph-mon[29774]: purged_snaps scrub ok 2024-09-20T22:50:55.431 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:55 smithi084 ceph-mon[29774]: pgmap v762: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 540 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 19 MiB/s rd, 7.3 MiB/s wr, 34 op/s 2024-09-20T22:50:55.431 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:55 smithi084 ceph-mon[29774]: osdmap e574: 8 total, 8 up, 8 in 2024-09-20T22:50:55.476 INFO:tasks.rados.rados.0.smithi084.stdout:2251: finishing copy_from racing read to smithi08477203-40 2024-09-20T22:50:55.476 INFO:tasks.rados.rados.0.smithi084.stdout:2251: finishing copy_from to smithi08477203-40 2024-09-20T22:50:55.476 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 40 v 402 (ObjNum 440 snap 115 seq_num 440) dirty exists 2024-09-20T22:50:55.476 INFO:tasks.rados.rados.0.smithi084.stdout:2253: expect (ObjNum 754 snap 238 seq_num 754) 2024-09-20T22:50:55.481 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:50:55.502 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:55.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:55.503 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:50:55.563 INFO:tasks.rados.rados.0.smithi084.stdout:2254: finishing write tid 1 to smithi08477203-29 2024-09-20T22:50:55.563 INFO:tasks.rados.rados.0.smithi084.stdout:2254: finishing write tid 2 to smithi08477203-29 2024-09-20T22:50:55.563 INFO:tasks.rados.rados.0.smithi084.stdout:2258: finishing rollback tid 1 to smithi08477203-3 2024-09-20T22:50:55.564 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 3 v 766 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-09-20T22:50:55.564 INFO:tasks.rados.rados.0.smithi084.stdout:2254: finishing write tid 3 to smithi08477203-29 2024-09-20T22:50:55.564 INFO:tasks.rados.rados.0.smithi084.stdout:2250: done (7 left) 2024-09-20T22:50:55.564 INFO:tasks.rados.rados.0.smithi084.stdout:2251: done (6 left) 2024-09-20T22:50:55.564 INFO:tasks.rados.rados.0.smithi084.stdout:2253: done (5 left) 2024-09-20T22:50:55.564 INFO:tasks.rados.rados.0.smithi084.stdout:2257: done (4 left) 2024-09-20T22:50:55.564 INFO:tasks.rados.rados.0.smithi084.stdout:2258: done (3 left) 2024-09-20T22:50:55.564 INFO:tasks.rados.rados.0.smithi084.stdout:2259: snap_create 2024-09-20T22:50:55.564 INFO:tasks.rados.rados.0.smithi084.stdout:2254: finishing write tid 4 to smithi08477203-29 2024-09-20T22:50:55.564 INFO:tasks.rados.rados.0.smithi084.stdout:2256: expect (ObjNum 556 snap 169 seq_num 556) 2024-09-20T22:50:55.603 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:50:55.624 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:55.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:55.625 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:50:55.695 INFO:teuthology.orchestra.run.smithi084.stderr:no valid command found; 10 closest matches: 2024-09-20T22:50:55.695 INFO:teuthology.orchestra.run.smithi084.stderr:0 2024-09-20T22:50:55.695 INFO:teuthology.orchestra.run.smithi084.stderr:1 2024-09-20T22:50:55.695 INFO:teuthology.orchestra.run.smithi084.stderr:2 2024-09-20T22:50:55.695 INFO:teuthology.orchestra.run.smithi084.stderr:abort 2024-09-20T22:50:55.695 INFO:teuthology.orchestra.run.smithi084.stderr:assert 2024-09-20T22:50:55.696 INFO:teuthology.orchestra.run.smithi084.stderr:bench [] [] [] [] 2024-09-20T22:50:55.696 INFO:teuthology.orchestra.run.smithi084.stderr:bluefs debug_inject_read_zeros 2024-09-20T22:50:55.696 INFO:teuthology.orchestra.run.smithi084.stderr:bluefs files list 2024-09-20T22:50:55.696 INFO:teuthology.orchestra.run.smithi084.stderr:bluefs stats 2024-09-20T22:50:55.696 INFO:teuthology.orchestra.run.smithi084.stderr:bluestore allocator dump block 2024-09-20T22:50:55.696 INFO:teuthology.orchestra.run.smithi084.stderr:Error EINVAL: invalid command 2024-09-20T22:50:55.726 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:50:55.748 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:55.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:55.749 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:50:55.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:55 smithi154 ceph-mon[33264]: purged_snaps scrub starts 2024-09-20T22:50:55.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:55 smithi154 ceph-mon[33264]: purged_snaps scrub ok 2024-09-20T22:50:55.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:55 smithi154 ceph-mon[33264]: pgmap v762: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 540 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 19 MiB/s rd, 7.3 MiB/s wr, 34 op/s 2024-09-20T22:50:55.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:55 smithi154 ceph-mon[33264]: osdmap e574: 8 total, 8 up, 8 in 2024-09-20T22:50:55.798 INFO:tasks.rados.rados.0.smithi084.stdout:2254: finishing write tid 5 to smithi08477203-29 2024-09-20T22:50:55.798 INFO:tasks.rados.rados.0.smithi084.stdout:2254: oid 29 updating version 0 to 1589 2024-09-20T22:50:55.798 INFO:tasks.rados.rados.0.smithi084.stdout:2254: oid 29 version 1589 is already newer than 1581 2024-09-20T22:50:55.799 INFO:tasks.rados.rados.0.smithi084.stdout:2254: oid 29 updating version 1589 to 1590 2024-09-20T22:50:55.799 INFO:tasks.rados.rados.0.smithi084.stdout:2254: oid 29 version 1590 is already newer than 1588 2024-09-20T22:50:55.799 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 29 v 1590 (ObjNum 778 snap 251 seq_num 778) dirty exists 2024-09-20T22:50:55.799 INFO:tasks.rados.rados.0.smithi084.stdout:2254: left oid 29 (ObjNum 778 snap 251 seq_num 778) 2024-09-20T22:50:55.799 INFO:tasks.rados.rados.0.smithi084.stdout:2255: expect (ObjNum 774 snap 248 seq_num 774) 2024-09-20T22:50:55.849 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:50:55.870 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:55.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:55.871 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:50:55.972 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:50:55.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:55.994 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:55.994 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:50:56.095 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:50:56.116 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:56.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:56.117 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:50:56.218 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:50:56.238 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:56.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:56.239 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:50:56.274 INFO:tasks.rados.rados.0.smithi084.stdout:2254: done (3 left) 2024-09-20T22:50:56.274 INFO:tasks.rados.rados.0.smithi084.stdout:2255: done (2 left) 2024-09-20T22:50:56.274 INFO:tasks.rados.rados.0.smithi084.stdout:2256: done (1 left) 2024-09-20T22:50:56.274 INFO:tasks.rados.rados.0.smithi084.stdout:2259: done (0 left) 2024-09-20T22:50:56.274 INFO:tasks.rados.rados.0.smithi084.stdout:2260: snap_remove snap 242 2024-09-20T22:50:56.340 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:50:56.361 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:56.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:56.362 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:50:56.405 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-20T22:50:56.405 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-20T22:50:56.406 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph tell osd.5 'trim stale osdmaps' 2024-09-20T22:50:56.435 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:50:56.450 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:50:56.463 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:50:56.484 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:56.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:56.485 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:50:56.559 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:56 smithi084 ceph-mon[40024]: osdmap e575: 8 total, 8 up, 8 in 2024-09-20T22:50:56.559 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:56 smithi084 ceph-mon[29774]: osdmap e575: 8 total, 8 up, 8 in 2024-09-20T22:50:56.586 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:50:56.607 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:56.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:56.608 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:50:56.708 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:50:56.730 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:56.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:56.732 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:50:56.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:56 smithi154 ceph-mon[33264]: osdmap e575: 8 total, 8 up, 8 in 2024-09-20T22:50:56.832 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:50:56.854 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:56.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:56.855 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:50:56.955 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:50:56.962 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:50:56.962 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:50:56.975 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:50:56.977 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:56.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:56.978 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:50:56.980 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:50:56.981 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:50:56.997 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:50:56.998 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:50:57.013 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-20T22:50:57.013 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T22:50:57.030 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:50:57.030 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:50:57.047 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:50:57.047 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:50:57.063 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:50:57.063 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:50:57.079 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:50:57.081 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:50:57.081 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:50:57.099 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:57.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:57.100 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:50:57.201 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd unset noscrub 2024-09-20T22:50:57.202 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:50:57.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:57.224 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:57.224 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:50:57.282 INFO:tasks.rados.rados.0.smithi084.stdout:2260: done (0 left) 2024-09-20T22:50:57.282 INFO:tasks.rados.rados.0.smithi084.stdout:2261: snap_remove snap 228 2024-09-20T22:50:57.325 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:50:57.346 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:57.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:57.347 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:50:57.364 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:57 smithi084 ceph-mon[29774]: pgmap v765: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 540 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 19 MiB/s rd, 7.3 MiB/s wr, 34 op/s 2024-09-20T22:50:57.364 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:57 smithi084 ceph-mon[29774]: osdmap e576: 8 total, 8 up, 8 in 2024-09-20T22:50:57.364 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:57 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/6731206' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:50:57.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:57 smithi084 ceph-mon[40024]: pgmap v765: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 540 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 19 MiB/s rd, 7.3 MiB/s wr, 34 op/s 2024-09-20T22:50:57.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:57 smithi084 ceph-mon[40024]: osdmap e576: 8 total, 8 up, 8 in 2024-09-20T22:50:57.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:57 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/6731206' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:50:57.447 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:50:57.468 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:57.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:57.469 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:50:57.570 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:50:57.592 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:57.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:57.593 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:50:57.694 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:50:57.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:57.715 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:57.715 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:50:57.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:57 smithi154 ceph-mon[33264]: pgmap v765: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 540 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 19 MiB/s rd, 7.3 MiB/s wr, 34 op/s 2024-09-20T22:50:57.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:57 smithi154 ceph-mon[33264]: osdmap e576: 8 total, 8 up, 8 in 2024-09-20T22:50:57.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:57 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/6731206' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:50:57.815 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:50:57.837 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:57.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:57.838 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:50:57.877 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-20T22:50:57.878 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:50:57.939 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:50:57.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:57.961 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:57.961 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:50:58.062 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:50:58.083 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:58.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:58.084 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:50:58.185 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:50:58.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:58.207 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:58.207 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:50:58.256 INFO:tasks.rados.rados.0.smithi084.stdout:2261: done (0 left) 2024-09-20T22:50:58.256 INFO:tasks.rados.rados.0.smithi084.stdout:2262: write (excl) oid 41 current snap is 252 2024-09-20T22:50:58.256 INFO:tasks.rados.rados.0.smithi084.stdout:2262: seq_num 779 ranges {692653=486977,1602061=647603,2992629=100024} 2024-09-20T22:50:58.263 INFO:tasks.rados.rados.0.smithi084.stdout:2262: writing smithi08477203-41 from 692653 to 1179630 tid 1 2024-09-20T22:50:58.270 INFO:tasks.rados.rados.0.smithi084.stdout:2262: writing smithi08477203-41 from 1602061 to 2249664 tid 2 2024-09-20T22:50:58.275 INFO:tasks.rados.rados.0.smithi084.stdout:2262: writing smithi08477203-41 from 2992629 to 3092653 tid 3 2024-09-20T22:50:58.275 INFO:tasks.rados.rados.0.smithi084.stdout:2263: read oid 31 snap -1 2024-09-20T22:50:58.275 INFO:tasks.rados.rados.0.smithi084.stdout:2263: expect deleted 2024-09-20T22:50:58.275 INFO:tasks.rados.rados.0.smithi084.stdout:2264: write oid 24 current snap is 252 2024-09-20T22:50:58.275 INFO:tasks.rados.rados.0.smithi084.stdout:2262: finishing write tid 1 to smithi08477203-41 2024-09-20T22:50:58.275 INFO:tasks.rados.rados.0.smithi084.stdout:2264: seq_num 780 ranges {709460=562226,2060543=791261} 2024-09-20T22:50:58.282 INFO:tasks.rados.rados.0.smithi084.stdout:2264: writing smithi08477203-24 from 709460 to 1271686 tid 1 2024-09-20T22:50:58.291 INFO:tasks.rados.rados.0.smithi084.stdout:2264: writing smithi08477203-24 from 2060543 to 2851804 tid 2 2024-09-20T22:50:58.291 INFO:tasks.rados.rados.0.smithi084.stdout:2265: read oid 13 snap -1 2024-09-20T22:50:58.291 INFO:tasks.rados.rados.0.smithi084.stdout:2265: expect (ObjNum 440 snap 115 seq_num 440) 2024-09-20T22:50:58.291 INFO:tasks.rados.rados.0.smithi084.stdout:2266: read oid 37 snap 195 2024-09-20T22:50:58.291 INFO:tasks.rados.rados.0.smithi084.stdout:2266: expect deleted 2024-09-20T22:50:58.291 INFO:tasks.rados.rados.0.smithi084.stdout:2267: snap_remove snap 209 2024-09-20T22:50:58.301 INFO:teuthology.orchestra.run.smithi084.stderr:no valid command found; 10 closest matches: 2024-09-20T22:50:58.301 INFO:teuthology.orchestra.run.smithi084.stderr:0 2024-09-20T22:50:58.301 INFO:teuthology.orchestra.run.smithi084.stderr:1 2024-09-20T22:50:58.301 INFO:teuthology.orchestra.run.smithi084.stderr:2 2024-09-20T22:50:58.301 INFO:teuthology.orchestra.run.smithi084.stderr:abort 2024-09-20T22:50:58.301 INFO:teuthology.orchestra.run.smithi084.stderr:assert 2024-09-20T22:50:58.301 INFO:teuthology.orchestra.run.smithi084.stderr:bench [] [] [] [] 2024-09-20T22:50:58.301 INFO:teuthology.orchestra.run.smithi084.stderr:bluefs debug_inject_read_zeros 2024-09-20T22:50:58.301 INFO:teuthology.orchestra.run.smithi084.stderr:bluefs files list 2024-09-20T22:50:58.301 INFO:teuthology.orchestra.run.smithi084.stderr:bluefs stats 2024-09-20T22:50:58.301 INFO:teuthology.orchestra.run.smithi084.stderr:bluestore allocator dump block 2024-09-20T22:50:58.301 INFO:teuthology.orchestra.run.smithi084.stderr:Error EINVAL: invalid command 2024-09-20T22:50:58.308 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:50:58.329 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:58.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:58.330 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:50:58.431 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:50:58.452 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:58.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:58.453 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:50:58.554 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:50:58.575 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:58.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:58.576 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:50:58.614 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:50:58 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: ::ffff:172.21.15.154 - - [20/Sep/2024:22:50:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-20T22:50:58.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:58 smithi084 ceph-mon[40024]: osdmap e577: 8 total, 8 up, 8 in 2024-09-20T22:50:58.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:58 smithi084 ceph-mon[40024]: osdmap e578: 8 total, 8 up, 8 in 2024-09-20T22:50:58.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:58 smithi084 ceph-mon[29774]: osdmap e577: 8 total, 8 up, 8 in 2024-09-20T22:50:58.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:58 smithi084 ceph-mon[29774]: osdmap e578: 8 total, 8 up, 8 in 2024-09-20T22:50:58.677 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:50:58.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:58.700 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:58.700 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:50:58.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:58 smithi154 ceph-mon[33264]: osdmap e577: 8 total, 8 up, 8 in 2024-09-20T22:50:58.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:58 smithi154 ceph-mon[33264]: osdmap e578: 8 total, 8 up, 8 in 2024-09-20T22:50:58.801 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:50:58.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:58.823 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:58.823 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:50:58.924 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:50:58.946 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:58.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:58.947 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:50:59.048 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:50:59.054 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-20T22:50:59.055 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-20T22:50:59.055 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph tell osd.6 'trim stale osdmaps' 2024-09-20T22:50:59.070 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:59.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:59.071 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:50:59.171 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:50:59.193 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:59.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:59.194 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:50:59.291 INFO:tasks.rados.rados.0.smithi084.stdout:2267: done (5 left) 2024-09-20T22:50:59.291 INFO:tasks.rados.rados.0.smithi084.stdout:2268: snap_remove snap 247 2024-09-20T22:50:59.295 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:50:59.317 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:59.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:59.318 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:50:59.419 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:50:59.440 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:59.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:59.441 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:50:59.542 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:50:59.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:59.571 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:59.571 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:50:59.582 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:50:59 smithi084 ceph-mon[40024]: pgmap v768: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 540 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-20T22:50:59.583 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:50:59 smithi084 ceph-mon[29774]: pgmap v768: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 540 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-20T22:50:59.672 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:50:59.693 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:59.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:59.694 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:50:59.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:50:59 smithi154 ceph-mon[33264]: pgmap v768: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 540 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-20T22:50:59.754 INFO:journalctl@ceph.iscsi.iscsi.a.smithi154.stdout:Sep 20 22:50:59 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-iscsi-iscsi-a[59730]: debug there is no tcmu-runner data available 2024-09-20T22:50:59.794 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:50:59.816 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:50:59.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:59.817 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:50:59.917 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:50:59.939 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:50:59.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:50:59.940 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:51:00.041 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:51:00.062 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:00.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:00.063 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:51:00.144 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.3 dump_blocked_ops 2024-09-20T22:51:00.145 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:51:00.164 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:51:00.207 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:00.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:00.208 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:51:00.302 INFO:tasks.rados.rados.0.smithi084.stdout:2268: done (5 left) 2024-09-20T22:51:00.302 INFO:tasks.rados.rados.0.smithi084.stdout:2269: read oid 47 snap 204 2024-09-20T22:51:00.302 INFO:tasks.rados.rados.0.smithi084.stdout:2269: expect (ObjNum 566 snap 170 seq_num 566) 2024-09-20T22:51:00.302 INFO:tasks.rados.rados.0.smithi084.stdout:2270: snap_create 2024-09-20T22:51:00.303 INFO:tasks.rados.rados.0.smithi084.stdout:2262: finishing write tid 2 to smithi08477203-41 2024-09-20T22:51:00.303 INFO:tasks.rados.rados.0.smithi084.stdout:2262: finishing write tid 3 to smithi08477203-41 2024-09-20T22:51:00.303 INFO:tasks.rados.rados.0.smithi084.stdout:2262: finishing write tid 4 to smithi08477203-41 2024-09-20T22:51:00.303 INFO:tasks.rados.rados.0.smithi084.stdout:2262: finishing write tid 5 to smithi08477203-41 2024-09-20T22:51:00.303 INFO:tasks.rados.rados.0.smithi084.stdout:2262: oid 41 updating version 0 to 1393 2024-09-20T22:51:00.303 INFO:tasks.rados.rados.0.smithi084.stdout:2262: oid 41 version 1393 is already newer than 1392 2024-09-20T22:51:00.303 INFO:tasks.rados.rados.0.smithi084.stdout:2262: oid 41 version 1393 is already newer than 1390 2024-09-20T22:51:00.303 INFO:tasks.rados.rados.0.smithi084.stdout:2262: oid 41 version 1393 is already newer than 1391 2024-09-20T22:51:00.303 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 41 v 1393 (ObjNum 779 snap 252 seq_num 779) dirty exists 2024-09-20T22:51:00.303 INFO:tasks.rados.rados.0.smithi084.stdout:2262: left oid 41 (ObjNum 779 snap 252 seq_num 779) 2024-09-20T22:51:00.303 INFO:tasks.rados.rados.0.smithi084.stdout:2264: finishing write tid 1 to smithi08477203-24 2024-09-20T22:51:00.304 INFO:tasks.rados.rados.0.smithi084.stdout:2264: finishing write tid 2 to smithi08477203-24 2024-09-20T22:51:00.304 INFO:tasks.rados.rados.0.smithi084.stdout:2264: finishing write tid 3 to smithi08477203-24 2024-09-20T22:51:00.304 INFO:tasks.rados.rados.0.smithi084.stdout:2264: finishing write tid 4 to smithi08477203-24 2024-09-20T22:51:00.304 INFO:tasks.rados.rados.0.smithi084.stdout:2264: oid 24 updating version 0 to 741 2024-09-20T22:51:00.304 INFO:tasks.rados.rados.0.smithi084.stdout:2264: oid 24 version 741 is already newer than 740 2024-09-20T22:51:00.304 INFO:tasks.rados.rados.0.smithi084.stdout:2264: oid 24 version 741 is already newer than 739 2024-09-20T22:51:00.304 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 24 v 741 (ObjNum 780 snap 252 seq_num 780) dirty exists 2024-09-20T22:51:00.304 INFO:tasks.rados.rados.0.smithi084.stdout:2264: left oid 24 (ObjNum 780 snap 252 seq_num 780) 2024-09-20T22:51:00.304 INFO:tasks.rados.rados.0.smithi084.stdout:2265: expect (ObjNum 440 snap 115 seq_num 440) 2024-09-20T22:51:00.309 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:51:00.331 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:00.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:00.332 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:51:00.364 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:00 smithi084 ceph-mon[40024]: osdmap e579: 8 total, 8 up, 8 in 2024-09-20T22:51:00.364 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:00 smithi084 ceph-mon[40024]: purged_snaps scrub starts 2024-09-20T22:51:00.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:00 smithi084 ceph-mon[40024]: purged_snaps scrub ok 2024-09-20T22:51:00.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:00 smithi084 ceph-mon[40024]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:51:00.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:00 smithi084 ceph-mon[29774]: osdmap e579: 8 total, 8 up, 8 in 2024-09-20T22:51:00.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:00 smithi084 ceph-mon[29774]: purged_snaps scrub starts 2024-09-20T22:51:00.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:00 smithi084 ceph-mon[29774]: purged_snaps scrub ok 2024-09-20T22:51:00.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:00 smithi084 ceph-mon[29774]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:51:00.432 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:51:00.456 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:00.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:00.457 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:51:00.558 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:51:00.577 INFO:tasks.rados.rados.0.smithi084.stdout:2269: expect (ObjNum 566 snap 170 seq_num 566) 2024-09-20T22:51:00.578 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:00.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:00.579 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:51:00.680 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:51:00.701 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:00.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:00.702 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:51:00.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:00 smithi154 ceph-mon[33264]: osdmap e579: 8 total, 8 up, 8 in 2024-09-20T22:51:00.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:00 smithi154 ceph-mon[33264]: purged_snaps scrub starts 2024-09-20T22:51:00.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:00 smithi154 ceph-mon[33264]: purged_snaps scrub ok 2024-09-20T22:51:00.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:00 smithi154 ceph-mon[33264]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:51:00.762 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:51:00.766 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:51:00.803 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:51:00.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:00.824 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:00.825 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:51:00.925 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:51:00.947 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:00.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:00.948 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:51:01.048 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:51:01.069 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:01.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:01.070 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:51:01.171 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:51:01.192 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:01.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:01.193 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:51:01.293 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:51:01.300 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is unset 2024-09-20T22:51:01.301 INFO:tasks.rados.rados.0.smithi084.stdout:2262: done (6 left) 2024-09-20T22:51:01.301 INFO:tasks.rados.rados.0.smithi084.stdout:2263: done (5 left) 2024-09-20T22:51:01.301 INFO:tasks.rados.rados.0.smithi084.stdout:2264: done (4 left) 2024-09-20T22:51:01.301 INFO:tasks.rados.rados.0.smithi084.stdout:2265: done (3 left) 2024-09-20T22:51:01.301 INFO:tasks.rados.rados.0.smithi084.stdout:2266: done (2 left) 2024-09-20T22:51:01.301 INFO:tasks.rados.rados.0.smithi084.stdout:2269: done (1 left) 2024-09-20T22:51:01.301 INFO:tasks.rados.rados.0.smithi084.stdout:2270: done (0 left) 2024-09-20T22:51:01.302 INFO:tasks.rados.rados.0.smithi084.stdout:2271: read oid 40 snap 189 2024-09-20T22:51:01.302 INFO:tasks.rados.rados.0.smithi084.stdout:2271: expect deleted 2024-09-20T22:51:01.302 INFO:tasks.rados.rados.0.smithi084.stdout:2272: read oid 6 snap -1 2024-09-20T22:51:01.302 INFO:tasks.rados.rados.0.smithi084.stdout:2272: expect (ObjNum 754 snap 238 seq_num 754) 2024-09-20T22:51:01.302 INFO:tasks.rados.rados.0.smithi084.stdout:2273: delete oid 46 current snap is 253 2024-09-20T22:51:01.310 INFO:tasks.rados.rados.0.smithi084.stdout:2271: done (2 left) 2024-09-20T22:51:01.310 INFO:tasks.rados.rados.0.smithi084.stdout:2273: done (1 left) 2024-09-20T22:51:01.310 INFO:tasks.rados.rados.0.smithi084.stdout:2274: delete oid 31 current snap is 253 2024-09-20T22:51:01.311 INFO:tasks.rados.rados.0.smithi084.stdout:2272: expect (ObjNum 754 snap 238 seq_num 754) 2024-09-20T22:51:01.317 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:01.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:01.317 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:51:01.354 INFO:teuthology.orchestra.run.smithi084.stderr:no valid command found; 10 closest matches: 2024-09-20T22:51:01.354 INFO:teuthology.orchestra.run.smithi084.stderr:0 2024-09-20T22:51:01.354 INFO:teuthology.orchestra.run.smithi084.stderr:1 2024-09-20T22:51:01.354 INFO:teuthology.orchestra.run.smithi084.stderr:2 2024-09-20T22:51:01.354 INFO:teuthology.orchestra.run.smithi084.stderr:abort 2024-09-20T22:51:01.354 INFO:teuthology.orchestra.run.smithi084.stderr:assert 2024-09-20T22:51:01.354 INFO:teuthology.orchestra.run.smithi084.stderr:bench [] [] [] [] 2024-09-20T22:51:01.354 INFO:teuthology.orchestra.run.smithi084.stderr:bluefs debug_inject_read_zeros 2024-09-20T22:51:01.354 INFO:teuthology.orchestra.run.smithi084.stderr:bluefs files list 2024-09-20T22:51:01.354 INFO:teuthology.orchestra.run.smithi084.stderr:bluefs stats 2024-09-20T22:51:01.354 INFO:teuthology.orchestra.run.smithi084.stderr:bluestore allocator dump block 2024-09-20T22:51:01.354 INFO:teuthology.orchestra.run.smithi084.stderr:Error EINVAL: invalid command 2024-09-20T22:51:01.405 INFO:tasks.rados.rados.0.smithi084.stdout:2272: done (1 left) 2024-09-20T22:51:01.405 INFO:tasks.rados.rados.0.smithi084.stdout:2274: done (0 left) 2024-09-20T22:51:01.405 INFO:tasks.rados.rados.0.smithi084.stdout:2275: read oid 11 snap -1 2024-09-20T22:51:01.405 INFO:tasks.rados.rados.0.smithi084.stdout:2275: expect (ObjNum 556 snap 169 seq_num 556) 2024-09-20T22:51:01.405 INFO:tasks.rados.rados.0.smithi084.stdout:2276: read oid 38 snap -1 2024-09-20T22:51:01.406 INFO:tasks.rados.rados.0.smithi084.stdout:2276: expect (ObjNum 771 snap 248 seq_num 771) 2024-09-20T22:51:01.406 INFO:tasks.rados.rados.0.smithi084.stdout:2277: delete oid 6 current snap is 253 2024-09-20T22:51:01.409 INFO:tasks.rados.rados.0.smithi084.stdout:2275: expect (ObjNum 556 snap 169 seq_num 556) 2024-09-20T22:51:01.418 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:51:01.439 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:01.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:01.440 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:51:01.541 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:51:01.562 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:01.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:01.563 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:51:01.610 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-20T22:51:01.614 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:01 smithi084 ceph-mon[40024]: pgmap v771: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 542 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 18 MiB/s rd, 1.8 MiB/s wr, 17 op/s 2024-09-20T22:51:01.614 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:01 smithi084 ceph-mon[40024]: osdmap e580: 8 total, 8 up, 8 in 2024-09-20T22:51:01.614 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:01 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/3270805028' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:51:01.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:01 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:51:01.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:01 smithi084 ceph-mon[29774]: pgmap v771: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 542 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 18 MiB/s rd, 1.8 MiB/s wr, 17 op/s 2024-09-20T22:51:01.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:01 smithi084 ceph-mon[29774]: osdmap e580: 8 total, 8 up, 8 in 2024-09-20T22:51:01.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:01 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/3270805028' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:51:01.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:01 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:51:01.637 INFO:tasks.rados.rados.0.smithi084.stdout:2276: expect (ObjNum 771 snap 248 seq_num 771) 2024-09-20T22:51:01.664 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:51:01.687 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:01.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:01.688 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:51:01.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:01 smithi154 ceph-mon[33264]: pgmap v771: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 542 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 18 MiB/s rd, 1.8 MiB/s wr, 17 op/s 2024-09-20T22:51:01.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:01 smithi154 ceph-mon[33264]: osdmap e580: 8 total, 8 up, 8 in 2024-09-20T22:51:01.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:01 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/3270805028' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:51:01.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:01 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:51:01.776 INFO:tasks.rados.rados.0.smithi084.stdout:2275: done (2 left) 2024-09-20T22:51:01.776 INFO:tasks.rados.rados.0.smithi084.stdout:2276: done (1 left) 2024-09-20T22:51:01.776 INFO:tasks.rados.rados.0.smithi084.stdout:2277: done (0 left) 2024-09-20T22:51:01.776 INFO:tasks.rados.rados.0.smithi084.stdout:2278: rollback oid 34 current snap is 253 2024-09-20T22:51:01.776 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 34 to 232 2024-09-20T22:51:01.776 INFO:tasks.rados.rados.0.smithi084.stdout:2279: write (excl) oid 5 current snap is 253 2024-09-20T22:51:01.776 INFO:tasks.rados.rados.0.smithi084.stdout:2279: seq_num 784 ranges {776688=579575,1894471=798599} 2024-09-20T22:51:01.783 INFO:tasks.rados.rados.0.smithi084.stdout:2279: writing smithi08477203-5 from 776688 to 1356263 tid 1 2024-09-20T22:51:01.788 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:51:01.791 INFO:tasks.rados.rados.0.smithi084.stdout:2279: writing smithi08477203-5 from 1894471 to 2693070 tid 2 2024-09-20T22:51:01.791 INFO:tasks.rados.rados.0.smithi084.stdout:2280: snap_remove snap 178 2024-09-20T22:51:01.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:01.811 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:01.811 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:51:01.912 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:51:01.933 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:01.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:01.933 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:51:02.034 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:51:02.055 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:02.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:02.056 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:51:02.156 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:51:02.178 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:02.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:02.179 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:51:02.280 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:51:02.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:02.303 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:02.303 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:51:02.315 INFO:tasks.rados.rados.0.smithi084.stdout:2280: done (2 left) 2024-09-20T22:51:02.316 INFO:tasks.rados.rados.0.smithi084.stdout:2281: read oid 25 snap -1 2024-09-20T22:51:02.316 INFO:tasks.rados.rados.0.smithi084.stdout:2281: expect (ObjNum 688 snap 212 seq_num 688) 2024-09-20T22:51:02.316 INFO:tasks.rados.rados.0.smithi084.stdout:2282: write oid 40 current snap is 253 2024-09-20T22:51:02.316 INFO:tasks.rados.rados.0.smithi084.stdout:2282: seq_num 785 ranges {793495=654824,1952953=658610,3157167=36328} 2024-09-20T22:51:02.324 INFO:tasks.rados.rados.0.smithi084.stdout:2282: writing smithi08477203-40 from 793495 to 1448319 tid 1 2024-09-20T22:51:02.331 INFO:tasks.rados.rados.0.smithi084.stdout:2282: writing smithi08477203-40 from 1952953 to 2611563 tid 2 2024-09-20T22:51:02.334 INFO:tasks.rados.rados.0.smithi084.stdout:2282: writing smithi08477203-40 from 3157167 to 3193495 tid 3 2024-09-20T22:51:02.334 INFO:tasks.rados.rados.0.smithi084.stdout:2283: read oid 44 snap -1 2024-09-20T22:51:02.334 INFO:tasks.rados.rados.0.smithi084.stdout:2283: expect (ObjNum 758 snap 241 seq_num 758) 2024-09-20T22:51:02.334 INFO:tasks.rados.rados.0.smithi084.stdout:2284: snap_create 2024-09-20T22:51:02.334 INFO:tasks.rados.rados.0.smithi084.stdout:2278: finishing rollback tid 0 to smithi08477203-34 2024-09-20T22:51:02.335 INFO:tasks.rados.rados.0.smithi084.stdout:2278: finishing rollback tid 1 to smithi08477203-34 2024-09-20T22:51:02.335 INFO:tasks.rados.rados.0.smithi084.stdout:2278: finishing rollback tid 2 to smithi08477203-34 2024-09-20T22:51:02.335 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 34 v 1216 (ObjNum 622 snap 186 seq_num 622) dirty exists 2024-09-20T22:51:02.335 INFO:tasks.rados.rados.0.smithi084.stdout:2279: finishing write tid 1 to smithi08477203-5 2024-09-20T22:51:02.335 INFO:tasks.rados.rados.0.smithi084.stdout:2279: finishing write tid 2 to smithi08477203-5 2024-09-20T22:51:02.335 INFO:tasks.rados.rados.0.smithi084.stdout:2279: finishing write tid 3 to smithi08477203-5 2024-09-20T22:51:02.335 INFO:tasks.rados.rados.0.smithi084.stdout:2279: finishing write tid 4 to smithi08477203-5 2024-09-20T22:51:02.335 INFO:tasks.rados.rados.0.smithi084.stdout:2279: oid 5 updating version 0 to 741 2024-09-20T22:51:02.335 INFO:tasks.rados.rados.0.smithi084.stdout:2279: oid 5 version 741 is already newer than 740 2024-09-20T22:51:02.335 INFO:tasks.rados.rados.0.smithi084.stdout:2279: oid 5 version 741 is already newer than 739 2024-09-20T22:51:02.335 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 5 v 741 (ObjNum 784 snap 253 seq_num 784) dirty exists 2024-09-20T22:51:02.335 INFO:tasks.rados.rados.0.smithi084.stdout:2279: left oid 5 (ObjNum 784 snap 253 seq_num 784) 2024-09-20T22:51:02.335 INFO:tasks.rados.rados.0.smithi084.stdout:2282: finishing write tid 1 to smithi08477203-40 2024-09-20T22:51:02.335 INFO:tasks.rados.rados.0.smithi084.stdout:2281: expect (ObjNum 688 snap 212 seq_num 688) 2024-09-20T22:51:02.404 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:51:02.413 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-20T22:51:02.413 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-20T22:51:02.413 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph tell osd.7 'trim stale osdmaps' 2024-09-20T22:51:02.425 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:02.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:02.426 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:51:02.452 INFO:tasks.rados.rados.0.smithi084.stdout:2282: finishing write tid 2 to smithi08477203-40 2024-09-20T22:51:02.452 INFO:tasks.rados.rados.0.smithi084.stdout:2282: finishing write tid 3 to smithi08477203-40 2024-09-20T22:51:02.452 INFO:tasks.rados.rados.0.smithi084.stdout:2282: finishing write tid 4 to smithi08477203-40 2024-09-20T22:51:02.452 INFO:tasks.rados.rados.0.smithi084.stdout:2282: finishing write tid 5 to smithi08477203-40 2024-09-20T22:51:02.452 INFO:tasks.rados.rados.0.smithi084.stdout:2282: oid 40 updating version 0 to 750 2024-09-20T22:51:02.452 INFO:tasks.rados.rados.0.smithi084.stdout:2282: oid 40 updating version 750 to 752 2024-09-20T22:51:02.452 INFO:tasks.rados.rados.0.smithi084.stdout:2282: oid 40 updating version 752 to 753 2024-09-20T22:51:02.452 INFO:tasks.rados.rados.0.smithi084.stdout:2282: oid 40 version 753 is already newer than 751 2024-09-20T22:51:02.452 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 40 v 753 (ObjNum 785 snap 253 seq_num 785) dirty exists 2024-09-20T22:51:02.452 INFO:tasks.rados.rados.0.smithi084.stdout:2282: left oid 40 (ObjNum 785 snap 253 seq_num 785) 2024-09-20T22:51:02.452 INFO:tasks.rados.rados.0.smithi084.stdout:2283: expect (ObjNum 758 snap 241 seq_num 758) 2024-09-20T22:51:02.527 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:51:02.549 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:02.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:02.549 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:51:02.614 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:02 smithi084 ceph-mon[40024]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:51:02.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:02 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T22:51:02.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:02 smithi084 ceph-mon[40024]: osdmap e581: 8 total, 8 up, 8 in 2024-09-20T22:51:02.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:02 smithi084 ceph-mon[40024]: 6.1b scrub starts 2024-09-20T22:51:02.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:02 smithi084 ceph-mon[40024]: 6.1b scrub ok 2024-09-20T22:51:02.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:02 smithi084 ceph-mon[29774]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:51:02.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:02 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T22:51:02.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:02 smithi084 ceph-mon[29774]: osdmap e581: 8 total, 8 up, 8 in 2024-09-20T22:51:02.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:02 smithi084 ceph-mon[29774]: 6.1b scrub starts 2024-09-20T22:51:02.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:02 smithi084 ceph-mon[29774]: 6.1b scrub ok 2024-09-20T22:51:02.650 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:51:02.671 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:02.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:02.672 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:51:02.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:02 smithi154 ceph-mon[33264]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:51:02.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:02 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T22:51:02.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:02 smithi154 ceph-mon[33264]: osdmap e581: 8 total, 8 up, 8 in 2024-09-20T22:51:02.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:02 smithi154 ceph-mon[33264]: 6.1b scrub starts 2024-09-20T22:51:02.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:02 smithi154 ceph-mon[33264]: 6.1b scrub ok 2024-09-20T22:51:02.773 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:51:02.795 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:02.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:02.796 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:51:02.896 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:51:02.917 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:02.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:02.918 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:51:03.018 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:51:03.037 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-20T22:51:03.037 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:51:03.041 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:03.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:03.042 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:51:03.142 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:51:03.165 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:03.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:03.165 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:51:03.254 INFO:tasks.rados.rados.0.smithi084.stdout:2278: done (5 left) 2024-09-20T22:51:03.255 INFO:tasks.rados.rados.0.smithi084.stdout:2279: done (4 left) 2024-09-20T22:51:03.255 INFO:tasks.rados.rados.0.smithi084.stdout:2281: done (3 left) 2024-09-20T22:51:03.255 INFO:tasks.rados.rados.0.smithi084.stdout:2282: done (2 left) 2024-09-20T22:51:03.255 INFO:tasks.rados.rados.0.smithi084.stdout:2283: done (1 left) 2024-09-20T22:51:03.255 INFO:tasks.rados.rados.0.smithi084.stdout:2284: done (0 left) 2024-09-20T22:51:03.255 INFO:tasks.rados.rados.0.smithi084.stdout:2285: write (excl) oid 6 current snap is 254 2024-09-20T22:51:03.255 INFO:tasks.rados.rados.0.smithi084.stdout:2285: seq_num 786 ranges {410302=730073,1895082=402268,2951884=258418} 2024-09-20T22:51:03.261 INFO:tasks.rados.rados.0.smithi084.stdout:2285: writing smithi08477203-6 from 410302 to 1140375 tid 1 2024-09-20T22:51:03.266 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:51:03.267 INFO:tasks.rados.rados.0.smithi084.stdout:2285: writing smithi08477203-6 from 1895082 to 2297350 tid 2 2024-09-20T22:51:03.272 INFO:tasks.rados.rados.0.smithi084.stdout:2285: writing smithi08477203-6 from 2951884 to 3210302 tid 3 2024-09-20T22:51:03.273 INFO:tasks.rados.rados.0.smithi084.stdout:2286: read oid 27 snap -1 2024-09-20T22:51:03.273 INFO:tasks.rados.rados.0.smithi084.stdout:2286: expect (ObjNum 714 snap 224 seq_num 714) 2024-09-20T22:51:03.273 INFO:tasks.rados.rados.0.smithi084.stdout:2287: write (excl) oid 35 current snap is 254 2024-09-20T22:51:03.273 INFO:tasks.rados.rados.0.smithi084.stdout:2287: seq_num 787 ranges {427109=405322,1553564=545926,2579307=647802} 2024-09-20T22:51:03.277 INFO:tasks.rados.rados.0.smithi084.stdout:2287: writing smithi08477203-35 from 427109 to 832431 tid 1 2024-09-20T22:51:03.284 INFO:tasks.rados.rados.0.smithi084.stdout:2287: writing smithi08477203-35 from 1553564 to 2099490 tid 2 2024-09-20T22:51:03.289 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:03.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:03.290 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:51:03.291 INFO:tasks.rados.rados.0.smithi084.stdout:2287: writing smithi08477203-35 from 2579307 to 3227109 tid 3 2024-09-20T22:51:03.291 INFO:tasks.rados.rados.0.smithi084.stdout:2288: copy_from oid 31 from oid 25 current snap is 254 2024-09-20T22:51:03.291 INFO:tasks.rados.rados.0.smithi084.stdout:2285: finishing write tid 1 to smithi08477203-6 2024-09-20T22:51:03.292 INFO:tasks.rados.rados.0.smithi084.stdout:2285: finishing write tid 2 to smithi08477203-6 2024-09-20T22:51:03.292 INFO:tasks.rados.rados.0.smithi084.stdout:2285: finishing write tid 3 to smithi08477203-6 2024-09-20T22:51:03.292 INFO:tasks.rados.rados.0.smithi084.stdout:2285: finishing write tid 4 to smithi08477203-6 2024-09-20T22:51:03.292 INFO:tasks.rados.rados.0.smithi084.stdout:2285: finishing write tid 5 to smithi08477203-6 2024-09-20T22:51:03.292 INFO:tasks.rados.rados.0.smithi084.stdout:2285: oid 6 updating version 0 to 764 2024-09-20T22:51:03.292 INFO:tasks.rados.rados.0.smithi084.stdout:2285: oid 6 version 764 is already newer than 762 2024-09-20T22:51:03.292 INFO:tasks.rados.rados.0.smithi084.stdout:2285: oid 6 version 764 is already newer than 763 2024-09-20T22:51:03.292 INFO:tasks.rados.rados.0.smithi084.stdout:2285: oid 6 version 764 is already newer than 761 2024-09-20T22:51:03.292 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 6 v 764 (ObjNum 786 snap 254 seq_num 786) dirty exists 2024-09-20T22:51:03.292 INFO:tasks.rados.rados.0.smithi084.stdout:2285: left oid 6 (ObjNum 786 snap 254 seq_num 786) 2024-09-20T22:51:03.292 INFO:tasks.rados.rados.0.smithi084.stdout:2286: expect (ObjNum 714 snap 224 seq_num 714) 2024-09-20T22:51:03.356 INFO:tasks.rados.rados.0.smithi084.stdout:2287: finishing write tid 1 to smithi08477203-35 2024-09-20T22:51:03.357 INFO:tasks.rados.rados.0.smithi084.stdout:2287: finishing write tid 2 to smithi08477203-35 2024-09-20T22:51:03.357 INFO:tasks.rados.rados.0.smithi084.stdout:2287: finishing write tid 3 to smithi08477203-35 2024-09-20T22:51:03.357 INFO:tasks.rados.rados.0.smithi084.stdout:2287: finishing write tid 4 to smithi08477203-35 2024-09-20T22:51:03.357 INFO:tasks.rados.rados.0.smithi084.stdout:2287: finishing write tid 5 to smithi08477203-35 2024-09-20T22:51:03.357 INFO:tasks.rados.rados.0.smithi084.stdout:2287: oid 35 updating version 0 to 745 2024-09-20T22:51:03.357 INFO:tasks.rados.rados.0.smithi084.stdout:2287: oid 35 version 745 is already newer than 742 2024-09-20T22:51:03.357 INFO:tasks.rados.rados.0.smithi084.stdout:2287: oid 35 version 745 is already newer than 744 2024-09-20T22:51:03.357 INFO:tasks.rados.rados.0.smithi084.stdout:2287: oid 35 version 745 is already newer than 743 2024-09-20T22:51:03.357 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 35 v 745 (ObjNum 787 snap 254 seq_num 787) dirty exists 2024-09-20T22:51:03.357 INFO:tasks.rados.rados.0.smithi084.stdout:2287: left oid 35 (ObjNum 787 snap 254 seq_num 787) 2024-09-20T22:51:03.357 INFO:tasks.rados.rados.0.smithi084.stdout:2288: finishing copy_from racing read to smithi08477203-31 2024-09-20T22:51:03.357 INFO:tasks.rados.rados.0.smithi084.stdout:2288: finishing copy_from to smithi08477203-31 2024-09-20T22:51:03.358 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 31 v 1305 (ObjNum 688 snap 212 seq_num 688) dirty exists 2024-09-20T22:51:03.358 INFO:tasks.rados.rados.0.smithi084.stdout:2285: done (3 left) 2024-09-20T22:51:03.358 INFO:tasks.rados.rados.0.smithi084.stdout:2286: done (2 left) 2024-09-20T22:51:03.358 INFO:tasks.rados.rados.0.smithi084.stdout:2287: done (1 left) 2024-09-20T22:51:03.358 INFO:tasks.rados.rados.0.smithi084.stdout:2288: done (0 left) 2024-09-20T22:51:03.358 INFO:tasks.rados.rados.0.smithi084.stdout:2289: rollback oid 12 current snap is 254 2024-09-20T22:51:03.358 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 12 to 189 2024-09-20T22:51:03.358 INFO:tasks.rados.rados.0.smithi084.stdout:2290: read oid 22 snap -1 2024-09-20T22:51:03.358 INFO:tasks.rados.rados.0.smithi084.stdout:2290: expect deleted 2024-09-20T22:51:03.358 INFO:tasks.rados.rados.0.smithi084.stdout:2291: write (excl) oid 20 current snap is 254 2024-09-20T22:51:03.358 INFO:tasks.rados.rados.0.smithi084.stdout:2291: seq_num 788 ranges {443916=480571,1612046=405937,2606730=495052} 2024-09-20T22:51:03.362 INFO:tasks.rados.rados.0.smithi084.stdout:2291: writing smithi08477203-20 from 443916 to 924487 tid 1 2024-09-20T22:51:03.368 INFO:tasks.rados.rados.0.smithi084.stdout:2291: writing smithi08477203-20 from 1612046 to 2017983 tid 2 2024-09-20T22:51:03.374 INFO:tasks.rados.rados.0.smithi084.stdout:2291: writing smithi08477203-20 from 2606730 to 3101782 tid 3 2024-09-20T22:51:03.374 INFO:tasks.rados.rados.0.smithi084.stdout:2292: snap_remove snap 217 2024-09-20T22:51:03.390 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:51:03.412 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:03.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:03.413 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:51:03.514 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:51:03.535 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:03.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:03.536 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:51:03.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:03 smithi084 ceph-mon[40024]: 6.12 scrub starts 2024-09-20T22:51:03.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:03 smithi084 ceph-mon[40024]: 6.12 scrub ok 2024-09-20T22:51:03.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:03 smithi084 ceph-mon[40024]: 7.6 scrub starts 2024-09-20T22:51:03.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:03 smithi084 ceph-mon[40024]: pgmap v774: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 542 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 18 MiB/s rd, 1.8 MiB/s wr, 17 op/s 2024-09-20T22:51:03.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:03 smithi084 ceph-mon[40024]: 7.6 scrub ok 2024-09-20T22:51:03.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:03 smithi084 ceph-mon[40024]: osdmap e582: 8 total, 8 up, 8 in 2024-09-20T22:51:03.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:03 smithi084 ceph-mon[40024]: 6.e scrub starts 2024-09-20T22:51:03.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:03 smithi084 ceph-mon[40024]: 6.e scrub ok 2024-09-20T22:51:03.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:03 smithi084 ceph-mon[40024]: osdmap e583: 8 total, 8 up, 8 in 2024-09-20T22:51:03.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:03 smithi084 ceph-mon[29774]: 6.12 scrub starts 2024-09-20T22:51:03.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:03 smithi084 ceph-mon[29774]: 6.12 scrub ok 2024-09-20T22:51:03.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:03 smithi084 ceph-mon[29774]: 7.6 scrub starts 2024-09-20T22:51:03.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:03 smithi084 ceph-mon[29774]: pgmap v774: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 542 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 18 MiB/s rd, 1.8 MiB/s wr, 17 op/s 2024-09-20T22:51:03.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:03 smithi084 ceph-mon[29774]: 7.6 scrub ok 2024-09-20T22:51:03.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:03 smithi084 ceph-mon[29774]: osdmap e582: 8 total, 8 up, 8 in 2024-09-20T22:51:03.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:03 smithi084 ceph-mon[29774]: 6.e scrub starts 2024-09-20T22:51:03.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:03 smithi084 ceph-mon[29774]: 6.e scrub ok 2024-09-20T22:51:03.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:03 smithi084 ceph-mon[29774]: osdmap e583: 8 total, 8 up, 8 in 2024-09-20T22:51:03.637 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:51:03.658 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:03.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:03.659 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:51:03.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:03 smithi154 ceph-mon[33264]: 6.12 scrub starts 2024-09-20T22:51:03.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:03 smithi154 ceph-mon[33264]: 6.12 scrub ok 2024-09-20T22:51:03.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:03 smithi154 ceph-mon[33264]: 7.6 scrub starts 2024-09-20T22:51:03.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:03 smithi154 ceph-mon[33264]: pgmap v774: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 542 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 18 MiB/s rd, 1.8 MiB/s wr, 17 op/s 2024-09-20T22:51:03.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:03 smithi154 ceph-mon[33264]: 7.6 scrub ok 2024-09-20T22:51:03.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:03 smithi154 ceph-mon[33264]: osdmap e582: 8 total, 8 up, 8 in 2024-09-20T22:51:03.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:03 smithi154 ceph-mon[33264]: 6.e scrub starts 2024-09-20T22:51:03.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:03 smithi154 ceph-mon[33264]: 6.e scrub ok 2024-09-20T22:51:03.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:03 smithi154 ceph-mon[33264]: osdmap e583: 8 total, 8 up, 8 in 2024-09-20T22:51:03.759 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.3 dump_historic_ops 2024-09-20T22:51:03.760 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:51:03.781 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:03.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:03.782 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:51:03.801 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:51:03.805 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:51:03.883 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:51:03.904 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:03.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:03.905 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:51:04.005 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:51:04.027 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:04.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:04.028 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:51:04.129 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:51:04.150 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:04.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:04.151 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:51:04.251 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:51:04.273 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:04.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:04.274 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:51:04.325 INFO:tasks.rados.rados.0.smithi084.stdout:2292: done (3 left) 2024-09-20T22:51:04.325 INFO:tasks.rados.rados.0.smithi084.stdout:2293: delete oid 10 current snap is 254 2024-09-20T22:51:04.325 INFO:tasks.rados.rados.0.smithi084.stdout:2289: finishing rollback tid 0 to smithi08477203-12 2024-09-20T22:51:04.325 INFO:tasks.rados.rados.0.smithi084.stdout:2289: finishing rollback tid 1 to smithi08477203-12 2024-09-20T22:51:04.325 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 12 v 1307 (ObjNum 305 snap 0 seq_num 1) dirty dne 2024-09-20T22:51:04.325 INFO:tasks.rados.rados.0.smithi084.stdout:2291: finishing write tid 1 to smithi08477203-20 2024-09-20T22:51:04.325 INFO:tasks.rados.rados.0.smithi084.stdout:2291: finishing write tid 2 to smithi08477203-20 2024-09-20T22:51:04.325 INFO:tasks.rados.rados.0.smithi084.stdout:2291: finishing write tid 3 to smithi08477203-20 2024-09-20T22:51:04.325 INFO:tasks.rados.rados.0.smithi084.stdout:2291: finishing write tid 4 to smithi08477203-20 2024-09-20T22:51:04.325 INFO:tasks.rados.rados.0.smithi084.stdout:2291: finishing write tid 5 to smithi08477203-20 2024-09-20T22:51:04.325 INFO:tasks.rados.rados.0.smithi084.stdout:2291: oid 20 updating version 0 to 1223 2024-09-20T22:51:04.326 INFO:tasks.rados.rados.0.smithi084.stdout:2291: oid 20 version 1223 is already newer than 1222 2024-09-20T22:51:04.326 INFO:tasks.rados.rados.0.smithi084.stdout:2291: oid 20 version 1223 is already newer than 1221 2024-09-20T22:51:04.326 INFO:tasks.rados.rados.0.smithi084.stdout:2291: oid 20 updating version 1223 to 1224 2024-09-20T22:51:04.326 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 20 v 1224 (ObjNum 788 snap 254 seq_num 788) dirty exists 2024-09-20T22:51:04.326 INFO:tasks.rados.rados.0.smithi084.stdout:2291: left oid 20 (ObjNum 788 snap 254 seq_num 788) 2024-09-20T22:51:04.341 INFO:tasks.rados.rados.0.smithi084.stdout:2289: done (3 left) 2024-09-20T22:51:04.341 INFO:tasks.rados.rados.0.smithi084.stdout:2290: done (2 left) 2024-09-20T22:51:04.341 INFO:tasks.rados.rados.0.smithi084.stdout:2291: done (1 left) 2024-09-20T22:51:04.341 INFO:tasks.rados.rados.0.smithi084.stdout:2293: done (0 left) 2024-09-20T22:51:04.341 INFO:tasks.rados.rados.0.smithi084.stdout:2294: read oid 15 snap -1 2024-09-20T22:51:04.341 INFO:tasks.rados.rados.0.smithi084.stdout:2294: expect deleted 2024-09-20T22:51:04.341 INFO:tasks.rados.rados.0.smithi084.stdout:2295: write (excl) oid 44 current snap is 254 2024-09-20T22:51:04.341 INFO:tasks.rados.rados.0.smithi084.stdout:2295: seq_num 790 ranges {477530=631069,1612657=409606,2545223=517596} 2024-09-20T22:51:04.347 INFO:tasks.rados.rados.0.smithi084.stdout:2295: writing smithi08477203-44 from 477530 to 1108599 tid 1 2024-09-20T22:51:04.352 INFO:tasks.rados.rados.0.smithi084.stdout:2295: writing smithi08477203-44 from 1612657 to 2022263 tid 2 2024-09-20T22:51:04.359 INFO:tasks.rados.rados.0.smithi084.stdout:2295: writing smithi08477203-44 from 2545223 to 3062819 tid 3 2024-09-20T22:51:04.359 INFO:tasks.rados.rados.0.smithi084.stdout:2296: copy_from oid 38 from oid 1 current snap is 254 2024-09-20T22:51:04.359 INFO:tasks.rados.rados.0.smithi084.stdout:2294: done (2 left) 2024-09-20T22:51:04.359 INFO:tasks.rados.rados.0.smithi084.stdout:2297: read oid 28 snap 216 2024-09-20T22:51:04.359 INFO:tasks.rados.rados.0.smithi084.stdout:2297: expect (ObjNum 656 snap 198 seq_num 656) 2024-09-20T22:51:04.359 INFO:tasks.rados.rados.0.smithi084.stdout:2298: read oid 48 snap -1 2024-09-20T22:51:04.359 INFO:tasks.rados.rados.0.smithi084.stdout:2298: expect (ObjNum 759 snap 243 seq_num 759) 2024-09-20T22:51:04.359 INFO:tasks.rados.rados.0.smithi084.stdout:2299: write (excl) oid 20 current snap is 254 2024-09-20T22:51:04.359 INFO:tasks.rados.rados.0.smithi084.stdout:2295: finishing write tid 1 to smithi08477203-44 2024-09-20T22:51:04.359 INFO:tasks.rados.rados.0.smithi084.stdout:2295: finishing write tid 2 to smithi08477203-44 2024-09-20T22:51:04.359 INFO:tasks.rados.rados.0.smithi084.stdout:2299: seq_num 791 ranges {494337=422671,1387492=553264,2688999=445221} 2024-09-20T22:51:04.363 INFO:tasks.rados.rados.0.smithi084.stdout:2299: writing smithi08477203-20 from 494337 to 917008 tid 1 2024-09-20T22:51:04.369 INFO:tasks.rados.rados.0.smithi084.stdout:2299: writing smithi08477203-20 from 1387492 to 1940756 tid 2 2024-09-20T22:51:04.375 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:51:04.376 INFO:tasks.rados.rados.0.smithi084.stdout:2299: writing smithi08477203-20 from 2688999 to 3134220 tid 3 2024-09-20T22:51:04.376 INFO:tasks.rados.rados.0.smithi084.stdout:2300: write oid 18 current snap is 254 2024-09-20T22:51:04.376 INFO:tasks.rados.rados.0.smithi084.stdout:2300: seq_num 792 ranges {511144=497920,1445974=413275,2316422=656493} 2024-09-20T22:51:04.382 INFO:tasks.rados.rados.0.smithi084.stdout:2300: writing smithi08477203-18 from 511144 to 1009064 tid 1 2024-09-20T22:51:04.386 INFO:tasks.rados.rados.0.smithi084.stdout:2300: writing smithi08477203-18 from 1445974 to 1859249 tid 2 2024-09-20T22:51:04.393 INFO:tasks.rados.rados.0.smithi084.stdout:2300: writing smithi08477203-18 from 2316422 to 2972915 tid 3 2024-09-20T22:51:04.393 INFO:tasks.rados.rados.0.smithi084.stdout:2301: read oid 7 snap -1 2024-09-20T22:51:04.393 INFO:tasks.rados.rados.0.smithi084.stdout:2301: expect (ObjNum 752 snap 238 seq_num 752) 2024-09-20T22:51:04.393 INFO:tasks.rados.rados.0.smithi084.stdout:2302: write (excl) oid 6 current snap is 254 2024-09-20T22:51:04.393 INFO:tasks.rados.rados.0.smithi084.stdout:2302: seq_num 793 ranges {527951=573169,1504456=556933,2743845=467765} 2024-09-20T22:51:04.399 INFO:tasks.rados.rados.0.smithi084.stdout:2302: writing smithi08477203-6 from 527951 to 1101120 tid 1 2024-09-20T22:51:04.400 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:04.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:04.402 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:51:04.404 INFO:tasks.rados.rados.0.smithi084.stdout:2302: writing smithi08477203-6 from 1504456 to 2061389 tid 2 2024-09-20T22:51:04.411 INFO:tasks.rados.rados.0.smithi084.stdout:2302: writing smithi08477203-6 from 2743845 to 3211610 tid 3 2024-09-20T22:51:04.411 INFO:tasks.rados.rados.0.smithi084.stdout:2303: write (excl) oid 19 current snap is 254 2024-09-20T22:51:04.411 INFO:tasks.rados.rados.0.smithi084.stdout:2298: expect (ObjNum 759 snap 243 seq_num 759) 2024-09-20T22:51:04.500 INFO:tasks.rados.rados.0.smithi084.stdout:2299: finishing write tid 1 to smithi08477203-20 2024-09-20T22:51:04.500 INFO:tasks.rados.rados.0.smithi084.stdout:2303: seq_num 794 ranges {544758=648418,1846585=416944,2771268=573490} 2024-09-20T22:51:04.502 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:51:04.506 INFO:tasks.rados.rados.0.smithi084.stdout:2303: writing smithi08477203-19 from 544758 to 1193176 tid 1 2024-09-20T22:51:04.512 INFO:tasks.rados.rados.0.smithi084.stdout:2303: writing smithi08477203-19 from 1846585 to 2263529 tid 2 2024-09-20T22:51:04.518 INFO:tasks.rados.rados.0.smithi084.stdout:2303: writing smithi08477203-19 from 2771268 to 3344758 tid 3 2024-09-20T22:51:04.518 INFO:tasks.rados.rados.0.smithi084.stdout:2298: done (8 left) 2024-09-20T22:51:04.518 INFO:tasks.rados.rados.0.smithi084.stdout:2304: rollback oid 16 current snap is 254 2024-09-20T22:51:04.518 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 16 to 216 2024-09-20T22:51:04.519 INFO:tasks.rados.rados.0.smithi084.stdout:2299: finishing write tid 2 to smithi08477203-20 2024-09-20T22:51:04.519 INFO:tasks.rados.rados.0.smithi084.stdout:2295: finishing write tid 3 to smithi08477203-44 2024-09-20T22:51:04.519 INFO:tasks.rados.rados.0.smithi084.stdout:2295: finishing write tid 4 to smithi08477203-44 2024-09-20T22:51:04.519 INFO:tasks.rados.rados.0.smithi084.stdout:2295: finishing write tid 5 to smithi08477203-44 2024-09-20T22:51:04.519 INFO:tasks.rados.rados.0.smithi084.stdout:2295: oid 44 updating version 0 to 1649 2024-09-20T22:51:04.519 INFO:tasks.rados.rados.0.smithi084.stdout:2295: oid 44 version 1649 is already newer than 1646 2024-09-20T22:51:04.519 INFO:tasks.rados.rados.0.smithi084.stdout:2295: oid 44 version 1649 is already newer than 1647 2024-09-20T22:51:04.519 INFO:tasks.rados.rados.0.smithi084.stdout:2295: oid 44 version 1649 is already newer than 1648 2024-09-20T22:51:04.519 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 44 v 1649 (ObjNum 790 snap 254 seq_num 790) dirty exists 2024-09-20T22:51:04.519 INFO:tasks.rados.rados.0.smithi084.stdout:2295: left oid 44 (ObjNum 790 snap 254 seq_num 790) 2024-09-20T22:51:04.519 INFO:tasks.rados.rados.0.smithi084.stdout:2297: expect (ObjNum 656 snap 198 seq_num 656) 2024-09-20T22:51:04.525 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:04.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:04.526 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:51:04.588 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:04 smithi084 ceph-mon[40024]: 4.1e scrub starts 2024-09-20T22:51:04.589 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:04 smithi084 ceph-mon[40024]: 4.1e scrub ok 2024-09-20T22:51:04.589 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:04 smithi084 ceph-mon[40024]: 4.1d scrub starts 2024-09-20T22:51:04.589 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:04 smithi084 ceph-mon[40024]: 4.1d scrub ok 2024-09-20T22:51:04.589 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:04 smithi084 ceph-mon[40024]: 5.1f scrub starts 2024-09-20T22:51:04.589 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:04 smithi084 ceph-mon[40024]: 5.1f scrub ok 2024-09-20T22:51:04.589 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:04 smithi084 ceph-mon[29774]: 4.1e scrub starts 2024-09-20T22:51:04.589 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:04 smithi084 ceph-mon[29774]: 4.1e scrub ok 2024-09-20T22:51:04.589 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:04 smithi084 ceph-mon[29774]: 4.1d scrub starts 2024-09-20T22:51:04.589 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:04 smithi084 ceph-mon[29774]: 4.1d scrub ok 2024-09-20T22:51:04.589 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:04 smithi084 ceph-mon[29774]: 5.1f scrub starts 2024-09-20T22:51:04.590 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:04 smithi084 ceph-mon[29774]: 5.1f scrub ok 2024-09-20T22:51:04.590 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:51:04.590 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:51:04.607 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:51:04.607 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:51:04.624 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:51:04.624 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:51:04.627 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:51:04.641 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-20T22:51:04.641 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T22:51:04.650 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:04.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:04.651 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:51:04.659 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:51:04.659 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:51:04.670 INFO:tasks.rados.rados.0.smithi084.stdout:2296: finishing copy_from racing read to smithi08477203-38 2024-09-20T22:51:04.670 INFO:tasks.rados.rados.0.smithi084.stdout:2299: finishing write tid 3 to smithi08477203-20 2024-09-20T22:51:04.670 INFO:tasks.rados.rados.0.smithi084.stdout:2299: finishing write tid 4 to smithi08477203-20 2024-09-20T22:51:04.670 INFO:tasks.rados.rados.0.smithi084.stdout:2299: finishing write tid 5 to smithi08477203-20 2024-09-20T22:51:04.670 INFO:tasks.rados.rados.0.smithi084.stdout:2299: oid 20 updating version 0 to 1225 2024-09-20T22:51:04.670 INFO:tasks.rados.rados.0.smithi084.stdout:2299: oid 20 updating version 1225 to 1228 2024-09-20T22:51:04.670 INFO:tasks.rados.rados.0.smithi084.stdout:2299: oid 20 version 1228 is already newer than 1226 2024-09-20T22:51:04.671 INFO:tasks.rados.rados.0.smithi084.stdout:2299: oid 20 version 1228 is already newer than 1227 2024-09-20T22:51:04.671 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 20 v 1228 (ObjNum 791 snap 254 seq_num 791) dirty exists 2024-09-20T22:51:04.671 INFO:tasks.rados.rados.0.smithi084.stdout:2299: left oid 20 (ObjNum 791 snap 254 seq_num 791) 2024-09-20T22:51:04.671 INFO:tasks.rados.rados.0.smithi084.stdout:2296: finishing copy_from to smithi08477203-38 2024-09-20T22:51:04.671 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 38 v 731 (ObjNum 470 snap 130 seq_num 470) dirty exists 2024-09-20T22:51:04.671 INFO:tasks.rados.rados.0.smithi084.stdout:2300: finishing write tid 1 to smithi08477203-18 2024-09-20T22:51:04.671 INFO:tasks.rados.rados.0.smithi084.stdout:2295: done (8 left) 2024-09-20T22:51:04.671 INFO:tasks.rados.rados.0.smithi084.stdout:2296: done (7 left) 2024-09-20T22:51:04.671 INFO:tasks.rados.rados.0.smithi084.stdout:2297: done (6 left) 2024-09-20T22:51:04.671 INFO:tasks.rados.rados.0.smithi084.stdout:2299: done (5 left) 2024-09-20T22:51:04.671 INFO:tasks.rados.rados.0.smithi084.stdout:2305: write (excl) oid 12 current snap is 254 2024-09-20T22:51:04.671 INFO:tasks.rados.rados.0.smithi084.stdout:2300: finishing write tid 2 to smithi08477203-18 2024-09-20T22:51:04.671 INFO:tasks.rados.rados.0.smithi084.stdout:2301: expect (ObjNum 752 snap 238 seq_num 752) 2024-09-20T22:51:04.675 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:51:04.675 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:51:04.692 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:51:04.692 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:51:04.709 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:51:04.709 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:51:04.752 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:51:04.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:04 smithi154 ceph-mon[33264]: 4.1e scrub starts 2024-09-20T22:51:04.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:04 smithi154 ceph-mon[33264]: 4.1e scrub ok 2024-09-20T22:51:04.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:04 smithi154 ceph-mon[33264]: 4.1d scrub starts 2024-09-20T22:51:04.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:04 smithi154 ceph-mon[33264]: 4.1d scrub ok 2024-09-20T22:51:04.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:04 smithi154 ceph-mon[33264]: 5.1f scrub starts 2024-09-20T22:51:04.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:04 smithi154 ceph-mon[33264]: 5.1f scrub ok 2024-09-20T22:51:04.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:04.773 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:04.773 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:51:04.850 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-20T22:51:04.873 INFO:teuthology.orchestra.run.smithi084.stderr:no valid command found; 10 closest matches: 2024-09-20T22:51:04.873 INFO:teuthology.orchestra.run.smithi084.stderr:0 2024-09-20T22:51:04.873 INFO:teuthology.orchestra.run.smithi084.stderr:1 2024-09-20T22:51:04.873 INFO:teuthology.orchestra.run.smithi084.stderr:2 2024-09-20T22:51:04.873 INFO:teuthology.orchestra.run.smithi084.stderr:abort 2024-09-20T22:51:04.873 INFO:teuthology.orchestra.run.smithi084.stderr:assert 2024-09-20T22:51:04.873 INFO:teuthology.orchestra.run.smithi084.stderr:bench [] [] [] [] 2024-09-20T22:51:04.873 INFO:teuthology.orchestra.run.smithi084.stderr:bluefs debug_inject_read_zeros 2024-09-20T22:51:04.874 INFO:teuthology.orchestra.run.smithi084.stderr:bluefs files list 2024-09-20T22:51:04.874 INFO:teuthology.orchestra.run.smithi084.stderr:bluefs stats 2024-09-20T22:51:04.874 INFO:teuthology.orchestra.run.smithi084.stderr:bluestore allocator dump block 2024-09-20T22:51:04.874 INFO:teuthology.orchestra.run.smithi084.stderr:Error EINVAL: invalid command 2024-09-20T22:51:04.874 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:51:04.895 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:04.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:04.896 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:51:04.908 INFO:tasks.rados.rados.0.smithi084.stdout:2300: finishing write tid 3 to smithi08477203-18 2024-09-20T22:51:04.908 INFO:tasks.rados.rados.0.smithi084.stdout:2300: finishing write tid 4 to smithi08477203-18 2024-09-20T22:51:04.908 INFO:tasks.rados.rados.0.smithi084.stdout:2305: seq_num 795 ranges {561565=723667,1905067=560602,3082338=279227} 2024-09-20T22:51:04.915 INFO:tasks.rados.rados.0.smithi084.stdout:2305: writing smithi08477203-12 from 561565 to 1285232 tid 1 2024-09-20T22:51:04.921 INFO:tasks.rados.rados.0.smithi084.stdout:2305: writing smithi08477203-12 from 1905067 to 2465669 tid 2 2024-09-20T22:51:04.926 INFO:tasks.rados.rados.0.smithi084.stdout:2305: writing smithi08477203-12 from 3082338 to 3361565 tid 3 2024-09-20T22:51:04.927 INFO:tasks.rados.rados.0.smithi084.stdout:2301: done (5 left) 2024-09-20T22:51:04.927 INFO:tasks.rados.rados.0.smithi084.stdout:2306: write (excl) oid 38 current snap is 254 2024-09-20T22:51:04.927 INFO:tasks.rados.rados.0.smithi084.stdout:2300: finishing write tid 5 to smithi08477203-18 2024-09-20T22:51:04.927 INFO:tasks.rados.rados.0.smithi084.stdout:2300: oid 18 updating version 0 to 1221 2024-09-20T22:51:04.927 INFO:tasks.rados.rados.0.smithi084.stdout:2300: oid 18 updating version 1221 to 1224 2024-09-20T22:51:04.927 INFO:tasks.rados.rados.0.smithi084.stdout:2300: oid 18 version 1224 is already newer than 1222 2024-09-20T22:51:04.927 INFO:tasks.rados.rados.0.smithi084.stdout:2300: oid 18 version 1224 is already newer than 1223 2024-09-20T22:51:04.927 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 18 v 1224 (ObjNum 792 snap 254 seq_num 792) dirty exists 2024-09-20T22:51:04.927 INFO:tasks.rados.rados.0.smithi084.stdout:2300: left oid 18 (ObjNum 792 snap 254 seq_num 792) 2024-09-20T22:51:04.927 INFO:tasks.rados.rados.0.smithi084.stdout:2302: finishing write tid 1 to smithi08477203-6 2024-09-20T22:51:04.927 INFO:tasks.rados.rados.0.smithi084.stdout:2306: seq_num 796 ranges {578372=798916,1963549=420613,2826114=417934} 2024-09-20T22:51:04.934 INFO:tasks.rados.rados.0.smithi084.stdout:2306: writing smithi08477203-38 from 578372 to 1377288 tid 1 2024-09-20T22:51:04.939 INFO:tasks.rados.rados.0.smithi084.stdout:2306: writing smithi08477203-38 from 1963549 to 2384162 tid 2 2024-09-20T22:51:04.945 INFO:tasks.rados.rados.0.smithi084.stdout:2306: writing smithi08477203-38 from 2826114 to 3244048 tid 3 2024-09-20T22:51:04.945 INFO:tasks.rados.rados.0.smithi084.stdout:2300: done (5 left) 2024-09-20T22:51:04.945 INFO:tasks.rados.rados.0.smithi084.stdout:2307: delete oid 49 current snap is 254 2024-09-20T22:51:04.945 INFO:tasks.rados.rados.0.smithi084.stdout:2302: finishing write tid 2 to smithi08477203-6 2024-09-20T22:51:04.945 INFO:tasks.rados.rados.0.smithi084.stdout:2302: finishing write tid 3 to smithi08477203-6 2024-09-20T22:51:04.945 INFO:tasks.rados.rados.0.smithi084.stdout:2302: finishing write tid 4 to smithi08477203-6 2024-09-20T22:51:04.945 INFO:tasks.rados.rados.0.smithi084.stdout:2302: finishing write tid 5 to smithi08477203-6 2024-09-20T22:51:04.945 INFO:tasks.rados.rados.0.smithi084.stdout:2302: oid 6 updating version 0 to 768 2024-09-20T22:51:04.945 INFO:tasks.rados.rados.0.smithi084.stdout:2302: oid 6 version 768 is already newer than 765 2024-09-20T22:51:04.945 INFO:tasks.rados.rados.0.smithi084.stdout:2302: oid 6 version 768 is already newer than 767 2024-09-20T22:51:04.945 INFO:tasks.rados.rados.0.smithi084.stdout:2302: oid 6 version 768 is already newer than 766 2024-09-20T22:51:04.945 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 6 v 768 (ObjNum 793 snap 254 seq_num 793) dirty exists 2024-09-20T22:51:04.945 INFO:tasks.rados.rados.0.smithi084.stdout:2302: left oid 6 (ObjNum 793 snap 254 seq_num 793) 2024-09-20T22:51:04.946 INFO:tasks.rados.rados.0.smithi084.stdout:2303: finishing write tid 1 to smithi08477203-19 2024-09-20T22:51:04.946 INFO:tasks.rados.rados.0.smithi084.stdout:2303: finishing write tid 2 to smithi08477203-19 2024-09-20T22:51:04.946 INFO:tasks.rados.rados.0.smithi084.stdout:2304: finishing rollback tid 0 to smithi08477203-16 2024-09-20T22:51:04.946 INFO:tasks.rados.rados.0.smithi084.stdout:2304: finishing rollback tid 1 to smithi08477203-16 2024-09-20T22:51:04.946 INFO:tasks.rados.rados.0.smithi084.stdout:2304: finishing rollback tid 2 to smithi08477203-16 2024-09-20T22:51:04.946 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 16 v 435 (ObjNum 419 snap 105 seq_num 419) dirty exists 2024-09-20T22:51:04.946 INFO:tasks.rados.rados.0.smithi084.stdout:2303: finishing write tid 3 to smithi08477203-19 2024-09-20T22:51:04.946 INFO:tasks.rados.rados.0.smithi084.stdout:2303: finishing write tid 4 to smithi08477203-19 2024-09-20T22:51:04.946 INFO:tasks.rados.rados.0.smithi084.stdout:2303: finishing write tid 5 to smithi08477203-19 2024-09-20T22:51:04.946 INFO:tasks.rados.rados.0.smithi084.stdout:2303: oid 19 updating version 0 to 1653 2024-09-20T22:51:04.946 INFO:tasks.rados.rados.0.smithi084.stdout:2303: oid 19 version 1653 is already newer than 1650 2024-09-20T22:51:04.946 INFO:tasks.rados.rados.0.smithi084.stdout:2303: oid 19 version 1653 is already newer than 1651 2024-09-20T22:51:04.946 INFO:tasks.rados.rados.0.smithi084.stdout:2303: oid 19 version 1653 is already newer than 1652 2024-09-20T22:51:04.946 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 19 v 1653 (ObjNum 794 snap 254 seq_num 794) dirty exists 2024-09-20T22:51:04.947 INFO:tasks.rados.rados.0.smithi084.stdout:2303: left oid 19 (ObjNum 794 snap 254 seq_num 794) 2024-09-20T22:51:04.947 INFO:tasks.rados.rados.0.smithi084.stdout:2305: finishing write tid 1 to smithi08477203-12 2024-09-20T22:51:04.947 INFO:tasks.rados.rados.0.smithi084.stdout:2305: finishing write tid 2 to smithi08477203-12 2024-09-20T22:51:04.947 INFO:tasks.rados.rados.0.smithi084.stdout:2305: finishing write tid 3 to smithi08477203-12 2024-09-20T22:51:04.947 INFO:tasks.rados.rados.0.smithi084.stdout:2305: finishing write tid 4 to smithi08477203-12 2024-09-20T22:51:04.947 INFO:tasks.rados.rados.0.smithi084.stdout:2305: finishing write tid 5 to smithi08477203-12 2024-09-20T22:51:04.947 INFO:tasks.rados.rados.0.smithi084.stdout:2305: oid 12 updating version 0 to 1309 2024-09-20T22:51:04.947 INFO:tasks.rados.rados.0.smithi084.stdout:2305: oid 12 updating version 1309 to 1310 2024-09-20T22:51:04.947 INFO:tasks.rados.rados.0.smithi084.stdout:2305: oid 12 version 1310 is already newer than 1308 2024-09-20T22:51:04.947 INFO:tasks.rados.rados.0.smithi084.stdout:2305: oid 12 updating version 1310 to 1311 2024-09-20T22:51:04.947 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 12 v 1311 (ObjNum 795 snap 254 seq_num 795) dirty exists 2024-09-20T22:51:04.947 INFO:tasks.rados.rados.0.smithi084.stdout:2305: left oid 12 (ObjNum 795 snap 254 seq_num 795) 2024-09-20T22:51:04.947 INFO:tasks.rados.rados.0.smithi084.stdout:2306: finishing write tid 1 to smithi08477203-38 2024-09-20T22:51:04.947 INFO:tasks.rados.rados.0.smithi084.stdout:2306: finishing write tid 2 to smithi08477203-38 2024-09-20T22:51:04.948 INFO:tasks.rados.rados.0.smithi084.stdout:2302: done (5 left) 2024-09-20T22:51:04.948 INFO:tasks.rados.rados.0.smithi084.stdout:2303: done (4 left) 2024-09-20T22:51:04.948 INFO:tasks.rados.rados.0.smithi084.stdout:2304: done (3 left) 2024-09-20T22:51:04.949 INFO:tasks.rados.rados.0.smithi084.stdout:2305: done (2 left) 2024-09-20T22:51:04.949 INFO:tasks.rados.rados.0.smithi084.stdout:2307: done (1 left) 2024-09-20T22:51:04.949 INFO:tasks.rados.rados.0.smithi084.stdout:2308: read oid 42 snap -1 2024-09-20T22:51:04.949 INFO:tasks.rados.rados.0.smithi084.stdout:2308: expect (ObjNum 775 snap 248 seq_num 775) 2024-09-20T22:51:04.949 INFO:tasks.rados.rados.0.smithi084.stdout:2309: delete oid 49 current snap is 254 2024-09-20T22:51:04.949 INFO:tasks.rados.rados.0.smithi084.stdout:2309: done (2 left) 2024-09-20T22:51:04.949 INFO:tasks.rados.rados.0.smithi084.stdout:2310: rollback oid 45 current snap is 254 2024-09-20T22:51:04.949 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 45 to 189 2024-09-20T22:51:04.949 INFO:tasks.rados.rados.0.smithi084.stdout:2311: read oid 49 snap -1 2024-09-20T22:51:04.949 INFO:tasks.rados.rados.0.smithi084.stdout:2311: expect deleted 2024-09-20T22:51:04.949 INFO:tasks.rados.rados.0.smithi084.stdout:2312: snap_remove snap 176 2024-09-20T22:51:04.997 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:51:05.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:05.019 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:05.019 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:51:05.120 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:51:05.141 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:05.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:05.141 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:51:05.242 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:51:05.263 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:05.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:05.264 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:51:05.329 INFO:tasks.rados.rados.0.smithi084.stdout:2312: done (4 left) 2024-09-20T22:51:05.329 INFO:tasks.rados.rados.0.smithi084.stdout:2313: snap_create 2024-09-20T22:51:05.329 INFO:tasks.rados.rados.0.smithi084.stdout:2310: finishing rollback tid 0 to smithi08477203-45 2024-09-20T22:51:05.329 INFO:tasks.rados.rados.0.smithi084.stdout:2310: finishing rollback tid 1 to smithi08477203-45 2024-09-20T22:51:05.329 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 45 v 743 (ObjNum 0 snap 0 seq_num 304) dirty dne 2024-09-20T22:51:05.329 INFO:tasks.rados.rados.0.smithi084.stdout:2308: expect (ObjNum 775 snap 248 seq_num 775) 2024-09-20T22:51:05.365 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:51:05.386 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:05.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:05.387 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:51:05.465 INFO:tasks.rados.rados.0.smithi084.stdout:2306: finishing write tid 3 to smithi08477203-38 2024-09-20T22:51:05.465 INFO:tasks.rados.rados.0.smithi084.stdout:2306: finishing write tid 4 to smithi08477203-38 2024-09-20T22:51:05.465 INFO:tasks.rados.rados.0.smithi084.stdout:2306: finishing write tid 5 to smithi08477203-38 2024-09-20T22:51:05.465 INFO:tasks.rados.rados.0.smithi084.stdout:2306: oid 38 updating version 0 to 1313 2024-09-20T22:51:05.465 INFO:tasks.rados.rados.0.smithi084.stdout:2306: oid 38 version 1313 is already newer than 1312 2024-09-20T22:51:05.465 INFO:tasks.rados.rados.0.smithi084.stdout:2306: oid 38 updating version 1313 to 1314 2024-09-20T22:51:05.465 INFO:tasks.rados.rados.0.smithi084.stdout:2306: oid 38 updating version 1314 to 1315 2024-09-20T22:51:05.465 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 38 v 1315 (ObjNum 796 snap 254 seq_num 796) dirty exists 2024-09-20T22:51:05.465 INFO:tasks.rados.rados.0.smithi084.stdout:2306: left oid 38 (ObjNum 796 snap 254 seq_num 796) 2024-09-20T22:51:05.488 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:51:05.508 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:05.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:05.509 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:51:05.610 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:51:05.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:05 smithi084 ceph-mon[40024]: pgmap v777: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 556 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 17 MiB/s rd, 2.5 MiB/s wr, 25 op/s 2024-09-20T22:51:05.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:05 smithi084 ceph-mon[40024]: 5.6 scrub starts 2024-09-20T22:51:05.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:05 smithi084 ceph-mon[40024]: 5.6 scrub ok 2024-09-20T22:51:05.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:05 smithi084 ceph-mon[40024]: osdmap e584: 8 total, 8 up, 8 in 2024-09-20T22:51:05.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:05 smithi084 ceph-mon[40024]: 6.13 scrub starts 2024-09-20T22:51:05.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:05 smithi084 ceph-mon[40024]: 6.13 scrub ok 2024-09-20T22:51:05.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:05 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/35975475' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:51:05.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:05 smithi084 ceph-mon[40024]: 4.7 scrub starts 2024-09-20T22:51:05.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:05 smithi084 ceph-mon[40024]: 4.7 scrub ok 2024-09-20T22:51:05.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:05 smithi084 ceph-mon[40024]: 7.c scrub starts 2024-09-20T22:51:05.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:05 smithi084 ceph-mon[40024]: 7.c scrub ok 2024-09-20T22:51:05.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:05 smithi084 ceph-mon[29774]: pgmap v777: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 556 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 17 MiB/s rd, 2.5 MiB/s wr, 25 op/s 2024-09-20T22:51:05.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:05 smithi084 ceph-mon[29774]: 5.6 scrub starts 2024-09-20T22:51:05.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:05 smithi084 ceph-mon[29774]: 5.6 scrub ok 2024-09-20T22:51:05.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:05 smithi084 ceph-mon[29774]: osdmap e584: 8 total, 8 up, 8 in 2024-09-20T22:51:05.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:05 smithi084 ceph-mon[29774]: 6.13 scrub starts 2024-09-20T22:51:05.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:05 smithi084 ceph-mon[29774]: 6.13 scrub ok 2024-09-20T22:51:05.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:05 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/35975475' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:51:05.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:05 smithi084 ceph-mon[29774]: 4.7 scrub starts 2024-09-20T22:51:05.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:05 smithi084 ceph-mon[29774]: 4.7 scrub ok 2024-09-20T22:51:05.617 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:05 smithi084 ceph-mon[29774]: 7.c scrub starts 2024-09-20T22:51:05.617 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:05 smithi084 ceph-mon[29774]: 7.c scrub ok 2024-09-20T22:51:05.632 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:05.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:05.633 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:51:05.646 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-20T22:51:05.646 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-20T22:51:05.647 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph tell osd.0 'trim stale osdmaps' 2024-09-20T22:51:05.733 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:51:05.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:05 smithi154 ceph-mon[33264]: pgmap v777: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 556 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 17 MiB/s rd, 2.5 MiB/s wr, 25 op/s 2024-09-20T22:51:05.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:05 smithi154 ceph-mon[33264]: 5.6 scrub starts 2024-09-20T22:51:05.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:05 smithi154 ceph-mon[33264]: 5.6 scrub ok 2024-09-20T22:51:05.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:05 smithi154 ceph-mon[33264]: osdmap e584: 8 total, 8 up, 8 in 2024-09-20T22:51:05.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:05 smithi154 ceph-mon[33264]: 6.13 scrub starts 2024-09-20T22:51:05.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:05 smithi154 ceph-mon[33264]: 6.13 scrub ok 2024-09-20T22:51:05.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:05 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/35975475' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:51:05.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:05 smithi154 ceph-mon[33264]: 4.7 scrub starts 2024-09-20T22:51:05.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:05 smithi154 ceph-mon[33264]: 4.7 scrub ok 2024-09-20T22:51:05.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:05 smithi154 ceph-mon[33264]: 7.c scrub starts 2024-09-20T22:51:05.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:05 smithi154 ceph-mon[33264]: 7.c scrub ok 2024-09-20T22:51:05.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:05.756 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:05.756 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:51:05.850 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:51:05.857 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:51:05.886 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:05.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:05.887 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:51:05.988 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:51:06.009 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:06.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:06.010 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:51:06.111 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:51:06.132 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:06.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:06.133 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:51:06.233 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:51:06.255 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:06.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:06.256 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:51:06.334 INFO:tasks.rados.rados.0.smithi084.stdout:2306: done (4 left) 2024-09-20T22:51:06.334 INFO:tasks.rados.rados.0.smithi084.stdout:2308: done (3 left) 2024-09-20T22:51:06.334 INFO:tasks.rados.rados.0.smithi084.stdout:2310: done (2 left) 2024-09-20T22:51:06.334 INFO:tasks.rados.rados.0.smithi084.stdout:2311: done (1 left) 2024-09-20T22:51:06.334 INFO:tasks.rados.rados.0.smithi084.stdout:2313: done (0 left) 2024-09-20T22:51:06.334 INFO:tasks.rados.rados.0.smithi084.stdout:2314: read oid 46 snap -1 2024-09-20T22:51:06.334 INFO:tasks.rados.rados.0.smithi084.stdout:2314: expect deleted 2024-09-20T22:51:06.334 INFO:tasks.rados.rados.0.smithi084.stdout:2315: write (excl) oid 13 current snap is 255 2024-09-20T22:51:06.335 INFO:tasks.rados.rados.0.smithi084.stdout:2315: seq_num 799 ranges {628793=741016,2138995=567940,3192030=236763} 2024-09-20T22:51:06.340 INFO:tasks.rados.rados.0.smithi084.stdout:2315: writing smithi08477203-13 from 628793 to 1369809 tid 1 2024-09-20T22:51:06.348 INFO:tasks.rados.rados.0.smithi084.stdout:2315: writing smithi08477203-13 from 2138995 to 2706935 tid 2 2024-09-20T22:51:06.352 INFO:tasks.rados.rados.0.smithi084.stdout:2315: writing smithi08477203-13 from 3192030 to 3428793 tid 3 2024-09-20T22:51:06.352 INFO:tasks.rados.rados.0.smithi084.stdout:2316: write oid 23 current snap is 255 2024-09-20T22:51:06.353 INFO:tasks.rados.rados.0.smithi084.stdout:2316: seq_num 800 ranges {645600=416265,1797477=711598,3219453=226147} 2024-09-20T22:51:06.356 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:51:06.358 INFO:tasks.rados.rados.0.smithi084.stdout:2316: writing smithi08477203-23 from 645600 to 1061865 tid 1 2024-09-20T22:51:06.366 INFO:tasks.rados.rados.0.smithi084.stdout:2316: writing smithi08477203-23 from 1797477 to 2509075 tid 2 2024-09-20T22:51:06.372 INFO:tasks.rados.rados.0.smithi084.stdout:2316: writing smithi08477203-23 from 3219453 to 3445600 tid 3 2024-09-20T22:51:06.372 INFO:tasks.rados.rados.0.smithi084.stdout:2317: snap_create 2024-09-20T22:51:06.372 INFO:tasks.rados.rados.0.smithi084.stdout:2315: finishing write tid 1 to smithi08477203-13 2024-09-20T22:51:06.372 INFO:tasks.rados.rados.0.smithi084.stdout:2315: finishing write tid 2 to smithi08477203-13 2024-09-20T22:51:06.372 INFO:tasks.rados.rados.0.smithi084.stdout:2315: finishing write tid 3 to smithi08477203-13 2024-09-20T22:51:06.372 INFO:tasks.rados.rados.0.smithi084.stdout:2315: finishing write tid 4 to smithi08477203-13 2024-09-20T22:51:06.372 INFO:tasks.rados.rados.0.smithi084.stdout:2315: finishing write tid 5 to smithi08477203-13 2024-09-20T22:51:06.372 INFO:tasks.rados.rados.0.smithi084.stdout:2315: oid 13 updating version 0 to 1663 2024-09-20T22:51:06.372 INFO:tasks.rados.rados.0.smithi084.stdout:2315: oid 13 updating version 1663 to 1665 2024-09-20T22:51:06.372 INFO:tasks.rados.rados.0.smithi084.stdout:2315: oid 13 version 1665 is already newer than 1664 2024-09-20T22:51:06.373 INFO:tasks.rados.rados.0.smithi084.stdout:2315: oid 13 updating version 1665 to 1666 2024-09-20T22:51:06.373 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 13 v 1666 (ObjNum 799 snap 255 seq_num 799) dirty exists 2024-09-20T22:51:06.373 INFO:tasks.rados.rados.0.smithi084.stdout:2315: left oid 13 (ObjNum 799 snap 255 seq_num 799) 2024-09-20T22:51:06.373 INFO:tasks.rados.rados.0.smithi084.stdout:2316: finishing write tid 1 to smithi08477203-23 2024-09-20T22:51:06.373 INFO:tasks.rados.rados.0.smithi084.stdout:2316: finishing write tid 2 to smithi08477203-23 2024-09-20T22:51:06.373 INFO:tasks.rados.rados.0.smithi084.stdout:2316: finishing write tid 3 to smithi08477203-23 2024-09-20T22:51:06.374 INFO:tasks.rados.rados.0.smithi084.stdout:2316: finishing write tid 4 to smithi08477203-23 2024-09-20T22:51:06.374 INFO:tasks.rados.rados.0.smithi084.stdout:2316: finishing write tid 5 to smithi08477203-23 2024-09-20T22:51:06.374 INFO:tasks.rados.rados.0.smithi084.stdout:2316: oid 23 updating version 0 to 755 2024-09-20T22:51:06.374 INFO:tasks.rados.rados.0.smithi084.stdout:2316: oid 23 updating version 755 to 757 2024-09-20T22:51:06.374 INFO:tasks.rados.rados.0.smithi084.stdout:2316: oid 23 version 757 is already newer than 756 2024-09-20T22:51:06.374 INFO:tasks.rados.rados.0.smithi084.stdout:2316: oid 23 version 757 is already newer than 754 2024-09-20T22:51:06.374 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 23 v 757 (ObjNum 800 snap 255 seq_num 800) dirty exists 2024-09-20T22:51:06.375 INFO:tasks.rados.rados.0.smithi084.stdout:2316: left oid 23 (ObjNum 800 snap 255 seq_num 800) 2024-09-20T22:51:06.381 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:06.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:06.382 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:51:06.482 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:51:06.506 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:06.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:06.507 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:51:06.608 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:51:06.613 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:51:06.614 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:06 smithi084 ceph-mon[29774]: 6.19 scrub starts 2024-09-20T22:51:06.614 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:06 smithi084 ceph-mon[29774]: 6.19 scrub ok 2024-09-20T22:51:06.614 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:06 smithi084 ceph-mon[29774]: 5.8 scrub starts 2024-09-20T22:51:06.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:06 smithi084 ceph-mon[29774]: 5.8 scrub ok 2024-09-20T22:51:06.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:06 smithi084 ceph-mon[29774]: osdmap e585: 8 total, 8 up, 8 in 2024-09-20T22:51:06.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:06 smithi084 ceph-mon[29774]: 6.4 scrub starts 2024-09-20T22:51:06.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:06 smithi084 ceph-mon[29774]: 6.4 scrub ok 2024-09-20T22:51:06.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:06 smithi084 ceph-mon[29774]: 4.1b scrub starts 2024-09-20T22:51:06.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:06 smithi084 ceph-mon[29774]: 4.1b scrub ok 2024-09-20T22:51:06.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:06 smithi084 ceph-mon[29774]: 1.6 scrub starts 2024-09-20T22:51:06.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:06 smithi084 ceph-mon[29774]: 1.6 scrub ok 2024-09-20T22:51:06.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:06 smithi084 ceph-mon[40024]: 6.19 scrub starts 2024-09-20T22:51:06.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:06 smithi084 ceph-mon[40024]: 6.19 scrub ok 2024-09-20T22:51:06.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:06 smithi084 ceph-mon[40024]: 5.8 scrub starts 2024-09-20T22:51:06.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:06 smithi084 ceph-mon[40024]: 5.8 scrub ok 2024-09-20T22:51:06.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:06 smithi084 ceph-mon[40024]: osdmap e585: 8 total, 8 up, 8 in 2024-09-20T22:51:06.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:06 smithi084 ceph-mon[40024]: 6.4 scrub starts 2024-09-20T22:51:06.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:06 smithi084 ceph-mon[40024]: 6.4 scrub ok 2024-09-20T22:51:06.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:06 smithi084 ceph-mon[40024]: 4.1b scrub starts 2024-09-20T22:51:06.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:06 smithi084 ceph-mon[40024]: 4.1b scrub ok 2024-09-20T22:51:06.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:06 smithi084 ceph-mon[40024]: 1.6 scrub starts 2024-09-20T22:51:06.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:06 smithi084 ceph-mon[40024]: 1.6 scrub ok 2024-09-20T22:51:06.631 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:06.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:06.632 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:51:06.733 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:51:06.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:06 smithi154 ceph-mon[33264]: 6.19 scrub starts 2024-09-20T22:51:06.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:06 smithi154 ceph-mon[33264]: 6.19 scrub ok 2024-09-20T22:51:06.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:06 smithi154 ceph-mon[33264]: 5.8 scrub starts 2024-09-20T22:51:06.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:06 smithi154 ceph-mon[33264]: 5.8 scrub ok 2024-09-20T22:51:06.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:06 smithi154 ceph-mon[33264]: osdmap e585: 8 total, 8 up, 8 in 2024-09-20T22:51:06.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:06 smithi154 ceph-mon[33264]: 6.4 scrub starts 2024-09-20T22:51:06.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:06 smithi154 ceph-mon[33264]: 6.4 scrub ok 2024-09-20T22:51:06.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:06 smithi154 ceph-mon[33264]: 4.1b scrub starts 2024-09-20T22:51:06.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:06 smithi154 ceph-mon[33264]: 4.1b scrub ok 2024-09-20T22:51:06.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:06 smithi154 ceph-mon[33264]: 1.6 scrub starts 2024-09-20T22:51:06.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:06 smithi154 ceph-mon[33264]: 1.6 scrub ok 2024-09-20T22:51:06.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:06.756 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:06.756 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:51:06.857 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:51:06.881 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:06.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:06.882 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:51:06.982 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:51:07.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:07.007 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:07.007 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:51:07.098 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-20T22:51:07.098 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2024-09-20T22:51:07.098 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2024-09-20T22:51:07.098 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [ 2024-09-20T22:51:07.098 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.099 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:554 7.a 7:5122292a:::smithi08477203-33:head [copy-from ver 6] snapc 13=[13,12,11,10,f,d] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-09-20T22:51:07.099 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:43:27.336824+0000", 2024-09-20T22:51:07.099 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 459.75828007000001, 2024-09-20T22:51:07.099 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.056403397000000001, 2024-09-20T22:51:07.099 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:51:07.099 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:51:07.099 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:51:07.099 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:51:07.100 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:51:07.100 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:51:07.100 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 554 2024-09-20T22:51:07.100 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.100 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:51:07.100 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.100 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:51:07.100 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:27.336824+0000", 2024-09-20T22:51:07.100 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:51:07.100 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.100 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.100 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:51:07.100 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:27.336824+0000", 2024-09-20T22:51:07.101 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:51:07.101 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.101 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.101 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:51:07.101 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:27.336826+0000", 2024-09-20T22:51:07.101 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.0339999999999999e-06 2024-09-20T22:51:07.101 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.101 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.101 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:51:07.101 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:27.336837+0000", 2024-09-20T22:51:07.101 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1243000000000001e-05 2024-09-20T22:51:07.101 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.101 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.101 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:51:07.101 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:27.336857+0000", 2024-09-20T22:51:07.101 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.9868999999999999e-05 2024-09-20T22:51:07.102 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.102 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.102 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:51:07.102 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:27.336869+0000", 2024-09-20T22:51:07.102 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1348999999999999e-05 2024-09-20T22:51:07.102 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.102 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.102 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:51:07.102 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:27.336946+0000", 2024-09-20T22:51:07.102 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.7683999999999997e-05 2024-09-20T22:51:07.102 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.102 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.102 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:51:07.103 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:27.337045+0000", 2024-09-20T22:51:07.103 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.9265999999999995e-05 2024-09-20T22:51:07.103 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.103 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.103 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 2,5", 2024-09-20T22:51:07.103 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:27.340603+0000", 2024-09-20T22:51:07.103 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.003557892 2024-09-20T22:51:07.103 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.103 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.103 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:51:07.103 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:27.343571+0000", 2024-09-20T22:51:07.103 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0029676609999999999 2024-09-20T22:51:07.103 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.103 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.104 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:51:07.104 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:27.346457+0000", 2024-09-20T22:51:07.104 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0028860040000000002 2024-09-20T22:51:07.104 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.104 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.104 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:51:07.104 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:27.393131+0000", 2024-09-20T22:51:07.104 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.046673813000000001 2024-09-20T22:51:07.104 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.104 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.104 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:51:07.104 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:27.393173+0000", 2024-09-20T22:51:07.104 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.2076999999999997e-05 2024-09-20T22:51:07.104 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.105 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.105 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:51:07.105 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:27.393227+0000", 2024-09-20T22:51:07.105 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.4505000000000001e-05 2024-09-20T22:51:07.105 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:51:07.105 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:51:07.105 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:51:07.106 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.106 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.106 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:663 7.12 7:4fa4b2b3:::smithi08477203-21:head [write 665577~781581,stat] snapc 17=[17,16,15,11,d] ondisk+write+known_if_redirected+supports_pool_eio e137)", 2024-09-20T22:51:07.106 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:43:36.161568+0000", 2024-09-20T22:51:07.106 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 450.933535893, 2024-09-20T22:51:07.106 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.99286350899999998, 2024-09-20T22:51:07.106 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:51:07.106 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:51:07.106 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:51:07.107 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:51:07.107 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:51:07.107 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:51:07.107 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 663 2024-09-20T22:51:07.107 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.107 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:51:07.107 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.107 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:51:07.108 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.161568+0000", 2024-09-20T22:51:07.108 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:51:07.108 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.108 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.108 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:51:07.108 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.161568+0000", 2024-09-20T22:51:07.109 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:51:07.109 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.109 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.109 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:51:07.109 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.161570+0000", 2024-09-20T22:51:07.109 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.0679999999999999e-06 2024-09-20T22:51:07.110 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.110 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.110 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:51:07.110 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.162310+0000", 2024-09-20T22:51:07.110 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00074005999999999996 2024-09-20T22:51:07.110 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.110 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.110 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:51:07.110 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.162349+0000", 2024-09-20T22:51:07.110 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.8778e-05 2024-09-20T22:51:07.110 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.111 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.111 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:51:07.111 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.162375+0000", 2024-09-20T22:51:07.111 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.6089e-05 2024-09-20T22:51:07.111 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.111 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.111 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:51:07.111 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.162434+0000", 2024-09-20T22:51:07.112 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.8764999999999997e-05 2024-09-20T22:51:07.112 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.112 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.112 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-09-20T22:51:07.112 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.162443+0000", 2024-09-20T22:51:07.112 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.054e-06 2024-09-20T22:51:07.112 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.112 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.112 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:51:07.112 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.143416+0000", 2024-09-20T22:51:07.112 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.98097291600000003 2024-09-20T22:51:07.112 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.112 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.112 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:51:07.113 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.143522+0000", 2024-09-20T22:51:07.113 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00010626 2024-09-20T22:51:07.113 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.113 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.113 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 1,2", 2024-09-20T22:51:07.113 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.143654+0000", 2024-09-20T22:51:07.113 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013218799999999999 2024-09-20T22:51:07.113 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.113 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.113 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:51:07.113 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.148086+0000", 2024-09-20T22:51:07.113 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0044314189999999998 2024-09-20T22:51:07.113 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.113 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.113 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:51:07.114 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.153546+0000", 2024-09-20T22:51:07.114 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0054599920000000003 2024-09-20T22:51:07.114 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.114 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.114 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:51:07.114 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.154317+0000", 2024-09-20T22:51:07.114 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00077125999999999996 2024-09-20T22:51:07.114 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.114 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.114 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:51:07.114 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.154380+0000", 2024-09-20T22:51:07.114 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.2526000000000004e-05 2024-09-20T22:51:07.114 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.114 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.115 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:51:07.115 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.154432+0000", 2024-09-20T22:51:07.115 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.2133999999999999e-05 2024-09-20T22:51:07.115 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:51:07.115 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:51:07.115 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:51:07.115 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.115 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.115 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:664 7.12 7:4fa4b2b3:::smithi08477203-21:head [write 2062560~563688] snapc 17=[17,16,15,11,d] ondisk+write+known_if_redirected+supports_pool_eio e137)", 2024-09-20T22:51:07.115 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:51:07.116 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:43:36.167947+0000", 2024-09-20T22:51:07.116 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 450.92715742299998, 2024-09-20T22:51:07.117 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.98665172599999995, 2024-09-20T22:51:07.117 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:51:07.117 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:51:07.117 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:51:07.117 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:51:07.117 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:51:07.117 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:51:07.117 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 664 2024-09-20T22:51:07.117 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.117 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:51:07.117 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.117 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:51:07.118 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.167947+0000", 2024-09-20T22:51:07.118 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:51:07.118 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.118 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.118 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:51:07.118 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.167947+0000", 2024-09-20T22:51:07.118 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:51:07.118 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.118 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.118 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:51:07.118 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.167948+0000", 2024-09-20T22:51:07.118 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.187e-06 2024-09-20T22:51:07.118 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.118 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.119 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:51:07.119 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.168193+0000", 2024-09-20T22:51:07.119 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00024496199999999999 2024-09-20T22:51:07.119 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.119 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.119 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:51:07.119 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.168203+0000", 2024-09-20T22:51:07.119 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0543999999999999e-05 2024-09-20T22:51:07.119 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.119 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.119 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:51:07.119 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.168209+0000", 2024-09-20T22:51:07.119 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.2839999999999997e-06 2024-09-20T22:51:07.119 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.119 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.120 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:51:07.120 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.168252+0000", 2024-09-20T22:51:07.120 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.3115000000000003e-05 2024-09-20T22:51:07.120 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.120 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.120 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-09-20T22:51:07.120 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.168257+0000", 2024-09-20T22:51:07.120 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.6860000000000003e-06 2024-09-20T22:51:07.120 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.120 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.120 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:51:07.120 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.144795+0000", 2024-09-20T22:51:07.120 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.97653758099999999 2024-09-20T22:51:07.120 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.120 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.121 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:51:07.121 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.144865+0000", 2024-09-20T22:51:07.121 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.9641000000000006e-05 2024-09-20T22:51:07.121 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.121 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.121 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 1,2", 2024-09-20T22:51:07.121 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.144986+0000", 2024-09-20T22:51:07.121 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000121248 2024-09-20T22:51:07.121 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.121 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.121 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:51:07.121 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.148091+0000", 2024-09-20T22:51:07.121 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0031050399999999999 2024-09-20T22:51:07.121 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.122 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.122 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:51:07.122 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.153603+0000", 2024-09-20T22:51:07.122 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0055118479999999997 2024-09-20T22:51:07.122 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.122 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.122 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:51:07.122 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.154508+0000", 2024-09-20T22:51:07.122 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00090530600000000004 2024-09-20T22:51:07.122 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.122 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.122 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:51:07.122 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.154560+0000", 2024-09-20T22:51:07.122 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.1474000000000002e-05 2024-09-20T22:51:07.123 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.123 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.123 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:51:07.123 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.154598+0000", 2024-09-20T22:51:07.123 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.8810000000000003e-05 2024-09-20T22:51:07.123 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:51:07.123 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:51:07.123 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:51:07.123 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.123 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.123 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:665 7.12 7:4fa4b2b3:::smithi08477203-21:head [write 3182305~643664] snapc 17=[17,16,15,11,d] ondisk+write+known_if_redirected+supports_pool_eio e137)", 2024-09-20T22:51:07.123 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:43:36.174520+0000", 2024-09-20T22:51:07.123 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 450.92058433300002, 2024-09-20T22:51:07.123 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.98020083400000002, 2024-09-20T22:51:07.123 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:51:07.124 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:51:07.124 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:51:07.124 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:51:07.124 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:51:07.124 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:51:07.124 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 665 2024-09-20T22:51:07.124 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.124 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:51:07.124 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.124 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:51:07.124 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174520+0000", 2024-09-20T22:51:07.124 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:51:07.125 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.125 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.125 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:51:07.125 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174520+0000", 2024-09-20T22:51:07.125 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:51:07.125 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.125 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.125 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:51:07.125 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174521+0000", 2024-09-20T22:51:07.125 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.234e-06 2024-09-20T22:51:07.125 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.125 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.125 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:51:07.125 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174816+0000", 2024-09-20T22:51:07.125 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000294682 2024-09-20T22:51:07.126 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.126 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.126 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:51:07.126 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174827+0000", 2024-09-20T22:51:07.126 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1025e-05 2024-09-20T22:51:07.126 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.126 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.126 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:51:07.126 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174832+0000", 2024-09-20T22:51:07.126 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.3990000000000003e-06 2024-09-20T22:51:07.126 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.126 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.126 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:51:07.126 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174868+0000", 2024-09-20T22:51:07.126 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.6225e-05 2024-09-20T22:51:07.127 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.127 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.127 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-09-20T22:51:07.127 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174873+0000", 2024-09-20T22:51:07.127 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.848e-06 2024-09-20T22:51:07.127 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.127 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.128 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:51:07.128 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.145844+0000", 2024-09-20T22:51:07.128 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.97097048900000005 2024-09-20T22:51:07.128 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.128 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.128 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:51:07.128 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.145917+0000", 2024-09-20T22:51:07.129 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.3248000000000001e-05 2024-09-20T22:51:07.129 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.129 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.129 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 1,2", 2024-09-20T22:51:07.129 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.146028+0000", 2024-09-20T22:51:07.129 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000110773 2024-09-20T22:51:07.130 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.130 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.130 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:51:07.130 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.148095+0000", 2024-09-20T22:51:07.130 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0020678179999999999 2024-09-20T22:51:07.131 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.131 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.131 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:51:07.131 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.153967+0000", 2024-09-20T22:51:07.131 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0058717350000000003 2024-09-20T22:51:07.131 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.132 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.132 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:51:07.132 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.154633+0000", 2024-09-20T22:51:07.132 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00066600600000000002 2024-09-20T22:51:07.132 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.133 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.133 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:51:07.133 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.154683+0000", 2024-09-20T22:51:07.133 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.9872e-05 2024-09-20T22:51:07.133 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.133 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.134 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:51:07.134 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.154721+0000", 2024-09-20T22:51:07.134 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.748e-05 2024-09-20T22:51:07.134 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:51:07.134 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:51:07.134 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:51:07.135 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.135 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.135 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:666 7.12 7:4fa4b2b3:::smithi08477203-21:head [setxattr _header (57) in=7b,truncate 3865577] snapc 17=[17,16,15,11,d] ondisk+write+known_if_redirected+supports_pool_eio e137)", 2024-09-20T22:51:07.135 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:43:36.174844+0000", 2024-09-20T22:51:07.135 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 450.92026013399999, 2024-09-20T22:51:07.136 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.98002428600000002, 2024-09-20T22:51:07.136 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:51:07.136 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:51:07.136 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:51:07.136 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:51:07.136 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:51:07.137 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:51:07.137 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 666 2024-09-20T22:51:07.137 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.137 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:51:07.137 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.137 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:51:07.138 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174844+0000", 2024-09-20T22:51:07.138 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:51:07.138 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.138 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.138 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:51:07.138 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174844+0000", 2024-09-20T22:51:07.138 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:51:07.139 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.139 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.139 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:51:07.139 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174845+0000", 2024-09-20T22:51:07.139 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.1999999999999999e-07 2024-09-20T22:51:07.140 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.140 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.140 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:51:07.140 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174853+0000", 2024-09-20T22:51:07.140 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.1950000000000003e-06 2024-09-20T22:51:07.140 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.140 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.141 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:51:07.141 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174858+0000", 2024-09-20T22:51:07.141 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.5650000000000001e-06 2024-09-20T22:51:07.141 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.141 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.142 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:51:07.142 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174863+0000", 2024-09-20T22:51:07.142 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.1649999999999997e-06 2024-09-20T22:51:07.142 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.142 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.142 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:51:07.143 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174915+0000", 2024-09-20T22:51:07.143 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.1045999999999999e-05 2024-09-20T22:51:07.143 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.143 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.143 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-09-20T22:51:07.143 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174921+0000", 2024-09-20T22:51:07.144 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.404e-06 2024-09-20T22:51:07.144 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.144 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.144 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:51:07.144 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.146528+0000", 2024-09-20T22:51:07.144 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.97160681199999999 2024-09-20T22:51:07.145 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.145 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.145 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:51:07.145 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.146881+0000", 2024-09-20T22:51:07.145 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00035348200000000001 2024-09-20T22:51:07.145 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.146 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.146 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 1,2", 2024-09-20T22:51:07.146 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.147011+0000", 2024-09-20T22:51:07.146 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013018900000000001 2024-09-20T22:51:07.146 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.146 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.147 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:51:07.147 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.148100+0000", 2024-09-20T22:51:07.147 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010881580000000001 2024-09-20T22:51:07.147 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.147 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.147 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:51:07.148 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.154278+0000", 2024-09-20T22:51:07.148 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0061781730000000003 2024-09-20T22:51:07.148 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.148 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.148 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:51:07.148 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.154764+0000", 2024-09-20T22:51:07.148 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00048589199999999998 2024-09-20T22:51:07.149 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.149 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.149 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:51:07.149 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.154815+0000", 2024-09-20T22:51:07.149 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.1428000000000003e-05 2024-09-20T22:51:07.150 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.150 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.150 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:51:07.150 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.154868+0000", 2024-09-20T22:51:07.150 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.3156999999999999e-05 2024-09-20T22:51:07.151 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:51:07.151 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:51:07.151 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:51:07.151 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.151 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.151 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:667 7.12 7:4fa4b2b3:::smithi08477203-21:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e137)", 2024-09-20T22:51:07.152 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:43:36.174872+0000", 2024-09-20T22:51:07.152 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 450.920232441, 2024-09-20T22:51:07.152 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.98024790299999998, 2024-09-20T22:51:07.152 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:51:07.152 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:51:07.152 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-20T22:51:07.153 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:51:07.153 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:51:07.153 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:51:07.153 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 667 2024-09-20T22:51:07.153 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.153 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:51:07.154 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.154 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:51:07.154 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174872+0000", 2024-09-20T22:51:07.154 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:51:07.154 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.154 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.155 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:51:07.155 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174872+0000", 2024-09-20T22:51:07.155 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:51:07.155 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.155 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.155 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:51:07.156 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174872+0000", 2024-09-20T22:51:07.156 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.2199999999999999e-07 2024-09-20T22:51:07.156 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.156 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.156 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:51:07.156 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174874+0000", 2024-09-20T22:51:07.157 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.187e-06 2024-09-20T22:51:07.157 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.157 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.157 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:51:07.157 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174881+0000", 2024-09-20T22:51:07.157 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.674e-06 2024-09-20T22:51:07.158 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.158 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.158 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:51:07.158 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174886+0000", 2024-09-20T22:51:07.158 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.2240000000000003e-06 2024-09-20T22:51:07.158 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.159 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.159 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:51:07.159 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174936+0000", 2024-09-20T22:51:07.159 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.9824999999999998e-05 2024-09-20T22:51:07.159 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.160 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.160 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-09-20T22:51:07.160 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174940+0000", 2024-09-20T22:51:07.160 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.5700000000000001e-06 2024-09-20T22:51:07.160 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.160 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.161 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:51:07.161 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.147523+0000", 2024-09-20T22:51:07.161 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.97258315399999995 2024-09-20T22:51:07.161 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.161 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.162 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for rw locks", 2024-09-20T22:51:07.162 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.147591+0000", 2024-09-20T22:51:07.162 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.7878000000000003e-05 2024-09-20T22:51:07.162 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.162 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.162 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:51:07.163 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.154925+0000", 2024-09-20T22:51:07.163 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0073344430000000004 2024-09-20T22:51:07.163 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.163 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.163 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:51:07.163 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.155020+0000", 2024-09-20T22:51:07.164 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.4767999999999998e-05 2024-09-20T22:51:07.164 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.164 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.164 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:51:07.164 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.155120+0000", 2024-09-20T22:51:07.164 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.9758000000000003e-05 2024-09-20T22:51:07.164 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:51:07.165 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:51:07.165 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:51:07.165 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.165 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.165 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.14478.0:1259 5.3 5:c609908c:::notify.5:head [watch reconnect cookie 94843898679808 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e169)", 2024-09-20T22:51:07.165 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:44:08.230723+0000", 2024-09-20T22:51:07.166 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 418.86438113600002, 2024-09-20T22:51:07.166 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.94983577399999997, 2024-09-20T22:51:07.166 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:51:07.166 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:51:07.166 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-20T22:51:07.166 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:51:07.167 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.14478", 2024-09-20T22:51:07.167 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:51:07.169 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/2994273271", 2024-09-20T22:51:07.169 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1259 2024-09-20T22:51:07.170 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.170 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:51:07.170 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.170 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:51:07.170 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:08.230723+0000", 2024-09-20T22:51:07.170 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:51:07.171 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.171 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.171 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:51:07.171 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:08.230723+0000", 2024-09-20T22:51:07.171 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:51:07.171 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.172 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.172 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:51:07.172 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:08.230724+0000", 2024-09-20T22:51:07.172 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.077e-06 2024-09-20T22:51:07.172 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.172 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.173 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:51:07.173 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:08.230729+0000", 2024-09-20T22:51:07.173 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.9930000000000003e-06 2024-09-20T22:51:07.173 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.173 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.173 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:51:07.173 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:08.230741+0000", 2024-09-20T22:51:07.174 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1897e-05 2024-09-20T22:51:07.174 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.174 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.174 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:51:07.174 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:08.230747+0000", 2024-09-20T22:51:07.174 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.9830000000000001e-06 2024-09-20T22:51:07.174 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.175 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.175 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:51:07.175 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:08.230802+0000", 2024-09-20T22:51:07.175 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.4657000000000002e-05 2024-09-20T22:51:07.175 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.175 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.176 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-09-20T22:51:07.176 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:08.230811+0000", 2024-09-20T22:51:07.176 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.1160000000000005e-06 2024-09-20T22:51:07.176 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.176 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.176 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:51:07.177 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:09.171968+0000", 2024-09-20T22:51:07.177 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.94115747299999997 2024-09-20T22:51:07.177 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.177 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.177 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for missing object", 2024-09-20T22:51:07.177 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:09.172098+0000", 2024-09-20T22:51:07.177 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00012937499999999999 2024-09-20T22:51:07.178 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.178 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.178 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:51:07.178 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:09.174248+0000", 2024-09-20T22:51:07.178 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0021508959999999998 2024-09-20T22:51:07.178 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.179 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.179 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for rw locks", 2024-09-20T22:51:07.179 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:09.174341+0000", 2024-09-20T22:51:07.179 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.2906999999999997e-05 2024-09-20T22:51:07.179 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.179 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.180 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:51:07.180 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:09.175083+0000", 2024-09-20T22:51:07.180 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00074120100000000003 2024-09-20T22:51:07.180 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.180 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.180 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:51:07.181 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:09.175149+0000", 2024-09-20T22:51:07.181 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.6877000000000004e-05 2024-09-20T22:51:07.181 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.181 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.181 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:51:07.181 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:09.180559+0000", 2024-09-20T22:51:07.182 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0054093220000000003 2024-09-20T22:51:07.182 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:51:07.182 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:51:07.182 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:51:07.182 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.182 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.182 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:1419 7.1 7:84a866f5:::smithi08477203-27:head [append 0~0] snapc 2f=[2f] ondisk+write+known_if_redirected+supports_pool_eio e189)", 2024-09-20T22:51:07.183 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:44:28.138055+0000", 2024-09-20T22:51:07.183 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 398.95704949700001, 2024-09-20T22:51:07.183 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.030290338999999999, 2024-09-20T22:51:07.183 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:51:07.183 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:51:07.183 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:51:07.183 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:51:07.183 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:51:07.183 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:51:07.183 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1419 2024-09-20T22:51:07.183 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.183 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:51:07.183 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.184 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:51:07.184 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.138055+0000", 2024-09-20T22:51:07.184 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:51:07.184 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.184 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.184 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:51:07.184 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.138055+0000", 2024-09-20T22:51:07.184 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:51:07.184 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.184 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.184 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:51:07.184 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.138056+0000", 2024-09-20T22:51:07.184 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.13e-06 2024-09-20T22:51:07.185 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.185 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.185 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:51:07.185 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.138060+0000", 2024-09-20T22:51:07.185 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.0269999999999997e-06 2024-09-20T22:51:07.185 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.185 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.185 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:51:07.185 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.138070+0000", 2024-09-20T22:51:07.185 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.961e-06 2024-09-20T22:51:07.185 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.185 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.185 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:51:07.185 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.138076+0000", 2024-09-20T22:51:07.186 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.1800000000000001e-06 2024-09-20T22:51:07.186 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.186 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.186 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:51:07.186 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.138137+0000", 2024-09-20T22:51:07.186 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.1159000000000002e-05 2024-09-20T22:51:07.186 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.186 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.186 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting_for_map not empty", 2024-09-20T22:51:07.186 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.138141+0000", 2024-09-20T22:51:07.186 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.189e-06 2024-09-20T22:51:07.186 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.186 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.187 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:51:07.187 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.154982+0000", 2024-09-20T22:51:07.187 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.016840738000000001 2024-09-20T22:51:07.187 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.187 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.187 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:51:07.187 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.155089+0000", 2024-09-20T22:51:07.187 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00010653399999999999 2024-09-20T22:51:07.187 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.187 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.187 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 1,6", 2024-09-20T22:51:07.187 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.155320+0000", 2024-09-20T22:51:07.187 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00023196299999999999 2024-09-20T22:51:07.187 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.187 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.188 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:51:07.188 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.158224+0000", 2024-09-20T22:51:07.188 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00290361 2024-09-20T22:51:07.188 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.188 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.188 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:51:07.188 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.167686+0000", 2024-09-20T22:51:07.188 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0094622360000000006 2024-09-20T22:51:07.188 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.188 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.188 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:51:07.188 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.168183+0000", 2024-09-20T22:51:07.188 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00049655999999999995 2024-09-20T22:51:07.188 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.188 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.189 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:51:07.189 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.168273+0000", 2024-09-20T22:51:07.189 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.9758000000000004e-05 2024-09-20T22:51:07.189 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.189 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.189 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:51:07.189 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.168345+0000", 2024-09-20T22:51:07.189 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.2293999999999996e-05 2024-09-20T22:51:07.189 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:51:07.189 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:51:07.189 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:51:07.189 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.189 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.189 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:2347 7.a 7:5122292a:::smithi08477203-33:head [sparse-read 0~2590699,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e260)", 2024-09-20T22:51:07.189 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:45:39.196440+0000", 2024-09-20T22:51:07.190 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 327.89866436900002, 2024-09-20T22:51:07.190 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.026701163, 2024-09-20T22:51:07.190 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:51:07.190 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:51:07.190 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-20T22:51:07.190 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:51:07.190 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:51:07.190 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:51:07.190 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 2347 2024-09-20T22:51:07.190 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.190 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:51:07.190 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.190 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:51:07.190 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.196440+0000", 2024-09-20T22:51:07.191 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:51:07.191 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.191 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.191 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:51:07.191 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.196440+0000", 2024-09-20T22:51:07.191 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:51:07.191 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.191 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.191 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:51:07.191 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.196441+0000", 2024-09-20T22:51:07.191 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.714e-06 2024-09-20T22:51:07.191 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.192 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.192 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:51:07.192 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.196449+0000", 2024-09-20T22:51:07.192 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.8730000000000007e-06 2024-09-20T22:51:07.192 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.192 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.192 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:51:07.192 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.196462+0000", 2024-09-20T22:51:07.192 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2649e-05 2024-09-20T22:51:07.192 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.192 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.192 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:51:07.192 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.196470+0000", 2024-09-20T22:51:07.192 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.4989999999999994e-06 2024-09-20T22:51:07.193 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.193 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.193 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:51:07.193 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.212172+0000", 2024-09-20T22:51:07.193 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.015701335 2024-09-20T22:51:07.193 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.193 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.193 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:51:07.193 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.212270+0000", 2024-09-20T22:51:07.193 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.8257999999999993e-05 2024-09-20T22:51:07.193 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.193 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.193 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:51:07.193 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.223141+0000", 2024-09-20T22:51:07.194 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010870835000000001 2024-09-20T22:51:07.194 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:51:07.194 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:51:07.194 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:51:07.194 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.194 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.194 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:2348 7.a 7:5122292a:::smithi08477203-33:head [read 0~3848803,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e260)", 2024-09-20T22:51:07.194 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:45:39.196485+0000", 2024-09-20T22:51:07.194 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 327.89861889600002, 2024-09-20T22:51:07.194 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0284703, 2024-09-20T22:51:07.194 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:51:07.194 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:51:07.194 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-20T22:51:07.194 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:51:07.195 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:51:07.195 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:51:07.195 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 2348 2024-09-20T22:51:07.195 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.195 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:51:07.195 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.195 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:51:07.195 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.196485+0000", 2024-09-20T22:51:07.195 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:51:07.195 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.195 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.195 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:51:07.195 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.196485+0000", 2024-09-20T22:51:07.195 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:51:07.196 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.196 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.196 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:51:07.196 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.196486+0000", 2024-09-20T22:51:07.196 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.9599999999999999e-07 2024-09-20T22:51:07.196 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.196 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.196 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:51:07.196 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.196489+0000", 2024-09-20T22:51:07.196 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.5470000000000001e-06 2024-09-20T22:51:07.196 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.196 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.196 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:51:07.196 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.196496+0000", 2024-09-20T22:51:07.197 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.3570000000000003e-06 2024-09-20T22:51:07.197 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.197 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.197 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:51:07.197 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.196501+0000", 2024-09-20T22:51:07.197 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.6239999999999999e-06 2024-09-20T22:51:07.197 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.197 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.197 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:51:07.197 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.223176+0000", 2024-09-20T22:51:07.197 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.026675285999999999 2024-09-20T22:51:07.197 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.197 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.197 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:51:07.197 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.223274+0000", 2024-09-20T22:51:07.198 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.7009999999999994e-05 2024-09-20T22:51:07.198 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.198 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.198 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:51:07.198 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.224955+0000", 2024-09-20T22:51:07.198 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00168198 2024-09-20T22:51:07.198 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:51:07.198 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:51:07.198 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:51:07.198 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.198 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.198 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:2349 7.a 7:5122292a:::smithi08477203-33:head [sparse-read 0~2590699] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e260)", 2024-09-20T22:51:07.198 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:45:39.196508+0000", 2024-09-20T22:51:07.198 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 327.89859560999997, 2024-09-20T22:51:07.199 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.028731293000000001, 2024-09-20T22:51:07.199 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:51:07.199 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:51:07.199 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-20T22:51:07.199 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:51:07.199 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:51:07.199 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:51:07.199 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 2349 2024-09-20T22:51:07.199 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.199 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:51:07.199 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.199 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:51:07.199 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.196508+0000", 2024-09-20T22:51:07.199 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:51:07.199 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.200 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.200 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:51:07.200 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.196508+0000", 2024-09-20T22:51:07.200 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:51:07.200 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.200 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.200 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:51:07.200 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.196509+0000", 2024-09-20T22:51:07.200 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.08e-07 2024-09-20T22:51:07.200 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.200 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.200 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:51:07.200 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.196511+0000", 2024-09-20T22:51:07.200 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.142e-06 2024-09-20T22:51:07.200 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.201 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.201 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:51:07.201 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.196517+0000", 2024-09-20T22:51:07.201 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.6330000000000001e-06 2024-09-20T22:51:07.201 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.201 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.201 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:51:07.201 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.196522+0000", 2024-09-20T22:51:07.201 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.3530000000000002e-06 2024-09-20T22:51:07.201 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.201 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.201 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:51:07.201 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.224990+0000", 2024-09-20T22:51:07.201 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.028467864999999998 2024-09-20T22:51:07.201 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.201 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.202 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:51:07.202 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.225088+0000", 2024-09-20T22:51:07.202 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.8281000000000003e-05 2024-09-20T22:51:07.202 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.202 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.202 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:51:07.202 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.225240+0000", 2024-09-20T22:51:07.202 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00015161100000000001 2024-09-20T22:51:07.202 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:51:07.202 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:51:07.202 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:51:07.202 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.202 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.202 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(osd.6.0:727 7.c e260/94 7:318c7512:::smithi08477203-44:4c v 260'428, mlcod=258'424)", 2024-09-20T22:51:07.202 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:45:39.202413+0000", 2024-09-20T22:51:07.203 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 327.892691071, 2024-09-20T22:51:07.203 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.022852897, 2024-09-20T22:51:07.203 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:51:07.203 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:51:07.203 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:51:07.203 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:51:07.203 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.203 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:51:07.203 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.202413+0000", 2024-09-20T22:51:07.203 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:51:07.203 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.203 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.203 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:51:07.203 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.202413+0000", 2024-09-20T22:51:07.203 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:51:07.204 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.204 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.204 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:51:07.204 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.202414+0000", 2024-09-20T22:51:07.204 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.9299999999999997e-07 2024-09-20T22:51:07.204 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.204 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.204 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:51:07.204 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.202421+0000", 2024-09-20T22:51:07.204 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.8109999999999997e-06 2024-09-20T22:51:07.204 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.204 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.204 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:51:07.204 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.202433+0000", 2024-09-20T22:51:07.205 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2411e-05 2024-09-20T22:51:07.205 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.205 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.205 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:51:07.205 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.202442+0000", 2024-09-20T22:51:07.205 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.7269999999999996e-06 2024-09-20T22:51:07.205 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.205 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.205 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:51:07.205 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.221764+0000", 2024-09-20T22:51:07.205 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.019322068000000001 2024-09-20T22:51:07.205 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.205 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.206 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:51:07.206 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.221786+0000", 2024-09-20T22:51:07.206 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.2167e-05 2024-09-20T22:51:07.206 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.206 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.206 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:51:07.206 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.225217+0000", 2024-09-20T22:51:07.206 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0034313709999999999 2024-09-20T22:51:07.206 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.206 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.206 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:51:07.206 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.225266+0000", 2024-09-20T22:51:07.206 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.8649000000000001e-05 2024-09-20T22:51:07.206 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:51:07.207 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:51:07.207 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:51:07.207 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.207 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.207 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:2525 7.1 7:8422f82d:::smithi08477203-22:head [copy-from ver 174] snapc 5d=[5d,5c,5a,50,47] ondisk+write+known_if_redirected+supports_pool_eio e278)", 2024-09-20T22:51:07.207 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:45:57.170348+0000", 2024-09-20T22:51:07.207 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 309.924756097, 2024-09-20T22:51:07.207 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.023333486000000001, 2024-09-20T22:51:07.207 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:51:07.207 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:51:07.207 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:51:07.207 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:51:07.207 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:51:07.207 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:51:07.208 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 2525 2024-09-20T22:51:07.208 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.208 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:51:07.208 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.208 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:51:07.208 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:57.170348+0000", 2024-09-20T22:51:07.208 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:51:07.208 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.208 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.208 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:51:07.208 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:57.170348+0000", 2024-09-20T22:51:07.208 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:51:07.208 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.209 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.209 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:51:07.209 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:57.170350+0000", 2024-09-20T22:51:07.209 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.1509999999999998e-06 2024-09-20T22:51:07.209 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.209 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.209 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:51:07.209 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:57.170361+0000", 2024-09-20T22:51:07.209 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0492999999999999e-05 2024-09-20T22:51:07.209 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.209 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.209 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:51:07.209 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:57.170381+0000", 2024-09-20T22:51:07.209 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.0509999999999998e-05 2024-09-20T22:51:07.209 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.210 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.210 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:51:07.210 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:57.170393+0000", 2024-09-20T22:51:07.210 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2009e-05 2024-09-20T22:51:07.210 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.210 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.210 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:51:07.210 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:57.170465+0000", 2024-09-20T22:51:07.210 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.1927000000000004e-05 2024-09-20T22:51:07.210 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.210 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.210 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op must wait for map", 2024-09-20T22:51:07.210 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:57.170473+0000", 2024-09-20T22:51:07.210 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.1890000000000007e-06 2024-09-20T22:51:07.211 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.211 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.211 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:51:07.211 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:57.172855+0000", 2024-09-20T22:51:07.211 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002382168 2024-09-20T22:51:07.211 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.211 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.211 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:51:07.211 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:57.172967+0000", 2024-09-20T22:51:07.211 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00011108299999999999 2024-09-20T22:51:07.211 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.211 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.211 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 1,6", 2024-09-20T22:51:07.211 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:57.181533+0000", 2024-09-20T22:51:07.211 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0085667499999999997 2024-09-20T22:51:07.212 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.212 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.212 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:51:07.212 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:57.191599+0000", 2024-09-20T22:51:07.212 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010065795000000001 2024-09-20T22:51:07.212 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.212 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.212 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:51:07.212 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:57.193469+0000", 2024-09-20T22:51:07.212 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0018695199999999999 2024-09-20T22:51:07.212 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.212 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.212 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:51:07.212 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:57.193550+0000", 2024-09-20T22:51:07.213 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.1891000000000006e-05 2024-09-20T22:51:07.213 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.213 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.213 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:51:07.213 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:57.193625+0000", 2024-09-20T22:51:07.213 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.4679999999999996e-05 2024-09-20T22:51:07.213 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.213 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.213 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:51:07.213 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:57.193681+0000", 2024-09-20T22:51:07.213 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.6320000000000003e-05 2024-09-20T22:51:07.213 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:51:07.213 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:51:07.213 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:51:07.213 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.214 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.214 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:2853 7.10 7:0f524219:::smithi08477203-50:head [copy-from ver 114] snapc 68=[68,67] ondisk+write+known_if_redirected+supports_pool_eio e303)", 2024-09-20T22:51:07.214 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:46:23.285507+0000", 2024-09-20T22:51:07.214 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 283.80959736800003, 2024-09-20T22:51:07.214 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.023440631, 2024-09-20T22:51:07.214 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:51:07.214 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:51:07.214 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:51:07.214 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:51:07.214 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:51:07.214 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:51:07.214 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 2853 2024-09-20T22:51:07.214 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.214 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:51:07.215 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.215 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:51:07.215 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.285507+0000", 2024-09-20T22:51:07.215 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:51:07.215 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.215 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.215 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:51:07.215 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.285507+0000", 2024-09-20T22:51:07.215 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:51:07.215 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.215 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.215 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:51:07.215 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.285508+0000", 2024-09-20T22:51:07.215 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.491e-06 2024-09-20T22:51:07.215 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.216 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.216 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:51:07.216 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.285514+0000", 2024-09-20T22:51:07.216 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.502e-06 2024-09-20T22:51:07.216 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.216 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.216 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:51:07.216 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.285526+0000", 2024-09-20T22:51:07.216 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2639e-05 2024-09-20T22:51:07.216 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.216 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.216 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:51:07.216 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.285535+0000", 2024-09-20T22:51:07.216 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.4689999999999997e-06 2024-09-20T22:51:07.216 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.217 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.217 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:51:07.217 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.285591+0000", 2024-09-20T22:51:07.217 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.6407999999999999e-05 2024-09-20T22:51:07.217 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.217 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.217 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:51:07.217 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.285681+0000", 2024-09-20T22:51:07.217 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.9845000000000005e-05 2024-09-20T22:51:07.217 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.217 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.217 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 1,2", 2024-09-20T22:51:07.217 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.293719+0000", 2024-09-20T22:51:07.217 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0080383950000000003 2024-09-20T22:51:07.218 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.218 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.218 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:51:07.218 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.307398+0000", 2024-09-20T22:51:07.218 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013678074 2024-09-20T22:51:07.218 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.218 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.218 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:51:07.218 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.308451+0000", 2024-09-20T22:51:07.218 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001052975 2024-09-20T22:51:07.218 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.218 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.218 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:51:07.218 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.308811+0000", 2024-09-20T22:51:07.219 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00036041399999999998 2024-09-20T22:51:07.219 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.219 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.219 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:51:07.219 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.308878+0000", 2024-09-20T22:51:07.219 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.7114999999999997e-05 2024-09-20T22:51:07.219 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.219 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.219 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:51:07.219 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.308947+0000", 2024-09-20T22:51:07.219 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.9303999999999999e-05 2024-09-20T22:51:07.219 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:51:07.219 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:51:07.219 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:51:07.220 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.220 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.220 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:3136 7.a 7:5122292a:::smithi08477203-33:head [copy-from ver 385] snapc 77=[77,76,75,74,73,70,6b] ondisk+write+known_if_redirected+supports_pool_eio e328)", 2024-09-20T22:51:07.220 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:46:48.210350+0000", 2024-09-20T22:51:07.220 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 258.88475423699998, 2024-09-20T22:51:07.220 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.030150626999999999, 2024-09-20T22:51:07.220 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:51:07.220 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:51:07.220 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:51:07.220 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:51:07.220 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:51:07.220 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:51:07.220 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 3136 2024-09-20T22:51:07.220 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.221 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:51:07.221 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.221 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:51:07.221 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.210350+0000", 2024-09-20T22:51:07.221 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:51:07.221 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.221 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.221 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:51:07.221 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.210350+0000", 2024-09-20T22:51:07.221 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:51:07.221 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.221 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.221 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:51:07.221 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.210352+0000", 2024-09-20T22:51:07.221 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.889e-06 2024-09-20T22:51:07.222 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.222 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.222 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:51:07.222 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.210360+0000", 2024-09-20T22:51:07.222 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.8890000000000002e-06 2024-09-20T22:51:07.222 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.222 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.222 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:51:07.222 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.210372+0000", 2024-09-20T22:51:07.222 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2662e-05 2024-09-20T22:51:07.222 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.222 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.222 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:51:07.223 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.210381+0000", 2024-09-20T22:51:07.223 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.8589999999999994e-06 2024-09-20T22:51:07.223 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.223 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.223 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:51:07.223 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.210470+0000", 2024-09-20T22:51:07.223 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.9034000000000002e-05 2024-09-20T22:51:07.223 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.223 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.223 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:51:07.223 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.210568+0000", 2024-09-20T22:51:07.223 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.7919999999999998e-05 2024-09-20T22:51:07.223 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.223 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.223 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 2,5", 2024-09-20T22:51:07.224 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.218491+0000", 2024-09-20T22:51:07.224 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0079229490000000003 2024-09-20T22:51:07.224 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.224 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.224 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:51:07.224 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.224053+0000", 2024-09-20T22:51:07.224 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0055624330000000003 2024-09-20T22:51:07.224 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.224 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.224 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:51:07.224 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.227143+0000", 2024-09-20T22:51:07.224 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0030896360000000002 2024-09-20T22:51:07.224 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.224 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.225 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:51:07.225 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.240355+0000", 2024-09-20T22:51:07.225 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013211624999999999 2024-09-20T22:51:07.225 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.225 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.225 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:51:07.225 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.240439+0000", 2024-09-20T22:51:07.225 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.4009000000000002e-05 2024-09-20T22:51:07.225 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.225 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.225 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:51:07.225 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.240500+0000", 2024-09-20T22:51:07.225 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.1722000000000003e-05 2024-09-20T22:51:07.225 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:51:07.225 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:51:07.226 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:51:07.226 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.226 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.226 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:3235 7.1 7:84a866f5:::smithi08477203-27:head [copy-from ver 114] snapc 7e=[7e,7d,7c,7b,7a,79,78,75,74,73,6b] ondisk+write+known_if_redirected+supports_pool_eio e338)", 2024-09-20T22:51:07.226 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:46:58.180892+0000", 2024-09-20T22:51:07.226 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 248.91421201399999, 2024-09-20T22:51:07.226 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.044574093000000002, 2024-09-20T22:51:07.226 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:51:07.226 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:51:07.226 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:51:07.226 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:51:07.226 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:51:07.226 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:51:07.226 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 3235 2024-09-20T22:51:07.227 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.227 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:51:07.227 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.227 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:51:07.227 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.180892+0000", 2024-09-20T22:51:07.227 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:51:07.227 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.227 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.227 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:51:07.227 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.180892+0000", 2024-09-20T22:51:07.227 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:51:07.227 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.227 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.228 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:51:07.228 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.180894+0000", 2024-09-20T22:51:07.228 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.1789999999999998e-06 2024-09-20T22:51:07.228 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.228 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.228 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:51:07.228 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.180915+0000", 2024-09-20T22:51:07.228 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.0859000000000001e-05 2024-09-20T22:51:07.228 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.228 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.228 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:51:07.228 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.180937+0000", 2024-09-20T22:51:07.228 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.2357999999999998e-05 2024-09-20T22:51:07.228 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.229 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.229 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:51:07.229 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.180947+0000", 2024-09-20T22:51:07.229 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.8090000000000004e-06 2024-09-20T22:51:07.229 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.229 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.229 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:51:07.229 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.181042+0000", 2024-09-20T22:51:07.229 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.4327999999999995e-05 2024-09-20T22:51:07.229 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.229 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.229 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op must wait for map", 2024-09-20T22:51:07.229 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.181049+0000", 2024-09-20T22:51:07.229 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.1640000000000004e-06 2024-09-20T22:51:07.229 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.229 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.230 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:51:07.230 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.185933+0000", 2024-09-20T22:51:07.230 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.004884645 2024-09-20T22:51:07.230 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.230 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.230 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:51:07.230 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.186177+0000", 2024-09-20T22:51:07.230 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000243107 2024-09-20T22:51:07.230 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.230 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.230 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 1,6", 2024-09-20T22:51:07.230 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.204218+0000", 2024-09-20T22:51:07.230 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.018041805000000001 2024-09-20T22:51:07.230 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.230 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.231 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:51:07.231 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.218275+0000", 2024-09-20T22:51:07.231 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014056528 2024-09-20T22:51:07.231 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.231 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.231 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:51:07.231 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.224283+0000", 2024-09-20T22:51:07.231 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0060084860000000004 2024-09-20T22:51:07.231 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.231 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.231 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:51:07.231 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.225379+0000", 2024-09-20T22:51:07.231 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010960830000000001 2024-09-20T22:51:07.231 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.231 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.232 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:51:07.232 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.225429+0000", 2024-09-20T22:51:07.232 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.9440000000000001e-05 2024-09-20T22:51:07.232 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.232 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.232 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:51:07.232 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.225466+0000", 2024-09-20T22:51:07.232 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.7302000000000003e-05 2024-09-20T22:51:07.232 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:51:07.232 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:51:07.232 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:51:07.232 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.232 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.232 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:3236 7.1 7:84a866f5:::smithi08477203-27:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e338)", 2024-09-20T22:51:07.233 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:46:58.180973+0000", 2024-09-20T22:51:07.233 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 248.914131392, 2024-09-20T22:51:07.233 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.028358540000000002, 2024-09-20T22:51:07.233 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:51:07.233 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:51:07.233 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-20T22:51:07.233 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:51:07.233 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:51:07.233 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:51:07.233 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 3236 2024-09-20T22:51:07.233 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.233 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:51:07.233 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.234 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:51:07.234 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.180973+0000", 2024-09-20T22:51:07.234 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:51:07.234 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.234 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.234 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:51:07.234 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.180973+0000", 2024-09-20T22:51:07.234 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:51:07.234 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.234 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.234 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:51:07.234 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.180973+0000", 2024-09-20T22:51:07.234 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.9100000000000004e-07 2024-09-20T22:51:07.234 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.234 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.235 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:51:07.235 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.180976+0000", 2024-09-20T22:51:07.235 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.118e-06 2024-09-20T22:51:07.235 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.235 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.235 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:51:07.235 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.180984+0000", 2024-09-20T22:51:07.235 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.43e-06 2024-09-20T22:51:07.235 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.235 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.235 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:51:07.235 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.180990+0000", 2024-09-20T22:51:07.235 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.1109999999999996e-06 2024-09-20T22:51:07.235 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.236 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.236 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:51:07.236 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.181102+0000", 2024-09-20T22:51:07.236 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000112193 2024-09-20T22:51:07.236 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.236 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.236 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting_for_map not empty", 2024-09-20T22:51:07.236 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.181107+0000", 2024-09-20T22:51:07.236 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.0490000000000003e-06 2024-09-20T22:51:07.236 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.236 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.236 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:51:07.237 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.186302+0000", 2024-09-20T22:51:07.237 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0051950240000000003 2024-09-20T22:51:07.237 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.237 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.237 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for blocked object", 2024-09-20T22:51:07.237 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.186344+0000", 2024-09-20T22:51:07.237 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.2168e-05 2024-09-20T22:51:07.237 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.237 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.237 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:51:07.237 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.209115+0000", 2024-09-20T22:51:07.237 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.022770291000000002 2024-09-20T22:51:07.237 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.237 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.238 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:51:07.238 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.209217+0000", 2024-09-20T22:51:07.238 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000102192 2024-09-20T22:51:07.238 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.238 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.238 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:51:07.238 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.209331+0000", 2024-09-20T22:51:07.238 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000114373 2024-09-20T22:51:07.238 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:51:07.238 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:51:07.238 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:51:07.238 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.239 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.239 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:4534 7.12 7:4b06cbed:::smithi08477203-49:head [copy-from ver 722] snapc bb=[bb,ba,b9,b8,b7,b5,b4,b2,b1,a7,a6,a5,9e,9c,95,87,86] ondisk+write+known_if_redirected+supports_pool_eio e456)", 2024-09-20T22:51:07.239 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:48:56.230481+0000", 2024-09-20T22:51:07.239 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 130.864623346, 2024-09-20T22:51:07.239 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.028199295999999999, 2024-09-20T22:51:07.239 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:51:07.239 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:51:07.239 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:51:07.239 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:51:07.239 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:51:07.239 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:51:07.239 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 4534 2024-09-20T22:51:07.239 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.239 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:51:07.239 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.240 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:51:07.240 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:56.230481+0000", 2024-09-20T22:51:07.240 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:51:07.240 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.240 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.240 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:51:07.240 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:56.230481+0000", 2024-09-20T22:51:07.240 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:51:07.240 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.240 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.240 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:51:07.240 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:56.230482+0000", 2024-09-20T22:51:07.240 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.674e-06 2024-09-20T22:51:07.240 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.241 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.241 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:51:07.241 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:56.230497+0000", 2024-09-20T22:51:07.241 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.4178e-05 2024-09-20T22:51:07.241 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.241 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.241 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:51:07.241 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:56.230517+0000", 2024-09-20T22:51:07.241 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.0336e-05 2024-09-20T22:51:07.241 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.241 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.241 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:51:07.242 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:56.230527+0000", 2024-09-20T22:51:07.242 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.6039999999999999e-06 2024-09-20T22:51:07.242 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.242 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.242 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:51:07.242 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:56.230631+0000", 2024-09-20T22:51:07.242 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00010445 2024-09-20T22:51:07.242 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.242 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.242 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op must wait for map", 2024-09-20T22:51:07.242 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:56.230639+0000", 2024-09-20T22:51:07.242 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.7759999999999997e-06 2024-09-20T22:51:07.242 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.243 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.243 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:51:07.243 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:56.237931+0000", 2024-09-20T22:51:07.243 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0072925630000000002 2024-09-20T22:51:07.243 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.243 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.243 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:51:07.243 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:56.238030+0000", 2024-09-20T22:51:07.243 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.8592999999999996e-05 2024-09-20T22:51:07.243 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.243 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.243 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 1,2", 2024-09-20T22:51:07.243 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:56.244691+0000", 2024-09-20T22:51:07.244 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0066614680000000003 2024-09-20T22:51:07.244 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.244 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.244 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:51:07.244 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:56.258211+0000", 2024-09-20T22:51:07.244 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013519721 2024-09-20T22:51:07.244 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.244 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.244 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:51:07.244 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:56.258404+0000", 2024-09-20T22:51:07.244 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00019257000000000001 2024-09-20T22:51:07.244 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.244 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.244 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:51:07.245 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:56.258595+0000", 2024-09-20T22:51:07.245 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000191387 2024-09-20T22:51:07.245 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.245 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.245 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:51:07.245 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:56.258635+0000", 2024-09-20T22:51:07.245 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.9918e-05 2024-09-20T22:51:07.245 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.245 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.245 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:51:07.245 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:56.258680+0000", 2024-09-20T22:51:07.245 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.5058000000000001e-05 2024-09-20T22:51:07.245 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:51:07.246 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:51:07.246 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:51:07.246 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.246 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.246 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:5677 7.2 7:4fa4b2b3:::smithi08477203-21:head [copy-from ver 1213] snapc fa=[fa,f9,f8,f7,f5,f4,f2,f1,ef,ed,ec,ea,e6,e4,e2,df,db,da,d7,d4,d3,ce,c5,c2,bf,b4,b2,a7] ondisk+write+known_if_redirected+supports_pool_eio e572)", 2024-09-20T22:51:07.246 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:50:52.384255+0000", 2024-09-20T22:51:07.246 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 14.710848598, 2024-09-20T22:51:07.246 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.027330337, 2024-09-20T22:51:07.246 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:51:07.246 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:51:07.246 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:51:07.246 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:51:07.246 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:51:07.246 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:51:07.247 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 5677 2024-09-20T22:51:07.247 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.247 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:51:07.247 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.247 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:51:07.247 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:52.384255+0000", 2024-09-20T22:51:07.247 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:51:07.247 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.247 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.247 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:51:07.247 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:52.384255+0000", 2024-09-20T22:51:07.247 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:51:07.247 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.247 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.248 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:51:07.248 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:52.384258+0000", 2024-09-20T22:51:07.248 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.0600000000000002e-06 2024-09-20T22:51:07.248 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.248 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.248 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:51:07.248 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:52.384267+0000", 2024-09-20T22:51:07.248 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.3519999999999996e-06 2024-09-20T22:51:07.248 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.248 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.248 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:51:07.248 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:52.384288+0000", 2024-09-20T22:51:07.248 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.1039e-05 2024-09-20T22:51:07.248 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.249 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.249 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:51:07.249 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:52.384298+0000", 2024-09-20T22:51:07.249 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.6630000000000005e-06 2024-09-20T22:51:07.249 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.249 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.249 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:51:07.249 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:52.384381+0000", 2024-09-20T22:51:07.249 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.3372000000000001e-05 2024-09-20T22:51:07.249 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.249 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.249 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:51:07.249 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:52.384455+0000", 2024-09-20T22:51:07.249 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.4414999999999998e-05 2024-09-20T22:51:07.249 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.250 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.250 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 1,2", 2024-09-20T22:51:07.250 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:52.398437+0000", 2024-09-20T22:51:07.250 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013981959 2024-09-20T22:51:07.250 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.250 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.250 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:51:07.250 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:52.410532+0000", 2024-09-20T22:51:07.250 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.012094898 2024-09-20T22:51:07.250 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.250 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.250 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:51:07.250 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:52.411064+0000", 2024-09-20T22:51:07.250 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00053131999999999995 2024-09-20T22:51:07.251 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.251 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.251 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:51:07.251 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:52.411446+0000", 2024-09-20T22:51:07.251 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00038282300000000002 2024-09-20T22:51:07.251 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.251 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.251 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:51:07.251 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:52.411525+0000", 2024-09-20T22:51:07.251 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.8339999999999999e-05 2024-09-20T22:51:07.251 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.251 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.251 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:51:07.251 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:52.411586+0000", 2024-09-20T22:51:07.252 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.1095999999999999e-05 2024-09-20T22:51:07.252 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:51:07.252 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:51:07.252 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:51:07.252 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.252 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.252 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:5695 7.a 7:5aaef466:::smithi08477203-40:head [copy-from ver 402] snapc fc=[fc,fb,fa,f9,f8,f7,f5,f4,f2,f1,ef,ed,ec,ea,e6,e4,e2,df,db,da,d7,d4,d3,ce,c5,c2,bf,b4,b2,a7] ondisk+write+known_if_redirected+supports_pool_eio e574)", 2024-09-20T22:51:07.252 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:50:54.262282+0000", 2024-09-20T22:51:07.252 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 12.832822261, 2024-09-20T22:51:07.252 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.022843216999999999, 2024-09-20T22:51:07.252 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:51:07.252 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:51:07.252 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:51:07.252 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:51:07.253 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:51:07.253 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:51:07.253 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 5695 2024-09-20T22:51:07.253 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.253 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:51:07.253 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.253 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:51:07.253 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:54.262282+0000", 2024-09-20T22:51:07.253 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:51:07.253 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.253 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.253 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:51:07.253 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:54.262282+0000", 2024-09-20T22:51:07.253 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:51:07.253 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.254 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.254 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:51:07.254 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:54.262284+0000", 2024-09-20T22:51:07.254 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.756e-06 2024-09-20T22:51:07.254 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.254 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.254 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:51:07.254 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:54.262294+0000", 2024-09-20T22:51:07.254 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0365e-05 2024-09-20T22:51:07.254 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.254 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.254 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:51:07.254 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:54.262313+0000", 2024-09-20T22:51:07.255 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.9363000000000001e-05 2024-09-20T22:51:07.255 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.255 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.255 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:51:07.255 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:54.262324+0000", 2024-09-20T22:51:07.255 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0688e-05 2024-09-20T22:51:07.255 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.255 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.255 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:51:07.255 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:54.262387+0000", 2024-09-20T22:51:07.255 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.2769999999999997e-05 2024-09-20T22:51:07.255 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.255 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.255 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op must wait for map", 2024-09-20T22:51:07.256 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:54.262397+0000", 2024-09-20T22:51:07.256 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.9420000000000007e-06 2024-09-20T22:51:07.256 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.256 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.256 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:51:07.256 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:54.264747+0000", 2024-09-20T22:51:07.256 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0023499910000000001 2024-09-20T22:51:07.256 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.256 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.256 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:51:07.256 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:54.264890+0000", 2024-09-20T22:51:07.256 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014352899999999999 2024-09-20T22:51:07.256 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.256 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.257 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 2,5", 2024-09-20T22:51:07.257 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:54.273677+0000", 2024-09-20T22:51:07.257 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0087865930000000005 2024-09-20T22:51:07.257 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.257 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.257 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:51:07.257 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:54.281449+0000", 2024-09-20T22:51:07.257 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.007771956 2024-09-20T22:51:07.257 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.257 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.257 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:51:07.257 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:54.283324+0000", 2024-09-20T22:51:07.257 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0018756689999999999 2024-09-20T22:51:07.258 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.258 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.258 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:51:07.258 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:54.285043+0000", 2024-09-20T22:51:07.258 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0017182409999999999 2024-09-20T22:51:07.258 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.258 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.258 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:51:07.258 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:54.285084+0000", 2024-09-20T22:51:07.258 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.1124999999999997e-05 2024-09-20T22:51:07.258 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:07.258 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:07.258 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:51:07.259 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:54.285125+0000", 2024-09-20T22:51:07.259 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.1229000000000001e-05 2024-09-20T22:51:07.259 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:51:07.259 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:51:07.259 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:51:07.259 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:51:07.259 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:51:07.259 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-20T22:51:07.286 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:07.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:07.287 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:51:07.338 INFO:tasks.rados.rados.0.smithi084.stdout:2314: done (3 left) 2024-09-20T22:51:07.339 INFO:tasks.rados.rados.0.smithi084.stdout:2315: done (2 left) 2024-09-20T22:51:07.339 INFO:tasks.rados.rados.0.smithi084.stdout:2316: done (1 left) 2024-09-20T22:51:07.339 INFO:tasks.rados.rados.0.smithi084.stdout:2317: done (0 left) 2024-09-20T22:51:07.339 INFO:tasks.rados.rados.0.smithi084.stdout:2318: snap_remove snap 237 2024-09-20T22:51:07.388 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:51:07.413 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:07.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:07.414 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:51:07.514 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:51:07.538 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:07.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:07.539 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:51:07.614 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:07 smithi084 ceph-mon[29774]: 6.15 scrub starts 2024-09-20T22:51:07.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:07 smithi084 ceph-mon[29774]: 6.15 scrub ok 2024-09-20T22:51:07.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:07 smithi084 ceph-mon[29774]: 4.15 scrub starts 2024-09-20T22:51:07.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:07 smithi084 ceph-mon[29774]: 4.15 scrub ok 2024-09-20T22:51:07.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:07 smithi084 ceph-mon[29774]: pgmap v780: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 556 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 17 MiB/s rd, 2.5 MiB/s wr, 25 op/s 2024-09-20T22:51:07.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:07 smithi084 ceph-mon[29774]: osdmap e586: 8 total, 8 up, 8 in 2024-09-20T22:51:07.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:07 smithi084 ceph-mon[29774]: 5.12 scrub starts 2024-09-20T22:51:07.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:07 smithi084 ceph-mon[29774]: 5.12 scrub ok 2024-09-20T22:51:07.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:07 smithi084 ceph-mon[29774]: 4.d scrub starts 2024-09-20T22:51:07.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:07 smithi084 ceph-mon[29774]: 4.d scrub ok 2024-09-20T22:51:07.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:07 smithi084 ceph-mon[40024]: 6.15 scrub starts 2024-09-20T22:51:07.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:07 smithi084 ceph-mon[40024]: 6.15 scrub ok 2024-09-20T22:51:07.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:07 smithi084 ceph-mon[40024]: 4.15 scrub starts 2024-09-20T22:51:07.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:07 smithi084 ceph-mon[40024]: 4.15 scrub ok 2024-09-20T22:51:07.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:07 smithi084 ceph-mon[40024]: pgmap v780: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 556 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 17 MiB/s rd, 2.5 MiB/s wr, 25 op/s 2024-09-20T22:51:07.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:07 smithi084 ceph-mon[40024]: osdmap e586: 8 total, 8 up, 8 in 2024-09-20T22:51:07.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:07 smithi084 ceph-mon[40024]: 5.12 scrub starts 2024-09-20T22:51:07.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:07 smithi084 ceph-mon[40024]: 5.12 scrub ok 2024-09-20T22:51:07.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:07 smithi084 ceph-mon[40024]: 4.d scrub starts 2024-09-20T22:51:07.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:07 smithi084 ceph-mon[40024]: 4.d scrub ok 2024-09-20T22:51:07.639 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:51:07.660 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:07.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:07.661 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:51:07.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:07 smithi154 ceph-mon[33264]: 6.15 scrub starts 2024-09-20T22:51:07.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:07 smithi154 ceph-mon[33264]: 6.15 scrub ok 2024-09-20T22:51:07.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:07 smithi154 ceph-mon[33264]: 4.15 scrub starts 2024-09-20T22:51:07.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:07 smithi154 ceph-mon[33264]: 4.15 scrub ok 2024-09-20T22:51:07.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:07 smithi154 ceph-mon[33264]: pgmap v780: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 556 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 17 MiB/s rd, 2.5 MiB/s wr, 25 op/s 2024-09-20T22:51:07.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:07 smithi154 ceph-mon[33264]: osdmap e586: 8 total, 8 up, 8 in 2024-09-20T22:51:07.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:07 smithi154 ceph-mon[33264]: 5.12 scrub starts 2024-09-20T22:51:07.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:07 smithi154 ceph-mon[33264]: 5.12 scrub ok 2024-09-20T22:51:07.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:07 smithi154 ceph-mon[33264]: 4.d scrub starts 2024-09-20T22:51:07.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:07 smithi154 ceph-mon[33264]: 4.d scrub ok 2024-09-20T22:51:07.762 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:51:07.783 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:07.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:07.784 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:51:07.885 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:51:07.910 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:07.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:07.911 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:51:08.012 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:51:08.033 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:08.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:08.034 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:51:08.135 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:51:08.161 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:08.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:08.162 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:51:08.257 INFO:tasks.rados.rados.0.smithi084.stdout:2318: done (0 left) 2024-09-20T22:51:08.257 INFO:tasks.rados.rados.0.smithi084.stdout:2319: read oid 29 snap -1 2024-09-20T22:51:08.257 INFO:tasks.rados.rados.0.smithi084.stdout:2319: expect (ObjNum 778 snap 251 seq_num 778) 2024-09-20T22:51:08.257 INFO:tasks.rados.rados.0.smithi084.stdout:2320: write oid 8 current snap is 256 2024-09-20T22:51:08.257 INFO:tasks.rados.rados.0.smithi084.stdout:2320: seq_num 801 ranges {662407=491514,1855959=571609,2846876=615531} 2024-09-20T22:51:08.263 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:51:08.264 INFO:tasks.rados.rados.0.smithi084.stdout:2320: writing smithi08477203-8 from 662407 to 1153921 tid 1 2024-09-20T22:51:08.272 INFO:tasks.rados.rados.0.smithi084.stdout:2320: writing smithi08477203-8 from 1855959 to 2427568 tid 2 2024-09-20T22:51:08.278 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:51:08 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: ::ffff:172.21.15.154 - - [20/Sep/2024:22:51:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-20T22:51:08.279 INFO:tasks.rados.rados.0.smithi084.stdout:2320: writing smithi08477203-8 from 2846876 to 3462407 tid 3 2024-09-20T22:51:08.285 INFO:tasks.rados.rados.0.smithi084.stdout:2319: expect (ObjNum 778 snap 251 seq_num 778) 2024-09-20T22:51:08.295 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:08.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:08.296 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:51:08.370 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-20T22:51:08.397 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:51:08.431 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-20T22:51:08.434 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:08.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:08.435 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:51:08.537 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:51:08.542 INFO:tasks.rados.rados.0.smithi084.stdout:2320: finishing write tid 1 to smithi08477203-8 2024-09-20T22:51:08.542 INFO:tasks.rados.rados.0.smithi084.stdout:2320: finishing write tid 2 to smithi08477203-8 2024-09-20T22:51:08.542 INFO:tasks.rados.rados.0.smithi084.stdout:2319: done (1 left) 2024-09-20T22:51:08.542 INFO:tasks.rados.rados.0.smithi084.stdout:2321: read oid 29 snap 216 2024-09-20T22:51:08.542 INFO:tasks.rados.rados.0.smithi084.stdout:2321: expect (ObjNum 692 snap 215 seq_num 692) 2024-09-20T22:51:08.542 INFO:tasks.rados.rados.0.smithi084.stdout:2322: write (excl) oid 43 current snap is 256 2024-09-20T22:51:08.542 INFO:tasks.rados.rados.0.smithi084.stdout:2322: seq_num 802 ranges {679214=566763,1798088=715267,3157946=321268} 2024-09-20T22:51:08.549 INFO:tasks.rados.rados.0.smithi084.stdout:2322: writing smithi08477203-43 from 679214 to 1245977 tid 1 2024-09-20T22:51:08.556 INFO:tasks.rados.rados.0.smithi084.stdout:2322: writing smithi08477203-43 from 1798088 to 2513355 tid 2 2024-09-20T22:51:08.561 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:08.562 INFO:tasks.rados.rados.0.smithi084.stdout:2322: writing smithi08477203-43 from 3157946 to 3479214 tid 3 2024-09-20T22:51:08.562 INFO:tasks.rados.rados.0.smithi084.stdout:2323: snap_create 2024-09-20T22:51:08.562 INFO:tasks.rados.rados.0.smithi084.stdout:2320: finishing write tid 3 to smithi08477203-8 2024-09-20T22:51:08.562 INFO:tasks.rados.rados.0.smithi084.stdout:2320: finishing write tid 4 to smithi08477203-8 2024-09-20T22:51:08.562 INFO:tasks.rados.rados.0.smithi084.stdout:2320: finishing write tid 5 to smithi08477203-8 2024-09-20T22:51:08.562 INFO:tasks.rados.rados.0.smithi084.stdout:2320: oid 8 updating version 0 to 812 2024-09-20T22:51:08.562 INFO:tasks.rados.rados.0.smithi084.stdout:2320: oid 8 version 812 is already newer than 811 2024-09-20T22:51:08.562 INFO:tasks.rados.rados.0.smithi084.stdout:2320: oid 8 version 812 is already newer than 809 2024-09-20T22:51:08.563 INFO:tasks.rados.rados.0.smithi084.stdout:2320: oid 8 version 812 is already newer than 810 2024-09-20T22:51:08.563 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 8 v 812 (ObjNum 801 snap 256 seq_num 801) dirty exists 2024-09-20T22:51:08.563 INFO:tasks.rados.rados.0.smithi084.stdout:2320: left oid 8 (ObjNum 801 snap 256 seq_num 801) 2024-09-20T22:51:08.563 INFO:tasks.rados.rados.0.smithi084.stdout:2322: finishing write tid 1 to smithi08477203-43 2024-09-20T22:51:08.563 INFO:tasks.rados.rados.0.smithi084.stdout:2321: expect (ObjNum 692 snap 215 seq_num 692) 2024-09-20T22:51:08.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:08.564 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:51:08.585 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:08 smithi084 ceph-mon[40024]: osdmap e587: 8 total, 8 up, 8 in 2024-09-20T22:51:08.585 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:08 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/2744459664' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:51:08.585 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:08 smithi084 ceph-mon[40024]: purged_snaps scrub starts 2024-09-20T22:51:08.585 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:08 smithi084 ceph-mon[40024]: purged_snaps scrub ok 2024-09-20T22:51:08.585 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:08 smithi084 ceph-mon[40024]: 4.13 scrub starts 2024-09-20T22:51:08.585 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:08 smithi084 ceph-mon[40024]: 4.13 scrub ok 2024-09-20T22:51:08.585 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:08 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:51:08.585 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:08 smithi084 ceph-mon[40024]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T22:51:08.586 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:08 smithi084 ceph-mon[40024]: Cluster is now healthy 2024-09-20T22:51:08.586 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:08 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/2744459664' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T22:51:08.586 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:08 smithi084 ceph-mon[40024]: osdmap e588: 8 total, 8 up, 8 in 2024-09-20T22:51:08.586 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:08 smithi084 ceph-mon[29774]: osdmap e587: 8 total, 8 up, 8 in 2024-09-20T22:51:08.586 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:08 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/2744459664' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:51:08.586 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:08 smithi084 ceph-mon[29774]: purged_snaps scrub starts 2024-09-20T22:51:08.586 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:08 smithi084 ceph-mon[29774]: purged_snaps scrub ok 2024-09-20T22:51:08.586 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:08 smithi084 ceph-mon[29774]: 4.13 scrub starts 2024-09-20T22:51:08.586 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:08 smithi084 ceph-mon[29774]: 4.13 scrub ok 2024-09-20T22:51:08.586 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:08 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:51:08.586 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:08 smithi084 ceph-mon[29774]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T22:51:08.586 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:08 smithi084 ceph-mon[29774]: Cluster is now healthy 2024-09-20T22:51:08.587 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:08 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/2744459664' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T22:51:08.587 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:08 smithi084 ceph-mon[29774]: osdmap e588: 8 total, 8 up, 8 in 2024-09-20T22:51:08.619 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:51:08.665 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:51:08.685 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:08.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:08.686 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:51:08.695 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is unset 2024-09-20T22:51:08.741 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:08 smithi154 ceph-mon[33264]: osdmap e587: 8 total, 8 up, 8 in 2024-09-20T22:51:08.741 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:08 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/2744459664' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:51:08.741 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:08 smithi154 ceph-mon[33264]: purged_snaps scrub starts 2024-09-20T22:51:08.741 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:08 smithi154 ceph-mon[33264]: purged_snaps scrub ok 2024-09-20T22:51:08.741 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:08 smithi154 ceph-mon[33264]: 4.13 scrub starts 2024-09-20T22:51:08.741 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:08 smithi154 ceph-mon[33264]: 4.13 scrub ok 2024-09-20T22:51:08.741 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:08 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:51:08.741 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:08 smithi154 ceph-mon[33264]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T22:51:08.741 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:08 smithi154 ceph-mon[33264]: Cluster is now healthy 2024-09-20T22:51:08.741 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:08 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/2744459664' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T22:51:08.741 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:08 smithi154 ceph-mon[33264]: osdmap e588: 8 total, 8 up, 8 in 2024-09-20T22:51:08.787 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:51:08.808 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:08.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:08.809 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:51:08.854 INFO:tasks.rados.rados.0.smithi084.stdout:2322: finishing write tid 2 to smithi08477203-43 2024-09-20T22:51:08.854 INFO:tasks.rados.rados.0.smithi084.stdout:2322: finishing write tid 3 to smithi08477203-43 2024-09-20T22:51:08.854 INFO:tasks.rados.rados.0.smithi084.stdout:2322: finishing write tid 4 to smithi08477203-43 2024-09-20T22:51:08.854 INFO:tasks.rados.rados.0.smithi084.stdout:2322: finishing write tid 5 to smithi08477203-43 2024-09-20T22:51:08.855 INFO:tasks.rados.rados.0.smithi084.stdout:2322: oid 43 updating version 0 to 1395 2024-09-20T22:51:08.855 INFO:tasks.rados.rados.0.smithi084.stdout:2322: oid 43 version 1395 is already newer than 1394 2024-09-20T22:51:08.855 INFO:tasks.rados.rados.0.smithi084.stdout:2322: oid 43 updating version 1395 to 1397 2024-09-20T22:51:08.855 INFO:tasks.rados.rados.0.smithi084.stdout:2322: oid 43 version 1397 is already newer than 1396 2024-09-20T22:51:08.855 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 43 v 1397 (ObjNum 802 snap 256 seq_num 802) dirty exists 2024-09-20T22:51:08.855 INFO:tasks.rados.rados.0.smithi084.stdout:2322: left oid 43 (ObjNum 802 snap 256 seq_num 802) 2024-09-20T22:51:08.910 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:51:08.931 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:08.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:08.932 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:51:09.033 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:51:09.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:09.057 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:09.057 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:51:09.158 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:51:09.173 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-20T22:51:09.173 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [], 2024-09-20T22:51:09.173 INFO:teuthology.orchestra.run.smithi154.stdout: "num_ops": 0 2024-09-20T22:51:09.173 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-20T22:51:09.180 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:09.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:09.181 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:51:09.282 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:51:09.303 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:09.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:09.304 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:51:09.349 INFO:tasks.rados.rados.0.smithi084.stdout:2320: done (3 left) 2024-09-20T22:51:09.349 INFO:tasks.rados.rados.0.smithi084.stdout:2321: done (2 left) 2024-09-20T22:51:09.349 INFO:tasks.rados.rados.0.smithi084.stdout:2322: done (1 left) 2024-09-20T22:51:09.349 INFO:tasks.rados.rados.0.smithi084.stdout:2323: done (0 left) 2024-09-20T22:51:09.349 INFO:tasks.rados.rados.0.smithi084.stdout:2324: snap_create 2024-09-20T22:51:09.405 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:51:09.428 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:09.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:09.429 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:51:09.529 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:51:09.551 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:09.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:09.552 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:51:09.559 INFO:journalctl@ceph.iscsi.iscsi.a.smithi154.stdout:Sep 20 22:51:09 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-iscsi-iscsi-a[59730]: debug there is no tcmu-runner data available 2024-09-20T22:51:09.559 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:09 smithi154 ceph-mon[33264]: pgmap v783: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 556 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-20T22:51:09.559 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:09 smithi154 ceph-mon[33264]: 4.1c scrub starts 2024-09-20T22:51:09.560 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:09 smithi154 ceph-mon[33264]: 4.1c scrub ok 2024-09-20T22:51:09.560 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:09 smithi154 ceph-mon[33264]: 4.19 scrub starts 2024-09-20T22:51:09.560 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:09 smithi154 ceph-mon[33264]: 4.19 scrub ok 2024-09-20T22:51:09.560 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:09 smithi154 ceph-mon[33264]: 3.14 scrub starts 2024-09-20T22:51:09.560 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:09 smithi154 ceph-mon[33264]: 3.14 scrub ok 2024-09-20T22:51:09.602 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:09 smithi084 ceph-mon[40024]: pgmap v783: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 556 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-20T22:51:09.602 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:09 smithi084 ceph-mon[40024]: 4.1c scrub starts 2024-09-20T22:51:09.602 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:09 smithi084 ceph-mon[40024]: 4.1c scrub ok 2024-09-20T22:51:09.602 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:09 smithi084 ceph-mon[40024]: 4.19 scrub starts 2024-09-20T22:51:09.602 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:09 smithi084 ceph-mon[40024]: 4.19 scrub ok 2024-09-20T22:51:09.602 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:09 smithi084 ceph-mon[40024]: 3.14 scrub starts 2024-09-20T22:51:09.602 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:09 smithi084 ceph-mon[40024]: 3.14 scrub ok 2024-09-20T22:51:09.603 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:09 smithi084 ceph-mon[29774]: pgmap v783: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 556 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-20T22:51:09.603 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:09 smithi084 ceph-mon[29774]: 4.1c scrub starts 2024-09-20T22:51:09.603 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:09 smithi084 ceph-mon[29774]: 4.1c scrub ok 2024-09-20T22:51:09.603 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:09 smithi084 ceph-mon[29774]: 4.19 scrub starts 2024-09-20T22:51:09.603 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:09 smithi084 ceph-mon[29774]: 4.19 scrub ok 2024-09-20T22:51:09.603 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:09 smithi084 ceph-mon[29774]: 3.14 scrub starts 2024-09-20T22:51:09.603 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:09 smithi084 ceph-mon[29774]: 3.14 scrub ok 2024-09-20T22:51:09.654 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:51:09.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:09.675 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:09.675 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:51:09.711 INFO:teuthology.orchestra.run.smithi084.stderr:no valid command found; 10 closest matches: 2024-09-20T22:51:09.711 INFO:teuthology.orchestra.run.smithi084.stderr:0 2024-09-20T22:51:09.711 INFO:teuthology.orchestra.run.smithi084.stderr:1 2024-09-20T22:51:09.712 INFO:teuthology.orchestra.run.smithi084.stderr:2 2024-09-20T22:51:09.712 INFO:teuthology.orchestra.run.smithi084.stderr:abort 2024-09-20T22:51:09.712 INFO:teuthology.orchestra.run.smithi084.stderr:assert 2024-09-20T22:51:09.712 INFO:teuthology.orchestra.run.smithi084.stderr:bench [] [] [] [] 2024-09-20T22:51:09.712 INFO:teuthology.orchestra.run.smithi084.stderr:bluefs debug_inject_read_zeros 2024-09-20T22:51:09.712 INFO:teuthology.orchestra.run.smithi084.stderr:bluefs files list 2024-09-20T22:51:09.712 INFO:teuthology.orchestra.run.smithi084.stderr:bluefs stats 2024-09-20T22:51:09.712 INFO:teuthology.orchestra.run.smithi084.stderr:bluestore allocator dump block 2024-09-20T22:51:09.712 INFO:teuthology.orchestra.run.smithi084.stderr:Error EINVAL: invalid command 2024-09-20T22:51:09.776 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:51:09.798 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:09.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:09.798 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:51:09.899 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:51:09.920 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:09.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:09.921 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:51:09.979 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.4 dump_blocked_ops 2024-09-20T22:51:10.022 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:51:10.043 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:10.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:10.044 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:51:10.145 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:51:10.166 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:10.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:10.167 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:51:10.245 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:51:10.268 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:51:10.290 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:10.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:10.291 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:51:10.356 INFO:tasks.rados.rados.0.smithi084.stdout:2324: done (0 left) 2024-09-20T22:51:10.356 INFO:tasks.rados.rados.0.smithi084.stdout:2325: snap_create 2024-09-20T22:51:10.392 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:51:10.414 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:10.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:10.415 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:51:10.415 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-20T22:51:10.416 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-20T22:51:10.416 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph tell osd.4 'trim stale osdmaps' 2024-09-20T22:51:10.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:10 smithi154 ceph-mon[33264]: purged_snaps scrub starts 2024-09-20T22:51:10.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:10 smithi154 ceph-mon[33264]: purged_snaps scrub ok 2024-09-20T22:51:10.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:10 smithi154 ceph-mon[33264]: osdmap e589: 8 total, 8 up, 8 in 2024-09-20T22:51:10.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:10 smithi154 ceph-mon[33264]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:51:10.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:10 smithi154 ceph-mon[33264]: 5.1 scrub starts 2024-09-20T22:51:10.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:10 smithi154 ceph-mon[33264]: 5.1 scrub ok 2024-09-20T22:51:10.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:10 smithi154 ceph-mon[33264]: 1.7 scrub starts 2024-09-20T22:51:10.514 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:10 smithi154 ceph-mon[33264]: 1.7 scrub ok 2024-09-20T22:51:10.516 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:51:10.531 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:51:10.537 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:10.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:10.538 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:51:10.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:10 smithi084 ceph-mon[40024]: purged_snaps scrub starts 2024-09-20T22:51:10.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:10 smithi084 ceph-mon[40024]: purged_snaps scrub ok 2024-09-20T22:51:10.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:10 smithi084 ceph-mon[40024]: osdmap e589: 8 total, 8 up, 8 in 2024-09-20T22:51:10.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:10 smithi084 ceph-mon[40024]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:51:10.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:10 smithi084 ceph-mon[40024]: 5.1 scrub starts 2024-09-20T22:51:10.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:10 smithi084 ceph-mon[40024]: 5.1 scrub ok 2024-09-20T22:51:10.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:10 smithi084 ceph-mon[40024]: 1.7 scrub starts 2024-09-20T22:51:10.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:10 smithi084 ceph-mon[40024]: 1.7 scrub ok 2024-09-20T22:51:10.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:10 smithi084 ceph-mon[29774]: purged_snaps scrub starts 2024-09-20T22:51:10.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:10 smithi084 ceph-mon[29774]: purged_snaps scrub ok 2024-09-20T22:51:10.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:10 smithi084 ceph-mon[29774]: osdmap e589: 8 total, 8 up, 8 in 2024-09-20T22:51:10.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:10 smithi084 ceph-mon[29774]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:51:10.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:10 smithi084 ceph-mon[29774]: 5.1 scrub starts 2024-09-20T22:51:10.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:10 smithi084 ceph-mon[29774]: 5.1 scrub ok 2024-09-20T22:51:10.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:10 smithi084 ceph-mon[29774]: 1.7 scrub starts 2024-09-20T22:51:10.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:10 smithi084 ceph-mon[29774]: 1.7 scrub ok 2024-09-20T22:51:10.639 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:51:10.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:10.661 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:10.661 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:51:10.762 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:51:10.765 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-20T22:51:10.765 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [], 2024-09-20T22:51:10.765 INFO:teuthology.orchestra.run.smithi154.stdout: "complaint_time": 30, 2024-09-20T22:51:10.765 INFO:teuthology.orchestra.run.smithi154.stdout: "num_blocked_ops": 0 2024-09-20T22:51:10.765 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-20T22:51:10.783 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:10.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:10.784 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:51:10.885 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:51:10.908 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:10.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:10.909 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:51:11.010 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:51:11.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:11.034 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:11.035 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:51:11.070 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:51:11.071 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:51:11.079 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:51:11.079 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:51:11.096 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:51:11.097 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:51:11.108 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:51:11.114 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-20T22:51:11.114 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T22:51:11.131 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:51:11.131 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:51:11.136 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:51:11.148 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:51:11.148 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:51:11.158 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:11.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:11.159 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:51:11.165 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:51:11.165 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:51:11.181 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:51:11.181 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:51:11.260 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:51:11.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:11.282 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:11.283 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:51:11.358 INFO:tasks.rados.rados.0.smithi084.stdout:2325: done (0 left) 2024-09-20T22:51:11.358 INFO:tasks.rados.rados.0.smithi084.stdout:2326: snap_remove snap 226 2024-09-20T22:51:11.375 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.4 dump_historic_ops 2024-09-20T22:51:11.383 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:51:11.406 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:11.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:11.425 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:51:11.445 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd set noscrub 2024-09-20T22:51:11.484 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:11 smithi084 ceph-mon[29774]: 3.1d scrub starts 2024-09-20T22:51:11.484 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:11 smithi084 ceph-mon[29774]: 3.1d scrub ok 2024-09-20T22:51:11.484 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:11 smithi084 ceph-mon[29774]: pgmap v786: 158 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 153 active+clean; 530 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 19 MiB/s rd, 5.0 MiB/s wr, 34 op/s 2024-09-20T22:51:11.484 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:11 smithi084 ceph-mon[29774]: osdmap e590: 8 total, 8 up, 8 in 2024-09-20T22:51:11.484 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:11 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/2332852237' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:51:11.485 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:11 smithi084 ceph-mon[40024]: 3.1d scrub starts 2024-09-20T22:51:11.485 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:11 smithi084 ceph-mon[40024]: 3.1d scrub ok 2024-09-20T22:51:11.485 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:11 smithi084 ceph-mon[40024]: pgmap v786: 158 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 153 active+clean; 530 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 19 MiB/s rd, 5.0 MiB/s wr, 34 op/s 2024-09-20T22:51:11.485 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:11 smithi084 ceph-mon[40024]: osdmap e590: 8 total, 8 up, 8 in 2024-09-20T22:51:11.485 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:11 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/2332852237' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:51:11.526 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:51:11.548 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:11.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:11.549 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:51:11.621 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:51:11.650 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:51:11.671 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:11.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:11.672 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:51:11.743 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:11 smithi154 ceph-mon[33264]: 3.1d scrub starts 2024-09-20T22:51:11.743 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:11 smithi154 ceph-mon[33264]: 3.1d scrub ok 2024-09-20T22:51:11.743 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:11 smithi154 ceph-mon[33264]: pgmap v786: 158 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 153 active+clean; 530 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 19 MiB/s rd, 5.0 MiB/s wr, 34 op/s 2024-09-20T22:51:11.743 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:11 smithi154 ceph-mon[33264]: osdmap e590: 8 total, 8 up, 8 in 2024-09-20T22:51:11.743 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:11 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/2332852237' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:51:11.772 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:51:11.794 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:11.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:11.795 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:51:11.895 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:51:11.917 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:11.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:11.918 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:51:12.018 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:51:12.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:12.042 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:12.042 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:51:12.143 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:51:12.163 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:12.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:12.164 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:51:12.248 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-20T22:51:12.248 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:51:12.265 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:51:12.276 INFO:teuthology.orchestra.run.smithi084.stderr:no valid command found; 10 closest matches: 2024-09-20T22:51:12.277 INFO:teuthology.orchestra.run.smithi084.stderr:0 2024-09-20T22:51:12.277 INFO:teuthology.orchestra.run.smithi084.stderr:1 2024-09-20T22:51:12.277 INFO:teuthology.orchestra.run.smithi084.stderr:2 2024-09-20T22:51:12.277 INFO:teuthology.orchestra.run.smithi084.stderr:abort 2024-09-20T22:51:12.277 INFO:teuthology.orchestra.run.smithi084.stderr:assert 2024-09-20T22:51:12.277 INFO:teuthology.orchestra.run.smithi084.stderr:bench [] [] [] [] 2024-09-20T22:51:12.277 INFO:teuthology.orchestra.run.smithi084.stderr:bluefs debug_inject_read_zeros 2024-09-20T22:51:12.277 INFO:teuthology.orchestra.run.smithi084.stderr:bluefs files list 2024-09-20T22:51:12.277 INFO:teuthology.orchestra.run.smithi084.stderr:bluefs stats 2024-09-20T22:51:12.277 INFO:teuthology.orchestra.run.smithi084.stderr:bluestore allocator dump block 2024-09-20T22:51:12.277 INFO:teuthology.orchestra.run.smithi084.stderr:Error EINVAL: invalid command 2024-09-20T22:51:12.287 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:12.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:12.288 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:51:12.370 INFO:tasks.rados.rados.0.smithi084.stdout:2326: done (0 left) 2024-09-20T22:51:12.371 INFO:tasks.rados.rados.0.smithi084.stdout:2327: snap_remove snap 195 2024-09-20T22:51:12.389 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:51:12.410 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:12.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:12.411 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:51:12.512 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:51:12.535 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:12.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:12.536 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:51:12.633 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:12 smithi084 ceph-mon[29774]: 5.10 scrub starts 2024-09-20T22:51:12.633 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:12 smithi084 ceph-mon[29774]: 5.10 scrub ok 2024-09-20T22:51:12.633 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:12 smithi084 ceph-mon[29774]: 7.9 deep-scrub starts 2024-09-20T22:51:12.633 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:12 smithi084 ceph-mon[29774]: 7.9 deep-scrub ok 2024-09-20T22:51:12.633 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:12 smithi084 ceph-mon[29774]: osdmap e591: 8 total, 8 up, 8 in 2024-09-20T22:51:12.633 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:12 smithi084 ceph-mon[40024]: 5.10 scrub starts 2024-09-20T22:51:12.633 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:12 smithi084 ceph-mon[40024]: 5.10 scrub ok 2024-09-20T22:51:12.633 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:12 smithi084 ceph-mon[40024]: 7.9 deep-scrub starts 2024-09-20T22:51:12.634 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:12 smithi084 ceph-mon[40024]: 7.9 deep-scrub ok 2024-09-20T22:51:12.634 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:12 smithi084 ceph-mon[40024]: osdmap e591: 8 total, 8 up, 8 in 2024-09-20T22:51:12.637 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:51:12.658 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:12.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:12.659 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:51:12.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:12 smithi154 ceph-mon[33264]: 5.10 scrub starts 2024-09-20T22:51:12.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:12 smithi154 ceph-mon[33264]: 5.10 scrub ok 2024-09-20T22:51:12.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:12 smithi154 ceph-mon[33264]: 7.9 deep-scrub starts 2024-09-20T22:51:12.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:12 smithi154 ceph-mon[33264]: 7.9 deep-scrub ok 2024-09-20T22:51:12.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:12 smithi154 ceph-mon[33264]: osdmap e591: 8 total, 8 up, 8 in 2024-09-20T22:51:12.760 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:51:12.781 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:12.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:12.782 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:51:12.839 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-20T22:51:12.883 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:51:12.905 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:12.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:12.906 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:51:12.968 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-20T22:51:12.968 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-20T22:51:13.007 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:51:13.029 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:13.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:13.030 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:51:13.055 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:51:13.089 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:51:13.131 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:51:13.153 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:13.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:13.154 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:51:13.255 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:51:13.276 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:13.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:13.277 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:51:13.378 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:51:13.379 INFO:tasks.rados.rados.0.smithi084.stdout:2327: done (0 left) 2024-09-20T22:51:13.380 INFO:tasks.rados.rados.0.smithi084.stdout:2328: read oid 7 snap 245 2024-09-20T22:51:13.380 INFO:tasks.rados.rados.0.smithi084.stdout:2328: expect (ObjNum 752 snap 238 seq_num 752) 2024-09-20T22:51:13.380 INFO:tasks.rados.rados.0.smithi084.stdout:2329: copy_from oid 45 from oid 30 current snap is 259 2024-09-20T22:51:13.380 INFO:tasks.rados.rados.0.smithi084.stdout:2330: rollback oid 8 current snap is 259 2024-09-20T22:51:13.380 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 8 to 213 2024-09-20T22:51:13.380 INFO:tasks.rados.rados.0.smithi084.stdout:2331: read oid 10 snap -1 2024-09-20T22:51:13.380 INFO:tasks.rados.rados.0.smithi084.stdout:2331: expect deleted 2024-09-20T22:51:13.380 INFO:tasks.rados.rados.0.smithi084.stdout:2332: write (excl) oid 11 current snap is 259 2024-09-20T22:51:13.380 INFO:tasks.rados.rados.0.smithi084.stdout:2332: seq_num 803 ranges {696021=642012,1856570=575278,3185369=310652} 2024-09-20T22:51:13.386 INFO:tasks.rados.rados.0.smithi084.stdout:2332: writing smithi08477203-11 from 696021 to 1338033 tid 1 2024-09-20T22:51:13.393 INFO:tasks.rados.rados.0.smithi084.stdout:2332: writing smithi08477203-11 from 1856570 to 2431848 tid 2 2024-09-20T22:51:13.400 INFO:tasks.rados.rados.0.smithi084.stdout:2332: writing smithi08477203-11 from 3185369 to 3496021 tid 3 2024-09-20T22:51:13.400 INFO:tasks.rados.rados.0.smithi084.stdout:2333: copy_from oid 44 from oid 42 current snap is 259 2024-09-20T22:51:13.400 INFO:tasks.rados.rados.0.smithi084.stdout:2328: expect (ObjNum 752 snap 238 seq_num 752) 2024-09-20T22:51:13.407 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:13.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:13.408 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:51:13.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:13 smithi154 ceph-mon[33264]: 6.5 scrub starts 2024-09-20T22:51:13.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:13 smithi154 ceph-mon[33264]: 6.5 scrub ok 2024-09-20T22:51:13.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:13 smithi154 ceph-mon[33264]: pgmap v789: 158 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 153 active+clean; 530 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 19 MiB/s rd, 5.0 MiB/s wr, 34 op/s 2024-09-20T22:51:13.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:13 smithi154 ceph-mon[33264]: 3.6 scrub starts 2024-09-20T22:51:13.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:13 smithi154 ceph-mon[33264]: 3.6 scrub ok 2024-09-20T22:51:13.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:13 smithi154 ceph-mon[33264]: osdmap e592: 8 total, 8 up, 8 in 2024-09-20T22:51:13.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:13 smithi154 ceph-mon[33264]: 3.1e scrub starts 2024-09-20T22:51:13.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:13 smithi154 ceph-mon[33264]: 3.1e scrub ok 2024-09-20T22:51:13.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:13 smithi154 ceph-mon[33264]: 1.5 scrub starts 2024-09-20T22:51:13.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:13 smithi154 ceph-mon[33264]: 1.5 scrub ok 2024-09-20T22:51:13.509 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:51:13.531 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:13.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:13.532 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:51:13.554 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:13 smithi084 ceph-mon[40024]: 6.5 scrub starts 2024-09-20T22:51:13.554 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:13 smithi084 ceph-mon[40024]: 6.5 scrub ok 2024-09-20T22:51:13.554 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:13 smithi084 ceph-mon[40024]: pgmap v789: 158 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 153 active+clean; 530 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 19 MiB/s rd, 5.0 MiB/s wr, 34 op/s 2024-09-20T22:51:13.554 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:13 smithi084 ceph-mon[40024]: 3.6 scrub starts 2024-09-20T22:51:13.554 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:13 smithi084 ceph-mon[40024]: 3.6 scrub ok 2024-09-20T22:51:13.554 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:13 smithi084 ceph-mon[40024]: osdmap e592: 8 total, 8 up, 8 in 2024-09-20T22:51:13.554 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:13 smithi084 ceph-mon[40024]: 3.1e scrub starts 2024-09-20T22:51:13.554 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:13 smithi084 ceph-mon[40024]: 3.1e scrub ok 2024-09-20T22:51:13.555 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:13 smithi084 ceph-mon[40024]: 1.5 scrub starts 2024-09-20T22:51:13.555 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:13 smithi084 ceph-mon[40024]: 1.5 scrub ok 2024-09-20T22:51:13.555 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:13 smithi084 ceph-mon[29774]: 6.5 scrub starts 2024-09-20T22:51:13.555 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:13 smithi084 ceph-mon[29774]: 6.5 scrub ok 2024-09-20T22:51:13.555 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:13 smithi084 ceph-mon[29774]: pgmap v789: 158 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 153 active+clean; 530 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 19 MiB/s rd, 5.0 MiB/s wr, 34 op/s 2024-09-20T22:51:13.555 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:13 smithi084 ceph-mon[29774]: 3.6 scrub starts 2024-09-20T22:51:13.555 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:13 smithi084 ceph-mon[29774]: 3.6 scrub ok 2024-09-20T22:51:13.555 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:13 smithi084 ceph-mon[29774]: osdmap e592: 8 total, 8 up, 8 in 2024-09-20T22:51:13.555 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:13 smithi084 ceph-mon[29774]: 3.1e scrub starts 2024-09-20T22:51:13.555 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:13 smithi084 ceph-mon[29774]: 3.1e scrub ok 2024-09-20T22:51:13.555 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:13 smithi084 ceph-mon[29774]: 1.5 scrub starts 2024-09-20T22:51:13.556 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:13 smithi084 ceph-mon[29774]: 1.5 scrub ok 2024-09-20T22:51:13.625 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-20T22:51:13.625 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:51:13.633 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:51:13.634 INFO:tasks.rados.rados.0.smithi084.stdout:2330: finishing rollback tid 0 to smithi08477203-8 2024-09-20T22:51:13.634 INFO:tasks.rados.rados.0.smithi084.stdout:2330: finishing rollback tid 1 to smithi08477203-8 2024-09-20T22:51:13.635 INFO:tasks.rados.rados.0.smithi084.stdout:2330: finishing rollback tid 2 to smithi08477203-8 2024-09-20T22:51:13.635 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 8 v 846 (ObjNum 625 snap 186 seq_num 625) dirty exists 2024-09-20T22:51:13.635 INFO:tasks.rados.rados.0.smithi084.stdout:2329: finishing copy_from racing read to smithi08477203-45 2024-09-20T22:51:13.635 INFO:tasks.rados.rados.0.smithi084.stdout:2332: finishing write tid 1 to smithi08477203-11 2024-09-20T22:51:13.635 INFO:tasks.rados.rados.0.smithi084.stdout:2328: done (5 left) 2024-09-20T22:51:13.635 INFO:tasks.rados.rados.0.smithi084.stdout:2330: done (4 left) 2024-09-20T22:51:13.635 INFO:tasks.rados.rados.0.smithi084.stdout:2331: done (3 left) 2024-09-20T22:51:13.635 INFO:tasks.rados.rados.0.smithi084.stdout:2334: snap_create 2024-09-20T22:51:13.635 INFO:tasks.rados.rados.0.smithi084.stdout:2332: finishing write tid 2 to smithi08477203-11 2024-09-20T22:51:13.635 INFO:tasks.rados.rados.0.smithi084.stdout:2332: finishing write tid 3 to smithi08477203-11 2024-09-20T22:51:13.635 INFO:tasks.rados.rados.0.smithi084.stdout:2332: finishing write tid 4 to smithi08477203-11 2024-09-20T22:51:13.635 INFO:tasks.rados.rados.0.smithi084.stdout:2332: finishing write tid 5 to smithi08477203-11 2024-09-20T22:51:13.636 INFO:tasks.rados.rados.0.smithi084.stdout:2332: oid 11 updating version 0 to 746 2024-09-20T22:51:13.636 INFO:tasks.rados.rados.0.smithi084.stdout:2332: oid 11 updating version 746 to 748 2024-09-20T22:51:13.636 INFO:tasks.rados.rados.0.smithi084.stdout:2332: oid 11 version 748 is already newer than 747 2024-09-20T22:51:13.636 INFO:tasks.rados.rados.0.smithi084.stdout:2332: oid 11 updating version 748 to 749 2024-09-20T22:51:13.636 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 11 v 749 (ObjNum 803 snap 259 seq_num 803) dirty exists 2024-09-20T22:51:13.636 INFO:tasks.rados.rados.0.smithi084.stdout:2332: left oid 11 (ObjNum 803 snap 259 seq_num 803) 2024-09-20T22:51:13.636 INFO:tasks.rados.rados.0.smithi084.stdout:2333: finishing copy_from racing read to smithi08477203-44 2024-09-20T22:51:13.636 INFO:tasks.rados.rados.0.smithi084.stdout:2333: finishing copy_from to smithi08477203-44 2024-09-20T22:51:13.636 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 44 v 1213 (ObjNum 775 snap 248 seq_num 775) dirty exists 2024-09-20T22:51:13.636 INFO:tasks.rados.rados.0.smithi084.stdout:2329: finishing copy_from to smithi08477203-45 2024-09-20T22:51:13.636 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 45 v 801 (ObjNum 750 snap 238 seq_num 750) dirty exists 2024-09-20T22:51:13.657 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:13.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:13.658 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:51:13.759 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:51:13.781 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:13.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:13.782 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:51:13.883 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:51:13.904 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:13.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:13.905 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:51:14.006 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:51:14.027 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:14.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:14.028 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:51:14.066 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-20T22:51:14.129 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:51:14.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:14.153 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:14.153 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:51:14.239 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.5 dump_blocked_ops 2024-09-20T22:51:14.253 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:51:14.303 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:14.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:14.304 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:51:14.314 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:51:14.377 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is set 2024-09-20T22:51:14.383 INFO:tasks.rados.rados.0.smithi084.stdout:2329: done (3 left) 2024-09-20T22:51:14.383 INFO:tasks.rados.rados.0.smithi084.stdout:2332: done (2 left) 2024-09-20T22:51:14.383 INFO:tasks.rados.rados.0.smithi084.stdout:2333: done (1 left) 2024-09-20T22:51:14.383 INFO:tasks.rados.rados.0.smithi084.stdout:2334: done (0 left) 2024-09-20T22:51:14.383 INFO:tasks.rados.rados.0.smithi084.stdout:2335: snap_create 2024-09-20T22:51:14.405 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:51:14.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:14.428 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:14.428 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:51:14.493 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:51:14.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:14 smithi154 ceph-mon[33264]: 3.11 scrub starts 2024-09-20T22:51:14.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:14 smithi154 ceph-mon[33264]: 3.11 scrub ok 2024-09-20T22:51:14.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:14 smithi154 ceph-mon[33264]: 3.c scrub starts 2024-09-20T22:51:14.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:14 smithi154 ceph-mon[33264]: 3.c scrub ok 2024-09-20T22:51:14.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:14 smithi154 ceph-mon[33264]: osdmap e593: 8 total, 8 up, 8 in 2024-09-20T22:51:14.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:14 smithi154 ceph-mon[33264]: 6.1f scrub starts 2024-09-20T22:51:14.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:14 smithi154 ceph-mon[33264]: 6.1f scrub ok 2024-09-20T22:51:14.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:14 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/2788906473' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-20T22:51:14.529 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:51:14.551 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:14.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:14.552 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:51:14.653 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:51:14.674 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:14.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:14.675 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:51:14.776 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:51:14.797 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:14.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:14.798 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:51:14.852 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:14 smithi084 ceph-mon[40024]: 3.11 scrub starts 2024-09-20T22:51:14.852 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:14 smithi084 ceph-mon[40024]: 3.11 scrub ok 2024-09-20T22:51:14.852 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:14 smithi084 ceph-mon[40024]: 3.c scrub starts 2024-09-20T22:51:14.852 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:14 smithi084 ceph-mon[40024]: 3.c scrub ok 2024-09-20T22:51:14.852 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:14 smithi084 ceph-mon[40024]: osdmap e593: 8 total, 8 up, 8 in 2024-09-20T22:51:14.852 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:14 smithi084 ceph-mon[40024]: 6.1f scrub starts 2024-09-20T22:51:14.852 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:14 smithi084 ceph-mon[40024]: 6.1f scrub ok 2024-09-20T22:51:14.852 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:14 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/2788906473' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-20T22:51:14.852 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:14 smithi084 ceph-mon[29774]: 3.11 scrub starts 2024-09-20T22:51:14.853 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:14 smithi084 ceph-mon[29774]: 3.11 scrub ok 2024-09-20T22:51:14.853 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:14 smithi084 ceph-mon[29774]: 3.c scrub starts 2024-09-20T22:51:14.853 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:14 smithi084 ceph-mon[29774]: 3.c scrub ok 2024-09-20T22:51:14.853 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:14 smithi084 ceph-mon[29774]: osdmap e593: 8 total, 8 up, 8 in 2024-09-20T22:51:14.853 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:14 smithi084 ceph-mon[29774]: 6.1f scrub starts 2024-09-20T22:51:14.853 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:14 smithi084 ceph-mon[29774]: 6.1f scrub ok 2024-09-20T22:51:14.853 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:14 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/2788906473' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-20T22:51:14.899 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:51:14.920 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:14.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:14.921 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:51:14.991 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-20T22:51:14.991 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:51:15.021 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:51:15.044 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:15.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:15.045 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:51:15.146 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:51:15.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:15.168 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:15.169 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:51:15.269 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:51:15.290 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:15.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:15.292 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:51:15.382 INFO:tasks.rados.rados.0.smithi084.stdout:2335: done (0 left) 2024-09-20T22:51:15.383 INFO:tasks.rados.rados.0.smithi084.stdout:2336: read oid 42 snap -1 2024-09-20T22:51:15.383 INFO:tasks.rados.rados.0.smithi084.stdout:2336: expect (ObjNum 775 snap 248 seq_num 775) 2024-09-20T22:51:15.383 INFO:tasks.rados.rados.0.smithi084.stdout:2337: read oid 12 snap -1 2024-09-20T22:51:15.383 INFO:tasks.rados.rados.0.smithi084.stdout:2337: expect (ObjNum 795 snap 254 seq_num 795) 2024-09-20T22:51:15.383 INFO:tasks.rados.rados.0.smithi084.stdout:2338: read oid 14 snap 254 2024-09-20T22:51:15.383 INFO:tasks.rados.rados.0.smithi084.stdout:2338: expect deleted 2024-09-20T22:51:15.383 INFO:tasks.rados.rados.0.smithi084.stdout:2339: copy_from oid 30 from oid 35 current snap is 261 2024-09-20T22:51:15.383 INFO:tasks.rados.rados.0.smithi084.stdout:2340: write (excl) oid 2 current snap is 261 2024-09-20T22:51:15.383 INFO:tasks.rados.rados.0.smithi084.stdout:2340: seq_num 804 ranges {712828=717261,1915052=718936,3212792=300036} 2024-09-20T22:51:15.390 INFO:tasks.rados.rados.0.smithi084.stdout:2340: writing smithi08477203-2 from 712828 to 1430089 tid 1 2024-09-20T22:51:15.393 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:51:15.398 INFO:tasks.rados.rados.0.smithi084.stdout:2340: writing smithi08477203-2 from 1915052 to 2633988 tid 2 2024-09-20T22:51:15.403 INFO:tasks.rados.rados.0.smithi084.stdout:2340: writing smithi08477203-2 from 3212792 to 3512828 tid 3 2024-09-20T22:51:15.403 INFO:tasks.rados.rados.0.smithi084.stdout:2341: write (excl) oid 16 current snap is 261 2024-09-20T22:51:15.403 INFO:tasks.rados.rados.0.smithi084.stdout:2341: seq_num 805 ranges {729635=792510,1973534=578947,3240215=289420} 2024-09-20T22:51:15.411 INFO:tasks.rados.rados.0.smithi084.stdout:2341: writing smithi08477203-16 from 729635 to 1522145 tid 1 2024-09-20T22:51:15.415 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:15.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:15.416 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:51:15.417 INFO:tasks.rados.rados.0.smithi084.stdout:2341: writing smithi08477203-16 from 1973534 to 2552481 tid 2 2024-09-20T22:51:15.423 INFO:tasks.rados.rados.0.smithi084.stdout:2341: writing smithi08477203-16 from 3240215 to 3529635 tid 3 2024-09-20T22:51:15.423 INFO:tasks.rados.rados.0.smithi084.stdout:2342: snap_remove snap 165 2024-09-20T22:51:15.516 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:51:15.538 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:15.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:15.538 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:51:15.639 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:51:15.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:15 smithi084 ceph-mon[29774]: pgmap v792: 158 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 153 active+clean; 531 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 2.4 MiB/s rd, 1.7 MiB/s wr, 6 op/s 2024-09-20T22:51:15.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:15 smithi084 ceph-mon[29774]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:51:15.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:15 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/2788906473' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T22:51:15.645 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:15 smithi084 ceph-mon[29774]: osdmap e594: 8 total, 8 up, 8 in 2024-09-20T22:51:15.646 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:15 smithi084 ceph-mon[40024]: pgmap v792: 158 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 153 active+clean; 531 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 2.4 MiB/s rd, 1.7 MiB/s wr, 6 op/s 2024-09-20T22:51:15.646 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:15 smithi084 ceph-mon[40024]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:51:15.646 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:15 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/2788906473' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T22:51:15.646 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:15 smithi084 ceph-mon[40024]: osdmap e594: 8 total, 8 up, 8 in 2024-09-20T22:51:15.662 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:15.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:15.663 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:51:15.687 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:15 smithi154 ceph-mon[33264]: pgmap v792: 158 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 153 active+clean; 531 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 2.4 MiB/s rd, 1.7 MiB/s wr, 6 op/s 2024-09-20T22:51:15.687 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:15 smithi154 ceph-mon[33264]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:51:15.687 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:15 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/2788906473' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T22:51:15.687 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:15 smithi154 ceph-mon[33264]: osdmap e594: 8 total, 8 up, 8 in 2024-09-20T22:51:15.714 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.5 dump_historic_ops 2024-09-20T22:51:15.763 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:51:15.770 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:51:15.770 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:51:15.771 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:51:15.771 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:51:15.776 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:51:15.776 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:51:15.785 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:15.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:15.785 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:51:15.793 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-20T22:51:15.793 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T22:51:15.809 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:51:15.809 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:51:15.827 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:51:15.827 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:51:15.843 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:51:15.843 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:51:15.859 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:51:15.860 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:51:15.886 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:51:15.907 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:15.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:15.908 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:51:16.004 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:51:16.009 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:51:16.030 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:16.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:16.031 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:51:16.132 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:51:16.154 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:16.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:16.155 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:51:16.256 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:51:16.295 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:16.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:16.296 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:51:16.396 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:51:16.405 INFO:tasks.rados.rados.0.smithi084.stdout:2342: done (6 left) 2024-09-20T22:51:16.406 INFO:tasks.rados.rados.0.smithi084.stdout:2343: delete oid 38 current snap is 261 2024-09-20T22:51:16.406 INFO:tasks.rados.rados.0.smithi084.stdout:2336: expect (ObjNum 775 snap 248 seq_num 775) 2024-09-20T22:51:16.425 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:16.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:16.426 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:51:16.527 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:51:16.540 INFO:tasks.rados.rados.0.smithi084.stdout:2339: finishing copy_from racing read to smithi08477203-30 2024-09-20T22:51:16.540 INFO:tasks.rados.rados.0.smithi084.stdout:2337: expect (ObjNum 795 snap 254 seq_num 795) 2024-09-20T22:51:16.548 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:16.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:16.549 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:51:16.550 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:16 smithi084 ceph-mon[40024]: 3.f scrub starts 2024-09-20T22:51:16.551 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:16 smithi084 ceph-mon[40024]: osdmap e595: 8 total, 8 up, 8 in 2024-09-20T22:51:16.551 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:16 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/3480728446' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:51:16.551 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:16 smithi084 ceph-mon[29774]: 3.f scrub starts 2024-09-20T22:51:16.551 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:16 smithi084 ceph-mon[29774]: osdmap e595: 8 total, 8 up, 8 in 2024-09-20T22:51:16.551 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:16 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/3480728446' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:51:16.650 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:51:16.651 INFO:tasks.rados.rados.0.smithi084.stdout:2339: finishing copy_from to smithi08477203-30 2024-09-20T22:51:16.651 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 30 v 745 (ObjNum 787 snap 254 seq_num 787) dirty exists 2024-09-20T22:51:16.651 INFO:tasks.rados.rados.0.smithi084.stdout:2340: finishing write tid 1 to smithi08477203-2 2024-09-20T22:51:16.651 INFO:tasks.rados.rados.0.smithi084.stdout:2340: finishing write tid 2 to smithi08477203-2 2024-09-20T22:51:16.651 INFO:tasks.rados.rados.0.smithi084.stdout:2336: done (6 left) 2024-09-20T22:51:16.651 INFO:tasks.rados.rados.0.smithi084.stdout:2337: done (5 left) 2024-09-20T22:51:16.652 INFO:tasks.rados.rados.0.smithi084.stdout:2338: done (4 left) 2024-09-20T22:51:16.652 INFO:tasks.rados.rados.0.smithi084.stdout:2339: done (3 left) 2024-09-20T22:51:16.652 INFO:tasks.rados.rados.0.smithi084.stdout:2343: done (2 left) 2024-09-20T22:51:16.652 INFO:tasks.rados.rados.0.smithi084.stdout:2344: snap_remove snap 254 2024-09-20T22:51:16.658 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-20T22:51:16.658 INFO:teuthology.orchestra.run.smithi154.stdout: "size": 20, 2024-09-20T22:51:16.658 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 600, 2024-09-20T22:51:16.658 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [ 2024-09-20T22:51:16.658 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.658 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24334.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-09-20T22:51:16.658 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:41:40.156321+0000", 2024-09-20T22:51:16.658 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 576.34244100700005, 2024-09-20T22:51:16.658 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.012506929, 2024-09-20T22:51:16.658 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:51:16.659 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:51:16.659 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:51:16.659 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:51:16.659 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24334", 2024-09-20T22:51:16.659 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/343304630", 2024-09-20T22:51:16.659 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 7 2024-09-20T22:51:16.659 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.659 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:51:16.659 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.659 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:51:16.659 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:40.156321+0000", 2024-09-20T22:51:16.659 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:51:16.659 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.660 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.660 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:51:16.660 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:40.156321+0000", 2024-09-20T22:51:16.660 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:51:16.660 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.660 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.660 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:51:16.660 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:40.156323+0000", 2024-09-20T22:51:16.660 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9700000000000002e-06 2024-09-20T22:51:16.660 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.660 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.661 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:51:16.661 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:40.156330+0000", 2024-09-20T22:51:16.661 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.9809999999999997e-06 2024-09-20T22:51:16.661 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.661 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.661 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:51:16.661 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:40.156348+0000", 2024-09-20T22:51:16.661 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.7938e-05 2024-09-20T22:51:16.661 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.661 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.661 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:51:16.661 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:40.156362+0000", 2024-09-20T22:51:16.661 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3641e-05 2024-09-20T22:51:16.661 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.661 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.662 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:51:16.662 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:40.156430+0000", 2024-09-20T22:51:16.662 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.7543e-05 2024-09-20T22:51:16.662 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.662 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.662 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:51:16.662 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:40.156543+0000", 2024-09-20T22:51:16.662 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00011390100000000001 2024-09-20T22:51:16.662 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.662 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.662 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 1,6", 2024-09-20T22:51:16.663 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:40.156712+0000", 2024-09-20T22:51:16.663 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00016825499999999999 2024-09-20T22:51:16.663 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.663 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.663 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:51:16.663 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:40.157640+0000", 2024-09-20T22:51:16.663 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00092863699999999995 2024-09-20T22:51:16.663 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.663 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.663 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:51:16.663 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:40.159647+0000", 2024-09-20T22:51:16.663 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0020068320000000001 2024-09-20T22:51:16.663 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.664 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.664 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:51:16.664 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:40.168687+0000", 2024-09-20T22:51:16.664 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0090397749999999999 2024-09-20T22:51:16.664 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.664 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.664 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:51:16.664 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:40.168768+0000", 2024-09-20T22:51:16.664 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.0819e-05 2024-09-20T22:51:16.664 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.664 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.664 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:51:16.664 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:41:40.168828+0000", 2024-09-20T22:51:16.664 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.0637000000000002e-05 2024-09-20T22:51:16.665 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:51:16.665 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:51:16.665 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:51:16.665 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.665 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.665 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "MOSDPGPush(1.4 93/92 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 77'128, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@77'128, size: 327680, copy_subset: [(0, 327680)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 327680, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-20T22:51:16.665 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:42:50.646393+0000", 2024-09-20T22:51:16.665 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 505.85236956799997, 2024-09-20T22:51:16.665 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.033640915, 2024-09-20T22:51:16.665 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:51:16.665 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:51:16.665 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:51:16.666 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:51:16.666 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.666 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:51:16.666 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:42:50.646393+0000", 2024-09-20T22:51:16.666 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:51:16.666 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.666 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.666 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:51:16.666 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:42:50.646393+0000", 2024-09-20T22:51:16.666 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:51:16.666 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.666 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.666 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:51:16.666 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:42:50.646393+0000", 2024-09-20T22:51:16.667 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.23e-07 2024-09-20T22:51:16.667 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.667 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.667 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:51:16.667 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:42:50.679431+0000", 2024-09-20T22:51:16.667 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.033038349000000002 2024-09-20T22:51:16.667 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.667 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.667 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:51:16.667 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:42:50.679478+0000", 2024-09-20T22:51:16.667 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.6828e-05 2024-09-20T22:51:16.667 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.667 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.667 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:51:16.668 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:42:50.679492+0000", 2024-09-20T22:51:16.668 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4062e-05 2024-09-20T22:51:16.668 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.668 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.668 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:51:16.668 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:42:50.679593+0000", 2024-09-20T22:51:16.668 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000100865 2024-09-20T22:51:16.668 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.668 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.668 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:51:16.668 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:42:50.679604+0000", 2024-09-20T22:51:16.668 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0348e-05 2024-09-20T22:51:16.668 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.668 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.668 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:51:16.669 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:42:50.680034+0000", 2024-09-20T22:51:16.669 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00043023999999999997 2024-09-20T22:51:16.669 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:51:16.669 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:51:16.669 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:51:16.669 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.669 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.669 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:554 7.a e128/94 7:5122292a:::smithi08477203-33:head v 128'32, mlcod=119'28)", 2024-09-20T22:51:16.669 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:27.341250+0000", 2024-09-20T22:51:16.669 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 469.157511984, 2024-09-20T22:51:16.669 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.051661319999999997, 2024-09-20T22:51:16.669 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:51:16.669 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:51:16.669 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:51:16.670 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:51:16.670 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.670 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:51:16.670 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.341250+0000", 2024-09-20T22:51:16.670 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:51:16.670 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.670 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.670 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:51:16.670 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.341250+0000", 2024-09-20T22:51:16.670 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:51:16.671 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.671 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.671 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:51:16.671 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.341251+0000", 2024-09-20T22:51:16.671 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.7299999999999996e-07 2024-09-20T22:51:16.671 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.671 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.671 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:51:16.671 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.390395+0000", 2024-09-20T22:51:16.671 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.049143539999999999 2024-09-20T22:51:16.671 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.671 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.671 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:51:16.672 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.390413+0000", 2024-09-20T22:51:16.672 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.8709999999999999e-05 2024-09-20T22:51:16.672 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.672 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.672 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:51:16.672 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.390423+0000", 2024-09-20T22:51:16.672 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.4660000000000005e-06 2024-09-20T22:51:16.672 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.672 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.672 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:51:16.672 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.390484+0000", 2024-09-20T22:51:16.672 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.0851999999999999e-05 2024-09-20T22:51:16.672 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.672 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.673 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:51:16.673 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.390504+0000", 2024-09-20T22:51:16.673 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9976999999999999e-05 2024-09-20T22:51:16.673 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.673 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.673 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:51:16.673 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.392875+0000", 2024-09-20T22:51:16.673 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0023713710000000002 2024-09-20T22:51:16.673 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.673 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.673 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:51:16.674 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.392912+0000", 2024-09-20T22:51:16.674 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.6831e-05 2024-09-20T22:51:16.674 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:51:16.674 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:51:16.674 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:51:16.674 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.674 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.674 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:786 7.17 e145/139 7:ed54065f:::smithi08477203-12:head v 145'119, mlcod=143'117)", 2024-09-20T22:51:16.674 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:44.345115+0000", 2024-09-20T22:51:16.674 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 452.15364750200001, 2024-09-20T22:51:16.674 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0098654899999999993, 2024-09-20T22:51:16.674 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:51:16.674 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:51:16.674 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:51:16.675 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:51:16.675 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.675 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:51:16.675 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.345115+0000", 2024-09-20T22:51:16.675 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:51:16.675 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.675 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.675 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:51:16.675 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.345115+0000", 2024-09-20T22:51:16.675 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:51:16.675 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.675 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.675 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:51:16.675 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.345116+0000", 2024-09-20T22:51:16.676 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.8299999999999995e-07 2024-09-20T22:51:16.676 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.676 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.676 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:51:16.676 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.347955+0000", 2024-09-20T22:51:16.676 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.002839028 2024-09-20T22:51:16.676 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.676 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.676 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:51:16.676 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.347977+0000", 2024-09-20T22:51:16.676 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.1897999999999999e-05 2024-09-20T22:51:16.676 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.676 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.676 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:51:16.677 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.347992+0000", 2024-09-20T22:51:16.677 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4761e-05 2024-09-20T22:51:16.677 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.677 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.677 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:51:16.677 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.348072+0000", 2024-09-20T22:51:16.677 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.0586000000000004e-05 2024-09-20T22:51:16.677 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.677 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.677 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:51:16.677 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.348097+0000", 2024-09-20T22:51:16.677 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.5026000000000001e-05 2024-09-20T22:51:16.677 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.677 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.677 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:51:16.678 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.354924+0000", 2024-09-20T22:51:16.678 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0068271240000000004 2024-09-20T22:51:16.678 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.678 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.678 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:51:16.678 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.354980+0000", 2024-09-20T22:51:16.678 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.6184000000000003e-05 2024-09-20T22:51:16.678 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:51:16.678 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:51:16.678 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:51:16.678 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.678 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.678 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:894 7.a e155/94 7:50097c2d:::smithi08477203-23:head v 155'54, mlcod=143'50)", 2024-09-20T22:51:16.678 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:54.450745+0000", 2024-09-20T22:51:16.679 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 442.04801741400001, 2024-09-20T22:51:16.679 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0091901029999999998, 2024-09-20T22:51:16.679 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:51:16.679 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:51:16.679 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:51:16.679 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:51:16.679 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.679 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:51:16.679 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:54.450745+0000", 2024-09-20T22:51:16.679 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:51:16.679 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.680 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.680 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:51:16.680 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:54.450745+0000", 2024-09-20T22:51:16.680 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:51:16.680 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.680 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.680 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:51:16.680 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:54.450745+0000", 2024-09-20T22:51:16.680 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.4200000000000002e-07 2024-09-20T22:51:16.680 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.680 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.680 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:51:16.680 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:54.456336+0000", 2024-09-20T22:51:16.681 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.005590589 2024-09-20T22:51:16.681 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.681 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.681 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:51:16.681 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:54.456353+0000", 2024-09-20T22:51:16.681 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.7187e-05 2024-09-20T22:51:16.681 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.681 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.681 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:51:16.681 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:54.456362+0000", 2024-09-20T22:51:16.681 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.7409999999999998e-06 2024-09-20T22:51:16.681 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.682 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.682 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:51:16.682 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:54.456418+0000", 2024-09-20T22:51:16.682 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.6431000000000002e-05 2024-09-20T22:51:16.682 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.682 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.682 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:51:16.682 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:54.456438+0000", 2024-09-20T22:51:16.682 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9718999999999999e-05 2024-09-20T22:51:16.682 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.682 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.682 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:51:16.682 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:54.459899+0000", 2024-09-20T22:51:16.683 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.003461254 2024-09-20T22:51:16.683 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.683 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.683 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:51:16.683 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:54.459935+0000", 2024-09-20T22:51:16.683 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.5939999999999998e-05 2024-09-20T22:51:16.683 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:51:16.683 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:51:16.683 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:51:16.683 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.683 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.683 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:1527 7.17 e197/139 7:ea720b85:::smithi08477203-26:head v 197'159, mlcod=187'156)", 2024-09-20T22:51:16.683 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:44:36.209560+0000", 2024-09-20T22:51:16.683 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 400.28920282299998, 2024-09-20T22:51:16.683 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.010263069999999999, 2024-09-20T22:51:16.684 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:51:16.684 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:51:16.684 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:51:16.684 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:51:16.684 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.684 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:51:16.684 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.209560+0000", 2024-09-20T22:51:16.684 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:51:16.684 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.684 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.684 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:51:16.684 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.209560+0000", 2024-09-20T22:51:16.684 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:51:16.684 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.684 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.685 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:51:16.685 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.209560+0000", 2024-09-20T22:51:16.685 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.0800000000000001e-07 2024-09-20T22:51:16.685 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.685 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.685 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:51:16.685 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.212844+0000", 2024-09-20T22:51:16.685 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.003283907 2024-09-20T22:51:16.685 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.685 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.685 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:51:16.685 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.212867+0000", 2024-09-20T22:51:16.685 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.3229000000000001e-05 2024-09-20T22:51:16.685 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.686 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.686 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:51:16.686 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.212878+0000", 2024-09-20T22:51:16.686 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0998e-05 2024-09-20T22:51:16.686 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.686 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.686 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:51:16.686 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.212971+0000", 2024-09-20T22:51:16.686 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.2872000000000001e-05 2024-09-20T22:51:16.686 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.686 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.686 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:51:16.687 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:16 smithi154 ceph-mon[33264]: 3.f scrub starts 2024-09-20T22:51:16.687 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:16 smithi154 ceph-mon[33264]: osdmap e595: 8 total, 8 up, 8 in 2024-09-20T22:51:16.687 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:16 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/3480728446' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:51:16.688 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.212996+0000", 2024-09-20T22:51:16.688 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.4921000000000001e-05 2024-09-20T22:51:16.688 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.688 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.688 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:51:16.688 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.219758+0000", 2024-09-20T22:51:16.688 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0067625139999999999 2024-09-20T22:51:16.688 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.688 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.688 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:51:16.688 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.219823+0000", 2024-09-20T22:51:16.688 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.4420999999999999e-05 2024-09-20T22:51:16.688 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:51:16.689 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:51:16.689 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:51:16.689 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.689 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.689 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:2350 7.0 7:064f66ab:::smithi08477203-45:head [stat,delete] snapc 56=[56,55,53,52,51,50,47,3e] ondisk+write+known_if_redirected+supports_pool_eio e260)", 2024-09-20T22:51:16.689 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:45:39.199566+0000", 2024-09-20T22:51:16.689 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 337.29919639399998, 2024-09-20T22:51:16.689 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.027897148, 2024-09-20T22:51:16.689 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:51:16.689 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:51:16.689 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:51:16.689 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:51:16.689 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:51:16.689 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:51:16.690 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 2350 2024-09-20T22:51:16.690 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.690 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:51:16.690 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.690 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:51:16.690 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.199566+0000", 2024-09-20T22:51:16.690 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:51:16.690 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.690 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.690 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:51:16.690 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.199566+0000", 2024-09-20T22:51:16.690 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:51:16.690 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.691 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.691 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:51:16.691 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.199568+0000", 2024-09-20T22:51:16.691 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9290000000000001e-06 2024-09-20T22:51:16.691 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.691 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.691 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:51:16.691 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.199574+0000", 2024-09-20T22:51:16.691 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.3409999999999999e-06 2024-09-20T22:51:16.691 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.692 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.692 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:51:16.692 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.199584+0000", 2024-09-20T22:51:16.692 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0091e-05 2024-09-20T22:51:16.692 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.692 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.692 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:51:16.692 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.199593+0000", 2024-09-20T22:51:16.692 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.1079999999999993e-06 2024-09-20T22:51:16.692 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.692 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.692 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:51:16.692 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.199661+0000", 2024-09-20T22:51:16.693 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.8767000000000001e-05 2024-09-20T22:51:16.693 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.693 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.693 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:51:16.693 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.199765+0000", 2024-09-20T22:51:16.693 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000103886 2024-09-20T22:51:16.693 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.693 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.693 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:51:16.693 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.227463+0000", 2024-09-20T22:51:16.693 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.027698026000000001 2024-09-20T22:51:16.693 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:51:16.693 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:51:16.693 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:51:16.694 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.694 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.694 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:3134 7.2 7:45b54b6b:::smithi08477203-2:head [copy-from ver 151] snapc 77=[77,76,75,74,73,70,6b] ondisk+write+known_if_redirected+supports_pool_eio e328)", 2024-09-20T22:51:16.694 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:46:48.211103+0000", 2024-09-20T22:51:16.694 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 268.28765988700002, 2024-09-20T22:51:16.694 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.030812519, 2024-09-20T22:51:16.694 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:51:16.694 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:51:16.694 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:51:16.694 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:51:16.694 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:51:16.694 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:51:16.694 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 3134 2024-09-20T22:51:16.694 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.695 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:51:16.695 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.695 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:51:16.695 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.211103+0000", 2024-09-20T22:51:16.695 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:51:16.695 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.695 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.695 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:51:16.695 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.211103+0000", 2024-09-20T22:51:16.695 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:51:16.695 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.695 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.695 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:51:16.695 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.211105+0000", 2024-09-20T22:51:16.696 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.3369999999999998e-06 2024-09-20T22:51:16.696 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.696 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.696 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:51:16.696 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.211111+0000", 2024-09-20T22:51:16.696 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.2759999999999997e-06 2024-09-20T22:51:16.696 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.696 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.696 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:51:16.696 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.211123+0000", 2024-09-20T22:51:16.696 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2119999999999999e-05 2024-09-20T22:51:16.696 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.696 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.697 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:51:16.697 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.211131+0000", 2024-09-20T22:51:16.697 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.2899999999999997e-06 2024-09-20T22:51:16.697 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.697 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.697 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:51:16.697 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.211213+0000", 2024-09-20T22:51:16.697 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.2756999999999994e-05 2024-09-20T22:51:16.697 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.697 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.697 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:51:16.698 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.211310+0000", 2024-09-20T22:51:16.698 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.6644999999999994e-05 2024-09-20T22:51:16.698 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.698 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.698 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 1,2", 2024-09-20T22:51:16.698 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.229109+0000", 2024-09-20T22:51:16.698 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.017798587000000001 2024-09-20T22:51:16.698 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.698 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.698 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:51:16.698 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.240265+0000", 2024-09-20T22:51:16.698 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.011156716000000001 2024-09-20T22:51:16.698 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.699 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.699 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:51:16.699 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.241608+0000", 2024-09-20T22:51:16.699 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001343033 2024-09-20T22:51:16.699 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.699 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.699 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:51:16.699 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.241831+0000", 2024-09-20T22:51:16.699 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000223148 2024-09-20T22:51:16.699 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.699 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.699 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:51:16.699 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.241885+0000", 2024-09-20T22:51:16.700 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.3161000000000001e-05 2024-09-20T22:51:16.700 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.700 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.700 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:51:16.700 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.241915+0000", 2024-09-20T22:51:16.700 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.0448999999999999e-05 2024-09-20T22:51:16.700 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:51:16.700 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:51:16.700 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:51:16.700 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.700 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.700 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:3135 7.2 7:45b54b6b:::smithi08477203-2:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e328)", 2024-09-20T22:51:16.700 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:46:48.211153+0000", 2024-09-20T22:51:16.700 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 268.28760991500002, 2024-09-20T22:51:16.701 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.022731534000000001, 2024-09-20T22:51:16.701 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:51:16.701 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:51:16.701 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:51:16.701 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:51:16.701 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:51:16.701 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:51:16.701 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 3135 2024-09-20T22:51:16.701 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.701 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:51:16.701 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.701 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:51:16.702 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.211153+0000", 2024-09-20T22:51:16.702 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:51:16.702 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.702 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.702 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:51:16.702 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.211153+0000", 2024-09-20T22:51:16.702 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:51:16.702 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.702 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.702 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:51:16.702 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.211153+0000", 2024-09-20T22:51:16.702 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.1600000000000001e-07 2024-09-20T22:51:16.703 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.703 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.703 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:51:16.703 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.211157+0000", 2024-09-20T22:51:16.703 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.5470000000000001e-06 2024-09-20T22:51:16.703 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.703 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.703 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:51:16.703 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.211164+0000", 2024-09-20T22:51:16.704 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.2799999999999998e-06 2024-09-20T22:51:16.704 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.704 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.704 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:51:16.704 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.211170+0000", 2024-09-20T22:51:16.704 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.0730000000000001e-06 2024-09-20T22:51:16.704 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.704 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.704 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:51:16.704 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.211634+0000", 2024-09-20T22:51:16.704 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00046401800000000002 2024-09-20T22:51:16.704 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.704 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.704 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for blocked object", 2024-09-20T22:51:16.705 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.211690+0000", 2024-09-20T22:51:16.705 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.5587999999999997e-05 2024-09-20T22:51:16.705 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.705 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.705 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:51:16.705 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.233669+0000", 2024-09-20T22:51:16.705 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.021979240000000001 2024-09-20T22:51:16.705 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.705 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.705 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:51:16.705 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.233762+0000", 2024-09-20T22:51:16.705 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.2658999999999995e-05 2024-09-20T22:51:16.705 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.705 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.705 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:51:16.706 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.233884+0000", 2024-09-20T22:51:16.706 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00012241300000000001 2024-09-20T22:51:16.706 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:51:16.706 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:51:16.706 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:51:16.706 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.706 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.706 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:3136 7.a e328/94 7:5122292a:::smithi08477203-33:head v 328'252, mlcod=322'248)", 2024-09-20T22:51:16.706 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:46:48.223492+0000", 2024-09-20T22:51:16.706 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 268.27527020399998, 2024-09-20T22:51:16.706 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.016662746999999999, 2024-09-20T22:51:16.706 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:51:16.707 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:51:16.707 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:51:16.707 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:51:16.707 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.707 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:51:16.707 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.223492+0000", 2024-09-20T22:51:16.707 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:51:16.707 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.707 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.707 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:51:16.707 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.223492+0000", 2024-09-20T22:51:16.707 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:51:16.707 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.707 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.707 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:51:16.708 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.223493+0000", 2024-09-20T22:51:16.708 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.4299999999999998e-07 2024-09-20T22:51:16.708 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.708 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.708 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:51:16.708 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.228060+0000", 2024-09-20T22:51:16.708 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0045669350000000003 2024-09-20T22:51:16.708 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.708 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.708 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:51:16.708 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.228107+0000", 2024-09-20T22:51:16.708 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.6932999999999999e-05 2024-09-20T22:51:16.708 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.708 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.708 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:51:16.709 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.228127+0000", 2024-09-20T22:51:16.709 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.0777000000000001e-05 2024-09-20T22:51:16.709 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.709 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.709 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:51:16.709 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.228206+0000", 2024-09-20T22:51:16.709 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.8660999999999998e-05 2024-09-20T22:51:16.709 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.709 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.709 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:51:16.709 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.228232+0000", 2024-09-20T22:51:16.709 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.6199000000000001e-05 2024-09-20T22:51:16.709 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.709 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.710 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:51:16.710 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.240103+0000", 2024-09-20T22:51:16.710 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.011871008000000001 2024-09-20T22:51:16.710 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.710 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.710 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:51:16.710 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.240155+0000", 2024-09-20T22:51:16.710 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.1790999999999999e-05 2024-09-20T22:51:16.710 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:51:16.710 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:51:16.710 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:51:16.710 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.711 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.711 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:3232 7.16 e337/260 7:6885a65f:::smithi08477203-14:head v 337'107, mlcod=331'104)", 2024-09-20T22:51:16.711 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:46:57.231995+0000", 2024-09-20T22:51:16.711 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 259.26676724999999, 2024-09-20T22:51:16.711 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0090885140000000007, 2024-09-20T22:51:16.711 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:51:16.711 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:51:16.711 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:51:16.711 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:51:16.711 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.711 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:51:16.711 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.231995+0000", 2024-09-20T22:51:16.711 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:51:16.711 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.712 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.712 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:51:16.712 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.231995+0000", 2024-09-20T22:51:16.712 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:51:16.712 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.712 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.712 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:51:16.712 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.231996+0000", 2024-09-20T22:51:16.712 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.5799999999999998e-07 2024-09-20T22:51:16.712 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.713 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.713 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:51:16.713 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.234590+0000", 2024-09-20T22:51:16.713 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.002594482 2024-09-20T22:51:16.713 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.713 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.713 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:51:16.713 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.234617+0000", 2024-09-20T22:51:16.713 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.6855e-05 2024-09-20T22:51:16.713 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.713 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.713 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:51:16.713 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.234636+0000", 2024-09-20T22:51:16.713 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.8289000000000001e-05 2024-09-20T22:51:16.713 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.714 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.714 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:51:16.714 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.234729+0000", 2024-09-20T22:51:16.714 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.3207000000000004e-05 2024-09-20T22:51:16.714 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.714 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.714 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:51:16.714 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.234758+0000", 2024-09-20T22:51:16.714 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.8994999999999999e-05 2024-09-20T22:51:16.714 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.714 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.714 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:51:16.714 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.241031+0000", 2024-09-20T22:51:16.714 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0062733279999999999 2024-09-20T22:51:16.715 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.715 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.715 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:51:16.715 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.241084+0000", 2024-09-20T22:51:16.715 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.2599999999999998e-05 2024-09-20T22:51:16.715 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:51:16.715 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:51:16.715 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:51:16.715 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.715 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.715 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:3589 7.b e363/94 7:dc95255e:::smithi08477203-1:head v 363'419, mlcod=363'417)", 2024-09-20T22:51:16.715 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:47:23.205271+0000", 2024-09-20T22:51:16.716 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 233.29349122299999, 2024-09-20T22:51:16.716 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.014301911, 2024-09-20T22:51:16.716 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:51:16.716 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:51:16.716 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:51:16.716 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:51:16.716 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.716 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:51:16.716 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:23.205271+0000", 2024-09-20T22:51:16.716 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:51:16.716 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.716 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.716 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:51:16.716 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:23.205271+0000", 2024-09-20T22:51:16.716 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:51:16.717 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.717 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.717 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:51:16.717 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:23.205273+0000", 2024-09-20T22:51:16.717 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.409e-06 2024-09-20T22:51:16.717 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.717 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.717 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:51:16.717 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:23.210044+0000", 2024-09-20T22:51:16.717 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0047712609999999997 2024-09-20T22:51:16.717 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.717 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.717 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:51:16.717 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:23.210064+0000", 2024-09-20T22:51:16.718 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.0421e-05 2024-09-20T22:51:16.718 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.718 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.718 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:51:16.718 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:23.210074+0000", 2024-09-20T22:51:16.718 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.4329999999999993e-06 2024-09-20T22:51:16.718 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.718 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.718 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:51:16.718 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:23.210130+0000", 2024-09-20T22:51:16.718 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.6286000000000003e-05 2024-09-20T22:51:16.718 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.718 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.719 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:51:16.719 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:23.210148+0000", 2024-09-20T22:51:16.719 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.7895999999999999e-05 2024-09-20T22:51:16.719 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.719 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.719 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:51:16.719 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:23.219532+0000", 2024-09-20T22:51:16.719 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.009383677 2024-09-20T22:51:16.719 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.719 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.719 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:51:16.719 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:23.219573+0000", 2024-09-20T22:51:16.719 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.1527999999999999e-05 2024-09-20T22:51:16.720 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:51:16.720 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:51:16.720 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:51:16.720 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.720 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.720 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:3760 7.13 e383/260 7:cae5792a:::smithi08477203-32:head v 383'78, mlcod=377'74)", 2024-09-20T22:51:16.720 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:47:43.291657+0000", 2024-09-20T22:51:16.720 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 213.207105345, 2024-09-20T22:51:16.720 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.011377195, 2024-09-20T22:51:16.720 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:51:16.720 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:51:16.720 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:51:16.720 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:51:16.721 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.721 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:51:16.721 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.291657+0000", 2024-09-20T22:51:16.721 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:51:16.721 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.721 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.721 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:51:16.721 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.291657+0000", 2024-09-20T22:51:16.721 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:51:16.721 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.721 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.721 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:51:16.721 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.291658+0000", 2024-09-20T22:51:16.721 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1009999999999999e-06 2024-09-20T22:51:16.722 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.722 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.722 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:51:16.722 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.296704+0000", 2024-09-20T22:51:16.722 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0050458070000000002 2024-09-20T22:51:16.722 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.722 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.722 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:51:16.722 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.296716+0000", 2024-09-20T22:51:16.722 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1525000000000001e-05 2024-09-20T22:51:16.722 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.722 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.723 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:51:16.723 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.296721+0000", 2024-09-20T22:51:16.723 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.8379999999999998e-06 2024-09-20T22:51:16.723 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.723 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.723 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:51:16.723 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.296769+0000", 2024-09-20T22:51:16.723 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.7120000000000003e-05 2024-09-20T22:51:16.723 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.723 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.724 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:51:16.724 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.296786+0000", 2024-09-20T22:51:16.724 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.7374e-05 2024-09-20T22:51:16.724 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.724 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.724 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:51:16.724 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.302998+0000", 2024-09-20T22:51:16.724 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0062118219999999997 2024-09-20T22:51:16.724 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.724 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.724 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:51:16.724 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.303034+0000", 2024-09-20T22:51:16.724 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.6607999999999999e-05 2024-09-20T22:51:16.724 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:51:16.725 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:51:16.725 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:51:16.725 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.725 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.725 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:4444 7.0 7:05bb442d:::smithi08477203-24:head [copy-from ver 421] snapc b8=[b8,b7,b6,b5,b4,b3,b2,b1,b0,ae,a9,a7,a6,a5,9e,9c,96,95,87,86] ondisk+write+known_if_redirected+supports_pool_eio e447)", 2024-09-20T22:51:16.725 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:48:47.227411+0000", 2024-09-20T22:51:16.725 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 149.27135177900001, 2024-09-20T22:51:16.725 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.033566952999999997, 2024-09-20T22:51:16.725 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:51:16.725 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:51:16.725 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:51:16.725 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:51:16.725 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:51:16.726 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:51:16.726 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 4444 2024-09-20T22:51:16.726 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.726 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:51:16.726 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.726 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:51:16.726 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.227411+0000", 2024-09-20T22:51:16.726 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:51:16.726 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.726 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.726 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:51:16.727 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.227411+0000", 2024-09-20T22:51:16.727 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:51:16.727 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.727 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.727 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:51:16.727 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.227413+0000", 2024-09-20T22:51:16.727 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.057e-06 2024-09-20T22:51:16.727 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.727 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.727 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:51:16.727 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.227422+0000", 2024-09-20T22:51:16.727 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.3219999999999999e-06 2024-09-20T22:51:16.728 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.728 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.728 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:51:16.728 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.227442+0000", 2024-09-20T22:51:16.728 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9449999999999998e-05 2024-09-20T22:51:16.728 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.728 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.728 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:51:16.728 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.227452+0000", 2024-09-20T22:51:16.728 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0699999999999999e-05 2024-09-20T22:51:16.729 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.729 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.729 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:51:16.729 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.227531+0000", 2024-09-20T22:51:16.729 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.8993999999999996e-05 2024-09-20T22:51:16.729 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.729 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.729 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op must wait for map", 2024-09-20T22:51:16.729 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.227542+0000", 2024-09-20T22:51:16.729 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0797e-05 2024-09-20T22:51:16.729 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.729 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.729 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:51:16.729 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.229501+0000", 2024-09-20T22:51:16.730 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0019594080000000002 2024-09-20T22:51:16.730 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.730 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.730 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:51:16.730 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.229616+0000", 2024-09-20T22:51:16.730 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000114577 2024-09-20T22:51:16.730 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.730 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.730 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 1,3", 2024-09-20T22:51:16.730 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.241752+0000", 2024-09-20T22:51:16.730 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.012135920999999999 2024-09-20T22:51:16.730 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.730 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.731 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:51:16.731 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.249601+0000", 2024-09-20T22:51:16.731 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0078493120000000006 2024-09-20T22:51:16.731 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.731 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.731 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:51:16.731 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.260731+0000", 2024-09-20T22:51:16.731 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.011129781 2024-09-20T22:51:16.731 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.731 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.731 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:51:16.731 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.260861+0000", 2024-09-20T22:51:16.732 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00013027899999999999 2024-09-20T22:51:16.732 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.732 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.732 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:51:16.732 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.260913+0000", 2024-09-20T22:51:16.732 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.2172999999999997e-05 2024-09-20T22:51:16.732 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.732 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.732 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:51:16.732 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.260978+0000", 2024-09-20T22:51:16.732 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.4182000000000003e-05 2024-09-20T22:51:16.732 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:51:16.732 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:51:16.732 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:51:16.733 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.733 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.733 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:4445 7.0 7:05bb442d:::smithi08477203-24:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e447)", 2024-09-20T22:51:16.733 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:48:47.227478+0000", 2024-09-20T22:51:16.733 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 149.271284842, 2024-09-20T22:51:16.733 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.019465758999999999, 2024-09-20T22:51:16.733 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:51:16.733 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:51:16.733 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:51:16.733 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:51:16.733 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:51:16.734 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:51:16.734 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 4445 2024-09-20T22:51:16.734 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.734 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:51:16.734 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.734 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:51:16.734 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.227478+0000", 2024-09-20T22:51:16.734 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:51:16.734 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.734 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.734 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:51:16.734 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.227478+0000", 2024-09-20T22:51:16.734 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:51:16.734 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.735 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.735 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:51:16.735 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.227478+0000", 2024-09-20T22:51:16.735 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.2200000000000004e-07 2024-09-20T22:51:16.735 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.735 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.735 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:51:16.735 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.227481+0000", 2024-09-20T22:51:16.735 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.0060000000000001e-06 2024-09-20T22:51:16.735 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.735 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.735 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:51:16.735 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.227500+0000", 2024-09-20T22:51:16.736 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9071000000000001e-05 2024-09-20T22:51:16.736 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.736 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.736 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:51:16.736 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.227507+0000", 2024-09-20T22:51:16.736 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.1779999999999997e-06 2024-09-20T22:51:16.736 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.736 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.736 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:51:16.736 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.227628+0000", 2024-09-20T22:51:16.736 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000120627 2024-09-20T22:51:16.736 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.736 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.736 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting_for_map not empty", 2024-09-20T22:51:16.737 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.227633+0000", 2024-09-20T22:51:16.737 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.4730000000000002e-06 2024-09-20T22:51:16.737 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.737 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.737 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:51:16.737 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.229766+0000", 2024-09-20T22:51:16.737 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0021332339999999999 2024-09-20T22:51:16.737 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.737 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.737 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for blocked object", 2024-09-20T22:51:16.737 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.229814+0000", 2024-09-20T22:51:16.737 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.7746e-05 2024-09-20T22:51:16.737 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.737 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.738 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:51:16.738 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.246707+0000", 2024-09-20T22:51:16.738 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.016893542000000001 2024-09-20T22:51:16.738 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.738 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.738 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:51:16.738 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.246863+0000", 2024-09-20T22:51:16.738 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000156105 2024-09-20T22:51:16.738 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.738 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.738 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:51:16.739 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.246943+0000", 2024-09-20T22:51:16.739 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.0154999999999994e-05 2024-09-20T22:51:16.739 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:51:16.739 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:51:16.739 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:51:16.739 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.739 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.739 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:5011 7.b e505/94 7:df7b2b73:::smithi08477203-10:head v 505'722, mlcod=503'718)", 2024-09-20T22:51:16.739 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:49:45.305974+0000", 2024-09-20T22:51:16.739 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 91.192788746000005, 2024-09-20T22:51:16.739 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.014321993, 2024-09-20T22:51:16.739 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:51:16.739 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:51:16.740 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:51:16.740 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:51:16.740 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.740 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:51:16.740 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:45.305974+0000", 2024-09-20T22:51:16.740 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:51:16.740 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.740 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.740 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:51:16.740 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:45.305974+0000", 2024-09-20T22:51:16.740 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:51:16.740 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.740 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.741 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:51:16.741 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:45.305974+0000", 2024-09-20T22:51:16.741 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.6199999999999997e-07 2024-09-20T22:51:16.741 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.741 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.741 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:51:16.741 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:45.311756+0000", 2024-09-20T22:51:16.741 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0057811019999999998 2024-09-20T22:51:16.741 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.741 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.741 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:51:16.741 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:45.311773+0000", 2024-09-20T22:51:16.741 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.7757999999999998e-05 2024-09-20T22:51:16.741 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.742 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.742 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:51:16.742 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:45.311783+0000", 2024-09-20T22:51:16.742 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.6639999999999993e-06 2024-09-20T22:51:16.742 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.742 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.742 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:51:16.742 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:45.311873+0000", 2024-09-20T22:51:16.742 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.0062000000000006e-05 2024-09-20T22:51:16.742 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.742 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.742 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:51:16.742 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:45.311900+0000", 2024-09-20T22:51:16.742 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.6475999999999999e-05 2024-09-20T22:51:16.742 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.743 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.743 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:51:16.743 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:45.320251+0000", 2024-09-20T22:51:16.743 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0083511560000000002 2024-09-20T22:51:16.743 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.743 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.743 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:51:16.743 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:45.320296+0000", 2024-09-20T22:51:16.743 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.5012999999999997e-05 2024-09-20T22:51:16.743 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:51:16.743 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:51:16.744 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:51:16.744 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.744 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.744 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:5151 7.0 7:05bb442d:::smithi08477203-24:head [write 749777~722868] snapc dd=[dd,dc,db,da,d9,d8,d7,d6,d5,d4,d3,d0,cf,ce,cd,ca,c7,c6,c5,c2,bf,ba,b4,b2,a7,9e,95,87] ondisk+write+known_if_redirected+supports_pool_eio e514)", 2024-09-20T22:51:16.744 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:49:54.638549+0000", 2024-09-20T22:51:16.744 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 81.860213603999995, 2024-09-20T22:51:16.744 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0092067450000000006, 2024-09-20T22:51:16.744 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:51:16.744 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:51:16.744 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:51:16.744 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:51:16.744 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:51:16.744 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:51:16.744 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 5151 2024-09-20T22:51:16.745 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.745 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:51:16.745 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.745 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:51:16.745 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:54.638549+0000", 2024-09-20T22:51:16.745 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:51:16.745 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.745 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.745 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:51:16.745 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:54.638549+0000", 2024-09-20T22:51:16.745 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:51:16.745 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.745 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.746 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:51:16.746 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:54.638551+0000", 2024-09-20T22:51:16.746 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9709999999999998e-06 2024-09-20T22:51:16.746 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.746 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.746 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:51:16.746 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:54.639792+0000", 2024-09-20T22:51:16.746 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0012410139999999999 2024-09-20T22:51:16.746 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.746 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.746 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:51:16.746 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:54.639813+0000", 2024-09-20T22:51:16.746 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.1148000000000001e-05 2024-09-20T22:51:16.747 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.747 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.747 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:51:16.747 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:54.639835+0000", 2024-09-20T22:51:16.747 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.1905000000000001e-05 2024-09-20T22:51:16.747 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.747 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.747 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:51:16.747 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:54.639901+0000", 2024-09-20T22:51:16.747 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.6548e-05 2024-09-20T22:51:16.747 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.747 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.747 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:51:16.747 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:54.639981+0000", 2024-09-20T22:51:16.747 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.9148999999999997e-05 2024-09-20T22:51:16.748 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.748 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.748 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 1,3", 2024-09-20T22:51:16.748 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:54.640098+0000", 2024-09-20T22:51:16.748 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000117337 2024-09-20T22:51:16.748 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.748 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.748 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:51:16.748 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:54.641401+0000", 2024-09-20T22:51:16.748 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0013027099999999999 2024-09-20T22:51:16.748 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.748 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.748 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:51:16.748 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:54.647561+0000", 2024-09-20T22:51:16.749 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.006160502 2024-09-20T22:51:16.749 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.749 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.749 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:51:16.749 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:54.647676+0000", 2024-09-20T22:51:16.749 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000114671 2024-09-20T22:51:16.749 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.749 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.749 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:51:16.749 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:54.647717+0000", 2024-09-20T22:51:16.749 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.1446000000000003e-05 2024-09-20T22:51:16.749 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.749 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.749 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:51:16.750 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:54.647756+0000", 2024-09-20T22:51:16.750 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.8343999999999997e-05 2024-09-20T22:51:16.750 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:51:16.750 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:51:16.750 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:51:16.750 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.750 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.750 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:5514 7.b e560/94 7:d3ca3619:::smithi08477203-39:head v 560'815, mlcod=559'812)", 2024-09-20T22:51:16.750 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:50:40.289035+0000", 2024-09-20T22:51:16.750 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 36.209727485999998, 2024-09-20T22:51:16.750 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0098817409999999994, 2024-09-20T22:51:16.750 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:51:16.750 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:51:16.751 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:51:16.751 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:51:16.751 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.751 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:51:16.751 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:40.289035+0000", 2024-09-20T22:51:16.751 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:51:16.751 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.751 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.751 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:51:16.751 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:40.289035+0000", 2024-09-20T22:51:16.751 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:51:16.751 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.751 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.751 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:51:16.751 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:40.289035+0000", 2024-09-20T22:51:16.752 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.8699999999999999e-07 2024-09-20T22:51:16.752 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.752 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.752 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:51:16.752 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:40.293026+0000", 2024-09-20T22:51:16.752 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.003991334 2024-09-20T22:51:16.752 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.752 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.752 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:51:16.752 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:40.293096+0000", 2024-09-20T22:51:16.752 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.9764999999999994e-05 2024-09-20T22:51:16.752 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.752 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.752 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:51:16.753 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:40.293107+0000", 2024-09-20T22:51:16.753 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.059e-05 2024-09-20T22:51:16.753 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.753 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.753 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:51:16.753 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:40.293195+0000", 2024-09-20T22:51:16.753 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.8052000000000003e-05 2024-09-20T22:51:16.753 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.753 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.753 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:51:16.753 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:40.293226+0000", 2024-09-20T22:51:16.753 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.1557999999999998e-05 2024-09-20T22:51:16.753 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.754 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.754 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:51:16.754 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:40.298884+0000", 2024-09-20T22:51:16.754 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0056575139999999998 2024-09-20T22:51:16.754 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.754 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.754 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:51:16.754 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:40.298917+0000", 2024-09-20T22:51:16.754 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.2740999999999997e-05 2024-09-20T22:51:16.755 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:51:16.755 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:51:16.755 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:51:16.755 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.755 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.755 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:5532 7.a e561/94 7:5aaef466:::smithi08477203-40:head v 561'565, mlcod=559'561)", 2024-09-20T22:51:16.755 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:50:41.456088+0000", 2024-09-20T22:51:16.755 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 35.042674357000003, 2024-09-20T22:51:16.755 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.009842557, 2024-09-20T22:51:16.755 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:51:16.755 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:51:16.755 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:51:16.755 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:51:16.755 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.756 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:51:16.756 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:41.456088+0000", 2024-09-20T22:51:16.756 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:51:16.756 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.756 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.756 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:51:16.756 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:41.456088+0000", 2024-09-20T22:51:16.756 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:51:16.756 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.756 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.756 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:51:16.756 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:41.456090+0000", 2024-09-20T22:51:16.756 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.916e-06 2024-09-20T22:51:16.757 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.757 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.757 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:51:16.757 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:41.459913+0000", 2024-09-20T22:51:16.757 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0038229879999999998 2024-09-20T22:51:16.757 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.757 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.757 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:51:16.757 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:41.459929+0000", 2024-09-20T22:51:16.757 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.6249e-05 2024-09-20T22:51:16.757 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.757 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.757 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:51:16.757 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:41.459938+0000", 2024-09-20T22:51:16.758 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.7390000000000005e-06 2024-09-20T22:51:16.758 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.758 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.758 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:51:16.758 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:41.460008+0000", 2024-09-20T22:51:16.758 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.9543999999999997e-05 2024-09-20T22:51:16.758 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.758 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.758 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:51:16.758 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:41.460034+0000", 2024-09-20T22:51:16.758 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.6341000000000001e-05 2024-09-20T22:51:16.758 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.758 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.759 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:51:16.759 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:41.465878+0000", 2024-09-20T22:51:16.759 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00584418 2024-09-20T22:51:16.759 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.759 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.759 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:51:16.759 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:41.465931+0000", 2024-09-20T22:51:16.759 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.2599999999999998e-05 2024-09-20T22:51:16.759 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:51:16.759 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:51:16.759 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:51:16.759 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.759 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.759 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:5659 7.b e570/94 7:dc95255e:::smithi08477203-1:head v 570'831, mlcod=570'827)", 2024-09-20T22:51:16.760 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:50:51.285735+0000", 2024-09-20T22:51:16.760 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 25.213027373999999, 2024-09-20T22:51:16.760 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.016481638999999999, 2024-09-20T22:51:16.760 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:51:16.760 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:51:16.760 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:51:16.760 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:51:16.760 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.760 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:51:16.760 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:51.285735+0000", 2024-09-20T22:51:16.760 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:51:16.760 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.760 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.760 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:51:16.760 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:51.285735+0000", 2024-09-20T22:51:16.761 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:51:16.761 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.761 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.761 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:51:16.761 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:51.285736+0000", 2024-09-20T22:51:16.761 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.5099999999999998e-07 2024-09-20T22:51:16.761 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.761 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.761 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:51:16.761 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:51.293158+0000", 2024-09-20T22:51:16.761 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0074220750000000002 2024-09-20T22:51:16.761 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.761 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.761 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:51:16.761 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:51.293175+0000", 2024-09-20T22:51:16.762 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.6401000000000001e-05 2024-09-20T22:51:16.762 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.762 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.762 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:51:16.762 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:51.293183+0000", 2024-09-20T22:51:16.762 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.8100000000000004e-06 2024-09-20T22:51:16.762 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.762 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.762 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:51:16.762 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:51.293262+0000", 2024-09-20T22:51:16.762 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.8527000000000002e-05 2024-09-20T22:51:16.762 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.762 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.762 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:51:16.763 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:51.293289+0000", 2024-09-20T22:51:16.763 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.7620999999999999e-05 2024-09-20T22:51:16.763 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.763 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.763 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:51:16.763 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:51.302173+0000", 2024-09-20T22:51:16.763 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0088831139999999992 2024-09-20T22:51:16.763 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:16.763 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:16.763 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:51:16.763 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:51.302217+0000", 2024-09-20T22:51:16.763 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.4140000000000001e-05 2024-09-20T22:51:16.763 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:51:16.763 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:51:16.763 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:51:16.764 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:51:16.764 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:51:16.764 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-20T22:51:16.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:16.765 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:16.765 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:51:16.865 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:51:16.887 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:16.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:16.888 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:51:16.989 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:51:17.010 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:17.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:17.011 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:51:17.112 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:51:17.133 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:17.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:17.134 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:51:17.235 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:51:17.256 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:17.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:17.257 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:51:17.358 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:51:17.379 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:17.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:17.380 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:51:17.399 INFO:tasks.rados.rados.0.smithi084.stdout:2344: done (2 left) 2024-09-20T22:51:17.399 INFO:tasks.rados.rados.0.smithi084.stdout:2345: read oid 20 snap -1 2024-09-20T22:51:17.399 INFO:tasks.rados.rados.0.smithi084.stdout:2345: expect (ObjNum 791 snap 254 seq_num 791) 2024-09-20T22:51:17.399 INFO:tasks.rados.rados.0.smithi084.stdout:2346: snap_remove snap 260 2024-09-20T22:51:17.446 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-20T22:51:17.458 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-20T22:51:17.480 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:51:17.521 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:17.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:17.522 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:51:17.623 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:51:17.645 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:17.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:17.646 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:51:17.693 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:51:17.705 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:51:17.747 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:51:17.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:17 smithi154 ceph-mon[33264]: pgmap v795: 158 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 153 active+clean; 531 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 2.4 MiB/s rd, 1.7 MiB/s wr, 6 op/s 2024-09-20T22:51:17.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:17 smithi154 ceph-mon[33264]: osdmap e596: 8 total, 8 up, 8 in 2024-09-20T22:51:17.768 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:17.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:17.769 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:51:17.814 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:17 smithi084 ceph-mon[40024]: pgmap v795: 158 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 153 active+clean; 531 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 2.4 MiB/s rd, 1.7 MiB/s wr, 6 op/s 2024-09-20T22:51:17.814 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:17 smithi084 ceph-mon[40024]: osdmap e596: 8 total, 8 up, 8 in 2024-09-20T22:51:17.814 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:17 smithi084 ceph-mon[29774]: pgmap v795: 158 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 153 active+clean; 531 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 2.4 MiB/s rd, 1.7 MiB/s wr, 6 op/s 2024-09-20T22:51:17.814 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:17 smithi084 ceph-mon[29774]: osdmap e596: 8 total, 8 up, 8 in 2024-09-20T22:51:17.870 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:51:17.892 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:17.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:17.893 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:51:17.969 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 5, 6, 7, 0, 4] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-20T22:51:17.969 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-20T22:51:17.970 INFO:tasks.thrashosds.thrasher:Removing osd 1, in_osds are: [1, 2, 3, 5, 6, 7, 0, 4] 2024-09-20T22:51:17.970 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd out 1 2024-09-20T22:51:17.994 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:51:18.015 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:18.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:18.016 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:51:18.117 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:51:18.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:18.140 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:18.140 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:51:18.201 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-20T22:51:18.201 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [], 2024-09-20T22:51:18.201 INFO:teuthology.orchestra.run.smithi154.stdout: "num_ops": 0 2024-09-20T22:51:18.201 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-20T22:51:18.241 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:51:18.260 INFO:tasks.rados.rados.0.smithi084.stdout:2346: done (3 left) 2024-09-20T22:51:18.261 INFO:tasks.rados.rados.0.smithi084.stdout:2347: snap_create 2024-09-20T22:51:18.261 INFO:tasks.rados.rados.0.smithi084.stdout:2340: finishing write tid 3 to smithi08477203-2 2024-09-20T22:51:18.261 INFO:tasks.rados.rados.0.smithi084.stdout:2340: finishing write tid 4 to smithi08477203-2 2024-09-20T22:51:18.261 INFO:tasks.rados.rados.0.smithi084.stdout:2340: finishing write tid 5 to smithi08477203-2 2024-09-20T22:51:18.261 INFO:tasks.rados.rados.0.smithi084.stdout:2340: oid 2 updating version 0 to 1221 2024-09-20T22:51:18.261 INFO:tasks.rados.rados.0.smithi084.stdout:2340: oid 2 version 1221 is already newer than 1218 2024-09-20T22:51:18.261 INFO:tasks.rados.rados.0.smithi084.stdout:2340: oid 2 version 1221 is already newer than 1220 2024-09-20T22:51:18.261 INFO:tasks.rados.rados.0.smithi084.stdout:2340: oid 2 version 1221 is already newer than 1219 2024-09-20T22:51:18.261 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 2 v 1221 (ObjNum 804 snap 261 seq_num 804) dirty exists 2024-09-20T22:51:18.261 INFO:tasks.rados.rados.0.smithi084.stdout:2340: left oid 2 (ObjNum 804 snap 261 seq_num 804) 2024-09-20T22:51:18.261 INFO:tasks.rados.rados.0.smithi084.stdout:2341: finishing write tid 1 to smithi08477203-16 2024-09-20T22:51:18.261 INFO:tasks.rados.rados.0.smithi084.stdout:2341: finishing write tid 2 to smithi08477203-16 2024-09-20T22:51:18.261 INFO:tasks.rados.rados.0.smithi084.stdout:2341: finishing write tid 3 to smithi08477203-16 2024-09-20T22:51:18.262 INFO:tasks.rados.rados.0.smithi084.stdout:2341: finishing write tid 4 to smithi08477203-16 2024-09-20T22:51:18.262 INFO:tasks.rados.rados.0.smithi084.stdout:2341: finishing write tid 5 to smithi08477203-16 2024-09-20T22:51:18.262 INFO:tasks.rados.rados.0.smithi084.stdout:2341: oid 16 updating version 0 to 439 2024-09-20T22:51:18.262 INFO:tasks.rados.rados.0.smithi084.stdout:2341: oid 16 version 439 is already newer than 438 2024-09-20T22:51:18.262 INFO:tasks.rados.rados.0.smithi084.stdout:2341: oid 16 version 439 is already newer than 436 2024-09-20T22:51:18.262 INFO:tasks.rados.rados.0.smithi084.stdout:2341: oid 16 version 439 is already newer than 437 2024-09-20T22:51:18.262 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 16 v 439 (ObjNum 805 snap 261 seq_num 805) dirty exists 2024-09-20T22:51:18.262 INFO:tasks.rados.rados.0.smithi084.stdout:2341: left oid 16 (ObjNum 805 snap 261 seq_num 805) 2024-09-20T22:51:18.262 INFO:tasks.rados.rados.0.smithi084.stdout:2345: expect (ObjNum 791 snap 254 seq_num 791) 2024-09-20T22:51:18.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:18.268 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:18.268 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:51:18.322 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:51:18.370 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:51:18.391 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:18.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:18.392 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:51:18.401 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:51:18 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: ::ffff:172.21.15.154 - - [20/Sep/2024:22:51:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-20T22:51:18.493 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:51:18.515 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:18.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:18.516 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:51:18.577 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-20T22:51:18.616 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:51:18.638 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:18.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:18.639 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:51:18.725 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:18 smithi084 ceph-mon[40024]: osdmap e597: 8 total, 8 up, 8 in 2024-09-20T22:51:18.725 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:18 smithi084 ceph-mon[40024]: osdmap e598: 8 total, 8 up, 8 in 2024-09-20T22:51:18.725 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:18 smithi084 ceph-mon[29774]: osdmap e597: 8 total, 8 up, 8 in 2024-09-20T22:51:18.725 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:18 smithi084 ceph-mon[29774]: osdmap e598: 8 total, 8 up, 8 in 2024-09-20T22:51:18.740 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:51:18.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:18 smithi154 ceph-mon[33264]: osdmap e597: 8 total, 8 up, 8 in 2024-09-20T22:51:18.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:18 smithi154 ceph-mon[33264]: osdmap e598: 8 total, 8 up, 8 in 2024-09-20T22:51:18.761 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:18.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:18.762 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:51:18.842 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.6 dump_blocked_ops 2024-09-20T22:51:18.863 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:51:18.884 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:18.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:18.885 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:51:18.907 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:51:18.986 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:51:19.007 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:19.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:19.008 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:51:19.086 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:51:19.109 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:51:19.131 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:19.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:19.132 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:51:19.233 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:51:19.254 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:19.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:19.255 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:51:19.257 INFO:teuthology.orchestra.run.smithi084.stderr:marked out osd.1. 2024-09-20T22:51:19.257 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:19 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mon-a[29770]: 2024-09-20T22:51:19.003+0000 7f26ad5e8640 -1 mon.a@0(leader).osd e598 definitely_dead 0 2024-09-20T22:51:19.264 INFO:tasks.rados.rados.0.smithi084.stdout:2340: done (3 left) 2024-09-20T22:51:19.264 INFO:tasks.rados.rados.0.smithi084.stdout:2341: done (2 left) 2024-09-20T22:51:19.264 INFO:tasks.rados.rados.0.smithi084.stdout:2345: done (1 left) 2024-09-20T22:51:19.264 INFO:tasks.rados.rados.0.smithi084.stdout:2347: done (0 left) 2024-09-20T22:51:19.264 INFO:tasks.rados.rados.0.smithi084.stdout:2348: write (excl) oid 50 current snap is 262 2024-09-20T22:51:19.265 INFO:tasks.rados.rados.0.smithi084.stdout:2348: seq_num 807 ranges {763249=659361,2090498=582616,3295061=268188} 2024-09-20T22:51:19.269 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is set 2024-09-20T22:51:19.273 INFO:tasks.rados.rados.0.smithi084.stdout:2348: writing smithi08477203-50 from 763249 to 1422610 tid 1 2024-09-20T22:51:19.283 INFO:tasks.rados.rados.0.smithi084.stdout:2348: writing smithi08477203-50 from 2090498 to 2673114 tid 2 2024-09-20T22:51:19.290 INFO:tasks.rados.rados.0.smithi084.stdout:2348: writing smithi08477203-50 from 3295061 to 3563249 tid 3 2024-09-20T22:51:19.290 INFO:tasks.rados.rados.0.smithi084.stdout:2349: delete oid 14 current snap is 262 2024-09-20T22:51:19.292 INFO:tasks.rados.rados.0.smithi084.stdout:2349: done (1 left) 2024-09-20T22:51:19.292 INFO:tasks.rados.rados.0.smithi084.stdout:2350: copy_from oid 43 from oid 37 current snap is 262 2024-09-20T22:51:19.292 INFO:tasks.rados.rados.0.smithi084.stdout:2351: rollback oid 47 current snap is 262 2024-09-20T22:51:19.292 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 47 to 252 2024-09-20T22:51:19.292 INFO:tasks.rados.rados.0.smithi084.stdout:2352: write oid 1 current snap is 262 2024-09-20T22:51:19.292 INFO:tasks.rados.rados.0.smithi084.stdout:2352: seq_num 809 ranges {796863=409859,1807462=586285,3066260=530603} 2024-09-20T22:51:19.300 INFO:tasks.rados.rados.0.smithi084.stdout:2352: writing smithi08477203-1 from 796863 to 1206722 tid 1 2024-09-20T22:51:19.306 INFO:tasks.rados.rados.0.smithi084.stdout:2352: writing smithi08477203-1 from 1807462 to 2393747 tid 2 2024-09-20T22:51:19.313 INFO:tasks.rados.rados.0.smithi084.stdout:2352: writing smithi08477203-1 from 3066260 to 3596863 tid 3 2024-09-20T22:51:19.313 INFO:tasks.rados.rados.0.smithi084.stdout:2353: write (excl) oid 4 current snap is 262 2024-09-20T22:51:19.313 INFO:tasks.rados.rados.0.smithi084.stdout:2353: seq_num 810 ranges {413670=485108,1465944=729943,2977330=524801} 2024-09-20T22:51:19.318 INFO:tasks.rados.rados.0.smithi084.stdout:2353: writing smithi08477203-4 from 413670 to 898778 tid 1 2024-09-20T22:51:19.326 INFO:tasks.rados.rados.0.smithi084.stdout:2353: writing smithi08477203-4 from 1465944 to 2195887 tid 2 2024-09-20T22:51:19.334 INFO:tasks.rados.rados.0.smithi084.stdout:2353: writing smithi08477203-4 from 2977330 to 3502131 tid 3 2024-09-20T22:51:19.334 INFO:tasks.rados.rados.0.smithi084.stdout:2354: delete oid 21 current snap is 262 2024-09-20T22:51:19.335 INFO:tasks.rados.rados.0.smithi084.stdout:2351: finishing rollback tid 0 to smithi08477203-47 2024-09-20T22:51:19.335 INFO:tasks.rados.rados.0.smithi084.stdout:2351: finishing rollback tid 1 to smithi08477203-47 2024-09-20T22:51:19.335 INFO:tasks.rados.rados.0.smithi084.stdout:2351: finishing rollback tid 2 to smithi08477203-47 2024-09-20T22:51:19.335 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 47 v 1319 (ObjNum 671 snap 205 seq_num 671) dirty exists 2024-09-20T22:51:19.335 INFO:tasks.rados.rados.0.smithi084.stdout:2353: finishing write tid 1 to smithi08477203-4 2024-09-20T22:51:19.335 INFO:tasks.rados.rados.0.smithi084.stdout:2353: finishing write tid 2 to smithi08477203-4 2024-09-20T22:51:19.337 INFO:tasks.rados.rados.0.smithi084.stdout:2353: finishing write tid 3 to smithi08477203-4 2024-09-20T22:51:19.338 INFO:tasks.rados.rados.0.smithi084.stdout:2353: finishing write tid 4 to smithi08477203-4 2024-09-20T22:51:19.338 INFO:tasks.rados.rados.0.smithi084.stdout:2353: finishing write tid 5 to smithi08477203-4 2024-09-20T22:51:19.338 INFO:tasks.rados.rados.0.smithi084.stdout:2353: oid 4 updating version 0 to 1706 2024-09-20T22:51:19.338 INFO:tasks.rados.rados.0.smithi084.stdout:2353: oid 4 updating version 1706 to 1707 2024-09-20T22:51:19.338 INFO:tasks.rados.rados.0.smithi084.stdout:2353: oid 4 version 1707 is already newer than 1704 2024-09-20T22:51:19.338 INFO:tasks.rados.rados.0.smithi084.stdout:2353: oid 4 version 1707 is already newer than 1705 2024-09-20T22:51:19.338 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 4 v 1707 (ObjNum 810 snap 262 seq_num 810) dirty exists 2024-09-20T22:51:19.338 INFO:tasks.rados.rados.0.smithi084.stdout:2353: left oid 4 (ObjNum 810 snap 262 seq_num 810) 2024-09-20T22:51:19.357 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:51:19.359 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:51:19.359 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:51:19.375 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:51:19.376 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:51:19.390 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:19.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:19.391 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:51:19.392 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:51:19.392 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:51:19.409 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-20T22:51:19.409 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T22:51:19.425 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:51:19.425 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:51:19.442 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:51:19.442 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:51:19.459 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:51:19.459 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:51:19.459 INFO:journalctl@ceph.iscsi.iscsi.a.smithi154.stdout:Sep 20 22:51:19 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-iscsi-iscsi-a[59730]: debug there is no tcmu-runner data available 2024-09-20T22:51:19.459 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:19 smithi154 ceph-mon[33264]: pgmap v798: 158 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 153 active+clean; 531 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-20T22:51:19.460 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:19 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/2130267518' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:51:19.460 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:19 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:51:19.460 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:19 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/1561786129' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["1"]}]: dispatch 2024-09-20T22:51:19.460 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:19 smithi154 ceph-mon[33264]: Client client.admin marked osd.1 out, while it was still marked up 2024-09-20T22:51:19.460 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:19 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T22:51:19.460 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:19 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/1561786129' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["1"]}]': finished 2024-09-20T22:51:19.460 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:19 smithi154 ceph-mon[33264]: osdmap e599: 8 total, 8 up, 7 in 2024-09-20T22:51:19.460 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:19 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/3568075423' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:51:19.475 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:51:19.475 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:51:19.492 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:51:19.514 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:19.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:19.515 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:51:19.538 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:19 smithi084 ceph-mon[40024]: pgmap v798: 158 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 153 active+clean; 531 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-20T22:51:19.539 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:19 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/2130267518' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:51:19.539 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:19 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:51:19.539 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:19 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/1561786129' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["1"]}]: dispatch 2024-09-20T22:51:19.539 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:19 smithi084 ceph-mon[40024]: Client client.admin marked osd.1 out, while it was still marked up 2024-09-20T22:51:19.539 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:19 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T22:51:19.539 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:19 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/1561786129' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["1"]}]': finished 2024-09-20T22:51:19.539 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:19 smithi084 ceph-mon[40024]: osdmap e599: 8 total, 8 up, 7 in 2024-09-20T22:51:19.539 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:19 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/3568075423' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:51:19.540 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:19 smithi084 ceph-mon[29774]: pgmap v798: 158 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 153 active+clean; 531 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail 2024-09-20T22:51:19.540 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:19 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/2130267518' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:51:19.540 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:19 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:51:19.540 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:19 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/1561786129' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["1"]}]: dispatch 2024-09-20T22:51:19.540 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:19 smithi084 ceph-mon[29774]: Client client.admin marked osd.1 out, while it was still marked up 2024-09-20T22:51:19.540 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:19 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T22:51:19.540 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:19 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/1561786129' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["1"]}]': finished 2024-09-20T22:51:19.540 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:19 smithi084 ceph-mon[29774]: osdmap e599: 8 total, 8 up, 7 in 2024-09-20T22:51:19.540 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:19 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/3568075423' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:51:19.561 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-20T22:51:19.561 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:51:19.616 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:51:19.637 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:19.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:19.638 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:51:19.738 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:51:19.760 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:19.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:19.761 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:51:19.861 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:51:19.882 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:19.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:19.883 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:51:19.984 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:51:20.006 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:20.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:20.007 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:51:20.108 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:51:20.131 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:20.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:20.132 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:51:20.157 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.6 dump_historic_ops 2024-09-20T22:51:20.234 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:51:20.255 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:20.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:20.256 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:51:20.357 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:51:20.378 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:20.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:20.379 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:51:20.472 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:51:20.481 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:51:20.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:20.504 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:20.504 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:51:20.605 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:51:20.626 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:20.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:20.627 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:51:20.716 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:20 smithi154 ceph-mon[33264]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:51:20.716 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:20 smithi154 ceph-mon[33264]: osdmap e600: 8 total, 8 up, 7 in 2024-09-20T22:51:20.728 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:51:20.749 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:20.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:20.750 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:51:20.851 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:51:20.864 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:20 smithi084 ceph-mon[29774]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:51:20.864 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:20 smithi084 ceph-mon[29774]: osdmap e600: 8 total, 8 up, 7 in 2024-09-20T22:51:20.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:20 smithi084 ceph-mon[40024]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:51:20.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:20 smithi084 ceph-mon[40024]: osdmap e600: 8 total, 8 up, 7 in 2024-09-20T22:51:20.872 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:20.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:20.873 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:51:20.926 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-20T22:51:20.926 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:51:20.974 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:51:20.996 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:20.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:20.997 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:51:21.097 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:51:21.118 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:21.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:21.119 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:51:21.220 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:51:21.241 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:21.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:21.242 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:51:21.271 INFO:tasks.rados.rados.0.smithi084.stdout:2350: finishing copy_from to smithi08477203-43 2024-09-20T22:51:21.271 INFO:tasks.rados.rados.0.smithi084.stdout:2350: got expected ENOENT (src dne) 2024-09-20T22:51:21.272 INFO:tasks.rados.rados.0.smithi084.stdout:2350: finishing copy_from racing read to smithi08477203-43 2024-09-20T22:51:21.279 INFO:tasks.rados.rados.0.smithi084.stdout:2348: finishing write tid 1 to smithi08477203-50 2024-09-20T22:51:21.280 INFO:tasks.rados.rados.0.smithi084.stdout:2348: finishing write tid 2 to smithi08477203-50 2024-09-20T22:51:21.281 INFO:tasks.rados.rados.0.smithi084.stdout:2348: finishing write tid 3 to smithi08477203-50 2024-09-20T22:51:21.281 INFO:tasks.rados.rados.0.smithi084.stdout:2348: finishing write tid 4 to smithi08477203-50 2024-09-20T22:51:21.282 INFO:tasks.rados.rados.0.smithi084.stdout:2348: finishing write tid 5 to smithi08477203-50 2024-09-20T22:51:21.282 INFO:tasks.rados.rados.0.smithi084.stdout:2348: oid 50 updating version 0 to 803 2024-09-20T22:51:21.282 INFO:tasks.rados.rados.0.smithi084.stdout:2348: oid 50 version 803 is already newer than 802 2024-09-20T22:51:21.282 INFO:tasks.rados.rados.0.smithi084.stdout:2348: oid 50 updating version 803 to 805 2024-09-20T22:51:21.282 INFO:tasks.rados.rados.0.smithi084.stdout:2348: oid 50 version 805 is already newer than 804 2024-09-20T22:51:21.282 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 50 v 805 (ObjNum 807 snap 262 seq_num 807) dirty exists 2024-09-20T22:51:21.282 INFO:tasks.rados.rados.0.smithi084.stdout:2348: left oid 50 (ObjNum 807 snap 262 seq_num 807) 2024-09-20T22:51:21.291 INFO:tasks.rados.rados.0.smithi084.stdout:2352: finishing write tid 1 to smithi08477203-1 2024-09-20T22:51:21.292 INFO:tasks.rados.rados.0.smithi084.stdout:2352: finishing write tid 2 to smithi08477203-1 2024-09-20T22:51:21.292 INFO:tasks.rados.rados.0.smithi084.stdout:2352: finishing write tid 3 to smithi08477203-1 2024-09-20T22:51:21.293 INFO:tasks.rados.rados.0.smithi084.stdout:2352: finishing write tid 4 to smithi08477203-1 2024-09-20T22:51:21.294 INFO:tasks.rados.rados.0.smithi084.stdout:2352: finishing write tid 5 to smithi08477203-1 2024-09-20T22:51:21.295 INFO:tasks.rados.rados.0.smithi084.stdout:2352: oid 1 updating version 0 to 1225 2024-09-20T22:51:21.295 INFO:tasks.rados.rados.0.smithi084.stdout:2352: oid 1 updating version 1225 to 1226 2024-09-20T22:51:21.295 INFO:tasks.rados.rados.0.smithi084.stdout:2352: oid 1 updating version 1226 to 1228 2024-09-20T22:51:21.295 INFO:tasks.rados.rados.0.smithi084.stdout:2352: oid 1 version 1228 is already newer than 1227 2024-09-20T22:51:21.295 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 1 v 1228 (ObjNum 809 snap 262 seq_num 809) dirty exists 2024-09-20T22:51:21.295 INFO:tasks.rados.rados.0.smithi084.stdout:2352: left oid 1 (ObjNum 809 snap 262 seq_num 809) 2024-09-20T22:51:21.342 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:51:21.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:21.368 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:21.368 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:51:21.468 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:51:21.491 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:21.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:21.492 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:51:21.513 INFO:tasks.rados.rados.0.smithi084.stdout:2348: done (5 left) 2024-09-20T22:51:21.513 INFO:tasks.rados.rados.0.smithi084.stdout:2350: done (4 left) 2024-09-20T22:51:21.513 INFO:tasks.rados.rados.0.smithi084.stdout:2351: done (3 left) 2024-09-20T22:51:21.513 INFO:tasks.rados.rados.0.smithi084.stdout:2352: done (2 left) 2024-09-20T22:51:21.513 INFO:tasks.rados.rados.0.smithi084.stdout:2353: done (1 left) 2024-09-20T22:51:21.513 INFO:tasks.rados.rados.0.smithi084.stdout:2354: done (0 left) 2024-09-20T22:51:21.513 INFO:tasks.rados.rados.0.smithi084.stdout:2355: copy_from oid 31 from oid 9 current snap is 262 2024-09-20T22:51:21.513 INFO:tasks.rados.rados.0.smithi084.stdout:2356: snap_remove snap 245 2024-09-20T22:51:21.587 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-20T22:51:21.593 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:51:21.652 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:21.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:21.653 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:51:21.753 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:51:21.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:21 smithi154 ceph-mon[33264]: pgmap v801: 158 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 153 active+clean; 468 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 10 MiB/s rd, 1.6 MiB/s wr, 11 op/s 2024-09-20T22:51:21.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:21 smithi154 ceph-mon[33264]: 7.9 starting backfill to osd.7 from (0'0,0'0] MAX to 598'862 2024-09-20T22:51:21.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:21 smithi154 ceph-mon[33264]: osdmap e601: 8 total, 8 up, 7 in 2024-09-20T22:51:21.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:21 smithi154 ceph-mon[33264]: 1.3 starting backfill to osd.5 from (0'0,0'0] MAX to 77'128 2024-09-20T22:51:21.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:21 smithi154 ceph-mon[33264]: 7.4 starting backfill to osd.6 from (0'0,0'0] MAX to 597'325 2024-09-20T22:51:21.776 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:21.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:21.777 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:51:21.849 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:51:21.864 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:21 smithi084 ceph-mon[29774]: pgmap v801: 158 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 153 active+clean; 468 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 10 MiB/s rd, 1.6 MiB/s wr, 11 op/s 2024-09-20T22:51:21.864 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:21 smithi084 ceph-mon[29774]: 7.9 starting backfill to osd.7 from (0'0,0'0] MAX to 598'862 2024-09-20T22:51:21.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:21 smithi084 ceph-mon[29774]: osdmap e601: 8 total, 8 up, 7 in 2024-09-20T22:51:21.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:21 smithi084 ceph-mon[29774]: 1.3 starting backfill to osd.5 from (0'0,0'0] MAX to 77'128 2024-09-20T22:51:21.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:21 smithi084 ceph-mon[29774]: 7.4 starting backfill to osd.6 from (0'0,0'0] MAX to 597'325 2024-09-20T22:51:21.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:21 smithi084 ceph-mon[40024]: pgmap v801: 158 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 153 active+clean; 468 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 10 MiB/s rd, 1.6 MiB/s wr, 11 op/s 2024-09-20T22:51:21.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:21 smithi084 ceph-mon[40024]: 7.9 starting backfill to osd.7 from (0'0,0'0] MAX to 598'862 2024-09-20T22:51:21.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:21 smithi084 ceph-mon[40024]: osdmap e601: 8 total, 8 up, 7 in 2024-09-20T22:51:21.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:21 smithi084 ceph-mon[40024]: 1.3 starting backfill to osd.5 from (0'0,0'0] MAX to 77'128 2024-09-20T22:51:21.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:21 smithi084 ceph-mon[40024]: 7.4 starting backfill to osd.6 from (0'0,0'0] MAX to 597'325 2024-09-20T22:51:21.877 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:51:21.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:21.900 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:21.900 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:51:22.001 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:51:22.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:22.023 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:22.023 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:51:22.124 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:51:22.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:22.146 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:22.146 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:51:22.247 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:51:22.268 INFO:tasks.rados.rados.0.smithi084.stdout:2356: done (1 left) 2024-09-20T22:51:22.268 INFO:tasks.rados.rados.0.smithi084.stdout:2357: write oid 48 current snap is 262 2024-09-20T22:51:22.268 INFO:tasks.rados.rados.0.smithi084.stdout:2355: finishing copy_from to smithi08477203-31 2024-09-20T22:51:22.268 INFO:tasks.rados.rados.0.smithi084.stdout:2355: got expected ENOENT (src dne) 2024-09-20T22:51:22.268 INFO:tasks.rados.rados.0.smithi084.stdout:2357: seq_num 812 ranges {447284=635606,1866555=733612,3315823=331461} 2024-09-20T22:51:22.273 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:22.274 INFO:tasks.rados.rados.0.smithi084.stdout:2357: writing smithi08477203-48 from 447284 to 1082890 tid 1 2024-09-20T22:51:22.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:22.274 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:51:22.283 INFO:tasks.rados.rados.0.smithi084.stdout:2357: writing smithi08477203-48 from 1866555 to 2600167 tid 2 2024-09-20T22:51:22.289 INFO:tasks.rados.rados.0.smithi084.stdout:2357: writing smithi08477203-48 from 3315823 to 3647284 tid 3 2024-09-20T22:51:22.289 INFO:tasks.rados.rados.0.smithi084.stdout:2358: snap_remove snap 204 2024-09-20T22:51:22.310 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-20T22:51:22.310 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:51:22.376 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:51:22.398 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:22.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:22.399 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:51:22.500 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:51:22.522 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:22.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:22.522 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:51:22.605 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:22 smithi154 ceph-mon[33264]: 7.2 starting backfill to osd.7 from (0'0,0'0] MAX to 597'644 2024-09-20T22:51:22.605 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:22 smithi154 ceph-mon[33264]: 7.0 starting backfill to osd.2 from (0'0,0'0] MAX to 596'434 2024-09-20T22:51:22.605 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:22 smithi154 ceph-mon[33264]: osdmap e602: 8 total, 8 up, 7 in 2024-09-20T22:51:22.623 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:51:22.645 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:22.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:22.646 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:51:22.747 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:51:22.767 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:22.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:22.769 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:51:22.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:22 smithi084 ceph-mon[29774]: 7.2 starting backfill to osd.7 from (0'0,0'0] MAX to 597'644 2024-09-20T22:51:22.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:22 smithi084 ceph-mon[29774]: 7.0 starting backfill to osd.2 from (0'0,0'0] MAX to 596'434 2024-09-20T22:51:22.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:22 smithi084 ceph-mon[29774]: osdmap e602: 8 total, 8 up, 7 in 2024-09-20T22:51:22.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:22 smithi084 ceph-mon[40024]: 7.2 starting backfill to osd.7 from (0'0,0'0] MAX to 597'644 2024-09-20T22:51:22.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:22 smithi084 ceph-mon[40024]: 7.0 starting backfill to osd.2 from (0'0,0'0] MAX to 596'434 2024-09-20T22:51:22.866 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:22 smithi084 ceph-mon[40024]: osdmap e602: 8 total, 8 up, 7 in 2024-09-20T22:51:22.869 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:51:22.891 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:22.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:22.892 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:51:22.957 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.7 dump_blocked_ops 2024-09-20T22:51:22.993 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-20T22:51:22.993 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:51:23.014 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:23.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:23.015 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:51:23.116 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:51:23.137 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:23.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:23.138 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:51:23.199 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:51:23.238 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:51:23.241 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:51:23.263 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:23.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:23.264 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:51:23.270 INFO:tasks.rados.rados.0.smithi084.stdout:2358: done (2 left) 2024-09-20T22:51:23.270 INFO:tasks.rados.rados.0.smithi084.stdout:2359: delete oid 44 current snap is 262 2024-09-20T22:51:23.270 INFO:tasks.rados.rados.0.smithi084.stdout:2355: finishing copy_from racing read to smithi08477203-31 2024-09-20T22:51:23.270 INFO:tasks.rados.rados.0.smithi084.stdout:2357: finishing write tid 1 to smithi08477203-48 2024-09-20T22:51:23.270 INFO:tasks.rados.rados.0.smithi084.stdout:2357: finishing write tid 2 to smithi08477203-48 2024-09-20T22:51:23.271 INFO:tasks.rados.rados.0.smithi084.stdout:2357: finishing write tid 3 to smithi08477203-48 2024-09-20T22:51:23.271 INFO:tasks.rados.rados.0.smithi084.stdout:2357: finishing write tid 4 to smithi08477203-48 2024-09-20T22:51:23.271 INFO:tasks.rados.rados.0.smithi084.stdout:2357: finishing write tid 5 to smithi08477203-48 2024-09-20T22:51:23.271 INFO:tasks.rados.rados.0.smithi084.stdout:2357: oid 48 updating version 0 to 862 2024-09-20T22:51:23.271 INFO:tasks.rados.rados.0.smithi084.stdout:2357: oid 48 version 862 is already newer than 861 2024-09-20T22:51:23.271 INFO:tasks.rados.rados.0.smithi084.stdout:2357: oid 48 updating version 862 to 863 2024-09-20T22:51:23.271 INFO:tasks.rados.rados.0.smithi084.stdout:2357: oid 48 version 863 is already newer than 860 2024-09-20T22:51:23.271 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 48 v 863 (ObjNum 812 snap 262 seq_num 812) dirty exists 2024-09-20T22:51:23.272 INFO:tasks.rados.rados.0.smithi084.stdout:2357: left oid 48 (ObjNum 812 snap 262 seq_num 812) 2024-09-20T22:51:23.276 INFO:tasks.rados.rados.0.smithi084.stdout:2355: done (2 left) 2024-09-20T22:51:23.276 INFO:tasks.rados.rados.0.smithi084.stdout:2357: done (1 left) 2024-09-20T22:51:23.277 INFO:tasks.rados.rados.0.smithi084.stdout:2359: done (0 left) 2024-09-20T22:51:23.277 INFO:tasks.rados.rados.0.smithi084.stdout:2360: read oid 17 snap -1 2024-09-20T22:51:23.277 INFO:tasks.rados.rados.0.smithi084.stdout:2360: expect (ObjNum 774 snap 248 seq_num 774) 2024-09-20T22:51:23.277 INFO:tasks.rados.rados.0.smithi084.stdout:2361: snap_remove snap 210 2024-09-20T22:51:23.365 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:51:23.387 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:23.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:23.389 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:51:23.489 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:51:23.511 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:23.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:23.511 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:51:23.596 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:23 smithi154 ceph-mon[33264]: pgmap v804: 158 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 153 active+clean; 468 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 10 MiB/s rd, 1.6 MiB/s wr, 11 op/s 2024-09-20T22:51:23.596 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:23 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:51:23.596 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:23 smithi154 ceph-mon[33264]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:51:23.596 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:23 smithi154 ceph-mon[33264]: osdmap e603: 8 total, 8 up, 7 in 2024-09-20T22:51:23.612 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:51:23.624 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd unset noscrub 2024-09-20T22:51:23.633 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:23.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:23.634 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:51:23.647 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:23 smithi084 ceph-mon[40024]: pgmap v804: 158 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 153 active+clean; 468 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 10 MiB/s rd, 1.6 MiB/s wr, 11 op/s 2024-09-20T22:51:23.647 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:23 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:51:23.647 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:23 smithi084 ceph-mon[40024]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:51:23.647 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:23 smithi084 ceph-mon[40024]: osdmap e603: 8 total, 8 up, 7 in 2024-09-20T22:51:23.647 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:23 smithi084 ceph-mon[29774]: pgmap v804: 158 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 153 active+clean; 468 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 10 MiB/s rd, 1.6 MiB/s wr, 11 op/s 2024-09-20T22:51:23.647 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:23 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:51:23.648 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:23 smithi084 ceph-mon[29774]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:51:23.648 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:23 smithi084 ceph-mon[29774]: osdmap e603: 8 total, 8 up, 7 in 2024-09-20T22:51:23.701 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-20T22:51:23.701 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:51:23.735 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:51:23.752 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:51:23.752 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:51:23.756 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:23.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:23.757 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:51:23.770 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:51:23.770 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:51:23.786 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:51:23.786 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:51:23.802 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-20T22:51:23.803 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T22:51:23.819 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:51:23.820 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:51:23.837 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:51:23.837 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:51:23.855 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:51:23.856 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:51:23.857 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:51:23.876 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:51:23.876 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:51:23.878 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:23.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:23.879 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:51:23.888 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:51:23.980 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:51:24.002 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:24.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:24.003 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:51:24.104 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:51:24.125 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:24.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:24.126 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:51:24.226 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:51:24.248 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:24.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:24.249 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:51:24.261 INFO:tasks.rados.rados.0.smithi084.stdout:2361: done (1 left) 2024-09-20T22:51:24.261 INFO:tasks.rados.rados.0.smithi084.stdout:2362: write (excl) oid 20 current snap is 262 2024-09-20T22:51:24.261 INFO:tasks.rados.rados.0.smithi084.stdout:2362: seq_num 814 ranges {480898=502457,1699872=737281,3087022=593876} 2024-09-20T22:51:24.267 INFO:tasks.rados.rados.0.smithi084.stdout:2362: writing smithi08477203-20 from 480898 to 983355 tid 1 2024-09-20T22:51:24.276 INFO:tasks.rados.rados.0.smithi084.stdout:2362: writing smithi08477203-20 from 1699872 to 2437153 tid 2 2024-09-20T22:51:24.283 INFO:tasks.rados.rados.0.smithi084.stdout:2362: writing smithi08477203-20 from 3087022 to 3680898 tid 3 2024-09-20T22:51:24.283 INFO:tasks.rados.rados.0.smithi084.stdout:2363: write oid 11 current snap is 262 2024-09-20T22:51:24.283 INFO:tasks.rados.rados.0.smithi084.stdout:2363: seq_num 815 ranges {497705=577706,1642001=480939,2598092=497514} 2024-09-20T22:51:24.289 INFO:tasks.rados.rados.0.smithi084.stdout:2363: writing smithi08477203-11 from 497705 to 1075411 tid 1 2024-09-20T22:51:24.294 INFO:tasks.rados.rados.0.smithi084.stdout:2363: writing smithi08477203-11 from 1642001 to 2122940 tid 2 2024-09-20T22:51:24.300 INFO:tasks.rados.rados.0.smithi084.stdout:2363: writing smithi08477203-11 from 2598092 to 3095606 tid 3 2024-09-20T22:51:24.300 INFO:tasks.rados.rados.0.smithi084.stdout:2364: copy_from oid 30 from oid 35 current snap is 262 2024-09-20T22:51:24.300 INFO:tasks.rados.rados.0.smithi084.stdout:2360: expect (ObjNum 774 snap 248 seq_num 774) 2024-09-20T22:51:24.303 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.7 dump_historic_ops 2024-09-20T22:51:24.350 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:51:24.372 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:24.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:24.374 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:51:24.434 INFO:tasks.rados.rados.0.smithi084.stdout:2362: finishing write tid 1 to smithi08477203-20 2024-09-20T22:51:24.435 INFO:tasks.rados.rados.0.smithi084.stdout:2362: finishing write tid 2 to smithi08477203-20 2024-09-20T22:51:24.435 INFO:tasks.rados.rados.0.smithi084.stdout:2362: finishing write tid 3 to smithi08477203-20 2024-09-20T22:51:24.435 INFO:tasks.rados.rados.0.smithi084.stdout:2362: finishing write tid 4 to smithi08477203-20 2024-09-20T22:51:24.435 INFO:tasks.rados.rados.0.smithi084.stdout:2362: finishing write tid 5 to smithi08477203-20 2024-09-20T22:51:24.435 INFO:tasks.rados.rados.0.smithi084.stdout:2362: oid 20 updating version 0 to 1231 2024-09-20T22:51:24.435 INFO:tasks.rados.rados.0.smithi084.stdout:2362: oid 20 version 1231 is already newer than 1229 2024-09-20T22:51:24.436 INFO:tasks.rados.rados.0.smithi084.stdout:2362: oid 20 version 1231 is already newer than 1230 2024-09-20T22:51:24.436 INFO:tasks.rados.rados.0.smithi084.stdout:2362: oid 20 updating version 1231 to 1232 2024-09-20T22:51:24.436 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 20 v 1232 (ObjNum 814 snap 262 seq_num 814) dirty exists 2024-09-20T22:51:24.436 INFO:tasks.rados.rados.0.smithi084.stdout:2362: left oid 20 (ObjNum 814 snap 262 seq_num 814) 2024-09-20T22:51:24.436 INFO:tasks.rados.rados.0.smithi084.stdout:2363: finishing write tid 1 to smithi08477203-11 2024-09-20T22:51:24.436 INFO:tasks.rados.rados.0.smithi084.stdout:2363: finishing write tid 2 to smithi08477203-11 2024-09-20T22:51:24.436 INFO:tasks.rados.rados.0.smithi084.stdout:2360: done (3 left) 2024-09-20T22:51:24.436 INFO:tasks.rados.rados.0.smithi084.stdout:2362: done (2 left) 2024-09-20T22:51:24.437 INFO:tasks.rados.rados.0.smithi084.stdout:2365: write (excl) oid 18 current snap is 262 2024-09-20T22:51:24.437 INFO:tasks.rados.rados.0.smithi084.stdout:2363: finishing write tid 3 to smithi08477203-11 2024-09-20T22:51:24.437 INFO:tasks.rados.rados.0.smithi084.stdout:2363: finishing write tid 4 to smithi08477203-11 2024-09-20T22:51:24.437 INFO:tasks.rados.rados.0.smithi084.stdout:2365: seq_num 816 ranges {514512=652955,1700483=740950,3025515=688997} 2024-09-20T22:51:24.440 INFO:tasks.rados.rados.0.smithi084.stdout:2365: writing smithi08477203-18 from 514512 to 1167467 tid 1 2024-09-20T22:51:24.447 INFO:tasks.rados.rados.0.smithi084.stdout:2365: writing smithi08477203-18 from 1700483 to 2441433 tid 2 2024-09-20T22:51:24.454 INFO:tasks.rados.rados.0.smithi084.stdout:2365: writing smithi08477203-18 from 3025515 to 3714512 tid 3 2024-09-20T22:51:24.455 INFO:tasks.rados.rados.0.smithi084.stdout:2366: snap_remove snap 239 2024-09-20T22:51:24.474 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:51:24.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:24.496 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:24.496 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:51:24.560 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:51:24.597 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:51:24.614 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:24 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/4116140042' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:51:24.614 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:24 smithi084 ceph-mon[40024]: osdmap e604: 8 total, 8 up, 7 in 2024-09-20T22:51:24.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:24 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/4116140042' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:51:24.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:24 smithi084 ceph-mon[29774]: osdmap e604: 8 total, 8 up, 7 in 2024-09-20T22:51:24.618 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:24.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:24.619 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:51:24.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:24 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/4116140042' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:51:24.681 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:24 smithi154 ceph-mon[33264]: osdmap e604: 8 total, 8 up, 7 in 2024-09-20T22:51:24.720 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:51:24.741 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:24.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:24.742 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:51:24.843 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:51:24.864 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:24.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:24.865 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:51:24.965 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:51:24.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:24.988 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:24.988 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:51:25.054 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-20T22:51:25.054 INFO:teuthology.orchestra.run.smithi154.stdout: "size": 20, 2024-09-20T22:51:25.054 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 600, 2024-09-20T22:51:25.054 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [ 2024-09-20T22:51:25.054 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.054 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:786 7.17 7:ed54065f:::smithi08477203-12:head [copy-from ver 61] snapc 19=[17] ondisk+write+known_if_redirected+supports_pool_eio e145)", 2024-09-20T22:51:25.054 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:44.333196+0000", 2024-09-20T22:51:25.054 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 460.717571572, 2024-09-20T22:51:25.054 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.023328503, 2024-09-20T22:51:25.055 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:51:25.055 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:51:25.055 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:51:25.055 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:51:25.055 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:51:25.055 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:51:25.055 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 786 2024-09-20T22:51:25.055 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.055 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:51:25.055 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.055 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:51:25.055 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.333196+0000", 2024-09-20T22:51:25.055 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:51:25.055 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.056 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.056 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:51:25.056 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.333196+0000", 2024-09-20T22:51:25.056 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:51:25.056 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.056 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.056 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:51:25.056 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.333197+0000", 2024-09-20T22:51:25.056 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.486e-06 2024-09-20T22:51:25.056 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.056 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.056 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:51:25.056 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.333207+0000", 2024-09-20T22:51:25.056 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0307e-05 2024-09-20T22:51:25.057 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.057 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.057 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:51:25.057 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.333220+0000", 2024-09-20T22:51:25.057 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.306e-05 2024-09-20T22:51:25.057 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.057 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.057 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:51:25.057 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.333229+0000", 2024-09-20T22:51:25.057 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.3450000000000006e-06 2024-09-20T22:51:25.057 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.057 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.058 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:51:25.058 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.333292+0000", 2024-09-20T22:51:25.058 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.3137000000000002e-05 2024-09-20T22:51:25.058 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.058 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.058 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op must wait for map", 2024-09-20T22:51:25.058 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.333298+0000", 2024-09-20T22:51:25.058 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.7780000000000004e-06 2024-09-20T22:51:25.058 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.058 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.058 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:51:25.058 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.336639+0000", 2024-09-20T22:51:25.058 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0033414479999999999 2024-09-20T22:51:25.058 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.059 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.059 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:51:25.059 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.336729+0000", 2024-09-20T22:51:25.059 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.9888000000000004e-05 2024-09-20T22:51:25.059 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.059 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.059 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 2,5", 2024-09-20T22:51:25.059 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.344876+0000", 2024-09-20T22:51:25.059 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0081473989999999996 2024-09-20T22:51:25.059 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.059 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.059 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:51:25.059 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.355138+0000", 2024-09-20T22:51:25.059 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.010261462000000001 2024-09-20T22:51:25.060 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.060 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.060 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:51:25.060 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.355972+0000", 2024-09-20T22:51:25.060 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00083377899999999999 2024-09-20T22:51:25.060 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.060 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.060 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:51:25.060 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.356451+0000", 2024-09-20T22:51:25.060 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00047985200000000001 2024-09-20T22:51:25.060 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.060 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.060 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:51:25.060 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.356486+0000", 2024-09-20T22:51:25.061 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.4653000000000002e-05 2024-09-20T22:51:25.061 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.061 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.061 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:51:25.061 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.356524+0000", 2024-09-20T22:51:25.061 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.7908999999999999e-05 2024-09-20T22:51:25.061 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:51:25.061 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:51:25.061 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:51:25.061 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.061 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.061 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:787 7.17 7:ed54065f:::smithi08477203-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e145)", 2024-09-20T22:51:25.061 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:44.333248+0000", 2024-09-20T22:51:25.062 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 460.717519477, 2024-09-20T22:51:25.062 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.016881184, 2024-09-20T22:51:25.062 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:51:25.062 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:51:25.062 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:51:25.062 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:51:25.062 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:51:25.062 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:51:25.062 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 787 2024-09-20T22:51:25.062 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.062 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:51:25.063 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.063 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:51:25.063 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.333248+0000", 2024-09-20T22:51:25.063 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:51:25.063 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.063 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.063 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:51:25.063 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.333248+0000", 2024-09-20T22:51:25.063 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:51:25.063 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.063 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.063 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:51:25.063 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.333248+0000", 2024-09-20T22:51:25.063 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.1399999999999997e-07 2024-09-20T22:51:25.063 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.064 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.064 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:51:25.064 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.333250+0000", 2024-09-20T22:51:25.064 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.8199999999999999e-06 2024-09-20T22:51:25.064 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.064 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.064 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:51:25.064 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.333257+0000", 2024-09-20T22:51:25.064 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.43e-06 2024-09-20T22:51:25.064 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.064 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.064 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:51:25.064 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.333261+0000", 2024-09-20T22:51:25.064 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.3670000000000001e-06 2024-09-20T22:51:25.065 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.065 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.065 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:51:25.065 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.333351+0000", 2024-09-20T22:51:25.065 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.0486000000000001e-05 2024-09-20T22:51:25.065 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.065 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.065 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting_for_map not empty", 2024-09-20T22:51:25.065 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.333357+0000", 2024-09-20T22:51:25.065 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.3520000000000003e-06 2024-09-20T22:51:25.065 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.065 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.065 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:51:25.065 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.336912+0000", 2024-09-20T22:51:25.066 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.003554612 2024-09-20T22:51:25.066 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.066 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.066 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for blocked object", 2024-09-20T22:51:25.066 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.336957+0000", 2024-09-20T22:51:25.066 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.4919000000000001e-05 2024-09-20T22:51:25.066 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.066 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.066 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:51:25.066 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.349989+0000", 2024-09-20T22:51:25.066 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.01303238 2024-09-20T22:51:25.066 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.066 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.066 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:51:25.067 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.350058+0000", 2024-09-20T22:51:25.067 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.8418999999999997e-05 2024-09-20T22:51:25.067 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.067 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.067 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:51:25.067 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.350129+0000", 2024-09-20T22:51:25.067 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.0984999999999999e-05 2024-09-20T22:51:25.067 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:51:25.067 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:51:25.067 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:51:25.067 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.067 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.067 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:804 7.17 7:ea720b85:::smithi08477203-26:head [copy-from ver 61] snapc 19=[17] ondisk+write+known_if_redirected+supports_pool_eio e145)", 2024-09-20T22:51:25.068 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:44.451241+0000", 2024-09-20T22:51:25.068 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 460.599525642, 2024-09-20T22:51:25.068 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.012865619999999999, 2024-09-20T22:51:25.068 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:51:25.068 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:51:25.068 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:51:25.068 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:51:25.068 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:51:25.068 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:51:25.068 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 804 2024-09-20T22:51:25.068 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.068 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:51:25.069 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.069 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:51:25.069 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.451241+0000", 2024-09-20T22:51:25.069 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:51:25.069 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.069 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.069 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:51:25.069 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.451241+0000", 2024-09-20T22:51:25.069 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:51:25.069 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.069 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.069 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:51:25.069 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.451244+0000", 2024-09-20T22:51:25.069 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.2689999999999998e-06 2024-09-20T22:51:25.069 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.070 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.070 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:51:25.070 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.451255+0000", 2024-09-20T22:51:25.070 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1151e-05 2024-09-20T22:51:25.070 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.070 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.070 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:51:25.070 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.451274+0000", 2024-09-20T22:51:25.070 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9389999999999999e-05 2024-09-20T22:51:25.070 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.070 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.070 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:51:25.070 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.451286+0000", 2024-09-20T22:51:25.070 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1703e-05 2024-09-20T22:51:25.071 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.071 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.071 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:51:25.071 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.451361+0000", 2024-09-20T22:51:25.071 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.5310999999999998e-05 2024-09-20T22:51:25.071 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.071 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.071 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:51:25.071 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.451462+0000", 2024-09-20T22:51:25.071 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000100809 2024-09-20T22:51:25.071 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.071 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.071 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 2,5", 2024-09-20T22:51:25.071 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.456603+0000", 2024-09-20T22:51:25.072 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0051411169999999997 2024-09-20T22:51:25.072 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.072 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.072 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:51:25.072 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.463017+0000", 2024-09-20T22:51:25.072 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.006413307 2024-09-20T22:51:25.072 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.072 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.072 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:51:25.072 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.463754+0000", 2024-09-20T22:51:25.072 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00073758000000000005 2024-09-20T22:51:25.072 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.072 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.072 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:51:25.073 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.464036+0000", 2024-09-20T22:51:25.073 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00028171000000000001 2024-09-20T22:51:25.073 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.073 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.073 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:51:25.073 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.464076+0000", 2024-09-20T22:51:25.073 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.9736e-05 2024-09-20T22:51:25.073 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.073 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.073 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:51:25.073 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.464107+0000", 2024-09-20T22:51:25.073 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.1537e-05 2024-09-20T22:51:25.073 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:51:25.073 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:51:25.074 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:51:25.074 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.074 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.074 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:1527 7.17 7:ea720b85:::smithi08477203-26:head [copy-from ver 83] snapc 33=[33] ondisk+write+known_if_redirected+supports_pool_eio e197)", 2024-09-20T22:51:25.074 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:44:36.198921+0000", 2024-09-20T22:51:25.074 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 408.85184631800001, 2024-09-20T22:51:25.074 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.021621703999999999, 2024-09-20T22:51:25.074 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:51:25.074 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:51:25.074 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:51:25.074 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:51:25.074 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:51:25.074 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:51:25.075 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 1527 2024-09-20T22:51:25.075 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.075 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:51:25.075 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.075 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:51:25.075 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.198921+0000", 2024-09-20T22:51:25.075 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:51:25.075 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.075 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.075 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:51:25.075 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.198921+0000", 2024-09-20T22:51:25.075 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:51:25.075 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.075 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.076 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:51:25.076 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.198923+0000", 2024-09-20T22:51:25.076 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.0080000000000001e-06 2024-09-20T22:51:25.076 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.076 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.076 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:51:25.076 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.198934+0000", 2024-09-20T22:51:25.076 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0923e-05 2024-09-20T22:51:25.076 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.076 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.076 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:51:25.076 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.198955+0000", 2024-09-20T22:51:25.076 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.0806e-05 2024-09-20T22:51:25.076 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.077 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.077 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:51:25.077 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.198966+0000", 2024-09-20T22:51:25.077 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1779e-05 2024-09-20T22:51:25.077 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.077 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.077 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:51:25.077 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.199051+0000", 2024-09-20T22:51:25.077 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.4357999999999994e-05 2024-09-20T22:51:25.077 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.077 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.077 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op must wait for map", 2024-09-20T22:51:25.077 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.199058+0000", 2024-09-20T22:51:25.078 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.8530000000000003e-06 2024-09-20T22:51:25.078 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.078 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.078 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:51:25.078 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.202420+0000", 2024-09-20T22:51:25.078 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0033623350000000002 2024-09-20T22:51:25.078 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.078 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.078 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:51:25.078 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.202498+0000", 2024-09-20T22:51:25.078 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.8316e-05 2024-09-20T22:51:25.078 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.078 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.079 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 2,5", 2024-09-20T22:51:25.079 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.209140+0000", 2024-09-20T22:51:25.079 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0066415759999999997 2024-09-20T22:51:25.079 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.079 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.079 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:51:25.079 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.214931+0000", 2024-09-20T22:51:25.079 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0057916520000000004 2024-09-20T22:51:25.079 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.079 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.079 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:51:25.079 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.220009+0000", 2024-09-20T22:51:25.079 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0050773750000000003 2024-09-20T22:51:25.080 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.080 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.080 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:51:25.080 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.220417+0000", 2024-09-20T22:51:25.080 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00040778599999999999 2024-09-20T22:51:25.080 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.080 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.080 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:51:25.080 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.220479+0000", 2024-09-20T22:51:25.080 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.1977000000000006e-05 2024-09-20T22:51:25.080 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.080 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.080 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:51:25.080 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.220542+0000", 2024-09-20T22:51:25.080 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.3960000000000004e-05 2024-09-20T22:51:25.081 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:51:25.081 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:51:25.081 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:51:25.081 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.081 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.081 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:1528 7.17 7:ea720b85:::smithi08477203-26:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e197)", 2024-09-20T22:51:25.081 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:44:36.198994+0000", 2024-09-20T22:51:25.081 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 408.85177359099998, 2024-09-20T22:51:25.081 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.015013924, 2024-09-20T22:51:25.081 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:51:25.081 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:51:25.081 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:51:25.081 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:51:25.081 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:51:25.082 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:51:25.082 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 1528 2024-09-20T22:51:25.082 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.082 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:51:25.082 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.082 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:51:25.082 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.198994+0000", 2024-09-20T22:51:25.082 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:51:25.082 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.084 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.084 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:51:25.084 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.198994+0000", 2024-09-20T22:51:25.084 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:51:25.084 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.084 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.084 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:51:25.084 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.198994+0000", 2024-09-20T22:51:25.084 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.8100000000000003e-07 2024-09-20T22:51:25.084 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.084 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.084 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:51:25.084 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.198996+0000", 2024-09-20T22:51:25.085 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.3e-06 2024-09-20T22:51:25.085 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.085 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.085 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:51:25.085 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.199008+0000", 2024-09-20T22:51:25.085 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1603999999999999e-05 2024-09-20T22:51:25.085 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.085 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.085 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:51:25.085 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.199013+0000", 2024-09-20T22:51:25.085 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.4500000000000003e-06 2024-09-20T22:51:25.085 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.085 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.085 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:51:25.086 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.199120+0000", 2024-09-20T22:51:25.086 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000106772 2024-09-20T22:51:25.086 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.086 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.086 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting_for_map not empty", 2024-09-20T22:51:25.086 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.199126+0000", 2024-09-20T22:51:25.086 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.8320000000000002e-06 2024-09-20T22:51:25.086 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.086 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.086 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:51:25.086 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.202625+0000", 2024-09-20T22:51:25.086 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0034986309999999999 2024-09-20T22:51:25.086 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.086 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.086 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for blocked object", 2024-09-20T22:51:25.087 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.202667+0000", 2024-09-20T22:51:25.087 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.2759000000000002e-05 2024-09-20T22:51:25.087 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.087 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.087 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:51:25.087 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.213787+0000", 2024-09-20T22:51:25.087 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.011119377 2024-09-20T22:51:25.087 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.087 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.087 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:51:25.087 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.213902+0000", 2024-09-20T22:51:25.087 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000115369 2024-09-20T22:51:25.087 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.087 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.087 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:51:25.088 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.214007+0000", 2024-09-20T22:51:25.088 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000105349 2024-09-20T22:51:25.088 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:51:25.088 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:51:25.088 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:51:25.088 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.088 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.088 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:1550 7.17 7:ed54065f:::smithi08477203-12:head [copy-from ver 88] snapc 34=[34,33] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2024-09-20T22:51:25.088 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:44:37.238376+0000", 2024-09-20T22:51:25.088 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 407.81239067899998, 2024-09-20T22:51:25.088 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.016634917999999999, 2024-09-20T22:51:25.088 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:51:25.089 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:51:25.089 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:51:25.089 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:51:25.089 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:51:25.089 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:51:25.089 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 1550 2024-09-20T22:51:25.089 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.089 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:51:25.089 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.089 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:51:25.089 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:37.238376+0000", 2024-09-20T22:51:25.089 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:51:25.089 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.089 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.089 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:51:25.090 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:37.238376+0000", 2024-09-20T22:51:25.090 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:51:25.090 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.090 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.090 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:51:25.090 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:37.238379+0000", 2024-09-20T22:51:25.090 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.7989999999999998e-06 2024-09-20T22:51:25.090 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.090 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.090 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:51:25.090 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:37.238408+0000", 2024-09-20T22:51:25.090 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.8980999999999999e-05 2024-09-20T22:51:25.090 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.090 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.090 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:51:25.090 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:37.238433+0000", 2024-09-20T22:51:25.091 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.5185999999999999e-05 2024-09-20T22:51:25.091 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.091 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.091 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:51:25.091 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:37.238446+0000", 2024-09-20T22:51:25.091 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2883e-05 2024-09-20T22:51:25.091 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.091 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.091 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:51:25.091 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:37.238523+0000", 2024-09-20T22:51:25.091 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.6396999999999994e-05 2024-09-20T22:51:25.091 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.091 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.091 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:51:25.091 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:37.238618+0000", 2024-09-20T22:51:25.092 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.5049000000000004e-05 2024-09-20T22:51:25.092 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.092 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.092 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 2,5", 2024-09-20T22:51:25.092 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:37.246556+0000", 2024-09-20T22:51:25.092 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0079378090000000005 2024-09-20T22:51:25.092 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.092 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.092 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:51:25.092 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:37.254025+0000", 2024-09-20T22:51:25.092 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0074697139999999997 2024-09-20T22:51:25.092 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.092 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.092 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:51:25.092 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:37.254033+0000", 2024-09-20T22:51:25.093 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.6979999999999998e-06 2024-09-20T22:51:25.093 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.093 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.093 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:51:25.093 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:37.254919+0000", 2024-09-20T22:51:25.093 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00088564700000000004 2024-09-20T22:51:25.093 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.093 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.093 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:51:25.093 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:37.254958+0000", 2024-09-20T22:51:25.093 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.9515999999999999e-05 2024-09-20T22:51:25.093 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.093 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.093 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:51:25.093 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:37.255011+0000", 2024-09-20T22:51:25.094 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.3239000000000003e-05 2024-09-20T22:51:25.094 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:51:25.094 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:51:25.094 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:51:25.094 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.094 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.094 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:1799 7.17 7:ea720b85:::smithi08477203-26:head [write 521083~714336,stat] snapc 46=[45,44,43,42,41,40,3e] ondisk+write+known_if_redirected+supports_pool_eio e229)", 2024-09-20T22:51:25.094 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:45:08.170663+0000", 2024-09-20T22:51:25.094 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 376.880104034, 2024-09-20T22:51:25.094 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.016368226999999999, 2024-09-20T22:51:25.094 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:51:25.094 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:51:25.094 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:51:25.094 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:51:25.095 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:51:25.095 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:51:25.095 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 1799 2024-09-20T22:51:25.095 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.095 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:51:25.095 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.095 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:51:25.095 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.170663+0000", 2024-09-20T22:51:25.095 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:51:25.095 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.095 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.095 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:51:25.095 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.170663+0000", 2024-09-20T22:51:25.096 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:51:25.096 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.096 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.096 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:51:25.096 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.170664+0000", 2024-09-20T22:51:25.096 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.085e-06 2024-09-20T22:51:25.096 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.096 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.096 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:51:25.096 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.171481+0000", 2024-09-20T22:51:25.096 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00081653900000000002 2024-09-20T22:51:25.096 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.096 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.096 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:51:25.096 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.171492+0000", 2024-09-20T22:51:25.097 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1565e-05 2024-09-20T22:51:25.097 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.097 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.097 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:51:25.097 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.171498+0000", 2024-09-20T22:51:25.097 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.925e-06 2024-09-20T22:51:25.097 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.097 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.097 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:51:25.097 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.171547+0000", 2024-09-20T22:51:25.097 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.8916999999999999e-05 2024-09-20T22:51:25.097 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.097 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.097 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:51:25.097 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.171611+0000", 2024-09-20T22:51:25.098 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.3838999999999996e-05 2024-09-20T22:51:25.098 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.098 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.098 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 2,5", 2024-09-20T22:51:25.098 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.171740+0000", 2024-09-20T22:51:25.098 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00012881500000000001 2024-09-20T22:51:25.098 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.098 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.098 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:51:25.098 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.173862+0000", 2024-09-20T22:51:25.098 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0021225649999999999 2024-09-20T22:51:25.098 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.098 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.098 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:51:25.099 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.179252+0000", 2024-09-20T22:51:25.099 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0053895490000000004 2024-09-20T22:51:25.099 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.099 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.099 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:51:25.099 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.186956+0000", 2024-09-20T22:51:25.099 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0077038050000000002 2024-09-20T22:51:25.099 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.099 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.099 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:51:25.099 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.186993+0000", 2024-09-20T22:51:25.099 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.7141000000000002e-05 2024-09-20T22:51:25.099 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.099 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.099 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:51:25.100 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.187031+0000", 2024-09-20T22:51:25.100 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.8482000000000002e-05 2024-09-20T22:51:25.100 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:51:25.100 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:51:25.100 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:51:25.100 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.100 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.100 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:1800 7.17 7:ea720b85:::smithi08477203-26:head [write 1724715~555421] snapc 46=[45,44,43,42,41,40,3e] ondisk+write+known_if_redirected+supports_pool_eio e229)", 2024-09-20T22:51:25.100 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:45:08.177237+0000", 2024-09-20T22:51:25.100 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 376.87353040900001, 2024-09-20T22:51:25.100 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.012771694, 2024-09-20T22:51:25.100 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:51:25.100 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:51:25.101 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:51:25.101 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:51:25.101 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:51:25.101 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:51:25.101 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 1800 2024-09-20T22:51:25.101 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.101 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:51:25.101 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.101 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:51:25.101 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.177237+0000", 2024-09-20T22:51:25.101 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:51:25.101 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.101 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.101 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:51:25.101 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.177237+0000", 2024-09-20T22:51:25.102 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:51:25.102 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.102 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.102 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:51:25.102 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.177238+0000", 2024-09-20T22:51:25.102 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.6080000000000001e-06 2024-09-20T22:51:25.102 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.102 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.102 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:51:25.102 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.177844+0000", 2024-09-20T22:51:25.102 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00060579999999999998 2024-09-20T22:51:25.103 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.103 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.103 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:51:25.103 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.177857+0000", 2024-09-20T22:51:25.103 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2529e-05 2024-09-20T22:51:25.103 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.103 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.103 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:51:25.103 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.177864+0000", 2024-09-20T22:51:25.103 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.2470000000000002e-06 2024-09-20T22:51:25.103 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.104 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.104 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:51:25.104 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.177908+0000", 2024-09-20T22:51:25.104 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.4329999999999997e-05 2024-09-20T22:51:25.104 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.104 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.104 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:51:25.104 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.177977+0000", 2024-09-20T22:51:25.104 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.8598999999999998e-05 2024-09-20T22:51:25.104 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.104 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.104 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 2,5", 2024-09-20T22:51:25.104 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.178079+0000", 2024-09-20T22:51:25.105 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000102074 2024-09-20T22:51:25.105 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.105 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.105 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:51:25.105 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.181043+0000", 2024-09-20T22:51:25.105 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0029639810000000001 2024-09-20T22:51:25.105 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.105 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.105 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:51:25.105 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.181481+0000", 2024-09-20T22:51:25.105 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00043855899999999999 2024-09-20T22:51:25.105 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.106 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.106 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:51:25.106 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.189928+0000", 2024-09-20T22:51:25.106 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0084463149999999994 2024-09-20T22:51:25.106 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.106 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.106 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:51:25.106 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.189968+0000", 2024-09-20T22:51:25.106 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.0268999999999997e-05 2024-09-20T22:51:25.106 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.106 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.106 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:51:25.107 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.190008+0000", 2024-09-20T22:51:25.107 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.0383000000000003e-05 2024-09-20T22:51:25.107 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:51:25.107 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:51:25.107 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:51:25.107 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.107 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.107 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:2318 7.5 7:a160b5d4:::smithi08477203-16:head [copy-from ver 168] snapc 55=[55,53,52,51,50,4b,47,3e] ondisk+write+known_if_redirected+supports_pool_eio e258)", 2024-09-20T22:51:25.107 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:45:37.175863+0000", 2024-09-20T22:51:25.107 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 347.87490395899999, 2024-09-20T22:51:25.107 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.030719883999999999, 2024-09-20T22:51:25.107 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:51:25.107 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:51:25.107 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:51:25.108 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:51:25.108 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:51:25.108 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:51:25.108 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 2318 2024-09-20T22:51:25.108 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.108 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:51:25.108 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.108 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:51:25.108 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.175863+0000", 2024-09-20T22:51:25.108 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:51:25.108 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.108 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.108 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:51:25.108 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.175863+0000", 2024-09-20T22:51:25.109 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:51:25.109 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.109 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.109 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:51:25.109 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.175865+0000", 2024-09-20T22:51:25.109 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.488e-06 2024-09-20T22:51:25.109 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.109 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.109 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:51:25.109 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.175871+0000", 2024-09-20T22:51:25.109 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.438e-06 2024-09-20T22:51:25.109 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.109 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.109 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:51:25.110 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.175881+0000", 2024-09-20T22:51:25.110 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.6430000000000007e-06 2024-09-20T22:51:25.110 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.110 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.110 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:51:25.110 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.175886+0000", 2024-09-20T22:51:25.110 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.3059999999999997e-06 2024-09-20T22:51:25.110 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.110 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.110 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:51:25.110 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.175944+0000", 2024-09-20T22:51:25.110 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.7982999999999997e-05 2024-09-20T22:51:25.110 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.110 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.110 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op must wait for map", 2024-09-20T22:51:25.110 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.175951+0000", 2024-09-20T22:51:25.111 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.9280000000000004e-06 2024-09-20T22:51:25.111 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.111 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.111 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:51:25.111 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.179794+0000", 2024-09-20T22:51:25.111 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0038435209999999999 2024-09-20T22:51:25.111 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.111 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.111 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:51:25.111 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.179940+0000", 2024-09-20T22:51:25.111 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00014550000000000001 2024-09-20T22:51:25.111 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.111 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.111 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 0,2", 2024-09-20T22:51:25.111 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.190992+0000", 2024-09-20T22:51:25.112 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.011051587999999999 2024-09-20T22:51:25.112 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.112 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.112 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:51:25.112 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.195070+0000", 2024-09-20T22:51:25.112 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0040784929999999999 2024-09-20T22:51:25.112 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.112 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.112 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:51:25.112 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.202908+0000", 2024-09-20T22:51:25.112 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0078376690000000002 2024-09-20T22:51:25.112 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.112 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.112 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:51:25.112 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.206517+0000", 2024-09-20T22:51:25.112 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0036088880000000002 2024-09-20T22:51:25.113 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.113 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.113 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:51:25.113 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.206553+0000", 2024-09-20T22:51:25.113 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.6363999999999999e-05 2024-09-20T22:51:25.113 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.113 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.113 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:51:25.113 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.206583+0000", 2024-09-20T22:51:25.113 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.0074999999999999e-05 2024-09-20T22:51:25.113 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:51:25.113 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:51:25.113 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:51:25.113 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.113 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.114 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:2319 7.5 7:a160b5d4:::smithi08477203-16:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e258)", 2024-09-20T22:51:25.114 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:45:37.175983+0000", 2024-09-20T22:51:25.114 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 347.87478420399998, 2024-09-20T22:51:25.114 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.018323547999999999, 2024-09-20T22:51:25.114 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:51:25.114 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:51:25.114 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:51:25.114 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:51:25.114 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:51:25.114 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:51:25.114 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 2319 2024-09-20T22:51:25.114 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.114 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:51:25.115 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.115 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:51:25.115 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.175983+0000", 2024-09-20T22:51:25.115 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:51:25.115 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.115 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.115 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:51:25.115 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.175983+0000", 2024-09-20T22:51:25.115 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:51:25.115 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.115 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.115 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:51:25.115 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.175984+0000", 2024-09-20T22:51:25.115 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.1099999999999995e-07 2024-09-20T22:51:25.115 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.116 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.116 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:51:25.116 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.175996+0000", 2024-09-20T22:51:25.116 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2121e-05 2024-09-20T22:51:25.116 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.116 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.116 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:51:25.116 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.176019+0000", 2024-09-20T22:51:25.116 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.3465999999999999e-05 2024-09-20T22:51:25.116 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.116 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.116 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:51:25.116 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.176026+0000", 2024-09-20T22:51:25.116 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.8190000000000003e-06 2024-09-20T22:51:25.116 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.117 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.117 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:51:25.117 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.176110+0000", 2024-09-20T22:51:25.117 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.3967000000000004e-05 2024-09-20T22:51:25.117 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.117 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.117 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting_for_map not empty", 2024-09-20T22:51:25.117 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.176116+0000", 2024-09-20T22:51:25.117 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.0249999999999999e-06 2024-09-20T22:51:25.117 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.117 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.117 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:51:25.117 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.180106+0000", 2024-09-20T22:51:25.117 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0039898980000000004 2024-09-20T22:51:25.117 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.118 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.118 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for blocked object", 2024-09-20T22:51:25.118 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.180135+0000", 2024-09-20T22:51:25.118 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.9595999999999999e-05 2024-09-20T22:51:25.118 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.118 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.118 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:51:25.118 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.194106+0000", 2024-09-20T22:51:25.118 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.013970761999999999 2024-09-20T22:51:25.118 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.118 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.118 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:51:25.118 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.194203+0000", 2024-09-20T22:51:25.118 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.7211000000000002e-05 2024-09-20T22:51:25.119 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.119 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.119 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:51:25.119 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.194306+0000", 2024-09-20T22:51:25.119 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00010307200000000001 2024-09-20T22:51:25.119 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:51:25.119 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:51:25.119 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:51:25.119 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.119 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.119 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24415.0:230 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch (1) in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e259)", 2024-09-20T22:51:25.119 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:45:39.207410+0000", 2024-09-20T22:51:25.119 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 345.84335713199999, 2024-09-20T22:51:25.120 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0072298289999999, 2024-09-20T22:51:25.120 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:51:25.120 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:51:25.120 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:51:25.120 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:51:25.120 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24415", 2024-09-20T22:51:25.120 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/2056244102", 2024-09-20T22:51:25.120 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 230 2024-09-20T22:51:25.120 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.120 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:51:25.120 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.120 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:51:25.120 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.207410+0000", 2024-09-20T22:51:25.120 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:51:25.120 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.120 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.121 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:51:25.121 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.207410+0000", 2024-09-20T22:51:25.121 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:51:25.121 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.121 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.121 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:51:25.121 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.207412+0000", 2024-09-20T22:51:25.121 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.7519999999999999e-06 2024-09-20T22:51:25.121 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.121 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.121 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:51:25.121 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.207419+0000", 2024-09-20T22:51:25.121 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.097e-06 2024-09-20T22:51:25.121 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.121 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.122 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:51:25.122 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.207432+0000", 2024-09-20T22:51:25.122 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2719e-05 2024-09-20T22:51:25.122 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.122 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.122 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:51:25.122 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.207464+0000", 2024-09-20T22:51:25.122 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.2820000000000001e-05 2024-09-20T22:51:25.122 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.122 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.122 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:51:25.122 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.207670+0000", 2024-09-20T22:51:25.122 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000205486 2024-09-20T22:51:25.122 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.122 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.123 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-20T22:51:25.123 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.207687+0000", 2024-09-20T22:51:25.123 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.7107000000000001e-05 2024-09-20T22:51:25.123 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.123 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.123 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:51:25.123 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:40.212672+0000", 2024-09-20T22:51:25.123 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.004984562 2024-09-20T22:51:25.123 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.123 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.123 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:51:25.123 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:40.213576+0000", 2024-09-20T22:51:25.123 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00090404799999999996 2024-09-20T22:51:25.124 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.124 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.124 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:51:25.124 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:40.214640+0000", 2024-09-20T22:51:25.124 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0010642379999999999 2024-09-20T22:51:25.124 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:51:25.124 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:51:25.124 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:51:25.124 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.124 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.124 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:2367 7.5 7:a160b5d4:::smithi08477203-16:head [stat,delete] snapc 56=[56,55,53,52,51,50,47,3e] ondisk+write+known_if_redirected+supports_pool_eio e260)", 2024-09-20T22:51:25.124 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:45:39.601299+0000", 2024-09-20T22:51:25.124 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 345.44946771100001, 2024-09-20T22:51:25.124 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.64714523300000004, 2024-09-20T22:51:25.125 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:51:25.125 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:51:25.125 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:51:25.125 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:51:25.125 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:51:25.125 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:51:25.125 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 2367 2024-09-20T22:51:25.125 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.125 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:51:25.125 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.125 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:51:25.125 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.601299+0000", 2024-09-20T22:51:25.125 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:51:25.125 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.126 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.126 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:51:25.126 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.601299+0000", 2024-09-20T22:51:25.126 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:51:25.126 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.126 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.126 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:51:25.126 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.601301+0000", 2024-09-20T22:51:25.126 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3850000000000001e-06 2024-09-20T22:51:25.126 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.126 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.126 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:51:25.126 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.601306+0000", 2024-09-20T22:51:25.126 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.2730000000000002e-06 2024-09-20T22:51:25.127 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.127 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.127 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:51:25.127 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.601318+0000", 2024-09-20T22:51:25.127 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2148e-05 2024-09-20T22:51:25.127 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.127 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.127 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:51:25.127 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.601324+0000", 2024-09-20T22:51:25.127 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.852e-06 2024-09-20T22:51:25.127 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.127 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.128 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:51:25.128 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.601383+0000", 2024-09-20T22:51:25.128 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.8570000000000003e-05 2024-09-20T22:51:25.128 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.128 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.128 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-20T22:51:25.128 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.601391+0000", 2024-09-20T22:51:25.128 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.5290000000000008e-06 2024-09-20T22:51:25.128 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.128 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.128 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:51:25.128 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:40.221227+0000", 2024-09-20T22:51:25.128 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.619835515 2024-09-20T22:51:25.128 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.129 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.129 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for degraded object", 2024-09-20T22:51:25.129 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:40.223919+0000", 2024-09-20T22:51:25.129 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.002692168 2024-09-20T22:51:25.129 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.129 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.129 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:51:25.129 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:40.244536+0000", 2024-09-20T22:51:25.129 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.020617613 2024-09-20T22:51:25.129 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.129 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.129 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:51:25.129 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:40.244653+0000", 2024-09-20T22:51:25.129 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000116796 2024-09-20T22:51:25.130 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.130 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.130 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 1,2", 2024-09-20T22:51:25.130 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:40.244832+0000", 2024-09-20T22:51:25.130 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00017833500000000001 2024-09-20T22:51:25.130 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.130 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.130 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:51:25.130 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:40.245494+0000", 2024-09-20T22:51:25.131 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00066241399999999995 2024-09-20T22:51:25.131 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.131 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.131 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:51:25.131 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:40.246501+0000", 2024-09-20T22:51:25.131 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0010066299999999999 2024-09-20T22:51:25.131 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.131 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.131 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:51:25.131 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:40.248326+0000", 2024-09-20T22:51:25.131 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001824895 2024-09-20T22:51:25.131 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.131 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.132 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:51:25.132 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:40.248370+0000", 2024-09-20T22:51:25.132 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.4298999999999997e-05 2024-09-20T22:51:25.132 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.132 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.132 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:51:25.132 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:40.248445+0000", 2024-09-20T22:51:25.132 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.4810999999999999e-05 2024-09-20T22:51:25.132 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:51:25.132 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:51:25.132 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:51:25.132 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.132 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.132 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:3132 7.19 e328/139 7:9eac4560:::smithi08477203-30:head v 328'198, mlcod=322'194)", 2024-09-20T22:51:25.133 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:46:48.223406+0000", 2024-09-20T22:51:25.133 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 276.82736127499999, 2024-09-20T22:51:25.133 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.016765688000000001, 2024-09-20T22:51:25.133 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:51:25.133 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:51:25.133 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:51:25.133 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:51:25.133 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.133 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:51:25.133 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.223406+0000", 2024-09-20T22:51:25.133 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:51:25.133 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.133 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.133 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:51:25.134 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.223406+0000", 2024-09-20T22:51:25.134 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:51:25.134 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.134 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.134 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:51:25.134 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.223406+0000", 2024-09-20T22:51:25.134 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.6699999999999999e-07 2024-09-20T22:51:25.134 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.134 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.134 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:51:25.134 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.226227+0000", 2024-09-20T22:51:25.134 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0028205690000000002 2024-09-20T22:51:25.134 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.135 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.135 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:51:25.135 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.226238+0000", 2024-09-20T22:51:25.135 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0879999999999999e-05 2024-09-20T22:51:25.135 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.135 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.135 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:51:25.135 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.226249+0000", 2024-09-20T22:51:25.135 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0912e-05 2024-09-20T22:51:25.135 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.135 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.135 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:51:25.135 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.227960+0000", 2024-09-20T22:51:25.136 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0017109619999999999 2024-09-20T22:51:25.136 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.136 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.136 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:51:25.136 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.227987+0000", 2024-09-20T22:51:25.136 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.7659999999999999e-05 2024-09-20T22:51:25.136 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.136 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.136 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:51:25.137 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.240119+0000", 2024-09-20T22:51:25.137 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.012131885 2024-09-20T22:51:25.137 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.137 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.137 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:51:25.137 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.240172+0000", 2024-09-20T22:51:25.138 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.2453000000000001e-05 2024-09-20T22:51:25.138 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:51:25.138 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:51:25.138 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:51:25.138 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.138 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.138 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:3232 7.16 7:6885a65f:::smithi08477203-14:head [copy-from ver 187] snapc 7e=[7e,7d,7c,7b,7a,79,78,75,74,73,70,6b] ondisk+write+known_if_redirected+supports_pool_eio e337)", 2024-09-20T22:51:25.139 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:46:57.213186+0000", 2024-09-20T22:51:25.139 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 267.83758160399998, 2024-09-20T22:51:25.139 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.028083105000000001, 2024-09-20T22:51:25.139 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:51:25.139 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:51:25.139 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:51:25.139 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:51:25.140 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:51:25.140 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:51:25.140 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 3232 2024-09-20T22:51:25.140 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.140 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:51:25.140 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.140 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:51:25.141 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.213186+0000", 2024-09-20T22:51:25.141 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:51:25.141 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.141 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.141 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:51:25.141 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.213186+0000", 2024-09-20T22:51:25.142 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:51:25.142 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.142 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.142 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:51:25.142 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.213188+0000", 2024-09-20T22:51:25.142 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.5129999999999999e-06 2024-09-20T22:51:25.143 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.143 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.143 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:51:25.143 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.213194+0000", 2024-09-20T22:51:25.143 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.4810000000000003e-06 2024-09-20T22:51:25.143 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.144 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.144 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:51:25.144 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.213215+0000", 2024-09-20T22:51:25.144 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.0455999999999999e-05 2024-09-20T22:51:25.144 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.144 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.145 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:51:25.145 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.213225+0000", 2024-09-20T22:51:25.145 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0212e-05 2024-09-20T22:51:25.145 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.145 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.145 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:51:25.146 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.213315+0000", 2024-09-20T22:51:25.146 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.9524000000000005e-05 2024-09-20T22:51:25.146 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.146 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.146 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op must wait for map", 2024-09-20T22:51:25.146 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.213324+0000", 2024-09-20T22:51:25.147 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.8890000000000008e-06 2024-09-20T22:51:25.147 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.147 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.147 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:51:25.147 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.220517+0000", 2024-09-20T22:51:25.148 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0071931219999999997 2024-09-20T22:51:25.148 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.148 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.148 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:51:25.148 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.220636+0000", 2024-09-20T22:51:25.148 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000118875 2024-09-20T22:51:25.148 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.149 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.149 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 2,5", 2024-09-20T22:51:25.149 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.230308+0000", 2024-09-20T22:51:25.149 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0096726299999999998 2024-09-20T22:51:25.149 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.149 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.150 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:51:25.150 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.238007+0000", 2024-09-20T22:51:25.150 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.007699287 2024-09-20T22:51:25.150 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.150 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.151 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:51:25.151 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.240863+0000", 2024-09-20T22:51:25.151 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0028554880000000002 2024-09-20T22:51:25.151 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.151 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.151 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:51:25.152 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.241194+0000", 2024-09-20T22:51:25.152 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00033087399999999999 2024-09-20T22:51:25.152 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.152 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.152 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:51:25.152 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.241233+0000", 2024-09-20T22:51:25.153 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.9338000000000002e-05 2024-09-20T22:51:25.153 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.153 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.153 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:51:25.154 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.241269+0000", 2024-09-20T22:51:25.154 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.5416000000000001e-05 2024-09-20T22:51:25.154 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:51:25.154 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:51:25.154 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:51:25.155 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.155 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.155 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:3233 7.16 7:6885a65f:::smithi08477203-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e337)", 2024-09-20T22:51:25.155 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:46:57.213253+0000", 2024-09-20T22:51:25.155 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 267.83751418000003, 2024-09-20T22:51:25.155 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.023594825, 2024-09-20T22:51:25.156 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:51:25.156 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:51:25.156 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:51:25.156 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:51:25.156 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:51:25.156 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:51:25.157 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 3233 2024-09-20T22:51:25.157 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.157 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:51:25.157 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.157 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:51:25.158 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.213253+0000", 2024-09-20T22:51:25.158 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:51:25.158 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.158 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.158 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:51:25.158 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.213253+0000", 2024-09-20T22:51:25.159 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:51:25.159 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.159 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.159 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:51:25.159 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.213254+0000", 2024-09-20T22:51:25.159 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.4799999999999998e-07 2024-09-20T22:51:25.160 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.160 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.160 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:51:25.160 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.213257+0000", 2024-09-20T22:51:25.160 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.3780000000000001e-06 2024-09-20T22:51:25.160 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.161 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.161 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:51:25.161 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.213267+0000", 2024-09-20T22:51:25.161 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.9480000000000003e-06 2024-09-20T22:51:25.161 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.161 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.162 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:51:25.162 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.213273+0000", 2024-09-20T22:51:25.162 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.1240000000000002e-06 2024-09-20T22:51:25.162 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.162 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.163 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:51:25.163 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.213394+0000", 2024-09-20T22:51:25.163 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000120717 2024-09-20T22:51:25.163 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.163 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.163 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting_for_map not empty", 2024-09-20T22:51:25.164 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.213400+0000", 2024-09-20T22:51:25.164 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.0769999999999996e-06 2024-09-20T22:51:25.164 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.164 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.164 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:51:25.164 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.220851+0000", 2024-09-20T22:51:25.165 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0074511309999999997 2024-09-20T22:51:25.165 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.165 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.165 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for blocked object", 2024-09-20T22:51:25.165 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.220903+0000", 2024-09-20T22:51:25.165 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.215e-05 2024-09-20T22:51:25.166 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.166 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.166 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:51:25.166 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.236698+0000", 2024-09-20T22:51:25.166 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.015794823 2024-09-20T22:51:25.166 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.167 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.167 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:51:25.167 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.236766+0000", 2024-09-20T22:51:25.167 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.8126999999999993e-05 2024-09-20T22:51:25.167 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.167 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.168 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:51:25.168 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.236848+0000", 2024-09-20T22:51:25.168 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.1701999999999998e-05 2024-09-20T22:51:25.168 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:51:25.168 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:51:25.168 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:51:25.169 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.169 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.169 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:3671 7.5 7:a160b5d4:::smithi08477203-16:head [copy-from ver 236] snapc 91=[91,8e,8d,8c,8b,8a,88,87,86,85,84,80,7f,7a,74,73,6b] ondisk+write+known_if_redirected+supports_pool_eio e370)", 2024-09-20T22:51:25.169 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:47:30.237235+0000", 2024-09-20T22:51:25.169 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 234.81353238099999, 2024-09-20T22:51:25.169 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.019719474000000001, 2024-09-20T22:51:25.170 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:51:25.170 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:51:25.170 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:51:25.170 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:51:25.170 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:51:25.170 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:51:25.171 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 3671 2024-09-20T22:51:25.171 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.171 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:51:25.171 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.171 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:51:25.171 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:30.237235+0000", 2024-09-20T22:51:25.171 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:51:25.171 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.172 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.172 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:51:25.172 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:30.237235+0000", 2024-09-20T22:51:25.172 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:51:25.172 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.172 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.172 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:51:25.172 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:30.237236+0000", 2024-09-20T22:51:25.172 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.5570000000000001e-06 2024-09-20T22:51:25.172 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.173 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.173 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:51:25.173 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:30.237241+0000", 2024-09-20T22:51:25.173 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.3279999999999999e-06 2024-09-20T22:51:25.173 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.173 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.173 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:51:25.173 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:30.237252+0000", 2024-09-20T22:51:25.173 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1004e-05 2024-09-20T22:51:25.173 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.174 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.174 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:51:25.174 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:30.237263+0000", 2024-09-20T22:51:25.174 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1341e-05 2024-09-20T22:51:25.174 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.174 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.174 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:51:25.174 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:30.237319+0000", 2024-09-20T22:51:25.174 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.5939000000000001e-05 2024-09-20T22:51:25.175 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.175 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.175 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:51:25.175 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:30.237436+0000", 2024-09-20T22:51:25.175 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000116866 2024-09-20T22:51:25.175 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.175 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.175 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 1,2", 2024-09-20T22:51:25.175 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:30.244330+0000", 2024-09-20T22:51:25.175 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0068945619999999999 2024-09-20T22:51:25.176 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.176 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.176 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:51:25.176 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:30.249103+0000", 2024-09-20T22:51:25.176 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0047726319999999997 2024-09-20T22:51:25.176 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.176 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.176 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:51:25.176 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:30.255771+0000", 2024-09-20T22:51:25.176 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.006667666 2024-09-20T22:51:25.177 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.177 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.177 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:51:25.177 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:30.256879+0000", 2024-09-20T22:51:25.177 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0011083110000000001 2024-09-20T22:51:25.177 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.177 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.177 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:51:25.177 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:30.256919+0000", 2024-09-20T22:51:25.177 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.9589000000000003e-05 2024-09-20T22:51:25.178 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.178 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.178 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:51:25.178 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:30.256954+0000", 2024-09-20T22:51:25.178 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.5679000000000002e-05 2024-09-20T22:51:25.178 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:51:25.178 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:51:25.178 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:51:25.178 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.179 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.179 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:3747 7.c e383/94 7:3f865ef5:::smithi08477203-28:head v 383'822, mlcod=382'818)", 2024-09-20T22:51:25.179 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:47:43.267267+0000", 2024-09-20T22:51:25.179 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 221.783499917, 2024-09-20T22:51:25.179 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.014405072, 2024-09-20T22:51:25.179 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:51:25.179 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:51:25.179 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:51:25.179 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:51:25.179 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.179 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:51:25.180 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.267267+0000", 2024-09-20T22:51:25.180 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:51:25.180 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.180 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.180 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:51:25.180 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.267267+0000", 2024-09-20T22:51:25.180 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:51:25.180 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.180 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.181 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:51:25.181 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.267267+0000", 2024-09-20T22:51:25.181 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.73e-07 2024-09-20T22:51:25.181 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.181 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.181 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:51:25.181 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.269537+0000", 2024-09-20T22:51:25.181 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0022696650000000001 2024-09-20T22:51:25.181 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.181 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.181 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:51:25.182 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.269564+0000", 2024-09-20T22:51:25.182 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.6653999999999999e-05 2024-09-20T22:51:25.182 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.182 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.182 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:51:25.182 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.269575+0000", 2024-09-20T22:51:25.182 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1661999999999999e-05 2024-09-20T22:51:25.182 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.183 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.183 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:51:25.183 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.269672+0000", 2024-09-20T22:51:25.183 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.6430999999999999e-05 2024-09-20T22:51:25.183 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.183 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.184 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:51:25.184 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.269716+0000", 2024-09-20T22:51:25.184 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.4518000000000003e-05 2024-09-20T22:51:25.184 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.184 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.184 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:51:25.185 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.281609+0000", 2024-09-20T22:51:25.185 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.011892982999999999 2024-09-20T22:51:25.185 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.185 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.185 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:51:25.185 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.281672+0000", 2024-09-20T22:51:25.186 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.2985999999999996e-05 2024-09-20T22:51:25.186 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:51:25.186 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:51:25.186 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:51:25.186 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.186 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.187 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:4129 7.c e417/94 7:3f865ef5:::smithi08477203-28:head v 417'903, mlcod=417'899)", 2024-09-20T22:51:25.187 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:48:17.254548+0000", 2024-09-20T22:51:25.187 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 187.79621940199999, 2024-09-20T22:51:25.187 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.012969022, 2024-09-20T22:51:25.187 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:51:25.187 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:51:25.188 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:51:25.188 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:51:25.188 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.188 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:51:25.188 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:17.254548+0000", 2024-09-20T22:51:25.188 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:51:25.189 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.189 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.189 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:51:25.189 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:17.254548+0000", 2024-09-20T22:51:25.189 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:51:25.189 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.190 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.190 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:51:25.190 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:17.254549+0000", 2024-09-20T22:51:25.190 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.446e-06 2024-09-20T22:51:25.190 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.190 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.191 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:51:25.191 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:17.257313+0000", 2024-09-20T22:51:25.191 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0027641319999999999 2024-09-20T22:51:25.191 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.191 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.191 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:51:25.191 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:17.257333+0000", 2024-09-20T22:51:25.192 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9950999999999999e-05 2024-09-20T22:51:25.192 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.192 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.192 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:51:25.192 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:17.257343+0000", 2024-09-20T22:51:25.192 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.2750000000000001e-06 2024-09-20T22:51:25.193 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.193 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.193 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:51:25.193 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:17.257416+0000", 2024-09-20T22:51:25.193 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.3041999999999996e-05 2024-09-20T22:51:25.193 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.194 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.194 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:51:25.194 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:17.257442+0000", 2024-09-20T22:51:25.194 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.6095999999999999e-05 2024-09-20T22:51:25.194 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.194 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.194 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:51:25.195 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:17.267454+0000", 2024-09-20T22:51:25.195 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.010012455999999999 2024-09-20T22:51:25.195 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.195 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.195 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:51:25.195 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:17.267517+0000", 2024-09-20T22:51:25.196 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.2624000000000002e-05 2024-09-20T22:51:25.196 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:51:25.196 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:51:25.196 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:51:25.196 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.196 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.197 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:5412 7.6 e549/446 7:64bcd2fc:::smithi08477203-20:head v 549'293, mlcod=548'290)", 2024-09-20T22:51:25.197 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:50:29.273467+0000", 2024-09-20T22:51:25.197 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 55.777300441999998, 2024-09-20T22:51:25.197 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.012864911999999999, 2024-09-20T22:51:25.197 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:51:25.197 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:51:25.198 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:51:25.198 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:51:25.198 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.198 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:51:25.198 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:29.273467+0000", 2024-09-20T22:51:25.198 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:51:25.199 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.199 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.199 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:51:25.199 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:29.273467+0000", 2024-09-20T22:51:25.199 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:51:25.201 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:51:25.201 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.201 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.201 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:51:25.202 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:29.273469+0000", 2024-09-20T22:51:25.202 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9149999999999999e-06 2024-09-20T22:51:25.202 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.202 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.202 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:51:25.202 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:29.279342+0000", 2024-09-20T22:51:25.203 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0058736170000000002 2024-09-20T22:51:25.203 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.203 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.203 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:51:25.203 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:29.279364+0000", 2024-09-20T22:51:25.203 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.19e-05 2024-09-20T22:51:25.203 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.204 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.204 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:51:25.204 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:29.279376+0000", 2024-09-20T22:51:25.204 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1719000000000001e-05 2024-09-20T22:51:25.204 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.204 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.204 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:51:25.204 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:29.279448+0000", 2024-09-20T22:51:25.204 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.1744999999999998e-05 2024-09-20T22:51:25.204 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.204 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.204 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:51:25.204 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:29.279476+0000", 2024-09-20T22:51:25.205 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.8764000000000001e-05 2024-09-20T22:51:25.205 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.205 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.205 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:51:25.205 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:29.286276+0000", 2024-09-20T22:51:25.205 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0067999749999999998 2024-09-20T22:51:25.205 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.205 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.205 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:51:25.205 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:29.286332+0000", 2024-09-20T22:51:25.205 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.5276999999999999e-05 2024-09-20T22:51:25.205 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:51:25.205 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:51:25.205 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:51:25.206 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.206 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.206 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:6000 7.9 e604/602 7:9eac4560:::smithi08477203-30:head v 604'886, mlcod=604'882)", 2024-09-20T22:51:25.206 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:51:24.304633+0000", 2024-09-20T22:51:25.206 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 0.74613368000000002, 2024-09-20T22:51:25.206 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.014258487, 2024-09-20T22:51:25.206 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:51:25.206 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:51:25.206 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:51:25.206 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:51:25.206 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.206 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:51:25.206 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:24.304633+0000", 2024-09-20T22:51:25.207 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:51:25.207 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.207 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.207 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:51:25.207 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:24.304633+0000", 2024-09-20T22:51:25.207 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:51:25.207 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.207 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.207 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:51:25.207 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:24.304634+0000", 2024-09-20T22:51:25.207 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.9999999999999996e-07 2024-09-20T22:51:25.207 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.207 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.207 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:51:25.208 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:24.310348+0000", 2024-09-20T22:51:25.208 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0057139629999999999 2024-09-20T22:51:25.208 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.208 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.208 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:51:25.208 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:24.310377+0000", 2024-09-20T22:51:25.208 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.8544e-05 2024-09-20T22:51:25.208 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.208 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.208 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:51:25.208 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:24.310388+0000", 2024-09-20T22:51:25.208 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1259e-05 2024-09-20T22:51:25.208 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.208 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.208 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:51:25.209 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:24.310470+0000", 2024-09-20T22:51:25.209 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.1730000000000005e-05 2024-09-20T22:51:25.209 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.209 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.209 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:51:25.209 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:24.310493+0000", 2024-09-20T22:51:25.209 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.3618e-05 2024-09-20T22:51:25.209 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.209 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.209 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:51:25.209 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:24.318852+0000", 2024-09-20T22:51:25.209 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0083582840000000005 2024-09-20T22:51:25.209 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:51:25.209 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:51:25.210 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:51:25.210 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:24.318892+0000", 2024-09-20T22:51:25.210 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.0289e-05 2024-09-20T22:51:25.210 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:51:25.210 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:51:25.210 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:51:25.210 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:51:25.210 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:51:25.210 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-20T22:51:25.221 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:25.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:25.222 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:51:25.323 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:51:25.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:25.344 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:25.344 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:51:25.421 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is unset 2024-09-20T22:51:25.426 INFO:tasks.rados.rados.0.smithi084.stdout:2366: done (3 left) 2024-09-20T22:51:25.426 INFO:tasks.rados.rados.0.smithi084.stdout:2367: copy_from oid 46 from oid 21 current snap is 262 2024-09-20T22:51:25.426 INFO:tasks.rados.rados.0.smithi084.stdout:2368: rollback oid 27 current snap is 262 2024-09-20T22:51:25.426 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 27 to 253 2024-09-20T22:51:25.426 INFO:tasks.rados.rados.0.smithi084.stdout:2369: copy_from oid 29 from oid 32 current snap is 262 2024-09-20T22:51:25.426 INFO:tasks.rados.rados.0.smithi084.stdout:2370: snap_create 2024-09-20T22:51:25.426 INFO:tasks.rados.rados.0.smithi084.stdout:2363: finishing write tid 5 to smithi08477203-11 2024-09-20T22:51:25.426 INFO:tasks.rados.rados.0.smithi084.stdout:2363: oid 11 updating version 0 to 753 2024-09-20T22:51:25.426 INFO:tasks.rados.rados.0.smithi084.stdout:2363: oid 11 version 753 is already newer than 752 2024-09-20T22:51:25.426 INFO:tasks.rados.rados.0.smithi084.stdout:2363: oid 11 version 753 is already newer than 751 2024-09-20T22:51:25.426 INFO:tasks.rados.rados.0.smithi084.stdout:2363: oid 11 version 753 is already newer than 750 2024-09-20T22:51:25.426 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 11 v 753 (ObjNum 815 snap 262 seq_num 815) dirty exists 2024-09-20T22:51:25.427 INFO:tasks.rados.rados.0.smithi084.stdout:2363: left oid 11 (ObjNum 815 snap 262 seq_num 815) 2024-09-20T22:51:25.427 INFO:tasks.rados.rados.0.smithi084.stdout:2364: finishing copy_from racing read to smithi08477203-30 2024-09-20T22:51:25.427 INFO:tasks.rados.rados.0.smithi084.stdout:2364: finishing copy_from to smithi08477203-30 2024-09-20T22:51:25.427 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 30 v 745 (ObjNum 787 snap 254 seq_num 787) dirty exists 2024-09-20T22:51:25.427 INFO:tasks.rados.rados.0.smithi084.stdout:2365: finishing write tid 1 to smithi08477203-18 2024-09-20T22:51:25.427 INFO:tasks.rados.rados.0.smithi084.stdout:2365: finishing write tid 2 to smithi08477203-18 2024-09-20T22:51:25.427 INFO:tasks.rados.rados.0.smithi084.stdout:2365: finishing write tid 3 to smithi08477203-18 2024-09-20T22:51:25.427 INFO:tasks.rados.rados.0.smithi084.stdout:2365: finishing write tid 4 to smithi08477203-18 2024-09-20T22:51:25.427 INFO:tasks.rados.rados.0.smithi084.stdout:2365: finishing write tid 5 to smithi08477203-18 2024-09-20T22:51:25.427 INFO:tasks.rados.rados.0.smithi084.stdout:2365: oid 18 updating version 0 to 1230 2024-09-20T22:51:25.427 INFO:tasks.rados.rados.0.smithi084.stdout:2365: oid 18 version 1230 is already newer than 1229 2024-09-20T22:51:25.427 INFO:tasks.rados.rados.0.smithi084.stdout:2365: oid 18 updating version 1230 to 1232 2024-09-20T22:51:25.427 INFO:tasks.rados.rados.0.smithi084.stdout:2365: oid 18 version 1232 is already newer than 1231 2024-09-20T22:51:25.427 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 18 v 1232 (ObjNum 816 snap 262 seq_num 816) dirty exists 2024-09-20T22:51:25.428 INFO:tasks.rados.rados.0.smithi084.stdout:2365: left oid 18 (ObjNum 816 snap 262 seq_num 816) 2024-09-20T22:51:25.428 INFO:tasks.rados.rados.0.smithi084.stdout:2367: finishing copy_from to smithi08477203-46 2024-09-20T22:51:25.428 INFO:tasks.rados.rados.0.smithi084.stdout:2367: got expected ENOENT (src dne) 2024-09-20T22:51:25.428 INFO:tasks.rados.rados.0.smithi084.stdout:2367: finishing copy_from racing read to smithi08477203-46 2024-09-20T22:51:25.430 INFO:tasks.rados.rados.0.smithi084.stdout:2368: finishing rollback tid 0 to smithi08477203-27 2024-09-20T22:51:25.430 INFO:tasks.rados.rados.0.smithi084.stdout:2368: finishing rollback tid 1 to smithi08477203-27 2024-09-20T22:51:25.430 INFO:tasks.rados.rados.0.smithi084.stdout:2368: finishing rollback tid 2 to smithi08477203-27 2024-09-20T22:51:25.430 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 27 v 734 (ObjNum 714 snap 224 seq_num 714) dirty exists 2024-09-20T22:51:25.442 INFO:tasks.rados.rados.0.smithi084.stdout:2369: finishing copy_from racing read to smithi08477203-29 2024-09-20T22:51:25.445 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:51:25.446 INFO:tasks.rados.rados.0.smithi084.stdout:2369: finishing copy_from to smithi08477203-29 2024-09-20T22:51:25.446 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 29 v 741 (ObjNum 769 snap 248 seq_num 769) dirty exists 2024-09-20T22:51:25.467 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:25.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:25.468 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:51:25.489 INFO:tasks.thrashosds.thrasher:in_osds: [2, 3, 5, 6, 7, 0, 4] out_osds: [1] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-20T22:51:25.489 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-20T22:51:25.489 INFO:tasks.thrashosds.thrasher:inject_pause on osd.4 2024-09-20T22:51:25.490 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-20T22:51:25.490 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-20T22:51:25.490 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.4 config set filestore_inject_stall 3 2024-09-20T22:51:25.568 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:51:25.589 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:25.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:25.590 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:51:25.691 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:51:25.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:25.713 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:25.713 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:51:25.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:25 smithi154 ceph-mon[33264]: pgmap v807: 158 pgs: 4 active+clean+snaptrim, 2 active+remapped+backfill_toofull, 152 active+clean; 445 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 2.1 MiB/s rd, 1.6 MiB/s wr, 13 op/s; 32/1233 objects misplaced (2.595%); 30 MiB/s, 52 objects/s recovering 2024-09-20T22:51:25.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:25 smithi154 ceph-mon[33264]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-20T22:51:25.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:25 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/812640784' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:51:25.815 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:51:25.836 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:25.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:25.837 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:51:25.864 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:25 smithi084 ceph-mon[29774]: pgmap v807: 158 pgs: 4 active+clean+snaptrim, 2 active+remapped+backfill_toofull, 152 active+clean; 445 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 2.1 MiB/s rd, 1.6 MiB/s wr, 13 op/s; 32/1233 objects misplaced (2.595%); 30 MiB/s, 52 objects/s recovering 2024-09-20T22:51:25.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:25 smithi084 ceph-mon[29774]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-20T22:51:25.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:25 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/812640784' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:51:25.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:25 smithi084 ceph-mon[40024]: pgmap v807: 158 pgs: 4 active+clean+snaptrim, 2 active+remapped+backfill_toofull, 152 active+clean; 445 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 2.1 MiB/s rd, 1.6 MiB/s wr, 13 op/s; 32/1233 objects misplaced (2.595%); 30 MiB/s, 52 objects/s recovering 2024-09-20T22:51:25.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:25 smithi084 ceph-mon[40024]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-20T22:51:25.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:25 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/812640784' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:51:25.937 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:51:25.958 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:25.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:25.959 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:51:26.060 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:51:26.082 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:26.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:26.083 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:51:26.133 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-20T22:51:26.172 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:51:26.183 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:51:26.204 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:26.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:26.205 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:51:26.307 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:51:26.329 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:26.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:26.330 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:51:26.386 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:51:26.426 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:26 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/812640784' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T22:51:26.426 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:26 smithi084 ceph-mon[29774]: osdmap e605: 8 total, 8 up, 7 in 2024-09-20T22:51:26.426 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:26 smithi084 ceph-mon[29774]: 7.5 scrub starts 2024-09-20T22:51:26.426 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:26 smithi084 ceph-mon[29774]: 7.5 scrub ok 2024-09-20T22:51:26.430 INFO:tasks.rados.rados.0.smithi084.stdout:2363: done (6 left) 2024-09-20T22:51:26.430 INFO:tasks.rados.rados.0.smithi084.stdout:2364: done (5 left) 2024-09-20T22:51:26.430 INFO:tasks.rados.rados.0.smithi084.stdout:2365: done (4 left) 2024-09-20T22:51:26.430 INFO:tasks.rados.rados.0.smithi084.stdout:2367: done (3 left) 2024-09-20T22:51:26.430 INFO:tasks.rados.rados.0.smithi084.stdout:2368: done (2 left) 2024-09-20T22:51:26.430 INFO:tasks.rados.rados.0.smithi084.stdout:2369: done (1 left) 2024-09-20T22:51:26.430 INFO:tasks.rados.rados.0.smithi084.stdout:2370: done (0 left) 2024-09-20T22:51:26.430 INFO:tasks.rados.rados.0.smithi084.stdout:2371: copy_from oid 31 from oid 28 current snap is 263 2024-09-20T22:51:26.430 INFO:tasks.rados.rados.0.smithi084.stdout:2372: copy_from oid 43 from oid 8 current snap is 263 2024-09-20T22:51:26.430 INFO:tasks.rados.rados.0.smithi084.stdout:2373: snap_remove snap 253 2024-09-20T22:51:26.430 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:51:26.453 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:26.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:26.454 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:51:26.555 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:51:26.576 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:26.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:26.577 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:51:26.636 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-20T22:51:26.636 INFO:teuthology.orchestra.run.smithi154.stdout: "success": "" 2024-09-20T22:51:26.636 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-20T22:51:26.644 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:26 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/812640784' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T22:51:26.644 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:26 smithi154 ceph-mon[33264]: osdmap e605: 8 total, 8 up, 7 in 2024-09-20T22:51:26.644 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:26 smithi154 ceph-mon[33264]: 7.5 scrub starts 2024-09-20T22:51:26.645 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:26 smithi154 ceph-mon[33264]: 7.5 scrub ok 2024-09-20T22:51:26.645 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:26 smithi154 ceph-mon[33264]: 5.c scrub starts 2024-09-20T22:51:26.645 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:26 smithi154 ceph-mon[33264]: 5.c scrub ok 2024-09-20T22:51:26.645 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:26 smithi154 ceph-mon[33264]: 4.12 scrub starts 2024-09-20T22:51:26.645 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:26 smithi154 ceph-mon[33264]: 4.12 scrub ok 2024-09-20T22:51:26.679 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:51:26.698 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-20T22:51:26.700 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:26.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:26.702 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:51:26.720 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:26 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/812640784' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T22:51:26.720 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:26 smithi084 ceph-mon[40024]: osdmap e605: 8 total, 8 up, 7 in 2024-09-20T22:51:26.720 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:26 smithi084 ceph-mon[40024]: 7.5 scrub starts 2024-09-20T22:51:26.720 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:26 smithi084 ceph-mon[40024]: 7.5 scrub ok 2024-09-20T22:51:26.720 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:26 smithi084 ceph-mon[40024]: 5.c scrub starts 2024-09-20T22:51:26.720 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:26 smithi084 ceph-mon[40024]: 5.c scrub ok 2024-09-20T22:51:26.721 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:26 smithi084 ceph-mon[40024]: 4.12 scrub starts 2024-09-20T22:51:26.721 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:26 smithi084 ceph-mon[40024]: 4.12 scrub ok 2024-09-20T22:51:26.721 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:26 smithi084 ceph-mon[29774]: 5.c scrub starts 2024-09-20T22:51:26.721 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:26 smithi084 ceph-mon[29774]: 5.c scrub ok 2024-09-20T22:51:26.721 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:26 smithi084 ceph-mon[29774]: 4.12 scrub starts 2024-09-20T22:51:26.721 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:26 smithi084 ceph-mon[29774]: 4.12 scrub ok 2024-09-20T22:51:26.802 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:51:26.823 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:26.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:26.824 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:51:26.849 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-20T22:51:26.849 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2024-09-20T22:51:26.849 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2024-09-20T22:51:26.849 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-20T22:51:26.925 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:51:26.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:26.947 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:26.948 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:51:27.048 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:51:27.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:27.071 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:27.072 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:51:27.172 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:51:27.194 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:27.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:27.195 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:51:27.296 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:51:27.317 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:27.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:27.318 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:51:27.419 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:51:27.437 INFO:tasks.rados.rados.0.smithi084.stdout:2373: done (2 left) 2024-09-20T22:51:27.437 INFO:tasks.rados.rados.0.smithi084.stdout:2374: delete oid 33 current snap is 263 2024-09-20T22:51:27.437 INFO:tasks.rados.rados.0.smithi084.stdout:2371: finishing copy_from to smithi08477203-31 2024-09-20T22:51:27.437 INFO:tasks.rados.rados.0.smithi084.stdout:2371: got expected ENOENT (src dne) 2024-09-20T22:51:27.437 INFO:tasks.rados.rados.0.smithi084.stdout:2371: finishing copy_from racing read to smithi08477203-31 2024-09-20T22:51:27.437 INFO:tasks.rados.rados.0.smithi084.stdout:2372: finishing copy_from racing read to smithi08477203-43 2024-09-20T22:51:27.437 INFO:tasks.rados.rados.0.smithi084.stdout:2372: finishing copy_from to smithi08477203-43 2024-09-20T22:51:27.437 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 43 v 846 (ObjNum 625 snap 186 seq_num 625) dirty exists 2024-09-20T22:51:27.443 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:27.444 INFO:tasks.rados.rados.0.smithi084.stdout:2371: done (2 left) 2024-09-20T22:51:27.444 INFO:tasks.rados.rados.0.smithi084.stdout:2372: done (1 left) 2024-09-20T22:51:27.444 INFO:tasks.rados.rados.0.smithi084.stdout:2374: done (0 left) 2024-09-20T22:51:27.444 INFO:tasks.rados.rados.0.smithi084.stdout:2375: read oid 18 snap -1 2024-09-20T22:51:27.444 INFO:tasks.rados.rados.0.smithi084.stdout:2375: expect (ObjNum 816 snap 262 seq_num 816) 2024-09-20T22:51:27.444 INFO:tasks.rados.rados.0.smithi084.stdout:2376: delete oid 8 current snap is 263 2024-09-20T22:51:27.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:27.445 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:51:27.447 INFO:tasks.rados.rados.0.smithi084.stdout:2376: done (1 left) 2024-09-20T22:51:27.447 INFO:tasks.rados.rados.0.smithi084.stdout:2377: rollback oid 19 current snap is 263 2024-09-20T22:51:27.447 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 19 to 216 2024-09-20T22:51:27.447 INFO:tasks.rados.rados.0.smithi084.stdout:2378: rollback oid 33 current snap is 263 2024-09-20T22:51:27.447 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 33 to 255 2024-09-20T22:51:27.447 INFO:tasks.rados.rados.0.smithi084.stdout:2379: write oid 14 current snap is 263 2024-09-20T22:51:27.447 INFO:tasks.rados.rados.0.smithi084.stdout:2379: seq_num 819 ranges {564933=478702,1759576=488277,2991431=658955} 2024-09-20T22:51:27.453 INFO:tasks.rados.rados.0.smithi084.stdout:2379: writing smithi08477203-14 from 564933 to 1043635 tid 1 2024-09-20T22:51:27.460 INFO:tasks.rados.rados.0.smithi084.stdout:2379: writing smithi08477203-14 from 1759576 to 2247853 tid 2 2024-09-20T22:51:27.468 INFO:tasks.rados.rados.0.smithi084.stdout:2379: writing smithi08477203-14 from 2991431 to 3650386 tid 3 2024-09-20T22:51:27.468 INFO:tasks.rados.rados.0.smithi084.stdout:2380: snap_create 2024-09-20T22:51:27.468 INFO:tasks.rados.rados.0.smithi084.stdout:2378: finishing rollback tid 1 to smithi08477203-33 2024-09-20T22:51:27.468 INFO:tasks.rados.rados.0.smithi084.stdout:2378: finishing rollback tid 2 to smithi08477203-33 2024-09-20T22:51:27.468 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 33 v 760 (ObjNum 738 snap 235 seq_num 738) dirty exists 2024-09-20T22:51:27.468 INFO:tasks.rados.rados.0.smithi084.stdout:2377: finishing rollback tid 0 to smithi08477203-19 2024-09-20T22:51:27.468 INFO:tasks.rados.rados.0.smithi084.stdout:2377: finishing rollback tid 1 to smithi08477203-19 2024-09-20T22:51:27.468 INFO:tasks.rados.rados.0.smithi084.stdout:2377: finishing rollback tid 2 to smithi08477203-19 2024-09-20T22:51:27.468 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 19 v 1757 (ObjNum 656 snap 198 seq_num 656) dirty exists 2024-09-20T22:51:27.468 INFO:tasks.rados.rados.0.smithi084.stdout:2379: finishing write tid 1 to smithi08477203-14 2024-09-20T22:51:27.468 INFO:tasks.rados.rados.0.smithi084.stdout:2375: expect (ObjNum 816 snap 262 seq_num 816) 2024-09-20T22:51:27.546 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:51:27.567 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:27.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:27.570 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:51:27.596 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:27 smithi084 ceph-mon[40024]: pgmap v810: 158 pgs: 4 active+clean+snaptrim, 2 active+remapped+backfill_toofull, 152 active+clean; 445 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 2.1 MiB/s rd, 1.6 MiB/s wr, 13 op/s; 32/1233 objects misplaced (2.595%); 30 MiB/s, 52 objects/s recovering 2024-09-20T22:51:27.596 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:27 smithi084 ceph-mon[40024]: 3.f scrub starts 2024-09-20T22:51:27.596 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:27 smithi084 ceph-mon[40024]: 3.f scrub ok 2024-09-20T22:51:27.596 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:27 smithi084 ceph-mon[40024]: osdmap e606: 8 total, 8 up, 7 in 2024-09-20T22:51:27.596 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:27 smithi084 ceph-mon[40024]: 5.3 scrub starts 2024-09-20T22:51:27.597 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:27 smithi084 ceph-mon[40024]: 5.3 scrub ok 2024-09-20T22:51:27.597 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:27 smithi084 ceph-mon[40024]: purged_snaps scrub starts 2024-09-20T22:51:27.597 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:27 smithi084 ceph-mon[40024]: purged_snaps scrub ok 2024-09-20T22:51:27.597 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:27 smithi084 ceph-mon[29774]: pgmap v810: 158 pgs: 4 active+clean+snaptrim, 2 active+remapped+backfill_toofull, 152 active+clean; 445 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 2.1 MiB/s rd, 1.6 MiB/s wr, 13 op/s; 32/1233 objects misplaced (2.595%); 30 MiB/s, 52 objects/s recovering 2024-09-20T22:51:27.597 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:27 smithi084 ceph-mon[29774]: 3.f scrub starts 2024-09-20T22:51:27.597 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:27 smithi084 ceph-mon[29774]: 3.f scrub ok 2024-09-20T22:51:27.597 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:27 smithi084 ceph-mon[29774]: osdmap e606: 8 total, 8 up, 7 in 2024-09-20T22:51:27.597 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:27 smithi084 ceph-mon[29774]: 5.3 scrub starts 2024-09-20T22:51:27.597 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:27 smithi084 ceph-mon[29774]: 5.3 scrub ok 2024-09-20T22:51:27.597 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:27 smithi084 ceph-mon[29774]: purged_snaps scrub starts 2024-09-20T22:51:27.598 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:27 smithi084 ceph-mon[29774]: purged_snaps scrub ok 2024-09-20T22:51:27.622 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.0 dump_blocked_ops 2024-09-20T22:51:27.664 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:51:27.671 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:51:27.693 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:27.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:27.694 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:51:27.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:27 smithi154 ceph-mon[33264]: pgmap v810: 158 pgs: 4 active+clean+snaptrim, 2 active+remapped+backfill_toofull, 152 active+clean; 445 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 2.1 MiB/s rd, 1.6 MiB/s wr, 13 op/s; 32/1233 objects misplaced (2.595%); 30 MiB/s, 52 objects/s recovering 2024-09-20T22:51:27.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:27 smithi154 ceph-mon[33264]: 3.f scrub starts 2024-09-20T22:51:27.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:27 smithi154 ceph-mon[33264]: 3.f scrub ok 2024-09-20T22:51:27.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:27 smithi154 ceph-mon[33264]: osdmap e606: 8 total, 8 up, 7 in 2024-09-20T22:51:27.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:27 smithi154 ceph-mon[33264]: 5.3 scrub starts 2024-09-20T22:51:27.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:27 smithi154 ceph-mon[33264]: 5.3 scrub ok 2024-09-20T22:51:27.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:27 smithi154 ceph-mon[33264]: purged_snaps scrub starts 2024-09-20T22:51:27.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:27 smithi154 ceph-mon[33264]: purged_snaps scrub ok 2024-09-20T22:51:27.767 INFO:tasks.rados.rados.0.smithi084.stdout:2379: finishing write tid 2 to smithi08477203-14 2024-09-20T22:51:27.767 INFO:tasks.rados.rados.0.smithi084.stdout:2379: finishing write tid 3 to smithi08477203-14 2024-09-20T22:51:27.767 INFO:tasks.rados.rados.0.smithi084.stdout:2379: finishing write tid 4 to smithi08477203-14 2024-09-20T22:51:27.768 INFO:tasks.rados.rados.0.smithi084.stdout:2379: finishing write tid 5 to smithi08477203-14 2024-09-20T22:51:27.768 INFO:tasks.rados.rados.0.smithi084.stdout:2379: oid 14 updating version 0 to 1233 2024-09-20T22:51:27.768 INFO:tasks.rados.rados.0.smithi084.stdout:2379: oid 14 updating version 1233 to 1236 2024-09-20T22:51:27.768 INFO:tasks.rados.rados.0.smithi084.stdout:2379: oid 14 version 1236 is already newer than 1235 2024-09-20T22:51:27.768 INFO:tasks.rados.rados.0.smithi084.stdout:2379: oid 14 version 1236 is already newer than 1234 2024-09-20T22:51:27.768 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 14 v 1236 (ObjNum 819 snap 263 seq_num 819) dirty exists 2024-09-20T22:51:27.768 INFO:tasks.rados.rados.0.smithi084.stdout:2379: left oid 14 (ObjNum 819 snap 263 seq_num 819) 2024-09-20T22:51:27.795 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:51:27.817 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:27.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:27.818 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:51:27.918 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:51:27.940 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:27.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:27.941 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:51:28.042 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:51:28.063 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:28.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:28.064 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:51:28.147 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:51:28.147 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:51:28.164 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:51:28.164 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:51:28.165 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:51:28.178 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:51:28.184 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:51:28.185 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:51:28.186 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:28.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:28.187 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:51:28.201 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-20T22:51:28.202 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T22:51:28.221 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:51:28.221 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:51:28.238 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:51:28.238 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:51:28.250 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-20T22:51:28.254 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:51:28.254 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:51:28.263 INFO:tasks.rados.rados.0.smithi084.stdout:2375: done (4 left) 2024-09-20T22:51:28.263 INFO:tasks.rados.rados.0.smithi084.stdout:2377: done (3 left) 2024-09-20T22:51:28.263 INFO:tasks.rados.rados.0.smithi084.stdout:2378: done (2 left) 2024-09-20T22:51:28.263 INFO:tasks.rados.rados.0.smithi084.stdout:2379: done (1 left) 2024-09-20T22:51:28.263 INFO:tasks.rados.rados.0.smithi084.stdout:2380: done (0 left) 2024-09-20T22:51:28.263 INFO:tasks.rados.rados.0.smithi084.stdout:2381: copy_from oid 33 from oid 42 current snap is 264 2024-09-20T22:51:28.263 INFO:tasks.rados.rados.0.smithi084.stdout:2382: snap_remove snap 221 2024-09-20T22:51:28.277 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:51:28.277 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:51:28.288 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:51:28.312 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:28.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:28.313 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:51:28.414 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:51:28.435 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:51:28 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: ::ffff:172.21.15.154 - - [20/Sep/2024:22:51:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-20T22:51:28.435 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:28 smithi084 ceph-mon[40024]: osdmap e607: 8 total, 8 up, 7 in 2024-09-20T22:51:28.435 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:28 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/1241598090' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:51:28.435 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:28 smithi084 ceph-mon[40024]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:51:28.435 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:28 smithi084 ceph-mon[40024]: osdmap e608: 8 total, 8 up, 7 in 2024-09-20T22:51:28.436 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:28 smithi084 ceph-mon[29774]: osdmap e607: 8 total, 8 up, 7 in 2024-09-20T22:51:28.436 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:28 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/1241598090' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:51:28.436 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:28 smithi084 ceph-mon[29774]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:51:28.436 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:28 smithi084 ceph-mon[29774]: osdmap e608: 8 total, 8 up, 7 in 2024-09-20T22:51:28.437 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:28.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:28.438 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:51:28.539 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:51:28.561 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:28.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:28.562 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:51:28.662 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:51:28.684 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:28.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:28.685 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:51:28.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:28 smithi154 ceph-mon[33264]: osdmap e607: 8 total, 8 up, 7 in 2024-09-20T22:51:28.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:28 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/1241598090' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:51:28.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:28 smithi154 ceph-mon[33264]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:51:28.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:28 smithi154 ceph-mon[33264]: osdmap e608: 8 total, 8 up, 7 in 2024-09-20T22:51:28.786 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:51:28.807 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:28.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:28.808 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:51:28.909 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:51:28.932 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:28.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:28.933 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:51:29.034 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:51:29.056 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:29.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:29.057 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:51:29.158 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:51:29.179 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:29.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:29.180 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:51:29.272 INFO:tasks.rados.rados.0.smithi084.stdout:2382: done (1 left) 2024-09-20T22:51:29.272 INFO:tasks.rados.rados.0.smithi084.stdout:2383: delete oid 21 current snap is 264 2024-09-20T22:51:29.272 INFO:tasks.rados.rados.0.smithi084.stdout:2381: finishing copy_from racing read to smithi08477203-33 2024-09-20T22:51:29.272 INFO:tasks.rados.rados.0.smithi084.stdout:2381: finishing copy_from to smithi08477203-33 2024-09-20T22:51:29.273 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 33 v 1213 (ObjNum 775 snap 248 seq_num 775) dirty exists 2024-09-20T22:51:29.278 INFO:tasks.rados.rados.0.smithi084.stdout:2381: done (1 left) 2024-09-20T22:51:29.278 INFO:tasks.rados.rados.0.smithi084.stdout:2383: done (0 left) 2024-09-20T22:51:29.278 INFO:tasks.rados.rados.0.smithi084.stdout:2384: read oid 30 snap -1 2024-09-20T22:51:29.279 INFO:tasks.rados.rados.0.smithi084.stdout:2384: expect (ObjNum 787 snap 254 seq_num 787) 2024-09-20T22:51:29.279 INFO:tasks.rados.rados.0.smithi084.stdout:2385: delete oid 15 current snap is 264 2024-09-20T22:51:29.281 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:51:29.286 INFO:tasks.rados.rados.0.smithi084.stdout:2384: expect (ObjNum 787 snap 254 seq_num 787) 2024-09-20T22:51:29.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:29.303 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:29.304 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:51:29.404 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:51:29.416 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-20T22:51:29.416 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:51:29.427 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:29.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:29.428 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:51:29.447 INFO:tasks.rados.rados.0.smithi084.stdout:2384: done (1 left) 2024-09-20T22:51:29.447 INFO:tasks.rados.rados.0.smithi084.stdout:2385: done (0 left) 2024-09-20T22:51:29.447 INFO:tasks.rados.rados.0.smithi084.stdout:2386: read oid 42 snap -1 2024-09-20T22:51:29.447 INFO:tasks.rados.rados.0.smithi084.stdout:2386: expect (ObjNum 775 snap 248 seq_num 775) 2024-09-20T22:51:29.447 INFO:tasks.rados.rados.0.smithi084.stdout:2387: delete oid 31 current snap is 264 2024-09-20T22:51:29.454 INFO:tasks.rados.rados.0.smithi084.stdout:2387: done (1 left) 2024-09-20T22:51:29.454 INFO:tasks.rados.rados.0.smithi084.stdout:2388: read oid 13 snap -1 2024-09-20T22:51:29.454 INFO:tasks.rados.rados.0.smithi084.stdout:2388: expect (ObjNum 799 snap 255 seq_num 799) 2024-09-20T22:51:29.454 INFO:tasks.rados.rados.0.smithi084.stdout:2389: snap_create 2024-09-20T22:51:29.454 INFO:tasks.rados.rados.0.smithi084.stdout:2386: expect (ObjNum 775 snap 248 seq_num 775) 2024-09-20T22:51:29.529 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:51:29.552 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:29.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:29.553 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:51:29.586 INFO:tasks.rados.rados.0.smithi084.stdout:2388: expect (ObjNum 799 snap 255 seq_num 799) 2024-09-20T22:51:29.654 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:51:29.676 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:29.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:29.677 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:51:29.744 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:29 smithi084 ceph-mon[40024]: pgmap v813: 158 pgs: 4 active+clean+snaptrim, 2 active+remapped+backfill_toofull, 152 active+clean; 445 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 32/1233 objects misplaced (2.595%) 2024-09-20T22:51:29.745 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:29 smithi084 ceph-mon[40024]: 1.3 scrub starts 2024-09-20T22:51:29.745 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:29 smithi084 ceph-mon[40024]: 1.3 scrub ok 2024-09-20T22:51:29.745 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:29 smithi084 ceph-mon[40024]: osdmap e609: 8 total, 8 up, 7 in 2024-09-20T22:51:29.745 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:29 smithi084 ceph-mon[29774]: pgmap v813: 158 pgs: 4 active+clean+snaptrim, 2 active+remapped+backfill_toofull, 152 active+clean; 445 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 32/1233 objects misplaced (2.595%) 2024-09-20T22:51:29.745 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:29 smithi084 ceph-mon[29774]: 1.3 scrub starts 2024-09-20T22:51:29.745 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:29 smithi084 ceph-mon[29774]: 1.3 scrub ok 2024-09-20T22:51:29.745 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:29 smithi084 ceph-mon[29774]: osdmap e609: 8 total, 8 up, 7 in 2024-09-20T22:51:29.754 INFO:journalctl@ceph.iscsi.iscsi.a.smithi154.stdout:Sep 20 22:51:29 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-iscsi-iscsi-a[59730]: debug there is no tcmu-runner data available 2024-09-20T22:51:29.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:29 smithi154 ceph-mon[33264]: pgmap v813: 158 pgs: 4 active+clean+snaptrim, 2 active+remapped+backfill_toofull, 152 active+clean; 445 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 32/1233 objects misplaced (2.595%) 2024-09-20T22:51:29.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:29 smithi154 ceph-mon[33264]: 1.3 scrub starts 2024-09-20T22:51:29.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:29 smithi154 ceph-mon[33264]: 1.3 scrub ok 2024-09-20T22:51:29.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:29 smithi154 ceph-mon[33264]: osdmap e609: 8 total, 8 up, 7 in 2024-09-20T22:51:29.778 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:51:29.801 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:29.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:29.802 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:51:29.903 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:51:29.924 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:29.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:29.925 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:51:30.026 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:51:30.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:30.049 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:30.049 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:51:30.087 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.0 dump_historic_ops 2024-09-20T22:51:30.098 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:51:30.150 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:51:30.170 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:30.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:30.268 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:51:30.369 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:51:30.391 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:30.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:30.392 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:51:30.456 INFO:tasks.rados.rados.0.smithi084.stdout:2386: done (2 left) 2024-09-20T22:51:30.456 INFO:tasks.rados.rados.0.smithi084.stdout:2388: done (1 left) 2024-09-20T22:51:30.456 INFO:tasks.rados.rados.0.smithi084.stdout:2389: done (0 left) 2024-09-20T22:51:30.457 INFO:tasks.rados.rados.0.smithi084.stdout:2390: write oid 33 current snap is 265 2024-09-20T22:51:30.457 INFO:tasks.rados.rados.0.smithi084.stdout:2390: seq_num 823 ranges {632161=496051,1593504=495615,2701123=420396,3611645=220516} 2024-09-20T22:51:30.461 INFO:tasks.rados.rados.0.smithi084.stdout:2390: writing smithi08477203-33 from 632161 to 1128212 tid 1 2024-09-20T22:51:30.467 INFO:tasks.rados.rados.0.smithi084.stdout:2390: writing smithi08477203-33 from 1593504 to 2089119 tid 2 2024-09-20T22:51:30.473 INFO:tasks.rados.rados.0.smithi084.stdout:2390: writing smithi08477203-33 from 2701123 to 3121519 tid 3 2024-09-20T22:51:30.477 INFO:tasks.rados.rados.0.smithi084.stdout:2390: writing smithi08477203-33 from 3611645 to 3832161 tid 4 2024-09-20T22:51:30.477 INFO:tasks.rados.rados.0.smithi084.stdout:2391: snap_create 2024-09-20T22:51:30.477 INFO:tasks.rados.rados.0.smithi084.stdout:2390: finishing write tid 1 to smithi08477203-33 2024-09-20T22:51:30.477 INFO:tasks.rados.rados.0.smithi084.stdout:2390: finishing write tid 2 to smithi08477203-33 2024-09-20T22:51:30.477 INFO:tasks.rados.rados.0.smithi084.stdout:2390: finishing write tid 3 to smithi08477203-33 2024-09-20T22:51:30.479 INFO:tasks.rados.rados.0.smithi084.stdout:2390: finishing write tid 4 to smithi08477203-33 2024-09-20T22:51:30.479 INFO:tasks.rados.rados.0.smithi084.stdout:2390: finishing write tid 5 to smithi08477203-33 2024-09-20T22:51:30.479 INFO:tasks.rados.rados.0.smithi084.stdout:2390: finishing write tid 6 to smithi08477203-33 2024-09-20T22:51:30.479 INFO:tasks.rados.rados.0.smithi084.stdout:2390: oid 33 updating version 0 to 1217 2024-09-20T22:51:30.479 INFO:tasks.rados.rados.0.smithi084.stdout:2390: oid 33 version 1217 is already newer than 1214 2024-09-20T22:51:30.479 INFO:tasks.rados.rados.0.smithi084.stdout:2390: oid 33 version 1217 is already newer than 1215 2024-09-20T22:51:30.479 INFO:tasks.rados.rados.0.smithi084.stdout:2390: oid 33 updating version 1217 to 1218 2024-09-20T22:51:30.479 INFO:tasks.rados.rados.0.smithi084.stdout:2390: oid 33 version 1218 is already newer than 1216 2024-09-20T22:51:30.480 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 33 v 1218 (ObjNum 823 snap 265 seq_num 823) dirty exists 2024-09-20T22:51:30.480 INFO:tasks.rados.rados.0.smithi084.stdout:2390: left oid 33 (ObjNum 823 snap 265 seq_num 823) 2024-09-20T22:51:30.493 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:51:30.515 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:30.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:30.516 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:51:30.617 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:51:30.620 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:51:30.641 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:30.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:30.642 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:51:30.743 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:30 smithi084 ceph-mon[29774]: 3.8 scrub starts 2024-09-20T22:51:30.743 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:30 smithi084 ceph-mon[29774]: 3.8 scrub ok 2024-09-20T22:51:30.743 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:30 smithi084 ceph-mon[29774]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:51:30.743 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:30 smithi084 ceph-mon[29774]: 3.9 scrub starts 2024-09-20T22:51:30.743 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:30 smithi084 ceph-mon[29774]: 3.9 scrub ok 2024-09-20T22:51:30.743 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:30 smithi084 ceph-mon[29774]: 5.2 scrub starts 2024-09-20T22:51:30.743 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:30 smithi084 ceph-mon[29774]: 5.2 scrub ok 2024-09-20T22:51:30.743 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:30 smithi084 ceph-mon[40024]: 3.8 scrub starts 2024-09-20T22:51:30.743 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:30 smithi084 ceph-mon[40024]: 3.8 scrub ok 2024-09-20T22:51:30.743 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:30 smithi084 ceph-mon[40024]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:51:30.743 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:30 smithi084 ceph-mon[40024]: 3.9 scrub starts 2024-09-20T22:51:30.744 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:30 smithi084 ceph-mon[40024]: 3.9 scrub ok 2024-09-20T22:51:30.744 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:30 smithi084 ceph-mon[40024]: 5.2 scrub starts 2024-09-20T22:51:30.744 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:30 smithi084 ceph-mon[40024]: 5.2 scrub ok 2024-09-20T22:51:30.744 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:51:30.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:30 smithi154 ceph-mon[33264]: 3.8 scrub starts 2024-09-20T22:51:30.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:30 smithi154 ceph-mon[33264]: 3.8 scrub ok 2024-09-20T22:51:30.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:30 smithi154 ceph-mon[33264]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:51:30.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:30 smithi154 ceph-mon[33264]: 3.9 scrub starts 2024-09-20T22:51:30.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:30 smithi154 ceph-mon[33264]: 3.9 scrub ok 2024-09-20T22:51:30.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:30 smithi154 ceph-mon[33264]: 5.2 scrub starts 2024-09-20T22:51:30.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:30 smithi154 ceph-mon[33264]: 5.2 scrub ok 2024-09-20T22:51:30.764 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:30.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:30.765 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:51:30.865 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:51:30.905 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:30.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:30.906 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:51:31.006 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:51:31.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:31.029 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:31.029 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:51:31.086 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-20T22:51:31.086 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:51:31.130 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:51:31.156 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:31.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:31.157 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:51:31.166 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-20T22:51:31.257 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:51:31.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:31.281 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:31.281 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:51:31.381 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:51:31.405 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:31.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:31.406 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:51:31.456 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is unset 2024-09-20T22:51:31.457 INFO:tasks.rados.rados.0.smithi084.stdout:2390: done (1 left) 2024-09-20T22:51:31.457 INFO:tasks.rados.rados.0.smithi084.stdout:2391: done (0 left) 2024-09-20T22:51:31.457 INFO:tasks.rados.rados.0.smithi084.stdout:2392: snap_remove snap 232 2024-09-20T22:51:31.507 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:51:31.532 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:31.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:31.533 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:51:31.634 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:51:31.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:31.658 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:31.658 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:51:31.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:31 smithi154 ceph-mon[33264]: pgmap v816: 158 pgs: 3 active+clean+snaptrim, 2 active+remapped+backfill_toofull, 153 active+clean; 457 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail; 4.4 MiB/s rd, 4.5 MiB/s wr, 14 op/s; 32/1242 objects misplaced (2.576%) 2024-09-20T22:51:31.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:31 smithi154 ceph-mon[33264]: 6.1d scrub starts 2024-09-20T22:51:31.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:31 smithi154 ceph-mon[33264]: 6.1d scrub ok 2024-09-20T22:51:31.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:31 smithi154 ceph-mon[33264]: osdmap e610: 8 total, 8 up, 7 in 2024-09-20T22:51:31.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:31 smithi154 ceph-mon[33264]: 4.1a scrub starts 2024-09-20T22:51:31.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:31 smithi154 ceph-mon[33264]: 4.1a scrub ok 2024-09-20T22:51:31.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:31 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/4183724007' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:51:31.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:31 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:51:31.759 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:51:31.774 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:31 smithi084 ceph-mon[40024]: pgmap v816: 158 pgs: 3 active+clean+snaptrim, 2 active+remapped+backfill_toofull, 153 active+clean; 457 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail; 4.4 MiB/s rd, 4.5 MiB/s wr, 14 op/s; 32/1242 objects misplaced (2.576%) 2024-09-20T22:51:31.774 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:31 smithi084 ceph-mon[40024]: 6.1d scrub starts 2024-09-20T22:51:31.774 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:31 smithi084 ceph-mon[40024]: 6.1d scrub ok 2024-09-20T22:51:31.774 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:31 smithi084 ceph-mon[40024]: osdmap e610: 8 total, 8 up, 7 in 2024-09-20T22:51:31.774 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:31 smithi084 ceph-mon[40024]: 4.1a scrub starts 2024-09-20T22:51:31.775 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:31 smithi084 ceph-mon[40024]: 4.1a scrub ok 2024-09-20T22:51:31.775 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:31 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/4183724007' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:51:31.775 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:31 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:51:31.775 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:31 smithi084 ceph-mon[29774]: pgmap v816: 158 pgs: 3 active+clean+snaptrim, 2 active+remapped+backfill_toofull, 153 active+clean; 457 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail; 4.4 MiB/s rd, 4.5 MiB/s wr, 14 op/s; 32/1242 objects misplaced (2.576%) 2024-09-20T22:51:31.775 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:31 smithi084 ceph-mon[29774]: 6.1d scrub starts 2024-09-20T22:51:31.775 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:31 smithi084 ceph-mon[29774]: 6.1d scrub ok 2024-09-20T22:51:31.775 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:31 smithi084 ceph-mon[29774]: osdmap e610: 8 total, 8 up, 7 in 2024-09-20T22:51:31.775 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:31 smithi084 ceph-mon[29774]: 4.1a scrub starts 2024-09-20T22:51:31.775 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:31 smithi084 ceph-mon[29774]: 4.1a scrub ok 2024-09-20T22:51:31.775 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:31 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/4183724007' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:51:31.775 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:31 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:51:31.781 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:31.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:31.782 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:51:31.801 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-20T22:51:31.884 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:51:31.910 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:31.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:31.911 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:51:32.011 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:51:32.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:32.038 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:32.038 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:51:32.139 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:51:32.165 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:32.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:32.166 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:51:32.267 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:51:32.293 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:32.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:32.294 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:51:32.369 INFO:tasks.thrashosds.thrasher:in_osds: [2, 3, 5, 6, 7, 0, 4] out_osds: [1] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-20T22:51:32.369 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-20T22:51:32.369 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-09-20T22:51:32.370 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd dump -f json-pretty 2024-09-20T22:51:32.395 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:51:32.433 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:32.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:32.434 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:51:32.461 INFO:tasks.rados.rados.0.smithi084.stdout:2392: done (0 left) 2024-09-20T22:51:32.461 INFO:tasks.rados.rados.0.smithi084.stdout:2393: delete oid 22 current snap is 266 2024-09-20T22:51:32.472 INFO:tasks.rados.rados.0.smithi084.stdout:2393: done (0 left) 2024-09-20T22:51:32.472 INFO:tasks.rados.rados.0.smithi084.stdout:2394: read oid 18 snap 261 2024-09-20T22:51:32.472 INFO:tasks.rados.rados.0.smithi084.stdout:2394: expect (ObjNum 792 snap 254 seq_num 792) 2024-09-20T22:51:32.472 INFO:tasks.rados.rados.0.smithi084.stdout:2395: read oid 26 snap 243 2024-09-20T22:51:32.473 INFO:tasks.rados.rados.0.smithi084.stdout:2395: expect (ObjNum 762 snap 243 seq_num 762) 2024-09-20T22:51:32.473 INFO:tasks.rados.rados.0.smithi084.stdout:2396: read oid 31 snap -1 2024-09-20T22:51:32.473 INFO:tasks.rados.rados.0.smithi084.stdout:2396: expect deleted 2024-09-20T22:51:32.473 INFO:tasks.rados.rados.0.smithi084.stdout:2397: read oid 6 snap -1 2024-09-20T22:51:32.473 INFO:tasks.rados.rados.0.smithi084.stdout:2397: expect (ObjNum 793 snap 254 seq_num 793) 2024-09-20T22:51:32.473 INFO:tasks.rados.rados.0.smithi084.stdout:2398: copy_from oid 2 from oid 1 current snap is 266 2024-09-20T22:51:32.473 INFO:tasks.rados.rados.0.smithi084.stdout:2399: read oid 40 snap 234 2024-09-20T22:51:32.473 INFO:tasks.rados.rados.0.smithi084.stdout:2399: expect (ObjNum 440 snap 115 seq_num 440) 2024-09-20T22:51:32.473 INFO:tasks.rados.rados.0.smithi084.stdout:2400: copy_from oid 8 from oid 4 current snap is 266 2024-09-20T22:51:32.473 INFO:tasks.rados.rados.0.smithi084.stdout:2401: read oid 3 snap -1 2024-09-20T22:51:32.473 INFO:tasks.rados.rados.0.smithi084.stdout:2401: expect deleted 2024-09-20T22:51:32.473 INFO:tasks.rados.rados.0.smithi084.stdout:2402: snap_create 2024-09-20T22:51:32.480 INFO:tasks.rados.rados.0.smithi084.stdout:2397: expect (ObjNum 793 snap 254 seq_num 793) 2024-09-20T22:51:32.535 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:51:32.557 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:32.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:32.558 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:51:32.593 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:51:32.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:32 smithi084 ceph-mon[29774]: 7.3 scrub starts 2024-09-20T22:51:32.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:32 smithi084 ceph-mon[29774]: 7.3 scrub ok 2024-09-20T22:51:32.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:32 smithi084 ceph-mon[29774]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T22:51:32.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:32 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T22:51:32.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:32 smithi084 ceph-mon[29774]: osdmap e611: 8 total, 8 up, 7 in 2024-09-20T22:51:32.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:32 smithi084 ceph-mon[29774]: 6.3 scrub starts 2024-09-20T22:51:32.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:32 smithi084 ceph-mon[29774]: 6.3 scrub ok 2024-09-20T22:51:32.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:32 smithi084 ceph-mon[40024]: 7.3 scrub starts 2024-09-20T22:51:32.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:32 smithi084 ceph-mon[40024]: 7.3 scrub ok 2024-09-20T22:51:32.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:32 smithi084 ceph-mon[40024]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T22:51:32.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:32 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T22:51:32.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:32 smithi084 ceph-mon[40024]: osdmap e611: 8 total, 8 up, 7 in 2024-09-20T22:51:32.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:32 smithi084 ceph-mon[40024]: 6.3 scrub starts 2024-09-20T22:51:32.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:32 smithi084 ceph-mon[40024]: 6.3 scrub ok 2024-09-20T22:51:32.627 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:51:32.659 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:51:32.660 INFO:tasks.rados.rados.0.smithi084.stdout:2399: expect (ObjNum 440 snap 115 seq_num 440) 2024-09-20T22:51:32.680 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:32.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:32.681 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:51:32.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:32 smithi154 ceph-mon[33264]: 7.3 scrub starts 2024-09-20T22:51:32.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:32 smithi154 ceph-mon[33264]: 7.3 scrub ok 2024-09-20T22:51:32.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:32 smithi154 ceph-mon[33264]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T22:51:32.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:32 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T22:51:32.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:32 smithi154 ceph-mon[33264]: osdmap e611: 8 total, 8 up, 7 in 2024-09-20T22:51:32.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:32 smithi154 ceph-mon[33264]: 6.3 scrub starts 2024-09-20T22:51:32.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:32 smithi154 ceph-mon[33264]: 6.3 scrub ok 2024-09-20T22:51:32.782 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:51:32.804 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:32.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:32.804 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:51:32.905 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:51:32.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:32.928 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:32.928 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:51:32.928 INFO:tasks.rados.rados.0.smithi084.stdout:2395: expect (ObjNum 762 snap 243 seq_num 762) 2024-09-20T22:51:33.029 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:51:33.049 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:33.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:33.050 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:51:33.054 INFO:tasks.rados.rados.0.smithi084.stdout:2394: expect (ObjNum 792 snap 254 seq_num 792) 2024-09-20T22:51:33.152 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:51:33.173 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:33.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:33.174 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:51:33.271 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:51:33.272 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:51:33.274 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:51:33.274 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:51:33.291 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:51:33.291 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:51:33.296 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:33.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:33.297 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:51:33.310 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:51:33.310 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:51:33.319 INFO:tasks.rados.rados.0.smithi084.stdout:2400: finishing copy_from racing read to smithi08477203-8 2024-09-20T22:51:33.319 INFO:tasks.rados.rados.0.smithi084.stdout:2398: finishing copy_from racing read to smithi08477203-2 2024-09-20T22:51:33.319 INFO:tasks.rados.rados.0.smithi084.stdout:2400: finishing copy_from to smithi08477203-8 2024-09-20T22:51:33.319 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 8 v 1707 (ObjNum 810 snap 262 seq_num 810) dirty exists 2024-09-20T22:51:33.319 INFO:tasks.rados.rados.0.smithi084.stdout:2398: finishing copy_from to smithi08477203-2 2024-09-20T22:51:33.319 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 2 v 1228 (ObjNum 809 snap 262 seq_num 809) dirty exists 2024-09-20T22:51:33.326 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-20T22:51:33.327 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T22:51:33.343 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:51:33.344 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:51:33.360 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:51:33.361 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:51:33.377 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:51:33.377 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:51:33.394 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:51:33.394 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:51:33.397 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:51:33.419 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:33.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:33.420 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:51:33.464 INFO:tasks.rados.rados.0.smithi084.stdout:2394: done (8 left) 2024-09-20T22:51:33.464 INFO:tasks.rados.rados.0.smithi084.stdout:2395: done (7 left) 2024-09-20T22:51:33.465 INFO:tasks.rados.rados.0.smithi084.stdout:2396: done (6 left) 2024-09-20T22:51:33.465 INFO:tasks.rados.rados.0.smithi084.stdout:2397: done (5 left) 2024-09-20T22:51:33.465 INFO:tasks.rados.rados.0.smithi084.stdout:2398: done (4 left) 2024-09-20T22:51:33.465 INFO:tasks.rados.rados.0.smithi084.stdout:2399: done (3 left) 2024-09-20T22:51:33.465 INFO:tasks.rados.rados.0.smithi084.stdout:2400: done (2 left) 2024-09-20T22:51:33.465 INFO:tasks.rados.rados.0.smithi084.stdout:2401: done (1 left) 2024-09-20T22:51:33.465 INFO:tasks.rados.rados.0.smithi084.stdout:2402: done (0 left) 2024-09-20T22:51:33.465 INFO:tasks.rados.rados.0.smithi084.stdout:2403: read oid 28 snap 257 2024-09-20T22:51:33.465 INFO:tasks.rados.rados.0.smithi084.stdout:2403: expect deleted 2024-09-20T22:51:33.465 INFO:tasks.rados.rados.0.smithi084.stdout:2404: rollback oid 6 current snap is 267 2024-09-20T22:51:33.465 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 6 to 266 2024-09-20T22:51:33.465 INFO:tasks.rados.rados.0.smithi084.stdout:2405: write (excl) oid 29 current snap is 267 2024-09-20T22:51:33.466 INFO:tasks.rados.rados.0.smithi084.stdout:2405: seq_num 825 ranges {665775=646549,2110468=499284,3155969=442940} 2024-09-20T22:51:33.472 INFO:tasks.rados.rados.0.smithi084.stdout:2405: writing smithi08477203-29 from 665775 to 1312324 tid 1 2024-09-20T22:51:33.479 INFO:tasks.rados.rados.0.smithi084.stdout:2405: writing smithi08477203-29 from 2110468 to 2609752 tid 2 2024-09-20T22:51:33.485 INFO:tasks.rados.rados.0.smithi084.stdout:2405: writing smithi08477203-29 from 3155969 to 3598909 tid 3 2024-09-20T22:51:33.485 INFO:tasks.rados.rados.0.smithi084.stdout:2406: write oid 36 current snap is 267 2024-09-20T22:51:33.485 INFO:tasks.rados.rados.0.smithi084.stdout:2406: seq_num 826 ranges {682582=721798,2168950=642942,3583392=299190} 2024-09-20T22:51:33.490 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-20T22:51:33.490 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2024-09-20T22:51:33.490 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2024-09-20T22:51:33.490 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-20T22:51:33.493 INFO:tasks.rados.rados.0.smithi084.stdout:2406: writing smithi08477203-36 from 682582 to 1404380 tid 1 2024-09-20T22:51:33.501 INFO:tasks.rados.rados.0.smithi084.stdout:2406: writing smithi08477203-36 from 2168950 to 2811892 tid 2 2024-09-20T22:51:33.507 INFO:tasks.rados.rados.0.smithi084.stdout:2406: writing smithi08477203-36 from 3583392 to 3882582 tid 3 2024-09-20T22:51:33.507 INFO:tasks.rados.rados.0.smithi084.stdout:2407: delete oid 3 current snap is 267 2024-09-20T22:51:33.507 INFO:tasks.rados.rados.0.smithi084.stdout:2404: finishing rollback tid 0 to smithi08477203-6 2024-09-20T22:51:33.507 INFO:tasks.rados.rados.0.smithi084.stdout:2404: finishing rollback tid 1 to smithi08477203-6 2024-09-20T22:51:33.507 INFO:tasks.rados.rados.0.smithi084.stdout:2404: finishing rollback tid 2 to smithi08477203-6 2024-09-20T22:51:33.508 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 6 v 771 (ObjNum 793 snap 254 seq_num 793) dirty exists 2024-09-20T22:51:33.508 INFO:tasks.rados.rados.0.smithi084.stdout:2405: finishing write tid 1 to smithi08477203-29 2024-09-20T22:51:33.508 INFO:tasks.rados.rados.0.smithi084.stdout:2405: finishing write tid 2 to smithi08477203-29 2024-09-20T22:51:33.508 INFO:tasks.rados.rados.0.smithi084.stdout:2405: finishing write tid 3 to smithi08477203-29 2024-09-20T22:51:33.508 INFO:tasks.rados.rados.0.smithi084.stdout:2405: finishing write tid 4 to smithi08477203-29 2024-09-20T22:51:33.508 INFO:tasks.rados.rados.0.smithi084.stdout:2405: finishing write tid 5 to smithi08477203-29 2024-09-20T22:51:33.508 INFO:tasks.rados.rados.0.smithi084.stdout:2405: oid 29 updating version 0 to 1774 2024-09-20T22:51:33.508 INFO:tasks.rados.rados.0.smithi084.stdout:2405: oid 29 version 1774 is already newer than 1772 2024-09-20T22:51:33.508 INFO:tasks.rados.rados.0.smithi084.stdout:2405: oid 29 version 1774 is already newer than 1771 2024-09-20T22:51:33.508 INFO:tasks.rados.rados.0.smithi084.stdout:2405: oid 29 version 1774 is already newer than 1773 2024-09-20T22:51:33.508 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 29 v 1774 (ObjNum 825 snap 267 seq_num 825) dirty exists 2024-09-20T22:51:33.508 INFO:tasks.rados.rados.0.smithi084.stdout:2405: left oid 29 (ObjNum 825 snap 267 seq_num 825) 2024-09-20T22:51:33.508 INFO:tasks.rados.rados.0.smithi084.stdout:2406: finishing write tid 1 to smithi08477203-36 2024-09-20T22:51:33.509 INFO:tasks.rados.rados.0.smithi084.stdout:2406: finishing write tid 2 to smithi08477203-36 2024-09-20T22:51:33.509 INFO:tasks.rados.rados.0.smithi084.stdout:2403: done (4 left) 2024-09-20T22:51:33.509 INFO:tasks.rados.rados.0.smithi084.stdout:2404: done (3 left) 2024-09-20T22:51:33.509 INFO:tasks.rados.rados.0.smithi084.stdout:2405: done (2 left) 2024-09-20T22:51:33.509 INFO:tasks.rados.rados.0.smithi084.stdout:2407: done (1 left) 2024-09-20T22:51:33.509 INFO:tasks.rados.rados.0.smithi084.stdout:2408: read oid 41 snap -1 2024-09-20T22:51:33.510 INFO:tasks.rados.rados.0.smithi084.stdout:2408: expect (ObjNum 779 snap 252 seq_num 779) 2024-09-20T22:51:33.510 INFO:tasks.rados.rados.0.smithi084.stdout:2409: snap_remove snap 252 2024-09-20T22:51:33.521 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:51:33.542 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:33.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:33.543 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:51:33.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:33 smithi084 ceph-mon[40024]: pgmap v819: 158 pgs: 3 active+clean+snaptrim, 2 active+remapped+backfill_toofull, 153 active+clean; 457 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail; 4.4 MiB/s rd, 4.5 MiB/s wr, 14 op/s; 32/1242 objects misplaced (2.576%) 2024-09-20T22:51:33.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:33 smithi084 ceph-mon[40024]: 6.d scrub starts 2024-09-20T22:51:33.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:33 smithi084 ceph-mon[40024]: 6.d scrub ok 2024-09-20T22:51:33.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:33 smithi084 ceph-mon[40024]: 3.0 scrub starts 2024-09-20T22:51:33.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:33 smithi084 ceph-mon[40024]: 3.0 scrub ok 2024-09-20T22:51:33.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:33 smithi084 ceph-mon[40024]: 5.0 scrub starts 2024-09-20T22:51:33.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:33 smithi084 ceph-mon[40024]: 5.0 scrub ok 2024-09-20T22:51:33.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:33 smithi084 ceph-mon[40024]: osdmap e612: 8 total, 8 up, 7 in 2024-09-20T22:51:33.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:33 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/3349574089' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:51:33.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:33 smithi084 ceph-mon[29774]: pgmap v819: 158 pgs: 3 active+clean+snaptrim, 2 active+remapped+backfill_toofull, 153 active+clean; 457 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail; 4.4 MiB/s rd, 4.5 MiB/s wr, 14 op/s; 32/1242 objects misplaced (2.576%) 2024-09-20T22:51:33.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:33 smithi084 ceph-mon[29774]: 6.d scrub starts 2024-09-20T22:51:33.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:33 smithi084 ceph-mon[29774]: 6.d scrub ok 2024-09-20T22:51:33.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:33 smithi084 ceph-mon[29774]: 3.0 scrub starts 2024-09-20T22:51:33.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:33 smithi084 ceph-mon[29774]: 3.0 scrub ok 2024-09-20T22:51:33.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:33 smithi084 ceph-mon[29774]: 5.0 scrub starts 2024-09-20T22:51:33.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:33 smithi084 ceph-mon[29774]: 5.0 scrub ok 2024-09-20T22:51:33.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:33 smithi084 ceph-mon[29774]: osdmap e612: 8 total, 8 up, 7 in 2024-09-20T22:51:33.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:33 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/3349574089' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:51:33.644 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:51:33.665 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:33.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:33.666 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:51:33.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:33 smithi154 ceph-mon[33264]: pgmap v819: 158 pgs: 3 active+clean+snaptrim, 2 active+remapped+backfill_toofull, 153 active+clean; 457 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail; 4.4 MiB/s rd, 4.5 MiB/s wr, 14 op/s; 32/1242 objects misplaced (2.576%) 2024-09-20T22:51:33.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:33 smithi154 ceph-mon[33264]: 6.d scrub starts 2024-09-20T22:51:33.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:33 smithi154 ceph-mon[33264]: 6.d scrub ok 2024-09-20T22:51:33.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:33 smithi154 ceph-mon[33264]: 3.0 scrub starts 2024-09-20T22:51:33.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:33 smithi154 ceph-mon[33264]: 3.0 scrub ok 2024-09-20T22:51:33.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:33 smithi154 ceph-mon[33264]: 5.0 scrub starts 2024-09-20T22:51:33.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:33 smithi154 ceph-mon[33264]: 5.0 scrub ok 2024-09-20T22:51:33.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:33 smithi154 ceph-mon[33264]: osdmap e612: 8 total, 8 up, 7 in 2024-09-20T22:51:33.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:33 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/3349574089' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:51:33.767 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:51:33.789 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:33.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:33.790 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:51:33.891 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:51:33.912 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:33.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:33.913 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:51:34.013 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:51:34.036 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:34.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:34.037 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:51:34.138 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:51:34.160 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:34.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:34.161 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:51:34.261 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:51:34.283 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:34.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:34.284 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:51:34.385 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:51:34.406 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:34.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:34.407 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:51:34.424 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.1 dump_blocked_ops 2024-09-20T22:51:34.484 INFO:tasks.rados.rados.0.smithi084.stdout:2409: done (2 left) 2024-09-20T22:51:34.484 INFO:tasks.rados.rados.0.smithi084.stdout:2410: delete oid 15 current snap is 267 2024-09-20T22:51:34.484 INFO:tasks.rados.rados.0.smithi084.stdout:2406: finishing write tid 3 to smithi08477203-36 2024-09-20T22:51:34.484 INFO:tasks.rados.rados.0.smithi084.stdout:2406: finishing write tid 4 to smithi08477203-36 2024-09-20T22:51:34.484 INFO:tasks.rados.rados.0.smithi084.stdout:2406: finishing write tid 5 to smithi08477203-36 2024-09-20T22:51:34.484 INFO:tasks.rados.rados.0.smithi084.stdout:2406: oid 36 updating version 0 to 1219 2024-09-20T22:51:34.484 INFO:tasks.rados.rados.0.smithi084.stdout:2406: oid 36 updating version 1219 to 1220 2024-09-20T22:51:34.484 INFO:tasks.rados.rados.0.smithi084.stdout:2406: oid 36 version 1220 is already newer than 1218 2024-09-20T22:51:34.484 INFO:tasks.rados.rados.0.smithi084.stdout:2406: oid 36 version 1220 is already newer than 1217 2024-09-20T22:51:34.485 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 36 v 1220 (ObjNum 826 snap 267 seq_num 826) dirty exists 2024-09-20T22:51:34.485 INFO:tasks.rados.rados.0.smithi084.stdout:2406: left oid 36 (ObjNum 826 snap 267 seq_num 826) 2024-09-20T22:51:34.485 INFO:tasks.rados.rados.0.smithi084.stdout:2408: expect (ObjNum 779 snap 252 seq_num 779) 2024-09-20T22:51:34.508 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:51:34.518 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd set noscrub 2024-09-20T22:51:34.529 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:34.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:34.530 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:51:34.539 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:34 smithi084 ceph-mon[29774]: osdmap e613: 8 total, 8 up, 7 in 2024-09-20T22:51:34.539 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:34 smithi084 ceph-mon[29774]: 4.5 scrub starts 2024-09-20T22:51:34.540 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:34 smithi084 ceph-mon[29774]: 4.5 scrub ok 2024-09-20T22:51:34.540 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:34 smithi084 ceph-mon[29774]: purged_snaps scrub starts 2024-09-20T22:51:34.540 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:34 smithi084 ceph-mon[29774]: purged_snaps scrub ok 2024-09-20T22:51:34.540 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:34 smithi084 ceph-mon[40024]: osdmap e613: 8 total, 8 up, 7 in 2024-09-20T22:51:34.540 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:34 smithi084 ceph-mon[40024]: 4.5 scrub starts 2024-09-20T22:51:34.540 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:34 smithi084 ceph-mon[40024]: 4.5 scrub ok 2024-09-20T22:51:34.540 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:34 smithi084 ceph-mon[40024]: purged_snaps scrub starts 2024-09-20T22:51:34.540 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:34 smithi084 ceph-mon[40024]: purged_snaps scrub ok 2024-09-20T22:51:34.593 INFO:tasks.rados.rados.0.smithi084.stdout:2406: done (2 left) 2024-09-20T22:51:34.593 INFO:tasks.rados.rados.0.smithi084.stdout:2408: done (1 left) 2024-09-20T22:51:34.593 INFO:tasks.rados.rados.0.smithi084.stdout:2410: done (0 left) 2024-09-20T22:51:34.593 INFO:tasks.rados.rados.0.smithi084.stdout:2411: snap_create 2024-09-20T22:51:34.631 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:51:34.651 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:34.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:34.652 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:51:34.753 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:51:34.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:34 smithi154 ceph-mon[33264]: osdmap e613: 8 total, 8 up, 7 in 2024-09-20T22:51:34.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:34 smithi154 ceph-mon[33264]: 4.5 scrub starts 2024-09-20T22:51:34.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:34 smithi154 ceph-mon[33264]: 4.5 scrub ok 2024-09-20T22:51:34.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:34 smithi154 ceph-mon[33264]: purged_snaps scrub starts 2024-09-20T22:51:34.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:34 smithi154 ceph-mon[33264]: purged_snaps scrub ok 2024-09-20T22:51:34.774 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:34.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:34.775 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:51:34.876 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:51:34.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:34.898 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:34.898 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:51:34.999 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:51:35.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:35.023 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:35.023 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:51:35.123 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:51:35.145 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:35.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:35.146 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:51:35.247 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:51:35.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:35.269 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:35.269 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:51:35.370 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:51:35.392 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:35.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:35.394 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:51:35.480 INFO:tasks.rados.rados.0.smithi084.stdout:2411: done (0 left) 2024-09-20T22:51:35.480 INFO:tasks.rados.rados.0.smithi084.stdout:2412: write oid 17 current snap is 268 2024-09-20T22:51:35.480 INFO:tasks.rados.rados.0.smithi084.stdout:2412: seq_num 829 ranges {733003=663898,1944396=506622,2865661=604381} 2024-09-20T22:51:35.487 INFO:tasks.rados.rados.0.smithi084.stdout:2412: writing smithi08477203-17 from 733003 to 1396901 tid 1 2024-09-20T22:51:35.493 INFO:tasks.rados.rados.0.smithi084.stdout:2412: writing smithi08477203-17 from 1944396 to 2451018 tid 2 2024-09-20T22:51:35.495 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:51:35.500 INFO:tasks.rados.rados.0.smithi084.stdout:2412: writing smithi08477203-17 from 2865661 to 3470042 tid 3 2024-09-20T22:51:35.500 INFO:tasks.rados.rados.0.smithi084.stdout:2413: rollback oid 10 current snap is 268 2024-09-20T22:51:35.500 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 10 to 213 2024-09-20T22:51:35.500 INFO:tasks.rados.rados.0.smithi084.stdout:2412: finishing write tid 1 to smithi08477203-17 2024-09-20T22:51:35.500 INFO:tasks.rados.rados.0.smithi084.stdout:2412: finishing write tid 2 to smithi08477203-17 2024-09-20T22:51:35.500 INFO:tasks.rados.rados.0.smithi084.stdout:2414: read oid 35 snap -1 2024-09-20T22:51:35.500 INFO:tasks.rados.rados.0.smithi084.stdout:2414: expect (ObjNum 787 snap 254 seq_num 787) 2024-09-20T22:51:35.500 INFO:tasks.rados.rados.0.smithi084.stdout:2415: snap_remove snap 216 2024-09-20T22:51:35.516 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:35.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:35.517 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:51:35.618 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:51:35.637 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:51:35.638 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:51:35.639 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:35.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:35.640 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:51:35.692 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:35 smithi084 ceph-mon[40024]: pgmap v822: 158 pgs: 4 active+clean+snaptrim, 2 active+remapped+backfill_toofull, 152 active+clean; 456 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 25 MiB/s rd, 2.9 MiB/s wr, 21 op/s; 33/1245 objects misplaced (2.651%) 2024-09-20T22:51:35.692 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:35 smithi084 ceph-mon[40024]: 5.1b scrub starts 2024-09-20T22:51:35.693 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:35 smithi084 ceph-mon[40024]: 5.1b scrub ok 2024-09-20T22:51:35.693 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:35 smithi084 ceph-mon[40024]: osdmap e614: 8 total, 8 up, 7 in 2024-09-20T22:51:35.693 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:35 smithi084 ceph-mon[40024]: 3.2 scrub starts 2024-09-20T22:51:35.693 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:35 smithi084 ceph-mon[40024]: 3.2 scrub ok 2024-09-20T22:51:35.693 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:35 smithi084 ceph-mon[29774]: pgmap v822: 158 pgs: 4 active+clean+snaptrim, 2 active+remapped+backfill_toofull, 152 active+clean; 456 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 25 MiB/s rd, 2.9 MiB/s wr, 21 op/s; 33/1245 objects misplaced (2.651%) 2024-09-20T22:51:35.693 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:35 smithi084 ceph-mon[29774]: 5.1b scrub starts 2024-09-20T22:51:35.693 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:35 smithi084 ceph-mon[29774]: 5.1b scrub ok 2024-09-20T22:51:35.693 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:35 smithi084 ceph-mon[29774]: osdmap e614: 8 total, 8 up, 7 in 2024-09-20T22:51:35.693 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:35 smithi084 ceph-mon[29774]: 3.2 scrub starts 2024-09-20T22:51:35.693 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:35 smithi084 ceph-mon[29774]: 3.2 scrub ok 2024-09-20T22:51:35.694 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-20T22:51:35.694 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-20T22:51:35.694 INFO:teuthology.orchestra.run.smithi084.stdout: "epoch": 615, 2024-09-20T22:51:35.694 INFO:teuthology.orchestra.run.smithi084.stdout: "fsid": "a9c0bc70-77a0-11ef-baf5-efdc52797490", 2024-09-20T22:51:35.694 INFO:teuthology.orchestra.run.smithi084.stdout: "created": "2024-09-20T22:36:30.240626+0000", 2024-09-20T22:51:35.694 INFO:teuthology.orchestra.run.smithi084.stdout: "modified": "2024-09-20T22:51:35.471275+0000", 2024-09-20T22:51:35.694 INFO:teuthology.orchestra.run.smithi084.stdout: "last_up_change": "2024-09-20T22:41:27.098565+0000", 2024-09-20T22:51:35.694 INFO:teuthology.orchestra.run.smithi084.stdout: "last_in_change": "2024-09-20T22:51:19.253681+0000", 2024-09-20T22:51:35.694 INFO:teuthology.orchestra.run.smithi084.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-20T22:51:35.694 INFO:teuthology.orchestra.run.smithi084.stdout: "flags_num": 5799936, 2024-09-20T22:51:35.694 INFO:teuthology.orchestra.run.smithi084.stdout: "flags_set": [ 2024-09-20T22:51:35.695 INFO:teuthology.orchestra.run.smithi084.stdout: "pglog_hardlimit", 2024-09-20T22:51:35.695 INFO:teuthology.orchestra.run.smithi084.stdout: "purged_snapdirs", 2024-09-20T22:51:35.695 INFO:teuthology.orchestra.run.smithi084.stdout: "recovery_deletes", 2024-09-20T22:51:35.695 INFO:teuthology.orchestra.run.smithi084.stdout: "sortbitwise" 2024-09-20T22:51:35.695 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2024-09-20T22:51:35.695 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_version": 18, 2024-09-20T22:51:35.695 INFO:teuthology.orchestra.run.smithi084.stdout: "full_ratio": 0.94999998807907104, 2024-09-20T22:51:35.695 INFO:teuthology.orchestra.run.smithi084.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-20T22:51:35.695 INFO:teuthology.orchestra.run.smithi084.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-20T22:51:35.695 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_snapshot": "", 2024-09-20T22:51:35.695 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_max": 7, 2024-09-20T22:51:35.695 INFO:teuthology.orchestra.run.smithi084.stdout: "max_osd": 8, 2024-09-20T22:51:35.695 INFO:teuthology.orchestra.run.smithi084.stdout: "require_min_compat_client": "luminous", 2024-09-20T22:51:35.695 INFO:teuthology.orchestra.run.smithi084.stdout: "min_compat_client": "luminous", 2024-09-20T22:51:35.695 INFO:teuthology.orchestra.run.smithi084.stdout: "require_osd_release": "squid", 2024-09-20T22:51:35.696 INFO:teuthology.orchestra.run.smithi084.stdout: "allow_crimson": false, 2024-09-20T22:51:35.696 INFO:teuthology.orchestra.run.smithi084.stdout: "pools": [ 2024-09-20T22:51:35.696 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:35.696 INFO:teuthology.orchestra.run.smithi084.stdout: "pool": 1, 2024-09-20T22:51:35.696 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_name": ".mgr", 2024-09-20T22:51:35.696 INFO:teuthology.orchestra.run.smithi084.stdout: "create_time": "2024-09-20T22:39:21.984381+0000", 2024-09-20T22:51:35.696 INFO:teuthology.orchestra.run.smithi084.stdout: "flags": 1, 2024-09-20T22:51:35.696 INFO:teuthology.orchestra.run.smithi084.stdout: "flags_names": "hashpspool", 2024-09-20T22:51:35.696 INFO:teuthology.orchestra.run.smithi084.stdout: "type": 1, 2024-09-20T22:51:35.696 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 3, 2024-09-20T22:51:35.696 INFO:teuthology.orchestra.run.smithi084.stdout: "min_size": 2, 2024-09-20T22:51:35.696 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_rule": 0, 2024-09-20T22:51:35.696 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_count": 0, 2024-09-20T22:51:35.696 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_target": 0, 2024-09-20T22:51:35.696 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_barrier": 0, 2024-09-20T22:51:35.696 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-20T22:51:35.697 INFO:teuthology.orchestra.run.smithi084.stdout: "is_stretch_pool": false, 2024-09-20T22:51:35.697 INFO:teuthology.orchestra.run.smithi084.stdout: "object_hash": 2, 2024-09-20T22:51:35.697 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_autoscale_mode": "off", 2024-09-20T22:51:35.697 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num": 11, 2024-09-20T22:51:35.697 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num": 11, 2024-09-20T22:51:35.697 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num_target": 11, 2024-09-20T22:51:35.697 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_target": 11, 2024-09-20T22:51:35.697 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_pending": 11, 2024-09-20T22:51:35.697 INFO:teuthology.orchestra.run.smithi084.stdout: "last_pg_merge_meta": { 2024-09-20T22:51:35.697 INFO:teuthology.orchestra.run.smithi084.stdout: "source_pgid": "1.b", 2024-09-20T22:51:35.697 INFO:teuthology.orchestra.run.smithi084.stdout: "ready_epoch": 299, 2024-09-20T22:51:35.697 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_started": 298, 2024-09-20T22:51:35.697 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_clean": 298, 2024-09-20T22:51:35.697 INFO:teuthology.orchestra.run.smithi084.stdout: "source_version": "77'128", 2024-09-20T22:51:35.697 INFO:teuthology.orchestra.run.smithi084.stdout: "target_version": "77'128" 2024-09-20T22:51:35.698 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:35.698 INFO:teuthology.orchestra.run.smithi084.stdout: "last_change": "542", 2024-09-20T22:51:35.698 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend": "0", 2024-09-20T22:51:35.698 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_prenautilus": "299", 2024-09-20T22:51:35.698 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_preluminous": "297", 2024-09-20T22:51:35.698 INFO:teuthology.orchestra.run.smithi084.stdout: "auid": 0, 2024-09-20T22:51:35.698 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_mode": "selfmanaged", 2024-09-20T22:51:35.698 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_seq": 0, 2024-09-20T22:51:35.698 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_epoch": 0, 2024-09-20T22:51:35.698 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_snaps": [], 2024-09-20T22:51:35.698 INFO:teuthology.orchestra.run.smithi084.stdout: "removed_snaps": "[]", 2024-09-20T22:51:35.698 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_bytes": 0, 2024-09-20T22:51:35.698 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_objects": 0, 2024-09-20T22:51:35.698 INFO:teuthology.orchestra.run.smithi084.stdout: "tiers": [], 2024-09-20T22:51:35.698 INFO:teuthology.orchestra.run.smithi084.stdout: "tier_of": -1, 2024-09-20T22:51:35.699 INFO:teuthology.orchestra.run.smithi084.stdout: "read_tier": -1, 2024-09-20T22:51:35.699 INFO:teuthology.orchestra.run.smithi084.stdout: "write_tier": -1, 2024-09-20T22:51:35.699 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_mode": "none", 2024-09-20T22:51:35.699 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_bytes": 0, 2024-09-20T22:51:35.699 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_objects": 0, 2024-09-20T22:51:35.699 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-20T22:51:35.699 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-20T22:51:35.699 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-20T22:51:35.699 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_flush_age": 0, 2024-09-20T22:51:35.699 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_evict_age": 0, 2024-09-20T22:51:35.699 INFO:teuthology.orchestra.run.smithi084.stdout: "erasure_code_profile": "", 2024-09-20T22:51:35.699 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_params": { 2024-09-20T22:51:35.699 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "none" 2024-09-20T22:51:35.700 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:35.700 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_period": 0, 2024-09-20T22:51:35.700 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_count": 0, 2024-09-20T22:51:35.700 INFO:teuthology.orchestra.run.smithi084.stdout: "use_gmt_hitset": true, 2024-09-20T22:51:35.700 INFO:teuthology.orchestra.run.smithi084.stdout: "min_read_recency_for_promote": 0, 2024-09-20T22:51:35.700 INFO:teuthology.orchestra.run.smithi084.stdout: "min_write_recency_for_promote": 0, 2024-09-20T22:51:35.700 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_grade_decay_rate": 0, 2024-09-20T22:51:35.700 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_search_last_n": 0, 2024-09-20T22:51:35.700 INFO:teuthology.orchestra.run.smithi084.stdout: "grade_table": [], 2024-09-20T22:51:35.700 INFO:teuthology.orchestra.run.smithi084.stdout: "stripe_width": 0, 2024-09-20T22:51:35.700 INFO:teuthology.orchestra.run.smithi084.stdout: "expected_num_objects": 0, 2024-09-20T22:51:35.700 INFO:teuthology.orchestra.run.smithi084.stdout: "fast_read": false, 2024-09-20T22:51:35.700 INFO:teuthology.orchestra.run.smithi084.stdout: "options": { 2024-09-20T22:51:35.700 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_max": 32, 2024-09-20T22:51:35.701 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_min": 1 2024-09-20T22:51:35.701 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:35.701 INFO:teuthology.orchestra.run.smithi084.stdout: "application_metadata": { 2024-09-20T22:51:35.701 INFO:teuthology.orchestra.run.smithi084.stdout: "mgr": {} 2024-09-20T22:51:35.701 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:35.701 INFO:teuthology.orchestra.run.smithi084.stdout: "read_balance": { 2024-09-20T22:51:35.701 INFO:teuthology.orchestra.run.smithi084.stdout: "score_type": "Fair distribution", 2024-09-20T22:51:35.701 INFO:teuthology.orchestra.run.smithi084.stdout: "score_acting": 2.1700000762939453, 2024-09-20T22:51:35.701 INFO:teuthology.orchestra.run.smithi084.stdout: "score_stable": 2.1700000762939453, 2024-09-20T22:51:35.701 INFO:teuthology.orchestra.run.smithi084.stdout: "optimal_score": 0.87999999523162842, 2024-09-20T22:51:35.701 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_acting": 1.9099999666213989, 2024-09-20T22:51:35.701 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_stable": 1.9099999666213989, 2024-09-20T22:51:35.701 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity_weighted": 1, 2024-09-20T22:51:35.702 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity": 1, 2024-09-20T22:51:35.702 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity_weighted": 1 2024-09-20T22:51:35.702 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:51:35.702 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:35.702 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:35.702 INFO:teuthology.orchestra.run.smithi084.stdout: "pool": 2, 2024-09-20T22:51:35.702 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_name": "datapool", 2024-09-20T22:51:35.702 INFO:teuthology.orchestra.run.smithi084.stdout: "create_time": "2024-09-20T22:41:35.820081+0000", 2024-09-20T22:51:35.702 INFO:teuthology.orchestra.run.smithi084.stdout: "flags": 8193, 2024-09-20T22:51:35.702 INFO:teuthology.orchestra.run.smithi084.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-20T22:51:35.702 INFO:teuthology.orchestra.run.smithi084.stdout: "type": 1, 2024-09-20T22:51:35.702 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 3, 2024-09-20T22:51:35.703 INFO:teuthology.orchestra.run.smithi084.stdout: "min_size": 2, 2024-09-20T22:51:35.703 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_rule": 0, 2024-09-20T22:51:35.703 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_count": 0, 2024-09-20T22:51:35.703 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_target": 0, 2024-09-20T22:51:35.703 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_barrier": 0, 2024-09-20T22:51:35.703 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-20T22:51:35.703 INFO:teuthology.orchestra.run.smithi084.stdout: "is_stretch_pool": false, 2024-09-20T22:51:35.703 INFO:teuthology.orchestra.run.smithi084.stdout: "object_hash": 2, 2024-09-20T22:51:35.703 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_autoscale_mode": "off", 2024-09-20T22:51:35.703 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num": 3, 2024-09-20T22:51:35.703 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num": 3, 2024-09-20T22:51:35.703 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num_target": 3, 2024-09-20T22:51:35.703 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_target": 3, 2024-09-20T22:51:35.703 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_pending": 3, 2024-09-20T22:51:35.703 INFO:teuthology.orchestra.run.smithi084.stdout: "last_pg_merge_meta": { 2024-09-20T22:51:35.703 INFO:teuthology.orchestra.run.smithi084.stdout: "source_pgid": "0.0", 2024-09-20T22:51:35.704 INFO:teuthology.orchestra.run.smithi084.stdout: "ready_epoch": 0, 2024-09-20T22:51:35.704 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_started": 0, 2024-09-20T22:51:35.704 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_clean": 0, 2024-09-20T22:51:35.704 INFO:teuthology.orchestra.run.smithi084.stdout: "source_version": "0'0", 2024-09-20T22:51:35.704 INFO:teuthology.orchestra.run.smithi084.stdout: "target_version": "0'0" 2024-09-20T22:51:35.704 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:35.704 INFO:teuthology.orchestra.run.smithi084.stdout: "last_change": "69", 2024-09-20T22:51:35.704 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend": "0", 2024-09-20T22:51:35.704 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-20T22:51:35.704 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_preluminous": "0", 2024-09-20T22:51:35.704 INFO:teuthology.orchestra.run.smithi084.stdout: "auid": 0, 2024-09-20T22:51:35.704 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_mode": "selfmanaged", 2024-09-20T22:51:35.704 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_seq": 2, 2024-09-20T22:51:35.704 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_epoch": 69, 2024-09-20T22:51:35.704 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_snaps": [], 2024-09-20T22:51:35.705 INFO:teuthology.orchestra.run.smithi084.stdout: "removed_snaps": "[]", 2024-09-20T22:51:35.705 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_bytes": 0, 2024-09-20T22:51:35.705 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_objects": 0, 2024-09-20T22:51:35.705 INFO:teuthology.orchestra.run.smithi084.stdout: "tiers": [], 2024-09-20T22:51:35.705 INFO:teuthology.orchestra.run.smithi084.stdout: "tier_of": -1, 2024-09-20T22:51:35.705 INFO:teuthology.orchestra.run.smithi084.stdout: "read_tier": -1, 2024-09-20T22:51:35.705 INFO:teuthology.orchestra.run.smithi084.stdout: "write_tier": -1, 2024-09-20T22:51:35.705 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_mode": "none", 2024-09-20T22:51:35.705 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_bytes": 0, 2024-09-20T22:51:35.705 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_objects": 0, 2024-09-20T22:51:35.705 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-20T22:51:35.705 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-20T22:51:35.705 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-20T22:51:35.705 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_flush_age": 0, 2024-09-20T22:51:35.705 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_evict_age": 0, 2024-09-20T22:51:35.706 INFO:teuthology.orchestra.run.smithi084.stdout: "erasure_code_profile": "", 2024-09-20T22:51:35.706 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_params": { 2024-09-20T22:51:35.706 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "none" 2024-09-20T22:51:35.706 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:35.706 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_period": 0, 2024-09-20T22:51:35.706 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_count": 0, 2024-09-20T22:51:35.706 INFO:teuthology.orchestra.run.smithi084.stdout: "use_gmt_hitset": true, 2024-09-20T22:51:35.706 INFO:teuthology.orchestra.run.smithi084.stdout: "min_read_recency_for_promote": 0, 2024-09-20T22:51:35.706 INFO:teuthology.orchestra.run.smithi084.stdout: "min_write_recency_for_promote": 0, 2024-09-20T22:51:35.706 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_grade_decay_rate": 0, 2024-09-20T22:51:35.706 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_search_last_n": 0, 2024-09-20T22:51:35.706 INFO:teuthology.orchestra.run.smithi084.stdout: "grade_table": [], 2024-09-20T22:51:35.706 INFO:teuthology.orchestra.run.smithi084.stdout: "stripe_width": 0, 2024-09-20T22:51:35.706 INFO:teuthology.orchestra.run.smithi084.stdout: "expected_num_objects": 0, 2024-09-20T22:51:35.706 INFO:teuthology.orchestra.run.smithi084.stdout: "fast_read": false, 2024-09-20T22:51:35.707 INFO:teuthology.orchestra.run.smithi084.stdout: "options": {}, 2024-09-20T22:51:35.707 INFO:teuthology.orchestra.run.smithi084.stdout: "application_metadata": { 2024-09-20T22:51:35.707 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd": {} 2024-09-20T22:51:35.707 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:35.707 INFO:teuthology.orchestra.run.smithi084.stdout: "read_balance": { 2024-09-20T22:51:35.707 INFO:teuthology.orchestra.run.smithi084.stdout: "score_type": "Fair distribution", 2024-09-20T22:51:35.707 INFO:teuthology.orchestra.run.smithi084.stdout: "score_acting": 2.6700000762939453, 2024-09-20T22:51:35.707 INFO:teuthology.orchestra.run.smithi084.stdout: "score_stable": 2.6700000762939453, 2024-09-20T22:51:35.707 INFO:teuthology.orchestra.run.smithi084.stdout: "optimal_score": 0.75, 2024-09-20T22:51:35.707 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_acting": 2, 2024-09-20T22:51:35.707 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_stable": 2, 2024-09-20T22:51:35.707 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity_weighted": 1, 2024-09-20T22:51:35.707 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity": 1, 2024-09-20T22:51:35.708 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity_weighted": 1 2024-09-20T22:51:35.708 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:51:35.708 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:35.708 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:35.708 INFO:teuthology.orchestra.run.smithi084.stdout: "pool": 3, 2024-09-20T22:51:35.708 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_name": ".rgw.root", 2024-09-20T22:51:35.708 INFO:teuthology.orchestra.run.smithi084.stdout: "create_time": "2024-09-20T22:41:36.546389+0000", 2024-09-20T22:51:35.708 INFO:teuthology.orchestra.run.smithi084.stdout: "flags": 1, 2024-09-20T22:51:35.708 INFO:teuthology.orchestra.run.smithi084.stdout: "flags_names": "hashpspool", 2024-09-20T22:51:35.708 INFO:teuthology.orchestra.run.smithi084.stdout: "type": 1, 2024-09-20T22:51:35.708 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 3, 2024-09-20T22:51:35.708 INFO:teuthology.orchestra.run.smithi084.stdout: "min_size": 2, 2024-09-20T22:51:35.708 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_rule": 0, 2024-09-20T22:51:35.709 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_count": 0, 2024-09-20T22:51:35.709 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_target": 0, 2024-09-20T22:51:35.709 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_barrier": 0, 2024-09-20T22:51:35.709 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-20T22:51:35.709 INFO:teuthology.orchestra.run.smithi084.stdout: "is_stretch_pool": false, 2024-09-20T22:51:35.709 INFO:teuthology.orchestra.run.smithi084.stdout: "object_hash": 2, 2024-09-20T22:51:35.709 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_autoscale_mode": "off", 2024-09-20T22:51:35.709 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num": 32, 2024-09-20T22:51:35.709 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num": 32, 2024-09-20T22:51:35.709 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num_target": 32, 2024-09-20T22:51:35.709 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_target": 32, 2024-09-20T22:51:35.709 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_pending": 32, 2024-09-20T22:51:35.709 INFO:teuthology.orchestra.run.smithi084.stdout: "last_pg_merge_meta": { 2024-09-20T22:51:35.709 INFO:teuthology.orchestra.run.smithi084.stdout: "source_pgid": "0.0", 2024-09-20T22:51:35.710 INFO:teuthology.orchestra.run.smithi084.stdout: "ready_epoch": 0, 2024-09-20T22:51:35.710 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_started": 0, 2024-09-20T22:51:35.710 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_clean": 0, 2024-09-20T22:51:35.710 INFO:teuthology.orchestra.run.smithi084.stdout: "source_version": "0'0", 2024-09-20T22:51:35.710 INFO:teuthology.orchestra.run.smithi084.stdout: "target_version": "0'0" 2024-09-20T22:51:35.710 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:35.710 INFO:teuthology.orchestra.run.smithi084.stdout: "last_change": "68", 2024-09-20T22:51:35.710 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend": "0", 2024-09-20T22:51:35.710 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-20T22:51:35.710 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_preluminous": "0", 2024-09-20T22:51:35.710 INFO:teuthology.orchestra.run.smithi084.stdout: "auid": 0, 2024-09-20T22:51:35.710 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_mode": "selfmanaged", 2024-09-20T22:51:35.710 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_seq": 0, 2024-09-20T22:51:35.710 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_epoch": 0, 2024-09-20T22:51:35.711 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_snaps": [], 2024-09-20T22:51:35.711 INFO:teuthology.orchestra.run.smithi084.stdout: "removed_snaps": "[]", 2024-09-20T22:51:35.711 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_bytes": 0, 2024-09-20T22:51:35.711 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_objects": 0, 2024-09-20T22:51:35.711 INFO:teuthology.orchestra.run.smithi084.stdout: "tiers": [], 2024-09-20T22:51:35.711 INFO:teuthology.orchestra.run.smithi084.stdout: "tier_of": -1, 2024-09-20T22:51:35.711 INFO:teuthology.orchestra.run.smithi084.stdout: "read_tier": -1, 2024-09-20T22:51:35.711 INFO:teuthology.orchestra.run.smithi084.stdout: "write_tier": -1, 2024-09-20T22:51:35.711 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_mode": "none", 2024-09-20T22:51:35.711 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_bytes": 0, 2024-09-20T22:51:35.711 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_objects": 0, 2024-09-20T22:51:35.711 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-20T22:51:35.711 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-20T22:51:35.711 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-20T22:51:35.712 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_flush_age": 0, 2024-09-20T22:51:35.712 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_evict_age": 0, 2024-09-20T22:51:35.712 INFO:teuthology.orchestra.run.smithi084.stdout: "erasure_code_profile": "", 2024-09-20T22:51:35.712 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_params": { 2024-09-20T22:51:35.712 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "none" 2024-09-20T22:51:35.712 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:35.712 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_period": 0, 2024-09-20T22:51:35.712 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_count": 0, 2024-09-20T22:51:35.712 INFO:teuthology.orchestra.run.smithi084.stdout: "use_gmt_hitset": true, 2024-09-20T22:51:35.712 INFO:teuthology.orchestra.run.smithi084.stdout: "min_read_recency_for_promote": 0, 2024-09-20T22:51:35.712 INFO:teuthology.orchestra.run.smithi084.stdout: "min_write_recency_for_promote": 0, 2024-09-20T22:51:35.712 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_grade_decay_rate": 0, 2024-09-20T22:51:35.712 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_search_last_n": 0, 2024-09-20T22:51:35.712 INFO:teuthology.orchestra.run.smithi084.stdout: "grade_table": [], 2024-09-20T22:51:35.712 INFO:teuthology.orchestra.run.smithi084.stdout: "stripe_width": 0, 2024-09-20T22:51:35.713 INFO:teuthology.orchestra.run.smithi084.stdout: "expected_num_objects": 0, 2024-09-20T22:51:35.713 INFO:teuthology.orchestra.run.smithi084.stdout: "fast_read": false, 2024-09-20T22:51:35.713 INFO:teuthology.orchestra.run.smithi084.stdout: "options": {}, 2024-09-20T22:51:35.713 INFO:teuthology.orchestra.run.smithi084.stdout: "application_metadata": { 2024-09-20T22:51:35.713 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw": {} 2024-09-20T22:51:35.713 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:35.713 INFO:teuthology.orchestra.run.smithi084.stdout: "read_balance": { 2024-09-20T22:51:35.713 INFO:teuthology.orchestra.run.smithi084.stdout: "score_type": "Fair distribution", 2024-09-20T22:51:35.713 INFO:teuthology.orchestra.run.smithi084.stdout: "score_acting": 1.7400000095367432, 2024-09-20T22:51:35.713 INFO:teuthology.orchestra.run.smithi084.stdout: "score_stable": 1.7400000095367432, 2024-09-20T22:51:35.713 INFO:teuthology.orchestra.run.smithi084.stdout: "optimal_score": 0.87999999523162842, 2024-09-20T22:51:35.713 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_acting": 1.5299999713897705, 2024-09-20T22:51:35.713 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_stable": 1.5299999713897705, 2024-09-20T22:51:35.713 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity_weighted": 1, 2024-09-20T22:51:35.713 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity": 1, 2024-09-20T22:51:35.714 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity_weighted": 1 2024-09-20T22:51:35.714 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:51:35.714 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:35.714 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:35.714 INFO:teuthology.orchestra.run.smithi084.stdout: "pool": 4, 2024-09-20T22:51:35.714 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_name": "default.rgw.log", 2024-09-20T22:51:35.714 INFO:teuthology.orchestra.run.smithi084.stdout: "create_time": "2024-09-20T22:41:38.454640+0000", 2024-09-20T22:51:35.714 INFO:teuthology.orchestra.run.smithi084.stdout: "flags": 1, 2024-09-20T22:51:35.714 INFO:teuthology.orchestra.run.smithi084.stdout: "flags_names": "hashpspool", 2024-09-20T22:51:35.714 INFO:teuthology.orchestra.run.smithi084.stdout: "type": 1, 2024-09-20T22:51:35.714 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 3, 2024-09-20T22:51:35.714 INFO:teuthology.orchestra.run.smithi084.stdout: "min_size": 2, 2024-09-20T22:51:35.714 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_rule": 0, 2024-09-20T22:51:35.714 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_count": 0, 2024-09-20T22:51:35.714 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_target": 0, 2024-09-20T22:51:35.714 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_barrier": 0, 2024-09-20T22:51:35.715 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-20T22:51:35.715 INFO:teuthology.orchestra.run.smithi084.stdout: "is_stretch_pool": false, 2024-09-20T22:51:35.715 INFO:teuthology.orchestra.run.smithi084.stdout: "object_hash": 2, 2024-09-20T22:51:35.715 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_autoscale_mode": "off", 2024-09-20T22:51:35.715 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num": 32, 2024-09-20T22:51:35.715 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num": 32, 2024-09-20T22:51:35.715 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num_target": 32, 2024-09-20T22:51:35.715 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_target": 32, 2024-09-20T22:51:35.715 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_pending": 32, 2024-09-20T22:51:35.715 INFO:teuthology.orchestra.run.smithi084.stdout: "last_pg_merge_meta": { 2024-09-20T22:51:35.715 INFO:teuthology.orchestra.run.smithi084.stdout: "source_pgid": "0.0", 2024-09-20T22:51:35.715 INFO:teuthology.orchestra.run.smithi084.stdout: "ready_epoch": 0, 2024-09-20T22:51:35.715 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_started": 0, 2024-09-20T22:51:35.715 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_clean": 0, 2024-09-20T22:51:35.715 INFO:teuthology.orchestra.run.smithi084.stdout: "source_version": "0'0", 2024-09-20T22:51:35.716 INFO:teuthology.orchestra.run.smithi084.stdout: "target_version": "0'0" 2024-09-20T22:51:35.716 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:35.716 INFO:teuthology.orchestra.run.smithi084.stdout: "last_change": "70", 2024-09-20T22:51:35.716 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend": "0", 2024-09-20T22:51:35.716 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-20T22:51:35.716 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_preluminous": "0", 2024-09-20T22:51:35.716 INFO:teuthology.orchestra.run.smithi084.stdout: "auid": 0, 2024-09-20T22:51:35.716 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_mode": "selfmanaged", 2024-09-20T22:51:35.716 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_seq": 0, 2024-09-20T22:51:35.716 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_epoch": 0, 2024-09-20T22:51:35.716 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_snaps": [], 2024-09-20T22:51:35.716 INFO:teuthology.orchestra.run.smithi084.stdout: "removed_snaps": "[]", 2024-09-20T22:51:35.716 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_bytes": 0, 2024-09-20T22:51:35.716 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_objects": 0, 2024-09-20T22:51:35.716 INFO:teuthology.orchestra.run.smithi084.stdout: "tiers": [], 2024-09-20T22:51:35.716 INFO:teuthology.orchestra.run.smithi084.stdout: "tier_of": -1, 2024-09-20T22:51:35.717 INFO:teuthology.orchestra.run.smithi084.stdout: "read_tier": -1, 2024-09-20T22:51:35.717 INFO:teuthology.orchestra.run.smithi084.stdout: "write_tier": -1, 2024-09-20T22:51:35.717 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_mode": "none", 2024-09-20T22:51:35.717 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_bytes": 0, 2024-09-20T22:51:35.717 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_objects": 0, 2024-09-20T22:51:35.717 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-20T22:51:35.717 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-20T22:51:35.717 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-20T22:51:35.717 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_flush_age": 0, 2024-09-20T22:51:35.717 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_evict_age": 0, 2024-09-20T22:51:35.717 INFO:teuthology.orchestra.run.smithi084.stdout: "erasure_code_profile": "", 2024-09-20T22:51:35.717 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_params": { 2024-09-20T22:51:35.717 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "none" 2024-09-20T22:51:35.717 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:35.717 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_period": 0, 2024-09-20T22:51:35.717 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_count": 0, 2024-09-20T22:51:35.718 INFO:teuthology.orchestra.run.smithi084.stdout: "use_gmt_hitset": true, 2024-09-20T22:51:35.718 INFO:teuthology.orchestra.run.smithi084.stdout: "min_read_recency_for_promote": 0, 2024-09-20T22:51:35.718 INFO:teuthology.orchestra.run.smithi084.stdout: "min_write_recency_for_promote": 0, 2024-09-20T22:51:35.718 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_grade_decay_rate": 0, 2024-09-20T22:51:35.718 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_search_last_n": 0, 2024-09-20T22:51:35.718 INFO:teuthology.orchestra.run.smithi084.stdout: "grade_table": [], 2024-09-20T22:51:35.718 INFO:teuthology.orchestra.run.smithi084.stdout: "stripe_width": 0, 2024-09-20T22:51:35.718 INFO:teuthology.orchestra.run.smithi084.stdout: "expected_num_objects": 0, 2024-09-20T22:51:35.718 INFO:teuthology.orchestra.run.smithi084.stdout: "fast_read": false, 2024-09-20T22:51:35.718 INFO:teuthology.orchestra.run.smithi084.stdout: "options": {}, 2024-09-20T22:51:35.718 INFO:teuthology.orchestra.run.smithi084.stdout: "application_metadata": { 2024-09-20T22:51:35.718 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw": {} 2024-09-20T22:51:35.718 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:35.718 INFO:teuthology.orchestra.run.smithi084.stdout: "read_balance": { 2024-09-20T22:51:35.718 INFO:teuthology.orchestra.run.smithi084.stdout: "score_type": "Fair distribution", 2024-09-20T22:51:35.719 INFO:teuthology.orchestra.run.smithi084.stdout: "score_acting": 2.2400000095367432, 2024-09-20T22:51:35.719 INFO:teuthology.orchestra.run.smithi084.stdout: "score_stable": 2.2400000095367432, 2024-09-20T22:51:35.719 INFO:teuthology.orchestra.run.smithi084.stdout: "optimal_score": 0.87999999523162842, 2024-09-20T22:51:35.719 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_acting": 1.9700000286102295, 2024-09-20T22:51:35.719 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_stable": 1.9700000286102295, 2024-09-20T22:51:35.719 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity_weighted": 1, 2024-09-20T22:51:35.719 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity": 1, 2024-09-20T22:51:35.719 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity_weighted": 1 2024-09-20T22:51:35.719 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:51:35.719 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:35.719 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:35.719 INFO:teuthology.orchestra.run.smithi084.stdout: "pool": 5, 2024-09-20T22:51:35.719 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_name": "default.rgw.control", 2024-09-20T22:51:35.719 INFO:teuthology.orchestra.run.smithi084.stdout: "create_time": "2024-09-20T22:41:40.407129+0000", 2024-09-20T22:51:35.719 INFO:teuthology.orchestra.run.smithi084.stdout: "flags": 1, 2024-09-20T22:51:35.719 INFO:teuthology.orchestra.run.smithi084.stdout: "flags_names": "hashpspool", 2024-09-20T22:51:35.720 INFO:teuthology.orchestra.run.smithi084.stdout: "type": 1, 2024-09-20T22:51:35.720 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 3, 2024-09-20T22:51:35.720 INFO:teuthology.orchestra.run.smithi084.stdout: "min_size": 2, 2024-09-20T22:51:35.720 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_rule": 0, 2024-09-20T22:51:35.720 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_count": 0, 2024-09-20T22:51:35.720 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_target": 0, 2024-09-20T22:51:35.720 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_barrier": 0, 2024-09-20T22:51:35.720 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-20T22:51:35.720 INFO:teuthology.orchestra.run.smithi084.stdout: "is_stretch_pool": false, 2024-09-20T22:51:35.720 INFO:teuthology.orchestra.run.smithi084.stdout: "object_hash": 2, 2024-09-20T22:51:35.720 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_autoscale_mode": "off", 2024-09-20T22:51:35.720 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num": 32, 2024-09-20T22:51:35.720 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num": 32, 2024-09-20T22:51:35.720 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num_target": 32, 2024-09-20T22:51:35.720 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_target": 32, 2024-09-20T22:51:35.721 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_pending": 32, 2024-09-20T22:51:35.721 INFO:teuthology.orchestra.run.smithi084.stdout: "last_pg_merge_meta": { 2024-09-20T22:51:35.721 INFO:teuthology.orchestra.run.smithi084.stdout: "source_pgid": "0.0", 2024-09-20T22:51:35.721 INFO:teuthology.orchestra.run.smithi084.stdout: "ready_epoch": 0, 2024-09-20T22:51:35.721 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_started": 0, 2024-09-20T22:51:35.721 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_clean": 0, 2024-09-20T22:51:35.721 INFO:teuthology.orchestra.run.smithi084.stdout: "source_version": "0'0", 2024-09-20T22:51:35.721 INFO:teuthology.orchestra.run.smithi084.stdout: "target_version": "0'0" 2024-09-20T22:51:35.721 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:35.721 INFO:teuthology.orchestra.run.smithi084.stdout: "last_change": "72", 2024-09-20T22:51:35.721 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend": "0", 2024-09-20T22:51:35.721 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-20T22:51:35.721 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_preluminous": "0", 2024-09-20T22:51:35.721 INFO:teuthology.orchestra.run.smithi084.stdout: "auid": 0, 2024-09-20T22:51:35.722 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_mode": "selfmanaged", 2024-09-20T22:51:35.722 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_seq": 0, 2024-09-20T22:51:35.722 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_epoch": 0, 2024-09-20T22:51:35.722 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_snaps": [], 2024-09-20T22:51:35.722 INFO:teuthology.orchestra.run.smithi084.stdout: "removed_snaps": "[]", 2024-09-20T22:51:35.722 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_bytes": 0, 2024-09-20T22:51:35.722 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_objects": 0, 2024-09-20T22:51:35.722 INFO:teuthology.orchestra.run.smithi084.stdout: "tiers": [], 2024-09-20T22:51:35.722 INFO:teuthology.orchestra.run.smithi084.stdout: "tier_of": -1, 2024-09-20T22:51:35.722 INFO:teuthology.orchestra.run.smithi084.stdout: "read_tier": -1, 2024-09-20T22:51:35.722 INFO:teuthology.orchestra.run.smithi084.stdout: "write_tier": -1, 2024-09-20T22:51:35.722 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_mode": "none", 2024-09-20T22:51:35.722 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_bytes": 0, 2024-09-20T22:51:35.722 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_objects": 0, 2024-09-20T22:51:35.722 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-20T22:51:35.723 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-20T22:51:35.723 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-20T22:51:35.723 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_flush_age": 0, 2024-09-20T22:51:35.723 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_evict_age": 0, 2024-09-20T22:51:35.723 INFO:teuthology.orchestra.run.smithi084.stdout: "erasure_code_profile": "", 2024-09-20T22:51:35.723 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_params": { 2024-09-20T22:51:35.723 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "none" 2024-09-20T22:51:35.723 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:35.723 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_period": 0, 2024-09-20T22:51:35.723 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_count": 0, 2024-09-20T22:51:35.723 INFO:teuthology.orchestra.run.smithi084.stdout: "use_gmt_hitset": true, 2024-09-20T22:51:35.723 INFO:teuthology.orchestra.run.smithi084.stdout: "min_read_recency_for_promote": 0, 2024-09-20T22:51:35.723 INFO:teuthology.orchestra.run.smithi084.stdout: "min_write_recency_for_promote": 0, 2024-09-20T22:51:35.724 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_grade_decay_rate": 0, 2024-09-20T22:51:35.724 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_search_last_n": 0, 2024-09-20T22:51:35.724 INFO:teuthology.orchestra.run.smithi084.stdout: "grade_table": [], 2024-09-20T22:51:35.724 INFO:teuthology.orchestra.run.smithi084.stdout: "stripe_width": 0, 2024-09-20T22:51:35.724 INFO:teuthology.orchestra.run.smithi084.stdout: "expected_num_objects": 0, 2024-09-20T22:51:35.724 INFO:teuthology.orchestra.run.smithi084.stdout: "fast_read": false, 2024-09-20T22:51:35.724 INFO:teuthology.orchestra.run.smithi084.stdout: "options": {}, 2024-09-20T22:51:35.724 INFO:teuthology.orchestra.run.smithi084.stdout: "application_metadata": { 2024-09-20T22:51:35.724 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw": {} 2024-09-20T22:51:35.724 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:35.724 INFO:teuthology.orchestra.run.smithi084.stdout: "read_balance": { 2024-09-20T22:51:35.724 INFO:teuthology.orchestra.run.smithi084.stdout: "score_type": "Fair distribution", 2024-09-20T22:51:35.724 INFO:teuthology.orchestra.run.smithi084.stdout: "score_acting": 1.7400000095367432, 2024-09-20T22:51:35.724 INFO:teuthology.orchestra.run.smithi084.stdout: "score_stable": 1.7400000095367432, 2024-09-20T22:51:35.724 INFO:teuthology.orchestra.run.smithi084.stdout: "optimal_score": 0.87999999523162842, 2024-09-20T22:51:35.724 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_acting": 1.5299999713897705, 2024-09-20T22:51:35.725 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_stable": 1.5299999713897705, 2024-09-20T22:51:35.725 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity_weighted": 1, 2024-09-20T22:51:35.725 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity": 1, 2024-09-20T22:51:35.726 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity_weighted": 1 2024-09-20T22:51:35.726 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:51:35.726 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:35.726 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:35.726 INFO:teuthology.orchestra.run.smithi084.stdout: "pool": 6, 2024-09-20T22:51:35.726 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_name": "default.rgw.meta", 2024-09-20T22:51:35.726 INFO:teuthology.orchestra.run.smithi084.stdout: "create_time": "2024-09-20T22:41:42.476716+0000", 2024-09-20T22:51:35.726 INFO:teuthology.orchestra.run.smithi084.stdout: "flags": 1, 2024-09-20T22:51:35.726 INFO:teuthology.orchestra.run.smithi084.stdout: "flags_names": "hashpspool", 2024-09-20T22:51:35.726 INFO:teuthology.orchestra.run.smithi084.stdout: "type": 1, 2024-09-20T22:51:35.726 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 3, 2024-09-20T22:51:35.726 INFO:teuthology.orchestra.run.smithi084.stdout: "min_size": 2, 2024-09-20T22:51:35.726 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_rule": 0, 2024-09-20T22:51:35.726 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_count": 0, 2024-09-20T22:51:35.727 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_target": 0, 2024-09-20T22:51:35.727 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_barrier": 0, 2024-09-20T22:51:35.727 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-20T22:51:35.727 INFO:teuthology.orchestra.run.smithi084.stdout: "is_stretch_pool": false, 2024-09-20T22:51:35.727 INFO:teuthology.orchestra.run.smithi084.stdout: "object_hash": 2, 2024-09-20T22:51:35.727 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_autoscale_mode": "off", 2024-09-20T22:51:35.727 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num": 32, 2024-09-20T22:51:35.727 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num": 32, 2024-09-20T22:51:35.727 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num_target": 32, 2024-09-20T22:51:35.727 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_target": 32, 2024-09-20T22:51:35.727 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_pending": 32, 2024-09-20T22:51:35.727 INFO:teuthology.orchestra.run.smithi084.stdout: "last_pg_merge_meta": { 2024-09-20T22:51:35.727 INFO:teuthology.orchestra.run.smithi084.stdout: "source_pgid": "0.0", 2024-09-20T22:51:35.727 INFO:teuthology.orchestra.run.smithi084.stdout: "ready_epoch": 0, 2024-09-20T22:51:35.727 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_started": 0, 2024-09-20T22:51:35.727 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_clean": 0, 2024-09-20T22:51:35.728 INFO:teuthology.orchestra.run.smithi084.stdout: "source_version": "0'0", 2024-09-20T22:51:35.728 INFO:teuthology.orchestra.run.smithi084.stdout: "target_version": "0'0" 2024-09-20T22:51:35.728 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:35.728 INFO:teuthology.orchestra.run.smithi084.stdout: "last_change": "74", 2024-09-20T22:51:35.728 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend": "0", 2024-09-20T22:51:35.728 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-20T22:51:35.728 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_preluminous": "0", 2024-09-20T22:51:35.728 INFO:teuthology.orchestra.run.smithi084.stdout: "auid": 0, 2024-09-20T22:51:35.728 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_mode": "selfmanaged", 2024-09-20T22:51:35.728 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_seq": 0, 2024-09-20T22:51:35.728 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_epoch": 0, 2024-09-20T22:51:35.728 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_snaps": [], 2024-09-20T22:51:35.728 INFO:teuthology.orchestra.run.smithi084.stdout: "removed_snaps": "[]", 2024-09-20T22:51:35.729 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_bytes": 0, 2024-09-20T22:51:35.729 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_objects": 0, 2024-09-20T22:51:35.729 INFO:teuthology.orchestra.run.smithi084.stdout: "tiers": [], 2024-09-20T22:51:35.729 INFO:teuthology.orchestra.run.smithi084.stdout: "tier_of": -1, 2024-09-20T22:51:35.729 INFO:teuthology.orchestra.run.smithi084.stdout: "read_tier": -1, 2024-09-20T22:51:35.729 INFO:teuthology.orchestra.run.smithi084.stdout: "write_tier": -1, 2024-09-20T22:51:35.729 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_mode": "none", 2024-09-20T22:51:35.729 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_bytes": 0, 2024-09-20T22:51:35.729 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_objects": 0, 2024-09-20T22:51:35.729 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-20T22:51:35.729 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-20T22:51:35.729 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-20T22:51:35.729 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_flush_age": 0, 2024-09-20T22:51:35.730 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_evict_age": 0, 2024-09-20T22:51:35.730 INFO:teuthology.orchestra.run.smithi084.stdout: "erasure_code_profile": "", 2024-09-20T22:51:35.730 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_params": { 2024-09-20T22:51:35.730 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "none" 2024-09-20T22:51:35.730 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:35.730 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_period": 0, 2024-09-20T22:51:35.730 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_count": 0, 2024-09-20T22:51:35.730 INFO:teuthology.orchestra.run.smithi084.stdout: "use_gmt_hitset": true, 2024-09-20T22:51:35.730 INFO:teuthology.orchestra.run.smithi084.stdout: "min_read_recency_for_promote": 0, 2024-09-20T22:51:35.730 INFO:teuthology.orchestra.run.smithi084.stdout: "min_write_recency_for_promote": 0, 2024-09-20T22:51:35.730 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_grade_decay_rate": 0, 2024-09-20T22:51:35.730 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_search_last_n": 0, 2024-09-20T22:51:35.730 INFO:teuthology.orchestra.run.smithi084.stdout: "grade_table": [], 2024-09-20T22:51:35.730 INFO:teuthology.orchestra.run.smithi084.stdout: "stripe_width": 0, 2024-09-20T22:51:35.730 INFO:teuthology.orchestra.run.smithi084.stdout: "expected_num_objects": 0, 2024-09-20T22:51:35.731 INFO:teuthology.orchestra.run.smithi084.stdout: "fast_read": false, 2024-09-20T22:51:35.731 INFO:teuthology.orchestra.run.smithi084.stdout: "options": { 2024-09-20T22:51:35.731 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_autoscale_bias": 4 2024-09-20T22:51:35.731 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:35.731 INFO:teuthology.orchestra.run.smithi084.stdout: "application_metadata": { 2024-09-20T22:51:35.731 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw": {} 2024-09-20T22:51:35.731 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:35.731 INFO:teuthology.orchestra.run.smithi084.stdout: "read_balance": { 2024-09-20T22:51:35.731 INFO:teuthology.orchestra.run.smithi084.stdout: "score_type": "Fair distribution", 2024-09-20T22:51:35.731 INFO:teuthology.orchestra.run.smithi084.stdout: "score_acting": 1.7400000095367432, 2024-09-20T22:51:35.731 INFO:teuthology.orchestra.run.smithi084.stdout: "score_stable": 1.7400000095367432, 2024-09-20T22:51:35.731 INFO:teuthology.orchestra.run.smithi084.stdout: "optimal_score": 0.87999999523162842, 2024-09-20T22:51:35.731 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_acting": 1.5299999713897705, 2024-09-20T22:51:35.731 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_stable": 1.5299999713897705, 2024-09-20T22:51:35.731 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity_weighted": 1, 2024-09-20T22:51:35.732 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity": 1, 2024-09-20T22:51:35.732 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity_weighted": 1 2024-09-20T22:51:35.732 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:51:35.732 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:35.732 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:35.732 INFO:teuthology.orchestra.run.smithi084.stdout: "pool": 7, 2024-09-20T22:51:35.732 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_name": "unique_pool_0", 2024-09-20T22:51:35.732 INFO:teuthology.orchestra.run.smithi084.stdout: "create_time": "2024-09-20T22:42:51.354358+0000", 2024-09-20T22:51:35.732 INFO:teuthology.orchestra.run.smithi084.stdout: "flags": 8193, 2024-09-20T22:51:35.732 INFO:teuthology.orchestra.run.smithi084.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-20T22:51:35.732 INFO:teuthology.orchestra.run.smithi084.stdout: "type": 1, 2024-09-20T22:51:35.732 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 3, 2024-09-20T22:51:35.732 INFO:teuthology.orchestra.run.smithi084.stdout: "min_size": 2, 2024-09-20T22:51:35.732 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_rule": 0, 2024-09-20T22:51:35.733 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_count": 0, 2024-09-20T22:51:35.733 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_target": 0, 2024-09-20T22:51:35.733 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_barrier": 0, 2024-09-20T22:51:35.733 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-20T22:51:35.733 INFO:teuthology.orchestra.run.smithi084.stdout: "is_stretch_pool": false, 2024-09-20T22:51:35.733 INFO:teuthology.orchestra.run.smithi084.stdout: "object_hash": 2, 2024-09-20T22:51:35.733 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_autoscale_mode": "off", 2024-09-20T22:51:35.733 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num": 16, 2024-09-20T22:51:35.733 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num": 16, 2024-09-20T22:51:35.733 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num_target": 16, 2024-09-20T22:51:35.733 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_target": 16, 2024-09-20T22:51:35.733 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_pending": 16, 2024-09-20T22:51:35.733 INFO:teuthology.orchestra.run.smithi084.stdout: "last_pg_merge_meta": { 2024-09-20T22:51:35.733 INFO:teuthology.orchestra.run.smithi084.stdout: "source_pgid": "7.10", 2024-09-20T22:51:35.733 INFO:teuthology.orchestra.run.smithi084.stdout: "ready_epoch": 484, 2024-09-20T22:51:35.733 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_started": 483, 2024-09-20T22:51:35.733 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_clean": 483, 2024-09-20T22:51:35.734 INFO:teuthology.orchestra.run.smithi084.stdout: "source_version": "480'172", 2024-09-20T22:51:35.734 INFO:teuthology.orchestra.run.smithi084.stdout: "target_version": "480'253" 2024-09-20T22:51:35.734 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:35.734 INFO:teuthology.orchestra.run.smithi084.stdout: "last_change": "615", 2024-09-20T22:51:35.734 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend": "0", 2024-09-20T22:51:35.734 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_prenautilus": "484", 2024-09-20T22:51:35.734 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_preluminous": "482", 2024-09-20T22:51:35.734 INFO:teuthology.orchestra.run.smithi084.stdout: "auid": 0, 2024-09-20T22:51:35.734 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_mode": "selfmanaged", 2024-09-20T22:51:35.734 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_seq": 269, 2024-09-20T22:51:35.734 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_epoch": 615, 2024-09-20T22:51:35.734 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_snaps": [], 2024-09-20T22:51:35.734 INFO:teuthology.orchestra.run.smithi084.stdout: "removed_snaps": "[]", 2024-09-20T22:51:35.734 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_bytes": 0, 2024-09-20T22:51:35.734 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_objects": 0, 2024-09-20T22:51:35.735 INFO:teuthology.orchestra.run.smithi084.stdout: "tiers": [], 2024-09-20T22:51:35.735 INFO:teuthology.orchestra.run.smithi084.stdout: "tier_of": -1, 2024-09-20T22:51:35.735 INFO:teuthology.orchestra.run.smithi084.stdout: "read_tier": -1, 2024-09-20T22:51:35.735 INFO:teuthology.orchestra.run.smithi084.stdout: "write_tier": -1, 2024-09-20T22:51:35.735 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_mode": "none", 2024-09-20T22:51:35.735 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_bytes": 0, 2024-09-20T22:51:35.735 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_objects": 0, 2024-09-20T22:51:35.735 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-20T22:51:35.735 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-20T22:51:35.735 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-20T22:51:35.735 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_flush_age": 0, 2024-09-20T22:51:35.735 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_evict_age": 0, 2024-09-20T22:51:35.735 INFO:teuthology.orchestra.run.smithi084.stdout: "erasure_code_profile": "", 2024-09-20T22:51:35.736 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_params": { 2024-09-20T22:51:35.736 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "none" 2024-09-20T22:51:35.736 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:35.736 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_period": 0, 2024-09-20T22:51:35.736 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_count": 0, 2024-09-20T22:51:35.736 INFO:teuthology.orchestra.run.smithi084.stdout: "use_gmt_hitset": true, 2024-09-20T22:51:35.736 INFO:teuthology.orchestra.run.smithi084.stdout: "min_read_recency_for_promote": 0, 2024-09-20T22:51:35.736 INFO:teuthology.orchestra.run.smithi084.stdout: "min_write_recency_for_promote": 0, 2024-09-20T22:51:35.736 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_grade_decay_rate": 0, 2024-09-20T22:51:35.736 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_search_last_n": 0, 2024-09-20T22:51:35.736 INFO:teuthology.orchestra.run.smithi084.stdout: "grade_table": [], 2024-09-20T22:51:35.736 INFO:teuthology.orchestra.run.smithi084.stdout: "stripe_width": 0, 2024-09-20T22:51:35.736 INFO:teuthology.orchestra.run.smithi084.stdout: "expected_num_objects": 0, 2024-09-20T22:51:35.736 INFO:teuthology.orchestra.run.smithi084.stdout: "fast_read": false, 2024-09-20T22:51:35.736 INFO:teuthology.orchestra.run.smithi084.stdout: "options": {}, 2024-09-20T22:51:35.737 INFO:teuthology.orchestra.run.smithi084.stdout: "application_metadata": { 2024-09-20T22:51:35.737 INFO:teuthology.orchestra.run.smithi084.stdout: "rados": {} 2024-09-20T22:51:35.737 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:35.737 INFO:teuthology.orchestra.run.smithi084.stdout: "read_balance": { 2024-09-20T22:51:35.737 INFO:teuthology.orchestra.run.smithi084.stdout: "score_type": "Fair distribution", 2024-09-20T22:51:35.737 INFO:teuthology.orchestra.run.smithi084.stdout: "score_acting": 1.9900000095367432, 2024-09-20T22:51:35.737 INFO:teuthology.orchestra.run.smithi084.stdout: "score_stable": 1.9900000095367432, 2024-09-20T22:51:35.737 INFO:teuthology.orchestra.run.smithi084.stdout: "optimal_score": 0.87999999523162842, 2024-09-20T22:51:35.737 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_acting": 1.75, 2024-09-20T22:51:35.737 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_stable": 1.75, 2024-09-20T22:51:35.737 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity_weighted": 1, 2024-09-20T22:51:35.737 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity": 1, 2024-09-20T22:51:35.737 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity_weighted": 1 2024-09-20T22:51:35.738 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:51:35.738 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:51:35.738 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2024-09-20T22:51:35.738 INFO:teuthology.orchestra.run.smithi084.stdout: "osds": [ 2024-09-20T22:51:35.738 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:35.738 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 0, 2024-09-20T22:51:35.738 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "ea36a489-6f91-4b2f-b784-b90506b9805d", 2024-09-20T22:51:35.738 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-09-20T22:51:35.738 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-09-20T22:51:35.738 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-09-20T22:51:35.738 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-09-20T22:51:35.738 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-09-20T22:51:35.738 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-09-20T22:51:35.738 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 8, 2024-09-20T22:51:35.739 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 602, 2024-09-20T22:51:35.739 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-09-20T22:51:35.739 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-09-20T22:51:35.739 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-09-20T22:51:35.739 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:51:35.739 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:35.739 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:51:35.739 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6801", 2024-09-20T22:51:35.739 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 268084626 2024-09-20T22:51:35.739 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:51:35.739 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:51:35.739 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:35.739 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-09-20T22:51:35.739 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:51:35.739 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:35.740 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:51:35.740 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6802", 2024-09-20T22:51:35.740 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 268084626 2024-09-20T22:51:35.740 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:51:35.740 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:51:35.740 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:35.740 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-09-20T22:51:35.740 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:51:35.740 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:35.740 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:51:35.740 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6804", 2024-09-20T22:51:35.740 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 268084626 2024-09-20T22:51:35.740 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:51:35.740 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:51:35.740 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:35.741 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-09-20T22:51:35.741 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:51:35.741 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:35.741 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:51:35.741 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6803", 2024-09-20T22:51:35.741 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 268084626 2024-09-20T22:51:35.741 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:51:35.741 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:51:35.741 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:35.741 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.84:6801/268084626", 2024-09-20T22:51:35.741 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.84:6802/268084626", 2024-09-20T22:51:35.741 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.84:6804/268084626", 2024-09-20T22:51:35.741 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.84:6803/268084626", 2024-09-20T22:51:35.741 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-09-20T22:51:35.741 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-09-20T22:51:35.742 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-09-20T22:51:35.742 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:51:35.742 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:35.742 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:35.742 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 1, 2024-09-20T22:51:35.742 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "c8649d2f-27ef-47f8-bc94-f1cd9e62bdb8", 2024-09-20T22:51:35.742 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-09-20T22:51:35.742 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 0, 2024-09-20T22:51:35.742 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 0, 2024-09-20T22:51:35.742 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-09-20T22:51:35.742 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-09-20T22:51:35.742 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-09-20T22:51:35.742 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 13, 2024-09-20T22:51:35.743 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 468, 2024-09-20T22:51:35.743 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-09-20T22:51:35.743 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-09-20T22:51:35.743 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-09-20T22:51:35.743 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:51:35.743 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:35.743 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:51:35.743 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6805", 2024-09-20T22:51:35.743 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 418374136 2024-09-20T22:51:35.743 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:51:35.743 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:51:35.743 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:35.743 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-09-20T22:51:35.744 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:51:35.744 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:35.744 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:51:35.744 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6806", 2024-09-20T22:51:35.744 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 418374136 2024-09-20T22:51:35.744 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:51:35.744 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:51:35.744 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:35.744 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-09-20T22:51:35.744 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:51:35.744 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:35.744 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:51:35.744 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6808", 2024-09-20T22:51:35.744 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 418374136 2024-09-20T22:51:35.744 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:51:35.745 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:51:35.745 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:35.745 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-09-20T22:51:35.745 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:51:35.745 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:35.745 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:51:35.745 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6807", 2024-09-20T22:51:35.745 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 418374136 2024-09-20T22:51:35.745 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:51:35.745 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:51:35.745 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:35.745 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.84:6805/418374136", 2024-09-20T22:51:35.745 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.84:6806/418374136", 2024-09-20T22:51:35.745 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.84:6808/418374136", 2024-09-20T22:51:35.745 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.84:6807/418374136", 2024-09-20T22:51:35.746 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-09-20T22:51:35.746 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-09-20T22:51:35.746 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-09-20T22:51:35.746 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:51:35.746 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:35.746 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:35.746 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 2, 2024-09-20T22:51:35.746 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "c728eb48-a0f0-4de4-a651-7f3bec867280", 2024-09-20T22:51:35.746 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-09-20T22:51:35.746 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-09-20T22:51:35.746 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-09-20T22:51:35.746 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-09-20T22:51:35.746 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-09-20T22:51:35.746 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-09-20T22:51:35.747 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 18, 2024-09-20T22:51:35.747 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 599, 2024-09-20T22:51:35.747 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-09-20T22:51:35.747 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-09-20T22:51:35.747 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-09-20T22:51:35.747 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:51:35.747 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:35.747 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:51:35.747 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6809", 2024-09-20T22:51:35.747 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3977168611 2024-09-20T22:51:35.747 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:51:35.747 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:51:35.747 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:35.748 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-09-20T22:51:35.748 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:51:35.748 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:35.748 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:51:35.748 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6810", 2024-09-20T22:51:35.748 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3977168611 2024-09-20T22:51:35.748 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:51:35.748 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:51:35.748 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:35.748 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-09-20T22:51:35.748 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:51:35.748 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:35.748 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:51:35.748 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6812", 2024-09-20T22:51:35.749 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3977168611 2024-09-20T22:51:35.749 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:51:35.749 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:51:35.749 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:35.749 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-09-20T22:51:35.749 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:51:35.749 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:35.749 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:51:35.749 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6811", 2024-09-20T22:51:35.749 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3977168611 2024-09-20T22:51:35.749 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:51:35.749 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:51:35.749 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:35.749 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.84:6809/3977168611", 2024-09-20T22:51:35.749 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.84:6810/3977168611", 2024-09-20T22:51:35.750 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.84:6812/3977168611", 2024-09-20T22:51:35.750 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.84:6811/3977168611", 2024-09-20T22:51:35.750 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-09-20T22:51:35.750 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-09-20T22:51:35.750 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-09-20T22:51:35.750 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:51:35.750 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:35.750 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:35.750 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 3, 2024-09-20T22:51:35.750 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "573f44eb-0853-4630-a7d8-9caaeb256f9b", 2024-09-20T22:51:35.750 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-09-20T22:51:35.750 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-09-20T22:51:35.750 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-09-20T22:51:35.750 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-09-20T22:51:35.750 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-09-20T22:51:35.751 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-09-20T22:51:35.751 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 25, 2024-09-20T22:51:35.751 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 600, 2024-09-20T22:51:35.751 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-09-20T22:51:35.751 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-09-20T22:51:35.751 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-09-20T22:51:35.751 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:51:35.751 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:35.751 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:51:35.751 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6813", 2024-09-20T22:51:35.751 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2192239945 2024-09-20T22:51:35.751 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:51:35.751 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:51:35.752 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:35.752 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-09-20T22:51:35.752 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:51:35.752 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:35.752 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:51:35.752 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6814", 2024-09-20T22:51:35.752 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2192239945 2024-09-20T22:51:35.752 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:51:35.752 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:51:35.752 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:35.752 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-09-20T22:51:35.752 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:51:35.752 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:35.752 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:51:35.753 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6816", 2024-09-20T22:51:35.753 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2192239945 2024-09-20T22:51:35.753 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:51:35.753 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:51:35.753 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:35.753 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-09-20T22:51:35.753 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:51:35.753 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:35.753 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:51:35.753 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6815", 2024-09-20T22:51:35.753 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2192239945 2024-09-20T22:51:35.753 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:51:35.753 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:51:35.753 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:35.754 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.84:6813/2192239945", 2024-09-20T22:51:35.754 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.84:6814/2192239945", 2024-09-20T22:51:35.754 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.84:6816/2192239945", 2024-09-20T22:51:35.754 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.84:6815/2192239945", 2024-09-20T22:51:35.754 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-09-20T22:51:35.754 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-09-20T22:51:35.754 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-09-20T22:51:35.754 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:51:35.754 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:35.754 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:35.754 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 4, 2024-09-20T22:51:35.754 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "a52c3b94-c5f3-490f-8ac8-09061b9b2abb", 2024-09-20T22:51:35.755 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-09-20T22:51:35.755 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-09-20T22:51:35.755 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-09-20T22:51:35.755 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-09-20T22:51:35.755 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-09-20T22:51:35.755 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-09-20T22:51:35.755 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 31, 2024-09-20T22:51:35.755 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 599, 2024-09-20T22:51:35.755 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-09-20T22:51:35.755 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-09-20T22:51:35.755 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-09-20T22:51:35.755 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:51:35.755 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:35.756 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:51:35.756 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.154:6800", 2024-09-20T22:51:35.756 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1793136002 2024-09-20T22:51:35.756 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:51:35.756 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:51:35.756 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:35.756 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-09-20T22:51:35.756 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:51:35.756 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:35.756 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:51:35.756 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.154:6801", 2024-09-20T22:51:35.756 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1793136002 2024-09-20T22:51:35.756 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:51:35.757 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:51:35.757 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:35.757 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-09-20T22:51:35.757 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:51:35.757 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:35.757 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:51:35.757 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.154:6803", 2024-09-20T22:51:35.757 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1793136002 2024-09-20T22:51:35.757 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:51:35.757 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:51:35.757 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:35.757 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-09-20T22:51:35.758 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:51:35.758 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:35.758 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:51:35.758 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.154:6802", 2024-09-20T22:51:35.758 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1793136002 2024-09-20T22:51:35.758 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:51:35.758 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:51:35.758 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:35.758 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.154:6800/1793136002", 2024-09-20T22:51:35.758 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.154:6801/1793136002", 2024-09-20T22:51:35.758 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.154:6803/1793136002", 2024-09-20T22:51:35.758 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.154:6802/1793136002", 2024-09-20T22:51:35.758 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-09-20T22:51:35.759 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-09-20T22:51:35.759 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-09-20T22:51:35.759 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:51:35.759 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:35.759 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:35.759 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 5, 2024-09-20T22:51:35.759 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "4fa0dc8e-25cc-46cd-96ff-33f20b6faec3", 2024-09-20T22:51:35.759 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-09-20T22:51:35.759 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-09-20T22:51:35.759 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-09-20T22:51:35.759 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-09-20T22:51:35.759 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-09-20T22:51:35.759 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-09-20T22:51:35.759 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 40, 2024-09-20T22:51:35.760 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 602, 2024-09-20T22:51:35.760 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-09-20T22:51:35.760 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-09-20T22:51:35.760 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-09-20T22:51:35.760 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:51:35.760 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:35.760 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:51:35.760 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.154:6804", 2024-09-20T22:51:35.760 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 493899927 2024-09-20T22:51:35.760 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:51:35.760 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:51:35.760 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:35.760 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-09-20T22:51:35.760 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:51:35.760 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:35.761 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:51:35.761 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.154:6805", 2024-09-20T22:51:35.761 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 493899927 2024-09-20T22:51:35.761 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:51:35.761 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:51:35.761 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:35.761 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-09-20T22:51:35.761 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:51:35.761 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:35.761 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:51:35.761 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.154:6807", 2024-09-20T22:51:35.761 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 493899927 2024-09-20T22:51:35.761 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:51:35.761 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:51:35.762 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:35.762 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-09-20T22:51:35.762 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:51:35.762 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:35.762 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:51:35.762 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.154:6806", 2024-09-20T22:51:35.762 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 493899927 2024-09-20T22:51:35.762 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:51:35.762 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:51:35.762 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:35.762 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.154:6804/493899927", 2024-09-20T22:51:35.762 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.154:6805/493899927", 2024-09-20T22:51:35.762 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.154:6807/493899927", 2024-09-20T22:51:35.762 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.154:6806/493899927", 2024-09-20T22:51:35.763 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-09-20T22:51:35.763 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-09-20T22:51:35.763 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-09-20T22:51:35.763 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:51:35.763 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:35.763 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:35.763 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 6, 2024-09-20T22:51:35.763 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "56788d74-810e-4827-a615-958c12f8a368", 2024-09-20T22:51:35.763 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-09-20T22:51:35.763 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-09-20T22:51:35.763 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-09-20T22:51:35.763 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-09-20T22:51:35.763 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-09-20T22:51:35.763 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-09-20T22:51:35.764 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 48, 2024-09-20T22:51:35.764 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 602, 2024-09-20T22:51:35.764 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-09-20T22:51:35.764 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-09-20T22:51:35.764 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-09-20T22:51:35.764 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:51:35.764 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:35.764 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:51:35.764 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.154:6808", 2024-09-20T22:51:35.764 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2375091033 2024-09-20T22:51:35.764 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:51:35.764 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:51:35.764 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:35.765 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-09-20T22:51:35.765 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:51:35.765 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:35.765 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:51:35.765 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.154:6809", 2024-09-20T22:51:35.765 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2375091033 2024-09-20T22:51:35.765 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:51:35.765 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:51:35.765 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:35.765 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-09-20T22:51:35.765 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:51:35.765 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:35.765 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:51:35.765 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.154:6811", 2024-09-20T22:51:35.765 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2375091033 2024-09-20T22:51:35.766 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:51:35.766 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:51:35.766 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:35.766 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-09-20T22:51:35.766 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:51:35.766 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:35.766 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:51:35.766 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.154:6810", 2024-09-20T22:51:35.766 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2375091033 2024-09-20T22:51:35.766 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:51:35.766 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:51:35.766 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:35.766 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.154:6808/2375091033", 2024-09-20T22:51:35.766 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.154:6809/2375091033", 2024-09-20T22:51:35.767 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.154:6811/2375091033", 2024-09-20T22:51:35.767 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.154:6810/2375091033", 2024-09-20T22:51:35.767 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-09-20T22:51:35.767 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-09-20T22:51:35.767 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-09-20T22:51:35.767 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:51:35.767 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:35.767 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:35.767 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 7, 2024-09-20T22:51:35.767 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "83503727-afc1-4b95-8b96-ecb63f3338c2", 2024-09-20T22:51:35.767 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-09-20T22:51:35.767 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-09-20T22:51:35.767 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-09-20T22:51:35.767 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-09-20T22:51:35.767 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-09-20T22:51:35.768 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-09-20T22:51:35.768 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 59, 2024-09-20T22:51:35.768 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 599, 2024-09-20T22:51:35.768 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-09-20T22:51:35.768 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-09-20T22:51:35.768 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-09-20T22:51:35.768 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:51:35.768 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:35.768 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:51:35.768 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.154:6812", 2024-09-20T22:51:35.768 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3717704491 2024-09-20T22:51:35.768 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:51:35.768 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:51:35.768 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:35.768 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-09-20T22:51:35.769 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:51:35.769 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:35.769 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:51:35.769 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.154:6813", 2024-09-20T22:51:35.769 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3717704491 2024-09-20T22:51:35.769 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:51:35.769 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:51:35.769 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:35.769 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-09-20T22:51:35.769 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:51:35.769 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:35.769 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:51:35.769 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.154:6815", 2024-09-20T22:51:35.770 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3717704491 2024-09-20T22:51:35.770 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:51:35.770 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:51:35.770 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:35.770 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-09-20T22:51:35.770 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:51:35.770 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:35.770 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:51:35.770 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.154:6814", 2024-09-20T22:51:35.770 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3717704491 2024-09-20T22:51:35.770 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:51:35.770 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:51:35.770 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:35.770 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.154:6812/3717704491", 2024-09-20T22:51:35.771 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.154:6813/3717704491", 2024-09-20T22:51:35.771 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.154:6815/3717704491", 2024-09-20T22:51:35.771 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.154:6814/3717704491", 2024-09-20T22:51:35.771 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-09-20T22:51:35.771 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-09-20T22:51:35.771 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-09-20T22:51:35.771 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:51:35.771 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:51:35.771 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2024-09-20T22:51:35.771 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_xinfo": [ 2024-09-20T22:51:35.771 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:35.771 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 0, 2024-09-20T22:51:35.771 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-09-20T22:51:35.771 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-09-20T22:51:35.771 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-09-20T22:51:35.772 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540701547738038271, 2024-09-20T22:51:35.772 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-09-20T22:51:35.772 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-09-20T22:51:34.189498+0000", 2024-09-20T22:51:35.772 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-09-20T22:51:35.772 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:35.772 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:35.772 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 1, 2024-09-20T22:51:35.772 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-09-20T22:51:35.772 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-09-20T22:51:35.772 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-09-20T22:51:35.772 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540701547738038271, 2024-09-20T22:51:35.772 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 65536, 2024-09-20T22:51:35.772 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-09-20T22:50:44.391721+0000", 2024-09-20T22:51:35.772 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-09-20T22:51:35.772 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:35.773 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:35.773 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 2, 2024-09-20T22:51:35.773 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-09-20T22:51:35.773 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-09-20T22:51:35.773 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-09-20T22:51:35.773 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540701547738038271, 2024-09-20T22:51:35.773 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-09-20T22:51:35.773 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-09-20T22:51:09.299267+0000", 2024-09-20T22:51:35.773 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-09-20T22:51:35.773 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:35.773 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:35.773 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 3, 2024-09-20T22:51:35.773 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-09-20T22:51:35.773 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-09-20T22:51:35.774 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-09-20T22:51:35.774 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540701547738038271, 2024-09-20T22:51:35.774 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-09-20T22:51:35.774 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-09-20T22:50:59.301860+0000", 2024-09-20T22:51:35.774 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-09-20T22:51:35.774 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:35.774 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:35.774 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 4, 2024-09-20T22:51:35.774 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-09-20T22:51:35.774 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-09-20T22:51:35.774 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-09-20T22:51:35.774 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540701547738038271, 2024-09-20T22:51:35.774 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-09-20T22:51:35.775 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-09-20T22:50:05.515503+0000", 2024-09-20T22:51:35.775 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-09-20T22:51:35.775 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:35.775 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:35.775 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 5, 2024-09-20T22:51:35.775 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-09-20T22:51:35.775 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-09-20T22:51:35.775 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-09-20T22:51:35.775 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540701547738038271, 2024-09-20T22:51:35.775 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-09-20T22:51:35.775 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-09-20T22:51:07.578742+0000", 2024-09-20T22:51:35.775 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-09-20T22:51:35.775 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:35.775 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:35.776 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 6, 2024-09-20T22:51:35.776 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-09-20T22:51:35.776 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-09-20T22:51:35.776 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-09-20T22:51:35.776 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540701547738038271, 2024-09-20T22:51:35.776 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-09-20T22:51:35.776 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-09-20T22:50:53.718609+0000", 2024-09-20T22:51:35.776 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-09-20T22:51:35.776 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:35.776 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:35.776 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 7, 2024-09-20T22:51:35.776 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-09-20T22:51:35.776 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-09-20T22:51:35.776 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-09-20T22:51:35.777 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540701547738038271, 2024-09-20T22:51:35.777 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-09-20T22:51:35.777 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-09-20T22:51:27.150337+0000", 2024-09-20T22:51:35.777 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-09-20T22:51:35.777 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:51:35.777 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2024-09-20T22:51:35.777 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_upmap": [ 2024-09-20T22:51:35.777 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:35.777 INFO:teuthology.orchestra.run.smithi084.stdout: "pgid": "1.9", 2024-09-20T22:51:35.777 INFO:teuthology.orchestra.run.smithi084.stdout: "osds": [ 2024-09-20T22:51:35.777 INFO:teuthology.orchestra.run.smithi084.stdout: 7, 2024-09-20T22:51:35.777 INFO:teuthology.orchestra.run.smithi084.stdout: 3, 2024-09-20T22:51:35.777 INFO:teuthology.orchestra.run.smithi084.stdout: 5 2024-09-20T22:51:35.777 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:51:35.778 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:35.778 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:35.778 INFO:teuthology.orchestra.run.smithi084.stdout: "pgid": "6.1e", 2024-09-20T22:51:35.778 INFO:teuthology.orchestra.run.smithi084.stdout: "osds": [ 2024-09-20T22:51:35.778 INFO:teuthology.orchestra.run.smithi084.stdout: 0, 2024-09-20T22:51:35.778 INFO:teuthology.orchestra.run.smithi084.stdout: 1, 2024-09-20T22:51:35.778 INFO:teuthology.orchestra.run.smithi084.stdout: 3 2024-09-20T22:51:35.778 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:51:35.778 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:51:35.778 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2024-09-20T22:51:35.778 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_upmap_items": [ 2024-09-20T22:51:35.778 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:35.778 INFO:teuthology.orchestra.run.smithi084.stdout: "pgid": "3.a", 2024-09-20T22:51:35.779 INFO:teuthology.orchestra.run.smithi084.stdout: "mappings": [ 2024-09-20T22:51:35.779 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:35.779 INFO:teuthology.orchestra.run.smithi084.stdout: "from": 6, 2024-09-20T22:51:35.779 INFO:teuthology.orchestra.run.smithi084.stdout: "to": 2 2024-09-20T22:51:35.779 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:51:35.779 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:51:35.779 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:35.779 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:35.779 INFO:teuthology.orchestra.run.smithi084.stdout: "pgid": "4.11", 2024-09-20T22:51:35.779 INFO:teuthology.orchestra.run.smithi084.stdout: "mappings": [ 2024-09-20T22:51:35.779 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:35.779 INFO:teuthology.orchestra.run.smithi084.stdout: "from": 3, 2024-09-20T22:51:35.779 INFO:teuthology.orchestra.run.smithi084.stdout: "to": 5 2024-09-20T22:51:35.779 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:51:35.780 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:51:35.780 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:51:35.780 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2024-09-20T22:51:35.780 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_upmap_primaries": [], 2024-09-20T22:51:35.780 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_temp": [ 2024-09-20T22:51:35.780 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:35.780 INFO:teuthology.orchestra.run.smithi084.stdout: "pgid": "7.0", 2024-09-20T22:51:35.780 INFO:teuthology.orchestra.run.smithi084.stdout: "osds": [ 2024-09-20T22:51:35.780 INFO:teuthology.orchestra.run.smithi084.stdout: 5, 2024-09-20T22:51:35.780 INFO:teuthology.orchestra.run.smithi084.stdout: 3, 2024-09-20T22:51:35.780 INFO:teuthology.orchestra.run.smithi084.stdout: 1 2024-09-20T22:51:35.780 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:51:35.780 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:35.780 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:35.780 INFO:teuthology.orchestra.run.smithi084.stdout: "pgid": "7.2", 2024-09-20T22:51:35.781 INFO:teuthology.orchestra.run.smithi084.stdout: "osds": [ 2024-09-20T22:51:35.781 INFO:teuthology.orchestra.run.smithi084.stdout: 3, 2024-09-20T22:51:35.781 INFO:teuthology.orchestra.run.smithi084.stdout: 2, 2024-09-20T22:51:35.781 INFO:teuthology.orchestra.run.smithi084.stdout: 1 2024-09-20T22:51:35.781 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:51:35.781 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:51:35.781 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2024-09-20T22:51:35.781 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_temp": [], 2024-09-20T22:51:35.781 INFO:teuthology.orchestra.run.smithi084.stdout: "blocklist": { 2024-09-20T22:51:35.781 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/542020545": "2024-09-21T22:42:07.596045+0000", 2024-09-20T22:51:35.781 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/25873236": "2024-09-21T22:36:54.917812+0000", 2024-09-20T22:51:35.781 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:6800/820908470": "2024-09-21T22:36:54.917812+0000", 2024-09-20T22:51:35.781 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/567614908": "2024-09-21T22:36:54.917812+0000", 2024-09-20T22:51:35.781 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/2963622459": "2024-09-21T22:42:07.596045+0000", 2024-09-20T22:51:35.781 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/1176849276": "2024-09-21T22:37:17.690104+0000", 2024-09-20T22:51:35.782 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:6800/2932420050": "2024-09-21T22:37:17.690104+0000", 2024-09-20T22:51:35.782 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/3879263635": "2024-09-21T22:37:17.690104+0000", 2024-09-20T22:51:35.782 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/969472279": "2024-09-21T22:36:54.917812+0000", 2024-09-20T22:51:35.782 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/1361143214": "2024-09-21T22:37:17.690104+0000", 2024-09-20T22:51:35.782 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/3353126955": "2024-09-21T22:42:07.596045+0000", 2024-09-20T22:51:35.782 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:6800/4024278029": "2024-09-21T22:42:07.596045+0000", 2024-09-20T22:51:35.782 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/547902262": "2024-09-21T22:42:07.596045+0000", 2024-09-20T22:51:35.782 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/1242929317": "2024-09-21T22:42:07.596045+0000" 2024-09-20T22:51:35.782 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:35.782 INFO:teuthology.orchestra.run.smithi084.stdout: "range_blocklist": {}, 2024-09-20T22:51:35.782 INFO:teuthology.orchestra.run.smithi084.stdout: "erasure_code_profiles": { 2024-09-20T22:51:35.782 INFO:teuthology.orchestra.run.smithi084.stdout: "default": { 2024-09-20T22:51:35.782 INFO:teuthology.orchestra.run.smithi084.stdout: "crush-failure-domain": "osd", 2024-09-20T22:51:35.782 INFO:teuthology.orchestra.run.smithi084.stdout: "k": "2", 2024-09-20T22:51:35.783 INFO:teuthology.orchestra.run.smithi084.stdout: "m": "1", 2024-09-20T22:51:35.783 INFO:teuthology.orchestra.run.smithi084.stdout: "plugin": "jerasure", 2024-09-20T22:51:35.783 INFO:teuthology.orchestra.run.smithi084.stdout: "technique": "reed_sol_van" 2024-09-20T22:51:35.783 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:51:35.783 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:35.783 INFO:teuthology.orchestra.run.smithi084.stdout: "removed_snaps_queue": [ 2024-09-20T22:51:35.783 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:35.783 INFO:teuthology.orchestra.run.smithi084.stdout: "pool": 7, 2024-09-20T22:51:35.783 INFO:teuthology.orchestra.run.smithi084.stdout: "snaps": [ 2024-09-20T22:51:35.783 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:35.783 INFO:teuthology.orchestra.run.smithi084.stdout: "begin": 167, 2024-09-20T22:51:35.783 INFO:teuthology.orchestra.run.smithi084.stdout: "length": 1 2024-09-20T22:51:35.783 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:35.783 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:35.784 INFO:teuthology.orchestra.run.smithi084.stdout: "begin": 197, 2024-09-20T22:51:35.784 INFO:teuthology.orchestra.run.smithi084.stdout: "length": 1 2024-09-20T22:51:35.784 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:35.784 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:35.784 INFO:teuthology.orchestra.run.smithi084.stdout: "begin": 206, 2024-09-20T22:51:35.784 INFO:teuthology.orchestra.run.smithi084.stdout: "length": 1 2024-09-20T22:51:35.784 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:35.784 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:35.784 INFO:teuthology.orchestra.run.smithi084.stdout: "begin": 212, 2024-09-20T22:51:35.784 INFO:teuthology.orchestra.run.smithi084.stdout: "length": 1 2024-09-20T22:51:35.784 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:35.784 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:35.784 INFO:teuthology.orchestra.run.smithi084.stdout: "begin": 223, 2024-09-20T22:51:35.784 INFO:teuthology.orchestra.run.smithi084.stdout: "length": 1 2024-09-20T22:51:35.784 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:35.784 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:35.785 INFO:teuthology.orchestra.run.smithi084.stdout: "begin": 227, 2024-09-20T22:51:35.785 INFO:teuthology.orchestra.run.smithi084.stdout: "length": 9 2024-09-20T22:51:35.785 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:35.785 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:35.785 INFO:teuthology.orchestra.run.smithi084.stdout: "begin": 238, 2024-09-20T22:51:35.785 INFO:teuthology.orchestra.run.smithi084.stdout: "length": 4 2024-09-20T22:51:35.785 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:35.785 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:35.785 INFO:teuthology.orchestra.run.smithi084.stdout: "begin": 243, 2024-09-20T22:51:35.785 INFO:teuthology.orchestra.run.smithi084.stdout: "length": 2 2024-09-20T22:51:35.785 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:35.786 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:35.786 INFO:teuthology.orchestra.run.smithi084.stdout: "begin": 246, 2024-09-20T22:51:35.786 INFO:teuthology.orchestra.run.smithi084.stdout: "length": 2 2024-09-20T22:51:35.786 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:35.786 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:35.786 INFO:teuthology.orchestra.run.smithi084.stdout: "begin": 249, 2024-09-20T22:51:35.786 INFO:teuthology.orchestra.run.smithi084.stdout: "length": 1 2024-09-20T22:51:35.786 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:35.786 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:35.786 INFO:teuthology.orchestra.run.smithi084.stdout: "begin": 254, 2024-09-20T22:51:35.786 INFO:teuthology.orchestra.run.smithi084.stdout: "length": 3 2024-09-20T22:51:35.786 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:35.786 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:35.787 INFO:teuthology.orchestra.run.smithi084.stdout: "begin": 262, 2024-09-20T22:51:35.787 INFO:teuthology.orchestra.run.smithi084.stdout: "length": 1 2024-09-20T22:51:35.787 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:51:35.787 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:51:35.787 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:51:35.787 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2024-09-20T22:51:35.787 INFO:teuthology.orchestra.run.smithi084.stdout: "new_removed_snaps": [], 2024-09-20T22:51:35.787 INFO:teuthology.orchestra.run.smithi084.stdout: "new_purged_snaps": [], 2024-09-20T22:51:35.787 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_node_flags": {}, 2024-09-20T22:51:35.787 INFO:teuthology.orchestra.run.smithi084.stdout: "device_class_flags": {}, 2024-09-20T22:51:35.787 INFO:teuthology.orchestra.run.smithi084.stdout: "stretch_mode": { 2024-09-20T22:51:35.787 INFO:teuthology.orchestra.run.smithi084.stdout: "stretch_mode_enabled": false, 2024-09-20T22:51:35.787 INFO:teuthology.orchestra.run.smithi084.stdout: "stretch_bucket_count": 0, 2024-09-20T22:51:35.787 INFO:teuthology.orchestra.run.smithi084.stdout: "degraded_stretch_mode": 0, 2024-09-20T22:51:35.788 INFO:teuthology.orchestra.run.smithi084.stdout: "recovering_stretch_mode": 0, 2024-09-20T22:51:35.788 INFO:teuthology.orchestra.run.smithi084.stdout: "stretch_mode_bucket": 0 2024-09-20T22:51:35.788 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:51:35.788 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-20T22:51:35.788 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:51:35.789 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:35 smithi154 ceph-mon[33264]: pgmap v822: 158 pgs: 4 active+clean+snaptrim, 2 active+remapped+backfill_toofull, 152 active+clean; 456 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 25 MiB/s rd, 2.9 MiB/s wr, 21 op/s; 33/1245 objects misplaced (2.651%) 2024-09-20T22:51:35.789 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:35 smithi154 ceph-mon[33264]: 5.1b scrub starts 2024-09-20T22:51:35.789 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:35 smithi154 ceph-mon[33264]: 5.1b scrub ok 2024-09-20T22:51:35.789 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:35 smithi154 ceph-mon[33264]: osdmap e614: 8 total, 8 up, 7 in 2024-09-20T22:51:35.789 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:35 smithi154 ceph-mon[33264]: 3.2 scrub starts 2024-09-20T22:51:35.789 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:35 smithi154 ceph-mon[33264]: 3.2 scrub ok 2024-09-20T22:51:35.810 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:35.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:35.811 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:51:35.911 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:51:35.934 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:35.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:35.934 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:51:36.035 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:51:36.055 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:36.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:36.056 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:51:36.157 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:51:36.177 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:36.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:36.178 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:51:36.279 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:51:36.300 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:36.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:36.302 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:51:36.402 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:51:36.425 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:36.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:36.426 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:51:36.486 INFO:tasks.rados.rados.0.smithi084.stdout:2415: done (3 left) 2024-09-20T22:51:36.487 INFO:tasks.rados.rados.0.smithi084.stdout:2416: write oid 6 current snap is 268 2024-09-20T22:51:36.487 INFO:tasks.rados.rados.0.smithi084.stdout:2416: seq_num 830 ranges {749810=739147,2002878=650280,3293084=415653} 2024-09-20T22:51:36.495 INFO:tasks.rados.rados.0.smithi084.stdout:2416: writing smithi08477203-6 from 749810 to 1488957 tid 1 2024-09-20T22:51:36.502 INFO:tasks.rados.rados.0.smithi084.stdout:2416: writing smithi08477203-6 from 2002878 to 2653158 tid 2 2024-09-20T22:51:36.508 INFO:tasks.rados.rados.0.smithi084.stdout:2416: writing smithi08477203-6 from 3293084 to 3708737 tid 3 2024-09-20T22:51:36.508 INFO:tasks.rados.rados.0.smithi084.stdout:2417: copy_from oid 2 from oid 45 current snap is 268 2024-09-20T22:51:36.508 INFO:tasks.rados.rados.0.smithi084.stdout:2414: expect (ObjNum 787 snap 254 seq_num 787) 2024-09-20T22:51:36.526 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:51:36.547 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:36.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:36.548 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:51:36.649 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:51:36.659 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:36 smithi084 ceph-mon[40024]: osdmap e615: 8 total, 8 up, 7 in 2024-09-20T22:51:36.659 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:36 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/2434853549' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-20T22:51:36.659 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:36 smithi084 ceph-mon[40024]: 6.18 scrub starts 2024-09-20T22:51:36.659 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:36 smithi084 ceph-mon[40024]: 6.18 scrub ok 2024-09-20T22:51:36.659 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:36 smithi084 ceph-mon[40024]: purged_snaps scrub starts 2024-09-20T22:51:36.659 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:36 smithi084 ceph-mon[40024]: purged_snaps scrub ok 2024-09-20T22:51:36.660 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:36 smithi084 ceph-mon[29774]: osdmap e615: 8 total, 8 up, 7 in 2024-09-20T22:51:36.660 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:36 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/2434853549' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-20T22:51:36.660 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:36 smithi084 ceph-mon[29774]: 6.18 scrub starts 2024-09-20T22:51:36.660 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:36 smithi084 ceph-mon[29774]: 6.18 scrub ok 2024-09-20T22:51:36.660 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:36 smithi084 ceph-mon[29774]: purged_snaps scrub starts 2024-09-20T22:51:36.660 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:36 smithi084 ceph-mon[29774]: purged_snaps scrub ok 2024-09-20T22:51:36.668 INFO:tasks.rados.rados.0.smithi084.stdout:2412: finishing write tid 3 to smithi08477203-17 2024-09-20T22:51:36.668 INFO:tasks.rados.rados.0.smithi084.stdout:2412: finishing write tid 4 to smithi08477203-17 2024-09-20T22:51:36.668 INFO:tasks.rados.rados.0.smithi084.stdout:2414: done (4 left) 2024-09-20T22:51:36.668 INFO:tasks.rados.rados.0.smithi084.stdout:2418: read oid 47 snap -1 2024-09-20T22:51:36.668 INFO:tasks.rados.rados.0.smithi084.stdout:2418: expect (ObjNum 671 snap 205 seq_num 671) 2024-09-20T22:51:36.668 INFO:tasks.rados.rados.0.smithi084.stdout:2419: rollback oid 46 current snap is 268 2024-09-20T22:51:36.668 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 46 to 263 2024-09-20T22:51:36.668 INFO:tasks.rados.rados.0.smithi084.stdout:2413: finishing rollback tid 1 to smithi08477203-10 2024-09-20T22:51:36.668 INFO:tasks.rados.rados.0.smithi084.stdout:2413: finishing rollback tid 2 to smithi08477203-10 2024-09-20T22:51:36.668 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 10 v 1234 (ObjNum 575 snap 172 seq_num 575) dirty exists 2024-09-20T22:51:36.668 INFO:tasks.rados.rados.0.smithi084.stdout:2412: finishing write tid 5 to smithi08477203-17 2024-09-20T22:51:36.668 INFO:tasks.rados.rados.0.smithi084.stdout:2412: oid 17 updating version 0 to 772 2024-09-20T22:51:36.668 INFO:tasks.rados.rados.0.smithi084.stdout:2412: oid 17 updating version 772 to 775 2024-09-20T22:51:36.668 INFO:tasks.rados.rados.0.smithi084.stdout:2412: oid 17 version 775 is already newer than 773 2024-09-20T22:51:36.669 INFO:tasks.rados.rados.0.smithi084.stdout:2412: oid 17 version 775 is already newer than 774 2024-09-20T22:51:36.669 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 17 v 775 (ObjNum 829 snap 268 seq_num 829) dirty exists 2024-09-20T22:51:36.669 INFO:tasks.rados.rados.0.smithi084.stdout:2412: left oid 17 (ObjNum 829 snap 268 seq_num 829) 2024-09-20T22:51:36.669 INFO:tasks.rados.rados.0.smithi084.stdout:2412: done (5 left) 2024-09-20T22:51:36.669 INFO:tasks.rados.rados.0.smithi084.stdout:2413: done (4 left) 2024-09-20T22:51:36.669 INFO:tasks.rados.rados.0.smithi084.stdout:2420: write (excl) oid 34 current snap is 268 2024-09-20T22:51:36.669 INFO:tasks.rados.rados.0.smithi084.stdout:2420: seq_num 831 ranges {766617=414396,1945007=510291,3204154=626925} 2024-09-20T22:51:36.670 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:36.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:36.671 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:51:36.674 INFO:tasks.rados.rados.0.smithi084.stdout:2420: writing smithi08477203-34 from 766617 to 1181013 tid 1 2024-09-20T22:51:36.681 INFO:tasks.rados.rados.0.smithi084.stdout:2420: writing smithi08477203-34 from 1945007 to 2455298 tid 2 2024-09-20T22:51:36.689 INFO:tasks.rados.rados.0.smithi084.stdout:2420: writing smithi08477203-34 from 3204154 to 3831079 tid 3 2024-09-20T22:51:36.690 INFO:tasks.rados.rados.0.smithi084.stdout:2421: read oid 16 snap -1 2024-09-20T22:51:36.690 INFO:tasks.rados.rados.0.smithi084.stdout:2421: expect (ObjNum 805 snap 261 seq_num 805) 2024-09-20T22:51:36.690 INFO:tasks.rados.rados.0.smithi084.stdout:2422: read oid 42 snap 243 2024-09-20T22:51:36.690 INFO:tasks.rados.rados.0.smithi084.stdout:2422: expect (ObjNum 714 snap 224 seq_num 714) 2024-09-20T22:51:36.690 INFO:tasks.rados.rados.0.smithi084.stdout:2423: write oid 13 current snap is 268 2024-09-20T22:51:36.690 INFO:tasks.rados.rados.0.smithi084.stdout:2416: finishing write tid 1 to smithi08477203-6 2024-09-20T22:51:36.690 INFO:tasks.rados.rados.0.smithi084.stdout:2416: finishing write tid 2 to smithi08477203-6 2024-09-20T22:51:36.690 INFO:tasks.rados.rados.0.smithi084.stdout:2416: finishing write tid 3 to smithi08477203-6 2024-09-20T22:51:36.690 INFO:tasks.rados.rados.0.smithi084.stdout:2416: finishing write tid 4 to smithi08477203-6 2024-09-20T22:51:36.690 INFO:tasks.rados.rados.0.smithi084.stdout:2416: finishing write tid 5 to smithi08477203-6 2024-09-20T22:51:36.690 INFO:tasks.rados.rados.0.smithi084.stdout:2416: oid 6 updating version 0 to 779 2024-09-20T22:51:36.690 INFO:tasks.rados.rados.0.smithi084.stdout:2416: oid 6 version 779 is already newer than 777 2024-09-20T22:51:36.690 INFO:tasks.rados.rados.0.smithi084.stdout:2416: oid 6 version 779 is already newer than 778 2024-09-20T22:51:36.691 INFO:tasks.rados.rados.0.smithi084.stdout:2416: oid 6 version 779 is already newer than 776 2024-09-20T22:51:36.691 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 6 v 779 (ObjNum 830 snap 268 seq_num 830) dirty exists 2024-09-20T22:51:36.691 INFO:tasks.rados.rados.0.smithi084.stdout:2416: left oid 6 (ObjNum 830 snap 268 seq_num 830) 2024-09-20T22:51:36.691 INFO:tasks.rados.rados.0.smithi084.stdout:2417: finishing copy_from racing read to smithi08477203-2 2024-09-20T22:51:36.691 INFO:tasks.rados.rados.0.smithi084.stdout:2417: finishing copy_from to smithi08477203-2 2024-09-20T22:51:36.691 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 2 v 801 (ObjNum 750 snap 238 seq_num 750) dirty exists 2024-09-20T22:51:36.691 INFO:tasks.rados.rados.0.smithi084.stdout:2423: seq_num 832 ranges {783424=489645,2003489=653949,3231577=554550} 2024-09-20T22:51:36.696 INFO:tasks.rados.rados.0.smithi084.stdout:2423: writing smithi08477203-13 from 783424 to 1273069 tid 1 2024-09-20T22:51:36.704 INFO:tasks.rados.rados.0.smithi084.stdout:2423: writing smithi08477203-13 from 2003489 to 2657438 tid 2 2024-09-20T22:51:36.710 INFO:tasks.rados.rados.0.smithi084.stdout:2423: writing smithi08477203-13 from 3231577 to 3786127 tid 3 2024-09-20T22:51:36.710 INFO:tasks.rados.rados.0.smithi084.stdout:2416: done (7 left) 2024-09-20T22:51:36.711 INFO:tasks.rados.rados.0.smithi084.stdout:2417: done (6 left) 2024-09-20T22:51:36.711 INFO:tasks.rados.rados.0.smithi084.stdout:2424: rollback oid 50 current snap is 268 2024-09-20T22:51:36.711 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 50 to 189 2024-09-20T22:51:36.711 INFO:tasks.rados.rados.0.smithi084.stdout:2419: finishing rollback tid 1 to smithi08477203-46 2024-09-20T22:51:36.711 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 46 v 808 (ObjNum 129 snap 0 seq_num 2091872608) dirty dne 2024-09-20T22:51:36.711 INFO:tasks.rados.rados.0.smithi084.stdout:2420: finishing write tid 1 to smithi08477203-34 2024-09-20T22:51:36.711 INFO:tasks.rados.rados.0.smithi084.stdout:2418: expect (ObjNum 671 snap 205 seq_num 671) 2024-09-20T22:51:36.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:36 smithi154 ceph-mon[33264]: osdmap e615: 8 total, 8 up, 7 in 2024-09-20T22:51:36.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:36 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/2434853549' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-20T22:51:36.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:36 smithi154 ceph-mon[33264]: 6.18 scrub starts 2024-09-20T22:51:36.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:36 smithi154 ceph-mon[33264]: 6.18 scrub ok 2024-09-20T22:51:36.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:36 smithi154 ceph-mon[33264]: purged_snaps scrub starts 2024-09-20T22:51:36.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:36 smithi154 ceph-mon[33264]: purged_snaps scrub ok 2024-09-20T22:51:36.772 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:51:36.793 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:36.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:36.794 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:51:36.894 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:51:36.915 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:36.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:36.916 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:51:36.930 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-20T22:51:36.930 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2024-09-20T22:51:36.930 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2024-09-20T22:51:36.930 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2024-09-20T22:51:36.930 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-20T22:51:37.017 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:51:37.038 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:37.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:37.039 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:51:37.049 INFO:tasks.rados.rados.0.smithi084.stdout:2420: finishing write tid 2 to smithi08477203-34 2024-09-20T22:51:37.049 INFO:tasks.rados.rados.0.smithi084.stdout:2420: finishing write tid 3 to smithi08477203-34 2024-09-20T22:51:37.050 INFO:tasks.rados.rados.0.smithi084.stdout:2420: finishing write tid 4 to smithi08477203-34 2024-09-20T22:51:37.050 INFO:tasks.rados.rados.0.smithi084.stdout:2420: finishing write tid 5 to smithi08477203-34 2024-09-20T22:51:37.050 INFO:tasks.rados.rados.0.smithi084.stdout:2420: oid 34 updating version 0 to 1229 2024-09-20T22:51:37.050 INFO:tasks.rados.rados.0.smithi084.stdout:2420: oid 34 updating version 1229 to 1231 2024-09-20T22:51:37.050 INFO:tasks.rados.rados.0.smithi084.stdout:2420: oid 34 version 1231 is already newer than 1230 2024-09-20T22:51:37.050 INFO:tasks.rados.rados.0.smithi084.stdout:2420: oid 34 updating version 1231 to 1232 2024-09-20T22:51:37.050 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 34 v 1232 (ObjNum 831 snap 268 seq_num 831) dirty exists 2024-09-20T22:51:37.050 INFO:tasks.rados.rados.0.smithi084.stdout:2420: left oid 34 (ObjNum 831 snap 268 seq_num 831) 2024-09-20T22:51:37.050 INFO:tasks.rados.rados.0.smithi084.stdout:2422: expect (ObjNum 714 snap 224 seq_num 714) 2024-09-20T22:51:37.110 INFO:tasks.rados.rados.0.smithi084.stdout:2423: finishing write tid 1 to smithi08477203-13 2024-09-20T22:51:37.110 INFO:tasks.rados.rados.0.smithi084.stdout:2421: expect (ObjNum 805 snap 261 seq_num 805) 2024-09-20T22:51:37.139 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:51:37.145 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-20T22:51:37.161 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:37.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:37.162 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:51:37.263 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:51:37.284 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:37.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:37.285 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:51:37.344 INFO:tasks.rados.rados.0.smithi084.stdout:2423: finishing write tid 2 to smithi08477203-13 2024-09-20T22:51:37.344 INFO:tasks.rados.rados.0.smithi084.stdout:2424: finishing rollback tid 0 to smithi08477203-50 2024-09-20T22:51:37.344 INFO:tasks.rados.rados.0.smithi084.stdout:2423: finishing write tid 3 to smithi08477203-13 2024-09-20T22:51:37.344 INFO:tasks.rados.rados.0.smithi084.stdout:2424: finishing rollback tid 1 to smithi08477203-50 2024-09-20T22:51:37.344 INFO:tasks.rados.rados.0.smithi084.stdout:2423: finishing write tid 4 to smithi08477203-13 2024-09-20T22:51:37.344 INFO:tasks.rados.rados.0.smithi084.stdout:2424: finishing rollback tid 2 to smithi08477203-50 2024-09-20T22:51:37.345 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 50 v 808 (ObjNum 515 snap 148 seq_num 515) dirty exists 2024-09-20T22:51:37.345 INFO:tasks.rados.rados.0.smithi084.stdout:2423: finishing write tid 5 to smithi08477203-13 2024-09-20T22:51:37.345 INFO:tasks.rados.rados.0.smithi084.stdout:2423: oid 13 updating version 0 to 1796 2024-09-20T22:51:37.345 INFO:tasks.rados.rados.0.smithi084.stdout:2423: oid 13 version 1796 is already newer than 1795 2024-09-20T22:51:37.345 INFO:tasks.rados.rados.0.smithi084.stdout:2423: oid 13 version 1796 is already newer than 1794 2024-09-20T22:51:37.345 INFO:tasks.rados.rados.0.smithi084.stdout:2423: oid 13 updating version 1796 to 1797 2024-09-20T22:51:37.346 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 13 v 1797 (ObjNum 832 snap 268 seq_num 832) dirty exists 2024-09-20T22:51:37.346 INFO:tasks.rados.rados.0.smithi084.stdout:2423: left oid 13 (ObjNum 832 snap 268 seq_num 832) 2024-09-20T22:51:37.346 INFO:tasks.rados.rados.0.smithi084.stdout:2418: done (6 left) 2024-09-20T22:51:37.346 INFO:tasks.rados.rados.0.smithi084.stdout:2419: done (5 left) 2024-09-20T22:51:37.346 INFO:tasks.rados.rados.0.smithi084.stdout:2420: done (4 left) 2024-09-20T22:51:37.346 INFO:tasks.rados.rados.0.smithi084.stdout:2421: done (3 left) 2024-09-20T22:51:37.346 INFO:tasks.rados.rados.0.smithi084.stdout:2422: done (2 left) 2024-09-20T22:51:37.347 INFO:tasks.rados.rados.0.smithi084.stdout:2423: done (1 left) 2024-09-20T22:51:37.347 INFO:tasks.rados.rados.0.smithi084.stdout:2424: done (0 left) 2024-09-20T22:51:37.347 INFO:tasks.rados.rados.0.smithi084.stdout:2425: snap_remove snap 263 2024-09-20T22:51:37.385 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:51:37.407 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:37.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:37.408 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:51:37.489 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is set 2024-09-20T22:51:37.489 INFO:tasks.rados.rados.0.smithi084.stdout:2425: done (0 left) 2024-09-20T22:51:37.490 INFO:tasks.rados.rados.0.smithi084.stdout:2426: snap_create 2024-09-20T22:51:37.508 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:51:37.530 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:37.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:37.531 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:51:37.632 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:51:37.654 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:37.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:37.655 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:51:37.664 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:37 smithi084 ceph-mon[40024]: pgmap v825: 158 pgs: 4 active+clean+snaptrim, 2 active+remapped+backfill_toofull, 152 active+clean; 456 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 25 MiB/s rd, 2.9 MiB/s wr, 21 op/s; 33/1245 objects misplaced (2.651%) 2024-09-20T22:51:37.664 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:37 smithi084 ceph-mon[40024]: 3.d scrub starts 2024-09-20T22:51:37.664 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:37 smithi084 ceph-mon[40024]: 3.d scrub ok 2024-09-20T22:51:37.664 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:37 smithi084 ceph-mon[40024]: osdmap e616: 8 total, 8 up, 7 in 2024-09-20T22:51:37.664 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:37 smithi084 ceph-mon[40024]: 6.1a deep-scrub starts 2024-09-20T22:51:37.664 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:37 smithi084 ceph-mon[40024]: 6.1a deep-scrub ok 2024-09-20T22:51:37.664 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:37 smithi084 ceph-mon[40024]: 7.4 scrub starts 2024-09-20T22:51:37.664 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:37 smithi084 ceph-mon[40024]: 7.4 scrub ok 2024-09-20T22:51:37.664 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:37 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/2010719087' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-20T22:51:37.664 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:37 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-20T22:51:37.665 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:37 smithi084 ceph-mon[29774]: pgmap v825: 158 pgs: 4 active+clean+snaptrim, 2 active+remapped+backfill_toofull, 152 active+clean; 456 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 25 MiB/s rd, 2.9 MiB/s wr, 21 op/s; 33/1245 objects misplaced (2.651%) 2024-09-20T22:51:37.665 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:37 smithi084 ceph-mon[29774]: 3.d scrub starts 2024-09-20T22:51:37.665 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:37 smithi084 ceph-mon[29774]: 3.d scrub ok 2024-09-20T22:51:37.665 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:37 smithi084 ceph-mon[29774]: osdmap e616: 8 total, 8 up, 7 in 2024-09-20T22:51:37.665 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:37 smithi084 ceph-mon[29774]: 6.1a deep-scrub starts 2024-09-20T22:51:37.665 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:37 smithi084 ceph-mon[29774]: 6.1a deep-scrub ok 2024-09-20T22:51:37.665 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:37 smithi084 ceph-mon[29774]: 7.4 scrub starts 2024-09-20T22:51:37.665 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:37 smithi084 ceph-mon[29774]: 7.4 scrub ok 2024-09-20T22:51:37.665 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:37 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/2010719087' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-20T22:51:37.665 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:37 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-20T22:51:37.666 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.1 dump_historic_ops 2024-09-20T22:51:37.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:37 smithi154 ceph-mon[33264]: pgmap v825: 158 pgs: 4 active+clean+snaptrim, 2 active+remapped+backfill_toofull, 152 active+clean; 456 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 25 MiB/s rd, 2.9 MiB/s wr, 21 op/s; 33/1245 objects misplaced (2.651%) 2024-09-20T22:51:37.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:37 smithi154 ceph-mon[33264]: 3.d scrub starts 2024-09-20T22:51:37.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:37 smithi154 ceph-mon[33264]: 3.d scrub ok 2024-09-20T22:51:37.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:37 smithi154 ceph-mon[33264]: osdmap e616: 8 total, 8 up, 7 in 2024-09-20T22:51:37.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:37 smithi154 ceph-mon[33264]: 6.1a deep-scrub starts 2024-09-20T22:51:37.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:37 smithi154 ceph-mon[33264]: 6.1a deep-scrub ok 2024-09-20T22:51:37.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:37 smithi154 ceph-mon[33264]: 7.4 scrub starts 2024-09-20T22:51:37.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:37 smithi154 ceph-mon[33264]: 7.4 scrub ok 2024-09-20T22:51:37.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:37 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/2010719087' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-20T22:51:37.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:37 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-20T22:51:37.756 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:51:37.777 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:37.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:37.778 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:51:37.879 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:51:37.900 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:37.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:37.901 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:51:38.003 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:51:38.025 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:38.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:38.027 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:51:38.128 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:51:38.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:38.151 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:38.151 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:51:38.252 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:51:38.267 INFO:tasks.rados.rados.0.smithi084.stdout:2426: done (0 left) 2024-09-20T22:51:38.267 INFO:tasks.rados.rados.0.smithi084.stdout:2427: snap_remove snap 264 2024-09-20T22:51:38.277 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:38.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:38.278 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:51:38.379 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:51:38.390 INFO:tasks.thrashosds.thrasher:j is {'epoch': 615, 'fsid': 'a9c0bc70-77a0-11ef-baf5-efdc52797490', 'created': '2024-09-20T22:36:30.240626+0000', 'modified': '2024-09-20T22:51:35.471275+0000', 'last_up_change': '2024-09-20T22:41:27.098565+0000', 'last_in_change': '2024-09-20T22:51:19.253681+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 18, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 7, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-20T22:39:21.984381+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 11, 'pg_placement_num': 11, 'pg_placement_num_target': 11, 'pg_num_target': 11, 'pg_num_pending': 11, 'last_pg_merge_meta': {'source_pgid': '1.b', 'ready_epoch': 299, 'last_epoch_started': 298, 'last_epoch_clean': 298, 'source_version': "77'128", 'target_version': "77'128"}, 'last_change': '542', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '299', 'last_force_op_resend_preluminous': '297', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.1700000762939453, 'score_stable': 2.1700000762939453, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.909999966621399, 'raw_score_stable': 1.909999966621399, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-09-20T22:41:35.820081+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '69', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 69, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.6700000762939453, 'score_stable': 2.6700000762939453, 'optimal_score': 0.75, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-09-20T22:41:36.546389+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.7400000095367432, 'score_stable': 1.7400000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.5299999713897705, 'raw_score_stable': 1.5299999713897705, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-09-20T22:41:38.454640+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '70', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.240000009536743, 'score_stable': 2.240000009536743, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.9700000286102295, 'raw_score_stable': 1.9700000286102295, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-09-20T22:41:40.407129+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '72', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.7400000095367432, 'score_stable': 1.7400000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.5299999713897705, 'raw_score_stable': 1.5299999713897705, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-09-20T22:41:42.476716+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '74', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.7400000095367432, 'score_stable': 1.7400000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.5299999713897705, 'raw_score_stable': 1.5299999713897705, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'unique_pool_0', 'create_time': '2024-09-20T22:42:51.354358+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '7.10', 'ready_epoch': 484, 'last_epoch_started': 483, 'last_epoch_clean': 483, 'source_version': "480'172", 'target_version': "480'253"}, 'last_change': '615', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '484', 'last_force_op_resend_preluminous': '482', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 269, 'snap_epoch': 615, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': 'ea36a489-6f91-4b2f-b784-b90506b9805d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 602, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6801', 'nonce': 268084626}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6802', 'nonce': 268084626}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6804', 'nonce': 268084626}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6803', 'nonce': 268084626}]}, 'public_addr': '172.21.15.84:6801/268084626', 'cluster_addr': '172.21.15.84:6802/268084626', 'heartbeat_back_addr': '172.21.15.84:6804/268084626', 'heartbeat_front_addr': '172.21.15.84:6803/268084626', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'c8649d2f-27ef-47f8-bc94-f1cd9e62bdb8', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 468, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6805', 'nonce': 418374136}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6806', 'nonce': 418374136}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6808', 'nonce': 418374136}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6807', 'nonce': 418374136}]}, 'public_addr': '172.21.15.84:6805/418374136', 'cluster_addr': '172.21.15.84:6806/418374136', 'heartbeat_back_addr': '172.21.15.84:6808/418374136', 'heartbeat_front_addr': '172.21.15.84:6807/418374136', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'c728eb48-a0f0-4de4-a651-7f3bec867280', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 599, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6809', 'nonce': 3977168611}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6810', 'nonce': 3977168611}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6812', 'nonce': 3977168611}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6811', 'nonce': 3977168611}]}, 'public_addr': '172.21.15.84:6809/3977168611', 'cluster_addr': '172.21.15.84:6810/3977168611', 'heartbeat_back_addr': '172.21.15.84:6812/3977168611', 'heartbeat_front_addr': '172.21.15.84:6811/3977168611', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '573f44eb-0853-4630-a7d8-9caaeb256f9b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 25, 'up_thru': 600, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6813', 'nonce': 2192239945}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6814', 'nonce': 2192239945}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6816', 'nonce': 2192239945}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6815', 'nonce': 2192239945}]}, 'public_addr': '172.21.15.84:6813/2192239945', 'cluster_addr': '172.21.15.84:6814/2192239945', 'heartbeat_back_addr': '172.21.15.84:6816/2192239945', 'heartbeat_front_addr': '172.21.15.84:6815/2192239945', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'a52c3b94-c5f3-490f-8ac8-09061b9b2abb', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 599, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6800', 'nonce': 1793136002}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6801', 'nonce': 1793136002}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6803', 'nonce': 1793136002}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6802', 'nonce': 1793136002}]}, 'public_addr': '172.21.15.154:6800/1793136002', 'cluster_addr': '172.21.15.154:6801/1793136002', 'heartbeat_back_addr': '172.21.15.154:6803/1793136002', 'heartbeat_front_addr': '172.21.15.154:6802/1793136002', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '4fa0dc8e-25cc-46cd-96ff-33f20b6faec3', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 40, 'up_thru': 602, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6804', 'nonce': 493899927}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6805', 'nonce': 493899927}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6807', 'nonce': 493899927}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6806', 'nonce': 493899927}]}, 'public_addr': '172.21.15.154:6804/493899927', 'cluster_addr': '172.21.15.154:6805/493899927', 'heartbeat_back_addr': '172.21.15.154:6807/493899927', 'heartbeat_front_addr': '172.21.15.154:6806/493899927', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '56788d74-810e-4827-a615-958c12f8a368', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 48, 'up_thru': 602, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6808', 'nonce': 2375091033}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6809', 'nonce': 2375091033}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6811', 'nonce': 2375091033}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6810', 'nonce': 2375091033}]}, 'public_addr': '172.21.15.154:6808/2375091033', 'cluster_addr': '172.21.15.154:6809/2375091033', 'heartbeat_back_addr': '172.21.15.154:6811/2375091033', 'heartbeat_front_addr': '172.21.15.154:6810/2375091033', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '83503727-afc1-4b95-8b96-ecb63f3338c2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 59, 'up_thru': 599, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6812', 'nonce': 3717704491}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6813', 'nonce': 3717704491}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6815', 'nonce': 3717704491}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6814', 'nonce': 3717704491}]}, 'public_addr': '172.21.15.154:6812/3717704491', 'cluster_addr': '172.21.15.154:6813/3717704491', 'heartbeat_back_addr': '172.21.15.154:6815/3717704491', 'heartbeat_front_addr': '172.21.15.154:6814/3717704491', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-20T22:51:34.189498+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-09-20T22:50:44.391721+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-20T22:51:09.299267+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-20T22:50:59.301860+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-20T22:50:05.515503+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-20T22:51:07.578742+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-20T22:50:53.718609+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-20T22:51:27.150337+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '1.9', 'osds': [7, 3, 5]}, {'pgid': '6.1e', 'osds': [0, 1, 3]}], 'pg_upmap_items': [{'pgid': '3.a', 'mappings': [{'from': 6, 'to': 2}]}, {'pgid': '4.11', 'mappings': [{'from': 3, 'to': 5}]}], 'pg_upmap_primaries': [], 'pg_temp': [{'pgid': '7.0', 'osds': [5, 3, 1]}, {'pgid': '7.2', 'osds': [3, 2, 1]}], 'primary_temp': [], 'blocklist': {'172.21.15.84:0/542020545': '2024-09-21T22:42:07.596045+0000', '172.21.15.84:0/25873236': '2024-09-21T22:36:54.917812+0000', '172.21.15.84:6800/820908470': '2024-09-21T22:36:54.917812+0000', '172.21.15.84:0/567614908': '2024-09-21T22:36:54.917812+0000', '172.21.15.84:0/2963622459': '2024-09-21T22:42:07.596045+0000', '172.21.15.84:0/1176849276': '2024-09-21T22:37:17.690104+0000', '172.21.15.84:6800/2932420050': '2024-09-21T22:37:17.690104+0000', '172.21.15.84:0/3879263635': '2024-09-21T22:37:17.690104+0000', '172.21.15.84:0/969472279': '2024-09-21T22:36:54.917812+0000', '172.21.15.84:0/1361143214': '2024-09-21T22:37:17.690104+0000', '172.21.15.84:0/3353126955': '2024-09-21T22:42:07.596045+0000', '172.21.15.84:6800/4024278029': '2024-09-21T22:42:07.596045+0000', '172.21.15.84:0/547902262': '2024-09-21T22:42:07.596045+0000', '172.21.15.84:0/1242929317': '2024-09-21T22:42:07.596045+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [{'pool': 7, 'snaps': [{'begin': 167, 'length': 1}, {'begin': 197, 'length': 1}, {'begin': 206, 'length': 1}, {'begin': 212, 'length': 1}, {'begin': 223, 'length': 1}, {'begin': 227, 'length': 9}, {'begin': 238, 'length': 4}, {'begin': 243, 'length': 2}, {'begin': 246, 'length': 2}, {'begin': 249, 'length': 1}, {'begin': 254, 'length': 3}, {'begin': 262, 'length': 1}]}], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-20T22:51:38.391 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph pg dump --format=json 2024-09-20T22:51:38.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:38.403 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:38.403 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:51:38.504 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:51:38.525 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:38.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:38.526 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:51:38.614 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:51:38 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: ::ffff:172.21.15.154 - - [20/Sep/2024:22:51:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-20T22:51:38.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:38 smithi084 ceph-mon[29774]: 2.2 scrub starts 2024-09-20T22:51:38.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:38 smithi084 ceph-mon[29774]: 2.2 scrub ok 2024-09-20T22:51:38.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:38 smithi084 ceph-mon[29774]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:51:38.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:38 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T22:51:38.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:38 smithi084 ceph-mon[29774]: osdmap e617: 8 total, 8 up, 7 in 2024-09-20T22:51:38.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:38 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:51:38.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:38 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:51:38.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:38 smithi084 ceph-mon[29774]: osdmap e618: 8 total, 8 up, 7 in 2024-09-20T22:51:38.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:38 smithi084 ceph-mon[40024]: 2.2 scrub starts 2024-09-20T22:51:38.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:38 smithi084 ceph-mon[40024]: 2.2 scrub ok 2024-09-20T22:51:38.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:38 smithi084 ceph-mon[40024]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:51:38.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:38 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T22:51:38.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:38 smithi084 ceph-mon[40024]: osdmap e617: 8 total, 8 up, 7 in 2024-09-20T22:51:38.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:38 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:51:38.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:38 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:51:38.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:38 smithi084 ceph-mon[40024]: osdmap e618: 8 total, 8 up, 7 in 2024-09-20T22:51:38.627 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:51:38.647 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:38.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:38.648 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:51:38.749 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:51:38.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:38 smithi154 ceph-mon[33264]: 2.2 scrub starts 2024-09-20T22:51:38.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:38 smithi154 ceph-mon[33264]: 2.2 scrub ok 2024-09-20T22:51:38.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:38 smithi154 ceph-mon[33264]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:51:38.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:38 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T22:51:38.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:38 smithi154 ceph-mon[33264]: osdmap e617: 8 total, 8 up, 7 in 2024-09-20T22:51:38.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:38 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:51:38.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:38 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:51:38.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:38 smithi154 ceph-mon[33264]: osdmap e618: 8 total, 8 up, 7 in 2024-09-20T22:51:38.771 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:38.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:38.772 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:51:38.873 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:51:38.893 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:38.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:38.894 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:51:38.995 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:51:39.018 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:39.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:39.019 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:51:39.057 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:51:39.121 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:51:39.142 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:39.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:39.143 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:51:39.244 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:51:39.265 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:39.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:39.266 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:51:39.274 INFO:tasks.rados.rados.0.smithi084.stdout:2427: done (0 left) 2024-09-20T22:51:39.274 INFO:tasks.rados.rados.0.smithi084.stdout:2428: read oid 25 snap -1 2024-09-20T22:51:39.274 INFO:tasks.rados.rados.0.smithi084.stdout:2428: expect (ObjNum 688 snap 212 seq_num 688) 2024-09-20T22:51:39.274 INFO:tasks.rados.rados.0.smithi084.stdout:2429: write (excl) oid 40 current snap is 269 2024-09-20T22:51:39.275 INFO:tasks.rados.rados.0.smithi084.stdout:2429: seq_num 833 ranges {400231=564894,1661971=338260} 2024-09-20T22:51:39.280 INFO:tasks.rados.rados.0.smithi084.stdout:2429: writing smithi08477203-40 from 400231 to 965125 tid 1 2024-09-20T22:51:39.287 INFO:tasks.rados.rados.0.smithi084.stdout:2429: writing smithi08477203-40 from 1661971 to 2000231 tid 2 2024-09-20T22:51:39.287 INFO:tasks.rados.rados.0.smithi084.stdout:2430: snap_remove snap 267 2024-09-20T22:51:39.367 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:51:39.388 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:39.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:39.389 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:51:39.490 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:51:39.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:39.512 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:39.512 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:51:39.545 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:51:39.559 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:51:39.559 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:51:39.559 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:39 smithi084 ceph-mon[40024]: pgmap v828: 158 pgs: 4 active+clean+snaptrim, 2 active+remapped+backfill_toofull, 152 active+clean; 456 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 33/1245 objects misplaced (2.651%) 2024-09-20T22:51:39.559 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:39 smithi084 ceph-mon[40024]: osdmap e619: 8 total, 8 up, 7 in 2024-09-20T22:51:39.560 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:39 smithi084 ceph-mon[29774]: pgmap v828: 158 pgs: 4 active+clean+snaptrim, 2 active+remapped+backfill_toofull, 152 active+clean; 456 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 33/1245 objects misplaced (2.651%) 2024-09-20T22:51:39.560 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:39 smithi084 ceph-mon[29774]: osdmap e619: 8 total, 8 up, 7 in 2024-09-20T22:51:39.578 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:51:39.578 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:51:39.595 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:51:39.595 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:51:39.611 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-20T22:51:39.611 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T22:51:39.613 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:51:39.628 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:51:39.628 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:51:39.637 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:39.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:39.638 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:51:39.645 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:51:39.645 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:51:39.661 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:51:39.662 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:51:39.679 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:51:39.679 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:51:39.738 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:51:39.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:39 smithi154 ceph-mon[33264]: pgmap v828: 158 pgs: 4 active+clean+snaptrim, 2 active+remapped+backfill_toofull, 152 active+clean; 456 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 33/1245 objects misplaced (2.651%) 2024-09-20T22:51:39.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:39 smithi154 ceph-mon[33264]: osdmap e619: 8 total, 8 up, 7 in 2024-09-20T22:51:39.754 INFO:journalctl@ceph.iscsi.iscsi.a.smithi154.stdout:Sep 20 22:51:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-iscsi-iscsi-a[59730]: debug there is no tcmu-runner data available 2024-09-20T22:51:39.760 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:39.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:39.761 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:51:39.861 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:51:39.882 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:39.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:39.883 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:51:39.984 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:51:40.010 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:40.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:40.011 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:51:40.111 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:51:40.132 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:40.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:40.133 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:51:40.210 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-20T22:51:40.210 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:51:40.234 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:51:40.260 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:40.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:40.261 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:51:40.276 INFO:tasks.rados.rados.0.smithi084.stdout:2430: done (2 left) 2024-09-20T22:51:40.276 INFO:tasks.rados.rados.0.smithi084.stdout:2431: snap_create 2024-09-20T22:51:40.276 INFO:tasks.rados.rados.0.smithi084.stdout:2429: finishing write tid 1 to smithi08477203-40 2024-09-20T22:51:40.276 INFO:tasks.rados.rados.0.smithi084.stdout:2429: finishing write tid 2 to smithi08477203-40 2024-09-20T22:51:40.277 INFO:tasks.rados.rados.0.smithi084.stdout:2429: finishing write tid 3 to smithi08477203-40 2024-09-20T22:51:40.277 INFO:tasks.rados.rados.0.smithi084.stdout:2428: expect (ObjNum 688 snap 212 seq_num 688) 2024-09-20T22:51:40.361 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:51:40.383 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:40.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:40.384 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:51:40.395 INFO:tasks.rados.rados.0.smithi084.stdout:2429: finishing write tid 4 to smithi08477203-40 2024-09-20T22:51:40.395 INFO:tasks.rados.rados.0.smithi084.stdout:2429: oid 40 updating version 0 to 1221 2024-09-20T22:51:40.395 INFO:tasks.rados.rados.0.smithi084.stdout:2429: oid 40 version 1221 is already newer than 1220 2024-09-20T22:51:40.395 INFO:tasks.rados.rados.0.smithi084.stdout:2429: oid 40 version 1221 is already newer than 1219 2024-09-20T22:51:40.395 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 40 v 1221 (ObjNum 833 snap 269 seq_num 833) dirty exists 2024-09-20T22:51:40.395 INFO:tasks.rados.rados.0.smithi084.stdout:2429: left oid 40 (ObjNum 833 snap 269 seq_num 833) 2024-09-20T22:51:40.485 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:51:40.512 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:40.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:40.513 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:51:40.613 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:51:40.639 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:40.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:40.641 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:51:40.741 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:51:40.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:40 smithi154 ceph-mon[33264]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:51:40.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:40 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/3213532572' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:51:40.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:40 smithi154 ceph-mon[33264]: osdmap e620: 8 total, 8 up, 7 in 2024-09-20T22:51:40.763 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:40.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:40.764 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:51:40.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:40 smithi084 ceph-mon[29774]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:51:40.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:40 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/3213532572' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:51:40.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:40 smithi084 ceph-mon[29774]: osdmap e620: 8 total, 8 up, 7 in 2024-09-20T22:51:40.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:40 smithi084 ceph-mon[40024]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:51:40.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:40 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/3213532572' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:51:40.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:40 smithi084 ceph-mon[40024]: osdmap e620: 8 total, 8 up, 7 in 2024-09-20T22:51:40.865 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:51:40.887 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:40.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:40.888 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:51:40.954 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:51:40.989 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:51:41.013 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:41.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:41.014 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:51:41.075 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-20T22:51:41.115 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:51:41.136 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:41.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:41.137 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:51:41.238 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:51:41.261 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:41.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:41.262 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:51:41.277 INFO:tasks.rados.rados.0.smithi084.stdout:2428: done (2 left) 2024-09-20T22:51:41.277 INFO:tasks.rados.rados.0.smithi084.stdout:2429: done (1 left) 2024-09-20T22:51:41.277 INFO:tasks.rados.rados.0.smithi084.stdout:2431: done (0 left) 2024-09-20T22:51:41.277 INFO:tasks.rados.rados.0.smithi084.stdout:2432: write (excl) oid 6 current snap is 270 2024-09-20T22:51:41.277 INFO:tasks.rados.rados.0.smithi084.stdout:2432: seq_num 834 ranges {417038=640143,1720453=296585} 2024-09-20T22:51:41.282 INFO:tasks.rados.rados.0.smithi084.stdout:2432: writing smithi08477203-6 from 417038 to 1057181 tid 1 2024-09-20T22:51:41.288 INFO:tasks.rados.rados.0.smithi084.stdout:2432: writing smithi08477203-6 from 1720453 to 2017038 tid 2 2024-09-20T22:51:41.288 INFO:tasks.rados.rados.0.smithi084.stdout:2433: write oid 16 current snap is 270 2024-09-20T22:51:41.288 INFO:tasks.rados.rados.0.smithi084.stdout:2432: finishing write tid 1 to smithi08477203-6 2024-09-20T22:51:41.288 INFO:tasks.rados.rados.0.smithi084.stdout:2433: seq_num 835 ranges {433845=715392,1662582=371263} 2024-09-20T22:51:41.294 INFO:tasks.rados.rados.0.smithi084.stdout:2433: writing smithi08477203-16 from 433845 to 1149237 tid 1 2024-09-20T22:51:41.300 INFO:tasks.rados.rados.0.smithi084.stdout:2433: writing smithi08477203-16 from 1662582 to 2033845 tid 2 2024-09-20T22:51:41.300 INFO:tasks.rados.rados.0.smithi084.stdout:2434: delete oid 29 current snap is 270 2024-09-20T22:51:41.300 INFO:tasks.rados.rados.0.smithi084.stdout:2432: finishing write tid 2 to smithi08477203-6 2024-09-20T22:51:41.300 INFO:tasks.rados.rados.0.smithi084.stdout:2432: finishing write tid 3 to smithi08477203-6 2024-09-20T22:51:41.300 INFO:tasks.rados.rados.0.smithi084.stdout:2432: finishing write tid 4 to smithi08477203-6 2024-09-20T22:51:41.300 INFO:tasks.rados.rados.0.smithi084.stdout:2432: oid 6 updating version 0 to 782 2024-09-20T22:51:41.300 INFO:tasks.rados.rados.0.smithi084.stdout:2432: oid 6 version 782 is already newer than 781 2024-09-20T22:51:41.300 INFO:tasks.rados.rados.0.smithi084.stdout:2432: oid 6 version 782 is already newer than 780 2024-09-20T22:51:41.300 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 6 v 782 (ObjNum 834 snap 270 seq_num 834) dirty exists 2024-09-20T22:51:41.300 INFO:tasks.rados.rados.0.smithi084.stdout:2432: left oid 6 (ObjNum 834 snap 270 seq_num 834) 2024-09-20T22:51:41.302 INFO:tasks.rados.rados.0.smithi084.stdout:2433: finishing write tid 1 to smithi08477203-16 2024-09-20T22:51:41.302 INFO:tasks.rados.rados.0.smithi084.stdout:2433: finishing write tid 2 to smithi08477203-16 2024-09-20T22:51:41.303 INFO:tasks.rados.rados.0.smithi084.stdout:2433: finishing write tid 3 to smithi08477203-16 2024-09-20T22:51:41.303 INFO:tasks.rados.rados.0.smithi084.stdout:2433: finishing write tid 4 to smithi08477203-16 2024-09-20T22:51:41.303 INFO:tasks.rados.rados.0.smithi084.stdout:2433: oid 16 updating version 0 to 440 2024-09-20T22:51:41.303 INFO:tasks.rados.rados.0.smithi084.stdout:2433: oid 16 updating version 440 to 441 2024-09-20T22:51:41.303 INFO:tasks.rados.rados.0.smithi084.stdout:2433: oid 16 updating version 441 to 442 2024-09-20T22:51:41.303 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 16 v 442 (ObjNum 835 snap 270 seq_num 835) dirty exists 2024-09-20T22:51:41.303 INFO:tasks.rados.rados.0.smithi084.stdout:2433: left oid 16 (ObjNum 835 snap 270 seq_num 835) 2024-09-20T22:51:41.304 INFO:tasks.rados.rados.0.smithi084.stdout:2432: done (2 left) 2024-09-20T22:51:41.304 INFO:tasks.rados.rados.0.smithi084.stdout:2433: done (1 left) 2024-09-20T22:51:41.304 INFO:tasks.rados.rados.0.smithi084.stdout:2434: done (0 left) 2024-09-20T22:51:41.304 INFO:tasks.rados.rados.0.smithi084.stdout:2435: rollback oid 25 current snap is 270 2024-09-20T22:51:41.304 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 25 to 250 2024-09-20T22:51:41.304 INFO:tasks.rados.rados.0.smithi084.stdout:2436: read oid 27 snap -1 2024-09-20T22:51:41.305 INFO:tasks.rados.rados.0.smithi084.stdout:2436: expect (ObjNum 714 snap 224 seq_num 714) 2024-09-20T22:51:41.305 INFO:tasks.rados.rados.0.smithi084.stdout:2437: read oid 16 snap -1 2024-09-20T22:51:41.305 INFO:tasks.rados.rados.0.smithi084.stdout:2437: expect (ObjNum 835 snap 270 seq_num 835) 2024-09-20T22:51:41.305 INFO:tasks.rados.rados.0.smithi084.stdout:2438: rollback oid 48 current snap is 270 2024-09-20T22:51:41.305 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 48 to 261 2024-09-20T22:51:41.305 INFO:tasks.rados.rados.0.smithi084.stdout:2439: snap_create 2024-09-20T22:51:41.307 INFO:tasks.rados.rados.0.smithi084.stdout:2438: finishing rollback tid 0 to smithi08477203-48 2024-09-20T22:51:41.307 INFO:tasks.rados.rados.0.smithi084.stdout:2438: finishing rollback tid 1 to smithi08477203-48 2024-09-20T22:51:41.307 INFO:tasks.rados.rados.0.smithi084.stdout:2438: finishing rollback tid 2 to smithi08477203-48 2024-09-20T22:51:41.307 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 48 v 866 (ObjNum 759 snap 243 seq_num 759) dirty exists 2024-09-20T22:51:41.309 INFO:tasks.rados.rados.0.smithi084.stdout:2435: finishing rollback tid 0 to smithi08477203-25 2024-09-20T22:51:41.309 INFO:tasks.rados.rados.0.smithi084.stdout:2435: finishing rollback tid 1 to smithi08477203-25 2024-09-20T22:51:41.310 INFO:tasks.rados.rados.0.smithi084.stdout:2435: finishing rollback tid 2 to smithi08477203-25 2024-09-20T22:51:41.310 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 25 v 1813 (ObjNum 688 snap 212 seq_num 688) dirty exists 2024-09-20T22:51:41.314 INFO:tasks.rados.rados.0.smithi084.stdout:2436: expect (ObjNum 714 snap 224 seq_num 714) 2024-09-20T22:51:41.363 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:51:41.377 INFO:tasks.rados.rados.0.smithi084.stdout:2437: expect (ObjNum 835 snap 270 seq_num 835) 2024-09-20T22:51:41.384 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:41.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:41.385 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:51:41.392 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-20T22:51:41.485 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:51:41.506 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:41.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:41.507 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:51:41.608 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:51:41.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:41 smithi084 ceph-mon[29774]: pgmap v831: 158 pgs: 2 active+clean+snaptrim, 2 active+remapped+backfill_toofull, 154 active+clean; 466 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 11 MiB/s rd, 2.3 MiB/s wr, 16 op/s; 36/1251 objects misplaced (2.878%) 2024-09-20T22:51:41.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:41 smithi084 ceph-mon[29774]: osdmap e621: 8 total, 8 up, 7 in 2024-09-20T22:51:41.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:41 smithi084 ceph-mon[40024]: pgmap v831: 158 pgs: 2 active+clean+snaptrim, 2 active+remapped+backfill_toofull, 154 active+clean; 466 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 11 MiB/s rd, 2.3 MiB/s wr, 16 op/s; 36/1251 objects misplaced (2.878%) 2024-09-20T22:51:41.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:41 smithi084 ceph-mon[40024]: osdmap e621: 8 total, 8 up, 7 in 2024-09-20T22:51:41.630 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:41.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:41.631 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:51:41.731 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:51:41.753 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:41.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:41 smithi154 ceph-mon[33264]: pgmap v831: 158 pgs: 2 active+clean+snaptrim, 2 active+remapped+backfill_toofull, 154 active+clean; 466 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 11 MiB/s rd, 2.3 MiB/s wr, 16 op/s; 36/1251 objects misplaced (2.878%) 2024-09-20T22:51:41.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:41 smithi154 ceph-mon[33264]: osdmap e621: 8 total, 8 up, 7 in 2024-09-20T22:51:41.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:41.755 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:51:41.855 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:51:41.877 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:41.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:41.878 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:51:41.884 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:51:41.925 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:51:41.979 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:51:42.000 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:42.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:42.001 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:51:42.038 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-20T22:51:42.040 INFO:teuthology.orchestra.run.smithi084.stderr:dumped all 2024-09-20T22:51:42.102 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:51:42.124 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:42.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:42.125 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:51:42.225 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:51:42.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:42.247 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:42.247 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:51:42.280 INFO:tasks.rados.rados.0.smithi084.stdout:2435: done (4 left) 2024-09-20T22:51:42.280 INFO:tasks.rados.rados.0.smithi084.stdout:2436: done (3 left) 2024-09-20T22:51:42.280 INFO:tasks.rados.rados.0.smithi084.stdout:2437: done (2 left) 2024-09-20T22:51:42.280 INFO:tasks.rados.rados.0.smithi084.stdout:2438: done (1 left) 2024-09-20T22:51:42.281 INFO:tasks.rados.rados.0.smithi084.stdout:2439: done (0 left) 2024-09-20T22:51:42.281 INFO:tasks.rados.rados.0.smithi084.stdout:2440: delete oid 3 current snap is 271 2024-09-20T22:51:42.282 INFO:tasks.rados.rados.0.smithi084.stdout:2440: done (0 left) 2024-09-20T22:51:42.282 INFO:tasks.rados.rados.0.smithi084.stdout:2441: copy_from oid 49 from oid 11 current snap is 271 2024-09-20T22:51:42.282 INFO:tasks.rados.rados.0.smithi084.stdout:2442: write (excl) oid 40 current snap is 271 2024-09-20T22:51:42.282 INFO:tasks.rados.rados.0.smithi084.stdout:2442: seq_num 838 ranges {484266=657492,1554381=529885} 2024-09-20T22:51:42.288 INFO:tasks.rados.rados.0.smithi084.stdout:2442: writing smithi08477203-40 from 484266 to 1141758 tid 1 2024-09-20T22:51:42.294 INFO:tasks.rados.rados.0.smithi084.stdout:2442: writing smithi08477203-40 from 1554381 to 2084266 tid 2 2024-09-20T22:51:42.294 INFO:tasks.rados.rados.0.smithi084.stdout:2443: write oid 27 current snap is 271 2024-09-20T22:51:42.294 INFO:tasks.rados.rados.0.smithi084.stdout:2443: seq_num 839 ranges {501073=732741,1896510=204563} 2024-09-20T22:51:42.302 INFO:tasks.rados.rados.0.smithi084.stdout:2443: writing smithi08477203-27 from 501073 to 1233814 tid 1 2024-09-20T22:51:42.307 INFO:tasks.rados.rados.0.smithi084.stdout:2443: writing smithi08477203-27 from 1896510 to 2101073 tid 2 2024-09-20T22:51:42.307 INFO:tasks.rados.rados.0.smithi084.stdout:2444: delete oid 46 current snap is 271 2024-09-20T22:51:42.307 INFO:tasks.rados.rados.0.smithi084.stdout:2442: finishing write tid 1 to smithi08477203-40 2024-09-20T22:51:42.308 INFO:tasks.rados.rados.0.smithi084.stdout:2441: finishing copy_from racing read to smithi08477203-49 2024-09-20T22:51:42.308 INFO:tasks.rados.rados.0.smithi084.stdout:2442: finishing write tid 2 to smithi08477203-40 2024-09-20T22:51:42.308 INFO:tasks.rados.rados.0.smithi084.stdout:2443: finishing write tid 1 to smithi08477203-27 2024-09-20T22:51:42.308 INFO:tasks.rados.rados.0.smithi084.stdout:2442: finishing write tid 3 to smithi08477203-40 2024-09-20T22:51:42.308 INFO:tasks.rados.rados.0.smithi084.stdout:2441: finishing copy_from to smithi08477203-49 2024-09-20T22:51:42.308 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 49 v 753 (ObjNum 815 snap 262 seq_num 815) dirty exists 2024-09-20T22:51:42.308 INFO:tasks.rados.rados.0.smithi084.stdout:2442: finishing write tid 4 to smithi08477203-40 2024-09-20T22:51:42.308 INFO:tasks.rados.rados.0.smithi084.stdout:2442: oid 40 updating version 0 to 1224 2024-09-20T22:51:42.308 INFO:tasks.rados.rados.0.smithi084.stdout:2442: oid 40 version 1224 is already newer than 1223 2024-09-20T22:51:42.308 INFO:tasks.rados.rados.0.smithi084.stdout:2442: oid 40 version 1224 is already newer than 1222 2024-09-20T22:51:42.308 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 40 v 1224 (ObjNum 838 snap 271 seq_num 838) dirty exists 2024-09-20T22:51:42.308 INFO:tasks.rados.rados.0.smithi084.stdout:2442: left oid 40 (ObjNum 838 snap 271 seq_num 838) 2024-09-20T22:51:42.309 INFO:tasks.rados.rados.0.smithi084.stdout:2443: finishing write tid 2 to smithi08477203-27 2024-09-20T22:51:42.309 INFO:tasks.rados.rados.0.smithi084.stdout:2443: finishing write tid 3 to smithi08477203-27 2024-09-20T22:51:42.309 INFO:tasks.rados.rados.0.smithi084.stdout:2441: done (3 left) 2024-09-20T22:51:42.309 INFO:tasks.rados.rados.0.smithi084.stdout:2442: done (2 left) 2024-09-20T22:51:42.309 INFO:tasks.rados.rados.0.smithi084.stdout:2444: done (1 left) 2024-09-20T22:51:42.310 INFO:tasks.rados.rados.0.smithi084.stdout:2445: rollback oid 15 current snap is 271 2024-09-20T22:51:42.310 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 15 to 250 2024-09-20T22:51:42.310 INFO:tasks.rados.rados.0.smithi084.stdout:2446: snap_create 2024-09-20T22:51:42.310 INFO:tasks.rados.rados.0.smithi084.stdout:2443: finishing write tid 4 to smithi08477203-27 2024-09-20T22:51:42.310 INFO:tasks.rados.rados.0.smithi084.stdout:2443: oid 27 updating version 0 to 736 2024-09-20T22:51:42.310 INFO:tasks.rados.rados.0.smithi084.stdout:2443: oid 27 version 736 is already newer than 735 2024-09-20T22:51:42.310 INFO:tasks.rados.rados.0.smithi084.stdout:2443: oid 27 updating version 736 to 737 2024-09-20T22:51:42.310 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 27 v 737 (ObjNum 839 snap 271 seq_num 839) dirty exists 2024-09-20T22:51:42.310 INFO:tasks.rados.rados.0.smithi084.stdout:2443: left oid 27 (ObjNum 839 snap 271 seq_num 839) 2024-09-20T22:51:42.311 INFO:tasks.rados.rados.0.smithi084.stdout:2445: finishing rollback tid 1 to smithi08477203-15 2024-09-20T22:51:42.311 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 15 v 798 (ObjNum 49 snap 0 seq_num 2090695488) dirty dne 2024-09-20T22:51:42.348 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:51:42.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:42.371 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:42.371 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:51:42.472 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:51:42.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:42.495 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:42.495 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:51:42.596 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:51:42.617 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:42.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:42.619 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:51:42.719 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:51:42.740 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:42.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:42.741 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:51:42.842 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:51:42.864 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:42.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:42.865 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:51:42.947 INFO:teuthology.orchestra.run.smithi084.stdout:{"pg_ready":true,"pg_map":{"version":834,"stamp":"2024-09-20T22:51:41.776809+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":488436395,"num_objects":417,"num_object_clones":171,"num_object_copies":1251,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":36,"num_objects_unfound":0,"num_objects_dirty":417,"num_whiteouts":10,"num_read":5556,"num_read_kb":4503084,"num_write":3432,"num_write_kb":1126152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1096,"num_bytes_recovered":428316364,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8420,"ondisk_log_size":8420,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":474,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":1391984,"kb_used_data":1052888,"kb_used_omap":10,"kb_used_meta":338933,"kb_avail":654824080,"statfs":{"total":671965249536,"available":670539857920,"internally_reserved":0,"allocated":1078157312,"data_stored":1505478768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11157,"internal_metadata":347067499},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":9813339,"num_objects":2,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":-1,"num_read":43,"num_read_kb":45819,"num_write":24,"num_write_kb":9221,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000778"},"pg_stats":[{"pgid":"3.1f","version":"86'2","reported_seq":482,"reported_epoch":618,"state":"active+clean","last_fresh":"2024-09-20T22:51:38.261485+0000","last_change":"2024-09-20T22:49:26.647633+0000","last_active":"2024-09-20T22:51:38.261485+0000","last_peered":"2024-09-20T22:51:38.261485+0000","last_clean":"2024-09-20T22:51:38.261485+0000","last_became_active":"2024-09-20T22:47:49.222114+0000","last_became_peered":"2024-09-20T22:47:49.222114+0000","last_unstale":"2024-09-20T22:51:38.261485+0000","last_undegraded":"2024-09-20T22:51:38.261485+0000","last_fullsized":"2024-09-20T22:51:38.261485+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'2","last_scrub_stamp":"2024-09-20T22:49:26.647557+0000","last_deep_scrub":"86'2","last_deep_scrub_stamp":"2024-09-20T22:42:39.664421+0000","last_clean_scrub_stamp":"2024-09-20T22:49:26.647557+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":436,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"74'9","reported_seq":472,"reported_epoch":618,"state":"active+clean","last_fresh":"2024-09-20T22:51:38.264956+0000","last_change":"2024-09-20T22:51:20.301883+0000","last_active":"2024-09-20T22:51:38.264956+0000","last_peered":"2024-09-20T22:51:38.264956+0000","last_clean":"2024-09-20T22:51:38.264956+0000","last_became_active":"2024-09-20T22:51:20.286651+0000","last_became_peered":"2024-09-20T22:51:20.286651+0000","last_unstale":"2024-09-20T22:51:38.264956+0000","last_undegraded":"2024-09-20T22:51:38.264956+0000","last_fullsized":"2024-09-20T22:51:38.264956+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-09-20T22:49:51.091416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:49:51.091416+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":46,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-20T22:51:37.491360+0000","last_change":"2024-09-20T22:51:20.291625+0000","last_active":"2024-09-20T22:51:37.491360+0000","last_peered":"2024-09-20T22:51:37.491360+0000","last_clean":"2024-09-20T22:51:37.491360+0000","last_became_active":"2024-09-20T22:51:20.291272+0000","last_became_peered":"2024-09-20T22:51:20.291272+0000","last_unstale":"2024-09-20T22:51:37.491360+0000","last_undegraded":"2024-09-20T22:51:37.491360+0000","last_fullsized":"2024-09-20T22:51:37.491360+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:50:12.718677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:50:12.718677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":460,"reported_epoch":618,"state":"active+clean","last_fresh":"2024-09-20T22:51:38.265307+0000","last_change":"2024-09-20T22:51:36.730087+0000","last_active":"2024-09-20T22:51:38.265307+0000","last_peered":"2024-09-20T22:51:38.265307+0000","last_clean":"2024-09-20T22:51:38.265307+0000","last_became_active":"2024-09-20T22:51:20.295608+0000","last_became_peered":"2024-09-20T22:51:20.295608+0000","last_unstale":"2024-09-20T22:51:38.265307+0000","last_undegraded":"2024-09-20T22:51:38.265307+0000","last_fullsized":"2024-09-20T22:51:38.265307+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:36.730015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:51:36.730015+0000","last_clean_scrub_stamp":"2024-09-20T22:51:36.730015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:41.980409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":1131,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-20T22:51:37.498140+0000","last_change":"2024-09-20T22:51:01.404127+0000","last_active":"2024-09-20T22:51:37.498140+0000","last_peered":"2024-09-20T22:51:37.498140+0000","last_clean":"2024-09-20T22:51:37.498140+0000","last_became_active":"2024-09-20T22:41:44.832099+0000","last_became_peered":"2024-09-20T22:41:44.832099+0000","last_unstale":"2024-09-20T22:51:37.498140+0000","last_undegraded":"2024-09-20T22:51:37.498140+0000","last_fullsized":"2024-09-20T22:51:37.498140+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:01.404063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:51:01.404063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:15.377506+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":1136,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-20T22:51:37.498996+0000","last_change":"2024-09-20T22:51:12.418041+0000","last_active":"2024-09-20T22:51:37.498996+0000","last_peered":"2024-09-20T22:51:37.498996+0000","last_clean":"2024-09-20T22:51:37.498996+0000","last_became_active":"2024-09-20T22:41:38.157298+0000","last_became_peered":"2024-09-20T22:41:38.157298+0000","last_unstale":"2024-09-20T22:51:37.498996+0000","last_undegraded":"2024-09-20T22:51:37.498996+0000","last_fullsized":"2024-09-20T22:51:37.498996+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:12.417920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:51:12.417920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:37.156764+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"74'15","reported_seq":1201,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-20T22:51:37.497965+0000","last_change":"2024-09-20T22:51:08.441927+0000","last_active":"2024-09-20T22:51:37.497965+0000","last_peered":"2024-09-20T22:51:37.497965+0000","last_clean":"2024-09-20T22:51:37.497965+0000","last_became_active":"2024-09-20T22:47:49.218798+0000","last_became_peered":"2024-09-20T22:47:49.218798+0000","last_unstale":"2024-09-20T22:51:37.497965+0000","last_undegraded":"2024-09-20T22:51:37.497965+0000","last_fullsized":"2024-09-20T22:51:37.497965+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-09-20T22:51:08.441788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:51:08.441788+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:25.517854+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":452,"reported_epoch":618,"state":"active+clean","last_fresh":"2024-09-20T22:51:38.264789+0000","last_change":"2024-09-20T22:51:20.287492+0000","last_active":"2024-09-20T22:51:38.264789+0000","last_peered":"2024-09-20T22:51:38.264789+0000","last_clean":"2024-09-20T22:51:38.264789+0000","last_became_active":"2024-09-20T22:51:20.287060+0000","last_became_peered":"2024-09-20T22:51:20.287060+0000","last_unstale":"2024-09-20T22:51:38.264789+0000","last_undegraded":"2024-09-20T22:51:38.264789+0000","last_fullsized":"2024-09-20T22:51:38.264789+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:59.011829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:49:59.011829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":1156,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-20T22:51:37.491912+0000","last_change":"2024-09-20T22:51:09.684125+0000","last_active":"2024-09-20T22:51:37.491912+0000","last_peered":"2024-09-20T22:51:37.491912+0000","last_clean":"2024-09-20T22:51:37.491912+0000","last_became_active":"2024-09-20T22:48:05.242059+0000","last_became_peered":"2024-09-20T22:48:05.242059+0000","last_unstale":"2024-09-20T22:51:37.491912+0000","last_undegraded":"2024-09-20T22:51:37.491912+0000","last_fullsized":"2024-09-20T22:51:37.491912+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:09.684053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:46:00.999939+0000","last_clean_scrub_stamp":"2024-09-20T22:51:09.684053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:35.986257+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"74'9","reported_seq":464,"reported_epoch":618,"state":"active+clean","last_fresh":"2024-09-20T22:51:38.265352+0000","last_change":"2024-09-20T22:51:30.710329+0000","last_active":"2024-09-20T22:51:38.265352+0000","last_peered":"2024-09-20T22:51:38.265352+0000","last_clean":"2024-09-20T22:51:38.265352+0000","last_became_active":"2024-09-20T22:48:05.233429+0000","last_became_peered":"2024-09-20T22:48:05.233429+0000","last_unstale":"2024-09-20T22:51:38.265352+0000","last_undegraded":"2024-09-20T22:51:38.265352+0000","last_fullsized":"2024-09-20T22:51:38.265352+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-09-20T22:51:30.710245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:51:30.710245+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:39.082734+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":1148,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-20T22:51:37.492023+0000","last_change":"2024-09-20T22:51:33.977328+0000","last_active":"2024-09-20T22:51:37.492023+0000","last_peered":"2024-09-20T22:51:37.492023+0000","last_clean":"2024-09-20T22:51:37.492023+0000","last_became_active":"2024-09-20T22:47:49.227379+0000","last_became_peered":"2024-09-20T22:47:49.227379+0000","last_unstale":"2024-09-20T22:51:37.492023+0000","last_undegraded":"2024-09-20T22:51:37.492023+0000","last_fullsized":"2024-09-20T22:51:37.492023+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:33.977257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:51:33.977257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:37.924731+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":492,"reported_epoch":618,"state":"active+clean","last_fresh":"2024-09-20T22:51:38.260221+0000","last_change":"2024-09-20T22:51:35.899942+0000","last_active":"2024-09-20T22:51:38.260221+0000","last_peered":"2024-09-20T22:51:38.260221+0000","last_clean":"2024-09-20T22:51:38.260221+0000","last_became_active":"2024-09-20T22:51:20.277465+0000","last_became_peered":"2024-09-20T22:51:20.277465+0000","last_unstale":"2024-09-20T22:51:38.260221+0000","last_undegraded":"2024-09-20T22:51:38.260221+0000","last_fullsized":"2024-09-20T22:51:38.260221+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:35.899860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:49:08.664407+0000","last_clean_scrub_stamp":"2024-09-20T22:51:35.899860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:38.548132+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":1163,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-20T22:51:37.491561+0000","last_change":"2024-09-20T22:51:20.298096+0000","last_active":"2024-09-20T22:51:37.491561+0000","last_peered":"2024-09-20T22:51:37.491561+0000","last_clean":"2024-09-20T22:51:37.491561+0000","last_became_active":"2024-09-20T22:51:20.297787+0000","last_became_peered":"2024-09-20T22:51:20.297787+0000","last_unstale":"2024-09-20T22:51:37.491561+0000","last_undegraded":"2024-09-20T22:51:37.491561+0000","last_fullsized":"2024-09-20T22:51:37.491561+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:50:17.735354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:50:17.735354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:51:44.179143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"181'7","reported_seq":472,"reported_epoch":618,"state":"active+clean","last_fresh":"2024-09-20T22:51:38.265950+0000","last_change":"2024-09-20T22:51:20.301034+0000","last_active":"2024-09-20T22:51:38.265950+0000","last_peered":"2024-09-20T22:51:38.265950+0000","last_clean":"2024-09-20T22:51:38.265950+0000","last_became_active":"2024-09-20T22:51:20.285795+0000","last_became_peered":"2024-09-20T22:51:20.285795+0000","last_unstale":"2024-09-20T22:51:38.265950+0000","last_undegraded":"2024-09-20T22:51:38.265950+0000","last_fullsized":"2024-09-20T22:51:38.265950+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"181'7","last_scrub_stamp":"2024-09-20T22:51:05.899205+0000","last_deep_scrub":"181'7","last_deep_scrub_stamp":"2024-09-20T22:48:41.049381+0000","last_clean_scrub_stamp":"2024-09-20T22:51:05.899205+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:14.403238+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":6,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":1169,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-20T22:51:37.538686+0000","last_change":"2024-09-20T22:51:20.285670+0000","last_active":"2024-09-20T22:51:37.538686+0000","last_peered":"2024-09-20T22:51:37.538686+0000","last_clean":"2024-09-20T22:51:37.538686+0000","last_became_active":"2024-09-20T22:51:20.285368+0000","last_became_peered":"2024-09-20T22:51:20.285368+0000","last_unstale":"2024-09-20T22:51:37.538686+0000","last_undegraded":"2024-09-20T22:51:37.538686+0000","last_fullsized":"2024-09-20T22:51:37.538686+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:48.313860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:47:40.361958+0000","last_clean_scrub_stamp":"2024-09-20T22:49:48.313860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:51:43.458719+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":1142,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-20T22:51:37.491393+0000","last_change":"2024-09-20T22:51:20.279625+0000","last_active":"2024-09-20T22:51:37.491393+0000","last_peered":"2024-09-20T22:51:37.491393+0000","last_clean":"2024-09-20T22:51:37.491393+0000","last_became_active":"2024-09-20T22:51:20.279307+0000","last_became_peered":"2024-09-20T22:51:20.279307+0000","last_unstale":"2024-09-20T22:51:37.491393+0000","last_undegraded":"2024-09-20T22:51:37.491393+0000","last_fullsized":"2024-09-20T22:51:37.491393+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:04.636900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:51:04.636900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:18.188059+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":47,"reported_epoch":618,"state":"active+clean","last_fresh":"2024-09-20T22:51:38.265648+0000","last_change":"2024-09-20T22:51:20.282749+0000","last_active":"2024-09-20T22:51:38.265648+0000","last_peered":"2024-09-20T22:51:38.265648+0000","last_clean":"2024-09-20T22:51:38.265648+0000","last_became_active":"2024-09-20T22:51:20.282393+0000","last_became_peered":"2024-09-20T22:51:20.282393+0000","last_unstale":"2024-09-20T22:51:38.265648+0000","last_undegraded":"2024-09-20T22:51:38.265648+0000","last_fullsized":"2024-09-20T22:51:38.265648+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:50:16.816835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:50:16.816835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":484,"reported_epoch":618,"state":"active+clean","last_fresh":"2024-09-20T22:51:38.260740+0000","last_change":"2024-09-20T22:49:53.933673+0000","last_active":"2024-09-20T22:51:38.260740+0000","last_peered":"2024-09-20T22:51:38.260740+0000","last_clean":"2024-09-20T22:51:38.260740+0000","last_became_active":"2024-09-20T22:48:05.233687+0000","last_became_peered":"2024-09-20T22:48:05.233687+0000","last_unstale":"2024-09-20T22:51:38.260740+0000","last_undegraded":"2024-09-20T22:51:38.260740+0000","last_fullsized":"2024-09-20T22:51:38.260740+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:53.933601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:49:53.933601+0000","last_clean_scrub_stamp":"2024-09-20T22:49:53.933601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"113'22","reported_seq":1209,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-20T22:51:37.489581+0000","last_change":"2024-09-20T22:51:20.289052+0000","last_active":"2024-09-20T22:51:37.489581+0000","last_peered":"2024-09-20T22:51:37.489581+0000","last_clean":"2024-09-20T22:51:37.489581+0000","last_became_active":"2024-09-20T22:51:20.276338+0000","last_became_peered":"2024-09-20T22:51:20.276338+0000","last_unstale":"2024-09-20T22:51:37.489581+0000","last_undegraded":"2024-09-20T22:51:37.489581+0000","last_fullsized":"2024-09-20T22:51:37.489581+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'22","last_scrub_stamp":"2024-09-20T22:51:08.280738+0000","last_deep_scrub":"113'22","last_deep_scrub_stamp":"2024-09-20T22:43:12.191878+0000","last_clean_scrub_stamp":"2024-09-20T22:51:08.280738+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:26.717275+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":759,"reported_epoch":618,"state":"active+clean","last_fresh":"2024-09-20T22:51:38.265131+0000","last_change":"2024-09-20T22:51:20.295989+0000","last_active":"2024-09-20T22:51:38.265131+0000","last_peered":"2024-09-20T22:51:38.265131+0000","last_clean":"2024-09-20T22:51:38.265131+0000","last_became_active":"2024-09-20T22:51:20.295564+0000","last_became_peered":"2024-09-20T22:51:20.295564+0000","last_unstale":"2024-09-20T22:51:38.265131+0000","last_undegraded":"2024-09-20T22:51:38.265131+0000","last_fullsized":"2024-09-20T22:51:38.265131+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:50:40.784154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:50:40.784154+0000","last_clean_scrub_stamp":"2024-09-20T22:50:40.784154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:51:44.309566+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":1124,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-20T22:51:37.498935+0000","last_change":"2024-09-20T22:51:13.385618+0000","last_active":"2024-09-20T22:51:37.498935+0000","last_peered":"2024-09-20T22:51:37.498935+0000","last_clean":"2024-09-20T22:51:37.498935+0000","last_became_active":"2024-09-20T22:41:44.832131+0000","last_became_peered":"2024-09-20T22:41:44.832131+0000","last_unstale":"2024-09-20T22:51:37.498935+0000","last_undegraded":"2024-09-20T22:51:37.498935+0000","last_fullsized":"2024-09-20T22:51:37.498935+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:13.385520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:45:25.002199+0000","last_clean_scrub_stamp":"2024-09-20T22:51:13.385520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:13.444102+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":460,"reported_epoch":618,"state":"active+clean","last_fresh":"2024-09-20T22:51:38.265505+0000","last_change":"2024-09-20T22:51:20.285019+0000","last_active":"2024-09-20T22:51:38.265505+0000","last_peered":"2024-09-20T22:51:38.265505+0000","last_clean":"2024-09-20T22:51:38.265505+0000","last_became_active":"2024-09-20T22:51:20.284621+0000","last_became_peered":"2024-09-20T22:51:20.284621+0000","last_unstale":"2024-09-20T22:51:38.265505+0000","last_undegraded":"2024-09-20T22:51:38.265505+0000","last_fullsized":"2024-09-20T22:51:38.265505+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:50:41.885844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:50:41.885844+0000","last_clean_scrub_stamp":"2024-09-20T22:50:41.885844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:11.439110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"74'12","reported_seq":1226,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-20T22:51:37.498615+0000","last_change":"2024-09-20T22:51:20.300828+0000","last_active":"2024-09-20T22:51:37.498615+0000","last_peered":"2024-09-20T22:51:37.498615+0000","last_clean":"2024-09-20T22:51:37.498615+0000","last_became_active":"2024-09-20T22:51:20.278886+0000","last_became_peered":"2024-09-20T22:51:20.278886+0000","last_unstale":"2024-09-20T22:51:37.498615+0000","last_undegraded":"2024-09-20T22:51:37.498615+0000","last_fullsized":"2024-09-20T22:51:37.498615+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-09-20T22:51:03.380934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:51:03.380934+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:30.819992+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":454,"reported_epoch":618,"state":"active+clean","last_fresh":"2024-09-20T22:51:38.265308+0000","last_change":"2024-09-20T22:50:42.864153+0000","last_active":"2024-09-20T22:51:38.265308+0000","last_peered":"2024-09-20T22:51:38.265308+0000","last_clean":"2024-09-20T22:51:38.265308+0000","last_became_active":"2024-09-20T22:48:05.250319+0000","last_became_peered":"2024-09-20T22:48:05.250319+0000","last_unstale":"2024-09-20T22:51:38.265308+0000","last_undegraded":"2024-09-20T22:51:38.265308+0000","last_fullsized":"2024-09-20T22:51:38.265308+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:50:42.864078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:50:42.864078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:51:59.926865+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"154'2","reported_seq":1133,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-20T22:51:37.537675+0000","last_change":"2024-09-20T22:50:34.375264+0000","last_active":"2024-09-20T22:51:37.537675+0000","last_peered":"2024-09-20T22:51:37.537675+0000","last_clean":"2024-09-20T22:51:37.537675+0000","last_became_active":"2024-09-20T22:41:44.199822+0000","last_became_peered":"2024-09-20T22:41:44.199822+0000","last_unstale":"2024-09-20T22:51:37.537675+0000","last_undegraded":"2024-09-20T22:51:37.537675+0000","last_fullsized":"2024-09-20T22:51:37.537675+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"154'2","last_scrub_stamp":"2024-09-20T22:50:34.375161+0000","last_deep_scrub":"154'2","last_deep_scrub_stamp":"2024-09-20T22:43:54.134066+0000","last_clean_scrub_stamp":"2024-09-20T22:50:34.375161+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:51:58.851029+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"67'1","reported_seq":57,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-20T22:51:37.497854+0000","last_change":"2024-09-20T22:51:20.303293+0000","last_active":"2024-09-20T22:51:37.497854+0000","last_peered":"2024-09-20T22:51:37.497854+0000","last_clean":"2024-09-20T22:51:37.497854+0000","last_became_active":"2024-09-20T22:51:20.279978+0000","last_became_peered":"2024-09-20T22:51:20.279978+0000","last_unstale":"2024-09-20T22:51:37.497854+0000","last_undegraded":"2024-09-20T22:51:37.497854+0000","last_fullsized":"2024-09-20T22:51:37.497854+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-09-20T22:50:42.717953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:50:42.717953+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:51:48.754094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"151'15","reported_seq":498,"reported_epoch":618,"state":"active+clean","last_fresh":"2024-09-20T22:51:38.261101+0000","last_change":"2024-09-20T22:51:02.707652+0000","last_active":"2024-09-20T22:51:38.261101+0000","last_peered":"2024-09-20T22:51:38.261101+0000","last_clean":"2024-09-20T22:51:38.261101+0000","last_became_active":"2024-09-20T22:47:49.232181+0000","last_became_peered":"2024-09-20T22:47:49.232181+0000","last_unstale":"2024-09-20T22:51:38.261101+0000","last_undegraded":"2024-09-20T22:51:38.261101+0000","last_fullsized":"2024-09-20T22:51:38.261101+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"151'15","last_scrub_stamp":"2024-09-20T22:51:02.707574+0000","last_deep_scrub":"151'15","last_deep_scrub_stamp":"2024-09-20T22:43:50.715761+0000","last_clean_scrub_stamp":"2024-09-20T22:51:02.707574+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:03.970959+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"408'10","reported_seq":1301,"reported_epoch":618,"state":"active+clean","last_fresh":"2024-09-20T22:51:38.581949+0000","last_change":"2024-09-20T22:51:03.967383+0000","last_active":"2024-09-20T22:51:38.581949+0000","last_peered":"2024-09-20T22:51:38.581949+0000","last_clean":"2024-09-20T22:51:38.581949+0000","last_became_active":"2024-09-20T22:48:05.244270+0000","last_became_peered":"2024-09-20T22:48:05.244270+0000","last_unstale":"2024-09-20T22:51:38.581949+0000","last_undegraded":"2024-09-20T22:51:38.581949+0000","last_fullsized":"2024-09-20T22:51:38.581949+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"408'10","last_scrub_stamp":"2024-09-20T22:51:03.967298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:51:03.967298+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:12.600235+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.c","version":"617'1801","reported_seq":4368,"reported_epoch":618,"state":"active+clean","last_fresh":"2024-09-20T22:51:38.581885+0000","last_change":"2024-09-20T22:51:37.541853+0000","last_active":"2024-09-20T22:51:38.581885+0000","last_peered":"2024-09-20T22:51:38.581885+0000","last_clean":"2024-09-20T22:51:38.581885+0000","last_became_active":"2024-09-20T22:42:52.404027+0000","last_became_peered":"2024-09-20T22:42:52.404027+0000","last_unstale":"2024-09-20T22:51:38.581885+0000","last_undegraded":"2024-09-20T22:51:38.581885+0000","last_fullsized":"2024-09-20T22:51:38.581885+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"584'1653","last_scrub_stamp":"2024-09-20T22:51:04.993550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:51:04.993550+0000","objects_scrubbed":39,"log_size":1801,"log_dups_size":0,"ondisk_log_size":1801,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:34.749662+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0.0030727459999999999,"stat_sum":{"num_bytes":76475341,"num_objects":34,"num_object_clones":27,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":2,"num_read":764,"num_read_kb":808528,"num_write":516,"num_write_kb":202441,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"a7","length":"1"},{"start":"c5","length":"1"},{"start":"ce","length":"1"},{"start":"d4","length":"1"},{"start":"da","length":"1"},{"start":"df","length":"1"},{"start":"e3","length":"9"},{"start":"ee","length":"4"},{"start":"f3","length":"2"},{"start":"f6","length":"2"},{"start":"f9","length":"1"},{"start":"fe","length":"3"},{"start":"106","length":"1"},{"start":"109","length":"1"}]},{"pgid":"1.a","version":"77'128","reported_seq":1405,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-20T22:51:37.491150+0000","last_change":"2024-09-20T22:50:19.676094+0000","last_active":"2024-09-20T22:51:37.491150+0000","last_peered":"2024-09-20T22:51:37.491150+0000","last_clean":"2024-09-20T22:51:37.491150+0000","last_became_active":"2024-09-20T22:42:52.407791+0000","last_became_peered":"2024-09-20T22:42:52.407791+0000","last_unstale":"2024-09-20T22:51:37.491150+0000","last_undegraded":"2024-09-20T22:51:37.491150+0000","last_fullsized":"2024-09-20T22:51:37.491150+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:50:19.675992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:50:19.675992+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:51:39.072482+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"74'15","reported_seq":105,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-20T22:51:37.499021+0000","last_change":"2024-09-20T22:51:20.309002+0000","last_active":"2024-09-20T22:51:37.499021+0000","last_peered":"2024-09-20T22:51:37.499021+0000","last_clean":"2024-09-20T22:51:37.499021+0000","last_became_active":"2024-09-20T22:51:20.287083+0000","last_became_peered":"2024-09-20T22:51:20.287083+0000","last_unstale":"2024-09-20T22:51:37.499021+0000","last_undegraded":"2024-09-20T22:51:37.499021+0000","last_fullsized":"2024-09-20T22:51:37.499021+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-09-20T22:49:55.709043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:49:55.709043+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":1142,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-20T22:51:37.497670+0000","last_change":"2024-09-20T22:51:29.394206+0000","last_active":"2024-09-20T22:51:37.497670+0000","last_peered":"2024-09-20T22:51:37.497670+0000","last_clean":"2024-09-20T22:51:37.497670+0000","last_became_active":"2024-09-20T22:51:20.279439+0000","last_became_peered":"2024-09-20T22:51:20.279439+0000","last_unstale":"2024-09-20T22:51:37.497670+0000","last_undegraded":"2024-09-20T22:51:37.497670+0000","last_fullsized":"2024-09-20T22:51:37.497670+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:29.394097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:49:12.070675+0000","last_clean_scrub_stamp":"2024-09-20T22:51:29.394097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:57.584257+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"408'12","reported_seq":493,"reported_epoch":618,"state":"active+clean","last_fresh":"2024-09-20T22:51:38.265867+0000","last_change":"2024-09-20T22:50:22.967349+0000","last_active":"2024-09-20T22:51:38.265867+0000","last_peered":"2024-09-20T22:51:38.265867+0000","last_clean":"2024-09-20T22:51:38.265867+0000","last_became_active":"2024-09-20T22:48:05.242914+0000","last_became_peered":"2024-09-20T22:48:05.242914+0000","last_unstale":"2024-09-20T22:51:38.265867+0000","last_undegraded":"2024-09-20T22:51:38.265867+0000","last_fullsized":"2024-09-20T22:51:38.265867+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"408'12","last_scrub_stamp":"2024-09-20T22:50:22.967270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:50:22.967270+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:51:47.860151+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":1151,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-20T22:51:37.491362+0000","last_change":"2024-09-20T22:51:31.960415+0000","last_active":"2024-09-20T22:51:37.491362+0000","last_peered":"2024-09-20T22:51:37.491362+0000","last_clean":"2024-09-20T22:51:37.491362+0000","last_became_active":"2024-09-20T22:51:20.280145+0000","last_became_peered":"2024-09-20T22:51:20.280145+0000","last_unstale":"2024-09-20T22:51:37.491362+0000","last_undegraded":"2024-09-20T22:51:37.491362+0000","last_fullsized":"2024-09-20T22:51:37.491362+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:31.960331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:51:31.960331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:46.560339+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3","version":"616'247","reported_seq":2731,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-20T22:51:38.493294+0000","last_change":"2024-09-20T22:51:38.493480+0000","last_active":"2024-09-20T22:51:38.493294+0000","last_peered":"2024-09-20T22:51:38.493294+0000","last_clean":"2024-09-20T22:51:38.493294+0000","last_became_active":"2024-09-20T22:49:05.221402+0000","last_became_peered":"2024-09-20T22:49:05.221402+0000","last_unstale":"2024-09-20T22:51:38.493294+0000","last_undegraded":"2024-09-20T22:51:38.493294+0000","last_fullsized":"2024-09-20T22:51:38.493294+0000","mapping_epoch":404,"log_start":"456'129","ondisk_log_start":"456'129","created":94,"last_epoch_clean":465,"parent":"0.0","parent_split_bits":0,"last_scrub":"609'243","last_scrub_stamp":"2024-09-20T22:51:31.402013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:51:31.402013+0000","objects_scrubbed":6,"log_size":118,"log_dups_size":149,"ondisk_log_size":118,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:41.128646+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.99525253000000002,"stat_sum":{"num_bytes":13665397,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":89,"num_read_kb":117349,"num_write":56,"num_write_kb":21007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":20196357,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"a7","length":"1"},{"start":"c5","length":"1"},{"start":"ce","length":"1"},{"start":"d4","length":"1"},{"start":"da","length":"1"},{"start":"df","length":"1"},{"start":"e3","length":"9"},{"start":"ee","length":"4"},{"start":"f3","length":"2"},{"start":"f6","length":"2"},{"start":"f9","length":"1"},{"start":"fe","length":"3"},{"start":"106","length":"1"},{"start":"109","length":"1"}]},{"pgid":"1.5","version":"77'128","reported_seq":1313,"reported_epoch":618,"state":"active+clean","last_fresh":"2024-09-20T22:51:38.265654+0000","last_change":"2024-09-20T22:51:12.927535+0000","last_active":"2024-09-20T22:51:38.265654+0000","last_peered":"2024-09-20T22:51:38.265654+0000","last_clean":"2024-09-20T22:51:38.265654+0000","last_became_active":"2024-09-20T22:48:08.207018+0000","last_became_peered":"2024-09-20T22:48:08.207018+0000","last_unstale":"2024-09-20T22:51:38.265654+0000","last_undegraded":"2024-09-20T22:51:38.265654+0000","last_fullsized":"2024-09-20T22:51:38.265654+0000","mapping_epoch":407,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":408,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:51:12.927463+0000","last_deep_scrub":"77'128","last_deep_scrub_stamp":"2024-09-20T22:45:41.608075+0000","last_clean_scrub_stamp":"2024-09-20T22:51:12.927463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:28.499830+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"74'18","reported_seq":1212,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-20T22:51:37.497830+0000","last_change":"2024-09-20T22:50:45.270518+0000","last_active":"2024-09-20T22:51:37.497830+0000","last_peered":"2024-09-20T22:51:37.497830+0000","last_clean":"2024-09-20T22:51:37.497830+0000","last_became_active":"2024-09-20T22:47:49.217605+0000","last_became_peered":"2024-09-20T22:47:49.217605+0000","last_unstale":"2024-09-20T22:51:37.497830+0000","last_undegraded":"2024-09-20T22:51:37.497830+0000","last_fullsized":"2024-09-20T22:51:37.497830+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'18","last_scrub_stamp":"2024-09-20T22:50:45.270399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:50:45.270399+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:03.171298+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":1148,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-20T22:51:37.499132+0000","last_change":"2024-09-20T22:50:22.220104+0000","last_active":"2024-09-20T22:51:37.499132+0000","last_peered":"2024-09-20T22:51:37.499132+0000","last_clean":"2024-09-20T22:51:37.499132+0000","last_became_active":"2024-09-20T22:47:49.227069+0000","last_became_peered":"2024-09-20T22:47:49.227069+0000","last_unstale":"2024-09-20T22:51:37.499132+0000","last_undegraded":"2024-09-20T22:51:37.499132+0000","last_fullsized":"2024-09-20T22:51:37.499132+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:50:22.219983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:50:22.219983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:51:42.667115+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":454,"reported_epoch":618,"state":"active+clean","last_fresh":"2024-09-20T22:51:38.265887+0000","last_change":"2024-09-20T22:51:09.901939+0000","last_active":"2024-09-20T22:51:38.265887+0000","last_peered":"2024-09-20T22:51:38.265887+0000","last_clean":"2024-09-20T22:51:38.265887+0000","last_became_active":"2024-09-20T22:48:05.230003+0000","last_became_peered":"2024-09-20T22:48:05.230003+0000","last_unstale":"2024-09-20T22:51:38.265887+0000","last_undegraded":"2024-09-20T22:51:38.265887+0000","last_fullsized":"2024-09-20T22:51:38.265887+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:09.901838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:07.947945+0000","last_clean_scrub_stamp":"2024-09-20T22:51:09.901838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:27.938229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":446,"reported_epoch":618,"state":"active+clean","last_fresh":"2024-09-20T22:51:38.265440+0000","last_change":"2024-09-20T22:50:16.143396+0000","last_active":"2024-09-20T22:51:38.265440+0000","last_peered":"2024-09-20T22:51:38.265440+0000","last_clean":"2024-09-20T22:51:38.265440+0000","last_became_active":"2024-09-20T22:48:05.239917+0000","last_became_peered":"2024-09-20T22:48:05.239917+0000","last_unstale":"2024-09-20T22:51:38.265440+0000","last_undegraded":"2024-09-20T22:51:38.265440+0000","last_fullsized":"2024-09-20T22:51:38.265440+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:50:16.143329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:53.036601+0000","last_clean_scrub_stamp":"2024-09-20T22:50:16.143329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2","version":"616'656","reported_seq":2436,"reported_epoch":617,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-20T22:51:37.498323+0000","last_change":"2024-09-20T22:51:21.511920+0000","last_active":"2024-09-20T22:51:37.498323+0000","last_peered":"2024-09-20T22:51:37.498323+0000","last_clean":"2024-09-20T22:51:18.665603+0000","last_became_active":"2024-09-20T22:51:21.509843+0000","last_became_peered":"2024-09-20T22:51:21.509843+0000","last_unstale":"2024-09-20T22:51:37.498323+0000","last_undegraded":"2024-09-20T22:51:37.498323+0000","last_fullsized":"2024-09-20T22:51:37.498323+0000","mapping_epoch":600,"log_start":"471'464","ondisk_log_start":"471'464","created":94,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"282'120","last_scrub_stamp":"2024-09-20T22:46:22.099642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:46:22.099642+0000","objects_scrubbed":1,"log_size":192,"log_dups_size":687,"ondisk_log_size":192,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":29,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.96265287200000005,"stat_sum":{"num_bytes":39045299,"num_objects":20,"num_object_clones":16,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":20,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":2,"num_read":361,"num_read_kb":386474,"num_write":218,"num_write_kb":79509,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":120,"num_bytes_recovered":49828787,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,1],"avail_no_missing":["3","1","2"],"object_location_counts":[{"shards":"1,2,3","objects":20}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"77'128","reported_seq":1429,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-20T22:51:37.498814+0000","last_change":"2024-09-20T22:49:58.063457+0000","last_active":"2024-09-20T22:51:37.498814+0000","last_peered":"2024-09-20T22:51:37.498814+0000","last_clean":"2024-09-20T22:51:37.498814+0000","last_became_active":"2024-09-20T22:46:15.181305+0000","last_became_peered":"2024-09-20T22:46:15.181305+0000","last_unstale":"2024-09-20T22:51:37.498814+0000","last_undegraded":"2024-09-20T22:51:37.498814+0000","last_fullsized":"2024-09-20T22:51:37.498814+0000","mapping_epoch":94,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:49:58.063382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:49:58.063382+0000","objects_scrubbed":1,"log_size":0,"log_dups_size":59,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"74'14","reported_seq":462,"reported_epoch":618,"state":"active+clean","last_fresh":"2024-09-20T22:51:38.265233+0000","last_change":"2024-09-20T22:49:57.026346+0000","last_active":"2024-09-20T22:51:38.265233+0000","last_peered":"2024-09-20T22:51:38.265233+0000","last_clean":"2024-09-20T22:51:38.265233+0000","last_became_active":"2024-09-20T22:48:05.235812+0000","last_became_peered":"2024-09-20T22:48:05.235812+0000","last_unstale":"2024-09-20T22:51:38.265233+0000","last_undegraded":"2024-09-20T22:51:38.265233+0000","last_fullsized":"2024-09-20T22:51:38.265233+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'14","last_scrub_stamp":"2024-09-20T22:49:57.026245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:49:57.026245+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"67'1","reported_seq":510,"reported_epoch":618,"state":"active+clean","last_fresh":"2024-09-20T22:51:38.260938+0000","last_change":"2024-09-20T22:51:20.292468+0000","last_active":"2024-09-20T22:51:38.260938+0000","last_peered":"2024-09-20T22:51:38.260938+0000","last_clean":"2024-09-20T22:51:38.260938+0000","last_became_active":"2024-09-20T22:51:20.283011+0000","last_became_peered":"2024-09-20T22:51:20.283011+0000","last_unstale":"2024-09-20T22:51:38.260938+0000","last_undegraded":"2024-09-20T22:51:38.260938+0000","last_fullsized":"2024-09-20T22:51:38.260938+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-09-20T22:51:11.806011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:51:11.806011+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:24.512033+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"600'13","reported_seq":1308,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-20T22:51:37.497980+0000","last_change":"2024-09-20T22:51:32.448600+0000","last_active":"2024-09-20T22:51:37.497980+0000","last_peered":"2024-09-20T22:51:37.497980+0000","last_clean":"2024-09-20T22:51:37.497980+0000","last_became_active":"2024-09-20T22:51:20.289873+0000","last_became_peered":"2024-09-20T22:51:20.289873+0000","last_unstale":"2024-09-20T22:51:37.497980+0000","last_undegraded":"2024-09-20T22:51:37.497980+0000","last_fullsized":"2024-09-20T22:51:37.497980+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"600'13","last_scrub_stamp":"2024-09-20T22:51:32.448512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:51:32.448512+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:52.962692+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":1132,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-20T22:51:37.538049+0000","last_change":"2024-09-20T22:51:31.487593+0000","last_active":"2024-09-20T22:51:37.538049+0000","last_peered":"2024-09-20T22:51:37.538049+0000","last_clean":"2024-09-20T22:51:37.538049+0000","last_became_active":"2024-09-20T22:41:44.831901+0000","last_became_peered":"2024-09-20T22:41:44.831901+0000","last_unstale":"2024-09-20T22:51:37.538049+0000","last_undegraded":"2024-09-20T22:51:37.538049+0000","last_fullsized":"2024-09-20T22:51:37.538049+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:31.487469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:51:31.487469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:01.276783+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1","version":"614'326","reported_seq":2889,"reported_epoch":618,"state":"active+clean","last_fresh":"2024-09-20T22:51:38.265242+0000","last_change":"2024-09-20T22:51:37.487282+0000","last_active":"2024-09-20T22:51:38.265242+0000","last_peered":"2024-09-20T22:51:38.265242+0000","last_clean":"2024-09-20T22:51:38.265242+0000","last_became_active":"2024-09-20T22:49:21.232750+0000","last_became_peered":"2024-09-20T22:49:21.232750+0000","last_unstale":"2024-09-20T22:51:38.265242+0000","last_undegraded":"2024-09-20T22:51:38.265242+0000","last_fullsized":"2024-09-20T22:51:38.265242+0000","mapping_epoch":407,"log_start":"474'259","ondisk_log_start":"474'259","created":94,"last_epoch_clean":481,"parent":"0.0","parent_split_bits":0,"last_scrub":"562'298","last_scrub_stamp":"2024-09-20T22:50:45.816540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:50:45.816540+0000","objects_scrubbed":4,"log_size":67,"log_dups_size":338,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:51:57.616336+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00055126800000000005,"stat_sum":{"num_bytes":8303087,"num_objects":5,"num_object_clones":3,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":81,"num_read_kb":85118,"num_write":72,"num_write_kb":42402,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":8732217,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"a7","length":"1"},{"start":"c5","length":"1"},{"start":"ce","length":"1"},{"start":"d4","length":"1"},{"start":"da","length":"1"},{"start":"df","length":"1"},{"start":"e3","length":"9"},{"start":"ee","length":"4"},{"start":"f3","length":"2"},{"start":"f6","length":"2"},{"start":"f9","length":"1"},{"start":"fe","length":"3"},{"start":"106","length":"1"},{"start":"109","length":"1"}]},{"pgid":"1.7","version":"77'128","reported_seq":1444,"reported_epoch":618,"state":"active+clean","last_fresh":"2024-09-20T22:51:38.581767+0000","last_change":"2024-09-20T22:51:10.000911+0000","last_active":"2024-09-20T22:51:38.581767+0000","last_peered":"2024-09-20T22:51:38.581767+0000","last_clean":"2024-09-20T22:51:38.581767+0000","last_became_active":"2024-09-20T22:46:03.169596+0000","last_became_peered":"2024-09-20T22:46:03.169596+0000","last_unstale":"2024-09-20T22:51:38.581767+0000","last_undegraded":"2024-09-20T22:51:38.581767+0000","last_fullsized":"2024-09-20T22:51:38.581767+0000","mapping_epoch":94,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:51:10.000823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:51:10.000823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:14.445311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"74'10","reported_seq":92,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-20T22:51:37.491572+0000","last_change":"2024-09-20T22:51:20.317396+0000","last_active":"2024-09-20T22:51:37.491572+0000","last_peered":"2024-09-20T22:51:37.491572+0000","last_clean":"2024-09-20T22:51:37.491572+0000","last_became_active":"2024-09-20T22:51:20.297933+0000","last_became_peered":"2024-09-20T22:51:20.297933+0000","last_unstale":"2024-09-20T22:51:37.491572+0000","last_undegraded":"2024-09-20T22:51:37.491572+0000","last_fullsized":"2024-09-20T22:51:37.491572+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-09-20T22:49:46.855673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:49:46.855673+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":1145,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-20T22:51:37.492017+0000","last_change":"2024-09-20T22:50:14.699223+0000","last_active":"2024-09-20T22:51:37.492017+0000","last_peered":"2024-09-20T22:51:37.492017+0000","last_clean":"2024-09-20T22:51:37.492017+0000","last_became_active":"2024-09-20T22:41:38.155629+0000","last_became_peered":"2024-09-20T22:41:38.155629+0000","last_unstale":"2024-09-20T22:51:37.492017+0000","last_undegraded":"2024-09-20T22:51:37.492017+0000","last_fullsized":"2024-09-20T22:51:37.492017+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:50:14.699152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:50:14.699152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"389'11","reported_seq":537,"reported_epoch":618,"state":"active+clean","last_fresh":"2024-09-20T22:51:38.260713+0000","last_change":"2024-09-20T22:51:26.968244+0000","last_active":"2024-09-20T22:51:38.260713+0000","last_peered":"2024-09-20T22:51:38.260713+0000","last_clean":"2024-09-20T22:51:38.260713+0000","last_became_active":"2024-09-20T22:47:49.227264+0000","last_became_peered":"2024-09-20T22:47:49.227264+0000","last_unstale":"2024-09-20T22:51:38.260713+0000","last_undegraded":"2024-09-20T22:51:38.260713+0000","last_fullsized":"2024-09-20T22:51:38.260713+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'11","last_scrub_stamp":"2024-09-20T22:51:26.968138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:51:26.968138+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:30.761523+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":478,"reported_epoch":618,"state":"active+clean","last_fresh":"2024-09-20T22:51:38.260067+0000","last_change":"2024-09-20T22:49:27.627363+0000","last_active":"2024-09-20T22:51:38.260067+0000","last_peered":"2024-09-20T22:51:38.260067+0000","last_clean":"2024-09-20T22:51:38.260067+0000","last_became_active":"2024-09-20T22:47:49.232380+0000","last_became_peered":"2024-09-20T22:47:49.232380+0000","last_unstale":"2024-09-20T22:51:38.260067+0000","last_undegraded":"2024-09-20T22:51:38.260067+0000","last_fullsized":"2024-09-20T22:51:38.260067+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:27.627295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:49:27.627295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.0","version":"616'443","reported_seq":3021,"reported_epoch":617,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-20T22:51:37.491458+0000","last_change":"2024-09-20T22:51:21.278706+0000","last_active":"2024-09-20T22:51:37.491458+0000","last_peered":"2024-09-20T22:51:37.491458+0000","last_clean":"2024-09-20T22:51:18.255785+0000","last_became_active":"2024-09-20T22:51:21.273895+0000","last_became_peered":"2024-09-20T22:51:21.273895+0000","last_unstale":"2024-09-20T22:51:37.491458+0000","last_undegraded":"2024-09-20T22:51:37.491458+0000","last_fullsized":"2024-09-20T22:51:37.491458+0000","mapping_epoch":600,"log_start":"480'253","ondisk_log_start":"480'253","created":94,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":0,"last_scrub":"562'368","last_scrub_stamp":"2024-09-20T22:50:44.669910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:50:44.669910+0000","objects_scrubbed":15,"log_size":190,"log_dups_size":416,"ondisk_log_size":190,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":14,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.91571902900000002,"stat_sum":{"num_bytes":34669284,"num_objects":16,"num_object_clones":13,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":16,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":181,"num_read_kb":195561,"num_write":137,"num_write_kb":47903,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":7603720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,1],"avail_no_missing":["5","1","3"],"object_location_counts":[{"shards":"1,3,5","objects":16}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"c5","length":"1"},{"start":"e3","length":"7"},{"start":"eb","length":"1"},{"start":"ee","length":"3"},{"start":"f3","length":"2"},{"start":"f6","length":"1"}]},{"pgid":"1.6","version":"77'128","reported_seq":1466,"reported_epoch":618,"state":"active+clean","last_fresh":"2024-09-20T22:51:38.581740+0000","last_change":"2024-09-20T22:51:06.036981+0000","last_active":"2024-09-20T22:51:38.581740+0000","last_peered":"2024-09-20T22:51:38.581740+0000","last_clean":"2024-09-20T22:51:38.581740+0000","last_became_active":"2024-09-20T22:47:52.212190+0000","last_became_peered":"2024-09-20T22:47:52.212190+0000","last_unstale":"2024-09-20T22:51:38.581740+0000","last_undegraded":"2024-09-20T22:51:38.581740+0000","last_fullsized":"2024-09-20T22:51:38.581740+0000","mapping_epoch":391,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:51:06.036900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:51:06.036900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:27.038972+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"515'27","reported_seq":504,"reported_epoch":618,"state":"active+clean","last_fresh":"2024-09-20T22:51:38.260484+0000","last_change":"2024-09-20T22:49:55.964247+0000","last_active":"2024-09-20T22:51:38.260484+0000","last_peered":"2024-09-20T22:51:38.260484+0000","last_clean":"2024-09-20T22:51:38.260484+0000","last_became_active":"2024-09-20T22:47:49.222009+0000","last_became_peered":"2024-09-20T22:47:49.222009+0000","last_unstale":"2024-09-20T22:51:38.260484+0000","last_undegraded":"2024-09-20T22:51:38.260484+0000","last_fullsized":"2024-09-20T22:51:38.260484+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"515'27","last_scrub_stamp":"2024-09-20T22:49:55.964153+0000","last_deep_scrub":"515'27","last_deep_scrub_stamp":"2024-09-20T22:49:55.964153+0000","last_clean_scrub_stamp":"2024-09-20T22:49:55.964153+0000","objects_scrubbed":8,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":38,"num_read_kb":24,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":46,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-20T22:51:37.489638+0000","last_change":"2024-09-20T22:51:20.287002+0000","last_active":"2024-09-20T22:51:37.489638+0000","last_peered":"2024-09-20T22:51:37.489638+0000","last_clean":"2024-09-20T22:51:37.489638+0000","last_became_active":"2024-09-20T22:51:20.286646+0000","last_became_peered":"2024-09-20T22:51:20.286646+0000","last_unstale":"2024-09-20T22:51:37.489638+0000","last_undegraded":"2024-09-20T22:51:37.489638+0000","last_fullsized":"2024-09-20T22:51:37.489638+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:28.084371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:48:23.320031+0000","last_clean_scrub_stamp":"2024-09-20T22:49:28.084371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":1158,"reported_epoch":618,"state":"active+clean","last_fresh":"2024-09-20T22:51:38.581284+0000","last_change":"2024-09-20T22:51:29.928160+0000","last_active":"2024-09-20T22:51:38.581284+0000","last_peered":"2024-09-20T22:51:38.581284+0000","last_clean":"2024-09-20T22:51:38.581284+0000","last_became_active":"2024-09-20T22:47:49.214055+0000","last_became_peered":"2024-09-20T22:47:49.214055+0000","last_unstale":"2024-09-20T22:51:38.581284+0000","last_undegraded":"2024-09-20T22:51:38.581284+0000","last_fullsized":"2024-09-20T22:51:38.581284+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:29.928092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:51:29.928092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:32.798183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":48,"reported_epoch":618,"state":"active+clean","last_fresh":"2024-09-20T22:51:38.581291+0000","last_change":"2024-09-20T22:51:20.285173+0000","last_active":"2024-09-20T22:51:38.581291+0000","last_peered":"2024-09-20T22:51:38.581291+0000","last_clean":"2024-09-20T22:51:38.581291+0000","last_became_active":"2024-09-20T22:51:20.284874+0000","last_became_peered":"2024-09-20T22:51:20.284874+0000","last_unstale":"2024-09-20T22:51:38.581291+0000","last_undegraded":"2024-09-20T22:51:38.581291+0000","last_fullsized":"2024-09-20T22:51:38.581291+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:50:21.729772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:48:41.208853+0000","last_clean_scrub_stamp":"2024-09-20T22:50:21.729772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:51:43.005227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.7","version":"617'960","reported_seq":3665,"reported_epoch":618,"state":"active+clean","last_fresh":"2024-09-20T22:51:38.582022+0000","last_change":"2024-09-20T22:51:37.542276+0000","last_active":"2024-09-20T22:51:38.582022+0000","last_peered":"2024-09-20T22:51:38.582022+0000","last_clean":"2024-09-20T22:51:38.582022+0000","last_became_active":"2024-09-20T22:48:43.224386+0000","last_became_peered":"2024-09-20T22:48:43.224386+0000","last_unstale":"2024-09-20T22:51:38.582022+0000","last_undegraded":"2024-09-20T22:51:38.582022+0000","last_fullsized":"2024-09-20T22:51:38.582022+0000","mapping_epoch":391,"log_start":"436'456","ondisk_log_start":"436'456","created":94,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"551'768","last_scrub_stamp":"2024-09-20T22:50:33.851339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:50:33.851339+0000","objects_scrubbed":27,"log_size":504,"log_dups_size":886,"ondisk_log_size":504,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0029917390000000002,"stat_sum":{"num_bytes":37065012,"num_objects":18,"num_object_clones":13,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":2,"num_read":407,"num_read_kb":426137,"num_write":312,"num_write_kb":104779,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":115,"num_bytes_recovered":51615167,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"a7","length":"1"},{"start":"c5","length":"1"},{"start":"ce","length":"1"},{"start":"d4","length":"1"},{"start":"da","length":"1"},{"start":"df","length":"1"},{"start":"e3","length":"9"},{"start":"ee","length":"4"},{"start":"f3","length":"2"},{"start":"f6","length":"2"},{"start":"f9","length":"1"},{"start":"fe","length":"3"},{"start":"106","length":"1"},{"start":"109","length":"1"}]},{"pgid":"1.1","version":"77'128","reported_seq":1453,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-20T22:51:37.491145+0000","last_change":"2024-09-20T22:50:20.639645+0000","last_active":"2024-09-20T22:51:37.491145+0000","last_peered":"2024-09-20T22:51:37.491145+0000","last_clean":"2024-09-20T22:51:37.491145+0000","last_became_active":"2024-09-20T22:47:52.208102+0000","last_became_peered":"2024-09-20T22:47:52.208102+0000","last_unstale":"2024-09-20T22:51:37.491145+0000","last_undegraded":"2024-09-20T22:51:37.491145+0000","last_fullsized":"2024-09-20T22:51:37.491145+0000","mapping_epoch":391,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:50:20.639561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:50:20.639561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"600'39","reported_seq":183,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-20T22:51:37.489610+0000","last_change":"2024-09-20T22:51:20.310742+0000","last_active":"2024-09-20T22:51:37.489610+0000","last_peered":"2024-09-20T22:51:37.489610+0000","last_clean":"2024-09-20T22:51:37.489610+0000","last_became_active":"2024-09-20T22:51:20.289376+0000","last_became_peered":"2024-09-20T22:51:20.289376+0000","last_unstale":"2024-09-20T22:51:37.489610+0000","last_undegraded":"2024-09-20T22:51:37.489610+0000","last_fullsized":"2024-09-20T22:51:37.489610+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'38","last_scrub_stamp":"2024-09-20T22:49:31.040362+0000","last_deep_scrub":"86'38","last_deep_scrub_stamp":"2024-09-20T22:42:43.935677+0000","last_clean_scrub_stamp":"2024-09-20T22:49:31.040362+0000","objects_scrubbed":10,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":446,"reported_epoch":618,"state":"active+clean","last_fresh":"2024-09-20T22:51:38.265851+0000","last_change":"2024-09-20T22:50:35.898050+0000","last_active":"2024-09-20T22:51:38.265851+0000","last_peered":"2024-09-20T22:51:38.265851+0000","last_clean":"2024-09-20T22:51:38.265851+0000","last_became_active":"2024-09-20T22:48:05.237756+0000","last_became_peered":"2024-09-20T22:48:05.237756+0000","last_unstale":"2024-09-20T22:51:38.265851+0000","last_undegraded":"2024-09-20T22:51:38.265851+0000","last_fullsized":"2024-09-20T22:51:38.265851+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:50:35.897979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:50:35.897979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"69'2","reported_seq":1173,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-20T22:51:37.491889+0000","last_change":"2024-09-20T22:51:37.070187+0000","last_active":"2024-09-20T22:51:37.491889+0000","last_peered":"2024-09-20T22:51:37.491889+0000","last_clean":"2024-09-20T22:51:37.491889+0000","last_became_active":"2024-09-20T22:51:20.286952+0000","last_became_peered":"2024-09-20T22:51:20.286952+0000","last_unstale":"2024-09-20T22:51:37.491889+0000","last_undegraded":"2024-09-20T22:51:37.491889+0000","last_fullsized":"2024-09-20T22:51:37.491889+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2024-09-20T22:51:37.070111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:36.130684+0000","last_clean_scrub_stamp":"2024-09-20T22:51:37.070111+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:05.960971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087280599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":490,"reported_epoch":618,"state":"active+clean","last_fresh":"2024-09-20T22:51:38.260671+0000","last_change":"2024-09-20T22:51:20.278834+0000","last_active":"2024-09-20T22:51:38.260671+0000","last_peered":"2024-09-20T22:51:38.260671+0000","last_clean":"2024-09-20T22:51:38.260671+0000","last_became_active":"2024-09-20T22:51:20.278427+0000","last_became_peered":"2024-09-20T22:51:20.278427+0000","last_unstale":"2024-09-20T22:51:38.260671+0000","last_undegraded":"2024-09-20T22:51:38.260671+0000","last_fullsized":"2024-09-20T22:51:38.260671+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:02.723689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:49:02.723689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":1136,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-20T22:51:37.498845+0000","last_change":"2024-09-20T22:49:57.022315+0000","last_active":"2024-09-20T22:51:37.498845+0000","last_peered":"2024-09-20T22:51:37.498845+0000","last_clean":"2024-09-20T22:51:37.498845+0000","last_became_active":"2024-09-20T22:48:05.234193+0000","last_became_peered":"2024-09-20T22:48:05.234193+0000","last_unstale":"2024-09-20T22:51:37.498845+0000","last_undegraded":"2024-09-20T22:51:37.498845+0000","last_fullsized":"2024-09-20T22:51:37.498845+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:57.022244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:49:57.022244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"616'341","reported_seq":1454,"reported_epoch":618,"state":"active+clean","last_fresh":"2024-09-20T22:51:38.261042+0000","last_change":"2024-09-20T22:51:37.488223+0000","last_active":"2024-09-20T22:51:38.261042+0000","last_peered":"2024-09-20T22:51:38.261042+0000","last_clean":"2024-09-20T22:51:38.261042+0000","last_became_active":"2024-09-20T22:51:23.267613+0000","last_became_peered":"2024-09-20T22:51:23.267613+0000","last_unstale":"2024-09-20T22:51:38.261042+0000","last_undegraded":"2024-09-20T22:51:38.261042+0000","last_fullsized":"2024-09-20T22:51:38.261042+0000","mapping_epoch":602,"log_start":"456'176","ondisk_log_start":"456'176","created":94,"last_epoch_clean":603,"parent":"0.0","parent_split_bits":0,"last_scrub":"616'341","last_scrub_stamp":"2024-09-20T22:51:36.893731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:51:36.893731+0000","objects_scrubbed":8,"log_size":165,"log_dups_size":318,"ondisk_log_size":165,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:55.486318+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.001487836,"stat_sum":{"num_bytes":18962578,"num_objects":8,"num_object_clones":6,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":134,"num_read_kb":128717,"num_write":111,"num_write_kb":40177,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":26566881,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"a7","length":"1"},{"start":"c5","length":"1"},{"start":"ce","length":"1"},{"start":"d4","length":"1"},{"start":"da","length":"1"},{"start":"df","length":"1"},{"start":"e3","length":"9"},{"start":"ee","length":"4"},{"start":"f3","length":"2"},{"start":"f6","length":"2"},{"start":"f9","length":"1"},{"start":"fe","length":"3"},{"start":"106","length":"1"},{"start":"109","length":"1"}]},{"pgid":"1.2","version":"77'128","reported_seq":1514,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-20T22:51:37.538459+0000","last_change":"2024-09-20T22:49:55.364495+0000","last_active":"2024-09-20T22:51:37.538459+0000","last_peered":"2024-09-20T22:51:37.538459+0000","last_clean":"2024-09-20T22:51:37.538459+0000","last_became_active":"2024-09-20T22:48:08.203028+0000","last_became_peered":"2024-09-20T22:48:08.203028+0000","last_unstale":"2024-09-20T22:51:37.538459+0000","last_undegraded":"2024-09-20T22:51:37.538459+0000","last_fullsized":"2024-09-20T22:51:37.538459+0000","mapping_epoch":407,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":408,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:49:55.364421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:49:55.364421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"443'18","reported_seq":901,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-20T22:51:37.492097+0000","last_change":"2024-09-20T22:51:20.312887+0000","last_active":"2024-09-20T22:51:37.492097+0000","last_peered":"2024-09-20T22:51:37.492097+0000","last_clean":"2024-09-20T22:51:37.492097+0000","last_became_active":"2024-09-20T22:51:20.293610+0000","last_became_peered":"2024-09-20T22:51:20.293610+0000","last_unstale":"2024-09-20T22:51:37.492097+0000","last_undegraded":"2024-09-20T22:51:37.492097+0000","last_fullsized":"2024-09-20T22:51:37.492097+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"443'18","last_scrub_stamp":"2024-09-20T22:51:04.643936+0000","last_deep_scrub":"443'18","last_deep_scrub_stamp":"2024-09-20T22:48:44.182517+0000","last_clean_scrub_stamp":"2024-09-20T22:51:04.643936+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:08.449352+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":990,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":54,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-20T22:51:37.487080+0000","last_change":"2024-09-20T22:51:32.279015+0000","last_active":"2024-09-20T22:51:37.487080+0000","last_peered":"2024-09-20T22:51:37.487080+0000","last_clean":"2024-09-20T22:51:37.487080+0000","last_became_active":"2024-09-20T22:51:20.278609+0000","last_became_peered":"2024-09-20T22:51:20.278609+0000","last_unstale":"2024-09-20T22:51:37.487080+0000","last_undegraded":"2024-09-20T22:51:37.487080+0000","last_fullsized":"2024-09-20T22:51:37.487080+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:32.278945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:51:32.278945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:54.818249+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"380'2","reported_seq":1190,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-20T22:51:37.488432+0000","last_change":"2024-09-20T22:50:39.347293+0000","last_active":"2024-09-20T22:51:37.488432+0000","last_peered":"2024-09-20T22:51:37.488432+0000","last_clean":"2024-09-20T22:51:37.488432+0000","last_became_active":"2024-09-20T22:47:49.222812+0000","last_became_peered":"2024-09-20T22:47:49.222812+0000","last_unstale":"2024-09-20T22:51:37.488432+0000","last_undegraded":"2024-09-20T22:51:37.488432+0000","last_fullsized":"2024-09-20T22:51:37.488432+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"380'2","last_scrub_stamp":"2024-09-20T22:50:39.347177+0000","last_deep_scrub":"380'2","last_deep_scrub_stamp":"2024-09-20T22:47:40.983039+0000","last_clean_scrub_stamp":"2024-09-20T22:50:39.347177+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:51:43.858386+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001301459,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":1144,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-20T22:51:37.489862+0000","last_change":"2024-09-20T22:51:04.254552+0000","last_active":"2024-09-20T22:51:37.489862+0000","last_peered":"2024-09-20T22:51:37.489862+0000","last_clean":"2024-09-20T22:51:37.489862+0000","last_became_active":"2024-09-20T22:41:42.185112+0000","last_became_peered":"2024-09-20T22:41:42.185112+0000","last_unstale":"2024-09-20T22:51:37.489862+0000","last_undegraded":"2024-09-20T22:51:37.489862+0000","last_fullsized":"2024-09-20T22:51:37.489862+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:04.254441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:47:22.881109+0000","last_clean_scrub_stamp":"2024-09-20T22:51:04.254441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:31.528165+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":1132,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-20T22:51:37.537905+0000","last_change":"2024-09-20T22:51:11.568798+0000","last_active":"2024-09-20T22:51:37.537905+0000","last_peered":"2024-09-20T22:51:37.537905+0000","last_clean":"2024-09-20T22:51:37.537905+0000","last_became_active":"2024-09-20T22:41:44.830515+0000","last_became_peered":"2024-09-20T22:41:44.830515+0000","last_unstale":"2024-09-20T22:51:37.537905+0000","last_undegraded":"2024-09-20T22:51:37.537905+0000","last_fullsized":"2024-09-20T22:51:37.537905+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:11.568712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:51:11.568712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:13.807768+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"616'288","reported_seq":2961,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-20T22:51:37.538668+0000","last_change":"2024-09-20T22:51:37.538935+0000","last_active":"2024-09-20T22:51:37.538668+0000","last_peered":"2024-09-20T22:51:37.538668+0000","last_clean":"2024-09-20T22:51:37.538668+0000","last_became_active":"2024-09-20T22:48:57.236417+0000","last_became_peered":"2024-09-20T22:48:57.236417+0000","last_unstale":"2024-09-20T22:51:37.538668+0000","last_undegraded":"2024-09-20T22:51:37.538668+0000","last_fullsized":"2024-09-20T22:51:37.538668+0000","mapping_epoch":391,"log_start":"450'190","ondisk_log_start":"450'190","created":94,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"605'278","last_scrub_stamp":"2024-09-20T22:51:25.467092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:51:25.467092+0000","objects_scrubbed":6,"log_size":98,"log_dups_size":240,"ondisk_log_size":98,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:27.184081+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001317197,"stat_sum":{"num_bytes":11580633,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":67,"num_read_kb":52122,"num_write":84,"num_write_kb":35181,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":19673726,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"a7","length":"1"},{"start":"c5","length":"1"},{"start":"ce","length":"1"},{"start":"d4","length":"1"},{"start":"da","length":"1"},{"start":"df","length":"1"},{"start":"e3","length":"9"},{"start":"ee","length":"4"},{"start":"f3","length":"2"},{"start":"f6","length":"2"},{"start":"f9","length":"1"},{"start":"fe","length":"3"},{"start":"106","length":"1"},{"start":"109","length":"1"}]},{"pgid":"1.3","version":"77'128","reported_seq":351,"reported_epoch":618,"state":"active+clean","last_fresh":"2024-09-20T22:51:38.581410+0000","last_change":"2024-09-20T22:51:28.897355+0000","last_active":"2024-09-20T22:51:38.581410+0000","last_peered":"2024-09-20T22:51:38.581410+0000","last_clean":"2024-09-20T22:51:38.581410+0000","last_became_active":"2024-09-20T22:51:23.266810+0000","last_became_peered":"2024-09-20T22:51:23.266810+0000","last_unstale":"2024-09-20T22:51:38.581410+0000","last_undegraded":"2024-09-20T22:51:38.581410+0000","last_fullsized":"2024-09-20T22:51:38.581410+0000","mapping_epoch":602,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":603,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:51:28.897273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:51:28.897273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:58.849420+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.6","version":"74'12","reported_seq":512,"reported_epoch":618,"state":"active+clean","last_fresh":"2024-09-20T22:51:38.261133+0000","last_change":"2024-09-20T22:51:20.300892+0000","last_active":"2024-09-20T22:51:38.261133+0000","last_peered":"2024-09-20T22:51:38.261133+0000","last_clean":"2024-09-20T22:51:38.261133+0000","last_became_active":"2024-09-20T22:51:20.283112+0000","last_became_peered":"2024-09-20T22:51:20.283112+0000","last_unstale":"2024-09-20T22:51:38.261133+0000","last_undegraded":"2024-09-20T22:51:38.261133+0000","last_fullsized":"2024-09-20T22:51:38.261133+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-09-20T22:50:14.745918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:50:14.745918+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":492,"reported_epoch":618,"state":"active+clean","last_fresh":"2024-09-20T22:51:38.260129+0000","last_change":"2024-09-20T22:50:39.786895+0000","last_active":"2024-09-20T22:51:38.260129+0000","last_peered":"2024-09-20T22:51:38.260129+0000","last_clean":"2024-09-20T22:51:38.260129+0000","last_became_active":"2024-09-20T22:48:05.233093+0000","last_became_peered":"2024-09-20T22:48:05.233093+0000","last_unstale":"2024-09-20T22:51:38.260129+0000","last_undegraded":"2024-09-20T22:51:38.260129+0000","last_fullsized":"2024-09-20T22:51:38.260129+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:50:39.786825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:44:19.834641+0000","last_clean_scrub_stamp":"2024-09-20T22:50:39.786825+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:51:58.231597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"75'5","reported_seq":1778,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-20T22:51:38.308454+0000","last_change":"2024-09-20T22:51:20.290148+0000","last_active":"2024-09-20T22:51:38.308454+0000","last_peered":"2024-09-20T22:51:38.308454+0000","last_clean":"2024-09-20T22:51:38.308454+0000","last_became_active":"2024-09-20T22:51:20.281329+0000","last_became_peered":"2024-09-20T22:51:20.281329+0000","last_unstale":"2024-09-20T22:51:38.308454+0000","last_undegraded":"2024-09-20T22:51:38.308454+0000","last_fullsized":"2024-09-20T22:51:38.308454+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-09-20T22:50:21.363477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:36.130684+0000","last_clean_scrub_stamp":"2024-09-20T22:50:21.363477+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:51:45.173573+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00078631700000000003,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":590,"num_read_kb":585,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1167,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":1155,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-20T22:51:37.492100+0000","last_change":"2024-09-20T22:51:20.274832+0000","last_active":"2024-09-20T22:51:37.492100+0000","last_peered":"2024-09-20T22:51:37.492100+0000","last_clean":"2024-09-20T22:51:37.492100+0000","last_became_active":"2024-09-20T22:51:20.274522+0000","last_became_peered":"2024-09-20T22:51:20.274522+0000","last_unstale":"2024-09-20T22:51:37.492100+0000","last_undegraded":"2024-09-20T22:51:37.492100+0000","last_fullsized":"2024-09-20T22:51:37.492100+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:50:18.695027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:50:18.695027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":46,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-20T22:51:37.491636+0000","last_change":"2024-09-20T22:51:20.288058+0000","last_active":"2024-09-20T22:51:37.491636+0000","last_peered":"2024-09-20T22:51:37.491636+0000","last_clean":"2024-09-20T22:51:37.491636+0000","last_became_active":"2024-09-20T22:51:20.287750+0000","last_became_peered":"2024-09-20T22:51:20.287750+0000","last_unstale":"2024-09-20T22:51:37.491636+0000","last_undegraded":"2024-09-20T22:51:37.491636+0000","last_fullsized":"2024-09-20T22:51:37.491636+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:05.621880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:51:05.621880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:10.146965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.6","version":"616'357","reported_seq":2961,"reported_epoch":618,"state":"active+clean","last_fresh":"2024-09-20T22:51:38.260800+0000","last_change":"2024-09-20T22:51:37.488805+0000","last_active":"2024-09-20T22:51:38.260800+0000","last_peered":"2024-09-20T22:51:38.260800+0000","last_clean":"2024-09-20T22:51:38.260800+0000","last_became_active":"2024-09-20T22:48:47.234531+0000","last_became_peered":"2024-09-20T22:48:47.234531+0000","last_unstale":"2024-09-20T22:51:38.260800+0000","last_undegraded":"2024-09-20T22:51:38.260800+0000","last_fullsized":"2024-09-20T22:51:38.260800+0000","mapping_epoch":391,"log_start":"443'181","ondisk_log_start":"443'181","created":94,"last_epoch_clean":447,"parent":"0.0","parent_split_bits":0,"last_scrub":"579'314","last_scrub_stamp":"2024-09-20T22:51:01.772358+0000","last_deep_scrub":"440'177","last_deep_scrub_stamp":"2024-09-20T22:48:40.602342+0000","last_clean_scrub_stamp":"2024-09-20T22:51:01.772358+0000","objects_scrubbed":9,"log_size":176,"log_dups_size":314,"ondisk_log_size":176,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:11.791242+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0.001321787,"stat_sum":{"num_bytes":20834358,"num_objects":7,"num_object_clones":5,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":141,"num_read_kb":142852,"num_write":114,"num_write_kb":43993,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":18662529,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"a7","length":"1"},{"start":"c5","length":"1"},{"start":"ce","length":"1"},{"start":"d4","length":"1"},{"start":"da","length":"1"},{"start":"df","length":"1"},{"start":"e3","length":"9"},{"start":"ee","length":"4"},{"start":"f3","length":"2"},{"start":"f6","length":"2"},{"start":"f9","length":"1"},{"start":"fe","length":"3"},{"start":"106","length":"1"},{"start":"109","length":"1"}]},{"pgid":"4.5","version":"408'20","reported_seq":1389,"reported_epoch":618,"state":"active+clean","last_fresh":"2024-09-20T22:51:38.581861+0000","last_change":"2024-09-20T22:51:33.961483+0000","last_active":"2024-09-20T22:51:38.581861+0000","last_peered":"2024-09-20T22:51:38.581861+0000","last_clean":"2024-09-20T22:51:38.581861+0000","last_became_active":"2024-09-20T22:48:05.243523+0000","last_became_peered":"2024-09-20T22:48:05.243523+0000","last_unstale":"2024-09-20T22:51:38.581861+0000","last_undegraded":"2024-09-20T22:51:38.581861+0000","last_fullsized":"2024-09-20T22:51:38.581861+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"408'20","last_scrub_stamp":"2024-09-20T22:51:33.961401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:51:33.961401+0000","objects_scrubbed":8,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:00.387882+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"67'1","reported_seq":1142,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-20T22:51:37.498711+0000","last_change":"2024-09-20T22:51:35.440516+0000","last_active":"2024-09-20T22:51:37.498711+0000","last_peered":"2024-09-20T22:51:37.498711+0000","last_clean":"2024-09-20T22:51:37.498711+0000","last_became_active":"2024-09-20T22:41:38.158917+0000","last_became_peered":"2024-09-20T22:41:38.158917+0000","last_unstale":"2024-09-20T22:51:37.498711+0000","last_undegraded":"2024-09-20T22:51:37.498711+0000","last_fullsized":"2024-09-20T22:51:37.498711+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-09-20T22:51:35.440416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:51:35.440416+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:53.239571+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"77'128","reported_seq":1486,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-20T22:51:37.537874+0000","last_change":"2024-09-20T22:49:46.391695+0000","last_active":"2024-09-20T22:51:37.537874+0000","last_peered":"2024-09-20T22:51:37.537874+0000","last_clean":"2024-09-20T22:51:37.537874+0000","last_became_active":"2024-09-20T22:47:52.206169+0000","last_became_peered":"2024-09-20T22:47:52.206169+0000","last_unstale":"2024-09-20T22:51:37.537874+0000","last_undegraded":"2024-09-20T22:51:37.537874+0000","last_fullsized":"2024-09-20T22:51:37.537874+0000","mapping_epoch":391,"log_start":"77'128","ondisk_log_start":"77'128","created":19,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:49:46.391614+0000","last_deep_scrub":"77'128","last_deep_scrub_stamp":"2024-09-20T22:47:22.417099+0000","last_clean_scrub_stamp":"2024-09-20T22:49:46.391614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":1136,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-20T22:51:37.537841+0000","last_change":"2024-09-20T22:50:41.381020+0000","last_active":"2024-09-20T22:51:37.537841+0000","last_peered":"2024-09-20T22:51:37.537841+0000","last_clean":"2024-09-20T22:51:37.537841+0000","last_became_active":"2024-09-20T22:41:42.184387+0000","last_became_peered":"2024-09-20T22:41:42.184387+0000","last_unstale":"2024-09-20T22:51:37.537841+0000","last_undegraded":"2024-09-20T22:51:37.537841+0000","last_fullsized":"2024-09-20T22:51:37.537841+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:50:41.380924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:50:41.380924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:51:50.365634+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":1145,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-20T22:51:37.491994+0000","last_change":"2024-09-20T22:49:57.715494+0000","last_active":"2024-09-20T22:51:37.491994+0000","last_peered":"2024-09-20T22:51:37.491994+0000","last_clean":"2024-09-20T22:51:37.491994+0000","last_became_active":"2024-09-20T22:48:05.241602+0000","last_became_peered":"2024-09-20T22:48:05.241602+0000","last_unstale":"2024-09-20T22:51:37.491994+0000","last_undegraded":"2024-09-20T22:51:37.491994+0000","last_fullsized":"2024-09-20T22:51:37.491994+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:57.715416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:49:57.715416+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"617'440","reported_seq":3165,"reported_epoch":618,"state":"active+clean","last_fresh":"2024-09-20T22:51:38.581695+0000","last_change":"2024-09-20T22:51:37.540704+0000","last_active":"2024-09-20T22:51:38.581695+0000","last_peered":"2024-09-20T22:51:38.581695+0000","last_clean":"2024-09-20T22:51:38.581695+0000","last_became_active":"2024-09-20T22:51:23.269216+0000","last_became_peered":"2024-09-20T22:51:23.269216+0000","last_unstale":"2024-09-20T22:51:38.581695+0000","last_undegraded":"2024-09-20T22:51:38.581695+0000","last_fullsized":"2024-09-20T22:51:38.581695+0000","mapping_epoch":602,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":603,"parent":"0.0","parent_split_bits":0,"last_scrub":"554'341","last_scrub_stamp":"2024-09-20T22:50:34.870361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:50:34.870361+0000","objects_scrubbed":11,"log_size":440,"log_dups_size":0,"ondisk_log_size":440,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:51:41.838578+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.0022855309999999999,"stat_sum":{"num_bytes":16398843,"num_objects":8,"num_object_clones":6,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":151,"num_read_kb":142455,"num_write":84,"num_write_kb":35211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":52,"num_bytes_recovered":30669453,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"a7","length":"1"},{"start":"c5","length":"1"},{"start":"ce","length":"1"},{"start":"d4","length":"1"},{"start":"da","length":"1"},{"start":"df","length":"1"},{"start":"e3","length":"9"},{"start":"ee","length":"4"},{"start":"f3","length":"2"},{"start":"f6","length":"2"},{"start":"f9","length":"1"},{"start":"fe","length":"3"},{"start":"106","length":"1"},{"start":"109","length":"1"}]},{"pgid":"4.e","version":"74'11","reported_seq":476,"reported_epoch":618,"state":"active+clean","last_fresh":"2024-09-20T22:51:38.265782+0000","last_change":"2024-09-20T22:51:20.310962+0000","last_active":"2024-09-20T22:51:38.265782+0000","last_peered":"2024-09-20T22:51:38.265782+0000","last_clean":"2024-09-20T22:51:38.265782+0000","last_became_active":"2024-09-20T22:51:20.288958+0000","last_became_peered":"2024-09-20T22:51:20.288958+0000","last_unstale":"2024-09-20T22:51:38.265782+0000","last_undegraded":"2024-09-20T22:51:38.265782+0000","last_fullsized":"2024-09-20T22:51:38.265782+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-09-20T22:50:38.879205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:50:38.879205+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:51:55.309483+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":454,"reported_epoch":618,"state":"active+clean","last_fresh":"2024-09-20T22:51:38.264948+0000","last_change":"2024-09-20T22:51:29.757136+0000","last_active":"2024-09-20T22:51:38.264948+0000","last_peered":"2024-09-20T22:51:38.264948+0000","last_clean":"2024-09-20T22:51:38.264948+0000","last_became_active":"2024-09-20T22:48:05.254688+0000","last_became_peered":"2024-09-20T22:48:05.254688+0000","last_unstale":"2024-09-20T22:51:38.264948+0000","last_undegraded":"2024-09-20T22:51:38.264948+0000","last_fullsized":"2024-09-20T22:51:38.264948+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:29.757030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:51:29.757030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:52.073822+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":1141,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-20T22:51:37.491847+0000","last_change":"2024-09-20T22:49:48.693437+0000","last_active":"2024-09-20T22:51:37.491847+0000","last_peered":"2024-09-20T22:51:37.491847+0000","last_clean":"2024-09-20T22:51:37.491847+0000","last_became_active":"2024-09-20T22:48:05.233778+0000","last_became_peered":"2024-09-20T22:48:05.233778+0000","last_unstale":"2024-09-20T22:51:37.491847+0000","last_undegraded":"2024-09-20T22:51:37.491847+0000","last_fullsized":"2024-09-20T22:51:37.491847+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:48.693333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:49:48.693333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":1124,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-20T22:51:37.498726+0000","last_change":"2024-09-20T22:50:47.230929+0000","last_active":"2024-09-20T22:51:37.498726+0000","last_peered":"2024-09-20T22:51:37.498726+0000","last_clean":"2024-09-20T22:51:37.498726+0000","last_became_active":"2024-09-20T22:41:44.832009+0000","last_became_peered":"2024-09-20T22:41:44.832009+0000","last_unstale":"2024-09-20T22:51:37.498726+0000","last_undegraded":"2024-09-20T22:51:37.498726+0000","last_fullsized":"2024-09-20T22:51:37.498726+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:50:47.230854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:48:40.127123+0000","last_clean_scrub_stamp":"2024-09-20T22:50:47.230854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:51:57.066298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.e","version":"617'532","reported_seq":3226,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-20T22:51:37.492144+0000","last_change":"2024-09-20T22:51:37.492322+0000","last_active":"2024-09-20T22:51:37.492144+0000","last_peered":"2024-09-20T22:51:37.492144+0000","last_clean":"2024-09-20T22:51:37.492144+0000","last_became_active":"2024-09-20T22:47:52.209452+0000","last_became_peered":"2024-09-20T22:47:52.209452+0000","last_unstale":"2024-09-20T22:51:37.492144+0000","last_undegraded":"2024-09-20T22:51:37.492144+0000","last_fullsized":"2024-09-20T22:51:37.492144+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"559'431","last_scrub_stamp":"2024-09-20T22:50:45.276861+0000","last_deep_scrub":"174'60","last_deep_scrub_stamp":"2024-09-20T22:44:18.301776+0000","last_clean_scrub_stamp":"2024-09-20T22:50:45.276861+0000","objects_scrubbed":8,"log_size":532,"log_dups_size":0,"ondisk_log_size":532,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:51:53.670491+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0.0059137440000000003,"stat_sum":{"num_bytes":21948475,"num_objects":12,"num_object_clones":10,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":224,"num_read_kb":228727,"num_write":147,"num_write_kb":53840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":23110741,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"a7","length":"1"},{"start":"c5","length":"1"},{"start":"ce","length":"1"},{"start":"d4","length":"1"},{"start":"da","length":"1"},{"start":"df","length":"1"},{"start":"e3","length":"9"},{"start":"ee","length":"4"},{"start":"f3","length":"2"},{"start":"f6","length":"2"},{"start":"f9","length":"1"},{"start":"fe","length":"3"},{"start":"106","length":"1"},{"start":"109","length":"1"}]},{"pgid":"1.8","version":"77'128","reported_seq":1405,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-20T22:51:37.491079+0000","last_change":"2024-09-20T22:49:56.715035+0000","last_active":"2024-09-20T22:51:37.491079+0000","last_peered":"2024-09-20T22:51:37.491079+0000","last_clean":"2024-09-20T22:51:37.491079+0000","last_became_active":"2024-09-20T22:42:52.405594+0000","last_became_peered":"2024-09-20T22:42:52.405594+0000","last_unstale":"2024-09-20T22:51:37.491079+0000","last_undegraded":"2024-09-20T22:51:37.491079+0000","last_fullsized":"2024-09-20T22:51:37.491079+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:49:56.714919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:49:56.714919+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"74'17","reported_seq":496,"reported_epoch":618,"state":"active+clean","last_fresh":"2024-09-20T22:51:38.265737+0000","last_change":"2024-09-20T22:51:20.314475+0000","last_active":"2024-09-20T22:51:38.265737+0000","last_peered":"2024-09-20T22:51:38.265737+0000","last_clean":"2024-09-20T22:51:38.265737+0000","last_became_active":"2024-09-20T22:51:20.288846+0000","last_became_peered":"2024-09-20T22:51:20.288846+0000","last_unstale":"2024-09-20T22:51:38.265737+0000","last_undegraded":"2024-09-20T22:51:38.265737+0000","last_fullsized":"2024-09-20T22:51:38.265737+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'17","last_scrub_stamp":"2024-09-20T22:51:06.946981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:51:06.946981+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:12.032715+0000","scrub_duration":57,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":28,"num_read_kb":18,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":699,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-20T22:51:37.486959+0000","last_change":"2024-09-20T22:51:20.294379+0000","last_active":"2024-09-20T22:51:37.486959+0000","last_peered":"2024-09-20T22:51:37.486959+0000","last_clean":"2024-09-20T22:51:37.486959+0000","last_became_active":"2024-09-20T22:51:20.294073+0000","last_became_peered":"2024-09-20T22:51:20.294073+0000","last_unstale":"2024-09-20T22:51:37.486959+0000","last_undegraded":"2024-09-20T22:51:37.486959+0000","last_fullsized":"2024-09-20T22:51:37.486959+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:50:47.268782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:35.492562+0000","last_clean_scrub_stamp":"2024-09-20T22:50:47.268782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:11.936481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":767,"reported_epoch":618,"state":"active+clean","last_fresh":"2024-09-20T22:51:38.265041+0000","last_change":"2024-09-20T22:51:25.738132+0000","last_active":"2024-09-20T22:51:38.265041+0000","last_peered":"2024-09-20T22:51:38.265041+0000","last_clean":"2024-09-20T22:51:38.265041+0000","last_became_active":"2024-09-20T22:51:20.281724+0000","last_became_peered":"2024-09-20T22:51:20.281724+0000","last_unstale":"2024-09-20T22:51:38.265041+0000","last_undegraded":"2024-09-20T22:51:38.265041+0000","last_fullsized":"2024-09-20T22:51:38.265041+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:25.738061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:51:25.738061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:41.584933+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":1160,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-20T22:51:37.489824+0000","last_change":"2024-09-20T22:50:41.284220+0000","last_active":"2024-09-20T22:51:37.489824+0000","last_peered":"2024-09-20T22:51:37.489824+0000","last_clean":"2024-09-20T22:51:37.489824+0000","last_became_active":"2024-09-20T22:48:05.239006+0000","last_became_peered":"2024-09-20T22:48:05.239006+0000","last_unstale":"2024-09-20T22:51:37.489824+0000","last_undegraded":"2024-09-20T22:51:37.489824+0000","last_fullsized":"2024-09-20T22:51:37.489824+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:50:41.284158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:50:41.284158+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:06.473276+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"616'587","reported_seq":3241,"reported_epoch":618,"state":"active+clean","last_fresh":"2024-09-20T22:51:38.261608+0000","last_change":"2024-09-20T22:51:37.488661+0000","last_active":"2024-09-20T22:51:38.261608+0000","last_peered":"2024-09-20T22:51:38.261608+0000","last_clean":"2024-09-20T22:51:38.261608+0000","last_became_active":"2024-09-20T22:47:52.216198+0000","last_became_peered":"2024-09-20T22:47:52.216198+0000","last_unstale":"2024-09-20T22:51:38.261608+0000","last_undegraded":"2024-09-20T22:51:38.261608+0000","last_fullsized":"2024-09-20T22:51:38.261608+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"481'456","last_scrub_stamp":"2024-09-20T22:49:22.637524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:49:22.637524+0000","objects_scrubbed":9,"log_size":587,"log_dups_size":0,"ondisk_log_size":587,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.000392431,"stat_sum":{"num_bytes":14214383,"num_objects":6,"num_object_clones":4,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":273,"num_read_kb":277178,"num_write":159,"num_write_kb":53436,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":53,"num_bytes_recovered":14983556,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"a7","length":"1"},{"start":"c5","length":"1"},{"start":"ce","length":"1"},{"start":"d4","length":"1"},{"start":"da","length":"1"},{"start":"df","length":"1"},{"start":"e3","length":"9"},{"start":"ee","length":"4"},{"start":"f3","length":"2"},{"start":"f6","length":"2"},{"start":"f9","length":"1"},{"start":"fe","length":"3"},{"start":"106","length":"1"},{"start":"109","length":"1"}]},{"pgid":"1.9","version":"77'128","reported_seq":1454,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-20T22:51:37.538319+0000","last_change":"2024-09-20T22:50:42.263313+0000","last_active":"2024-09-20T22:51:37.538319+0000","last_peered":"2024-09-20T22:51:37.538319+0000","last_clean":"2024-09-20T22:51:37.538319+0000","last_became_active":"2024-09-20T22:50:42.262862+0000","last_became_peered":"2024-09-20T22:50:42.262862+0000","last_unstale":"2024-09-20T22:51:37.538319+0000","last_undegraded":"2024-09-20T22:51:37.538319+0000","last_fullsized":"2024-09-20T22:51:37.538319+0000","mapping_epoch":561,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":562,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:49:52.301348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:49:52.301348+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"74'10","reported_seq":458,"reported_epoch":618,"state":"active+clean","last_fresh":"2024-09-20T22:51:38.265384+0000","last_change":"2024-09-20T22:50:34.897059+0000","last_active":"2024-09-20T22:51:38.265384+0000","last_peered":"2024-09-20T22:51:38.265384+0000","last_clean":"2024-09-20T22:51:38.265384+0000","last_became_active":"2024-09-20T22:48:05.242811+0000","last_became_peered":"2024-09-20T22:48:05.242811+0000","last_unstale":"2024-09-20T22:51:38.265384+0000","last_undegraded":"2024-09-20T22:51:38.265384+0000","last_fullsized":"2024-09-20T22:51:38.265384+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-09-20T22:50:34.896923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:50:34.896923+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":1160,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-20T22:51:37.498409+0000","last_change":"2024-09-20T22:50:13.205821+0000","last_active":"2024-09-20T22:51:37.498409+0000","last_peered":"2024-09-20T22:51:37.498409+0000","last_clean":"2024-09-20T22:51:37.498409+0000","last_became_active":"2024-09-20T22:48:05.233999+0000","last_became_peered":"2024-09-20T22:48:05.233999+0000","last_unstale":"2024-09-20T22:51:37.498409+0000","last_undegraded":"2024-09-20T22:51:37.498409+0000","last_fullsized":"2024-09-20T22:51:37.498409+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:50:13.205743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:50:13.205743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:51:41.953774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"261'9","reported_seq":1274,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-20T22:51:37.489410+0000","last_change":"2024-09-20T22:49:57.205107+0000","last_active":"2024-09-20T22:51:37.489410+0000","last_peered":"2024-09-20T22:51:37.489410+0000","last_clean":"2024-09-20T22:51:37.489410+0000","last_became_active":"2024-09-20T22:41:42.185863+0000","last_became_peered":"2024-09-20T22:41:42.185863+0000","last_unstale":"2024-09-20T22:51:37.489410+0000","last_undegraded":"2024-09-20T22:51:37.489410+0000","last_fullsized":"2024-09-20T22:51:37.489410+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'9","last_scrub_stamp":"2024-09-20T22:49:57.204987+0000","last_deep_scrub":"261'9","last_deep_scrub_stamp":"2024-09-20T22:46:50.974735+0000","last_clean_scrub_stamp":"2024-09-20T22:49:57.204987+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":460,"reported_epoch":618,"state":"active+clean","last_fresh":"2024-09-20T22:51:38.265627+0000","last_change":"2024-09-20T22:51:20.296567+0000","last_active":"2024-09-20T22:51:38.265627+0000","last_peered":"2024-09-20T22:51:38.265627+0000","last_clean":"2024-09-20T22:51:38.265627+0000","last_became_active":"2024-09-20T22:51:20.296204+0000","last_became_peered":"2024-09-20T22:51:20.296204+0000","last_unstale":"2024-09-20T22:51:38.265627+0000","last_undegraded":"2024-09-20T22:51:38.265627+0000","last_fullsized":"2024-09-20T22:51:38.265627+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:02.932364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:51:02.932364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:17.218936+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.8","version":"616'646","reported_seq":3152,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-20T22:51:37.486821+0000","last_change":"2024-09-20T22:51:37.487325+0000","last_active":"2024-09-20T22:51:37.486821+0000","last_peered":"2024-09-20T22:51:37.486821+0000","last_clean":"2024-09-20T22:51:37.486821+0000","last_became_active":"2024-09-20T22:48:31.224359+0000","last_became_peered":"2024-09-20T22:48:31.224359+0000","last_unstale":"2024-09-20T22:51:37.486821+0000","last_undegraded":"2024-09-20T22:51:37.486821+0000","last_fullsized":"2024-09-20T22:51:37.486821+0000","mapping_epoch":407,"log_start":"426'268","ondisk_log_start":"426'268","created":94,"last_epoch_clean":431,"parent":"0.0","parent_split_bits":0,"last_scrub":"534'447","last_scrub_stamp":"2024-09-20T22:50:14.324477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:50:14.324477+0000","objects_scrubbed":15,"log_size":378,"log_dups_size":445,"ondisk_log_size":378,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:51:39.171588+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.001102488,"stat_sum":{"num_bytes":34024569,"num_objects":15,"num_object_clones":12,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":243,"num_read_kb":244936,"num_write":190,"num_write_kb":65464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":26027409,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"a7","length":"1"},{"start":"c5","length":"1"},{"start":"ce","length":"1"},{"start":"d4","length":"1"},{"start":"da","length":"1"},{"start":"df","length":"1"},{"start":"e3","length":"9"},{"start":"ee","length":"4"},{"start":"f3","length":"2"},{"start":"f6","length":"2"},{"start":"f9","length":"1"},{"start":"fe","length":"3"},{"start":"106","length":"1"},{"start":"109","length":"1"}]},{"pgid":"4.b","version":"74'9","reported_seq":528,"reported_epoch":618,"state":"active+clean","last_fresh":"2024-09-20T22:51:38.260623+0000","last_change":"2024-09-20T22:51:20.296532+0000","last_active":"2024-09-20T22:51:38.260623+0000","last_peered":"2024-09-20T22:51:38.260623+0000","last_clean":"2024-09-20T22:51:38.260623+0000","last_became_active":"2024-09-20T22:51:20.273094+0000","last_became_peered":"2024-09-20T22:51:20.273094+0000","last_unstale":"2024-09-20T22:51:38.260623+0000","last_undegraded":"2024-09-20T22:51:38.260623+0000","last_fullsized":"2024-09-20T22:51:38.260623+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-09-20T22:50:36.883811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:50:36.883811+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:05.901934+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":1145,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-20T22:51:37.491633+0000","last_change":"2024-09-20T22:51:12.755628+0000","last_active":"2024-09-20T22:51:37.491633+0000","last_peered":"2024-09-20T22:51:37.491633+0000","last_clean":"2024-09-20T22:51:37.491633+0000","last_became_active":"2024-09-20T22:41:38.155953+0000","last_became_peered":"2024-09-20T22:41:38.155953+0000","last_unstale":"2024-09-20T22:51:37.491633+0000","last_undegraded":"2024-09-20T22:51:37.491633+0000","last_fullsized":"2024-09-20T22:51:37.491633+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:12.755526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:51:12.755526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:32.975772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":1168,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-20T22:51:37.487149+0000","last_change":"2024-09-20T22:50:37.347748+0000","last_active":"2024-09-20T22:51:37.487149+0000","last_peered":"2024-09-20T22:51:37.487149+0000","last_clean":"2024-09-20T22:51:37.487149+0000","last_became_active":"2024-09-20T22:48:05.240613+0000","last_became_peered":"2024-09-20T22:48:05.240613+0000","last_unstale":"2024-09-20T22:51:37.487149+0000","last_undegraded":"2024-09-20T22:51:37.487149+0000","last_fullsized":"2024-09-20T22:51:37.487149+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:50:37.347636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:50:37.347636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:51:51.117109+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":478,"reported_epoch":618,"state":"active+clean","last_fresh":"2024-09-20T22:51:38.260218+0000","last_change":"2024-09-20T22:49:31.666061+0000","last_active":"2024-09-20T22:51:38.260218+0000","last_peered":"2024-09-20T22:51:38.260218+0000","last_clean":"2024-09-20T22:51:38.260218+0000","last_became_active":"2024-09-20T22:47:49.219486+0000","last_became_peered":"2024-09-20T22:47:49.219486+0000","last_unstale":"2024-09-20T22:51:38.260218+0000","last_undegraded":"2024-09-20T22:51:38.260218+0000","last_fullsized":"2024-09-20T22:51:38.260218+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:31.665954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:49:31.665954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"616'929","reported_seq":3639,"reported_epoch":618,"state":"active+clean","last_fresh":"2024-09-20T22:51:38.260423+0000","last_change":"2024-09-20T22:51:37.488168+0000","last_active":"2024-09-20T22:51:38.260423+0000","last_peered":"2024-09-20T22:51:38.260423+0000","last_clean":"2024-09-20T22:51:38.260423+0000","last_became_active":"2024-09-20T22:51:23.270962+0000","last_became_peered":"2024-09-20T22:51:23.270962+0000","last_unstale":"2024-09-20T22:51:38.260423+0000","last_undegraded":"2024-09-20T22:51:38.260423+0000","last_fullsized":"2024-09-20T22:51:38.260423+0000","mapping_epoch":602,"log_start":"423'280","ondisk_log_start":"423'280","created":94,"last_epoch_clean":603,"parent":"0.0","parent_split_bits":0,"last_scrub":"590'828","last_scrub_stamp":"2024-09-20T22:51:10.909364+0000","last_deep_scrub":"590'828","last_deep_scrub_stamp":"2024-09-20T22:51:10.909364+0000","last_clean_scrub_stamp":"2024-09-20T22:51:10.909364+0000","objects_scrubbed":27,"log_size":649,"log_dups_size":649,"ondisk_log_size":649,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:17.371854+0000","scrub_duration":120,"objects_trimmed":0,"snaptrim_duration":0.00089167800000000002,"stat_sum":{"num_bytes":48987863,"num_objects":23,"num_object_clones":19,"num_object_copies":69,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":2,"num_read":356,"num_read_kb":365041,"num_write":267,"num_write_kb":107327,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":91020358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"a7","length":"1"},{"start":"c5","length":"1"},{"start":"ce","length":"1"},{"start":"d4","length":"1"},{"start":"da","length":"1"},{"start":"df","length":"1"},{"start":"e3","length":"9"},{"start":"ee","length":"4"},{"start":"f3","length":"2"},{"start":"f6","length":"2"},{"start":"f9","length":"1"},{"start":"fe","length":"3"},{"start":"106","length":"1"},{"start":"109","length":"1"}]},{"pgid":"4.a","version":"113'28","reported_seq":1216,"reported_epoch":618,"state":"active+clean","last_fresh":"2024-09-20T22:51:38.581314+0000","last_change":"2024-09-20T22:51:20.311212+0000","last_active":"2024-09-20T22:51:38.581314+0000","last_peered":"2024-09-20T22:51:38.581314+0000","last_clean":"2024-09-20T22:51:38.581314+0000","last_became_active":"2024-09-20T22:51:20.287620+0000","last_became_peered":"2024-09-20T22:51:20.287620+0000","last_unstale":"2024-09-20T22:51:38.581314+0000","last_undegraded":"2024-09-20T22:51:38.581314+0000","last_fullsized":"2024-09-20T22:51:38.581314+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'28","last_scrub_stamp":"2024-09-20T22:50:43.980368+0000","last_deep_scrub":"113'28","last_deep_scrub_stamp":"2024-09-20T22:43:12.725900+0000","last_clean_scrub_stamp":"2024-09-20T22:50:43.980368+0000","objects_scrubbed":9,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:51:54.553676+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":1144,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-20T22:51:37.538243+0000","last_change":"2024-09-20T22:51:36.440271+0000","last_active":"2024-09-20T22:51:37.538243+0000","last_peered":"2024-09-20T22:51:37.538243+0000","last_clean":"2024-09-20T22:51:37.538243+0000","last_became_active":"2024-09-20T22:41:38.161524+0000","last_became_peered":"2024-09-20T22:41:38.161524+0000","last_unstale":"2024-09-20T22:51:37.538243+0000","last_undegraded":"2024-09-20T22:51:37.538243+0000","last_fullsized":"2024-09-20T22:51:37.538243+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:36.440160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:51:36.440160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:51.318747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":1156,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-20T22:51:37.489168+0000","last_change":"2024-09-20T22:50:15.301586+0000","last_active":"2024-09-20T22:51:37.489168+0000","last_peered":"2024-09-20T22:51:37.489168+0000","last_clean":"2024-09-20T22:51:37.489168+0000","last_became_active":"2024-09-20T22:47:49.223220+0000","last_became_peered":"2024-09-20T22:47:49.223220+0000","last_unstale":"2024-09-20T22:51:37.489168+0000","last_undegraded":"2024-09-20T22:51:37.489168+0000","last_fullsized":"2024-09-20T22:51:37.489168+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:50:15.301506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:50:15.301506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:51:44.987795+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":1132,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-20T22:51:37.537772+0000","last_change":"2024-09-20T22:49:30.310175+0000","last_active":"2024-09-20T22:51:37.537772+0000","last_peered":"2024-09-20T22:51:37.537772+0000","last_clean":"2024-09-20T22:51:37.537772+0000","last_became_active":"2024-09-20T22:41:44.199395+0000","last_became_peered":"2024-09-20T22:41:44.199395+0000","last_unstale":"2024-09-20T22:51:37.537772+0000","last_undegraded":"2024-09-20T22:51:37.537772+0000","last_fullsized":"2024-09-20T22:51:37.537772+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:30.310069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:47:18.329209+0000","last_clean_scrub_stamp":"2024-09-20T22:49:30.310069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.a","version":"617'667","reported_seq":3229,"reported_epoch":617,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T22:51:38.387657+0000","last_change":"2024-09-20T22:51:38.385665+0000","last_active":"2024-09-20T22:51:38.387657+0000","last_peered":"2024-09-20T22:51:38.387657+0000","last_clean":"2024-09-20T22:51:38.387657+0000","last_became_active":"2024-09-20T22:42:52.405085+0000","last_became_peered":"2024-09-20T22:42:52.405085+0000","last_unstale":"2024-09-20T22:51:38.387657+0000","last_undegraded":"2024-09-20T22:51:38.387657+0000","last_fullsized":"2024-09-20T22:51:38.387657+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"505'466","last_scrub_stamp":"2024-09-20T22:49:47.088165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:49:47.088165+0000","objects_scrubbed":13,"log_size":667,"log_dups_size":0,"ondisk_log_size":667,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":8,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":1,"snaptrim_duration":0.88719968199999999,"stat_sum":{"num_bytes":44686047,"num_objects":17,"num_object_clones":14,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":317,"num_read_kb":351927,"num_write":221,"num_write_kb":90451,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"a7","length":"1"},{"start":"c5","length":"1"},{"start":"ce","length":"1"},{"start":"d4","length":"1"},{"start":"df","length":"1"},{"start":"e3","length":"9"},{"start":"ee","length":"4"},{"start":"f3","length":"2"},{"start":"f6","length":"1"}]},{"pgid":"4.9","version":"74'12","reported_seq":476,"reported_epoch":618,"state":"active+clean","last_fresh":"2024-09-20T22:51:38.265097+0000","last_change":"2024-09-20T22:51:20.321309+0000","last_active":"2024-09-20T22:51:38.265097+0000","last_peered":"2024-09-20T22:51:38.265097+0000","last_clean":"2024-09-20T22:51:38.265097+0000","last_became_active":"2024-09-20T22:51:20.290218+0000","last_became_peered":"2024-09-20T22:51:20.290218+0000","last_unstale":"2024-09-20T22:51:38.265097+0000","last_undegraded":"2024-09-20T22:51:38.265097+0000","last_fullsized":"2024-09-20T22:51:38.265097+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-09-20T22:50:40.914303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:50:40.914303+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:51:45.201583+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":21,"num_read_kb":12,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":1174,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-20T22:51:37.538759+0000","last_change":"2024-09-20T22:51:20.294005+0000","last_active":"2024-09-20T22:51:37.538759+0000","last_peered":"2024-09-20T22:51:37.538759+0000","last_clean":"2024-09-20T22:51:37.538759+0000","last_became_active":"2024-09-20T22:51:20.293701+0000","last_became_peered":"2024-09-20T22:51:20.293701+0000","last_unstale":"2024-09-20T22:51:37.538759+0000","last_undegraded":"2024-09-20T22:51:37.538759+0000","last_fullsized":"2024-09-20T22:51:37.538759+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:24.300441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:46:52.465940+0000","last_clean_scrub_stamp":"2024-09-20T22:49:24.300441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":1170,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-20T22:51:37.487026+0000","last_change":"2024-09-20T22:51:20.286960+0000","last_active":"2024-09-20T22:51:37.487026+0000","last_peered":"2024-09-20T22:51:37.487026+0000","last_clean":"2024-09-20T22:51:37.487026+0000","last_became_active":"2024-09-20T22:51:20.286626+0000","last_became_peered":"2024-09-20T22:51:20.286626+0000","last_unstale":"2024-09-20T22:51:37.487026+0000","last_undegraded":"2024-09-20T22:51:37.487026+0000","last_fullsized":"2024-09-20T22:51:37.487026+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:05.254319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:46:01.828080+0000","last_clean_scrub_stamp":"2024-09-20T22:51:05.254319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:11.920665+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":1130,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-20T22:51:37.498233+0000","last_change":"2024-09-20T22:51:20.279408+0000","last_active":"2024-09-20T22:51:37.498233+0000","last_peered":"2024-09-20T22:51:37.498233+0000","last_clean":"2024-09-20T22:51:37.498233+0000","last_became_active":"2024-09-20T22:51:20.279088+0000","last_became_peered":"2024-09-20T22:51:20.279088+0000","last_unstale":"2024-09-20T22:51:37.498233+0000","last_undegraded":"2024-09-20T22:51:37.498233+0000","last_fullsized":"2024-09-20T22:51:37.498233+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:50:39.266034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:55.170379+0000","last_clean_scrub_stamp":"2024-09-20T22:50:39.266034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:07.923642+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"616'916","reported_seq":3655,"reported_epoch":617,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T22:51:37.491886+0000","last_change":"2024-09-20T22:51:37.293269+0000","last_active":"2024-09-20T22:51:37.491886+0000","last_peered":"2024-09-20T22:51:37.491886+0000","last_clean":"2024-09-20T22:51:37.491886+0000","last_became_active":"2024-09-20T22:51:23.271244+0000","last_became_peered":"2024-09-20T22:51:23.271244+0000","last_unstale":"2024-09-20T22:51:37.491886+0000","last_undegraded":"2024-09-20T22:51:37.491886+0000","last_fullsized":"2024-09-20T22:51:37.491886+0000","mapping_epoch":602,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":603,"parent":"0.0","parent_split_bits":0,"last_scrub":"194'116","last_scrub_stamp":"2024-09-20T22:44:37.097674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:44:37.097674+0000","objects_scrubbed":3,"log_size":916,"log_dups_size":0,"ondisk_log_size":916,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":7,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.80270893899999995,"stat_sum":{"num_bytes":46978719,"num_objects":18,"num_object_clones":14,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":1,"num_read":473,"num_read_kb":548916,"num_write":308,"num_write_kb":102986,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":76,"num_bytes_recovered":39288293,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"a7","length":"1"},{"start":"c5","length":"1"},{"start":"ce","length":"1"},{"start":"d4","length":"1"},{"start":"df","length":"1"},{"start":"e3","length":"9"},{"start":"ee","length":"4"},{"start":"f3","length":"2"},{"start":"f6","length":"1"},{"start":"f9","length":"1"}]},{"pgid":"4.8","version":"74'15","reported_seq":1158,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-20T22:51:37.491459+0000","last_change":"2024-09-20T22:49:47.687063+0000","last_active":"2024-09-20T22:51:37.491459+0000","last_peered":"2024-09-20T22:51:37.491459+0000","last_clean":"2024-09-20T22:51:37.491459+0000","last_became_active":"2024-09-20T22:41:40.252113+0000","last_became_peered":"2024-09-20T22:41:40.252113+0000","last_unstale":"2024-09-20T22:51:37.491459+0000","last_undegraded":"2024-09-20T22:51:37.491459+0000","last_fullsized":"2024-09-20T22:51:37.491459+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-09-20T22:49:47.686984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:49:47.686984+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'2","reported_seq":1207,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-20T22:51:37.539659+0000","last_change":"2024-09-20T22:51:26.421985+0000","last_active":"2024-09-20T22:51:37.539659+0000","last_peered":"2024-09-20T22:51:37.539659+0000","last_clean":"2024-09-20T22:51:37.539659+0000","last_became_active":"2024-09-20T22:48:05.238098+0000","last_became_peered":"2024-09-20T22:48:05.238098+0000","last_unstale":"2024-09-20T22:51:37.539659+0000","last_undegraded":"2024-09-20T22:51:37.539659+0000","last_fullsized":"2024-09-20T22:51:37.539659+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-09-20T22:51:26.421896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:51:26.421896+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:53.826247+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"408'10","reported_seq":1285,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-20T22:51:37.538551+0000","last_change":"2024-09-20T22:49:47.363588+0000","last_active":"2024-09-20T22:51:37.538551+0000","last_peered":"2024-09-20T22:51:37.538551+0000","last_clean":"2024-09-20T22:51:37.538551+0000","last_became_active":"2024-09-20T22:48:05.239344+0000","last_became_peered":"2024-09-20T22:48:05.239344+0000","last_unstale":"2024-09-20T22:51:37.538551+0000","last_undegraded":"2024-09-20T22:51:37.538551+0000","last_fullsized":"2024-09-20T22:51:37.538551+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"408'10","last_scrub_stamp":"2024-09-20T22:49:47.363515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:49:47.363515+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":1137,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-20T22:51:37.491318+0000","last_change":"2024-09-20T22:49:50.693099+0000","last_active":"2024-09-20T22:51:37.491318+0000","last_peered":"2024-09-20T22:51:37.491318+0000","last_clean":"2024-09-20T22:51:37.491318+0000","last_became_active":"2024-09-20T22:47:49.227569+0000","last_became_peered":"2024-09-20T22:47:49.227569+0000","last_unstale":"2024-09-20T22:51:37.491318+0000","last_undegraded":"2024-09-20T22:51:37.491318+0000","last_fullsized":"2024-09-20T22:51:37.491318+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:50.693003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:49:50.693003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":1166,"reported_epoch":618,"state":"active+clean","last_fresh":"2024-09-20T22:51:38.581398+0000","last_change":"2024-09-20T22:50:42.949760+0000","last_active":"2024-09-20T22:51:38.581398+0000","last_peered":"2024-09-20T22:51:38.581398+0000","last_clean":"2024-09-20T22:51:38.581398+0000","last_became_active":"2024-09-20T22:47:49.213292+0000","last_became_peered":"2024-09-20T22:47:49.213292+0000","last_unstale":"2024-09-20T22:51:38.581398+0000","last_undegraded":"2024-09-20T22:51:38.581398+0000","last_fullsized":"2024-09-20T22:51:38.581398+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:50:42.949671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:50:42.949671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:51:47.981810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"74'6","reported_seq":1155,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-20T22:51:37.499096+0000","last_change":"2024-09-20T22:49:48.113810+0000","last_active":"2024-09-20T22:51:37.499096+0000","last_peered":"2024-09-20T22:51:37.499096+0000","last_clean":"2024-09-20T22:51:37.499096+0000","last_became_active":"2024-09-20T22:43:09.116887+0000","last_became_peered":"2024-09-20T22:43:09.116887+0000","last_unstale":"2024-09-20T22:51:37.499096+0000","last_undegraded":"2024-09-20T22:51:37.499096+0000","last_fullsized":"2024-09-20T22:51:37.499096+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'6","last_scrub_stamp":"2024-09-20T22:49:48.113740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:49:48.113740+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":1143,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-20T22:51:37.491781+0000","last_change":"2024-09-20T22:51:20.293964+0000","last_active":"2024-09-20T22:51:37.491781+0000","last_peered":"2024-09-20T22:51:37.491781+0000","last_clean":"2024-09-20T22:51:37.491781+0000","last_became_active":"2024-09-20T22:51:20.293232+0000","last_became_peered":"2024-09-20T22:51:20.293232+0000","last_unstale":"2024-09-20T22:51:37.491781+0000","last_undegraded":"2024-09-20T22:51:37.491781+0000","last_fullsized":"2024-09-20T22:51:37.491781+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:50:22.597796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:09.705632+0000","last_clean_scrub_stamp":"2024-09-20T22:50:22.597796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"74'1","reported_seq":1153,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-20T22:51:37.537794+0000","last_change":"2024-09-20T22:51:05.527388+0000","last_active":"2024-09-20T22:51:37.537794+0000","last_peered":"2024-09-20T22:51:37.537794+0000","last_clean":"2024-09-20T22:51:37.537794+0000","last_became_active":"2024-09-20T22:48:05.237882+0000","last_became_peered":"2024-09-20T22:48:05.237882+0000","last_unstale":"2024-09-20T22:51:37.537794+0000","last_undegraded":"2024-09-20T22:51:37.537794+0000","last_fullsized":"2024-09-20T22:51:37.537794+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-09-20T22:51:05.527264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:51:05.527264+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:12.071495+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"74'9","reported_seq":488,"reported_epoch":618,"state":"active+clean","last_fresh":"2024-09-20T22:51:38.261042+0000","last_change":"2024-09-20T22:49:11.739149+0000","last_active":"2024-09-20T22:51:38.261042+0000","last_peered":"2024-09-20T22:51:38.261042+0000","last_clean":"2024-09-20T22:51:38.261042+0000","last_became_active":"2024-09-20T22:47:49.231315+0000","last_became_peered":"2024-09-20T22:47:49.231315+0000","last_unstale":"2024-09-20T22:51:38.261042+0000","last_undegraded":"2024-09-20T22:51:38.261042+0000","last_fullsized":"2024-09-20T22:51:38.261042+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-09-20T22:49:11.739076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:49:11.739076+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":1168,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-20T22:51:37.538031+0000","last_change":"2024-09-20T22:51:12.541531+0000","last_active":"2024-09-20T22:51:37.538031+0000","last_peered":"2024-09-20T22:51:37.538031+0000","last_clean":"2024-09-20T22:51:37.538031+0000","last_became_active":"2024-09-20T22:48:05.238801+0000","last_became_peered":"2024-09-20T22:48:05.238801+0000","last_unstale":"2024-09-20T22:51:37.538031+0000","last_undegraded":"2024-09-20T22:51:37.538031+0000","last_fullsized":"2024-09-20T22:51:37.538031+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:12.541457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:51:12.541457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:38.408641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":1134,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-20T22:51:37.498986+0000","last_change":"2024-09-20T22:51:20.279846+0000","last_active":"2024-09-20T22:51:37.498986+0000","last_peered":"2024-09-20T22:51:37.498986+0000","last_clean":"2024-09-20T22:51:37.498986+0000","last_became_active":"2024-09-20T22:51:20.279397+0000","last_became_peered":"2024-09-20T22:51:20.279397+0000","last_unstale":"2024-09-20T22:51:37.498986+0000","last_undegraded":"2024-09-20T22:51:37.498986+0000","last_fullsized":"2024-09-20T22:51:37.498986+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:50:17.192813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:50:17.192813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":1164,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-20T22:51:37.489421+0000","last_change":"2024-09-20T22:50:16.268746+0000","last_active":"2024-09-20T22:51:37.489421+0000","last_peered":"2024-09-20T22:51:37.489421+0000","last_clean":"2024-09-20T22:51:37.489421+0000","last_became_active":"2024-09-20T22:48:05.239710+0000","last_became_peered":"2024-09-20T22:48:05.239710+0000","last_unstale":"2024-09-20T22:51:37.489421+0000","last_undegraded":"2024-09-20T22:51:37.489421+0000","last_fullsized":"2024-09-20T22:51:37.489421+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:50:16.268655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:50:16.268655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"74'9","reported_seq":1157,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-20T22:51:37.491762+0000","last_change":"2024-09-20T22:51:05.675215+0000","last_active":"2024-09-20T22:51:37.491762+0000","last_peered":"2024-09-20T22:51:37.491762+0000","last_clean":"2024-09-20T22:51:37.491762+0000","last_became_active":"2024-09-20T22:41:40.252486+0000","last_became_peered":"2024-09-20T22:41:40.252486+0000","last_unstale":"2024-09-20T22:51:37.491762+0000","last_undegraded":"2024-09-20T22:51:37.491762+0000","last_fullsized":"2024-09-20T22:51:37.491762+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-09-20T22:51:05.675095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:51:05.675095+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:18.851214+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":486,"reported_epoch":618,"state":"active+clean","last_fresh":"2024-09-20T22:51:38.260891+0000","last_change":"2024-09-20T22:50:44.761567+0000","last_active":"2024-09-20T22:51:38.260891+0000","last_peered":"2024-09-20T22:51:38.260891+0000","last_clean":"2024-09-20T22:51:38.260891+0000","last_became_active":"2024-09-20T22:47:49.229936+0000","last_became_peered":"2024-09-20T22:47:49.229936+0000","last_unstale":"2024-09-20T22:51:38.260891+0000","last_undegraded":"2024-09-20T22:51:38.260891+0000","last_fullsized":"2024-09-20T22:51:38.260891+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:50:44.761495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:50:44.761495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:10.339412+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"74'8","reported_seq":1260,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-20T22:51:37.498574+0000","last_change":"2024-09-20T22:50:38.269118+0000","last_active":"2024-09-20T22:51:37.498574+0000","last_peered":"2024-09-20T22:51:37.498574+0000","last_clean":"2024-09-20T22:51:37.498574+0000","last_became_active":"2024-09-20T22:41:42.186352+0000","last_became_peered":"2024-09-20T22:41:42.186352+0000","last_unstale":"2024-09-20T22:51:37.498574+0000","last_undegraded":"2024-09-20T22:51:37.498574+0000","last_fullsized":"2024-09-20T22:51:37.498574+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-09-20T22:50:38.269010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:50:38.269010+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:08.016477+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":446,"reported_epoch":618,"state":"active+clean","last_fresh":"2024-09-20T22:51:38.265881+0000","last_change":"2024-09-20T22:50:21.999590+0000","last_active":"2024-09-20T22:51:38.265881+0000","last_peered":"2024-09-20T22:51:38.265881+0000","last_clean":"2024-09-20T22:51:38.265881+0000","last_became_active":"2024-09-20T22:48:05.249656+0000","last_became_peered":"2024-09-20T22:48:05.249656+0000","last_unstale":"2024-09-20T22:51:38.265881+0000","last_undegraded":"2024-09-20T22:51:38.265881+0000","last_fullsized":"2024-09-20T22:51:38.265881+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:50:21.999521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:50:21.999521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"74'10","reported_seq":1165,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-20T22:51:37.498246+0000","last_change":"2024-09-20T22:51:20.298112+0000","last_active":"2024-09-20T22:51:37.498246+0000","last_peered":"2024-09-20T22:51:37.498246+0000","last_clean":"2024-09-20T22:51:37.498246+0000","last_became_active":"2024-09-20T22:51:20.281859+0000","last_became_peered":"2024-09-20T22:51:20.281859+0000","last_unstale":"2024-09-20T22:51:37.498246+0000","last_undegraded":"2024-09-20T22:51:37.498246+0000","last_fullsized":"2024-09-20T22:51:37.498246+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-09-20T22:50:15.240945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:50:15.240945+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:51:45.124881+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":1156,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-20T22:51:37.538424+0000","last_change":"2024-09-20T22:49:54.381836+0000","last_active":"2024-09-20T22:51:37.538424+0000","last_peered":"2024-09-20T22:51:37.538424+0000","last_clean":"2024-09-20T22:51:37.538424+0000","last_became_active":"2024-09-20T22:48:05.240231+0000","last_became_peered":"2024-09-20T22:48:05.240231+0000","last_unstale":"2024-09-20T22:51:37.538424+0000","last_undegraded":"2024-09-20T22:51:37.538424+0000","last_fullsized":"2024-09-20T22:51:37.538424+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:54.381721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:49:54.381721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"600'11","reported_seq":1300,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-20T22:51:37.491539+0000","last_change":"2024-09-20T22:51:20.303774+0000","last_active":"2024-09-20T22:51:37.491539+0000","last_peered":"2024-09-20T22:51:37.491539+0000","last_clean":"2024-09-20T22:51:37.491539+0000","last_became_active":"2024-09-20T22:51:20.283393+0000","last_became_peered":"2024-09-20T22:51:20.283393+0000","last_unstale":"2024-09-20T22:51:37.491539+0000","last_undegraded":"2024-09-20T22:51:37.491539+0000","last_fullsized":"2024-09-20T22:51:37.491539+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'10","last_scrub_stamp":"2024-09-20T22:49:51.715471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:49:51.715471+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:51:43.100811+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":478,"reported_epoch":618,"state":"active+clean","last_fresh":"2024-09-20T22:51:38.260614+0000","last_change":"2024-09-20T22:50:21.853169+0000","last_active":"2024-09-20T22:51:38.260614+0000","last_peered":"2024-09-20T22:51:38.260614+0000","last_clean":"2024-09-20T22:51:38.260614+0000","last_became_active":"2024-09-20T22:47:49.230674+0000","last_became_peered":"2024-09-20T22:47:49.230674+0000","last_unstale":"2024-09-20T22:51:38.260614+0000","last_undegraded":"2024-09-20T22:51:38.260614+0000","last_fullsized":"2024-09-20T22:51:38.260614+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:50:21.853070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:50:21.853070+0000","last_clean_scrub_stamp":"2024-09-20T22:50:21.853070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"74'11","reported_seq":484,"reported_epoch":618,"state":"active+clean","last_fresh":"2024-09-20T22:51:38.265905+0000","last_change":"2024-09-20T22:51:20.317969+0000","last_active":"2024-09-20T22:51:38.265905+0000","last_peered":"2024-09-20T22:51:38.265905+0000","last_clean":"2024-09-20T22:51:38.265905+0000","last_became_active":"2024-09-20T22:51:20.289584+0000","last_became_peered":"2024-09-20T22:51:20.289584+0000","last_unstale":"2024-09-20T22:51:38.265905+0000","last_undegraded":"2024-09-20T22:51:38.265905+0000","last_fullsized":"2024-09-20T22:51:38.265905+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-09-20T22:51:07.926382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:51:07.926382+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:30.537714+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":454,"reported_epoch":618,"state":"active+clean","last_fresh":"2024-09-20T22:51:38.265255+0000","last_change":"2024-09-20T22:51:08.874933+0000","last_active":"2024-09-20T22:51:38.265255+0000","last_peered":"2024-09-20T22:51:38.265255+0000","last_clean":"2024-09-20T22:51:38.265255+0000","last_became_active":"2024-09-20T22:48:05.252026+0000","last_became_peered":"2024-09-20T22:48:05.252026+0000","last_unstale":"2024-09-20T22:51:38.265255+0000","last_undegraded":"2024-09-20T22:51:38.265255+0000","last_fullsized":"2024-09-20T22:51:38.265255+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:08.874812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:51:08.874812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:10.748714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":46,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-20T22:51:37.491471+0000","last_change":"2024-09-20T22:51:20.291857+0000","last_active":"2024-09-20T22:51:37.491471+0000","last_peered":"2024-09-20T22:51:37.491471+0000","last_clean":"2024-09-20T22:51:37.491471+0000","last_became_active":"2024-09-20T22:51:20.291264+0000","last_became_peered":"2024-09-20T22:51:20.291264+0000","last_unstale":"2024-09-20T22:51:37.491471+0000","last_undegraded":"2024-09-20T22:51:37.491471+0000","last_fullsized":"2024-09-20T22:51:37.491471+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:06.651167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:51:06.651167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:34.034653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":1136,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-20T22:51:37.497735+0000","last_change":"2024-09-20T22:50:20.207052+0000","last_active":"2024-09-20T22:51:37.497735+0000","last_peered":"2024-09-20T22:51:37.497735+0000","last_clean":"2024-09-20T22:51:37.497735+0000","last_became_active":"2024-09-20T22:47:49.216141+0000","last_became_peered":"2024-09-20T22:47:49.216141+0000","last_unstale":"2024-09-20T22:51:37.497735+0000","last_undegraded":"2024-09-20T22:51:37.497735+0000","last_fullsized":"2024-09-20T22:51:37.497735+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:50:20.206984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:50:20.206984+0000","last_clean_scrub_stamp":"2024-09-20T22:50:20.206984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"404'13","reported_seq":783,"reported_epoch":618,"state":"active+clean","last_fresh":"2024-09-20T22:51:38.581459+0000","last_change":"2024-09-20T22:51:25.879860+0000","last_active":"2024-09-20T22:51:38.581459+0000","last_peered":"2024-09-20T22:51:38.581459+0000","last_clean":"2024-09-20T22:51:38.581459+0000","last_became_active":"2024-09-20T22:51:20.283868+0000","last_became_peered":"2024-09-20T22:51:20.283868+0000","last_unstale":"2024-09-20T22:51:38.581459+0000","last_undegraded":"2024-09-20T22:51:38.581459+0000","last_fullsized":"2024-09-20T22:51:38.581459+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"404'13","last_scrub_stamp":"2024-09-20T22:51:25.879750+0000","last_deep_scrub":"404'13","last_deep_scrub_stamp":"2024-09-20T22:48:04.276282+0000","last_clean_scrub_stamp":"2024-09-20T22:51:25.879750+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:35.823216+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,0],"acting":[6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":1171,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-20T22:51:37.538094+0000","last_change":"2024-09-20T22:49:56.396435+0000","last_active":"2024-09-20T22:51:37.538094+0000","last_peered":"2024-09-20T22:51:37.538094+0000","last_clean":"2024-09-20T22:51:37.538094+0000","last_became_active":"2024-09-20T22:48:05.238812+0000","last_became_peered":"2024-09-20T22:48:05.238812+0000","last_unstale":"2024-09-20T22:51:37.538094+0000","last_undegraded":"2024-09-20T22:51:37.538094+0000","last_fullsized":"2024-09-20T22:51:37.538094+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:56.396347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:31.138067+0000","last_clean_scrub_stamp":"2024-09-20T22:49:56.396347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":1146,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-20T22:51:37.498317+0000","last_change":"2024-09-20T22:51:20.288308+0000","last_active":"2024-09-20T22:51:37.498317+0000","last_peered":"2024-09-20T22:51:37.498317+0000","last_clean":"2024-09-20T22:51:37.498317+0000","last_became_active":"2024-09-20T22:51:20.288000+0000","last_became_peered":"2024-09-20T22:51:20.288000+0000","last_unstale":"2024-09-20T22:51:37.498317+0000","last_undegraded":"2024-09-20T22:51:37.498317+0000","last_fullsized":"2024-09-20T22:51:37.498317+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:50:44.313182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:50:44.313182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:01.030703+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":484,"reported_epoch":618,"state":"active+clean","last_fresh":"2024-09-20T22:51:38.260312+0000","last_change":"2024-09-20T22:51:20.278471+0000","last_active":"2024-09-20T22:51:38.260312+0000","last_peered":"2024-09-20T22:51:38.260312+0000","last_clean":"2024-09-20T22:51:38.260312+0000","last_became_active":"2024-09-20T22:51:20.278147+0000","last_became_peered":"2024-09-20T22:51:20.278147+0000","last_unstale":"2024-09-20T22:51:38.260312+0000","last_undegraded":"2024-09-20T22:51:38.260312+0000","last_fullsized":"2024-09-20T22:51:38.260312+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:50:35.895483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:50:35.895483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:51:51.402592+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"470'16","reported_seq":955,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-20T22:51:37.491104+0000","last_change":"2024-09-20T22:50:39.665781+0000","last_active":"2024-09-20T22:51:37.491104+0000","last_peered":"2024-09-20T22:51:37.491104+0000","last_clean":"2024-09-20T22:51:37.491104+0000","last_became_active":"2024-09-20T22:44:09.170785+0000","last_became_peered":"2024-09-20T22:44:09.170785+0000","last_unstale":"2024-09-20T22:51:37.491104+0000","last_undegraded":"2024-09-20T22:51:37.491104+0000","last_fullsized":"2024-09-20T22:51:37.491104+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"470'16","last_scrub_stamp":"2024-09-20T22:50:39.665682+0000","last_deep_scrub":"470'16","last_deep_scrub_stamp":"2024-09-20T22:49:10.659868+0000","last_clean_scrub_stamp":"2024-09-20T22:50:39.665682+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:51:59.605546+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":1151,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-20T22:51:37.491888+0000","last_change":"2024-09-20T22:51:20.294181+0000","last_active":"2024-09-20T22:51:37.491888+0000","last_peered":"2024-09-20T22:51:37.491888+0000","last_clean":"2024-09-20T22:51:37.491888+0000","last_became_active":"2024-09-20T22:51:20.293645+0000","last_became_peered":"2024-09-20T22:51:20.293645+0000","last_unstale":"2024-09-20T22:51:37.491888+0000","last_undegraded":"2024-09-20T22:51:37.491888+0000","last_fullsized":"2024-09-20T22:51:37.491888+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:50:13.688169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:45:24.765414+0000","last_clean_scrub_stamp":"2024-09-20T22:50:13.688169+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":1160,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-20T22:51:37.537341+0000","last_change":"2024-09-20T22:51:10.610718+0000","last_active":"2024-09-20T22:51:37.537341+0000","last_peered":"2024-09-20T22:51:37.537341+0000","last_clean":"2024-09-20T22:51:37.537341+0000","last_became_active":"2024-09-20T22:48:05.238334+0000","last_became_peered":"2024-09-20T22:48:05.238334+0000","last_unstale":"2024-09-20T22:51:37.537341+0000","last_undegraded":"2024-09-20T22:51:37.537341+0000","last_fullsized":"2024-09-20T22:51:37.537341+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:10.610588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:50.174861+0000","last_clean_scrub_stamp":"2024-09-20T22:51:10.610588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:16.089343+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":1136,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-20T22:51:37.498406+0000","last_change":"2024-09-20T22:51:04.379662+0000","last_active":"2024-09-20T22:51:37.498406+0000","last_peered":"2024-09-20T22:51:37.498406+0000","last_clean":"2024-09-20T22:51:37.498406+0000","last_became_active":"2024-09-20T22:47:49.214487+0000","last_became_peered":"2024-09-20T22:47:49.214487+0000","last_unstale":"2024-09-20T22:51:37.498406+0000","last_undegraded":"2024-09-20T22:51:37.498406+0000","last_fullsized":"2024-09-20T22:51:37.498406+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:04.379558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:51:04.379558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:19.515045+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"74'4","reported_seq":1150,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-20T22:51:37.497822+0000","last_change":"2024-09-20T22:51:20.314500+0000","last_active":"2024-09-20T22:51:37.497822+0000","last_peered":"2024-09-20T22:51:37.497822+0000","last_clean":"2024-09-20T22:51:37.497822+0000","last_became_active":"2024-09-20T22:51:20.290269+0000","last_became_peered":"2024-09-20T22:51:20.290269+0000","last_unstale":"2024-09-20T22:51:37.497822+0000","last_undegraded":"2024-09-20T22:51:37.497822+0000","last_fullsized":"2024-09-20T22:51:37.497822+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'4","last_scrub_stamp":"2024-09-20T22:50:40.292964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:50:40.292964+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:51:46.387686+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":486,"reported_epoch":618,"state":"active+clean","last_fresh":"2024-09-20T22:51:38.261396+0000","last_change":"2024-09-20T22:50:43.756048+0000","last_active":"2024-09-20T22:51:38.261396+0000","last_peered":"2024-09-20T22:51:38.261396+0000","last_clean":"2024-09-20T22:51:38.261396+0000","last_became_active":"2024-09-20T22:47:49.231964+0000","last_became_peered":"2024-09-20T22:47:49.231964+0000","last_unstale":"2024-09-20T22:51:38.261396+0000","last_undegraded":"2024-09-20T22:51:38.261396+0000","last_fullsized":"2024-09-20T22:51:38.261396+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:50:43.755981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:50:43.755981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:51:48.026339+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":1170,"reported_epoch":618,"state":"active+clean","last_fresh":"2024-09-20T22:51:38.581312+0000","last_change":"2024-09-20T22:50:36.933344+0000","last_active":"2024-09-20T22:51:38.581312+0000","last_peered":"2024-09-20T22:51:38.581312+0000","last_clean":"2024-09-20T22:51:38.581312+0000","last_became_active":"2024-09-20T22:48:05.244659+0000","last_became_peered":"2024-09-20T22:48:05.244659+0000","last_unstale":"2024-09-20T22:51:38.581312+0000","last_undegraded":"2024-09-20T22:51:38.581312+0000","last_fullsized":"2024-09-20T22:51:38.581312+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:50:36.933255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:50:36.933255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:06.390897+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":1152,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-20T22:51:37.539227+0000","last_change":"2024-09-20T22:51:01.446349+0000","last_active":"2024-09-20T22:51:37.539227+0000","last_peered":"2024-09-20T22:51:37.539227+0000","last_clean":"2024-09-20T22:51:37.539227+0000","last_became_active":"2024-09-20T22:48:05.237765+0000","last_became_peered":"2024-09-20T22:48:05.237765+0000","last_unstale":"2024-09-20T22:51:37.539227+0000","last_undegraded":"2024-09-20T22:51:37.539227+0000","last_fullsized":"2024-09-20T22:51:37.539227+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:01.446278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:10.233487+0000","last_clean_scrub_stamp":"2024-09-20T22:51:01.446278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:12.163458+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":749,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-20T22:51:37.491601+0000","last_change":"2024-09-20T22:51:29.904430+0000","last_active":"2024-09-20T22:51:37.491601+0000","last_peered":"2024-09-20T22:51:37.491601+0000","last_clean":"2024-09-20T22:51:37.491601+0000","last_became_active":"2024-09-20T22:51:20.297739+0000","last_became_peered":"2024-09-20T22:51:20.297739+0000","last_unstale":"2024-09-20T22:51:37.491601+0000","last_undegraded":"2024-09-20T22:51:37.491601+0000","last_fullsized":"2024-09-20T22:51:37.491601+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:29.904355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:51:29.904355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:36.892277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":1168,"reported_epoch":617,"state":"active+clean","last_fresh":"2024-09-20T22:51:37.497697+0000","last_change":"2024-09-20T22:51:20.275893+0000","last_active":"2024-09-20T22:51:37.497697+0000","last_peered":"2024-09-20T22:51:37.497697+0000","last_clean":"2024-09-20T22:51:37.497697+0000","last_became_active":"2024-09-20T22:51:20.274891+0000","last_became_peered":"2024-09-20T22:51:20.274891+0000","last_unstale":"2024-09-20T22:51:37.497697+0000","last_undegraded":"2024-09-20T22:51:37.497697+0000","last_fullsized":"2024-09-20T22:51:37.497697+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:50:41.281470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:50:41.281470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:51:59.753596+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"74'11","reported_seq":1190,"reported_epoch":618,"state":"active+clean","last_fresh":"2024-09-20T22:51:38.582059+0000","last_change":"2024-09-20T22:51:20.304208+0000","last_active":"2024-09-20T22:51:38.582059+0000","last_peered":"2024-09-20T22:51:38.582059+0000","last_clean":"2024-09-20T22:51:38.582059+0000","last_became_active":"2024-09-20T22:51:20.284937+0000","last_became_peered":"2024-09-20T22:51:20.284937+0000","last_unstale":"2024-09-20T22:51:38.582059+0000","last_undegraded":"2024-09-20T22:51:38.582059+0000","last_fullsized":"2024-09-20T22:51:38.582059+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-09-20T22:50:35.897330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:50:35.897330+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:02.547748+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":478,"reported_epoch":618,"state":"active+clean","last_fresh":"2024-09-20T22:51:38.260671+0000","last_change":"2024-09-20T22:49:23.660691+0000","last_active":"2024-09-20T22:51:38.260671+0000","last_peered":"2024-09-20T22:51:38.260671+0000","last_clean":"2024-09-20T22:51:38.260671+0000","last_became_active":"2024-09-20T22:47:49.220971+0000","last_became_peered":"2024-09-20T22:47:49.220971+0000","last_unstale":"2024-09-20T22:51:38.260671+0000","last_undegraded":"2024-09-20T22:51:38.260671+0000","last_fullsized":"2024-09-20T22:51:38.260671+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:23.660625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:51.667341+0000","last_clean_scrub_stamp":"2024-09-20T22:49:23.660625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":487839888,"num_objects":218,"num_object_clones":171,"num_object_copies":654,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":36,"num_objects_unfound":0,"num_objects_dirty":218,"num_whiteouts":10,"num_read":4262,"num_read_kb":4502038,"num_write":2996,"num_write_kb":1126107,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":749,"num_bytes_recovered":427979194,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1088466944,"data_stored":1554806037,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7480,"ondisk_log_size":7480,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":84,"ondisk_log_size":84,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":680,"num_read_kb":437,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":307,"num_bytes_recovered":7244,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":499712,"data_stored":13224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":453,"ondisk_log_size":453,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":1034,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":4931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":590,"num_read_kb":585,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1186,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":384,"ondisk_log_size":384,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070588,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":216532,"kb_used_data":171456,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":93528620,"statfs":{"total":95995035648,"available":95773306880,"internally_reserved":0,"allocated":175570944,"data_stored":223886456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":46135754},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430338,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":193432,"kb_used_data":148352,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":93551720,"statfs":{"total":95995035648,"available":95796961280,"internally_reserved":0,"allocated":151912448,"data_stored":207317434,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691975,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":159304,"kb_used_data":105908,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93585848,"statfs":{"total":95995035648,"available":95831908352,"internally_reserved":0,"allocated":108449792,"data_stored":165681997,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986316,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":121344,"kb_used_data":94308,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93623808,"statfs":{"total":95995035648,"available":95870779392,"internally_reserved":0,"allocated":96571392,"data_stored":132022192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182546,"num_pgs":77,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":279928,"kb_used_data":230692,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":93465224,"statfs":{"total":95995035648,"available":95708389376,"internally_reserved":0,"allocated":236228608,"data_stored":321666974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":50395591},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411479,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":219872,"kb_used_data":162316,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":93525280,"statfs":{"total":95995035648,"available":95769886720,"internally_reserved":0,"allocated":166211584,"data_stored":241116968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":55834575004,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738529,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":201572,"kb_used_data":139856,"kb_used_omap":1,"kb_used_meta":61694,"kb_avail":93543580,"statfs":{"total":95995035648,"available":95788625920,"internally_reserved":0,"allocated":143212544,"data_stored":213786747,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":63175109},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":137846784,"data_stored":210544448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49590272,"data_stored":81038267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":160620544,"data_stored":233990962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":230260736,"data_stored":317833393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":91217920,"data_stored":128780854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":102510592,"data_stored":158226887,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":146542592,"data_stored":204074754,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":169877504,"data_stored":220316472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-20T22:51:42.953 INFO:tasks.thrashosds.thrasher:Setting 6.e to [1, 2, 5] 2024-09-20T22:51:42.953 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '6.e', '1', '2', '5'] 2024-09-20T22:51:42.953 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd pg-upmap 6.e 1 2 5 2024-09-20T22:51:42.965 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:51:42.983 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-20T22:51:42.986 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:42.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:42.987 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:51:43.087 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:51:43.109 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:43.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:43.110 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:51:43.211 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:51:43.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:43.233 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:43.233 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:51:43.297 INFO:tasks.rados.rados.0.smithi084.stdout:2443: done (2 left) 2024-09-20T22:51:43.297 INFO:tasks.rados.rados.0.smithi084.stdout:2445: done (1 left) 2024-09-20T22:51:43.298 INFO:tasks.rados.rados.0.smithi084.stdout:2446: done (0 left) 2024-09-20T22:51:43.298 INFO:tasks.rados.rados.0.smithi084.stdout:2447: snap_remove snap 250 2024-09-20T22:51:43.333 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:51:43.355 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:43.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:43.356 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:51:43.456 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:51:43.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:43.479 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:43.479 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:51:43.540 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:43 smithi084 ceph-mon[40024]: pgmap v834: 158 pgs: 2 active+clean+snaptrim, 2 active+remapped+backfill_toofull, 154 active+clean; 466 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 11 MiB/s rd, 2.3 MiB/s wr, 16 op/s; 36/1251 objects misplaced (2.878%) 2024-09-20T22:51:43.540 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:43 smithi084 ceph-mon[40024]: from='client.16473 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T22:51:43.541 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:43 smithi084 ceph-mon[40024]: osdmap e622: 8 total, 8 up, 7 in 2024-09-20T22:51:43.541 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:43 smithi084 ceph-mon[29774]: pgmap v834: 158 pgs: 2 active+clean+snaptrim, 2 active+remapped+backfill_toofull, 154 active+clean; 466 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 11 MiB/s rd, 2.3 MiB/s wr, 16 op/s; 36/1251 objects misplaced (2.878%) 2024-09-20T22:51:43.541 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:43 smithi084 ceph-mon[29774]: from='client.16473 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T22:51:43.541 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:43 smithi084 ceph-mon[29774]: osdmap e622: 8 total, 8 up, 7 in 2024-09-20T22:51:43.581 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:51:43.602 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:43.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:43.603 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:51:43.640 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-20T22:51:43.641 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:51:43.704 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:51:43.726 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:43.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:43.727 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:51:43.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:43 smithi154 ceph-mon[33264]: pgmap v834: 158 pgs: 2 active+clean+snaptrim, 2 active+remapped+backfill_toofull, 154 active+clean; 466 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 11 MiB/s rd, 2.3 MiB/s wr, 16 op/s; 36/1251 objects misplaced (2.878%) 2024-09-20T22:51:43.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:43 smithi154 ceph-mon[33264]: from='client.16473 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T22:51:43.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:43 smithi154 ceph-mon[33264]: osdmap e622: 8 total, 8 up, 7 in 2024-09-20T22:51:43.828 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:51:43.850 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:43.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:43.851 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:51:43.952 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:51:43.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:43.975 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:43.975 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:51:44.076 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:51:44.098 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:44.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:44.099 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:51:44.200 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:51:44.221 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:44.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:44.222 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:51:44.286 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is set 2024-09-20T22:51:44.297 INFO:tasks.rados.rados.0.smithi084.stdout:2447: done (0 left) 2024-09-20T22:51:44.297 INFO:tasks.rados.rados.0.smithi084.stdout:2448: rollback oid 40 current snap is 272 2024-09-20T22:51:44.297 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 40 to 269 2024-09-20T22:51:44.297 INFO:tasks.rados.rados.0.smithi084.stdout:2449: snap_create 2024-09-20T22:51:44.307 INFO:tasks.rados.rados.0.smithi084.stdout:2448: finishing rollback tid 0 to smithi08477203-40 2024-09-20T22:51:44.307 INFO:tasks.rados.rados.0.smithi084.stdout:2448: finishing rollback tid 1 to smithi08477203-40 2024-09-20T22:51:44.307 INFO:tasks.rados.rados.0.smithi084.stdout:2448: finishing rollback tid 2 to smithi08477203-40 2024-09-20T22:51:44.307 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 40 v 1227 (ObjNum 833 snap 269 seq_num 833) dirty exists 2024-09-20T22:51:44.323 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:51:44.343 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:44.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:44.344 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:51:44.356 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.2 dump_blocked_ops 2024-09-20T22:51:44.445 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:51:44.466 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:44.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:44.467 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:51:44.557 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:44 smithi084 ceph-mon[40024]: osdmap e623: 8 total, 8 up, 7 in 2024-09-20T22:51:44.558 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:44 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/2887108861' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:51:44.558 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:44 smithi084 ceph-mon[29774]: osdmap e623: 8 total, 8 up, 7 in 2024-09-20T22:51:44.558 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:44 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/2887108861' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:51:44.567 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:51:44.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:44.589 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:44.589 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:51:44.690 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:51:44.711 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:44.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:44.712 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:51:44.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:44 smithi154 ceph-mon[33264]: osdmap e623: 8 total, 8 up, 7 in 2024-09-20T22:51:44.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:44 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/2887108861' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:51:44.813 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:51:44.833 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:44.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:44.834 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:51:44.935 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:51:44.935 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:51:44.935 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:51:44.939 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:51:44.956 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:44.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:44.957 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:51:45.057 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:51:45.078 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:45.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:45.079 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:51:45.180 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:51:45.201 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:45.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:45.202 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:51:45.295 INFO:tasks.rados.rados.0.smithi084.stdout:2448: done (1 left) 2024-09-20T22:51:45.295 INFO:tasks.rados.rados.0.smithi084.stdout:2449: done (0 left) 2024-09-20T22:51:45.295 INFO:tasks.rados.rados.0.smithi084.stdout:2450: write (excl) oid 5 current snap is 273 2024-09-20T22:51:45.295 INFO:tasks.rados.rados.0.smithi084.stdout:2450: seq_num 841 ranges {534687=483239,1613474=412283} 2024-09-20T22:51:45.300 INFO:tasks.rados.rados.0.smithi084.stdout:2450: writing smithi08477203-5 from 534687 to 1017926 tid 1 2024-09-20T22:51:45.302 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:51:45.306 INFO:tasks.rados.rados.0.smithi084.stdout:2450: writing smithi08477203-5 from 1613474 to 2025757 tid 2 2024-09-20T22:51:45.306 INFO:tasks.rados.rados.0.smithi084.stdout:2451: snap_create 2024-09-20T22:51:45.307 INFO:tasks.rados.rados.0.smithi084.stdout:2450: finishing write tid 1 to smithi08477203-5 2024-09-20T22:51:45.309 INFO:tasks.rados.rados.0.smithi084.stdout:2450: finishing write tid 2 to smithi08477203-5 2024-09-20T22:51:45.309 INFO:tasks.rados.rados.0.smithi084.stdout:2450: finishing write tid 3 to smithi08477203-5 2024-09-20T22:51:45.310 INFO:tasks.rados.rados.0.smithi084.stdout:2450: finishing write tid 4 to smithi08477203-5 2024-09-20T22:51:45.310 INFO:tasks.rados.rados.0.smithi084.stdout:2450: oid 5 updating version 0 to 756 2024-09-20T22:51:45.310 INFO:tasks.rados.rados.0.smithi084.stdout:2450: oid 5 version 756 is already newer than 755 2024-09-20T22:51:45.310 INFO:tasks.rados.rados.0.smithi084.stdout:2450: oid 5 version 756 is already newer than 754 2024-09-20T22:51:45.310 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 5 v 756 (ObjNum 841 snap 273 seq_num 841) dirty exists 2024-09-20T22:51:45.310 INFO:tasks.rados.rados.0.smithi084.stdout:2450: left oid 5 (ObjNum 841 snap 273 seq_num 841) 2024-09-20T22:51:45.326 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:45.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:45.327 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:51:45.364 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:45 smithi084 ceph-mon[29774]: pgmap v837: 158 pgs: 2 active+clean+snaptrim, 2 active+remapped+backfill_toofull, 154 active+clean; 472 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 8.5 MiB/s rd, 2.1 MiB/s wr, 16 op/s; 36/1269 objects misplaced (2.837%) 2024-09-20T22:51:45.364 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:45 smithi084 ceph-mon[29774]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:51:45.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:45 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/2887108861' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T22:51:45.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:45 smithi084 ceph-mon[29774]: osdmap e624: 8 total, 8 up, 7 in 2024-09-20T22:51:45.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:45 smithi084 ceph-mon[40024]: pgmap v837: 158 pgs: 2 active+clean+snaptrim, 2 active+remapped+backfill_toofull, 154 active+clean; 472 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 8.5 MiB/s rd, 2.1 MiB/s wr, 16 op/s; 36/1269 objects misplaced (2.837%) 2024-09-20T22:51:45.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:45 smithi084 ceph-mon[40024]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:51:45.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:45 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/2887108861' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T22:51:45.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:45 smithi084 ceph-mon[40024]: osdmap e624: 8 total, 8 up, 7 in 2024-09-20T22:51:45.428 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:51:45.449 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:45.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:45.450 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:51:45.551 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:51:45.571 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:45.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:45.572 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:51:45.673 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:51:45.696 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:45.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:45.697 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:51:45.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:45 smithi154 ceph-mon[33264]: pgmap v837: 158 pgs: 2 active+clean+snaptrim, 2 active+remapped+backfill_toofull, 154 active+clean; 472 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 8.5 MiB/s rd, 2.1 MiB/s wr, 16 op/s; 36/1269 objects misplaced (2.837%) 2024-09-20T22:51:45.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:45 smithi154 ceph-mon[33264]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:51:45.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:45 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/2887108861' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T22:51:45.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:45 smithi154 ceph-mon[33264]: osdmap e624: 8 total, 8 up, 7 in 2024-09-20T22:51:45.798 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:51:45.821 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:45.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:45.822 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:51:45.922 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:51:45.943 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:45.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:45.944 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:51:45.987 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:51:45.987 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:51:46.004 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:51:46.004 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:51:46.019 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:51:46.020 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:51:46.035 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-20T22:51:46.035 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T22:51:46.045 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:51:46.052 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:51:46.053 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:51:46.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:46.066 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:46.066 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:51:46.070 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:51:46.070 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:51:46.086 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:51:46.086 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:51:46.103 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:51:46.103 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:51:46.167 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:51:46.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:46.189 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:46.190 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:51:46.290 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:51:46.307 INFO:teuthology.orchestra.run.smithi084.stderr:set 6.e pg_upmap mapping to [1,2,5] 2024-09-20T22:51:46.313 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:46.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:46.314 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:51:46.314 INFO:tasks.rados.rados.0.smithi084.stdout:2450: done (1 left) 2024-09-20T22:51:46.314 INFO:tasks.rados.rados.0.smithi084.stdout:2451: done (0 left) 2024-09-20T22:51:46.314 INFO:tasks.rados.rados.0.smithi084.stdout:2452: delete oid 5 current snap is 274 2024-09-20T22:51:46.319 INFO:tasks.rados.rados.0.smithi084.stdout:2452: done (0 left) 2024-09-20T22:51:46.319 INFO:tasks.rados.rados.0.smithi084.stdout:2453: write oid 18 current snap is 274 2024-09-20T22:51:46.319 INFO:tasks.rados.rados.0.smithi084.stdout:2453: seq_num 843 ranges {568301=633737,1614085=415952} 2024-09-20T22:51:46.326 INFO:tasks.rados.rados.0.smithi084.stdout:2453: writing smithi08477203-18 from 568301 to 1202038 tid 1 2024-09-20T22:51:46.331 INFO:tasks.rados.rados.0.smithi084.stdout:2453: writing smithi08477203-18 from 1614085 to 2030037 tid 2 2024-09-20T22:51:46.331 INFO:tasks.rados.rados.0.smithi084.stdout:2454: write oid 14 current snap is 274 2024-09-20T22:51:46.331 INFO:tasks.rados.rados.0.smithi084.stdout:2453: finishing write tid 1 to smithi08477203-18 2024-09-20T22:51:46.331 INFO:tasks.rados.rados.0.smithi084.stdout:2454: seq_num 844 ranges {585108=425339,1788920=396188} 2024-09-20T22:51:46.336 INFO:tasks.rados.rados.0.smithi084.stdout:2454: writing smithi08477203-14 from 585108 to 1010447 tid 1 2024-09-20T22:51:46.344 INFO:tasks.rados.rados.0.smithi084.stdout:2454: writing smithi08477203-14 from 1788920 to 2185108 tid 2 2024-09-20T22:51:46.344 INFO:tasks.rados.rados.0.smithi084.stdout:2455: delete oid 23 current snap is 274 2024-09-20T22:51:46.344 INFO:tasks.rados.rados.0.smithi084.stdout:2453: finishing write tid 2 to smithi08477203-18 2024-09-20T22:51:46.344 INFO:tasks.rados.rados.0.smithi084.stdout:2453: finishing write tid 3 to smithi08477203-18 2024-09-20T22:51:46.344 INFO:tasks.rados.rados.0.smithi084.stdout:2453: finishing write tid 4 to smithi08477203-18 2024-09-20T22:51:46.344 INFO:tasks.rados.rados.0.smithi084.stdout:2453: oid 18 updating version 0 to 1236 2024-09-20T22:51:46.344 INFO:tasks.rados.rados.0.smithi084.stdout:2453: oid 18 updating version 1236 to 1237 2024-09-20T22:51:46.344 INFO:tasks.rados.rados.0.smithi084.stdout:2453: oid 18 version 1237 is already newer than 1235 2024-09-20T22:51:46.344 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 18 v 1237 (ObjNum 843 snap 274 seq_num 843) dirty exists 2024-09-20T22:51:46.344 INFO:tasks.rados.rados.0.smithi084.stdout:2453: left oid 18 (ObjNum 843 snap 274 seq_num 843) 2024-09-20T22:51:46.344 INFO:tasks.rados.rados.0.smithi084.stdout:2454: finishing write tid 1 to smithi08477203-14 2024-09-20T22:51:46.346 INFO:tasks.rados.rados.0.smithi084.stdout:2454: finishing write tid 2 to smithi08477203-14 2024-09-20T22:51:46.346 INFO:tasks.rados.rados.0.smithi084.stdout:2453: done (2 left) 2024-09-20T22:51:46.346 INFO:tasks.rados.rados.0.smithi084.stdout:2455: done (1 left) 2024-09-20T22:51:46.346 INFO:tasks.rados.rados.0.smithi084.stdout:2456: write oid 45 current snap is 274 2024-09-20T22:51:46.346 INFO:tasks.rados.rados.0.smithi084.stdout:2454: finishing write tid 3 to smithi08477203-14 2024-09-20T22:51:46.346 INFO:tasks.rados.rados.0.smithi084.stdout:2456: seq_num 846 ranges {618722=575837,1905884=312838} 2024-09-20T22:51:46.352 INFO:tasks.rados.rados.0.smithi084.stdout:2456: writing smithi08477203-45 from 618722 to 1194559 tid 1 2024-09-20T22:51:46.358 INFO:tasks.rados.rados.0.smithi084.stdout:2456: writing smithi08477203-45 from 1905884 to 2218722 tid 2 2024-09-20T22:51:46.358 INFO:tasks.rados.rados.0.smithi084.stdout:2457: snap_remove snap 256 2024-09-20T22:51:46.415 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:51:46.436 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:46.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:46.437 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:51:46.538 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:51:46.559 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:46.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:46.560 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:51:46.614 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:46 smithi084 ceph-mon[40024]: osdmap e625: 8 total, 8 up, 7 in 2024-09-20T22:51:46.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:46 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/3981253228' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "6.e", "id": [1, 2, 5]}]: dispatch 2024-09-20T22:51:46.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:46 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/1835013053' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:51:46.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:46 smithi084 ceph-mon[29774]: osdmap e625: 8 total, 8 up, 7 in 2024-09-20T22:51:46.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:46 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/3981253228' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "6.e", "id": [1, 2, 5]}]: dispatch 2024-09-20T22:51:46.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:46 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/1835013053' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:51:46.660 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:51:46.682 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:46.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:46.683 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:51:46.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:46 smithi154 ceph-mon[33264]: osdmap e625: 8 total, 8 up, 7 in 2024-09-20T22:51:46.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:46 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/3981253228' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "6.e", "id": [1, 2, 5]}]: dispatch 2024-09-20T22:51:46.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:46 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/1835013053' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:51:46.783 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:51:46.805 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:46.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:46.806 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:51:46.867 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd unset noscrub 2024-09-20T22:51:46.907 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:51:46.931 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:46.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:46.932 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:51:47.033 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:51:47.055 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:47.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:47.056 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:51:47.157 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:51:47.179 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:47.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:47.180 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:51:47.281 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:51:47.303 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:47.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:47.304 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:51:47.324 INFO:tasks.rados.rados.0.smithi084.stdout:2457: done (2 left) 2024-09-20T22:51:47.324 INFO:tasks.rados.rados.0.smithi084.stdout:2458: delete oid 46 current snap is 274 2024-09-20T22:51:47.324 INFO:tasks.rados.rados.0.smithi084.stdout:2454: finishing write tid 4 to smithi08477203-14 2024-09-20T22:51:47.325 INFO:tasks.rados.rados.0.smithi084.stdout:2454: oid 14 updating version 0 to 1239 2024-09-20T22:51:47.325 INFO:tasks.rados.rados.0.smithi084.stdout:2454: oid 14 version 1239 is already newer than 1238 2024-09-20T22:51:47.325 INFO:tasks.rados.rados.0.smithi084.stdout:2454: oid 14 version 1239 is already newer than 1237 2024-09-20T22:51:47.325 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 14 v 1239 (ObjNum 844 snap 274 seq_num 844) dirty exists 2024-09-20T22:51:47.325 INFO:tasks.rados.rados.0.smithi084.stdout:2454: left oid 14 (ObjNum 844 snap 274 seq_num 844) 2024-09-20T22:51:47.325 INFO:tasks.rados.rados.0.smithi084.stdout:2456: finishing write tid 1 to smithi08477203-45 2024-09-20T22:51:47.325 INFO:tasks.rados.rados.0.smithi084.stdout:2456: finishing write tid 2 to smithi08477203-45 2024-09-20T22:51:47.325 INFO:tasks.rados.rados.0.smithi084.stdout:2456: finishing write tid 3 to smithi08477203-45 2024-09-20T22:51:47.325 INFO:tasks.rados.rados.0.smithi084.stdout:2456: finishing write tid 4 to smithi08477203-45 2024-09-20T22:51:47.325 INFO:tasks.rados.rados.0.smithi084.stdout:2456: oid 45 updating version 0 to 810 2024-09-20T22:51:47.325 INFO:tasks.rados.rados.0.smithi084.stdout:2456: oid 45 version 810 is already newer than 809 2024-09-20T22:51:47.325 INFO:tasks.rados.rados.0.smithi084.stdout:2456: oid 45 updating version 810 to 811 2024-09-20T22:51:47.325 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 45 v 811 (ObjNum 846 snap 274 seq_num 846) dirty exists 2024-09-20T22:51:47.326 INFO:tasks.rados.rados.0.smithi084.stdout:2456: left oid 45 (ObjNum 846 snap 274 seq_num 846) 2024-09-20T22:51:47.327 INFO:tasks.rados.rados.0.smithi084.stdout:2454: done (2 left) 2024-09-20T22:51:47.327 INFO:tasks.rados.rados.0.smithi084.stdout:2456: done (1 left) 2024-09-20T22:51:47.327 INFO:tasks.rados.rados.0.smithi084.stdout:2458: done (0 left) 2024-09-20T22:51:47.327 INFO:tasks.rados.rados.0.smithi084.stdout:2459: write oid 21 current snap is 274 2024-09-20T22:51:47.327 INFO:tasks.rados.rados.0.smithi084.stdout:2459: seq_num 848 ranges {652336=726335,1906495=345841} 2024-09-20T22:51:47.334 INFO:tasks.rados.rados.0.smithi084.stdout:2459: writing smithi08477203-21 from 652336 to 1378671 tid 1 2024-09-20T22:51:47.339 INFO:tasks.rados.rados.0.smithi084.stdout:2459: writing smithi08477203-21 from 1906495 to 2252336 tid 2 2024-09-20T22:51:47.340 INFO:tasks.rados.rados.0.smithi084.stdout:2460: write (excl) oid 46 current snap is 274 2024-09-20T22:51:47.340 INFO:tasks.rados.rados.0.smithi084.stdout:2459: finishing write tid 1 to smithi08477203-21 2024-09-20T22:51:47.340 INFO:tasks.rados.rados.0.smithi084.stdout:2460: seq_num 849 ranges {669143=401584,1564977=426959} 2024-09-20T22:51:47.345 INFO:tasks.rados.rados.0.smithi084.stdout:2460: writing smithi08477203-46 from 669143 to 1070727 tid 1 2024-09-20T22:51:47.350 INFO:tasks.rados.rados.0.smithi084.stdout:2460: writing smithi08477203-46 from 1564977 to 1991936 tid 2 2024-09-20T22:51:47.351 INFO:tasks.rados.rados.0.smithi084.stdout:2461: snap_remove snap 269 2024-09-20T22:51:47.405 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:51:47.426 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:47.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:47.427 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:51:47.528 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:51:47.549 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:47.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:47.550 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:51:47.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:47 smithi084 ceph-mon[29774]: pgmap v840: 158 pgs: 2 active+clean+snaptrim, 2 active+remapped+backfill_toofull, 154 active+clean; 472 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 8.5 MiB/s rd, 2.1 MiB/s wr, 16 op/s; 36/1269 objects misplaced (2.837%) 2024-09-20T22:51:47.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:47 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/3981253228' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "6.e", "id": [1, 2, 5]}]': finished 2024-09-20T22:51:47.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:47 smithi084 ceph-mon[29774]: osdmap e626: 8 total, 8 up, 7 in 2024-09-20T22:51:47.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:47 smithi084 ceph-mon[40024]: pgmap v840: 158 pgs: 2 active+clean+snaptrim, 2 active+remapped+backfill_toofull, 154 active+clean; 472 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 8.5 MiB/s rd, 2.1 MiB/s wr, 16 op/s; 36/1269 objects misplaced (2.837%) 2024-09-20T22:51:47.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:47 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/3981253228' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "6.e", "id": [1, 2, 5]}]': finished 2024-09-20T22:51:47.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:47 smithi084 ceph-mon[40024]: osdmap e626: 8 total, 8 up, 7 in 2024-09-20T22:51:47.651 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:51:47.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:47.674 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:47.674 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:51:47.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:47 smithi154 ceph-mon[33264]: pgmap v840: 158 pgs: 2 active+clean+snaptrim, 2 active+remapped+backfill_toofull, 154 active+clean; 472 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 8.5 MiB/s rd, 2.1 MiB/s wr, 16 op/s; 36/1269 objects misplaced (2.837%) 2024-09-20T22:51:47.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:47 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/3981253228' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "6.e", "id": [1, 2, 5]}]': finished 2024-09-20T22:51:47.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:47 smithi154 ceph-mon[33264]: osdmap e626: 8 total, 8 up, 7 in 2024-09-20T22:51:47.775 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:51:47.796 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:47.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:47.797 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:51:47.897 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:51:47.903 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-20T22:51:47.904 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2024-09-20T22:51:47.904 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2024-09-20T22:51:47.904 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2024-09-20T22:51:47.904 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-20T22:51:47.918 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:47.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:47.919 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:51:48.020 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:51:48.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:48.042 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:48.042 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:51:48.143 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:51:48.163 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:51:48.166 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:48.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:48.167 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:51:48.267 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:51:48.273 INFO:tasks.rados.rados.0.smithi084.stdout:2461: done (2 left) 2024-09-20T22:51:48.273 INFO:tasks.rados.rados.0.smithi084.stdout:2462: rollback oid 29 current snap is 274 2024-09-20T22:51:48.273 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 29 to 240 2024-09-20T22:51:48.273 INFO:tasks.rados.rados.0.smithi084.stdout:2459: finishing write tid 2 to smithi08477203-21 2024-09-20T22:51:48.273 INFO:tasks.rados.rados.0.smithi084.stdout:2459: finishing write tid 3 to smithi08477203-21 2024-09-20T22:51:48.273 INFO:tasks.rados.rados.0.smithi084.stdout:2459: finishing write tid 4 to smithi08477203-21 2024-09-20T22:51:48.273 INFO:tasks.rados.rados.0.smithi084.stdout:2459: oid 21 updating version 0 to 1234 2024-09-20T22:51:48.274 INFO:tasks.rados.rados.0.smithi084.stdout:2459: oid 21 updating version 1234 to 1235 2024-09-20T22:51:48.274 INFO:tasks.rados.rados.0.smithi084.stdout:2459: oid 21 version 1235 is already newer than 1233 2024-09-20T22:51:48.274 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 21 v 1235 (ObjNum 848 snap 274 seq_num 848) dirty exists 2024-09-20T22:51:48.274 INFO:tasks.rados.rados.0.smithi084.stdout:2459: left oid 21 (ObjNum 848 snap 274 seq_num 848) 2024-09-20T22:51:48.274 INFO:tasks.rados.rados.0.smithi084.stdout:2460: finishing write tid 1 to smithi08477203-46 2024-09-20T22:51:48.274 INFO:tasks.rados.rados.0.smithi084.stdout:2460: finishing write tid 2 to smithi08477203-46 2024-09-20T22:51:48.274 INFO:tasks.rados.rados.0.smithi084.stdout:2460: finishing write tid 3 to smithi08477203-46 2024-09-20T22:51:48.274 INFO:tasks.rados.rados.0.smithi084.stdout:2460: finishing write tid 4 to smithi08477203-46 2024-09-20T22:51:48.274 INFO:tasks.rados.rados.0.smithi084.stdout:2460: oid 46 updating version 0 to 1709 2024-09-20T22:51:48.274 INFO:tasks.rados.rados.0.smithi084.stdout:2460: oid 46 version 1709 is already newer than 1708 2024-09-20T22:51:48.274 INFO:tasks.rados.rados.0.smithi084.stdout:2460: oid 46 updating version 1709 to 1710 2024-09-20T22:51:48.274 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 46 v 1710 (ObjNum 849 snap 274 seq_num 849) dirty exists 2024-09-20T22:51:48.274 INFO:tasks.rados.rados.0.smithi084.stdout:2460: left oid 46 (ObjNum 849 snap 274 seq_num 849) 2024-09-20T22:51:48.274 INFO:tasks.rados.rados.0.smithi084.stdout:2459: done (2 left) 2024-09-20T22:51:48.275 INFO:tasks.rados.rados.0.smithi084.stdout:2460: done (1 left) 2024-09-20T22:51:48.275 INFO:tasks.rados.rados.0.smithi084.stdout:2463: snap_create 2024-09-20T22:51:48.280 INFO:tasks.rados.rados.0.smithi084.stdout:2462: finishing rollback tid 1 to smithi08477203-29 2024-09-20T22:51:48.280 INFO:tasks.rados.rados.0.smithi084.stdout:2462: finishing rollback tid 2 to smithi08477203-29 2024-09-20T22:51:48.280 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 29 v 1841 (ObjNum 440 snap 115 seq_num 440) dirty exists 2024-09-20T22:51:48.297 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:48.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:48.298 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:51:48.399 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:51:48.420 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:48.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:48.421 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:51:48.522 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:51:48.544 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:48.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:48.545 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:51:48.614 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:51:48 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: ::ffff:172.21.15.154 - - [20/Sep/2024:22:51:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-20T22:51:48.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:48 smithi084 ceph-mon[40024]: osdmap e627: 8 total, 8 up, 7 in 2024-09-20T22:51:48.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:48 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:51:48.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:48 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:51:48.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:48 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:51:48.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:48 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:51:48.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:48 smithi084 ceph-mon[40024]: osdmap e628: 8 total, 8 up, 7 in 2024-09-20T22:51:48.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:48 smithi084 ceph-mon[29774]: osdmap e627: 8 total, 8 up, 7 in 2024-09-20T22:51:48.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:48 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:51:48.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:48 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:51:48.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:48 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:51:48.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:48 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:51:48.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:48 smithi084 ceph-mon[29774]: osdmap e628: 8 total, 8 up, 7 in 2024-09-20T22:51:48.645 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:51:48.667 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:48.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:48.668 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:51:48.692 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.2 dump_historic_ops 2024-09-20T22:51:48.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:48 smithi154 ceph-mon[33264]: osdmap e627: 8 total, 8 up, 7 in 2024-09-20T22:51:48.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:48 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:51:48.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:48 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:51:48.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:48 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:51:48.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:48 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:51:48.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:48 smithi154 ceph-mon[33264]: osdmap e628: 8 total, 8 up, 7 in 2024-09-20T22:51:48.769 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:51:48.791 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:48.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:48.792 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:51:48.893 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:51:48.915 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:48.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:48.916 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:51:48.974 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-20T22:51:49.017 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:51:49.039 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:49.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:49.040 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:51:49.141 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:51:49.164 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:49.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:49.165 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:51:49.205 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:51:49.244 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:51:49.266 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:51:49.271 INFO:tasks.rados.rados.0.smithi084.stdout:2462: done (1 left) 2024-09-20T22:51:49.271 INFO:tasks.rados.rados.0.smithi084.stdout:2463: done (0 left) 2024-09-20T22:51:49.271 INFO:tasks.rados.rados.0.smithi084.stdout:2464: read oid 41 snap -1 2024-09-20T22:51:49.271 INFO:tasks.rados.rados.0.smithi084.stdout:2464: expect (ObjNum 779 snap 252 seq_num 779) 2024-09-20T22:51:49.271 INFO:tasks.rados.rados.0.smithi084.stdout:2465: delete oid 2 current snap is 275 2024-09-20T22:51:49.276 INFO:tasks.rados.rados.0.smithi084.stdout:2464: expect (ObjNum 779 snap 252 seq_num 779) 2024-09-20T22:51:49.292 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:49.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:49.293 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:51:49.385 INFO:tasks.rados.rados.0.smithi084.stdout:2464: done (1 left) 2024-09-20T22:51:49.385 INFO:tasks.rados.rados.0.smithi084.stdout:2465: done (0 left) 2024-09-20T22:51:49.385 INFO:tasks.rados.rados.0.smithi084.stdout:2466: rollback oid 9 current snap is 275 2024-09-20T22:51:49.385 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 9 to 255 2024-09-20T22:51:49.386 INFO:tasks.rados.rados.0.smithi084.stdout:2467: read oid 45 snap -1 2024-09-20T22:51:49.386 INFO:tasks.rados.rados.0.smithi084.stdout:2467: expect (ObjNum 846 snap 274 seq_num 846) 2024-09-20T22:51:49.386 INFO:tasks.rados.rados.0.smithi084.stdout:2468: write (excl) oid 47 current snap is 275 2024-09-20T22:51:49.386 INFO:tasks.rados.rados.0.smithi084.stdout:2468: seq_num 851 ranges {702757=552082,1681941=430628} 2024-09-20T22:51:49.392 INFO:tasks.rados.rados.0.smithi084.stdout:2468: writing smithi08477203-47 from 702757 to 1254839 tid 1 2024-09-20T22:51:49.394 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:51:49.397 INFO:tasks.rados.rados.0.smithi084.stdout:2468: writing smithi08477203-47 from 1681941 to 2112569 tid 2 2024-09-20T22:51:49.397 INFO:tasks.rados.rados.0.smithi084.stdout:2469: rollback oid 41 current snap is 275 2024-09-20T22:51:49.397 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 41 to 189 2024-09-20T22:51:49.398 INFO:tasks.rados.rados.0.smithi084.stdout:2467: expect (ObjNum 846 snap 274 seq_num 846) 2024-09-20T22:51:49.416 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:49.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:49.417 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:51:49.518 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:51:49.539 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:49.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:49.539 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:51:49.597 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:49 smithi084 ceph-mon[29774]: pgmap v843: 158 pgs: 2 active+clean+snaptrim, 2 active+remapped+backfill_toofull, 154 active+clean; 472 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 36/1269 objects misplaced (2.837%) 2024-09-20T22:51:49.597 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:49 smithi084 ceph-mon[29774]: osdmap e629: 8 total, 8 up, 7 in 2024-09-20T22:51:49.598 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:49 smithi084 ceph-mon[40024]: pgmap v843: 158 pgs: 2 active+clean+snaptrim, 2 active+remapped+backfill_toofull, 154 active+clean; 472 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 36/1269 objects misplaced (2.837%) 2024-09-20T22:51:49.598 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:49 smithi084 ceph-mon[40024]: osdmap e629: 8 total, 8 up, 7 in 2024-09-20T22:51:49.606 INFO:tasks.rados.rados.0.smithi084.stdout:2466: finishing rollback tid 1 to smithi08477203-9 2024-09-20T22:51:49.606 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 9 v 0 (ObjNum 97 snap 0 seq_num 2090873488) dirty dne 2024-09-20T22:51:49.606 INFO:tasks.rados.rados.0.smithi084.stdout:2468: finishing write tid 1 to smithi08477203-47 2024-09-20T22:51:49.606 INFO:tasks.rados.rados.0.smithi084.stdout:2469: finishing rollback tid 0 to smithi08477203-41 2024-09-20T22:51:49.606 INFO:tasks.rados.rados.0.smithi084.stdout:2469: finishing rollback tid 1 to smithi08477203-41 2024-09-20T22:51:49.607 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 41 v 1399 (ObjNum 209 snap 0 seq_num 2091824080) dirty dne 2024-09-20T22:51:49.607 INFO:tasks.rados.rados.0.smithi084.stdout:2468: finishing write tid 2 to smithi08477203-47 2024-09-20T22:51:49.607 INFO:tasks.rados.rados.0.smithi084.stdout:2466: done (3 left) 2024-09-20T22:51:49.607 INFO:tasks.rados.rados.0.smithi084.stdout:2467: done (2 left) 2024-09-20T22:51:49.607 INFO:tasks.rados.rados.0.smithi084.stdout:2469: done (1 left) 2024-09-20T22:51:49.607 INFO:tasks.rados.rados.0.smithi084.stdout:2470: write oid 38 current snap is 275 2024-09-20T22:51:49.607 INFO:tasks.rados.rados.0.smithi084.stdout:2470: seq_num 852 ranges {719564=743684,2140423=179141} 2024-09-20T22:51:49.613 INFO:tasks.rados.rados.0.smithi084.stdout:2470: writing smithi08477203-38 from 719564 to 1463248 tid 1 2024-09-20T22:51:49.619 INFO:tasks.rados.rados.0.smithi084.stdout:2470: writing smithi08477203-38 from 2140423 to 2319564 tid 2 2024-09-20T22:51:49.620 INFO:tasks.rados.rados.0.smithi084.stdout:2471: copy_from oid 8 from oid 13 current snap is 275 2024-09-20T22:51:49.620 INFO:tasks.rados.rados.0.smithi084.stdout:2468: finishing write tid 3 to smithi08477203-47 2024-09-20T22:51:49.620 INFO:tasks.rados.rados.0.smithi084.stdout:2468: finishing write tid 4 to smithi08477203-47 2024-09-20T22:51:49.620 INFO:tasks.rados.rados.0.smithi084.stdout:2468: oid 47 updating version 0 to 1321 2024-09-20T22:51:49.620 INFO:tasks.rados.rados.0.smithi084.stdout:2468: oid 47 updating version 1321 to 1322 2024-09-20T22:51:49.620 INFO:tasks.rados.rados.0.smithi084.stdout:2468: oid 47 updating version 1322 to 1323 2024-09-20T22:51:49.620 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 47 v 1323 (ObjNum 851 snap 275 seq_num 851) dirty exists 2024-09-20T22:51:49.620 INFO:tasks.rados.rados.0.smithi084.stdout:2468: left oid 47 (ObjNum 851 snap 275 seq_num 851) 2024-09-20T22:51:49.620 INFO:tasks.rados.rados.0.smithi084.stdout:2468: done (2 left) 2024-09-20T22:51:49.620 INFO:tasks.rados.rados.0.smithi084.stdout:2472: write (excl) oid 48 current snap is 275 2024-09-20T22:51:49.620 INFO:tasks.rados.rados.0.smithi084.stdout:2470: finishing write tid 1 to smithi08477203-38 2024-09-20T22:51:49.620 INFO:tasks.rados.rados.0.smithi084.stdout:2472: seq_num 853 ranges {736371=418933,1798905=434297} 2024-09-20T22:51:49.626 INFO:tasks.rados.rados.0.smithi084.stdout:2472: writing smithi08477203-48 from 736371 to 1155304 tid 1 2024-09-20T22:51:49.632 INFO:tasks.rados.rados.0.smithi084.stdout:2472: writing smithi08477203-48 from 1798905 to 2233202 tid 2 2024-09-20T22:51:49.632 INFO:tasks.rados.rados.0.smithi084.stdout:2473: snap_remove snap 255 2024-09-20T22:51:49.641 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:51:49.663 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:49.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:49.664 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:51:49.748 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-20T22:51:49.748 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2024-09-20T22:51:49.748 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2024-09-20T22:51:49.749 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [ 2024-09-20T22:51:49.749 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.749 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:901 7.e 7:7c62ced4:::smithi08477203-17:head [copy-from ver 48] snapc 1e=[] ondisk+write+known_if_redirected+supports_pool_eio e156)", 2024-09-20T22:51:49.749 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:43:55.624762+0000", 2024-09-20T22:51:49.749 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 474.12098558500003, 2024-09-20T22:51:49.749 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.021835856000000001, 2024-09-20T22:51:49.749 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:51:49.749 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:51:49.749 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:51:49.749 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:51:49.749 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:51:49.749 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:51:49.749 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 901 2024-09-20T22:51:49.749 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.750 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:51:49.750 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.750 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:51:49.750 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.624762+0000", 2024-09-20T22:51:49.750 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:51:49.750 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.750 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.750 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:51:49.750 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.624762+0000", 2024-09-20T22:51:49.750 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:51:49.750 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.750 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.750 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:51:49.750 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.624765+0000", 2024-09-20T22:51:49.750 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.57e-06 2024-09-20T22:51:49.751 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.751 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.751 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:51:49.751 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.624776+0000", 2024-09-20T22:51:49.751 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.078e-05 2024-09-20T22:51:49.751 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.751 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.751 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:51:49.751 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.624803+0000", 2024-09-20T22:51:49.751 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.6769999999999999e-05 2024-09-20T22:51:49.751 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.751 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.751 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:51:49.751 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.624815+0000", 2024-09-20T22:51:49.752 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2459e-05 2024-09-20T22:51:49.752 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.752 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.752 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:51:49.752 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.624936+0000", 2024-09-20T22:51:49.752 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000120467 2024-09-20T22:51:49.752 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.752 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.752 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:51:49.752 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.625046+0000", 2024-09-20T22:51:49.752 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000110222 2024-09-20T22:51:49.752 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.752 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.753 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 0,5", 2024-09-20T22:51:49.753 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.635547+0000", 2024-09-20T22:51:49.753 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010501765999999999 2024-09-20T22:51:49.753 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.753 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.753 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:51:49.753 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.644332+0000", 2024-09-20T22:51:49.753 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0087847729999999992 2024-09-20T22:51:49.753 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.753 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.753 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:51:49.753 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.646168+0000", 2024-09-20T22:51:49.753 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0018360010000000001 2024-09-20T22:51:49.753 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.754 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.754 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:51:49.754 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.646512+0000", 2024-09-20T22:51:49.754 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000343269 2024-09-20T22:51:49.754 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.754 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.754 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:51:49.754 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.646550+0000", 2024-09-20T22:51:49.754 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.8383000000000002e-05 2024-09-20T22:51:49.754 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.754 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.754 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:51:49.754 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.646598+0000", 2024-09-20T22:51:49.755 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.8396000000000002e-05 2024-09-20T22:51:49.755 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:51:49.755 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:51:49.755 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:51:49.755 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.755 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.755 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:902 7.e 7:7c62ced4:::smithi08477203-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e156)", 2024-09-20T22:51:49.755 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:43:55.624853+0000", 2024-09-20T22:51:49.755 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 474.120895507, 2024-09-20T22:51:49.755 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.015323603999999999, 2024-09-20T22:51:49.755 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:51:49.755 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:51:49.755 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-20T22:51:49.755 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:51:49.755 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:51:49.756 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:51:49.756 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 902 2024-09-20T22:51:49.756 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.756 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:51:49.756 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.756 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:51:49.756 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.624853+0000", 2024-09-20T22:51:49.756 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:51:49.756 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.756 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.756 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:51:49.756 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.624853+0000", 2024-09-20T22:51:49.756 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:51:49.756 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.756 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.757 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:51:49.757 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.624853+0000", 2024-09-20T22:51:49.757 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.1399999999999997e-07 2024-09-20T22:51:49.757 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.757 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.757 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:51:49.757 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.624859+0000", 2024-09-20T22:51:49.757 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.9610000000000002e-06 2024-09-20T22:51:49.757 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.757 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.757 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:51:49.757 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.624887+0000", 2024-09-20T22:51:49.757 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.7597e-05 2024-09-20T22:51:49.757 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.757 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.757 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:51:49.758 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.624894+0000", 2024-09-20T22:51:49.758 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.469e-06 2024-09-20T22:51:49.758 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.758 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.758 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:51:49.758 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.625219+0000", 2024-09-20T22:51:49.758 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00032538499999999998 2024-09-20T22:51:49.758 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.758 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.758 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for blocked object", 2024-09-20T22:51:49.758 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.625258+0000", 2024-09-20T22:51:49.758 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.8467000000000003e-05 2024-09-20T22:51:49.758 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.758 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.758 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:51:49.759 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.640050+0000", 2024-09-20T22:51:49.759 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0147923 2024-09-20T22:51:49.759 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.759 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.759 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:51:49.759 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.640112+0000", 2024-09-20T22:51:49.759 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.1723000000000005e-05 2024-09-20T22:51:49.759 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.759 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.759 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:51:49.759 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.640176+0000", 2024-09-20T22:51:49.759 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.4188000000000003e-05 2024-09-20T22:51:49.759 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:51:49.759 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:51:49.759 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:51:49.760 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.760 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.760 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24652.0:1420 7.19 e189/139 7:9cc0b7e2:::smithi08477203-46:head v 189'99, mlcod=174'95)", 2024-09-20T22:51:49.760 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:44:28.150258+0000", 2024-09-20T22:51:49.760 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 441.59548981099999, 2024-09-20T22:51:49.760 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014180626, 2024-09-20T22:51:49.760 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:51:49.760 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:51:49.760 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:51:49.760 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:51:49.760 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.760 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:51:49.760 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.150258+0000", 2024-09-20T22:51:49.761 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:51:49.761 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.761 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.761 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:51:49.761 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.150258+0000", 2024-09-20T22:51:49.761 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:51:49.761 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.761 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.761 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:51:49.761 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.150259+0000", 2024-09-20T22:51:49.761 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.99e-07 2024-09-20T22:51:49.761 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.761 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.761 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:51:49.762 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.150674+0000", 2024-09-20T22:51:49.762 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00041528000000000002 2024-09-20T22:51:49.762 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.762 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.762 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:51:49.762 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.150688+0000", 2024-09-20T22:51:49.762 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3399e-05 2024-09-20T22:51:49.763 INFO:journalctl@ceph.iscsi.iscsi.a.smithi154.stdout:Sep 20 22:51:49 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-iscsi-iscsi-a[59730]: debug there is no tcmu-runner data available 2024-09-20T22:51:49.763 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:49 smithi154 ceph-mon[33264]: pgmap v843: 158 pgs: 2 active+clean+snaptrim, 2 active+remapped+backfill_toofull, 154 active+clean; 472 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 36/1269 objects misplaced (2.837%) 2024-09-20T22:51:49.763 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:49 smithi154 ceph-mon[33264]: osdmap e629: 8 total, 8 up, 7 in 2024-09-20T22:51:49.763 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.764 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.764 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:51:49.764 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.150699+0000", 2024-09-20T22:51:49.764 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1022999999999999e-05 2024-09-20T22:51:49.764 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.764 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.764 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:51:49.764 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.150772+0000", 2024-09-20T22:51:49.764 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.3455999999999996e-05 2024-09-20T22:51:49.764 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.764 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.765 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:51:49.765 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.150798+0000", 2024-09-20T22:51:49.765 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.5777000000000001e-05 2024-09-20T22:51:49.765 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.765 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.765 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:51:49.765 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.164390+0000", 2024-09-20T22:51:49.765 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013591789999999999 2024-09-20T22:51:49.765 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.765 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.765 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:51:49.765 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.164439+0000", 2024-09-20T22:51:49.765 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.9202000000000001e-05 2024-09-20T22:51:49.765 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:51:49.766 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:51:49.766 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:51:49.766 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.766 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.766 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24652.0:1421 7.19 e189/139 7:9cc0b7e2:::smithi08477203-46:head v 189'100, mlcod=174'95)", 2024-09-20T22:51:49.766 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:44:28.156805+0000", 2024-09-20T22:51:49.766 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 441.58894295300001, 2024-09-20T22:51:49.766 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013180661999999999, 2024-09-20T22:51:49.766 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:51:49.766 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:51:49.766 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:51:49.766 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:51:49.766 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.767 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:51:49.767 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.156805+0000", 2024-09-20T22:51:49.767 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:51:49.767 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.767 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.767 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:51:49.767 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.156805+0000", 2024-09-20T22:51:49.767 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:51:49.767 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.767 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.767 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:51:49.767 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.156806+0000", 2024-09-20T22:51:49.767 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.074e-06 2024-09-20T22:51:49.767 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.768 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.768 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:51:49.768 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.157126+0000", 2024-09-20T22:51:49.768 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00031991399999999997 2024-09-20T22:51:49.768 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.768 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.768 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:51:49.768 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.157136+0000", 2024-09-20T22:51:49.768 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.4499999999999993e-06 2024-09-20T22:51:49.768 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.768 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.768 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:51:49.768 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.157143+0000", 2024-09-20T22:51:49.768 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.7649999999999993e-06 2024-09-20T22:51:49.768 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.768 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.769 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:51:49.769 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.157188+0000", 2024-09-20T22:51:49.769 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.4885000000000001e-05 2024-09-20T22:51:49.769 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.769 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.769 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:51:49.769 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.157208+0000", 2024-09-20T22:51:49.769 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.0015000000000001e-05 2024-09-20T22:51:49.769 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.769 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.769 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:51:49.769 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.169936+0000", 2024-09-20T22:51:49.769 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.012728204 2024-09-20T22:51:49.769 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.769 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.770 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:51:49.770 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.169986+0000", 2024-09-20T22:51:49.770 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.9354999999999998e-05 2024-09-20T22:51:49.770 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:51:49.770 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:51:49.770 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:51:49.770 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.770 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.770 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24652.0:1654 7.19 e206/139 7:9970b578:::smithi08477203-15:head v 206'114, mlcod=205'111)", 2024-09-20T22:51:49.770 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:44:45.167994+0000", 2024-09-20T22:51:49.770 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 424.57775440299997, 2024-09-20T22:51:49.770 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.01345262, 2024-09-20T22:51:49.770 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:51:49.770 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:51:49.770 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:51:49.771 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:51:49.771 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.771 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:51:49.771 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:45.167994+0000", 2024-09-20T22:51:49.771 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:51:49.771 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.771 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.771 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:51:49.771 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:45.167994+0000", 2024-09-20T22:51:49.771 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:51:49.771 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.771 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.771 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:51:49.771 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:45.167996+0000", 2024-09-20T22:51:49.771 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.5100000000000001e-06 2024-09-20T22:51:49.772 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.772 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.772 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:51:49.772 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:45.175169+0000", 2024-09-20T22:51:49.772 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0071732740000000003 2024-09-20T22:51:49.772 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.772 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.772 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:51:49.772 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:45.175208+0000", 2024-09-20T22:51:49.772 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.8597000000000003e-05 2024-09-20T22:51:49.772 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.773 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.773 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:51:49.773 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:45.175223+0000", 2024-09-20T22:51:49.773 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.4627000000000001e-05 2024-09-20T22:51:49.773 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.773 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.773 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:51:49.773 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:45.175304+0000", 2024-09-20T22:51:49.773 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.1111000000000003e-05 2024-09-20T22:51:49.773 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.773 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.773 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:51:49.773 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:45.175336+0000", 2024-09-20T22:51:49.774 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.1945e-05 2024-09-20T22:51:49.774 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.774 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.774 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:51:49.774 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:45.181392+0000", 2024-09-20T22:51:49.774 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0060566630000000003 2024-09-20T22:51:49.774 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.774 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.774 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:51:49.774 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:45.181446+0000", 2024-09-20T22:51:49.774 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.3893000000000001e-05 2024-09-20T22:51:49.774 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:51:49.774 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:51:49.775 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:51:49.775 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.775 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.775 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24652.0:1799 7.17 e229/139 7:ea720b85:::smithi08477203-26:head v 229'168, mlcod=222'164)", 2024-09-20T22:51:49.775 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:45:08.171867+0000", 2024-09-20T22:51:49.775 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 401.573881185, 2024-09-20T22:51:49.775 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014913454, 2024-09-20T22:51:49.775 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:51:49.775 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:51:49.775 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:51:49.775 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:51:49.775 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.775 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:51:49.775 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.171867+0000", 2024-09-20T22:51:49.776 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:51:49.776 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.776 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.776 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:51:49.776 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.171867+0000", 2024-09-20T22:51:49.776 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:51:49.776 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.776 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.776 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:51:49.776 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.171867+0000", 2024-09-20T22:51:49.776 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.3799999999999999e-07 2024-09-20T22:51:49.776 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.776 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.776 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:51:49.776 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.173016+0000", 2024-09-20T22:51:49.777 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0011485740000000001 2024-09-20T22:51:49.777 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.777 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.777 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:51:49.777 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.173024+0000", 2024-09-20T22:51:49.777 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.4780000000000008e-06 2024-09-20T22:51:49.777 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.777 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.777 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:51:49.777 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.173034+0000", 2024-09-20T22:51:49.777 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.7860000000000008e-06 2024-09-20T22:51:49.777 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.777 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.777 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:51:49.778 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.173098+0000", 2024-09-20T22:51:49.778 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.4405000000000004e-05 2024-09-20T22:51:49.778 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.778 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.778 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:51:49.778 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.173125+0000", 2024-09-20T22:51:49.778 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.6950000000000001e-05 2024-09-20T22:51:49.778 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.778 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.778 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:51:49.778 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.186711+0000", 2024-09-20T22:51:49.778 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.01358608 2024-09-20T22:51:49.779 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.779 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.779 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:51:49.779 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.186780+0000", 2024-09-20T22:51:49.779 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.8942999999999994e-05 2024-09-20T22:51:49.779 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:51:49.779 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:51:49.779 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:51:49.779 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.779 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.779 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:2322 7.8 7:12e9c8db:::smithi08477203-11:head [append 0~0] snapc 55=[55,53,52,51,50,4b,47,3e] ondisk+write+known_if_redirected+supports_pool_eio e258)", 2024-09-20T22:51:49.779 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:45:37.176152+0000", 2024-09-20T22:51:49.779 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 372.56959639500002, 2024-09-20T22:51:49.779 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013033472000000001, 2024-09-20T22:51:49.779 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:51:49.780 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:51:49.780 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:51:49.780 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:51:49.780 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:51:49.780 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:51:49.780 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 2322 2024-09-20T22:51:49.780 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.780 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:51:49.780 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.780 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:51:49.780 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.176152+0000", 2024-09-20T22:51:49.780 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:51:49.780 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.781 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.781 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:51:49.781 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.176152+0000", 2024-09-20T22:51:49.781 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:51:49.781 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.781 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.781 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:51:49.781 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.176152+0000", 2024-09-20T22:51:49.781 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.8899999999999999e-07 2024-09-20T22:51:49.781 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.781 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.781 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:51:49.781 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.176156+0000", 2024-09-20T22:51:49.781 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.2669999999999998e-06 2024-09-20T22:51:49.781 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.782 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.782 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:51:49.782 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.176162+0000", 2024-09-20T22:51:49.782 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.1589999999999998e-06 2024-09-20T22:51:49.782 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.782 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.782 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:51:49.782 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.176167+0000", 2024-09-20T22:51:49.782 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.3040000000000004e-06 2024-09-20T22:51:49.782 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.782 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.782 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:51:49.782 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.176212+0000", 2024-09-20T22:51:49.783 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.5448999999999998e-05 2024-09-20T22:51:49.783 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.783 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.783 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting_for_map not empty", 2024-09-20T22:51:49.783 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.176218+0000", 2024-09-20T22:51:49.783 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.5520000000000004e-06 2024-09-20T22:51:49.783 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.783 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.783 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:51:49.783 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.182721+0000", 2024-09-20T22:51:49.783 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0065028550000000001 2024-09-20T22:51:49.783 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.784 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.784 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:51:49.784 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.182787+0000", 2024-09-20T22:51:49.784 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.5995999999999996e-05 2024-09-20T22:51:49.784 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.784 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.784 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 3,7", 2024-09-20T22:51:49.784 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.183002+0000", 2024-09-20T22:51:49.784 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00021516699999999999 2024-09-20T22:51:49.784 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.784 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.784 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:51:49.785 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.185137+0000", 2024-09-20T22:51:49.785 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0021350570000000001 2024-09-20T22:51:49.785 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.785 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.785 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:51:49.785 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.185875+0000", 2024-09-20T22:51:49.785 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00073794799999999999 2024-09-20T22:51:49.785 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.785 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.785 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:51:49.785 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.188620+0000", 2024-09-20T22:51:49.785 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0027446229999999999 2024-09-20T22:51:49.786 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.786 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.786 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:51:49.786 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.189088+0000", 2024-09-20T22:51:49.786 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00046794499999999999 2024-09-20T22:51:49.786 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.786 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.786 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:51:49.786 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.189185+0000", 2024-09-20T22:51:49.786 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.7560999999999996e-05 2024-09-20T22:51:49.786 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:51:49.786 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:51:49.786 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:51:49.787 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.787 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.787 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24652.0:2318 7.5 e258/137 7:a160b5d4:::smithi08477203-16:head v 258'43, mlcod=239'39)", 2024-09-20T22:51:49.787 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:45:37.192419+0000", 2024-09-20T22:51:49.787 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 372.55332909800001, 2024-09-20T22:51:49.787 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013974723, 2024-09-20T22:51:49.787 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:51:49.787 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:51:49.787 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:51:49.787 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:51:49.787 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.787 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:51:49.787 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.192419+0000", 2024-09-20T22:51:49.787 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:51:49.787 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.788 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.788 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:51:49.788 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.192419+0000", 2024-09-20T22:51:49.788 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:51:49.788 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.788 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.788 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:51:49.788 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.192420+0000", 2024-09-20T22:51:49.788 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.3799999999999997e-07 2024-09-20T22:51:49.788 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.788 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.788 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:51:49.788 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.204088+0000", 2024-09-20T22:51:49.788 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011668883 2024-09-20T22:51:49.789 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.789 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.789 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:51:49.789 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.204104+0000", 2024-09-20T22:51:49.789 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.5437000000000001e-05 2024-09-20T22:51:49.789 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.789 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.789 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:51:49.789 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.204112+0000", 2024-09-20T22:51:49.789 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.2519999999999999e-06 2024-09-20T22:51:49.789 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.789 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.789 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:51:49.789 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.204168+0000", 2024-09-20T22:51:49.789 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.5977999999999998e-05 2024-09-20T22:51:49.790 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.790 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.790 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:51:49.790 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.204188+0000", 2024-09-20T22:51:49.790 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.9400999999999999e-05 2024-09-20T22:51:49.790 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.790 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.790 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:51:49.790 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.206340+0000", 2024-09-20T22:51:49.790 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0021527159999999998 2024-09-20T22:51:49.790 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.790 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.790 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:51:49.791 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.206394+0000", 2024-09-20T22:51:49.791 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.3418000000000003e-05 2024-09-20T22:51:49.791 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:51:49.791 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:51:49.791 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:51:49.791 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.791 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.791 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(osd.6.0:726 7.19 e260/139 7:9eac4560:::smithi08477203-30:4c v 260'159, mlcod=259'156)", 2024-09-20T22:51:49.791 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:45:39.197025+0000", 2024-09-20T22:51:49.791 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 370.548723188, 2024-09-20T22:51:49.791 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.018150454, 2024-09-20T22:51:49.791 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:51:49.791 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:51:49.791 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:51:49.791 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:51:49.792 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.792 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:51:49.792 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.197025+0000", 2024-09-20T22:51:49.792 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:51:49.792 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.792 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.792 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:51:49.792 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.197025+0000", 2024-09-20T22:51:49.792 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:51:49.792 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.792 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.792 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:51:49.792 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.197025+0000", 2024-09-20T22:51:49.793 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.7599999999999998e-07 2024-09-20T22:51:49.793 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.793 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.793 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:51:49.793 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.197033+0000", 2024-09-20T22:51:49.793 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.9119999999999998e-06 2024-09-20T22:51:49.793 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.793 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.793 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:51:49.793 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.197042+0000", 2024-09-20T22:51:49.793 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.3349999999999996e-06 2024-09-20T22:51:49.793 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.793 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.793 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:51:49.794 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.197050+0000", 2024-09-20T22:51:49.794 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.7230000000000004e-06 2024-09-20T22:51:49.794 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.794 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.794 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:51:49.794 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.200756+0000", 2024-09-20T22:51:49.794 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0037059749999999998 2024-09-20T22:51:49.794 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.794 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.794 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:51:49.794 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.200778+0000", 2024-09-20T22:51:49.794 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.2140000000000001e-05 2024-09-20T22:51:49.794 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.794 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.795 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:51:49.795 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.215131+0000", 2024-09-20T22:51:49.795 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014353054000000001 2024-09-20T22:51:49.795 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.795 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.795 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:51:49.795 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.215175+0000", 2024-09-20T22:51:49.795 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.4039000000000003e-05 2024-09-20T22:51:49.795 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:51:49.795 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:51:49.795 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:51:49.795 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.795 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.795 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(osd.7.0:152 7.5 e269/260 7:a160b5d4:::smithi08477203-16:55 v 269'55, mlcod=263'51)", 2024-09-20T22:51:49.796 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:45:48.173893+0000", 2024-09-20T22:51:49.796 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 361.57185544700002, 2024-09-20T22:51:49.796 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014936935, 2024-09-20T22:51:49.796 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:51:49.796 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:51:49.796 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:51:49.796 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:51:49.796 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.796 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:51:49.796 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.173893+0000", 2024-09-20T22:51:49.796 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:51:49.796 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.796 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.796 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:51:49.797 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.173893+0000", 2024-09-20T22:51:49.797 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:51:49.797 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.797 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.797 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:51:49.797 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.173893+0000", 2024-09-20T22:51:49.797 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.8800000000000002e-07 2024-09-20T22:51:49.797 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.797 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.797 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:51:49.797 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.173916+0000", 2024-09-20T22:51:49.797 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.2889999999999999e-05 2024-09-20T22:51:49.797 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.797 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.797 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:51:49.798 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.173924+0000", 2024-09-20T22:51:49.798 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.4420000000000001e-06 2024-09-20T22:51:49.798 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.798 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.798 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:51:49.798 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.173932+0000", 2024-09-20T22:51:49.798 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.5299999999999996e-06 2024-09-20T22:51:49.798 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.798 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.798 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:51:49.798 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.174011+0000", 2024-09-20T22:51:49.798 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.9342e-05 2024-09-20T22:51:49.798 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.798 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.798 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:51:49.799 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.174047+0000", 2024-09-20T22:51:49.799 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.5237999999999997e-05 2024-09-20T22:51:49.799 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.799 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.799 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:51:49.799 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.188782+0000", 2024-09-20T22:51:49.799 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014735599 2024-09-20T22:51:49.799 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.799 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.799 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:51:49.799 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.188830+0000", 2024-09-20T22:51:49.799 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.7206000000000002e-05 2024-09-20T22:51:49.799 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:51:49.799 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:51:49.799 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:51:49.800 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.800 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.800 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(osd.7.0:153 7.17 e269/139 7:ea720b85:::smithi08477203-26:55 v 269'225, mlcod=263'221)", 2024-09-20T22:51:49.800 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:45:48.173991+0000", 2024-09-20T22:51:49.800 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 361.57175696600001, 2024-09-20T22:51:49.800 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013064868, 2024-09-20T22:51:49.800 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:51:49.800 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:51:49.800 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:51:49.800 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:51:49.800 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.800 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:51:49.800 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.173991+0000", 2024-09-20T22:51:49.800 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:51:49.801 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.801 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.801 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:51:49.801 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.173991+0000", 2024-09-20T22:51:49.801 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:51:49.801 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.801 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.801 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:51:49.801 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.173991+0000", 2024-09-20T22:51:49.801 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.9999999999999999e-07 2024-09-20T22:51:49.801 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.801 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.801 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:51:49.801 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.173996+0000", 2024-09-20T22:51:49.802 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.3080000000000001e-06 2024-09-20T22:51:49.802 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.802 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.802 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:51:49.802 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.174001+0000", 2024-09-20T22:51:49.802 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.203e-06 2024-09-20T22:51:49.802 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.802 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.802 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:51:49.802 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.174005+0000", 2024-09-20T22:51:49.802 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.4880000000000001e-06 2024-09-20T22:51:49.802 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.802 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.802 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:51:49.803 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.175600+0000", 2024-09-20T22:51:49.803 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015942459999999999 2024-09-20T22:51:49.803 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.803 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.803 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:51:49.803 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.175631+0000", 2024-09-20T22:51:49.803 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.1816000000000002e-05 2024-09-20T22:51:49.803 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.803 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.803 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:51:49.803 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.187003+0000", 2024-09-20T22:51:49.803 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011372006 2024-09-20T22:51:49.803 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.804 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.804 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:51:49.804 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.187056+0000", 2024-09-20T22:51:49.804 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.2500999999999998e-05 2024-09-20T22:51:49.804 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:51:49.804 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:51:49.804 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:51:49.804 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.804 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.804 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(osd.6.0:767 7.12 e269/263 7:4fa4b2b3:::smithi08477203-21:53 v 269'262, mlcod=265'259)", 2024-09-20T22:51:49.804 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:45:48.175130+0000", 2024-09-20T22:51:49.804 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 361.570617777, 2024-09-20T22:51:49.804 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013842727000000001, 2024-09-20T22:51:49.805 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:51:49.805 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:51:49.805 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:51:49.805 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:51:49.805 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.805 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:51:49.805 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.175130+0000", 2024-09-20T22:51:49.805 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:51:49.805 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.805 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.805 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:51:49.805 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.175130+0000", 2024-09-20T22:51:49.805 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:51:49.805 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.806 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.806 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:51:49.806 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.175130+0000", 2024-09-20T22:51:49.806 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.98e-07 2024-09-20T22:51:49.806 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.806 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.806 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:51:49.806 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.175139+0000", 2024-09-20T22:51:49.806 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.5909999999999996e-06 2024-09-20T22:51:49.806 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.806 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.806 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:51:49.806 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.175144+0000", 2024-09-20T22:51:49.806 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.361e-06 2024-09-20T22:51:49.807 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.807 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.807 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:51:49.807 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.175156+0000", 2024-09-20T22:51:49.807 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1236999999999999e-05 2024-09-20T22:51:49.807 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.807 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.807 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:51:49.807 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.176359+0000", 2024-09-20T22:51:49.807 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0012036060000000001 2024-09-20T22:51:49.807 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.807 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.807 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:51:49.808 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.176380+0000", 2024-09-20T22:51:49.808 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.0497999999999999e-05 2024-09-20T22:51:49.808 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.808 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.808 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:51:49.808 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.188936+0000", 2024-09-20T22:51:49.808 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.012556286999999999 2024-09-20T22:51:49.808 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.808 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.808 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:51:49.808 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.188973+0000", 2024-09-20T22:51:49.808 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.6949000000000001e-05 2024-09-20T22:51:49.808 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:51:49.808 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:51:49.808 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:51:49.808 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.809 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.809 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24652.0:2853 7.10 e303/139 7:0f524219:::smithi08477203-50:head v 303'85, mlcod=301'82)", 2024-09-20T22:51:49.809 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:46:23.293837+0000", 2024-09-20T22:51:49.809 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 326.45191106300001, 2024-09-20T22:51:49.809 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013368555000000001, 2024-09-20T22:51:49.809 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:51:49.809 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:51:49.809 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:51:49.809 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:51:49.809 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.809 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:51:49.809 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.293837+0000", 2024-09-20T22:51:49.809 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:51:49.809 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.809 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.809 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:51:49.810 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.293837+0000", 2024-09-20T22:51:49.810 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:51:49.810 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.810 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.810 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:51:49.810 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.293839+0000", 2024-09-20T22:51:49.810 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.7239999999999999e-06 2024-09-20T22:51:49.810 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.810 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.810 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:51:49.810 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.297226+0000", 2024-09-20T22:51:49.810 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.003387583 2024-09-20T22:51:49.810 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.810 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.810 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:51:49.811 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.297253+0000", 2024-09-20T22:51:49.811 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.6849e-05 2024-09-20T22:51:49.811 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.811 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.811 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:51:49.811 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.297266+0000", 2024-09-20T22:51:49.811 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2481e-05 2024-09-20T22:51:49.811 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.811 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.811 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:51:49.811 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.297362+0000", 2024-09-20T22:51:49.811 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.6347000000000004e-05 2024-09-20T22:51:49.811 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.811 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.812 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:51:49.812 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.297394+0000", 2024-09-20T22:51:49.812 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.1888999999999999e-05 2024-09-20T22:51:49.812 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.812 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.812 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:51:49.812 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.307147+0000", 2024-09-20T22:51:49.812 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0097533719999999997 2024-09-20T22:51:49.812 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.812 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.812 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:51:49.812 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.307206+0000", 2024-09-20T22:51:49.812 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.8310000000000002e-05 2024-09-20T22:51:49.812 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:51:49.812 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:51:49.812 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:51:49.813 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.813 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.813 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:3030 7.e 7:71b5c915:::smithi08477203-6:head [copy-from ver 216] snapc 72=[72,71,70,6e,6c,6b,6a,68] ondisk+write+known_if_redirected+supports_pool_eio e317)", 2024-09-20T22:51:49.813 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:46:37.293029+0000", 2024-09-20T22:51:49.813 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 312.452718705, 2024-09-20T22:51:49.813 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.019929916999999998, 2024-09-20T22:51:49.813 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:51:49.813 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:51:49.813 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:51:49.813 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:51:49.813 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:51:49.813 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:51:49.813 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 3030 2024-09-20T22:51:49.813 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.814 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:51:49.814 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.814 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:51:49.814 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.293029+0000", 2024-09-20T22:51:49.814 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:51:49.814 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.814 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.814 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:51:49.814 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.293029+0000", 2024-09-20T22:51:49.814 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:51:49.814 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.814 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.814 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:51:49.815 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.293031+0000", 2024-09-20T22:51:49.815 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.4190000000000001e-06 2024-09-20T22:51:49.815 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.815 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.815 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:51:49.815 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.293040+0000", 2024-09-20T22:51:49.815 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.6039999999999999e-06 2024-09-20T22:51:49.815 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.815 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.815 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:51:49.815 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.293054+0000", 2024-09-20T22:51:49.815 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.4014e-05 2024-09-20T22:51:49.815 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.815 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.816 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:51:49.816 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.293062+0000", 2024-09-20T22:51:49.816 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.8820000000000001e-06 2024-09-20T22:51:49.816 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.816 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.816 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:51:49.816 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.293121+0000", 2024-09-20T22:51:49.816 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.8977999999999997e-05 2024-09-20T22:51:49.816 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.816 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.816 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op must wait for map", 2024-09-20T22:51:49.816 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.293125+0000", 2024-09-20T22:51:49.816 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.2089999999999999e-06 2024-09-20T22:51:49.816 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.816 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.817 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:51:49.817 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.298805+0000", 2024-09-20T22:51:49.817 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0056790870000000002 2024-09-20T22:51:49.817 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.817 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.817 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:51:49.817 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.298886+0000", 2024-09-20T22:51:49.817 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.1363e-05 2024-09-20T22:51:49.817 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.817 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.817 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 3,5", 2024-09-20T22:51:49.817 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.305883+0000", 2024-09-20T22:51:49.817 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0069966159999999998 2024-09-20T22:51:49.817 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.817 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.818 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:51:49.818 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.312556+0000", 2024-09-20T22:51:49.818 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0066737009999999998 2024-09-20T22:51:49.818 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.818 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.818 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:51:49.818 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.312587+0000", 2024-09-20T22:51:49.818 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.0286000000000001e-05 2024-09-20T22:51:49.818 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.818 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.818 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:51:49.818 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.312890+0000", 2024-09-20T22:51:49.818 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00030374800000000001 2024-09-20T22:51:49.818 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.818 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.819 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:51:49.819 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.312932+0000", 2024-09-20T22:51:49.819 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.1621e-05 2024-09-20T22:51:49.819 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.819 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.819 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:51:49.819 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.312959+0000", 2024-09-20T22:51:49.819 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.7389000000000002e-05 2024-09-20T22:51:49.819 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:51:49.820 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:51:49.820 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:51:49.821 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:51:49.821 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.821 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.821 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:3031 7.e 7:71b5c915:::smithi08477203-6:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e317)", 2024-09-20T22:51:49.821 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:46:37.293082+0000", 2024-09-20T22:51:49.821 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 312.45266632699997, 2024-09-20T22:51:49.821 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.016134757, 2024-09-20T22:51:49.821 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:51:49.821 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:51:49.821 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-20T22:51:49.821 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:51:49.821 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:51:49.821 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:51:49.822 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 3031 2024-09-20T22:51:49.822 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.822 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:51:49.822 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.822 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:51:49.822 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.293082+0000", 2024-09-20T22:51:49.822 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:51:49.822 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.822 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.822 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:51:49.822 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.293082+0000", 2024-09-20T22:51:49.822 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:51:49.822 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.822 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.823 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:51:49.823 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.293082+0000", 2024-09-20T22:51:49.823 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.1799999999999995e-07 2024-09-20T22:51:49.823 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.823 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.823 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:51:49.823 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.293085+0000", 2024-09-20T22:51:49.823 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.937e-06 2024-09-20T22:51:49.823 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.823 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.823 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:51:49.823 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.293093+0000", 2024-09-20T22:51:49.823 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.554e-06 2024-09-20T22:51:49.823 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.823 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.824 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:51:49.824 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.293098+0000", 2024-09-20T22:51:49.824 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.643e-06 2024-09-20T22:51:49.824 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.824 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.824 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:51:49.824 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.293164+0000", 2024-09-20T22:51:49.824 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.5806e-05 2024-09-20T22:51:49.824 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.824 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.824 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting_for_map not empty", 2024-09-20T22:51:49.824 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.293168+0000", 2024-09-20T22:51:49.824 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.2459999999999999e-06 2024-09-20T22:51:49.824 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.824 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.825 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:51:49.825 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.299033+0000", 2024-09-20T22:51:49.825 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0058656699999999999 2024-09-20T22:51:49.825 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.825 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.825 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for blocked object", 2024-09-20T22:51:49.825 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.299069+0000", 2024-09-20T22:51:49.825 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.5392999999999998e-05 2024-09-20T22:51:49.825 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.825 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.825 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:51:49.825 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.309088+0000", 2024-09-20T22:51:49.825 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010019287999999999 2024-09-20T22:51:49.825 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.825 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.825 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:51:49.826 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.309150+0000", 2024-09-20T22:51:49.826 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.2008000000000007e-05 2024-09-20T22:51:49.826 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.826 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.826 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:51:49.826 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.309216+0000", 2024-09-20T22:51:49.826 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.6594000000000006e-05 2024-09-20T22:51:49.826 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:51:49.826 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:51:49.826 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:51:49.826 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.826 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.827 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:3290 7.8 7:12e9c8db:::smithi08477203-11:head [copy-from ver 675] snapc 7f=[7f,7e,7d,7c,7b,7a,79,78,74,73,6b] ondisk+write+known_if_redirected+supports_pool_eio e340)", 2024-09-20T22:51:49.827 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:47:00.706846+0000", 2024-09-20T22:51:49.827 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 289.038902468, 2024-09-20T22:51:49.827 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.018408746, 2024-09-20T22:51:49.827 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:51:49.827 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:51:49.827 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:51:49.827 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:51:49.827 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:51:49.827 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:51:49.827 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 3290 2024-09-20T22:51:49.827 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.827 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:51:49.827 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.827 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:51:49.828 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.706846+0000", 2024-09-20T22:51:49.828 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:51:49.828 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.828 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.828 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:51:49.828 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.706846+0000", 2024-09-20T22:51:49.828 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:51:49.828 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.828 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.828 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:51:49.828 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.706848+0000", 2024-09-20T22:51:49.828 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.2340000000000001e-06 2024-09-20T22:51:49.828 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.828 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.828 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:51:49.829 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.706859+0000", 2024-09-20T22:51:49.829 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1001e-05 2024-09-20T22:51:49.829 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.829 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.829 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:51:49.829 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.706879+0000", 2024-09-20T22:51:49.829 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.0556000000000001e-05 2024-09-20T22:51:49.829 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.829 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.829 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:51:49.829 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.706890+0000", 2024-09-20T22:51:49.829 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0328e-05 2024-09-20T22:51:49.829 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.829 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.829 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:51:49.830 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.706973+0000", 2024-09-20T22:51:49.830 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.3574999999999999e-05 2024-09-20T22:51:49.830 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.830 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.830 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:51:49.830 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.707042+0000", 2024-09-20T22:51:49.830 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.8987999999999997e-05 2024-09-20T22:51:49.830 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.830 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.830 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 3,7", 2024-09-20T22:51:49.830 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.717491+0000", 2024-09-20T22:51:49.830 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010448618 2024-09-20T22:51:49.830 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.830 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.830 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:51:49.831 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.722875+0000", 2024-09-20T22:51:49.831 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0053846190000000002 2024-09-20T22:51:49.831 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.831 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.831 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:51:49.831 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.724841+0000", 2024-09-20T22:51:49.831 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0019651579999999998 2024-09-20T22:51:49.831 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.831 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.831 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:51:49.831 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.725168+0000", 2024-09-20T22:51:49.831 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00032734499999999999 2024-09-20T22:51:49.831 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.831 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.831 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:51:49.832 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.725218+0000", 2024-09-20T22:51:49.832 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.0123000000000001e-05 2024-09-20T22:51:49.832 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.832 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.832 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:51:49.832 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.725254+0000", 2024-09-20T22:51:49.832 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.6201000000000001e-05 2024-09-20T22:51:49.832 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:51:49.832 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:51:49.832 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:51:49.832 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.832 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.832 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:3291 7.8 7:12e9c8db:::smithi08477203-11:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e340)", 2024-09-20T22:51:49.832 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:47:00.706927+0000", 2024-09-20T22:51:49.833 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 289.03882082899997, 2024-09-20T22:51:49.833 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014159255000000001, 2024-09-20T22:51:49.833 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:51:49.833 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:51:49.833 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-20T22:51:49.833 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:51:49.833 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:51:49.833 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:51:49.833 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 3291 2024-09-20T22:51:49.833 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.833 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:51:49.833 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.833 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:51:49.833 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.706927+0000", 2024-09-20T22:51:49.834 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:51:49.834 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.834 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.834 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:51:49.834 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.706927+0000", 2024-09-20T22:51:49.834 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:51:49.834 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.834 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.834 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:51:49.834 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.706928+0000", 2024-09-20T22:51:49.834 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.4099999999999999e-07 2024-09-20T22:51:49.834 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.834 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.834 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:51:49.835 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.706941+0000", 2024-09-20T22:51:49.835 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3335e-05 2024-09-20T22:51:49.835 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.835 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.835 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:51:49.835 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.706948+0000", 2024-09-20T22:51:49.835 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.3350000000000001e-06 2024-09-20T22:51:49.835 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.835 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.835 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:51:49.835 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.706954+0000", 2024-09-20T22:51:49.835 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.9190000000000004e-06 2024-09-20T22:51:49.835 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.835 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.836 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:51:49.836 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.707166+0000", 2024-09-20T22:51:49.836 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00021159 2024-09-20T22:51:49.836 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.836 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.836 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for blocked object", 2024-09-20T22:51:49.836 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.707217+0000", 2024-09-20T22:51:49.836 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.1167e-05 2024-09-20T22:51:49.836 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.836 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.836 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:51:49.836 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.720939+0000", 2024-09-20T22:51:49.836 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013721458000000001 2024-09-20T22:51:49.836 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.836 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.837 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:51:49.837 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.721010+0000", 2024-09-20T22:51:49.837 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.1570999999999996e-05 2024-09-20T22:51:49.837 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.837 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.837 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:51:49.837 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.721086+0000", 2024-09-20T22:51:49.837 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.6339000000000002e-05 2024-09-20T22:51:49.837 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:51:49.837 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:51:49.837 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:51:49.837 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.837 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.837 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(osd.5.0:167 7.2 e347/169 7:45b54b6b:::smithi08477203-2:81 v 347'140, mlcod=345'136)", 2024-09-20T22:51:49.837 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:47:08.204691+0000", 2024-09-20T22:51:49.837 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 281.541056728, 2024-09-20T22:51:49.838 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.01335918, 2024-09-20T22:51:49.838 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:51:49.838 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:51:49.838 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:51:49.838 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:51:49.838 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.838 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:51:49.838 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:08.204691+0000", 2024-09-20T22:51:49.838 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:51:49.838 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.838 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.838 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:51:49.838 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:08.204691+0000", 2024-09-20T22:51:49.838 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:51:49.838 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.839 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.839 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:51:49.839 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:08.204692+0000", 2024-09-20T22:51:49.839 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.5400000000000001e-07 2024-09-20T22:51:49.839 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.839 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.839 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:51:49.839 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:08.204697+0000", 2024-09-20T22:51:49.839 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.614e-06 2024-09-20T22:51:49.839 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.839 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.839 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:51:49.840 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:08.204705+0000", 2024-09-20T22:51:49.840 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.8259999999999993e-06 2024-09-20T22:51:49.840 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.840 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.840 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:51:49.840 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:08.204715+0000", 2024-09-20T22:51:49.840 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.8279999999999998e-06 2024-09-20T22:51:49.840 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.840 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.840 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:51:49.840 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:08.204801+0000", 2024-09-20T22:51:49.840 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.6263999999999999e-05 2024-09-20T22:51:49.840 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.840 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.841 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:51:49.841 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:08.204823+0000", 2024-09-20T22:51:49.841 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.1971999999999999e-05 2024-09-20T22:51:49.841 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.841 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.841 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:51:49.841 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:08.217961+0000", 2024-09-20T22:51:49.841 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013137618 2024-09-20T22:51:49.841 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.841 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.841 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:51:49.841 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:08.218050+0000", 2024-09-20T22:51:49.841 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.9504000000000002e-05 2024-09-20T22:51:49.841 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:51:49.842 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:51:49.842 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:51:49.842 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.842 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.842 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24652.0:3760 7.13 e383/260 7:cae5792a:::smithi08477203-32:head v 383'78, mlcod=377'74)", 2024-09-20T22:51:49.842 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:47:43.291645+0000", 2024-09-20T22:51:49.842 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 246.45410285599999, 2024-09-20T22:51:49.842 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.012696648, 2024-09-20T22:51:49.842 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:51:49.842 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:51:49.842 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:51:49.842 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:51:49.842 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.842 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:51:49.843 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:43.291645+0000", 2024-09-20T22:51:49.843 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:51:49.843 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.843 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.843 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:51:49.843 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:43.291645+0000", 2024-09-20T22:51:49.843 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:51:49.843 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.843 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.843 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:51:49.843 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:43.291645+0000", 2024-09-20T22:51:49.843 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.0800000000000001e-07 2024-09-20T22:51:49.843 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.843 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.844 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:51:49.844 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:43.294764+0000", 2024-09-20T22:51:49.844 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0031183420000000001 2024-09-20T22:51:49.844 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.844 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.844 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:51:49.844 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:43.294777+0000", 2024-09-20T22:51:49.844 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3465999999999999e-05 2024-09-20T22:51:49.844 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.844 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.844 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:51:49.844 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:43.294788+0000", 2024-09-20T22:51:49.844 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0662999999999999e-05 2024-09-20T22:51:49.845 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.845 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.845 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:51:49.845 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:43.294952+0000", 2024-09-20T22:51:49.845 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00016369799999999999 2024-09-20T22:51:49.845 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.845 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.845 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:51:49.845 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:43.294978+0000", 2024-09-20T22:51:49.845 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.6414000000000002e-05 2024-09-20T22:51:49.845 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.845 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.846 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:51:49.846 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:43.304313+0000", 2024-09-20T22:51:49.846 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0093352039999999997 2024-09-20T22:51:49.846 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.846 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.846 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:51:49.846 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:43.304342+0000", 2024-09-20T22:51:49.846 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.8552999999999999e-05 2024-09-20T22:51:49.846 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:51:49.846 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:51:49.846 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:51:49.846 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.846 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.846 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24652.0:4534 7.12 e456/421 7:4b06cbed:::smithi08477203-49:head v 456'446, mlcod=454'443)", 2024-09-20T22:51:49.846 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:48:56.245037+0000", 2024-09-20T22:51:49.847 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 173.50071083500001, 2024-09-20T22:51:49.847 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013221933999999999, 2024-09-20T22:51:49.847 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:51:49.847 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:51:49.847 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:51:49.847 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:51:49.847 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.847 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:51:49.847 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:56.245037+0000", 2024-09-20T22:51:49.847 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:51:49.847 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.847 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.847 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:51:49.847 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:56.245037+0000", 2024-09-20T22:51:49.848 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:51:49.848 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.848 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.848 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:51:49.848 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:56.245037+0000", 2024-09-20T22:51:49.848 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.4600000000000001e-07 2024-09-20T22:51:49.848 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.848 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.848 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:51:49.848 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:56.248142+0000", 2024-09-20T22:51:49.848 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0031044839999999998 2024-09-20T22:51:49.848 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.848 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.848 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:51:49.849 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:56.248174+0000", 2024-09-20T22:51:49.849 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.2555000000000003e-05 2024-09-20T22:51:49.849 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.849 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.849 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:51:49.849 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:56.248188+0000", 2024-09-20T22:51:49.849 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.308e-05 2024-09-20T22:51:49.849 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.849 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.849 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:51:49.849 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:56.248292+0000", 2024-09-20T22:51:49.849 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000104182 2024-09-20T22:51:49.849 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.849 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.850 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:51:49.850 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:56.248318+0000", 2024-09-20T22:51:49.850 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.652e-05 2024-09-20T22:51:49.850 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.850 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.850 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:51:49.850 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:56.258207+0000", 2024-09-20T22:51:49.850 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0098883909999999998 2024-09-20T22:51:49.850 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:51:49.850 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:51:49.850 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:51:49.850 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:56.258259+0000", 2024-09-20T22:51:49.850 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.2475999999999997e-05 2024-09-20T22:51:49.850 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:51:49.850 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:51:49.851 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:51:49.851 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:51:49.851 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:51:49.851 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-20T22:51:49.871 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:49.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:49.872 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:51:49.973 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:51:49.999 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:50.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:50.000 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:51:50.101 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:51:50.150 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:50.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:50.151 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:51:50.252 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:51:50.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:50.281 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:50.281 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:51:50.327 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is unset 2024-09-20T22:51:50.332 INFO:tasks.rados.rados.0.smithi084.stdout:2473: done (3 left) 2024-09-20T22:51:50.332 INFO:tasks.rados.rados.0.smithi084.stdout:2474: snap_remove snap 265 2024-09-20T22:51:50.381 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:51:50.406 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:50.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:50.407 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:51:50.508 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:51:50.529 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:50.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:50.530 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:51:50.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:50 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/144870320' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:51:50.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:50 smithi084 ceph-mon[29774]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:51:50.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:50 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/144870320' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:51:50.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:50 smithi084 ceph-mon[40024]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:51:50.631 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:51:50.652 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:50.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:50.653 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:51:50.729 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:51:50.729 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:51:50.744 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:51:50.744 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:51:50.754 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:51:50.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:50 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/144870320' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:51:50.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:50 smithi154 ceph-mon[33264]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:51:50.760 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:51:50.760 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:51:50.777 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-20T22:51:50.777 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T22:51:50.779 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:50.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:50.780 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:51:50.794 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:51:50.794 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:51:50.811 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:51:50.811 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:51:50.828 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:51:50.828 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:51:50.845 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:51:50.845 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:51:50.881 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:51:50.902 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:50.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:50.903 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:51:51.004 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:51:51.025 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:51.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:51.026 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:51:51.126 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:51:51.153 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:51.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:51.154 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:51:51.255 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:51:51.276 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:51.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:51.277 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:51:51.346 INFO:tasks.rados.rados.0.smithi084.stdout:2474: done (3 left) 2024-09-20T22:51:51.346 INFO:tasks.rados.rados.0.smithi084.stdout:2475: delete oid 27 current snap is 275 2024-09-20T22:51:51.346 INFO:tasks.rados.rados.0.smithi084.stdout:2470: finishing write tid 2 to smithi08477203-38 2024-09-20T22:51:51.346 INFO:tasks.rados.rados.0.smithi084.stdout:2470: finishing write tid 3 to smithi08477203-38 2024-09-20T22:51:51.346 INFO:tasks.rados.rados.0.smithi084.stdout:2470: finishing write tid 4 to smithi08477203-38 2024-09-20T22:51:51.346 INFO:tasks.rados.rados.0.smithi084.stdout:2470: oid 38 updating version 0 to 1325 2024-09-20T22:51:51.346 INFO:tasks.rados.rados.0.smithi084.stdout:2470: oid 38 updating version 1325 to 1326 2024-09-20T22:51:51.346 INFO:tasks.rados.rados.0.smithi084.stdout:2470: oid 38 version 1326 is already newer than 1324 2024-09-20T22:51:51.346 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 38 v 1326 (ObjNum 852 snap 275 seq_num 852) dirty exists 2024-09-20T22:51:51.346 INFO:tasks.rados.rados.0.smithi084.stdout:2470: left oid 38 (ObjNum 852 snap 275 seq_num 852) 2024-09-20T22:51:51.347 INFO:tasks.rados.rados.0.smithi084.stdout:2471: finishing copy_from racing read to smithi08477203-8 2024-09-20T22:51:51.347 INFO:tasks.rados.rados.0.smithi084.stdout:2472: finishing write tid 1 to smithi08477203-48 2024-09-20T22:51:51.347 INFO:tasks.rados.rados.0.smithi084.stdout:2472: finishing write tid 2 to smithi08477203-48 2024-09-20T22:51:51.347 INFO:tasks.rados.rados.0.smithi084.stdout:2472: finishing write tid 3 to smithi08477203-48 2024-09-20T22:51:51.347 INFO:tasks.rados.rados.0.smithi084.stdout:2472: finishing write tid 4 to smithi08477203-48 2024-09-20T22:51:51.347 INFO:tasks.rados.rados.0.smithi084.stdout:2472: oid 48 updating version 0 to 868 2024-09-20T22:51:51.347 INFO:tasks.rados.rados.0.smithi084.stdout:2472: oid 48 updating version 868 to 869 2024-09-20T22:51:51.347 INFO:tasks.rados.rados.0.smithi084.stdout:2472: oid 48 version 869 is already newer than 867 2024-09-20T22:51:51.347 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 48 v 869 (ObjNum 853 snap 275 seq_num 853) dirty exists 2024-09-20T22:51:51.347 INFO:tasks.rados.rados.0.smithi084.stdout:2472: left oid 48 (ObjNum 853 snap 275 seq_num 853) 2024-09-20T22:51:51.347 INFO:tasks.rados.rados.0.smithi084.stdout:2471: finishing copy_from to smithi08477203-8 2024-09-20T22:51:51.347 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 8 v 1797 (ObjNum 832 snap 268 seq_num 832) dirty exists 2024-09-20T22:51:51.349 INFO:tasks.rados.rados.0.smithi084.stdout:2470: done (3 left) 2024-09-20T22:51:51.349 INFO:tasks.rados.rados.0.smithi084.stdout:2471: done (2 left) 2024-09-20T22:51:51.349 INFO:tasks.rados.rados.0.smithi084.stdout:2472: done (1 left) 2024-09-20T22:51:51.349 INFO:tasks.rados.rados.0.smithi084.stdout:2475: done (0 left) 2024-09-20T22:51:51.349 INFO:tasks.rados.rados.0.smithi084.stdout:2476: read oid 3 snap -1 2024-09-20T22:51:51.349 INFO:tasks.rados.rados.0.smithi084.stdout:2476: expect deleted 2024-09-20T22:51:51.349 INFO:tasks.rados.rados.0.smithi084.stdout:2477: delete oid 14 current snap is 275 2024-09-20T22:51:51.351 INFO:tasks.rados.rados.0.smithi084.stdout:2476: done (1 left) 2024-09-20T22:51:51.351 INFO:tasks.rados.rados.0.smithi084.stdout:2477: done (0 left) 2024-09-20T22:51:51.351 INFO:tasks.rados.rados.0.smithi084.stdout:2478: write (excl) oid 27 current snap is 275 2024-09-20T22:51:51.351 INFO:tasks.rados.rados.0.smithi084.stdout:2478: seq_num 856 ranges {786792=644680,1857998=528794} 2024-09-20T22:51:51.359 INFO:tasks.rados.rados.0.smithi084.stdout:2478: writing smithi08477203-27 from 786792 to 1431472 tid 1 2024-09-20T22:51:51.365 INFO:tasks.rados.rados.0.smithi084.stdout:2478: writing smithi08477203-27 from 1857998 to 2386792 tid 2 2024-09-20T22:51:51.365 INFO:tasks.rados.rados.0.smithi084.stdout:2479: read oid 32 snap -1 2024-09-20T22:51:51.365 INFO:tasks.rados.rados.0.smithi084.stdout:2479: expect (ObjNum 769 snap 248 seq_num 769) 2024-09-20T22:51:51.365 INFO:tasks.rados.rados.0.smithi084.stdout:2480: write (excl) oid 23 current snap is 275 2024-09-20T22:51:51.365 INFO:tasks.rados.rados.0.smithi084.stdout:2478: finishing write tid 1 to smithi08477203-27 2024-09-20T22:51:51.365 INFO:tasks.rados.rados.0.smithi084.stdout:2480: seq_num 857 ranges {403599=719929,1916480=441635} 2024-09-20T22:51:51.371 INFO:tasks.rados.rados.0.smithi084.stdout:2480: writing smithi08477203-23 from 403599 to 1123528 tid 1 2024-09-20T22:51:51.378 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:51:51.379 INFO:tasks.rados.rados.0.smithi084.stdout:2480: writing smithi08477203-23 from 1916480 to 2358115 tid 2 2024-09-20T22:51:51.379 INFO:tasks.rados.rados.0.smithi084.stdout:2481: write oid 26 current snap is 275 2024-09-20T22:51:51.379 INFO:tasks.rados.rados.0.smithi084.stdout:2481: seq_num 858 ranges {420406=795178,1974962=445444} 2024-09-20T22:51:51.385 INFO:tasks.rados.rados.0.smithi084.stdout:2481: writing smithi08477203-26 from 420406 to 1215584 tid 1 2024-09-20T22:51:51.392 INFO:tasks.rados.rados.0.smithi084.stdout:2481: writing smithi08477203-26 from 1974962 to 2420406 tid 2 2024-09-20T22:51:51.392 INFO:tasks.rados.rados.0.smithi084.stdout:2482: write oid 41 current snap is 275 2024-09-20T22:51:51.392 INFO:tasks.rados.rados.0.smithi084.stdout:2482: seq_num 859 ranges {437213=470427,1633444=728951} 2024-09-20T22:51:51.397 INFO:tasks.rados.rados.0.smithi084.stdout:2482: writing smithi08477203-41 from 437213 to 907640 tid 1 2024-09-20T22:51:51.402 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:51.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:51.403 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:51:51.405 INFO:tasks.rados.rados.0.smithi084.stdout:2482: writing smithi08477203-41 from 1633444 to 2362395 tid 2 2024-09-20T22:51:51.406 INFO:tasks.rados.rados.0.smithi084.stdout:2483: rollback oid 46 current snap is 275 2024-09-20T22:51:51.406 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 46 to 189 2024-09-20T22:51:51.406 INFO:tasks.rados.rados.0.smithi084.stdout:2484: snap_create 2024-09-20T22:51:51.406 INFO:tasks.rados.rados.0.smithi084.stdout:2479: expect (ObjNum 769 snap 248 seq_num 769) 2024-09-20T22:51:51.504 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:51:51.525 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:51.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:51.526 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:51:51.610 INFO:tasks.rados.rados.0.smithi084.stdout:2478: finishing write tid 2 to smithi08477203-27 2024-09-20T22:51:51.610 INFO:tasks.rados.rados.0.smithi084.stdout:2478: finishing write tid 3 to smithi08477203-27 2024-09-20T22:51:51.610 INFO:tasks.rados.rados.0.smithi084.stdout:2478: finishing write tid 4 to smithi08477203-27 2024-09-20T22:51:51.610 INFO:tasks.rados.rados.0.smithi084.stdout:2478: oid 27 updating version 0 to 741 2024-09-20T22:51:51.610 INFO:tasks.rados.rados.0.smithi084.stdout:2478: oid 27 version 741 is already newer than 739 2024-09-20T22:51:51.610 INFO:tasks.rados.rados.0.smithi084.stdout:2478: oid 27 version 741 is already newer than 740 2024-09-20T22:51:51.610 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 27 v 741 (ObjNum 856 snap 275 seq_num 856) dirty exists 2024-09-20T22:51:51.610 INFO:tasks.rados.rados.0.smithi084.stdout:2478: left oid 27 (ObjNum 856 snap 275 seq_num 856) 2024-09-20T22:51:51.610 INFO:tasks.rados.rados.0.smithi084.stdout:2480: finishing write tid 1 to smithi08477203-23 2024-09-20T22:51:51.611 INFO:tasks.rados.rados.0.smithi084.stdout:2480: finishing write tid 2 to smithi08477203-23 2024-09-20T22:51:51.611 INFO:tasks.rados.rados.0.smithi084.stdout:2480: finishing write tid 3 to smithi08477203-23 2024-09-20T22:51:51.611 INFO:tasks.rados.rados.0.smithi084.stdout:2480: finishing write tid 4 to smithi08477203-23 2024-09-20T22:51:51.611 INFO:tasks.rados.rados.0.smithi084.stdout:2480: oid 23 updating version 0 to 1231 2024-09-20T22:51:51.611 INFO:tasks.rados.rados.0.smithi084.stdout:2480: oid 23 version 1231 is already newer than 1229 2024-09-20T22:51:51.611 INFO:tasks.rados.rados.0.smithi084.stdout:2480: oid 23 version 1231 is already newer than 1230 2024-09-20T22:51:51.611 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 23 v 1231 (ObjNum 857 snap 275 seq_num 857) dirty exists 2024-09-20T22:51:51.611 INFO:tasks.rados.rados.0.smithi084.stdout:2480: left oid 23 (ObjNum 857 snap 275 seq_num 857) 2024-09-20T22:51:51.611 INFO:tasks.rados.rados.0.smithi084.stdout:2481: finishing write tid 1 to smithi08477203-26 2024-09-20T22:51:51.611 INFO:tasks.rados.rados.0.smithi084.stdout:2481: finishing write tid 2 to smithi08477203-26 2024-09-20T22:51:51.611 INFO:tasks.rados.rados.0.smithi084.stdout:2481: finishing write tid 3 to smithi08477203-26 2024-09-20T22:51:51.611 INFO:tasks.rados.rados.0.smithi084.stdout:2481: finishing write tid 4 to smithi08477203-26 2024-09-20T22:51:51.611 INFO:tasks.rados.rados.0.smithi084.stdout:2481: oid 26 updating version 0 to 1327 2024-09-20T22:51:51.611 INFO:tasks.rados.rados.0.smithi084.stdout:2481: oid 26 updating version 1327 to 1329 2024-09-20T22:51:51.611 INFO:tasks.rados.rados.0.smithi084.stdout:2481: oid 26 version 1329 is already newer than 1328 2024-09-20T22:51:51.611 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 26 v 1329 (ObjNum 858 snap 275 seq_num 858) dirty exists 2024-09-20T22:51:51.612 INFO:tasks.rados.rados.0.smithi084.stdout:2481: left oid 26 (ObjNum 858 snap 275 seq_num 858) 2024-09-20T22:51:51.612 INFO:tasks.rados.rados.0.smithi084.stdout:2482: finishing write tid 1 to smithi08477203-41 2024-09-20T22:51:51.612 INFO:tasks.rados.rados.0.smithi084.stdout:2482: finishing write tid 2 to smithi08477203-41 2024-09-20T22:51:51.612 INFO:tasks.rados.rados.0.smithi084.stdout:2482: finishing write tid 3 to smithi08477203-41 2024-09-20T22:51:51.612 INFO:tasks.rados.rados.0.smithi084.stdout:2482: finishing write tid 4 to smithi08477203-41 2024-09-20T22:51:51.612 INFO:tasks.rados.rados.0.smithi084.stdout:2482: oid 41 updating version 0 to 1400 2024-09-20T22:51:51.612 INFO:tasks.rados.rados.0.smithi084.stdout:2482: oid 41 updating version 1400 to 1401 2024-09-20T22:51:51.612 INFO:tasks.rados.rados.0.smithi084.stdout:2482: oid 41 updating version 1401 to 1402 2024-09-20T22:51:51.612 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 41 v 1402 (ObjNum 859 snap 275 seq_num 859) dirty exists 2024-09-20T22:51:51.612 INFO:tasks.rados.rados.0.smithi084.stdout:2482: left oid 41 (ObjNum 859 snap 275 seq_num 859) 2024-09-20T22:51:51.612 INFO:tasks.rados.rados.0.smithi084.stdout:2483: finishing rollback tid 0 to smithi08477203-46 2024-09-20T22:51:51.612 INFO:tasks.rados.rados.0.smithi084.stdout:2483: finishing rollback tid 1 to smithi08477203-46 2024-09-20T22:51:51.612 INFO:tasks.rados.rados.0.smithi084.stdout:2483: finishing rollback tid 2 to smithi08477203-46 2024-09-20T22:51:51.612 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 46 v 1800 (ObjNum 470 snap 130 seq_num 470) dirty exists 2024-09-20T22:51:51.614 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:51 smithi084 ceph-mon[40024]: pgmap v846: 158 pgs: 3 active+clean+snaptrim, 2 active+remapped+backfill_toofull, 153 active+clean; 484 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 110 KiB/s rd, 1.3 MiB/s wr, 189 op/s; 37/1290 objects misplaced (2.868%) 2024-09-20T22:51:51.614 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:51 smithi084 ceph-mon[40024]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:51:51.614 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:51 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/144870320' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T22:51:51.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:51 smithi084 ceph-mon[40024]: osdmap e630: 8 total, 8 up, 7 in 2024-09-20T22:51:51.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:51 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/1670545875' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:51:51.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:51 smithi084 ceph-mon[40024]: 5.e scrub starts 2024-09-20T22:51:51.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:51 smithi084 ceph-mon[40024]: 5.e scrub ok 2024-09-20T22:51:51.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:51 smithi084 ceph-mon[40024]: 6.1 scrub starts 2024-09-20T22:51:51.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:51 smithi084 ceph-mon[40024]: 6.1 scrub ok 2024-09-20T22:51:51.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:51 smithi084 ceph-mon[29774]: pgmap v846: 158 pgs: 3 active+clean+snaptrim, 2 active+remapped+backfill_toofull, 153 active+clean; 484 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 110 KiB/s rd, 1.3 MiB/s wr, 189 op/s; 37/1290 objects misplaced (2.868%) 2024-09-20T22:51:51.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:51 smithi084 ceph-mon[29774]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:51:51.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:51 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/144870320' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T22:51:51.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:51 smithi084 ceph-mon[29774]: osdmap e630: 8 total, 8 up, 7 in 2024-09-20T22:51:51.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:51 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/1670545875' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:51:51.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:51 smithi084 ceph-mon[29774]: 5.e scrub starts 2024-09-20T22:51:51.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:51 smithi084 ceph-mon[29774]: 5.e scrub ok 2024-09-20T22:51:51.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:51 smithi084 ceph-mon[29774]: 6.1 scrub starts 2024-09-20T22:51:51.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:51 smithi084 ceph-mon[29774]: 6.1 scrub ok 2024-09-20T22:51:51.627 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:51:51.648 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:51.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:51.649 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:51:51.737 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:51 smithi154 ceph-mon[33264]: pgmap v846: 158 pgs: 3 active+clean+snaptrim, 2 active+remapped+backfill_toofull, 153 active+clean; 484 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 110 KiB/s rd, 1.3 MiB/s wr, 189 op/s; 37/1290 objects misplaced (2.868%) 2024-09-20T22:51:51.738 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:51 smithi154 ceph-mon[33264]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:51:51.738 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:51 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/144870320' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T22:51:51.738 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:51 smithi154 ceph-mon[33264]: osdmap e630: 8 total, 8 up, 7 in 2024-09-20T22:51:51.738 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:51 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/1670545875' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:51:51.738 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:51 smithi154 ceph-mon[33264]: 5.e scrub starts 2024-09-20T22:51:51.738 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:51 smithi154 ceph-mon[33264]: 5.e scrub ok 2024-09-20T22:51:51.738 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:51 smithi154 ceph-mon[33264]: 6.1 scrub starts 2024-09-20T22:51:51.738 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:51 smithi154 ceph-mon[33264]: 6.1 scrub ok 2024-09-20T22:51:51.750 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:51:51.771 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:51.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:51.772 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:51:51.873 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:51:51.894 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:51.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:51.895 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:51:51.996 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:51:52.018 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:52.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:52.019 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:51:52.120 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:51:52.142 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:52.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:52.143 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:51:52.244 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:51:52.267 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:52.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:52.268 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:51:52.349 INFO:tasks.rados.rados.0.smithi084.stdout:2478: done (6 left) 2024-09-20T22:51:52.350 INFO:tasks.rados.rados.0.smithi084.stdout:2479: done (5 left) 2024-09-20T22:51:52.350 INFO:tasks.rados.rados.0.smithi084.stdout:2480: done (4 left) 2024-09-20T22:51:52.350 INFO:tasks.rados.rados.0.smithi084.stdout:2481: done (3 left) 2024-09-20T22:51:52.350 INFO:tasks.rados.rados.0.smithi084.stdout:2482: done (2 left) 2024-09-20T22:51:52.350 INFO:tasks.rados.rados.0.smithi084.stdout:2483: done (1 left) 2024-09-20T22:51:52.350 INFO:tasks.rados.rados.0.smithi084.stdout:2484: done (0 left) 2024-09-20T22:51:52.350 INFO:tasks.rados.rados.0.smithi084.stdout:2485: read oid 29 snap -1 2024-09-20T22:51:52.350 INFO:tasks.rados.rados.0.smithi084.stdout:2485: expect (ObjNum 440 snap 115 seq_num 440) 2024-09-20T22:51:52.350 INFO:tasks.rados.rados.0.smithi084.stdout:2486: copy_from oid 30 from oid 44 current snap is 276 2024-09-20T22:51:52.350 INFO:tasks.rados.rados.0.smithi084.stdout:2487: write (excl) oid 10 current snap is 276 2024-09-20T22:51:52.350 INFO:tasks.rados.rados.0.smithi084.stdout:2487: seq_num 860 ranges {454020=662029,1691926=588962} 2024-09-20T22:51:52.355 INFO:tasks.rados.rados.0.smithi084.stdout:2487: writing smithi08477203-10 from 454020 to 1116049 tid 1 2024-09-20T22:51:52.362 INFO:tasks.rados.rados.0.smithi084.stdout:2487: writing smithi08477203-10 from 1691926 to 2280888 tid 2 2024-09-20T22:51:52.362 INFO:tasks.rados.rados.0.smithi084.stdout:2488: write (excl) oid 5 current snap is 276 2024-09-20T22:51:52.363 INFO:tasks.rados.rados.0.smithi084.stdout:2486: finishing copy_from to smithi08477203-30 2024-09-20T22:51:52.363 INFO:tasks.rados.rados.0.smithi084.stdout:2486: got expected ENOENT (src dne) 2024-09-20T22:51:52.363 INFO:tasks.rados.rados.0.smithi084.stdout:2486: finishing copy_from racing read to smithi08477203-30 2024-09-20T22:51:52.363 INFO:tasks.rados.rados.0.smithi084.stdout:2487: finishing write tid 1 to smithi08477203-10 2024-09-20T22:51:52.363 INFO:tasks.rados.rados.0.smithi084.stdout:2488: seq_num 861 ranges {470827=737278,1750408=720419} 2024-09-20T22:51:52.368 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:51:52.369 INFO:tasks.rados.rados.0.smithi084.stdout:2488: writing smithi08477203-5 from 470827 to 1208105 tid 1 2024-09-20T22:51:52.376 INFO:tasks.rados.rados.0.smithi084.stdout:2488: writing smithi08477203-5 from 1750408 to 2470827 tid 2 2024-09-20T22:51:52.376 INFO:tasks.rados.rados.0.smithi084.stdout:2486: done (3 left) 2024-09-20T22:51:52.376 INFO:tasks.rados.rados.0.smithi084.stdout:2489: write (excl) oid 33 current snap is 276 2024-09-20T22:51:52.377 INFO:tasks.rados.rados.0.smithi084.stdout:2489: seq_num 862 ranges {487634=412527,1408890=592631} 2024-09-20T22:51:52.381 INFO:tasks.rados.rados.0.smithi084.stdout:2489: writing smithi08477203-33 from 487634 to 900161 tid 1 2024-09-20T22:51:52.388 INFO:tasks.rados.rados.0.smithi084.stdout:2489: writing smithi08477203-33 from 1408890 to 2001521 tid 2 2024-09-20T22:51:52.388 INFO:tasks.rados.rados.0.smithi084.stdout:2490: read oid 28 snap 213 2024-09-20T22:51:52.388 INFO:tasks.rados.rados.0.smithi084.stdout:2490: expect (ObjNum 656 snap 198 seq_num 656) 2024-09-20T22:51:52.388 INFO:tasks.rados.rados.0.smithi084.stdout:2491: write oid 8 current snap is 276 2024-09-20T22:51:52.388 INFO:tasks.rados.rados.0.smithi084.stdout:2487: finishing write tid 2 to smithi08477203-10 2024-09-20T22:51:52.388 INFO:tasks.rados.rados.0.smithi084.stdout:2487: finishing write tid 3 to smithi08477203-10 2024-09-20T22:51:52.388 INFO:tasks.rados.rados.0.smithi084.stdout:2485: expect (ObjNum 440 snap 115 seq_num 440) 2024-09-20T22:51:52.391 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:52.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:52.392 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:51:52.493 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:51:52.514 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:52.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:52.515 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:51:52.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:52 smithi084 ceph-mon[40024]: 5.4 scrub starts 2024-09-20T22:51:52.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:52 smithi084 ceph-mon[40024]: 5.4 scrub ok 2024-09-20T22:51:52.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:52 smithi084 ceph-mon[40024]: 7.f scrub starts 2024-09-20T22:51:52.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:52 smithi084 ceph-mon[40024]: 7.f scrub ok 2024-09-20T22:51:52.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:52 smithi084 ceph-mon[40024]: 5.f scrub starts 2024-09-20T22:51:52.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:52 smithi084 ceph-mon[40024]: 5.f scrub ok 2024-09-20T22:51:52.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:52 smithi084 ceph-mon[40024]: osdmap e631: 8 total, 8 up, 7 in 2024-09-20T22:51:52.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:52 smithi084 ceph-mon[40024]: 4.18 scrub starts 2024-09-20T22:51:52.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:52 smithi084 ceph-mon[40024]: 4.18 scrub ok 2024-09-20T22:51:52.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:52 smithi084 ceph-mon[40024]: 7.d scrub starts 2024-09-20T22:51:52.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:52 smithi084 ceph-mon[40024]: 7.d scrub ok 2024-09-20T22:51:52.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:52 smithi084 ceph-mon[29774]: 5.4 scrub starts 2024-09-20T22:51:52.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:52 smithi084 ceph-mon[29774]: 5.4 scrub ok 2024-09-20T22:51:52.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:52 smithi084 ceph-mon[29774]: 7.f scrub starts 2024-09-20T22:51:52.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:52 smithi084 ceph-mon[29774]: 7.f scrub ok 2024-09-20T22:51:52.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:52 smithi084 ceph-mon[29774]: 5.f scrub starts 2024-09-20T22:51:52.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:52 smithi084 ceph-mon[29774]: 5.f scrub ok 2024-09-20T22:51:52.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:52 smithi084 ceph-mon[29774]: osdmap e631: 8 total, 8 up, 7 in 2024-09-20T22:51:52.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:52 smithi084 ceph-mon[29774]: 4.18 scrub starts 2024-09-20T22:51:52.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:52 smithi084 ceph-mon[29774]: 4.18 scrub ok 2024-09-20T22:51:52.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:52 smithi084 ceph-mon[29774]: 7.d scrub starts 2024-09-20T22:51:52.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:52 smithi084 ceph-mon[29774]: 7.d scrub ok 2024-09-20T22:51:52.617 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:51:52.636 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:52.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:52.637 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:51:52.654 INFO:tasks.rados.rados.0.smithi084.stdout:2487: finishing write tid 4 to smithi08477203-10 2024-09-20T22:51:52.654 INFO:tasks.rados.rados.0.smithi084.stdout:2487: oid 10 updating version 0 to 1238 2024-09-20T22:51:52.654 INFO:tasks.rados.rados.0.smithi084.stdout:2487: oid 10 updating version 1238 to 1239 2024-09-20T22:51:52.654 INFO:tasks.rados.rados.0.smithi084.stdout:2487: oid 10 updating version 1239 to 1240 2024-09-20T22:51:52.654 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 10 v 1240 (ObjNum 860 snap 276 seq_num 860) dirty exists 2024-09-20T22:51:52.654 INFO:tasks.rados.rados.0.smithi084.stdout:2487: left oid 10 (ObjNum 860 snap 276 seq_num 860) 2024-09-20T22:51:52.654 INFO:tasks.rados.rados.0.smithi084.stdout:2488: finishing write tid 1 to smithi08477203-5 2024-09-20T22:51:52.654 INFO:tasks.rados.rados.0.smithi084.stdout:2491: seq_num 863 ranges {504441=487776,1467372=736289} 2024-09-20T22:51:52.659 INFO:tasks.rados.rados.0.smithi084.stdout:2491: writing smithi08477203-8 from 504441 to 992217 tid 1 2024-09-20T22:51:52.666 INFO:tasks.rados.rados.0.smithi084.stdout:2491: writing smithi08477203-8 from 1467372 to 2203661 tid 2 2024-09-20T22:51:52.666 INFO:tasks.rados.rados.0.smithi084.stdout:2485: done (5 left) 2024-09-20T22:51:52.666 INFO:tasks.rados.rados.0.smithi084.stdout:2487: done (4 left) 2024-09-20T22:51:52.666 INFO:tasks.rados.rados.0.smithi084.stdout:2492: snap_create 2024-09-20T22:51:52.666 INFO:tasks.rados.rados.0.smithi084.stdout:2488: finishing write tid 2 to smithi08477203-5 2024-09-20T22:51:52.666 INFO:tasks.rados.rados.0.smithi084.stdout:2488: finishing write tid 3 to smithi08477203-5 2024-09-20T22:51:52.666 INFO:tasks.rados.rados.0.smithi084.stdout:2488: finishing write tid 4 to smithi08477203-5 2024-09-20T22:51:52.667 INFO:tasks.rados.rados.0.smithi084.stdout:2488: oid 5 updating version 0 to 760 2024-09-20T22:51:52.667 INFO:tasks.rados.rados.0.smithi084.stdout:2488: oid 5 version 760 is already newer than 758 2024-09-20T22:51:52.667 INFO:tasks.rados.rados.0.smithi084.stdout:2488: oid 5 version 760 is already newer than 759 2024-09-20T22:51:52.667 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 5 v 760 (ObjNum 861 snap 276 seq_num 861) dirty exists 2024-09-20T22:51:52.667 INFO:tasks.rados.rados.0.smithi084.stdout:2488: left oid 5 (ObjNum 861 snap 276 seq_num 861) 2024-09-20T22:51:52.667 INFO:tasks.rados.rados.0.smithi084.stdout:2489: finishing write tid 1 to smithi08477203-33 2024-09-20T22:51:52.667 INFO:tasks.rados.rados.0.smithi084.stdout:2489: finishing write tid 2 to smithi08477203-33 2024-09-20T22:51:52.667 INFO:tasks.rados.rados.0.smithi084.stdout:2489: finishing write tid 3 to smithi08477203-33 2024-09-20T22:51:52.667 INFO:tasks.rados.rados.0.smithi084.stdout:2489: finishing write tid 4 to smithi08477203-33 2024-09-20T22:51:52.667 INFO:tasks.rados.rados.0.smithi084.stdout:2489: oid 33 updating version 0 to 1233 2024-09-20T22:51:52.667 INFO:tasks.rados.rados.0.smithi084.stdout:2489: oid 33 updating version 1233 to 1234 2024-09-20T22:51:52.667 INFO:tasks.rados.rados.0.smithi084.stdout:2489: oid 33 version 1234 is already newer than 1232 2024-09-20T22:51:52.667 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 33 v 1234 (ObjNum 862 snap 276 seq_num 862) dirty exists 2024-09-20T22:51:52.667 INFO:tasks.rados.rados.0.smithi084.stdout:2489: left oid 33 (ObjNum 862 snap 276 seq_num 862) 2024-09-20T22:51:52.668 INFO:tasks.rados.rados.0.smithi084.stdout:2490: expect (ObjNum 656 snap 198 seq_num 656) 2024-09-20T22:51:52.689 INFO:tasks.thrashosds.thrasher:in_osds: [2, 3, 5, 6, 7, 0, 4] out_osds: [1] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-20T22:51:52.689 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd reweight 2 1 2024-09-20T22:51:52.738 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:51:52.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:52 smithi154 ceph-mon[33264]: 5.4 scrub starts 2024-09-20T22:51:52.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:52 smithi154 ceph-mon[33264]: 5.4 scrub ok 2024-09-20T22:51:52.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:52 smithi154 ceph-mon[33264]: 7.f scrub starts 2024-09-20T22:51:52.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:52 smithi154 ceph-mon[33264]: 7.f scrub ok 2024-09-20T22:51:52.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:52 smithi154 ceph-mon[33264]: 5.f scrub starts 2024-09-20T22:51:52.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:52 smithi154 ceph-mon[33264]: 5.f scrub ok 2024-09-20T22:51:52.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:52 smithi154 ceph-mon[33264]: osdmap e631: 8 total, 8 up, 7 in 2024-09-20T22:51:52.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:52 smithi154 ceph-mon[33264]: 4.18 scrub starts 2024-09-20T22:51:52.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:52 smithi154 ceph-mon[33264]: 4.18 scrub ok 2024-09-20T22:51:52.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:52 smithi154 ceph-mon[33264]: 7.d scrub starts 2024-09-20T22:51:52.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:52 smithi154 ceph-mon[33264]: 7.d scrub ok 2024-09-20T22:51:52.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:52.759 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:52.759 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:51:52.809 INFO:tasks.rados.rados.0.smithi084.stdout:2491: finishing write tid 1 to smithi08477203-8 2024-09-20T22:51:52.809 INFO:tasks.rados.rados.0.smithi084.stdout:2491: finishing write tid 2 to smithi08477203-8 2024-09-20T22:51:52.809 INFO:tasks.rados.rados.0.smithi084.stdout:2491: finishing write tid 3 to smithi08477203-8 2024-09-20T22:51:52.810 INFO:tasks.rados.rados.0.smithi084.stdout:2491: finishing write tid 4 to smithi08477203-8 2024-09-20T22:51:52.810 INFO:tasks.rados.rados.0.smithi084.stdout:2491: oid 8 updating version 0 to 1801 2024-09-20T22:51:52.810 INFO:tasks.rados.rados.0.smithi084.stdout:2491: oid 8 updating version 1801 to 1803 2024-09-20T22:51:52.810 INFO:tasks.rados.rados.0.smithi084.stdout:2491: oid 8 version 1803 is already newer than 1802 2024-09-20T22:51:52.810 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 8 v 1803 (ObjNum 863 snap 276 seq_num 863) dirty exists 2024-09-20T22:51:52.810 INFO:tasks.rados.rados.0.smithi084.stdout:2491: left oid 8 (ObjNum 863 snap 276 seq_num 863) 2024-09-20T22:51:52.822 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-20T22:51:52.861 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:51:52.886 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:52.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:52.887 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:51:52.958 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:51:52.987 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:51:53.009 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:53.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:53.010 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:51:53.111 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:51:53.133 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:53.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:53.133 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:51:53.234 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:51:53.256 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:53.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:53.257 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:51:53.358 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:51:53.359 INFO:tasks.rados.rados.0.smithi084.stdout:2488: done (4 left) 2024-09-20T22:51:53.359 INFO:tasks.rados.rados.0.smithi084.stdout:2489: done (3 left) 2024-09-20T22:51:53.360 INFO:tasks.rados.rados.0.smithi084.stdout:2490: done (2 left) 2024-09-20T22:51:53.360 INFO:tasks.rados.rados.0.smithi084.stdout:2491: done (1 left) 2024-09-20T22:51:53.360 INFO:tasks.rados.rados.0.smithi084.stdout:2492: done (0 left) 2024-09-20T22:51:53.360 INFO:tasks.rados.rados.0.smithi084.stdout:2493: snap_create 2024-09-20T22:51:53.386 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:53.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:53.387 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:51:53.488 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:51:53.508 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:53.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:53.509 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:51:53.610 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:51:53.614 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:53 smithi084 ceph-mon[40024]: pgmap v849: 158 pgs: 3 active+clean+snaptrim, 2 active+remapped+backfill_toofull, 153 active+clean; 484 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 110 KiB/s rd, 1.3 MiB/s wr, 189 op/s; 37/1290 objects misplaced (2.868%) 2024-09-20T22:51:53.614 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:53 smithi084 ceph-mon[40024]: osdmap e632: 8 total, 8 up, 7 in 2024-09-20T22:51:53.614 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:53 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:51:53.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:53 smithi084 ceph-mon[29774]: pgmap v849: 158 pgs: 3 active+clean+snaptrim, 2 active+remapped+backfill_toofull, 153 active+clean; 484 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 110 KiB/s rd, 1.3 MiB/s wr, 189 op/s; 37/1290 objects misplaced (2.868%) 2024-09-20T22:51:53.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:53 smithi084 ceph-mon[29774]: osdmap e632: 8 total, 8 up, 7 in 2024-09-20T22:51:53.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:53 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:51:53.615 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:51:53.633 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:53.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:53.634 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:51:53.735 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:51:53.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:53 smithi154 ceph-mon[33264]: pgmap v849: 158 pgs: 3 active+clean+snaptrim, 2 active+remapped+backfill_toofull, 153 active+clean; 484 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 110 KiB/s rd, 1.3 MiB/s wr, 189 op/s; 37/1290 objects misplaced (2.868%) 2024-09-20T22:51:53.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:53 smithi154 ceph-mon[33264]: osdmap e632: 8 total, 8 up, 7 in 2024-09-20T22:51:53.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:53 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:51:53.755 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-20T22:51:53.756 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:53.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:53.757 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:51:53.858 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:51:53.881 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:53.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:53.882 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:51:53.982 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:51:54.003 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:54.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:54.004 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:51:54.105 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:51:54.126 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:54.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:54.126 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:51:54.150 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-20T22:51:54.150 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:51:54.156 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-20T22:51:54.227 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:51:54.252 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:54.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:54.253 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:51:54.354 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:51:54.365 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.2 to 1 (10000) 2024-09-20T22:51:54.365 INFO:tasks.rados.rados.0.smithi084.stdout:2493: done (0 left) 2024-09-20T22:51:54.366 INFO:tasks.rados.rados.0.smithi084.stdout:2494: read oid 14 snap 243 2024-09-20T22:51:54.366 INFO:tasks.rados.rados.0.smithi084.stdout:2494: expect deleted 2024-09-20T22:51:54.366 INFO:tasks.rados.rados.0.smithi084.stdout:2495: read oid 35 snap -1 2024-09-20T22:51:54.366 INFO:tasks.rados.rados.0.smithi084.stdout:2495: expect (ObjNum 787 snap 254 seq_num 787) 2024-09-20T22:51:54.366 INFO:tasks.rados.rados.0.smithi084.stdout:2496: copy_from oid 2 from oid 23 current snap is 278 2024-09-20T22:51:54.366 INFO:tasks.rados.rados.0.smithi084.stdout:2497: write oid 6 current snap is 278 2024-09-20T22:51:54.366 INFO:tasks.rados.rados.0.smithi084.stdout:2497: seq_num 864 ranges {521248=563025,1809501=596300} 2024-09-20T22:51:54.372 INFO:tasks.rados.rados.0.smithi084.stdout:2497: writing smithi08477203-6 from 521248 to 1084273 tid 1 2024-09-20T22:51:54.379 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:54.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:54.380 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:51:54.380 INFO:tasks.rados.rados.0.smithi084.stdout:2497: writing smithi08477203-6 from 1809501 to 2405801 tid 2 2024-09-20T22:51:54.380 INFO:tasks.rados.rados.0.smithi084.stdout:2498: rollback oid 3 current snap is 278 2024-09-20T22:51:54.381 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 3 to 272 2024-09-20T22:51:54.381 INFO:tasks.rados.rados.0.smithi084.stdout:2499: write oid 41 current snap is 278 2024-09-20T22:51:54.381 INFO:tasks.rados.rados.0.smithi084.stdout:2499: seq_num 865 ranges {538055=638274,1867983=670072} 2024-09-20T22:51:54.387 INFO:tasks.rados.rados.0.smithi084.stdout:2499: writing smithi08477203-41 from 538055 to 1176329 tid 1 2024-09-20T22:51:54.395 INFO:tasks.rados.rados.0.smithi084.stdout:2499: writing smithi08477203-41 from 1867983 to 2538055 tid 2 2024-09-20T22:51:54.395 INFO:tasks.rados.rados.0.smithi084.stdout:2495: expect (ObjNum 787 snap 254 seq_num 787) 2024-09-20T22:51:54.481 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:51:54.504 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:54.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:54.505 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:51:54.551 INFO:tasks.rados.rados.0.smithi084.stdout:2496: finishing copy_from racing read to smithi08477203-2 2024-09-20T22:51:54.551 INFO:tasks.rados.rados.0.smithi084.stdout:2498: finishing rollback tid 1 to smithi08477203-3 2024-09-20T22:51:54.551 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 3 v 0 (ObjNum 49 snap 0 seq_num 1953066355) dirty dne 2024-09-20T22:51:54.551 INFO:tasks.rados.rados.0.smithi084.stdout:2496: finishing copy_from to smithi08477203-2 2024-09-20T22:51:54.551 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 2 v 1231 (ObjNum 857 snap 275 seq_num 857) dirty exists 2024-09-20T22:51:54.551 INFO:tasks.rados.rados.0.smithi084.stdout:2497: finishing write tid 1 to smithi08477203-6 2024-09-20T22:51:54.551 INFO:tasks.rados.rados.0.smithi084.stdout:2494: done (5 left) 2024-09-20T22:51:54.551 INFO:tasks.rados.rados.0.smithi084.stdout:2495: done (4 left) 2024-09-20T22:51:54.551 INFO:tasks.rados.rados.0.smithi084.stdout:2496: done (3 left) 2024-09-20T22:51:54.551 INFO:tasks.rados.rados.0.smithi084.stdout:2498: done (2 left) 2024-09-20T22:51:54.551 INFO:tasks.rados.rados.0.smithi084.stdout:2500: snap_remove snap 262 2024-09-20T22:51:54.606 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:51:54.614 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:54 smithi084 ceph-mon[40024]: 4.16 scrub starts 2024-09-20T22:51:54.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:54 smithi084 ceph-mon[40024]: 4.16 scrub ok 2024-09-20T22:51:54.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:54 smithi084 ceph-mon[40024]: 4.8 scrub starts 2024-09-20T22:51:54.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:54 smithi084 ceph-mon[40024]: 4.8 scrub ok 2024-09-20T22:51:54.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:54 smithi084 ceph-mon[40024]: osdmap e633: 8 total, 8 up, 7 in 2024-09-20T22:51:54.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:54 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/298953321' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-09-20T22:51:54.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:54 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-09-20T22:51:54.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:54 smithi084 ceph-mon[40024]: 7.7 scrub starts 2024-09-20T22:51:54.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:54 smithi084 ceph-mon[40024]: 7.7 scrub ok 2024-09-20T22:51:54.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:54 smithi084 ceph-mon[29774]: 4.16 scrub starts 2024-09-20T22:51:54.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:54 smithi084 ceph-mon[29774]: 4.16 scrub ok 2024-09-20T22:51:54.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:54 smithi084 ceph-mon[29774]: 4.8 scrub starts 2024-09-20T22:51:54.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:54 smithi084 ceph-mon[29774]: 4.8 scrub ok 2024-09-20T22:51:54.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:54 smithi084 ceph-mon[29774]: osdmap e633: 8 total, 8 up, 7 in 2024-09-20T22:51:54.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:54 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/298953321' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-09-20T22:51:54.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:54 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-09-20T22:51:54.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:54 smithi084 ceph-mon[29774]: 7.7 scrub starts 2024-09-20T22:51:54.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:54 smithi084 ceph-mon[29774]: 7.7 scrub ok 2024-09-20T22:51:54.628 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:54.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:54.629 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:51:54.730 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:51:54.752 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:54.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:54.753 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:51:54.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:54 smithi154 ceph-mon[33264]: 4.16 scrub starts 2024-09-20T22:51:54.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:54 smithi154 ceph-mon[33264]: 4.16 scrub ok 2024-09-20T22:51:54.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:54 smithi154 ceph-mon[33264]: 4.8 scrub starts 2024-09-20T22:51:54.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:54 smithi154 ceph-mon[33264]: 4.8 scrub ok 2024-09-20T22:51:54.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:54 smithi154 ceph-mon[33264]: osdmap e633: 8 total, 8 up, 7 in 2024-09-20T22:51:54.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:54 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/298953321' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-09-20T22:51:54.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:54 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-09-20T22:51:54.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:54 smithi154 ceph-mon[33264]: 7.7 scrub starts 2024-09-20T22:51:54.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:54 smithi154 ceph-mon[33264]: 7.7 scrub ok 2024-09-20T22:51:54.854 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:51:54.876 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:54.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:54.877 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:51:54.977 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:51:54.998 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:54.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:54.999 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:51:55.100 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:51:55.107 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd reweight 3 1 2024-09-20T22:51:55.121 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:55.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:55.122 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:51:55.223 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:51:55.244 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:55.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:55.245 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:51:55.346 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:51:55.368 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:55.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:55.369 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:51:55.374 INFO:tasks.rados.rados.0.smithi084.stdout:2500: done (2 left) 2024-09-20T22:51:55.374 INFO:tasks.rados.rados.0.smithi084.stdout:2501: copy_from oid 17 from oid 49 current snap is 278 2024-09-20T22:51:55.374 INFO:tasks.rados.rados.0.smithi084.stdout:2497: finishing write tid 2 to smithi08477203-6 2024-09-20T22:51:55.374 INFO:tasks.rados.rados.0.smithi084.stdout:2497: finishing write tid 3 to smithi08477203-6 2024-09-20T22:51:55.374 INFO:tasks.rados.rados.0.smithi084.stdout:2497: finishing write tid 4 to smithi08477203-6 2024-09-20T22:51:55.374 INFO:tasks.rados.rados.0.smithi084.stdout:2497: oid 6 updating version 0 to 784 2024-09-20T22:51:55.374 INFO:tasks.rados.rados.0.smithi084.stdout:2497: oid 6 updating version 784 to 785 2024-09-20T22:51:55.374 INFO:tasks.rados.rados.0.smithi084.stdout:2497: oid 6 version 785 is already newer than 783 2024-09-20T22:51:55.375 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 6 v 785 (ObjNum 864 snap 278 seq_num 864) dirty exists 2024-09-20T22:51:55.375 INFO:tasks.rados.rados.0.smithi084.stdout:2497: left oid 6 (ObjNum 864 snap 278 seq_num 864) 2024-09-20T22:51:55.375 INFO:tasks.rados.rados.0.smithi084.stdout:2497: done (2 left) 2024-09-20T22:51:55.375 INFO:tasks.rados.rados.0.smithi084.stdout:2502: write oid 26 current snap is 278 2024-09-20T22:51:55.375 INFO:tasks.rados.rados.0.smithi084.stdout:2502: seq_num 866 ranges {554862=713523,1926465=599969} 2024-09-20T22:51:55.381 INFO:tasks.rados.rados.0.smithi084.stdout:2502: writing smithi08477203-26 from 554862 to 1268385 tid 1 2024-09-20T22:51:55.390 INFO:tasks.rados.rados.0.smithi084.stdout:2502: writing smithi08477203-26 from 1926465 to 2526434 tid 2 2024-09-20T22:51:55.390 INFO:tasks.rados.rados.0.smithi084.stdout:2503: snap_create 2024-09-20T22:51:55.390 INFO:tasks.rados.rados.0.smithi084.stdout:2499: finishing write tid 1 to smithi08477203-41 2024-09-20T22:51:55.391 INFO:tasks.rados.rados.0.smithi084.stdout:2499: finishing write tid 2 to smithi08477203-41 2024-09-20T22:51:55.391 INFO:tasks.rados.rados.0.smithi084.stdout:2499: finishing write tid 3 to smithi08477203-41 2024-09-20T22:51:55.391 INFO:tasks.rados.rados.0.smithi084.stdout:2499: finishing write tid 4 to smithi08477203-41 2024-09-20T22:51:55.391 INFO:tasks.rados.rados.0.smithi084.stdout:2499: oid 41 updating version 0 to 1404 2024-09-20T22:51:55.391 INFO:tasks.rados.rados.0.smithi084.stdout:2499: oid 41 version 1404 is already newer than 1403 2024-09-20T22:51:55.391 INFO:tasks.rados.rados.0.smithi084.stdout:2499: oid 41 updating version 1404 to 1405 2024-09-20T22:51:55.391 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 41 v 1405 (ObjNum 865 snap 278 seq_num 865) dirty exists 2024-09-20T22:51:55.391 INFO:tasks.rados.rados.0.smithi084.stdout:2499: left oid 41 (ObjNum 865 snap 278 seq_num 865) 2024-09-20T22:51:55.391 INFO:tasks.rados.rados.0.smithi084.stdout:2502: finishing write tid 1 to smithi08477203-26 2024-09-20T22:51:55.392 INFO:tasks.rados.rados.0.smithi084.stdout:2501: finishing copy_from racing read to smithi08477203-17 2024-09-20T22:51:55.399 INFO:tasks.rados.rados.0.smithi084.stdout:2502: finishing write tid 2 to smithi08477203-26 2024-09-20T22:51:55.399 INFO:tasks.rados.rados.0.smithi084.stdout:2502: finishing write tid 3 to smithi08477203-26 2024-09-20T22:51:55.399 INFO:tasks.rados.rados.0.smithi084.stdout:2501: finishing copy_from to smithi08477203-17 2024-09-20T22:51:55.399 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 17 v 753 (ObjNum 815 snap 262 seq_num 815) dirty exists 2024-09-20T22:51:55.399 INFO:tasks.rados.rados.0.smithi084.stdout:2502: finishing write tid 4 to smithi08477203-26 2024-09-20T22:51:55.400 INFO:tasks.rados.rados.0.smithi084.stdout:2502: oid 26 updating version 0 to 1332 2024-09-20T22:51:55.400 INFO:tasks.rados.rados.0.smithi084.stdout:2502: oid 26 version 1332 is already newer than 1331 2024-09-20T22:51:55.400 INFO:tasks.rados.rados.0.smithi084.stdout:2502: oid 26 version 1332 is already newer than 1330 2024-09-20T22:51:55.400 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 26 v 1332 (ObjNum 866 snap 278 seq_num 866) dirty exists 2024-09-20T22:51:55.400 INFO:tasks.rados.rados.0.smithi084.stdout:2502: left oid 26 (ObjNum 866 snap 278 seq_num 866) 2024-09-20T22:51:55.470 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:51:55.494 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:55.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:55.495 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:51:55.568 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:51:55.588 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:51:55.590 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.3 dump_blocked_ops 2024-09-20T22:51:55.596 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:51:55.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:55 smithi084 ceph-mon[29774]: 5.5 scrub starts 2024-09-20T22:51:55.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:55 smithi084 ceph-mon[29774]: 5.5 scrub ok 2024-09-20T22:51:55.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:55 smithi084 ceph-mon[29774]: pgmap v852: 158 pgs: 4 active+clean+snaptrim, 1 active+remapped+backfill_toofull, 153 active+clean; 508 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 13 MiB/s rd, 3.9 MiB/s wr, 29 op/s; 21/1323 objects misplaced (1.587%); 8.4 MiB/s, 4 objects/s recovering 2024-09-20T22:51:55.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:55 smithi084 ceph-mon[29774]: 6.a scrub starts 2024-09-20T22:51:55.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:55 smithi084 ceph-mon[29774]: 6.a scrub ok 2024-09-20T22:51:55.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:55 smithi084 ceph-mon[29774]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-09-20T22:51:55.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:55 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-20T22:51:55.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:55 smithi084 ceph-mon[29774]: osdmap e634: 8 total, 8 up, 7 in 2024-09-20T22:51:55.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:55 smithi084 ceph-mon[29774]: purged_snaps scrub starts 2024-09-20T22:51:55.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:55 smithi084 ceph-mon[29774]: purged_snaps scrub ok 2024-09-20T22:51:55.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:55 smithi084 ceph-mon[40024]: 5.5 scrub starts 2024-09-20T22:51:55.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:55 smithi084 ceph-mon[40024]: 5.5 scrub ok 2024-09-20T22:51:55.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:55 smithi084 ceph-mon[40024]: pgmap v852: 158 pgs: 4 active+clean+snaptrim, 1 active+remapped+backfill_toofull, 153 active+clean; 508 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 13 MiB/s rd, 3.9 MiB/s wr, 29 op/s; 21/1323 objects misplaced (1.587%); 8.4 MiB/s, 4 objects/s recovering 2024-09-20T22:51:55.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:55 smithi084 ceph-mon[40024]: 6.a scrub starts 2024-09-20T22:51:55.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:55 smithi084 ceph-mon[40024]: 6.a scrub ok 2024-09-20T22:51:55.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:55 smithi084 ceph-mon[40024]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-09-20T22:51:55.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:55 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-20T22:51:55.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:55 smithi084 ceph-mon[40024]: osdmap e634: 8 total, 8 up, 7 in 2024-09-20T22:51:55.617 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:55 smithi084 ceph-mon[40024]: purged_snaps scrub starts 2024-09-20T22:51:55.617 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:55 smithi084 ceph-mon[40024]: purged_snaps scrub ok 2024-09-20T22:51:55.638 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:55.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:55.639 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:51:55.740 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:51:55.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:55 smithi154 ceph-mon[33264]: 5.5 scrub starts 2024-09-20T22:51:55.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:55 smithi154 ceph-mon[33264]: 5.5 scrub ok 2024-09-20T22:51:55.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:55 smithi154 ceph-mon[33264]: pgmap v852: 158 pgs: 4 active+clean+snaptrim, 1 active+remapped+backfill_toofull, 153 active+clean; 508 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 13 MiB/s rd, 3.9 MiB/s wr, 29 op/s; 21/1323 objects misplaced (1.587%); 8.4 MiB/s, 4 objects/s recovering 2024-09-20T22:51:55.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:55 smithi154 ceph-mon[33264]: 6.a scrub starts 2024-09-20T22:51:55.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:55 smithi154 ceph-mon[33264]: 6.a scrub ok 2024-09-20T22:51:55.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:55 smithi154 ceph-mon[33264]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-09-20T22:51:55.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:55 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-20T22:51:55.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:55 smithi154 ceph-mon[33264]: osdmap e634: 8 total, 8 up, 7 in 2024-09-20T22:51:55.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:55 smithi154 ceph-mon[33264]: purged_snaps scrub starts 2024-09-20T22:51:55.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:55 smithi154 ceph-mon[33264]: purged_snaps scrub ok 2024-09-20T22:51:55.761 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:55.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:55.762 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:51:55.863 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:51:55.889 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:55.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:55.890 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:51:55.991 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:51:56.016 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:56.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:56.017 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:51:56.118 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:51:56.143 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:56.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:56.144 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:51:56.244 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:51:56.266 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:56.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:56.267 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:51:56.367 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:51:56.372 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is unset 2024-09-20T22:51:56.380 INFO:tasks.rados.rados.0.smithi084.stdout:2499: done (3 left) 2024-09-20T22:51:56.380 INFO:tasks.rados.rados.0.smithi084.stdout:2501: done (2 left) 2024-09-20T22:51:56.380 INFO:tasks.rados.rados.0.smithi084.stdout:2502: done (1 left) 2024-09-20T22:51:56.381 INFO:tasks.rados.rados.0.smithi084.stdout:2503: done (0 left) 2024-09-20T22:51:56.381 INFO:tasks.rados.rados.0.smithi084.stdout:2504: delete oid 30 current snap is 279 2024-09-20T22:51:56.385 INFO:tasks.rados.rados.0.smithi084.stdout:2504: done (0 left) 2024-09-20T22:51:56.385 INFO:tasks.rados.rados.0.smithi084.stdout:2505: read oid 11 snap 192 2024-09-20T22:51:56.385 INFO:tasks.rados.rados.0.smithi084.stdout:2505: expect (ObjNum 411 snap 105 seq_num 411) 2024-09-20T22:51:56.385 INFO:tasks.rados.rados.0.smithi084.stdout:2506: write (excl) oid 26 current snap is 279 2024-09-20T22:51:56.385 INFO:tasks.rados.rados.0.smithi084.stdout:2506: seq_num 868 ranges {588476=580374,1643429=603638} 2024-09-20T22:51:56.392 INFO:tasks.rados.rados.0.smithi084.stdout:2506: writing smithi08477203-26 from 588476 to 1168850 tid 1 2024-09-20T22:51:56.395 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:56.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:56.397 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:51:56.398 INFO:tasks.rados.rados.0.smithi084.stdout:2506: writing smithi08477203-26 from 1643429 to 2247067 tid 2 2024-09-20T22:51:56.400 INFO:tasks.rados.rados.0.smithi084.stdout:2505: expect (ObjNum 411 snap 105 seq_num 411) 2024-09-20T22:51:56.498 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:51:56.522 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:56.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:56.523 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:51:56.552 INFO:tasks.rados.rados.0.smithi084.stdout:2506: finishing write tid 1 to smithi08477203-26 2024-09-20T22:51:56.552 INFO:tasks.rados.rados.0.smithi084.stdout:2506: finishing write tid 2 to smithi08477203-26 2024-09-20T22:51:56.552 INFO:tasks.rados.rados.0.smithi084.stdout:2506: finishing write tid 3 to smithi08477203-26 2024-09-20T22:51:56.552 INFO:tasks.rados.rados.0.smithi084.stdout:2506: finishing write tid 4 to smithi08477203-26 2024-09-20T22:51:56.552 INFO:tasks.rados.rados.0.smithi084.stdout:2506: oid 26 updating version 0 to 1333 2024-09-20T22:51:56.552 INFO:tasks.rados.rados.0.smithi084.stdout:2506: oid 26 updating version 1333 to 1335 2024-09-20T22:51:56.552 INFO:tasks.rados.rados.0.smithi084.stdout:2506: oid 26 version 1335 is already newer than 1334 2024-09-20T22:51:56.552 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 26 v 1335 (ObjNum 868 snap 279 seq_num 868) dirty exists 2024-09-20T22:51:56.553 INFO:tasks.rados.rados.0.smithi084.stdout:2506: left oid 26 (ObjNum 868 snap 279 seq_num 868) 2024-09-20T22:51:56.553 INFO:tasks.rados.rados.0.smithi084.stdout:2505: done (1 left) 2024-09-20T22:51:56.553 INFO:tasks.rados.rados.0.smithi084.stdout:2506: done (0 left) 2024-09-20T22:51:56.553 INFO:tasks.rados.rados.0.smithi084.stdout:2507: read oid 21 snap 243 2024-09-20T22:51:56.553 INFO:tasks.rados.rados.0.smithi084.stdout:2507: expect (ObjNum 636 snap 193 seq_num 636) 2024-09-20T22:51:56.553 INFO:tasks.rados.rados.0.smithi084.stdout:2508: snap_remove snap 234 2024-09-20T22:51:56.614 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:56 smithi084 ceph-mon[29774]: osdmap e635: 8 total, 8 up, 7 in 2024-09-20T22:51:56.614 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:56 smithi084 ceph-mon[29774]: 6.2 scrub starts 2024-09-20T22:51:56.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:56 smithi084 ceph-mon[29774]: 6.2 scrub ok 2024-09-20T22:51:56.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:56 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/547087786' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:51:56.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:56 smithi084 ceph-mon[40024]: osdmap e635: 8 total, 8 up, 7 in 2024-09-20T22:51:56.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:56 smithi084 ceph-mon[40024]: 6.2 scrub starts 2024-09-20T22:51:56.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:56 smithi084 ceph-mon[40024]: 6.2 scrub ok 2024-09-20T22:51:56.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:56 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/547087786' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:51:56.623 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:51:56.648 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:56.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:56.649 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:51:56.750 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:51:56.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:56 smithi154 ceph-mon[33264]: osdmap e635: 8 total, 8 up, 7 in 2024-09-20T22:51:56.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:56 smithi154 ceph-mon[33264]: 6.2 scrub starts 2024-09-20T22:51:56.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:56 smithi154 ceph-mon[33264]: 6.2 scrub ok 2024-09-20T22:51:56.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:56 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/547087786' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:51:56.774 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:56.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:56.775 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:51:56.876 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:51:56.899 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:56.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:56.901 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:51:57.001 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:51:57.023 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:57.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:57.024 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:51:57.042 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:51:57.042 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:51:57.045 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:51:57.045 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:51:57.064 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:51:57.064 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:51:57.085 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:51:57.085 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:51:57.101 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-20T22:51:57.101 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T22:51:57.118 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:51:57.118 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:51:57.125 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:51:57.135 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:51:57.135 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:51:57.150 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:57.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:57.152 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:51:57.152 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:51:57.152 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:51:57.169 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:51:57.169 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:51:57.252 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:51:57.273 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:57.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:57.275 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:51:57.376 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:51:57.397 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:57.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:57.398 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:51:57.406 INFO:tasks.rados.rados.0.smithi084.stdout:2508: done (1 left) 2024-09-20T22:51:57.406 INFO:tasks.rados.rados.0.smithi084.stdout:2509: copy_from oid 34 from oid 15 current snap is 279 2024-09-20T22:51:57.406 INFO:tasks.rados.rados.0.smithi084.stdout:2510: snap_create 2024-09-20T22:51:57.406 INFO:tasks.rados.rados.0.smithi084.stdout:2507: expect (ObjNum 636 snap 193 seq_num 636) 2024-09-20T22:51:57.499 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:51:57.524 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:57.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:57.525 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:51:57.561 INFO:tasks.rados.rados.0.smithi084.stdout:2509: finishing copy_from to smithi08477203-34 2024-09-20T22:51:57.561 INFO:tasks.rados.rados.0.smithi084.stdout:2509: got expected ENOENT (src dne) 2024-09-20T22:51:57.562 INFO:tasks.rados.rados.0.smithi084.stdout:2509: finishing copy_from racing read to smithi08477203-34 2024-09-20T22:51:57.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:57 smithi084 ceph-mon[29774]: 3.e scrub starts 2024-09-20T22:51:57.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:57 smithi084 ceph-mon[29774]: 3.e scrub ok 2024-09-20T22:51:57.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:57 smithi084 ceph-mon[29774]: 6.0 scrub starts 2024-09-20T22:51:57.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:57 smithi084 ceph-mon[29774]: 6.0 scrub ok 2024-09-20T22:51:57.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:57 smithi084 ceph-mon[29774]: pgmap v855: 158 pgs: 4 active+clean+snaptrim, 1 active+remapped+backfill_toofull, 153 active+clean; 508 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 13 MiB/s rd, 3.9 MiB/s wr, 29 op/s; 21/1323 objects misplaced (1.587%); 8.4 MiB/s, 4 objects/s recovering 2024-09-20T22:51:57.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:57 smithi084 ceph-mon[29774]: 1.8 scrub starts 2024-09-20T22:51:57.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:57 smithi084 ceph-mon[29774]: 1.8 scrub ok 2024-09-20T22:51:57.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:57 smithi084 ceph-mon[29774]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T22:51:57.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:57 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/547087786' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T22:51:57.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:57 smithi084 ceph-mon[29774]: osdmap e636: 8 total, 8 up, 7 in 2024-09-20T22:51:57.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:57 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/2689642513' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:51:57.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:57 smithi084 ceph-mon[40024]: 3.e scrub starts 2024-09-20T22:51:57.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:57 smithi084 ceph-mon[40024]: 3.e scrub ok 2024-09-20T22:51:57.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:57 smithi084 ceph-mon[40024]: 6.0 scrub starts 2024-09-20T22:51:57.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:57 smithi084 ceph-mon[40024]: 6.0 scrub ok 2024-09-20T22:51:57.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:57 smithi084 ceph-mon[40024]: pgmap v855: 158 pgs: 4 active+clean+snaptrim, 1 active+remapped+backfill_toofull, 153 active+clean; 508 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 13 MiB/s rd, 3.9 MiB/s wr, 29 op/s; 21/1323 objects misplaced (1.587%); 8.4 MiB/s, 4 objects/s recovering 2024-09-20T22:51:57.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:57 smithi084 ceph-mon[40024]: 1.8 scrub starts 2024-09-20T22:51:57.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:57 smithi084 ceph-mon[40024]: 1.8 scrub ok 2024-09-20T22:51:57.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:57 smithi084 ceph-mon[40024]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T22:51:57.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:57 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/547087786' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T22:51:57.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:57 smithi084 ceph-mon[40024]: osdmap e636: 8 total, 8 up, 7 in 2024-09-20T22:51:57.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:57 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/2689642513' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:51:57.626 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:51:57.649 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:57.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:57.650 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:51:57.750 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:51:57.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:57 smithi154 ceph-mon[33264]: 3.e scrub starts 2024-09-20T22:51:57.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:57 smithi154 ceph-mon[33264]: 3.e scrub ok 2024-09-20T22:51:57.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:57 smithi154 ceph-mon[33264]: 6.0 scrub starts 2024-09-20T22:51:57.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:57 smithi154 ceph-mon[33264]: 6.0 scrub ok 2024-09-20T22:51:57.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:57 smithi154 ceph-mon[33264]: pgmap v855: 158 pgs: 4 active+clean+snaptrim, 1 active+remapped+backfill_toofull, 153 active+clean; 508 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 13 MiB/s rd, 3.9 MiB/s wr, 29 op/s; 21/1323 objects misplaced (1.587%); 8.4 MiB/s, 4 objects/s recovering 2024-09-20T22:51:57.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:57 smithi154 ceph-mon[33264]: 1.8 scrub starts 2024-09-20T22:51:57.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:57 smithi154 ceph-mon[33264]: 1.8 scrub ok 2024-09-20T22:51:57.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:57 smithi154 ceph-mon[33264]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T22:51:57.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:57 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/547087786' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T22:51:57.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:57 smithi154 ceph-mon[33264]: osdmap e636: 8 total, 8 up, 7 in 2024-09-20T22:51:57.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:57 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/2689642513' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:51:57.773 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:57.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:57.774 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:51:57.875 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:51:57.898 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:57.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:57.899 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:51:57.999 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:51:58.023 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:58.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:58.024 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:51:58.124 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:51:58.146 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:58.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:58.147 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:51:58.247 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:51:58.269 INFO:tasks.rados.rados.0.smithi084.stdout:2507: done (2 left) 2024-09-20T22:51:58.269 INFO:tasks.rados.rados.0.smithi084.stdout:2509: done (1 left) 2024-09-20T22:51:58.269 INFO:tasks.rados.rados.0.smithi084.stdout:2510: done (0 left) 2024-09-20T22:51:58.270 INFO:tasks.rados.rados.0.smithi084.stdout:2511: delete oid 23 current snap is 280 2024-09-20T22:51:58.274 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:58.275 INFO:tasks.rados.rados.0.smithi084.stdout:2511: done (0 left) 2024-09-20T22:51:58.275 INFO:tasks.rados.rados.0.smithi084.stdout:2512: copy_from oid 4 from oid 8 current snap is 280 2024-09-20T22:51:58.275 INFO:tasks.rados.rados.0.smithi084.stdout:2513: delete oid 10 current snap is 280 2024-09-20T22:51:58.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:58.276 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:51:58.278 INFO:tasks.rados.rados.0.smithi084.stdout:2513: done (1 left) 2024-09-20T22:51:58.278 INFO:tasks.rados.rados.0.smithi084.stdout:2514: delete oid 36 current snap is 280 2024-09-20T22:51:58.280 INFO:tasks.rados.rados.0.smithi084.stdout:2514: done (1 left) 2024-09-20T22:51:58.280 INFO:tasks.rados.rados.0.smithi084.stdout:2515: write (excl) oid 5 current snap is 280 2024-09-20T22:51:58.280 INFO:tasks.rados.rados.0.smithi084.stdout:2515: seq_num 872 ranges {655704=481370,1761004=494623} 2024-09-20T22:51:58.296 INFO:tasks.rados.rados.0.smithi084.stdout:2515: writing smithi08477203-5 from 655704 to 1137074 tid 1 2024-09-20T22:51:58.296 INFO:tasks.rados.rados.0.smithi084.stdout:2515: writing smithi08477203-5 from 1761004 to 2255627 tid 2 2024-09-20T22:51:58.296 INFO:tasks.rados.rados.0.smithi084.stdout:2516: write (excl) oid 29 current snap is 280 2024-09-20T22:51:58.296 INFO:tasks.rados.rados.0.smithi084.stdout:2516: seq_num 873 ranges {672511=556619,1819486=754634} 2024-09-20T22:51:58.299 INFO:tasks.rados.rados.0.smithi084.stdout:2516: writing smithi08477203-29 from 672511 to 1229130 tid 1 2024-09-20T22:51:58.307 INFO:tasks.rados.rados.0.smithi084.stdout:2516: writing smithi08477203-29 from 1819486 to 2574120 tid 2 2024-09-20T22:51:58.307 INFO:tasks.rados.rados.0.smithi084.stdout:2517: write (excl) oid 20 current snap is 280 2024-09-20T22:51:58.307 INFO:tasks.rados.rados.0.smithi084.stdout:2515: finishing write tid 1 to smithi08477203-5 2024-09-20T22:51:58.307 INFO:tasks.rados.rados.0.smithi084.stdout:2517: seq_num 874 ranges {689318=631868,1877968=498292} 2024-09-20T22:51:58.314 INFO:tasks.rados.rados.0.smithi084.stdout:2517: writing smithi08477203-20 from 689318 to 1321186 tid 1 2024-09-20T22:51:58.321 INFO:tasks.rados.rados.0.smithi084.stdout:2517: writing smithi08477203-20 from 1877968 to 2376260 tid 2 2024-09-20T22:51:58.321 INFO:tasks.rados.rados.0.smithi084.stdout:2518: delete oid 24 current snap is 280 2024-09-20T22:51:58.321 INFO:tasks.rados.rados.0.smithi084.stdout:2515: finishing write tid 2 to smithi08477203-5 2024-09-20T22:51:58.321 INFO:tasks.rados.rados.0.smithi084.stdout:2515: finishing write tid 3 to smithi08477203-5 2024-09-20T22:51:58.321 INFO:tasks.rados.rados.0.smithi084.stdout:2515: finishing write tid 4 to smithi08477203-5 2024-09-20T22:51:58.321 INFO:tasks.rados.rados.0.smithi084.stdout:2515: oid 5 updating version 0 to 763 2024-09-20T22:51:58.321 INFO:tasks.rados.rados.0.smithi084.stdout:2515: oid 5 version 763 is already newer than 761 2024-09-20T22:51:58.321 INFO:tasks.rados.rados.0.smithi084.stdout:2515: oid 5 version 763 is already newer than 762 2024-09-20T22:51:58.321 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 5 v 763 (ObjNum 872 snap 280 seq_num 872) dirty exists 2024-09-20T22:51:58.321 INFO:tasks.rados.rados.0.smithi084.stdout:2515: left oid 5 (ObjNum 872 snap 280 seq_num 872) 2024-09-20T22:51:58.321 INFO:tasks.rados.rados.0.smithi084.stdout:2512: finishing copy_from racing read to smithi08477203-4 2024-09-20T22:51:58.321 INFO:tasks.rados.rados.0.smithi084.stdout:2516: finishing write tid 1 to smithi08477203-29 2024-09-20T22:51:58.321 INFO:tasks.rados.rados.0.smithi084.stdout:2512: finishing copy_from to smithi08477203-4 2024-09-20T22:51:58.321 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 4 v 1803 (ObjNum 863 snap 276 seq_num 863) dirty exists 2024-09-20T22:51:58.322 INFO:tasks.rados.rados.0.smithi084.stdout:2516: finishing write tid 2 to smithi08477203-29 2024-09-20T22:51:58.322 INFO:tasks.rados.rados.0.smithi084.stdout:2516: finishing write tid 3 to smithi08477203-29 2024-09-20T22:51:58.322 INFO:tasks.rados.rados.0.smithi084.stdout:2516: finishing write tid 4 to smithi08477203-29 2024-09-20T22:51:58.322 INFO:tasks.rados.rados.0.smithi084.stdout:2516: oid 29 updating version 0 to 1881 2024-09-20T22:51:58.322 INFO:tasks.rados.rados.0.smithi084.stdout:2516: oid 29 version 1881 is already newer than 1880 2024-09-20T22:51:58.322 INFO:tasks.rados.rados.0.smithi084.stdout:2516: oid 29 version 1881 is already newer than 1877 2024-09-20T22:51:58.322 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 29 v 1881 (ObjNum 873 snap 280 seq_num 873) dirty exists 2024-09-20T22:51:58.322 INFO:tasks.rados.rados.0.smithi084.stdout:2516: left oid 29 (ObjNum 873 snap 280 seq_num 873) 2024-09-20T22:51:58.322 INFO:tasks.rados.rados.0.smithi084.stdout:2517: finishing write tid 1 to smithi08477203-20 2024-09-20T22:51:58.322 INFO:tasks.rados.rados.0.smithi084.stdout:2512: done (4 left) 2024-09-20T22:51:58.322 INFO:tasks.rados.rados.0.smithi084.stdout:2515: done (3 left) 2024-09-20T22:51:58.322 INFO:tasks.rados.rados.0.smithi084.stdout:2516: done (2 left) 2024-09-20T22:51:58.323 INFO:tasks.rados.rados.0.smithi084.stdout:2518: done (1 left) 2024-09-20T22:51:58.323 INFO:tasks.rados.rados.0.smithi084.stdout:2519: write oid 30 current snap is 280 2024-09-20T22:51:58.323 INFO:tasks.rados.rados.0.smithi084.stdout:2519: seq_num 876 ranges {722932=498719,1994932=501961} 2024-09-20T22:51:58.329 INFO:tasks.rados.rados.0.smithi084.stdout:2519: writing smithi08477203-30 from 722932 to 1221651 tid 1 2024-09-20T22:51:58.336 INFO:tasks.rados.rados.0.smithi084.stdout:2519: writing smithi08477203-30 from 1994932 to 2496893 tid 2 2024-09-20T22:51:58.336 INFO:tasks.rados.rados.0.smithi084.stdout:2520: snap_create 2024-09-20T22:51:58.336 INFO:tasks.rados.rados.0.smithi084.stdout:2517: finishing write tid 2 to smithi08477203-20 2024-09-20T22:51:58.336 INFO:tasks.rados.rados.0.smithi084.stdout:2517: finishing write tid 3 to smithi08477203-20 2024-09-20T22:51:58.336 INFO:tasks.rados.rados.0.smithi084.stdout:2517: finishing write tid 4 to smithi08477203-20 2024-09-20T22:51:58.337 INFO:tasks.rados.rados.0.smithi084.stdout:2517: oid 20 updating version 0 to 1242 2024-09-20T22:51:58.337 INFO:tasks.rados.rados.0.smithi084.stdout:2517: oid 20 updating version 1242 to 1243 2024-09-20T22:51:58.337 INFO:tasks.rados.rados.0.smithi084.stdout:2517: oid 20 version 1243 is already newer than 1241 2024-09-20T22:51:58.337 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 20 v 1243 (ObjNum 874 snap 280 seq_num 874) dirty exists 2024-09-20T22:51:58.337 INFO:tasks.rados.rados.0.smithi084.stdout:2517: left oid 20 (ObjNum 874 snap 280 seq_num 874) 2024-09-20T22:51:58.337 INFO:tasks.rados.rados.0.smithi084.stdout:2519: finishing write tid 1 to smithi08477203-30 2024-09-20T22:51:58.339 INFO:tasks.rados.rados.0.smithi084.stdout:2519: finishing write tid 2 to smithi08477203-30 2024-09-20T22:51:58.339 INFO:tasks.rados.rados.0.smithi084.stdout:2519: finishing write tid 3 to smithi08477203-30 2024-09-20T22:51:58.339 INFO:tasks.rados.rados.0.smithi084.stdout:2519: finishing write tid 4 to smithi08477203-30 2024-09-20T22:51:58.339 INFO:tasks.rados.rados.0.smithi084.stdout:2519: oid 30 updating version 0 to 1806 2024-09-20T22:51:58.339 INFO:tasks.rados.rados.0.smithi084.stdout:2519: oid 30 updating version 1806 to 1807 2024-09-20T22:51:58.339 INFO:tasks.rados.rados.0.smithi084.stdout:2519: oid 30 version 1807 is already newer than 1805 2024-09-20T22:51:58.339 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 30 v 1807 (ObjNum 876 snap 280 seq_num 876) dirty exists 2024-09-20T22:51:58.339 INFO:tasks.rados.rados.0.smithi084.stdout:2519: left oid 30 (ObjNum 876 snap 280 seq_num 876) 2024-09-20T22:51:58.377 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:51:58.397 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:58.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:58.398 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:51:58.499 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:51:58.520 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:58.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:58.521 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:51:58.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:58 smithi084 ceph-mon[29774]: osdmap e637: 8 total, 8 up, 7 in 2024-09-20T22:51:58.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:58 smithi084 ceph-mon[29774]: 3.10 scrub starts 2024-09-20T22:51:58.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:58 smithi084 ceph-mon[29774]: 3.10 scrub ok 2024-09-20T22:51:58.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:58 smithi084 ceph-mon[29774]: osdmap e638: 8 total, 8 up, 7 in 2024-09-20T22:51:58.615 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:51:58 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: ::ffff:172.21.15.154 - - [20/Sep/2024:22:51:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-20T22:51:58.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:58 smithi084 ceph-mon[40024]: osdmap e637: 8 total, 8 up, 7 in 2024-09-20T22:51:58.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:58 smithi084 ceph-mon[40024]: 3.10 scrub starts 2024-09-20T22:51:58.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:58 smithi084 ceph-mon[40024]: 3.10 scrub ok 2024-09-20T22:51:58.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:58 smithi084 ceph-mon[40024]: osdmap e638: 8 total, 8 up, 7 in 2024-09-20T22:51:58.622 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:51:58.643 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:58.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:58.644 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:51:58.744 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:51:58.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:58 smithi154 ceph-mon[33264]: osdmap e637: 8 total, 8 up, 7 in 2024-09-20T22:51:58.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:58 smithi154 ceph-mon[33264]: 3.10 scrub starts 2024-09-20T22:51:58.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:58 smithi154 ceph-mon[33264]: 3.10 scrub ok 2024-09-20T22:51:58.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:58 smithi154 ceph-mon[33264]: osdmap e638: 8 total, 8 up, 7 in 2024-09-20T22:51:58.770 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:58.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:58.771 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:51:58.872 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:51:58.893 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:58.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:58.894 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:51:58.975 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-20T22:51:58.975 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2024-09-20T22:51:58.975 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2024-09-20T22:51:58.975 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2024-09-20T22:51:58.975 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-20T22:51:58.994 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:51:59.020 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:59.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:59.021 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:51:59.121 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:51:59.147 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:59.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:59.148 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:51:59.249 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:51:59.270 INFO:tasks.rados.rados.0.smithi084.stdout:2517: done (2 left) 2024-09-20T22:51:59.271 INFO:tasks.rados.rados.0.smithi084.stdout:2519: done (1 left) 2024-09-20T22:51:59.271 INFO:tasks.rados.rados.0.smithi084.stdout:2520: done (0 left) 2024-09-20T22:51:59.271 INFO:tasks.rados.rados.0.smithi084.stdout:2521: copy_from oid 34 from oid 46 current snap is 281 2024-09-20T22:51:59.271 INFO:tasks.rados.rados.0.smithi084.stdout:2522: copy_from oid 5 from oid 37 current snap is 281 2024-09-20T22:51:59.271 INFO:tasks.rados.rados.0.smithi084.stdout:2523: snap_create 2024-09-20T22:51:59.271 INFO:tasks.rados.rados.0.smithi084.stdout:2522: finishing copy_from to smithi08477203-5 2024-09-20T22:51:59.271 INFO:tasks.rados.rados.0.smithi084.stdout:2522: got expected ENOENT (src dne) 2024-09-20T22:51:59.272 INFO:tasks.rados.rados.0.smithi084.stdout:2522: finishing copy_from racing read to smithi08477203-5 2024-09-20T22:51:59.273 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:59.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:59.274 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:51:59.285 INFO:tasks.rados.rados.0.smithi084.stdout:2521: finishing copy_from racing read to smithi08477203-34 2024-09-20T22:51:59.294 INFO:tasks.rados.rados.0.smithi084.stdout:2521: finishing copy_from to smithi08477203-34 2024-09-20T22:51:59.294 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 34 v 1800 (ObjNum 470 snap 130 seq_num 470) dirty exists 2024-09-20T22:51:59.375 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:51:59.396 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:59.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:59.397 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:51:59.497 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:51:59.518 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:51:59.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:59.519 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:51:59.614 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:59 smithi084 ceph-mon[29774]: pgmap v858: 158 pgs: 4 active+clean+snaptrim, 1 active+remapped+backfill_toofull, 153 active+clean; 508 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 21/1323 objects misplaced (1.587%) 2024-09-20T22:51:59.614 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:59 smithi084 ceph-mon[29774]: 4.1 scrub starts 2024-09-20T22:51:59.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:59 smithi084 ceph-mon[29774]: 4.1 scrub ok 2024-09-20T22:51:59.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:51:59 smithi084 ceph-mon[29774]: osdmap e639: 8 total, 8 up, 7 in 2024-09-20T22:51:59.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:59 smithi084 ceph-mon[40024]: pgmap v858: 158 pgs: 4 active+clean+snaptrim, 1 active+remapped+backfill_toofull, 153 active+clean; 508 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 21/1323 objects misplaced (1.587%) 2024-09-20T22:51:59.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:59 smithi084 ceph-mon[40024]: 4.1 scrub starts 2024-09-20T22:51:59.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:59 smithi084 ceph-mon[40024]: 4.1 scrub ok 2024-09-20T22:51:59.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:51:59 smithi084 ceph-mon[40024]: osdmap e639: 8 total, 8 up, 7 in 2024-09-20T22:51:59.619 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:51:59.645 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:59.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:59.646 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:51:59.746 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:51:59.754 INFO:journalctl@ceph.iscsi.iscsi.a.smithi154.stdout:Sep 20 22:51:59 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-iscsi-iscsi-a[59730]: debug there is no tcmu-runner data available 2024-09-20T22:51:59.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:59 smithi154 ceph-mon[33264]: pgmap v858: 158 pgs: 4 active+clean+snaptrim, 1 active+remapped+backfill_toofull, 153 active+clean; 508 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 21/1323 objects misplaced (1.587%) 2024-09-20T22:51:59.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:59 smithi154 ceph-mon[33264]: 4.1 scrub starts 2024-09-20T22:51:59.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:59 smithi154 ceph-mon[33264]: 4.1 scrub ok 2024-09-20T22:51:59.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:51:59 smithi154 ceph-mon[33264]: osdmap e639: 8 total, 8 up, 7 in 2024-09-20T22:51:59.772 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:59.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:59.773 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:51:59.873 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:51:59.898 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:51:59.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:51:59.899 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:51:59.915 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.3 dump_historic_ops 2024-09-20T22:52:00.000 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:52:00.024 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:00.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:00.025 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:52:00.039 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-20T22:52:00.126 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:52:00.150 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:00.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:00.151 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:52:00.166 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:52:00.252 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:52:00.271 INFO:tasks.rados.rados.0.smithi084.stdout:2521: done (2 left) 2024-09-20T22:52:00.271 INFO:tasks.rados.rados.0.smithi084.stdout:2522: done (1 left) 2024-09-20T22:52:00.271 INFO:tasks.rados.rados.0.smithi084.stdout:2523: done (0 left) 2024-09-20T22:52:00.271 INFO:tasks.rados.rados.0.smithi084.stdout:2524: snap_create 2024-09-20T22:52:00.272 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.3 to 1 (10000) 2024-09-20T22:52:00.276 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:00.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:00.277 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:52:00.378 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:52:00.400 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:00.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:00.401 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:52:00.501 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:52:00.525 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:00.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:00.526 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:52:00.617 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:00 smithi084 ceph-mon[29774]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:52:00.617 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:00 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/1305830256' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-09-20T22:52:00.617 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:00 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-09-20T22:52:00.617 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:00 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-20T22:52:00.617 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:00 smithi084 ceph-mon[29774]: osdmap e640: 8 total, 8 up, 7 in 2024-09-20T22:52:00.618 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:00 smithi084 ceph-mon[40024]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:52:00.618 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:00 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/1305830256' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-09-20T22:52:00.618 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:00 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-09-20T22:52:00.618 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:00 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-20T22:52:00.618 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:00 smithi084 ceph-mon[40024]: osdmap e640: 8 total, 8 up, 7 in 2024-09-20T22:52:00.626 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:52:00.650 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:00.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:00.651 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:52:00.751 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:52:00.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:00 smithi154 ceph-mon[33264]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:52:00.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:00 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/1305830256' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-09-20T22:52:00.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:00 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-09-20T22:52:00.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:00 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-20T22:52:00.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:00 smithi154 ceph-mon[33264]: osdmap e640: 8 total, 8 up, 7 in 2024-09-20T22:52:00.768 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd set noscrub 2024-09-20T22:52:00.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:00.774 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:00.774 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:52:00.875 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:52:00.900 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:00.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:00.908 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:52:01.009 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:52:01.033 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:01.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:01.034 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:52:01.099 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd reweight 5 1 2024-09-20T22:52:01.135 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:52:01.162 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:01.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:01.163 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:52:01.263 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:52:01.274 INFO:tasks.rados.rados.0.smithi084.stdout:2524: done (0 left) 2024-09-20T22:52:01.275 INFO:tasks.rados.rados.0.smithi084.stdout:2525: snap_create 2024-09-20T22:52:01.289 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:01.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:01.290 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:52:01.356 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-20T22:52:01.356 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2024-09-20T22:52:01.356 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2024-09-20T22:52:01.356 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [ 2024-09-20T22:52:01.356 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.356 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:554 7.a 7:5122292a:::smithi08477203-33:head [copy-from ver 6] snapc 13=[13,12,11,10,f,d] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-09-20T22:52:01.356 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:43:27.336824+0000", 2024-09-20T22:52:01.356 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 514.01595044099997, 2024-09-20T22:52:01.356 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.056403397000000001, 2024-09-20T22:52:01.357 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:52:01.357 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:52:01.357 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:52:01.357 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:52:01.357 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:52:01.357 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:52:01.357 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 554 2024-09-20T22:52:01.357 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.357 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:52:01.357 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.357 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:52:01.357 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:27.336824+0000", 2024-09-20T22:52:01.357 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:52:01.357 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.357 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.358 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:52:01.358 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:27.336824+0000", 2024-09-20T22:52:01.358 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:52:01.358 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.358 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.358 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:52:01.358 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:27.336826+0000", 2024-09-20T22:52:01.358 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.0339999999999999e-06 2024-09-20T22:52:01.358 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.358 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.358 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:52:01.358 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:27.336837+0000", 2024-09-20T22:52:01.358 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1243000000000001e-05 2024-09-20T22:52:01.358 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.359 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.359 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:52:01.359 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:27.336857+0000", 2024-09-20T22:52:01.359 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.9868999999999999e-05 2024-09-20T22:52:01.359 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.359 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.359 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:52:01.359 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:27.336869+0000", 2024-09-20T22:52:01.359 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1348999999999999e-05 2024-09-20T22:52:01.359 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.359 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.359 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:52:01.359 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:27.336946+0000", 2024-09-20T22:52:01.359 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.7683999999999997e-05 2024-09-20T22:52:01.359 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.360 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.360 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:52:01.360 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:27.337045+0000", 2024-09-20T22:52:01.360 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.9265999999999995e-05 2024-09-20T22:52:01.360 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.360 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.360 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 2,5", 2024-09-20T22:52:01.360 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:27.340603+0000", 2024-09-20T22:52:01.360 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.003557892 2024-09-20T22:52:01.360 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.360 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.360 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:52:01.360 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:27.343571+0000", 2024-09-20T22:52:01.360 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0029676609999999999 2024-09-20T22:52:01.360 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.361 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.361 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:52:01.361 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:27.346457+0000", 2024-09-20T22:52:01.361 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0028860040000000002 2024-09-20T22:52:01.361 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.361 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.361 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:52:01.361 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:27.393131+0000", 2024-09-20T22:52:01.361 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.046673813000000001 2024-09-20T22:52:01.361 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.361 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.361 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:52:01.361 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:27.393173+0000", 2024-09-20T22:52:01.361 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.2076999999999997e-05 2024-09-20T22:52:01.361 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.361 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.362 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:52:01.362 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:27.393227+0000", 2024-09-20T22:52:01.362 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.4505000000000001e-05 2024-09-20T22:52:01.362 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:52:01.362 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:52:01.362 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:52:01.362 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.362 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.362 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:663 7.12 7:4fa4b2b3:::smithi08477203-21:head [write 665577~781581,stat] snapc 17=[17,16,15,11,d] ondisk+write+known_if_redirected+supports_pool_eio e137)", 2024-09-20T22:52:01.362 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:43:36.161568+0000", 2024-09-20T22:52:01.362 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 505.19120626400002, 2024-09-20T22:52:01.363 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.99286350899999998, 2024-09-20T22:52:01.363 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:52:01.363 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:52:01.363 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:52:01.363 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:52:01.363 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:52:01.363 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:52:01.363 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 663 2024-09-20T22:52:01.363 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.363 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:52:01.363 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.363 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:52:01.363 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.161568+0000", 2024-09-20T22:52:01.363 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:52:01.364 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.364 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.364 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:52:01.364 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.161568+0000", 2024-09-20T22:52:01.364 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:52:01.364 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.364 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.364 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:52:01.364 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.161570+0000", 2024-09-20T22:52:01.364 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.0679999999999999e-06 2024-09-20T22:52:01.364 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.364 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.364 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:52:01.364 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.162310+0000", 2024-09-20T22:52:01.364 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00074005999999999996 2024-09-20T22:52:01.365 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.365 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.365 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:52:01.365 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.162349+0000", 2024-09-20T22:52:01.365 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.8778e-05 2024-09-20T22:52:01.365 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.365 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.365 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:52:01.365 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.162375+0000", 2024-09-20T22:52:01.365 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.6089e-05 2024-09-20T22:52:01.365 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.365 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.365 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:52:01.365 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.162434+0000", 2024-09-20T22:52:01.365 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.8764999999999997e-05 2024-09-20T22:52:01.366 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.366 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.366 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-09-20T22:52:01.366 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.162443+0000", 2024-09-20T22:52:01.366 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.054e-06 2024-09-20T22:52:01.366 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.366 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.366 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:52:01.366 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.143416+0000", 2024-09-20T22:52:01.366 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.98097291600000003 2024-09-20T22:52:01.366 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.366 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.366 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:52:01.366 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.143522+0000", 2024-09-20T22:52:01.367 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00010626 2024-09-20T22:52:01.367 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.367 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.367 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 1,2", 2024-09-20T22:52:01.367 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.143654+0000", 2024-09-20T22:52:01.367 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013218799999999999 2024-09-20T22:52:01.367 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.367 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.367 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:52:01.367 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.148086+0000", 2024-09-20T22:52:01.367 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0044314189999999998 2024-09-20T22:52:01.367 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.367 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.367 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:52:01.367 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.153546+0000", 2024-09-20T22:52:01.368 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0054599920000000003 2024-09-20T22:52:01.368 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.368 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.368 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:52:01.368 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.154317+0000", 2024-09-20T22:52:01.368 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00077125999999999996 2024-09-20T22:52:01.368 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.368 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.368 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:52:01.368 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.154380+0000", 2024-09-20T22:52:01.368 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.2526000000000004e-05 2024-09-20T22:52:01.368 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.368 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.368 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:52:01.369 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.154432+0000", 2024-09-20T22:52:01.369 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.2133999999999999e-05 2024-09-20T22:52:01.369 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:52:01.369 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:52:01.369 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:52:01.369 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.369 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.369 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:664 7.12 7:4fa4b2b3:::smithi08477203-21:head [write 2062560~563688] snapc 17=[17,16,15,11,d] ondisk+write+known_if_redirected+supports_pool_eio e137)", 2024-09-20T22:52:01.369 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:43:36.167947+0000", 2024-09-20T22:52:01.369 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 505.184827794, 2024-09-20T22:52:01.369 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.98665172599999995, 2024-09-20T22:52:01.369 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:52:01.369 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:52:01.369 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:52:01.369 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:52:01.370 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:52:01.370 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:52:01.370 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 664 2024-09-20T22:52:01.370 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.370 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:52:01.370 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.370 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:52:01.370 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.167947+0000", 2024-09-20T22:52:01.370 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:52:01.370 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.370 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.370 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:52:01.370 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.167947+0000", 2024-09-20T22:52:01.370 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:52:01.370 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.371 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.371 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:52:01.371 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.167948+0000", 2024-09-20T22:52:01.371 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.187e-06 2024-09-20T22:52:01.371 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.371 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.371 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:52:01.371 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.168193+0000", 2024-09-20T22:52:01.371 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00024496199999999999 2024-09-20T22:52:01.371 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.371 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.371 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:52:01.371 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.168203+0000", 2024-09-20T22:52:01.371 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0543999999999999e-05 2024-09-20T22:52:01.372 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.372 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.372 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:52:01.372 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.168209+0000", 2024-09-20T22:52:01.372 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.2839999999999997e-06 2024-09-20T22:52:01.372 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.372 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.372 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:52:01.372 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.168252+0000", 2024-09-20T22:52:01.372 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.3115000000000003e-05 2024-09-20T22:52:01.372 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.372 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.372 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-09-20T22:52:01.372 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.168257+0000", 2024-09-20T22:52:01.372 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.6860000000000003e-06 2024-09-20T22:52:01.373 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.373 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.373 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:52:01.373 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.144795+0000", 2024-09-20T22:52:01.373 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.97653758099999999 2024-09-20T22:52:01.373 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.373 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.373 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:52:01.373 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.144865+0000", 2024-09-20T22:52:01.373 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.9641000000000006e-05 2024-09-20T22:52:01.373 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.373 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.373 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 1,2", 2024-09-20T22:52:01.373 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.144986+0000", 2024-09-20T22:52:01.373 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000121248 2024-09-20T22:52:01.374 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.374 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.374 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:52:01.374 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.148091+0000", 2024-09-20T22:52:01.374 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0031050399999999999 2024-09-20T22:52:01.374 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.374 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.374 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:52:01.374 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.153603+0000", 2024-09-20T22:52:01.374 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0055118479999999997 2024-09-20T22:52:01.374 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.374 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.374 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:52:01.374 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.154508+0000", 2024-09-20T22:52:01.374 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00090530600000000004 2024-09-20T22:52:01.375 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.375 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.375 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:52:01.375 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.154560+0000", 2024-09-20T22:52:01.375 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.1474000000000002e-05 2024-09-20T22:52:01.375 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.375 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.375 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:52:01.375 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.154598+0000", 2024-09-20T22:52:01.375 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.8810000000000003e-05 2024-09-20T22:52:01.375 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:52:01.375 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:52:01.375 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:52:01.375 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.375 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.376 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:665 7.12 7:4fa4b2b3:::smithi08477203-21:head [write 3182305~643664] snapc 17=[17,16,15,11,d] ondisk+write+known_if_redirected+supports_pool_eio e137)", 2024-09-20T22:52:01.376 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:43:36.174520+0000", 2024-09-20T22:52:01.376 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 505.17825470399998, 2024-09-20T22:52:01.376 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.98020083400000002, 2024-09-20T22:52:01.376 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:52:01.376 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:52:01.376 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:52:01.376 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:52:01.376 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:52:01.376 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:52:01.376 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 665 2024-09-20T22:52:01.376 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.376 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:52:01.376 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.377 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:52:01.377 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174520+0000", 2024-09-20T22:52:01.377 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:52:01.377 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.377 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.377 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:52:01.377 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174520+0000", 2024-09-20T22:52:01.377 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:52:01.377 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.377 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.377 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:52:01.377 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174521+0000", 2024-09-20T22:52:01.377 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.234e-06 2024-09-20T22:52:01.377 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.377 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.378 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:52:01.378 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174816+0000", 2024-09-20T22:52:01.378 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000294682 2024-09-20T22:52:01.378 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.378 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.378 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:52:01.378 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174827+0000", 2024-09-20T22:52:01.378 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1025e-05 2024-09-20T22:52:01.378 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.378 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.378 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:52:01.378 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174832+0000", 2024-09-20T22:52:01.378 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.3990000000000003e-06 2024-09-20T22:52:01.378 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.378 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.379 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:52:01.379 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174868+0000", 2024-09-20T22:52:01.379 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.6225e-05 2024-09-20T22:52:01.379 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.379 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.379 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-09-20T22:52:01.379 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174873+0000", 2024-09-20T22:52:01.379 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.848e-06 2024-09-20T22:52:01.379 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.379 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.379 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:52:01.379 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.145844+0000", 2024-09-20T22:52:01.379 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.97097048900000005 2024-09-20T22:52:01.379 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.380 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.380 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:52:01.380 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.145917+0000", 2024-09-20T22:52:01.380 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.3248000000000001e-05 2024-09-20T22:52:01.380 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.380 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.380 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 1,2", 2024-09-20T22:52:01.380 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.146028+0000", 2024-09-20T22:52:01.380 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000110773 2024-09-20T22:52:01.380 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.380 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.380 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:52:01.380 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.148095+0000", 2024-09-20T22:52:01.380 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0020678179999999999 2024-09-20T22:52:01.380 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.380 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.381 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:52:01.381 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.153967+0000", 2024-09-20T22:52:01.381 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0058717350000000003 2024-09-20T22:52:01.381 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.381 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.381 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:52:01.381 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.154633+0000", 2024-09-20T22:52:01.381 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00066600600000000002 2024-09-20T22:52:01.381 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.381 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.381 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:52:01.381 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.154683+0000", 2024-09-20T22:52:01.381 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.9872e-05 2024-09-20T22:52:01.381 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.381 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.382 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:52:01.382 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.154721+0000", 2024-09-20T22:52:01.382 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.748e-05 2024-09-20T22:52:01.382 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:52:01.382 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:52:01.382 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:52:01.382 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.382 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.383 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:666 7.12 7:4fa4b2b3:::smithi08477203-21:head [setxattr _header (57) in=7b,truncate 3865577] snapc 17=[17,16,15,11,d] ondisk+write+known_if_redirected+supports_pool_eio e137)", 2024-09-20T22:52:01.383 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:43:36.174844+0000", 2024-09-20T22:52:01.383 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 505.17793050500001, 2024-09-20T22:52:01.383 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.98002428600000002, 2024-09-20T22:52:01.383 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:52:01.383 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:52:01.383 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:52:01.383 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:52:01.383 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:52:01.384 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:52:01.384 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 666 2024-09-20T22:52:01.384 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.384 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:52:01.384 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.384 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:52:01.384 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174844+0000", 2024-09-20T22:52:01.384 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:52:01.384 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.384 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.384 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:52:01.384 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174844+0000", 2024-09-20T22:52:01.384 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:52:01.384 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.384 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.385 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:52:01.385 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174845+0000", 2024-09-20T22:52:01.385 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.1999999999999999e-07 2024-09-20T22:52:01.385 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.385 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.385 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:52:01.385 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174853+0000", 2024-09-20T22:52:01.385 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.1950000000000003e-06 2024-09-20T22:52:01.385 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.385 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.385 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:52:01.385 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174858+0000", 2024-09-20T22:52:01.385 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.5650000000000001e-06 2024-09-20T22:52:01.385 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.385 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.386 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:52:01.386 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174863+0000", 2024-09-20T22:52:01.386 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.1649999999999997e-06 2024-09-20T22:52:01.386 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.386 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.386 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:52:01.386 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174915+0000", 2024-09-20T22:52:01.386 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.1045999999999999e-05 2024-09-20T22:52:01.386 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.386 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.386 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-09-20T22:52:01.386 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174921+0000", 2024-09-20T22:52:01.386 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.404e-06 2024-09-20T22:52:01.386 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.387 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.387 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:52:01.387 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.146528+0000", 2024-09-20T22:52:01.387 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.97160681199999999 2024-09-20T22:52:01.387 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.387 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.387 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:52:01.387 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.146881+0000", 2024-09-20T22:52:01.387 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00035348200000000001 2024-09-20T22:52:01.387 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.387 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.387 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 1,2", 2024-09-20T22:52:01.387 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.147011+0000", 2024-09-20T22:52:01.387 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013018900000000001 2024-09-20T22:52:01.388 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.388 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.388 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:52:01.388 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.148100+0000", 2024-09-20T22:52:01.388 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010881580000000001 2024-09-20T22:52:01.388 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.388 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.388 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:52:01.388 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.154278+0000", 2024-09-20T22:52:01.388 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0061781730000000003 2024-09-20T22:52:01.388 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.388 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.388 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:52:01.388 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.154764+0000", 2024-09-20T22:52:01.388 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00048589199999999998 2024-09-20T22:52:01.389 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.389 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.389 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:52:01.389 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.154815+0000", 2024-09-20T22:52:01.389 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.1428000000000003e-05 2024-09-20T22:52:01.389 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.389 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.389 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:52:01.389 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.154868+0000", 2024-09-20T22:52:01.389 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.3156999999999999e-05 2024-09-20T22:52:01.389 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:52:01.389 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:52:01.389 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:52:01.389 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.389 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.390 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:667 7.12 7:4fa4b2b3:::smithi08477203-21:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e137)", 2024-09-20T22:52:01.390 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:43:36.174872+0000", 2024-09-20T22:52:01.390 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 505.17790281200001, 2024-09-20T22:52:01.390 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.98024790299999998, 2024-09-20T22:52:01.390 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:52:01.390 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:52:01.390 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-20T22:52:01.390 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:52:01.390 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:52:01.390 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:52:01.390 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 667 2024-09-20T22:52:01.390 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.390 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:52:01.390 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.391 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:52:01.391 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174872+0000", 2024-09-20T22:52:01.391 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:52:01.391 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.391 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.391 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:52:01.391 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174872+0000", 2024-09-20T22:52:01.391 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:52:01.391 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.391 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.391 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:52:01.391 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174872+0000", 2024-09-20T22:52:01.391 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.2199999999999999e-07 2024-09-20T22:52:01.391 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.392 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.392 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:52:01.392 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174874+0000", 2024-09-20T22:52:01.392 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.187e-06 2024-09-20T22:52:01.392 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.392 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.392 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:52:01.392 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174881+0000", 2024-09-20T22:52:01.392 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.674e-06 2024-09-20T22:52:01.392 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.392 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.392 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:52:01.392 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174886+0000", 2024-09-20T22:52:01.392 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.2240000000000003e-06 2024-09-20T22:52:01.392 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.393 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.393 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:52:01.393 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174936+0000", 2024-09-20T22:52:01.393 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.9824999999999998e-05 2024-09-20T22:52:01.393 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.393 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.393 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-09-20T22:52:01.393 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174940+0000", 2024-09-20T22:52:01.393 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.5700000000000001e-06 2024-09-20T22:52:01.393 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.393 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.393 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:52:01.393 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.147523+0000", 2024-09-20T22:52:01.393 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.97258315399999995 2024-09-20T22:52:01.393 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.394 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.394 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for rw locks", 2024-09-20T22:52:01.394 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.147591+0000", 2024-09-20T22:52:01.394 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.7878000000000003e-05 2024-09-20T22:52:01.394 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.394 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.394 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:52:01.394 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.154925+0000", 2024-09-20T22:52:01.394 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0073344430000000004 2024-09-20T22:52:01.394 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.394 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.394 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:52:01.394 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.155020+0000", 2024-09-20T22:52:01.394 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.4767999999999998e-05 2024-09-20T22:52:01.395 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.395 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.395 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:52:01.395 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.155120+0000", 2024-09-20T22:52:01.395 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.9758000000000003e-05 2024-09-20T22:52:01.395 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:52:01.395 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:52:01.395 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:52:01.395 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.395 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.395 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.14478.0:1259 5.3 5:c609908c:::notify.5:head [watch reconnect cookie 94843898679808 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e169)", 2024-09-20T22:52:01.395 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:44:08.230723+0000", 2024-09-20T22:52:01.395 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 473.12205150699998, 2024-09-20T22:52:01.395 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.94983577399999997, 2024-09-20T22:52:01.395 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:52:01.396 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:52:01.396 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-20T22:52:01.396 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:52:01.396 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.14478", 2024-09-20T22:52:01.396 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/2994273271", 2024-09-20T22:52:01.396 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1259 2024-09-20T22:52:01.396 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.396 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:52:01.396 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.396 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:52:01.396 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:08.230723+0000", 2024-09-20T22:52:01.396 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:52:01.396 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.396 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.397 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:52:01.397 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:08.230723+0000", 2024-09-20T22:52:01.397 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:52:01.397 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.397 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.397 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:52:01.397 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:08.230724+0000", 2024-09-20T22:52:01.397 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.077e-06 2024-09-20T22:52:01.397 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.397 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.397 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:52:01.397 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:08.230729+0000", 2024-09-20T22:52:01.397 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.9930000000000003e-06 2024-09-20T22:52:01.397 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.398 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.398 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:52:01.398 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:08.230741+0000", 2024-09-20T22:52:01.398 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1897e-05 2024-09-20T22:52:01.398 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.398 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.398 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:52:01.398 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:08.230747+0000", 2024-09-20T22:52:01.398 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.9830000000000001e-06 2024-09-20T22:52:01.398 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.398 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.398 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:52:01.398 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:08.230802+0000", 2024-09-20T22:52:01.398 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.4657000000000002e-05 2024-09-20T22:52:01.398 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.399 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.399 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-09-20T22:52:01.399 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:08.230811+0000", 2024-09-20T22:52:01.399 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.1160000000000005e-06 2024-09-20T22:52:01.399 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.399 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.399 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:52:01.399 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:09.171968+0000", 2024-09-20T22:52:01.399 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.94115747299999997 2024-09-20T22:52:01.399 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.399 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.399 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for missing object", 2024-09-20T22:52:01.399 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:09.172098+0000", 2024-09-20T22:52:01.399 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00012937499999999999 2024-09-20T22:52:01.399 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.400 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.400 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:52:01.400 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:09.174248+0000", 2024-09-20T22:52:01.400 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0021508959999999998 2024-09-20T22:52:01.400 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.400 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.400 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for rw locks", 2024-09-20T22:52:01.400 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:09.174341+0000", 2024-09-20T22:52:01.400 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.2906999999999997e-05 2024-09-20T22:52:01.400 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.400 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.400 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:52:01.400 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:09.175083+0000", 2024-09-20T22:52:01.400 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00074120100000000003 2024-09-20T22:52:01.401 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.401 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.401 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:52:01.401 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:09.175149+0000", 2024-09-20T22:52:01.401 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.6877000000000004e-05 2024-09-20T22:52:01.401 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.401 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.401 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:52:01.401 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:09.180559+0000", 2024-09-20T22:52:01.401 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0054093220000000003 2024-09-20T22:52:01.401 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:52:01.401 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:52:01.401 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:52:01.401 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.401 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.402 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:1419 7.1 7:84a866f5:::smithi08477203-27:head [append 0~0] snapc 2f=[2f] ondisk+write+known_if_redirected+supports_pool_eio e189)", 2024-09-20T22:52:01.402 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:44:28.138055+0000", 2024-09-20T22:52:01.402 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 453.21471986799997, 2024-09-20T22:52:01.402 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.030290338999999999, 2024-09-20T22:52:01.402 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:52:01.402 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:52:01.402 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:52:01.402 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:52:01.402 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:52:01.402 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:52:01.402 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1419 2024-09-20T22:52:01.402 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.402 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:52:01.402 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.403 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:52:01.403 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.138055+0000", 2024-09-20T22:52:01.403 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:52:01.403 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.403 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.403 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:52:01.403 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.138055+0000", 2024-09-20T22:52:01.403 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:52:01.403 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.403 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.403 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:52:01.403 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.138056+0000", 2024-09-20T22:52:01.403 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.13e-06 2024-09-20T22:52:01.403 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.404 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.404 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:52:01.404 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.138060+0000", 2024-09-20T22:52:01.404 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.0269999999999997e-06 2024-09-20T22:52:01.404 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.404 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.404 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:52:01.404 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.138070+0000", 2024-09-20T22:52:01.404 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.961e-06 2024-09-20T22:52:01.404 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.404 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.404 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:52:01.404 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.138076+0000", 2024-09-20T22:52:01.404 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.1800000000000001e-06 2024-09-20T22:52:01.404 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.405 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.405 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:52:01.405 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.138137+0000", 2024-09-20T22:52:01.405 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.1159000000000002e-05 2024-09-20T22:52:01.405 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.405 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.405 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting_for_map not empty", 2024-09-20T22:52:01.405 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.138141+0000", 2024-09-20T22:52:01.405 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.189e-06 2024-09-20T22:52:01.405 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.405 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.405 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:52:01.405 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.154982+0000", 2024-09-20T22:52:01.405 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.016840738000000001 2024-09-20T22:52:01.406 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.406 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.406 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:52:01.406 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.155089+0000", 2024-09-20T22:52:01.406 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00010653399999999999 2024-09-20T22:52:01.406 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.406 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.406 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 1,6", 2024-09-20T22:52:01.406 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.155320+0000", 2024-09-20T22:52:01.406 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00023196299999999999 2024-09-20T22:52:01.406 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.406 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.406 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:52:01.406 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.158224+0000", 2024-09-20T22:52:01.406 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00290361 2024-09-20T22:52:01.407 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.407 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.407 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:52:01.407 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.167686+0000", 2024-09-20T22:52:01.407 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0094622360000000006 2024-09-20T22:52:01.407 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.407 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.407 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:52:01.407 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.168183+0000", 2024-09-20T22:52:01.407 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00049655999999999995 2024-09-20T22:52:01.407 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.407 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.407 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:52:01.407 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.168273+0000", 2024-09-20T22:52:01.408 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.9758000000000004e-05 2024-09-20T22:52:01.408 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.408 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.408 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:52:01.408 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.168345+0000", 2024-09-20T22:52:01.408 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.2293999999999996e-05 2024-09-20T22:52:01.408 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:52:01.408 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:52:01.408 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:52:01.408 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.408 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.408 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:2347 7.a 7:5122292a:::smithi08477203-33:head [sparse-read 0~2590699,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e260)", 2024-09-20T22:52:01.408 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:45:39.196440+0000", 2024-09-20T22:52:01.408 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 382.15633473999998, 2024-09-20T22:52:01.409 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.026701163, 2024-09-20T22:52:01.409 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:52:01.409 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:52:01.409 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-20T22:52:01.409 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:52:01.409 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:52:01.409 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:52:01.409 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 2347 2024-09-20T22:52:01.409 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.409 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:52:01.409 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.409 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:52:01.409 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.196440+0000", 2024-09-20T22:52:01.409 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:52:01.410 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.410 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.410 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:52:01.410 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.196440+0000", 2024-09-20T22:52:01.410 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:52:01.410 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.410 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.410 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:52:01.410 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.196441+0000", 2024-09-20T22:52:01.410 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.714e-06 2024-09-20T22:52:01.410 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.410 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.410 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:52:01.411 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.196449+0000", 2024-09-20T22:52:01.411 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.8730000000000007e-06 2024-09-20T22:52:01.411 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.411 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.411 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:52:01.411 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.196462+0000", 2024-09-20T22:52:01.411 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2649e-05 2024-09-20T22:52:01.411 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.411 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.411 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:52:01.411 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.196470+0000", 2024-09-20T22:52:01.411 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.4989999999999994e-06 2024-09-20T22:52:01.411 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.411 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.411 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:52:01.412 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.212172+0000", 2024-09-20T22:52:01.412 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.015701335 2024-09-20T22:52:01.412 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.412 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.412 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:52:01.412 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.212270+0000", 2024-09-20T22:52:01.412 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.8257999999999993e-05 2024-09-20T22:52:01.412 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.412 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.412 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:52:01.412 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.223141+0000", 2024-09-20T22:52:01.412 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010870835000000001 2024-09-20T22:52:01.412 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:52:01.412 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:52:01.412 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:52:01.413 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.413 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.413 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:2348 7.a 7:5122292a:::smithi08477203-33:head [read 0~3848803,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e260)", 2024-09-20T22:52:01.413 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:45:39.196485+0000", 2024-09-20T22:52:01.413 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 382.15628926699998, 2024-09-20T22:52:01.413 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0284703, 2024-09-20T22:52:01.413 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:52:01.413 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:52:01.413 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-20T22:52:01.413 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:52:01.413 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:52:01.413 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:52:01.413 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 2348 2024-09-20T22:52:01.413 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.414 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:52:01.414 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.414 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:52:01.414 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.196485+0000", 2024-09-20T22:52:01.414 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:52:01.414 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.414 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.414 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:52:01.414 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.196485+0000", 2024-09-20T22:52:01.414 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:52:01.414 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.414 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.414 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:52:01.414 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.196486+0000", 2024-09-20T22:52:01.415 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.9599999999999999e-07 2024-09-20T22:52:01.415 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.415 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.415 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:52:01.415 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.196489+0000", 2024-09-20T22:52:01.415 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.5470000000000001e-06 2024-09-20T22:52:01.415 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.415 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.415 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:52:01.415 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.196496+0000", 2024-09-20T22:52:01.415 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.3570000000000003e-06 2024-09-20T22:52:01.415 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.415 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.415 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:52:01.415 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.196501+0000", 2024-09-20T22:52:01.416 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.6239999999999999e-06 2024-09-20T22:52:01.416 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.416 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.416 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:52:01.416 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.223176+0000", 2024-09-20T22:52:01.416 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.026675285999999999 2024-09-20T22:52:01.416 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.416 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.416 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:52:01.416 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.223274+0000", 2024-09-20T22:52:01.416 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.7009999999999994e-05 2024-09-20T22:52:01.416 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.416 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.417 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:52:01.417 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.224955+0000", 2024-09-20T22:52:01.417 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00168198 2024-09-20T22:52:01.417 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:52:01.417 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:52:01.417 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:52:01.417 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.417 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.417 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:2349 7.a 7:5122292a:::smithi08477203-33:head [sparse-read 0~2590699] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e260)", 2024-09-20T22:52:01.417 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:45:39.196508+0000", 2024-09-20T22:52:01.417 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 382.15626598099999, 2024-09-20T22:52:01.417 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.028731293000000001, 2024-09-20T22:52:01.417 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:52:01.417 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:52:01.417 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-20T22:52:01.418 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:52:01.418 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:52:01.418 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:52:01.418 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 2349 2024-09-20T22:52:01.418 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.418 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:52:01.418 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.418 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:52:01.418 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.196508+0000", 2024-09-20T22:52:01.418 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:52:01.418 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.418 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.418 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:52:01.418 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.196508+0000", 2024-09-20T22:52:01.419 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:52:01.419 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.419 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.419 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:52:01.419 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.196509+0000", 2024-09-20T22:52:01.419 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.08e-07 2024-09-20T22:52:01.419 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.419 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.419 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:52:01.419 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.196511+0000", 2024-09-20T22:52:01.419 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.142e-06 2024-09-20T22:52:01.419 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.419 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.419 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:52:01.420 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.196517+0000", 2024-09-20T22:52:01.420 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.6330000000000001e-06 2024-09-20T22:52:01.420 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.420 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.420 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:52:01.420 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.196522+0000", 2024-09-20T22:52:01.420 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.3530000000000002e-06 2024-09-20T22:52:01.420 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.420 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.420 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:52:01.420 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.224990+0000", 2024-09-20T22:52:01.420 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.028467864999999998 2024-09-20T22:52:01.420 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.420 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.420 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:52:01.421 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.225088+0000", 2024-09-20T22:52:01.421 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.8281000000000003e-05 2024-09-20T22:52:01.421 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.421 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.421 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:52:01.421 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.225240+0000", 2024-09-20T22:52:01.421 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00015161100000000001 2024-09-20T22:52:01.421 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:52:01.421 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:52:01.421 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:52:01.421 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.421 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.421 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(osd.6.0:727 7.c e260/94 7:318c7512:::smithi08477203-44:4c v 260'428, mlcod=258'424)", 2024-09-20T22:52:01.421 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:45:39.202413+0000", 2024-09-20T22:52:01.421 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 382.15036144200002, 2024-09-20T22:52:01.421 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.022852897, 2024-09-20T22:52:01.422 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:52:01.422 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:52:01.422 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:52:01.422 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:52:01.422 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.422 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:52:01.422 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.202413+0000", 2024-09-20T22:52:01.422 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:52:01.422 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.422 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.422 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:52:01.422 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.202413+0000", 2024-09-20T22:52:01.422 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:52:01.422 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.422 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.423 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:52:01.423 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.202414+0000", 2024-09-20T22:52:01.423 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.9299999999999997e-07 2024-09-20T22:52:01.423 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.423 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.423 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:52:01.423 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.202421+0000", 2024-09-20T22:52:01.423 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.8109999999999997e-06 2024-09-20T22:52:01.423 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.423 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.423 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:52:01.423 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.202433+0000", 2024-09-20T22:52:01.423 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2411e-05 2024-09-20T22:52:01.424 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.424 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.424 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:52:01.424 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.202442+0000", 2024-09-20T22:52:01.424 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.7269999999999996e-06 2024-09-20T22:52:01.424 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.424 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.424 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:52:01.424 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.221764+0000", 2024-09-20T22:52:01.424 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.019322068000000001 2024-09-20T22:52:01.424 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.424 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.424 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:52:01.424 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.221786+0000", 2024-09-20T22:52:01.424 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.2167e-05 2024-09-20T22:52:01.425 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.425 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.425 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:52:01.425 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.225217+0000", 2024-09-20T22:52:01.425 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0034313709999999999 2024-09-20T22:52:01.425 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.425 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.425 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:52:01.425 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.225266+0000", 2024-09-20T22:52:01.425 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.8649000000000001e-05 2024-09-20T22:52:01.425 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:52:01.425 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:52:01.425 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:52:01.425 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.425 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.426 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:2525 7.1 7:8422f82d:::smithi08477203-22:head [copy-from ver 174] snapc 5d=[5d,5c,5a,50,47] ondisk+write+known_if_redirected+supports_pool_eio e278)", 2024-09-20T22:52:01.426 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:45:57.170348+0000", 2024-09-20T22:52:01.426 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 364.18242646800002, 2024-09-20T22:52:01.426 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.023333486000000001, 2024-09-20T22:52:01.426 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:52:01.426 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:52:01.426 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:52:01.426 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:52:01.426 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:52:01.426 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:52:01.426 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 2525 2024-09-20T22:52:01.426 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.426 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:52:01.426 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.426 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:52:01.427 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:57.170348+0000", 2024-09-20T22:52:01.427 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:52:01.427 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.427 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.427 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:52:01.427 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:57.170348+0000", 2024-09-20T22:52:01.427 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:52:01.427 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.427 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.427 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:52:01.427 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:57.170350+0000", 2024-09-20T22:52:01.427 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.1509999999999998e-06 2024-09-20T22:52:01.427 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.427 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.428 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:52:01.428 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:57.170361+0000", 2024-09-20T22:52:01.428 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0492999999999999e-05 2024-09-20T22:52:01.428 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.428 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.428 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:52:01.428 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:57.170381+0000", 2024-09-20T22:52:01.428 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.0509999999999998e-05 2024-09-20T22:52:01.428 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.428 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.428 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:52:01.428 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:57.170393+0000", 2024-09-20T22:52:01.428 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2009e-05 2024-09-20T22:52:01.428 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.428 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.429 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:52:01.429 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:57.170465+0000", 2024-09-20T22:52:01.429 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.1927000000000004e-05 2024-09-20T22:52:01.429 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.429 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.429 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op must wait for map", 2024-09-20T22:52:01.429 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:57.170473+0000", 2024-09-20T22:52:01.429 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.1890000000000007e-06 2024-09-20T22:52:01.429 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.429 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.429 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:52:01.429 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:57.172855+0000", 2024-09-20T22:52:01.429 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002382168 2024-09-20T22:52:01.429 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.430 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.430 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:52:01.430 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:57.172967+0000", 2024-09-20T22:52:01.430 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00011108299999999999 2024-09-20T22:52:01.430 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.430 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.430 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 1,6", 2024-09-20T22:52:01.430 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:57.181533+0000", 2024-09-20T22:52:01.430 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0085667499999999997 2024-09-20T22:52:01.430 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.430 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.430 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:52:01.430 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:57.191599+0000", 2024-09-20T22:52:01.430 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010065795000000001 2024-09-20T22:52:01.430 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.431 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.431 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:52:01.431 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:57.193469+0000", 2024-09-20T22:52:01.431 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0018695199999999999 2024-09-20T22:52:01.431 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.431 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.431 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:52:01.431 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:57.193550+0000", 2024-09-20T22:52:01.431 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.1891000000000006e-05 2024-09-20T22:52:01.431 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.431 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.431 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:52:01.431 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:57.193625+0000", 2024-09-20T22:52:01.431 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.4679999999999996e-05 2024-09-20T22:52:01.432 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.432 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.432 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:52:01.432 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:57.193681+0000", 2024-09-20T22:52:01.432 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.6320000000000003e-05 2024-09-20T22:52:01.432 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:52:01.432 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:52:01.432 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:52:01.432 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.432 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.432 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:2853 7.10 7:0f524219:::smithi08477203-50:head [copy-from ver 114] snapc 68=[68,67] ondisk+write+known_if_redirected+supports_pool_eio e303)", 2024-09-20T22:52:01.432 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:46:23.285507+0000", 2024-09-20T22:52:01.432 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 338.06726773899999, 2024-09-20T22:52:01.432 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.023440631, 2024-09-20T22:52:01.433 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:52:01.433 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:52:01.433 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:52:01.433 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:52:01.433 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:52:01.433 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:52:01.433 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 2853 2024-09-20T22:52:01.433 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.433 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:52:01.433 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.433 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:52:01.433 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.285507+0000", 2024-09-20T22:52:01.433 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:52:01.433 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.433 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.434 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:52:01.434 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.285507+0000", 2024-09-20T22:52:01.434 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:52:01.434 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.434 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.434 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:52:01.434 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.285508+0000", 2024-09-20T22:52:01.434 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.491e-06 2024-09-20T22:52:01.434 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.434 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.434 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:52:01.434 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.285514+0000", 2024-09-20T22:52:01.434 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.502e-06 2024-09-20T22:52:01.434 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.434 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.435 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:52:01.435 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.285526+0000", 2024-09-20T22:52:01.435 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2639e-05 2024-09-20T22:52:01.435 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.435 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.435 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:52:01.435 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.285535+0000", 2024-09-20T22:52:01.435 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.4689999999999997e-06 2024-09-20T22:52:01.435 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.435 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.435 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:52:01.437 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.285591+0000", 2024-09-20T22:52:01.437 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.6407999999999999e-05 2024-09-20T22:52:01.437 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.437 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.437 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:52:01.437 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.285681+0000", 2024-09-20T22:52:01.438 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.9845000000000005e-05 2024-09-20T22:52:01.438 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.438 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.438 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 1,2", 2024-09-20T22:52:01.438 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:52:01.439 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.293719+0000", 2024-09-20T22:52:01.439 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0080383950000000003 2024-09-20T22:52:01.439 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.439 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.439 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:52:01.439 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.307398+0000", 2024-09-20T22:52:01.439 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013678074 2024-09-20T22:52:01.439 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.439 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.440 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:52:01.440 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.308451+0000", 2024-09-20T22:52:01.440 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001052975 2024-09-20T22:52:01.440 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.440 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.440 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:52:01.440 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.308811+0000", 2024-09-20T22:52:01.440 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00036041399999999998 2024-09-20T22:52:01.440 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.440 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.440 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:52:01.440 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.308878+0000", 2024-09-20T22:52:01.440 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.7114999999999997e-05 2024-09-20T22:52:01.440 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.440 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.441 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:52:01.441 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.308947+0000", 2024-09-20T22:52:01.441 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.9303999999999999e-05 2024-09-20T22:52:01.441 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:52:01.441 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:52:01.441 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:52:01.441 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.441 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.441 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:3136 7.a 7:5122292a:::smithi08477203-33:head [copy-from ver 385] snapc 77=[77,76,75,74,73,70,6b] ondisk+write+known_if_redirected+supports_pool_eio e328)", 2024-09-20T22:52:01.441 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:46:48.210350+0000", 2024-09-20T22:52:01.441 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 313.142424608, 2024-09-20T22:52:01.441 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.030150626999999999, 2024-09-20T22:52:01.441 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:52:01.441 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:52:01.442 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:52:01.442 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:52:01.442 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:52:01.442 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:52:01.442 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 3136 2024-09-20T22:52:01.442 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.442 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:52:01.442 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.442 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:52:01.442 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.210350+0000", 2024-09-20T22:52:01.442 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:52:01.442 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.442 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.442 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:52:01.442 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.210350+0000", 2024-09-20T22:52:01.443 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:52:01.443 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.443 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.443 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:52:01.443 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.210352+0000", 2024-09-20T22:52:01.443 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.889e-06 2024-09-20T22:52:01.443 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.443 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.443 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:52:01.443 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.210360+0000", 2024-09-20T22:52:01.443 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.8890000000000002e-06 2024-09-20T22:52:01.443 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.443 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.443 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:52:01.444 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.210372+0000", 2024-09-20T22:52:01.444 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2662e-05 2024-09-20T22:52:01.444 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.444 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.444 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:52:01.444 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.210381+0000", 2024-09-20T22:52:01.444 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.8589999999999994e-06 2024-09-20T22:52:01.444 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.444 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.444 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:52:01.444 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.210470+0000", 2024-09-20T22:52:01.444 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.9034000000000002e-05 2024-09-20T22:52:01.444 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.444 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.445 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:52:01.445 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.210568+0000", 2024-09-20T22:52:01.445 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.7919999999999998e-05 2024-09-20T22:52:01.445 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.445 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.445 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 2,5", 2024-09-20T22:52:01.445 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.218491+0000", 2024-09-20T22:52:01.445 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0079229490000000003 2024-09-20T22:52:01.445 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.445 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.445 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:52:01.445 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.224053+0000", 2024-09-20T22:52:01.445 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0055624330000000003 2024-09-20T22:52:01.445 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.445 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.446 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:52:01.446 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.227143+0000", 2024-09-20T22:52:01.446 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0030896360000000002 2024-09-20T22:52:01.446 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.446 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.446 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:52:01.446 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.240355+0000", 2024-09-20T22:52:01.446 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013211624999999999 2024-09-20T22:52:01.446 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.446 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.446 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:52:01.446 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.240439+0000", 2024-09-20T22:52:01.446 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.4009000000000002e-05 2024-09-20T22:52:01.446 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.447 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.447 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:52:01.447 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.240500+0000", 2024-09-20T22:52:01.447 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.1722000000000003e-05 2024-09-20T22:52:01.447 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:52:01.447 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:52:01.447 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:52:01.447 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.447 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.447 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:3235 7.1 7:84a866f5:::smithi08477203-27:head [copy-from ver 114] snapc 7e=[7e,7d,7c,7b,7a,79,78,75,74,73,6b] ondisk+write+known_if_redirected+supports_pool_eio e338)", 2024-09-20T22:52:01.447 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:46:58.180892+0000", 2024-09-20T22:52:01.447 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 303.171882385, 2024-09-20T22:52:01.447 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.044574093000000002, 2024-09-20T22:52:01.447 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:52:01.447 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:52:01.448 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:52:01.448 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:52:01.448 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:52:01.448 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:52:01.448 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 3235 2024-09-20T22:52:01.448 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.448 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:52:01.448 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.448 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:52:01.448 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.180892+0000", 2024-09-20T22:52:01.448 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:52:01.448 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.448 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.448 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:52:01.449 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.180892+0000", 2024-09-20T22:52:01.449 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:52:01.449 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.449 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.449 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:52:01.449 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.180894+0000", 2024-09-20T22:52:01.449 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.1789999999999998e-06 2024-09-20T22:52:01.449 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.449 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.449 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:52:01.449 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.180915+0000", 2024-09-20T22:52:01.449 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.0859000000000001e-05 2024-09-20T22:52:01.449 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.449 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.449 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:52:01.450 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.180937+0000", 2024-09-20T22:52:01.450 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.2357999999999998e-05 2024-09-20T22:52:01.450 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.450 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.450 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:52:01.450 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.180947+0000", 2024-09-20T22:52:01.450 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.8090000000000004e-06 2024-09-20T22:52:01.450 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.450 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.450 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:52:01.450 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.181042+0000", 2024-09-20T22:52:01.450 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.4327999999999995e-05 2024-09-20T22:52:01.450 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.450 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.450 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op must wait for map", 2024-09-20T22:52:01.451 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.181049+0000", 2024-09-20T22:52:01.451 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.1640000000000004e-06 2024-09-20T22:52:01.451 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.451 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.451 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:52:01.451 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.185933+0000", 2024-09-20T22:52:01.451 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.004884645 2024-09-20T22:52:01.451 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.451 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.451 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:52:01.451 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.186177+0000", 2024-09-20T22:52:01.451 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000243107 2024-09-20T22:52:01.451 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.451 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.452 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 1,6", 2024-09-20T22:52:01.452 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.204218+0000", 2024-09-20T22:52:01.452 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.018041805000000001 2024-09-20T22:52:01.452 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.452 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.452 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:52:01.452 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:52:01.453 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.218275+0000", 2024-09-20T22:52:01.453 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014056528 2024-09-20T22:52:01.453 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.453 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.453 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:52:01.453 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.224283+0000", 2024-09-20T22:52:01.453 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0060084860000000004 2024-09-20T22:52:01.453 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.453 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.453 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:52:01.453 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.225379+0000", 2024-09-20T22:52:01.453 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010960830000000001 2024-09-20T22:52:01.453 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.454 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.454 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:52:01.454 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.225429+0000", 2024-09-20T22:52:01.454 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.9440000000000001e-05 2024-09-20T22:52:01.454 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.454 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.454 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:52:01.454 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.225466+0000", 2024-09-20T22:52:01.454 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.7302000000000003e-05 2024-09-20T22:52:01.454 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:52:01.454 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:52:01.454 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:52:01.454 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.454 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.455 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:3236 7.1 7:84a866f5:::smithi08477203-27:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e338)", 2024-09-20T22:52:01.455 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:46:58.180973+0000", 2024-09-20T22:52:01.455 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 303.17180176300002, 2024-09-20T22:52:01.455 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.028358540000000002, 2024-09-20T22:52:01.455 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:52:01.455 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:52:01.455 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-20T22:52:01.455 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:52:01.455 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:52:01.455 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:52:01.455 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 3236 2024-09-20T22:52:01.455 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.455 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:52:01.455 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.456 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:52:01.456 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.180973+0000", 2024-09-20T22:52:01.456 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:52:01.456 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.456 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.456 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:52:01.456 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.180973+0000", 2024-09-20T22:52:01.456 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:52:01.456 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.456 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.456 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:52:01.456 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.180973+0000", 2024-09-20T22:52:01.456 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.9100000000000004e-07 2024-09-20T22:52:01.456 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.456 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.457 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:52:01.457 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.180976+0000", 2024-09-20T22:52:01.457 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.118e-06 2024-09-20T22:52:01.457 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.457 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.457 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:52:01.457 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.180984+0000", 2024-09-20T22:52:01.457 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.43e-06 2024-09-20T22:52:01.457 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.457 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.457 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:52:01.457 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.180990+0000", 2024-09-20T22:52:01.457 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.1109999999999996e-06 2024-09-20T22:52:01.457 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.457 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.458 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:52:01.458 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.181102+0000", 2024-09-20T22:52:01.458 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000112193 2024-09-20T22:52:01.458 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.458 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.458 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting_for_map not empty", 2024-09-20T22:52:01.458 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.181107+0000", 2024-09-20T22:52:01.458 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.0490000000000003e-06 2024-09-20T22:52:01.458 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.458 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.458 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:52:01.458 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.186302+0000", 2024-09-20T22:52:01.458 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0051950240000000003 2024-09-20T22:52:01.458 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.458 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.459 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for blocked object", 2024-09-20T22:52:01.459 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.186344+0000", 2024-09-20T22:52:01.459 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.2168e-05 2024-09-20T22:52:01.459 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.459 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.459 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:52:01.459 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.209115+0000", 2024-09-20T22:52:01.459 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.022770291000000002 2024-09-20T22:52:01.459 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.459 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.459 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:52:01.459 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.209217+0000", 2024-09-20T22:52:01.459 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000102192 2024-09-20T22:52:01.459 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.460 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.460 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:52:01.460 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.209331+0000", 2024-09-20T22:52:01.460 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000114373 2024-09-20T22:52:01.460 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:52:01.460 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:52:01.460 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:52:01.460 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.460 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.460 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:4534 7.12 7:4b06cbed:::smithi08477203-49:head [copy-from ver 722] snapc bb=[bb,ba,b9,b8,b7,b5,b4,b2,b1,a7,a6,a5,9e,9c,95,87,86] ondisk+write+known_if_redirected+supports_pool_eio e456)", 2024-09-20T22:52:01.460 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:48:56.230481+0000", 2024-09-20T22:52:01.460 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 185.12229371699999, 2024-09-20T22:52:01.460 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.028199295999999999, 2024-09-20T22:52:01.460 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:52:01.461 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:52:01.461 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:52:01.461 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:52:01.461 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:52:01.461 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:52:01.461 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 4534 2024-09-20T22:52:01.461 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.461 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:52:01.461 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.461 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:52:01.461 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:56.230481+0000", 2024-09-20T22:52:01.461 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:52:01.461 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.461 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.462 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:52:01.462 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:56.230481+0000", 2024-09-20T22:52:01.462 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:52:01.462 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.462 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.462 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:52:01.462 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:56.230482+0000", 2024-09-20T22:52:01.462 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.674e-06 2024-09-20T22:52:01.462 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.462 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.462 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:52:01.462 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:56.230497+0000", 2024-09-20T22:52:01.462 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.4178e-05 2024-09-20T22:52:01.463 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.463 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.463 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:52:01.463 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:56.230517+0000", 2024-09-20T22:52:01.463 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.0336e-05 2024-09-20T22:52:01.463 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.463 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.463 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:52:01.463 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:56.230527+0000", 2024-09-20T22:52:01.463 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.6039999999999999e-06 2024-09-20T22:52:01.463 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.463 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.463 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:52:01.463 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:56.230631+0000", 2024-09-20T22:52:01.464 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00010445 2024-09-20T22:52:01.464 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.464 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.464 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op must wait for map", 2024-09-20T22:52:01.464 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:56.230639+0000", 2024-09-20T22:52:01.464 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.7759999999999997e-06 2024-09-20T22:52:01.464 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.464 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.464 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:52:01.464 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:56.237931+0000", 2024-09-20T22:52:01.464 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0072925630000000002 2024-09-20T22:52:01.464 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.464 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.464 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:52:01.465 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:56.238030+0000", 2024-09-20T22:52:01.465 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.8592999999999996e-05 2024-09-20T22:52:01.465 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.465 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.465 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 1,2", 2024-09-20T22:52:01.465 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:56.244691+0000", 2024-09-20T22:52:01.465 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0066614680000000003 2024-09-20T22:52:01.465 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.465 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.465 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:52:01.465 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:56.258211+0000", 2024-09-20T22:52:01.465 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013519721 2024-09-20T22:52:01.465 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.465 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.465 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:52:01.466 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:56.258404+0000", 2024-09-20T22:52:01.466 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00019257000000000001 2024-09-20T22:52:01.466 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.466 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.466 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:52:01.466 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:56.258595+0000", 2024-09-20T22:52:01.466 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000191387 2024-09-20T22:52:01.466 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.466 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.466 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:52:01.466 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:56.258635+0000", 2024-09-20T22:52:01.466 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.9918e-05 2024-09-20T22:52:01.466 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.466 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.466 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:52:01.467 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:56.258680+0000", 2024-09-20T22:52:01.467 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.5058000000000001e-05 2024-09-20T22:52:01.467 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:52:01.467 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:52:01.467 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:52:01.467 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.467 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.467 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:5677 7.2 7:4fa4b2b3:::smithi08477203-21:head [copy-from ver 1213] snapc fa=[fa,f9,f8,f7,f5,f4,f2,f1,ef,ed,ec,ea,e6,e4,e2,df,db,da,d7,d4,d3,ce,c5,c2,bf,b4,b2,a7] ondisk+write+known_if_redirected+supports_pool_eio e572)", 2024-09-20T22:52:01.467 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:50:52.384255+0000", 2024-09-20T22:52:01.467 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 68.968518969000002, 2024-09-20T22:52:01.467 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.027330337, 2024-09-20T22:52:01.467 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:52:01.467 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:52:01.467 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:52:01.468 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:52:01.468 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:52:01.468 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:52:01.468 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 5677 2024-09-20T22:52:01.468 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.468 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:52:01.468 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.468 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:52:01.468 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:52.384255+0000", 2024-09-20T22:52:01.468 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:52:01.468 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.468 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.468 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:52:01.468 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:52.384255+0000", 2024-09-20T22:52:01.468 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:52:01.469 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.469 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.469 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:52:01.469 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:52.384258+0000", 2024-09-20T22:52:01.469 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.0600000000000002e-06 2024-09-20T22:52:01.469 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.469 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.469 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:52:01.469 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:52.384267+0000", 2024-09-20T22:52:01.469 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.3519999999999996e-06 2024-09-20T22:52:01.469 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.469 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.469 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:52:01.469 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:52.384288+0000", 2024-09-20T22:52:01.469 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.1039e-05 2024-09-20T22:52:01.470 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.470 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.470 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:52:01.470 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:52.384298+0000", 2024-09-20T22:52:01.470 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.6630000000000005e-06 2024-09-20T22:52:01.470 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.470 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.470 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:52:01.470 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:52.384381+0000", 2024-09-20T22:52:01.470 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.3372000000000001e-05 2024-09-20T22:52:01.470 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.470 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.470 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:52:01.470 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:52.384455+0000", 2024-09-20T22:52:01.470 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.4414999999999998e-05 2024-09-20T22:52:01.471 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.471 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.471 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 1,2", 2024-09-20T22:52:01.471 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:52.398437+0000", 2024-09-20T22:52:01.471 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013981959 2024-09-20T22:52:01.471 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.471 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.471 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:52:01.471 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:52.410532+0000", 2024-09-20T22:52:01.471 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.012094898 2024-09-20T22:52:01.471 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.471 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.471 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:52:01.471 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:52.411064+0000", 2024-09-20T22:52:01.471 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00053131999999999995 2024-09-20T22:52:01.472 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.472 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.472 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:52:01.472 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:52.411446+0000", 2024-09-20T22:52:01.472 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00038282300000000002 2024-09-20T22:52:01.472 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.472 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.472 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:52:01.472 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:52.411525+0000", 2024-09-20T22:52:01.472 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.8339999999999999e-05 2024-09-20T22:52:01.472 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.472 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.472 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:52:01.472 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:52.411586+0000", 2024-09-20T22:52:01.472 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.1095999999999999e-05 2024-09-20T22:52:01.472 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:52:01.473 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:52:01.473 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:52:01.473 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.473 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.473 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:6378 7.2 7:4d20232a:::smithi08477203-34:head [copy-from ver 1800] snapc 11a=[11a,119,118,117,116,115,114,113,112,111,110,10e,10c,107,105,104,103,fd,fb,fa,f8,f5,f2,ed,d7,c2,bf] ondisk+write+known_if_redirected+supports_pool_eio e639)", 2024-09-20T22:52:01.473 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:51:59.270245+0000", 2024-09-20T22:52:01.473 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 2.0825291039999998, 2024-09-20T22:52:01.473 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.024049938999999999, 2024-09-20T22:52:01.473 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:52:01.473 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:52:01.473 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:52:01.473 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:52:01.473 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:52:01.473 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:52:01.473 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 6378 2024-09-20T22:52:01.473 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.474 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:52:01.474 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.474 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:52:01.474 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:59.270245+0000", 2024-09-20T22:52:01.474 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:52:01.474 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.474 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.474 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:52:01.474 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:59.270245+0000", 2024-09-20T22:52:01.474 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:52:01.474 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.474 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.474 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:52:01.474 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:59.270248+0000", 2024-09-20T22:52:01.474 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.8159999999999998e-06 2024-09-20T22:52:01.475 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.475 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.475 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:52:01.475 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:59.270262+0000", 2024-09-20T22:52:01.475 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3429e-05 2024-09-20T22:52:01.475 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.475 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.475 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:52:01.475 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:59.270279+0000", 2024-09-20T22:52:01.475 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.7269e-05 2024-09-20T22:52:01.475 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.475 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.475 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:52:01.475 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:59.270288+0000", 2024-09-20T22:52:01.475 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.3610000000000007e-06 2024-09-20T22:52:01.476 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.476 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.476 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:52:01.476 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:59.270383+0000", 2024-09-20T22:52:01.476 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.5199999999999997e-05 2024-09-20T22:52:01.476 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.476 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.476 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op must wait for map", 2024-09-20T22:52:01.476 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:59.270393+0000", 2024-09-20T22:52:01.476 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.4190000000000008e-06 2024-09-20T22:52:01.476 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.476 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.476 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:52:01.476 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:59.272509+0000", 2024-09-20T22:52:01.476 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0021163839999999998 2024-09-20T22:52:01.476 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.477 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.477 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:52:01.477 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:59.272677+0000", 2024-09-20T22:52:01.477 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00016751 2024-09-20T22:52:01.477 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.477 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.477 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 1,2,7", 2024-09-20T22:52:01.477 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:59.278535+0000", 2024-09-20T22:52:01.477 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0058579849999999996 2024-09-20T22:52:01.477 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.477 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.477 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:52:01.477 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:59.285175+0000", 2024-09-20T22:52:01.477 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0066406829999999997 2024-09-20T22:52:01.477 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.477 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.478 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:52:01.478 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:59.293955+0000", 2024-09-20T22:52:01.478 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0087799990000000001 2024-09-20T22:52:01.478 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.478 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.478 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:52:01.478 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:59.294123+0000", 2024-09-20T22:52:01.478 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00016759699999999999 2024-09-20T22:52:01.478 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.478 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.478 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:52:01.478 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:59.294156+0000", 2024-09-20T22:52:01.478 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.2756000000000003e-05 2024-09-20T22:52:01.478 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.478 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.479 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:52:01.479 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:59.294197+0000", 2024-09-20T22:52:01.479 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.1712000000000003e-05 2024-09-20T22:52:01.479 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:01.479 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:01.479 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:52:01.479 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:59.294295+0000", 2024-09-20T22:52:01.479 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.7819000000000007e-05 2024-09-20T22:52:01.479 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:52:01.479 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:52:01.479 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:52:01.479 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:52:01.479 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:52:01.479 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-20T22:52:01.504 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:01.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:01.505 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:52:01.606 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:52:01.614 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:01 smithi084 ceph-mon[29774]: pgmap v861: 158 pgs: 5 active+clean+snaptrim, 1 active+remapped+backfill_toofull, 152 active+clean; 519 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 8.9 MiB/s rd, 4.3 MiB/s wr, 22 op/s; 21/1356 objects misplaced (1.549%) 2024-09-20T22:52:01.614 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:01 smithi084 ceph-mon[29774]: osdmap e641: 8 total, 8 up, 7 in 2024-09-20T22:52:01.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:01 smithi084 ceph-mon[40024]: pgmap v861: 158 pgs: 5 active+clean+snaptrim, 1 active+remapped+backfill_toofull, 152 active+clean; 519 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 8.9 MiB/s rd, 4.3 MiB/s wr, 22 op/s; 21/1356 objects misplaced (1.549%) 2024-09-20T22:52:01.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:01 smithi084 ceph-mon[40024]: osdmap e641: 8 total, 8 up, 7 in 2024-09-20T22:52:01.631 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:01.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:01.632 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:52:01.733 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:52:01.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:01 smithi154 ceph-mon[33264]: pgmap v861: 158 pgs: 5 active+clean+snaptrim, 1 active+remapped+backfill_toofull, 152 active+clean; 519 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 8.9 MiB/s rd, 4.3 MiB/s wr, 22 op/s; 21/1356 objects misplaced (1.549%) 2024-09-20T22:52:01.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:01 smithi154 ceph-mon[33264]: osdmap e641: 8 total, 8 up, 7 in 2024-09-20T22:52:01.757 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:01.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:01.758 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:52:01.859 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:52:01.883 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:01.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:01.884 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:52:01.985 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:52:02.006 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:02.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:02.007 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:52:02.107 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:52:02.128 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:02.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:02.129 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:52:02.229 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:52:02.255 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:02.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:02.256 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:52:02.275 INFO:tasks.rados.rados.0.smithi084.stdout:2525: done (0 left) 2024-09-20T22:52:02.275 INFO:tasks.rados.rados.0.smithi084.stdout:2526: write oid 19 current snap is 284 2024-09-20T22:52:02.276 INFO:tasks.rados.rados.0.smithi084.stdout:2526: seq_num 877 ranges {739739=573968,2053414=686325} 2024-09-20T22:52:02.282 INFO:tasks.rados.rados.0.smithi084.stdout:2526: writing smithi08477203-19 from 739739 to 1313707 tid 1 2024-09-20T22:52:02.291 INFO:tasks.rados.rados.0.smithi084.stdout:2526: writing smithi08477203-19 from 2053414 to 2739739 tid 2 2024-09-20T22:52:02.291 INFO:tasks.rados.rados.0.smithi084.stdout:2527: snap_remove snap 258 2024-09-20T22:52:02.356 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:52:02.376 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-20T22:52:02.377 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:02.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:02.378 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:52:02.479 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:52:02.501 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:02.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:02.502 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:52:02.602 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:52:02.614 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:02 smithi084 ceph-mon[40024]: purged_snaps scrub starts 2024-09-20T22:52:02.614 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:02 smithi084 ceph-mon[40024]: purged_snaps scrub ok 2024-09-20T22:52:02.614 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:02 smithi084 ceph-mon[40024]: osdmap e642: 8 total, 8 up, 7 in 2024-09-20T22:52:02.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:02 smithi084 ceph-mon[29774]: purged_snaps scrub starts 2024-09-20T22:52:02.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:02 smithi084 ceph-mon[29774]: purged_snaps scrub ok 2024-09-20T22:52:02.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:02 smithi084 ceph-mon[29774]: osdmap e642: 8 total, 8 up, 7 in 2024-09-20T22:52:02.624 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:02.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:02.625 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:52:02.634 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:52:02.698 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:52:02.726 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:52:02.748 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:02.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:02.749 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:52:02.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:02 smithi154 ceph-mon[33264]: purged_snaps scrub starts 2024-09-20T22:52:02.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:02 smithi154 ceph-mon[33264]: purged_snaps scrub ok 2024-09-20T22:52:02.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:02 smithi154 ceph-mon[33264]: osdmap e642: 8 total, 8 up, 7 in 2024-09-20T22:52:02.850 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:52:02.864 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:52:02.872 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:02.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:02.873 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:52:02.974 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:52:02.995 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:02.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:02.996 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:52:03.097 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:52:03.118 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:03.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:03.119 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:52:03.163 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-20T22:52:03.163 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [], 2024-09-20T22:52:03.163 INFO:teuthology.orchestra.run.smithi154.stdout: "num_ops": 0 2024-09-20T22:52:03.163 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-20T22:52:03.220 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:52:03.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:03.243 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:03.243 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:52:03.277 INFO:tasks.rados.rados.0.smithi084.stdout:2527: done (1 left) 2024-09-20T22:52:03.277 INFO:tasks.rados.rados.0.smithi084.stdout:2528: read oid 26 snap -1 2024-09-20T22:52:03.277 INFO:tasks.rados.rados.0.smithi084.stdout:2528: expect (ObjNum 868 snap 279 seq_num 868) 2024-09-20T22:52:03.277 INFO:tasks.rados.rados.0.smithi084.stdout:2529: read oid 18 snap 240 2024-09-20T22:52:03.277 INFO:tasks.rados.rados.0.smithi084.stdout:2529: expect (ObjNum 657 snap 199 seq_num 657) 2024-09-20T22:52:03.277 INFO:tasks.rados.rados.0.smithi084.stdout:2530: delete oid 44 current snap is 284 2024-09-20T22:52:03.277 INFO:tasks.rados.rados.0.smithi084.stdout:2526: finishing write tid 1 to smithi08477203-19 2024-09-20T22:52:03.277 INFO:tasks.rados.rados.0.smithi084.stdout:2526: finishing write tid 2 to smithi08477203-19 2024-09-20T22:52:03.277 INFO:tasks.rados.rados.0.smithi084.stdout:2526: finishing write tid 3 to smithi08477203-19 2024-09-20T22:52:03.277 INFO:tasks.rados.rados.0.smithi084.stdout:2526: finishing write tid 4 to smithi08477203-19 2024-09-20T22:52:03.277 INFO:tasks.rados.rados.0.smithi084.stdout:2526: oid 19 updating version 0 to 1884 2024-09-20T22:52:03.278 INFO:tasks.rados.rados.0.smithi084.stdout:2526: oid 19 updating version 1884 to 1885 2024-09-20T22:52:03.278 INFO:tasks.rados.rados.0.smithi084.stdout:2526: oid 19 version 1885 is already newer than 1883 2024-09-20T22:52:03.278 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 19 v 1885 (ObjNum 877 snap 284 seq_num 877) dirty exists 2024-09-20T22:52:03.278 INFO:tasks.rados.rados.0.smithi084.stdout:2526: left oid 19 (ObjNum 877 snap 284 seq_num 877) 2024-09-20T22:52:03.278 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is set 2024-09-20T22:52:03.297 INFO:tasks.rados.rados.0.smithi084.stdout:2529: expect (ObjNum 657 snap 199 seq_num 657) 2024-09-20T22:52:03.344 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:52:03.365 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:03.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:03.365 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:52:03.466 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:52:03.488 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:03.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:03.489 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:52:03.541 INFO:tasks.rados.rados.0.smithi084.stdout:2528: expect (ObjNum 868 snap 279 seq_num 868) 2024-09-20T22:52:03.590 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:52:03.607 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:03 smithi084 ceph-mon[29774]: 6.8 scrub starts 2024-09-20T22:52:03.607 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:03 smithi084 ceph-mon[29774]: 6.8 scrub ok 2024-09-20T22:52:03.607 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:03 smithi084 ceph-mon[29774]: pgmap v864: 158 pgs: 5 active+clean+snaptrim, 1 active+remapped+backfill_toofull, 152 active+clean; 519 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 8.9 MiB/s rd, 4.3 MiB/s wr, 22 op/s; 21/1356 objects misplaced (1.549%) 2024-09-20T22:52:03.607 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:03 smithi084 ceph-mon[29774]: 1.4 scrub starts 2024-09-20T22:52:03.607 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:03 smithi084 ceph-mon[29774]: 1.4 scrub ok 2024-09-20T22:52:03.607 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:03 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/1356463319' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-20T22:52:03.608 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:03 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-20T22:52:03.608 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:03 smithi084 ceph-mon[29774]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:52:03.608 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:03 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T22:52:03.608 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:03 smithi084 ceph-mon[29774]: osdmap e643: 8 total, 8 up, 7 in 2024-09-20T22:52:03.608 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:03 smithi084 ceph-mon[40024]: 6.8 scrub starts 2024-09-20T22:52:03.608 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:03 smithi084 ceph-mon[40024]: 6.8 scrub ok 2024-09-20T22:52:03.608 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:03 smithi084 ceph-mon[40024]: pgmap v864: 158 pgs: 5 active+clean+snaptrim, 1 active+remapped+backfill_toofull, 152 active+clean; 519 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 8.9 MiB/s rd, 4.3 MiB/s wr, 22 op/s; 21/1356 objects misplaced (1.549%) 2024-09-20T22:52:03.608 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:03 smithi084 ceph-mon[40024]: 1.4 scrub starts 2024-09-20T22:52:03.608 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:03 smithi084 ceph-mon[40024]: 1.4 scrub ok 2024-09-20T22:52:03.608 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:03 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/1356463319' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-20T22:52:03.608 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:03 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-20T22:52:03.609 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:03 smithi084 ceph-mon[40024]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:52:03.609 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:03 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T22:52:03.609 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:03 smithi084 ceph-mon[40024]: osdmap e643: 8 total, 8 up, 7 in 2024-09-20T22:52:03.612 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:03.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:03.613 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:52:03.713 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:52:03.724 INFO:tasks.rados.rados.0.smithi084.stdout:2526: done (3 left) 2024-09-20T22:52:03.724 INFO:tasks.rados.rados.0.smithi084.stdout:2528: done (2 left) 2024-09-20T22:52:03.724 INFO:tasks.rados.rados.0.smithi084.stdout:2529: done (1 left) 2024-09-20T22:52:03.724 INFO:tasks.rados.rados.0.smithi084.stdout:2530: done (0 left) 2024-09-20T22:52:03.724 INFO:tasks.rados.rados.0.smithi084.stdout:2531: read oid 6 snap 278 2024-09-20T22:52:03.724 INFO:tasks.rados.rados.0.smithi084.stdout:2531: expect (ObjNum 864 snap 278 seq_num 864) 2024-09-20T22:52:03.724 INFO:tasks.rados.rados.0.smithi084.stdout:2532: snap_create 2024-09-20T22:52:03.727 INFO:tasks.rados.rados.0.smithi084.stdout:2531: expect (ObjNum 864 snap 278 seq_num 864) 2024-09-20T22:52:03.734 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:03.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:03.735 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:52:03.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:03 smithi154 ceph-mon[33264]: 6.8 scrub starts 2024-09-20T22:52:03.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:03 smithi154 ceph-mon[33264]: 6.8 scrub ok 2024-09-20T22:52:03.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:03 smithi154 ceph-mon[33264]: pgmap v864: 158 pgs: 5 active+clean+snaptrim, 1 active+remapped+backfill_toofull, 152 active+clean; 519 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 8.9 MiB/s rd, 4.3 MiB/s wr, 22 op/s; 21/1356 objects misplaced (1.549%) 2024-09-20T22:52:03.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:03 smithi154 ceph-mon[33264]: 1.4 scrub starts 2024-09-20T22:52:03.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:03 smithi154 ceph-mon[33264]: 1.4 scrub ok 2024-09-20T22:52:03.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:03 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/1356463319' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-20T22:52:03.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:03 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-20T22:52:03.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:03 smithi154 ceph-mon[33264]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:52:03.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:03 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T22:52:03.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:03 smithi154 ceph-mon[33264]: osdmap e643: 8 total, 8 up, 7 in 2024-09-20T22:52:03.836 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:52:03.858 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:03.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:03.859 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:52:03.959 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:52:03.981 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:03.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:03.981 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:52:04.082 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:52:04.103 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:04.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:04.104 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:52:04.181 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.4 dump_blocked_ops 2024-09-20T22:52:04.205 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:52:04.226 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:04.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:04.227 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:52:04.328 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:52:04.350 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:04.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:04.351 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:52:04.419 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.5 to 1 (10000) 2024-09-20T22:52:04.425 INFO:tasks.rados.rados.0.smithi084.stdout:2531: done (1 left) 2024-09-20T22:52:04.426 INFO:tasks.rados.rados.0.smithi084.stdout:2532: done (0 left) 2024-09-20T22:52:04.426 INFO:tasks.rados.rados.0.smithi084.stdout:2533: read oid 36 snap 274 2024-09-20T22:52:04.426 INFO:tasks.rados.rados.0.smithi084.stdout:2533: expect (ObjNum 826 snap 267 seq_num 826) 2024-09-20T22:52:04.426 INFO:tasks.rados.rados.0.smithi084.stdout:2534: read oid 25 snap 257 2024-09-20T22:52:04.426 INFO:tasks.rados.rados.0.smithi084.stdout:2534: expect (ObjNum 688 snap 212 seq_num 688) 2024-09-20T22:52:04.426 INFO:tasks.rados.rados.0.smithi084.stdout:2535: write (excl) oid 44 current snap is 285 2024-09-20T22:52:04.426 INFO:tasks.rados.rados.0.smithi084.stdout:2535: seq_num 879 ranges {773353=724466,2170378=602975} 2024-09-20T22:52:04.434 INFO:tasks.rados.rados.0.smithi084.stdout:2535: writing smithi08477203-44 from 773353 to 1497819 tid 1 2024-09-20T22:52:04.442 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:52:04.443 INFO:tasks.rados.rados.0.smithi084.stdout:2535: writing smithi08477203-44 from 2170378 to 2773353 tid 2 2024-09-20T22:52:04.443 INFO:tasks.rados.rados.0.smithi084.stdout:2536: snap_remove snap 277 2024-09-20T22:52:04.451 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:52:04.472 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:04.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:04.473 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:52:04.530 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:52:04.530 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:52:04.547 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:52:04.547 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:52:04.548 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:04 smithi084 ceph-mon[40024]: 1.2 scrub starts 2024-09-20T22:52:04.548 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:04 smithi084 ceph-mon[40024]: 1.2 scrub ok 2024-09-20T22:52:04.548 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:04 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/158139921' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-20T22:52:04.548 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:04 smithi084 ceph-mon[29774]: 1.2 scrub starts 2024-09-20T22:52:04.548 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:04 smithi084 ceph-mon[29774]: 1.2 scrub ok 2024-09-20T22:52:04.548 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:04 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/158139921' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-20T22:52:04.563 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:52:04.563 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:52:04.574 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:52:04.579 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-20T22:52:04.579 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T22:52:04.594 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:04.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:04.595 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:52:04.596 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:52:04.596 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:52:04.613 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:52:04.613 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:52:04.629 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:52:04.630 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:52:04.645 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:52:04.646 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:52:04.696 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:52:04.717 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:04.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:04.718 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:52:04.720 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:04 smithi154 ceph-mon[33264]: 1.2 scrub starts 2024-09-20T22:52:04.721 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:04 smithi154 ceph-mon[33264]: 1.2 scrub ok 2024-09-20T22:52:04.721 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:04 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/158139921' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-20T22:52:04.819 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:52:04.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:04.842 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:04.842 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:52:04.942 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:52:04.947 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-20T22:52:04.947 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:52:04.964 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:04.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:04.965 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:52:05.065 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:52:05.086 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:05.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:05.087 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:52:05.188 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:52:05.209 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:05.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:05.209 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:52:05.310 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:52:05.331 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:05.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:05.332 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:52:05.433 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:52:05.443 INFO:tasks.rados.rados.0.smithi084.stdout:2536: done (3 left) 2024-09-20T22:52:05.443 INFO:tasks.rados.rados.0.smithi084.stdout:2537: copy_from oid 9 from oid 13 current snap is 285 2024-09-20T22:52:05.443 INFO:tasks.rados.rados.0.smithi084.stdout:2534: expect (ObjNum 688 snap 212 seq_num 688) 2024-09-20T22:52:05.461 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:05.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:05.463 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:52:05.562 INFO:tasks.rados.rados.0.smithi084.stdout:2533: expect (ObjNum 826 snap 267 seq_num 826) 2024-09-20T22:52:05.564 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:52:05.569 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:05 smithi154 ceph-mon[33264]: pgmap v867: 158 pgs: 4 active+clean+snaptrim, 1 active+remapped+backfill_toofull, 153 active+clean; 522 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 4.8 MiB/s rd, 1.2 MiB/s wr, 7 op/s; 22/1359 objects misplaced (1.619%) 2024-09-20T22:52:05.569 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:05 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/158139921' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-20T22:52:05.570 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:05 smithi154 ceph-mon[33264]: osdmap e644: 8 total, 8 up, 7 in 2024-09-20T22:52:05.570 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:05 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/4122618265' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:52:05.584 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:05.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:05.585 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:52:05.596 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.4 dump_historic_ops 2024-09-20T22:52:05.686 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:52:05.706 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:05.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:05.707 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:52:05.808 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:52:05.830 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:05.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:05.831 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:52:05.841 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:52:05.857 INFO:tasks.rados.rados.0.smithi084.stdout:2535: finishing write tid 1 to smithi08477203-44 2024-09-20T22:52:05.857 INFO:tasks.rados.rados.0.smithi084.stdout:2535: finishing write tid 2 to smithi08477203-44 2024-09-20T22:52:05.857 INFO:tasks.rados.rados.0.smithi084.stdout:2535: finishing write tid 3 to smithi08477203-44 2024-09-20T22:52:05.857 INFO:tasks.rados.rados.0.smithi084.stdout:2535: finishing write tid 4 to smithi08477203-44 2024-09-20T22:52:05.857 INFO:tasks.rados.rados.0.smithi084.stdout:2535: oid 44 updating version 0 to 1900 2024-09-20T22:52:05.857 INFO:tasks.rados.rados.0.smithi084.stdout:2535: oid 44 version 1900 is already newer than 1899 2024-09-20T22:52:05.857 INFO:tasks.rados.rados.0.smithi084.stdout:2535: oid 44 updating version 1900 to 1901 2024-09-20T22:52:05.857 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 44 v 1901 (ObjNum 879 snap 285 seq_num 879) dirty exists 2024-09-20T22:52:05.857 INFO:tasks.rados.rados.0.smithi084.stdout:2535: left oid 44 (ObjNum 879 snap 285 seq_num 879) 2024-09-20T22:52:05.857 INFO:tasks.rados.rados.0.smithi084.stdout:2533: done (3 left) 2024-09-20T22:52:05.858 INFO:tasks.rados.rados.0.smithi084.stdout:2534: done (2 left) 2024-09-20T22:52:05.858 INFO:tasks.rados.rados.0.smithi084.stdout:2535: done (1 left) 2024-09-20T22:52:05.858 INFO:tasks.rados.rados.0.smithi084.stdout:2538: rollback oid 38 current snap is 285 2024-09-20T22:52:05.858 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 38 to 283 2024-09-20T22:52:05.858 INFO:tasks.rados.rados.0.smithi084.stdout:2537: finishing copy_from racing read to smithi08477203-9 2024-09-20T22:52:05.858 INFO:tasks.rados.rados.0.smithi084.stdout:2537: finishing copy_from to smithi08477203-9 2024-09-20T22:52:05.858 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 9 v 1797 (ObjNum 832 snap 268 seq_num 832) dirty exists 2024-09-20T22:52:05.858 INFO:tasks.rados.rados.0.smithi084.stdout:2537: done (1 left) 2024-09-20T22:52:05.858 INFO:tasks.rados.rados.0.smithi084.stdout:2539: delete oid 18 current snap is 285 2024-09-20T22:52:05.859 INFO:tasks.rados.rados.0.smithi084.stdout:2539: done (1 left) 2024-09-20T22:52:05.859 INFO:tasks.rados.rados.0.smithi084.stdout:2540: write (excl) oid 37 current snap is 285 2024-09-20T22:52:05.859 INFO:tasks.rados.rados.0.smithi084.stdout:2540: seq_num 881 ranges {406967=474964,1370989=769310,2677186=129781} 2024-09-20T22:52:05.864 INFO:tasks.rados.rados.0.smithi084.stdout:2540: writing smithi08477203-37 from 406967 to 881931 tid 1 2024-09-20T22:52:05.864 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:05 smithi084 ceph-mon[29774]: pgmap v867: 158 pgs: 4 active+clean+snaptrim, 1 active+remapped+backfill_toofull, 153 active+clean; 522 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 4.8 MiB/s rd, 1.2 MiB/s wr, 7 op/s; 22/1359 objects misplaced (1.619%) 2024-09-20T22:52:05.864 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:05 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/158139921' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-20T22:52:05.864 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:05 smithi084 ceph-mon[29774]: osdmap e644: 8 total, 8 up, 7 in 2024-09-20T22:52:05.864 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:05 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/4122618265' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:52:05.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:05 smithi084 ceph-mon[40024]: pgmap v867: 158 pgs: 4 active+clean+snaptrim, 1 active+remapped+backfill_toofull, 153 active+clean; 522 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 4.8 MiB/s rd, 1.2 MiB/s wr, 7 op/s; 22/1359 objects misplaced (1.619%) 2024-09-20T22:52:05.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:05 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/158139921' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-20T22:52:05.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:05 smithi084 ceph-mon[40024]: osdmap e644: 8 total, 8 up, 7 in 2024-09-20T22:52:05.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:05 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/4122618265' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:52:05.870 INFO:tasks.rados.rados.0.smithi084.stdout:2540: writing smithi08477203-37 from 1370989 to 2140299 tid 2 2024-09-20T22:52:05.875 INFO:tasks.rados.rados.0.smithi084.stdout:2540: writing smithi08477203-37 from 2677186 to 2806967 tid 3 2024-09-20T22:52:05.875 INFO:tasks.rados.rados.0.smithi084.stdout:2541: rollback oid 18 current snap is 285 2024-09-20T22:52:05.875 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 18 to 248 2024-09-20T22:52:05.875 INFO:tasks.rados.rados.0.smithi084.stdout:2538: finishing rollback tid 0 to smithi08477203-38 2024-09-20T22:52:05.875 INFO:tasks.rados.rados.0.smithi084.stdout:2542: write oid 24 current snap is 285 2024-09-20T22:52:05.875 INFO:tasks.rados.rados.0.smithi084.stdout:2538: finishing rollback tid 1 to smithi08477203-38 2024-09-20T22:52:05.875 INFO:tasks.rados.rados.0.smithi084.stdout:2542: seq_num 882 ranges {423774=666566,1545824=512968,2704609=119165} 2024-09-20T22:52:05.880 INFO:tasks.rados.rados.0.smithi084.stdout:2542: writing smithi08477203-24 from 423774 to 1090340 tid 1 2024-09-20T22:52:05.886 INFO:tasks.rados.rados.0.smithi084.stdout:2542: writing smithi08477203-24 from 1545824 to 2058792 tid 2 2024-09-20T22:52:05.891 INFO:tasks.rados.rados.0.smithi084.stdout:2542: writing smithi08477203-24 from 2704609 to 2823774 tid 3 2024-09-20T22:52:05.891 INFO:tasks.rados.rados.0.smithi084.stdout:2543: rollback oid 11 current snap is 285 2024-09-20T22:52:05.891 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 11 to 279 2024-09-20T22:52:05.891 INFO:tasks.rados.rados.0.smithi084.stdout:2538: finishing rollback tid 2 to smithi08477203-38 2024-09-20T22:52:05.891 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 38 v 1338 (ObjNum 852 snap 275 seq_num 852) dirty exists 2024-09-20T22:52:05.891 INFO:tasks.rados.rados.0.smithi084.stdout:2540: finishing write tid 1 to smithi08477203-37 2024-09-20T22:52:05.891 INFO:tasks.rados.rados.0.smithi084.stdout:2540: finishing write tid 2 to smithi08477203-37 2024-09-20T22:52:05.891 INFO:tasks.rados.rados.0.smithi084.stdout:2538: done (4 left) 2024-09-20T22:52:05.891 INFO:tasks.rados.rados.0.smithi084.stdout:2544: write oid 7 current snap is 285 2024-09-20T22:52:05.891 INFO:tasks.rados.rados.0.smithi084.stdout:2540: finishing write tid 3 to smithi08477203-37 2024-09-20T22:52:05.891 INFO:tasks.rados.rados.0.smithi084.stdout:2541: finishing rollback tid 1 to smithi08477203-18 2024-09-20T22:52:05.892 INFO:tasks.rados.rados.0.smithi084.stdout:2540: finishing write tid 4 to smithi08477203-37 2024-09-20T22:52:05.892 INFO:tasks.rados.rados.0.smithi084.stdout:2541: finishing rollback tid 2 to smithi08477203-18 2024-09-20T22:52:05.892 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 18 v 1800 (ObjNum 657 snap 199 seq_num 657) dirty exists 2024-09-20T22:52:05.892 INFO:tasks.rados.rados.0.smithi084.stdout:2540: finishing write tid 5 to smithi08477203-37 2024-09-20T22:52:05.892 INFO:tasks.rados.rados.0.smithi084.stdout:2540: oid 37 updating version 0 to 872 2024-09-20T22:52:05.892 INFO:tasks.rados.rados.0.smithi084.stdout:2540: oid 37 version 872 is already newer than 871 2024-09-20T22:52:05.892 INFO:tasks.rados.rados.0.smithi084.stdout:2540: oid 37 updating version 872 to 873 2024-09-20T22:52:05.892 INFO:tasks.rados.rados.0.smithi084.stdout:2540: oid 37 version 873 is already newer than 870 2024-09-20T22:52:05.892 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 37 v 873 (ObjNum 881 snap 285 seq_num 881) dirty exists 2024-09-20T22:52:05.892 INFO:tasks.rados.rados.0.smithi084.stdout:2540: left oid 37 (ObjNum 881 snap 285 seq_num 881) 2024-09-20T22:52:05.892 INFO:tasks.rados.rados.0.smithi084.stdout:2542: finishing write tid 1 to smithi08477203-24 2024-09-20T22:52:05.892 INFO:tasks.rados.rados.0.smithi084.stdout:2542: finishing write tid 2 to smithi08477203-24 2024-09-20T22:52:05.892 INFO:tasks.rados.rados.0.smithi084.stdout:2544: seq_num 883 ranges {440581=741815,1604306=656626,2732032=108549} 2024-09-20T22:52:05.896 INFO:tasks.rados.rados.0.smithi084.stdout:2544: writing smithi08477203-7 from 440581 to 1182396 tid 1 2024-09-20T22:52:05.902 INFO:tasks.rados.rados.0.smithi084.stdout:2544: writing smithi08477203-7 from 1604306 to 2260932 tid 2 2024-09-20T22:52:05.906 INFO:tasks.rados.rados.0.smithi084.stdout:2544: writing smithi08477203-7 from 2732032 to 2840581 tid 3 2024-09-20T22:52:05.906 INFO:tasks.rados.rados.0.smithi084.stdout:2540: done (4 left) 2024-09-20T22:52:05.907 INFO:tasks.rados.rados.0.smithi084.stdout:2541: done (3 left) 2024-09-20T22:52:05.907 INFO:tasks.rados.rados.0.smithi084.stdout:2545: snap_create 2024-09-20T22:52:05.907 INFO:tasks.rados.rados.0.smithi084.stdout:2543: finishing rollback tid 0 to smithi08477203-11 2024-09-20T22:52:05.907 INFO:tasks.rados.rados.0.smithi084.stdout:2542: finishing write tid 3 to smithi08477203-24 2024-09-20T22:52:05.907 INFO:tasks.rados.rados.0.smithi084.stdout:2543: finishing rollback tid 1 to smithi08477203-11 2024-09-20T22:52:05.907 INFO:tasks.rados.rados.0.smithi084.stdout:2542: finishing write tid 4 to smithi08477203-24 2024-09-20T22:52:05.907 INFO:tasks.rados.rados.0.smithi084.stdout:2543: finishing rollback tid 2 to smithi08477203-11 2024-09-20T22:52:05.907 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 11 v 766 (ObjNum 815 snap 262 seq_num 815) dirty exists 2024-09-20T22:52:05.907 INFO:tasks.rados.rados.0.smithi084.stdout:2542: finishing write tid 5 to smithi08477203-24 2024-09-20T22:52:05.907 INFO:tasks.rados.rados.0.smithi084.stdout:2542: oid 24 updating version 0 to 814 2024-09-20T22:52:05.907 INFO:tasks.rados.rados.0.smithi084.stdout:2542: oid 24 updating version 814 to 816 2024-09-20T22:52:05.907 INFO:tasks.rados.rados.0.smithi084.stdout:2542: oid 24 version 816 is already newer than 815 2024-09-20T22:52:05.907 INFO:tasks.rados.rados.0.smithi084.stdout:2542: oid 24 version 816 is already newer than 813 2024-09-20T22:52:05.908 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 24 v 816 (ObjNum 882 snap 285 seq_num 882) dirty exists 2024-09-20T22:52:05.908 INFO:tasks.rados.rados.0.smithi084.stdout:2542: left oid 24 (ObjNum 882 snap 285 seq_num 882) 2024-09-20T22:52:05.908 INFO:tasks.rados.rados.0.smithi084.stdout:2544: finishing write tid 1 to smithi08477203-7 2024-09-20T22:52:05.908 INFO:tasks.rados.rados.0.smithi084.stdout:2544: finishing write tid 2 to smithi08477203-7 2024-09-20T22:52:05.908 INFO:tasks.rados.rados.0.smithi084.stdout:2544: finishing write tid 3 to smithi08477203-7 2024-09-20T22:52:05.908 INFO:tasks.rados.rados.0.smithi084.stdout:2544: finishing write tid 4 to smithi08477203-7 2024-09-20T22:52:05.909 INFO:tasks.rados.rados.0.smithi084.stdout:2544: finishing write tid 5 to smithi08477203-7 2024-09-20T22:52:05.909 INFO:tasks.rados.rados.0.smithi084.stdout:2544: oid 7 updating version 0 to 876 2024-09-20T22:52:05.909 INFO:tasks.rados.rados.0.smithi084.stdout:2544: oid 7 version 876 is already newer than 875 2024-09-20T22:52:05.909 INFO:tasks.rados.rados.0.smithi084.stdout:2544: oid 7 updating version 876 to 877 2024-09-20T22:52:05.909 INFO:tasks.rados.rados.0.smithi084.stdout:2544: oid 7 version 877 is already newer than 874 2024-09-20T22:52:05.909 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 7 v 877 (ObjNum 883 snap 285 seq_num 883) dirty exists 2024-09-20T22:52:05.909 INFO:tasks.rados.rados.0.smithi084.stdout:2544: left oid 7 (ObjNum 883 snap 285 seq_num 883) 2024-09-20T22:52:05.931 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:52:05.953 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:05.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:05.954 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:52:06.025 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-20T22:52:06.054 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:52:06.075 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:06.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:06.076 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:52:06.124 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd reweight 6 1 2024-09-20T22:52:06.177 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:52:06.197 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:06.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:06.198 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:52:06.279 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:52:06.299 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:52:06.321 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:06.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:06.322 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:52:06.390 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-20T22:52:06.390 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:52:06.423 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:52:06.447 INFO:tasks.rados.rados.0.smithi084.stdout:2542: done (3 left) 2024-09-20T22:52:06.447 INFO:tasks.rados.rados.0.smithi084.stdout:2543: done (2 left) 2024-09-20T22:52:06.447 INFO:tasks.rados.rados.0.smithi084.stdout:2544: done (1 left) 2024-09-20T22:52:06.447 INFO:tasks.rados.rados.0.smithi084.stdout:2545: done (0 left) 2024-09-20T22:52:06.447 INFO:tasks.rados.rados.0.smithi084.stdout:2546: write (excl) oid 3 current snap is 286 2024-09-20T22:52:06.447 INFO:tasks.rados.rados.0.smithi084.stdout:2546: seq_num 884 ranges {457388=417064,1546435=516637,2643102=214286} 2024-09-20T22:52:06.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:06.448 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:06.448 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:52:06.451 INFO:tasks.rados.rados.0.smithi084.stdout:2546: writing smithi08477203-3 from 457388 to 874452 tid 1 2024-09-20T22:52:06.458 INFO:tasks.rados.rados.0.smithi084.stdout:2546: writing smithi08477203-3 from 1546435 to 2063072 tid 2 2024-09-20T22:52:06.463 INFO:tasks.rados.rados.0.smithi084.stdout:2546: writing smithi08477203-3 from 2643102 to 2857388 tid 3 2024-09-20T22:52:06.463 INFO:tasks.rados.rados.0.smithi084.stdout:2547: snap_create 2024-09-20T22:52:06.463 INFO:tasks.rados.rados.0.smithi084.stdout:2546: finishing write tid 1 to smithi08477203-3 2024-09-20T22:52:06.463 INFO:tasks.rados.rados.0.smithi084.stdout:2546: finishing write tid 2 to smithi08477203-3 2024-09-20T22:52:06.464 INFO:tasks.rados.rados.0.smithi084.stdout:2546: finishing write tid 3 to smithi08477203-3 2024-09-20T22:52:06.465 INFO:tasks.rados.rados.0.smithi084.stdout:2546: finishing write tid 4 to smithi08477203-3 2024-09-20T22:52:06.465 INFO:tasks.rados.rados.0.smithi084.stdout:2546: finishing write tid 5 to smithi08477203-3 2024-09-20T22:52:06.465 INFO:tasks.rados.rados.0.smithi084.stdout:2546: oid 3 updating version 0 to 1810 2024-09-20T22:52:06.465 INFO:tasks.rados.rados.0.smithi084.stdout:2546: oid 3 updating version 1810 to 1811 2024-09-20T22:52:06.465 INFO:tasks.rados.rados.0.smithi084.stdout:2546: oid 3 version 1811 is already newer than 1808 2024-09-20T22:52:06.465 INFO:tasks.rados.rados.0.smithi084.stdout:2546: oid 3 version 1811 is already newer than 1809 2024-09-20T22:52:06.465 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 3 v 1811 (ObjNum 884 snap 286 seq_num 884) dirty exists 2024-09-20T22:52:06.465 INFO:tasks.rados.rados.0.smithi084.stdout:2546: left oid 3 (ObjNum 884 snap 286 seq_num 884) 2024-09-20T22:52:06.549 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:52:06.570 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:06.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:06.571 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:52:06.671 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:52:06.692 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:06.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:06.693 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:52:06.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:06 smithi154 ceph-mon[33264]: osdmap e645: 8 total, 8 up, 7 in 2024-09-20T22:52:06.758 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:06 smithi084 ceph-mon[29774]: osdmap e645: 8 total, 8 up, 7 in 2024-09-20T22:52:06.758 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:06 smithi084 ceph-mon[40024]: osdmap e645: 8 total, 8 up, 7 in 2024-09-20T22:52:06.794 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:52:06.794 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:52:06.816 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:06.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:06.817 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:52:06.918 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:52:06.939 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:06.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:06.940 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:52:07.041 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:52:07.063 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:07.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:07.063 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:52:07.139 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-20T22:52:07.164 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:52:07.203 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:07.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:07.204 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:52:07.305 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:52:07.327 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:07.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:07.328 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:52:07.386 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:52:07.429 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:52:07.444 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is set 2024-09-20T22:52:07.449 INFO:tasks.rados.rados.0.smithi084.stdout:2546: done (1 left) 2024-09-20T22:52:07.449 INFO:tasks.rados.rados.0.smithi084.stdout:2547: done (0 left) 2024-09-20T22:52:07.449 INFO:tasks.rados.rados.0.smithi084.stdout:2548: delete oid 48 current snap is 287 2024-09-20T22:52:07.453 INFO:tasks.rados.rados.0.smithi084.stdout:2548: done (0 left) 2024-09-20T22:52:07.453 INFO:tasks.rados.rados.0.smithi084.stdout:2549: copy_from oid 21 from oid 40 current snap is 287 2024-09-20T22:52:07.453 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:07.454 INFO:tasks.rados.rados.0.smithi084.stdout:2550: rollback oid 24 current snap is 287 2024-09-20T22:52:07.454 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 24 to 286 2024-09-20T22:52:07.454 INFO:tasks.rados.rados.0.smithi084.stdout:2551: rollback oid 32 current snap is 287 2024-09-20T22:52:07.454 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 32 to 278 2024-09-20T22:52:07.455 INFO:tasks.rados.rados.0.smithi084.stdout:2552: write (excl) oid 25 current snap is 287 2024-09-20T22:52:07.455 INFO:tasks.rados.rados.0.smithi084.stdout:2552: seq_num 886 ranges {491002=567562,1663399=520306,2814301=76701} 2024-09-20T22:52:07.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:07.455 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:52:07.460 INFO:tasks.rados.rados.0.smithi084.stdout:2552: writing smithi08477203-25 from 491002 to 1058564 tid 1 2024-09-20T22:52:07.467 INFO:tasks.rados.rados.0.smithi084.stdout:2552: writing smithi08477203-25 from 1663399 to 2183705 tid 2 2024-09-20T22:52:07.472 INFO:tasks.rados.rados.0.smithi084.stdout:2552: writing smithi08477203-25 from 2814301 to 2891002 tid 3 2024-09-20T22:52:07.472 INFO:tasks.rados.rados.0.smithi084.stdout:2553: rollback oid 23 current snap is 287 2024-09-20T22:52:07.472 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 23 to 285 2024-09-20T22:52:07.472 INFO:tasks.rados.rados.0.smithi084.stdout:2550: finishing rollback tid 0 to smithi08477203-24 2024-09-20T22:52:07.472 INFO:tasks.rados.rados.0.smithi084.stdout:2551: finishing rollback tid 0 to smithi08477203-32 2024-09-20T22:52:07.472 INFO:tasks.rados.rados.0.smithi084.stdout:2551: finishing rollback tid 1 to smithi08477203-32 2024-09-20T22:52:07.473 INFO:tasks.rados.rados.0.smithi084.stdout:2550: finishing rollback tid 1 to smithi08477203-24 2024-09-20T22:52:07.473 INFO:tasks.rados.rados.0.smithi084.stdout:2550: finishing rollback tid 2 to smithi08477203-24 2024-09-20T22:52:07.473 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 24 v 819 (ObjNum 882 snap 285 seq_num 882) dirty exists 2024-09-20T22:52:07.473 INFO:tasks.rados.rados.0.smithi084.stdout:2551: finishing rollback tid 2 to smithi08477203-32 2024-09-20T22:52:07.473 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 32 v 744 (ObjNum 769 snap 248 seq_num 769) dirty exists 2024-09-20T22:52:07.473 INFO:tasks.rados.rados.0.smithi084.stdout:2549: finishing copy_from racing read to smithi08477203-21 2024-09-20T22:52:07.473 INFO:tasks.rados.rados.0.smithi084.stdout:2550: done (4 left) 2024-09-20T22:52:07.473 INFO:tasks.rados.rados.0.smithi084.stdout:2551: done (3 left) 2024-09-20T22:52:07.473 INFO:tasks.rados.rados.0.smithi084.stdout:2554: snap_remove snap 271 2024-09-20T22:52:07.481 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-20T22:52:07.556 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:52:07.578 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:07.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:07.579 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:52:07.680 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:52:07.701 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:07.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:07.702 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:52:07.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:07 smithi154 ceph-mon[33264]: pgmap v870: 158 pgs: 4 active+clean+snaptrim, 1 active+remapped+backfill_toofull, 153 active+clean; 522 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 4.8 MiB/s rd, 1.2 MiB/s wr, 7 op/s; 22/1359 objects misplaced (1.619%) 2024-09-20T22:52:07.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:07 smithi154 ceph-mon[33264]: osdmap e646: 8 total, 8 up, 7 in 2024-09-20T22:52:07.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:07 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/1682861199' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:52:07.803 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:52:07.826 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:07.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:07.827 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:52:07.839 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:07 smithi084 ceph-mon[40024]: pgmap v870: 158 pgs: 4 active+clean+snaptrim, 1 active+remapped+backfill_toofull, 153 active+clean; 522 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 4.8 MiB/s rd, 1.2 MiB/s wr, 7 op/s; 22/1359 objects misplaced (1.619%) 2024-09-20T22:52:07.839 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:07 smithi084 ceph-mon[40024]: osdmap e646: 8 total, 8 up, 7 in 2024-09-20T22:52:07.839 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:07 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/1682861199' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:52:07.840 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:07 smithi084 ceph-mon[29774]: pgmap v870: 158 pgs: 4 active+clean+snaptrim, 1 active+remapped+backfill_toofull, 153 active+clean; 522 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 4.8 MiB/s rd, 1.2 MiB/s wr, 7 op/s; 22/1359 objects misplaced (1.619%) 2024-09-20T22:52:07.840 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:07 smithi084 ceph-mon[29774]: osdmap e646: 8 total, 8 up, 7 in 2024-09-20T22:52:07.840 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:07 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/1682861199' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:52:07.875 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-20T22:52:07.875 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:52:07.928 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:52:07.950 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:07.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:07.950 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:52:08.051 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:52:08.072 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:08.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:08.073 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:52:08.173 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:52:08.195 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:08.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:08.196 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:52:08.266 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.6 to 1 (10000) 2024-09-20T22:52:08.267 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:52:08.280 INFO:tasks.rados.rados.0.smithi084.stdout:2554: done (3 left) 2024-09-20T22:52:08.280 INFO:tasks.rados.rados.0.smithi084.stdout:2555: snap_remove snap 282 2024-09-20T22:52:08.296 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:52:08.319 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:08.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:08.320 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:52:08.421 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:52:08.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:08.443 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:08.443 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:52:08.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:08 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/1682861199' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T22:52:08.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:08 smithi154 ceph-mon[33264]: osdmap e647: 8 total, 8 up, 7 in 2024-09-20T22:52:08.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:08 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/3817588552' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-09-20T22:52:08.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:08 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:52:08.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:08 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/3817588552' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-20T22:52:08.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:08 smithi154 ceph-mon[33264]: osdmap e648: 8 total, 8 up, 7 in 2024-09-20T22:52:08.544 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:52:08.562 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.5 dump_blocked_ops 2024-09-20T22:52:08.565 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:08.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:08.566 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:52:08.614 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:52:08 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: ::ffff:172.21.15.154 - - [20/Sep/2024:22:52:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-20T22:52:08.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:08 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/1682861199' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T22:52:08.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:08 smithi084 ceph-mon[29774]: osdmap e647: 8 total, 8 up, 7 in 2024-09-20T22:52:08.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:08 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/3817588552' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-09-20T22:52:08.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:08 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:52:08.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:08 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/3817588552' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-20T22:52:08.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:08 smithi084 ceph-mon[29774]: osdmap e648: 8 total, 8 up, 7 in 2024-09-20T22:52:08.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:08 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/1682861199' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T22:52:08.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:08 smithi084 ceph-mon[40024]: osdmap e647: 8 total, 8 up, 7 in 2024-09-20T22:52:08.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:08 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/3817588552' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-09-20T22:52:08.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:08 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:52:08.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:08 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/3817588552' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-20T22:52:08.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:08 smithi084 ceph-mon[40024]: osdmap e648: 8 total, 8 up, 7 in 2024-09-20T22:52:08.667 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:52:08.688 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:08.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:08.689 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:52:08.790 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:52:08.811 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:52:08.811 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:08.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:08.812 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:52:08.913 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:52:08.934 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:08.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:08.935 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:52:09.035 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:52:09.057 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:09.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:09.058 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:52:09.159 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:52:09.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:09.180 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:09.180 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:52:09.250 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd reweight 7 1 2024-09-20T22:52:09.279 INFO:tasks.rados.rados.0.smithi084.stdout:2555: done (3 left) 2024-09-20T22:52:09.279 INFO:tasks.rados.rados.0.smithi084.stdout:2556: snap_remove snap 280 2024-09-20T22:52:09.281 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:52:09.302 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:09.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:09.303 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:52:09.308 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-20T22:52:09.309 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:52:09.404 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:52:09.427 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:09.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:09.428 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:52:09.488 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:09 smithi154 ceph-mon[33264]: pgmap v873: 158 pgs: 4 active+clean+snaptrim, 1 active+remapped+backfill_toofull, 153 active+clean; 522 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 22/1359 objects misplaced (1.619%) 2024-09-20T22:52:09.488 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:09 smithi154 ceph-mon[33264]: osdmap e649: 8 total, 8 up, 7 in 2024-09-20T22:52:09.488 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:09 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [3, 2]}]: dispatch 2024-09-20T22:52:09.488 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:09 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [3, 2]}]: dispatch 2024-09-20T22:52:09.507 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:09 smithi084 ceph-mon[29774]: pgmap v873: 158 pgs: 4 active+clean+snaptrim, 1 active+remapped+backfill_toofull, 153 active+clean; 522 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 22/1359 objects misplaced (1.619%) 2024-09-20T22:52:09.507 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:09 smithi084 ceph-mon[29774]: osdmap e649: 8 total, 8 up, 7 in 2024-09-20T22:52:09.507 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:09 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [3, 2]}]: dispatch 2024-09-20T22:52:09.507 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:09 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [3, 2]}]: dispatch 2024-09-20T22:52:09.507 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:09 smithi084 ceph-mon[40024]: pgmap v873: 158 pgs: 4 active+clean+snaptrim, 1 active+remapped+backfill_toofull, 153 active+clean; 522 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 22/1359 objects misplaced (1.619%) 2024-09-20T22:52:09.507 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:09 smithi084 ceph-mon[40024]: osdmap e649: 8 total, 8 up, 7 in 2024-09-20T22:52:09.508 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:09 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [3, 2]}]: dispatch 2024-09-20T22:52:09.508 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:09 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [3, 2]}]: dispatch 2024-09-20T22:52:09.508 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:52:09.508 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:52:09.525 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:52:09.525 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:52:09.528 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:52:09.545 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:52:09.545 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:52:09.552 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:09.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:09.552 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:52:09.562 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-20T22:52:09.562 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T22:52:09.580 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:52:09.580 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:52:09.590 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:52:09.598 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:52:09.598 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:52:09.616 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:52:09.616 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:52:09.632 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:52:09.633 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:52:09.653 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:52:09.675 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:09.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:09.676 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:52:09.754 INFO:journalctl@ceph.iscsi.iscsi.a.smithi154.stdout:Sep 20 22:52:09 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-iscsi-iscsi-a[59730]: debug there is no tcmu-runner data available 2024-09-20T22:52:09.776 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:52:09.798 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:09.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:09.799 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:52:09.900 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:52:09.921 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:09.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:09.922 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:52:10.005 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.5 dump_historic_ops 2024-09-20T22:52:10.022 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:52:10.069 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:10.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:10.070 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:52:10.171 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:52:10.193 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:10.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:10.194 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:52:10.253 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:52:10.259 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd unset noscrub 2024-09-20T22:52:10.289 INFO:tasks.rados.rados.0.smithi084.stdout:2556: done (3 left) 2024-09-20T22:52:10.290 INFO:tasks.rados.rados.0.smithi084.stdout:2557: read oid 4 snap -1 2024-09-20T22:52:10.290 INFO:tasks.rados.rados.0.smithi084.stdout:2557: expect (ObjNum 863 snap 276 seq_num 863) 2024-09-20T22:52:10.290 INFO:tasks.rados.rados.0.smithi084.stdout:2558: snap_remove snap 240 2024-09-20T22:52:10.294 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:52:10.316 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:10.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:10.317 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:52:10.418 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:52:10.439 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:10.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:10.440 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:52:10.457 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:10 smithi084 ceph-mon[29774]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:52:10.457 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:10 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/330448921' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:52:10.457 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:10 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [3, 2]}]': finished 2024-09-20T22:52:10.458 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:10 smithi084 ceph-mon[29774]: osdmap e650: 8 total, 8 up, 7 in 2024-09-20T22:52:10.518 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:10 smithi154 ceph-mon[33264]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:52:10.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:10 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/330448921' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:52:10.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:10 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [3, 2]}]': finished 2024-09-20T22:52:10.519 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:10 smithi154 ceph-mon[33264]: osdmap e650: 8 total, 8 up, 7 in 2024-09-20T22:52:10.541 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:52:10.562 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:10.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:10.563 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:52:10.664 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:52:10.686 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:10.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:10.687 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:52:10.717 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-20T22:52:10.717 INFO:teuthology.orchestra.run.smithi154.stdout: "size": 20, 2024-09-20T22:52:10.717 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 600, 2024-09-20T22:52:10.717 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [ 2024-09-20T22:52:10.718 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.718 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "MOSDPGPush(1.4 93/92 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 77'128, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@77'128, size: 327680, copy_subset: [(0, 327680)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 327680, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-20T22:52:10.718 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:42:50.646393+0000", 2024-09-20T22:52:10.718 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 560.06835805699995, 2024-09-20T22:52:10.718 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.033640915, 2024-09-20T22:52:10.718 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:52:10.718 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:52:10.718 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:52:10.718 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:52:10.718 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.718 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:52:10.718 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:42:50.646393+0000", 2024-09-20T22:52:10.718 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:52:10.718 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.719 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.719 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:52:10.719 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:42:50.646393+0000", 2024-09-20T22:52:10.719 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:52:10.719 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.719 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.719 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:52:10.719 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:42:50.646393+0000", 2024-09-20T22:52:10.719 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.23e-07 2024-09-20T22:52:10.719 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.719 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.719 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:52:10.719 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:42:50.679431+0000", 2024-09-20T22:52:10.719 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.033038349000000002 2024-09-20T22:52:10.719 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.720 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.720 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:52:10.720 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:42:50.679478+0000", 2024-09-20T22:52:10.720 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.6828e-05 2024-09-20T22:52:10.720 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.720 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.720 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:52:10.720 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:42:50.679492+0000", 2024-09-20T22:52:10.720 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4062e-05 2024-09-20T22:52:10.720 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.720 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.720 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:52:10.720 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:42:50.679593+0000", 2024-09-20T22:52:10.720 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000100865 2024-09-20T22:52:10.720 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.721 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.721 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:52:10.721 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:42:50.679604+0000", 2024-09-20T22:52:10.721 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0348e-05 2024-09-20T22:52:10.721 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.721 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.721 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:52:10.721 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:42:50.680034+0000", 2024-09-20T22:52:10.721 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00043023999999999997 2024-09-20T22:52:10.721 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:52:10.721 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:52:10.721 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:52:10.721 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.721 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.721 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:554 7.a e128/94 7:5122292a:::smithi08477203-33:head v 128'32, mlcod=119'28)", 2024-09-20T22:52:10.722 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:27.341250+0000", 2024-09-20T22:52:10.722 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 523.37350047300004, 2024-09-20T22:52:10.722 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.051661319999999997, 2024-09-20T22:52:10.722 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:52:10.722 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:52:10.722 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:52:10.722 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:52:10.722 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.722 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:52:10.722 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.341250+0000", 2024-09-20T22:52:10.722 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:52:10.722 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.722 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.722 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:52:10.722 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.341250+0000", 2024-09-20T22:52:10.723 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:52:10.723 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.723 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.723 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:52:10.723 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.341251+0000", 2024-09-20T22:52:10.723 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.7299999999999996e-07 2024-09-20T22:52:10.723 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.723 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.723 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:52:10.723 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.390395+0000", 2024-09-20T22:52:10.723 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.049143539999999999 2024-09-20T22:52:10.723 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.723 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.723 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:52:10.724 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.390413+0000", 2024-09-20T22:52:10.724 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.8709999999999999e-05 2024-09-20T22:52:10.724 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.724 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.724 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:52:10.724 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.390423+0000", 2024-09-20T22:52:10.724 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.4660000000000005e-06 2024-09-20T22:52:10.724 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.724 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.724 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:52:10.724 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.390484+0000", 2024-09-20T22:52:10.724 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.0851999999999999e-05 2024-09-20T22:52:10.724 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.724 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.724 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:52:10.725 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.390504+0000", 2024-09-20T22:52:10.725 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9976999999999999e-05 2024-09-20T22:52:10.725 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.725 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.725 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:52:10.725 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.392875+0000", 2024-09-20T22:52:10.725 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0023713710000000002 2024-09-20T22:52:10.725 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.725 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.725 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:52:10.725 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.392912+0000", 2024-09-20T22:52:10.725 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.6831e-05 2024-09-20T22:52:10.725 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:52:10.725 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:52:10.725 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:52:10.725 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.726 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.726 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:786 7.17 e145/139 7:ed54065f:::smithi08477203-12:head v 145'119, mlcod=143'117)", 2024-09-20T22:52:10.726 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:44.345115+0000", 2024-09-20T22:52:10.726 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 506.369635991, 2024-09-20T22:52:10.726 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0098654899999999993, 2024-09-20T22:52:10.726 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:52:10.726 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:52:10.726 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:52:10.726 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:52:10.726 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.726 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:52:10.726 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.345115+0000", 2024-09-20T22:52:10.726 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:52:10.726 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.726 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.726 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:52:10.727 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.345115+0000", 2024-09-20T22:52:10.727 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:52:10.727 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.727 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.727 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:52:10.727 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.345116+0000", 2024-09-20T22:52:10.727 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.8299999999999995e-07 2024-09-20T22:52:10.727 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.727 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.727 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:52:10.727 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.347955+0000", 2024-09-20T22:52:10.727 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.002839028 2024-09-20T22:52:10.727 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.727 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.727 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:52:10.727 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.347977+0000", 2024-09-20T22:52:10.727 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.1897999999999999e-05 2024-09-20T22:52:10.728 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.728 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.728 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:52:10.728 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.347992+0000", 2024-09-20T22:52:10.728 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4761e-05 2024-09-20T22:52:10.728 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.728 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.728 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:52:10.728 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.348072+0000", 2024-09-20T22:52:10.728 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.0586000000000004e-05 2024-09-20T22:52:10.728 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.728 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.728 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:52:10.729 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.348097+0000", 2024-09-20T22:52:10.729 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.5026000000000001e-05 2024-09-20T22:52:10.729 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.729 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.729 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:52:10.729 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.354924+0000", 2024-09-20T22:52:10.729 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0068271240000000004 2024-09-20T22:52:10.729 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.729 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.729 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:52:10.729 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.354980+0000", 2024-09-20T22:52:10.729 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.6184000000000003e-05 2024-09-20T22:52:10.729 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:52:10.729 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:52:10.730 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:52:10.730 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.730 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.730 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:894 7.a e155/94 7:50097c2d:::smithi08477203-23:head v 155'54, mlcod=143'50)", 2024-09-20T22:52:10.730 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:54.450745+0000", 2024-09-20T22:52:10.731 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 496.264005903, 2024-09-20T22:52:10.731 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0091901029999999998, 2024-09-20T22:52:10.731 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:52:10.731 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:52:10.731 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:52:10.731 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:52:10.731 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.731 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:52:10.731 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:54.450745+0000", 2024-09-20T22:52:10.731 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:52:10.731 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.731 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.731 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:52:10.731 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:54.450745+0000", 2024-09-20T22:52:10.732 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:52:10.732 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.732 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.732 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:52:10.732 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:54.450745+0000", 2024-09-20T22:52:10.732 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.4200000000000002e-07 2024-09-20T22:52:10.732 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.732 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.732 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:52:10.732 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:54.456336+0000", 2024-09-20T22:52:10.732 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.005590589 2024-09-20T22:52:10.732 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.732 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.732 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:52:10.732 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:54.456353+0000", 2024-09-20T22:52:10.732 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.7187e-05 2024-09-20T22:52:10.732 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.733 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.733 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:52:10.733 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:54.456362+0000", 2024-09-20T22:52:10.733 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.7409999999999998e-06 2024-09-20T22:52:10.733 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.733 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.733 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:52:10.733 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:54.456418+0000", 2024-09-20T22:52:10.733 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.6431000000000002e-05 2024-09-20T22:52:10.733 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.733 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.733 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:52:10.733 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:54.456438+0000", 2024-09-20T22:52:10.733 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9718999999999999e-05 2024-09-20T22:52:10.733 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.733 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.734 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:52:10.734 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:54.459899+0000", 2024-09-20T22:52:10.734 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.003461254 2024-09-20T22:52:10.734 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.734 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.734 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:52:10.734 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:54.459935+0000", 2024-09-20T22:52:10.734 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.5939999999999998e-05 2024-09-20T22:52:10.734 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:52:10.734 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:52:10.734 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:52:10.734 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.734 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.734 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:1527 7.17 e197/139 7:ea720b85:::smithi08477203-26:head v 197'159, mlcod=187'156)", 2024-09-20T22:52:10.734 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:44:36.209560+0000", 2024-09-20T22:52:10.734 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 454.50519131200002, 2024-09-20T22:52:10.734 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.010263069999999999, 2024-09-20T22:52:10.735 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:52:10.735 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:52:10.735 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:52:10.735 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:52:10.735 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.735 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:52:10.735 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.209560+0000", 2024-09-20T22:52:10.735 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:52:10.735 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.735 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.735 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:52:10.735 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.209560+0000", 2024-09-20T22:52:10.735 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:52:10.735 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.735 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.735 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:52:10.735 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.209560+0000", 2024-09-20T22:52:10.736 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.0800000000000001e-07 2024-09-20T22:52:10.736 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.736 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.736 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:52:10.736 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.212844+0000", 2024-09-20T22:52:10.736 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.003283907 2024-09-20T22:52:10.736 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.736 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.736 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:52:10.736 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.212867+0000", 2024-09-20T22:52:10.736 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.3229000000000001e-05 2024-09-20T22:52:10.736 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.736 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.736 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:52:10.736 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.212878+0000", 2024-09-20T22:52:10.736 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0998e-05 2024-09-20T22:52:10.737 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.737 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.737 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:52:10.737 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.212971+0000", 2024-09-20T22:52:10.737 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.2872000000000001e-05 2024-09-20T22:52:10.737 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.737 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.737 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:52:10.737 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.212996+0000", 2024-09-20T22:52:10.737 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.4921000000000001e-05 2024-09-20T22:52:10.737 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.737 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.737 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:52:10.737 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.219758+0000", 2024-09-20T22:52:10.737 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0067625139999999999 2024-09-20T22:52:10.737 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.737 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.738 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:52:10.738 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.219823+0000", 2024-09-20T22:52:10.738 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.4420999999999999e-05 2024-09-20T22:52:10.738 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:52:10.738 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:52:10.738 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:52:10.738 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.738 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.738 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:2350 7.0 7:064f66ab:::smithi08477203-45:head [stat,delete] snapc 56=[56,55,53,52,51,50,47,3e] ondisk+write+known_if_redirected+supports_pool_eio e260)", 2024-09-20T22:52:10.738 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:45:39.199566+0000", 2024-09-20T22:52:10.738 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 391.51518488300002, 2024-09-20T22:52:10.738 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.027897148, 2024-09-20T22:52:10.738 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:52:10.738 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:52:10.738 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:52:10.739 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:52:10.739 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:52:10.739 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:52:10.739 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 2350 2024-09-20T22:52:10.739 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.739 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:52:10.739 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.739 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:52:10.739 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.199566+0000", 2024-09-20T22:52:10.739 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:52:10.739 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.739 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.739 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:52:10.739 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.199566+0000", 2024-09-20T22:52:10.739 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:52:10.740 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.740 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.740 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:52:10.740 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.199568+0000", 2024-09-20T22:52:10.740 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9290000000000001e-06 2024-09-20T22:52:10.740 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.740 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.740 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:52:10.740 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.199574+0000", 2024-09-20T22:52:10.740 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.3409999999999999e-06 2024-09-20T22:52:10.740 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.740 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.740 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:52:10.740 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.199584+0000", 2024-09-20T22:52:10.741 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0091e-05 2024-09-20T22:52:10.741 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.741 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.741 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:52:10.741 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.199593+0000", 2024-09-20T22:52:10.741 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.1079999999999993e-06 2024-09-20T22:52:10.741 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.741 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.741 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:52:10.741 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.199661+0000", 2024-09-20T22:52:10.741 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.8767000000000001e-05 2024-09-20T22:52:10.741 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.741 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.741 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:52:10.741 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.199765+0000", 2024-09-20T22:52:10.741 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000103886 2024-09-20T22:52:10.741 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.742 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.742 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:52:10.742 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.227463+0000", 2024-09-20T22:52:10.742 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.027698026000000001 2024-09-20T22:52:10.742 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:52:10.742 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:52:10.742 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:52:10.742 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.742 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.742 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:3134 7.2 7:45b54b6b:::smithi08477203-2:head [copy-from ver 151] snapc 77=[77,76,75,74,73,70,6b] ondisk+write+known_if_redirected+supports_pool_eio e328)", 2024-09-20T22:52:10.742 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:46:48.211103+0000", 2024-09-20T22:52:10.742 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 322.503648376, 2024-09-20T22:52:10.742 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.030812519, 2024-09-20T22:52:10.742 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:52:10.742 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:52:10.743 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:52:10.743 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:52:10.743 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:52:10.743 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:52:10.743 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 3134 2024-09-20T22:52:10.743 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.743 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:52:10.743 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.743 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:52:10.743 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.211103+0000", 2024-09-20T22:52:10.743 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:52:10.743 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.743 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.743 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:52:10.743 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.211103+0000", 2024-09-20T22:52:10.743 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:52:10.744 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.744 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.744 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:52:10.744 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.211105+0000", 2024-09-20T22:52:10.744 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.3369999999999998e-06 2024-09-20T22:52:10.744 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.744 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.744 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:52:10.744 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.211111+0000", 2024-09-20T22:52:10.744 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.2759999999999997e-06 2024-09-20T22:52:10.744 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.744 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.744 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:52:10.744 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.211123+0000", 2024-09-20T22:52:10.744 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2119999999999999e-05 2024-09-20T22:52:10.744 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.745 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.745 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:52:10.745 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.211131+0000", 2024-09-20T22:52:10.745 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.2899999999999997e-06 2024-09-20T22:52:10.745 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.745 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.745 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:52:10.745 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.211213+0000", 2024-09-20T22:52:10.745 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.2756999999999994e-05 2024-09-20T22:52:10.745 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.745 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.745 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:52:10.745 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.211310+0000", 2024-09-20T22:52:10.745 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.6644999999999994e-05 2024-09-20T22:52:10.745 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.745 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.745 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 1,2", 2024-09-20T22:52:10.746 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.229109+0000", 2024-09-20T22:52:10.746 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.017798587000000001 2024-09-20T22:52:10.746 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.746 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.746 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:52:10.746 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.240265+0000", 2024-09-20T22:52:10.746 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.011156716000000001 2024-09-20T22:52:10.746 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.746 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.746 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:52:10.746 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.241608+0000", 2024-09-20T22:52:10.746 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001343033 2024-09-20T22:52:10.746 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.746 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.746 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:52:10.746 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.241831+0000", 2024-09-20T22:52:10.747 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000223148 2024-09-20T22:52:10.747 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.747 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.747 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:52:10.747 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.241885+0000", 2024-09-20T22:52:10.747 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.3161000000000001e-05 2024-09-20T22:52:10.747 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.747 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.747 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:52:10.747 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.241915+0000", 2024-09-20T22:52:10.747 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.0448999999999999e-05 2024-09-20T22:52:10.747 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:52:10.747 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:52:10.747 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:52:10.747 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.747 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.748 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:3135 7.2 7:45b54b6b:::smithi08477203-2:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e328)", 2024-09-20T22:52:10.748 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:46:48.211153+0000", 2024-09-20T22:52:10.748 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 322.503598404, 2024-09-20T22:52:10.748 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.022731534000000001, 2024-09-20T22:52:10.748 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:52:10.748 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:52:10.748 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:52:10.748 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:52:10.748 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:52:10.748 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:52:10.748 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 3135 2024-09-20T22:52:10.748 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.748 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:52:10.748 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.749 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:52:10.749 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.211153+0000", 2024-09-20T22:52:10.749 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:52:10.749 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.749 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.749 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:52:10.749 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.211153+0000", 2024-09-20T22:52:10.749 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:52:10.749 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.749 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.749 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:52:10.749 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.211153+0000", 2024-09-20T22:52:10.749 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.1600000000000001e-07 2024-09-20T22:52:10.749 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.749 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.750 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:52:10.750 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.211157+0000", 2024-09-20T22:52:10.750 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.5470000000000001e-06 2024-09-20T22:52:10.750 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.750 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.750 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:52:10.750 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.211164+0000", 2024-09-20T22:52:10.750 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.2799999999999998e-06 2024-09-20T22:52:10.750 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.750 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.750 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:52:10.750 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.211170+0000", 2024-09-20T22:52:10.750 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.0730000000000001e-06 2024-09-20T22:52:10.750 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.750 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.750 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:52:10.751 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.211634+0000", 2024-09-20T22:52:10.751 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00046401800000000002 2024-09-20T22:52:10.751 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.751 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.751 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for blocked object", 2024-09-20T22:52:10.751 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.211690+0000", 2024-09-20T22:52:10.751 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.5587999999999997e-05 2024-09-20T22:52:10.751 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.751 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.751 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:52:10.751 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.233669+0000", 2024-09-20T22:52:10.751 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.021979240000000001 2024-09-20T22:52:10.751 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.751 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.751 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:52:10.751 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.233762+0000", 2024-09-20T22:52:10.752 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.2658999999999995e-05 2024-09-20T22:52:10.752 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.752 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.752 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:52:10.752 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.233884+0000", 2024-09-20T22:52:10.752 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00012241300000000001 2024-09-20T22:52:10.752 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:52:10.752 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:52:10.752 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:52:10.752 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.752 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.752 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:3136 7.a e328/94 7:5122292a:::smithi08477203-33:head v 328'252, mlcod=322'248)", 2024-09-20T22:52:10.752 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:46:48.223492+0000", 2024-09-20T22:52:10.752 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 322.49125869300002, 2024-09-20T22:52:10.752 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.016662746999999999, 2024-09-20T22:52:10.752 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:52:10.752 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:52:10.752 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:52:10.753 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:52:10.753 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.753 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:52:10.753 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.223492+0000", 2024-09-20T22:52:10.753 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:52:10.753 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.753 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.753 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:52:10.753 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.223492+0000", 2024-09-20T22:52:10.753 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:52:10.753 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.753 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.753 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:52:10.753 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.223493+0000", 2024-09-20T22:52:10.753 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.4299999999999998e-07 2024-09-20T22:52:10.754 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.754 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.754 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:52:10.754 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.228060+0000", 2024-09-20T22:52:10.754 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0045669350000000003 2024-09-20T22:52:10.754 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.754 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.754 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:52:10.754 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.228107+0000", 2024-09-20T22:52:10.754 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.6932999999999999e-05 2024-09-20T22:52:10.754 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.754 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.754 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:52:10.754 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.228127+0000", 2024-09-20T22:52:10.754 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.0777000000000001e-05 2024-09-20T22:52:10.755 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.755 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.755 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:52:10.755 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.228206+0000", 2024-09-20T22:52:10.755 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.8660999999999998e-05 2024-09-20T22:52:10.755 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.755 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.755 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:52:10.755 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.228232+0000", 2024-09-20T22:52:10.755 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.6199000000000001e-05 2024-09-20T22:52:10.755 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.755 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.755 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:52:10.755 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.240103+0000", 2024-09-20T22:52:10.755 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.011871008000000001 2024-09-20T22:52:10.755 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.756 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.756 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:52:10.756 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.240155+0000", 2024-09-20T22:52:10.756 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.1790999999999999e-05 2024-09-20T22:52:10.756 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:52:10.756 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:52:10.756 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:52:10.756 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.756 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.756 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:3589 7.b e363/94 7:dc95255e:::smithi08477203-1:head v 363'419, mlcod=363'417)", 2024-09-20T22:52:10.756 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:47:23.205271+0000", 2024-09-20T22:52:10.756 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 287.50947971199997, 2024-09-20T22:52:10.756 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.014301911, 2024-09-20T22:52:10.756 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:52:10.757 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:52:10.757 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:52:10.757 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:52:10.757 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.757 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:52:10.757 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:23.205271+0000", 2024-09-20T22:52:10.757 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:52:10.757 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.757 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.757 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:52:10.757 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:23.205271+0000", 2024-09-20T22:52:10.757 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:52:10.757 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.757 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.757 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:52:10.757 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:23.205273+0000", 2024-09-20T22:52:10.758 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.409e-06 2024-09-20T22:52:10.758 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.758 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.758 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:52:10.758 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:23.210044+0000", 2024-09-20T22:52:10.758 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0047712609999999997 2024-09-20T22:52:10.758 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.758 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.758 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:52:10.758 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:23.210064+0000", 2024-09-20T22:52:10.758 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.0421e-05 2024-09-20T22:52:10.758 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.758 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.758 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:52:10.758 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:23.210074+0000", 2024-09-20T22:52:10.758 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.4329999999999993e-06 2024-09-20T22:52:10.759 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.759 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.759 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:52:10.759 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:23.210130+0000", 2024-09-20T22:52:10.759 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.6286000000000003e-05 2024-09-20T22:52:10.759 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.759 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.759 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:52:10.759 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:23.210148+0000", 2024-09-20T22:52:10.759 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.7895999999999999e-05 2024-09-20T22:52:10.759 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.759 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.759 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:52:10.759 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:23.219532+0000", 2024-09-20T22:52:10.759 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.009383677 2024-09-20T22:52:10.760 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.760 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.760 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:52:10.760 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:23.219573+0000", 2024-09-20T22:52:10.760 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.1527999999999999e-05 2024-09-20T22:52:10.760 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:52:10.760 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:52:10.760 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:52:10.760 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.760 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.760 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:3760 7.13 e383/260 7:cae5792a:::smithi08477203-32:head v 383'78, mlcod=377'74)", 2024-09-20T22:52:10.760 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:47:43.291657+0000", 2024-09-20T22:52:10.760 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 267.42309383399999, 2024-09-20T22:52:10.760 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.011377195, 2024-09-20T22:52:10.760 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:52:10.760 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:52:10.760 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:52:10.761 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:52:10.761 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.761 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:52:10.761 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.291657+0000", 2024-09-20T22:52:10.761 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:52:10.761 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.761 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.761 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:52:10.761 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.291657+0000", 2024-09-20T22:52:10.761 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:52:10.761 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.761 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.761 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:52:10.761 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.291658+0000", 2024-09-20T22:52:10.762 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1009999999999999e-06 2024-09-20T22:52:10.762 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.762 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.762 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:52:10.762 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.296704+0000", 2024-09-20T22:52:10.762 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0050458070000000002 2024-09-20T22:52:10.762 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.762 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.762 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:52:10.762 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.296716+0000", 2024-09-20T22:52:10.762 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1525000000000001e-05 2024-09-20T22:52:10.762 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.762 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.762 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:52:10.762 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.296721+0000", 2024-09-20T22:52:10.762 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.8379999999999998e-06 2024-09-20T22:52:10.763 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.763 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.763 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:52:10.763 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.296769+0000", 2024-09-20T22:52:10.763 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.7120000000000003e-05 2024-09-20T22:52:10.763 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.763 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.763 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:52:10.763 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.296786+0000", 2024-09-20T22:52:10.763 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.7374e-05 2024-09-20T22:52:10.763 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.763 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.763 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:52:10.763 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.302998+0000", 2024-09-20T22:52:10.763 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0062118219999999997 2024-09-20T22:52:10.763 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.764 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.764 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:52:10.764 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.303034+0000", 2024-09-20T22:52:10.764 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.6607999999999999e-05 2024-09-20T22:52:10.764 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:52:10.764 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:52:10.764 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:52:10.764 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.764 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.764 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:4444 7.0 7:05bb442d:::smithi08477203-24:head [copy-from ver 421] snapc b8=[b8,b7,b6,b5,b4,b3,b2,b1,b0,ae,a9,a7,a6,a5,9e,9c,96,95,87,86] ondisk+write+known_if_redirected+supports_pool_eio e447)", 2024-09-20T22:52:10.764 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:48:47.227411+0000", 2024-09-20T22:52:10.764 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 203.487340268, 2024-09-20T22:52:10.764 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.033566952999999997, 2024-09-20T22:52:10.764 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:52:10.764 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:52:10.764 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:52:10.764 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:52:10.765 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:52:10.765 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:52:10.765 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 4444 2024-09-20T22:52:10.765 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.765 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:52:10.765 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.765 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:52:10.765 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.227411+0000", 2024-09-20T22:52:10.765 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:52:10.765 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.765 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.765 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:52:10.765 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.227411+0000", 2024-09-20T22:52:10.765 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:52:10.765 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.765 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.766 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:52:10.766 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.227413+0000", 2024-09-20T22:52:10.766 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.057e-06 2024-09-20T22:52:10.766 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.766 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.766 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:52:10.766 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.227422+0000", 2024-09-20T22:52:10.766 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.3219999999999999e-06 2024-09-20T22:52:10.766 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.766 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.766 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:52:10.766 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.227442+0000", 2024-09-20T22:52:10.766 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9449999999999998e-05 2024-09-20T22:52:10.766 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.767 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.767 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:52:10.767 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.227452+0000", 2024-09-20T22:52:10.767 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0699999999999999e-05 2024-09-20T22:52:10.767 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.767 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.767 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:52:10.767 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.227531+0000", 2024-09-20T22:52:10.767 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.8993999999999996e-05 2024-09-20T22:52:10.767 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.767 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.767 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op must wait for map", 2024-09-20T22:52:10.767 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.227542+0000", 2024-09-20T22:52:10.768 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0797e-05 2024-09-20T22:52:10.768 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.768 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.768 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:52:10.768 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.229501+0000", 2024-09-20T22:52:10.768 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0019594080000000002 2024-09-20T22:52:10.768 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.768 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.768 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:52:10.768 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.229616+0000", 2024-09-20T22:52:10.768 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000114577 2024-09-20T22:52:10.768 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.768 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.769 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 1,3", 2024-09-20T22:52:10.769 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.241752+0000", 2024-09-20T22:52:10.769 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.012135920999999999 2024-09-20T22:52:10.769 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.769 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.769 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:52:10.769 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.249601+0000", 2024-09-20T22:52:10.769 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0078493120000000006 2024-09-20T22:52:10.769 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.769 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.769 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:52:10.769 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.260731+0000", 2024-09-20T22:52:10.769 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.011129781 2024-09-20T22:52:10.769 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.769 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.769 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:52:10.770 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.260861+0000", 2024-09-20T22:52:10.770 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00013027899999999999 2024-09-20T22:52:10.770 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.770 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.770 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:52:10.770 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.260913+0000", 2024-09-20T22:52:10.770 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.2172999999999997e-05 2024-09-20T22:52:10.770 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.770 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.770 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:52:10.770 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.260978+0000", 2024-09-20T22:52:10.770 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.4182000000000003e-05 2024-09-20T22:52:10.770 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:52:10.770 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:52:10.771 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:52:10.771 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.771 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.771 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:4445 7.0 7:05bb442d:::smithi08477203-24:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e447)", 2024-09-20T22:52:10.771 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:48:47.227478+0000", 2024-09-20T22:52:10.771 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 203.48727333100001, 2024-09-20T22:52:10.771 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.019465758999999999, 2024-09-20T22:52:10.771 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:52:10.771 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:52:10.771 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:52:10.771 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:52:10.771 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:52:10.771 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:52:10.771 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 4445 2024-09-20T22:52:10.772 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.772 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:52:10.772 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.772 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:52:10.772 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.227478+0000", 2024-09-20T22:52:10.772 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:52:10.772 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.772 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.772 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:52:10.772 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.227478+0000", 2024-09-20T22:52:10.772 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:52:10.772 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.772 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.773 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:52:10.773 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.227478+0000", 2024-09-20T22:52:10.773 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.2200000000000004e-07 2024-09-20T22:52:10.773 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.773 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.773 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:52:10.773 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.227481+0000", 2024-09-20T22:52:10.773 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.0060000000000001e-06 2024-09-20T22:52:10.773 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.773 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.773 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:52:10.773 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.227500+0000", 2024-09-20T22:52:10.773 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9071000000000001e-05 2024-09-20T22:52:10.773 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.773 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.774 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:52:10.774 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.227507+0000", 2024-09-20T22:52:10.774 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.1779999999999997e-06 2024-09-20T22:52:10.774 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.774 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.774 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:52:10.774 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.227628+0000", 2024-09-20T22:52:10.774 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000120627 2024-09-20T22:52:10.774 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.774 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.774 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting_for_map not empty", 2024-09-20T22:52:10.774 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.227633+0000", 2024-09-20T22:52:10.774 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.4730000000000002e-06 2024-09-20T22:52:10.774 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.774 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.775 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:52:10.775 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.229766+0000", 2024-09-20T22:52:10.775 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0021332339999999999 2024-09-20T22:52:10.775 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.775 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.775 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for blocked object", 2024-09-20T22:52:10.775 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.229814+0000", 2024-09-20T22:52:10.775 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.7746e-05 2024-09-20T22:52:10.775 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.775 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.775 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:52:10.775 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.246707+0000", 2024-09-20T22:52:10.775 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.016893542000000001 2024-09-20T22:52:10.775 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.775 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.775 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:52:10.776 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.246863+0000", 2024-09-20T22:52:10.776 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000156105 2024-09-20T22:52:10.776 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.776 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.776 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:52:10.776 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.246943+0000", 2024-09-20T22:52:10.776 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.0154999999999994e-05 2024-09-20T22:52:10.776 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:52:10.776 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:52:10.776 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:52:10.776 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.776 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.776 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:5011 7.b e505/94 7:df7b2b73:::smithi08477203-10:head v 505'722, mlcod=503'718)", 2024-09-20T22:52:10.776 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:49:45.305974+0000", 2024-09-20T22:52:10.777 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 145.408777235, 2024-09-20T22:52:10.777 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.014321993, 2024-09-20T22:52:10.777 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:52:10.777 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:52:10.777 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:52:10.777 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:52:10.777 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.777 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:52:10.778 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:45.305974+0000", 2024-09-20T22:52:10.778 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:52:10.778 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.778 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.778 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:52:10.778 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:45.305974+0000", 2024-09-20T22:52:10.778 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:52:10.778 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.778 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.778 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:52:10.778 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:45.305974+0000", 2024-09-20T22:52:10.778 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.6199999999999997e-07 2024-09-20T22:52:10.778 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.779 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.779 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:52:10.779 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:45.311756+0000", 2024-09-20T22:52:10.779 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0057811019999999998 2024-09-20T22:52:10.779 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.779 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.779 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:52:10.779 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:45.311773+0000", 2024-09-20T22:52:10.779 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.7757999999999998e-05 2024-09-20T22:52:10.779 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.779 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.779 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:52:10.779 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:45.311783+0000", 2024-09-20T22:52:10.779 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.6639999999999993e-06 2024-09-20T22:52:10.779 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.779 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.780 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:52:10.780 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:45.311873+0000", 2024-09-20T22:52:10.780 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.0062000000000006e-05 2024-09-20T22:52:10.780 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.780 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.780 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:52:10.780 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:45.311900+0000", 2024-09-20T22:52:10.780 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.6475999999999999e-05 2024-09-20T22:52:10.780 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.780 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.780 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:52:10.780 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:45.320251+0000", 2024-09-20T22:52:10.780 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0083511560000000002 2024-09-20T22:52:10.780 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.780 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.781 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:52:10.781 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:45.320296+0000", 2024-09-20T22:52:10.781 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.5012999999999997e-05 2024-09-20T22:52:10.781 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:52:10.781 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:52:10.781 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:52:10.781 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.781 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.782 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:10 smithi084 ceph-mon[40024]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:52:10.782 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:10 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/330448921' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:52:10.782 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:10 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [3, 2]}]': finished 2024-09-20T22:52:10.782 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:10 smithi084 ceph-mon[40024]: osdmap e650: 8 total, 8 up, 7 in 2024-09-20T22:52:10.782 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:5151 7.0 7:05bb442d:::smithi08477203-24:head [write 749777~722868] snapc dd=[dd,dc,db,da,d9,d8,d7,d6,d5,d4,d3,d0,cf,ce,cd,ca,c7,c6,c5,c2,bf,ba,b4,b2,a7,9e,95,87] ondisk+write+known_if_redirected+supports_pool_eio e514)", 2024-09-20T22:52:10.782 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:49:54.638549+0000", 2024-09-20T22:52:10.783 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 136.07620209300001, 2024-09-20T22:52:10.783 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0092067450000000006, 2024-09-20T22:52:10.783 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:52:10.783 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:52:10.783 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:52:10.783 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:52:10.783 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:52:10.783 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:52:10.783 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 5151 2024-09-20T22:52:10.783 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.783 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:52:10.784 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.784 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:52:10.784 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:54.638549+0000", 2024-09-20T22:52:10.784 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:52:10.784 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.784 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.784 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:52:10.784 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:54.638549+0000", 2024-09-20T22:52:10.784 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:52:10.784 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.784 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.784 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:52:10.784 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:54.638551+0000", 2024-09-20T22:52:10.784 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9709999999999998e-06 2024-09-20T22:52:10.785 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.785 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.785 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:52:10.785 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:54.639792+0000", 2024-09-20T22:52:10.785 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0012410139999999999 2024-09-20T22:52:10.785 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.785 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.785 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:52:10.785 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:54.639813+0000", 2024-09-20T22:52:10.785 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.1148000000000001e-05 2024-09-20T22:52:10.785 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.785 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.785 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:52:10.785 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:54.639835+0000", 2024-09-20T22:52:10.785 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.1905000000000001e-05 2024-09-20T22:52:10.786 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.786 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.786 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:52:10.786 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:54.639901+0000", 2024-09-20T22:52:10.786 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.6548e-05 2024-09-20T22:52:10.786 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.786 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.786 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:52:10.786 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:54.639981+0000", 2024-09-20T22:52:10.786 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.9148999999999997e-05 2024-09-20T22:52:10.786 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.786 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.786 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 1,3", 2024-09-20T22:52:10.786 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:54.640098+0000", 2024-09-20T22:52:10.786 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000117337 2024-09-20T22:52:10.786 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.787 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.787 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:52:10.787 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:54.641401+0000", 2024-09-20T22:52:10.787 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0013027099999999999 2024-09-20T22:52:10.787 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.787 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.787 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:52:10.787 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:54.647561+0000", 2024-09-20T22:52:10.787 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.006160502 2024-09-20T22:52:10.787 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.787 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.787 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:52:10.787 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:54.647676+0000", 2024-09-20T22:52:10.787 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000114671 2024-09-20T22:52:10.787 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.787 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.787 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:52:10.788 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:54.647717+0000", 2024-09-20T22:52:10.788 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.1446000000000003e-05 2024-09-20T22:52:10.788 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.788 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.788 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:52:10.788 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:54.647756+0000", 2024-09-20T22:52:10.788 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.8343999999999997e-05 2024-09-20T22:52:10.788 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:52:10.788 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:52:10.788 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:52:10.788 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.788 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.788 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:5514 7.b e560/94 7:d3ca3619:::smithi08477203-39:head v 560'815, mlcod=559'812)", 2024-09-20T22:52:10.788 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:50:40.289035+0000", 2024-09-20T22:52:10.788 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 90.425715975000003, 2024-09-20T22:52:10.788 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0098817409999999994, 2024-09-20T22:52:10.789 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:52:10.789 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:52:10.789 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:52:10.789 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:52:10.789 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.789 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:52:10.789 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:40.289035+0000", 2024-09-20T22:52:10.789 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:52:10.789 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.789 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.789 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:52:10.789 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:40.289035+0000", 2024-09-20T22:52:10.789 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:52:10.789 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.789 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.789 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:52:10.789 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:40.289035+0000", 2024-09-20T22:52:10.790 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.8699999999999999e-07 2024-09-20T22:52:10.790 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.790 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.790 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:52:10.790 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:40.293026+0000", 2024-09-20T22:52:10.790 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.003991334 2024-09-20T22:52:10.790 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.790 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.790 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:52:10.790 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:40.293096+0000", 2024-09-20T22:52:10.790 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.9764999999999994e-05 2024-09-20T22:52:10.790 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.790 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.790 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:52:10.790 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:40.293107+0000", 2024-09-20T22:52:10.790 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.059e-05 2024-09-20T22:52:10.791 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.791 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.791 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:52:10.791 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:40.293195+0000", 2024-09-20T22:52:10.791 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.8052000000000003e-05 2024-09-20T22:52:10.791 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.791 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.791 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:52:10.791 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:40.293226+0000", 2024-09-20T22:52:10.791 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.1557999999999998e-05 2024-09-20T22:52:10.791 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.791 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.791 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:52:10.791 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:40.298884+0000", 2024-09-20T22:52:10.791 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0056575139999999998 2024-09-20T22:52:10.791 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.792 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.792 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:52:10.792 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:40.298917+0000", 2024-09-20T22:52:10.792 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.2740999999999997e-05 2024-09-20T22:52:10.792 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:52:10.792 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:52:10.792 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:52:10.792 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.792 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.792 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:5532 7.a e561/94 7:5aaef466:::smithi08477203-40:head v 561'565, mlcod=559'561)", 2024-09-20T22:52:10.792 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:50:41.456088+0000", 2024-09-20T22:52:10.792 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 89.258662845999993, 2024-09-20T22:52:10.792 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.009842557, 2024-09-20T22:52:10.792 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:52:10.792 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:52:10.792 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:52:10.792 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:52:10.793 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.793 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:52:10.793 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:41.456088+0000", 2024-09-20T22:52:10.793 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:52:10.793 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.793 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.793 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:52:10.793 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:41.456088+0000", 2024-09-20T22:52:10.793 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:52:10.793 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.793 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.793 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:52:10.793 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:41.456090+0000", 2024-09-20T22:52:10.793 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.916e-06 2024-09-20T22:52:10.793 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.794 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.794 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:52:10.794 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:41.459913+0000", 2024-09-20T22:52:10.794 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0038229879999999998 2024-09-20T22:52:10.794 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.794 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.794 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:52:10.794 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:41.459929+0000", 2024-09-20T22:52:10.794 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.6249e-05 2024-09-20T22:52:10.794 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.794 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.794 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:52:10.794 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:41.459938+0000", 2024-09-20T22:52:10.794 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.7390000000000005e-06 2024-09-20T22:52:10.794 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.794 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.795 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:52:10.795 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:41.460008+0000", 2024-09-20T22:52:10.795 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.9543999999999997e-05 2024-09-20T22:52:10.795 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.795 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.795 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:52:10.795 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:41.460034+0000", 2024-09-20T22:52:10.795 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.6341000000000001e-05 2024-09-20T22:52:10.795 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.795 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.795 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:52:10.795 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:41.465878+0000", 2024-09-20T22:52:10.795 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00584418 2024-09-20T22:52:10.795 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.795 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.796 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:52:10.796 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:41.465931+0000", 2024-09-20T22:52:10.796 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.2599999999999998e-05 2024-09-20T22:52:10.796 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:52:10.796 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:52:10.796 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:52:10.796 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.796 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.796 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:5659 7.b e570/94 7:dc95255e:::smithi08477203-1:head v 570'831, mlcod=570'827)", 2024-09-20T22:52:10.796 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:50:51.285735+0000", 2024-09-20T22:52:10.796 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 79.429015863000004, 2024-09-20T22:52:10.796 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.016481638999999999, 2024-09-20T22:52:10.796 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:52:10.796 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:52:10.797 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:52:10.797 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:52:10.797 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.797 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:52:10.797 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:51.285735+0000", 2024-09-20T22:52:10.797 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:52:10.797 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.797 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.797 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:52:10.797 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:51.285735+0000", 2024-09-20T22:52:10.797 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:52:10.797 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.797 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.797 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:52:10.798 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:51.285736+0000", 2024-09-20T22:52:10.798 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.5099999999999998e-07 2024-09-20T22:52:10.798 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.798 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.798 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:52:10.798 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:51.293158+0000", 2024-09-20T22:52:10.798 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0074220750000000002 2024-09-20T22:52:10.798 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.798 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.798 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:52:10.798 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:51.293175+0000", 2024-09-20T22:52:10.798 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.6401000000000001e-05 2024-09-20T22:52:10.798 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.798 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.798 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:52:10.798 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:51.293183+0000", 2024-09-20T22:52:10.799 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.8100000000000004e-06 2024-09-20T22:52:10.799 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.799 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.799 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:52:10.799 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:51.293262+0000", 2024-09-20T22:52:10.799 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.8527000000000002e-05 2024-09-20T22:52:10.799 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.799 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.799 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:52:10.799 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:51.293289+0000", 2024-09-20T22:52:10.799 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.7620999999999999e-05 2024-09-20T22:52:10.799 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.799 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.799 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:52:10.799 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:51.302173+0000", 2024-09-20T22:52:10.799 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0088831139999999992 2024-09-20T22:52:10.800 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.800 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.800 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:52:10.800 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:51.302217+0000", 2024-09-20T22:52:10.800 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.4140000000000001e-05 2024-09-20T22:52:10.800 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:52:10.800 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:52:10.800 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:52:10.800 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.800 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.800 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(osd.2.0:732 7.e e643/391 7:7c62ced4:::smithi08477203-17:10d v 643'576, mlcod=635'572)", 2024-09-20T22:52:10.800 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:52:03.284360+0000", 2024-09-20T22:52:10.800 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 7.4303910560000004, 2024-09-20T22:52:10.800 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.011646446, 2024-09-20T22:52:10.800 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:52:10.800 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:52:10.800 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:52:10.801 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:52:10.801 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.801 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:52:10.801 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:03.284360+0000", 2024-09-20T22:52:10.801 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:52:10.801 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.801 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.801 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:52:10.801 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:03.284360+0000", 2024-09-20T22:52:10.801 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:52:10.801 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.801 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.801 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:52:10.801 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:03.284360+0000", 2024-09-20T22:52:10.801 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.2899999999999999e-07 2024-09-20T22:52:10.802 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.802 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.802 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:52:10.802 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:03.284366+0000", 2024-09-20T22:52:10.802 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.0979999999999999e-06 2024-09-20T22:52:10.802 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.802 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.802 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:52:10.802 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:03.284373+0000", 2024-09-20T22:52:10.802 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.6889999999999998e-06 2024-09-20T22:52:10.802 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.802 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.802 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:52:10.803 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:03.284381+0000", 2024-09-20T22:52:10.803 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.8099999999999998e-06 2024-09-20T22:52:10.803 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.803 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.803 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:52:10.803 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:03.284442+0000", 2024-09-20T22:52:10.803 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.1328999999999995e-05 2024-09-20T22:52:10.803 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.803 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.804 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:52:10.804 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:03.284461+0000", 2024-09-20T22:52:10.804 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9239e-05 2024-09-20T22:52:10.804 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.804 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.804 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:52:10.804 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:03.295936+0000", 2024-09-20T22:52:10.805 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.011474544 2024-09-20T22:52:10.805 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.805 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.805 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:52:10.805 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:03.296006+0000", 2024-09-20T22:52:10.805 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.0308000000000005e-05 2024-09-20T22:52:10.805 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:52:10.805 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:52:10.805 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:52:10.805 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.805 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.805 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(osd.2.0:733 7.e e643/391 7:71b5c915:::smithi08477203-6:10c v 643'578, mlcod=635'572)", 2024-09-20T22:52:10.805 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:52:03.284684+0000", 2024-09-20T22:52:10.806 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 7.430066751, 2024-09-20T22:52:10.806 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.011653251999999999, 2024-09-20T22:52:10.806 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:52:10.806 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:52:10.806 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:52:10.806 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:52:10.806 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.807 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:52:10.807 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:03.284684+0000", 2024-09-20T22:52:10.807 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:52:10.807 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.807 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.807 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:52:10.807 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:03.284684+0000", 2024-09-20T22:52:10.807 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:52:10.807 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.807 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.807 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:52:10.807 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:03.284685+0000", 2024-09-20T22:52:10.807 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.4000000000000002e-07 2024-09-20T22:52:10.808 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.808 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.808 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:52:10.808 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:03.284696+0000", 2024-09-20T22:52:10.808 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1847e-05 2024-09-20T22:52:10.808 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.808 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.808 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:52:10.808 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:03.284707+0000", 2024-09-20T22:52:10.808 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0540000000000001e-05 2024-09-20T22:52:10.808 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.808 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.808 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:52:10.808 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:03.284713+0000", 2024-09-20T22:52:10.808 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.1959999999999996e-06 2024-09-20T22:52:10.808 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.809 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.809 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:52:10.809 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:03.285051+0000", 2024-09-20T22:52:10.809 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00033754400000000001 2024-09-20T22:52:10.809 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.809 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.809 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:52:10.809 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:03.285070+0000", 2024-09-20T22:52:10.809 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9451000000000001e-05 2024-09-20T22:52:10.809 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.809 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.809 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:52:10.809 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:03.296284+0000", 2024-09-20T22:52:10.809 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.011214277 2024-09-20T22:52:10.809 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:52:10.809 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:52:10.810 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:52:10.810 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:03.296337+0000", 2024-09-20T22:52:10.810 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.2957000000000001e-05 2024-09-20T22:52:10.810 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:52:10.810 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:52:10.810 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:52:10.810 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:52:10.810 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:52:10.810 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-20T22:52:10.810 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:52:10.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:10.833 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:10.833 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:52:10.934 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:52:10.949 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:52:10.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:10.957 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:10.957 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:52:11.058 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:52:11.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:11.080 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:11.080 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:52:11.181 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:52:11.202 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:11.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:11.203 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:52:11.281 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.7 to 1 (10000) 2024-09-20T22:52:11.281 INFO:tasks.rados.rados.0.smithi084.stdout:2558: done (4 left) 2024-09-20T22:52:11.282 INFO:tasks.rados.rados.0.smithi084.stdout:2559: rollback oid 28 current snap is 287 2024-09-20T22:52:11.282 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 28 to 281 2024-09-20T22:52:11.282 INFO:tasks.rados.rados.0.smithi084.stdout:2560: rollback oid 26 current snap is 287 2024-09-20T22:52:11.282 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 26 to 213 2024-09-20T22:52:11.282 INFO:tasks.rados.rados.0.smithi084.stdout:2561: snap_remove snap 273 2024-09-20T22:52:11.304 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:52:11.326 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:11.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:11.327 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:52:11.427 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:52:11.449 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:11.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:11.450 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:52:11.541 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-20T22:52:11.551 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:52:11.605 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:11.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:11.606 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:52:11.614 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:11 smithi084 ceph-mon[40024]: pgmap v876: 158 pgs: 5 active+clean+snaptrim, 1 active+remapped+backfill_toofull, 152 active+clean; 544 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 7.9 MiB/s rd, 3.3 MiB/s wr, 20 op/s; 23/1395 objects misplaced (1.649%) 2024-09-20T22:52:11.614 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:11 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/967383948' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-09-20T22:52:11.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:11 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-09-20T22:52:11.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:11 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-20T22:52:11.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:11 smithi084 ceph-mon[40024]: osdmap e651: 8 total, 8 up, 7 in 2024-09-20T22:52:11.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:11 smithi084 ceph-mon[29774]: pgmap v876: 158 pgs: 5 active+clean+snaptrim, 1 active+remapped+backfill_toofull, 152 active+clean; 544 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 7.9 MiB/s rd, 3.3 MiB/s wr, 20 op/s; 23/1395 objects misplaced (1.649%) 2024-09-20T22:52:11.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:11 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/967383948' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-09-20T22:52:11.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:11 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-09-20T22:52:11.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:11 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-20T22:52:11.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:11 smithi084 ceph-mon[29774]: osdmap e651: 8 total, 8 up, 7 in 2024-09-20T22:52:11.707 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:52:11.734 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:11.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:11.735 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:52:11.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:11 smithi154 ceph-mon[33264]: pgmap v876: 158 pgs: 5 active+clean+snaptrim, 1 active+remapped+backfill_toofull, 152 active+clean; 544 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 7.9 MiB/s rd, 3.3 MiB/s wr, 20 op/s; 23/1395 objects misplaced (1.649%) 2024-09-20T22:52:11.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:11 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/967383948' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-09-20T22:52:11.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:11 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-09-20T22:52:11.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:11 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-20T22:52:11.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:11 smithi154 ceph-mon[33264]: osdmap e651: 8 total, 8 up, 7 in 2024-09-20T22:52:11.787 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:52:11.835 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:52:11.856 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:11.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:11.857 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:52:11.958 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:52:11.979 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:11.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:11.980 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:52:12.081 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:52:12.102 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:12.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:12.103 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:52:12.204 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:52:12.226 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:12.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:12.227 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:52:12.230 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd reweight 0 1 2024-09-20T22:52:12.284 INFO:tasks.rados.rados.0.smithi084.stdout:2561: done (6 left) 2024-09-20T22:52:12.285 INFO:tasks.rados.rados.0.smithi084.stdout:2562: rollback oid 41 current snap is 287 2024-09-20T22:52:12.285 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 41 to 257 2024-09-20T22:52:12.285 INFO:tasks.rados.rados.0.smithi084.stdout:2563: delete oid 34 current snap is 287 2024-09-20T22:52:12.285 INFO:tasks.rados.rados.0.smithi084.stdout:2549: finishing copy_from to smithi08477203-21 2024-09-20T22:52:12.285 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 21 v 1227 (ObjNum 833 snap 269 seq_num 833) dirty exists 2024-09-20T22:52:12.285 INFO:tasks.rados.rados.0.smithi084.stdout:2553: finishing rollback tid 1 to smithi08477203-23 2024-09-20T22:52:12.285 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 23 v 1235 (ObjNum 33 snap 0 seq_num 2085656408) dirty dne 2024-09-20T22:52:12.285 INFO:tasks.rados.rados.0.smithi084.stdout:2552: finishing write tid 1 to smithi08477203-25 2024-09-20T22:52:12.285 INFO:tasks.rados.rados.0.smithi084.stdout:2552: finishing write tid 2 to smithi08477203-25 2024-09-20T22:52:12.285 INFO:tasks.rados.rados.0.smithi084.stdout:2552: finishing write tid 3 to smithi08477203-25 2024-09-20T22:52:12.285 INFO:tasks.rados.rados.0.smithi084.stdout:2552: finishing write tid 4 to smithi08477203-25 2024-09-20T22:52:12.285 INFO:tasks.rados.rados.0.smithi084.stdout:2552: finishing write tid 5 to smithi08477203-25 2024-09-20T22:52:12.285 INFO:tasks.rados.rados.0.smithi084.stdout:2552: oid 25 updating version 0 to 1912 2024-09-20T22:52:12.285 INFO:tasks.rados.rados.0.smithi084.stdout:2552: oid 25 version 1912 is already newer than 1911 2024-09-20T22:52:12.286 INFO:tasks.rados.rados.0.smithi084.stdout:2552: oid 25 version 1912 is already newer than 1909 2024-09-20T22:52:12.286 INFO:tasks.rados.rados.0.smithi084.stdout:2552: oid 25 version 1912 is already newer than 1910 2024-09-20T22:52:12.286 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 25 v 1912 (ObjNum 886 snap 287 seq_num 886) dirty exists 2024-09-20T22:52:12.286 INFO:tasks.rados.rados.0.smithi084.stdout:2552: left oid 25 (ObjNum 886 snap 287 seq_num 886) 2024-09-20T22:52:12.286 INFO:tasks.rados.rados.0.smithi084.stdout:2557: expect (ObjNum 863 snap 276 seq_num 863) 2024-09-20T22:52:12.286 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is unset 2024-09-20T22:52:12.307 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-20T22:52:12.307 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [], 2024-09-20T22:52:12.307 INFO:teuthology.orchestra.run.smithi154.stdout: "num_ops": 0 2024-09-20T22:52:12.307 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-20T22:52:12.328 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:52:12.349 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:12.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:12.350 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:52:12.450 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:52:12.462 INFO:tasks.rados.rados.0.smithi084.stdout:2559: finishing rollback tid 1 to smithi08477203-28 2024-09-20T22:52:12.462 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 28 v 1565 (ObjNum 289 snap 0 seq_num 1) dirty dne 2024-09-20T22:52:12.462 INFO:tasks.rados.rados.0.smithi084.stdout:2560: finishing rollback tid 0 to smithi08477203-26 2024-09-20T22:52:12.462 INFO:tasks.rados.rados.0.smithi084.stdout:2549: done (7 left) 2024-09-20T22:52:12.462 INFO:tasks.rados.rados.0.smithi084.stdout:2552: done (6 left) 2024-09-20T22:52:12.462 INFO:tasks.rados.rados.0.smithi084.stdout:2553: done (5 left) 2024-09-20T22:52:12.462 INFO:tasks.rados.rados.0.smithi084.stdout:2557: done (4 left) 2024-09-20T22:52:12.462 INFO:tasks.rados.rados.0.smithi084.stdout:2559: done (3 left) 2024-09-20T22:52:12.462 INFO:tasks.rados.rados.0.smithi084.stdout:2563: done (2 left) 2024-09-20T22:52:12.462 INFO:tasks.rados.rados.0.smithi084.stdout:2564: write oid 19 current snap is 287 2024-09-20T22:52:12.462 INFO:tasks.rados.rados.0.smithi084.stdout:2560: finishing rollback tid 1 to smithi08477203-26 2024-09-20T22:52:12.463 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 26 v 1340 (ObjNum 33 snap 0 seq_num 2091185344) dirty dne 2024-09-20T22:52:12.463 INFO:tasks.rados.rados.0.smithi084.stdout:2564: seq_num 888 ranges {524616=718060,1664010=523975,2752794=171822} 2024-09-20T22:52:12.467 INFO:tasks.rados.rados.0.smithi084.stdout:2564: writing smithi08477203-19 from 524616 to 1242676 tid 1 2024-09-20T22:52:12.468 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:12 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/2920754550' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:52:12.468 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:12 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:52:12.468 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:12 smithi154 ceph-mon[33264]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:52:12.468 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:12 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T22:52:12.468 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:12 smithi154 ceph-mon[33264]: osdmap e652: 8 total, 8 up, 7 in 2024-09-20T22:52:12.473 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:12.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:12.473 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:52:12.474 INFO:tasks.rados.rados.0.smithi084.stdout:2564: writing smithi08477203-19 from 1664010 to 2187985 tid 2 2024-09-20T22:52:12.478 INFO:tasks.rados.rados.0.smithi084.stdout:2564: writing smithi08477203-19 from 2752794 to 2924616 tid 3 2024-09-20T22:52:12.478 INFO:tasks.rados.rados.0.smithi084.stdout:2560: done (2 left) 2024-09-20T22:52:12.478 INFO:tasks.rados.rados.0.smithi084.stdout:2565: rollback oid 32 current snap is 287 2024-09-20T22:52:12.478 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 32 to 235 2024-09-20T22:52:12.478 INFO:tasks.rados.rados.0.smithi084.stdout:2562: finishing rollback tid 0 to smithi08477203-41 2024-09-20T22:52:12.478 INFO:tasks.rados.rados.0.smithi084.stdout:2562: finishing rollback tid 1 to smithi08477203-41 2024-09-20T22:52:12.478 INFO:tasks.rados.rados.0.smithi084.stdout:2562: finishing rollback tid 2 to smithi08477203-41 2024-09-20T22:52:12.478 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 41 v 1408 (ObjNum 779 snap 252 seq_num 779) dirty exists 2024-09-20T22:52:12.478 INFO:tasks.rados.rados.0.smithi084.stdout:2564: finishing write tid 1 to smithi08477203-19 2024-09-20T22:52:12.478 INFO:tasks.rados.rados.0.smithi084.stdout:2562: done (2 left) 2024-09-20T22:52:12.479 INFO:tasks.rados.rados.0.smithi084.stdout:2566: snap_create 2024-09-20T22:52:12.479 INFO:tasks.rados.rados.0.smithi084.stdout:2564: finishing write tid 2 to smithi08477203-19 2024-09-20T22:52:12.480 INFO:tasks.rados.rados.0.smithi084.stdout:2565: finishing rollback tid 0 to smithi08477203-32 2024-09-20T22:52:12.480 INFO:tasks.rados.rados.0.smithi084.stdout:2565: finishing rollback tid 1 to smithi08477203-32 2024-09-20T22:52:12.480 INFO:tasks.rados.rados.0.smithi084.stdout:2565: finishing rollback tid 2 to smithi08477203-32 2024-09-20T22:52:12.480 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 32 v 747 (ObjNum 466 snap 128 seq_num 466) dirty exists 2024-09-20T22:52:12.480 INFO:tasks.rados.rados.0.smithi084.stdout:2564: finishing write tid 3 to smithi08477203-19 2024-09-20T22:52:12.480 INFO:tasks.rados.rados.0.smithi084.stdout:2564: finishing write tid 4 to smithi08477203-19 2024-09-20T22:52:12.481 INFO:tasks.rados.rados.0.smithi084.stdout:2564: finishing write tid 5 to smithi08477203-19 2024-09-20T22:52:12.481 INFO:tasks.rados.rados.0.smithi084.stdout:2564: oid 19 updating version 0 to 1947 2024-09-20T22:52:12.481 INFO:tasks.rados.rados.0.smithi084.stdout:2564: oid 19 version 1947 is already newer than 1946 2024-09-20T22:52:12.481 INFO:tasks.rados.rados.0.smithi084.stdout:2564: oid 19 updating version 1947 to 1948 2024-09-20T22:52:12.481 INFO:tasks.rados.rados.0.smithi084.stdout:2564: oid 19 version 1948 is already newer than 1945 2024-09-20T22:52:12.481 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 19 v 1948 (ObjNum 888 snap 287 seq_num 888) dirty exists 2024-09-20T22:52:12.481 INFO:tasks.rados.rados.0.smithi084.stdout:2564: left oid 19 (ObjNum 888 snap 287 seq_num 888) 2024-09-20T22:52:12.574 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:52:12.594 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:12.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:12.595 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:52:12.696 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:52:12.717 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:12.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:12.718 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:52:12.818 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:52:12.834 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-20T22:52:12.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:12.841 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:12.841 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:52:12.855 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:12 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/2920754550' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:52:12.855 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:12 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:52:12.856 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:12 smithi084 ceph-mon[40024]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:52:12.856 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:12 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T22:52:12.856 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:12 smithi084 ceph-mon[40024]: osdmap e652: 8 total, 8 up, 7 in 2024-09-20T22:52:12.856 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:12 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/2920754550' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:52:12.856 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:12 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:52:12.856 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:12 smithi084 ceph-mon[29774]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:52:12.856 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:12 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T22:52:12.856 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:12 smithi084 ceph-mon[29774]: osdmap e652: 8 total, 8 up, 7 in 2024-09-20T22:52:12.899 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.6 dump_blocked_ops 2024-09-20T22:52:12.942 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:52:12.963 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:12.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:12.964 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:52:13.064 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:52:13.085 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:13.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:13.086 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:52:13.144 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:52:13.149 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:52:13.164 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:52:13.187 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:52:13.209 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:13.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:13.209 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:52:13.310 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:52:13.331 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:13.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:13.332 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:52:13.433 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:52:13.453 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:13.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:13.454 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:52:13.481 INFO:tasks.rados.rados.0.smithi084.stdout:2564: done (2 left) 2024-09-20T22:52:13.481 INFO:tasks.rados.rados.0.smithi084.stdout:2565: done (1 left) 2024-09-20T22:52:13.481 INFO:tasks.rados.rados.0.smithi084.stdout:2566: done (0 left) 2024-09-20T22:52:13.481 INFO:tasks.rados.rados.0.smithi084.stdout:2567: read oid 25 snap -1 2024-09-20T22:52:13.481 INFO:tasks.rados.rados.0.smithi084.stdout:2567: expect (ObjNum 886 snap 287 seq_num 886) 2024-09-20T22:52:13.481 INFO:tasks.rados.rados.0.smithi084.stdout:2568: copy_from oid 39 from oid 2 current snap is 288 2024-09-20T22:52:13.481 INFO:tasks.rados.rados.0.smithi084.stdout:2569: rollback oid 28 current snap is 288 2024-09-20T22:52:13.481 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 28 to 246 2024-09-20T22:52:13.481 INFO:tasks.rados.rados.0.smithi084.stdout:2570: read oid 15 snap -1 2024-09-20T22:52:13.481 INFO:tasks.rados.rados.0.smithi084.stdout:2570: expect deleted 2024-09-20T22:52:13.481 INFO:tasks.rados.rados.0.smithi084.stdout:2571: read oid 33 snap -1 2024-09-20T22:52:13.481 INFO:tasks.rados.rados.0.smithi084.stdout:2571: expect (ObjNum 862 snap 276 seq_num 862) 2024-09-20T22:52:13.481 INFO:tasks.rados.rados.0.smithi084.stdout:2572: rollback oid 24 current snap is 288 2024-09-20T22:52:13.481 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 24 to 279 2024-09-20T22:52:13.482 INFO:tasks.rados.rados.0.smithi084.stdout:2573: snap_create 2024-09-20T22:52:13.484 INFO:tasks.rados.rados.0.smithi084.stdout:2572: finishing rollback tid 0 to smithi08477203-24 2024-09-20T22:52:13.484 INFO:tasks.rados.rados.0.smithi084.stdout:2572: finishing rollback tid 1 to smithi08477203-24 2024-09-20T22:52:13.485 INFO:tasks.rados.rados.0.smithi084.stdout:2572: finishing rollback tid 2 to smithi08477203-24 2024-09-20T22:52:13.485 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 24 v 822 (ObjNum 780 snap 252 seq_num 780) dirty exists 2024-09-20T22:52:13.491 INFO:tasks.rados.rados.0.smithi084.stdout:2571: expect (ObjNum 862 snap 276 seq_num 862) 2024-09-20T22:52:13.543 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:13 smithi154 ceph-mon[33264]: pgmap v879: 158 pgs: 5 active+clean+snaptrim, 1 active+remapped+backfill_toofull, 152 active+clean; 544 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 7.9 MiB/s rd, 3.3 MiB/s wr, 20 op/s; 23/1395 objects misplaced (1.649%) 2024-09-20T22:52:13.555 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:52:13.575 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:13.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:13.576 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:52:13.649 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:13 smithi084 ceph-mon[40024]: pgmap v879: 158 pgs: 5 active+clean+snaptrim, 1 active+remapped+backfill_toofull, 152 active+clean; 544 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 7.9 MiB/s rd, 3.3 MiB/s wr, 20 op/s; 23/1395 objects misplaced (1.649%) 2024-09-20T22:52:13.649 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:13 smithi084 ceph-mon[29774]: pgmap v879: 158 pgs: 5 active+clean+snaptrim, 1 active+remapped+backfill_toofull, 152 active+clean; 544 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 7.9 MiB/s rd, 3.3 MiB/s wr, 20 op/s; 23/1395 objects misplaced (1.649%) 2024-09-20T22:52:13.659 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-20T22:52:13.659 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [], 2024-09-20T22:52:13.659 INFO:teuthology.orchestra.run.smithi154.stdout: "complaint_time": 30, 2024-09-20T22:52:13.660 INFO:teuthology.orchestra.run.smithi154.stdout: "num_blocked_ops": 0 2024-09-20T22:52:13.660 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-20T22:52:13.677 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:52:13.681 INFO:tasks.rados.rados.0.smithi084.stdout:2567: expect (ObjNum 886 snap 287 seq_num 886) 2024-09-20T22:52:13.700 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:13.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:13.700 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:52:13.801 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:52:13.809 INFO:tasks.rados.rados.0.smithi084.stdout:2569: finishing rollback tid 1 to smithi08477203-28 2024-09-20T22:52:13.809 INFO:tasks.rados.rados.0.smithi084.stdout:2569: finishing rollback tid 2 to smithi08477203-28 2024-09-20T22:52:13.809 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 28 v 1950 (ObjNum 751 snap 238 seq_num 751) dirty exists 2024-09-20T22:52:13.809 INFO:tasks.rados.rados.0.smithi084.stdout:2568: finishing copy_from racing read to smithi08477203-39 2024-09-20T22:52:13.809 INFO:tasks.rados.rados.0.smithi084.stdout:2568: finishing copy_from to smithi08477203-39 2024-09-20T22:52:13.809 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 39 v 1231 (ObjNum 857 snap 275 seq_num 857) dirty exists 2024-09-20T22:52:13.822 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:13.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:13.823 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:52:13.924 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:52:13.945 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:13.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:13.946 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:52:14.047 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:52:14.069 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:14.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:14.069 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:52:14.170 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:52:14.193 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:14.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:14.194 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:52:14.194 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:52:14.194 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:52:14.194 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:52:14.194 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:52:14.201 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:52:14.201 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:52:14.217 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-20T22:52:14.217 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T22:52:14.235 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:52:14.235 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:52:14.251 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:52:14.252 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:52:14.265 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.6 dump_historic_ops 2024-09-20T22:52:14.268 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:52:14.268 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:52:14.284 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:52:14.285 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:52:14.295 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:52:14.330 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:14.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:14.331 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:52:14.431 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:52:14.454 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:14.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:14.454 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:52:14.478 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.0 to 1 (10000) 2024-09-20T22:52:14.486 INFO:tasks.rados.rados.0.smithi084.stdout:2567: done (6 left) 2024-09-20T22:52:14.486 INFO:tasks.rados.rados.0.smithi084.stdout:2568: done (5 left) 2024-09-20T22:52:14.486 INFO:tasks.rados.rados.0.smithi084.stdout:2569: done (4 left) 2024-09-20T22:52:14.486 INFO:tasks.rados.rados.0.smithi084.stdout:2570: done (3 left) 2024-09-20T22:52:14.486 INFO:tasks.rados.rados.0.smithi084.stdout:2571: done (2 left) 2024-09-20T22:52:14.486 INFO:tasks.rados.rados.0.smithi084.stdout:2572: done (1 left) 2024-09-20T22:52:14.486 INFO:tasks.rados.rados.0.smithi084.stdout:2573: done (0 left) 2024-09-20T22:52:14.486 INFO:tasks.rados.rados.0.smithi084.stdout:2574: delete oid 14 current snap is 289 2024-09-20T22:52:14.487 INFO:tasks.rados.rados.0.smithi084.stdout:2574: done (0 left) 2024-09-20T22:52:14.487 INFO:tasks.rados.rados.0.smithi084.stdout:2575: rollback oid 31 current snap is 289 2024-09-20T22:52:14.487 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 31 to 257 2024-09-20T22:52:14.487 INFO:tasks.rados.rados.0.smithi084.stdout:2576: write oid 15 current snap is 289 2024-09-20T22:52:14.487 INFO:tasks.rados.rados.0.smithi084.stdout:2576: seq_num 890 ranges {558230=584911,1897327=527644,2923993=34237} 2024-09-20T22:52:14.494 INFO:tasks.rados.rados.0.smithi084.stdout:2576: writing smithi08477203-15 from 558230 to 1143141 tid 1 2024-09-20T22:52:14.501 INFO:tasks.rados.rados.0.smithi084.stdout:2576: writing smithi08477203-15 from 1897327 to 2424971 tid 2 2024-09-20T22:52:14.504 INFO:tasks.rados.rados.0.smithi084.stdout:2576: writing smithi08477203-15 from 2923993 to 2958230 tid 3 2024-09-20T22:52:14.504 INFO:tasks.rados.rados.0.smithi084.stdout:2577: rollback oid 3 current snap is 289 2024-09-20T22:52:14.504 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 3 to 274 2024-09-20T22:52:14.504 INFO:tasks.rados.rados.0.smithi084.stdout:2575: finishing rollback tid 1 to smithi08477203-31 2024-09-20T22:52:14.504 INFO:tasks.rados.rados.0.smithi084.stdout:2578: delete oid 13 current snap is 289 2024-09-20T22:52:14.505 INFO:tasks.rados.rados.0.smithi084.stdout:2575: finishing rollback tid 2 to smithi08477203-31 2024-09-20T22:52:14.505 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 31 v 1342 (ObjNum 688 snap 212 seq_num 688) dirty exists 2024-09-20T22:52:14.505 INFO:tasks.rados.rados.0.smithi084.stdout:2576: finishing write tid 1 to smithi08477203-15 2024-09-20T22:52:14.505 INFO:tasks.rados.rados.0.smithi084.stdout:2576: finishing write tid 2 to smithi08477203-15 2024-09-20T22:52:14.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:14 smithi154 ceph-mon[33264]: osdmap e653: 8 total, 8 up, 7 in 2024-09-20T22:52:14.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:14 smithi154 ceph-mon[33264]: 4.d scrub starts 2024-09-20T22:52:14.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:14 smithi154 ceph-mon[33264]: 4.d scrub ok 2024-09-20T22:52:14.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:14 smithi154 ceph-mon[33264]: 3.12 scrub starts 2024-09-20T22:52:14.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:14 smithi154 ceph-mon[33264]: 3.12 scrub ok 2024-09-20T22:52:14.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:14 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/4268502350' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-09-20T22:52:14.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:14 smithi154 ceph-mon[33264]: 4.a scrub starts 2024-09-20T22:52:14.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:14 smithi154 ceph-mon[33264]: 4.a scrub ok 2024-09-20T22:52:14.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:14 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/2709921031' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:52:14.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:14 smithi154 ceph-mon[33264]: purged_snaps scrub starts 2024-09-20T22:52:14.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:14 smithi154 ceph-mon[33264]: purged_snaps scrub ok 2024-09-20T22:52:14.508 INFO:tasks.rados.rados.0.smithi084.stdout:2576: finishing write tid 3 to smithi08477203-15 2024-09-20T22:52:14.508 INFO:tasks.rados.rados.0.smithi084.stdout:2576: finishing write tid 4 to smithi08477203-15 2024-09-20T22:52:14.508 INFO:tasks.rados.rados.0.smithi084.stdout:2576: finishing write tid 5 to smithi08477203-15 2024-09-20T22:52:14.508 INFO:tasks.rados.rados.0.smithi084.stdout:2576: oid 15 updating version 0 to 1814 2024-09-20T22:52:14.508 INFO:tasks.rados.rados.0.smithi084.stdout:2576: oid 15 version 1814 is already newer than 1813 2024-09-20T22:52:14.509 INFO:tasks.rados.rados.0.smithi084.stdout:2576: oid 15 updating version 1814 to 1815 2024-09-20T22:52:14.509 INFO:tasks.rados.rados.0.smithi084.stdout:2576: oid 15 version 1815 is already newer than 1812 2024-09-20T22:52:14.509 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 15 v 1815 (ObjNum 890 snap 289 seq_num 890) dirty exists 2024-09-20T22:52:14.509 INFO:tasks.rados.rados.0.smithi084.stdout:2576: left oid 15 (ObjNum 890 snap 289 seq_num 890) 2024-09-20T22:52:14.509 INFO:tasks.rados.rados.0.smithi084.stdout:2577: finishing rollback tid 0 to smithi08477203-3 2024-09-20T22:52:14.509 INFO:tasks.rados.rados.0.smithi084.stdout:2577: finishing rollback tid 1 to smithi08477203-3 2024-09-20T22:52:14.509 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 3 v 1817 (ObjNum 65 snap 0 seq_num 2091898880) dirty dne 2024-09-20T22:52:14.509 INFO:tasks.rados.rados.0.smithi084.stdout:2575: done (3 left) 2024-09-20T22:52:14.509 INFO:tasks.rados.rados.0.smithi084.stdout:2576: done (2 left) 2024-09-20T22:52:14.509 INFO:tasks.rados.rados.0.smithi084.stdout:2577: done (1 left) 2024-09-20T22:52:14.509 INFO:tasks.rados.rados.0.smithi084.stdout:2578: done (0 left) 2024-09-20T22:52:14.509 INFO:tasks.rados.rados.0.smithi084.stdout:2579: read oid 4 snap -1 2024-09-20T22:52:14.510 INFO:tasks.rados.rados.0.smithi084.stdout:2579: expect (ObjNum 863 snap 276 seq_num 863) 2024-09-20T22:52:14.510 INFO:tasks.rados.rados.0.smithi084.stdout:2580: snap_remove snap 259 2024-09-20T22:52:14.519 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:52:14.555 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:52:14.576 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:14.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:14.577 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:52:14.678 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:52:14.699 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:14.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:14.700 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:52:14.791 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:14 smithi084 ceph-mon[29774]: osdmap e653: 8 total, 8 up, 7 in 2024-09-20T22:52:14.791 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:14 smithi084 ceph-mon[29774]: 4.d scrub starts 2024-09-20T22:52:14.791 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:14 smithi084 ceph-mon[29774]: 4.d scrub ok 2024-09-20T22:52:14.791 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:14 smithi084 ceph-mon[29774]: 3.12 scrub starts 2024-09-20T22:52:14.791 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:14 smithi084 ceph-mon[29774]: 3.12 scrub ok 2024-09-20T22:52:14.791 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:14 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/4268502350' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-09-20T22:52:14.791 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:14 smithi084 ceph-mon[29774]: 4.a scrub starts 2024-09-20T22:52:14.791 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:14 smithi084 ceph-mon[29774]: 4.a scrub ok 2024-09-20T22:52:14.791 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:14 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/2709921031' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:52:14.791 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:14 smithi084 ceph-mon[29774]: purged_snaps scrub starts 2024-09-20T22:52:14.791 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:14 smithi084 ceph-mon[29774]: purged_snaps scrub ok 2024-09-20T22:52:14.792 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:14 smithi084 ceph-mon[40024]: osdmap e653: 8 total, 8 up, 7 in 2024-09-20T22:52:14.792 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:14 smithi084 ceph-mon[40024]: 4.d scrub starts 2024-09-20T22:52:14.792 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:14 smithi084 ceph-mon[40024]: 4.d scrub ok 2024-09-20T22:52:14.792 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:14 smithi084 ceph-mon[40024]: 3.12 scrub starts 2024-09-20T22:52:14.792 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:14 smithi084 ceph-mon[40024]: 3.12 scrub ok 2024-09-20T22:52:14.792 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:14 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/4268502350' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-09-20T22:52:14.792 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:14 smithi084 ceph-mon[40024]: 4.a scrub starts 2024-09-20T22:52:14.792 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:14 smithi084 ceph-mon[40024]: 4.a scrub ok 2024-09-20T22:52:14.792 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:14 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/2709921031' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:52:14.792 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:14 smithi084 ceph-mon[40024]: purged_snaps scrub starts 2024-09-20T22:52:14.792 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:14 smithi084 ceph-mon[40024]: purged_snaps scrub ok 2024-09-20T22:52:14.801 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:52:14.822 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:14.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:14.823 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:52:14.923 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:52:14.944 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-20T22:52:14.944 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:52:14.946 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:14.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:14.947 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:52:15.048 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:52:15.070 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:15.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:15.070 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:52:15.133 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-20T22:52:15.172 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:52:15.186 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd reweight 4 1 2024-09-20T22:52:15.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:15.194 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:15.194 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:52:15.295 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:52:15.316 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:15.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:15.317 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:52:15.417 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:52:15.438 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:15.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:15.439 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:52:15.492 INFO:tasks.rados.rados.0.smithi084.stdout:2580: done (1 left) 2024-09-20T22:52:15.492 INFO:tasks.rados.rados.0.smithi084.stdout:2581: delete oid 11 current snap is 289 2024-09-20T22:52:15.492 INFO:tasks.rados.rados.0.smithi084.stdout:2579: expect (ObjNum 863 snap 276 seq_num 863) 2024-09-20T22:52:15.540 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:52:15.561 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:15.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:15.562 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:52:15.624 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-20T22:52:15.655 INFO:tasks.rados.rados.0.smithi084.stdout:2579: done (1 left) 2024-09-20T22:52:15.655 INFO:tasks.rados.rados.0.smithi084.stdout:2581: done (0 left) 2024-09-20T22:52:15.655 INFO:tasks.rados.rados.0.smithi084.stdout:2582: snap_create 2024-09-20T22:52:15.663 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:52:15.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:15.684 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:15.684 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:52:15.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:15 smithi154 ceph-mon[33264]: pgmap v882: 158 pgs: 5 active+clean+snaptrim, 1 active+remapped+backfill_toofull, 152 active+clean; 556 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 9.5 MiB/s rd, 932 KiB/s wr, 12 op/s; 24/1401 objects misplaced (1.713%); 54 B/s, 1 objects/s recovering 2024-09-20T22:52:15.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:15 smithi154 ceph-mon[33264]: 3.4 scrub starts 2024-09-20T22:52:15.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:15 smithi154 ceph-mon[33264]: 3.4 scrub ok 2024-09-20T22:52:15.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:15 smithi154 ceph-mon[33264]: 4.17 scrub starts 2024-09-20T22:52:15.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:15 smithi154 ceph-mon[33264]: 4.17 scrub ok 2024-09-20T22:52:15.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:15 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/4268502350' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-20T22:52:15.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:15 smithi154 ceph-mon[33264]: osdmap e654: 8 total, 8 up, 7 in 2024-09-20T22:52:15.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:15 smithi154 ceph-mon[33264]: 5.18 scrub starts 2024-09-20T22:52:15.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:15 smithi154 ceph-mon[33264]: 5.18 scrub ok 2024-09-20T22:52:15.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:15 smithi154 ceph-mon[33264]: 7.6 scrub starts 2024-09-20T22:52:15.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:15 smithi154 ceph-mon[33264]: 7.6 scrub ok 2024-09-20T22:52:15.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:15 smithi154 ceph-mon[33264]: 4.f scrub starts 2024-09-20T22:52:15.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:15 smithi154 ceph-mon[33264]: 4.f scrub ok 2024-09-20T22:52:15.769 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:52:15.785 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:52:15.809 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:15.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:15.810 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:52:15.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:15 smithi084 ceph-mon[29774]: pgmap v882: 158 pgs: 5 active+clean+snaptrim, 1 active+remapped+backfill_toofull, 152 active+clean; 556 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 9.5 MiB/s rd, 932 KiB/s wr, 12 op/s; 24/1401 objects misplaced (1.713%); 54 B/s, 1 objects/s recovering 2024-09-20T22:52:15.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:15 smithi084 ceph-mon[29774]: 3.4 scrub starts 2024-09-20T22:52:15.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:15 smithi084 ceph-mon[29774]: 3.4 scrub ok 2024-09-20T22:52:15.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:15 smithi084 ceph-mon[29774]: 4.17 scrub starts 2024-09-20T22:52:15.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:15 smithi084 ceph-mon[29774]: 4.17 scrub ok 2024-09-20T22:52:15.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:15 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/4268502350' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-20T22:52:15.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:15 smithi084 ceph-mon[29774]: osdmap e654: 8 total, 8 up, 7 in 2024-09-20T22:52:15.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:15 smithi084 ceph-mon[29774]: 5.18 scrub starts 2024-09-20T22:52:15.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:15 smithi084 ceph-mon[29774]: 5.18 scrub ok 2024-09-20T22:52:15.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:15 smithi084 ceph-mon[29774]: 7.6 scrub starts 2024-09-20T22:52:15.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:15 smithi084 ceph-mon[29774]: 7.6 scrub ok 2024-09-20T22:52:15.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:15 smithi084 ceph-mon[29774]: 4.f scrub starts 2024-09-20T22:52:15.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:15 smithi084 ceph-mon[29774]: 4.f scrub ok 2024-09-20T22:52:15.866 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:15 smithi084 ceph-mon[40024]: pgmap v882: 158 pgs: 5 active+clean+snaptrim, 1 active+remapped+backfill_toofull, 152 active+clean; 556 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 9.5 MiB/s rd, 932 KiB/s wr, 12 op/s; 24/1401 objects misplaced (1.713%); 54 B/s, 1 objects/s recovering 2024-09-20T22:52:15.866 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:15 smithi084 ceph-mon[40024]: 3.4 scrub starts 2024-09-20T22:52:15.866 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:15 smithi084 ceph-mon[40024]: 3.4 scrub ok 2024-09-20T22:52:15.866 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:15 smithi084 ceph-mon[40024]: 4.17 scrub starts 2024-09-20T22:52:15.866 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:15 smithi084 ceph-mon[40024]: 4.17 scrub ok 2024-09-20T22:52:15.866 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:15 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/4268502350' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-20T22:52:15.866 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:15 smithi084 ceph-mon[40024]: osdmap e654: 8 total, 8 up, 7 in 2024-09-20T22:52:15.866 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:15 smithi084 ceph-mon[40024]: 5.18 scrub starts 2024-09-20T22:52:15.866 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:15 smithi084 ceph-mon[40024]: 5.18 scrub ok 2024-09-20T22:52:15.866 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:15 smithi084 ceph-mon[40024]: 7.6 scrub starts 2024-09-20T22:52:15.866 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:15 smithi084 ceph-mon[40024]: 7.6 scrub ok 2024-09-20T22:52:15.866 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:15 smithi084 ceph-mon[40024]: 4.f scrub starts 2024-09-20T22:52:15.866 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:15 smithi084 ceph-mon[40024]: 4.f scrub ok 2024-09-20T22:52:15.870 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:52:15.911 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:52:15.933 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:15.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:15.934 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:52:16.034 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:52:16.056 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:16.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:16.057 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:52:16.158 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:52:16.179 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:16.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:16.180 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:52:16.262 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:52:16.280 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:52:16.304 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:16.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:16.305 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:52:16.334 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-20T22:52:16.334 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:52:16.406 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:52:16.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:16.427 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:16.427 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:52:16.491 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.4 to 1 (10000) 2024-09-20T22:52:16.498 INFO:tasks.rados.rados.0.smithi084.stdout:2582: done (0 left) 2024-09-20T22:52:16.498 INFO:tasks.rados.rados.0.smithi084.stdout:2583: snap_remove snap 276 2024-09-20T22:52:16.527 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:52:16.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:16.550 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:16.550 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:52:16.651 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:52:16.671 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:16.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:16.672 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:52:16.672 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:16 smithi154 ceph-mon[33264]: purged_snaps scrub starts 2024-09-20T22:52:16.673 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:16 smithi154 ceph-mon[33264]: purged_snaps scrub ok 2024-09-20T22:52:16.673 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:16 smithi154 ceph-mon[33264]: osdmap e655: 8 total, 8 up, 7 in 2024-09-20T22:52:16.673 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:16 smithi154 ceph-mon[33264]: 5.9 scrub starts 2024-09-20T22:52:16.673 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:16 smithi154 ceph-mon[33264]: 5.9 scrub ok 2024-09-20T22:52:16.673 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:16 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/848246856' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-09-20T22:52:16.773 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:52:16.795 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:16.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:16.796 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:52:16.861 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:16 smithi084 ceph-mon[40024]: purged_snaps scrub starts 2024-09-20T22:52:16.861 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:16 smithi084 ceph-mon[40024]: purged_snaps scrub ok 2024-09-20T22:52:16.861 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:16 smithi084 ceph-mon[40024]: osdmap e655: 8 total, 8 up, 7 in 2024-09-20T22:52:16.861 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:16 smithi084 ceph-mon[40024]: 5.9 scrub starts 2024-09-20T22:52:16.861 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:16 smithi084 ceph-mon[40024]: 5.9 scrub ok 2024-09-20T22:52:16.861 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:16 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/848246856' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-09-20T22:52:16.861 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:16 smithi084 ceph-mon[29774]: purged_snaps scrub starts 2024-09-20T22:52:16.861 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:16 smithi084 ceph-mon[29774]: purged_snaps scrub ok 2024-09-20T22:52:16.861 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:16 smithi084 ceph-mon[29774]: osdmap e655: 8 total, 8 up, 7 in 2024-09-20T22:52:16.861 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:16 smithi084 ceph-mon[29774]: 5.9 scrub starts 2024-09-20T22:52:16.861 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:16 smithi084 ceph-mon[29774]: 5.9 scrub ok 2024-09-20T22:52:16.862 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:16 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/848246856' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-09-20T22:52:16.897 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:52:16.920 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:16.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:16.920 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:52:16.941 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.7 dump_blocked_ops 2024-09-20T22:52:17.021 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:52:17.042 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:17.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:17.043 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:52:17.143 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:52:17.164 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:17.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:17.165 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:52:17.225 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:52:17.266 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:52:17.289 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:17.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:17.290 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:52:17.391 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:52:17.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:17.413 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:17.413 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:52:17.499 INFO:tasks.rados.rados.0.smithi084.stdout:2583: done (0 left) 2024-09-20T22:52:17.499 INFO:tasks.rados.rados.0.smithi084.stdout:2584: read oid 44 snap 261 2024-09-20T22:52:17.499 INFO:tasks.rados.rados.0.smithi084.stdout:2584: expect (ObjNum 775 snap 248 seq_num 775) 2024-09-20T22:52:17.499 INFO:tasks.rados.rados.0.smithi084.stdout:2585: read oid 29 snap -1 2024-09-20T22:52:17.499 INFO:tasks.rados.rados.0.smithi084.stdout:2585: expect (ObjNum 873 snap 280 seq_num 873) 2024-09-20T22:52:17.499 INFO:tasks.rados.rados.0.smithi084.stdout:2586: rollback oid 9 current snap is 290 2024-09-20T22:52:17.499 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 9 to 283 2024-09-20T22:52:17.499 INFO:tasks.rados.rados.0.smithi084.stdout:2587: snap_create 2024-09-20T22:52:17.500 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is unset 2024-09-20T22:52:17.512 INFO:tasks.rados.rados.0.smithi084.stdout:2584: expect (ObjNum 775 snap 248 seq_num 775) 2024-09-20T22:52:17.514 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:52:17.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:17.538 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:17.538 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:52:17.549 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-09-20T22:52:17.549 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph pg dump --format=json 2024-09-20T22:52:17.610 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:17 smithi154 ceph-mon[33264]: pgmap v885: 158 pgs: 5 active+clean+snaptrim, 1 active+remapped+backfill_toofull, 152 active+clean; 556 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 9.5 MiB/s rd, 932 KiB/s wr, 12 op/s; 24/1401 objects misplaced (1.713%); 54 B/s, 1 objects/s recovering 2024-09-20T22:52:17.610 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:17 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/848246856' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-20T22:52:17.610 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:17 smithi154 ceph-mon[33264]: osdmap e656: 8 total, 8 up, 7 in 2024-09-20T22:52:17.610 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:17 smithi154 ceph-mon[33264]: 6.17 scrub starts 2024-09-20T22:52:17.610 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:17 smithi154 ceph-mon[33264]: 6.17 scrub ok 2024-09-20T22:52:17.611 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:17 smithi154 ceph-mon[33264]: 5.1a scrub starts 2024-09-20T22:52:17.611 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:17 smithi154 ceph-mon[33264]: 6.9 scrub starts 2024-09-20T22:52:17.611 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:17 smithi154 ceph-mon[33264]: 5.1a scrub ok 2024-09-20T22:52:17.611 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:17 smithi154 ceph-mon[33264]: 6.9 scrub ok 2024-09-20T22:52:17.611 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:17 smithi154 ceph-mon[33264]: 4.1f scrub starts 2024-09-20T22:52:17.611 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:17 smithi154 ceph-mon[33264]: 4.1f scrub ok 2024-09-20T22:52:17.611 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:17 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/217536820' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:52:17.611 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:17 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:52:17.639 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:52:17.651 INFO:tasks.rados.rados.0.smithi084.stdout:2585: expect (ObjNum 873 snap 280 seq_num 873) 2024-09-20T22:52:17.660 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:17.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:17.661 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:52:17.710 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-20T22:52:17.710 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:52:17.714 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-20T22:52:17.740 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:17 smithi084 ceph-mon[29774]: pgmap v885: 158 pgs: 5 active+clean+snaptrim, 1 active+remapped+backfill_toofull, 152 active+clean; 556 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 9.5 MiB/s rd, 932 KiB/s wr, 12 op/s; 24/1401 objects misplaced (1.713%); 54 B/s, 1 objects/s recovering 2024-09-20T22:52:17.740 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:17 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/848246856' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-20T22:52:17.740 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:17 smithi084 ceph-mon[29774]: osdmap e656: 8 total, 8 up, 7 in 2024-09-20T22:52:17.740 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:17 smithi084 ceph-mon[29774]: 6.17 scrub starts 2024-09-20T22:52:17.741 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:17 smithi084 ceph-mon[29774]: 6.17 scrub ok 2024-09-20T22:52:17.741 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:17 smithi084 ceph-mon[29774]: 5.1a scrub starts 2024-09-20T22:52:17.741 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:17 smithi084 ceph-mon[29774]: 6.9 scrub starts 2024-09-20T22:52:17.741 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:17 smithi084 ceph-mon[29774]: 5.1a scrub ok 2024-09-20T22:52:17.741 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:17 smithi084 ceph-mon[29774]: 6.9 scrub ok 2024-09-20T22:52:17.741 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:17 smithi084 ceph-mon[29774]: 4.1f scrub starts 2024-09-20T22:52:17.741 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:17 smithi084 ceph-mon[29774]: 4.1f scrub ok 2024-09-20T22:52:17.741 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:17 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/217536820' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:52:17.741 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:17 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:52:17.741 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:17 smithi084 ceph-mon[40024]: pgmap v885: 158 pgs: 5 active+clean+snaptrim, 1 active+remapped+backfill_toofull, 152 active+clean; 556 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 9.5 MiB/s rd, 932 KiB/s wr, 12 op/s; 24/1401 objects misplaced (1.713%); 54 B/s, 1 objects/s recovering 2024-09-20T22:52:17.741 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:17 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/848246856' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-20T22:52:17.741 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:17 smithi084 ceph-mon[40024]: osdmap e656: 8 total, 8 up, 7 in 2024-09-20T22:52:17.742 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:17 smithi084 ceph-mon[40024]: 6.17 scrub starts 2024-09-20T22:52:17.742 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:17 smithi084 ceph-mon[40024]: 6.17 scrub ok 2024-09-20T22:52:17.742 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:17 smithi084 ceph-mon[40024]: 5.1a scrub starts 2024-09-20T22:52:17.742 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:17 smithi084 ceph-mon[40024]: 6.9 scrub starts 2024-09-20T22:52:17.742 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:17 smithi084 ceph-mon[40024]: 5.1a scrub ok 2024-09-20T22:52:17.742 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:17 smithi084 ceph-mon[40024]: 6.9 scrub ok 2024-09-20T22:52:17.742 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:17 smithi084 ceph-mon[40024]: 4.1f scrub starts 2024-09-20T22:52:17.742 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:17 smithi084 ceph-mon[40024]: 4.1f scrub ok 2024-09-20T22:52:17.742 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:17 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/217536820' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:52:17.742 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:17 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:52:17.762 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:52:17.783 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:17.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:17.784 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:52:17.885 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:52:17.907 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:17.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:17.908 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:52:17.915 INFO:tasks.rados.rados.0.smithi084.stdout:2586: finishing rollback tid 0 to smithi08477203-9 2024-09-20T22:52:17.915 INFO:tasks.rados.rados.0.smithi084.stdout:2586: finishing rollback tid 1 to smithi08477203-9 2024-09-20T22:52:17.915 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 9 v 1802 (ObjNum 289 snap 0 seq_num 0) dirty dne 2024-09-20T22:52:18.009 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:52:18.030 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:18.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:18.031 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:52:18.131 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:52:18.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:18.153 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:18.153 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:52:18.254 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:52:18.273 INFO:tasks.rados.rados.0.smithi084.stdout:2584: done (3 left) 2024-09-20T22:52:18.274 INFO:tasks.rados.rados.0.smithi084.stdout:2585: done (2 left) 2024-09-20T22:52:18.274 INFO:tasks.rados.rados.0.smithi084.stdout:2586: done (1 left) 2024-09-20T22:52:18.274 INFO:tasks.rados.rados.0.smithi084.stdout:2587: done (0 left) 2024-09-20T22:52:18.274 INFO:tasks.rados.rados.0.smithi084.stdout:2588: snap_create 2024-09-20T22:52:18.279 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:18.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:18.281 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:52:18.308 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:52:18.345 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:52:18.351 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.7 dump_historic_ops 2024-09-20T22:52:18.381 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:52:18.403 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:18.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:18.403 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:52:18.504 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:52:18.507 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:52:18 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: ::ffff:172.21.15.154 - - [20/Sep/2024:22:52:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-20T22:52:18.525 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:18.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:18.526 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:52:18.597 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:52:18.627 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:52:18.648 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:18.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:18.649 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:52:18.718 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:18 smithi154 ceph-mon[33264]: 4.4 scrub starts 2024-09-20T22:52:18.718 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:18 smithi154 ceph-mon[33264]: 4.4 scrub ok 2024-09-20T22:52:18.718 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:18 smithi154 ceph-mon[33264]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T22:52:18.718 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:18 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T22:52:18.719 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:18 smithi154 ceph-mon[33264]: osdmap e657: 8 total, 8 up, 7 in 2024-09-20T22:52:18.719 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:18 smithi154 ceph-mon[33264]: osdmap e658: 8 total, 8 up, 7 in 2024-09-20T22:52:18.750 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:52:18.771 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:18.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:18.772 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:52:18.778 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:18 smithi084 ceph-mon[40024]: 4.4 scrub starts 2024-09-20T22:52:18.778 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:18 smithi084 ceph-mon[40024]: 4.4 scrub ok 2024-09-20T22:52:18.778 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:18 smithi084 ceph-mon[40024]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T22:52:18.778 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:18 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T22:52:18.778 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:18 smithi084 ceph-mon[40024]: osdmap e657: 8 total, 8 up, 7 in 2024-09-20T22:52:18.778 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:18 smithi084 ceph-mon[40024]: osdmap e658: 8 total, 8 up, 7 in 2024-09-20T22:52:18.778 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:18 smithi084 ceph-mon[29774]: 4.4 scrub starts 2024-09-20T22:52:18.778 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:18 smithi084 ceph-mon[29774]: 4.4 scrub ok 2024-09-20T22:52:18.778 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:18 smithi084 ceph-mon[29774]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T22:52:18.778 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:18 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T22:52:18.779 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:18 smithi084 ceph-mon[29774]: osdmap e657: 8 total, 8 up, 7 in 2024-09-20T22:52:18.779 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:18 smithi084 ceph-mon[29774]: osdmap e658: 8 total, 8 up, 7 in 2024-09-20T22:52:18.872 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:52:18.893 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:18.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:18.893 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:52:18.991 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:52:18.991 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:52:18.994 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:52:19.005 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:52:19.005 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:52:19.015 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:19.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:19.016 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:52:19.022 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:52:19.022 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:52:19.038 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-20T22:52:19.038 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T22:52:19.055 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:52:19.055 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:52:19.072 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:52:19.073 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:52:19.089 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:52:19.089 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:52:19.095 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-20T22:52:19.095 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:52:19.106 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:52:19.106 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:52:19.118 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:52:19.139 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:19.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:19.140 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:52:19.240 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:52:19.262 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:19.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:19.263 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:52:19.274 INFO:tasks.rados.rados.0.smithi084.stdout:2588: done (0 left) 2024-09-20T22:52:19.274 INFO:tasks.rados.rados.0.smithi084.stdout:2589: delete oid 46 current snap is 292 2024-09-20T22:52:19.276 INFO:tasks.rados.rados.0.smithi084.stdout:2589: done (0 left) 2024-09-20T22:52:19.277 INFO:tasks.rados.rados.0.smithi084.stdout:2590: delete oid 19 current snap is 292 2024-09-20T22:52:19.282 INFO:tasks.rados.rados.0.smithi084.stdout:2590: done (0 left) 2024-09-20T22:52:19.282 INFO:tasks.rados.rados.0.smithi084.stdout:2591: snap_remove snap 291 2024-09-20T22:52:19.363 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:52:19.384 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:19.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:19.385 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:52:19.473 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-20T22:52:19.475 INFO:teuthology.orchestra.run.smithi084.stderr:dumped all 2024-09-20T22:52:19.485 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:52:19.507 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:19.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:19.508 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:52:19.609 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:52:19.630 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:19.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:19.631 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:52:19.631 INFO:journalctl@ceph.iscsi.iscsi.a.smithi154.stdout:Sep 20 22:52:19 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-iscsi-iscsi-a[59730]: debug there is no tcmu-runner data available 2024-09-20T22:52:19.631 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:19 smithi154 ceph-mon[33264]: pgmap v888: 158 pgs: 5 active+clean+snaptrim, 1 active+remapped+backfill_toofull, 152 active+clean; 556 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 24/1401 objects misplaced (1.713%) 2024-09-20T22:52:19.631 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:19 smithi154 ceph-mon[33264]: 5.d scrub starts 2024-09-20T22:52:19.631 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:19 smithi154 ceph-mon[33264]: 5.d scrub ok 2024-09-20T22:52:19.632 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:19 smithi154 ceph-mon[33264]: 1.0 scrub starts 2024-09-20T22:52:19.632 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:19 smithi154 ceph-mon[33264]: 1.0 scrub ok 2024-09-20T22:52:19.632 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:19 smithi154 ceph-mon[33264]: 5.1e scrub starts 2024-09-20T22:52:19.632 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:19 smithi154 ceph-mon[33264]: 5.1e scrub ok 2024-09-20T22:52:19.632 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:19 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/1074783146' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:52:19.632 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:19 smithi154 ceph-mon[33264]: osdmap e659: 8 total, 8 up, 7 in 2024-09-20T22:52:19.657 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-20T22:52:19.679 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:19 smithi084 ceph-mon[40024]: pgmap v888: 158 pgs: 5 active+clean+snaptrim, 1 active+remapped+backfill_toofull, 152 active+clean; 556 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 24/1401 objects misplaced (1.713%) 2024-09-20T22:52:19.679 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:19 smithi084 ceph-mon[40024]: 5.d scrub starts 2024-09-20T22:52:19.679 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:19 smithi084 ceph-mon[40024]: 5.d scrub ok 2024-09-20T22:52:19.679 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:19 smithi084 ceph-mon[40024]: 1.0 scrub starts 2024-09-20T22:52:19.679 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:19 smithi084 ceph-mon[40024]: 1.0 scrub ok 2024-09-20T22:52:19.679 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:19 smithi084 ceph-mon[40024]: 5.1e scrub starts 2024-09-20T22:52:19.679 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:19 smithi084 ceph-mon[40024]: 5.1e scrub ok 2024-09-20T22:52:19.679 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:19 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/1074783146' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:52:19.679 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:19 smithi084 ceph-mon[40024]: osdmap e659: 8 total, 8 up, 7 in 2024-09-20T22:52:19.680 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:19 smithi084 ceph-mon[29774]: pgmap v888: 158 pgs: 5 active+clean+snaptrim, 1 active+remapped+backfill_toofull, 152 active+clean; 556 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 24/1401 objects misplaced (1.713%) 2024-09-20T22:52:19.680 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:19 smithi084 ceph-mon[29774]: 5.d scrub starts 2024-09-20T22:52:19.680 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:19 smithi084 ceph-mon[29774]: 5.d scrub ok 2024-09-20T22:52:19.680 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:19 smithi084 ceph-mon[29774]: 1.0 scrub starts 2024-09-20T22:52:19.680 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:19 smithi084 ceph-mon[29774]: 1.0 scrub ok 2024-09-20T22:52:19.680 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:19 smithi084 ceph-mon[29774]: 5.1e scrub starts 2024-09-20T22:52:19.680 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:19 smithi084 ceph-mon[29774]: 5.1e scrub ok 2024-09-20T22:52:19.680 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:19 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/1074783146' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:52:19.680 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:19 smithi084 ceph-mon[29774]: osdmap e659: 8 total, 8 up, 7 in 2024-09-20T22:52:19.732 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:52:19.752 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:19.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:19.753 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:52:19.854 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:52:19.874 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:19.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:19.876 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:52:19.976 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:52:19.999 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:20.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:20.000 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:52:20.101 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:52:20.123 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:20.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:20.124 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:52:20.220 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd set noscrub 2024-09-20T22:52:20.225 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:52:20.278 INFO:tasks.rados.rados.0.smithi084.stdout:2591: done (0 left) 2024-09-20T22:52:20.278 INFO:tasks.rados.rados.0.smithi084.stdout:2592: snap_create 2024-09-20T22:52:20.287 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:20.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:20.288 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:52:20.388 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:52:20.412 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:20.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:20.413 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:52:20.514 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:52:20.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:20.537 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:20.537 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:52:20.638 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:52:20.659 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:20.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:20.660 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:52:20.760 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:52:20.781 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:20.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:20.782 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:52:20.794 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:20 smithi084 ceph-mon[40024]: 4.2 scrub starts 2024-09-20T22:52:20.795 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:20 smithi084 ceph-mon[40024]: 4.2 scrub ok 2024-09-20T22:52:20.795 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:20 smithi084 ceph-mon[40024]: 6.14 scrub starts 2024-09-20T22:52:20.795 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:20 smithi084 ceph-mon[40024]: 6.14 scrub ok 2024-09-20T22:52:20.795 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:20 smithi084 ceph-mon[40024]: from='client.26510 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T22:52:20.795 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:20 smithi084 ceph-mon[40024]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:52:20.795 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:20 smithi084 ceph-mon[40024]: purged_snaps scrub starts 2024-09-20T22:52:20.795 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:20 smithi084 ceph-mon[40024]: purged_snaps scrub ok 2024-09-20T22:52:20.795 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:20 smithi084 ceph-mon[40024]: osdmap e660: 8 total, 8 up, 7 in 2024-09-20T22:52:20.795 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:20 smithi084 ceph-mon[40024]: 5.17 scrub starts 2024-09-20T22:52:20.795 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:20 smithi084 ceph-mon[40024]: 5.17 scrub ok 2024-09-20T22:52:20.796 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:20 smithi084 ceph-mon[29774]: 4.2 scrub starts 2024-09-20T22:52:20.796 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:20 smithi084 ceph-mon[29774]: 4.2 scrub ok 2024-09-20T22:52:20.796 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:20 smithi084 ceph-mon[29774]: 6.14 scrub starts 2024-09-20T22:52:20.796 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:20 smithi084 ceph-mon[29774]: 6.14 scrub ok 2024-09-20T22:52:20.796 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:20 smithi084 ceph-mon[29774]: from='client.26510 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T22:52:20.796 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:20 smithi084 ceph-mon[29774]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:52:20.796 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:20 smithi084 ceph-mon[29774]: purged_snaps scrub starts 2024-09-20T22:52:20.796 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:20 smithi084 ceph-mon[29774]: purged_snaps scrub ok 2024-09-20T22:52:20.796 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:20 smithi084 ceph-mon[29774]: osdmap e660: 8 total, 8 up, 7 in 2024-09-20T22:52:20.796 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:20 smithi084 ceph-mon[29774]: 5.17 scrub starts 2024-09-20T22:52:20.796 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:20 smithi084 ceph-mon[29774]: 5.17 scrub ok 2024-09-20T22:52:20.820 INFO:teuthology.orchestra.run.smithi084.stdout:{"pg_ready":true,"pg_map":{"version":890,"stamp":"2024-09-20T22:52:19.270621+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":587895311,"num_objects":470,"num_object_clones":221,"num_object_copies":1410,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":24,"num_objects_unfound":0,"num_objects_dirty":470,"num_whiteouts":12,"num_read":6287,"num_read_kb":4754951,"num_write":3940,"num_write_kb":1197176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1118,"num_bytes_recovered":463656067,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9696,"ondisk_log_size":9696,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":474,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":1594632,"kb_used_data":1255512,"kb_used_omap":10,"kb_used_meta":338933,"kb_avail":654621432,"statfs":{"total":671965249536,"available":670332346368,"internally_reserved":0,"allocated":1285644288,"data_stored":1842563552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11157,"internal_metadata":347067499},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":4723778,"num_objects":3,"num_object_clones":3,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":2,"num_read":33,"num_read_kb":37463,"num_write":12,"num_write_kb":1122,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.783260"},"pg_stats":[{"pgid":"3.1f","version":"86'2","reported_seq":563,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-09-20T22:52:18.273705+0000","last_change":"2024-09-20T22:49:26.647633+0000","last_active":"2024-09-20T22:52:18.273705+0000","last_peered":"2024-09-20T22:52:18.273705+0000","last_clean":"2024-09-20T22:52:18.273705+0000","last_became_active":"2024-09-20T22:47:49.222114+0000","last_became_peered":"2024-09-20T22:47:49.222114+0000","last_unstale":"2024-09-20T22:52:18.273705+0000","last_undegraded":"2024-09-20T22:52:18.273705+0000","last_fullsized":"2024-09-20T22:52:18.273705+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'2","last_scrub_stamp":"2024-09-20T22:49:26.647557+0000","last_deep_scrub":"86'2","last_deep_scrub_stamp":"2024-09-20T22:42:39.664421+0000","last_clean_scrub_stamp":"2024-09-20T22:49:26.647557+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":436,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"625'15","reported_seq":569,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-09-20T22:52:18.270440+0000","last_change":"2024-09-20T22:51:51.813358+0000","last_active":"2024-09-20T22:52:18.270440+0000","last_peered":"2024-09-20T22:52:18.270440+0000","last_clean":"2024-09-20T22:52:18.270440+0000","last_became_active":"2024-09-20T22:51:20.286651+0000","last_became_peered":"2024-09-20T22:51:20.286651+0000","last_unstale":"2024-09-20T22:52:18.270440+0000","last_undegraded":"2024-09-20T22:52:18.270440+0000","last_fullsized":"2024-09-20T22:52:18.270440+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'15","last_scrub_stamp":"2024-09-20T22:51:51.813269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:51:51.813269+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:09.354334+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":128,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-09-20T22:52:18.439906+0000","last_change":"2024-09-20T22:51:20.291625+0000","last_active":"2024-09-20T22:52:18.439906+0000","last_peered":"2024-09-20T22:52:18.439906+0000","last_clean":"2024-09-20T22:52:18.439906+0000","last_became_active":"2024-09-20T22:51:20.291272+0000","last_became_peered":"2024-09-20T22:51:20.291272+0000","last_unstale":"2024-09-20T22:52:18.439906+0000","last_undegraded":"2024-09-20T22:52:18.439906+0000","last_fullsized":"2024-09-20T22:52:18.439906+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:50:12.718677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:50:12.718677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":540,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-09-20T22:52:18.270556+0000","last_change":"2024-09-20T22:51:36.730087+0000","last_active":"2024-09-20T22:52:18.270556+0000","last_peered":"2024-09-20T22:52:18.270556+0000","last_clean":"2024-09-20T22:52:18.270556+0000","last_became_active":"2024-09-20T22:51:20.295608+0000","last_became_peered":"2024-09-20T22:51:20.295608+0000","last_unstale":"2024-09-20T22:52:18.270556+0000","last_undegraded":"2024-09-20T22:52:18.270556+0000","last_fullsized":"2024-09-20T22:52:18.270556+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:36.730015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:51:36.730015+0000","last_clean_scrub_stamp":"2024-09-20T22:51:36.730015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:41.980409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":1214,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-09-20T22:52:18.304342+0000","last_change":"2024-09-20T22:51:01.404127+0000","last_active":"2024-09-20T22:52:18.304342+0000","last_peered":"2024-09-20T22:52:18.304342+0000","last_clean":"2024-09-20T22:52:18.304342+0000","last_became_active":"2024-09-20T22:41:44.832099+0000","last_became_peered":"2024-09-20T22:41:44.832099+0000","last_unstale":"2024-09-20T22:52:18.304342+0000","last_undegraded":"2024-09-20T22:52:18.304342+0000","last_fullsized":"2024-09-20T22:52:18.304342+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:01.404063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:51:01.404063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":1219,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-09-20T22:52:18.305570+0000","last_change":"2024-09-20T22:51:12.418041+0000","last_active":"2024-09-20T22:52:18.305570+0000","last_peered":"2024-09-20T22:52:18.305570+0000","last_clean":"2024-09-20T22:52:18.305570+0000","last_became_active":"2024-09-20T22:41:38.157298+0000","last_became_peered":"2024-09-20T22:41:38.157298+0000","last_unstale":"2024-09-20T22:52:18.305570+0000","last_undegraded":"2024-09-20T22:52:18.305570+0000","last_fullsized":"2024-09-20T22:52:18.305570+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:12.417920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:51:12.417920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:37.156764+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"625'27","reported_seq":1301,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-09-20T22:52:18.304441+0000","last_change":"2024-09-20T22:51:08.441927+0000","last_active":"2024-09-20T22:52:18.304441+0000","last_peered":"2024-09-20T22:52:18.304441+0000","last_clean":"2024-09-20T22:52:18.304441+0000","last_became_active":"2024-09-20T22:47:49.218798+0000","last_became_peered":"2024-09-20T22:47:49.218798+0000","last_unstale":"2024-09-20T22:52:18.304441+0000","last_undegraded":"2024-09-20T22:52:18.304441+0000","last_fullsized":"2024-09-20T22:52:18.304441+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-09-20T22:51:08.441788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:51:08.441788+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:25.517854+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":540,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-09-20T22:52:18.270315+0000","last_change":"2024-09-20T22:52:14.594895+0000","last_active":"2024-09-20T22:52:18.270315+0000","last_peered":"2024-09-20T22:52:18.270315+0000","last_clean":"2024-09-20T22:52:18.270315+0000","last_became_active":"2024-09-20T22:51:20.287060+0000","last_became_peered":"2024-09-20T22:51:20.287060+0000","last_unstale":"2024-09-20T22:52:18.270315+0000","last_undegraded":"2024-09-20T22:52:18.270315+0000","last_fullsized":"2024-09-20T22:52:18.270315+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:14.594823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:52:14.594823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:30.113157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":1239,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-09-20T22:52:18.440848+0000","last_change":"2024-09-20T22:51:09.684125+0000","last_active":"2024-09-20T22:52:18.440848+0000","last_peered":"2024-09-20T22:52:18.440848+0000","last_clean":"2024-09-20T22:52:18.440848+0000","last_became_active":"2024-09-20T22:48:05.242059+0000","last_became_peered":"2024-09-20T22:48:05.242059+0000","last_unstale":"2024-09-20T22:52:18.440848+0000","last_undegraded":"2024-09-20T22:52:18.440848+0000","last_fullsized":"2024-09-20T22:52:18.440848+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:09.684053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:46:00.999939+0000","last_clean_scrub_stamp":"2024-09-20T22:51:09.684053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:35.986257+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"625'15","reported_seq":553,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-09-20T22:52:18.270588+0000","last_change":"2024-09-20T22:51:30.710329+0000","last_active":"2024-09-20T22:52:18.270588+0000","last_peered":"2024-09-20T22:52:18.270588+0000","last_clean":"2024-09-20T22:52:18.270588+0000","last_became_active":"2024-09-20T22:48:05.233429+0000","last_became_peered":"2024-09-20T22:48:05.233429+0000","last_unstale":"2024-09-20T22:52:18.270588+0000","last_undegraded":"2024-09-20T22:52:18.270588+0000","last_fullsized":"2024-09-20T22:52:18.270588+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-09-20T22:51:30.710245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:51:30.710245+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:39.082734+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":1231,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-09-20T22:52:18.440384+0000","last_change":"2024-09-20T22:51:33.977328+0000","last_active":"2024-09-20T22:52:18.440384+0000","last_peered":"2024-09-20T22:52:18.440384+0000","last_clean":"2024-09-20T22:52:18.440384+0000","last_became_active":"2024-09-20T22:47:49.227379+0000","last_became_peered":"2024-09-20T22:47:49.227379+0000","last_unstale":"2024-09-20T22:52:18.440384+0000","last_undegraded":"2024-09-20T22:52:18.440384+0000","last_fullsized":"2024-09-20T22:52:18.440384+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:33.977257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:51:33.977257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:37.924731+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":572,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-09-20T22:52:18.272383+0000","last_change":"2024-09-20T22:51:35.899942+0000","last_active":"2024-09-20T22:52:18.272383+0000","last_peered":"2024-09-20T22:52:18.272383+0000","last_clean":"2024-09-20T22:52:18.272383+0000","last_became_active":"2024-09-20T22:51:20.277465+0000","last_became_peered":"2024-09-20T22:51:20.277465+0000","last_unstale":"2024-09-20T22:52:18.272383+0000","last_undegraded":"2024-09-20T22:52:18.272383+0000","last_fullsized":"2024-09-20T22:52:18.272383+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:35.899860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:49:08.664407+0000","last_clean_scrub_stamp":"2024-09-20T22:51:35.899860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:38.548132+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":1245,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-09-20T22:52:18.440484+0000","last_change":"2024-09-20T22:51:20.298096+0000","last_active":"2024-09-20T22:52:18.440484+0000","last_peered":"2024-09-20T22:52:18.440484+0000","last_clean":"2024-09-20T22:52:18.440484+0000","last_became_active":"2024-09-20T22:51:20.297787+0000","last_became_peered":"2024-09-20T22:51:20.297787+0000","last_unstale":"2024-09-20T22:52:18.440484+0000","last_undegraded":"2024-09-20T22:52:18.440484+0000","last_fullsized":"2024-09-20T22:52:18.440484+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:50:17.735354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:50:17.735354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"625'9","reported_seq":555,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-09-20T22:52:18.270805+0000","last_change":"2024-09-20T22:51:20.301034+0000","last_active":"2024-09-20T22:52:18.270805+0000","last_peered":"2024-09-20T22:52:18.270805+0000","last_clean":"2024-09-20T22:52:18.270805+0000","last_became_active":"2024-09-20T22:51:20.285795+0000","last_became_peered":"2024-09-20T22:51:20.285795+0000","last_unstale":"2024-09-20T22:52:18.270805+0000","last_undegraded":"2024-09-20T22:52:18.270805+0000","last_fullsized":"2024-09-20T22:52:18.270805+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"181'7","last_scrub_stamp":"2024-09-20T22:51:05.899205+0000","last_deep_scrub":"181'7","last_deep_scrub_stamp":"2024-09-20T22:48:41.049381+0000","last_clean_scrub_stamp":"2024-09-20T22:51:05.899205+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":9,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":1257,"reported_epoch":657,"state":"active+clean","last_fresh":"2024-09-20T22:52:17.496759+0000","last_change":"2024-09-20T22:52:16.705733+0000","last_active":"2024-09-20T22:52:17.496759+0000","last_peered":"2024-09-20T22:52:17.496759+0000","last_clean":"2024-09-20T22:52:17.496759+0000","last_became_active":"2024-09-20T22:51:20.285368+0000","last_became_peered":"2024-09-20T22:51:20.285368+0000","last_unstale":"2024-09-20T22:52:17.496759+0000","last_undegraded":"2024-09-20T22:52:17.496759+0000","last_fullsized":"2024-09-20T22:52:17.496759+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:16.705645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:47:40.361958+0000","last_clean_scrub_stamp":"2024-09-20T22:52:16.705645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:40.490856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":1224,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-09-20T22:52:18.439944+0000","last_change":"2024-09-20T22:51:20.279625+0000","last_active":"2024-09-20T22:52:18.439944+0000","last_peered":"2024-09-20T22:52:18.439944+0000","last_clean":"2024-09-20T22:52:18.439944+0000","last_became_active":"2024-09-20T22:51:20.279307+0000","last_became_peered":"2024-09-20T22:51:20.279307+0000","last_unstale":"2024-09-20T22:52:18.439944+0000","last_undegraded":"2024-09-20T22:52:18.439944+0000","last_fullsized":"2024-09-20T22:52:18.439944+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:04.636900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:51:04.636900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":127,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-09-20T22:52:18.271096+0000","last_change":"2024-09-20T22:51:20.282749+0000","last_active":"2024-09-20T22:52:18.271096+0000","last_peered":"2024-09-20T22:52:18.271096+0000","last_clean":"2024-09-20T22:52:18.271096+0000","last_became_active":"2024-09-20T22:51:20.282393+0000","last_became_peered":"2024-09-20T22:51:20.282393+0000","last_unstale":"2024-09-20T22:52:18.271096+0000","last_undegraded":"2024-09-20T22:52:18.271096+0000","last_fullsized":"2024-09-20T22:52:18.271096+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:50:16.816835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:50:16.816835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":564,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-09-20T22:52:18.273118+0000","last_change":"2024-09-20T22:49:53.933673+0000","last_active":"2024-09-20T22:52:18.273118+0000","last_peered":"2024-09-20T22:52:18.273118+0000","last_clean":"2024-09-20T22:52:18.273118+0000","last_became_active":"2024-09-20T22:48:05.233687+0000","last_became_peered":"2024-09-20T22:48:05.233687+0000","last_unstale":"2024-09-20T22:52:18.273118+0000","last_undegraded":"2024-09-20T22:52:18.273118+0000","last_fullsized":"2024-09-20T22:52:18.273118+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:53.933601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:49:53.933601+0000","last_clean_scrub_stamp":"2024-09-20T22:49:53.933601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"625'34","reported_seq":1310,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-09-20T22:52:18.440616+0000","last_change":"2024-09-20T22:51:20.289052+0000","last_active":"2024-09-20T22:52:18.440616+0000","last_peered":"2024-09-20T22:52:18.440616+0000","last_clean":"2024-09-20T22:52:18.440616+0000","last_became_active":"2024-09-20T22:51:20.276338+0000","last_became_peered":"2024-09-20T22:51:20.276338+0000","last_unstale":"2024-09-20T22:52:18.440616+0000","last_undegraded":"2024-09-20T22:52:18.440616+0000","last_fullsized":"2024-09-20T22:52:18.440616+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'22","last_scrub_stamp":"2024-09-20T22:51:08.280738+0000","last_deep_scrub":"113'22","last_deep_scrub_stamp":"2024-09-20T22:43:12.191878+0000","last_clean_scrub_stamp":"2024-09-20T22:51:08.280738+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:26.717275+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":839,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-09-20T22:52:18.270912+0000","last_change":"2024-09-20T22:51:20.295989+0000","last_active":"2024-09-20T22:52:18.270912+0000","last_peered":"2024-09-20T22:52:18.270912+0000","last_clean":"2024-09-20T22:52:18.270912+0000","last_became_active":"2024-09-20T22:51:20.295564+0000","last_became_peered":"2024-09-20T22:51:20.295564+0000","last_unstale":"2024-09-20T22:52:18.270912+0000","last_undegraded":"2024-09-20T22:52:18.270912+0000","last_fullsized":"2024-09-20T22:52:18.270912+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:50:40.784154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:50:40.784154+0000","last_clean_scrub_stamp":"2024-09-20T22:50:40.784154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":1207,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-09-20T22:52:18.305836+0000","last_change":"2024-09-20T22:51:13.385618+0000","last_active":"2024-09-20T22:52:18.305836+0000","last_peered":"2024-09-20T22:52:18.305836+0000","last_clean":"2024-09-20T22:52:18.305836+0000","last_became_active":"2024-09-20T22:41:44.832131+0000","last_became_peered":"2024-09-20T22:41:44.832131+0000","last_unstale":"2024-09-20T22:52:18.305836+0000","last_undegraded":"2024-09-20T22:52:18.305836+0000","last_fullsized":"2024-09-20T22:52:18.305836+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:13.385520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:45:25.002199+0000","last_clean_scrub_stamp":"2024-09-20T22:51:13.385520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":540,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-09-20T22:52:18.270735+0000","last_change":"2024-09-20T22:51:20.285019+0000","last_active":"2024-09-20T22:52:18.270735+0000","last_peered":"2024-09-20T22:52:18.270735+0000","last_clean":"2024-09-20T22:52:18.270735+0000","last_became_active":"2024-09-20T22:51:20.284621+0000","last_became_peered":"2024-09-20T22:51:20.284621+0000","last_unstale":"2024-09-20T22:52:18.270735+0000","last_undegraded":"2024-09-20T22:52:18.270735+0000","last_fullsized":"2024-09-20T22:52:18.270735+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:50:41.885844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:50:41.885844+0000","last_clean_scrub_stamp":"2024-09-20T22:50:41.885844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"625'18","reported_seq":1318,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-09-20T22:52:18.305103+0000","last_change":"2024-09-20T22:51:20.300828+0000","last_active":"2024-09-20T22:52:18.305103+0000","last_peered":"2024-09-20T22:52:18.305103+0000","last_clean":"2024-09-20T22:52:18.305103+0000","last_became_active":"2024-09-20T22:51:20.278886+0000","last_became_peered":"2024-09-20T22:51:20.278886+0000","last_unstale":"2024-09-20T22:52:18.305103+0000","last_undegraded":"2024-09-20T22:52:18.305103+0000","last_fullsized":"2024-09-20T22:52:18.305103+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-09-20T22:51:03.380934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:51:03.380934+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:30.819992+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":534,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-09-20T22:52:18.271078+0000","last_change":"2024-09-20T22:50:42.864153+0000","last_active":"2024-09-20T22:52:18.271078+0000","last_peered":"2024-09-20T22:52:18.271078+0000","last_clean":"2024-09-20T22:52:18.271078+0000","last_became_active":"2024-09-20T22:48:05.250319+0000","last_became_peered":"2024-09-20T22:48:05.250319+0000","last_unstale":"2024-09-20T22:52:18.271078+0000","last_undegraded":"2024-09-20T22:52:18.271078+0000","last_fullsized":"2024-09-20T22:52:18.271078+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:50:42.864078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:50:42.864078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"154'2","reported_seq":1214,"reported_epoch":657,"state":"active+clean","last_fresh":"2024-09-20T22:52:17.497268+0000","last_change":"2024-09-20T22:50:34.375264+0000","last_active":"2024-09-20T22:52:17.497268+0000","last_peered":"2024-09-20T22:52:17.497268+0000","last_clean":"2024-09-20T22:52:17.497268+0000","last_became_active":"2024-09-20T22:41:44.199822+0000","last_became_peered":"2024-09-20T22:41:44.199822+0000","last_unstale":"2024-09-20T22:52:17.497268+0000","last_undegraded":"2024-09-20T22:52:17.497268+0000","last_fullsized":"2024-09-20T22:52:17.497268+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"154'2","last_scrub_stamp":"2024-09-20T22:50:34.375161+0000","last_deep_scrub":"154'2","last_deep_scrub_stamp":"2024-09-20T22:43:54.134066+0000","last_clean_scrub_stamp":"2024-09-20T22:50:34.375161+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"67'1","reported_seq":139,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-09-20T22:52:18.304293+0000","last_change":"2024-09-20T22:51:20.303293+0000","last_active":"2024-09-20T22:52:18.304293+0000","last_peered":"2024-09-20T22:52:18.304293+0000","last_clean":"2024-09-20T22:52:18.304293+0000","last_became_active":"2024-09-20T22:51:20.279978+0000","last_became_peered":"2024-09-20T22:51:20.279978+0000","last_unstale":"2024-09-20T22:52:18.304293+0000","last_undegraded":"2024-09-20T22:52:18.304293+0000","last_fullsized":"2024-09-20T22:52:18.304293+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-09-20T22:50:42.717953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:50:42.717953+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"625'25","reported_seq":594,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-09-20T22:52:18.273812+0000","last_change":"2024-09-20T22:51:02.707652+0000","last_active":"2024-09-20T22:52:18.273812+0000","last_peered":"2024-09-20T22:52:18.273812+0000","last_clean":"2024-09-20T22:52:18.273812+0000","last_became_active":"2024-09-20T22:47:49.232181+0000","last_became_peered":"2024-09-20T22:47:49.232181+0000","last_unstale":"2024-09-20T22:52:18.273812+0000","last_undegraded":"2024-09-20T22:52:18.273812+0000","last_fullsized":"2024-09-20T22:52:18.273812+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"151'15","last_scrub_stamp":"2024-09-20T22:51:02.707574+0000","last_deep_scrub":"151'15","last_deep_scrub_stamp":"2024-09-20T22:43:50.715761+0000","last_clean_scrub_stamp":"2024-09-20T22:51:02.707574+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"408'10","reported_seq":1388,"reported_epoch":657,"state":"active+clean","last_fresh":"2024-09-20T22:52:17.494456+0000","last_change":"2024-09-20T22:51:03.967383+0000","last_active":"2024-09-20T22:52:17.494456+0000","last_peered":"2024-09-20T22:52:17.494456+0000","last_clean":"2024-09-20T22:52:17.494456+0000","last_became_active":"2024-09-20T22:48:05.244270+0000","last_became_peered":"2024-09-20T22:48:05.244270+0000","last_unstale":"2024-09-20T22:52:17.494456+0000","last_undegraded":"2024-09-20T22:52:17.494456+0000","last_fullsized":"2024-09-20T22:52:17.494456+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"408'10","last_scrub_stamp":"2024-09-20T22:51:03.967298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:51:03.967298+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.c","version":"657'1974","reported_seq":4728,"reported_epoch":657,"state":"active+clean","last_fresh":"2024-09-20T22:52:17.519438+0000","last_change":"2024-09-20T22:52:17.519662+0000","last_active":"2024-09-20T22:52:17.519438+0000","last_peered":"2024-09-20T22:52:17.519438+0000","last_clean":"2024-09-20T22:52:17.519438+0000","last_became_active":"2024-09-20T22:42:52.404027+0000","last_became_peered":"2024-09-20T22:42:52.404027+0000","last_unstale":"2024-09-20T22:52:17.519438+0000","last_undegraded":"2024-09-20T22:52:17.519438+0000","last_fullsized":"2024-09-20T22:52:17.519438+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"584'1653","last_scrub_stamp":"2024-09-20T22:51:04.993550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:51:04.993550+0000","objects_scrubbed":39,"log_size":1974,"log_dups_size":0,"ondisk_log_size":1974,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0.025310839000000002,"stat_sum":{"num_bytes":90680996,"num_objects":41,"num_object_clones":34,"num_object_copies":123,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":1,"num_read":851,"num_read_kb":923855,"num_write":545,"num_write_kb":211218,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"a7","length":"1"},{"start":"c5","length":"1"},{"start":"ce","length":"1"},{"start":"d4","length":"1"},{"start":"da","length":"1"},{"start":"df","length":"1"},{"start":"e3","length":"a"},{"start":"ee","length":"7"},{"start":"f6","length":"2"},{"start":"f9","length":"1"},{"start":"fc","length":"1"},{"start":"fe","length":"5"},{"start":"104","length":"3"},{"start":"108","length":"4"},{"start":"10d","length":"1"},{"start":"10f","length":"1"},{"start":"111","length":"1"},{"start":"113","length":"1"},{"start":"116","length":"2"},{"start":"11a","length":"1"},{"start":"11c","length":"1"}]},{"pgid":"1.a","version":"77'128","reported_seq":1487,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-09-20T22:52:18.439834+0000","last_change":"2024-09-20T22:50:19.676094+0000","last_active":"2024-09-20T22:52:18.439834+0000","last_peered":"2024-09-20T22:52:18.439834+0000","last_clean":"2024-09-20T22:52:18.439834+0000","last_became_active":"2024-09-20T22:42:52.407791+0000","last_became_peered":"2024-09-20T22:42:52.407791+0000","last_unstale":"2024-09-20T22:52:18.439834+0000","last_undegraded":"2024-09-20T22:52:18.439834+0000","last_fullsized":"2024-09-20T22:52:18.439834+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:50:19.675992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:50:19.675992+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"625'27","reported_seq":213,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-09-20T22:52:18.305997+0000","last_change":"2024-09-20T22:52:15.358218+0000","last_active":"2024-09-20T22:52:18.305997+0000","last_peered":"2024-09-20T22:52:18.305997+0000","last_clean":"2024-09-20T22:52:18.305997+0000","last_became_active":"2024-09-20T22:51:20.287083+0000","last_became_peered":"2024-09-20T22:51:20.287083+0000","last_unstale":"2024-09-20T22:52:18.305997+0000","last_undegraded":"2024-09-20T22:52:18.305997+0000","last_fullsized":"2024-09-20T22:52:18.305997+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'27","last_scrub_stamp":"2024-09-20T22:52:15.358100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:52:15.358100+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:27.239349+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":1224,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-09-20T22:52:18.303870+0000","last_change":"2024-09-20T22:51:29.394206+0000","last_active":"2024-09-20T22:52:18.303870+0000","last_peered":"2024-09-20T22:52:18.303870+0000","last_clean":"2024-09-20T22:52:18.303870+0000","last_became_active":"2024-09-20T22:51:20.279439+0000","last_became_peered":"2024-09-20T22:51:20.279439+0000","last_unstale":"2024-09-20T22:52:18.303870+0000","last_undegraded":"2024-09-20T22:52:18.303870+0000","last_fullsized":"2024-09-20T22:52:18.303870+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:29.394097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:49:12.070675+0000","last_clean_scrub_stamp":"2024-09-20T22:51:29.394097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:57.584257+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"408'12","reported_seq":589,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-09-20T22:52:18.271258+0000","last_change":"2024-09-20T22:51:50.785575+0000","last_active":"2024-09-20T22:52:18.271258+0000","last_peered":"2024-09-20T22:52:18.271258+0000","last_clean":"2024-09-20T22:52:18.271258+0000","last_became_active":"2024-09-20T22:48:05.242914+0000","last_became_peered":"2024-09-20T22:48:05.242914+0000","last_unstale":"2024-09-20T22:52:18.271258+0000","last_undegraded":"2024-09-20T22:52:18.271258+0000","last_fullsized":"2024-09-20T22:52:18.271258+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"408'12","last_scrub_stamp":"2024-09-20T22:51:50.785478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:51:50.785478+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:06.586894+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":1233,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-09-20T22:52:18.440625+0000","last_change":"2024-09-20T22:51:31.960415+0000","last_active":"2024-09-20T22:52:18.440625+0000","last_peered":"2024-09-20T22:52:18.440625+0000","last_clean":"2024-09-20T22:52:18.440625+0000","last_became_active":"2024-09-20T22:51:20.280145+0000","last_became_peered":"2024-09-20T22:51:20.280145+0000","last_unstale":"2024-09-20T22:52:18.440625+0000","last_undegraded":"2024-09-20T22:52:18.440625+0000","last_fullsized":"2024-09-20T22:52:18.440625+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:31.960331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:51:31.960331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:46.560339+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3","version":"657'268","reported_seq":2943,"reported_epoch":658,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T22:52:18.304224+0000","last_change":"2024-09-20T22:52:18.302315+0000","last_active":"2024-09-20T22:52:18.304224+0000","last_peered":"2024-09-20T22:52:18.304224+0000","last_clean":"2024-09-20T22:52:18.304224+0000","last_became_active":"2024-09-20T22:49:05.221402+0000","last_became_peered":"2024-09-20T22:49:05.221402+0000","last_unstale":"2024-09-20T22:52:18.304224+0000","last_undegraded":"2024-09-20T22:52:18.304224+0000","last_fullsized":"2024-09-20T22:52:18.304224+0000","mapping_epoch":404,"log_start":"456'129","ondisk_log_start":"456'129","created":94,"last_epoch_clean":465,"parent":"0.0","parent_split_bits":0,"last_scrub":"609'243","last_scrub_stamp":"2024-09-20T22:51:31.402013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:51:31.402013+0000","objects_scrubbed":6,"log_size":139,"log_dups_size":149,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:57.361067+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.80582715100000002,"stat_sum":{"num_bytes":16392605,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":97,"num_read_kb":128299,"num_write":60,"num_write_kb":21007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":20196357,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"a7","length":"1"},{"start":"c5","length":"1"},{"start":"ce","length":"1"},{"start":"d4","length":"1"},{"start":"da","length":"1"},{"start":"df","length":"1"},{"start":"e3","length":"a"},{"start":"ee","length":"7"},{"start":"f6","length":"2"},{"start":"f9","length":"1"},{"start":"fc","length":"1"},{"start":"fe","length":"5"},{"start":"104","length":"3"},{"start":"108","length":"4"},{"start":"10d","length":"1"},{"start":"10f","length":"1"},{"start":"111","length":"1"},{"start":"113","length":"1"},{"start":"117","length":"1"},{"start":"11c","length":"1"}]},{"pgid":"1.5","version":"77'128","reported_seq":1394,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-09-20T22:52:18.271349+0000","last_change":"2024-09-20T22:51:12.927535+0000","last_active":"2024-09-20T22:52:18.271349+0000","last_peered":"2024-09-20T22:52:18.271349+0000","last_clean":"2024-09-20T22:52:18.271349+0000","last_became_active":"2024-09-20T22:48:08.207018+0000","last_became_peered":"2024-09-20T22:48:08.207018+0000","last_unstale":"2024-09-20T22:52:18.271349+0000","last_undegraded":"2024-09-20T22:52:18.271349+0000","last_fullsized":"2024-09-20T22:52:18.271349+0000","mapping_epoch":407,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":408,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:51:12.927463+0000","last_deep_scrub":"77'128","last_deep_scrub_stamp":"2024-09-20T22:45:41.608075+0000","last_clean_scrub_stamp":"2024-09-20T22:51:12.927463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:28.499830+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"625'30","reported_seq":1313,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-09-20T22:52:18.303974+0000","last_change":"2024-09-20T22:50:45.270518+0000","last_active":"2024-09-20T22:52:18.303974+0000","last_peered":"2024-09-20T22:52:18.303974+0000","last_clean":"2024-09-20T22:52:18.303974+0000","last_became_active":"2024-09-20T22:47:49.217605+0000","last_became_peered":"2024-09-20T22:47:49.217605+0000","last_unstale":"2024-09-20T22:52:18.303974+0000","last_undegraded":"2024-09-20T22:52:18.303974+0000","last_fullsized":"2024-09-20T22:52:18.303974+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'18","last_scrub_stamp":"2024-09-20T22:50:45.270399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:50:45.270399+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":1230,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-09-20T22:52:18.306305+0000","last_change":"2024-09-20T22:50:22.220104+0000","last_active":"2024-09-20T22:52:18.306305+0000","last_peered":"2024-09-20T22:52:18.306305+0000","last_clean":"2024-09-20T22:52:18.306305+0000","last_became_active":"2024-09-20T22:47:49.227069+0000","last_became_peered":"2024-09-20T22:47:49.227069+0000","last_unstale":"2024-09-20T22:52:18.306305+0000","last_undegraded":"2024-09-20T22:52:18.306305+0000","last_fullsized":"2024-09-20T22:52:18.306305+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:50:22.219983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:50:22.219983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":535,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-09-20T22:52:18.271245+0000","last_change":"2024-09-20T22:51:09.901939+0000","last_active":"2024-09-20T22:52:18.271245+0000","last_peered":"2024-09-20T22:52:18.271245+0000","last_clean":"2024-09-20T22:52:18.271245+0000","last_became_active":"2024-09-20T22:48:05.230003+0000","last_became_peered":"2024-09-20T22:48:05.230003+0000","last_unstale":"2024-09-20T22:52:18.271245+0000","last_undegraded":"2024-09-20T22:52:18.271245+0000","last_fullsized":"2024-09-20T22:52:18.271245+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:09.901838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:07.947945+0000","last_clean_scrub_stamp":"2024-09-20T22:51:09.901838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:27.938229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":535,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-09-20T22:52:18.270719+0000","last_change":"2024-09-20T22:51:55.748527+0000","last_active":"2024-09-20T22:52:18.270719+0000","last_peered":"2024-09-20T22:52:18.270719+0000","last_clean":"2024-09-20T22:52:18.270719+0000","last_became_active":"2024-09-20T22:48:05.239917+0000","last_became_peered":"2024-09-20T22:48:05.239917+0000","last_unstale":"2024-09-20T22:52:18.270719+0000","last_undegraded":"2024-09-20T22:52:18.270719+0000","last_fullsized":"2024-09-20T22:52:18.270719+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:55.748425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:53.036601+0000","last_clean_scrub_stamp":"2024-09-20T22:51:55.748425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:19.609464+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2","version":"652'669","reported_seq":2583,"reported_epoch":658,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-20T22:52:18.304531+0000","last_change":"2024-09-20T22:51:51.513081+0000","last_active":"2024-09-20T22:52:18.304531+0000","last_peered":"2024-09-20T22:52:18.304531+0000","last_clean":"2024-09-20T22:51:18.665603+0000","last_became_active":"2024-09-20T22:51:21.509843+0000","last_became_peered":"2024-09-20T22:51:21.509843+0000","last_unstale":"2024-09-20T22:52:18.304531+0000","last_undegraded":"2024-09-20T22:52:18.304531+0000","last_fullsized":"2024-09-20T22:52:18.304531+0000","mapping_epoch":600,"log_start":"471'464","ondisk_log_start":"471'464","created":94,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":0,"last_scrub":"282'120","last_scrub_stamp":"2024-09-20T22:46:22.099642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:46:22.099642+0000","objects_scrubbed":1,"log_size":205,"log_dups_size":687,"ondisk_log_size":205,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":47,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.96265287200000005,"stat_sum":{"num_bytes":53298460,"num_objects":24,"num_object_clones":20,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":24,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":1,"num_read":376,"num_read_kb":396277,"num_write":228,"num_write_kb":92279,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":120,"num_bytes_recovered":49828787,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,1],"avail_no_missing":["3","1","2"],"object_location_counts":[{"shards":"1,2,3","objects":24}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"77'128","reported_seq":1519,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-09-20T22:52:18.306531+0000","last_change":"2024-09-20T22:52:02.474080+0000","last_active":"2024-09-20T22:52:18.306531+0000","last_peered":"2024-09-20T22:52:18.306531+0000","last_clean":"2024-09-20T22:52:18.306531+0000","last_became_active":"2024-09-20T22:46:15.181305+0000","last_became_peered":"2024-09-20T22:46:15.181305+0000","last_unstale":"2024-09-20T22:52:18.306531+0000","last_undegraded":"2024-09-20T22:52:18.306531+0000","last_fullsized":"2024-09-20T22:52:18.306531+0000","mapping_epoch":94,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:52:02.473979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:52:02.473979+0000","objects_scrubbed":1,"log_size":0,"log_dups_size":59,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:05.099562+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"625'28","reported_seq":572,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-09-20T22:52:18.271181+0000","last_change":"2024-09-20T22:51:58.712602+0000","last_active":"2024-09-20T22:52:18.271181+0000","last_peered":"2024-09-20T22:52:18.271181+0000","last_clean":"2024-09-20T22:52:18.271181+0000","last_became_active":"2024-09-20T22:48:05.235812+0000","last_became_peered":"2024-09-20T22:48:05.235812+0000","last_unstale":"2024-09-20T22:52:18.271181+0000","last_undegraded":"2024-09-20T22:52:18.271181+0000","last_fullsized":"2024-09-20T22:52:18.271181+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'28","last_scrub_stamp":"2024-09-20T22:51:58.712488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:51:58.712488+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:18.921188+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":41,"num_read_kb":34,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"67'1","reported_seq":591,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-09-20T22:52:18.273585+0000","last_change":"2024-09-20T22:51:20.292468+0000","last_active":"2024-09-20T22:52:18.273585+0000","last_peered":"2024-09-20T22:52:18.273585+0000","last_clean":"2024-09-20T22:52:18.273585+0000","last_became_active":"2024-09-20T22:51:20.283011+0000","last_became_peered":"2024-09-20T22:51:20.283011+0000","last_unstale":"2024-09-20T22:52:18.273585+0000","last_undegraded":"2024-09-20T22:52:18.273585+0000","last_fullsized":"2024-09-20T22:52:18.273585+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-09-20T22:51:11.806011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:51:11.806011+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:24.512033+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"600'13","reported_seq":1399,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-09-20T22:52:18.304188+0000","last_change":"2024-09-20T22:51:32.448600+0000","last_active":"2024-09-20T22:52:18.304188+0000","last_peered":"2024-09-20T22:52:18.304188+0000","last_clean":"2024-09-20T22:52:18.304188+0000","last_became_active":"2024-09-20T22:51:20.289873+0000","last_became_peered":"2024-09-20T22:51:20.289873+0000","last_unstale":"2024-09-20T22:52:18.304188+0000","last_undegraded":"2024-09-20T22:52:18.304188+0000","last_fullsized":"2024-09-20T22:52:18.304188+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"600'13","last_scrub_stamp":"2024-09-20T22:51:32.448512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:51:32.448512+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:52.962692+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":1213,"reported_epoch":657,"state":"active+clean","last_fresh":"2024-09-20T22:52:17.496652+0000","last_change":"2024-09-20T22:51:31.487593+0000","last_active":"2024-09-20T22:52:17.496652+0000","last_peered":"2024-09-20T22:52:17.496652+0000","last_clean":"2024-09-20T22:52:17.496652+0000","last_became_active":"2024-09-20T22:41:44.831901+0000","last_became_peered":"2024-09-20T22:41:44.831901+0000","last_unstale":"2024-09-20T22:52:17.496652+0000","last_undegraded":"2024-09-20T22:52:17.496652+0000","last_fullsized":"2024-09-20T22:52:17.496652+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:31.487469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:51:31.487469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:01.276783+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1","version":"655'361","reported_seq":3119,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-09-20T22:52:18.270846+0000","last_change":"2024-09-20T22:52:17.512713+0000","last_active":"2024-09-20T22:52:18.270846+0000","last_peered":"2024-09-20T22:52:18.270846+0000","last_clean":"2024-09-20T22:52:18.270846+0000","last_became_active":"2024-09-20T22:49:21.232750+0000","last_became_peered":"2024-09-20T22:49:21.232750+0000","last_unstale":"2024-09-20T22:52:18.270846+0000","last_undegraded":"2024-09-20T22:52:18.270846+0000","last_fullsized":"2024-09-20T22:52:18.270846+0000","mapping_epoch":407,"log_start":"474'259","ondisk_log_start":"474'259","created":94,"last_epoch_clean":481,"parent":"0.0","parent_split_bits":0,"last_scrub":"562'298","last_scrub_stamp":"2024-09-20T22:50:45.816540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:50:45.816540+0000","objects_scrubbed":4,"log_size":102,"log_dups_size":338,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0013613690000000001,"stat_sum":{"num_bytes":11627183,"num_objects":7,"num_object_clones":5,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":92,"num_read_kb":91248,"num_write":81,"num_write_kb":44465,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":8732217,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"a7","length":"1"},{"start":"c5","length":"1"},{"start":"ce","length":"1"},{"start":"d4","length":"1"},{"start":"da","length":"1"},{"start":"df","length":"1"},{"start":"e3","length":"a"},{"start":"ee","length":"7"},{"start":"f6","length":"2"},{"start":"f9","length":"1"},{"start":"fc","length":"1"},{"start":"fe","length":"5"},{"start":"104","length":"3"},{"start":"108","length":"4"},{"start":"10d","length":"1"},{"start":"10f","length":"1"},{"start":"111","length":"1"},{"start":"113","length":"1"},{"start":"116","length":"2"},{"start":"11a","length":"1"},{"start":"11c","length":"1"}]},{"pgid":"1.7","version":"77'128","reported_seq":1522,"reported_epoch":657,"state":"active+clean","last_fresh":"2024-09-20T22:52:17.494024+0000","last_change":"2024-09-20T22:51:10.000911+0000","last_active":"2024-09-20T22:52:17.494024+0000","last_peered":"2024-09-20T22:52:17.494024+0000","last_clean":"2024-09-20T22:52:17.494024+0000","last_became_active":"2024-09-20T22:46:03.169596+0000","last_became_peered":"2024-09-20T22:46:03.169596+0000","last_unstale":"2024-09-20T22:52:17.494024+0000","last_undegraded":"2024-09-20T22:52:17.494024+0000","last_fullsized":"2024-09-20T22:52:17.494024+0000","mapping_epoch":94,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:51:10.000823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:51:10.000823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"625'20","reported_seq":189,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-09-20T22:52:18.440358+0000","last_change":"2024-09-20T22:51:20.317396+0000","last_active":"2024-09-20T22:52:18.440358+0000","last_peered":"2024-09-20T22:52:18.440358+0000","last_clean":"2024-09-20T22:52:18.440358+0000","last_became_active":"2024-09-20T22:51:20.297933+0000","last_became_peered":"2024-09-20T22:51:20.297933+0000","last_unstale":"2024-09-20T22:52:18.440358+0000","last_undegraded":"2024-09-20T22:52:18.440358+0000","last_fullsized":"2024-09-20T22:52:18.440358+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-09-20T22:49:46.855673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:49:46.855673+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":1228,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-09-20T22:52:18.440907+0000","last_change":"2024-09-20T22:50:14.699223+0000","last_active":"2024-09-20T22:52:18.440907+0000","last_peered":"2024-09-20T22:52:18.440907+0000","last_clean":"2024-09-20T22:52:18.440907+0000","last_became_active":"2024-09-20T22:41:38.155629+0000","last_became_peered":"2024-09-20T22:41:38.155629+0000","last_unstale":"2024-09-20T22:52:18.440907+0000","last_undegraded":"2024-09-20T22:52:18.440907+0000","last_fullsized":"2024-09-20T22:52:18.440907+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:50:14.699152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:50:14.699152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"389'11","reported_seq":626,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-09-20T22:52:18.273043+0000","last_change":"2024-09-20T22:51:26.968244+0000","last_active":"2024-09-20T22:52:18.273043+0000","last_peered":"2024-09-20T22:52:18.273043+0000","last_clean":"2024-09-20T22:52:18.273043+0000","last_became_active":"2024-09-20T22:47:49.227264+0000","last_became_peered":"2024-09-20T22:47:49.227264+0000","last_unstale":"2024-09-20T22:52:18.273043+0000","last_undegraded":"2024-09-20T22:52:18.273043+0000","last_fullsized":"2024-09-20T22:52:18.273043+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'11","last_scrub_stamp":"2024-09-20T22:51:26.968138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:51:26.968138+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:30.761523+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":567,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-09-20T22:52:18.272230+0000","last_change":"2024-09-20T22:51:55.755550+0000","last_active":"2024-09-20T22:52:18.272230+0000","last_peered":"2024-09-20T22:52:18.272230+0000","last_clean":"2024-09-20T22:52:18.272230+0000","last_became_active":"2024-09-20T22:47:49.232380+0000","last_became_peered":"2024-09-20T22:47:49.232380+0000","last_unstale":"2024-09-20T22:52:18.272230+0000","last_undegraded":"2024-09-20T22:52:18.272230+0000","last_fullsized":"2024-09-20T22:52:18.272230+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:55.755432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:51:55.755432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:07.619648+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.0","version":"656'499","reported_seq":3325,"reported_epoch":658,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T22:52:18.439799+0000","last_change":"2024-09-20T22:52:17.396407+0000","last_active":"2024-09-20T22:52:18.439799+0000","last_peered":"2024-09-20T22:52:18.439799+0000","last_clean":"2024-09-20T22:52:18.439799+0000","last_became_active":"2024-09-20T22:51:53.366096+0000","last_became_peered":"2024-09-20T22:51:53.366096+0000","last_unstale":"2024-09-20T22:52:18.439799+0000","last_undegraded":"2024-09-20T22:52:18.439799+0000","last_fullsized":"2024-09-20T22:52:18.439799+0000","mapping_epoch":632,"log_start":"480'253","ondisk_log_start":"480'253","created":94,"last_epoch_clean":633,"parent":"0.0","parent_split_bits":0,"last_scrub":"562'368","last_scrub_stamp":"2024-09-20T22:50:44.669910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:50:44.669910+0000","objects_scrubbed":15,"log_size":246,"log_dups_size":416,"ondisk_log_size":246,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":12,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.77459749200000005,"stat_sum":{"num_bytes":35905141,"num_objects":18,"num_object_clones":15,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":190,"num_read_kb":202068,"num_write":151,"num_write_kb":50041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":42943203,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"a7","length":"1"},{"start":"c5","length":"1"},{"start":"ce","length":"1"},{"start":"d4","length":"1"},{"start":"da","length":"1"},{"start":"df","length":"1"},{"start":"e3","length":"a"},{"start":"ee","length":"7"},{"start":"f6","length":"2"},{"start":"f9","length":"1"},{"start":"fc","length":"1"},{"start":"fe","length":"5"},{"start":"104","length":"1"},{"start":"106","length":"1"},{"start":"108","length":"1"}]},{"pgid":"1.6","version":"77'128","reported_seq":1544,"reported_epoch":657,"state":"active+clean","last_fresh":"2024-09-20T22:52:17.494107+0000","last_change":"2024-09-20T22:51:06.036981+0000","last_active":"2024-09-20T22:52:17.494107+0000","last_peered":"2024-09-20T22:52:17.494107+0000","last_clean":"2024-09-20T22:52:17.494107+0000","last_became_active":"2024-09-20T22:47:52.212190+0000","last_became_peered":"2024-09-20T22:47:52.212190+0000","last_unstale":"2024-09-20T22:52:17.494107+0000","last_undegraded":"2024-09-20T22:52:17.494107+0000","last_fullsized":"2024-09-20T22:52:17.494107+0000","mapping_epoch":391,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:51:06.036900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:51:06.036900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:27.038972+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"625'37","reported_seq":600,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-09-20T22:52:18.272712+0000","last_change":"2024-09-20T22:49:55.964247+0000","last_active":"2024-09-20T22:52:18.272712+0000","last_peered":"2024-09-20T22:52:18.272712+0000","last_clean":"2024-09-20T22:52:18.272712+0000","last_became_active":"2024-09-20T22:47:49.222009+0000","last_became_peered":"2024-09-20T22:47:49.222009+0000","last_unstale":"2024-09-20T22:52:18.272712+0000","last_undegraded":"2024-09-20T22:52:18.272712+0000","last_fullsized":"2024-09-20T22:52:18.272712+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"515'27","last_scrub_stamp":"2024-09-20T22:49:55.964153+0000","last_deep_scrub":"515'27","last_deep_scrub_stamp":"2024-09-20T22:49:55.964153+0000","last_clean_scrub_stamp":"2024-09-20T22:49:55.964153+0000","objects_scrubbed":8,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":53,"num_read_kb":39,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":136,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-09-20T22:52:18.440664+0000","last_change":"2024-09-20T22:52:14.225251+0000","last_active":"2024-09-20T22:52:18.440664+0000","last_peered":"2024-09-20T22:52:18.440664+0000","last_clean":"2024-09-20T22:52:18.440664+0000","last_became_active":"2024-09-20T22:51:20.286646+0000","last_became_peered":"2024-09-20T22:51:20.286646+0000","last_unstale":"2024-09-20T22:52:18.440664+0000","last_undegraded":"2024-09-20T22:52:18.440664+0000","last_fullsized":"2024-09-20T22:52:18.440664+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:14.225111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:48:23.320031+0000","last_clean_scrub_stamp":"2024-09-20T22:52:14.225111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:38.894957+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":1237,"reported_epoch":657,"state":"active+clean","last_fresh":"2024-09-20T22:52:17.493644+0000","last_change":"2024-09-20T22:51:29.928160+0000","last_active":"2024-09-20T22:52:17.493644+0000","last_peered":"2024-09-20T22:52:17.493644+0000","last_clean":"2024-09-20T22:52:17.493644+0000","last_became_active":"2024-09-20T22:47:49.214055+0000","last_became_peered":"2024-09-20T22:47:49.214055+0000","last_unstale":"2024-09-20T22:52:17.493644+0000","last_undegraded":"2024-09-20T22:52:17.493644+0000","last_fullsized":"2024-09-20T22:52:17.493644+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:29.928092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:51:29.928092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:32.798183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":134,"reported_epoch":657,"state":"active+clean","last_fresh":"2024-09-20T22:52:17.493637+0000","last_change":"2024-09-20T22:51:51.100969+0000","last_active":"2024-09-20T22:52:17.493637+0000","last_peered":"2024-09-20T22:52:17.493637+0000","last_clean":"2024-09-20T22:52:17.493637+0000","last_became_active":"2024-09-20T22:51:20.284874+0000","last_became_peered":"2024-09-20T22:51:20.284874+0000","last_unstale":"2024-09-20T22:52:17.493637+0000","last_undegraded":"2024-09-20T22:52:17.493637+0000","last_fullsized":"2024-09-20T22:52:17.493637+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:51.100878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:48:41.208853+0000","last_clean_scrub_stamp":"2024-09-20T22:51:51.100878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:54.933631+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.7","version":"657'1058","reported_seq":3949,"reported_epoch":657,"state":"active+clean","last_fresh":"2024-09-20T22:52:17.500290+0000","last_change":"2024-09-20T22:52:17.500478+0000","last_active":"2024-09-20T22:52:17.500290+0000","last_peered":"2024-09-20T22:52:17.500290+0000","last_clean":"2024-09-20T22:52:17.500290+0000","last_became_active":"2024-09-20T22:48:43.224386+0000","last_became_peered":"2024-09-20T22:48:43.224386+0000","last_unstale":"2024-09-20T22:52:17.500290+0000","last_undegraded":"2024-09-20T22:52:17.500290+0000","last_fullsized":"2024-09-20T22:52:17.500290+0000","mapping_epoch":391,"log_start":"436'456","ondisk_log_start":"436'456","created":94,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"631'991","last_scrub_stamp":"2024-09-20T22:51:54.106963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:51:54.106963+0000","objects_scrubbed":20,"log_size":602,"log_dups_size":886,"ondisk_log_size":602,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:00.537117+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.006045412,"stat_sum":{"num_bytes":47730670,"num_objects":24,"num_object_clones":19,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":1,"num_read":422,"num_read_kb":434616,"num_write":337,"num_write_kb":110295,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":115,"num_bytes_recovered":51615167,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"a7","length":"1"},{"start":"c5","length":"1"},{"start":"ce","length":"1"},{"start":"d4","length":"1"},{"start":"da","length":"1"},{"start":"df","length":"1"},{"start":"e3","length":"a"},{"start":"ee","length":"7"},{"start":"f6","length":"2"},{"start":"f9","length":"1"},{"start":"fc","length":"1"},{"start":"fe","length":"5"},{"start":"104","length":"3"},{"start":"108","length":"4"},{"start":"10d","length":"1"},{"start":"10f","length":"1"},{"start":"111","length":"1"},{"start":"113","length":"1"},{"start":"116","length":"2"},{"start":"11a","length":"1"},{"start":"11c","length":"1"}]},{"pgid":"1.1","version":"77'128","reported_seq":1535,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-09-20T22:52:18.439834+0000","last_change":"2024-09-20T22:50:20.639645+0000","last_active":"2024-09-20T22:52:18.439834+0000","last_peered":"2024-09-20T22:52:18.439834+0000","last_clean":"2024-09-20T22:52:18.439834+0000","last_became_active":"2024-09-20T22:47:52.208102+0000","last_became_peered":"2024-09-20T22:47:52.208102+0000","last_unstale":"2024-09-20T22:52:18.439834+0000","last_undegraded":"2024-09-20T22:52:18.439834+0000","last_fullsized":"2024-09-20T22:52:18.439834+0000","mapping_epoch":391,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:50:20.639561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:50:20.639561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"625'51","reported_seq":299,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-09-20T22:52:18.440560+0000","last_change":"2024-09-20T22:52:17.199465+0000","last_active":"2024-09-20T22:52:18.440560+0000","last_peered":"2024-09-20T22:52:18.440560+0000","last_clean":"2024-09-20T22:52:18.440560+0000","last_became_active":"2024-09-20T22:51:20.289376+0000","last_became_peered":"2024-09-20T22:51:20.289376+0000","last_unstale":"2024-09-20T22:52:18.440560+0000","last_undegraded":"2024-09-20T22:52:18.440560+0000","last_fullsized":"2024-09-20T22:52:18.440560+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'51","last_scrub_stamp":"2024-09-20T22:52:17.199367+0000","last_deep_scrub":"86'38","last_deep_scrub_stamp":"2024-09-20T22:42:43.935677+0000","last_clean_scrub_stamp":"2024-09-20T22:52:17.199367+0000","objects_scrubbed":10,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:38.979686+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":527,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-09-20T22:52:18.270755+0000","last_change":"2024-09-20T22:50:35.898050+0000","last_active":"2024-09-20T22:52:18.270755+0000","last_peered":"2024-09-20T22:52:18.270755+0000","last_clean":"2024-09-20T22:52:18.270755+0000","last_became_active":"2024-09-20T22:48:05.237756+0000","last_became_peered":"2024-09-20T22:48:05.237756+0000","last_unstale":"2024-09-20T22:52:18.270755+0000","last_undegraded":"2024-09-20T22:52:18.270755+0000","last_fullsized":"2024-09-20T22:52:18.270755+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:50:35.897979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:50:35.897979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"69'2","reported_seq":1256,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-09-20T22:52:18.440471+0000","last_change":"2024-09-20T22:51:37.070187+0000","last_active":"2024-09-20T22:52:18.440471+0000","last_peered":"2024-09-20T22:52:18.440471+0000","last_clean":"2024-09-20T22:52:18.440471+0000","last_became_active":"2024-09-20T22:51:20.286952+0000","last_became_peered":"2024-09-20T22:51:20.286952+0000","last_unstale":"2024-09-20T22:52:18.440471+0000","last_undegraded":"2024-09-20T22:52:18.440471+0000","last_fullsized":"2024-09-20T22:52:18.440471+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2024-09-20T22:51:37.070111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:36.130684+0000","last_clean_scrub_stamp":"2024-09-20T22:51:37.070111+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:05.960971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087280599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":579,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-09-20T22:52:18.272995+0000","last_change":"2024-09-20T22:51:53.772969+0000","last_active":"2024-09-20T22:52:18.272995+0000","last_peered":"2024-09-20T22:52:18.272995+0000","last_clean":"2024-09-20T22:52:18.272995+0000","last_became_active":"2024-09-20T22:51:20.278427+0000","last_became_peered":"2024-09-20T22:51:20.278427+0000","last_unstale":"2024-09-20T22:52:18.272995+0000","last_undegraded":"2024-09-20T22:52:18.272995+0000","last_fullsized":"2024-09-20T22:52:18.272995+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:53.772885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:51:53.772885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:59.965395+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":1218,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-09-20T22:52:18.305330+0000","last_change":"2024-09-20T22:49:57.022315+0000","last_active":"2024-09-20T22:52:18.305330+0000","last_peered":"2024-09-20T22:52:18.305330+0000","last_clean":"2024-09-20T22:52:18.305330+0000","last_became_active":"2024-09-20T22:48:05.234193+0000","last_became_peered":"2024-09-20T22:48:05.234193+0000","last_unstale":"2024-09-20T22:52:18.305330+0000","last_undegraded":"2024-09-20T22:52:18.305330+0000","last_fullsized":"2024-09-20T22:52:18.305330+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:57.022244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:49:57.022244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"657'389","reported_seq":1693,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-09-20T22:52:18.272985+0000","last_change":"2024-09-20T22:52:17.513416+0000","last_active":"2024-09-20T22:52:18.272985+0000","last_peered":"2024-09-20T22:52:18.272985+0000","last_clean":"2024-09-20T22:52:18.272985+0000","last_became_active":"2024-09-20T22:51:23.267613+0000","last_became_peered":"2024-09-20T22:51:23.267613+0000","last_unstale":"2024-09-20T22:52:18.272985+0000","last_undegraded":"2024-09-20T22:52:18.272985+0000","last_fullsized":"2024-09-20T22:52:18.272985+0000","mapping_epoch":602,"log_start":"456'176","ondisk_log_start":"456'176","created":94,"last_epoch_clean":603,"parent":"0.0","parent_split_bits":0,"last_scrub":"616'341","last_scrub_stamp":"2024-09-20T22:51:36.893731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:51:36.893731+0000","objects_scrubbed":8,"log_size":213,"log_dups_size":318,"ondisk_log_size":213,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:05.962782+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.016479002999999999,"stat_sum":{"num_bytes":24989903,"num_objects":10,"num_object_clones":8,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":144,"num_read_kb":137177,"num_write":123,"num_write_kb":42628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":26566881,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"a7","length":"1"},{"start":"c5","length":"1"},{"start":"ce","length":"1"},{"start":"d4","length":"1"},{"start":"da","length":"1"},{"start":"df","length":"1"},{"start":"e3","length":"a"},{"start":"ee","length":"7"},{"start":"f6","length":"2"},{"start":"f9","length":"1"},{"start":"fc","length":"1"},{"start":"fe","length":"5"},{"start":"104","length":"3"},{"start":"108","length":"4"},{"start":"10d","length":"1"},{"start":"10f","length":"1"},{"start":"111","length":"1"},{"start":"113","length":"1"},{"start":"116","length":"2"},{"start":"11a","length":"1"},{"start":"11c","length":"1"}]},{"pgid":"1.2","version":"77'128","reported_seq":1602,"reported_epoch":657,"state":"active+clean","last_fresh":"2024-09-20T22:52:17.496561+0000","last_change":"2024-09-20T22:52:02.623634+0000","last_active":"2024-09-20T22:52:17.496561+0000","last_peered":"2024-09-20T22:52:17.496561+0000","last_clean":"2024-09-20T22:52:17.496561+0000","last_became_active":"2024-09-20T22:48:08.203028+0000","last_became_peered":"2024-09-20T22:48:08.203028+0000","last_unstale":"2024-09-20T22:52:17.496561+0000","last_undegraded":"2024-09-20T22:52:17.496561+0000","last_fullsized":"2024-09-20T22:52:17.496561+0000","mapping_epoch":407,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":408,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:52:02.623562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:52:02.623562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:31.354752+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"625'22","reported_seq":989,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-09-20T22:52:18.441037+0000","last_change":"2024-09-20T22:51:20.312887+0000","last_active":"2024-09-20T22:52:18.441037+0000","last_peered":"2024-09-20T22:52:18.441037+0000","last_clean":"2024-09-20T22:52:18.441037+0000","last_became_active":"2024-09-20T22:51:20.293610+0000","last_became_peered":"2024-09-20T22:51:20.293610+0000","last_unstale":"2024-09-20T22:52:18.441037+0000","last_undegraded":"2024-09-20T22:52:18.441037+0000","last_fullsized":"2024-09-20T22:52:18.441037+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"443'18","last_scrub_stamp":"2024-09-20T22:51:04.643936+0000","last_deep_scrub":"443'18","last_deep_scrub_stamp":"2024-09-20T22:48:44.182517+0000","last_clean_scrub_stamp":"2024-09-20T22:51:04.643936+0000","objects_scrubbed":5,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":990,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":136,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-09-20T22:52:18.440487+0000","last_change":"2024-09-20T22:51:32.279015+0000","last_active":"2024-09-20T22:52:18.440487+0000","last_peered":"2024-09-20T22:52:18.440487+0000","last_clean":"2024-09-20T22:52:18.440487+0000","last_became_active":"2024-09-20T22:51:20.278609+0000","last_became_peered":"2024-09-20T22:51:20.278609+0000","last_unstale":"2024-09-20T22:52:18.440487+0000","last_undegraded":"2024-09-20T22:52:18.440487+0000","last_fullsized":"2024-09-20T22:52:18.440487+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:32.278945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:51:32.278945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:54.818249+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"380'2","reported_seq":1273,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-09-20T22:52:18.440387+0000","last_change":"2024-09-20T22:50:39.347293+0000","last_active":"2024-09-20T22:52:18.440387+0000","last_peered":"2024-09-20T22:52:18.440387+0000","last_clean":"2024-09-20T22:52:18.440387+0000","last_became_active":"2024-09-20T22:47:49.222812+0000","last_became_peered":"2024-09-20T22:47:49.222812+0000","last_unstale":"2024-09-20T22:52:18.440387+0000","last_undegraded":"2024-09-20T22:52:18.440387+0000","last_fullsized":"2024-09-20T22:52:18.440387+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"380'2","last_scrub_stamp":"2024-09-20T22:50:39.347177+0000","last_deep_scrub":"380'2","last_deep_scrub_stamp":"2024-09-20T22:47:40.983039+0000","last_clean_scrub_stamp":"2024-09-20T22:50:39.347177+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001301459,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":1226,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-09-20T22:52:18.440738+0000","last_change":"2024-09-20T22:51:04.254552+0000","last_active":"2024-09-20T22:52:18.440738+0000","last_peered":"2024-09-20T22:52:18.440738+0000","last_clean":"2024-09-20T22:52:18.440738+0000","last_became_active":"2024-09-20T22:41:42.185112+0000","last_became_peered":"2024-09-20T22:41:42.185112+0000","last_unstale":"2024-09-20T22:52:18.440738+0000","last_undegraded":"2024-09-20T22:52:18.440738+0000","last_fullsized":"2024-09-20T22:52:18.440738+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:04.254441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:47:22.881109+0000","last_clean_scrub_stamp":"2024-09-20T22:51:04.254441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:31.528165+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":1213,"reported_epoch":657,"state":"active+clean","last_fresh":"2024-09-20T22:52:17.497254+0000","last_change":"2024-09-20T22:51:11.568798+0000","last_active":"2024-09-20T22:52:17.497254+0000","last_peered":"2024-09-20T22:52:17.497254+0000","last_clean":"2024-09-20T22:52:17.497254+0000","last_became_active":"2024-09-20T22:41:44.830515+0000","last_became_peered":"2024-09-20T22:41:44.830515+0000","last_unstale":"2024-09-20T22:52:17.497254+0000","last_undegraded":"2024-09-20T22:52:17.497254+0000","last_fullsized":"2024-09-20T22:52:17.497254+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:11.568712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:51:11.568712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"655'312","reported_seq":3180,"reported_epoch":657,"state":"active+clean","last_fresh":"2024-09-20T22:52:17.497527+0000","last_change":"2024-09-20T22:52:17.497862+0000","last_active":"2024-09-20T22:52:17.497527+0000","last_peered":"2024-09-20T22:52:17.497527+0000","last_clean":"2024-09-20T22:52:17.497527+0000","last_became_active":"2024-09-20T22:48:57.236417+0000","last_became_peered":"2024-09-20T22:48:57.236417+0000","last_unstale":"2024-09-20T22:52:17.497527+0000","last_undegraded":"2024-09-20T22:52:17.497527+0000","last_fullsized":"2024-09-20T22:52:17.497527+0000","mapping_epoch":391,"log_start":"450'190","ondisk_log_start":"450'190","created":94,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"605'278","last_scrub_stamp":"2024-09-20T22:51:25.467092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:51:25.467092+0000","objects_scrubbed":6,"log_size":122,"log_dups_size":240,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:55.415940+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00075512599999999995,"stat_sum":{"num_bytes":12667288,"num_objects":7,"num_object_clones":6,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":76,"num_read_kb":60075,"num_write":88,"num_write_kb":36243,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":19673726,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"a7","length":"1"},{"start":"c5","length":"1"},{"start":"ce","length":"1"},{"start":"d4","length":"1"},{"start":"da","length":"1"},{"start":"df","length":"1"},{"start":"e3","length":"a"},{"start":"ee","length":"7"},{"start":"f6","length":"2"},{"start":"f9","length":"1"},{"start":"fc","length":"1"},{"start":"fe","length":"5"},{"start":"104","length":"3"},{"start":"108","length":"4"},{"start":"10d","length":"1"},{"start":"10f","length":"1"},{"start":"111","length":"1"},{"start":"113","length":"1"},{"start":"116","length":"2"},{"start":"11a","length":"1"},{"start":"11c","length":"1"}]},{"pgid":"1.3","version":"77'128","reported_seq":429,"reported_epoch":657,"state":"active+clean","last_fresh":"2024-09-20T22:52:17.493742+0000","last_change":"2024-09-20T22:51:28.897355+0000","last_active":"2024-09-20T22:52:17.493742+0000","last_peered":"2024-09-20T22:52:17.493742+0000","last_clean":"2024-09-20T22:52:17.493742+0000","last_became_active":"2024-09-20T22:51:23.266810+0000","last_became_peered":"2024-09-20T22:51:23.266810+0000","last_unstale":"2024-09-20T22:52:17.493742+0000","last_undegraded":"2024-09-20T22:52:17.493742+0000","last_fullsized":"2024-09-20T22:52:17.493742+0000","mapping_epoch":602,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":603,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:51:28.897273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:51:28.897273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:58.849420+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.6","version":"625'24","reported_seq":611,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-09-20T22:52:18.273857+0000","last_change":"2024-09-20T22:51:20.300892+0000","last_active":"2024-09-20T22:52:18.273857+0000","last_peered":"2024-09-20T22:52:18.273857+0000","last_clean":"2024-09-20T22:52:18.273857+0000","last_became_active":"2024-09-20T22:51:20.283112+0000","last_became_peered":"2024-09-20T22:51:20.283112+0000","last_unstale":"2024-09-20T22:52:18.273857+0000","last_undegraded":"2024-09-20T22:52:18.273857+0000","last_fullsized":"2024-09-20T22:52:18.273857+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-09-20T22:50:14.745918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:50:14.745918+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":573,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-09-20T22:52:18.272573+0000","last_change":"2024-09-20T22:50:39.786895+0000","last_active":"2024-09-20T22:52:18.272573+0000","last_peered":"2024-09-20T22:52:18.272573+0000","last_clean":"2024-09-20T22:52:18.272573+0000","last_became_active":"2024-09-20T22:48:05.233093+0000","last_became_peered":"2024-09-20T22:48:05.233093+0000","last_unstale":"2024-09-20T22:52:18.272573+0000","last_undegraded":"2024-09-20T22:52:18.272573+0000","last_fullsized":"2024-09-20T22:52:18.272573+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:50:39.786825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:44:19.834641+0000","last_clean_scrub_stamp":"2024-09-20T22:50:39.786825+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"75'5","reported_seq":1898,"reported_epoch":657,"state":"active+clean","last_fresh":"2024-09-20T22:52:17.497895+0000","last_change":"2024-09-20T22:51:20.290148+0000","last_active":"2024-09-20T22:52:17.497895+0000","last_peered":"2024-09-20T22:52:17.497895+0000","last_clean":"2024-09-20T22:52:17.497895+0000","last_became_active":"2024-09-20T22:51:20.281329+0000","last_became_peered":"2024-09-20T22:51:20.281329+0000","last_unstale":"2024-09-20T22:52:17.497895+0000","last_undegraded":"2024-09-20T22:52:17.497895+0000","last_fullsized":"2024-09-20T22:52:17.497895+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-09-20T22:50:21.363477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:36.130684+0000","last_clean_scrub_stamp":"2024-09-20T22:50:21.363477+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00078631700000000003,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":630,"num_read_kb":625,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1167,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":1237,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-09-20T22:52:18.441084+0000","last_change":"2024-09-20T22:51:20.274832+0000","last_active":"2024-09-20T22:52:18.441084+0000","last_peered":"2024-09-20T22:52:18.441084+0000","last_clean":"2024-09-20T22:52:18.441084+0000","last_became_active":"2024-09-20T22:51:20.274522+0000","last_became_peered":"2024-09-20T22:51:20.274522+0000","last_unstale":"2024-09-20T22:52:18.441084+0000","last_undegraded":"2024-09-20T22:52:18.441084+0000","last_fullsized":"2024-09-20T22:52:18.441084+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:50:18.695027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:50:18.695027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":128,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-09-20T22:52:18.440537+0000","last_change":"2024-09-20T22:51:20.288058+0000","last_active":"2024-09-20T22:52:18.440537+0000","last_peered":"2024-09-20T22:52:18.440537+0000","last_clean":"2024-09-20T22:52:18.440537+0000","last_became_active":"2024-09-20T22:51:20.287750+0000","last_became_peered":"2024-09-20T22:51:20.287750+0000","last_unstale":"2024-09-20T22:52:18.440537+0000","last_undegraded":"2024-09-20T22:52:18.440537+0000","last_fullsized":"2024-09-20T22:52:18.440537+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:05.621880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:51:05.621880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.6","version":"657'396","reported_seq":3198,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-09-20T22:52:18.273533+0000","last_change":"2024-09-20T22:52:17.522689+0000","last_active":"2024-09-20T22:52:18.273533+0000","last_peered":"2024-09-20T22:52:18.273533+0000","last_clean":"2024-09-20T22:52:18.273533+0000","last_became_active":"2024-09-20T22:48:47.234531+0000","last_became_peered":"2024-09-20T22:48:47.234531+0000","last_unstale":"2024-09-20T22:52:18.273533+0000","last_undegraded":"2024-09-20T22:52:18.273533+0000","last_fullsized":"2024-09-20T22:52:18.273533+0000","mapping_epoch":391,"log_start":"443'181","ondisk_log_start":"443'181","created":94,"last_epoch_clean":447,"parent":"0.0","parent_split_bits":0,"last_scrub":"654'392","last_scrub_stamp":"2024-09-20T22:52:14.706591+0000","last_deep_scrub":"440'177","last_deep_scrub_stamp":"2024-09-20T22:48:40.602342+0000","last_clean_scrub_stamp":"2024-09-20T22:52:14.706591+0000","objects_scrubbed":10,"log_size":215,"log_dups_size":314,"ondisk_log_size":215,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:18.195453+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.025228581999999999,"stat_sum":{"num_bytes":22786045,"num_objects":10,"num_object_clones":8,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":1,"num_read":145,"num_read_kb":142854,"num_write":123,"num_write_kb":45901,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":18662529,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"a7","length":"1"},{"start":"c5","length":"1"},{"start":"ce","length":"1"},{"start":"d4","length":"1"},{"start":"da","length":"1"},{"start":"df","length":"1"},{"start":"e3","length":"a"},{"start":"ee","length":"7"},{"start":"f6","length":"2"},{"start":"f9","length":"1"},{"start":"fc","length":"1"},{"start":"fe","length":"5"},{"start":"104","length":"3"},{"start":"108","length":"4"},{"start":"10d","length":"1"},{"start":"10f","length":"1"},{"start":"111","length":"1"},{"start":"113","length":"1"},{"start":"116","length":"2"},{"start":"11a","length":"1"},{"start":"11c","length":"1"}]},{"pgid":"4.5","version":"625'32","reported_seq":1494,"reported_epoch":657,"state":"active+clean","last_fresh":"2024-09-20T22:52:17.494890+0000","last_change":"2024-09-20T22:51:33.961483+0000","last_active":"2024-09-20T22:52:17.494890+0000","last_peered":"2024-09-20T22:52:17.494890+0000","last_clean":"2024-09-20T22:52:17.494890+0000","last_became_active":"2024-09-20T22:48:05.243523+0000","last_became_peered":"2024-09-20T22:48:05.243523+0000","last_unstale":"2024-09-20T22:52:17.494890+0000","last_undegraded":"2024-09-20T22:52:17.494890+0000","last_fullsized":"2024-09-20T22:52:17.494890+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"408'20","last_scrub_stamp":"2024-09-20T22:51:33.961401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:51:33.961401+0000","objects_scrubbed":8,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:00.387882+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"67'1","reported_seq":1225,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-09-20T22:52:18.304989+0000","last_change":"2024-09-20T22:51:35.440516+0000","last_active":"2024-09-20T22:52:18.304989+0000","last_peered":"2024-09-20T22:52:18.304989+0000","last_clean":"2024-09-20T22:52:18.304989+0000","last_became_active":"2024-09-20T22:41:38.158917+0000","last_became_peered":"2024-09-20T22:41:38.158917+0000","last_unstale":"2024-09-20T22:52:18.304989+0000","last_undegraded":"2024-09-20T22:52:18.304989+0000","last_fullsized":"2024-09-20T22:52:18.304989+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-09-20T22:51:35.440416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:51:35.440416+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:53.239571+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"77'128","reported_seq":1566,"reported_epoch":657,"state":"active+clean","last_fresh":"2024-09-20T22:52:17.497946+0000","last_change":"2024-09-20T22:49:46.391695+0000","last_active":"2024-09-20T22:52:17.497946+0000","last_peered":"2024-09-20T22:52:17.497946+0000","last_clean":"2024-09-20T22:52:17.497946+0000","last_became_active":"2024-09-20T22:47:52.206169+0000","last_became_peered":"2024-09-20T22:47:52.206169+0000","last_unstale":"2024-09-20T22:52:17.497946+0000","last_undegraded":"2024-09-20T22:52:17.497946+0000","last_fullsized":"2024-09-20T22:52:17.497946+0000","mapping_epoch":391,"log_start":"77'128","ondisk_log_start":"77'128","created":19,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:49:46.391614+0000","last_deep_scrub":"77'128","last_deep_scrub_stamp":"2024-09-20T22:47:22.417099+0000","last_clean_scrub_stamp":"2024-09-20T22:49:46.391614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":1225,"reported_epoch":657,"state":"active+clean","last_fresh":"2024-09-20T22:52:17.497316+0000","last_change":"2024-09-20T22:51:50.585717+0000","last_active":"2024-09-20T22:52:17.497316+0000","last_peered":"2024-09-20T22:52:17.497316+0000","last_clean":"2024-09-20T22:52:17.497316+0000","last_became_active":"2024-09-20T22:41:42.184387+0000","last_became_peered":"2024-09-20T22:41:42.184387+0000","last_unstale":"2024-09-20T22:52:17.497316+0000","last_undegraded":"2024-09-20T22:52:17.497316+0000","last_fullsized":"2024-09-20T22:52:17.497316+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:50.585627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:51:50.585627+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:03.411628+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":1227,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-09-20T22:52:18.440965+0000","last_change":"2024-09-20T22:49:57.715494+0000","last_active":"2024-09-20T22:52:18.440965+0000","last_peered":"2024-09-20T22:52:18.440965+0000","last_clean":"2024-09-20T22:52:18.440965+0000","last_became_active":"2024-09-20T22:48:05.241602+0000","last_became_peered":"2024-09-20T22:48:05.241602+0000","last_unstale":"2024-09-20T22:52:18.440965+0000","last_undegraded":"2024-09-20T22:52:18.440965+0000","last_fullsized":"2024-09-20T22:52:18.440965+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:57.715416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:49:57.715416+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"657'474","reported_seq":3397,"reported_epoch":657,"state":"active+clean","last_fresh":"2024-09-20T22:52:17.501936+0000","last_change":"2024-09-20T22:52:17.502112+0000","last_active":"2024-09-20T22:52:17.501936+0000","last_peered":"2024-09-20T22:52:17.501936+0000","last_clean":"2024-09-20T22:52:17.501936+0000","last_became_active":"2024-09-20T22:51:23.269216+0000","last_became_peered":"2024-09-20T22:51:23.269216+0000","last_unstale":"2024-09-20T22:52:17.501936+0000","last_undegraded":"2024-09-20T22:52:17.501936+0000","last_fullsized":"2024-09-20T22:52:17.501936+0000","mapping_epoch":602,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":603,"parent":"0.0","parent_split_bits":0,"last_scrub":"631'450","last_scrub_stamp":"2024-09-20T22:51:52.091217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:51:52.091217+0000","objects_scrubbed":8,"log_size":474,"log_dups_size":0,"ondisk_log_size":474,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:15.578170+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.007513133,"stat_sum":{"num_bytes":16398843,"num_objects":9,"num_object_clones":7,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":1,"num_read":161,"num_read_kb":161235,"num_write":85,"num_write_kb":35211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":52,"num_bytes_recovered":30669453,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"a7","length":"1"},{"start":"c5","length":"1"},{"start":"ce","length":"1"},{"start":"d4","length":"1"},{"start":"da","length":"1"},{"start":"df","length":"1"},{"start":"e3","length":"a"},{"start":"ee","length":"7"},{"start":"f6","length":"2"},{"start":"f9","length":"1"},{"start":"fc","length":"1"},{"start":"fe","length":"5"},{"start":"104","length":"3"},{"start":"108","length":"4"},{"start":"10d","length":"1"},{"start":"10f","length":"1"},{"start":"111","length":"1"},{"start":"113","length":"1"},{"start":"116","length":"2"},{"start":"11a","length":"1"},{"start":"11c","length":"1"}]},{"pgid":"4.e","version":"625'19","reported_seq":568,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-09-20T22:52:18.271318+0000","last_change":"2024-09-20T22:51:20.310962+0000","last_active":"2024-09-20T22:52:18.271318+0000","last_peered":"2024-09-20T22:52:18.271318+0000","last_clean":"2024-09-20T22:52:18.271318+0000","last_became_active":"2024-09-20T22:51:20.288958+0000","last_became_peered":"2024-09-20T22:51:20.288958+0000","last_unstale":"2024-09-20T22:52:18.271318+0000","last_undegraded":"2024-09-20T22:52:18.271318+0000","last_fullsized":"2024-09-20T22:52:18.271318+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-09-20T22:50:38.879205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:50:38.879205+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":534,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-09-20T22:52:18.270447+0000","last_change":"2024-09-20T22:51:29.757136+0000","last_active":"2024-09-20T22:52:18.270447+0000","last_peered":"2024-09-20T22:52:18.270447+0000","last_clean":"2024-09-20T22:52:18.270447+0000","last_became_active":"2024-09-20T22:48:05.254688+0000","last_became_peered":"2024-09-20T22:48:05.254688+0000","last_unstale":"2024-09-20T22:52:18.270447+0000","last_undegraded":"2024-09-20T22:52:18.270447+0000","last_fullsized":"2024-09-20T22:52:18.270447+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:29.757030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:51:29.757030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:52.073822+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":1231,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-09-20T22:52:18.440852+0000","last_change":"2024-09-20T22:51:51.002353+0000","last_active":"2024-09-20T22:52:18.440852+0000","last_peered":"2024-09-20T22:52:18.440852+0000","last_clean":"2024-09-20T22:52:18.440852+0000","last_became_active":"2024-09-20T22:48:05.233778+0000","last_became_peered":"2024-09-20T22:48:05.233778+0000","last_unstale":"2024-09-20T22:52:18.440852+0000","last_undegraded":"2024-09-20T22:52:18.440852+0000","last_fullsized":"2024-09-20T22:52:18.440852+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:51.002226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:51:51.002226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:20.564071+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":1206,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-09-20T22:52:18.306473+0000","last_change":"2024-09-20T22:50:47.230929+0000","last_active":"2024-09-20T22:52:18.306473+0000","last_peered":"2024-09-20T22:52:18.306473+0000","last_clean":"2024-09-20T22:52:18.306473+0000","last_became_active":"2024-09-20T22:41:44.832009+0000","last_became_peered":"2024-09-20T22:41:44.832009+0000","last_unstale":"2024-09-20T22:52:18.306473+0000","last_undegraded":"2024-09-20T22:52:18.306473+0000","last_fullsized":"2024-09-20T22:52:18.306473+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:50:47.230854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:48:40.127123+0000","last_clean_scrub_stamp":"2024-09-20T22:50:47.230854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.e","version":"657'602","reported_seq":3481,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-09-20T22:52:18.440875+0000","last_change":"2024-09-20T22:52:17.524777+0000","last_active":"2024-09-20T22:52:18.440875+0000","last_peered":"2024-09-20T22:52:18.440875+0000","last_clean":"2024-09-20T22:52:18.440875+0000","last_became_active":"2024-09-20T22:47:52.209452+0000","last_became_peered":"2024-09-20T22:47:52.209452+0000","last_unstale":"2024-09-20T22:52:18.440875+0000","last_undegraded":"2024-09-20T22:52:18.440875+0000","last_fullsized":"2024-09-20T22:52:18.440875+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"559'431","last_scrub_stamp":"2024-09-20T22:50:45.276861+0000","last_deep_scrub":"174'60","last_deep_scrub_stamp":"2024-09-20T22:44:18.301776+0000","last_clean_scrub_stamp":"2024-09-20T22:50:45.276861+0000","objects_scrubbed":8,"log_size":602,"log_dups_size":0,"ondisk_log_size":602,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0.0046410749999999997,"stat_sum":{"num_bytes":27857680,"num_objects":14,"num_object_clones":12,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":236,"num_read_kb":237527,"num_write":156,"num_write_kb":59501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":23110741,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"a7","length":"1"},{"start":"c5","length":"1"},{"start":"ce","length":"1"},{"start":"d4","length":"1"},{"start":"da","length":"1"},{"start":"df","length":"1"},{"start":"e3","length":"a"},{"start":"ee","length":"7"},{"start":"f6","length":"2"},{"start":"f9","length":"1"},{"start":"fc","length":"1"},{"start":"fe","length":"5"},{"start":"104","length":"3"},{"start":"108","length":"4"},{"start":"10d","length":"1"},{"start":"10f","length":"1"},{"start":"111","length":"1"},{"start":"113","length":"1"},{"start":"116","length":"2"},{"start":"11a","length":"1"},{"start":"11c","length":"1"}]},{"pgid":"1.8","version":"77'128","reported_seq":1495,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-09-20T22:52:18.439693+0000","last_change":"2024-09-20T22:51:56.108038+0000","last_active":"2024-09-20T22:52:18.439693+0000","last_peered":"2024-09-20T22:52:18.439693+0000","last_clean":"2024-09-20T22:52:18.439693+0000","last_became_active":"2024-09-20T22:42:52.405594+0000","last_became_peered":"2024-09-20T22:42:52.405594+0000","last_unstale":"2024-09-20T22:52:18.439693+0000","last_undegraded":"2024-09-20T22:52:18.439693+0000","last_fullsized":"2024-09-20T22:52:18.439693+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:51:56.107944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:51:56.107944+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:19.128079+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"625'31","reported_seq":605,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-09-20T22:52:18.271228+0000","last_change":"2024-09-20T22:52:13.620897+0000","last_active":"2024-09-20T22:52:18.271228+0000","last_peered":"2024-09-20T22:52:18.271228+0000","last_clean":"2024-09-20T22:52:18.271228+0000","last_became_active":"2024-09-20T22:51:20.288846+0000","last_became_peered":"2024-09-20T22:51:20.288846+0000","last_unstale":"2024-09-20T22:52:18.271228+0000","last_undegraded":"2024-09-20T22:52:18.271228+0000","last_fullsized":"2024-09-20T22:52:18.271228+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'31","last_scrub_stamp":"2024-09-20T22:52:13.620754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:52:13.620754+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:26.544483+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":49,"num_read_kb":39,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":781,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-09-20T22:52:18.440297+0000","last_change":"2024-09-20T22:51:20.294379+0000","last_active":"2024-09-20T22:52:18.440297+0000","last_peered":"2024-09-20T22:52:18.440297+0000","last_clean":"2024-09-20T22:52:18.440297+0000","last_became_active":"2024-09-20T22:51:20.294073+0000","last_became_peered":"2024-09-20T22:51:20.294073+0000","last_unstale":"2024-09-20T22:52:18.440297+0000","last_undegraded":"2024-09-20T22:52:18.440297+0000","last_fullsized":"2024-09-20T22:52:18.440297+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:50:47.268782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:35.492562+0000","last_clean_scrub_stamp":"2024-09-20T22:50:47.268782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":847,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-09-20T22:52:18.270669+0000","last_change":"2024-09-20T22:51:25.738132+0000","last_active":"2024-09-20T22:52:18.270669+0000","last_peered":"2024-09-20T22:52:18.270669+0000","last_clean":"2024-09-20T22:52:18.270669+0000","last_became_active":"2024-09-20T22:51:20.281724+0000","last_became_peered":"2024-09-20T22:51:20.281724+0000","last_unstale":"2024-09-20T22:52:18.270669+0000","last_undegraded":"2024-09-20T22:52:18.270669+0000","last_fullsized":"2024-09-20T22:52:18.270669+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:25.738061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:51:25.738061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:41.584933+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":1242,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-09-20T22:52:18.440699+0000","last_change":"2024-09-20T22:50:41.284220+0000","last_active":"2024-09-20T22:52:18.440699+0000","last_peered":"2024-09-20T22:52:18.440699+0000","last_clean":"2024-09-20T22:52:18.440699+0000","last_became_active":"2024-09-20T22:48:05.239006+0000","last_became_peered":"2024-09-20T22:48:05.239006+0000","last_unstale":"2024-09-20T22:52:18.440699+0000","last_undegraded":"2024-09-20T22:52:18.440699+0000","last_fullsized":"2024-09-20T22:52:18.440699+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:50:41.284158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:50:41.284158+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"657'648","reported_seq":3496,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-09-20T22:52:18.273968+0000","last_change":"2024-09-20T22:52:17.520290+0000","last_active":"2024-09-20T22:52:18.273968+0000","last_peered":"2024-09-20T22:52:18.273968+0000","last_clean":"2024-09-20T22:52:18.273968+0000","last_became_active":"2024-09-20T22:47:52.216198+0000","last_became_peered":"2024-09-20T22:47:52.216198+0000","last_unstale":"2024-09-20T22:52:18.273968+0000","last_undegraded":"2024-09-20T22:52:18.273968+0000","last_fullsized":"2024-09-20T22:52:18.273968+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"630'603","last_scrub_stamp":"2024-09-20T22:51:50.769050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:51:50.769050+0000","objects_scrubbed":8,"log_size":648,"log_dups_size":0,"ondisk_log_size":648,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:06.771716+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.021258768000000001,"stat_sum":{"num_bytes":20843956,"num_objects":11,"num_object_clones":8,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":1,"num_read":279,"num_read_kb":277181,"num_write":176,"num_write_kb":57088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":53,"num_bytes_recovered":14983556,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"a7","length":"1"},{"start":"c5","length":"1"},{"start":"ce","length":"1"},{"start":"d4","length":"1"},{"start":"da","length":"1"},{"start":"df","length":"1"},{"start":"e3","length":"a"},{"start":"ee","length":"7"},{"start":"f6","length":"2"},{"start":"f9","length":"1"},{"start":"fc","length":"1"},{"start":"fe","length":"5"},{"start":"104","length":"3"},{"start":"108","length":"4"},{"start":"10d","length":"1"},{"start":"10f","length":"1"},{"start":"111","length":"1"},{"start":"113","length":"1"},{"start":"116","length":"2"},{"start":"11a","length":"1"},{"start":"11c","length":"1"}]},{"pgid":"1.9","version":"77'128","reported_seq":1534,"reported_epoch":657,"state":"active+clean","last_fresh":"2024-09-20T22:52:17.497912+0000","last_change":"2024-09-20T22:50:42.263313+0000","last_active":"2024-09-20T22:52:17.497912+0000","last_peered":"2024-09-20T22:52:17.497912+0000","last_clean":"2024-09-20T22:52:17.497912+0000","last_became_active":"2024-09-20T22:50:42.262862+0000","last_became_peered":"2024-09-20T22:50:42.262862+0000","last_unstale":"2024-09-20T22:52:17.497912+0000","last_undegraded":"2024-09-20T22:52:17.497912+0000","last_fullsized":"2024-09-20T22:52:17.497912+0000","mapping_epoch":561,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":562,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:49:52.301348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:49:52.301348+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"625'20","reported_seq":553,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-09-20T22:52:18.271140+0000","last_change":"2024-09-20T22:50:34.897059+0000","last_active":"2024-09-20T22:52:18.271140+0000","last_peered":"2024-09-20T22:52:18.271140+0000","last_clean":"2024-09-20T22:52:18.271140+0000","last_became_active":"2024-09-20T22:48:05.242811+0000","last_became_peered":"2024-09-20T22:48:05.242811+0000","last_unstale":"2024-09-20T22:52:18.271140+0000","last_undegraded":"2024-09-20T22:52:18.271140+0000","last_fullsized":"2024-09-20T22:52:18.271140+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-09-20T22:50:34.896923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:50:34.896923+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":1242,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-09-20T22:52:18.304733+0000","last_change":"2024-09-20T22:50:13.205821+0000","last_active":"2024-09-20T22:52:18.304733+0000","last_peered":"2024-09-20T22:52:18.304733+0000","last_clean":"2024-09-20T22:52:18.304733+0000","last_became_active":"2024-09-20T22:48:05.233999+0000","last_became_peered":"2024-09-20T22:48:05.233999+0000","last_unstale":"2024-09-20T22:52:18.304733+0000","last_undegraded":"2024-09-20T22:52:18.304733+0000","last_fullsized":"2024-09-20T22:52:18.304733+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:50:13.205743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:50:13.205743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"261'9","reported_seq":1372,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-09-20T22:52:18.440768+0000","last_change":"2024-09-20T22:52:18.218722+0000","last_active":"2024-09-20T22:52:18.440768+0000","last_peered":"2024-09-20T22:52:18.440768+0000","last_clean":"2024-09-20T22:52:18.440768+0000","last_became_active":"2024-09-20T22:41:42.185863+0000","last_became_peered":"2024-09-20T22:41:42.185863+0000","last_unstale":"2024-09-20T22:52:18.440768+0000","last_undegraded":"2024-09-20T22:52:18.440768+0000","last_fullsized":"2024-09-20T22:52:18.440768+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'9","last_scrub_stamp":"2024-09-20T22:52:18.218644+0000","last_deep_scrub":"261'9","last_deep_scrub_stamp":"2024-09-20T22:46:50.974735+0000","last_clean_scrub_stamp":"2024-09-20T22:52:18.218644+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:38.055984+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":540,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-09-20T22:52:18.271090+0000","last_change":"2024-09-20T22:51:20.296567+0000","last_active":"2024-09-20T22:52:18.271090+0000","last_peered":"2024-09-20T22:52:18.271090+0000","last_clean":"2024-09-20T22:52:18.271090+0000","last_became_active":"2024-09-20T22:51:20.296204+0000","last_became_peered":"2024-09-20T22:51:20.296204+0000","last_unstale":"2024-09-20T22:52:18.271090+0000","last_undegraded":"2024-09-20T22:52:18.271090+0000","last_fullsized":"2024-09-20T22:52:18.271090+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:02.932364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:51:02.932364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.8","version":"657'721","reported_seq":3418,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-09-20T22:52:18.440083+0000","last_change":"2024-09-20T22:52:17.529684+0000","last_active":"2024-09-20T22:52:18.440083+0000","last_peered":"2024-09-20T22:52:18.440083+0000","last_clean":"2024-09-20T22:52:18.440083+0000","last_became_active":"2024-09-20T22:48:31.224359+0000","last_became_peered":"2024-09-20T22:48:31.224359+0000","last_unstale":"2024-09-20T22:52:18.440083+0000","last_undegraded":"2024-09-20T22:52:18.440083+0000","last_fullsized":"2024-09-20T22:52:18.440083+0000","mapping_epoch":407,"log_start":"426'268","ondisk_log_start":"426'268","created":94,"last_epoch_clean":431,"parent":"0.0","parent_split_bits":0,"last_scrub":"534'447","last_scrub_stamp":"2024-09-20T22:50:14.324477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:50:14.324477+0000","objects_scrubbed":15,"log_size":453,"log_dups_size":445,"ondisk_log_size":453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.0095137409999999992,"stat_sum":{"num_bytes":37656266,"num_objects":19,"num_object_clones":16,"num_object_copies":57,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":1,"num_read":268,"num_read_kb":271649,"num_write":206,"num_write_kb":68718,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":26027409,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"a7","length":"1"},{"start":"c5","length":"1"},{"start":"ce","length":"1"},{"start":"d4","length":"1"},{"start":"da","length":"1"},{"start":"df","length":"1"},{"start":"e3","length":"a"},{"start":"ee","length":"7"},{"start":"f6","length":"2"},{"start":"f9","length":"1"},{"start":"fc","length":"1"},{"start":"fe","length":"5"},{"start":"104","length":"3"},{"start":"108","length":"4"},{"start":"10d","length":"1"},{"start":"10f","length":"1"},{"start":"111","length":"1"},{"start":"113","length":"1"},{"start":"116","length":"2"},{"start":"11a","length":"1"},{"start":"11c","length":"1"}]},{"pgid":"4.b","version":"625'15","reported_seq":618,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-09-20T22:52:18.272863+0000","last_change":"2024-09-20T22:51:20.296532+0000","last_active":"2024-09-20T22:52:18.272863+0000","last_peered":"2024-09-20T22:52:18.272863+0000","last_clean":"2024-09-20T22:52:18.272863+0000","last_became_active":"2024-09-20T22:51:20.273094+0000","last_became_peered":"2024-09-20T22:51:20.273094+0000","last_unstale":"2024-09-20T22:52:18.272863+0000","last_undegraded":"2024-09-20T22:52:18.272863+0000","last_fullsized":"2024-09-20T22:52:18.272863+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-09-20T22:50:36.883811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:50:36.883811+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":1227,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-09-20T22:52:18.440563+0000","last_change":"2024-09-20T22:51:12.755628+0000","last_active":"2024-09-20T22:52:18.440563+0000","last_peered":"2024-09-20T22:52:18.440563+0000","last_clean":"2024-09-20T22:52:18.440563+0000","last_became_active":"2024-09-20T22:41:38.155953+0000","last_became_peered":"2024-09-20T22:41:38.155953+0000","last_unstale":"2024-09-20T22:52:18.440563+0000","last_undegraded":"2024-09-20T22:52:18.440563+0000","last_fullsized":"2024-09-20T22:52:18.440563+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:12.755526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:51:12.755526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:32.975772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":1250,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-09-20T22:52:18.440514+0000","last_change":"2024-09-20T22:50:37.347748+0000","last_active":"2024-09-20T22:52:18.440514+0000","last_peered":"2024-09-20T22:52:18.440514+0000","last_clean":"2024-09-20T22:52:18.440514+0000","last_became_active":"2024-09-20T22:48:05.240613+0000","last_became_peered":"2024-09-20T22:48:05.240613+0000","last_unstale":"2024-09-20T22:52:18.440514+0000","last_undegraded":"2024-09-20T22:52:18.440514+0000","last_fullsized":"2024-09-20T22:52:18.440514+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:50:37.347636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:50:37.347636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":567,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-09-20T22:52:18.272574+0000","last_change":"2024-09-20T22:52:16.706497+0000","last_active":"2024-09-20T22:52:18.272574+0000","last_peered":"2024-09-20T22:52:18.272574+0000","last_clean":"2024-09-20T22:52:18.272574+0000","last_became_active":"2024-09-20T22:47:49.219486+0000","last_became_peered":"2024-09-20T22:47:49.219486+0000","last_unstale":"2024-09-20T22:52:18.272574+0000","last_undegraded":"2024-09-20T22:52:18.272574+0000","last_fullsized":"2024-09-20T22:52:18.272574+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:16.706429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:52:16.706429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:26.931246+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"657'1012","reported_seq":3910,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-09-20T22:52:18.272840+0000","last_change":"2024-09-20T22:52:17.513700+0000","last_active":"2024-09-20T22:52:18.272840+0000","last_peered":"2024-09-20T22:52:18.272840+0000","last_clean":"2024-09-20T22:52:18.272840+0000","last_became_active":"2024-09-20T22:51:23.270962+0000","last_became_peered":"2024-09-20T22:51:23.270962+0000","last_unstale":"2024-09-20T22:52:18.272840+0000","last_undegraded":"2024-09-20T22:52:18.272840+0000","last_fullsized":"2024-09-20T22:52:18.272840+0000","mapping_epoch":602,"log_start":"423'280","ondisk_log_start":"423'280","created":94,"last_epoch_clean":603,"parent":"0.0","parent_split_bits":0,"last_scrub":"590'828","last_scrub_stamp":"2024-09-20T22:51:10.909364+0000","last_deep_scrub":"590'828","last_deep_scrub_stamp":"2024-09-20T22:51:10.909364+0000","last_clean_scrub_stamp":"2024-09-20T22:51:10.909364+0000","objects_scrubbed":27,"log_size":732,"log_dups_size":649,"ondisk_log_size":732,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:21.652352+0000","scrub_duration":120,"objects_trimmed":0,"snaptrim_duration":0.017993539999999999,"stat_sum":{"num_bytes":68902335,"num_objects":29,"num_object_clones":24,"num_object_copies":87,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":1,"num_read":366,"num_read_kb":365046,"num_write":295,"num_write_kb":116449,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":91020358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"a7","length":"1"},{"start":"c5","length":"1"},{"start":"ce","length":"1"},{"start":"d4","length":"1"},{"start":"da","length":"1"},{"start":"df","length":"1"},{"start":"e3","length":"a"},{"start":"ee","length":"7"},{"start":"f6","length":"2"},{"start":"f9","length":"1"},{"start":"fc","length":"1"},{"start":"fe","length":"5"},{"start":"104","length":"3"},{"start":"108","length":"4"},{"start":"10d","length":"1"},{"start":"10f","length":"1"},{"start":"111","length":"1"},{"start":"113","length":"1"},{"start":"116","length":"2"},{"start":"11a","length":"1"},{"start":"11c","length":"1"}]},{"pgid":"4.a","version":"625'44","reported_seq":1326,"reported_epoch":657,"state":"active+clean","last_fresh":"2024-09-20T22:52:17.493582+0000","last_change":"2024-09-20T22:52:14.038882+0000","last_active":"2024-09-20T22:52:17.493582+0000","last_peered":"2024-09-20T22:52:17.493582+0000","last_clean":"2024-09-20T22:52:17.493582+0000","last_became_active":"2024-09-20T22:51:20.287620+0000","last_became_peered":"2024-09-20T22:51:20.287620+0000","last_unstale":"2024-09-20T22:52:17.493582+0000","last_undegraded":"2024-09-20T22:52:17.493582+0000","last_fullsized":"2024-09-20T22:52:17.493582+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'44","last_scrub_stamp":"2024-09-20T22:52:14.038788+0000","last_deep_scrub":"113'28","last_deep_scrub_stamp":"2024-09-20T22:43:12.725900+0000","last_clean_scrub_stamp":"2024-09-20T22:52:14.038788+0000","objects_scrubbed":9,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:32.613295+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":1224,"reported_epoch":657,"state":"active+clean","last_fresh":"2024-09-20T22:52:17.497601+0000","last_change":"2024-09-20T22:51:36.440271+0000","last_active":"2024-09-20T22:52:17.497601+0000","last_peered":"2024-09-20T22:52:17.497601+0000","last_clean":"2024-09-20T22:52:17.497601+0000","last_became_active":"2024-09-20T22:41:38.161524+0000","last_became_peered":"2024-09-20T22:41:38.161524+0000","last_unstale":"2024-09-20T22:52:17.497601+0000","last_undegraded":"2024-09-20T22:52:17.497601+0000","last_fullsized":"2024-09-20T22:52:17.497601+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:36.440160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:51:36.440160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:51.318747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":1238,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-09-20T22:52:18.440386+0000","last_change":"2024-09-20T22:50:15.301586+0000","last_active":"2024-09-20T22:52:18.440386+0000","last_peered":"2024-09-20T22:52:18.440386+0000","last_clean":"2024-09-20T22:52:18.440386+0000","last_became_active":"2024-09-20T22:47:49.223220+0000","last_became_peered":"2024-09-20T22:47:49.223220+0000","last_unstale":"2024-09-20T22:52:18.440386+0000","last_undegraded":"2024-09-20T22:52:18.440386+0000","last_fullsized":"2024-09-20T22:52:18.440386+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:50:15.301506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:50:15.301506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":1220,"reported_epoch":657,"state":"active+clean","last_fresh":"2024-09-20T22:52:17.497941+0000","last_change":"2024-09-20T22:52:01.643899+0000","last_active":"2024-09-20T22:52:17.497941+0000","last_peered":"2024-09-20T22:52:17.497941+0000","last_clean":"2024-09-20T22:52:17.497941+0000","last_became_active":"2024-09-20T22:41:44.199395+0000","last_became_peered":"2024-09-20T22:41:44.199395+0000","last_unstale":"2024-09-20T22:52:17.497941+0000","last_undegraded":"2024-09-20T22:52:17.497941+0000","last_fullsized":"2024-09-20T22:52:17.497941+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:01.643803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:47:18.329209+0000","last_clean_scrub_stamp":"2024-09-20T22:52:01.643803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:04.584695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.a","version":"658'751","reported_seq":3509,"reported_epoch":658,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T22:52:18.441045+0000","last_change":"2024-09-20T22:52:18.438331+0000","last_active":"2024-09-20T22:52:18.441045+0000","last_peered":"2024-09-20T22:52:18.441045+0000","last_clean":"2024-09-20T22:52:18.441045+0000","last_became_active":"2024-09-20T22:42:52.405085+0000","last_became_peered":"2024-09-20T22:42:52.405085+0000","last_unstale":"2024-09-20T22:52:18.441045+0000","last_undegraded":"2024-09-20T22:52:18.441045+0000","last_fullsized":"2024-09-20T22:52:18.441045+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"505'466","last_scrub_stamp":"2024-09-20T22:49:47.088165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:49:47.088165+0000","objects_scrubbed":13,"log_size":751,"log_dups_size":0,"ondisk_log_size":751,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":6,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.13339543600000001,"stat_sum":{"num_bytes":44584098,"num_objects":19,"num_object_clones":16,"num_object_copies":57,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":1,"num_read":334,"num_read_kb":364085,"num_write":241,"num_write_kb":94613,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"a7","length":"1"},{"start":"c5","length":"1"},{"start":"ce","length":"1"},{"start":"d4","length":"1"},{"start":"da","length":"1"},{"start":"df","length":"1"},{"start":"e3","length":"a"},{"start":"ee","length":"7"},{"start":"f6","length":"2"},{"start":"f9","length":"1"},{"start":"fc","length":"1"},{"start":"fe","length":"5"},{"start":"104","length":"1"},{"start":"106","length":"1"},{"start":"108","length":"4"},{"start":"10d","length":"1"},{"start":"10f","length":"1"},{"start":"111","length":"1"}]},{"pgid":"4.9","version":"625'18","reported_seq":583,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-09-20T22:52:18.270603+0000","last_change":"2024-09-20T22:52:11.295320+0000","last_active":"2024-09-20T22:52:18.270603+0000","last_peered":"2024-09-20T22:52:18.270603+0000","last_clean":"2024-09-20T22:52:18.270603+0000","last_became_active":"2024-09-20T22:52:11.286549+0000","last_became_peered":"2024-09-20T22:52:11.286549+0000","last_unstale":"2024-09-20T22:52:18.270603+0000","last_undegraded":"2024-09-20T22:52:18.270603+0000","last_fullsized":"2024-09-20T22:52:18.270603+0000","mapping_epoch":650,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":651,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-09-20T22:50:40.914303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:50:40.914303+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":21,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":1262,"reported_epoch":657,"state":"active+clean","last_fresh":"2024-09-20T22:52:17.497091+0000","last_change":"2024-09-20T22:51:55.653755+0000","last_active":"2024-09-20T22:52:17.497091+0000","last_peered":"2024-09-20T22:52:17.497091+0000","last_clean":"2024-09-20T22:52:17.497091+0000","last_became_active":"2024-09-20T22:51:20.293701+0000","last_became_peered":"2024-09-20T22:51:20.293701+0000","last_unstale":"2024-09-20T22:52:17.497091+0000","last_undegraded":"2024-09-20T22:52:17.497091+0000","last_fullsized":"2024-09-20T22:52:17.497091+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:55.653662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:46:52.465940+0000","last_clean_scrub_stamp":"2024-09-20T22:51:55.653662+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:56.990647+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":1252,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-09-20T22:52:18.440317+0000","last_change":"2024-09-20T22:51:20.286960+0000","last_active":"2024-09-20T22:52:18.440317+0000","last_peered":"2024-09-20T22:52:18.440317+0000","last_clean":"2024-09-20T22:52:18.440317+0000","last_became_active":"2024-09-20T22:51:20.286626+0000","last_became_peered":"2024-09-20T22:51:20.286626+0000","last_unstale":"2024-09-20T22:52:18.440317+0000","last_undegraded":"2024-09-20T22:52:18.440317+0000","last_fullsized":"2024-09-20T22:52:18.440317+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:05.254319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:46:01.828080+0000","last_clean_scrub_stamp":"2024-09-20T22:51:05.254319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":1212,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-09-20T22:52:18.304427+0000","last_change":"2024-09-20T22:51:20.279408+0000","last_active":"2024-09-20T22:52:18.304427+0000","last_peered":"2024-09-20T22:52:18.304427+0000","last_clean":"2024-09-20T22:52:18.304427+0000","last_became_active":"2024-09-20T22:51:20.279088+0000","last_became_peered":"2024-09-20T22:51:20.279088+0000","last_unstale":"2024-09-20T22:52:18.304427+0000","last_undegraded":"2024-09-20T22:52:18.304427+0000","last_fullsized":"2024-09-20T22:52:18.304427+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:50:39.266034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:55.170379+0000","last_clean_scrub_stamp":"2024-09-20T22:50:39.266034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"657'992","reported_seq":3934,"reported_epoch":658,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T22:52:18.440321+0000","last_change":"2024-09-20T22:52:18.303924+0000","last_active":"2024-09-20T22:52:18.440321+0000","last_peered":"2024-09-20T22:52:18.440321+0000","last_clean":"2024-09-20T22:52:18.440321+0000","last_became_active":"2024-09-20T22:51:23.271244+0000","last_became_peered":"2024-09-20T22:51:23.271244+0000","last_unstale":"2024-09-20T22:52:18.440321+0000","last_undegraded":"2024-09-20T22:52:18.440321+0000","last_fullsized":"2024-09-20T22:52:18.440321+0000","mapping_epoch":602,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":603,"parent":"0.0","parent_split_bits":0,"last_scrub":"194'116","last_scrub_stamp":"2024-09-20T22:44:37.097674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:44:37.097674+0000","objects_scrubbed":3,"log_size":992,"log_dups_size":0,"ondisk_log_size":992,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.79076015799999999,"stat_sum":{"num_bytes":54977335,"num_objects":23,"num_object_clones":18,"num_object_copies":69,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":2,"num_read":487,"num_read_kb":560244,"num_write":323,"num_write_kb":111474,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":76,"num_bytes_recovered":39288293,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"a7","length":"1"},{"start":"c5","length":"1"},{"start":"ce","length":"1"},{"start":"d4","length":"1"},{"start":"da","length":"1"},{"start":"df","length":"1"},{"start":"e3","length":"a"},{"start":"ee","length":"7"},{"start":"f6","length":"2"},{"start":"f9","length":"1"},{"start":"fc","length":"1"},{"start":"fe","length":"5"},{"start":"104","length":"3"},{"start":"108","length":"4"},{"start":"10d","length":"1"},{"start":"10f","length":"1"},{"start":"111","length":"1"},{"start":"117","length":"1"},{"start":"11c","length":"1"}]},{"pgid":"4.8","version":"625'27","reported_seq":1266,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-09-20T22:52:18.439884+0000","last_change":"2024-09-20T22:51:53.020721+0000","last_active":"2024-09-20T22:52:18.439884+0000","last_peered":"2024-09-20T22:52:18.439884+0000","last_clean":"2024-09-20T22:52:18.439884+0000","last_became_active":"2024-09-20T22:41:40.252113+0000","last_became_peered":"2024-09-20T22:41:40.252113+0000","last_unstale":"2024-09-20T22:52:18.439884+0000","last_undegraded":"2024-09-20T22:52:18.439884+0000","last_fullsized":"2024-09-20T22:52:18.439884+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'27","last_scrub_stamp":"2024-09-20T22:51:53.020628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:51:53.020628+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:14.005659+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'2","reported_seq":1287,"reported_epoch":657,"state":"active+clean","last_fresh":"2024-09-20T22:52:17.497218+0000","last_change":"2024-09-20T22:51:26.421985+0000","last_active":"2024-09-20T22:52:17.497218+0000","last_peered":"2024-09-20T22:52:17.497218+0000","last_clean":"2024-09-20T22:52:17.497218+0000","last_became_active":"2024-09-20T22:48:05.238098+0000","last_became_peered":"2024-09-20T22:48:05.238098+0000","last_unstale":"2024-09-20T22:52:17.497218+0000","last_undegraded":"2024-09-20T22:52:17.497218+0000","last_fullsized":"2024-09-20T22:52:17.497218+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-09-20T22:51:26.421896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:51:26.421896+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:53.826247+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"408'10","reported_seq":1381,"reported_epoch":657,"state":"active+clean","last_fresh":"2024-09-20T22:52:17.498074+0000","last_change":"2024-09-20T22:52:15.703572+0000","last_active":"2024-09-20T22:52:17.498074+0000","last_peered":"2024-09-20T22:52:17.498074+0000","last_clean":"2024-09-20T22:52:17.498074+0000","last_became_active":"2024-09-20T22:48:05.239344+0000","last_became_peered":"2024-09-20T22:48:05.239344+0000","last_unstale":"2024-09-20T22:52:17.498074+0000","last_undegraded":"2024-09-20T22:52:17.498074+0000","last_fullsized":"2024-09-20T22:52:17.498074+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"408'10","last_scrub_stamp":"2024-09-20T22:52:15.703464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:52:15.703464+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:23.273102+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":1227,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-09-20T22:52:18.440119+0000","last_change":"2024-09-20T22:51:54.066995+0000","last_active":"2024-09-20T22:52:18.440119+0000","last_peered":"2024-09-20T22:52:18.440119+0000","last_clean":"2024-09-20T22:52:18.440119+0000","last_became_active":"2024-09-20T22:47:49.227569+0000","last_became_peered":"2024-09-20T22:47:49.227569+0000","last_unstale":"2024-09-20T22:52:18.440119+0000","last_undegraded":"2024-09-20T22:52:18.440119+0000","last_fullsized":"2024-09-20T22:52:18.440119+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:54.066913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:51:54.066913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:18.059705+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":1252,"reported_epoch":657,"state":"active+clean","last_fresh":"2024-09-20T22:52:17.493900+0000","last_change":"2024-09-20T22:51:58.129589+0000","last_active":"2024-09-20T22:52:17.493900+0000","last_peered":"2024-09-20T22:52:17.493900+0000","last_clean":"2024-09-20T22:52:17.493900+0000","last_became_active":"2024-09-20T22:47:49.213292+0000","last_became_peered":"2024-09-20T22:47:49.213292+0000","last_unstale":"2024-09-20T22:52:17.493900+0000","last_undegraded":"2024-09-20T22:52:17.493900+0000","last_fullsized":"2024-09-20T22:52:17.493900+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:58.129502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:51:58.129502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:11.276401+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"625'12","reported_seq":1254,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-09-20T22:52:18.306169+0000","last_change":"2024-09-20T22:52:14.322926+0000","last_active":"2024-09-20T22:52:18.306169+0000","last_peered":"2024-09-20T22:52:18.306169+0000","last_clean":"2024-09-20T22:52:18.306169+0000","last_became_active":"2024-09-20T22:43:09.116887+0000","last_became_peered":"2024-09-20T22:43:09.116887+0000","last_unstale":"2024-09-20T22:52:18.306169+0000","last_undegraded":"2024-09-20T22:52:18.306169+0000","last_fullsized":"2024-09-20T22:52:18.306169+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'12","last_scrub_stamp":"2024-09-20T22:52:14.322811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:52:14.322811+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:23.704685+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":1225,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-09-20T22:52:18.440869+0000","last_change":"2024-09-20T22:51:20.293964+0000","last_active":"2024-09-20T22:52:18.440869+0000","last_peered":"2024-09-20T22:52:18.440869+0000","last_clean":"2024-09-20T22:52:18.440869+0000","last_became_active":"2024-09-20T22:51:20.293232+0000","last_became_peered":"2024-09-20T22:51:20.293232+0000","last_unstale":"2024-09-20T22:52:18.440869+0000","last_undegraded":"2024-09-20T22:52:18.440869+0000","last_fullsized":"2024-09-20T22:52:18.440869+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:50:22.597796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:09.705632+0000","last_clean_scrub_stamp":"2024-09-20T22:50:22.597796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"74'1","reported_seq":1233,"reported_epoch":657,"state":"active+clean","last_fresh":"2024-09-20T22:52:17.497014+0000","last_change":"2024-09-20T22:51:05.527388+0000","last_active":"2024-09-20T22:52:17.497014+0000","last_peered":"2024-09-20T22:52:17.497014+0000","last_clean":"2024-09-20T22:52:17.497014+0000","last_became_active":"2024-09-20T22:48:05.237882+0000","last_became_peered":"2024-09-20T22:48:05.237882+0000","last_unstale":"2024-09-20T22:52:17.497014+0000","last_undegraded":"2024-09-20T22:52:17.497014+0000","last_fullsized":"2024-09-20T22:52:17.497014+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-09-20T22:51:05.527264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:51:05.527264+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"625'15","reported_seq":585,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-09-20T22:52:18.273759+0000","last_change":"2024-09-20T22:51:52.816826+0000","last_active":"2024-09-20T22:52:18.273759+0000","last_peered":"2024-09-20T22:52:18.273759+0000","last_clean":"2024-09-20T22:52:18.273759+0000","last_became_active":"2024-09-20T22:47:49.231315+0000","last_became_peered":"2024-09-20T22:47:49.231315+0000","last_unstale":"2024-09-20T22:52:18.273759+0000","last_undegraded":"2024-09-20T22:52:18.273759+0000","last_fullsized":"2024-09-20T22:52:18.273759+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'15","last_scrub_stamp":"2024-09-20T22:51:52.816714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:51:52.816714+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:55.354430+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":1248,"reported_epoch":657,"state":"active+clean","last_fresh":"2024-09-20T22:52:17.497824+0000","last_change":"2024-09-20T22:51:12.541531+0000","last_active":"2024-09-20T22:52:17.497824+0000","last_peered":"2024-09-20T22:52:17.497824+0000","last_clean":"2024-09-20T22:52:17.497824+0000","last_became_active":"2024-09-20T22:48:05.238801+0000","last_became_peered":"2024-09-20T22:48:05.238801+0000","last_unstale":"2024-09-20T22:52:17.497824+0000","last_undegraded":"2024-09-20T22:52:17.497824+0000","last_fullsized":"2024-09-20T22:52:17.497824+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:12.541457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:51:12.541457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:38.408641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":1216,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-09-20T22:52:18.305919+0000","last_change":"2024-09-20T22:51:20.279846+0000","last_active":"2024-09-20T22:52:18.305919+0000","last_peered":"2024-09-20T22:52:18.305919+0000","last_clean":"2024-09-20T22:52:18.305919+0000","last_became_active":"2024-09-20T22:51:20.279397+0000","last_became_peered":"2024-09-20T22:51:20.279397+0000","last_unstale":"2024-09-20T22:52:18.305919+0000","last_undegraded":"2024-09-20T22:52:18.305919+0000","last_fullsized":"2024-09-20T22:52:18.305919+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:50:17.192813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:50:17.192813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":1246,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-09-20T22:52:18.440410+0000","last_change":"2024-09-20T22:50:16.268746+0000","last_active":"2024-09-20T22:52:18.440410+0000","last_peered":"2024-09-20T22:52:18.440410+0000","last_clean":"2024-09-20T22:52:18.440410+0000","last_became_active":"2024-09-20T22:48:05.239710+0000","last_became_peered":"2024-09-20T22:48:05.239710+0000","last_unstale":"2024-09-20T22:52:18.440410+0000","last_undegraded":"2024-09-20T22:52:18.440410+0000","last_fullsized":"2024-09-20T22:52:18.440410+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:50:16.268655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:50:16.268655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"625'15","reported_seq":1248,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-09-20T22:52:18.440805+0000","last_change":"2024-09-20T22:51:05.675215+0000","last_active":"2024-09-20T22:52:18.440805+0000","last_peered":"2024-09-20T22:52:18.440805+0000","last_clean":"2024-09-20T22:52:18.440805+0000","last_became_active":"2024-09-20T22:41:40.252486+0000","last_became_peered":"2024-09-20T22:41:40.252486+0000","last_unstale":"2024-09-20T22:52:18.440805+0000","last_undegraded":"2024-09-20T22:52:18.440805+0000","last_fullsized":"2024-09-20T22:52:18.440805+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-09-20T22:51:05.675095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:51:05.675095+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:18.851214+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":574,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-09-20T22:52:18.272304+0000","last_change":"2024-09-20T22:52:13.735451+0000","last_active":"2024-09-20T22:52:18.272304+0000","last_peered":"2024-09-20T22:52:18.272304+0000","last_clean":"2024-09-20T22:52:18.272304+0000","last_became_active":"2024-09-20T22:47:49.229936+0000","last_became_peered":"2024-09-20T22:47:49.229936+0000","last_unstale":"2024-09-20T22:52:18.272304+0000","last_undegraded":"2024-09-20T22:52:18.272304+0000","last_fullsized":"2024-09-20T22:52:18.272304+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:13.735382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:52:13.735382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:22.644334+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"74'8","reported_seq":1350,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-09-20T22:52:18.306474+0000","last_change":"2024-09-20T22:50:38.269118+0000","last_active":"2024-09-20T22:52:18.306474+0000","last_peered":"2024-09-20T22:52:18.306474+0000","last_clean":"2024-09-20T22:52:18.306474+0000","last_became_active":"2024-09-20T22:41:42.186352+0000","last_became_peered":"2024-09-20T22:41:42.186352+0000","last_unstale":"2024-09-20T22:52:18.306474+0000","last_undegraded":"2024-09-20T22:52:18.306474+0000","last_fullsized":"2024-09-20T22:52:18.306474+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-09-20T22:50:38.269010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:50:38.269010+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":534,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-09-20T22:52:18.271410+0000","last_change":"2024-09-20T22:52:16.623463+0000","last_active":"2024-09-20T22:52:18.271410+0000","last_peered":"2024-09-20T22:52:18.271410+0000","last_clean":"2024-09-20T22:52:18.271410+0000","last_became_active":"2024-09-20T22:48:05.249656+0000","last_became_peered":"2024-09-20T22:48:05.249656+0000","last_unstale":"2024-09-20T22:52:18.271410+0000","last_undegraded":"2024-09-20T22:52:18.271410+0000","last_fullsized":"2024-09-20T22:52:18.271410+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:16.623368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:52:16.623368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:22.519570+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"625'20","reported_seq":1262,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-09-20T22:52:18.306205+0000","last_change":"2024-09-20T22:51:20.298112+0000","last_active":"2024-09-20T22:52:18.306205+0000","last_peered":"2024-09-20T22:52:18.306205+0000","last_clean":"2024-09-20T22:52:18.306205+0000","last_became_active":"2024-09-20T22:51:20.281859+0000","last_became_peered":"2024-09-20T22:51:20.281859+0000","last_unstale":"2024-09-20T22:52:18.306205+0000","last_undegraded":"2024-09-20T22:52:18.306205+0000","last_fullsized":"2024-09-20T22:52:18.306205+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-09-20T22:50:15.240945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:50:15.240945+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":1236,"reported_epoch":657,"state":"active+clean","last_fresh":"2024-09-20T22:52:17.496921+0000","last_change":"2024-09-20T22:49:54.381836+0000","last_active":"2024-09-20T22:52:17.496921+0000","last_peered":"2024-09-20T22:52:17.496921+0000","last_clean":"2024-09-20T22:52:17.496921+0000","last_became_active":"2024-09-20T22:48:05.240231+0000","last_became_peered":"2024-09-20T22:48:05.240231+0000","last_unstale":"2024-09-20T22:52:17.496921+0000","last_undegraded":"2024-09-20T22:52:17.496921+0000","last_fullsized":"2024-09-20T22:52:17.496921+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:54.381721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:49:54.381721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"600'11","reported_seq":1390,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-09-20T22:52:18.440654+0000","last_change":"2024-09-20T22:51:20.303774+0000","last_active":"2024-09-20T22:52:18.440654+0000","last_peered":"2024-09-20T22:52:18.440654+0000","last_clean":"2024-09-20T22:52:18.440654+0000","last_became_active":"2024-09-20T22:51:20.283393+0000","last_became_peered":"2024-09-20T22:51:20.283393+0000","last_unstale":"2024-09-20T22:52:18.440654+0000","last_undegraded":"2024-09-20T22:52:18.440654+0000","last_fullsized":"2024-09-20T22:52:18.440654+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'10","last_scrub_stamp":"2024-09-20T22:49:51.715471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:49:51.715471+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":558,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-09-20T22:52:18.273490+0000","last_change":"2024-09-20T22:50:21.853169+0000","last_active":"2024-09-20T22:52:18.273490+0000","last_peered":"2024-09-20T22:52:18.273490+0000","last_clean":"2024-09-20T22:52:18.273490+0000","last_became_active":"2024-09-20T22:47:49.230674+0000","last_became_peered":"2024-09-20T22:47:49.230674+0000","last_unstale":"2024-09-20T22:52:18.273490+0000","last_undegraded":"2024-09-20T22:52:18.273490+0000","last_fullsized":"2024-09-20T22:52:18.273490+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:50:21.853070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:50:21.853070+0000","last_clean_scrub_stamp":"2024-09-20T22:50:21.853070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"625'19","reported_seq":576,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-09-20T22:52:18.270764+0000","last_change":"2024-09-20T22:51:20.317969+0000","last_active":"2024-09-20T22:52:18.270764+0000","last_peered":"2024-09-20T22:52:18.270764+0000","last_clean":"2024-09-20T22:52:18.270764+0000","last_became_active":"2024-09-20T22:51:20.289584+0000","last_became_peered":"2024-09-20T22:51:20.289584+0000","last_unstale":"2024-09-20T22:52:18.270764+0000","last_undegraded":"2024-09-20T22:52:18.270764+0000","last_fullsized":"2024-09-20T22:52:18.270764+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-09-20T22:51:07.926382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:51:07.926382+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:30.537714+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":534,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-09-20T22:52:18.271028+0000","last_change":"2024-09-20T22:51:08.874933+0000","last_active":"2024-09-20T22:52:18.271028+0000","last_peered":"2024-09-20T22:52:18.271028+0000","last_clean":"2024-09-20T22:52:18.271028+0000","last_became_active":"2024-09-20T22:48:05.252026+0000","last_became_peered":"2024-09-20T22:48:05.252026+0000","last_unstale":"2024-09-20T22:52:18.271028+0000","last_undegraded":"2024-09-20T22:52:18.271028+0000","last_fullsized":"2024-09-20T22:52:18.271028+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:08.874812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:51:08.874812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":128,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-09-20T22:52:18.440259+0000","last_change":"2024-09-20T22:51:20.291857+0000","last_active":"2024-09-20T22:52:18.440259+0000","last_peered":"2024-09-20T22:52:18.440259+0000","last_clean":"2024-09-20T22:52:18.440259+0000","last_became_active":"2024-09-20T22:51:20.291264+0000","last_became_peered":"2024-09-20T22:51:20.291264+0000","last_unstale":"2024-09-20T22:52:18.440259+0000","last_undegraded":"2024-09-20T22:52:18.440259+0000","last_fullsized":"2024-09-20T22:52:18.440259+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:06.651167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:51:06.651167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:34.034653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":1218,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-09-20T22:52:18.304080+0000","last_change":"2024-09-20T22:50:20.207052+0000","last_active":"2024-09-20T22:52:18.304080+0000","last_peered":"2024-09-20T22:52:18.304080+0000","last_clean":"2024-09-20T22:52:18.304080+0000","last_became_active":"2024-09-20T22:47:49.216141+0000","last_became_peered":"2024-09-20T22:47:49.216141+0000","last_unstale":"2024-09-20T22:52:18.304080+0000","last_undegraded":"2024-09-20T22:52:18.304080+0000","last_fullsized":"2024-09-20T22:52:18.304080+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:50:20.206984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:50:20.206984+0000","last_clean_scrub_stamp":"2024-09-20T22:50:20.206984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"625'19","reported_seq":870,"reported_epoch":657,"state":"active+clean","last_fresh":"2024-09-20T22:52:17.493872+0000","last_change":"2024-09-20T22:51:25.879860+0000","last_active":"2024-09-20T22:52:17.493872+0000","last_peered":"2024-09-20T22:52:17.493872+0000","last_clean":"2024-09-20T22:52:17.493872+0000","last_became_active":"2024-09-20T22:51:20.283868+0000","last_became_peered":"2024-09-20T22:51:20.283868+0000","last_unstale":"2024-09-20T22:52:17.493872+0000","last_undegraded":"2024-09-20T22:52:17.493872+0000","last_fullsized":"2024-09-20T22:52:17.493872+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"404'13","last_scrub_stamp":"2024-09-20T22:51:25.879750+0000","last_deep_scrub":"404'13","last_deep_scrub_stamp":"2024-09-20T22:48:04.276282+0000","last_clean_scrub_stamp":"2024-09-20T22:51:25.879750+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:35.823216+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,0],"acting":[6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":1251,"reported_epoch":657,"state":"active+clean","last_fresh":"2024-09-20T22:52:17.497462+0000","last_change":"2024-09-20T22:49:56.396435+0000","last_active":"2024-09-20T22:52:17.497462+0000","last_peered":"2024-09-20T22:52:17.497462+0000","last_clean":"2024-09-20T22:52:17.497462+0000","last_became_active":"2024-09-20T22:48:05.238812+0000","last_became_peered":"2024-09-20T22:48:05.238812+0000","last_unstale":"2024-09-20T22:52:17.497462+0000","last_undegraded":"2024-09-20T22:52:17.497462+0000","last_fullsized":"2024-09-20T22:52:17.497462+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:56.396347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:31.138067+0000","last_clean_scrub_stamp":"2024-09-20T22:49:56.396347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":1228,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-09-20T22:52:18.304509+0000","last_change":"2024-09-20T22:51:20.288308+0000","last_active":"2024-09-20T22:52:18.304509+0000","last_peered":"2024-09-20T22:52:18.304509+0000","last_clean":"2024-09-20T22:52:18.304509+0000","last_became_active":"2024-09-20T22:51:20.288000+0000","last_became_peered":"2024-09-20T22:51:20.288000+0000","last_unstale":"2024-09-20T22:52:18.304509+0000","last_undegraded":"2024-09-20T22:52:18.304509+0000","last_fullsized":"2024-09-20T22:52:18.304509+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:50:44.313182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:50:44.313182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":564,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-09-20T22:52:18.272383+0000","last_change":"2024-09-20T22:51:20.278471+0000","last_active":"2024-09-20T22:52:18.272383+0000","last_peered":"2024-09-20T22:52:18.272383+0000","last_clean":"2024-09-20T22:52:18.272383+0000","last_became_active":"2024-09-20T22:51:20.278147+0000","last_became_peered":"2024-09-20T22:51:20.278147+0000","last_unstale":"2024-09-20T22:52:18.272383+0000","last_undegraded":"2024-09-20T22:52:18.272383+0000","last_fullsized":"2024-09-20T22:52:18.272383+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:50:35.895483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:50:35.895483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"625'24","reported_seq":1049,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-09-20T22:52:18.439714+0000","last_change":"2024-09-20T22:50:39.665781+0000","last_active":"2024-09-20T22:52:18.439714+0000","last_peered":"2024-09-20T22:52:18.439714+0000","last_clean":"2024-09-20T22:52:18.439714+0000","last_became_active":"2024-09-20T22:44:09.170785+0000","last_became_peered":"2024-09-20T22:44:09.170785+0000","last_unstale":"2024-09-20T22:52:18.439714+0000","last_undegraded":"2024-09-20T22:52:18.439714+0000","last_fullsized":"2024-09-20T22:52:18.439714+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"470'16","last_scrub_stamp":"2024-09-20T22:50:39.665682+0000","last_deep_scrub":"470'16","last_deep_scrub_stamp":"2024-09-20T22:49:10.659868+0000","last_clean_scrub_stamp":"2024-09-20T22:50:39.665682+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":1233,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-09-20T22:52:18.440940+0000","last_change":"2024-09-20T22:51:20.294181+0000","last_active":"2024-09-20T22:52:18.440940+0000","last_peered":"2024-09-20T22:52:18.440940+0000","last_clean":"2024-09-20T22:52:18.440940+0000","last_became_active":"2024-09-20T22:51:20.293645+0000","last_became_peered":"2024-09-20T22:51:20.293645+0000","last_unstale":"2024-09-20T22:52:18.440940+0000","last_undegraded":"2024-09-20T22:52:18.440940+0000","last_fullsized":"2024-09-20T22:52:18.440940+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:50:13.688169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:45:24.765414+0000","last_clean_scrub_stamp":"2024-09-20T22:50:13.688169+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":1240,"reported_epoch":657,"state":"active+clean","last_fresh":"2024-09-20T22:52:17.497580+0000","last_change":"2024-09-20T22:51:10.610718+0000","last_active":"2024-09-20T22:52:17.497580+0000","last_peered":"2024-09-20T22:52:17.497580+0000","last_clean":"2024-09-20T22:52:17.497580+0000","last_became_active":"2024-09-20T22:48:05.238334+0000","last_became_peered":"2024-09-20T22:48:05.238334+0000","last_unstale":"2024-09-20T22:52:17.497580+0000","last_undegraded":"2024-09-20T22:52:17.497580+0000","last_fullsized":"2024-09-20T22:52:17.497580+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:10.610588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:50.174861+0000","last_clean_scrub_stamp":"2024-09-20T22:51:10.610588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":1218,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-09-20T22:52:18.304648+0000","last_change":"2024-09-20T22:51:04.379662+0000","last_active":"2024-09-20T22:52:18.304648+0000","last_peered":"2024-09-20T22:52:18.304648+0000","last_clean":"2024-09-20T22:52:18.304648+0000","last_became_active":"2024-09-20T22:47:49.214487+0000","last_became_peered":"2024-09-20T22:47:49.214487+0000","last_unstale":"2024-09-20T22:52:18.304648+0000","last_undegraded":"2024-09-20T22:52:18.304648+0000","last_fullsized":"2024-09-20T22:52:18.304648+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:04.379558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:51:04.379558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:19.515045+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"625'8","reported_seq":1238,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-09-20T22:52:18.304043+0000","last_change":"2024-09-20T22:51:20.314500+0000","last_active":"2024-09-20T22:52:18.304043+0000","last_peered":"2024-09-20T22:52:18.304043+0000","last_clean":"2024-09-20T22:52:18.304043+0000","last_became_active":"2024-09-20T22:51:20.290269+0000","last_became_peered":"2024-09-20T22:51:20.290269+0000","last_unstale":"2024-09-20T22:52:18.304043+0000","last_undegraded":"2024-09-20T22:52:18.304043+0000","last_fullsized":"2024-09-20T22:52:18.304043+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'4","last_scrub_stamp":"2024-09-20T22:50:40.292964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:50:40.292964+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":566,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-09-20T22:52:18.273565+0000","last_change":"2024-09-20T22:50:43.756048+0000","last_active":"2024-09-20T22:52:18.273565+0000","last_peered":"2024-09-20T22:52:18.273565+0000","last_clean":"2024-09-20T22:52:18.273565+0000","last_became_active":"2024-09-20T22:47:49.231964+0000","last_became_peered":"2024-09-20T22:47:49.231964+0000","last_unstale":"2024-09-20T22:52:18.273565+0000","last_undegraded":"2024-09-20T22:52:18.273565+0000","last_fullsized":"2024-09-20T22:52:18.273565+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:50:43.755981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:50:43.755981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":1248,"reported_epoch":657,"state":"active+clean","last_fresh":"2024-09-20T22:52:17.493676+0000","last_change":"2024-09-20T22:50:36.933344+0000","last_active":"2024-09-20T22:52:17.493676+0000","last_peered":"2024-09-20T22:52:17.493676+0000","last_clean":"2024-09-20T22:52:17.493676+0000","last_became_active":"2024-09-20T22:48:05.244659+0000","last_became_peered":"2024-09-20T22:48:05.244659+0000","last_unstale":"2024-09-20T22:52:17.493676+0000","last_undegraded":"2024-09-20T22:52:17.493676+0000","last_fullsized":"2024-09-20T22:52:17.493676+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:50:36.933255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:50:36.933255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":1232,"reported_epoch":657,"state":"active+clean","last_fresh":"2024-09-20T22:52:17.496585+0000","last_change":"2024-09-20T22:51:01.446349+0000","last_active":"2024-09-20T22:52:17.496585+0000","last_peered":"2024-09-20T22:52:17.496585+0000","last_clean":"2024-09-20T22:52:17.496585+0000","last_became_active":"2024-09-20T22:48:05.237765+0000","last_became_peered":"2024-09-20T22:48:05.237765+0000","last_unstale":"2024-09-20T22:52:17.496585+0000","last_undegraded":"2024-09-20T22:52:17.496585+0000","last_fullsized":"2024-09-20T22:52:17.496585+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:01.446278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:10.233487+0000","last_clean_scrub_stamp":"2024-09-20T22:51:01.446278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":831,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-09-20T22:52:18.440729+0000","last_change":"2024-09-20T22:51:29.904430+0000","last_active":"2024-09-20T22:52:18.440729+0000","last_peered":"2024-09-20T22:52:18.440729+0000","last_clean":"2024-09-20T22:52:18.440729+0000","last_became_active":"2024-09-20T22:51:20.297739+0000","last_became_peered":"2024-09-20T22:51:20.297739+0000","last_unstale":"2024-09-20T22:52:18.440729+0000","last_undegraded":"2024-09-20T22:52:18.440729+0000","last_fullsized":"2024-09-20T22:52:18.440729+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:29.904355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:51:29.904355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:36.892277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":1250,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-09-20T22:52:18.303874+0000","last_change":"2024-09-20T22:51:20.275893+0000","last_active":"2024-09-20T22:52:18.303874+0000","last_peered":"2024-09-20T22:52:18.303874+0000","last_clean":"2024-09-20T22:52:18.303874+0000","last_became_active":"2024-09-20T22:51:20.274891+0000","last_became_peered":"2024-09-20T22:51:20.274891+0000","last_unstale":"2024-09-20T22:52:18.303874+0000","last_undegraded":"2024-09-20T22:52:18.303874+0000","last_fullsized":"2024-09-20T22:52:18.303874+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:50:41.281470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:50:41.281470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"625'19","reported_seq":1289,"reported_epoch":657,"state":"active+clean","last_fresh":"2024-09-20T22:52:17.494514+0000","last_change":"2024-09-20T22:52:16.953034+0000","last_active":"2024-09-20T22:52:17.494514+0000","last_peered":"2024-09-20T22:52:17.494514+0000","last_clean":"2024-09-20T22:52:17.494514+0000","last_became_active":"2024-09-20T22:51:20.284937+0000","last_became_peered":"2024-09-20T22:51:20.284937+0000","last_unstale":"2024-09-20T22:52:17.494514+0000","last_undegraded":"2024-09-20T22:52:17.494514+0000","last_fullsized":"2024-09-20T22:52:17.494514+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'19","last_scrub_stamp":"2024-09-20T22:52:16.952955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:52:16.952955+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:27.426198+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":559,"reported_epoch":658,"state":"active+clean","last_fresh":"2024-09-20T22:52:18.273385+0000","last_change":"2024-09-20T22:49:23.660691+0000","last_active":"2024-09-20T22:52:18.273385+0000","last_peered":"2024-09-20T22:52:18.273385+0000","last_clean":"2024-09-20T22:52:18.273385+0000","last_became_active":"2024-09-20T22:47:49.220971+0000","last_became_peered":"2024-09-20T22:47:49.220971+0000","last_unstale":"2024-09-20T22:52:18.273385+0000","last_undegraded":"2024-09-20T22:52:18.273385+0000","last_fullsized":"2024-09-20T22:52:18.273385+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:23.660625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:51.667341+0000","last_clean_scrub_stamp":"2024-09-20T22:49:23.660625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":587298804,"num_objects":271,"num_object_clones":221,"num_object_copies":813,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":24,"num_objects_unfound":0,"num_objects_dirty":271,"num_whiteouts":12,"num_read":4524,"num_read_kb":4753436,"num_write":3218,"num_write_kb":1197131,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":766,"num_bytes_recovered":463318677,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1286283264,"data_stored":1872465612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8470,"ondisk_log_size":8470,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":84,"ondisk_log_size":84,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1109,"num_read_kb":866,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":312,"num_bytes_recovered":7464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":499712,"data_stored":13224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":739,"ondisk_log_size":739,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":1034,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":4931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":630,"num_read_kb":625,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1186,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":384,"ondisk_log_size":384,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070596,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":243180,"kb_used_data":198100,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":93501972,"statfs":{"total":95995035648,"available":95746019328,"internally_reserved":0,"allocated":202854400,"data_stored":277878021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":46135754},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430346,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":218728,"kb_used_data":169484,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":93526424,"statfs":{"total":95995035648,"available":95771058176,"internally_reserved":0,"allocated":173551616,"data_stored":255165910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691983,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":176620,"kb_used_data":119060,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":93568532,"statfs":{"total":95995035648,"available":95814176768,"internally_reserved":0,"allocated":121917440,"data_stored":180365604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986324,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":141412,"kb_used_data":110220,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93603740,"statfs":{"total":95995035648,"available":95850229760,"internally_reserved":0,"allocated":112865280,"data_stored":161501486,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182554,"num_pgs":76,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":316492,"kb_used_data":267248,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":93428660,"statfs":{"total":95995035648,"available":95670947840,"internally_reserved":0,"allocated":273661952,"data_stored":388430654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":50395591},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411487,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":283928,"kb_used_data":222208,"kb_used_omap":1,"kb_used_meta":61694,"kb_avail":93461224,"statfs":{"total":95995035648,"available":95704293376,"internally_reserved":0,"allocated":227540992,"data_stored":318648142,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":63175109},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":55834575012,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738537,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":214272,"kb_used_data":169192,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":93530880,"statfs":{"total":95995035648,"available":95775621120,"internally_reserved":0,"allocated":173252608,"data_stored":260573735,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":167395328,"data_stored":257025739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":43384832,"data_stored":56524481,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":221450240,"data_stored":314838123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":267198464,"data_stored":384283920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":107008000,"data_stored":157946775,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":115486720,"data_stored":176226701,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":167690240,"data_stored":251617533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":196669440,"data_stored":274002340,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-20T22:52:20.827 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph pg dump --format=json 2024-09-20T22:52:20.869 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:52:20.883 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:52:20.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:20.910 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:20.910 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:52:21.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:20 smithi154 ceph-mon[33264]: 4.2 scrub starts 2024-09-20T22:52:21.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:20 smithi154 ceph-mon[33264]: 4.2 scrub ok 2024-09-20T22:52:21.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:20 smithi154 ceph-mon[33264]: 6.14 scrub starts 2024-09-20T22:52:21.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:20 smithi154 ceph-mon[33264]: 6.14 scrub ok 2024-09-20T22:52:21.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:20 smithi154 ceph-mon[33264]: from='client.26510 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T22:52:21.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:20 smithi154 ceph-mon[33264]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:52:21.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:20 smithi154 ceph-mon[33264]: purged_snaps scrub starts 2024-09-20T22:52:21.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:20 smithi154 ceph-mon[33264]: purged_snaps scrub ok 2024-09-20T22:52:21.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:20 smithi154 ceph-mon[33264]: osdmap e660: 8 total, 8 up, 7 in 2024-09-20T22:52:21.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:20 smithi154 ceph-mon[33264]: 5.17 scrub starts 2024-09-20T22:52:21.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:20 smithi154 ceph-mon[33264]: 5.17 scrub ok 2024-09-20T22:52:21.011 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:52:21.036 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:21.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:21.037 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:52:21.138 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:52:21.158 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:21.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:21.159 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:52:21.259 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:52:21.280 INFO:tasks.rados.rados.0.smithi084.stdout:2592: done (0 left) 2024-09-20T22:52:21.281 INFO:tasks.rados.rados.0.smithi084.stdout:2593: write (excl) oid 10 current snap is 293 2024-09-20T22:52:21.281 INFO:tasks.rados.rados.0.smithi084.stdout:2593: seq_num 895 ranges {642265=561156,1673384=678640,2828402=213863} 2024-09-20T22:52:21.285 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:21.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:21.285 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:52:21.287 INFO:tasks.rados.rados.0.smithi084.stdout:2593: writing smithi08477203-10 from 642265 to 1203421 tid 1 2024-09-20T22:52:21.294 INFO:tasks.rados.rados.0.smithi084.stdout:2593: writing smithi08477203-10 from 1673384 to 2352024 tid 2 2024-09-20T22:52:21.298 INFO:tasks.rados.rados.0.smithi084.stdout:2593: writing smithi08477203-10 from 2828402 to 3042265 tid 3 2024-09-20T22:52:21.298 INFO:tasks.rados.rados.0.smithi084.stdout:2594: delete oid 13 current snap is 293 2024-09-20T22:52:21.298 INFO:tasks.rados.rados.0.smithi084.stdout:2593: finishing write tid 1 to smithi08477203-10 2024-09-20T22:52:21.298 INFO:tasks.rados.rados.0.smithi084.stdout:2593: finishing write tid 2 to smithi08477203-10 2024-09-20T22:52:21.301 INFO:tasks.rados.rados.0.smithi084.stdout:2593: finishing write tid 3 to smithi08477203-10 2024-09-20T22:52:21.301 INFO:tasks.rados.rados.0.smithi084.stdout:2594: done (1 left) 2024-09-20T22:52:21.301 INFO:tasks.rados.rados.0.smithi084.stdout:2595: delete oid 12 current snap is 293 2024-09-20T22:52:21.302 INFO:tasks.rados.rados.0.smithi084.stdout:2593: finishing write tid 4 to smithi08477203-10 2024-09-20T22:52:21.302 INFO:tasks.rados.rados.0.smithi084.stdout:2593: finishing write tid 5 to smithi08477203-10 2024-09-20T22:52:21.302 INFO:tasks.rados.rados.0.smithi084.stdout:2593: oid 10 updating version 0 to 1806 2024-09-20T22:52:21.302 INFO:tasks.rados.rados.0.smithi084.stdout:2593: oid 10 version 1806 is already newer than 1804 2024-09-20T22:52:21.302 INFO:tasks.rados.rados.0.smithi084.stdout:2593: oid 10 version 1806 is already newer than 1803 2024-09-20T22:52:21.302 INFO:tasks.rados.rados.0.smithi084.stdout:2593: oid 10 version 1806 is already newer than 1805 2024-09-20T22:52:21.302 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 10 v 1806 (ObjNum 895 snap 293 seq_num 895) dirty exists 2024-09-20T22:52:21.302 INFO:tasks.rados.rados.0.smithi084.stdout:2593: left oid 10 (ObjNum 895 snap 293 seq_num 895) 2024-09-20T22:52:21.303 INFO:tasks.rados.rados.0.smithi084.stdout:2593: done (1 left) 2024-09-20T22:52:21.303 INFO:tasks.rados.rados.0.smithi084.stdout:2595: done (0 left) 2024-09-20T22:52:21.303 INFO:tasks.rados.rados.0.smithi084.stdout:2596: snap_create 2024-09-20T22:52:21.346 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:52:21.346 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-20T22:52:21.346 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2024-09-20T22:52:21.346 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2024-09-20T22:52:21.346 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-20T22:52:21.386 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:52:21.408 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:21.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:21.408 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:52:21.509 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:52:21.531 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:21.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:21.532 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:52:21.614 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:21 smithi084 ceph-mon[40024]: pgmap v891: 158 pgs: 4 active+clean+snaptrim, 1 active+remapped+backfill_toofull, 153 active+clean; 561 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 9.1 MiB/s rd, 280 KiB/s wr, 11 op/s; 24/1410 objects misplaced (1.702%) 2024-09-20T22:52:21.614 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:21 smithi084 ceph-mon[40024]: 3.13 scrub starts 2024-09-20T22:52:21.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:21 smithi084 ceph-mon[40024]: 3.13 scrub ok 2024-09-20T22:52:21.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:21 smithi084 ceph-mon[40024]: 5.11 scrub starts 2024-09-20T22:52:21.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:21 smithi084 ceph-mon[40024]: 5.11 scrub ok 2024-09-20T22:52:21.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:21 smithi084 ceph-mon[40024]: osdmap e661: 8 total, 8 up, 7 in 2024-09-20T22:52:21.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:21 smithi084 ceph-mon[40024]: 6.6 scrub starts 2024-09-20T22:52:21.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:21 smithi084 ceph-mon[40024]: 6.6 scrub ok 2024-09-20T22:52:21.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:21 smithi084 ceph-mon[29774]: pgmap v891: 158 pgs: 4 active+clean+snaptrim, 1 active+remapped+backfill_toofull, 153 active+clean; 561 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 9.1 MiB/s rd, 280 KiB/s wr, 11 op/s; 24/1410 objects misplaced (1.702%) 2024-09-20T22:52:21.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:21 smithi084 ceph-mon[29774]: 3.13 scrub starts 2024-09-20T22:52:21.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:21 smithi084 ceph-mon[29774]: 3.13 scrub ok 2024-09-20T22:52:21.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:21 smithi084 ceph-mon[29774]: 5.11 scrub starts 2024-09-20T22:52:21.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:21 smithi084 ceph-mon[29774]: 5.11 scrub ok 2024-09-20T22:52:21.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:21 smithi084 ceph-mon[29774]: osdmap e661: 8 total, 8 up, 7 in 2024-09-20T22:52:21.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:21 smithi084 ceph-mon[29774]: 6.6 scrub starts 2024-09-20T22:52:21.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:21 smithi084 ceph-mon[29774]: 6.6 scrub ok 2024-09-20T22:52:21.633 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:52:21.655 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:21.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:21.656 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:52:21.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:21 smithi154 ceph-mon[33264]: pgmap v891: 158 pgs: 4 active+clean+snaptrim, 1 active+remapped+backfill_toofull, 153 active+clean; 561 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 9.1 MiB/s rd, 280 KiB/s wr, 11 op/s; 24/1410 objects misplaced (1.702%) 2024-09-20T22:52:21.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:21 smithi154 ceph-mon[33264]: 3.13 scrub starts 2024-09-20T22:52:21.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:21 smithi154 ceph-mon[33264]: 3.13 scrub ok 2024-09-20T22:52:21.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:21 smithi154 ceph-mon[33264]: 5.11 scrub starts 2024-09-20T22:52:21.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:21 smithi154 ceph-mon[33264]: 5.11 scrub ok 2024-09-20T22:52:21.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:21 smithi154 ceph-mon[33264]: osdmap e661: 8 total, 8 up, 7 in 2024-09-20T22:52:21.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:21 smithi154 ceph-mon[33264]: 6.6 scrub starts 2024-09-20T22:52:21.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:21 smithi154 ceph-mon[33264]: 6.6 scrub ok 2024-09-20T22:52:21.757 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:52:21.778 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:21.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:21.779 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:52:21.879 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:52:21.900 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:21.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:21.901 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:52:22.002 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:52:22.023 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:22.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:22.024 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:52:22.032 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.0 dump_blocked_ops 2024-09-20T22:52:22.047 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-20T22:52:22.109 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:52:22.125 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:52:22.147 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:22.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:22.147 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:52:22.248 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:52:22.269 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:22.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:22.270 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:52:22.284 INFO:tasks.rados.rados.0.smithi084.stdout:2596: done (0 left) 2024-09-20T22:52:22.285 INFO:tasks.rados.rados.0.smithi084.stdout:2597: snap_create 2024-09-20T22:52:22.370 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:52:22.392 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:22.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:22.393 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:52:22.493 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:52:22.516 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:22.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:22.517 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:52:22.617 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:52:22.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:22.640 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:22.640 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:52:22.718 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:52:22.723 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:52:22.741 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:52:22.762 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:22.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:22.763 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:52:22.863 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:52:22.885 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:22.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:22.886 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:52:22.986 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:52:23.008 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:23.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:23.009 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:52:23.110 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:52:23.132 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:23.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:23.133 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:52:23.233 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:52:23.257 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:23.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:23.258 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:52:23.284 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is set 2024-09-20T22:52:23.285 INFO:tasks.rados.rados.0.smithi084.stdout:2597: done (0 left) 2024-09-20T22:52:23.285 INFO:tasks.rados.rados.0.smithi084.stdout:2598: write oid 27 current snap is 295 2024-09-20T22:52:23.286 INFO:tasks.rados.rados.0.smithi084.stdout:2598: seq_num 898 ranges {692686=503256,1848830=425967,2910671=182015} 2024-09-20T22:52:23.292 INFO:tasks.rados.rados.0.smithi084.stdout:2598: writing smithi08477203-27 from 692686 to 1195942 tid 1 2024-09-20T22:52:23.298 INFO:tasks.rados.rados.0.smithi084.stdout:2598: writing smithi08477203-27 from 1848830 to 2274797 tid 2 2024-09-20T22:52:23.302 INFO:tasks.rados.rados.0.smithi084.stdout:2598: writing smithi08477203-27 from 2910671 to 3092686 tid 3 2024-09-20T22:52:23.303 INFO:tasks.rados.rados.0.smithi084.stdout:2599: snap_remove snap 243 2024-09-20T22:52:23.346 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:23 smithi084 ceph-mon[29774]: 3.1f scrub starts 2024-09-20T22:52:23.346 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:23 smithi084 ceph-mon[29774]: 3.1f scrub ok 2024-09-20T22:52:23.346 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:23 smithi084 ceph-mon[29774]: pgmap v894: 158 pgs: 4 active+clean+snaptrim, 1 active+remapped+backfill_toofull, 153 active+clean; 561 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 9.1 MiB/s rd, 280 KiB/s wr, 11 op/s; 24/1410 objects misplaced (1.702%) 2024-09-20T22:52:23.346 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:23 smithi084 ceph-mon[29774]: 5.1f deep-scrub starts 2024-09-20T22:52:23.346 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:23 smithi084 ceph-mon[29774]: 5.1f deep-scrub ok 2024-09-20T22:52:23.346 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:23 smithi084 ceph-mon[29774]: osdmap e662: 8 total, 8 up, 7 in 2024-09-20T22:52:23.346 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:23 smithi084 ceph-mon[29774]: 4.c scrub starts 2024-09-20T22:52:23.346 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:23 smithi084 ceph-mon[29774]: 4.c scrub ok 2024-09-20T22:52:23.346 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:23 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/2474972449' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-20T22:52:23.346 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:23 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-20T22:52:23.346 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:23 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:52:23.346 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:23 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:52:23.347 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:23 smithi084 ceph-mon[40024]: 3.1f scrub starts 2024-09-20T22:52:23.347 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:23 smithi084 ceph-mon[40024]: 3.1f scrub ok 2024-09-20T22:52:23.347 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:23 smithi084 ceph-mon[40024]: pgmap v894: 158 pgs: 4 active+clean+snaptrim, 1 active+remapped+backfill_toofull, 153 active+clean; 561 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 9.1 MiB/s rd, 280 KiB/s wr, 11 op/s; 24/1410 objects misplaced (1.702%) 2024-09-20T22:52:23.347 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:23 smithi084 ceph-mon[40024]: 5.1f deep-scrub starts 2024-09-20T22:52:23.347 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:23 smithi084 ceph-mon[40024]: 5.1f deep-scrub ok 2024-09-20T22:52:23.347 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:23 smithi084 ceph-mon[40024]: osdmap e662: 8 total, 8 up, 7 in 2024-09-20T22:52:23.347 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:23 smithi084 ceph-mon[40024]: 4.c scrub starts 2024-09-20T22:52:23.347 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:23 smithi084 ceph-mon[40024]: 4.c scrub ok 2024-09-20T22:52:23.347 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:23 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/2474972449' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-20T22:52:23.347 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:23 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-20T22:52:23.347 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:23 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:52:23.347 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:23 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:52:23.359 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:52:23.380 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:23.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:23.381 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:52:23.444 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:52:23.445 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:52:23.461 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:52:23.461 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:52:23.477 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:52:23.477 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:52:23.481 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:52:23.493 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-20T22:52:23.493 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T22:52:23.502 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:23.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:23.503 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:52:23.509 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:52:23.509 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:52:23.525 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:52:23.525 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:52:23.542 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:52:23.542 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:52:23.558 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:52:23.558 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:52:23.604 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:52:23.625 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:23.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:23.626 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:52:23.726 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:52:23.748 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:23.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:23.749 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:52:23.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:23 smithi154 ceph-mon[33264]: 3.1f scrub starts 2024-09-20T22:52:23.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:23 smithi154 ceph-mon[33264]: 3.1f scrub ok 2024-09-20T22:52:23.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:23 smithi154 ceph-mon[33264]: pgmap v894: 158 pgs: 4 active+clean+snaptrim, 1 active+remapped+backfill_toofull, 153 active+clean; 561 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 9.1 MiB/s rd, 280 KiB/s wr, 11 op/s; 24/1410 objects misplaced (1.702%) 2024-09-20T22:52:23.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:23 smithi154 ceph-mon[33264]: 5.1f deep-scrub starts 2024-09-20T22:52:23.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:23 smithi154 ceph-mon[33264]: 5.1f deep-scrub ok 2024-09-20T22:52:23.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:23 smithi154 ceph-mon[33264]: osdmap e662: 8 total, 8 up, 7 in 2024-09-20T22:52:23.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:23 smithi154 ceph-mon[33264]: 4.c scrub starts 2024-09-20T22:52:23.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:23 smithi154 ceph-mon[33264]: 4.c scrub ok 2024-09-20T22:52:23.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:23 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/2474972449' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-20T22:52:23.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:23 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-20T22:52:23.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:23 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:52:23.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:23 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:52:23.849 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:52:23.870 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:23.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:23.871 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:52:23.972 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:52:23.993 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:23.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:23.994 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:52:24.095 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:52:24.116 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:24.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:24.117 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:52:24.218 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:52:24.239 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:24.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:24.240 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:52:24.292 INFO:tasks.rados.rados.0.smithi084.stdout:2599: done (1 left) 2024-09-20T22:52:24.292 INFO:tasks.rados.rados.0.smithi084.stdout:2600: write (excl) oid 11 current snap is 295 2024-09-20T22:52:24.292 INFO:tasks.rados.rados.0.smithi084.stdout:2598: finishing write tid 1 to smithi08477203-27 2024-09-20T22:52:24.292 INFO:tasks.rados.rados.0.smithi084.stdout:2598: finishing write tid 2 to smithi08477203-27 2024-09-20T22:52:24.292 INFO:tasks.rados.rados.0.smithi084.stdout:2598: finishing write tid 3 to smithi08477203-27 2024-09-20T22:52:24.292 INFO:tasks.rados.rados.0.smithi084.stdout:2598: finishing write tid 4 to smithi08477203-27 2024-09-20T22:52:24.292 INFO:tasks.rados.rados.0.smithi084.stdout:2598: finishing write tid 5 to smithi08477203-27 2024-09-20T22:52:24.293 INFO:tasks.rados.rados.0.smithi084.stdout:2598: oid 27 updating version 0 to 742 2024-09-20T22:52:24.293 INFO:tasks.rados.rados.0.smithi084.stdout:2598: oid 27 updating version 742 to 744 2024-09-20T22:52:24.293 INFO:tasks.rados.rados.0.smithi084.stdout:2598: oid 27 version 744 is already newer than 743 2024-09-20T22:52:24.293 INFO:tasks.rados.rados.0.smithi084.stdout:2598: oid 27 updating version 744 to 745 2024-09-20T22:52:24.293 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 27 v 745 (ObjNum 898 snap 295 seq_num 898) dirty exists 2024-09-20T22:52:24.293 INFO:tasks.rados.rados.0.smithi084.stdout:2598: left oid 27 (ObjNum 898 snap 295 seq_num 898) 2024-09-20T22:52:24.293 INFO:tasks.rados.rados.0.smithi084.stdout:2600: seq_num 899 ranges {709493=578505,1907312=685978} 2024-09-20T22:52:24.300 INFO:tasks.rados.rados.0.smithi084.stdout:2600: writing smithi08477203-11 from 709493 to 1287998 tid 1 2024-09-20T22:52:24.308 INFO:tasks.rados.rados.0.smithi084.stdout:2600: writing smithi08477203-11 from 1907312 to 2593290 tid 2 2024-09-20T22:52:24.308 INFO:tasks.rados.rados.0.smithi084.stdout:2598: done (1 left) 2024-09-20T22:52:24.308 INFO:tasks.rados.rados.0.smithi084.stdout:2601: write oid 26 current snap is 295 2024-09-20T22:52:24.308 INFO:tasks.rados.rados.0.smithi084.stdout:2601: seq_num 900 ranges {726300=653754,1965794=429636,2965517=160783} 2024-09-20T22:52:24.315 INFO:tasks.rados.rados.0.smithi084.stdout:2601: writing smithi08477203-26 from 726300 to 1380054 tid 1 2024-09-20T22:52:24.321 INFO:tasks.rados.rados.0.smithi084.stdout:2601: writing smithi08477203-26 from 1965794 to 2395430 tid 2 2024-09-20T22:52:24.326 INFO:tasks.rados.rados.0.smithi084.stdout:2601: writing smithi08477203-26 from 2965517 to 3126300 tid 3 2024-09-20T22:52:24.326 INFO:tasks.rados.rados.0.smithi084.stdout:2602: write oid 31 current snap is 295 2024-09-20T22:52:24.326 INFO:tasks.rados.rados.0.smithi084.stdout:2600: finishing write tid 1 to smithi08477203-11 2024-09-20T22:52:24.326 INFO:tasks.rados.rados.0.smithi084.stdout:2600: finishing write tid 2 to smithi08477203-11 2024-09-20T22:52:24.326 INFO:tasks.rados.rados.0.smithi084.stdout:2600: finishing write tid 3 to smithi08477203-11 2024-09-20T22:52:24.326 INFO:tasks.rados.rados.0.smithi084.stdout:2602: seq_num 901 ranges {743107=729003,1907923=689647} 2024-09-20T22:52:24.333 INFO:tasks.rados.rados.0.smithi084.stdout:2602: writing smithi08477203-31 from 743107 to 1472110 tid 1 2024-09-20T22:52:24.340 INFO:tasks.rados.rados.0.smithi084.stdout:2602: writing smithi08477203-31 from 1907923 to 2597570 tid 2 2024-09-20T22:52:24.340 INFO:tasks.rados.rados.0.smithi084.stdout:2603: copy_from oid 19 from oid 43 current snap is 295 2024-09-20T22:52:24.340 INFO:tasks.rados.rados.0.smithi084.stdout:2600: finishing write tid 4 to smithi08477203-11 2024-09-20T22:52:24.340 INFO:tasks.rados.rados.0.smithi084.stdout:2600: oid 11 updating version 0 to 770 2024-09-20T22:52:24.340 INFO:tasks.rados.rados.0.smithi084.stdout:2600: oid 11 version 770 is already newer than 769 2024-09-20T22:52:24.340 INFO:tasks.rados.rados.0.smithi084.stdout:2600: oid 11 version 770 is already newer than 768 2024-09-20T22:52:24.340 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 11 v 770 (ObjNum 899 snap 295 seq_num 899) dirty exists 2024-09-20T22:52:24.341 INFO:tasks.rados.rados.0.smithi084.stdout:2600: left oid 11 (ObjNum 899 snap 295 seq_num 899) 2024-09-20T22:52:24.341 INFO:tasks.rados.rados.0.smithi084.stdout:2601: finishing write tid 1 to smithi08477203-26 2024-09-20T22:52:24.341 INFO:tasks.rados.rados.0.smithi084.stdout:2600: done (3 left) 2024-09-20T22:52:24.341 INFO:tasks.rados.rados.0.smithi084.stdout:2604: snap_remove snap 289 2024-09-20T22:52:24.341 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:52:24.363 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:24.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:24.364 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:52:24.364 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:24 smithi084 ceph-mon[29774]: 7.b scrub starts 2024-09-20T22:52:24.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:24 smithi084 ceph-mon[29774]: 7.b scrub ok 2024-09-20T22:52:24.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:24 smithi084 ceph-mon[29774]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:52:24.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:24 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T22:52:24.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:24 smithi084 ceph-mon[29774]: osdmap e663: 8 total, 8 up, 7 in 2024-09-20T22:52:24.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:24 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/2033148074' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:52:24.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:24 smithi084 ceph-mon[40024]: 7.b scrub starts 2024-09-20T22:52:24.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:24 smithi084 ceph-mon[40024]: 7.b scrub ok 2024-09-20T22:52:24.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:24 smithi084 ceph-mon[40024]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:52:24.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:24 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T22:52:24.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:24 smithi084 ceph-mon[40024]: osdmap e663: 8 total, 8 up, 7 in 2024-09-20T22:52:24.365 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:24 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/2033148074' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:52:24.465 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:52:24.486 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:24.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:24.487 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:52:24.588 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:52:24.608 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:24.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:24.609 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:52:24.710 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:52:24.731 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:24.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:24.732 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:52:24.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:24 smithi154 ceph-mon[33264]: 7.b scrub starts 2024-09-20T22:52:24.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:24 smithi154 ceph-mon[33264]: 7.b scrub ok 2024-09-20T22:52:24.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:24 smithi154 ceph-mon[33264]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:52:24.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:24 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T22:52:24.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:24 smithi154 ceph-mon[33264]: osdmap e663: 8 total, 8 up, 7 in 2024-09-20T22:52:24.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:24 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/2033148074' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:52:24.833 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:52:24.839 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-20T22:52:24.841 INFO:teuthology.orchestra.run.smithi084.stderr:dumped all 2024-09-20T22:52:24.854 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:24.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:24.855 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:52:24.955 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:52:24.976 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:24.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:24.976 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:52:25.077 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:52:25.098 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:25.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:25.099 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:52:25.200 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:52:25.222 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:25.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:25.223 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:52:25.295 INFO:tasks.rados.rados.0.smithi084.stdout:2604: done (3 left) 2024-09-20T22:52:25.296 INFO:tasks.rados.rados.0.smithi084.stdout:2605: snap_create 2024-09-20T22:52:25.296 INFO:tasks.rados.rados.0.smithi084.stdout:2601: finishing write tid 2 to smithi08477203-26 2024-09-20T22:52:25.296 INFO:tasks.rados.rados.0.smithi084.stdout:2601: finishing write tid 3 to smithi08477203-26 2024-09-20T22:52:25.296 INFO:tasks.rados.rados.0.smithi084.stdout:2601: finishing write tid 4 to smithi08477203-26 2024-09-20T22:52:25.296 INFO:tasks.rados.rados.0.smithi084.stdout:2601: finishing write tid 5 to smithi08477203-26 2024-09-20T22:52:25.296 INFO:tasks.rados.rados.0.smithi084.stdout:2601: oid 26 updating version 0 to 1344 2024-09-20T22:52:25.296 INFO:tasks.rados.rados.0.smithi084.stdout:2601: oid 26 updating version 1344 to 1346 2024-09-20T22:52:25.296 INFO:tasks.rados.rados.0.smithi084.stdout:2601: oid 26 version 1346 is already newer than 1345 2024-09-20T22:52:25.296 INFO:tasks.rados.rados.0.smithi084.stdout:2601: oid 26 updating version 1346 to 1347 2024-09-20T22:52:25.296 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 26 v 1347 (ObjNum 900 snap 295 seq_num 900) dirty exists 2024-09-20T22:52:25.296 INFO:tasks.rados.rados.0.smithi084.stdout:2601: left oid 26 (ObjNum 900 snap 295 seq_num 900) 2024-09-20T22:52:25.296 INFO:tasks.rados.rados.0.smithi084.stdout:2602: finishing write tid 1 to smithi08477203-31 2024-09-20T22:52:25.297 INFO:tasks.rados.rados.0.smithi084.stdout:2602: finishing write tid 2 to smithi08477203-31 2024-09-20T22:52:25.297 INFO:tasks.rados.rados.0.smithi084.stdout:2602: finishing write tid 3 to smithi08477203-31 2024-09-20T22:52:25.297 INFO:tasks.rados.rados.0.smithi084.stdout:2602: finishing write tid 4 to smithi08477203-31 2024-09-20T22:52:25.297 INFO:tasks.rados.rados.0.smithi084.stdout:2602: oid 31 updating version 0 to 1349 2024-09-20T22:52:25.297 INFO:tasks.rados.rados.0.smithi084.stdout:2602: oid 31 updating version 1349 to 1350 2024-09-20T22:52:25.297 INFO:tasks.rados.rados.0.smithi084.stdout:2602: oid 31 version 1350 is already newer than 1348 2024-09-20T22:52:25.297 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 31 v 1350 (ObjNum 901 snap 295 seq_num 901) dirty exists 2024-09-20T22:52:25.297 INFO:tasks.rados.rados.0.smithi084.stdout:2602: left oid 31 (ObjNum 901 snap 295 seq_num 901) 2024-09-20T22:52:25.297 INFO:tasks.rados.rados.0.smithi084.stdout:2603: finishing copy_from racing read to smithi08477203-19 2024-09-20T22:52:25.297 INFO:tasks.rados.rados.0.smithi084.stdout:2603: finishing copy_from to smithi08477203-19 2024-09-20T22:52:25.297 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 19 v 846 (ObjNum 625 snap 186 seq_num 625) dirty exists 2024-09-20T22:52:25.324 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:52:25.344 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:25.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:25.345 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:52:25.349 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-20T22:52:25.349 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:52:25.447 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:52:25.468 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:25.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:25.469 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:52:25.570 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:52:25.590 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:25.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:25.591 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:52:25.614 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:25 smithi084 ceph-mon[40024]: pgmap v897: 158 pgs: 1 active+clean+snaptrim, 157 active+clean; 565 MiB data, 1.6 GiB used, 624 GiB / 626 GiB avail; 1.7 KiB/s rd, 627 KiB/s wr, 5 op/s; 13 MiB/s, 5 objects/s recovering 2024-09-20T22:52:25.614 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:25 smithi084 ceph-mon[40024]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-09-20T22:52:25.614 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:25 smithi084 ceph-mon[40024]: osdmap e664: 8 total, 8 up, 7 in 2024-09-20T22:52:25.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:25 smithi084 ceph-mon[29774]: pgmap v897: 158 pgs: 1 active+clean+snaptrim, 157 active+clean; 565 MiB data, 1.6 GiB used, 624 GiB / 626 GiB avail; 1.7 KiB/s rd, 627 KiB/s wr, 5 op/s; 13 MiB/s, 5 objects/s recovering 2024-09-20T22:52:25.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:25 smithi084 ceph-mon[29774]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-09-20T22:52:25.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:25 smithi084 ceph-mon[29774]: osdmap e664: 8 total, 8 up, 7 in 2024-09-20T22:52:25.692 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:52:25.716 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:25.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:25.716 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:52:25.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:25 smithi154 ceph-mon[33264]: pgmap v897: 158 pgs: 1 active+clean+snaptrim, 157 active+clean; 565 MiB data, 1.6 GiB used, 624 GiB / 626 GiB avail; 1.7 KiB/s rd, 627 KiB/s wr, 5 op/s; 13 MiB/s, 5 objects/s recovering 2024-09-20T22:52:25.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:25 smithi154 ceph-mon[33264]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-09-20T22:52:25.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:25 smithi154 ceph-mon[33264]: osdmap e664: 8 total, 8 up, 7 in 2024-09-20T22:52:25.818 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:52:25.841 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:25.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:25.841 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:52:25.942 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:52:25.964 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:25.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:25.965 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:52:26.063 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.0 dump_historic_ops 2024-09-20T22:52:26.065 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:52:26.085 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:26.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:26.086 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:52:26.187 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:52:26.211 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:26.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:26.212 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:52:26.298 INFO:tasks.rados.rados.0.smithi084.stdout:2601: done (3 left) 2024-09-20T22:52:26.298 INFO:tasks.rados.rados.0.smithi084.stdout:2602: done (2 left) 2024-09-20T22:52:26.298 INFO:tasks.rados.rados.0.smithi084.stdout:2603: done (1 left) 2024-09-20T22:52:26.298 INFO:tasks.rados.rados.0.smithi084.stdout:2605: done (0 left) 2024-09-20T22:52:26.298 INFO:tasks.rados.rados.0.smithi084.stdout:2606: snap_remove snap 283 2024-09-20T22:52:26.312 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:52:26.333 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:26.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:26.334 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:52:26.435 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:52:26.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:26.459 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:26.459 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:52:26.478 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-20T22:52:26.561 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:52:26.582 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:26.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:26.584 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:52:26.614 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:26 smithi084 ceph-mon[40024]: from='client.26528 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T22:52:26.614 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:26 smithi084 ceph-mon[40024]: osdmap e665: 8 total, 8 up, 7 in 2024-09-20T22:52:26.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:26 smithi084 ceph-mon[29774]: from='client.26528 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T22:52:26.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:26 smithi084 ceph-mon[29774]: osdmap e665: 8 total, 8 up, 7 in 2024-09-20T22:52:26.684 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:52:26.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:26.706 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:26.706 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:52:26.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:26 smithi154 ceph-mon[33264]: from='client.26528 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T22:52:26.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:26 smithi154 ceph-mon[33264]: osdmap e665: 8 total, 8 up, 7 in 2024-09-20T22:52:26.807 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:52:26.827 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:26.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:26.828 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:52:26.929 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:52:26.954 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:26.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:26.955 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:52:27.055 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:52:27.079 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:27.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:27.080 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:52:27.154 INFO:teuthology.orchestra.run.smithi084.stdout:{"pg_ready":true,"pg_map":{"version":898,"stamp":"2024-09-20T22:52:24.290955+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592572693,"num_objects":474,"num_object_clones":225,"num_object_copies":1422,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":474,"num_whiteouts":14,"num_read":6295,"num_read_kb":4754958,"num_write":3953,"num_write_kb":1199683,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1142,"num_bytes_recovered":520180548,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9733,"ondisk_log_size":9733,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":474,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":1656848,"kb_used_data":1317744,"kb_used_omap":10,"kb_used_meta":338933,"kb_avail":654559216,"statfs":{"total":671965249536,"available":670268637184,"internally_reserved":0,"allocated":1349369856,"data_stored":1906643874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11157,"internal_metadata":347067499},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":4677382,"num_objects":4,"num_object_clones":4,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-24,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":2,"num_read":8,"num_read_kb":7,"num_write":13,"num_write_kb":2507,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":56524481,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.017300"},"pg_stats":[{"pgid":"3.1f","version":"86'2","reported_seq":581,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-09-20T22:52:23.283826+0000","last_change":"2024-09-20T22:52:21.630419+0000","last_active":"2024-09-20T22:52:23.283826+0000","last_peered":"2024-09-20T22:52:23.283826+0000","last_clean":"2024-09-20T22:52:23.283826+0000","last_became_active":"2024-09-20T22:47:49.222114+0000","last_became_peered":"2024-09-20T22:47:49.222114+0000","last_unstale":"2024-09-20T22:52:23.283826+0000","last_undegraded":"2024-09-20T22:52:23.283826+0000","last_fullsized":"2024-09-20T22:52:23.283826+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'2","last_scrub_stamp":"2024-09-20T22:52:21.630304+0000","last_deep_scrub":"86'2","last_deep_scrub_stamp":"2024-09-20T22:42:39.664421+0000","last_clean_scrub_stamp":"2024-09-20T22:52:21.630304+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:38.958732+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":436,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"625'15","reported_seq":579,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-09-20T22:52:23.281499+0000","last_change":"2024-09-20T22:51:51.813358+0000","last_active":"2024-09-20T22:52:23.281499+0000","last_peered":"2024-09-20T22:52:23.281499+0000","last_clean":"2024-09-20T22:52:23.281499+0000","last_became_active":"2024-09-20T22:51:20.286651+0000","last_became_peered":"2024-09-20T22:51:20.286651+0000","last_unstale":"2024-09-20T22:52:23.281499+0000","last_undegraded":"2024-09-20T22:52:23.281499+0000","last_fullsized":"2024-09-20T22:52:23.281499+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'15","last_scrub_stamp":"2024-09-20T22:51:51.813269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:51:51.813269+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:09.354334+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":138,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-09-20T22:52:23.630691+0000","last_change":"2024-09-20T22:51:20.291625+0000","last_active":"2024-09-20T22:52:23.630691+0000","last_peered":"2024-09-20T22:52:23.630691+0000","last_clean":"2024-09-20T22:52:23.630691+0000","last_became_active":"2024-09-20T22:51:20.291272+0000","last_became_peered":"2024-09-20T22:51:20.291272+0000","last_unstale":"2024-09-20T22:52:23.630691+0000","last_undegraded":"2024-09-20T22:52:23.630691+0000","last_fullsized":"2024-09-20T22:52:23.630691+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:50:12.718677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:50:12.718677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":550,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-09-20T22:52:23.281977+0000","last_change":"2024-09-20T22:51:36.730087+0000","last_active":"2024-09-20T22:52:23.281977+0000","last_peered":"2024-09-20T22:52:23.281977+0000","last_clean":"2024-09-20T22:52:23.281977+0000","last_became_active":"2024-09-20T22:51:20.295608+0000","last_became_peered":"2024-09-20T22:51:20.295608+0000","last_unstale":"2024-09-20T22:52:23.281977+0000","last_undegraded":"2024-09-20T22:52:23.281977+0000","last_fullsized":"2024-09-20T22:52:23.281977+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:36.730015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:51:36.730015+0000","last_clean_scrub_stamp":"2024-09-20T22:51:36.730015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:41.980409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":1224,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-09-20T22:52:23.281735+0000","last_change":"2024-09-20T22:51:01.404127+0000","last_active":"2024-09-20T22:52:23.281735+0000","last_peered":"2024-09-20T22:52:23.281735+0000","last_clean":"2024-09-20T22:52:23.281735+0000","last_became_active":"2024-09-20T22:41:44.832099+0000","last_became_peered":"2024-09-20T22:41:44.832099+0000","last_unstale":"2024-09-20T22:52:23.281735+0000","last_undegraded":"2024-09-20T22:52:23.281735+0000","last_fullsized":"2024-09-20T22:52:23.281735+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:01.404063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:51:01.404063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":1229,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-09-20T22:52:23.283212+0000","last_change":"2024-09-20T22:51:12.418041+0000","last_active":"2024-09-20T22:52:23.283212+0000","last_peered":"2024-09-20T22:52:23.283212+0000","last_clean":"2024-09-20T22:52:23.283212+0000","last_became_active":"2024-09-20T22:41:38.157298+0000","last_became_peered":"2024-09-20T22:41:38.157298+0000","last_unstale":"2024-09-20T22:52:23.283212+0000","last_undegraded":"2024-09-20T22:52:23.283212+0000","last_fullsized":"2024-09-20T22:52:23.283212+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:12.417920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:51:12.417920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:37.156764+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"625'27","reported_seq":1311,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-09-20T22:52:23.281629+0000","last_change":"2024-09-20T22:51:08.441927+0000","last_active":"2024-09-20T22:52:23.281629+0000","last_peered":"2024-09-20T22:52:23.281629+0000","last_clean":"2024-09-20T22:52:23.281629+0000","last_became_active":"2024-09-20T22:47:49.218798+0000","last_became_peered":"2024-09-20T22:47:49.218798+0000","last_unstale":"2024-09-20T22:52:23.281629+0000","last_undegraded":"2024-09-20T22:52:23.281629+0000","last_fullsized":"2024-09-20T22:52:23.281629+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-09-20T22:51:08.441788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:51:08.441788+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:25.517854+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":550,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-09-20T22:52:23.281396+0000","last_change":"2024-09-20T22:52:14.594895+0000","last_active":"2024-09-20T22:52:23.281396+0000","last_peered":"2024-09-20T22:52:23.281396+0000","last_clean":"2024-09-20T22:52:23.281396+0000","last_became_active":"2024-09-20T22:51:20.287060+0000","last_became_peered":"2024-09-20T22:51:20.287060+0000","last_unstale":"2024-09-20T22:52:23.281396+0000","last_undegraded":"2024-09-20T22:52:23.281396+0000","last_fullsized":"2024-09-20T22:52:23.281396+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:14.594823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:52:14.594823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:30.113157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":1249,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-09-20T22:52:23.631299+0000","last_change":"2024-09-20T22:51:09.684125+0000","last_active":"2024-09-20T22:52:23.631299+0000","last_peered":"2024-09-20T22:52:23.631299+0000","last_clean":"2024-09-20T22:52:23.631299+0000","last_became_active":"2024-09-20T22:48:05.242059+0000","last_became_peered":"2024-09-20T22:48:05.242059+0000","last_unstale":"2024-09-20T22:52:23.631299+0000","last_undegraded":"2024-09-20T22:52:23.631299+0000","last_fullsized":"2024-09-20T22:52:23.631299+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:09.684053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:46:00.999939+0000","last_clean_scrub_stamp":"2024-09-20T22:51:09.684053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:35.986257+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"625'15","reported_seq":563,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-09-20T22:52:23.281765+0000","last_change":"2024-09-20T22:51:30.710329+0000","last_active":"2024-09-20T22:52:23.281765+0000","last_peered":"2024-09-20T22:52:23.281765+0000","last_clean":"2024-09-20T22:52:23.281765+0000","last_became_active":"2024-09-20T22:48:05.233429+0000","last_became_peered":"2024-09-20T22:48:05.233429+0000","last_unstale":"2024-09-20T22:52:23.281765+0000","last_undegraded":"2024-09-20T22:52:23.281765+0000","last_fullsized":"2024-09-20T22:52:23.281765+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-09-20T22:51:30.710245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:51:30.710245+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:39.082734+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":1241,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-09-20T22:52:23.631212+0000","last_change":"2024-09-20T22:51:33.977328+0000","last_active":"2024-09-20T22:52:23.631212+0000","last_peered":"2024-09-20T22:52:23.631212+0000","last_clean":"2024-09-20T22:52:23.631212+0000","last_became_active":"2024-09-20T22:47:49.227379+0000","last_became_peered":"2024-09-20T22:47:49.227379+0000","last_unstale":"2024-09-20T22:52:23.631212+0000","last_undegraded":"2024-09-20T22:52:23.631212+0000","last_fullsized":"2024-09-20T22:52:23.631212+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:33.977257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:51:33.977257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:37.924731+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":582,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-09-20T22:52:23.281263+0000","last_change":"2024-09-20T22:51:35.899942+0000","last_active":"2024-09-20T22:52:23.281263+0000","last_peered":"2024-09-20T22:52:23.281263+0000","last_clean":"2024-09-20T22:52:23.281263+0000","last_became_active":"2024-09-20T22:51:20.277465+0000","last_became_peered":"2024-09-20T22:51:20.277465+0000","last_unstale":"2024-09-20T22:52:23.281263+0000","last_undegraded":"2024-09-20T22:52:23.281263+0000","last_fullsized":"2024-09-20T22:52:23.281263+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:35.899860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:49:08.664407+0000","last_clean_scrub_stamp":"2024-09-20T22:51:35.899860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:38.548132+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":1255,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-09-20T22:52:23.630970+0000","last_change":"2024-09-20T22:51:20.298096+0000","last_active":"2024-09-20T22:52:23.630970+0000","last_peered":"2024-09-20T22:52:23.630970+0000","last_clean":"2024-09-20T22:52:23.630970+0000","last_became_active":"2024-09-20T22:51:20.297787+0000","last_became_peered":"2024-09-20T22:51:20.297787+0000","last_unstale":"2024-09-20T22:52:23.630970+0000","last_undegraded":"2024-09-20T22:52:23.630970+0000","last_fullsized":"2024-09-20T22:52:23.630970+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:50:17.735354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:50:17.735354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"625'9","reported_seq":565,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-09-20T22:52:23.282332+0000","last_change":"2024-09-20T22:51:20.301034+0000","last_active":"2024-09-20T22:52:23.282332+0000","last_peered":"2024-09-20T22:52:23.282332+0000","last_clean":"2024-09-20T22:52:23.282332+0000","last_became_active":"2024-09-20T22:51:20.285795+0000","last_became_peered":"2024-09-20T22:51:20.285795+0000","last_unstale":"2024-09-20T22:52:23.282332+0000","last_undegraded":"2024-09-20T22:52:23.282332+0000","last_fullsized":"2024-09-20T22:52:23.282332+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"181'7","last_scrub_stamp":"2024-09-20T22:51:05.899205+0000","last_deep_scrub":"181'7","last_deep_scrub_stamp":"2024-09-20T22:48:41.049381+0000","last_clean_scrub_stamp":"2024-09-20T22:51:05.899205+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":9,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":1269,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-09-20T22:52:23.284749+0000","last_change":"2024-09-20T22:52:16.705733+0000","last_active":"2024-09-20T22:52:23.284749+0000","last_peered":"2024-09-20T22:52:23.284749+0000","last_clean":"2024-09-20T22:52:23.284749+0000","last_became_active":"2024-09-20T22:51:20.285368+0000","last_became_peered":"2024-09-20T22:51:20.285368+0000","last_unstale":"2024-09-20T22:52:23.284749+0000","last_undegraded":"2024-09-20T22:52:23.284749+0000","last_fullsized":"2024-09-20T22:52:23.284749+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:16.705645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:47:40.361958+0000","last_clean_scrub_stamp":"2024-09-20T22:52:16.705645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:40.490856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":1234,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-09-20T22:52:23.630744+0000","last_change":"2024-09-20T22:51:20.279625+0000","last_active":"2024-09-20T22:52:23.630744+0000","last_peered":"2024-09-20T22:52:23.630744+0000","last_clean":"2024-09-20T22:52:23.630744+0000","last_became_active":"2024-09-20T22:51:20.279307+0000","last_became_peered":"2024-09-20T22:51:20.279307+0000","last_unstale":"2024-09-20T22:52:23.630744+0000","last_undegraded":"2024-09-20T22:52:23.630744+0000","last_fullsized":"2024-09-20T22:52:23.630744+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:04.636900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:51:04.636900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":137,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-09-20T22:52:23.282585+0000","last_change":"2024-09-20T22:51:20.282749+0000","last_active":"2024-09-20T22:52:23.282585+0000","last_peered":"2024-09-20T22:52:23.282585+0000","last_clean":"2024-09-20T22:52:23.282585+0000","last_became_active":"2024-09-20T22:51:20.282393+0000","last_became_peered":"2024-09-20T22:51:20.282393+0000","last_unstale":"2024-09-20T22:52:23.282585+0000","last_undegraded":"2024-09-20T22:52:23.282585+0000","last_fullsized":"2024-09-20T22:52:23.282585+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:50:16.816835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:50:16.816835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":574,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-09-20T22:52:23.282637+0000","last_change":"2024-09-20T22:49:53.933673+0000","last_active":"2024-09-20T22:52:23.282637+0000","last_peered":"2024-09-20T22:52:23.282637+0000","last_clean":"2024-09-20T22:52:23.282637+0000","last_became_active":"2024-09-20T22:48:05.233687+0000","last_became_peered":"2024-09-20T22:48:05.233687+0000","last_unstale":"2024-09-20T22:52:23.282637+0000","last_undegraded":"2024-09-20T22:52:23.282637+0000","last_fullsized":"2024-09-20T22:52:23.282637+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:53.933601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:49:53.933601+0000","last_clean_scrub_stamp":"2024-09-20T22:49:53.933601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"625'34","reported_seq":1320,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-09-20T22:52:23.285527+0000","last_change":"2024-09-20T22:51:20.289052+0000","last_active":"2024-09-20T22:52:23.285527+0000","last_peered":"2024-09-20T22:52:23.285527+0000","last_clean":"2024-09-20T22:52:23.285527+0000","last_became_active":"2024-09-20T22:51:20.276338+0000","last_became_peered":"2024-09-20T22:51:20.276338+0000","last_unstale":"2024-09-20T22:52:23.285527+0000","last_undegraded":"2024-09-20T22:52:23.285527+0000","last_fullsized":"2024-09-20T22:52:23.285527+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'22","last_scrub_stamp":"2024-09-20T22:51:08.280738+0000","last_deep_scrub":"113'22","last_deep_scrub_stamp":"2024-09-20T22:43:12.191878+0000","last_clean_scrub_stamp":"2024-09-20T22:51:08.280738+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:26.717275+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":849,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-09-20T22:52:23.282270+0000","last_change":"2024-09-20T22:51:20.295989+0000","last_active":"2024-09-20T22:52:23.282270+0000","last_peered":"2024-09-20T22:52:23.282270+0000","last_clean":"2024-09-20T22:52:23.282270+0000","last_became_active":"2024-09-20T22:51:20.295564+0000","last_became_peered":"2024-09-20T22:51:20.295564+0000","last_unstale":"2024-09-20T22:52:23.282270+0000","last_undegraded":"2024-09-20T22:52:23.282270+0000","last_fullsized":"2024-09-20T22:52:23.282270+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:50:40.784154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:50:40.784154+0000","last_clean_scrub_stamp":"2024-09-20T22:50:40.784154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":1217,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-09-20T22:52:23.283124+0000","last_change":"2024-09-20T22:51:13.385618+0000","last_active":"2024-09-20T22:52:23.283124+0000","last_peered":"2024-09-20T22:52:23.283124+0000","last_clean":"2024-09-20T22:52:23.283124+0000","last_became_active":"2024-09-20T22:41:44.832131+0000","last_became_peered":"2024-09-20T22:41:44.832131+0000","last_unstale":"2024-09-20T22:52:23.283124+0000","last_undegraded":"2024-09-20T22:52:23.283124+0000","last_fullsized":"2024-09-20T22:52:23.283124+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:13.385520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:45:25.002199+0000","last_clean_scrub_stamp":"2024-09-20T22:51:13.385520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":550,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-09-20T22:52:23.281982+0000","last_change":"2024-09-20T22:51:20.285019+0000","last_active":"2024-09-20T22:52:23.281982+0000","last_peered":"2024-09-20T22:52:23.281982+0000","last_clean":"2024-09-20T22:52:23.281982+0000","last_became_active":"2024-09-20T22:51:20.284621+0000","last_became_peered":"2024-09-20T22:51:20.284621+0000","last_unstale":"2024-09-20T22:52:23.281982+0000","last_undegraded":"2024-09-20T22:52:23.281982+0000","last_fullsized":"2024-09-20T22:52:23.281982+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:50:41.885844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:50:41.885844+0000","last_clean_scrub_stamp":"2024-09-20T22:50:41.885844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"625'18","reported_seq":1328,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-09-20T22:52:23.282844+0000","last_change":"2024-09-20T22:51:20.300828+0000","last_active":"2024-09-20T22:52:23.282844+0000","last_peered":"2024-09-20T22:52:23.282844+0000","last_clean":"2024-09-20T22:52:23.282844+0000","last_became_active":"2024-09-20T22:51:20.278886+0000","last_became_peered":"2024-09-20T22:51:20.278886+0000","last_unstale":"2024-09-20T22:52:23.282844+0000","last_undegraded":"2024-09-20T22:52:23.282844+0000","last_fullsized":"2024-09-20T22:52:23.282844+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-09-20T22:51:03.380934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:51:03.380934+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:30.819992+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":544,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-09-20T22:52:23.282406+0000","last_change":"2024-09-20T22:50:42.864153+0000","last_active":"2024-09-20T22:52:23.282406+0000","last_peered":"2024-09-20T22:52:23.282406+0000","last_clean":"2024-09-20T22:52:23.282406+0000","last_became_active":"2024-09-20T22:48:05.250319+0000","last_became_peered":"2024-09-20T22:48:05.250319+0000","last_unstale":"2024-09-20T22:52:23.282406+0000","last_undegraded":"2024-09-20T22:52:23.282406+0000","last_fullsized":"2024-09-20T22:52:23.282406+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:50:42.864078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:50:42.864078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"154'2","reported_seq":1226,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-09-20T22:52:23.284400+0000","last_change":"2024-09-20T22:50:34.375264+0000","last_active":"2024-09-20T22:52:23.284400+0000","last_peered":"2024-09-20T22:52:23.284400+0000","last_clean":"2024-09-20T22:52:23.284400+0000","last_became_active":"2024-09-20T22:41:44.199822+0000","last_became_peered":"2024-09-20T22:41:44.199822+0000","last_unstale":"2024-09-20T22:52:23.284400+0000","last_undegraded":"2024-09-20T22:52:23.284400+0000","last_fullsized":"2024-09-20T22:52:23.284400+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"154'2","last_scrub_stamp":"2024-09-20T22:50:34.375161+0000","last_deep_scrub":"154'2","last_deep_scrub_stamp":"2024-09-20T22:43:54.134066+0000","last_clean_scrub_stamp":"2024-09-20T22:50:34.375161+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"67'1","reported_seq":149,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-09-20T22:52:23.281485+0000","last_change":"2024-09-20T22:51:20.303293+0000","last_active":"2024-09-20T22:52:23.281485+0000","last_peered":"2024-09-20T22:52:23.281485+0000","last_clean":"2024-09-20T22:52:23.281485+0000","last_became_active":"2024-09-20T22:51:20.279978+0000","last_became_peered":"2024-09-20T22:51:20.279978+0000","last_unstale":"2024-09-20T22:52:23.281485+0000","last_undegraded":"2024-09-20T22:52:23.281485+0000","last_fullsized":"2024-09-20T22:52:23.281485+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-09-20T22:50:42.717953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:50:42.717953+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"625'25","reported_seq":604,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-09-20T22:52:23.283438+0000","last_change":"2024-09-20T22:51:02.707652+0000","last_active":"2024-09-20T22:52:23.283438+0000","last_peered":"2024-09-20T22:52:23.283438+0000","last_clean":"2024-09-20T22:52:23.283438+0000","last_became_active":"2024-09-20T22:47:49.232181+0000","last_became_peered":"2024-09-20T22:47:49.232181+0000","last_unstale":"2024-09-20T22:52:23.283438+0000","last_undegraded":"2024-09-20T22:52:23.283438+0000","last_fullsized":"2024-09-20T22:52:23.283438+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"151'15","last_scrub_stamp":"2024-09-20T22:51:02.707574+0000","last_deep_scrub":"151'15","last_deep_scrub_stamp":"2024-09-20T22:43:50.715761+0000","last_clean_scrub_stamp":"2024-09-20T22:51:02.707574+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"661'11","reported_seq":1410,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-09-20T22:52:23.294943+0000","last_change":"2024-09-20T22:52:21.908015+0000","last_active":"2024-09-20T22:52:23.294943+0000","last_peered":"2024-09-20T22:52:23.294943+0000","last_clean":"2024-09-20T22:52:23.294943+0000","last_became_active":"2024-09-20T22:48:05.244270+0000","last_became_peered":"2024-09-20T22:48:05.244270+0000","last_unstale":"2024-09-20T22:52:23.294943+0000","last_undegraded":"2024-09-20T22:52:23.294943+0000","last_fullsized":"2024-09-20T22:52:23.294943+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"661'11","last_scrub_stamp":"2024-09-20T22:52:21.907943+0000","last_deep_scrub":"661'11","last_deep_scrub_stamp":"2024-09-20T22:52:21.907943+0000","last_clean_scrub_stamp":"2024-09-20T22:52:21.907943+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:27.456801+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.c","version":"661'1979","reported_seq":4754,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-09-20T22:52:23.294693+0000","last_change":"2024-09-20T22:52:20.676049+0000","last_active":"2024-09-20T22:52:23.294693+0000","last_peered":"2024-09-20T22:52:23.294693+0000","last_clean":"2024-09-20T22:52:23.294693+0000","last_became_active":"2024-09-20T22:42:52.404027+0000","last_became_peered":"2024-09-20T22:42:52.404027+0000","last_unstale":"2024-09-20T22:52:23.294693+0000","last_undegraded":"2024-09-20T22:52:23.294693+0000","last_fullsized":"2024-09-20T22:52:23.294693+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"584'1653","last_scrub_stamp":"2024-09-20T22:51:04.993550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:51:04.993550+0000","objects_scrubbed":39,"log_size":1979,"log_dups_size":0,"ondisk_log_size":1979,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0.00302506,"stat_sum":{"num_bytes":90680996,"num_objects":42,"num_object_clones":35,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":2,"num_read":851,"num_read_kb":923855,"num_write":546,"num_write_kb":211218,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"a7","length":"1"},{"start":"c5","length":"1"},{"start":"ce","length":"1"},{"start":"d4","length":"1"},{"start":"da","length":"1"},{"start":"df","length":"1"},{"start":"e3","length":"a"},{"start":"ee","length":"7"},{"start":"f6","length":"2"},{"start":"f9","length":"1"},{"start":"fc","length":"1"},{"start":"fe","length":"5"},{"start":"104","length":"3"},{"start":"108","length":"4"},{"start":"10d","length":"1"},{"start":"10f","length":"1"},{"start":"111","length":"1"},{"start":"113","length":"1"},{"start":"116","length":"2"},{"start":"11a","length":"1"},{"start":"11c","length":"1"},{"start":"125","length":"1"}]},{"pgid":"1.a","version":"77'128","reported_seq":1497,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-09-20T22:52:23.630597+0000","last_change":"2024-09-20T22:50:19.676094+0000","last_active":"2024-09-20T22:52:23.630597+0000","last_peered":"2024-09-20T22:52:23.630597+0000","last_clean":"2024-09-20T22:52:23.630597+0000","last_became_active":"2024-09-20T22:42:52.407791+0000","last_became_peered":"2024-09-20T22:42:52.407791+0000","last_unstale":"2024-09-20T22:52:23.630597+0000","last_undegraded":"2024-09-20T22:52:23.630597+0000","last_fullsized":"2024-09-20T22:52:23.630597+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:50:19.675992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:50:19.675992+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"625'27","reported_seq":223,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-09-20T22:52:23.283237+0000","last_change":"2024-09-20T22:52:15.358218+0000","last_active":"2024-09-20T22:52:23.283237+0000","last_peered":"2024-09-20T22:52:23.283237+0000","last_clean":"2024-09-20T22:52:23.283237+0000","last_became_active":"2024-09-20T22:51:20.287083+0000","last_became_peered":"2024-09-20T22:51:20.287083+0000","last_unstale":"2024-09-20T22:52:23.283237+0000","last_undegraded":"2024-09-20T22:52:23.283237+0000","last_fullsized":"2024-09-20T22:52:23.283237+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'27","last_scrub_stamp":"2024-09-20T22:52:15.358100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:52:15.358100+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:27.239349+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":1234,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-09-20T22:52:23.280949+0000","last_change":"2024-09-20T22:51:29.394206+0000","last_active":"2024-09-20T22:52:23.280949+0000","last_peered":"2024-09-20T22:52:23.280949+0000","last_clean":"2024-09-20T22:52:23.280949+0000","last_became_active":"2024-09-20T22:51:20.279439+0000","last_became_peered":"2024-09-20T22:51:20.279439+0000","last_unstale":"2024-09-20T22:52:23.280949+0000","last_undegraded":"2024-09-20T22:52:23.280949+0000","last_fullsized":"2024-09-20T22:52:23.280949+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:29.394097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:49:12.070675+0000","last_clean_scrub_stamp":"2024-09-20T22:51:29.394097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:57.584257+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"408'12","reported_seq":600,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-09-20T22:52:23.282727+0000","last_change":"2024-09-20T22:51:50.785575+0000","last_active":"2024-09-20T22:52:23.282727+0000","last_peered":"2024-09-20T22:52:23.282727+0000","last_clean":"2024-09-20T22:52:23.282727+0000","last_became_active":"2024-09-20T22:48:05.242914+0000","last_became_peered":"2024-09-20T22:48:05.242914+0000","last_unstale":"2024-09-20T22:52:23.282727+0000","last_undegraded":"2024-09-20T22:52:23.282727+0000","last_fullsized":"2024-09-20T22:52:23.282727+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"408'12","last_scrub_stamp":"2024-09-20T22:51:50.785478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:51:50.785478+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:06.586894+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":1243,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-09-20T22:52:23.631017+0000","last_change":"2024-09-20T22:51:31.960415+0000","last_active":"2024-09-20T22:52:23.631017+0000","last_peered":"2024-09-20T22:52:23.631017+0000","last_clean":"2024-09-20T22:52:23.631017+0000","last_became_active":"2024-09-20T22:51:20.280145+0000","last_became_peered":"2024-09-20T22:51:20.280145+0000","last_unstale":"2024-09-20T22:52:23.631017+0000","last_undegraded":"2024-09-20T22:52:23.631017+0000","last_fullsized":"2024-09-20T22:52:23.631017+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:31.960331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:51:31.960331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:46.560339+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3","version":"657'268","reported_seq":2968,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-09-20T22:52:23.281620+0000","last_change":"2024-09-20T22:52:20.409140+0000","last_active":"2024-09-20T22:52:23.281620+0000","last_peered":"2024-09-20T22:52:23.281620+0000","last_clean":"2024-09-20T22:52:23.281620+0000","last_became_active":"2024-09-20T22:49:05.221402+0000","last_became_peered":"2024-09-20T22:49:05.221402+0000","last_unstale":"2024-09-20T22:52:23.281620+0000","last_undegraded":"2024-09-20T22:52:23.281620+0000","last_fullsized":"2024-09-20T22:52:23.281620+0000","mapping_epoch":404,"log_start":"456'129","ondisk_log_start":"456'129","created":94,"last_epoch_clean":465,"parent":"0.0","parent_split_bits":0,"last_scrub":"609'243","last_scrub_stamp":"2024-09-20T22:51:31.402013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:51:31.402013+0000","objects_scrubbed":6,"log_size":139,"log_dups_size":149,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:38.488791+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.00067579499999999998,"stat_sum":{"num_bytes":16392605,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":97,"num_read_kb":128299,"num_write":60,"num_write_kb":21007,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":20196357,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"a7","length":"1"},{"start":"c5","length":"1"},{"start":"ce","length":"1"},{"start":"d4","length":"1"},{"start":"da","length":"1"},{"start":"df","length":"1"},{"start":"e3","length":"a"},{"start":"ee","length":"7"},{"start":"f6","length":"2"},{"start":"f9","length":"1"},{"start":"fc","length":"1"},{"start":"fe","length":"5"},{"start":"104","length":"3"},{"start":"108","length":"4"},{"start":"10d","length":"1"},{"start":"10f","length":"1"},{"start":"111","length":"1"},{"start":"113","length":"1"},{"start":"116","length":"2"},{"start":"11a","length":"1"},{"start":"11c","length":"1"},{"start":"125","length":"1"}]},{"pgid":"1.5","version":"77'128","reported_seq":1404,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-09-20T22:52:23.282648+0000","last_change":"2024-09-20T22:51:12.927535+0000","last_active":"2024-09-20T22:52:23.282648+0000","last_peered":"2024-09-20T22:52:23.282648+0000","last_clean":"2024-09-20T22:52:23.282648+0000","last_became_active":"2024-09-20T22:48:08.207018+0000","last_became_peered":"2024-09-20T22:48:08.207018+0000","last_unstale":"2024-09-20T22:52:23.282648+0000","last_undegraded":"2024-09-20T22:52:23.282648+0000","last_fullsized":"2024-09-20T22:52:23.282648+0000","mapping_epoch":407,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":408,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:51:12.927463+0000","last_deep_scrub":"77'128","last_deep_scrub_stamp":"2024-09-20T22:45:41.608075+0000","last_clean_scrub_stamp":"2024-09-20T22:51:12.927463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:28.499830+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"625'30","reported_seq":1323,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-09-20T22:52:23.281192+0000","last_change":"2024-09-20T22:50:45.270518+0000","last_active":"2024-09-20T22:52:23.281192+0000","last_peered":"2024-09-20T22:52:23.281192+0000","last_clean":"2024-09-20T22:52:23.281192+0000","last_became_active":"2024-09-20T22:47:49.217605+0000","last_became_peered":"2024-09-20T22:47:49.217605+0000","last_unstale":"2024-09-20T22:52:23.281192+0000","last_undegraded":"2024-09-20T22:52:23.281192+0000","last_fullsized":"2024-09-20T22:52:23.281192+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'18","last_scrub_stamp":"2024-09-20T22:50:45.270399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:50:45.270399+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":1240,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-09-20T22:52:23.283396+0000","last_change":"2024-09-20T22:50:22.220104+0000","last_active":"2024-09-20T22:52:23.283396+0000","last_peered":"2024-09-20T22:52:23.283396+0000","last_clean":"2024-09-20T22:52:23.283396+0000","last_became_active":"2024-09-20T22:47:49.227069+0000","last_became_peered":"2024-09-20T22:47:49.227069+0000","last_unstale":"2024-09-20T22:52:23.283396+0000","last_undegraded":"2024-09-20T22:52:23.283396+0000","last_fullsized":"2024-09-20T22:52:23.283396+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:50:22.219983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:50:22.219983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":545,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-09-20T22:52:23.281877+0000","last_change":"2024-09-20T22:51:09.901939+0000","last_active":"2024-09-20T22:52:23.281877+0000","last_peered":"2024-09-20T22:52:23.281877+0000","last_clean":"2024-09-20T22:52:23.281877+0000","last_became_active":"2024-09-20T22:48:05.230003+0000","last_became_peered":"2024-09-20T22:48:05.230003+0000","last_unstale":"2024-09-20T22:52:23.281877+0000","last_undegraded":"2024-09-20T22:52:23.281877+0000","last_fullsized":"2024-09-20T22:52:23.281877+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:09.901838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:07.947945+0000","last_clean_scrub_stamp":"2024-09-20T22:51:09.901838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:27.938229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":545,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-09-20T22:52:23.281883+0000","last_change":"2024-09-20T22:51:55.748527+0000","last_active":"2024-09-20T22:52:23.281883+0000","last_peered":"2024-09-20T22:52:23.281883+0000","last_clean":"2024-09-20T22:52:23.281883+0000","last_became_active":"2024-09-20T22:48:05.239917+0000","last_became_peered":"2024-09-20T22:48:05.239917+0000","last_unstale":"2024-09-20T22:52:23.281883+0000","last_undegraded":"2024-09-20T22:52:23.281883+0000","last_fullsized":"2024-09-20T22:52:23.281883+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:55.748425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:53.036601+0000","last_clean_scrub_stamp":"2024-09-20T22:51:55.748425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:19.609464+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2","version":"663'675","reported_seq":2670,"reported_epoch":663,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T22:52:23.289191+0000","last_change":"2024-09-20T22:52:23.287751+0000","last_active":"2024-09-20T22:52:23.289191+0000","last_peered":"2024-09-20T22:52:23.289191+0000","last_clean":"2024-09-20T22:52:23.289191+0000","last_became_active":"2024-09-20T22:52:23.286461+0000","last_became_peered":"2024-09-20T22:52:23.286461+0000","last_unstale":"2024-09-20T22:52:23.289191+0000","last_undegraded":"2024-09-20T22:52:23.289191+0000","last_fullsized":"2024-09-20T22:52:23.289191+0000","mapping_epoch":662,"log_start":"471'464","ondisk_log_start":"471'464","created":94,"last_epoch_clean":663,"parent":"0.0","parent_split_bits":0,"last_scrub":"282'120","last_scrub_stamp":"2024-09-20T22:46:22.099642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:46:22.099642+0000","objects_scrubbed":1,"log_size":211,"log_dups_size":687,"ondisk_log_size":211,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":47,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00043656800000000003,"stat_sum":{"num_bytes":53298460,"num_objects":24,"num_object_clones":20,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":1,"num_read":376,"num_read_kb":396277,"num_write":228,"num_write_kb":92279,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":144,"num_bytes_recovered":106353268,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"a7","length":"1"}]},{"pgid":"1.4","version":"77'128","reported_seq":1529,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-09-20T22:52:23.283488+0000","last_change":"2024-09-20T22:52:02.474080+0000","last_active":"2024-09-20T22:52:23.283488+0000","last_peered":"2024-09-20T22:52:23.283488+0000","last_clean":"2024-09-20T22:52:23.283488+0000","last_became_active":"2024-09-20T22:46:15.181305+0000","last_became_peered":"2024-09-20T22:46:15.181305+0000","last_unstale":"2024-09-20T22:52:23.283488+0000","last_undegraded":"2024-09-20T22:52:23.283488+0000","last_fullsized":"2024-09-20T22:52:23.283488+0000","mapping_epoch":94,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:52:02.473979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:52:02.473979+0000","objects_scrubbed":1,"log_size":0,"log_dups_size":59,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:05.099562+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"625'28","reported_seq":582,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-09-20T22:52:23.281870+0000","last_change":"2024-09-20T22:51:58.712602+0000","last_active":"2024-09-20T22:52:23.281870+0000","last_peered":"2024-09-20T22:52:23.281870+0000","last_clean":"2024-09-20T22:52:23.281870+0000","last_became_active":"2024-09-20T22:48:05.235812+0000","last_became_peered":"2024-09-20T22:48:05.235812+0000","last_unstale":"2024-09-20T22:52:23.281870+0000","last_undegraded":"2024-09-20T22:52:23.281870+0000","last_fullsized":"2024-09-20T22:52:23.281870+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'28","last_scrub_stamp":"2024-09-20T22:51:58.712488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:51:58.712488+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:18.921188+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":41,"num_read_kb":34,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"67'1","reported_seq":601,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-09-20T22:52:23.283299+0000","last_change":"2024-09-20T22:51:20.292468+0000","last_active":"2024-09-20T22:52:23.283299+0000","last_peered":"2024-09-20T22:52:23.283299+0000","last_clean":"2024-09-20T22:52:23.283299+0000","last_became_active":"2024-09-20T22:51:20.283011+0000","last_became_peered":"2024-09-20T22:51:20.283011+0000","last_unstale":"2024-09-20T22:52:23.283299+0000","last_undegraded":"2024-09-20T22:52:23.283299+0000","last_fullsized":"2024-09-20T22:52:23.283299+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-09-20T22:51:11.806011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:51:11.806011+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:24.512033+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"600'13","reported_seq":1410,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-09-20T22:52:23.281522+0000","last_change":"2024-09-20T22:51:32.448600+0000","last_active":"2024-09-20T22:52:23.281522+0000","last_peered":"2024-09-20T22:52:23.281522+0000","last_clean":"2024-09-20T22:52:23.281522+0000","last_became_active":"2024-09-20T22:51:20.289873+0000","last_became_peered":"2024-09-20T22:51:20.289873+0000","last_unstale":"2024-09-20T22:52:23.281522+0000","last_undegraded":"2024-09-20T22:52:23.281522+0000","last_fullsized":"2024-09-20T22:52:23.281522+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"600'13","last_scrub_stamp":"2024-09-20T22:51:32.448512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:51:32.448512+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:52.962692+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":1225,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-09-20T22:52:23.284123+0000","last_change":"2024-09-20T22:51:31.487593+0000","last_active":"2024-09-20T22:52:23.284123+0000","last_peered":"2024-09-20T22:52:23.284123+0000","last_clean":"2024-09-20T22:52:23.284123+0000","last_became_active":"2024-09-20T22:41:44.831901+0000","last_became_peered":"2024-09-20T22:41:44.831901+0000","last_unstale":"2024-09-20T22:52:23.284123+0000","last_undegraded":"2024-09-20T22:52:23.284123+0000","last_fullsized":"2024-09-20T22:52:23.284123+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:31.487469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:51:31.487469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:01.276783+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1","version":"663'366","reported_seq":3143,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-09-20T22:52:23.303892+0000","last_change":"2024-09-20T22:52:20.274435+0000","last_active":"2024-09-20T22:52:23.303892+0000","last_peered":"2024-09-20T22:52:23.303892+0000","last_clean":"2024-09-20T22:52:23.303892+0000","last_became_active":"2024-09-20T22:49:21.232750+0000","last_became_peered":"2024-09-20T22:49:21.232750+0000","last_unstale":"2024-09-20T22:52:23.303892+0000","last_undegraded":"2024-09-20T22:52:23.303892+0000","last_fullsized":"2024-09-20T22:52:23.303892+0000","mapping_epoch":407,"log_start":"474'259","ondisk_log_start":"474'259","created":94,"last_epoch_clean":481,"parent":"0.0","parent_split_bits":0,"last_scrub":"562'298","last_scrub_stamp":"2024-09-20T22:50:45.816540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:50:45.816540+0000","objects_scrubbed":4,"log_size":107,"log_dups_size":338,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00050010400000000002,"stat_sum":{"num_bytes":13262300,"num_objects":8,"num_object_clones":6,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":1,"num_read":93,"num_read_kb":91249,"num_write":86,"num_write_kb":45551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":8732217,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"a7","length":"1"},{"start":"c5","length":"1"},{"start":"ce","length":"1"},{"start":"d4","length":"1"},{"start":"da","length":"1"},{"start":"df","length":"1"},{"start":"e3","length":"a"},{"start":"ee","length":"7"},{"start":"f6","length":"2"},{"start":"f9","length":"1"},{"start":"fc","length":"1"},{"start":"fe","length":"5"},{"start":"104","length":"3"},{"start":"108","length":"4"},{"start":"10d","length":"1"},{"start":"10f","length":"1"},{"start":"111","length":"1"},{"start":"113","length":"1"},{"start":"116","length":"2"},{"start":"11a","length":"1"},{"start":"11c","length":"1"},{"start":"125","length":"1"}]},{"pgid":"1.7","version":"77'128","reported_seq":1534,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-09-20T22:52:23.294832+0000","last_change":"2024-09-20T22:51:10.000911+0000","last_active":"2024-09-20T22:52:23.294832+0000","last_peered":"2024-09-20T22:52:23.294832+0000","last_clean":"2024-09-20T22:52:23.294832+0000","last_became_active":"2024-09-20T22:46:03.169596+0000","last_became_peered":"2024-09-20T22:46:03.169596+0000","last_unstale":"2024-09-20T22:52:23.294832+0000","last_undegraded":"2024-09-20T22:52:23.294832+0000","last_fullsized":"2024-09-20T22:52:23.294832+0000","mapping_epoch":94,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:51:10.000823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:51:10.000823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"625'20","reported_seq":207,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-09-20T22:52:23.630946+0000","last_change":"2024-09-20T22:52:19.125548+0000","last_active":"2024-09-20T22:52:23.630946+0000","last_peered":"2024-09-20T22:52:23.630946+0000","last_clean":"2024-09-20T22:52:23.630946+0000","last_became_active":"2024-09-20T22:51:20.297933+0000","last_became_peered":"2024-09-20T22:51:20.297933+0000","last_unstale":"2024-09-20T22:52:23.630946+0000","last_undegraded":"2024-09-20T22:52:23.630946+0000","last_fullsized":"2024-09-20T22:52:23.630946+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'20","last_scrub_stamp":"2024-09-20T22:52:19.125462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:52:19.125462+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:45.473839+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":1238,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-09-20T22:52:23.631397+0000","last_change":"2024-09-20T22:50:14.699223+0000","last_active":"2024-09-20T22:52:23.631397+0000","last_peered":"2024-09-20T22:52:23.631397+0000","last_clean":"2024-09-20T22:52:23.631397+0000","last_became_active":"2024-09-20T22:41:38.155629+0000","last_became_peered":"2024-09-20T22:41:38.155629+0000","last_unstale":"2024-09-20T22:52:23.631397+0000","last_undegraded":"2024-09-20T22:52:23.631397+0000","last_fullsized":"2024-09-20T22:52:23.631397+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:50:14.699152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:50:14.699152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"389'11","reported_seq":637,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-09-20T22:52:23.282622+0000","last_change":"2024-09-20T22:51:26.968244+0000","last_active":"2024-09-20T22:52:23.282622+0000","last_peered":"2024-09-20T22:52:23.282622+0000","last_clean":"2024-09-20T22:52:23.282622+0000","last_became_active":"2024-09-20T22:47:49.227264+0000","last_became_peered":"2024-09-20T22:47:49.227264+0000","last_unstale":"2024-09-20T22:52:23.282622+0000","last_undegraded":"2024-09-20T22:52:23.282622+0000","last_fullsized":"2024-09-20T22:52:23.282622+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'11","last_scrub_stamp":"2024-09-20T22:51:26.968138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:51:26.968138+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:30.761523+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":577,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-09-20T22:52:23.281096+0000","last_change":"2024-09-20T22:51:55.755550+0000","last_active":"2024-09-20T22:52:23.281096+0000","last_peered":"2024-09-20T22:52:23.281096+0000","last_clean":"2024-09-20T22:52:23.281096+0000","last_became_active":"2024-09-20T22:47:49.232380+0000","last_became_peered":"2024-09-20T22:47:49.232380+0000","last_unstale":"2024-09-20T22:52:23.281096+0000","last_undegraded":"2024-09-20T22:52:23.281096+0000","last_fullsized":"2024-09-20T22:52:23.281096+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:55.755432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:51:55.755432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:07.619648+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.0","version":"661'503","reported_seq":3394,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-09-20T22:52:23.630573+0000","last_change":"2024-09-20T22:52:23.406161+0000","last_active":"2024-09-20T22:52:23.630573+0000","last_peered":"2024-09-20T22:52:23.630573+0000","last_clean":"2024-09-20T22:52:23.630573+0000","last_became_active":"2024-09-20T22:51:53.366096+0000","last_became_peered":"2024-09-20T22:51:53.366096+0000","last_unstale":"2024-09-20T22:52:23.630573+0000","last_undegraded":"2024-09-20T22:52:23.630573+0000","last_fullsized":"2024-09-20T22:52:23.630573+0000","mapping_epoch":632,"log_start":"480'253","ondisk_log_start":"480'253","created":94,"last_epoch_clean":633,"parent":"0.0","parent_split_bits":0,"last_scrub":"562'368","last_scrub_stamp":"2024-09-20T22:50:44.669910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:50:44.669910+0000","objects_scrubbed":15,"log_size":250,"log_dups_size":416,"ondisk_log_size":250,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.24817879900000001,"stat_sum":{"num_bytes":35905141,"num_objects":18,"num_object_clones":15,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":190,"num_read_kb":202068,"num_write":151,"num_write_kb":50041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":42943203,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"a7","length":"1"},{"start":"c5","length":"1"},{"start":"ce","length":"1"},{"start":"d4","length":"1"},{"start":"da","length":"1"},{"start":"df","length":"1"},{"start":"e3","length":"a"},{"start":"ee","length":"7"},{"start":"f6","length":"2"},{"start":"f9","length":"1"},{"start":"fc","length":"1"},{"start":"fe","length":"5"},{"start":"104","length":"3"},{"start":"108","length":"4"},{"start":"10d","length":"1"},{"start":"10f","length":"1"},{"start":"111","length":"1"},{"start":"113","length":"1"},{"start":"116","length":"2"},{"start":"11a","length":"1"},{"start":"11c","length":"1"},{"start":"125","length":"1"}]},{"pgid":"1.6","version":"77'128","reported_seq":1556,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-09-20T22:52:23.294499+0000","last_change":"2024-09-20T22:51:06.036981+0000","last_active":"2024-09-20T22:52:23.294499+0000","last_peered":"2024-09-20T22:52:23.294499+0000","last_clean":"2024-09-20T22:52:23.294499+0000","last_became_active":"2024-09-20T22:47:52.212190+0000","last_became_peered":"2024-09-20T22:47:52.212190+0000","last_unstale":"2024-09-20T22:52:23.294499+0000","last_undegraded":"2024-09-20T22:52:23.294499+0000","last_fullsized":"2024-09-20T22:52:23.294499+0000","mapping_epoch":391,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:51:06.036900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:51:06.036900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:27.038972+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"625'37","reported_seq":610,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-09-20T22:52:23.281894+0000","last_change":"2024-09-20T22:49:55.964247+0000","last_active":"2024-09-20T22:52:23.281894+0000","last_peered":"2024-09-20T22:52:23.281894+0000","last_clean":"2024-09-20T22:52:23.281894+0000","last_became_active":"2024-09-20T22:47:49.222009+0000","last_became_peered":"2024-09-20T22:47:49.222009+0000","last_unstale":"2024-09-20T22:52:23.281894+0000","last_undegraded":"2024-09-20T22:52:23.281894+0000","last_fullsized":"2024-09-20T22:52:23.281894+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"515'27","last_scrub_stamp":"2024-09-20T22:49:55.964153+0000","last_deep_scrub":"515'27","last_deep_scrub_stamp":"2024-09-20T22:49:55.964153+0000","last_clean_scrub_stamp":"2024-09-20T22:49:55.964153+0000","objects_scrubbed":8,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":53,"num_read_kb":39,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":146,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-09-20T22:52:23.285629+0000","last_change":"2024-09-20T22:52:14.225251+0000","last_active":"2024-09-20T22:52:23.285629+0000","last_peered":"2024-09-20T22:52:23.285629+0000","last_clean":"2024-09-20T22:52:23.285629+0000","last_became_active":"2024-09-20T22:51:20.286646+0000","last_became_peered":"2024-09-20T22:51:20.286646+0000","last_unstale":"2024-09-20T22:52:23.285629+0000","last_undegraded":"2024-09-20T22:52:23.285629+0000","last_fullsized":"2024-09-20T22:52:23.285629+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:14.225111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:48:23.320031+0000","last_clean_scrub_stamp":"2024-09-20T22:52:14.225111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:38.894957+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":1249,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-09-20T22:52:23.294061+0000","last_change":"2024-09-20T22:51:29.928160+0000","last_active":"2024-09-20T22:52:23.294061+0000","last_peered":"2024-09-20T22:52:23.294061+0000","last_clean":"2024-09-20T22:52:23.294061+0000","last_became_active":"2024-09-20T22:47:49.214055+0000","last_became_peered":"2024-09-20T22:47:49.214055+0000","last_unstale":"2024-09-20T22:52:23.294061+0000","last_undegraded":"2024-09-20T22:52:23.294061+0000","last_fullsized":"2024-09-20T22:52:23.294061+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:29.928092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:51:29.928092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:32.798183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":146,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-09-20T22:52:23.294066+0000","last_change":"2024-09-20T22:51:51.100969+0000","last_active":"2024-09-20T22:52:23.294066+0000","last_peered":"2024-09-20T22:52:23.294066+0000","last_clean":"2024-09-20T22:52:23.294066+0000","last_became_active":"2024-09-20T22:51:20.284874+0000","last_became_peered":"2024-09-20T22:51:20.284874+0000","last_unstale":"2024-09-20T22:52:23.294066+0000","last_undegraded":"2024-09-20T22:52:23.294066+0000","last_fullsized":"2024-09-20T22:52:23.294066+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:51.100878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:48:41.208853+0000","last_clean_scrub_stamp":"2024-09-20T22:51:51.100878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:54.933631+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.7","version":"661'1060","reported_seq":3972,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-09-20T22:52:23.294934+0000","last_change":"2024-09-20T22:52:20.674641+0000","last_active":"2024-09-20T22:52:23.294934+0000","last_peered":"2024-09-20T22:52:23.294934+0000","last_clean":"2024-09-20T22:52:23.294934+0000","last_became_active":"2024-09-20T22:48:43.224386+0000","last_became_peered":"2024-09-20T22:48:43.224386+0000","last_unstale":"2024-09-20T22:52:23.294934+0000","last_undegraded":"2024-09-20T22:52:23.294934+0000","last_fullsized":"2024-09-20T22:52:23.294934+0000","mapping_epoch":391,"log_start":"436'456","ondisk_log_start":"436'456","created":94,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"631'991","last_scrub_stamp":"2024-09-20T22:51:54.106963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:51:54.106963+0000","objects_scrubbed":20,"log_size":604,"log_dups_size":886,"ondisk_log_size":604,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:20.725545+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.00037115899999999997,"stat_sum":{"num_bytes":47730670,"num_objects":25,"num_object_clones":20,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":2,"num_read":422,"num_read_kb":434616,"num_write":338,"num_write_kb":110295,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":115,"num_bytes_recovered":51615167,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"a7","length":"1"},{"start":"c5","length":"1"},{"start":"ce","length":"1"},{"start":"d4","length":"1"},{"start":"da","length":"1"},{"start":"df","length":"1"},{"start":"e3","length":"a"},{"start":"ee","length":"7"},{"start":"f6","length":"2"},{"start":"f9","length":"1"},{"start":"fc","length":"1"},{"start":"fe","length":"5"},{"start":"104","length":"3"},{"start":"108","length":"4"},{"start":"10d","length":"1"},{"start":"10f","length":"1"},{"start":"111","length":"1"},{"start":"113","length":"1"},{"start":"116","length":"2"},{"start":"11a","length":"1"},{"start":"11c","length":"1"},{"start":"125","length":"1"}]},{"pgid":"1.1","version":"77'128","reported_seq":1545,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-09-20T22:52:23.630601+0000","last_change":"2024-09-20T22:50:20.639645+0000","last_active":"2024-09-20T22:52:23.630601+0000","last_peered":"2024-09-20T22:52:23.630601+0000","last_clean":"2024-09-20T22:52:23.630601+0000","last_became_active":"2024-09-20T22:47:52.208102+0000","last_became_peered":"2024-09-20T22:47:52.208102+0000","last_unstale":"2024-09-20T22:52:23.630601+0000","last_undegraded":"2024-09-20T22:52:23.630601+0000","last_fullsized":"2024-09-20T22:52:23.630601+0000","mapping_epoch":391,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:50:20.639561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:50:20.639561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"625'51","reported_seq":310,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-09-20T22:52:23.285468+0000","last_change":"2024-09-20T22:52:17.199465+0000","last_active":"2024-09-20T22:52:23.285468+0000","last_peered":"2024-09-20T22:52:23.285468+0000","last_clean":"2024-09-20T22:52:23.285468+0000","last_became_active":"2024-09-20T22:51:20.289376+0000","last_became_peered":"2024-09-20T22:51:20.289376+0000","last_unstale":"2024-09-20T22:52:23.285468+0000","last_undegraded":"2024-09-20T22:52:23.285468+0000","last_fullsized":"2024-09-20T22:52:23.285468+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'51","last_scrub_stamp":"2024-09-20T22:52:17.199367+0000","last_deep_scrub":"86'38","last_deep_scrub_stamp":"2024-09-20T22:42:43.935677+0000","last_clean_scrub_stamp":"2024-09-20T22:52:17.199367+0000","objects_scrubbed":10,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:38.979686+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":537,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-09-20T22:52:23.282219+0000","last_change":"2024-09-20T22:50:35.898050+0000","last_active":"2024-09-20T22:52:23.282219+0000","last_peered":"2024-09-20T22:52:23.282219+0000","last_clean":"2024-09-20T22:52:23.282219+0000","last_became_active":"2024-09-20T22:48:05.237756+0000","last_became_peered":"2024-09-20T22:48:05.237756+0000","last_unstale":"2024-09-20T22:52:23.282219+0000","last_undegraded":"2024-09-20T22:52:23.282219+0000","last_fullsized":"2024-09-20T22:52:23.282219+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:50:35.897979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:50:35.897979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"69'2","reported_seq":1266,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-09-20T22:52:23.631160+0000","last_change":"2024-09-20T22:51:37.070187+0000","last_active":"2024-09-20T22:52:23.631160+0000","last_peered":"2024-09-20T22:52:23.631160+0000","last_clean":"2024-09-20T22:52:23.631160+0000","last_became_active":"2024-09-20T22:51:20.286952+0000","last_became_peered":"2024-09-20T22:51:20.286952+0000","last_unstale":"2024-09-20T22:52:23.631160+0000","last_undegraded":"2024-09-20T22:52:23.631160+0000","last_fullsized":"2024-09-20T22:52:23.631160+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2024-09-20T22:51:37.070111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:36.130684+0000","last_clean_scrub_stamp":"2024-09-20T22:51:37.070111+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:05.960971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087280599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":589,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-09-20T22:52:23.282294+0000","last_change":"2024-09-20T22:51:53.772969+0000","last_active":"2024-09-20T22:52:23.282294+0000","last_peered":"2024-09-20T22:52:23.282294+0000","last_clean":"2024-09-20T22:52:23.282294+0000","last_became_active":"2024-09-20T22:51:20.278427+0000","last_became_peered":"2024-09-20T22:51:20.278427+0000","last_unstale":"2024-09-20T22:52:23.282294+0000","last_undegraded":"2024-09-20T22:52:23.282294+0000","last_fullsized":"2024-09-20T22:52:23.282294+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:53.772885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:51:53.772885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:59.965395+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":1236,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-09-20T22:52:23.282949+0000","last_change":"2024-09-20T22:52:21.363432+0000","last_active":"2024-09-20T22:52:23.282949+0000","last_peered":"2024-09-20T22:52:23.282949+0000","last_clean":"2024-09-20T22:52:23.282949+0000","last_became_active":"2024-09-20T22:48:05.234193+0000","last_became_peered":"2024-09-20T22:48:05.234193+0000","last_unstale":"2024-09-20T22:52:23.282949+0000","last_undegraded":"2024-09-20T22:52:23.282949+0000","last_fullsized":"2024-09-20T22:52:23.282949+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:21.363359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:52:21.363359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:28.660587+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"657'389","reported_seq":1712,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-09-20T22:52:23.281576+0000","last_change":"2024-09-20T22:52:20.276842+0000","last_active":"2024-09-20T22:52:23.281576+0000","last_peered":"2024-09-20T22:52:23.281576+0000","last_clean":"2024-09-20T22:52:23.281576+0000","last_became_active":"2024-09-20T22:51:23.267613+0000","last_became_peered":"2024-09-20T22:51:23.267613+0000","last_unstale":"2024-09-20T22:52:23.281576+0000","last_undegraded":"2024-09-20T22:52:23.281576+0000","last_fullsized":"2024-09-20T22:52:23.281576+0000","mapping_epoch":602,"log_start":"456'176","ondisk_log_start":"456'176","created":94,"last_epoch_clean":603,"parent":"0.0","parent_split_bits":0,"last_scrub":"616'341","last_scrub_stamp":"2024-09-20T22:51:36.893731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:51:36.893731+0000","objects_scrubbed":8,"log_size":213,"log_dups_size":318,"ondisk_log_size":213,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:59.190070+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.001046834,"stat_sum":{"num_bytes":24989903,"num_objects":10,"num_object_clones":8,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":144,"num_read_kb":137177,"num_write":123,"num_write_kb":42628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":26566881,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"a7","length":"1"},{"start":"c5","length":"1"},{"start":"ce","length":"1"},{"start":"d4","length":"1"},{"start":"da","length":"1"},{"start":"df","length":"1"},{"start":"e3","length":"a"},{"start":"ee","length":"7"},{"start":"f6","length":"2"},{"start":"f9","length":"1"},{"start":"fc","length":"1"},{"start":"fe","length":"5"},{"start":"104","length":"3"},{"start":"108","length":"4"},{"start":"10d","length":"1"},{"start":"10f","length":"1"},{"start":"111","length":"1"},{"start":"113","length":"1"},{"start":"116","length":"2"},{"start":"11a","length":"1"},{"start":"11c","length":"1"},{"start":"125","length":"1"}]},{"pgid":"1.2","version":"77'128","reported_seq":1614,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-09-20T22:52:23.284244+0000","last_change":"2024-09-20T22:52:02.623634+0000","last_active":"2024-09-20T22:52:23.284244+0000","last_peered":"2024-09-20T22:52:23.284244+0000","last_clean":"2024-09-20T22:52:23.284244+0000","last_became_active":"2024-09-20T22:48:08.203028+0000","last_became_peered":"2024-09-20T22:48:08.203028+0000","last_unstale":"2024-09-20T22:52:23.284244+0000","last_undegraded":"2024-09-20T22:52:23.284244+0000","last_fullsized":"2024-09-20T22:52:23.284244+0000","mapping_epoch":407,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":408,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:52:02.623562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:52:02.623562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:31.354752+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"625'22","reported_seq":999,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-09-20T22:52:23.631456+0000","last_change":"2024-09-20T22:51:20.312887+0000","last_active":"2024-09-20T22:52:23.631456+0000","last_peered":"2024-09-20T22:52:23.631456+0000","last_clean":"2024-09-20T22:52:23.631456+0000","last_became_active":"2024-09-20T22:51:20.293610+0000","last_became_peered":"2024-09-20T22:51:20.293610+0000","last_unstale":"2024-09-20T22:52:23.631456+0000","last_undegraded":"2024-09-20T22:52:23.631456+0000","last_fullsized":"2024-09-20T22:52:23.631456+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"443'18","last_scrub_stamp":"2024-09-20T22:51:04.643936+0000","last_deep_scrub":"443'18","last_deep_scrub_stamp":"2024-09-20T22:48:44.182517+0000","last_clean_scrub_stamp":"2024-09-20T22:51:04.643936+0000","objects_scrubbed":5,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":990,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":146,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-09-20T22:52:23.285174+0000","last_change":"2024-09-20T22:51:32.279015+0000","last_active":"2024-09-20T22:52:23.285174+0000","last_peered":"2024-09-20T22:52:23.285174+0000","last_clean":"2024-09-20T22:52:23.285174+0000","last_became_active":"2024-09-20T22:51:20.278609+0000","last_became_peered":"2024-09-20T22:51:20.278609+0000","last_unstale":"2024-09-20T22:52:23.285174+0000","last_undegraded":"2024-09-20T22:52:23.285174+0000","last_fullsized":"2024-09-20T22:52:23.285174+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:32.278945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:51:32.278945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:54.818249+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"380'2","reported_seq":1283,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-09-20T22:52:23.285688+0000","last_change":"2024-09-20T22:50:39.347293+0000","last_active":"2024-09-20T22:52:23.285688+0000","last_peered":"2024-09-20T22:52:23.285688+0000","last_clean":"2024-09-20T22:52:23.285688+0000","last_became_active":"2024-09-20T22:47:49.222812+0000","last_became_peered":"2024-09-20T22:47:49.222812+0000","last_unstale":"2024-09-20T22:52:23.285688+0000","last_undegraded":"2024-09-20T22:52:23.285688+0000","last_fullsized":"2024-09-20T22:52:23.285688+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"380'2","last_scrub_stamp":"2024-09-20T22:50:39.347177+0000","last_deep_scrub":"380'2","last_deep_scrub_stamp":"2024-09-20T22:47:40.983039+0000","last_clean_scrub_stamp":"2024-09-20T22:50:39.347177+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001301459,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":1236,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-09-20T22:52:23.285688+0000","last_change":"2024-09-20T22:51:04.254552+0000","last_active":"2024-09-20T22:52:23.285688+0000","last_peered":"2024-09-20T22:52:23.285688+0000","last_clean":"2024-09-20T22:52:23.285688+0000","last_became_active":"2024-09-20T22:41:42.185112+0000","last_became_peered":"2024-09-20T22:41:42.185112+0000","last_unstale":"2024-09-20T22:52:23.285688+0000","last_undegraded":"2024-09-20T22:52:23.285688+0000","last_fullsized":"2024-09-20T22:52:23.285688+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:04.254441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:47:22.881109+0000","last_clean_scrub_stamp":"2024-09-20T22:51:04.254441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:31.528165+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":1225,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-09-20T22:52:23.284668+0000","last_change":"2024-09-20T22:51:11.568798+0000","last_active":"2024-09-20T22:52:23.284668+0000","last_peered":"2024-09-20T22:52:23.284668+0000","last_clean":"2024-09-20T22:52:23.284668+0000","last_became_active":"2024-09-20T22:41:44.830515+0000","last_became_peered":"2024-09-20T22:41:44.830515+0000","last_unstale":"2024-09-20T22:52:23.284668+0000","last_undegraded":"2024-09-20T22:52:23.284668+0000","last_fullsized":"2024-09-20T22:52:23.284668+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:11.568712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:51:11.568712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"655'312","reported_seq":3202,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-09-20T22:52:23.284551+0000","last_change":"2024-09-20T22:52:20.280703+0000","last_active":"2024-09-20T22:52:23.284551+0000","last_peered":"2024-09-20T22:52:23.284551+0000","last_clean":"2024-09-20T22:52:23.284551+0000","last_became_active":"2024-09-20T22:48:57.236417+0000","last_became_peered":"2024-09-20T22:48:57.236417+0000","last_unstale":"2024-09-20T22:52:23.284551+0000","last_undegraded":"2024-09-20T22:52:23.284551+0000","last_fullsized":"2024-09-20T22:52:23.284551+0000","mapping_epoch":391,"log_start":"450'190","ondisk_log_start":"450'190","created":94,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"605'278","last_scrub_stamp":"2024-09-20T22:51:25.467092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:51:25.467092+0000","objects_scrubbed":6,"log_size":122,"log_dups_size":240,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:26.264436+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00058659000000000001,"stat_sum":{"num_bytes":12667288,"num_objects":7,"num_object_clones":6,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":76,"num_read_kb":60075,"num_write":88,"num_write_kb":36243,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":19673726,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"a7","length":"1"},{"start":"c5","length":"1"},{"start":"ce","length":"1"},{"start":"d4","length":"1"},{"start":"da","length":"1"},{"start":"df","length":"1"},{"start":"e3","length":"a"},{"start":"ee","length":"7"},{"start":"f6","length":"2"},{"start":"f9","length":"1"},{"start":"fc","length":"1"},{"start":"fe","length":"5"},{"start":"104","length":"3"},{"start":"108","length":"4"},{"start":"10d","length":"1"},{"start":"10f","length":"1"},{"start":"111","length":"1"},{"start":"113","length":"1"},{"start":"116","length":"2"},{"start":"11a","length":"1"},{"start":"11c","length":"1"},{"start":"125","length":"1"}]},{"pgid":"1.3","version":"77'128","reported_seq":441,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-09-20T22:52:23.294237+0000","last_change":"2024-09-20T22:51:28.897355+0000","last_active":"2024-09-20T22:52:23.294237+0000","last_peered":"2024-09-20T22:52:23.294237+0000","last_clean":"2024-09-20T22:52:23.294237+0000","last_became_active":"2024-09-20T22:51:23.266810+0000","last_became_peered":"2024-09-20T22:51:23.266810+0000","last_unstale":"2024-09-20T22:52:23.294237+0000","last_undegraded":"2024-09-20T22:52:23.294237+0000","last_fullsized":"2024-09-20T22:52:23.294237+0000","mapping_epoch":602,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":603,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:51:28.897273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:51:28.897273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:58.849420+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.6","version":"625'24","reported_seq":621,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-09-20T22:52:23.283474+0000","last_change":"2024-09-20T22:51:20.300892+0000","last_active":"2024-09-20T22:52:23.283474+0000","last_peered":"2024-09-20T22:52:23.283474+0000","last_clean":"2024-09-20T22:52:23.283474+0000","last_became_active":"2024-09-20T22:51:20.283112+0000","last_became_peered":"2024-09-20T22:51:20.283112+0000","last_unstale":"2024-09-20T22:52:23.283474+0000","last_undegraded":"2024-09-20T22:52:23.283474+0000","last_fullsized":"2024-09-20T22:52:23.283474+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-09-20T22:50:14.745918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:50:14.745918+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":583,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-09-20T22:52:23.281939+0000","last_change":"2024-09-20T22:50:39.786895+0000","last_active":"2024-09-20T22:52:23.281939+0000","last_peered":"2024-09-20T22:52:23.281939+0000","last_clean":"2024-09-20T22:52:23.281939+0000","last_became_active":"2024-09-20T22:48:05.233093+0000","last_became_peered":"2024-09-20T22:48:05.233093+0000","last_unstale":"2024-09-20T22:52:23.281939+0000","last_undegraded":"2024-09-20T22:52:23.281939+0000","last_fullsized":"2024-09-20T22:52:23.281939+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:50:39.786825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:44:19.834641+0000","last_clean_scrub_stamp":"2024-09-20T22:50:39.786825+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"75'5","reported_seq":1915,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-09-20T22:52:23.284152+0000","last_change":"2024-09-20T22:51:20.290148+0000","last_active":"2024-09-20T22:52:23.284152+0000","last_peered":"2024-09-20T22:52:23.284152+0000","last_clean":"2024-09-20T22:52:23.284152+0000","last_became_active":"2024-09-20T22:51:20.281329+0000","last_became_peered":"2024-09-20T22:51:20.281329+0000","last_unstale":"2024-09-20T22:52:23.284152+0000","last_undegraded":"2024-09-20T22:52:23.284152+0000","last_fullsized":"2024-09-20T22:52:23.284152+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-09-20T22:50:21.363477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:36.130684+0000","last_clean_scrub_stamp":"2024-09-20T22:50:21.363477+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00078631700000000003,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":635,"num_read_kb":630,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1167,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":1247,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-09-20T22:52:23.631490+0000","last_change":"2024-09-20T22:51:20.274832+0000","last_active":"2024-09-20T22:52:23.631490+0000","last_peered":"2024-09-20T22:52:23.631490+0000","last_clean":"2024-09-20T22:52:23.631490+0000","last_became_active":"2024-09-20T22:51:20.274522+0000","last_became_peered":"2024-09-20T22:51:20.274522+0000","last_unstale":"2024-09-20T22:52:23.631490+0000","last_undegraded":"2024-09-20T22:52:23.631490+0000","last_fullsized":"2024-09-20T22:52:23.631490+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:50:18.695027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:50:18.695027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":138,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-09-20T22:52:23.630997+0000","last_change":"2024-09-20T22:51:20.288058+0000","last_active":"2024-09-20T22:52:23.630997+0000","last_peered":"2024-09-20T22:52:23.630997+0000","last_clean":"2024-09-20T22:52:23.630997+0000","last_became_active":"2024-09-20T22:51:20.287750+0000","last_became_peered":"2024-09-20T22:51:20.287750+0000","last_unstale":"2024-09-20T22:52:23.630997+0000","last_undegraded":"2024-09-20T22:52:23.630997+0000","last_fullsized":"2024-09-20T22:52:23.630997+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:05.621880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:51:05.621880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.6","version":"657'396","reported_seq":3217,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-09-20T22:52:23.283211+0000","last_change":"2024-09-20T22:52:20.277965+0000","last_active":"2024-09-20T22:52:23.283211+0000","last_peered":"2024-09-20T22:52:23.283211+0000","last_clean":"2024-09-20T22:52:23.283211+0000","last_became_active":"2024-09-20T22:48:47.234531+0000","last_became_peered":"2024-09-20T22:48:47.234531+0000","last_unstale":"2024-09-20T22:52:23.283211+0000","last_undegraded":"2024-09-20T22:52:23.283211+0000","last_fullsized":"2024-09-20T22:52:23.283211+0000","mapping_epoch":391,"log_start":"443'181","ondisk_log_start":"443'181","created":94,"last_epoch_clean":447,"parent":"0.0","parent_split_bits":0,"last_scrub":"654'392","last_scrub_stamp":"2024-09-20T22:52:14.706591+0000","last_deep_scrub":"440'177","last_deep_scrub_stamp":"2024-09-20T22:48:40.602342+0000","last_clean_scrub_stamp":"2024-09-20T22:52:14.706591+0000","objects_scrubbed":10,"log_size":215,"log_dups_size":314,"ondisk_log_size":215,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:20.455127+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.00045786299999999998,"stat_sum":{"num_bytes":22786045,"num_objects":10,"num_object_clones":8,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":1,"num_read":145,"num_read_kb":142854,"num_write":123,"num_write_kb":45901,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":18662529,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"a7","length":"1"},{"start":"c5","length":"1"},{"start":"ce","length":"1"},{"start":"d4","length":"1"},{"start":"da","length":"1"},{"start":"df","length":"1"},{"start":"e3","length":"a"},{"start":"ee","length":"7"},{"start":"f6","length":"2"},{"start":"f9","length":"1"},{"start":"fc","length":"1"},{"start":"fe","length":"5"},{"start":"104","length":"3"},{"start":"108","length":"4"},{"start":"10d","length":"1"},{"start":"10f","length":"1"},{"start":"111","length":"1"},{"start":"113","length":"1"},{"start":"116","length":"2"},{"start":"11a","length":"1"},{"start":"11c","length":"1"},{"start":"125","length":"1"}]},{"pgid":"4.5","version":"625'32","reported_seq":1507,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-09-20T22:52:23.294849+0000","last_change":"2024-09-20T22:51:33.961483+0000","last_active":"2024-09-20T22:52:23.294849+0000","last_peered":"2024-09-20T22:52:23.294849+0000","last_clean":"2024-09-20T22:52:23.294849+0000","last_became_active":"2024-09-20T22:48:05.243523+0000","last_became_peered":"2024-09-20T22:48:05.243523+0000","last_unstale":"2024-09-20T22:52:23.294849+0000","last_undegraded":"2024-09-20T22:52:23.294849+0000","last_fullsized":"2024-09-20T22:52:23.294849+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"408'20","last_scrub_stamp":"2024-09-20T22:51:33.961401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:51:33.961401+0000","objects_scrubbed":8,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:00.387882+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"67'1","reported_seq":1235,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-09-20T22:52:23.282216+0000","last_change":"2024-09-20T22:51:35.440516+0000","last_active":"2024-09-20T22:52:23.282216+0000","last_peered":"2024-09-20T22:52:23.282216+0000","last_clean":"2024-09-20T22:52:23.282216+0000","last_became_active":"2024-09-20T22:41:38.158917+0000","last_became_peered":"2024-09-20T22:41:38.158917+0000","last_unstale":"2024-09-20T22:52:23.282216+0000","last_undegraded":"2024-09-20T22:52:23.282216+0000","last_fullsized":"2024-09-20T22:52:23.282216+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-09-20T22:51:35.440416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:51:35.440416+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:53.239571+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"77'128","reported_seq":1586,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-09-20T22:52:23.284207+0000","last_change":"2024-09-20T22:52:18.688292+0000","last_active":"2024-09-20T22:52:23.284207+0000","last_peered":"2024-09-20T22:52:23.284207+0000","last_clean":"2024-09-20T22:52:23.284207+0000","last_became_active":"2024-09-20T22:47:52.206169+0000","last_became_peered":"2024-09-20T22:47:52.206169+0000","last_unstale":"2024-09-20T22:52:23.284207+0000","last_undegraded":"2024-09-20T22:52:23.284207+0000","last_fullsized":"2024-09-20T22:52:23.284207+0000","mapping_epoch":391,"log_start":"77'128","ondisk_log_start":"77'128","created":19,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:52:18.688135+0000","last_deep_scrub":"77'128","last_deep_scrub_stamp":"2024-09-20T22:47:22.417099+0000","last_clean_scrub_stamp":"2024-09-20T22:52:18.688135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:47.016821+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":1237,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-09-20T22:52:23.284510+0000","last_change":"2024-09-20T22:51:50.585717+0000","last_active":"2024-09-20T22:52:23.284510+0000","last_peered":"2024-09-20T22:52:23.284510+0000","last_clean":"2024-09-20T22:52:23.284510+0000","last_became_active":"2024-09-20T22:41:42.184387+0000","last_became_peered":"2024-09-20T22:41:42.184387+0000","last_unstale":"2024-09-20T22:52:23.284510+0000","last_undegraded":"2024-09-20T22:52:23.284510+0000","last_fullsized":"2024-09-20T22:52:23.284510+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:50.585627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:51:50.585627+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:03.411628+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":1237,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-09-20T22:52:23.631365+0000","last_change":"2024-09-20T22:49:57.715494+0000","last_active":"2024-09-20T22:52:23.631365+0000","last_peered":"2024-09-20T22:52:23.631365+0000","last_clean":"2024-09-20T22:52:23.631365+0000","last_became_active":"2024-09-20T22:48:05.241602+0000","last_became_peered":"2024-09-20T22:48:05.241602+0000","last_unstale":"2024-09-20T22:52:23.631365+0000","last_undegraded":"2024-09-20T22:52:23.631365+0000","last_fullsized":"2024-09-20T22:52:23.631365+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:57.715416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:49:57.715416+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"657'474","reported_seq":3419,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-09-20T22:52:23.294647+0000","last_change":"2024-09-20T22:52:20.674329+0000","last_active":"2024-09-20T22:52:23.294647+0000","last_peered":"2024-09-20T22:52:23.294647+0000","last_clean":"2024-09-20T22:52:23.294647+0000","last_became_active":"2024-09-20T22:51:23.269216+0000","last_became_peered":"2024-09-20T22:51:23.269216+0000","last_unstale":"2024-09-20T22:52:23.294647+0000","last_undegraded":"2024-09-20T22:52:23.294647+0000","last_fullsized":"2024-09-20T22:52:23.294647+0000","mapping_epoch":602,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":603,"parent":"0.0","parent_split_bits":0,"last_scrub":"631'450","last_scrub_stamp":"2024-09-20T22:51:52.091217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:51:52.091217+0000","objects_scrubbed":8,"log_size":474,"log_dups_size":0,"ondisk_log_size":474,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:52.119921+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.000479968,"stat_sum":{"num_bytes":16398843,"num_objects":9,"num_object_clones":7,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":1,"num_read":161,"num_read_kb":161235,"num_write":85,"num_write_kb":35211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":52,"num_bytes_recovered":30669453,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"a7","length":"1"},{"start":"c5","length":"1"},{"start":"ce","length":"1"},{"start":"d4","length":"1"},{"start":"da","length":"1"},{"start":"df","length":"1"},{"start":"e3","length":"a"},{"start":"ee","length":"7"},{"start":"f6","length":"2"},{"start":"f9","length":"1"},{"start":"fc","length":"1"},{"start":"fe","length":"5"},{"start":"104","length":"3"},{"start":"108","length":"4"},{"start":"10d","length":"1"},{"start":"10f","length":"1"},{"start":"111","length":"1"},{"start":"113","length":"1"},{"start":"116","length":"2"},{"start":"11a","length":"1"},{"start":"11c","length":"1"},{"start":"125","length":"1"}]},{"pgid":"4.e","version":"625'19","reported_seq":578,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-09-20T22:52:23.282696+0000","last_change":"2024-09-20T22:51:20.310962+0000","last_active":"2024-09-20T22:52:23.282696+0000","last_peered":"2024-09-20T22:52:23.282696+0000","last_clean":"2024-09-20T22:52:23.282696+0000","last_became_active":"2024-09-20T22:51:20.288958+0000","last_became_peered":"2024-09-20T22:51:20.288958+0000","last_unstale":"2024-09-20T22:52:23.282696+0000","last_undegraded":"2024-09-20T22:52:23.282696+0000","last_fullsized":"2024-09-20T22:52:23.282696+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-09-20T22:50:38.879205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:50:38.879205+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":544,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-09-20T22:52:23.281528+0000","last_change":"2024-09-20T22:51:29.757136+0000","last_active":"2024-09-20T22:52:23.281528+0000","last_peered":"2024-09-20T22:52:23.281528+0000","last_clean":"2024-09-20T22:52:23.281528+0000","last_became_active":"2024-09-20T22:48:05.254688+0000","last_became_peered":"2024-09-20T22:48:05.254688+0000","last_unstale":"2024-09-20T22:52:23.281528+0000","last_undegraded":"2024-09-20T22:52:23.281528+0000","last_fullsized":"2024-09-20T22:52:23.281528+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:29.757030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:51:29.757030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:52.073822+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":1241,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-09-20T22:52:23.631343+0000","last_change":"2024-09-20T22:51:51.002353+0000","last_active":"2024-09-20T22:52:23.631343+0000","last_peered":"2024-09-20T22:52:23.631343+0000","last_clean":"2024-09-20T22:52:23.631343+0000","last_became_active":"2024-09-20T22:48:05.233778+0000","last_became_peered":"2024-09-20T22:48:05.233778+0000","last_unstale":"2024-09-20T22:52:23.631343+0000","last_undegraded":"2024-09-20T22:52:23.631343+0000","last_fullsized":"2024-09-20T22:52:23.631343+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:51.002226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:51:51.002226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:20.564071+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":1216,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-09-20T22:52:23.283403+0000","last_change":"2024-09-20T22:50:47.230929+0000","last_active":"2024-09-20T22:52:23.283403+0000","last_peered":"2024-09-20T22:52:23.283403+0000","last_clean":"2024-09-20T22:52:23.283403+0000","last_became_active":"2024-09-20T22:41:44.832009+0000","last_became_peered":"2024-09-20T22:41:44.832009+0000","last_unstale":"2024-09-20T22:52:23.283403+0000","last_undegraded":"2024-09-20T22:52:23.283403+0000","last_fullsized":"2024-09-20T22:52:23.283403+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:50:47.230854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:48:40.127123+0000","last_clean_scrub_stamp":"2024-09-20T22:50:47.230854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.e","version":"657'602","reported_seq":3500,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-09-20T22:52:23.285461+0000","last_change":"2024-09-20T22:52:20.444780+0000","last_active":"2024-09-20T22:52:23.285461+0000","last_peered":"2024-09-20T22:52:23.285461+0000","last_clean":"2024-09-20T22:52:23.285461+0000","last_became_active":"2024-09-20T22:47:52.209452+0000","last_became_peered":"2024-09-20T22:47:52.209452+0000","last_unstale":"2024-09-20T22:52:23.285461+0000","last_undegraded":"2024-09-20T22:52:23.285461+0000","last_fullsized":"2024-09-20T22:52:23.285461+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"559'431","last_scrub_stamp":"2024-09-20T22:50:45.276861+0000","last_deep_scrub":"174'60","last_deep_scrub_stamp":"2024-09-20T22:44:18.301776+0000","last_clean_scrub_stamp":"2024-09-20T22:50:45.276861+0000","objects_scrubbed":8,"log_size":602,"log_dups_size":0,"ondisk_log_size":602,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0.00032937100000000001,"stat_sum":{"num_bytes":27857680,"num_objects":14,"num_object_clones":12,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":236,"num_read_kb":237527,"num_write":156,"num_write_kb":59501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":23110741,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"a7","length":"1"},{"start":"c5","length":"1"},{"start":"ce","length":"1"},{"start":"d4","length":"1"},{"start":"da","length":"1"},{"start":"df","length":"1"},{"start":"e3","length":"a"},{"start":"ee","length":"7"},{"start":"f6","length":"2"},{"start":"f9","length":"1"},{"start":"fc","length":"1"},{"start":"fe","length":"5"},{"start":"104","length":"3"},{"start":"108","length":"4"},{"start":"10d","length":"1"},{"start":"10f","length":"1"},{"start":"111","length":"1"},{"start":"113","length":"1"},{"start":"116","length":"2"},{"start":"11a","length":"1"},{"start":"11c","length":"1"},{"start":"125","length":"1"}]},{"pgid":"1.8","version":"77'128","reported_seq":1505,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-09-20T22:52:23.630532+0000","last_change":"2024-09-20T22:51:56.108038+0000","last_active":"2024-09-20T22:52:23.630532+0000","last_peered":"2024-09-20T22:52:23.630532+0000","last_clean":"2024-09-20T22:52:23.630532+0000","last_became_active":"2024-09-20T22:42:52.405594+0000","last_became_peered":"2024-09-20T22:42:52.405594+0000","last_unstale":"2024-09-20T22:52:23.630532+0000","last_undegraded":"2024-09-20T22:52:23.630532+0000","last_fullsized":"2024-09-20T22:52:23.630532+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:51:56.107944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:51:56.107944+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:19.128079+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"625'31","reported_seq":615,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-09-20T22:52:23.282571+0000","last_change":"2024-09-20T22:52:13.620897+0000","last_active":"2024-09-20T22:52:23.282571+0000","last_peered":"2024-09-20T22:52:23.282571+0000","last_clean":"2024-09-20T22:52:23.282571+0000","last_became_active":"2024-09-20T22:51:20.288846+0000","last_became_peered":"2024-09-20T22:51:20.288846+0000","last_unstale":"2024-09-20T22:52:23.282571+0000","last_undegraded":"2024-09-20T22:52:23.282571+0000","last_fullsized":"2024-09-20T22:52:23.282571+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'31","last_scrub_stamp":"2024-09-20T22:52:13.620754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:52:13.620754+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:26.544483+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":49,"num_read_kb":39,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":791,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-09-20T22:52:23.284964+0000","last_change":"2024-09-20T22:51:20.294379+0000","last_active":"2024-09-20T22:52:23.284964+0000","last_peered":"2024-09-20T22:52:23.284964+0000","last_clean":"2024-09-20T22:52:23.284964+0000","last_became_active":"2024-09-20T22:51:20.294073+0000","last_became_peered":"2024-09-20T22:51:20.294073+0000","last_unstale":"2024-09-20T22:52:23.284964+0000","last_undegraded":"2024-09-20T22:52:23.284964+0000","last_fullsized":"2024-09-20T22:52:23.284964+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:50:47.268782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:35.492562+0000","last_clean_scrub_stamp":"2024-09-20T22:50:47.268782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":857,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-09-20T22:52:23.282200+0000","last_change":"2024-09-20T22:51:25.738132+0000","last_active":"2024-09-20T22:52:23.282200+0000","last_peered":"2024-09-20T22:52:23.282200+0000","last_clean":"2024-09-20T22:52:23.282200+0000","last_became_active":"2024-09-20T22:51:20.281724+0000","last_became_peered":"2024-09-20T22:51:20.281724+0000","last_unstale":"2024-09-20T22:52:23.282200+0000","last_undegraded":"2024-09-20T22:52:23.282200+0000","last_fullsized":"2024-09-20T22:52:23.282200+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:25.738061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:51:25.738061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:41.584933+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":1252,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-09-20T22:52:23.285714+0000","last_change":"2024-09-20T22:50:41.284220+0000","last_active":"2024-09-20T22:52:23.285714+0000","last_peered":"2024-09-20T22:52:23.285714+0000","last_clean":"2024-09-20T22:52:23.285714+0000","last_became_active":"2024-09-20T22:48:05.239006+0000","last_became_peered":"2024-09-20T22:48:05.239006+0000","last_unstale":"2024-09-20T22:52:23.285714+0000","last_undegraded":"2024-09-20T22:52:23.285714+0000","last_fullsized":"2024-09-20T22:52:23.285714+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:50:41.284158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:50:41.284158+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"657'648","reported_seq":3515,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-09-20T22:52:23.283965+0000","last_change":"2024-09-20T22:52:20.278219+0000","last_active":"2024-09-20T22:52:23.283965+0000","last_peered":"2024-09-20T22:52:23.283965+0000","last_clean":"2024-09-20T22:52:23.283965+0000","last_became_active":"2024-09-20T22:47:52.216198+0000","last_became_peered":"2024-09-20T22:47:52.216198+0000","last_unstale":"2024-09-20T22:52:23.283965+0000","last_undegraded":"2024-09-20T22:52:23.283965+0000","last_fullsized":"2024-09-20T22:52:23.283965+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"630'603","last_scrub_stamp":"2024-09-20T22:51:50.769050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:51:50.769050+0000","objects_scrubbed":8,"log_size":648,"log_dups_size":0,"ondisk_log_size":648,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:05.630371+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00061398099999999999,"stat_sum":{"num_bytes":20843956,"num_objects":11,"num_object_clones":8,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":1,"num_read":279,"num_read_kb":277181,"num_write":176,"num_write_kb":57088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":53,"num_bytes_recovered":14983556,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"a7","length":"1"},{"start":"c5","length":"1"},{"start":"ce","length":"1"},{"start":"d4","length":"1"},{"start":"da","length":"1"},{"start":"df","length":"1"},{"start":"e3","length":"a"},{"start":"ee","length":"7"},{"start":"f6","length":"2"},{"start":"f9","length":"1"},{"start":"fc","length":"1"},{"start":"fe","length":"5"},{"start":"104","length":"3"},{"start":"108","length":"4"},{"start":"10d","length":"1"},{"start":"10f","length":"1"},{"start":"111","length":"1"},{"start":"113","length":"1"},{"start":"116","length":"2"},{"start":"11a","length":"1"},{"start":"11c","length":"1"},{"start":"125","length":"1"}]},{"pgid":"1.9","version":"77'128","reported_seq":1546,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-09-20T22:52:23.284142+0000","last_change":"2024-09-20T22:50:42.263313+0000","last_active":"2024-09-20T22:52:23.284142+0000","last_peered":"2024-09-20T22:52:23.284142+0000","last_clean":"2024-09-20T22:52:23.284142+0000","last_became_active":"2024-09-20T22:50:42.262862+0000","last_became_peered":"2024-09-20T22:50:42.262862+0000","last_unstale":"2024-09-20T22:52:23.284142+0000","last_undegraded":"2024-09-20T22:52:23.284142+0000","last_fullsized":"2024-09-20T22:52:23.284142+0000","mapping_epoch":561,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":562,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:49:52.301348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:49:52.301348+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"625'20","reported_seq":571,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-09-20T22:52:23.282448+0000","last_change":"2024-09-20T22:52:22.646547+0000","last_active":"2024-09-20T22:52:23.282448+0000","last_peered":"2024-09-20T22:52:23.282448+0000","last_clean":"2024-09-20T22:52:23.282448+0000","last_became_active":"2024-09-20T22:48:05.242811+0000","last_became_peered":"2024-09-20T22:48:05.242811+0000","last_unstale":"2024-09-20T22:52:23.282448+0000","last_undegraded":"2024-09-20T22:52:23.282448+0000","last_fullsized":"2024-09-20T22:52:23.282448+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'20","last_scrub_stamp":"2024-09-20T22:52:22.646468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:52:22.646468+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:48.440427+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":1252,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-09-20T22:52:23.282228+0000","last_change":"2024-09-20T22:50:13.205821+0000","last_active":"2024-09-20T22:52:23.282228+0000","last_peered":"2024-09-20T22:52:23.282228+0000","last_clean":"2024-09-20T22:52:23.282228+0000","last_became_active":"2024-09-20T22:48:05.233999+0000","last_became_peered":"2024-09-20T22:48:05.233999+0000","last_unstale":"2024-09-20T22:52:23.282228+0000","last_undegraded":"2024-09-20T22:52:23.282228+0000","last_fullsized":"2024-09-20T22:52:23.282228+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:50:13.205743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:50:13.205743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"261'9","reported_seq":1383,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-09-20T22:52:23.285517+0000","last_change":"2024-09-20T22:52:18.218722+0000","last_active":"2024-09-20T22:52:23.285517+0000","last_peered":"2024-09-20T22:52:23.285517+0000","last_clean":"2024-09-20T22:52:23.285517+0000","last_became_active":"2024-09-20T22:41:42.185863+0000","last_became_peered":"2024-09-20T22:41:42.185863+0000","last_unstale":"2024-09-20T22:52:23.285517+0000","last_undegraded":"2024-09-20T22:52:23.285517+0000","last_fullsized":"2024-09-20T22:52:23.285517+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'9","last_scrub_stamp":"2024-09-20T22:52:18.218644+0000","last_deep_scrub":"261'9","last_deep_scrub_stamp":"2024-09-20T22:46:50.974735+0000","last_clean_scrub_stamp":"2024-09-20T22:52:18.218644+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:38.055984+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":550,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-09-20T22:52:23.282489+0000","last_change":"2024-09-20T22:51:20.296567+0000","last_active":"2024-09-20T22:52:23.282489+0000","last_peered":"2024-09-20T22:52:23.282489+0000","last_clean":"2024-09-20T22:52:23.282489+0000","last_became_active":"2024-09-20T22:51:20.296204+0000","last_became_peered":"2024-09-20T22:51:20.296204+0000","last_unstale":"2024-09-20T22:52:23.282489+0000","last_undegraded":"2024-09-20T22:52:23.282489+0000","last_fullsized":"2024-09-20T22:52:23.282489+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:02.932364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:51:02.932364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.8","version":"657'721","reported_seq":3437,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-09-20T22:52:23.284832+0000","last_change":"2024-09-20T22:52:20.444019+0000","last_active":"2024-09-20T22:52:23.284832+0000","last_peered":"2024-09-20T22:52:23.284832+0000","last_clean":"2024-09-20T22:52:23.284832+0000","last_became_active":"2024-09-20T22:48:31.224359+0000","last_became_peered":"2024-09-20T22:48:31.224359+0000","last_unstale":"2024-09-20T22:52:23.284832+0000","last_undegraded":"2024-09-20T22:52:23.284832+0000","last_fullsized":"2024-09-20T22:52:23.284832+0000","mapping_epoch":407,"log_start":"426'268","ondisk_log_start":"426'268","created":94,"last_epoch_clean":431,"parent":"0.0","parent_split_bits":0,"last_scrub":"534'447","last_scrub_stamp":"2024-09-20T22:50:14.324477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:50:14.324477+0000","objects_scrubbed":15,"log_size":453,"log_dups_size":445,"ondisk_log_size":453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.00050000399999999996,"stat_sum":{"num_bytes":37656266,"num_objects":19,"num_object_clones":16,"num_object_copies":57,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":1,"num_read":268,"num_read_kb":271649,"num_write":206,"num_write_kb":68718,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":26027409,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"a7","length":"1"},{"start":"c5","length":"1"},{"start":"ce","length":"1"},{"start":"d4","length":"1"},{"start":"da","length":"1"},{"start":"df","length":"1"},{"start":"e3","length":"a"},{"start":"ee","length":"7"},{"start":"f6","length":"2"},{"start":"f9","length":"1"},{"start":"fc","length":"1"},{"start":"fe","length":"5"},{"start":"104","length":"3"},{"start":"108","length":"4"},{"start":"10d","length":"1"},{"start":"10f","length":"1"},{"start":"111","length":"1"},{"start":"113","length":"1"},{"start":"116","length":"2"},{"start":"11a","length":"1"},{"start":"11c","length":"1"},{"start":"125","length":"1"}]},{"pgid":"4.b","version":"625'15","reported_seq":628,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-09-20T22:52:23.282406+0000","last_change":"2024-09-20T22:51:20.296532+0000","last_active":"2024-09-20T22:52:23.282406+0000","last_peered":"2024-09-20T22:52:23.282406+0000","last_clean":"2024-09-20T22:52:23.282406+0000","last_became_active":"2024-09-20T22:51:20.273094+0000","last_became_peered":"2024-09-20T22:51:20.273094+0000","last_unstale":"2024-09-20T22:52:23.282406+0000","last_undegraded":"2024-09-20T22:52:23.282406+0000","last_fullsized":"2024-09-20T22:52:23.282406+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-09-20T22:50:36.883811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:50:36.883811+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":1237,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-09-20T22:52:23.631110+0000","last_change":"2024-09-20T22:51:12.755628+0000","last_active":"2024-09-20T22:52:23.631110+0000","last_peered":"2024-09-20T22:52:23.631110+0000","last_clean":"2024-09-20T22:52:23.631110+0000","last_became_active":"2024-09-20T22:41:38.155953+0000","last_became_peered":"2024-09-20T22:41:38.155953+0000","last_unstale":"2024-09-20T22:52:23.631110+0000","last_undegraded":"2024-09-20T22:52:23.631110+0000","last_fullsized":"2024-09-20T22:52:23.631110+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:12.755526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:51:12.755526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:32.975772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":1260,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-09-20T22:52:23.285153+0000","last_change":"2024-09-20T22:50:37.347748+0000","last_active":"2024-09-20T22:52:23.285153+0000","last_peered":"2024-09-20T22:52:23.285153+0000","last_clean":"2024-09-20T22:52:23.285153+0000","last_became_active":"2024-09-20T22:48:05.240613+0000","last_became_peered":"2024-09-20T22:48:05.240613+0000","last_unstale":"2024-09-20T22:52:23.285153+0000","last_undegraded":"2024-09-20T22:52:23.285153+0000","last_fullsized":"2024-09-20T22:52:23.285153+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:50:37.347636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:50:37.347636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":577,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-09-20T22:52:23.282033+0000","last_change":"2024-09-20T22:52:16.706497+0000","last_active":"2024-09-20T22:52:23.282033+0000","last_peered":"2024-09-20T22:52:23.282033+0000","last_clean":"2024-09-20T22:52:23.282033+0000","last_became_active":"2024-09-20T22:47:49.219486+0000","last_became_peered":"2024-09-20T22:47:49.219486+0000","last_unstale":"2024-09-20T22:52:23.282033+0000","last_undegraded":"2024-09-20T22:52:23.282033+0000","last_fullsized":"2024-09-20T22:52:23.282033+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:16.706429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:52:16.706429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:26.931246+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"660'1016","reported_seq":3932,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-09-20T22:52:23.282194+0000","last_change":"2024-09-20T22:52:20.279143+0000","last_active":"2024-09-20T22:52:23.282194+0000","last_peered":"2024-09-20T22:52:23.282194+0000","last_clean":"2024-09-20T22:52:23.282194+0000","last_became_active":"2024-09-20T22:51:23.270962+0000","last_became_peered":"2024-09-20T22:51:23.270962+0000","last_unstale":"2024-09-20T22:52:23.282194+0000","last_undegraded":"2024-09-20T22:52:23.282194+0000","last_fullsized":"2024-09-20T22:52:23.282194+0000","mapping_epoch":602,"log_start":"423'280","ondisk_log_start":"423'280","created":94,"last_epoch_clean":603,"parent":"0.0","parent_split_bits":0,"last_scrub":"590'828","last_scrub_stamp":"2024-09-20T22:51:10.909364+0000","last_deep_scrub":"590'828","last_deep_scrub_stamp":"2024-09-20T22:51:10.909364+0000","last_clean_scrub_stamp":"2024-09-20T22:51:10.909364+0000","objects_scrubbed":27,"log_size":736,"log_dups_size":649,"ondisk_log_size":736,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:35.541603+0000","scrub_duration":120,"objects_trimmed":0,"snaptrim_duration":0.0035389969999999999,"stat_sum":{"num_bytes":68902335,"num_objects":30,"num_object_clones":25,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":2,"num_read":366,"num_read_kb":365046,"num_write":296,"num_write_kb":116449,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":91020358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"a7","length":"1"},{"start":"c5","length":"1"},{"start":"ce","length":"1"},{"start":"d4","length":"1"},{"start":"da","length":"1"},{"start":"df","length":"1"},{"start":"e3","length":"a"},{"start":"ee","length":"7"},{"start":"f6","length":"2"},{"start":"f9","length":"1"},{"start":"fc","length":"1"},{"start":"fe","length":"5"},{"start":"104","length":"3"},{"start":"108","length":"4"},{"start":"10d","length":"1"},{"start":"10f","length":"1"},{"start":"111","length":"1"},{"start":"113","length":"1"},{"start":"116","length":"2"},{"start":"11a","length":"1"},{"start":"11c","length":"1"},{"start":"125","length":"1"}]},{"pgid":"4.a","version":"625'44","reported_seq":1338,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-09-20T22:52:23.294117+0000","last_change":"2024-09-20T22:52:14.038882+0000","last_active":"2024-09-20T22:52:23.294117+0000","last_peered":"2024-09-20T22:52:23.294117+0000","last_clean":"2024-09-20T22:52:23.294117+0000","last_became_active":"2024-09-20T22:51:20.287620+0000","last_became_peered":"2024-09-20T22:51:20.287620+0000","last_unstale":"2024-09-20T22:52:23.294117+0000","last_undegraded":"2024-09-20T22:52:23.294117+0000","last_fullsized":"2024-09-20T22:52:23.294117+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'44","last_scrub_stamp":"2024-09-20T22:52:14.038788+0000","last_deep_scrub":"113'28","last_deep_scrub_stamp":"2024-09-20T22:43:12.725900+0000","last_clean_scrub_stamp":"2024-09-20T22:52:14.038788+0000","objects_scrubbed":9,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:32.613295+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":1236,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-09-20T22:52:23.284899+0000","last_change":"2024-09-20T22:51:36.440271+0000","last_active":"2024-09-20T22:52:23.284899+0000","last_peered":"2024-09-20T22:52:23.284899+0000","last_clean":"2024-09-20T22:52:23.284899+0000","last_became_active":"2024-09-20T22:41:38.161524+0000","last_became_peered":"2024-09-20T22:41:38.161524+0000","last_unstale":"2024-09-20T22:52:23.284899+0000","last_undegraded":"2024-09-20T22:52:23.284899+0000","last_fullsized":"2024-09-20T22:52:23.284899+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:36.440160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:51:36.440160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:51.318747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":1248,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-09-20T22:52:23.285290+0000","last_change":"2024-09-20T22:50:15.301586+0000","last_active":"2024-09-20T22:52:23.285290+0000","last_peered":"2024-09-20T22:52:23.285290+0000","last_clean":"2024-09-20T22:52:23.285290+0000","last_became_active":"2024-09-20T22:47:49.223220+0000","last_became_peered":"2024-09-20T22:47:49.223220+0000","last_unstale":"2024-09-20T22:52:23.285290+0000","last_undegraded":"2024-09-20T22:52:23.285290+0000","last_fullsized":"2024-09-20T22:52:23.285290+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:50:15.301506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:50:15.301506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":1232,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-09-20T22:52:23.284135+0000","last_change":"2024-09-20T22:52:01.643899+0000","last_active":"2024-09-20T22:52:23.284135+0000","last_peered":"2024-09-20T22:52:23.284135+0000","last_clean":"2024-09-20T22:52:23.284135+0000","last_became_active":"2024-09-20T22:41:44.199395+0000","last_became_peered":"2024-09-20T22:41:44.199395+0000","last_unstale":"2024-09-20T22:52:23.284135+0000","last_undegraded":"2024-09-20T22:52:23.284135+0000","last_fullsized":"2024-09-20T22:52:23.284135+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:01.643803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:47:18.329209+0000","last_clean_scrub_stamp":"2024-09-20T22:52:01.643803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:04.584695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.a","version":"660'753","reported_seq":3552,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-09-20T22:52:23.281852+0000","last_change":"2024-09-20T22:52:22.445906+0000","last_active":"2024-09-20T22:52:23.281852+0000","last_peered":"2024-09-20T22:52:23.281852+0000","last_clean":"2024-09-20T22:52:23.281852+0000","last_became_active":"2024-09-20T22:42:52.405085+0000","last_became_peered":"2024-09-20T22:42:52.405085+0000","last_unstale":"2024-09-20T22:52:23.281852+0000","last_undegraded":"2024-09-20T22:52:23.281852+0000","last_fullsized":"2024-09-20T22:52:23.281852+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"505'466","last_scrub_stamp":"2024-09-20T22:49:47.088165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:49:47.088165+0000","objects_scrubbed":13,"log_size":753,"log_dups_size":0,"ondisk_log_size":753,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.160975702,"stat_sum":{"num_bytes":44584098,"num_objects":19,"num_object_clones":16,"num_object_copies":57,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":1,"num_read":334,"num_read_kb":364085,"num_write":241,"num_write_kb":94613,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"a7","length":"1"},{"start":"c5","length":"1"},{"start":"ce","length":"1"},{"start":"d4","length":"1"},{"start":"da","length":"1"},{"start":"df","length":"1"},{"start":"e3","length":"a"},{"start":"ee","length":"7"},{"start":"f6","length":"2"},{"start":"f9","length":"1"},{"start":"fc","length":"1"},{"start":"fe","length":"5"},{"start":"104","length":"3"},{"start":"108","length":"4"},{"start":"10d","length":"1"},{"start":"10f","length":"1"},{"start":"111","length":"1"},{"start":"113","length":"1"},{"start":"116","length":"2"},{"start":"11a","length":"1"},{"start":"11c","length":"1"},{"start":"125","length":"1"}]},{"pgid":"4.9","version":"625'18","reported_seq":593,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-09-20T22:52:23.281617+0000","last_change":"2024-09-20T22:52:11.295320+0000","last_active":"2024-09-20T22:52:23.281617+0000","last_peered":"2024-09-20T22:52:23.281617+0000","last_clean":"2024-09-20T22:52:23.281617+0000","last_became_active":"2024-09-20T22:52:11.286549+0000","last_became_peered":"2024-09-20T22:52:11.286549+0000","last_unstale":"2024-09-20T22:52:23.281617+0000","last_undegraded":"2024-09-20T22:52:23.281617+0000","last_fullsized":"2024-09-20T22:52:23.281617+0000","mapping_epoch":650,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":651,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-09-20T22:50:40.914303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:50:40.914303+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":21,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":1274,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-09-20T22:52:23.284746+0000","last_change":"2024-09-20T22:51:55.653755+0000","last_active":"2024-09-20T22:52:23.284746+0000","last_peered":"2024-09-20T22:52:23.284746+0000","last_clean":"2024-09-20T22:52:23.284746+0000","last_became_active":"2024-09-20T22:51:20.293701+0000","last_became_peered":"2024-09-20T22:51:20.293701+0000","last_unstale":"2024-09-20T22:52:23.284746+0000","last_undegraded":"2024-09-20T22:52:23.284746+0000","last_fullsized":"2024-09-20T22:52:23.284746+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:55.653662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:46:52.465940+0000","last_clean_scrub_stamp":"2024-09-20T22:51:55.653662+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:56.990647+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":1262,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-09-20T22:52:23.285032+0000","last_change":"2024-09-20T22:51:20.286960+0000","last_active":"2024-09-20T22:52:23.285032+0000","last_peered":"2024-09-20T22:52:23.285032+0000","last_clean":"2024-09-20T22:52:23.285032+0000","last_became_active":"2024-09-20T22:51:20.286626+0000","last_became_peered":"2024-09-20T22:51:20.286626+0000","last_unstale":"2024-09-20T22:52:23.285032+0000","last_undegraded":"2024-09-20T22:52:23.285032+0000","last_fullsized":"2024-09-20T22:52:23.285032+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:05.254319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:46:01.828080+0000","last_clean_scrub_stamp":"2024-09-20T22:51:05.254319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":1222,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-09-20T22:52:23.281870+0000","last_change":"2024-09-20T22:51:20.279408+0000","last_active":"2024-09-20T22:52:23.281870+0000","last_peered":"2024-09-20T22:52:23.281870+0000","last_clean":"2024-09-20T22:52:23.281870+0000","last_became_active":"2024-09-20T22:51:20.279088+0000","last_became_peered":"2024-09-20T22:51:20.279088+0000","last_unstale":"2024-09-20T22:52:23.281870+0000","last_undegraded":"2024-09-20T22:52:23.281870+0000","last_fullsized":"2024-09-20T22:52:23.281870+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:50:39.266034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:55.170379+0000","last_clean_scrub_stamp":"2024-09-20T22:50:39.266034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"661'1000","reported_seq":3979,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-09-20T22:52:23.630979+0000","last_change":"2024-09-20T22:52:23.164132+0000","last_active":"2024-09-20T22:52:23.630979+0000","last_peered":"2024-09-20T22:52:23.630979+0000","last_clean":"2024-09-20T22:52:23.630979+0000","last_became_active":"2024-09-20T22:51:23.271244+0000","last_became_peered":"2024-09-20T22:51:23.271244+0000","last_unstale":"2024-09-20T22:52:23.630979+0000","last_undegraded":"2024-09-20T22:52:23.630979+0000","last_fullsized":"2024-09-20T22:52:23.630979+0000","mapping_epoch":602,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":603,"parent":"0.0","parent_split_bits":0,"last_scrub":"661'1000","last_scrub_stamp":"2024-09-20T22:52:23.164028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:52:23.164028+0000","objects_scrubbed":23,"log_size":1000,"log_dups_size":0,"ondisk_log_size":1000,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:26.427068+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":1.0204075969999999,"stat_sum":{"num_bytes":58019600,"num_objects":23,"num_object_clones":18,"num_object_copies":69,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":1,"num_read":489,"num_read_kb":560245,"num_write":328,"num_write_kb":112895,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":76,"num_bytes_recovered":39288293,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"a7","length":"1"},{"start":"c5","length":"1"},{"start":"ce","length":"1"},{"start":"d4","length":"1"},{"start":"da","length":"1"},{"start":"df","length":"1"},{"start":"e3","length":"a"},{"start":"ee","length":"7"},{"start":"f6","length":"2"},{"start":"f9","length":"1"},{"start":"fc","length":"1"},{"start":"fe","length":"5"},{"start":"104","length":"3"},{"start":"108","length":"4"},{"start":"10d","length":"1"},{"start":"10f","length":"1"},{"start":"111","length":"1"},{"start":"113","length":"1"},{"start":"116","length":"2"},{"start":"11a","length":"1"},{"start":"11c","length":"1"},{"start":"125","length":"1"}]},{"pgid":"4.8","version":"625'27","reported_seq":1276,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-09-20T22:52:23.630701+0000","last_change":"2024-09-20T22:51:53.020721+0000","last_active":"2024-09-20T22:52:23.630701+0000","last_peered":"2024-09-20T22:52:23.630701+0000","last_clean":"2024-09-20T22:52:23.630701+0000","last_became_active":"2024-09-20T22:41:40.252113+0000","last_became_peered":"2024-09-20T22:41:40.252113+0000","last_unstale":"2024-09-20T22:52:23.630701+0000","last_undegraded":"2024-09-20T22:52:23.630701+0000","last_fullsized":"2024-09-20T22:52:23.630701+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'27","last_scrub_stamp":"2024-09-20T22:51:53.020628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:51:53.020628+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:14.005659+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'2","reported_seq":1299,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-09-20T22:52:23.285119+0000","last_change":"2024-09-20T22:51:26.421985+0000","last_active":"2024-09-20T22:52:23.285119+0000","last_peered":"2024-09-20T22:52:23.285119+0000","last_clean":"2024-09-20T22:52:23.285119+0000","last_became_active":"2024-09-20T22:48:05.238098+0000","last_became_peered":"2024-09-20T22:48:05.238098+0000","last_unstale":"2024-09-20T22:52:23.285119+0000","last_undegraded":"2024-09-20T22:52:23.285119+0000","last_fullsized":"2024-09-20T22:52:23.285119+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-09-20T22:51:26.421896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:51:26.421896+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:53.826247+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"408'10","reported_seq":1394,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-09-20T22:52:23.284456+0000","last_change":"2024-09-20T22:52:15.703572+0000","last_active":"2024-09-20T22:52:23.284456+0000","last_peered":"2024-09-20T22:52:23.284456+0000","last_clean":"2024-09-20T22:52:23.284456+0000","last_became_active":"2024-09-20T22:48:05.239344+0000","last_became_peered":"2024-09-20T22:48:05.239344+0000","last_unstale":"2024-09-20T22:52:23.284456+0000","last_undegraded":"2024-09-20T22:52:23.284456+0000","last_fullsized":"2024-09-20T22:52:23.284456+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"408'10","last_scrub_stamp":"2024-09-20T22:52:15.703464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:52:15.703464+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:23.273102+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":1237,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-09-20T22:52:23.630682+0000","last_change":"2024-09-20T22:51:54.066995+0000","last_active":"2024-09-20T22:52:23.630682+0000","last_peered":"2024-09-20T22:52:23.630682+0000","last_clean":"2024-09-20T22:52:23.630682+0000","last_became_active":"2024-09-20T22:47:49.227569+0000","last_became_peered":"2024-09-20T22:47:49.227569+0000","last_unstale":"2024-09-20T22:52:23.630682+0000","last_undegraded":"2024-09-20T22:52:23.630682+0000","last_fullsized":"2024-09-20T22:52:23.630682+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:54.066913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:51:54.066913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:18.059705+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":1264,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-09-20T22:52:23.294063+0000","last_change":"2024-09-20T22:51:58.129589+0000","last_active":"2024-09-20T22:52:23.294063+0000","last_peered":"2024-09-20T22:52:23.294063+0000","last_clean":"2024-09-20T22:52:23.294063+0000","last_became_active":"2024-09-20T22:47:49.213292+0000","last_became_peered":"2024-09-20T22:47:49.213292+0000","last_unstale":"2024-09-20T22:52:23.294063+0000","last_undegraded":"2024-09-20T22:52:23.294063+0000","last_fullsized":"2024-09-20T22:52:23.294063+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:58.129502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:51:58.129502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:11.276401+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"625'12","reported_seq":1264,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-09-20T22:52:23.283302+0000","last_change":"2024-09-20T22:52:14.322926+0000","last_active":"2024-09-20T22:52:23.283302+0000","last_peered":"2024-09-20T22:52:23.283302+0000","last_clean":"2024-09-20T22:52:23.283302+0000","last_became_active":"2024-09-20T22:43:09.116887+0000","last_became_peered":"2024-09-20T22:43:09.116887+0000","last_unstale":"2024-09-20T22:52:23.283302+0000","last_undegraded":"2024-09-20T22:52:23.283302+0000","last_fullsized":"2024-09-20T22:52:23.283302+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'12","last_scrub_stamp":"2024-09-20T22:52:14.322811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:52:14.322811+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:23.704685+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":1235,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-09-20T22:52:23.631353+0000","last_change":"2024-09-20T22:51:20.293964+0000","last_active":"2024-09-20T22:52:23.631353+0000","last_peered":"2024-09-20T22:52:23.631353+0000","last_clean":"2024-09-20T22:52:23.631353+0000","last_became_active":"2024-09-20T22:51:20.293232+0000","last_became_peered":"2024-09-20T22:51:20.293232+0000","last_unstale":"2024-09-20T22:52:23.631353+0000","last_undegraded":"2024-09-20T22:52:23.631353+0000","last_fullsized":"2024-09-20T22:52:23.631353+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:50:22.597796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:09.705632+0000","last_clean_scrub_stamp":"2024-09-20T22:50:22.597796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"74'1","reported_seq":1245,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-09-20T22:52:23.284508+0000","last_change":"2024-09-20T22:51:05.527388+0000","last_active":"2024-09-20T22:52:23.284508+0000","last_peered":"2024-09-20T22:52:23.284508+0000","last_clean":"2024-09-20T22:52:23.284508+0000","last_became_active":"2024-09-20T22:48:05.237882+0000","last_became_peered":"2024-09-20T22:48:05.237882+0000","last_unstale":"2024-09-20T22:52:23.284508+0000","last_undegraded":"2024-09-20T22:52:23.284508+0000","last_fullsized":"2024-09-20T22:52:23.284508+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-09-20T22:51:05.527264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:51:05.527264+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"625'15","reported_seq":595,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-09-20T22:52:23.283337+0000","last_change":"2024-09-20T22:51:52.816826+0000","last_active":"2024-09-20T22:52:23.283337+0000","last_peered":"2024-09-20T22:52:23.283337+0000","last_clean":"2024-09-20T22:52:23.283337+0000","last_became_active":"2024-09-20T22:47:49.231315+0000","last_became_peered":"2024-09-20T22:47:49.231315+0000","last_unstale":"2024-09-20T22:52:23.283337+0000","last_undegraded":"2024-09-20T22:52:23.283337+0000","last_fullsized":"2024-09-20T22:52:23.283337+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'15","last_scrub_stamp":"2024-09-20T22:51:52.816714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:51:52.816714+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:55.354430+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":1260,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-09-20T22:52:23.283976+0000","last_change":"2024-09-20T22:51:12.541531+0000","last_active":"2024-09-20T22:52:23.283976+0000","last_peered":"2024-09-20T22:52:23.283976+0000","last_clean":"2024-09-20T22:52:23.283976+0000","last_became_active":"2024-09-20T22:48:05.238801+0000","last_became_peered":"2024-09-20T22:48:05.238801+0000","last_unstale":"2024-09-20T22:52:23.283976+0000","last_undegraded":"2024-09-20T22:52:23.283976+0000","last_fullsized":"2024-09-20T22:52:23.283976+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:12.541457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:51:12.541457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:38.408641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":1234,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-09-20T22:52:23.283150+0000","last_change":"2024-09-20T22:52:20.410227+0000","last_active":"2024-09-20T22:52:23.283150+0000","last_peered":"2024-09-20T22:52:23.283150+0000","last_clean":"2024-09-20T22:52:23.283150+0000","last_became_active":"2024-09-20T22:51:20.279397+0000","last_became_peered":"2024-09-20T22:51:20.279397+0000","last_unstale":"2024-09-20T22:52:23.283150+0000","last_undegraded":"2024-09-20T22:52:23.283150+0000","last_fullsized":"2024-09-20T22:52:23.283150+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:20.410164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:52:20.410164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:50.341745+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":1264,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-09-20T22:52:23.285067+0000","last_change":"2024-09-20T22:52:19.253411+0000","last_active":"2024-09-20T22:52:23.285067+0000","last_peered":"2024-09-20T22:52:23.285067+0000","last_clean":"2024-09-20T22:52:23.285067+0000","last_became_active":"2024-09-20T22:48:05.239710+0000","last_became_peered":"2024-09-20T22:48:05.239710+0000","last_unstale":"2024-09-20T22:52:23.285067+0000","last_undegraded":"2024-09-20T22:52:23.285067+0000","last_fullsized":"2024-09-20T22:52:23.285067+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:19.253343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:52:19.253343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:44.397693+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"625'15","reported_seq":1258,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-09-20T22:52:23.631339+0000","last_change":"2024-09-20T22:51:05.675215+0000","last_active":"2024-09-20T22:52:23.631339+0000","last_peered":"2024-09-20T22:52:23.631339+0000","last_clean":"2024-09-20T22:52:23.631339+0000","last_became_active":"2024-09-20T22:41:40.252486+0000","last_became_peered":"2024-09-20T22:41:40.252486+0000","last_unstale":"2024-09-20T22:52:23.631339+0000","last_undegraded":"2024-09-20T22:52:23.631339+0000","last_fullsized":"2024-09-20T22:52:23.631339+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-09-20T22:51:05.675095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:51:05.675095+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":584,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-09-20T22:52:23.281455+0000","last_change":"2024-09-20T22:52:13.735451+0000","last_active":"2024-09-20T22:52:23.281455+0000","last_peered":"2024-09-20T22:52:23.281455+0000","last_clean":"2024-09-20T22:52:23.281455+0000","last_became_active":"2024-09-20T22:47:49.229936+0000","last_became_peered":"2024-09-20T22:47:49.229936+0000","last_unstale":"2024-09-20T22:52:23.281455+0000","last_undegraded":"2024-09-20T22:52:23.281455+0000","last_fullsized":"2024-09-20T22:52:23.281455+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:13.735382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:52:13.735382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:22.644334+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"74'8","reported_seq":1361,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-09-20T22:52:23.283290+0000","last_change":"2024-09-20T22:50:38.269118+0000","last_active":"2024-09-20T22:52:23.283290+0000","last_peered":"2024-09-20T22:52:23.283290+0000","last_clean":"2024-09-20T22:52:23.283290+0000","last_became_active":"2024-09-20T22:41:42.186352+0000","last_became_peered":"2024-09-20T22:41:42.186352+0000","last_unstale":"2024-09-20T22:52:23.283290+0000","last_undegraded":"2024-09-20T22:52:23.283290+0000","last_fullsized":"2024-09-20T22:52:23.283290+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-09-20T22:50:38.269010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:50:38.269010+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":544,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-09-20T22:52:23.282854+0000","last_change":"2024-09-20T22:52:16.623463+0000","last_active":"2024-09-20T22:52:23.282854+0000","last_peered":"2024-09-20T22:52:23.282854+0000","last_clean":"2024-09-20T22:52:23.282854+0000","last_became_active":"2024-09-20T22:48:05.249656+0000","last_became_peered":"2024-09-20T22:48:05.249656+0000","last_unstale":"2024-09-20T22:52:23.282854+0000","last_undegraded":"2024-09-20T22:52:23.282854+0000","last_fullsized":"2024-09-20T22:52:23.282854+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:16.623368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:52:16.623368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:22.519570+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"625'20","reported_seq":1272,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-09-20T22:52:23.283278+0000","last_change":"2024-09-20T22:51:20.298112+0000","last_active":"2024-09-20T22:52:23.283278+0000","last_peered":"2024-09-20T22:52:23.283278+0000","last_clean":"2024-09-20T22:52:23.283278+0000","last_became_active":"2024-09-20T22:51:20.281859+0000","last_became_peered":"2024-09-20T22:51:20.281859+0000","last_unstale":"2024-09-20T22:52:23.283278+0000","last_undegraded":"2024-09-20T22:52:23.283278+0000","last_fullsized":"2024-09-20T22:52:23.283278+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-09-20T22:50:15.240945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:50:15.240945+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":1256,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-09-20T22:52:23.284453+0000","last_change":"2024-09-20T22:52:20.621799+0000","last_active":"2024-09-20T22:52:23.284453+0000","last_peered":"2024-09-20T22:52:23.284453+0000","last_clean":"2024-09-20T22:52:23.284453+0000","last_became_active":"2024-09-20T22:48:05.240231+0000","last_became_peered":"2024-09-20T22:48:05.240231+0000","last_unstale":"2024-09-20T22:52:23.284453+0000","last_undegraded":"2024-09-20T22:52:23.284453+0000","last_fullsized":"2024-09-20T22:52:23.284453+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:20.621718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:52:20.621718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:30.275424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"600'11","reported_seq":1401,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-09-20T22:52:23.631139+0000","last_change":"2024-09-20T22:51:20.303774+0000","last_active":"2024-09-20T22:52:23.631139+0000","last_peered":"2024-09-20T22:52:23.631139+0000","last_clean":"2024-09-20T22:52:23.631139+0000","last_became_active":"2024-09-20T22:51:20.283393+0000","last_became_peered":"2024-09-20T22:51:20.283393+0000","last_unstale":"2024-09-20T22:52:23.631139+0000","last_undegraded":"2024-09-20T22:52:23.631139+0000","last_fullsized":"2024-09-20T22:52:23.631139+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'10","last_scrub_stamp":"2024-09-20T22:49:51.715471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:49:51.715471+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":568,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-09-20T22:52:23.283019+0000","last_change":"2024-09-20T22:50:21.853169+0000","last_active":"2024-09-20T22:52:23.283019+0000","last_peered":"2024-09-20T22:52:23.283019+0000","last_clean":"2024-09-20T22:52:23.283019+0000","last_became_active":"2024-09-20T22:47:49.230674+0000","last_became_peered":"2024-09-20T22:47:49.230674+0000","last_unstale":"2024-09-20T22:52:23.283019+0000","last_undegraded":"2024-09-20T22:52:23.283019+0000","last_fullsized":"2024-09-20T22:52:23.283019+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:50:21.853070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:50:21.853070+0000","last_clean_scrub_stamp":"2024-09-20T22:50:21.853070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"625'19","reported_seq":586,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-09-20T22:52:23.282227+0000","last_change":"2024-09-20T22:51:20.317969+0000","last_active":"2024-09-20T22:52:23.282227+0000","last_peered":"2024-09-20T22:52:23.282227+0000","last_clean":"2024-09-20T22:52:23.282227+0000","last_became_active":"2024-09-20T22:51:20.289584+0000","last_became_peered":"2024-09-20T22:51:20.289584+0000","last_unstale":"2024-09-20T22:52:23.282227+0000","last_undegraded":"2024-09-20T22:52:23.282227+0000","last_fullsized":"2024-09-20T22:52:23.282227+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-09-20T22:51:07.926382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:51:07.926382+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:30.537714+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":544,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-09-20T22:52:23.282327+0000","last_change":"2024-09-20T22:51:08.874933+0000","last_active":"2024-09-20T22:52:23.282327+0000","last_peered":"2024-09-20T22:52:23.282327+0000","last_clean":"2024-09-20T22:52:23.282327+0000","last_became_active":"2024-09-20T22:48:05.252026+0000","last_became_peered":"2024-09-20T22:48:05.252026+0000","last_unstale":"2024-09-20T22:52:23.282327+0000","last_undegraded":"2024-09-20T22:52:23.282327+0000","last_fullsized":"2024-09-20T22:52:23.282327+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:08.874812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:51:08.874812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":138,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-09-20T22:52:23.630824+0000","last_change":"2024-09-20T22:51:20.291857+0000","last_active":"2024-09-20T22:52:23.630824+0000","last_peered":"2024-09-20T22:52:23.630824+0000","last_clean":"2024-09-20T22:52:23.630824+0000","last_became_active":"2024-09-20T22:51:20.291264+0000","last_became_peered":"2024-09-20T22:51:20.291264+0000","last_unstale":"2024-09-20T22:52:23.630824+0000","last_undegraded":"2024-09-20T22:52:23.630824+0000","last_fullsized":"2024-09-20T22:52:23.630824+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:06.651167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:51:06.651167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:34.034653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":1228,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-09-20T22:52:23.281320+0000","last_change":"2024-09-20T22:50:20.207052+0000","last_active":"2024-09-20T22:52:23.281320+0000","last_peered":"2024-09-20T22:52:23.281320+0000","last_clean":"2024-09-20T22:52:23.281320+0000","last_became_active":"2024-09-20T22:47:49.216141+0000","last_became_peered":"2024-09-20T22:47:49.216141+0000","last_unstale":"2024-09-20T22:52:23.281320+0000","last_undegraded":"2024-09-20T22:52:23.281320+0000","last_fullsized":"2024-09-20T22:52:23.281320+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:50:20.206984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:50:20.206984+0000","last_clean_scrub_stamp":"2024-09-20T22:50:20.206984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"625'19","reported_seq":882,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-09-20T22:52:23.294287+0000","last_change":"2024-09-20T22:51:25.879860+0000","last_active":"2024-09-20T22:52:23.294287+0000","last_peered":"2024-09-20T22:52:23.294287+0000","last_clean":"2024-09-20T22:52:23.294287+0000","last_became_active":"2024-09-20T22:51:20.283868+0000","last_became_peered":"2024-09-20T22:51:20.283868+0000","last_unstale":"2024-09-20T22:52:23.294287+0000","last_undegraded":"2024-09-20T22:52:23.294287+0000","last_fullsized":"2024-09-20T22:52:23.294287+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"404'13","last_scrub_stamp":"2024-09-20T22:51:25.879750+0000","last_deep_scrub":"404'13","last_deep_scrub_stamp":"2024-09-20T22:48:04.276282+0000","last_clean_scrub_stamp":"2024-09-20T22:51:25.879750+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:35.823216+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,0],"acting":[6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":1263,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-09-20T22:52:23.284796+0000","last_change":"2024-09-20T22:49:56.396435+0000","last_active":"2024-09-20T22:52:23.284796+0000","last_peered":"2024-09-20T22:52:23.284796+0000","last_clean":"2024-09-20T22:52:23.284796+0000","last_became_active":"2024-09-20T22:48:05.238812+0000","last_became_peered":"2024-09-20T22:48:05.238812+0000","last_unstale":"2024-09-20T22:52:23.284796+0000","last_undegraded":"2024-09-20T22:52:23.284796+0000","last_fullsized":"2024-09-20T22:52:23.284796+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:56.396347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:31.138067+0000","last_clean_scrub_stamp":"2024-09-20T22:49:56.396347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":1238,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-09-20T22:52:23.281991+0000","last_change":"2024-09-20T22:51:20.288308+0000","last_active":"2024-09-20T22:52:23.281991+0000","last_peered":"2024-09-20T22:52:23.281991+0000","last_clean":"2024-09-20T22:52:23.281991+0000","last_became_active":"2024-09-20T22:51:20.288000+0000","last_became_peered":"2024-09-20T22:51:20.288000+0000","last_unstale":"2024-09-20T22:52:23.281991+0000","last_undegraded":"2024-09-20T22:52:23.281991+0000","last_fullsized":"2024-09-20T22:52:23.281991+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:50:44.313182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:50:44.313182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":574,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-09-20T22:52:23.281360+0000","last_change":"2024-09-20T22:51:20.278471+0000","last_active":"2024-09-20T22:52:23.281360+0000","last_peered":"2024-09-20T22:52:23.281360+0000","last_clean":"2024-09-20T22:52:23.281360+0000","last_became_active":"2024-09-20T22:51:20.278147+0000","last_became_peered":"2024-09-20T22:51:20.278147+0000","last_unstale":"2024-09-20T22:52:23.281360+0000","last_undegraded":"2024-09-20T22:52:23.281360+0000","last_fullsized":"2024-09-20T22:52:23.281360+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:50:35.895483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:50:35.895483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"625'24","reported_seq":1059,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-09-20T22:52:23.630536+0000","last_change":"2024-09-20T22:50:39.665781+0000","last_active":"2024-09-20T22:52:23.630536+0000","last_peered":"2024-09-20T22:52:23.630536+0000","last_clean":"2024-09-20T22:52:23.630536+0000","last_became_active":"2024-09-20T22:44:09.170785+0000","last_became_peered":"2024-09-20T22:44:09.170785+0000","last_unstale":"2024-09-20T22:52:23.630536+0000","last_undegraded":"2024-09-20T22:52:23.630536+0000","last_fullsized":"2024-09-20T22:52:23.630536+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"470'16","last_scrub_stamp":"2024-09-20T22:50:39.665682+0000","last_deep_scrub":"470'16","last_deep_scrub_stamp":"2024-09-20T22:49:10.659868+0000","last_clean_scrub_stamp":"2024-09-20T22:50:39.665682+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":1243,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-09-20T22:52:23.631451+0000","last_change":"2024-09-20T22:51:20.294181+0000","last_active":"2024-09-20T22:52:23.631451+0000","last_peered":"2024-09-20T22:52:23.631451+0000","last_clean":"2024-09-20T22:52:23.631451+0000","last_became_active":"2024-09-20T22:51:20.293645+0000","last_became_peered":"2024-09-20T22:51:20.293645+0000","last_unstale":"2024-09-20T22:52:23.631451+0000","last_undegraded":"2024-09-20T22:52:23.631451+0000","last_fullsized":"2024-09-20T22:52:23.631451+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:50:13.688169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:45:24.765414+0000","last_clean_scrub_stamp":"2024-09-20T22:50:13.688169+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":1252,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-09-20T22:52:23.283837+0000","last_change":"2024-09-20T22:51:10.610718+0000","last_active":"2024-09-20T22:52:23.283837+0000","last_peered":"2024-09-20T22:52:23.283837+0000","last_clean":"2024-09-20T22:52:23.283837+0000","last_became_active":"2024-09-20T22:48:05.238334+0000","last_became_peered":"2024-09-20T22:48:05.238334+0000","last_unstale":"2024-09-20T22:52:23.283837+0000","last_undegraded":"2024-09-20T22:52:23.283837+0000","last_fullsized":"2024-09-20T22:52:23.283837+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:10.610588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:50.174861+0000","last_clean_scrub_stamp":"2024-09-20T22:51:10.610588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":1228,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-09-20T22:52:23.282140+0000","last_change":"2024-09-20T22:51:04.379662+0000","last_active":"2024-09-20T22:52:23.282140+0000","last_peered":"2024-09-20T22:52:23.282140+0000","last_clean":"2024-09-20T22:52:23.282140+0000","last_became_active":"2024-09-20T22:47:49.214487+0000","last_became_peered":"2024-09-20T22:47:49.214487+0000","last_unstale":"2024-09-20T22:52:23.282140+0000","last_undegraded":"2024-09-20T22:52:23.282140+0000","last_fullsized":"2024-09-20T22:52:23.282140+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:04.379558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:51:04.379558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"625'8","reported_seq":1248,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-09-20T22:52:23.281287+0000","last_change":"2024-09-20T22:51:20.314500+0000","last_active":"2024-09-20T22:52:23.281287+0000","last_peered":"2024-09-20T22:52:23.281287+0000","last_clean":"2024-09-20T22:52:23.281287+0000","last_became_active":"2024-09-20T22:51:20.290269+0000","last_became_peered":"2024-09-20T22:51:20.290269+0000","last_unstale":"2024-09-20T22:52:23.281287+0000","last_undegraded":"2024-09-20T22:52:23.281287+0000","last_fullsized":"2024-09-20T22:52:23.281287+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'4","last_scrub_stamp":"2024-09-20T22:50:40.292964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:50:40.292964+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":576,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-09-20T22:52:23.283735+0000","last_change":"2024-09-20T22:50:43.756048+0000","last_active":"2024-09-20T22:52:23.283735+0000","last_peered":"2024-09-20T22:52:23.283735+0000","last_clean":"2024-09-20T22:52:23.283735+0000","last_became_active":"2024-09-20T22:47:49.231964+0000","last_became_peered":"2024-09-20T22:47:49.231964+0000","last_unstale":"2024-09-20T22:52:23.283735+0000","last_undegraded":"2024-09-20T22:52:23.283735+0000","last_fullsized":"2024-09-20T22:52:23.283735+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:50:43.755981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:50:43.755981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":1268,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-09-20T22:52:23.294186+0000","last_change":"2024-09-20T22:52:20.955721+0000","last_active":"2024-09-20T22:52:23.294186+0000","last_peered":"2024-09-20T22:52:23.294186+0000","last_clean":"2024-09-20T22:52:23.294186+0000","last_became_active":"2024-09-20T22:48:05.244659+0000","last_became_peered":"2024-09-20T22:48:05.244659+0000","last_unstale":"2024-09-20T22:52:23.294186+0000","last_undegraded":"2024-09-20T22:52:23.294186+0000","last_fullsized":"2024-09-20T22:52:23.294186+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:20.955598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:52:20.955598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:32.677353+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":1244,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-09-20T22:52:23.284399+0000","last_change":"2024-09-20T22:51:01.446349+0000","last_active":"2024-09-20T22:52:23.284399+0000","last_peered":"2024-09-20T22:52:23.284399+0000","last_clean":"2024-09-20T22:52:23.284399+0000","last_became_active":"2024-09-20T22:48:05.237765+0000","last_became_peered":"2024-09-20T22:48:05.237765+0000","last_unstale":"2024-09-20T22:52:23.284399+0000","last_undegraded":"2024-09-20T22:52:23.284399+0000","last_fullsized":"2024-09-20T22:52:23.284399+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:01.446278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:10.233487+0000","last_clean_scrub_stamp":"2024-09-20T22:51:01.446278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":841,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-09-20T22:52:23.631141+0000","last_change":"2024-09-20T22:51:29.904430+0000","last_active":"2024-09-20T22:52:23.631141+0000","last_peered":"2024-09-20T22:52:23.631141+0000","last_clean":"2024-09-20T22:52:23.631141+0000","last_became_active":"2024-09-20T22:51:20.297739+0000","last_became_peered":"2024-09-20T22:51:20.297739+0000","last_unstale":"2024-09-20T22:52:23.631141+0000","last_undegraded":"2024-09-20T22:52:23.631141+0000","last_fullsized":"2024-09-20T22:52:23.631141+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:29.904355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:51:29.904355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:36.892277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":1260,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-09-20T22:52:23.281046+0000","last_change":"2024-09-20T22:51:20.275893+0000","last_active":"2024-09-20T22:52:23.281046+0000","last_peered":"2024-09-20T22:52:23.281046+0000","last_clean":"2024-09-20T22:52:23.281046+0000","last_became_active":"2024-09-20T22:51:20.274891+0000","last_became_peered":"2024-09-20T22:51:20.274891+0000","last_unstale":"2024-09-20T22:52:23.281046+0000","last_undegraded":"2024-09-20T22:52:23.281046+0000","last_fullsized":"2024-09-20T22:52:23.281046+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:50:41.281470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:50:41.281470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"625'19","reported_seq":1301,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-09-20T22:52:23.294974+0000","last_change":"2024-09-20T22:52:16.953034+0000","last_active":"2024-09-20T22:52:23.294974+0000","last_peered":"2024-09-20T22:52:23.294974+0000","last_clean":"2024-09-20T22:52:23.294974+0000","last_became_active":"2024-09-20T22:51:20.284937+0000","last_became_peered":"2024-09-20T22:51:20.284937+0000","last_unstale":"2024-09-20T22:52:23.294974+0000","last_undegraded":"2024-09-20T22:52:23.294974+0000","last_fullsized":"2024-09-20T22:52:23.294974+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'19","last_scrub_stamp":"2024-09-20T22:52:16.952955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:52:16.952955+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:27.426198+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":577,"reported_epoch":663,"state":"active+clean","last_fresh":"2024-09-20T22:52:23.283095+0000","last_change":"2024-09-20T22:52:18.698154+0000","last_active":"2024-09-20T22:52:23.283095+0000","last_peered":"2024-09-20T22:52:23.283095+0000","last_clean":"2024-09-20T22:52:23.283095+0000","last_became_active":"2024-09-20T22:47:49.220971+0000","last_became_peered":"2024-09-20T22:47:49.220971+0000","last_unstale":"2024-09-20T22:52:23.283095+0000","last_undegraded":"2024-09-20T22:52:23.283095+0000","last_fullsized":"2024-09-20T22:52:23.283095+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:18.698087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:51.667341+0000","last_clean_scrub_stamp":"2024-09-20T22:52:18.698087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:47.971098+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":591976186,"num_objects":275,"num_object_clones":225,"num_object_copies":825,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":275,"num_whiteouts":14,"num_read":4527,"num_read_kb":4753438,"num_write":3231,"num_write_kb":1199638,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":790,"num_bytes_recovered":519843158,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1307303936,"data_stored":1881202878,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8506,"ondisk_log_size":8506,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85,"ondisk_log_size":85,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1109,"num_read_kb":866,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":312,"num_bytes_recovered":7464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":499712,"data_stored":13224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":739,"ondisk_log_size":739,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":1034,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":4931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":635,"num_read_kb":630,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1186,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":384,"ondisk_log_size":384,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070597,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298564,"kb_used_data":253484,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":93446588,"statfs":{"total":95995035648,"available":95689306112,"internally_reserved":0,"allocated":259567616,"data_stored":334448699,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":46135754},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430347,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":220216,"kb_used_data":170976,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":93524936,"statfs":{"total":95995035648,"available":95769534464,"internally_reserved":0,"allocated":175079424,"data_stored":256658099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691984,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":178100,"kb_used_data":120540,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":93567052,"statfs":{"total":95995035648,"available":95812661248,"internally_reserved":0,"allocated":123432960,"data_stored":181850105,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986325,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":142564,"kb_used_data":111376,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93602588,"statfs":{"total":95995035648,"available":95849050112,"internally_reserved":0,"allocated":114049024,"data_stored":162998770,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182555,"num_pgs":76,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":317644,"kb_used_data":268404,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":93427508,"statfs":{"total":95995035648,"available":95669768192,"internally_reserved":0,"allocated":274845696,"data_stored":389927938,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":50395591},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411488,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":285416,"kb_used_data":223700,"kb_used_omap":1,"kb_used_meta":61694,"kb_avail":93459736,"statfs":{"total":95995035648,"available":95702769664,"internally_reserved":0,"allocated":229068800,"data_stored":320140331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":63175109},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":55834575013,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738538,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":214344,"kb_used_data":169264,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":93530808,"statfs":{"total":95995035648,"available":95775547392,"internally_reserved":0,"allocated":173326336,"data_stored":260619932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":167395328,"data_stored":257025739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":222916608,"data_stored":316291782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":268320768,"data_stored":385742683,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":108130304,"data_stored":159405538,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":116953088,"data_stored":177680360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":170278912,"data_stored":254529955,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":253308928,"data_stored":330526821,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-20T22:52:27.161 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-09-20T22:52:27.161 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-20T22:52:27.161 INFO:tasks.thrashosds.thrasher:Shrinking pool .mgr 2024-09-20T22:52:27.161 INFO:tasks.cephadm.ceph_manager.ceph:contract_pool .mgr by 10 min 8 2024-09-20T22:52:27.161 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph pg dump --format=json 2024-09-20T22:52:27.181 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:52:27.205 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:27.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:27.205 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:52:27.252 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:52:27.270 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:52:27.306 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:52:27.309 INFO:tasks.rados.rados.0.smithi084.stdout:2606: done (0 left) 2024-09-20T22:52:27.309 INFO:tasks.rados.rados.0.smithi084.stdout:2607: write (excl) oid 39 current snap is 296 2024-09-20T22:52:27.309 INFO:tasks.rados.rados.0.smithi084.stdout:2607: seq_num 902 ranges {759914=404252,1566405=433305,2504010=655904} 2024-09-20T22:52:27.316 INFO:tasks.rados.rados.0.smithi084.stdout:2607: writing smithi08477203-39 from 759914 to 1164166 tid 1 2024-09-20T22:52:27.321 INFO:tasks.rados.rados.0.smithi084.stdout:2607: writing smithi08477203-39 from 1566405 to 1999710 tid 2 2024-09-20T22:52:27.328 INFO:tasks.rados.rados.0.smithi084.stdout:2607: writing smithi08477203-39 from 2504010 to 3159914 tid 3 2024-09-20T22:52:27.328 INFO:tasks.rados.rados.0.smithi084.stdout:2608: copy_from oid 26 from oid 1 current snap is 296 2024-09-20T22:52:27.328 INFO:tasks.rados.rados.0.smithi084.stdout:2607: finishing write tid 1 to smithi08477203-39 2024-09-20T22:52:27.328 INFO:tasks.rados.rados.0.smithi084.stdout:2607: finishing write tid 2 to smithi08477203-39 2024-09-20T22:52:27.328 INFO:tasks.rados.rados.0.smithi084.stdout:2609: delete oid 35 current snap is 296 2024-09-20T22:52:27.331 INFO:tasks.rados.rados.0.smithi084.stdout:2609: done (2 left) 2024-09-20T22:52:27.331 INFO:tasks.rados.rados.0.smithi084.stdout:2610: copy_from oid 7 from oid 40 current snap is 296 2024-09-20T22:52:27.331 INFO:tasks.rados.rados.0.smithi084.stdout:2611: snap_create 2024-09-20T22:52:27.332 INFO:tasks.rados.rados.0.smithi084.stdout:2607: finishing write tid 3 to smithi08477203-39 2024-09-20T22:52:27.333 INFO:tasks.rados.rados.0.smithi084.stdout:2607: finishing write tid 4 to smithi08477203-39 2024-09-20T22:52:27.334 INFO:tasks.rados.rados.0.smithi084.stdout:2607: finishing write tid 5 to smithi08477203-39 2024-09-20T22:52:27.335 INFO:tasks.rados.rados.0.smithi084.stdout:2607: oid 39 updating version 0 to 1810 2024-09-20T22:52:27.335 INFO:tasks.rados.rados.0.smithi084.stdout:2607: oid 39 version 1810 is already newer than 1809 2024-09-20T22:52:27.335 INFO:tasks.rados.rados.0.smithi084.stdout:2607: oid 39 version 1810 is already newer than 1808 2024-09-20T22:52:27.335 INFO:tasks.rados.rados.0.smithi084.stdout:2607: oid 39 version 1810 is already newer than 1807 2024-09-20T22:52:27.335 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 39 v 1810 (ObjNum 902 snap 296 seq_num 902) dirty exists 2024-09-20T22:52:27.335 INFO:tasks.rados.rados.0.smithi084.stdout:2607: left oid 39 (ObjNum 902 snap 296 seq_num 902) 2024-09-20T22:52:27.336 INFO:tasks.rados.rados.0.smithi084.stdout:2610: finishing copy_from racing read to smithi08477203-7 2024-09-20T22:52:27.337 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:27.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:27.338 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:52:27.339 INFO:tasks.rados.rados.0.smithi084.stdout:2608: finishing copy_from racing read to smithi08477203-26 2024-09-20T22:52:27.342 INFO:tasks.rados.rados.0.smithi084.stdout:2610: finishing copy_from to smithi08477203-7 2024-09-20T22:52:27.342 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 7 v 1227 (ObjNum 833 snap 269 seq_num 833) dirty exists 2024-09-20T22:52:27.346 INFO:tasks.rados.rados.0.smithi084.stdout:2608: finishing copy_from to smithi08477203-26 2024-09-20T22:52:27.346 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 26 v 1228 (ObjNum 809 snap 262 seq_num 809) dirty exists 2024-09-20T22:52:27.381 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:27 smithi084 ceph-mon[40024]: pgmap v900: 158 pgs: 1 active+clean+snaptrim, 157 active+clean; 565 MiB data, 1.6 GiB used, 624 GiB / 626 GiB avail; 1.7 KiB/s rd, 627 KiB/s wr, 5 op/s; 13 MiB/s, 5 objects/s recovering 2024-09-20T22:52:27.382 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:27 smithi084 ceph-mon[40024]: osdmap e666: 8 total, 8 up, 7 in 2024-09-20T22:52:27.382 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:27 smithi084 ceph-mon[29774]: pgmap v900: 158 pgs: 1 active+clean+snaptrim, 157 active+clean; 565 MiB data, 1.6 GiB used, 624 GiB / 626 GiB avail; 1.7 KiB/s rd, 627 KiB/s wr, 5 op/s; 13 MiB/s, 5 objects/s recovering 2024-09-20T22:52:27.382 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:27 smithi084 ceph-mon[29774]: osdmap e666: 8 total, 8 up, 7 in 2024-09-20T22:52:27.438 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:52:27.461 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:27.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:27.462 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:52:27.562 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:52:27.583 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:27.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:27.584 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:52:27.685 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:52:27.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:27.707 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:27.707 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:52:27.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:27 smithi154 ceph-mon[33264]: pgmap v900: 158 pgs: 1 active+clean+snaptrim, 157 active+clean; 565 MiB data, 1.6 GiB used, 624 GiB / 626 GiB avail; 1.7 KiB/s rd, 627 KiB/s wr, 5 op/s; 13 MiB/s, 5 objects/s recovering 2024-09-20T22:52:27.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:27 smithi154 ceph-mon[33264]: osdmap e666: 8 total, 8 up, 7 in 2024-09-20T22:52:27.780 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-20T22:52:27.780 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:52:27.808 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:52:27.832 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:27.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:27.833 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:52:27.933 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:52:27.956 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:27.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:27.956 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:52:28.057 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:52:28.078 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:28.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:28.079 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:52:28.180 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:52:28.200 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:28.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:28.201 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:52:28.277 INFO:tasks.rados.rados.0.smithi084.stdout:2607: done (3 left) 2024-09-20T22:52:28.277 INFO:tasks.rados.rados.0.smithi084.stdout:2608: done (2 left) 2024-09-20T22:52:28.277 INFO:tasks.rados.rados.0.smithi084.stdout:2610: done (1 left) 2024-09-20T22:52:28.277 INFO:tasks.rados.rados.0.smithi084.stdout:2611: done (0 left) 2024-09-20T22:52:28.277 INFO:tasks.rados.rados.0.smithi084.stdout:2612: delete oid 10 current snap is 297 2024-09-20T22:52:28.286 INFO:tasks.rados.rados.0.smithi084.stdout:2612: done (0 left) 2024-09-20T22:52:28.286 INFO:tasks.rados.rados.0.smithi084.stdout:2613: write oid 39 current snap is 297 2024-09-20T22:52:28.286 INFO:tasks.rados.rados.0.smithi084.stdout:2613: seq_num 905 ranges {410335=746352,1741851=696985,3102632=107703} 2024-09-20T22:52:28.293 INFO:tasks.rados.rados.0.smithi084.stdout:2613: writing smithi08477203-39 from 410335 to 1156687 tid 1 2024-09-20T22:52:28.300 INFO:tasks.rados.rados.0.smithi084.stdout:2613: writing smithi08477203-39 from 1741851 to 2438836 tid 2 2024-09-20T22:52:28.302 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:52:28.304 INFO:tasks.rados.rados.0.smithi084.stdout:2613: writing smithi08477203-39 from 3102632 to 3210335 tid 3 2024-09-20T22:52:28.304 INFO:tasks.rados.rados.0.smithi084.stdout:2614: rollback oid 28 current snap is 297 2024-09-20T22:52:28.304 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 28 to 261 2024-09-20T22:52:28.305 INFO:tasks.rados.rados.0.smithi084.stdout:2615: write oid 26 current snap is 297 2024-09-20T22:52:28.305 INFO:tasks.rados.rados.0.smithi084.stdout:2615: seq_num 906 ranges {427142=421601,1400333=440643,2613702=450326} 2024-09-20T22:52:28.309 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:52:28 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: ::ffff:172.21.15.154 - - [20/Sep/2024:22:52:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-20T22:52:28.309 INFO:tasks.rados.rados.0.smithi084.stdout:2615: writing smithi08477203-26 from 427142 to 848743 tid 1 2024-09-20T22:52:28.315 INFO:tasks.rados.rados.0.smithi084.stdout:2615: writing smithi08477203-26 from 1400333 to 1840976 tid 2 2024-09-20T22:52:28.323 INFO:tasks.rados.rados.0.smithi084.stdout:2615: writing smithi08477203-26 from 2613702 to 3064028 tid 3 2024-09-20T22:52:28.323 INFO:tasks.rados.rados.0.smithi084.stdout:2616: write oid 24 current snap is 297 2024-09-20T22:52:28.323 INFO:tasks.rados.rados.0.smithi084.stdout:2616: seq_num 907 ranges {443949=496850,1458815=584301,2641125=602824} 2024-09-20T22:52:28.328 INFO:tasks.rados.rados.0.smithi084.stdout:2616: writing smithi08477203-24 from 443949 to 940799 tid 1 2024-09-20T22:52:28.330 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:28.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:28.331 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:52:28.335 INFO:tasks.rados.rados.0.smithi084.stdout:2616: writing smithi08477203-24 from 1458815 to 2043116 tid 2 2024-09-20T22:52:28.341 INFO:tasks.rados.rados.0.smithi084.stdout:2616: writing smithi08477203-24 from 2641125 to 3243949 tid 3 2024-09-20T22:52:28.341 INFO:tasks.rados.rados.0.smithi084.stdout:2617: write (excl) oid 3 current snap is 297 2024-09-20T22:52:28.341 INFO:tasks.rados.rados.0.smithi084.stdout:2617: seq_num 908 ranges {460756=572099,1517297=444312,2668548=472870} 2024-09-20T22:52:28.347 INFO:tasks.rados.rados.0.smithi084.stdout:2617: writing smithi08477203-3 from 460756 to 1032855 tid 1 2024-09-20T22:52:28.352 INFO:tasks.rados.rados.0.smithi084.stdout:2617: writing smithi08477203-3 from 1517297 to 1961609 tid 2 2024-09-20T22:52:28.359 INFO:tasks.rados.rados.0.smithi084.stdout:2617: writing smithi08477203-3 from 2668548 to 3141418 tid 3 2024-09-20T22:52:28.360 INFO:tasks.rados.rados.0.smithi084.stdout:2618: read oid 33 snap 235 2024-09-20T22:52:28.360 INFO:tasks.rados.rados.0.smithi084.stdout:2618: expect (ObjNum 738 snap 235 seq_num 738) 2024-09-20T22:52:28.360 INFO:tasks.rados.rados.0.smithi084.stdout:2619: rollback oid 14 current snap is 297 2024-09-20T22:52:28.360 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 14 to 189 2024-09-20T22:52:28.360 INFO:tasks.rados.rados.0.smithi084.stdout:2613: finishing write tid 1 to smithi08477203-39 2024-09-20T22:52:28.360 INFO:tasks.rados.rados.0.smithi084.stdout:2614: finishing rollback tid 0 to smithi08477203-28 2024-09-20T22:52:28.360 INFO:tasks.rados.rados.0.smithi084.stdout:2620: write (excl) oid 32 current snap is 297 2024-09-20T22:52:28.360 INFO:tasks.rados.rados.0.smithi084.stdout:2613: finishing write tid 2 to smithi08477203-39 2024-09-20T22:52:28.360 INFO:tasks.rados.rados.0.smithi084.stdout:2614: finishing rollback tid 1 to smithi08477203-28 2024-09-20T22:52:28.360 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 28 v 2003 (ObjNum 273 snap 0 seq_num 709484942) dirty dne 2024-09-20T22:52:28.360 INFO:tasks.rados.rados.0.smithi084.stdout:2613: finishing write tid 3 to smithi08477203-39 2024-09-20T22:52:28.360 INFO:tasks.rados.rados.0.smithi084.stdout:2613: finishing write tid 4 to smithi08477203-39 2024-09-20T22:52:28.360 INFO:tasks.rados.rados.0.smithi084.stdout:2613: finishing write tid 5 to smithi08477203-39 2024-09-20T22:52:28.360 INFO:tasks.rados.rados.0.smithi084.stdout:2613: oid 39 updating version 0 to 1812 2024-09-20T22:52:28.361 INFO:tasks.rados.rados.0.smithi084.stdout:2613: oid 39 updating version 1812 to 1813 2024-09-20T22:52:28.361 INFO:tasks.rados.rados.0.smithi084.stdout:2613: oid 39 updating version 1813 to 1815 2024-09-20T22:52:28.361 INFO:tasks.rados.rados.0.smithi084.stdout:2613: oid 39 version 1815 is already newer than 1814 2024-09-20T22:52:28.361 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 39 v 1815 (ObjNum 905 snap 297 seq_num 905) dirty exists 2024-09-20T22:52:28.361 INFO:tasks.rados.rados.0.smithi084.stdout:2613: left oid 39 (ObjNum 905 snap 297 seq_num 905) 2024-09-20T22:52:28.361 INFO:tasks.rados.rados.0.smithi084.stdout:2615: finishing write tid 1 to smithi08477203-26 2024-09-20T22:52:28.361 INFO:tasks.rados.rados.0.smithi084.stdout:2620: seq_num 909 ranges {477563=647348,1859426=587970,2979618=297945} 2024-09-20T22:52:28.365 INFO:tasks.rados.rados.0.smithi084.stdout:2620: writing smithi08477203-32 from 477563 to 1124911 tid 1 2024-09-20T22:52:28.372 INFO:tasks.rados.rados.0.smithi084.stdout:2620: writing smithi08477203-32 from 1859426 to 2447396 tid 2 2024-09-20T22:52:28.377 INFO:tasks.rados.rados.0.smithi084.stdout:2620: writing smithi08477203-32 from 2979618 to 3277563 tid 3 2024-09-20T22:52:28.377 INFO:tasks.rados.rados.0.smithi084.stdout:2613: done (7 left) 2024-09-20T22:52:28.377 INFO:tasks.rados.rados.0.smithi084.stdout:2614: done (6 left) 2024-09-20T22:52:28.378 INFO:tasks.rados.rados.0.smithi084.stdout:2621: write oid 31 current snap is 297 2024-09-20T22:52:28.378 INFO:tasks.rados.rados.0.smithi084.stdout:2615: finishing write tid 2 to smithi08477203-26 2024-09-20T22:52:28.378 INFO:tasks.rados.rados.0.smithi084.stdout:2615: finishing write tid 3 to smithi08477203-26 2024-09-20T22:52:28.378 INFO:tasks.rados.rados.0.smithi084.stdout:2615: finishing write tid 4 to smithi08477203-26 2024-09-20T22:52:28.378 INFO:tasks.rados.rados.0.smithi084.stdout:2621: seq_num 910 ranges {494370=722597,1917908=447981,3007041=287329} 2024-09-20T22:52:28.384 INFO:tasks.rados.rados.0.smithi084.stdout:2621: writing smithi08477203-31 from 494370 to 1216967 tid 1 2024-09-20T22:52:28.390 INFO:tasks.rados.rados.0.smithi084.stdout:2621: writing smithi08477203-31 from 1917908 to 2365889 tid 2 2024-09-20T22:52:28.396 INFO:tasks.rados.rados.0.smithi084.stdout:2621: writing smithi08477203-31 from 3007041 to 3294370 tid 3 2024-09-20T22:52:28.396 INFO:tasks.rados.rados.0.smithi084.stdout:2622: read oid 19 snap -1 2024-09-20T22:52:28.396 INFO:tasks.rados.rados.0.smithi084.stdout:2622: expect (ObjNum 625 snap 186 seq_num 625) 2024-09-20T22:52:28.396 INFO:tasks.rados.rados.0.smithi084.stdout:2615: finishing write tid 5 to smithi08477203-26 2024-09-20T22:52:28.396 INFO:tasks.rados.rados.0.smithi084.stdout:2615: oid 26 updating version 0 to 1353 2024-09-20T22:52:28.396 INFO:tasks.rados.rados.0.smithi084.stdout:2615: oid 26 updating version 1353 to 1354 2024-09-20T22:52:28.396 INFO:tasks.rados.rados.0.smithi084.stdout:2615: oid 26 version 1354 is already newer than 1352 2024-09-20T22:52:28.397 INFO:tasks.rados.rados.0.smithi084.stdout:2615: oid 26 version 1354 is already newer than 1351 2024-09-20T22:52:28.397 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 26 v 1354 (ObjNum 906 snap 297 seq_num 906) dirty exists 2024-09-20T22:52:28.397 INFO:tasks.rados.rados.0.smithi084.stdout:2615: left oid 26 (ObjNum 906 snap 297 seq_num 906) 2024-09-20T22:52:28.397 INFO:tasks.rados.rados.0.smithi084.stdout:2623: read oid 10 snap 287 2024-09-20T22:52:28.397 INFO:tasks.rados.rados.0.smithi084.stdout:2623: expect deleted 2024-09-20T22:52:28.397 INFO:tasks.rados.rados.0.smithi084.stdout:2615: done (8 left) 2024-09-20T22:52:28.397 INFO:tasks.rados.rados.0.smithi084.stdout:2624: snap_remove snap 274 2024-09-20T22:52:28.432 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:52:28.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:28.456 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:28.456 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:52:28.555 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-20T22:52:28.557 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:52:28.576 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:28 smithi084 ceph-mon[40024]: osdmap e667: 8 total, 8 up, 7 in 2024-09-20T22:52:28.577 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:28 smithi084 ceph-mon[40024]: osdmap e668: 8 total, 8 up, 7 in 2024-09-20T22:52:28.577 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:28 smithi084 ceph-mon[29774]: osdmap e667: 8 total, 8 up, 7 in 2024-09-20T22:52:28.577 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:28 smithi084 ceph-mon[29774]: osdmap e668: 8 total, 8 up, 7 in 2024-09-20T22:52:28.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:28.578 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:28.578 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:52:28.679 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:52:28.702 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:28.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:28.703 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:52:28.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:28 smithi154 ceph-mon[33264]: osdmap e667: 8 total, 8 up, 7 in 2024-09-20T22:52:28.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:28 smithi154 ceph-mon[33264]: osdmap e668: 8 total, 8 up, 7 in 2024-09-20T22:52:28.804 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:52:28.825 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:28.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:28.826 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:52:28.927 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:52:28.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:28.948 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:28.949 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:52:29.049 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:52:29.070 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:29.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:29.071 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:52:29.161 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:52:29.172 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:52:29.191 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-20T22:52:29.197 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:29.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:29.198 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:52:29.299 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:52:29.322 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:29.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:29.323 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:52:29.331 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is set 2024-09-20T22:52:29.341 INFO:tasks.rados.rados.0.smithi084.stdout:2624: done (8 left) 2024-09-20T22:52:29.341 INFO:tasks.rados.rados.0.smithi084.stdout:2625: read oid 37 snap -1 2024-09-20T22:52:29.341 INFO:tasks.rados.rados.0.smithi084.stdout:2625: expect (ObjNum 881 snap 285 seq_num 881) 2024-09-20T22:52:29.341 INFO:tasks.rados.rados.0.smithi084.stdout:2626: snap_create 2024-09-20T22:52:29.342 INFO:tasks.rados.rados.0.smithi084.stdout:2616: finishing write tid 1 to smithi08477203-24 2024-09-20T22:52:29.342 INFO:tasks.rados.rados.0.smithi084.stdout:2616: finishing write tid 2 to smithi08477203-24 2024-09-20T22:52:29.342 INFO:tasks.rados.rados.0.smithi084.stdout:2616: finishing write tid 3 to smithi08477203-24 2024-09-20T22:52:29.342 INFO:tasks.rados.rados.0.smithi084.stdout:2616: finishing write tid 4 to smithi08477203-24 2024-09-20T22:52:29.342 INFO:tasks.rados.rados.0.smithi084.stdout:2616: finishing write tid 5 to smithi08477203-24 2024-09-20T22:52:29.342 INFO:tasks.rados.rados.0.smithi084.stdout:2616: oid 24 updating version 0 to 825 2024-09-20T22:52:29.342 INFO:tasks.rados.rados.0.smithi084.stdout:2616: oid 24 version 825 is already newer than 824 2024-09-20T22:52:29.342 INFO:tasks.rados.rados.0.smithi084.stdout:2616: oid 24 updating version 825 to 826 2024-09-20T22:52:29.342 INFO:tasks.rados.rados.0.smithi084.stdout:2616: oid 24 version 826 is already newer than 823 2024-09-20T22:52:29.342 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 24 v 826 (ObjNum 907 snap 297 seq_num 907) dirty exists 2024-09-20T22:52:29.343 INFO:tasks.rados.rados.0.smithi084.stdout:2616: left oid 24 (ObjNum 907 snap 297 seq_num 907) 2024-09-20T22:52:29.343 INFO:tasks.rados.rados.0.smithi084.stdout:2617: finishing write tid 1 to smithi08477203-3 2024-09-20T22:52:29.343 INFO:tasks.rados.rados.0.smithi084.stdout:2617: finishing write tid 2 to smithi08477203-3 2024-09-20T22:52:29.343 INFO:tasks.rados.rados.0.smithi084.stdout:2619: finishing rollback tid 1 to smithi08477203-14 2024-09-20T22:52:29.343 INFO:tasks.rados.rados.0.smithi084.stdout:2617: finishing write tid 3 to smithi08477203-3 2024-09-20T22:52:29.343 INFO:tasks.rados.rados.0.smithi084.stdout:2619: finishing rollback tid 2 to smithi08477203-14 2024-09-20T22:52:29.343 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 14 v 1245 (ObjNum 618 snap 184 seq_num 618) dirty exists 2024-09-20T22:52:29.343 INFO:tasks.rados.rados.0.smithi084.stdout:2617: finishing write tid 4 to smithi08477203-3 2024-09-20T22:52:29.343 INFO:tasks.rados.rados.0.smithi084.stdout:2617: finishing write tid 5 to smithi08477203-3 2024-09-20T22:52:29.343 INFO:tasks.rados.rados.0.smithi084.stdout:2617: oid 3 updating version 0 to 1821 2024-09-20T22:52:29.343 INFO:tasks.rados.rados.0.smithi084.stdout:2617: oid 3 version 1821 is already newer than 1820 2024-09-20T22:52:29.343 INFO:tasks.rados.rados.0.smithi084.stdout:2617: oid 3 version 1821 is already newer than 1819 2024-09-20T22:52:29.343 INFO:tasks.rados.rados.0.smithi084.stdout:2617: oid 3 updating version 1821 to 1822 2024-09-20T22:52:29.343 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 3 v 1822 (ObjNum 908 snap 297 seq_num 908) dirty exists 2024-09-20T22:52:29.343 INFO:tasks.rados.rados.0.smithi084.stdout:2617: left oid 3 (ObjNum 908 snap 297 seq_num 908) 2024-09-20T22:52:29.344 INFO:tasks.rados.rados.0.smithi084.stdout:2618: expect (ObjNum 738 snap 235 seq_num 738) 2024-09-20T22:52:29.424 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:52:29.427 INFO:tasks.rados.rados.0.smithi084.stdout:2620: finishing write tid 1 to smithi08477203-32 2024-09-20T22:52:29.427 INFO:tasks.rados.rados.0.smithi084.stdout:2620: finishing write tid 2 to smithi08477203-32 2024-09-20T22:52:29.427 INFO:tasks.rados.rados.0.smithi084.stdout:2620: finishing write tid 3 to smithi08477203-32 2024-09-20T22:52:29.427 INFO:tasks.rados.rados.0.smithi084.stdout:2620: finishing write tid 4 to smithi08477203-32 2024-09-20T22:52:29.427 INFO:tasks.rados.rados.0.smithi084.stdout:2620: finishing write tid 5 to smithi08477203-32 2024-09-20T22:52:29.427 INFO:tasks.rados.rados.0.smithi084.stdout:2620: oid 32 updating version 0 to 748 2024-09-20T22:52:29.427 INFO:tasks.rados.rados.0.smithi084.stdout:2620: oid 32 updating version 748 to 750 2024-09-20T22:52:29.427 INFO:tasks.rados.rados.0.smithi084.stdout:2620: oid 32 version 750 is already newer than 749 2024-09-20T22:52:29.427 INFO:tasks.rados.rados.0.smithi084.stdout:2620: oid 32 updating version 750 to 751 2024-09-20T22:52:29.427 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 32 v 751 (ObjNum 909 snap 297 seq_num 909) dirty exists 2024-09-20T22:52:29.427 INFO:tasks.rados.rados.0.smithi084.stdout:2620: left oid 32 (ObjNum 909 snap 297 seq_num 909) 2024-09-20T22:52:29.428 INFO:tasks.rados.rados.0.smithi084.stdout:2621: finishing write tid 1 to smithi08477203-31 2024-09-20T22:52:29.428 INFO:tasks.rados.rados.0.smithi084.stdout:2621: finishing write tid 2 to smithi08477203-31 2024-09-20T22:52:29.428 INFO:tasks.rados.rados.0.smithi084.stdout:2621: finishing write tid 3 to smithi08477203-31 2024-09-20T22:52:29.428 INFO:tasks.rados.rados.0.smithi084.stdout:2621: finishing write tid 4 to smithi08477203-31 2024-09-20T22:52:29.428 INFO:tasks.rados.rados.0.smithi084.stdout:2621: finishing write tid 5 to smithi08477203-31 2024-09-20T22:52:29.428 INFO:tasks.rados.rados.0.smithi084.stdout:2621: oid 31 updating version 0 to 1358 2024-09-20T22:52:29.428 INFO:tasks.rados.rados.0.smithi084.stdout:2621: oid 31 version 1358 is already newer than 1356 2024-09-20T22:52:29.428 INFO:tasks.rados.rados.0.smithi084.stdout:2621: oid 31 version 1358 is already newer than 1355 2024-09-20T22:52:29.428 INFO:tasks.rados.rados.0.smithi084.stdout:2621: oid 31 version 1358 is already newer than 1357 2024-09-20T22:52:29.428 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 31 v 1358 (ObjNum 910 snap 297 seq_num 910) dirty exists 2024-09-20T22:52:29.428 INFO:tasks.rados.rados.0.smithi084.stdout:2621: left oid 31 (ObjNum 910 snap 297 seq_num 910) 2024-09-20T22:52:29.428 INFO:tasks.rados.rados.0.smithi084.stdout:2622: expect (ObjNum 625 snap 186 seq_num 625) 2024-09-20T22:52:29.445 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:29.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:29.446 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:52:29.546 INFO:tasks.rados.rados.0.smithi084.stdout:2625: expect (ObjNum 881 snap 285 seq_num 881) 2024-09-20T22:52:29.546 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:52:29.567 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:29.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:29.568 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:52:29.614 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:29 smithi084 ceph-mon[40024]: pgmap v903: 158 pgs: 1 active+clean+snaptrim, 157 active+clean; 565 MiB data, 1.6 GiB used, 624 GiB / 626 GiB avail 2024-09-20T22:52:29.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:29 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/4027864765' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:52:29.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:29 smithi084 ceph-mon[29774]: pgmap v903: 158 pgs: 1 active+clean+snaptrim, 157 active+clean; 565 MiB data, 1.6 GiB used, 624 GiB / 626 GiB avail 2024-09-20T22:52:29.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:29 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/4027864765' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:52:29.669 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:52:29.691 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:29.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:29.692 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:52:29.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:29 smithi154 ceph-mon[33264]: pgmap v903: 158 pgs: 1 active+clean+snaptrim, 157 active+clean; 565 MiB data, 1.6 GiB used, 624 GiB / 626 GiB avail 2024-09-20T22:52:29.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:29 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/4027864765' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:52:29.754 INFO:journalctl@ceph.iscsi.iscsi.a.smithi154.stdout:Sep 20 22:52:29 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-iscsi-iscsi-a[59730]: debug there is no tcmu-runner data available 2024-09-20T22:52:29.793 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:52:29.815 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:29.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:29.816 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:52:29.916 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:52:29.937 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:29.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:29.938 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:52:30.024 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-20T22:52:30.026 INFO:teuthology.orchestra.run.smithi084.stderr:dumped all 2024-09-20T22:52:30.039 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:52:30.063 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:30.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:30.064 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:52:30.165 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:52:30.186 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:30.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:30.187 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:52:30.288 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:52:30.314 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:30.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:30.315 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:52:30.346 INFO:tasks.rados.rados.0.smithi084.stdout:2616: done (9 left) 2024-09-20T22:52:30.346 INFO:tasks.rados.rados.0.smithi084.stdout:2617: done (8 left) 2024-09-20T22:52:30.346 INFO:tasks.rados.rados.0.smithi084.stdout:2618: done (7 left) 2024-09-20T22:52:30.346 INFO:tasks.rados.rados.0.smithi084.stdout:2619: done (6 left) 2024-09-20T22:52:30.346 INFO:tasks.rados.rados.0.smithi084.stdout:2620: done (5 left) 2024-09-20T22:52:30.347 INFO:tasks.rados.rados.0.smithi084.stdout:2621: done (4 left) 2024-09-20T22:52:30.347 INFO:tasks.rados.rados.0.smithi084.stdout:2622: done (3 left) 2024-09-20T22:52:30.347 INFO:tasks.rados.rados.0.smithi084.stdout:2623: done (2 left) 2024-09-20T22:52:30.347 INFO:tasks.rados.rados.0.smithi084.stdout:2625: done (1 left) 2024-09-20T22:52:30.347 INFO:tasks.rados.rados.0.smithi084.stdout:2626: done (0 left) 2024-09-20T22:52:30.347 INFO:tasks.rados.rados.0.smithi084.stdout:2627: copy_from oid 40 from oid 5 current snap is 298 2024-09-20T22:52:30.347 INFO:tasks.rados.rados.0.smithi084.stdout:2628: delete oid 37 current snap is 298 2024-09-20T22:52:30.350 INFO:tasks.rados.rados.0.smithi084.stdout:2628: done (1 left) 2024-09-20T22:52:30.350 INFO:tasks.rados.rados.0.smithi084.stdout:2629: snap_remove snap 295 2024-09-20T22:52:30.416 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:52:30.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:30.440 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:30.440 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:52:30.541 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:52:30.564 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:30.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:30.565 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:52:30.614 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:30 smithi084 ceph-mon[29774]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:52:30.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:30 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/4027864765' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T22:52:30.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:30 smithi084 ceph-mon[29774]: osdmap e669: 8 total, 8 up, 7 in 2024-09-20T22:52:30.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:30 smithi084 ceph-mon[29774]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:52:30.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:30 smithi084 ceph-mon[40024]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:52:30.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:30 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/4027864765' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T22:52:30.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:30 smithi084 ceph-mon[40024]: osdmap e669: 8 total, 8 up, 7 in 2024-09-20T22:52:30.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:30 smithi084 ceph-mon[40024]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:52:30.665 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:52:30.689 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:30.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:30.690 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:52:30.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:30 smithi154 ceph-mon[33264]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:52:30.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:30 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/4027864765' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T22:52:30.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:30 smithi154 ceph-mon[33264]: osdmap e669: 8 total, 8 up, 7 in 2024-09-20T22:52:30.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:30 smithi154 ceph-mon[33264]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:52:30.790 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:52:30.815 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:30.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:30.816 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:52:30.916 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:52:30.939 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:30.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:30.940 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:52:31.040 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:52:31.062 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:31.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:31.063 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:52:31.153 INFO:teuthology.orchestra.run.smithi084.stdout:{"pg_ready":true,"pg_map":{"version":906,"stamp":"2024-09-20T22:52:29.787995+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":618186719,"num_objects":483,"num_object_clones":234,"num_object_copies":1449,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":483,"num_whiteouts":12,"num_read":6334,"num_read_kb":4771351,"num_write":4009,"num_write_kb":1221718,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1142,"num_bytes_recovered":520180548,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9908,"ondisk_log_size":9908,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":474,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":1708168,"kb_used_data":1360672,"kb_used_omap":10,"kb_used_meta":347317,"kb_avail":654507896,"statfs":{"total":671965249536,"available":670216085504,"internally_reserved":0,"allocated":1393328128,"data_stored":1964797292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11157,"internal_metadata":355652715},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":25614026,"num_objects":9,"num_object_clones":9,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":-2,"num_read":39,"num_read_kb":16393,"num_write":56,"num_write_kb":22035,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000878"},"pg_stats":[{"pgid":"3.1f","version":"86'2","reported_seq":591,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-09-20T22:52:28.274443+0000","last_change":"2024-09-20T22:52:21.630419+0000","last_active":"2024-09-20T22:52:28.274443+0000","last_peered":"2024-09-20T22:52:28.274443+0000","last_clean":"2024-09-20T22:52:28.274443+0000","last_became_active":"2024-09-20T22:47:49.222114+0000","last_became_peered":"2024-09-20T22:47:49.222114+0000","last_unstale":"2024-09-20T22:52:28.274443+0000","last_undegraded":"2024-09-20T22:52:28.274443+0000","last_fullsized":"2024-09-20T22:52:28.274443+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'2","last_scrub_stamp":"2024-09-20T22:52:21.630304+0000","last_deep_scrub":"86'2","last_deep_scrub_stamp":"2024-09-20T22:42:39.664421+0000","last_clean_scrub_stamp":"2024-09-20T22:52:21.630304+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:38.958732+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":436,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"625'15","reported_seq":589,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-09-20T22:52:28.273952+0000","last_change":"2024-09-20T22:51:51.813358+0000","last_active":"2024-09-20T22:52:28.273952+0000","last_peered":"2024-09-20T22:52:28.273952+0000","last_clean":"2024-09-20T22:52:28.273952+0000","last_became_active":"2024-09-20T22:51:20.286651+0000","last_became_peered":"2024-09-20T22:51:20.286651+0000","last_unstale":"2024-09-20T22:52:28.273952+0000","last_undegraded":"2024-09-20T22:52:28.273952+0000","last_fullsized":"2024-09-20T22:52:28.273952+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'15","last_scrub_stamp":"2024-09-20T22:51:51.813269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:51:51.813269+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:09.354334+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":148,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-09-20T22:52:28.282309+0000","last_change":"2024-09-20T22:51:20.291625+0000","last_active":"2024-09-20T22:52:28.282309+0000","last_peered":"2024-09-20T22:52:28.282309+0000","last_clean":"2024-09-20T22:52:28.282309+0000","last_became_active":"2024-09-20T22:51:20.291272+0000","last_became_peered":"2024-09-20T22:51:20.291272+0000","last_unstale":"2024-09-20T22:52:28.282309+0000","last_undegraded":"2024-09-20T22:52:28.282309+0000","last_fullsized":"2024-09-20T22:52:28.282309+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:50:12.718677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:50:12.718677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":560,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-09-20T22:52:28.274371+0000","last_change":"2024-09-20T22:51:36.730087+0000","last_active":"2024-09-20T22:52:28.274371+0000","last_peered":"2024-09-20T22:52:28.274371+0000","last_clean":"2024-09-20T22:52:28.274371+0000","last_became_active":"2024-09-20T22:51:20.295608+0000","last_became_peered":"2024-09-20T22:51:20.295608+0000","last_unstale":"2024-09-20T22:52:28.274371+0000","last_undegraded":"2024-09-20T22:52:28.274371+0000","last_fullsized":"2024-09-20T22:52:28.274371+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:36.730015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:51:36.730015+0000","last_clean_scrub_stamp":"2024-09-20T22:51:36.730015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:41.980409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":1234,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-09-20T22:52:28.274958+0000","last_change":"2024-09-20T22:51:01.404127+0000","last_active":"2024-09-20T22:52:28.274958+0000","last_peered":"2024-09-20T22:52:28.274958+0000","last_clean":"2024-09-20T22:52:28.274958+0000","last_became_active":"2024-09-20T22:41:44.832099+0000","last_became_peered":"2024-09-20T22:41:44.832099+0000","last_unstale":"2024-09-20T22:52:28.274958+0000","last_undegraded":"2024-09-20T22:52:28.274958+0000","last_fullsized":"2024-09-20T22:52:28.274958+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:01.404063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:51:01.404063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:28.385187+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":1239,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-09-20T22:52:28.277789+0000","last_change":"2024-09-20T22:51:12.418041+0000","last_active":"2024-09-20T22:52:28.277789+0000","last_peered":"2024-09-20T22:52:28.277789+0000","last_clean":"2024-09-20T22:52:28.277789+0000","last_became_active":"2024-09-20T22:41:38.157298+0000","last_became_peered":"2024-09-20T22:41:38.157298+0000","last_unstale":"2024-09-20T22:52:28.277789+0000","last_undegraded":"2024-09-20T22:52:28.277789+0000","last_fullsized":"2024-09-20T22:52:28.277789+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:12.417920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:51:12.417920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:37.156764+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"625'27","reported_seq":1321,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-09-20T22:52:28.274781+0000","last_change":"2024-09-20T22:51:08.441927+0000","last_active":"2024-09-20T22:52:28.274781+0000","last_peered":"2024-09-20T22:52:28.274781+0000","last_clean":"2024-09-20T22:52:28.274781+0000","last_became_active":"2024-09-20T22:47:49.218798+0000","last_became_peered":"2024-09-20T22:47:49.218798+0000","last_unstale":"2024-09-20T22:52:28.274781+0000","last_undegraded":"2024-09-20T22:52:28.274781+0000","last_fullsized":"2024-09-20T22:52:28.274781+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-09-20T22:51:08.441788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:51:08.441788+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:31.369783+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":560,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-09-20T22:52:28.273883+0000","last_change":"2024-09-20T22:52:14.594895+0000","last_active":"2024-09-20T22:52:28.273883+0000","last_peered":"2024-09-20T22:52:28.273883+0000","last_clean":"2024-09-20T22:52:28.273883+0000","last_became_active":"2024-09-20T22:51:20.287060+0000","last_became_peered":"2024-09-20T22:51:20.287060+0000","last_unstale":"2024-09-20T22:52:28.273883+0000","last_undegraded":"2024-09-20T22:52:28.273883+0000","last_fullsized":"2024-09-20T22:52:28.273883+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:14.594823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:52:14.594823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:30.113157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":1259,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-09-20T22:52:28.283158+0000","last_change":"2024-09-20T22:51:09.684125+0000","last_active":"2024-09-20T22:52:28.283158+0000","last_peered":"2024-09-20T22:52:28.283158+0000","last_clean":"2024-09-20T22:52:28.283158+0000","last_became_active":"2024-09-20T22:48:05.242059+0000","last_became_peered":"2024-09-20T22:48:05.242059+0000","last_unstale":"2024-09-20T22:52:28.283158+0000","last_undegraded":"2024-09-20T22:52:28.283158+0000","last_fullsized":"2024-09-20T22:52:28.283158+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:09.684053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:46:00.999939+0000","last_clean_scrub_stamp":"2024-09-20T22:51:09.684053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:35.986257+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"625'15","reported_seq":573,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-09-20T22:52:28.274538+0000","last_change":"2024-09-20T22:51:30.710329+0000","last_active":"2024-09-20T22:52:28.274538+0000","last_peered":"2024-09-20T22:52:28.274538+0000","last_clean":"2024-09-20T22:52:28.274538+0000","last_became_active":"2024-09-20T22:48:05.233429+0000","last_became_peered":"2024-09-20T22:48:05.233429+0000","last_unstale":"2024-09-20T22:52:28.274538+0000","last_undegraded":"2024-09-20T22:52:28.274538+0000","last_fullsized":"2024-09-20T22:52:28.274538+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-09-20T22:51:30.710245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:51:30.710245+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:39.082734+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":1251,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-09-20T22:52:28.282969+0000","last_change":"2024-09-20T22:51:33.977328+0000","last_active":"2024-09-20T22:52:28.282969+0000","last_peered":"2024-09-20T22:52:28.282969+0000","last_clean":"2024-09-20T22:52:28.282969+0000","last_became_active":"2024-09-20T22:47:49.227379+0000","last_became_peered":"2024-09-20T22:47:49.227379+0000","last_unstale":"2024-09-20T22:52:28.282969+0000","last_undegraded":"2024-09-20T22:52:28.282969+0000","last_fullsized":"2024-09-20T22:52:28.282969+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:33.977257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:51:33.977257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:37.924731+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":592,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-09-20T22:52:28.272452+0000","last_change":"2024-09-20T22:51:35.899942+0000","last_active":"2024-09-20T22:52:28.272452+0000","last_peered":"2024-09-20T22:52:28.272452+0000","last_clean":"2024-09-20T22:52:28.272452+0000","last_became_active":"2024-09-20T22:51:20.277465+0000","last_became_peered":"2024-09-20T22:51:20.277465+0000","last_unstale":"2024-09-20T22:52:28.272452+0000","last_undegraded":"2024-09-20T22:52:28.272452+0000","last_fullsized":"2024-09-20T22:52:28.272452+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:35.899860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:49:08.664407+0000","last_clean_scrub_stamp":"2024-09-20T22:51:35.899860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:38.548132+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":1265,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-09-20T22:52:28.282751+0000","last_change":"2024-09-20T22:51:20.298096+0000","last_active":"2024-09-20T22:52:28.282751+0000","last_peered":"2024-09-20T22:52:28.282751+0000","last_clean":"2024-09-20T22:52:28.282751+0000","last_became_active":"2024-09-20T22:51:20.297787+0000","last_became_peered":"2024-09-20T22:51:20.297787+0000","last_unstale":"2024-09-20T22:52:28.282751+0000","last_undegraded":"2024-09-20T22:52:28.282751+0000","last_fullsized":"2024-09-20T22:52:28.282751+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:50:17.735354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:50:17.735354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"625'9","reported_seq":575,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-09-20T22:52:28.274891+0000","last_change":"2024-09-20T22:51:20.301034+0000","last_active":"2024-09-20T22:52:28.274891+0000","last_peered":"2024-09-20T22:52:28.274891+0000","last_clean":"2024-09-20T22:52:28.274891+0000","last_became_active":"2024-09-20T22:51:20.285795+0000","last_became_peered":"2024-09-20T22:51:20.285795+0000","last_unstale":"2024-09-20T22:52:28.274891+0000","last_undegraded":"2024-09-20T22:52:28.274891+0000","last_fullsized":"2024-09-20T22:52:28.274891+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"181'7","last_scrub_stamp":"2024-09-20T22:51:05.899205+0000","last_deep_scrub":"181'7","last_deep_scrub_stamp":"2024-09-20T22:48:41.049381+0000","last_clean_scrub_stamp":"2024-09-20T22:51:05.899205+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:31.564278+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":9,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":1279,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-09-20T22:52:28.315229+0000","last_change":"2024-09-20T22:52:16.705733+0000","last_active":"2024-09-20T22:52:28.315229+0000","last_peered":"2024-09-20T22:52:28.315229+0000","last_clean":"2024-09-20T22:52:28.315229+0000","last_became_active":"2024-09-20T22:51:20.285368+0000","last_became_peered":"2024-09-20T22:51:20.285368+0000","last_unstale":"2024-09-20T22:52:28.315229+0000","last_undegraded":"2024-09-20T22:52:28.315229+0000","last_fullsized":"2024-09-20T22:52:28.315229+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:16.705645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:47:40.361958+0000","last_clean_scrub_stamp":"2024-09-20T22:52:16.705645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:40.490856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":1244,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-09-20T22:52:28.282408+0000","last_change":"2024-09-20T22:51:20.279625+0000","last_active":"2024-09-20T22:52:28.282408+0000","last_peered":"2024-09-20T22:52:28.282408+0000","last_clean":"2024-09-20T22:52:28.282408+0000","last_became_active":"2024-09-20T22:51:20.279307+0000","last_became_peered":"2024-09-20T22:51:20.279307+0000","last_unstale":"2024-09-20T22:52:28.282408+0000","last_undegraded":"2024-09-20T22:52:28.282408+0000","last_fullsized":"2024-09-20T22:52:28.282408+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:04.636900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:51:04.636900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":147,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-09-20T22:52:28.274660+0000","last_change":"2024-09-20T22:51:20.282749+0000","last_active":"2024-09-20T22:52:28.274660+0000","last_peered":"2024-09-20T22:52:28.274660+0000","last_clean":"2024-09-20T22:52:28.274660+0000","last_became_active":"2024-09-20T22:51:20.282393+0000","last_became_peered":"2024-09-20T22:51:20.282393+0000","last_unstale":"2024-09-20T22:52:28.274660+0000","last_undegraded":"2024-09-20T22:52:28.274660+0000","last_fullsized":"2024-09-20T22:52:28.274660+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:50:16.816835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:50:16.816835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:31.564154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":584,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-09-20T22:52:28.273330+0000","last_change":"2024-09-20T22:49:53.933673+0000","last_active":"2024-09-20T22:52:28.273330+0000","last_peered":"2024-09-20T22:52:28.273330+0000","last_clean":"2024-09-20T22:52:28.273330+0000","last_became_active":"2024-09-20T22:48:05.233687+0000","last_became_peered":"2024-09-20T22:48:05.233687+0000","last_unstale":"2024-09-20T22:52:28.273330+0000","last_undegraded":"2024-09-20T22:52:28.273330+0000","last_fullsized":"2024-09-20T22:52:28.273330+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:53.933601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:49:53.933601+0000","last_clean_scrub_stamp":"2024-09-20T22:49:53.933601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:28.623022+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"625'34","reported_seq":1330,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-09-20T22:52:28.276095+0000","last_change":"2024-09-20T22:51:20.289052+0000","last_active":"2024-09-20T22:52:28.276095+0000","last_peered":"2024-09-20T22:52:28.276095+0000","last_clean":"2024-09-20T22:52:28.276095+0000","last_became_active":"2024-09-20T22:51:20.276338+0000","last_became_peered":"2024-09-20T22:51:20.276338+0000","last_unstale":"2024-09-20T22:52:28.276095+0000","last_undegraded":"2024-09-20T22:52:28.276095+0000","last_fullsized":"2024-09-20T22:52:28.276095+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'22","last_scrub_stamp":"2024-09-20T22:51:08.280738+0000","last_deep_scrub":"113'22","last_deep_scrub_stamp":"2024-09-20T22:43:12.191878+0000","last_clean_scrub_stamp":"2024-09-20T22:51:08.280738+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:33.054284+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":859,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-09-20T22:52:28.274175+0000","last_change":"2024-09-20T22:51:20.295989+0000","last_active":"2024-09-20T22:52:28.274175+0000","last_peered":"2024-09-20T22:52:28.274175+0000","last_clean":"2024-09-20T22:52:28.274175+0000","last_became_active":"2024-09-20T22:51:20.295564+0000","last_became_peered":"2024-09-20T22:51:20.295564+0000","last_unstale":"2024-09-20T22:52:28.274175+0000","last_undegraded":"2024-09-20T22:52:28.274175+0000","last_fullsized":"2024-09-20T22:52:28.274175+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:50:40.784154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:50:40.784154+0000","last_clean_scrub_stamp":"2024-09-20T22:50:40.784154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:31.564170+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":1227,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-09-20T22:52:28.277215+0000","last_change":"2024-09-20T22:51:13.385618+0000","last_active":"2024-09-20T22:52:28.277215+0000","last_peered":"2024-09-20T22:52:28.277215+0000","last_clean":"2024-09-20T22:52:28.277215+0000","last_became_active":"2024-09-20T22:41:44.832131+0000","last_became_peered":"2024-09-20T22:41:44.832131+0000","last_unstale":"2024-09-20T22:52:28.277215+0000","last_undegraded":"2024-09-20T22:52:28.277215+0000","last_fullsized":"2024-09-20T22:52:28.277215+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:13.385520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:45:25.002199+0000","last_clean_scrub_stamp":"2024-09-20T22:51:13.385520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:28.385173+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":560,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-09-20T22:52:28.274977+0000","last_change":"2024-09-20T22:51:20.285019+0000","last_active":"2024-09-20T22:52:28.274977+0000","last_peered":"2024-09-20T22:52:28.274977+0000","last_clean":"2024-09-20T22:52:28.274977+0000","last_became_active":"2024-09-20T22:51:20.284621+0000","last_became_peered":"2024-09-20T22:51:20.284621+0000","last_unstale":"2024-09-20T22:52:28.274977+0000","last_undegraded":"2024-09-20T22:52:28.274977+0000","last_fullsized":"2024-09-20T22:52:28.274977+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:50:41.885844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:50:41.885844+0000","last_clean_scrub_stamp":"2024-09-20T22:50:41.885844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:31.564259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"625'18","reported_seq":1338,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-09-20T22:52:28.277512+0000","last_change":"2024-09-20T22:51:20.300828+0000","last_active":"2024-09-20T22:52:28.277512+0000","last_peered":"2024-09-20T22:52:28.277512+0000","last_clean":"2024-09-20T22:52:28.277512+0000","last_became_active":"2024-09-20T22:51:20.278886+0000","last_became_peered":"2024-09-20T22:51:20.278886+0000","last_unstale":"2024-09-20T22:52:28.277512+0000","last_undegraded":"2024-09-20T22:52:28.277512+0000","last_fullsized":"2024-09-20T22:52:28.277512+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-09-20T22:51:03.380934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:51:03.380934+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:30.819992+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":554,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-09-20T22:52:28.274577+0000","last_change":"2024-09-20T22:50:42.864153+0000","last_active":"2024-09-20T22:52:28.274577+0000","last_peered":"2024-09-20T22:52:28.274577+0000","last_clean":"2024-09-20T22:52:28.274577+0000","last_became_active":"2024-09-20T22:48:05.250319+0000","last_became_peered":"2024-09-20T22:48:05.250319+0000","last_unstale":"2024-09-20T22:52:28.274577+0000","last_undegraded":"2024-09-20T22:52:28.274577+0000","last_fullsized":"2024-09-20T22:52:28.274577+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:50:42.864078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:50:42.864078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:31.564213+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"154'2","reported_seq":1236,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-09-20T22:52:28.314795+0000","last_change":"2024-09-20T22:50:34.375264+0000","last_active":"2024-09-20T22:52:28.314795+0000","last_peered":"2024-09-20T22:52:28.314795+0000","last_clean":"2024-09-20T22:52:28.314795+0000","last_became_active":"2024-09-20T22:41:44.199822+0000","last_became_peered":"2024-09-20T22:41:44.199822+0000","last_unstale":"2024-09-20T22:52:28.314795+0000","last_undegraded":"2024-09-20T22:52:28.314795+0000","last_fullsized":"2024-09-20T22:52:28.314795+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"154'2","last_scrub_stamp":"2024-09-20T22:50:34.375161+0000","last_deep_scrub":"154'2","last_deep_scrub_stamp":"2024-09-20T22:43:54.134066+0000","last_clean_scrub_stamp":"2024-09-20T22:50:34.375161+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:28.653493+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"67'1","reported_seq":159,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-09-20T22:52:28.274639+0000","last_change":"2024-09-20T22:51:20.303293+0000","last_active":"2024-09-20T22:52:28.274639+0000","last_peered":"2024-09-20T22:52:28.274639+0000","last_clean":"2024-09-20T22:52:28.274639+0000","last_became_active":"2024-09-20T22:51:20.279978+0000","last_became_peered":"2024-09-20T22:51:20.279978+0000","last_unstale":"2024-09-20T22:52:28.274639+0000","last_undegraded":"2024-09-20T22:52:28.274639+0000","last_fullsized":"2024-09-20T22:52:28.274639+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-09-20T22:50:42.717953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:50:42.717953+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:28.385068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"625'25","reported_seq":614,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-09-20T22:52:28.274658+0000","last_change":"2024-09-20T22:51:02.707652+0000","last_active":"2024-09-20T22:52:28.274658+0000","last_peered":"2024-09-20T22:52:28.274658+0000","last_clean":"2024-09-20T22:52:28.274658+0000","last_became_active":"2024-09-20T22:47:49.232181+0000","last_became_peered":"2024-09-20T22:47:49.232181+0000","last_unstale":"2024-09-20T22:52:28.274658+0000","last_undegraded":"2024-09-20T22:52:28.274658+0000","last_fullsized":"2024-09-20T22:52:28.274658+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"151'15","last_scrub_stamp":"2024-09-20T22:51:02.707574+0000","last_deep_scrub":"151'15","last_deep_scrub_stamp":"2024-09-20T22:43:50.715761+0000","last_clean_scrub_stamp":"2024-09-20T22:51:02.707574+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:28.623115+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"661'11","reported_seq":1421,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-09-20T22:52:28.286996+0000","last_change":"2024-09-20T22:52:21.908015+0000","last_active":"2024-09-20T22:52:28.286996+0000","last_peered":"2024-09-20T22:52:28.286996+0000","last_clean":"2024-09-20T22:52:28.286996+0000","last_became_active":"2024-09-20T22:48:05.244270+0000","last_became_peered":"2024-09-20T22:48:05.244270+0000","last_unstale":"2024-09-20T22:52:28.286996+0000","last_undegraded":"2024-09-20T22:52:28.286996+0000","last_fullsized":"2024-09-20T22:52:28.286996+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"661'11","last_scrub_stamp":"2024-09-20T22:52:21.907943+0000","last_deep_scrub":"661'11","last_deep_scrub_stamp":"2024-09-20T22:52:21.907943+0000","last_clean_scrub_stamp":"2024-09-20T22:52:21.907943+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:27.456801+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.c","version":"668'2003","reported_seq":4804,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-09-20T22:52:28.398197+0000","last_change":"2024-09-20T22:52:27.324941+0000","last_active":"2024-09-20T22:52:28.398197+0000","last_peered":"2024-09-20T22:52:28.398197+0000","last_clean":"2024-09-20T22:52:28.398197+0000","last_became_active":"2024-09-20T22:42:52.404027+0000","last_became_peered":"2024-09-20T22:42:52.404027+0000","last_unstale":"2024-09-20T22:52:28.398197+0000","last_undegraded":"2024-09-20T22:52:28.398197+0000","last_fullsized":"2024-09-20T22:52:28.398197+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"584'1653","last_scrub_stamp":"2024-09-20T22:51:04.993550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:51:04.993550+0000","objects_scrubbed":39,"log_size":2003,"log_dups_size":0,"ondisk_log_size":2003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0.012900962,"stat_sum":{"num_bytes":93185371,"num_objects":42,"num_object_clones":35,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":2,"num_read":860,"num_read_kb":933643,"num_write":549,"num_write_kb":213664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"c5","length":"1"},{"start":"ce","length":"1"},{"start":"d4","length":"1"},{"start":"da","length":"1"},{"start":"df","length":"1"},{"start":"e3","length":"a"},{"start":"ee","length":"a"},{"start":"f9","length":"1"},{"start":"fc","length":"1"},{"start":"fe","length":"5"},{"start":"104","length":"3"},{"start":"108","length":"4"},{"start":"10d","length":"1"},{"start":"10f","length":"1"},{"start":"111","length":"1"},{"start":"113","length":"1"},{"start":"116","length":"2"},{"start":"11a","length":"1"},{"start":"11c","length":"2"},{"start":"123","length":"1"},{"start":"125","length":"1"}]},{"pgid":"1.a","version":"77'128","reported_seq":1507,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-09-20T22:52:28.282248+0000","last_change":"2024-09-20T22:50:19.676094+0000","last_active":"2024-09-20T22:52:28.282248+0000","last_peered":"2024-09-20T22:52:28.282248+0000","last_clean":"2024-09-20T22:52:28.282248+0000","last_became_active":"2024-09-20T22:42:52.407791+0000","last_became_peered":"2024-09-20T22:42:52.407791+0000","last_unstale":"2024-09-20T22:52:28.282248+0000","last_undegraded":"2024-09-20T22:52:28.282248+0000","last_fullsized":"2024-09-20T22:52:28.282248+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:50:19.675992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:50:19.675992+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"625'27","reported_seq":233,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-09-20T22:52:28.277393+0000","last_change":"2024-09-20T22:52:15.358218+0000","last_active":"2024-09-20T22:52:28.277393+0000","last_peered":"2024-09-20T22:52:28.277393+0000","last_clean":"2024-09-20T22:52:28.277393+0000","last_became_active":"2024-09-20T22:51:20.287083+0000","last_became_peered":"2024-09-20T22:51:20.287083+0000","last_unstale":"2024-09-20T22:52:28.277393+0000","last_undegraded":"2024-09-20T22:52:28.277393+0000","last_fullsized":"2024-09-20T22:52:28.277393+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'27","last_scrub_stamp":"2024-09-20T22:52:15.358100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:52:15.358100+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:27.239349+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":1244,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-09-20T22:52:28.274249+0000","last_change":"2024-09-20T22:51:29.394206+0000","last_active":"2024-09-20T22:52:28.274249+0000","last_peered":"2024-09-20T22:52:28.274249+0000","last_clean":"2024-09-20T22:52:28.274249+0000","last_became_active":"2024-09-20T22:51:20.279439+0000","last_became_peered":"2024-09-20T22:51:20.279439+0000","last_unstale":"2024-09-20T22:52:28.274249+0000","last_undegraded":"2024-09-20T22:52:28.274249+0000","last_fullsized":"2024-09-20T22:52:28.274249+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:29.394097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:49:12.070675+0000","last_clean_scrub_stamp":"2024-09-20T22:51:29.394097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:57.584257+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"408'12","reported_seq":611,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-09-20T22:52:28.274908+0000","last_change":"2024-09-20T22:51:50.785575+0000","last_active":"2024-09-20T22:52:28.274908+0000","last_peered":"2024-09-20T22:52:28.274908+0000","last_clean":"2024-09-20T22:52:28.274908+0000","last_became_active":"2024-09-20T22:48:05.242914+0000","last_became_peered":"2024-09-20T22:48:05.242914+0000","last_unstale":"2024-09-20T22:52:28.274908+0000","last_undegraded":"2024-09-20T22:52:28.274908+0000","last_fullsized":"2024-09-20T22:52:28.274908+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"408'12","last_scrub_stamp":"2024-09-20T22:51:50.785478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:51:50.785478+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:06.586894+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":1253,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-09-20T22:52:28.282940+0000","last_change":"2024-09-20T22:51:31.960415+0000","last_active":"2024-09-20T22:52:28.282940+0000","last_peered":"2024-09-20T22:52:28.282940+0000","last_clean":"2024-09-20T22:52:28.282940+0000","last_became_active":"2024-09-20T22:51:20.280145+0000","last_became_peered":"2024-09-20T22:51:20.280145+0000","last_unstale":"2024-09-20T22:52:28.282940+0000","last_undegraded":"2024-09-20T22:52:28.282940+0000","last_fullsized":"2024-09-20T22:52:28.282940+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:31.960331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:51:31.960331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:46.560339+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3","version":"668'275","reported_seq":2999,"reported_epoch":668,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-20T22:52:28.379104+0000","last_change":"2024-09-20T22:52:27.313911+0000","last_active":"2024-09-20T22:52:28.379104+0000","last_peered":"2024-09-20T22:52:28.379104+0000","last_clean":"2024-09-20T22:52:28.379104+0000","last_became_active":"2024-09-20T22:49:05.221402+0000","last_became_peered":"2024-09-20T22:49:05.221402+0000","last_unstale":"2024-09-20T22:52:28.379104+0000","last_undegraded":"2024-09-20T22:52:28.379104+0000","last_fullsized":"2024-09-20T22:52:28.379104+0000","mapping_epoch":404,"log_start":"456'129","ondisk_log_start":"456'129","created":94,"last_epoch_clean":465,"parent":"0.0","parent_split_bits":0,"last_scrub":"609'243","last_scrub_stamp":"2024-09-20T22:51:31.402013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:51:31.402013+0000","objects_scrubbed":6,"log_size":146,"log_dups_size":149,"ondisk_log_size":146,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:55.249653+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.98643658300000003,"stat_sum":{"num_bytes":17925868,"num_objects":7,"num_object_clones":6,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":99,"num_read_kb":128300,"num_write":65,"num_write_kb":22506,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":20196357,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"c5","length":"1"},{"start":"ce","length":"1"},{"start":"d4","length":"1"},{"start":"da","length":"1"},{"start":"df","length":"1"},{"start":"e3","length":"a"},{"start":"ee","length":"a"},{"start":"f9","length":"1"},{"start":"fc","length":"1"},{"start":"fe","length":"5"},{"start":"104","length":"3"},{"start":"108","length":"4"},{"start":"10d","length":"1"},{"start":"10f","length":"1"},{"start":"111","length":"1"},{"start":"113","length":"1"},{"start":"116","length":"2"},{"start":"11a","length":"1"},{"start":"11c","length":"1"},{"start":"123","length":"1"},{"start":"125","length":"1"}]},{"pgid":"1.5","version":"77'128","reported_seq":1414,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-09-20T22:52:28.275256+0000","last_change":"2024-09-20T22:51:12.927535+0000","last_active":"2024-09-20T22:52:28.275256+0000","last_peered":"2024-09-20T22:52:28.275256+0000","last_clean":"2024-09-20T22:52:28.275256+0000","last_became_active":"2024-09-20T22:48:08.207018+0000","last_became_peered":"2024-09-20T22:48:08.207018+0000","last_unstale":"2024-09-20T22:52:28.275256+0000","last_undegraded":"2024-09-20T22:52:28.275256+0000","last_fullsized":"2024-09-20T22:52:28.275256+0000","mapping_epoch":407,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":408,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:51:12.927463+0000","last_deep_scrub":"77'128","last_deep_scrub_stamp":"2024-09-20T22:45:41.608075+0000","last_clean_scrub_stamp":"2024-09-20T22:51:12.927463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:28.499830+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"625'30","reported_seq":1333,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-09-20T22:52:28.274483+0000","last_change":"2024-09-20T22:50:45.270518+0000","last_active":"2024-09-20T22:52:28.274483+0000","last_peered":"2024-09-20T22:52:28.274483+0000","last_clean":"2024-09-20T22:52:28.274483+0000","last_became_active":"2024-09-20T22:47:49.217605+0000","last_became_peered":"2024-09-20T22:47:49.217605+0000","last_unstale":"2024-09-20T22:52:28.274483+0000","last_undegraded":"2024-09-20T22:52:28.274483+0000","last_fullsized":"2024-09-20T22:52:28.274483+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'18","last_scrub_stamp":"2024-09-20T22:50:45.270399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:50:45.270399+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:28.385128+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":1250,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-09-20T22:52:28.277687+0000","last_change":"2024-09-20T22:50:22.220104+0000","last_active":"2024-09-20T22:52:28.277687+0000","last_peered":"2024-09-20T22:52:28.277687+0000","last_clean":"2024-09-20T22:52:28.277687+0000","last_became_active":"2024-09-20T22:47:49.227069+0000","last_became_peered":"2024-09-20T22:47:49.227069+0000","last_unstale":"2024-09-20T22:52:28.277687+0000","last_undegraded":"2024-09-20T22:52:28.277687+0000","last_fullsized":"2024-09-20T22:52:28.277687+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:50:22.219983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:50:22.219983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:28.385018+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":555,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-09-20T22:52:28.275570+0000","last_change":"2024-09-20T22:51:09.901939+0000","last_active":"2024-09-20T22:52:28.275570+0000","last_peered":"2024-09-20T22:52:28.275570+0000","last_clean":"2024-09-20T22:52:28.275570+0000","last_became_active":"2024-09-20T22:48:05.230003+0000","last_became_peered":"2024-09-20T22:48:05.230003+0000","last_unstale":"2024-09-20T22:52:28.275570+0000","last_undegraded":"2024-09-20T22:52:28.275570+0000","last_fullsized":"2024-09-20T22:52:28.275570+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:09.901838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:07.947945+0000","last_clean_scrub_stamp":"2024-09-20T22:51:09.901838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":555,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-09-20T22:52:28.274664+0000","last_change":"2024-09-20T22:51:55.748527+0000","last_active":"2024-09-20T22:52:28.274664+0000","last_peered":"2024-09-20T22:52:28.274664+0000","last_clean":"2024-09-20T22:52:28.274664+0000","last_became_active":"2024-09-20T22:48:05.239917+0000","last_became_peered":"2024-09-20T22:48:05.239917+0000","last_unstale":"2024-09-20T22:52:28.274664+0000","last_undegraded":"2024-09-20T22:52:28.274664+0000","last_fullsized":"2024-09-20T22:52:28.274664+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:55.748425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:53.036601+0000","last_clean_scrub_stamp":"2024-09-20T22:51:55.748425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:19.609464+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2","version":"666'679","reported_seq":2693,"reported_epoch":668,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T22:52:28.275692+0000","last_change":"2024-09-20T22:52:27.291828+0000","last_active":"2024-09-20T22:52:28.275692+0000","last_peered":"2024-09-20T22:52:28.275692+0000","last_clean":"2024-09-20T22:52:28.275692+0000","last_became_active":"2024-09-20T22:52:23.286461+0000","last_became_peered":"2024-09-20T22:52:23.286461+0000","last_unstale":"2024-09-20T22:52:28.275692+0000","last_undegraded":"2024-09-20T22:52:28.275692+0000","last_fullsized":"2024-09-20T22:52:28.275692+0000","mapping_epoch":662,"log_start":"471'464","ondisk_log_start":"471'464","created":94,"last_epoch_clean":663,"parent":"0.0","parent_split_bits":0,"last_scrub":"282'120","last_scrub_stamp":"2024-09-20T22:46:22.099642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:46:22.099642+0000","objects_scrubbed":1,"log_size":215,"log_dups_size":687,"ondisk_log_size":215,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":49,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.98646230099999999,"stat_sum":{"num_bytes":53298460,"num_objects":24,"num_object_clones":20,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":1,"num_read":376,"num_read_kb":396277,"num_write":228,"num_write_kb":92279,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":144,"num_bytes_recovered":106353268,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"c5","length":"1"}]},{"pgid":"1.4","version":"77'128","reported_seq":1539,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-09-20T22:52:28.277221+0000","last_change":"2024-09-20T22:52:02.474080+0000","last_active":"2024-09-20T22:52:28.277221+0000","last_peered":"2024-09-20T22:52:28.277221+0000","last_clean":"2024-09-20T22:52:28.277221+0000","last_became_active":"2024-09-20T22:46:15.181305+0000","last_became_peered":"2024-09-20T22:46:15.181305+0000","last_unstale":"2024-09-20T22:52:28.277221+0000","last_undegraded":"2024-09-20T22:52:28.277221+0000","last_fullsized":"2024-09-20T22:52:28.277221+0000","mapping_epoch":94,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:52:02.473979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:52:02.473979+0000","objects_scrubbed":1,"log_size":0,"log_dups_size":59,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:05.099562+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"625'28","reported_seq":592,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-09-20T22:52:28.275517+0000","last_change":"2024-09-20T22:51:58.712602+0000","last_active":"2024-09-20T22:52:28.275517+0000","last_peered":"2024-09-20T22:52:28.275517+0000","last_clean":"2024-09-20T22:52:28.275517+0000","last_became_active":"2024-09-20T22:48:05.235812+0000","last_became_peered":"2024-09-20T22:48:05.235812+0000","last_unstale":"2024-09-20T22:52:28.275517+0000","last_undegraded":"2024-09-20T22:52:28.275517+0000","last_fullsized":"2024-09-20T22:52:28.275517+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'28","last_scrub_stamp":"2024-09-20T22:51:58.712488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:51:58.712488+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:18.921188+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":41,"num_read_kb":34,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"67'1","reported_seq":611,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-09-20T22:52:28.274511+0000","last_change":"2024-09-20T22:51:20.292468+0000","last_active":"2024-09-20T22:52:28.274511+0000","last_peered":"2024-09-20T22:52:28.274511+0000","last_clean":"2024-09-20T22:52:28.274511+0000","last_became_active":"2024-09-20T22:51:20.283011+0000","last_became_peered":"2024-09-20T22:51:20.283011+0000","last_unstale":"2024-09-20T22:52:28.274511+0000","last_undegraded":"2024-09-20T22:52:28.274511+0000","last_fullsized":"2024-09-20T22:52:28.274511+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-09-20T22:51:11.806011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:51:11.806011+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"600'13","reported_seq":1421,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-09-20T22:52:28.274800+0000","last_change":"2024-09-20T22:51:32.448600+0000","last_active":"2024-09-20T22:52:28.274800+0000","last_peered":"2024-09-20T22:52:28.274800+0000","last_clean":"2024-09-20T22:52:28.274800+0000","last_became_active":"2024-09-20T22:51:20.289873+0000","last_became_peered":"2024-09-20T22:51:20.289873+0000","last_unstale":"2024-09-20T22:52:28.274800+0000","last_undegraded":"2024-09-20T22:52:28.274800+0000","last_fullsized":"2024-09-20T22:52:28.274800+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"600'13","last_scrub_stamp":"2024-09-20T22:51:32.448512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:51:32.448512+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:52.962692+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":1235,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-09-20T22:52:28.314412+0000","last_change":"2024-09-20T22:51:31.487593+0000","last_active":"2024-09-20T22:52:28.314412+0000","last_peered":"2024-09-20T22:52:28.314412+0000","last_clean":"2024-09-20T22:52:28.314412+0000","last_became_active":"2024-09-20T22:41:44.831901+0000","last_became_peered":"2024-09-20T22:41:44.831901+0000","last_unstale":"2024-09-20T22:52:28.314412+0000","last_undegraded":"2024-09-20T22:52:28.314412+0000","last_fullsized":"2024-09-20T22:52:28.314412+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:31.487469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:51:31.487469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:01.276783+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1","version":"668'374","reported_seq":3175,"reported_epoch":668,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T22:52:28.305302+0000","last_change":"2024-09-20T22:52:28.305302+0000","last_active":"2024-09-20T22:52:28.305302+0000","last_peered":"2024-09-20T22:52:28.305302+0000","last_clean":"2024-09-20T22:52:28.305302+0000","last_became_active":"2024-09-20T22:49:21.232750+0000","last_became_peered":"2024-09-20T22:49:21.232750+0000","last_unstale":"2024-09-20T22:52:28.305302+0000","last_undegraded":"2024-09-20T22:52:28.305302+0000","last_fullsized":"2024-09-20T22:52:28.305302+0000","mapping_epoch":407,"log_start":"474'259","ondisk_log_start":"474'259","created":94,"last_epoch_clean":481,"parent":"0.0","parent_split_bits":0,"last_scrub":"562'298","last_scrub_stamp":"2024-09-20T22:50:45.816540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:50:45.816540+0000","objects_scrubbed":4,"log_size":115,"log_dups_size":338,"ondisk_log_size":115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.029870242000000002,"stat_sum":{"num_bytes":13262300,"num_objects":8,"num_object_clones":6,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":1,"num_read":93,"num_read_kb":91249,"num_write":86,"num_write_kb":45551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":8732217,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"c5","length":"1"},{"start":"ce","length":"1"},{"start":"d4","length":"1"},{"start":"da","length":"1"},{"start":"df","length":"1"},{"start":"e3","length":"a"},{"start":"ee","length":"a"},{"start":"f9","length":"1"},{"start":"fc","length":"1"},{"start":"fe","length":"5"},{"start":"104","length":"3"},{"start":"108","length":"4"},{"start":"10d","length":"1"},{"start":"10f","length":"1"},{"start":"111","length":"1"},{"start":"113","length":"1"},{"start":"116","length":"2"},{"start":"11a","length":"1"},{"start":"11c","length":"1"},{"start":"123","length":"1"},{"start":"125","length":"1"}]},{"pgid":"1.7","version":"77'128","reported_seq":1544,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-09-20T22:52:28.286569+0000","last_change":"2024-09-20T22:51:10.000911+0000","last_active":"2024-09-20T22:52:28.286569+0000","last_peered":"2024-09-20T22:52:28.286569+0000","last_clean":"2024-09-20T22:52:28.286569+0000","last_became_active":"2024-09-20T22:46:03.169596+0000","last_became_peered":"2024-09-20T22:46:03.169596+0000","last_unstale":"2024-09-20T22:52:28.286569+0000","last_undegraded":"2024-09-20T22:52:28.286569+0000","last_fullsized":"2024-09-20T22:52:28.286569+0000","mapping_epoch":94,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:51:10.000823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:51:10.000823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:28.849579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"625'20","reported_seq":217,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-09-20T22:52:28.282649+0000","last_change":"2024-09-20T22:52:19.125548+0000","last_active":"2024-09-20T22:52:28.282649+0000","last_peered":"2024-09-20T22:52:28.282649+0000","last_clean":"2024-09-20T22:52:28.282649+0000","last_became_active":"2024-09-20T22:51:20.297933+0000","last_became_peered":"2024-09-20T22:51:20.297933+0000","last_unstale":"2024-09-20T22:52:28.282649+0000","last_undegraded":"2024-09-20T22:52:28.282649+0000","last_fullsized":"2024-09-20T22:52:28.282649+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'20","last_scrub_stamp":"2024-09-20T22:52:19.125462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:52:19.125462+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:45.473839+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":1248,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-09-20T22:52:28.283192+0000","last_change":"2024-09-20T22:50:14.699223+0000","last_active":"2024-09-20T22:52:28.283192+0000","last_peered":"2024-09-20T22:52:28.283192+0000","last_clean":"2024-09-20T22:52:28.283192+0000","last_became_active":"2024-09-20T22:41:38.155629+0000","last_became_peered":"2024-09-20T22:41:38.155629+0000","last_unstale":"2024-09-20T22:52:28.283192+0000","last_undegraded":"2024-09-20T22:52:28.283192+0000","last_fullsized":"2024-09-20T22:52:28.283192+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:50:14.699152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:50:14.699152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"389'11","reported_seq":648,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-09-20T22:52:28.273318+0000","last_change":"2024-09-20T22:51:26.968244+0000","last_active":"2024-09-20T22:52:28.273318+0000","last_peered":"2024-09-20T22:52:28.273318+0000","last_clean":"2024-09-20T22:52:28.273318+0000","last_became_active":"2024-09-20T22:47:49.227264+0000","last_became_peered":"2024-09-20T22:47:49.227264+0000","last_unstale":"2024-09-20T22:52:28.273318+0000","last_undegraded":"2024-09-20T22:52:28.273318+0000","last_fullsized":"2024-09-20T22:52:28.273318+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'11","last_scrub_stamp":"2024-09-20T22:51:26.968138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:51:26.968138+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:30.761523+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":587,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-09-20T22:52:28.272364+0000","last_change":"2024-09-20T22:51:55.755550+0000","last_active":"2024-09-20T22:52:28.272364+0000","last_peered":"2024-09-20T22:52:28.272364+0000","last_clean":"2024-09-20T22:52:28.272364+0000","last_became_active":"2024-09-20T22:47:49.232380+0000","last_became_peered":"2024-09-20T22:47:49.232380+0000","last_unstale":"2024-09-20T22:52:28.272364+0000","last_undegraded":"2024-09-20T22:52:28.272364+0000","last_fullsized":"2024-09-20T22:52:28.272364+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:55.755432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:51:55.755432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:07.619648+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.0","version":"668'514","reported_seq":3431,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-09-20T22:52:28.344449+0000","last_change":"2024-09-20T22:52:28.308801+0000","last_active":"2024-09-20T22:52:28.344449+0000","last_peered":"2024-09-20T22:52:28.344449+0000","last_clean":"2024-09-20T22:52:28.344449+0000","last_became_active":"2024-09-20T22:51:53.366096+0000","last_became_peered":"2024-09-20T22:51:53.366096+0000","last_unstale":"2024-09-20T22:52:28.344449+0000","last_undegraded":"2024-09-20T22:52:28.344449+0000","last_fullsized":"2024-09-20T22:52:28.344449+0000","mapping_epoch":632,"log_start":"480'253","ondisk_log_start":"480'253","created":94,"last_epoch_clean":633,"parent":"0.0","parent_split_bits":0,"last_scrub":"562'368","last_scrub_stamp":"2024-09-20T22:50:44.669910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:50:44.669910+0000","objects_scrubbed":15,"log_size":261,"log_dups_size":416,"ondisk_log_size":261,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.026157293000000002,"stat_sum":{"num_bytes":37589116,"num_objects":19,"num_object_clones":16,"num_object_copies":57,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":191,"num_read_kb":202069,"num_write":156,"num_write_kb":51687,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":42943203,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"c5","length":"1"},{"start":"ce","length":"1"},{"start":"d4","length":"1"},{"start":"da","length":"1"},{"start":"df","length":"1"},{"start":"e3","length":"a"},{"start":"ee","length":"a"},{"start":"f9","length":"1"},{"start":"fc","length":"1"},{"start":"fe","length":"5"},{"start":"104","length":"3"},{"start":"108","length":"4"},{"start":"10d","length":"1"},{"start":"10f","length":"1"},{"start":"111","length":"1"},{"start":"113","length":"1"},{"start":"116","length":"2"},{"start":"11a","length":"1"},{"start":"11c","length":"2"},{"start":"123","length":"1"},{"start":"125","length":"1"}]},{"pgid":"1.6","version":"77'128","reported_seq":1566,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-09-20T22:52:28.286497+0000","last_change":"2024-09-20T22:51:06.036981+0000","last_active":"2024-09-20T22:52:28.286497+0000","last_peered":"2024-09-20T22:52:28.286497+0000","last_clean":"2024-09-20T22:52:28.286497+0000","last_became_active":"2024-09-20T22:47:52.212190+0000","last_became_peered":"2024-09-20T22:47:52.212190+0000","last_unstale":"2024-09-20T22:52:28.286497+0000","last_undegraded":"2024-09-20T22:52:28.286497+0000","last_fullsized":"2024-09-20T22:52:28.286497+0000","mapping_epoch":391,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:51:06.036900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:51:06.036900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:32.934556+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"625'37","reported_seq":620,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-09-20T22:52:28.273032+0000","last_change":"2024-09-20T22:49:55.964247+0000","last_active":"2024-09-20T22:52:28.273032+0000","last_peered":"2024-09-20T22:52:28.273032+0000","last_clean":"2024-09-20T22:52:28.273032+0000","last_became_active":"2024-09-20T22:47:49.222009+0000","last_became_peered":"2024-09-20T22:47:49.222009+0000","last_unstale":"2024-09-20T22:52:28.273032+0000","last_undegraded":"2024-09-20T22:52:28.273032+0000","last_fullsized":"2024-09-20T22:52:28.273032+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"515'27","last_scrub_stamp":"2024-09-20T22:49:55.964153+0000","last_deep_scrub":"515'27","last_deep_scrub_stamp":"2024-09-20T22:49:55.964153+0000","last_clean_scrub_stamp":"2024-09-20T22:49:55.964153+0000","objects_scrubbed":8,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:28.623005+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":53,"num_read_kb":39,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":156,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-09-20T22:52:28.276274+0000","last_change":"2024-09-20T22:52:14.225251+0000","last_active":"2024-09-20T22:52:28.276274+0000","last_peered":"2024-09-20T22:52:28.276274+0000","last_clean":"2024-09-20T22:52:28.276274+0000","last_became_active":"2024-09-20T22:51:20.286646+0000","last_became_peered":"2024-09-20T22:51:20.286646+0000","last_unstale":"2024-09-20T22:52:28.276274+0000","last_undegraded":"2024-09-20T22:52:28.276274+0000","last_fullsized":"2024-09-20T22:52:28.276274+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:14.225111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:48:23.320031+0000","last_clean_scrub_stamp":"2024-09-20T22:52:14.225111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:38.894957+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":1259,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-09-20T22:52:28.285940+0000","last_change":"2024-09-20T22:51:29.928160+0000","last_active":"2024-09-20T22:52:28.285940+0000","last_peered":"2024-09-20T22:52:28.285940+0000","last_clean":"2024-09-20T22:52:28.285940+0000","last_became_active":"2024-09-20T22:47:49.214055+0000","last_became_peered":"2024-09-20T22:47:49.214055+0000","last_unstale":"2024-09-20T22:52:28.285940+0000","last_undegraded":"2024-09-20T22:52:28.285940+0000","last_fullsized":"2024-09-20T22:52:28.285940+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:29.928092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:51:29.928092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:32.798183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":156,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-09-20T22:52:28.285626+0000","last_change":"2024-09-20T22:51:51.100969+0000","last_active":"2024-09-20T22:52:28.285626+0000","last_peered":"2024-09-20T22:52:28.285626+0000","last_clean":"2024-09-20T22:52:28.285626+0000","last_became_active":"2024-09-20T22:51:20.284874+0000","last_became_peered":"2024-09-20T22:51:20.284874+0000","last_unstale":"2024-09-20T22:52:28.285626+0000","last_undegraded":"2024-09-20T22:52:28.285626+0000","last_fullsized":"2024-09-20T22:52:28.285626+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:51.100878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:48:41.208853+0000","last_clean_scrub_stamp":"2024-09-20T22:51:51.100878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:54.933631+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.7","version":"668'1098","reported_seq":4034,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-09-20T22:52:28.397610+0000","last_change":"2024-09-20T22:52:27.325076+0000","last_active":"2024-09-20T22:52:28.397610+0000","last_peered":"2024-09-20T22:52:28.397610+0000","last_clean":"2024-09-20T22:52:28.397610+0000","last_became_active":"2024-09-20T22:48:43.224386+0000","last_became_peered":"2024-09-20T22:48:43.224386+0000","last_unstale":"2024-09-20T22:52:28.397610+0000","last_undegraded":"2024-09-20T22:52:28.397610+0000","last_fullsized":"2024-09-20T22:52:28.397610+0000","mapping_epoch":391,"log_start":"436'456","ondisk_log_start":"436'456","created":94,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"631'991","last_scrub_stamp":"2024-09-20T22:51:54.106963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:51:54.106963+0000","objects_scrubbed":20,"log_size":642,"log_dups_size":886,"ondisk_log_size":642,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:23.260795+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.011364201000000001,"stat_sum":{"num_bytes":58642960,"num_objects":29,"num_object_clones":24,"num_object_copies":87,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":1,"num_read":427,"num_read_kb":434620,"num_write":358,"num_write_kb":119119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":115,"num_bytes_recovered":51615167,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"c5","length":"1"},{"start":"ce","length":"1"},{"start":"d4","length":"1"},{"start":"da","length":"1"},{"start":"df","length":"1"},{"start":"e3","length":"a"},{"start":"ee","length":"a"},{"start":"f9","length":"1"},{"start":"fc","length":"1"},{"start":"fe","length":"5"},{"start":"104","length":"3"},{"start":"108","length":"4"},{"start":"10d","length":"1"},{"start":"10f","length":"1"},{"start":"111","length":"1"},{"start":"113","length":"1"},{"start":"116","length":"2"},{"start":"11a","length":"1"},{"start":"11c","length":"2"},{"start":"123","length":"1"},{"start":"125","length":"1"}]},{"pgid":"1.1","version":"77'128","reported_seq":1555,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-09-20T22:52:28.282192+0000","last_change":"2024-09-20T22:50:20.639645+0000","last_active":"2024-09-20T22:52:28.282192+0000","last_peered":"2024-09-20T22:52:28.282192+0000","last_clean":"2024-09-20T22:52:28.282192+0000","last_became_active":"2024-09-20T22:47:52.208102+0000","last_became_peered":"2024-09-20T22:47:52.208102+0000","last_unstale":"2024-09-20T22:52:28.282192+0000","last_undegraded":"2024-09-20T22:52:28.282192+0000","last_fullsized":"2024-09-20T22:52:28.282192+0000","mapping_epoch":391,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:50:20.639561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:50:20.639561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"625'51","reported_seq":321,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-09-20T22:52:28.276236+0000","last_change":"2024-09-20T22:52:17.199465+0000","last_active":"2024-09-20T22:52:28.276236+0000","last_peered":"2024-09-20T22:52:28.276236+0000","last_clean":"2024-09-20T22:52:28.276236+0000","last_became_active":"2024-09-20T22:51:20.289376+0000","last_became_peered":"2024-09-20T22:51:20.289376+0000","last_unstale":"2024-09-20T22:52:28.276236+0000","last_undegraded":"2024-09-20T22:52:28.276236+0000","last_fullsized":"2024-09-20T22:52:28.276236+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'51","last_scrub_stamp":"2024-09-20T22:52:17.199367+0000","last_deep_scrub":"86'38","last_deep_scrub_stamp":"2024-09-20T22:42:43.935677+0000","last_clean_scrub_stamp":"2024-09-20T22:52:17.199367+0000","objects_scrubbed":10,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:38.979686+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":547,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-09-20T22:52:28.274734+0000","last_change":"2024-09-20T22:50:35.898050+0000","last_active":"2024-09-20T22:52:28.274734+0000","last_peered":"2024-09-20T22:52:28.274734+0000","last_clean":"2024-09-20T22:52:28.274734+0000","last_became_active":"2024-09-20T22:48:05.237756+0000","last_became_peered":"2024-09-20T22:48:05.237756+0000","last_unstale":"2024-09-20T22:52:28.274734+0000","last_undegraded":"2024-09-20T22:52:28.274734+0000","last_fullsized":"2024-09-20T22:52:28.274734+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:50:35.897979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:50:35.897979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:31.564136+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"69'2","reported_seq":1276,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-09-20T22:52:28.282947+0000","last_change":"2024-09-20T22:51:37.070187+0000","last_active":"2024-09-20T22:52:28.282947+0000","last_peered":"2024-09-20T22:52:28.282947+0000","last_clean":"2024-09-20T22:52:28.282947+0000","last_became_active":"2024-09-20T22:51:20.286952+0000","last_became_peered":"2024-09-20T22:51:20.286952+0000","last_unstale":"2024-09-20T22:52:28.282947+0000","last_undegraded":"2024-09-20T22:52:28.282947+0000","last_fullsized":"2024-09-20T22:52:28.282947+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2024-09-20T22:51:37.070111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:36.130684+0000","last_clean_scrub_stamp":"2024-09-20T22:51:37.070111+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:05.960971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087280599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":599,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-09-20T22:52:28.273939+0000","last_change":"2024-09-20T22:51:53.772969+0000","last_active":"2024-09-20T22:52:28.273939+0000","last_peered":"2024-09-20T22:52:28.273939+0000","last_clean":"2024-09-20T22:52:28.273939+0000","last_became_active":"2024-09-20T22:51:20.278427+0000","last_became_peered":"2024-09-20T22:51:20.278427+0000","last_unstale":"2024-09-20T22:52:28.273939+0000","last_undegraded":"2024-09-20T22:52:28.273939+0000","last_fullsized":"2024-09-20T22:52:28.273939+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:53.772885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:51:53.772885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:59.965395+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":1246,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-09-20T22:52:28.277524+0000","last_change":"2024-09-20T22:52:21.363432+0000","last_active":"2024-09-20T22:52:28.277524+0000","last_peered":"2024-09-20T22:52:28.277524+0000","last_clean":"2024-09-20T22:52:28.277524+0000","last_became_active":"2024-09-20T22:48:05.234193+0000","last_became_peered":"2024-09-20T22:48:05.234193+0000","last_unstale":"2024-09-20T22:52:28.277524+0000","last_undegraded":"2024-09-20T22:52:28.277524+0000","last_fullsized":"2024-09-20T22:52:28.277524+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:21.363359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:52:21.363359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:28.660587+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"667'393","reported_seq":1744,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-09-20T22:52:28.273737+0000","last_change":"2024-09-20T22:52:27.310587+0000","last_active":"2024-09-20T22:52:28.273737+0000","last_peered":"2024-09-20T22:52:28.273737+0000","last_clean":"2024-09-20T22:52:28.273737+0000","last_became_active":"2024-09-20T22:51:23.267613+0000","last_became_peered":"2024-09-20T22:51:23.267613+0000","last_unstale":"2024-09-20T22:52:28.273737+0000","last_undegraded":"2024-09-20T22:52:28.273737+0000","last_fullsized":"2024-09-20T22:52:28.273737+0000","mapping_epoch":602,"log_start":"456'176","ondisk_log_start":"456'176","created":94,"last_epoch_clean":603,"parent":"0.0","parent_split_bits":0,"last_scrub":"616'341","last_scrub_stamp":"2024-09-20T22:51:36.893731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:51:36.893731+0000","objects_scrubbed":8,"log_size":217,"log_dups_size":318,"ondisk_log_size":217,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:06.874669+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.002268642,"stat_sum":{"num_bytes":24989903,"num_objects":10,"num_object_clones":8,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":144,"num_read_kb":137177,"num_write":123,"num_write_kb":42628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":26566881,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"c5","length":"1"},{"start":"ce","length":"1"},{"start":"d4","length":"1"},{"start":"da","length":"1"},{"start":"df","length":"1"},{"start":"e3","length":"a"},{"start":"ee","length":"a"},{"start":"f9","length":"1"},{"start":"fc","length":"1"},{"start":"fe","length":"5"},{"start":"104","length":"3"},{"start":"108","length":"4"},{"start":"10d","length":"1"},{"start":"10f","length":"1"},{"start":"111","length":"1"},{"start":"113","length":"1"},{"start":"116","length":"2"},{"start":"11a","length":"1"},{"start":"11c","length":"2"},{"start":"123","length":"1"},{"start":"125","length":"1"}]},{"pgid":"1.2","version":"77'128","reported_seq":1624,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-09-20T22:52:28.315080+0000","last_change":"2024-09-20T22:52:02.623634+0000","last_active":"2024-09-20T22:52:28.315080+0000","last_peered":"2024-09-20T22:52:28.315080+0000","last_clean":"2024-09-20T22:52:28.315080+0000","last_became_active":"2024-09-20T22:48:08.203028+0000","last_became_peered":"2024-09-20T22:48:08.203028+0000","last_unstale":"2024-09-20T22:52:28.315080+0000","last_undegraded":"2024-09-20T22:52:28.315080+0000","last_fullsized":"2024-09-20T22:52:28.315080+0000","mapping_epoch":407,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":408,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:52:02.623562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:52:02.623562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:31.354752+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"625'22","reported_seq":1009,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-09-20T22:52:28.283749+0000","last_change":"2024-09-20T22:51:20.312887+0000","last_active":"2024-09-20T22:52:28.283749+0000","last_peered":"2024-09-20T22:52:28.283749+0000","last_clean":"2024-09-20T22:52:28.283749+0000","last_became_active":"2024-09-20T22:51:20.293610+0000","last_became_peered":"2024-09-20T22:51:20.293610+0000","last_unstale":"2024-09-20T22:52:28.283749+0000","last_undegraded":"2024-09-20T22:52:28.283749+0000","last_fullsized":"2024-09-20T22:52:28.283749+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"443'18","last_scrub_stamp":"2024-09-20T22:51:04.643936+0000","last_deep_scrub":"443'18","last_deep_scrub_stamp":"2024-09-20T22:48:44.182517+0000","last_clean_scrub_stamp":"2024-09-20T22:51:04.643936+0000","objects_scrubbed":5,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":990,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":156,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-09-20T22:52:28.277659+0000","last_change":"2024-09-20T22:51:32.279015+0000","last_active":"2024-09-20T22:52:28.277659+0000","last_peered":"2024-09-20T22:52:28.277659+0000","last_clean":"2024-09-20T22:52:28.277659+0000","last_became_active":"2024-09-20T22:51:20.278609+0000","last_became_peered":"2024-09-20T22:51:20.278609+0000","last_unstale":"2024-09-20T22:52:28.277659+0000","last_undegraded":"2024-09-20T22:52:28.277659+0000","last_fullsized":"2024-09-20T22:52:28.277659+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:32.278945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:51:32.278945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:54.818249+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"380'2","reported_seq":1293,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-09-20T22:52:28.275405+0000","last_change":"2024-09-20T22:50:39.347293+0000","last_active":"2024-09-20T22:52:28.275405+0000","last_peered":"2024-09-20T22:52:28.275405+0000","last_clean":"2024-09-20T22:52:28.275405+0000","last_became_active":"2024-09-20T22:47:49.222812+0000","last_became_peered":"2024-09-20T22:47:49.222812+0000","last_unstale":"2024-09-20T22:52:28.275405+0000","last_undegraded":"2024-09-20T22:52:28.275405+0000","last_fullsized":"2024-09-20T22:52:28.275405+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"380'2","last_scrub_stamp":"2024-09-20T22:50:39.347177+0000","last_deep_scrub":"380'2","last_deep_scrub_stamp":"2024-09-20T22:47:40.983039+0000","last_clean_scrub_stamp":"2024-09-20T22:50:39.347177+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:31.128396+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001301459,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":1247,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-09-20T22:52:28.275807+0000","last_change":"2024-09-20T22:51:04.254552+0000","last_active":"2024-09-20T22:52:28.275807+0000","last_peered":"2024-09-20T22:52:28.275807+0000","last_clean":"2024-09-20T22:52:28.275807+0000","last_became_active":"2024-09-20T22:41:42.185112+0000","last_became_peered":"2024-09-20T22:41:42.185112+0000","last_unstale":"2024-09-20T22:52:28.275807+0000","last_undegraded":"2024-09-20T22:52:28.275807+0000","last_fullsized":"2024-09-20T22:52:28.275807+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:04.254441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:47:22.881109+0000","last_clean_scrub_stamp":"2024-09-20T22:51:04.254441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:31.528165+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":1235,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-09-20T22:52:28.315253+0000","last_change":"2024-09-20T22:51:11.568798+0000","last_active":"2024-09-20T22:52:28.315253+0000","last_peered":"2024-09-20T22:52:28.315253+0000","last_clean":"2024-09-20T22:52:28.315253+0000","last_became_active":"2024-09-20T22:41:44.830515+0000","last_became_peered":"2024-09-20T22:41:44.830515+0000","last_unstale":"2024-09-20T22:52:28.315253+0000","last_undegraded":"2024-09-20T22:52:28.315253+0000","last_fullsized":"2024-09-20T22:52:28.315253+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:11.568712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:51:11.568712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:28.653536+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"664'314","reported_seq":3231,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-09-20T22:52:28.314846+0000","last_change":"2024-09-20T22:52:27.313678+0000","last_active":"2024-09-20T22:52:28.314846+0000","last_peered":"2024-09-20T22:52:28.314846+0000","last_clean":"2024-09-20T22:52:28.314846+0000","last_became_active":"2024-09-20T22:48:57.236417+0000","last_became_peered":"2024-09-20T22:48:57.236417+0000","last_unstale":"2024-09-20T22:52:28.314846+0000","last_undegraded":"2024-09-20T22:52:28.314846+0000","last_fullsized":"2024-09-20T22:52:28.314846+0000","mapping_epoch":391,"log_start":"450'190","ondisk_log_start":"450'190","created":94,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"605'278","last_scrub_stamp":"2024-09-20T22:51:25.467092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:51:25.467092+0000","objects_scrubbed":6,"log_size":124,"log_dups_size":240,"ondisk_log_size":124,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:34.503525+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00079967700000000005,"stat_sum":{"num_bytes":12667288,"num_objects":7,"num_object_clones":6,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":76,"num_read_kb":60075,"num_write":88,"num_write_kb":36243,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":19673726,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"c5","length":"1"},{"start":"ce","length":"1"},{"start":"d4","length":"1"},{"start":"da","length":"1"},{"start":"df","length":"1"},{"start":"e3","length":"a"},{"start":"ee","length":"a"},{"start":"f9","length":"1"},{"start":"fc","length":"1"},{"start":"fe","length":"5"},{"start":"104","length":"3"},{"start":"108","length":"4"},{"start":"10d","length":"1"},{"start":"10f","length":"1"},{"start":"111","length":"1"},{"start":"113","length":"1"},{"start":"116","length":"2"},{"start":"11a","length":"1"},{"start":"11c","length":"2"},{"start":"123","length":"1"},{"start":"125","length":"1"}]},{"pgid":"1.3","version":"77'128","reported_seq":451,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-09-20T22:52:28.286041+0000","last_change":"2024-09-20T22:51:28.897355+0000","last_active":"2024-09-20T22:52:28.286041+0000","last_peered":"2024-09-20T22:52:28.286041+0000","last_clean":"2024-09-20T22:52:28.286041+0000","last_became_active":"2024-09-20T22:51:23.266810+0000","last_became_peered":"2024-09-20T22:51:23.266810+0000","last_unstale":"2024-09-20T22:52:28.286041+0000","last_undegraded":"2024-09-20T22:52:28.286041+0000","last_fullsized":"2024-09-20T22:52:28.286041+0000","mapping_epoch":602,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":603,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:51:28.897273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:51:28.897273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:58.849420+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.6","version":"625'24","reported_seq":631,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-09-20T22:52:28.274698+0000","last_change":"2024-09-20T22:51:20.300892+0000","last_active":"2024-09-20T22:52:28.274698+0000","last_peered":"2024-09-20T22:52:28.274698+0000","last_clean":"2024-09-20T22:52:28.274698+0000","last_became_active":"2024-09-20T22:51:20.283112+0000","last_became_peered":"2024-09-20T22:51:20.283112+0000","last_unstale":"2024-09-20T22:52:28.274698+0000","last_undegraded":"2024-09-20T22:52:28.274698+0000","last_fullsized":"2024-09-20T22:52:28.274698+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-09-20T22:50:14.745918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:50:14.745918+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:28.623054+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":593,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-09-20T22:52:28.272820+0000","last_change":"2024-09-20T22:50:39.786895+0000","last_active":"2024-09-20T22:52:28.272820+0000","last_peered":"2024-09-20T22:52:28.272820+0000","last_clean":"2024-09-20T22:52:28.272820+0000","last_became_active":"2024-09-20T22:48:05.233093+0000","last_became_peered":"2024-09-20T22:48:05.233093+0000","last_unstale":"2024-09-20T22:52:28.272820+0000","last_undegraded":"2024-09-20T22:52:28.272820+0000","last_fullsized":"2024-09-20T22:52:28.272820+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:50:39.786825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:44:19.834641+0000","last_clean_scrub_stamp":"2024-09-20T22:50:39.786825+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:28.623098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"75'5","reported_seq":1930,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-09-20T22:52:28.314134+0000","last_change":"2024-09-20T22:51:20.290148+0000","last_active":"2024-09-20T22:52:28.314134+0000","last_peered":"2024-09-20T22:52:28.314134+0000","last_clean":"2024-09-20T22:52:28.314134+0000","last_became_active":"2024-09-20T22:51:20.281329+0000","last_became_peered":"2024-09-20T22:51:20.281329+0000","last_unstale":"2024-09-20T22:52:28.314134+0000","last_undegraded":"2024-09-20T22:52:28.314134+0000","last_fullsized":"2024-09-20T22:52:28.314134+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-09-20T22:50:21.363477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:36.130684+0000","last_clean_scrub_stamp":"2024-09-20T22:50:21.363477+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:28.653477+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00078631700000000003,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":640,"num_read_kb":635,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1167,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":1257,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-09-20T22:52:28.283762+0000","last_change":"2024-09-20T22:51:20.274832+0000","last_active":"2024-09-20T22:52:28.283762+0000","last_peered":"2024-09-20T22:52:28.283762+0000","last_clean":"2024-09-20T22:52:28.283762+0000","last_became_active":"2024-09-20T22:51:20.274522+0000","last_became_peered":"2024-09-20T22:51:20.274522+0000","last_unstale":"2024-09-20T22:52:28.283762+0000","last_undegraded":"2024-09-20T22:52:28.283762+0000","last_fullsized":"2024-09-20T22:52:28.283762+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:50:18.695027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:50:18.695027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":148,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-09-20T22:52:28.282873+0000","last_change":"2024-09-20T22:51:20.288058+0000","last_active":"2024-09-20T22:52:28.282873+0000","last_peered":"2024-09-20T22:52:28.282873+0000","last_clean":"2024-09-20T22:52:28.282873+0000","last_became_active":"2024-09-20T22:51:20.287750+0000","last_became_peered":"2024-09-20T22:51:20.287750+0000","last_unstale":"2024-09-20T22:52:28.282873+0000","last_undegraded":"2024-09-20T22:52:28.282873+0000","last_fullsized":"2024-09-20T22:52:28.282873+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:05.621880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:51:05.621880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.6","version":"668'400","reported_seq":3248,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-09-20T22:52:28.361199+0000","last_change":"2024-09-20T22:52:27.310937+0000","last_active":"2024-09-20T22:52:28.361199+0000","last_peered":"2024-09-20T22:52:28.361199+0000","last_clean":"2024-09-20T22:52:28.361199+0000","last_became_active":"2024-09-20T22:48:47.234531+0000","last_became_peered":"2024-09-20T22:48:47.234531+0000","last_unstale":"2024-09-20T22:52:28.361199+0000","last_undegraded":"2024-09-20T22:52:28.361199+0000","last_fullsized":"2024-09-20T22:52:28.361199+0000","mapping_epoch":391,"log_start":"443'181","ondisk_log_start":"443'181","created":94,"last_epoch_clean":447,"parent":"0.0","parent_split_bits":0,"last_scrub":"654'392","last_scrub_stamp":"2024-09-20T22:52:14.706591+0000","last_deep_scrub":"440'177","last_deep_scrub_stamp":"2024-09-20T22:48:40.602342+0000","last_clean_scrub_stamp":"2024-09-20T22:52:14.706591+0000","objects_scrubbed":10,"log_size":219,"log_dups_size":314,"ondisk_log_size":219,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:44.598364+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.0015264180000000001,"stat_sum":{"num_bytes":21273481,"num_objects":9,"num_object_clones":7,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":145,"num_read_kb":142854,"num_write":124,"num_write_kb":45901,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":18662529,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"c5","length":"1"},{"start":"ce","length":"1"},{"start":"d4","length":"1"},{"start":"da","length":"1"},{"start":"df","length":"1"},{"start":"e3","length":"a"},{"start":"ee","length":"a"},{"start":"f9","length":"1"},{"start":"fc","length":"1"},{"start":"fe","length":"5"},{"start":"104","length":"3"},{"start":"108","length":"4"},{"start":"10d","length":"1"},{"start":"10f","length":"1"},{"start":"111","length":"1"},{"start":"113","length":"1"},{"start":"116","length":"2"},{"start":"11a","length":"1"},{"start":"11c","length":"2"},{"start":"123","length":"1"},{"start":"125","length":"1"}]},{"pgid":"4.5","version":"625'32","reported_seq":1518,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-09-20T22:52:28.286845+0000","last_change":"2024-09-20T22:51:33.961483+0000","last_active":"2024-09-20T22:52:28.286845+0000","last_peered":"2024-09-20T22:52:28.286845+0000","last_clean":"2024-09-20T22:52:28.286845+0000","last_became_active":"2024-09-20T22:48:05.243523+0000","last_became_peered":"2024-09-20T22:48:05.243523+0000","last_unstale":"2024-09-20T22:52:28.286845+0000","last_undegraded":"2024-09-20T22:52:28.286845+0000","last_fullsized":"2024-09-20T22:52:28.286845+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"408'20","last_scrub_stamp":"2024-09-20T22:51:33.961401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:51:33.961401+0000","objects_scrubbed":8,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:00.387882+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"67'1","reported_seq":1245,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-09-20T22:52:28.276450+0000","last_change":"2024-09-20T22:51:35.440516+0000","last_active":"2024-09-20T22:52:28.276450+0000","last_peered":"2024-09-20T22:52:28.276450+0000","last_clean":"2024-09-20T22:52:28.276450+0000","last_became_active":"2024-09-20T22:41:38.158917+0000","last_became_peered":"2024-09-20T22:41:38.158917+0000","last_unstale":"2024-09-20T22:52:28.276450+0000","last_undegraded":"2024-09-20T22:52:28.276450+0000","last_fullsized":"2024-09-20T22:52:28.276450+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-09-20T22:51:35.440416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:51:35.440416+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:53.239571+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"77'128","reported_seq":1596,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-09-20T22:52:28.314192+0000","last_change":"2024-09-20T22:52:18.688292+0000","last_active":"2024-09-20T22:52:28.314192+0000","last_peered":"2024-09-20T22:52:28.314192+0000","last_clean":"2024-09-20T22:52:28.314192+0000","last_became_active":"2024-09-20T22:47:52.206169+0000","last_became_peered":"2024-09-20T22:47:52.206169+0000","last_unstale":"2024-09-20T22:52:28.314192+0000","last_undegraded":"2024-09-20T22:52:28.314192+0000","last_fullsized":"2024-09-20T22:52:28.314192+0000","mapping_epoch":391,"log_start":"77'128","ondisk_log_start":"77'128","created":19,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:52:18.688135+0000","last_deep_scrub":"77'128","last_deep_scrub_stamp":"2024-09-20T22:47:22.417099+0000","last_clean_scrub_stamp":"2024-09-20T22:52:18.688135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:47.016821+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":1247,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-09-20T22:52:28.315000+0000","last_change":"2024-09-20T22:51:50.585717+0000","last_active":"2024-09-20T22:52:28.315000+0000","last_peered":"2024-09-20T22:52:28.315000+0000","last_clean":"2024-09-20T22:52:28.315000+0000","last_became_active":"2024-09-20T22:41:42.184387+0000","last_became_peered":"2024-09-20T22:41:42.184387+0000","last_unstale":"2024-09-20T22:52:28.315000+0000","last_undegraded":"2024-09-20T22:52:28.315000+0000","last_fullsized":"2024-09-20T22:52:28.315000+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:50.585627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:51:50.585627+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:03.411628+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":1247,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-09-20T22:52:28.283656+0000","last_change":"2024-09-20T22:49:57.715494+0000","last_active":"2024-09-20T22:52:28.283656+0000","last_peered":"2024-09-20T22:52:28.283656+0000","last_clean":"2024-09-20T22:52:28.283656+0000","last_became_active":"2024-09-20T22:48:05.241602+0000","last_became_peered":"2024-09-20T22:48:05.241602+0000","last_unstale":"2024-09-20T22:52:28.283656+0000","last_undegraded":"2024-09-20T22:52:28.283656+0000","last_fullsized":"2024-09-20T22:52:28.283656+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:57.715416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:49:57.715416+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"664'478","reported_seq":3449,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-09-20T22:52:28.286814+0000","last_change":"2024-09-20T22:52:27.314273+0000","last_active":"2024-09-20T22:52:28.286814+0000","last_peered":"2024-09-20T22:52:28.286814+0000","last_clean":"2024-09-20T22:52:28.286814+0000","last_became_active":"2024-09-20T22:51:23.269216+0000","last_became_peered":"2024-09-20T22:51:23.269216+0000","last_unstale":"2024-09-20T22:52:28.286814+0000","last_undegraded":"2024-09-20T22:52:28.286814+0000","last_fullsized":"2024-09-20T22:52:28.286814+0000","mapping_epoch":602,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":603,"parent":"0.0","parent_split_bits":0,"last_scrub":"631'450","last_scrub_stamp":"2024-09-20T22:51:52.091217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:51:52.091217+0000","objects_scrubbed":8,"log_size":478,"log_dups_size":0,"ondisk_log_size":478,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:19.082046+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0014679579999999999,"stat_sum":{"num_bytes":16398843,"num_objects":9,"num_object_clones":7,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":1,"num_read":161,"num_read_kb":161235,"num_write":85,"num_write_kb":35211,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":52,"num_bytes_recovered":30669453,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"c5","length":"1"},{"start":"ce","length":"1"},{"start":"d4","length":"1"},{"start":"da","length":"1"},{"start":"df","length":"1"},{"start":"e3","length":"a"},{"start":"ee","length":"a"},{"start":"f9","length":"1"},{"start":"fc","length":"1"},{"start":"fe","length":"5"},{"start":"104","length":"3"},{"start":"108","length":"4"},{"start":"10d","length":"1"},{"start":"10f","length":"1"},{"start":"111","length":"1"},{"start":"113","length":"1"},{"start":"116","length":"2"},{"start":"11a","length":"1"},{"start":"11c","length":"2"},{"start":"123","length":"1"},{"start":"125","length":"1"}]},{"pgid":"4.e","version":"625'19","reported_seq":588,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-09-20T22:52:28.274919+0000","last_change":"2024-09-20T22:51:20.310962+0000","last_active":"2024-09-20T22:52:28.274919+0000","last_peered":"2024-09-20T22:52:28.274919+0000","last_clean":"2024-09-20T22:52:28.274919+0000","last_became_active":"2024-09-20T22:51:20.288958+0000","last_became_peered":"2024-09-20T22:51:20.288958+0000","last_unstale":"2024-09-20T22:52:28.274919+0000","last_undegraded":"2024-09-20T22:52:28.274919+0000","last_fullsized":"2024-09-20T22:52:28.274919+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-09-20T22:50:38.879205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:50:38.879205+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:31.564198+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":555,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-09-20T22:52:28.274350+0000","last_change":"2024-09-20T22:51:29.757136+0000","last_active":"2024-09-20T22:52:28.274350+0000","last_peered":"2024-09-20T22:52:28.274350+0000","last_clean":"2024-09-20T22:52:28.274350+0000","last_became_active":"2024-09-20T22:48:05.254688+0000","last_became_peered":"2024-09-20T22:48:05.254688+0000","last_unstale":"2024-09-20T22:52:28.274350+0000","last_undegraded":"2024-09-20T22:52:28.274350+0000","last_fullsized":"2024-09-20T22:52:28.274350+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:29.757030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:51:29.757030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:52.073822+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":1251,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-09-20T22:52:28.283596+0000","last_change":"2024-09-20T22:51:51.002353+0000","last_active":"2024-09-20T22:52:28.283596+0000","last_peered":"2024-09-20T22:52:28.283596+0000","last_clean":"2024-09-20T22:52:28.283596+0000","last_became_active":"2024-09-20T22:48:05.233778+0000","last_became_peered":"2024-09-20T22:48:05.233778+0000","last_unstale":"2024-09-20T22:52:28.283596+0000","last_undegraded":"2024-09-20T22:52:28.283596+0000","last_fullsized":"2024-09-20T22:52:28.283596+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:51.002226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:51:51.002226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:20.564071+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":1226,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-09-20T22:52:28.277129+0000","last_change":"2024-09-20T22:50:47.230929+0000","last_active":"2024-09-20T22:52:28.277129+0000","last_peered":"2024-09-20T22:52:28.277129+0000","last_clean":"2024-09-20T22:52:28.277129+0000","last_became_active":"2024-09-20T22:41:44.832009+0000","last_became_peered":"2024-09-20T22:41:44.832009+0000","last_unstale":"2024-09-20T22:52:28.277129+0000","last_undegraded":"2024-09-20T22:52:28.277129+0000","last_fullsized":"2024-09-20T22:52:28.277129+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:50:47.230854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:48:40.127123+0000","last_clean_scrub_stamp":"2024-09-20T22:50:47.230854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:28.385083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.e","version":"664'606","reported_seq":3530,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-09-20T22:52:28.275665+0000","last_change":"2024-09-20T22:52:27.323762+0000","last_active":"2024-09-20T22:52:28.275665+0000","last_peered":"2024-09-20T22:52:28.275665+0000","last_clean":"2024-09-20T22:52:28.275665+0000","last_became_active":"2024-09-20T22:47:52.209452+0000","last_became_peered":"2024-09-20T22:47:52.209452+0000","last_unstale":"2024-09-20T22:52:28.275665+0000","last_undegraded":"2024-09-20T22:52:28.275665+0000","last_fullsized":"2024-09-20T22:52:28.275665+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"559'431","last_scrub_stamp":"2024-09-20T22:50:45.276861+0000","last_deep_scrub":"174'60","last_deep_scrub_stamp":"2024-09-20T22:44:18.301776+0000","last_clean_scrub_stamp":"2024-09-20T22:50:45.276861+0000","objects_scrubbed":8,"log_size":606,"log_dups_size":0,"ondisk_log_size":606,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0.00055340499999999998,"stat_sum":{"num_bytes":26834975,"num_objects":13,"num_object_clones":11,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":236,"num_read_kb":237527,"num_write":156,"num_write_kb":59501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":23110741,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"c5","length":"1"},{"start":"ce","length":"1"},{"start":"d4","length":"1"},{"start":"da","length":"1"},{"start":"df","length":"1"},{"start":"e3","length":"a"},{"start":"ee","length":"a"},{"start":"f9","length":"1"},{"start":"fc","length":"1"},{"start":"fe","length":"5"},{"start":"104","length":"3"},{"start":"108","length":"4"},{"start":"10d","length":"1"},{"start":"10f","length":"1"},{"start":"111","length":"1"},{"start":"113","length":"1"},{"start":"116","length":"2"},{"start":"11a","length":"1"},{"start":"11c","length":"2"},{"start":"123","length":"1"},{"start":"125","length":"1"}]},{"pgid":"1.8","version":"77'128","reported_seq":1515,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-09-20T22:52:28.282104+0000","last_change":"2024-09-20T22:51:56.108038+0000","last_active":"2024-09-20T22:52:28.282104+0000","last_peered":"2024-09-20T22:52:28.282104+0000","last_clean":"2024-09-20T22:52:28.282104+0000","last_became_active":"2024-09-20T22:42:52.405594+0000","last_became_peered":"2024-09-20T22:42:52.405594+0000","last_unstale":"2024-09-20T22:52:28.282104+0000","last_undegraded":"2024-09-20T22:52:28.282104+0000","last_fullsized":"2024-09-20T22:52:28.282104+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:51:56.107944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:51:56.107944+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:19.128079+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"625'31","reported_seq":625,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-09-20T22:52:28.275137+0000","last_change":"2024-09-20T22:52:13.620897+0000","last_active":"2024-09-20T22:52:28.275137+0000","last_peered":"2024-09-20T22:52:28.275137+0000","last_clean":"2024-09-20T22:52:28.275137+0000","last_became_active":"2024-09-20T22:51:20.288846+0000","last_became_peered":"2024-09-20T22:51:20.288846+0000","last_unstale":"2024-09-20T22:52:28.275137+0000","last_undegraded":"2024-09-20T22:52:28.275137+0000","last_fullsized":"2024-09-20T22:52:28.275137+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'31","last_scrub_stamp":"2024-09-20T22:52:13.620754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:52:13.620754+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:26.544483+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":49,"num_read_kb":39,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":801,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-09-20T22:52:28.275836+0000","last_change":"2024-09-20T22:51:20.294379+0000","last_active":"2024-09-20T22:52:28.275836+0000","last_peered":"2024-09-20T22:52:28.275836+0000","last_clean":"2024-09-20T22:52:28.275836+0000","last_became_active":"2024-09-20T22:51:20.294073+0000","last_became_peered":"2024-09-20T22:51:20.294073+0000","last_unstale":"2024-09-20T22:52:28.275836+0000","last_undegraded":"2024-09-20T22:52:28.275836+0000","last_fullsized":"2024-09-20T22:52:28.275836+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:50:47.268782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:35.492562+0000","last_clean_scrub_stamp":"2024-09-20T22:50:47.268782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:31.128490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":867,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-09-20T22:52:28.274115+0000","last_change":"2024-09-20T22:51:25.738132+0000","last_active":"2024-09-20T22:52:28.274115+0000","last_peered":"2024-09-20T22:52:28.274115+0000","last_clean":"2024-09-20T22:52:28.274115+0000","last_became_active":"2024-09-20T22:51:20.281724+0000","last_became_peered":"2024-09-20T22:51:20.281724+0000","last_unstale":"2024-09-20T22:52:28.274115+0000","last_undegraded":"2024-09-20T22:52:28.274115+0000","last_fullsized":"2024-09-20T22:52:28.274115+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:25.738061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:51:25.738061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:41.584933+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":1262,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-09-20T22:52:28.275620+0000","last_change":"2024-09-20T22:50:41.284220+0000","last_active":"2024-09-20T22:52:28.275620+0000","last_peered":"2024-09-20T22:52:28.275620+0000","last_clean":"2024-09-20T22:52:28.275620+0000","last_became_active":"2024-09-20T22:48:05.239006+0000","last_became_peered":"2024-09-20T22:48:05.239006+0000","last_unstale":"2024-09-20T22:52:28.275620+0000","last_undegraded":"2024-09-20T22:52:28.275620+0000","last_fullsized":"2024-09-20T22:52:28.275620+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:50:41.284158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:50:41.284158+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:31.128443+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"667'658","reported_seq":3550,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-09-20T22:52:28.274611+0000","last_change":"2024-09-20T22:52:27.321476+0000","last_active":"2024-09-20T22:52:28.274611+0000","last_peered":"2024-09-20T22:52:28.274611+0000","last_clean":"2024-09-20T22:52:28.274611+0000","last_became_active":"2024-09-20T22:47:52.216198+0000","last_became_peered":"2024-09-20T22:47:52.216198+0000","last_unstale":"2024-09-20T22:52:28.274611+0000","last_undegraded":"2024-09-20T22:52:28.274611+0000","last_fullsized":"2024-09-20T22:52:28.274611+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"630'603","last_scrub_stamp":"2024-09-20T22:51:50.769050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:51:50.769050+0000","objects_scrubbed":8,"log_size":658,"log_dups_size":0,"ondisk_log_size":658,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:06.692448+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.011325988,"stat_sum":{"num_bytes":22844187,"num_objects":12,"num_object_clones":9,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":1,"num_read":280,"num_read_kb":277181,"num_write":177,"num_write_kb":59042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":53,"num_bytes_recovered":14983556,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"c5","length":"1"},{"start":"ce","length":"1"},{"start":"d4","length":"1"},{"start":"da","length":"1"},{"start":"df","length":"1"},{"start":"e3","length":"a"},{"start":"ee","length":"a"},{"start":"f9","length":"1"},{"start":"fc","length":"1"},{"start":"fe","length":"5"},{"start":"104","length":"3"},{"start":"108","length":"4"},{"start":"10d","length":"1"},{"start":"10f","length":"1"},{"start":"111","length":"1"},{"start":"113","length":"1"},{"start":"116","length":"2"},{"start":"11a","length":"1"},{"start":"11c","length":"2"},{"start":"123","length":"1"},{"start":"125","length":"1"}]},{"pgid":"1.9","version":"77'128","reported_seq":1556,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-09-20T22:52:28.314354+0000","last_change":"2024-09-20T22:50:42.263313+0000","last_active":"2024-09-20T22:52:28.314354+0000","last_peered":"2024-09-20T22:52:28.314354+0000","last_clean":"2024-09-20T22:52:28.314354+0000","last_became_active":"2024-09-20T22:50:42.262862+0000","last_became_peered":"2024-09-20T22:50:42.262862+0000","last_unstale":"2024-09-20T22:52:28.314354+0000","last_undegraded":"2024-09-20T22:52:28.314354+0000","last_fullsized":"2024-09-20T22:52:28.314354+0000","mapping_epoch":561,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":562,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:49:52.301348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:49:52.301348+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:28.653441+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"625'20","reported_seq":581,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-09-20T22:52:28.274709+0000","last_change":"2024-09-20T22:52:22.646547+0000","last_active":"2024-09-20T22:52:28.274709+0000","last_peered":"2024-09-20T22:52:28.274709+0000","last_clean":"2024-09-20T22:52:28.274709+0000","last_became_active":"2024-09-20T22:48:05.242811+0000","last_became_peered":"2024-09-20T22:48:05.242811+0000","last_unstale":"2024-09-20T22:52:28.274709+0000","last_undegraded":"2024-09-20T22:52:28.274709+0000","last_fullsized":"2024-09-20T22:52:28.274709+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'20","last_scrub_stamp":"2024-09-20T22:52:22.646468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:52:22.646468+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:48.440427+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":1262,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-09-20T22:52:28.275287+0000","last_change":"2024-09-20T22:50:13.205821+0000","last_active":"2024-09-20T22:52:28.275287+0000","last_peered":"2024-09-20T22:52:28.275287+0000","last_clean":"2024-09-20T22:52:28.275287+0000","last_became_active":"2024-09-20T22:48:05.233999+0000","last_became_peered":"2024-09-20T22:48:05.233999+0000","last_unstale":"2024-09-20T22:52:28.275287+0000","last_undegraded":"2024-09-20T22:52:28.275287+0000","last_fullsized":"2024-09-20T22:52:28.275287+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:50:13.205743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:50:13.205743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:28.385004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"261'9","reported_seq":1394,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-09-20T22:52:28.275655+0000","last_change":"2024-09-20T22:52:18.218722+0000","last_active":"2024-09-20T22:52:28.275655+0000","last_peered":"2024-09-20T22:52:28.275655+0000","last_clean":"2024-09-20T22:52:28.275655+0000","last_became_active":"2024-09-20T22:41:42.185863+0000","last_became_peered":"2024-09-20T22:41:42.185863+0000","last_unstale":"2024-09-20T22:52:28.275655+0000","last_undegraded":"2024-09-20T22:52:28.275655+0000","last_fullsized":"2024-09-20T22:52:28.275655+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'9","last_scrub_stamp":"2024-09-20T22:52:18.218644+0000","last_deep_scrub":"261'9","last_deep_scrub_stamp":"2024-09-20T22:46:50.974735+0000","last_clean_scrub_stamp":"2024-09-20T22:52:18.218644+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:38.055984+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":560,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-09-20T22:52:28.274615+0000","last_change":"2024-09-20T22:51:20.296567+0000","last_active":"2024-09-20T22:52:28.274615+0000","last_peered":"2024-09-20T22:52:28.274615+0000","last_clean":"2024-09-20T22:52:28.274615+0000","last_became_active":"2024-09-20T22:51:20.296204+0000","last_became_peered":"2024-09-20T22:51:20.296204+0000","last_unstale":"2024-09-20T22:52:28.274615+0000","last_undegraded":"2024-09-20T22:52:28.274615+0000","last_fullsized":"2024-09-20T22:52:28.274615+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:02.932364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:51:02.932364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:31.564292+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.8","version":"667'734","reported_seq":3476,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-09-20T22:52:28.277152+0000","last_change":"2024-09-20T22:52:27.324786+0000","last_active":"2024-09-20T22:52:28.277152+0000","last_peered":"2024-09-20T22:52:28.277152+0000","last_clean":"2024-09-20T22:52:28.277152+0000","last_became_active":"2024-09-20T22:48:31.224359+0000","last_became_peered":"2024-09-20T22:48:31.224359+0000","last_unstale":"2024-09-20T22:52:28.277152+0000","last_undegraded":"2024-09-20T22:52:28.277152+0000","last_fullsized":"2024-09-20T22:52:28.277152+0000","mapping_epoch":407,"log_start":"426'268","ondisk_log_start":"426'268","created":94,"last_epoch_clean":431,"parent":"0.0","parent_split_bits":0,"last_scrub":"534'447","last_scrub_stamp":"2024-09-20T22:50:14.324477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:50:14.324477+0000","objects_scrubbed":15,"log_size":466,"log_dups_size":445,"ondisk_log_size":466,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.0026792159999999999,"stat_sum":{"num_bytes":40765759,"num_objects":20,"num_object_clones":17,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":1,"num_read":270,"num_read_kb":271650,"num_write":211,"num_write_kb":69953,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":26027409,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"c5","length":"1"},{"start":"ce","length":"1"},{"start":"d4","length":"1"},{"start":"da","length":"1"},{"start":"df","length":"1"},{"start":"e3","length":"a"},{"start":"ee","length":"a"},{"start":"f9","length":"1"},{"start":"fc","length":"1"},{"start":"fe","length":"5"},{"start":"104","length":"3"},{"start":"108","length":"4"},{"start":"10d","length":"1"},{"start":"10f","length":"1"},{"start":"111","length":"1"},{"start":"113","length":"1"},{"start":"116","length":"2"},{"start":"11a","length":"1"},{"start":"11c","length":"2"},{"start":"123","length":"1"},{"start":"125","length":"1"}]},{"pgid":"4.b","version":"625'15","reported_seq":638,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-09-20T22:52:28.273134+0000","last_change":"2024-09-20T22:51:20.296532+0000","last_active":"2024-09-20T22:52:28.273134+0000","last_peered":"2024-09-20T22:52:28.273134+0000","last_clean":"2024-09-20T22:52:28.273134+0000","last_became_active":"2024-09-20T22:51:20.273094+0000","last_became_peered":"2024-09-20T22:51:20.273094+0000","last_unstale":"2024-09-20T22:52:28.273134+0000","last_undegraded":"2024-09-20T22:52:28.273134+0000","last_fullsized":"2024-09-20T22:52:28.273134+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-09-20T22:50:36.883811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:50:36.883811+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:28.623131+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":1247,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-09-20T22:52:28.282903+0000","last_change":"2024-09-20T22:51:12.755628+0000","last_active":"2024-09-20T22:52:28.282903+0000","last_peered":"2024-09-20T22:52:28.282903+0000","last_clean":"2024-09-20T22:52:28.282903+0000","last_became_active":"2024-09-20T22:41:38.155953+0000","last_became_peered":"2024-09-20T22:41:38.155953+0000","last_unstale":"2024-09-20T22:52:28.282903+0000","last_undegraded":"2024-09-20T22:52:28.282903+0000","last_fullsized":"2024-09-20T22:52:28.282903+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:12.755526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:51:12.755526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:32.975772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":1270,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-09-20T22:52:28.276133+0000","last_change":"2024-09-20T22:50:37.347748+0000","last_active":"2024-09-20T22:52:28.276133+0000","last_peered":"2024-09-20T22:52:28.276133+0000","last_clean":"2024-09-20T22:52:28.276133+0000","last_became_active":"2024-09-20T22:48:05.240613+0000","last_became_peered":"2024-09-20T22:48:05.240613+0000","last_unstale":"2024-09-20T22:52:28.276133+0000","last_undegraded":"2024-09-20T22:52:28.276133+0000","last_fullsized":"2024-09-20T22:52:28.276133+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:50:37.347636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:50:37.347636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:31.128427+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":587,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-09-20T22:52:28.272921+0000","last_change":"2024-09-20T22:52:16.706497+0000","last_active":"2024-09-20T22:52:28.272921+0000","last_peered":"2024-09-20T22:52:28.272921+0000","last_clean":"2024-09-20T22:52:28.272921+0000","last_became_active":"2024-09-20T22:47:49.219486+0000","last_became_peered":"2024-09-20T22:47:49.219486+0000","last_unstale":"2024-09-20T22:52:28.272921+0000","last_undegraded":"2024-09-20T22:52:28.272921+0000","last_fullsized":"2024-09-20T22:52:28.272921+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:16.706429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:52:16.706429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:26.931246+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"668'1030","reported_seq":3973,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-09-20T22:52:28.361504+0000","last_change":"2024-09-20T22:52:27.315737+0000","last_active":"2024-09-20T22:52:28.361504+0000","last_peered":"2024-09-20T22:52:28.361504+0000","last_clean":"2024-09-20T22:52:28.361504+0000","last_became_active":"2024-09-20T22:51:23.270962+0000","last_became_peered":"2024-09-20T22:51:23.270962+0000","last_unstale":"2024-09-20T22:52:28.361504+0000","last_undegraded":"2024-09-20T22:52:28.361504+0000","last_fullsized":"2024-09-20T22:52:28.361504+0000","mapping_epoch":602,"log_start":"423'280","ondisk_log_start":"423'280","created":94,"last_epoch_clean":603,"parent":"0.0","parent_split_bits":0,"last_scrub":"590'828","last_scrub_stamp":"2024-09-20T22:51:10.909364+0000","last_deep_scrub":"590'828","last_deep_scrub_stamp":"2024-09-20T22:51:10.909364+0000","last_clean_scrub_stamp":"2024-09-20T22:51:10.909364+0000","objects_scrubbed":27,"log_size":750,"log_dups_size":649,"ondisk_log_size":750,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:35.189302+0000","scrub_duration":120,"objects_trimmed":0,"snaptrim_duration":0.0071147349999999996,"stat_sum":{"num_bytes":72163091,"num_objects":30,"num_object_clones":25,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":1,"num_read":368,"num_read_kb":365047,"num_write":301,"num_write_kb":117904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":91020358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"c5","length":"1"},{"start":"ce","length":"1"},{"start":"d4","length":"1"},{"start":"da","length":"1"},{"start":"df","length":"1"},{"start":"e3","length":"a"},{"start":"ee","length":"a"},{"start":"f9","length":"1"},{"start":"fc","length":"1"},{"start":"fe","length":"5"},{"start":"104","length":"3"},{"start":"108","length":"4"},{"start":"10d","length":"1"},{"start":"10f","length":"1"},{"start":"111","length":"1"},{"start":"113","length":"1"},{"start":"116","length":"2"},{"start":"11a","length":"1"},{"start":"11c","length":"2"},{"start":"123","length":"1"},{"start":"125","length":"1"}]},{"pgid":"4.a","version":"625'44","reported_seq":1348,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-09-20T22:52:28.285987+0000","last_change":"2024-09-20T22:52:14.038882+0000","last_active":"2024-09-20T22:52:28.285987+0000","last_peered":"2024-09-20T22:52:28.285987+0000","last_clean":"2024-09-20T22:52:28.285987+0000","last_became_active":"2024-09-20T22:51:20.287620+0000","last_became_peered":"2024-09-20T22:51:20.287620+0000","last_unstale":"2024-09-20T22:52:28.285987+0000","last_undegraded":"2024-09-20T22:52:28.285987+0000","last_fullsized":"2024-09-20T22:52:28.285987+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'44","last_scrub_stamp":"2024-09-20T22:52:14.038788+0000","last_deep_scrub":"113'28","last_deep_scrub_stamp":"2024-09-20T22:43:12.725900+0000","last_clean_scrub_stamp":"2024-09-20T22:52:14.038788+0000","objects_scrubbed":9,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:32.613295+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":1246,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-09-20T22:52:28.315661+0000","last_change":"2024-09-20T22:51:36.440271+0000","last_active":"2024-09-20T22:52:28.315661+0000","last_peered":"2024-09-20T22:52:28.315661+0000","last_clean":"2024-09-20T22:52:28.315661+0000","last_became_active":"2024-09-20T22:41:38.161524+0000","last_became_peered":"2024-09-20T22:41:38.161524+0000","last_unstale":"2024-09-20T22:52:28.315661+0000","last_undegraded":"2024-09-20T22:52:28.315661+0000","last_fullsized":"2024-09-20T22:52:28.315661+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:36.440160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:51:36.440160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:51.318747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":1258,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-09-20T22:52:28.276689+0000","last_change":"2024-09-20T22:50:15.301586+0000","last_active":"2024-09-20T22:52:28.276689+0000","last_peered":"2024-09-20T22:52:28.276689+0000","last_clean":"2024-09-20T22:52:28.276689+0000","last_became_active":"2024-09-20T22:47:49.223220+0000","last_became_peered":"2024-09-20T22:47:49.223220+0000","last_unstale":"2024-09-20T22:52:28.276689+0000","last_undegraded":"2024-09-20T22:52:28.276689+0000","last_fullsized":"2024-09-20T22:52:28.276689+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:50:15.301506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:50:15.301506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:31.128412+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":1242,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-09-20T22:52:28.314140+0000","last_change":"2024-09-20T22:52:01.643899+0000","last_active":"2024-09-20T22:52:28.314140+0000","last_peered":"2024-09-20T22:52:28.314140+0000","last_clean":"2024-09-20T22:52:28.314140+0000","last_became_active":"2024-09-20T22:41:44.199395+0000","last_became_peered":"2024-09-20T22:41:44.199395+0000","last_unstale":"2024-09-20T22:52:28.314140+0000","last_undegraded":"2024-09-20T22:52:28.314140+0000","last_fullsized":"2024-09-20T22:52:28.314140+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:01.643803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:47:18.329209+0000","last_clean_scrub_stamp":"2024-09-20T22:52:01.643803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:04.584695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.a","version":"668'759","reported_seq":3578,"reported_epoch":668,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T22:52:28.360469+0000","last_change":"2024-09-20T22:52:26.299703+0000","last_active":"2024-09-20T22:52:28.360469+0000","last_peered":"2024-09-20T22:52:28.360469+0000","last_clean":"2024-09-20T22:52:28.360469+0000","last_became_active":"2024-09-20T22:42:52.405085+0000","last_became_peered":"2024-09-20T22:42:52.405085+0000","last_unstale":"2024-09-20T22:52:28.360469+0000","last_undegraded":"2024-09-20T22:52:28.360469+0000","last_fullsized":"2024-09-20T22:52:28.360469+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"505'466","last_scrub_stamp":"2024-09-20T22:49:47.088165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:49:47.088165+0000","objects_scrubbed":13,"log_size":759,"log_dups_size":0,"ondisk_log_size":759,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.160975702,"stat_sum":{"num_bytes":44584098,"num_objects":19,"num_object_clones":16,"num_object_copies":57,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":1,"num_read":342,"num_read_kb":370675,"num_write":241,"num_write_kb":94613,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"c5","length":"1"},{"start":"ce","length":"1"},{"start":"d4","length":"1"},{"start":"da","length":"1"},{"start":"df","length":"1"},{"start":"e3","length":"a"},{"start":"ee","length":"7"},{"start":"f6","length":"2"},{"start":"f9","length":"1"},{"start":"fc","length":"1"},{"start":"fe","length":"5"},{"start":"104","length":"3"},{"start":"108","length":"4"},{"start":"10d","length":"1"},{"start":"10f","length":"1"},{"start":"111","length":"1"},{"start":"113","length":"1"},{"start":"116","length":"2"},{"start":"11a","length":"1"},{"start":"11c","length":"1"},{"start":"125","length":"1"}]},{"pgid":"4.9","version":"625'18","reported_seq":604,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-09-20T22:52:28.274811+0000","last_change":"2024-09-20T22:52:11.295320+0000","last_active":"2024-09-20T22:52:28.274811+0000","last_peered":"2024-09-20T22:52:28.274811+0000","last_clean":"2024-09-20T22:52:28.274811+0000","last_became_active":"2024-09-20T22:52:11.286549+0000","last_became_peered":"2024-09-20T22:52:11.286549+0000","last_unstale":"2024-09-20T22:52:28.274811+0000","last_undegraded":"2024-09-20T22:52:28.274811+0000","last_fullsized":"2024-09-20T22:52:28.274811+0000","mapping_epoch":650,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":651,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-09-20T22:50:40.914303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:50:40.914303+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:31.564229+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":21,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":1284,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-09-20T22:52:28.315969+0000","last_change":"2024-09-20T22:51:55.653755+0000","last_active":"2024-09-20T22:52:28.315969+0000","last_peered":"2024-09-20T22:52:28.315969+0000","last_clean":"2024-09-20T22:52:28.315969+0000","last_became_active":"2024-09-20T22:51:20.293701+0000","last_became_peered":"2024-09-20T22:51:20.293701+0000","last_unstale":"2024-09-20T22:52:28.315969+0000","last_undegraded":"2024-09-20T22:52:28.315969+0000","last_fullsized":"2024-09-20T22:52:28.315969+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:55.653662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:46:52.465940+0000","last_clean_scrub_stamp":"2024-09-20T22:51:55.653662+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:56.990647+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":1272,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-09-20T22:52:28.277520+0000","last_change":"2024-09-20T22:51:20.286960+0000","last_active":"2024-09-20T22:52:28.277520+0000","last_peered":"2024-09-20T22:52:28.277520+0000","last_clean":"2024-09-20T22:52:28.277520+0000","last_became_active":"2024-09-20T22:51:20.286626+0000","last_became_peered":"2024-09-20T22:51:20.286626+0000","last_unstale":"2024-09-20T22:52:28.277520+0000","last_undegraded":"2024-09-20T22:52:28.277520+0000","last_fullsized":"2024-09-20T22:52:28.277520+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:05.254319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:46:01.828080+0000","last_clean_scrub_stamp":"2024-09-20T22:51:05.254319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:31.128474+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":1232,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-09-20T22:52:28.275050+0000","last_change":"2024-09-20T22:51:20.279408+0000","last_active":"2024-09-20T22:52:28.275050+0000","last_peered":"2024-09-20T22:52:28.275050+0000","last_clean":"2024-09-20T22:52:28.275050+0000","last_became_active":"2024-09-20T22:51:20.279088+0000","last_became_peered":"2024-09-20T22:51:20.279088+0000","last_unstale":"2024-09-20T22:52:28.275050+0000","last_undegraded":"2024-09-20T22:52:28.275050+0000","last_fullsized":"2024-09-20T22:52:28.275050+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:50:39.266034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:55.170379+0000","last_clean_scrub_stamp":"2024-09-20T22:50:39.266034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:28.385143+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"668'1022","reported_seq":4020,"reported_epoch":668,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T22:52:28.317088+0000","last_change":"2024-09-20T22:52:28.309877+0000","last_active":"2024-09-20T22:52:28.317088+0000","last_peered":"2024-09-20T22:52:28.317088+0000","last_clean":"2024-09-20T22:52:28.317088+0000","last_became_active":"2024-09-20T22:51:23.271244+0000","last_became_peered":"2024-09-20T22:51:23.271244+0000","last_unstale":"2024-09-20T22:52:28.317088+0000","last_undegraded":"2024-09-20T22:52:28.317088+0000","last_fullsized":"2024-09-20T22:52:28.317088+0000","mapping_epoch":602,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":603,"parent":"0.0","parent_split_bits":0,"last_scrub":"661'1000","last_scrub_stamp":"2024-09-20T22:52:23.164028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:52:23.164028+0000","objects_scrubbed":23,"log_size":1022,"log_dups_size":0,"ondisk_log_size":1022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:42.897047+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.026479084999999999,"stat_sum":{"num_bytes":61164512,"num_objects":26,"num_object_clones":21,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":2,"num_read":493,"num_read_kb":560247,"num_write":339,"num_write_kb":115871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":76,"num_bytes_recovered":39288293,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"c5","length":"1"},{"start":"ce","length":"1"},{"start":"d4","length":"1"},{"start":"da","length":"1"},{"start":"df","length":"1"},{"start":"e3","length":"a"},{"start":"ee","length":"a"},{"start":"f9","length":"1"},{"start":"fc","length":"1"},{"start":"fe","length":"5"},{"start":"104","length":"3"},{"start":"108","length":"4"},{"start":"10d","length":"1"},{"start":"10f","length":"1"},{"start":"111","length":"1"},{"start":"113","length":"1"},{"start":"116","length":"2"},{"start":"11a","length":"1"},{"start":"11c","length":"1"},{"start":"125","length":"1"}]},{"pgid":"4.8","version":"625'27","reported_seq":1287,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-09-20T22:52:28.282774+0000","last_change":"2024-09-20T22:51:53.020721+0000","last_active":"2024-09-20T22:52:28.282774+0000","last_peered":"2024-09-20T22:52:28.282774+0000","last_clean":"2024-09-20T22:52:28.282774+0000","last_became_active":"2024-09-20T22:41:40.252113+0000","last_became_peered":"2024-09-20T22:41:40.252113+0000","last_unstale":"2024-09-20T22:52:28.282774+0000","last_undegraded":"2024-09-20T22:52:28.282774+0000","last_fullsized":"2024-09-20T22:52:28.282774+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'27","last_scrub_stamp":"2024-09-20T22:51:53.020628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:51:53.020628+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:14.005659+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'2","reported_seq":1309,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-09-20T22:52:28.316274+0000","last_change":"2024-09-20T22:51:26.421985+0000","last_active":"2024-09-20T22:52:28.316274+0000","last_peered":"2024-09-20T22:52:28.316274+0000","last_clean":"2024-09-20T22:52:28.316274+0000","last_became_active":"2024-09-20T22:48:05.238098+0000","last_became_peered":"2024-09-20T22:48:05.238098+0000","last_unstale":"2024-09-20T22:52:28.316274+0000","last_undegraded":"2024-09-20T22:52:28.316274+0000","last_fullsized":"2024-09-20T22:52:28.316274+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-09-20T22:51:26.421896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:51:26.421896+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:53.826247+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"408'10","reported_seq":1405,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-09-20T22:52:28.315304+0000","last_change":"2024-09-20T22:52:15.703572+0000","last_active":"2024-09-20T22:52:28.315304+0000","last_peered":"2024-09-20T22:52:28.315304+0000","last_clean":"2024-09-20T22:52:28.315304+0000","last_became_active":"2024-09-20T22:48:05.239344+0000","last_became_peered":"2024-09-20T22:48:05.239344+0000","last_unstale":"2024-09-20T22:52:28.315304+0000","last_undegraded":"2024-09-20T22:52:28.315304+0000","last_fullsized":"2024-09-20T22:52:28.315304+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"408'10","last_scrub_stamp":"2024-09-20T22:52:15.703464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:52:15.703464+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:23.273102+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":1247,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-09-20T22:52:28.282432+0000","last_change":"2024-09-20T22:51:54.066995+0000","last_active":"2024-09-20T22:52:28.282432+0000","last_peered":"2024-09-20T22:52:28.282432+0000","last_clean":"2024-09-20T22:52:28.282432+0000","last_became_active":"2024-09-20T22:47:49.227569+0000","last_became_peered":"2024-09-20T22:47:49.227569+0000","last_unstale":"2024-09-20T22:52:28.282432+0000","last_undegraded":"2024-09-20T22:52:28.282432+0000","last_fullsized":"2024-09-20T22:52:28.282432+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:54.066913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:51:54.066913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:18.059705+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":1274,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-09-20T22:52:28.285691+0000","last_change":"2024-09-20T22:51:58.129589+0000","last_active":"2024-09-20T22:52:28.285691+0000","last_peered":"2024-09-20T22:52:28.285691+0000","last_clean":"2024-09-20T22:52:28.285691+0000","last_became_active":"2024-09-20T22:47:49.213292+0000","last_became_peered":"2024-09-20T22:47:49.213292+0000","last_unstale":"2024-09-20T22:52:28.285691+0000","last_undegraded":"2024-09-20T22:52:28.285691+0000","last_fullsized":"2024-09-20T22:52:28.285691+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:58.129502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:51:58.129502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:11.276401+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"625'12","reported_seq":1274,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-09-20T22:52:28.277543+0000","last_change":"2024-09-20T22:52:14.322926+0000","last_active":"2024-09-20T22:52:28.277543+0000","last_peered":"2024-09-20T22:52:28.277543+0000","last_clean":"2024-09-20T22:52:28.277543+0000","last_became_active":"2024-09-20T22:43:09.116887+0000","last_became_peered":"2024-09-20T22:43:09.116887+0000","last_unstale":"2024-09-20T22:52:28.277543+0000","last_undegraded":"2024-09-20T22:52:28.277543+0000","last_fullsized":"2024-09-20T22:52:28.277543+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'12","last_scrub_stamp":"2024-09-20T22:52:14.322811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:52:14.322811+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:23.704685+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":1245,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-09-20T22:52:28.283448+0000","last_change":"2024-09-20T22:51:20.293964+0000","last_active":"2024-09-20T22:52:28.283448+0000","last_peered":"2024-09-20T22:52:28.283448+0000","last_clean":"2024-09-20T22:52:28.283448+0000","last_became_active":"2024-09-20T22:51:20.293232+0000","last_became_peered":"2024-09-20T22:51:20.293232+0000","last_unstale":"2024-09-20T22:52:28.283448+0000","last_undegraded":"2024-09-20T22:52:28.283448+0000","last_fullsized":"2024-09-20T22:52:28.283448+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:50:22.597796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:09.705632+0000","last_clean_scrub_stamp":"2024-09-20T22:50:22.597796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"74'1","reported_seq":1255,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-09-20T22:52:28.315044+0000","last_change":"2024-09-20T22:51:05.527388+0000","last_active":"2024-09-20T22:52:28.315044+0000","last_peered":"2024-09-20T22:52:28.315044+0000","last_clean":"2024-09-20T22:52:28.315044+0000","last_became_active":"2024-09-20T22:48:05.237882+0000","last_became_peered":"2024-09-20T22:48:05.237882+0000","last_unstale":"2024-09-20T22:52:28.315044+0000","last_undegraded":"2024-09-20T22:52:28.315044+0000","last_fullsized":"2024-09-20T22:52:28.315044+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-09-20T22:51:05.527264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:51:05.527264+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:28.653508+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"625'15","reported_seq":605,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-09-20T22:52:28.274605+0000","last_change":"2024-09-20T22:51:52.816826+0000","last_active":"2024-09-20T22:52:28.274605+0000","last_peered":"2024-09-20T22:52:28.274605+0000","last_clean":"2024-09-20T22:52:28.274605+0000","last_became_active":"2024-09-20T22:47:49.231315+0000","last_became_peered":"2024-09-20T22:47:49.231315+0000","last_unstale":"2024-09-20T22:52:28.274605+0000","last_undegraded":"2024-09-20T22:52:28.274605+0000","last_fullsized":"2024-09-20T22:52:28.274605+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'15","last_scrub_stamp":"2024-09-20T22:51:52.816714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:51:52.816714+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:55.354430+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":1270,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-09-20T22:52:28.314246+0000","last_change":"2024-09-20T22:51:12.541531+0000","last_active":"2024-09-20T22:52:28.314246+0000","last_peered":"2024-09-20T22:52:28.314246+0000","last_clean":"2024-09-20T22:52:28.314246+0000","last_became_active":"2024-09-20T22:48:05.238801+0000","last_became_peered":"2024-09-20T22:48:05.238801+0000","last_unstale":"2024-09-20T22:52:28.314246+0000","last_undegraded":"2024-09-20T22:52:28.314246+0000","last_fullsized":"2024-09-20T22:52:28.314246+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:12.541457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:51:12.541457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:38.408641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":1244,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-09-20T22:52:28.277305+0000","last_change":"2024-09-20T22:52:20.410227+0000","last_active":"2024-09-20T22:52:28.277305+0000","last_peered":"2024-09-20T22:52:28.277305+0000","last_clean":"2024-09-20T22:52:28.277305+0000","last_became_active":"2024-09-20T22:51:20.279397+0000","last_became_peered":"2024-09-20T22:51:20.279397+0000","last_unstale":"2024-09-20T22:52:28.277305+0000","last_undegraded":"2024-09-20T22:52:28.277305+0000","last_fullsized":"2024-09-20T22:52:28.277305+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:20.410164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:52:20.410164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:50.341745+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":1274,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-09-20T22:52:28.275151+0000","last_change":"2024-09-20T22:52:19.253411+0000","last_active":"2024-09-20T22:52:28.275151+0000","last_peered":"2024-09-20T22:52:28.275151+0000","last_clean":"2024-09-20T22:52:28.275151+0000","last_became_active":"2024-09-20T22:48:05.239710+0000","last_became_peered":"2024-09-20T22:48:05.239710+0000","last_unstale":"2024-09-20T22:52:28.275151+0000","last_undegraded":"2024-09-20T22:52:28.275151+0000","last_fullsized":"2024-09-20T22:52:28.275151+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:19.253343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:52:19.253343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:44.397693+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"625'15","reported_seq":1268,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-09-20T22:52:28.283068+0000","last_change":"2024-09-20T22:51:05.675215+0000","last_active":"2024-09-20T22:52:28.283068+0000","last_peered":"2024-09-20T22:52:28.283068+0000","last_clean":"2024-09-20T22:52:28.283068+0000","last_became_active":"2024-09-20T22:41:40.252486+0000","last_became_peered":"2024-09-20T22:41:40.252486+0000","last_unstale":"2024-09-20T22:52:28.283068+0000","last_undegraded":"2024-09-20T22:52:28.283068+0000","last_fullsized":"2024-09-20T22:52:28.283068+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-09-20T22:51:05.675095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:51:05.675095+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":595,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-09-20T22:52:28.273221+0000","last_change":"2024-09-20T22:52:13.735451+0000","last_active":"2024-09-20T22:52:28.273221+0000","last_peered":"2024-09-20T22:52:28.273221+0000","last_clean":"2024-09-20T22:52:28.273221+0000","last_became_active":"2024-09-20T22:47:49.229936+0000","last_became_peered":"2024-09-20T22:47:49.229936+0000","last_unstale":"2024-09-20T22:52:28.273221+0000","last_undegraded":"2024-09-20T22:52:28.273221+0000","last_fullsized":"2024-09-20T22:52:28.273221+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:13.735382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:52:13.735382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:22.644334+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"74'8","reported_seq":1372,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-09-20T22:52:28.277029+0000","last_change":"2024-09-20T22:50:38.269118+0000","last_active":"2024-09-20T22:52:28.277029+0000","last_peered":"2024-09-20T22:52:28.277029+0000","last_clean":"2024-09-20T22:52:28.277029+0000","last_became_active":"2024-09-20T22:41:42.186352+0000","last_became_peered":"2024-09-20T22:41:42.186352+0000","last_unstale":"2024-09-20T22:52:28.277029+0000","last_undegraded":"2024-09-20T22:52:28.277029+0000","last_fullsized":"2024-09-20T22:52:28.277029+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-09-20T22:50:38.269010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:50:38.269010+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:28.385159+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":554,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-09-20T22:52:28.275328+0000","last_change":"2024-09-20T22:52:16.623463+0000","last_active":"2024-09-20T22:52:28.275328+0000","last_peered":"2024-09-20T22:52:28.275328+0000","last_clean":"2024-09-20T22:52:28.275328+0000","last_became_active":"2024-09-20T22:48:05.249656+0000","last_became_peered":"2024-09-20T22:48:05.249656+0000","last_unstale":"2024-09-20T22:52:28.275328+0000","last_undegraded":"2024-09-20T22:52:28.275328+0000","last_fullsized":"2024-09-20T22:52:28.275328+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:16.623368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:52:16.623368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:22.519570+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"625'20","reported_seq":1282,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-09-20T22:52:28.276582+0000","last_change":"2024-09-20T22:51:20.298112+0000","last_active":"2024-09-20T22:52:28.276582+0000","last_peered":"2024-09-20T22:52:28.276582+0000","last_clean":"2024-09-20T22:52:28.276582+0000","last_became_active":"2024-09-20T22:51:20.281859+0000","last_became_peered":"2024-09-20T22:51:20.281859+0000","last_unstale":"2024-09-20T22:52:28.276582+0000","last_undegraded":"2024-09-20T22:52:28.276582+0000","last_fullsized":"2024-09-20T22:52:28.276582+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-09-20T22:50:15.240945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:50:15.240945+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:28.385035+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":1266,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-09-20T22:52:28.314675+0000","last_change":"2024-09-20T22:52:20.621799+0000","last_active":"2024-09-20T22:52:28.314675+0000","last_peered":"2024-09-20T22:52:28.314675+0000","last_clean":"2024-09-20T22:52:28.314675+0000","last_became_active":"2024-09-20T22:48:05.240231+0000","last_became_peered":"2024-09-20T22:48:05.240231+0000","last_unstale":"2024-09-20T22:52:28.314675+0000","last_undegraded":"2024-09-20T22:52:28.314675+0000","last_fullsized":"2024-09-20T22:52:28.314675+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:20.621718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:52:20.621718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:30.275424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"600'11","reported_seq":1412,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-09-20T22:52:28.282807+0000","last_change":"2024-09-20T22:51:20.303774+0000","last_active":"2024-09-20T22:52:28.282807+0000","last_peered":"2024-09-20T22:52:28.282807+0000","last_clean":"2024-09-20T22:52:28.282807+0000","last_became_active":"2024-09-20T22:51:20.283393+0000","last_became_peered":"2024-09-20T22:51:20.283393+0000","last_unstale":"2024-09-20T22:52:28.282807+0000","last_undegraded":"2024-09-20T22:52:28.282807+0000","last_fullsized":"2024-09-20T22:52:28.282807+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'10","last_scrub_stamp":"2024-09-20T22:49:51.715471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:49:51.715471+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":578,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-09-20T22:52:28.274529+0000","last_change":"2024-09-20T22:50:21.853169+0000","last_active":"2024-09-20T22:52:28.274529+0000","last_peered":"2024-09-20T22:52:28.274529+0000","last_clean":"2024-09-20T22:52:28.274529+0000","last_became_active":"2024-09-20T22:47:49.230674+0000","last_became_peered":"2024-09-20T22:47:49.230674+0000","last_unstale":"2024-09-20T22:52:28.274529+0000","last_undegraded":"2024-09-20T22:52:28.274529+0000","last_fullsized":"2024-09-20T22:52:28.274529+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:50:21.853070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:50:21.853070+0000","last_clean_scrub_stamp":"2024-09-20T22:50:21.853070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:28.623037+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"625'19","reported_seq":596,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-09-20T22:52:28.274652+0000","last_change":"2024-09-20T22:51:20.317969+0000","last_active":"2024-09-20T22:52:28.274652+0000","last_peered":"2024-09-20T22:52:28.274652+0000","last_clean":"2024-09-20T22:52:28.274652+0000","last_became_active":"2024-09-20T22:51:20.289584+0000","last_became_peered":"2024-09-20T22:51:20.289584+0000","last_unstale":"2024-09-20T22:52:28.274652+0000","last_undegraded":"2024-09-20T22:52:28.274652+0000","last_fullsized":"2024-09-20T22:52:28.274652+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-09-20T22:51:07.926382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:51:07.926382+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:30.537714+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":554,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-09-20T22:52:28.274438+0000","last_change":"2024-09-20T22:51:08.874933+0000","last_active":"2024-09-20T22:52:28.274438+0000","last_peered":"2024-09-20T22:52:28.274438+0000","last_clean":"2024-09-20T22:52:28.274438+0000","last_became_active":"2024-09-20T22:48:05.252026+0000","last_became_peered":"2024-09-20T22:48:05.252026+0000","last_unstale":"2024-09-20T22:52:28.274438+0000","last_undegraded":"2024-09-20T22:52:28.274438+0000","last_fullsized":"2024-09-20T22:52:28.274438+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:08.874812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:51:08.874812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:31.564244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":148,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-09-20T22:52:28.282580+0000","last_change":"2024-09-20T22:51:20.291857+0000","last_active":"2024-09-20T22:52:28.282580+0000","last_peered":"2024-09-20T22:52:28.282580+0000","last_clean":"2024-09-20T22:52:28.282580+0000","last_became_active":"2024-09-20T22:51:20.291264+0000","last_became_peered":"2024-09-20T22:51:20.291264+0000","last_unstale":"2024-09-20T22:52:28.282580+0000","last_undegraded":"2024-09-20T22:52:28.282580+0000","last_fullsized":"2024-09-20T22:52:28.282580+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:06.651167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:51:06.651167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:34.034653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":1238,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-09-20T22:52:28.274447+0000","last_change":"2024-09-20T22:50:20.207052+0000","last_active":"2024-09-20T22:52:28.274447+0000","last_peered":"2024-09-20T22:52:28.274447+0000","last_clean":"2024-09-20T22:52:28.274447+0000","last_became_active":"2024-09-20T22:47:49.216141+0000","last_became_peered":"2024-09-20T22:47:49.216141+0000","last_unstale":"2024-09-20T22:52:28.274447+0000","last_undegraded":"2024-09-20T22:52:28.274447+0000","last_fullsized":"2024-09-20T22:52:28.274447+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:50:20.206984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:50:20.206984+0000","last_clean_scrub_stamp":"2024-09-20T22:50:20.206984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:28.384988+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"625'19","reported_seq":892,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-09-20T22:52:28.286210+0000","last_change":"2024-09-20T22:51:25.879860+0000","last_active":"2024-09-20T22:52:28.286210+0000","last_peered":"2024-09-20T22:52:28.286210+0000","last_clean":"2024-09-20T22:52:28.286210+0000","last_became_active":"2024-09-20T22:51:20.283868+0000","last_became_peered":"2024-09-20T22:51:20.283868+0000","last_unstale":"2024-09-20T22:52:28.286210+0000","last_undegraded":"2024-09-20T22:52:28.286210+0000","last_fullsized":"2024-09-20T22:52:28.286210+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"404'13","last_scrub_stamp":"2024-09-20T22:51:25.879750+0000","last_deep_scrub":"404'13","last_deep_scrub_stamp":"2024-09-20T22:48:04.276282+0000","last_clean_scrub_stamp":"2024-09-20T22:51:25.879750+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:35.823216+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,0],"acting":[6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":1273,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-09-20T22:52:28.315505+0000","last_change":"2024-09-20T22:49:56.396435+0000","last_active":"2024-09-20T22:52:28.315505+0000","last_peered":"2024-09-20T22:52:28.315505+0000","last_clean":"2024-09-20T22:52:28.315505+0000","last_became_active":"2024-09-20T22:48:05.238812+0000","last_became_peered":"2024-09-20T22:48:05.238812+0000","last_unstale":"2024-09-20T22:52:28.315505+0000","last_undegraded":"2024-09-20T22:52:28.315505+0000","last_fullsized":"2024-09-20T22:52:28.315505+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:56.396347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:31.138067+0000","last_clean_scrub_stamp":"2024-09-20T22:49:56.396347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:28.653459+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":1248,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-09-20T22:52:28.275135+0000","last_change":"2024-09-20T22:51:20.288308+0000","last_active":"2024-09-20T22:52:28.275135+0000","last_peered":"2024-09-20T22:52:28.275135+0000","last_clean":"2024-09-20T22:52:28.275135+0000","last_became_active":"2024-09-20T22:51:20.288000+0000","last_became_peered":"2024-09-20T22:51:20.288000+0000","last_unstale":"2024-09-20T22:52:28.275135+0000","last_undegraded":"2024-09-20T22:52:28.275135+0000","last_fullsized":"2024-09-20T22:52:28.275135+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:50:44.313182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:50:44.313182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:28.385112+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":585,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-09-20T22:52:28.272513+0000","last_change":"2024-09-20T22:51:20.278471+0000","last_active":"2024-09-20T22:52:28.272513+0000","last_peered":"2024-09-20T22:52:28.272513+0000","last_clean":"2024-09-20T22:52:28.272513+0000","last_became_active":"2024-09-20T22:51:20.278147+0000","last_became_peered":"2024-09-20T22:51:20.278147+0000","last_unstale":"2024-09-20T22:52:28.272513+0000","last_undegraded":"2024-09-20T22:52:28.272513+0000","last_fullsized":"2024-09-20T22:52:28.272513+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:50:35.895483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:50:35.895483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"625'24","reported_seq":1069,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-09-20T22:52:28.282135+0000","last_change":"2024-09-20T22:50:39.665781+0000","last_active":"2024-09-20T22:52:28.282135+0000","last_peered":"2024-09-20T22:52:28.282135+0000","last_clean":"2024-09-20T22:52:28.282135+0000","last_became_active":"2024-09-20T22:44:09.170785+0000","last_became_peered":"2024-09-20T22:44:09.170785+0000","last_unstale":"2024-09-20T22:52:28.282135+0000","last_undegraded":"2024-09-20T22:52:28.282135+0000","last_fullsized":"2024-09-20T22:52:28.282135+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"470'16","last_scrub_stamp":"2024-09-20T22:50:39.665682+0000","last_deep_scrub":"470'16","last_deep_scrub_stamp":"2024-09-20T22:49:10.659868+0000","last_clean_scrub_stamp":"2024-09-20T22:50:39.665682+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":1253,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-09-20T22:52:28.283556+0000","last_change":"2024-09-20T22:51:20.294181+0000","last_active":"2024-09-20T22:52:28.283556+0000","last_peered":"2024-09-20T22:52:28.283556+0000","last_clean":"2024-09-20T22:52:28.283556+0000","last_became_active":"2024-09-20T22:51:20.293645+0000","last_became_peered":"2024-09-20T22:51:20.293645+0000","last_unstale":"2024-09-20T22:52:28.283556+0000","last_undegraded":"2024-09-20T22:52:28.283556+0000","last_fullsized":"2024-09-20T22:52:28.283556+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:50:13.688169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:45:24.765414+0000","last_clean_scrub_stamp":"2024-09-20T22:50:13.688169+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":1262,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-09-20T22:52:28.313756+0000","last_change":"2024-09-20T22:51:10.610718+0000","last_active":"2024-09-20T22:52:28.313756+0000","last_peered":"2024-09-20T22:52:28.313756+0000","last_clean":"2024-09-20T22:52:28.313756+0000","last_became_active":"2024-09-20T22:48:05.238334+0000","last_became_peered":"2024-09-20T22:48:05.238334+0000","last_unstale":"2024-09-20T22:52:28.313756+0000","last_undegraded":"2024-09-20T22:52:28.313756+0000","last_fullsized":"2024-09-20T22:52:28.313756+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:10.610588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:50.174861+0000","last_clean_scrub_stamp":"2024-09-20T22:51:10.610588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:28.653550+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":1238,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-09-20T22:52:28.275253+0000","last_change":"2024-09-20T22:51:04.379662+0000","last_active":"2024-09-20T22:52:28.275253+0000","last_peered":"2024-09-20T22:52:28.275253+0000","last_clean":"2024-09-20T22:52:28.275253+0000","last_became_active":"2024-09-20T22:47:49.214487+0000","last_became_peered":"2024-09-20T22:47:49.214487+0000","last_unstale":"2024-09-20T22:52:28.275253+0000","last_undegraded":"2024-09-20T22:52:28.275253+0000","last_fullsized":"2024-09-20T22:52:28.275253+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:04.379558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:51:04.379558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:28.385201+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"625'8","reported_seq":1258,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-09-20T22:52:28.274581+0000","last_change":"2024-09-20T22:51:20.314500+0000","last_active":"2024-09-20T22:52:28.274581+0000","last_peered":"2024-09-20T22:52:28.274581+0000","last_clean":"2024-09-20T22:52:28.274581+0000","last_became_active":"2024-09-20T22:51:20.290269+0000","last_became_peered":"2024-09-20T22:51:20.290269+0000","last_unstale":"2024-09-20T22:52:28.274581+0000","last_undegraded":"2024-09-20T22:52:28.274581+0000","last_fullsized":"2024-09-20T22:52:28.274581+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'4","last_scrub_stamp":"2024-09-20T22:50:40.292964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:50:40.292964+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:28.385052+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":586,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-09-20T22:52:28.274362+0000","last_change":"2024-09-20T22:50:43.756048+0000","last_active":"2024-09-20T22:52:28.274362+0000","last_peered":"2024-09-20T22:52:28.274362+0000","last_clean":"2024-09-20T22:52:28.274362+0000","last_became_active":"2024-09-20T22:47:49.231964+0000","last_became_peered":"2024-09-20T22:47:49.231964+0000","last_unstale":"2024-09-20T22:52:28.274362+0000","last_undegraded":"2024-09-20T22:52:28.274362+0000","last_fullsized":"2024-09-20T22:52:28.274362+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:50:43.755981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:50:43.755981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:28.623068+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":1278,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-09-20T22:52:28.285636+0000","last_change":"2024-09-20T22:52:20.955721+0000","last_active":"2024-09-20T22:52:28.285636+0000","last_peered":"2024-09-20T22:52:28.285636+0000","last_clean":"2024-09-20T22:52:28.285636+0000","last_became_active":"2024-09-20T22:48:05.244659+0000","last_became_peered":"2024-09-20T22:48:05.244659+0000","last_unstale":"2024-09-20T22:52:28.285636+0000","last_undegraded":"2024-09-20T22:52:28.285636+0000","last_fullsized":"2024-09-20T22:52:28.285636+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:20.955598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:52:20.955598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:32.677353+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":1254,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-09-20T22:52:28.315193+0000","last_change":"2024-09-20T22:51:01.446349+0000","last_active":"2024-09-20T22:52:28.315193+0000","last_peered":"2024-09-20T22:52:28.315193+0000","last_clean":"2024-09-20T22:52:28.315193+0000","last_became_active":"2024-09-20T22:48:05.237765+0000","last_became_peered":"2024-09-20T22:48:05.237765+0000","last_unstale":"2024-09-20T22:52:28.315193+0000","last_undegraded":"2024-09-20T22:52:28.315193+0000","last_fullsized":"2024-09-20T22:52:28.315193+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:01.446278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:10.233487+0000","last_clean_scrub_stamp":"2024-09-20T22:51:01.446278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:28.653522+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":851,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-09-20T22:52:28.282888+0000","last_change":"2024-09-20T22:51:29.904430+0000","last_active":"2024-09-20T22:52:28.282888+0000","last_peered":"2024-09-20T22:52:28.282888+0000","last_clean":"2024-09-20T22:52:28.282888+0000","last_became_active":"2024-09-20T22:51:20.297739+0000","last_became_peered":"2024-09-20T22:51:20.297739+0000","last_unstale":"2024-09-20T22:52:28.282888+0000","last_undegraded":"2024-09-20T22:52:28.282888+0000","last_fullsized":"2024-09-20T22:52:28.282888+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:29.904355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:51:29.904355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:36.892277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":1270,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-09-20T22:52:28.274343+0000","last_change":"2024-09-20T22:51:20.275893+0000","last_active":"2024-09-20T22:52:28.274343+0000","last_peered":"2024-09-20T22:52:28.274343+0000","last_clean":"2024-09-20T22:52:28.274343+0000","last_became_active":"2024-09-20T22:51:20.274891+0000","last_became_peered":"2024-09-20T22:51:20.274891+0000","last_unstale":"2024-09-20T22:52:28.274343+0000","last_undegraded":"2024-09-20T22:52:28.274343+0000","last_fullsized":"2024-09-20T22:52:28.274343+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:50:41.281470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:50:41.281470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:52:28.385098+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"625'19","reported_seq":1311,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-09-20T22:52:28.287058+0000","last_change":"2024-09-20T22:52:16.953034+0000","last_active":"2024-09-20T22:52:28.287058+0000","last_peered":"2024-09-20T22:52:28.287058+0000","last_clean":"2024-09-20T22:52:28.287058+0000","last_became_active":"2024-09-20T22:51:20.284937+0000","last_became_peered":"2024-09-20T22:51:20.284937+0000","last_unstale":"2024-09-20T22:52:28.287058+0000","last_undegraded":"2024-09-20T22:52:28.287058+0000","last_fullsized":"2024-09-20T22:52:28.287058+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'19","last_scrub_stamp":"2024-09-20T22:52:16.952955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:52:16.952955+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:27.426198+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":587,"reported_epoch":668,"state":"active+clean","last_fresh":"2024-09-20T22:52:28.274141+0000","last_change":"2024-09-20T22:52:18.698154+0000","last_active":"2024-09-20T22:52:28.274141+0000","last_peered":"2024-09-20T22:52:28.274141+0000","last_clean":"2024-09-20T22:52:28.274141+0000","last_became_active":"2024-09-20T22:47:49.220971+0000","last_became_peered":"2024-09-20T22:47:49.220971+0000","last_unstale":"2024-09-20T22:52:28.274141+0000","last_undegraded":"2024-09-20T22:52:28.274141+0000","last_fullsized":"2024-09-20T22:52:28.274141+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:18.698087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:51.667341+0000","last_clean_scrub_stamp":"2024-09-20T22:52:18.698087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:47.971098+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":617590212,"num_objects":284,"num_object_clones":234,"num_object_copies":852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":284,"num_whiteouts":12,"num_read":4561,"num_read_kb":4769826,"num_write":3287,"num_write_kb":1221673,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":790,"num_bytes_recovered":519843158,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1363619840,"data_stored":1965688083,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8681,"ondisk_log_size":8681,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85,"ondisk_log_size":85,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1109,"num_read_kb":866,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":312,"num_bytes_recovered":7464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":499712,"data_stored":13224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":739,"ondisk_log_size":739,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":1034,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":4931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":640,"num_read_kb":635,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1186,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":384,"ondisk_log_size":384,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070598,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":304616,"kb_used_data":255372,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":93440536,"statfs":{"total":95995035648,"available":95683108864,"internally_reserved":0,"allocated":261500928,"data_stored":333727229,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":50395594},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430348,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":231424,"kb_used_data":182188,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":93513728,"statfs":{"total":95995035648,"available":95758057472,"internally_reserved":0,"allocated":186560512,"data_stored":269081281,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691985,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":179644,"kb_used_data":122084,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":93565508,"statfs":{"total":95995035648,"available":95811080192,"internally_reserved":0,"allocated":125014016,"data_stored":183839141,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986326,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":146852,"kb_used_data":115656,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93598300,"statfs":{"total":95995035648,"available":95844659200,"internally_reserved":0,"allocated":118431744,"data_stored":169068584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182556,"num_pgs":76,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":330328,"kb_used_data":281092,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":93414824,"statfs":{"total":95995035648,"available":95656779776,"internally_reserved":0,"allocated":287838208,"data_stored":410286342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":50395591},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411489,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":292424,"kb_used_data":226476,"kb_used_omap":1,"kb_used_meta":65918,"kb_avail":93452728,"statfs":{"total":95995035648,"available":95695593472,"internally_reserved":0,"allocated":231911424,"data_stored":323386162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":67500485},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":0,"seq":55834575014,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738539,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":222880,"kb_used_data":177804,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":93522272,"statfs":{"total":95995035648,"available":95766806528,"internally_reserved":0,"allocated":182071296,"data_stored":275408553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":176078848,"data_stored":271775873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":230498304,"data_stored":328704455,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":281251840,"data_stored":406062600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":112451584,"data_stored":165436865,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":121712640,"data_stored":185550987,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":184934400,"data_stored":275122592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":256692224,"data_stored":333034711,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-20T22:52:31.159 INFO:tasks.cephadm.ceph_manager.ceph:would drop below min_pgs, proj 1, currently 11 2024-09-20T22:52:31.164 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:52:31.185 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:31.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:31.186 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:52:31.203 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:52:31.232 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:52:31.287 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:52:31.308 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:31.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:31.309 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:52:31.351 INFO:tasks.rados.rados.0.smithi084.stdout:2629: done (1 left) 2024-09-20T22:52:31.352 INFO:tasks.rados.rados.0.smithi084.stdout:2630: snap_create 2024-09-20T22:52:31.352 INFO:tasks.rados.rados.0.smithi084.stdout:2627: finishing copy_from racing read to smithi08477203-40 2024-09-20T22:52:31.352 INFO:tasks.rados.rados.0.smithi084.stdout:2627: finishing copy_from to smithi08477203-40 2024-09-20T22:52:31.352 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 40 v 763 (ObjNum 872 snap 280 seq_num 872) dirty exists 2024-09-20T22:52:31.410 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:52:31.432 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:31.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:31.433 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:52:31.533 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:52:31.555 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:31.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:31.556 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:52:31.576 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:31 smithi084 ceph-mon[40024]: pgmap v906: 158 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 153 active+clean; 590 MiB data, 1.6 GiB used, 624 GiB / 626 GiB avail; 4.0 MiB/s rd, 5.4 MiB/s wr, 23 op/s 2024-09-20T22:52:31.577 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:31 smithi084 ceph-mon[40024]: from='client.16641 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T22:52:31.577 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:31 smithi084 ceph-mon[40024]: osdmap e670: 8 total, 8 up, 7 in 2024-09-20T22:52:31.577 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:31 smithi084 ceph-mon[29774]: pgmap v906: 158 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 153 active+clean; 590 MiB data, 1.6 GiB used, 624 GiB / 626 GiB avail; 4.0 MiB/s rd, 5.4 MiB/s wr, 23 op/s 2024-09-20T22:52:31.577 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:31 smithi084 ceph-mon[29774]: from='client.16641 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T22:52:31.577 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:31 smithi084 ceph-mon[29774]: osdmap e670: 8 total, 8 up, 7 in 2024-09-20T22:52:31.657 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:52:31.730 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:31.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:31.731 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:52:31.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:31 smithi154 ceph-mon[33264]: pgmap v906: 158 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 153 active+clean; 590 MiB data, 1.6 GiB used, 624 GiB / 626 GiB avail; 4.0 MiB/s rd, 5.4 MiB/s wr, 23 op/s 2024-09-20T22:52:31.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:31 smithi154 ceph-mon[33264]: from='client.16641 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T22:52:31.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:31 smithi154 ceph-mon[33264]: osdmap e670: 8 total, 8 up, 7 in 2024-09-20T22:52:31.832 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:52:31.854 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:31.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:31.855 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:52:31.956 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:52:31.982 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:31.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:31.982 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:52:32.016 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:52:32.016 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:52:32.033 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:52:32.033 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:52:32.050 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:52:32.050 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:52:32.064 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-20T22:52:32.064 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T22:52:32.081 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:52:32.082 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:52:32.083 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:52:32.099 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:52:32.099 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:52:32.108 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:32.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:32.109 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:52:32.116 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:52:32.116 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:52:32.133 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:52:32.133 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:52:32.209 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:52:32.210 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-20T22:52:32.211 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2024-09-20T22:52:32.211 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2024-09-20T22:52:32.211 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-20T22:52:32.231 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:32.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:32.232 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:52:32.333 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:52:32.357 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:32.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:32.358 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:52:32.360 INFO:tasks.rados.rados.0.smithi084.stdout:2627: done (1 left) 2024-09-20T22:52:32.360 INFO:tasks.rados.rados.0.smithi084.stdout:2630: done (0 left) 2024-09-20T22:52:32.361 INFO:tasks.rados.rados.0.smithi084.stdout:2631: copy_from oid 20 from oid 37 current snap is 299 2024-09-20T22:52:32.361 INFO:tasks.rados.rados.0.smithi084.stdout:2632: rollback oid 25 current snap is 299 2024-09-20T22:52:32.361 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 25 to 270 2024-09-20T22:52:32.361 INFO:tasks.rados.rados.0.smithi084.stdout:2633: rollback oid 28 current snap is 299 2024-09-20T22:52:32.361 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 28 to 270 2024-09-20T22:52:32.361 INFO:tasks.rados.rados.0.smithi084.stdout:2634: read oid 14 snap -1 2024-09-20T22:52:32.361 INFO:tasks.rados.rados.0.smithi084.stdout:2634: expect (ObjNum 618 snap 184 seq_num 618) 2024-09-20T22:52:32.361 INFO:tasks.rados.rados.0.smithi084.stdout:2635: read oid 41 snap -1 2024-09-20T22:52:32.361 INFO:tasks.rados.rados.0.smithi084.stdout:2635: expect (ObjNum 779 snap 252 seq_num 779) 2024-09-20T22:52:32.361 INFO:tasks.rados.rados.0.smithi084.stdout:2636: snap_create 2024-09-20T22:52:32.365 INFO:tasks.rados.rados.0.smithi084.stdout:2631: finishing copy_from to smithi08477203-20 2024-09-20T22:52:32.365 INFO:tasks.rados.rados.0.smithi084.stdout:2631: got expected ENOENT (src dne) 2024-09-20T22:52:32.366 INFO:tasks.rados.rados.0.smithi084.stdout:2631: finishing copy_from racing read to smithi08477203-20 2024-09-20T22:52:32.366 INFO:tasks.rados.rados.0.smithi084.stdout:2635: expect (ObjNum 779 snap 252 seq_num 779) 2024-09-20T22:52:32.459 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:52:32.470 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:32 smithi084 ceph-mon[40024]: osdmap e671: 8 total, 8 up, 7 in 2024-09-20T22:52:32.470 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:32 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/1126467080' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:52:32.470 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:32 smithi084 ceph-mon[29774]: osdmap e671: 8 total, 8 up, 7 in 2024-09-20T22:52:32.470 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:32 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/1126467080' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:52:32.471 INFO:tasks.rados.rados.0.smithi084.stdout:2632: finishing rollback tid 0 to smithi08477203-25 2024-09-20T22:52:32.471 INFO:tasks.rados.rados.0.smithi084.stdout:2634: expect (ObjNum 618 snap 184 seq_num 618) 2024-09-20T22:52:32.482 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:32.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:32.483 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:52:32.491 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd unset noscrub 2024-09-20T22:52:32.583 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:52:32.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:32.606 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:32.606 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:52:32.610 INFO:tasks.rados.rados.0.smithi084.stdout:2632: finishing rollback tid 1 to smithi08477203-25 2024-09-20T22:52:32.610 INFO:tasks.rados.rados.0.smithi084.stdout:2632: finishing rollback tid 2 to smithi08477203-25 2024-09-20T22:52:32.611 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 25 v 2019 (ObjNum 688 snap 212 seq_num 688) dirty exists 2024-09-20T22:52:32.611 INFO:tasks.rados.rados.0.smithi084.stdout:2633: finishing rollback tid 1 to smithi08477203-28 2024-09-20T22:52:32.611 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 28 v 2003 (ObjNum 289 snap 0 seq_num 0) dirty dne 2024-09-20T22:52:32.707 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:52:32.730 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:32.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:32.731 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:52:32.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:32 smithi154 ceph-mon[33264]: osdmap e671: 8 total, 8 up, 7 in 2024-09-20T22:52:32.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:32 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/1126467080' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:52:32.831 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:52:32.853 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:32.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:32.854 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:52:32.955 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:52:32.978 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:32.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:32.979 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:52:33.080 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:52:33.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:33.103 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:33.104 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:52:33.204 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:52:33.226 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:33.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:33.227 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:52:33.277 INFO:tasks.rados.rados.0.smithi084.stdout:2631: done (5 left) 2024-09-20T22:52:33.277 INFO:tasks.rados.rados.0.smithi084.stdout:2632: done (4 left) 2024-09-20T22:52:33.277 INFO:tasks.rados.rados.0.smithi084.stdout:2633: done (3 left) 2024-09-20T22:52:33.277 INFO:tasks.rados.rados.0.smithi084.stdout:2634: done (2 left) 2024-09-20T22:52:33.277 INFO:tasks.rados.rados.0.smithi084.stdout:2635: done (1 left) 2024-09-20T22:52:33.277 INFO:tasks.rados.rados.0.smithi084.stdout:2636: done (0 left) 2024-09-20T22:52:33.277 INFO:tasks.rados.rados.0.smithi084.stdout:2637: read oid 20 snap -1 2024-09-20T22:52:33.277 INFO:tasks.rados.rados.0.smithi084.stdout:2637: expect (ObjNum 874 snap 280 seq_num 874) 2024-09-20T22:52:33.278 INFO:tasks.rados.rados.0.smithi084.stdout:2638: delete oid 9 current snap is 300 2024-09-20T22:52:33.281 INFO:tasks.rados.rados.0.smithi084.stdout:2638: done (1 left) 2024-09-20T22:52:33.281 INFO:tasks.rados.rados.0.smithi084.stdout:2639: write (excl) oid 41 current snap is 300 2024-09-20T22:52:33.281 INFO:tasks.rados.rados.0.smithi084.stdout:2639: seq_num 913 ranges {544791=664697,1693354=595308,2689310=445583} 2024-09-20T22:52:33.288 INFO:tasks.rados.rados.0.smithi084.stdout:2639: writing smithi08477203-41 from 544791 to 1209488 tid 1 2024-09-20T22:52:33.294 INFO:tasks.rados.rados.0.smithi084.stdout:2639: writing smithi08477203-41 from 1693354 to 2288662 tid 2 2024-09-20T22:52:33.299 INFO:tasks.rados.rados.0.smithi084.stdout:2639: writing smithi08477203-41 from 2689310 to 3134893 tid 3 2024-09-20T22:52:33.299 INFO:tasks.rados.rados.0.smithi084.stdout:2640: rollback oid 42 current snap is 300 2024-09-20T22:52:33.299 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 42 to 246 2024-09-20T22:52:33.299 INFO:tasks.rados.rados.0.smithi084.stdout:2637: expect (ObjNum 874 snap 280 seq_num 874) 2024-09-20T22:52:33.328 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:52:33.348 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:33.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:33.349 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:52:33.450 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:52:33.472 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:33.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:33.473 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:52:33.502 INFO:tasks.rados.rados.0.smithi084.stdout:2639: finishing write tid 1 to smithi08477203-41 2024-09-20T22:52:33.502 INFO:tasks.rados.rados.0.smithi084.stdout:2639: finishing write tid 2 to smithi08477203-41 2024-09-20T22:52:33.502 INFO:tasks.rados.rados.0.smithi084.stdout:2640: finishing rollback tid 0 to smithi08477203-42 2024-09-20T22:52:33.503 INFO:tasks.rados.rados.0.smithi084.stdout:2640: finishing rollback tid 1 to smithi08477203-42 2024-09-20T22:52:33.503 INFO:tasks.rados.rados.0.smithi084.stdout:2640: finishing rollback tid 2 to smithi08477203-42 2024-09-20T22:52:33.503 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 42 v 1224 (ObjNum 714 snap 224 seq_num 714) dirty exists 2024-09-20T22:52:33.503 INFO:tasks.rados.rados.0.smithi084.stdout:2637: done (2 left) 2024-09-20T22:52:33.503 INFO:tasks.rados.rados.0.smithi084.stdout:2640: done (1 left) 2024-09-20T22:52:33.503 INFO:tasks.rados.rados.0.smithi084.stdout:2641: delete oid 19 current snap is 300 2024-09-20T22:52:33.503 INFO:tasks.rados.rados.0.smithi084.stdout:2639: finishing write tid 3 to smithi08477203-41 2024-09-20T22:52:33.503 INFO:tasks.rados.rados.0.smithi084.stdout:2639: finishing write tid 4 to smithi08477203-41 2024-09-20T22:52:33.503 INFO:tasks.rados.rados.0.smithi084.stdout:2639: finishing write tid 5 to smithi08477203-41 2024-09-20T22:52:33.503 INFO:tasks.rados.rados.0.smithi084.stdout:2639: oid 41 updating version 0 to 1410 2024-09-20T22:52:33.503 INFO:tasks.rados.rados.0.smithi084.stdout:2639: oid 41 updating version 1410 to 1412 2024-09-20T22:52:33.503 INFO:tasks.rados.rados.0.smithi084.stdout:2639: oid 41 version 1412 is already newer than 1411 2024-09-20T22:52:33.503 INFO:tasks.rados.rados.0.smithi084.stdout:2639: oid 41 version 1412 is already newer than 1409 2024-09-20T22:52:33.503 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 41 v 1412 (ObjNum 913 snap 300 seq_num 913) dirty exists 2024-09-20T22:52:33.504 INFO:tasks.rados.rados.0.smithi084.stdout:2639: left oid 41 (ObjNum 913 snap 300 seq_num 913) 2024-09-20T22:52:33.505 INFO:tasks.rados.rados.0.smithi084.stdout:2639: done (1 left) 2024-09-20T22:52:33.505 INFO:tasks.rados.rados.0.smithi084.stdout:2641: done (0 left) 2024-09-20T22:52:33.506 INFO:tasks.rados.rados.0.smithi084.stdout:2642: snap_remove snap 270 2024-09-20T22:52:33.573 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:52:33.594 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:33.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:33.596 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:52:33.608 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:33 smithi084 ceph-mon[40024]: pgmap v909: 158 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 153 active+clean; 590 MiB data, 1.6 GiB used, 624 GiB / 626 GiB avail; 4.0 MiB/s rd, 5.4 MiB/s wr, 23 op/s 2024-09-20T22:52:33.608 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:33 smithi084 ceph-mon[40024]: osdmap e672: 8 total, 8 up, 7 in 2024-09-20T22:52:33.608 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:33 smithi084 ceph-mon[40024]: osdmap e673: 8 total, 8 up, 7 in 2024-09-20T22:52:33.608 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:33 smithi084 ceph-mon[29774]: pgmap v909: 158 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 153 active+clean; 590 MiB data, 1.6 GiB used, 624 GiB / 626 GiB avail; 4.0 MiB/s rd, 5.4 MiB/s wr, 23 op/s 2024-09-20T22:52:33.608 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:33 smithi084 ceph-mon[29774]: osdmap e672: 8 total, 8 up, 7 in 2024-09-20T22:52:33.608 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:33 smithi084 ceph-mon[29774]: osdmap e673: 8 total, 8 up, 7 in 2024-09-20T22:52:33.634 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.1 dump_blocked_ops 2024-09-20T22:52:33.696 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:52:33.699 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:52:33.719 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:33.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:33.720 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:52:33.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:33 smithi154 ceph-mon[33264]: pgmap v909: 158 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 153 active+clean; 590 MiB data, 1.6 GiB used, 624 GiB / 626 GiB avail; 4.0 MiB/s rd, 5.4 MiB/s wr, 23 op/s 2024-09-20T22:52:33.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:33 smithi154 ceph-mon[33264]: osdmap e672: 8 total, 8 up, 7 in 2024-09-20T22:52:33.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:33 smithi154 ceph-mon[33264]: osdmap e673: 8 total, 8 up, 7 in 2024-09-20T22:52:33.821 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:52:33.843 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:33.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:33.844 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:52:33.944 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:52:33.967 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:33.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:33.968 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:52:34.069 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:52:34.089 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:34.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:34.090 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:52:34.183 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:52:34.191 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:52:34.214 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:34.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:34.214 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:52:34.315 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:52:34.337 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:34.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:34.338 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:52:34.363 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is unset 2024-09-20T22:52:34.364 INFO:tasks.rados.rados.0.smithi084.stdout:2642: done (0 left) 2024-09-20T22:52:34.364 INFO:tasks.rados.rados.0.smithi084.stdout:2643: delete oid 40 current snap is 300 2024-09-20T22:52:34.370 INFO:tasks.rados.rados.0.smithi084.stdout:2643: done (0 left) 2024-09-20T22:52:34.371 INFO:tasks.rados.rados.0.smithi084.stdout:2644: copy_from oid 16 from oid 30 current snap is 300 2024-09-20T22:52:34.371 INFO:tasks.rados.rados.0.smithi084.stdout:2645: delete oid 22 current snap is 300 2024-09-20T22:52:34.372 INFO:tasks.rados.rados.0.smithi084.stdout:2645: done (1 left) 2024-09-20T22:52:34.372 INFO:tasks.rados.rados.0.smithi084.stdout:2646: snap_create 2024-09-20T22:52:34.382 INFO:tasks.rados.rados.0.smithi084.stdout:2644: finishing copy_from racing read to smithi08477203-16 2024-09-20T22:52:34.389 INFO:tasks.rados.rados.0.smithi084.stdout:2644: finishing copy_from to smithi08477203-16 2024-09-20T22:52:34.389 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 16 v 1807 (ObjNum 876 snap 280 seq_num 876) dirty exists 2024-09-20T22:52:34.439 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:52:34.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:34.461 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:34.462 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:52:34.562 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:52:34.583 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:34.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:34.584 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:52:34.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:34 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/372286671' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:52:34.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:34 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:52:34.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:34 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/372286671' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:52:34.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:34 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:52:34.685 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:52:34.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:34.708 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:34.708 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:52:34.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:34 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/372286671' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:52:34.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:34 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:52:34.809 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:52:34.834 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:34.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:34.835 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:52:34.856 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-20T22:52:34.935 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:52:34.959 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:34.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:34.960 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:52:35.060 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:52:35.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:35.082 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:35.082 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:52:35.183 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:52:35.206 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:35.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:35.207 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:52:35.308 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:52:35.330 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:35.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:35.331 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:52:35.368 INFO:tasks.rados.rados.0.smithi084.stdout:2644: done (1 left) 2024-09-20T22:52:35.368 INFO:tasks.rados.rados.0.smithi084.stdout:2646: done (0 left) 2024-09-20T22:52:35.368 INFO:tasks.rados.rados.0.smithi084.stdout:2647: write oid 41 current snap is 301 2024-09-20T22:52:35.368 INFO:tasks.rados.rados.0.smithi084.stdout:2647: seq_num 917 ranges {612019=565693,1810929=602646,3082649=329370} 2024-09-20T22:52:35.375 INFO:tasks.rados.rados.0.smithi084.stdout:2647: writing smithi08477203-41 from 612019 to 1177712 tid 1 2024-09-20T22:52:35.382 INFO:tasks.rados.rados.0.smithi084.stdout:2647: writing smithi08477203-41 from 1810929 to 2413575 tid 2 2024-09-20T22:52:35.388 INFO:tasks.rados.rados.0.smithi084.stdout:2647: writing smithi08477203-41 from 3082649 to 3412019 tid 3 2024-09-20T22:52:35.388 INFO:tasks.rados.rados.0.smithi084.stdout:2648: snap_create 2024-09-20T22:52:35.388 INFO:tasks.rados.rados.0.smithi084.stdout:2647: finishing write tid 1 to smithi08477203-41 2024-09-20T22:52:35.388 INFO:tasks.rados.rados.0.smithi084.stdout:2647: finishing write tid 2 to smithi08477203-41 2024-09-20T22:52:35.390 INFO:tasks.rados.rados.0.smithi084.stdout:2647: finishing write tid 3 to smithi08477203-41 2024-09-20T22:52:35.390 INFO:tasks.rados.rados.0.smithi084.stdout:2647: finishing write tid 4 to smithi08477203-41 2024-09-20T22:52:35.390 INFO:tasks.rados.rados.0.smithi084.stdout:2647: finishing write tid 5 to smithi08477203-41 2024-09-20T22:52:35.391 INFO:tasks.rados.rados.0.smithi084.stdout:2647: oid 41 updating version 0 to 1415 2024-09-20T22:52:35.391 INFO:tasks.rados.rados.0.smithi084.stdout:2647: oid 41 version 1415 is already newer than 1413 2024-09-20T22:52:35.391 INFO:tasks.rados.rados.0.smithi084.stdout:2647: oid 41 version 1415 is already newer than 1414 2024-09-20T22:52:35.391 INFO:tasks.rados.rados.0.smithi084.stdout:2647: oid 41 updating version 1415 to 1416 2024-09-20T22:52:35.391 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 41 v 1416 (ObjNum 917 snap 301 seq_num 917) dirty exists 2024-09-20T22:52:35.391 INFO:tasks.rados.rados.0.smithi084.stdout:2647: left oid 41 (ObjNum 917 snap 301 seq_num 917) 2024-09-20T22:52:35.432 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:52:35.454 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:35.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:35.455 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:52:35.461 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:35 smithi084 ceph-mon[29774]: pgmap v912: 158 pgs: 1 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 154 active+clean; 590 MiB data, 1.6 GiB used, 624 GiB / 626 GiB avail; 12 MiB/s rd, 1.0 MiB/s wr, 14 op/s 2024-09-20T22:52:35.462 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:35 smithi084 ceph-mon[29774]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:52:35.462 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:35 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T22:52:35.462 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:35 smithi084 ceph-mon[29774]: osdmap e674: 8 total, 8 up, 7 in 2024-09-20T22:52:35.462 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:35 smithi084 ceph-mon[29774]: 1.7 scrub starts 2024-09-20T22:52:35.462 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:35 smithi084 ceph-mon[29774]: 1.7 scrub ok 2024-09-20T22:52:35.462 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:35 smithi084 ceph-mon[40024]: pgmap v912: 158 pgs: 1 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 154 active+clean; 590 MiB data, 1.6 GiB used, 624 GiB / 626 GiB avail; 12 MiB/s rd, 1.0 MiB/s wr, 14 op/s 2024-09-20T22:52:35.462 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:35 smithi084 ceph-mon[40024]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:52:35.462 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:35 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T22:52:35.462 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:35 smithi084 ceph-mon[40024]: osdmap e674: 8 total, 8 up, 7 in 2024-09-20T22:52:35.462 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:35 smithi084 ceph-mon[40024]: 1.7 scrub starts 2024-09-20T22:52:35.462 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:35 smithi084 ceph-mon[40024]: 1.7 scrub ok 2024-09-20T22:52:35.464 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-20T22:52:35.464 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2024-09-20T22:52:35.464 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2024-09-20T22:52:35.464 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2024-09-20T22:52:35.464 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-20T22:52:35.555 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:52:35.577 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:35.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:35.578 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:52:35.678 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:52:35.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:35.701 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:35.701 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:52:35.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:35 smithi154 ceph-mon[33264]: pgmap v912: 158 pgs: 1 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 154 active+clean; 590 MiB data, 1.6 GiB used, 624 GiB / 626 GiB avail; 12 MiB/s rd, 1.0 MiB/s wr, 14 op/s 2024-09-20T22:52:35.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:35 smithi154 ceph-mon[33264]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:52:35.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:35 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T22:52:35.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:35 smithi154 ceph-mon[33264]: osdmap e674: 8 total, 8 up, 7 in 2024-09-20T22:52:35.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:35 smithi154 ceph-mon[33264]: 1.7 scrub starts 2024-09-20T22:52:35.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:35 smithi154 ceph-mon[33264]: 1.7 scrub ok 2024-09-20T22:52:35.802 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:52:35.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:35.823 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:35.823 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:52:35.924 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:52:35.949 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:35.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:35.950 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:52:36.050 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:52:36.072 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:36.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:36.073 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:52:36.101 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.1 dump_historic_ops 2024-09-20T22:52:36.128 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:52:36.160 INFO:tasks.thrashosds.thrasher:in_osds: [2, 3, 5, 6, 7, 0, 4] out_osds: [1] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-20T22:52:36.160 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd reweight 2 1 2024-09-20T22:52:36.174 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:52:36.234 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:36.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:36.235 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:52:36.336 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:52:36.358 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:36.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:36.359 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:52:36.373 INFO:tasks.rados.rados.0.smithi084.stdout:2647: done (1 left) 2024-09-20T22:52:36.373 INFO:tasks.rados.rados.0.smithi084.stdout:2648: done (0 left) 2024-09-20T22:52:36.374 INFO:tasks.rados.rados.0.smithi084.stdout:2649: write (excl) oid 2 current snap is 302 2024-09-20T22:52:36.374 INFO:tasks.rados.rados.0.smithi084.stdout:2649: seq_num 918 ranges {628826=640942,1869411=462657,2826425=534649} 2024-09-20T22:52:36.380 INFO:tasks.rados.rados.0.smithi084.stdout:2649: writing smithi08477203-2 from 628826 to 1269768 tid 1 2024-09-20T22:52:36.386 INFO:tasks.rados.rados.0.smithi084.stdout:2649: writing smithi08477203-2 from 1869411 to 2332068 tid 2 2024-09-20T22:52:36.392 INFO:tasks.rados.rados.0.smithi084.stdout:2649: writing smithi08477203-2 from 2826425 to 3361074 tid 3 2024-09-20T22:52:36.392 INFO:tasks.rados.rados.0.smithi084.stdout:2650: write oid 10 current snap is 302 2024-09-20T22:52:36.392 INFO:tasks.rados.rados.0.smithi084.stdout:2650: seq_num 919 ranges {645633=716191,1927893=606315,3137495=308138} 2024-09-20T22:52:36.399 INFO:tasks.rados.rados.0.smithi084.stdout:2650: writing smithi08477203-10 from 645633 to 1361824 tid 1 2024-09-20T22:52:36.406 INFO:tasks.rados.rados.0.smithi084.stdout:2650: writing smithi08477203-10 from 1927893 to 2534208 tid 2 2024-09-20T22:52:36.412 INFO:tasks.rados.rados.0.smithi084.stdout:2650: writing smithi08477203-10 from 3137495 to 3445633 tid 3 2024-09-20T22:52:36.412 INFO:tasks.rados.rados.0.smithi084.stdout:2651: snap_create 2024-09-20T22:52:36.412 INFO:tasks.rados.rados.0.smithi084.stdout:2649: finishing write tid 1 to smithi08477203-2 2024-09-20T22:52:36.412 INFO:tasks.rados.rados.0.smithi084.stdout:2649: finishing write tid 2 to smithi08477203-2 2024-09-20T22:52:36.412 INFO:tasks.rados.rados.0.smithi084.stdout:2649: finishing write tid 3 to smithi08477203-2 2024-09-20T22:52:36.412 INFO:tasks.rados.rados.0.smithi084.stdout:2649: finishing write tid 4 to smithi08477203-2 2024-09-20T22:52:36.412 INFO:tasks.rados.rados.0.smithi084.stdout:2649: finishing write tid 5 to smithi08477203-2 2024-09-20T22:52:36.412 INFO:tasks.rados.rados.0.smithi084.stdout:2649: oid 2 updating version 0 to 1804 2024-09-20T22:52:36.413 INFO:tasks.rados.rados.0.smithi084.stdout:2649: oid 2 updating version 1804 to 1805 2024-09-20T22:52:36.413 INFO:tasks.rados.rados.0.smithi084.stdout:2649: oid 2 version 1805 is already newer than 1803 2024-09-20T22:52:36.413 INFO:tasks.rados.rados.0.smithi084.stdout:2649: oid 2 version 1805 is already newer than 1802 2024-09-20T22:52:36.413 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 2 v 1805 (ObjNum 918 snap 302 seq_num 918) dirty exists 2024-09-20T22:52:36.413 INFO:tasks.rados.rados.0.smithi084.stdout:2649: left oid 2 (ObjNum 918 snap 302 seq_num 918) 2024-09-20T22:52:36.413 INFO:tasks.rados.rados.0.smithi084.stdout:2650: finishing write tid 1 to smithi08477203-10 2024-09-20T22:52:36.413 INFO:tasks.rados.rados.0.smithi084.stdout:2650: finishing write tid 2 to smithi08477203-10 2024-09-20T22:52:36.415 INFO:tasks.rados.rados.0.smithi084.stdout:2650: finishing write tid 3 to smithi08477203-10 2024-09-20T22:52:36.415 INFO:tasks.rados.rados.0.smithi084.stdout:2650: finishing write tid 4 to smithi08477203-10 2024-09-20T22:52:36.415 INFO:tasks.rados.rados.0.smithi084.stdout:2650: finishing write tid 5 to smithi08477203-10 2024-09-20T22:52:36.415 INFO:tasks.rados.rados.0.smithi084.stdout:2650: oid 10 updating version 0 to 1818 2024-09-20T22:52:36.415 INFO:tasks.rados.rados.0.smithi084.stdout:2650: oid 10 version 1818 is already newer than 1816 2024-09-20T22:52:36.415 INFO:tasks.rados.rados.0.smithi084.stdout:2650: oid 10 updating version 1818 to 1819 2024-09-20T22:52:36.416 INFO:tasks.rados.rados.0.smithi084.stdout:2650: oid 10 version 1819 is already newer than 1817 2024-09-20T22:52:36.416 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 10 v 1819 (ObjNum 919 snap 302 seq_num 919) dirty exists 2024-09-20T22:52:36.416 INFO:tasks.rados.rados.0.smithi084.stdout:2650: left oid 10 (ObjNum 919 snap 302 seq_num 919) 2024-09-20T22:52:36.460 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:52:36.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:36.482 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:36.483 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:52:36.583 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:52:36.604 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:36.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:36.605 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:52:36.685 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:36 smithi084 ceph-mon[40024]: 7.9 scrub starts 2024-09-20T22:52:36.685 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:36 smithi084 ceph-mon[40024]: 7.9 scrub ok 2024-09-20T22:52:36.685 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:36 smithi084 ceph-mon[40024]: 7.8 scrub starts 2024-09-20T22:52:36.685 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:36 smithi084 ceph-mon[40024]: 7.8 scrub ok 2024-09-20T22:52:36.685 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:36 smithi084 ceph-mon[40024]: osdmap e675: 8 total, 8 up, 7 in 2024-09-20T22:52:36.686 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:36 smithi084 ceph-mon[29774]: 7.9 scrub starts 2024-09-20T22:52:36.686 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:36 smithi084 ceph-mon[29774]: 7.9 scrub ok 2024-09-20T22:52:36.686 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:36 smithi084 ceph-mon[29774]: 7.8 scrub starts 2024-09-20T22:52:36.686 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:36 smithi084 ceph-mon[29774]: 7.8 scrub ok 2024-09-20T22:52:36.686 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:36 smithi084 ceph-mon[29774]: osdmap e675: 8 total, 8 up, 7 in 2024-09-20T22:52:36.686 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:52:36.686 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:52:36.705 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:52:36.706 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:52:36.706 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:52:36.723 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:52:36.723 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:52:36.732 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:52:36.732 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:36.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:36.733 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:52:36.734 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:52:36.742 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-20T22:52:36.743 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T22:52:36.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:36 smithi154 ceph-mon[33264]: 7.9 scrub starts 2024-09-20T22:52:36.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:36 smithi154 ceph-mon[33264]: 7.9 scrub ok 2024-09-20T22:52:36.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:36 smithi154 ceph-mon[33264]: 7.8 scrub starts 2024-09-20T22:52:36.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:36 smithi154 ceph-mon[33264]: 7.8 scrub ok 2024-09-20T22:52:36.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:36 smithi154 ceph-mon[33264]: osdmap e675: 8 total, 8 up, 7 in 2024-09-20T22:52:36.763 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:52:36.763 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:52:36.781 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:52:36.781 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:52:36.797 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:52:36.798 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:52:36.814 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:52:36.815 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:52:36.834 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:52:36.855 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:36.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:36.856 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:52:36.956 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:52:36.978 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:36.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:36.979 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:52:37.079 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:52:37.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:37.101 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:37.101 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:52:37.201 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-20T22:52:37.202 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:52:37.222 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:37.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:37.223 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:52:37.324 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:52:37.349 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:37.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:37.350 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:52:37.382 INFO:tasks.rados.rados.0.smithi084.stdout:2649: done (2 left) 2024-09-20T22:52:37.382 INFO:tasks.rados.rados.0.smithi084.stdout:2650: done (1 left) 2024-09-20T22:52:37.382 INFO:tasks.rados.rados.0.smithi084.stdout:2651: done (0 left) 2024-09-20T22:52:37.382 INFO:tasks.rados.rados.0.smithi084.stdout:2652: snap_remove snap 284 2024-09-20T22:52:37.451 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:52:37.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:37.472 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:37.472 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:52:37.518 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-20T22:52:37.518 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:52:37.527 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:37 smithi084 ceph-mon[40024]: 7.5 scrub starts 2024-09-20T22:52:37.527 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:37 smithi084 ceph-mon[40024]: 7.5 scrub ok 2024-09-20T22:52:37.527 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:37 smithi084 ceph-mon[40024]: pgmap v915: 158 pgs: 1 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 154 active+clean; 590 MiB data, 1.6 GiB used, 624 GiB / 626 GiB avail; 12 MiB/s rd, 1.0 MiB/s wr, 14 op/s 2024-09-20T22:52:37.527 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:37 smithi084 ceph-mon[40024]: 5.15 scrub starts 2024-09-20T22:52:37.527 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:37 smithi084 ceph-mon[40024]: 5.15 scrub ok 2024-09-20T22:52:37.527 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:37 smithi084 ceph-mon[40024]: osdmap e676: 8 total, 8 up, 7 in 2024-09-20T22:52:37.528 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:37 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/3395932793' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:52:37.528 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:37 smithi084 ceph-mon[40024]: 3.3 scrub starts 2024-09-20T22:52:37.528 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:37 smithi084 ceph-mon[40024]: 3.3 scrub ok 2024-09-20T22:52:37.528 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:37 smithi084 ceph-mon[40024]: 7.c scrub starts 2024-09-20T22:52:37.528 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:37 smithi084 ceph-mon[40024]: 7.c scrub ok 2024-09-20T22:52:37.528 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:37 smithi084 ceph-mon[40024]: 2.1 scrub starts 2024-09-20T22:52:37.528 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:37 smithi084 ceph-mon[40024]: 2.1 scrub ok 2024-09-20T22:52:37.528 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:37 smithi084 ceph-mon[29774]: 7.5 scrub starts 2024-09-20T22:52:37.528 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:37 smithi084 ceph-mon[29774]: 7.5 scrub ok 2024-09-20T22:52:37.528 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:37 smithi084 ceph-mon[29774]: pgmap v915: 158 pgs: 1 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 154 active+clean; 590 MiB data, 1.6 GiB used, 624 GiB / 626 GiB avail; 12 MiB/s rd, 1.0 MiB/s wr, 14 op/s 2024-09-20T22:52:37.528 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:37 smithi084 ceph-mon[29774]: 5.15 scrub starts 2024-09-20T22:52:37.528 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:37 smithi084 ceph-mon[29774]: 5.15 scrub ok 2024-09-20T22:52:37.528 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:37 smithi084 ceph-mon[29774]: osdmap e676: 8 total, 8 up, 7 in 2024-09-20T22:52:37.529 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:37 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/3395932793' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:52:37.529 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:37 smithi084 ceph-mon[29774]: 3.3 scrub starts 2024-09-20T22:52:37.529 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:37 smithi084 ceph-mon[29774]: 3.3 scrub ok 2024-09-20T22:52:37.529 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:37 smithi084 ceph-mon[29774]: 7.c scrub starts 2024-09-20T22:52:37.529 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:37 smithi084 ceph-mon[29774]: 7.c scrub ok 2024-09-20T22:52:37.529 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:37 smithi084 ceph-mon[29774]: 2.1 scrub starts 2024-09-20T22:52:37.529 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:37 smithi084 ceph-mon[29774]: 2.1 scrub ok 2024-09-20T22:52:37.574 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:52:37.599 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:37.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:37.600 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:52:37.701 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:52:37.722 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:37.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:37.723 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:52:37.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:37 smithi154 ceph-mon[33264]: 7.5 scrub starts 2024-09-20T22:52:37.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:37 smithi154 ceph-mon[33264]: 7.5 scrub ok 2024-09-20T22:52:37.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:37 smithi154 ceph-mon[33264]: pgmap v915: 158 pgs: 1 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 154 active+clean; 590 MiB data, 1.6 GiB used, 624 GiB / 626 GiB avail; 12 MiB/s rd, 1.0 MiB/s wr, 14 op/s 2024-09-20T22:52:37.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:37 smithi154 ceph-mon[33264]: 5.15 scrub starts 2024-09-20T22:52:37.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:37 smithi154 ceph-mon[33264]: 5.15 scrub ok 2024-09-20T22:52:37.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:37 smithi154 ceph-mon[33264]: osdmap e676: 8 total, 8 up, 7 in 2024-09-20T22:52:37.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:37 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/3395932793' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:52:37.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:37 smithi154 ceph-mon[33264]: 3.3 scrub starts 2024-09-20T22:52:37.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:37 smithi154 ceph-mon[33264]: 3.3 scrub ok 2024-09-20T22:52:37.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:37 smithi154 ceph-mon[33264]: 7.c scrub starts 2024-09-20T22:52:37.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:37 smithi154 ceph-mon[33264]: 7.c scrub ok 2024-09-20T22:52:37.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:37 smithi154 ceph-mon[33264]: 2.1 scrub starts 2024-09-20T22:52:37.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:37 smithi154 ceph-mon[33264]: 2.1 scrub ok 2024-09-20T22:52:37.824 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:52:37.844 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:37.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:37.845 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:52:37.946 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:52:37.972 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:37.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:37.973 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:52:38.073 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:52:38.098 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:38.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:38.099 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:52:38.200 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:52:38.210 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-20T22:52:38.221 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:38.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:38.222 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:52:38.290 INFO:tasks.rados.rados.0.smithi084.stdout:2652: done (0 left) 2024-09-20T22:52:38.290 INFO:tasks.rados.rados.0.smithi084.stdout:2653: snap_create 2024-09-20T22:52:38.323 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:52:38.345 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:38.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:38.346 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:52:38.446 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:52:38.470 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:38.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:38.470 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:52:38.571 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:52:38.592 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:38.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:38.593 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:52:38.600 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:52:38 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: ::ffff:172.21.15.154 - - [20/Sep/2024:22:52:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-20T22:52:38.600 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:38 smithi084 ceph-mon[29774]: osdmap e677: 8 total, 8 up, 7 in 2024-09-20T22:52:38.600 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:38 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:52:38.600 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:38 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:52:38.600 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:38 smithi084 ceph-mon[29774]: osdmap e678: 8 total, 8 up, 7 in 2024-09-20T22:52:38.601 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:38 smithi084 ceph-mon[40024]: osdmap e677: 8 total, 8 up, 7 in 2024-09-20T22:52:38.601 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:38 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:52:38.601 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:38 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:52:38.601 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:38 smithi084 ceph-mon[40024]: osdmap e678: 8 total, 8 up, 7 in 2024-09-20T22:52:38.693 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:52:38.715 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:38.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:38.716 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:52:38.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:38 smithi154 ceph-mon[33264]: osdmap e677: 8 total, 8 up, 7 in 2024-09-20T22:52:38.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:38 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:52:38.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:38 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:52:38.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:38 smithi154 ceph-mon[33264]: osdmap e678: 8 total, 8 up, 7 in 2024-09-20T22:52:38.817 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:52:38.837 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:38.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:38.838 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:52:38.939 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:52:38.960 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:38.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:38.961 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:52:39.062 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:52:39.083 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:39.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:39.084 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:52:39.184 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:52:39.211 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:39.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:39.211 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:52:39.294 INFO:tasks.rados.rados.0.smithi084.stdout:2653: done (0 left) 2024-09-20T22:52:39.294 INFO:tasks.rados.rados.0.smithi084.stdout:2654: write oid 36 current snap is 304 2024-09-20T22:52:39.294 INFO:tasks.rados.rados.0.smithi084.stdout:2654: seq_num 920 ranges {662440=507793,1702728=749973,2881271=557193} 2024-09-20T22:52:39.300 INFO:tasks.rados.rados.0.smithi084.stdout:2654: writing smithi08477203-36 from 662440 to 1170233 tid 1 2024-09-20T22:52:39.308 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:52:39.309 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:52:39.310 INFO:tasks.rados.rados.0.smithi084.stdout:2654: writing smithi08477203-36 from 1702728 to 2452701 tid 2 2024-09-20T22:52:39.312 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:52:39.318 INFO:tasks.rados.rados.0.smithi084.stdout:2654: writing smithi08477203-36 from 2881271 to 3438464 tid 3 2024-09-20T22:52:39.318 INFO:tasks.rados.rados.0.smithi084.stdout:2655: snap_create 2024-09-20T22:52:39.318 INFO:tasks.rados.rados.0.smithi084.stdout:2654: finishing write tid 1 to smithi08477203-36 2024-09-20T22:52:39.318 INFO:tasks.rados.rados.0.smithi084.stdout:2654: finishing write tid 2 to smithi08477203-36 2024-09-20T22:52:39.322 INFO:tasks.rados.rados.0.smithi084.stdout:2654: finishing write tid 3 to smithi08477203-36 2024-09-20T22:52:39.322 INFO:tasks.rados.rados.0.smithi084.stdout:2654: finishing write tid 4 to smithi08477203-36 2024-09-20T22:52:39.322 INFO:tasks.rados.rados.0.smithi084.stdout:2654: finishing write tid 5 to smithi08477203-36 2024-09-20T22:52:39.322 INFO:tasks.rados.rados.0.smithi084.stdout:2654: oid 36 updating version 0 to 1227 2024-09-20T22:52:39.322 INFO:tasks.rados.rados.0.smithi084.stdout:2654: oid 36 version 1227 is already newer than 1225 2024-09-20T22:52:39.322 INFO:tasks.rados.rados.0.smithi084.stdout:2654: oid 36 updating version 1227 to 1228 2024-09-20T22:52:39.323 INFO:tasks.rados.rados.0.smithi084.stdout:2654: oid 36 version 1228 is already newer than 1226 2024-09-20T22:52:39.323 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 36 v 1228 (ObjNum 920 snap 304 seq_num 920) dirty exists 2024-09-20T22:52:39.323 INFO:tasks.rados.rados.0.smithi084.stdout:2654: left oid 36 (ObjNum 920 snap 304 seq_num 920) 2024-09-20T22:52:39.337 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:39.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:39.338 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:52:39.439 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:52:39.460 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:39.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:39.461 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:52:39.561 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:52:39.582 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:39.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:39.583 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:52:39.642 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:39 smithi084 ceph-mon[29774]: pgmap v918: 158 pgs: 1 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 154 active+clean; 590 MiB data, 1.6 GiB used, 624 GiB / 626 GiB avail 2024-09-20T22:52:39.642 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:39 smithi084 ceph-mon[29774]: 1.6 scrub starts 2024-09-20T22:52:39.642 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:39 smithi084 ceph-mon[29774]: 1.6 scrub ok 2024-09-20T22:52:39.642 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:39 smithi084 ceph-mon[29774]: osdmap e679: 8 total, 8 up, 7 in 2024-09-20T22:52:39.642 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:39 smithi084 ceph-mon[40024]: pgmap v918: 158 pgs: 1 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 154 active+clean; 590 MiB data, 1.6 GiB used, 624 GiB / 626 GiB avail 2024-09-20T22:52:39.642 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:39 smithi084 ceph-mon[40024]: 1.6 scrub starts 2024-09-20T22:52:39.642 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:39 smithi084 ceph-mon[40024]: 1.6 scrub ok 2024-09-20T22:52:39.643 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:39 smithi084 ceph-mon[40024]: osdmap e679: 8 total, 8 up, 7 in 2024-09-20T22:52:39.684 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:52:39.710 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:39.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:39.711 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:52:39.754 INFO:journalctl@ceph.iscsi.iscsi.a.smithi154.stdout:Sep 20 22:52:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-iscsi-iscsi-a[59730]: debug there is no tcmu-runner data available 2024-09-20T22:52:39.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:39 smithi154 ceph-mon[33264]: pgmap v918: 158 pgs: 1 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 154 active+clean; 590 MiB data, 1.6 GiB used, 624 GiB / 626 GiB avail 2024-09-20T22:52:39.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:39 smithi154 ceph-mon[33264]: 1.6 scrub starts 2024-09-20T22:52:39.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:39 smithi154 ceph-mon[33264]: 1.6 scrub ok 2024-09-20T22:52:39.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:39 smithi154 ceph-mon[33264]: osdmap e679: 8 total, 8 up, 7 in 2024-09-20T22:52:39.812 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:52:39.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:39.834 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:39.834 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:52:39.935 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:52:39.956 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:39.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:39.956 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:52:40.057 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:52:40.084 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:40.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:40.085 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:52:40.133 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-20T22:52:40.133 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:52:40.186 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:52:40.208 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:40.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:40.209 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:52:40.290 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.2 to 1 (10000) 2024-09-20T22:52:40.295 INFO:tasks.rados.rados.0.smithi084.stdout:2654: done (1 left) 2024-09-20T22:52:40.295 INFO:tasks.rados.rados.0.smithi084.stdout:2655: done (0 left) 2024-09-20T22:52:40.295 INFO:tasks.rados.rados.0.smithi084.stdout:2656: delete oid 29 current snap is 305 2024-09-20T22:52:40.302 INFO:tasks.rados.rados.0.smithi084.stdout:2656: done (0 left) 2024-09-20T22:52:40.302 INFO:tasks.rados.rados.0.smithi084.stdout:2657: snap_remove snap 275 2024-09-20T22:52:40.309 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:52:40.332 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:40.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:40.334 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:52:40.434 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:52:40.459 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:40.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:40.460 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:52:40.561 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:52:40.583 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:40.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:40.584 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:52:40.642 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:40 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/273656920' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-09-20T22:52:40.642 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:40 smithi084 ceph-mon[40024]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:52:40.642 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:40 smithi084 ceph-mon[40024]: 4.3 scrub starts 2024-09-20T22:52:40.642 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:40 smithi084 ceph-mon[40024]: 4.3 scrub ok 2024-09-20T22:52:40.642 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:40 smithi084 ceph-mon[40024]: 6.1e scrub starts 2024-09-20T22:52:40.642 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:40 smithi084 ceph-mon[40024]: 6.1e scrub ok 2024-09-20T22:52:40.643 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:40 smithi084 ceph-mon[40024]: 5.2 scrub starts 2024-09-20T22:52:40.643 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:40 smithi084 ceph-mon[40024]: 5.2 scrub ok 2024-09-20T22:52:40.643 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:40 smithi084 ceph-mon[40024]: 5.b scrub starts 2024-09-20T22:52:40.643 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:40 smithi084 ceph-mon[40024]: 5.b scrub ok 2024-09-20T22:52:40.643 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:40 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/273656920' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-20T22:52:40.643 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:40 smithi084 ceph-mon[40024]: osdmap e680: 8 total, 8 up, 7 in 2024-09-20T22:52:40.643 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:40 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/273656920' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-09-20T22:52:40.643 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:40 smithi084 ceph-mon[29774]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:52:40.643 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:40 smithi084 ceph-mon[29774]: 4.3 scrub starts 2024-09-20T22:52:40.643 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:40 smithi084 ceph-mon[29774]: 4.3 scrub ok 2024-09-20T22:52:40.643 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:40 smithi084 ceph-mon[29774]: 6.1e scrub starts 2024-09-20T22:52:40.643 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:40 smithi084 ceph-mon[29774]: 6.1e scrub ok 2024-09-20T22:52:40.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:40 smithi084 ceph-mon[29774]: 5.2 scrub starts 2024-09-20T22:52:40.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:40 smithi084 ceph-mon[29774]: 5.2 scrub ok 2024-09-20T22:52:40.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:40 smithi084 ceph-mon[29774]: 5.b scrub starts 2024-09-20T22:52:40.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:40 smithi084 ceph-mon[29774]: 5.b scrub ok 2024-09-20T22:52:40.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:40 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/273656920' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-20T22:52:40.644 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:40 smithi084 ceph-mon[29774]: osdmap e680: 8 total, 8 up, 7 in 2024-09-20T22:52:40.656 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:40 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/273656920' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-09-20T22:52:40.656 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:40 smithi154 ceph-mon[33264]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:52:40.656 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:40 smithi154 ceph-mon[33264]: 4.3 scrub starts 2024-09-20T22:52:40.656 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:40 smithi154 ceph-mon[33264]: 4.3 scrub ok 2024-09-20T22:52:40.656 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:40 smithi154 ceph-mon[33264]: 6.1e scrub starts 2024-09-20T22:52:40.656 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:40 smithi154 ceph-mon[33264]: 6.1e scrub ok 2024-09-20T22:52:40.656 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:40 smithi154 ceph-mon[33264]: 5.2 scrub starts 2024-09-20T22:52:40.656 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:40 smithi154 ceph-mon[33264]: 5.2 scrub ok 2024-09-20T22:52:40.656 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:40 smithi154 ceph-mon[33264]: 5.b scrub starts 2024-09-20T22:52:40.656 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:40 smithi154 ceph-mon[33264]: 5.b scrub ok 2024-09-20T22:52:40.657 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:40 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/273656920' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-20T22:52:40.657 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:40 smithi154 ceph-mon[33264]: osdmap e680: 8 total, 8 up, 7 in 2024-09-20T22:52:40.685 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:52:40.711 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:40.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:40.712 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:52:40.813 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:52:40.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:40.834 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:40.835 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:52:40.935 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:52:40.958 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:40.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:40.959 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:52:41.053 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-20T22:52:41.059 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:52:41.080 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:41.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:41.081 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:52:41.091 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd reweight 3 1 2024-09-20T22:52:41.182 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:52:41.202 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:41.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:41.204 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:52:41.254 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:52:41 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=cleanup t=2024-09-20T22:52:41.153632091Z level=info msg="Completed cleanup jobs" duration=4.278602ms 2024-09-20T22:52:41.304 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:52:41.308 INFO:tasks.rados.rados.0.smithi084.stdout:2657: done (0 left) 2024-09-20T22:52:41.308 INFO:tasks.rados.rados.0.smithi084.stdout:2658: rollback oid 49 current snap is 305 2024-09-20T22:52:41.308 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 49 to 300 2024-09-20T22:52:41.308 INFO:tasks.rados.rados.0.smithi084.stdout:2659: rollback oid 48 current snap is 305 2024-09-20T22:52:41.309 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 48 to 248 2024-09-20T22:52:41.309 INFO:tasks.rados.rados.0.smithi084.stdout:2660: write oid 5 current snap is 305 2024-09-20T22:52:41.309 INFO:tasks.rados.rados.0.smithi084.stdout:2660: seq_num 922 ranges {696054=658291,2103339=753642} 2024-09-20T22:52:41.311 INFO:tasks.rados.rados.0.smithi084.stdout:2660: writing smithi08477203-5 from 696054 to 1354345 tid 1 2024-09-20T22:52:41.321 INFO:tasks.rados.rados.0.smithi084.stdout:2660: writing smithi08477203-5 from 2103339 to 2856981 tid 2 2024-09-20T22:52:41.321 INFO:tasks.rados.rados.0.smithi084.stdout:2659: finishing rollback tid 1 to smithi08477203-48 2024-09-20T22:52:41.321 INFO:tasks.rados.rados.0.smithi084.stdout:2659: finishing rollback tid 2 to smithi08477203-48 2024-09-20T22:52:41.321 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 48 v 1230 (ObjNum 759 snap 243 seq_num 759) dirty exists 2024-09-20T22:52:41.321 INFO:tasks.rados.rados.0.smithi084.stdout:2658: finishing rollback tid 0 to smithi08477203-49 2024-09-20T22:52:41.321 INFO:tasks.rados.rados.0.smithi084.stdout:2658: finishing rollback tid 1 to smithi08477203-49 2024-09-20T22:52:41.321 INFO:tasks.rados.rados.0.smithi084.stdout:2658: finishing rollback tid 2 to smithi08477203-49 2024-09-20T22:52:41.321 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 49 v 1808 (ObjNum 815 snap 262 seq_num 815) dirty exists 2024-09-20T22:52:41.321 INFO:tasks.rados.rados.0.smithi084.stdout:2660: finishing write tid 1 to smithi08477203-5 2024-09-20T22:52:41.321 INFO:tasks.rados.rados.0.smithi084.stdout:2658: done (2 left) 2024-09-20T22:52:41.322 INFO:tasks.rados.rados.0.smithi084.stdout:2659: done (1 left) 2024-09-20T22:52:41.322 INFO:tasks.rados.rados.0.smithi084.stdout:2661: rollback oid 47 current snap is 305 2024-09-20T22:52:41.322 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 47 to 266 2024-09-20T22:52:41.322 INFO:tasks.rados.rados.0.smithi084.stdout:2662: write (excl) oid 8 current snap is 305 2024-09-20T22:52:41.322 INFO:tasks.rados.rados.0.smithi084.stdout:2662: seq_num 923 ranges {712861=733540,2161821=613653,3247187=265674} 2024-09-20T22:52:41.329 INFO:tasks.rados.rados.0.smithi084.stdout:2662: writing smithi08477203-8 from 712861 to 1446401 tid 1 2024-09-20T22:52:41.335 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:41.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:41.336 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:52:41.336 INFO:tasks.rados.rados.0.smithi084.stdout:2662: writing smithi08477203-8 from 2161821 to 2775474 tid 2 2024-09-20T22:52:41.341 INFO:tasks.rados.rados.0.smithi084.stdout:2662: writing smithi08477203-8 from 3247187 to 3512861 tid 3 2024-09-20T22:52:41.341 INFO:tasks.rados.rados.0.smithi084.stdout:2663: snap_remove snap 278 2024-09-20T22:52:41.437 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:52:41.458 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:41.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:41.461 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:52:41.562 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:52:41.583 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:41.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:41.584 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:52:41.646 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:41 smithi084 ceph-mon[29774]: 1.9 scrub starts 2024-09-20T22:52:41.646 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:41 smithi084 ceph-mon[29774]: 1.9 scrub ok 2024-09-20T22:52:41.646 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:41 smithi084 ceph-mon[29774]: pgmap v921: 158 pgs: 1 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 154 active+clean; 599 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail; 2.0 KiB/s rd, 1.8 MiB/s wr, 6 op/s 2024-09-20T22:52:41.646 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:41 smithi084 ceph-mon[29774]: 4.12 scrub starts 2024-09-20T22:52:41.647 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:41 smithi084 ceph-mon[29774]: 4.12 scrub ok 2024-09-20T22:52:41.647 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:41 smithi084 ceph-mon[29774]: osdmap e681: 8 total, 8 up, 7 in 2024-09-20T22:52:41.647 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:41 smithi084 ceph-mon[40024]: 1.9 scrub starts 2024-09-20T22:52:41.647 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:41 smithi084 ceph-mon[40024]: 1.9 scrub ok 2024-09-20T22:52:41.647 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:41 smithi084 ceph-mon[40024]: pgmap v921: 158 pgs: 1 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 154 active+clean; 599 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail; 2.0 KiB/s rd, 1.8 MiB/s wr, 6 op/s 2024-09-20T22:52:41.647 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:41 smithi084 ceph-mon[40024]: 4.12 scrub starts 2024-09-20T22:52:41.647 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:41 smithi084 ceph-mon[40024]: 4.12 scrub ok 2024-09-20T22:52:41.647 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:41 smithi084 ceph-mon[40024]: osdmap e681: 8 total, 8 up, 7 in 2024-09-20T22:52:41.648 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.2 dump_blocked_ops 2024-09-20T22:52:41.685 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:52:41.705 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:41.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:41.706 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:52:41.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:41 smithi154 ceph-mon[33264]: 1.9 scrub starts 2024-09-20T22:52:41.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:41 smithi154 ceph-mon[33264]: 1.9 scrub ok 2024-09-20T22:52:41.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:41 smithi154 ceph-mon[33264]: pgmap v921: 158 pgs: 1 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 154 active+clean; 599 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail; 2.0 KiB/s rd, 1.8 MiB/s wr, 6 op/s 2024-09-20T22:52:41.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:41 smithi154 ceph-mon[33264]: 4.12 scrub starts 2024-09-20T22:52:41.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:41 smithi154 ceph-mon[33264]: 4.12 scrub ok 2024-09-20T22:52:41.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:41 smithi154 ceph-mon[33264]: osdmap e681: 8 total, 8 up, 7 in 2024-09-20T22:52:41.807 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:52:41.828 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:41.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:41.829 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:52:41.929 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:52:41.950 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:41.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:41.951 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:52:41.960 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:52:41.964 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:52:41.967 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:52:42.052 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:52:42.074 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:42.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:42.075 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:52:42.176 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:52:42.198 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:42.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:42.199 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:52:42.292 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is unset 2024-09-20T22:52:42.297 INFO:tasks.rados.rados.0.smithi084.stdout:2663: done (3 left) 2024-09-20T22:52:42.297 INFO:tasks.rados.rados.0.smithi084.stdout:2664: write oid 29 current snap is 305 2024-09-20T22:52:42.297 INFO:tasks.rados.rados.0.smithi084.stdout:2664: seq_num 924 ranges {729668=408789,1820303=757311,3274610=255058} 2024-09-20T22:52:42.299 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:52:42.303 INFO:tasks.rados.rados.0.smithi084.stdout:2664: writing smithi08477203-29 from 729668 to 1138457 tid 1 2024-09-20T22:52:42.312 INFO:tasks.rados.rados.0.smithi084.stdout:2664: writing smithi08477203-29 from 1820303 to 2577614 tid 2 2024-09-20T22:52:42.318 INFO:tasks.rados.rados.0.smithi084.stdout:2664: writing smithi08477203-29 from 3274610 to 3529668 tid 3 2024-09-20T22:52:42.318 INFO:tasks.rados.rados.0.smithi084.stdout:2665: delete oid 12 current snap is 305 2024-09-20T22:52:42.319 INFO:tasks.rados.rados.0.smithi084.stdout:2661: finishing rollback tid 0 to smithi08477203-47 2024-09-20T22:52:42.319 INFO:tasks.rados.rados.0.smithi084.stdout:2661: finishing rollback tid 1 to smithi08477203-47 2024-09-20T22:52:42.319 INFO:tasks.rados.rados.0.smithi084.stdout:2661: finishing rollback tid 2 to smithi08477203-47 2024-09-20T22:52:42.319 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 47 v 1361 (ObjNum 671 snap 205 seq_num 671) dirty exists 2024-09-20T22:52:42.319 INFO:tasks.rados.rados.0.smithi084.stdout:2660: finishing write tid 2 to smithi08477203-5 2024-09-20T22:52:42.319 INFO:tasks.rados.rados.0.smithi084.stdout:2660: finishing write tid 3 to smithi08477203-5 2024-09-20T22:52:42.319 INFO:tasks.rados.rados.0.smithi084.stdout:2660: finishing write tid 4 to smithi08477203-5 2024-09-20T22:52:42.319 INFO:tasks.rados.rados.0.smithi084.stdout:2660: oid 5 updating version 0 to 773 2024-09-20T22:52:42.319 INFO:tasks.rados.rados.0.smithi084.stdout:2660: oid 5 updating version 773 to 774 2024-09-20T22:52:42.319 INFO:tasks.rados.rados.0.smithi084.stdout:2660: oid 5 version 774 is already newer than 772 2024-09-20T22:52:42.319 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 5 v 774 (ObjNum 922 snap 305 seq_num 922) dirty exists 2024-09-20T22:52:42.319 INFO:tasks.rados.rados.0.smithi084.stdout:2660: left oid 5 (ObjNum 922 snap 305 seq_num 922) 2024-09-20T22:52:42.319 INFO:tasks.rados.rados.0.smithi084.stdout:2662: finishing write tid 1 to smithi08477203-8 2024-09-20T22:52:42.320 INFO:tasks.rados.rados.0.smithi084.stdout:2662: finishing write tid 2 to smithi08477203-8 2024-09-20T22:52:42.320 INFO:tasks.rados.rados.0.smithi084.stdout:2662: finishing write tid 3 to smithi08477203-8 2024-09-20T22:52:42.320 INFO:tasks.rados.rados.0.smithi084.stdout:2662: finishing write tid 4 to smithi08477203-8 2024-09-20T22:52:42.320 INFO:tasks.rados.rados.0.smithi084.stdout:2662: finishing write tid 5 to smithi08477203-8 2024-09-20T22:52:42.320 INFO:tasks.rados.rados.0.smithi084.stdout:2662: oid 8 updating version 0 to 1823 2024-09-20T22:52:42.320 INFO:tasks.rados.rados.0.smithi084.stdout:2662: oid 8 updating version 1823 to 1824 2024-09-20T22:52:42.320 INFO:tasks.rados.rados.0.smithi084.stdout:2662: oid 8 updating version 1824 to 1825 2024-09-20T22:52:42.320 INFO:tasks.rados.rados.0.smithi084.stdout:2662: oid 8 updating version 1825 to 1826 2024-09-20T22:52:42.320 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 8 v 1826 (ObjNum 923 snap 305 seq_num 923) dirty exists 2024-09-20T22:52:42.320 INFO:tasks.rados.rados.0.smithi084.stdout:2662: left oid 8 (ObjNum 923 snap 305 seq_num 923) 2024-09-20T22:52:42.320 INFO:tasks.rados.rados.0.smithi084.stdout:2664: finishing write tid 1 to smithi08477203-29 2024-09-20T22:52:42.320 INFO:tasks.rados.rados.0.smithi084.stdout:2664: finishing write tid 2 to smithi08477203-29 2024-09-20T22:52:42.321 INFO:tasks.rados.rados.0.smithi084.stdout:2660: done (4 left) 2024-09-20T22:52:42.321 INFO:tasks.rados.rados.0.smithi084.stdout:2661: done (3 left) 2024-09-20T22:52:42.321 INFO:tasks.rados.rados.0.smithi084.stdout:2662: done (2 left) 2024-09-20T22:52:42.321 INFO:tasks.rados.rados.0.smithi084.stdout:2665: done (1 left) 2024-09-20T22:52:42.321 INFO:tasks.rados.rados.0.smithi084.stdout:2666: write (excl) oid 13 current snap is 305 2024-09-20T22:52:42.321 INFO:tasks.rados.rados.0.smithi084.stdout:2666: seq_num 926 ranges {763282=559287,1820914=760980,3213103=350179} 2024-09-20T22:52:42.328 INFO:tasks.rados.rados.0.smithi084.stdout:2666: writing smithi08477203-13 from 763282 to 1322569 tid 1 2024-09-20T22:52:42.330 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:42.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:42.332 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:52:42.336 INFO:tasks.rados.rados.0.smithi084.stdout:2666: writing smithi08477203-13 from 1820914 to 2581894 tid 2 2024-09-20T22:52:42.342 INFO:tasks.rados.rados.0.smithi084.stdout:2666: writing smithi08477203-13 from 3213103 to 3563282 tid 3 2024-09-20T22:52:42.342 INFO:tasks.rados.rados.0.smithi084.stdout:2667: read oid 44 snap -1 2024-09-20T22:52:42.342 INFO:tasks.rados.rados.0.smithi084.stdout:2667: expect (ObjNum 879 snap 285 seq_num 879) 2024-09-20T22:52:42.342 INFO:tasks.rados.rados.0.smithi084.stdout:2668: copy_from oid 14 from oid 9 current snap is 305 2024-09-20T22:52:42.342 INFO:tasks.rados.rados.0.smithi084.stdout:2664: finishing write tid 3 to smithi08477203-29 2024-09-20T22:52:42.342 INFO:tasks.rados.rados.0.smithi084.stdout:2664: finishing write tid 4 to smithi08477203-29 2024-09-20T22:52:42.342 INFO:tasks.rados.rados.0.smithi084.stdout:2664: finishing write tid 5 to smithi08477203-29 2024-09-20T22:52:42.342 INFO:tasks.rados.rados.0.smithi084.stdout:2664: oid 29 updating version 0 to 2061 2024-09-20T22:52:42.342 INFO:tasks.rados.rados.0.smithi084.stdout:2664: oid 29 version 2061 is already newer than 2060 2024-09-20T22:52:42.342 INFO:tasks.rados.rados.0.smithi084.stdout:2664: oid 29 updating version 2061 to 2062 2024-09-20T22:52:42.342 INFO:tasks.rados.rados.0.smithi084.stdout:2664: oid 29 version 2062 is already newer than 2049 2024-09-20T22:52:42.343 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 29 v 2062 (ObjNum 924 snap 305 seq_num 924) dirty exists 2024-09-20T22:52:42.343 INFO:tasks.rados.rados.0.smithi084.stdout:2664: left oid 29 (ObjNum 924 snap 305 seq_num 924) 2024-09-20T22:52:42.343 INFO:tasks.rados.rados.0.smithi084.stdout:2664: done (3 left) 2024-09-20T22:52:42.343 INFO:tasks.rados.rados.0.smithi084.stdout:2669: rollback oid 27 current snap is 305 2024-09-20T22:52:42.343 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 27 to 298 2024-09-20T22:52:42.343 INFO:tasks.rados.rados.0.smithi084.stdout:2666: finishing write tid 1 to smithi08477203-13 2024-09-20T22:52:42.343 INFO:tasks.rados.rados.0.smithi084.stdout:2666: finishing write tid 2 to smithi08477203-13 2024-09-20T22:52:42.343 INFO:tasks.rados.rados.0.smithi084.stdout:2670: snap_create 2024-09-20T22:52:42.343 INFO:tasks.rados.rados.0.smithi084.stdout:2668: finishing copy_from to smithi08477203-14 2024-09-20T22:52:42.344 INFO:tasks.rados.rados.0.smithi084.stdout:2668: got expected ENOENT (src dne) 2024-09-20T22:52:42.344 INFO:tasks.rados.rados.0.smithi084.stdout:2668: finishing copy_from racing read to smithi08477203-14 2024-09-20T22:52:42.344 INFO:tasks.rados.rados.0.smithi084.stdout:2669: finishing rollback tid 0 to smithi08477203-27 2024-09-20T22:52:42.344 INFO:tasks.rados.rados.0.smithi084.stdout:2669: finishing rollback tid 1 to smithi08477203-27 2024-09-20T22:52:42.344 INFO:tasks.rados.rados.0.smithi084.stdout:2669: finishing rollback tid 2 to smithi08477203-27 2024-09-20T22:52:42.345 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 27 v 748 (ObjNum 898 snap 295 seq_num 898) dirty exists 2024-09-20T22:52:42.345 INFO:tasks.rados.rados.0.smithi084.stdout:2666: finishing write tid 3 to smithi08477203-13 2024-09-20T22:52:42.348 INFO:tasks.rados.rados.0.smithi084.stdout:2666: finishing write tid 4 to smithi08477203-13 2024-09-20T22:52:42.348 INFO:tasks.rados.rados.0.smithi084.stdout:2666: finishing write tid 5 to smithi08477203-13 2024-09-20T22:52:42.348 INFO:tasks.rados.rados.0.smithi084.stdout:2666: oid 13 updating version 0 to 2065 2024-09-20T22:52:42.348 INFO:tasks.rados.rados.0.smithi084.stdout:2666: oid 13 version 2065 is already newer than 2064 2024-09-20T22:52:42.348 INFO:tasks.rados.rados.0.smithi084.stdout:2666: oid 13 updating version 2065 to 2066 2024-09-20T22:52:42.348 INFO:tasks.rados.rados.0.smithi084.stdout:2666: oid 13 version 2066 is already newer than 2063 2024-09-20T22:52:42.348 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 13 v 2066 (ObjNum 926 snap 305 seq_num 926) dirty exists 2024-09-20T22:52:42.348 INFO:tasks.rados.rados.0.smithi084.stdout:2666: left oid 13 (ObjNum 926 snap 305 seq_num 926) 2024-09-20T22:52:42.353 INFO:tasks.rados.rados.0.smithi084.stdout:2667: expect (ObjNum 879 snap 285 seq_num 879) 2024-09-20T22:52:42.433 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:52:42.454 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:42.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:42.455 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:52:42.556 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:52:42.577 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:42.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:42.578 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:52:42.679 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:52:42.700 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:42.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:42.701 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:52:42.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:42 smithi154 ceph-mon[33264]: 5.19 scrub starts 2024-09-20T22:52:42.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:42 smithi154 ceph-mon[33264]: 5.19 scrub ok 2024-09-20T22:52:42.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:42 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/2082370852' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:52:42.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:42 smithi154 ceph-mon[33264]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T22:52:42.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:42 smithi154 ceph-mon[33264]: Cluster is now healthy 2024-09-20T22:52:42.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:42 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/2082370852' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T22:52:42.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:42 smithi154 ceph-mon[33264]: osdmap e682: 8 total, 8 up, 7 in 2024-09-20T22:52:42.802 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:52:42.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:42.825 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:42.825 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:52:42.861 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:42 smithi084 ceph-mon[40024]: 5.19 scrub starts 2024-09-20T22:52:42.861 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:42 smithi084 ceph-mon[40024]: 5.19 scrub ok 2024-09-20T22:52:42.861 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:42 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/2082370852' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:52:42.861 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:42 smithi084 ceph-mon[40024]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T22:52:42.862 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:42 smithi084 ceph-mon[40024]: Cluster is now healthy 2024-09-20T22:52:42.862 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:42 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/2082370852' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T22:52:42.862 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:42 smithi084 ceph-mon[40024]: osdmap e682: 8 total, 8 up, 7 in 2024-09-20T22:52:42.862 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:42 smithi084 ceph-mon[29774]: 5.19 scrub starts 2024-09-20T22:52:42.862 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:42 smithi084 ceph-mon[29774]: 5.19 scrub ok 2024-09-20T22:52:42.862 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:42 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/2082370852' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:52:42.862 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:42 smithi084 ceph-mon[29774]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T22:52:42.862 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:42 smithi084 ceph-mon[29774]: Cluster is now healthy 2024-09-20T22:52:42.862 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:42 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/2082370852' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T22:52:42.862 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:42 smithi084 ceph-mon[29774]: osdmap e682: 8 total, 8 up, 7 in 2024-09-20T22:52:42.926 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:52:42.948 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:42.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:42.949 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:52:43.050 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:52:43.071 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:43.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:43.072 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:52:43.173 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:52:43.194 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:43.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:43.195 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:52:43.276 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.3 to 1 (10000) 2024-09-20T22:52:43.282 INFO:tasks.rados.rados.0.smithi084.stdout:2666: done (4 left) 2024-09-20T22:52:43.282 INFO:tasks.rados.rados.0.smithi084.stdout:2667: done (3 left) 2024-09-20T22:52:43.282 INFO:tasks.rados.rados.0.smithi084.stdout:2668: done (2 left) 2024-09-20T22:52:43.282 INFO:tasks.rados.rados.0.smithi084.stdout:2669: done (1 left) 2024-09-20T22:52:43.282 INFO:tasks.rados.rados.0.smithi084.stdout:2670: done (0 left) 2024-09-20T22:52:43.282 INFO:tasks.rados.rados.0.smithi084.stdout:2671: snap_remove snap 286 2024-09-20T22:52:43.297 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:52:43.321 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:43.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:43.322 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:52:43.423 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:52:43.444 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:43.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:43.445 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:52:43.546 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:52:43.567 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:43.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:43.568 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:52:43.668 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:52:43.691 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:43.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:43.692 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:52:43.734 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:43 smithi084 ceph-mon[29774]: pgmap v924: 158 pgs: 1 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 154 active+clean; 599 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail; 2.0 KiB/s rd, 1.8 MiB/s wr, 6 op/s 2024-09-20T22:52:43.734 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:43 smithi084 ceph-mon[29774]: 1.1 scrub starts 2024-09-20T22:52:43.734 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:43 smithi084 ceph-mon[29774]: 1.1 scrub ok 2024-09-20T22:52:43.734 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:43 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/2669687718' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-09-20T22:52:43.734 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:43 smithi084 ceph-mon[29774]: 3.1b scrub starts 2024-09-20T22:52:43.734 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:43 smithi084 ceph-mon[29774]: 3.1b scrub ok 2024-09-20T22:52:43.734 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:43 smithi084 ceph-mon[29774]: 5.a scrub starts 2024-09-20T22:52:43.734 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:43 smithi084 ceph-mon[29774]: 5.a scrub ok 2024-09-20T22:52:43.734 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:43 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/2669687718' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-20T22:52:43.734 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:43 smithi084 ceph-mon[29774]: osdmap e683: 8 total, 8 up, 7 in 2024-09-20T22:52:43.735 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:43 smithi084 ceph-mon[40024]: pgmap v924: 158 pgs: 1 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 154 active+clean; 599 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail; 2.0 KiB/s rd, 1.8 MiB/s wr, 6 op/s 2024-09-20T22:52:43.735 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:43 smithi084 ceph-mon[40024]: 1.1 scrub starts 2024-09-20T22:52:43.735 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:43 smithi084 ceph-mon[40024]: 1.1 scrub ok 2024-09-20T22:52:43.735 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:43 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/2669687718' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-09-20T22:52:43.735 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:43 smithi084 ceph-mon[40024]: 3.1b scrub starts 2024-09-20T22:52:43.735 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:43 smithi084 ceph-mon[40024]: 3.1b scrub ok 2024-09-20T22:52:43.735 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:43 smithi084 ceph-mon[40024]: 5.a scrub starts 2024-09-20T22:52:43.735 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:43 smithi084 ceph-mon[40024]: 5.a scrub ok 2024-09-20T22:52:43.735 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:43 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/2669687718' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-20T22:52:43.735 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:43 smithi084 ceph-mon[40024]: osdmap e683: 8 total, 8 up, 7 in 2024-09-20T22:52:43.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:43 smithi154 ceph-mon[33264]: pgmap v924: 158 pgs: 1 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 154 active+clean; 599 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail; 2.0 KiB/s rd, 1.8 MiB/s wr, 6 op/s 2024-09-20T22:52:43.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:43 smithi154 ceph-mon[33264]: 1.1 scrub starts 2024-09-20T22:52:43.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:43 smithi154 ceph-mon[33264]: 1.1 scrub ok 2024-09-20T22:52:43.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:43 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/2669687718' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-09-20T22:52:43.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:43 smithi154 ceph-mon[33264]: 3.1b scrub starts 2024-09-20T22:52:43.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:43 smithi154 ceph-mon[33264]: 3.1b scrub ok 2024-09-20T22:52:43.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:43 smithi154 ceph-mon[33264]: 5.a scrub starts 2024-09-20T22:52:43.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:43 smithi154 ceph-mon[33264]: 5.a scrub ok 2024-09-20T22:52:43.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:43 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/2669687718' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-20T22:52:43.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:43 smithi154 ceph-mon[33264]: osdmap e683: 8 total, 8 up, 7 in 2024-09-20T22:52:43.792 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:52:43.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:43.815 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:43.815 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:52:43.916 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:52:43.938 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:43.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:43.938 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:52:44.018 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-20T22:52:44.019 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:52:44.039 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:52:44.060 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:44.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:44.061 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:52:44.161 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:52:44.183 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:44.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:44.183 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:52:44.283 INFO:tasks.rados.rados.0.smithi084.stdout:2671: done (0 left) 2024-09-20T22:52:44.283 INFO:tasks.rados.rados.0.smithi084.stdout:2672: read oid 25 snap 287 2024-09-20T22:52:44.283 INFO:tasks.rados.rados.0.smithi084.stdout:2672: expect (ObjNum 886 snap 287 seq_num 886) 2024-09-20T22:52:44.283 INFO:tasks.rados.rados.0.smithi084.stdout:2673: write oid 23 current snap is 306 2024-09-20T22:52:44.283 INFO:tasks.rados.rados.0.smithi084.stdout:2673: seq_num 927 ranges {780089=634536,1879396=620991,3240526=339563} 2024-09-20T22:52:44.284 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:52:44.291 INFO:tasks.rados.rados.0.smithi084.stdout:2673: writing smithi08477203-23 from 780089 to 1414625 tid 1 2024-09-20T22:52:44.298 INFO:tasks.rados.rados.0.smithi084.stdout:2673: writing smithi08477203-23 from 1879396 to 2500387 tid 2 2024-09-20T22:52:44.304 INFO:tasks.rados.rados.0.smithi084.stdout:2673: writing smithi08477203-23 from 3240526 to 3580089 tid 3 2024-09-20T22:52:44.304 INFO:tasks.rados.rados.0.smithi084.stdout:2674: read oid 45 snap -1 2024-09-20T22:52:44.305 INFO:tasks.rados.rados.0.smithi084.stdout:2674: expect (ObjNum 846 snap 274 seq_num 846) 2024-09-20T22:52:44.305 INFO:tasks.rados.rados.0.smithi084.stdout:2675: snap_create 2024-09-20T22:52:44.305 INFO:tasks.rados.rados.0.smithi084.stdout:2672: expect (ObjNum 886 snap 287 seq_num 886) 2024-09-20T22:52:44.310 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:44.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:44.311 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:52:44.412 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:52:44.433 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:44.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:44.434 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:52:44.435 INFO:tasks.rados.rados.0.smithi084.stdout:2673: finishing write tid 1 to smithi08477203-23 2024-09-20T22:52:44.435 INFO:tasks.rados.rados.0.smithi084.stdout:2673: finishing write tid 2 to smithi08477203-23 2024-09-20T22:52:44.435 INFO:tasks.rados.rados.0.smithi084.stdout:2673: finishing write tid 3 to smithi08477203-23 2024-09-20T22:52:44.435 INFO:tasks.rados.rados.0.smithi084.stdout:2673: finishing write tid 4 to smithi08477203-23 2024-09-20T22:52:44.435 INFO:tasks.rados.rados.0.smithi084.stdout:2673: finishing write tid 5 to smithi08477203-23 2024-09-20T22:52:44.435 INFO:tasks.rados.rados.0.smithi084.stdout:2673: oid 23 updating version 0 to 1239 2024-09-20T22:52:44.435 INFO:tasks.rados.rados.0.smithi084.stdout:2673: oid 23 updating version 1239 to 1240 2024-09-20T22:52:44.435 INFO:tasks.rados.rados.0.smithi084.stdout:2673: oid 23 version 1240 is already newer than 1237 2024-09-20T22:52:44.435 INFO:tasks.rados.rados.0.smithi084.stdout:2673: oid 23 version 1240 is already newer than 1238 2024-09-20T22:52:44.435 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 23 v 1240 (ObjNum 927 snap 306 seq_num 927) dirty exists 2024-09-20T22:52:44.435 INFO:tasks.rados.rados.0.smithi084.stdout:2673: left oid 23 (ObjNum 927 snap 306 seq_num 927) 2024-09-20T22:52:44.435 INFO:tasks.rados.rados.0.smithi084.stdout:2674: expect (ObjNum 846 snap 274 seq_num 846) 2024-09-20T22:52:44.535 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:52:44.555 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:44.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:44.556 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:52:44.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:44 smithi084 ceph-mon[29774]: 3.15 scrub starts 2024-09-20T22:52:44.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:44 smithi084 ceph-mon[29774]: 3.15 scrub ok 2024-09-20T22:52:44.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:44 smithi084 ceph-mon[29774]: 6.11 scrub starts 2024-09-20T22:52:44.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:44 smithi084 ceph-mon[29774]: 6.11 scrub ok 2024-09-20T22:52:44.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:44 smithi084 ceph-mon[29774]: 5.1d scrub starts 2024-09-20T22:52:44.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:44 smithi084 ceph-mon[29774]: 5.1d scrub ok 2024-09-20T22:52:44.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:44 smithi084 ceph-mon[29774]: osdmap e684: 8 total, 8 up, 7 in 2024-09-20T22:52:44.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:44 smithi084 ceph-mon[40024]: 3.15 scrub starts 2024-09-20T22:52:44.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:44 smithi084 ceph-mon[40024]: 3.15 scrub ok 2024-09-20T22:52:44.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:44 smithi084 ceph-mon[40024]: 6.11 scrub starts 2024-09-20T22:52:44.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:44 smithi084 ceph-mon[40024]: 6.11 scrub ok 2024-09-20T22:52:44.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:44 smithi084 ceph-mon[40024]: 5.1d scrub starts 2024-09-20T22:52:44.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:44 smithi084 ceph-mon[40024]: 5.1d scrub ok 2024-09-20T22:52:44.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:44 smithi084 ceph-mon[40024]: osdmap e684: 8 total, 8 up, 7 in 2024-09-20T22:52:44.657 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:52:44.678 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:44.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:44.679 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:52:44.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:44 smithi154 ceph-mon[33264]: 3.15 scrub starts 2024-09-20T22:52:44.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:44 smithi154 ceph-mon[33264]: 3.15 scrub ok 2024-09-20T22:52:44.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:44 smithi154 ceph-mon[33264]: 6.11 scrub starts 2024-09-20T22:52:44.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:44 smithi154 ceph-mon[33264]: 6.11 scrub ok 2024-09-20T22:52:44.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:44 smithi154 ceph-mon[33264]: 5.1d scrub starts 2024-09-20T22:52:44.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:44 smithi154 ceph-mon[33264]: 5.1d scrub ok 2024-09-20T22:52:44.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:44 smithi154 ceph-mon[33264]: osdmap e684: 8 total, 8 up, 7 in 2024-09-20T22:52:44.780 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:52:44.801 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:44.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:44.802 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:52:44.885 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.2 dump_historic_ops 2024-09-20T22:52:44.903 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:52:44.948 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:44.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:44.949 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:52:45.050 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:52:45.074 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:45.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:45.075 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:52:45.130 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:52:45.130 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:52:45.147 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:52:45.147 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:52:45.164 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:52:45.164 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:52:45.175 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:52:45.180 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-20T22:52:45.180 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T22:52:45.196 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:45.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:45.197 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:52:45.197 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:52:45.197 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:52:45.214 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:52:45.214 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:52:45.231 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:52:45.231 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:52:45.249 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:52:45.250 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:52:45.285 INFO:tasks.rados.rados.0.smithi084.stdout:2672: done (3 left) 2024-09-20T22:52:45.285 INFO:tasks.rados.rados.0.smithi084.stdout:2673: done (2 left) 2024-09-20T22:52:45.285 INFO:tasks.rados.rados.0.smithi084.stdout:2674: done (1 left) 2024-09-20T22:52:45.286 INFO:tasks.rados.rados.0.smithi084.stdout:2675: done (0 left) 2024-09-20T22:52:45.286 INFO:tasks.rados.rados.0.smithi084.stdout:2676: read oid 11 snap -1 2024-09-20T22:52:45.286 INFO:tasks.rados.rados.0.smithi084.stdout:2676: expect (ObjNum 899 snap 295 seq_num 899) 2024-09-20T22:52:45.286 INFO:tasks.rados.rados.0.smithi084.stdout:2677: rollback oid 47 current snap is 307 2024-09-20T22:52:45.286 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 47 to 189 2024-09-20T22:52:45.286 INFO:tasks.rados.rados.0.smithi084.stdout:2678: snap_create 2024-09-20T22:52:45.291 INFO:tasks.rados.rados.0.smithi084.stdout:2677: finishing rollback tid 0 to smithi08477203-47 2024-09-20T22:52:45.292 INFO:tasks.rados.rados.0.smithi084.stdout:2677: finishing rollback tid 1 to smithi08477203-47 2024-09-20T22:52:45.293 INFO:tasks.rados.rados.0.smithi084.stdout:2677: finishing rollback tid 2 to smithi08477203-47 2024-09-20T22:52:45.293 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 47 v 1364 (ObjNum 566 snap 170 seq_num 566) dirty exists 2024-09-20T22:52:45.293 INFO:tasks.rados.rados.0.smithi084.stdout:2676: expect (ObjNum 899 snap 295 seq_num 899) 2024-09-20T22:52:45.298 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:52:45.320 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:45.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:45.321 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:52:45.422 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:52:45.443 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:45.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:45.444 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:52:45.545 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:52:45.567 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:45.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:45.568 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:52:45.668 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:52:45.690 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:45.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:45.691 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:52:45.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:45 smithi154 ceph-mon[33264]: pgmap v927: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 612 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail; 3.0 MiB/s rd, 1.9 MiB/s wr, 13 op/s 2024-09-20T22:52:45.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:45 smithi154 ceph-mon[33264]: 4.e scrub starts 2024-09-20T22:52:45.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:45 smithi154 ceph-mon[33264]: 4.e scrub ok 2024-09-20T22:52:45.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:45 smithi154 ceph-mon[33264]: 6.16 scrub starts 2024-09-20T22:52:45.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:45 smithi154 ceph-mon[33264]: 6.16 scrub ok 2024-09-20T22:52:45.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:45 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/4080013855' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:52:45.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:45 smithi154 ceph-mon[33264]: osdmap e685: 8 total, 8 up, 7 in 2024-09-20T22:52:45.792 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:52:45.813 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:45.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:45.814 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:52:45.825 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd set noscrub 2024-09-20T22:52:45.847 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:45 smithi084 ceph-mon[40024]: pgmap v927: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 612 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail; 3.0 MiB/s rd, 1.9 MiB/s wr, 13 op/s 2024-09-20T22:52:45.847 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:45 smithi084 ceph-mon[40024]: 4.e scrub starts 2024-09-20T22:52:45.847 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:45 smithi084 ceph-mon[40024]: 4.e scrub ok 2024-09-20T22:52:45.847 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:45 smithi084 ceph-mon[40024]: 6.16 scrub starts 2024-09-20T22:52:45.847 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:45 smithi084 ceph-mon[40024]: 6.16 scrub ok 2024-09-20T22:52:45.847 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:45 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/4080013855' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:52:45.847 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:45 smithi084 ceph-mon[40024]: osdmap e685: 8 total, 8 up, 7 in 2024-09-20T22:52:45.847 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:45 smithi084 ceph-mon[29774]: pgmap v927: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 612 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail; 3.0 MiB/s rd, 1.9 MiB/s wr, 13 op/s 2024-09-20T22:52:45.848 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:45 smithi084 ceph-mon[29774]: 4.e scrub starts 2024-09-20T22:52:45.848 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:45 smithi084 ceph-mon[29774]: 4.e scrub ok 2024-09-20T22:52:45.848 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:45 smithi084 ceph-mon[29774]: 6.16 scrub starts 2024-09-20T22:52:45.848 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:45 smithi084 ceph-mon[29774]: 6.16 scrub ok 2024-09-20T22:52:45.848 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:45 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/4080013855' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:52:45.848 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:45 smithi084 ceph-mon[29774]: osdmap e685: 8 total, 8 up, 7 in 2024-09-20T22:52:45.902 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd reweight 5 1 2024-09-20T22:52:45.915 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:52:45.964 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:45.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:45.965 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:52:46.065 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:52:46.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:46.088 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:46.088 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:52:46.189 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:52:46.211 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:46.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:46.212 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:52:46.285 INFO:tasks.rados.rados.0.smithi084.stdout:2676: done (2 left) 2024-09-20T22:52:46.286 INFO:tasks.rados.rados.0.smithi084.stdout:2677: done (1 left) 2024-09-20T22:52:46.286 INFO:tasks.rados.rados.0.smithi084.stdout:2678: done (0 left) 2024-09-20T22:52:46.286 INFO:tasks.rados.rados.0.smithi084.stdout:2679: write oid 49 current snap is 308 2024-09-20T22:52:46.286 INFO:tasks.rados.rados.0.smithi084.stdout:2679: seq_num 928 ranges {796896=426138,1654231=764649,2984302=480075} 2024-09-20T22:52:46.292 INFO:tasks.rados.rados.0.smithi084.stdout:2679: writing smithi08477203-49 from 796896 to 1223034 tid 1 2024-09-20T22:52:46.299 INFO:tasks.rados.rados.0.smithi084.stdout:2679: writing smithi08477203-49 from 1654231 to 2418880 tid 2 2024-09-20T22:52:46.305 INFO:tasks.rados.rados.0.smithi084.stdout:2679: writing smithi08477203-49 from 2984302 to 3464377 tid 3 2024-09-20T22:52:46.305 INFO:tasks.rados.rados.0.smithi084.stdout:2680: write (excl) oid 34 current snap is 308 2024-09-20T22:52:46.305 INFO:tasks.rados.rados.0.smithi084.stdout:2680: seq_num 929 ranges {413703=501387,1596360=624660,2895372=691347} 2024-09-20T22:52:46.310 INFO:tasks.rados.rados.0.smithi084.stdout:2680: writing smithi08477203-34 from 413703 to 915090 tid 1 2024-09-20T22:52:46.313 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:52:46.317 INFO:tasks.rados.rados.0.smithi084.stdout:2680: writing smithi08477203-34 from 1596360 to 2221020 tid 2 2024-09-20T22:52:46.325 INFO:tasks.rados.rados.0.smithi084.stdout:2680: writing smithi08477203-34 from 2895372 to 3586719 tid 3 2024-09-20T22:52:46.325 INFO:tasks.rados.rados.0.smithi084.stdout:2681: write oid 47 current snap is 308 2024-09-20T22:52:46.325 INFO:tasks.rados.rados.0.smithi084.stdout:2679: finishing write tid 1 to smithi08477203-49 2024-09-20T22:52:46.325 INFO:tasks.rados.rados.0.smithi084.stdout:2681: seq_num 930 ranges {430510=576636,1654842=768318,2922795=502619} 2024-09-20T22:52:46.330 INFO:tasks.rados.rados.0.smithi084.stdout:2681: writing smithi08477203-47 from 430510 to 1007146 tid 1 2024-09-20T22:52:46.335 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:46.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:46.336 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:52:46.338 INFO:tasks.rados.rados.0.smithi084.stdout:2681: writing smithi08477203-47 from 1654842 to 2423160 tid 2 2024-09-20T22:52:46.344 INFO:tasks.rados.rados.0.smithi084.stdout:2681: writing smithi08477203-47 from 2922795 to 3425414 tid 3 2024-09-20T22:52:46.344 INFO:tasks.rados.rados.0.smithi084.stdout:2682: snap_remove snap 251 2024-09-20T22:52:46.437 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:52:46.458 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:46.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:46.459 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:52:46.560 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:52:46.578 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:46 smithi084 ceph-mon[40024]: 3.b deep-scrub starts 2024-09-20T22:52:46.579 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:46 smithi084 ceph-mon[40024]: 3.b deep-scrub ok 2024-09-20T22:52:46.579 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:46 smithi084 ceph-mon[40024]: 2.0 scrub starts 2024-09-20T22:52:46.579 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:46 smithi084 ceph-mon[40024]: 2.0 scrub ok 2024-09-20T22:52:46.579 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:46 smithi084 ceph-mon[40024]: 5.1c scrub starts 2024-09-20T22:52:46.579 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:46 smithi084 ceph-mon[40024]: 5.1c scrub ok 2024-09-20T22:52:46.579 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:46 smithi084 ceph-mon[40024]: 7.e scrub starts 2024-09-20T22:52:46.579 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:46 smithi084 ceph-mon[40024]: 7.e scrub ok 2024-09-20T22:52:46.579 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:46 smithi084 ceph-mon[40024]: osdmap e686: 8 total, 8 up, 7 in 2024-09-20T22:52:46.579 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:46 smithi084 ceph-mon[29774]: 3.b deep-scrub starts 2024-09-20T22:52:46.580 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:46 smithi084 ceph-mon[29774]: 3.b deep-scrub ok 2024-09-20T22:52:46.580 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:46 smithi084 ceph-mon[29774]: 2.0 scrub starts 2024-09-20T22:52:46.580 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:46 smithi084 ceph-mon[29774]: 2.0 scrub ok 2024-09-20T22:52:46.580 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:46 smithi084 ceph-mon[29774]: 5.1c scrub starts 2024-09-20T22:52:46.580 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:46 smithi084 ceph-mon[29774]: 5.1c scrub ok 2024-09-20T22:52:46.580 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:46 smithi084 ceph-mon[29774]: 7.e scrub starts 2024-09-20T22:52:46.580 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:46 smithi084 ceph-mon[29774]: 7.e scrub ok 2024-09-20T22:52:46.580 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:46 smithi084 ceph-mon[29774]: osdmap e686: 8 total, 8 up, 7 in 2024-09-20T22:52:46.581 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:46.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:46.582 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:52:46.666 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:52:46.669 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:52:46.683 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:52:46.704 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:46.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:46.705 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:52:46.709 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:52:46.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:46 smithi154 ceph-mon[33264]: 3.b deep-scrub starts 2024-09-20T22:52:46.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:46 smithi154 ceph-mon[33264]: 3.b deep-scrub ok 2024-09-20T22:52:46.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:46 smithi154 ceph-mon[33264]: 2.0 scrub starts 2024-09-20T22:52:46.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:46 smithi154 ceph-mon[33264]: 2.0 scrub ok 2024-09-20T22:52:46.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:46 smithi154 ceph-mon[33264]: 5.1c scrub starts 2024-09-20T22:52:46.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:46 smithi154 ceph-mon[33264]: 5.1c scrub ok 2024-09-20T22:52:46.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:46 smithi154 ceph-mon[33264]: 7.e scrub starts 2024-09-20T22:52:46.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:46 smithi154 ceph-mon[33264]: 7.e scrub ok 2024-09-20T22:52:46.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:46 smithi154 ceph-mon[33264]: osdmap e686: 8 total, 8 up, 7 in 2024-09-20T22:52:46.806 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:52:46.828 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:46.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:46.829 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:52:46.929 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:52:46.951 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:46.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:46.952 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:52:47.052 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:52:47.075 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:47.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:47.076 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:52:47.176 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:52:47.197 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:47.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:47.199 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:52:47.285 INFO:tasks.rados.rados.0.smithi084.stdout:2682: done (3 left) 2024-09-20T22:52:47.286 INFO:tasks.rados.rados.0.smithi084.stdout:2683: read oid 5 snap -1 2024-09-20T22:52:47.286 INFO:tasks.rados.rados.0.smithi084.stdout:2683: expect (ObjNum 922 snap 305 seq_num 922) 2024-09-20T22:52:47.286 INFO:tasks.rados.rados.0.smithi084.stdout:2684: read oid 36 snap -1 2024-09-20T22:52:47.286 INFO:tasks.rados.rados.0.smithi084.stdout:2684: expect (ObjNum 920 snap 304 seq_num 920) 2024-09-20T22:52:47.286 INFO:tasks.rados.rados.0.smithi084.stdout:2685: copy_from oid 45 from oid 38 current snap is 308 2024-09-20T22:52:47.286 INFO:tasks.rados.rados.0.smithi084.stdout:2686: read oid 30 snap -1 2024-09-20T22:52:47.286 INFO:tasks.rados.rados.0.smithi084.stdout:2686: expect (ObjNum 876 snap 280 seq_num 876) 2024-09-20T22:52:47.286 INFO:tasks.rados.rados.0.smithi084.stdout:2687: delete oid 1 current snap is 308 2024-09-20T22:52:47.287 INFO:tasks.rados.rados.0.smithi084.stdout:2679: finishing write tid 2 to smithi08477203-49 2024-09-20T22:52:47.287 INFO:tasks.rados.rados.0.smithi084.stdout:2679: finishing write tid 3 to smithi08477203-49 2024-09-20T22:52:47.287 INFO:tasks.rados.rados.0.smithi084.stdout:2679: finishing write tid 4 to smithi08477203-49 2024-09-20T22:52:47.287 INFO:tasks.rados.rados.0.smithi084.stdout:2679: finishing write tid 5 to smithi08477203-49 2024-09-20T22:52:47.287 INFO:tasks.rados.rados.0.smithi084.stdout:2679: oid 49 updating version 0 to 1811 2024-09-20T22:52:47.287 INFO:tasks.rados.rados.0.smithi084.stdout:2679: oid 49 updating version 1811 to 1812 2024-09-20T22:52:47.287 INFO:tasks.rados.rados.0.smithi084.stdout:2679: oid 49 version 1812 is already newer than 1810 2024-09-20T22:52:47.287 INFO:tasks.rados.rados.0.smithi084.stdout:2679: oid 49 version 1812 is already newer than 1809 2024-09-20T22:52:47.287 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 49 v 1812 (ObjNum 928 snap 308 seq_num 928) dirty exists 2024-09-20T22:52:47.287 INFO:tasks.rados.rados.0.smithi084.stdout:2679: left oid 49 (ObjNum 928 snap 308 seq_num 928) 2024-09-20T22:52:47.287 INFO:tasks.rados.rados.0.smithi084.stdout:2680: finishing write tid 1 to smithi08477203-34 2024-09-20T22:52:47.287 INFO:tasks.rados.rados.0.smithi084.stdout:2680: finishing write tid 2 to smithi08477203-34 2024-09-20T22:52:47.288 INFO:tasks.rados.rados.0.smithi084.stdout:2680: finishing write tid 3 to smithi08477203-34 2024-09-20T22:52:47.288 INFO:tasks.rados.rados.0.smithi084.stdout:2680: finishing write tid 4 to smithi08477203-34 2024-09-20T22:52:47.288 INFO:tasks.rados.rados.0.smithi084.stdout:2680: finishing write tid 5 to smithi08477203-34 2024-09-20T22:52:47.288 INFO:tasks.rados.rados.0.smithi084.stdout:2680: oid 34 updating version 0 to 1815 2024-09-20T22:52:47.288 INFO:tasks.rados.rados.0.smithi084.stdout:2680: oid 34 updating version 1815 to 1816 2024-09-20T22:52:47.288 INFO:tasks.rados.rados.0.smithi084.stdout:2680: oid 34 version 1816 is already newer than 1814 2024-09-20T22:52:47.288 INFO:tasks.rados.rados.0.smithi084.stdout:2680: oid 34 version 1816 is already newer than 1813 2024-09-20T22:52:47.288 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 34 v 1816 (ObjNum 929 snap 308 seq_num 929) dirty exists 2024-09-20T22:52:47.288 INFO:tasks.rados.rados.0.smithi084.stdout:2680: left oid 34 (ObjNum 929 snap 308 seq_num 929) 2024-09-20T22:52:47.288 INFO:tasks.rados.rados.0.smithi084.stdout:2681: finishing write tid 1 to smithi08477203-47 2024-09-20T22:52:47.288 INFO:tasks.rados.rados.0.smithi084.stdout:2681: finishing write tid 2 to smithi08477203-47 2024-09-20T22:52:47.288 INFO:tasks.rados.rados.0.smithi084.stdout:2681: finishing write tid 3 to smithi08477203-47 2024-09-20T22:52:47.288 INFO:tasks.rados.rados.0.smithi084.stdout:2681: finishing write tid 4 to smithi08477203-47 2024-09-20T22:52:47.288 INFO:tasks.rados.rados.0.smithi084.stdout:2681: finishing write tid 5 to smithi08477203-47 2024-09-20T22:52:47.288 INFO:tasks.rados.rados.0.smithi084.stdout:2681: oid 47 updating version 0 to 1368 2024-09-20T22:52:47.288 INFO:tasks.rados.rados.0.smithi084.stdout:2681: oid 47 version 1368 is already newer than 1365 2024-09-20T22:52:47.289 INFO:tasks.rados.rados.0.smithi084.stdout:2681: oid 47 version 1368 is already newer than 1366 2024-09-20T22:52:47.289 INFO:tasks.rados.rados.0.smithi084.stdout:2681: oid 47 version 1368 is already newer than 1367 2024-09-20T22:52:47.289 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 47 v 1368 (ObjNum 930 snap 308 seq_num 930) dirty exists 2024-09-20T22:52:47.289 INFO:tasks.rados.rados.0.smithi084.stdout:2681: left oid 47 (ObjNum 930 snap 308 seq_num 930) 2024-09-20T22:52:47.296 INFO:tasks.rados.rados.0.smithi084.stdout:2683: expect (ObjNum 922 snap 305 seq_num 922) 2024-09-20T22:52:47.299 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:52:47.323 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:47.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:47.323 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:52:47.424 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:52:47.446 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:47.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:47.447 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:52:47.464 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-20T22:52:47.464 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2024-09-20T22:52:47.464 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2024-09-20T22:52:47.464 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [ 2024-09-20T22:52:47.464 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.464 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:901 7.e 7:7c62ced4:::smithi08477203-17:head [copy-from ver 48] snapc 1e=[] ondisk+write+known_if_redirected+supports_pool_eio e156)", 2024-09-20T22:52:47.464 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:43:55.624762+0000", 2024-09-20T22:52:47.464 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 531.83654605499999, 2024-09-20T22:52:47.464 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.021835856000000001, 2024-09-20T22:52:47.465 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:52:47.465 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:52:47.465 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:52:47.465 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:52:47.465 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:52:47.465 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:52:47.465 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 901 2024-09-20T22:52:47.465 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.465 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:52:47.465 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.465 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:52:47.465 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.624762+0000", 2024-09-20T22:52:47.465 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:52:47.466 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.466 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.466 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:52:47.466 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.624762+0000", 2024-09-20T22:52:47.466 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:52:47.466 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.466 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.466 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:52:47.466 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.624765+0000", 2024-09-20T22:52:47.466 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.57e-06 2024-09-20T22:52:47.466 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.466 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.466 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:52:47.466 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.624776+0000", 2024-09-20T22:52:47.467 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.078e-05 2024-09-20T22:52:47.467 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.467 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.467 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:52:47.467 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.624803+0000", 2024-09-20T22:52:47.467 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.6769999999999999e-05 2024-09-20T22:52:47.467 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.467 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.467 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:52:47.467 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.624815+0000", 2024-09-20T22:52:47.467 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2459e-05 2024-09-20T22:52:47.467 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.467 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.468 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:52:47.468 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.624936+0000", 2024-09-20T22:52:47.468 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000120467 2024-09-20T22:52:47.468 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.468 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.468 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:52:47.468 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.625046+0000", 2024-09-20T22:52:47.468 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000110222 2024-09-20T22:52:47.468 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.468 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.468 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 0,5", 2024-09-20T22:52:47.468 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.635547+0000", 2024-09-20T22:52:47.468 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010501765999999999 2024-09-20T22:52:47.468 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.468 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.469 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:52:47.469 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.644332+0000", 2024-09-20T22:52:47.469 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0087847729999999992 2024-09-20T22:52:47.469 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.469 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.469 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:52:47.469 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.646168+0000", 2024-09-20T22:52:47.469 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0018360010000000001 2024-09-20T22:52:47.469 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.469 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.469 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:52:47.469 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.646512+0000", 2024-09-20T22:52:47.469 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000343269 2024-09-20T22:52:47.469 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.470 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.470 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:52:47.470 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.646550+0000", 2024-09-20T22:52:47.470 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.8383000000000002e-05 2024-09-20T22:52:47.470 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.470 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.470 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:52:47.470 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.646598+0000", 2024-09-20T22:52:47.470 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.8396000000000002e-05 2024-09-20T22:52:47.470 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:52:47.470 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:52:47.470 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:52:47.470 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.471 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.471 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:902 7.e 7:7c62ced4:::smithi08477203-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e156)", 2024-09-20T22:52:47.471 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:43:55.624853+0000", 2024-09-20T22:52:47.471 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 531.83645597700001, 2024-09-20T22:52:47.471 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.015323603999999999, 2024-09-20T22:52:47.471 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:52:47.471 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:52:47.471 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-20T22:52:47.471 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:52:47.471 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:52:47.471 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:52:47.471 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 902 2024-09-20T22:52:47.471 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.471 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:52:47.471 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.472 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:52:47.472 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.624853+0000", 2024-09-20T22:52:47.472 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:52:47.472 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.472 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.472 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:52:47.472 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.624853+0000", 2024-09-20T22:52:47.472 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:52:47.472 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.472 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.472 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:52:47.472 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.624853+0000", 2024-09-20T22:52:47.472 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.1399999999999997e-07 2024-09-20T22:52:47.472 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.472 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.473 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:52:47.473 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.624859+0000", 2024-09-20T22:52:47.473 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.9610000000000002e-06 2024-09-20T22:52:47.473 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.473 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.473 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:52:47.473 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.624887+0000", 2024-09-20T22:52:47.473 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.7597e-05 2024-09-20T22:52:47.473 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.473 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.473 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:52:47.473 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.624894+0000", 2024-09-20T22:52:47.473 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.469e-06 2024-09-20T22:52:47.473 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.474 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.474 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:52:47.474 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.625219+0000", 2024-09-20T22:52:47.474 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00032538499999999998 2024-09-20T22:52:47.474 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.474 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.474 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for blocked object", 2024-09-20T22:52:47.474 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.625258+0000", 2024-09-20T22:52:47.474 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.8467000000000003e-05 2024-09-20T22:52:47.474 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.474 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.474 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:52:47.475 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.640050+0000", 2024-09-20T22:52:47.475 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0147923 2024-09-20T22:52:47.475 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.475 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.475 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:52:47.475 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.640112+0000", 2024-09-20T22:52:47.475 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.1723000000000005e-05 2024-09-20T22:52:47.475 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.475 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.475 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:52:47.475 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:55.640176+0000", 2024-09-20T22:52:47.475 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.4188000000000003e-05 2024-09-20T22:52:47.475 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:52:47.476 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:52:47.476 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:52:47.476 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.476 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.476 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24652.0:1420 7.19 e189/139 7:9cc0b7e2:::smithi08477203-46:head v 189'99, mlcod=174'95)", 2024-09-20T22:52:47.476 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:44:28.150258+0000", 2024-09-20T22:52:47.476 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 499.31105028100001, 2024-09-20T22:52:47.476 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014180626, 2024-09-20T22:52:47.476 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:52:47.476 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:52:47.476 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:52:47.476 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:52:47.476 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.476 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:52:47.477 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.150258+0000", 2024-09-20T22:52:47.477 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:52:47.477 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.477 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.477 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:52:47.477 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.150258+0000", 2024-09-20T22:52:47.477 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:52:47.477 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.477 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.477 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:52:47.477 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.150259+0000", 2024-09-20T22:52:47.477 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.99e-07 2024-09-20T22:52:47.477 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.477 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.478 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:52:47.478 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.150674+0000", 2024-09-20T22:52:47.478 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00041528000000000002 2024-09-20T22:52:47.478 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.478 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.478 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:52:47.478 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.150688+0000", 2024-09-20T22:52:47.479 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3399e-05 2024-09-20T22:52:47.479 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.479 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.479 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:52:47.479 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.150699+0000", 2024-09-20T22:52:47.479 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1022999999999999e-05 2024-09-20T22:52:47.479 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.479 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.479 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:52:47.480 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.150772+0000", 2024-09-20T22:52:47.480 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.3455999999999996e-05 2024-09-20T22:52:47.480 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.480 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.480 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:52:47.480 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.150798+0000", 2024-09-20T22:52:47.480 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.5777000000000001e-05 2024-09-20T22:52:47.480 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.480 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.480 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:52:47.480 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.164390+0000", 2024-09-20T22:52:47.480 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013591789999999999 2024-09-20T22:52:47.480 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.480 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.480 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:52:47.480 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.164439+0000", 2024-09-20T22:52:47.481 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.9202000000000001e-05 2024-09-20T22:52:47.481 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:52:47.481 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:52:47.481 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:52:47.481 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.481 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.481 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24652.0:1654 7.19 e206/139 7:9970b578:::smithi08477203-15:head v 206'114, mlcod=205'111)", 2024-09-20T22:52:47.481 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:44:45.167994+0000", 2024-09-20T22:52:47.481 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 482.29331487299999, 2024-09-20T22:52:47.481 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.01345262, 2024-09-20T22:52:47.481 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:52:47.481 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:52:47.481 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:52:47.481 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:52:47.482 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.482 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:52:47.482 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:45.167994+0000", 2024-09-20T22:52:47.482 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:52:47.482 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.482 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.482 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:52:47.482 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:45.167994+0000", 2024-09-20T22:52:47.482 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:52:47.482 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.482 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.482 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:52:47.482 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:45.167996+0000", 2024-09-20T22:52:47.482 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.5100000000000001e-06 2024-09-20T22:52:47.483 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.483 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.483 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:52:47.483 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:45.175169+0000", 2024-09-20T22:52:47.483 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0071732740000000003 2024-09-20T22:52:47.483 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.483 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.483 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:52:47.483 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:45.175208+0000", 2024-09-20T22:52:47.483 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.8597000000000003e-05 2024-09-20T22:52:47.483 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.483 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.483 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:52:47.483 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:45.175223+0000", 2024-09-20T22:52:47.483 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.4627000000000001e-05 2024-09-20T22:52:47.483 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.484 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.484 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:52:47.484 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:45.175304+0000", 2024-09-20T22:52:47.484 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.1111000000000003e-05 2024-09-20T22:52:47.484 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.484 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.484 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:52:47.484 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:45.175336+0000", 2024-09-20T22:52:47.484 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.1945e-05 2024-09-20T22:52:47.484 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.484 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.484 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:52:47.484 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:45.181392+0000", 2024-09-20T22:52:47.484 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0060566630000000003 2024-09-20T22:52:47.484 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.485 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.485 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:52:47.485 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:45.181446+0000", 2024-09-20T22:52:47.485 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.3893000000000001e-05 2024-09-20T22:52:47.485 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:52:47.485 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:52:47.485 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:52:47.485 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.485 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.485 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24652.0:1799 7.17 e229/139 7:ea720b85:::smithi08477203-26:head v 229'168, mlcod=222'164)", 2024-09-20T22:52:47.485 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:45:08.171867+0000", 2024-09-20T22:52:47.485 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 459.28944165500002, 2024-09-20T22:52:47.485 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014913454, 2024-09-20T22:52:47.485 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:52:47.485 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:52:47.486 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:52:47.486 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:52:47.486 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.486 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:52:47.486 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.171867+0000", 2024-09-20T22:52:47.486 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:52:47.486 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.486 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.486 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:52:47.486 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.171867+0000", 2024-09-20T22:52:47.486 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:52:47.486 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.486 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.487 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:52:47.487 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.171867+0000", 2024-09-20T22:52:47.487 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.3799999999999999e-07 2024-09-20T22:52:47.487 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.487 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.487 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:52:47.487 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.173016+0000", 2024-09-20T22:52:47.487 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0011485740000000001 2024-09-20T22:52:47.487 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.487 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.487 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:52:47.487 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.173024+0000", 2024-09-20T22:52:47.487 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.4780000000000008e-06 2024-09-20T22:52:47.487 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.488 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.488 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:52:47.488 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.173034+0000", 2024-09-20T22:52:47.488 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.7860000000000008e-06 2024-09-20T22:52:47.488 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.488 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.488 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:52:47.488 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.173098+0000", 2024-09-20T22:52:47.488 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.4405000000000004e-05 2024-09-20T22:52:47.488 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.488 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.488 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:52:47.488 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.173125+0000", 2024-09-20T22:52:47.488 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.6950000000000001e-05 2024-09-20T22:52:47.488 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.489 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.489 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:52:47.489 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.186711+0000", 2024-09-20T22:52:47.489 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.01358608 2024-09-20T22:52:47.489 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.489 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.489 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:52:47.489 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.186780+0000", 2024-09-20T22:52:47.489 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.8942999999999994e-05 2024-09-20T22:52:47.489 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:52:47.489 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:52:47.489 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:52:47.489 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.489 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.489 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24652.0:2318 7.5 e258/137 7:a160b5d4:::smithi08477203-16:head v 258'43, mlcod=239'39)", 2024-09-20T22:52:47.490 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:45:37.192419+0000", 2024-09-20T22:52:47.490 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 430.26888956800002, 2024-09-20T22:52:47.490 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013974723, 2024-09-20T22:52:47.490 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:52:47.490 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:52:47.490 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:52:47.490 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:52:47.490 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.490 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:52:47.490 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.192419+0000", 2024-09-20T22:52:47.490 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:52:47.490 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.490 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.491 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:52:47.491 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.192419+0000", 2024-09-20T22:52:47.491 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:52:47.491 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.491 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.491 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:52:47.491 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.192420+0000", 2024-09-20T22:52:47.491 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.3799999999999997e-07 2024-09-20T22:52:47.491 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.491 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.491 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:52:47.491 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.204088+0000", 2024-09-20T22:52:47.491 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011668883 2024-09-20T22:52:47.491 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.491 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.492 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:52:47.492 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.204104+0000", 2024-09-20T22:52:47.492 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.5437000000000001e-05 2024-09-20T22:52:47.492 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.492 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.492 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:52:47.492 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.204112+0000", 2024-09-20T22:52:47.492 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.2519999999999999e-06 2024-09-20T22:52:47.492 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.492 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.492 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:52:47.492 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.204168+0000", 2024-09-20T22:52:47.492 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.5977999999999998e-05 2024-09-20T22:52:47.492 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.492 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.493 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:52:47.493 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.204188+0000", 2024-09-20T22:52:47.493 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.9400999999999999e-05 2024-09-20T22:52:47.493 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.493 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.493 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:52:47.493 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.206340+0000", 2024-09-20T22:52:47.493 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0021527159999999998 2024-09-20T22:52:47.493 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.493 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.493 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:52:47.493 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.206394+0000", 2024-09-20T22:52:47.493 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.3418000000000003e-05 2024-09-20T22:52:47.493 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:52:47.493 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:52:47.494 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:52:47.494 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.494 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.494 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(osd.6.0:726 7.19 e260/139 7:9eac4560:::smithi08477203-30:4c v 260'159, mlcod=259'156)", 2024-09-20T22:52:47.494 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:45:39.197025+0000", 2024-09-20T22:52:47.494 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 428.26428365800001, 2024-09-20T22:52:47.494 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.018150454, 2024-09-20T22:52:47.494 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:52:47.494 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:52:47.494 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:52:47.494 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:52:47.494 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.494 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:52:47.494 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.197025+0000", 2024-09-20T22:52:47.495 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:52:47.495 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.495 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.495 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:52:47.495 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.197025+0000", 2024-09-20T22:52:47.495 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:52:47.495 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.495 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.495 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:52:47.495 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.197025+0000", 2024-09-20T22:52:47.496 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.7599999999999998e-07 2024-09-20T22:52:47.496 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.497 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.497 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:52:47.497 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.197033+0000", 2024-09-20T22:52:47.497 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.9119999999999998e-06 2024-09-20T22:52:47.497 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.497 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.497 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:52:47.497 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.197042+0000", 2024-09-20T22:52:47.497 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.3349999999999996e-06 2024-09-20T22:52:47.497 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.497 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.497 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:52:47.497 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.197050+0000", 2024-09-20T22:52:47.497 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.7230000000000004e-06 2024-09-20T22:52:47.498 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.498 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.498 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:52:47.498 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.200756+0000", 2024-09-20T22:52:47.498 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0037059749999999998 2024-09-20T22:52:47.498 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.498 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.498 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:52:47.498 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.200778+0000", 2024-09-20T22:52:47.498 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.2140000000000001e-05 2024-09-20T22:52:47.498 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.498 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.499 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:52:47.499 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.215131+0000", 2024-09-20T22:52:47.499 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014353054000000001 2024-09-20T22:52:47.499 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.499 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.499 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:52:47.499 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.215175+0000", 2024-09-20T22:52:47.499 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.4039000000000003e-05 2024-09-20T22:52:47.499 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:52:47.499 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:52:47.499 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:52:47.499 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.499 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.499 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(osd.7.0:152 7.5 e269/260 7:a160b5d4:::smithi08477203-16:55 v 269'55, mlcod=263'51)", 2024-09-20T22:52:47.500 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:45:48.173893+0000", 2024-09-20T22:52:47.500 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 419.28741591699998, 2024-09-20T22:52:47.500 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014936935, 2024-09-20T22:52:47.500 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:52:47.500 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:52:47.500 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:52:47.500 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:52:47.500 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.500 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:52:47.500 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.173893+0000", 2024-09-20T22:52:47.500 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:52:47.500 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.500 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.501 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:52:47.501 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.173893+0000", 2024-09-20T22:52:47.501 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:52:47.501 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.501 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.501 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:52:47.501 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.173893+0000", 2024-09-20T22:52:47.501 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.8800000000000002e-07 2024-09-20T22:52:47.501 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.501 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.501 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:52:47.501 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.173916+0000", 2024-09-20T22:52:47.501 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.2889999999999999e-05 2024-09-20T22:52:47.501 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.502 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.502 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:52:47.502 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.173924+0000", 2024-09-20T22:52:47.502 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.4420000000000001e-06 2024-09-20T22:52:47.502 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.502 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.502 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:52:47.502 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.173932+0000", 2024-09-20T22:52:47.502 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.5299999999999996e-06 2024-09-20T22:52:47.502 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.502 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.502 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:52:47.502 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.174011+0000", 2024-09-20T22:52:47.502 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.9342e-05 2024-09-20T22:52:47.503 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.503 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.503 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:52:47.503 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.174047+0000", 2024-09-20T22:52:47.503 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.5237999999999997e-05 2024-09-20T22:52:47.503 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.503 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.503 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:52:47.503 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.188782+0000", 2024-09-20T22:52:47.503 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014735599 2024-09-20T22:52:47.503 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.503 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.503 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:52:47.503 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.188830+0000", 2024-09-20T22:52:47.503 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.7206000000000002e-05 2024-09-20T22:52:47.504 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:52:47.504 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:52:47.504 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:52:47.504 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.504 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.504 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(osd.6.0:767 7.12 e269/263 7:4fa4b2b3:::smithi08477203-21:53 v 269'262, mlcod=265'259)", 2024-09-20T22:52:47.504 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:45:48.175130+0000", 2024-09-20T22:52:47.504 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 419.28617824700001, 2024-09-20T22:52:47.504 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013842727000000001, 2024-09-20T22:52:47.504 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:52:47.504 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:52:47.504 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:52:47.504 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:52:47.505 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.505 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:52:47.505 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.175130+0000", 2024-09-20T22:52:47.505 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:52:47.505 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.505 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.505 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:52:47.505 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.175130+0000", 2024-09-20T22:52:47.505 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:52:47.505 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.505 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.505 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:52:47.505 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.175130+0000", 2024-09-20T22:52:47.505 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.98e-07 2024-09-20T22:52:47.505 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.506 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.506 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:52:47.506 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.175139+0000", 2024-09-20T22:52:47.506 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.5909999999999996e-06 2024-09-20T22:52:47.506 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.506 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.506 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:52:47.506 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.175144+0000", 2024-09-20T22:52:47.506 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.361e-06 2024-09-20T22:52:47.506 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.506 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.506 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:52:47.506 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.175156+0000", 2024-09-20T22:52:47.506 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1236999999999999e-05 2024-09-20T22:52:47.506 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.506 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.507 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:52:47.507 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.176359+0000", 2024-09-20T22:52:47.507 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0012036060000000001 2024-09-20T22:52:47.507 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.507 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.507 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:52:47.507 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.176380+0000", 2024-09-20T22:52:47.507 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.0497999999999999e-05 2024-09-20T22:52:47.507 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.507 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.507 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:52:47.507 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.188936+0000", 2024-09-20T22:52:47.507 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.012556286999999999 2024-09-20T22:52:47.507 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.508 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.508 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:52:47.508 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.188973+0000", 2024-09-20T22:52:47.508 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.6949000000000001e-05 2024-09-20T22:52:47.508 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:52:47.508 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:52:47.508 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:52:47.508 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.508 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.508 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24652.0:2853 7.10 e303/139 7:0f524219:::smithi08477203-50:head v 303'85, mlcod=301'82)", 2024-09-20T22:52:47.508 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:46:23.293837+0000", 2024-09-20T22:52:47.508 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 384.16747153300003, 2024-09-20T22:52:47.508 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013368555000000001, 2024-09-20T22:52:47.508 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:52:47.509 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:52:47.509 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:52:47.509 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:52:47.509 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.509 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:52:47.509 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.293837+0000", 2024-09-20T22:52:47.509 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:52:47.509 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.509 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.509 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:52:47.509 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.293837+0000", 2024-09-20T22:52:47.509 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:52:47.509 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.509 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.509 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:52:47.510 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.293839+0000", 2024-09-20T22:52:47.510 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.7239999999999999e-06 2024-09-20T22:52:47.510 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.510 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.510 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:52:47.510 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.297226+0000", 2024-09-20T22:52:47.510 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.003387583 2024-09-20T22:52:47.510 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.510 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.510 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:52:47.510 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.297253+0000", 2024-09-20T22:52:47.510 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.6849e-05 2024-09-20T22:52:47.510 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.510 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.510 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:52:47.511 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.297266+0000", 2024-09-20T22:52:47.511 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2481e-05 2024-09-20T22:52:47.511 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.511 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.511 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:52:47.511 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.297362+0000", 2024-09-20T22:52:47.511 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.6347000000000004e-05 2024-09-20T22:52:47.511 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.511 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.511 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:52:47.511 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.297394+0000", 2024-09-20T22:52:47.511 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.1888999999999999e-05 2024-09-20T22:52:47.511 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.511 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.511 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:52:47.511 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.307147+0000", 2024-09-20T22:52:47.512 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0097533719999999997 2024-09-20T22:52:47.512 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.512 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.512 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:52:47.512 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.307206+0000", 2024-09-20T22:52:47.512 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.8310000000000002e-05 2024-09-20T22:52:47.512 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:52:47.512 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:52:47.512 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:52:47.512 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.512 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.512 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:3030 7.e 7:71b5c915:::smithi08477203-6:head [copy-from ver 216] snapc 72=[72,71,70,6e,6c,6b,6a,68] ondisk+write+known_if_redirected+supports_pool_eio e317)", 2024-09-20T22:52:47.512 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:46:37.293029+0000", 2024-09-20T22:52:47.512 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 370.16827917500001, 2024-09-20T22:52:47.512 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.019929916999999998, 2024-09-20T22:52:47.513 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:52:47.513 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:52:47.513 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:52:47.513 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:52:47.513 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:52:47.513 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:52:47.513 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 3030 2024-09-20T22:52:47.513 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.513 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:52:47.513 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.513 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:52:47.513 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.293029+0000", 2024-09-20T22:52:47.513 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:52:47.513 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.514 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.514 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:52:47.514 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.293029+0000", 2024-09-20T22:52:47.514 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:52:47.514 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.514 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.514 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:52:47.514 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.293031+0000", 2024-09-20T22:52:47.514 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.4190000000000001e-06 2024-09-20T22:52:47.514 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.514 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.514 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:52:47.514 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.293040+0000", 2024-09-20T22:52:47.514 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.6039999999999999e-06 2024-09-20T22:52:47.515 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.515 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.515 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:52:47.515 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.293054+0000", 2024-09-20T22:52:47.515 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.4014e-05 2024-09-20T22:52:47.515 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.515 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.515 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:52:47.515 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.293062+0000", 2024-09-20T22:52:47.515 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.8820000000000001e-06 2024-09-20T22:52:47.515 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.515 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.515 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:52:47.515 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.293121+0000", 2024-09-20T22:52:47.515 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.8977999999999997e-05 2024-09-20T22:52:47.516 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.516 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.516 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op must wait for map", 2024-09-20T22:52:47.516 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.293125+0000", 2024-09-20T22:52:47.516 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.2089999999999999e-06 2024-09-20T22:52:47.516 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.516 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.516 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:52:47.516 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.298805+0000", 2024-09-20T22:52:47.516 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0056790870000000002 2024-09-20T22:52:47.516 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.516 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.516 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:52:47.516 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.298886+0000", 2024-09-20T22:52:47.516 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.1363e-05 2024-09-20T22:52:47.517 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.517 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.517 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 3,5", 2024-09-20T22:52:47.517 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.305883+0000", 2024-09-20T22:52:47.517 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0069966159999999998 2024-09-20T22:52:47.517 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.517 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.517 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:52:47.517 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.312556+0000", 2024-09-20T22:52:47.517 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0066737009999999998 2024-09-20T22:52:47.517 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.517 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.517 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:52:47.517 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.312587+0000", 2024-09-20T22:52:47.518 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.0286000000000001e-05 2024-09-20T22:52:47.518 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.518 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.518 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:52:47.518 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.312890+0000", 2024-09-20T22:52:47.518 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00030374800000000001 2024-09-20T22:52:47.518 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.518 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.518 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:52:47.518 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.312932+0000", 2024-09-20T22:52:47.518 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.1621e-05 2024-09-20T22:52:47.518 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.518 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.518 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:52:47.518 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.312959+0000", 2024-09-20T22:52:47.519 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.7389000000000002e-05 2024-09-20T22:52:47.519 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:52:47.519 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:52:47.519 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:52:47.519 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.519 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.519 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:3031 7.e 7:71b5c915:::smithi08477203-6:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e317)", 2024-09-20T22:52:47.519 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:46:37.293082+0000", 2024-09-20T22:52:47.519 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 370.16822679699999, 2024-09-20T22:52:47.519 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.016134757, 2024-09-20T22:52:47.519 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:52:47.519 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:52:47.519 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-20T22:52:47.519 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:52:47.520 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:52:47.520 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:52:47.520 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 3031 2024-09-20T22:52:47.520 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.520 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:52:47.520 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.520 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:52:47.520 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.293082+0000", 2024-09-20T22:52:47.520 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:52:47.520 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.520 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.520 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:52:47.520 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.293082+0000", 2024-09-20T22:52:47.520 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:52:47.520 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.521 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.521 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:52:47.521 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.293082+0000", 2024-09-20T22:52:47.521 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.1799999999999995e-07 2024-09-20T22:52:47.521 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.521 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.521 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:52:47.521 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.293085+0000", 2024-09-20T22:52:47.521 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.937e-06 2024-09-20T22:52:47.521 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.521 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.521 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:52:47.521 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.293093+0000", 2024-09-20T22:52:47.521 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.554e-06 2024-09-20T22:52:47.522 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.522 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.522 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:52:47.522 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.293098+0000", 2024-09-20T22:52:47.522 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.643e-06 2024-09-20T22:52:47.522 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.522 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.522 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:52:47.522 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.293164+0000", 2024-09-20T22:52:47.522 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.5806e-05 2024-09-20T22:52:47.522 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.522 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.522 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting_for_map not empty", 2024-09-20T22:52:47.523 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.293168+0000", 2024-09-20T22:52:47.523 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.2459999999999999e-06 2024-09-20T22:52:47.523 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.523 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.523 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:52:47.523 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.299033+0000", 2024-09-20T22:52:47.523 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0058656699999999999 2024-09-20T22:52:47.523 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.523 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.523 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for blocked object", 2024-09-20T22:52:47.523 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.299069+0000", 2024-09-20T22:52:47.523 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.5392999999999998e-05 2024-09-20T22:52:47.523 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.523 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.523 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:52:47.524 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.309088+0000", 2024-09-20T22:52:47.524 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010019287999999999 2024-09-20T22:52:47.524 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.524 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.524 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:52:47.524 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.309150+0000", 2024-09-20T22:52:47.524 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.2008000000000007e-05 2024-09-20T22:52:47.524 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.524 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.524 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:52:47.524 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.309216+0000", 2024-09-20T22:52:47.524 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.6594000000000006e-05 2024-09-20T22:52:47.524 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:52:47.524 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:52:47.525 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:52:47.525 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.525 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.525 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:3290 7.8 7:12e9c8db:::smithi08477203-11:head [copy-from ver 675] snapc 7f=[7f,7e,7d,7c,7b,7a,79,78,74,73,6b] ondisk+write+known_if_redirected+supports_pool_eio e340)", 2024-09-20T22:52:47.525 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:47:00.706846+0000", 2024-09-20T22:52:47.525 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 346.75446293800002, 2024-09-20T22:52:47.525 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.018408746, 2024-09-20T22:52:47.525 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:52:47.525 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:52:47.525 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:52:47.525 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:52:47.525 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:52:47.525 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:52:47.525 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 3290 2024-09-20T22:52:47.526 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.526 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:52:47.526 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.526 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:52:47.526 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.706846+0000", 2024-09-20T22:52:47.526 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:52:47.526 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.526 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.526 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:52:47.526 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.706846+0000", 2024-09-20T22:52:47.526 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:52:47.526 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.526 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.526 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:52:47.527 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.706848+0000", 2024-09-20T22:52:47.527 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.2340000000000001e-06 2024-09-20T22:52:47.527 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.527 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.527 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:52:47.527 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.706859+0000", 2024-09-20T22:52:47.527 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1001e-05 2024-09-20T22:52:47.527 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.527 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.527 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:52:47.527 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.706879+0000", 2024-09-20T22:52:47.527 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.0556000000000001e-05 2024-09-20T22:52:47.527 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.527 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.527 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:52:47.528 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.706890+0000", 2024-09-20T22:52:47.528 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0328e-05 2024-09-20T22:52:47.528 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.528 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.528 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:52:47.528 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.706973+0000", 2024-09-20T22:52:47.528 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.3574999999999999e-05 2024-09-20T22:52:47.528 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.528 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.528 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:52:47.528 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.707042+0000", 2024-09-20T22:52:47.528 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.8987999999999997e-05 2024-09-20T22:52:47.528 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.528 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.529 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 3,7", 2024-09-20T22:52:47.529 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.717491+0000", 2024-09-20T22:52:47.529 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010448618 2024-09-20T22:52:47.529 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.529 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.529 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:52:47.529 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.722875+0000", 2024-09-20T22:52:47.529 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0053846190000000002 2024-09-20T22:52:47.529 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.529 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.529 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:52:47.529 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.724841+0000", 2024-09-20T22:52:47.529 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0019651579999999998 2024-09-20T22:52:47.529 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.530 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.530 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:52:47.530 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.725168+0000", 2024-09-20T22:52:47.530 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00032734499999999999 2024-09-20T22:52:47.530 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.530 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.530 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:52:47.530 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.725218+0000", 2024-09-20T22:52:47.530 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.0123000000000001e-05 2024-09-20T22:52:47.530 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.530 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.530 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:52:47.530 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.725254+0000", 2024-09-20T22:52:47.530 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.6201000000000001e-05 2024-09-20T22:52:47.531 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:52:47.531 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:52:47.531 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:52:47.531 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.531 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.531 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:3291 7.8 7:12e9c8db:::smithi08477203-11:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e340)", 2024-09-20T22:52:47.531 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:47:00.706927+0000", 2024-09-20T22:52:47.531 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 346.75438129899999, 2024-09-20T22:52:47.531 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014159255000000001, 2024-09-20T22:52:47.531 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:52:47.531 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:52:47.531 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-20T22:52:47.531 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:52:47.531 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:52:47.531 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:52:47.532 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 3291 2024-09-20T22:52:47.532 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.532 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:52:47.532 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.532 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:52:47.532 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.706927+0000", 2024-09-20T22:52:47.532 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:52:47.532 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.532 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.532 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:52:47.532 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.706927+0000", 2024-09-20T22:52:47.532 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:52:47.532 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.532 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.532 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:52:47.533 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.706928+0000", 2024-09-20T22:52:47.533 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.4099999999999999e-07 2024-09-20T22:52:47.533 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.533 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.533 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:52:47.533 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.706941+0000", 2024-09-20T22:52:47.533 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3335e-05 2024-09-20T22:52:47.533 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.533 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.533 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:52:47.533 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.706948+0000", 2024-09-20T22:52:47.533 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.3350000000000001e-06 2024-09-20T22:52:47.533 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.533 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.534 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:52:47.534 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.706954+0000", 2024-09-20T22:52:47.534 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.9190000000000004e-06 2024-09-20T22:52:47.534 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.534 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.534 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:52:47.534 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.707166+0000", 2024-09-20T22:52:47.534 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00021159 2024-09-20T22:52:47.535 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.535 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.535 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for blocked object", 2024-09-20T22:52:47.535 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.707217+0000", 2024-09-20T22:52:47.535 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.1167e-05 2024-09-20T22:52:47.535 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.535 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.536 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:52:47.536 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.720939+0000", 2024-09-20T22:52:47.536 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013721458000000001 2024-09-20T22:52:47.536 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.536 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.536 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:52:47.536 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.721010+0000", 2024-09-20T22:52:47.536 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.1570999999999996e-05 2024-09-20T22:52:47.536 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.536 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.536 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:52:47.536 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.721086+0000", 2024-09-20T22:52:47.536 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.6339000000000002e-05 2024-09-20T22:52:47.536 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:52:47.537 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:52:47.537 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:52:47.537 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.537 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.537 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(osd.5.0:167 7.2 e347/169 7:45b54b6b:::smithi08477203-2:81 v 347'140, mlcod=345'136)", 2024-09-20T22:52:47.537 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:47:08.204691+0000", 2024-09-20T22:52:47.537 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 339.25661719800001, 2024-09-20T22:52:47.537 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.01335918, 2024-09-20T22:52:47.537 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:52:47.537 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:52:47.537 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:52:47.537 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:52:47.537 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.537 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:52:47.538 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:08.204691+0000", 2024-09-20T22:52:47.538 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:52:47.538 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.538 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.538 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:52:47.538 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:08.204691+0000", 2024-09-20T22:52:47.538 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:52:47.538 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.538 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.538 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:52:47.538 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:08.204692+0000", 2024-09-20T22:52:47.538 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.5400000000000001e-07 2024-09-20T22:52:47.538 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.539 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.539 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:52:47.539 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:08.204697+0000", 2024-09-20T22:52:47.539 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.614e-06 2024-09-20T22:52:47.539 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.539 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.539 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:52:47.539 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:08.204705+0000", 2024-09-20T22:52:47.539 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.8259999999999993e-06 2024-09-20T22:52:47.539 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.539 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.539 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:52:47.539 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:08.204715+0000", 2024-09-20T22:52:47.539 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.8279999999999998e-06 2024-09-20T22:52:47.539 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.540 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.540 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:52:47.540 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:08.204801+0000", 2024-09-20T22:52:47.540 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.6263999999999999e-05 2024-09-20T22:52:47.540 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.540 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.540 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:52:47.540 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:08.204823+0000", 2024-09-20T22:52:47.540 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.1971999999999999e-05 2024-09-20T22:52:47.540 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.540 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.540 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:52:47.540 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:08.217961+0000", 2024-09-20T22:52:47.540 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013137618 2024-09-20T22:52:47.541 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.541 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.541 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:52:47.541 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:08.218050+0000", 2024-09-20T22:52:47.541 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.9504000000000002e-05 2024-09-20T22:52:47.541 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:52:47.541 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:52:47.541 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:52:47.541 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.541 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.541 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24652.0:4534 7.12 e456/421 7:4b06cbed:::smithi08477203-49:head v 456'446, mlcod=454'443)", 2024-09-20T22:52:47.541 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:48:56.245037+0000", 2024-09-20T22:52:47.541 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 231.21627130499999, 2024-09-20T22:52:47.541 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013221933999999999, 2024-09-20T22:52:47.542 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:52:47.542 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:52:47.542 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:52:47.542 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:52:47.542 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.542 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:52:47.542 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:56.245037+0000", 2024-09-20T22:52:47.542 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:52:47.542 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.542 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.542 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:52:47.542 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:56.245037+0000", 2024-09-20T22:52:47.542 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:52:47.542 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.542 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.543 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:52:47.543 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:56.245037+0000", 2024-09-20T22:52:47.543 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.4600000000000001e-07 2024-09-20T22:52:47.543 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.543 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.543 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:52:47.543 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:56.248142+0000", 2024-09-20T22:52:47.543 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0031044839999999998 2024-09-20T22:52:47.543 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.543 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.543 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:52:47.543 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:56.248174+0000", 2024-09-20T22:52:47.543 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.2555000000000003e-05 2024-09-20T22:52:47.543 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.543 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.543 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:52:47.544 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:56.248188+0000", 2024-09-20T22:52:47.544 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.308e-05 2024-09-20T22:52:47.544 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.544 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.544 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:52:47.544 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:56.248292+0000", 2024-09-20T22:52:47.544 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000104182 2024-09-20T22:52:47.544 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.544 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.544 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:52:47.544 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:56.248318+0000", 2024-09-20T22:52:47.544 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.652e-05 2024-09-20T22:52:47.544 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.544 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.544 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:52:47.544 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:56.258207+0000", 2024-09-20T22:52:47.544 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0098883909999999998 2024-09-20T22:52:47.545 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.545 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.545 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:52:47.545 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:56.258259+0000", 2024-09-20T22:52:47.545 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.2475999999999997e-05 2024-09-20T22:52:47.545 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:52:47.545 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:52:47.545 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:52:47.545 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.545 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.545 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:6360 7.8 7:1e9e2524:::smithi08477203-5:head [write 655704~481370,stat] snapc 119=[119,118,117,116,115,114,113,112,111,110,10e,10c,107,105,104,103,fd,fb,fa,f8,f5,f2,ed,d7,c2,bf] ondisk+write+known_if_redirected+supports_pool_eio e638)", 2024-09-20T22:52:47.545 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:51:58.286036+0000", 2024-09-20T22:52:47.545 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 49.175272616999997, 2024-09-20T22:52:47.545 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013234065999999999, 2024-09-20T22:52:47.545 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:52:47.546 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:52:47.546 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:52:47.546 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:52:47.546 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:52:47.546 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:52:47.546 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 6360 2024-09-20T22:52:47.546 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.546 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:52:47.546 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.546 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:52:47.546 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:58.286036+0000", 2024-09-20T22:52:47.546 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:52:47.546 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.546 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.546 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:52:47.546 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:58.286036+0000", 2024-09-20T22:52:47.547 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:52:47.547 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.547 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.547 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:52:47.547 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:58.286038+0000", 2024-09-20T22:52:47.547 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.3209999999999999e-06 2024-09-20T22:52:47.547 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.547 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.547 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:52:47.547 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:58.286324+0000", 2024-09-20T22:52:47.547 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00028558499999999998 2024-09-20T22:52:47.547 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.547 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.547 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:52:47.547 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:58.286342+0000", 2024-09-20T22:52:47.548 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.8084000000000002e-05 2024-09-20T22:52:47.548 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.548 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.548 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:52:47.548 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:58.286351+0000", 2024-09-20T22:52:47.548 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.1560000000000001e-06 2024-09-20T22:52:47.548 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.548 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.548 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:52:47.548 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:58.286407+0000", 2024-09-20T22:52:47.548 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.6183000000000001e-05 2024-09-20T22:52:47.548 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.548 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.548 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:52:47.549 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:58.286475+0000", 2024-09-20T22:52:47.549 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.8015000000000005e-05 2024-09-20T22:52:47.549 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.549 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.549 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 4,7", 2024-09-20T22:52:47.549 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:58.286599+0000", 2024-09-20T22:52:47.549 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000123931 2024-09-20T22:52:47.549 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.549 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.549 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:52:47.549 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:58.287919+0000", 2024-09-20T22:52:47.549 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0013193899999999999 2024-09-20T22:52:47.549 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.549 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.549 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:52:47.550 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:58.299104+0000", 2024-09-20T22:52:47.550 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.01118513 2024-09-20T22:52:47.550 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.550 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.550 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:52:47.550 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:58.299180+0000", 2024-09-20T22:52:47.550 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.5819e-05 2024-09-20T22:52:47.550 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.550 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.550 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:52:47.550 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:58.299219+0000", 2024-09-20T22:52:47.550 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.9359e-05 2024-09-20T22:52:47.550 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.550 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.551 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:52:47.551 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:58.299270+0000", 2024-09-20T22:52:47.551 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.1093e-05 2024-09-20T22:52:47.551 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:52:47.551 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:52:47.551 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:52:47.551 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.551 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.551 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24652.0:6378 7.2 e639/600 7:4d20232a:::smithi08477203-34:head v 639'665, mlcod=629'662)", 2024-09-20T22:52:47.551 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:51:59.278714+0000", 2024-09-20T22:52:47.551 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 48.182594643000002, 2024-09-20T22:52:47.551 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.015218334, 2024-09-20T22:52:47.551 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:52:47.551 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:52:47.552 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:52:47.552 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:52:47.552 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.552 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:52:47.552 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:59.278714+0000", 2024-09-20T22:52:47.552 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:52:47.552 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.552 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.552 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:52:47.552 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:59.278714+0000", 2024-09-20T22:52:47.552 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:52:47.552 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.552 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.552 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:52:47.553 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:59.278714+0000", 2024-09-20T22:52:47.553 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.6e-07 2024-09-20T22:52:47.553 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.553 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.553 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:52:47.553 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:59.281091+0000", 2024-09-20T22:52:47.553 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0023769239999999999 2024-09-20T22:52:47.553 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.553 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.553 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:52:47.553 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:59.281110+0000", 2024-09-20T22:52:47.553 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.9230000000000001e-05 2024-09-20T22:52:47.553 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.553 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.554 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:52:47.554 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:59.281122+0000", 2024-09-20T22:52:47.554 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1997999999999999e-05 2024-09-20T22:52:47.554 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.554 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.554 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:52:47.554 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:59.281186+0000", 2024-09-20T22:52:47.554 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.3998999999999995e-05 2024-09-20T22:52:47.554 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.554 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.554 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:52:47.554 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:59.281209+0000", 2024-09-20T22:52:47.554 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.2654999999999999e-05 2024-09-20T22:52:47.554 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.555 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.555 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:52:47.555 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:59.293866+0000", 2024-09-20T22:52:47.555 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.012656745 2024-09-20T22:52:47.555 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.555 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.555 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:52:47.555 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:59.293932+0000", 2024-09-20T22:52:47.555 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.6523e-05 2024-09-20T22:52:47.555 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:52:47.555 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:52:47.555 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:52:47.555 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.555 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.556 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(osd.5.0:520 7.b e643/602 7:dc95255e:::smithi08477203-1:107 v 643'959, mlcod=637'955)", 2024-09-20T22:52:47.556 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:52:03.289043+0000", 2024-09-20T22:52:47.556 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 44.172265533999997, 2024-09-20T22:52:47.556 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.015367396, 2024-09-20T22:52:47.556 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:52:47.556 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:52:47.556 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:52:47.556 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:52:47.556 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.556 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:52:47.556 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:52:03.289043+0000", 2024-09-20T22:52:47.556 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:52:47.556 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.556 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.557 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:52:47.557 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:52:03.289043+0000", 2024-09-20T22:52:47.557 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:52:47.557 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.557 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.557 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:52:47.557 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:52:03.289044+0000", 2024-09-20T22:52:47.557 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.8100000000000002e-07 2024-09-20T22:52:47.557 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.557 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.557 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:52:47.557 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:52:03.289056+0000", 2024-09-20T22:52:47.557 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2408999999999999e-05 2024-09-20T22:52:47.558 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.558 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.558 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:52:47.558 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:52:03.289071+0000", 2024-09-20T22:52:47.558 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.4331000000000001e-05 2024-09-20T22:52:47.558 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.558 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.558 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:52:47.558 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:52:03.289079+0000", 2024-09-20T22:52:47.558 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.1009999999999992e-06 2024-09-20T22:52:47.558 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.558 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.558 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:52:47.558 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:52:03.289168+0000", 2024-09-20T22:52:47.559 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.9614e-05 2024-09-20T22:52:47.559 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.559 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.559 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:52:47.559 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:52:03.289189+0000", 2024-09-20T22:52:47.559 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.0701999999999999e-05 2024-09-20T22:52:47.559 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.559 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.559 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:52:47.559 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:52:03.304364+0000", 2024-09-20T22:52:47.559 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.015175414 2024-09-20T22:52:47.559 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.559 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.559 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:52:47.560 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:52:03.304410+0000", 2024-09-20T22:52:47.560 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.6044000000000002e-05 2024-09-20T22:52:47.560 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:52:47.560 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:52:47.560 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:52:47.560 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.560 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.560 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(osd.5.0:521 7.b e643/602 7:d960cad4:::smithi08477203-18:107 v 643'961, mlcod=637'955)", 2024-09-20T22:52:47.560 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:52:03.289097+0000", 2024-09-20T22:52:47.560 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 44.172211887000003, 2024-09-20T22:52:47.560 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.016179973, 2024-09-20T22:52:47.560 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:52:47.560 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:52:47.561 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:52:47.561 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:52:47.561 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.561 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:52:47.561 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:52:03.289097+0000", 2024-09-20T22:52:47.561 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:52:47.561 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.561 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.561 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:52:47.561 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:52:03.289097+0000", 2024-09-20T22:52:47.561 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:52:47.561 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.561 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.561 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:52:47.562 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:52:03.289097+0000", 2024-09-20T22:52:47.562 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.3599999999999999e-07 2024-09-20T22:52:47.562 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.562 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.562 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:52:47.562 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:52:03.289105+0000", 2024-09-20T22:52:47.562 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.8869999999999992e-06 2024-09-20T22:52:47.562 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.562 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.562 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:52:47.562 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:52:03.289111+0000", 2024-09-20T22:52:47.562 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.2419999999999997e-06 2024-09-20T22:52:47.562 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.562 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.563 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:52:47.563 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:52:03.289116+0000", 2024-09-20T22:52:47.563 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.6569999999999996e-06 2024-09-20T22:52:47.563 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.563 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.563 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:52:47.563 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:52:03.291170+0000", 2024-09-20T22:52:47.563 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0020537989999999998 2024-09-20T22:52:47.563 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.563 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.563 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:52:47.563 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:52:03.291192+0000", 2024-09-20T22:52:47.563 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.2548000000000002e-05 2024-09-20T22:52:47.563 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.564 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.564 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:52:47.564 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:52:03.305241+0000", 2024-09-20T22:52:47.564 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.01404832 2024-09-20T22:52:47.564 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:47.564 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:47.564 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:52:47.564 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:52:03.305277+0000", 2024-09-20T22:52:47.564 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.6084000000000002e-05 2024-09-20T22:52:47.564 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:52:47.564 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:52:47.564 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:52:47.564 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:52:47.564 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:52:47.564 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-20T22:52:47.565 INFO:tasks.rados.rados.0.smithi084.stdout:2684: expect (ObjNum 920 snap 304 seq_num 920) 2024-09-20T22:52:47.565 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:52:47.586 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:47.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:47.587 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:52:47.606 INFO:tasks.rados.rados.0.smithi084.stdout:2686: expect (ObjNum 876 snap 280 seq_num 876) 2024-09-20T22:52:47.614 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:47 smithi084 ceph-mon[29774]: 3.7 scrub starts 2024-09-20T22:52:47.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:47 smithi084 ceph-mon[29774]: 3.7 scrub ok 2024-09-20T22:52:47.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:47 smithi084 ceph-mon[29774]: 6.1c scrub starts 2024-09-20T22:52:47.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:47 smithi084 ceph-mon[29774]: 6.1c scrub ok 2024-09-20T22:52:47.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:47 smithi084 ceph-mon[29774]: pgmap v930: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 612 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail; 3.0 MiB/s rd, 1.9 MiB/s wr, 13 op/s 2024-09-20T22:52:47.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:47 smithi084 ceph-mon[29774]: purged_snaps scrub starts 2024-09-20T22:52:47.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:47 smithi084 ceph-mon[29774]: purged_snaps scrub ok 2024-09-20T22:52:47.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:47 smithi084 ceph-mon[29774]: 3.5 scrub starts 2024-09-20T22:52:47.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:47 smithi084 ceph-mon[29774]: 3.5 scrub ok 2024-09-20T22:52:47.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:47 smithi084 ceph-mon[29774]: 4.9 deep-scrub starts 2024-09-20T22:52:47.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:47 smithi084 ceph-mon[29774]: 4.9 deep-scrub ok 2024-09-20T22:52:47.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:47 smithi084 ceph-mon[29774]: osdmap e687: 8 total, 8 up, 7 in 2024-09-20T22:52:47.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:47 smithi084 ceph-mon[40024]: 3.7 scrub starts 2024-09-20T22:52:47.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:47 smithi084 ceph-mon[40024]: 3.7 scrub ok 2024-09-20T22:52:47.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:47 smithi084 ceph-mon[40024]: 6.1c scrub starts 2024-09-20T22:52:47.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:47 smithi084 ceph-mon[40024]: 6.1c scrub ok 2024-09-20T22:52:47.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:47 smithi084 ceph-mon[40024]: pgmap v930: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 612 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail; 3.0 MiB/s rd, 1.9 MiB/s wr, 13 op/s 2024-09-20T22:52:47.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:47 smithi084 ceph-mon[40024]: purged_snaps scrub starts 2024-09-20T22:52:47.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:47 smithi084 ceph-mon[40024]: purged_snaps scrub ok 2024-09-20T22:52:47.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:47 smithi084 ceph-mon[40024]: 3.5 scrub starts 2024-09-20T22:52:47.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:47 smithi084 ceph-mon[40024]: 3.5 scrub ok 2024-09-20T22:52:47.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:47 smithi084 ceph-mon[40024]: 4.9 deep-scrub starts 2024-09-20T22:52:47.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:47 smithi084 ceph-mon[40024]: 4.9 deep-scrub ok 2024-09-20T22:52:47.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:47 smithi084 ceph-mon[40024]: osdmap e687: 8 total, 8 up, 7 in 2024-09-20T22:52:47.688 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:52:47.699 INFO:tasks.rados.rados.0.smithi084.stdout:2685: finishing copy_from racing read to smithi08477203-45 2024-09-20T22:52:47.699 INFO:tasks.rados.rados.0.smithi084.stdout:2685: finishing copy_from to smithi08477203-45 2024-09-20T22:52:47.700 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 45 v 1338 (ObjNum 852 snap 275 seq_num 852) dirty exists 2024-09-20T22:52:47.700 INFO:tasks.rados.rados.0.smithi084.stdout:2679: done (7 left) 2024-09-20T22:52:47.700 INFO:tasks.rados.rados.0.smithi084.stdout:2680: done (6 left) 2024-09-20T22:52:47.700 INFO:tasks.rados.rados.0.smithi084.stdout:2681: done (5 left) 2024-09-20T22:52:47.700 INFO:tasks.rados.rados.0.smithi084.stdout:2683: done (4 left) 2024-09-20T22:52:47.700 INFO:tasks.rados.rados.0.smithi084.stdout:2684: done (3 left) 2024-09-20T22:52:47.700 INFO:tasks.rados.rados.0.smithi084.stdout:2685: done (2 left) 2024-09-20T22:52:47.700 INFO:tasks.rados.rados.0.smithi084.stdout:2686: done (1 left) 2024-09-20T22:52:47.700 INFO:tasks.rados.rados.0.smithi084.stdout:2687: done (0 left) 2024-09-20T22:52:47.700 INFO:tasks.rados.rados.0.smithi084.stdout:2688: rollback oid 4 current snap is 308 2024-09-20T22:52:47.700 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 4 to 293 2024-09-20T22:52:47.700 INFO:tasks.rados.rados.0.smithi084.stdout:2689: read oid 20 snap -1 2024-09-20T22:52:47.700 INFO:tasks.rados.rados.0.smithi084.stdout:2689: expect (ObjNum 874 snap 280 seq_num 874) 2024-09-20T22:52:47.700 INFO:tasks.rados.rados.0.smithi084.stdout:2690: rollback oid 35 current snap is 308 2024-09-20T22:52:47.701 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 35 to 266 2024-09-20T22:52:47.701 INFO:tasks.rados.rados.0.smithi084.stdout:2691: read oid 29 snap 305 2024-09-20T22:52:47.701 INFO:tasks.rados.rados.0.smithi084.stdout:2691: expect (ObjNum 924 snap 305 seq_num 924) 2024-09-20T22:52:47.701 INFO:tasks.rados.rados.0.smithi084.stdout:2692: rollback oid 7 current snap is 308 2024-09-20T22:52:47.701 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 7 to 294 2024-09-20T22:52:47.701 INFO:tasks.rados.rados.0.smithi084.stdout:2693: copy_from oid 45 from oid 39 current snap is 308 2024-09-20T22:52:47.701 INFO:tasks.rados.rados.0.smithi084.stdout:2694: read oid 14 snap 292 2024-09-20T22:52:47.701 INFO:tasks.rados.rados.0.smithi084.stdout:2694: expect deleted 2024-09-20T22:52:47.701 INFO:tasks.rados.rados.0.smithi084.stdout:2695: delete oid 36 current snap is 308 2024-09-20T22:52:47.702 INFO:tasks.rados.rados.0.smithi084.stdout:2689: expect (ObjNum 874 snap 280 seq_num 874) 2024-09-20T22:52:47.714 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:47.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:47.715 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:52:47.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:47 smithi154 ceph-mon[33264]: 3.7 scrub starts 2024-09-20T22:52:47.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:47 smithi154 ceph-mon[33264]: 3.7 scrub ok 2024-09-20T22:52:47.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:47 smithi154 ceph-mon[33264]: 6.1c scrub starts 2024-09-20T22:52:47.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:47 smithi154 ceph-mon[33264]: 6.1c scrub ok 2024-09-20T22:52:47.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:47 smithi154 ceph-mon[33264]: pgmap v930: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 612 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail; 3.0 MiB/s rd, 1.9 MiB/s wr, 13 op/s 2024-09-20T22:52:47.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:47 smithi154 ceph-mon[33264]: purged_snaps scrub starts 2024-09-20T22:52:47.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:47 smithi154 ceph-mon[33264]: purged_snaps scrub ok 2024-09-20T22:52:47.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:47 smithi154 ceph-mon[33264]: 3.5 scrub starts 2024-09-20T22:52:47.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:47 smithi154 ceph-mon[33264]: 3.5 scrub ok 2024-09-20T22:52:47.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:47 smithi154 ceph-mon[33264]: 4.9 deep-scrub starts 2024-09-20T22:52:47.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:47 smithi154 ceph-mon[33264]: 4.9 deep-scrub ok 2024-09-20T22:52:47.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:47 smithi154 ceph-mon[33264]: osdmap e687: 8 total, 8 up, 7 in 2024-09-20T22:52:47.816 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:52:47.838 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:47.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:47.839 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:52:47.910 INFO:tasks.rados.rados.0.smithi084.stdout:2692: finishing rollback tid 0 to smithi08477203-7 2024-09-20T22:52:47.910 INFO:tasks.rados.rados.0.smithi084.stdout:2692: finishing rollback tid 1 to smithi08477203-7 2024-09-20T22:52:47.910 INFO:tasks.rados.rados.0.smithi084.stdout:2690: finishing rollback tid 1 to smithi08477203-35 2024-09-20T22:52:47.910 INFO:tasks.rados.rados.0.smithi084.stdout:2690: finishing rollback tid 2 to smithi08477203-35 2024-09-20T22:52:47.910 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 35 v 778 (ObjNum 787 snap 254 seq_num 787) dirty exists 2024-09-20T22:52:47.910 INFO:tasks.rados.rados.0.smithi084.stdout:2692: finishing rollback tid 2 to smithi08477203-7 2024-09-20T22:52:47.911 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 7 v 1233 (ObjNum 883 snap 285 seq_num 883) dirty exists 2024-09-20T22:52:47.911 INFO:tasks.rados.rados.0.smithi084.stdout:2688: finishing rollback tid 0 to smithi08477203-4 2024-09-20T22:52:47.911 INFO:tasks.rados.rados.0.smithi084.stdout:2688: finishing rollback tid 1 to smithi08477203-4 2024-09-20T22:52:47.911 INFO:tasks.rados.rados.0.smithi084.stdout:2688: finishing rollback tid 2 to smithi08477203-4 2024-09-20T22:52:47.911 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 4 v 2088 (ObjNum 863 snap 276 seq_num 863) dirty exists 2024-09-20T22:52:47.911 INFO:tasks.rados.rados.0.smithi084.stdout:2693: finishing copy_from racing read to smithi08477203-45 2024-09-20T22:52:47.911 INFO:tasks.rados.rados.0.smithi084.stdout:2691: expect (ObjNum 924 snap 305 seq_num 924) 2024-09-20T22:52:47.940 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:52:47.961 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:47.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:47.963 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:52:48.030 INFO:tasks.rados.rados.0.smithi084.stdout:2693: finishing copy_from to smithi08477203-45 2024-09-20T22:52:48.030 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 45 v 1815 (ObjNum 905 snap 297 seq_num 905) dirty exists 2024-09-20T22:52:48.030 INFO:tasks.rados.rados.0.smithi084.stdout:2688: done (7 left) 2024-09-20T22:52:48.030 INFO:tasks.rados.rados.0.smithi084.stdout:2689: done (6 left) 2024-09-20T22:52:48.030 INFO:tasks.rados.rados.0.smithi084.stdout:2690: done (5 left) 2024-09-20T22:52:48.030 INFO:tasks.rados.rados.0.smithi084.stdout:2691: done (4 left) 2024-09-20T22:52:48.030 INFO:tasks.rados.rados.0.smithi084.stdout:2692: done (3 left) 2024-09-20T22:52:48.030 INFO:tasks.rados.rados.0.smithi084.stdout:2693: done (2 left) 2024-09-20T22:52:48.030 INFO:tasks.rados.rados.0.smithi084.stdout:2694: done (1 left) 2024-09-20T22:52:48.030 INFO:tasks.rados.rados.0.smithi084.stdout:2695: done (0 left) 2024-09-20T22:52:48.030 INFO:tasks.rados.rados.0.smithi084.stdout:2696: write (excl) oid 20 current snap is 308 2024-09-20T22:52:48.031 INFO:tasks.rados.rados.0.smithi084.stdout:2696: seq_num 933 ranges {480931=402383,1313935=515645,2488711=452788,3443230=237701} 2024-09-20T22:52:48.034 INFO:tasks.rados.rados.0.smithi084.stdout:2696: writing smithi08477203-20 from 480931 to 883314 tid 1 2024-09-20T22:52:48.039 INFO:tasks.rados.rados.0.smithi084.stdout:2696: writing smithi08477203-20 from 1313935 to 1829580 tid 2 2024-09-20T22:52:48.046 INFO:tasks.rados.rados.0.smithi084.stdout:2696: writing smithi08477203-20 from 2488711 to 2941499 tid 3 2024-09-20T22:52:48.050 INFO:tasks.rados.rados.0.smithi084.stdout:2696: writing smithi08477203-20 from 3443230 to 3680931 tid 4 2024-09-20T22:52:48.050 INFO:tasks.rados.rados.0.smithi084.stdout:2697: read oid 36 snap -1 2024-09-20T22:52:48.050 INFO:tasks.rados.rados.0.smithi084.stdout:2697: expect deleted 2024-09-20T22:52:48.050 INFO:tasks.rados.rados.0.smithi084.stdout:2698: write (excl) oid 3 current snap is 308 2024-09-20T22:52:48.050 INFO:tasks.rados.rados.0.smithi084.stdout:2696: finishing write tid 1 to smithi08477203-20 2024-09-20T22:52:48.051 INFO:tasks.rados.rados.0.smithi084.stdout:2698: seq_num 934 ranges {497738=477632,1772417=775656,3316134=381604} 2024-09-20T22:52:48.055 INFO:tasks.rados.rados.0.smithi084.stdout:2698: writing smithi08477203-3 from 497738 to 975370 tid 1 2024-09-20T22:52:48.064 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:52:48.064 INFO:tasks.rados.rados.0.smithi084.stdout:2698: writing smithi08477203-3 from 1772417 to 2548073 tid 2 2024-09-20T22:52:48.071 INFO:tasks.rados.rados.0.smithi084.stdout:2698: writing smithi08477203-3 from 3316134 to 3697738 tid 3 2024-09-20T22:52:48.071 INFO:tasks.rados.rados.0.smithi084.stdout:2699: copy_from oid 14 from oid 7 current snap is 308 2024-09-20T22:52:48.071 INFO:tasks.rados.rados.0.smithi084.stdout:2696: finishing write tid 2 to smithi08477203-20 2024-09-20T22:52:48.071 INFO:tasks.rados.rados.0.smithi084.stdout:2696: finishing write tid 3 to smithi08477203-20 2024-09-20T22:52:48.071 INFO:tasks.rados.rados.0.smithi084.stdout:2700: rollback oid 9 current snap is 308 2024-09-20T22:52:48.071 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 9 to 279 2024-09-20T22:52:48.072 INFO:tasks.rados.rados.0.smithi084.stdout:2696: finishing write tid 4 to smithi08477203-20 2024-09-20T22:52:48.072 INFO:tasks.rados.rados.0.smithi084.stdout:2696: finishing write tid 5 to smithi08477203-20 2024-09-20T22:52:48.072 INFO:tasks.rados.rados.0.smithi084.stdout:2696: finishing write tid 6 to smithi08477203-20 2024-09-20T22:52:48.072 INFO:tasks.rados.rados.0.smithi084.stdout:2696: oid 20 updating version 0 to 1246 2024-09-20T22:52:48.072 INFO:tasks.rados.rados.0.smithi084.stdout:2696: oid 20 updating version 1246 to 1250 2024-09-20T22:52:48.072 INFO:tasks.rados.rados.0.smithi084.stdout:2696: oid 20 version 1250 is already newer than 1247 2024-09-20T22:52:48.072 INFO:tasks.rados.rados.0.smithi084.stdout:2696: oid 20 version 1250 is already newer than 1248 2024-09-20T22:52:48.072 INFO:tasks.rados.rados.0.smithi084.stdout:2696: oid 20 version 1250 is already newer than 1249 2024-09-20T22:52:48.072 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 20 v 1250 (ObjNum 933 snap 308 seq_num 933) dirty exists 2024-09-20T22:52:48.072 INFO:tasks.rados.rados.0.smithi084.stdout:2696: left oid 20 (ObjNum 933 snap 308 seq_num 933) 2024-09-20T22:52:48.072 INFO:tasks.rados.rados.0.smithi084.stdout:2698: finishing write tid 1 to smithi08477203-3 2024-09-20T22:52:48.072 INFO:tasks.rados.rados.0.smithi084.stdout:2698: finishing write tid 2 to smithi08477203-3 2024-09-20T22:52:48.072 INFO:tasks.rados.rados.0.smithi084.stdout:2696: done (4 left) 2024-09-20T22:52:48.072 INFO:tasks.rados.rados.0.smithi084.stdout:2697: done (3 left) 2024-09-20T22:52:48.073 INFO:tasks.rados.rados.0.smithi084.stdout:2701: snap_create 2024-09-20T22:52:48.073 INFO:tasks.rados.rados.0.smithi084.stdout:2698: finishing write tid 3 to smithi08477203-3 2024-09-20T22:52:48.073 INFO:tasks.rados.rados.0.smithi084.stdout:2700: finishing rollback tid 1 to smithi08477203-9 2024-09-20T22:52:48.074 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 9 v 1802 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-09-20T22:52:48.074 INFO:tasks.rados.rados.0.smithi084.stdout:2698: finishing write tid 4 to smithi08477203-3 2024-09-20T22:52:48.074 INFO:tasks.rados.rados.0.smithi084.stdout:2698: finishing write tid 5 to smithi08477203-3 2024-09-20T22:52:48.074 INFO:tasks.rados.rados.0.smithi084.stdout:2698: oid 3 updating version 0 to 1829 2024-09-20T22:52:48.074 INFO:tasks.rados.rados.0.smithi084.stdout:2698: oid 3 updating version 1829 to 1830 2024-09-20T22:52:48.074 INFO:tasks.rados.rados.0.smithi084.stdout:2698: oid 3 version 1830 is already newer than 1828 2024-09-20T22:52:48.074 INFO:tasks.rados.rados.0.smithi084.stdout:2698: oid 3 version 1830 is already newer than 1827 2024-09-20T22:52:48.074 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 3 v 1830 (ObjNum 934 snap 308 seq_num 934) dirty exists 2024-09-20T22:52:48.074 INFO:tasks.rados.rados.0.smithi084.stdout:2698: left oid 3 (ObjNum 934 snap 308 seq_num 934) 2024-09-20T22:52:48.080 INFO:tasks.rados.rados.0.smithi084.stdout:2699: finishing copy_from racing read to smithi08477203-14 2024-09-20T22:52:48.083 INFO:tasks.rados.rados.0.smithi084.stdout:2699: finishing copy_from to smithi08477203-14 2024-09-20T22:52:48.083 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 14 v 1233 (ObjNum 883 snap 285 seq_num 883) dirty exists 2024-09-20T22:52:48.088 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:48.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:48.089 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:52:48.190 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:52:48.211 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:48.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:48.212 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:52:48.282 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.5 to 1 (10000) 2024-09-20T22:52:48.283 INFO:tasks.rados.rados.0.smithi084.stdout:2698: done (3 left) 2024-09-20T22:52:48.283 INFO:tasks.rados.rados.0.smithi084.stdout:2699: done (2 left) 2024-09-20T22:52:48.283 INFO:tasks.rados.rados.0.smithi084.stdout:2700: done (1 left) 2024-09-20T22:52:48.283 INFO:tasks.rados.rados.0.smithi084.stdout:2701: done (0 left) 2024-09-20T22:52:48.283 INFO:tasks.rados.rados.0.smithi084.stdout:2702: rollback oid 12 current snap is 309 2024-09-20T22:52:48.283 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 12 to 292 2024-09-20T22:52:48.284 INFO:tasks.rados.rados.0.smithi084.stdout:2703: copy_from oid 1 from oid 32 current snap is 309 2024-09-20T22:52:48.284 INFO:tasks.rados.rados.0.smithi084.stdout:2704: read oid 19 snap -1 2024-09-20T22:52:48.284 INFO:tasks.rados.rados.0.smithi084.stdout:2704: expect deleted 2024-09-20T22:52:48.284 INFO:tasks.rados.rados.0.smithi084.stdout:2705: rollback oid 21 current snap is 309 2024-09-20T22:52:48.284 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 21 to 235 2024-09-20T22:52:48.284 INFO:tasks.rados.rados.0.smithi084.stdout:2706: delete oid 34 current snap is 309 2024-09-20T22:52:48.294 INFO:tasks.rados.rados.0.smithi084.stdout:2702: finishing rollback tid 1 to smithi08477203-12 2024-09-20T22:52:48.295 INFO:tasks.rados.rados.0.smithi084.stdout:2702: finishing rollback tid 2 to smithi08477203-12 2024-09-20T22:52:48.295 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 12 v 1370 (ObjNum 795 snap 254 seq_num 795) dirty exists 2024-09-20T22:52:48.298 INFO:tasks.rados.rados.0.smithi084.stdout:2705: finishing rollback tid 0 to smithi08477203-21 2024-09-20T22:52:48.298 INFO:tasks.rados.rados.0.smithi084.stdout:2705: finishing rollback tid 1 to smithi08477203-21 2024-09-20T22:52:48.298 INFO:tasks.rados.rados.0.smithi084.stdout:2705: finishing rollback tid 2 to smithi08477203-21 2024-09-20T22:52:48.298 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 21 v 1819 (ObjNum 636 snap 193 seq_num 636) dirty exists 2024-09-20T22:52:48.298 INFO:tasks.rados.rados.0.smithi084.stdout:2702: done (4 left) 2024-09-20T22:52:48.298 INFO:tasks.rados.rados.0.smithi084.stdout:2704: done (3 left) 2024-09-20T22:52:48.298 INFO:tasks.rados.rados.0.smithi084.stdout:2705: done (2 left) 2024-09-20T22:52:48.298 INFO:tasks.rados.rados.0.smithi084.stdout:2706: done (1 left) 2024-09-20T22:52:48.299 INFO:tasks.rados.rados.0.smithi084.stdout:2707: rollback oid 17 current snap is 309 2024-09-20T22:52:48.299 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 17 to 298 2024-09-20T22:52:48.299 INFO:tasks.rados.rados.0.smithi084.stdout:2708: delete oid 11 current snap is 309 2024-09-20T22:52:48.302 INFO:tasks.rados.rados.0.smithi084.stdout:2708: done (2 left) 2024-09-20T22:52:48.302 INFO:tasks.rados.rados.0.smithi084.stdout:2709: write (excl) oid 16 current snap is 309 2024-09-20T22:52:48.302 INFO:tasks.rados.rados.0.smithi084.stdout:2709: seq_num 937 ranges {548159=419732,1664216=522983,2714756=614229} 2024-09-20T22:52:48.308 INFO:tasks.rados.rados.0.smithi084.stdout:2709: writing smithi08477203-16 from 548159 to 967891 tid 1 2024-09-20T22:52:48.313 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:52:48.315 INFO:tasks.rados.rados.0.smithi084.stdout:2709: writing smithi08477203-16 from 1664216 to 2187199 tid 2 2024-09-20T22:52:48.322 INFO:tasks.rados.rados.0.smithi084.stdout:2709: writing smithi08477203-16 from 2714756 to 3328985 tid 3 2024-09-20T22:52:48.323 INFO:tasks.rados.rados.0.smithi084.stdout:2707: finishing rollback tid 0 to smithi08477203-17 2024-09-20T22:52:48.323 INFO:tasks.rados.rados.0.smithi084.stdout:2703: finishing copy_from racing read to smithi08477203-1 2024-09-20T22:52:48.323 INFO:tasks.rados.rados.0.smithi084.stdout:2707: finishing rollback tid 1 to smithi08477203-17 2024-09-20T22:52:48.323 INFO:tasks.rados.rados.0.smithi084.stdout:2707: finishing rollback tid 2 to smithi08477203-17 2024-09-20T22:52:48.323 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 17 v 788 (ObjNum 815 snap 262 seq_num 815) dirty exists 2024-09-20T22:52:48.323 INFO:tasks.rados.rados.0.smithi084.stdout:2703: finishing copy_from to smithi08477203-1 2024-09-20T22:52:48.323 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 1 v 751 (ObjNum 909 snap 297 seq_num 909) dirty exists 2024-09-20T22:52:48.323 INFO:tasks.rados.rados.0.smithi084.stdout:2703: done (2 left) 2024-09-20T22:52:48.323 INFO:tasks.rados.rados.0.smithi084.stdout:2707: done (1 left) 2024-09-20T22:52:48.323 INFO:tasks.rados.rados.0.smithi084.stdout:2710: write (excl) oid 45 current snap is 309 2024-09-20T22:52:48.323 INFO:tasks.rados.rados.0.smithi084.stdout:2710: seq_num 938 ranges {564966=494981,1606345=782994,3025826=425501} 2024-09-20T22:52:48.328 INFO:tasks.rados.rados.0.smithi084.stdout:2710: writing smithi08477203-45 from 564966 to 1059947 tid 1 2024-09-20T22:52:48.336 INFO:tasks.rados.rados.0.smithi084.stdout:2710: writing smithi08477203-45 from 1606345 to 2389339 tid 2 2024-09-20T22:52:48.336 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:48.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:48.337 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:52:48.342 INFO:tasks.rados.rados.0.smithi084.stdout:2710: writing smithi08477203-45 from 3025826 to 3451327 tid 3 2024-09-20T22:52:48.343 INFO:tasks.rados.rados.0.smithi084.stdout:2709: finishing write tid 1 to smithi08477203-16 2024-09-20T22:52:48.343 INFO:tasks.rados.rados.0.smithi084.stdout:2709: finishing write tid 2 to smithi08477203-16 2024-09-20T22:52:48.343 INFO:tasks.rados.rados.0.smithi084.stdout:2709: finishing write tid 3 to smithi08477203-16 2024-09-20T22:52:48.343 INFO:tasks.rados.rados.0.smithi084.stdout:2709: finishing write tid 4 to smithi08477203-16 2024-09-20T22:52:48.343 INFO:tasks.rados.rados.0.smithi084.stdout:2709: finishing write tid 5 to smithi08477203-16 2024-09-20T22:52:48.343 INFO:tasks.rados.rados.0.smithi084.stdout:2709: oid 16 updating version 0 to 1808 2024-09-20T22:52:48.343 INFO:tasks.rados.rados.0.smithi084.stdout:2709: oid 16 updating version 1808 to 1811 2024-09-20T22:52:48.343 INFO:tasks.rados.rados.0.smithi084.stdout:2709: oid 16 version 1811 is already newer than 1809 2024-09-20T22:52:48.343 INFO:tasks.rados.rados.0.smithi084.stdout:2709: oid 16 version 1811 is already newer than 1810 2024-09-20T22:52:48.343 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 16 v 1811 (ObjNum 937 snap 309 seq_num 937) dirty exists 2024-09-20T22:52:48.343 INFO:tasks.rados.rados.0.smithi084.stdout:2709: left oid 16 (ObjNum 937 snap 309 seq_num 937) 2024-09-20T22:52:48.343 INFO:tasks.rados.rados.0.smithi084.stdout:2710: finishing write tid 1 to smithi08477203-45 2024-09-20T22:52:48.343 INFO:tasks.rados.rados.0.smithi084.stdout:2710: finishing write tid 2 to smithi08477203-45 2024-09-20T22:52:48.343 INFO:tasks.rados.rados.0.smithi084.stdout:2709: done (1 left) 2024-09-20T22:52:48.344 INFO:tasks.rados.rados.0.smithi084.stdout:2711: snap_remove snap 305 2024-09-20T22:52:48.364 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:52:48 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: ::ffff:172.21.15.154 - - [20/Sep/2024:22:52:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-20T22:52:48.438 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:52:48.459 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:48.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:48.460 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:52:48.561 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:52:48.582 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:48.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:48.583 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:52:48.605 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-20T22:52:48.683 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:52:48.705 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:48.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:48.706 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:52:48.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:48 smithi154 ceph-mon[33264]: 3.14 deep-scrub starts 2024-09-20T22:52:48.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:48 smithi154 ceph-mon[33264]: 3.14 deep-scrub ok 2024-09-20T22:52:48.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:48 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/2715945528' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-20T22:52:48.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:48 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-20T22:52:48.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:48 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-20T22:52:48.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:48 smithi154 ceph-mon[33264]: osdmap e688: 8 total, 8 up, 7 in 2024-09-20T22:52:48.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:48 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:52:48.807 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:52:48.828 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:48.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:48.829 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:52:48.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:48 smithi084 ceph-mon[29774]: 3.14 deep-scrub starts 2024-09-20T22:52:48.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:48 smithi084 ceph-mon[29774]: 3.14 deep-scrub ok 2024-09-20T22:52:48.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:48 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/2715945528' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-20T22:52:48.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:48 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-20T22:52:48.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:48 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-20T22:52:48.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:48 smithi084 ceph-mon[29774]: osdmap e688: 8 total, 8 up, 7 in 2024-09-20T22:52:48.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:48 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:52:48.866 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:48 smithi084 ceph-mon[40024]: 3.14 deep-scrub starts 2024-09-20T22:52:48.866 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:48 smithi084 ceph-mon[40024]: 3.14 deep-scrub ok 2024-09-20T22:52:48.866 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:48 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/2715945528' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-20T22:52:48.866 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:48 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-20T22:52:48.866 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:48 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-20T22:52:48.866 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:48 smithi084 ceph-mon[40024]: osdmap e688: 8 total, 8 up, 7 in 2024-09-20T22:52:48.866 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:48 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:52:48.930 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:52:48.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:48.952 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:48.952 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:52:49.053 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:52:49.075 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:49.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:49.076 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:52:49.177 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:52:49.198 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:49.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:49.199 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:52:49.298 INFO:tasks.rados.rados.0.smithi084.stdout:2711: done (1 left) 2024-09-20T22:52:49.298 INFO:tasks.rados.rados.0.smithi084.stdout:2712: copy_from oid 12 from oid 17 current snap is 309 2024-09-20T22:52:49.298 INFO:tasks.rados.rados.0.smithi084.stdout:2713: copy_from oid 50 from oid 38 current snap is 309 2024-09-20T22:52:49.298 INFO:tasks.rados.rados.0.smithi084.stdout:2714: snap_remove snap 261 2024-09-20T22:52:49.299 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:52:49.307 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-20T22:52:49.323 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:49.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:49.324 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:52:49.425 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:52:49.446 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:49.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:49.447 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:52:49.531 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:49 smithi084 ceph-mon[40024]: pgmap v933: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 612 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail 2024-09-20T22:52:49.532 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:49 smithi084 ceph-mon[40024]: 5.7 scrub starts 2024-09-20T22:52:49.532 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:49 smithi084 ceph-mon[40024]: 5.7 scrub ok 2024-09-20T22:52:49.532 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:49 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:52:49.532 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:49 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:52:49.532 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:49 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:52:49.532 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:49 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:52:49.532 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:49 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:52:49.532 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:49 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:52:49.532 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:49 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:52:49.532 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:49 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:52:49.532 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:49 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:52:49.532 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:49 smithi084 ceph-mon[40024]: 4.6 scrub starts 2024-09-20T22:52:49.532 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:49 smithi084 ceph-mon[40024]: 4.6 scrub ok 2024-09-20T22:52:49.533 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:49 smithi084 ceph-mon[40024]: osdmap e689: 8 total, 8 up, 7 in 2024-09-20T22:52:49.533 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:49 smithi084 ceph-mon[29774]: pgmap v933: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 612 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail 2024-09-20T22:52:49.533 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:49 smithi084 ceph-mon[29774]: 5.7 scrub starts 2024-09-20T22:52:49.533 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:49 smithi084 ceph-mon[29774]: 5.7 scrub ok 2024-09-20T22:52:49.533 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:49 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:52:49.533 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:49 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:52:49.533 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:49 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:52:49.533 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:49 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:52:49.533 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:49 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:52:49.534 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:49 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:52:49.534 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:49 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:52:49.534 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:49 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:52:49.534 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:49 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:52:49.534 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:49 smithi084 ceph-mon[29774]: 4.6 scrub starts 2024-09-20T22:52:49.534 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:49 smithi084 ceph-mon[29774]: 4.6 scrub ok 2024-09-20T22:52:49.534 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:49 smithi084 ceph-mon[29774]: osdmap e689: 8 total, 8 up, 7 in 2024-09-20T22:52:49.548 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:52:49.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:49.577 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:49.578 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:52:49.678 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:52:49.700 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:49.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:49.701 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:52:49.754 INFO:journalctl@ceph.iscsi.iscsi.a.smithi154.stdout:Sep 20 22:52:49 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-iscsi-iscsi-a[59730]: debug there is no tcmu-runner data available 2024-09-20T22:52:49.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:49 smithi154 ceph-mon[33264]: pgmap v933: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 612 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail 2024-09-20T22:52:49.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:49 smithi154 ceph-mon[33264]: 5.7 scrub starts 2024-09-20T22:52:49.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:49 smithi154 ceph-mon[33264]: 5.7 scrub ok 2024-09-20T22:52:49.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:49 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:52:49.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:49 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:52:49.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:49 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:52:49.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:49 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:52:49.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:49 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:52:49.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:49 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:52:49.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:49 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:52:49.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:49 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:52:49.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:49 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:52:49.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:49 smithi154 ceph-mon[33264]: 4.6 scrub starts 2024-09-20T22:52:49.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:49 smithi154 ceph-mon[33264]: 4.6 scrub ok 2024-09-20T22:52:49.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:49 smithi154 ceph-mon[33264]: osdmap e689: 8 total, 8 up, 7 in 2024-09-20T22:52:49.802 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:52:49.824 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:49.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:49.825 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:52:49.925 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:52:49.947 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:49.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:49.948 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:52:50.003 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd reweight 6 1 2024-09-20T22:52:50.016 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:52:50.048 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:52:50.051 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:52:50.070 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:50.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:50.071 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:52:50.172 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:52:50.193 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:50.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:50.195 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:52:50.288 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is set 2024-09-20T22:52:50.296 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:52:50.298 INFO:tasks.rados.rados.0.smithi084.stdout:2714: done (3 left) 2024-09-20T22:52:50.299 INFO:tasks.rados.rados.0.smithi084.stdout:2715: delete oid 11 current snap is 309 2024-09-20T22:52:50.299 INFO:tasks.rados.rados.0.smithi084.stdout:2710: finishing write tid 3 to smithi08477203-45 2024-09-20T22:52:50.299 INFO:tasks.rados.rados.0.smithi084.stdout:2710: finishing write tid 4 to smithi08477203-45 2024-09-20T22:52:50.299 INFO:tasks.rados.rados.0.smithi084.stdout:2710: finishing write tid 5 to smithi08477203-45 2024-09-20T22:52:50.299 INFO:tasks.rados.rados.0.smithi084.stdout:2710: oid 45 updating version 0 to 1816 2024-09-20T22:52:50.299 INFO:tasks.rados.rados.0.smithi084.stdout:2710: oid 45 updating version 1816 to 1817 2024-09-20T22:52:50.299 INFO:tasks.rados.rados.0.smithi084.stdout:2710: oid 45 updating version 1817 to 1819 2024-09-20T22:52:50.299 INFO:tasks.rados.rados.0.smithi084.stdout:2710: oid 45 version 1819 is already newer than 1818 2024-09-20T22:52:50.299 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 45 v 1819 (ObjNum 938 snap 309 seq_num 938) dirty exists 2024-09-20T22:52:50.299 INFO:tasks.rados.rados.0.smithi084.stdout:2710: left oid 45 (ObjNum 938 snap 309 seq_num 938) 2024-09-20T22:52:50.299 INFO:tasks.rados.rados.0.smithi084.stdout:2713: finishing copy_from racing read to smithi08477203-50 2024-09-20T22:52:50.299 INFO:tasks.rados.rados.0.smithi084.stdout:2712: finishing copy_from racing read to smithi08477203-12 2024-09-20T22:52:50.299 INFO:tasks.rados.rados.0.smithi084.stdout:2713: finishing copy_from to smithi08477203-50 2024-09-20T22:52:50.299 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 50 v 1338 (ObjNum 852 snap 275 seq_num 852) dirty exists 2024-09-20T22:52:50.300 INFO:tasks.rados.rados.0.smithi084.stdout:2712: finishing copy_from to smithi08477203-12 2024-09-20T22:52:50.300 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 12 v 788 (ObjNum 815 snap 262 seq_num 815) dirty exists 2024-09-20T22:52:50.312 INFO:tasks.rados.rados.0.smithi084.stdout:2710: done (3 left) 2024-09-20T22:52:50.312 INFO:tasks.rados.rados.0.smithi084.stdout:2712: done (2 left) 2024-09-20T22:52:50.312 INFO:tasks.rados.rados.0.smithi084.stdout:2713: done (1 left) 2024-09-20T22:52:50.312 INFO:tasks.rados.rados.0.smithi084.stdout:2715: done (0 left) 2024-09-20T22:52:50.312 INFO:tasks.rados.rados.0.smithi084.stdout:2716: snap_create 2024-09-20T22:52:50.327 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:50.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:50.327 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:52:50.428 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:52:50.449 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:50.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:50.450 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:52:50.551 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:52:50.573 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:50.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:50.574 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:52:50.614 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:50 smithi084 ceph-mon[29774]: Adjusting osd_memory_target on smithi154 to 3248M 2024-09-20T22:52:50.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:50 smithi084 ceph-mon[29774]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:52:50.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:50 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/274809845' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-20T22:52:50.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:50 smithi084 ceph-mon[29774]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:52:50.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:50 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/274809845' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T22:52:50.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:50 smithi084 ceph-mon[29774]: osdmap e690: 8 total, 8 up, 7 in 2024-09-20T22:52:50.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:50 smithi084 ceph-mon[40024]: Adjusting osd_memory_target on smithi154 to 3248M 2024-09-20T22:52:50.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:50 smithi084 ceph-mon[40024]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:52:50.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:50 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/274809845' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-20T22:52:50.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:50 smithi084 ceph-mon[40024]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:52:50.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:50 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/274809845' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T22:52:50.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:50 smithi084 ceph-mon[40024]: osdmap e690: 8 total, 8 up, 7 in 2024-09-20T22:52:50.675 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:52:50.696 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:50.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:50.697 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:52:50.731 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:52:50.731 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:52:50.738 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:52:50.738 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:52:50.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:50 smithi154 ceph-mon[33264]: Adjusting osd_memory_target on smithi154 to 3248M 2024-09-20T22:52:50.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:50 smithi154 ceph-mon[33264]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:52:50.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:50 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/274809845' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-20T22:52:50.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:50 smithi154 ceph-mon[33264]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:52:50.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:50 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/274809845' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T22:52:50.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:50 smithi154 ceph-mon[33264]: osdmap e690: 8 total, 8 up, 7 in 2024-09-20T22:52:50.761 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:52:50.762 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:52:50.788 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-20T22:52:50.788 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T22:52:50.798 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:52:50.801 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:52:50.809 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:52:50.809 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:52:50.819 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:50.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:50.820 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:52:50.835 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:52:50.835 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:52:50.852 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:52:50.852 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:52:50.869 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:52:50.869 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:52:50.921 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:52:50.942 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:50.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:50.943 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:52:51.043 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:52:51.064 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:51.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:51.065 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:52:51.166 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:52:51.187 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:51.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:51.188 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:52:51.289 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:52:51.295 INFO:tasks.rados.rados.0.smithi084.stdout:2716: done (0 left) 2024-09-20T22:52:51.295 INFO:tasks.rados.rados.0.smithi084.stdout:2717: rollback oid 6 current snap is 310 2024-09-20T22:52:51.295 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 6 to 266 2024-09-20T22:52:51.295 INFO:tasks.rados.rados.0.smithi084.stdout:2718: snap_create 2024-09-20T22:52:51.299 INFO:tasks.rados.rados.0.smithi084.stdout:2717: finishing rollback tid 0 to smithi08477203-6 2024-09-20T22:52:51.299 INFO:tasks.rados.rados.0.smithi084.stdout:2717: finishing rollback tid 1 to smithi08477203-6 2024-09-20T22:52:51.299 INFO:tasks.rados.rados.0.smithi084.stdout:2717: finishing rollback tid 2 to smithi08477203-6 2024-09-20T22:52:51.299 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 6 v 791 (ObjNum 793 snap 254 seq_num 793) dirty exists 2024-09-20T22:52:51.312 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:51.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:51.313 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:52:51.413 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:52:51.435 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:51.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:51.436 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:52:51.536 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:52:51.557 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:51.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:51.558 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:52:51.614 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:51 smithi084 ceph-mon[29774]: 6.15 scrub starts 2024-09-20T22:52:51.614 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:51 smithi084 ceph-mon[29774]: 6.15 scrub ok 2024-09-20T22:52:51.614 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:51 smithi084 ceph-mon[29774]: pgmap v936: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 651 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail; 22 MiB/s rd, 6.0 MiB/s wr, 38 op/s 2024-09-20T22:52:51.614 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:51 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/2216298140' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:52:51.614 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:51 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:52:51.614 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:51 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:52:51.614 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:51 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:52:51.614 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:51 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:52:51.614 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:51 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:52:51.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:51 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:52:51.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:51 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:52:51.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:51 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:52:51.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:51 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:52:51.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:51 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:52:51.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:51 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:52:51.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:51 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:52:51.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:51 smithi084 ceph-mon[29774]: osdmap e691: 8 total, 8 up, 7 in 2024-09-20T22:52:51.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:51 smithi084 ceph-mon[40024]: 6.15 scrub starts 2024-09-20T22:52:51.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:51 smithi084 ceph-mon[40024]: 6.15 scrub ok 2024-09-20T22:52:51.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:51 smithi084 ceph-mon[40024]: pgmap v936: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 651 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail; 22 MiB/s rd, 6.0 MiB/s wr, 38 op/s 2024-09-20T22:52:51.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:51 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/2216298140' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:52:51.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:51 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:52:51.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:51 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:52:51.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:51 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:52:51.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:51 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:52:51.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:51 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:52:51.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:51 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:52:51.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:51 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:52:51.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:51 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:52:51.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:51 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:52:51.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:51 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:52:51.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:51 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:52:51.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:51 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:52:51.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:51 smithi084 ceph-mon[40024]: osdmap e691: 8 total, 8 up, 7 in 2024-09-20T22:52:51.659 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:52:51.680 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:51.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:51.681 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:52:51.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:51 smithi154 ceph-mon[33264]: 6.15 scrub starts 2024-09-20T22:52:51.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:51 smithi154 ceph-mon[33264]: 6.15 scrub ok 2024-09-20T22:52:51.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:51 smithi154 ceph-mon[33264]: pgmap v936: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 651 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail; 22 MiB/s rd, 6.0 MiB/s wr, 38 op/s 2024-09-20T22:52:51.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:51 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/2216298140' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:52:51.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:51 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:52:51.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:51 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:52:51.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:51 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:52:51.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:51 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:52:51.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:51 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:52:51.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:51 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:52:51.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:51 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:52:51.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:51 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-20T22:52:51.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:51 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:52:51.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:51 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:52:51.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:51 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:52:51.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:51 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:52:51.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:51 smithi154 ceph-mon[33264]: osdmap e691: 8 total, 8 up, 7 in 2024-09-20T22:52:51.782 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:52:51.803 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:51.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:51.804 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:52:51.904 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:52:51.925 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:51.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:51.926 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:52:52.027 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:52:52.048 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:52.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:52.049 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:52:52.150 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:52:52.201 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-20T22:52:52.201 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:52:52.222 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:52.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:52.223 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:52:52.300 INFO:tasks.rados.rados.0.smithi084.stdout:2717: done (1 left) 2024-09-20T22:52:52.301 INFO:tasks.rados.rados.0.smithi084.stdout:2718: done (0 left) 2024-09-20T22:52:52.301 INFO:tasks.rados.rados.0.smithi084.stdout:2719: snap_remove snap 306 2024-09-20T22:52:52.324 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:52:52.346 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:52.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:52.347 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:52:52.447 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:52:52.469 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:52.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:52.470 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:52:52.570 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:52:52.592 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:52.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:52.593 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:52:52.694 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:52:52.716 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:52.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:52.716 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:52:52.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:52 smithi154 ceph-mon[33264]: Adjusting osd_memory_target on smithi084 to 3248M 2024-09-20T22:52:52.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:52 smithi154 ceph-mon[33264]: osdmap e692: 8 total, 8 up, 7 in 2024-09-20T22:52:52.817 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:52:52.829 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:52 smithi084 ceph-mon[40024]: Adjusting osd_memory_target on smithi084 to 3248M 2024-09-20T22:52:52.830 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:52 smithi084 ceph-mon[40024]: osdmap e692: 8 total, 8 up, 7 in 2024-09-20T22:52:52.830 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:52 smithi084 ceph-mon[29774]: Adjusting osd_memory_target on smithi084 to 3248M 2024-09-20T22:52:52.830 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:52 smithi084 ceph-mon[29774]: osdmap e692: 8 total, 8 up, 7 in 2024-09-20T22:52:52.839 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:52.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:52.840 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:52:52.857 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.3 dump_blocked_ops 2024-09-20T22:52:52.940 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:52:52.961 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:52.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:52.962 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:52:53.063 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:52:53.086 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:53.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:53.087 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:52:53.188 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:52:53.210 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:53.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:53.211 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:52:53.285 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.6 to 1 (10000) 2024-09-20T22:52:53.286 INFO:tasks.rados.rados.0.smithi084.stdout:2719: done (0 left) 2024-09-20T22:52:53.286 INFO:tasks.rados.rados.0.smithi084.stdout:2720: read oid 7 snap -1 2024-09-20T22:52:53.286 INFO:tasks.rados.rados.0.smithi084.stdout:2720: expect (ObjNum 883 snap 285 seq_num 883) 2024-09-20T22:52:53.286 INFO:tasks.rados.rados.0.smithi084.stdout:2721: rollback oid 29 current snap is 311 2024-09-20T22:52:53.286 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 29 to 235 2024-09-20T22:52:53.286 INFO:tasks.rados.rados.0.smithi084.stdout:2722: write oid 40 current snap is 311 2024-09-20T22:52:53.286 INFO:tasks.rados.rados.0.smithi084.stdout:2722: seq_num 940 ranges {598580=645479,1723309=786663,3080672=448045} 2024-09-20T22:52:53.293 INFO:tasks.rados.rados.0.smithi084.stdout:2722: writing smithi08477203-40 from 598580 to 1244059 tid 1 2024-09-20T22:52:53.301 INFO:tasks.rados.rados.0.smithi084.stdout:2722: writing smithi08477203-40 from 1723309 to 2509972 tid 2 2024-09-20T22:52:53.307 INFO:tasks.rados.rados.0.smithi084.stdout:2722: writing smithi08477203-40 from 3080672 to 3528717 tid 3 2024-09-20T22:52:53.307 INFO:tasks.rados.rados.0.smithi084.stdout:2723: write (excl) oid 33 current snap is 311 2024-09-20T22:52:53.307 INFO:tasks.rados.rados.0.smithi084.stdout:2723: seq_num 941 ranges {615387=720728,1781791=530321,3108095=707292} 2024-09-20T22:52:53.312 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:52:53.316 INFO:tasks.rados.rados.0.smithi084.stdout:2723: writing smithi08477203-33 from 615387 to 1336115 tid 1 2024-09-20T22:52:53.321 INFO:tasks.rados.rados.0.smithi084.stdout:2723: writing smithi08477203-33 from 1781791 to 2312112 tid 2 2024-09-20T22:52:53.326 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-20T22:52:53.330 INFO:tasks.rados.rados.0.smithi084.stdout:2723: writing smithi08477203-33 from 3108095 to 3815387 tid 3 2024-09-20T22:52:53.330 INFO:tasks.rados.rados.0.smithi084.stdout:2724: copy_from oid 42 from oid 43 current snap is 311 2024-09-20T22:52:53.330 INFO:tasks.rados.rados.0.smithi084.stdout:2720: expect (ObjNum 883 snap 285 seq_num 883) 2024-09-20T22:52:53.341 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:53.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:53.342 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:52:53.443 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:52:53.464 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:53.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:53.465 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:52:53.551 INFO:tasks.rados.rados.0.smithi084.stdout:2722: finishing write tid 1 to smithi08477203-40 2024-09-20T22:52:53.551 INFO:tasks.rados.rados.0.smithi084.stdout:2721: finishing rollback tid 0 to smithi08477203-29 2024-09-20T22:52:53.551 INFO:tasks.rados.rados.0.smithi084.stdout:2720: done (4 left) 2024-09-20T22:52:53.551 INFO:tasks.rados.rados.0.smithi084.stdout:2725: read oid 30 snap -1 2024-09-20T22:52:53.551 INFO:tasks.rados.rados.0.smithi084.stdout:2725: expect (ObjNum 876 snap 280 seq_num 876) 2024-09-20T22:52:53.551 INFO:tasks.rados.rados.0.smithi084.stdout:2726: write (excl) oid 34 current snap is 311 2024-09-20T22:52:53.552 INFO:tasks.rados.rados.0.smithi084.stdout:2721: finishing rollback tid 1 to smithi08477203-29 2024-09-20T22:52:53.552 INFO:tasks.rados.rados.0.smithi084.stdout:2726: seq_num 942 ranges {632194=795977,2123920=790332,3419165=413029} 2024-09-20T22:52:53.558 INFO:tasks.rados.rados.0.smithi084.stdout:2726: writing smithi08477203-34 from 632194 to 1428171 tid 1 2024-09-20T22:52:53.566 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:52:53.567 INFO:tasks.rados.rados.0.smithi084.stdout:2726: writing smithi08477203-34 from 2123920 to 2914252 tid 2 2024-09-20T22:52:53.573 INFO:tasks.rados.rados.0.smithi084.stdout:2726: writing smithi08477203-34 from 3419165 to 3832194 tid 3 2024-09-20T22:52:53.573 INFO:tasks.rados.rados.0.smithi084.stdout:2727: read oid 2 snap 307 2024-09-20T22:52:53.573 INFO:tasks.rados.rados.0.smithi084.stdout:2727: expect (ObjNum 918 snap 302 seq_num 918) 2024-09-20T22:52:53.573 INFO:tasks.rados.rados.0.smithi084.stdout:2728: read oid 7 snap -1 2024-09-20T22:52:53.573 INFO:tasks.rados.rados.0.smithi084.stdout:2728: expect (ObjNum 883 snap 285 seq_num 883) 2024-09-20T22:52:53.573 INFO:tasks.rados.rados.0.smithi084.stdout:2729: delete oid 39 current snap is 311 2024-09-20T22:52:53.573 INFO:tasks.rados.rados.0.smithi084.stdout:2721: finishing rollback tid 2 to smithi08477203-29 2024-09-20T22:52:53.573 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 29 v 2106 (ObjNum 440 snap 115 seq_num 440) dirty exists 2024-09-20T22:52:53.573 INFO:tasks.rados.rados.0.smithi084.stdout:2722: finishing write tid 2 to smithi08477203-40 2024-09-20T22:52:53.574 INFO:tasks.rados.rados.0.smithi084.stdout:2722: finishing write tid 3 to smithi08477203-40 2024-09-20T22:52:53.574 INFO:tasks.rados.rados.0.smithi084.stdout:2722: finishing write tid 4 to smithi08477203-40 2024-09-20T22:52:53.574 INFO:tasks.rados.rados.0.smithi084.stdout:2722: finishing write tid 5 to smithi08477203-40 2024-09-20T22:52:53.574 INFO:tasks.rados.rados.0.smithi084.stdout:2722: oid 40 updating version 0 to 1241 2024-09-20T22:52:53.574 INFO:tasks.rados.rados.0.smithi084.stdout:2722: oid 40 updating version 1241 to 1242 2024-09-20T22:52:53.574 INFO:tasks.rados.rados.0.smithi084.stdout:2722: oid 40 updating version 1242 to 1243 2024-09-20T22:52:53.574 INFO:tasks.rados.rados.0.smithi084.stdout:2722: oid 40 updating version 1243 to 1244 2024-09-20T22:52:53.574 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 40 v 1244 (ObjNum 940 snap 311 seq_num 940) dirty exists 2024-09-20T22:52:53.574 INFO:tasks.rados.rados.0.smithi084.stdout:2722: left oid 40 (ObjNum 940 snap 311 seq_num 940) 2024-09-20T22:52:53.574 INFO:tasks.rados.rados.0.smithi084.stdout:2724: finishing copy_from racing read to smithi08477203-42 2024-09-20T22:52:53.574 INFO:tasks.rados.rados.0.smithi084.stdout:2723: finishing write tid 1 to smithi08477203-33 2024-09-20T22:52:53.574 INFO:tasks.rados.rados.0.smithi084.stdout:2723: finishing write tid 2 to smithi08477203-33 2024-09-20T22:52:53.574 INFO:tasks.rados.rados.0.smithi084.stdout:2723: finishing write tid 3 to smithi08477203-33 2024-09-20T22:52:53.574 INFO:tasks.rados.rados.0.smithi084.stdout:2724: finishing copy_from to smithi08477203-42 2024-09-20T22:52:53.575 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 42 v 846 (ObjNum 625 snap 186 seq_num 625) dirty exists 2024-09-20T22:52:53.575 INFO:tasks.rados.rados.0.smithi084.stdout:2723: finishing write tid 4 to smithi08477203-33 2024-09-20T22:52:53.575 INFO:tasks.rados.rados.0.smithi084.stdout:2723: finishing write tid 5 to smithi08477203-33 2024-09-20T22:52:53.575 INFO:tasks.rados.rados.0.smithi084.stdout:2723: oid 33 updating version 0 to 1247 2024-09-20T22:52:53.575 INFO:tasks.rados.rados.0.smithi084.stdout:2723: oid 33 version 1247 is already newer than 1245 2024-09-20T22:52:53.575 INFO:tasks.rados.rados.0.smithi084.stdout:2723: oid 33 updating version 1247 to 1248 2024-09-20T22:52:53.575 INFO:tasks.rados.rados.0.smithi084.stdout:2723: oid 33 version 1248 is already newer than 1246 2024-09-20T22:52:53.575 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 33 v 1248 (ObjNum 941 snap 311 seq_num 941) dirty exists 2024-09-20T22:52:53.575 INFO:tasks.rados.rados.0.smithi084.stdout:2723: left oid 33 (ObjNum 941 snap 311 seq_num 941) 2024-09-20T22:52:53.575 INFO:tasks.rados.rados.0.smithi084.stdout:2725: expect (ObjNum 876 snap 280 seq_num 876) 2024-09-20T22:52:53.587 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:53.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:53.588 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:52:53.601 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:53 smithi084 ceph-mon[29774]: pgmap v939: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 651 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail; 22 MiB/s rd, 6.0 MiB/s wr, 38 op/s 2024-09-20T22:52:53.602 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:53 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:52:53.602 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:53 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/1979332268' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-09-20T22:52:53.602 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:53 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-09-20T22:52:53.602 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:53 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-20T22:52:53.602 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:53 smithi084 ceph-mon[29774]: osdmap e693: 8 total, 8 up, 7 in 2024-09-20T22:52:53.602 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:53 smithi084 ceph-mon[40024]: pgmap v939: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 651 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail; 22 MiB/s rd, 6.0 MiB/s wr, 38 op/s 2024-09-20T22:52:53.602 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:53 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:52:53.602 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:53 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/1979332268' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-09-20T22:52:53.602 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:53 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-09-20T22:52:53.602 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:53 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-20T22:52:53.602 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:53 smithi084 ceph-mon[40024]: osdmap e693: 8 total, 8 up, 7 in 2024-09-20T22:52:53.666 INFO:tasks.rados.rados.0.smithi084.stdout:2726: finishing write tid 1 to smithi08477203-34 2024-09-20T22:52:53.667 INFO:tasks.rados.rados.0.smithi084.stdout:2726: finishing write tid 2 to smithi08477203-34 2024-09-20T22:52:53.667 INFO:tasks.rados.rados.0.smithi084.stdout:2721: done (8 left) 2024-09-20T22:52:53.667 INFO:tasks.rados.rados.0.smithi084.stdout:2722: done (7 left) 2024-09-20T22:52:53.667 INFO:tasks.rados.rados.0.smithi084.stdout:2723: done (6 left) 2024-09-20T22:52:53.667 INFO:tasks.rados.rados.0.smithi084.stdout:2724: done (5 left) 2024-09-20T22:52:53.667 INFO:tasks.rados.rados.0.smithi084.stdout:2725: done (4 left) 2024-09-20T22:52:53.667 INFO:tasks.rados.rados.0.smithi084.stdout:2729: done (3 left) 2024-09-20T22:52:53.667 INFO:tasks.rados.rados.0.smithi084.stdout:2730: copy_from oid 10 from oid 28 current snap is 311 2024-09-20T22:52:53.667 INFO:tasks.rados.rados.0.smithi084.stdout:2728: expect (ObjNum 883 snap 285 seq_num 883) 2024-09-20T22:52:53.688 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:52:53.710 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:53.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:53.711 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:52:53.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:53 smithi154 ceph-mon[33264]: pgmap v939: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 651 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail; 22 MiB/s rd, 6.0 MiB/s wr, 38 op/s 2024-09-20T22:52:53.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:53 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:52:53.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:53 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/1979332268' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-09-20T22:52:53.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:53 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-09-20T22:52:53.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:53 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-20T22:52:53.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:53 smithi154 ceph-mon[33264]: osdmap e693: 8 total, 8 up, 7 in 2024-09-20T22:52:53.812 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:52:53.836 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:53.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:53.837 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:52:53.857 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:52:53.884 INFO:tasks.rados.rados.0.smithi084.stdout:2726: finishing write tid 3 to smithi08477203-34 2024-09-20T22:52:53.884 INFO:tasks.rados.rados.0.smithi084.stdout:2728: done (3 left) 2024-09-20T22:52:53.884 INFO:tasks.rados.rados.0.smithi084.stdout:2731: delete oid 41 current snap is 311 2024-09-20T22:52:53.884 INFO:tasks.rados.rados.0.smithi084.stdout:2726: finishing write tid 4 to smithi08477203-34 2024-09-20T22:52:53.884 INFO:tasks.rados.rados.0.smithi084.stdout:2726: finishing write tid 5 to smithi08477203-34 2024-09-20T22:52:53.884 INFO:tasks.rados.rados.0.smithi084.stdout:2726: oid 34 updating version 0 to 1823 2024-09-20T22:52:53.884 INFO:tasks.rados.rados.0.smithi084.stdout:2726: oid 34 version 1823 is already newer than 1822 2024-09-20T22:52:53.884 INFO:tasks.rados.rados.0.smithi084.stdout:2726: oid 34 updating version 1823 to 1824 2024-09-20T22:52:53.884 INFO:tasks.rados.rados.0.smithi084.stdout:2726: oid 34 version 1824 is already newer than 1821 2024-09-20T22:52:53.884 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 34 v 1824 (ObjNum 942 snap 311 seq_num 942) dirty exists 2024-09-20T22:52:53.885 INFO:tasks.rados.rados.0.smithi084.stdout:2726: left oid 34 (ObjNum 942 snap 311 seq_num 942) 2024-09-20T22:52:53.885 INFO:tasks.rados.rados.0.smithi084.stdout:2727: expect (ObjNum 918 snap 302 seq_num 918) 2024-09-20T22:52:53.938 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:52:53.959 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:53.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:53.960 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:52:54.045 INFO:tasks.rados.rados.0.smithi084.stdout:2730: finishing copy_from to smithi08477203-10 2024-09-20T22:52:54.045 INFO:tasks.rados.rados.0.smithi084.stdout:2730: got expected ENOENT (src dne) 2024-09-20T22:52:54.045 INFO:tasks.rados.rados.0.smithi084.stdout:2730: finishing copy_from racing read to smithi08477203-10 2024-09-20T22:52:54.045 INFO:tasks.rados.rados.0.smithi084.stdout:2726: done (3 left) 2024-09-20T22:52:54.045 INFO:tasks.rados.rados.0.smithi084.stdout:2727: done (2 left) 2024-09-20T22:52:54.045 INFO:tasks.rados.rados.0.smithi084.stdout:2730: done (1 left) 2024-09-20T22:52:54.046 INFO:tasks.rados.rados.0.smithi084.stdout:2731: done (0 left) 2024-09-20T22:52:54.046 INFO:tasks.rados.rados.0.smithi084.stdout:2732: write (excl) oid 36 current snap is 311 2024-09-20T22:52:54.046 INFO:tasks.rados.rados.0.smithi084.stdout:2732: seq_num 945 ranges {682615=738077,2015719=537659,3217787=420758} 2024-09-20T22:52:54.052 INFO:tasks.rados.rados.0.smithi084.stdout:2732: writing smithi08477203-36 from 682615 to 1420692 tid 1 2024-09-20T22:52:54.059 INFO:tasks.rados.rados.0.smithi084.stdout:2732: writing smithi08477203-36 from 2015719 to 2553378 tid 2 2024-09-20T22:52:54.061 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:52:54.065 INFO:tasks.rados.rados.0.smithi084.stdout:2732: writing smithi08477203-36 from 3217787 to 3638545 tid 3 2024-09-20T22:52:54.065 INFO:tasks.rados.rados.0.smithi084.stdout:2733: delete oid 30 current snap is 311 2024-09-20T22:52:54.065 INFO:tasks.rados.rados.0.smithi084.stdout:2732: finishing write tid 1 to smithi08477203-36 2024-09-20T22:52:54.065 INFO:tasks.rados.rados.0.smithi084.stdout:2732: finishing write tid 2 to smithi08477203-36 2024-09-20T22:52:54.068 INFO:tasks.rados.rados.0.smithi084.stdout:2733: done (1 left) 2024-09-20T22:52:54.068 INFO:tasks.rados.rados.0.smithi084.stdout:2734: read oid 39 snap -1 2024-09-20T22:52:54.068 INFO:tasks.rados.rados.0.smithi084.stdout:2734: expect deleted 2024-09-20T22:52:54.068 INFO:tasks.rados.rados.0.smithi084.stdout:2735: write oid 37 current snap is 311 2024-09-20T22:52:54.069 INFO:tasks.rados.rados.0.smithi084.stdout:2735: seq_num 947 ranges {716229=488575,1616330=541328,2756280=559655} 2024-09-20T22:52:54.075 INFO:tasks.rados.rados.0.smithi084.stdout:2735: writing smithi08477203-37 from 716229 to 1204804 tid 1 2024-09-20T22:52:54.080 INFO:tasks.rados.rados.0.smithi084.stdout:2735: writing smithi08477203-37 from 1616330 to 2157658 tid 2 2024-09-20T22:52:54.086 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:54.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:54.087 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:52:54.088 INFO:tasks.rados.rados.0.smithi084.stdout:2735: writing smithi08477203-37 from 2756280 to 3315935 tid 3 2024-09-20T22:52:54.088 INFO:tasks.rados.rados.0.smithi084.stdout:2736: write (excl) oid 6 current snap is 311 2024-09-20T22:52:54.088 INFO:tasks.rados.rados.0.smithi084.stdout:2732: finishing write tid 3 to smithi08477203-36 2024-09-20T22:52:54.088 INFO:tasks.rados.rados.0.smithi084.stdout:2732: finishing write tid 4 to smithi08477203-36 2024-09-20T22:52:54.088 INFO:tasks.rados.rados.0.smithi084.stdout:2732: finishing write tid 5 to smithi08477203-36 2024-09-20T22:52:54.088 INFO:tasks.rados.rados.0.smithi084.stdout:2732: oid 36 updating version 0 to 1231 2024-09-20T22:52:54.088 INFO:tasks.rados.rados.0.smithi084.stdout:2732: oid 36 version 1231 is already newer than 1230 2024-09-20T22:52:54.088 INFO:tasks.rados.rados.0.smithi084.stdout:2732: oid 36 updating version 1231 to 1233 2024-09-20T22:52:54.088 INFO:tasks.rados.rados.0.smithi084.stdout:2732: oid 36 version 1233 is already newer than 1232 2024-09-20T22:52:54.089 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 36 v 1233 (ObjNum 945 snap 311 seq_num 945) dirty exists 2024-09-20T22:52:54.089 INFO:tasks.rados.rados.0.smithi084.stdout:2732: left oid 36 (ObjNum 945 snap 311 seq_num 945) 2024-09-20T22:52:54.089 INFO:tasks.rados.rados.0.smithi084.stdout:2735: finishing write tid 1 to smithi08477203-37 2024-09-20T22:52:54.089 INFO:tasks.rados.rados.0.smithi084.stdout:2736: seq_num 948 ranges {733036=563824,2074812=684986,3183703=749333} 2024-09-20T22:52:54.094 INFO:tasks.rados.rados.0.smithi084.stdout:2736: writing smithi08477203-6 from 733036 to 1296860 tid 1 2024-09-20T22:52:54.102 INFO:tasks.rados.rados.0.smithi084.stdout:2736: writing smithi08477203-6 from 2074812 to 2759798 tid 2 2024-09-20T22:52:54.109 INFO:tasks.rados.rados.0.smithi084.stdout:2736: writing smithi08477203-6 from 3183703 to 3933036 tid 3 2024-09-20T22:52:54.109 INFO:tasks.rados.rados.0.smithi084.stdout:2732: done (3 left) 2024-09-20T22:52:54.109 INFO:tasks.rados.rados.0.smithi084.stdout:2734: done (2 left) 2024-09-20T22:52:54.109 INFO:tasks.rados.rados.0.smithi084.stdout:2737: read oid 48 snap 298 2024-09-20T22:52:54.110 INFO:tasks.rados.rados.0.smithi084.stdout:2737: expect deleted 2024-09-20T22:52:54.110 INFO:tasks.rados.rados.0.smithi084.stdout:2738: rollback oid 47 current snap is 311 2024-09-20T22:52:54.110 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 47 to 249 2024-09-20T22:52:54.110 INFO:tasks.rados.rados.0.smithi084.stdout:2735: finishing write tid 2 to smithi08477203-37 2024-09-20T22:52:54.110 INFO:tasks.rados.rados.0.smithi084.stdout:2735: finishing write tid 3 to smithi08477203-37 2024-09-20T22:52:54.110 INFO:tasks.rados.rados.0.smithi084.stdout:2735: finishing write tid 4 to smithi08477203-37 2024-09-20T22:52:54.110 INFO:tasks.rados.rados.0.smithi084.stdout:2735: finishing write tid 5 to smithi08477203-37 2024-09-20T22:52:54.110 INFO:tasks.rados.rados.0.smithi084.stdout:2735: oid 37 updating version 0 to 1236 2024-09-20T22:52:54.110 INFO:tasks.rados.rados.0.smithi084.stdout:2735: oid 37 version 1236 is already newer than 1235 2024-09-20T22:52:54.110 INFO:tasks.rados.rados.0.smithi084.stdout:2735: oid 37 updating version 1236 to 1237 2024-09-20T22:52:54.110 INFO:tasks.rados.rados.0.smithi084.stdout:2735: oid 37 version 1237 is already newer than 1234 2024-09-20T22:52:54.110 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 37 v 1237 (ObjNum 947 snap 311 seq_num 947) dirty exists 2024-09-20T22:52:54.110 INFO:tasks.rados.rados.0.smithi084.stdout:2735: left oid 37 (ObjNum 947 snap 311 seq_num 947) 2024-09-20T22:52:54.110 INFO:tasks.rados.rados.0.smithi084.stdout:2736: finishing write tid 1 to smithi08477203-6 2024-09-20T22:52:54.111 INFO:tasks.rados.rados.0.smithi084.stdout:2736: finishing write tid 2 to smithi08477203-6 2024-09-20T22:52:54.111 INFO:tasks.rados.rados.0.smithi084.stdout:2735: done (3 left) 2024-09-20T22:52:54.111 INFO:tasks.rados.rados.0.smithi084.stdout:2739: rollback oid 28 current snap is 311 2024-09-20T22:52:54.111 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 28 to 304 2024-09-20T22:52:54.111 INFO:tasks.rados.rados.0.smithi084.stdout:2740: write (excl) oid 29 current snap is 311 2024-09-20T22:52:54.111 INFO:tasks.rados.rados.0.smithi084.stdout:2740: seq_num 949 ranges {749843=639073,2133294=544997,3211126=582199} 2024-09-20T22:52:54.116 INFO:tasks.rados.rados.0.smithi084.stdout:2740: writing smithi08477203-29 from 749843 to 1388916 tid 1 2024-09-20T22:52:54.123 INFO:tasks.rados.rados.0.smithi084.stdout:2740: writing smithi08477203-29 from 2133294 to 2678291 tid 2 2024-09-20T22:52:54.130 INFO:tasks.rados.rados.0.smithi084.stdout:2740: writing smithi08477203-29 from 3211126 to 3793325 tid 3 2024-09-20T22:52:54.130 INFO:tasks.rados.rados.0.smithi084.stdout:2741: write (excl) oid 24 current snap is 311 2024-09-20T22:52:54.130 INFO:tasks.rados.rados.0.smithi084.stdout:2741: seq_num 950 ranges {766650=714322,2075423=688655,3522196=444454} 2024-09-20T22:52:54.138 INFO:tasks.rados.rados.0.smithi084.stdout:2741: writing smithi08477203-24 from 766650 to 1480972 tid 1 2024-09-20T22:52:54.145 INFO:tasks.rados.rados.0.smithi084.stdout:2741: writing smithi08477203-24 from 2075423 to 2764078 tid 2 2024-09-20T22:52:54.152 INFO:tasks.rados.rados.0.smithi084.stdout:2741: writing smithi08477203-24 from 3522196 to 3966650 tid 3 2024-09-20T22:52:54.153 INFO:tasks.rados.rados.0.smithi084.stdout:2742: delete oid 37 current snap is 311 2024-09-20T22:52:54.153 INFO:tasks.rados.rados.0.smithi084.stdout:2736: finishing write tid 3 to smithi08477203-6 2024-09-20T22:52:54.153 INFO:tasks.rados.rados.0.smithi084.stdout:2739: finishing rollback tid 1 to smithi08477203-28 2024-09-20T22:52:54.154 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 28 v 2003 (ObjNum 129 snap 0 seq_num 2091341176) dirty dne 2024-09-20T22:52:54.154 INFO:tasks.rados.rados.0.smithi084.stdout:2736: finishing write tid 4 to smithi08477203-6 2024-09-20T22:52:54.154 INFO:tasks.rados.rados.0.smithi084.stdout:2738: finishing rollback tid 0 to smithi08477203-47 2024-09-20T22:52:54.154 INFO:tasks.rados.rados.0.smithi084.stdout:2736: finishing write tid 5 to smithi08477203-6 2024-09-20T22:52:54.154 INFO:tasks.rados.rados.0.smithi084.stdout:2736: oid 6 updating version 0 to 792 2024-09-20T22:52:54.154 INFO:tasks.rados.rados.0.smithi084.stdout:2736: oid 6 updating version 792 to 794 2024-09-20T22:52:54.154 INFO:tasks.rados.rados.0.smithi084.stdout:2736: oid 6 version 794 is already newer than 793 2024-09-20T22:52:54.154 INFO:tasks.rados.rados.0.smithi084.stdout:2736: oid 6 updating version 794 to 795 2024-09-20T22:52:54.154 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 6 v 795 (ObjNum 948 snap 311 seq_num 948) dirty exists 2024-09-20T22:52:54.154 INFO:tasks.rados.rados.0.smithi084.stdout:2736: left oid 6 (ObjNum 948 snap 311 seq_num 948) 2024-09-20T22:52:54.154 INFO:tasks.rados.rados.0.smithi084.stdout:2738: finishing rollback tid 1 to smithi08477203-47 2024-09-20T22:52:54.154 INFO:tasks.rados.rados.0.smithi084.stdout:2738: finishing rollback tid 2 to smithi08477203-47 2024-09-20T22:52:54.154 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 47 v 1373 (ObjNum 671 snap 205 seq_num 671) dirty exists 2024-09-20T22:52:54.155 INFO:tasks.rados.rados.0.smithi084.stdout:2740: finishing write tid 1 to smithi08477203-29 2024-09-20T22:52:54.155 INFO:tasks.rados.rados.0.smithi084.stdout:2740: finishing write tid 2 to smithi08477203-29 2024-09-20T22:52:54.155 INFO:tasks.rados.rados.0.smithi084.stdout:2740: finishing write tid 3 to smithi08477203-29 2024-09-20T22:52:54.155 INFO:tasks.rados.rados.0.smithi084.stdout:2740: finishing write tid 4 to smithi08477203-29 2024-09-20T22:52:54.155 INFO:tasks.rados.rados.0.smithi084.stdout:2740: finishing write tid 5 to smithi08477203-29 2024-09-20T22:52:54.155 INFO:tasks.rados.rados.0.smithi084.stdout:2740: oid 29 updating version 0 to 2113 2024-09-20T22:52:54.155 INFO:tasks.rados.rados.0.smithi084.stdout:2740: oid 29 version 2113 is already newer than 2111 2024-09-20T22:52:54.155 INFO:tasks.rados.rados.0.smithi084.stdout:2740: oid 29 version 2113 is already newer than 2110 2024-09-20T22:52:54.155 INFO:tasks.rados.rados.0.smithi084.stdout:2740: oid 29 version 2113 is already newer than 2112 2024-09-20T22:52:54.155 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 29 v 2113 (ObjNum 949 snap 311 seq_num 949) dirty exists 2024-09-20T22:52:54.155 INFO:tasks.rados.rados.0.smithi084.stdout:2740: left oid 29 (ObjNum 949 snap 311 seq_num 949) 2024-09-20T22:52:54.155 INFO:tasks.rados.rados.0.smithi084.stdout:2741: finishing write tid 1 to smithi08477203-24 2024-09-20T22:52:54.155 INFO:tasks.rados.rados.0.smithi084.stdout:2741: finishing write tid 2 to smithi08477203-24 2024-09-20T22:52:54.156 INFO:tasks.rados.rados.0.smithi084.stdout:2736: done (6 left) 2024-09-20T22:52:54.156 INFO:tasks.rados.rados.0.smithi084.stdout:2737: done (5 left) 2024-09-20T22:52:54.157 INFO:tasks.rados.rados.0.smithi084.stdout:2738: done (4 left) 2024-09-20T22:52:54.157 INFO:tasks.rados.rados.0.smithi084.stdout:2739: done (3 left) 2024-09-20T22:52:54.157 INFO:tasks.rados.rados.0.smithi084.stdout:2740: done (2 left) 2024-09-20T22:52:54.157 INFO:tasks.rados.rados.0.smithi084.stdout:2742: done (1 left) 2024-09-20T22:52:54.157 INFO:tasks.rados.rados.0.smithi084.stdout:2743: read oid 8 snap -1 2024-09-20T22:52:54.157 INFO:tasks.rados.rados.0.smithi084.stdout:2743: expect (ObjNum 923 snap 305 seq_num 923) 2024-09-20T22:52:54.157 INFO:tasks.rados.rados.0.smithi084.stdout:2744: delete oid 3 current snap is 311 2024-09-20T22:52:54.157 INFO:tasks.rados.rados.0.smithi084.stdout:2741: finishing write tid 3 to smithi08477203-24 2024-09-20T22:52:54.157 INFO:tasks.rados.rados.0.smithi084.stdout:2741: finishing write tid 4 to smithi08477203-24 2024-09-20T22:52:54.157 INFO:tasks.rados.rados.0.smithi084.stdout:2741: finishing write tid 5 to smithi08477203-24 2024-09-20T22:52:54.157 INFO:tasks.rados.rados.0.smithi084.stdout:2741: oid 24 updating version 0 to 1822 2024-09-20T22:52:54.157 INFO:tasks.rados.rados.0.smithi084.stdout:2741: oid 24 updating version 1822 to 1823 2024-09-20T22:52:54.157 INFO:tasks.rados.rados.0.smithi084.stdout:2741: oid 24 version 1823 is already newer than 1821 2024-09-20T22:52:54.158 INFO:tasks.rados.rados.0.smithi084.stdout:2741: oid 24 version 1823 is already newer than 1820 2024-09-20T22:52:54.158 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 24 v 1823 (ObjNum 950 snap 311 seq_num 950) dirty exists 2024-09-20T22:52:54.158 INFO:tasks.rados.rados.0.smithi084.stdout:2741: left oid 24 (ObjNum 950 snap 311 seq_num 950) 2024-09-20T22:52:54.160 INFO:tasks.rados.rados.0.smithi084.stdout:2743: expect (ObjNum 923 snap 305 seq_num 923) 2024-09-20T22:52:54.188 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:52:54.209 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:54.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:54.210 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:52:54.262 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:52:54.298 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd reweight 7 1 2024-09-20T22:52:54.310 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:52:54.357 INFO:tasks.rados.rados.0.smithi084.stdout:2741: done (2 left) 2024-09-20T22:52:54.357 INFO:tasks.rados.rados.0.smithi084.stdout:2743: done (1 left) 2024-09-20T22:52:54.357 INFO:tasks.rados.rados.0.smithi084.stdout:2744: done (0 left) 2024-09-20T22:52:54.358 INFO:tasks.rados.rados.0.smithi084.stdout:2745: copy_from oid 21 from oid 32 current snap is 311 2024-09-20T22:52:54.358 INFO:tasks.rados.rados.0.smithi084.stdout:2746: snap_create 2024-09-20T22:52:54.362 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:54.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:54.364 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:52:54.365 INFO:tasks.rados.rados.0.smithi084.stdout:2745: finishing copy_from racing read to smithi08477203-21 2024-09-20T22:52:54.371 INFO:tasks.rados.rados.0.smithi084.stdout:2745: finishing copy_from to smithi08477203-21 2024-09-20T22:52:54.371 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 21 v 751 (ObjNum 909 snap 297 seq_num 909) dirty exists 2024-09-20T22:52:54.464 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:52:54.486 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:54.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:54.487 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:52:54.587 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:52:54.609 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:54.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:54.610 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:52:54.711 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:52:54.732 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:54.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:54.733 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:52:54.795 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:52:54.834 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:52:54.854 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:54.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:54.855 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:52:54.956 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:52:54.977 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:54.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:54.978 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:52:55.079 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:52:55.100 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:55.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:55.101 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:52:55.202 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:52:55.223 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:55.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:55.224 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:52:55.296 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is set 2024-09-20T22:52:55.300 INFO:tasks.rados.rados.0.smithi084.stdout:2745: done (1 left) 2024-09-20T22:52:55.300 INFO:tasks.rados.rados.0.smithi084.stdout:2746: done (0 left) 2024-09-20T22:52:55.300 INFO:tasks.rados.rados.0.smithi084.stdout:2747: read oid 9 snap -1 2024-09-20T22:52:55.300 INFO:tasks.rados.rados.0.smithi084.stdout:2747: expect deleted 2024-09-20T22:52:55.301 INFO:tasks.rados.rados.0.smithi084.stdout:2748: snap_remove snap 249 2024-09-20T22:52:55.305 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-20T22:52:55.305 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:52:55.325 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:52:55.347 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:55.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:55.348 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:52:55.449 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:52:55.470 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:55.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:55.471 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:52:55.540 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:55 smithi084 ceph-mon[40024]: purged_snaps scrub starts 2024-09-20T22:52:55.540 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:55 smithi084 ceph-mon[40024]: purged_snaps scrub ok 2024-09-20T22:52:55.540 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:55 smithi084 ceph-mon[40024]: pgmap v942: 158 pgs: 4 active+clean+snaptrim, 154 active+clean; 659 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail; 10 MiB/s rd, 3.4 MiB/s wr, 17 op/s 2024-09-20T22:52:55.540 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:55 smithi084 ceph-mon[40024]: osdmap e694: 8 total, 8 up, 7 in 2024-09-20T22:52:55.540 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:55 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/1415156576' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:52:55.541 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:55 smithi084 ceph-mon[29774]: purged_snaps scrub starts 2024-09-20T22:52:55.541 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:55 smithi084 ceph-mon[29774]: purged_snaps scrub ok 2024-09-20T22:52:55.541 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:55 smithi084 ceph-mon[29774]: pgmap v942: 158 pgs: 4 active+clean+snaptrim, 154 active+clean; 659 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail; 10 MiB/s rd, 3.4 MiB/s wr, 17 op/s 2024-09-20T22:52:55.541 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:55 smithi084 ceph-mon[29774]: osdmap e694: 8 total, 8 up, 7 in 2024-09-20T22:52:55.541 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:55 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/1415156576' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:52:55.572 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:52:55.593 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:55.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:55.594 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:52:55.628 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-20T22:52:55.695 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:52:55.716 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:55.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:55.717 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:52:55.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:55 smithi154 ceph-mon[33264]: purged_snaps scrub starts 2024-09-20T22:52:55.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:55 smithi154 ceph-mon[33264]: purged_snaps scrub ok 2024-09-20T22:52:55.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:55 smithi154 ceph-mon[33264]: pgmap v942: 158 pgs: 4 active+clean+snaptrim, 154 active+clean; 659 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail; 10 MiB/s rd, 3.4 MiB/s wr, 17 op/s 2024-09-20T22:52:55.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:55 smithi154 ceph-mon[33264]: osdmap e694: 8 total, 8 up, 7 in 2024-09-20T22:52:55.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:55 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/1415156576' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:52:55.818 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:52:55.839 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:55.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:55.840 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:52:55.941 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:52:55.967 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:55.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:55.968 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:52:56.069 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:52:56.094 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:56.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:56.095 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:52:56.196 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:52:56.220 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:56.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:56.221 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:52:56.289 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.3 dump_historic_ops 2024-09-20T22:52:56.305 INFO:tasks.rados.rados.0.smithi084.stdout:2748: done (1 left) 2024-09-20T22:52:56.305 INFO:tasks.rados.rados.0.smithi084.stdout:2749: delete oid 34 current snap is 312 2024-09-20T22:52:56.312 INFO:tasks.rados.rados.0.smithi084.stdout:2747: done (1 left) 2024-09-20T22:52:56.312 INFO:tasks.rados.rados.0.smithi084.stdout:2749: done (0 left) 2024-09-20T22:52:56.312 INFO:tasks.rados.rados.0.smithi084.stdout:2750: copy_from oid 19 from oid 35 current snap is 312 2024-09-20T22:52:56.312 INFO:tasks.rados.rados.0.smithi084.stdout:2751: rollback oid 32 current snap is 312 2024-09-20T22:52:56.312 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 32 to 235 2024-09-20T22:52:56.312 INFO:tasks.rados.rados.0.smithi084.stdout:2752: rollback oid 11 current snap is 312 2024-09-20T22:52:56.312 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 11 to 285 2024-09-20T22:52:56.312 INFO:tasks.rados.rados.0.smithi084.stdout:2753: snap_remove snap 268 2024-09-20T22:52:56.322 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:52:56.345 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:56.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:56.346 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:52:56.446 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:52:56.468 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:56.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:56.469 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:52:56.504 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:56 smithi084 ceph-mon[40024]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:52:56.504 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:56 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/1415156576' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T22:52:56.504 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:56 smithi084 ceph-mon[40024]: osdmap e695: 8 total, 8 up, 7 in 2024-09-20T22:52:56.504 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:56 smithi084 ceph-mon[29774]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:52:56.504 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:56 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/1415156576' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T22:52:56.504 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:56 smithi084 ceph-mon[29774]: osdmap e695: 8 total, 8 up, 7 in 2024-09-20T22:52:56.570 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:52:56.591 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:56.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:56.592 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:52:56.693 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:52:56.715 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:56.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:56.716 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:52:56.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:56 smithi154 ceph-mon[33264]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:52:56.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:56 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/1415156576' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T22:52:56.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:56 smithi154 ceph-mon[33264]: osdmap e695: 8 total, 8 up, 7 in 2024-09-20T22:52:56.816 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:52:56.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:56.839 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:56.839 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:52:56.940 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:52:56.962 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:56.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:56.963 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:52:57.063 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:52:57.084 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:57.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:57.085 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:52:57.186 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:52:57.207 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:57.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:57.208 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:52:57.268 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:52:57.306 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:52:57.309 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:52:57.324 INFO:tasks.rados.rados.0.smithi084.stdout:2753: done (3 left) 2024-09-20T22:52:57.324 INFO:tasks.rados.rados.0.smithi084.stdout:2754: write (excl) oid 24 current snap is 312 2024-09-20T22:52:57.324 INFO:tasks.rados.rados.0.smithi084.stdout:2754: seq_num 954 ranges {433878=731671,1909351=124527} 2024-09-20T22:52:57.331 INFO:tasks.rados.rados.0.smithi084.stdout:2754: writing smithi08477203-24 from 433878 to 1165549 tid 1 2024-09-20T22:52:57.337 INFO:tasks.rados.rados.0.smithi084.stdout:2754: writing smithi08477203-24 from 1909351 to 2033878 tid 2 2024-09-20T22:52:57.338 INFO:tasks.rados.rados.0.smithi084.stdout:2752: finishing rollback tid 1 to smithi08477203-11 2024-09-20T22:52:57.339 INFO:tasks.rados.rados.0.smithi084.stdout:2751: finishing rollback tid 0 to smithi08477203-32 2024-09-20T22:52:57.339 INFO:tasks.rados.rados.0.smithi084.stdout:2752: finishing rollback tid 2 to smithi08477203-11 2024-09-20T22:52:57.339 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 11 v 797 (ObjNum 815 snap 262 seq_num 815) dirty exists 2024-09-20T22:52:57.339 INFO:tasks.rados.rados.0.smithi084.stdout:2751: finishing rollback tid 1 to smithi08477203-32 2024-09-20T22:52:57.339 INFO:tasks.rados.rados.0.smithi084.stdout:2751: finishing rollback tid 2 to smithi08477203-32 2024-09-20T22:52:57.339 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 32 v 754 (ObjNum 466 snap 128 seq_num 466) dirty exists 2024-09-20T22:52:57.339 INFO:tasks.rados.rados.0.smithi084.stdout:2750: finishing copy_from racing read to smithi08477203-19 2024-09-20T22:52:57.339 INFO:tasks.rados.rados.0.smithi084.stdout:2750: finishing copy_from to smithi08477203-19 2024-09-20T22:52:57.339 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 19 v 778 (ObjNum 787 snap 254 seq_num 787) dirty exists 2024-09-20T22:52:57.339 INFO:tasks.rados.rados.0.smithi084.stdout:2754: finishing write tid 1 to smithi08477203-24 2024-09-20T22:52:57.339 INFO:tasks.rados.rados.0.smithi084.stdout:2750: done (3 left) 2024-09-20T22:52:57.339 INFO:tasks.rados.rados.0.smithi084.stdout:2751: done (2 left) 2024-09-20T22:52:57.339 INFO:tasks.rados.rados.0.smithi084.stdout:2752: done (1 left) 2024-09-20T22:52:57.340 INFO:tasks.rados.rados.0.smithi084.stdout:2755: read oid 50 snap -1 2024-09-20T22:52:57.340 INFO:tasks.rados.rados.0.smithi084.stdout:2755: expect (ObjNum 852 snap 275 seq_num 852) 2024-09-20T22:52:57.340 INFO:tasks.rados.rados.0.smithi084.stdout:2756: snap_remove snap 246 2024-09-20T22:52:57.344 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:57.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:57.345 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:52:57.428 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:57 smithi084 ceph-mon[40024]: pgmap v945: 158 pgs: 4 active+clean+snaptrim, 154 active+clean; 659 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail; 10 MiB/s rd, 3.4 MiB/s wr, 17 op/s 2024-09-20T22:52:57.428 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:57 smithi084 ceph-mon[40024]: osdmap e696: 8 total, 8 up, 7 in 2024-09-20T22:52:57.428 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:57 smithi084 ceph-mon[29774]: pgmap v945: 158 pgs: 4 active+clean+snaptrim, 154 active+clean; 659 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail; 10 MiB/s rd, 3.4 MiB/s wr, 17 op/s 2024-09-20T22:52:57.428 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:57 smithi084 ceph-mon[29774]: osdmap e696: 8 total, 8 up, 7 in 2024-09-20T22:52:57.446 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:52:57.466 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:57.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:57.468 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:52:57.568 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:52:57.590 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:57.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:57.591 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:52:57.692 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:52:57.713 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:57.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:57.714 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:52:57.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:57 smithi154 ceph-mon[33264]: pgmap v945: 158 pgs: 4 active+clean+snaptrim, 154 active+clean; 659 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail; 10 MiB/s rd, 3.4 MiB/s wr, 17 op/s 2024-09-20T22:52:57.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:57 smithi154 ceph-mon[33264]: osdmap e696: 8 total, 8 up, 7 in 2024-09-20T22:52:57.815 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:52:57.836 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:57.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:57.837 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:52:57.938 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:52:57.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:57.959 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:57.959 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:52:58.060 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:52:58.082 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:58.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:58.083 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:52:58.184 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:52:58.189 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:52:58.189 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:52:58.206 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:58.206 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:52:58.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:58.207 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:52:58.207 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:52:58.222 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:52:58.222 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:52:58.238 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-20T22:52:58.238 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T22:52:58.255 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:52:58.255 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:52:58.272 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:52:58.272 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:52:58.287 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.7 to 1 (10000) 2024-09-20T22:52:58.289 INFO:tasks.rados.rados.0.smithi084.stdout:2756: done (2 left) 2024-09-20T22:52:58.289 INFO:tasks.rados.rados.0.smithi084.stdout:2757: rollback oid 13 current snap is 312 2024-09-20T22:52:58.290 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 13 to 308 2024-09-20T22:52:58.290 INFO:tasks.rados.rados.0.smithi084.stdout:2758: snap_create 2024-09-20T22:52:58.290 INFO:tasks.rados.rados.0.smithi084.stdout:2754: finishing write tid 2 to smithi08477203-24 2024-09-20T22:52:58.290 INFO:tasks.rados.rados.0.smithi084.stdout:2754: finishing write tid 3 to smithi08477203-24 2024-09-20T22:52:58.290 INFO:tasks.rados.rados.0.smithi084.stdout:2754: finishing write tid 4 to smithi08477203-24 2024-09-20T22:52:58.290 INFO:tasks.rados.rados.0.smithi084.stdout:2754: oid 24 updating version 0 to 1824 2024-09-20T22:52:58.290 INFO:tasks.rados.rados.0.smithi084.stdout:2754: oid 24 updating version 1824 to 1825 2024-09-20T22:52:58.290 INFO:tasks.rados.rados.0.smithi084.stdout:2754: oid 24 updating version 1825 to 1826 2024-09-20T22:52:58.290 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 24 v 1826 (ObjNum 954 snap 312 seq_num 954) dirty exists 2024-09-20T22:52:58.290 INFO:tasks.rados.rados.0.smithi084.stdout:2754: left oid 24 (ObjNum 954 snap 312 seq_num 954) 2024-09-20T22:52:58.290 INFO:tasks.rados.rados.0.smithi084.stdout:2755: expect (ObjNum 852 snap 275 seq_num 852) 2024-09-20T22:52:58.298 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:52:58.298 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:52:58.307 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:52:58.322 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:52:58.322 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:52:58.334 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:58.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:58.335 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:52:58.382 INFO:tasks.rados.rados.0.smithi084.stdout:2757: finishing rollback tid 0 to smithi08477203-13 2024-09-20T22:52:58.382 INFO:tasks.rados.rados.0.smithi084.stdout:2757: finishing rollback tid 1 to smithi08477203-13 2024-09-20T22:52:58.383 INFO:tasks.rados.rados.0.smithi084.stdout:2757: finishing rollback tid 2 to smithi08477203-13 2024-09-20T22:52:58.383 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 13 v 2138 (ObjNum 926 snap 305 seq_num 926) dirty exists 2024-09-20T22:52:58.432 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-20T22:52:58.432 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2024-09-20T22:52:58.433 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2024-09-20T22:52:58.433 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [ 2024-09-20T22:52:58.433 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.433 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:554 7.a 7:5122292a:::smithi08477203-33:head [copy-from ver 6] snapc 13=[13,12,11,10,f,d] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-09-20T22:52:58.433 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:43:27.336824+0000", 2024-09-20T22:52:58.433 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 571.09297981999998, 2024-09-20T22:52:58.433 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.056403397000000001, 2024-09-20T22:52:58.433 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:52:58.433 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:52:58.433 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:52:58.433 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:52:58.433 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:52:58.433 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:52:58.433 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 554 2024-09-20T22:52:58.434 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.434 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:52:58.434 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.434 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:52:58.434 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:27.336824+0000", 2024-09-20T22:52:58.434 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:52:58.434 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.434 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.434 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:52:58.434 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:27.336824+0000", 2024-09-20T22:52:58.434 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:52:58.434 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.434 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.435 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:52:58.435 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:27.336826+0000", 2024-09-20T22:52:58.435 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.0339999999999999e-06 2024-09-20T22:52:58.435 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.435 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.435 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:52:58.435 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:27.336837+0000", 2024-09-20T22:52:58.435 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1243000000000001e-05 2024-09-20T22:52:58.435 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.435 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.435 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:52:58.435 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:27.336857+0000", 2024-09-20T22:52:58.435 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.9868999999999999e-05 2024-09-20T22:52:58.435 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.436 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.436 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:52:58.436 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:27.336869+0000", 2024-09-20T22:52:58.436 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1348999999999999e-05 2024-09-20T22:52:58.436 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.436 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.436 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:52:58.436 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:27.336946+0000", 2024-09-20T22:52:58.436 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.7683999999999997e-05 2024-09-20T22:52:58.436 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.436 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.436 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:52:58.436 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:27.337045+0000", 2024-09-20T22:52:58.436 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.9265999999999995e-05 2024-09-20T22:52:58.437 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.437 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.437 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 2,5", 2024-09-20T22:52:58.437 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:27.340603+0000", 2024-09-20T22:52:58.437 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.003557892 2024-09-20T22:52:58.437 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.437 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.437 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:52:58.437 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:27.343571+0000", 2024-09-20T22:52:58.437 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0029676609999999999 2024-09-20T22:52:58.437 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.437 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.437 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:52:58.437 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:27.346457+0000", 2024-09-20T22:52:58.438 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0028860040000000002 2024-09-20T22:52:58.438 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.438 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.438 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:52:58.438 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:27.393131+0000", 2024-09-20T22:52:58.438 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.046673813000000001 2024-09-20T22:52:58.438 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.438 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.438 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:52:58.438 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:27.393173+0000", 2024-09-20T22:52:58.438 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.2076999999999997e-05 2024-09-20T22:52:58.438 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.438 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.438 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:52:58.439 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:27.393227+0000", 2024-09-20T22:52:58.439 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.4505000000000001e-05 2024-09-20T22:52:58.439 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:52:58.439 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:52:58.439 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:52:58.439 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.439 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.439 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:663 7.12 7:4fa4b2b3:::smithi08477203-21:head [write 665577~781581,stat] snapc 17=[17,16,15,11,d] ondisk+write+known_if_redirected+supports_pool_eio e137)", 2024-09-20T22:52:58.439 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:43:36.161568+0000", 2024-09-20T22:52:58.439 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 562.26823564300003, 2024-09-20T22:52:58.439 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.99286350899999998, 2024-09-20T22:52:58.439 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:52:58.439 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:52:58.440 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:52:58.440 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:52:58.440 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:52:58.440 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:52:58.440 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 663 2024-09-20T22:52:58.440 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.440 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:52:58.440 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.440 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:52:58.440 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.161568+0000", 2024-09-20T22:52:58.440 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:52:58.440 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.440 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.440 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:52:58.441 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.161568+0000", 2024-09-20T22:52:58.441 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:52:58.441 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.441 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.441 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:52:58.441 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.161570+0000", 2024-09-20T22:52:58.441 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.0679999999999999e-06 2024-09-20T22:52:58.441 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.441 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.441 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:52:58.441 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.162310+0000", 2024-09-20T22:52:58.441 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00074005999999999996 2024-09-20T22:52:58.441 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.441 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.441 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:52:58.442 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.162349+0000", 2024-09-20T22:52:58.442 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.8778e-05 2024-09-20T22:52:58.442 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.442 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.442 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:52:58.442 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.162375+0000", 2024-09-20T22:52:58.442 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.6089e-05 2024-09-20T22:52:58.442 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.442 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.442 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:52:58.442 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.162434+0000", 2024-09-20T22:52:58.442 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.8764999999999997e-05 2024-09-20T22:52:58.442 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.442 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.442 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-09-20T22:52:58.443 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.162443+0000", 2024-09-20T22:52:58.443 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.054e-06 2024-09-20T22:52:58.443 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.443 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.443 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:52:58.443 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.143416+0000", 2024-09-20T22:52:58.443 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.98097291600000003 2024-09-20T22:52:58.443 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.443 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.443 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:52:58.443 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.143522+0000", 2024-09-20T22:52:58.443 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00010626 2024-09-20T22:52:58.443 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.443 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.444 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 1,2", 2024-09-20T22:52:58.444 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.143654+0000", 2024-09-20T22:52:58.444 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013218799999999999 2024-09-20T22:52:58.444 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.444 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.444 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:52:58.444 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.148086+0000", 2024-09-20T22:52:58.444 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0044314189999999998 2024-09-20T22:52:58.444 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.444 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.444 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:52:58.444 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.153546+0000", 2024-09-20T22:52:58.444 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0054599920000000003 2024-09-20T22:52:58.444 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.445 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.445 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:52:58.445 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.154317+0000", 2024-09-20T22:52:58.445 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00077125999999999996 2024-09-20T22:52:58.445 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.445 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.445 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:52:58.445 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.154380+0000", 2024-09-20T22:52:58.445 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.2526000000000004e-05 2024-09-20T22:52:58.445 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.445 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.445 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:52:58.445 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.154432+0000", 2024-09-20T22:52:58.445 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.2133999999999999e-05 2024-09-20T22:52:58.445 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:52:58.446 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:52:58.446 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:52:58.446 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.446 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.446 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:52:58.447 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:664 7.12 7:4fa4b2b3:::smithi08477203-21:head [write 2062560~563688] snapc 17=[17,16,15,11,d] ondisk+write+known_if_redirected+supports_pool_eio e137)", 2024-09-20T22:52:58.447 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:43:36.167947+0000", 2024-09-20T22:52:58.447 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 562.26185717299995, 2024-09-20T22:52:58.447 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.98665172599999995, 2024-09-20T22:52:58.447 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:52:58.447 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:52:58.447 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:52:58.447 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:52:58.448 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:52:58.448 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:52:58.448 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 664 2024-09-20T22:52:58.448 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.448 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:52:58.448 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.448 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:52:58.448 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.167947+0000", 2024-09-20T22:52:58.448 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:52:58.448 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.448 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.448 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:52:58.448 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.167947+0000", 2024-09-20T22:52:58.448 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:52:58.449 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.449 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.449 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:52:58.449 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.167948+0000", 2024-09-20T22:52:58.449 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.187e-06 2024-09-20T22:52:58.449 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.449 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.449 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:52:58.449 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.168193+0000", 2024-09-20T22:52:58.449 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00024496199999999999 2024-09-20T22:52:58.449 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.449 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.449 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:52:58.449 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.168203+0000", 2024-09-20T22:52:58.449 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0543999999999999e-05 2024-09-20T22:52:58.450 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.450 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.450 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:52:58.450 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.168209+0000", 2024-09-20T22:52:58.450 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.2839999999999997e-06 2024-09-20T22:52:58.450 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.450 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.450 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:52:58.450 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.168252+0000", 2024-09-20T22:52:58.450 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.3115000000000003e-05 2024-09-20T22:52:58.450 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.450 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.450 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-09-20T22:52:58.450 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.168257+0000", 2024-09-20T22:52:58.451 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.6860000000000003e-06 2024-09-20T22:52:58.451 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.451 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.451 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:52:58.451 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.144795+0000", 2024-09-20T22:52:58.451 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.97653758099999999 2024-09-20T22:52:58.451 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.451 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.451 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:52:58.451 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.144865+0000", 2024-09-20T22:52:58.451 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.9641000000000006e-05 2024-09-20T22:52:58.451 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.451 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.451 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 1,2", 2024-09-20T22:52:58.452 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.144986+0000", 2024-09-20T22:52:58.452 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000121248 2024-09-20T22:52:58.452 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.452 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.452 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:52:58.452 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.148091+0000", 2024-09-20T22:52:58.452 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0031050399999999999 2024-09-20T22:52:58.452 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.452 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.452 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:52:58.452 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.153603+0000", 2024-09-20T22:52:58.452 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0055118479999999997 2024-09-20T22:52:58.452 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.452 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.453 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:52:58.453 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.154508+0000", 2024-09-20T22:52:58.453 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00090530600000000004 2024-09-20T22:52:58.453 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.453 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.453 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:52:58.453 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.154560+0000", 2024-09-20T22:52:58.453 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.1474000000000002e-05 2024-09-20T22:52:58.453 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.453 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.453 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:52:58.453 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.154598+0000", 2024-09-20T22:52:58.453 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.8810000000000003e-05 2024-09-20T22:52:58.453 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:52:58.454 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:52:58.454 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:52:58.454 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.454 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.454 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:665 7.12 7:4fa4b2b3:::smithi08477203-21:head [write 3182305~643664] snapc 17=[17,16,15,11,d] ondisk+write+known_if_redirected+supports_pool_eio e137)", 2024-09-20T22:52:58.454 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:43:36.174520+0000", 2024-09-20T22:52:58.454 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 562.25528408299999, 2024-09-20T22:52:58.454 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.98020083400000002, 2024-09-20T22:52:58.454 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:52:58.454 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:52:58.454 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:52:58.454 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:52:58.454 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:52:58.454 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:52:58.455 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 665 2024-09-20T22:52:58.455 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.455 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:52:58.455 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.455 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:52:58.455 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174520+0000", 2024-09-20T22:52:58.455 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:52:58.455 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.455 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.455 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:52:58.455 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174520+0000", 2024-09-20T22:52:58.455 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:52:58.455 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.455 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.455 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:52:58.456 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174521+0000", 2024-09-20T22:52:58.456 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.234e-06 2024-09-20T22:52:58.456 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.456 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.456 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:52:58.456 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174816+0000", 2024-09-20T22:52:58.456 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000294682 2024-09-20T22:52:58.456 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.456 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.456 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:52:58.456 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174827+0000", 2024-09-20T22:52:58.456 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1025e-05 2024-09-20T22:52:58.456 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.456 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.457 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:52:58.457 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174832+0000", 2024-09-20T22:52:58.457 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.3990000000000003e-06 2024-09-20T22:52:58.457 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.457 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.457 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:52:58.457 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174868+0000", 2024-09-20T22:52:58.457 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.6225e-05 2024-09-20T22:52:58.457 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.457 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.457 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-09-20T22:52:58.458 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174873+0000", 2024-09-20T22:52:58.458 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.848e-06 2024-09-20T22:52:58.458 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.458 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.458 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:52:58.458 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.145844+0000", 2024-09-20T22:52:58.458 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.97097048900000005 2024-09-20T22:52:58.458 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.458 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.458 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:52:58.458 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.145917+0000", 2024-09-20T22:52:58.458 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.3248000000000001e-05 2024-09-20T22:52:58.458 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.459 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.459 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 1,2", 2024-09-20T22:52:58.459 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.146028+0000", 2024-09-20T22:52:58.459 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000110773 2024-09-20T22:52:58.459 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.459 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.459 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:52:58.459 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.148095+0000", 2024-09-20T22:52:58.459 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0020678179999999999 2024-09-20T22:52:58.459 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.459 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.459 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:52:58.459 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.153967+0000", 2024-09-20T22:52:58.460 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0058717350000000003 2024-09-20T22:52:58.460 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.460 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.460 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:52:58.460 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.154633+0000", 2024-09-20T22:52:58.460 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00066600600000000002 2024-09-20T22:52:58.460 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.460 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.460 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:52:58.460 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.154683+0000", 2024-09-20T22:52:58.460 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.9872e-05 2024-09-20T22:52:58.460 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.460 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.460 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:52:58.460 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.154721+0000", 2024-09-20T22:52:58.461 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.748e-05 2024-09-20T22:52:58.461 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:52:58.461 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:52:58.461 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:52:58.461 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.461 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.461 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:666 7.12 7:4fa4b2b3:::smithi08477203-21:head [setxattr _header (57) in=7b,truncate 3865577] snapc 17=[17,16,15,11,d] ondisk+write+known_if_redirected+supports_pool_eio e137)", 2024-09-20T22:52:58.461 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:43:36.174844+0000", 2024-09-20T22:52:58.461 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 562.25495988399996, 2024-09-20T22:52:58.461 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.98002428600000002, 2024-09-20T22:52:58.461 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:52:58.461 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:52:58.461 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:52:58.461 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:52:58.461 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:52:58.462 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:52:58.462 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 666 2024-09-20T22:52:58.462 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.462 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:52:58.462 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.462 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:52:58.462 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174844+0000", 2024-09-20T22:52:58.462 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:52:58.462 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.462 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.462 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:52:58.462 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174844+0000", 2024-09-20T22:52:58.462 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:52:58.463 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.463 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.463 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:52:58.463 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174845+0000", 2024-09-20T22:52:58.463 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.1999999999999999e-07 2024-09-20T22:52:58.463 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.463 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.463 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:52:58.463 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174853+0000", 2024-09-20T22:52:58.463 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.1950000000000003e-06 2024-09-20T22:52:58.463 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.463 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.463 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:52:58.463 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174858+0000", 2024-09-20T22:52:58.463 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.5650000000000001e-06 2024-09-20T22:52:58.464 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.464 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.464 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:52:58.464 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174863+0000", 2024-09-20T22:52:58.464 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.1649999999999997e-06 2024-09-20T22:52:58.464 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.464 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.464 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:52:58.464 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174915+0000", 2024-09-20T22:52:58.464 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.1045999999999999e-05 2024-09-20T22:52:58.464 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.464 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.464 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-09-20T22:52:58.464 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174921+0000", 2024-09-20T22:52:58.465 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.404e-06 2024-09-20T22:52:58.465 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.465 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.465 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:52:58.465 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.146528+0000", 2024-09-20T22:52:58.465 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.97160681199999999 2024-09-20T22:52:58.465 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.465 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.465 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:52:58.465 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.146881+0000", 2024-09-20T22:52:58.465 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00035348200000000001 2024-09-20T22:52:58.465 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.465 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.465 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 1,2", 2024-09-20T22:52:58.466 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.147011+0000", 2024-09-20T22:52:58.466 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013018900000000001 2024-09-20T22:52:58.466 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.466 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.466 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:52:58.466 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.148100+0000", 2024-09-20T22:52:58.466 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010881580000000001 2024-09-20T22:52:58.466 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.466 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.466 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:52:58.466 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.154278+0000", 2024-09-20T22:52:58.466 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0061781730000000003 2024-09-20T22:52:58.466 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.466 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.467 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:52:58.467 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.154764+0000", 2024-09-20T22:52:58.467 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00048589199999999998 2024-09-20T22:52:58.467 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.467 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.467 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:52:58.467 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.154815+0000", 2024-09-20T22:52:58.467 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.1428000000000003e-05 2024-09-20T22:52:58.467 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.467 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.467 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:52:58.467 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.154868+0000", 2024-09-20T22:52:58.467 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.3156999999999999e-05 2024-09-20T22:52:58.467 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:52:58.468 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:52:58.468 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:52:58.468 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.468 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.468 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:667 7.12 7:4fa4b2b3:::smithi08477203-21:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e137)", 2024-09-20T22:52:58.468 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:43:36.174872+0000", 2024-09-20T22:52:58.468 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 562.25493219099997, 2024-09-20T22:52:58.468 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.98024790299999998, 2024-09-20T22:52:58.468 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:52:58.468 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:52:58.468 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-20T22:52:58.468 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:52:58.468 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:52:58.468 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:52:58.469 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 667 2024-09-20T22:52:58.469 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.469 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:52:58.469 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.469 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:52:58.469 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174872+0000", 2024-09-20T22:52:58.469 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:52:58.469 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.469 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.469 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:52:58.469 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174872+0000", 2024-09-20T22:52:58.469 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:52:58.469 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.469 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.469 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:52:58.470 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174872+0000", 2024-09-20T22:52:58.470 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.2199999999999999e-07 2024-09-20T22:52:58.470 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.470 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.470 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:52:58.470 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174874+0000", 2024-09-20T22:52:58.470 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.187e-06 2024-09-20T22:52:58.470 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.470 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.470 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:52:58.470 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174881+0000", 2024-09-20T22:52:58.470 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.674e-06 2024-09-20T22:52:58.470 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.470 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.471 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:52:58.471 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174886+0000", 2024-09-20T22:52:58.471 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.2240000000000003e-06 2024-09-20T22:52:58.471 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.471 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.471 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:52:58.471 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174936+0000", 2024-09-20T22:52:58.471 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.9824999999999998e-05 2024-09-20T22:52:58.471 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.471 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.471 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-09-20T22:52:58.471 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:36.174940+0000", 2024-09-20T22:52:58.471 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.5700000000000001e-06 2024-09-20T22:52:58.471 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.471 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.472 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:52:58.472 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.147523+0000", 2024-09-20T22:52:58.472 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.97258315399999995 2024-09-20T22:52:58.472 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.472 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.472 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for rw locks", 2024-09-20T22:52:58.472 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.147591+0000", 2024-09-20T22:52:58.472 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.7878000000000003e-05 2024-09-20T22:52:58.472 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.472 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.472 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:52:58.472 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.154925+0000", 2024-09-20T22:52:58.472 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0073344430000000004 2024-09-20T22:52:58.473 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.473 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.473 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:52:58.473 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.155020+0000", 2024-09-20T22:52:58.473 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.4767999999999998e-05 2024-09-20T22:52:58.473 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.473 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.473 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:52:58.473 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:43:37.155120+0000", 2024-09-20T22:52:58.473 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.9758000000000003e-05 2024-09-20T22:52:58.473 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:52:58.473 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:52:58.473 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:52:58.473 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.473 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.474 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.14478.0:1259 5.3 5:c609908c:::notify.5:head [watch reconnect cookie 94843898679808 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e169)", 2024-09-20T22:52:58.474 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:44:08.230723+0000", 2024-09-20T22:52:58.474 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 530.19908088600005, 2024-09-20T22:52:58.474 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.94983577399999997, 2024-09-20T22:52:58.474 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:52:58.474 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:52:58.474 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-20T22:52:58.474 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:52:58.474 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.14478", 2024-09-20T22:52:58.474 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/2994273271", 2024-09-20T22:52:58.474 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1259 2024-09-20T22:52:58.474 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.474 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:52:58.474 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.475 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:52:58.475 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:08.230723+0000", 2024-09-20T22:52:58.475 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:52:58.475 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.475 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.475 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:52:58.475 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:08.230723+0000", 2024-09-20T22:52:58.475 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:52:58.475 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.475 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.475 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:52:58.475 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:08.230724+0000", 2024-09-20T22:52:58.475 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.077e-06 2024-09-20T22:52:58.475 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.475 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.476 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:52:58.476 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:08.230729+0000", 2024-09-20T22:52:58.476 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.9930000000000003e-06 2024-09-20T22:52:58.476 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.476 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.476 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:52:58.476 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:08.230741+0000", 2024-09-20T22:52:58.476 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1897e-05 2024-09-20T22:52:58.476 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.476 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.476 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:52:58.476 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:08.230747+0000", 2024-09-20T22:52:58.476 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.9830000000000001e-06 2024-09-20T22:52:58.476 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.476 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.477 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:52:58.477 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:08.230802+0000", 2024-09-20T22:52:58.477 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.4657000000000002e-05 2024-09-20T22:52:58.477 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.477 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.477 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-09-20T22:52:58.477 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:08.230811+0000", 2024-09-20T22:52:58.477 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.1160000000000005e-06 2024-09-20T22:52:58.477 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.477 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.477 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:52:58.477 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:09.171968+0000", 2024-09-20T22:52:58.478 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.94115747299999997 2024-09-20T22:52:58.478 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.478 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.478 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for missing object", 2024-09-20T22:52:58.478 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:09.172098+0000", 2024-09-20T22:52:58.478 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00012937499999999999 2024-09-20T22:52:58.478 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.478 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.478 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:52:58.478 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:09.174248+0000", 2024-09-20T22:52:58.478 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0021508959999999998 2024-09-20T22:52:58.478 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.478 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.478 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for rw locks", 2024-09-20T22:52:58.478 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:09.174341+0000", 2024-09-20T22:52:58.479 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.2906999999999997e-05 2024-09-20T22:52:58.479 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.479 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.479 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:52:58.479 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:09.175083+0000", 2024-09-20T22:52:58.479 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00074120100000000003 2024-09-20T22:52:58.479 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.479 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.479 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:52:58.479 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:09.175149+0000", 2024-09-20T22:52:58.479 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.6877000000000004e-05 2024-09-20T22:52:58.479 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.479 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.479 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:52:58.479 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:09.180559+0000", 2024-09-20T22:52:58.480 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0054093220000000003 2024-09-20T22:52:58.480 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:52:58.480 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:52:58.480 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:52:58.480 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.480 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.480 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:1419 7.1 7:84a866f5:::smithi08477203-27:head [append 0~0] snapc 2f=[2f] ondisk+write+known_if_redirected+supports_pool_eio e189)", 2024-09-20T22:52:58.480 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:44:28.138055+0000", 2024-09-20T22:52:58.480 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 510.29174924699998, 2024-09-20T22:52:58.480 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.030290338999999999, 2024-09-20T22:52:58.480 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:52:58.480 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:52:58.480 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:52:58.481 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:52:58.481 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:52:58.481 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:52:58.481 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1419 2024-09-20T22:52:58.481 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.481 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:52:58.481 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.481 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:52:58.481 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.138055+0000", 2024-09-20T22:52:58.481 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:52:58.481 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.481 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.481 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:52:58.481 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.138055+0000", 2024-09-20T22:52:58.481 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:52:58.482 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.482 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.482 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:52:58.482 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.138056+0000", 2024-09-20T22:52:58.482 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.13e-06 2024-09-20T22:52:58.482 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.482 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.482 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:52:58.482 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.138060+0000", 2024-09-20T22:52:58.482 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.0269999999999997e-06 2024-09-20T22:52:58.482 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.482 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.482 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:52:58.482 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.138070+0000", 2024-09-20T22:52:58.483 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.961e-06 2024-09-20T22:52:58.483 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.483 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.483 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:52:58.483 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.138076+0000", 2024-09-20T22:52:58.483 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.1800000000000001e-06 2024-09-20T22:52:58.483 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.483 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.483 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:52:58.483 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.138137+0000", 2024-09-20T22:52:58.483 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.1159000000000002e-05 2024-09-20T22:52:58.483 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.483 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.484 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting_for_map not empty", 2024-09-20T22:52:58.484 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.138141+0000", 2024-09-20T22:52:58.484 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.189e-06 2024-09-20T22:52:58.484 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.484 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.484 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:52:58.484 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.154982+0000", 2024-09-20T22:52:58.484 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.016840738000000001 2024-09-20T22:52:58.484 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.484 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.484 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:52:58.484 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.155089+0000", 2024-09-20T22:52:58.484 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00010653399999999999 2024-09-20T22:52:58.484 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.484 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.485 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 1,6", 2024-09-20T22:52:58.485 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.155320+0000", 2024-09-20T22:52:58.485 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00023196299999999999 2024-09-20T22:52:58.485 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.485 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.485 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:52:58.485 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.158224+0000", 2024-09-20T22:52:58.485 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00290361 2024-09-20T22:52:58.485 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.485 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.485 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:52:58.485 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.167686+0000", 2024-09-20T22:52:58.485 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0094622360000000006 2024-09-20T22:52:58.485 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.486 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.486 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:52:58.486 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.168183+0000", 2024-09-20T22:52:58.486 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00049655999999999995 2024-09-20T22:52:58.486 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.486 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.486 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:52:58.486 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.168273+0000", 2024-09-20T22:52:58.486 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.9758000000000004e-05 2024-09-20T22:52:58.486 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.486 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.486 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:52:58.486 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:28.168345+0000", 2024-09-20T22:52:58.486 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.2293999999999996e-05 2024-09-20T22:52:58.487 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:52:58.487 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:52:58.487 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:52:58.487 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.487 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.487 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:2347 7.a 7:5122292a:::smithi08477203-33:head [sparse-read 0~2590699,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e260)", 2024-09-20T22:52:58.487 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:45:39.196440+0000", 2024-09-20T22:52:58.487 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 439.23336411899999, 2024-09-20T22:52:58.487 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.026701163, 2024-09-20T22:52:58.487 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:52:58.487 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:52:58.487 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-20T22:52:58.487 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:52:58.487 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:52:58.488 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:52:58.488 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:52:58 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: ::ffff:172.21.15.154 - - [20/Sep/2024:22:52:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-20T22:52:58.488 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:58 smithi084 ceph-mon[40024]: osdmap e697: 8 total, 8 up, 7 in 2024-09-20T22:52:58.488 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:58 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/258403835' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-09-20T22:52:58.488 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:58 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-09-20T22:52:58.488 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:58 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/3807117380' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:52:58.489 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:58 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-20T22:52:58.489 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:58 smithi084 ceph-mon[40024]: osdmap e698: 8 total, 8 up, 7 in 2024-09-20T22:52:58.489 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:58 smithi084 ceph-mon[29774]: osdmap e697: 8 total, 8 up, 7 in 2024-09-20T22:52:58.489 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:58 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/258403835' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-09-20T22:52:58.489 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:58 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-09-20T22:52:58.489 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:58 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/3807117380' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:52:58.489 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:58 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-20T22:52:58.489 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:58 smithi084 ceph-mon[29774]: osdmap e698: 8 total, 8 up, 7 in 2024-09-20T22:52:58.491 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 2347 2024-09-20T22:52:58.491 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.491 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:52:58.491 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.491 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:52:58.491 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.196440+0000", 2024-09-20T22:52:58.491 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:52:58.491 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.491 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.491 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:52:58.491 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.196440+0000", 2024-09-20T22:52:58.491 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:52:58.492 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.492 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.492 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:52:58.492 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.196441+0000", 2024-09-20T22:52:58.492 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.714e-06 2024-09-20T22:52:58.492 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.492 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.492 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:52:58.492 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.196449+0000", 2024-09-20T22:52:58.492 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.8730000000000007e-06 2024-09-20T22:52:58.492 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.492 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.492 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:52:58.493 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.196462+0000", 2024-09-20T22:52:58.493 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2649e-05 2024-09-20T22:52:58.493 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.493 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.493 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:52:58.493 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.196470+0000", 2024-09-20T22:52:58.493 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.4989999999999994e-06 2024-09-20T22:52:58.493 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.493 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.493 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:52:58.493 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.212172+0000", 2024-09-20T22:52:58.493 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.015701335 2024-09-20T22:52:58.493 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.493 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.494 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:52:58.494 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.212270+0000", 2024-09-20T22:52:58.494 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.8257999999999993e-05 2024-09-20T22:52:58.494 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.494 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.494 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:52:58.494 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.223141+0000", 2024-09-20T22:52:58.494 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010870835000000001 2024-09-20T22:52:58.494 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:52:58.494 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:52:58.494 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:52:58.494 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.494 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.494 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:2348 7.a 7:5122292a:::smithi08477203-33:head [read 0~3848803,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e260)", 2024-09-20T22:52:58.495 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:45:39.196485+0000", 2024-09-20T22:52:58.495 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 439.23331864599999, 2024-09-20T22:52:58.495 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0284703, 2024-09-20T22:52:58.495 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:52:58.495 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:52:58.495 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-20T22:52:58.495 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:52:58.495 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:52:58.495 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:52:58.495 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 2348 2024-09-20T22:52:58.495 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.495 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:52:58.495 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.495 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:52:58.496 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.196485+0000", 2024-09-20T22:52:58.496 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:52:58.496 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.496 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.496 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:52:58.496 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.196485+0000", 2024-09-20T22:52:58.496 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:52:58.496 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.496 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.496 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:52:58.496 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.196486+0000", 2024-09-20T22:52:58.496 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.9599999999999999e-07 2024-09-20T22:52:58.496 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.496 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.497 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:52:58.497 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.196489+0000", 2024-09-20T22:52:58.497 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.5470000000000001e-06 2024-09-20T22:52:58.497 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.497 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.497 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:52:58.497 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.196496+0000", 2024-09-20T22:52:58.497 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.3570000000000003e-06 2024-09-20T22:52:58.497 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.497 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.497 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:52:58.497 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.196501+0000", 2024-09-20T22:52:58.497 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.6239999999999999e-06 2024-09-20T22:52:58.497 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.497 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.498 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:52:58.498 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.223176+0000", 2024-09-20T22:52:58.498 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.026675285999999999 2024-09-20T22:52:58.498 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.498 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.498 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:52:58.498 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.223274+0000", 2024-09-20T22:52:58.498 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.7009999999999994e-05 2024-09-20T22:52:58.498 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.498 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.498 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:52:58.498 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.224955+0000", 2024-09-20T22:52:58.499 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00168198 2024-09-20T22:52:58.499 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:52:58.499 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:52:58.499 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:52:58.499 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.499 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.499 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:2349 7.a 7:5122292a:::smithi08477203-33:head [sparse-read 0~2590699] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e260)", 2024-09-20T22:52:58.499 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:45:39.196508+0000", 2024-09-20T22:52:58.499 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 439.23329536, 2024-09-20T22:52:58.499 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.028731293000000001, 2024-09-20T22:52:58.499 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:52:58.499 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:52:58.499 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-20T22:52:58.499 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:52:58.499 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:52:58.500 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:52:58.500 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 2349 2024-09-20T22:52:58.500 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.500 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:52:58.500 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.500 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:52:58.500 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.196508+0000", 2024-09-20T22:52:58.500 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:52:58.500 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.500 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.500 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:52:58.500 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.196508+0000", 2024-09-20T22:52:58.500 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:52:58.500 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.501 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.501 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:52:58.501 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.196509+0000", 2024-09-20T22:52:58.501 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.08e-07 2024-09-20T22:52:58.501 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.501 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.501 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:52:58.501 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.196511+0000", 2024-09-20T22:52:58.501 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.142e-06 2024-09-20T22:52:58.501 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.501 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.501 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:52:58.501 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.196517+0000", 2024-09-20T22:52:58.501 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.6330000000000001e-06 2024-09-20T22:52:58.501 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.502 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.502 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:52:58.502 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.196522+0000", 2024-09-20T22:52:58.502 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.3530000000000002e-06 2024-09-20T22:52:58.502 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.502 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.502 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:52:58.502 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.224990+0000", 2024-09-20T22:52:58.502 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.028467864999999998 2024-09-20T22:52:58.502 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.502 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.502 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:52:58.502 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.225088+0000", 2024-09-20T22:52:58.502 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.8281000000000003e-05 2024-09-20T22:52:58.503 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.503 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.503 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:52:58.503 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.225240+0000", 2024-09-20T22:52:58.503 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00015161100000000001 2024-09-20T22:52:58.503 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:52:58.503 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:52:58.503 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:52:58.503 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.503 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.503 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(osd.6.0:727 7.c e260/94 7:318c7512:::smithi08477203-44:4c v 260'428, mlcod=258'424)", 2024-09-20T22:52:58.503 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:45:39.202413+0000", 2024-09-20T22:52:58.503 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 439.22739082099997, 2024-09-20T22:52:58.503 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.022852897, 2024-09-20T22:52:58.503 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:52:58.504 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:52:58.504 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:52:58.504 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:52:58.504 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.504 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:52:58.504 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.202413+0000", 2024-09-20T22:52:58.504 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:52:58.504 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.504 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.504 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:52:58.504 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.202413+0000", 2024-09-20T22:52:58.504 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:52:58.504 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.504 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.504 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:52:58.505 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.202414+0000", 2024-09-20T22:52:58.505 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.9299999999999997e-07 2024-09-20T22:52:58.505 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.505 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.505 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:52:58.505 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.202421+0000", 2024-09-20T22:52:58.505 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.8109999999999997e-06 2024-09-20T22:52:58.505 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.505 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.505 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:52:58.505 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.202433+0000", 2024-09-20T22:52:58.505 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2411e-05 2024-09-20T22:52:58.505 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.505 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.506 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:52:58.506 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.202442+0000", 2024-09-20T22:52:58.506 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.7269999999999996e-06 2024-09-20T22:52:58.506 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.506 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.506 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:52:58.506 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.221764+0000", 2024-09-20T22:52:58.506 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.019322068000000001 2024-09-20T22:52:58.506 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.506 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.506 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:52:58.506 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.221786+0000", 2024-09-20T22:52:58.506 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.2167e-05 2024-09-20T22:52:58.507 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.507 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.507 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:52:58.507 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.225217+0000", 2024-09-20T22:52:58.507 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0034313709999999999 2024-09-20T22:52:58.507 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.507 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.507 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:52:58.507 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.225266+0000", 2024-09-20T22:52:58.507 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.8649000000000001e-05 2024-09-20T22:52:58.507 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:52:58.507 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:52:58.507 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:52:58.507 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.507 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.508 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:2525 7.1 7:8422f82d:::smithi08477203-22:head [copy-from ver 174] snapc 5d=[5d,5c,5a,50,47] ondisk+write+known_if_redirected+supports_pool_eio e278)", 2024-09-20T22:52:58.508 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:45:57.170348+0000", 2024-09-20T22:52:58.508 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 421.25945584700003, 2024-09-20T22:52:58.508 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.023333486000000001, 2024-09-20T22:52:58.508 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:52:58.508 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:52:58.508 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:52:58.508 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:52:58.508 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:52:58.508 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:52:58.508 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 2525 2024-09-20T22:52:58.508 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.508 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:52:58.508 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.509 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:52:58.509 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:57.170348+0000", 2024-09-20T22:52:58.509 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:52:58.509 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.509 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.509 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:52:58.509 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:57.170348+0000", 2024-09-20T22:52:58.509 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:52:58.509 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.509 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.509 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:52:58.509 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:57.170350+0000", 2024-09-20T22:52:58.509 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.1509999999999998e-06 2024-09-20T22:52:58.509 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.510 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.510 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:52:58.510 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:57.170361+0000", 2024-09-20T22:52:58.510 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0492999999999999e-05 2024-09-20T22:52:58.510 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.510 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.510 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:52:58.510 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:57.170381+0000", 2024-09-20T22:52:58.510 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.0509999999999998e-05 2024-09-20T22:52:58.510 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.510 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.510 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:52:58.510 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:57.170393+0000", 2024-09-20T22:52:58.511 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2009e-05 2024-09-20T22:52:58.511 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.511 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.511 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:52:58.511 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:57.170465+0000", 2024-09-20T22:52:58.511 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.1927000000000004e-05 2024-09-20T22:52:58.511 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.511 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.511 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op must wait for map", 2024-09-20T22:52:58.511 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:57.170473+0000", 2024-09-20T22:52:58.511 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.1890000000000007e-06 2024-09-20T22:52:58.511 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.511 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.512 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:52:58.512 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:57.172855+0000", 2024-09-20T22:52:58.512 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002382168 2024-09-20T22:52:58.512 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.512 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.512 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:52:58.512 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:57.172967+0000", 2024-09-20T22:52:58.512 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00011108299999999999 2024-09-20T22:52:58.512 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.512 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.512 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 1,6", 2024-09-20T22:52:58.512 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:57.181533+0000", 2024-09-20T22:52:58.512 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0085667499999999997 2024-09-20T22:52:58.512 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.513 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.513 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:52:58.513 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:57.191599+0000", 2024-09-20T22:52:58.513 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010065795000000001 2024-09-20T22:52:58.513 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.513 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.513 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:52:58.513 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:57.193469+0000", 2024-09-20T22:52:58.513 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0018695199999999999 2024-09-20T22:52:58.513 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.513 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.513 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:52:58.513 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:57.193550+0000", 2024-09-20T22:52:58.514 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.1891000000000006e-05 2024-09-20T22:52:58.514 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.514 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.514 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:52:58.514 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:57.193625+0000", 2024-09-20T22:52:58.514 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.4679999999999996e-05 2024-09-20T22:52:58.514 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.514 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.514 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:52:58.514 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:57.193681+0000", 2024-09-20T22:52:58.514 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.6320000000000003e-05 2024-09-20T22:52:58.514 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:52:58.514 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:52:58.514 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:52:58.515 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.515 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.515 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:2853 7.10 7:0f524219:::smithi08477203-50:head [copy-from ver 114] snapc 68=[68,67] ondisk+write+known_if_redirected+supports_pool_eio e303)", 2024-09-20T22:52:58.515 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:46:23.285507+0000", 2024-09-20T22:52:58.515 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 395.144297118, 2024-09-20T22:52:58.515 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.023440631, 2024-09-20T22:52:58.515 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:52:58.515 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:52:58.515 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:52:58.515 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:52:58.515 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:52:58.515 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:52:58.515 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 2853 2024-09-20T22:52:58.515 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.515 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:52:58.516 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.516 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:52:58.516 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.285507+0000", 2024-09-20T22:52:58.516 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:52:58.516 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.516 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.516 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:52:58.516 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.285507+0000", 2024-09-20T22:52:58.516 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:52:58.516 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.516 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.516 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:52:58.516 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.285508+0000", 2024-09-20T22:52:58.516 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.491e-06 2024-09-20T22:52:58.517 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.517 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.517 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:52:58.517 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.285514+0000", 2024-09-20T22:52:58.517 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.502e-06 2024-09-20T22:52:58.517 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.517 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.517 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:52:58.517 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.285526+0000", 2024-09-20T22:52:58.517 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2639e-05 2024-09-20T22:52:58.517 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.517 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.517 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:52:58.517 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.285535+0000", 2024-09-20T22:52:58.518 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.4689999999999997e-06 2024-09-20T22:52:58.518 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.518 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.518 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:52:58.518 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.285591+0000", 2024-09-20T22:52:58.518 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.6407999999999999e-05 2024-09-20T22:52:58.518 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.518 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.518 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:52:58.518 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.285681+0000", 2024-09-20T22:52:58.518 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.9845000000000005e-05 2024-09-20T22:52:58.518 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.518 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.518 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 1,2", 2024-09-20T22:52:58.519 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.293719+0000", 2024-09-20T22:52:58.519 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0080383950000000003 2024-09-20T22:52:58.519 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.519 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.519 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:52:58.519 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.307398+0000", 2024-09-20T22:52:58.519 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013678074 2024-09-20T22:52:58.519 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.519 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.519 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:52:58.519 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.308451+0000", 2024-09-20T22:52:58.519 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001052975 2024-09-20T22:52:58.519 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.519 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.520 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:52:58.520 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.308811+0000", 2024-09-20T22:52:58.520 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00036041399999999998 2024-09-20T22:52:58.520 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.520 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.520 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:52:58.520 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.308878+0000", 2024-09-20T22:52:58.520 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.7114999999999997e-05 2024-09-20T22:52:58.520 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.520 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.520 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:52:58.520 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.308947+0000", 2024-09-20T22:52:58.520 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.9303999999999999e-05 2024-09-20T22:52:58.520 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:52:58.520 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:52:58.521 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:52:58.521 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.521 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.521 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:3136 7.a 7:5122292a:::smithi08477203-33:head [copy-from ver 385] snapc 77=[77,76,75,74,73,70,6b] ondisk+write+known_if_redirected+supports_pool_eio e328)", 2024-09-20T22:52:58.521 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:46:48.210350+0000", 2024-09-20T22:52:58.521 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 370.21945398700001, 2024-09-20T22:52:58.521 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.030150626999999999, 2024-09-20T22:52:58.521 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:52:58.521 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:52:58.521 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:52:58.521 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:52:58.521 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:52:58.521 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:52:58.521 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 3136 2024-09-20T22:52:58.522 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.522 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:52:58.522 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.522 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:52:58.522 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.210350+0000", 2024-09-20T22:52:58.522 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:52:58.522 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.522 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.522 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:52:58.522 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.210350+0000", 2024-09-20T22:52:58.522 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:52:58.522 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.522 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.523 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:52:58.523 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.210352+0000", 2024-09-20T22:52:58.523 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.889e-06 2024-09-20T22:52:58.523 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.523 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.523 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:52:58.523 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.210360+0000", 2024-09-20T22:52:58.523 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.8890000000000002e-06 2024-09-20T22:52:58.523 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.523 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.523 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:52:58.523 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.210372+0000", 2024-09-20T22:52:58.523 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2662e-05 2024-09-20T22:52:58.523 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.524 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.524 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:52:58.524 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.210381+0000", 2024-09-20T22:52:58.524 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.8589999999999994e-06 2024-09-20T22:52:58.524 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.524 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.524 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:52:58.524 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.210470+0000", 2024-09-20T22:52:58.524 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.9034000000000002e-05 2024-09-20T22:52:58.524 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.524 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.524 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:52:58.524 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.210568+0000", 2024-09-20T22:52:58.524 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.7919999999999998e-05 2024-09-20T22:52:58.524 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.525 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.525 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 2,5", 2024-09-20T22:52:58.525 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.218491+0000", 2024-09-20T22:52:58.525 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0079229490000000003 2024-09-20T22:52:58.525 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.525 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.525 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:52:58.525 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.224053+0000", 2024-09-20T22:52:58.525 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0055624330000000003 2024-09-20T22:52:58.525 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.525 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.525 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:52:58.525 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.227143+0000", 2024-09-20T22:52:58.525 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0030896360000000002 2024-09-20T22:52:58.526 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.526 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.526 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:52:58.526 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.240355+0000", 2024-09-20T22:52:58.526 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013211624999999999 2024-09-20T22:52:58.526 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.526 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.526 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:52:58.526 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.240439+0000", 2024-09-20T22:52:58.526 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.4009000000000002e-05 2024-09-20T22:52:58.526 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.526 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.526 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:52:58.526 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.240500+0000", 2024-09-20T22:52:58.526 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.1722000000000003e-05 2024-09-20T22:52:58.527 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:52:58.527 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:52:58.527 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:52:58.527 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.527 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.527 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:3235 7.1 7:84a866f5:::smithi08477203-27:head [copy-from ver 114] snapc 7e=[7e,7d,7c,7b,7a,79,78,75,74,73,6b] ondisk+write+known_if_redirected+supports_pool_eio e338)", 2024-09-20T22:52:58.527 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:46:58.180892+0000", 2024-09-20T22:52:58.527 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 360.24891176400001, 2024-09-20T22:52:58.527 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.044574093000000002, 2024-09-20T22:52:58.527 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:52:58.527 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:52:58.527 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:52:58.527 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:52:58.527 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:52:58.528 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:52:58.528 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 3235 2024-09-20T22:52:58.528 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.528 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:52:58.528 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.528 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:52:58.528 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.180892+0000", 2024-09-20T22:52:58.528 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:52:58.528 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.528 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.528 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:52:58.528 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.180892+0000", 2024-09-20T22:52:58.528 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:52:58.528 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.529 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.529 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:52:58.529 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.180894+0000", 2024-09-20T22:52:58.529 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.1789999999999998e-06 2024-09-20T22:52:58.529 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.529 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.529 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:52:58.529 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.180915+0000", 2024-09-20T22:52:58.529 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.0859000000000001e-05 2024-09-20T22:52:58.529 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.529 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.529 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:52:58.529 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.180937+0000", 2024-09-20T22:52:58.529 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.2357999999999998e-05 2024-09-20T22:52:58.530 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.530 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.530 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:52:58.530 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.180947+0000", 2024-09-20T22:52:58.530 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.8090000000000004e-06 2024-09-20T22:52:58.530 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.530 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.530 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:52:58.530 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.181042+0000", 2024-09-20T22:52:58.530 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.4327999999999995e-05 2024-09-20T22:52:58.530 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.530 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.530 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op must wait for map", 2024-09-20T22:52:58.530 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.181049+0000", 2024-09-20T22:52:58.530 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.1640000000000004e-06 2024-09-20T22:52:58.531 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.531 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.531 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:52:58.531 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.185933+0000", 2024-09-20T22:52:58.531 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.004884645 2024-09-20T22:52:58.531 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.531 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.531 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:52:58.531 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.186177+0000", 2024-09-20T22:52:58.531 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000243107 2024-09-20T22:52:58.531 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.531 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.531 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 1,6", 2024-09-20T22:52:58.532 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.204218+0000", 2024-09-20T22:52:58.532 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.018041805000000001 2024-09-20T22:52:58.532 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.532 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.532 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:52:58.532 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.218275+0000", 2024-09-20T22:52:58.532 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014056528 2024-09-20T22:52:58.532 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.532 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.532 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:52:58.532 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.224283+0000", 2024-09-20T22:52:58.532 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0060084860000000004 2024-09-20T22:52:58.532 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.532 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.533 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:52:58.533 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.225379+0000", 2024-09-20T22:52:58.533 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010960830000000001 2024-09-20T22:52:58.533 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.533 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.533 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:52:58.533 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.225429+0000", 2024-09-20T22:52:58.533 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.9440000000000001e-05 2024-09-20T22:52:58.533 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.533 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.533 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:52:58.533 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.225466+0000", 2024-09-20T22:52:58.533 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.7302000000000003e-05 2024-09-20T22:52:58.533 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:52:58.534 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:52:58.534 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:52:58.534 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.534 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.534 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:3236 7.1 7:84a866f5:::smithi08477203-27:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e338)", 2024-09-20T22:52:58.534 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:46:58.180973+0000", 2024-09-20T22:52:58.534 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 360.24883114199997, 2024-09-20T22:52:58.534 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.028358540000000002, 2024-09-20T22:52:58.534 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:52:58.534 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:52:58.534 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-20T22:52:58.534 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:52:58.534 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:52:58.535 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:52:58.535 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 3236 2024-09-20T22:52:58.535 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.535 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:52:58.535 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.535 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:52:58.535 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.180973+0000", 2024-09-20T22:52:58.535 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:52:58.535 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.535 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.535 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:52:58.535 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.180973+0000", 2024-09-20T22:52:58.535 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:52:58.535 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.536 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.536 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:52:58.536 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.180973+0000", 2024-09-20T22:52:58.536 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.9100000000000004e-07 2024-09-20T22:52:58.536 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.536 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.536 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:52:58.536 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.180976+0000", 2024-09-20T22:52:58.536 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.118e-06 2024-09-20T22:52:58.536 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.536 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.536 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:52:58.536 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.180984+0000", 2024-09-20T22:52:58.536 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.43e-06 2024-09-20T22:52:58.537 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.537 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.537 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:52:58.537 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.180990+0000", 2024-09-20T22:52:58.537 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.1109999999999996e-06 2024-09-20T22:52:58.537 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.537 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.537 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:52:58.537 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.181102+0000", 2024-09-20T22:52:58.537 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000112193 2024-09-20T22:52:58.537 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.537 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.537 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting_for_map not empty", 2024-09-20T22:52:58.537 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.181107+0000", 2024-09-20T22:52:58.538 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.0490000000000003e-06 2024-09-20T22:52:58.538 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.538 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.538 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:52:58.538 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.186302+0000", 2024-09-20T22:52:58.538 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0051950240000000003 2024-09-20T22:52:58.538 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.538 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.538 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for blocked object", 2024-09-20T22:52:58.538 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.186344+0000", 2024-09-20T22:52:58.538 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.2168e-05 2024-09-20T22:52:58.538 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.538 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.539 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:52:58.539 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.209115+0000", 2024-09-20T22:52:58.539 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.022770291000000002 2024-09-20T22:52:58.539 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.539 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.539 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:52:58.539 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.209217+0000", 2024-09-20T22:52:58.539 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000102192 2024-09-20T22:52:58.539 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.539 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.539 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:52:58.539 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.209331+0000", 2024-09-20T22:52:58.539 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000114373 2024-09-20T22:52:58.539 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:52:58.540 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:52:58.540 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:52:58.540 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.540 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.540 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:4534 7.12 7:4b06cbed:::smithi08477203-49:head [copy-from ver 722] snapc bb=[bb,ba,b9,b8,b7,b5,b4,b2,b1,a7,a6,a5,9e,9c,95,87,86] ondisk+write+known_if_redirected+supports_pool_eio e456)", 2024-09-20T22:52:58.540 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:48:56.230481+0000", 2024-09-20T22:52:58.540 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 242.199323096, 2024-09-20T22:52:58.540 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.028199295999999999, 2024-09-20T22:52:58.540 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:52:58.540 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:52:58.540 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:52:58.540 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:52:58.540 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:52:58.540 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:52:58.541 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 4534 2024-09-20T22:52:58.541 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.541 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:52:58.541 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.541 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:52:58.541 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:56.230481+0000", 2024-09-20T22:52:58.541 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:52:58.541 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.541 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.541 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:52:58.541 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:56.230481+0000", 2024-09-20T22:52:58.541 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:52:58.541 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.541 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.542 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:52:58.542 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:56.230482+0000", 2024-09-20T22:52:58.542 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.674e-06 2024-09-20T22:52:58.542 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.542 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.542 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:52:58.542 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:56.230497+0000", 2024-09-20T22:52:58.542 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.4178e-05 2024-09-20T22:52:58.542 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.542 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.542 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:52:58.542 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:56.230517+0000", 2024-09-20T22:52:58.542 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.0336e-05 2024-09-20T22:52:58.542 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.542 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.543 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:52:58.543 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:56.230527+0000", 2024-09-20T22:52:58.543 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.6039999999999999e-06 2024-09-20T22:52:58.543 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.543 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.543 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:52:58.543 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:56.230631+0000", 2024-09-20T22:52:58.543 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00010445 2024-09-20T22:52:58.543 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.543 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.543 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op must wait for map", 2024-09-20T22:52:58.543 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:56.230639+0000", 2024-09-20T22:52:58.543 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.7759999999999997e-06 2024-09-20T22:52:58.544 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.544 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.544 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:52:58.544 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:56.237931+0000", 2024-09-20T22:52:58.544 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0072925630000000002 2024-09-20T22:52:58.544 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.544 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.544 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:52:58.544 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:56.238030+0000", 2024-09-20T22:52:58.544 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.8592999999999996e-05 2024-09-20T22:52:58.544 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.544 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.544 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 1,2", 2024-09-20T22:52:58.544 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:56.244691+0000", 2024-09-20T22:52:58.545 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0066614680000000003 2024-09-20T22:52:58.545 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.545 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.545 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:52:58.545 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:56.258211+0000", 2024-09-20T22:52:58.545 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013519721 2024-09-20T22:52:58.545 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.545 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.545 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:52:58.545 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:56.258404+0000", 2024-09-20T22:52:58.545 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00019257000000000001 2024-09-20T22:52:58.545 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.545 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.545 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:52:58.546 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:56.258595+0000", 2024-09-20T22:52:58.546 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000191387 2024-09-20T22:52:58.546 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.546 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.546 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:52:58.546 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:56.258635+0000", 2024-09-20T22:52:58.546 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.9918e-05 2024-09-20T22:52:58.546 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.546 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.546 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:52:58.546 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:56.258680+0000", 2024-09-20T22:52:58.546 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.5058000000000001e-05 2024-09-20T22:52:58.546 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:52:58.546 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:52:58.547 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:52:58.547 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.547 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.547 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:5677 7.2 7:4fa4b2b3:::smithi08477203-21:head [copy-from ver 1213] snapc fa=[fa,f9,f8,f7,f5,f4,f2,f1,ef,ed,ec,ea,e6,e4,e2,df,db,da,d7,d4,d3,ce,c5,c2,bf,b4,b2,a7] ondisk+write+known_if_redirected+supports_pool_eio e572)", 2024-09-20T22:52:58.547 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:50:52.384255+0000", 2024-09-20T22:52:58.547 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 126.045548348, 2024-09-20T22:52:58.547 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.027330337, 2024-09-20T22:52:58.547 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:52:58.547 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:52:58.547 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:52:58.547 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:52:58.547 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:52:58.547 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:52:58.548 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 5677 2024-09-20T22:52:58.548 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.548 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:52:58.548 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.548 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:52:58.548 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:52.384255+0000", 2024-09-20T22:52:58.548 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:52:58.548 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.548 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.548 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:52:58.548 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:52.384255+0000", 2024-09-20T22:52:58.548 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:52:58.548 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.548 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.549 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:52:58.549 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:52.384258+0000", 2024-09-20T22:52:58.549 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.0600000000000002e-06 2024-09-20T22:52:58.549 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.549 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.549 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:52:58.549 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:52.384267+0000", 2024-09-20T22:52:58.549 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.3519999999999996e-06 2024-09-20T22:52:58.549 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.549 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.549 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:52:58.549 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:52.384288+0000", 2024-09-20T22:52:58.549 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.1039e-05 2024-09-20T22:52:58.549 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.549 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.550 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:52:58.550 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:52.384298+0000", 2024-09-20T22:52:58.550 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.6630000000000005e-06 2024-09-20T22:52:58.550 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.550 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.550 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:52:58.550 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:52.384381+0000", 2024-09-20T22:52:58.550 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.3372000000000001e-05 2024-09-20T22:52:58.550 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.550 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.550 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:52:58.550 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:52.384455+0000", 2024-09-20T22:52:58.550 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.4414999999999998e-05 2024-09-20T22:52:58.550 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.551 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.551 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 1,2", 2024-09-20T22:52:58.551 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:52.398437+0000", 2024-09-20T22:52:58.551 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013981959 2024-09-20T22:52:58.551 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.551 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.551 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:52:58.551 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:52.410532+0000", 2024-09-20T22:52:58.551 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.012094898 2024-09-20T22:52:58.551 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.551 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.551 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:52:58.551 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:52.411064+0000", 2024-09-20T22:52:58.551 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00053131999999999995 2024-09-20T22:52:58.552 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.552 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.552 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:52:58.552 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:52.411446+0000", 2024-09-20T22:52:58.552 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00038282300000000002 2024-09-20T22:52:58.552 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.552 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.552 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:52:58.552 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:52.411525+0000", 2024-09-20T22:52:58.552 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.8339999999999999e-05 2024-09-20T22:52:58.552 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.552 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.552 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:52:58.552 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:52.411586+0000", 2024-09-20T22:52:58.553 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.1095999999999999e-05 2024-09-20T22:52:58.553 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:52:58.553 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:52:58.553 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:52:58.553 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.553 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.553 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:6378 7.2 7:4d20232a:::smithi08477203-34:head [copy-from ver 1800] snapc 11a=[11a,119,118,117,116,115,114,113,112,111,110,10e,10c,107,105,104,103,fd,fb,fa,f8,f5,f2,ed,d7,c2,bf] ondisk+write+known_if_redirected+supports_pool_eio e639)", 2024-09-20T22:52:58.553 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:51:59.270245+0000", 2024-09-20T22:52:58.553 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 59.159558482999998, 2024-09-20T22:52:58.553 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.024049938999999999, 2024-09-20T22:52:58.553 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:52:58.553 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:52:58.553 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:52:58.553 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:52:58.554 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:52:58.554 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:52:58.554 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 6378 2024-09-20T22:52:58.554 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.554 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:52:58.554 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.554 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:52:58.554 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:59.270245+0000", 2024-09-20T22:52:58.554 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:52:58.554 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.554 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.554 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:52:58.554 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:59.270245+0000", 2024-09-20T22:52:58.554 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:52:58.555 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.555 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.555 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:52:58.555 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:59.270248+0000", 2024-09-20T22:52:58.555 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.8159999999999998e-06 2024-09-20T22:52:58.555 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.555 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.555 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:52:58.555 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:59.270262+0000", 2024-09-20T22:52:58.555 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3429e-05 2024-09-20T22:52:58.555 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.555 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.555 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:52:58.555 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:59.270279+0000", 2024-09-20T22:52:58.555 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.7269e-05 2024-09-20T22:52:58.556 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.556 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.556 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:52:58.556 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:59.270288+0000", 2024-09-20T22:52:58.556 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.3610000000000007e-06 2024-09-20T22:52:58.556 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.556 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.556 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:52:58.556 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:59.270383+0000", 2024-09-20T22:52:58.556 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.5199999999999997e-05 2024-09-20T22:52:58.556 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.556 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.556 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op must wait for map", 2024-09-20T22:52:58.557 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:59.270393+0000", 2024-09-20T22:52:58.557 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.4190000000000008e-06 2024-09-20T22:52:58.557 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.557 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.557 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:52:58.557 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:59.272509+0000", 2024-09-20T22:52:58.557 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0021163839999999998 2024-09-20T22:52:58.557 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.557 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.557 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:52:58.557 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:59.272677+0000", 2024-09-20T22:52:58.557 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00016751 2024-09-20T22:52:58.557 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.557 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.558 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 1,2,7", 2024-09-20T22:52:58.558 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:59.278535+0000", 2024-09-20T22:52:58.558 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0058579849999999996 2024-09-20T22:52:58.558 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.558 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.558 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:52:58.558 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:59.285175+0000", 2024-09-20T22:52:58.558 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0066406829999999997 2024-09-20T22:52:58.558 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.558 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.558 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:52:58.558 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:59.293955+0000", 2024-09-20T22:52:58.558 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0087799990000000001 2024-09-20T22:52:58.558 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.559 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.559 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:52:58.559 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:59.294123+0000", 2024-09-20T22:52:58.559 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00016759699999999999 2024-09-20T22:52:58.559 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.559 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.559 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:52:58.559 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:59.294156+0000", 2024-09-20T22:52:58.559 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.2756000000000003e-05 2024-09-20T22:52:58.559 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.559 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.559 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:52:58.559 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:59.294197+0000", 2024-09-20T22:52:58.559 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.1712000000000003e-05 2024-09-20T22:52:58.560 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:52:58.560 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:52:58.560 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:52:58.560 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:59.294295+0000", 2024-09-20T22:52:58.560 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.7819000000000007e-05 2024-09-20T22:52:58.560 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:52:58.560 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:52:58.560 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:52:58.560 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:52:58.560 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:52:58.560 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-20T22:52:58.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:58.562 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:58.562 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:52:58.663 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:52:58.685 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:58.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:58.686 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:52:58.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:58 smithi154 ceph-mon[33264]: osdmap e697: 8 total, 8 up, 7 in 2024-09-20T22:52:58.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:58 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/258403835' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-09-20T22:52:58.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:58 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-09-20T22:52:58.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:58 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/3807117380' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:52:58.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:58 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-20T22:52:58.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:58 smithi154 ceph-mon[33264]: osdmap e698: 8 total, 8 up, 7 in 2024-09-20T22:52:58.787 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:52:58.809 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:58.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:58.810 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:52:58.910 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:52:58.932 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:58.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:58.933 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:52:59.011 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd unset noscrub 2024-09-20T22:52:59.034 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:52:59.076 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:59.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:59.077 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:52:59.177 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:52:59.199 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:59.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:59.200 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:52:59.289 INFO:tasks.rados.rados.0.smithi084.stdout:2754: done (3 left) 2024-09-20T22:52:59.290 INFO:tasks.rados.rados.0.smithi084.stdout:2755: done (2 left) 2024-09-20T22:52:59.290 INFO:tasks.rados.rados.0.smithi084.stdout:2757: done (1 left) 2024-09-20T22:52:59.290 INFO:tasks.rados.rados.0.smithi084.stdout:2758: done (0 left) 2024-09-20T22:52:59.290 INFO:tasks.rados.rados.0.smithi084.stdout:2759: delete oid 40 current snap is 313 2024-09-20T22:52:59.294 INFO:tasks.rados.rados.0.smithi084.stdout:2759: done (0 left) 2024-09-20T22:52:59.294 INFO:tasks.rados.rados.0.smithi084.stdout:2760: read oid 22 snap 294 2024-09-20T22:52:59.294 INFO:tasks.rados.rados.0.smithi084.stdout:2760: expect deleted 2024-09-20T22:52:59.295 INFO:tasks.rados.rados.0.smithi084.stdout:2761: read oid 18 snap 302 2024-09-20T22:52:59.295 INFO:tasks.rados.rados.0.smithi084.stdout:2761: expect (ObjNum 657 snap 199 seq_num 657) 2024-09-20T22:52:59.295 INFO:tasks.rados.rados.0.smithi084.stdout:2762: snap_remove snap 189 2024-09-20T22:52:59.301 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:52:59.322 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:59.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:59.323 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:52:59.424 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:52:59.444 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:52:59.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:59.445 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:52:59.546 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:52:59.569 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:59.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:59.570 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:52:59.586 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:59 smithi084 ceph-mon[29774]: pgmap v948: 158 pgs: 4 active+clean+snaptrim, 154 active+clean; 659 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail 2024-09-20T22:52:59.586 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:52:59 smithi084 ceph-mon[29774]: osdmap e699: 8 total, 8 up, 7 in 2024-09-20T22:52:59.586 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:59 smithi084 ceph-mon[40024]: pgmap v948: 158 pgs: 4 active+clean+snaptrim, 154 active+clean; 659 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail 2024-09-20T22:52:59.586 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:52:59 smithi084 ceph-mon[40024]: osdmap e699: 8 total, 8 up, 7 in 2024-09-20T22:52:59.587 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd reweight 0 1 2024-09-20T22:52:59.670 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:52:59.692 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:59.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:59.693 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:52:59.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:59 smithi154 ceph-mon[33264]: pgmap v948: 158 pgs: 4 active+clean+snaptrim, 154 active+clean; 659 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail 2024-09-20T22:52:59.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:52:59 smithi154 ceph-mon[33264]: osdmap e699: 8 total, 8 up, 7 in 2024-09-20T22:52:59.754 INFO:journalctl@ceph.iscsi.iscsi.a.smithi154.stdout:Sep 20 22:52:59 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-iscsi-iscsi-a[59730]: debug there is no tcmu-runner data available 2024-09-20T22:52:59.795 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:52:59.817 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:59.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:59.818 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:52:59.919 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:52:59.942 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:52:59.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:52:59.943 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:53:00.044 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:53:00.067 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:00.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:00.068 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:53:00.168 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:53:00.192 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:00.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:00.193 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:53:00.293 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:53:00.294 INFO:tasks.rados.rados.0.smithi084.stdout:2762: done (2 left) 2024-09-20T22:53:00.294 INFO:tasks.rados.rados.0.smithi084.stdout:2763: snap_remove snap 292 2024-09-20T22:53:00.319 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:00.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:00.320 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:53:00.416 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:00 smithi084 ceph-mon[40024]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:53:00.416 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:00 smithi084 ceph-mon[40024]: osdmap e700: 8 total, 8 up, 7 in 2024-09-20T22:53:00.417 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:00 smithi084 ceph-mon[29774]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:53:00.417 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:00 smithi084 ceph-mon[29774]: osdmap e700: 8 total, 8 up, 7 in 2024-09-20T22:53:00.421 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:53:00.443 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:00.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:00.444 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:53:00.449 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-20T22:53:00.545 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:53:00.566 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:00.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:00.567 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:53:00.668 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:53:00.689 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:00.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:00.690 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:53:00.702 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:53:00.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:00 smithi154 ceph-mon[33264]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:53:00.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:00 smithi154 ceph-mon[33264]: osdmap e700: 8 total, 8 up, 7 in 2024-09-20T22:53:00.792 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:53:00.799 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:53:00.813 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:00.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:00.814 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:53:00.831 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:53:00.915 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:53:00.937 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:00.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:00.937 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:53:01.038 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:53:01.061 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:01.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:01.062 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:53:01.153 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-20T22:53:01.153 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [], 2024-09-20T22:53:01.153 INFO:teuthology.orchestra.run.smithi154.stdout: "num_ops": 0 2024-09-20T22:53:01.153 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-20T22:53:01.162 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:53:01.185 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:01.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:01.186 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:53:01.287 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:53:01.290 INFO:tasks.rados.rados.0.smithi084.stdout:2763: done (2 left) 2024-09-20T22:53:01.291 INFO:tasks.rados.rados.0.smithi084.stdout:2764: delete oid 45 current snap is 313 2024-09-20T22:53:01.291 INFO:tasks.rados.rados.0.smithi084.stdout:2761: expect (ObjNum 657 snap 199 seq_num 657) 2024-09-20T22:53:01.315 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:01.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:01.316 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:53:01.416 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:53:01.424 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:01 smithi154 ceph-mon[33264]: pgmap v951: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 663 MiB data, 1.8 GiB used, 624 GiB / 626 GiB avail; 4.9 MiB/s rd, 3.9 MiB/s wr, 20 op/s 2024-09-20T22:53:01.424 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:01 smithi154 ceph-mon[33264]: osdmap e701: 8 total, 8 up, 7 in 2024-09-20T22:53:01.439 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:01.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:01.440 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:53:01.535 INFO:tasks.rados.rados.0.smithi084.stdout:2760: done (2 left) 2024-09-20T22:53:01.535 INFO:tasks.rados.rados.0.smithi084.stdout:2761: done (1 left) 2024-09-20T22:53:01.535 INFO:tasks.rados.rados.0.smithi084.stdout:2764: done (0 left) 2024-09-20T22:53:01.535 INFO:tasks.rados.rados.0.smithi084.stdout:2765: read oid 28 snap -1 2024-09-20T22:53:01.535 INFO:tasks.rados.rados.0.smithi084.stdout:2765: expect deleted 2024-09-20T22:53:01.535 INFO:tasks.rados.rados.0.smithi084.stdout:2766: write (excl) oid 23 current snap is 313 2024-09-20T22:53:01.536 INFO:tasks.rados.rados.0.smithi084.stdout:2766: seq_num 957 ranges {484299=557418,1684797=399502} 2024-09-20T22:53:01.540 INFO:tasks.rados.rados.0.smithi084.stdout:2766: writing smithi08477203-23 from 484299 to 1041717 tid 1 2024-09-20T22:53:01.540 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:53:01.546 INFO:tasks.rados.rados.0.smithi084.stdout:2766: writing smithi08477203-23 from 1684797 to 2084299 tid 2 2024-09-20T22:53:01.546 INFO:tasks.rados.rados.0.smithi084.stdout:2767: snap_remove snap 279 2024-09-20T22:53:01.562 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:01.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:01.563 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:53:01.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:01 smithi084 ceph-mon[29774]: pgmap v951: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 663 MiB data, 1.8 GiB used, 624 GiB / 626 GiB avail; 4.9 MiB/s rd, 3.9 MiB/s wr, 20 op/s 2024-09-20T22:53:01.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:01 smithi084 ceph-mon[29774]: osdmap e701: 8 total, 8 up, 7 in 2024-09-20T22:53:01.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:01 smithi084 ceph-mon[40024]: pgmap v951: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 663 MiB data, 1.8 GiB used, 624 GiB / 626 GiB avail; 4.9 MiB/s rd, 3.9 MiB/s wr, 20 op/s 2024-09-20T22:53:01.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:01 smithi084 ceph-mon[40024]: osdmap e701: 8 total, 8 up, 7 in 2024-09-20T22:53:01.663 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:53:01.684 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:01.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:01.685 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:53:01.786 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:53:01.807 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:01.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:01.808 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:53:01.878 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.4 dump_blocked_ops 2024-09-20T22:53:01.909 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:53:01.942 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:01.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:01.943 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:53:02.044 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:53:02.065 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:02.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:02.066 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:53:02.132 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:53:02.167 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:53:02.188 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:02.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:02.189 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:53:02.289 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:53:02.311 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:02.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:02.312 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:53:02.329 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is unset 2024-09-20T22:53:02.335 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.0 to 1 (10000) 2024-09-20T22:53:02.336 INFO:tasks.rados.rados.0.smithi084.stdout:2767: done (2 left) 2024-09-20T22:53:02.336 INFO:tasks.rados.rados.0.smithi084.stdout:2768: write oid 38 current snap is 313 2024-09-20T22:53:02.336 INFO:tasks.rados.rados.0.smithi084.stdout:2768: seq_num 958 ranges {501106=749020,1743279=357827} 2024-09-20T22:53:02.341 INFO:tasks.rados.rados.0.smithi084.stdout:2768: writing smithi08477203-38 from 501106 to 1250126 tid 1 2024-09-20T22:53:02.347 INFO:tasks.rados.rados.0.smithi084.stdout:2768: writing smithi08477203-38 from 1743279 to 2101106 tid 2 2024-09-20T22:53:02.347 INFO:tasks.rados.rados.0.smithi084.stdout:2769: snap_create 2024-09-20T22:53:02.347 INFO:tasks.rados.rados.0.smithi084.stdout:2766: finishing write tid 1 to smithi08477203-23 2024-09-20T22:53:02.347 INFO:tasks.rados.rados.0.smithi084.stdout:2766: finishing write tid 2 to smithi08477203-23 2024-09-20T22:53:02.347 INFO:tasks.rados.rados.0.smithi084.stdout:2766: finishing write tid 3 to smithi08477203-23 2024-09-20T22:53:02.347 INFO:tasks.rados.rados.0.smithi084.stdout:2766: finishing write tid 4 to smithi08477203-23 2024-09-20T22:53:02.347 INFO:tasks.rados.rados.0.smithi084.stdout:2766: oid 23 updating version 0 to 1252 2024-09-20T22:53:02.347 INFO:tasks.rados.rados.0.smithi084.stdout:2766: oid 23 version 1252 is already newer than 1251 2024-09-20T22:53:02.347 INFO:tasks.rados.rados.0.smithi084.stdout:2766: oid 23 version 1252 is already newer than 1250 2024-09-20T22:53:02.347 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 23 v 1252 (ObjNum 957 snap 313 seq_num 957) dirty exists 2024-09-20T22:53:02.347 INFO:tasks.rados.rados.0.smithi084.stdout:2766: left oid 23 (ObjNum 957 snap 313 seq_num 957) 2024-09-20T22:53:02.350 INFO:tasks.rados.rados.0.smithi084.stdout:2768: finishing write tid 1 to smithi08477203-38 2024-09-20T22:53:02.352 INFO:tasks.rados.rados.0.smithi084.stdout:2768: finishing write tid 2 to smithi08477203-38 2024-09-20T22:53:02.353 INFO:tasks.rados.rados.0.smithi084.stdout:2768: finishing write tid 3 to smithi08477203-38 2024-09-20T22:53:02.353 INFO:tasks.rados.rados.0.smithi084.stdout:2768: finishing write tid 4 to smithi08477203-38 2024-09-20T22:53:02.353 INFO:tasks.rados.rados.0.smithi084.stdout:2768: oid 38 updating version 0 to 1376 2024-09-20T22:53:02.353 INFO:tasks.rados.rados.0.smithi084.stdout:2768: oid 38 version 1376 is already newer than 1375 2024-09-20T22:53:02.353 INFO:tasks.rados.rados.0.smithi084.stdout:2768: oid 38 version 1376 is already newer than 1374 2024-09-20T22:53:02.353 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 38 v 1376 (ObjNum 958 snap 313 seq_num 958) dirty exists 2024-09-20T22:53:02.353 INFO:tasks.rados.rados.0.smithi084.stdout:2768: left oid 38 (ObjNum 958 snap 313 seq_num 958) 2024-09-20T22:53:02.413 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:53:02.436 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:02.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:02.436 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:53:02.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:02 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/3544368541' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-09-20T22:53:02.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:02 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-09-20T22:53:02.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:02 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/379049017' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:53:02.538 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:53:02.559 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:02.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:02.560 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:53:02.584 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:02 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/3544368541' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-09-20T22:53:02.584 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:02 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-09-20T22:53:02.584 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:02 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/379049017' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:53:02.584 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:02 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/3544368541' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-09-20T22:53:02.584 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:02 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-09-20T22:53:02.584 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:02 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/379049017' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:53:02.662 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:53:02.672 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-20T22:53:02.672 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [], 2024-09-20T22:53:02.672 INFO:teuthology.orchestra.run.smithi154.stdout: "complaint_time": 30, 2024-09-20T22:53:02.672 INFO:teuthology.orchestra.run.smithi154.stdout: "num_blocked_ops": 0 2024-09-20T22:53:02.672 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-20T22:53:02.683 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:02.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:02.684 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:53:02.785 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:53:02.796 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-20T22:53:02.807 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:02.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:02.808 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:53:02.909 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:53:02.930 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:02.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:02.931 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:53:03.032 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:53:03.054 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:03.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:03.055 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:53:03.156 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:53:03.176 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:03.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:03.177 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:53:03.278 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:53:03.296 INFO:tasks.rados.rados.0.smithi084.stdout:2765: done (3 left) 2024-09-20T22:53:03.296 INFO:tasks.rados.rados.0.smithi084.stdout:2766: done (2 left) 2024-09-20T22:53:03.296 INFO:tasks.rados.rados.0.smithi084.stdout:2768: done (1 left) 2024-09-20T22:53:03.296 INFO:tasks.rados.rados.0.smithi084.stdout:2769: done (0 left) 2024-09-20T22:53:03.296 INFO:tasks.rados.rados.0.smithi084.stdout:2770: read oid 20 snap -1 2024-09-20T22:53:03.296 INFO:tasks.rados.rados.0.smithi084.stdout:2770: expect (ObjNum 933 snap 308 seq_num 933) 2024-09-20T22:53:03.296 INFO:tasks.rados.rados.0.smithi084.stdout:2771: write oid 18 current snap is 314 2024-09-20T22:53:03.296 INFO:tasks.rados.rados.0.smithi084.stdout:2771: seq_num 959 ranges {517913=424269,1401761=446989} 2024-09-20T22:53:03.301 INFO:tasks.rados.rados.0.smithi084.stdout:2771: writing smithi08477203-18 from 517913 to 942182 tid 1 2024-09-20T22:53:03.303 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:03.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:03.305 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:53:03.307 INFO:tasks.rados.rados.0.smithi084.stdout:2771: writing smithi08477203-18 from 1401761 to 1848750 tid 2 2024-09-20T22:53:03.307 INFO:tasks.rados.rados.0.smithi084.stdout:2772: write oid 38 current snap is 314 2024-09-20T22:53:03.307 INFO:tasks.rados.rados.0.smithi084.stdout:2772: seq_num 960 ranges {534720=499518,1460243=674477} 2024-09-20T22:53:03.312 INFO:tasks.rados.rados.0.smithi084.stdout:2772: writing smithi08477203-38 from 534720 to 1034238 tid 1 2024-09-20T22:53:03.319 INFO:tasks.rados.rados.0.smithi084.stdout:2772: writing smithi08477203-38 from 1460243 to 2134720 tid 2 2024-09-20T22:53:03.319 INFO:tasks.rados.rados.0.smithi084.stdout:2773: delete oid 21 current snap is 314 2024-09-20T22:53:03.319 INFO:tasks.rados.rados.0.smithi084.stdout:2770: expect (ObjNum 933 snap 308 seq_num 933) 2024-09-20T22:53:03.399 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:03 smithi154 ceph-mon[33264]: pgmap v954: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 663 MiB data, 1.8 GiB used, 624 GiB / 626 GiB avail; 4.9 MiB/s rd, 3.9 MiB/s wr, 20 op/s 2024-09-20T22:53:03.399 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:03 smithi154 ceph-mon[33264]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:53:03.399 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:03 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-20T22:53:03.399 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:03 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/379049017' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T22:53:03.399 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:03 smithi154 ceph-mon[33264]: osdmap e702: 8 total, 8 up, 7 in 2024-09-20T22:53:03.400 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:03 smithi154 ceph-mon[33264]: 6.1 scrub starts 2024-09-20T22:53:03.400 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:03 smithi154 ceph-mon[33264]: 6.1 scrub ok 2024-09-20T22:53:03.400 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:03 smithi154 ceph-mon[33264]: osdmap e703: 8 total, 8 up, 7 in 2024-09-20T22:53:03.406 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:53:03.428 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.4 dump_historic_ops 2024-09-20T22:53:03.429 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:03.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:03.430 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:53:03.522 INFO:tasks.rados.rados.0.smithi084.stdout:2771: finishing write tid 1 to smithi08477203-18 2024-09-20T22:53:03.522 INFO:tasks.rados.rados.0.smithi084.stdout:2771: finishing write tid 2 to smithi08477203-18 2024-09-20T22:53:03.522 INFO:tasks.rados.rados.0.smithi084.stdout:2771: finishing write tid 3 to smithi08477203-18 2024-09-20T22:53:03.523 INFO:tasks.rados.rados.0.smithi084.stdout:2771: finishing write tid 4 to smithi08477203-18 2024-09-20T22:53:03.523 INFO:tasks.rados.rados.0.smithi084.stdout:2771: oid 18 updating version 0 to 1823 2024-09-20T22:53:03.523 INFO:tasks.rados.rados.0.smithi084.stdout:2771: oid 18 updating version 1823 to 1824 2024-09-20T22:53:03.523 INFO:tasks.rados.rados.0.smithi084.stdout:2771: oid 18 version 1824 is already newer than 1822 2024-09-20T22:53:03.523 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 18 v 1824 (ObjNum 959 snap 314 seq_num 959) dirty exists 2024-09-20T22:53:03.523 INFO:tasks.rados.rados.0.smithi084.stdout:2771: left oid 18 (ObjNum 959 snap 314 seq_num 959) 2024-09-20T22:53:03.523 INFO:tasks.rados.rados.0.smithi084.stdout:2772: finishing write tid 1 to smithi08477203-38 2024-09-20T22:53:03.523 INFO:tasks.rados.rados.0.smithi084.stdout:2772: finishing write tid 2 to smithi08477203-38 2024-09-20T22:53:03.523 INFO:tasks.rados.rados.0.smithi084.stdout:2772: finishing write tid 3 to smithi08477203-38 2024-09-20T22:53:03.524 INFO:tasks.rados.rados.0.smithi084.stdout:2772: finishing write tid 4 to smithi08477203-38 2024-09-20T22:53:03.524 INFO:tasks.rados.rados.0.smithi084.stdout:2772: oid 38 updating version 0 to 1379 2024-09-20T22:53:03.524 INFO:tasks.rados.rados.0.smithi084.stdout:2772: oid 38 version 1379 is already newer than 1378 2024-09-20T22:53:03.524 INFO:tasks.rados.rados.0.smithi084.stdout:2772: oid 38 version 1379 is already newer than 1377 2024-09-20T22:53:03.524 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 38 v 1379 (ObjNum 960 snap 314 seq_num 960) dirty exists 2024-09-20T22:53:03.524 INFO:tasks.rados.rados.0.smithi084.stdout:2772: left oid 38 (ObjNum 960 snap 314 seq_num 960) 2024-09-20T22:53:03.524 INFO:tasks.rados.rados.0.smithi084.stdout:2770: done (3 left) 2024-09-20T22:53:03.524 INFO:tasks.rados.rados.0.smithi084.stdout:2771: done (2 left) 2024-09-20T22:53:03.524 INFO:tasks.rados.rados.0.smithi084.stdout:2772: done (1 left) 2024-09-20T22:53:03.524 INFO:tasks.rados.rados.0.smithi084.stdout:2773: done (0 left) 2024-09-20T22:53:03.524 INFO:tasks.rados.rados.0.smithi084.stdout:2774: write (excl) oid 28 current snap is 314 2024-09-20T22:53:03.524 INFO:tasks.rados.rados.0.smithi084.stdout:2774: seq_num 962 ranges {568334=650016,1860854=307480} 2024-09-20T22:53:03.528 INFO:tasks.rados.rados.0.smithi084.stdout:2774: writing smithi08477203-28 from 568334 to 1218350 tid 1 2024-09-20T22:53:03.531 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:53:03.535 INFO:tasks.rados.rados.0.smithi084.stdout:2774: writing smithi08477203-28 from 1860854 to 2168334 tid 2 2024-09-20T22:53:03.535 INFO:tasks.rados.rados.0.smithi084.stdout:2775: write (excl) oid 41 current snap is 314 2024-09-20T22:53:03.535 INFO:tasks.rados.rados.0.smithi084.stdout:2774: finishing write tid 1 to smithi08477203-28 2024-09-20T22:53:03.535 INFO:tasks.rados.rados.0.smithi084.stdout:2775: seq_num 963 ranges {585141=725265,1919336=265805} 2024-09-20T22:53:03.542 INFO:tasks.rados.rados.0.smithi084.stdout:2775: writing smithi08477203-41 from 585141 to 1310406 tid 1 2024-09-20T22:53:03.548 INFO:tasks.rados.rados.0.smithi084.stdout:2775: writing smithi08477203-41 from 1919336 to 2185141 tid 2 2024-09-20T22:53:03.548 INFO:tasks.rados.rados.0.smithi084.stdout:2776: copy_from oid 8 from oid 34 current snap is 314 2024-09-20T22:53:03.548 INFO:tasks.rados.rados.0.smithi084.stdout:2774: finishing write tid 2 to smithi08477203-28 2024-09-20T22:53:03.548 INFO:tasks.rados.rados.0.smithi084.stdout:2774: finishing write tid 3 to smithi08477203-28 2024-09-20T22:53:03.548 INFO:tasks.rados.rados.0.smithi084.stdout:2774: finishing write tid 4 to smithi08477203-28 2024-09-20T22:53:03.548 INFO:tasks.rados.rados.0.smithi084.stdout:2774: oid 28 updating version 0 to 2183 2024-09-20T22:53:03.548 INFO:tasks.rados.rados.0.smithi084.stdout:2774: oid 28 version 2183 is already newer than 2182 2024-09-20T22:53:03.549 INFO:tasks.rados.rados.0.smithi084.stdout:2774: oid 28 version 2183 is already newer than 2181 2024-09-20T22:53:03.549 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 28 v 2183 (ObjNum 962 snap 314 seq_num 962) dirty exists 2024-09-20T22:53:03.549 INFO:tasks.rados.rados.0.smithi084.stdout:2774: left oid 28 (ObjNum 962 snap 314 seq_num 962) 2024-09-20T22:53:03.549 INFO:tasks.rados.rados.0.smithi084.stdout:2775: finishing write tid 1 to smithi08477203-41 2024-09-20T22:53:03.549 INFO:tasks.rados.rados.0.smithi084.stdout:2774: done (2 left) 2024-09-20T22:53:03.549 INFO:tasks.rados.rados.0.smithi084.stdout:2777: write oid 20 current snap is 314 2024-09-20T22:53:03.549 INFO:tasks.rados.rados.0.smithi084.stdout:2777: seq_num 964 ranges {601948=400514,1577818=624130} 2024-09-20T22:53:03.553 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:03.553 INFO:tasks.rados.rados.0.smithi084.stdout:2777: writing smithi08477203-20 from 601948 to 1002462 tid 1 2024-09-20T22:53:03.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:03.554 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:53:03.560 INFO:tasks.rados.rados.0.smithi084.stdout:2777: writing smithi08477203-20 from 1577818 to 2201948 tid 2 2024-09-20T22:53:03.560 INFO:tasks.rados.rados.0.smithi084.stdout:2778: read oid 21 snap -1 2024-09-20T22:53:03.560 INFO:tasks.rados.rados.0.smithi084.stdout:2778: expect deleted 2024-09-20T22:53:03.561 INFO:tasks.rados.rados.0.smithi084.stdout:2779: snap_create 2024-09-20T22:53:03.561 INFO:tasks.rados.rados.0.smithi084.stdout:2776: finishing copy_from to smithi08477203-8 2024-09-20T22:53:03.561 INFO:tasks.rados.rados.0.smithi084.stdout:2776: got expected ENOENT (src dne) 2024-09-20T22:53:03.561 INFO:tasks.rados.rados.0.smithi084.stdout:2776: finishing copy_from racing read to smithi08477203-8 2024-09-20T22:53:03.561 INFO:tasks.rados.rados.0.smithi084.stdout:2775: finishing write tid 2 to smithi08477203-41 2024-09-20T22:53:03.561 INFO:tasks.rados.rados.0.smithi084.stdout:2775: finishing write tid 3 to smithi08477203-41 2024-09-20T22:53:03.561 INFO:tasks.rados.rados.0.smithi084.stdout:2775: finishing write tid 4 to smithi08477203-41 2024-09-20T22:53:03.561 INFO:tasks.rados.rados.0.smithi084.stdout:2775: oid 41 updating version 0 to 1419 2024-09-20T22:53:03.562 INFO:tasks.rados.rados.0.smithi084.stdout:2775: oid 41 version 1419 is already newer than 1418 2024-09-20T22:53:03.562 INFO:tasks.rados.rados.0.smithi084.stdout:2775: oid 41 updating version 1419 to 1420 2024-09-20T22:53:03.562 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 41 v 1420 (ObjNum 963 snap 314 seq_num 963) dirty exists 2024-09-20T22:53:03.562 INFO:tasks.rados.rados.0.smithi084.stdout:2775: left oid 41 (ObjNum 963 snap 314 seq_num 963) 2024-09-20T22:53:03.562 INFO:tasks.rados.rados.0.smithi084.stdout:2777: finishing write tid 1 to smithi08477203-20 2024-09-20T22:53:03.563 INFO:tasks.rados.rados.0.smithi084.stdout:2777: finishing write tid 2 to smithi08477203-20 2024-09-20T22:53:03.563 INFO:tasks.rados.rados.0.smithi084.stdout:2777: finishing write tid 3 to smithi08477203-20 2024-09-20T22:53:03.563 INFO:tasks.rados.rados.0.smithi084.stdout:2777: finishing write tid 4 to smithi08477203-20 2024-09-20T22:53:03.563 INFO:tasks.rados.rados.0.smithi084.stdout:2777: oid 20 updating version 0 to 1251 2024-09-20T22:53:03.563 INFO:tasks.rados.rados.0.smithi084.stdout:2777: oid 20 updating version 1251 to 1253 2024-09-20T22:53:03.563 INFO:tasks.rados.rados.0.smithi084.stdout:2777: oid 20 version 1253 is already newer than 1252 2024-09-20T22:53:03.563 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 20 v 1253 (ObjNum 964 snap 314 seq_num 964) dirty exists 2024-09-20T22:53:03.563 INFO:tasks.rados.rados.0.smithi084.stdout:2777: left oid 20 (ObjNum 964 snap 314 seq_num 964) 2024-09-20T22:53:03.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:03 smithi084 ceph-mon[40024]: pgmap v954: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 663 MiB data, 1.8 GiB used, 624 GiB / 626 GiB avail; 4.9 MiB/s rd, 3.9 MiB/s wr, 20 op/s 2024-09-20T22:53:03.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:03 smithi084 ceph-mon[40024]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:53:03.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:03 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-20T22:53:03.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:03 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/379049017' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T22:53:03.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:03 smithi084 ceph-mon[40024]: osdmap e702: 8 total, 8 up, 7 in 2024-09-20T22:53:03.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:03 smithi084 ceph-mon[40024]: 6.1 scrub starts 2024-09-20T22:53:03.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:03 smithi084 ceph-mon[40024]: 6.1 scrub ok 2024-09-20T22:53:03.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:03 smithi084 ceph-mon[40024]: osdmap e703: 8 total, 8 up, 7 in 2024-09-20T22:53:03.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:03 smithi084 ceph-mon[29774]: pgmap v954: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 663 MiB data, 1.8 GiB used, 624 GiB / 626 GiB avail; 4.9 MiB/s rd, 3.9 MiB/s wr, 20 op/s 2024-09-20T22:53:03.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:03 smithi084 ceph-mon[29774]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:53:03.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:03 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-20T22:53:03.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:03 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/379049017' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T22:53:03.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:03 smithi084 ceph-mon[29774]: osdmap e702: 8 total, 8 up, 7 in 2024-09-20T22:53:03.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:03 smithi084 ceph-mon[29774]: 6.1 scrub starts 2024-09-20T22:53:03.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:03 smithi084 ceph-mon[29774]: 6.1 scrub ok 2024-09-20T22:53:03.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:03 smithi084 ceph-mon[29774]: osdmap e703: 8 total, 8 up, 7 in 2024-09-20T22:53:03.655 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:53:03.679 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:03.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:03.680 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:53:03.684 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:53:03.780 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:53:03.789 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd reweight 4 1 2024-09-20T22:53:03.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:03.803 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:03.803 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:53:03.880 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:53:03.904 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:53:03.926 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:03.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:03.927 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:53:04.028 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:53:04.049 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:04.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:04.050 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:53:04.136 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-20T22:53:04.136 INFO:teuthology.orchestra.run.smithi154.stdout: "size": 20, 2024-09-20T22:53:04.136 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 600, 2024-09-20T22:53:04.136 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [ 2024-09-20T22:53:04.136 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.136 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:522 7.9 e124/94 7:9970b578:::smithi08477203-15:head v 124'40, mlcod=118'36)", 2024-09-20T22:53:04.136 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:23.137032+0000", 2024-09-20T22:53:04.136 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 580.99069494299999, 2024-09-20T22:53:04.137 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0095727380000000008, 2024-09-20T22:53:04.137 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:53:04.137 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:53:04.137 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:53:04.137 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:53:04.137 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.137 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:53:04.137 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:23.137032+0000", 2024-09-20T22:53:04.137 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:53:04.137 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.137 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.137 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:53:04.137 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:23.137032+0000", 2024-09-20T22:53:04.138 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:53:04.138 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.138 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.138 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:53:04.138 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:23.137033+0000", 2024-09-20T22:53:04.138 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.034e-06 2024-09-20T22:53:04.138 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.138 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.138 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:53:04.138 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:23.140240+0000", 2024-09-20T22:53:04.139 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0032075419999999999 2024-09-20T22:53:04.139 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.139 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.139 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:53:04.139 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:23.140261+0000", 2024-09-20T22:53:04.139 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.1155999999999999e-05 2024-09-20T22:53:04.139 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.139 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.139 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:53:04.139 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:23.140271+0000", 2024-09-20T22:53:04.139 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.6970000000000005e-06 2024-09-20T22:53:04.139 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.139 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.140 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:53:04.140 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:23.140339+0000", 2024-09-20T22:53:04.140 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.7964000000000002e-05 2024-09-20T22:53:04.140 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.140 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.140 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:53:04.140 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:23.140361+0000", 2024-09-20T22:53:04.140 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.196e-05 2024-09-20T22:53:04.140 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.140 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.140 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:53:04.140 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:23.146563+0000", 2024-09-20T22:53:04.141 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0062014879999999998 2024-09-20T22:53:04.141 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.141 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.141 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:53:04.141 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:23.146604+0000", 2024-09-20T22:53:04.141 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.1897000000000002e-05 2024-09-20T22:53:04.141 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:53:04.141 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:53:04.141 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:53:04.141 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.141 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.141 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:4229 7.14 e423/421 7:2f8647a2:::smithi08477203-41:head v 423'133, mlcod=422'129)", 2024-09-20T22:53:04.142 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:48:23.296695+0000", 2024-09-20T22:53:04.142 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 280.83103157099998, 2024-09-20T22:53:04.142 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.011591664, 2024-09-20T22:53:04.142 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:53:04.142 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:53:04.142 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:53:04.142 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:53:04.142 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.142 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:53:04.142 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:23.296695+0000", 2024-09-20T22:53:04.142 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:53:04.142 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.143 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.143 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:53:04.143 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:23.296695+0000", 2024-09-20T22:53:04.143 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:53:04.143 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.143 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.143 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:53:04.143 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:23.296697+0000", 2024-09-20T22:53:04.143 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.8509999999999999e-06 2024-09-20T22:53:04.143 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.143 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.143 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:53:04.143 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:23.302610+0000", 2024-09-20T22:53:04.143 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0059131959999999999 2024-09-20T22:53:04.144 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.144 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.144 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:53:04.144 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:23.302632+0000", 2024-09-20T22:53:04.144 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.2146000000000001e-05 2024-09-20T22:53:04.144 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.144 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.144 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:53:04.144 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:23.302641+0000", 2024-09-20T22:53:04.144 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.9069999999999996e-06 2024-09-20T22:53:04.144 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.144 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.144 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:53:04.144 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:23.302711+0000", 2024-09-20T22:53:04.145 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.0038999999999998e-05 2024-09-20T22:53:04.145 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.145 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.145 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:53:04.145 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:23.302732+0000", 2024-09-20T22:53:04.145 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.0922000000000001e-05 2024-09-20T22:53:04.145 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.145 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.145 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:53:04.145 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:23.308245+0000", 2024-09-20T22:53:04.145 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0055130889999999997 2024-09-20T22:53:04.145 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.145 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.145 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:53:04.146 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:23.308287+0000", 2024-09-20T22:53:04.146 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.1514000000000003e-05 2024-09-20T22:53:04.146 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:53:04.146 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:53:04.146 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:53:04.146 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.146 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.146 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:4413 7.4 e444/404 7:24b1a612:::smithi08477203-43:head v 444'131, mlcod=435'127)", 2024-09-20T22:53:04.146 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:48:44.270681+0000", 2024-09-20T22:53:04.146 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 259.85704589099998, 2024-09-20T22:53:04.146 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0069886260000000004, 2024-09-20T22:53:04.146 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:53:04.146 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:53:04.147 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:53:04.147 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:53:04.147 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.147 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:53:04.147 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:44.270681+0000", 2024-09-20T22:53:04.147 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:53:04.147 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.147 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.147 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:53:04.147 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:44.270681+0000", 2024-09-20T22:53:04.147 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:53:04.147 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.147 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.148 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:53:04.148 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:44.270681+0000", 2024-09-20T22:53:04.148 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.4200000000000002e-07 2024-09-20T22:53:04.148 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.148 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.148 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:53:04.148 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:44.274116+0000", 2024-09-20T22:53:04.148 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0034349670000000001 2024-09-20T22:53:04.148 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.148 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.148 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:53:04.148 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:44.274133+0000", 2024-09-20T22:53:04.148 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.6718000000000002e-05 2024-09-20T22:53:04.149 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.149 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.149 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:53:04.149 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:44.274142+0000", 2024-09-20T22:53:04.149 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.7610000000000002e-06 2024-09-20T22:53:04.149 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.149 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.149 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:53:04.149 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:44.274226+0000", 2024-09-20T22:53:04.149 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.3642999999999999e-05 2024-09-20T22:53:04.149 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.149 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.149 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:53:04.150 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:44.274257+0000", 2024-09-20T22:53:04.150 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.1000999999999997e-05 2024-09-20T22:53:04.150 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.150 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.150 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:53:04.150 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:44.277618+0000", 2024-09-20T22:53:04.150 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.003360516 2024-09-20T22:53:04.150 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.150 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.150 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:53:04.150 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:44.277669+0000", 2024-09-20T22:53:04.150 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.1777999999999998e-05 2024-09-20T22:53:04.150 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:53:04.150 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:53:04.151 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:53:04.151 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.151 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.151 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(osd.0.0:936 7.9 e508/426 7:9970b578:::smithi08477203-15:d8 v 508'552, mlcod=505'547)", 2024-09-20T22:53:04.151 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:49:48.234292+0000", 2024-09-20T22:53:04.151 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 195.893434574, 2024-09-20T22:53:04.151 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0068743279999999999, 2024-09-20T22:53:04.151 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:53:04.151 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:53:04.151 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:53:04.151 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:53:04.152 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.152 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:53:04.152 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:48.234292+0000", 2024-09-20T22:53:04.152 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:53:04.152 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.152 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.152 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:53:04.152 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:48.234292+0000", 2024-09-20T22:53:04.152 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:53:04.152 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.152 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.152 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:53:04.152 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:48.234293+0000", 2024-09-20T22:53:04.153 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.0900000000000001e-07 2024-09-20T22:53:04.153 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.153 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.153 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:53:04.153 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:48.240496+0000", 2024-09-20T22:53:04.153 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0062036340000000004 2024-09-20T22:53:04.153 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.153 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.153 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:53:04.153 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:48.240508+0000", 2024-09-20T22:53:04.153 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1296e-05 2024-09-20T22:53:04.153 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.153 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.153 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:53:04.154 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:48.240516+0000", 2024-09-20T22:53:04.154 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.9710000000000004e-06 2024-09-20T22:53:04.154 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.154 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.154 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:53:04.154 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:48.240582+0000", 2024-09-20T22:53:04.154 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.614e-05 2024-09-20T22:53:04.154 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.154 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.154 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:53:04.154 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:48.240604+0000", 2024-09-20T22:53:04.154 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.2283e-05 2024-09-20T22:53:04.155 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.155 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.155 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:53:04.155 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:48.241107+0000", 2024-09-20T22:53:04.155 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00050252800000000002 2024-09-20T22:53:04.155 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.155 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.155 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:53:04.155 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:48.241166+0000", 2024-09-20T22:53:04.155 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.9867000000000001e-05 2024-09-20T22:53:04.155 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:53:04.155 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:53:04.155 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:53:04.156 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.156 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.156 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:5057 7.9 e508/426 7:9eac4560:::smithi08477203-30:head v 508'554, mlcod=505'548)", 2024-09-20T22:53:04.156 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:49:48.245783+0000", 2024-09-20T22:53:04.156 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 195.88194361699999, 2024-09-20T22:53:04.156 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.035976553000000001, 2024-09-20T22:53:04.156 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:53:04.156 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:53:04.156 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:53:04.156 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:53:04.156 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.156 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:53:04.156 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:48.245783+0000", 2024-09-20T22:53:04.157 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:53:04.157 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.157 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.157 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:53:04.157 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:48.245783+0000", 2024-09-20T22:53:04.157 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:53:04.157 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.157 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.157 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:53:04.157 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:48.245784+0000", 2024-09-20T22:53:04.157 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.08e-06 2024-09-20T22:53:04.157 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.157 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.157 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:53:04.157 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:48.279966+0000", 2024-09-20T22:53:04.158 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.034181493 2024-09-20T22:53:04.158 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.158 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.158 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:53:04.158 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:48.279981+0000", 2024-09-20T22:53:04.158 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.5092000000000001e-05 2024-09-20T22:53:04.158 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.158 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.158 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:53:04.158 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:48.280000+0000", 2024-09-20T22:53:04.158 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9493000000000001e-05 2024-09-20T22:53:04.158 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.158 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.158 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:53:04.159 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:48.280054+0000", 2024-09-20T22:53:04.159 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.3687000000000003e-05 2024-09-20T22:53:04.159 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.159 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.159 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:53:04.159 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:48.280072+0000", 2024-09-20T22:53:04.159 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.8581e-05 2024-09-20T22:53:04.159 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.159 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.159 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:53:04.159 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:48.281731+0000", 2024-09-20T22:53:04.159 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0016588569999999999 2024-09-20T22:53:04.159 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.159 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.159 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:53:04.160 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:48.281760+0000", 2024-09-20T22:53:04.160 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.8269999999999999e-05 2024-09-20T22:53:04.160 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:53:04.160 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:53:04.160 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:53:04.160 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.160 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.160 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:5414 7.8 e549/430 7:1cd232b2:::smithi08477203-35:head v 549'467, mlcod=548'459)", 2024-09-20T22:53:04.160 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:50:29.260963+0000", 2024-09-20T22:53:04.160 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 154.86676411799999, 2024-09-20T22:53:04.160 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0085861400000000008, 2024-09-20T22:53:04.160 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:53:04.160 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:53:04.161 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:53:04.161 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:53:04.161 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.161 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:53:04.161 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:29.260963+0000", 2024-09-20T22:53:04.161 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:53:04.161 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.161 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.161 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:53:04.161 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:29.260963+0000", 2024-09-20T22:53:04.161 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:53:04.161 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.161 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.161 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:53:04.161 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:29.260963+0000", 2024-09-20T22:53:04.162 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.0900000000000002e-07 2024-09-20T22:53:04.162 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.162 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.162 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:53:04.162 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:29.266981+0000", 2024-09-20T22:53:04.162 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0060184360000000003 2024-09-20T22:53:04.162 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.162 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.162 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:53:04.162 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:29.266992+0000", 2024-09-20T22:53:04.162 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0254e-05 2024-09-20T22:53:04.162 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.162 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.162 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:53:04.163 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:29.266999+0000", 2024-09-20T22:53:04.163 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.0369999999999997e-06 2024-09-20T22:53:04.163 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.163 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.163 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:53:04.163 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:29.267052+0000", 2024-09-20T22:53:04.163 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.3377000000000001e-05 2024-09-20T22:53:04.163 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.163 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.163 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:53:04.163 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:29.267073+0000", 2024-09-20T22:53:04.163 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.0604000000000001e-05 2024-09-20T22:53:04.163 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.163 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.163 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:53:04.164 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:29.269512+0000", 2024-09-20T22:53:04.164 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0024397749999999999 2024-09-20T22:53:04.164 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.164 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.164 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:53:04.164 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:29.269549+0000", 2024-09-20T22:53:04.164 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.6248000000000003e-05 2024-09-20T22:53:04.164 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:53:04.164 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:53:04.164 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:53:04.164 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.164 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.165 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:5418 7.1 7:8422f82d:::smithi08477203-22:head [copy-from ver 726] snapc ef=[ef,ed,ec,eb,ea,e8,e7,e6,e5,e4,e2,df,db,da,d7,d5,d4,d3,d0,cf,ce,ca,c5,c2,bf,b4,b2,a7,95] ondisk+write+known_if_redirected+supports_pool_eio e549)", 2024-09-20T22:53:04.165 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:50:29.266643+0000", 2024-09-20T22:53:04.165 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 154.86108345, 2024-09-20T22:53:04.165 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.022362705, 2024-09-20T22:53:04.165 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:53:04.165 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:53:04.165 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:53:04.165 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:53:04.165 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:53:04.165 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:53:04.165 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 5418 2024-09-20T22:53:04.165 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.166 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:53:04.166 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.166 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:53:04.166 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:29.266643+0000", 2024-09-20T22:53:04.166 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:53:04.166 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.166 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.166 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:53:04.166 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:29.266643+0000", 2024-09-20T22:53:04.166 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:53:04.166 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.166 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.166 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:53:04.167 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:29.266646+0000", 2024-09-20T22:53:04.167 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.835e-06 2024-09-20T22:53:04.167 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.167 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.167 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:53:04.167 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:29.266654+0000", 2024-09-20T22:53:04.167 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.7540000000000006e-06 2024-09-20T22:53:04.167 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.167 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.167 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:53:04.167 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:29.266672+0000", 2024-09-20T22:53:04.167 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.8187000000000001e-05 2024-09-20T22:53:04.167 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.167 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.168 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:53:04.168 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:29.266681+0000", 2024-09-20T22:53:04.168 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.5199999999999997e-06 2024-09-20T22:53:04.168 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.168 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.168 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:53:04.168 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:29.266788+0000", 2024-09-20T22:53:04.168 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00010746 2024-09-20T22:53:04.168 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.168 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.168 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:53:04.168 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:29.266936+0000", 2024-09-20T22:53:04.168 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000148238 2024-09-20T22:53:04.168 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.169 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.169 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 3,6", 2024-09-20T22:53:04.169 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:29.274529+0000", 2024-09-20T22:53:04.169 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0075932170000000002 2024-09-20T22:53:04.169 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.169 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.169 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:53:04.169 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:29.279305+0000", 2024-09-20T22:53:04.169 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0047758990000000001 2024-09-20T22:53:04.169 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.169 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.169 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:53:04.169 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:29.281410+0000", 2024-09-20T22:53:04.169 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.002104341 2024-09-20T22:53:04.169 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.170 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.170 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:53:04.170 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:29.288915+0000", 2024-09-20T22:53:04.170 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0075048900000000002 2024-09-20T22:53:04.170 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.170 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.170 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:53:04.170 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:29.288966+0000", 2024-09-20T22:53:04.170 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.1669000000000003e-05 2024-09-20T22:53:04.170 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.170 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.170 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:53:04.170 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:29.289006+0000", 2024-09-20T22:53:04.171 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.9694999999999999e-05 2024-09-20T22:53:04.171 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:53:04.171 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:53:04.171 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:53:04.171 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.171 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.171 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:5419 7.1 7:8422f82d:::smithi08477203-22:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e549)", 2024-09-20T22:53:04.171 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:50:29.266705+0000", 2024-09-20T22:53:04.171 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 154.861022117, 2024-09-20T22:53:04.171 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.012031030999999999, 2024-09-20T22:53:04.171 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:53:04.171 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:53:04.171 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:53:04.171 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:53:04.172 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:53:04.172 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:53:04.172 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 5419 2024-09-20T22:53:04.172 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.172 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:53:04.172 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.172 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:53:04.172 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:29.266705+0000", 2024-09-20T22:53:04.172 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:53:04.172 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.172 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.172 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:53:04.172 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:29.266705+0000", 2024-09-20T22:53:04.172 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:53:04.173 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.173 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.173 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:53:04.173 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:29.266705+0000", 2024-09-20T22:53:04.173 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.5000000000000002e-07 2024-09-20T22:53:04.173 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.173 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.173 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:53:04.173 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:29.266708+0000", 2024-09-20T22:53:04.173 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.242e-06 2024-09-20T22:53:04.173 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.173 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.173 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:53:04.173 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:29.266717+0000", 2024-09-20T22:53:04.174 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.9390000000000003e-06 2024-09-20T22:53:04.174 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.174 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.174 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:53:04.174 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:29.266723+0000", 2024-09-20T22:53:04.174 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.7289999999999997e-06 2024-09-20T22:53:04.174 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.174 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.174 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:53:04.174 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:29.267079+0000", 2024-09-20T22:53:04.174 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00035592600000000001 2024-09-20T22:53:04.174 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.174 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.174 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for blocked object", 2024-09-20T22:53:04.175 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:29.267125+0000", 2024-09-20T22:53:04.175 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.5859000000000003e-05 2024-09-20T22:53:04.175 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.175 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.175 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:53:04.175 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:29.278512+0000", 2024-09-20T22:53:04.175 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.011386871999999999 2024-09-20T22:53:04.175 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.175 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.175 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:53:04.175 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:29.278620+0000", 2024-09-20T22:53:04.175 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000108729 2024-09-20T22:53:04.175 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.176 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.176 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:53:04.176 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:29.278736+0000", 2024-09-20T22:53:04.176 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000115085 2024-09-20T22:53:04.176 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:53:04.176 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:53:04.176 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:53:04.176 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.176 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.176 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:5679 7.9 e572/426 7:9cc0b7e2:::smithi08477203-46:head v 572'739, mlcod=570'735)", 2024-09-20T22:53:04.176 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:50:52.389509+0000", 2024-09-20T22:53:04.176 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 131.73821774000001, 2024-09-20T22:53:04.176 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0062941530000000002, 2024-09-20T22:53:04.177 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:53:04.177 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:53:04.177 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:53:04.177 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:53:04.177 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.177 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:53:04.177 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:52.389509+0000", 2024-09-20T22:53:04.177 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:53:04.177 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.177 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.177 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:53:04.178 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:52.389509+0000", 2024-09-20T22:53:04.178 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:53:04.178 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.178 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.178 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:53:04.178 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:52.389509+0000", 2024-09-20T22:53:04.178 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.8099999999999999e-07 2024-09-20T22:53:04.178 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.178 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.178 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:53:04.178 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:52.392957+0000", 2024-09-20T22:53:04.178 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0034480219999999998 2024-09-20T22:53:04.178 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.179 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.179 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:53:04.179 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:52.392972+0000", 2024-09-20T22:53:04.179 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4324e-05 2024-09-20T22:53:04.179 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.179 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.179 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:53:04.179 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:52.392978+0000", 2024-09-20T22:53:04.179 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.9439999999999999e-06 2024-09-20T22:53:04.179 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.179 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.179 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:53:04.179 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:52.393041+0000", 2024-09-20T22:53:04.179 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.2144999999999995e-05 2024-09-20T22:53:04.179 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.180 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.180 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:53:04.180 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:52.393057+0000", 2024-09-20T22:53:04.180 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.6273000000000002e-05 2024-09-20T22:53:04.180 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.180 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.180 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:53:04.182 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:53:04.182 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:52.395766+0000", 2024-09-20T22:53:04.183 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0027090339999999999 2024-09-20T22:53:04.183 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.183 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.183 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:53:04.183 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:52.395803+0000", 2024-09-20T22:53:04.183 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.7129999999999999e-05 2024-09-20T22:53:04.183 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:53:04.183 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:53:04.183 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:53:04.183 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.183 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.183 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(osd.0.0:1215 7.9 e588/426 7:91f2d47c:::smithi08477203-8:ef v 588'801, mlcod=588'794)", 2024-09-20T22:53:04.183 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:51:08.278858+0000", 2024-09-20T22:53:04.184 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 115.848868818, 2024-09-20T22:53:04.184 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0079378230000000001, 2024-09-20T22:53:04.184 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:53:04.185 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:53:04.185 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:53:04.185 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:53:04.185 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.185 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:53:04.185 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:08.278858+0000", 2024-09-20T22:53:04.185 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:53:04.186 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.186 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.186 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:53:04.186 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:08.278858+0000", 2024-09-20T22:53:04.186 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:53:04.186 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.186 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.186 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:53:04.186 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:08.278859+0000", 2024-09-20T22:53:04.186 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4249999999999999e-06 2024-09-20T22:53:04.186 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.187 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.187 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:53:04.187 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:08.278875+0000", 2024-09-20T22:53:04.187 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.5698000000000001e-05 2024-09-20T22:53:04.187 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.187 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.187 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:53:04.187 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:08.278888+0000", 2024-09-20T22:53:04.187 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3377000000000001e-05 2024-09-20T22:53:04.187 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.187 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.187 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:53:04.187 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:08.278895+0000", 2024-09-20T22:53:04.188 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.1990000000000003e-06 2024-09-20T22:53:04.188 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.188 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.188 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:53:04.188 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:08.278963+0000", 2024-09-20T22:53:04.188 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.8042999999999999e-05 2024-09-20T22:53:04.188 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.188 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.188 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:53:04.188 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:08.278994+0000", 2024-09-20T22:53:04.188 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.1343999999999997e-05 2024-09-20T22:53:04.188 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.189 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.189 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:53:04.189 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:08.286741+0000", 2024-09-20T22:53:04.189 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0077470780000000001 2024-09-20T22:53:04.189 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.189 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.189 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:53:04.189 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:08.286796+0000", 2024-09-20T22:53:04.189 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.4659e-05 2024-09-20T22:53:04.189 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:53:04.189 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:53:04.189 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:53:04.189 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.190 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.190 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(osd.0.0:1265 7.9 e596/426 7:907aec0e:::smithi08477203-3:aa v 596'852, mlcod=593'848)", 2024-09-20T22:53:04.190 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:51:16.390167+0000", 2024-09-20T22:53:04.190 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 107.737559551, 2024-09-20T22:53:04.190 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0085014630000000008, 2024-09-20T22:53:04.190 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:53:04.190 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:53:04.190 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:53:04.190 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:53:04.190 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.190 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:53:04.190 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:16.390167+0000", 2024-09-20T22:53:04.190 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:53:04.190 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.191 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.191 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:53:04.191 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:16.390167+0000", 2024-09-20T22:53:04.191 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:53:04.191 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.191 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.191 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:53:04.191 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:16.390167+0000", 2024-09-20T22:53:04.191 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9999999999999999e-07 2024-09-20T22:53:04.191 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.191 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.191 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:53:04.191 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:16.390172+0000", 2024-09-20T22:53:04.191 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.7480000000000001e-06 2024-09-20T22:53:04.192 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.192 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.192 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:53:04.192 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:16.390188+0000", 2024-09-20T22:53:04.192 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.6246999999999999e-05 2024-09-20T22:53:04.192 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.192 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.192 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:53:04.192 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:16.390196+0000", 2024-09-20T22:53:04.192 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.4549999999999998e-06 2024-09-20T22:53:04.192 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.192 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.192 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:53:04.192 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:16.390261+0000", 2024-09-20T22:53:04.193 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.4888000000000006e-05 2024-09-20T22:53:04.193 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.193 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.193 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:53:04.193 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:16.390279+0000", 2024-09-20T22:53:04.193 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.8553999999999999e-05 2024-09-20T22:53:04.193 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.193 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.193 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:53:04.193 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:16.398617+0000", 2024-09-20T22:53:04.193 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0083380550000000005 2024-09-20T22:53:04.194 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.194 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.194 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:53:04.194 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:16.398669+0000", 2024-09-20T22:53:04.194 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.1316000000000001e-05 2024-09-20T22:53:04.194 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:53:04.194 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:53:04.194 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:53:04.194 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.194 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.195 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(osd.0.0:1266 7.9 e596/426 7:9eac4560:::smithi08477203-30:ae v 596'854, mlcod=593'848)", 2024-09-20T22:53:04.195 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:51:16.392066+0000", 2024-09-20T22:53:04.195 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 107.73566086, 2024-09-20T22:53:04.195 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0099900849999999992, 2024-09-20T22:53:04.195 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:53:04.195 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:53:04.195 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:53:04.195 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:53:04.195 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.195 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:53:04.195 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:16.392066+0000", 2024-09-20T22:53:04.195 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:53:04.195 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.196 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.196 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:53:04.196 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:16.392066+0000", 2024-09-20T22:53:04.196 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:53:04.196 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.196 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.196 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:53:04.196 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:16.392067+0000", 2024-09-20T22:53:04.196 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.2600000000000001e-07 2024-09-20T22:53:04.196 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.196 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.196 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:53:04.196 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:16.392073+0000", 2024-09-20T22:53:04.196 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.6989999999999997e-06 2024-09-20T22:53:04.196 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.197 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.197 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:53:04.197 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:16.392082+0000", 2024-09-20T22:53:04.197 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.8200000000000003e-06 2024-09-20T22:53:04.197 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.197 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.197 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:53:04.197 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:16.392090+0000", 2024-09-20T22:53:04.197 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.7160000000000003e-06 2024-09-20T22:53:04.197 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.197 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.197 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:53:04.197 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:16.392637+0000", 2024-09-20T22:53:04.197 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00054682600000000004 2024-09-20T22:53:04.198 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.198 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.198 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:53:04.198 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:16.392657+0000", 2024-09-20T22:53:04.198 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9956e-05 2024-09-20T22:53:04.198 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.198 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.198 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:53:04.198 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:16.401999+0000", 2024-09-20T22:53:04.198 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0093422560000000002 2024-09-20T22:53:04.198 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.198 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.199 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:53:04.199 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:16.402056+0000", 2024-09-20T22:53:04.199 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.6986e-05 2024-09-20T22:53:04.199 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:53:04.199 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:53:04.199 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:53:04.199 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.199 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.199 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(osd.0.0:1267 7.4 e596/460 7:24b1a612:::smithi08477203-43:aa v 596'321, mlcod=592'317)", 2024-09-20T22:53:04.199 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:51:16.392654+0000", 2024-09-20T22:53:04.199 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 107.73507312300001, 2024-09-20T22:53:04.200 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0093429540000000005, 2024-09-20T22:53:04.200 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:53:04.200 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:53:04.200 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:53:04.200 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:53:04.200 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.200 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:53:04.200 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:16.392654+0000", 2024-09-20T22:53:04.200 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:53:04.200 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.200 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.200 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:53:04.200 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:16.392654+0000", 2024-09-20T22:53:04.200 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:53:04.201 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.201 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.201 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:53:04.201 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:16.392654+0000", 2024-09-20T22:53:04.201 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.4300000000000002e-07 2024-09-20T22:53:04.201 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.201 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.201 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:53:04.201 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:16.392664+0000", 2024-09-20T22:53:04.201 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.2830000000000007e-06 2024-09-20T22:53:04.201 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.201 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.202 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:53:04.202 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:16.392671+0000", 2024-09-20T22:53:04.202 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.3499999999999999e-06 2024-09-20T22:53:04.202 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.202 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.202 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:53:04.202 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:16.392677+0000", 2024-09-20T22:53:04.202 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.4590000000000003e-06 2024-09-20T22:53:04.202 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.202 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.202 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:53:04.202 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:16.392753+0000", 2024-09-20T22:53:04.202 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.5387000000000002e-05 2024-09-20T22:53:04.202 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.203 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.203 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:53:04.203 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:16.392776+0000", 2024-09-20T22:53:04.203 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.3703e-05 2024-09-20T22:53:04.203 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.203 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.203 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:53:04.203 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:16.401944+0000", 2024-09-20T22:53:04.203 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0091675100000000002 2024-09-20T22:53:04.203 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.203 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.203 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:53:04.203 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:16.401996+0000", 2024-09-20T22:53:04.204 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.2519000000000003e-05 2024-09-20T22:53:04.204 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:53:04.204 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:53:04.204 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:53:04.204 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.204 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.204 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(osd.0.0:1268 7.4 e596/460 7:2f8647a2:::smithi08477203-41:aa v 596'323, mlcod=592'317)", 2024-09-20T22:53:04.204 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:51:16.392986+0000", 2024-09-20T22:53:04.204 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 107.73474101799999, 2024-09-20T22:53:04.204 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0091905690000000009, 2024-09-20T22:53:04.204 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:53:04.204 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:53:04.204 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:53:04.204 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:53:04.205 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.205 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:53:04.205 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:16.392986+0000", 2024-09-20T22:53:04.205 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:53:04.205 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.205 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.205 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:53:04.205 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:16.392986+0000", 2024-09-20T22:53:04.205 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:53:04.205 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.205 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.205 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:53:04.205 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:16.392986+0000", 2024-09-20T22:53:04.205 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.1099999999999995e-07 2024-09-20T22:53:04.205 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.206 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.206 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:53:04.206 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:16.392992+0000", 2024-09-20T22:53:04.206 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.7089999999999999e-06 2024-09-20T22:53:04.206 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.206 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.206 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:53:04.206 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:16.393000+0000", 2024-09-20T22:53:04.206 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.5229999999999995e-06 2024-09-20T22:53:04.206 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.206 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.207 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:53:04.207 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:16.393007+0000", 2024-09-20T22:53:04.207 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.5169999999999996e-06 2024-09-20T22:53:04.207 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.207 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.207 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:53:04.207 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:16.393223+0000", 2024-09-20T22:53:04.207 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00021552600000000001 2024-09-20T22:53:04.207 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.207 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.207 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:53:04.207 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:16.393245+0000", 2024-09-20T22:53:04.207 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.2172e-05 2024-09-20T22:53:04.207 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.208 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.208 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:53:04.208 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:16.402143+0000", 2024-09-20T22:53:04.208 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.008898418 2024-09-20T22:53:04.208 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.208 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.208 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:53:04.208 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:16.402176+0000", 2024-09-20T22:53:04.208 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.3093000000000003e-05 2024-09-20T22:53:04.208 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:53:04.208 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:53:04.208 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:53:04.209 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.209 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.209 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:6000 7.9 e604/602 7:9eac4560:::smithi08477203-30:head v 604'886, mlcod=604'882)", 2024-09-20T22:53:04.209 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:51:24.304331+0000", 2024-09-20T22:53:04.209 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 99.823395985999994, 2024-09-20T22:53:04.209 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.014326934, 2024-09-20T22:53:04.209 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:53:04.209 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:53:04.209 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:53:04.209 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:53:04.209 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.209 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:53:04.209 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:24.304331+0000", 2024-09-20T22:53:04.209 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:53:04.210 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.210 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.210 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:53:04.210 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:24.304331+0000", 2024-09-20T22:53:04.210 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:53:04.210 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.210 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.210 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:53:04.210 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:24.304332+0000", 2024-09-20T22:53:04.210 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.8400000000000002e-07 2024-09-20T22:53:04.210 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.210 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.210 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:53:04.211 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:24.310538+0000", 2024-09-20T22:53:04.211 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0062066710000000004 2024-09-20T22:53:04.211 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.211 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.211 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:53:04.211 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:24.310568+0000", 2024-09-20T22:53:04.211 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.9283000000000001e-05 2024-09-20T22:53:04.211 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.211 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.211 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:53:04.211 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:24.310576+0000", 2024-09-20T22:53:04.211 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.3639999999999999e-06 2024-09-20T22:53:04.212 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.212 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.212 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:53:04.212 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:24.310641+0000", 2024-09-20T22:53:04.212 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.4690000000000006e-05 2024-09-20T22:53:04.212 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.212 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.212 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:53:04.212 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:24.310665+0000", 2024-09-20T22:53:04.212 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.4635999999999999e-05 2024-09-20T22:53:04.212 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.212 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.212 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:53:04.213 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:24.318613+0000", 2024-09-20T22:53:04.213 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0079475429999999996 2024-09-20T22:53:04.213 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.213 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.213 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:53:04.213 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:24.318658+0000", 2024-09-20T22:53:04.213 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.4762999999999998e-05 2024-09-20T22:53:04.213 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:53:04.213 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:53:04.213 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:53:04.213 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.213 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.213 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:6018 7.4 e606/602 7:24b1a612:::smithi08477203-43:head v 606'331, mlcod=603'327)", 2024-09-20T22:53:04.213 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:51:26.433795+0000", 2024-09-20T22:53:04.214 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 97.693931229, 2024-09-20T22:53:04.214 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0066411250000000003, 2024-09-20T22:53:04.214 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:53:04.214 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:53:04.214 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:53:04.214 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:53:04.214 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.214 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:53:04.214 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:26.433795+0000", 2024-09-20T22:53:04.214 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:53:04.214 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.214 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.214 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:53:04.215 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:26.433795+0000", 2024-09-20T22:53:04.215 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:53:04.215 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.215 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.215 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:53:04.215 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:26.433796+0000", 2024-09-20T22:53:04.215 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.5699999999999999e-07 2024-09-20T22:53:04.215 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.215 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.215 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:53:04.215 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:26.437596+0000", 2024-09-20T22:53:04.215 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0038002859999999999 2024-09-20T22:53:04.215 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.215 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.216 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:53:04.216 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:26.437614+0000", 2024-09-20T22:53:04.216 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.8233999999999999e-05 2024-09-20T22:53:04.216 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.216 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.216 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:53:04.216 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:26.437622+0000", 2024-09-20T22:53:04.216 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.9179999999999994e-06 2024-09-20T22:53:04.216 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.216 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.216 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:53:04.216 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:26.437700+0000", 2024-09-20T22:53:04.216 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.7498999999999998e-05 2024-09-20T22:53:04.217 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.217 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.217 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:53:04.217 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:26.437727+0000", 2024-09-20T22:53:04.217 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.7549999999999999e-05 2024-09-20T22:53:04.217 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.217 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.217 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:53:04.217 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:26.440389+0000", 2024-09-20T22:53:04.217 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0026614049999999999 2024-09-20T22:53:04.217 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.217 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.217 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:53:04.217 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:26.440437+0000", 2024-09-20T22:53:04.218 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.7976000000000003e-05 2024-09-20T22:53:04.218 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:53:04.218 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:53:04.218 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:53:04.218 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.218 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.218 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:6253 7.9 e629/602 7:91f2d47c:::smithi08477203-8:head v 629'948, mlcod=627'945)", 2024-09-20T22:53:04.218 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:51:49.625685+0000", 2024-09-20T22:53:04.218 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 74.502041758000004, 2024-09-20T22:53:04.219 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.017501778999999999, 2024-09-20T22:53:04.219 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:53:04.219 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:53:04.219 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:53:04.219 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:53:04.219 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.219 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:53:04.219 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:49.625685+0000", 2024-09-20T22:53:04.219 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:53:04.219 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.219 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.219 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:53:04.219 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:49.625685+0000", 2024-09-20T22:53:04.220 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:53:04.220 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.220 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.220 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:53:04.220 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:49.625686+0000", 2024-09-20T22:53:04.220 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.457e-06 2024-09-20T22:53:04.220 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.220 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.220 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:53:04.220 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:49.632417+0000", 2024-09-20T22:53:04.221 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.006730453 2024-09-20T22:53:04.221 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.221 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.221 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:53:04.221 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:49.632440+0000", 2024-09-20T22:53:04.221 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.2999000000000001e-05 2024-09-20T22:53:04.221 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.221 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.221 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:53:04.221 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:49.632450+0000", 2024-09-20T22:53:04.221 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0210999999999999e-05 2024-09-20T22:53:04.221 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.221 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.222 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:53:04.222 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:49.632524+0000", 2024-09-20T22:53:04.222 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.4234999999999997e-05 2024-09-20T22:53:04.222 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.222 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.222 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:53:04.222 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:49.632549+0000", 2024-09-20T22:53:04.222 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.4572000000000001e-05 2024-09-20T22:53:04.222 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.222 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.222 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:53:04.222 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:49.643131+0000", 2024-09-20T22:53:04.223 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.010582446000000001 2024-09-20T22:53:04.223 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.223 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.223 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:53:04.223 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:49.643187+0000", 2024-09-20T22:53:04.223 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.5405999999999998e-05 2024-09-20T22:53:04.223 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:53:04.223 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:53:04.223 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:53:04.223 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.223 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.223 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:6360 7.8 e638/430 7:1e9e2524:::smithi08477203-5:head v 638'681, mlcod=637'677)", 2024-09-20T22:53:04.223 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:51:58.286846+0000", 2024-09-20T22:53:04.224 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 65.840880341000002, 2024-09-20T22:53:04.224 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.012344572999999999, 2024-09-20T22:53:04.224 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:53:04.224 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:53:04.224 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:53:04.224 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:53:04.224 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.224 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:53:04.224 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:58.286846+0000", 2024-09-20T22:53:04.224 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:53:04.224 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.224 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.225 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:53:04.225 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:58.286846+0000", 2024-09-20T22:53:04.225 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:53:04.225 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.225 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.225 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:53:04.225 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:58.286847+0000", 2024-09-20T22:53:04.225 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.3700000000000001e-07 2024-09-20T22:53:04.225 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.225 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.225 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:53:04.225 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:58.297345+0000", 2024-09-20T22:53:04.225 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.010498535 2024-09-20T22:53:04.226 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.226 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.226 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:53:04.226 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:58.297359+0000", 2024-09-20T22:53:04.226 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3631999999999999e-05 2024-09-20T22:53:04.226 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.226 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.226 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:53:04.226 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:58.297365+0000", 2024-09-20T22:53:04.226 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.4489999999999996e-06 2024-09-20T22:53:04.227 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.227 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.227 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:53:04.227 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:58.297433+0000", 2024-09-20T22:53:04.227 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.7206000000000007e-05 2024-09-20T22:53:04.227 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.227 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.227 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:53:04.227 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:58.297455+0000", 2024-09-20T22:53:04.227 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.2711999999999999e-05 2024-09-20T22:53:04.227 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.227 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.227 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:53:04.227 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:58.299091+0000", 2024-09-20T22:53:04.228 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0016354010000000001 2024-09-20T22:53:04.228 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.228 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.228 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:53:04.228 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:58.299191+0000", 2024-09-20T22:53:04.228 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000100201 2024-09-20T22:53:04.228 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:53:04.228 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:53:04.228 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:53:04.228 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.228 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.228 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(osd.0.0:1463 7.9 e643/602 7:9eac4560:::smithi08477203-30:106 v 643'977, mlcod=638'974)", 2024-09-20T22:53:04.228 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:52:03.281106+0000", 2024-09-20T22:53:04.228 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 60.846620434999998, 2024-09-20T22:53:04.229 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0074291940000000001, 2024-09-20T22:53:04.229 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:53:04.229 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:53:04.229 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:53:04.229 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:53:04.229 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.229 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:53:04.229 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:03.281106+0000", 2024-09-20T22:53:04.229 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:53:04.229 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.229 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.229 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:53:04.229 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:03.281106+0000", 2024-09-20T22:53:04.229 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:53:04.230 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.230 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.230 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:53:04.230 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:03.281107+0000", 2024-09-20T22:53:04.230 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.9800000000000004e-07 2024-09-20T22:53:04.230 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.230 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.230 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:53:04.230 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:03.281112+0000", 2024-09-20T22:53:04.230 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.4709999999999998e-06 2024-09-20T22:53:04.230 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.230 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.230 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:53:04.230 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:03.281120+0000", 2024-09-20T22:53:04.231 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.486e-06 2024-09-20T22:53:04.231 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.231 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.231 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:53:04.231 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:03.281126+0000", 2024-09-20T22:53:04.231 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.5849999999999996e-06 2024-09-20T22:53:04.231 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.231 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.231 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:53:04.231 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:03.281211+0000", 2024-09-20T22:53:04.231 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.4299e-05 2024-09-20T22:53:04.231 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.231 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.231 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:53:04.232 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:03.281230+0000", 2024-09-20T22:53:04.232 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9318e-05 2024-09-20T22:53:04.232 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.232 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.232 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:53:04.232 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:03.288489+0000", 2024-09-20T22:53:04.232 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.007259255 2024-09-20T22:53:04.232 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.232 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.232 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:53:04.232 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:03.288535+0000", 2024-09-20T22:53:04.232 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.6282000000000002e-05 2024-09-20T22:53:04.232 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:53:04.233 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:53:04.233 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:53:04.233 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.233 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.233 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(osd.0.0:1636 7.9 e690/602 7:91f2d47c:::smithi08477203-8:108 v 690'1082, mlcod=687'1076)", 2024-09-20T22:53:04.233 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:52:50.292726+0000", 2024-09-20T22:53:04.233 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 13.835000240999999, 2024-09-20T22:53:04.233 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0062741430000000003, 2024-09-20T22:53:04.233 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:53:04.233 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:53:04.233 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:53:04.233 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:53:04.233 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.233 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:53:04.234 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:50.292726+0000", 2024-09-20T22:53:04.234 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:53:04.234 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.234 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.234 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:53:04.234 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:50.292726+0000", 2024-09-20T22:53:04.234 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:53:04.234 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.234 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.234 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:53:04.234 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:50.292727+0000", 2024-09-20T22:53:04.234 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.7700000000000004e-07 2024-09-20T22:53:04.234 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.235 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.235 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:53:04.235 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:50.292733+0000", 2024-09-20T22:53:04.235 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.5890000000000002e-06 2024-09-20T22:53:04.235 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.235 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.235 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:53:04.235 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:50.292743+0000", 2024-09-20T22:53:04.235 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0239e-05 2024-09-20T22:53:04.235 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.235 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.235 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:53:04.235 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:50.292749+0000", 2024-09-20T22:53:04.235 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.4110000000000001e-06 2024-09-20T22:53:04.236 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.236 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.236 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:53:04.236 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:50.293359+0000", 2024-09-20T22:53:04.236 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00060966899999999997 2024-09-20T22:53:04.236 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.236 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.236 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:53:04.236 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:50.293384+0000", 2024-09-20T22:53:04.236 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.4916e-05 2024-09-20T22:53:04.236 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.236 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.236 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:53:04.236 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:50.298966+0000", 2024-09-20T22:53:04.237 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0055820720000000004 2024-09-20T22:53:04.237 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:04.237 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:04.237 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:53:04.237 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:50.299001+0000", 2024-09-20T22:53:04.237 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.4669999999999998e-05 2024-09-20T22:53:04.237 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:53:04.237 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:53:04.237 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:53:04.237 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:53:04.237 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:53:04.237 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-20T22:53:04.258 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:04.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:04.260 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:53:04.343 INFO:tasks.rados.rados.0.smithi084.stdout:2775: done (4 left) 2024-09-20T22:53:04.344 INFO:tasks.rados.rados.0.smithi084.stdout:2776: done (3 left) 2024-09-20T22:53:04.344 INFO:tasks.rados.rados.0.smithi084.stdout:2777: done (2 left) 2024-09-20T22:53:04.344 INFO:tasks.rados.rados.0.smithi084.stdout:2778: done (1 left) 2024-09-20T22:53:04.344 INFO:tasks.rados.rados.0.smithi084.stdout:2779: done (0 left) 2024-09-20T22:53:04.344 INFO:tasks.rados.rados.0.smithi084.stdout:2780: write (excl) oid 42 current snap is 315 2024-09-20T22:53:04.344 INFO:tasks.rados.rados.0.smithi084.stdout:2780: seq_num 965 ranges {618755=475763,1636300=457996} 2024-09-20T22:53:04.350 INFO:tasks.rados.rados.0.smithi084.stdout:2780: writing smithi08477203-42 from 618755 to 1094518 tid 1 2024-09-20T22:53:04.357 INFO:tasks.rados.rados.0.smithi084.stdout:2780: writing smithi08477203-42 from 1636300 to 2094296 tid 2 2024-09-20T22:53:04.357 INFO:tasks.rados.rados.0.smithi084.stdout:2781: rollback oid 24 current snap is 315 2024-09-20T22:53:04.357 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 24 to 296 2024-09-20T22:53:04.358 INFO:tasks.rados.rados.0.smithi084.stdout:2780: finishing write tid 1 to smithi08477203-42 2024-09-20T22:53:04.358 INFO:tasks.rados.rados.0.smithi084.stdout:2782: write oid 11 current snap is 315 2024-09-20T22:53:04.358 INFO:tasks.rados.rados.0.smithi084.stdout:2782: seq_num 966 ranges {635562=667365,2094782=140780} 2024-09-20T22:53:04.360 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:53:04.365 INFO:tasks.rados.rados.0.smithi084.stdout:2782: writing smithi08477203-11 from 635562 to 1302927 tid 1 2024-09-20T22:53:04.371 INFO:tasks.rados.rados.0.smithi084.stdout:2782: writing smithi08477203-11 from 2094782 to 2235562 tid 2 2024-09-20T22:53:04.371 INFO:tasks.rados.rados.0.smithi084.stdout:2783: delete oid 12 current snap is 315 2024-09-20T22:53:04.371 INFO:tasks.rados.rados.0.smithi084.stdout:2781: finishing rollback tid 0 to smithi08477203-24 2024-09-20T22:53:04.371 INFO:tasks.rados.rados.0.smithi084.stdout:2780: finishing write tid 2 to smithi08477203-42 2024-09-20T22:53:04.371 INFO:tasks.rados.rados.0.smithi084.stdout:2781: finishing rollback tid 1 to smithi08477203-24 2024-09-20T22:53:04.372 INFO:tasks.rados.rados.0.smithi084.stdout:2780: finishing write tid 3 to smithi08477203-42 2024-09-20T22:53:04.372 INFO:tasks.rados.rados.0.smithi084.stdout:2781: finishing rollback tid 2 to smithi08477203-24 2024-09-20T22:53:04.372 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 24 v 1830 (ObjNum 780 snap 252 seq_num 780) dirty exists 2024-09-20T22:53:04.372 INFO:tasks.rados.rados.0.smithi084.stdout:2780: finishing write tid 4 to smithi08477203-42 2024-09-20T22:53:04.372 INFO:tasks.rados.rados.0.smithi084.stdout:2780: oid 42 updating version 0 to 1235 2024-09-20T22:53:04.372 INFO:tasks.rados.rados.0.smithi084.stdout:2780: oid 42 updating version 1235 to 1236 2024-09-20T22:53:04.372 INFO:tasks.rados.rados.0.smithi084.stdout:2780: oid 42 version 1236 is already newer than 1234 2024-09-20T22:53:04.372 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 42 v 1236 (ObjNum 965 snap 315 seq_num 965) dirty exists 2024-09-20T22:53:04.373 INFO:tasks.rados.rados.0.smithi084.stdout:2780: left oid 42 (ObjNum 965 snap 315 seq_num 965) 2024-09-20T22:53:04.373 INFO:tasks.rados.rados.0.smithi084.stdout:2780: done (3 left) 2024-09-20T22:53:04.373 INFO:tasks.rados.rados.0.smithi084.stdout:2781: done (2 left) 2024-09-20T22:53:04.373 INFO:tasks.rados.rados.0.smithi084.stdout:2783: done (1 left) 2024-09-20T22:53:04.373 INFO:tasks.rados.rados.0.smithi084.stdout:2784: copy_from oid 32 from oid 37 current snap is 315 2024-09-20T22:53:04.373 INFO:tasks.rados.rados.0.smithi084.stdout:2785: read oid 10 snap -1 2024-09-20T22:53:04.374 INFO:tasks.rados.rados.0.smithi084.stdout:2785: expect (ObjNum 919 snap 302 seq_num 919) 2024-09-20T22:53:04.374 INFO:tasks.rados.rados.0.smithi084.stdout:2786: delete oid 30 current snap is 315 2024-09-20T22:53:04.374 INFO:tasks.rados.rados.0.smithi084.stdout:2782: finishing write tid 1 to smithi08477203-11 2024-09-20T22:53:04.374 INFO:tasks.rados.rados.0.smithi084.stdout:2782: finishing write tid 2 to smithi08477203-11 2024-09-20T22:53:04.374 INFO:tasks.rados.rados.0.smithi084.stdout:2782: finishing write tid 3 to smithi08477203-11 2024-09-20T22:53:04.375 INFO:tasks.rados.rados.0.smithi084.stdout:2782: finishing write tid 4 to smithi08477203-11 2024-09-20T22:53:04.375 INFO:tasks.rados.rados.0.smithi084.stdout:2782: oid 11 updating version 0 to 828 2024-09-20T22:53:04.375 INFO:tasks.rados.rados.0.smithi084.stdout:2782: oid 11 version 828 is already newer than 827 2024-09-20T22:53:04.375 INFO:tasks.rados.rados.0.smithi084.stdout:2782: oid 11 updating version 828 to 829 2024-09-20T22:53:04.375 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 11 v 829 (ObjNum 966 snap 315 seq_num 966) dirty exists 2024-09-20T22:53:04.375 INFO:tasks.rados.rados.0.smithi084.stdout:2782: left oid 11 (ObjNum 966 snap 315 seq_num 966) 2024-09-20T22:53:04.376 INFO:tasks.rados.rados.0.smithi084.stdout:2784: finishing copy_from to smithi08477203-32 2024-09-20T22:53:04.376 INFO:tasks.rados.rados.0.smithi084.stdout:2784: got expected ENOENT (src dne) 2024-09-20T22:53:04.376 INFO:tasks.rados.rados.0.smithi084.stdout:2784: finishing copy_from racing read to smithi08477203-32 2024-09-20T22:53:04.376 INFO:tasks.rados.rados.0.smithi084.stdout:2782: done (3 left) 2024-09-20T22:53:04.376 INFO:tasks.rados.rados.0.smithi084.stdout:2784: done (2 left) 2024-09-20T22:53:04.376 INFO:tasks.rados.rados.0.smithi084.stdout:2786: done (1 left) 2024-09-20T22:53:04.376 INFO:tasks.rados.rados.0.smithi084.stdout:2787: rollback oid 35 current snap is 315 2024-09-20T22:53:04.376 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 35 to 313 2024-09-20T22:53:04.376 INFO:tasks.rados.rados.0.smithi084.stdout:2788: read oid 6 snap -1 2024-09-20T22:53:04.377 INFO:tasks.rados.rados.0.smithi084.stdout:2788: expect (ObjNum 948 snap 311 seq_num 948) 2024-09-20T22:53:04.377 INFO:tasks.rados.rados.0.smithi084.stdout:2789: delete oid 4 current snap is 315 2024-09-20T22:53:04.380 INFO:tasks.rados.rados.0.smithi084.stdout:2789: done (3 left) 2024-09-20T22:53:04.380 INFO:tasks.rados.rados.0.smithi084.stdout:2790: write oid 40 current snap is 315 2024-09-20T22:53:04.380 INFO:tasks.rados.rados.0.smithi084.stdout:2790: seq_num 970 ranges {702790=568361,1812357=490433} 2024-09-20T22:53:04.387 INFO:tasks.rados.rados.0.smithi084.stdout:2790: writing smithi08477203-40 from 702790 to 1271151 tid 1 2024-09-20T22:53:04.391 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:04.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:04.392 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:53:04.393 INFO:tasks.rados.rados.0.smithi084.stdout:2790: writing smithi08477203-40 from 1812357 to 2302790 tid 2 2024-09-20T22:53:04.393 INFO:tasks.rados.rados.0.smithi084.stdout:2791: write (excl) oid 42 current snap is 315 2024-09-20T22:53:04.393 INFO:tasks.rados.rados.0.smithi084.stdout:2791: seq_num 971 ranges {719597=643610,1870839=448758} 2024-09-20T22:53:04.397 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:53:04.397 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:53:04.401 INFO:tasks.rados.rados.0.smithi084.stdout:2791: writing smithi08477203-42 from 719597 to 1363207 tid 1 2024-09-20T22:53:04.407 INFO:tasks.rados.rados.0.smithi084.stdout:2791: writing smithi08477203-42 from 1870839 to 2319597 tid 2 2024-09-20T22:53:04.407 INFO:tasks.rados.rados.0.smithi084.stdout:2792: delete oid 47 current snap is 315 2024-09-20T22:53:04.407 INFO:tasks.rados.rados.0.smithi084.stdout:2787: finishing rollback tid 0 to smithi08477203-35 2024-09-20T22:53:04.407 INFO:tasks.rados.rados.0.smithi084.stdout:2787: finishing rollback tid 1 to smithi08477203-35 2024-09-20T22:53:04.407 INFO:tasks.rados.rados.0.smithi084.stdout:2788: expect (ObjNum 948 snap 311 seq_num 948) 2024-09-20T22:53:04.419 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:53:04.420 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:53:04.421 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:53:04.437 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:53:04.437 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:53:04.456 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-20T22:53:04.456 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T22:53:04.472 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:53:04.473 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:53:04.489 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:53:04.489 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:53:04.494 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:53:04.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:04 smithi154 ceph-mon[33264]: 4.14 scrub starts 2024-09-20T22:53:04.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:04 smithi154 ceph-mon[33264]: 4.14 scrub ok 2024-09-20T22:53:04.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:04 smithi154 ceph-mon[33264]: 6.3 scrub starts 2024-09-20T22:53:04.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:04 smithi154 ceph-mon[33264]: 6.3 scrub ok 2024-09-20T22:53:04.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:04 smithi154 ceph-mon[33264]: purged_snaps scrub starts 2024-09-20T22:53:04.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:04 smithi154 ceph-mon[33264]: purged_snaps scrub ok 2024-09-20T22:53:04.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:04 smithi154 ceph-mon[33264]: 7.d scrub starts 2024-09-20T22:53:04.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:04 smithi154 ceph-mon[33264]: 7.d scrub ok 2024-09-20T22:53:04.506 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:53:04.506 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:53:04.514 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:04.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:04.515 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:53:04.522 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:53:04.523 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:53:04.532 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:04 smithi084 ceph-mon[40024]: 4.14 scrub starts 2024-09-20T22:53:04.532 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:04 smithi084 ceph-mon[40024]: 4.14 scrub ok 2024-09-20T22:53:04.532 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:04 smithi084 ceph-mon[40024]: 6.3 scrub starts 2024-09-20T22:53:04.532 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:04 smithi084 ceph-mon[40024]: 6.3 scrub ok 2024-09-20T22:53:04.532 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:04 smithi084 ceph-mon[40024]: purged_snaps scrub starts 2024-09-20T22:53:04.532 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:04 smithi084 ceph-mon[40024]: purged_snaps scrub ok 2024-09-20T22:53:04.533 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:04 smithi084 ceph-mon[40024]: 7.d scrub starts 2024-09-20T22:53:04.533 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:04 smithi084 ceph-mon[40024]: 7.d scrub ok 2024-09-20T22:53:04.533 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:04 smithi084 ceph-mon[29774]: 4.14 scrub starts 2024-09-20T22:53:04.533 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:04 smithi084 ceph-mon[29774]: 4.14 scrub ok 2024-09-20T22:53:04.533 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:04 smithi084 ceph-mon[29774]: 6.3 scrub starts 2024-09-20T22:53:04.533 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:04 smithi084 ceph-mon[29774]: 6.3 scrub ok 2024-09-20T22:53:04.533 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:04 smithi084 ceph-mon[29774]: purged_snaps scrub starts 2024-09-20T22:53:04.533 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:04 smithi084 ceph-mon[29774]: purged_snaps scrub ok 2024-09-20T22:53:04.533 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:04 smithi084 ceph-mon[29774]: 7.d scrub starts 2024-09-20T22:53:04.533 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:04 smithi084 ceph-mon[29774]: 7.d scrub ok 2024-09-20T22:53:04.610 INFO:tasks.rados.rados.0.smithi084.stdout:2787: finishing rollback tid 2 to smithi08477203-35 2024-09-20T22:53:04.611 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 35 v 833 (ObjNum 787 snap 254 seq_num 787) dirty exists 2024-09-20T22:53:04.611 INFO:tasks.rados.rados.0.smithi084.stdout:2785: expect (ObjNum 919 snap 302 seq_num 919) 2024-09-20T22:53:04.616 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:53:04.637 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:04.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:04.639 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:53:04.723 INFO:tasks.rados.rados.0.smithi084.stdout:2790: finishing write tid 1 to smithi08477203-40 2024-09-20T22:53:04.723 INFO:tasks.rados.rados.0.smithi084.stdout:2785: done (5 left) 2024-09-20T22:53:04.723 INFO:tasks.rados.rados.0.smithi084.stdout:2787: done (4 left) 2024-09-20T22:53:04.723 INFO:tasks.rados.rados.0.smithi084.stdout:2788: done (3 left) 2024-09-20T22:53:04.723 INFO:tasks.rados.rados.0.smithi084.stdout:2792: done (2 left) 2024-09-20T22:53:04.723 INFO:tasks.rados.rados.0.smithi084.stdout:2793: write (excl) oid 14 current snap is 315 2024-09-20T22:53:04.723 INFO:tasks.rados.rados.0.smithi084.stdout:2790: finishing write tid 2 to smithi08477203-40 2024-09-20T22:53:04.723 INFO:tasks.rados.rados.0.smithi084.stdout:2793: seq_num 973 ranges {753211=794108,1987803=365408} 2024-09-20T22:53:04.731 INFO:tasks.rados.rados.0.smithi084.stdout:2793: writing smithi08477203-14 from 753211 to 1547319 tid 1 2024-09-20T22:53:04.736 INFO:tasks.rados.rados.0.smithi084.stdout:2793: writing smithi08477203-14 from 1987803 to 2353211 tid 2 2024-09-20T22:53:04.736 INFO:tasks.rados.rados.0.smithi084.stdout:2794: write (excl) oid 50 current snap is 315 2024-09-20T22:53:04.736 INFO:tasks.rados.rados.0.smithi084.stdout:2790: finishing write tid 3 to smithi08477203-40 2024-09-20T22:53:04.736 INFO:tasks.rados.rados.0.smithi084.stdout:2790: finishing write tid 4 to smithi08477203-40 2024-09-20T22:53:04.737 INFO:tasks.rados.rados.0.smithi084.stdout:2790: oid 40 updating version 0 to 1253 2024-09-20T22:53:04.737 INFO:tasks.rados.rados.0.smithi084.stdout:2790: oid 40 updating version 1253 to 1255 2024-09-20T22:53:04.737 INFO:tasks.rados.rados.0.smithi084.stdout:2790: oid 40 version 1255 is already newer than 1254 2024-09-20T22:53:04.737 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 40 v 1255 (ObjNum 970 snap 315 seq_num 970) dirty exists 2024-09-20T22:53:04.737 INFO:tasks.rados.rados.0.smithi084.stdout:2790: left oid 40 (ObjNum 970 snap 315 seq_num 970) 2024-09-20T22:53:04.737 INFO:tasks.rados.rados.0.smithi084.stdout:2791: finishing write tid 1 to smithi08477203-42 2024-09-20T22:53:04.737 INFO:tasks.rados.rados.0.smithi084.stdout:2791: finishing write tid 2 to smithi08477203-42 2024-09-20T22:53:04.737 INFO:tasks.rados.rados.0.smithi084.stdout:2791: finishing write tid 3 to smithi08477203-42 2024-09-20T22:53:04.737 INFO:tasks.rados.rados.0.smithi084.stdout:2791: finishing write tid 4 to smithi08477203-42 2024-09-20T22:53:04.737 INFO:tasks.rados.rados.0.smithi084.stdout:2791: oid 42 updating version 0 to 1237 2024-09-20T22:53:04.737 INFO:tasks.rados.rados.0.smithi084.stdout:2791: oid 42 updating version 1237 to 1239 2024-09-20T22:53:04.737 INFO:tasks.rados.rados.0.smithi084.stdout:2791: oid 42 version 1239 is already newer than 1238 2024-09-20T22:53:04.737 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 42 v 1239 (ObjNum 971 snap 315 seq_num 971) dirty exists 2024-09-20T22:53:04.737 INFO:tasks.rados.rados.0.smithi084.stdout:2791: left oid 42 (ObjNum 971 snap 315 seq_num 971) 2024-09-20T22:53:04.737 INFO:tasks.rados.rados.0.smithi084.stdout:2793: finishing write tid 1 to smithi08477203-14 2024-09-20T22:53:04.738 INFO:tasks.rados.rados.0.smithi084.stdout:2794: seq_num 974 ranges {770018=585710,2046285=323733} 2024-09-20T22:53:04.739 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:53:04.743 INFO:tasks.rados.rados.0.smithi084.stdout:2794: writing smithi08477203-50 from 770018 to 1355728 tid 1 2024-09-20T22:53:04.748 INFO:tasks.rados.rados.0.smithi084.stdout:2794: writing smithi08477203-50 from 2046285 to 2370018 tid 2 2024-09-20T22:53:04.757 INFO:tasks.rados.rados.0.smithi084.stdout:2790: done (3 left) 2024-09-20T22:53:04.757 INFO:tasks.rados.rados.0.smithi084.stdout:2791: done (2 left) 2024-09-20T22:53:04.758 INFO:tasks.rados.rados.0.smithi084.stdout:2795: write (excl) oid 45 current snap is 315 2024-09-20T22:53:04.758 INFO:tasks.rados.rados.0.smithi084.stdout:2793: finishing write tid 2 to smithi08477203-14 2024-09-20T22:53:04.758 INFO:tasks.rados.rados.0.smithi084.stdout:2795: seq_num 975 ranges {786825=660959,2104767=282058} 2024-09-20T22:53:04.758 INFO:tasks.rados.rados.0.smithi084.stdout:2795: writing smithi08477203-45 from 786825 to 1447784 tid 1 2024-09-20T22:53:04.763 INFO:tasks.rados.rados.0.smithi084.stdout:2795: writing smithi08477203-45 from 2104767 to 2386825 tid 2 2024-09-20T22:53:04.763 INFO:tasks.rados.rados.0.smithi084.stdout:2796: read oid 31 snap -1 2024-09-20T22:53:04.763 INFO:tasks.rados.rados.0.smithi084.stdout:2796: expect (ObjNum 910 snap 297 seq_num 910) 2024-09-20T22:53:04.763 INFO:tasks.rados.rados.0.smithi084.stdout:2797: snap_create 2024-09-20T22:53:04.763 INFO:tasks.rados.rados.0.smithi084.stdout:2793: finishing write tid 3 to smithi08477203-14 2024-09-20T22:53:04.764 INFO:tasks.rados.rados.0.smithi084.stdout:2793: finishing write tid 4 to smithi08477203-14 2024-09-20T22:53:04.764 INFO:tasks.rados.rados.0.smithi084.stdout:2793: oid 14 updating version 0 to 1255 2024-09-20T22:53:04.764 INFO:tasks.rados.rados.0.smithi084.stdout:2793: oid 14 version 1255 is already newer than 1254 2024-09-20T22:53:04.764 INFO:tasks.rados.rados.0.smithi084.stdout:2793: oid 14 updating version 1255 to 1256 2024-09-20T22:53:04.764 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 14 v 1256 (ObjNum 973 snap 315 seq_num 973) dirty exists 2024-09-20T22:53:04.764 INFO:tasks.rados.rados.0.smithi084.stdout:2793: left oid 14 (ObjNum 973 snap 315 seq_num 973) 2024-09-20T22:53:04.764 INFO:tasks.rados.rados.0.smithi084.stdout:2794: finishing write tid 1 to smithi08477203-50 2024-09-20T22:53:04.764 INFO:tasks.rados.rados.0.smithi084.stdout:2794: finishing write tid 2 to smithi08477203-50 2024-09-20T22:53:04.764 INFO:tasks.rados.rados.0.smithi084.stdout:2794: finishing write tid 3 to smithi08477203-50 2024-09-20T22:53:04.764 INFO:tasks.rados.rados.0.smithi084.stdout:2794: finishing write tid 4 to smithi08477203-50 2024-09-20T22:53:04.764 INFO:tasks.rados.rados.0.smithi084.stdout:2794: oid 50 updating version 0 to 1833 2024-09-20T22:53:04.764 INFO:tasks.rados.rados.0.smithi084.stdout:2794: oid 50 version 1833 is already newer than 1832 2024-09-20T22:53:04.764 INFO:tasks.rados.rados.0.smithi084.stdout:2794: oid 50 version 1833 is already newer than 1831 2024-09-20T22:53:04.764 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 50 v 1833 (ObjNum 974 snap 315 seq_num 974) dirty exists 2024-09-20T22:53:04.765 INFO:tasks.rados.rados.0.smithi084.stdout:2794: left oid 50 (ObjNum 974 snap 315 seq_num 974) 2024-09-20T22:53:04.765 INFO:tasks.rados.rados.0.smithi084.stdout:2795: finishing write tid 1 to smithi08477203-45 2024-09-20T22:53:04.766 INFO:tasks.rados.rados.0.smithi084.stdout:2795: finishing write tid 2 to smithi08477203-45 2024-09-20T22:53:04.766 INFO:tasks.rados.rados.0.smithi084.stdout:2795: finishing write tid 3 to smithi08477203-45 2024-09-20T22:53:04.766 INFO:tasks.rados.rados.0.smithi084.stdout:2795: finishing write tid 4 to smithi08477203-45 2024-09-20T22:53:04.766 INFO:tasks.rados.rados.0.smithi084.stdout:2795: oid 45 updating version 0 to 1835 2024-09-20T22:53:04.766 INFO:tasks.rados.rados.0.smithi084.stdout:2795: oid 45 updating version 1835 to 1836 2024-09-20T22:53:04.766 INFO:tasks.rados.rados.0.smithi084.stdout:2795: oid 45 version 1836 is already newer than 1834 2024-09-20T22:53:04.766 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 45 v 1836 (ObjNum 975 snap 315 seq_num 975) dirty exists 2024-09-20T22:53:04.766 INFO:tasks.rados.rados.0.smithi084.stdout:2795: left oid 45 (ObjNum 975 snap 315 seq_num 975) 2024-09-20T22:53:04.776 INFO:tasks.rados.rados.0.smithi084.stdout:2796: expect (ObjNum 910 snap 297 seq_num 910) 2024-09-20T22:53:04.780 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:04.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:04.781 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:53:04.882 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:53:04.903 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:04.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:04.904 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:53:04.926 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-20T22:53:05.005 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:53:05.026 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:05.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:05.027 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:53:05.128 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:53:05.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:05.150 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:05.151 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:53:05.173 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:53:05.252 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:53:05.273 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:05.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:05.274 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:53:05.350 INFO:tasks.rados.rados.0.smithi084.stdout:2793: done (4 left) 2024-09-20T22:53:05.350 INFO:tasks.rados.rados.0.smithi084.stdout:2794: done (3 left) 2024-09-20T22:53:05.350 INFO:tasks.rados.rados.0.smithi084.stdout:2795: done (2 left) 2024-09-20T22:53:05.350 INFO:tasks.rados.rados.0.smithi084.stdout:2796: done (1 left) 2024-09-20T22:53:05.350 INFO:tasks.rados.rados.0.smithi084.stdout:2797: done (0 left) 2024-09-20T22:53:05.350 INFO:tasks.rados.rados.0.smithi084.stdout:2798: rollback oid 24 current snap is 316 2024-09-20T22:53:05.350 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 24 to 293 2024-09-20T22:53:05.350 INFO:tasks.rados.rados.0.smithi084.stdout:2799: read oid 50 snap -1 2024-09-20T22:53:05.351 INFO:tasks.rados.rados.0.smithi084.stdout:2799: expect (ObjNum 974 snap 315 seq_num 974) 2024-09-20T22:53:05.351 INFO:tasks.rados.rados.0.smithi084.stdout:2800: copy_from oid 48 from oid 4 current snap is 316 2024-09-20T22:53:05.351 INFO:tasks.rados.rados.0.smithi084.stdout:2801: rollback oid 37 current snap is 316 2024-09-20T22:53:05.351 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 37 to 300 2024-09-20T22:53:05.351 INFO:tasks.rados.rados.0.smithi084.stdout:2802: delete oid 3 current snap is 316 2024-09-20T22:53:05.351 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-20T22:53:05.355 INFO:tasks.rados.rados.0.smithi084.stdout:2800: finishing copy_from to smithi08477203-48 2024-09-20T22:53:05.355 INFO:tasks.rados.rados.0.smithi084.stdout:2800: got expected ENOENT (src dne) 2024-09-20T22:53:05.355 INFO:tasks.rados.rados.0.smithi084.stdout:2800: finishing copy_from racing read to smithi08477203-48 2024-09-20T22:53:05.355 INFO:tasks.rados.rados.0.smithi084.stdout:2801: finishing rollback tid 1 to smithi08477203-37 2024-09-20T22:53:05.355 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 37 v 1238 (ObjNum 81 snap 0 seq_num 2091261232) dirty dne 2024-09-20T22:53:05.355 INFO:tasks.rados.rados.0.smithi084.stdout:2800: done (4 left) 2024-09-20T22:53:05.355 INFO:tasks.rados.rados.0.smithi084.stdout:2801: done (3 left) 2024-09-20T22:53:05.356 INFO:tasks.rados.rados.0.smithi084.stdout:2802: done (2 left) 2024-09-20T22:53:05.356 INFO:tasks.rados.rados.0.smithi084.stdout:2803: rollback oid 45 current snap is 316 2024-09-20T22:53:05.356 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 45 to 285 2024-09-20T22:53:05.356 INFO:tasks.rados.rados.0.smithi084.stdout:2804: delete oid 13 current snap is 316 2024-09-20T22:53:05.357 INFO:tasks.rados.rados.0.smithi084.stdout:2804: done (3 left) 2024-09-20T22:53:05.357 INFO:tasks.rados.rados.0.smithi084.stdout:2805: delete oid 35 current snap is 316 2024-09-20T22:53:05.359 INFO:tasks.rados.rados.0.smithi084.stdout:2798: finishing rollback tid 0 to smithi08477203-24 2024-09-20T22:53:05.360 INFO:tasks.rados.rados.0.smithi084.stdout:2805: done (3 left) 2024-09-20T22:53:05.360 INFO:tasks.rados.rados.0.smithi084.stdout:2806: write (excl) oid 3 current snap is 316 2024-09-20T22:53:05.360 INFO:tasks.rados.rados.0.smithi084.stdout:2806: seq_num 979 ranges {454053=561955,1422342=483679,2368404=85649} 2024-09-20T22:53:05.366 INFO:tasks.rados.rados.0.smithi084.stdout:2806: writing smithi08477203-3 from 454053 to 1016008 tid 1 2024-09-20T22:53:05.371 INFO:tasks.rados.rados.0.smithi084.stdout:2806: writing smithi08477203-3 from 1422342 to 1906021 tid 2 2024-09-20T22:53:05.374 INFO:tasks.rados.rados.0.smithi084.stdout:2806: writing smithi08477203-3 from 2368404 to 2454053 tid 3 2024-09-20T22:53:05.374 INFO:tasks.rados.rados.0.smithi084.stdout:2807: write (excl) oid 33 current snap is 316 2024-09-20T22:53:05.374 INFO:tasks.rados.rados.0.smithi084.stdout:2807: seq_num 980 ranges {470860=637204,1880824=590036} 2024-09-20T22:53:05.375 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:53:05.380 INFO:tasks.rados.rados.0.smithi084.stdout:2807: writing smithi08477203-33 from 470860 to 1108064 tid 1 2024-09-20T22:53:05.388 INFO:tasks.rados.rados.0.smithi084.stdout:2807: writing smithi08477203-33 from 1880824 to 2470860 tid 2 2024-09-20T22:53:05.388 INFO:tasks.rados.rados.0.smithi084.stdout:2808: snap_create 2024-09-20T22:53:05.388 INFO:tasks.rados.rados.0.smithi084.stdout:2798: finishing rollback tid 1 to smithi08477203-24 2024-09-20T22:53:05.388 INFO:tasks.rados.rados.0.smithi084.stdout:2798: finishing rollback tid 2 to smithi08477203-24 2024-09-20T22:53:05.388 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 24 v 1839 (ObjNum 780 snap 252 seq_num 780) dirty exists 2024-09-20T22:53:05.388 INFO:tasks.rados.rados.0.smithi084.stdout:2799: expect (ObjNum 974 snap 315 seq_num 974) 2024-09-20T22:53:05.399 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:05.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:05.400 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:53:05.496 INFO:tasks.rados.rados.0.smithi084.stdout:2803: finishing rollback tid 0 to smithi08477203-45 2024-09-20T22:53:05.496 INFO:tasks.rados.rados.0.smithi084.stdout:2803: finishing rollback tid 1 to smithi08477203-45 2024-09-20T22:53:05.496 INFO:tasks.rados.rados.0.smithi084.stdout:2803: finishing rollback tid 2 to smithi08477203-45 2024-09-20T22:53:05.496 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 45 v 1842 (ObjNum 846 snap 274 seq_num 846) dirty exists 2024-09-20T22:53:05.496 INFO:tasks.rados.rados.0.smithi084.stdout:2806: finishing write tid 1 to smithi08477203-3 2024-09-20T22:53:05.496 INFO:tasks.rados.rados.0.smithi084.stdout:2806: finishing write tid 2 to smithi08477203-3 2024-09-20T22:53:05.496 INFO:tasks.rados.rados.0.smithi084.stdout:2806: finishing write tid 3 to smithi08477203-3 2024-09-20T22:53:05.496 INFO:tasks.rados.rados.0.smithi084.stdout:2806: finishing write tid 4 to smithi08477203-3 2024-09-20T22:53:05.496 INFO:tasks.rados.rados.0.smithi084.stdout:2806: finishing write tid 5 to smithi08477203-3 2024-09-20T22:53:05.496 INFO:tasks.rados.rados.0.smithi084.stdout:2806: oid 3 updating version 0 to 1835 2024-09-20T22:53:05.497 INFO:tasks.rados.rados.0.smithi084.stdout:2806: oid 3 version 1835 is already newer than 1834 2024-09-20T22:53:05.497 INFO:tasks.rados.rados.0.smithi084.stdout:2806: oid 3 updating version 1835 to 1836 2024-09-20T22:53:05.497 INFO:tasks.rados.rados.0.smithi084.stdout:2806: oid 3 version 1836 is already newer than 1833 2024-09-20T22:53:05.497 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 3 v 1836 (ObjNum 979 snap 316 seq_num 979) dirty exists 2024-09-20T22:53:05.497 INFO:tasks.rados.rados.0.smithi084.stdout:2806: left oid 3 (ObjNum 979 snap 316 seq_num 979) 2024-09-20T22:53:05.497 INFO:tasks.rados.rados.0.smithi084.stdout:2807: finishing write tid 1 to smithi08477203-33 2024-09-20T22:53:05.497 INFO:tasks.rados.rados.0.smithi084.stdout:2807: finishing write tid 2 to smithi08477203-33 2024-09-20T22:53:05.497 INFO:tasks.rados.rados.0.smithi084.stdout:2807: finishing write tid 3 to smithi08477203-33 2024-09-20T22:53:05.497 INFO:tasks.rados.rados.0.smithi084.stdout:2807: finishing write tid 4 to smithi08477203-33 2024-09-20T22:53:05.497 INFO:tasks.rados.rados.0.smithi084.stdout:2807: oid 33 updating version 0 to 1256 2024-09-20T22:53:05.497 INFO:tasks.rados.rados.0.smithi084.stdout:2807: oid 33 updating version 1256 to 1257 2024-09-20T22:53:05.497 INFO:tasks.rados.rados.0.smithi084.stdout:2807: oid 33 updating version 1257 to 1258 2024-09-20T22:53:05.497 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 33 v 1258 (ObjNum 980 snap 316 seq_num 980) dirty exists 2024-09-20T22:53:05.497 INFO:tasks.rados.rados.0.smithi084.stdout:2807: left oid 33 (ObjNum 980 snap 316 seq_num 980) 2024-09-20T22:53:05.501 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:53:05.522 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:05.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:05.523 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:53:05.597 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:05 smithi154 ceph-mon[33264]: 5.4 scrub starts 2024-09-20T22:53:05.597 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:05 smithi154 ceph-mon[33264]: 5.4 scrub ok 2024-09-20T22:53:05.597 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:05 smithi154 ceph-mon[33264]: pgmap v957: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 650 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail; 6.0 MiB/s rd, 1.7 MiB/s wr, 16 op/s 2024-09-20T22:53:05.597 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:05 smithi154 ceph-mon[33264]: 5.16 scrub starts 2024-09-20T22:53:05.597 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:05 smithi154 ceph-mon[33264]: 5.16 scrub ok 2024-09-20T22:53:05.597 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:05 smithi154 ceph-mon[33264]: osdmap e704: 8 total, 8 up, 7 in 2024-09-20T22:53:05.597 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:05 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/2464736084' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:53:05.597 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:05 smithi154 ceph-mon[33264]: 3.1a scrub starts 2024-09-20T22:53:05.597 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:05 smithi154 ceph-mon[33264]: 3.1a scrub ok 2024-09-20T22:53:05.614 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:05 smithi084 ceph-mon[40024]: 5.4 scrub starts 2024-09-20T22:53:05.614 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:05 smithi084 ceph-mon[40024]: 5.4 scrub ok 2024-09-20T22:53:05.614 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:05 smithi084 ceph-mon[40024]: pgmap v957: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 650 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail; 6.0 MiB/s rd, 1.7 MiB/s wr, 16 op/s 2024-09-20T22:53:05.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:05 smithi084 ceph-mon[40024]: 5.16 scrub starts 2024-09-20T22:53:05.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:05 smithi084 ceph-mon[40024]: 5.16 scrub ok 2024-09-20T22:53:05.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:05 smithi084 ceph-mon[40024]: osdmap e704: 8 total, 8 up, 7 in 2024-09-20T22:53:05.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:05 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/2464736084' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:53:05.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:05 smithi084 ceph-mon[40024]: 3.1a scrub starts 2024-09-20T22:53:05.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:05 smithi084 ceph-mon[40024]: 3.1a scrub ok 2024-09-20T22:53:05.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:05 smithi084 ceph-mon[29774]: 5.4 scrub starts 2024-09-20T22:53:05.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:05 smithi084 ceph-mon[29774]: 5.4 scrub ok 2024-09-20T22:53:05.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:05 smithi084 ceph-mon[29774]: pgmap v957: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 650 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail; 6.0 MiB/s rd, 1.7 MiB/s wr, 16 op/s 2024-09-20T22:53:05.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:05 smithi084 ceph-mon[29774]: 5.16 scrub starts 2024-09-20T22:53:05.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:05 smithi084 ceph-mon[29774]: 5.16 scrub ok 2024-09-20T22:53:05.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:05 smithi084 ceph-mon[29774]: osdmap e704: 8 total, 8 up, 7 in 2024-09-20T22:53:05.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:05 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/2464736084' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:53:05.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:05 smithi084 ceph-mon[29774]: 3.1a scrub starts 2024-09-20T22:53:05.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:05 smithi084 ceph-mon[29774]: 3.1a scrub ok 2024-09-20T22:53:05.623 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:53:05.627 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-20T22:53:05.627 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:53:05.646 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:05.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:05.647 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:53:05.695 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:53:05.747 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:53:05.769 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:05.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:05.770 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:53:05.870 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:53:05.892 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:05.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:05.893 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:53:05.993 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:53:06.015 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:06.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:06.016 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:53:06.117 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:53:06.138 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:06.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:06.139 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:53:06.239 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:53:06.261 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:06.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:06.262 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:53:06.354 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:06 smithi084 ceph-mon[40024]: 6.f scrub starts 2024-09-20T22:53:06.354 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:06 smithi084 ceph-mon[40024]: 6.f scrub ok 2024-09-20T22:53:06.358 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is unset 2024-09-20T22:53:06.358 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.4 to 1 (10000) 2024-09-20T22:53:06.360 INFO:tasks.rados.rados.0.smithi084.stdout:2798: done (5 left) 2024-09-20T22:53:06.360 INFO:tasks.rados.rados.0.smithi084.stdout:2799: done (4 left) 2024-09-20T22:53:06.361 INFO:tasks.rados.rados.0.smithi084.stdout:2803: done (3 left) 2024-09-20T22:53:06.361 INFO:tasks.rados.rados.0.smithi084.stdout:2806: done (2 left) 2024-09-20T22:53:06.361 INFO:tasks.rados.rados.0.smithi084.stdout:2807: done (1 left) 2024-09-20T22:53:06.361 INFO:tasks.rados.rados.0.smithi084.stdout:2808: done (0 left) 2024-09-20T22:53:06.361 INFO:tasks.rados.rados.0.smithi084.stdout:2809: delete oid 30 current snap is 317 2024-09-20T22:53:06.362 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:53:06.363 INFO:tasks.rados.rados.0.smithi084.stdout:2809: done (0 left) 2024-09-20T22:53:06.363 INFO:tasks.rados.rados.0.smithi084.stdout:2810: copy_from oid 50 from oid 48 current snap is 317 2024-09-20T22:53:06.363 INFO:tasks.rados.rados.0.smithi084.stdout:2811: snap_create 2024-09-20T22:53:06.376 INFO:tasks.rados.rados.0.smithi084.stdout:2810: finishing copy_from racing read to smithi08477203-50 2024-09-20T22:53:06.384 INFO:tasks.rados.rados.0.smithi084.stdout:2810: finishing copy_from to smithi08477203-50 2024-09-20T22:53:06.384 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 50 v 1230 (ObjNum 759 snap 243 seq_num 759) dirty exists 2024-09-20T22:53:06.388 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:06.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:06.389 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:53:06.489 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:53:06.511 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:06.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:06.512 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:53:06.612 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:53:06.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:06 smithi084 ceph-mon[29774]: 6.f scrub starts 2024-09-20T22:53:06.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:06 smithi084 ceph-mon[29774]: 6.f scrub ok 2024-09-20T22:53:06.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:06 smithi084 ceph-mon[29774]: osdmap e705: 8 total, 8 up, 7 in 2024-09-20T22:53:06.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:06 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/3236793828' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-09-20T22:53:06.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:06 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-09-20T22:53:06.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:06 smithi084 ceph-mon[29774]: 1.3 scrub starts 2024-09-20T22:53:06.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:06 smithi084 ceph-mon[29774]: 1.3 scrub ok 2024-09-20T22:53:06.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:06 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/1249633447' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:53:06.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:06 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:53:06.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:06 smithi084 ceph-mon[40024]: osdmap e705: 8 total, 8 up, 7 in 2024-09-20T22:53:06.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:06 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/3236793828' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-09-20T22:53:06.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:06 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-09-20T22:53:06.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:06 smithi084 ceph-mon[40024]: 1.3 scrub starts 2024-09-20T22:53:06.617 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:06 smithi084 ceph-mon[40024]: 1.3 scrub ok 2024-09-20T22:53:06.617 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:06 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/1249633447' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:53:06.617 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:06 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:53:06.635 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:06.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:06.636 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:53:06.736 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:53:06.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:06 smithi154 ceph-mon[33264]: 6.f scrub starts 2024-09-20T22:53:06.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:06 smithi154 ceph-mon[33264]: 6.f scrub ok 2024-09-20T22:53:06.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:06 smithi154 ceph-mon[33264]: osdmap e705: 8 total, 8 up, 7 in 2024-09-20T22:53:06.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:06 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/3236793828' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-09-20T22:53:06.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:06 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-09-20T22:53:06.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:06 smithi154 ceph-mon[33264]: 1.3 scrub starts 2024-09-20T22:53:06.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:06 smithi154 ceph-mon[33264]: 1.3 scrub ok 2024-09-20T22:53:06.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:06 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/1249633447' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:53:06.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:06 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:53:06.759 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:06.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:06.760 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:53:06.819 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.5 dump_blocked_ops 2024-09-20T22:53:06.861 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:53:06.883 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:06.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:06.884 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:53:06.985 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:53:07.007 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:07.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:07.008 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:53:07.068 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:53:07.108 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:53:07.131 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:07.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:07.132 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:53:07.137 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-09-20T22:53:07.138 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph pg dump --format=json 2024-09-20T22:53:07.233 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:53:07.255 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:07.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:07.255 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:53:07.356 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:53:07.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:07 smithi084 ceph-mon[29774]: 4.10 scrub starts 2024-09-20T22:53:07.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:07 smithi084 ceph-mon[29774]: 4.10 scrub ok 2024-09-20T22:53:07.365 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:07 smithi084 ceph-mon[29774]: 6.12 scrub starts 2024-09-20T22:53:07.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:07 smithi084 ceph-mon[29774]: 6.12 scrub ok 2024-09-20T22:53:07.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:07 smithi084 ceph-mon[29774]: pgmap v960: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 650 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail; 6.0 MiB/s rd, 1.7 MiB/s wr, 16 op/s 2024-09-20T22:53:07.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:07 smithi084 ceph-mon[29774]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T22:53:07.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:07 smithi084 ceph-mon[29774]: Cluster is now healthy 2024-09-20T22:53:07.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:07 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-20T22:53:07.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:07 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T22:53:07.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:07 smithi084 ceph-mon[29774]: osdmap e706: 8 total, 8 up, 7 in 2024-09-20T22:53:07.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:07 smithi084 ceph-mon[29774]: 7.1 scrub starts 2024-09-20T22:53:07.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:07 smithi084 ceph-mon[29774]: 7.1 scrub ok 2024-09-20T22:53:07.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:07 smithi084 ceph-mon[29774]: 7.7 scrub starts 2024-09-20T22:53:07.366 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:07 smithi084 ceph-mon[29774]: 7.7 scrub ok 2024-09-20T22:53:07.366 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:07 smithi084 ceph-mon[40024]: 4.10 scrub starts 2024-09-20T22:53:07.367 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:07 smithi084 ceph-mon[40024]: 4.10 scrub ok 2024-09-20T22:53:07.367 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:07 smithi084 ceph-mon[40024]: 6.12 scrub starts 2024-09-20T22:53:07.367 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:07 smithi084 ceph-mon[40024]: 6.12 scrub ok 2024-09-20T22:53:07.367 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:07 smithi084 ceph-mon[40024]: pgmap v960: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 650 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail; 6.0 MiB/s rd, 1.7 MiB/s wr, 16 op/s 2024-09-20T22:53:07.367 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:07 smithi084 ceph-mon[40024]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T22:53:07.367 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:07 smithi084 ceph-mon[40024]: Cluster is now healthy 2024-09-20T22:53:07.367 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:07 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-20T22:53:07.367 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:07 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T22:53:07.367 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:07 smithi084 ceph-mon[40024]: osdmap e706: 8 total, 8 up, 7 in 2024-09-20T22:53:07.367 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:07 smithi084 ceph-mon[40024]: 7.1 scrub starts 2024-09-20T22:53:07.367 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:07 smithi084 ceph-mon[40024]: 7.1 scrub ok 2024-09-20T22:53:07.367 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:07 smithi084 ceph-mon[40024]: 7.7 scrub starts 2024-09-20T22:53:07.367 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:07 smithi084 ceph-mon[40024]: 7.7 scrub ok 2024-09-20T22:53:07.368 INFO:tasks.rados.rados.0.smithi084.stdout:2810: done (1 left) 2024-09-20T22:53:07.368 INFO:tasks.rados.rados.0.smithi084.stdout:2811: done (0 left) 2024-09-20T22:53:07.368 INFO:tasks.rados.rados.0.smithi084.stdout:2812: snap_remove snap 297 2024-09-20T22:53:07.379 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:07.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:07.380 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:53:07.469 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-20T22:53:07.481 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:53:07.483 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:07 smithi154 ceph-mon[33264]: 4.10 scrub starts 2024-09-20T22:53:07.484 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:07 smithi154 ceph-mon[33264]: 4.10 scrub ok 2024-09-20T22:53:07.484 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:07 smithi154 ceph-mon[33264]: 6.12 scrub starts 2024-09-20T22:53:07.484 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:07 smithi154 ceph-mon[33264]: 6.12 scrub ok 2024-09-20T22:53:07.484 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:07 smithi154 ceph-mon[33264]: pgmap v960: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 650 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail; 6.0 MiB/s rd, 1.7 MiB/s wr, 16 op/s 2024-09-20T22:53:07.484 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:07 smithi154 ceph-mon[33264]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T22:53:07.484 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:07 smithi154 ceph-mon[33264]: Cluster is now healthy 2024-09-20T22:53:07.484 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:07 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-20T22:53:07.484 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:07 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T22:53:07.484 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:07 smithi154 ceph-mon[33264]: osdmap e706: 8 total, 8 up, 7 in 2024-09-20T22:53:07.484 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:07 smithi154 ceph-mon[33264]: 7.1 scrub starts 2024-09-20T22:53:07.484 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:07 smithi154 ceph-mon[33264]: 7.1 scrub ok 2024-09-20T22:53:07.484 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:07 smithi154 ceph-mon[33264]: 7.7 scrub starts 2024-09-20T22:53:07.484 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:07 smithi154 ceph-mon[33264]: 7.7 scrub ok 2024-09-20T22:53:07.533 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:07.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:07.534 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:53:07.588 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-20T22:53:07.588 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:53:07.635 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:53:07.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:07.658 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:07.658 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:53:07.759 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:53:07.781 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:07.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:07.782 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:53:07.796 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:53:07.883 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:53:07.905 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:07.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:07.906 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:53:08.006 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:53:08.028 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:08.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:08.028 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:53:08.129 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:53:08.151 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:08.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:08.152 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:53:08.237 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:53:08.252 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:53:08.272 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.5 dump_historic_ops 2024-09-20T22:53:08.273 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:53:08.273 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:53:08.276 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:08.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:08.277 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:53:08.292 INFO:tasks.rados.rados.0.smithi084.stdout:2812: done (0 left) 2024-09-20T22:53:08.293 INFO:tasks.rados.rados.0.smithi084.stdout:2813: read oid 9 snap 301 2024-09-20T22:53:08.293 INFO:tasks.rados.rados.0.smithi084.stdout:2813: expect deleted 2024-09-20T22:53:08.293 INFO:tasks.rados.rados.0.smithi084.stdout:2814: delete oid 28 current snap is 318 2024-09-20T22:53:08.294 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:53:08.294 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:53:08.296 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:53:08 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: ::ffff:172.21.15.154 - - [20/Sep/2024:22:53:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-20T22:53:08.303 INFO:tasks.rados.rados.0.smithi084.stdout:2813: done (1 left) 2024-09-20T22:53:08.303 INFO:tasks.rados.rados.0.smithi084.stdout:2814: done (0 left) 2024-09-20T22:53:08.303 INFO:tasks.rados.rados.0.smithi084.stdout:2815: delete oid 13 current snap is 318 2024-09-20T22:53:08.307 INFO:tasks.rados.rados.0.smithi084.stdout:2815: done (0 left) 2024-09-20T22:53:08.307 INFO:tasks.rados.rados.0.smithi084.stdout:2816: snap_remove snap 266 2024-09-20T22:53:08.318 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:53:08.318 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:53:08.335 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-20T22:53:08.335 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T22:53:08.352 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:53:08.352 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:53:08.370 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:53:08.371 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:53:08.378 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:53:08.387 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:53:08.388 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:53:08.399 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:08.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:08.400 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:53:08.405 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:53:08.405 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:53:08.500 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:53:08.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:08 smithi154 ceph-mon[33264]: 6.5 scrub starts 2024-09-20T22:53:08.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:08 smithi154 ceph-mon[33264]: 6.5 scrub ok 2024-09-20T22:53:08.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:08 smithi154 ceph-mon[33264]: 5.8 deep-scrub starts 2024-09-20T22:53:08.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:08 smithi154 ceph-mon[33264]: 5.8 deep-scrub ok 2024-09-20T22:53:08.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:08 smithi154 ceph-mon[33264]: osdmap e707: 8 total, 8 up, 7 in 2024-09-20T22:53:08.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:08 smithi154 ceph-mon[33264]: 4.5 scrub starts 2024-09-20T22:53:08.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:08 smithi154 ceph-mon[33264]: 4.5 scrub ok 2024-09-20T22:53:08.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:08 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:53:08.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:08 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:53:08.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:08 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/3120902249' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:53:08.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:08 smithi154 ceph-mon[33264]: osdmap e708: 8 total, 8 up, 7 in 2024-09-20T22:53:08.523 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:08.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:08.524 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:53:08.535 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:53:08.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:08 smithi084 ceph-mon[29774]: 6.5 scrub starts 2024-09-20T22:53:08.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:08 smithi084 ceph-mon[29774]: 6.5 scrub ok 2024-09-20T22:53:08.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:08 smithi084 ceph-mon[29774]: 5.8 deep-scrub starts 2024-09-20T22:53:08.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:08 smithi084 ceph-mon[29774]: 5.8 deep-scrub ok 2024-09-20T22:53:08.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:08 smithi084 ceph-mon[29774]: osdmap e707: 8 total, 8 up, 7 in 2024-09-20T22:53:08.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:08 smithi084 ceph-mon[29774]: 4.5 scrub starts 2024-09-20T22:53:08.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:08 smithi084 ceph-mon[29774]: 4.5 scrub ok 2024-09-20T22:53:08.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:08 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:53:08.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:08 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:53:08.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:08 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/3120902249' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:53:08.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:08 smithi084 ceph-mon[29774]: osdmap e708: 8 total, 8 up, 7 in 2024-09-20T22:53:08.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:08 smithi084 ceph-mon[40024]: 6.5 scrub starts 2024-09-20T22:53:08.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:08 smithi084 ceph-mon[40024]: 6.5 scrub ok 2024-09-20T22:53:08.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:08 smithi084 ceph-mon[40024]: 5.8 deep-scrub starts 2024-09-20T22:53:08.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:08 smithi084 ceph-mon[40024]: 5.8 deep-scrub ok 2024-09-20T22:53:08.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:08 smithi084 ceph-mon[40024]: osdmap e707: 8 total, 8 up, 7 in 2024-09-20T22:53:08.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:08 smithi084 ceph-mon[40024]: 4.5 scrub starts 2024-09-20T22:53:08.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:08 smithi084 ceph-mon[40024]: 4.5 scrub ok 2024-09-20T22:53:08.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:08 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:53:08.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:08 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:53:08.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:08 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/3120902249' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:53:08.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:08 smithi084 ceph-mon[40024]: osdmap e708: 8 total, 8 up, 7 in 2024-09-20T22:53:08.625 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:53:08.646 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:08.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:08.647 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:53:08.748 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:53:08.769 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:08.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:08.770 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:53:08.871 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:53:08.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:08.893 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:08.893 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:53:08.994 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:53:09.015 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:09.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:09.016 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:53:09.017 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-20T22:53:09.018 INFO:teuthology.orchestra.run.smithi154.stdout: "size": 20, 2024-09-20T22:53:09.018 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 600, 2024-09-20T22:53:09.018 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [ 2024-09-20T22:53:09.018 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.018 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:554 7.a e128/94 7:5122292a:::smithi08477203-33:head v 128'32, mlcod=119'28)", 2024-09-20T22:53:09.018 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:27.341250+0000", 2024-09-20T22:53:09.018 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 581.67127875799997, 2024-09-20T22:53:09.018 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.051661319999999997, 2024-09-20T22:53:09.018 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:53:09.018 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:53:09.019 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:53:09.019 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:53:09.019 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.019 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:53:09.019 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.341250+0000", 2024-09-20T22:53:09.019 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:53:09.019 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.020 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.020 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:53:09.020 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.341250+0000", 2024-09-20T22:53:09.020 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:53:09.020 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.020 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.020 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:53:09.020 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.341251+0000", 2024-09-20T22:53:09.020 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.7299999999999996e-07 2024-09-20T22:53:09.020 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.021 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.021 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:53:09.021 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.390395+0000", 2024-09-20T22:53:09.021 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.049143539999999999 2024-09-20T22:53:09.021 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.021 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.021 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:53:09.021 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.390413+0000", 2024-09-20T22:53:09.021 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.8709999999999999e-05 2024-09-20T22:53:09.021 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.021 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.021 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:53:09.021 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.390423+0000", 2024-09-20T22:53:09.021 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.4660000000000005e-06 2024-09-20T22:53:09.021 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.022 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.022 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:53:09.022 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.390484+0000", 2024-09-20T22:53:09.022 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.0851999999999999e-05 2024-09-20T22:53:09.022 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.022 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.022 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:53:09.022 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.390504+0000", 2024-09-20T22:53:09.022 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9976999999999999e-05 2024-09-20T22:53:09.022 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.022 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.022 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:53:09.022 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.392875+0000", 2024-09-20T22:53:09.023 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0023713710000000002 2024-09-20T22:53:09.023 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.023 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.023 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:53:09.023 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:27.392912+0000", 2024-09-20T22:53:09.023 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.6831e-05 2024-09-20T22:53:09.023 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:53:09.023 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:53:09.023 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:53:09.023 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.023 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.023 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:2350 7.0 7:064f66ab:::smithi08477203-45:head [stat,delete] snapc 56=[56,55,53,52,51,50,47,3e] ondisk+write+known_if_redirected+supports_pool_eio e260)", 2024-09-20T22:53:09.023 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:45:39.199566+0000", 2024-09-20T22:53:09.024 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 449.81296316800001, 2024-09-20T22:53:09.024 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.027897148, 2024-09-20T22:53:09.024 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:53:09.024 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:53:09.024 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:53:09.024 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:53:09.024 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:53:09.024 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:53:09.024 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 2350 2024-09-20T22:53:09.024 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.024 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:53:09.024 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.024 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:53:09.024 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.199566+0000", 2024-09-20T22:53:09.024 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:53:09.025 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.025 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.025 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:53:09.025 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.199566+0000", 2024-09-20T22:53:09.025 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:53:09.025 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.025 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.025 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:53:09.025 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.199568+0000", 2024-09-20T22:53:09.025 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9290000000000001e-06 2024-09-20T22:53:09.026 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.026 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.026 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:53:09.026 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.199574+0000", 2024-09-20T22:53:09.026 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.3409999999999999e-06 2024-09-20T22:53:09.026 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.027 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.027 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:53:09.027 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.199584+0000", 2024-09-20T22:53:09.027 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0091e-05 2024-09-20T22:53:09.027 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.028 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.028 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:53:09.028 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.199593+0000", 2024-09-20T22:53:09.028 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.1079999999999993e-06 2024-09-20T22:53:09.028 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.028 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.028 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:53:09.028 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.199661+0000", 2024-09-20T22:53:09.028 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.8767000000000001e-05 2024-09-20T22:53:09.028 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.028 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.028 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:53:09.028 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.199765+0000", 2024-09-20T22:53:09.029 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000103886 2024-09-20T22:53:09.029 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.029 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.029 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:53:09.029 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.227463+0000", 2024-09-20T22:53:09.029 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.027698026000000001 2024-09-20T22:53:09.029 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:53:09.029 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:53:09.029 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:53:09.029 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.029 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.029 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:3134 7.2 7:45b54b6b:::smithi08477203-2:head [copy-from ver 151] snapc 77=[77,76,75,74,73,70,6b] ondisk+write+known_if_redirected+supports_pool_eio e328)", 2024-09-20T22:53:09.029 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:46:48.211103+0000", 2024-09-20T22:53:09.029 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 380.80142666099999, 2024-09-20T22:53:09.029 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.030812519, 2024-09-20T22:53:09.030 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:53:09.030 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:53:09.030 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:53:09.030 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:53:09.030 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:53:09.030 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:53:09.030 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 3134 2024-09-20T22:53:09.030 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.030 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:53:09.030 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.030 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:53:09.030 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.211103+0000", 2024-09-20T22:53:09.032 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:53:09.032 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.032 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.032 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:53:09.032 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.211103+0000", 2024-09-20T22:53:09.032 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:53:09.032 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.032 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.032 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:53:09.032 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.211105+0000", 2024-09-20T22:53:09.032 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.3369999999999998e-06 2024-09-20T22:53:09.033 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.033 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.033 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:53:09.033 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.211111+0000", 2024-09-20T22:53:09.034 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.2759999999999997e-06 2024-09-20T22:53:09.034 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.034 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.034 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:53:09.034 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.211123+0000", 2024-09-20T22:53:09.034 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2119999999999999e-05 2024-09-20T22:53:09.034 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.034 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.034 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:53:09.034 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.211131+0000", 2024-09-20T22:53:09.034 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.2899999999999997e-06 2024-09-20T22:53:09.034 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.035 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.035 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:53:09.035 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.211213+0000", 2024-09-20T22:53:09.035 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.2756999999999994e-05 2024-09-20T22:53:09.035 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.035 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.035 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:53:09.035 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.211310+0000", 2024-09-20T22:53:09.035 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.6644999999999994e-05 2024-09-20T22:53:09.035 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.035 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.035 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 1,2", 2024-09-20T22:53:09.035 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.229109+0000", 2024-09-20T22:53:09.036 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.017798587000000001 2024-09-20T22:53:09.036 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.036 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.036 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:53:09.036 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.240265+0000", 2024-09-20T22:53:09.036 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.011156716000000001 2024-09-20T22:53:09.036 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.036 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.036 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:53:09.036 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.241608+0000", 2024-09-20T22:53:09.036 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001343033 2024-09-20T22:53:09.036 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.036 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.037 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:53:09.037 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.241831+0000", 2024-09-20T22:53:09.037 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000223148 2024-09-20T22:53:09.037 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.037 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.037 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:53:09.037 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.241885+0000", 2024-09-20T22:53:09.037 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.3161000000000001e-05 2024-09-20T22:53:09.037 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.037 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.037 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:53:09.037 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.241915+0000", 2024-09-20T22:53:09.037 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.0448999999999999e-05 2024-09-20T22:53:09.037 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:53:09.037 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:53:09.038 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:53:09.038 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.038 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.038 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:3135 7.2 7:45b54b6b:::smithi08477203-2:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e328)", 2024-09-20T22:53:09.038 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:46:48.211153+0000", 2024-09-20T22:53:09.038 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 380.80137668899999, 2024-09-20T22:53:09.038 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.022731534000000001, 2024-09-20T22:53:09.038 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:53:09.038 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:53:09.038 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:53:09.038 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:53:09.038 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:53:09.039 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:53:09.039 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 3135 2024-09-20T22:53:09.039 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.039 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:53:09.039 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.039 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:53:09.039 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.211153+0000", 2024-09-20T22:53:09.039 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:53:09.039 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.039 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.039 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:53:09.039 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.211153+0000", 2024-09-20T22:53:09.039 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:53:09.039 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.040 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.040 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:53:09.040 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.211153+0000", 2024-09-20T22:53:09.040 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.1600000000000001e-07 2024-09-20T22:53:09.040 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.040 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.040 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:53:09.040 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.211157+0000", 2024-09-20T22:53:09.040 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.5470000000000001e-06 2024-09-20T22:53:09.040 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.040 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.040 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:53:09.040 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.211164+0000", 2024-09-20T22:53:09.040 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.2799999999999998e-06 2024-09-20T22:53:09.040 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.041 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.041 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:53:09.041 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.211170+0000", 2024-09-20T22:53:09.041 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.0730000000000001e-06 2024-09-20T22:53:09.041 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.041 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.041 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:53:09.041 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.211634+0000", 2024-09-20T22:53:09.041 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00046401800000000002 2024-09-20T22:53:09.041 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.041 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.041 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for blocked object", 2024-09-20T22:53:09.041 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.211690+0000", 2024-09-20T22:53:09.042 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.5587999999999997e-05 2024-09-20T22:53:09.042 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.042 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.042 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:53:09.042 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.233669+0000", 2024-09-20T22:53:09.042 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.021979240000000001 2024-09-20T22:53:09.042 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.042 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.042 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:53:09.042 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.233762+0000", 2024-09-20T22:53:09.042 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.2658999999999995e-05 2024-09-20T22:53:09.042 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.042 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.042 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:53:09.043 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.233884+0000", 2024-09-20T22:53:09.043 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00012241300000000001 2024-09-20T22:53:09.043 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:53:09.043 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:53:09.043 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:53:09.043 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.043 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.043 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:3136 7.a e328/94 7:5122292a:::smithi08477203-33:head v 328'252, mlcod=322'248)", 2024-09-20T22:53:09.043 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:46:48.223492+0000", 2024-09-20T22:53:09.043 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 380.78903697800001, 2024-09-20T22:53:09.043 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.016662746999999999, 2024-09-20T22:53:09.043 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:53:09.043 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:53:09.043 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:53:09.044 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:53:09.044 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.044 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:53:09.044 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.223492+0000", 2024-09-20T22:53:09.044 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:53:09.044 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.044 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.044 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:53:09.044 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.223492+0000", 2024-09-20T22:53:09.044 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:53:09.044 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.044 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.044 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:53:09.044 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.223493+0000", 2024-09-20T22:53:09.044 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.4299999999999998e-07 2024-09-20T22:53:09.044 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.045 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.045 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:53:09.045 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.228060+0000", 2024-09-20T22:53:09.045 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0045669350000000003 2024-09-20T22:53:09.045 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.045 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.045 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:53:09.045 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.228107+0000", 2024-09-20T22:53:09.045 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.6932999999999999e-05 2024-09-20T22:53:09.045 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.045 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.045 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:53:09.045 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.228127+0000", 2024-09-20T22:53:09.045 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.0777000000000001e-05 2024-09-20T22:53:09.045 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.045 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.046 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:53:09.046 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.228206+0000", 2024-09-20T22:53:09.046 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.8660999999999998e-05 2024-09-20T22:53:09.046 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.046 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.046 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:53:09.046 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.228232+0000", 2024-09-20T22:53:09.046 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.6199000000000001e-05 2024-09-20T22:53:09.046 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.046 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.046 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:53:09.046 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.240103+0000", 2024-09-20T22:53:09.046 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.011871008000000001 2024-09-20T22:53:09.046 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.046 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.047 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:53:09.047 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.240155+0000", 2024-09-20T22:53:09.047 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.1790999999999999e-05 2024-09-20T22:53:09.047 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:53:09.047 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:53:09.047 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:53:09.047 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.047 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.047 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:3589 7.b e363/94 7:dc95255e:::smithi08477203-1:head v 363'419, mlcod=363'417)", 2024-09-20T22:53:09.047 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:47:23.205271+0000", 2024-09-20T22:53:09.047 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 345.80725799700002, 2024-09-20T22:53:09.047 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.014301911, 2024-09-20T22:53:09.047 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:53:09.047 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:53:09.048 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:53:09.048 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:53:09.048 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.048 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:53:09.048 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:23.205271+0000", 2024-09-20T22:53:09.048 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:53:09.048 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.048 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.048 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:53:09.048 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:23.205271+0000", 2024-09-20T22:53:09.048 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:53:09.048 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.048 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.048 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:53:09.049 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:23.205273+0000", 2024-09-20T22:53:09.049 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.409e-06 2024-09-20T22:53:09.049 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.049 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.049 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:53:09.049 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:23.210044+0000", 2024-09-20T22:53:09.049 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0047712609999999997 2024-09-20T22:53:09.049 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.049 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.049 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:53:09.049 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:23.210064+0000", 2024-09-20T22:53:09.049 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.0421e-05 2024-09-20T22:53:09.049 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.050 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.050 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:53:09.050 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:23.210074+0000", 2024-09-20T22:53:09.050 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.4329999999999993e-06 2024-09-20T22:53:09.050 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.050 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.050 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:53:09.050 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:23.210130+0000", 2024-09-20T22:53:09.050 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.6286000000000003e-05 2024-09-20T22:53:09.050 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.050 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.050 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:53:09.050 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:23.210148+0000", 2024-09-20T22:53:09.050 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.7895999999999999e-05 2024-09-20T22:53:09.051 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.051 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.051 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:53:09.051 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:23.219532+0000", 2024-09-20T22:53:09.051 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.009383677 2024-09-20T22:53:09.051 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.051 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.051 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:53:09.051 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:23.219573+0000", 2024-09-20T22:53:09.051 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.1527999999999999e-05 2024-09-20T22:53:09.051 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:53:09.051 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:53:09.051 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:53:09.051 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.051 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.052 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:4444 7.0 7:05bb442d:::smithi08477203-24:head [copy-from ver 421] snapc b8=[b8,b7,b6,b5,b4,b3,b2,b1,b0,ae,a9,a7,a6,a5,9e,9c,96,95,87,86] ondisk+write+known_if_redirected+supports_pool_eio e447)", 2024-09-20T22:53:09.052 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:48:47.227411+0000", 2024-09-20T22:53:09.052 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 261.78511855300002, 2024-09-20T22:53:09.052 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.033566952999999997, 2024-09-20T22:53:09.052 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:53:09.052 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:53:09.052 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:53:09.052 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:53:09.052 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:53:09.052 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:53:09.052 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 4444 2024-09-20T22:53:09.052 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.052 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:53:09.052 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.052 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:53:09.053 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.227411+0000", 2024-09-20T22:53:09.053 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:53:09.053 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.053 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.053 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:53:09.053 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.227411+0000", 2024-09-20T22:53:09.053 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:53:09.053 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.053 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.053 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:53:09.053 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.227413+0000", 2024-09-20T22:53:09.053 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.057e-06 2024-09-20T22:53:09.053 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.053 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.053 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:53:09.054 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.227422+0000", 2024-09-20T22:53:09.054 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.3219999999999999e-06 2024-09-20T22:53:09.054 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.054 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.054 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:53:09.054 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.227442+0000", 2024-09-20T22:53:09.054 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9449999999999998e-05 2024-09-20T22:53:09.054 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.054 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.054 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:53:09.054 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.227452+0000", 2024-09-20T22:53:09.054 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0699999999999999e-05 2024-09-20T22:53:09.054 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.054 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.055 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:53:09.055 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.227531+0000", 2024-09-20T22:53:09.055 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.8993999999999996e-05 2024-09-20T22:53:09.055 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.055 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.055 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op must wait for map", 2024-09-20T22:53:09.055 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.227542+0000", 2024-09-20T22:53:09.055 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0797e-05 2024-09-20T22:53:09.055 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.055 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.055 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:53:09.055 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.229501+0000", 2024-09-20T22:53:09.055 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0019594080000000002 2024-09-20T22:53:09.055 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.056 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.056 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:53:09.056 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.229616+0000", 2024-09-20T22:53:09.056 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000114577 2024-09-20T22:53:09.056 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.056 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.056 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 1,3", 2024-09-20T22:53:09.056 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.241752+0000", 2024-09-20T22:53:09.056 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.012135920999999999 2024-09-20T22:53:09.056 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.056 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.057 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:53:09.057 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.249601+0000", 2024-09-20T22:53:09.057 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0078493120000000006 2024-09-20T22:53:09.057 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.057 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.057 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:53:09.057 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.260731+0000", 2024-09-20T22:53:09.057 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.011129781 2024-09-20T22:53:09.057 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.057 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.057 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:53:09.057 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.260861+0000", 2024-09-20T22:53:09.057 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00013027899999999999 2024-09-20T22:53:09.057 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.058 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.058 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:53:09.058 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.260913+0000", 2024-09-20T22:53:09.058 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.2172999999999997e-05 2024-09-20T22:53:09.058 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.058 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.058 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:53:09.058 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.260978+0000", 2024-09-20T22:53:09.058 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.4182000000000003e-05 2024-09-20T22:53:09.058 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:53:09.058 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:53:09.058 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:53:09.058 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.059 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.059 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:4445 7.0 7:05bb442d:::smithi08477203-24:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e447)", 2024-09-20T22:53:09.059 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:48:47.227478+0000", 2024-09-20T22:53:09.059 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 261.78505161599998, 2024-09-20T22:53:09.059 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.019465758999999999, 2024-09-20T22:53:09.059 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:53:09.059 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:53:09.059 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:53:09.059 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:53:09.059 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:53:09.059 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:53:09.059 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 4445 2024-09-20T22:53:09.059 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.059 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:53:09.059 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.060 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:53:09.060 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.227478+0000", 2024-09-20T22:53:09.060 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:53:09.060 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.060 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.060 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:53:09.060 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.227478+0000", 2024-09-20T22:53:09.060 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:53:09.060 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.060 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.060 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:53:09.060 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.227478+0000", 2024-09-20T22:53:09.061 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.2200000000000004e-07 2024-09-20T22:53:09.061 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.061 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.061 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:53:09.061 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.227481+0000", 2024-09-20T22:53:09.061 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.0060000000000001e-06 2024-09-20T22:53:09.061 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.061 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.061 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:53:09.061 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.227500+0000", 2024-09-20T22:53:09.061 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9071000000000001e-05 2024-09-20T22:53:09.061 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.061 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.061 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:53:09.062 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.227507+0000", 2024-09-20T22:53:09.062 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.1779999999999997e-06 2024-09-20T22:53:09.062 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.062 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.062 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:53:09.062 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.227628+0000", 2024-09-20T22:53:09.062 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000120627 2024-09-20T22:53:09.062 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.062 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.062 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting_for_map not empty", 2024-09-20T22:53:09.062 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.227633+0000", 2024-09-20T22:53:09.062 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.4730000000000002e-06 2024-09-20T22:53:09.062 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.062 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.063 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:53:09.063 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.229766+0000", 2024-09-20T22:53:09.063 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0021332339999999999 2024-09-20T22:53:09.063 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.063 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.063 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for blocked object", 2024-09-20T22:53:09.063 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.229814+0000", 2024-09-20T22:53:09.063 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.7746e-05 2024-09-20T22:53:09.063 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.063 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.063 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:53:09.063 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.246707+0000", 2024-09-20T22:53:09.063 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.016893542000000001 2024-09-20T22:53:09.063 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.064 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.064 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:53:09.065 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.246863+0000", 2024-09-20T22:53:09.065 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000156105 2024-09-20T22:53:09.065 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.065 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.065 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:53:09.065 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.246943+0000", 2024-09-20T22:53:09.065 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.0154999999999994e-05 2024-09-20T22:53:09.065 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:53:09.065 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:53:09.066 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:53:09.066 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.066 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.066 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:5011 7.b e505/94 7:df7b2b73:::smithi08477203-10:head v 505'722, mlcod=503'718)", 2024-09-20T22:53:09.066 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:49:45.305974+0000", 2024-09-20T22:53:09.066 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 203.70655551999999, 2024-09-20T22:53:09.066 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.014321993, 2024-09-20T22:53:09.066 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:53:09.066 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:53:09.066 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:53:09.066 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:53:09.066 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.066 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:53:09.066 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:45.305974+0000", 2024-09-20T22:53:09.067 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:53:09.067 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.067 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.067 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:53:09.067 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:45.305974+0000", 2024-09-20T22:53:09.067 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:53:09.067 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.067 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.067 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:53:09.067 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:45.305974+0000", 2024-09-20T22:53:09.067 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.6199999999999997e-07 2024-09-20T22:53:09.067 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.067 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.067 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:53:09.068 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:45.311756+0000", 2024-09-20T22:53:09.068 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0057811019999999998 2024-09-20T22:53:09.068 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.068 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.068 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:53:09.068 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:45.311773+0000", 2024-09-20T22:53:09.068 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.7757999999999998e-05 2024-09-20T22:53:09.068 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.068 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.068 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:53:09.068 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:45.311783+0000", 2024-09-20T22:53:09.069 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.6639999999999993e-06 2024-09-20T22:53:09.069 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.069 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.069 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:53:09.069 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:45.311873+0000", 2024-09-20T22:53:09.069 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.0062000000000006e-05 2024-09-20T22:53:09.069 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.069 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.069 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:53:09.069 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:45.311900+0000", 2024-09-20T22:53:09.069 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.6475999999999999e-05 2024-09-20T22:53:09.070 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.070 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.070 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:53:09.070 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:45.320251+0000", 2024-09-20T22:53:09.070 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0083511560000000002 2024-09-20T22:53:09.070 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.070 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.070 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:53:09.070 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:45.320296+0000", 2024-09-20T22:53:09.070 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.5012999999999997e-05 2024-09-20T22:53:09.070 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:53:09.070 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:53:09.071 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:53:09.071 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.071 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.071 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:5659 7.b e570/94 7:dc95255e:::smithi08477203-1:head v 570'831, mlcod=570'827)", 2024-09-20T22:53:09.071 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:50:51.285735+0000", 2024-09-20T22:53:09.071 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 137.72679414800001, 2024-09-20T22:53:09.071 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.016481638999999999, 2024-09-20T22:53:09.071 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:53:09.071 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:53:09.071 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:53:09.071 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:53:09.071 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.071 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:53:09.071 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:51.285735+0000", 2024-09-20T22:53:09.072 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:53:09.072 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.072 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.072 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:53:09.072 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:51.285735+0000", 2024-09-20T22:53:09.072 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:53:09.072 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.072 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.072 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:53:09.072 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:51.285736+0000", 2024-09-20T22:53:09.072 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.5099999999999998e-07 2024-09-20T22:53:09.072 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.072 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.072 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:53:09.072 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:51.293158+0000", 2024-09-20T22:53:09.073 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0074220750000000002 2024-09-20T22:53:09.073 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.073 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.073 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:53:09.073 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:51.293175+0000", 2024-09-20T22:53:09.073 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.6401000000000001e-05 2024-09-20T22:53:09.073 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.073 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.073 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:53:09.073 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:51.293183+0000", 2024-09-20T22:53:09.073 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.8100000000000004e-06 2024-09-20T22:53:09.074 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.074 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.074 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:53:09.074 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:51.293262+0000", 2024-09-20T22:53:09.074 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.8527000000000002e-05 2024-09-20T22:53:09.074 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.074 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.074 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:53:09.074 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:51.293289+0000", 2024-09-20T22:53:09.074 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.7620999999999999e-05 2024-09-20T22:53:09.074 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.074 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.074 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:53:09.074 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:51.302173+0000", 2024-09-20T22:53:09.075 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0088831139999999992 2024-09-20T22:53:09.075 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.075 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.075 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:53:09.075 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:51.302217+0000", 2024-09-20T22:53:09.075 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.4140000000000001e-05 2024-09-20T22:53:09.075 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:53:09.075 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:53:09.075 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:53:09.075 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.075 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.075 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:6487 7.b 7:d3ca3619:::smithi08477203-39:head [copy-from ver 1231] snapc 121=[121,120,11f,11e,11d,11b,119,118,116,115,114,112,110,10e,10c,107,105,103,fd,fb,fa,f8,f5,ed,d7,c2,bf] ondisk+write+known_if_redirected+supports_pool_eio e653)", 2024-09-20T22:53:09.075 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:52:13.479898+0000", 2024-09-20T22:53:09.076 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 55.532631039000002, 2024-09-20T22:53:09.076 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.023339255, 2024-09-20T22:53:09.076 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:53:09.076 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:53:09.076 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:53:09.076 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:53:09.076 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:53:09.076 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:53:09.076 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 6487 2024-09-20T22:53:09.076 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.076 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:53:09.076 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.076 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:53:09.076 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:13.479898+0000", 2024-09-20T22:53:09.076 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:53:09.077 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.077 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.077 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:53:09.077 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:13.479898+0000", 2024-09-20T22:53:09.077 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:53:09.077 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.077 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.077 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:53:09.077 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:13.479900+0000", 2024-09-20T22:53:09.077 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.415e-06 2024-09-20T22:53:09.077 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.077 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.077 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:53:09.077 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:13.479907+0000", 2024-09-20T22:53:09.078 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.0740000000000004e-06 2024-09-20T22:53:09.078 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.078 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.078 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:53:09.078 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:13.479916+0000", 2024-09-20T22:53:09.078 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.6420000000000002e-06 2024-09-20T22:53:09.078 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.078 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.078 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:53:09.078 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:13.479922+0000", 2024-09-20T22:53:09.078 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.1070000000000001e-06 2024-09-20T22:53:09.078 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.078 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.079 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:53:09.079 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:13.479981+0000", 2024-09-20T22:53:09.079 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.8613000000000003e-05 2024-09-20T22:53:09.079 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.079 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.079 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:53:09.079 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:13.480095+0000", 2024-09-20T22:53:09.079 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00011437799999999999 2024-09-20T22:53:09.079 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.079 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.079 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 2,6", 2024-09-20T22:53:09.079 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:13.496711+0000", 2024-09-20T22:53:09.080 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.016615938 2024-09-20T22:53:09.080 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.080 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.080 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:53:09.080 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:13.502323+0000", 2024-09-20T22:53:09.080 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0056115799999999997 2024-09-20T22:53:09.080 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.080 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.080 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:53:09.080 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:13.502878+0000", 2024-09-20T22:53:09.080 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00055555000000000003 2024-09-20T22:53:09.080 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.080 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.080 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:53:09.081 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:13.503163+0000", 2024-09-20T22:53:09.081 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000284037 2024-09-20T22:53:09.081 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.081 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.081 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:53:09.081 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:13.503201+0000", 2024-09-20T22:53:09.081 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.8569000000000003e-05 2024-09-20T22:53:09.081 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.081 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.081 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:53:09.081 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:13.503237+0000", 2024-09-20T22:53:09.081 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.6352e-05 2024-09-20T22:53:09.081 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:53:09.081 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:53:09.082 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:53:09.082 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.082 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.082 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:6488 7.b 7:d3ca3619:::smithi08477203-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e653)", 2024-09-20T22:53:09.082 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:52:13.479939+0000", 2024-09-20T22:53:09.082 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 55.532590362000001, 2024-09-20T22:53:09.082 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.019688051000000002, 2024-09-20T22:53:09.082 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:53:09.082 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:53:09.082 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:53:09.082 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:53:09.082 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:53:09.082 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:53:09.083 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 6488 2024-09-20T22:53:09.083 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.083 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:53:09.083 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.083 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:53:09.083 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:13.479939+0000", 2024-09-20T22:53:09.083 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:53:09.083 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.083 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.083 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:53:09.083 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:13.479939+0000", 2024-09-20T22:53:09.083 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:53:09.083 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.083 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.083 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:53:09.084 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:13.479940+0000", 2024-09-20T22:53:09.084 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.7199999999999998e-07 2024-09-20T22:53:09.084 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.084 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.084 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:53:09.084 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:13.479942+0000", 2024-09-20T22:53:09.084 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.853e-06 2024-09-20T22:53:09.084 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.084 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.084 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:53:09.084 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:13.479951+0000", 2024-09-20T22:53:09.084 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.1100000000000003e-06 2024-09-20T22:53:09.084 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.084 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.084 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:53:09.085 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:13.479956+0000", 2024-09-20T22:53:09.085 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.4190000000000001e-06 2024-09-20T22:53:09.085 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.085 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.085 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:53:09.085 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:13.480258+0000", 2024-09-20T22:53:09.085 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00030201400000000003 2024-09-20T22:53:09.085 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.085 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.085 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for blocked object", 2024-09-20T22:53:09.085 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:13.480300+0000", 2024-09-20T22:53:09.085 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.1816000000000001e-05 2024-09-20T22:53:09.086 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.086 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.086 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:53:09.086 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:13.499455+0000", 2024-09-20T22:53:09.086 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.019155583 2024-09-20T22:53:09.086 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.086 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.086 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:53:09.086 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:13.499533+0000", 2024-09-20T22:53:09.086 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.7972000000000005e-05 2024-09-20T22:53:09.086 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.086 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.086 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:53:09.086 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:13.499627+0000", 2024-09-20T22:53:09.087 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.3611999999999998e-05 2024-09-20T22:53:09.087 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:53:09.087 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:53:09.087 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:53:09.087 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.087 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.087 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:6770 7.0 7:064f66ab:::smithi08477203-45:head [copy-from ver 1338] snapc 135=[135,134,133,132,131,130,12f,12e,12d,12c,12b,12a,128,127,126,124,122,121,11f,11b,119,112,10e,10c,107,103,fb,fa,f8,ed,d7,c2,bf] ondisk+write+known_if_redirected+supports_pool_eio e687)", 2024-09-20T22:53:09.087 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:52:47.286358+0000", 2024-09-20T22:53:09.087 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 21.726171709999999, 2024-09-20T22:53:09.087 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.024891383, 2024-09-20T22:53:09.087 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:53:09.087 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:53:09.087 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:53:09.087 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:53:09.088 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:53:09.088 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:53:09.088 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 6770 2024-09-20T22:53:09.088 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.088 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:53:09.088 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.088 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:53:09.088 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:47.286358+0000", 2024-09-20T22:53:09.088 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:53:09.088 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.088 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.088 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:53:09.088 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:47.286358+0000", 2024-09-20T22:53:09.088 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:53:09.088 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.089 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.089 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:53:09.089 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:47.286360+0000", 2024-09-20T22:53:09.089 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.1260000000000001e-06 2024-09-20T22:53:09.089 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.089 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.089 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:53:09.089 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:47.286369+0000", 2024-09-20T22:53:09.089 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.431e-06 2024-09-20T22:53:09.089 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.089 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.089 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:53:09.089 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:47.286391+0000", 2024-09-20T22:53:09.089 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.1824e-05 2024-09-20T22:53:09.089 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.090 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.090 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:53:09.090 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:47.286402+0000", 2024-09-20T22:53:09.090 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0801e-05 2024-09-20T22:53:09.090 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.090 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.090 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:53:09.090 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:47.286608+0000", 2024-09-20T22:53:09.090 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00020635400000000001 2024-09-20T22:53:09.090 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.090 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.090 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op must wait for map", 2024-09-20T22:53:09.090 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:47.286616+0000", 2024-09-20T22:53:09.091 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.7389999999999999e-06 2024-09-20T22:53:09.091 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.091 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.091 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:53:09.091 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:47.289915+0000", 2024-09-20T22:53:09.091 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.003299022 2024-09-20T22:53:09.091 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.091 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.091 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:53:09.091 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:47.290021+0000", 2024-09-20T22:53:09.091 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000106267 2024-09-20T22:53:09.091 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.091 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.091 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 2,3", 2024-09-20T22:53:09.091 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:47.299972+0000", 2024-09-20T22:53:09.092 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0099504399999999996 2024-09-20T22:53:09.092 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.092 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.092 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:53:09.092 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:47.304349+0000", 2024-09-20T22:53:09.092 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0043777549999999997 2024-09-20T22:53:09.092 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.092 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.092 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:53:09.092 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:47.310757+0000", 2024-09-20T22:53:09.092 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0064077650000000002 2024-09-20T22:53:09.092 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.092 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.092 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:53:09.092 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:47.311157+0000", 2024-09-20T22:53:09.093 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00039990000000000002 2024-09-20T22:53:09.093 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.093 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.093 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:53:09.093 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:47.311206+0000", 2024-09-20T22:53:09.093 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.8906000000000002e-05 2024-09-20T22:53:09.093 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.093 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.093 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:53:09.093 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:47.311249+0000", 2024-09-20T22:53:09.093 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.3053000000000003e-05 2024-09-20T22:53:09.093 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:53:09.093 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:53:09.093 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:53:09.093 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.093 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.094 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:6771 7.0 7:064f66ab:::smithi08477203-45:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e687)", 2024-09-20T22:53:09.094 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:52:47.286426+0000", 2024-09-20T22:53:09.094 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 21.726102856000001, 2024-09-20T22:53:09.094 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.017336328000000002, 2024-09-20T22:53:09.094 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:53:09.094 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:53:09.094 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:53:09.094 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:53:09.094 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:53:09.094 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:53:09.094 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 6771 2024-09-20T22:53:09.094 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.094 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:53:09.094 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.094 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:53:09.095 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:47.286426+0000", 2024-09-20T22:53:09.095 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:53:09.095 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.095 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.095 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:53:09.095 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:47.286426+0000", 2024-09-20T22:53:09.095 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:53:09.095 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.095 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.095 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:53:09.095 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:47.286427+0000", 2024-09-20T22:53:09.095 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.2099999999999996e-07 2024-09-20T22:53:09.095 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.095 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.096 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:53:09.096 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:47.286432+0000", 2024-09-20T22:53:09.096 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.1030000000000001e-06 2024-09-20T22:53:09.096 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.097 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.097 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:53:09.097 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:47.286440+0000", 2024-09-20T22:53:09.097 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.8990000000000001e-06 2024-09-20T22:53:09.097 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.098 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.098 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:53:09.098 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:47.286446+0000", 2024-09-20T22:53:09.098 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.8089999999999998e-06 2024-09-20T22:53:09.098 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.098 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.098 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:53:09.098 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:47.286640+0000", 2024-09-20T22:53:09.098 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00019450899999999999 2024-09-20T22:53:09.098 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.098 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.098 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting_for_map not empty", 2024-09-20T22:53:09.098 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:47.286648+0000", 2024-09-20T22:53:09.099 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.8150000000000006e-06 2024-09-20T22:53:09.099 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.099 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.099 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:53:09.099 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:47.290185+0000", 2024-09-20T22:53:09.099 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0035371030000000002 2024-09-20T22:53:09.099 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.099 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.099 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for blocked object", 2024-09-20T22:53:09.099 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:47.290239+0000", 2024-09-20T22:53:09.099 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.3671000000000002e-05 2024-09-20T22:53:09.099 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.099 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.100 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:53:09.100 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:47.303495+0000", 2024-09-20T22:53:09.100 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.013256396 2024-09-20T22:53:09.100 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.100 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.100 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:53:09.100 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:47.303636+0000", 2024-09-20T22:53:09.100 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00014082600000000001 2024-09-20T22:53:09.100 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.100 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.100 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:53:09.100 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:47.303763+0000", 2024-09-20T22:53:09.100 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00012657599999999999 2024-09-20T22:53:09.100 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:53:09.101 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:53:09.101 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:53:09.101 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.101 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.101 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:6790 7.0 7:064f66ab:::smithi08477203-45:head [copy-from ver 1815] snapc 135=[135,134,133,132,131,130,12f,12e,12d,12c,12b,12a,128,127,126,124,122,121,11f,11b,119,112,10e,10c,107,103,fb,fa,f8,ed,d7,c2,bf] ondisk+write+known_if_redirected+supports_pool_eio e687)", 2024-09-20T22:53:09.101 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:52:47.698638+0000", 2024-09-20T22:53:09.101 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 21.313891664, 2024-09-20T22:53:09.101 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.029923735, 2024-09-20T22:53:09.101 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:53:09.101 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:53:09.101 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:53:09.101 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:53:09.101 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:53:09.101 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:53:09.101 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 6790 2024-09-20T22:53:09.102 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.102 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:53:09.102 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.102 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:53:09.102 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:47.698638+0000", 2024-09-20T22:53:09.102 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:53:09.102 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.102 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.102 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:53:09.102 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:47.698638+0000", 2024-09-20T22:53:09.102 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:53:09.102 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.102 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.102 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:53:09.103 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:47.698639+0000", 2024-09-20T22:53:09.103 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.5880000000000001e-06 2024-09-20T22:53:09.103 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.103 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.103 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:53:09.103 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:47.698647+0000", 2024-09-20T22:53:09.103 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.6040000000000004e-06 2024-09-20T22:53:09.103 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.103 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.103 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:53:09.103 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:47.698662+0000", 2024-09-20T22:53:09.103 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4997999999999999e-05 2024-09-20T22:53:09.103 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.103 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.104 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:53:09.104 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:47.698669+0000", 2024-09-20T22:53:09.104 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.0400000000000004e-06 2024-09-20T22:53:09.104 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.104 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.104 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:53:09.104 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:47.698744+0000", 2024-09-20T22:53:09.104 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.5018999999999994e-05 2024-09-20T22:53:09.104 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.104 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.104 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:53:09.104 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:47.698841+0000", 2024-09-20T22:53:09.104 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.7229e-05 2024-09-20T22:53:09.104 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.104 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.105 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 2,3", 2024-09-20T22:53:09.105 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:47.711667+0000", 2024-09-20T22:53:09.105 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.012825731999999999 2024-09-20T22:53:09.105 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.105 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.105 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:53:09.105 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:47.717229+0000", 2024-09-20T22:53:09.105 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0055624200000000002 2024-09-20T22:53:09.105 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.105 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.105 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:53:09.106 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:47.726665+0000", 2024-09-20T22:53:09.106 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0094358689999999995 2024-09-20T22:53:09.106 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.106 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.106 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:53:09.106 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:47.728457+0000", 2024-09-20T22:53:09.106 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0017914789999999999 2024-09-20T22:53:09.106 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.106 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.106 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:53:09.106 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:47.728511+0000", 2024-09-20T22:53:09.106 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.3953000000000003e-05 2024-09-20T22:53:09.106 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.107 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.107 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:53:09.107 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:47.728561+0000", 2024-09-20T22:53:09.107 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.0803999999999997e-05 2024-09-20T22:53:09.107 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:53:09.107 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:53:09.107 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:53:09.107 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.107 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.107 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:6791 7.0 7:064f66ab:::smithi08477203-45:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e687)", 2024-09-20T22:53:09.107 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:52:47.698687+0000", 2024-09-20T22:53:09.107 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 21.313842374, 2024-09-20T22:53:09.107 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.017803079999999999, 2024-09-20T22:53:09.107 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:53:09.107 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:53:09.108 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:53:09.108 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:53:09.108 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:53:09.108 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:53:09.108 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 6791 2024-09-20T22:53:09.108 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.108 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:53:09.108 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.108 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:53:09.108 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:47.698687+0000", 2024-09-20T22:53:09.108 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:53:09.108 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.108 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.108 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:53:09.109 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:47.698687+0000", 2024-09-20T22:53:09.109 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:53:09.109 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.109 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.109 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:53:09.109 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:47.698687+0000", 2024-09-20T22:53:09.109 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.2099999999999997e-07 2024-09-20T22:53:09.109 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.109 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.109 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:53:09.109 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:47.698690+0000", 2024-09-20T22:53:09.109 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.067e-06 2024-09-20T22:53:09.109 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.109 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.109 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:53:09.110 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:47.698702+0000", 2024-09-20T22:53:09.110 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1222e-05 2024-09-20T22:53:09.110 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.110 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.110 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:53:09.110 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:47.698708+0000", 2024-09-20T22:53:09.110 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.8479999999999997e-06 2024-09-20T22:53:09.110 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.110 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.110 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:53:09.110 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:47.699009+0000", 2024-09-20T22:53:09.110 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00030181000000000001 2024-09-20T22:53:09.110 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.111 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.111 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for blocked object", 2024-09-20T22:53:09.111 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:47.699075+0000", 2024-09-20T22:53:09.111 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.6021999999999999e-05 2024-09-20T22:53:09.111 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.111 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.111 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:53:09.111 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:47.716226+0000", 2024-09-20T22:53:09.111 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.017150657 2024-09-20T22:53:09.111 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.111 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.111 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:53:09.111 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:47.716352+0000", 2024-09-20T22:53:09.111 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00012609599999999999 2024-09-20T22:53:09.112 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.112 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.112 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:53:09.112 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:47.716490+0000", 2024-09-20T22:53:09.112 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000137837 2024-09-20T22:53:09.112 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:53:09.112 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:53:09.112 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:53:09.112 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.112 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.112 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:6816 7.b 7:dc95255e:::smithi08477203-1:head [copy-from ver 751] snapc 136=[136,135,134,133,132,131,130,12f,12e,12d,12c,12b,12a,128,127,126,124,122,121,11f,11b,119,112,10e,10c,107,103,fb,fa,f8,ed,d7,c2,bf] ondisk+write+known_if_redirected+supports_pool_eio e688)", 2024-09-20T22:53:09.112 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:52:48.282781+0000", 2024-09-20T22:53:09.112 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 20.729748067999999, 2024-09-20T22:53:09.112 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.036288688999999999, 2024-09-20T22:53:09.112 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:53:09.113 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:53:09.113 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:53:09.113 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:53:09.113 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:53:09.113 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:53:09.113 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 6816 2024-09-20T22:53:09.113 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.113 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:53:09.113 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.113 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:53:09.113 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.282781+0000", 2024-09-20T22:53:09.113 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:53:09.113 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.113 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.113 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:53:09.114 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.282781+0000", 2024-09-20T22:53:09.114 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:53:09.114 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.114 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.114 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:53:09.114 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.282783+0000", 2024-09-20T22:53:09.114 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.381e-06 2024-09-20T22:53:09.114 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.114 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.114 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:53:09.114 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.282789+0000", 2024-09-20T22:53:09.114 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.2210000000000002e-06 2024-09-20T22:53:09.115 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.115 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.115 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:53:09.115 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.282800+0000", 2024-09-20T22:53:09.115 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1581e-05 2024-09-20T22:53:09.115 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.115 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.115 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:53:09.115 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.282812+0000", 2024-09-20T22:53:09.115 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1797000000000001e-05 2024-09-20T22:53:09.115 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.115 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.115 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:53:09.116 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.282886+0000", 2024-09-20T22:53:09.116 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.4327999999999997e-05 2024-09-20T22:53:09.116 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.116 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.116 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op must wait for map", 2024-09-20T22:53:09.116 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.282896+0000", 2024-09-20T22:53:09.116 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.7410000000000004e-06 2024-09-20T22:53:09.116 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.116 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.116 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:53:09.116 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.285446+0000", 2024-09-20T22:53:09.116 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0025500319999999998 2024-09-20T22:53:09.116 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.116 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.117 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:53:09.117 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.285598+0000", 2024-09-20T22:53:09.117 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000151302 2024-09-20T22:53:09.117 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.117 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.117 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 2,6", 2024-09-20T22:53:09.117 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.300505+0000", 2024-09-20T22:53:09.117 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.014907204 2024-09-20T22:53:09.117 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.117 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.117 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:53:09.117 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.309549+0000", 2024-09-20T22:53:09.117 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0090438819999999996 2024-09-20T22:53:09.117 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.117 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.118 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:53:09.118 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.318842+0000", 2024-09-20T22:53:09.118 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0092930649999999997 2024-09-20T22:53:09.118 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.118 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.118 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:53:09.118 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.318947+0000", 2024-09-20T22:53:09.118 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00010507500000000001 2024-09-20T22:53:09.118 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.118 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.118 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:53:09.118 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.319009+0000", 2024-09-20T22:53:09.118 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.2607000000000006e-05 2024-09-20T22:53:09.118 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.119 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.119 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:53:09.119 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.319070+0000", 2024-09-20T22:53:09.119 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.0473000000000002e-05 2024-09-20T22:53:09.119 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:53:09.119 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:53:09.119 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:53:09.119 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.119 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.119 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:6817 7.b 7:dc95255e:::smithi08477203-1:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e688)", 2024-09-20T22:53:09.119 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:52:48.282837+0000", 2024-09-20T22:53:09.119 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 20.729692500999999, 2024-09-20T22:53:09.119 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.022790719000000001, 2024-09-20T22:53:09.119 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:53:09.119 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:53:09.119 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:53:09.120 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:53:09.120 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:53:09.120 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:53:09.120 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 6817 2024-09-20T22:53:09.120 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.120 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:53:09.120 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.120 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:53:09.120 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.282837+0000", 2024-09-20T22:53:09.120 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:53:09.120 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.120 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.120 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:53:09.120 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.282837+0000", 2024-09-20T22:53:09.121 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:53:09.121 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.121 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.121 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:53:09.121 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.282837+0000", 2024-09-20T22:53:09.121 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.0200000000000002e-07 2024-09-20T22:53:09.121 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.121 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.121 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:53:09.121 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.282840+0000", 2024-09-20T22:53:09.121 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.157e-06 2024-09-20T22:53:09.121 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.121 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.121 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:53:09.121 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.282847+0000", 2024-09-20T22:53:09.121 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.0159999999999997e-06 2024-09-20T22:53:09.122 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.122 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.122 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:53:09.122 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.282852+0000", 2024-09-20T22:53:09.122 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.6250000000000004e-06 2024-09-20T22:53:09.122 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.122 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.122 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:53:09.122 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.282955+0000", 2024-09-20T22:53:09.122 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000103056 2024-09-20T22:53:09.123 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.123 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.123 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting_for_map not empty", 2024-09-20T22:53:09.123 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.282963+0000", 2024-09-20T22:53:09.123 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.1650000000000006e-06 2024-09-20T22:53:09.123 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.123 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.123 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:53:09.123 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.285785+0000", 2024-09-20T22:53:09.123 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0028221259999999999 2024-09-20T22:53:09.123 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.123 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.123 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for blocked object", 2024-09-20T22:53:09.123 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.285873+0000", 2024-09-20T22:53:09.124 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.7231999999999994e-05 2024-09-20T22:53:09.124 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.124 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.124 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:53:09.124 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.305344+0000", 2024-09-20T22:53:09.124 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.019471366 2024-09-20T22:53:09.124 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.124 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.124 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:53:09.124 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.305484+0000", 2024-09-20T22:53:09.124 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00013981900000000001 2024-09-20T22:53:09.124 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.124 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.124 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:53:09.125 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.305627+0000", 2024-09-20T22:53:09.125 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00014355499999999999 2024-09-20T22:53:09.125 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:53:09.125 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:53:09.125 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:53:09.125 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.125 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.125 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:6842 7.0 7:0f524219:::smithi08477203-50:head [copy-from ver 1338] snapc 136=[136,135,134,132,131,130,12f,12e,12d,12c,12b,12a,128,127,126,124,122,121,11f,11b,119,112,10e,10c,107,103,fb,fa,f8,ed,d7,c2,bf] ondisk+write+known_if_redirected+supports_pool_eio e689)", 2024-09-20T22:53:09.125 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:52:49.294403+0000", 2024-09-20T22:53:09.125 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 19.718125932, 2024-09-20T22:53:09.125 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.020667376000000001, 2024-09-20T22:53:09.125 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:53:09.125 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:53:09.125 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:53:09.126 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:53:09.126 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:53:09.126 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:53:09.126 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 6842 2024-09-20T22:53:09.126 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.126 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:53:09.126 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.126 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:53:09.126 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:49.294403+0000", 2024-09-20T22:53:09.126 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:53:09.126 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.126 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.126 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:53:09.127 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:49.294403+0000", 2024-09-20T22:53:09.127 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:53:09.127 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.127 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.127 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:53:09.127 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:49.294406+0000", 2024-09-20T22:53:09.127 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.7199999999999998e-06 2024-09-20T22:53:09.127 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.127 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.127 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:53:09.127 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:49.294416+0000", 2024-09-20T22:53:09.127 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.7219999999999994e-06 2024-09-20T22:53:09.127 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.127 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.128 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:53:09.128 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:49.294437+0000", 2024-09-20T22:53:09.128 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.1514000000000001e-05 2024-09-20T22:53:09.128 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.128 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.128 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:53:09.128 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:49.294447+0000", 2024-09-20T22:53:09.128 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.6860000000000001e-06 2024-09-20T22:53:09.128 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.128 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.128 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:53:09.129 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:49.294546+0000", 2024-09-20T22:53:09.129 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.9258000000000004e-05 2024-09-20T22:53:09.129 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.129 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.129 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op must wait for map", 2024-09-20T22:53:09.129 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:49.294555+0000", 2024-09-20T22:53:09.129 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.7269999999999996e-06 2024-09-20T22:53:09.129 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.129 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.129 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:53:09.129 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:49.297887+0000", 2024-09-20T22:53:09.129 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0033321370000000002 2024-09-20T22:53:09.129 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.129 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.129 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:53:09.130 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:49.298017+0000", 2024-09-20T22:53:09.130 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00013003099999999999 2024-09-20T22:53:09.130 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.130 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.130 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 2,3", 2024-09-20T22:53:09.130 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:49.301096+0000", 2024-09-20T22:53:09.130 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0030791289999999999 2024-09-20T22:53:09.130 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.130 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.130 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:53:09.130 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:49.305218+0000", 2024-09-20T22:53:09.130 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0041216530000000003 2024-09-20T22:53:09.130 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.130 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.131 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:53:09.131 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:49.308776+0000", 2024-09-20T22:53:09.131 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0035582809999999999 2024-09-20T22:53:09.131 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.131 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.131 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:53:09.131 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:49.314968+0000", 2024-09-20T22:53:09.131 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.006191378 2024-09-20T22:53:09.131 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.131 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.131 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:53:09.131 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:49.315022+0000", 2024-09-20T22:53:09.131 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.4185999999999999e-05 2024-09-20T22:53:09.131 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.132 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.132 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:53:09.132 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:49.315071+0000", 2024-09-20T22:53:09.132 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.8953999999999999e-05 2024-09-20T22:53:09.132 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:53:09.132 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:53:09.132 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:53:09.132 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.132 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.132 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:6966 7.0 7:064f66ab:::smithi08477203-45:head [delete] snapc 13a=[13a,139,138,137,136,135,132,131,130,12f,12e,12d,12c,12b,12a,128,127,124,122,121,11f,11b,119,112,10c,103,fa,ed,d7,c2] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e701)", 2024-09-20T22:53:09.132 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:53:01.292657+0000", 2024-09-20T22:53:09.133 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 7.7198726410000003, 2024-09-20T22:53:09.133 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.010221263, 2024-09-20T22:53:09.133 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:53:09.133 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:53:09.133 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:53:09.133 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:53:09.133 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:53:09.133 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:53:09.133 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 6966 2024-09-20T22:53:09.133 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.133 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:53:09.133 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.133 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:53:09.133 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:01.292657+0000", 2024-09-20T22:53:09.133 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:53:09.133 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.134 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.134 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:53:09.134 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:01.292657+0000", 2024-09-20T22:53:09.134 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:53:09.134 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.134 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.134 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:53:09.134 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:01.292659+0000", 2024-09-20T22:53:09.134 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.3709999999999998e-06 2024-09-20T22:53:09.134 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.134 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.134 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:53:09.134 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:01.292667+0000", 2024-09-20T22:53:09.134 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.3680000000000002e-06 2024-09-20T22:53:09.134 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.135 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.135 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:53:09.135 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:01.292692+0000", 2024-09-20T22:53:09.135 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.5026999999999999e-05 2024-09-20T22:53:09.135 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.135 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.135 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:53:09.135 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:01.292703+0000", 2024-09-20T22:53:09.135 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.026e-05 2024-09-20T22:53:09.135 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.135 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.135 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:53:09.135 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:01.292791+0000", 2024-09-20T22:53:09.135 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.8250000000000004e-05 2024-09-20T22:53:09.135 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.136 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.136 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op must wait for map", 2024-09-20T22:53:09.136 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:01.292802+0000", 2024-09-20T22:53:09.136 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0983e-05 2024-09-20T22:53:09.136 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.136 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.136 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:53:09.136 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:01.295037+0000", 2024-09-20T22:53:09.136 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0022354010000000001 2024-09-20T22:53:09.136 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.136 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.136 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:53:09.136 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:01.295170+0000", 2024-09-20T22:53:09.136 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00013318499999999999 2024-09-20T22:53:09.137 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.137 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.137 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 2,3", 2024-09-20T22:53:09.137 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:01.295448+0000", 2024-09-20T22:53:09.137 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00027764800000000002 2024-09-20T22:53:09.137 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.137 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.137 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:53:09.137 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:01.296607+0000", 2024-09-20T22:53:09.137 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001159174 2024-09-20T22:53:09.137 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.137 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.137 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:53:09.137 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:01.298060+0000", 2024-09-20T22:53:09.137 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0014531430000000001 2024-09-20T22:53:09.137 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.137 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.138 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:53:09.138 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:01.302717+0000", 2024-09-20T22:53:09.138 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0046563739999999996 2024-09-20T22:53:09.138 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.138 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.138 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:53:09.138 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:01.302802+0000", 2024-09-20T22:53:09.138 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.4948000000000001e-05 2024-09-20T22:53:09.138 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:09.138 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:09.138 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:53:09.138 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:01.302878+0000", 2024-09-20T22:53:09.138 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.6130999999999993e-05 2024-09-20T22:53:09.138 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:53:09.139 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:53:09.139 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:53:09.139 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:53:09.139 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:53:09.139 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-20T22:53:09.139 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:53:09.160 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:09.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:09.161 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:53:09.261 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:53:09.282 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:09.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:09.283 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:53:09.302 INFO:tasks.rados.rados.0.smithi084.stdout:2816: done (0 left) 2024-09-20T22:53:09.302 INFO:tasks.rados.rados.0.smithi084.stdout:2817: snap_create 2024-09-20T22:53:09.384 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:53:09.405 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:09.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:09.407 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:53:09.482 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:09 smithi084 ceph-mon[29774]: pgmap v963: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 650 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail 2024-09-20T22:53:09.482 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:09 smithi084 ceph-mon[29774]: 3.17 scrub starts 2024-09-20T22:53:09.483 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:09 smithi084 ceph-mon[29774]: 3.17 scrub ok 2024-09-20T22:53:09.483 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:09 smithi084 ceph-mon[29774]: 3.16 scrub starts 2024-09-20T22:53:09.483 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:09 smithi084 ceph-mon[29774]: 3.16 scrub ok 2024-09-20T22:53:09.483 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:09 smithi084 ceph-mon[29774]: 3.a scrub starts 2024-09-20T22:53:09.483 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:09 smithi084 ceph-mon[29774]: 3.a scrub ok 2024-09-20T22:53:09.483 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:09 smithi084 ceph-mon[29774]: osdmap e709: 8 total, 8 up, 7 in 2024-09-20T22:53:09.483 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:09 smithi084 ceph-mon[40024]: pgmap v963: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 650 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail 2024-09-20T22:53:09.483 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:09 smithi084 ceph-mon[40024]: 3.17 scrub starts 2024-09-20T22:53:09.483 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:09 smithi084 ceph-mon[40024]: 3.17 scrub ok 2024-09-20T22:53:09.483 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:09 smithi084 ceph-mon[40024]: 3.16 scrub starts 2024-09-20T22:53:09.483 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:09 smithi084 ceph-mon[40024]: 3.16 scrub ok 2024-09-20T22:53:09.484 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:09 smithi084 ceph-mon[40024]: 3.a scrub starts 2024-09-20T22:53:09.484 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:09 smithi084 ceph-mon[40024]: 3.a scrub ok 2024-09-20T22:53:09.484 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:09 smithi084 ceph-mon[40024]: osdmap e709: 8 total, 8 up, 7 in 2024-09-20T22:53:09.508 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:53:09.529 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:09.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:09.530 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:53:09.631 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:53:09.647 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-20T22:53:09.650 INFO:teuthology.orchestra.run.smithi084.stderr:dumped all 2024-09-20T22:53:09.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:09.653 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:09.653 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:53:09.688 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:09 smithi154 ceph-mon[33264]: pgmap v963: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 650 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail 2024-09-20T22:53:09.688 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:09 smithi154 ceph-mon[33264]: 3.17 scrub starts 2024-09-20T22:53:09.688 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:09 smithi154 ceph-mon[33264]: 3.17 scrub ok 2024-09-20T22:53:09.689 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:09 smithi154 ceph-mon[33264]: 3.16 scrub starts 2024-09-20T22:53:09.689 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:09 smithi154 ceph-mon[33264]: 3.16 scrub ok 2024-09-20T22:53:09.689 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:09 smithi154 ceph-mon[33264]: 3.a scrub starts 2024-09-20T22:53:09.689 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:09 smithi154 ceph-mon[33264]: 3.a scrub ok 2024-09-20T22:53:09.689 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:09 smithi154 ceph-mon[33264]: osdmap e709: 8 total, 8 up, 7 in 2024-09-20T22:53:09.689 INFO:journalctl@ceph.iscsi.iscsi.a.smithi154.stdout:Sep 20 22:53:09 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-iscsi-iscsi-a[59730]: debug there is no tcmu-runner data available 2024-09-20T22:53:09.754 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:53:09.776 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:09.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:09.777 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:53:09.785 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd set noscrub 2024-09-20T22:53:09.878 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:53:09.900 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:09.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:09.901 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:53:09.937 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-20T22:53:10.002 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:53:10.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:10.025 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:10.025 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:53:10.126 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:53:10.147 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:10.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:10.148 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:53:10.181 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:53:10.249 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:53:10.270 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:10.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:10.271 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:53:10.304 INFO:tasks.rados.rados.0.smithi084.stdout:2817: done (0 left) 2024-09-20T22:53:10.304 INFO:tasks.rados.rados.0.smithi084.stdout:2818: read oid 28 snap -1 2024-09-20T22:53:10.304 INFO:tasks.rados.rados.0.smithi084.stdout:2818: expect deleted 2024-09-20T22:53:10.304 INFO:tasks.rados.rados.0.smithi084.stdout:2819: write oid 21 current snap is 319 2024-09-20T22:53:10.304 INFO:tasks.rados.rados.0.smithi084.stdout:2819: seq_num 984 ranges {538088=654553,1714752=634675} 2024-09-20T22:53:10.310 INFO:tasks.rados.rados.0.smithi084.stdout:2819: writing smithi08477203-21 from 538088 to 1192641 tid 1 2024-09-20T22:53:10.317 INFO:tasks.rados.rados.0.smithi084.stdout:2819: writing smithi08477203-21 from 1714752 to 2349427 tid 2 2024-09-20T22:53:10.317 INFO:tasks.rados.rados.0.smithi084.stdout:2820: snap_remove snap 213 2024-09-20T22:53:10.372 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:53:10.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:10.394 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:10.394 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:53:10.494 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:53:10.506 INFO:teuthology.orchestra.run.smithi084.stdout:{"pg_ready":true,"pg_map":{"version":965,"stamp":"2024-09-20T22:53:09.296090+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":697136492,"num_objects":513,"num_object_clones":265,"num_object_copies":1539,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":513,"num_whiteouts":13,"num_read":6686,"num_read_kb":5066461,"num_write":4280,"num_write_kb":1314150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1142,"num_bytes_recovered":520180548,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":11199,"ondisk_log_size":11199,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":474,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":1822920,"kb_used_data":1487980,"kb_used_omap":10,"kb_used_meta":334773,"kb_avail":654393144,"statfs":{"total":671965249536,"available":670098579456,"internally_reserved":0,"allocated":1523691520,"data_stored":2228661935,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11157,"internal_metadata":342807659},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":15790086,"num_objects":13,"num_object_clones":13,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":3,"num_read":64,"num_read_kb":59035,"num_write":52,"num_write_kb":11748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.949957"},"pg_stats":[{"pgid":"3.1f","version":"86'2","reported_seq":671,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.289682+0000","last_change":"2024-09-20T22:52:21.630419+0000","last_active":"2024-09-20T22:53:08.289682+0000","last_peered":"2024-09-20T22:53:08.289682+0000","last_clean":"2024-09-20T22:53:08.289682+0000","last_became_active":"2024-09-20T22:47:49.222114+0000","last_became_peered":"2024-09-20T22:47:49.222114+0000","last_unstale":"2024-09-20T22:53:08.289682+0000","last_undegraded":"2024-09-20T22:53:08.289682+0000","last_fullsized":"2024-09-20T22:53:08.289682+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'2","last_scrub_stamp":"2024-09-20T22:52:21.630304+0000","last_deep_scrub":"86'2","last_deep_scrub_stamp":"2024-09-20T22:42:39.664421+0000","last_clean_scrub_stamp":"2024-09-20T22:52:21.630304+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:38.958732+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":436,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"625'15","reported_seq":669,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.291859+0000","last_change":"2024-09-20T22:51:51.813358+0000","last_active":"2024-09-20T22:53:08.291859+0000","last_peered":"2024-09-20T22:53:08.291859+0000","last_clean":"2024-09-20T22:53:08.291859+0000","last_became_active":"2024-09-20T22:51:20.286651+0000","last_became_peered":"2024-09-20T22:51:20.286651+0000","last_unstale":"2024-09-20T22:53:08.291859+0000","last_undegraded":"2024-09-20T22:53:08.291859+0000","last_fullsized":"2024-09-20T22:53:08.291859+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'15","last_scrub_stamp":"2024-09-20T22:51:51.813269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:51:51.813269+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:09.354334+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":236,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.285555+0000","last_change":"2024-09-20T22:52:41.184388+0000","last_active":"2024-09-20T22:53:08.285555+0000","last_peered":"2024-09-20T22:53:08.285555+0000","last_clean":"2024-09-20T22:53:08.285555+0000","last_became_active":"2024-09-20T22:51:20.291272+0000","last_became_peered":"2024-09-20T22:51:20.291272+0000","last_unstale":"2024-09-20T22:53:08.285555+0000","last_undegraded":"2024-09-20T22:53:08.285555+0000","last_fullsized":"2024-09-20T22:53:08.285555+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:41.184318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:52:41.184318+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:04.632649+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":640,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.291519+0000","last_change":"2024-09-20T22:51:36.730087+0000","last_active":"2024-09-20T22:53:08.291519+0000","last_peered":"2024-09-20T22:53:08.291519+0000","last_clean":"2024-09-20T22:53:08.291519+0000","last_became_active":"2024-09-20T22:51:20.295608+0000","last_became_peered":"2024-09-20T22:51:20.295608+0000","last_unstale":"2024-09-20T22:53:08.291519+0000","last_undegraded":"2024-09-20T22:53:08.291519+0000","last_fullsized":"2024-09-20T22:53:08.291519+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:36.730015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:51:36.730015+0000","last_clean_scrub_stamp":"2024-09-20T22:51:36.730015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":1314,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.297938+0000","last_change":"2024-09-20T22:51:01.404127+0000","last_active":"2024-09-20T22:53:08.297938+0000","last_peered":"2024-09-20T22:53:08.297938+0000","last_clean":"2024-09-20T22:53:08.297938+0000","last_became_active":"2024-09-20T22:41:44.832099+0000","last_became_peered":"2024-09-20T22:41:44.832099+0000","last_unstale":"2024-09-20T22:53:08.297938+0000","last_undegraded":"2024-09-20T22:53:08.297938+0000","last_fullsized":"2024-09-20T22:53:08.297938+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:01.404063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:51:01.404063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":1319,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.296487+0000","last_change":"2024-09-20T22:51:12.418041+0000","last_active":"2024-09-20T22:53:08.296487+0000","last_peered":"2024-09-20T22:53:08.296487+0000","last_clean":"2024-09-20T22:53:08.296487+0000","last_became_active":"2024-09-20T22:41:38.157298+0000","last_became_peered":"2024-09-20T22:41:38.157298+0000","last_unstale":"2024-09-20T22:53:08.296487+0000","last_undegraded":"2024-09-20T22:53:08.296487+0000","last_fullsized":"2024-09-20T22:53:08.296487+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:12.417920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:51:12.417920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"625'27","reported_seq":1401,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.296684+0000","last_change":"2024-09-20T22:51:08.441927+0000","last_active":"2024-09-20T22:53:08.296684+0000","last_peered":"2024-09-20T22:53:08.296684+0000","last_clean":"2024-09-20T22:53:08.296684+0000","last_became_active":"2024-09-20T22:47:49.218798+0000","last_became_peered":"2024-09-20T22:47:49.218798+0000","last_unstale":"2024-09-20T22:53:08.296684+0000","last_undegraded":"2024-09-20T22:53:08.296684+0000","last_fullsized":"2024-09-20T22:53:08.296684+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-09-20T22:51:08.441788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:51:08.441788+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":640,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.291746+0000","last_change":"2024-09-20T22:52:14.594895+0000","last_active":"2024-09-20T22:53:08.291746+0000","last_peered":"2024-09-20T22:53:08.291746+0000","last_clean":"2024-09-20T22:53:08.291746+0000","last_became_active":"2024-09-20T22:51:20.287060+0000","last_became_peered":"2024-09-20T22:51:20.287060+0000","last_unstale":"2024-09-20T22:53:08.291746+0000","last_undegraded":"2024-09-20T22:53:08.291746+0000","last_fullsized":"2024-09-20T22:53:08.291746+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:14.594823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:52:14.594823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:30.113157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":1339,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.286369+0000","last_change":"2024-09-20T22:51:09.684125+0000","last_active":"2024-09-20T22:53:08.286369+0000","last_peered":"2024-09-20T22:53:08.286369+0000","last_clean":"2024-09-20T22:53:08.286369+0000","last_became_active":"2024-09-20T22:48:05.242059+0000","last_became_peered":"2024-09-20T22:48:05.242059+0000","last_unstale":"2024-09-20T22:53:08.286369+0000","last_undegraded":"2024-09-20T22:53:08.286369+0000","last_fullsized":"2024-09-20T22:53:08.286369+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:09.684053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:46:00.999939+0000","last_clean_scrub_stamp":"2024-09-20T22:51:09.684053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"625'15","reported_seq":653,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.291891+0000","last_change":"2024-09-20T22:51:30.710329+0000","last_active":"2024-09-20T22:53:08.291891+0000","last_peered":"2024-09-20T22:53:08.291891+0000","last_clean":"2024-09-20T22:53:08.291891+0000","last_became_active":"2024-09-20T22:48:05.233429+0000","last_became_peered":"2024-09-20T22:48:05.233429+0000","last_unstale":"2024-09-20T22:53:08.291891+0000","last_undegraded":"2024-09-20T22:53:08.291891+0000","last_fullsized":"2024-09-20T22:53:08.291891+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-09-20T22:51:30.710245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:51:30.710245+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":1331,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.286118+0000","last_change":"2024-09-20T22:51:33.977328+0000","last_active":"2024-09-20T22:53:08.286118+0000","last_peered":"2024-09-20T22:53:08.286118+0000","last_clean":"2024-09-20T22:53:08.286118+0000","last_became_active":"2024-09-20T22:47:49.227379+0000","last_became_peered":"2024-09-20T22:47:49.227379+0000","last_unstale":"2024-09-20T22:53:08.286118+0000","last_undegraded":"2024-09-20T22:53:08.286118+0000","last_fullsized":"2024-09-20T22:53:08.286118+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:33.977257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:51:33.977257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":673,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.286640+0000","last_change":"2024-09-20T22:51:35.899942+0000","last_active":"2024-09-20T22:53:08.286640+0000","last_peered":"2024-09-20T22:53:08.286640+0000","last_clean":"2024-09-20T22:53:08.286640+0000","last_became_active":"2024-09-20T22:51:20.277465+0000","last_became_peered":"2024-09-20T22:51:20.277465+0000","last_unstale":"2024-09-20T22:53:08.286640+0000","last_undegraded":"2024-09-20T22:53:08.286640+0000","last_fullsized":"2024-09-20T22:53:08.286640+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:35.899860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:49:08.664407+0000","last_clean_scrub_stamp":"2024-09-20T22:51:35.899860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":1345,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.286280+0000","last_change":"2024-09-20T22:51:20.298096+0000","last_active":"2024-09-20T22:53:08.286280+0000","last_peered":"2024-09-20T22:53:08.286280+0000","last_clean":"2024-09-20T22:53:08.286280+0000","last_became_active":"2024-09-20T22:51:20.297787+0000","last_became_peered":"2024-09-20T22:51:20.297787+0000","last_unstale":"2024-09-20T22:53:08.286280+0000","last_undegraded":"2024-09-20T22:53:08.286280+0000","last_fullsized":"2024-09-20T22:53:08.286280+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:50:17.735354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:50:17.735354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"625'9","reported_seq":655,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.290258+0000","last_change":"2024-09-20T22:51:20.301034+0000","last_active":"2024-09-20T22:53:08.290258+0000","last_peered":"2024-09-20T22:53:08.290258+0000","last_clean":"2024-09-20T22:53:08.290258+0000","last_became_active":"2024-09-20T22:51:20.285795+0000","last_became_peered":"2024-09-20T22:51:20.285795+0000","last_unstale":"2024-09-20T22:53:08.290258+0000","last_undegraded":"2024-09-20T22:53:08.290258+0000","last_fullsized":"2024-09-20T22:53:08.290258+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"181'7","last_scrub_stamp":"2024-09-20T22:51:05.899205+0000","last_deep_scrub":"181'7","last_deep_scrub_stamp":"2024-09-20T22:48:41.049381+0000","last_clean_scrub_stamp":"2024-09-20T22:51:05.899205+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":9,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":1358,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.290125+0000","last_change":"2024-09-20T22:52:16.705733+0000","last_active":"2024-09-20T22:53:08.290125+0000","last_peered":"2024-09-20T22:53:08.290125+0000","last_clean":"2024-09-20T22:53:08.290125+0000","last_became_active":"2024-09-20T22:51:20.285368+0000","last_became_peered":"2024-09-20T22:51:20.285368+0000","last_unstale":"2024-09-20T22:53:08.290125+0000","last_undegraded":"2024-09-20T22:53:08.290125+0000","last_fullsized":"2024-09-20T22:53:08.290125+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:16.705645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:47:40.361958+0000","last_clean_scrub_stamp":"2024-09-20T22:52:16.705645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:40.490856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":1324,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.285573+0000","last_change":"2024-09-20T22:51:20.279625+0000","last_active":"2024-09-20T22:53:08.285573+0000","last_peered":"2024-09-20T22:53:08.285573+0000","last_clean":"2024-09-20T22:53:08.285573+0000","last_became_active":"2024-09-20T22:51:20.279307+0000","last_became_peered":"2024-09-20T22:51:20.279307+0000","last_unstale":"2024-09-20T22:53:08.285573+0000","last_undegraded":"2024-09-20T22:53:08.285573+0000","last_fullsized":"2024-09-20T22:53:08.285573+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:04.636900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:51:04.636900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":234,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.292547+0000","last_change":"2024-09-20T22:52:39.724071+0000","last_active":"2024-09-20T22:53:08.292547+0000","last_peered":"2024-09-20T22:53:08.292547+0000","last_clean":"2024-09-20T22:53:08.292547+0000","last_became_active":"2024-09-20T22:51:20.282393+0000","last_became_peered":"2024-09-20T22:51:20.282393+0000","last_unstale":"2024-09-20T22:53:08.292547+0000","last_undegraded":"2024-09-20T22:53:08.292547+0000","last_fullsized":"2024-09-20T22:53:08.292547+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:39.723977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:52:39.723977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:47.638711+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":673,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.288228+0000","last_change":"2024-09-20T22:52:42.745105+0000","last_active":"2024-09-20T22:53:08.288228+0000","last_peered":"2024-09-20T22:53:08.288228+0000","last_clean":"2024-09-20T22:53:08.288228+0000","last_became_active":"2024-09-20T22:48:05.233687+0000","last_became_peered":"2024-09-20T22:48:05.233687+0000","last_unstale":"2024-09-20T22:53:08.288228+0000","last_undegraded":"2024-09-20T22:53:08.288228+0000","last_fullsized":"2024-09-20T22:53:08.288228+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:42.745032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:49:53.933601+0000","last_clean_scrub_stamp":"2024-09-20T22:52:42.745032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:59.532706+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"625'34","reported_seq":1410,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.287747+0000","last_change":"2024-09-20T22:51:20.289052+0000","last_active":"2024-09-20T22:53:08.287747+0000","last_peered":"2024-09-20T22:53:08.287747+0000","last_clean":"2024-09-20T22:53:08.287747+0000","last_became_active":"2024-09-20T22:51:20.276338+0000","last_became_peered":"2024-09-20T22:51:20.276338+0000","last_unstale":"2024-09-20T22:53:08.287747+0000","last_undegraded":"2024-09-20T22:53:08.287747+0000","last_fullsized":"2024-09-20T22:53:08.287747+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'22","last_scrub_stamp":"2024-09-20T22:51:08.280738+0000","last_deep_scrub":"113'22","last_deep_scrub_stamp":"2024-09-20T22:43:12.191878+0000","last_clean_scrub_stamp":"2024-09-20T22:51:08.280738+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":946,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.292198+0000","last_change":"2024-09-20T22:52:43.758236+0000","last_active":"2024-09-20T22:53:08.292198+0000","last_peered":"2024-09-20T22:53:08.292198+0000","last_clean":"2024-09-20T22:53:08.292198+0000","last_became_active":"2024-09-20T22:51:20.295564+0000","last_became_peered":"2024-09-20T22:51:20.295564+0000","last_unstale":"2024-09-20T22:53:08.292198+0000","last_undegraded":"2024-09-20T22:53:08.292198+0000","last_fullsized":"2024-09-20T22:53:08.292198+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:43.758162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:50:40.784154+0000","last_clean_scrub_stamp":"2024-09-20T22:52:43.758162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:10.526451+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":1307,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.296301+0000","last_change":"2024-09-20T22:51:13.385618+0000","last_active":"2024-09-20T22:53:08.296301+0000","last_peered":"2024-09-20T22:53:08.296301+0000","last_clean":"2024-09-20T22:53:08.296301+0000","last_became_active":"2024-09-20T22:41:44.832131+0000","last_became_peered":"2024-09-20T22:41:44.832131+0000","last_unstale":"2024-09-20T22:53:08.296301+0000","last_undegraded":"2024-09-20T22:53:08.296301+0000","last_fullsized":"2024-09-20T22:53:08.296301+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:13.385520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:45:25.002199+0000","last_clean_scrub_stamp":"2024-09-20T22:51:13.385520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":648,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.292061+0000","last_change":"2024-09-20T22:53:04.583753+0000","last_active":"2024-09-20T22:53:08.292061+0000","last_peered":"2024-09-20T22:53:08.292061+0000","last_clean":"2024-09-20T22:53:08.292061+0000","last_became_active":"2024-09-20T22:51:20.284621+0000","last_became_peered":"2024-09-20T22:51:20.284621+0000","last_unstale":"2024-09-20T22:53:08.292061+0000","last_undegraded":"2024-09-20T22:53:08.292061+0000","last_fullsized":"2024-09-20T22:53:08.292061+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:04.583683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:50:41.885844+0000","last_clean_scrub_stamp":"2024-09-20T22:53:04.583683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:27.888402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"625'18","reported_seq":1418,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.295670+0000","last_change":"2024-09-20T22:51:20.300828+0000","last_active":"2024-09-20T22:53:08.295670+0000","last_peered":"2024-09-20T22:53:08.295670+0000","last_clean":"2024-09-20T22:53:08.295670+0000","last_became_active":"2024-09-20T22:51:20.278886+0000","last_became_peered":"2024-09-20T22:51:20.278886+0000","last_unstale":"2024-09-20T22:53:08.295670+0000","last_undegraded":"2024-09-20T22:53:08.295670+0000","last_fullsized":"2024-09-20T22:53:08.295670+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-09-20T22:51:03.380934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:51:03.380934+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":642,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.292553+0000","last_change":"2024-09-20T22:52:45.738376+0000","last_active":"2024-09-20T22:53:08.292553+0000","last_peered":"2024-09-20T22:53:08.292553+0000","last_clean":"2024-09-20T22:53:08.292553+0000","last_became_active":"2024-09-20T22:48:05.250319+0000","last_became_peered":"2024-09-20T22:48:05.250319+0000","last_unstale":"2024-09-20T22:53:08.292553+0000","last_undegraded":"2024-09-20T22:53:08.292553+0000","last_fullsized":"2024-09-20T22:53:08.292553+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:45.738275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:52:45.738275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:02.329689+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"154'2","reported_seq":1323,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.290596+0000","last_change":"2024-09-20T22:52:45.578519+0000","last_active":"2024-09-20T22:53:08.290596+0000","last_peered":"2024-09-20T22:53:08.290596+0000","last_clean":"2024-09-20T22:53:08.290596+0000","last_became_active":"2024-09-20T22:41:44.199822+0000","last_became_peered":"2024-09-20T22:41:44.199822+0000","last_unstale":"2024-09-20T22:53:08.290596+0000","last_undegraded":"2024-09-20T22:53:08.290596+0000","last_fullsized":"2024-09-20T22:53:08.290596+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"154'2","last_scrub_stamp":"2024-09-20T22:52:45.578416+0000","last_deep_scrub":"154'2","last_deep_scrub_stamp":"2024-09-20T22:43:54.134066+0000","last_clean_scrub_stamp":"2024-09-20T22:52:45.578416+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:53.480152+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"67'1","reported_seq":247,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.612447+0000","last_change":"2024-09-20T22:53:08.612473+0000","last_active":"2024-09-20T22:53:08.612447+0000","last_peered":"2024-09-20T22:53:08.612447+0000","last_clean":"2024-09-20T22:53:08.612447+0000","last_became_active":"2024-09-20T22:51:20.279978+0000","last_became_peered":"2024-09-20T22:51:20.279978+0000","last_unstale":"2024-09-20T22:53:08.612447+0000","last_undegraded":"2024-09-20T22:53:08.612447+0000","last_fullsized":"2024-09-20T22:53:08.612447+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-09-20T22:53:08.612395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:53:08.612395+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:13.547026+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"625'25","reported_seq":694,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.289276+0000","last_change":"2024-09-20T22:51:02.707652+0000","last_active":"2024-09-20T22:53:08.289276+0000","last_peered":"2024-09-20T22:53:08.289276+0000","last_clean":"2024-09-20T22:53:08.289276+0000","last_became_active":"2024-09-20T22:47:49.232181+0000","last_became_peered":"2024-09-20T22:47:49.232181+0000","last_unstale":"2024-09-20T22:53:08.289276+0000","last_undegraded":"2024-09-20T22:53:08.289276+0000","last_fullsized":"2024-09-20T22:53:08.289276+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"151'15","last_scrub_stamp":"2024-09-20T22:51:02.707574+0000","last_deep_scrub":"151'15","last_deep_scrub_stamp":"2024-09-20T22:43:50.715761+0000","last_clean_scrub_stamp":"2024-09-20T22:51:02.707574+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"661'11","reported_seq":1509,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.293101+0000","last_change":"2024-09-20T22:52:21.908015+0000","last_active":"2024-09-20T22:53:08.293101+0000","last_peered":"2024-09-20T22:53:08.293101+0000","last_clean":"2024-09-20T22:53:08.293101+0000","last_became_active":"2024-09-20T22:48:05.244270+0000","last_became_peered":"2024-09-20T22:48:05.244270+0000","last_unstale":"2024-09-20T22:53:08.293101+0000","last_undegraded":"2024-09-20T22:53:08.293101+0000","last_fullsized":"2024-09-20T22:53:08.293101+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"661'11","last_scrub_stamp":"2024-09-20T22:52:21.907943+0000","last_deep_scrub":"661'11","last_deep_scrub_stamp":"2024-09-20T22:52:21.907943+0000","last_clean_scrub_stamp":"2024-09-20T22:52:21.907943+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:27.456801+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.c","version":"708'2198","reported_seq":5171,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.307978+0000","last_change":"2024-09-20T22:53:08.308187+0000","last_active":"2024-09-20T22:53:08.307978+0000","last_peered":"2024-09-20T22:53:08.307978+0000","last_clean":"2024-09-20T22:53:08.307978+0000","last_became_active":"2024-09-20T22:42:52.404027+0000","last_became_peered":"2024-09-20T22:42:52.404027+0000","last_unstale":"2024-09-20T22:53:08.307978+0000","last_undegraded":"2024-09-20T22:53:08.307978+0000","last_fullsized":"2024-09-20T22:53:08.307978+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"674'2030","last_scrub_stamp":"2024-09-20T22:52:36.892383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:52:36.892383+0000","objects_scrubbed":44,"log_size":2198,"log_dups_size":0,"ondisk_log_size":2198,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:01.737318+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.015309313,"stat_sum":{"num_bytes":98091786,"num_objects":44,"num_object_clones":37,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":3,"num_read":895,"num_read_kb":966419,"num_write":582,"num_write_kb":222502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"bf","length":"1"},{"start":"f8","length":"1"},{"start":"fb","length":"1"},{"start":"fd","length":"6"},{"start":"104","length":"8"},{"start":"10d","length":"5"},{"start":"113","length":"8"},{"start":"11c","length":"3"},{"start":"120","length":"1"},{"start":"123","length":"1"},{"start":"125","length":"2"},{"start":"129","length":"1"},{"start":"12b","length":"1"},{"start":"133","length":"2"}]},{"pgid":"1.a","version":"77'128","reported_seq":1587,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.285440+0000","last_change":"2024-09-20T22:50:19.676094+0000","last_active":"2024-09-20T22:53:08.285440+0000","last_peered":"2024-09-20T22:53:08.285440+0000","last_clean":"2024-09-20T22:53:08.285440+0000","last_became_active":"2024-09-20T22:42:52.407791+0000","last_became_peered":"2024-09-20T22:42:52.407791+0000","last_unstale":"2024-09-20T22:53:08.285440+0000","last_undegraded":"2024-09-20T22:53:08.285440+0000","last_fullsized":"2024-09-20T22:53:08.285440+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:50:19.675992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:50:19.675992+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"625'27","reported_seq":313,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.296462+0000","last_change":"2024-09-20T22:52:15.358218+0000","last_active":"2024-09-20T22:53:08.296462+0000","last_peered":"2024-09-20T22:53:08.296462+0000","last_clean":"2024-09-20T22:53:08.296462+0000","last_became_active":"2024-09-20T22:51:20.287083+0000","last_became_peered":"2024-09-20T22:51:20.287083+0000","last_unstale":"2024-09-20T22:53:08.296462+0000","last_undegraded":"2024-09-20T22:53:08.296462+0000","last_fullsized":"2024-09-20T22:53:08.296462+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'27","last_scrub_stamp":"2024-09-20T22:52:15.358100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:52:15.358100+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:27.239349+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":1325,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.295228+0000","last_change":"2024-09-20T22:51:29.394206+0000","last_active":"2024-09-20T22:53:08.295228+0000","last_peered":"2024-09-20T22:53:08.295228+0000","last_clean":"2024-09-20T22:53:08.295228+0000","last_became_active":"2024-09-20T22:51:20.279439+0000","last_became_peered":"2024-09-20T22:51:20.279439+0000","last_unstale":"2024-09-20T22:53:08.295228+0000","last_undegraded":"2024-09-20T22:53:08.295228+0000","last_fullsized":"2024-09-20T22:53:08.295228+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:29.394097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:49:12.070675+0000","last_clean_scrub_stamp":"2024-09-20T22:51:29.394097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"408'12","reported_seq":699,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.292685+0000","last_change":"2024-09-20T22:51:50.785575+0000","last_active":"2024-09-20T22:53:08.292685+0000","last_peered":"2024-09-20T22:53:08.292685+0000","last_clean":"2024-09-20T22:53:08.292685+0000","last_became_active":"2024-09-20T22:48:05.242914+0000","last_became_peered":"2024-09-20T22:48:05.242914+0000","last_unstale":"2024-09-20T22:53:08.292685+0000","last_undegraded":"2024-09-20T22:53:08.292685+0000","last_fullsized":"2024-09-20T22:53:08.292685+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"408'12","last_scrub_stamp":"2024-09-20T22:51:50.785478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:51:50.785478+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":1334,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.286054+0000","last_change":"2024-09-20T22:51:31.960415+0000","last_active":"2024-09-20T22:53:08.286054+0000","last_peered":"2024-09-20T22:53:08.286054+0000","last_clean":"2024-09-20T22:53:08.286054+0000","last_became_active":"2024-09-20T22:51:20.280145+0000","last_became_peered":"2024-09-20T22:51:20.280145+0000","last_unstale":"2024-09-20T22:53:08.286054+0000","last_undegraded":"2024-09-20T22:53:08.286054+0000","last_fullsized":"2024-09-20T22:53:08.286054+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:31.960331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:51:31.960331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3","version":"706'307","reported_seq":3216,"reported_epoch":708,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-20T22:53:08.365089+0000","last_change":"2024-09-20T22:53:08.365150+0000","last_active":"2024-09-20T22:53:08.365089+0000","last_peered":"2024-09-20T22:53:08.365089+0000","last_clean":"2024-09-20T22:53:08.365089+0000","last_became_active":"2024-09-20T22:49:05.221402+0000","last_became_peered":"2024-09-20T22:49:05.221402+0000","last_unstale":"2024-09-20T22:53:08.365089+0000","last_undegraded":"2024-09-20T22:53:08.365089+0000","last_fullsized":"2024-09-20T22:53:08.365089+0000","mapping_epoch":404,"log_start":"456'129","ondisk_log_start":"456'129","created":94,"last_epoch_clean":465,"parent":"0.0","parent_split_bits":0,"last_scrub":"609'243","last_scrub_stamp":"2024-09-20T22:51:31.402013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:51:31.402013+0000","objects_scrubbed":6,"log_size":178,"log_dups_size":149,"ondisk_log_size":178,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.067059514000000001,"stat_sum":{"num_bytes":19863277,"num_objects":7,"num_object_clones":6,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":100,"num_read_kb":128300,"num_write":67,"num_write_kb":22506,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":20196357,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"bf","length":"1"},{"start":"f8","length":"1"},{"start":"fb","length":"1"},{"start":"fd","length":"6"},{"start":"104","length":"8"},{"start":"10d","length":"5"},{"start":"113","length":"6"},{"start":"11a","length":"1"},{"start":"11c","length":"3"},{"start":"120","length":"1"},{"start":"123","length":"1"},{"start":"125","length":"1"},{"start":"129","length":"1"},{"start":"133","length":"2"}]},{"pgid":"1.5","version":"77'128","reported_seq":1493,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.293102+0000","last_change":"2024-09-20T22:51:12.927535+0000","last_active":"2024-09-20T22:53:08.293102+0000","last_peered":"2024-09-20T22:53:08.293102+0000","last_clean":"2024-09-20T22:53:08.293102+0000","last_became_active":"2024-09-20T22:48:08.207018+0000","last_became_peered":"2024-09-20T22:48:08.207018+0000","last_unstale":"2024-09-20T22:53:08.293102+0000","last_undegraded":"2024-09-20T22:53:08.293102+0000","last_fullsized":"2024-09-20T22:53:08.293102+0000","mapping_epoch":407,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":408,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:51:12.927463+0000","last_deep_scrub":"77'128","last_deep_scrub_stamp":"2024-09-20T22:45:41.608075+0000","last_clean_scrub_stamp":"2024-09-20T22:51:12.927463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"625'30","reported_seq":1413,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.295459+0000","last_change":"2024-09-20T22:50:45.270518+0000","last_active":"2024-09-20T22:53:08.295459+0000","last_peered":"2024-09-20T22:53:08.295459+0000","last_clean":"2024-09-20T22:53:08.295459+0000","last_became_active":"2024-09-20T22:47:49.217605+0000","last_became_peered":"2024-09-20T22:47:49.217605+0000","last_unstale":"2024-09-20T22:53:08.295459+0000","last_undegraded":"2024-09-20T22:53:08.295459+0000","last_fullsized":"2024-09-20T22:53:08.295459+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'18","last_scrub_stamp":"2024-09-20T22:50:45.270399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:50:45.270399+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":1339,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.296758+0000","last_change":"2024-09-20T22:52:45.537243+0000","last_active":"2024-09-20T22:53:08.296758+0000","last_peered":"2024-09-20T22:53:08.296758+0000","last_clean":"2024-09-20T22:53:08.296758+0000","last_became_active":"2024-09-20T22:47:49.227069+0000","last_became_peered":"2024-09-20T22:47:49.227069+0000","last_unstale":"2024-09-20T22:53:08.296758+0000","last_undegraded":"2024-09-20T22:53:08.296758+0000","last_fullsized":"2024-09-20T22:53:08.296758+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:45.537173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:52:45.537173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:54.445275+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":634,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.294186+0000","last_change":"2024-09-20T22:51:09.901939+0000","last_active":"2024-09-20T22:53:08.294186+0000","last_peered":"2024-09-20T22:53:08.294186+0000","last_clean":"2024-09-20T22:53:08.294186+0000","last_became_active":"2024-09-20T22:48:05.230003+0000","last_became_peered":"2024-09-20T22:48:05.230003+0000","last_unstale":"2024-09-20T22:53:08.294186+0000","last_undegraded":"2024-09-20T22:53:08.294186+0000","last_fullsized":"2024-09-20T22:53:08.294186+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:09.901838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:07.947945+0000","last_clean_scrub_stamp":"2024-09-20T22:51:09.901838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":634,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.291849+0000","last_change":"2024-09-20T22:51:55.748527+0000","last_active":"2024-09-20T22:53:08.291849+0000","last_peered":"2024-09-20T22:53:08.291849+0000","last_clean":"2024-09-20T22:53:08.291849+0000","last_became_active":"2024-09-20T22:48:05.239917+0000","last_became_peered":"2024-09-20T22:48:05.239917+0000","last_unstale":"2024-09-20T22:53:08.291849+0000","last_undegraded":"2024-09-20T22:53:08.291849+0000","last_fullsized":"2024-09-20T22:53:08.291849+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:55.748425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:53.036601+0000","last_clean_scrub_stamp":"2024-09-20T22:51:55.748425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:19.609464+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2","version":"708'747","reported_seq":2986,"reported_epoch":708,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T22:53:08.367301+0000","last_change":"2024-09-20T22:53:08.365196+0000","last_active":"2024-09-20T22:53:08.367301+0000","last_peered":"2024-09-20T22:53:08.367301+0000","last_clean":"2024-09-20T22:53:08.367301+0000","last_became_active":"2024-09-20T22:52:23.286461+0000","last_became_peered":"2024-09-20T22:52:23.286461+0000","last_unstale":"2024-09-20T22:53:08.367301+0000","last_undegraded":"2024-09-20T22:53:08.367301+0000","last_fullsized":"2024-09-20T22:53:08.367301+0000","mapping_epoch":662,"log_start":"471'464","ondisk_log_start":"471'464","created":94,"last_epoch_clean":663,"parent":"0.0","parent_split_bits":0,"last_scrub":"282'120","last_scrub_stamp":"2024-09-20T22:46:22.099642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:46:22.099642+0000","objects_scrubbed":1,"log_size":283,"log_dups_size":687,"ondisk_log_size":283,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":39,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.034743356000000003,"stat_sum":{"num_bytes":67843488,"num_objects":30,"num_object_clones":26,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":2,"num_read":396,"num_read_kb":412551,"num_write":256,"num_write_kb":102445,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":144,"num_bytes_recovered":106353268,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"bf","length":"1"},{"start":"f8","length":"1"}]},{"pgid":"1.4","version":"77'128","reported_seq":1619,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.296226+0000","last_change":"2024-09-20T22:52:02.474080+0000","last_active":"2024-09-20T22:53:08.296226+0000","last_peered":"2024-09-20T22:53:08.296226+0000","last_clean":"2024-09-20T22:53:08.296226+0000","last_became_active":"2024-09-20T22:46:15.181305+0000","last_became_peered":"2024-09-20T22:46:15.181305+0000","last_unstale":"2024-09-20T22:53:08.296226+0000","last_undegraded":"2024-09-20T22:53:08.296226+0000","last_fullsized":"2024-09-20T22:53:08.296226+0000","mapping_epoch":94,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:52:02.473979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:52:02.473979+0000","objects_scrubbed":1,"log_size":0,"log_dups_size":59,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"625'28","reported_seq":671,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.294095+0000","last_change":"2024-09-20T22:51:58.712602+0000","last_active":"2024-09-20T22:53:08.294095+0000","last_peered":"2024-09-20T22:53:08.294095+0000","last_clean":"2024-09-20T22:53:08.294095+0000","last_became_active":"2024-09-20T22:48:05.235812+0000","last_became_peered":"2024-09-20T22:48:05.235812+0000","last_unstale":"2024-09-20T22:53:08.294095+0000","last_undegraded":"2024-09-20T22:53:08.294095+0000","last_fullsized":"2024-09-20T22:53:08.294095+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'28","last_scrub_stamp":"2024-09-20T22:51:58.712488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:51:58.712488+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:18.921188+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":41,"num_read_kb":34,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"67'1","reported_seq":691,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.289122+0000","last_change":"2024-09-20T22:51:20.292468+0000","last_active":"2024-09-20T22:53:08.289122+0000","last_peered":"2024-09-20T22:53:08.289122+0000","last_clean":"2024-09-20T22:53:08.289122+0000","last_became_active":"2024-09-20T22:51:20.283011+0000","last_became_peered":"2024-09-20T22:51:20.283011+0000","last_unstale":"2024-09-20T22:53:08.289122+0000","last_undegraded":"2024-09-20T22:53:08.289122+0000","last_fullsized":"2024-09-20T22:53:08.289122+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-09-20T22:51:11.806011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:51:11.806011+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"600'13","reported_seq":1509,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.295805+0000","last_change":"2024-09-20T22:51:32.448600+0000","last_active":"2024-09-20T22:53:08.295805+0000","last_peered":"2024-09-20T22:53:08.295805+0000","last_clean":"2024-09-20T22:53:08.295805+0000","last_became_active":"2024-09-20T22:51:20.289873+0000","last_became_peered":"2024-09-20T22:51:20.289873+0000","last_unstale":"2024-09-20T22:53:08.295805+0000","last_undegraded":"2024-09-20T22:53:08.295805+0000","last_fullsized":"2024-09-20T22:53:08.295805+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"600'13","last_scrub_stamp":"2024-09-20T22:51:32.448512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:51:32.448512+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":1322,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.289842+0000","last_change":"2024-09-20T22:53:02.610452+0000","last_active":"2024-09-20T22:53:08.289842+0000","last_peered":"2024-09-20T22:53:08.289842+0000","last_clean":"2024-09-20T22:53:08.289842+0000","last_became_active":"2024-09-20T22:41:44.831901+0000","last_became_peered":"2024-09-20T22:41:44.831901+0000","last_unstale":"2024-09-20T22:53:08.289842+0000","last_undegraded":"2024-09-20T22:53:08.289842+0000","last_fullsized":"2024-09-20T22:53:08.289842+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:02.610359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:53:02.610359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:32.529803+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1","version":"708'409","reported_seq":3405,"reported_epoch":708,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T22:53:08.296838+0000","last_change":"2024-09-20T22:53:08.296838+0000","last_active":"2024-09-20T22:53:08.296838+0000","last_peered":"2024-09-20T22:53:08.296838+0000","last_clean":"2024-09-20T22:53:08.296838+0000","last_became_active":"2024-09-20T22:49:21.232750+0000","last_became_peered":"2024-09-20T22:49:21.232750+0000","last_unstale":"2024-09-20T22:53:08.296838+0000","last_undegraded":"2024-09-20T22:53:08.296838+0000","last_fullsized":"2024-09-20T22:53:08.296838+0000","mapping_epoch":407,"log_start":"474'259","ondisk_log_start":"474'259","created":94,"last_epoch_clean":481,"parent":"0.0","parent_split_bits":0,"last_scrub":"703'407","last_scrub_stamp":"2024-09-20T22:53:06.625437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:53:06.625437+0000","objects_scrubbed":6,"log_size":150,"log_dups_size":338,"ondisk_log_size":150,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:19.539132+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.96718284099999996,"stat_sum":{"num_bytes":6959411,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":93,"num_read_kb":91249,"num_write":88,"num_write_kb":45551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":8732217,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"bf","length":"1"},{"start":"f8","length":"1"},{"start":"fb","length":"1"},{"start":"fd","length":"6"},{"start":"104","length":"8"},{"start":"10d","length":"5"},{"start":"113","length":"8"},{"start":"11c","length":"3"},{"start":"120","length":"1"},{"start":"123","length":"1"},{"start":"125","length":"2"},{"start":"129","length":"1"},{"start":"133","length":"2"}]},{"pgid":"1.7","version":"77'128","reported_seq":1632,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.292050+0000","last_change":"2024-09-20T22:52:34.898681+0000","last_active":"2024-09-20T22:53:08.292050+0000","last_peered":"2024-09-20T22:53:08.292050+0000","last_clean":"2024-09-20T22:53:08.292050+0000","last_became_active":"2024-09-20T22:46:03.169596+0000","last_became_peered":"2024-09-20T22:46:03.169596+0000","last_unstale":"2024-09-20T22:53:08.292050+0000","last_undegraded":"2024-09-20T22:53:08.292050+0000","last_fullsized":"2024-09-20T22:53:08.292050+0000","mapping_epoch":94,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:52:34.898595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:52:34.898595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:51.834169+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"625'20","reported_seq":297,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.285758+0000","last_change":"2024-09-20T22:52:19.125548+0000","last_active":"2024-09-20T22:53:08.285758+0000","last_peered":"2024-09-20T22:53:08.285758+0000","last_clean":"2024-09-20T22:53:08.285758+0000","last_became_active":"2024-09-20T22:51:20.297933+0000","last_became_peered":"2024-09-20T22:51:20.297933+0000","last_unstale":"2024-09-20T22:53:08.285758+0000","last_undegraded":"2024-09-20T22:53:08.285758+0000","last_fullsized":"2024-09-20T22:53:08.285758+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'20","last_scrub_stamp":"2024-09-20T22:52:19.125462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:52:19.125462+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:45.473839+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":1336,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.286406+0000","last_change":"2024-09-20T22:52:46.062448+0000","last_active":"2024-09-20T22:53:08.286406+0000","last_peered":"2024-09-20T22:53:08.286406+0000","last_clean":"2024-09-20T22:53:08.286406+0000","last_became_active":"2024-09-20T22:41:38.155629+0000","last_became_peered":"2024-09-20T22:41:38.155629+0000","last_unstale":"2024-09-20T22:53:08.286406+0000","last_undegraded":"2024-09-20T22:53:08.286406+0000","last_fullsized":"2024-09-20T22:53:08.286406+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:46.062352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:52:46.062352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:15.728943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"389'11","reported_seq":736,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.288183+0000","last_change":"2024-09-20T22:51:26.968244+0000","last_active":"2024-09-20T22:53:08.288183+0000","last_peered":"2024-09-20T22:53:08.288183+0000","last_clean":"2024-09-20T22:53:08.288183+0000","last_became_active":"2024-09-20T22:47:49.227264+0000","last_became_peered":"2024-09-20T22:47:49.227264+0000","last_unstale":"2024-09-20T22:53:08.288183+0000","last_undegraded":"2024-09-20T22:53:08.288183+0000","last_fullsized":"2024-09-20T22:53:08.288183+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'11","last_scrub_stamp":"2024-09-20T22:51:26.968138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:51:26.968138+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":667,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.286491+0000","last_change":"2024-09-20T22:51:55.755550+0000","last_active":"2024-09-20T22:53:08.286491+0000","last_peered":"2024-09-20T22:53:08.286491+0000","last_clean":"2024-09-20T22:53:08.286491+0000","last_became_active":"2024-09-20T22:47:49.232380+0000","last_became_peered":"2024-09-20T22:47:49.232380+0000","last_unstale":"2024-09-20T22:53:08.286491+0000","last_undegraded":"2024-09-20T22:53:08.286491+0000","last_fullsized":"2024-09-20T22:53:08.286491+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:55.755432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:51:55.755432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.0","version":"706'604","reported_seq":3695,"reported_epoch":708,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T22:53:08.285514+0000","last_change":"2024-09-20T22:53:07.362433+0000","last_active":"2024-09-20T22:53:08.285514+0000","last_peered":"2024-09-20T22:53:08.285514+0000","last_clean":"2024-09-20T22:53:08.285514+0000","last_became_active":"2024-09-20T22:51:53.366096+0000","last_became_peered":"2024-09-20T22:51:53.366096+0000","last_unstale":"2024-09-20T22:53:08.285514+0000","last_undegraded":"2024-09-20T22:53:08.285514+0000","last_fullsized":"2024-09-20T22:53:08.285514+0000","mapping_epoch":632,"log_start":"480'253","ondisk_log_start":"480'253","created":94,"last_epoch_clean":633,"parent":"0.0","parent_split_bits":0,"last_scrub":"562'368","last_scrub_stamp":"2024-09-20T22:50:44.669910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:50:44.669910+0000","objects_scrubbed":15,"log_size":351,"log_dups_size":416,"ondisk_log_size":351,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":5,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.99780654700000004,"stat_sum":{"num_bytes":60418945,"num_objects":29,"num_object_clones":26,"num_object_copies":87,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":0,"num_read":231,"num_read_kb":233511,"num_write":189,"num_write_kb":68166,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":42943203,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"bf","length":"1"},{"start":"f8","length":"1"},{"start":"fb","length":"1"},{"start":"fd","length":"6"},{"start":"104","length":"8"},{"start":"10d","length":"1"},{"start":"10f","length":"3"},{"start":"113","length":"6"},{"start":"11a","length":"1"},{"start":"11c","length":"3"},{"start":"120","length":"1"},{"start":"123","length":"1"},{"start":"125","length":"1"},{"start":"129","length":"1"},{"start":"133","length":"1"}]},{"pgid":"1.6","version":"77'128","reported_seq":1654,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.292062+0000","last_change":"2024-09-20T22:52:38.843686+0000","last_active":"2024-09-20T22:53:08.292062+0000","last_peered":"2024-09-20T22:53:08.292062+0000","last_clean":"2024-09-20T22:53:08.292062+0000","last_became_active":"2024-09-20T22:47:52.212190+0000","last_became_peered":"2024-09-20T22:47:52.212190+0000","last_unstale":"2024-09-20T22:53:08.292062+0000","last_undegraded":"2024-09-20T22:53:08.292062+0000","last_fullsized":"2024-09-20T22:53:08.292062+0000","mapping_epoch":391,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:52:38.843586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:52:38.843586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:07.967280+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"625'37","reported_seq":708,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.286855+0000","last_change":"2024-09-20T22:52:39.703172+0000","last_active":"2024-09-20T22:53:08.286855+0000","last_peered":"2024-09-20T22:53:08.286855+0000","last_clean":"2024-09-20T22:53:08.286855+0000","last_became_active":"2024-09-20T22:47:49.222009+0000","last_became_peered":"2024-09-20T22:47:49.222009+0000","last_unstale":"2024-09-20T22:53:08.286855+0000","last_undegraded":"2024-09-20T22:53:08.286855+0000","last_fullsized":"2024-09-20T22:53:08.286855+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'37","last_scrub_stamp":"2024-09-20T22:52:39.703093+0000","last_deep_scrub":"515'27","last_deep_scrub_stamp":"2024-09-20T22:49:55.964153+0000","last_clean_scrub_stamp":"2024-09-20T22:52:39.703093+0000","objects_scrubbed":8,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:49.380387+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":53,"num_read_kb":39,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":236,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.287842+0000","last_change":"2024-09-20T22:52:14.225251+0000","last_active":"2024-09-20T22:53:08.287842+0000","last_peered":"2024-09-20T22:53:08.287842+0000","last_clean":"2024-09-20T22:53:08.287842+0000","last_became_active":"2024-09-20T22:51:20.286646+0000","last_became_peered":"2024-09-20T22:51:20.286646+0000","last_unstale":"2024-09-20T22:53:08.287842+0000","last_undegraded":"2024-09-20T22:53:08.287842+0000","last_fullsized":"2024-09-20T22:53:08.287842+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:14.225111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:48:23.320031+0000","last_clean_scrub_stamp":"2024-09-20T22:52:14.225111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:38.894957+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":1347,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.291300+0000","last_change":"2024-09-20T22:52:39.814499+0000","last_active":"2024-09-20T22:53:08.291300+0000","last_peered":"2024-09-20T22:53:08.291300+0000","last_clean":"2024-09-20T22:53:08.291300+0000","last_became_active":"2024-09-20T22:47:49.214055+0000","last_became_peered":"2024-09-20T22:47:49.214055+0000","last_unstale":"2024-09-20T22:53:08.291300+0000","last_undegraded":"2024-09-20T22:53:08.291300+0000","last_fullsized":"2024-09-20T22:53:08.291300+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:39.814381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:52:39.814381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:52.010799+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":244,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.291182+0000","last_change":"2024-09-20T22:53:02.916523+0000","last_active":"2024-09-20T22:53:08.291182+0000","last_peered":"2024-09-20T22:53:08.291182+0000","last_clean":"2024-09-20T22:53:08.291182+0000","last_became_active":"2024-09-20T22:51:20.284874+0000","last_became_peered":"2024-09-20T22:51:20.284874+0000","last_unstale":"2024-09-20T22:53:08.291182+0000","last_undegraded":"2024-09-20T22:53:08.291182+0000","last_fullsized":"2024-09-20T22:53:08.291182+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:02.916449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:48:41.208853+0000","last_clean_scrub_stamp":"2024-09-20T22:53:02.916449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:15.012347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.7","version":"708'1239","reported_seq":4353,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.302283+0000","last_change":"2024-09-20T22:53:08.302510+0000","last_active":"2024-09-20T22:53:08.302283+0000","last_peered":"2024-09-20T22:53:08.302283+0000","last_clean":"2024-09-20T22:53:08.302283+0000","last_became_active":"2024-09-20T22:48:43.224386+0000","last_became_peered":"2024-09-20T22:48:43.224386+0000","last_unstale":"2024-09-20T22:53:08.302283+0000","last_undegraded":"2024-09-20T22:53:08.302283+0000","last_fullsized":"2024-09-20T22:53:08.302283+0000","mapping_epoch":391,"log_start":"436'456","ondisk_log_start":"436'456","created":94,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"704'1235","last_scrub_stamp":"2024-09-20T22:53:06.830408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:53:06.830408+0000","objects_scrubbed":31,"log_size":783,"log_dups_size":886,"ondisk_log_size":783,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:22.349820+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0096448039999999999,"stat_sum":{"num_bytes":63153091,"num_objects":31,"num_object_clones":26,"num_object_copies":93,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":2,"num_read":442,"num_read_kb":450717,"num_write":381,"num_write_kb":126766,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":115,"num_bytes_recovered":51615167,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"bf","length":"1"},{"start":"f8","length":"1"},{"start":"fb","length":"1"},{"start":"fd","length":"6"},{"start":"104","length":"8"},{"start":"10d","length":"5"},{"start":"113","length":"8"},{"start":"11c","length":"3"},{"start":"120","length":"1"},{"start":"123","length":"1"},{"start":"125","length":"2"},{"start":"129","length":"1"},{"start":"12b","length":"1"},{"start":"133","length":"2"}]},{"pgid":"1.1","version":"77'128","reported_seq":1643,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.285445+0000","last_change":"2024-09-20T22:52:42.146147+0000","last_active":"2024-09-20T22:53:08.285445+0000","last_peered":"2024-09-20T22:53:08.285445+0000","last_clean":"2024-09-20T22:53:08.285445+0000","last_became_active":"2024-09-20T22:47:52.208102+0000","last_became_peered":"2024-09-20T22:47:52.208102+0000","last_unstale":"2024-09-20T22:53:08.285445+0000","last_undegraded":"2024-09-20T22:53:08.285445+0000","last_fullsized":"2024-09-20T22:53:08.285445+0000","mapping_epoch":391,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:52:42.146069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:52:42.146069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:50.657914+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"625'51","reported_seq":409,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.287693+0000","last_change":"2024-09-20T22:52:17.199465+0000","last_active":"2024-09-20T22:53:08.287693+0000","last_peered":"2024-09-20T22:53:08.287693+0000","last_clean":"2024-09-20T22:53:08.287693+0000","last_became_active":"2024-09-20T22:51:20.289376+0000","last_became_peered":"2024-09-20T22:51:20.289376+0000","last_unstale":"2024-09-20T22:53:08.287693+0000","last_undegraded":"2024-09-20T22:53:08.287693+0000","last_fullsized":"2024-09-20T22:53:08.287693+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'51","last_scrub_stamp":"2024-09-20T22:52:17.199367+0000","last_deep_scrub":"86'38","last_deep_scrub_stamp":"2024-09-20T22:42:43.935677+0000","last_clean_scrub_stamp":"2024-09-20T22:52:17.199367+0000","objects_scrubbed":10,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:38.979686+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":634,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.290046+0000","last_change":"2024-09-20T22:52:36.695537+0000","last_active":"2024-09-20T22:53:08.290046+0000","last_peered":"2024-09-20T22:53:08.290046+0000","last_clean":"2024-09-20T22:53:08.290046+0000","last_became_active":"2024-09-20T22:48:05.237756+0000","last_became_peered":"2024-09-20T22:48:05.237756+0000","last_unstale":"2024-09-20T22:53:08.290046+0000","last_undegraded":"2024-09-20T22:53:08.290046+0000","last_fullsized":"2024-09-20T22:53:08.290046+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:36.695446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:52:36.695446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:38.713978+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"69'2","reported_seq":1356,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.286043+0000","last_change":"2024-09-20T22:51:37.070187+0000","last_active":"2024-09-20T22:53:08.286043+0000","last_peered":"2024-09-20T22:53:08.286043+0000","last_clean":"2024-09-20T22:53:08.286043+0000","last_became_active":"2024-09-20T22:51:20.286952+0000","last_became_peered":"2024-09-20T22:51:20.286952+0000","last_unstale":"2024-09-20T22:53:08.286043+0000","last_undegraded":"2024-09-20T22:53:08.286043+0000","last_fullsized":"2024-09-20T22:53:08.286043+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2024-09-20T22:51:37.070111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:36.130684+0000","last_clean_scrub_stamp":"2024-09-20T22:51:37.070111+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087280599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":679,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.287664+0000","last_change":"2024-09-20T22:51:53.772969+0000","last_active":"2024-09-20T22:53:08.287664+0000","last_peered":"2024-09-20T22:53:08.287664+0000","last_clean":"2024-09-20T22:53:08.287664+0000","last_became_active":"2024-09-20T22:51:20.278427+0000","last_became_peered":"2024-09-20T22:51:20.278427+0000","last_unstale":"2024-09-20T22:53:08.287664+0000","last_undegraded":"2024-09-20T22:53:08.287664+0000","last_fullsized":"2024-09-20T22:53:08.287664+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:53.772885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:51:53.772885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":1327,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.296223+0000","last_change":"2024-09-20T22:52:21.363432+0000","last_active":"2024-09-20T22:53:08.296223+0000","last_peered":"2024-09-20T22:53:08.296223+0000","last_clean":"2024-09-20T22:53:08.296223+0000","last_became_active":"2024-09-20T22:48:05.234193+0000","last_became_peered":"2024-09-20T22:48:05.234193+0000","last_unstale":"2024-09-20T22:53:08.296223+0000","last_undegraded":"2024-09-20T22:53:08.296223+0000","last_fullsized":"2024-09-20T22:53:08.296223+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:21.363359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:52:21.363359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:28.660587+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"708'440","reported_seq":1985,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.298859+0000","last_change":"2024-09-20T22:53:08.299149+0000","last_active":"2024-09-20T22:53:08.298859+0000","last_peered":"2024-09-20T22:53:08.298859+0000","last_clean":"2024-09-20T22:53:08.298859+0000","last_became_active":"2024-09-20T22:51:23.267613+0000","last_became_peered":"2024-09-20T22:51:23.267613+0000","last_unstale":"2024-09-20T22:53:08.298859+0000","last_undegraded":"2024-09-20T22:53:08.298859+0000","last_fullsized":"2024-09-20T22:53:08.298859+0000","mapping_epoch":602,"log_start":"456'176","ondisk_log_start":"456'176","created":94,"last_epoch_clean":603,"parent":"0.0","parent_split_bits":0,"last_scrub":"616'341","last_scrub_stamp":"2024-09-20T22:51:36.893731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:51:36.893731+0000","objects_scrubbed":8,"log_size":264,"log_dups_size":318,"ondisk_log_size":264,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.011398122,"stat_sum":{"num_bytes":26187339,"num_objects":11,"num_object_clones":9,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":158,"num_read_kb":145638,"num_write":138,"num_write_kb":46729,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":26566881,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"bf","length":"1"},{"start":"f8","length":"1"},{"start":"fb","length":"1"},{"start":"fd","length":"6"},{"start":"104","length":"8"},{"start":"10d","length":"5"},{"start":"113","length":"8"},{"start":"11c","length":"3"},{"start":"120","length":"1"},{"start":"123","length":"1"},{"start":"125","length":"2"},{"start":"129","length":"1"},{"start":"12b","length":"1"},{"start":"133","length":"2"}]},{"pgid":"1.2","version":"77'128","reported_seq":1703,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.289865+0000","last_change":"2024-09-20T22:52:02.623634+0000","last_active":"2024-09-20T22:53:08.289865+0000","last_peered":"2024-09-20T22:53:08.289865+0000","last_clean":"2024-09-20T22:53:08.289865+0000","last_became_active":"2024-09-20T22:48:08.203028+0000","last_became_peered":"2024-09-20T22:48:08.203028+0000","last_unstale":"2024-09-20T22:53:08.289865+0000","last_undegraded":"2024-09-20T22:53:08.289865+0000","last_fullsized":"2024-09-20T22:53:08.289865+0000","mapping_epoch":407,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":408,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:52:02.623562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:52:02.623562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:31.354752+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"625'22","reported_seq":1089,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.286611+0000","last_change":"2024-09-20T22:51:20.312887+0000","last_active":"2024-09-20T22:53:08.286611+0000","last_peered":"2024-09-20T22:53:08.286611+0000","last_clean":"2024-09-20T22:53:08.286611+0000","last_became_active":"2024-09-20T22:51:20.293610+0000","last_became_peered":"2024-09-20T22:51:20.293610+0000","last_unstale":"2024-09-20T22:53:08.286611+0000","last_undegraded":"2024-09-20T22:53:08.286611+0000","last_fullsized":"2024-09-20T22:53:08.286611+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"443'18","last_scrub_stamp":"2024-09-20T22:51:04.643936+0000","last_deep_scrub":"443'18","last_deep_scrub_stamp":"2024-09-20T22:48:44.182517+0000","last_clean_scrub_stamp":"2024-09-20T22:51:04.643936+0000","objects_scrubbed":5,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":990,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":236,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.288224+0000","last_change":"2024-09-20T22:51:32.279015+0000","last_active":"2024-09-20T22:53:08.288224+0000","last_peered":"2024-09-20T22:53:08.288224+0000","last_clean":"2024-09-20T22:53:08.288224+0000","last_became_active":"2024-09-20T22:51:20.278609+0000","last_became_peered":"2024-09-20T22:51:20.278609+0000","last_unstale":"2024-09-20T22:53:08.288224+0000","last_undegraded":"2024-09-20T22:53:08.288224+0000","last_fullsized":"2024-09-20T22:53:08.288224+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:32.278945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:51:32.278945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"380'2","reported_seq":1381,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.289195+0000","last_change":"2024-09-20T22:52:37.033309+0000","last_active":"2024-09-20T22:53:08.289195+0000","last_peered":"2024-09-20T22:53:08.289195+0000","last_clean":"2024-09-20T22:53:08.289195+0000","last_became_active":"2024-09-20T22:47:49.222812+0000","last_became_peered":"2024-09-20T22:47:49.222812+0000","last_unstale":"2024-09-20T22:53:08.289195+0000","last_undegraded":"2024-09-20T22:53:08.289195+0000","last_fullsized":"2024-09-20T22:53:08.289195+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"380'2","last_scrub_stamp":"2024-09-20T22:52:37.033222+0000","last_deep_scrub":"380'2","last_deep_scrub_stamp":"2024-09-20T22:47:40.983039+0000","last_clean_scrub_stamp":"2024-09-20T22:52:37.033222+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:56.197067+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001301459,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":1327,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.290069+0000","last_change":"2024-09-20T22:51:04.254552+0000","last_active":"2024-09-20T22:53:08.290069+0000","last_peered":"2024-09-20T22:53:08.290069+0000","last_clean":"2024-09-20T22:53:08.290069+0000","last_became_active":"2024-09-20T22:41:42.185112+0000","last_became_peered":"2024-09-20T22:41:42.185112+0000","last_unstale":"2024-09-20T22:53:08.290069+0000","last_undegraded":"2024-09-20T22:53:08.290069+0000","last_fullsized":"2024-09-20T22:53:08.290069+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:04.254441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:47:22.881109+0000","last_clean_scrub_stamp":"2024-09-20T22:51:04.254441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":1322,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.290445+0000","last_change":"2024-09-20T22:53:06.644045+0000","last_active":"2024-09-20T22:53:08.290445+0000","last_peered":"2024-09-20T22:53:08.290445+0000","last_clean":"2024-09-20T22:53:08.290445+0000","last_became_active":"2024-09-20T22:41:44.830515+0000","last_became_peered":"2024-09-20T22:41:44.830515+0000","last_unstale":"2024-09-20T22:53:08.290445+0000","last_undegraded":"2024-09-20T22:53:08.290445+0000","last_fullsized":"2024-09-20T22:53:08.290445+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:06.643947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:53:06.643947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:16.583688+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"708'351","reported_seq":3465,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.300848+0000","last_change":"2024-09-20T22:53:08.301106+0000","last_active":"2024-09-20T22:53:08.300848+0000","last_peered":"2024-09-20T22:53:08.300848+0000","last_clean":"2024-09-20T22:53:08.300848+0000","last_became_active":"2024-09-20T22:48:57.236417+0000","last_became_peered":"2024-09-20T22:48:57.236417+0000","last_unstale":"2024-09-20T22:53:08.300848+0000","last_undegraded":"2024-09-20T22:53:08.300848+0000","last_fullsized":"2024-09-20T22:53:08.300848+0000","mapping_epoch":391,"log_start":"450'190","ondisk_log_start":"450'190","created":94,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"674'316","last_scrub_stamp":"2024-09-20T22:52:35.685017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:52:35.685017+0000","objects_scrubbed":8,"log_size":161,"log_dups_size":240,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:46.348449+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.010728626999999999,"stat_sum":{"num_bytes":17366338,"num_objects":9,"num_object_clones":8,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":79,"num_read_kb":60076,"num_write":94,"num_write_kb":40424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":19673726,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"bf","length":"1"},{"start":"f8","length":"1"},{"start":"fb","length":"1"},{"start":"fd","length":"6"},{"start":"104","length":"8"},{"start":"10d","length":"5"},{"start":"113","length":"8"},{"start":"11c","length":"3"},{"start":"120","length":"1"},{"start":"123","length":"1"},{"start":"125","length":"2"},{"start":"129","length":"1"},{"start":"12b","length":"1"},{"start":"133","length":"2"}]},{"pgid":"1.3","version":"77'128","reported_seq":539,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.291480+0000","last_change":"2024-09-20T22:53:05.866046+0000","last_active":"2024-09-20T22:53:08.291480+0000","last_peered":"2024-09-20T22:53:08.291480+0000","last_clean":"2024-09-20T22:53:08.291480+0000","last_became_active":"2024-09-20T22:51:23.266810+0000","last_became_peered":"2024-09-20T22:51:23.266810+0000","last_unstale":"2024-09-20T22:53:08.291480+0000","last_undegraded":"2024-09-20T22:53:08.291480+0000","last_fullsized":"2024-09-20T22:53:08.291480+0000","mapping_epoch":602,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":603,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:53:05.865942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:53:05.865942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:28.578041+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.6","version":"625'24","reported_seq":719,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.289317+0000","last_change":"2024-09-20T22:52:48.829973+0000","last_active":"2024-09-20T22:53:08.289317+0000","last_peered":"2024-09-20T22:53:08.289317+0000","last_clean":"2024-09-20T22:53:08.289317+0000","last_became_active":"2024-09-20T22:51:20.283112+0000","last_became_peered":"2024-09-20T22:51:20.283112+0000","last_unstale":"2024-09-20T22:53:08.289317+0000","last_undegraded":"2024-09-20T22:53:08.289317+0000","last_fullsized":"2024-09-20T22:53:08.289317+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'24","last_scrub_stamp":"2024-09-20T22:52:48.829882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:52:48.829882+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:17.822454+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":673,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.286396+0000","last_change":"2024-09-20T22:50:39.786895+0000","last_active":"2024-09-20T22:53:08.286396+0000","last_peered":"2024-09-20T22:53:08.286396+0000","last_clean":"2024-09-20T22:53:08.286396+0000","last_became_active":"2024-09-20T22:48:05.233093+0000","last_became_peered":"2024-09-20T22:48:05.233093+0000","last_unstale":"2024-09-20T22:53:08.286396+0000","last_undegraded":"2024-09-20T22:53:08.286396+0000","last_fullsized":"2024-09-20T22:53:08.286396+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:50:39.786825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:44:19.834641+0000","last_clean_scrub_stamp":"2024-09-20T22:50:39.786825+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"75'5","reported_seq":2057,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.289960+0000","last_change":"2024-09-20T22:52:44.568232+0000","last_active":"2024-09-20T22:53:08.289960+0000","last_peered":"2024-09-20T22:53:08.289960+0000","last_clean":"2024-09-20T22:53:08.289960+0000","last_became_active":"2024-09-20T22:51:20.281329+0000","last_became_peered":"2024-09-20T22:51:20.281329+0000","last_unstale":"2024-09-20T22:53:08.289960+0000","last_undegraded":"2024-09-20T22:53:08.289960+0000","last_fullsized":"2024-09-20T22:53:08.289960+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-09-20T22:52:44.568129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:36.130684+0000","last_clean_scrub_stamp":"2024-09-20T22:52:44.568129+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:05.662332+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00078631700000000003,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":680,"num_read_kb":675,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1167,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":1346,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.286646+0000","last_change":"2024-09-20T22:52:48.082097+0000","last_active":"2024-09-20T22:53:08.286646+0000","last_peered":"2024-09-20T22:53:08.286646+0000","last_clean":"2024-09-20T22:53:08.286646+0000","last_became_active":"2024-09-20T22:51:20.274522+0000","last_became_peered":"2024-09-20T22:51:20.274522+0000","last_unstale":"2024-09-20T22:53:08.286646+0000","last_undegraded":"2024-09-20T22:53:08.286646+0000","last_fullsized":"2024-09-20T22:53:08.286646+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:48.082027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:52:48.082027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:51.217095+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":228,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.286349+0000","last_change":"2024-09-20T22:51:20.288058+0000","last_active":"2024-09-20T22:53:08.286349+0000","last_peered":"2024-09-20T22:53:08.286349+0000","last_clean":"2024-09-20T22:53:08.286349+0000","last_became_active":"2024-09-20T22:51:20.287750+0000","last_became_peered":"2024-09-20T22:51:20.287750+0000","last_unstale":"2024-09-20T22:53:08.286349+0000","last_undegraded":"2024-09-20T22:53:08.286349+0000","last_fullsized":"2024-09-20T22:53:08.286349+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:05.621880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:51:05.621880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.6","version":"708'456","reported_seq":3503,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.303246+0000","last_change":"2024-09-20T22:53:08.303511+0000","last_active":"2024-09-20T22:53:08.303246+0000","last_peered":"2024-09-20T22:53:08.303246+0000","last_clean":"2024-09-20T22:53:08.303246+0000","last_became_active":"2024-09-20T22:48:47.234531+0000","last_became_peered":"2024-09-20T22:48:47.234531+0000","last_unstale":"2024-09-20T22:53:08.303246+0000","last_undegraded":"2024-09-20T22:53:08.303246+0000","last_fullsized":"2024-09-20T22:53:08.303246+0000","mapping_epoch":391,"log_start":"443'181","ondisk_log_start":"443'181","created":94,"last_epoch_clean":447,"parent":"0.0","parent_split_bits":0,"last_scrub":"654'392","last_scrub_stamp":"2024-09-20T22:52:14.706591+0000","last_deep_scrub":"440'177","last_deep_scrub_stamp":"2024-09-20T22:48:40.602342+0000","last_clean_scrub_stamp":"2024-09-20T22:52:14.706591+0000","objects_scrubbed":10,"log_size":275,"log_dups_size":314,"ondisk_log_size":275,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:31.145032+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.014507139,"stat_sum":{"num_bytes":27586943,"num_objects":12,"num_object_clones":10,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":188,"num_read_kb":193597,"num_write":139,"num_write_kb":52384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":18662529,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"bf","length":"1"},{"start":"f8","length":"1"},{"start":"fb","length":"1"},{"start":"fd","length":"6"},{"start":"104","length":"8"},{"start":"10d","length":"5"},{"start":"113","length":"8"},{"start":"11c","length":"3"},{"start":"120","length":"1"},{"start":"123","length":"1"},{"start":"125","length":"2"},{"start":"129","length":"1"},{"start":"12b","length":"1"},{"start":"133","length":"2"}]},{"pgid":"4.5","version":"625'32","reported_seq":1614,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.291963+0000","last_change":"2024-09-20T22:53:07.867174+0000","last_active":"2024-09-20T22:53:08.291963+0000","last_peered":"2024-09-20T22:53:08.291963+0000","last_clean":"2024-09-20T22:53:08.291963+0000","last_became_active":"2024-09-20T22:48:05.243523+0000","last_became_peered":"2024-09-20T22:48:05.243523+0000","last_unstale":"2024-09-20T22:53:08.291963+0000","last_undegraded":"2024-09-20T22:53:08.291963+0000","last_fullsized":"2024-09-20T22:53:08.291963+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'32","last_scrub_stamp":"2024-09-20T22:53:07.867088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:53:07.867088+0000","objects_scrubbed":8,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:23.661414+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"67'1","reported_seq":1325,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.297739+0000","last_change":"2024-09-20T22:51:35.440516+0000","last_active":"2024-09-20T22:53:08.297739+0000","last_peered":"2024-09-20T22:53:08.297739+0000","last_clean":"2024-09-20T22:53:08.297739+0000","last_became_active":"2024-09-20T22:41:38.158917+0000","last_became_peered":"2024-09-20T22:41:38.158917+0000","last_unstale":"2024-09-20T22:53:08.297739+0000","last_undegraded":"2024-09-20T22:53:08.297739+0000","last_fullsized":"2024-09-20T22:53:08.297739+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-09-20T22:51:35.440416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:51:35.440416+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"77'128","reported_seq":1675,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.290127+0000","last_change":"2024-09-20T22:52:18.688292+0000","last_active":"2024-09-20T22:53:08.290127+0000","last_peered":"2024-09-20T22:53:08.290127+0000","last_clean":"2024-09-20T22:53:08.290127+0000","last_became_active":"2024-09-20T22:47:52.206169+0000","last_became_peered":"2024-09-20T22:47:52.206169+0000","last_unstale":"2024-09-20T22:53:08.290127+0000","last_undegraded":"2024-09-20T22:53:08.290127+0000","last_fullsized":"2024-09-20T22:53:08.290127+0000","mapping_epoch":391,"log_start":"77'128","ondisk_log_start":"77'128","created":19,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:52:18.688135+0000","last_deep_scrub":"77'128","last_deep_scrub_stamp":"2024-09-20T22:47:22.417099+0000","last_clean_scrub_stamp":"2024-09-20T22:52:18.688135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:47.016821+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":1334,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.290740+0000","last_change":"2024-09-20T22:53:03.564792+0000","last_active":"2024-09-20T22:53:08.290740+0000","last_peered":"2024-09-20T22:53:08.290740+0000","last_clean":"2024-09-20T22:53:08.290740+0000","last_became_active":"2024-09-20T22:41:42.184387+0000","last_became_peered":"2024-09-20T22:41:42.184387+0000","last_unstale":"2024-09-20T22:53:08.290740+0000","last_undegraded":"2024-09-20T22:53:08.290740+0000","last_fullsized":"2024-09-20T22:53:08.290740+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:03.564726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:53:03.564726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:25.313703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":1328,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.286334+0000","last_change":"2024-09-20T22:49:57.715494+0000","last_active":"2024-09-20T22:53:08.286334+0000","last_peered":"2024-09-20T22:53:08.286334+0000","last_clean":"2024-09-20T22:53:08.286334+0000","last_became_active":"2024-09-20T22:48:05.241602+0000","last_became_peered":"2024-09-20T22:48:05.241602+0000","last_unstale":"2024-09-20T22:53:08.286334+0000","last_undegraded":"2024-09-20T22:53:08.286334+0000","last_fullsized":"2024-09-20T22:53:08.286334+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:57.715416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:49:57.715416+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"708'553","reported_seq":3718,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.296812+0000","last_change":"2024-09-20T22:53:08.297116+0000","last_active":"2024-09-20T22:53:08.296812+0000","last_peered":"2024-09-20T22:53:08.296812+0000","last_clean":"2024-09-20T22:53:08.296812+0000","last_became_active":"2024-09-20T22:51:23.269216+0000","last_became_peered":"2024-09-20T22:51:23.269216+0000","last_unstale":"2024-09-20T22:53:08.296812+0000","last_undegraded":"2024-09-20T22:53:08.296812+0000","last_fullsized":"2024-09-20T22:53:08.296812+0000","mapping_epoch":602,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":603,"parent":"0.0","parent_split_bits":0,"last_scrub":"702'544","last_scrub_stamp":"2024-09-20T22:53:03.918130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:53:03.918130+0000","objects_scrubbed":10,"log_size":553,"log_dups_size":0,"ondisk_log_size":553,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:28.183552+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.0051844819999999998,"stat_sum":{"num_bytes":24280942,"num_objects":11,"num_object_clones":9,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":178,"num_read_kb":171553,"num_write":107,"num_write_kb":43070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":52,"num_bytes_recovered":30669453,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"bf","length":"1"},{"start":"f8","length":"1"},{"start":"fb","length":"1"},{"start":"fd","length":"6"},{"start":"104","length":"8"},{"start":"10d","length":"5"},{"start":"113","length":"8"},{"start":"11c","length":"3"},{"start":"120","length":"1"},{"start":"123","length":"1"},{"start":"125","length":"2"},{"start":"129","length":"1"},{"start":"12b","length":"1"},{"start":"133","length":"2"}]},{"pgid":"4.e","version":"625'19","reported_seq":676,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.292716+0000","last_change":"2024-09-20T22:52:44.783087+0000","last_active":"2024-09-20T22:53:08.292716+0000","last_peered":"2024-09-20T22:53:08.292716+0000","last_clean":"2024-09-20T22:53:08.292716+0000","last_became_active":"2024-09-20T22:51:20.288958+0000","last_became_peered":"2024-09-20T22:51:20.288958+0000","last_unstale":"2024-09-20T22:53:08.292716+0000","last_undegraded":"2024-09-20T22:53:08.292716+0000","last_fullsized":"2024-09-20T22:53:08.292716+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'19","last_scrub_stamp":"2024-09-20T22:52:44.782975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:52:44.782975+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:48.007796+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":634,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.293537+0000","last_change":"2024-09-20T22:51:29.757136+0000","last_active":"2024-09-20T22:53:08.293537+0000","last_peered":"2024-09-20T22:53:08.293537+0000","last_clean":"2024-09-20T22:53:08.293537+0000","last_became_active":"2024-09-20T22:48:05.254688+0000","last_became_peered":"2024-09-20T22:48:05.254688+0000","last_unstale":"2024-09-20T22:53:08.293537+0000","last_undegraded":"2024-09-20T22:53:08.293537+0000","last_fullsized":"2024-09-20T22:53:08.293537+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:29.757030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:51:29.757030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":1332,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.286242+0000","last_change":"2024-09-20T22:51:51.002353+0000","last_active":"2024-09-20T22:53:08.286242+0000","last_peered":"2024-09-20T22:53:08.286242+0000","last_clean":"2024-09-20T22:53:08.286242+0000","last_became_active":"2024-09-20T22:48:05.233778+0000","last_became_peered":"2024-09-20T22:48:05.233778+0000","last_unstale":"2024-09-20T22:53:08.286242+0000","last_undegraded":"2024-09-20T22:53:08.286242+0000","last_fullsized":"2024-09-20T22:53:08.286242+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:51.002226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:51:51.002226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:20.564071+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":1307,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.296194+0000","last_change":"2024-09-20T22:50:47.230929+0000","last_active":"2024-09-20T22:53:08.296194+0000","last_peered":"2024-09-20T22:53:08.296194+0000","last_clean":"2024-09-20T22:53:08.296194+0000","last_became_active":"2024-09-20T22:41:44.832009+0000","last_became_peered":"2024-09-20T22:41:44.832009+0000","last_unstale":"2024-09-20T22:53:08.296194+0000","last_undegraded":"2024-09-20T22:53:08.296194+0000","last_fullsized":"2024-09-20T22:53:08.296194+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:50:47.230854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:48:40.127123+0000","last_clean_scrub_stamp":"2024-09-20T22:50:47.230854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.e","version":"708'673","reported_seq":3785,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.297878+0000","last_change":"2024-09-20T22:53:08.298106+0000","last_active":"2024-09-20T22:53:08.297878+0000","last_peered":"2024-09-20T22:53:08.297878+0000","last_clean":"2024-09-20T22:53:08.297878+0000","last_became_active":"2024-09-20T22:47:52.209452+0000","last_became_peered":"2024-09-20T22:47:52.209452+0000","last_unstale":"2024-09-20T22:53:08.297878+0000","last_undegraded":"2024-09-20T22:53:08.297878+0000","last_fullsized":"2024-09-20T22:53:08.297878+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"681'618","last_scrub_stamp":"2024-09-20T22:52:45.935130+0000","last_deep_scrub":"174'60","last_deep_scrub_stamp":"2024-09-20T22:44:18.301776+0000","last_clean_scrub_stamp":"2024-09-20T22:52:45.935130+0000","objects_scrubbed":13,"log_size":673,"log_dups_size":0,"ondisk_log_size":673,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:47.673915+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.0085316109999999997,"stat_sum":{"num_bytes":28690576,"num_objects":14,"num_object_clones":12,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":247,"num_read_kb":256134,"num_write":165,"num_write_kb":61453,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":23110741,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"bf","length":"1"},{"start":"f8","length":"1"},{"start":"fb","length":"1"},{"start":"fd","length":"6"},{"start":"104","length":"8"},{"start":"10d","length":"5"},{"start":"113","length":"8"},{"start":"11c","length":"3"},{"start":"120","length":"1"},{"start":"123","length":"1"},{"start":"125","length":"2"},{"start":"129","length":"1"},{"start":"12b","length":"1"},{"start":"133","length":"2"}]},{"pgid":"1.8","version":"77'128","reported_seq":1595,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.285428+0000","last_change":"2024-09-20T22:51:56.108038+0000","last_active":"2024-09-20T22:53:08.285428+0000","last_peered":"2024-09-20T22:53:08.285428+0000","last_clean":"2024-09-20T22:53:08.285428+0000","last_became_active":"2024-09-20T22:42:52.405594+0000","last_became_peered":"2024-09-20T22:42:52.405594+0000","last_unstale":"2024-09-20T22:53:08.285428+0000","last_undegraded":"2024-09-20T22:53:08.285428+0000","last_fullsized":"2024-09-20T22:53:08.285428+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:51:56.107944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:51:56.107944+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:19.128079+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"625'31","reported_seq":705,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.292941+0000","last_change":"2024-09-20T22:52:13.620897+0000","last_active":"2024-09-20T22:53:08.292941+0000","last_peered":"2024-09-20T22:53:08.292941+0000","last_clean":"2024-09-20T22:53:08.292941+0000","last_became_active":"2024-09-20T22:51:20.288846+0000","last_became_peered":"2024-09-20T22:51:20.288846+0000","last_unstale":"2024-09-20T22:53:08.292941+0000","last_undegraded":"2024-09-20T22:53:08.292941+0000","last_fullsized":"2024-09-20T22:53:08.292941+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'31","last_scrub_stamp":"2024-09-20T22:52:13.620754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:52:13.620754+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:26.544483+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":49,"num_read_kb":39,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":890,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.289932+0000","last_change":"2024-09-20T22:53:08.105116+0000","last_active":"2024-09-20T22:53:08.289932+0000","last_peered":"2024-09-20T22:53:08.289932+0000","last_clean":"2024-09-20T22:53:08.289932+0000","last_became_active":"2024-09-20T22:51:20.294073+0000","last_became_peered":"2024-09-20T22:51:20.294073+0000","last_unstale":"2024-09-20T22:53:08.289932+0000","last_undegraded":"2024-09-20T22:53:08.289932+0000","last_fullsized":"2024-09-20T22:53:08.289932+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:08.105010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:35.492562+0000","last_clean_scrub_stamp":"2024-09-20T22:53:08.105010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:35.105470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":946,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.289528+0000","last_change":"2024-09-20T22:51:25.738132+0000","last_active":"2024-09-20T22:53:08.289528+0000","last_peered":"2024-09-20T22:53:08.289528+0000","last_clean":"2024-09-20T22:53:08.289528+0000","last_became_active":"2024-09-20T22:51:20.281724+0000","last_became_peered":"2024-09-20T22:51:20.281724+0000","last_unstale":"2024-09-20T22:53:08.289528+0000","last_undegraded":"2024-09-20T22:53:08.289528+0000","last_fullsized":"2024-09-20T22:53:08.289528+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:25.738061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:51:25.738061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":1351,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.288065+0000","last_change":"2024-09-20T22:53:05.106323+0000","last_active":"2024-09-20T22:53:08.288065+0000","last_peered":"2024-09-20T22:53:08.288065+0000","last_clean":"2024-09-20T22:53:08.288065+0000","last_became_active":"2024-09-20T22:48:05.239006+0000","last_became_peered":"2024-09-20T22:48:05.239006+0000","last_unstale":"2024-09-20T22:53:08.288065+0000","last_undegraded":"2024-09-20T22:53:08.288065+0000","last_fullsized":"2024-09-20T22:53:08.288065+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:05.106224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:53:05.106224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:05.128923+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"708'740","reported_seq":3819,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.305218+0000","last_change":"2024-09-20T22:53:08.305449+0000","last_active":"2024-09-20T22:53:08.305218+0000","last_peered":"2024-09-20T22:53:08.305218+0000","last_clean":"2024-09-20T22:53:08.305218+0000","last_became_active":"2024-09-20T22:47:52.216198+0000","last_became_peered":"2024-09-20T22:47:52.216198+0000","last_unstale":"2024-09-20T22:53:08.305218+0000","last_undegraded":"2024-09-20T22:53:08.305218+0000","last_fullsized":"2024-09-20T22:53:08.305218+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"630'603","last_scrub_stamp":"2024-09-20T22:51:50.769050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:51:50.769050+0000","objects_scrubbed":8,"log_size":740,"log_dups_size":0,"ondisk_log_size":740,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.015620185999999999,"stat_sum":{"num_bytes":28441281,"num_objects":14,"num_object_clones":11,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":1,"num_read":306,"num_read_kb":304372,"num_write":187,"num_write_kb":60596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":53,"num_bytes_recovered":14983556,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"bf","length":"1"},{"start":"f8","length":"1"},{"start":"fb","length":"1"},{"start":"fd","length":"6"},{"start":"104","length":"8"},{"start":"10d","length":"5"},{"start":"113","length":"8"},{"start":"11c","length":"3"},{"start":"120","length":"1"},{"start":"123","length":"1"},{"start":"125","length":"2"},{"start":"129","length":"1"},{"start":"12b","length":"1"},{"start":"133","length":"2"}]},{"pgid":"1.9","version":"77'128","reported_seq":1643,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.289917+0000","last_change":"2024-09-20T22:52:39.568327+0000","last_active":"2024-09-20T22:53:08.289917+0000","last_peered":"2024-09-20T22:53:08.289917+0000","last_clean":"2024-09-20T22:53:08.289917+0000","last_became_active":"2024-09-20T22:50:42.262862+0000","last_became_peered":"2024-09-20T22:50:42.262862+0000","last_unstale":"2024-09-20T22:53:08.289917+0000","last_undegraded":"2024-09-20T22:53:08.289917+0000","last_fullsized":"2024-09-20T22:53:08.289917+0000","mapping_epoch":561,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":562,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:52:39.568240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:52:39.568240+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:39.745450+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"625'20","reported_seq":661,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.292601+0000","last_change":"2024-09-20T22:52:22.646547+0000","last_active":"2024-09-20T22:53:08.292601+0000","last_peered":"2024-09-20T22:53:08.292601+0000","last_clean":"2024-09-20T22:53:08.292601+0000","last_became_active":"2024-09-20T22:48:05.242811+0000","last_became_peered":"2024-09-20T22:48:05.242811+0000","last_unstale":"2024-09-20T22:53:08.292601+0000","last_undegraded":"2024-09-20T22:53:08.292601+0000","last_fullsized":"2024-09-20T22:53:08.292601+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'20","last_scrub_stamp":"2024-09-20T22:52:22.646468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:52:22.646468+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:48.440427+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":1351,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.298416+0000","last_change":"2024-09-20T22:52:44.527004+0000","last_active":"2024-09-20T22:53:08.298416+0000","last_peered":"2024-09-20T22:53:08.298416+0000","last_clean":"2024-09-20T22:53:08.298416+0000","last_became_active":"2024-09-20T22:48:05.233999+0000","last_became_peered":"2024-09-20T22:48:05.233999+0000","last_unstale":"2024-09-20T22:53:08.298416+0000","last_undegraded":"2024-09-20T22:53:08.298416+0000","last_fullsized":"2024-09-20T22:53:08.298416+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:44.526935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:52:44.526935+0000","last_clean_scrub_stamp":"2024-09-20T22:52:44.526935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:45.123238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"261'9","reported_seq":1483,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.288543+0000","last_change":"2024-09-20T22:52:18.218722+0000","last_active":"2024-09-20T22:53:08.288543+0000","last_peered":"2024-09-20T22:53:08.288543+0000","last_clean":"2024-09-20T22:53:08.288543+0000","last_became_active":"2024-09-20T22:41:42.185863+0000","last_became_peered":"2024-09-20T22:41:42.185863+0000","last_unstale":"2024-09-20T22:53:08.288543+0000","last_undegraded":"2024-09-20T22:53:08.288543+0000","last_fullsized":"2024-09-20T22:53:08.288543+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'9","last_scrub_stamp":"2024-09-20T22:52:18.218644+0000","last_deep_scrub":"261'9","last_deep_scrub_stamp":"2024-09-20T22:46:50.974735+0000","last_clean_scrub_stamp":"2024-09-20T22:52:18.218644+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:38.055984+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":640,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.292454+0000","last_change":"2024-09-20T22:51:20.296567+0000","last_active":"2024-09-20T22:53:08.292454+0000","last_peered":"2024-09-20T22:53:08.292454+0000","last_clean":"2024-09-20T22:53:08.292454+0000","last_became_active":"2024-09-20T22:51:20.296204+0000","last_became_peered":"2024-09-20T22:51:20.296204+0000","last_unstale":"2024-09-20T22:53:08.292454+0000","last_undegraded":"2024-09-20T22:53:08.292454+0000","last_fullsized":"2024-09-20T22:53:08.292454+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:02.932364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:51:02.932364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.8","version":"708'839","reported_seq":3770,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.297189+0000","last_change":"2024-09-20T22:53:08.298454+0000","last_active":"2024-09-20T22:53:08.297189+0000","last_peered":"2024-09-20T22:53:08.297189+0000","last_clean":"2024-09-20T22:53:08.297189+0000","last_became_active":"2024-09-20T22:48:31.224359+0000","last_became_peered":"2024-09-20T22:48:31.224359+0000","last_unstale":"2024-09-20T22:53:08.297189+0000","last_undegraded":"2024-09-20T22:53:08.297189+0000","last_fullsized":"2024-09-20T22:53:08.297189+0000","mapping_epoch":407,"log_start":"426'268","ondisk_log_start":"426'268","created":94,"last_epoch_clean":431,"parent":"0.0","parent_split_bits":0,"last_scrub":"674'746","last_scrub_stamp":"2024-09-20T22:52:35.065256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:52:35.065256+0000","objects_scrubbed":20,"log_size":571,"log_dups_size":445,"ondisk_log_size":571,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:50.034752+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.011046392,"stat_sum":{"num_bytes":44032348,"num_objects":20,"num_object_clones":17,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":1,"num_read":291,"num_read_kb":300600,"num_write":225,"num_write_kb":72122,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":26027409,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"bf","length":"1"},{"start":"f8","length":"1"},{"start":"fb","length":"1"},{"start":"fd","length":"6"},{"start":"104","length":"8"},{"start":"10d","length":"5"},{"start":"113","length":"8"},{"start":"11c","length":"3"},{"start":"120","length":"1"},{"start":"123","length":"1"},{"start":"125","length":"2"},{"start":"129","length":"1"},{"start":"12b","length":"1"},{"start":"133","length":"2"}]},{"pgid":"4.b","version":"625'15","reported_seq":718,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.287994+0000","last_change":"2024-09-20T22:51:20.296532+0000","last_active":"2024-09-20T22:53:08.287994+0000","last_peered":"2024-09-20T22:53:08.287994+0000","last_clean":"2024-09-20T22:53:08.287994+0000","last_became_active":"2024-09-20T22:51:20.273094+0000","last_became_peered":"2024-09-20T22:51:20.273094+0000","last_unstale":"2024-09-20T22:53:08.287994+0000","last_undegraded":"2024-09-20T22:53:08.287994+0000","last_fullsized":"2024-09-20T22:53:08.287994+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-09-20T22:50:36.883811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:50:36.883811+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":1328,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.286591+0000","last_change":"2024-09-20T22:51:12.755628+0000","last_active":"2024-09-20T22:53:08.286591+0000","last_peered":"2024-09-20T22:53:08.286591+0000","last_clean":"2024-09-20T22:53:08.286591+0000","last_became_active":"2024-09-20T22:41:38.155953+0000","last_became_peered":"2024-09-20T22:41:38.155953+0000","last_unstale":"2024-09-20T22:53:08.286591+0000","last_undegraded":"2024-09-20T22:53:08.286591+0000","last_fullsized":"2024-09-20T22:53:08.286591+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:12.755526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:51:12.755526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":1359,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.290038+0000","last_change":"2024-09-20T22:52:42.998852+0000","last_active":"2024-09-20T22:53:08.290038+0000","last_peered":"2024-09-20T22:53:08.290038+0000","last_clean":"2024-09-20T22:53:08.290038+0000","last_became_active":"2024-09-20T22:48:05.240613+0000","last_became_peered":"2024-09-20T22:48:05.240613+0000","last_unstale":"2024-09-20T22:53:08.290038+0000","last_undegraded":"2024-09-20T22:53:08.290038+0000","last_fullsized":"2024-09-20T22:53:08.290038+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:42.998756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:52:42.998756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:11.946882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":667,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.286485+0000","last_change":"2024-09-20T22:52:16.706497+0000","last_active":"2024-09-20T22:53:08.286485+0000","last_peered":"2024-09-20T22:53:08.286485+0000","last_clean":"2024-09-20T22:53:08.286485+0000","last_became_active":"2024-09-20T22:47:49.219486+0000","last_became_peered":"2024-09-20T22:47:49.219486+0000","last_unstale":"2024-09-20T22:53:08.286485+0000","last_undegraded":"2024-09-20T22:53:08.286485+0000","last_fullsized":"2024-09-20T22:53:08.286485+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:16.706429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:52:16.706429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:26.931246+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"708'1133","reported_seq":4270,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.302259+0000","last_change":"2024-09-20T22:53:08.302628+0000","last_active":"2024-09-20T22:53:08.302259+0000","last_peered":"2024-09-20T22:53:08.302259+0000","last_clean":"2024-09-20T22:53:08.302259+0000","last_became_active":"2024-09-20T22:51:23.270962+0000","last_became_peered":"2024-09-20T22:51:23.270962+0000","last_unstale":"2024-09-20T22:53:08.302259+0000","last_undegraded":"2024-09-20T22:53:08.302259+0000","last_fullsized":"2024-09-20T22:53:08.302259+0000","mapping_epoch":602,"log_start":"423'280","ondisk_log_start":"423'280","created":94,"last_epoch_clean":603,"parent":"0.0","parent_split_bits":0,"last_scrub":"674'1040","last_scrub_stamp":"2024-09-20T22:52:34.647394+0000","last_deep_scrub":"590'828","last_deep_scrub_stamp":"2024-09-20T22:51:10.909364+0000","last_clean_scrub_stamp":"2024-09-20T22:52:34.647394+0000","objects_scrubbed":29,"log_size":853,"log_dups_size":649,"ondisk_log_size":853,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:48.620185+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.015822605999999999,"stat_sum":{"num_bytes":65083069,"num_objects":28,"num_object_clones":23,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":2,"num_read":401,"num_read_kb":392866,"num_write":318,"num_write_kb":122185,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":91020358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"bf","length":"1"},{"start":"f8","length":"1"},{"start":"fb","length":"1"},{"start":"fd","length":"6"},{"start":"104","length":"8"},{"start":"10d","length":"5"},{"start":"113","length":"8"},{"start":"11c","length":"3"},{"start":"120","length":"1"},{"start":"123","length":"1"},{"start":"125","length":"2"},{"start":"129","length":"1"},{"start":"12b","length":"1"},{"start":"133","length":"2"}]},{"pgid":"4.a","version":"625'44","reported_seq":1429,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.291399+0000","last_change":"2024-09-20T22:52:14.038882+0000","last_active":"2024-09-20T22:53:08.291399+0000","last_peered":"2024-09-20T22:53:08.291399+0000","last_clean":"2024-09-20T22:53:08.291399+0000","last_became_active":"2024-09-20T22:51:20.287620+0000","last_became_peered":"2024-09-20T22:51:20.287620+0000","last_unstale":"2024-09-20T22:53:08.291399+0000","last_undegraded":"2024-09-20T22:53:08.291399+0000","last_fullsized":"2024-09-20T22:53:08.291399+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'44","last_scrub_stamp":"2024-09-20T22:52:14.038788+0000","last_deep_scrub":"113'28","last_deep_scrub_stamp":"2024-09-20T22:43:12.725900+0000","last_clean_scrub_stamp":"2024-09-20T22:52:14.038788+0000","objects_scrubbed":9,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:32.613295+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":1326,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.290667+0000","last_change":"2024-09-20T22:51:36.440271+0000","last_active":"2024-09-20T22:53:08.290667+0000","last_peered":"2024-09-20T22:53:08.290667+0000","last_clean":"2024-09-20T22:53:08.290667+0000","last_became_active":"2024-09-20T22:41:38.161524+0000","last_became_peered":"2024-09-20T22:41:38.161524+0000","last_unstale":"2024-09-20T22:53:08.290667+0000","last_undegraded":"2024-09-20T22:53:08.290667+0000","last_fullsized":"2024-09-20T22:53:08.290667+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:36.440160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:51:36.440160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":1347,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.287070+0000","last_change":"2024-09-20T22:52:40.004730+0000","last_active":"2024-09-20T22:53:08.287070+0000","last_peered":"2024-09-20T22:53:08.287070+0000","last_clean":"2024-09-20T22:53:08.287070+0000","last_became_active":"2024-09-20T22:47:49.223220+0000","last_became_peered":"2024-09-20T22:47:49.223220+0000","last_unstale":"2024-09-20T22:53:08.287070+0000","last_undegraded":"2024-09-20T22:53:08.287070+0000","last_fullsized":"2024-09-20T22:53:08.287070+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:40.004661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:52:40.004661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:04.999859+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":1322,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.290026+0000","last_change":"2024-09-20T22:52:01.643899+0000","last_active":"2024-09-20T22:53:08.290026+0000","last_peered":"2024-09-20T22:53:08.290026+0000","last_clean":"2024-09-20T22:53:08.290026+0000","last_became_active":"2024-09-20T22:41:44.199395+0000","last_became_peered":"2024-09-20T22:41:44.199395+0000","last_unstale":"2024-09-20T22:53:08.290026+0000","last_undegraded":"2024-09-20T22:53:08.290026+0000","last_fullsized":"2024-09-20T22:53:08.290026+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:01.643803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:47:18.329209+0000","last_clean_scrub_stamp":"2024-09-20T22:52:01.643803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.a","version":"708'835","reported_seq":3834,"reported_epoch":708,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T22:53:08.334861+0000","last_change":"2024-09-20T22:53:08.331844+0000","last_active":"2024-09-20T22:53:08.334861+0000","last_peered":"2024-09-20T22:53:08.334861+0000","last_clean":"2024-09-20T22:53:08.334861+0000","last_became_active":"2024-09-20T22:42:52.405085+0000","last_became_peered":"2024-09-20T22:42:52.405085+0000","last_unstale":"2024-09-20T22:53:08.334861+0000","last_undegraded":"2024-09-20T22:53:08.334861+0000","last_fullsized":"2024-09-20T22:53:08.334861+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"505'466","last_scrub_stamp":"2024-09-20T22:49:47.088165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:49:47.088165+0000","objects_scrubbed":13,"log_size":835,"log_dups_size":0,"ondisk_log_size":835,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.0063310459999999999,"stat_sum":{"num_bytes":57160920,"num_objects":22,"num_object_clones":19,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":353,"num_read_kb":370681,"num_write":271,"num_write_kb":105688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"bf","length":"1"},{"start":"f8","length":"1"},{"start":"fb","length":"1"},{"start":"fd","length":"6"},{"start":"104","length":"8"},{"start":"10d","length":"1"},{"start":"10f","length":"3"},{"start":"113","length":"6"},{"start":"11a","length":"1"},{"start":"11c","length":"3"},{"start":"120","length":"1"},{"start":"123","length":"1"},{"start":"125","length":"1"},{"start":"129","length":"1"},{"start":"133","length":"2"}]},{"pgid":"4.9","version":"686'23","reported_seq":696,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.293678+0000","last_change":"2024-09-20T22:52:46.768898+0000","last_active":"2024-09-20T22:53:08.293678+0000","last_peered":"2024-09-20T22:53:08.293678+0000","last_clean":"2024-09-20T22:53:08.293678+0000","last_became_active":"2024-09-20T22:52:11.286549+0000","last_became_peered":"2024-09-20T22:52:11.286549+0000","last_unstale":"2024-09-20T22:53:08.293678+0000","last_undegraded":"2024-09-20T22:53:08.293678+0000","last_fullsized":"2024-09-20T22:53:08.293678+0000","mapping_epoch":650,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":651,"parent":"0.0","parent_split_bits":0,"last_scrub":"686'23","last_scrub_stamp":"2024-09-20T22:52:46.768812+0000","last_deep_scrub":"686'23","last_deep_scrub_stamp":"2024-09-20T22:52:46.768812+0000","last_clean_scrub_stamp":"2024-09-20T22:52:46.768812+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:50.916744+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":21,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":1364,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.290468+0000","last_change":"2024-09-20T22:51:55.653755+0000","last_active":"2024-09-20T22:53:08.290468+0000","last_peered":"2024-09-20T22:53:08.290468+0000","last_clean":"2024-09-20T22:53:08.290468+0000","last_became_active":"2024-09-20T22:51:20.293701+0000","last_became_peered":"2024-09-20T22:51:20.293701+0000","last_unstale":"2024-09-20T22:53:08.290468+0000","last_undegraded":"2024-09-20T22:53:08.290468+0000","last_fullsized":"2024-09-20T22:53:08.290468+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:55.653662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:46:52.465940+0000","last_clean_scrub_stamp":"2024-09-20T22:51:55.653662+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":1361,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.287537+0000","last_change":"2024-09-20T22:53:07.104977+0000","last_active":"2024-09-20T22:53:08.287537+0000","last_peered":"2024-09-20T22:53:08.287537+0000","last_clean":"2024-09-20T22:53:08.287537+0000","last_became_active":"2024-09-20T22:51:20.286626+0000","last_became_peered":"2024-09-20T22:51:20.286626+0000","last_unstale":"2024-09-20T22:53:08.287537+0000","last_undegraded":"2024-09-20T22:53:08.287537+0000","last_fullsized":"2024-09-20T22:53:08.287537+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:07.104891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:53:07.104891+0000","last_clean_scrub_stamp":"2024-09-20T22:53:07.104891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:17.124586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":1313,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.298027+0000","last_change":"2024-09-20T22:51:20.279408+0000","last_active":"2024-09-20T22:53:08.298027+0000","last_peered":"2024-09-20T22:53:08.298027+0000","last_clean":"2024-09-20T22:53:08.298027+0000","last_became_active":"2024-09-20T22:51:20.279088+0000","last_became_peered":"2024-09-20T22:51:20.279088+0000","last_unstale":"2024-09-20T22:53:08.298027+0000","last_undegraded":"2024-09-20T22:53:08.298027+0000","last_fullsized":"2024-09-20T22:53:08.298027+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:50:39.266034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:55.170379+0000","last_clean_scrub_stamp":"2024-09-20T22:50:39.266034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"708'1099","reported_seq":4276,"reported_epoch":708,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T22:53:08.334605+0000","last_change":"2024-09-20T22:53:08.331990+0000","last_active":"2024-09-20T22:53:08.334605+0000","last_peered":"2024-09-20T22:53:08.334605+0000","last_clean":"2024-09-20T22:53:08.334605+0000","last_became_active":"2024-09-20T22:51:23.271244+0000","last_became_peered":"2024-09-20T22:51:23.271244+0000","last_unstale":"2024-09-20T22:53:08.334605+0000","last_undegraded":"2024-09-20T22:53:08.334605+0000","last_fullsized":"2024-09-20T22:53:08.334605+0000","mapping_epoch":602,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":603,"parent":"0.0","parent_split_bits":0,"last_scrub":"661'1000","last_scrub_stamp":"2024-09-20T22:52:23.164028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:52:23.164028+0000","objects_scrubbed":23,"log_size":1099,"log_dups_size":0,"ondisk_log_size":1099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:35.866726+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.044929170999999997,"stat_sum":{"num_bytes":61380231,"num_objects":26,"num_object_clones":21,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":2,"num_read":515,"num_read_kb":586632,"num_write":351,"num_write_kb":121518,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":76,"num_bytes_recovered":39288293,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"bf","length":"1"},{"start":"f8","length":"1"},{"start":"fb","length":"1"},{"start":"fd","length":"6"},{"start":"104","length":"8"},{"start":"10d","length":"1"},{"start":"10f","length":"3"},{"start":"113","length":"6"},{"start":"11a","length":"1"},{"start":"11c","length":"3"},{"start":"120","length":"1"},{"start":"123","length":"1"},{"start":"125","length":"1"},{"start":"129","length":"1"},{"start":"133","length":"2"}]},{"pgid":"4.8","version":"625'27","reported_seq":1367,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.285577+0000","last_change":"2024-09-20T22:51:53.020721+0000","last_active":"2024-09-20T22:53:08.285577+0000","last_peered":"2024-09-20T22:53:08.285577+0000","last_clean":"2024-09-20T22:53:08.285577+0000","last_became_active":"2024-09-20T22:41:40.252113+0000","last_became_peered":"2024-09-20T22:41:40.252113+0000","last_unstale":"2024-09-20T22:53:08.285577+0000","last_undegraded":"2024-09-20T22:53:08.285577+0000","last_fullsized":"2024-09-20T22:53:08.285577+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'27","last_scrub_stamp":"2024-09-20T22:51:53.020628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:51:53.020628+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:14.005659+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'2","reported_seq":1389,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.295544+0000","last_change":"2024-09-20T22:51:26.421985+0000","last_active":"2024-09-20T22:53:08.295544+0000","last_peered":"2024-09-20T22:53:08.295544+0000","last_clean":"2024-09-20T22:53:08.295544+0000","last_became_active":"2024-09-20T22:48:05.238098+0000","last_became_peered":"2024-09-20T22:48:05.238098+0000","last_unstale":"2024-09-20T22:53:08.295544+0000","last_undegraded":"2024-09-20T22:53:08.295544+0000","last_fullsized":"2024-09-20T22:53:08.295544+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-09-20T22:51:26.421896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:51:26.421896+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"408'10","reported_seq":1493,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.290179+0000","last_change":"2024-09-20T22:52:15.703572+0000","last_active":"2024-09-20T22:53:08.290179+0000","last_peered":"2024-09-20T22:53:08.290179+0000","last_clean":"2024-09-20T22:53:08.290179+0000","last_became_active":"2024-09-20T22:48:05.239344+0000","last_became_peered":"2024-09-20T22:48:05.239344+0000","last_unstale":"2024-09-20T22:53:08.290179+0000","last_undegraded":"2024-09-20T22:53:08.290179+0000","last_fullsized":"2024-09-20T22:53:08.290179+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"408'10","last_scrub_stamp":"2024-09-20T22:52:15.703464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:52:15.703464+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:23.273102+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":1328,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.285642+0000","last_change":"2024-09-20T22:51:54.066995+0000","last_active":"2024-09-20T22:53:08.285642+0000","last_peered":"2024-09-20T22:53:08.285642+0000","last_clean":"2024-09-20T22:53:08.285642+0000","last_became_active":"2024-09-20T22:47:49.227569+0000","last_became_peered":"2024-09-20T22:47:49.227569+0000","last_unstale":"2024-09-20T22:53:08.285642+0000","last_undegraded":"2024-09-20T22:53:08.285642+0000","last_fullsized":"2024-09-20T22:53:08.285642+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:54.066913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:51:54.066913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:18.059705+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":1354,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.291225+0000","last_change":"2024-09-20T22:51:58.129589+0000","last_active":"2024-09-20T22:53:08.291225+0000","last_peered":"2024-09-20T22:53:08.291225+0000","last_clean":"2024-09-20T22:53:08.291225+0000","last_became_active":"2024-09-20T22:47:49.213292+0000","last_became_peered":"2024-09-20T22:47:49.213292+0000","last_unstale":"2024-09-20T22:53:08.291225+0000","last_undegraded":"2024-09-20T22:53:08.291225+0000","last_fullsized":"2024-09-20T22:53:08.291225+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:58.129502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:51:58.129502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:11.276401+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"625'12","reported_seq":1354,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.296627+0000","last_change":"2024-09-20T22:52:14.322926+0000","last_active":"2024-09-20T22:53:08.296627+0000","last_peered":"2024-09-20T22:53:08.296627+0000","last_clean":"2024-09-20T22:53:08.296627+0000","last_became_active":"2024-09-20T22:43:09.116887+0000","last_became_peered":"2024-09-20T22:43:09.116887+0000","last_unstale":"2024-09-20T22:53:08.296627+0000","last_undegraded":"2024-09-20T22:53:08.296627+0000","last_fullsized":"2024-09-20T22:53:08.296627+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'12","last_scrub_stamp":"2024-09-20T22:52:14.322811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:52:14.322811+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:23.704685+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":1333,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.286645+0000","last_change":"2024-09-20T22:53:03.888115+0000","last_active":"2024-09-20T22:53:08.286645+0000","last_peered":"2024-09-20T22:53:08.286645+0000","last_clean":"2024-09-20T22:53:08.286645+0000","last_became_active":"2024-09-20T22:51:20.293232+0000","last_became_peered":"2024-09-20T22:51:20.293232+0000","last_unstale":"2024-09-20T22:53:08.286645+0000","last_undegraded":"2024-09-20T22:53:08.286645+0000","last_fullsized":"2024-09-20T22:53:08.286645+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:03.888040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:09.705632+0000","last_clean_scrub_stamp":"2024-09-20T22:53:03.888040+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:19.352989+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"74'1","reported_seq":1342,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.290335+0000","last_change":"2024-09-20T22:52:49.575819+0000","last_active":"2024-09-20T22:53:08.290335+0000","last_peered":"2024-09-20T22:53:08.290335+0000","last_clean":"2024-09-20T22:53:08.290335+0000","last_became_active":"2024-09-20T22:48:05.237882+0000","last_became_peered":"2024-09-20T22:48:05.237882+0000","last_unstale":"2024-09-20T22:53:08.290335+0000","last_undegraded":"2024-09-20T22:53:08.290335+0000","last_fullsized":"2024-09-20T22:53:08.290335+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-09-20T22:52:49.575702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:52:49.575702+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:06.855774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"625'15","reported_seq":686,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.289171+0000","last_change":"2024-09-20T22:51:52.816826+0000","last_active":"2024-09-20T22:53:08.289171+0000","last_peered":"2024-09-20T22:53:08.289171+0000","last_clean":"2024-09-20T22:53:08.289171+0000","last_became_active":"2024-09-20T22:47:49.231315+0000","last_became_peered":"2024-09-20T22:47:49.231315+0000","last_unstale":"2024-09-20T22:53:08.289171+0000","last_undegraded":"2024-09-20T22:53:08.289171+0000","last_fullsized":"2024-09-20T22:53:08.289171+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'15","last_scrub_stamp":"2024-09-20T22:51:52.816714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:51:52.816714+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":1349,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.289692+0000","last_change":"2024-09-20T22:51:12.541531+0000","last_active":"2024-09-20T22:53:08.289692+0000","last_peered":"2024-09-20T22:53:08.289692+0000","last_clean":"2024-09-20T22:53:08.289692+0000","last_became_active":"2024-09-20T22:48:05.238801+0000","last_became_peered":"2024-09-20T22:48:05.238801+0000","last_unstale":"2024-09-20T22:53:08.289692+0000","last_undegraded":"2024-09-20T22:53:08.289692+0000","last_fullsized":"2024-09-20T22:53:08.289692+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:12.541457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:51:12.541457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":1324,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.296382+0000","last_change":"2024-09-20T22:52:20.410227+0000","last_active":"2024-09-20T22:53:08.296382+0000","last_peered":"2024-09-20T22:53:08.296382+0000","last_clean":"2024-09-20T22:53:08.296382+0000","last_became_active":"2024-09-20T22:51:20.279397+0000","last_became_peered":"2024-09-20T22:51:20.279397+0000","last_unstale":"2024-09-20T22:53:08.296382+0000","last_undegraded":"2024-09-20T22:53:08.296382+0000","last_fullsized":"2024-09-20T22:53:08.296382+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:20.410164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:52:20.410164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:50.341745+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":1354,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.287296+0000","last_change":"2024-09-20T22:52:19.253411+0000","last_active":"2024-09-20T22:53:08.287296+0000","last_peered":"2024-09-20T22:53:08.287296+0000","last_clean":"2024-09-20T22:53:08.287296+0000","last_became_active":"2024-09-20T22:48:05.239710+0000","last_became_peered":"2024-09-20T22:48:05.239710+0000","last_unstale":"2024-09-20T22:53:08.287296+0000","last_undegraded":"2024-09-20T22:53:08.287296+0000","last_fullsized":"2024-09-20T22:53:08.287296+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:19.253343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:52:19.253343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:44.397693+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"625'15","reported_seq":1348,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.286301+0000","last_change":"2024-09-20T22:51:05.675215+0000","last_active":"2024-09-20T22:53:08.286301+0000","last_peered":"2024-09-20T22:53:08.286301+0000","last_clean":"2024-09-20T22:53:08.286301+0000","last_became_active":"2024-09-20T22:41:40.252486+0000","last_became_peered":"2024-09-20T22:41:40.252486+0000","last_unstale":"2024-09-20T22:53:08.286301+0000","last_undegraded":"2024-09-20T22:53:08.286301+0000","last_fullsized":"2024-09-20T22:53:08.286301+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-09-20T22:51:05.675095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:51:05.675095+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":675,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.287907+0000","last_change":"2024-09-20T22:52:13.735451+0000","last_active":"2024-09-20T22:53:08.287907+0000","last_peered":"2024-09-20T22:53:08.287907+0000","last_clean":"2024-09-20T22:53:08.287907+0000","last_became_active":"2024-09-20T22:47:49.229936+0000","last_became_peered":"2024-09-20T22:47:49.229936+0000","last_unstale":"2024-09-20T22:53:08.287907+0000","last_undegraded":"2024-09-20T22:53:08.287907+0000","last_fullsized":"2024-09-20T22:53:08.287907+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:13.735382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:52:13.735382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:22.644334+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"74'8","reported_seq":1460,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.296145+0000","last_change":"2024-09-20T22:50:38.269118+0000","last_active":"2024-09-20T22:53:08.296145+0000","last_peered":"2024-09-20T22:53:08.296145+0000","last_clean":"2024-09-20T22:53:08.296145+0000","last_became_active":"2024-09-20T22:41:42.186352+0000","last_became_peered":"2024-09-20T22:41:42.186352+0000","last_unstale":"2024-09-20T22:53:08.296145+0000","last_undegraded":"2024-09-20T22:53:08.296145+0000","last_fullsized":"2024-09-20T22:53:08.296145+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-09-20T22:50:38.269010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:50:38.269010+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":634,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.293130+0000","last_change":"2024-09-20T22:52:16.623463+0000","last_active":"2024-09-20T22:53:08.293130+0000","last_peered":"2024-09-20T22:53:08.293130+0000","last_clean":"2024-09-20T22:53:08.293130+0000","last_became_active":"2024-09-20T22:48:05.249656+0000","last_became_peered":"2024-09-20T22:48:05.249656+0000","last_unstale":"2024-09-20T22:53:08.293130+0000","last_undegraded":"2024-09-20T22:53:08.293130+0000","last_fullsized":"2024-09-20T22:53:08.293130+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:16.623368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:52:16.623368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:22.519570+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"625'20","reported_seq":1370,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.295767+0000","last_change":"2024-09-20T22:53:02.491402+0000","last_active":"2024-09-20T22:53:08.295767+0000","last_peered":"2024-09-20T22:53:08.295767+0000","last_clean":"2024-09-20T22:53:08.295767+0000","last_became_active":"2024-09-20T22:51:20.281859+0000","last_became_peered":"2024-09-20T22:51:20.281859+0000","last_unstale":"2024-09-20T22:53:08.295767+0000","last_undegraded":"2024-09-20T22:53:08.295767+0000","last_fullsized":"2024-09-20T22:53:08.295767+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'20","last_scrub_stamp":"2024-09-20T22:53:02.491320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:53:02.491320+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:10.839410+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":1345,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.290142+0000","last_change":"2024-09-20T22:52:20.621799+0000","last_active":"2024-09-20T22:53:08.290142+0000","last_peered":"2024-09-20T22:53:08.290142+0000","last_clean":"2024-09-20T22:53:08.290142+0000","last_became_active":"2024-09-20T22:48:05.240231+0000","last_became_peered":"2024-09-20T22:48:05.240231+0000","last_unstale":"2024-09-20T22:53:08.290142+0000","last_undegraded":"2024-09-20T22:53:08.290142+0000","last_fullsized":"2024-09-20T22:53:08.290142+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:20.621718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:52:20.621718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:30.275424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"600'11","reported_seq":1508,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.286174+0000","last_change":"2024-09-20T22:52:36.264515+0000","last_active":"2024-09-20T22:53:08.286174+0000","last_peered":"2024-09-20T22:53:08.286174+0000","last_clean":"2024-09-20T22:53:08.286174+0000","last_became_active":"2024-09-20T22:51:20.283393+0000","last_became_peered":"2024-09-20T22:51:20.283393+0000","last_unstale":"2024-09-20T22:53:08.286174+0000","last_undegraded":"2024-09-20T22:53:08.286174+0000","last_fullsized":"2024-09-20T22:53:08.286174+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"600'11","last_scrub_stamp":"2024-09-20T22:52:36.264410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:52:36.264410+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:52.246125+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":667,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.288688+0000","last_change":"2024-09-20T22:52:44.834418+0000","last_active":"2024-09-20T22:53:08.288688+0000","last_peered":"2024-09-20T22:53:08.288688+0000","last_clean":"2024-09-20T22:53:08.288688+0000","last_became_active":"2024-09-20T22:47:49.230674+0000","last_became_peered":"2024-09-20T22:47:49.230674+0000","last_unstale":"2024-09-20T22:53:08.288688+0000","last_undegraded":"2024-09-20T22:53:08.288688+0000","last_fullsized":"2024-09-20T22:53:08.288688+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:44.834352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:50:21.853070+0000","last_clean_scrub_stamp":"2024-09-20T22:52:44.834352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:57.888226+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"625'19","reported_seq":676,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.290133+0000","last_change":"2024-09-20T22:51:20.317969+0000","last_active":"2024-09-20T22:53:08.290133+0000","last_peered":"2024-09-20T22:53:08.290133+0000","last_clean":"2024-09-20T22:53:08.290133+0000","last_became_active":"2024-09-20T22:51:20.289584+0000","last_became_peered":"2024-09-20T22:51:20.289584+0000","last_unstale":"2024-09-20T22:53:08.290133+0000","last_undegraded":"2024-09-20T22:53:08.290133+0000","last_fullsized":"2024-09-20T22:53:08.290133+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-09-20T22:51:07.926382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:51:07.926382+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":642,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.292423+0000","last_change":"2024-09-20T22:52:47.717598+0000","last_active":"2024-09-20T22:53:08.292423+0000","last_peered":"2024-09-20T22:53:08.292423+0000","last_clean":"2024-09-20T22:53:08.292423+0000","last_became_active":"2024-09-20T22:48:05.252026+0000","last_became_peered":"2024-09-20T22:48:05.252026+0000","last_unstale":"2024-09-20T22:53:08.292423+0000","last_undegraded":"2024-09-20T22:53:08.292423+0000","last_fullsized":"2024-09-20T22:53:08.292423+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:47.717490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:52:47.717490+0000","last_clean_scrub_stamp":"2024-09-20T22:52:47.717490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:59.413281+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":228,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.285734+0000","last_change":"2024-09-20T22:51:20.291857+0000","last_active":"2024-09-20T22:53:08.285734+0000","last_peered":"2024-09-20T22:53:08.285734+0000","last_clean":"2024-09-20T22:53:08.285734+0000","last_became_active":"2024-09-20T22:51:20.291264+0000","last_became_peered":"2024-09-20T22:51:20.291264+0000","last_unstale":"2024-09-20T22:53:08.285734+0000","last_undegraded":"2024-09-20T22:53:08.285734+0000","last_fullsized":"2024-09-20T22:53:08.285734+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:06.651167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:51:06.651167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":1326,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.296192+0000","last_change":"2024-09-20T22:52:42.582474+0000","last_active":"2024-09-20T22:53:08.296192+0000","last_peered":"2024-09-20T22:53:08.296192+0000","last_clean":"2024-09-20T22:53:08.296192+0000","last_became_active":"2024-09-20T22:47:49.216141+0000","last_became_peered":"2024-09-20T22:47:49.216141+0000","last_unstale":"2024-09-20T22:53:08.296192+0000","last_undegraded":"2024-09-20T22:53:08.296192+0000","last_fullsized":"2024-09-20T22:53:08.296192+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:42.582398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:50:20.206984+0000","last_clean_scrub_stamp":"2024-09-20T22:52:42.582398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:57.745331+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"625'19","reported_seq":980,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.291624+0000","last_change":"2024-09-20T22:52:40.792541+0000","last_active":"2024-09-20T22:53:08.291624+0000","last_peered":"2024-09-20T22:53:08.291624+0000","last_clean":"2024-09-20T22:53:08.291624+0000","last_became_active":"2024-09-20T22:51:20.283868+0000","last_became_peered":"2024-09-20T22:51:20.283868+0000","last_unstale":"2024-09-20T22:53:08.291624+0000","last_undegraded":"2024-09-20T22:53:08.291624+0000","last_fullsized":"2024-09-20T22:53:08.291624+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'19","last_scrub_stamp":"2024-09-20T22:52:40.792457+0000","last_deep_scrub":"404'13","last_deep_scrub_stamp":"2024-09-20T22:48:04.276282+0000","last_clean_scrub_stamp":"2024-09-20T22:52:40.792457+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:55.252741+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,0],"acting":[6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":1360,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.290585+0000","last_change":"2024-09-20T22:52:42.505280+0000","last_active":"2024-09-20T22:53:08.290585+0000","last_peered":"2024-09-20T22:53:08.290585+0000","last_clean":"2024-09-20T22:53:08.290585+0000","last_became_active":"2024-09-20T22:48:05.238812+0000","last_became_peered":"2024-09-20T22:48:05.238812+0000","last_unstale":"2024-09-20T22:53:08.290585+0000","last_undegraded":"2024-09-20T22:53:08.290585+0000","last_fullsized":"2024-09-20T22:53:08.290585+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:42.505211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:31.138067+0000","last_clean_scrub_stamp":"2024-09-20T22:52:42.505211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:44.598851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":1328,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.298172+0000","last_change":"2024-09-20T22:51:20.288308+0000","last_active":"2024-09-20T22:53:08.298172+0000","last_peered":"2024-09-20T22:53:08.298172+0000","last_clean":"2024-09-20T22:53:08.298172+0000","last_became_active":"2024-09-20T22:51:20.288000+0000","last_became_peered":"2024-09-20T22:51:20.288000+0000","last_unstale":"2024-09-20T22:53:08.298172+0000","last_undegraded":"2024-09-20T22:53:08.298172+0000","last_fullsized":"2024-09-20T22:53:08.298172+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:50:44.313182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:50:44.313182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":665,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.286727+0000","last_change":"2024-09-20T22:51:20.278471+0000","last_active":"2024-09-20T22:53:08.286727+0000","last_peered":"2024-09-20T22:53:08.286727+0000","last_clean":"2024-09-20T22:53:08.286727+0000","last_became_active":"2024-09-20T22:51:20.278147+0000","last_became_peered":"2024-09-20T22:51:20.278147+0000","last_unstale":"2024-09-20T22:53:08.286727+0000","last_undegraded":"2024-09-20T22:53:08.286727+0000","last_fullsized":"2024-09-20T22:53:08.286727+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:50:35.895483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:50:35.895483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"625'24","reported_seq":1149,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.285379+0000","last_change":"2024-09-20T22:50:39.665781+0000","last_active":"2024-09-20T22:53:08.285379+0000","last_peered":"2024-09-20T22:53:08.285379+0000","last_clean":"2024-09-20T22:53:08.285379+0000","last_became_active":"2024-09-20T22:44:09.170785+0000","last_became_peered":"2024-09-20T22:44:09.170785+0000","last_unstale":"2024-09-20T22:53:08.285379+0000","last_undegraded":"2024-09-20T22:53:08.285379+0000","last_fullsized":"2024-09-20T22:53:08.285379+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"470'16","last_scrub_stamp":"2024-09-20T22:50:39.665682+0000","last_deep_scrub":"470'16","last_deep_scrub_stamp":"2024-09-20T22:49:10.659868+0000","last_clean_scrub_stamp":"2024-09-20T22:50:39.665682+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":1341,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.286544+0000","last_change":"2024-09-20T22:53:07.945728+0000","last_active":"2024-09-20T22:53:08.286544+0000","last_peered":"2024-09-20T22:53:08.286544+0000","last_clean":"2024-09-20T22:53:08.286544+0000","last_became_active":"2024-09-20T22:51:20.293645+0000","last_became_peered":"2024-09-20T22:51:20.293645+0000","last_unstale":"2024-09-20T22:53:08.286544+0000","last_undegraded":"2024-09-20T22:53:08.286544+0000","last_fullsized":"2024-09-20T22:53:08.286544+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:07.945664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:45:24.765414+0000","last_clean_scrub_stamp":"2024-09-20T22:53:07.945664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:21.914259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":1341,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.289376+0000","last_change":"2024-09-20T22:51:10.610718+0000","last_active":"2024-09-20T22:53:08.289376+0000","last_peered":"2024-09-20T22:53:08.289376+0000","last_clean":"2024-09-20T22:53:08.289376+0000","last_became_active":"2024-09-20T22:48:05.238334+0000","last_became_peered":"2024-09-20T22:48:05.238334+0000","last_unstale":"2024-09-20T22:53:08.289376+0000","last_undegraded":"2024-09-20T22:53:08.289376+0000","last_fullsized":"2024-09-20T22:53:08.289376+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:10.610588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:50.174861+0000","last_clean_scrub_stamp":"2024-09-20T22:51:10.610588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":1318,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.298329+0000","last_change":"2024-09-20T22:51:04.379662+0000","last_active":"2024-09-20T22:53:08.298329+0000","last_peered":"2024-09-20T22:53:08.298329+0000","last_clean":"2024-09-20T22:53:08.298329+0000","last_became_active":"2024-09-20T22:47:49.214487+0000","last_became_peered":"2024-09-20T22:47:49.214487+0000","last_unstale":"2024-09-20T22:53:08.298329+0000","last_undegraded":"2024-09-20T22:53:08.298329+0000","last_fullsized":"2024-09-20T22:53:08.298329+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:04.379558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:51:04.379558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"625'8","reported_seq":1346,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.295555+0000","last_change":"2024-09-20T22:53:05.489357+0000","last_active":"2024-09-20T22:53:08.295555+0000","last_peered":"2024-09-20T22:53:08.295555+0000","last_clean":"2024-09-20T22:53:08.295555+0000","last_became_active":"2024-09-20T22:51:20.290269+0000","last_became_peered":"2024-09-20T22:51:20.290269+0000","last_unstale":"2024-09-20T22:53:08.295555+0000","last_undegraded":"2024-09-20T22:53:08.295555+0000","last_fullsized":"2024-09-20T22:53:08.295555+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'8","last_scrub_stamp":"2024-09-20T22:53:05.489279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:53:05.489279+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:30.438834+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":675,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.289544+0000","last_change":"2024-09-20T22:53:07.918178+0000","last_active":"2024-09-20T22:53:08.289544+0000","last_peered":"2024-09-20T22:53:08.289544+0000","last_clean":"2024-09-20T22:53:08.289544+0000","last_became_active":"2024-09-20T22:47:49.231964+0000","last_became_peered":"2024-09-20T22:47:49.231964+0000","last_unstale":"2024-09-20T22:53:08.289544+0000","last_undegraded":"2024-09-20T22:53:08.289544+0000","last_fullsized":"2024-09-20T22:53:08.289544+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:07.918108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:53:07.918108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:20.150259+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":1358,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.290937+0000","last_change":"2024-09-20T22:52:20.955721+0000","last_active":"2024-09-20T22:53:08.290937+0000","last_peered":"2024-09-20T22:53:08.290937+0000","last_clean":"2024-09-20T22:53:08.290937+0000","last_became_active":"2024-09-20T22:48:05.244659+0000","last_became_peered":"2024-09-20T22:48:05.244659+0000","last_unstale":"2024-09-20T22:53:08.290937+0000","last_undegraded":"2024-09-20T22:53:08.290937+0000","last_fullsized":"2024-09-20T22:53:08.290937+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:20.955598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:52:20.955598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:32.677353+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":1341,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.289957+0000","last_change":"2024-09-20T22:53:05.612142+0000","last_active":"2024-09-20T22:53:08.289957+0000","last_peered":"2024-09-20T22:53:08.289957+0000","last_clean":"2024-09-20T22:53:08.289957+0000","last_became_active":"2024-09-20T22:48:05.237765+0000","last_became_peered":"2024-09-20T22:48:05.237765+0000","last_unstale":"2024-09-20T22:53:08.289957+0000","last_undegraded":"2024-09-20T22:53:08.289957+0000","last_fullsized":"2024-09-20T22:53:08.289957+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:05.612030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:10.233487+0000","last_clean_scrub_stamp":"2024-09-20T22:53:05.612030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:09.643188+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":931,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.286151+0000","last_change":"2024-09-20T22:51:29.904430+0000","last_active":"2024-09-20T22:53:08.286151+0000","last_peered":"2024-09-20T22:53:08.286151+0000","last_clean":"2024-09-20T22:53:08.286151+0000","last_became_active":"2024-09-20T22:51:20.297739+0000","last_became_peered":"2024-09-20T22:51:20.297739+0000","last_unstale":"2024-09-20T22:53:08.286151+0000","last_undegraded":"2024-09-20T22:53:08.286151+0000","last_fullsized":"2024-09-20T22:53:08.286151+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:29.904355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:51:29.904355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":1350,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.295322+0000","last_change":"2024-09-20T22:51:20.275893+0000","last_active":"2024-09-20T22:53:08.295322+0000","last_peered":"2024-09-20T22:53:08.295322+0000","last_clean":"2024-09-20T22:53:08.295322+0000","last_became_active":"2024-09-20T22:51:20.274891+0000","last_became_peered":"2024-09-20T22:51:20.274891+0000","last_unstale":"2024-09-20T22:53:08.295322+0000","last_undegraded":"2024-09-20T22:53:08.295322+0000","last_fullsized":"2024-09-20T22:53:08.295322+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:50:41.281470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:50:41.281470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"625'19","reported_seq":1391,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.293149+0000","last_change":"2024-09-20T22:52:16.953034+0000","last_active":"2024-09-20T22:53:08.293149+0000","last_peered":"2024-09-20T22:53:08.293149+0000","last_clean":"2024-09-20T22:53:08.293149+0000","last_became_active":"2024-09-20T22:51:20.284937+0000","last_became_peered":"2024-09-20T22:51:20.284937+0000","last_unstale":"2024-09-20T22:53:08.293149+0000","last_undegraded":"2024-09-20T22:53:08.293149+0000","last_fullsized":"2024-09-20T22:53:08.293149+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'19","last_scrub_stamp":"2024-09-20T22:52:16.952955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:52:16.952955+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:27.426198+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":667,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.288784+0000","last_change":"2024-09-20T22:52:18.698154+0000","last_active":"2024-09-20T22:53:08.288784+0000","last_peered":"2024-09-20T22:53:08.288784+0000","last_clean":"2024-09-20T22:53:08.288784+0000","last_became_active":"2024-09-20T22:47:49.220971+0000","last_became_peered":"2024-09-20T22:47:49.220971+0000","last_unstale":"2024-09-20T22:53:08.288784+0000","last_undegraded":"2024-09-20T22:53:08.288784+0000","last_fullsized":"2024-09-20T22:53:08.288784+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:18.698087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:51.667341+0000","last_clean_scrub_stamp":"2024-09-20T22:52:18.698087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:47.971098+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":696539985,"num_objects":314,"num_object_clones":265,"num_object_copies":942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":314,"num_whiteouts":13,"num_read":4873,"num_read_kb":5064896,"num_write":3558,"num_write_kb":1314105,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":790,"num_bytes_recovered":519843158,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1476657152,"data_stored":2199379794,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9967,"ondisk_log_size":9967,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85,"ondisk_log_size":85,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1109,"num_read_kb":866,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":312,"num_bytes_recovered":7464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":499712,"data_stored":13224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":744,"ondisk_log_size":744,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":1034,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":4931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":680,"num_read_kb":675,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1186,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":384,"ondisk_log_size":384,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070606,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":315080,"kb_used_data":265836,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":93430072,"statfs":{"total":95995035648,"available":95672393728,"internally_reserved":0,"allocated":272216064,"data_stored":367528106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":50395594},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430356,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":244992,"kb_used_data":195752,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":93500160,"statfs":{"total":95995035648,"available":95744163840,"internally_reserved":0,"allocated":200450048,"data_stored":294351817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691993,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":204484,"kb_used_data":146928,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":93540668,"statfs":{"total":95995035648,"available":95785644032,"internally_reserved":0,"allocated":150454272,"data_stored":224054236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986334,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":136900,"kb_used_data":105712,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93608252,"statfs":{"total":95995035648,"available":95854850048,"internally_reserved":0,"allocated":108249088,"data_stored":163305840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182564,"num_pgs":76,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":372052,"kb_used_data":318652,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93373100,"statfs":{"total":95995035648,"available":95614054400,"internally_reserved":0,"allocated":326299648,"data_stored":480390430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411497,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":307232,"kb_used_data":262156,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":93437920,"statfs":{"total":95995035648,"available":95680430080,"internally_reserved":0,"allocated":268447744,"data_stored":392183892,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":55834575022,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738547,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":242180,"kb_used_data":192944,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":93502972,"statfs":{"total":95995035648,"available":95747043328,"internally_reserved":0,"allocated":197574656,"data_stored":306847614,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":191102976,"data_stored":302918559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":261742592,"data_stored":387992814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319221760,"data_stored":475862726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":101777408,"data_stored":159370159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":143409152,"data_stored":219534274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":193974272,"data_stored":290422381,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":265428992,"data_stored":363278881,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-20T22:53:10.513 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph pg dump --format=json 2024-09-20T22:53:10.535 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:10.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:10.536 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:53:10.551 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:10 smithi154 ceph-mon[33264]: 3.19 scrub starts 2024-09-20T22:53:10.551 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:10 smithi154 ceph-mon[33264]: 3.19 scrub ok 2024-09-20T22:53:10.551 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:10 smithi154 ceph-mon[33264]: 4.1c scrub starts 2024-09-20T22:53:10.551 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:10 smithi154 ceph-mon[33264]: 4.1c scrub ok 2024-09-20T22:53:10.551 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:10 smithi154 ceph-mon[33264]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:53:10.551 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:10 smithi154 ceph-mon[33264]: 4.1b scrub starts 2024-09-20T22:53:10.551 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:10 smithi154 ceph-mon[33264]: 4.1b scrub ok 2024-09-20T22:53:10.551 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:10 smithi154 ceph-mon[33264]: from='client.26666 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T22:53:10.551 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:10 smithi154 ceph-mon[33264]: osdmap e710: 8 total, 8 up, 7 in 2024-09-20T22:53:10.593 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:53:10.637 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:53:10.652 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-20T22:53:10.652 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [], 2024-09-20T22:53:10.652 INFO:teuthology.orchestra.run.smithi154.stdout: "num_ops": 0 2024-09-20T22:53:10.652 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-20T22:53:10.660 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:10.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:10.661 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:53:10.715 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:10 smithi084 ceph-mon[40024]: 3.19 scrub starts 2024-09-20T22:53:10.715 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:10 smithi084 ceph-mon[40024]: 3.19 scrub ok 2024-09-20T22:53:10.715 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:10 smithi084 ceph-mon[40024]: 4.1c scrub starts 2024-09-20T22:53:10.715 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:10 smithi084 ceph-mon[40024]: 4.1c scrub ok 2024-09-20T22:53:10.715 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:10 smithi084 ceph-mon[40024]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:53:10.716 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:10 smithi084 ceph-mon[40024]: 4.1b scrub starts 2024-09-20T22:53:10.716 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:10 smithi084 ceph-mon[40024]: 4.1b scrub ok 2024-09-20T22:53:10.716 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:10 smithi084 ceph-mon[40024]: from='client.26666 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T22:53:10.716 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:10 smithi084 ceph-mon[40024]: osdmap e710: 8 total, 8 up, 7 in 2024-09-20T22:53:10.716 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:10 smithi084 ceph-mon[29774]: 3.19 scrub starts 2024-09-20T22:53:10.716 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:10 smithi084 ceph-mon[29774]: 3.19 scrub ok 2024-09-20T22:53:10.716 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:10 smithi084 ceph-mon[29774]: 4.1c scrub starts 2024-09-20T22:53:10.716 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:10 smithi084 ceph-mon[29774]: 4.1c scrub ok 2024-09-20T22:53:10.716 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:10 smithi084 ceph-mon[29774]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:53:10.716 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:10 smithi084 ceph-mon[29774]: 4.1b scrub starts 2024-09-20T22:53:10.716 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:10 smithi084 ceph-mon[29774]: 4.1b scrub ok 2024-09-20T22:53:10.716 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:10 smithi084 ceph-mon[29774]: from='client.26666 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T22:53:10.716 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:10 smithi084 ceph-mon[29774]: osdmap e710: 8 total, 8 up, 7 in 2024-09-20T22:53:10.761 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:53:10.782 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:10.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:10.783 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:53:10.884 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:53:10.905 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:10.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:10.906 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:53:11.007 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:53:11.028 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:11.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:11.029 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:53:11.130 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:53:11.151 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:11.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:11.152 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:53:11.253 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:53:11.260 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-20T22:53:11.261 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.6 dump_blocked_ops 2024-09-20T22:53:11.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:11.276 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:11.277 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:53:11.298 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:53:11.302 INFO:tasks.rados.rados.0.smithi084.stdout:2820: done (2 left) 2024-09-20T22:53:11.302 INFO:tasks.rados.rados.0.smithi084.stdout:2821: snap_create 2024-09-20T22:53:11.302 INFO:tasks.rados.rados.0.smithi084.stdout:2819: finishing write tid 1 to smithi08477203-21 2024-09-20T22:53:11.303 INFO:tasks.rados.rados.0.smithi084.stdout:2819: finishing write tid 2 to smithi08477203-21 2024-09-20T22:53:11.303 INFO:tasks.rados.rados.0.smithi084.stdout:2819: finishing write tid 3 to smithi08477203-21 2024-09-20T22:53:11.303 INFO:tasks.rados.rados.0.smithi084.stdout:2819: finishing write tid 4 to smithi08477203-21 2024-09-20T22:53:11.303 INFO:tasks.rados.rados.0.smithi084.stdout:2819: oid 21 updating version 0 to 1828 2024-09-20T22:53:11.303 INFO:tasks.rados.rados.0.smithi084.stdout:2819: oid 21 updating version 1828 to 1829 2024-09-20T22:53:11.303 INFO:tasks.rados.rados.0.smithi084.stdout:2819: oid 21 version 1829 is already newer than 1827 2024-09-20T22:53:11.303 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 21 v 1829 (ObjNum 984 snap 319 seq_num 984) dirty exists 2024-09-20T22:53:11.304 INFO:tasks.rados.rados.0.smithi084.stdout:2819: left oid 21 (ObjNum 984 snap 319 seq_num 984) 2024-09-20T22:53:11.377 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:53:11.398 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:11.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:11.399 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:53:11.500 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:53:11.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:11 smithi154 ceph-mon[33264]: 5.10 scrub starts 2024-09-20T22:53:11.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:11 smithi154 ceph-mon[33264]: 5.10 scrub ok 2024-09-20T22:53:11.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:11 smithi154 ceph-mon[33264]: pgmap v966: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 665 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail; 14 MiB/s rd, 2.9 MiB/s wr, 28 op/s 2024-09-20T22:53:11.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:11 smithi154 ceph-mon[33264]: 6.10 scrub starts 2024-09-20T22:53:11.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:11 smithi154 ceph-mon[33264]: 6.10 scrub ok 2024-09-20T22:53:11.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:11 smithi154 ceph-mon[33264]: 6.7 scrub starts 2024-09-20T22:53:11.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:11 smithi154 ceph-mon[33264]: 6.7 scrub ok 2024-09-20T22:53:11.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:11 smithi154 ceph-mon[33264]: 6.e scrub starts 2024-09-20T22:53:11.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:11 smithi154 ceph-mon[33264]: 6.e scrub ok 2024-09-20T22:53:11.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:11 smithi154 ceph-mon[33264]: osdmap e711: 8 total, 8 up, 7 in 2024-09-20T22:53:11.522 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:11.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:11.523 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:53:11.533 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:53:11.554 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:11 smithi084 ceph-mon[29774]: 5.10 scrub starts 2024-09-20T22:53:11.554 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:11 smithi084 ceph-mon[29774]: 5.10 scrub ok 2024-09-20T22:53:11.554 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:11 smithi084 ceph-mon[29774]: pgmap v966: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 665 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail; 14 MiB/s rd, 2.9 MiB/s wr, 28 op/s 2024-09-20T22:53:11.554 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:11 smithi084 ceph-mon[29774]: 6.10 scrub starts 2024-09-20T22:53:11.554 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:11 smithi084 ceph-mon[29774]: 6.10 scrub ok 2024-09-20T22:53:11.554 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:11 smithi084 ceph-mon[29774]: 6.7 scrub starts 2024-09-20T22:53:11.554 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:11 smithi084 ceph-mon[29774]: 6.7 scrub ok 2024-09-20T22:53:11.554 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:11 smithi084 ceph-mon[29774]: 6.e scrub starts 2024-09-20T22:53:11.554 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:11 smithi084 ceph-mon[29774]: 6.e scrub ok 2024-09-20T22:53:11.554 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:11 smithi084 ceph-mon[29774]: osdmap e711: 8 total, 8 up, 7 in 2024-09-20T22:53:11.555 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:11 smithi084 ceph-mon[40024]: 5.10 scrub starts 2024-09-20T22:53:11.555 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:11 smithi084 ceph-mon[40024]: 5.10 scrub ok 2024-09-20T22:53:11.555 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:11 smithi084 ceph-mon[40024]: pgmap v966: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 665 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail; 14 MiB/s rd, 2.9 MiB/s wr, 28 op/s 2024-09-20T22:53:11.555 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:11 smithi084 ceph-mon[40024]: 6.10 scrub starts 2024-09-20T22:53:11.555 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:11 smithi084 ceph-mon[40024]: 6.10 scrub ok 2024-09-20T22:53:11.555 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:11 smithi084 ceph-mon[40024]: 6.7 scrub starts 2024-09-20T22:53:11.555 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:11 smithi084 ceph-mon[40024]: 6.7 scrub ok 2024-09-20T22:53:11.555 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:11 smithi084 ceph-mon[40024]: 6.e scrub starts 2024-09-20T22:53:11.555 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:11 smithi084 ceph-mon[40024]: 6.e scrub ok 2024-09-20T22:53:11.555 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:11 smithi084 ceph-mon[40024]: osdmap e711: 8 total, 8 up, 7 in 2024-09-20T22:53:11.624 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:53:11.645 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:11.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:11.646 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:53:11.747 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:53:11.769 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:11.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:11.770 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:53:11.799 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:53:11.872 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:53:11.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:11.894 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:11.894 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:53:11.959 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-20T22:53:11.961 INFO:teuthology.orchestra.run.smithi084.stderr:dumped all 2024-09-20T22:53:11.995 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:53:12.017 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:12.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:12.018 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:53:12.058 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-20T22:53:12.059 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [], 2024-09-20T22:53:12.059 INFO:teuthology.orchestra.run.smithi154.stdout: "complaint_time": 30, 2024-09-20T22:53:12.059 INFO:teuthology.orchestra.run.smithi154.stdout: "num_blocked_ops": 0 2024-09-20T22:53:12.059 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-20T22:53:12.118 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:53:12.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:12.142 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:12.142 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:53:12.242 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:53:12.264 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:12.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:12.264 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:53:12.298 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is set 2024-09-20T22:53:12.303 INFO:tasks.rados.rados.0.smithi084.stdout:2818: done (2 left) 2024-09-20T22:53:12.303 INFO:tasks.rados.rados.0.smithi084.stdout:2819: done (1 left) 2024-09-20T22:53:12.303 INFO:tasks.rados.rados.0.smithi084.stdout:2821: done (0 left) 2024-09-20T22:53:12.303 INFO:tasks.rados.rados.0.smithi084.stdout:2822: snap_create 2024-09-20T22:53:12.366 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:53:12.390 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:12.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:12.391 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:53:12.491 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:53:12.513 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:12.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:12.514 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:53:12.614 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:53:12.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:12 smithi084 ceph-mon[29774]: 6.c scrub starts 2024-09-20T22:53:12.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:12 smithi084 ceph-mon[29774]: 6.c scrub ok 2024-09-20T22:53:12.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:12 smithi084 ceph-mon[29774]: 3.11 scrub starts 2024-09-20T22:53:12.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:12 smithi084 ceph-mon[29774]: 3.11 scrub ok 2024-09-20T22:53:12.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:12 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/792979969' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-20T22:53:12.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:12 smithi084 ceph-mon[29774]: 3.10 scrub starts 2024-09-20T22:53:12.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:12 smithi084 ceph-mon[29774]: 3.10 scrub ok 2024-09-20T22:53:12.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:12 smithi084 ceph-mon[29774]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:53:12.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:12 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/792979969' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T22:53:12.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:12 smithi084 ceph-mon[29774]: osdmap e712: 8 total, 8 up, 7 in 2024-09-20T22:53:12.617 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:12 smithi084 ceph-mon[40024]: 6.c scrub starts 2024-09-20T22:53:12.617 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:12 smithi084 ceph-mon[40024]: 6.c scrub ok 2024-09-20T22:53:12.617 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:12 smithi084 ceph-mon[40024]: 3.11 scrub starts 2024-09-20T22:53:12.617 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:12 smithi084 ceph-mon[40024]: 3.11 scrub ok 2024-09-20T22:53:12.617 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:12 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/792979969' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-20T22:53:12.617 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:12 smithi084 ceph-mon[40024]: 3.10 scrub starts 2024-09-20T22:53:12.617 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:12 smithi084 ceph-mon[40024]: 3.10 scrub ok 2024-09-20T22:53:12.617 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:12 smithi084 ceph-mon[40024]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:53:12.617 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:12 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/792979969' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T22:53:12.617 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:12 smithi084 ceph-mon[40024]: osdmap e712: 8 total, 8 up, 7 in 2024-09-20T22:53:12.637 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:12.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:12.638 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:53:12.671 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:12 smithi154 ceph-mon[33264]: 6.c scrub starts 2024-09-20T22:53:12.672 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:12 smithi154 ceph-mon[33264]: 6.c scrub ok 2024-09-20T22:53:12.672 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:12 smithi154 ceph-mon[33264]: 3.11 scrub starts 2024-09-20T22:53:12.672 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:12 smithi154 ceph-mon[33264]: 3.11 scrub ok 2024-09-20T22:53:12.672 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:12 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/792979969' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-20T22:53:12.672 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:12 smithi154 ceph-mon[33264]: 3.10 scrub starts 2024-09-20T22:53:12.672 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:12 smithi154 ceph-mon[33264]: 3.10 scrub ok 2024-09-20T22:53:12.672 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:12 smithi154 ceph-mon[33264]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:53:12.672 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:12 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/792979969' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T22:53:12.672 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:12 smithi154 ceph-mon[33264]: osdmap e712: 8 total, 8 up, 7 in 2024-09-20T22:53:12.698 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.6 dump_historic_ops 2024-09-20T22:53:12.738 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:53:12.759 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:12.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:12.760 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:53:12.789 INFO:teuthology.orchestra.run.smithi084.stdout:{"pg_ready":true,"pg_map":{"version":969,"stamp":"2024-09-20T22:53:11.797796+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":697136492,"num_objects":513,"num_object_clones":265,"num_object_copies":1539,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":513,"num_whiteouts":13,"num_read":6686,"num_read_kb":5066461,"num_write":4280,"num_write_kb":1314150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1142,"num_bytes_recovered":520180548,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":11199,"ondisk_log_size":11199,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":474,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":1822920,"kb_used_data":1487980,"kb_used_omap":10,"kb_used_meta":334773,"kb_avail":654393144,"statfs":{"total":671965249536,"available":670098579456,"internally_reserved":0,"allocated":1523691520,"data_stored":2228661935,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11157,"internal_metadata":342807659},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":15790086,"num_objects":13,"num_object_clones":13,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":3,"num_read":64,"num_read_kb":59035,"num_write":52,"num_write_kb":11748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000690"},"pg_stats":[{"pgid":"3.1f","version":"86'2","reported_seq":671,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.289682+0000","last_change":"2024-09-20T22:52:21.630419+0000","last_active":"2024-09-20T22:53:08.289682+0000","last_peered":"2024-09-20T22:53:08.289682+0000","last_clean":"2024-09-20T22:53:08.289682+0000","last_became_active":"2024-09-20T22:47:49.222114+0000","last_became_peered":"2024-09-20T22:47:49.222114+0000","last_unstale":"2024-09-20T22:53:08.289682+0000","last_undegraded":"2024-09-20T22:53:08.289682+0000","last_fullsized":"2024-09-20T22:53:08.289682+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'2","last_scrub_stamp":"2024-09-20T22:52:21.630304+0000","last_deep_scrub":"86'2","last_deep_scrub_stamp":"2024-09-20T22:42:39.664421+0000","last_clean_scrub_stamp":"2024-09-20T22:52:21.630304+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:38.958732+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":436,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"625'15","reported_seq":669,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.291859+0000","last_change":"2024-09-20T22:51:51.813358+0000","last_active":"2024-09-20T22:53:08.291859+0000","last_peered":"2024-09-20T22:53:08.291859+0000","last_clean":"2024-09-20T22:53:08.291859+0000","last_became_active":"2024-09-20T22:51:20.286651+0000","last_became_peered":"2024-09-20T22:51:20.286651+0000","last_unstale":"2024-09-20T22:53:08.291859+0000","last_undegraded":"2024-09-20T22:53:08.291859+0000","last_fullsized":"2024-09-20T22:53:08.291859+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'15","last_scrub_stamp":"2024-09-20T22:51:51.813269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:51:51.813269+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:09.354334+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":236,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.285555+0000","last_change":"2024-09-20T22:52:41.184388+0000","last_active":"2024-09-20T22:53:08.285555+0000","last_peered":"2024-09-20T22:53:08.285555+0000","last_clean":"2024-09-20T22:53:08.285555+0000","last_became_active":"2024-09-20T22:51:20.291272+0000","last_became_peered":"2024-09-20T22:51:20.291272+0000","last_unstale":"2024-09-20T22:53:08.285555+0000","last_undegraded":"2024-09-20T22:53:08.285555+0000","last_fullsized":"2024-09-20T22:53:08.285555+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:41.184318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:52:41.184318+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:04.632649+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":640,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.291519+0000","last_change":"2024-09-20T22:51:36.730087+0000","last_active":"2024-09-20T22:53:08.291519+0000","last_peered":"2024-09-20T22:53:08.291519+0000","last_clean":"2024-09-20T22:53:08.291519+0000","last_became_active":"2024-09-20T22:51:20.295608+0000","last_became_peered":"2024-09-20T22:51:20.295608+0000","last_unstale":"2024-09-20T22:53:08.291519+0000","last_undegraded":"2024-09-20T22:53:08.291519+0000","last_fullsized":"2024-09-20T22:53:08.291519+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:36.730015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:51:36.730015+0000","last_clean_scrub_stamp":"2024-09-20T22:51:36.730015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":1314,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.297938+0000","last_change":"2024-09-20T22:51:01.404127+0000","last_active":"2024-09-20T22:53:08.297938+0000","last_peered":"2024-09-20T22:53:08.297938+0000","last_clean":"2024-09-20T22:53:08.297938+0000","last_became_active":"2024-09-20T22:41:44.832099+0000","last_became_peered":"2024-09-20T22:41:44.832099+0000","last_unstale":"2024-09-20T22:53:08.297938+0000","last_undegraded":"2024-09-20T22:53:08.297938+0000","last_fullsized":"2024-09-20T22:53:08.297938+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:01.404063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:51:01.404063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":1319,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.296487+0000","last_change":"2024-09-20T22:51:12.418041+0000","last_active":"2024-09-20T22:53:08.296487+0000","last_peered":"2024-09-20T22:53:08.296487+0000","last_clean":"2024-09-20T22:53:08.296487+0000","last_became_active":"2024-09-20T22:41:38.157298+0000","last_became_peered":"2024-09-20T22:41:38.157298+0000","last_unstale":"2024-09-20T22:53:08.296487+0000","last_undegraded":"2024-09-20T22:53:08.296487+0000","last_fullsized":"2024-09-20T22:53:08.296487+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:12.417920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:51:12.417920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"625'27","reported_seq":1401,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.296684+0000","last_change":"2024-09-20T22:51:08.441927+0000","last_active":"2024-09-20T22:53:08.296684+0000","last_peered":"2024-09-20T22:53:08.296684+0000","last_clean":"2024-09-20T22:53:08.296684+0000","last_became_active":"2024-09-20T22:47:49.218798+0000","last_became_peered":"2024-09-20T22:47:49.218798+0000","last_unstale":"2024-09-20T22:53:08.296684+0000","last_undegraded":"2024-09-20T22:53:08.296684+0000","last_fullsized":"2024-09-20T22:53:08.296684+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-09-20T22:51:08.441788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:51:08.441788+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":640,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.291746+0000","last_change":"2024-09-20T22:52:14.594895+0000","last_active":"2024-09-20T22:53:08.291746+0000","last_peered":"2024-09-20T22:53:08.291746+0000","last_clean":"2024-09-20T22:53:08.291746+0000","last_became_active":"2024-09-20T22:51:20.287060+0000","last_became_peered":"2024-09-20T22:51:20.287060+0000","last_unstale":"2024-09-20T22:53:08.291746+0000","last_undegraded":"2024-09-20T22:53:08.291746+0000","last_fullsized":"2024-09-20T22:53:08.291746+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:14.594823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:52:14.594823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:30.113157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":1339,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.286369+0000","last_change":"2024-09-20T22:51:09.684125+0000","last_active":"2024-09-20T22:53:08.286369+0000","last_peered":"2024-09-20T22:53:08.286369+0000","last_clean":"2024-09-20T22:53:08.286369+0000","last_became_active":"2024-09-20T22:48:05.242059+0000","last_became_peered":"2024-09-20T22:48:05.242059+0000","last_unstale":"2024-09-20T22:53:08.286369+0000","last_undegraded":"2024-09-20T22:53:08.286369+0000","last_fullsized":"2024-09-20T22:53:08.286369+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:09.684053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:46:00.999939+0000","last_clean_scrub_stamp":"2024-09-20T22:51:09.684053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"625'15","reported_seq":653,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.291891+0000","last_change":"2024-09-20T22:51:30.710329+0000","last_active":"2024-09-20T22:53:08.291891+0000","last_peered":"2024-09-20T22:53:08.291891+0000","last_clean":"2024-09-20T22:53:08.291891+0000","last_became_active":"2024-09-20T22:48:05.233429+0000","last_became_peered":"2024-09-20T22:48:05.233429+0000","last_unstale":"2024-09-20T22:53:08.291891+0000","last_undegraded":"2024-09-20T22:53:08.291891+0000","last_fullsized":"2024-09-20T22:53:08.291891+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-09-20T22:51:30.710245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:51:30.710245+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":1331,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.286118+0000","last_change":"2024-09-20T22:51:33.977328+0000","last_active":"2024-09-20T22:53:08.286118+0000","last_peered":"2024-09-20T22:53:08.286118+0000","last_clean":"2024-09-20T22:53:08.286118+0000","last_became_active":"2024-09-20T22:47:49.227379+0000","last_became_peered":"2024-09-20T22:47:49.227379+0000","last_unstale":"2024-09-20T22:53:08.286118+0000","last_undegraded":"2024-09-20T22:53:08.286118+0000","last_fullsized":"2024-09-20T22:53:08.286118+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:33.977257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:51:33.977257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":673,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.286640+0000","last_change":"2024-09-20T22:51:35.899942+0000","last_active":"2024-09-20T22:53:08.286640+0000","last_peered":"2024-09-20T22:53:08.286640+0000","last_clean":"2024-09-20T22:53:08.286640+0000","last_became_active":"2024-09-20T22:51:20.277465+0000","last_became_peered":"2024-09-20T22:51:20.277465+0000","last_unstale":"2024-09-20T22:53:08.286640+0000","last_undegraded":"2024-09-20T22:53:08.286640+0000","last_fullsized":"2024-09-20T22:53:08.286640+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:35.899860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:49:08.664407+0000","last_clean_scrub_stamp":"2024-09-20T22:51:35.899860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":1345,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.286280+0000","last_change":"2024-09-20T22:51:20.298096+0000","last_active":"2024-09-20T22:53:08.286280+0000","last_peered":"2024-09-20T22:53:08.286280+0000","last_clean":"2024-09-20T22:53:08.286280+0000","last_became_active":"2024-09-20T22:51:20.297787+0000","last_became_peered":"2024-09-20T22:51:20.297787+0000","last_unstale":"2024-09-20T22:53:08.286280+0000","last_undegraded":"2024-09-20T22:53:08.286280+0000","last_fullsized":"2024-09-20T22:53:08.286280+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:50:17.735354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:50:17.735354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"625'9","reported_seq":655,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.290258+0000","last_change":"2024-09-20T22:51:20.301034+0000","last_active":"2024-09-20T22:53:08.290258+0000","last_peered":"2024-09-20T22:53:08.290258+0000","last_clean":"2024-09-20T22:53:08.290258+0000","last_became_active":"2024-09-20T22:51:20.285795+0000","last_became_peered":"2024-09-20T22:51:20.285795+0000","last_unstale":"2024-09-20T22:53:08.290258+0000","last_undegraded":"2024-09-20T22:53:08.290258+0000","last_fullsized":"2024-09-20T22:53:08.290258+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"181'7","last_scrub_stamp":"2024-09-20T22:51:05.899205+0000","last_deep_scrub":"181'7","last_deep_scrub_stamp":"2024-09-20T22:48:41.049381+0000","last_clean_scrub_stamp":"2024-09-20T22:51:05.899205+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":9,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":1358,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.290125+0000","last_change":"2024-09-20T22:52:16.705733+0000","last_active":"2024-09-20T22:53:08.290125+0000","last_peered":"2024-09-20T22:53:08.290125+0000","last_clean":"2024-09-20T22:53:08.290125+0000","last_became_active":"2024-09-20T22:51:20.285368+0000","last_became_peered":"2024-09-20T22:51:20.285368+0000","last_unstale":"2024-09-20T22:53:08.290125+0000","last_undegraded":"2024-09-20T22:53:08.290125+0000","last_fullsized":"2024-09-20T22:53:08.290125+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:16.705645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:47:40.361958+0000","last_clean_scrub_stamp":"2024-09-20T22:52:16.705645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:40.490856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":1324,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.285573+0000","last_change":"2024-09-20T22:51:20.279625+0000","last_active":"2024-09-20T22:53:08.285573+0000","last_peered":"2024-09-20T22:53:08.285573+0000","last_clean":"2024-09-20T22:53:08.285573+0000","last_became_active":"2024-09-20T22:51:20.279307+0000","last_became_peered":"2024-09-20T22:51:20.279307+0000","last_unstale":"2024-09-20T22:53:08.285573+0000","last_undegraded":"2024-09-20T22:53:08.285573+0000","last_fullsized":"2024-09-20T22:53:08.285573+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:04.636900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:51:04.636900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":234,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.292547+0000","last_change":"2024-09-20T22:52:39.724071+0000","last_active":"2024-09-20T22:53:08.292547+0000","last_peered":"2024-09-20T22:53:08.292547+0000","last_clean":"2024-09-20T22:53:08.292547+0000","last_became_active":"2024-09-20T22:51:20.282393+0000","last_became_peered":"2024-09-20T22:51:20.282393+0000","last_unstale":"2024-09-20T22:53:08.292547+0000","last_undegraded":"2024-09-20T22:53:08.292547+0000","last_fullsized":"2024-09-20T22:53:08.292547+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:39.723977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:52:39.723977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:47.638711+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":673,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.288228+0000","last_change":"2024-09-20T22:52:42.745105+0000","last_active":"2024-09-20T22:53:08.288228+0000","last_peered":"2024-09-20T22:53:08.288228+0000","last_clean":"2024-09-20T22:53:08.288228+0000","last_became_active":"2024-09-20T22:48:05.233687+0000","last_became_peered":"2024-09-20T22:48:05.233687+0000","last_unstale":"2024-09-20T22:53:08.288228+0000","last_undegraded":"2024-09-20T22:53:08.288228+0000","last_fullsized":"2024-09-20T22:53:08.288228+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:42.745032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:49:53.933601+0000","last_clean_scrub_stamp":"2024-09-20T22:52:42.745032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:59.532706+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"625'34","reported_seq":1410,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.287747+0000","last_change":"2024-09-20T22:51:20.289052+0000","last_active":"2024-09-20T22:53:08.287747+0000","last_peered":"2024-09-20T22:53:08.287747+0000","last_clean":"2024-09-20T22:53:08.287747+0000","last_became_active":"2024-09-20T22:51:20.276338+0000","last_became_peered":"2024-09-20T22:51:20.276338+0000","last_unstale":"2024-09-20T22:53:08.287747+0000","last_undegraded":"2024-09-20T22:53:08.287747+0000","last_fullsized":"2024-09-20T22:53:08.287747+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'22","last_scrub_stamp":"2024-09-20T22:51:08.280738+0000","last_deep_scrub":"113'22","last_deep_scrub_stamp":"2024-09-20T22:43:12.191878+0000","last_clean_scrub_stamp":"2024-09-20T22:51:08.280738+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":946,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.292198+0000","last_change":"2024-09-20T22:52:43.758236+0000","last_active":"2024-09-20T22:53:08.292198+0000","last_peered":"2024-09-20T22:53:08.292198+0000","last_clean":"2024-09-20T22:53:08.292198+0000","last_became_active":"2024-09-20T22:51:20.295564+0000","last_became_peered":"2024-09-20T22:51:20.295564+0000","last_unstale":"2024-09-20T22:53:08.292198+0000","last_undegraded":"2024-09-20T22:53:08.292198+0000","last_fullsized":"2024-09-20T22:53:08.292198+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:43.758162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:50:40.784154+0000","last_clean_scrub_stamp":"2024-09-20T22:52:43.758162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:10.526451+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":1307,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.296301+0000","last_change":"2024-09-20T22:51:13.385618+0000","last_active":"2024-09-20T22:53:08.296301+0000","last_peered":"2024-09-20T22:53:08.296301+0000","last_clean":"2024-09-20T22:53:08.296301+0000","last_became_active":"2024-09-20T22:41:44.832131+0000","last_became_peered":"2024-09-20T22:41:44.832131+0000","last_unstale":"2024-09-20T22:53:08.296301+0000","last_undegraded":"2024-09-20T22:53:08.296301+0000","last_fullsized":"2024-09-20T22:53:08.296301+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:13.385520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:45:25.002199+0000","last_clean_scrub_stamp":"2024-09-20T22:51:13.385520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":648,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.292061+0000","last_change":"2024-09-20T22:53:04.583753+0000","last_active":"2024-09-20T22:53:08.292061+0000","last_peered":"2024-09-20T22:53:08.292061+0000","last_clean":"2024-09-20T22:53:08.292061+0000","last_became_active":"2024-09-20T22:51:20.284621+0000","last_became_peered":"2024-09-20T22:51:20.284621+0000","last_unstale":"2024-09-20T22:53:08.292061+0000","last_undegraded":"2024-09-20T22:53:08.292061+0000","last_fullsized":"2024-09-20T22:53:08.292061+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:04.583683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:50:41.885844+0000","last_clean_scrub_stamp":"2024-09-20T22:53:04.583683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:27.888402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"625'18","reported_seq":1418,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.295670+0000","last_change":"2024-09-20T22:51:20.300828+0000","last_active":"2024-09-20T22:53:08.295670+0000","last_peered":"2024-09-20T22:53:08.295670+0000","last_clean":"2024-09-20T22:53:08.295670+0000","last_became_active":"2024-09-20T22:51:20.278886+0000","last_became_peered":"2024-09-20T22:51:20.278886+0000","last_unstale":"2024-09-20T22:53:08.295670+0000","last_undegraded":"2024-09-20T22:53:08.295670+0000","last_fullsized":"2024-09-20T22:53:08.295670+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-09-20T22:51:03.380934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:51:03.380934+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":642,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.292553+0000","last_change":"2024-09-20T22:52:45.738376+0000","last_active":"2024-09-20T22:53:08.292553+0000","last_peered":"2024-09-20T22:53:08.292553+0000","last_clean":"2024-09-20T22:53:08.292553+0000","last_became_active":"2024-09-20T22:48:05.250319+0000","last_became_peered":"2024-09-20T22:48:05.250319+0000","last_unstale":"2024-09-20T22:53:08.292553+0000","last_undegraded":"2024-09-20T22:53:08.292553+0000","last_fullsized":"2024-09-20T22:53:08.292553+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:45.738275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:52:45.738275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:02.329689+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"154'2","reported_seq":1323,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.290596+0000","last_change":"2024-09-20T22:52:45.578519+0000","last_active":"2024-09-20T22:53:08.290596+0000","last_peered":"2024-09-20T22:53:08.290596+0000","last_clean":"2024-09-20T22:53:08.290596+0000","last_became_active":"2024-09-20T22:41:44.199822+0000","last_became_peered":"2024-09-20T22:41:44.199822+0000","last_unstale":"2024-09-20T22:53:08.290596+0000","last_undegraded":"2024-09-20T22:53:08.290596+0000","last_fullsized":"2024-09-20T22:53:08.290596+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"154'2","last_scrub_stamp":"2024-09-20T22:52:45.578416+0000","last_deep_scrub":"154'2","last_deep_scrub_stamp":"2024-09-20T22:43:54.134066+0000","last_clean_scrub_stamp":"2024-09-20T22:52:45.578416+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:53.480152+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"67'1","reported_seq":247,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.612447+0000","last_change":"2024-09-20T22:53:08.612473+0000","last_active":"2024-09-20T22:53:08.612447+0000","last_peered":"2024-09-20T22:53:08.612447+0000","last_clean":"2024-09-20T22:53:08.612447+0000","last_became_active":"2024-09-20T22:51:20.279978+0000","last_became_peered":"2024-09-20T22:51:20.279978+0000","last_unstale":"2024-09-20T22:53:08.612447+0000","last_undegraded":"2024-09-20T22:53:08.612447+0000","last_fullsized":"2024-09-20T22:53:08.612447+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-09-20T22:53:08.612395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:53:08.612395+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:13.547026+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"625'25","reported_seq":694,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.289276+0000","last_change":"2024-09-20T22:51:02.707652+0000","last_active":"2024-09-20T22:53:08.289276+0000","last_peered":"2024-09-20T22:53:08.289276+0000","last_clean":"2024-09-20T22:53:08.289276+0000","last_became_active":"2024-09-20T22:47:49.232181+0000","last_became_peered":"2024-09-20T22:47:49.232181+0000","last_unstale":"2024-09-20T22:53:08.289276+0000","last_undegraded":"2024-09-20T22:53:08.289276+0000","last_fullsized":"2024-09-20T22:53:08.289276+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"151'15","last_scrub_stamp":"2024-09-20T22:51:02.707574+0000","last_deep_scrub":"151'15","last_deep_scrub_stamp":"2024-09-20T22:43:50.715761+0000","last_clean_scrub_stamp":"2024-09-20T22:51:02.707574+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"661'11","reported_seq":1509,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.293101+0000","last_change":"2024-09-20T22:52:21.908015+0000","last_active":"2024-09-20T22:53:08.293101+0000","last_peered":"2024-09-20T22:53:08.293101+0000","last_clean":"2024-09-20T22:53:08.293101+0000","last_became_active":"2024-09-20T22:48:05.244270+0000","last_became_peered":"2024-09-20T22:48:05.244270+0000","last_unstale":"2024-09-20T22:53:08.293101+0000","last_undegraded":"2024-09-20T22:53:08.293101+0000","last_fullsized":"2024-09-20T22:53:08.293101+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"661'11","last_scrub_stamp":"2024-09-20T22:52:21.907943+0000","last_deep_scrub":"661'11","last_deep_scrub_stamp":"2024-09-20T22:52:21.907943+0000","last_clean_scrub_stamp":"2024-09-20T22:52:21.907943+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:27.456801+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.c","version":"708'2198","reported_seq":5171,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.307978+0000","last_change":"2024-09-20T22:53:08.308187+0000","last_active":"2024-09-20T22:53:08.307978+0000","last_peered":"2024-09-20T22:53:08.307978+0000","last_clean":"2024-09-20T22:53:08.307978+0000","last_became_active":"2024-09-20T22:42:52.404027+0000","last_became_peered":"2024-09-20T22:42:52.404027+0000","last_unstale":"2024-09-20T22:53:08.307978+0000","last_undegraded":"2024-09-20T22:53:08.307978+0000","last_fullsized":"2024-09-20T22:53:08.307978+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"674'2030","last_scrub_stamp":"2024-09-20T22:52:36.892383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:52:36.892383+0000","objects_scrubbed":44,"log_size":2198,"log_dups_size":0,"ondisk_log_size":2198,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:01.737318+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.015309313,"stat_sum":{"num_bytes":98091786,"num_objects":44,"num_object_clones":37,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":3,"num_read":895,"num_read_kb":966419,"num_write":582,"num_write_kb":222502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"bf","length":"1"},{"start":"f8","length":"1"},{"start":"fb","length":"1"},{"start":"fd","length":"6"},{"start":"104","length":"8"},{"start":"10d","length":"5"},{"start":"113","length":"8"},{"start":"11c","length":"3"},{"start":"120","length":"1"},{"start":"123","length":"1"},{"start":"125","length":"2"},{"start":"129","length":"1"},{"start":"12b","length":"1"},{"start":"133","length":"2"}]},{"pgid":"1.a","version":"77'128","reported_seq":1587,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.285440+0000","last_change":"2024-09-20T22:50:19.676094+0000","last_active":"2024-09-20T22:53:08.285440+0000","last_peered":"2024-09-20T22:53:08.285440+0000","last_clean":"2024-09-20T22:53:08.285440+0000","last_became_active":"2024-09-20T22:42:52.407791+0000","last_became_peered":"2024-09-20T22:42:52.407791+0000","last_unstale":"2024-09-20T22:53:08.285440+0000","last_undegraded":"2024-09-20T22:53:08.285440+0000","last_fullsized":"2024-09-20T22:53:08.285440+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:50:19.675992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:50:19.675992+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"625'27","reported_seq":313,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.296462+0000","last_change":"2024-09-20T22:52:15.358218+0000","last_active":"2024-09-20T22:53:08.296462+0000","last_peered":"2024-09-20T22:53:08.296462+0000","last_clean":"2024-09-20T22:53:08.296462+0000","last_became_active":"2024-09-20T22:51:20.287083+0000","last_became_peered":"2024-09-20T22:51:20.287083+0000","last_unstale":"2024-09-20T22:53:08.296462+0000","last_undegraded":"2024-09-20T22:53:08.296462+0000","last_fullsized":"2024-09-20T22:53:08.296462+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'27","last_scrub_stamp":"2024-09-20T22:52:15.358100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:52:15.358100+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:27.239349+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":1325,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.295228+0000","last_change":"2024-09-20T22:51:29.394206+0000","last_active":"2024-09-20T22:53:08.295228+0000","last_peered":"2024-09-20T22:53:08.295228+0000","last_clean":"2024-09-20T22:53:08.295228+0000","last_became_active":"2024-09-20T22:51:20.279439+0000","last_became_peered":"2024-09-20T22:51:20.279439+0000","last_unstale":"2024-09-20T22:53:08.295228+0000","last_undegraded":"2024-09-20T22:53:08.295228+0000","last_fullsized":"2024-09-20T22:53:08.295228+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:29.394097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:49:12.070675+0000","last_clean_scrub_stamp":"2024-09-20T22:51:29.394097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"408'12","reported_seq":699,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.292685+0000","last_change":"2024-09-20T22:51:50.785575+0000","last_active":"2024-09-20T22:53:08.292685+0000","last_peered":"2024-09-20T22:53:08.292685+0000","last_clean":"2024-09-20T22:53:08.292685+0000","last_became_active":"2024-09-20T22:48:05.242914+0000","last_became_peered":"2024-09-20T22:48:05.242914+0000","last_unstale":"2024-09-20T22:53:08.292685+0000","last_undegraded":"2024-09-20T22:53:08.292685+0000","last_fullsized":"2024-09-20T22:53:08.292685+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"408'12","last_scrub_stamp":"2024-09-20T22:51:50.785478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:51:50.785478+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":1334,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.286054+0000","last_change":"2024-09-20T22:51:31.960415+0000","last_active":"2024-09-20T22:53:08.286054+0000","last_peered":"2024-09-20T22:53:08.286054+0000","last_clean":"2024-09-20T22:53:08.286054+0000","last_became_active":"2024-09-20T22:51:20.280145+0000","last_became_peered":"2024-09-20T22:51:20.280145+0000","last_unstale":"2024-09-20T22:53:08.286054+0000","last_undegraded":"2024-09-20T22:53:08.286054+0000","last_fullsized":"2024-09-20T22:53:08.286054+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:31.960331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:51:31.960331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3","version":"706'307","reported_seq":3216,"reported_epoch":708,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-20T22:53:08.365089+0000","last_change":"2024-09-20T22:53:08.365150+0000","last_active":"2024-09-20T22:53:08.365089+0000","last_peered":"2024-09-20T22:53:08.365089+0000","last_clean":"2024-09-20T22:53:08.365089+0000","last_became_active":"2024-09-20T22:49:05.221402+0000","last_became_peered":"2024-09-20T22:49:05.221402+0000","last_unstale":"2024-09-20T22:53:08.365089+0000","last_undegraded":"2024-09-20T22:53:08.365089+0000","last_fullsized":"2024-09-20T22:53:08.365089+0000","mapping_epoch":404,"log_start":"456'129","ondisk_log_start":"456'129","created":94,"last_epoch_clean":465,"parent":"0.0","parent_split_bits":0,"last_scrub":"609'243","last_scrub_stamp":"2024-09-20T22:51:31.402013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:51:31.402013+0000","objects_scrubbed":6,"log_size":178,"log_dups_size":149,"ondisk_log_size":178,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.067059514000000001,"stat_sum":{"num_bytes":19863277,"num_objects":7,"num_object_clones":6,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":100,"num_read_kb":128300,"num_write":67,"num_write_kb":22506,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":20196357,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"bf","length":"1"},{"start":"f8","length":"1"},{"start":"fb","length":"1"},{"start":"fd","length":"6"},{"start":"104","length":"8"},{"start":"10d","length":"5"},{"start":"113","length":"6"},{"start":"11a","length":"1"},{"start":"11c","length":"3"},{"start":"120","length":"1"},{"start":"123","length":"1"},{"start":"125","length":"1"},{"start":"129","length":"1"},{"start":"133","length":"2"}]},{"pgid":"1.5","version":"77'128","reported_seq":1493,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.293102+0000","last_change":"2024-09-20T22:51:12.927535+0000","last_active":"2024-09-20T22:53:08.293102+0000","last_peered":"2024-09-20T22:53:08.293102+0000","last_clean":"2024-09-20T22:53:08.293102+0000","last_became_active":"2024-09-20T22:48:08.207018+0000","last_became_peered":"2024-09-20T22:48:08.207018+0000","last_unstale":"2024-09-20T22:53:08.293102+0000","last_undegraded":"2024-09-20T22:53:08.293102+0000","last_fullsized":"2024-09-20T22:53:08.293102+0000","mapping_epoch":407,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":408,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:51:12.927463+0000","last_deep_scrub":"77'128","last_deep_scrub_stamp":"2024-09-20T22:45:41.608075+0000","last_clean_scrub_stamp":"2024-09-20T22:51:12.927463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"625'30","reported_seq":1413,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.295459+0000","last_change":"2024-09-20T22:50:45.270518+0000","last_active":"2024-09-20T22:53:08.295459+0000","last_peered":"2024-09-20T22:53:08.295459+0000","last_clean":"2024-09-20T22:53:08.295459+0000","last_became_active":"2024-09-20T22:47:49.217605+0000","last_became_peered":"2024-09-20T22:47:49.217605+0000","last_unstale":"2024-09-20T22:53:08.295459+0000","last_undegraded":"2024-09-20T22:53:08.295459+0000","last_fullsized":"2024-09-20T22:53:08.295459+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'18","last_scrub_stamp":"2024-09-20T22:50:45.270399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:50:45.270399+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":1339,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.296758+0000","last_change":"2024-09-20T22:52:45.537243+0000","last_active":"2024-09-20T22:53:08.296758+0000","last_peered":"2024-09-20T22:53:08.296758+0000","last_clean":"2024-09-20T22:53:08.296758+0000","last_became_active":"2024-09-20T22:47:49.227069+0000","last_became_peered":"2024-09-20T22:47:49.227069+0000","last_unstale":"2024-09-20T22:53:08.296758+0000","last_undegraded":"2024-09-20T22:53:08.296758+0000","last_fullsized":"2024-09-20T22:53:08.296758+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:45.537173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:52:45.537173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:54.445275+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":634,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.294186+0000","last_change":"2024-09-20T22:51:09.901939+0000","last_active":"2024-09-20T22:53:08.294186+0000","last_peered":"2024-09-20T22:53:08.294186+0000","last_clean":"2024-09-20T22:53:08.294186+0000","last_became_active":"2024-09-20T22:48:05.230003+0000","last_became_peered":"2024-09-20T22:48:05.230003+0000","last_unstale":"2024-09-20T22:53:08.294186+0000","last_undegraded":"2024-09-20T22:53:08.294186+0000","last_fullsized":"2024-09-20T22:53:08.294186+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:09.901838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:07.947945+0000","last_clean_scrub_stamp":"2024-09-20T22:51:09.901838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":634,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.291849+0000","last_change":"2024-09-20T22:51:55.748527+0000","last_active":"2024-09-20T22:53:08.291849+0000","last_peered":"2024-09-20T22:53:08.291849+0000","last_clean":"2024-09-20T22:53:08.291849+0000","last_became_active":"2024-09-20T22:48:05.239917+0000","last_became_peered":"2024-09-20T22:48:05.239917+0000","last_unstale":"2024-09-20T22:53:08.291849+0000","last_undegraded":"2024-09-20T22:53:08.291849+0000","last_fullsized":"2024-09-20T22:53:08.291849+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:55.748425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:53.036601+0000","last_clean_scrub_stamp":"2024-09-20T22:51:55.748425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:19.609464+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2","version":"708'747","reported_seq":2986,"reported_epoch":708,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T22:53:08.367301+0000","last_change":"2024-09-20T22:53:08.365196+0000","last_active":"2024-09-20T22:53:08.367301+0000","last_peered":"2024-09-20T22:53:08.367301+0000","last_clean":"2024-09-20T22:53:08.367301+0000","last_became_active":"2024-09-20T22:52:23.286461+0000","last_became_peered":"2024-09-20T22:52:23.286461+0000","last_unstale":"2024-09-20T22:53:08.367301+0000","last_undegraded":"2024-09-20T22:53:08.367301+0000","last_fullsized":"2024-09-20T22:53:08.367301+0000","mapping_epoch":662,"log_start":"471'464","ondisk_log_start":"471'464","created":94,"last_epoch_clean":663,"parent":"0.0","parent_split_bits":0,"last_scrub":"282'120","last_scrub_stamp":"2024-09-20T22:46:22.099642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:46:22.099642+0000","objects_scrubbed":1,"log_size":283,"log_dups_size":687,"ondisk_log_size":283,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":39,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.034743356000000003,"stat_sum":{"num_bytes":67843488,"num_objects":30,"num_object_clones":26,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":2,"num_read":396,"num_read_kb":412551,"num_write":256,"num_write_kb":102445,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":144,"num_bytes_recovered":106353268,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"bf","length":"1"},{"start":"f8","length":"1"}]},{"pgid":"1.4","version":"77'128","reported_seq":1619,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.296226+0000","last_change":"2024-09-20T22:52:02.474080+0000","last_active":"2024-09-20T22:53:08.296226+0000","last_peered":"2024-09-20T22:53:08.296226+0000","last_clean":"2024-09-20T22:53:08.296226+0000","last_became_active":"2024-09-20T22:46:15.181305+0000","last_became_peered":"2024-09-20T22:46:15.181305+0000","last_unstale":"2024-09-20T22:53:08.296226+0000","last_undegraded":"2024-09-20T22:53:08.296226+0000","last_fullsized":"2024-09-20T22:53:08.296226+0000","mapping_epoch":94,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:52:02.473979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:52:02.473979+0000","objects_scrubbed":1,"log_size":0,"log_dups_size":59,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"625'28","reported_seq":671,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.294095+0000","last_change":"2024-09-20T22:51:58.712602+0000","last_active":"2024-09-20T22:53:08.294095+0000","last_peered":"2024-09-20T22:53:08.294095+0000","last_clean":"2024-09-20T22:53:08.294095+0000","last_became_active":"2024-09-20T22:48:05.235812+0000","last_became_peered":"2024-09-20T22:48:05.235812+0000","last_unstale":"2024-09-20T22:53:08.294095+0000","last_undegraded":"2024-09-20T22:53:08.294095+0000","last_fullsized":"2024-09-20T22:53:08.294095+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'28","last_scrub_stamp":"2024-09-20T22:51:58.712488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:51:58.712488+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:18.921188+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":41,"num_read_kb":34,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"67'1","reported_seq":691,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.289122+0000","last_change":"2024-09-20T22:51:20.292468+0000","last_active":"2024-09-20T22:53:08.289122+0000","last_peered":"2024-09-20T22:53:08.289122+0000","last_clean":"2024-09-20T22:53:08.289122+0000","last_became_active":"2024-09-20T22:51:20.283011+0000","last_became_peered":"2024-09-20T22:51:20.283011+0000","last_unstale":"2024-09-20T22:53:08.289122+0000","last_undegraded":"2024-09-20T22:53:08.289122+0000","last_fullsized":"2024-09-20T22:53:08.289122+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-09-20T22:51:11.806011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:51:11.806011+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"600'13","reported_seq":1509,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.295805+0000","last_change":"2024-09-20T22:51:32.448600+0000","last_active":"2024-09-20T22:53:08.295805+0000","last_peered":"2024-09-20T22:53:08.295805+0000","last_clean":"2024-09-20T22:53:08.295805+0000","last_became_active":"2024-09-20T22:51:20.289873+0000","last_became_peered":"2024-09-20T22:51:20.289873+0000","last_unstale":"2024-09-20T22:53:08.295805+0000","last_undegraded":"2024-09-20T22:53:08.295805+0000","last_fullsized":"2024-09-20T22:53:08.295805+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"600'13","last_scrub_stamp":"2024-09-20T22:51:32.448512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:51:32.448512+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":1322,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.289842+0000","last_change":"2024-09-20T22:53:02.610452+0000","last_active":"2024-09-20T22:53:08.289842+0000","last_peered":"2024-09-20T22:53:08.289842+0000","last_clean":"2024-09-20T22:53:08.289842+0000","last_became_active":"2024-09-20T22:41:44.831901+0000","last_became_peered":"2024-09-20T22:41:44.831901+0000","last_unstale":"2024-09-20T22:53:08.289842+0000","last_undegraded":"2024-09-20T22:53:08.289842+0000","last_fullsized":"2024-09-20T22:53:08.289842+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:02.610359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:53:02.610359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:32.529803+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1","version":"708'409","reported_seq":3405,"reported_epoch":708,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T22:53:08.296838+0000","last_change":"2024-09-20T22:53:08.296838+0000","last_active":"2024-09-20T22:53:08.296838+0000","last_peered":"2024-09-20T22:53:08.296838+0000","last_clean":"2024-09-20T22:53:08.296838+0000","last_became_active":"2024-09-20T22:49:21.232750+0000","last_became_peered":"2024-09-20T22:49:21.232750+0000","last_unstale":"2024-09-20T22:53:08.296838+0000","last_undegraded":"2024-09-20T22:53:08.296838+0000","last_fullsized":"2024-09-20T22:53:08.296838+0000","mapping_epoch":407,"log_start":"474'259","ondisk_log_start":"474'259","created":94,"last_epoch_clean":481,"parent":"0.0","parent_split_bits":0,"last_scrub":"703'407","last_scrub_stamp":"2024-09-20T22:53:06.625437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:53:06.625437+0000","objects_scrubbed":6,"log_size":150,"log_dups_size":338,"ondisk_log_size":150,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:19.539132+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.96718284099999996,"stat_sum":{"num_bytes":6959411,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":93,"num_read_kb":91249,"num_write":88,"num_write_kb":45551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":8732217,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"bf","length":"1"},{"start":"f8","length":"1"},{"start":"fb","length":"1"},{"start":"fd","length":"6"},{"start":"104","length":"8"},{"start":"10d","length":"5"},{"start":"113","length":"8"},{"start":"11c","length":"3"},{"start":"120","length":"1"},{"start":"123","length":"1"},{"start":"125","length":"2"},{"start":"129","length":"1"},{"start":"133","length":"2"}]},{"pgid":"1.7","version":"77'128","reported_seq":1632,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.292050+0000","last_change":"2024-09-20T22:52:34.898681+0000","last_active":"2024-09-20T22:53:08.292050+0000","last_peered":"2024-09-20T22:53:08.292050+0000","last_clean":"2024-09-20T22:53:08.292050+0000","last_became_active":"2024-09-20T22:46:03.169596+0000","last_became_peered":"2024-09-20T22:46:03.169596+0000","last_unstale":"2024-09-20T22:53:08.292050+0000","last_undegraded":"2024-09-20T22:53:08.292050+0000","last_fullsized":"2024-09-20T22:53:08.292050+0000","mapping_epoch":94,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:52:34.898595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:52:34.898595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:51.834169+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"625'20","reported_seq":297,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.285758+0000","last_change":"2024-09-20T22:52:19.125548+0000","last_active":"2024-09-20T22:53:08.285758+0000","last_peered":"2024-09-20T22:53:08.285758+0000","last_clean":"2024-09-20T22:53:08.285758+0000","last_became_active":"2024-09-20T22:51:20.297933+0000","last_became_peered":"2024-09-20T22:51:20.297933+0000","last_unstale":"2024-09-20T22:53:08.285758+0000","last_undegraded":"2024-09-20T22:53:08.285758+0000","last_fullsized":"2024-09-20T22:53:08.285758+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'20","last_scrub_stamp":"2024-09-20T22:52:19.125462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:52:19.125462+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:45.473839+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":1336,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.286406+0000","last_change":"2024-09-20T22:52:46.062448+0000","last_active":"2024-09-20T22:53:08.286406+0000","last_peered":"2024-09-20T22:53:08.286406+0000","last_clean":"2024-09-20T22:53:08.286406+0000","last_became_active":"2024-09-20T22:41:38.155629+0000","last_became_peered":"2024-09-20T22:41:38.155629+0000","last_unstale":"2024-09-20T22:53:08.286406+0000","last_undegraded":"2024-09-20T22:53:08.286406+0000","last_fullsized":"2024-09-20T22:53:08.286406+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:46.062352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:52:46.062352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:15.728943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"389'11","reported_seq":736,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.288183+0000","last_change":"2024-09-20T22:51:26.968244+0000","last_active":"2024-09-20T22:53:08.288183+0000","last_peered":"2024-09-20T22:53:08.288183+0000","last_clean":"2024-09-20T22:53:08.288183+0000","last_became_active":"2024-09-20T22:47:49.227264+0000","last_became_peered":"2024-09-20T22:47:49.227264+0000","last_unstale":"2024-09-20T22:53:08.288183+0000","last_undegraded":"2024-09-20T22:53:08.288183+0000","last_fullsized":"2024-09-20T22:53:08.288183+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'11","last_scrub_stamp":"2024-09-20T22:51:26.968138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:51:26.968138+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":667,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.286491+0000","last_change":"2024-09-20T22:51:55.755550+0000","last_active":"2024-09-20T22:53:08.286491+0000","last_peered":"2024-09-20T22:53:08.286491+0000","last_clean":"2024-09-20T22:53:08.286491+0000","last_became_active":"2024-09-20T22:47:49.232380+0000","last_became_peered":"2024-09-20T22:47:49.232380+0000","last_unstale":"2024-09-20T22:53:08.286491+0000","last_undegraded":"2024-09-20T22:53:08.286491+0000","last_fullsized":"2024-09-20T22:53:08.286491+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:55.755432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:51:55.755432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.0","version":"706'604","reported_seq":3695,"reported_epoch":708,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T22:53:08.285514+0000","last_change":"2024-09-20T22:53:07.362433+0000","last_active":"2024-09-20T22:53:08.285514+0000","last_peered":"2024-09-20T22:53:08.285514+0000","last_clean":"2024-09-20T22:53:08.285514+0000","last_became_active":"2024-09-20T22:51:53.366096+0000","last_became_peered":"2024-09-20T22:51:53.366096+0000","last_unstale":"2024-09-20T22:53:08.285514+0000","last_undegraded":"2024-09-20T22:53:08.285514+0000","last_fullsized":"2024-09-20T22:53:08.285514+0000","mapping_epoch":632,"log_start":"480'253","ondisk_log_start":"480'253","created":94,"last_epoch_clean":633,"parent":"0.0","parent_split_bits":0,"last_scrub":"562'368","last_scrub_stamp":"2024-09-20T22:50:44.669910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:50:44.669910+0000","objects_scrubbed":15,"log_size":351,"log_dups_size":416,"ondisk_log_size":351,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":5,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.99780654700000004,"stat_sum":{"num_bytes":60418945,"num_objects":29,"num_object_clones":26,"num_object_copies":87,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":0,"num_read":231,"num_read_kb":233511,"num_write":189,"num_write_kb":68166,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":42943203,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"bf","length":"1"},{"start":"f8","length":"1"},{"start":"fb","length":"1"},{"start":"fd","length":"6"},{"start":"104","length":"8"},{"start":"10d","length":"1"},{"start":"10f","length":"3"},{"start":"113","length":"6"},{"start":"11a","length":"1"},{"start":"11c","length":"3"},{"start":"120","length":"1"},{"start":"123","length":"1"},{"start":"125","length":"1"},{"start":"129","length":"1"},{"start":"133","length":"1"}]},{"pgid":"1.6","version":"77'128","reported_seq":1654,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.292062+0000","last_change":"2024-09-20T22:52:38.843686+0000","last_active":"2024-09-20T22:53:08.292062+0000","last_peered":"2024-09-20T22:53:08.292062+0000","last_clean":"2024-09-20T22:53:08.292062+0000","last_became_active":"2024-09-20T22:47:52.212190+0000","last_became_peered":"2024-09-20T22:47:52.212190+0000","last_unstale":"2024-09-20T22:53:08.292062+0000","last_undegraded":"2024-09-20T22:53:08.292062+0000","last_fullsized":"2024-09-20T22:53:08.292062+0000","mapping_epoch":391,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:52:38.843586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:52:38.843586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:07.967280+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"625'37","reported_seq":708,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.286855+0000","last_change":"2024-09-20T22:52:39.703172+0000","last_active":"2024-09-20T22:53:08.286855+0000","last_peered":"2024-09-20T22:53:08.286855+0000","last_clean":"2024-09-20T22:53:08.286855+0000","last_became_active":"2024-09-20T22:47:49.222009+0000","last_became_peered":"2024-09-20T22:47:49.222009+0000","last_unstale":"2024-09-20T22:53:08.286855+0000","last_undegraded":"2024-09-20T22:53:08.286855+0000","last_fullsized":"2024-09-20T22:53:08.286855+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'37","last_scrub_stamp":"2024-09-20T22:52:39.703093+0000","last_deep_scrub":"515'27","last_deep_scrub_stamp":"2024-09-20T22:49:55.964153+0000","last_clean_scrub_stamp":"2024-09-20T22:52:39.703093+0000","objects_scrubbed":8,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:49.380387+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":53,"num_read_kb":39,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":236,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.287842+0000","last_change":"2024-09-20T22:52:14.225251+0000","last_active":"2024-09-20T22:53:08.287842+0000","last_peered":"2024-09-20T22:53:08.287842+0000","last_clean":"2024-09-20T22:53:08.287842+0000","last_became_active":"2024-09-20T22:51:20.286646+0000","last_became_peered":"2024-09-20T22:51:20.286646+0000","last_unstale":"2024-09-20T22:53:08.287842+0000","last_undegraded":"2024-09-20T22:53:08.287842+0000","last_fullsized":"2024-09-20T22:53:08.287842+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:14.225111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:48:23.320031+0000","last_clean_scrub_stamp":"2024-09-20T22:52:14.225111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:38.894957+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":1347,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.291300+0000","last_change":"2024-09-20T22:52:39.814499+0000","last_active":"2024-09-20T22:53:08.291300+0000","last_peered":"2024-09-20T22:53:08.291300+0000","last_clean":"2024-09-20T22:53:08.291300+0000","last_became_active":"2024-09-20T22:47:49.214055+0000","last_became_peered":"2024-09-20T22:47:49.214055+0000","last_unstale":"2024-09-20T22:53:08.291300+0000","last_undegraded":"2024-09-20T22:53:08.291300+0000","last_fullsized":"2024-09-20T22:53:08.291300+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:39.814381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:52:39.814381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:52.010799+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":244,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.291182+0000","last_change":"2024-09-20T22:53:02.916523+0000","last_active":"2024-09-20T22:53:08.291182+0000","last_peered":"2024-09-20T22:53:08.291182+0000","last_clean":"2024-09-20T22:53:08.291182+0000","last_became_active":"2024-09-20T22:51:20.284874+0000","last_became_peered":"2024-09-20T22:51:20.284874+0000","last_unstale":"2024-09-20T22:53:08.291182+0000","last_undegraded":"2024-09-20T22:53:08.291182+0000","last_fullsized":"2024-09-20T22:53:08.291182+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:02.916449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:48:41.208853+0000","last_clean_scrub_stamp":"2024-09-20T22:53:02.916449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:15.012347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.7","version":"708'1239","reported_seq":4353,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.302283+0000","last_change":"2024-09-20T22:53:08.302510+0000","last_active":"2024-09-20T22:53:08.302283+0000","last_peered":"2024-09-20T22:53:08.302283+0000","last_clean":"2024-09-20T22:53:08.302283+0000","last_became_active":"2024-09-20T22:48:43.224386+0000","last_became_peered":"2024-09-20T22:48:43.224386+0000","last_unstale":"2024-09-20T22:53:08.302283+0000","last_undegraded":"2024-09-20T22:53:08.302283+0000","last_fullsized":"2024-09-20T22:53:08.302283+0000","mapping_epoch":391,"log_start":"436'456","ondisk_log_start":"436'456","created":94,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"704'1235","last_scrub_stamp":"2024-09-20T22:53:06.830408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:53:06.830408+0000","objects_scrubbed":31,"log_size":783,"log_dups_size":886,"ondisk_log_size":783,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:22.349820+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0096448039999999999,"stat_sum":{"num_bytes":63153091,"num_objects":31,"num_object_clones":26,"num_object_copies":93,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":2,"num_read":442,"num_read_kb":450717,"num_write":381,"num_write_kb":126766,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":115,"num_bytes_recovered":51615167,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"bf","length":"1"},{"start":"f8","length":"1"},{"start":"fb","length":"1"},{"start":"fd","length":"6"},{"start":"104","length":"8"},{"start":"10d","length":"5"},{"start":"113","length":"8"},{"start":"11c","length":"3"},{"start":"120","length":"1"},{"start":"123","length":"1"},{"start":"125","length":"2"},{"start":"129","length":"1"},{"start":"12b","length":"1"},{"start":"133","length":"2"}]},{"pgid":"1.1","version":"77'128","reported_seq":1643,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.285445+0000","last_change":"2024-09-20T22:52:42.146147+0000","last_active":"2024-09-20T22:53:08.285445+0000","last_peered":"2024-09-20T22:53:08.285445+0000","last_clean":"2024-09-20T22:53:08.285445+0000","last_became_active":"2024-09-20T22:47:52.208102+0000","last_became_peered":"2024-09-20T22:47:52.208102+0000","last_unstale":"2024-09-20T22:53:08.285445+0000","last_undegraded":"2024-09-20T22:53:08.285445+0000","last_fullsized":"2024-09-20T22:53:08.285445+0000","mapping_epoch":391,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:52:42.146069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:52:42.146069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:50.657914+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"625'51","reported_seq":409,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.287693+0000","last_change":"2024-09-20T22:52:17.199465+0000","last_active":"2024-09-20T22:53:08.287693+0000","last_peered":"2024-09-20T22:53:08.287693+0000","last_clean":"2024-09-20T22:53:08.287693+0000","last_became_active":"2024-09-20T22:51:20.289376+0000","last_became_peered":"2024-09-20T22:51:20.289376+0000","last_unstale":"2024-09-20T22:53:08.287693+0000","last_undegraded":"2024-09-20T22:53:08.287693+0000","last_fullsized":"2024-09-20T22:53:08.287693+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'51","last_scrub_stamp":"2024-09-20T22:52:17.199367+0000","last_deep_scrub":"86'38","last_deep_scrub_stamp":"2024-09-20T22:42:43.935677+0000","last_clean_scrub_stamp":"2024-09-20T22:52:17.199367+0000","objects_scrubbed":10,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:38.979686+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":634,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.290046+0000","last_change":"2024-09-20T22:52:36.695537+0000","last_active":"2024-09-20T22:53:08.290046+0000","last_peered":"2024-09-20T22:53:08.290046+0000","last_clean":"2024-09-20T22:53:08.290046+0000","last_became_active":"2024-09-20T22:48:05.237756+0000","last_became_peered":"2024-09-20T22:48:05.237756+0000","last_unstale":"2024-09-20T22:53:08.290046+0000","last_undegraded":"2024-09-20T22:53:08.290046+0000","last_fullsized":"2024-09-20T22:53:08.290046+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:36.695446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:52:36.695446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:38.713978+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"69'2","reported_seq":1356,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.286043+0000","last_change":"2024-09-20T22:51:37.070187+0000","last_active":"2024-09-20T22:53:08.286043+0000","last_peered":"2024-09-20T22:53:08.286043+0000","last_clean":"2024-09-20T22:53:08.286043+0000","last_became_active":"2024-09-20T22:51:20.286952+0000","last_became_peered":"2024-09-20T22:51:20.286952+0000","last_unstale":"2024-09-20T22:53:08.286043+0000","last_undegraded":"2024-09-20T22:53:08.286043+0000","last_fullsized":"2024-09-20T22:53:08.286043+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2024-09-20T22:51:37.070111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:36.130684+0000","last_clean_scrub_stamp":"2024-09-20T22:51:37.070111+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087280599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":679,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.287664+0000","last_change":"2024-09-20T22:51:53.772969+0000","last_active":"2024-09-20T22:53:08.287664+0000","last_peered":"2024-09-20T22:53:08.287664+0000","last_clean":"2024-09-20T22:53:08.287664+0000","last_became_active":"2024-09-20T22:51:20.278427+0000","last_became_peered":"2024-09-20T22:51:20.278427+0000","last_unstale":"2024-09-20T22:53:08.287664+0000","last_undegraded":"2024-09-20T22:53:08.287664+0000","last_fullsized":"2024-09-20T22:53:08.287664+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:53.772885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:51:53.772885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":1327,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.296223+0000","last_change":"2024-09-20T22:52:21.363432+0000","last_active":"2024-09-20T22:53:08.296223+0000","last_peered":"2024-09-20T22:53:08.296223+0000","last_clean":"2024-09-20T22:53:08.296223+0000","last_became_active":"2024-09-20T22:48:05.234193+0000","last_became_peered":"2024-09-20T22:48:05.234193+0000","last_unstale":"2024-09-20T22:53:08.296223+0000","last_undegraded":"2024-09-20T22:53:08.296223+0000","last_fullsized":"2024-09-20T22:53:08.296223+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:21.363359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:52:21.363359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:28.660587+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"708'440","reported_seq":1985,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.298859+0000","last_change":"2024-09-20T22:53:08.299149+0000","last_active":"2024-09-20T22:53:08.298859+0000","last_peered":"2024-09-20T22:53:08.298859+0000","last_clean":"2024-09-20T22:53:08.298859+0000","last_became_active":"2024-09-20T22:51:23.267613+0000","last_became_peered":"2024-09-20T22:51:23.267613+0000","last_unstale":"2024-09-20T22:53:08.298859+0000","last_undegraded":"2024-09-20T22:53:08.298859+0000","last_fullsized":"2024-09-20T22:53:08.298859+0000","mapping_epoch":602,"log_start":"456'176","ondisk_log_start":"456'176","created":94,"last_epoch_clean":603,"parent":"0.0","parent_split_bits":0,"last_scrub":"616'341","last_scrub_stamp":"2024-09-20T22:51:36.893731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:51:36.893731+0000","objects_scrubbed":8,"log_size":264,"log_dups_size":318,"ondisk_log_size":264,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.011398122,"stat_sum":{"num_bytes":26187339,"num_objects":11,"num_object_clones":9,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":158,"num_read_kb":145638,"num_write":138,"num_write_kb":46729,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":26566881,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"bf","length":"1"},{"start":"f8","length":"1"},{"start":"fb","length":"1"},{"start":"fd","length":"6"},{"start":"104","length":"8"},{"start":"10d","length":"5"},{"start":"113","length":"8"},{"start":"11c","length":"3"},{"start":"120","length":"1"},{"start":"123","length":"1"},{"start":"125","length":"2"},{"start":"129","length":"1"},{"start":"12b","length":"1"},{"start":"133","length":"2"}]},{"pgid":"1.2","version":"77'128","reported_seq":1703,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.289865+0000","last_change":"2024-09-20T22:52:02.623634+0000","last_active":"2024-09-20T22:53:08.289865+0000","last_peered":"2024-09-20T22:53:08.289865+0000","last_clean":"2024-09-20T22:53:08.289865+0000","last_became_active":"2024-09-20T22:48:08.203028+0000","last_became_peered":"2024-09-20T22:48:08.203028+0000","last_unstale":"2024-09-20T22:53:08.289865+0000","last_undegraded":"2024-09-20T22:53:08.289865+0000","last_fullsized":"2024-09-20T22:53:08.289865+0000","mapping_epoch":407,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":408,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:52:02.623562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:52:02.623562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:31.354752+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"625'22","reported_seq":1089,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.286611+0000","last_change":"2024-09-20T22:51:20.312887+0000","last_active":"2024-09-20T22:53:08.286611+0000","last_peered":"2024-09-20T22:53:08.286611+0000","last_clean":"2024-09-20T22:53:08.286611+0000","last_became_active":"2024-09-20T22:51:20.293610+0000","last_became_peered":"2024-09-20T22:51:20.293610+0000","last_unstale":"2024-09-20T22:53:08.286611+0000","last_undegraded":"2024-09-20T22:53:08.286611+0000","last_fullsized":"2024-09-20T22:53:08.286611+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"443'18","last_scrub_stamp":"2024-09-20T22:51:04.643936+0000","last_deep_scrub":"443'18","last_deep_scrub_stamp":"2024-09-20T22:48:44.182517+0000","last_clean_scrub_stamp":"2024-09-20T22:51:04.643936+0000","objects_scrubbed":5,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":990,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":236,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.288224+0000","last_change":"2024-09-20T22:51:32.279015+0000","last_active":"2024-09-20T22:53:08.288224+0000","last_peered":"2024-09-20T22:53:08.288224+0000","last_clean":"2024-09-20T22:53:08.288224+0000","last_became_active":"2024-09-20T22:51:20.278609+0000","last_became_peered":"2024-09-20T22:51:20.278609+0000","last_unstale":"2024-09-20T22:53:08.288224+0000","last_undegraded":"2024-09-20T22:53:08.288224+0000","last_fullsized":"2024-09-20T22:53:08.288224+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:32.278945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:51:32.278945+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"380'2","reported_seq":1381,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.289195+0000","last_change":"2024-09-20T22:52:37.033309+0000","last_active":"2024-09-20T22:53:08.289195+0000","last_peered":"2024-09-20T22:53:08.289195+0000","last_clean":"2024-09-20T22:53:08.289195+0000","last_became_active":"2024-09-20T22:47:49.222812+0000","last_became_peered":"2024-09-20T22:47:49.222812+0000","last_unstale":"2024-09-20T22:53:08.289195+0000","last_undegraded":"2024-09-20T22:53:08.289195+0000","last_fullsized":"2024-09-20T22:53:08.289195+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"380'2","last_scrub_stamp":"2024-09-20T22:52:37.033222+0000","last_deep_scrub":"380'2","last_deep_scrub_stamp":"2024-09-20T22:47:40.983039+0000","last_clean_scrub_stamp":"2024-09-20T22:52:37.033222+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:56.197067+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001301459,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":1327,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.290069+0000","last_change":"2024-09-20T22:51:04.254552+0000","last_active":"2024-09-20T22:53:08.290069+0000","last_peered":"2024-09-20T22:53:08.290069+0000","last_clean":"2024-09-20T22:53:08.290069+0000","last_became_active":"2024-09-20T22:41:42.185112+0000","last_became_peered":"2024-09-20T22:41:42.185112+0000","last_unstale":"2024-09-20T22:53:08.290069+0000","last_undegraded":"2024-09-20T22:53:08.290069+0000","last_fullsized":"2024-09-20T22:53:08.290069+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:04.254441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:47:22.881109+0000","last_clean_scrub_stamp":"2024-09-20T22:51:04.254441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":1322,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.290445+0000","last_change":"2024-09-20T22:53:06.644045+0000","last_active":"2024-09-20T22:53:08.290445+0000","last_peered":"2024-09-20T22:53:08.290445+0000","last_clean":"2024-09-20T22:53:08.290445+0000","last_became_active":"2024-09-20T22:41:44.830515+0000","last_became_peered":"2024-09-20T22:41:44.830515+0000","last_unstale":"2024-09-20T22:53:08.290445+0000","last_undegraded":"2024-09-20T22:53:08.290445+0000","last_fullsized":"2024-09-20T22:53:08.290445+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:06.643947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:53:06.643947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:16.583688+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"708'351","reported_seq":3465,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.300848+0000","last_change":"2024-09-20T22:53:08.301106+0000","last_active":"2024-09-20T22:53:08.300848+0000","last_peered":"2024-09-20T22:53:08.300848+0000","last_clean":"2024-09-20T22:53:08.300848+0000","last_became_active":"2024-09-20T22:48:57.236417+0000","last_became_peered":"2024-09-20T22:48:57.236417+0000","last_unstale":"2024-09-20T22:53:08.300848+0000","last_undegraded":"2024-09-20T22:53:08.300848+0000","last_fullsized":"2024-09-20T22:53:08.300848+0000","mapping_epoch":391,"log_start":"450'190","ondisk_log_start":"450'190","created":94,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"674'316","last_scrub_stamp":"2024-09-20T22:52:35.685017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:52:35.685017+0000","objects_scrubbed":8,"log_size":161,"log_dups_size":240,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:46.348449+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.010728626999999999,"stat_sum":{"num_bytes":17366338,"num_objects":9,"num_object_clones":8,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":79,"num_read_kb":60076,"num_write":94,"num_write_kb":40424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":19673726,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"bf","length":"1"},{"start":"f8","length":"1"},{"start":"fb","length":"1"},{"start":"fd","length":"6"},{"start":"104","length":"8"},{"start":"10d","length":"5"},{"start":"113","length":"8"},{"start":"11c","length":"3"},{"start":"120","length":"1"},{"start":"123","length":"1"},{"start":"125","length":"2"},{"start":"129","length":"1"},{"start":"12b","length":"1"},{"start":"133","length":"2"}]},{"pgid":"1.3","version":"77'128","reported_seq":539,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.291480+0000","last_change":"2024-09-20T22:53:05.866046+0000","last_active":"2024-09-20T22:53:08.291480+0000","last_peered":"2024-09-20T22:53:08.291480+0000","last_clean":"2024-09-20T22:53:08.291480+0000","last_became_active":"2024-09-20T22:51:23.266810+0000","last_became_peered":"2024-09-20T22:51:23.266810+0000","last_unstale":"2024-09-20T22:53:08.291480+0000","last_undegraded":"2024-09-20T22:53:08.291480+0000","last_fullsized":"2024-09-20T22:53:08.291480+0000","mapping_epoch":602,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":603,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:53:05.865942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:53:05.865942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:28.578041+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.6","version":"625'24","reported_seq":719,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.289317+0000","last_change":"2024-09-20T22:52:48.829973+0000","last_active":"2024-09-20T22:53:08.289317+0000","last_peered":"2024-09-20T22:53:08.289317+0000","last_clean":"2024-09-20T22:53:08.289317+0000","last_became_active":"2024-09-20T22:51:20.283112+0000","last_became_peered":"2024-09-20T22:51:20.283112+0000","last_unstale":"2024-09-20T22:53:08.289317+0000","last_undegraded":"2024-09-20T22:53:08.289317+0000","last_fullsized":"2024-09-20T22:53:08.289317+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'24","last_scrub_stamp":"2024-09-20T22:52:48.829882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:52:48.829882+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:17.822454+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":673,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.286396+0000","last_change":"2024-09-20T22:50:39.786895+0000","last_active":"2024-09-20T22:53:08.286396+0000","last_peered":"2024-09-20T22:53:08.286396+0000","last_clean":"2024-09-20T22:53:08.286396+0000","last_became_active":"2024-09-20T22:48:05.233093+0000","last_became_peered":"2024-09-20T22:48:05.233093+0000","last_unstale":"2024-09-20T22:53:08.286396+0000","last_undegraded":"2024-09-20T22:53:08.286396+0000","last_fullsized":"2024-09-20T22:53:08.286396+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:50:39.786825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:44:19.834641+0000","last_clean_scrub_stamp":"2024-09-20T22:50:39.786825+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"75'5","reported_seq":2057,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.289960+0000","last_change":"2024-09-20T22:52:44.568232+0000","last_active":"2024-09-20T22:53:08.289960+0000","last_peered":"2024-09-20T22:53:08.289960+0000","last_clean":"2024-09-20T22:53:08.289960+0000","last_became_active":"2024-09-20T22:51:20.281329+0000","last_became_peered":"2024-09-20T22:51:20.281329+0000","last_unstale":"2024-09-20T22:53:08.289960+0000","last_undegraded":"2024-09-20T22:53:08.289960+0000","last_fullsized":"2024-09-20T22:53:08.289960+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-09-20T22:52:44.568129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:36.130684+0000","last_clean_scrub_stamp":"2024-09-20T22:52:44.568129+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:05.662332+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00078631700000000003,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":680,"num_read_kb":675,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1167,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":1346,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.286646+0000","last_change":"2024-09-20T22:52:48.082097+0000","last_active":"2024-09-20T22:53:08.286646+0000","last_peered":"2024-09-20T22:53:08.286646+0000","last_clean":"2024-09-20T22:53:08.286646+0000","last_became_active":"2024-09-20T22:51:20.274522+0000","last_became_peered":"2024-09-20T22:51:20.274522+0000","last_unstale":"2024-09-20T22:53:08.286646+0000","last_undegraded":"2024-09-20T22:53:08.286646+0000","last_fullsized":"2024-09-20T22:53:08.286646+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:48.082027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:52:48.082027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:51.217095+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":228,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.286349+0000","last_change":"2024-09-20T22:51:20.288058+0000","last_active":"2024-09-20T22:53:08.286349+0000","last_peered":"2024-09-20T22:53:08.286349+0000","last_clean":"2024-09-20T22:53:08.286349+0000","last_became_active":"2024-09-20T22:51:20.287750+0000","last_became_peered":"2024-09-20T22:51:20.287750+0000","last_unstale":"2024-09-20T22:53:08.286349+0000","last_undegraded":"2024-09-20T22:53:08.286349+0000","last_fullsized":"2024-09-20T22:53:08.286349+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:05.621880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:51:05.621880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.6","version":"708'456","reported_seq":3503,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.303246+0000","last_change":"2024-09-20T22:53:08.303511+0000","last_active":"2024-09-20T22:53:08.303246+0000","last_peered":"2024-09-20T22:53:08.303246+0000","last_clean":"2024-09-20T22:53:08.303246+0000","last_became_active":"2024-09-20T22:48:47.234531+0000","last_became_peered":"2024-09-20T22:48:47.234531+0000","last_unstale":"2024-09-20T22:53:08.303246+0000","last_undegraded":"2024-09-20T22:53:08.303246+0000","last_fullsized":"2024-09-20T22:53:08.303246+0000","mapping_epoch":391,"log_start":"443'181","ondisk_log_start":"443'181","created":94,"last_epoch_clean":447,"parent":"0.0","parent_split_bits":0,"last_scrub":"654'392","last_scrub_stamp":"2024-09-20T22:52:14.706591+0000","last_deep_scrub":"440'177","last_deep_scrub_stamp":"2024-09-20T22:48:40.602342+0000","last_clean_scrub_stamp":"2024-09-20T22:52:14.706591+0000","objects_scrubbed":10,"log_size":275,"log_dups_size":314,"ondisk_log_size":275,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:31.145032+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.014507139,"stat_sum":{"num_bytes":27586943,"num_objects":12,"num_object_clones":10,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":188,"num_read_kb":193597,"num_write":139,"num_write_kb":52384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":18662529,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"bf","length":"1"},{"start":"f8","length":"1"},{"start":"fb","length":"1"},{"start":"fd","length":"6"},{"start":"104","length":"8"},{"start":"10d","length":"5"},{"start":"113","length":"8"},{"start":"11c","length":"3"},{"start":"120","length":"1"},{"start":"123","length":"1"},{"start":"125","length":"2"},{"start":"129","length":"1"},{"start":"12b","length":"1"},{"start":"133","length":"2"}]},{"pgid":"4.5","version":"625'32","reported_seq":1614,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.291963+0000","last_change":"2024-09-20T22:53:07.867174+0000","last_active":"2024-09-20T22:53:08.291963+0000","last_peered":"2024-09-20T22:53:08.291963+0000","last_clean":"2024-09-20T22:53:08.291963+0000","last_became_active":"2024-09-20T22:48:05.243523+0000","last_became_peered":"2024-09-20T22:48:05.243523+0000","last_unstale":"2024-09-20T22:53:08.291963+0000","last_undegraded":"2024-09-20T22:53:08.291963+0000","last_fullsized":"2024-09-20T22:53:08.291963+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'32","last_scrub_stamp":"2024-09-20T22:53:07.867088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:53:07.867088+0000","objects_scrubbed":8,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:23.661414+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"67'1","reported_seq":1325,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.297739+0000","last_change":"2024-09-20T22:51:35.440516+0000","last_active":"2024-09-20T22:53:08.297739+0000","last_peered":"2024-09-20T22:53:08.297739+0000","last_clean":"2024-09-20T22:53:08.297739+0000","last_became_active":"2024-09-20T22:41:38.158917+0000","last_became_peered":"2024-09-20T22:41:38.158917+0000","last_unstale":"2024-09-20T22:53:08.297739+0000","last_undegraded":"2024-09-20T22:53:08.297739+0000","last_fullsized":"2024-09-20T22:53:08.297739+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-09-20T22:51:35.440416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:51:35.440416+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"77'128","reported_seq":1675,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.290127+0000","last_change":"2024-09-20T22:52:18.688292+0000","last_active":"2024-09-20T22:53:08.290127+0000","last_peered":"2024-09-20T22:53:08.290127+0000","last_clean":"2024-09-20T22:53:08.290127+0000","last_became_active":"2024-09-20T22:47:52.206169+0000","last_became_peered":"2024-09-20T22:47:52.206169+0000","last_unstale":"2024-09-20T22:53:08.290127+0000","last_undegraded":"2024-09-20T22:53:08.290127+0000","last_fullsized":"2024-09-20T22:53:08.290127+0000","mapping_epoch":391,"log_start":"77'128","ondisk_log_start":"77'128","created":19,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:52:18.688135+0000","last_deep_scrub":"77'128","last_deep_scrub_stamp":"2024-09-20T22:47:22.417099+0000","last_clean_scrub_stamp":"2024-09-20T22:52:18.688135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:47.016821+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":1334,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.290740+0000","last_change":"2024-09-20T22:53:03.564792+0000","last_active":"2024-09-20T22:53:08.290740+0000","last_peered":"2024-09-20T22:53:08.290740+0000","last_clean":"2024-09-20T22:53:08.290740+0000","last_became_active":"2024-09-20T22:41:42.184387+0000","last_became_peered":"2024-09-20T22:41:42.184387+0000","last_unstale":"2024-09-20T22:53:08.290740+0000","last_undegraded":"2024-09-20T22:53:08.290740+0000","last_fullsized":"2024-09-20T22:53:08.290740+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:03.564726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:53:03.564726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:25.313703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":1328,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.286334+0000","last_change":"2024-09-20T22:49:57.715494+0000","last_active":"2024-09-20T22:53:08.286334+0000","last_peered":"2024-09-20T22:53:08.286334+0000","last_clean":"2024-09-20T22:53:08.286334+0000","last_became_active":"2024-09-20T22:48:05.241602+0000","last_became_peered":"2024-09-20T22:48:05.241602+0000","last_unstale":"2024-09-20T22:53:08.286334+0000","last_undegraded":"2024-09-20T22:53:08.286334+0000","last_fullsized":"2024-09-20T22:53:08.286334+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:49:57.715416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:49:57.715416+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"708'553","reported_seq":3718,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.296812+0000","last_change":"2024-09-20T22:53:08.297116+0000","last_active":"2024-09-20T22:53:08.296812+0000","last_peered":"2024-09-20T22:53:08.296812+0000","last_clean":"2024-09-20T22:53:08.296812+0000","last_became_active":"2024-09-20T22:51:23.269216+0000","last_became_peered":"2024-09-20T22:51:23.269216+0000","last_unstale":"2024-09-20T22:53:08.296812+0000","last_undegraded":"2024-09-20T22:53:08.296812+0000","last_fullsized":"2024-09-20T22:53:08.296812+0000","mapping_epoch":602,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":603,"parent":"0.0","parent_split_bits":0,"last_scrub":"702'544","last_scrub_stamp":"2024-09-20T22:53:03.918130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:53:03.918130+0000","objects_scrubbed":10,"log_size":553,"log_dups_size":0,"ondisk_log_size":553,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:28.183552+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.0051844819999999998,"stat_sum":{"num_bytes":24280942,"num_objects":11,"num_object_clones":9,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":178,"num_read_kb":171553,"num_write":107,"num_write_kb":43070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":52,"num_bytes_recovered":30669453,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"bf","length":"1"},{"start":"f8","length":"1"},{"start":"fb","length":"1"},{"start":"fd","length":"6"},{"start":"104","length":"8"},{"start":"10d","length":"5"},{"start":"113","length":"8"},{"start":"11c","length":"3"},{"start":"120","length":"1"},{"start":"123","length":"1"},{"start":"125","length":"2"},{"start":"129","length":"1"},{"start":"12b","length":"1"},{"start":"133","length":"2"}]},{"pgid":"4.e","version":"625'19","reported_seq":676,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.292716+0000","last_change":"2024-09-20T22:52:44.783087+0000","last_active":"2024-09-20T22:53:08.292716+0000","last_peered":"2024-09-20T22:53:08.292716+0000","last_clean":"2024-09-20T22:53:08.292716+0000","last_became_active":"2024-09-20T22:51:20.288958+0000","last_became_peered":"2024-09-20T22:51:20.288958+0000","last_unstale":"2024-09-20T22:53:08.292716+0000","last_undegraded":"2024-09-20T22:53:08.292716+0000","last_fullsized":"2024-09-20T22:53:08.292716+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'19","last_scrub_stamp":"2024-09-20T22:52:44.782975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:52:44.782975+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:48.007796+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":634,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.293537+0000","last_change":"2024-09-20T22:51:29.757136+0000","last_active":"2024-09-20T22:53:08.293537+0000","last_peered":"2024-09-20T22:53:08.293537+0000","last_clean":"2024-09-20T22:53:08.293537+0000","last_became_active":"2024-09-20T22:48:05.254688+0000","last_became_peered":"2024-09-20T22:48:05.254688+0000","last_unstale":"2024-09-20T22:53:08.293537+0000","last_undegraded":"2024-09-20T22:53:08.293537+0000","last_fullsized":"2024-09-20T22:53:08.293537+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:29.757030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:51:29.757030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":1332,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.286242+0000","last_change":"2024-09-20T22:51:51.002353+0000","last_active":"2024-09-20T22:53:08.286242+0000","last_peered":"2024-09-20T22:53:08.286242+0000","last_clean":"2024-09-20T22:53:08.286242+0000","last_became_active":"2024-09-20T22:48:05.233778+0000","last_became_peered":"2024-09-20T22:48:05.233778+0000","last_unstale":"2024-09-20T22:53:08.286242+0000","last_undegraded":"2024-09-20T22:53:08.286242+0000","last_fullsized":"2024-09-20T22:53:08.286242+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:51.002226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:51:51.002226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:20.564071+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":1307,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.296194+0000","last_change":"2024-09-20T22:50:47.230929+0000","last_active":"2024-09-20T22:53:08.296194+0000","last_peered":"2024-09-20T22:53:08.296194+0000","last_clean":"2024-09-20T22:53:08.296194+0000","last_became_active":"2024-09-20T22:41:44.832009+0000","last_became_peered":"2024-09-20T22:41:44.832009+0000","last_unstale":"2024-09-20T22:53:08.296194+0000","last_undegraded":"2024-09-20T22:53:08.296194+0000","last_fullsized":"2024-09-20T22:53:08.296194+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:50:47.230854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:48:40.127123+0000","last_clean_scrub_stamp":"2024-09-20T22:50:47.230854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.e","version":"708'673","reported_seq":3785,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.297878+0000","last_change":"2024-09-20T22:53:08.298106+0000","last_active":"2024-09-20T22:53:08.297878+0000","last_peered":"2024-09-20T22:53:08.297878+0000","last_clean":"2024-09-20T22:53:08.297878+0000","last_became_active":"2024-09-20T22:47:52.209452+0000","last_became_peered":"2024-09-20T22:47:52.209452+0000","last_unstale":"2024-09-20T22:53:08.297878+0000","last_undegraded":"2024-09-20T22:53:08.297878+0000","last_fullsized":"2024-09-20T22:53:08.297878+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"681'618","last_scrub_stamp":"2024-09-20T22:52:45.935130+0000","last_deep_scrub":"174'60","last_deep_scrub_stamp":"2024-09-20T22:44:18.301776+0000","last_clean_scrub_stamp":"2024-09-20T22:52:45.935130+0000","objects_scrubbed":13,"log_size":673,"log_dups_size":0,"ondisk_log_size":673,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:47.673915+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.0085316109999999997,"stat_sum":{"num_bytes":28690576,"num_objects":14,"num_object_clones":12,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":247,"num_read_kb":256134,"num_write":165,"num_write_kb":61453,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":23110741,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"bf","length":"1"},{"start":"f8","length":"1"},{"start":"fb","length":"1"},{"start":"fd","length":"6"},{"start":"104","length":"8"},{"start":"10d","length":"5"},{"start":"113","length":"8"},{"start":"11c","length":"3"},{"start":"120","length":"1"},{"start":"123","length":"1"},{"start":"125","length":"2"},{"start":"129","length":"1"},{"start":"12b","length":"1"},{"start":"133","length":"2"}]},{"pgid":"1.8","version":"77'128","reported_seq":1595,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.285428+0000","last_change":"2024-09-20T22:51:56.108038+0000","last_active":"2024-09-20T22:53:08.285428+0000","last_peered":"2024-09-20T22:53:08.285428+0000","last_clean":"2024-09-20T22:53:08.285428+0000","last_became_active":"2024-09-20T22:42:52.405594+0000","last_became_peered":"2024-09-20T22:42:52.405594+0000","last_unstale":"2024-09-20T22:53:08.285428+0000","last_undegraded":"2024-09-20T22:53:08.285428+0000","last_fullsized":"2024-09-20T22:53:08.285428+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:51:56.107944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:51:56.107944+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:19.128079+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"625'31","reported_seq":705,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.292941+0000","last_change":"2024-09-20T22:52:13.620897+0000","last_active":"2024-09-20T22:53:08.292941+0000","last_peered":"2024-09-20T22:53:08.292941+0000","last_clean":"2024-09-20T22:53:08.292941+0000","last_became_active":"2024-09-20T22:51:20.288846+0000","last_became_peered":"2024-09-20T22:51:20.288846+0000","last_unstale":"2024-09-20T22:53:08.292941+0000","last_undegraded":"2024-09-20T22:53:08.292941+0000","last_fullsized":"2024-09-20T22:53:08.292941+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'31","last_scrub_stamp":"2024-09-20T22:52:13.620754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:52:13.620754+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:26.544483+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":49,"num_read_kb":39,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":890,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.289932+0000","last_change":"2024-09-20T22:53:08.105116+0000","last_active":"2024-09-20T22:53:08.289932+0000","last_peered":"2024-09-20T22:53:08.289932+0000","last_clean":"2024-09-20T22:53:08.289932+0000","last_became_active":"2024-09-20T22:51:20.294073+0000","last_became_peered":"2024-09-20T22:51:20.294073+0000","last_unstale":"2024-09-20T22:53:08.289932+0000","last_undegraded":"2024-09-20T22:53:08.289932+0000","last_fullsized":"2024-09-20T22:53:08.289932+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:08.105010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:35.492562+0000","last_clean_scrub_stamp":"2024-09-20T22:53:08.105010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:35.105470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":946,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.289528+0000","last_change":"2024-09-20T22:51:25.738132+0000","last_active":"2024-09-20T22:53:08.289528+0000","last_peered":"2024-09-20T22:53:08.289528+0000","last_clean":"2024-09-20T22:53:08.289528+0000","last_became_active":"2024-09-20T22:51:20.281724+0000","last_became_peered":"2024-09-20T22:51:20.281724+0000","last_unstale":"2024-09-20T22:53:08.289528+0000","last_undegraded":"2024-09-20T22:53:08.289528+0000","last_fullsized":"2024-09-20T22:53:08.289528+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:25.738061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:51:25.738061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":1351,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.288065+0000","last_change":"2024-09-20T22:53:05.106323+0000","last_active":"2024-09-20T22:53:08.288065+0000","last_peered":"2024-09-20T22:53:08.288065+0000","last_clean":"2024-09-20T22:53:08.288065+0000","last_became_active":"2024-09-20T22:48:05.239006+0000","last_became_peered":"2024-09-20T22:48:05.239006+0000","last_unstale":"2024-09-20T22:53:08.288065+0000","last_undegraded":"2024-09-20T22:53:08.288065+0000","last_fullsized":"2024-09-20T22:53:08.288065+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:05.106224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:53:05.106224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:05.128923+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"708'740","reported_seq":3819,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.305218+0000","last_change":"2024-09-20T22:53:08.305449+0000","last_active":"2024-09-20T22:53:08.305218+0000","last_peered":"2024-09-20T22:53:08.305218+0000","last_clean":"2024-09-20T22:53:08.305218+0000","last_became_active":"2024-09-20T22:47:52.216198+0000","last_became_peered":"2024-09-20T22:47:52.216198+0000","last_unstale":"2024-09-20T22:53:08.305218+0000","last_undegraded":"2024-09-20T22:53:08.305218+0000","last_fullsized":"2024-09-20T22:53:08.305218+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"630'603","last_scrub_stamp":"2024-09-20T22:51:50.769050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:51:50.769050+0000","objects_scrubbed":8,"log_size":740,"log_dups_size":0,"ondisk_log_size":740,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.015620185999999999,"stat_sum":{"num_bytes":28441281,"num_objects":14,"num_object_clones":11,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":1,"num_read":306,"num_read_kb":304372,"num_write":187,"num_write_kb":60596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":53,"num_bytes_recovered":14983556,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"bf","length":"1"},{"start":"f8","length":"1"},{"start":"fb","length":"1"},{"start":"fd","length":"6"},{"start":"104","length":"8"},{"start":"10d","length":"5"},{"start":"113","length":"8"},{"start":"11c","length":"3"},{"start":"120","length":"1"},{"start":"123","length":"1"},{"start":"125","length":"2"},{"start":"129","length":"1"},{"start":"12b","length":"1"},{"start":"133","length":"2"}]},{"pgid":"1.9","version":"77'128","reported_seq":1643,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.289917+0000","last_change":"2024-09-20T22:52:39.568327+0000","last_active":"2024-09-20T22:53:08.289917+0000","last_peered":"2024-09-20T22:53:08.289917+0000","last_clean":"2024-09-20T22:53:08.289917+0000","last_became_active":"2024-09-20T22:50:42.262862+0000","last_became_peered":"2024-09-20T22:50:42.262862+0000","last_unstale":"2024-09-20T22:53:08.289917+0000","last_undegraded":"2024-09-20T22:53:08.289917+0000","last_fullsized":"2024-09-20T22:53:08.289917+0000","mapping_epoch":561,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":562,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:52:39.568240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:52:39.568240+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:39.745450+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"625'20","reported_seq":661,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.292601+0000","last_change":"2024-09-20T22:52:22.646547+0000","last_active":"2024-09-20T22:53:08.292601+0000","last_peered":"2024-09-20T22:53:08.292601+0000","last_clean":"2024-09-20T22:53:08.292601+0000","last_became_active":"2024-09-20T22:48:05.242811+0000","last_became_peered":"2024-09-20T22:48:05.242811+0000","last_unstale":"2024-09-20T22:53:08.292601+0000","last_undegraded":"2024-09-20T22:53:08.292601+0000","last_fullsized":"2024-09-20T22:53:08.292601+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'20","last_scrub_stamp":"2024-09-20T22:52:22.646468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:52:22.646468+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:48.440427+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":1351,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.298416+0000","last_change":"2024-09-20T22:52:44.527004+0000","last_active":"2024-09-20T22:53:08.298416+0000","last_peered":"2024-09-20T22:53:08.298416+0000","last_clean":"2024-09-20T22:53:08.298416+0000","last_became_active":"2024-09-20T22:48:05.233999+0000","last_became_peered":"2024-09-20T22:48:05.233999+0000","last_unstale":"2024-09-20T22:53:08.298416+0000","last_undegraded":"2024-09-20T22:53:08.298416+0000","last_fullsized":"2024-09-20T22:53:08.298416+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:44.526935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:52:44.526935+0000","last_clean_scrub_stamp":"2024-09-20T22:52:44.526935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:45.123238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"261'9","reported_seq":1483,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.288543+0000","last_change":"2024-09-20T22:52:18.218722+0000","last_active":"2024-09-20T22:53:08.288543+0000","last_peered":"2024-09-20T22:53:08.288543+0000","last_clean":"2024-09-20T22:53:08.288543+0000","last_became_active":"2024-09-20T22:41:42.185863+0000","last_became_peered":"2024-09-20T22:41:42.185863+0000","last_unstale":"2024-09-20T22:53:08.288543+0000","last_undegraded":"2024-09-20T22:53:08.288543+0000","last_fullsized":"2024-09-20T22:53:08.288543+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'9","last_scrub_stamp":"2024-09-20T22:52:18.218644+0000","last_deep_scrub":"261'9","last_deep_scrub_stamp":"2024-09-20T22:46:50.974735+0000","last_clean_scrub_stamp":"2024-09-20T22:52:18.218644+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:38.055984+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":640,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.292454+0000","last_change":"2024-09-20T22:51:20.296567+0000","last_active":"2024-09-20T22:53:08.292454+0000","last_peered":"2024-09-20T22:53:08.292454+0000","last_clean":"2024-09-20T22:53:08.292454+0000","last_became_active":"2024-09-20T22:51:20.296204+0000","last_became_peered":"2024-09-20T22:51:20.296204+0000","last_unstale":"2024-09-20T22:53:08.292454+0000","last_undegraded":"2024-09-20T22:53:08.292454+0000","last_fullsized":"2024-09-20T22:53:08.292454+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:02.932364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:51:02.932364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.8","version":"708'839","reported_seq":3770,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.297189+0000","last_change":"2024-09-20T22:53:08.298454+0000","last_active":"2024-09-20T22:53:08.297189+0000","last_peered":"2024-09-20T22:53:08.297189+0000","last_clean":"2024-09-20T22:53:08.297189+0000","last_became_active":"2024-09-20T22:48:31.224359+0000","last_became_peered":"2024-09-20T22:48:31.224359+0000","last_unstale":"2024-09-20T22:53:08.297189+0000","last_undegraded":"2024-09-20T22:53:08.297189+0000","last_fullsized":"2024-09-20T22:53:08.297189+0000","mapping_epoch":407,"log_start":"426'268","ondisk_log_start":"426'268","created":94,"last_epoch_clean":431,"parent":"0.0","parent_split_bits":0,"last_scrub":"674'746","last_scrub_stamp":"2024-09-20T22:52:35.065256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:52:35.065256+0000","objects_scrubbed":20,"log_size":571,"log_dups_size":445,"ondisk_log_size":571,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:50.034752+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.011046392,"stat_sum":{"num_bytes":44032348,"num_objects":20,"num_object_clones":17,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":1,"num_read":291,"num_read_kb":300600,"num_write":225,"num_write_kb":72122,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":26027409,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"bf","length":"1"},{"start":"f8","length":"1"},{"start":"fb","length":"1"},{"start":"fd","length":"6"},{"start":"104","length":"8"},{"start":"10d","length":"5"},{"start":"113","length":"8"},{"start":"11c","length":"3"},{"start":"120","length":"1"},{"start":"123","length":"1"},{"start":"125","length":"2"},{"start":"129","length":"1"},{"start":"12b","length":"1"},{"start":"133","length":"2"}]},{"pgid":"4.b","version":"625'15","reported_seq":718,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.287994+0000","last_change":"2024-09-20T22:51:20.296532+0000","last_active":"2024-09-20T22:53:08.287994+0000","last_peered":"2024-09-20T22:53:08.287994+0000","last_clean":"2024-09-20T22:53:08.287994+0000","last_became_active":"2024-09-20T22:51:20.273094+0000","last_became_peered":"2024-09-20T22:51:20.273094+0000","last_unstale":"2024-09-20T22:53:08.287994+0000","last_undegraded":"2024-09-20T22:53:08.287994+0000","last_fullsized":"2024-09-20T22:53:08.287994+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-09-20T22:50:36.883811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:50:36.883811+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":1328,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.286591+0000","last_change":"2024-09-20T22:51:12.755628+0000","last_active":"2024-09-20T22:53:08.286591+0000","last_peered":"2024-09-20T22:53:08.286591+0000","last_clean":"2024-09-20T22:53:08.286591+0000","last_became_active":"2024-09-20T22:41:38.155953+0000","last_became_peered":"2024-09-20T22:41:38.155953+0000","last_unstale":"2024-09-20T22:53:08.286591+0000","last_undegraded":"2024-09-20T22:53:08.286591+0000","last_fullsized":"2024-09-20T22:53:08.286591+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:12.755526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:51:12.755526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":1359,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.290038+0000","last_change":"2024-09-20T22:52:42.998852+0000","last_active":"2024-09-20T22:53:08.290038+0000","last_peered":"2024-09-20T22:53:08.290038+0000","last_clean":"2024-09-20T22:53:08.290038+0000","last_became_active":"2024-09-20T22:48:05.240613+0000","last_became_peered":"2024-09-20T22:48:05.240613+0000","last_unstale":"2024-09-20T22:53:08.290038+0000","last_undegraded":"2024-09-20T22:53:08.290038+0000","last_fullsized":"2024-09-20T22:53:08.290038+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:42.998756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:52:42.998756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:11.946882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":667,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.286485+0000","last_change":"2024-09-20T22:52:16.706497+0000","last_active":"2024-09-20T22:53:08.286485+0000","last_peered":"2024-09-20T22:53:08.286485+0000","last_clean":"2024-09-20T22:53:08.286485+0000","last_became_active":"2024-09-20T22:47:49.219486+0000","last_became_peered":"2024-09-20T22:47:49.219486+0000","last_unstale":"2024-09-20T22:53:08.286485+0000","last_undegraded":"2024-09-20T22:53:08.286485+0000","last_fullsized":"2024-09-20T22:53:08.286485+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:16.706429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:52:16.706429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:26.931246+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"708'1133","reported_seq":4270,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.302259+0000","last_change":"2024-09-20T22:53:08.302628+0000","last_active":"2024-09-20T22:53:08.302259+0000","last_peered":"2024-09-20T22:53:08.302259+0000","last_clean":"2024-09-20T22:53:08.302259+0000","last_became_active":"2024-09-20T22:51:23.270962+0000","last_became_peered":"2024-09-20T22:51:23.270962+0000","last_unstale":"2024-09-20T22:53:08.302259+0000","last_undegraded":"2024-09-20T22:53:08.302259+0000","last_fullsized":"2024-09-20T22:53:08.302259+0000","mapping_epoch":602,"log_start":"423'280","ondisk_log_start":"423'280","created":94,"last_epoch_clean":603,"parent":"0.0","parent_split_bits":0,"last_scrub":"674'1040","last_scrub_stamp":"2024-09-20T22:52:34.647394+0000","last_deep_scrub":"590'828","last_deep_scrub_stamp":"2024-09-20T22:51:10.909364+0000","last_clean_scrub_stamp":"2024-09-20T22:52:34.647394+0000","objects_scrubbed":29,"log_size":853,"log_dups_size":649,"ondisk_log_size":853,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:48.620185+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.015822605999999999,"stat_sum":{"num_bytes":65083069,"num_objects":28,"num_object_clones":23,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":2,"num_read":401,"num_read_kb":392866,"num_write":318,"num_write_kb":122185,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":91020358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"bf","length":"1"},{"start":"f8","length":"1"},{"start":"fb","length":"1"},{"start":"fd","length":"6"},{"start":"104","length":"8"},{"start":"10d","length":"5"},{"start":"113","length":"8"},{"start":"11c","length":"3"},{"start":"120","length":"1"},{"start":"123","length":"1"},{"start":"125","length":"2"},{"start":"129","length":"1"},{"start":"12b","length":"1"},{"start":"133","length":"2"}]},{"pgid":"4.a","version":"625'44","reported_seq":1429,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.291399+0000","last_change":"2024-09-20T22:52:14.038882+0000","last_active":"2024-09-20T22:53:08.291399+0000","last_peered":"2024-09-20T22:53:08.291399+0000","last_clean":"2024-09-20T22:53:08.291399+0000","last_became_active":"2024-09-20T22:51:20.287620+0000","last_became_peered":"2024-09-20T22:51:20.287620+0000","last_unstale":"2024-09-20T22:53:08.291399+0000","last_undegraded":"2024-09-20T22:53:08.291399+0000","last_fullsized":"2024-09-20T22:53:08.291399+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'44","last_scrub_stamp":"2024-09-20T22:52:14.038788+0000","last_deep_scrub":"113'28","last_deep_scrub_stamp":"2024-09-20T22:43:12.725900+0000","last_clean_scrub_stamp":"2024-09-20T22:52:14.038788+0000","objects_scrubbed":9,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:32.613295+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":1326,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.290667+0000","last_change":"2024-09-20T22:51:36.440271+0000","last_active":"2024-09-20T22:53:08.290667+0000","last_peered":"2024-09-20T22:53:08.290667+0000","last_clean":"2024-09-20T22:53:08.290667+0000","last_became_active":"2024-09-20T22:41:38.161524+0000","last_became_peered":"2024-09-20T22:41:38.161524+0000","last_unstale":"2024-09-20T22:53:08.290667+0000","last_undegraded":"2024-09-20T22:53:08.290667+0000","last_fullsized":"2024-09-20T22:53:08.290667+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:36.440160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:51:36.440160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":1347,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.287070+0000","last_change":"2024-09-20T22:52:40.004730+0000","last_active":"2024-09-20T22:53:08.287070+0000","last_peered":"2024-09-20T22:53:08.287070+0000","last_clean":"2024-09-20T22:53:08.287070+0000","last_became_active":"2024-09-20T22:47:49.223220+0000","last_became_peered":"2024-09-20T22:47:49.223220+0000","last_unstale":"2024-09-20T22:53:08.287070+0000","last_undegraded":"2024-09-20T22:53:08.287070+0000","last_fullsized":"2024-09-20T22:53:08.287070+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:40.004661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:52:40.004661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:04.999859+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":1322,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.290026+0000","last_change":"2024-09-20T22:52:01.643899+0000","last_active":"2024-09-20T22:53:08.290026+0000","last_peered":"2024-09-20T22:53:08.290026+0000","last_clean":"2024-09-20T22:53:08.290026+0000","last_became_active":"2024-09-20T22:41:44.199395+0000","last_became_peered":"2024-09-20T22:41:44.199395+0000","last_unstale":"2024-09-20T22:53:08.290026+0000","last_undegraded":"2024-09-20T22:53:08.290026+0000","last_fullsized":"2024-09-20T22:53:08.290026+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:01.643803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:47:18.329209+0000","last_clean_scrub_stamp":"2024-09-20T22:52:01.643803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.a","version":"708'835","reported_seq":3834,"reported_epoch":708,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T22:53:08.334861+0000","last_change":"2024-09-20T22:53:08.331844+0000","last_active":"2024-09-20T22:53:08.334861+0000","last_peered":"2024-09-20T22:53:08.334861+0000","last_clean":"2024-09-20T22:53:08.334861+0000","last_became_active":"2024-09-20T22:42:52.405085+0000","last_became_peered":"2024-09-20T22:42:52.405085+0000","last_unstale":"2024-09-20T22:53:08.334861+0000","last_undegraded":"2024-09-20T22:53:08.334861+0000","last_fullsized":"2024-09-20T22:53:08.334861+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"505'466","last_scrub_stamp":"2024-09-20T22:49:47.088165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:49:47.088165+0000","objects_scrubbed":13,"log_size":835,"log_dups_size":0,"ondisk_log_size":835,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.0063310459999999999,"stat_sum":{"num_bytes":57160920,"num_objects":22,"num_object_clones":19,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":353,"num_read_kb":370681,"num_write":271,"num_write_kb":105688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"bf","length":"1"},{"start":"f8","length":"1"},{"start":"fb","length":"1"},{"start":"fd","length":"6"},{"start":"104","length":"8"},{"start":"10d","length":"1"},{"start":"10f","length":"3"},{"start":"113","length":"6"},{"start":"11a","length":"1"},{"start":"11c","length":"3"},{"start":"120","length":"1"},{"start":"123","length":"1"},{"start":"125","length":"1"},{"start":"129","length":"1"},{"start":"133","length":"2"}]},{"pgid":"4.9","version":"686'23","reported_seq":696,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.293678+0000","last_change":"2024-09-20T22:52:46.768898+0000","last_active":"2024-09-20T22:53:08.293678+0000","last_peered":"2024-09-20T22:53:08.293678+0000","last_clean":"2024-09-20T22:53:08.293678+0000","last_became_active":"2024-09-20T22:52:11.286549+0000","last_became_peered":"2024-09-20T22:52:11.286549+0000","last_unstale":"2024-09-20T22:53:08.293678+0000","last_undegraded":"2024-09-20T22:53:08.293678+0000","last_fullsized":"2024-09-20T22:53:08.293678+0000","mapping_epoch":650,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":651,"parent":"0.0","parent_split_bits":0,"last_scrub":"686'23","last_scrub_stamp":"2024-09-20T22:52:46.768812+0000","last_deep_scrub":"686'23","last_deep_scrub_stamp":"2024-09-20T22:52:46.768812+0000","last_clean_scrub_stamp":"2024-09-20T22:52:46.768812+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:50.916744+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":21,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":1364,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.290468+0000","last_change":"2024-09-20T22:51:55.653755+0000","last_active":"2024-09-20T22:53:08.290468+0000","last_peered":"2024-09-20T22:53:08.290468+0000","last_clean":"2024-09-20T22:53:08.290468+0000","last_became_active":"2024-09-20T22:51:20.293701+0000","last_became_peered":"2024-09-20T22:51:20.293701+0000","last_unstale":"2024-09-20T22:53:08.290468+0000","last_undegraded":"2024-09-20T22:53:08.290468+0000","last_fullsized":"2024-09-20T22:53:08.290468+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:55.653662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:46:52.465940+0000","last_clean_scrub_stamp":"2024-09-20T22:51:55.653662+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":1361,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.287537+0000","last_change":"2024-09-20T22:53:07.104977+0000","last_active":"2024-09-20T22:53:08.287537+0000","last_peered":"2024-09-20T22:53:08.287537+0000","last_clean":"2024-09-20T22:53:08.287537+0000","last_became_active":"2024-09-20T22:51:20.286626+0000","last_became_peered":"2024-09-20T22:51:20.286626+0000","last_unstale":"2024-09-20T22:53:08.287537+0000","last_undegraded":"2024-09-20T22:53:08.287537+0000","last_fullsized":"2024-09-20T22:53:08.287537+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:07.104891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:53:07.104891+0000","last_clean_scrub_stamp":"2024-09-20T22:53:07.104891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:17.124586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":1313,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.298027+0000","last_change":"2024-09-20T22:51:20.279408+0000","last_active":"2024-09-20T22:53:08.298027+0000","last_peered":"2024-09-20T22:53:08.298027+0000","last_clean":"2024-09-20T22:53:08.298027+0000","last_became_active":"2024-09-20T22:51:20.279088+0000","last_became_peered":"2024-09-20T22:51:20.279088+0000","last_unstale":"2024-09-20T22:53:08.298027+0000","last_undegraded":"2024-09-20T22:53:08.298027+0000","last_fullsized":"2024-09-20T22:53:08.298027+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:50:39.266034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:55.170379+0000","last_clean_scrub_stamp":"2024-09-20T22:50:39.266034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"708'1099","reported_seq":4276,"reported_epoch":708,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T22:53:08.334605+0000","last_change":"2024-09-20T22:53:08.331990+0000","last_active":"2024-09-20T22:53:08.334605+0000","last_peered":"2024-09-20T22:53:08.334605+0000","last_clean":"2024-09-20T22:53:08.334605+0000","last_became_active":"2024-09-20T22:51:23.271244+0000","last_became_peered":"2024-09-20T22:51:23.271244+0000","last_unstale":"2024-09-20T22:53:08.334605+0000","last_undegraded":"2024-09-20T22:53:08.334605+0000","last_fullsized":"2024-09-20T22:53:08.334605+0000","mapping_epoch":602,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":603,"parent":"0.0","parent_split_bits":0,"last_scrub":"661'1000","last_scrub_stamp":"2024-09-20T22:52:23.164028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:52:23.164028+0000","objects_scrubbed":23,"log_size":1099,"log_dups_size":0,"ondisk_log_size":1099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:35.866726+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.044929170999999997,"stat_sum":{"num_bytes":61380231,"num_objects":26,"num_object_clones":21,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":2,"num_read":515,"num_read_kb":586632,"num_write":351,"num_write_kb":121518,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":76,"num_bytes_recovered":39288293,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"bf","length":"1"},{"start":"f8","length":"1"},{"start":"fb","length":"1"},{"start":"fd","length":"6"},{"start":"104","length":"8"},{"start":"10d","length":"1"},{"start":"10f","length":"3"},{"start":"113","length":"6"},{"start":"11a","length":"1"},{"start":"11c","length":"3"},{"start":"120","length":"1"},{"start":"123","length":"1"},{"start":"125","length":"1"},{"start":"129","length":"1"},{"start":"133","length":"2"}]},{"pgid":"4.8","version":"625'27","reported_seq":1367,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.285577+0000","last_change":"2024-09-20T22:51:53.020721+0000","last_active":"2024-09-20T22:53:08.285577+0000","last_peered":"2024-09-20T22:53:08.285577+0000","last_clean":"2024-09-20T22:53:08.285577+0000","last_became_active":"2024-09-20T22:41:40.252113+0000","last_became_peered":"2024-09-20T22:41:40.252113+0000","last_unstale":"2024-09-20T22:53:08.285577+0000","last_undegraded":"2024-09-20T22:53:08.285577+0000","last_fullsized":"2024-09-20T22:53:08.285577+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'27","last_scrub_stamp":"2024-09-20T22:51:53.020628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:51:53.020628+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:14.005659+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'2","reported_seq":1389,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.295544+0000","last_change":"2024-09-20T22:51:26.421985+0000","last_active":"2024-09-20T22:53:08.295544+0000","last_peered":"2024-09-20T22:53:08.295544+0000","last_clean":"2024-09-20T22:53:08.295544+0000","last_became_active":"2024-09-20T22:48:05.238098+0000","last_became_peered":"2024-09-20T22:48:05.238098+0000","last_unstale":"2024-09-20T22:53:08.295544+0000","last_undegraded":"2024-09-20T22:53:08.295544+0000","last_fullsized":"2024-09-20T22:53:08.295544+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-09-20T22:51:26.421896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:51:26.421896+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"408'10","reported_seq":1493,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.290179+0000","last_change":"2024-09-20T22:52:15.703572+0000","last_active":"2024-09-20T22:53:08.290179+0000","last_peered":"2024-09-20T22:53:08.290179+0000","last_clean":"2024-09-20T22:53:08.290179+0000","last_became_active":"2024-09-20T22:48:05.239344+0000","last_became_peered":"2024-09-20T22:48:05.239344+0000","last_unstale":"2024-09-20T22:53:08.290179+0000","last_undegraded":"2024-09-20T22:53:08.290179+0000","last_fullsized":"2024-09-20T22:53:08.290179+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"408'10","last_scrub_stamp":"2024-09-20T22:52:15.703464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:52:15.703464+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:23.273102+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":1328,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.285642+0000","last_change":"2024-09-20T22:51:54.066995+0000","last_active":"2024-09-20T22:53:08.285642+0000","last_peered":"2024-09-20T22:53:08.285642+0000","last_clean":"2024-09-20T22:53:08.285642+0000","last_became_active":"2024-09-20T22:47:49.227569+0000","last_became_peered":"2024-09-20T22:47:49.227569+0000","last_unstale":"2024-09-20T22:53:08.285642+0000","last_undegraded":"2024-09-20T22:53:08.285642+0000","last_fullsized":"2024-09-20T22:53:08.285642+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:54.066913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:51:54.066913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:18.059705+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":1354,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.291225+0000","last_change":"2024-09-20T22:51:58.129589+0000","last_active":"2024-09-20T22:53:08.291225+0000","last_peered":"2024-09-20T22:53:08.291225+0000","last_clean":"2024-09-20T22:53:08.291225+0000","last_became_active":"2024-09-20T22:47:49.213292+0000","last_became_peered":"2024-09-20T22:47:49.213292+0000","last_unstale":"2024-09-20T22:53:08.291225+0000","last_undegraded":"2024-09-20T22:53:08.291225+0000","last_fullsized":"2024-09-20T22:53:08.291225+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:58.129502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:51:58.129502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:11.276401+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"625'12","reported_seq":1354,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.296627+0000","last_change":"2024-09-20T22:52:14.322926+0000","last_active":"2024-09-20T22:53:08.296627+0000","last_peered":"2024-09-20T22:53:08.296627+0000","last_clean":"2024-09-20T22:53:08.296627+0000","last_became_active":"2024-09-20T22:43:09.116887+0000","last_became_peered":"2024-09-20T22:43:09.116887+0000","last_unstale":"2024-09-20T22:53:08.296627+0000","last_undegraded":"2024-09-20T22:53:08.296627+0000","last_fullsized":"2024-09-20T22:53:08.296627+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'12","last_scrub_stamp":"2024-09-20T22:52:14.322811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:52:14.322811+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:23.704685+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":1333,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.286645+0000","last_change":"2024-09-20T22:53:03.888115+0000","last_active":"2024-09-20T22:53:08.286645+0000","last_peered":"2024-09-20T22:53:08.286645+0000","last_clean":"2024-09-20T22:53:08.286645+0000","last_became_active":"2024-09-20T22:51:20.293232+0000","last_became_peered":"2024-09-20T22:51:20.293232+0000","last_unstale":"2024-09-20T22:53:08.286645+0000","last_undegraded":"2024-09-20T22:53:08.286645+0000","last_fullsized":"2024-09-20T22:53:08.286645+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:03.888040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:09.705632+0000","last_clean_scrub_stamp":"2024-09-20T22:53:03.888040+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:19.352989+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"74'1","reported_seq":1342,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.290335+0000","last_change":"2024-09-20T22:52:49.575819+0000","last_active":"2024-09-20T22:53:08.290335+0000","last_peered":"2024-09-20T22:53:08.290335+0000","last_clean":"2024-09-20T22:53:08.290335+0000","last_became_active":"2024-09-20T22:48:05.237882+0000","last_became_peered":"2024-09-20T22:48:05.237882+0000","last_unstale":"2024-09-20T22:53:08.290335+0000","last_undegraded":"2024-09-20T22:53:08.290335+0000","last_fullsized":"2024-09-20T22:53:08.290335+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-09-20T22:52:49.575702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:52:49.575702+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:06.855774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"625'15","reported_seq":686,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.289171+0000","last_change":"2024-09-20T22:51:52.816826+0000","last_active":"2024-09-20T22:53:08.289171+0000","last_peered":"2024-09-20T22:53:08.289171+0000","last_clean":"2024-09-20T22:53:08.289171+0000","last_became_active":"2024-09-20T22:47:49.231315+0000","last_became_peered":"2024-09-20T22:47:49.231315+0000","last_unstale":"2024-09-20T22:53:08.289171+0000","last_undegraded":"2024-09-20T22:53:08.289171+0000","last_fullsized":"2024-09-20T22:53:08.289171+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'15","last_scrub_stamp":"2024-09-20T22:51:52.816714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:51:52.816714+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":1349,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.289692+0000","last_change":"2024-09-20T22:51:12.541531+0000","last_active":"2024-09-20T22:53:08.289692+0000","last_peered":"2024-09-20T22:53:08.289692+0000","last_clean":"2024-09-20T22:53:08.289692+0000","last_became_active":"2024-09-20T22:48:05.238801+0000","last_became_peered":"2024-09-20T22:48:05.238801+0000","last_unstale":"2024-09-20T22:53:08.289692+0000","last_undegraded":"2024-09-20T22:53:08.289692+0000","last_fullsized":"2024-09-20T22:53:08.289692+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:12.541457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:51:12.541457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":1324,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.296382+0000","last_change":"2024-09-20T22:52:20.410227+0000","last_active":"2024-09-20T22:53:08.296382+0000","last_peered":"2024-09-20T22:53:08.296382+0000","last_clean":"2024-09-20T22:53:08.296382+0000","last_became_active":"2024-09-20T22:51:20.279397+0000","last_became_peered":"2024-09-20T22:51:20.279397+0000","last_unstale":"2024-09-20T22:53:08.296382+0000","last_undegraded":"2024-09-20T22:53:08.296382+0000","last_fullsized":"2024-09-20T22:53:08.296382+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:20.410164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:52:20.410164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:50.341745+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":1354,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.287296+0000","last_change":"2024-09-20T22:52:19.253411+0000","last_active":"2024-09-20T22:53:08.287296+0000","last_peered":"2024-09-20T22:53:08.287296+0000","last_clean":"2024-09-20T22:53:08.287296+0000","last_became_active":"2024-09-20T22:48:05.239710+0000","last_became_peered":"2024-09-20T22:48:05.239710+0000","last_unstale":"2024-09-20T22:53:08.287296+0000","last_undegraded":"2024-09-20T22:53:08.287296+0000","last_fullsized":"2024-09-20T22:53:08.287296+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:19.253343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:52:19.253343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:44.397693+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"625'15","reported_seq":1348,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.286301+0000","last_change":"2024-09-20T22:51:05.675215+0000","last_active":"2024-09-20T22:53:08.286301+0000","last_peered":"2024-09-20T22:53:08.286301+0000","last_clean":"2024-09-20T22:53:08.286301+0000","last_became_active":"2024-09-20T22:41:40.252486+0000","last_became_peered":"2024-09-20T22:41:40.252486+0000","last_unstale":"2024-09-20T22:53:08.286301+0000","last_undegraded":"2024-09-20T22:53:08.286301+0000","last_fullsized":"2024-09-20T22:53:08.286301+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-09-20T22:51:05.675095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:51:05.675095+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":675,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.287907+0000","last_change":"2024-09-20T22:52:13.735451+0000","last_active":"2024-09-20T22:53:08.287907+0000","last_peered":"2024-09-20T22:53:08.287907+0000","last_clean":"2024-09-20T22:53:08.287907+0000","last_became_active":"2024-09-20T22:47:49.229936+0000","last_became_peered":"2024-09-20T22:47:49.229936+0000","last_unstale":"2024-09-20T22:53:08.287907+0000","last_undegraded":"2024-09-20T22:53:08.287907+0000","last_fullsized":"2024-09-20T22:53:08.287907+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:13.735382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:52:13.735382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:22.644334+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"74'8","reported_seq":1460,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.296145+0000","last_change":"2024-09-20T22:50:38.269118+0000","last_active":"2024-09-20T22:53:08.296145+0000","last_peered":"2024-09-20T22:53:08.296145+0000","last_clean":"2024-09-20T22:53:08.296145+0000","last_became_active":"2024-09-20T22:41:42.186352+0000","last_became_peered":"2024-09-20T22:41:42.186352+0000","last_unstale":"2024-09-20T22:53:08.296145+0000","last_undegraded":"2024-09-20T22:53:08.296145+0000","last_fullsized":"2024-09-20T22:53:08.296145+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-09-20T22:50:38.269010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:50:38.269010+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":634,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.293130+0000","last_change":"2024-09-20T22:52:16.623463+0000","last_active":"2024-09-20T22:53:08.293130+0000","last_peered":"2024-09-20T22:53:08.293130+0000","last_clean":"2024-09-20T22:53:08.293130+0000","last_became_active":"2024-09-20T22:48:05.249656+0000","last_became_peered":"2024-09-20T22:48:05.249656+0000","last_unstale":"2024-09-20T22:53:08.293130+0000","last_undegraded":"2024-09-20T22:53:08.293130+0000","last_fullsized":"2024-09-20T22:53:08.293130+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:16.623368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:52:16.623368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:22.519570+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"625'20","reported_seq":1370,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.295767+0000","last_change":"2024-09-20T22:53:02.491402+0000","last_active":"2024-09-20T22:53:08.295767+0000","last_peered":"2024-09-20T22:53:08.295767+0000","last_clean":"2024-09-20T22:53:08.295767+0000","last_became_active":"2024-09-20T22:51:20.281859+0000","last_became_peered":"2024-09-20T22:51:20.281859+0000","last_unstale":"2024-09-20T22:53:08.295767+0000","last_undegraded":"2024-09-20T22:53:08.295767+0000","last_fullsized":"2024-09-20T22:53:08.295767+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'20","last_scrub_stamp":"2024-09-20T22:53:02.491320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:53:02.491320+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:10.839410+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":1345,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.290142+0000","last_change":"2024-09-20T22:52:20.621799+0000","last_active":"2024-09-20T22:53:08.290142+0000","last_peered":"2024-09-20T22:53:08.290142+0000","last_clean":"2024-09-20T22:53:08.290142+0000","last_became_active":"2024-09-20T22:48:05.240231+0000","last_became_peered":"2024-09-20T22:48:05.240231+0000","last_unstale":"2024-09-20T22:53:08.290142+0000","last_undegraded":"2024-09-20T22:53:08.290142+0000","last_fullsized":"2024-09-20T22:53:08.290142+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:20.621718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:52:20.621718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:30.275424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"600'11","reported_seq":1508,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.286174+0000","last_change":"2024-09-20T22:52:36.264515+0000","last_active":"2024-09-20T22:53:08.286174+0000","last_peered":"2024-09-20T22:53:08.286174+0000","last_clean":"2024-09-20T22:53:08.286174+0000","last_became_active":"2024-09-20T22:51:20.283393+0000","last_became_peered":"2024-09-20T22:51:20.283393+0000","last_unstale":"2024-09-20T22:53:08.286174+0000","last_undegraded":"2024-09-20T22:53:08.286174+0000","last_fullsized":"2024-09-20T22:53:08.286174+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"600'11","last_scrub_stamp":"2024-09-20T22:52:36.264410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:52:36.264410+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:52.246125+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":667,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.288688+0000","last_change":"2024-09-20T22:52:44.834418+0000","last_active":"2024-09-20T22:53:08.288688+0000","last_peered":"2024-09-20T22:53:08.288688+0000","last_clean":"2024-09-20T22:53:08.288688+0000","last_became_active":"2024-09-20T22:47:49.230674+0000","last_became_peered":"2024-09-20T22:47:49.230674+0000","last_unstale":"2024-09-20T22:53:08.288688+0000","last_undegraded":"2024-09-20T22:53:08.288688+0000","last_fullsized":"2024-09-20T22:53:08.288688+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:44.834352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:50:21.853070+0000","last_clean_scrub_stamp":"2024-09-20T22:52:44.834352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:57.888226+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"625'19","reported_seq":676,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.290133+0000","last_change":"2024-09-20T22:51:20.317969+0000","last_active":"2024-09-20T22:53:08.290133+0000","last_peered":"2024-09-20T22:53:08.290133+0000","last_clean":"2024-09-20T22:53:08.290133+0000","last_became_active":"2024-09-20T22:51:20.289584+0000","last_became_peered":"2024-09-20T22:51:20.289584+0000","last_unstale":"2024-09-20T22:53:08.290133+0000","last_undegraded":"2024-09-20T22:53:08.290133+0000","last_fullsized":"2024-09-20T22:53:08.290133+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-09-20T22:51:07.926382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:51:07.926382+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":642,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.292423+0000","last_change":"2024-09-20T22:52:47.717598+0000","last_active":"2024-09-20T22:53:08.292423+0000","last_peered":"2024-09-20T22:53:08.292423+0000","last_clean":"2024-09-20T22:53:08.292423+0000","last_became_active":"2024-09-20T22:48:05.252026+0000","last_became_peered":"2024-09-20T22:48:05.252026+0000","last_unstale":"2024-09-20T22:53:08.292423+0000","last_undegraded":"2024-09-20T22:53:08.292423+0000","last_fullsized":"2024-09-20T22:53:08.292423+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:47.717490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:52:47.717490+0000","last_clean_scrub_stamp":"2024-09-20T22:52:47.717490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:59.413281+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":228,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.285734+0000","last_change":"2024-09-20T22:51:20.291857+0000","last_active":"2024-09-20T22:53:08.285734+0000","last_peered":"2024-09-20T22:53:08.285734+0000","last_clean":"2024-09-20T22:53:08.285734+0000","last_became_active":"2024-09-20T22:51:20.291264+0000","last_became_peered":"2024-09-20T22:51:20.291264+0000","last_unstale":"2024-09-20T22:53:08.285734+0000","last_undegraded":"2024-09-20T22:53:08.285734+0000","last_fullsized":"2024-09-20T22:53:08.285734+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:06.651167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:51:06.651167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":1326,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.296192+0000","last_change":"2024-09-20T22:52:42.582474+0000","last_active":"2024-09-20T22:53:08.296192+0000","last_peered":"2024-09-20T22:53:08.296192+0000","last_clean":"2024-09-20T22:53:08.296192+0000","last_became_active":"2024-09-20T22:47:49.216141+0000","last_became_peered":"2024-09-20T22:47:49.216141+0000","last_unstale":"2024-09-20T22:53:08.296192+0000","last_undegraded":"2024-09-20T22:53:08.296192+0000","last_fullsized":"2024-09-20T22:53:08.296192+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:42.582398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:50:20.206984+0000","last_clean_scrub_stamp":"2024-09-20T22:52:42.582398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:57.745331+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"625'19","reported_seq":980,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.291624+0000","last_change":"2024-09-20T22:52:40.792541+0000","last_active":"2024-09-20T22:53:08.291624+0000","last_peered":"2024-09-20T22:53:08.291624+0000","last_clean":"2024-09-20T22:53:08.291624+0000","last_became_active":"2024-09-20T22:51:20.283868+0000","last_became_peered":"2024-09-20T22:51:20.283868+0000","last_unstale":"2024-09-20T22:53:08.291624+0000","last_undegraded":"2024-09-20T22:53:08.291624+0000","last_fullsized":"2024-09-20T22:53:08.291624+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'19","last_scrub_stamp":"2024-09-20T22:52:40.792457+0000","last_deep_scrub":"404'13","last_deep_scrub_stamp":"2024-09-20T22:48:04.276282+0000","last_clean_scrub_stamp":"2024-09-20T22:52:40.792457+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:55.252741+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,0],"acting":[6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":1360,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.290585+0000","last_change":"2024-09-20T22:52:42.505280+0000","last_active":"2024-09-20T22:53:08.290585+0000","last_peered":"2024-09-20T22:53:08.290585+0000","last_clean":"2024-09-20T22:53:08.290585+0000","last_became_active":"2024-09-20T22:48:05.238812+0000","last_became_peered":"2024-09-20T22:48:05.238812+0000","last_unstale":"2024-09-20T22:53:08.290585+0000","last_undegraded":"2024-09-20T22:53:08.290585+0000","last_fullsized":"2024-09-20T22:53:08.290585+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:42.505211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:31.138067+0000","last_clean_scrub_stamp":"2024-09-20T22:52:42.505211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:44.598851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":1328,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.298172+0000","last_change":"2024-09-20T22:51:20.288308+0000","last_active":"2024-09-20T22:53:08.298172+0000","last_peered":"2024-09-20T22:53:08.298172+0000","last_clean":"2024-09-20T22:53:08.298172+0000","last_became_active":"2024-09-20T22:51:20.288000+0000","last_became_peered":"2024-09-20T22:51:20.288000+0000","last_unstale":"2024-09-20T22:53:08.298172+0000","last_undegraded":"2024-09-20T22:53:08.298172+0000","last_fullsized":"2024-09-20T22:53:08.298172+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:50:44.313182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:50:44.313182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":665,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.286727+0000","last_change":"2024-09-20T22:51:20.278471+0000","last_active":"2024-09-20T22:53:08.286727+0000","last_peered":"2024-09-20T22:53:08.286727+0000","last_clean":"2024-09-20T22:53:08.286727+0000","last_became_active":"2024-09-20T22:51:20.278147+0000","last_became_peered":"2024-09-20T22:51:20.278147+0000","last_unstale":"2024-09-20T22:53:08.286727+0000","last_undegraded":"2024-09-20T22:53:08.286727+0000","last_fullsized":"2024-09-20T22:53:08.286727+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:50:35.895483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:50:35.895483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"625'24","reported_seq":1149,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.285379+0000","last_change":"2024-09-20T22:50:39.665781+0000","last_active":"2024-09-20T22:53:08.285379+0000","last_peered":"2024-09-20T22:53:08.285379+0000","last_clean":"2024-09-20T22:53:08.285379+0000","last_became_active":"2024-09-20T22:44:09.170785+0000","last_became_peered":"2024-09-20T22:44:09.170785+0000","last_unstale":"2024-09-20T22:53:08.285379+0000","last_undegraded":"2024-09-20T22:53:08.285379+0000","last_fullsized":"2024-09-20T22:53:08.285379+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"470'16","last_scrub_stamp":"2024-09-20T22:50:39.665682+0000","last_deep_scrub":"470'16","last_deep_scrub_stamp":"2024-09-20T22:49:10.659868+0000","last_clean_scrub_stamp":"2024-09-20T22:50:39.665682+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":1341,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.286544+0000","last_change":"2024-09-20T22:53:07.945728+0000","last_active":"2024-09-20T22:53:08.286544+0000","last_peered":"2024-09-20T22:53:08.286544+0000","last_clean":"2024-09-20T22:53:08.286544+0000","last_became_active":"2024-09-20T22:51:20.293645+0000","last_became_peered":"2024-09-20T22:51:20.293645+0000","last_unstale":"2024-09-20T22:53:08.286544+0000","last_undegraded":"2024-09-20T22:53:08.286544+0000","last_fullsized":"2024-09-20T22:53:08.286544+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:07.945664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:45:24.765414+0000","last_clean_scrub_stamp":"2024-09-20T22:53:07.945664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:21.914259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":1341,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.289376+0000","last_change":"2024-09-20T22:51:10.610718+0000","last_active":"2024-09-20T22:53:08.289376+0000","last_peered":"2024-09-20T22:53:08.289376+0000","last_clean":"2024-09-20T22:53:08.289376+0000","last_became_active":"2024-09-20T22:48:05.238334+0000","last_became_peered":"2024-09-20T22:48:05.238334+0000","last_unstale":"2024-09-20T22:53:08.289376+0000","last_undegraded":"2024-09-20T22:53:08.289376+0000","last_fullsized":"2024-09-20T22:53:08.289376+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:10.610588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:50.174861+0000","last_clean_scrub_stamp":"2024-09-20T22:51:10.610588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":1318,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.298329+0000","last_change":"2024-09-20T22:51:04.379662+0000","last_active":"2024-09-20T22:53:08.298329+0000","last_peered":"2024-09-20T22:53:08.298329+0000","last_clean":"2024-09-20T22:53:08.298329+0000","last_became_active":"2024-09-20T22:47:49.214487+0000","last_became_peered":"2024-09-20T22:47:49.214487+0000","last_unstale":"2024-09-20T22:53:08.298329+0000","last_undegraded":"2024-09-20T22:53:08.298329+0000","last_fullsized":"2024-09-20T22:53:08.298329+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:04.379558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:51:04.379558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"625'8","reported_seq":1346,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.295555+0000","last_change":"2024-09-20T22:53:05.489357+0000","last_active":"2024-09-20T22:53:08.295555+0000","last_peered":"2024-09-20T22:53:08.295555+0000","last_clean":"2024-09-20T22:53:08.295555+0000","last_became_active":"2024-09-20T22:51:20.290269+0000","last_became_peered":"2024-09-20T22:51:20.290269+0000","last_unstale":"2024-09-20T22:53:08.295555+0000","last_undegraded":"2024-09-20T22:53:08.295555+0000","last_fullsized":"2024-09-20T22:53:08.295555+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'8","last_scrub_stamp":"2024-09-20T22:53:05.489279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:53:05.489279+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:30.438834+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":675,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.289544+0000","last_change":"2024-09-20T22:53:07.918178+0000","last_active":"2024-09-20T22:53:08.289544+0000","last_peered":"2024-09-20T22:53:08.289544+0000","last_clean":"2024-09-20T22:53:08.289544+0000","last_became_active":"2024-09-20T22:47:49.231964+0000","last_became_peered":"2024-09-20T22:47:49.231964+0000","last_unstale":"2024-09-20T22:53:08.289544+0000","last_undegraded":"2024-09-20T22:53:08.289544+0000","last_fullsized":"2024-09-20T22:53:08.289544+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:07.918108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:53:07.918108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:20.150259+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":1358,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.290937+0000","last_change":"2024-09-20T22:52:20.955721+0000","last_active":"2024-09-20T22:53:08.290937+0000","last_peered":"2024-09-20T22:53:08.290937+0000","last_clean":"2024-09-20T22:53:08.290937+0000","last_became_active":"2024-09-20T22:48:05.244659+0000","last_became_peered":"2024-09-20T22:48:05.244659+0000","last_unstale":"2024-09-20T22:53:08.290937+0000","last_undegraded":"2024-09-20T22:53:08.290937+0000","last_fullsized":"2024-09-20T22:53:08.290937+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:20.955598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:52:20.955598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:32.677353+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":1341,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.289957+0000","last_change":"2024-09-20T22:53:05.612142+0000","last_active":"2024-09-20T22:53:08.289957+0000","last_peered":"2024-09-20T22:53:08.289957+0000","last_clean":"2024-09-20T22:53:08.289957+0000","last_became_active":"2024-09-20T22:48:05.237765+0000","last_became_peered":"2024-09-20T22:48:05.237765+0000","last_unstale":"2024-09-20T22:53:08.289957+0000","last_undegraded":"2024-09-20T22:53:08.289957+0000","last_fullsized":"2024-09-20T22:53:08.289957+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:05.612030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:10.233487+0000","last_clean_scrub_stamp":"2024-09-20T22:53:05.612030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:09.643188+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":931,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.286151+0000","last_change":"2024-09-20T22:51:29.904430+0000","last_active":"2024-09-20T22:53:08.286151+0000","last_peered":"2024-09-20T22:53:08.286151+0000","last_clean":"2024-09-20T22:53:08.286151+0000","last_became_active":"2024-09-20T22:51:20.297739+0000","last_became_peered":"2024-09-20T22:51:20.297739+0000","last_unstale":"2024-09-20T22:53:08.286151+0000","last_undegraded":"2024-09-20T22:53:08.286151+0000","last_fullsized":"2024-09-20T22:53:08.286151+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:29.904355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:51:29.904355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":1350,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.295322+0000","last_change":"2024-09-20T22:51:20.275893+0000","last_active":"2024-09-20T22:53:08.295322+0000","last_peered":"2024-09-20T22:53:08.295322+0000","last_clean":"2024-09-20T22:53:08.295322+0000","last_became_active":"2024-09-20T22:51:20.274891+0000","last_became_peered":"2024-09-20T22:51:20.274891+0000","last_unstale":"2024-09-20T22:53:08.295322+0000","last_undegraded":"2024-09-20T22:53:08.295322+0000","last_fullsized":"2024-09-20T22:53:08.295322+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:50:41.281470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:50:41.281470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"625'19","reported_seq":1391,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.293149+0000","last_change":"2024-09-20T22:52:16.953034+0000","last_active":"2024-09-20T22:53:08.293149+0000","last_peered":"2024-09-20T22:53:08.293149+0000","last_clean":"2024-09-20T22:53:08.293149+0000","last_became_active":"2024-09-20T22:51:20.284937+0000","last_became_peered":"2024-09-20T22:51:20.284937+0000","last_unstale":"2024-09-20T22:53:08.293149+0000","last_undegraded":"2024-09-20T22:53:08.293149+0000","last_fullsized":"2024-09-20T22:53:08.293149+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'19","last_scrub_stamp":"2024-09-20T22:52:16.952955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:52:16.952955+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:27.426198+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":667,"reported_epoch":708,"state":"active+clean","last_fresh":"2024-09-20T22:53:08.288784+0000","last_change":"2024-09-20T22:52:18.698154+0000","last_active":"2024-09-20T22:53:08.288784+0000","last_peered":"2024-09-20T22:53:08.288784+0000","last_clean":"2024-09-20T22:53:08.288784+0000","last_became_active":"2024-09-20T22:47:49.220971+0000","last_became_peered":"2024-09-20T22:47:49.220971+0000","last_unstale":"2024-09-20T22:53:08.288784+0000","last_undegraded":"2024-09-20T22:53:08.288784+0000","last_fullsized":"2024-09-20T22:53:08.288784+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:18.698087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:51.667341+0000","last_clean_scrub_stamp":"2024-09-20T22:52:18.698087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:47.971098+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":696539985,"num_objects":314,"num_object_clones":265,"num_object_copies":942,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":314,"num_whiteouts":13,"num_read":4873,"num_read_kb":5064896,"num_write":3558,"num_write_kb":1314105,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":790,"num_bytes_recovered":519843158,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1476657152,"data_stored":2199379794,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9967,"ondisk_log_size":9967,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85,"ondisk_log_size":85,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1109,"num_read_kb":866,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":312,"num_bytes_recovered":7464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":499712,"data_stored":13224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":744,"ondisk_log_size":744,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":1034,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":4931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":680,"num_read_kb":675,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1186,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":384,"ondisk_log_size":384,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070606,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":315080,"kb_used_data":265836,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":93430072,"statfs":{"total":95995035648,"available":95672393728,"internally_reserved":0,"allocated":272216064,"data_stored":367528106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":50395594},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430356,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":244992,"kb_used_data":195752,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":93500160,"statfs":{"total":95995035648,"available":95744163840,"internally_reserved":0,"allocated":200450048,"data_stored":294351817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798691993,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":204484,"kb_used_data":146928,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":93540668,"statfs":{"total":95995035648,"available":95785644032,"internally_reserved":0,"allocated":150454272,"data_stored":224054236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986334,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":136900,"kb_used_data":105712,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93608252,"statfs":{"total":95995035648,"available":95854850048,"internally_reserved":0,"allocated":108249088,"data_stored":163305840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182564,"num_pgs":76,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":372052,"kb_used_data":318652,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93373100,"statfs":{"total":95995035648,"available":95614054400,"internally_reserved":0,"allocated":326299648,"data_stored":480390430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411497,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":307232,"kb_used_data":262156,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":93437920,"statfs":{"total":95995035648,"available":95680430080,"internally_reserved":0,"allocated":268447744,"data_stored":392183892,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":55834575022,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738547,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":242180,"kb_used_data":192944,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":93502972,"statfs":{"total":95995035648,"available":95747043328,"internally_reserved":0,"allocated":197574656,"data_stored":306847614,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":191102976,"data_stored":302918559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":261742592,"data_stored":387992814,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319221760,"data_stored":475862726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":101777408,"data_stored":159370159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":143409152,"data_stored":219534274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":193974272,"data_stored":290422381,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":265428992,"data_stored":363278881,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-20T22:53:12.798 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-09-20T22:53:12.798 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-20T22:53:12.798 INFO:tasks.thrashosds.thrasher:inject_pause on osd.2 2024-09-20T22:53:12.798 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-20T22:53:12.798 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-20T22:53:12.799 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.2 config set filestore_inject_stall 3 2024-09-20T22:53:12.860 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:53:12.882 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:12.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:12.883 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:53:12.984 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:53:12.987 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:53:13.004 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:13.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:13.005 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:53:13.106 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:53:13.131 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:13.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:13.132 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:53:13.141 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:53:13.141 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:53:13.158 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:53:13.158 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:53:13.174 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:53:13.174 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:53:13.190 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-20T22:53:13.190 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T22:53:13.206 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:53:13.206 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:53:13.222 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:53:13.223 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:53:13.232 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:53:13.239 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:53:13.239 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:53:13.254 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:13.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:13.255 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:53:13.256 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:53:13.256 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:53:13.292 INFO:tasks.rados.rados.0.smithi084.stdout:2822: done (0 left) 2024-09-20T22:53:13.292 INFO:tasks.rados.rados.0.smithi084.stdout:2823: copy_from oid 38 from oid 23 current snap is 321 2024-09-20T22:53:13.292 INFO:tasks.rados.rados.0.smithi084.stdout:2824: read oid 15 snap -1 2024-09-20T22:53:13.292 INFO:tasks.rados.rados.0.smithi084.stdout:2824: expect (ObjNum 890 snap 289 seq_num 890) 2024-09-20T22:53:13.292 INFO:tasks.rados.rados.0.smithi084.stdout:2825: read oid 44 snap -1 2024-09-20T22:53:13.292 INFO:tasks.rados.rados.0.smithi084.stdout:2825: expect (ObjNum 879 snap 285 seq_num 879) 2024-09-20T22:53:13.292 INFO:tasks.rados.rados.0.smithi084.stdout:2826: read oid 7 snap -1 2024-09-20T22:53:13.292 INFO:tasks.rados.rados.0.smithi084.stdout:2826: expect (ObjNum 883 snap 285 seq_num 883) 2024-09-20T22:53:13.292 INFO:tasks.rados.rados.0.smithi084.stdout:2827: read oid 27 snap -1 2024-09-20T22:53:13.292 INFO:tasks.rados.rados.0.smithi084.stdout:2827: expect (ObjNum 898 snap 295 seq_num 898) 2024-09-20T22:53:13.292 INFO:tasks.rados.rados.0.smithi084.stdout:2828: rollback oid 33 current snap is 321 2024-09-20T22:53:13.292 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 33 to 248 2024-09-20T22:53:13.292 INFO:tasks.rados.rados.0.smithi084.stdout:2829: snap_create 2024-09-20T22:53:13.298 INFO:tasks.rados.rados.0.smithi084.stdout:2825: expect (ObjNum 879 snap 285 seq_num 879) 2024-09-20T22:53:13.355 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:53:13.376 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:13.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:13.377 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:53:13.401 INFO:tasks.rados.rados.0.smithi084.stdout:2826: expect (ObjNum 883 snap 285 seq_num 883) 2024-09-20T22:53:13.478 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:53:13.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:13.500 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:13.500 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:53:13.566 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-20T22:53:13.566 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:53:13.600 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:53:13.616 INFO:tasks.rados.rados.0.smithi084.stdout:2824: expect (ObjNum 890 snap 289 seq_num 890) 2024-09-20T22:53:13.621 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:13.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:13.622 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:53:13.715 INFO:tasks.rados.rados.0.smithi084.stdout:2827: expect (ObjNum 898 snap 295 seq_num 898) 2024-09-20T22:53:13.723 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:53:13.744 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:13.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:13.745 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:53:13.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:13 smithi154 ceph-mon[33264]: pgmap v969: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 665 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail; 14 MiB/s rd, 2.9 MiB/s wr, 28 op/s 2024-09-20T22:53:13.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:13 smithi154 ceph-mon[33264]: from='client.16779 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T22:53:13.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:13 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/1055007476' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:53:13.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:13 smithi154 ceph-mon[33264]: osdmap e713: 8 total, 8 up, 7 in 2024-09-20T22:53:13.768 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:13 smithi084 ceph-mon[29774]: pgmap v969: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 665 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail; 14 MiB/s rd, 2.9 MiB/s wr, 28 op/s 2024-09-20T22:53:13.768 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:13 smithi084 ceph-mon[29774]: from='client.16779 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T22:53:13.768 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:13 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/1055007476' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:53:13.768 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:13 smithi084 ceph-mon[29774]: osdmap e713: 8 total, 8 up, 7 in 2024-09-20T22:53:13.768 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:13 smithi084 ceph-mon[40024]: pgmap v969: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 665 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail; 14 MiB/s rd, 2.9 MiB/s wr, 28 op/s 2024-09-20T22:53:13.768 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:13 smithi084 ceph-mon[40024]: from='client.16779 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T22:53:13.768 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:13 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/1055007476' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:53:13.768 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:13 smithi084 ceph-mon[40024]: osdmap e713: 8 total, 8 up, 7 in 2024-09-20T22:53:13.846 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:53:13.854 INFO:tasks.rados.rados.0.smithi084.stdout:2828: finishing rollback tid 0 to smithi08477203-33 2024-09-20T22:53:13.854 INFO:tasks.rados.rados.0.smithi084.stdout:2828: finishing rollback tid 1 to smithi08477203-33 2024-09-20T22:53:13.854 INFO:tasks.rados.rados.0.smithi084.stdout:2828: finishing rollback tid 2 to smithi08477203-33 2024-09-20T22:53:13.855 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 33 v 1261 (ObjNum 738 snap 235 seq_num 738) dirty exists 2024-09-20T22:53:13.855 INFO:tasks.rados.rados.0.smithi084.stdout:2823: finishing copy_from racing read to smithi08477203-38 2024-09-20T22:53:13.855 INFO:tasks.rados.rados.0.smithi084.stdout:2823: finishing copy_from to smithi08477203-38 2024-09-20T22:53:13.855 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 38 v 1252 (ObjNum 957 snap 313 seq_num 957) dirty exists 2024-09-20T22:53:13.866 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:13.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:13.867 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:53:13.968 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:53:13.991 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:13.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:13.992 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:53:14.093 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:53:14.114 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:14.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:14.115 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:53:14.216 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:53:14.238 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:14.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:14.239 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:53:14.292 INFO:tasks.rados.rados.0.smithi084.stdout:2823: done (6 left) 2024-09-20T22:53:14.292 INFO:tasks.rados.rados.0.smithi084.stdout:2824: done (5 left) 2024-09-20T22:53:14.292 INFO:tasks.rados.rados.0.smithi084.stdout:2825: done (4 left) 2024-09-20T22:53:14.292 INFO:tasks.rados.rados.0.smithi084.stdout:2826: done (3 left) 2024-09-20T22:53:14.292 INFO:tasks.rados.rados.0.smithi084.stdout:2827: done (2 left) 2024-09-20T22:53:14.292 INFO:tasks.rados.rados.0.smithi084.stdout:2828: done (1 left) 2024-09-20T22:53:14.293 INFO:tasks.rados.rados.0.smithi084.stdout:2829: done (0 left) 2024-09-20T22:53:14.293 INFO:tasks.rados.rados.0.smithi084.stdout:2830: snap_create 2024-09-20T22:53:14.333 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-20T22:53:14.340 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:53:14.396 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:14.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:14.397 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:53:14.498 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:53:14.520 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:14.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:14.521 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:53:14.578 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:53:14.584 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:53:14.622 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:53:14.643 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:14.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:14.644 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:53:14.698 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:14 smithi084 ceph-mon[29774]: purged_snaps scrub starts 2024-09-20T22:53:14.698 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:14 smithi084 ceph-mon[29774]: purged_snaps scrub ok 2024-09-20T22:53:14.698 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:14 smithi084 ceph-mon[29774]: osdmap e714: 8 total, 8 up, 7 in 2024-09-20T22:53:14.699 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:14 smithi084 ceph-mon[40024]: purged_snaps scrub starts 2024-09-20T22:53:14.699 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:14 smithi084 ceph-mon[40024]: purged_snaps scrub ok 2024-09-20T22:53:14.699 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:14 smithi084 ceph-mon[40024]: osdmap e714: 8 total, 8 up, 7 in 2024-09-20T22:53:14.705 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:14 smithi154 ceph-mon[33264]: purged_snaps scrub starts 2024-09-20T22:53:14.706 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:14 smithi154 ceph-mon[33264]: purged_snaps scrub ok 2024-09-20T22:53:14.706 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:14 smithi154 ceph-mon[33264]: osdmap e714: 8 total, 8 up, 7 in 2024-09-20T22:53:14.745 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:53:14.766 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:14.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:14.767 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:53:14.868 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:53:14.889 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:14.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:14.890 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:53:14.991 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:53:15.012 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:15.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:15.013 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:53:15.113 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:53:15.118 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-20T22:53:15.118 INFO:teuthology.orchestra.run.smithi084.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-20T22:53:15.118 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-20T22:53:15.136 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:15.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:15.136 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:53:15.185 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-20T22:53:15.185 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:53:15.237 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:53:15.258 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:15.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:15.259 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:53:15.292 INFO:tasks.rados.rados.0.smithi084.stdout:2830: done (0 left) 2024-09-20T22:53:15.292 INFO:tasks.rados.rados.0.smithi084.stdout:2831: write (excl) oid 48 current snap is 323 2024-09-20T22:53:15.292 INFO:tasks.rados.rados.0.smithi084.stdout:2831: seq_num 985 ranges {554895=729802,1773234=778333} 2024-09-20T22:53:15.299 INFO:tasks.rados.rados.0.smithi084.stdout:2831: writing smithi08477203-48 from 554895 to 1284697 tid 1 2024-09-20T22:53:15.306 INFO:tasks.rados.rados.0.smithi084.stdout:2831: writing smithi08477203-48 from 1773234 to 2551567 tid 2 2024-09-20T22:53:15.306 INFO:tasks.rados.rados.0.smithi084.stdout:2832: snap_remove snap 309 2024-09-20T22:53:15.360 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:53:15.381 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:15.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:15.382 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:53:15.482 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:53:15.503 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:15.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:15.504 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:53:15.605 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:53:15.626 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:15.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:15.627 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:53:15.728 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:53:15.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:15.751 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:15.752 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:53:15.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:15 smithi154 ceph-mon[33264]: pgmap v972: 158 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 153 active+clean; 621 MiB data, 1.6 GiB used, 624 GiB / 626 GiB avail; 14 MiB/s rd, 824 KiB/s wr, 12 op/s 2024-09-20T22:53:15.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:15 smithi154 ceph-mon[33264]: osdmap e715: 8 total, 8 up, 7 in 2024-09-20T22:53:15.772 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:15 smithi084 ceph-mon[40024]: pgmap v972: 158 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 153 active+clean; 621 MiB data, 1.6 GiB used, 624 GiB / 626 GiB avail; 14 MiB/s rd, 824 KiB/s wr, 12 op/s 2024-09-20T22:53:15.772 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:15 smithi084 ceph-mon[40024]: osdmap e715: 8 total, 8 up, 7 in 2024-09-20T22:53:15.772 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:15 smithi084 ceph-mon[29774]: pgmap v972: 158 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 153 active+clean; 621 MiB data, 1.6 GiB used, 624 GiB / 626 GiB avail; 14 MiB/s rd, 824 KiB/s wr, 12 op/s 2024-09-20T22:53:15.772 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:15 smithi084 ceph-mon[29774]: osdmap e715: 8 total, 8 up, 7 in 2024-09-20T22:53:15.795 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-20T22:53:15.853 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:53:15.874 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:15.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:15.875 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:53:15.976 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:53:15.998 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:15.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:15.999 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.7 dump_blocked_ops 2024-09-20T22:53:16.000 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:53:16.040 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:53:16.101 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:53:16.122 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:16.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:16.123 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:53:16.224 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:53:16.245 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:16.245 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:53:16.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:16.246 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:53:16.333 INFO:tasks.rados.rados.0.smithi084.stdout:2832: done (1 left) 2024-09-20T22:53:16.333 INFO:tasks.rados.rados.0.smithi084.stdout:2833: snap_remove snap 321 2024-09-20T22:53:16.348 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:53:16.369 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:16.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:16.370 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:53:16.471 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:53:16.492 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:16.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:16.493 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:53:16.520 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-20T22:53:16.594 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:53:16.615 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:16.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:16.616 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:53:16.717 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:53:16.726 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-20T22:53:16.726 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:53:16.740 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:16.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:16.741 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:53:16.767 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:53:16.842 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:53:16.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:16.864 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:16.864 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:53:16.964 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:53:16.986 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:16.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:16.987 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:53:17.088 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:53:17.110 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:17.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:17.111 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:53:17.212 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:53:17.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:17.235 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:17.236 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:53:17.238 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:53:17.238 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:53:17.255 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:53:17.256 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:53:17.271 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:53:17.271 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:53:17.287 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-20T22:53:17.287 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T22:53:17.299 INFO:tasks.rados.rados.0.smithi084.stdout:2833: done (1 left) 2024-09-20T22:53:17.299 INFO:tasks.rados.rados.0.smithi084.stdout:2834: snap_remove snap 192 2024-09-20T22:53:17.301 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is set 2024-09-20T22:53:17.309 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:53:17.309 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:53:17.331 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:53:17.331 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:53:17.337 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:53:17.349 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:53:17.349 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:53:17.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:17.359 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:17.359 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:53:17.366 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:53:17.366 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:53:17.460 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:53:17.472 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:17 smithi154 ceph-mon[33264]: pgmap v975: 158 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 153 active+clean; 621 MiB data, 1.6 GiB used, 624 GiB / 626 GiB avail; 14 MiB/s rd, 824 KiB/s wr, 12 op/s 2024-09-20T22:53:17.472 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:17 smithi154 ceph-mon[33264]: osdmap e716: 8 total, 8 up, 7 in 2024-09-20T22:53:17.472 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:17 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/1152333925' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:53:17.472 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:17 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:53:17.472 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:17 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/831563150' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:53:17.482 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:17.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:17.483 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:53:17.500 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.7 dump_historic_ops 2024-09-20T22:53:17.504 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:17 smithi084 ceph-mon[40024]: pgmap v975: 158 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 153 active+clean; 621 MiB data, 1.6 GiB used, 624 GiB / 626 GiB avail; 14 MiB/s rd, 824 KiB/s wr, 12 op/s 2024-09-20T22:53:17.505 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:17 smithi084 ceph-mon[40024]: osdmap e716: 8 total, 8 up, 7 in 2024-09-20T22:53:17.505 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:17 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/1152333925' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:53:17.505 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:17 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:53:17.505 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:17 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/831563150' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:53:17.505 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:17 smithi084 ceph-mon[29774]: pgmap v975: 158 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 153 active+clean; 621 MiB data, 1.6 GiB used, 624 GiB / 626 GiB avail; 14 MiB/s rd, 824 KiB/s wr, 12 op/s 2024-09-20T22:53:17.505 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:17 smithi084 ceph-mon[29774]: osdmap e716: 8 total, 8 up, 7 in 2024-09-20T22:53:17.505 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:17 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/1152333925' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:53:17.505 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:17 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:53:17.505 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:17 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/831563150' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:53:17.584 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:53:17.605 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:17.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:17.606 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:53:17.706 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:53:17.729 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:17.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:17.730 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:53:17.788 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:53:17.831 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:53:17.852 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:17.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:17.853 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:53:17.954 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:53:17.975 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:17.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:17.976 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:53:18.077 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:53:18.099 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:18.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:18.100 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:53:18.201 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:53:18.222 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:18.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:18.223 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:53:18.270 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-20T22:53:18.270 INFO:teuthology.orchestra.run.smithi154.stdout: "size": 20, 2024-09-20T22:53:18.270 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 600, 2024-09-20T22:53:18.270 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [ 2024-09-20T22:53:18.270 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.270 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:786 7.17 7:ed54065f:::smithi08477203-12:head [copy-from ver 61] snapc 19=[17] ondisk+write+known_if_redirected+supports_pool_eio e145)", 2024-09-20T22:53:18.270 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:44.333196+0000", 2024-09-20T22:53:18.270 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 573.93388948899997, 2024-09-20T22:53:18.271 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.023328503, 2024-09-20T22:53:18.271 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:53:18.271 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:53:18.271 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:53:18.271 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:53:18.271 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:53:18.271 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:53:18.271 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 786 2024-09-20T22:53:18.271 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.271 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:53:18.271 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.271 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:53:18.271 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.333196+0000", 2024-09-20T22:53:18.272 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:53:18.272 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.272 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.272 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:53:18.272 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.333196+0000", 2024-09-20T22:53:18.272 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:53:18.272 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.272 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.272 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:53:18.272 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.333197+0000", 2024-09-20T22:53:18.272 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.486e-06 2024-09-20T22:53:18.272 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.273 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.273 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:53:18.273 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.333207+0000", 2024-09-20T22:53:18.273 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0307e-05 2024-09-20T22:53:18.273 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.273 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.273 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:53:18.273 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.333220+0000", 2024-09-20T22:53:18.273 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.306e-05 2024-09-20T22:53:18.273 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.273 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.273 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:53:18.273 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.333229+0000", 2024-09-20T22:53:18.273 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.3450000000000006e-06 2024-09-20T22:53:18.274 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.274 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.274 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:53:18.274 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.333292+0000", 2024-09-20T22:53:18.274 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.3137000000000002e-05 2024-09-20T22:53:18.274 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.274 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.274 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op must wait for map", 2024-09-20T22:53:18.274 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.333298+0000", 2024-09-20T22:53:18.274 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.7780000000000004e-06 2024-09-20T22:53:18.274 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.274 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.274 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:53:18.274 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.336639+0000", 2024-09-20T22:53:18.275 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0033414479999999999 2024-09-20T22:53:18.275 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.275 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.275 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:53:18.275 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.336729+0000", 2024-09-20T22:53:18.275 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.9888000000000004e-05 2024-09-20T22:53:18.275 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.275 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.275 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 2,5", 2024-09-20T22:53:18.275 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.344876+0000", 2024-09-20T22:53:18.275 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0081473989999999996 2024-09-20T22:53:18.275 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.275 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.276 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:53:18.276 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.355138+0000", 2024-09-20T22:53:18.276 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.010261462000000001 2024-09-20T22:53:18.276 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.276 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.276 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:53:18.276 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.355972+0000", 2024-09-20T22:53:18.276 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00083377899999999999 2024-09-20T22:53:18.276 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.276 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.276 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:53:18.276 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.356451+0000", 2024-09-20T22:53:18.276 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00047985200000000001 2024-09-20T22:53:18.276 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.277 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.277 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:53:18.277 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.356486+0000", 2024-09-20T22:53:18.277 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.4653000000000002e-05 2024-09-20T22:53:18.277 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.277 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.277 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:53:18.277 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.356524+0000", 2024-09-20T22:53:18.277 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.7908999999999999e-05 2024-09-20T22:53:18.277 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:53:18.277 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:53:18.277 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:53:18.277 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.277 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.277 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:787 7.17 7:ed54065f:::smithi08477203-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e145)", 2024-09-20T22:53:18.277 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:43:44.333248+0000", 2024-09-20T22:53:18.278 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 573.93383739399997, 2024-09-20T22:53:18.278 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.016881184, 2024-09-20T22:53:18.278 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:53:18.278 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:53:18.278 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:53:18.278 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:53:18.278 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:53:18.278 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:53:18.278 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 787 2024-09-20T22:53:18.278 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.278 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:53:18.278 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.278 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:53:18.278 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.333248+0000", 2024-09-20T22:53:18.278 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:53:18.279 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.279 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.279 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:53:18.279 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.333248+0000", 2024-09-20T22:53:18.279 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:53:18.279 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.279 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.279 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:53:18.279 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.333248+0000", 2024-09-20T22:53:18.279 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.1399999999999997e-07 2024-09-20T22:53:18.279 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.279 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.279 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:53:18.279 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.333250+0000", 2024-09-20T22:53:18.280 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.8199999999999999e-06 2024-09-20T22:53:18.280 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.280 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.280 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:53:18.280 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.333257+0000", 2024-09-20T22:53:18.280 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.43e-06 2024-09-20T22:53:18.280 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.280 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.280 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:53:18.280 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.333261+0000", 2024-09-20T22:53:18.280 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.3670000000000001e-06 2024-09-20T22:53:18.280 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.280 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.280 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:53:18.281 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.333351+0000", 2024-09-20T22:53:18.281 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.0486000000000001e-05 2024-09-20T22:53:18.281 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.281 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.281 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting_for_map not empty", 2024-09-20T22:53:18.281 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.333357+0000", 2024-09-20T22:53:18.281 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.3520000000000003e-06 2024-09-20T22:53:18.281 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.281 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.281 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:53:18.281 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.336912+0000", 2024-09-20T22:53:18.281 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.003554612 2024-09-20T22:53:18.281 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.281 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.282 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for blocked object", 2024-09-20T22:53:18.282 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.336957+0000", 2024-09-20T22:53:18.282 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.4919000000000001e-05 2024-09-20T22:53:18.282 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.282 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.282 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:53:18.282 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.349989+0000", 2024-09-20T22:53:18.282 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.01303238 2024-09-20T22:53:18.282 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.282 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.282 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:53:18.282 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.350058+0000", 2024-09-20T22:53:18.282 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.8418999999999997e-05 2024-09-20T22:53:18.282 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.283 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.283 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:53:18.283 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:43:44.350129+0000", 2024-09-20T22:53:18.283 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.0984999999999999e-05 2024-09-20T22:53:18.283 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:53:18.283 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:53:18.283 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:53:18.283 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.283 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.283 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:1527 7.17 7:ea720b85:::smithi08477203-26:head [copy-from ver 83] snapc 33=[33] ondisk+write+known_if_redirected+supports_pool_eio e197)", 2024-09-20T22:53:18.283 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:44:36.198921+0000", 2024-09-20T22:53:18.283 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 522.06816423500004, 2024-09-20T22:53:18.284 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.021621703999999999, 2024-09-20T22:53:18.284 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:53:18.284 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:53:18.284 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:53:18.284 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:53:18.285 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:53:18.285 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:53:18.285 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 1527 2024-09-20T22:53:18.285 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.285 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:53:18.285 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.285 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:53:18.285 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.198921+0000", 2024-09-20T22:53:18.285 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:53:18.285 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.285 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.286 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:53:18.286 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.198921+0000", 2024-09-20T22:53:18.286 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:53:18.286 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.286 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.286 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:53:18.286 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.198923+0000", 2024-09-20T22:53:18.286 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.0080000000000001e-06 2024-09-20T22:53:18.286 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.286 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.286 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:53:18.286 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.198934+0000", 2024-09-20T22:53:18.286 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0923e-05 2024-09-20T22:53:18.286 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.287 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.287 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:53:18.287 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.198955+0000", 2024-09-20T22:53:18.287 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.0806e-05 2024-09-20T22:53:18.287 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.287 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.287 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:53:18.287 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.198966+0000", 2024-09-20T22:53:18.287 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1779e-05 2024-09-20T22:53:18.287 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.287 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.287 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:53:18.287 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.199051+0000", 2024-09-20T22:53:18.287 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.4357999999999994e-05 2024-09-20T22:53:18.288 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.288 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.288 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op must wait for map", 2024-09-20T22:53:18.288 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.199058+0000", 2024-09-20T22:53:18.288 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.8530000000000003e-06 2024-09-20T22:53:18.288 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.288 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.288 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:53:18.288 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.202420+0000", 2024-09-20T22:53:18.288 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0033623350000000002 2024-09-20T22:53:18.288 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.288 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.288 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:53:18.288 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.202498+0000", 2024-09-20T22:53:18.288 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.8316e-05 2024-09-20T22:53:18.289 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.289 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.289 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 2,5", 2024-09-20T22:53:18.289 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.209140+0000", 2024-09-20T22:53:18.289 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0066415759999999997 2024-09-20T22:53:18.289 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.289 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.289 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:53:18.289 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.214931+0000", 2024-09-20T22:53:18.289 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0057916520000000004 2024-09-20T22:53:18.289 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.289 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.289 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:53:18.289 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.220009+0000", 2024-09-20T22:53:18.290 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0050773750000000003 2024-09-20T22:53:18.290 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.290 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.290 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:53:18.290 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.220417+0000", 2024-09-20T22:53:18.290 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00040778599999999999 2024-09-20T22:53:18.290 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.290 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.290 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:53:18.290 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.220479+0000", 2024-09-20T22:53:18.290 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.1977000000000006e-05 2024-09-20T22:53:18.290 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.290 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.290 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:53:18.291 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.220542+0000", 2024-09-20T22:53:18.291 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.3960000000000004e-05 2024-09-20T22:53:18.291 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:53:18.291 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:53:18.292 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:53:18.292 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.292 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.292 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:1528 7.17 7:ea720b85:::smithi08477203-26:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e197)", 2024-09-20T22:53:18.292 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:44:36.198994+0000", 2024-09-20T22:53:18.292 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 522.06809150799995, 2024-09-20T22:53:18.292 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.015013924, 2024-09-20T22:53:18.292 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:53:18.292 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:53:18.292 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:53:18.292 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:53:18.292 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:53:18.292 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:53:18.292 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 1528 2024-09-20T22:53:18.293 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.293 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:53:18.293 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.293 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:53:18.293 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.198994+0000", 2024-09-20T22:53:18.293 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:53:18.293 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.293 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.293 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:53:18.293 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.198994+0000", 2024-09-20T22:53:18.293 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:53:18.293 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.293 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.293 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:53:18.294 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.198994+0000", 2024-09-20T22:53:18.294 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.8100000000000003e-07 2024-09-20T22:53:18.294 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.294 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.294 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:53:18.294 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.198996+0000", 2024-09-20T22:53:18.294 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.3e-06 2024-09-20T22:53:18.294 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.294 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.294 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:53:18.294 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.199008+0000", 2024-09-20T22:53:18.294 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1603999999999999e-05 2024-09-20T22:53:18.294 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.294 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.295 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:53:18.295 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.199013+0000", 2024-09-20T22:53:18.295 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.4500000000000003e-06 2024-09-20T22:53:18.295 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.295 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.295 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:53:18.295 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.199120+0000", 2024-09-20T22:53:18.295 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000106772 2024-09-20T22:53:18.295 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.295 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.295 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting_for_map not empty", 2024-09-20T22:53:18.295 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.199126+0000", 2024-09-20T22:53:18.295 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.8320000000000002e-06 2024-09-20T22:53:18.295 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.295 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.296 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:53:18.296 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.202625+0000", 2024-09-20T22:53:18.296 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0034986309999999999 2024-09-20T22:53:18.296 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.296 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.296 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for blocked object", 2024-09-20T22:53:18.296 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.202667+0000", 2024-09-20T22:53:18.296 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.2759000000000002e-05 2024-09-20T22:53:18.296 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.296 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.296 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:53:18.296 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.213787+0000", 2024-09-20T22:53:18.296 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.011119377 2024-09-20T22:53:18.296 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.297 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.297 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:53:18.297 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.213902+0000", 2024-09-20T22:53:18.297 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000115369 2024-09-20T22:53:18.297 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.297 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.297 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:53:18.297 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.214007+0000", 2024-09-20T22:53:18.297 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000105349 2024-09-20T22:53:18.297 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:53:18.297 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:53:18.297 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:53:18.297 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.297 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.297 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:1550 7.17 7:ed54065f:::smithi08477203-12:head [copy-from ver 88] snapc 34=[34,33] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2024-09-20T22:53:18.297 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:44:37.238376+0000", 2024-09-20T22:53:18.298 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 521.028708596, 2024-09-20T22:53:18.298 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.016634917999999999, 2024-09-20T22:53:18.298 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:53:18.298 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:53:18.298 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:53:18.298 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:53:18.298 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:53:18.298 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:53:18.298 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 1550 2024-09-20T22:53:18.298 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.298 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:53:18.298 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.298 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:53:18.298 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:37.238376+0000", 2024-09-20T22:53:18.298 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:53:18.299 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.299 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.299 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:53:18.299 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:37.238376+0000", 2024-09-20T22:53:18.299 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:53:18.299 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.299 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.299 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:53:18.299 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:37.238379+0000", 2024-09-20T22:53:18.299 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.7989999999999998e-06 2024-09-20T22:53:18.299 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.299 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.299 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:53:18.299 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:37.238408+0000", 2024-09-20T22:53:18.299 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.8980999999999999e-05 2024-09-20T22:53:18.300 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.300 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.300 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:53:18.300 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:37.238433+0000", 2024-09-20T22:53:18.300 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.5185999999999999e-05 2024-09-20T22:53:18.300 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.300 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.300 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:53:18.300 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:37.238446+0000", 2024-09-20T22:53:18.300 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2883e-05 2024-09-20T22:53:18.300 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.300 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.300 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:53:18.300 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:37.238523+0000", 2024-09-20T22:53:18.301 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.6396999999999994e-05 2024-09-20T22:53:18.301 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.301 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.301 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:53:18.301 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:37.238618+0000", 2024-09-20T22:53:18.301 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.5049000000000004e-05 2024-09-20T22:53:18.301 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.301 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.301 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 2,5", 2024-09-20T22:53:18.301 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:37.246556+0000", 2024-09-20T22:53:18.301 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0079378090000000005 2024-09-20T22:53:18.301 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.301 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.301 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:53:18.302 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:37.254025+0000", 2024-09-20T22:53:18.302 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0074697139999999997 2024-09-20T22:53:18.302 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.302 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.302 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:53:18.302 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:37.254033+0000", 2024-09-20T22:53:18.302 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.6979999999999998e-06 2024-09-20T22:53:18.302 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.302 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.302 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:53:18.302 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:37.254919+0000", 2024-09-20T22:53:18.302 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00088564700000000004 2024-09-20T22:53:18.303 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.303 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.303 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:53:18.303 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:37.254958+0000", 2024-09-20T22:53:18.303 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.9515999999999999e-05 2024-09-20T22:53:18.303 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.303 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.303 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:53:18.303 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:37.255011+0000", 2024-09-20T22:53:18.303 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.3239000000000003e-05 2024-09-20T22:53:18.303 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:53:18.303 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:53:18.303 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:53:18.303 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.304 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.304 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:1799 7.17 7:ea720b85:::smithi08477203-26:head [write 521083~714336,stat] snapc 46=[45,44,43,42,41,40,3e] ondisk+write+known_if_redirected+supports_pool_eio e229)", 2024-09-20T22:53:18.304 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:45:08.170663+0000", 2024-09-20T22:53:18.304 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 490.09642195100002, 2024-09-20T22:53:18.304 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.016368226999999999, 2024-09-20T22:53:18.304 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:53:18.304 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:53:18.304 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:53:18.304 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:53:18.304 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:53:18.304 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:53:18.304 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 1799 2024-09-20T22:53:18.304 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.305 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:53:18.305 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.305 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:53:18.305 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.170663+0000", 2024-09-20T22:53:18.305 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:53:18.305 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.305 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.305 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:53:18.305 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.170663+0000", 2024-09-20T22:53:18.305 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:53:18.305 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.305 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.305 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:53:18.305 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.170664+0000", 2024-09-20T22:53:18.306 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.085e-06 2024-09-20T22:53:18.306 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.306 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.306 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:53:18.306 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.171481+0000", 2024-09-20T22:53:18.306 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00081653900000000002 2024-09-20T22:53:18.306 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.306 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.306 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:53:18.306 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.171492+0000", 2024-09-20T22:53:18.306 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1565e-05 2024-09-20T22:53:18.306 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.306 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.307 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:53:18.307 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.171498+0000", 2024-09-20T22:53:18.307 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.925e-06 2024-09-20T22:53:18.307 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.307 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.307 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:53:18.307 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.171547+0000", 2024-09-20T22:53:18.307 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.8916999999999999e-05 2024-09-20T22:53:18.307 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.307 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.307 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:53:18.307 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.171611+0000", 2024-09-20T22:53:18.307 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.3838999999999996e-05 2024-09-20T22:53:18.307 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.308 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.308 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 2,5", 2024-09-20T22:53:18.308 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.171740+0000", 2024-09-20T22:53:18.308 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00012881500000000001 2024-09-20T22:53:18.308 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.308 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.308 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:53:18.308 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.173862+0000", 2024-09-20T22:53:18.308 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0021225649999999999 2024-09-20T22:53:18.308 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.308 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.308 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:53:18.308 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.179252+0000", 2024-09-20T22:53:18.309 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0053895490000000004 2024-09-20T22:53:18.309 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.309 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.309 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:53:18.309 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.186956+0000", 2024-09-20T22:53:18.309 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0077038050000000002 2024-09-20T22:53:18.309 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.309 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.309 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:53:18.309 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.186993+0000", 2024-09-20T22:53:18.309 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.7141000000000002e-05 2024-09-20T22:53:18.309 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.309 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.309 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:53:18.309 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:08.187031+0000", 2024-09-20T22:53:18.309 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.8482000000000002e-05 2024-09-20T22:53:18.309 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:53:18.310 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:53:18.310 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:53:18.310 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.310 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.310 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:2318 7.5 7:a160b5d4:::smithi08477203-16:head [copy-from ver 168] snapc 55=[55,53,52,51,50,4b,47,3e] ondisk+write+known_if_redirected+supports_pool_eio e258)", 2024-09-20T22:53:18.310 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:45:37.175863+0000", 2024-09-20T22:53:18.310 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 461.09122187600002, 2024-09-20T22:53:18.310 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.030719883999999999, 2024-09-20T22:53:18.310 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:53:18.310 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:53:18.310 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:53:18.310 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:53:18.310 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:53:18.310 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:53:18.310 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 2318 2024-09-20T22:53:18.311 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.311 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:53:18.311 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.311 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:53:18.311 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.175863+0000", 2024-09-20T22:53:18.311 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:53:18.311 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.311 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.311 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:53:18.311 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.175863+0000", 2024-09-20T22:53:18.311 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:53:18.311 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.311 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.311 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:53:18.311 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.175865+0000", 2024-09-20T22:53:18.311 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.488e-06 2024-09-20T22:53:18.312 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.312 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.312 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:53:18.312 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.175871+0000", 2024-09-20T22:53:18.312 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.438e-06 2024-09-20T22:53:18.312 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.312 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.312 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:53:18.312 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.175881+0000", 2024-09-20T22:53:18.312 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.6430000000000007e-06 2024-09-20T22:53:18.312 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.312 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.312 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:53:18.312 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.175886+0000", 2024-09-20T22:53:18.313 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.3059999999999997e-06 2024-09-20T22:53:18.313 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.313 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.313 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:53:18.313 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.175944+0000", 2024-09-20T22:53:18.313 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.7982999999999997e-05 2024-09-20T22:53:18.313 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.313 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.313 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op must wait for map", 2024-09-20T22:53:18.313 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.175951+0000", 2024-09-20T22:53:18.313 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.9280000000000004e-06 2024-09-20T22:53:18.313 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.313 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.313 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:53:18.313 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.179794+0000", 2024-09-20T22:53:18.313 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0038435209999999999 2024-09-20T22:53:18.313 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.314 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.314 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:53:18.314 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.179940+0000", 2024-09-20T22:53:18.314 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00014550000000000001 2024-09-20T22:53:18.314 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.314 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.314 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 0,2", 2024-09-20T22:53:18.314 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.190992+0000", 2024-09-20T22:53:18.314 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.011051587999999999 2024-09-20T22:53:18.314 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.314 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.314 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:53:18.314 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.195070+0000", 2024-09-20T22:53:18.314 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0040784929999999999 2024-09-20T22:53:18.314 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.315 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.315 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:53:18.315 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.202908+0000", 2024-09-20T22:53:18.315 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0078376690000000002 2024-09-20T22:53:18.315 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.315 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.315 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:53:18.315 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.206517+0000", 2024-09-20T22:53:18.315 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0036088880000000002 2024-09-20T22:53:18.315 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.315 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.315 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:53:18.315 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.206553+0000", 2024-09-20T22:53:18.315 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.6363999999999999e-05 2024-09-20T22:53:18.316 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.316 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.316 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:53:18.316 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.206583+0000", 2024-09-20T22:53:18.316 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.0074999999999999e-05 2024-09-20T22:53:18.316 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:53:18.316 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:53:18.316 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:53:18.316 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.316 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.316 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:2319 7.5 7:a160b5d4:::smithi08477203-16:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e258)", 2024-09-20T22:53:18.316 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:45:37.175983+0000", 2024-09-20T22:53:18.316 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 461.09110212100001, 2024-09-20T22:53:18.316 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.018323547999999999, 2024-09-20T22:53:18.316 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:53:18.316 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:53:18.317 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:53:18.317 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:53:18.317 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:53:18.317 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:53:18.317 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 2319 2024-09-20T22:53:18.317 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.317 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:53:18.317 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.317 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:53:18.317 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.175983+0000", 2024-09-20T22:53:18.317 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:53:18.317 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.317 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.317 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:53:18.318 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.175983+0000", 2024-09-20T22:53:18.318 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:53:18.318 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.318 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.318 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:53:18.318 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.175984+0000", 2024-09-20T22:53:18.318 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.1099999999999995e-07 2024-09-20T22:53:18.318 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.318 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.318 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:53:18.318 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.175996+0000", 2024-09-20T22:53:18.318 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2121e-05 2024-09-20T22:53:18.318 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.318 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.318 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:53:18.318 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.176019+0000", 2024-09-20T22:53:18.319 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.3465999999999999e-05 2024-09-20T22:53:18.319 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.319 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.319 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:53:18.319 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.176026+0000", 2024-09-20T22:53:18.319 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.8190000000000003e-06 2024-09-20T22:53:18.319 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.319 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.319 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:53:18.319 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.176110+0000", 2024-09-20T22:53:18.319 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.3967000000000004e-05 2024-09-20T22:53:18.319 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.319 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.319 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting_for_map not empty", 2024-09-20T22:53:18.320 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.176116+0000", 2024-09-20T22:53:18.320 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.0249999999999999e-06 2024-09-20T22:53:18.320 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.320 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.320 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:53:18.320 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.180106+0000", 2024-09-20T22:53:18.320 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0039898980000000004 2024-09-20T22:53:18.320 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.320 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.320 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for blocked object", 2024-09-20T22:53:18.320 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.180135+0000", 2024-09-20T22:53:18.320 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.9595999999999999e-05 2024-09-20T22:53:18.320 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.320 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.320 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:53:18.320 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.194106+0000", 2024-09-20T22:53:18.321 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.013970761999999999 2024-09-20T22:53:18.321 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.321 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.321 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:53:18.321 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.194203+0000", 2024-09-20T22:53:18.321 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.7211000000000002e-05 2024-09-20T22:53:18.321 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.321 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.321 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:53:18.321 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.194306+0000", 2024-09-20T22:53:18.321 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00010307200000000001 2024-09-20T22:53:18.321 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:53:18.321 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:53:18.321 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:53:18.321 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.322 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.322 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24415.0:230 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch (1) in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e259)", 2024-09-20T22:53:18.322 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:45:39.207410+0000", 2024-09-20T22:53:18.322 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 459.05967504900002, 2024-09-20T22:53:18.322 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0072298289999999, 2024-09-20T22:53:18.322 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:53:18.322 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:53:18.322 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:53:18.322 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:53:18.322 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24415", 2024-09-20T22:53:18.322 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/2056244102", 2024-09-20T22:53:18.322 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 230 2024-09-20T22:53:18.322 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.322 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:53:18.322 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.323 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:53:18.323 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.207410+0000", 2024-09-20T22:53:18.323 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:53:18.323 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.323 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.323 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:53:18.323 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.207410+0000", 2024-09-20T22:53:18.323 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:53:18.323 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.323 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.323 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:53:18.323 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.207412+0000", 2024-09-20T22:53:18.323 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.7519999999999999e-06 2024-09-20T22:53:18.323 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.323 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.324 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:53:18.324 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.207419+0000", 2024-09-20T22:53:18.324 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.097e-06 2024-09-20T22:53:18.324 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.324 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.324 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:53:18.324 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.207432+0000", 2024-09-20T22:53:18.324 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2719e-05 2024-09-20T22:53:18.324 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.324 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.324 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:53:18.324 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.207464+0000", 2024-09-20T22:53:18.324 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.2820000000000001e-05 2024-09-20T22:53:18.324 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.324 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.325 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:53:18.325 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.207670+0000", 2024-09-20T22:53:18.325 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000205486 2024-09-20T22:53:18.325 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.325 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.325 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-20T22:53:18.325 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.207687+0000", 2024-09-20T22:53:18.325 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:53:18.327 INFO:tasks.rados.rados.0.smithi084.stdout:2834: done (1 left) 2024-09-20T22:53:18.327 INFO:tasks.rados.rados.0.smithi084.stdout:2835: read oid 13 snap -1 2024-09-20T22:53:18.327 INFO:tasks.rados.rados.0.smithi084.stdout:2835: expect deleted 2024-09-20T22:53:18.327 INFO:tasks.rados.rados.0.smithi084.stdout:2836: snap_create 2024-09-20T22:53:18.327 INFO:tasks.rados.rados.0.smithi084.stdout:2831: finishing write tid 1 to smithi08477203-48 2024-09-20T22:53:18.327 INFO:tasks.rados.rados.0.smithi084.stdout:2831: finishing write tid 2 to smithi08477203-48 2024-09-20T22:53:18.327 INFO:tasks.rados.rados.0.smithi084.stdout:2831: finishing write tid 3 to smithi08477203-48 2024-09-20T22:53:18.328 INFO:tasks.rados.rados.0.smithi084.stdout:2831: finishing write tid 4 to smithi08477203-48 2024-09-20T22:53:18.328 INFO:tasks.rados.rados.0.smithi084.stdout:2831: oid 48 updating version 0 to 1241 2024-09-20T22:53:18.328 INFO:tasks.rados.rados.0.smithi084.stdout:2831: oid 48 version 1241 is already newer than 1240 2024-09-20T22:53:18.328 INFO:tasks.rados.rados.0.smithi084.stdout:2831: oid 48 version 1241 is already newer than 1239 2024-09-20T22:53:18.328 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 48 v 1241 (ObjNum 985 snap 323 seq_num 985) dirty exists 2024-09-20T22:53:18.328 INFO:tasks.rados.rados.0.smithi084.stdout:2831: left oid 48 (ObjNum 985 snap 323 seq_num 985) 2024-09-20T22:53:18.328 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.7107000000000001e-05 2024-09-20T22:53:18.328 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.328 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.328 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:53:18.328 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:40.212672+0000", 2024-09-20T22:53:18.328 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.004984562 2024-09-20T22:53:18.329 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.329 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.329 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:53:18.329 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:40.213576+0000", 2024-09-20T22:53:18.329 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00090404799999999996 2024-09-20T22:53:18.329 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.329 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.329 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:53:18.329 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:40.214640+0000", 2024-09-20T22:53:18.329 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0010642379999999999 2024-09-20T22:53:18.329 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:53:18.329 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:53:18.329 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:53:18.330 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.330 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.330 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:2367 7.5 7:a160b5d4:::smithi08477203-16:head [stat,delete] snapc 56=[56,55,53,52,51,50,47,3e] ondisk+write+known_if_redirected+supports_pool_eio e260)", 2024-09-20T22:53:18.330 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:45:39.601299+0000", 2024-09-20T22:53:18.330 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 458.66578562799998, 2024-09-20T22:53:18.330 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.64714523300000004, 2024-09-20T22:53:18.330 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:53:18.330 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:53:18.330 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:53:18.330 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:53:18.330 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:53:18.330 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:53:18.330 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 2367 2024-09-20T22:53:18.331 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.331 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:53:18.331 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.331 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:53:18.331 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.601299+0000", 2024-09-20T22:53:18.331 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:53:18.331 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.331 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.331 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:53:18.331 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.601299+0000", 2024-09-20T22:53:18.331 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:53:18.331 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.332 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.332 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:53:18.332 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.601301+0000", 2024-09-20T22:53:18.332 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3850000000000001e-06 2024-09-20T22:53:18.332 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.332 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.332 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:53:18.332 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.601306+0000", 2024-09-20T22:53:18.332 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.2730000000000002e-06 2024-09-20T22:53:18.332 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.332 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.332 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:53:18.332 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.601318+0000", 2024-09-20T22:53:18.332 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2148e-05 2024-09-20T22:53:18.333 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.333 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.333 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:53:18.333 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.601324+0000", 2024-09-20T22:53:18.333 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.852e-06 2024-09-20T22:53:18.333 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.333 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.333 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:53:18.333 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.601383+0000", 2024-09-20T22:53:18.333 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.8570000000000003e-05 2024-09-20T22:53:18.333 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.333 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.333 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-20T22:53:18.334 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.601391+0000", 2024-09-20T22:53:18.334 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.5290000000000008e-06 2024-09-20T22:53:18.334 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.334 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.334 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:53:18.334 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:40.221227+0000", 2024-09-20T22:53:18.334 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.619835515 2024-09-20T22:53:18.334 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.334 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.334 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for degraded object", 2024-09-20T22:53:18.334 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:40.223919+0000", 2024-09-20T22:53:18.334 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.002692168 2024-09-20T22:53:18.334 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.335 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.335 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:53:18.335 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:40.244536+0000", 2024-09-20T22:53:18.335 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.020617613 2024-09-20T22:53:18.335 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.335 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.335 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:53:18.335 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:40.244653+0000", 2024-09-20T22:53:18.335 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000116796 2024-09-20T22:53:18.335 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.335 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.335 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 1,2", 2024-09-20T22:53:18.336 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:40.244832+0000", 2024-09-20T22:53:18.336 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00017833500000000001 2024-09-20T22:53:18.336 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.336 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.336 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:53:18.336 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:40.245494+0000", 2024-09-20T22:53:18.336 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00066241399999999995 2024-09-20T22:53:18.336 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.336 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.336 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:53:18.336 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:40.246501+0000", 2024-09-20T22:53:18.336 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0010066299999999999 2024-09-20T22:53:18.336 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.336 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.336 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:53:18.337 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:40.248326+0000", 2024-09-20T22:53:18.337 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001824895 2024-09-20T22:53:18.337 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.337 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.337 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:53:18.337 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:40.248370+0000", 2024-09-20T22:53:18.337 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.4298999999999997e-05 2024-09-20T22:53:18.337 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.337 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.337 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:53:18.337 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:40.248445+0000", 2024-09-20T22:53:18.337 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.4810999999999999e-05 2024-09-20T22:53:18.337 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:53:18.337 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:53:18.337 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:53:18.337 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.338 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.338 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:3132 7.19 e328/139 7:9eac4560:::smithi08477203-30:head v 328'198, mlcod=322'194)", 2024-09-20T22:53:18.338 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:46:48.223406+0000", 2024-09-20T22:53:18.338 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 390.04367919200001, 2024-09-20T22:53:18.338 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.016765688000000001, 2024-09-20T22:53:18.338 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:53:18.338 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:53:18.338 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:53:18.338 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:53:18.338 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.338 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:53:18.338 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.223406+0000", 2024-09-20T22:53:18.338 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:53:18.338 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.338 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.339 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:53:18.339 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.223406+0000", 2024-09-20T22:53:18.339 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:53:18.339 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.339 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.339 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:53:18.339 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.223406+0000", 2024-09-20T22:53:18.339 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.6699999999999999e-07 2024-09-20T22:53:18.339 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.339 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.339 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:53:18.339 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.226227+0000", 2024-09-20T22:53:18.339 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0028205690000000002 2024-09-20T22:53:18.339 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.340 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.340 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:53:18.340 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.226238+0000", 2024-09-20T22:53:18.340 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0879999999999999e-05 2024-09-20T22:53:18.340 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.340 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.340 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:53:18.340 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.226249+0000", 2024-09-20T22:53:18.340 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0912e-05 2024-09-20T22:53:18.340 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.340 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.340 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:53:18.340 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.227960+0000", 2024-09-20T22:53:18.340 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0017109619999999999 2024-09-20T22:53:18.340 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.341 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.341 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:53:18.341 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.227987+0000", 2024-09-20T22:53:18.341 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.7659999999999999e-05 2024-09-20T22:53:18.341 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.341 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.341 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:53:18.341 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.240119+0000", 2024-09-20T22:53:18.341 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.012131885 2024-09-20T22:53:18.341 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.341 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.341 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:53:18.341 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.240172+0000", 2024-09-20T22:53:18.341 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.2453000000000001e-05 2024-09-20T22:53:18.341 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:53:18.341 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:53:18.342 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:53:18.342 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.342 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.342 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:3232 7.16 7:6885a65f:::smithi08477203-14:head [copy-from ver 187] snapc 7e=[7e,7d,7c,7b,7a,79,78,75,74,73,70,6b] ondisk+write+known_if_redirected+supports_pool_eio e337)", 2024-09-20T22:53:18.342 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:46:57.213186+0000", 2024-09-20T22:53:18.342 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 381.05389952100001, 2024-09-20T22:53:18.342 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.028083105000000001, 2024-09-20T22:53:18.342 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:53:18.342 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:53:18.342 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:53:18.342 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:53:18.342 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:53:18.342 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:53:18.342 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 3232 2024-09-20T22:53:18.343 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.343 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:53:18.343 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.343 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:53:18.343 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.213186+0000", 2024-09-20T22:53:18.343 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:53:18.343 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.343 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.343 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:53:18.343 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.213186+0000", 2024-09-20T22:53:18.343 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:53:18.343 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.343 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.343 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:53:18.343 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.213188+0000", 2024-09-20T22:53:18.343 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.5129999999999999e-06 2024-09-20T22:53:18.344 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.344 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.344 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:53:18.344 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.213194+0000", 2024-09-20T22:53:18.344 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.4810000000000003e-06 2024-09-20T22:53:18.344 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.344 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.344 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:53:18.344 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.213215+0000", 2024-09-20T22:53:18.344 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.0455999999999999e-05 2024-09-20T22:53:18.344 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.344 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.344 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:53:18.344 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.213225+0000", 2024-09-20T22:53:18.344 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0212e-05 2024-09-20T22:53:18.344 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.345 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.345 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:53:18.345 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.213315+0000", 2024-09-20T22:53:18.345 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.9524000000000005e-05 2024-09-20T22:53:18.345 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.345 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.345 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op must wait for map", 2024-09-20T22:53:18.345 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.213324+0000", 2024-09-20T22:53:18.345 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.8890000000000008e-06 2024-09-20T22:53:18.345 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.345 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.345 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:53:18.345 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.220517+0000", 2024-09-20T22:53:18.345 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0071931219999999997 2024-09-20T22:53:18.346 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.346 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.346 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:53:18.346 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.220636+0000", 2024-09-20T22:53:18.346 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000118875 2024-09-20T22:53:18.346 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.346 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.346 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 2,5", 2024-09-20T22:53:18.346 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.230308+0000", 2024-09-20T22:53:18.346 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0096726299999999998 2024-09-20T22:53:18.346 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.346 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.346 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:53:18.347 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.238007+0000", 2024-09-20T22:53:18.347 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.007699287 2024-09-20T22:53:18.347 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.347 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.347 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:53:18.347 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.240863+0000", 2024-09-20T22:53:18.347 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0028554880000000002 2024-09-20T22:53:18.347 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.347 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.347 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:53:18.347 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.241194+0000", 2024-09-20T22:53:18.347 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00033087399999999999 2024-09-20T22:53:18.347 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.347 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.347 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:53:18.348 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.241233+0000", 2024-09-20T22:53:18.348 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.9338000000000002e-05 2024-09-20T22:53:18.348 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.348 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.348 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:53:18.348 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.241269+0000", 2024-09-20T22:53:18.348 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.5416000000000001e-05 2024-09-20T22:53:18.348 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:53:18.348 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:53:18.348 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:53:18.348 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.348 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.348 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:3233 7.16 7:6885a65f:::smithi08477203-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e337)", 2024-09-20T22:53:18.348 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:46:57.213253+0000", 2024-09-20T22:53:18.349 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 381.053832097, 2024-09-20T22:53:18.349 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.023594825, 2024-09-20T22:53:18.349 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:53:18.349 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:53:18.349 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:53:18.349 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:53:18.349 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:53:18.349 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:53:18.349 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 3233 2024-09-20T22:53:18.349 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.349 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:53:18.349 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.349 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:53:18.349 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.213253+0000", 2024-09-20T22:53:18.350 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:53:18.350 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.350 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.350 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:53:18.350 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.213253+0000", 2024-09-20T22:53:18.350 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:53:18.350 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.350 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.350 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:53:18.350 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.213254+0000", 2024-09-20T22:53:18.350 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.4799999999999998e-07 2024-09-20T22:53:18.350 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.350 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.350 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:53:18.350 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.213257+0000", 2024-09-20T22:53:18.351 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.3780000000000001e-06 2024-09-20T22:53:18.351 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.351 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.351 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:53:18.351 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.213267+0000", 2024-09-20T22:53:18.351 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.9480000000000003e-06 2024-09-20T22:53:18.351 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.351 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.351 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:53:18.351 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.213273+0000", 2024-09-20T22:53:18.351 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.1240000000000002e-06 2024-09-20T22:53:18.351 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.351 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.351 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:53:18.351 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.213394+0000", 2024-09-20T22:53:18.352 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000120717 2024-09-20T22:53:18.352 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.352 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.352 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting_for_map not empty", 2024-09-20T22:53:18.352 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.213400+0000", 2024-09-20T22:53:18.352 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.0769999999999996e-06 2024-09-20T22:53:18.352 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.352 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.352 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:53:18.352 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.220851+0000", 2024-09-20T22:53:18.352 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0074511309999999997 2024-09-20T22:53:18.352 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.352 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.352 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for blocked object", 2024-09-20T22:53:18.352 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.220903+0000", 2024-09-20T22:53:18.352 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.215e-05 2024-09-20T22:53:18.353 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.353 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.353 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:53:18.353 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.236698+0000", 2024-09-20T22:53:18.353 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.015794823 2024-09-20T22:53:18.353 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.353 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.353 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:53:18.353 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.236766+0000", 2024-09-20T22:53:18.353 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.8126999999999993e-05 2024-09-20T22:53:18.353 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.353 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.353 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:53:18.353 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.236848+0000", 2024-09-20T22:53:18.353 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.1701999999999998e-05 2024-09-20T22:53:18.354 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:53:18.354 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:53:18.354 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:53:18.354 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.354 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.354 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:3671 7.5 7:a160b5d4:::smithi08477203-16:head [copy-from ver 236] snapc 91=[91,8e,8d,8c,8b,8a,88,87,86,85,84,80,7f,7a,74,73,6b] ondisk+write+known_if_redirected+supports_pool_eio e370)", 2024-09-20T22:53:18.354 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:47:30.237235+0000", 2024-09-20T22:53:18.354 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 348.02985029799999, 2024-09-20T22:53:18.354 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.019719474000000001, 2024-09-20T22:53:18.354 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:53:18.354 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:53:18.354 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:53:18.354 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:53:18.354 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:53:18.355 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:53:18.355 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 3671 2024-09-20T22:53:18.355 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.355 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:53:18.355 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.355 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:53:18.355 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:30.237235+0000", 2024-09-20T22:53:18.355 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:53:18.355 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.355 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.355 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:53:18.355 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:30.237235+0000", 2024-09-20T22:53:18.355 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:53:18.355 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.356 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.356 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:53:18.356 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:30.237236+0000", 2024-09-20T22:53:18.356 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.5570000000000001e-06 2024-09-20T22:53:18.356 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.356 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.356 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:53:18.356 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:30.237241+0000", 2024-09-20T22:53:18.356 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.3279999999999999e-06 2024-09-20T22:53:18.356 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.356 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.356 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:53:18.356 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:30.237252+0000", 2024-09-20T22:53:18.356 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1004e-05 2024-09-20T22:53:18.356 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.357 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.357 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:53:18.357 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:30.237263+0000", 2024-09-20T22:53:18.357 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1341e-05 2024-09-20T22:53:18.357 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.357 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.357 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:53:18.357 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:30.237319+0000", 2024-09-20T22:53:18.357 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.5939000000000001e-05 2024-09-20T22:53:18.357 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.357 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.357 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:53:18.357 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:30.237436+0000", 2024-09-20T22:53:18.357 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000116866 2024-09-20T22:53:18.357 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.358 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.358 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 1,2", 2024-09-20T22:53:18.358 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:30.244330+0000", 2024-09-20T22:53:18.358 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0068945619999999999 2024-09-20T22:53:18.358 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.358 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.358 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:53:18.358 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:30.249103+0000", 2024-09-20T22:53:18.358 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0047726319999999997 2024-09-20T22:53:18.358 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.358 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.358 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:53:18.358 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:30.255771+0000", 2024-09-20T22:53:18.358 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.006667666 2024-09-20T22:53:18.358 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.358 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.359 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:53:18.359 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:30.256879+0000", 2024-09-20T22:53:18.359 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0011083110000000001 2024-09-20T22:53:18.359 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.359 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.359 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:53:18.359 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:30.256919+0000", 2024-09-20T22:53:18.359 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.9589000000000003e-05 2024-09-20T22:53:18.359 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.359 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.359 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:53:18.359 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:30.256954+0000", 2024-09-20T22:53:18.359 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.5679000000000002e-05 2024-09-20T22:53:18.360 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:53:18.360 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:53:18.360 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:53:18.360 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.360 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.360 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:3747 7.c e383/94 7:3f865ef5:::smithi08477203-28:head v 383'822, mlcod=382'818)", 2024-09-20T22:53:18.360 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:47:43.267267+0000", 2024-09-20T22:53:18.360 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 334.999817834, 2024-09-20T22:53:18.360 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.014405072, 2024-09-20T22:53:18.360 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:53:18.360 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:53:18.360 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:53:18.360 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:53:18.361 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.361 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:53:18.361 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.267267+0000", 2024-09-20T22:53:18.361 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:53:18.361 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.361 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.361 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:53:18.361 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.267267+0000", 2024-09-20T22:53:18.361 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:53:18.361 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.361 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.362 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:53:18.362 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.267267+0000", 2024-09-20T22:53:18.362 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.73e-07 2024-09-20T22:53:18.362 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.362 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.362 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:53:18.362 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.269537+0000", 2024-09-20T22:53:18.362 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0022696650000000001 2024-09-20T22:53:18.362 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.362 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.362 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:53:18.362 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.269564+0000", 2024-09-20T22:53:18.362 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.6653999999999999e-05 2024-09-20T22:53:18.363 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.363 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.363 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:53:18.363 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.269575+0000", 2024-09-20T22:53:18.363 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1661999999999999e-05 2024-09-20T22:53:18.363 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.363 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.363 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:53:18.363 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.269672+0000", 2024-09-20T22:53:18.363 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.6430999999999999e-05 2024-09-20T22:53:18.363 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.363 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.363 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:53:18.363 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.269716+0000", 2024-09-20T22:53:18.364 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.4518000000000003e-05 2024-09-20T22:53:18.364 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.364 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.364 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:53:18.364 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.281609+0000", 2024-09-20T22:53:18.364 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.011892982999999999 2024-09-20T22:53:18.364 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.364 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.364 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:53:18.364 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.281672+0000", 2024-09-20T22:53:18.364 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.2985999999999996e-05 2024-09-20T22:53:18.364 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:53:18.364 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:53:18.364 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:53:18.365 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.365 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.365 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:6000 7.9 e604/602 7:9eac4560:::smithi08477203-30:head v 604'886, mlcod=604'882)", 2024-09-20T22:53:18.365 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:51:24.304633+0000", 2024-09-20T22:53:18.365 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 113.962451597, 2024-09-20T22:53:18.365 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.014258487, 2024-09-20T22:53:18.365 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:53:18.365 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:53:18.365 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:53:18.365 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:53:18.365 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.365 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:53:18.365 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:24.304633+0000", 2024-09-20T22:53:18.365 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:53:18.365 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.365 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.366 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:53:18.366 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:24.304633+0000", 2024-09-20T22:53:18.366 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:53:18.366 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.366 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.366 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:53:18.366 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:24.304634+0000", 2024-09-20T22:53:18.366 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.9999999999999996e-07 2024-09-20T22:53:18.366 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.366 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.366 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:53:18.366 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:24.310348+0000", 2024-09-20T22:53:18.366 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0057139629999999999 2024-09-20T22:53:18.367 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.367 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.367 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:53:18.367 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:24.310377+0000", 2024-09-20T22:53:18.367 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.8544e-05 2024-09-20T22:53:18.367 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.367 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.367 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:53:18.367 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:24.310388+0000", 2024-09-20T22:53:18.367 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1259e-05 2024-09-20T22:53:18.367 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.367 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.367 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:53:18.368 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:24.310470+0000", 2024-09-20T22:53:18.368 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.1730000000000005e-05 2024-09-20T22:53:18.368 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.368 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.368 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:53:18.368 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:24.310493+0000", 2024-09-20T22:53:18.368 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.3618e-05 2024-09-20T22:53:18.368 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.368 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.368 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:53:18.368 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:24.318852+0000", 2024-09-20T22:53:18.368 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0083582840000000005 2024-09-20T22:53:18.368 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.368 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.369 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:53:18.369 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:24.318892+0000", 2024-09-20T22:53:18.369 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.0289e-05 2024-09-20T22:53:18.369 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:53:18.369 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:53:18.369 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:53:18.369 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.369 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.369 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:6253 7.9 e629/602 7:91f2d47c:::smithi08477203-8:head v 629'948, mlcod=627'945)", 2024-09-20T22:53:18.369 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:51:49.626110+0000", 2024-09-20T22:53:18.369 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 88.640974944000007, 2024-09-20T22:53:18.369 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.017238303, 2024-09-20T22:53:18.369 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:53:18.369 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:53:18.370 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:53:18.370 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:53:18.370 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.370 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:53:18.370 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:49.626110+0000", 2024-09-20T22:53:18.370 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:53:18.370 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.370 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.370 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:53:18.370 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:49.626110+0000", 2024-09-20T22:53:18.370 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:53:18.370 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.370 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.370 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:53:18.371 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:49.626111+0000", 2024-09-20T22:53:18.371 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.33e-06 2024-09-20T22:53:18.371 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.371 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.371 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:53:18.371 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:49.633470+0000", 2024-09-20T22:53:18.371 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0073589700000000003 2024-09-20T22:53:18.371 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.371 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.371 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:53:18.371 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:49.633492+0000", 2024-09-20T22:53:18.371 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.1443000000000001e-05 2024-09-20T22:53:18.371 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.371 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.372 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:53:18.372 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:49.633503+0000", 2024-09-20T22:53:18.372 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1610000000000001e-05 2024-09-20T22:53:18.372 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.372 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.372 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:53:18.372 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:49.633575+0000", 2024-09-20T22:53:18.372 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.1433999999999994e-05 2024-09-20T22:53:18.372 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.372 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.372 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:53:18.372 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:49.633600+0000", 2024-09-20T22:53:18.372 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.4960000000000002e-05 2024-09-20T22:53:18.372 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.373 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.373 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:53:18.373 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:49.643306+0000", 2024-09-20T22:53:18.373 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0097065109999999993 2024-09-20T22:53:18.373 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.373 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.373 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:53:18.373 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:49.643348+0000", 2024-09-20T22:53:18.373 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.2045000000000001e-05 2024-09-20T22:53:18.373 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:53:18.373 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:53:18.373 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:53:18.373 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.373 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.373 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:6659 7.5 7:a160b5d4:::smithi08477203-16:head [copy-from ver 1807] snapc 12d=[12d,12c,12b,12a,128,127,126,124,122,121,120,11f,11e,11b,119,118,115,112,10e,10c,107,103,fd,fb,fa,f8,ed,d7,c2,bf] ondisk+write+known_if_redirected+supports_pool_eio e674)", 2024-09-20T22:53:18.374 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:52:34.370352+0000", 2024-09-20T22:53:18.374 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 43.896733449999999, 2024-09-20T22:53:18.374 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.018083765000000002, 2024-09-20T22:53:18.374 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:53:18.374 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:53:18.374 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:53:18.374 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:53:18.374 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:53:18.374 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:53:18.374 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 6659 2024-09-20T22:53:18.374 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.374 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:53:18.374 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.374 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:53:18.374 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:34.370352+0000", 2024-09-20T22:53:18.375 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:53:18.375 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.375 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.375 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:53:18.375 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:34.370352+0000", 2024-09-20T22:53:18.375 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:53:18.375 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.375 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.375 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:53:18.375 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:34.370353+0000", 2024-09-20T22:53:18.375 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.61e-06 2024-09-20T22:53:18.375 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.375 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.375 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:53:18.376 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:34.370360+0000", 2024-09-20T22:53:18.376 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.6760000000000001e-06 2024-09-20T22:53:18.376 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.376 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.376 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:53:18.376 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:34.370371+0000", 2024-09-20T22:53:18.376 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1155000000000001e-05 2024-09-20T22:53:18.376 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.376 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.376 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:53:18.376 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:34.370380+0000", 2024-09-20T22:53:18.376 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.9039999999999998e-06 2024-09-20T22:53:18.376 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.376 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.376 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:53:18.377 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:34.370461+0000", 2024-09-20T22:53:18.377 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.0667000000000005e-05 2024-09-20T22:53:18.377 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.377 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.377 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:53:18.377 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:34.370571+0000", 2024-09-20T22:53:18.377 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000110129 2024-09-20T22:53:18.377 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.377 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.377 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 0,2", 2024-09-20T22:53:18.377 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:34.378118+0000", 2024-09-20T22:53:18.377 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0075469489999999998 2024-09-20T22:53:18.377 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.377 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.377 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:53:18.378 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:34.382656+0000", 2024-09-20T22:53:18.378 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0045383239999999998 2024-09-20T22:53:18.378 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.378 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.378 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:53:18.378 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:34.385168+0000", 2024-09-20T22:53:18.378 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0025123770000000001 2024-09-20T22:53:18.378 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.378 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.378 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:53:18.378 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:34.388343+0000", 2024-09-20T22:53:18.378 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.003174713 2024-09-20T22:53:18.378 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.378 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.378 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:53:18.379 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:34.388382+0000", 2024-09-20T22:53:18.379 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.8550000000000002e-05 2024-09-20T22:53:18.379 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.379 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.379 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:53:18.379 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:34.388435+0000", 2024-09-20T22:53:18.379 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.3711000000000001e-05 2024-09-20T22:53:18.379 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:53:18.379 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:53:18.379 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:53:18.379 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.379 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.379 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:6829 7.5 7:a160b5d4:::smithi08477203-16:head [write 548159~419732,stat] snapc 136=[136,135,134,133,132,131,130,12f,12e,12d,12c,12b,12a,128,127,126,124,122,121,11f,11b,119,112,10e,10c,107,103,fb,fa,f8,ed,d7,c2,bf] ondisk+write+known_if_redirected+supports_pool_eio e688)", 2024-09-20T22:53:18.379 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:52:48.308503+0000", 2024-09-20T22:53:18.379 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 29.958582125, 2024-09-20T22:53:18.380 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.020227210999999998, 2024-09-20T22:53:18.380 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:53:18.380 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:53:18.380 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:53:18.380 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:53:18.380 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:53:18.380 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:53:18.380 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 6829 2024-09-20T22:53:18.380 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.380 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:53:18.380 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.380 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:53:18.380 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.308503+0000", 2024-09-20T22:53:18.380 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:53:18.380 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.381 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.381 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:53:18.381 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.308503+0000", 2024-09-20T22:53:18.381 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:53:18.381 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.381 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.381 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:53:18.381 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.308505+0000", 2024-09-20T22:53:18.381 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.765e-06 2024-09-20T22:53:18.381 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.381 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.381 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:53:18.381 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.308957+0000", 2024-09-20T22:53:18.381 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00045254399999999998 2024-09-20T22:53:18.381 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.382 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.382 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:53:18.382 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.308975+0000", 2024-09-20T22:53:18.382 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.7489000000000001e-05 2024-09-20T22:53:18.382 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.382 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.382 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:53:18.382 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.308983+0000", 2024-09-20T22:53:18.382 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.4500000000000004e-06 2024-09-20T22:53:18.382 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.382 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.382 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:53:18.383 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.309063+0000", 2024-09-20T22:53:18.383 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.0264000000000003e-05 2024-09-20T22:53:18.383 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.383 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.383 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:53:18.383 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.309200+0000", 2024-09-20T22:53:18.383 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00013697500000000001 2024-09-20T22:53:18.383 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.383 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.383 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 0,2", 2024-09-20T22:53:18.383 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.309374+0000", 2024-09-20T22:53:18.383 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000173597 2024-09-20T22:53:18.383 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.383 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.383 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:53:18.384 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.318972+0000", 2024-09-20T22:53:18.384 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0095980430000000005 2024-09-20T22:53:18.384 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.384 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.384 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:53:18.384 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.326825+0000", 2024-09-20T22:53:18.384 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0078531720000000003 2024-09-20T22:53:18.384 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.384 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.384 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:53:18.384 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.328638+0000", 2024-09-20T22:53:18.384 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0018131969999999999 2024-09-20T22:53:18.384 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.385 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.385 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:53:18.385 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.328685+0000", 2024-09-20T22:53:18.385 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.6536999999999998e-05 2024-09-20T22:53:18.385 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.385 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.385 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:53:18.385 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.328730+0000", 2024-09-20T22:53:18.385 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.5178e-05 2024-09-20T22:53:18.385 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:53:18.385 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:53:18.385 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:53:18.385 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.385 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.385 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:6830 7.5 7:a160b5d4:::smithi08477203-16:head [write 1664216~522983] snapc 136=[136,135,134,133,132,131,130,12f,12e,12d,12c,12b,12a,128,127,126,124,122,121,11f,11b,119,112,10e,10c,107,103,fb,fa,f8,ed,d7,c2,bf] ondisk+write+known_if_redirected+supports_pool_eio e688)", 2024-09-20T22:53:18.386 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:52:48.315593+0000", 2024-09-20T22:53:18.386 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 29.951492033000001, 2024-09-20T22:53:18.386 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.013624783, 2024-09-20T22:53:18.386 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:53:18.386 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:53:18.386 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:53:18.386 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:53:18.386 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:53:18.386 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:53:18.386 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 6830 2024-09-20T22:53:18.386 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.386 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:53:18.386 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.386 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:53:18.387 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.315593+0000", 2024-09-20T22:53:18.387 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:53:18.387 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.387 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.387 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:53:18.387 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.315593+0000", 2024-09-20T22:53:18.387 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:53:18.387 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.387 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.387 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:53:18.387 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.315595+0000", 2024-09-20T22:53:18.387 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.86e-06 2024-09-20T22:53:18.388 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.388 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.388 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:53:18.388 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.316152+0000", 2024-09-20T22:53:18.388 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00055723899999999995 2024-09-20T22:53:18.388 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.388 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.388 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:53:18.388 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.316163+0000", 2024-09-20T22:53:18.388 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.057e-05 2024-09-20T22:53:18.388 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.388 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.388 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:53:18.388 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.316168+0000", 2024-09-20T22:53:18.389 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.3070000000000002e-06 2024-09-20T22:53:18.389 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.389 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.389 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:53:18.389 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.316233+0000", 2024-09-20T22:53:18.389 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.4653000000000006e-05 2024-09-20T22:53:18.389 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.389 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.389 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:53:18.389 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.316358+0000", 2024-09-20T22:53:18.389 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00012565699999999999 2024-09-20T22:53:18.389 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.389 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.389 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 0,2", 2024-09-20T22:53:18.390 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.316605+0000", 2024-09-20T22:53:18.390 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00024712399999999999 2024-09-20T22:53:18.390 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.390 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.390 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:53:18.390 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.319135+0000", 2024-09-20T22:53:18.390 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0025298510000000001 2024-09-20T22:53:18.390 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.390 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.390 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:53:18.390 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.328407+0000", 2024-09-20T22:53:18.390 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0092714389999999994 2024-09-20T22:53:18.390 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.391 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.391 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:53:18.391 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.329114+0000", 2024-09-20T22:53:18.391 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00070755299999999998 2024-09-20T22:53:18.391 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.391 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.391 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:53:18.391 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.329180+0000", 2024-09-20T22:53:18.391 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.6038999999999996e-05 2024-09-20T22:53:18.391 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:53:18.391 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:53:18.391 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:53:18.391 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.329218+0000", 2024-09-20T22:53:18.391 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.7490999999999997e-05 2024-09-20T22:53:18.391 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:53:18.392 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:53:18.392 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:53:18.392 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:53:18.392 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:53:18.392 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-20T22:53:18.413 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:18.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:18.414 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:53:18.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:18 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T22:53:18.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:18 smithi154 ceph-mon[33264]: osdmap e717: 8 total, 8 up, 7 in 2024-09-20T22:53:18.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:18 smithi154 ceph-mon[33264]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:53:18.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:18 smithi154 ceph-mon[33264]: osdmap e718: 8 total, 8 up, 7 in 2024-09-20T22:53:18.515 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:53:18.536 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:18.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:18.537 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:53:18.614 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:53:18 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: ::ffff:172.21.15.154 - - [20/Sep/2024:22:53:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-20T22:53:18.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:18 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T22:53:18.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:18 smithi084 ceph-mon[40024]: osdmap e717: 8 total, 8 up, 7 in 2024-09-20T22:53:18.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:18 smithi084 ceph-mon[40024]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:53:18.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:18 smithi084 ceph-mon[40024]: osdmap e718: 8 total, 8 up, 7 in 2024-09-20T22:53:18.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:18 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T22:53:18.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:18 smithi084 ceph-mon[29774]: osdmap e717: 8 total, 8 up, 7 in 2024-09-20T22:53:18.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:18 smithi084 ceph-mon[29774]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:53:18.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:18 smithi084 ceph-mon[29774]: osdmap e718: 8 total, 8 up, 7 in 2024-09-20T22:53:18.638 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:53:18.659 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:18.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:18.660 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:53:18.760 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:53:18.781 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:18.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:18.782 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:53:18.882 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:53:18.904 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:18.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:18.905 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:53:19.006 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:53:19.028 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:19.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:19.028 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:53:19.129 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:53:19.150 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:19.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:19.151 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:53:19.252 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:53:19.272 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:19.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:19.273 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:53:19.295 INFO:tasks.rados.rados.0.smithi084.stdout:2831: done (2 left) 2024-09-20T22:53:19.295 INFO:tasks.rados.rados.0.smithi084.stdout:2835: done (1 left) 2024-09-20T22:53:19.296 INFO:tasks.rados.rados.0.smithi084.stdout:2836: done (0 left) 2024-09-20T22:53:19.296 INFO:tasks.rados.rados.0.smithi084.stdout:2837: read oid 39 snap 323 2024-09-20T22:53:19.296 INFO:tasks.rados.rados.0.smithi084.stdout:2837: expect deleted 2024-09-20T22:53:19.296 INFO:tasks.rados.rados.0.smithi084.stdout:2838: read oid 47 snap -1 2024-09-20T22:53:19.296 INFO:tasks.rados.rados.0.smithi084.stdout:2838: expect deleted 2024-09-20T22:53:19.296 INFO:tasks.rados.rados.0.smithi084.stdout:2839: copy_from oid 50 from oid 8 current snap is 324 2024-09-20T22:53:19.296 INFO:tasks.rados.rados.0.smithi084.stdout:2840: write oid 32 current snap is 324 2024-09-20T22:53:19.296 INFO:tasks.rados.rados.0.smithi084.stdout:2840: seq_num 986 ranges {571702=405051,1431716=638344,2560365=11337} 2024-09-20T22:53:19.301 INFO:tasks.rados.rados.0.smithi084.stdout:2840: writing smithi08477203-32 from 571702 to 976753 tid 1 2024-09-20T22:53:19.308 INFO:tasks.rados.rados.0.smithi084.stdout:2840: writing smithi08477203-32 from 1431716 to 2070060 tid 2 2024-09-20T22:53:19.311 INFO:tasks.rados.rados.0.smithi084.stdout:2840: writing smithi08477203-32 from 2560365 to 2571702 tid 3 2024-09-20T22:53:19.311 INFO:tasks.rados.rados.0.smithi084.stdout:2841: read oid 38 snap -1 2024-09-20T22:53:19.311 INFO:tasks.rados.rados.0.smithi084.stdout:2841: expect (ObjNum 957 snap 313 seq_num 957) 2024-09-20T22:53:19.312 INFO:tasks.rados.rados.0.smithi084.stdout:2842: delete oid 17 current snap is 324 2024-09-20T22:53:19.312 INFO:tasks.rados.rados.0.smithi084.stdout:2840: finishing write tid 1 to smithi08477203-32 2024-09-20T22:53:19.312 INFO:tasks.rados.rados.0.smithi084.stdout:2839: finishing copy_from racing read to smithi08477203-50 2024-09-20T22:53:19.318 INFO:tasks.rados.rados.0.smithi084.stdout:2837: done (5 left) 2024-09-20T22:53:19.318 INFO:tasks.rados.rados.0.smithi084.stdout:2838: done (4 left) 2024-09-20T22:53:19.318 INFO:tasks.rados.rados.0.smithi084.stdout:2842: done (3 left) 2024-09-20T22:53:19.319 INFO:tasks.rados.rados.0.smithi084.stdout:2843: copy_from oid 44 from oid 13 current snap is 324 2024-09-20T22:53:19.319 INFO:tasks.rados.rados.0.smithi084.stdout:2839: finishing copy_from to smithi08477203-50 2024-09-20T22:53:19.319 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 50 v 1826 (ObjNum 923 snap 305 seq_num 923) dirty exists 2024-09-20T22:53:19.319 INFO:tasks.rados.rados.0.smithi084.stdout:2840: finishing write tid 2 to smithi08477203-32 2024-09-20T22:53:19.319 INFO:tasks.rados.rados.0.smithi084.stdout:2839: done (3 left) 2024-09-20T22:53:19.319 INFO:tasks.rados.rados.0.smithi084.stdout:2844: delete oid 16 current snap is 324 2024-09-20T22:53:19.319 INFO:tasks.rados.rados.0.smithi084.stdout:2840: finishing write tid 3 to smithi08477203-32 2024-09-20T22:53:19.319 INFO:tasks.rados.rados.0.smithi084.stdout:2840: finishing write tid 4 to smithi08477203-32 2024-09-20T22:53:19.319 INFO:tasks.rados.rados.0.smithi084.stdout:2840: finishing write tid 5 to smithi08477203-32 2024-09-20T22:53:19.319 INFO:tasks.rados.rados.0.smithi084.stdout:2840: oid 32 updating version 0 to 755 2024-09-20T22:53:19.319 INFO:tasks.rados.rados.0.smithi084.stdout:2840: oid 32 updating version 755 to 758 2024-09-20T22:53:19.319 INFO:tasks.rados.rados.0.smithi084.stdout:2840: oid 32 version 758 is already newer than 756 2024-09-20T22:53:19.319 INFO:tasks.rados.rados.0.smithi084.stdout:2840: oid 32 version 758 is already newer than 757 2024-09-20T22:53:19.319 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 32 v 758 (ObjNum 986 snap 324 seq_num 986) dirty exists 2024-09-20T22:53:19.319 INFO:tasks.rados.rados.0.smithi084.stdout:2840: left oid 32 (ObjNum 986 snap 324 seq_num 986) 2024-09-20T22:53:19.320 INFO:tasks.rados.rados.0.smithi084.stdout:2841: expect (ObjNum 957 snap 313 seq_num 957) 2024-09-20T22:53:19.374 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-20T22:53:19.374 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:53:19.395 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:19.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:19.396 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:53:19.405 INFO:tasks.rados.rados.0.smithi084.stdout:2843: finishing copy_from to smithi08477203-44 2024-09-20T22:53:19.405 INFO:tasks.rados.rados.0.smithi084.stdout:2843: got expected ENOENT (src dne) 2024-09-20T22:53:19.405 INFO:tasks.rados.rados.0.smithi084.stdout:2843: finishing copy_from racing read to smithi08477203-44 2024-09-20T22:53:19.405 INFO:tasks.rados.rados.0.smithi084.stdout:2840: done (3 left) 2024-09-20T22:53:19.405 INFO:tasks.rados.rados.0.smithi084.stdout:2841: done (2 left) 2024-09-20T22:53:19.405 INFO:tasks.rados.rados.0.smithi084.stdout:2843: done (1 left) 2024-09-20T22:53:19.405 INFO:tasks.rados.rados.0.smithi084.stdout:2844: done (0 left) 2024-09-20T22:53:19.405 INFO:tasks.rados.rados.0.smithi084.stdout:2845: copy_from oid 25 from oid 5 current snap is 324 2024-09-20T22:53:19.405 INFO:tasks.rados.rados.0.smithi084.stdout:2846: snap_create 2024-09-20T22:53:19.417 INFO:tasks.rados.rados.0.smithi084.stdout:2845: finishing copy_from racing read to smithi08477203-25 2024-09-20T22:53:19.424 INFO:tasks.rados.rados.0.smithi084.stdout:2845: finishing copy_from to smithi08477203-25 2024-09-20T22:53:19.425 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 25 v 774 (ObjNum 922 snap 305 seq_num 922) dirty exists 2024-09-20T22:53:19.497 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:53:19.519 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:19.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:19.520 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:53:19.614 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:19 smithi084 ceph-mon[29774]: pgmap v978: 158 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 153 active+clean; 621 MiB data, 1.6 GiB used, 624 GiB / 626 GiB avail 2024-09-20T22:53:19.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:19 smithi084 ceph-mon[29774]: osdmap e719: 8 total, 8 up, 7 in 2024-09-20T22:53:19.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:19 smithi084 ceph-mon[40024]: pgmap v978: 158 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 153 active+clean; 621 MiB data, 1.6 GiB used, 624 GiB / 626 GiB avail 2024-09-20T22:53:19.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:19 smithi084 ceph-mon[40024]: osdmap e719: 8 total, 8 up, 7 in 2024-09-20T22:53:19.620 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:53:19.623 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:53:19.642 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:19.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:19.643 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:53:19.744 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:53:19.754 INFO:journalctl@ceph.iscsi.iscsi.a.smithi154.stdout:Sep 20 22:53:19 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-iscsi-iscsi-a[59730]: debug there is no tcmu-runner data available 2024-09-20T22:53:19.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:19 smithi154 ceph-mon[33264]: pgmap v978: 158 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 153 active+clean; 621 MiB data, 1.6 GiB used, 624 GiB / 626 GiB avail 2024-09-20T22:53:19.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:19 smithi154 ceph-mon[33264]: osdmap e719: 8 total, 8 up, 7 in 2024-09-20T22:53:19.766 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:19.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:19.767 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:53:19.868 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:53:19.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:19.890 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:19.890 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:53:19.990 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:53:20.012 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:20.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:20.013 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:53:20.084 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-20T22:53:20.085 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2024-09-20T22:53:20.085 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2024-09-20T22:53:20.085 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-20T22:53:20.113 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:53:20.134 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:20.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:20.135 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:53:20.171 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-20T22:53:20.235 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:53:20.257 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:20.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:20.258 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:53:20.318 INFO:tasks.rados.rados.0.smithi084.stdout:2845: done (1 left) 2024-09-20T22:53:20.318 INFO:tasks.rados.rados.0.smithi084.stdout:2846: done (0 left) 2024-09-20T22:53:20.318 INFO:tasks.rados.rados.0.smithi084.stdout:2847: copy_from oid 38 from oid 2 current snap is 325 2024-09-20T22:53:20.318 INFO:tasks.rados.rados.0.smithi084.stdout:2848: copy_from oid 14 from oid 44 current snap is 325 2024-09-20T22:53:20.318 INFO:tasks.rados.rados.0.smithi084.stdout:2849: snap_remove snap 288 2024-09-20T22:53:20.359 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:53:20.364 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:20 smithi084 ceph-mon[29774]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:53:20.364 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:20 smithi084 ceph-mon[40024]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:53:20.380 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:20.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:20.381 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:53:20.482 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:53:20.504 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:20.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:20.505 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:53:20.606 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:53:20.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:20.629 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:20.629 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:53:20.721 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd unset noscrub 2024-09-20T22:53:20.730 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:53:20.739 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.0 dump_blocked_ops 2024-09-20T22:53:20.752 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:53:20.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:20 smithi154 ceph-mon[33264]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:53:20.772 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:20.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:20.773 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:53:20.800 INFO:tasks.thrashosds.thrasher:in_osds: [2, 3, 5, 6, 7, 0, 4] out_osds: [1] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-20T22:53:20.800 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd reweight 2 1 2024-09-20T22:53:20.874 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:53:20.897 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:20.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:20.898 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:53:20.998 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:53:21.031 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:21.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:21.032 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:53:21.133 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:53:21.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:21.156 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:21.156 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:53:21.208 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:53:21.208 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:53:21.230 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:53:21.231 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:53:21.253 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:53:21.253 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:53:21.257 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:53:21.265 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:53:21.283 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:53:21.283 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:21.283 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-20T22:53:21.284 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T22:53:21.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:21.284 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:53:21.286 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:53:21.306 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:53:21.307 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:53:21.327 INFO:tasks.rados.rados.0.smithi084.stdout:2849: done (2 left) 2024-09-20T22:53:21.327 INFO:tasks.rados.rados.0.smithi084.stdout:2850: read oid 28 snap -1 2024-09-20T22:53:21.327 INFO:tasks.rados.rados.0.smithi084.stdout:2850: expect deleted 2024-09-20T22:53:21.327 INFO:tasks.rados.rados.0.smithi084.stdout:2851: rollback oid 41 current snap is 325 2024-09-20T22:53:21.327 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 41 to 290 2024-09-20T22:53:21.327 INFO:tasks.rados.rados.0.smithi084.stdout:2852: copy_from oid 31 from oid 12 current snap is 325 2024-09-20T22:53:21.328 INFO:tasks.rados.rados.0.smithi084.stdout:2853: copy_from oid 5 from oid 6 current snap is 325 2024-09-20T22:53:21.328 INFO:tasks.rados.rados.0.smithi084.stdout:2848: finishing copy_from racing read to smithi08477203-14 2024-09-20T22:53:21.328 INFO:tasks.rados.rados.0.smithi084.stdout:2848: finishing copy_from to smithi08477203-14 2024-09-20T22:53:21.328 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 14 v 1901 (ObjNum 879 snap 285 seq_num 879) dirty exists 2024-09-20T22:53:21.328 INFO:tasks.rados.rados.0.smithi084.stdout:2848: done (5 left) 2024-09-20T22:53:21.328 INFO:tasks.rados.rados.0.smithi084.stdout:2854: snap_remove snap 303 2024-09-20T22:53:21.332 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:53:21.332 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:53:21.354 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:53:21.354 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:53:21.370 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:53:21.371 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:53:21.385 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:53:21.407 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:21.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:21.408 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:53:21.508 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:53:21.534 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:21.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:21.535 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:53:21.542 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:21 smithi084 ceph-mon[40024]: pgmap v981: 158 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 153 active+clean; 561 MiB data, 1.6 GiB used, 624 GiB / 626 GiB avail; 1.5 KiB/s rd, 368 KiB/s wr, 2 op/s 2024-09-20T22:53:21.542 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:21 smithi084 ceph-mon[40024]: osdmap e720: 8 total, 8 up, 7 in 2024-09-20T22:53:21.542 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:21 smithi084 ceph-mon[40024]: purged_snaps scrub starts 2024-09-20T22:53:21.543 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:21 smithi084 ceph-mon[40024]: purged_snaps scrub ok 2024-09-20T22:53:21.543 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:21 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/3166034760' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:53:21.543 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:21 smithi084 ceph-mon[29774]: pgmap v981: 158 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 153 active+clean; 561 MiB data, 1.6 GiB used, 624 GiB / 626 GiB avail; 1.5 KiB/s rd, 368 KiB/s wr, 2 op/s 2024-09-20T22:53:21.543 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:21 smithi084 ceph-mon[29774]: osdmap e720: 8 total, 8 up, 7 in 2024-09-20T22:53:21.543 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:21 smithi084 ceph-mon[29774]: purged_snaps scrub starts 2024-09-20T22:53:21.543 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:21 smithi084 ceph-mon[29774]: purged_snaps scrub ok 2024-09-20T22:53:21.543 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:21 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/3166034760' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:53:21.637 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:53:21.660 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:21.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:21.661 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:53:21.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:21 smithi154 ceph-mon[33264]: pgmap v981: 158 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 153 active+clean; 561 MiB data, 1.6 GiB used, 624 GiB / 626 GiB avail; 1.5 KiB/s rd, 368 KiB/s wr, 2 op/s 2024-09-20T22:53:21.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:21 smithi154 ceph-mon[33264]: osdmap e720: 8 total, 8 up, 7 in 2024-09-20T22:53:21.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:21 smithi154 ceph-mon[33264]: purged_snaps scrub starts 2024-09-20T22:53:21.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:21 smithi154 ceph-mon[33264]: purged_snaps scrub ok 2024-09-20T22:53:21.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:21 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/3166034760' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:53:21.762 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:53:21.784 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:21.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:21.785 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:53:21.886 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:53:21.906 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:21.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:21.907 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:53:22.008 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:53:22.029 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:22.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:22.029 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:53:22.130 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:53:22.156 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:22.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:22.157 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:53:22.257 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:53:22.279 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:22.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:22.280 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:53:22.333 INFO:tasks.rados.rados.0.smithi084.stdout:2854: done (5 left) 2024-09-20T22:53:22.334 INFO:tasks.rados.rados.0.smithi084.stdout:2855: copy_from oid 42 from oid 40 current snap is 325 2024-09-20T22:53:22.334 INFO:tasks.rados.rados.0.smithi084.stdout:2847: finishing copy_from racing read to smithi08477203-38 2024-09-20T22:53:22.334 INFO:tasks.rados.rados.0.smithi084.stdout:2847: finishing copy_from to smithi08477203-38 2024-09-20T22:53:22.334 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 38 v 1805 (ObjNum 918 snap 302 seq_num 918) dirty exists 2024-09-20T22:53:22.334 INFO:tasks.rados.rados.0.smithi084.stdout:2852: finishing copy_from to smithi08477203-31 2024-09-20T22:53:22.334 INFO:tasks.rados.rados.0.smithi084.stdout:2852: got expected ENOENT (src dne) 2024-09-20T22:53:22.334 INFO:tasks.rados.rados.0.smithi084.stdout:2852: finishing copy_from racing read to smithi08477203-31 2024-09-20T22:53:22.335 INFO:tasks.rados.rados.0.smithi084.stdout:2851: finishing rollback tid 0 to smithi08477203-41 2024-09-20T22:53:22.335 INFO:tasks.rados.rados.0.smithi084.stdout:2851: finishing rollback tid 1 to smithi08477203-41 2024-09-20T22:53:22.335 INFO:tasks.rados.rados.0.smithi084.stdout:2851: finishing rollback tid 2 to smithi08477203-41 2024-09-20T22:53:22.335 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 41 v 1423 (ObjNum 779 snap 252 seq_num 779) dirty exists 2024-09-20T22:53:22.335 INFO:tasks.rados.rados.0.smithi084.stdout:2853: finishing copy_from racing read to smithi08477203-5 2024-09-20T22:53:22.335 INFO:tasks.rados.rados.0.smithi084.stdout:2853: finishing copy_from to smithi08477203-5 2024-09-20T22:53:22.336 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 5 v 795 (ObjNum 948 snap 311 seq_num 948) dirty exists 2024-09-20T22:53:22.336 INFO:tasks.rados.rados.0.smithi084.stdout:2847: done (5 left) 2024-09-20T22:53:22.336 INFO:tasks.rados.rados.0.smithi084.stdout:2850: done (4 left) 2024-09-20T22:53:22.336 INFO:tasks.rados.rados.0.smithi084.stdout:2851: done (3 left) 2024-09-20T22:53:22.336 INFO:tasks.rados.rados.0.smithi084.stdout:2852: done (2 left) 2024-09-20T22:53:22.336 INFO:tasks.rados.rados.0.smithi084.stdout:2853: done (1 left) 2024-09-20T22:53:22.336 INFO:tasks.rados.rados.0.smithi084.stdout:2856: read oid 21 snap -1 2024-09-20T22:53:22.336 INFO:tasks.rados.rados.0.smithi084.stdout:2856: expect (ObjNum 984 snap 319 seq_num 984) 2024-09-20T22:53:22.336 INFO:tasks.rados.rados.0.smithi084.stdout:2857: read oid 10 snap 301 2024-09-20T22:53:22.336 INFO:tasks.rados.rados.0.smithi084.stdout:2857: expect deleted 2024-09-20T22:53:22.337 INFO:tasks.rados.rados.0.smithi084.stdout:2858: rollback oid 3 current snap is 325 2024-09-20T22:53:22.337 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 3 to 308 2024-09-20T22:53:22.337 INFO:tasks.rados.rados.0.smithi084.stdout:2859: delete oid 25 current snap is 325 2024-09-20T22:53:22.339 INFO:tasks.rados.rados.0.smithi084.stdout:2856: expect (ObjNum 984 snap 319 seq_num 984) 2024-09-20T22:53:22.381 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:53:22.404 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:22.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:22.405 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:53:22.426 INFO:tasks.rados.rados.0.smithi084.stdout:2856: done (4 left) 2024-09-20T22:53:22.426 INFO:tasks.rados.rados.0.smithi084.stdout:2857: done (3 left) 2024-09-20T22:53:22.427 INFO:tasks.rados.rados.0.smithi084.stdout:2859: done (2 left) 2024-09-20T22:53:22.427 INFO:tasks.rados.rados.0.smithi084.stdout:2860: rollback oid 27 current snap is 325 2024-09-20T22:53:22.427 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 27 to 300 2024-09-20T22:53:22.427 INFO:tasks.rados.rados.0.smithi084.stdout:2858: finishing rollback tid 0 to smithi08477203-3 2024-09-20T22:53:22.427 INFO:tasks.rados.rados.0.smithi084.stdout:2858: finishing rollback tid 1 to smithi08477203-3 2024-09-20T22:53:22.427 INFO:tasks.rados.rados.0.smithi084.stdout:2858: finishing rollback tid 2 to smithi08477203-3 2024-09-20T22:53:22.427 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 3 v 1839 (ObjNum 934 snap 308 seq_num 934) dirty exists 2024-09-20T22:53:22.427 INFO:tasks.rados.rados.0.smithi084.stdout:2855: finishing copy_from racing read to smithi08477203-42 2024-09-20T22:53:22.427 INFO:tasks.rados.rados.0.smithi084.stdout:2855: finishing copy_from to smithi08477203-42 2024-09-20T22:53:22.427 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 42 v 1255 (ObjNum 970 snap 315 seq_num 970) dirty exists 2024-09-20T22:53:22.427 INFO:tasks.rados.rados.0.smithi084.stdout:2855: done (2 left) 2024-09-20T22:53:22.427 INFO:tasks.rados.rados.0.smithi084.stdout:2858: done (1 left) 2024-09-20T22:53:22.427 INFO:tasks.rados.rados.0.smithi084.stdout:2861: snap_remove snap 317 2024-09-20T22:53:22.506 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:53:22.513 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:22 smithi084 ceph-mon[40024]: osdmap e721: 8 total, 8 up, 7 in 2024-09-20T22:53:22.513 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:22 smithi084 ceph-mon[29774]: osdmap e721: 8 total, 8 up, 7 in 2024-09-20T22:53:22.528 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:22.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:22.529 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:53:22.629 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:53:22.650 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:22.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:22.651 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:53:22.752 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:53:22.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:22 smithi154 ceph-mon[33264]: osdmap e721: 8 total, 8 up, 7 in 2024-09-20T22:53:22.773 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:22.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:22.774 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:53:22.875 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:53:22.896 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:22.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:22.898 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:53:22.999 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:53:23.020 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:23.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:23.021 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:53:23.122 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:53:23.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:23.145 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:23.145 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:53:23.246 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:53:23.269 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:23.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:23.270 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:53:23.289 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is unset 2024-09-20T22:53:23.294 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.2 to 1 (10000) 2024-09-20T22:53:23.297 INFO:tasks.rados.rados.0.smithi084.stdout:2861: done (1 left) 2024-09-20T22:53:23.297 INFO:tasks.rados.rados.0.smithi084.stdout:2862: delete oid 6 current snap is 325 2024-09-20T22:53:23.297 INFO:tasks.rados.rados.0.smithi084.stdout:2860: finishing rollback tid 0 to smithi08477203-27 2024-09-20T22:53:23.298 INFO:tasks.rados.rados.0.smithi084.stdout:2860: finishing rollback tid 1 to smithi08477203-27 2024-09-20T22:53:23.298 INFO:tasks.rados.rados.0.smithi084.stdout:2860: finishing rollback tid 2 to smithi08477203-27 2024-09-20T22:53:23.298 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 27 v 751 (ObjNum 898 snap 295 seq_num 898) dirty exists 2024-09-20T22:53:23.305 INFO:tasks.rados.rados.0.smithi084.stdout:2860: done (1 left) 2024-09-20T22:53:23.305 INFO:tasks.rados.rados.0.smithi084.stdout:2862: done (0 left) 2024-09-20T22:53:23.305 INFO:tasks.rados.rados.0.smithi084.stdout:2863: write oid 11 current snap is 325 2024-09-20T22:53:23.305 INFO:tasks.rados.rados.0.smithi084.stdout:2863: seq_num 991 ranges {655737=497649,1607773=789340} 2024-09-20T22:53:23.311 INFO:tasks.rados.rados.0.smithi084.stdout:2863: writing smithi08477203-11 from 655737 to 1153386 tid 1 2024-09-20T22:53:23.319 INFO:tasks.rados.rados.0.smithi084.stdout:2863: writing smithi08477203-11 from 1607773 to 2397113 tid 2 2024-09-20T22:53:23.319 INFO:tasks.rados.rados.0.smithi084.stdout:2864: rollback oid 16 current snap is 325 2024-09-20T22:53:23.319 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 16 to 308 2024-09-20T22:53:23.319 INFO:tasks.rados.rados.0.smithi084.stdout:2863: finishing write tid 1 to smithi08477203-11 2024-09-20T22:53:23.319 INFO:tasks.rados.rados.0.smithi084.stdout:2865: copy_from oid 1 from oid 33 current snap is 325 2024-09-20T22:53:23.319 INFO:tasks.rados.rados.0.smithi084.stdout:2866: snap_remove snap 281 2024-09-20T22:53:23.370 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:53:23.391 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:23.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:23.392 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:53:23.426 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-20T22:53:23.426 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:53:23.493 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:53:23.514 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:23.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:23.515 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:53:23.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:23 smithi084 ceph-mon[40024]: pgmap v984: 158 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 153 active+clean; 561 MiB data, 1.6 GiB used, 624 GiB / 626 GiB avail; 1.5 KiB/s rd, 368 KiB/s wr, 2 op/s 2024-09-20T22:53:23.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:23 smithi084 ceph-mon[40024]: osdmap e722: 8 total, 8 up, 7 in 2024-09-20T22:53:23.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:23 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/3544366978' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-09-20T22:53:23.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:23 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-09-20T22:53:23.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:23 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/117886743' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:53:23.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:23 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:53:23.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:23 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:53:23.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:23 smithi084 ceph-mon[40024]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:53:23.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:23 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-20T22:53:23.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:23 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/117886743' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T22:53:23.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:23 smithi084 ceph-mon[40024]: osdmap e723: 8 total, 8 up, 7 in 2024-09-20T22:53:23.617 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:23 smithi084 ceph-mon[29774]: pgmap v984: 158 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 153 active+clean; 561 MiB data, 1.6 GiB used, 624 GiB / 626 GiB avail; 1.5 KiB/s rd, 368 KiB/s wr, 2 op/s 2024-09-20T22:53:23.617 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:23 smithi084 ceph-mon[29774]: osdmap e722: 8 total, 8 up, 7 in 2024-09-20T22:53:23.617 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:23 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/3544366978' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-09-20T22:53:23.617 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:23 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-09-20T22:53:23.617 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:23 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/117886743' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:53:23.617 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:23 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:53:23.617 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:23 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:53:23.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:23 smithi084 ceph-mon[29774]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:53:23.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:23 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-20T22:53:23.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:23 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/117886743' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T22:53:23.618 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:23 smithi084 ceph-mon[29774]: osdmap e723: 8 total, 8 up, 7 in 2024-09-20T22:53:23.618 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:53:23.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:23.641 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:23.641 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:53:23.742 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:53:23.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:23 smithi154 ceph-mon[33264]: pgmap v984: 158 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 153 active+clean; 561 MiB data, 1.6 GiB used, 624 GiB / 626 GiB avail; 1.5 KiB/s rd, 368 KiB/s wr, 2 op/s 2024-09-20T22:53:23.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:23 smithi154 ceph-mon[33264]: osdmap e722: 8 total, 8 up, 7 in 2024-09-20T22:53:23.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:23 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/3544366978' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-09-20T22:53:23.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:23 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-09-20T22:53:23.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:23 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/117886743' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:53:23.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:23 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:53:23.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:23 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:53:23.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:23 smithi154 ceph-mon[33264]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:53:23.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:23 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-20T22:53:23.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:23 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/117886743' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T22:53:23.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:23 smithi154 ceph-mon[33264]: osdmap e723: 8 total, 8 up, 7 in 2024-09-20T22:53:23.763 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:23.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:23.764 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:53:23.865 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:53:23.886 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:23.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:23.887 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:53:23.988 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:53:24.010 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:24.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:24.011 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:53:24.042 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.0 dump_historic_ops 2024-09-20T22:53:24.112 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:53:24.134 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:24.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:24.135 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:53:24.236 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:53:24.258 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:24.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:24.259 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:53:24.294 INFO:tasks.rados.rados.0.smithi084.stdout:2866: done (3 left) 2024-09-20T22:53:24.295 INFO:tasks.rados.rados.0.smithi084.stdout:2867: write (excl) oid 41 current snap is 325 2024-09-20T22:53:24.295 INFO:tasks.rados.rados.0.smithi084.stdout:2864: finishing rollback tid 1 to smithi08477203-16 2024-09-20T22:53:24.295 INFO:tasks.rados.rados.0.smithi084.stdout:2867: seq_num 992 ranges {672544=572898,1666255=532998} 2024-09-20T22:53:24.302 INFO:tasks.rados.rados.0.smithi084.stdout:2867: writing smithi08477203-41 from 672544 to 1245442 tid 1 2024-09-20T22:53:24.309 INFO:tasks.rados.rados.0.smithi084.stdout:2867: writing smithi08477203-41 from 1666255 to 2199253 tid 2 2024-09-20T22:53:24.309 INFO:tasks.rados.rados.0.smithi084.stdout:2868: read oid 35 snap -1 2024-09-20T22:53:24.309 INFO:tasks.rados.rados.0.smithi084.stdout:2868: expect deleted 2024-09-20T22:53:24.309 INFO:tasks.rados.rados.0.smithi084.stdout:2869: read oid 27 snap -1 2024-09-20T22:53:24.309 INFO:tasks.rados.rados.0.smithi084.stdout:2869: expect (ObjNum 898 snap 295 seq_num 898) 2024-09-20T22:53:24.309 INFO:tasks.rados.rados.0.smithi084.stdout:2870: snap_create 2024-09-20T22:53:24.309 INFO:tasks.rados.rados.0.smithi084.stdout:2864: finishing rollback tid 2 to smithi08477203-16 2024-09-20T22:53:24.310 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 16 v 1814 (ObjNum 876 snap 280 seq_num 876) dirty exists 2024-09-20T22:53:24.310 INFO:tasks.rados.rados.0.smithi084.stdout:2863: finishing write tid 2 to smithi08477203-11 2024-09-20T22:53:24.310 INFO:tasks.rados.rados.0.smithi084.stdout:2863: finishing write tid 3 to smithi08477203-11 2024-09-20T22:53:24.310 INFO:tasks.rados.rados.0.smithi084.stdout:2863: finishing write tid 4 to smithi08477203-11 2024-09-20T22:53:24.310 INFO:tasks.rados.rados.0.smithi084.stdout:2863: oid 11 updating version 0 to 874 2024-09-20T22:53:24.310 INFO:tasks.rados.rados.0.smithi084.stdout:2863: oid 11 updating version 874 to 875 2024-09-20T22:53:24.310 INFO:tasks.rados.rados.0.smithi084.stdout:2863: oid 11 version 875 is already newer than 873 2024-09-20T22:53:24.310 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 11 v 875 (ObjNum 991 snap 325 seq_num 991) dirty exists 2024-09-20T22:53:24.310 INFO:tasks.rados.rados.0.smithi084.stdout:2863: left oid 11 (ObjNum 991 snap 325 seq_num 991) 2024-09-20T22:53:24.310 INFO:tasks.rados.rados.0.smithi084.stdout:2865: finishing copy_from racing read to smithi08477203-1 2024-09-20T22:53:24.310 INFO:tasks.rados.rados.0.smithi084.stdout:2865: finishing copy_from to smithi08477203-1 2024-09-20T22:53:24.310 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 1 v 1261 (ObjNum 738 snap 235 seq_num 738) dirty exists 2024-09-20T22:53:24.310 INFO:tasks.rados.rados.0.smithi084.stdout:2867: finishing write tid 1 to smithi08477203-41 2024-09-20T22:53:24.312 INFO:tasks.rados.rados.0.smithi084.stdout:2867: finishing write tid 2 to smithi08477203-41 2024-09-20T22:53:24.312 INFO:tasks.rados.rados.0.smithi084.stdout:2867: finishing write tid 3 to smithi08477203-41 2024-09-20T22:53:24.312 INFO:tasks.rados.rados.0.smithi084.stdout:2867: finishing write tid 4 to smithi08477203-41 2024-09-20T22:53:24.312 INFO:tasks.rados.rados.0.smithi084.stdout:2867: oid 41 updating version 0 to 1425 2024-09-20T22:53:24.312 INFO:tasks.rados.rados.0.smithi084.stdout:2867: oid 41 updating version 1425 to 1426 2024-09-20T22:53:24.312 INFO:tasks.rados.rados.0.smithi084.stdout:2867: oid 41 version 1426 is already newer than 1424 2024-09-20T22:53:24.312 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 41 v 1426 (ObjNum 992 snap 325 seq_num 992) dirty exists 2024-09-20T22:53:24.312 INFO:tasks.rados.rados.0.smithi084.stdout:2867: left oid 41 (ObjNum 992 snap 325 seq_num 992) 2024-09-20T22:53:24.324 INFO:tasks.rados.rados.0.smithi084.stdout:2869: expect (ObjNum 898 snap 295 seq_num 898) 2024-09-20T22:53:24.359 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:53:24.380 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:24.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:24.381 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:53:24.482 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:53:24.503 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:24.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:24.504 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:53:24.605 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:53:24.626 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:24.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:24.627 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:53:24.728 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:53:24.749 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:24.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:24.750 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:53:24.851 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:53:24.873 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:24.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:24.874 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:53:24.975 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:53:24.997 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:24.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:24.998 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:53:25.098 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:53:25.120 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:25.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:25.121 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:53:25.221 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:53:25.243 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:25.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:25.244 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:53:25.298 INFO:tasks.rados.rados.0.smithi084.stdout:2863: done (6 left) 2024-09-20T22:53:25.298 INFO:tasks.rados.rados.0.smithi084.stdout:2864: done (5 left) 2024-09-20T22:53:25.298 INFO:tasks.rados.rados.0.smithi084.stdout:2865: done (4 left) 2024-09-20T22:53:25.299 INFO:tasks.rados.rados.0.smithi084.stdout:2867: done (3 left) 2024-09-20T22:53:25.299 INFO:tasks.rados.rados.0.smithi084.stdout:2868: done (2 left) 2024-09-20T22:53:25.299 INFO:tasks.rados.rados.0.smithi084.stdout:2869: done (1 left) 2024-09-20T22:53:25.299 INFO:tasks.rados.rados.0.smithi084.stdout:2870: done (0 left) 2024-09-20T22:53:25.299 INFO:tasks.rados.rados.0.smithi084.stdout:2871: write (excl) oid 2 current snap is 326 2024-09-20T22:53:25.299 INFO:tasks.rados.rados.0.smithi084.stdout:2871: seq_num 993 ranges {689351=648147,2124737=564614} 2024-09-20T22:53:25.305 INFO:tasks.rados.rados.0.smithi084.stdout:2871: writing smithi08477203-2 from 689351 to 1337498 tid 1 2024-09-20T22:53:25.313 INFO:tasks.rados.rados.0.smithi084.stdout:2871: writing smithi08477203-2 from 2124737 to 2689351 tid 2 2024-09-20T22:53:25.313 INFO:tasks.rados.rados.0.smithi084.stdout:2872: rollback oid 9 current snap is 326 2024-09-20T22:53:25.313 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 9 to 301 2024-09-20T22:53:25.313 INFO:tasks.rados.rados.0.smithi084.stdout:2871: finishing write tid 1 to smithi08477203-2 2024-09-20T22:53:25.313 INFO:tasks.rados.rados.0.smithi084.stdout:2873: rollback oid 17 current snap is 326 2024-09-20T22:53:25.314 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 17 to 323 2024-09-20T22:53:25.314 INFO:tasks.rados.rados.0.smithi084.stdout:2874: snap_remove snap 248 2024-09-20T22:53:25.345 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:53:25.348 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-20T22:53:25.366 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:25.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:25.367 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:53:25.432 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd reweight 3 1 2024-09-20T22:53:25.437 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:53:25.468 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:53:25.490 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:25.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:25.491 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:53:25.564 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:25 smithi084 ceph-mon[40024]: pgmap v987: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 573 MiB data, 1.6 GiB used, 624 GiB / 626 GiB avail; 3.4 MiB/s rd, 5.8 MiB/s wr, 15 op/s 2024-09-20T22:53:25.564 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:25 smithi084 ceph-mon[40024]: osdmap e724: 8 total, 8 up, 7 in 2024-09-20T22:53:25.565 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:25 smithi084 ceph-mon[29774]: pgmap v987: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 573 MiB data, 1.6 GiB used, 624 GiB / 626 GiB avail; 3.4 MiB/s rd, 5.8 MiB/s wr, 15 op/s 2024-09-20T22:53:25.565 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:25 smithi084 ceph-mon[29774]: osdmap e724: 8 total, 8 up, 7 in 2024-09-20T22:53:25.593 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:53:25.614 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:25.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:25.615 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:53:25.716 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:53:25.738 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:25.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:25.739 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:53:25.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:25 smithi154 ceph-mon[33264]: pgmap v987: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 573 MiB data, 1.6 GiB used, 624 GiB / 626 GiB avail; 3.4 MiB/s rd, 5.8 MiB/s wr, 15 op/s 2024-09-20T22:53:25.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:25 smithi154 ceph-mon[33264]: osdmap e724: 8 total, 8 up, 7 in 2024-09-20T22:53:25.839 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:53:25.860 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:25.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:25.861 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:53:25.917 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-20T22:53:25.917 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:53:25.962 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:53:25.984 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:25.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:25.985 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:53:26.086 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:53:26.107 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:26.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:26.108 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:53:26.188 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:53:26.209 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:53:26.230 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:26.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:26.231 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:53:26.315 INFO:tasks.rados.rados.0.smithi084.stdout:2874: done (3 left) 2024-09-20T22:53:26.315 INFO:tasks.rados.rados.0.smithi084.stdout:2875: write oid 37 current snap is 326 2024-09-20T22:53:26.315 INFO:tasks.rados.rados.0.smithi084.stdout:2873: finishing rollback tid 1 to smithi08477203-17 2024-09-20T22:53:26.315 INFO:tasks.rados.rados.0.smithi084.stdout:2871: finishing write tid 2 to smithi08477203-2 2024-09-20T22:53:26.315 INFO:tasks.rados.rados.0.smithi084.stdout:2871: finishing write tid 3 to smithi08477203-2 2024-09-20T22:53:26.315 INFO:tasks.rados.rados.0.smithi084.stdout:2873: finishing rollback tid 2 to smithi08477203-17 2024-09-20T22:53:26.315 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 17 v 799 (ObjNum 815 snap 262 seq_num 815) dirty exists 2024-09-20T22:53:26.315 INFO:tasks.rados.rados.0.smithi084.stdout:2871: finishing write tid 4 to smithi08477203-2 2024-09-20T22:53:26.315 INFO:tasks.rados.rados.0.smithi084.stdout:2871: oid 2 updating version 0 to 1832 2024-09-20T22:53:26.315 INFO:tasks.rados.rados.0.smithi084.stdout:2871: oid 2 version 1832 is already newer than 1831 2024-09-20T22:53:26.315 INFO:tasks.rados.rados.0.smithi084.stdout:2871: oid 2 version 1832 is already newer than 1830 2024-09-20T22:53:26.316 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 2 v 1832 (ObjNum 993 snap 326 seq_num 993) dirty exists 2024-09-20T22:53:26.316 INFO:tasks.rados.rados.0.smithi084.stdout:2871: left oid 2 (ObjNum 993 snap 326 seq_num 993) 2024-09-20T22:53:26.316 INFO:tasks.rados.rados.0.smithi084.stdout:2872: finishing rollback tid 1 to smithi08477203-9 2024-09-20T22:53:26.316 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 9 v 1802 (ObjNum 33 snap 0 seq_num 2085660880) dirty dne 2024-09-20T22:53:26.316 INFO:tasks.rados.rados.0.smithi084.stdout:2875: seq_num 994 ranges {706158=723396,2183219=522939} 2024-09-20T22:53:26.323 INFO:tasks.rados.rados.0.smithi084.stdout:2875: writing smithi08477203-37 from 706158 to 1429554 tid 1 2024-09-20T22:53:26.330 INFO:tasks.rados.rados.0.smithi084.stdout:2875: writing smithi08477203-37 from 2183219 to 2706158 tid 2 2024-09-20T22:53:26.330 INFO:tasks.rados.rados.0.smithi084.stdout:2871: done (3 left) 2024-09-20T22:53:26.330 INFO:tasks.rados.rados.0.smithi084.stdout:2872: done (2 left) 2024-09-20T22:53:26.330 INFO:tasks.rados.rados.0.smithi084.stdout:2873: done (1 left) 2024-09-20T22:53:26.330 INFO:tasks.rados.rados.0.smithi084.stdout:2876: delete oid 2 current snap is 326 2024-09-20T22:53:26.330 INFO:tasks.rados.rados.0.smithi084.stdout:2875: finishing write tid 1 to smithi08477203-37 2024-09-20T22:53:26.332 INFO:tasks.rados.rados.0.smithi084.stdout:2876: done (1 left) 2024-09-20T22:53:26.332 INFO:tasks.rados.rados.0.smithi084.stdout:2877: read oid 50 snap -1 2024-09-20T22:53:26.332 INFO:tasks.rados.rados.0.smithi084.stdout:2877: expect (ObjNum 923 snap 305 seq_num 923) 2024-09-20T22:53:26.332 INFO:tasks.rados.rados.0.smithi084.stdout:2878: write oid 7 current snap is 326 2024-09-20T22:53:26.332 INFO:tasks.rados.rados.0.smithi084.stdout:2878: seq_num 996 ranges {739772=590247,1900183=540336} 2024-09-20T22:53:26.332 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:53:26.339 INFO:tasks.rados.rados.0.smithi084.stdout:2878: writing smithi08477203-7 from 739772 to 1330019 tid 1 2024-09-20T22:53:26.346 INFO:tasks.rados.rados.0.smithi084.stdout:2878: writing smithi08477203-7 from 1900183 to 2440519 tid 2 2024-09-20T22:53:26.346 INFO:tasks.rados.rados.0.smithi084.stdout:2879: delete oid 43 current snap is 326 2024-09-20T22:53:26.346 INFO:tasks.rados.rados.0.smithi084.stdout:2875: finishing write tid 2 to smithi08477203-37 2024-09-20T22:53:26.346 INFO:tasks.rados.rados.0.smithi084.stdout:2875: finishing write tid 3 to smithi08477203-37 2024-09-20T22:53:26.346 INFO:tasks.rados.rados.0.smithi084.stdout:2875: finishing write tid 4 to smithi08477203-37 2024-09-20T22:53:26.346 INFO:tasks.rados.rados.0.smithi084.stdout:2875: oid 37 updating version 0 to 1244 2024-09-20T22:53:26.346 INFO:tasks.rados.rados.0.smithi084.stdout:2875: oid 37 version 1244 is already newer than 1242 2024-09-20T22:53:26.346 INFO:tasks.rados.rados.0.smithi084.stdout:2875: oid 37 version 1244 is already newer than 1243 2024-09-20T22:53:26.346 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 37 v 1244 (ObjNum 994 snap 326 seq_num 994) dirty exists 2024-09-20T22:53:26.346 INFO:tasks.rados.rados.0.smithi084.stdout:2875: left oid 37 (ObjNum 994 snap 326 seq_num 994) 2024-09-20T22:53:26.346 INFO:tasks.rados.rados.0.smithi084.stdout:2878: finishing write tid 1 to smithi08477203-7 2024-09-20T22:53:26.347 INFO:tasks.rados.rados.0.smithi084.stdout:2877: expect (ObjNum 923 snap 305 seq_num 923) 2024-09-20T22:53:26.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:26.356 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:26.356 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:53:26.457 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:53:26.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:26.479 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:26.480 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:53:26.521 INFO:tasks.rados.rados.0.smithi084.stdout:2878: finishing write tid 2 to smithi08477203-7 2024-09-20T22:53:26.522 INFO:tasks.rados.rados.0.smithi084.stdout:2878: finishing write tid 3 to smithi08477203-7 2024-09-20T22:53:26.522 INFO:tasks.rados.rados.0.smithi084.stdout:2878: finishing write tid 4 to smithi08477203-7 2024-09-20T22:53:26.522 INFO:tasks.rados.rados.0.smithi084.stdout:2878: oid 7 updating version 0 to 1246 2024-09-20T22:53:26.522 INFO:tasks.rados.rados.0.smithi084.stdout:2878: oid 7 version 1246 is already newer than 1245 2024-09-20T22:53:26.522 INFO:tasks.rados.rados.0.smithi084.stdout:2878: oid 7 updating version 1246 to 1247 2024-09-20T22:53:26.522 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 7 v 1247 (ObjNum 996 snap 326 seq_num 996) dirty exists 2024-09-20T22:53:26.522 INFO:tasks.rados.rados.0.smithi084.stdout:2878: left oid 7 (ObjNum 996 snap 326 seq_num 996) 2024-09-20T22:53:26.522 INFO:tasks.rados.rados.0.smithi084.stdout:2875: done (3 left) 2024-09-20T22:53:26.522 INFO:tasks.rados.rados.0.smithi084.stdout:2877: done (2 left) 2024-09-20T22:53:26.522 INFO:tasks.rados.rados.0.smithi084.stdout:2878: done (1 left) 2024-09-20T22:53:26.522 INFO:tasks.rados.rados.0.smithi084.stdout:2879: done (0 left) 2024-09-20T22:53:26.523 INFO:tasks.rados.rados.0.smithi084.stdout:2880: read oid 16 snap -1 2024-09-20T22:53:26.523 INFO:tasks.rados.rados.0.smithi084.stdout:2880: expect (ObjNum 876 snap 280 seq_num 876) 2024-09-20T22:53:26.523 INFO:tasks.rados.rados.0.smithi084.stdout:2881: copy_from oid 45 from oid 24 current snap is 326 2024-09-20T22:53:26.523 INFO:tasks.rados.rados.0.smithi084.stdout:2882: read oid 3 snap -1 2024-09-20T22:53:26.523 INFO:tasks.rados.rados.0.smithi084.stdout:2882: expect (ObjNum 934 snap 308 seq_num 934) 2024-09-20T22:53:26.523 INFO:tasks.rados.rados.0.smithi084.stdout:2883: rollback oid 10 current snap is 326 2024-09-20T22:53:26.523 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 10 to 290 2024-09-20T22:53:26.523 INFO:tasks.rados.rados.0.smithi084.stdout:2884: delete oid 34 current snap is 326 2024-09-20T22:53:26.524 INFO:tasks.rados.rados.0.smithi084.stdout:2884: done (4 left) 2024-09-20T22:53:26.524 INFO:tasks.rados.rados.0.smithi084.stdout:2885: rollback oid 5 current snap is 326 2024-09-20T22:53:26.524 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 5 to 311 2024-09-20T22:53:26.524 INFO:tasks.rados.rados.0.smithi084.stdout:2886: delete oid 46 current snap is 326 2024-09-20T22:53:26.524 INFO:tasks.rados.rados.0.smithi084.stdout:2883: finishing rollback tid 0 to smithi08477203-10 2024-09-20T22:53:26.525 INFO:tasks.rados.rados.0.smithi084.stdout:2883: finishing rollback tid 1 to smithi08477203-10 2024-09-20T22:53:26.525 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 10 v 1826 (ObjNum 129 snap 0 seq_num 2091192896) dirty dne 2024-09-20T22:53:26.527 INFO:tasks.rados.rados.0.smithi084.stdout:2885: finishing rollback tid 0 to smithi08477203-5 2024-09-20T22:53:26.527 INFO:tasks.rados.rados.0.smithi084.stdout:2882: expect (ObjNum 934 snap 308 seq_num 934) 2024-09-20T22:53:26.580 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:53:26.601 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:26.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:26.602 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:53:26.614 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:26 smithi084 ceph-mon[29774]: 5.9 scrub starts 2024-09-20T22:53:26.614 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:26 smithi084 ceph-mon[29774]: 5.9 scrub ok 2024-09-20T22:53:26.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:26 smithi084 ceph-mon[29774]: 3.1 scrub starts 2024-09-20T22:53:26.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:26 smithi084 ceph-mon[29774]: 3.1 scrub ok 2024-09-20T22:53:26.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:26 smithi084 ceph-mon[29774]: 5.6 scrub starts 2024-09-20T22:53:26.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:26 smithi084 ceph-mon[29774]: 5.6 scrub ok 2024-09-20T22:53:26.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:26 smithi084 ceph-mon[29774]: osdmap e725: 8 total, 8 up, 7 in 2024-09-20T22:53:26.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:26 smithi084 ceph-mon[40024]: 5.9 scrub starts 2024-09-20T22:53:26.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:26 smithi084 ceph-mon[40024]: 5.9 scrub ok 2024-09-20T22:53:26.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:26 smithi084 ceph-mon[40024]: 3.1 scrub starts 2024-09-20T22:53:26.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:26 smithi084 ceph-mon[40024]: 3.1 scrub ok 2024-09-20T22:53:26.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:26 smithi084 ceph-mon[40024]: 5.6 scrub starts 2024-09-20T22:53:26.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:26 smithi084 ceph-mon[40024]: 5.6 scrub ok 2024-09-20T22:53:26.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:26 smithi084 ceph-mon[40024]: osdmap e725: 8 total, 8 up, 7 in 2024-09-20T22:53:26.673 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-20T22:53:26.692 INFO:tasks.rados.rados.0.smithi084.stdout:2885: finishing rollback tid 1 to smithi08477203-5 2024-09-20T22:53:26.692 INFO:tasks.rados.rados.0.smithi084.stdout:2885: finishing rollback tid 2 to smithi08477203-5 2024-09-20T22:53:26.692 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 5 v 891 (ObjNum 922 snap 305 seq_num 922) dirty exists 2024-09-20T22:53:26.692 INFO:tasks.rados.rados.0.smithi084.stdout:2882: done (5 left) 2024-09-20T22:53:26.693 INFO:tasks.rados.rados.0.smithi084.stdout:2883: done (4 left) 2024-09-20T22:53:26.693 INFO:tasks.rados.rados.0.smithi084.stdout:2885: done (3 left) 2024-09-20T22:53:26.693 INFO:tasks.rados.rados.0.smithi084.stdout:2886: done (2 left) 2024-09-20T22:53:26.693 INFO:tasks.rados.rados.0.smithi084.stdout:2887: snap_create 2024-09-20T22:53:26.693 INFO:tasks.rados.rados.0.smithi084.stdout:2881: finishing copy_from racing read to smithi08477203-45 2024-09-20T22:53:26.693 INFO:tasks.rados.rados.0.smithi084.stdout:2880: expect (ObjNum 876 snap 280 seq_num 876) 2024-09-20T22:53:26.703 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:53:26.724 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:26.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:26.725 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:53:26.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:26 smithi154 ceph-mon[33264]: 5.9 scrub starts 2024-09-20T22:53:26.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:26 smithi154 ceph-mon[33264]: 5.9 scrub ok 2024-09-20T22:53:26.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:26 smithi154 ceph-mon[33264]: 3.1 scrub starts 2024-09-20T22:53:26.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:26 smithi154 ceph-mon[33264]: 3.1 scrub ok 2024-09-20T22:53:26.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:26 smithi154 ceph-mon[33264]: 5.6 scrub starts 2024-09-20T22:53:26.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:26 smithi154 ceph-mon[33264]: 5.6 scrub ok 2024-09-20T22:53:26.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:26 smithi154 ceph-mon[33264]: osdmap e725: 8 total, 8 up, 7 in 2024-09-20T22:53:26.790 INFO:tasks.rados.rados.0.smithi084.stdout:2881: finishing copy_from to smithi08477203-45 2024-09-20T22:53:26.790 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 45 v 1839 (ObjNum 780 snap 252 seq_num 780) dirty exists 2024-09-20T22:53:26.826 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:53:26.847 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:26.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:26.848 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:53:26.886 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:53:26.896 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-20T22:53:26.948 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:53:26.969 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:26.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:26.970 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:53:27.071 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:53:27.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:27.093 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:27.093 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:53:27.194 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:53:27.217 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:27.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:27.218 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:53:27.308 INFO:tasks.rados.rados.0.smithi084.stdout:2880: done (2 left) 2024-09-20T22:53:27.308 INFO:tasks.rados.rados.0.smithi084.stdout:2881: done (1 left) 2024-09-20T22:53:27.308 INFO:tasks.rados.rados.0.smithi084.stdout:2887: done (0 left) 2024-09-20T22:53:27.308 INFO:tasks.rados.rados.0.smithi084.stdout:2888: delete oid 28 current snap is 327 2024-09-20T22:53:27.318 INFO:tasks.rados.rados.0.smithi084.stdout:2888: done (0 left) 2024-09-20T22:53:27.318 INFO:tasks.rados.rados.0.smithi084.stdout:2889: snap_create 2024-09-20T22:53:27.318 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:53:27.340 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:27.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:27.341 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:53:27.442 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:53:27.463 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:27.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:27.464 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:53:27.528 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:27 smithi084 ceph-mon[40024]: pgmap v990: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 573 MiB data, 1.6 GiB used, 624 GiB / 626 GiB avail; 3.4 MiB/s rd, 5.8 MiB/s wr, 15 op/s 2024-09-20T22:53:27.528 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:27 smithi084 ceph-mon[40024]: 4.1e scrub starts 2024-09-20T22:53:27.528 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:27 smithi084 ceph-mon[40024]: 4.1e scrub ok 2024-09-20T22:53:27.528 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:27 smithi084 ceph-mon[40024]: osdmap e726: 8 total, 8 up, 7 in 2024-09-20T22:53:27.528 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:27 smithi084 ceph-mon[40024]: 4.17 scrub starts 2024-09-20T22:53:27.528 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:27 smithi084 ceph-mon[40024]: 4.17 scrub ok 2024-09-20T22:53:27.528 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:27 smithi084 ceph-mon[40024]: 5.1 scrub starts 2024-09-20T22:53:27.528 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:27 smithi084 ceph-mon[40024]: 5.1 scrub ok 2024-09-20T22:53:27.528 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:27 smithi084 ceph-mon[29774]: pgmap v990: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 573 MiB data, 1.6 GiB used, 624 GiB / 626 GiB avail; 3.4 MiB/s rd, 5.8 MiB/s wr, 15 op/s 2024-09-20T22:53:27.529 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:27 smithi084 ceph-mon[29774]: 4.1e scrub starts 2024-09-20T22:53:27.529 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:27 smithi084 ceph-mon[29774]: 4.1e scrub ok 2024-09-20T22:53:27.529 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:27 smithi084 ceph-mon[29774]: osdmap e726: 8 total, 8 up, 7 in 2024-09-20T22:53:27.529 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:27 smithi084 ceph-mon[29774]: 4.17 scrub starts 2024-09-20T22:53:27.529 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:27 smithi084 ceph-mon[29774]: 4.17 scrub ok 2024-09-20T22:53:27.529 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:27 smithi084 ceph-mon[29774]: 5.1 scrub starts 2024-09-20T22:53:27.529 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:27 smithi084 ceph-mon[29774]: 5.1 scrub ok 2024-09-20T22:53:27.529 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:53:27.529 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:53:27.545 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:53:27.545 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:53:27.561 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:53:27.561 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:53:27.564 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:53:27.576 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-20T22:53:27.577 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T22:53:27.586 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:27.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:27.587 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:53:27.593 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:53:27.593 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:53:27.609 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:53:27.609 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:53:27.626 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:53:27.626 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:53:27.642 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:53:27.642 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:53:27.687 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:53:27.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:27.710 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:27.710 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:53:27.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:27 smithi154 ceph-mon[33264]: pgmap v990: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 573 MiB data, 1.6 GiB used, 624 GiB / 626 GiB avail; 3.4 MiB/s rd, 5.8 MiB/s wr, 15 op/s 2024-09-20T22:53:27.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:27 smithi154 ceph-mon[33264]: 4.1e scrub starts 2024-09-20T22:53:27.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:27 smithi154 ceph-mon[33264]: 4.1e scrub ok 2024-09-20T22:53:27.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:27 smithi154 ceph-mon[33264]: osdmap e726: 8 total, 8 up, 7 in 2024-09-20T22:53:27.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:27 smithi154 ceph-mon[33264]: 4.17 scrub starts 2024-09-20T22:53:27.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:27 smithi154 ceph-mon[33264]: 4.17 scrub ok 2024-09-20T22:53:27.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:27 smithi154 ceph-mon[33264]: 5.1 scrub starts 2024-09-20T22:53:27.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:27 smithi154 ceph-mon[33264]: 5.1 scrub ok 2024-09-20T22:53:27.811 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:53:27.833 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:27.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:27.834 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:53:27.934 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:53:27.956 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:27.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:27.957 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:53:28.057 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:53:28.079 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:28.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:28.080 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:53:28.180 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:53:28.202 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:28.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:28.203 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:53:28.290 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.3 to 1 (10000) 2024-09-20T22:53:28.297 INFO:tasks.rados.rados.0.smithi084.stdout:2889: done (0 left) 2024-09-20T22:53:28.297 INFO:tasks.rados.rados.0.smithi084.stdout:2890: rollback oid 15 current snap is 328 2024-09-20T22:53:28.297 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 15 to 316 2024-09-20T22:53:28.297 INFO:tasks.rados.rados.0.smithi084.stdout:2891: write (excl) oid 12 current snap is 328 2024-09-20T22:53:28.297 INFO:tasks.rados.rados.0.smithi084.stdout:2891: seq_num 1001 ranges {423807=566492,1676240=407685,2739004=84803} 2024-09-20T22:53:28.303 INFO:tasks.rados.rados.0.smithi084.stdout:2891: writing smithi08477203-12 from 423807 to 990299 tid 1 2024-09-20T22:53:28.303 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:53:28.310 INFO:tasks.rados.rados.0.smithi084.stdout:2891: writing smithi08477203-12 from 1676240 to 2083925 tid 2 2024-09-20T22:53:28.315 INFO:tasks.rados.rados.0.smithi084.stdout:2891: writing smithi08477203-12 from 2739004 to 2823807 tid 3 2024-09-20T22:53:28.315 INFO:tasks.rados.rados.0.smithi084.stdout:2892: write oid 45 current snap is 328 2024-09-20T22:53:28.315 INFO:tasks.rados.rados.0.smithi084.stdout:2892: seq_num 1002 ranges {440614=641741,1734722=551343} 2024-09-20T22:53:28.322 INFO:tasks.rados.rados.0.smithi084.stdout:2892: writing smithi08477203-45 from 440614 to 1082355 tid 1 2024-09-20T22:53:28.329 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:28.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:28.330 INFO:tasks.rados.rados.0.smithi084.stdout:2892: writing smithi08477203-45 from 1734722 to 2286065 tid 2 2024-09-20T22:53:28.330 INFO:tasks.rados.rados.0.smithi084.stdout:2893: read oid 8 snap -1 2024-09-20T22:53:28.330 INFO:tasks.rados.rados.0.smithi084.stdout:2893: expect (ObjNum 923 snap 305 seq_num 923) 2024-09-20T22:53:28.330 INFO:tasks.rados.rados.0.smithi084.stdout:2894: copy_from oid 27 from oid 38 current snap is 328 2024-09-20T22:53:28.330 INFO:tasks.rados.rados.0.smithi084.stdout:2890: finishing rollback tid 0 to smithi08477203-15 2024-09-20T22:53:28.330 INFO:tasks.rados.rados.0.smithi084.stdout:2890: finishing rollback tid 1 to smithi08477203-15 2024-09-20T22:53:28.330 INFO:tasks.rados.rados.0.smithi084.stdout:2890: finishing rollback tid 2 to smithi08477203-15 2024-09-20T22:53:28.331 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 15 v 1842 (ObjNum 890 snap 289 seq_num 890) dirty exists 2024-09-20T22:53:28.331 INFO:tasks.rados.rados.0.smithi084.stdout:2890: done (4 left) 2024-09-20T22:53:28.331 INFO:tasks.rados.rados.0.smithi084.stdout:2895: delete oid 34 current snap is 328 2024-09-20T22:53:28.331 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:53:28.337 INFO:tasks.rados.rados.0.smithi084.stdout:2893: expect (ObjNum 923 snap 305 seq_num 923) 2024-09-20T22:53:28.432 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:53:28.453 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:28.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:28.454 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:53:28.516 INFO:tasks.rados.rados.0.smithi084.stdout:2894: finishing copy_from racing read to smithi08477203-27 2024-09-20T22:53:28.516 INFO:tasks.rados.rados.0.smithi084.stdout:2892: finishing write tid 1 to smithi08477203-45 2024-09-20T22:53:28.516 INFO:tasks.rados.rados.0.smithi084.stdout:2892: finishing write tid 2 to smithi08477203-45 2024-09-20T22:53:28.517 INFO:tasks.rados.rados.0.smithi084.stdout:2892: finishing write tid 3 to smithi08477203-45 2024-09-20T22:53:28.517 INFO:tasks.rados.rados.0.smithi084.stdout:2891: finishing write tid 1 to smithi08477203-12 2024-09-20T22:53:28.517 INFO:tasks.rados.rados.0.smithi084.stdout:2893: done (4 left) 2024-09-20T22:53:28.517 INFO:tasks.rados.rados.0.smithi084.stdout:2895: done (3 left) 2024-09-20T22:53:28.517 INFO:tasks.rados.rados.0.smithi084.stdout:2896: write oid 6 current snap is 328 2024-09-20T22:53:28.517 INFO:tasks.rados.rados.0.smithi084.stdout:2891: finishing write tid 2 to smithi08477203-12 2024-09-20T22:53:28.517 INFO:tasks.rados.rados.0.smithi084.stdout:2891: finishing write tid 3 to smithi08477203-12 2024-09-20T22:53:28.517 INFO:tasks.rados.rados.0.smithi084.stdout:2891: finishing write tid 4 to smithi08477203-12 2024-09-20T22:53:28.517 INFO:tasks.rados.rados.0.smithi084.stdout:2891: finishing write tid 5 to smithi08477203-12 2024-09-20T22:53:28.517 INFO:tasks.rados.rados.0.smithi084.stdout:2891: oid 12 updating version 0 to 1808 2024-09-20T22:53:28.517 INFO:tasks.rados.rados.0.smithi084.stdout:2891: oid 12 version 1808 is already newer than 1806 2024-09-20T22:53:28.517 INFO:tasks.rados.rados.0.smithi084.stdout:2891: oid 12 updating version 1808 to 1809 2024-09-20T22:53:28.517 INFO:tasks.rados.rados.0.smithi084.stdout:2891: oid 12 version 1809 is already newer than 1807 2024-09-20T22:53:28.517 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 12 v 1809 (ObjNum 1001 snap 328 seq_num 1001) dirty exists 2024-09-20T22:53:28.518 INFO:tasks.rados.rados.0.smithi084.stdout:2891: left oid 12 (ObjNum 1001 snap 328 seq_num 1001) 2024-09-20T22:53:28.518 INFO:tasks.rados.rados.0.smithi084.stdout:2892: finishing write tid 4 to smithi08477203-45 2024-09-20T22:53:28.518 INFO:tasks.rados.rados.0.smithi084.stdout:2892: oid 45 updating version 0 to 1843 2024-09-20T22:53:28.518 INFO:tasks.rados.rados.0.smithi084.stdout:2892: oid 45 updating version 1843 to 1845 2024-09-20T22:53:28.518 INFO:tasks.rados.rados.0.smithi084.stdout:2892: oid 45 version 1845 is already newer than 1844 2024-09-20T22:53:28.518 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 45 v 1845 (ObjNum 1002 snap 328 seq_num 1002) dirty exists 2024-09-20T22:53:28.518 INFO:tasks.rados.rados.0.smithi084.stdout:2892: left oid 45 (ObjNum 1002 snap 328 seq_num 1002) 2024-09-20T22:53:28.518 INFO:tasks.rados.rados.0.smithi084.stdout:2894: finishing copy_from to smithi08477203-27 2024-09-20T22:53:28.518 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 27 v 1805 (ObjNum 918 snap 302 seq_num 918) dirty exists 2024-09-20T22:53:28.518 INFO:tasks.rados.rados.0.smithi084.stdout:2896: seq_num 1004 ranges {474228=508592,1451686=555012,2704920=169308} 2024-09-20T22:53:28.521 INFO:tasks.rados.rados.0.smithi084.stdout:2896: writing smithi08477203-6 from 474228 to 982820 tid 1 2024-09-20T22:53:28.527 INFO:tasks.rados.rados.0.smithi084.stdout:2896: writing smithi08477203-6 from 1451686 to 2006698 tid 2 2024-09-20T22:53:28.532 INFO:tasks.rados.rados.0.smithi084.stdout:2896: writing smithi08477203-6 from 2704920 to 2874228 tid 3 2024-09-20T22:53:28.532 INFO:tasks.rados.rados.0.smithi084.stdout:2896: finishing write tid 1 to smithi08477203-6 2024-09-20T22:53:28.532 INFO:tasks.rados.rados.0.smithi084.stdout:2896: finishing write tid 2 to smithi08477203-6 2024-09-20T22:53:28.532 INFO:tasks.rados.rados.0.smithi084.stdout:2891: done (3 left) 2024-09-20T22:53:28.533 INFO:tasks.rados.rados.0.smithi084.stdout:2892: done (2 left) 2024-09-20T22:53:28.533 INFO:tasks.rados.rados.0.smithi084.stdout:2894: done (1 left) 2024-09-20T22:53:28.533 INFO:tasks.rados.rados.0.smithi084.stdout:2897: read oid 24 snap 298 2024-09-20T22:53:28.533 INFO:tasks.rados.rados.0.smithi084.stdout:2897: expect (ObjNum 907 snap 297 seq_num 907) 2024-09-20T22:53:28.533 INFO:tasks.rados.rados.0.smithi084.stdout:2898: read oid 40 snap 290 2024-09-20T22:53:28.533 INFO:tasks.rados.rados.0.smithi084.stdout:2898: expect (ObjNum 833 snap 269 seq_num 833) 2024-09-20T22:53:28.533 INFO:tasks.rados.rados.0.smithi084.stdout:2899: snap_create 2024-09-20T22:53:28.534 INFO:tasks.rados.rados.0.smithi084.stdout:2896: finishing write tid 3 to smithi08477203-6 2024-09-20T22:53:28.534 INFO:tasks.rados.rados.0.smithi084.stdout:2896: finishing write tid 4 to smithi08477203-6 2024-09-20T22:53:28.535 INFO:tasks.rados.rados.0.smithi084.stdout:2896: finishing write tid 5 to smithi08477203-6 2024-09-20T22:53:28.535 INFO:tasks.rados.rados.0.smithi084.stdout:2896: oid 6 updating version 0 to 801 2024-09-20T22:53:28.535 INFO:tasks.rados.rados.0.smithi084.stdout:2896: oid 6 updating version 801 to 803 2024-09-20T22:53:28.535 INFO:tasks.rados.rados.0.smithi084.stdout:2896: oid 6 version 803 is already newer than 802 2024-09-20T22:53:28.535 INFO:tasks.rados.rados.0.smithi084.stdout:2896: oid 6 version 803 is already newer than 800 2024-09-20T22:53:28.535 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 6 v 803 (ObjNum 1004 snap 328 seq_num 1004) dirty exists 2024-09-20T22:53:28.535 INFO:tasks.rados.rados.0.smithi084.stdout:2896: left oid 6 (ObjNum 1004 snap 328 seq_num 1004) 2024-09-20T22:53:28.536 INFO:tasks.rados.rados.0.smithi084.stdout:2898: expect (ObjNum 833 snap 269 seq_num 833) 2024-09-20T22:53:28.555 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:53:28.575 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:28.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:28.576 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:53:28.614 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:53:28 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: ::ffff:172.21.15.154 - - [20/Sep/2024:22:53:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-20T22:53:28.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:28 smithi084 ceph-mon[40024]: purged_snaps scrub starts 2024-09-20T22:53:28.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:28 smithi084 ceph-mon[40024]: purged_snaps scrub ok 2024-09-20T22:53:28.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:28 smithi084 ceph-mon[40024]: osdmap e727: 8 total, 8 up, 7 in 2024-09-20T22:53:28.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:28 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/498863861' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:53:28.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:28 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/397098945' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-09-20T22:53:28.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:28 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/397098945' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-20T22:53:28.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:28 smithi084 ceph-mon[40024]: osdmap e728: 8 total, 8 up, 7 in 2024-09-20T22:53:28.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:28 smithi084 ceph-mon[29774]: purged_snaps scrub starts 2024-09-20T22:53:28.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:28 smithi084 ceph-mon[29774]: purged_snaps scrub ok 2024-09-20T22:53:28.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:28 smithi084 ceph-mon[29774]: osdmap e727: 8 total, 8 up, 7 in 2024-09-20T22:53:28.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:28 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/498863861' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:53:28.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:28 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/397098945' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-09-20T22:53:28.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:28 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/397098945' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-20T22:53:28.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:28 smithi084 ceph-mon[29774]: osdmap e728: 8 total, 8 up, 7 in 2024-09-20T22:53:28.671 INFO:tasks.rados.rados.0.smithi084.stdout:2897: expect (ObjNum 907 snap 297 seq_num 907) 2024-09-20T22:53:28.678 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:53:28.699 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:28.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:28.700 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:53:28.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:28 smithi154 ceph-mon[33264]: purged_snaps scrub starts 2024-09-20T22:53:28.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:28 smithi154 ceph-mon[33264]: purged_snaps scrub ok 2024-09-20T22:53:28.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:28 smithi154 ceph-mon[33264]: osdmap e727: 8 total, 8 up, 7 in 2024-09-20T22:53:28.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:28 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/498863861' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:53:28.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:28 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/397098945' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-09-20T22:53:28.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:28 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/397098945' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-20T22:53:28.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:28 smithi154 ceph-mon[33264]: osdmap e728: 8 total, 8 up, 7 in 2024-09-20T22:53:28.801 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:53:28.822 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:28.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:28.823 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:53:28.924 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:53:28.946 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:28.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:28.947 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:53:29.048 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:53:29.069 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:29.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:29.070 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:53:29.094 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:53:29.100 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd reweight 5 1 2024-09-20T22:53:29.134 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:53:29.171 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:53:29.192 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:29.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:29.192 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:53:29.293 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:53:29.315 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:29.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:29.315 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:53:29.333 INFO:tasks.rados.rados.0.smithi084.stdout:2896: done (3 left) 2024-09-20T22:53:29.333 INFO:tasks.rados.rados.0.smithi084.stdout:2897: done (2 left) 2024-09-20T22:53:29.333 INFO:tasks.rados.rados.0.smithi084.stdout:2898: done (1 left) 2024-09-20T22:53:29.333 INFO:tasks.rados.rados.0.smithi084.stdout:2899: done (0 left) 2024-09-20T22:53:29.333 INFO:tasks.rados.rados.0.smithi084.stdout:2900: read oid 21 snap -1 2024-09-20T22:53:29.333 INFO:tasks.rados.rados.0.smithi084.stdout:2900: expect (ObjNum 984 snap 319 seq_num 984) 2024-09-20T22:53:29.333 INFO:tasks.rados.rados.0.smithi084.stdout:2901: snap_create 2024-09-20T22:53:29.339 INFO:tasks.rados.rados.0.smithi084.stdout:2900: expect (ObjNum 984 snap 319 seq_num 984) 2024-09-20T22:53:29.343 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:29 smithi084 ceph-mon[40024]: 3.d scrub starts 2024-09-20T22:53:29.343 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:29 smithi084 ceph-mon[40024]: 3.d scrub ok 2024-09-20T22:53:29.343 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:29 smithi084 ceph-mon[40024]: pgmap v993: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 573 MiB data, 1.6 GiB used, 624 GiB / 626 GiB avail 2024-09-20T22:53:29.343 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:29 smithi084 ceph-mon[40024]: 1.a scrub starts 2024-09-20T22:53:29.343 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:29 smithi084 ceph-mon[40024]: 1.a scrub ok 2024-09-20T22:53:29.343 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:29 smithi084 ceph-mon[29774]: 3.d scrub starts 2024-09-20T22:53:29.343 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:29 smithi084 ceph-mon[29774]: 3.d scrub ok 2024-09-20T22:53:29.343 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:29 smithi084 ceph-mon[29774]: pgmap v993: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 573 MiB data, 1.6 GiB used, 624 GiB / 626 GiB avail 2024-09-20T22:53:29.343 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:29 smithi084 ceph-mon[29774]: 1.a scrub starts 2024-09-20T22:53:29.343 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:29 smithi084 ceph-mon[29774]: 1.a scrub ok 2024-09-20T22:53:29.416 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:53:29.437 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:29.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:29.438 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:53:29.539 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:53:29.560 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:29.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:29.561 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:53:29.662 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:53:29.684 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:29.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:29.685 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:53:29.754 INFO:journalctl@ceph.iscsi.iscsi.a.smithi154.stdout:Sep 20 22:53:29 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-iscsi-iscsi-a[59730]: debug there is no tcmu-runner data available 2024-09-20T22:53:29.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:29 smithi154 ceph-mon[33264]: 3.d scrub starts 2024-09-20T22:53:29.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:29 smithi154 ceph-mon[33264]: 3.d scrub ok 2024-09-20T22:53:29.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:29 smithi154 ceph-mon[33264]: pgmap v993: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 573 MiB data, 1.6 GiB used, 624 GiB / 626 GiB avail 2024-09-20T22:53:29.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:29 smithi154 ceph-mon[33264]: 1.a scrub starts 2024-09-20T22:53:29.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:29 smithi154 ceph-mon[33264]: 1.a scrub ok 2024-09-20T22:53:29.786 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:53:29.807 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:29.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:29.808 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:53:29.909 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:53:29.930 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:29.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:29.931 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:53:30.032 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:53:30.035 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-20T22:53:30.035 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2024-09-20T22:53:30.035 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2024-09-20T22:53:30.035 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-20T22:53:30.054 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:30.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:30.055 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:53:30.157 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:53:30.179 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:30.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:30.180 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:53:30.281 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:53:30.302 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:30.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:30.303 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:53:30.333 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is unset 2024-09-20T22:53:30.342 INFO:tasks.rados.rados.0.smithi084.stdout:2900: done (1 left) 2024-09-20T22:53:30.342 INFO:tasks.rados.rados.0.smithi084.stdout:2901: done (0 left) 2024-09-20T22:53:30.342 INFO:tasks.rados.rados.0.smithi084.stdout:2902: delete oid 4 current snap is 330 2024-09-20T22:53:30.346 INFO:tasks.rados.rados.0.smithi084.stdout:2902: done (0 left) 2024-09-20T22:53:30.346 INFO:tasks.rados.rados.0.smithi084.stdout:2903: read oid 45 snap -1 2024-09-20T22:53:30.346 INFO:tasks.rados.rados.0.smithi084.stdout:2903: expect (ObjNum 1002 snap 328 seq_num 1002) 2024-09-20T22:53:30.346 INFO:tasks.rados.rados.0.smithi084.stdout:2904: write oid 50 current snap is 330 2024-09-20T22:53:30.346 INFO:tasks.rados.rados.0.smithi084.stdout:2904: seq_num 1006 ranges {507842=659090,1568650=558681,2759766=148076} 2024-09-20T22:53:30.353 INFO:tasks.rados.rados.0.smithi084.stdout:2904: writing smithi08477203-50 from 507842 to 1166932 tid 1 2024-09-20T22:53:30.359 INFO:tasks.rados.rados.0.smithi084.stdout:2904: writing smithi08477203-50 from 1568650 to 2127331 tid 2 2024-09-20T22:53:30.364 INFO:tasks.rados.rados.0.smithi084.stdout:2904: writing smithi08477203-50 from 2759766 to 2907842 tid 3 2024-09-20T22:53:30.364 INFO:tasks.rados.rados.0.smithi084.stdout:2905: read oid 25 snap -1 2024-09-20T22:53:30.364 INFO:tasks.rados.rados.0.smithi084.stdout:2905: expect deleted 2024-09-20T22:53:30.365 INFO:tasks.rados.rados.0.smithi084.stdout:2903: expect (ObjNum 1002 snap 328 seq_num 1002) 2024-09-20T22:53:30.403 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:53:30.419 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-20T22:53:30.425 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:30.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:30.425 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:53:30.519 INFO:tasks.rados.rados.0.smithi084.stdout:2904: finishing write tid 1 to smithi08477203-50 2024-09-20T22:53:30.519 INFO:tasks.rados.rados.0.smithi084.stdout:2904: finishing write tid 2 to smithi08477203-50 2024-09-20T22:53:30.519 INFO:tasks.rados.rados.0.smithi084.stdout:2904: finishing write tid 3 to smithi08477203-50 2024-09-20T22:53:30.519 INFO:tasks.rados.rados.0.smithi084.stdout:2904: finishing write tid 4 to smithi08477203-50 2024-09-20T22:53:30.519 INFO:tasks.rados.rados.0.smithi084.stdout:2904: finishing write tid 5 to smithi08477203-50 2024-09-20T22:53:30.519 INFO:tasks.rados.rados.0.smithi084.stdout:2904: oid 50 updating version 0 to 1849 2024-09-20T22:53:30.519 INFO:tasks.rados.rados.0.smithi084.stdout:2904: oid 50 version 1849 is already newer than 1848 2024-09-20T22:53:30.519 INFO:tasks.rados.rados.0.smithi084.stdout:2904: oid 50 version 1849 is already newer than 1846 2024-09-20T22:53:30.519 INFO:tasks.rados.rados.0.smithi084.stdout:2904: oid 50 version 1849 is already newer than 1847 2024-09-20T22:53:30.519 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 50 v 1849 (ObjNum 1006 snap 330 seq_num 1006) dirty exists 2024-09-20T22:53:30.520 INFO:tasks.rados.rados.0.smithi084.stdout:2904: left oid 50 (ObjNum 1006 snap 330 seq_num 1006) 2024-09-20T22:53:30.520 INFO:tasks.rados.rados.0.smithi084.stdout:2903: done (2 left) 2024-09-20T22:53:30.520 INFO:tasks.rados.rados.0.smithi084.stdout:2904: done (1 left) 2024-09-20T22:53:30.520 INFO:tasks.rados.rados.0.smithi084.stdout:2906: snap_remove snap 329 2024-09-20T22:53:30.526 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:53:30.547 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:30.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:30.548 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:53:30.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:30 smithi084 ceph-mon[29774]: osdmap e729: 8 total, 8 up, 7 in 2024-09-20T22:53:30.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:30 smithi084 ceph-mon[29774]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:53:30.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:30 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/2102769555' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:53:30.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:30 smithi084 ceph-mon[40024]: osdmap e729: 8 total, 8 up, 7 in 2024-09-20T22:53:30.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:30 smithi084 ceph-mon[40024]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:53:30.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:30 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/2102769555' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:53:30.649 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:53:30.670 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:30.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:30.671 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:53:30.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:30 smithi154 ceph-mon[33264]: osdmap e729: 8 total, 8 up, 7 in 2024-09-20T22:53:30.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:30 smithi154 ceph-mon[33264]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:53:30.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:30 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/2102769555' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:53:30.774 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:53:30.795 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:30.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:30.796 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:53:30.897 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:53:30.901 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.1 dump_blocked_ops 2024-09-20T22:53:30.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:30.918 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:30.918 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:53:31.020 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:53:31.042 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:31.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:31.043 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:53:31.144 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:53:31.167 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:31.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:31.168 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:53:31.268 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:53:31.290 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:31.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:31.291 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:53:31.339 INFO:tasks.rados.rados.0.smithi084.stdout:2906: done (1 left) 2024-09-20T22:53:31.339 INFO:tasks.rados.rados.0.smithi084.stdout:2907: snap_create 2024-09-20T22:53:31.393 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:53:31.414 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:31.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:31.415 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:53:31.513 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:31 smithi084 ceph-mon[29774]: pgmap v996: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 562 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 22 MiB/s rd, 3.5 MiB/s wr, 29 op/s 2024-09-20T22:53:31.513 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:31 smithi084 ceph-mon[29774]: 3.0 scrub starts 2024-09-20T22:53:31.513 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:31 smithi084 ceph-mon[29774]: 3.0 scrub ok 2024-09-20T22:53:31.513 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:31 smithi084 ceph-mon[29774]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T22:53:31.513 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:31 smithi084 ceph-mon[29774]: Cluster is now healthy 2024-09-20T22:53:31.513 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:31 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/2102769555' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T22:53:31.513 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:31 smithi084 ceph-mon[29774]: osdmap e730: 8 total, 8 up, 7 in 2024-09-20T22:53:31.513 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:31 smithi084 ceph-mon[29774]: 4.1f scrub starts 2024-09-20T22:53:31.513 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:31 smithi084 ceph-mon[29774]: 4.1f scrub ok 2024-09-20T22:53:31.514 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:31 smithi084 ceph-mon[40024]: pgmap v996: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 562 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 22 MiB/s rd, 3.5 MiB/s wr, 29 op/s 2024-09-20T22:53:31.514 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:31 smithi084 ceph-mon[40024]: 3.0 scrub starts 2024-09-20T22:53:31.514 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:31 smithi084 ceph-mon[40024]: 3.0 scrub ok 2024-09-20T22:53:31.514 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:31 smithi084 ceph-mon[40024]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T22:53:31.514 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:31 smithi084 ceph-mon[40024]: Cluster is now healthy 2024-09-20T22:53:31.514 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:31 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/2102769555' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T22:53:31.514 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:31 smithi084 ceph-mon[40024]: osdmap e730: 8 total, 8 up, 7 in 2024-09-20T22:53:31.514 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:31 smithi084 ceph-mon[40024]: 4.1f scrub starts 2024-09-20T22:53:31.514 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:31 smithi084 ceph-mon[40024]: 4.1f scrub ok 2024-09-20T22:53:31.517 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:53:31.540 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:31.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:31.541 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:53:31.590 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:53:31.620 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:53:31.627 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:53:31.642 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:53:31.663 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:31.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:31.664 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:53:31.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:31 smithi154 ceph-mon[33264]: pgmap v996: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 562 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 22 MiB/s rd, 3.5 MiB/s wr, 29 op/s 2024-09-20T22:53:31.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:31 smithi154 ceph-mon[33264]: 3.0 scrub starts 2024-09-20T22:53:31.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:31 smithi154 ceph-mon[33264]: 3.0 scrub ok 2024-09-20T22:53:31.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:31 smithi154 ceph-mon[33264]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T22:53:31.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:31 smithi154 ceph-mon[33264]: Cluster is now healthy 2024-09-20T22:53:31.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:31 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/2102769555' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T22:53:31.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:31 smithi154 ceph-mon[33264]: osdmap e730: 8 total, 8 up, 7 in 2024-09-20T22:53:31.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:31 smithi154 ceph-mon[33264]: 4.1f scrub starts 2024-09-20T22:53:31.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:31 smithi154 ceph-mon[33264]: 4.1f scrub ok 2024-09-20T22:53:31.765 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:53:31.789 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:31.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:31.790 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:53:31.891 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:53:31.914 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:31.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:31.915 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:53:32.015 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:53:32.039 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:32.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:32.040 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:53:32.141 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:53:32.162 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:32.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:32.163 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:53:32.264 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:53:32.284 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:32.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:32.285 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:53:32.355 INFO:tasks.rados.rados.0.smithi084.stdout:2905: done (1 left) 2024-09-20T22:53:32.355 INFO:tasks.rados.rados.0.smithi084.stdout:2907: done (0 left) 2024-09-20T22:53:32.355 INFO:tasks.rados.rados.0.smithi084.stdout:2908: rollback oid 28 current snap is 331 2024-09-20T22:53:32.355 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 28 to 322 2024-09-20T22:53:32.355 INFO:tasks.rados.rados.0.smithi084.stdout:2909: snap_create 2024-09-20T22:53:32.359 INFO:tasks.rados.rados.0.smithi084.stdout:2908: finishing rollback tid 1 to smithi08477203-28 2024-09-20T22:53:32.359 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 28 v 2189 (ObjNum 129 snap 0 seq_num 2090735872) dirty dne 2024-09-20T22:53:32.368 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:53:32.368 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:53:32.387 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:53:32.387 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:53:32.388 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:53:32.404 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:53:32.404 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:53:32.411 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:32.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:32.411 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:53:32.421 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-20T22:53:32.422 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T22:53:32.422 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:32 smithi084 ceph-mon[29774]: 4.b scrub starts 2024-09-20T22:53:32.422 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:32 smithi084 ceph-mon[29774]: 4.b scrub ok 2024-09-20T22:53:32.422 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:32 smithi084 ceph-mon[29774]: 3.1c scrub starts 2024-09-20T22:53:32.422 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:32 smithi084 ceph-mon[29774]: 3.1c scrub ok 2024-09-20T22:53:32.422 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:32 smithi084 ceph-mon[29774]: 3.18 scrub starts 2024-09-20T22:53:32.422 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:32 smithi084 ceph-mon[29774]: 3.18 scrub ok 2024-09-20T22:53:32.422 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:32 smithi084 ceph-mon[29774]: osdmap e731: 8 total, 8 up, 7 in 2024-09-20T22:53:32.423 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:32 smithi084 ceph-mon[40024]: 4.b scrub starts 2024-09-20T22:53:32.423 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:32 smithi084 ceph-mon[40024]: 4.b scrub ok 2024-09-20T22:53:32.423 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:32 smithi084 ceph-mon[40024]: 3.1c scrub starts 2024-09-20T22:53:32.423 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:32 smithi084 ceph-mon[40024]: 3.1c scrub ok 2024-09-20T22:53:32.423 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:32 smithi084 ceph-mon[40024]: 3.18 scrub starts 2024-09-20T22:53:32.423 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:32 smithi084 ceph-mon[40024]: 3.18 scrub ok 2024-09-20T22:53:32.423 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:32 smithi084 ceph-mon[40024]: osdmap e731: 8 total, 8 up, 7 in 2024-09-20T22:53:32.438 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:53:32.439 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:53:32.455 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:53:32.455 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:53:32.472 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:53:32.472 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:53:32.487 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:53:32.488 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:53:32.512 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:53:32.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:32.534 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:32.534 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:53:32.634 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:53:32.658 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:32.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:32.659 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:53:32.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:32 smithi154 ceph-mon[33264]: 4.b scrub starts 2024-09-20T22:53:32.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:32 smithi154 ceph-mon[33264]: 4.b scrub ok 2024-09-20T22:53:32.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:32 smithi154 ceph-mon[33264]: 3.1c scrub starts 2024-09-20T22:53:32.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:32 smithi154 ceph-mon[33264]: 3.1c scrub ok 2024-09-20T22:53:32.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:32 smithi154 ceph-mon[33264]: 3.18 scrub starts 2024-09-20T22:53:32.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:32 smithi154 ceph-mon[33264]: 3.18 scrub ok 2024-09-20T22:53:32.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:32 smithi154 ceph-mon[33264]: osdmap e731: 8 total, 8 up, 7 in 2024-09-20T22:53:32.760 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:53:32.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:32.783 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:32.783 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:53:32.883 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:53:32.904 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:32.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:32.905 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:53:33.006 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:53:33.032 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:33.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:33.033 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:53:33.133 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:53:33.159 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:33.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:33.160 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:53:33.260 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:53:33.284 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:33.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:33.285 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:53:33.292 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.5 to 1 (10000) 2024-09-20T22:53:33.299 INFO:tasks.rados.rados.0.smithi084.stdout:2908: done (1 left) 2024-09-20T22:53:33.300 INFO:tasks.rados.rados.0.smithi084.stdout:2909: done (0 left) 2024-09-20T22:53:33.300 INFO:tasks.rados.rados.0.smithi084.stdout:2910: delete oid 34 current snap is 332 2024-09-20T22:53:33.303 INFO:tasks.rados.rados.0.smithi084.stdout:2910: done (0 left) 2024-09-20T22:53:33.303 INFO:tasks.rados.rados.0.smithi084.stdout:2911: read oid 10 snap 326 2024-09-20T22:53:33.303 INFO:tasks.rados.rados.0.smithi084.stdout:2911: expect deleted 2024-09-20T22:53:33.304 INFO:tasks.rados.rados.0.smithi084.stdout:2912: snap_remove snap 322 2024-09-20T22:53:33.385 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:53:33.408 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:33.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:33.409 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:53:33.509 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:53:33.530 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:33.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:33.531 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:53:33.539 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd set noscrub 2024-09-20T22:53:33.592 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:33 smithi084 ceph-mon[40024]: pgmap v999: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 562 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 22 MiB/s rd, 3.5 MiB/s wr, 29 op/s 2024-09-20T22:53:33.592 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:33 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/3078805384' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:53:33.593 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:33 smithi084 ceph-mon[40024]: osdmap e732: 8 total, 8 up, 7 in 2024-09-20T22:53:33.593 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:33 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/885347899' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-20T22:53:33.593 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:33 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/885347899' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-20T22:53:33.593 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:33 smithi084 ceph-mon[40024]: osdmap e733: 8 total, 8 up, 7 in 2024-09-20T22:53:33.593 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:33 smithi084 ceph-mon[29774]: pgmap v999: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 562 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 22 MiB/s rd, 3.5 MiB/s wr, 29 op/s 2024-09-20T22:53:33.593 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:33 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/3078805384' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:53:33.593 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:33 smithi084 ceph-mon[29774]: osdmap e732: 8 total, 8 up, 7 in 2024-09-20T22:53:33.593 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:33 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/885347899' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-20T22:53:33.593 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:33 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/885347899' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-20T22:53:33.593 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:33 smithi084 ceph-mon[29774]: osdmap e733: 8 total, 8 up, 7 in 2024-09-20T22:53:33.632 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:53:33.653 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:33.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:33.654 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:53:33.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:33 smithi154 ceph-mon[33264]: pgmap v999: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 562 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 22 MiB/s rd, 3.5 MiB/s wr, 29 op/s 2024-09-20T22:53:33.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:33 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/3078805384' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:53:33.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:33 smithi154 ceph-mon[33264]: osdmap e732: 8 total, 8 up, 7 in 2024-09-20T22:53:33.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:33 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/885347899' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-20T22:53:33.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:33 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/885347899' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-20T22:53:33.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:33 smithi154 ceph-mon[33264]: osdmap e733: 8 total, 8 up, 7 in 2024-09-20T22:53:33.755 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:53:33.775 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:33.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:33.776 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:53:33.877 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:53:33.898 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:33.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:33.899 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:53:33.999 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:53:34.020 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:34.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:34.021 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:53:34.122 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:53:34.143 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:34.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:34.144 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:53:34.183 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd reweight 6 1 2024-09-20T22:53:34.245 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:53:34.287 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:34.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:34.288 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:53:34.301 INFO:tasks.rados.rados.0.smithi084.stdout:2912: done (1 left) 2024-09-20T22:53:34.301 INFO:tasks.rados.rados.0.smithi084.stdout:2913: write oid 19 current snap is 332 2024-09-20T22:53:34.301 INFO:tasks.rados.rados.0.smithi084.stdout:2913: seq_num 1008 ranges {541456=409588,1569261=562350,2814612=126844} 2024-09-20T22:53:34.307 INFO:tasks.rados.rados.0.smithi084.stdout:2913: writing smithi08477203-19 from 541456 to 951044 tid 1 2024-09-20T22:53:34.314 INFO:tasks.rados.rados.0.smithi084.stdout:2913: writing smithi08477203-19 from 1569261 to 2131611 tid 2 2024-09-20T22:53:34.319 INFO:tasks.rados.rados.0.smithi084.stdout:2913: writing smithi08477203-19 from 2814612 to 2941456 tid 3 2024-09-20T22:53:34.319 INFO:tasks.rados.rados.0.smithi084.stdout:2914: copy_from oid 32 from oid 21 current snap is 332 2024-09-20T22:53:34.319 INFO:tasks.rados.rados.0.smithi084.stdout:2913: finishing write tid 1 to smithi08477203-19 2024-09-20T22:53:34.319 INFO:tasks.rados.rados.0.smithi084.stdout:2913: finishing write tid 2 to smithi08477203-19 2024-09-20T22:53:34.319 INFO:tasks.rados.rados.0.smithi084.stdout:2911: done (2 left) 2024-09-20T22:53:34.319 INFO:tasks.rados.rados.0.smithi084.stdout:2915: write (excl) oid 30 current snap is 332 2024-09-20T22:53:34.319 INFO:tasks.rados.rados.0.smithi084.stdout:2915: seq_num 1009 ranges {558263=484837,1627743=706008} 2024-09-20T22:53:34.325 INFO:tasks.rados.rados.0.smithi084.stdout:2915: writing smithi08477203-30 from 558263 to 1043100 tid 1 2024-09-20T22:53:34.333 INFO:tasks.rados.rados.0.smithi084.stdout:2915: writing smithi08477203-30 from 1627743 to 2333751 tid 2 2024-09-20T22:53:34.333 INFO:tasks.rados.rados.0.smithi084.stdout:2916: delete oid 26 current snap is 332 2024-09-20T22:53:34.333 INFO:tasks.rados.rados.0.smithi084.stdout:2913: finishing write tid 3 to smithi08477203-19 2024-09-20T22:53:34.333 INFO:tasks.rados.rados.0.smithi084.stdout:2913: finishing write tid 4 to smithi08477203-19 2024-09-20T22:53:34.333 INFO:tasks.rados.rados.0.smithi084.stdout:2913: finishing write tid 5 to smithi08477203-19 2024-09-20T22:53:34.333 INFO:tasks.rados.rados.0.smithi084.stdout:2913: oid 19 updating version 0 to 2290 2024-09-20T22:53:34.333 INFO:tasks.rados.rados.0.smithi084.stdout:2913: oid 19 updating version 2290 to 2292 2024-09-20T22:53:34.333 INFO:tasks.rados.rados.0.smithi084.stdout:2913: oid 19 version 2292 is already newer than 2289 2024-09-20T22:53:34.333 INFO:tasks.rados.rados.0.smithi084.stdout:2913: oid 19 version 2292 is already newer than 2291 2024-09-20T22:53:34.333 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 19 v 2292 (ObjNum 1008 snap 332 seq_num 1008) dirty exists 2024-09-20T22:53:34.333 INFO:tasks.rados.rados.0.smithi084.stdout:2913: left oid 19 (ObjNum 1008 snap 332 seq_num 1008) 2024-09-20T22:53:34.333 INFO:tasks.rados.rados.0.smithi084.stdout:2914: finishing copy_from racing read to smithi08477203-32 2024-09-20T22:53:34.334 INFO:tasks.rados.rados.0.smithi084.stdout:2915: finishing write tid 1 to smithi08477203-30 2024-09-20T22:53:34.334 INFO:tasks.rados.rados.0.smithi084.stdout:2914: finishing copy_from to smithi08477203-32 2024-09-20T22:53:34.334 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 32 v 1829 (ObjNum 984 snap 319 seq_num 984) dirty exists 2024-09-20T22:53:34.335 INFO:tasks.rados.rados.0.smithi084.stdout:2913: done (3 left) 2024-09-20T22:53:34.335 INFO:tasks.rados.rados.0.smithi084.stdout:2914: done (2 left) 2024-09-20T22:53:34.335 INFO:tasks.rados.rados.0.smithi084.stdout:2916: done (1 left) 2024-09-20T22:53:34.335 INFO:tasks.rados.rados.0.smithi084.stdout:2917: snap_create 2024-09-20T22:53:34.336 INFO:tasks.rados.rados.0.smithi084.stdout:2915: finishing write tid 2 to smithi08477203-30 2024-09-20T22:53:34.336 INFO:tasks.rados.rados.0.smithi084.stdout:2915: finishing write tid 3 to smithi08477203-30 2024-09-20T22:53:34.337 INFO:tasks.rados.rados.0.smithi084.stdout:2915: finishing write tid 4 to smithi08477203-30 2024-09-20T22:53:34.337 INFO:tasks.rados.rados.0.smithi084.stdout:2915: oid 30 updating version 0 to 1844 2024-09-20T22:53:34.337 INFO:tasks.rados.rados.0.smithi084.stdout:2915: oid 30 updating version 1844 to 1845 2024-09-20T22:53:34.337 INFO:tasks.rados.rados.0.smithi084.stdout:2915: oid 30 version 1845 is already newer than 1843 2024-09-20T22:53:34.337 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 30 v 1845 (ObjNum 1009 snap 332 seq_num 1009) dirty exists 2024-09-20T22:53:34.337 INFO:tasks.rados.rados.0.smithi084.stdout:2915: left oid 30 (ObjNum 1009 snap 332 seq_num 1009) 2024-09-20T22:53:34.389 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:53:34.411 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:34.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:34.412 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:53:34.440 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:34 smithi084 ceph-mon[40024]: 3.6 scrub starts 2024-09-20T22:53:34.440 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:34 smithi084 ceph-mon[40024]: 3.6 scrub ok 2024-09-20T22:53:34.440 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:34 smithi084 ceph-mon[40024]: 4.11 scrub starts 2024-09-20T22:53:34.440 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:34 smithi084 ceph-mon[40024]: 4.11 scrub ok 2024-09-20T22:53:34.440 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:34 smithi084 ceph-mon[40024]: purged_snaps scrub starts 2024-09-20T22:53:34.440 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:34 smithi084 ceph-mon[40024]: purged_snaps scrub ok 2024-09-20T22:53:34.440 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:34 smithi084 ceph-mon[40024]: osdmap e734: 8 total, 8 up, 7 in 2024-09-20T22:53:34.440 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:34 smithi084 ceph-mon[29774]: 3.6 scrub starts 2024-09-20T22:53:34.441 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:34 smithi084 ceph-mon[29774]: 3.6 scrub ok 2024-09-20T22:53:34.441 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:34 smithi084 ceph-mon[29774]: 4.11 scrub starts 2024-09-20T22:53:34.441 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:34 smithi084 ceph-mon[29774]: 4.11 scrub ok 2024-09-20T22:53:34.441 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:34 smithi084 ceph-mon[29774]: purged_snaps scrub starts 2024-09-20T22:53:34.441 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:34 smithi084 ceph-mon[29774]: purged_snaps scrub ok 2024-09-20T22:53:34.441 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:34 smithi084 ceph-mon[29774]: osdmap e734: 8 total, 8 up, 7 in 2024-09-20T22:53:34.441 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-20T22:53:34.441 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:53:34.513 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:53:34.534 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:34.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:34.535 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:53:34.636 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:53:34.659 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:34.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:34.660 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:53:34.661 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:53:34.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:34 smithi154 ceph-mon[33264]: 3.6 scrub starts 2024-09-20T22:53:34.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:34 smithi154 ceph-mon[33264]: 3.6 scrub ok 2024-09-20T22:53:34.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:34 smithi154 ceph-mon[33264]: 4.11 scrub starts 2024-09-20T22:53:34.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:34 smithi154 ceph-mon[33264]: 4.11 scrub ok 2024-09-20T22:53:34.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:34 smithi154 ceph-mon[33264]: purged_snaps scrub starts 2024-09-20T22:53:34.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:34 smithi154 ceph-mon[33264]: purged_snaps scrub ok 2024-09-20T22:53:34.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:34 smithi154 ceph-mon[33264]: osdmap e734: 8 total, 8 up, 7 in 2024-09-20T22:53:34.761 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:53:34.783 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:34.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:34.785 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:53:34.885 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:53:34.906 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:34.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:34.907 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:53:35.008 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:53:35.031 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:35.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:35.032 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:53:35.133 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:53:35.156 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:35.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:35.157 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:53:35.190 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.1 dump_historic_ops 2024-09-20T22:53:35.193 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:53:35.257 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:53:35.281 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:35.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:35.283 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:53:35.301 INFO:tasks.rados.rados.0.smithi084.stdout:2915: done (1 left) 2024-09-20T22:53:35.301 INFO:tasks.rados.rados.0.smithi084.stdout:2917: done (0 left) 2024-09-20T22:53:35.301 INFO:tasks.rados.rados.0.smithi084.stdout:2918: delete oid 13 current snap is 333 2024-09-20T22:53:35.307 INFO:tasks.rados.rados.0.smithi084.stdout:2918: done (0 left) 2024-09-20T22:53:35.307 INFO:tasks.rados.rados.0.smithi084.stdout:2919: read oid 48 snap -1 2024-09-20T22:53:35.308 INFO:tasks.rados.rados.0.smithi084.stdout:2919: expect (ObjNum 985 snap 323 seq_num 985) 2024-09-20T22:53:35.308 INFO:tasks.rados.rados.0.smithi084.stdout:2920: snap_remove snap 299 2024-09-20T22:53:35.383 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:53:35.405 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:35.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:35.406 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:53:35.457 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:35 smithi084 ceph-mon[40024]: 3.f scrub starts 2024-09-20T22:53:35.457 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:35 smithi084 ceph-mon[40024]: 3.f scrub ok 2024-09-20T22:53:35.457 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:35 smithi084 ceph-mon[40024]: pgmap v1002: 158 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 153 active+clean; 562 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 3.9 MiB/s rd, 334 KiB/s wr, 6 op/s 2024-09-20T22:53:35.457 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:35 smithi084 ceph-mon[40024]: 5.3 deep-scrub starts 2024-09-20T22:53:35.457 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:35 smithi084 ceph-mon[40024]: 5.3 deep-scrub ok 2024-09-20T22:53:35.457 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:35 smithi084 ceph-mon[40024]: 5.13 scrub starts 2024-09-20T22:53:35.457 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:35 smithi084 ceph-mon[40024]: 5.13 scrub ok 2024-09-20T22:53:35.457 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:35 smithi084 ceph-mon[40024]: 5.1f scrub starts 2024-09-20T22:53:35.457 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:35 smithi084 ceph-mon[40024]: 5.1f scrub ok 2024-09-20T22:53:35.457 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:35 smithi084 ceph-mon[40024]: osdmap e735: 8 total, 8 up, 7 in 2024-09-20T22:53:35.457 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:35 smithi084 ceph-mon[29774]: 3.f scrub starts 2024-09-20T22:53:35.458 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:35 smithi084 ceph-mon[29774]: 3.f scrub ok 2024-09-20T22:53:35.458 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:35 smithi084 ceph-mon[29774]: pgmap v1002: 158 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 153 active+clean; 562 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 3.9 MiB/s rd, 334 KiB/s wr, 6 op/s 2024-09-20T22:53:35.458 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:35 smithi084 ceph-mon[29774]: 5.3 deep-scrub starts 2024-09-20T22:53:35.458 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:35 smithi084 ceph-mon[29774]: 5.3 deep-scrub ok 2024-09-20T22:53:35.458 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:35 smithi084 ceph-mon[29774]: 5.13 scrub starts 2024-09-20T22:53:35.458 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:35 smithi084 ceph-mon[29774]: 5.13 scrub ok 2024-09-20T22:53:35.458 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:35 smithi084 ceph-mon[29774]: 5.1f scrub starts 2024-09-20T22:53:35.458 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:35 smithi084 ceph-mon[29774]: 5.1f scrub ok 2024-09-20T22:53:35.458 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:35 smithi084 ceph-mon[29774]: osdmap e735: 8 total, 8 up, 7 in 2024-09-20T22:53:35.480 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-20T22:53:35.507 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:53:35.543 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:35.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:35.544 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:53:35.645 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:53:35.668 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:35.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:35.669 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:53:35.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:35 smithi154 ceph-mon[33264]: 3.f scrub starts 2024-09-20T22:53:35.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:35 smithi154 ceph-mon[33264]: 3.f scrub ok 2024-09-20T22:53:35.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:35 smithi154 ceph-mon[33264]: pgmap v1002: 158 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 153 active+clean; 562 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 3.9 MiB/s rd, 334 KiB/s wr, 6 op/s 2024-09-20T22:53:35.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:35 smithi154 ceph-mon[33264]: 5.3 deep-scrub starts 2024-09-20T22:53:35.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:35 smithi154 ceph-mon[33264]: 5.3 deep-scrub ok 2024-09-20T22:53:35.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:35 smithi154 ceph-mon[33264]: 5.13 scrub starts 2024-09-20T22:53:35.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:35 smithi154 ceph-mon[33264]: 5.13 scrub ok 2024-09-20T22:53:35.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:35 smithi154 ceph-mon[33264]: 5.1f scrub starts 2024-09-20T22:53:35.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:35 smithi154 ceph-mon[33264]: 5.1f scrub ok 2024-09-20T22:53:35.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:35 smithi154 ceph-mon[33264]: osdmap e735: 8 total, 8 up, 7 in 2024-09-20T22:53:35.770 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:53:35.792 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:35.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:35.793 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:53:35.894 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:53:35.916 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:35.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:35.917 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:53:36.017 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:53:36.039 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:36.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:36.040 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:53:36.123 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:53:36.140 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:53:36.166 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:36.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:36.167 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:53:36.268 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:53:36.291 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:36.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:36.292 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:53:36.299 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is set 2024-09-20T22:53:36.305 INFO:tasks.rados.rados.0.smithi084.stdout:2920: done (1 left) 2024-09-20T22:53:36.305 INFO:tasks.rados.rados.0.smithi084.stdout:2921: write (excl) oid 5 current snap is 333 2024-09-20T22:53:36.305 INFO:tasks.rados.rados.0.smithi084.stdout:2921: seq_num 1012 ranges {608684=426937,1803189=569688,2924304=84380} 2024-09-20T22:53:36.306 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.6 to 1 (10000) 2024-09-20T22:53:36.312 INFO:tasks.rados.rados.0.smithi084.stdout:2921: writing smithi08477203-5 from 608684 to 1035621 tid 1 2024-09-20T22:53:36.320 INFO:tasks.rados.rados.0.smithi084.stdout:2921: writing smithi08477203-5 from 1803189 to 2372877 tid 2 2024-09-20T22:53:36.324 INFO:tasks.rados.rados.0.smithi084.stdout:2921: writing smithi08477203-5 from 2924304 to 3008684 tid 3 2024-09-20T22:53:36.324 INFO:tasks.rados.rados.0.smithi084.stdout:2922: write oid 40 current snap is 333 2024-09-20T22:53:36.324 INFO:tasks.rados.rados.0.smithi084.stdout:2922: seq_num 1013 ranges {625491=502186,1861671=713346} 2024-09-20T22:53:36.331 INFO:tasks.rados.rados.0.smithi084.stdout:2922: writing smithi08477203-40 from 625491 to 1127677 tid 1 2024-09-20T22:53:36.340 INFO:tasks.rados.rados.0.smithi084.stdout:2922: writing smithi08477203-40 from 1861671 to 2575017 tid 2 2024-09-20T22:53:36.340 INFO:tasks.rados.rados.0.smithi084.stdout:2923: read oid 36 snap 331 2024-09-20T22:53:36.341 INFO:tasks.rados.rados.0.smithi084.stdout:2923: expect (ObjNum 945 snap 311 seq_num 945) 2024-09-20T22:53:36.341 INFO:tasks.rados.rados.0.smithi084.stdout:2924: read oid 16 snap -1 2024-09-20T22:53:36.341 INFO:tasks.rados.rados.0.smithi084.stdout:2924: expect (ObjNum 876 snap 280 seq_num 876) 2024-09-20T22:53:36.341 INFO:tasks.rados.rados.0.smithi084.stdout:2925: write (excl) oid 37 current snap is 333 2024-09-20T22:53:36.341 INFO:tasks.rados.rados.0.smithi084.stdout:2919: expect (ObjNum 985 snap 323 seq_num 985) 2024-09-20T22:53:36.393 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:53:36.415 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:36.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:36.416 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:53:36.472 INFO:tasks.rados.rados.0.smithi084.stdout:2921: finishing write tid 1 to smithi08477203-5 2024-09-20T22:53:36.472 INFO:tasks.rados.rados.0.smithi084.stdout:2925: seq_num 1014 ranges {642298=577435,1920153=573357,2979150=63148} 2024-09-20T22:53:36.479 INFO:tasks.rados.rados.0.smithi084.stdout:2925: writing smithi08477203-37 from 642298 to 1219733 tid 1 2024-09-20T22:53:36.486 INFO:tasks.rados.rados.0.smithi084.stdout:2925: writing smithi08477203-37 from 1920153 to 2493510 tid 2 2024-09-20T22:53:36.489 INFO:tasks.rados.rados.0.smithi084.stdout:2925: writing smithi08477203-37 from 2979150 to 3042298 tid 3 2024-09-20T22:53:36.489 INFO:tasks.rados.rados.0.smithi084.stdout:2919: done (5 left) 2024-09-20T22:53:36.489 INFO:tasks.rados.rados.0.smithi084.stdout:2926: read oid 43 snap -1 2024-09-20T22:53:36.489 INFO:tasks.rados.rados.0.smithi084.stdout:2926: expect deleted 2024-09-20T22:53:36.489 INFO:tasks.rados.rados.0.smithi084.stdout:2921: finishing write tid 2 to smithi08477203-5 2024-09-20T22:53:36.489 INFO:tasks.rados.rados.0.smithi084.stdout:2927: read oid 14 snap -1 2024-09-20T22:53:36.490 INFO:tasks.rados.rados.0.smithi084.stdout:2927: expect (ObjNum 879 snap 285 seq_num 879) 2024-09-20T22:53:36.490 INFO:tasks.rados.rados.0.smithi084.stdout:2928: snap_remove snap 325 2024-09-20T22:53:36.517 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:53:36.538 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:36.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:36.539 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:53:36.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:36 smithi084 ceph-mon[29774]: 4.7 scrub starts 2024-09-20T22:53:36.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:36 smithi084 ceph-mon[29774]: 4.7 scrub ok 2024-09-20T22:53:36.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:36 smithi084 ceph-mon[29774]: 1.5 scrub starts 2024-09-20T22:53:36.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:36 smithi084 ceph-mon[29774]: 1.5 scrub ok 2024-09-20T22:53:36.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:36 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/1526400182' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-20T22:53:36.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:36 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/4177056539' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-09-20T22:53:36.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:36 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-09-20T22:53:36.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:36 smithi084 ceph-mon[29774]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:53:36.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:36 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/1526400182' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T22:53:36.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:36 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-20T22:53:36.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:36 smithi084 ceph-mon[29774]: osdmap e736: 8 total, 8 up, 7 in 2024-09-20T22:53:36.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:36 smithi084 ceph-mon[40024]: 4.7 scrub starts 2024-09-20T22:53:36.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:36 smithi084 ceph-mon[40024]: 4.7 scrub ok 2024-09-20T22:53:36.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:36 smithi084 ceph-mon[40024]: 1.5 scrub starts 2024-09-20T22:53:36.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:36 smithi084 ceph-mon[40024]: 1.5 scrub ok 2024-09-20T22:53:36.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:36 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/1526400182' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-20T22:53:36.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:36 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/4177056539' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-09-20T22:53:36.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:36 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-09-20T22:53:36.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:36 smithi084 ceph-mon[40024]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:53:36.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:36 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/1526400182' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T22:53:36.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:36 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-20T22:53:36.617 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:36 smithi084 ceph-mon[40024]: osdmap e736: 8 total, 8 up, 7 in 2024-09-20T22:53:36.640 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:53:36.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:36.663 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:36.663 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:53:36.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:36 smithi154 ceph-mon[33264]: 4.7 scrub starts 2024-09-20T22:53:36.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:36 smithi154 ceph-mon[33264]: 4.7 scrub ok 2024-09-20T22:53:36.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:36 smithi154 ceph-mon[33264]: 1.5 scrub starts 2024-09-20T22:53:36.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:36 smithi154 ceph-mon[33264]: 1.5 scrub ok 2024-09-20T22:53:36.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:36 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/1526400182' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-20T22:53:36.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:36 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/4177056539' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-09-20T22:53:36.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:36 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-09-20T22:53:36.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:36 smithi154 ceph-mon[33264]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:53:36.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:36 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/1526400182' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T22:53:36.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:36 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-20T22:53:36.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:36 smithi154 ceph-mon[33264]: osdmap e736: 8 total, 8 up, 7 in 2024-09-20T22:53:36.764 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:53:36.786 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:36.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:36.787 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:53:36.880 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:53:36.881 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:53:36.888 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:53:36.898 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:53:36.899 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:53:36.910 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:36.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:36.911 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:53:36.915 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:53:36.915 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:53:36.931 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-20T22:53:36.932 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T22:53:36.948 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:53:36.948 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:53:36.965 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:53:36.965 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:53:36.982 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:53:36.982 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:53:36.998 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:53:36.998 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:53:37.012 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:53:37.033 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:37.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:37.034 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:53:37.135 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:53:37.156 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:37.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:37.157 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:53:37.257 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:53:37.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:37.280 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:37.280 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:53:37.370 INFO:tasks.rados.rados.0.smithi084.stdout:2928: done (7 left) 2024-09-20T22:53:37.370 INFO:tasks.rados.rados.0.smithi084.stdout:2929: rollback oid 50 current snap is 333 2024-09-20T22:53:37.370 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 50 to 324 2024-09-20T22:53:37.370 INFO:tasks.rados.rados.0.smithi084.stdout:2921: finishing write tid 3 to smithi08477203-5 2024-09-20T22:53:37.370 INFO:tasks.rados.rados.0.smithi084.stdout:2921: finishing write tid 4 to smithi08477203-5 2024-09-20T22:53:37.370 INFO:tasks.rados.rados.0.smithi084.stdout:2921: finishing write tid 5 to smithi08477203-5 2024-09-20T22:53:37.370 INFO:tasks.rados.rados.0.smithi084.stdout:2921: oid 5 updating version 0 to 901 2024-09-20T22:53:37.370 INFO:tasks.rados.rados.0.smithi084.stdout:2921: oid 5 updating version 901 to 903 2024-09-20T22:53:37.371 INFO:tasks.rados.rados.0.smithi084.stdout:2921: oid 5 version 903 is already newer than 902 2024-09-20T22:53:37.371 INFO:tasks.rados.rados.0.smithi084.stdout:2921: oid 5 updating version 903 to 904 2024-09-20T22:53:37.371 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 5 v 904 (ObjNum 1012 snap 333 seq_num 1012) dirty exists 2024-09-20T22:53:37.371 INFO:tasks.rados.rados.0.smithi084.stdout:2921: left oid 5 (ObjNum 1012 snap 333 seq_num 1012) 2024-09-20T22:53:37.371 INFO:tasks.rados.rados.0.smithi084.stdout:2922: finishing write tid 1 to smithi08477203-40 2024-09-20T22:53:37.371 INFO:tasks.rados.rados.0.smithi084.stdout:2922: finishing write tid 2 to smithi08477203-40 2024-09-20T22:53:37.371 INFO:tasks.rados.rados.0.smithi084.stdout:2922: finishing write tid 3 to smithi08477203-40 2024-09-20T22:53:37.371 INFO:tasks.rados.rados.0.smithi084.stdout:2922: finishing write tid 4 to smithi08477203-40 2024-09-20T22:53:37.371 INFO:tasks.rados.rados.0.smithi084.stdout:2922: oid 40 updating version 0 to 1263 2024-09-20T22:53:37.371 INFO:tasks.rados.rados.0.smithi084.stdout:2922: oid 40 updating version 1263 to 1264 2024-09-20T22:53:37.371 INFO:tasks.rados.rados.0.smithi084.stdout:2922: oid 40 version 1264 is already newer than 1262 2024-09-20T22:53:37.371 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 40 v 1264 (ObjNum 1013 snap 333 seq_num 1013) dirty exists 2024-09-20T22:53:37.371 INFO:tasks.rados.rados.0.smithi084.stdout:2922: left oid 40 (ObjNum 1013 snap 333 seq_num 1013) 2024-09-20T22:53:37.371 INFO:tasks.rados.rados.0.smithi084.stdout:2924: expect (ObjNum 876 snap 280 seq_num 876) 2024-09-20T22:53:37.381 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:53:37.402 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:37.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:37.403 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:53:37.467 INFO:tasks.rados.rados.0.smithi084.stdout:2923: expect (ObjNum 945 snap 311 seq_num 945) 2024-09-20T22:53:37.504 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:53:37.524 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:37.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:37.526 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:53:37.593 INFO:tasks.rados.rados.0.smithi084.stdout:2927: expect (ObjNum 879 snap 285 seq_num 879) 2024-09-20T22:53:37.614 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:37 smithi084 ceph-mon[40024]: 3.e scrub starts 2024-09-20T22:53:37.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:37 smithi084 ceph-mon[40024]: 3.e scrub ok 2024-09-20T22:53:37.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:37 smithi084 ceph-mon[40024]: pgmap v1005: 158 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 153 active+clean; 562 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 3.9 MiB/s rd, 334 KiB/s wr, 6 op/s 2024-09-20T22:53:37.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:37 smithi084 ceph-mon[40024]: 6.18 scrub starts 2024-09-20T22:53:37.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:37 smithi084 ceph-mon[40024]: 6.18 scrub ok 2024-09-20T22:53:37.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:37 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/2350527660' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:53:37.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:37 smithi084 ceph-mon[29774]: 3.e scrub starts 2024-09-20T22:53:37.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:37 smithi084 ceph-mon[29774]: 3.e scrub ok 2024-09-20T22:53:37.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:37 smithi084 ceph-mon[29774]: pgmap v1005: 158 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 153 active+clean; 562 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 3.9 MiB/s rd, 334 KiB/s wr, 6 op/s 2024-09-20T22:53:37.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:37 smithi084 ceph-mon[29774]: 6.18 scrub starts 2024-09-20T22:53:37.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:37 smithi084 ceph-mon[29774]: 6.18 scrub ok 2024-09-20T22:53:37.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:37 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/2350527660' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:53:37.627 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:53:37.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:37.650 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:37.650 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:53:37.683 INFO:tasks.rados.rados.0.smithi084.stdout:2925: finishing write tid 1 to smithi08477203-37 2024-09-20T22:53:37.683 INFO:tasks.rados.rados.0.smithi084.stdout:2925: finishing write tid 2 to smithi08477203-37 2024-09-20T22:53:37.683 INFO:tasks.rados.rados.0.smithi084.stdout:2921: done (7 left) 2024-09-20T22:53:37.683 INFO:tasks.rados.rados.0.smithi084.stdout:2922: done (6 left) 2024-09-20T22:53:37.683 INFO:tasks.rados.rados.0.smithi084.stdout:2923: done (5 left) 2024-09-20T22:53:37.683 INFO:tasks.rados.rados.0.smithi084.stdout:2924: done (4 left) 2024-09-20T22:53:37.683 INFO:tasks.rados.rados.0.smithi084.stdout:2926: done (3 left) 2024-09-20T22:53:37.683 INFO:tasks.rados.rados.0.smithi084.stdout:2927: done (2 left) 2024-09-20T22:53:37.683 INFO:tasks.rados.rados.0.smithi084.stdout:2930: snap_remove snap 235 2024-09-20T22:53:37.751 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:53:37.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:37 smithi154 ceph-mon[33264]: 3.e scrub starts 2024-09-20T22:53:37.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:37 smithi154 ceph-mon[33264]: 3.e scrub ok 2024-09-20T22:53:37.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:37 smithi154 ceph-mon[33264]: pgmap v1005: 158 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 153 active+clean; 562 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 3.9 MiB/s rd, 334 KiB/s wr, 6 op/s 2024-09-20T22:53:37.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:37 smithi154 ceph-mon[33264]: 6.18 scrub starts 2024-09-20T22:53:37.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:37 smithi154 ceph-mon[33264]: 6.18 scrub ok 2024-09-20T22:53:37.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:37 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/2350527660' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:53:37.772 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:37.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:37.773 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:53:37.874 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:53:37.895 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:37.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:37.897 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:53:37.997 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:53:38.018 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:38.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:38.019 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:53:38.120 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:53:38.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:38.141 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:38.141 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:53:38.242 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:53:38.263 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:38.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:38.264 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:53:38.298 INFO:tasks.rados.rados.0.smithi084.stdout:2930: done (2 left) 2024-09-20T22:53:38.299 INFO:tasks.rados.rados.0.smithi084.stdout:2931: copy_from oid 40 from oid 14 current snap is 333 2024-09-20T22:53:38.299 INFO:tasks.rados.rados.0.smithi084.stdout:2925: finishing write tid 3 to smithi08477203-37 2024-09-20T22:53:38.299 INFO:tasks.rados.rados.0.smithi084.stdout:2925: finishing write tid 4 to smithi08477203-37 2024-09-20T22:53:38.299 INFO:tasks.rados.rados.0.smithi084.stdout:2925: finishing write tid 5 to smithi08477203-37 2024-09-20T22:53:38.299 INFO:tasks.rados.rados.0.smithi084.stdout:2925: oid 37 updating version 0 to 1251 2024-09-20T22:53:38.299 INFO:tasks.rados.rados.0.smithi084.stdout:2925: oid 37 version 1251 is already newer than 1250 2024-09-20T22:53:38.299 INFO:tasks.rados.rados.0.smithi084.stdout:2925: oid 37 version 1251 is already newer than 1249 2024-09-20T22:53:38.299 INFO:tasks.rados.rados.0.smithi084.stdout:2925: oid 37 version 1251 is already newer than 1248 2024-09-20T22:53:38.299 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 37 v 1251 (ObjNum 1014 snap 333 seq_num 1014) dirty exists 2024-09-20T22:53:38.299 INFO:tasks.rados.rados.0.smithi084.stdout:2925: left oid 37 (ObjNum 1014 snap 333 seq_num 1014) 2024-09-20T22:53:38.299 INFO:tasks.rados.rados.0.smithi084.stdout:2929: finishing rollback tid 0 to smithi08477203-50 2024-09-20T22:53:38.299 INFO:tasks.rados.rados.0.smithi084.stdout:2929: finishing rollback tid 1 to smithi08477203-50 2024-09-20T22:53:38.300 INFO:tasks.rados.rados.0.smithi084.stdout:2929: finishing rollback tid 2 to smithi08477203-50 2024-09-20T22:53:38.300 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 50 v 1852 (ObjNum 923 snap 305 seq_num 923) dirty exists 2024-09-20T22:53:38.300 INFO:tasks.rados.rados.0.smithi084.stdout:2925: done (2 left) 2024-09-20T22:53:38.300 INFO:tasks.rados.rados.0.smithi084.stdout:2929: done (1 left) 2024-09-20T22:53:38.300 INFO:tasks.rados.rados.0.smithi084.stdout:2932: rollback oid 24 current snap is 333 2024-09-20T22:53:38.300 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 24 to 326 2024-09-20T22:53:38.300 INFO:tasks.rados.rados.0.smithi084.stdout:2933: snap_remove snap 308 2024-09-20T22:53:38.348 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd reweight 7 1 2024-09-20T22:53:38.365 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:53:38.397 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:53:38.412 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:38.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:38.414 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:53:38.514 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:53:38.529 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:53:38 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: ::ffff:172.21.15.154 - - [20/Sep/2024:22:53:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-20T22:53:38.529 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:38 smithi084 ceph-mon[29774]: osdmap e737: 8 total, 8 up, 7 in 2024-09-20T22:53:38.529 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:38 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:53:38.529 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:38 smithi084 ceph-mon[29774]: osdmap e738: 8 total, 8 up, 7 in 2024-09-20T22:53:38.530 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:38 smithi084 ceph-mon[40024]: osdmap e737: 8 total, 8 up, 7 in 2024-09-20T22:53:38.530 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:38 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:53:38.530 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:38 smithi084 ceph-mon[40024]: osdmap e738: 8 total, 8 up, 7 in 2024-09-20T22:53:38.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:38.538 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:38.538 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:53:38.639 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:53:38.660 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:38.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:38.661 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:53:38.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:38 smithi154 ceph-mon[33264]: osdmap e737: 8 total, 8 up, 7 in 2024-09-20T22:53:38.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:38 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:53:38.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:38 smithi154 ceph-mon[33264]: osdmap e738: 8 total, 8 up, 7 in 2024-09-20T22:53:38.762 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:53:38.784 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:38.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:38.785 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:53:38.885 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:53:38.906 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:38.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:38.907 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:53:39.008 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:53:39.031 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:39.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:39.032 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:53:39.133 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:53:39.156 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:39.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:39.157 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:53:39.258 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:53:39.280 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:39.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:39.281 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:53:39.300 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-20T22:53:39.300 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2024-09-20T22:53:39.300 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2024-09-20T22:53:39.300 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [ 2024-09-20T22:53:39.300 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.301 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:2351 7.13 7:cae5792a:::smithi08477203-32:head [copy-from ver 376] snapc 56=[56,55,53,52,51,50,47,3e] ondisk+write+known_if_redirected+supports_pool_eio e260)", 2024-09-20T22:53:39.301 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:45:39.362090+0000", 2024-09-20T22:53:39.301 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 479.93560824899998, 2024-09-20T22:53:39.301 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.88649603300000002, 2024-09-20T22:53:39.301 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:53:39.301 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:53:39.301 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:53:39.301 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:53:39.301 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:53:39.301 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:53:39.301 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 2351 2024-09-20T22:53:39.301 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.301 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:53:39.301 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.302 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:53:39.302 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.362090+0000", 2024-09-20T22:53:39.302 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:53:39.302 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.302 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.302 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:53:39.302 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.362090+0000", 2024-09-20T22:53:39.302 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:53:39.302 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.302 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.302 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:53:39.302 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.362091+0000", 2024-09-20T22:53:39.302 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.5570000000000001e-06 2024-09-20T22:53:39.303 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.303 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.303 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:53:39.303 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.362099+0000", 2024-09-20T22:53:39.303 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.7030000000000006e-06 2024-09-20T22:53:39.303 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.303 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.303 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:53:39.303 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.362113+0000", 2024-09-20T22:53:39.303 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3492999999999999e-05 2024-09-20T22:53:39.303 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.303 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.303 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:53:39.303 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.362119+0000", 2024-09-20T22:53:39.303 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.7700000000000004e-06 2024-09-20T22:53:39.304 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.304 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.304 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:53:39.304 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.362190+0000", 2024-09-20T22:53:39.304 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.0576999999999999e-05 2024-09-20T22:53:39.304 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.304 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.304 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-09-20T22:53:39.304 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.362203+0000", 2024-09-20T22:53:39.304 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3484e-05 2024-09-20T22:53:39.304 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.304 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.304 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:53:39.304 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:40.226702+0000", 2024-09-20T22:53:39.304 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.86449815799999996 2024-09-20T22:53:39.304 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.305 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.305 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for missing object", 2024-09-20T22:53:39.305 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:40.227275+0000", 2024-09-20T22:53:39.305 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00057360899999999999 2024-09-20T22:53:39.305 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.305 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.305 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:53:39.305 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:40.232018+0000", 2024-09-20T22:53:39.305 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0047427839999999999 2024-09-20T22:53:39.305 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.305 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.305 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:53:39.305 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:40.232136+0000", 2024-09-20T22:53:39.305 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000118068 2024-09-20T22:53:39.306 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.306 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.306 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 2,5", 2024-09-20T22:53:39.306 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:40.240483+0000", 2024-09-20T22:53:39.306 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0083470390000000005 2024-09-20T22:53:39.306 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.306 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.306 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:53:39.306 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:40.245573+0000", 2024-09-20T22:53:39.306 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00508998 2024-09-20T22:53:39.306 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.306 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.306 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:53:39.306 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:40.248086+0000", 2024-09-20T22:53:39.307 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025128870000000001 2024-09-20T22:53:39.307 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.307 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.307 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:53:39.307 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:40.248398+0000", 2024-09-20T22:53:39.307 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00031257600000000001 2024-09-20T22:53:39.307 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.307 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.307 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:53:39.307 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:40.248476+0000", 2024-09-20T22:53:39.307 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.7491999999999996e-05 2024-09-20T22:53:39.307 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.307 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.307 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:53:39.307 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:40.248586+0000", 2024-09-20T22:53:39.308 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000109856 2024-09-20T22:53:39.308 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:53:39.308 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:53:39.308 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:53:39.308 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.308 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.308 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:2352 7.13 7:cae5792a:::smithi08477203-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e260)", 2024-09-20T22:53:39.308 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:45:39.362137+0000", 2024-09-20T22:53:39.308 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 479.935561382, 2024-09-20T22:53:39.308 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.883015891, 2024-09-20T22:53:39.308 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:53:39.308 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:53:39.308 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-20T22:53:39.308 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:53:39.308 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:53:39.308 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:53:39.308 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 2352 2024-09-20T22:53:39.309 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.309 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:53:39.309 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.309 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:53:39.309 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.362137+0000", 2024-09-20T22:53:39.309 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:53:39.309 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.309 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.309 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:53:39.309 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.362137+0000", 2024-09-20T22:53:39.309 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:53:39.309 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.309 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.309 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:53:39.309 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.362137+0000", 2024-09-20T22:53:39.310 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.6300000000000001e-07 2024-09-20T22:53:39.310 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.310 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.310 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:53:39.310 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.362139+0000", 2024-09-20T22:53:39.310 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.7460000000000001e-06 2024-09-20T22:53:39.310 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.310 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.310 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:53:39.310 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.362144+0000", 2024-09-20T22:53:39.310 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.1080000000000001e-06 2024-09-20T22:53:39.310 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.310 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.310 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:53:39.310 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.362147+0000", 2024-09-20T22:53:39.310 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.2109999999999998e-06 2024-09-20T22:53:39.311 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.311 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.311 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:53:39.311 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.362233+0000", 2024-09-20T22:53:39.311 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.5736999999999996e-05 2024-09-20T22:53:39.311 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.311 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.311 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-09-20T22:53:39.311 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.362242+0000", 2024-09-20T22:53:39.311 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.8179999999999993e-06 2024-09-20T22:53:39.311 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.311 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.311 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:53:39.311 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:40.227702+0000", 2024-09-20T22:53:39.311 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.86546061299999999 2024-09-20T22:53:39.311 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.312 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.312 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for missing object", 2024-09-20T22:53:39.312 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:40.227751+0000", 2024-09-20T22:53:39.312 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.9128000000000001e-05 2024-09-20T22:53:39.312 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.312 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.312 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:53:39.312 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:40.232300+0000", 2024-09-20T22:53:39.312 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0045490260000000003 2024-09-20T22:53:39.312 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.312 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.312 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for blocked object", 2024-09-20T22:53:39.312 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:40.232354+0000", 2024-09-20T22:53:39.312 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.3183000000000002e-05 2024-09-20T22:53:39.312 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.313 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.313 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:53:39.313 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:40.244838+0000", 2024-09-20T22:53:39.313 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.012484004 2024-09-20T22:53:39.313 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.313 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.313 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:53:39.313 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:40.244974+0000", 2024-09-20T22:53:39.314 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013651399999999999 2024-09-20T22:53:39.314 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.314 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.314 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:53:39.314 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:40.245153+0000", 2024-09-20T22:53:39.314 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00017844000000000001 2024-09-20T22:53:39.314 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:53:39.314 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:53:39.314 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:53:39.314 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.314 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.314 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(osd.5.0:136 7.2 e269/169 7:45b54b6b:::smithi08477203-2:54 v 269'114, mlcod=265'111)", 2024-09-20T22:53:39.314 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:45:48.170083+0000", 2024-09-20T22:53:39.314 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 471.12761546899998, 2024-09-20T22:53:39.315 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.017111619000000002, 2024-09-20T22:53:39.315 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:53:39.315 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:53:39.315 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:53:39.315 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:53:39.315 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.315 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:53:39.315 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.170083+0000", 2024-09-20T22:53:39.315 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:53:39.315 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.315 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.315 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:53:39.315 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.170083+0000", 2024-09-20T22:53:39.315 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:53:39.316 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.316 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.316 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:53:39.316 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.170083+0000", 2024-09-20T22:53:39.316 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.2500000000000001e-07 2024-09-20T22:53:39.316 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.316 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.316 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:53:39.316 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.170096+0000", 2024-09-20T22:53:39.316 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3101999999999999e-05 2024-09-20T22:53:39.316 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.316 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.316 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:53:39.316 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.170101+0000", 2024-09-20T22:53:39.317 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.1379999999999998e-06 2024-09-20T22:53:39.317 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.317 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.317 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:53:39.317 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.170111+0000", 2024-09-20T22:53:39.317 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0305e-05 2024-09-20T22:53:39.317 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.317 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.317 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:53:39.317 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.173555+0000", 2024-09-20T22:53:39.317 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0034440389999999999 2024-09-20T22:53:39.317 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.317 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.318 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:53:39.318 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.173580+0000", 2024-09-20T22:53:39.318 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.4142e-05 2024-09-20T22:53:39.318 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.318 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.318 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:53:39.318 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.187133+0000", 2024-09-20T22:53:39.318 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013553170999999999 2024-09-20T22:53:39.318 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.318 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.318 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:53:39.318 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.187194+0000", 2024-09-20T22:53:39.318 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.1396999999999995e-05 2024-09-20T22:53:39.319 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:53:39.319 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:53:39.319 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:53:39.319 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.319 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.319 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:3124 7.b 7:df7b2b73:::smithi08477203-10:head [write 695922~709528] snapc 77=[77,76,75,74,73,70,6b] ondisk+write+known_if_redirected+supports_pool_eio e328)", 2024-09-20T22:53:39.319 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:46:48.190758+0000", 2024-09-20T22:53:39.319 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 411.106940275, 2024-09-20T22:53:39.319 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.030625260000000001, 2024-09-20T22:53:39.319 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:53:39.319 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:53:39.319 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:53:39.319 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:53:39.319 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:53:39.320 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:53:39.320 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 3124 2024-09-20T22:53:39.320 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.320 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:53:39.320 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.320 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:53:39.320 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.190758+0000", 2024-09-20T22:53:39.320 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:53:39.320 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.320 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.320 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:53:39.320 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.190758+0000", 2024-09-20T22:53:39.320 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:53:39.320 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.321 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.321 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:53:39.321 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.190760+0000", 2024-09-20T22:53:39.321 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.0109999999999999e-06 2024-09-20T22:53:39.321 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.321 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.321 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:53:39.321 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.191218+0000", 2024-09-20T22:53:39.321 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00045814 2024-09-20T22:53:39.321 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.321 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.321 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:53:39.321 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.191242+0000", 2024-09-20T22:53:39.321 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.4264e-05 2024-09-20T22:53:39.322 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.322 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.322 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:53:39.322 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.191256+0000", 2024-09-20T22:53:39.322 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.345e-05 2024-09-20T22:53:39.322 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.322 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.322 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:53:39.322 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.191362+0000", 2024-09-20T22:53:39.322 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00010649699999999999 2024-09-20T22:53:39.322 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.322 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.322 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op must wait for map", 2024-09-20T22:53:39.322 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.191370+0000", 2024-09-20T22:53:39.323 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.9049999999999997e-06 2024-09-20T22:53:39.323 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.323 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.323 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:53:39.323 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.202536+0000", 2024-09-20T22:53:39.323 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011165552 2024-09-20T22:53:39.323 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.323 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.323 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:53:39.323 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.202637+0000", 2024-09-20T22:53:39.323 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000101648 2024-09-20T22:53:39.323 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.323 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.323 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5,6", 2024-09-20T22:53:39.324 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.202824+0000", 2024-09-20T22:53:39.324 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00018637699999999999 2024-09-20T22:53:39.324 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.324 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.324 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:53:39.324 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.211036+0000", 2024-09-20T22:53:39.324 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0082124829999999996 2024-09-20T22:53:39.324 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.324 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.324 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:53:39.324 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.217521+0000", 2024-09-20T22:53:39.324 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0064849080000000002 2024-09-20T22:53:39.324 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.324 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.325 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:53:39.325 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.221251+0000", 2024-09-20T22:53:39.325 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0037300200000000001 2024-09-20T22:53:39.325 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.325 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.325 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:53:39.325 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.221325+0000", 2024-09-20T22:53:39.325 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.3737000000000002e-05 2024-09-20T22:53:39.325 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.325 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.325 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:53:39.325 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.221383+0000", 2024-09-20T22:53:39.325 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.8267999999999998e-05 2024-09-20T22:53:39.325 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:53:39.326 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:53:39.326 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:53:39.326 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.326 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.326 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:3125 7.b 7:df7b2b73:::smithi08477203-10:head [write 2032616~607480] snapc 77=[77,76,75,74,73,70,6b] ondisk+write+known_if_redirected+supports_pool_eio e328)", 2024-09-20T22:53:39.326 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:46:48.204698+0000", 2024-09-20T22:53:39.326 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 411.09300050500002, 2024-09-20T22:53:39.326 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.016777378999999999, 2024-09-20T22:53:39.326 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:53:39.326 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:53:39.326 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:53:39.326 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:53:39.326 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:53:39.326 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:53:39.327 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 3125 2024-09-20T22:53:39.327 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.327 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:53:39.327 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.327 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:53:39.327 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.204698+0000", 2024-09-20T22:53:39.327 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:53:39.327 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.327 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.327 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:53:39.327 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.204698+0000", 2024-09-20T22:53:39.327 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:53:39.327 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.327 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.327 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:53:39.328 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.204700+0000", 2024-09-20T22:53:39.328 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.1890000000000001e-06 2024-09-20T22:53:39.328 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.328 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.328 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:53:39.328 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.204872+0000", 2024-09-20T22:53:39.328 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00017232799999999999 2024-09-20T22:53:39.328 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.328 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.328 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:53:39.328 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.204885+0000", 2024-09-20T22:53:39.328 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3135e-05 2024-09-20T22:53:39.328 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.328 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.329 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:53:39.329 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.204896+0000", 2024-09-20T22:53:39.329 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1048e-05 2024-09-20T22:53:39.329 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.329 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.329 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:53:39.329 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.205292+0000", 2024-09-20T22:53:39.329 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00039570499999999999 2024-09-20T22:53:39.329 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.329 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.329 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:53:39.329 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.205404+0000", 2024-09-20T22:53:39.329 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000112224 2024-09-20T22:53:39.329 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.329 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.330 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5,6", 2024-09-20T22:53:39.330 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.206058+0000", 2024-09-20T22:53:39.330 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00065375100000000005 2024-09-20T22:53:39.330 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.330 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.330 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:53:39.330 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.211942+0000", 2024-09-20T22:53:39.330 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0058842690000000001 2024-09-20T22:53:39.330 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.330 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.330 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:53:39.330 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.217575+0000", 2024-09-20T22:53:39.330 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.005633201 2024-09-20T22:53:39.331 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.331 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.331 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:53:39.331 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.221393+0000", 2024-09-20T22:53:39.331 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00381797 2024-09-20T22:53:39.331 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.331 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.331 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:53:39.331 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.221438+0000", 2024-09-20T22:53:39.331 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.4347000000000001e-05 2024-09-20T22:53:39.331 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.331 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.331 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:53:39.331 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.221475+0000", 2024-09-20T22:53:39.332 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.7212000000000002e-05 2024-09-20T22:53:39.332 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:53:39.332 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:53:39.332 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:53:39.332 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.332 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.332 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:3237 7.b 7:d960cad4:::smithi08477203-18:head [copy-from ver 252] snapc 7e=[7e,7d,7c,7b,7a,79,78,75,74,73,6b] ondisk+write+known_if_redirected+supports_pool_eio e338)", 2024-09-20T22:53:39.332 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:46:58.180925+0000", 2024-09-20T22:53:39.332 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 401.11677294600003, 2024-09-20T22:53:39.332 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.029255882, 2024-09-20T22:53:39.332 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:53:39.332 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:53:39.332 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:53:39.332 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:53:39.333 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:53:39.333 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:53:39.333 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 3237 2024-09-20T22:53:39.333 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.333 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:53:39.333 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.333 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:53:39.333 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.180925+0000", 2024-09-20T22:53:39.333 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:53:39.333 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.333 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.333 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:53:39.333 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.180925+0000", 2024-09-20T22:53:39.333 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:53:39.334 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.334 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.334 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:53:39.334 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.180927+0000", 2024-09-20T22:53:39.334 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.945e-06 2024-09-20T22:53:39.334 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.334 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.334 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:53:39.334 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.180933+0000", 2024-09-20T22:53:39.334 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.7400000000000001e-06 2024-09-20T22:53:39.334 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.334 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.334 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:53:39.334 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.180947+0000", 2024-09-20T22:53:39.335 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3947999999999999e-05 2024-09-20T22:53:39.335 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.335 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.335 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:53:39.335 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.180954+0000", 2024-09-20T22:53:39.335 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.25e-06 2024-09-20T22:53:39.335 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.335 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.335 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:53:39.335 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.181002+0000", 2024-09-20T22:53:39.335 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.8331000000000002e-05 2024-09-20T22:53:39.335 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.335 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.335 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:53:39.336 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.181101+0000", 2024-09-20T22:53:39.336 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.8553000000000003e-05 2024-09-20T22:53:39.336 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.336 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.336 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5,6", 2024-09-20T22:53:39.336 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.196353+0000", 2024-09-20T22:53:39.336 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.015251789 2024-09-20T22:53:39.336 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.336 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.336 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:53:39.336 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.202094+0000", 2024-09-20T22:53:39.336 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0057409829999999999 2024-09-20T22:53:39.336 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.336 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.336 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:53:39.337 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.202682+0000", 2024-09-20T22:53:39.337 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000587898 2024-09-20T22:53:39.337 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.337 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.337 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:53:39.337 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.210104+0000", 2024-09-20T22:53:39.337 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0074227900000000003 2024-09-20T22:53:39.337 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.337 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.337 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:53:39.337 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.210145+0000", 2024-09-20T22:53:39.337 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.1050999999999997e-05 2024-09-20T22:53:39.337 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.337 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.337 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:53:39.338 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.210181+0000", 2024-09-20T22:53:39.338 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.5604e-05 2024-09-20T22:53:39.338 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:53:39.338 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:53:39.338 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:53:39.338 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.338 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.338 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:3238 7.b 7:d960cad4:::smithi08477203-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e338)", 2024-09-20T22:53:39.338 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:46:58.180973+0000", 2024-09-20T22:53:39.338 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 401.116725261, 2024-09-20T22:53:39.338 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.020749782000000001, 2024-09-20T22:53:39.338 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:53:39.338 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:53:39.338 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-20T22:53:39.339 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:53:39.339 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:53:39.339 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:53:39.339 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 3238 2024-09-20T22:53:39.339 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.339 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:53:39.339 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.339 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:53:39.339 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.180973+0000", 2024-09-20T22:53:39.339 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:53:39.339 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.339 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.339 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:53:39.339 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.180973+0000", 2024-09-20T22:53:39.340 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:53:39.340 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.340 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.340 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:53:39.340 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.180973+0000", 2024-09-20T22:53:39.340 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.4000000000000002e-07 2024-09-20T22:53:39.340 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.340 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.340 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:53:39.340 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.180976+0000", 2024-09-20T22:53:39.340 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.576e-06 2024-09-20T22:53:39.340 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.340 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.340 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:53:39.341 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.180982+0000", 2024-09-20T22:53:39.341 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.0700000000000003e-06 2024-09-20T22:53:39.341 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.341 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.341 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:53:39.341 INFO:tasks.rados.rados.0.smithi084.stdout:2933: done (2 left) 2024-09-20T22:53:39.341 INFO:tasks.rados.rados.0.smithi084.stdout:2934: snap_remove snap 310 2024-09-20T22:53:39.342 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.180988+0000", 2024-09-20T22:53:39.342 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.5929999999999996e-06 2024-09-20T22:53:39.342 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.342 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.342 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:53:39.342 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.182094+0000", 2024-09-20T22:53:39.342 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001106604 2024-09-20T22:53:39.342 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.343 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.343 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for blocked object", 2024-09-20T22:53:39.343 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.182223+0000", 2024-09-20T22:53:39.343 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000129152 2024-09-20T22:53:39.343 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.343 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.343 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:53:39.343 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.201400+0000", 2024-09-20T22:53:39.343 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.019176192000000002 2024-09-20T22:53:39.343 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.343 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.343 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:53:39.343 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.201563+0000", 2024-09-20T22:53:39.344 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00016328999999999999 2024-09-20T22:53:39.344 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.344 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.344 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:53:39.344 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.201723+0000", 2024-09-20T22:53:39.344 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000159765 2024-09-20T22:53:39.344 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:53:39.344 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:53:39.344 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:53:39.344 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.344 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.344 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24652.0:3235 7.1 e338/157 7:84a866f5:::smithi08477203-27:head v 338'167, mlcod=320'164)", 2024-09-20T22:53:39.344 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:46:58.205374+0000", 2024-09-20T22:53:39.344 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 401.092324389, 2024-09-20T22:53:39.344 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.019862464999999999, 2024-09-20T22:53:39.344 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:53:39.345 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:53:39.345 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:53:39.345 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:53:39.345 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.345 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:53:39.345 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.205374+0000", 2024-09-20T22:53:39.345 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:53:39.345 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.345 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.345 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:53:39.345 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.205374+0000", 2024-09-20T22:53:39.345 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:53:39.345 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.345 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.345 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:53:39.345 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.205374+0000", 2024-09-20T22:53:39.346 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.3200000000000001e-07 2024-09-20T22:53:39.346 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.346 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.346 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:53:39.346 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.208223+0000", 2024-09-20T22:53:39.346 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0028490820000000002 2024-09-20T22:53:39.346 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.346 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.346 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:53:39.346 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.208249+0000", 2024-09-20T22:53:39.346 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.584e-05 2024-09-20T22:53:39.346 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.346 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.346 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:53:39.346 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.208262+0000", 2024-09-20T22:53:39.347 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3166999999999999e-05 2024-09-20T22:53:39.347 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.347 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.347 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:53:39.347 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.208344+0000", 2024-09-20T22:53:39.347 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.1567e-05 2024-09-20T22:53:39.347 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.347 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.347 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:53:39.347 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.208370+0000", 2024-09-20T22:53:39.347 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.5973999999999999e-05 2024-09-20T22:53:39.347 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.348 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.348 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:53:39.348 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.225185+0000", 2024-09-20T22:53:39.348 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.016814976999999998 2024-09-20T22:53:39.348 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.348 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.348 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:53:39.348 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.225236+0000", 2024-09-20T22:53:39.348 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.1526000000000001e-05 2024-09-20T22:53:39.348 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:53:39.348 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:53:39.348 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:53:39.348 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.348 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.349 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:3589 7.b 7:dc95255e:::smithi08477203-1:head [copy-from ver 516] snapc 8d=[8d,8c,8b,8a,89,88,87,86,85,84,80,7f,7a,74,73,6b] ondisk+write+known_if_redirected+supports_pool_eio e363)", 2024-09-20T22:53:39.349 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:47:23.191965+0000", 2024-09-20T22:53:39.349 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 376.10573298000003, 2024-09-20T22:53:39.349 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.027894147000000001, 2024-09-20T22:53:39.349 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:53:39.349 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:53:39.349 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:53:39.349 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:53:39.349 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:53:39.349 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:53:39.349 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 3589 2024-09-20T22:53:39.349 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.349 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:53:39.349 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.350 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:53:39.350 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:23.191965+0000", 2024-09-20T22:53:39.350 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:53:39.350 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.350 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.350 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:53:39.350 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:23.191965+0000", 2024-09-20T22:53:39.350 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:53:39.350 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.350 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.350 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:53:39.350 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:23.191967+0000", 2024-09-20T22:53:39.350 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.6920000000000001e-06 2024-09-20T22:53:39.351 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.351 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.351 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:53:39.351 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:23.191972+0000", 2024-09-20T22:53:39.351 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.3770000000000004e-06 2024-09-20T22:53:39.351 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.351 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.351 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:53:39.351 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:23.191985+0000", 2024-09-20T22:53:39.351 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2509e-05 2024-09-20T22:53:39.351 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.351 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.351 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:53:39.351 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:23.191993+0000", 2024-09-20T22:53:39.351 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.14e-06 2024-09-20T22:53:39.351 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.352 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.352 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:53:39.352 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:23.192151+0000", 2024-09-20T22:53:39.352 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00015796699999999999 2024-09-20T22:53:39.352 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.352 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.352 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:53:39.352 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:23.192343+0000", 2024-09-20T22:53:39.352 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00019251000000000001 2024-09-20T22:53:39.352 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.352 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.352 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5,6", 2024-09-20T22:53:39.352 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:23.205044+0000", 2024-09-20T22:53:39.352 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.012700751 2024-09-20T22:53:39.353 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.353 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.353 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:53:39.353 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:23.209060+0000", 2024-09-20T22:53:39.353 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0040161959999999997 2024-09-20T22:53:39.353 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.353 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.353 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:53:39.353 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:23.219722+0000", 2024-09-20T22:53:39.353 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010661976 2024-09-20T22:53:39.353 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.353 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.353 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:53:39.353 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:23.219767+0000", 2024-09-20T22:53:39.353 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.4768999999999998e-05 2024-09-20T22:53:39.354 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.354 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.354 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:53:39.354 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:23.219810+0000", 2024-09-20T22:53:39.354 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.3544000000000002e-05 2024-09-20T22:53:39.354 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.354 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.354 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:53:39.354 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:23.219859+0000", 2024-09-20T22:53:39.354 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.8715999999999999e-05 2024-09-20T22:53:39.354 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:53:39.354 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:53:39.354 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:53:39.354 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.354 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.355 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:3590 7.b 7:dc95255e:::smithi08477203-1:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e363)", 2024-09-20T22:53:39.355 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:47:23.192017+0000", 2024-09-20T22:53:39.355 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 376.10568094199999, 2024-09-20T22:53:39.355 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.016178512999999999, 2024-09-20T22:53:39.355 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:53:39.355 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:53:39.355 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-20T22:53:39.355 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:53:39.355 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:53:39.355 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:53:39.355 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 3590 2024-09-20T22:53:39.355 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.355 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:53:39.355 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.355 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:53:39.355 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:23.192017+0000", 2024-09-20T22:53:39.356 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:53:39.356 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.356 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.356 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:53:39.356 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:23.192017+0000", 2024-09-20T22:53:39.356 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:53:39.356 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.356 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.356 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:53:39.356 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:23.192018+0000", 2024-09-20T22:53:39.356 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.4300000000000003e-07 2024-09-20T22:53:39.356 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.356 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.356 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:53:39.356 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:23.192020+0000", 2024-09-20T22:53:39.356 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.553e-06 2024-09-20T22:53:39.356 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.357 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.357 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:53:39.357 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:23.192027+0000", 2024-09-20T22:53:39.357 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.3990000000000001e-06 2024-09-20T22:53:39.357 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.357 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.357 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:53:39.357 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:23.192032+0000", 2024-09-20T22:53:39.357 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.3709999999999999e-06 2024-09-20T22:53:39.357 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.357 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.357 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:53:39.357 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:23.192511+0000", 2024-09-20T22:53:39.357 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00047946700000000002 2024-09-20T22:53:39.357 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.357 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.358 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for blocked object", 2024-09-20T22:53:39.358 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:23.192563+0000", 2024-09-20T22:53:39.358 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.1158e-05 2024-09-20T22:53:39.358 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.358 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.358 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:53:39.358 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:23.207970+0000", 2024-09-20T22:53:39.358 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.015407522999999999 2024-09-20T22:53:39.358 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.358 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.358 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:53:39.358 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:23.208078+0000", 2024-09-20T22:53:39.358 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00010755999999999999 2024-09-20T22:53:39.358 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.359 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.359 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:53:39.359 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:23.208196+0000", 2024-09-20T22:53:39.359 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000117939 2024-09-20T22:53:39.359 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:53:39.359 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:53:39.359 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:53:39.359 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.359 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.359 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:3760 7.13 7:cae5792a:::smithi08477203-32:head [copy-from ver 131] snapc 98=[98,97,96,95,94,93,92,8e,8d,8c,8b,8a,88,87,86,80,7f,73] ondisk+write+known_if_redirected+supports_pool_eio e383)", 2024-09-20T22:53:39.359 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:47:43.282985+0000", 2024-09-20T22:53:39.359 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 356.01471258200002, 2024-09-20T22:53:39.359 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.021555008000000001, 2024-09-20T22:53:39.359 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:53:39.360 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:53:39.360 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:53:39.360 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:53:39.360 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:53:39.360 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:53:39.360 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 3760 2024-09-20T22:53:39.360 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.360 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:53:39.360 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.360 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:53:39.360 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:43.282985+0000", 2024-09-20T22:53:39.360 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:53:39.361 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.361 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.361 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:53:39.361 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:43.282985+0000", 2024-09-20T22:53:39.361 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:53:39.361 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.361 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.361 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:53:39.361 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:43.282989+0000", 2024-09-20T22:53:39.361 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.4570000000000001e-06 2024-09-20T22:53:39.361 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.361 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.361 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:53:39.361 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:43.283000+0000", 2024-09-20T22:53:39.362 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1012999999999999e-05 2024-09-20T22:53:39.362 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.362 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.362 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:53:39.362 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:43.283026+0000", 2024-09-20T22:53:39.362 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.6313e-05 2024-09-20T22:53:39.362 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.362 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.362 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:53:39.362 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:43.283038+0000", 2024-09-20T22:53:39.362 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1976e-05 2024-09-20T22:53:39.362 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.362 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.363 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:53:39.363 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:43.283136+0000", 2024-09-20T22:53:39.363 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.7569e-05 2024-09-20T22:53:39.363 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.363 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.363 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:53:39.363 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:43.283257+0000", 2024-09-20T22:53:39.363 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000121408 2024-09-20T22:53:39.363 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.363 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.363 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 2,5", 2024-09-20T22:53:39.363 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:43.291446+0000", 2024-09-20T22:53:39.363 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0081890850000000005 2024-09-20T22:53:39.363 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.363 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.364 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:53:39.364 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:43.303163+0000", 2024-09-20T22:53:39.364 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011716809 2024-09-20T22:53:39.364 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.364 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.364 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:53:39.364 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:43.304043+0000", 2024-09-20T22:53:39.364 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00087947299999999997 2024-09-20T22:53:39.364 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.364 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.364 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:53:39.364 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:43.304457+0000", 2024-09-20T22:53:39.364 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00041483299999999998 2024-09-20T22:53:39.364 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.364 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.365 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:53:39.365 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:43.304500+0000", 2024-09-20T22:53:39.365 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.2484000000000002e-05 2024-09-20T22:53:39.365 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.365 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.365 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:53:39.365 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:43.304540+0000", 2024-09-20T22:53:39.365 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.0587999999999998e-05 2024-09-20T22:53:39.365 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:53:39.365 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:53:39.365 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:53:39.365 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.365 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.365 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(osd.5.0:213 7.2 e404/169 7:45b54b6b:::smithi08477203-2:a5 v 404'178, mlcod=402'175)", 2024-09-20T22:53:39.365 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:48:04.218296+0000", 2024-09-20T22:53:39.365 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 335.07940185500001, 2024-09-20T22:53:39.366 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.016469849000000002, 2024-09-20T22:53:39.366 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:53:39.366 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:53:39.366 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:53:39.366 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:53:39.366 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.366 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:53:39.366 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:04.218296+0000", 2024-09-20T22:53:39.366 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:53:39.366 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.366 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.366 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:53:39.366 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:04.218296+0000", 2024-09-20T22:53:39.366 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:53:39.366 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.366 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.367 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:53:39.367 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:04.218297+0000", 2024-09-20T22:53:39.367 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.3799999999999998e-07 2024-09-20T22:53:39.367 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.367 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.367 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:53:39.367 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:04.218306+0000", 2024-09-20T22:53:39.367 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.9429999999999995e-06 2024-09-20T22:53:39.367 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.367 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.367 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:53:39.367 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:04.218312+0000", 2024-09-20T22:53:39.367 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.7679999999999997e-06 2024-09-20T22:53:39.367 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.368 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.368 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:53:39.368 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:04.218321+0000", 2024-09-20T22:53:39.368 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.6619999999999994e-06 2024-09-20T22:53:39.368 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.368 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.368 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:53:39.368 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:04.233566+0000", 2024-09-20T22:53:39.368 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.015245491999999999 2024-09-20T22:53:39.368 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.368 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.368 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:53:39.368 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:04.233589+0000", 2024-09-20T22:53:39.368 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.2317e-05 2024-09-20T22:53:39.369 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.369 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.369 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:53:39.369 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:04.234725+0000", 2024-09-20T22:53:39.369 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0011366460000000001 2024-09-20T22:53:39.369 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.369 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.369 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:53:39.369 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:04.234766+0000", 2024-09-20T22:53:39.369 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.0683000000000003e-05 2024-09-20T22:53:39.369 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:53:39.369 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:53:39.369 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:53:39.369 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.369 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.370 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24652.0:4444 7.0 e447/137 7:05bb442d:::smithi08477203-24:head v 447'216, mlcod=444'212)", 2024-09-20T22:53:39.370 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:48:47.242826+0000", 2024-09-20T22:53:39.370 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 292.05487173400002, 2024-09-20T22:53:39.370 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.017527779, 2024-09-20T22:53:39.370 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:53:39.370 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:53:39.370 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:53:39.370 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:53:39.370 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.370 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:53:39.370 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:47.242826+0000", 2024-09-20T22:53:39.370 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:53:39.370 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.371 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.371 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:53:39.371 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:47.242826+0000", 2024-09-20T22:53:39.371 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:53:39.371 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.371 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.371 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:53:39.371 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:47.242827+0000", 2024-09-20T22:53:39.371 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.9999999999999998e-07 2024-09-20T22:53:39.371 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.371 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.371 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:53:39.371 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:47.251760+0000", 2024-09-20T22:53:39.371 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0089333339999999994 2024-09-20T22:53:39.371 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.372 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.372 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:53:39.372 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:47.251788+0000", 2024-09-20T22:53:39.372 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.8343e-05 2024-09-20T22:53:39.372 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.372 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.372 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:53:39.372 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:47.251806+0000", 2024-09-20T22:53:39.372 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.7909000000000001e-05 2024-09-20T22:53:39.372 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.372 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.372 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:53:39.372 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:47.251920+0000", 2024-09-20T22:53:39.373 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000113165 2024-09-20T22:53:39.373 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.373 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.373 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:53:39.373 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:47.251944+0000", 2024-09-20T22:53:39.373 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.4507000000000001e-05 2024-09-20T22:53:39.373 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.373 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.373 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:53:39.373 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:47.260296+0000", 2024-09-20T22:53:39.373 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0083519419999999994 2024-09-20T22:53:39.373 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.373 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.374 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:53:39.374 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:47.260354+0000", 2024-09-20T22:53:39.374 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.8078999999999997e-05 2024-09-20T22:53:39.374 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:53:39.374 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:53:39.374 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:53:39.374 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.374 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.374 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24652.0:4640 7.9 e468/426 7:907aec0e:::smithi08477203-3:head v 468'434, mlcod=468'431)", 2024-09-20T22:53:39.374 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:49:08.245712+0000", 2024-09-20T22:53:39.374 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 271.05198560399998, 2024-09-20T22:53:39.374 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.01546574, 2024-09-20T22:53:39.374 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:53:39.374 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:53:39.375 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:53:39.375 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:53:39.375 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.375 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:53:39.375 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:49:08.245712+0000", 2024-09-20T22:53:39.375 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:53:39.375 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.375 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.375 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:53:39.375 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:49:08.245712+0000", 2024-09-20T22:53:39.375 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:53:39.375 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.375 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.375 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:53:39.376 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:49:08.245713+0000", 2024-09-20T22:53:39.376 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.6300000000000004e-07 2024-09-20T22:53:39.376 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.376 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.376 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:53:39.376 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:49:08.246012+0000", 2024-09-20T22:53:39.376 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000298521 2024-09-20T22:53:39.376 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.376 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.376 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:53:39.376 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:49:08.246024+0000", 2024-09-20T22:53:39.376 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2014999999999999e-05 2024-09-20T22:53:39.376 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.376 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.376 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:53:39.377 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:49:08.246034+0000", 2024-09-20T22:53:39.377 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0147e-05 2024-09-20T22:53:39.377 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.377 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.377 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:53:39.377 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:49:08.246097+0000", 2024-09-20T22:53:39.377 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.2874999999999997e-05 2024-09-20T22:53:39.377 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.377 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.377 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:53:39.377 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:49:08.246123+0000", 2024-09-20T22:53:39.377 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.6211999999999999e-05 2024-09-20T22:53:39.377 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.377 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.377 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:53:39.377 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:49:08.261118+0000", 2024-09-20T22:53:39.378 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014995517999999999 2024-09-20T22:53:39.378 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.378 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.378 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:53:39.378 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:49:08.261178+0000", 2024-09-20T22:53:39.378 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.9688999999999997e-05 2024-09-20T22:53:39.378 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:53:39.378 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:53:39.378 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:53:39.378 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.378 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.378 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24652.0:4638 7.d e468/391 7:b30e4612:::smithi08477203-42:head v 468'257, mlcod=466'252)", 2024-09-20T22:53:39.378 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:49:08.256560+0000", 2024-09-20T22:53:39.378 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 271.04113840100001, 2024-09-20T22:53:39.378 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.015962423999999999, 2024-09-20T22:53:39.379 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:53:39.379 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:53:39.379 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:53:39.379 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:53:39.379 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.379 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:53:39.379 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:49:08.256560+0000", 2024-09-20T22:53:39.379 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:53:39.379 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.379 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.379 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:53:39.379 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:49:08.256560+0000", 2024-09-20T22:53:39.379 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:53:39.380 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.380 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.380 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:53:39.380 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:49:08.256560+0000", 2024-09-20T22:53:39.380 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.7100000000000001e-07 2024-09-20T22:53:39.380 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.380 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.380 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:53:39.380 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:49:08.263534+0000", 2024-09-20T22:53:39.380 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0069739069999999997 2024-09-20T22:53:39.380 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.380 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.380 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:53:39.380 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:49:08.263550+0000", 2024-09-20T22:53:39.380 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.5974999999999999e-05 2024-09-20T22:53:39.380 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.381 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.381 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:53:39.381 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:49:08.263560+0000", 2024-09-20T22:53:39.381 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.5889999999999992e-06 2024-09-20T22:53:39.381 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.381 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.381 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:53:39.381 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:49:08.263643+0000", 2024-09-20T22:53:39.381 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.3369999999999996e-05 2024-09-20T22:53:39.381 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.381 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.381 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:53:39.381 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:49:08.263666+0000", 2024-09-20T22:53:39.381 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.3224e-05 2024-09-20T22:53:39.382 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.382 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.382 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:53:39.382 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:49:08.272477+0000", 2024-09-20T22:53:39.382 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0088104729999999992 2024-09-20T22:53:39.382 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.382 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.382 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:53:39.382 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:49:08.272522+0000", 2024-09-20T22:53:39.382 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.5114999999999997e-05 2024-09-20T22:53:39.382 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:53:39.382 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:53:39.382 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:53:39.383 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.383 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.383 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:5011 7.b 7:df7b2b73:::smithi08477203-10:head [copy-from ver 1207] snapc d8=[d8,d7,d6,d5,d4,d3,d2,d1,d0,cf,ce,cd,cc,ca,c7,c6,c5,c2,bf,ba,b4,b2,a7,9e,9c,95,87] ondisk+write+known_if_redirected+supports_pool_eio e505)", 2024-09-20T22:53:39.383 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:49:45.297952+0000", 2024-09-20T22:53:39.383 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 233.999745687, 2024-09-20T22:53:39.383 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.022999847, 2024-09-20T22:53:39.383 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:53:39.383 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:53:39.383 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:53:39.383 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:53:39.383 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:53:39.383 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:53:39.383 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 5011 2024-09-20T22:53:39.383 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.383 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:53:39.384 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.384 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:53:39.384 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:49:45.297952+0000", 2024-09-20T22:53:39.384 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:53:39.384 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.384 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.384 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:53:39.384 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:49:45.297952+0000", 2024-09-20T22:53:39.384 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:53:39.384 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.384 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.384 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:53:39.384 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:49:45.297954+0000", 2024-09-20T22:53:39.385 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.9190000000000002e-06 2024-09-20T22:53:39.385 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.385 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.385 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:53:39.385 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:49:45.297967+0000", 2024-09-20T22:53:39.385 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.242e-05 2024-09-20T22:53:39.385 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.385 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.385 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:53:39.385 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:49:45.297982+0000", 2024-09-20T22:53:39.385 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.5183e-05 2024-09-20T22:53:39.385 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.385 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.385 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:53:39.386 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:49:45.297990+0000", 2024-09-20T22:53:39.386 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.0020000000000006e-06 2024-09-20T22:53:39.386 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.386 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.386 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:53:39.386 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:49:45.298075+0000", 2024-09-20T22:53:39.386 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.4852999999999995e-05 2024-09-20T22:53:39.386 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.386 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.386 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:53:39.386 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:49:45.298178+0000", 2024-09-20T22:53:39.386 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000103553 2024-09-20T22:53:39.386 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.387 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.387 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5,6", 2024-09-20T22:53:39.387 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:49:45.305815+0000", 2024-09-20T22:53:39.387 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0076365349999999999 2024-09-20T22:53:39.387 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.387 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.387 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:53:39.387 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:49:45.310882+0000", 2024-09-20T22:53:39.387 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.005066966 2024-09-20T22:53:39.387 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.387 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.387 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:53:39.387 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:49:45.320429+0000", 2024-09-20T22:53:39.388 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0095475709999999995 2024-09-20T22:53:39.388 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.388 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.388 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:53:39.388 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:49:45.320842+0000", 2024-09-20T22:53:39.388 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00041306300000000003 2024-09-20T22:53:39.388 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.388 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.388 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:53:39.388 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:49:45.320886+0000", 2024-09-20T22:53:39.388 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.3204999999999997e-05 2024-09-20T22:53:39.388 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.388 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.388 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:53:39.388 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:49:45.320952+0000", 2024-09-20T22:53:39.388 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.6576999999999996e-05 2024-09-20T22:53:39.389 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:53:39.389 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:53:39.389 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:53:39.389 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.389 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.389 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:5514 7.b 7:d3ca3619:::smithi08477203-39:head [copy-from ver 729] snapc f4=[f4,f3,f2,f1,f0,ef,ed,ec,ea,e8,e6,e4,e2,df,db,da,d7,d5,d4,d3,cf,ce,c5,c2,bf,b4,b2,a7] ondisk+write+known_if_redirected+supports_pool_eio e560)", 2024-09-20T22:53:39.389 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:50:40.272011+0000", 2024-09-20T22:53:39.389 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 179.02568692099999, 2024-09-20T22:53:39.389 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.028606645, 2024-09-20T22:53:39.389 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:53:39.389 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:53:39.389 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:53:39.390 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:53:39.390 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:53:39.390 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:53:39.390 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 5514 2024-09-20T22:53:39.390 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.390 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:53:39.390 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.390 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:53:39.390 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:40.272011+0000", 2024-09-20T22:53:39.390 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:53:39.390 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.390 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.390 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:53:39.390 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:40.272011+0000", 2024-09-20T22:53:39.390 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:53:39.391 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.391 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.391 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:53:39.391 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:40.272013+0000", 2024-09-20T22:53:39.391 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.3159999999999999e-06 2024-09-20T22:53:39.391 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.391 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.391 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:53:39.391 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:40.272023+0000", 2024-09-20T22:53:39.391 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.9229999999999997e-06 2024-09-20T22:53:39.391 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.391 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.391 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:53:39.391 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:40.272044+0000", 2024-09-20T22:53:39.392 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.0590000000000001e-05 2024-09-20T22:53:39.392 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.392 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.392 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:53:39.392 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:40.272054+0000", 2024-09-20T22:53:39.392 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.9760000000000007e-06 2024-09-20T22:53:39.392 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.392 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.392 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:53:39.392 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:40.272138+0000", 2024-09-20T22:53:39.392 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.4122999999999994e-05 2024-09-20T22:53:39.392 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.392 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.392 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op must wait for map", 2024-09-20T22:53:39.393 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:40.272147+0000", 2024-09-20T22:53:39.393 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.1919999999999994e-06 2024-09-20T22:53:39.393 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.393 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.393 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:53:39.393 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:40.277061+0000", 2024-09-20T22:53:39.393 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0049135070000000001 2024-09-20T22:53:39.393 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.393 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.393 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:53:39.393 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:40.277135+0000", 2024-09-20T22:53:39.394 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.4307999999999994e-05 2024-09-20T22:53:39.394 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.394 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.394 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5,6", 2024-09-20T22:53:39.394 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:40.288318+0000", 2024-09-20T22:53:39.394 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011183324 2024-09-20T22:53:39.394 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.394 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.394 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:53:39.394 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:40.296101+0000", 2024-09-20T22:53:39.394 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0077823500000000004 2024-09-20T22:53:39.394 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.394 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.394 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:53:39.394 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:40.299263+0000", 2024-09-20T22:53:39.395 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0031626279999999998 2024-09-20T22:53:39.395 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.395 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.395 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:53:39.395 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:40.300471+0000", 2024-09-20T22:53:39.395 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00120729 2024-09-20T22:53:39.395 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.395 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.395 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:53:39.395 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:40.300554+0000", 2024-09-20T22:53:39.395 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.3728999999999997e-05 2024-09-20T22:53:39.395 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.395 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.395 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:53:39.395 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:40.300618+0000", 2024-09-20T22:53:39.395 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.3388999999999999e-05 2024-09-20T22:53:39.396 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:53:39.396 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:53:39.396 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:53:39.396 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.396 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:53:39.397 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.397 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:5515 7.b 7:d3ca3619:::smithi08477203-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e560)", 2024-09-20T22:53:39.397 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:50:40.272081+0000", 2024-09-20T22:53:39.397 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 179.02561710399999, 2024-09-20T22:53:39.397 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.020994938000000001, 2024-09-20T22:53:39.397 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:53:39.397 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:53:39.397 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-20T22:53:39.397 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:53:39.397 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:53:39.397 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:53:39.397 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 5515 2024-09-20T22:53:39.398 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.398 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:53:39.398 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.398 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:53:39.398 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:40.272081+0000", 2024-09-20T22:53:39.398 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:53:39.398 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.398 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.398 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:53:39.398 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:40.272081+0000", 2024-09-20T22:53:39.398 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:53:39.398 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.398 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.398 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:53:39.398 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:40.272082+0000", 2024-09-20T22:53:39.399 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.7700000000000004e-07 2024-09-20T22:53:39.399 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.399 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.399 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:53:39.399 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:40.272085+0000", 2024-09-20T22:53:39.399 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.5240000000000001e-06 2024-09-20T22:53:39.399 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.399 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.399 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:53:39.399 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:40.272093+0000", 2024-09-20T22:53:39.399 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.8790000000000003e-06 2024-09-20T22:53:39.399 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.399 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.399 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:53:39.399 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:40.272099+0000", 2024-09-20T22:53:39.400 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.879e-06 2024-09-20T22:53:39.400 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.400 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.400 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:53:39.400 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:40.272222+0000", 2024-09-20T22:53:39.400 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00012285 2024-09-20T22:53:39.400 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.400 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.400 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting_for_map not empty", 2024-09-20T22:53:39.400 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:40.272235+0000", 2024-09-20T22:53:39.400 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3788000000000001e-05 2024-09-20T22:53:39.400 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.400 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.400 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:53:39.401 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:40.277248+0000", 2024-09-20T22:53:39.401 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0050127210000000004 2024-09-20T22:53:39.401 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.401 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.401 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for blocked object", 2024-09-20T22:53:39.401 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:40.277277+0000", 2024-09-20T22:53:39.401 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.8498000000000001e-05 2024-09-20T22:53:39.401 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.401 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.401 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:53:39.401 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:40.292787+0000", 2024-09-20T22:53:39.401 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.015510636 2024-09-20T22:53:39.401 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.402 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.402 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:53:39.402 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:40.292928+0000", 2024-09-20T22:53:39.402 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014066400000000001 2024-09-20T22:53:39.402 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.402 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.402 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:53:39.402 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:40.293076+0000", 2024-09-20T22:53:39.402 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014792200000000001 2024-09-20T22:53:39.402 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:53:39.402 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:53:39.402 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:53:39.402 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.403 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.403 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:5659 7.b 7:dc95255e:::smithi08477203-1:head [copy-from ver 731] snapc f9=[f9,f8,f7,f6,f5,f4,f2,f1,ef,ed,ec,ea,e6,e4,e2,df,db,da,d7,d4,d3,ce,c5,c2,bf,b4,b2,a7] ondisk+write+known_if_redirected+supports_pool_eio e570)", 2024-09-20T22:53:39.403 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:50:51.277671+0000", 2024-09-20T22:53:39.403 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 168.02002654, 2024-09-20T22:53:39.403 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.024917069999999999, 2024-09-20T22:53:39.403 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:53:39.403 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:53:39.403 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:53:39.403 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:53:39.403 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:53:39.403 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:53:39.403 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 5659 2024-09-20T22:53:39.403 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.403 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:53:39.404 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.404 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:53:39.404 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:51.277671+0000", 2024-09-20T22:53:39.404 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:53:39.404 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.404 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.404 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:53:39.404 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:51.277671+0000", 2024-09-20T22:53:39.404 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:53:39.404 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.404 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.404 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:53:39.404 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:51.277672+0000", 2024-09-20T22:53:39.404 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.2399999999999999e-07 2024-09-20T22:53:39.404 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.404 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.405 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:53:39.405 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:51.277676+0000", 2024-09-20T22:53:39.405 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.0879999999999997e-06 2024-09-20T22:53:39.405 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.405 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.405 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:53:39.405 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:51.277682+0000", 2024-09-20T22:53:39.405 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.8130000000000001e-06 2024-09-20T22:53:39.405 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.405 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.405 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:53:39.405 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:51.277688+0000", 2024-09-20T22:53:39.405 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.9699999999999996e-06 2024-09-20T22:53:39.405 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.406 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.406 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:53:39.406 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:51.278165+0000", 2024-09-20T22:53:39.406 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00047736300000000002 2024-09-20T22:53:39.406 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.406 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.406 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:53:39.406 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:51.278262+0000", 2024-09-20T22:53:39.406 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.6638000000000005e-05 2024-09-20T22:53:39.406 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.406 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.406 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5,6", 2024-09-20T22:53:39.406 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:51.285554+0000", 2024-09-20T22:53:39.407 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0072918219999999999 2024-09-20T22:53:39.407 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.407 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.407 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:53:39.407 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:51.294109+0000", 2024-09-20T22:53:39.407 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0085554729999999992 2024-09-20T22:53:39.407 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.407 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.407 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:53:39.407 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:51.302364+0000", 2024-09-20T22:53:39.407 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0082549540000000001 2024-09-20T22:53:39.407 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.407 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.407 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:53:39.407 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:51.302426+0000", 2024-09-20T22:53:39.407 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.224e-05 2024-09-20T22:53:39.408 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.408 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.408 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:53:39.408 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:51.302507+0000", 2024-09-20T22:53:39.408 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.0797999999999994e-05 2024-09-20T22:53:39.408 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.408 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.408 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:53:39.408 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:51.302589+0000", 2024-09-20T22:53:39.408 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.1487000000000001e-05 2024-09-20T22:53:39.408 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:53:39.408 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:53:39.408 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:53:39.408 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.408 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.408 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24652.0:6378 7.2 e639/600 7:4d20232a:::smithi08477203-34:head v 639'665, mlcod=629'662)", 2024-09-20T22:53:39.408 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:51:59.278699+0000", 2024-09-20T22:53:39.409 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 100.018999067, 2024-09-20T22:53:39.409 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.015272501000000001, 2024-09-20T22:53:39.409 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:53:39.409 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:53:39.409 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:53:39.409 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:53:39.409 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.409 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:53:39.409 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:59.278699+0000", 2024-09-20T22:53:39.409 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:53:39.409 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.409 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.409 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:53:39.409 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:59.278699+0000", 2024-09-20T22:53:39.409 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:53:39.409 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.410 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.410 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:53:39.410 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:59.278699+0000", 2024-09-20T22:53:39.410 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.6399999999999998e-07 2024-09-20T22:53:39.410 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.410 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.410 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:53:39.410 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:59.281173+0000", 2024-09-20T22:53:39.410 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002473523 2024-09-20T22:53:39.410 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.410 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.410 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:53:39.410 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:59.281190+0000", 2024-09-20T22:53:39.410 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.7391e-05 2024-09-20T22:53:39.411 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.411 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.411 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:53:39.411 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:59.281199+0000", 2024-09-20T22:53:39.411 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.9900000000000003e-06 2024-09-20T22:53:39.411 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.411 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.411 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:53:39.411 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:59.281278+0000", 2024-09-20T22:53:39.411 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.8770999999999995e-05 2024-09-20T22:53:39.411 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.411 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.411 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:53:39.411 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:59.281313+0000", 2024-09-20T22:53:39.412 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.4573000000000003e-05 2024-09-20T22:53:39.412 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.412 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.412 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:53:39.412 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:59.293921+0000", 2024-09-20T22:53:39.412 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.012608263 2024-09-20T22:53:39.412 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:53:39.412 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:53:39.412 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:53:39.412 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:59.293971+0000", 2024-09-20T22:53:39.412 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.0625999999999999e-05 2024-09-20T22:53:39.412 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:53:39.412 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:53:39.412 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:53:39.413 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:53:39.413 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:53:39.413 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-20T22:53:39.434 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:39.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:39.435 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:53:39.441 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-20T22:53:39.536 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:53:39.557 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:39.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:39.557 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:53:39.614 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:39 smithi084 ceph-mon[40024]: pgmap v1008: 158 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 153 active+clean; 562 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail 2024-09-20T22:53:39.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:39 smithi084 ceph-mon[40024]: osdmap e739: 8 total, 8 up, 7 in 2024-09-20T22:53:39.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:39 smithi084 ceph-mon[29774]: pgmap v1008: 158 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 153 active+clean; 562 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail 2024-09-20T22:53:39.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:39 smithi084 ceph-mon[29774]: osdmap e739: 8 total, 8 up, 7 in 2024-09-20T22:53:39.658 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:53:39.681 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:39.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:39.682 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:53:39.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:39 smithi154 ceph-mon[33264]: pgmap v1008: 158 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 153 active+clean; 562 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail 2024-09-20T22:53:39.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:39 smithi154 ceph-mon[33264]: osdmap e739: 8 total, 8 up, 7 in 2024-09-20T22:53:39.754 INFO:journalctl@ceph.iscsi.iscsi.a.smithi154.stdout:Sep 20 22:53:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-iscsi-iscsi-a[59730]: debug there is no tcmu-runner data available 2024-09-20T22:53:39.783 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:53:39.805 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:39.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:39.806 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:53:39.907 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:53:39.928 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:39.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:39.929 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:53:40.030 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:53:40.051 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:40.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:40.052 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:53:40.153 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:53:40.173 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:40.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:40.174 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:53:40.275 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:53:40.301 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:40.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:40.302 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:53:40.307 INFO:tasks.rados.rados.0.smithi084.stdout:2934: done (2 left) 2024-09-20T22:53:40.307 INFO:tasks.rados.rados.0.smithi084.stdout:2935: rollback oid 46 current snap is 333 2024-09-20T22:53:40.307 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 46 to 257 2024-09-20T22:53:40.307 INFO:tasks.rados.rados.0.smithi084.stdout:2932: finishing rollback tid 0 to smithi08477203-24 2024-09-20T22:53:40.307 INFO:tasks.rados.rados.0.smithi084.stdout:2932: finishing rollback tid 1 to smithi08477203-24 2024-09-20T22:53:40.307 INFO:tasks.rados.rados.0.smithi084.stdout:2932: finishing rollback tid 2 to smithi08477203-24 2024-09-20T22:53:40.308 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 24 v 1855 (ObjNum 780 snap 252 seq_num 780) dirty exists 2024-09-20T22:53:40.308 INFO:tasks.rados.rados.0.smithi084.stdout:2931: finishing copy_from racing read to smithi08477203-40 2024-09-20T22:53:40.308 INFO:tasks.rados.rados.0.smithi084.stdout:2931: finishing copy_from to smithi08477203-40 2024-09-20T22:53:40.308 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 40 v 1901 (ObjNum 879 snap 285 seq_num 879) dirty exists 2024-09-20T22:53:40.308 INFO:tasks.rados.rados.0.smithi084.stdout:2931: done (2 left) 2024-09-20T22:53:40.308 INFO:tasks.rados.rados.0.smithi084.stdout:2932: done (1 left) 2024-09-20T22:53:40.308 INFO:tasks.rados.rados.0.smithi084.stdout:2936: snap_remove snap 324 2024-09-20T22:53:40.402 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:53:40.425 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:40.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:40.426 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:53:40.444 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-20T22:53:40.487 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:53:40.523 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:53:40.527 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:53:40.549 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:40.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:40.551 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:53:40.651 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:53:40.655 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:40 smithi154 ceph-mon[33264]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:53:40.655 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:40 smithi154 ceph-mon[33264]: osdmap e740: 8 total, 8 up, 7 in 2024-09-20T22:53:40.672 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:40.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:40.674 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:53:40.719 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:40 smithi084 ceph-mon[29774]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:53:40.719 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:40 smithi084 ceph-mon[29774]: osdmap e740: 8 total, 8 up, 7 in 2024-09-20T22:53:40.719 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:40 smithi084 ceph-mon[40024]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:53:40.719 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:40 smithi084 ceph-mon[40024]: osdmap e740: 8 total, 8 up, 7 in 2024-09-20T22:53:40.775 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:53:40.796 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:40.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:40.797 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:53:40.848 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-20T22:53:40.898 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:53:40.918 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:40.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:40.920 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:53:41.020 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:53:41.042 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:41.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:41.042 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:53:41.143 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:53:41.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:41.165 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:41.165 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:53:41.266 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:53:41.287 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:41.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:41.288 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:53:41.303 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.7 to 1 (10000) 2024-09-20T22:53:41.309 INFO:tasks.rados.rados.0.smithi084.stdout:2936: done (1 left) 2024-09-20T22:53:41.309 INFO:tasks.rados.rados.0.smithi084.stdout:2937: snap_remove snap 331 2024-09-20T22:53:41.389 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:53:41.410 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:41.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:41.411 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:53:41.512 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:53:41.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:41.534 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:41.534 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:53:41.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:41 smithi084 ceph-mon[40024]: pgmap v1011: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 535 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 13 MiB/s rd, 2.7 MiB/s wr, 20 op/s 2024-09-20T22:53:41.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:41 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/1809722217' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-09-20T22:53:41.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:41 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/1809722217' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-20T22:53:41.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:41 smithi084 ceph-mon[40024]: osdmap e741: 8 total, 8 up, 7 in 2024-09-20T22:53:41.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:41 smithi084 ceph-mon[29774]: pgmap v1011: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 535 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 13 MiB/s rd, 2.7 MiB/s wr, 20 op/s 2024-09-20T22:53:41.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:41 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/1809722217' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-09-20T22:53:41.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:41 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/1809722217' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-20T22:53:41.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:41 smithi084 ceph-mon[29774]: osdmap e741: 8 total, 8 up, 7 in 2024-09-20T22:53:41.635 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:53:41.656 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:41.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:41.657 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:53:41.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:41 smithi154 ceph-mon[33264]: pgmap v1011: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 535 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 13 MiB/s rd, 2.7 MiB/s wr, 20 op/s 2024-09-20T22:53:41.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:41 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/1809722217' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-09-20T22:53:41.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:41 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/1809722217' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-20T22:53:41.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:41 smithi154 ceph-mon[33264]: osdmap e741: 8 total, 8 up, 7 in 2024-09-20T22:53:41.758 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:53:41.781 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:41.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:41.782 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:53:41.882 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:53:41.904 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:41.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:41.904 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:53:42.005 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:53:42.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:42.027 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:42.027 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:53:42.128 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:53:42.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:42.150 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:42.150 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:53:42.251 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:53:42.273 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:42.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:42.273 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:53:42.299 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd reweight 0 1 2024-09-20T22:53:42.300 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:53:42.308 INFO:tasks.rados.rados.0.smithi084.stdout:2937: done (1 left) 2024-09-20T22:53:42.308 INFO:tasks.rados.rados.0.smithi084.stdout:2938: rollback oid 16 current snap is 333 2024-09-20T22:53:42.309 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 16 to 257 2024-09-20T22:53:42.309 INFO:tasks.rados.rados.0.smithi084.stdout:2939: read oid 10 snap -1 2024-09-20T22:53:42.309 INFO:tasks.rados.rados.0.smithi084.stdout:2939: expect deleted 2024-09-20T22:53:42.309 INFO:tasks.rados.rados.0.smithi084.stdout:2940: read oid 38 snap -1 2024-09-20T22:53:42.309 INFO:tasks.rados.rados.0.smithi084.stdout:2940: expect (ObjNum 918 snap 302 seq_num 918) 2024-09-20T22:53:42.309 INFO:tasks.rados.rados.0.smithi084.stdout:2941: rollback oid 4 current snap is 333 2024-09-20T22:53:42.309 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 4 to 293 2024-09-20T22:53:42.309 INFO:tasks.rados.rados.0.smithi084.stdout:2942: delete oid 14 current snap is 333 2024-09-20T22:53:42.309 INFO:tasks.rados.rados.0.smithi084.stdout:2935: finishing rollback tid 1 to smithi08477203-46 2024-09-20T22:53:42.309 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 46 v 1818 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-09-20T22:53:42.310 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is set 2024-09-20T22:53:42.315 INFO:tasks.rados.rados.0.smithi084.stdout:2938: finishing rollback tid 0 to smithi08477203-16 2024-09-20T22:53:42.319 INFO:tasks.rados.rados.0.smithi084.stdout:2938: finishing rollback tid 1 to smithi08477203-16 2024-09-20T22:53:42.319 INFO:tasks.rados.rados.0.smithi084.stdout:2938: finishing rollback tid 2 to smithi08477203-16 2024-09-20T22:53:42.319 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 16 v 1817 (ObjNum 419 snap 105 seq_num 419) dirty exists 2024-09-20T22:53:42.321 INFO:tasks.rados.rados.0.smithi084.stdout:2935: done (5 left) 2024-09-20T22:53:42.321 INFO:tasks.rados.rados.0.smithi084.stdout:2938: done (4 left) 2024-09-20T22:53:42.321 INFO:tasks.rados.rados.0.smithi084.stdout:2939: done (3 left) 2024-09-20T22:53:42.321 INFO:tasks.rados.rados.0.smithi084.stdout:2942: done (2 left) 2024-09-20T22:53:42.321 INFO:tasks.rados.rados.0.smithi084.stdout:2943: write oid 27 current snap is 333 2024-09-20T22:53:42.321 INFO:tasks.rados.rados.0.smithi084.stdout:2943: seq_num 1016 ranges {675912=727933,1920764=460673,2801290=274622} 2024-09-20T22:53:42.330 INFO:tasks.rados.rados.0.smithi084.stdout:2943: writing smithi08477203-27 from 675912 to 1403845 tid 1 2024-09-20T22:53:42.336 INFO:tasks.rados.rados.0.smithi084.stdout:2943: writing smithi08477203-27 from 1920764 to 2381437 tid 2 2024-09-20T22:53:42.340 INFO:tasks.rados.rados.0.smithi084.stdout:2943: writing smithi08477203-27 from 2801290 to 3075912 tid 3 2024-09-20T22:53:42.340 INFO:tasks.rados.rados.0.smithi084.stdout:2940: expect (ObjNum 918 snap 302 seq_num 918) 2024-09-20T22:53:42.374 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:53:42.397 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:42.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:42.398 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:53:42.499 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:53:42.507 INFO:tasks.rados.rados.0.smithi084.stdout:2941: finishing rollback tid 1 to smithi08477203-4 2024-09-20T22:53:42.507 INFO:tasks.rados.rados.0.smithi084.stdout:2941: finishing rollback tid 2 to smithi08477203-4 2024-09-20T22:53:42.507 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 4 v 2331 (ObjNum 863 snap 276 seq_num 863) dirty exists 2024-09-20T22:53:42.507 INFO:tasks.rados.rados.0.smithi084.stdout:2943: finishing write tid 1 to smithi08477203-27 2024-09-20T22:53:42.507 INFO:tasks.rados.rados.0.smithi084.stdout:2943: finishing write tid 2 to smithi08477203-27 2024-09-20T22:53:42.507 INFO:tasks.rados.rados.0.smithi084.stdout:2940: done (2 left) 2024-09-20T22:53:42.508 INFO:tasks.rados.rados.0.smithi084.stdout:2941: done (1 left) 2024-09-20T22:53:42.508 INFO:tasks.rados.rados.0.smithi084.stdout:2944: rollback oid 23 current snap is 333 2024-09-20T22:53:42.508 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 23 to 316 2024-09-20T22:53:42.508 INFO:tasks.rados.rados.0.smithi084.stdout:2943: finishing write tid 3 to smithi08477203-27 2024-09-20T22:53:42.508 INFO:tasks.rados.rados.0.smithi084.stdout:2943: finishing write tid 4 to smithi08477203-27 2024-09-20T22:53:42.508 INFO:tasks.rados.rados.0.smithi084.stdout:2943: finishing write tid 5 to smithi08477203-27 2024-09-20T22:53:42.508 INFO:tasks.rados.rados.0.smithi084.stdout:2943: oid 27 updating version 0 to 1809 2024-09-20T22:53:42.508 INFO:tasks.rados.rados.0.smithi084.stdout:2943: oid 27 version 1809 is already newer than 1806 2024-09-20T22:53:42.508 INFO:tasks.rados.rados.0.smithi084.stdout:2943: oid 27 version 1809 is already newer than 1808 2024-09-20T22:53:42.508 INFO:tasks.rados.rados.0.smithi084.stdout:2943: oid 27 version 1809 is already newer than 1807 2024-09-20T22:53:42.508 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 27 v 1809 (ObjNum 1016 snap 333 seq_num 1016) dirty exists 2024-09-20T22:53:42.508 INFO:tasks.rados.rados.0.smithi084.stdout:2943: left oid 27 (ObjNum 1016 snap 333 seq_num 1016) 2024-09-20T22:53:42.508 INFO:tasks.rados.rados.0.smithi084.stdout:2943: done (1 left) 2024-09-20T22:53:42.508 INFO:tasks.rados.rados.0.smithi084.stdout:2945: snap_remove snap 318 2024-09-20T22:53:42.521 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:42.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:42.522 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:53:42.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:42 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/4280414426' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:53:42.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:42 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:53:42.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:42 smithi084 ceph-mon[29774]: purged_snaps scrub starts 2024-09-20T22:53:42.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:42 smithi084 ceph-mon[29774]: purged_snaps scrub ok 2024-09-20T22:53:42.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:42 smithi084 ceph-mon[29774]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:53:42.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:42 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T22:53:42.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:42 smithi084 ceph-mon[29774]: osdmap e742: 8 total, 8 up, 7 in 2024-09-20T22:53:42.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:42 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/4280414426' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:53:42.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:42 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:53:42.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:42 smithi084 ceph-mon[40024]: purged_snaps scrub starts 2024-09-20T22:53:42.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:42 smithi084 ceph-mon[40024]: purged_snaps scrub ok 2024-09-20T22:53:42.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:42 smithi084 ceph-mon[40024]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:53:42.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:42 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T22:53:42.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:42 smithi084 ceph-mon[40024]: osdmap e742: 8 total, 8 up, 7 in 2024-09-20T22:53:42.622 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:53:42.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:42.645 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:42.645 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:53:42.746 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:53:42.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:42 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/4280414426' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:53:42.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:42 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:53:42.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:42 smithi154 ceph-mon[33264]: purged_snaps scrub starts 2024-09-20T22:53:42.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:42 smithi154 ceph-mon[33264]: purged_snaps scrub ok 2024-09-20T22:53:42.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:42 smithi154 ceph-mon[33264]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:53:42.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:42 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T22:53:42.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:42 smithi154 ceph-mon[33264]: osdmap e742: 8 total, 8 up, 7 in 2024-09-20T22:53:42.767 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:42.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:42.768 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:53:42.869 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:53:42.890 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:42.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:42.891 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:53:42.992 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:53:43.013 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:43.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:43.014 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:53:43.115 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:53:43.136 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:43.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:43.137 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:53:43.237 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:53:43.259 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:43.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:43.260 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:53:43.301 INFO:tasks.rados.rados.0.smithi084.stdout:2945: done (1 left) 2024-09-20T22:53:43.338 INFO:tasks.rados.rados.0.smithi084.stdout:2946: write (excl) oid 3 current snap is 333 2024-09-20T22:53:43.338 INFO:tasks.rados.rados.0.smithi084.stdout:2946: seq_num 1017 ranges {692719=403182,1579246=720684,2828713=264006} 2024-09-20T22:53:43.340 INFO:tasks.rados.rados.0.smithi084.stdout:2946: writing smithi08477203-3 from 692719 to 1095901 tid 1 2024-09-20T22:53:43.340 INFO:tasks.rados.rados.0.smithi084.stdout:2946: writing smithi08477203-3 from 1579246 to 2299930 tid 2 2024-09-20T22:53:43.340 INFO:tasks.rados.rados.0.smithi084.stdout:2946: writing smithi08477203-3 from 2828713 to 3092719 tid 3 2024-09-20T22:53:43.340 INFO:tasks.rados.rados.0.smithi084.stdout:2947: write oid 32 current snap is 333 2024-09-20T22:53:43.340 INFO:tasks.rados.rados.0.smithi084.stdout:2944: finishing rollback tid 0 to smithi08477203-23 2024-09-20T22:53:43.341 INFO:tasks.rados.rados.0.smithi084.stdout:2947: seq_num 1018 ranges {709526=478431,1637728=464342,2856136=253390} 2024-09-20T22:53:43.341 INFO:tasks.rados.rados.0.smithi084.stdout:2947: writing smithi08477203-32 from 709526 to 1187957 tid 1 2024-09-20T22:53:43.341 INFO:tasks.rados.rados.0.smithi084.stdout:2947: writing smithi08477203-32 from 1637728 to 2102070 tid 2 2024-09-20T22:53:43.341 INFO:tasks.rados.rados.0.smithi084.stdout:2947: writing smithi08477203-32 from 2856136 to 3109526 tid 3 2024-09-20T22:53:43.341 INFO:tasks.rados.rados.0.smithi084.stdout:2948: write (excl) oid 43 current snap is 333 2024-09-20T22:53:43.341 INFO:tasks.rados.rados.0.smithi084.stdout:2944: finishing rollback tid 1 to smithi08477203-23 2024-09-20T22:53:43.341 INFO:tasks.rados.rados.0.smithi084.stdout:2948: seq_num 1019 ranges {726333=670033,2096210=724353} 2024-09-20T22:53:43.346 INFO:tasks.rados.rados.0.smithi084.stdout:2948: writing smithi08477203-43 from 726333 to 1396366 tid 1 2024-09-20T22:53:43.354 INFO:tasks.rados.rados.0.smithi084.stdout:2948: writing smithi08477203-43 from 2096210 to 2820563 tid 2 2024-09-20T22:53:43.354 INFO:tasks.rados.rados.0.smithi084.stdout:2949: write (excl) oid 4 current snap is 333 2024-09-20T22:53:43.354 INFO:tasks.rados.rados.0.smithi084.stdout:2944: finishing rollback tid 2 to smithi08477203-23 2024-09-20T22:53:43.354 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 23 v 1904 (ObjNum 957 snap 313 seq_num 957) dirty exists 2024-09-20T22:53:43.354 INFO:tasks.rados.rados.0.smithi084.stdout:2949: seq_num 1020 ranges {743140=745282,2154692=468011} 2024-09-20T22:53:43.361 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:53:43.362 INFO:tasks.rados.rados.0.smithi084.stdout:2949: writing smithi08477203-4 from 743140 to 1488422 tid 1 2024-09-20T22:53:43.369 INFO:tasks.rados.rados.0.smithi084.stdout:2949: writing smithi08477203-4 from 2154692 to 2622703 tid 2 2024-09-20T22:53:43.369 INFO:tasks.rados.rados.0.smithi084.stdout:2944: done (4 left) 2024-09-20T22:53:43.369 INFO:tasks.rados.rados.0.smithi084.stdout:2950: write oid 45 current snap is 333 2024-09-20T22:53:43.369 INFO:tasks.rados.rados.0.smithi084.stdout:2947: finishing write tid 1 to smithi08477203-32 2024-09-20T22:53:43.369 INFO:tasks.rados.rados.0.smithi084.stdout:2947: finishing write tid 2 to smithi08477203-32 2024-09-20T22:53:43.369 INFO:tasks.rados.rados.0.smithi084.stdout:2947: finishing write tid 3 to smithi08477203-32 2024-09-20T22:53:43.369 INFO:tasks.rados.rados.0.smithi084.stdout:2947: finishing write tid 4 to smithi08477203-32 2024-09-20T22:53:43.369 INFO:tasks.rados.rados.0.smithi084.stdout:2946: finishing write tid 1 to smithi08477203-3 2024-09-20T22:53:43.369 INFO:tasks.rados.rados.0.smithi084.stdout:2946: finishing write tid 2 to smithi08477203-3 2024-09-20T22:53:43.369 INFO:tasks.rados.rados.0.smithi084.stdout:2947: finishing write tid 5 to smithi08477203-32 2024-09-20T22:53:43.370 INFO:tasks.rados.rados.0.smithi084.stdout:2947: oid 32 updating version 0 to 1832 2024-09-20T22:53:43.370 INFO:tasks.rados.rados.0.smithi084.stdout:2947: oid 32 updating version 1832 to 1833 2024-09-20T22:53:43.370 INFO:tasks.rados.rados.0.smithi084.stdout:2947: oid 32 version 1833 is already newer than 1831 2024-09-20T22:53:43.370 INFO:tasks.rados.rados.0.smithi084.stdout:2947: oid 32 version 1833 is already newer than 1830 2024-09-20T22:53:43.370 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 32 v 1833 (ObjNum 1018 snap 333 seq_num 1018) dirty exists 2024-09-20T22:53:43.370 INFO:tasks.rados.rados.0.smithi084.stdout:2947: left oid 32 (ObjNum 1018 snap 333 seq_num 1018) 2024-09-20T22:53:43.370 INFO:tasks.rados.rados.0.smithi084.stdout:2946: finishing write tid 3 to smithi08477203-3 2024-09-20T22:53:43.370 INFO:tasks.rados.rados.0.smithi084.stdout:2946: finishing write tid 4 to smithi08477203-3 2024-09-20T22:53:43.370 INFO:tasks.rados.rados.0.smithi084.stdout:2946: finishing write tid 5 to smithi08477203-3 2024-09-20T22:53:43.370 INFO:tasks.rados.rados.0.smithi084.stdout:2946: oid 3 updating version 0 to 1848 2024-09-20T22:53:43.370 INFO:tasks.rados.rados.0.smithi084.stdout:2946: oid 3 version 1848 is already newer than 1846 2024-09-20T22:53:43.370 INFO:tasks.rados.rados.0.smithi084.stdout:2946: oid 3 version 1848 is already newer than 1847 2024-09-20T22:53:43.370 INFO:tasks.rados.rados.0.smithi084.stdout:2946: oid 3 updating version 1848 to 1849 2024-09-20T22:53:43.370 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 3 v 1849 (ObjNum 1017 snap 333 seq_num 1017) dirty exists 2024-09-20T22:53:43.371 INFO:tasks.rados.rados.0.smithi084.stdout:2946: left oid 3 (ObjNum 1017 snap 333 seq_num 1017) 2024-09-20T22:53:43.371 INFO:tasks.rados.rados.0.smithi084.stdout:2948: finishing write tid 1 to smithi08477203-43 2024-09-20T22:53:43.371 INFO:tasks.rados.rados.0.smithi084.stdout:2950: seq_num 1021 ranges {759947=420531,1813174=728022} 2024-09-20T22:53:43.375 INFO:tasks.rados.rados.0.smithi084.stdout:2950: writing smithi08477203-45 from 759947 to 1180478 tid 1 2024-09-20T22:53:43.382 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:43.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:43.383 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:53:43.384 INFO:tasks.rados.rados.0.smithi084.stdout:2950: writing smithi08477203-45 from 1813174 to 2541196 tid 2 2024-09-20T22:53:43.384 INFO:tasks.rados.rados.0.smithi084.stdout:2946: done (4 left) 2024-09-20T22:53:43.384 INFO:tasks.rados.rados.0.smithi084.stdout:2947: done (3 left) 2024-09-20T22:53:43.384 INFO:tasks.rados.rados.0.smithi084.stdout:2951: read oid 12 snap -1 2024-09-20T22:53:43.384 INFO:tasks.rados.rados.0.smithi084.stdout:2951: expect (ObjNum 1001 snap 328 seq_num 1001) 2024-09-20T22:53:43.384 INFO:tasks.rados.rados.0.smithi084.stdout:2952: write (excl) oid 17 current snap is 333 2024-09-20T22:53:43.384 INFO:tasks.rados.rados.0.smithi084.stdout:2948: finishing write tid 2 to smithi08477203-43 2024-09-20T22:53:43.384 INFO:tasks.rados.rados.0.smithi084.stdout:2948: finishing write tid 3 to smithi08477203-43 2024-09-20T22:53:43.384 INFO:tasks.rados.rados.0.smithi084.stdout:2948: finishing write tid 4 to smithi08477203-43 2024-09-20T22:53:43.384 INFO:tasks.rados.rados.0.smithi084.stdout:2948: oid 43 updating version 0 to 1430 2024-09-20T22:53:43.384 INFO:tasks.rados.rados.0.smithi084.stdout:2948: oid 43 version 1430 is already newer than 1429 2024-09-20T22:53:43.385 INFO:tasks.rados.rados.0.smithi084.stdout:2948: oid 43 version 1430 is already newer than 1428 2024-09-20T22:53:43.385 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 43 v 1430 (ObjNum 1019 snap 333 seq_num 1019) dirty exists 2024-09-20T22:53:43.385 INFO:tasks.rados.rados.0.smithi084.stdout:2948: left oid 43 (ObjNum 1019 snap 333 seq_num 1019) 2024-09-20T22:53:43.385 INFO:tasks.rados.rados.0.smithi084.stdout:2949: finishing write tid 1 to smithi08477203-4 2024-09-20T22:53:43.385 INFO:tasks.rados.rados.0.smithi084.stdout:2952: seq_num 1022 ranges {776754=495780,1871656=471680,2965828=210926} 2024-09-20T22:53:43.390 INFO:tasks.rados.rados.0.smithi084.stdout:2952: writing smithi08477203-17 from 776754 to 1272534 tid 1 2024-09-20T22:53:43.396 INFO:tasks.rados.rados.0.smithi084.stdout:2952: writing smithi08477203-17 from 1871656 to 2343336 tid 2 2024-09-20T22:53:43.401 INFO:tasks.rados.rados.0.smithi084.stdout:2952: writing smithi08477203-17 from 2965828 to 3176754 tid 3 2024-09-20T22:53:43.401 INFO:tasks.rados.rados.0.smithi084.stdout:2948: done (4 left) 2024-09-20T22:53:43.401 INFO:tasks.rados.rados.0.smithi084.stdout:2953: delete oid 29 current snap is 333 2024-09-20T22:53:43.402 INFO:tasks.rados.rados.0.smithi084.stdout:2949: finishing write tid 2 to smithi08477203-4 2024-09-20T22:53:43.402 INFO:tasks.rados.rados.0.smithi084.stdout:2949: finishing write tid 3 to smithi08477203-4 2024-09-20T22:53:43.402 INFO:tasks.rados.rados.0.smithi084.stdout:2949: finishing write tid 4 to smithi08477203-4 2024-09-20T22:53:43.402 INFO:tasks.rados.rados.0.smithi084.stdout:2949: oid 4 updating version 0 to 2339 2024-09-20T22:53:43.402 INFO:tasks.rados.rados.0.smithi084.stdout:2949: oid 4 version 2339 is already newer than 2338 2024-09-20T22:53:43.402 INFO:tasks.rados.rados.0.smithi084.stdout:2949: oid 4 updating version 2339 to 2340 2024-09-20T22:53:43.402 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 4 v 2340 (ObjNum 1020 snap 333 seq_num 1020) dirty exists 2024-09-20T22:53:43.402 INFO:tasks.rados.rados.0.smithi084.stdout:2949: left oid 4 (ObjNum 1020 snap 333 seq_num 1020) 2024-09-20T22:53:43.402 INFO:tasks.rados.rados.0.smithi084.stdout:2950: finishing write tid 1 to smithi08477203-45 2024-09-20T22:53:43.402 INFO:tasks.rados.rados.0.smithi084.stdout:2950: finishing write tid 2 to smithi08477203-45 2024-09-20T22:53:43.402 INFO:tasks.rados.rados.0.smithi084.stdout:2950: finishing write tid 3 to smithi08477203-45 2024-09-20T22:53:43.402 INFO:tasks.rados.rados.0.smithi084.stdout:2950: finishing write tid 4 to smithi08477203-45 2024-09-20T22:53:43.402 INFO:tasks.rados.rados.0.smithi084.stdout:2950: oid 45 updating version 0 to 1856 2024-09-20T22:53:43.403 INFO:tasks.rados.rados.0.smithi084.stdout:2950: oid 45 updating version 1856 to 1857 2024-09-20T22:53:43.403 INFO:tasks.rados.rados.0.smithi084.stdout:2950: oid 45 updating version 1857 to 1858 2024-09-20T22:53:43.403 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 45 v 1858 (ObjNum 1021 snap 333 seq_num 1021) dirty exists 2024-09-20T22:53:43.403 INFO:tasks.rados.rados.0.smithi084.stdout:2950: left oid 45 (ObjNum 1021 snap 333 seq_num 1021) 2024-09-20T22:53:43.403 INFO:tasks.rados.rados.0.smithi084.stdout:2951: expect (ObjNum 1001 snap 328 seq_num 1001) 2024-09-20T22:53:43.484 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:53:43.500 INFO:tasks.rados.rados.0.smithi084.stdout:2952: finishing write tid 1 to smithi08477203-17 2024-09-20T22:53:43.500 INFO:tasks.rados.rados.0.smithi084.stdout:2952: finishing write tid 2 to smithi08477203-17 2024-09-20T22:53:43.500 INFO:tasks.rados.rados.0.smithi084.stdout:2949: done (4 left) 2024-09-20T22:53:43.500 INFO:tasks.rados.rados.0.smithi084.stdout:2950: done (3 left) 2024-09-20T22:53:43.500 INFO:tasks.rados.rados.0.smithi084.stdout:2951: done (2 left) 2024-09-20T22:53:43.500 INFO:tasks.rados.rados.0.smithi084.stdout:2953: done (1 left) 2024-09-20T22:53:43.500 INFO:tasks.rados.rados.0.smithi084.stdout:2954: copy_from oid 4 from oid 19 current snap is 333 2024-09-20T22:53:43.501 INFO:tasks.rados.rados.0.smithi084.stdout:2952: finishing write tid 3 to smithi08477203-17 2024-09-20T22:53:43.501 INFO:tasks.rados.rados.0.smithi084.stdout:2952: finishing write tid 4 to smithi08477203-17 2024-09-20T22:53:43.501 INFO:tasks.rados.rados.0.smithi084.stdout:2952: finishing write tid 5 to smithi08477203-17 2024-09-20T22:53:43.501 INFO:tasks.rados.rados.0.smithi084.stdout:2952: oid 17 updating version 0 to 804 2024-09-20T22:53:43.501 INFO:tasks.rados.rados.0.smithi084.stdout:2952: oid 17 updating version 804 to 807 2024-09-20T22:53:43.501 INFO:tasks.rados.rados.0.smithi084.stdout:2952: oid 17 version 807 is already newer than 805 2024-09-20T22:53:43.501 INFO:tasks.rados.rados.0.smithi084.stdout:2952: oid 17 version 807 is already newer than 806 2024-09-20T22:53:43.501 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 17 v 807 (ObjNum 1022 snap 333 seq_num 1022) dirty exists 2024-09-20T22:53:43.501 INFO:tasks.rados.rados.0.smithi084.stdout:2952: left oid 17 (ObjNum 1022 snap 333 seq_num 1022) 2024-09-20T22:53:43.501 INFO:tasks.rados.rados.0.smithi084.stdout:2952: done (1 left) 2024-09-20T22:53:43.501 INFO:tasks.rados.rados.0.smithi084.stdout:2955: write (excl) oid 48 current snap is 333 2024-09-20T22:53:43.501 INFO:tasks.rados.rados.0.smithi084.stdout:2955: seq_num 1024 ranges {410368=646278,1472267=475349,2504321=405600} 2024-09-20T22:53:43.505 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:43.505 INFO:tasks.rados.rados.0.smithi084.stdout:2955: writing smithi08477203-48 from 410368 to 1056646 tid 1 2024-09-20T22:53:43.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:43.506 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:53:43.510 INFO:tasks.rados.rados.0.smithi084.stdout:2955: writing smithi08477203-48 from 1472267 to 1947616 tid 2 2024-09-20T22:53:43.516 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-20T22:53:43.516 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2024-09-20T22:53:43.516 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2024-09-20T22:53:43.516 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-20T22:53:43.516 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:43 smithi084 ceph-mon[29774]: pgmap v1014: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 535 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 13 MiB/s rd, 2.7 MiB/s wr, 20 op/s 2024-09-20T22:53:43.517 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:43 smithi084 ceph-mon[29774]: osdmap e743: 8 total, 8 up, 7 in 2024-09-20T22:53:43.517 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:43 smithi084 ceph-mon[40024]: pgmap v1014: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 535 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 13 MiB/s rd, 2.7 MiB/s wr, 20 op/s 2024-09-20T22:53:43.517 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:43 smithi084 ceph-mon[40024]: osdmap e743: 8 total, 8 up, 7 in 2024-09-20T22:53:43.517 INFO:tasks.rados.rados.0.smithi084.stdout:2955: writing smithi08477203-48 from 2504321 to 2909921 tid 3 2024-09-20T22:53:43.518 INFO:tasks.rados.rados.0.smithi084.stdout:2956: read oid 43 snap 316 2024-09-20T22:53:43.518 INFO:tasks.rados.rados.0.smithi084.stdout:2956: expect (ObjNum 625 snap 186 seq_num 625) 2024-09-20T22:53:43.518 INFO:tasks.rados.rados.0.smithi084.stdout:2957: snap_remove snap 307 2024-09-20T22:53:43.520 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:53:43.607 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:53:43.629 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:43.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:43.630 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:53:43.730 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:53:43.751 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:43.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:43.752 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:53:43.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:43 smithi154 ceph-mon[33264]: pgmap v1014: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 535 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 13 MiB/s rd, 2.7 MiB/s wr, 20 op/s 2024-09-20T22:53:43.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:43 smithi154 ceph-mon[33264]: osdmap e743: 8 total, 8 up, 7 in 2024-09-20T22:53:43.853 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:53:43.874 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:43.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:43.875 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:53:43.976 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:53:43.996 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:43.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:43.997 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:53:44.098 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:53:44.119 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:44.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:44.120 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:53:44.167 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.2 dump_blocked_ops 2024-09-20T22:53:44.184 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:53:44.220 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:53:44.246 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:44.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:44.247 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:53:44.347 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:53:44.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:44.369 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:44.369 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:53:44.420 INFO:tasks.rados.rados.0.smithi084.stdout:2957: done (3 left) 2024-09-20T22:53:44.420 INFO:tasks.rados.rados.0.smithi084.stdout:2958: rollback oid 30 current snap is 333 2024-09-20T22:53:44.420 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 30 to 272 2024-09-20T22:53:44.420 INFO:tasks.rados.rados.0.smithi084.stdout:2959: write (excl) oid 45 current snap is 333 2024-09-20T22:53:44.421 INFO:tasks.rados.rados.0.smithi084.stdout:2959: seq_num 1025 ranges {427175=721527,1930749=735360} 2024-09-20T22:53:44.427 INFO:tasks.rados.rados.0.smithi084.stdout:2959: writing smithi08477203-45 from 427175 to 1148702 tid 1 2024-09-20T22:53:44.436 INFO:tasks.rados.rados.0.smithi084.stdout:2959: writing smithi08477203-45 from 1930749 to 2666109 tid 2 2024-09-20T22:53:44.436 INFO:tasks.rados.rados.0.smithi084.stdout:2960: rollback oid 12 current snap is 333 2024-09-20T22:53:44.436 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 12 to 290 2024-09-20T22:53:44.436 INFO:tasks.rados.rados.0.smithi084.stdout:2955: finishing write tid 1 to smithi08477203-48 2024-09-20T22:53:44.436 INFO:tasks.rados.rados.0.smithi084.stdout:2954: finishing copy_from racing read to smithi08477203-4 2024-09-20T22:53:44.436 INFO:tasks.rados.rados.0.smithi084.stdout:2954: finishing copy_from to smithi08477203-4 2024-09-20T22:53:44.436 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 4 v 2292 (ObjNum 1008 snap 332 seq_num 1008) dirty exists 2024-09-20T22:53:44.437 INFO:tasks.rados.rados.0.smithi084.stdout:2955: finishing write tid 2 to smithi08477203-48 2024-09-20T22:53:44.437 INFO:tasks.rados.rados.0.smithi084.stdout:2955: finishing write tid 3 to smithi08477203-48 2024-09-20T22:53:44.437 INFO:tasks.rados.rados.0.smithi084.stdout:2955: finishing write tid 4 to smithi08477203-48 2024-09-20T22:53:44.437 INFO:tasks.rados.rados.0.smithi084.stdout:2955: finishing write tid 5 to smithi08477203-48 2024-09-20T22:53:44.437 INFO:tasks.rados.rados.0.smithi084.stdout:2955: oid 48 updating version 0 to 1253 2024-09-20T22:53:44.437 INFO:tasks.rados.rados.0.smithi084.stdout:2955: oid 48 version 1253 is already newer than 1252 2024-09-20T22:53:44.437 INFO:tasks.rados.rados.0.smithi084.stdout:2955: oid 48 updating version 1253 to 1255 2024-09-20T22:53:44.437 INFO:tasks.rados.rados.0.smithi084.stdout:2955: oid 48 version 1255 is already newer than 1254 2024-09-20T22:53:44.437 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 48 v 1255 (ObjNum 1024 snap 333 seq_num 1024) dirty exists 2024-09-20T22:53:44.437 INFO:tasks.rados.rados.0.smithi084.stdout:2955: left oid 48 (ObjNum 1024 snap 333 seq_num 1024) 2024-09-20T22:53:44.437 INFO:tasks.rados.rados.0.smithi084.stdout:2956: expect (ObjNum 625 snap 186 seq_num 625) 2024-09-20T22:53:44.470 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:53:44.493 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:44.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:44.494 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:53:44.559 INFO:tasks.rados.rados.0.smithi084.stdout:2958: finishing rollback tid 0 to smithi08477203-30 2024-09-20T22:53:44.559 INFO:tasks.rados.rados.0.smithi084.stdout:2958: finishing rollback tid 1 to smithi08477203-30 2024-09-20T22:53:44.559 INFO:tasks.rados.rados.0.smithi084.stdout:2954: done (5 left) 2024-09-20T22:53:44.559 INFO:tasks.rados.rados.0.smithi084.stdout:2955: done (4 left) 2024-09-20T22:53:44.559 INFO:tasks.rados.rados.0.smithi084.stdout:2956: done (3 left) 2024-09-20T22:53:44.559 INFO:tasks.rados.rados.0.smithi084.stdout:2961: snap_create 2024-09-20T22:53:44.559 INFO:tasks.rados.rados.0.smithi084.stdout:2958: finishing rollback tid 2 to smithi08477203-30 2024-09-20T22:53:44.559 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 30 v 1852 (ObjNum 787 snap 254 seq_num 787) dirty exists 2024-09-20T22:53:44.559 INFO:tasks.rados.rados.0.smithi084.stdout:2960: finishing rollback tid 0 to smithi08477203-12 2024-09-20T22:53:44.559 INFO:tasks.rados.rados.0.smithi084.stdout:2960: finishing rollback tid 1 to smithi08477203-12 2024-09-20T22:53:44.559 INFO:tasks.rados.rados.0.smithi084.stdout:2960: finishing rollback tid 2 to smithi08477203-12 2024-09-20T22:53:44.559 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 12 v 1813 (ObjNum 795 snap 254 seq_num 795) dirty exists 2024-09-20T22:53:44.559 INFO:tasks.rados.rados.0.smithi084.stdout:2959: finishing write tid 1 to smithi08477203-45 2024-09-20T22:53:44.560 INFO:tasks.rados.rados.0.smithi084.stdout:2959: finishing write tid 2 to smithi08477203-45 2024-09-20T22:53:44.560 INFO:tasks.rados.rados.0.smithi084.stdout:2959: finishing write tid 3 to smithi08477203-45 2024-09-20T22:53:44.560 INFO:tasks.rados.rados.0.smithi084.stdout:2959: finishing write tid 4 to smithi08477203-45 2024-09-20T22:53:44.560 INFO:tasks.rados.rados.0.smithi084.stdout:2959: oid 45 updating version 0 to 1861 2024-09-20T22:53:44.560 INFO:tasks.rados.rados.0.smithi084.stdout:2959: oid 45 version 1861 is already newer than 1859 2024-09-20T22:53:44.560 INFO:tasks.rados.rados.0.smithi084.stdout:2959: oid 45 version 1861 is already newer than 1860 2024-09-20T22:53:44.560 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 45 v 1861 (ObjNum 1025 snap 333 seq_num 1025) dirty exists 2024-09-20T22:53:44.560 INFO:tasks.rados.rados.0.smithi084.stdout:2959: left oid 45 (ObjNum 1025 snap 333 seq_num 1025) 2024-09-20T22:53:44.594 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:53:44.616 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:44.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:44.617 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:53:44.717 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:53:44.741 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:44.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:44.742 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:53:44.804 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:53:44.804 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:53:44.820 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:53:44.820 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:53:44.836 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:53:44.836 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:53:44.843 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:53:44.852 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-20T22:53:44.852 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T22:53:44.864 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:44.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:44.865 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:53:44.870 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:53:44.870 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:53:44.887 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:53:44.887 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:53:44.904 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:53:44.904 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:53:44.922 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:53:44.922 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:53:44.966 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:53:44.989 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:44.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:44.990 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:53:45.038 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd unset noscrub 2024-09-20T22:53:45.090 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:53:45.111 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:45.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:45.112 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:53:45.213 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:53:45.234 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:45.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:45.235 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:53:45.336 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:53:45.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:45.358 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:45.358 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:53:45.414 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.0 to 1 (10000) 2024-09-20T22:53:45.415 INFO:tasks.rados.rados.0.smithi084.stdout:2958: done (3 left) 2024-09-20T22:53:45.415 INFO:tasks.rados.rados.0.smithi084.stdout:2959: done (2 left) 2024-09-20T22:53:45.415 INFO:tasks.rados.rados.0.smithi084.stdout:2960: done (1 left) 2024-09-20T22:53:45.415 INFO:tasks.rados.rados.0.smithi084.stdout:2961: done (0 left) 2024-09-20T22:53:45.416 INFO:tasks.rados.rados.0.smithi084.stdout:2962: read oid 3 snap -1 2024-09-20T22:53:45.416 INFO:tasks.rados.rados.0.smithi084.stdout:2962: expect (ObjNum 1017 snap 333 seq_num 1017) 2024-09-20T22:53:45.416 INFO:tasks.rados.rados.0.smithi084.stdout:2963: delete oid 29 current snap is 334 2024-09-20T22:53:45.421 INFO:tasks.rados.rados.0.smithi084.stdout:2962: expect (ObjNum 1017 snap 333 seq_num 1017) 2024-09-20T22:53:45.458 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:53:45.479 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:45.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:45.480 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:53:45.581 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:53:45.601 INFO:tasks.rados.rados.0.smithi084.stdout:2962: done (1 left) 2024-09-20T22:53:45.601 INFO:tasks.rados.rados.0.smithi084.stdout:2963: done (0 left) 2024-09-20T22:53:45.601 INFO:tasks.rados.rados.0.smithi084.stdout:2964: rollback oid 2 current snap is 334 2024-09-20T22:53:45.601 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 2 to 312 2024-09-20T22:53:45.601 INFO:tasks.rados.rados.0.smithi084.stdout:2965: snap_create 2024-09-20T22:53:45.602 INFO:tasks.rados.rados.0.smithi084.stdout:2964: finishing rollback tid 1 to smithi08477203-2 2024-09-20T22:53:45.603 INFO:tasks.rados.rados.0.smithi084.stdout:2964: finishing rollback tid 2 to smithi08477203-2 2024-09-20T22:53:45.603 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 2 v 1835 (ObjNum 918 snap 302 seq_num 918) dirty exists 2024-09-20T22:53:45.607 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:45.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:45.608 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:53:45.709 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:53:45.731 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:45.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:45.732 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:53:45.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:45 smithi154 ceph-mon[33264]: pgmap v1017: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 522 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 8.6 MiB/s rd, 3.2 MiB/s wr, 22 op/s 2024-09-20T22:53:45.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:45 smithi154 ceph-mon[33264]: osdmap e744: 8 total, 8 up, 7 in 2024-09-20T22:53:45.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:45 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/2882828216' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:53:45.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:45 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/1632315235' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-09-20T22:53:45.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:45 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-09-20T22:53:45.804 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:45 smithi084 ceph-mon[29774]: pgmap v1017: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 522 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 8.6 MiB/s rd, 3.2 MiB/s wr, 22 op/s 2024-09-20T22:53:45.805 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:45 smithi084 ceph-mon[29774]: osdmap e744: 8 total, 8 up, 7 in 2024-09-20T22:53:45.805 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:45 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/2882828216' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:53:45.805 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:45 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/1632315235' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-09-20T22:53:45.805 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:45 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-09-20T22:53:45.805 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:45 smithi084 ceph-mon[40024]: pgmap v1017: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 522 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 8.6 MiB/s rd, 3.2 MiB/s wr, 22 op/s 2024-09-20T22:53:45.805 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:45 smithi084 ceph-mon[40024]: osdmap e744: 8 total, 8 up, 7 in 2024-09-20T22:53:45.805 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:45 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/2882828216' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:53:45.805 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:45 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/1632315235' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-09-20T22:53:45.805 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:45 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-09-20T22:53:45.833 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:53:45.858 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:45.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:45.859 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:53:45.959 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:53:45.981 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:45.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:45.982 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:53:46.062 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:53:46.083 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:53:46.108 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:46.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:46.109 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:53:46.210 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:53:46.231 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:46.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:46.232 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:53:46.333 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:53:46.358 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:46.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:46.359 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:53:46.424 INFO:tasks.rados.rados.0.smithi084.stdout:2964: done (1 left) 2024-09-20T22:53:46.425 INFO:tasks.rados.rados.0.smithi084.stdout:2965: done (0 left) 2024-09-20T22:53:46.425 INFO:tasks.rados.rados.0.smithi084.stdout:2966: read oid 30 snap -1 2024-09-20T22:53:46.425 INFO:tasks.rados.rados.0.smithi084.stdout:2966: expect (ObjNum 787 snap 254 seq_num 787) 2024-09-20T22:53:46.425 INFO:tasks.rados.rados.0.smithi084.stdout:2967: snap_create 2024-09-20T22:53:46.430 INFO:tasks.rados.rados.0.smithi084.stdout:2966: expect (ObjNum 787 snap 254 seq_num 787) 2024-09-20T22:53:46.459 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:53:46.479 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:46.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:46.480 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:53:46.581 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:53:46.584 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:46 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-20T22:53:46.584 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:46 smithi084 ceph-mon[29774]: osdmap e745: 8 total, 8 up, 7 in 2024-09-20T22:53:46.584 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:46 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-20T22:53:46.584 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:46 smithi084 ceph-mon[40024]: osdmap e745: 8 total, 8 up, 7 in 2024-09-20T22:53:46.605 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:46.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:46.607 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:53:46.610 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd reweight 4 1 2024-09-20T22:53:46.624 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:53:46.707 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:53:46.730 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:46.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:46.731 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:53:46.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:46 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-20T22:53:46.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:46 smithi154 ceph-mon[33264]: osdmap e745: 8 total, 8 up, 7 in 2024-09-20T22:53:46.832 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:53:46.854 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:46.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:46.855 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:53:46.956 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:53:46.978 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:46.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:46.979 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:53:47.079 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:53:47.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:47.102 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:47.102 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:53:47.203 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:53:47.225 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:47.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:47.226 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:53:47.327 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:53:47.348 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:47.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:47.349 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:53:47.356 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-20T22:53:47.356 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:53:47.426 INFO:tasks.rados.rados.0.smithi084.stdout:2966: done (1 left) 2024-09-20T22:53:47.426 INFO:tasks.rados.rados.0.smithi084.stdout:2967: done (0 left) 2024-09-20T22:53:47.426 INFO:tasks.rados.rados.0.smithi084.stdout:2968: snap_remove snap 334 2024-09-20T22:53:47.450 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:53:47.472 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:47.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:47.473 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:53:47.541 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:47 smithi084 ceph-mon[40024]: pgmap v1020: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 522 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 8.6 MiB/s rd, 3.2 MiB/s wr, 22 op/s 2024-09-20T22:53:47.541 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:47 smithi084 ceph-mon[40024]: osdmap e746: 8 total, 8 up, 7 in 2024-09-20T22:53:47.541 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:47 smithi084 ceph-mon[29774]: pgmap v1020: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 522 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 8.6 MiB/s rd, 3.2 MiB/s wr, 22 op/s 2024-09-20T22:53:47.541 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:47 smithi084 ceph-mon[29774]: osdmap e746: 8 total, 8 up, 7 in 2024-09-20T22:53:47.574 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:53:47.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:47.595 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:47.595 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:53:47.696 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:53:47.718 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:47.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:47.719 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:53:47.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:47 smithi154 ceph-mon[33264]: pgmap v1020: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 522 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 8.6 MiB/s rd, 3.2 MiB/s wr, 22 op/s 2024-09-20T22:53:47.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:47 smithi154 ceph-mon[33264]: osdmap e746: 8 total, 8 up, 7 in 2024-09-20T22:53:47.819 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:53:47.831 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-20T22:53:47.842 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:47.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:47.843 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:53:47.944 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:53:47.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:47.967 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:47.967 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:53:48.067 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:53:48.088 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:48.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:48.089 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:53:48.190 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:53:48.212 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:48.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:48.213 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:53:48.238 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.2 dump_historic_ops 2024-09-20T22:53:48.302 INFO:tasks.rados.rados.0.smithi084.stdout:2968: done (0 left) 2024-09-20T22:53:48.302 INFO:tasks.rados.rados.0.smithi084.stdout:2969: snap_create 2024-09-20T22:53:48.303 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is unset 2024-09-20T22:53:48.307 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:53:48.314 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:53:48.316 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:53:48.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:48.338 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:48.338 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:53:48.438 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:53:48.460 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:48.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:48.461 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:53:48.562 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:53:48.583 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:48.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:48.584 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:53:48.614 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:53:48 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: ::ffff:172.21.15.154 - - [20/Sep/2024:22:53:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-20T22:53:48.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:48 smithi084 ceph-mon[29774]: osdmap e747: 8 total, 8 up, 7 in 2024-09-20T22:53:48.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:48 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/4211534159' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:53:48.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:48 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:53:48.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:48 smithi084 ceph-mon[29774]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:53:48.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:48 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T22:53:48.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:48 smithi084 ceph-mon[29774]: osdmap e748: 8 total, 8 up, 7 in 2024-09-20T22:53:48.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:48 smithi084 ceph-mon[40024]: osdmap e747: 8 total, 8 up, 7 in 2024-09-20T22:53:48.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:48 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/4211534159' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:53:48.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:48 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:53:48.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:48 smithi084 ceph-mon[40024]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:53:48.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:48 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T22:53:48.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:48 smithi084 ceph-mon[40024]: osdmap e748: 8 total, 8 up, 7 in 2024-09-20T22:53:48.684 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:53:48.705 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:48.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:48.706 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:53:48.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:48 smithi154 ceph-mon[33264]: osdmap e747: 8 total, 8 up, 7 in 2024-09-20T22:53:48.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:48 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/4211534159' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:53:48.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:48 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:53:48.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:48 smithi154 ceph-mon[33264]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:53:48.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:48 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T22:53:48.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:48 smithi154 ceph-mon[33264]: osdmap e748: 8 total, 8 up, 7 in 2024-09-20T22:53:48.807 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:53:48.828 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:48.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:48.829 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:53:48.930 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:53:48.949 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:53:48.969 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:48.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:48.970 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:53:49.071 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:53:49.093 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:49.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:49.095 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:53:49.195 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:53:49.217 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:49.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:49.218 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:53:49.311 INFO:tasks.rados.rados.0.smithi084.stdout:2969: done (0 left) 2024-09-20T22:53:49.311 INFO:tasks.rados.rados.0.smithi084.stdout:2970: delete oid 8 current snap is 337 2024-09-20T22:53:49.316 INFO:tasks.rados.rados.0.smithi084.stdout:2970: done (0 left) 2024-09-20T22:53:49.316 INFO:tasks.rados.rados.0.smithi084.stdout:2971: snap_create 2024-09-20T22:53:49.319 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:53:49.340 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:49.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:49.341 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:53:49.442 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:53:49.463 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:49.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:49.464 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:53:49.506 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:49 smithi084 ceph-mon[29774]: pgmap v1023: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 522 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail 2024-09-20T22:53:49.507 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:49 smithi084 ceph-mon[29774]: 4.0 scrub starts 2024-09-20T22:53:49.507 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:49 smithi084 ceph-mon[29774]: 4.0 scrub ok 2024-09-20T22:53:49.507 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:49 smithi084 ceph-mon[29774]: 6.1e scrub starts 2024-09-20T22:53:49.507 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:49 smithi084 ceph-mon[29774]: 6.1e scrub ok 2024-09-20T22:53:49.507 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:49 smithi084 ceph-mon[29774]: osdmap e749: 8 total, 8 up, 7 in 2024-09-20T22:53:49.507 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:49 smithi084 ceph-mon[40024]: pgmap v1023: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 522 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail 2024-09-20T22:53:49.507 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:49 smithi084 ceph-mon[40024]: 4.0 scrub starts 2024-09-20T22:53:49.507 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:49 smithi084 ceph-mon[40024]: 4.0 scrub ok 2024-09-20T22:53:49.507 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:49 smithi084 ceph-mon[40024]: 6.1e scrub starts 2024-09-20T22:53:49.507 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:49 smithi084 ceph-mon[40024]: 6.1e scrub ok 2024-09-20T22:53:49.508 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:49 smithi084 ceph-mon[40024]: osdmap e749: 8 total, 8 up, 7 in 2024-09-20T22:53:49.565 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:53:49.587 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:49.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:49.588 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:53:49.689 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:53:49.710 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:49.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:49.711 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:53:49.754 INFO:journalctl@ceph.iscsi.iscsi.a.smithi154.stdout:Sep 20 22:53:49 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-iscsi-iscsi-a[59730]: debug there is no tcmu-runner data available 2024-09-20T22:53:49.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:49 smithi154 ceph-mon[33264]: pgmap v1023: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 522 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail 2024-09-20T22:53:49.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:49 smithi154 ceph-mon[33264]: 4.0 scrub starts 2024-09-20T22:53:49.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:49 smithi154 ceph-mon[33264]: 4.0 scrub ok 2024-09-20T22:53:49.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:49 smithi154 ceph-mon[33264]: 6.1e scrub starts 2024-09-20T22:53:49.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:49 smithi154 ceph-mon[33264]: 6.1e scrub ok 2024-09-20T22:53:49.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:49 smithi154 ceph-mon[33264]: osdmap e749: 8 total, 8 up, 7 in 2024-09-20T22:53:49.788 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-20T22:53:49.788 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:53:49.812 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:53:49.833 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:49.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:49.834 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:53:49.934 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:53:49.956 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:49.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:49.957 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:53:50.057 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:53:50.080 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:50.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:50.081 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:53:50.181 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:53:50.203 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:50.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:50.204 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:53:50.305 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:53:50.307 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.4 to 1 (10000) 2024-09-20T22:53:50.313 INFO:tasks.rados.rados.0.smithi084.stdout:2971: done (0 left) 2024-09-20T22:53:50.313 INFO:tasks.rados.rados.0.smithi084.stdout:2972: read oid 9 snap -1 2024-09-20T22:53:50.313 INFO:tasks.rados.rados.0.smithi084.stdout:2972: expect deleted 2024-09-20T22:53:50.314 INFO:tasks.rados.rados.0.smithi084.stdout:2973: read oid 27 snap -1 2024-09-20T22:53:50.314 INFO:tasks.rados.rados.0.smithi084.stdout:2973: expect (ObjNum 1016 snap 333 seq_num 1016) 2024-09-20T22:53:50.314 INFO:tasks.rados.rados.0.smithi084.stdout:2974: copy_from oid 7 from oid 20 current snap is 338 2024-09-20T22:53:50.314 INFO:tasks.rados.rados.0.smithi084.stdout:2975: write (excl) oid 4 current snap is 338 2024-09-20T22:53:50.314 INFO:tasks.rados.rados.0.smithi084.stdout:2975: seq_num 1028 ranges {477596=663627,1706195=482687,2614013=567041} 2024-09-20T22:53:50.320 INFO:tasks.rados.rados.0.smithi084.stdout:2975: writing smithi08477203-4 from 477596 to 1141223 tid 1 2024-09-20T22:53:50.326 INFO:tasks.rados.rados.0.smithi084.stdout:2975: writing smithi08477203-4 from 1706195 to 2188882 tid 2 2024-09-20T22:53:50.332 INFO:tasks.rados.rados.0.smithi084.stdout:2975: writing smithi08477203-4 from 2614013 to 3181054 tid 3 2024-09-20T22:53:50.332 INFO:tasks.rados.rados.0.smithi084.stdout:2976: rollback oid 24 current snap is 338 2024-09-20T22:53:50.332 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 24 to 323 2024-09-20T22:53:50.332 INFO:tasks.rados.rados.0.smithi084.stdout:2974: finishing copy_from racing read to smithi08477203-7 2024-09-20T22:53:50.332 INFO:tasks.rados.rados.0.smithi084.stdout:2974: finishing copy_from to smithi08477203-7 2024-09-20T22:53:50.332 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 7 v 1253 (ObjNum 964 snap 314 seq_num 964) dirty exists 2024-09-20T22:53:50.332 INFO:tasks.rados.rados.0.smithi084.stdout:2973: expect (ObjNum 1016 snap 333 seq_num 1016) 2024-09-20T22:53:50.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:50.336 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:50.337 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:53:50.437 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:53:50.459 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:50.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:50.460 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:53:50.503 INFO:tasks.rados.rados.0.smithi084.stdout:2975: finishing write tid 1 to smithi08477203-4 2024-09-20T22:53:50.503 INFO:tasks.rados.rados.0.smithi084.stdout:2975: finishing write tid 2 to smithi08477203-4 2024-09-20T22:53:50.503 INFO:tasks.rados.rados.0.smithi084.stdout:2972: done (4 left) 2024-09-20T22:53:50.503 INFO:tasks.rados.rados.0.smithi084.stdout:2973: done (3 left) 2024-09-20T22:53:50.503 INFO:tasks.rados.rados.0.smithi084.stdout:2974: done (2 left) 2024-09-20T22:53:50.503 INFO:tasks.rados.rados.0.smithi084.stdout:2977: read oid 36 snap -1 2024-09-20T22:53:50.504 INFO:tasks.rados.rados.0.smithi084.stdout:2977: expect (ObjNum 945 snap 311 seq_num 945) 2024-09-20T22:53:50.504 INFO:tasks.rados.rados.0.smithi084.stdout:2978: write oid 37 current snap is 338 2024-09-20T22:53:50.504 INFO:tasks.rados.rados.0.smithi084.stdout:2976: finishing rollback tid 0 to smithi08477203-24 2024-09-20T22:53:50.504 INFO:tasks.rados.rados.0.smithi084.stdout:2976: finishing rollback tid 1 to smithi08477203-24 2024-09-20T22:53:50.504 INFO:tasks.rados.rados.0.smithi084.stdout:2976: finishing rollback tid 2 to smithi08477203-24 2024-09-20T22:53:50.504 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 24 v 1864 (ObjNum 780 snap 252 seq_num 780) dirty exists 2024-09-20T22:53:50.504 INFO:tasks.rados.rados.0.smithi084.stdout:2975: finishing write tid 3 to smithi08477203-4 2024-09-20T22:53:50.504 INFO:tasks.rados.rados.0.smithi084.stdout:2975: finishing write tid 4 to smithi08477203-4 2024-09-20T22:53:50.504 INFO:tasks.rados.rados.0.smithi084.stdout:2975: finishing write tid 5 to smithi08477203-4 2024-09-20T22:53:50.504 INFO:tasks.rados.rados.0.smithi084.stdout:2975: oid 4 updating version 0 to 2356 2024-09-20T22:53:50.504 INFO:tasks.rados.rados.0.smithi084.stdout:2975: oid 4 version 2356 is already newer than 2354 2024-09-20T22:53:50.504 INFO:tasks.rados.rados.0.smithi084.stdout:2975: oid 4 version 2356 is already newer than 2355 2024-09-20T22:53:50.504 INFO:tasks.rados.rados.0.smithi084.stdout:2975: oid 4 updating version 2356 to 2357 2024-09-20T22:53:50.504 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 4 v 2357 (ObjNum 1028 snap 338 seq_num 1028) dirty exists 2024-09-20T22:53:50.504 INFO:tasks.rados.rados.0.smithi084.stdout:2975: left oid 4 (ObjNum 1028 snap 338 seq_num 1028) 2024-09-20T22:53:50.505 INFO:tasks.rados.rados.0.smithi084.stdout:2978: seq_num 1029 ranges {494403=738876,1764677=626345,3041436=252967} 2024-09-20T22:53:50.509 INFO:tasks.rados.rados.0.smithi084.stdout:2978: writing smithi08477203-37 from 494403 to 1233279 tid 1 2024-09-20T22:53:50.516 INFO:tasks.rados.rados.0.smithi084.stdout:2978: writing smithi08477203-37 from 1764677 to 2391022 tid 2 2024-09-20T22:53:50.521 INFO:tasks.rados.rados.0.smithi084.stdout:2978: writing smithi08477203-37 from 3041436 to 3294403 tid 3 2024-09-20T22:53:50.521 INFO:tasks.rados.rados.0.smithi084.stdout:2975: done (3 left) 2024-09-20T22:53:50.521 INFO:tasks.rados.rados.0.smithi084.stdout:2976: done (2 left) 2024-09-20T22:53:50.522 INFO:tasks.rados.rados.0.smithi084.stdout:2979: delete oid 26 current snap is 338 2024-09-20T22:53:50.522 INFO:tasks.rados.rados.0.smithi084.stdout:2977: expect (ObjNum 945 snap 311 seq_num 945) 2024-09-20T22:53:50.543 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-20T22:53:50.561 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:53:50.582 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:50.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:50.583 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:53:50.652 INFO:tasks.rados.rados.0.smithi084.stdout:2978: finishing write tid 1 to smithi08477203-37 2024-09-20T22:53:50.652 INFO:tasks.rados.rados.0.smithi084.stdout:2978: finishing write tid 2 to smithi08477203-37 2024-09-20T22:53:50.652 INFO:tasks.rados.rados.0.smithi084.stdout:2977: done (2 left) 2024-09-20T22:53:50.652 INFO:tasks.rados.rados.0.smithi084.stdout:2979: done (1 left) 2024-09-20T22:53:50.653 INFO:tasks.rados.rados.0.smithi084.stdout:2980: snap_remove snap 298 2024-09-20T22:53:50.684 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:53:50.705 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:50.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:50.706 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:53:50.754 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:50 smithi084 ceph-mon[40024]: 6.4 scrub starts 2024-09-20T22:53:50.754 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:50 smithi084 ceph-mon[40024]: 6.4 scrub ok 2024-09-20T22:53:50.754 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:50 smithi084 ceph-mon[40024]: 4.13 scrub starts 2024-09-20T22:53:50.754 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:50 smithi084 ceph-mon[40024]: 4.13 scrub ok 2024-09-20T22:53:50.754 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:50 smithi084 ceph-mon[40024]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:53:50.754 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:50 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/815166640' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-09-20T22:53:50.754 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:50 smithi084 ceph-mon[40024]: 7.4 scrub starts 2024-09-20T22:53:50.754 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:50 smithi084 ceph-mon[40024]: 7.4 scrub ok 2024-09-20T22:53:50.754 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:50 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/815166640' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-20T22:53:50.755 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:50 smithi084 ceph-mon[40024]: osdmap e750: 8 total, 8 up, 7 in 2024-09-20T22:53:50.755 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:50 smithi084 ceph-mon[29774]: 6.4 scrub starts 2024-09-20T22:53:50.755 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:50 smithi084 ceph-mon[29774]: 6.4 scrub ok 2024-09-20T22:53:50.755 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:50 smithi084 ceph-mon[29774]: 4.13 scrub starts 2024-09-20T22:53:50.755 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:50 smithi084 ceph-mon[29774]: 4.13 scrub ok 2024-09-20T22:53:50.755 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:50 smithi084 ceph-mon[29774]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:53:50.755 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:50 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/815166640' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-09-20T22:53:50.755 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:50 smithi084 ceph-mon[29774]: 7.4 scrub starts 2024-09-20T22:53:50.755 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:50 smithi084 ceph-mon[29774]: 7.4 scrub ok 2024-09-20T22:53:50.755 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:50 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/815166640' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-20T22:53:50.755 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:50 smithi084 ceph-mon[29774]: osdmap e750: 8 total, 8 up, 7 in 2024-09-20T22:53:50.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:50 smithi154 ceph-mon[33264]: 6.4 scrub starts 2024-09-20T22:53:50.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:50 smithi154 ceph-mon[33264]: 6.4 scrub ok 2024-09-20T22:53:50.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:50 smithi154 ceph-mon[33264]: 4.13 scrub starts 2024-09-20T22:53:50.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:50 smithi154 ceph-mon[33264]: 4.13 scrub ok 2024-09-20T22:53:50.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:50 smithi154 ceph-mon[33264]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:53:50.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:50 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/815166640' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-09-20T22:53:50.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:50 smithi154 ceph-mon[33264]: 7.4 scrub starts 2024-09-20T22:53:50.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:50 smithi154 ceph-mon[33264]: 7.4 scrub ok 2024-09-20T22:53:50.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:50 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/815166640' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-20T22:53:50.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:50 smithi154 ceph-mon[33264]: osdmap e750: 8 total, 8 up, 7 in 2024-09-20T22:53:50.808 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:53:50.830 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:50.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:50.831 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:53:50.907 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-20T22:53:50.932 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:53:50.954 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:50.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:50.955 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:53:51.055 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:53:51.077 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:51.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:51.078 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:53:51.179 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:53:51.200 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:51.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:51.201 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:53:51.302 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:53:51.324 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:51.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:51.325 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:53:51.425 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:53:51.448 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:51.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:51.450 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:53:51.456 INFO:tasks.rados.rados.0.smithi084.stdout:2980: done (1 left) 2024-09-20T22:53:51.456 INFO:tasks.rados.rados.0.smithi084.stdout:2981: read oid 8 snap 293 2024-09-20T22:53:51.456 INFO:tasks.rados.rados.0.smithi084.stdout:2981: expect (ObjNum 863 snap 276 seq_num 863) 2024-09-20T22:53:51.456 INFO:tasks.rados.rados.0.smithi084.stdout:2982: rollback oid 28 current snap is 338 2024-09-20T22:53:51.456 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 28 to 336 2024-09-20T22:53:51.456 INFO:tasks.rados.rados.0.smithi084.stdout:2978: finishing write tid 3 to smithi08477203-37 2024-09-20T22:53:51.456 INFO:tasks.rados.rados.0.smithi084.stdout:2978: finishing write tid 4 to smithi08477203-37 2024-09-20T22:53:51.456 INFO:tasks.rados.rados.0.smithi084.stdout:2978: finishing write tid 5 to smithi08477203-37 2024-09-20T22:53:51.456 INFO:tasks.rados.rados.0.smithi084.stdout:2978: oid 37 updating version 0 to 1259 2024-09-20T22:53:51.456 INFO:tasks.rados.rados.0.smithi084.stdout:2978: oid 37 version 1259 is already newer than 1257 2024-09-20T22:53:51.456 INFO:tasks.rados.rados.0.smithi084.stdout:2978: oid 37 version 1259 is already newer than 1256 2024-09-20T22:53:51.456 INFO:tasks.rados.rados.0.smithi084.stdout:2978: oid 37 version 1259 is already newer than 1258 2024-09-20T22:53:51.456 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 37 v 1259 (ObjNum 1029 snap 338 seq_num 1029) dirty exists 2024-09-20T22:53:51.457 INFO:tasks.rados.rados.0.smithi084.stdout:2978: left oid 37 (ObjNum 1029 snap 338 seq_num 1029) 2024-09-20T22:53:51.457 INFO:tasks.rados.rados.0.smithi084.stdout:2978: done (2 left) 2024-09-20T22:53:51.457 INFO:tasks.rados.rados.0.smithi084.stdout:2983: rollback oid 26 current snap is 338 2024-09-20T22:53:51.457 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 26 to 300 2024-09-20T22:53:51.457 INFO:tasks.rados.rados.0.smithi084.stdout:2984: write oid 38 current snap is 338 2024-09-20T22:53:51.457 INFO:tasks.rados.rados.0.smithi084.stdout:2984: seq_num 1031 ranges {528017=489374,1481641=630014,2696282=400857} 2024-09-20T22:53:51.462 INFO:tasks.rados.rados.0.smithi084.stdout:2984: writing smithi08477203-38 from 528017 to 1017391 tid 1 2024-09-20T22:53:51.468 INFO:tasks.rados.rados.0.smithi084.stdout:2984: writing smithi08477203-38 from 1481641 to 2111655 tid 2 2024-09-20T22:53:51.474 INFO:tasks.rados.rados.0.smithi084.stdout:2984: writing smithi08477203-38 from 2696282 to 3097139 tid 3 2024-09-20T22:53:51.474 INFO:tasks.rados.rados.0.smithi084.stdout:2985: copy_from oid 15 from oid 5 current snap is 338 2024-09-20T22:53:51.474 INFO:tasks.rados.rados.0.smithi084.stdout:2982: finishing rollback tid 1 to smithi08477203-28 2024-09-20T22:53:51.474 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 28 v 2189 (ObjNum 49 snap 0 seq_num 2091720904) dirty dne 2024-09-20T22:53:51.474 INFO:tasks.rados.rados.0.smithi084.stdout:2981: expect (ObjNum 863 snap 276 seq_num 863) 2024-09-20T22:53:51.550 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:53:51.572 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:51.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:51.573 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:53:51.614 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-09-20T22:53:51.614 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph pg dump --format=json 2024-09-20T22:53:51.649 INFO:tasks.rados.rados.0.smithi084.stdout:2983: finishing rollback tid 1 to smithi08477203-26 2024-09-20T22:53:51.649 INFO:tasks.rados.rados.0.smithi084.stdout:2983: finishing rollback tid 2 to smithi08477203-26 2024-09-20T22:53:51.649 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 26 v 1815 (ObjNum 906 snap 297 seq_num 906) dirty exists 2024-09-20T22:53:51.649 INFO:tasks.rados.rados.0.smithi084.stdout:2984: finishing write tid 1 to smithi08477203-38 2024-09-20T22:53:51.649 INFO:tasks.rados.rados.0.smithi084.stdout:2984: finishing write tid 2 to smithi08477203-38 2024-09-20T22:53:51.650 INFO:tasks.rados.rados.0.smithi084.stdout:2981: done (4 left) 2024-09-20T22:53:51.650 INFO:tasks.rados.rados.0.smithi084.stdout:2982: done (3 left) 2024-09-20T22:53:51.650 INFO:tasks.rados.rados.0.smithi084.stdout:2983: done (2 left) 2024-09-20T22:53:51.650 INFO:tasks.rados.rados.0.smithi084.stdout:2986: read oid 23 snap 302 2024-09-20T22:53:51.650 INFO:tasks.rados.rados.0.smithi084.stdout:2986: expect deleted 2024-09-20T22:53:51.650 INFO:tasks.rados.rados.0.smithi084.stdout:2987: delete oid 47 current snap is 338 2024-09-20T22:53:51.650 INFO:tasks.rados.rados.0.smithi084.stdout:2984: finishing write tid 3 to smithi08477203-38 2024-09-20T22:53:51.650 INFO:tasks.rados.rados.0.smithi084.stdout:2984: finishing write tid 4 to smithi08477203-38 2024-09-20T22:53:51.650 INFO:tasks.rados.rados.0.smithi084.stdout:2984: finishing write tid 5 to smithi08477203-38 2024-09-20T22:53:51.650 INFO:tasks.rados.rados.0.smithi084.stdout:2984: oid 38 updating version 0 to 1816 2024-09-20T22:53:51.650 INFO:tasks.rados.rados.0.smithi084.stdout:2984: oid 38 updating version 1816 to 1819 2024-09-20T22:53:51.650 INFO:tasks.rados.rados.0.smithi084.stdout:2984: oid 38 version 1819 is already newer than 1818 2024-09-20T22:53:51.650 INFO:tasks.rados.rados.0.smithi084.stdout:2984: oid 38 version 1819 is already newer than 1817 2024-09-20T22:53:51.650 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 38 v 1819 (ObjNum 1031 snap 338 seq_num 1031) dirty exists 2024-09-20T22:53:51.651 INFO:tasks.rados.rados.0.smithi084.stdout:2984: left oid 38 (ObjNum 1031 snap 338 seq_num 1031) 2024-09-20T22:53:51.651 INFO:tasks.rados.rados.0.smithi084.stdout:2985: finishing copy_from racing read to smithi08477203-15 2024-09-20T22:53:51.651 INFO:tasks.rados.rados.0.smithi084.stdout:2985: finishing copy_from to smithi08477203-15 2024-09-20T22:53:51.651 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 15 v 904 (ObjNum 1012 snap 333 seq_num 1012) dirty exists 2024-09-20T22:53:51.651 INFO:tasks.rados.rados.0.smithi084.stdout:2984: done (3 left) 2024-09-20T22:53:51.651 INFO:tasks.rados.rados.0.smithi084.stdout:2985: done (2 left) 2024-09-20T22:53:51.651 INFO:tasks.rados.rados.0.smithi084.stdout:2986: done (1 left) 2024-09-20T22:53:51.651 INFO:tasks.rados.rados.0.smithi084.stdout:2987: done (0 left) 2024-09-20T22:53:51.651 INFO:tasks.rados.rados.0.smithi084.stdout:2988: delete oid 48 current snap is 338 2024-09-20T22:53:51.653 INFO:tasks.rados.rados.0.smithi084.stdout:2988: done (0 left) 2024-09-20T22:53:51.653 INFO:tasks.rados.rados.0.smithi084.stdout:2989: rollback oid 28 current snap is 338 2024-09-20T22:53:51.653 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 28 to 336 2024-09-20T22:53:51.653 INFO:tasks.rados.rados.0.smithi084.stdout:2990: delete oid 25 current snap is 338 2024-09-20T22:53:51.656 INFO:tasks.rados.rados.0.smithi084.stdout:2989: finishing rollback tid 1 to smithi08477203-28 2024-09-20T22:53:51.657 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 28 v 2189 (ObjNum 49 snap 0 seq_num 2091023424) dirty dne 2024-09-20T22:53:51.657 INFO:tasks.rados.rados.0.smithi084.stdout:2989: done (1 left) 2024-09-20T22:53:51.657 INFO:tasks.rados.rados.0.smithi084.stdout:2990: done (0 left) 2024-09-20T22:53:51.657 INFO:tasks.rados.rados.0.smithi084.stdout:2991: copy_from oid 46 from oid 31 current snap is 338 2024-09-20T22:53:51.657 INFO:tasks.rados.rados.0.smithi084.stdout:2992: write (excl) oid 6 current snap is 338 2024-09-20T22:53:51.657 INFO:tasks.rados.rados.0.smithi084.stdout:2992: seq_num 1035 ranges {595245=506723,1715569=637352,2805974=562298} 2024-09-20T22:53:51.662 INFO:tasks.rados.rados.0.smithi084.stdout:2992: writing smithi08477203-6 from 595245 to 1101968 tid 1 2024-09-20T22:53:51.670 INFO:tasks.rados.rados.0.smithi084.stdout:2992: writing smithi08477203-6 from 1715569 to 2352921 tid 2 2024-09-20T22:53:51.673 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:53:51.676 INFO:tasks.rados.rados.0.smithi084.stdout:2992: writing smithi08477203-6 from 2805974 to 3368272 tid 3 2024-09-20T22:53:51.676 INFO:tasks.rados.rados.0.smithi084.stdout:2993: read oid 39 snap -1 2024-09-20T22:53:51.676 INFO:tasks.rados.rados.0.smithi084.stdout:2993: expect deleted 2024-09-20T22:53:51.676 INFO:tasks.rados.rados.0.smithi084.stdout:2994: snap_remove snap 314 2024-09-20T22:53:51.694 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:51.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:51.695 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:53:51.738 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:51 smithi084 ceph-mon[40024]: pgmap v1026: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 522 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 6.5 MiB/s rd, 356 KiB/s wr, 8 op/s 2024-09-20T22:53:51.738 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:51 smithi084 ceph-mon[40024]: 1.9 scrub starts 2024-09-20T22:53:51.738 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:51 smithi084 ceph-mon[40024]: 1.9 scrub ok 2024-09-20T22:53:51.738 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:51 smithi084 ceph-mon[40024]: 4.16 scrub starts 2024-09-20T22:53:51.738 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:51 smithi084 ceph-mon[40024]: 4.16 scrub ok 2024-09-20T22:53:51.738 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:51 smithi084 ceph-mon[40024]: 4.a scrub starts 2024-09-20T22:53:51.738 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:51 smithi084 ceph-mon[40024]: 4.a scrub ok 2024-09-20T22:53:51.738 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:51 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:53:51.738 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:51 smithi084 ceph-mon[29774]: pgmap v1026: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 522 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 6.5 MiB/s rd, 356 KiB/s wr, 8 op/s 2024-09-20T22:53:51.739 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:51 smithi084 ceph-mon[29774]: 1.9 scrub starts 2024-09-20T22:53:51.739 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:51 smithi084 ceph-mon[29774]: 1.9 scrub ok 2024-09-20T22:53:51.739 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:51 smithi084 ceph-mon[29774]: 4.16 scrub starts 2024-09-20T22:53:51.739 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:51 smithi084 ceph-mon[29774]: 4.16 scrub ok 2024-09-20T22:53:51.739 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:51 smithi084 ceph-mon[29774]: 4.a scrub starts 2024-09-20T22:53:51.739 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:51 smithi084 ceph-mon[29774]: 4.a scrub ok 2024-09-20T22:53:51.739 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:51 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:53:51.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:51 smithi154 ceph-mon[33264]: pgmap v1026: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 522 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 6.5 MiB/s rd, 356 KiB/s wr, 8 op/s 2024-09-20T22:53:51.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:51 smithi154 ceph-mon[33264]: 1.9 scrub starts 2024-09-20T22:53:51.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:51 smithi154 ceph-mon[33264]: 1.9 scrub ok 2024-09-20T22:53:51.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:51 smithi154 ceph-mon[33264]: 4.16 scrub starts 2024-09-20T22:53:51.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:51 smithi154 ceph-mon[33264]: 4.16 scrub ok 2024-09-20T22:53:51.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:51 smithi154 ceph-mon[33264]: 4.a scrub starts 2024-09-20T22:53:51.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:51 smithi154 ceph-mon[33264]: 4.a scrub ok 2024-09-20T22:53:51.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:51 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:53:51.795 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:53:51.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:51.816 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:51.817 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:53:51.854 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:53:51.854 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:53:51.864 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:53:51.864 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:53:51.891 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:53:51.891 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:53:51.902 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:53:51.911 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-20T22:53:51.911 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T22:53:51.917 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:53:51.933 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:53:51.933 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:53:51.934 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:53:51.939 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:51.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:51.940 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:53:51.954 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:53:51.954 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:53:51.970 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:53:51.971 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:53:51.987 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:53:51.987 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:53:52.041 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:53:52.062 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:52.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:52.063 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:53:52.164 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:53:52.185 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:52.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:52.186 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:53:52.286 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:53:52.308 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:52.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:52.309 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:53:52.410 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:53:52.432 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:52.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:52.432 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:53:52.457 INFO:tasks.rados.rados.0.smithi084.stdout:2994: done (3 left) 2024-09-20T22:53:52.457 INFO:tasks.rados.rados.0.smithi084.stdout:2995: write (excl) oid 24 current snap is 338 2024-09-20T22:53:52.457 INFO:tasks.rados.rados.0.smithi084.stdout:2991: finishing copy_from racing read to smithi08477203-46 2024-09-20T22:53:52.457 INFO:tasks.rados.rados.0.smithi084.stdout:2995: seq_num 1036 ranges {612052=581972,1657698=497363,2717044=695008} 2024-09-20T22:53:52.464 INFO:tasks.rados.rados.0.smithi084.stdout:2995: writing smithi08477203-24 from 612052 to 1194024 tid 1 2024-09-20T22:53:52.470 INFO:tasks.rados.rados.0.smithi084.stdout:2995: writing smithi08477203-24 from 1657698 to 2155061 tid 2 2024-09-20T22:53:52.478 INFO:tasks.rados.rados.0.smithi084.stdout:2995: writing smithi08477203-24 from 2717044 to 3412052 tid 3 2024-09-20T22:53:52.478 INFO:tasks.rados.rados.0.smithi084.stdout:2996: delete oid 47 current snap is 338 2024-09-20T22:53:52.478 INFO:tasks.rados.rados.0.smithi084.stdout:2992: finishing write tid 1 to smithi08477203-6 2024-09-20T22:53:52.478 INFO:tasks.rados.rados.0.smithi084.stdout:2992: finishing write tid 2 to smithi08477203-6 2024-09-20T22:53:52.478 INFO:tasks.rados.rados.0.smithi084.stdout:2991: finishing copy_from to smithi08477203-46 2024-09-20T22:53:52.478 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 46 v 1358 (ObjNum 910 snap 297 seq_num 910) dirty exists 2024-09-20T22:53:52.478 INFO:tasks.rados.rados.0.smithi084.stdout:2992: finishing write tid 3 to smithi08477203-6 2024-09-20T22:53:52.478 INFO:tasks.rados.rados.0.smithi084.stdout:2992: finishing write tid 4 to smithi08477203-6 2024-09-20T22:53:52.478 INFO:tasks.rados.rados.0.smithi084.stdout:2992: finishing write tid 5 to smithi08477203-6 2024-09-20T22:53:52.479 INFO:tasks.rados.rados.0.smithi084.stdout:2992: oid 6 updating version 0 to 809 2024-09-20T22:53:52.479 INFO:tasks.rados.rados.0.smithi084.stdout:2992: oid 6 version 809 is already newer than 808 2024-09-20T22:53:52.479 INFO:tasks.rados.rados.0.smithi084.stdout:2992: oid 6 updating version 809 to 810 2024-09-20T22:53:52.479 INFO:tasks.rados.rados.0.smithi084.stdout:2992: oid 6 updating version 810 to 811 2024-09-20T22:53:52.479 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 6 v 811 (ObjNum 1035 snap 338 seq_num 1035) dirty exists 2024-09-20T22:53:52.479 INFO:tasks.rados.rados.0.smithi084.stdout:2992: left oid 6 (ObjNum 1035 snap 338 seq_num 1035) 2024-09-20T22:53:52.479 INFO:tasks.rados.rados.0.smithi084.stdout:2995: finishing write tid 1 to smithi08477203-24 2024-09-20T22:53:52.479 INFO:tasks.rados.rados.0.smithi084.stdout:2995: finishing write tid 2 to smithi08477203-24 2024-09-20T22:53:52.479 INFO:tasks.rados.rados.0.smithi084.stdout:2991: done (4 left) 2024-09-20T22:53:52.480 INFO:tasks.rados.rados.0.smithi084.stdout:2992: done (3 left) 2024-09-20T22:53:52.480 INFO:tasks.rados.rados.0.smithi084.stdout:2993: done (2 left) 2024-09-20T22:53:52.480 INFO:tasks.rados.rados.0.smithi084.stdout:2996: done (1 left) 2024-09-20T22:53:52.480 INFO:tasks.rados.rados.0.smithi084.stdout:2997: snap_create 2024-09-20T22:53:52.481 INFO:tasks.rados.rados.0.smithi084.stdout:2995: finishing write tid 3 to smithi08477203-24 2024-09-20T22:53:52.481 INFO:tasks.rados.rados.0.smithi084.stdout:2995: finishing write tid 4 to smithi08477203-24 2024-09-20T22:53:52.482 INFO:tasks.rados.rados.0.smithi084.stdout:2995: finishing write tid 5 to smithi08477203-24 2024-09-20T22:53:52.482 INFO:tasks.rados.rados.0.smithi084.stdout:2995: oid 24 updating version 0 to 1866 2024-09-20T22:53:52.482 INFO:tasks.rados.rados.0.smithi084.stdout:2995: oid 24 version 1866 is already newer than 1865 2024-09-20T22:53:52.482 INFO:tasks.rados.rados.0.smithi084.stdout:2995: oid 24 updating version 1866 to 1868 2024-09-20T22:53:52.482 INFO:tasks.rados.rados.0.smithi084.stdout:2995: oid 24 version 1868 is already newer than 1867 2024-09-20T22:53:52.482 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 24 v 1868 (ObjNum 1036 snap 338 seq_num 1036) dirty exists 2024-09-20T22:53:52.482 INFO:tasks.rados.rados.0.smithi084.stdout:2995: left oid 24 (ObjNum 1036 snap 338 seq_num 1036) 2024-09-20T22:53:52.533 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:53:52.554 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:52.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:52.555 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:53:52.568 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:52 smithi084 ceph-mon[29774]: osdmap e751: 8 total, 8 up, 7 in 2024-09-20T22:53:52.568 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:52 smithi084 ceph-mon[29774]: 4.1a scrub starts 2024-09-20T22:53:52.568 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:52 smithi084 ceph-mon[29774]: 4.1a scrub ok 2024-09-20T22:53:52.568 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:52 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:53:52.568 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:52 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:53:52.568 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:52 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:53:52.568 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:52 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/3983994917' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:53:52.568 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:52 smithi084 ceph-mon[29774]: 5.5 scrub starts 2024-09-20T22:53:52.568 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:52 smithi084 ceph-mon[29774]: 5.5 scrub ok 2024-09-20T22:53:52.568 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:52 smithi084 ceph-mon[29774]: 5.11 scrub starts 2024-09-20T22:53:52.568 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:52 smithi084 ceph-mon[29774]: 5.11 scrub ok 2024-09-20T22:53:52.569 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:52 smithi084 ceph-mon[40024]: osdmap e751: 8 total, 8 up, 7 in 2024-09-20T22:53:52.569 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:52 smithi084 ceph-mon[40024]: 4.1a scrub starts 2024-09-20T22:53:52.569 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:52 smithi084 ceph-mon[40024]: 4.1a scrub ok 2024-09-20T22:53:52.569 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:52 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:53:52.569 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:52 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:53:52.569 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:52 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:53:52.569 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:52 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/3983994917' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:53:52.569 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:52 smithi084 ceph-mon[40024]: 5.5 scrub starts 2024-09-20T22:53:52.569 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:52 smithi084 ceph-mon[40024]: 5.5 scrub ok 2024-09-20T22:53:52.569 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:52 smithi084 ceph-mon[40024]: 5.11 scrub starts 2024-09-20T22:53:52.569 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:52 smithi084 ceph-mon[40024]: 5.11 scrub ok 2024-09-20T22:53:52.586 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-20T22:53:52.586 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2024-09-20T22:53:52.586 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2024-09-20T22:53:52.586 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-20T22:53:52.656 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:53:52.678 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:52.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:52.679 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:53:52.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:52 smithi154 ceph-mon[33264]: osdmap e751: 8 total, 8 up, 7 in 2024-09-20T22:53:52.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:52 smithi154 ceph-mon[33264]: 4.1a scrub starts 2024-09-20T22:53:52.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:52 smithi154 ceph-mon[33264]: 4.1a scrub ok 2024-09-20T22:53:52.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:52 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:53:52.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:52 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:53:52.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:52 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:53:52.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:52 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/3983994917' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:53:52.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:52 smithi154 ceph-mon[33264]: 5.5 scrub starts 2024-09-20T22:53:52.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:52 smithi154 ceph-mon[33264]: 5.5 scrub ok 2024-09-20T22:53:52.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:52 smithi154 ceph-mon[33264]: 5.11 scrub starts 2024-09-20T22:53:52.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:52 smithi154 ceph-mon[33264]: 5.11 scrub ok 2024-09-20T22:53:52.779 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:53:52.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:52.800 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:52.800 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:53:52.901 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:53:52.923 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:52.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:52.924 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:53:53.025 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:53:53.045 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:53.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:53.046 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:53:53.147 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:53:53.168 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:53.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:53.169 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:53:53.184 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-20T22:53:53.186 INFO:teuthology.orchestra.run.smithi084.stderr:dumped all 2024-09-20T22:53:53.270 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:53:53.291 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:53.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:53.292 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:53:53.305 INFO:tasks.rados.rados.0.smithi084.stdout:2995: done (1 left) 2024-09-20T22:53:53.306 INFO:tasks.rados.rados.0.smithi084.stdout:2997: done (0 left) 2024-09-20T22:53:53.306 INFO:tasks.rados.rados.0.smithi084.stdout:2998: read oid 46 snap -1 2024-09-20T22:53:53.306 INFO:tasks.rados.rados.0.smithi084.stdout:2998: expect (ObjNum 910 snap 297 seq_num 910) 2024-09-20T22:53:53.306 INFO:tasks.rados.rados.0.smithi084.stdout:2999: delete oid 3 current snap is 339 2024-09-20T22:53:53.316 INFO:tasks.rados.rados.0.smithi084.stdout:2998: expect (ObjNum 910 snap 297 seq_num 910) 2024-09-20T22:53:53.392 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:53:53.414 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:53.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:53.415 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:53:53.493 INFO:tasks.rados.rados.0.smithi084.stdout:2998: done (1 left) 2024-09-20T22:53:53.493 INFO:tasks.rados.rados.0.smithi084.stdout:2999: done (0 left) 2024-09-20T22:53:53.494 INFO:tasks.rados.rados.0.smithi084.stdout:3000: copy_from oid 41 from oid 8 current snap is 339 2024-09-20T22:53:53.494 INFO:tasks.rados.rados.0.smithi084.stdout:3001: read oid 5 snap -1 2024-09-20T22:53:53.494 INFO:tasks.rados.rados.0.smithi084.stdout:3001: expect (ObjNum 1012 snap 333 seq_num 1012) 2024-09-20T22:53:53.494 INFO:tasks.rados.rados.0.smithi084.stdout:3002: read oid 46 snap 335 2024-09-20T22:53:53.494 INFO:tasks.rados.rados.0.smithi084.stdout:3002: expect deleted 2024-09-20T22:53:53.494 INFO:tasks.rados.rados.0.smithi084.stdout:3003: rollback oid 26 current snap is 339 2024-09-20T22:53:53.494 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 26 to 257 2024-09-20T22:53:53.494 INFO:tasks.rados.rados.0.smithi084.stdout:3004: delete oid 39 current snap is 339 2024-09-20T22:53:53.495 INFO:tasks.rados.rados.0.smithi084.stdout:3000: finishing copy_from to smithi08477203-41 2024-09-20T22:53:53.495 INFO:tasks.rados.rados.0.smithi084.stdout:3000: got expected ENOENT (src dne) 2024-09-20T22:53:53.495 INFO:tasks.rados.rados.0.smithi084.stdout:3000: finishing copy_from racing read to smithi08477203-41 2024-09-20T22:53:53.495 INFO:tasks.rados.rados.0.smithi084.stdout:3000: done (4 left) 2024-09-20T22:53:53.496 INFO:tasks.rados.rados.0.smithi084.stdout:3002: done (3 left) 2024-09-20T22:53:53.496 INFO:tasks.rados.rados.0.smithi084.stdout:3004: done (2 left) 2024-09-20T22:53:53.496 INFO:tasks.rados.rados.0.smithi084.stdout:3005: snap_remove snap 294 2024-09-20T22:53:53.515 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:53:53.536 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:53.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:53.537 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:53:53.638 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:53:53.658 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:53.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:53.659 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:53:53.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:53 smithi154 ceph-mon[33264]: pgmap v1029: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 522 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 6.5 MiB/s rd, 356 KiB/s wr, 8 op/s 2024-09-20T22:53:53.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:53 smithi154 ceph-mon[33264]: 6.19 scrub starts 2024-09-20T22:53:53.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:53 smithi154 ceph-mon[33264]: 6.19 scrub ok 2024-09-20T22:53:53.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:53 smithi154 ceph-mon[33264]: osdmap e752: 8 total, 8 up, 7 in 2024-09-20T22:53:53.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:53 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:53:53.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:53 smithi154 ceph-mon[33264]: osdmap e753: 8 total, 8 up, 7 in 2024-09-20T22:53:53.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:53 smithi154 ceph-mon[33264]: 6.8 scrub starts 2024-09-20T22:53:53.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:53 smithi154 ceph-mon[33264]: 6.8 scrub ok 2024-09-20T22:53:53.760 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:53:53.782 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:53.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:53.783 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:53:53.864 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:53 smithi084 ceph-mon[29774]: pgmap v1029: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 522 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 6.5 MiB/s rd, 356 KiB/s wr, 8 op/s 2024-09-20T22:53:53.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:53 smithi084 ceph-mon[29774]: 6.19 scrub starts 2024-09-20T22:53:53.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:53 smithi084 ceph-mon[29774]: 6.19 scrub ok 2024-09-20T22:53:53.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:53 smithi084 ceph-mon[29774]: osdmap e752: 8 total, 8 up, 7 in 2024-09-20T22:53:53.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:53 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:53:53.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:53 smithi084 ceph-mon[29774]: osdmap e753: 8 total, 8 up, 7 in 2024-09-20T22:53:53.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:53 smithi084 ceph-mon[29774]: 6.8 scrub starts 2024-09-20T22:53:53.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:53 smithi084 ceph-mon[29774]: 6.8 scrub ok 2024-09-20T22:53:53.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:53 smithi084 ceph-mon[40024]: pgmap v1029: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 522 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 6.5 MiB/s rd, 356 KiB/s wr, 8 op/s 2024-09-20T22:53:53.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:53 smithi084 ceph-mon[40024]: 6.19 scrub starts 2024-09-20T22:53:53.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:53 smithi084 ceph-mon[40024]: 6.19 scrub ok 2024-09-20T22:53:53.866 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:53 smithi084 ceph-mon[40024]: osdmap e752: 8 total, 8 up, 7 in 2024-09-20T22:53:53.866 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:53 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:53:53.866 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:53 smithi084 ceph-mon[40024]: osdmap e753: 8 total, 8 up, 7 in 2024-09-20T22:53:53.866 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:53 smithi084 ceph-mon[40024]: 6.8 scrub starts 2024-09-20T22:53:53.866 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:53 smithi084 ceph-mon[40024]: 6.8 scrub ok 2024-09-20T22:53:53.884 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:53:53.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:53.906 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:53.906 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:53:54.007 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:53:54.028 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:54.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:54.029 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:53:54.129 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:53:54.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:54.152 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:54.152 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:53:54.253 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:53:54.274 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:54.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:54.275 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:53:54.376 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:53:54.386 INFO:teuthology.orchestra.run.smithi084.stdout:{"pg_ready":true,"pg_map":{"version":1030,"stamp":"2024-09-20T22:53:52.452255+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":547684579,"num_objects":444,"num_object_clones":196,"num_object_copies":1332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":444,"num_whiteouts":13,"num_read":6992,"num_read_kb":5358646,"num_write":4448,"num_write_kb":1383696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1142,"num_bytes_recovered":520180548,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":12376,"ondisk_log_size":12376,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":474,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":1545020,"kb_used_data":1175364,"kb_used_omap":10,"kb_used_meta":369461,"kb_avail":654671044,"statfs":{"total":671965249536,"available":670383149056,"internally_reserved":0,"allocated":1203572736,"data_stored":1705499215,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11157,"internal_metadata":378328171},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":807719,"num_objects":-1,"num_object_clones":-1,"num_object_copies":-3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1,"num_whiteouts":-1,"num_read":24,"num_read_kb":26748,"num_write":9,"num_write_kb":1424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.154811"},"pg_stats":[{"pgid":"3.1f","version":"86'2","reported_seq":751,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-09-20T22:53:48.301972+0000","last_change":"2024-09-20T22:52:21.630419+0000","last_active":"2024-09-20T22:53:48.301972+0000","last_peered":"2024-09-20T22:53:48.301972+0000","last_clean":"2024-09-20T22:53:48.301972+0000","last_became_active":"2024-09-20T22:47:49.222114+0000","last_became_peered":"2024-09-20T22:47:49.222114+0000","last_unstale":"2024-09-20T22:53:48.301972+0000","last_undegraded":"2024-09-20T22:53:48.301972+0000","last_fullsized":"2024-09-20T22:53:48.301972+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'2","last_scrub_stamp":"2024-09-20T22:52:21.630304+0000","last_deep_scrub":"86'2","last_deep_scrub_stamp":"2024-09-20T22:42:39.664421+0000","last_clean_scrub_stamp":"2024-09-20T22:52:21.630304+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":436,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"625'15","reported_seq":749,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-09-20T22:53:48.299094+0000","last_change":"2024-09-20T22:51:51.813358+0000","last_active":"2024-09-20T22:53:48.299094+0000","last_peered":"2024-09-20T22:53:48.299094+0000","last_clean":"2024-09-20T22:53:48.299094+0000","last_became_active":"2024-09-20T22:51:20.286651+0000","last_became_peered":"2024-09-20T22:51:20.286651+0000","last_unstale":"2024-09-20T22:53:48.299094+0000","last_undegraded":"2024-09-20T22:53:48.299094+0000","last_fullsized":"2024-09-20T22:53:48.299094+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'15","last_scrub_stamp":"2024-09-20T22:51:51.813269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:51:51.813269+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:49.514461+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":317,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-09-20T22:53:48.394712+0000","last_change":"2024-09-20T22:52:41.184388+0000","last_active":"2024-09-20T22:53:48.394712+0000","last_peered":"2024-09-20T22:53:48.394712+0000","last_clean":"2024-09-20T22:53:48.394712+0000","last_became_active":"2024-09-20T22:51:20.291272+0000","last_became_peered":"2024-09-20T22:51:20.291272+0000","last_unstale":"2024-09-20T22:53:48.394712+0000","last_undegraded":"2024-09-20T22:53:48.394712+0000","last_fullsized":"2024-09-20T22:53:48.394712+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:41.184318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:52:41.184318+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:04.632649+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":720,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-09-20T22:53:48.299244+0000","last_change":"2024-09-20T22:51:36.730087+0000","last_active":"2024-09-20T22:53:48.299244+0000","last_peered":"2024-09-20T22:53:48.299244+0000","last_clean":"2024-09-20T22:53:48.299244+0000","last_became_active":"2024-09-20T22:51:20.295608+0000","last_became_peered":"2024-09-20T22:51:20.295608+0000","last_unstale":"2024-09-20T22:53:48.299244+0000","last_undegraded":"2024-09-20T22:53:48.299244+0000","last_fullsized":"2024-09-20T22:53:48.299244+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:36.730015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:51:36.730015+0000","last_clean_scrub_stamp":"2024-09-20T22:51:36.730015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:49.514413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":1394,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-09-20T22:53:48.335241+0000","last_change":"2024-09-20T22:51:01.404127+0000","last_active":"2024-09-20T22:53:48.335241+0000","last_peered":"2024-09-20T22:53:48.335241+0000","last_clean":"2024-09-20T22:53:48.335241+0000","last_became_active":"2024-09-20T22:41:44.832099+0000","last_became_peered":"2024-09-20T22:41:44.832099+0000","last_unstale":"2024-09-20T22:53:48.335241+0000","last_undegraded":"2024-09-20T22:53:48.335241+0000","last_fullsized":"2024-09-20T22:53:48.335241+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:01.404063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:51:01.404063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":1400,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-09-20T22:53:48.336883+0000","last_change":"2024-09-20T22:51:12.418041+0000","last_active":"2024-09-20T22:53:48.336883+0000","last_peered":"2024-09-20T22:53:48.336883+0000","last_clean":"2024-09-20T22:53:48.336883+0000","last_became_active":"2024-09-20T22:41:38.157298+0000","last_became_peered":"2024-09-20T22:41:38.157298+0000","last_unstale":"2024-09-20T22:53:48.336883+0000","last_undegraded":"2024-09-20T22:53:48.336883+0000","last_fullsized":"2024-09-20T22:53:48.336883+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:12.417920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:51:12.417920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"625'27","reported_seq":1482,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-09-20T22:53:48.335601+0000","last_change":"2024-09-20T22:51:08.441927+0000","last_active":"2024-09-20T22:53:48.335601+0000","last_peered":"2024-09-20T22:53:48.335601+0000","last_clean":"2024-09-20T22:53:48.335601+0000","last_became_active":"2024-09-20T22:47:49.218798+0000","last_became_peered":"2024-09-20T22:47:49.218798+0000","last_unstale":"2024-09-20T22:53:48.335601+0000","last_undegraded":"2024-09-20T22:53:48.335601+0000","last_fullsized":"2024-09-20T22:53:48.335601+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-09-20T22:51:08.441788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:51:08.441788+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":720,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-09-20T22:53:48.298956+0000","last_change":"2024-09-20T22:52:14.594895+0000","last_active":"2024-09-20T22:53:48.298956+0000","last_peered":"2024-09-20T22:53:48.298956+0000","last_clean":"2024-09-20T22:53:48.298956+0000","last_became_active":"2024-09-20T22:51:20.287060+0000","last_became_peered":"2024-09-20T22:51:20.287060+0000","last_unstale":"2024-09-20T22:53:48.298956+0000","last_undegraded":"2024-09-20T22:53:48.298956+0000","last_fullsized":"2024-09-20T22:53:48.298956+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:14.594823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:52:14.594823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:49.514536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":1419,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-09-20T22:53:48.395253+0000","last_change":"2024-09-20T22:51:09.684125+0000","last_active":"2024-09-20T22:53:48.395253+0000","last_peered":"2024-09-20T22:53:48.395253+0000","last_clean":"2024-09-20T22:53:48.395253+0000","last_became_active":"2024-09-20T22:48:05.242059+0000","last_became_peered":"2024-09-20T22:48:05.242059+0000","last_unstale":"2024-09-20T22:53:48.395253+0000","last_undegraded":"2024-09-20T22:53:48.395253+0000","last_fullsized":"2024-09-20T22:53:48.395253+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:09.684053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:46:00.999939+0000","last_clean_scrub_stamp":"2024-09-20T22:51:09.684053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:48.982671+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"625'15","reported_seq":733,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-09-20T22:53:48.299345+0000","last_change":"2024-09-20T22:51:30.710329+0000","last_active":"2024-09-20T22:53:48.299345+0000","last_peered":"2024-09-20T22:53:48.299345+0000","last_clean":"2024-09-20T22:53:48.299345+0000","last_became_active":"2024-09-20T22:48:05.233429+0000","last_became_peered":"2024-09-20T22:48:05.233429+0000","last_unstale":"2024-09-20T22:53:48.299345+0000","last_undegraded":"2024-09-20T22:53:48.299345+0000","last_fullsized":"2024-09-20T22:53:48.299345+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-09-20T22:51:30.710245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:51:30.710245+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:49.514383+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":1411,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-09-20T22:53:48.395746+0000","last_change":"2024-09-20T22:51:33.977328+0000","last_active":"2024-09-20T22:53:48.395746+0000","last_peered":"2024-09-20T22:53:48.395746+0000","last_clean":"2024-09-20T22:53:48.395746+0000","last_became_active":"2024-09-20T22:47:49.227379+0000","last_became_peered":"2024-09-20T22:47:49.227379+0000","last_unstale":"2024-09-20T22:53:48.395746+0000","last_undegraded":"2024-09-20T22:53:48.395746+0000","last_fullsized":"2024-09-20T22:53:48.395746+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:33.977257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:51:33.977257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:48.982706+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":761,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-09-20T22:53:48.298806+0000","last_change":"2024-09-20T22:53:35.921037+0000","last_active":"2024-09-20T22:53:48.298806+0000","last_peered":"2024-09-20T22:53:48.298806+0000","last_clean":"2024-09-20T22:53:48.298806+0000","last_became_active":"2024-09-20T22:51:20.277465+0000","last_became_peered":"2024-09-20T22:51:20.277465+0000","last_unstale":"2024-09-20T22:53:48.298806+0000","last_undegraded":"2024-09-20T22:53:48.298806+0000","last_fullsized":"2024-09-20T22:53:48.298806+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:35.920968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:49:08.664407+0000","last_clean_scrub_stamp":"2024-09-20T22:53:35.920968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:03.583550+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":1433,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-09-20T22:53:48.394919+0000","last_change":"2024-09-20T22:53:31.092991+0000","last_active":"2024-09-20T22:53:48.394919+0000","last_peered":"2024-09-20T22:53:48.394919+0000","last_clean":"2024-09-20T22:53:48.394919+0000","last_became_active":"2024-09-20T22:51:20.297787+0000","last_became_peered":"2024-09-20T22:51:20.297787+0000","last_unstale":"2024-09-20T22:53:48.394919+0000","last_undegraded":"2024-09-20T22:53:48.394919+0000","last_fullsized":"2024-09-20T22:53:48.394919+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:31.092915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:53:31.092915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:41.403638+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"625'9","reported_seq":743,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-09-20T22:53:48.299731+0000","last_change":"2024-09-20T22:53:09.623541+0000","last_active":"2024-09-20T22:53:48.299731+0000","last_peered":"2024-09-20T22:53:48.299731+0000","last_clean":"2024-09-20T22:53:48.299731+0000","last_became_active":"2024-09-20T22:51:20.285795+0000","last_became_peered":"2024-09-20T22:51:20.285795+0000","last_unstale":"2024-09-20T22:53:48.299731+0000","last_undegraded":"2024-09-20T22:53:48.299731+0000","last_fullsized":"2024-09-20T22:53:48.299731+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'9","last_scrub_stamp":"2024-09-20T22:53:09.623458+0000","last_deep_scrub":"181'7","last_deep_scrub_stamp":"2024-09-20T22:48:41.049381+0000","last_clean_scrub_stamp":"2024-09-20T22:53:09.623458+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:34.232945+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":9,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":1438,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-09-20T22:53:48.344016+0000","last_change":"2024-09-20T22:52:16.705733+0000","last_active":"2024-09-20T22:53:48.344016+0000","last_peered":"2024-09-20T22:53:48.344016+0000","last_clean":"2024-09-20T22:53:48.344016+0000","last_became_active":"2024-09-20T22:51:20.285368+0000","last_became_peered":"2024-09-20T22:51:20.285368+0000","last_unstale":"2024-09-20T22:53:48.344016+0000","last_undegraded":"2024-09-20T22:53:48.344016+0000","last_fullsized":"2024-09-20T22:53:48.344016+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:16.705645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:47:40.361958+0000","last_clean_scrub_stamp":"2024-09-20T22:52:16.705645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":1404,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-09-20T22:53:48.394624+0000","last_change":"2024-09-20T22:51:20.279625+0000","last_active":"2024-09-20T22:53:48.394624+0000","last_peered":"2024-09-20T22:53:48.394624+0000","last_clean":"2024-09-20T22:53:48.394624+0000","last_became_active":"2024-09-20T22:51:20.279307+0000","last_became_peered":"2024-09-20T22:51:20.279307+0000","last_unstale":"2024-09-20T22:53:48.394624+0000","last_undegraded":"2024-09-20T22:53:48.394624+0000","last_fullsized":"2024-09-20T22:53:48.394624+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:04.636900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:51:04.636900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:48.982601+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":323,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-09-20T22:53:48.558669+0000","last_change":"2024-09-20T22:53:48.558689+0000","last_active":"2024-09-20T22:53:48.558669+0000","last_peered":"2024-09-20T22:53:48.558669+0000","last_clean":"2024-09-20T22:53:48.558669+0000","last_became_active":"2024-09-20T22:51:20.282393+0000","last_became_peered":"2024-09-20T22:51:20.282393+0000","last_unstale":"2024-09-20T22:53:48.558669+0000","last_undegraded":"2024-09-20T22:53:48.558669+0000","last_fullsized":"2024-09-20T22:53:48.558669+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:48.558619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:53:48.558619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:02.645042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":753,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-09-20T22:53:48.300806+0000","last_change":"2024-09-20T22:52:42.745105+0000","last_active":"2024-09-20T22:53:48.300806+0000","last_peered":"2024-09-20T22:53:48.300806+0000","last_clean":"2024-09-20T22:53:48.300806+0000","last_became_active":"2024-09-20T22:48:05.233687+0000","last_became_peered":"2024-09-20T22:48:05.233687+0000","last_unstale":"2024-09-20T22:53:48.300806+0000","last_undegraded":"2024-09-20T22:53:48.300806+0000","last_fullsized":"2024-09-20T22:53:48.300806+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:42.745032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:49:53.933601+0000","last_clean_scrub_stamp":"2024-09-20T22:52:42.745032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:59.532706+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"625'34","reported_seq":1498,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-09-20T22:53:48.394911+0000","last_change":"2024-09-20T22:53:09.119883+0000","last_active":"2024-09-20T22:53:48.394911+0000","last_peered":"2024-09-20T22:53:48.394911+0000","last_clean":"2024-09-20T22:53:48.394911+0000","last_became_active":"2024-09-20T22:51:20.276338+0000","last_became_peered":"2024-09-20T22:51:20.276338+0000","last_unstale":"2024-09-20T22:53:48.394911+0000","last_undegraded":"2024-09-20T22:53:48.394911+0000","last_fullsized":"2024-09-20T22:53:48.394911+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'34","last_scrub_stamp":"2024-09-20T22:53:09.119791+0000","last_deep_scrub":"113'22","last_deep_scrub_stamp":"2024-09-20T22:43:12.191878+0000","last_clean_scrub_stamp":"2024-09-20T22:53:09.119791+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:36.423122+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":1026,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-09-20T22:53:48.299340+0000","last_change":"2024-09-20T22:52:43.758236+0000","last_active":"2024-09-20T22:53:48.299340+0000","last_peered":"2024-09-20T22:53:48.299340+0000","last_clean":"2024-09-20T22:53:48.299340+0000","last_became_active":"2024-09-20T22:51:20.295564+0000","last_became_peered":"2024-09-20T22:51:20.295564+0000","last_unstale":"2024-09-20T22:53:48.299340+0000","last_undegraded":"2024-09-20T22:53:48.299340+0000","last_fullsized":"2024-09-20T22:53:48.299340+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:43.758162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:50:40.784154+0000","last_clean_scrub_stamp":"2024-09-20T22:52:43.758162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:10.526451+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":1387,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-09-20T22:53:48.336550+0000","last_change":"2024-09-20T22:51:13.385618+0000","last_active":"2024-09-20T22:53:48.336550+0000","last_peered":"2024-09-20T22:53:48.336550+0000","last_clean":"2024-09-20T22:53:48.336550+0000","last_became_active":"2024-09-20T22:41:44.832131+0000","last_became_peered":"2024-09-20T22:41:44.832131+0000","last_unstale":"2024-09-20T22:53:48.336550+0000","last_undegraded":"2024-09-20T22:53:48.336550+0000","last_fullsized":"2024-09-20T22:53:48.336550+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:13.385520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:45:25.002199+0000","last_clean_scrub_stamp":"2024-09-20T22:51:13.385520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":728,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-09-20T22:53:48.299481+0000","last_change":"2024-09-20T22:53:04.583753+0000","last_active":"2024-09-20T22:53:48.299481+0000","last_peered":"2024-09-20T22:53:48.299481+0000","last_clean":"2024-09-20T22:53:48.299481+0000","last_became_active":"2024-09-20T22:51:20.284621+0000","last_became_peered":"2024-09-20T22:51:20.284621+0000","last_unstale":"2024-09-20T22:53:48.299481+0000","last_undegraded":"2024-09-20T22:53:48.299481+0000","last_fullsized":"2024-09-20T22:53:48.299481+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:04.583683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:50:41.885844+0000","last_clean_scrub_stamp":"2024-09-20T22:53:04.583683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:27.888402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"625'18","reported_seq":1498,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-09-20T22:53:48.336433+0000","last_change":"2024-09-20T22:51:20.300828+0000","last_active":"2024-09-20T22:53:48.336433+0000","last_peered":"2024-09-20T22:53:48.336433+0000","last_clean":"2024-09-20T22:53:48.336433+0000","last_became_active":"2024-09-20T22:51:20.278886+0000","last_became_peered":"2024-09-20T22:51:20.278886+0000","last_unstale":"2024-09-20T22:53:48.336433+0000","last_undegraded":"2024-09-20T22:53:48.336433+0000","last_fullsized":"2024-09-20T22:53:48.336433+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-09-20T22:51:03.380934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:51:03.380934+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":722,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-09-20T22:53:48.299474+0000","last_change":"2024-09-20T22:52:45.738376+0000","last_active":"2024-09-20T22:53:48.299474+0000","last_peered":"2024-09-20T22:53:48.299474+0000","last_clean":"2024-09-20T22:53:48.299474+0000","last_became_active":"2024-09-20T22:48:05.250319+0000","last_became_peered":"2024-09-20T22:48:05.250319+0000","last_unstale":"2024-09-20T22:53:48.299474+0000","last_undegraded":"2024-09-20T22:53:48.299474+0000","last_fullsized":"2024-09-20T22:53:48.299474+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:45.738275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:52:45.738275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:02.329689+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"154'2","reported_seq":1403,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-09-20T22:53:48.344187+0000","last_change":"2024-09-20T22:52:45.578519+0000","last_active":"2024-09-20T22:53:48.344187+0000","last_peered":"2024-09-20T22:53:48.344187+0000","last_clean":"2024-09-20T22:53:48.344187+0000","last_became_active":"2024-09-20T22:41:44.199822+0000","last_became_peered":"2024-09-20T22:41:44.199822+0000","last_unstale":"2024-09-20T22:53:48.344187+0000","last_undegraded":"2024-09-20T22:53:48.344187+0000","last_fullsized":"2024-09-20T22:53:48.344187+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"154'2","last_scrub_stamp":"2024-09-20T22:52:45.578416+0000","last_deep_scrub":"154'2","last_deep_scrub_stamp":"2024-09-20T22:43:54.134066+0000","last_clean_scrub_stamp":"2024-09-20T22:52:45.578416+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:53.480152+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"67'1","reported_seq":328,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-09-20T22:53:48.335454+0000","last_change":"2024-09-20T22:53:08.612473+0000","last_active":"2024-09-20T22:53:48.335454+0000","last_peered":"2024-09-20T22:53:48.335454+0000","last_clean":"2024-09-20T22:53:48.335454+0000","last_became_active":"2024-09-20T22:51:20.279978+0000","last_became_peered":"2024-09-20T22:51:20.279978+0000","last_unstale":"2024-09-20T22:53:48.335454+0000","last_undegraded":"2024-09-20T22:53:48.335454+0000","last_fullsized":"2024-09-20T22:53:48.335454+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-09-20T22:53:08.612395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:53:08.612395+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:13.547026+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"625'25","reported_seq":783,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-09-20T22:53:48.301287+0000","last_change":"2024-09-20T22:53:26.146547+0000","last_active":"2024-09-20T22:53:48.301287+0000","last_peered":"2024-09-20T22:53:48.301287+0000","last_clean":"2024-09-20T22:53:48.301287+0000","last_became_active":"2024-09-20T22:47:49.232181+0000","last_became_peered":"2024-09-20T22:47:49.232181+0000","last_unstale":"2024-09-20T22:53:48.301287+0000","last_undegraded":"2024-09-20T22:53:48.301287+0000","last_fullsized":"2024-09-20T22:53:48.301287+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'25","last_scrub_stamp":"2024-09-20T22:53:26.146466+0000","last_deep_scrub":"151'15","last_deep_scrub_stamp":"2024-09-20T22:43:50.715761+0000","last_clean_scrub_stamp":"2024-09-20T22:53:26.146466+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:51.749507+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"661'11","reported_seq":1605,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-09-20T22:53:48.331202+0000","last_change":"2024-09-20T22:53:34.944681+0000","last_active":"2024-09-20T22:53:48.331202+0000","last_peered":"2024-09-20T22:53:48.331202+0000","last_clean":"2024-09-20T22:53:48.331202+0000","last_became_active":"2024-09-20T22:48:05.244270+0000","last_became_peered":"2024-09-20T22:48:05.244270+0000","last_unstale":"2024-09-20T22:53:48.331202+0000","last_undegraded":"2024-09-20T22:53:48.331202+0000","last_fullsized":"2024-09-20T22:53:48.331202+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"661'11","last_scrub_stamp":"2024-09-20T22:53:34.944591+0000","last_deep_scrub":"661'11","last_deep_scrub_stamp":"2024-09-20T22:52:21.907943+0000","last_clean_scrub_stamp":"2024-09-20T22:53:34.944591+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:38.596007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.c","version":"745'2352","reported_seq":5511,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-09-20T22:53:48.331076+0000","last_change":"2024-09-20T22:53:48.331369+0000","last_active":"2024-09-20T22:53:48.331076+0000","last_peered":"2024-09-20T22:53:48.331076+0000","last_clean":"2024-09-20T22:53:48.331076+0000","last_became_active":"2024-09-20T22:42:52.404027+0000","last_became_peered":"2024-09-20T22:42:52.404027+0000","last_unstale":"2024-09-20T22:53:48.331076+0000","last_undegraded":"2024-09-20T22:53:48.331076+0000","last_fullsized":"2024-09-20T22:53:48.331076+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"674'2030","last_scrub_stamp":"2024-09-20T22:52:36.892383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:52:36.892383+0000","objects_scrubbed":44,"log_size":2352,"log_dups_size":0,"ondisk_log_size":2352,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.00073071099999999999,"stat_sum":{"num_bytes":62218699,"num_objects":31,"num_object_clones":24,"num_object_copies":93,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":4,"num_read":911,"num_read_kb":974437,"num_write":596,"num_write_kb":231050,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"ed","length":"1"},{"start":"10a","length":"8"},{"start":"113","length":"c"},{"start":"120","length":"1"},{"start":"122","length":"2"},{"start":"125","length":"2"},{"start":"129","length":"1"},{"start":"12b","length":"1"},{"start":"12d","length":"1"},{"start":"131","length":"1"},{"start":"133","length":"6"},{"start":"13f","length":"2"},{"start":"143","length":"2"},{"start":"146","length":"2"},{"start":"14b","length":"1"},{"start":"14d","length":"1"},{"start":"150","length":"1"}]},{"pgid":"1.a","version":"77'128","reported_seq":1676,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-09-20T22:53:48.394334+0000","last_change":"2024-09-20T22:53:28.095343+0000","last_active":"2024-09-20T22:53:48.394334+0000","last_peered":"2024-09-20T22:53:48.394334+0000","last_clean":"2024-09-20T22:53:48.394334+0000","last_became_active":"2024-09-20T22:42:52.407791+0000","last_became_peered":"2024-09-20T22:42:52.407791+0000","last_unstale":"2024-09-20T22:53:48.394334+0000","last_undegraded":"2024-09-20T22:53:48.394334+0000","last_fullsized":"2024-09-20T22:53:48.394334+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:53:28.095266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:53:28.095266+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:45.776903+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"625'27","reported_seq":394,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-09-20T22:53:48.337095+0000","last_change":"2024-09-20T22:52:15.358218+0000","last_active":"2024-09-20T22:53:48.337095+0000","last_peered":"2024-09-20T22:53:48.337095+0000","last_clean":"2024-09-20T22:53:48.337095+0000","last_became_active":"2024-09-20T22:51:20.287083+0000","last_became_peered":"2024-09-20T22:51:20.287083+0000","last_unstale":"2024-09-20T22:53:48.337095+0000","last_undegraded":"2024-09-20T22:53:48.337095+0000","last_fullsized":"2024-09-20T22:53:48.337095+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'27","last_scrub_stamp":"2024-09-20T22:52:15.358100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:52:15.358100+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":1405,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-09-20T22:53:48.335227+0000","last_change":"2024-09-20T22:51:29.394206+0000","last_active":"2024-09-20T22:53:48.335227+0000","last_peered":"2024-09-20T22:53:48.335227+0000","last_clean":"2024-09-20T22:53:48.335227+0000","last_became_active":"2024-09-20T22:51:20.279439+0000","last_became_peered":"2024-09-20T22:51:20.279439+0000","last_unstale":"2024-09-20T22:53:48.335227+0000","last_undegraded":"2024-09-20T22:53:48.335227+0000","last_fullsized":"2024-09-20T22:53:48.335227+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:29.394097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:49:12.070675+0000","last_clean_scrub_stamp":"2024-09-20T22:51:29.394097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"408'12","reported_seq":787,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-09-20T22:53:48.300175+0000","last_change":"2024-09-20T22:51:50.785575+0000","last_active":"2024-09-20T22:53:48.300175+0000","last_peered":"2024-09-20T22:53:48.300175+0000","last_clean":"2024-09-20T22:53:48.300175+0000","last_became_active":"2024-09-20T22:48:05.242914+0000","last_became_peered":"2024-09-20T22:48:05.242914+0000","last_unstale":"2024-09-20T22:53:48.300175+0000","last_undegraded":"2024-09-20T22:53:48.300175+0000","last_fullsized":"2024-09-20T22:53:48.300175+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"408'12","last_scrub_stamp":"2024-09-20T22:51:50.785478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:51:50.785478+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:49.514444+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":1414,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-09-20T22:53:48.394876+0000","last_change":"2024-09-20T22:51:31.960415+0000","last_active":"2024-09-20T22:53:48.394876+0000","last_peered":"2024-09-20T22:53:48.394876+0000","last_clean":"2024-09-20T22:53:48.394876+0000","last_became_active":"2024-09-20T22:51:20.280145+0000","last_became_peered":"2024-09-20T22:51:20.280145+0000","last_unstale":"2024-09-20T22:53:48.394876+0000","last_undegraded":"2024-09-20T22:53:48.394876+0000","last_fullsized":"2024-09-20T22:53:48.394876+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:31.960331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:51:31.960331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:48.982723+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3","version":"745'345","reported_seq":3451,"reported_epoch":748,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-20T22:53:48.412674+0000","last_change":"2024-09-20T22:53:48.412735+0000","last_active":"2024-09-20T22:53:48.412674+0000","last_peered":"2024-09-20T22:53:48.412674+0000","last_clean":"2024-09-20T22:53:48.412674+0000","last_became_active":"2024-09-20T22:49:05.221402+0000","last_became_peered":"2024-09-20T22:49:05.221402+0000","last_unstale":"2024-09-20T22:53:48.412674+0000","last_undegraded":"2024-09-20T22:53:48.412674+0000","last_fullsized":"2024-09-20T22:53:48.412674+0000","mapping_epoch":404,"log_start":"456'129","ondisk_log_start":"456'129","created":94,"last_epoch_clean":465,"parent":"0.0","parent_split_bits":0,"last_scrub":"609'243","last_scrub_stamp":"2024-09-20T22:51:31.402013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:51:31.402013+0000","objects_scrubbed":6,"log_size":216,"log_dups_size":149,"ondisk_log_size":216,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":6,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.077244917999999996,"stat_sum":{"num_bytes":15066313,"num_objects":7,"num_object_clones":6,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":103,"num_read_kb":128302,"num_write":78,"num_write_kb":27187,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":20196357,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"ed","length":"1"},{"start":"10a","length":"8"},{"start":"113","length":"c"},{"start":"120","length":"1"},{"start":"122","length":"2"},{"start":"125","length":"2"},{"start":"129","length":"1"},{"start":"12b","length":"1"},{"start":"12d","length":"1"},{"start":"131","length":"1"},{"start":"133","length":"6"},{"start":"13f","length":"2"},{"start":"143","length":"1"}]},{"pgid":"1.5","version":"77'128","reported_seq":1581,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-09-20T22:53:48.299684+0000","last_change":"2024-09-20T22:53:35.470320+0000","last_active":"2024-09-20T22:53:48.299684+0000","last_peered":"2024-09-20T22:53:48.299684+0000","last_clean":"2024-09-20T22:53:48.299684+0000","last_became_active":"2024-09-20T22:48:08.207018+0000","last_became_peered":"2024-09-20T22:48:08.207018+0000","last_unstale":"2024-09-20T22:53:48.299684+0000","last_undegraded":"2024-09-20T22:53:48.299684+0000","last_fullsized":"2024-09-20T22:53:48.299684+0000","mapping_epoch":407,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":408,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:53:35.470248+0000","last_deep_scrub":"77'128","last_deep_scrub_stamp":"2024-09-20T22:45:41.608075+0000","last_clean_scrub_stamp":"2024-09-20T22:53:35.470248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:57.007268+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"625'30","reported_seq":1502,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-09-20T22:53:48.346394+0000","last_change":"2024-09-20T22:53:48.346424+0000","last_active":"2024-09-20T22:53:48.346394+0000","last_peered":"2024-09-20T22:53:48.346394+0000","last_clean":"2024-09-20T22:53:48.346394+0000","last_became_active":"2024-09-20T22:47:49.217605+0000","last_became_peered":"2024-09-20T22:47:49.217605+0000","last_unstale":"2024-09-20T22:53:48.346394+0000","last_undegraded":"2024-09-20T22:53:48.346394+0000","last_fullsized":"2024-09-20T22:53:48.346394+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'30","last_scrub_stamp":"2024-09-20T22:53:48.346341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:53:48.346341+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:02.270695+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":1419,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-09-20T22:53:48.337301+0000","last_change":"2024-09-20T22:52:45.537243+0000","last_active":"2024-09-20T22:53:48.337301+0000","last_peered":"2024-09-20T22:53:48.337301+0000","last_clean":"2024-09-20T22:53:48.337301+0000","last_became_active":"2024-09-20T22:47:49.227069+0000","last_became_peered":"2024-09-20T22:47:49.227069+0000","last_unstale":"2024-09-20T22:53:48.337301+0000","last_undegraded":"2024-09-20T22:53:48.337301+0000","last_fullsized":"2024-09-20T22:53:48.337301+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:45.537173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:52:45.537173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:54.445275+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":722,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-09-20T22:53:48.299892+0000","last_change":"2024-09-20T22:53:26.623524+0000","last_active":"2024-09-20T22:53:48.299892+0000","last_peered":"2024-09-20T22:53:48.299892+0000","last_clean":"2024-09-20T22:53:48.299892+0000","last_became_active":"2024-09-20T22:48:05.230003+0000","last_became_peered":"2024-09-20T22:48:05.230003+0000","last_unstale":"2024-09-20T22:53:48.299892+0000","last_undegraded":"2024-09-20T22:53:48.299892+0000","last_fullsized":"2024-09-20T22:53:48.299892+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:26.623455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:07.947945+0000","last_clean_scrub_stamp":"2024-09-20T22:53:26.623455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:36.274129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":714,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-09-20T22:53:48.299431+0000","last_change":"2024-09-20T22:51:55.748527+0000","last_active":"2024-09-20T22:53:48.299431+0000","last_peered":"2024-09-20T22:53:48.299431+0000","last_clean":"2024-09-20T22:53:48.299431+0000","last_became_active":"2024-09-20T22:48:05.239917+0000","last_became_peered":"2024-09-20T22:48:05.239917+0000","last_unstale":"2024-09-20T22:53:48.299431+0000","last_undegraded":"2024-09-20T22:53:48.299431+0000","last_fullsized":"2024-09-20T22:53:48.299431+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:55.748425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:53.036601+0000","last_clean_scrub_stamp":"2024-09-20T22:51:55.748425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:49.514491+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2","version":"748'802","reported_seq":3246,"reported_epoch":748,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T22:53:48.394259+0000","last_change":"2024-09-20T22:53:48.392561+0000","last_active":"2024-09-20T22:53:48.394259+0000","last_peered":"2024-09-20T22:53:48.394259+0000","last_clean":"2024-09-20T22:53:48.394259+0000","last_became_active":"2024-09-20T22:52:23.286461+0000","last_became_peered":"2024-09-20T22:52:23.286461+0000","last_unstale":"2024-09-20T22:53:48.394259+0000","last_undegraded":"2024-09-20T22:53:48.394259+0000","last_fullsized":"2024-09-20T22:53:48.394259+0000","mapping_epoch":662,"log_start":"471'464","ondisk_log_start":"471'464","created":94,"last_epoch_clean":663,"parent":"0.0","parent_split_bits":0,"last_scrub":"282'120","last_scrub_stamp":"2024-09-20T22:46:22.099642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:46:22.099642+0000","objects_scrubbed":1,"log_size":338,"log_dups_size":687,"ondisk_log_size":338,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":40,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:53.341139+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.98299088899999998,"stat_sum":{"num_bytes":51331449,"num_objects":20,"num_object_clones":16,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":1,"num_read":414,"num_read_kb":423819,"num_write":266,"num_write_kb":104890,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":144,"num_bytes_recovered":106353268,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"ed","length":"1"},{"start":"10a","length":"4"}]},{"pgid":"1.4","version":"77'128","reported_seq":1700,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-09-20T22:53:48.337114+0000","last_change":"2024-09-20T22:52:02.474080+0000","last_active":"2024-09-20T22:53:48.337114+0000","last_peered":"2024-09-20T22:53:48.337114+0000","last_clean":"2024-09-20T22:53:48.337114+0000","last_became_active":"2024-09-20T22:46:15.181305+0000","last_became_peered":"2024-09-20T22:46:15.181305+0000","last_unstale":"2024-09-20T22:53:48.337114+0000","last_undegraded":"2024-09-20T22:53:48.337114+0000","last_fullsized":"2024-09-20T22:53:48.337114+0000","mapping_epoch":94,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:52:02.473979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:52:02.473979+0000","objects_scrubbed":1,"log_size":0,"log_dups_size":59,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"625'28","reported_seq":751,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-09-20T22:53:48.299750+0000","last_change":"2024-09-20T22:51:58.712602+0000","last_active":"2024-09-20T22:53:48.299750+0000","last_peered":"2024-09-20T22:53:48.299750+0000","last_clean":"2024-09-20T22:53:48.299750+0000","last_became_active":"2024-09-20T22:48:05.235812+0000","last_became_peered":"2024-09-20T22:48:05.235812+0000","last_unstale":"2024-09-20T22:53:48.299750+0000","last_undegraded":"2024-09-20T22:53:48.299750+0000","last_fullsized":"2024-09-20T22:53:48.299750+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'28","last_scrub_stamp":"2024-09-20T22:51:58.712488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:51:58.712488+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:49.514477+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":41,"num_read_kb":34,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"67'1","reported_seq":779,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-09-20T22:53:48.300993+0000","last_change":"2024-09-20T22:53:32.995298+0000","last_active":"2024-09-20T22:53:48.300993+0000","last_peered":"2024-09-20T22:53:48.300993+0000","last_clean":"2024-09-20T22:53:48.300993+0000","last_became_active":"2024-09-20T22:51:20.283011+0000","last_became_peered":"2024-09-20T22:51:20.283011+0000","last_unstale":"2024-09-20T22:53:48.300993+0000","last_undegraded":"2024-09-20T22:53:48.300993+0000","last_fullsized":"2024-09-20T22:53:48.300993+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-09-20T22:53:32.995211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:53:32.995211+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:59.820181+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"600'13","reported_seq":1598,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-09-20T22:53:48.335917+0000","last_change":"2024-09-20T22:51:32.448600+0000","last_active":"2024-09-20T22:53:48.335917+0000","last_peered":"2024-09-20T22:53:48.335917+0000","last_clean":"2024-09-20T22:53:48.335917+0000","last_became_active":"2024-09-20T22:51:20.289873+0000","last_became_peered":"2024-09-20T22:51:20.289873+0000","last_unstale":"2024-09-20T22:53:48.335917+0000","last_undegraded":"2024-09-20T22:53:48.335917+0000","last_fullsized":"2024-09-20T22:53:48.335917+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"600'13","last_scrub_stamp":"2024-09-20T22:51:32.448512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:51:32.448512+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":1402,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-09-20T22:53:48.344075+0000","last_change":"2024-09-20T22:53:02.610452+0000","last_active":"2024-09-20T22:53:48.344075+0000","last_peered":"2024-09-20T22:53:48.344075+0000","last_clean":"2024-09-20T22:53:48.344075+0000","last_became_active":"2024-09-20T22:41:44.831901+0000","last_became_peered":"2024-09-20T22:41:44.831901+0000","last_unstale":"2024-09-20T22:53:48.344075+0000","last_undegraded":"2024-09-20T22:53:48.344075+0000","last_fullsized":"2024-09-20T22:53:48.344075+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:02.610359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:53:02.610359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:32.529803+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1","version":"748'448","reported_seq":3649,"reported_epoch":748,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T22:53:48.324406+0000","last_change":"2024-09-20T22:53:48.324406+0000","last_active":"2024-09-20T22:53:48.324406+0000","last_peered":"2024-09-20T22:53:48.324406+0000","last_clean":"2024-09-20T22:53:48.324406+0000","last_became_active":"2024-09-20T22:49:21.232750+0000","last_became_peered":"2024-09-20T22:49:21.232750+0000","last_unstale":"2024-09-20T22:53:48.324406+0000","last_undegraded":"2024-09-20T22:53:48.324406+0000","last_fullsized":"2024-09-20T22:53:48.324406+0000","mapping_epoch":407,"log_start":"474'259","ondisk_log_start":"474'259","created":94,"last_epoch_clean":481,"parent":"0.0","parent_split_bits":0,"last_scrub":"703'407","last_scrub_stamp":"2024-09-20T22:53:06.625437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:53:06.625437+0000","objects_scrubbed":6,"log_size":189,"log_dups_size":338,"ondisk_log_size":189,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:27.835925+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.024293697999999999,"stat_sum":{"num_bytes":11851465,"num_objects":8,"num_object_clones":7,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":115,"num_read_kb":127510,"num_write":96,"num_write_kb":50330,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":8732217,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"ed","length":"1"},{"start":"10a","length":"8"},{"start":"113","length":"c"},{"start":"120","length":"1"},{"start":"122","length":"2"},{"start":"125","length":"2"},{"start":"129","length":"1"},{"start":"12b","length":"1"},{"start":"12d","length":"1"},{"start":"131","length":"1"},{"start":"133","length":"6"},{"start":"13f","length":"2"},{"start":"143","length":"2"},{"start":"146","length":"1"},{"start":"14b","length":"1"}]},{"pgid":"1.7","version":"77'128","reported_seq":1713,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-09-20T22:53:48.330599+0000","last_change":"2024-09-20T22:52:34.898681+0000","last_active":"2024-09-20T22:53:48.330599+0000","last_peered":"2024-09-20T22:53:48.330599+0000","last_clean":"2024-09-20T22:53:48.330599+0000","last_became_active":"2024-09-20T22:46:03.169596+0000","last_became_peered":"2024-09-20T22:46:03.169596+0000","last_unstale":"2024-09-20T22:53:48.330599+0000","last_undegraded":"2024-09-20T22:53:48.330599+0000","last_fullsized":"2024-09-20T22:53:48.330599+0000","mapping_epoch":94,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:52:34.898595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:52:34.898595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:51.834169+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"625'20","reported_seq":378,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-09-20T22:53:48.394740+0000","last_change":"2024-09-20T22:52:19.125548+0000","last_active":"2024-09-20T22:53:48.394740+0000","last_peered":"2024-09-20T22:53:48.394740+0000","last_clean":"2024-09-20T22:53:48.394740+0000","last_became_active":"2024-09-20T22:51:20.297933+0000","last_became_peered":"2024-09-20T22:51:20.297933+0000","last_unstale":"2024-09-20T22:53:48.394740+0000","last_undegraded":"2024-09-20T22:53:48.394740+0000","last_fullsized":"2024-09-20T22:53:48.394740+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'20","last_scrub_stamp":"2024-09-20T22:52:19.125462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:52:19.125462+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":1416,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-09-20T22:53:48.395364+0000","last_change":"2024-09-20T22:52:46.062448+0000","last_active":"2024-09-20T22:53:48.395364+0000","last_peered":"2024-09-20T22:53:48.395364+0000","last_clean":"2024-09-20T22:53:48.395364+0000","last_became_active":"2024-09-20T22:41:38.155629+0000","last_became_peered":"2024-09-20T22:41:38.155629+0000","last_unstale":"2024-09-20T22:53:48.395364+0000","last_undegraded":"2024-09-20T22:53:48.395364+0000","last_fullsized":"2024-09-20T22:53:48.395364+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:46.062352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:52:46.062352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:15.728943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"733'12","reported_seq":833,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-09-20T22:53:48.300760+0000","last_change":"2024-09-20T22:53:33.928576+0000","last_active":"2024-09-20T22:53:48.300760+0000","last_peered":"2024-09-20T22:53:48.300760+0000","last_clean":"2024-09-20T22:53:48.300760+0000","last_became_active":"2024-09-20T22:47:49.227264+0000","last_became_peered":"2024-09-20T22:47:49.227264+0000","last_unstale":"2024-09-20T22:53:48.300760+0000","last_undegraded":"2024-09-20T22:53:48.300760+0000","last_fullsized":"2024-09-20T22:53:48.300760+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"733'12","last_scrub_stamp":"2024-09-20T22:53:33.928499+0000","last_deep_scrub":"733'12","last_deep_scrub_stamp":"2024-09-20T22:53:33.928499+0000","last_clean_scrub_stamp":"2024-09-20T22:53:33.928499+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:02.678336+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":748,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-09-20T22:53:48.298746+0000","last_change":"2024-09-20T22:51:55.755550+0000","last_active":"2024-09-20T22:53:48.298746+0000","last_peered":"2024-09-20T22:53:48.298746+0000","last_clean":"2024-09-20T22:53:48.298746+0000","last_became_active":"2024-09-20T22:47:49.232380+0000","last_became_peered":"2024-09-20T22:47:49.232380+0000","last_unstale":"2024-09-20T22:53:48.298746+0000","last_undegraded":"2024-09-20T22:53:48.298746+0000","last_fullsized":"2024-09-20T22:53:48.298746+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:55.755432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:51:55.755432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.0","version":"746'692","reported_seq":3970,"reported_epoch":748,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T22:53:48.394466+0000","last_change":"2024-09-20T22:53:45.436020+0000","last_active":"2024-09-20T22:53:48.394466+0000","last_peered":"2024-09-20T22:53:48.394466+0000","last_clean":"2024-09-20T22:53:48.394466+0000","last_became_active":"2024-09-20T22:51:53.366096+0000","last_became_peered":"2024-09-20T22:51:53.366096+0000","last_unstale":"2024-09-20T22:53:48.394466+0000","last_undegraded":"2024-09-20T22:53:48.394466+0000","last_fullsized":"2024-09-20T22:53:48.394466+0000","mapping_epoch":632,"log_start":"480'253","ondisk_log_start":"480'253","created":94,"last_epoch_clean":633,"parent":"0.0","parent_split_bits":0,"last_scrub":"562'368","last_scrub_stamp":"2024-09-20T22:50:44.669910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:50:44.669910+0000","objects_scrubbed":15,"log_size":439,"log_dups_size":416,"ondisk_log_size":439,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":10,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":1.0145607329999999,"stat_sum":{"num_bytes":55065179,"num_objects":26,"num_object_clones":23,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":261,"num_read_kb":267348,"num_write":212,"num_write_kb":79681,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":42943203,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"ed","length":"1"},{"start":"10a","length":"8"},{"start":"113","length":"c"},{"start":"120","length":"1"},{"start":"122","length":"2"},{"start":"125","length":"2"},{"start":"129","length":"1"},{"start":"12b","length":"1"},{"start":"12d","length":"1"},{"start":"131","length":"1"},{"start":"133","length":"2"},{"start":"136","length":"2"},{"start":"13f","length":"1"}]},{"pgid":"1.6","version":"77'128","reported_seq":1735,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-09-20T22:53:48.330568+0000","last_change":"2024-09-20T22:52:38.843686+0000","last_active":"2024-09-20T22:53:48.330568+0000","last_peered":"2024-09-20T22:53:48.330568+0000","last_clean":"2024-09-20T22:53:48.330568+0000","last_became_active":"2024-09-20T22:47:52.212190+0000","last_became_peered":"2024-09-20T22:47:52.212190+0000","last_unstale":"2024-09-20T22:53:48.330568+0000","last_undegraded":"2024-09-20T22:53:48.330568+0000","last_fullsized":"2024-09-20T22:53:48.330568+0000","mapping_epoch":391,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:52:38.843586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:52:38.843586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:07.967280+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"625'37","reported_seq":788,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-09-20T22:53:48.300494+0000","last_change":"2024-09-20T22:52:39.703172+0000","last_active":"2024-09-20T22:53:48.300494+0000","last_peered":"2024-09-20T22:53:48.300494+0000","last_clean":"2024-09-20T22:53:48.300494+0000","last_became_active":"2024-09-20T22:47:49.222009+0000","last_became_peered":"2024-09-20T22:47:49.222009+0000","last_unstale":"2024-09-20T22:53:48.300494+0000","last_undegraded":"2024-09-20T22:53:48.300494+0000","last_fullsized":"2024-09-20T22:53:48.300494+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'37","last_scrub_stamp":"2024-09-20T22:52:39.703093+0000","last_deep_scrub":"515'27","last_deep_scrub_stamp":"2024-09-20T22:49:55.964153+0000","last_clean_scrub_stamp":"2024-09-20T22:52:39.703093+0000","objects_scrubbed":8,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:49.380387+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":53,"num_read_kb":39,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":317,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-09-20T22:53:48.395035+0000","last_change":"2024-09-20T22:52:14.225251+0000","last_active":"2024-09-20T22:53:48.395035+0000","last_peered":"2024-09-20T22:53:48.395035+0000","last_clean":"2024-09-20T22:53:48.395035+0000","last_became_active":"2024-09-20T22:51:20.286646+0000","last_became_peered":"2024-09-20T22:51:20.286646+0000","last_unstale":"2024-09-20T22:53:48.395035+0000","last_undegraded":"2024-09-20T22:53:48.395035+0000","last_fullsized":"2024-09-20T22:53:48.395035+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:14.225111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:48:23.320031+0000","last_clean_scrub_stamp":"2024-09-20T22:52:14.225111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:50.351856+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":1427,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-09-20T22:53:48.330085+0000","last_change":"2024-09-20T22:52:39.814499+0000","last_active":"2024-09-20T22:53:48.330085+0000","last_peered":"2024-09-20T22:53:48.330085+0000","last_clean":"2024-09-20T22:53:48.330085+0000","last_became_active":"2024-09-20T22:47:49.214055+0000","last_became_peered":"2024-09-20T22:47:49.214055+0000","last_unstale":"2024-09-20T22:53:48.330085+0000","last_undegraded":"2024-09-20T22:53:48.330085+0000","last_fullsized":"2024-09-20T22:53:48.330085+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:39.814381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:52:39.814381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:52.010799+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":325,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-09-20T22:53:48.329970+0000","last_change":"2024-09-20T22:53:02.916523+0000","last_active":"2024-09-20T22:53:48.329970+0000","last_peered":"2024-09-20T22:53:48.329970+0000","last_clean":"2024-09-20T22:53:48.329970+0000","last_became_active":"2024-09-20T22:51:20.284874+0000","last_became_peered":"2024-09-20T22:51:20.284874+0000","last_unstale":"2024-09-20T22:53:48.329970+0000","last_undegraded":"2024-09-20T22:53:48.329970+0000","last_fullsized":"2024-09-20T22:53:48.329970+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:02.916449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:48:41.208853+0000","last_clean_scrub_stamp":"2024-09-20T22:53:02.916449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:15.012347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.7","version":"744'1347","reported_seq":4662,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-09-20T22:53:48.331589+0000","last_change":"2024-09-20T22:53:48.331797+0000","last_active":"2024-09-20T22:53:48.331589+0000","last_peered":"2024-09-20T22:53:48.331589+0000","last_clean":"2024-09-20T22:53:48.331589+0000","last_became_active":"2024-09-20T22:48:43.224386+0000","last_became_peered":"2024-09-20T22:48:43.224386+0000","last_unstale":"2024-09-20T22:53:48.331589+0000","last_undegraded":"2024-09-20T22:53:48.331589+0000","last_fullsized":"2024-09-20T22:53:48.331589+0000","mapping_epoch":391,"log_start":"436'456","ondisk_log_start":"436'456","created":94,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"704'1235","last_scrub_stamp":"2024-09-20T22:53:06.830408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:53:06.830408+0000","objects_scrubbed":31,"log_size":891,"log_dups_size":886,"ondisk_log_size":891,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:27.877879+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.00077127399999999996,"stat_sum":{"num_bytes":52956118,"num_objects":24,"num_object_clones":19,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":2,"num_read":473,"num_read_kb":486373,"num_write":391,"num_write_kb":133187,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":115,"num_bytes_recovered":51615167,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"ed","length":"1"},{"start":"10a","length":"8"},{"start":"113","length":"c"},{"start":"120","length":"1"},{"start":"122","length":"2"},{"start":"125","length":"2"},{"start":"129","length":"1"},{"start":"12b","length":"1"},{"start":"12d","length":"1"},{"start":"131","length":"1"},{"start":"133","length":"6"},{"start":"13f","length":"2"},{"start":"143","length":"2"},{"start":"146","length":"2"},{"start":"14b","length":"1"},{"start":"14d","length":"1"},{"start":"150","length":"1"}]},{"pgid":"1.1","version":"77'128","reported_seq":1724,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-09-20T22:53:48.394399+0000","last_change":"2024-09-20T22:52:42.146147+0000","last_active":"2024-09-20T22:53:48.394399+0000","last_peered":"2024-09-20T22:53:48.394399+0000","last_clean":"2024-09-20T22:53:48.394399+0000","last_became_active":"2024-09-20T22:47:52.208102+0000","last_became_peered":"2024-09-20T22:47:52.208102+0000","last_unstale":"2024-09-20T22:53:48.394399+0000","last_undegraded":"2024-09-20T22:53:48.394399+0000","last_fullsized":"2024-09-20T22:53:48.394399+0000","mapping_epoch":391,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:52:42.146069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:52:42.146069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:50.657914+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"625'51","reported_seq":498,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-09-20T22:53:48.394895+0000","last_change":"2024-09-20T22:52:17.199465+0000","last_active":"2024-09-20T22:53:48.394895+0000","last_peered":"2024-09-20T22:53:48.394895+0000","last_clean":"2024-09-20T22:53:48.394895+0000","last_became_active":"2024-09-20T22:51:20.289376+0000","last_became_peered":"2024-09-20T22:51:20.289376+0000","last_unstale":"2024-09-20T22:53:48.394895+0000","last_undegraded":"2024-09-20T22:53:48.394895+0000","last_fullsized":"2024-09-20T22:53:48.394895+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'51","last_scrub_stamp":"2024-09-20T22:52:17.199367+0000","last_deep_scrub":"86'38","last_deep_scrub_stamp":"2024-09-20T22:42:43.935677+0000","last_clean_scrub_stamp":"2024-09-20T22:52:17.199367+0000","objects_scrubbed":10,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:50.351894+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":714,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-09-20T22:53:48.299690+0000","last_change":"2024-09-20T22:52:36.695537+0000","last_active":"2024-09-20T22:53:48.299690+0000","last_peered":"2024-09-20T22:53:48.299690+0000","last_clean":"2024-09-20T22:53:48.299690+0000","last_became_active":"2024-09-20T22:48:05.237756+0000","last_became_peered":"2024-09-20T22:48:05.237756+0000","last_unstale":"2024-09-20T22:53:48.299690+0000","last_undegraded":"2024-09-20T22:53:48.299690+0000","last_fullsized":"2024-09-20T22:53:48.299690+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:36.695446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:52:36.695446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:51.545377+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"69'2","reported_seq":1436,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-09-20T22:53:48.395355+0000","last_change":"2024-09-20T22:51:37.070187+0000","last_active":"2024-09-20T22:53:48.395355+0000","last_peered":"2024-09-20T22:53:48.395355+0000","last_clean":"2024-09-20T22:53:48.395355+0000","last_became_active":"2024-09-20T22:51:20.286952+0000","last_became_peered":"2024-09-20T22:51:20.286952+0000","last_unstale":"2024-09-20T22:53:48.395355+0000","last_undegraded":"2024-09-20T22:53:48.395355+0000","last_fullsized":"2024-09-20T22:53:48.395355+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2024-09-20T22:51:37.070111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:36.130684+0000","last_clean_scrub_stamp":"2024-09-20T22:51:37.070111+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:48.982742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087280599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":759,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-09-20T22:53:48.300269+0000","last_change":"2024-09-20T22:51:53.772969+0000","last_active":"2024-09-20T22:53:48.300269+0000","last_peered":"2024-09-20T22:53:48.300269+0000","last_clean":"2024-09-20T22:53:48.300269+0000","last_became_active":"2024-09-20T22:51:20.278427+0000","last_became_peered":"2024-09-20T22:51:20.278427+0000","last_unstale":"2024-09-20T22:53:48.300269+0000","last_undegraded":"2024-09-20T22:53:48.300269+0000","last_fullsized":"2024-09-20T22:53:48.300269+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:53.772885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:51:53.772885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":1407,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-09-20T22:53:48.336607+0000","last_change":"2024-09-20T22:52:21.363432+0000","last_active":"2024-09-20T22:53:48.336607+0000","last_peered":"2024-09-20T22:53:48.336607+0000","last_clean":"2024-09-20T22:53:48.336607+0000","last_became_active":"2024-09-20T22:48:05.234193+0000","last_became_peered":"2024-09-20T22:48:05.234193+0000","last_unstale":"2024-09-20T22:53:48.336607+0000","last_undegraded":"2024-09-20T22:53:48.336607+0000","last_fullsized":"2024-09-20T22:53:48.336607+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:21.363359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:52:21.363359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"744'500","reported_seq":2249,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-09-20T22:53:48.300301+0000","last_change":"2024-09-20T22:53:48.300492+0000","last_active":"2024-09-20T22:53:48.300301+0000","last_peered":"2024-09-20T22:53:48.300301+0000","last_clean":"2024-09-20T22:53:48.300301+0000","last_became_active":"2024-09-20T22:51:23.267613+0000","last_became_peered":"2024-09-20T22:51:23.267613+0000","last_unstale":"2024-09-20T22:53:48.300301+0000","last_undegraded":"2024-09-20T22:53:48.300301+0000","last_fullsized":"2024-09-20T22:53:48.300301+0000","mapping_epoch":602,"log_start":"456'176","ondisk_log_start":"456'176","created":94,"last_epoch_clean":603,"parent":"0.0","parent_split_bits":0,"last_scrub":"616'341","last_scrub_stamp":"2024-09-20T22:51:36.893731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:51:36.893731+0000","objects_scrubbed":8,"log_size":324,"log_dups_size":318,"ondisk_log_size":324,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.00045702399999999999,"stat_sum":{"num_bytes":26104163,"num_objects":10,"num_object_clones":8,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":171,"num_read_kb":155428,"num_write":149,"num_write_kb":49173,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":26566881,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"ed","length":"1"},{"start":"10a","length":"8"},{"start":"113","length":"c"},{"start":"120","length":"1"},{"start":"122","length":"2"},{"start":"125","length":"2"},{"start":"129","length":"1"},{"start":"12b","length":"1"},{"start":"12d","length":"1"},{"start":"131","length":"1"},{"start":"133","length":"6"},{"start":"13f","length":"2"},{"start":"143","length":"2"},{"start":"146","length":"2"},{"start":"14b","length":"1"},{"start":"14d","length":"1"},{"start":"150","length":"1"}]},{"pgid":"1.2","version":"77'128","reported_seq":1784,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-09-20T22:53:48.343837+0000","last_change":"2024-09-20T22:52:02.623634+0000","last_active":"2024-09-20T22:53:48.343837+0000","last_peered":"2024-09-20T22:53:48.343837+0000","last_clean":"2024-09-20T22:53:48.343837+0000","last_became_active":"2024-09-20T22:48:08.203028+0000","last_became_peered":"2024-09-20T22:48:08.203028+0000","last_unstale":"2024-09-20T22:53:48.343837+0000","last_undegraded":"2024-09-20T22:53:48.343837+0000","last_fullsized":"2024-09-20T22:53:48.343837+0000","mapping_epoch":407,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":408,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:52:02.623562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:52:02.623562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:51.391693+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"625'22","reported_seq":1178,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-09-20T22:53:48.395763+0000","last_change":"2024-09-20T22:53:35.117496+0000","last_active":"2024-09-20T22:53:48.395763+0000","last_peered":"2024-09-20T22:53:48.395763+0000","last_clean":"2024-09-20T22:53:48.395763+0000","last_became_active":"2024-09-20T22:51:20.293610+0000","last_became_peered":"2024-09-20T22:51:20.293610+0000","last_unstale":"2024-09-20T22:53:48.395763+0000","last_undegraded":"2024-09-20T22:53:48.395763+0000","last_fullsized":"2024-09-20T22:53:48.395763+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'22","last_scrub_stamp":"2024-09-20T22:53:35.117387+0000","last_deep_scrub":"443'18","last_deep_scrub_stamp":"2024-09-20T22:48:44.182517+0000","last_clean_scrub_stamp":"2024-09-20T22:53:35.117387+0000","objects_scrubbed":5,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:45.532207+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":990,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":325,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-09-20T22:53:48.394792+0000","last_change":"2024-09-20T22:53:30.210048+0000","last_active":"2024-09-20T22:53:48.394792+0000","last_peered":"2024-09-20T22:53:48.394792+0000","last_clean":"2024-09-20T22:53:48.394792+0000","last_became_active":"2024-09-20T22:51:20.278609+0000","last_became_peered":"2024-09-20T22:51:20.278609+0000","last_unstale":"2024-09-20T22:53:48.394792+0000","last_undegraded":"2024-09-20T22:53:48.394792+0000","last_fullsized":"2024-09-20T22:53:48.394792+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:30.209986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:53:30.209986+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:35.918431+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"380'2","reported_seq":1461,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-09-20T22:53:48.394670+0000","last_change":"2024-09-20T22:52:37.033309+0000","last_active":"2024-09-20T22:53:48.394670+0000","last_peered":"2024-09-20T22:53:48.394670+0000","last_clean":"2024-09-20T22:53:48.394670+0000","last_became_active":"2024-09-20T22:47:49.222812+0000","last_became_peered":"2024-09-20T22:47:49.222812+0000","last_unstale":"2024-09-20T22:53:48.394670+0000","last_undegraded":"2024-09-20T22:53:48.394670+0000","last_fullsized":"2024-09-20T22:53:48.394670+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"380'2","last_scrub_stamp":"2024-09-20T22:52:37.033222+0000","last_deep_scrub":"380'2","last_deep_scrub_stamp":"2024-09-20T22:47:40.983039+0000","last_clean_scrub_stamp":"2024-09-20T22:52:37.033222+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:56.197067+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001301459,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":1415,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-09-20T22:53:48.395850+0000","last_change":"2024-09-20T22:53:25.223722+0000","last_active":"2024-09-20T22:53:48.395850+0000","last_peered":"2024-09-20T22:53:48.395850+0000","last_clean":"2024-09-20T22:53:48.395850+0000","last_became_active":"2024-09-20T22:41:42.185112+0000","last_became_peered":"2024-09-20T22:41:42.185112+0000","last_unstale":"2024-09-20T22:53:48.395850+0000","last_undegraded":"2024-09-20T22:53:48.395850+0000","last_fullsized":"2024-09-20T22:53:48.395850+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:25.223611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:47:22.881109+0000","last_clean_scrub_stamp":"2024-09-20T22:53:25.223611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:53.455503+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":1402,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-09-20T22:53:48.344321+0000","last_change":"2024-09-20T22:53:06.644045+0000","last_active":"2024-09-20T22:53:48.344321+0000","last_peered":"2024-09-20T22:53:48.344321+0000","last_clean":"2024-09-20T22:53:48.344321+0000","last_became_active":"2024-09-20T22:41:44.830515+0000","last_became_peered":"2024-09-20T22:41:44.830515+0000","last_unstale":"2024-09-20T22:53:48.344321+0000","last_undegraded":"2024-09-20T22:53:48.344321+0000","last_fullsized":"2024-09-20T22:53:48.344321+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:06.643947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:53:06.643947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:16.583688+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"744'389","reported_seq":3716,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-09-20T22:53:48.344781+0000","last_change":"2024-09-20T22:53:48.344945+0000","last_active":"2024-09-20T22:53:48.344781+0000","last_peered":"2024-09-20T22:53:48.344781+0000","last_clean":"2024-09-20T22:53:48.344781+0000","last_became_active":"2024-09-20T22:48:57.236417+0000","last_became_peered":"2024-09-20T22:48:57.236417+0000","last_unstale":"2024-09-20T22:53:48.344781+0000","last_undegraded":"2024-09-20T22:53:48.344781+0000","last_fullsized":"2024-09-20T22:53:48.344781+0000","mapping_epoch":391,"log_start":"450'190","ondisk_log_start":"450'190","created":94,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"674'316","last_scrub_stamp":"2024-09-20T22:52:35.685017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:52:35.685017+0000","objects_scrubbed":8,"log_size":199,"log_dups_size":240,"ondisk_log_size":199,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:52.520029+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.00047783100000000001,"stat_sum":{"num_bytes":14965811,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":99,"num_read_kb":89344,"num_write":98,"num_write_kb":40424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":19673726,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"ed","length":"1"},{"start":"10a","length":"8"},{"start":"113","length":"c"},{"start":"120","length":"1"},{"start":"122","length":"2"},{"start":"125","length":"2"},{"start":"129","length":"1"},{"start":"12b","length":"1"},{"start":"12d","length":"1"},{"start":"131","length":"1"},{"start":"133","length":"6"},{"start":"13f","length":"2"},{"start":"143","length":"2"},{"start":"146","length":"2"},{"start":"14b","length":"1"},{"start":"14d","length":"1"},{"start":"150","length":"1"}]},{"pgid":"1.3","version":"77'128","reported_seq":620,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-09-20T22:53:48.330087+0000","last_change":"2024-09-20T22:53:05.866046+0000","last_active":"2024-09-20T22:53:48.330087+0000","last_peered":"2024-09-20T22:53:48.330087+0000","last_clean":"2024-09-20T22:53:48.330087+0000","last_became_active":"2024-09-20T22:51:23.266810+0000","last_became_peered":"2024-09-20T22:51:23.266810+0000","last_unstale":"2024-09-20T22:53:48.330087+0000","last_undegraded":"2024-09-20T22:53:48.330087+0000","last_fullsized":"2024-09-20T22:53:48.330087+0000","mapping_epoch":602,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":603,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:53:05.865942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:53:05.865942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:28.578041+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.6","version":"625'24","reported_seq":799,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-09-20T22:53:48.301339+0000","last_change":"2024-09-20T22:52:48.829973+0000","last_active":"2024-09-20T22:53:48.301339+0000","last_peered":"2024-09-20T22:53:48.301339+0000","last_clean":"2024-09-20T22:53:48.301339+0000","last_became_active":"2024-09-20T22:51:20.283112+0000","last_became_peered":"2024-09-20T22:51:20.283112+0000","last_unstale":"2024-09-20T22:53:48.301339+0000","last_undegraded":"2024-09-20T22:53:48.301339+0000","last_fullsized":"2024-09-20T22:53:48.301339+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'24","last_scrub_stamp":"2024-09-20T22:52:48.829882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:52:48.829882+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:17.822454+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":761,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-09-20T22:53:48.299134+0000","last_change":"2024-09-20T22:53:25.172560+0000","last_active":"2024-09-20T22:53:48.299134+0000","last_peered":"2024-09-20T22:53:48.299134+0000","last_clean":"2024-09-20T22:53:48.299134+0000","last_became_active":"2024-09-20T22:48:05.233093+0000","last_became_peered":"2024-09-20T22:48:05.233093+0000","last_unstale":"2024-09-20T22:53:48.299134+0000","last_undegraded":"2024-09-20T22:53:48.299134+0000","last_fullsized":"2024-09-20T22:53:48.299134+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:25.172487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:44:19.834641+0000","last_clean_scrub_stamp":"2024-09-20T22:53:25.172487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:39.242526+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"75'5","reported_seq":2178,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-09-20T22:53:48.343784+0000","last_change":"2024-09-20T22:52:44.568232+0000","last_active":"2024-09-20T22:53:48.343784+0000","last_peered":"2024-09-20T22:53:48.343784+0000","last_clean":"2024-09-20T22:53:48.343784+0000","last_became_active":"2024-09-20T22:51:20.281329+0000","last_became_peered":"2024-09-20T22:51:20.281329+0000","last_unstale":"2024-09-20T22:53:48.343784+0000","last_undegraded":"2024-09-20T22:53:48.343784+0000","last_fullsized":"2024-09-20T22:53:48.343784+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-09-20T22:52:44.568129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:36.130684+0000","last_clean_scrub_stamp":"2024-09-20T22:52:44.568129+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:05.662332+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00078631700000000003,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":720,"num_read_kb":715,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1167,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":1426,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-09-20T22:53:48.395831+0000","last_change":"2024-09-20T22:52:48.082097+0000","last_active":"2024-09-20T22:53:48.395831+0000","last_peered":"2024-09-20T22:53:48.395831+0000","last_clean":"2024-09-20T22:53:48.395831+0000","last_became_active":"2024-09-20T22:51:20.274522+0000","last_became_peered":"2024-09-20T22:51:20.274522+0000","last_unstale":"2024-09-20T22:53:48.395831+0000","last_undegraded":"2024-09-20T22:53:48.395831+0000","last_fullsized":"2024-09-20T22:53:48.395831+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:48.082027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:52:48.082027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:51.217095+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":309,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-09-20T22:53:48.394966+0000","last_change":"2024-09-20T22:51:20.288058+0000","last_active":"2024-09-20T22:53:48.394966+0000","last_peered":"2024-09-20T22:53:48.394966+0000","last_clean":"2024-09-20T22:53:48.394966+0000","last_became_active":"2024-09-20T22:51:20.287750+0000","last_became_peered":"2024-09-20T22:51:20.287750+0000","last_unstale":"2024-09-20T22:53:48.394966+0000","last_undegraded":"2024-09-20T22:53:48.394966+0000","last_fullsized":"2024-09-20T22:53:48.394966+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:05.621880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:51:05.621880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:48.982582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.6","version":"744'506","reported_seq":3757,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-09-20T22:53:48.301487+0000","last_change":"2024-09-20T22:53:48.301839+0000","last_active":"2024-09-20T22:53:48.301487+0000","last_peered":"2024-09-20T22:53:48.301487+0000","last_clean":"2024-09-20T22:53:48.301487+0000","last_became_active":"2024-09-20T22:48:47.234531+0000","last_became_peered":"2024-09-20T22:48:47.234531+0000","last_unstale":"2024-09-20T22:53:48.301487+0000","last_undegraded":"2024-09-20T22:53:48.301487+0000","last_fullsized":"2024-09-20T22:53:48.301487+0000","mapping_epoch":391,"log_start":"443'181","ondisk_log_start":"443'181","created":94,"last_epoch_clean":447,"parent":"0.0","parent_split_bits":0,"last_scrub":"654'392","last_scrub_stamp":"2024-09-20T22:52:14.706591+0000","last_deep_scrub":"440'177","last_deep_scrub_stamp":"2024-09-20T22:48:40.602342+0000","last_clean_scrub_stamp":"2024-09-20T22:52:14.706591+0000","objects_scrubbed":10,"log_size":325,"log_dups_size":314,"ondisk_log_size":325,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.00069533799999999999,"stat_sum":{"num_bytes":22318112,"num_objects":11,"num_object_clones":9,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":1,"num_read":199,"num_read_kb":209855,"num_write":141,"num_write_kb":55093,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":18662529,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"ed","length":"1"},{"start":"10a","length":"8"},{"start":"113","length":"c"},{"start":"120","length":"1"},{"start":"122","length":"2"},{"start":"125","length":"2"},{"start":"129","length":"1"},{"start":"12b","length":"1"},{"start":"12d","length":"1"},{"start":"131","length":"1"},{"start":"133","length":"6"},{"start":"13f","length":"2"},{"start":"143","length":"2"},{"start":"146","length":"2"},{"start":"14b","length":"1"},{"start":"14d","length":"1"},{"start":"150","length":"1"}]},{"pgid":"4.5","version":"625'32","reported_seq":1702,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-09-20T22:53:48.330790+0000","last_change":"2024-09-20T22:53:07.867174+0000","last_active":"2024-09-20T22:53:48.330790+0000","last_peered":"2024-09-20T22:53:48.330790+0000","last_clean":"2024-09-20T22:53:48.330790+0000","last_became_active":"2024-09-20T22:48:05.243523+0000","last_became_peered":"2024-09-20T22:48:05.243523+0000","last_unstale":"2024-09-20T22:53:48.330790+0000","last_undegraded":"2024-09-20T22:53:48.330790+0000","last_fullsized":"2024-09-20T22:53:48.330790+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'32","last_scrub_stamp":"2024-09-20T22:53:07.867088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:53:07.867088+0000","objects_scrubbed":8,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:23.661414+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"67'1","reported_seq":1405,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-09-20T22:53:48.337317+0000","last_change":"2024-09-20T22:51:35.440516+0000","last_active":"2024-09-20T22:53:48.337317+0000","last_peered":"2024-09-20T22:53:48.337317+0000","last_clean":"2024-09-20T22:53:48.337317+0000","last_became_active":"2024-09-20T22:41:38.158917+0000","last_became_peered":"2024-09-20T22:41:38.158917+0000","last_unstale":"2024-09-20T22:53:48.337317+0000","last_undegraded":"2024-09-20T22:53:48.337317+0000","last_fullsized":"2024-09-20T22:53:48.337317+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-09-20T22:51:35.440416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:51:35.440416+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"77'128","reported_seq":1756,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-09-20T22:53:48.343966+0000","last_change":"2024-09-20T22:52:18.688292+0000","last_active":"2024-09-20T22:53:48.343966+0000","last_peered":"2024-09-20T22:53:48.343966+0000","last_clean":"2024-09-20T22:53:48.343966+0000","last_became_active":"2024-09-20T22:47:52.206169+0000","last_became_peered":"2024-09-20T22:47:52.206169+0000","last_unstale":"2024-09-20T22:53:48.343966+0000","last_undegraded":"2024-09-20T22:53:48.343966+0000","last_fullsized":"2024-09-20T22:53:48.343966+0000","mapping_epoch":391,"log_start":"77'128","ondisk_log_start":"77'128","created":19,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:52:18.688135+0000","last_deep_scrub":"77'128","last_deep_scrub_stamp":"2024-09-20T22:47:22.417099+0000","last_clean_scrub_stamp":"2024-09-20T22:52:18.688135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":1414,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-09-20T22:53:48.344256+0000","last_change":"2024-09-20T22:53:03.564792+0000","last_active":"2024-09-20T22:53:48.344256+0000","last_peered":"2024-09-20T22:53:48.344256+0000","last_clean":"2024-09-20T22:53:48.344256+0000","last_became_active":"2024-09-20T22:41:42.184387+0000","last_became_peered":"2024-09-20T22:41:42.184387+0000","last_unstale":"2024-09-20T22:53:48.344256+0000","last_undegraded":"2024-09-20T22:53:48.344256+0000","last_fullsized":"2024-09-20T22:53:48.344256+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:03.564726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:53:03.564726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:25.313703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":1416,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-09-20T22:53:48.395672+0000","last_change":"2024-09-20T22:53:09.957459+0000","last_active":"2024-09-20T22:53:48.395672+0000","last_peered":"2024-09-20T22:53:48.395672+0000","last_clean":"2024-09-20T22:53:48.395672+0000","last_became_active":"2024-09-20T22:48:05.241602+0000","last_became_peered":"2024-09-20T22:48:05.241602+0000","last_unstale":"2024-09-20T22:53:48.395672+0000","last_undegraded":"2024-09-20T22:53:48.395672+0000","last_fullsized":"2024-09-20T22:53:48.395672+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:09.957374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:53:09.957374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:12.536331+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"744'599","reported_seq":3970,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-09-20T22:53:48.331185+0000","last_change":"2024-09-20T22:53:48.331398+0000","last_active":"2024-09-20T22:53:48.331185+0000","last_peered":"2024-09-20T22:53:48.331185+0000","last_clean":"2024-09-20T22:53:48.331185+0000","last_became_active":"2024-09-20T22:51:23.269216+0000","last_became_peered":"2024-09-20T22:51:23.269216+0000","last_unstale":"2024-09-20T22:53:48.331185+0000","last_undegraded":"2024-09-20T22:53:48.331185+0000","last_fullsized":"2024-09-20T22:53:48.331185+0000","mapping_epoch":602,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":603,"parent":"0.0","parent_split_bits":0,"last_scrub":"702'544","last_scrub_stamp":"2024-09-20T22:53:03.918130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:53:03.918130+0000","objects_scrubbed":10,"log_size":599,"log_dups_size":0,"ondisk_log_size":599,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:29.798604+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.00063856999999999998,"stat_sum":{"num_bytes":22554409,"num_objects":9,"num_object_clones":7,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":188,"num_read_kb":188382,"num_write":108,"num_write_kb":45319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":52,"num_bytes_recovered":30669453,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"ed","length":"1"},{"start":"10a","length":"8"},{"start":"113","length":"c"},{"start":"120","length":"1"},{"start":"122","length":"2"},{"start":"125","length":"2"},{"start":"129","length":"1"},{"start":"12b","length":"1"},{"start":"12d","length":"1"},{"start":"131","length":"1"},{"start":"133","length":"6"},{"start":"13f","length":"2"},{"start":"143","length":"2"},{"start":"146","length":"2"},{"start":"14b","length":"1"},{"start":"14d","length":"1"},{"start":"150","length":"1"}]},{"pgid":"4.e","version":"625'19","reported_seq":756,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-09-20T22:53:48.300168+0000","last_change":"2024-09-20T22:52:44.783087+0000","last_active":"2024-09-20T22:53:48.300168+0000","last_peered":"2024-09-20T22:53:48.300168+0000","last_clean":"2024-09-20T22:53:48.300168+0000","last_became_active":"2024-09-20T22:51:20.288958+0000","last_became_peered":"2024-09-20T22:51:20.288958+0000","last_unstale":"2024-09-20T22:53:48.300168+0000","last_undegraded":"2024-09-20T22:53:48.300168+0000","last_fullsized":"2024-09-20T22:53:48.300168+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'19","last_scrub_stamp":"2024-09-20T22:52:44.782975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:52:44.782975+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":714,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-09-20T22:53:48.299223+0000","last_change":"2024-09-20T22:51:29.757136+0000","last_active":"2024-09-20T22:53:48.299223+0000","last_peered":"2024-09-20T22:53:48.299223+0000","last_clean":"2024-09-20T22:53:48.299223+0000","last_became_active":"2024-09-20T22:48:05.254688+0000","last_became_peered":"2024-09-20T22:48:05.254688+0000","last_unstale":"2024-09-20T22:53:48.299223+0000","last_undegraded":"2024-09-20T22:53:48.299223+0000","last_fullsized":"2024-09-20T22:53:48.299223+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:29.757030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:51:29.757030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:49.514427+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":1412,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-09-20T22:53:48.395589+0000","last_change":"2024-09-20T22:51:51.002353+0000","last_active":"2024-09-20T22:53:48.395589+0000","last_peered":"2024-09-20T22:53:48.395589+0000","last_clean":"2024-09-20T22:53:48.395589+0000","last_became_active":"2024-09-20T22:48:05.233778+0000","last_became_peered":"2024-09-20T22:48:05.233778+0000","last_unstale":"2024-09-20T22:53:48.395589+0000","last_undegraded":"2024-09-20T22:53:48.395589+0000","last_fullsized":"2024-09-20T22:53:48.395589+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:51.002226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:51:51.002226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:48.982810+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":1395,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-09-20T22:53:48.337023+0000","last_change":"2024-09-20T22:53:10.647075+0000","last_active":"2024-09-20T22:53:48.337023+0000","last_peered":"2024-09-20T22:53:48.337023+0000","last_clean":"2024-09-20T22:53:48.337023+0000","last_became_active":"2024-09-20T22:41:44.832009+0000","last_became_peered":"2024-09-20T22:41:44.832009+0000","last_unstale":"2024-09-20T22:53:48.337023+0000","last_undegraded":"2024-09-20T22:53:48.337023+0000","last_fullsized":"2024-09-20T22:53:48.337023+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:10.647005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:48:40.127123+0000","last_clean_scrub_stamp":"2024-09-20T22:53:10.647005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:32.128560+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.e","version":"744'746","reported_seq":4054,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-09-20T22:53:48.396215+0000","last_change":"2024-09-20T22:53:48.396345+0000","last_active":"2024-09-20T22:53:48.396215+0000","last_peered":"2024-09-20T22:53:48.396215+0000","last_clean":"2024-09-20T22:53:48.396215+0000","last_became_active":"2024-09-20T22:47:52.209452+0000","last_became_peered":"2024-09-20T22:47:52.209452+0000","last_unstale":"2024-09-20T22:53:48.396215+0000","last_undegraded":"2024-09-20T22:53:48.396215+0000","last_fullsized":"2024-09-20T22:53:48.396215+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"681'618","last_scrub_stamp":"2024-09-20T22:52:45.935130+0000","last_deep_scrub":"174'60","last_deep_scrub_stamp":"2024-09-20T22:44:18.301776+0000","last_clean_scrub_stamp":"2024-09-20T22:52:45.935130+0000","objects_scrubbed":13,"log_size":746,"log_dups_size":0,"ondisk_log_size":746,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:50.207722+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.000415602,"stat_sum":{"num_bytes":25950233,"num_objects":11,"num_object_clones":9,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":252,"num_read_kb":256136,"num_write":178,"num_write_kb":63811,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":23110741,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"ed","length":"1"},{"start":"10a","length":"8"},{"start":"113","length":"c"},{"start":"120","length":"1"},{"start":"122","length":"2"},{"start":"125","length":"2"},{"start":"129","length":"1"},{"start":"12b","length":"1"},{"start":"12d","length":"1"},{"start":"131","length":"1"},{"start":"133","length":"6"},{"start":"13f","length":"2"},{"start":"143","length":"2"},{"start":"146","length":"2"},{"start":"14b","length":"1"},{"start":"14d","length":"1"},{"start":"150","length":"1"}]},{"pgid":"1.8","version":"77'128","reported_seq":1676,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-09-20T22:53:48.394329+0000","last_change":"2024-09-20T22:51:56.108038+0000","last_active":"2024-09-20T22:53:48.394329+0000","last_peered":"2024-09-20T22:53:48.394329+0000","last_clean":"2024-09-20T22:53:48.394329+0000","last_became_active":"2024-09-20T22:42:52.405594+0000","last_became_peered":"2024-09-20T22:42:52.405594+0000","last_unstale":"2024-09-20T22:53:48.394329+0000","last_undegraded":"2024-09-20T22:53:48.394329+0000","last_fullsized":"2024-09-20T22:53:48.394329+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:51:56.107944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:51:56.107944+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:48.982793+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"625'31","reported_seq":785,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-09-20T22:53:48.299597+0000","last_change":"2024-09-20T22:52:13.620897+0000","last_active":"2024-09-20T22:53:48.299597+0000","last_peered":"2024-09-20T22:53:48.299597+0000","last_clean":"2024-09-20T22:53:48.299597+0000","last_became_active":"2024-09-20T22:51:20.288846+0000","last_became_peered":"2024-09-20T22:51:20.288846+0000","last_unstale":"2024-09-20T22:53:48.299597+0000","last_undegraded":"2024-09-20T22:53:48.299597+0000","last_fullsized":"2024-09-20T22:53:48.299597+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'31","last_scrub_stamp":"2024-09-20T22:52:13.620754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:52:13.620754+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:49.514521+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":49,"num_read_kb":39,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":970,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-09-20T22:53:48.395330+0000","last_change":"2024-09-20T22:53:08.105116+0000","last_active":"2024-09-20T22:53:48.395330+0000","last_peered":"2024-09-20T22:53:48.395330+0000","last_clean":"2024-09-20T22:53:48.395330+0000","last_became_active":"2024-09-20T22:51:20.294073+0000","last_became_peered":"2024-09-20T22:51:20.294073+0000","last_unstale":"2024-09-20T22:53:48.395330+0000","last_undegraded":"2024-09-20T22:53:48.395330+0000","last_fullsized":"2024-09-20T22:53:48.395330+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:08.105010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:35.492562+0000","last_clean_scrub_stamp":"2024-09-20T22:53:08.105010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:35.105470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":1027,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-09-20T22:53:48.299074+0000","last_change":"2024-09-20T22:51:25.738132+0000","last_active":"2024-09-20T22:53:48.299074+0000","last_peered":"2024-09-20T22:53:48.299074+0000","last_clean":"2024-09-20T22:53:48.299074+0000","last_became_active":"2024-09-20T22:51:20.281724+0000","last_became_peered":"2024-09-20T22:51:20.281724+0000","last_unstale":"2024-09-20T22:53:48.299074+0000","last_undegraded":"2024-09-20T22:53:48.299074+0000","last_fullsized":"2024-09-20T22:53:48.299074+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:25.738061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:51:25.738061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:49.514398+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":1431,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-09-20T22:53:48.395506+0000","last_change":"2024-09-20T22:53:05.106323+0000","last_active":"2024-09-20T22:53:48.395506+0000","last_peered":"2024-09-20T22:53:48.395506+0000","last_clean":"2024-09-20T22:53:48.395506+0000","last_became_active":"2024-09-20T22:48:05.239006+0000","last_became_peered":"2024-09-20T22:48:05.239006+0000","last_unstale":"2024-09-20T22:53:48.395506+0000","last_undegraded":"2024-09-20T22:53:48.395506+0000","last_fullsized":"2024-09-20T22:53:48.395506+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:05.106224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:53:05.106224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:05.128923+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"744'819","reported_seq":4104,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-09-20T22:53:48.302530+0000","last_change":"2024-09-20T22:53:48.302884+0000","last_active":"2024-09-20T22:53:48.302530+0000","last_peered":"2024-09-20T22:53:48.302530+0000","last_clean":"2024-09-20T22:53:48.302530+0000","last_became_active":"2024-09-20T22:47:52.216198+0000","last_became_peered":"2024-09-20T22:47:52.216198+0000","last_unstale":"2024-09-20T22:53:48.302530+0000","last_undegraded":"2024-09-20T22:53:48.302530+0000","last_fullsized":"2024-09-20T22:53:48.302530+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"630'603","last_scrub_stamp":"2024-09-20T22:51:50.769050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:51:50.769050+0000","objects_scrubbed":8,"log_size":819,"log_dups_size":0,"ondisk_log_size":819,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00067838000000000002,"stat_sum":{"num_bytes":28281757,"num_objects":14,"num_object_clones":11,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":332,"num_read_kb":329185,"num_write":209,"num_write_kb":67073,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":53,"num_bytes_recovered":14983556,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"ed","length":"1"},{"start":"10a","length":"8"},{"start":"113","length":"c"},{"start":"120","length":"1"},{"start":"122","length":"2"},{"start":"125","length":"2"},{"start":"129","length":"1"},{"start":"12b","length":"1"},{"start":"12d","length":"1"},{"start":"131","length":"1"},{"start":"133","length":"6"},{"start":"13f","length":"2"},{"start":"143","length":"2"},{"start":"146","length":"2"},{"start":"14b","length":"1"},{"start":"14d","length":"1"},{"start":"150","length":"1"}]},{"pgid":"1.9","version":"77'128","reported_seq":1724,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-09-20T22:53:48.343745+0000","last_change":"2024-09-20T22:52:39.568327+0000","last_active":"2024-09-20T22:53:48.343745+0000","last_peered":"2024-09-20T22:53:48.343745+0000","last_clean":"2024-09-20T22:53:48.343745+0000","last_became_active":"2024-09-20T22:50:42.262862+0000","last_became_peered":"2024-09-20T22:50:42.262862+0000","last_unstale":"2024-09-20T22:53:48.343745+0000","last_undegraded":"2024-09-20T22:53:48.343745+0000","last_fullsized":"2024-09-20T22:53:48.343745+0000","mapping_epoch":561,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":562,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:52:39.568240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:52:39.568240+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"625'20","reported_seq":741,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-09-20T22:53:48.299953+0000","last_change":"2024-09-20T22:52:22.646547+0000","last_active":"2024-09-20T22:53:48.299953+0000","last_peered":"2024-09-20T22:53:48.299953+0000","last_clean":"2024-09-20T22:53:48.299953+0000","last_became_active":"2024-09-20T22:48:05.242811+0000","last_became_peered":"2024-09-20T22:48:05.242811+0000","last_unstale":"2024-09-20T22:53:48.299953+0000","last_undegraded":"2024-09-20T22:53:48.299953+0000","last_fullsized":"2024-09-20T22:53:48.299953+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'20","last_scrub_stamp":"2024-09-20T22:52:22.646468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:52:22.646468+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:48.440427+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":1431,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-09-20T22:53:48.336196+0000","last_change":"2024-09-20T22:52:44.527004+0000","last_active":"2024-09-20T22:53:48.336196+0000","last_peered":"2024-09-20T22:53:48.336196+0000","last_clean":"2024-09-20T22:53:48.336196+0000","last_became_active":"2024-09-20T22:48:05.233999+0000","last_became_peered":"2024-09-20T22:48:05.233999+0000","last_unstale":"2024-09-20T22:53:48.336196+0000","last_undegraded":"2024-09-20T22:53:48.336196+0000","last_fullsized":"2024-09-20T22:53:48.336196+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:44.526935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:52:44.526935+0000","last_clean_scrub_stamp":"2024-09-20T22:52:44.526935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:52.383797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"261'9","reported_seq":1571,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-09-20T22:53:48.395335+0000","last_change":"2024-09-20T22:52:18.218722+0000","last_active":"2024-09-20T22:53:48.395335+0000","last_peered":"2024-09-20T22:53:48.395335+0000","last_clean":"2024-09-20T22:53:48.395335+0000","last_became_active":"2024-09-20T22:41:42.185863+0000","last_became_peered":"2024-09-20T22:41:42.185863+0000","last_unstale":"2024-09-20T22:53:48.395335+0000","last_undegraded":"2024-09-20T22:53:48.395335+0000","last_fullsized":"2024-09-20T22:53:48.395335+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'9","last_scrub_stamp":"2024-09-20T22:52:18.218644+0000","last_deep_scrub":"261'9","last_deep_scrub_stamp":"2024-09-20T22:46:50.974735+0000","last_clean_scrub_stamp":"2024-09-20T22:52:18.218644+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:50.351821+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":728,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-09-20T22:53:48.299963+0000","last_change":"2024-09-20T22:53:10.650499+0000","last_active":"2024-09-20T22:53:48.299963+0000","last_peered":"2024-09-20T22:53:48.299963+0000","last_clean":"2024-09-20T22:53:48.299963+0000","last_became_active":"2024-09-20T22:51:20.296204+0000","last_became_peered":"2024-09-20T22:51:20.296204+0000","last_unstale":"2024-09-20T22:53:48.299963+0000","last_undegraded":"2024-09-20T22:53:48.299963+0000","last_fullsized":"2024-09-20T22:53:48.299963+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:10.650428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:53:10.650428+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:28.093442+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.8","version":"744'934","reported_seq":4059,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-09-20T22:53:48.394818+0000","last_change":"2024-09-20T22:53:48.395280+0000","last_active":"2024-09-20T22:53:48.394818+0000","last_peered":"2024-09-20T22:53:48.394818+0000","last_clean":"2024-09-20T22:53:48.394818+0000","last_became_active":"2024-09-20T22:48:31.224359+0000","last_became_peered":"2024-09-20T22:48:31.224359+0000","last_unstale":"2024-09-20T22:53:48.394818+0000","last_undegraded":"2024-09-20T22:53:48.394818+0000","last_fullsized":"2024-09-20T22:53:48.394818+0000","mapping_epoch":407,"log_start":"426'268","ondisk_log_start":"426'268","created":94,"last_epoch_clean":431,"parent":"0.0","parent_split_bits":0,"last_scrub":"674'746","last_scrub_stamp":"2024-09-20T22:52:35.065256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:52:35.065256+0000","objects_scrubbed":20,"log_size":666,"log_dups_size":445,"ondisk_log_size":666,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:52.715088+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.00089697799999999999,"stat_sum":{"num_bytes":34060348,"num_objects":16,"num_object_clones":13,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":1,"num_read":295,"num_read_kb":300602,"num_write":237,"num_write_kb":78277,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":26027409,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"ed","length":"1"},{"start":"10a","length":"8"},{"start":"113","length":"c"},{"start":"120","length":"1"},{"start":"122","length":"2"},{"start":"125","length":"2"},{"start":"129","length":"1"},{"start":"12b","length":"1"},{"start":"12d","length":"1"},{"start":"131","length":"1"},{"start":"133","length":"6"},{"start":"13f","length":"2"},{"start":"143","length":"2"},{"start":"146","length":"2"},{"start":"14b","length":"1"},{"start":"14d","length":"1"},{"start":"150","length":"1"}]},{"pgid":"4.b","version":"625'15","reported_seq":806,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-09-20T22:53:48.300589+0000","last_change":"2024-09-20T22:53:30.976691+0000","last_active":"2024-09-20T22:53:48.300589+0000","last_peered":"2024-09-20T22:53:48.300589+0000","last_clean":"2024-09-20T22:53:48.300589+0000","last_became_active":"2024-09-20T22:51:20.273094+0000","last_became_peered":"2024-09-20T22:51:20.273094+0000","last_unstale":"2024-09-20T22:53:48.300589+0000","last_undegraded":"2024-09-20T22:53:48.300589+0000","last_fullsized":"2024-09-20T22:53:48.300589+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'15","last_scrub_stamp":"2024-09-20T22:53:30.976566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:53:30.976566+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:51.015430+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":1408,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-09-20T22:53:48.395013+0000","last_change":"2024-09-20T22:51:12.755628+0000","last_active":"2024-09-20T22:53:48.395013+0000","last_peered":"2024-09-20T22:53:48.395013+0000","last_clean":"2024-09-20T22:53:48.395013+0000","last_became_active":"2024-09-20T22:41:38.155953+0000","last_became_peered":"2024-09-20T22:41:38.155953+0000","last_unstale":"2024-09-20T22:53:48.395013+0000","last_undegraded":"2024-09-20T22:53:48.395013+0000","last_fullsized":"2024-09-20T22:53:48.395013+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:12.755526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:51:12.755526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:48.982637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":1439,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-09-20T22:53:48.395575+0000","last_change":"2024-09-20T22:52:42.998852+0000","last_active":"2024-09-20T22:53:48.395575+0000","last_peered":"2024-09-20T22:53:48.395575+0000","last_clean":"2024-09-20T22:53:48.395575+0000","last_became_active":"2024-09-20T22:48:05.240613+0000","last_became_peered":"2024-09-20T22:48:05.240613+0000","last_unstale":"2024-09-20T22:53:48.395575+0000","last_undegraded":"2024-09-20T22:53:48.395575+0000","last_fullsized":"2024-09-20T22:53:48.395575+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:42.998756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:52:42.998756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:11.946882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":747,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-09-20T22:53:48.299220+0000","last_change":"2024-09-20T22:52:16.706497+0000","last_active":"2024-09-20T22:53:48.299220+0000","last_peered":"2024-09-20T22:53:48.299220+0000","last_clean":"2024-09-20T22:53:48.299220+0000","last_became_active":"2024-09-20T22:47:49.219486+0000","last_became_peered":"2024-09-20T22:47:49.219486+0000","last_unstale":"2024-09-20T22:53:48.299220+0000","last_undegraded":"2024-09-20T22:53:48.299220+0000","last_fullsized":"2024-09-20T22:53:48.299220+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:16.706429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:52:16.706429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"744'1255","reported_seq":4593,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-09-20T22:53:48.299909+0000","last_change":"2024-09-20T22:53:48.300231+0000","last_active":"2024-09-20T22:53:48.299909+0000","last_peered":"2024-09-20T22:53:48.299909+0000","last_clean":"2024-09-20T22:53:48.299909+0000","last_became_active":"2024-09-20T22:51:23.270962+0000","last_became_peered":"2024-09-20T22:51:23.270962+0000","last_unstale":"2024-09-20T22:53:48.299909+0000","last_undegraded":"2024-09-20T22:53:48.299909+0000","last_fullsized":"2024-09-20T22:53:48.299909+0000","mapping_epoch":602,"log_start":"423'280","ondisk_log_start":"423'280","created":94,"last_epoch_clean":603,"parent":"0.0","parent_split_bits":0,"last_scrub":"674'1040","last_scrub_stamp":"2024-09-20T22:52:34.647394+0000","last_deep_scrub":"590'828","last_deep_scrub_stamp":"2024-09-20T22:51:10.909364+0000","last_clean_scrub_stamp":"2024-09-20T22:52:34.647394+0000","objects_scrubbed":29,"log_size":975,"log_dups_size":649,"ondisk_log_size":975,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.000708174,"stat_sum":{"num_bytes":46921088,"num_objects":18,"num_object_clones":13,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":1,"num_read":446,"num_read_kb":453232,"num_write":333,"num_write_kb":124705,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":91020358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"ed","length":"1"},{"start":"10a","length":"8"},{"start":"113","length":"c"},{"start":"120","length":"1"},{"start":"122","length":"2"},{"start":"125","length":"2"},{"start":"129","length":"1"},{"start":"12b","length":"1"},{"start":"12d","length":"1"},{"start":"131","length":"1"},{"start":"133","length":"6"},{"start":"13f","length":"2"},{"start":"143","length":"2"},{"start":"146","length":"2"},{"start":"14b","length":"1"},{"start":"14d","length":"1"},{"start":"150","length":"1"}]},{"pgid":"4.a","version":"625'44","reported_seq":1509,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-09-20T22:53:48.330100+0000","last_change":"2024-09-20T22:52:14.038882+0000","last_active":"2024-09-20T22:53:48.330100+0000","last_peered":"2024-09-20T22:53:48.330100+0000","last_clean":"2024-09-20T22:53:48.330100+0000","last_became_active":"2024-09-20T22:51:20.287620+0000","last_became_peered":"2024-09-20T22:51:20.287620+0000","last_unstale":"2024-09-20T22:53:48.330100+0000","last_undegraded":"2024-09-20T22:53:48.330100+0000","last_fullsized":"2024-09-20T22:53:48.330100+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'44","last_scrub_stamp":"2024-09-20T22:52:14.038788+0000","last_deep_scrub":"113'28","last_deep_scrub_stamp":"2024-09-20T22:43:12.725900+0000","last_clean_scrub_stamp":"2024-09-20T22:52:14.038788+0000","objects_scrubbed":9,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:49.921537+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":1414,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-09-20T22:53:48.344738+0000","last_change":"2024-09-20T22:53:27.585334+0000","last_active":"2024-09-20T22:53:48.344738+0000","last_peered":"2024-09-20T22:53:48.344738+0000","last_clean":"2024-09-20T22:53:48.344738+0000","last_became_active":"2024-09-20T22:41:38.161524+0000","last_became_peered":"2024-09-20T22:41:38.161524+0000","last_unstale":"2024-09-20T22:53:48.344738+0000","last_undegraded":"2024-09-20T22:53:48.344738+0000","last_fullsized":"2024-09-20T22:53:48.344738+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:27.585262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:53:27.585262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:52.375797+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":1427,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-09-20T22:53:48.394966+0000","last_change":"2024-09-20T22:52:40.004730+0000","last_active":"2024-09-20T22:53:48.394966+0000","last_peered":"2024-09-20T22:53:48.394966+0000","last_clean":"2024-09-20T22:53:48.394966+0000","last_became_active":"2024-09-20T22:47:49.223220+0000","last_became_peered":"2024-09-20T22:47:49.223220+0000","last_unstale":"2024-09-20T22:53:48.394966+0000","last_undegraded":"2024-09-20T22:53:48.394966+0000","last_fullsized":"2024-09-20T22:53:48.394966+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:40.004661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:52:40.004661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:04.999859+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":1402,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-09-20T22:53:48.343853+0000","last_change":"2024-09-20T22:52:01.643899+0000","last_active":"2024-09-20T22:53:48.343853+0000","last_peered":"2024-09-20T22:53:48.343853+0000","last_clean":"2024-09-20T22:53:48.343853+0000","last_became_active":"2024-09-20T22:41:44.199395+0000","last_became_peered":"2024-09-20T22:41:44.199395+0000","last_unstale":"2024-09-20T22:53:48.343853+0000","last_undegraded":"2024-09-20T22:53:48.343853+0000","last_fullsized":"2024-09-20T22:53:48.343853+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:01.643803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:47:18.329209+0000","last_clean_scrub_stamp":"2024-09-20T22:52:01.643803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:51.391625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.a","version":"748'894","reported_seq":4079,"reported_epoch":748,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T22:53:48.412772+0000","last_change":"2024-09-20T22:53:48.412772+0000","last_active":"2024-09-20T22:53:48.412772+0000","last_peered":"2024-09-20T22:53:48.412772+0000","last_clean":"2024-09-20T22:53:48.412772+0000","last_became_active":"2024-09-20T22:42:52.405085+0000","last_became_peered":"2024-09-20T22:42:52.405085+0000","last_unstale":"2024-09-20T22:53:48.412772+0000","last_undegraded":"2024-09-20T22:53:48.412772+0000","last_fullsized":"2024-09-20T22:53:48.412772+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"505'466","last_scrub_stamp":"2024-09-20T22:49:47.088165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:49:47.088165+0000","objects_scrubbed":13,"log_size":894,"log_dups_size":0,"ondisk_log_size":894,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":9,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:53.341168+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.075725961999999994,"stat_sum":{"num_bytes":37344401,"num_objects":16,"num_object_clones":13,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":364,"num_read_kb":380456,"num_write":280,"num_write_kb":109585,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"ed","length":"1"},{"start":"10a","length":"8"},{"start":"113","length":"c"},{"start":"120","length":"1"},{"start":"122","length":"2"},{"start":"125","length":"2"},{"start":"129","length":"1"},{"start":"12b","length":"1"},{"start":"12d","length":"1"},{"start":"131","length":"1"},{"start":"133","length":"6"}]},{"pgid":"4.9","version":"686'23","reported_seq":776,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-09-20T22:53:48.299417+0000","last_change":"2024-09-20T22:52:46.768898+0000","last_active":"2024-09-20T22:53:48.299417+0000","last_peered":"2024-09-20T22:53:48.299417+0000","last_clean":"2024-09-20T22:53:48.299417+0000","last_became_active":"2024-09-20T22:52:11.286549+0000","last_became_peered":"2024-09-20T22:52:11.286549+0000","last_unstale":"2024-09-20T22:53:48.299417+0000","last_undegraded":"2024-09-20T22:53:48.299417+0000","last_fullsized":"2024-09-20T22:53:48.299417+0000","mapping_epoch":650,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":651,"parent":"0.0","parent_split_bits":0,"last_scrub":"686'23","last_scrub_stamp":"2024-09-20T22:52:46.768812+0000","last_deep_scrub":"686'23","last_deep_scrub_stamp":"2024-09-20T22:52:46.768812+0000","last_clean_scrub_stamp":"2024-09-20T22:52:46.768812+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:50.916744+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":21,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":1452,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-09-20T22:53:48.344426+0000","last_change":"2024-09-20T22:53:35.478160+0000","last_active":"2024-09-20T22:53:48.344426+0000","last_peered":"2024-09-20T22:53:48.344426+0000","last_clean":"2024-09-20T22:53:48.344426+0000","last_became_active":"2024-09-20T22:51:20.293701+0000","last_became_peered":"2024-09-20T22:51:20.293701+0000","last_unstale":"2024-09-20T22:53:48.344426+0000","last_undegraded":"2024-09-20T22:53:48.344426+0000","last_fullsized":"2024-09-20T22:53:48.344426+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:35.478053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:46:52.465940+0000","last_clean_scrub_stamp":"2024-09-20T22:53:35.478053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:46.753808+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":1441,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-09-20T22:53:48.394705+0000","last_change":"2024-09-20T22:53:07.104977+0000","last_active":"2024-09-20T22:53:48.394705+0000","last_peered":"2024-09-20T22:53:48.394705+0000","last_clean":"2024-09-20T22:53:48.394705+0000","last_became_active":"2024-09-20T22:51:20.286626+0000","last_became_peered":"2024-09-20T22:51:20.286626+0000","last_unstale":"2024-09-20T22:53:48.394705+0000","last_undegraded":"2024-09-20T22:53:48.394705+0000","last_fullsized":"2024-09-20T22:53:48.394705+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:07.104891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:53:07.104891+0000","last_clean_scrub_stamp":"2024-09-20T22:53:07.104891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:17.124586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":1393,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-09-20T22:53:48.335356+0000","last_change":"2024-09-20T22:51:20.279408+0000","last_active":"2024-09-20T22:53:48.335356+0000","last_peered":"2024-09-20T22:53:48.335356+0000","last_clean":"2024-09-20T22:53:48.335356+0000","last_became_active":"2024-09-20T22:51:20.279088+0000","last_became_peered":"2024-09-20T22:51:20.279088+0000","last_unstale":"2024-09-20T22:53:48.335356+0000","last_undegraded":"2024-09-20T22:53:48.335356+0000","last_fullsized":"2024-09-20T22:53:48.335356+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:50:39.266034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:55.170379+0000","last_clean_scrub_stamp":"2024-09-20T22:50:39.266034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"747'1171","reported_seq":4511,"reported_epoch":748,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T22:53:48.395944+0000","last_change":"2024-09-20T22:53:46.389635+0000","last_active":"2024-09-20T22:53:48.395944+0000","last_peered":"2024-09-20T22:53:48.395944+0000","last_clean":"2024-09-20T22:53:48.395944+0000","last_became_active":"2024-09-20T22:51:23.271244+0000","last_became_peered":"2024-09-20T22:51:23.271244+0000","last_unstale":"2024-09-20T22:53:48.395944+0000","last_undegraded":"2024-09-20T22:53:48.395944+0000","last_fullsized":"2024-09-20T22:53:48.395944+0000","mapping_epoch":602,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":603,"parent":"0.0","parent_split_bits":0,"last_scrub":"661'1000","last_scrub_stamp":"2024-09-20T22:52:23.164028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:52:23.164028+0000","objects_scrubbed":23,"log_size":1171,"log_dups_size":0,"ondisk_log_size":1171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":12,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:50.268878+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.95127280599999997,"stat_sum":{"num_bytes":40098527,"num_objects":18,"num_object_clones":13,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":3,"num_read":516,"num_read_kb":586632,"num_write":354,"num_write_kb":123866,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":76,"num_bytes_recovered":39288293,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"ed","length":"1"},{"start":"10a","length":"8"},{"start":"113","length":"c"},{"start":"120","length":"1"},{"start":"122","length":"2"},{"start":"125","length":"2"},{"start":"129","length":"1"},{"start":"12b","length":"1"},{"start":"12d","length":"1"},{"start":"131","length":"1"},{"start":"133","length":"2"},{"start":"136","length":"1"}]},{"pgid":"4.8","version":"625'27","reported_seq":1447,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-09-20T22:53:48.394482+0000","last_change":"2024-09-20T22:51:53.020721+0000","last_active":"2024-09-20T22:53:48.394482+0000","last_peered":"2024-09-20T22:53:48.394482+0000","last_clean":"2024-09-20T22:53:48.394482+0000","last_became_active":"2024-09-20T22:41:40.252113+0000","last_became_peered":"2024-09-20T22:41:40.252113+0000","last_unstale":"2024-09-20T22:53:48.394482+0000","last_undegraded":"2024-09-20T22:53:48.394482+0000","last_fullsized":"2024-09-20T22:53:48.394482+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'27","last_scrub_stamp":"2024-09-20T22:51:53.020628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:51:53.020628+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:48.982759+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'2","reported_seq":1477,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-09-20T22:53:48.344663+0000","last_change":"2024-09-20T22:53:33.526131+0000","last_active":"2024-09-20T22:53:48.344663+0000","last_peered":"2024-09-20T22:53:48.344663+0000","last_clean":"2024-09-20T22:53:48.344663+0000","last_became_active":"2024-09-20T22:48:05.238098+0000","last_became_peered":"2024-09-20T22:48:05.238098+0000","last_unstale":"2024-09-20T22:53:48.344663+0000","last_undegraded":"2024-09-20T22:53:48.344663+0000","last_fullsized":"2024-09-20T22:53:48.344663+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-09-20T22:53:33.526049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:53:33.526049+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:58.400288+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"408'10","reported_seq":1589,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-09-20T22:53:48.343889+0000","last_change":"2024-09-20T22:53:24.628363+0000","last_active":"2024-09-20T22:53:48.343889+0000","last_peered":"2024-09-20T22:53:48.343889+0000","last_clean":"2024-09-20T22:53:48.343889+0000","last_became_active":"2024-09-20T22:48:05.239344+0000","last_became_peered":"2024-09-20T22:48:05.239344+0000","last_unstale":"2024-09-20T22:53:48.343889+0000","last_undegraded":"2024-09-20T22:53:48.343889+0000","last_fullsized":"2024-09-20T22:53:48.343889+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"408'10","last_scrub_stamp":"2024-09-20T22:53:24.628269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:53:24.628269+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:33.393650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":1408,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-09-20T22:53:48.394567+0000","last_change":"2024-09-20T22:51:54.066995+0000","last_active":"2024-09-20T22:53:48.394567+0000","last_peered":"2024-09-20T22:53:48.394567+0000","last_clean":"2024-09-20T22:53:48.394567+0000","last_became_active":"2024-09-20T22:47:49.227569+0000","last_became_peered":"2024-09-20T22:47:49.227569+0000","last_unstale":"2024-09-20T22:53:48.394567+0000","last_undegraded":"2024-09-20T22:53:48.394567+0000","last_fullsized":"2024-09-20T22:53:48.394567+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:54.066913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:51:54.066913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:48.982776+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":1443,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-09-20T22:53:48.329877+0000","last_change":"2024-09-20T22:53:11.873951+0000","last_active":"2024-09-20T22:53:48.329877+0000","last_peered":"2024-09-20T22:53:48.329877+0000","last_clean":"2024-09-20T22:53:48.329877+0000","last_became_active":"2024-09-20T22:47:49.213292+0000","last_became_peered":"2024-09-20T22:47:49.213292+0000","last_unstale":"2024-09-20T22:53:48.329877+0000","last_undegraded":"2024-09-20T22:53:48.329877+0000","last_fullsized":"2024-09-20T22:53:48.329877+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:11.873813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:53:11.873813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:18.510382+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"625'12","reported_seq":1443,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-09-20T22:53:48.337251+0000","last_change":"2024-09-20T22:53:26.319275+0000","last_active":"2024-09-20T22:53:48.337251+0000","last_peered":"2024-09-20T22:53:48.337251+0000","last_clean":"2024-09-20T22:53:48.337251+0000","last_became_active":"2024-09-20T22:43:09.116887+0000","last_became_peered":"2024-09-20T22:43:09.116887+0000","last_unstale":"2024-09-20T22:53:48.337251+0000","last_undegraded":"2024-09-20T22:53:48.337251+0000","last_fullsized":"2024-09-20T22:53:48.337251+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'12","last_scrub_stamp":"2024-09-20T22:53:26.319144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:53:26.319144+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:37.907176+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":1414,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-09-20T22:53:48.395541+0000","last_change":"2024-09-20T22:53:03.888115+0000","last_active":"2024-09-20T22:53:48.395541+0000","last_peered":"2024-09-20T22:53:48.395541+0000","last_clean":"2024-09-20T22:53:48.395541+0000","last_became_active":"2024-09-20T22:51:20.293232+0000","last_became_peered":"2024-09-20T22:51:20.293232+0000","last_unstale":"2024-09-20T22:53:48.395541+0000","last_undegraded":"2024-09-20T22:53:48.395541+0000","last_fullsized":"2024-09-20T22:53:48.395541+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:03.888040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:09.705632+0000","last_clean_scrub_stamp":"2024-09-20T22:53:03.888040+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:19.352989+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"74'1","reported_seq":1423,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-09-20T22:53:48.344244+0000","last_change":"2024-09-20T22:52:49.575819+0000","last_active":"2024-09-20T22:53:48.344244+0000","last_peered":"2024-09-20T22:53:48.344244+0000","last_clean":"2024-09-20T22:53:48.344244+0000","last_became_active":"2024-09-20T22:48:05.237882+0000","last_became_peered":"2024-09-20T22:48:05.237882+0000","last_unstale":"2024-09-20T22:53:48.344244+0000","last_undegraded":"2024-09-20T22:53:48.344244+0000","last_fullsized":"2024-09-20T22:53:48.344244+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-09-20T22:52:49.575702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:52:49.575702+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:06.855774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"625'15","reported_seq":766,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-09-20T22:53:48.301094+0000","last_change":"2024-09-20T22:51:52.816826+0000","last_active":"2024-09-20T22:53:48.301094+0000","last_peered":"2024-09-20T22:53:48.301094+0000","last_clean":"2024-09-20T22:53:48.301094+0000","last_became_active":"2024-09-20T22:47:49.231315+0000","last_became_peered":"2024-09-20T22:47:49.231315+0000","last_unstale":"2024-09-20T22:53:48.301094+0000","last_undegraded":"2024-09-20T22:53:48.301094+0000","last_fullsized":"2024-09-20T22:53:48.301094+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'15","last_scrub_stamp":"2024-09-20T22:51:52.816714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:51:52.816714+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":1438,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-09-20T22:53:48.343648+0000","last_change":"2024-09-20T22:53:10.778908+0000","last_active":"2024-09-20T22:53:48.343648+0000","last_peered":"2024-09-20T22:53:48.343648+0000","last_clean":"2024-09-20T22:53:48.343648+0000","last_became_active":"2024-09-20T22:48:05.238801+0000","last_became_peered":"2024-09-20T22:48:05.238801+0000","last_unstale":"2024-09-20T22:53:48.343648+0000","last_undegraded":"2024-09-20T22:53:48.343648+0000","last_fullsized":"2024-09-20T22:53:48.343648+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:10.778836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:53:10.778836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:25.023394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":1405,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-09-20T22:53:48.337036+0000","last_change":"2024-09-20T22:52:20.410227+0000","last_active":"2024-09-20T22:53:48.337036+0000","last_peered":"2024-09-20T22:53:48.337036+0000","last_clean":"2024-09-20T22:53:48.337036+0000","last_became_active":"2024-09-20T22:51:20.279397+0000","last_became_peered":"2024-09-20T22:51:20.279397+0000","last_unstale":"2024-09-20T22:53:48.337036+0000","last_undegraded":"2024-09-20T22:53:48.337036+0000","last_fullsized":"2024-09-20T22:53:48.337036+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:20.410164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:52:20.410164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:50.341745+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":1435,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-09-20T22:53:48.394547+0000","last_change":"2024-09-20T22:52:19.253411+0000","last_active":"2024-09-20T22:53:48.394547+0000","last_peered":"2024-09-20T22:53:48.394547+0000","last_clean":"2024-09-20T22:53:48.394547+0000","last_became_active":"2024-09-20T22:48:05.239710+0000","last_became_peered":"2024-09-20T22:48:05.239710+0000","last_unstale":"2024-09-20T22:53:48.394547+0000","last_undegraded":"2024-09-20T22:53:48.394547+0000","last_fullsized":"2024-09-20T22:53:48.394547+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:19.253343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:52:19.253343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:50.351933+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"625'15","reported_seq":1429,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-09-20T22:53:48.395249+0000","last_change":"2024-09-20T22:51:05.675215+0000","last_active":"2024-09-20T22:53:48.395249+0000","last_peered":"2024-09-20T22:53:48.395249+0000","last_clean":"2024-09-20T22:53:48.395249+0000","last_became_active":"2024-09-20T22:41:40.252486+0000","last_became_peered":"2024-09-20T22:41:40.252486+0000","last_unstale":"2024-09-20T22:53:48.395249+0000","last_undegraded":"2024-09-20T22:53:48.395249+0000","last_fullsized":"2024-09-20T22:53:48.395249+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-09-20T22:51:05.675095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:51:05.675095+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:48.982620+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":755,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-09-20T22:53:48.299524+0000","last_change":"2024-09-20T22:52:13.735451+0000","last_active":"2024-09-20T22:53:48.299524+0000","last_peered":"2024-09-20T22:53:48.299524+0000","last_clean":"2024-09-20T22:53:48.299524+0000","last_became_active":"2024-09-20T22:47:49.229936+0000","last_became_peered":"2024-09-20T22:47:49.229936+0000","last_unstale":"2024-09-20T22:53:48.299524+0000","last_undegraded":"2024-09-20T22:53:48.299524+0000","last_fullsized":"2024-09-20T22:53:48.299524+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:13.735382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:52:13.735382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"74'8","reported_seq":1549,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-09-20T22:53:48.337241+0000","last_change":"2024-09-20T22:50:38.269118+0000","last_active":"2024-09-20T22:53:48.337241+0000","last_peered":"2024-09-20T22:53:48.337241+0000","last_clean":"2024-09-20T22:53:48.337241+0000","last_became_active":"2024-09-20T22:41:42.186352+0000","last_became_peered":"2024-09-20T22:41:42.186352+0000","last_unstale":"2024-09-20T22:53:48.337241+0000","last_undegraded":"2024-09-20T22:53:48.337241+0000","last_fullsized":"2024-09-20T22:53:48.337241+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-09-20T22:50:38.269010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:50:38.269010+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":714,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-09-20T22:53:48.300326+0000","last_change":"2024-09-20T22:52:16.623463+0000","last_active":"2024-09-20T22:53:48.300326+0000","last_peered":"2024-09-20T22:53:48.300326+0000","last_clean":"2024-09-20T22:53:48.300326+0000","last_became_active":"2024-09-20T22:48:05.249656+0000","last_became_peered":"2024-09-20T22:48:05.249656+0000","last_unstale":"2024-09-20T22:53:48.300326+0000","last_undegraded":"2024-09-20T22:53:48.300326+0000","last_fullsized":"2024-09-20T22:53:48.300326+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:16.623368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:52:16.623368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:49.514505+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"625'20","reported_seq":1451,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-09-20T22:53:48.336018+0000","last_change":"2024-09-20T22:53:02.491402+0000","last_active":"2024-09-20T22:53:48.336018+0000","last_peered":"2024-09-20T22:53:48.336018+0000","last_clean":"2024-09-20T22:53:48.336018+0000","last_became_active":"2024-09-20T22:51:20.281859+0000","last_became_peered":"2024-09-20T22:51:20.281859+0000","last_unstale":"2024-09-20T22:53:48.336018+0000","last_undegraded":"2024-09-20T22:53:48.336018+0000","last_fullsized":"2024-09-20T22:53:48.336018+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'20","last_scrub_stamp":"2024-09-20T22:53:02.491320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:53:02.491320+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:10.839410+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":1426,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-09-20T22:53:48.344263+0000","last_change":"2024-09-20T22:52:20.621799+0000","last_active":"2024-09-20T22:53:48.344263+0000","last_peered":"2024-09-20T22:53:48.344263+0000","last_clean":"2024-09-20T22:53:48.344263+0000","last_became_active":"2024-09-20T22:48:05.240231+0000","last_became_peered":"2024-09-20T22:48:05.240231+0000","last_unstale":"2024-09-20T22:53:48.344263+0000","last_undegraded":"2024-09-20T22:53:48.344263+0000","last_fullsized":"2024-09-20T22:53:48.344263+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:20.621718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:52:20.621718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:51.391662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"600'11","reported_seq":1597,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-09-20T22:53:48.395036+0000","last_change":"2024-09-20T22:52:36.264515+0000","last_active":"2024-09-20T22:53:48.395036+0000","last_peered":"2024-09-20T22:53:48.395036+0000","last_clean":"2024-09-20T22:53:48.395036+0000","last_became_active":"2024-09-20T22:51:20.283393+0000","last_became_peered":"2024-09-20T22:51:20.283393+0000","last_unstale":"2024-09-20T22:53:48.395036+0000","last_undegraded":"2024-09-20T22:53:48.395036+0000","last_fullsized":"2024-09-20T22:53:48.395036+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"600'11","last_scrub_stamp":"2024-09-20T22:52:36.264410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:52:36.264410+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:52.246125+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":747,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-09-20T22:53:48.300659+0000","last_change":"2024-09-20T22:52:44.834418+0000","last_active":"2024-09-20T22:53:48.300659+0000","last_peered":"2024-09-20T22:53:48.300659+0000","last_clean":"2024-09-20T22:53:48.300659+0000","last_became_active":"2024-09-20T22:47:49.230674+0000","last_became_peered":"2024-09-20T22:47:49.230674+0000","last_unstale":"2024-09-20T22:53:48.300659+0000","last_undegraded":"2024-09-20T22:53:48.300659+0000","last_fullsized":"2024-09-20T22:53:48.300659+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:44.834352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:50:21.853070+0000","last_clean_scrub_stamp":"2024-09-20T22:52:44.834352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:57.888226+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"625'19","reported_seq":756,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-09-20T22:53:48.299612+0000","last_change":"2024-09-20T22:51:20.317969+0000","last_active":"2024-09-20T22:53:48.299612+0000","last_peered":"2024-09-20T22:53:48.299612+0000","last_clean":"2024-09-20T22:53:48.299612+0000","last_became_active":"2024-09-20T22:51:20.289584+0000","last_became_peered":"2024-09-20T22:51:20.289584+0000","last_unstale":"2024-09-20T22:53:48.299612+0000","last_undegraded":"2024-09-20T22:53:48.299612+0000","last_fullsized":"2024-09-20T22:53:48.299612+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-09-20T22:51:07.926382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:51:07.926382+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:49.514366+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":722,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-09-20T22:53:48.299386+0000","last_change":"2024-09-20T22:52:47.717598+0000","last_active":"2024-09-20T22:53:48.299386+0000","last_peered":"2024-09-20T22:53:48.299386+0000","last_clean":"2024-09-20T22:53:48.299386+0000","last_became_active":"2024-09-20T22:48:05.252026+0000","last_became_peered":"2024-09-20T22:48:05.252026+0000","last_unstale":"2024-09-20T22:53:48.299386+0000","last_undegraded":"2024-09-20T22:53:48.299386+0000","last_fullsized":"2024-09-20T22:53:48.299386+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:47.717490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:52:47.717490+0000","last_clean_scrub_stamp":"2024-09-20T22:52:47.717490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:59.413281+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":309,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-09-20T22:53:48.394662+0000","last_change":"2024-09-20T22:51:20.291857+0000","last_active":"2024-09-20T22:53:48.394662+0000","last_peered":"2024-09-20T22:53:48.394662+0000","last_clean":"2024-09-20T22:53:48.394662+0000","last_became_active":"2024-09-20T22:51:20.291264+0000","last_became_peered":"2024-09-20T22:51:20.291264+0000","last_unstale":"2024-09-20T22:53:48.394662+0000","last_undegraded":"2024-09-20T22:53:48.394662+0000","last_fullsized":"2024-09-20T22:53:48.394662+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:06.651167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:51:06.651167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:48.982654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":1407,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-09-20T22:53:48.335197+0000","last_change":"2024-09-20T22:52:42.582474+0000","last_active":"2024-09-20T22:53:48.335197+0000","last_peered":"2024-09-20T22:53:48.335197+0000","last_clean":"2024-09-20T22:53:48.335197+0000","last_became_active":"2024-09-20T22:47:49.216141+0000","last_became_peered":"2024-09-20T22:47:49.216141+0000","last_unstale":"2024-09-20T22:53:48.335197+0000","last_undegraded":"2024-09-20T22:53:48.335197+0000","last_fullsized":"2024-09-20T22:53:48.335197+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:42.582398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:50:20.206984+0000","last_clean_scrub_stamp":"2024-09-20T22:52:42.582398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:57.745331+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"625'19","reported_seq":1061,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-09-20T22:53:48.330220+0000","last_change":"2024-09-20T22:52:40.792541+0000","last_active":"2024-09-20T22:53:48.330220+0000","last_peered":"2024-09-20T22:53:48.330220+0000","last_clean":"2024-09-20T22:53:48.330220+0000","last_became_active":"2024-09-20T22:51:20.283868+0000","last_became_peered":"2024-09-20T22:51:20.283868+0000","last_unstale":"2024-09-20T22:53:48.330220+0000","last_undegraded":"2024-09-20T22:53:48.330220+0000","last_fullsized":"2024-09-20T22:53:48.330220+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'19","last_scrub_stamp":"2024-09-20T22:52:40.792457+0000","last_deep_scrub":"404'13","last_deep_scrub_stamp":"2024-09-20T22:48:04.276282+0000","last_clean_scrub_stamp":"2024-09-20T22:52:40.792457+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:55.252741+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,0],"acting":[6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":1441,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-09-20T22:53:48.344558+0000","last_change":"2024-09-20T22:52:42.505280+0000","last_active":"2024-09-20T22:53:48.344558+0000","last_peered":"2024-09-20T22:53:48.344558+0000","last_clean":"2024-09-20T22:53:48.344558+0000","last_became_active":"2024-09-20T22:48:05.238812+0000","last_became_peered":"2024-09-20T22:48:05.238812+0000","last_unstale":"2024-09-20T22:53:48.344558+0000","last_undegraded":"2024-09-20T22:53:48.344558+0000","last_fullsized":"2024-09-20T22:53:48.344558+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:42.505211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:31.138067+0000","last_clean_scrub_stamp":"2024-09-20T22:52:42.505211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:51.391714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":1417,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-09-20T22:53:48.335448+0000","last_change":"2024-09-20T22:53:34.345146+0000","last_active":"2024-09-20T22:53:48.335448+0000","last_peered":"2024-09-20T22:53:48.335448+0000","last_clean":"2024-09-20T22:53:48.335448+0000","last_became_active":"2024-09-20T22:51:20.288000+0000","last_became_peered":"2024-09-20T22:51:20.288000+0000","last_unstale":"2024-09-20T22:53:48.335448+0000","last_undegraded":"2024-09-20T22:53:48.335448+0000","last_fullsized":"2024-09-20T22:53:48.335448+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:34.345077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:53:34.345077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:58.604576+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":753,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-09-20T22:53:48.298817+0000","last_change":"2024-09-20T22:53:09.890579+0000","last_active":"2024-09-20T22:53:48.298817+0000","last_peered":"2024-09-20T22:53:48.298817+0000","last_clean":"2024-09-20T22:53:48.298817+0000","last_became_active":"2024-09-20T22:51:20.278147+0000","last_became_peered":"2024-09-20T22:51:20.278147+0000","last_unstale":"2024-09-20T22:53:48.298817+0000","last_undegraded":"2024-09-20T22:53:48.298817+0000","last_fullsized":"2024-09-20T22:53:48.298817+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:09.890501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:53:09.890501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:36.773749+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"625'24","reported_seq":1238,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-09-20T22:53:48.394428+0000","last_change":"2024-09-20T22:53:33.089688+0000","last_active":"2024-09-20T22:53:48.394428+0000","last_peered":"2024-09-20T22:53:48.394428+0000","last_clean":"2024-09-20T22:53:48.394428+0000","last_became_active":"2024-09-20T22:44:09.170785+0000","last_became_peered":"2024-09-20T22:44:09.170785+0000","last_unstale":"2024-09-20T22:53:48.394428+0000","last_undegraded":"2024-09-20T22:53:48.394428+0000","last_fullsized":"2024-09-20T22:53:48.394428+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'24","last_scrub_stamp":"2024-09-20T22:53:33.089580+0000","last_deep_scrub":"470'16","last_deep_scrub_stamp":"2024-09-20T22:49:10.659868+0000","last_clean_scrub_stamp":"2024-09-20T22:53:33.089580+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:46.655555+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":1422,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-09-20T22:53:48.395588+0000","last_change":"2024-09-20T22:53:07.945728+0000","last_active":"2024-09-20T22:53:48.395588+0000","last_peered":"2024-09-20T22:53:48.395588+0000","last_clean":"2024-09-20T22:53:48.395588+0000","last_became_active":"2024-09-20T22:51:20.293645+0000","last_became_peered":"2024-09-20T22:51:20.293645+0000","last_unstale":"2024-09-20T22:53:48.395588+0000","last_undegraded":"2024-09-20T22:53:48.395588+0000","last_fullsized":"2024-09-20T22:53:48.395588+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:07.945664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:45:24.765414+0000","last_clean_scrub_stamp":"2024-09-20T22:53:07.945664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:21.914259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":1430,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-09-20T22:53:48.343518+0000","last_change":"2024-09-20T22:53:09.740214+0000","last_active":"2024-09-20T22:53:48.343518+0000","last_peered":"2024-09-20T22:53:48.343518+0000","last_clean":"2024-09-20T22:53:48.343518+0000","last_became_active":"2024-09-20T22:48:05.238334+0000","last_became_peered":"2024-09-20T22:48:05.238334+0000","last_unstale":"2024-09-20T22:53:48.343518+0000","last_undegraded":"2024-09-20T22:53:48.343518+0000","last_fullsized":"2024-09-20T22:53:48.343518+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:09.740145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:50.174861+0000","last_clean_scrub_stamp":"2024-09-20T22:53:09.740145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:33.910113+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":1399,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-09-20T22:53:48.335896+0000","last_change":"2024-09-20T22:51:04.379662+0000","last_active":"2024-09-20T22:53:48.335896+0000","last_peered":"2024-09-20T22:53:48.335896+0000","last_clean":"2024-09-20T22:53:48.335896+0000","last_became_active":"2024-09-20T22:47:49.214487+0000","last_became_peered":"2024-09-20T22:47:49.214487+0000","last_unstale":"2024-09-20T22:53:48.335896+0000","last_undegraded":"2024-09-20T22:53:48.335896+0000","last_fullsized":"2024-09-20T22:53:48.335896+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:04.379558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:51:04.379558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"625'8","reported_seq":1427,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-09-20T22:53:48.335580+0000","last_change":"2024-09-20T22:53:05.489357+0000","last_active":"2024-09-20T22:53:48.335580+0000","last_peered":"2024-09-20T22:53:48.335580+0000","last_clean":"2024-09-20T22:53:48.335580+0000","last_became_active":"2024-09-20T22:51:20.290269+0000","last_became_peered":"2024-09-20T22:51:20.290269+0000","last_unstale":"2024-09-20T22:53:48.335580+0000","last_undegraded":"2024-09-20T22:53:48.335580+0000","last_fullsized":"2024-09-20T22:53:48.335580+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'8","last_scrub_stamp":"2024-09-20T22:53:05.489279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:53:05.489279+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:30.438834+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":755,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-09-20T22:53:48.301814+0000","last_change":"2024-09-20T22:53:07.918178+0000","last_active":"2024-09-20T22:53:48.301814+0000","last_peered":"2024-09-20T22:53:48.301814+0000","last_clean":"2024-09-20T22:53:48.301814+0000","last_became_active":"2024-09-20T22:47:49.231964+0000","last_became_peered":"2024-09-20T22:47:49.231964+0000","last_unstale":"2024-09-20T22:53:48.301814+0000","last_undegraded":"2024-09-20T22:53:48.301814+0000","last_fullsized":"2024-09-20T22:53:48.301814+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:07.918108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:53:07.918108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:20.150259+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":1439,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-09-20T22:53:48.330018+0000","last_change":"2024-09-20T22:52:20.955721+0000","last_active":"2024-09-20T22:53:48.330018+0000","last_peered":"2024-09-20T22:53:48.330018+0000","last_clean":"2024-09-20T22:53:48.330018+0000","last_became_active":"2024-09-20T22:48:05.244659+0000","last_became_peered":"2024-09-20T22:48:05.244659+0000","last_unstale":"2024-09-20T22:53:48.330018+0000","last_undegraded":"2024-09-20T22:53:48.330018+0000","last_fullsized":"2024-09-20T22:53:48.330018+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:20.955598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:52:20.955598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:49.921556+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":1422,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-09-20T22:53:48.343965+0000","last_change":"2024-09-20T22:53:05.612142+0000","last_active":"2024-09-20T22:53:48.343965+0000","last_peered":"2024-09-20T22:53:48.343965+0000","last_clean":"2024-09-20T22:53:48.343965+0000","last_became_active":"2024-09-20T22:48:05.237765+0000","last_became_peered":"2024-09-20T22:48:05.237765+0000","last_unstale":"2024-09-20T22:53:48.343965+0000","last_undegraded":"2024-09-20T22:53:48.343965+0000","last_fullsized":"2024-09-20T22:53:48.343965+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:05.612030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:10.233487+0000","last_clean_scrub_stamp":"2024-09-20T22:53:05.612030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:09.643188+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":1011,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-09-20T22:53:48.395146+0000","last_change":"2024-09-20T22:51:29.904430+0000","last_active":"2024-09-20T22:53:48.395146+0000","last_peered":"2024-09-20T22:53:48.395146+0000","last_clean":"2024-09-20T22:53:48.395146+0000","last_became_active":"2024-09-20T22:51:20.297739+0000","last_became_peered":"2024-09-20T22:51:20.297739+0000","last_unstale":"2024-09-20T22:53:48.395146+0000","last_undegraded":"2024-09-20T22:53:48.395146+0000","last_fullsized":"2024-09-20T22:53:48.395146+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:29.904355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:51:29.904355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:48.982689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":1439,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-09-20T22:53:48.335360+0000","last_change":"2024-09-20T22:53:31.274457+0000","last_active":"2024-09-20T22:53:48.335360+0000","last_peered":"2024-09-20T22:53:48.335360+0000","last_clean":"2024-09-20T22:53:48.335360+0000","last_became_active":"2024-09-20T22:51:20.274891+0000","last_became_peered":"2024-09-20T22:51:20.274891+0000","last_unstale":"2024-09-20T22:53:48.335360+0000","last_undegraded":"2024-09-20T22:53:48.335360+0000","last_fullsized":"2024-09-20T22:53:48.335360+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:31.274390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:53:31.274390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:36.584363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"625'19","reported_seq":1479,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-09-20T22:53:48.331244+0000","last_change":"2024-09-20T22:53:30.953158+0000","last_active":"2024-09-20T22:53:48.331244+0000","last_peered":"2024-09-20T22:53:48.331244+0000","last_clean":"2024-09-20T22:53:48.331244+0000","last_became_active":"2024-09-20T22:51:20.284937+0000","last_became_peered":"2024-09-20T22:51:20.284937+0000","last_unstale":"2024-09-20T22:53:48.331244+0000","last_undegraded":"2024-09-20T22:53:48.331244+0000","last_fullsized":"2024-09-20T22:53:48.331244+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'19","last_scrub_stamp":"2024-09-20T22:53:30.953067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:53:30.953067+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:59.104815+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":748,"reported_epoch":748,"state":"active+clean","last_fresh":"2024-09-20T22:53:48.300876+0000","last_change":"2024-09-20T22:52:18.698154+0000","last_active":"2024-09-20T22:53:48.300876+0000","last_peered":"2024-09-20T22:53:48.300876+0000","last_clean":"2024-09-20T22:53:48.300876+0000","last_became_active":"2024-09-20T22:47:49.220971+0000","last_became_peered":"2024-09-20T22:47:49.220971+0000","last_unstale":"2024-09-20T22:53:48.300876+0000","last_undegraded":"2024-09-20T22:53:48.300876+0000","last_fullsized":"2024-09-20T22:53:48.300876+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:18.698087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:51.667341+0000","last_clean_scrub_stamp":"2024-09-20T22:52:18.698087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":547088072,"num_objects":245,"num_object_clones":196,"num_object_copies":735,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":245,"num_whiteouts":13,"num_read":5139,"num_read_kb":5357041,"num_write":3726,"num_write_kb":1383651,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":790,"num_bytes_recovered":519843158,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1153060864,"data_stored":1674067131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":11143,"ondisk_log_size":11143,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86,"ondisk_log_size":86,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1109,"num_read_kb":866,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":312,"num_bytes_recovered":7464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":499712,"data_stored":13224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":744,"ondisk_log_size":744,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":1034,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":4931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":720,"num_read_kb":715,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1186,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":384,"ondisk_log_size":384,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070614,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":235240,"kb_used_data":181836,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93509912,"statfs":{"total":95995035648,"available":95754149888,"internally_reserved":0,"allocated":186200064,"data_stored":262181887,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":54655434},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430364,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":217416,"kb_used_data":164012,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93527736,"statfs":{"total":95995035648,"available":95772401664,"internally_reserved":0,"allocated":167948288,"data_stored":223101321,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798692001,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":175460,"kb_used_data":117900,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":93569692,"statfs":{"total":95995035648,"available":95815364608,"internally_reserved":0,"allocated":120729600,"data_stored":173098080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986342,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":120916,"kb_used_data":89724,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93624236,"statfs":{"total":95995035648,"available":95871217664,"internally_reserved":0,"allocated":91877376,"data_stored":132783338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182572,"num_pgs":76,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":336592,"kb_used_data":260980,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93408560,"statfs":{"total":95995035648,"available":95650365440,"internally_reserved":0,"allocated":267243520,"data_stored":379935239,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":77396423},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411505,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":244652,"kb_used_data":195408,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":93500500,"statfs":{"total":95995035648,"available":95744512000,"internally_reserved":0,"allocated":200097792,"data_stored":297438537,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":55834575030,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738555,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":214744,"kb_used_data":165504,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":93530408,"statfs":{"total":95995035648,"available":95775137792,"internally_reserved":0,"allocated":169476096,"data_stored":236960813,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":162512896,"data_stored":232727924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":192888832,"data_stored":292935934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":259674112,"data_stored":375103597,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":84914176,"data_stored":128543719,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":113192960,"data_stored":168274284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":160980992,"data_stored":218868051,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":178896896,"data_stored":257613622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-20T22:53:54.393 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph pg dump --format=json 2024-09-20T22:53:54.397 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:54.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:54.398 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:53:54.465 INFO:tasks.rados.rados.0.smithi084.stdout:3005: done (2 left) 2024-09-20T22:53:54.465 INFO:tasks.rados.rados.0.smithi084.stdout:3006: delete oid 39 current snap is 339 2024-09-20T22:53:54.465 INFO:tasks.rados.rados.0.smithi084.stdout:3001: expect (ObjNum 1012 snap 333 seq_num 1012) 2024-09-20T22:53:54.499 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:53:54.519 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:54.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:54.520 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:53:54.614 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:54 smithi084 ceph-mon[29774]: 4.15 scrub starts 2024-09-20T22:53:54.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:54 smithi084 ceph-mon[29774]: 4.15 scrub ok 2024-09-20T22:53:54.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:54 smithi084 ceph-mon[29774]: from='client.16911 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T22:53:54.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:54 smithi084 ceph-mon[29774]: 6.b scrub starts 2024-09-20T22:53:54.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:54 smithi084 ceph-mon[29774]: 6.b scrub ok 2024-09-20T22:53:54.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:54 smithi084 ceph-mon[40024]: 4.15 scrub starts 2024-09-20T22:53:54.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:54 smithi084 ceph-mon[40024]: 4.15 scrub ok 2024-09-20T22:53:54.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:54 smithi084 ceph-mon[40024]: from='client.16911 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T22:53:54.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:54 smithi084 ceph-mon[40024]: 6.b scrub starts 2024-09-20T22:53:54.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:54 smithi084 ceph-mon[40024]: 6.b scrub ok 2024-09-20T22:53:54.622 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:53:54.643 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:54.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:54.644 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:53:54.679 INFO:tasks.rados.rados.0.smithi084.stdout:3003: finishing rollback tid 0 to smithi08477203-26 2024-09-20T22:53:54.679 INFO:tasks.rados.rados.0.smithi084.stdout:3003: finishing rollback tid 1 to smithi08477203-26 2024-09-20T22:53:54.679 INFO:tasks.rados.rados.0.smithi084.stdout:3003: finishing rollback tid 2 to smithi08477203-26 2024-09-20T22:53:54.679 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 26 v 1822 (ObjNum 762 snap 243 seq_num 762) dirty exists 2024-09-20T22:53:54.679 INFO:tasks.rados.rados.0.smithi084.stdout:3001: done (2 left) 2024-09-20T22:53:54.679 INFO:tasks.rados.rados.0.smithi084.stdout:3003: done (1 left) 2024-09-20T22:53:54.679 INFO:tasks.rados.rados.0.smithi084.stdout:3006: done (0 left) 2024-09-20T22:53:54.679 INFO:tasks.rados.rados.0.smithi084.stdout:3007: snap_remove snap 338 2024-09-20T22:53:54.746 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:53:54.753 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.3 dump_blocked_ops 2024-09-20T22:53:54.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:54 smithi154 ceph-mon[33264]: 4.15 scrub starts 2024-09-20T22:53:54.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:54 smithi154 ceph-mon[33264]: 4.15 scrub ok 2024-09-20T22:53:54.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:54 smithi154 ceph-mon[33264]: from='client.16911 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T22:53:54.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:54 smithi154 ceph-mon[33264]: 6.b scrub starts 2024-09-20T22:53:54.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:54 smithi154 ceph-mon[33264]: 6.b scrub ok 2024-09-20T22:53:54.767 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:54.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:54.768 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:53:54.823 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:53:54.843 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:53:54.868 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:53:54.889 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:54.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:54.890 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:53:54.991 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:53:55.012 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:55.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:55.013 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:53:55.114 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:53:55.136 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:55.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:55.136 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:53:55.237 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:53:55.258 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:55.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:55.259 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:53:55.360 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:53:55.382 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:55.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:55.383 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:53:55.472 INFO:tasks.rados.rados.0.smithi084.stdout:3007: done (0 left) 2024-09-20T22:53:55.472 INFO:tasks.rados.rados.0.smithi084.stdout:3008: rollback oid 34 current snap is 339 2024-09-20T22:53:55.472 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 34 to 330 2024-09-20T22:53:55.472 INFO:tasks.rados.rados.0.smithi084.stdout:3009: write (excl) oid 32 current snap is 339 2024-09-20T22:53:55.472 INFO:tasks.rados.rados.0.smithi084.stdout:3009: seq_num 1041 ranges {696087=558217,1833755=648359,3137806=358281} 2024-09-20T22:53:55.480 INFO:tasks.rados.rados.0.smithi084.stdout:3009: writing smithi08477203-32 from 696087 to 1254304 tid 1 2024-09-20T22:53:55.484 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:53:55.487 INFO:tasks.rados.rados.0.smithi084.stdout:3009: writing smithi08477203-32 from 1833755 to 2482114 tid 2 2024-09-20T22:53:55.492 INFO:tasks.rados.rados.0.smithi084.stdout:3009: writing smithi08477203-32 from 3137806 to 3496087 tid 3 2024-09-20T22:53:55.492 INFO:tasks.rados.rados.0.smithi084.stdout:3010: write oid 14 current snap is 339 2024-09-20T22:53:55.493 INFO:tasks.rados.rados.0.smithi084.stdout:3008: finishing rollback tid 1 to smithi08477203-34 2024-09-20T22:53:55.493 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 34 v 1825 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-09-20T22:53:55.493 INFO:tasks.rados.rados.0.smithi084.stdout:3010: seq_num 1042 ranges {712894=749819,2008590=792017,3281582=231312} 2024-09-20T22:53:55.497 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:53:55.500 INFO:tasks.rados.rados.0.smithi084.stdout:3010: writing smithi08477203-14 from 712894 to 1462713 tid 1 2024-09-20T22:53:55.506 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:55.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:55.507 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:53:55.508 INFO:tasks.rados.rados.0.smithi084.stdout:3010: writing smithi08477203-14 from 2008590 to 2800607 tid 2 2024-09-20T22:53:55.513 INFO:tasks.rados.rados.0.smithi084.stdout:3010: writing smithi08477203-14 from 3281582 to 3512894 tid 3 2024-09-20T22:53:55.513 INFO:tasks.rados.rados.0.smithi084.stdout:3008: done (2 left) 2024-09-20T22:53:55.513 INFO:tasks.rados.rados.0.smithi084.stdout:3011: copy_from oid 41 from oid 21 current snap is 339 2024-09-20T22:53:55.513 INFO:tasks.rados.rados.0.smithi084.stdout:3009: finishing write tid 1 to smithi08477203-32 2024-09-20T22:53:55.513 INFO:tasks.rados.rados.0.smithi084.stdout:3009: finishing write tid 2 to smithi08477203-32 2024-09-20T22:53:55.513 INFO:tasks.rados.rados.0.smithi084.stdout:3009: finishing write tid 3 to smithi08477203-32 2024-09-20T22:53:55.513 INFO:tasks.rados.rados.0.smithi084.stdout:3009: finishing write tid 4 to smithi08477203-32 2024-09-20T22:53:55.513 INFO:tasks.rados.rados.0.smithi084.stdout:3009: finishing write tid 5 to smithi08477203-32 2024-09-20T22:53:55.513 INFO:tasks.rados.rados.0.smithi084.stdout:3009: oid 32 updating version 0 to 1834 2024-09-20T22:53:55.513 INFO:tasks.rados.rados.0.smithi084.stdout:3009: oid 32 updating version 1834 to 1837 2024-09-20T22:53:55.513 INFO:tasks.rados.rados.0.smithi084.stdout:3009: oid 32 version 1837 is already newer than 1835 2024-09-20T22:53:55.513 INFO:tasks.rados.rados.0.smithi084.stdout:3009: oid 32 version 1837 is already newer than 1836 2024-09-20T22:53:55.513 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 32 v 1837 (ObjNum 1041 snap 339 seq_num 1041) dirty exists 2024-09-20T22:53:55.514 INFO:tasks.rados.rados.0.smithi084.stdout:3009: left oid 32 (ObjNum 1041 snap 339 seq_num 1041) 2024-09-20T22:53:55.514 INFO:tasks.rados.rados.0.smithi084.stdout:3010: finishing write tid 1 to smithi08477203-14 2024-09-20T22:53:55.514 INFO:tasks.rados.rados.0.smithi084.stdout:3010: finishing write tid 2 to smithi08477203-14 2024-09-20T22:53:55.514 INFO:tasks.rados.rados.0.smithi084.stdout:3009: done (2 left) 2024-09-20T22:53:55.514 INFO:tasks.rados.rados.0.smithi084.stdout:3012: snap_create 2024-09-20T22:53:55.515 INFO:tasks.rados.rados.0.smithi084.stdout:3010: finishing write tid 3 to smithi08477203-14 2024-09-20T22:53:55.515 INFO:tasks.rados.rados.0.smithi084.stdout:3010: finishing write tid 4 to smithi08477203-14 2024-09-20T22:53:55.516 INFO:tasks.rados.rados.0.smithi084.stdout:3010: finishing write tid 5 to smithi08477203-14 2024-09-20T22:53:55.516 INFO:tasks.rados.rados.0.smithi084.stdout:3010: oid 14 updating version 0 to 1904 2024-09-20T22:53:55.516 INFO:tasks.rados.rados.0.smithi084.stdout:3010: oid 14 version 1904 is already newer than 1903 2024-09-20T22:53:55.516 INFO:tasks.rados.rados.0.smithi084.stdout:3010: oid 14 updating version 1904 to 1905 2024-09-20T22:53:55.516 INFO:tasks.rados.rados.0.smithi084.stdout:3010: oid 14 updating version 1905 to 1906 2024-09-20T22:53:55.516 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 14 v 1906 (ObjNum 1042 snap 339 seq_num 1042) dirty exists 2024-09-20T22:53:55.516 INFO:tasks.rados.rados.0.smithi084.stdout:3010: left oid 14 (ObjNum 1042 snap 339 seq_num 1042) 2024-09-20T22:53:55.520 INFO:tasks.rados.rados.0.smithi084.stdout:3011: finishing copy_from racing read to smithi08477203-41 2024-09-20T22:53:55.526 INFO:tasks.rados.rados.0.smithi084.stdout:3011: finishing copy_from to smithi08477203-41 2024-09-20T22:53:55.526 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 41 v 1829 (ObjNum 984 snap 319 seq_num 984) dirty exists 2024-09-20T22:53:55.608 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:53:55.611 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:55 smithi084 ceph-mon[40024]: pgmap v1032: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 542 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 16 MiB/s rd, 4.0 MiB/s wr, 24 op/s 2024-09-20T22:53:55.611 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:55 smithi084 ceph-mon[40024]: 5.d scrub starts 2024-09-20T22:53:55.611 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:55 smithi084 ceph-mon[40024]: 5.d scrub ok 2024-09-20T22:53:55.611 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:55 smithi084 ceph-mon[40024]: osdmap e754: 8 total, 8 up, 7 in 2024-09-20T22:53:55.611 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:55 smithi084 ceph-mon[40024]: 7.c scrub starts 2024-09-20T22:53:55.611 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:55 smithi084 ceph-mon[40024]: 7.c scrub ok 2024-09-20T22:53:55.612 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:55 smithi084 ceph-mon[40024]: 7.f scrub starts 2024-09-20T22:53:55.612 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:55 smithi084 ceph-mon[40024]: 7.f scrub ok 2024-09-20T22:53:55.612 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:55 smithi084 ceph-mon[29774]: pgmap v1032: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 542 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 16 MiB/s rd, 4.0 MiB/s wr, 24 op/s 2024-09-20T22:53:55.612 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:55 smithi084 ceph-mon[29774]: 5.d scrub starts 2024-09-20T22:53:55.612 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:55 smithi084 ceph-mon[29774]: 5.d scrub ok 2024-09-20T22:53:55.612 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:55 smithi084 ceph-mon[29774]: osdmap e754: 8 total, 8 up, 7 in 2024-09-20T22:53:55.612 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:55 smithi084 ceph-mon[29774]: 7.c scrub starts 2024-09-20T22:53:55.612 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:55 smithi084 ceph-mon[29774]: 7.c scrub ok 2024-09-20T22:53:55.612 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:55 smithi084 ceph-mon[29774]: 7.f scrub starts 2024-09-20T22:53:55.612 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:55 smithi084 ceph-mon[29774]: 7.f scrub ok 2024-09-20T22:53:55.629 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:55.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:55.631 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:53:55.731 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:53:55.753 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:55.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:55.754 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:53:55.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:55 smithi154 ceph-mon[33264]: pgmap v1032: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 542 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 16 MiB/s rd, 4.0 MiB/s wr, 24 op/s 2024-09-20T22:53:55.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:55 smithi154 ceph-mon[33264]: 5.d scrub starts 2024-09-20T22:53:55.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:55 smithi154 ceph-mon[33264]: 5.d scrub ok 2024-09-20T22:53:55.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:55 smithi154 ceph-mon[33264]: osdmap e754: 8 total, 8 up, 7 in 2024-09-20T22:53:55.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:55 smithi154 ceph-mon[33264]: 7.c scrub starts 2024-09-20T22:53:55.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:55 smithi154 ceph-mon[33264]: 7.c scrub ok 2024-09-20T22:53:55.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:55 smithi154 ceph-mon[33264]: 7.f scrub starts 2024-09-20T22:53:55.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:55 smithi154 ceph-mon[33264]: 7.f scrub ok 2024-09-20T22:53:55.855 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:53:55.876 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:55.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:55.877 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:53:55.940 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-20T22:53:55.978 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:53:56.005 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:56.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:56.006 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:53:56.107 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:53:56.128 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:56.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:56.130 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:53:56.230 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:53:56.252 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:56.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:56.253 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:53:56.353 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-20T22:53:56.355 INFO:teuthology.orchestra.run.smithi084.stderr:dumped all 2024-09-20T22:53:56.356 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:53:56.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:56.382 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:56.382 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:53:56.476 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is unset 2024-09-20T22:53:56.476 INFO:tasks.rados.rados.0.smithi084.stdout:3010: done (2 left) 2024-09-20T22:53:56.477 INFO:tasks.rados.rados.0.smithi084.stdout:3011: done (1 left) 2024-09-20T22:53:56.477 INFO:tasks.rados.rados.0.smithi084.stdout:3012: done (0 left) 2024-09-20T22:53:56.477 INFO:tasks.rados.rados.0.smithi084.stdout:3013: copy_from oid 4 from oid 20 current snap is 340 2024-09-20T22:53:56.477 INFO:tasks.rados.rados.0.smithi084.stdout:3014: snap_remove snap 285 2024-09-20T22:53:56.481 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-20T22:53:56.481 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2024-09-20T22:53:56.481 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2024-09-20T22:53:56.481 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2024-09-20T22:53:56.481 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-20T22:53:56.484 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:53:56.507 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:56.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:56.508 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:53:56.609 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:53:56.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:56 smithi084 ceph-mon[40024]: 3.c scrub starts 2024-09-20T22:53:56.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:56 smithi084 ceph-mon[40024]: 3.c scrub ok 2024-09-20T22:53:56.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:56 smithi084 ceph-mon[40024]: 3.4 scrub starts 2024-09-20T22:53:56.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:56 smithi084 ceph-mon[40024]: 3.4 scrub ok 2024-09-20T22:53:56.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:56 smithi084 ceph-mon[40024]: osdmap e755: 8 total, 8 up, 7 in 2024-09-20T22:53:56.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:56 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/1450003704' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:53:56.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:56 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:53:56.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:56 smithi084 ceph-mon[40024]: 3.13 scrub starts 2024-09-20T22:53:56.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:56 smithi084 ceph-mon[40024]: 3.13 scrub ok 2024-09-20T22:53:56.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:56 smithi084 ceph-mon[29774]: 3.c scrub starts 2024-09-20T22:53:56.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:56 smithi084 ceph-mon[29774]: 3.c scrub ok 2024-09-20T22:53:56.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:56 smithi084 ceph-mon[29774]: 3.4 scrub starts 2024-09-20T22:53:56.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:56 smithi084 ceph-mon[29774]: 3.4 scrub ok 2024-09-20T22:53:56.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:56 smithi084 ceph-mon[29774]: osdmap e755: 8 total, 8 up, 7 in 2024-09-20T22:53:56.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:56 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/1450003704' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:53:56.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:56 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:53:56.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:56 smithi084 ceph-mon[29774]: 3.13 scrub starts 2024-09-20T22:53:56.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:56 smithi084 ceph-mon[29774]: 3.13 scrub ok 2024-09-20T22:53:56.630 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:56.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:56.631 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:53:56.731 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:53:56.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:56.754 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:56.755 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:53:56.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:56 smithi154 ceph-mon[33264]: 3.c scrub starts 2024-09-20T22:53:56.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:56 smithi154 ceph-mon[33264]: 3.c scrub ok 2024-09-20T22:53:56.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:56 smithi154 ceph-mon[33264]: 3.4 scrub starts 2024-09-20T22:53:56.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:56 smithi154 ceph-mon[33264]: 3.4 scrub ok 2024-09-20T22:53:56.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:56 smithi154 ceph-mon[33264]: osdmap e755: 8 total, 8 up, 7 in 2024-09-20T22:53:56.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:56 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/1450003704' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:53:56.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:56 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:53:56.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:56 smithi154 ceph-mon[33264]: 3.13 scrub starts 2024-09-20T22:53:56.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:56 smithi154 ceph-mon[33264]: 3.13 scrub ok 2024-09-20T22:53:56.855 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:53:56.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:56.878 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:56.879 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:53:56.979 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:53:57.001 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:57.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:57.002 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:53:57.103 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:53:57.125 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:57.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:57.126 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:53:57.226 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:53:57.249 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:57.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:57.250 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:53:57.351 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:53:57.364 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.3 dump_historic_ops 2024-09-20T22:53:57.372 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:57.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:57.373 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:53:57.474 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:53:57.483 INFO:tasks.rados.rados.0.smithi084.stdout:3014: done (1 left) 2024-09-20T22:53:57.483 INFO:tasks.rados.rados.0.smithi084.stdout:3015: snap_create 2024-09-20T22:53:57.484 INFO:tasks.rados.rados.0.smithi084.stdout:3013: finishing copy_from racing read to smithi08477203-4 2024-09-20T22:53:57.484 INFO:tasks.rados.rados.0.smithi084.stdout:3013: finishing copy_from to smithi08477203-4 2024-09-20T22:53:57.484 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 4 v 1253 (ObjNum 964 snap 314 seq_num 964) dirty exists 2024-09-20T22:53:57.503 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:57.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:57.504 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:53:57.605 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:53:57.614 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:57 smithi084 ceph-mon[40024]: pgmap v1035: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 542 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 16 MiB/s rd, 4.0 MiB/s wr, 24 op/s 2024-09-20T22:53:57.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:57 smithi084 ceph-mon[40024]: 5.12 scrub starts 2024-09-20T22:53:57.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:57 smithi084 ceph-mon[40024]: 5.12 scrub ok 2024-09-20T22:53:57.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:57 smithi084 ceph-mon[40024]: from='client.26822 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T22:53:57.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:57 smithi084 ceph-mon[40024]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T22:53:57.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:57 smithi084 ceph-mon[40024]: Cluster is now healthy 2024-09-20T22:53:57.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:57 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T22:53:57.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:57 smithi084 ceph-mon[40024]: osdmap e756: 8 total, 8 up, 7 in 2024-09-20T22:53:57.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:57 smithi084 ceph-mon[40024]: 1.7 scrub starts 2024-09-20T22:53:57.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:57 smithi084 ceph-mon[40024]: 1.7 scrub ok 2024-09-20T22:53:57.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:57 smithi084 ceph-mon[29774]: pgmap v1035: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 542 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 16 MiB/s rd, 4.0 MiB/s wr, 24 op/s 2024-09-20T22:53:57.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:57 smithi084 ceph-mon[29774]: 5.12 scrub starts 2024-09-20T22:53:57.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:57 smithi084 ceph-mon[29774]: 5.12 scrub ok 2024-09-20T22:53:57.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:57 smithi084 ceph-mon[29774]: from='client.26822 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T22:53:57.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:57 smithi084 ceph-mon[29774]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T22:53:57.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:57 smithi084 ceph-mon[29774]: Cluster is now healthy 2024-09-20T22:53:57.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:57 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T22:53:57.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:57 smithi084 ceph-mon[29774]: osdmap e756: 8 total, 8 up, 7 in 2024-09-20T22:53:57.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:57 smithi084 ceph-mon[29774]: 1.7 scrub starts 2024-09-20T22:53:57.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:57 smithi084 ceph-mon[29774]: 1.7 scrub ok 2024-09-20T22:53:57.626 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:57.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:57.627 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:53:57.728 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:53:57.749 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:57.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:57.750 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:53:57.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:57 smithi154 ceph-mon[33264]: pgmap v1035: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 542 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 16 MiB/s rd, 4.0 MiB/s wr, 24 op/s 2024-09-20T22:53:57.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:57 smithi154 ceph-mon[33264]: 5.12 scrub starts 2024-09-20T22:53:57.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:57 smithi154 ceph-mon[33264]: 5.12 scrub ok 2024-09-20T22:53:57.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:57 smithi154 ceph-mon[33264]: from='client.26822 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T22:53:57.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:57 smithi154 ceph-mon[33264]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T22:53:57.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:57 smithi154 ceph-mon[33264]: Cluster is now healthy 2024-09-20T22:53:57.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:57 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T22:53:57.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:57 smithi154 ceph-mon[33264]: osdmap e756: 8 total, 8 up, 7 in 2024-09-20T22:53:57.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:57 smithi154 ceph-mon[33264]: 1.7 scrub starts 2024-09-20T22:53:57.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:57 smithi154 ceph-mon[33264]: 1.7 scrub ok 2024-09-20T22:53:57.851 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:53:57.873 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:57.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:57.873 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:53:57.974 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:53:57.995 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:57.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:57.996 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:53:58.097 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:53:58.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:58.119 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:58.119 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:53:58.220 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:53:58.243 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:58.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:58.245 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:53:58.305 INFO:tasks.rados.rados.0.smithi084.stdout:3013: done (1 left) 2024-09-20T22:53:58.306 INFO:tasks.rados.rados.0.smithi084.stdout:3015: done (0 left) 2024-09-20T22:53:58.306 INFO:tasks.rados.rados.0.smithi084.stdout:3016: read oid 3 snap 335 2024-09-20T22:53:58.306 INFO:tasks.rados.rados.0.smithi084.stdout:3016: expect (ObjNum 1017 snap 333 seq_num 1017) 2024-09-20T22:53:58.306 INFO:tasks.rados.rados.0.smithi084.stdout:3017: rollback oid 6 current snap is 341 2024-09-20T22:53:58.306 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 6 to 339 2024-09-20T22:53:58.306 INFO:tasks.rados.rados.0.smithi084.stdout:3018: write (excl) oid 11 current snap is 341 2024-09-20T22:53:58.306 INFO:tasks.rados.rados.0.smithi084.stdout:3018: seq_num 1043 ranges {729701=425068,1667072=652028,2909005=485180} 2024-09-20T22:53:58.312 INFO:tasks.rados.rados.0.smithi084.stdout:3018: writing smithi08477203-11 from 729701 to 1154769 tid 1 2024-09-20T22:53:58.318 INFO:tasks.rados.rados.0.smithi084.stdout:3018: writing smithi08477203-11 from 1667072 to 2319100 tid 2 2024-09-20T22:53:58.325 INFO:tasks.rados.rados.0.smithi084.stdout:3018: writing smithi08477203-11 from 2909005 to 3394185 tid 3 2024-09-20T22:53:58.325 INFO:tasks.rados.rados.0.smithi084.stdout:3019: snap_create 2024-09-20T22:53:58.325 INFO:tasks.rados.rados.0.smithi084.stdout:3017: finishing rollback tid 0 to smithi08477203-6 2024-09-20T22:53:58.325 INFO:tasks.rados.rados.0.smithi084.stdout:3017: finishing rollback tid 1 to smithi08477203-6 2024-09-20T22:53:58.325 INFO:tasks.rados.rados.0.smithi084.stdout:3017: finishing rollback tid 2 to smithi08477203-6 2024-09-20T22:53:58.325 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 6 v 814 (ObjNum 1035 snap 338 seq_num 1035) dirty exists 2024-09-20T22:53:58.325 INFO:tasks.rados.rados.0.smithi084.stdout:3016: expect (ObjNum 1017 snap 333 seq_num 1017) 2024-09-20T22:53:58.345 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:53:58.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:58.367 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:58.367 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:53:58.468 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:53:58.481 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:53:58 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: ::ffff:172.21.15.154 - - [20/Sep/2024:22:53:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-20T22:53:58.492 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:58.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:58.493 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:53:58.506 INFO:tasks.rados.rados.0.smithi084.stdout:3018: finishing write tid 1 to smithi08477203-11 2024-09-20T22:53:58.506 INFO:tasks.rados.rados.0.smithi084.stdout:3018: finishing write tid 2 to smithi08477203-11 2024-09-20T22:53:58.506 INFO:tasks.rados.rados.0.smithi084.stdout:3018: finishing write tid 3 to smithi08477203-11 2024-09-20T22:53:58.506 INFO:tasks.rados.rados.0.smithi084.stdout:3018: finishing write tid 4 to smithi08477203-11 2024-09-20T22:53:58.506 INFO:tasks.rados.rados.0.smithi084.stdout:3018: finishing write tid 5 to smithi08477203-11 2024-09-20T22:53:58.506 INFO:tasks.rados.rados.0.smithi084.stdout:3018: oid 11 updating version 0 to 959 2024-09-20T22:53:58.506 INFO:tasks.rados.rados.0.smithi084.stdout:3018: oid 11 version 959 is already newer than 958 2024-09-20T22:53:58.506 INFO:tasks.rados.rados.0.smithi084.stdout:3018: oid 11 version 959 is already newer than 957 2024-09-20T22:53:58.506 INFO:tasks.rados.rados.0.smithi084.stdout:3018: oid 11 version 959 is already newer than 956 2024-09-20T22:53:58.507 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 11 v 959 (ObjNum 1043 snap 341 seq_num 1043) dirty exists 2024-09-20T22:53:58.507 INFO:tasks.rados.rados.0.smithi084.stdout:3018: left oid 11 (ObjNum 1043 snap 341 seq_num 1043) 2024-09-20T22:53:58.566 INFO:teuthology.orchestra.run.smithi084.stdout:{"pg_ready":true,"pg_map":{"version":1035,"stamp":"2024-09-20T22:53:55.807920+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":568023972,"num_objects":454,"num_object_clones":206,"num_object_copies":1362,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":454,"num_whiteouts":14,"num_read":7053,"num_read_kb":5425553,"num_write":4484,"num_write_kb":1400149,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1142,"num_bytes_recovered":520180548,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":12552,"ondisk_log_size":12552,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":474,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":1587160,"kb_used_data":1217532,"kb_used_omap":10,"kb_used_meta":369461,"kb_avail":654628904,"statfs":{"total":671965249536,"available":670339997696,"internally_reserved":0,"allocated":1246752768,"data_stored":1773504540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11157,"internal_metadata":378328171},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":20339393,"num_objects":10,"num_object_clones":10,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":1,"num_read":61,"num_read_kb":66907,"num_write":36,"num_write_kb":16453,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001108"},"pg_stats":[{"pgid":"3.1f","version":"86'2","reported_seq":761,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-20T22:53:53.303385+0000","last_change":"2024-09-20T22:52:21.630419+0000","last_active":"2024-09-20T22:53:53.303385+0000","last_peered":"2024-09-20T22:53:53.303385+0000","last_clean":"2024-09-20T22:53:53.303385+0000","last_became_active":"2024-09-20T22:47:49.222114+0000","last_became_peered":"2024-09-20T22:47:49.222114+0000","last_unstale":"2024-09-20T22:53:53.303385+0000","last_undegraded":"2024-09-20T22:53:53.303385+0000","last_fullsized":"2024-09-20T22:53:53.303385+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'2","last_scrub_stamp":"2024-09-20T22:52:21.630304+0000","last_deep_scrub":"86'2","last_deep_scrub_stamp":"2024-09-20T22:42:39.664421+0000","last_clean_scrub_stamp":"2024-09-20T22:52:21.630304+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":436,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"625'15","reported_seq":759,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-20T22:53:53.301061+0000","last_change":"2024-09-20T22:51:51.813358+0000","last_active":"2024-09-20T22:53:53.301061+0000","last_peered":"2024-09-20T22:53:53.301061+0000","last_clean":"2024-09-20T22:53:53.301061+0000","last_became_active":"2024-09-20T22:51:20.286651+0000","last_became_peered":"2024-09-20T22:51:20.286651+0000","last_unstale":"2024-09-20T22:53:53.301061+0000","last_undegraded":"2024-09-20T22:53:53.301061+0000","last_fullsized":"2024-09-20T22:53:53.301061+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'15","last_scrub_stamp":"2024-09-20T22:51:51.813269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:51:51.813269+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":327,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-20T22:53:53.452926+0000","last_change":"2024-09-20T22:52:41.184388+0000","last_active":"2024-09-20T22:53:53.452926+0000","last_peered":"2024-09-20T22:53:53.452926+0000","last_clean":"2024-09-20T22:53:53.452926+0000","last_became_active":"2024-09-20T22:51:20.291272+0000","last_became_peered":"2024-09-20T22:51:20.291272+0000","last_unstale":"2024-09-20T22:53:53.452926+0000","last_undegraded":"2024-09-20T22:53:53.452926+0000","last_fullsized":"2024-09-20T22:53:53.452926+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:41.184318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:52:41.184318+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:04.632649+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":730,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-20T22:53:53.301164+0000","last_change":"2024-09-20T22:51:36.730087+0000","last_active":"2024-09-20T22:53:53.301164+0000","last_peered":"2024-09-20T22:53:53.301164+0000","last_clean":"2024-09-20T22:53:53.301164+0000","last_became_active":"2024-09-20T22:51:20.295608+0000","last_became_peered":"2024-09-20T22:51:20.295608+0000","last_unstale":"2024-09-20T22:53:53.301164+0000","last_undegraded":"2024-09-20T22:53:53.301164+0000","last_fullsized":"2024-09-20T22:53:53.301164+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:36.730015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:51:36.730015+0000","last_clean_scrub_stamp":"2024-09-20T22:51:36.730015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":1404,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-20T22:53:53.364432+0000","last_change":"2024-09-20T22:51:01.404127+0000","last_active":"2024-09-20T22:53:53.364432+0000","last_peered":"2024-09-20T22:53:53.364432+0000","last_clean":"2024-09-20T22:53:53.364432+0000","last_became_active":"2024-09-20T22:41:44.832099+0000","last_became_peered":"2024-09-20T22:41:44.832099+0000","last_unstale":"2024-09-20T22:53:53.364432+0000","last_undegraded":"2024-09-20T22:53:53.364432+0000","last_fullsized":"2024-09-20T22:53:53.364432+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:01.404063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:51:01.404063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":1410,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-20T22:53:53.365107+0000","last_change":"2024-09-20T22:51:12.418041+0000","last_active":"2024-09-20T22:53:53.365107+0000","last_peered":"2024-09-20T22:53:53.365107+0000","last_clean":"2024-09-20T22:53:53.365107+0000","last_became_active":"2024-09-20T22:41:38.157298+0000","last_became_peered":"2024-09-20T22:41:38.157298+0000","last_unstale":"2024-09-20T22:53:53.365107+0000","last_undegraded":"2024-09-20T22:53:53.365107+0000","last_fullsized":"2024-09-20T22:53:53.365107+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:12.417920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:51:12.417920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"625'27","reported_seq":1492,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-20T22:53:53.364176+0000","last_change":"2024-09-20T22:51:08.441927+0000","last_active":"2024-09-20T22:53:53.364176+0000","last_peered":"2024-09-20T22:53:53.364176+0000","last_clean":"2024-09-20T22:53:53.364176+0000","last_became_active":"2024-09-20T22:47:49.218798+0000","last_became_peered":"2024-09-20T22:47:49.218798+0000","last_unstale":"2024-09-20T22:53:53.364176+0000","last_undegraded":"2024-09-20T22:53:53.364176+0000","last_fullsized":"2024-09-20T22:53:53.364176+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-09-20T22:51:08.441788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:51:08.441788+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":730,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-20T22:53:53.300948+0000","last_change":"2024-09-20T22:52:14.594895+0000","last_active":"2024-09-20T22:53:53.300948+0000","last_peered":"2024-09-20T22:53:53.300948+0000","last_clean":"2024-09-20T22:53:53.300948+0000","last_became_active":"2024-09-20T22:51:20.287060+0000","last_became_peered":"2024-09-20T22:51:20.287060+0000","last_unstale":"2024-09-20T22:53:53.300948+0000","last_undegraded":"2024-09-20T22:53:53.300948+0000","last_fullsized":"2024-09-20T22:53:53.300948+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:14.594823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:52:14.594823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":1429,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-20T22:53:53.454137+0000","last_change":"2024-09-20T22:51:09.684125+0000","last_active":"2024-09-20T22:53:53.454137+0000","last_peered":"2024-09-20T22:53:53.454137+0000","last_clean":"2024-09-20T22:53:53.454137+0000","last_became_active":"2024-09-20T22:48:05.242059+0000","last_became_peered":"2024-09-20T22:48:05.242059+0000","last_unstale":"2024-09-20T22:53:53.454137+0000","last_undegraded":"2024-09-20T22:53:53.454137+0000","last_fullsized":"2024-09-20T22:53:53.454137+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:09.684053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:46:00.999939+0000","last_clean_scrub_stamp":"2024-09-20T22:51:09.684053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"625'15","reported_seq":751,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-20T22:53:53.301185+0000","last_change":"2024-09-20T22:53:51.486324+0000","last_active":"2024-09-20T22:53:53.301185+0000","last_peered":"2024-09-20T22:53:53.301185+0000","last_clean":"2024-09-20T22:53:53.301185+0000","last_became_active":"2024-09-20T22:48:05.233429+0000","last_became_peered":"2024-09-20T22:48:05.233429+0000","last_unstale":"2024-09-20T22:53:53.301185+0000","last_undegraded":"2024-09-20T22:53:53.301185+0000","last_fullsized":"2024-09-20T22:53:53.301185+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'15","last_scrub_stamp":"2024-09-20T22:53:51.486202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:53:51.486202+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:15.973503+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":1421,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-20T22:53:53.453925+0000","last_change":"2024-09-20T22:51:33.977328+0000","last_active":"2024-09-20T22:53:53.453925+0000","last_peered":"2024-09-20T22:53:53.453925+0000","last_clean":"2024-09-20T22:53:53.453925+0000","last_became_active":"2024-09-20T22:47:49.227379+0000","last_became_peered":"2024-09-20T22:47:49.227379+0000","last_unstale":"2024-09-20T22:53:53.453925+0000","last_undegraded":"2024-09-20T22:53:53.453925+0000","last_fullsized":"2024-09-20T22:53:53.453925+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:33.977257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:51:33.977257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":771,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-20T22:53:53.302689+0000","last_change":"2024-09-20T22:53:35.921037+0000","last_active":"2024-09-20T22:53:53.302689+0000","last_peered":"2024-09-20T22:53:53.302689+0000","last_clean":"2024-09-20T22:53:53.302689+0000","last_became_active":"2024-09-20T22:51:20.277465+0000","last_became_peered":"2024-09-20T22:51:20.277465+0000","last_unstale":"2024-09-20T22:53:53.302689+0000","last_undegraded":"2024-09-20T22:53:53.302689+0000","last_fullsized":"2024-09-20T22:53:53.302689+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:35.920968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:49:08.664407+0000","last_clean_scrub_stamp":"2024-09-20T22:53:35.920968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:03.583550+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":1443,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-20T22:53:53.453214+0000","last_change":"2024-09-20T22:53:31.092991+0000","last_active":"2024-09-20T22:53:53.453214+0000","last_peered":"2024-09-20T22:53:53.453214+0000","last_clean":"2024-09-20T22:53:53.453214+0000","last_became_active":"2024-09-20T22:51:20.297787+0000","last_became_peered":"2024-09-20T22:51:20.297787+0000","last_unstale":"2024-09-20T22:53:53.453214+0000","last_undegraded":"2024-09-20T22:53:53.453214+0000","last_fullsized":"2024-09-20T22:53:53.453214+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:31.092915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:53:31.092915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:41.403638+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"625'9","reported_seq":753,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-20T22:53:53.302248+0000","last_change":"2024-09-20T22:53:09.623541+0000","last_active":"2024-09-20T22:53:53.302248+0000","last_peered":"2024-09-20T22:53:53.302248+0000","last_clean":"2024-09-20T22:53:53.302248+0000","last_became_active":"2024-09-20T22:51:20.285795+0000","last_became_peered":"2024-09-20T22:51:20.285795+0000","last_unstale":"2024-09-20T22:53:53.302248+0000","last_undegraded":"2024-09-20T22:53:53.302248+0000","last_fullsized":"2024-09-20T22:53:53.302248+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'9","last_scrub_stamp":"2024-09-20T22:53:09.623458+0000","last_deep_scrub":"181'7","last_deep_scrub_stamp":"2024-09-20T22:48:41.049381+0000","last_clean_scrub_stamp":"2024-09-20T22:53:09.623458+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:34.232945+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":9,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":1448,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-20T22:53:53.314384+0000","last_change":"2024-09-20T22:52:16.705733+0000","last_active":"2024-09-20T22:53:53.314384+0000","last_peered":"2024-09-20T22:53:53.314384+0000","last_clean":"2024-09-20T22:53:53.314384+0000","last_became_active":"2024-09-20T22:51:20.285368+0000","last_became_peered":"2024-09-20T22:51:20.285368+0000","last_unstale":"2024-09-20T22:53:53.314384+0000","last_undegraded":"2024-09-20T22:53:53.314384+0000","last_fullsized":"2024-09-20T22:53:53.314384+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:16.705645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:47:40.361958+0000","last_clean_scrub_stamp":"2024-09-20T22:52:16.705645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":1423,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-20T22:53:53.453067+0000","last_change":"2024-09-20T22:53:52.116314+0000","last_active":"2024-09-20T22:53:53.453067+0000","last_peered":"2024-09-20T22:53:53.453067+0000","last_clean":"2024-09-20T22:53:53.453067+0000","last_became_active":"2024-09-20T22:51:20.279307+0000","last_became_peered":"2024-09-20T22:51:20.279307+0000","last_unstale":"2024-09-20T22:53:53.453067+0000","last_undegraded":"2024-09-20T22:53:53.453067+0000","last_fullsized":"2024-09-20T22:53:53.453067+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:52.116243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:53:52.116243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:12.673739+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":333,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-20T22:53:53.302027+0000","last_change":"2024-09-20T22:53:48.558689+0000","last_active":"2024-09-20T22:53:53.302027+0000","last_peered":"2024-09-20T22:53:53.302027+0000","last_clean":"2024-09-20T22:53:53.302027+0000","last_became_active":"2024-09-20T22:51:20.282393+0000","last_became_peered":"2024-09-20T22:51:20.282393+0000","last_unstale":"2024-09-20T22:53:53.302027+0000","last_undegraded":"2024-09-20T22:53:53.302027+0000","last_fullsized":"2024-09-20T22:53:53.302027+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:48.558619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:53:48.558619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:02.645042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":763,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-20T22:53:53.303166+0000","last_change":"2024-09-20T22:52:42.745105+0000","last_active":"2024-09-20T22:53:53.303166+0000","last_peered":"2024-09-20T22:53:53.303166+0000","last_clean":"2024-09-20T22:53:53.303166+0000","last_became_active":"2024-09-20T22:48:05.233687+0000","last_became_peered":"2024-09-20T22:48:05.233687+0000","last_unstale":"2024-09-20T22:53:53.303166+0000","last_undegraded":"2024-09-20T22:53:53.303166+0000","last_fullsized":"2024-09-20T22:53:53.303166+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:42.745032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:49:53.933601+0000","last_clean_scrub_stamp":"2024-09-20T22:52:42.745032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:59.532706+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"625'34","reported_seq":1508,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-20T22:53:53.362139+0000","last_change":"2024-09-20T22:53:09.119883+0000","last_active":"2024-09-20T22:53:53.362139+0000","last_peered":"2024-09-20T22:53:53.362139+0000","last_clean":"2024-09-20T22:53:53.362139+0000","last_became_active":"2024-09-20T22:51:20.276338+0000","last_became_peered":"2024-09-20T22:51:20.276338+0000","last_unstale":"2024-09-20T22:53:53.362139+0000","last_undegraded":"2024-09-20T22:53:53.362139+0000","last_fullsized":"2024-09-20T22:53:53.362139+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'34","last_scrub_stamp":"2024-09-20T22:53:09.119791+0000","last_deep_scrub":"113'22","last_deep_scrub_stamp":"2024-09-20T22:43:12.191878+0000","last_clean_scrub_stamp":"2024-09-20T22:53:09.119791+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:36.423122+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":1037,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-20T22:53:53.301666+0000","last_change":"2024-09-20T22:52:43.758236+0000","last_active":"2024-09-20T22:53:53.301666+0000","last_peered":"2024-09-20T22:53:53.301666+0000","last_clean":"2024-09-20T22:53:53.301666+0000","last_became_active":"2024-09-20T22:51:20.295564+0000","last_became_peered":"2024-09-20T22:51:20.295564+0000","last_unstale":"2024-09-20T22:53:53.301666+0000","last_undegraded":"2024-09-20T22:53:53.301666+0000","last_fullsized":"2024-09-20T22:53:53.301666+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:43.758162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:50:40.784154+0000","last_clean_scrub_stamp":"2024-09-20T22:52:43.758162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:10.526451+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":1397,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-20T22:53:53.365312+0000","last_change":"2024-09-20T22:51:13.385618+0000","last_active":"2024-09-20T22:53:53.365312+0000","last_peered":"2024-09-20T22:53:53.365312+0000","last_clean":"2024-09-20T22:53:53.365312+0000","last_became_active":"2024-09-20T22:41:44.832131+0000","last_became_peered":"2024-09-20T22:41:44.832131+0000","last_unstale":"2024-09-20T22:53:53.365312+0000","last_undegraded":"2024-09-20T22:53:53.365312+0000","last_fullsized":"2024-09-20T22:53:53.365312+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:13.385520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:45:25.002199+0000","last_clean_scrub_stamp":"2024-09-20T22:51:13.385520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":738,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-20T22:53:53.301298+0000","last_change":"2024-09-20T22:53:04.583753+0000","last_active":"2024-09-20T22:53:53.301298+0000","last_peered":"2024-09-20T22:53:53.301298+0000","last_clean":"2024-09-20T22:53:53.301298+0000","last_became_active":"2024-09-20T22:51:20.284621+0000","last_became_peered":"2024-09-20T22:51:20.284621+0000","last_unstale":"2024-09-20T22:53:53.301298+0000","last_undegraded":"2024-09-20T22:53:53.301298+0000","last_fullsized":"2024-09-20T22:53:53.301298+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:04.583683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:50:41.885844+0000","last_clean_scrub_stamp":"2024-09-20T22:53:04.583683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:27.888402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"625'18","reported_seq":1508,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-20T22:53:53.364740+0000","last_change":"2024-09-20T22:51:20.300828+0000","last_active":"2024-09-20T22:53:53.364740+0000","last_peered":"2024-09-20T22:53:53.364740+0000","last_clean":"2024-09-20T22:53:53.364740+0000","last_became_active":"2024-09-20T22:51:20.278886+0000","last_became_peered":"2024-09-20T22:51:20.278886+0000","last_unstale":"2024-09-20T22:53:53.364740+0000","last_undegraded":"2024-09-20T22:53:53.364740+0000","last_fullsized":"2024-09-20T22:53:53.364740+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-09-20T22:51:03.380934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:51:03.380934+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":732,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-20T22:53:53.301723+0000","last_change":"2024-09-20T22:52:45.738376+0000","last_active":"2024-09-20T22:53:53.301723+0000","last_peered":"2024-09-20T22:53:53.301723+0000","last_clean":"2024-09-20T22:53:53.301723+0000","last_became_active":"2024-09-20T22:48:05.250319+0000","last_became_peered":"2024-09-20T22:48:05.250319+0000","last_unstale":"2024-09-20T22:53:53.301723+0000","last_undegraded":"2024-09-20T22:53:53.301723+0000","last_fullsized":"2024-09-20T22:53:53.301723+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:45.738275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:52:45.738275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:02.329689+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"154'2","reported_seq":1413,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-20T22:53:53.314749+0000","last_change":"2024-09-20T22:52:45.578519+0000","last_active":"2024-09-20T22:53:53.314749+0000","last_peered":"2024-09-20T22:53:53.314749+0000","last_clean":"2024-09-20T22:53:53.314749+0000","last_became_active":"2024-09-20T22:41:44.199822+0000","last_became_peered":"2024-09-20T22:41:44.199822+0000","last_unstale":"2024-09-20T22:53:53.314749+0000","last_undegraded":"2024-09-20T22:53:53.314749+0000","last_fullsized":"2024-09-20T22:53:53.314749+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"154'2","last_scrub_stamp":"2024-09-20T22:52:45.578416+0000","last_deep_scrub":"154'2","last_deep_scrub_stamp":"2024-09-20T22:43:54.134066+0000","last_clean_scrub_stamp":"2024-09-20T22:52:45.578416+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:53.480152+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"67'1","reported_seq":338,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-20T22:53:53.364076+0000","last_change":"2024-09-20T22:53:08.612473+0000","last_active":"2024-09-20T22:53:53.364076+0000","last_peered":"2024-09-20T22:53:53.364076+0000","last_clean":"2024-09-20T22:53:53.364076+0000","last_became_active":"2024-09-20T22:51:20.279978+0000","last_became_peered":"2024-09-20T22:51:20.279978+0000","last_unstale":"2024-09-20T22:53:53.364076+0000","last_undegraded":"2024-09-20T22:53:53.364076+0000","last_fullsized":"2024-09-20T22:53:53.364076+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-09-20T22:53:08.612395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:53:08.612395+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:13.547026+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"625'25","reported_seq":793,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-20T22:53:53.303815+0000","last_change":"2024-09-20T22:53:26.146547+0000","last_active":"2024-09-20T22:53:53.303815+0000","last_peered":"2024-09-20T22:53:53.303815+0000","last_clean":"2024-09-20T22:53:53.303815+0000","last_became_active":"2024-09-20T22:47:49.232181+0000","last_became_peered":"2024-09-20T22:47:49.232181+0000","last_unstale":"2024-09-20T22:53:53.303815+0000","last_undegraded":"2024-09-20T22:53:53.303815+0000","last_fullsized":"2024-09-20T22:53:53.303815+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'25","last_scrub_stamp":"2024-09-20T22:53:26.146466+0000","last_deep_scrub":"151'15","last_deep_scrub_stamp":"2024-09-20T22:43:50.715761+0000","last_clean_scrub_stamp":"2024-09-20T22:53:26.146466+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:51.749507+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"661'11","reported_seq":1616,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-20T22:53:53.456141+0000","last_change":"2024-09-20T22:53:34.944681+0000","last_active":"2024-09-20T22:53:53.456141+0000","last_peered":"2024-09-20T22:53:53.456141+0000","last_clean":"2024-09-20T22:53:53.456141+0000","last_became_active":"2024-09-20T22:48:05.244270+0000","last_became_peered":"2024-09-20T22:48:05.244270+0000","last_unstale":"2024-09-20T22:53:53.456141+0000","last_undegraded":"2024-09-20T22:53:53.456141+0000","last_fullsized":"2024-09-20T22:53:53.456141+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"661'11","last_scrub_stamp":"2024-09-20T22:53:34.944591+0000","last_deep_scrub":"661'11","last_deep_scrub_stamp":"2024-09-20T22:52:21.907943+0000","last_clean_scrub_stamp":"2024-09-20T22:53:34.944591+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:38.596007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.c","version":"752'2380","reported_seq":5557,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-20T22:53:53.456226+0000","last_change":"2024-09-20T22:53:52.476962+0000","last_active":"2024-09-20T22:53:53.456226+0000","last_peered":"2024-09-20T22:53:53.456226+0000","last_clean":"2024-09-20T22:53:53.456226+0000","last_became_active":"2024-09-20T22:42:52.404027+0000","last_became_peered":"2024-09-20T22:42:52.404027+0000","last_unstale":"2024-09-20T22:53:53.456226+0000","last_undegraded":"2024-09-20T22:53:53.456226+0000","last_fullsized":"2024-09-20T22:53:53.456226+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"674'2030","last_scrub_stamp":"2024-09-20T22:52:36.892383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:52:36.892383+0000","objects_scrubbed":44,"log_size":2380,"log_dups_size":0,"ondisk_log_size":2380,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:03.723946+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.017885304000000001,"stat_sum":{"num_bytes":64028596,"num_objects":32,"num_object_clones":25,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":4,"num_read":913,"num_read_kb":974438,"num_write":601,"num_write_kb":232725,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"10e","length":"4"},{"start":"113","length":"c"},{"start":"120","length":"1"},{"start":"122","length":"2"},{"start":"125","length":"2"},{"start":"129","length":"1"},{"start":"12b","length":"3"},{"start":"131","length":"1"},{"start":"133","length":"6"},{"start":"13c","length":"1"},{"start":"13f","length":"2"},{"start":"143","length":"2"},{"start":"146","length":"2"},{"start":"14b","length":"1"},{"start":"14d","length":"1"},{"start":"150","length":"1"}]},{"pgid":"1.a","version":"77'128","reported_seq":1686,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-20T22:53:53.452827+0000","last_change":"2024-09-20T22:53:28.095343+0000","last_active":"2024-09-20T22:53:53.452827+0000","last_peered":"2024-09-20T22:53:53.452827+0000","last_clean":"2024-09-20T22:53:53.452827+0000","last_became_active":"2024-09-20T22:42:52.407791+0000","last_became_peered":"2024-09-20T22:42:52.407791+0000","last_unstale":"2024-09-20T22:53:53.452827+0000","last_undegraded":"2024-09-20T22:53:53.452827+0000","last_fullsized":"2024-09-20T22:53:53.452827+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:53:28.095266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:53:28.095266+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:45.776903+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"625'27","reported_seq":404,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-20T22:53:53.365431+0000","last_change":"2024-09-20T22:52:15.358218+0000","last_active":"2024-09-20T22:53:53.365431+0000","last_peered":"2024-09-20T22:53:53.365431+0000","last_clean":"2024-09-20T22:53:53.365431+0000","last_became_active":"2024-09-20T22:51:20.287083+0000","last_became_peered":"2024-09-20T22:51:20.287083+0000","last_unstale":"2024-09-20T22:53:53.365431+0000","last_undegraded":"2024-09-20T22:53:53.365431+0000","last_fullsized":"2024-09-20T22:53:53.365431+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'27","last_scrub_stamp":"2024-09-20T22:52:15.358100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:52:15.358100+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":1415,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-20T22:53:53.363881+0000","last_change":"2024-09-20T22:51:29.394206+0000","last_active":"2024-09-20T22:53:53.363881+0000","last_peered":"2024-09-20T22:53:53.363881+0000","last_clean":"2024-09-20T22:53:53.363881+0000","last_became_active":"2024-09-20T22:51:20.279439+0000","last_became_peered":"2024-09-20T22:51:20.279439+0000","last_unstale":"2024-09-20T22:53:53.363881+0000","last_undegraded":"2024-09-20T22:53:53.363881+0000","last_fullsized":"2024-09-20T22:53:53.363881+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:29.394097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:49:12.070675+0000","last_clean_scrub_stamp":"2024-09-20T22:51:29.394097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"408'12","reported_seq":798,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-20T22:53:53.302170+0000","last_change":"2024-09-20T22:51:50.785575+0000","last_active":"2024-09-20T22:53:53.302170+0000","last_peered":"2024-09-20T22:53:53.302170+0000","last_clean":"2024-09-20T22:53:53.302170+0000","last_became_active":"2024-09-20T22:48:05.242914+0000","last_became_peered":"2024-09-20T22:48:05.242914+0000","last_unstale":"2024-09-20T22:53:53.302170+0000","last_undegraded":"2024-09-20T22:53:53.302170+0000","last_fullsized":"2024-09-20T22:53:53.302170+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"408'12","last_scrub_stamp":"2024-09-20T22:51:50.785478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:51:50.785478+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":1424,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-20T22:53:53.453126+0000","last_change":"2024-09-20T22:51:31.960415+0000","last_active":"2024-09-20T22:53:53.453126+0000","last_peered":"2024-09-20T22:53:53.453126+0000","last_clean":"2024-09-20T22:53:53.453126+0000","last_became_active":"2024-09-20T22:51:20.280145+0000","last_became_peered":"2024-09-20T22:51:20.280145+0000","last_unstale":"2024-09-20T22:53:53.453126+0000","last_undegraded":"2024-09-20T22:53:53.453126+0000","last_fullsized":"2024-09-20T22:53:53.453126+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:31.960331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:51:31.960331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3","version":"751'349","reported_seq":3479,"reported_epoch":753,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T22:53:53.364354+0000","last_change":"2024-09-20T22:53:52.399283+0000","last_active":"2024-09-20T22:53:53.364354+0000","last_peered":"2024-09-20T22:53:53.364354+0000","last_clean":"2024-09-20T22:53:53.364354+0000","last_became_active":"2024-09-20T22:49:05.221402+0000","last_became_peered":"2024-09-20T22:49:05.221402+0000","last_unstale":"2024-09-20T22:53:53.364354+0000","last_undegraded":"2024-09-20T22:53:53.364354+0000","last_fullsized":"2024-09-20T22:53:53.364354+0000","mapping_epoch":404,"log_start":"456'129","ondisk_log_start":"456'129","created":94,"last_epoch_clean":465,"parent":"0.0","parent_split_bits":0,"last_scrub":"609'243","last_scrub_stamp":"2024-09-20T22:51:31.402013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:51:31.402013+0000","objects_scrubbed":6,"log_size":220,"log_dups_size":149,"ondisk_log_size":220,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":6,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.946585801,"stat_sum":{"num_bytes":15066313,"num_objects":7,"num_object_clones":6,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":103,"num_read_kb":128302,"num_write":78,"num_write_kb":27187,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":20196357,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"10e","length":"4"},{"start":"113","length":"c"},{"start":"120","length":"1"},{"start":"122","length":"2"},{"start":"125","length":"2"},{"start":"129","length":"1"},{"start":"12b","length":"1"},{"start":"12d","length":"1"},{"start":"131","length":"1"},{"start":"133","length":"6"},{"start":"13f","length":"2"},{"start":"143","length":"2"},{"start":"146","length":"1"}]},{"pgid":"1.5","version":"77'128","reported_seq":1591,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-20T22:53:53.302071+0000","last_change":"2024-09-20T22:53:35.470320+0000","last_active":"2024-09-20T22:53:53.302071+0000","last_peered":"2024-09-20T22:53:53.302071+0000","last_clean":"2024-09-20T22:53:53.302071+0000","last_became_active":"2024-09-20T22:48:08.207018+0000","last_became_peered":"2024-09-20T22:48:08.207018+0000","last_unstale":"2024-09-20T22:53:53.302071+0000","last_undegraded":"2024-09-20T22:53:53.302071+0000","last_fullsized":"2024-09-20T22:53:53.302071+0000","mapping_epoch":407,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":408,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:53:35.470248+0000","last_deep_scrub":"77'128","last_deep_scrub_stamp":"2024-09-20T22:45:41.608075+0000","last_clean_scrub_stamp":"2024-09-20T22:53:35.470248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:57.007268+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"625'30","reported_seq":1512,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-20T22:53:53.364003+0000","last_change":"2024-09-20T22:53:48.346424+0000","last_active":"2024-09-20T22:53:53.364003+0000","last_peered":"2024-09-20T22:53:53.364003+0000","last_clean":"2024-09-20T22:53:53.364003+0000","last_became_active":"2024-09-20T22:47:49.217605+0000","last_became_peered":"2024-09-20T22:47:49.217605+0000","last_unstale":"2024-09-20T22:53:53.364003+0000","last_undegraded":"2024-09-20T22:53:53.364003+0000","last_fullsized":"2024-09-20T22:53:53.364003+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'30","last_scrub_stamp":"2024-09-20T22:53:48.346341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:53:48.346341+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:02.270695+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":1429,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-20T22:53:53.365602+0000","last_change":"2024-09-20T22:52:45.537243+0000","last_active":"2024-09-20T22:53:53.365602+0000","last_peered":"2024-09-20T22:53:53.365602+0000","last_clean":"2024-09-20T22:53:53.365602+0000","last_became_active":"2024-09-20T22:47:49.227069+0000","last_became_peered":"2024-09-20T22:47:49.227069+0000","last_unstale":"2024-09-20T22:53:53.365602+0000","last_undegraded":"2024-09-20T22:53:53.365602+0000","last_fullsized":"2024-09-20T22:53:53.365602+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:45.537173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:52:45.537173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:54.445275+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":732,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-20T22:53:53.301259+0000","last_change":"2024-09-20T22:53:26.623524+0000","last_active":"2024-09-20T22:53:53.301259+0000","last_peered":"2024-09-20T22:53:53.301259+0000","last_clean":"2024-09-20T22:53:53.301259+0000","last_became_active":"2024-09-20T22:48:05.230003+0000","last_became_peered":"2024-09-20T22:48:05.230003+0000","last_unstale":"2024-09-20T22:53:53.301259+0000","last_undegraded":"2024-09-20T22:53:53.301259+0000","last_fullsized":"2024-09-20T22:53:53.301259+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:26.623455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:07.947945+0000","last_clean_scrub_stamp":"2024-09-20T22:53:26.623455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:36.274129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":724,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-20T22:53:53.301255+0000","last_change":"2024-09-20T22:51:55.748527+0000","last_active":"2024-09-20T22:53:53.301255+0000","last_peered":"2024-09-20T22:53:53.301255+0000","last_clean":"2024-09-20T22:53:53.301255+0000","last_became_active":"2024-09-20T22:48:05.239917+0000","last_became_peered":"2024-09-20T22:48:05.239917+0000","last_unstale":"2024-09-20T22:53:53.301255+0000","last_undegraded":"2024-09-20T22:53:53.301255+0000","last_fullsized":"2024-09-20T22:53:53.301255+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:55.748425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:53.036601+0000","last_clean_scrub_stamp":"2024-09-20T22:51:55.748425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2","version":"748'802","reported_seq":3280,"reported_epoch":753,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-20T22:53:53.364779+0000","last_change":"2024-09-20T22:53:52.417575+0000","last_active":"2024-09-20T22:53:53.364779+0000","last_peered":"2024-09-20T22:53:53.364779+0000","last_clean":"2024-09-20T22:53:53.364779+0000","last_became_active":"2024-09-20T22:52:23.286461+0000","last_became_peered":"2024-09-20T22:52:23.286461+0000","last_unstale":"2024-09-20T22:53:53.364779+0000","last_undegraded":"2024-09-20T22:53:53.364779+0000","last_fullsized":"2024-09-20T22:53:53.364779+0000","mapping_epoch":662,"log_start":"471'464","ondisk_log_start":"471'464","created":94,"last_epoch_clean":663,"parent":"0.0","parent_split_bits":0,"last_scrub":"282'120","last_scrub_stamp":"2024-09-20T22:46:22.099642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:46:22.099642+0000","objects_scrubbed":1,"log_size":338,"log_dups_size":687,"ondisk_log_size":338,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":37,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.96526718199999995,"stat_sum":{"num_bytes":51331449,"num_objects":20,"num_object_clones":16,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":1,"num_read":414,"num_read_kb":423819,"num_write":266,"num_write_kb":104890,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":144,"num_bytes_recovered":106353268,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"10e","length":"4"},{"start":"113","length":"1"}]},{"pgid":"1.4","version":"77'128","reported_seq":1710,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-20T22:53:53.365053+0000","last_change":"2024-09-20T22:52:02.474080+0000","last_active":"2024-09-20T22:53:53.365053+0000","last_peered":"2024-09-20T22:53:53.365053+0000","last_clean":"2024-09-20T22:53:53.365053+0000","last_became_active":"2024-09-20T22:46:15.181305+0000","last_became_peered":"2024-09-20T22:46:15.181305+0000","last_unstale":"2024-09-20T22:53:53.365053+0000","last_undegraded":"2024-09-20T22:53:53.365053+0000","last_fullsized":"2024-09-20T22:53:53.365053+0000","mapping_epoch":94,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:52:02.473979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:52:02.473979+0000","objects_scrubbed":1,"log_size":0,"log_dups_size":59,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"625'28","reported_seq":761,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-20T22:53:53.301183+0000","last_change":"2024-09-20T22:51:58.712602+0000","last_active":"2024-09-20T22:53:53.301183+0000","last_peered":"2024-09-20T22:53:53.301183+0000","last_clean":"2024-09-20T22:53:53.301183+0000","last_became_active":"2024-09-20T22:48:05.235812+0000","last_became_peered":"2024-09-20T22:48:05.235812+0000","last_unstale":"2024-09-20T22:53:53.301183+0000","last_undegraded":"2024-09-20T22:53:53.301183+0000","last_fullsized":"2024-09-20T22:53:53.301183+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'28","last_scrub_stamp":"2024-09-20T22:51:58.712488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:51:58.712488+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":41,"num_read_kb":34,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"67'1","reported_seq":789,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-20T22:53:53.303556+0000","last_change":"2024-09-20T22:53:32.995298+0000","last_active":"2024-09-20T22:53:53.303556+0000","last_peered":"2024-09-20T22:53:53.303556+0000","last_clean":"2024-09-20T22:53:53.303556+0000","last_became_active":"2024-09-20T22:51:20.283011+0000","last_became_peered":"2024-09-20T22:51:20.283011+0000","last_unstale":"2024-09-20T22:53:53.303556+0000","last_undegraded":"2024-09-20T22:53:53.303556+0000","last_fullsized":"2024-09-20T22:53:53.303556+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-09-20T22:53:32.995211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:53:32.995211+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:59.820181+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"600'13","reported_seq":1609,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-20T22:53:53.364207+0000","last_change":"2024-09-20T22:51:32.448600+0000","last_active":"2024-09-20T22:53:53.364207+0000","last_peered":"2024-09-20T22:53:53.364207+0000","last_clean":"2024-09-20T22:53:53.364207+0000","last_became_active":"2024-09-20T22:51:20.289873+0000","last_became_peered":"2024-09-20T22:51:20.289873+0000","last_unstale":"2024-09-20T22:53:53.364207+0000","last_undegraded":"2024-09-20T22:53:53.364207+0000","last_fullsized":"2024-09-20T22:53:53.364207+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"600'13","last_scrub_stamp":"2024-09-20T22:51:32.448512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:51:32.448512+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":1412,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-20T22:53:53.314662+0000","last_change":"2024-09-20T22:53:02.610452+0000","last_active":"2024-09-20T22:53:53.314662+0000","last_peered":"2024-09-20T22:53:53.314662+0000","last_clean":"2024-09-20T22:53:53.314662+0000","last_became_active":"2024-09-20T22:41:44.831901+0000","last_became_peered":"2024-09-20T22:41:44.831901+0000","last_unstale":"2024-09-20T22:53:53.314662+0000","last_undegraded":"2024-09-20T22:53:53.314662+0000","last_fullsized":"2024-09-20T22:53:53.314662+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:02.610359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:53:02.610359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:32.529803+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1","version":"753'452","reported_seq":3687,"reported_epoch":753,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T22:53:53.452354+0000","last_change":"2024-09-20T22:53:53.452354+0000","last_active":"2024-09-20T22:53:53.452354+0000","last_peered":"2024-09-20T22:53:53.452354+0000","last_clean":"2024-09-20T22:53:53.452354+0000","last_became_active":"2024-09-20T22:49:21.232750+0000","last_became_peered":"2024-09-20T22:49:21.232750+0000","last_unstale":"2024-09-20T22:53:53.452354+0000","last_undegraded":"2024-09-20T22:53:53.452354+0000","last_fullsized":"2024-09-20T22:53:53.452354+0000","mapping_epoch":407,"log_start":"474'259","ondisk_log_start":"474'259","created":94,"last_epoch_clean":481,"parent":"0.0","parent_split_bits":0,"last_scrub":"703'407","last_scrub_stamp":"2024-09-20T22:53:06.625437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:53:06.625437+0000","objects_scrubbed":6,"log_size":193,"log_dups_size":338,"ondisk_log_size":193,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:32.600535+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.150833193,"stat_sum":{"num_bytes":11851465,"num_objects":8,"num_object_clones":7,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":124,"num_read_kb":142534,"num_write":96,"num_write_kb":50330,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":8732217,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"10e","length":"4"},{"start":"113","length":"c"},{"start":"120","length":"1"},{"start":"122","length":"2"},{"start":"125","length":"2"},{"start":"129","length":"1"},{"start":"12b","length":"3"},{"start":"131","length":"1"},{"start":"133","length":"6"},{"start":"13f","length":"2"},{"start":"143","length":"2"},{"start":"146","length":"2"},{"start":"14b","length":"1"},{"start":"14d","length":"1"},{"start":"150","length":"1"}]},{"pgid":"1.7","version":"77'128","reported_seq":1723,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-20T22:53:53.455917+0000","last_change":"2024-09-20T22:52:34.898681+0000","last_active":"2024-09-20T22:53:53.455917+0000","last_peered":"2024-09-20T22:53:53.455917+0000","last_clean":"2024-09-20T22:53:53.455917+0000","last_became_active":"2024-09-20T22:46:03.169596+0000","last_became_peered":"2024-09-20T22:46:03.169596+0000","last_unstale":"2024-09-20T22:53:53.455917+0000","last_undegraded":"2024-09-20T22:53:53.455917+0000","last_fullsized":"2024-09-20T22:53:53.455917+0000","mapping_epoch":94,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:52:34.898595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:52:34.898595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"625'20","reported_seq":388,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-20T22:53:53.453391+0000","last_change":"2024-09-20T22:52:19.125548+0000","last_active":"2024-09-20T22:53:53.453391+0000","last_peered":"2024-09-20T22:53:53.453391+0000","last_clean":"2024-09-20T22:53:53.453391+0000","last_became_active":"2024-09-20T22:51:20.297933+0000","last_became_peered":"2024-09-20T22:51:20.297933+0000","last_unstale":"2024-09-20T22:53:53.453391+0000","last_undegraded":"2024-09-20T22:53:53.453391+0000","last_fullsized":"2024-09-20T22:53:53.453391+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'20","last_scrub_stamp":"2024-09-20T22:52:19.125462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:52:19.125462+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":1426,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-20T22:53:53.454217+0000","last_change":"2024-09-20T22:52:46.062448+0000","last_active":"2024-09-20T22:53:53.454217+0000","last_peered":"2024-09-20T22:53:53.454217+0000","last_clean":"2024-09-20T22:53:53.454217+0000","last_became_active":"2024-09-20T22:41:38.155629+0000","last_became_peered":"2024-09-20T22:41:38.155629+0000","last_unstale":"2024-09-20T22:53:53.454217+0000","last_undegraded":"2024-09-20T22:53:53.454217+0000","last_fullsized":"2024-09-20T22:53:53.454217+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:46.062352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:52:46.062352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:15.728943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"733'12","reported_seq":844,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-20T22:53:53.303120+0000","last_change":"2024-09-20T22:53:33.928576+0000","last_active":"2024-09-20T22:53:53.303120+0000","last_peered":"2024-09-20T22:53:53.303120+0000","last_clean":"2024-09-20T22:53:53.303120+0000","last_became_active":"2024-09-20T22:47:49.227264+0000","last_became_peered":"2024-09-20T22:47:49.227264+0000","last_unstale":"2024-09-20T22:53:53.303120+0000","last_undegraded":"2024-09-20T22:53:53.303120+0000","last_fullsized":"2024-09-20T22:53:53.303120+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"733'12","last_scrub_stamp":"2024-09-20T22:53:33.928499+0000","last_deep_scrub":"733'12","last_deep_scrub_stamp":"2024-09-20T22:53:33.928499+0000","last_clean_scrub_stamp":"2024-09-20T22:53:33.928499+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:02.678336+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":758,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-20T22:53:53.302514+0000","last_change":"2024-09-20T22:51:55.755550+0000","last_active":"2024-09-20T22:53:53.302514+0000","last_peered":"2024-09-20T22:53:53.302514+0000","last_clean":"2024-09-20T22:53:53.302514+0000","last_became_active":"2024-09-20T22:47:49.232380+0000","last_became_peered":"2024-09-20T22:47:49.232380+0000","last_unstale":"2024-09-20T22:53:53.302514+0000","last_undegraded":"2024-09-20T22:53:53.302514+0000","last_fullsized":"2024-09-20T22:53:53.302514+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:55.755432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:51:55.755432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.0","version":"752'710","reported_seq":4009,"reported_epoch":753,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T22:53:53.452887+0000","last_change":"2024-09-20T22:53:51.443568+0000","last_active":"2024-09-20T22:53:53.452887+0000","last_peered":"2024-09-20T22:53:53.452887+0000","last_clean":"2024-09-20T22:53:53.452887+0000","last_became_active":"2024-09-20T22:51:53.366096+0000","last_became_peered":"2024-09-20T22:51:53.366096+0000","last_unstale":"2024-09-20T22:53:53.452887+0000","last_undegraded":"2024-09-20T22:53:53.452887+0000","last_fullsized":"2024-09-20T22:53:53.452887+0000","mapping_epoch":632,"log_start":"480'253","ondisk_log_start":"480'253","created":94,"last_epoch_clean":633,"parent":"0.0","parent_split_bits":0,"last_scrub":"562'368","last_scrub_stamp":"2024-09-20T22:50:44.669910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:50:44.669910+0000","objects_scrubbed":15,"log_size":457,"log_dups_size":416,"ondisk_log_size":457,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":10,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":1.1283459330000001,"stat_sum":{"num_bytes":56839522,"num_objects":27,"num_object_clones":24,"num_object_copies":81,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":263,"num_read_kb":267349,"num_write":219,"num_write_kb":81415,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":42943203,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"10e","length":"4"},{"start":"113","length":"c"},{"start":"120","length":"1"},{"start":"122","length":"2"},{"start":"125","length":"2"},{"start":"129","length":"1"},{"start":"12b","length":"1"},{"start":"12d","length":"1"},{"start":"131","length":"1"},{"start":"133","length":"6"},{"start":"13f","length":"1"}]},{"pgid":"1.6","version":"77'128","reported_seq":1745,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-20T22:53:53.455791+0000","last_change":"2024-09-20T22:52:38.843686+0000","last_active":"2024-09-20T22:53:53.455791+0000","last_peered":"2024-09-20T22:53:53.455791+0000","last_clean":"2024-09-20T22:53:53.455791+0000","last_became_active":"2024-09-20T22:47:52.212190+0000","last_became_peered":"2024-09-20T22:47:52.212190+0000","last_unstale":"2024-09-20T22:53:53.455791+0000","last_undegraded":"2024-09-20T22:53:53.455791+0000","last_fullsized":"2024-09-20T22:53:53.455791+0000","mapping_epoch":391,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:52:38.843586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:52:38.843586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:07.967280+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"625'37","reported_seq":798,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-20T22:53:53.302759+0000","last_change":"2024-09-20T22:52:39.703172+0000","last_active":"2024-09-20T22:53:53.302759+0000","last_peered":"2024-09-20T22:53:53.302759+0000","last_clean":"2024-09-20T22:53:53.302759+0000","last_became_active":"2024-09-20T22:47:49.222009+0000","last_became_peered":"2024-09-20T22:47:49.222009+0000","last_unstale":"2024-09-20T22:53:53.302759+0000","last_undegraded":"2024-09-20T22:53:53.302759+0000","last_fullsized":"2024-09-20T22:53:53.302759+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'37","last_scrub_stamp":"2024-09-20T22:52:39.703093+0000","last_deep_scrub":"515'27","last_deep_scrub_stamp":"2024-09-20T22:49:55.964153+0000","last_clean_scrub_stamp":"2024-09-20T22:52:39.703093+0000","objects_scrubbed":8,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":53,"num_read_kb":39,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":327,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-20T22:53:53.362234+0000","last_change":"2024-09-20T22:52:14.225251+0000","last_active":"2024-09-20T22:53:53.362234+0000","last_peered":"2024-09-20T22:53:53.362234+0000","last_clean":"2024-09-20T22:53:53.362234+0000","last_became_active":"2024-09-20T22:51:20.286646+0000","last_became_peered":"2024-09-20T22:51:20.286646+0000","last_unstale":"2024-09-20T22:53:53.362234+0000","last_undegraded":"2024-09-20T22:53:53.362234+0000","last_fullsized":"2024-09-20T22:53:53.362234+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:14.225111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:48:23.320031+0000","last_clean_scrub_stamp":"2024-09-20T22:52:14.225111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":1437,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-20T22:53:53.455172+0000","last_change":"2024-09-20T22:52:39.814499+0000","last_active":"2024-09-20T22:53:53.455172+0000","last_peered":"2024-09-20T22:53:53.455172+0000","last_clean":"2024-09-20T22:53:53.455172+0000","last_became_active":"2024-09-20T22:47:49.214055+0000","last_became_peered":"2024-09-20T22:47:49.214055+0000","last_unstale":"2024-09-20T22:53:53.455172+0000","last_undegraded":"2024-09-20T22:53:53.455172+0000","last_fullsized":"2024-09-20T22:53:53.455172+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:39.814381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:52:39.814381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":335,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-20T22:53:53.455213+0000","last_change":"2024-09-20T22:53:02.916523+0000","last_active":"2024-09-20T22:53:53.455213+0000","last_peered":"2024-09-20T22:53:53.455213+0000","last_clean":"2024-09-20T22:53:53.455213+0000","last_became_active":"2024-09-20T22:51:20.284874+0000","last_became_peered":"2024-09-20T22:51:20.284874+0000","last_unstale":"2024-09-20T22:53:53.455213+0000","last_undegraded":"2024-09-20T22:53:53.455213+0000","last_fullsized":"2024-09-20T22:53:53.455213+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:02.916449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:48:41.208853+0000","last_clean_scrub_stamp":"2024-09-20T22:53:02.916449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:15.012347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.7","version":"753'1375","reported_seq":4710,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-20T22:53:53.497589+0000","last_change":"2024-09-20T22:53:52.471894+0000","last_active":"2024-09-20T22:53:53.497589+0000","last_peered":"2024-09-20T22:53:53.497589+0000","last_clean":"2024-09-20T22:53:53.497589+0000","last_became_active":"2024-09-20T22:48:43.224386+0000","last_became_peered":"2024-09-20T22:48:43.224386+0000","last_unstale":"2024-09-20T22:53:53.497589+0000","last_undegraded":"2024-09-20T22:53:53.497589+0000","last_fullsized":"2024-09-20T22:53:53.497589+0000","mapping_epoch":391,"log_start":"436'456","ondisk_log_start":"436'456","created":94,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"704'1235","last_scrub_stamp":"2024-09-20T22:53:06.830408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:53:06.830408+0000","objects_scrubbed":31,"log_size":919,"log_dups_size":886,"ondisk_log_size":919,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:27.765191+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.013143699999999999,"stat_sum":{"num_bytes":60510439,"num_objects":26,"num_object_clones":21,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":1,"num_read":474,"num_read_kb":486374,"num_write":399,"num_write_kb":134673,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":115,"num_bytes_recovered":51615167,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"10e","length":"4"},{"start":"113","length":"c"},{"start":"120","length":"1"},{"start":"122","length":"2"},{"start":"125","length":"2"},{"start":"129","length":"1"},{"start":"12b","length":"3"},{"start":"131","length":"1"},{"start":"133","length":"6"},{"start":"13c","length":"1"},{"start":"13f","length":"2"},{"start":"143","length":"2"},{"start":"146","length":"2"},{"start":"14b","length":"1"},{"start":"14d","length":"1"},{"start":"150","length":"1"}]},{"pgid":"1.1","version":"77'128","reported_seq":1734,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-20T22:53:53.452791+0000","last_change":"2024-09-20T22:52:42.146147+0000","last_active":"2024-09-20T22:53:53.452791+0000","last_peered":"2024-09-20T22:53:53.452791+0000","last_clean":"2024-09-20T22:53:53.452791+0000","last_became_active":"2024-09-20T22:47:52.208102+0000","last_became_peered":"2024-09-20T22:47:52.208102+0000","last_unstale":"2024-09-20T22:53:53.452791+0000","last_undegraded":"2024-09-20T22:53:53.452791+0000","last_fullsized":"2024-09-20T22:53:53.452791+0000","mapping_epoch":391,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:52:42.146069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:52:42.146069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"625'51","reported_seq":509,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-20T22:53:53.362130+0000","last_change":"2024-09-20T22:52:17.199465+0000","last_active":"2024-09-20T22:53:53.362130+0000","last_peered":"2024-09-20T22:53:53.362130+0000","last_clean":"2024-09-20T22:53:53.362130+0000","last_became_active":"2024-09-20T22:51:20.289376+0000","last_became_peered":"2024-09-20T22:51:20.289376+0000","last_unstale":"2024-09-20T22:53:53.362130+0000","last_undegraded":"2024-09-20T22:53:53.362130+0000","last_fullsized":"2024-09-20T22:53:53.362130+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'51","last_scrub_stamp":"2024-09-20T22:52:17.199367+0000","last_deep_scrub":"86'38","last_deep_scrub_stamp":"2024-09-20T22:42:43.935677+0000","last_clean_scrub_stamp":"2024-09-20T22:52:17.199367+0000","objects_scrubbed":10,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":724,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-20T22:53:53.301576+0000","last_change":"2024-09-20T22:52:36.695537+0000","last_active":"2024-09-20T22:53:53.301576+0000","last_peered":"2024-09-20T22:53:53.301576+0000","last_clean":"2024-09-20T22:53:53.301576+0000","last_became_active":"2024-09-20T22:48:05.237756+0000","last_became_peered":"2024-09-20T22:48:05.237756+0000","last_unstale":"2024-09-20T22:53:53.301576+0000","last_undegraded":"2024-09-20T22:53:53.301576+0000","last_fullsized":"2024-09-20T22:53:53.301576+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:36.695446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:52:36.695446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"69'2","reported_seq":1446,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-20T22:53:53.453654+0000","last_change":"2024-09-20T22:51:37.070187+0000","last_active":"2024-09-20T22:53:53.453654+0000","last_peered":"2024-09-20T22:53:53.453654+0000","last_clean":"2024-09-20T22:53:53.453654+0000","last_became_active":"2024-09-20T22:51:20.286952+0000","last_became_peered":"2024-09-20T22:51:20.286952+0000","last_unstale":"2024-09-20T22:53:53.453654+0000","last_undegraded":"2024-09-20T22:53:53.453654+0000","last_fullsized":"2024-09-20T22:53:53.453654+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2024-09-20T22:51:37.070111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:36.130684+0000","last_clean_scrub_stamp":"2024-09-20T22:51:37.070111+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087280599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":777,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-20T22:53:53.303149+0000","last_change":"2024-09-20T22:53:51.849903+0000","last_active":"2024-09-20T22:53:53.303149+0000","last_peered":"2024-09-20T22:53:53.303149+0000","last_clean":"2024-09-20T22:53:53.303149+0000","last_became_active":"2024-09-20T22:51:20.278427+0000","last_became_peered":"2024-09-20T22:51:20.278427+0000","last_unstale":"2024-09-20T22:53:53.303149+0000","last_undegraded":"2024-09-20T22:53:53.303149+0000","last_fullsized":"2024-09-20T22:53:53.303149+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:51.849772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:53:51.849772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:20.262243+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":1417,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-20T22:53:53.364931+0000","last_change":"2024-09-20T22:52:21.363432+0000","last_active":"2024-09-20T22:53:53.364931+0000","last_peered":"2024-09-20T22:53:53.364931+0000","last_clean":"2024-09-20T22:53:53.364931+0000","last_became_active":"2024-09-20T22:48:05.234193+0000","last_became_peered":"2024-09-20T22:48:05.234193+0000","last_unstale":"2024-09-20T22:53:53.364931+0000","last_undegraded":"2024-09-20T22:53:53.364931+0000","last_fullsized":"2024-09-20T22:53:53.364931+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:21.363359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:52:21.363359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"752'508","reported_seq":2287,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-20T22:53:53.303525+0000","last_change":"2024-09-20T22:53:52.457608+0000","last_active":"2024-09-20T22:53:53.303525+0000","last_peered":"2024-09-20T22:53:53.303525+0000","last_clean":"2024-09-20T22:53:53.303525+0000","last_became_active":"2024-09-20T22:51:23.267613+0000","last_became_peered":"2024-09-20T22:51:23.267613+0000","last_unstale":"2024-09-20T22:53:53.303525+0000","last_undegraded":"2024-09-20T22:53:53.303525+0000","last_fullsized":"2024-09-20T22:53:53.303525+0000","mapping_epoch":602,"log_start":"456'176","ondisk_log_start":"456'176","created":94,"last_epoch_clean":603,"parent":"0.0","parent_split_bits":0,"last_scrub":"744'500","last_scrub_stamp":"2024-09-20T22:53:49.903129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:53:49.903129+0000","objects_scrubbed":10,"log_size":332,"log_dups_size":318,"ondisk_log_size":332,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:52.601275+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0036966770000000002,"stat_sum":{"num_bytes":26104163,"num_objects":10,"num_object_clones":8,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":172,"num_read_kb":155428,"num_write":149,"num_write_kb":49173,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":26566881,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"10e","length":"4"},{"start":"113","length":"c"},{"start":"120","length":"1"},{"start":"122","length":"2"},{"start":"125","length":"2"},{"start":"129","length":"1"},{"start":"12b","length":"3"},{"start":"131","length":"1"},{"start":"133","length":"6"},{"start":"13c","length":"1"},{"start":"13f","length":"2"},{"start":"143","length":"2"},{"start":"146","length":"2"},{"start":"14b","length":"1"},{"start":"14d","length":"1"},{"start":"150","length":"1"}]},{"pgid":"1.2","version":"77'128","reported_seq":1794,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-20T22:53:53.314266+0000","last_change":"2024-09-20T22:52:02.623634+0000","last_active":"2024-09-20T22:53:53.314266+0000","last_peered":"2024-09-20T22:53:53.314266+0000","last_clean":"2024-09-20T22:53:53.314266+0000","last_became_active":"2024-09-20T22:48:08.203028+0000","last_became_peered":"2024-09-20T22:48:08.203028+0000","last_unstale":"2024-09-20T22:53:53.314266+0000","last_undegraded":"2024-09-20T22:53:53.314266+0000","last_fullsized":"2024-09-20T22:53:53.314266+0000","mapping_epoch":407,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":408,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:52:02.623562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:52:02.623562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"625'22","reported_seq":1188,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-20T22:53:53.453602+0000","last_change":"2024-09-20T22:53:35.117496+0000","last_active":"2024-09-20T22:53:53.453602+0000","last_peered":"2024-09-20T22:53:53.453602+0000","last_clean":"2024-09-20T22:53:53.453602+0000","last_became_active":"2024-09-20T22:51:20.293610+0000","last_became_peered":"2024-09-20T22:51:20.293610+0000","last_unstale":"2024-09-20T22:53:53.453602+0000","last_undegraded":"2024-09-20T22:53:53.453602+0000","last_fullsized":"2024-09-20T22:53:53.453602+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'22","last_scrub_stamp":"2024-09-20T22:53:35.117387+0000","last_deep_scrub":"443'18","last_deep_scrub_stamp":"2024-09-20T22:48:44.182517+0000","last_clean_scrub_stamp":"2024-09-20T22:53:35.117387+0000","objects_scrubbed":5,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:45.532207+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":990,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":335,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-20T22:53:53.362303+0000","last_change":"2024-09-20T22:53:30.210048+0000","last_active":"2024-09-20T22:53:53.362303+0000","last_peered":"2024-09-20T22:53:53.362303+0000","last_clean":"2024-09-20T22:53:53.362303+0000","last_became_active":"2024-09-20T22:51:20.278609+0000","last_became_peered":"2024-09-20T22:51:20.278609+0000","last_unstale":"2024-09-20T22:53:53.362303+0000","last_undegraded":"2024-09-20T22:53:53.362303+0000","last_fullsized":"2024-09-20T22:53:53.362303+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:30.209986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:53:30.209986+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:35.918431+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"380'2","reported_seq":1471,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-20T22:53:53.361920+0000","last_change":"2024-09-20T22:52:37.033309+0000","last_active":"2024-09-20T22:53:53.361920+0000","last_peered":"2024-09-20T22:53:53.361920+0000","last_clean":"2024-09-20T22:53:53.361920+0000","last_became_active":"2024-09-20T22:47:49.222812+0000","last_became_peered":"2024-09-20T22:47:49.222812+0000","last_unstale":"2024-09-20T22:53:53.361920+0000","last_undegraded":"2024-09-20T22:53:53.361920+0000","last_fullsized":"2024-09-20T22:53:53.361920+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"380'2","last_scrub_stamp":"2024-09-20T22:52:37.033222+0000","last_deep_scrub":"380'2","last_deep_scrub_stamp":"2024-09-20T22:47:40.983039+0000","last_clean_scrub_stamp":"2024-09-20T22:52:37.033222+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:56.197067+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001301459,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":1425,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-20T22:53:53.362628+0000","last_change":"2024-09-20T22:53:25.223722+0000","last_active":"2024-09-20T22:53:53.362628+0000","last_peered":"2024-09-20T22:53:53.362628+0000","last_clean":"2024-09-20T22:53:53.362628+0000","last_became_active":"2024-09-20T22:41:42.185112+0000","last_became_peered":"2024-09-20T22:41:42.185112+0000","last_unstale":"2024-09-20T22:53:53.362628+0000","last_undegraded":"2024-09-20T22:53:53.362628+0000","last_fullsized":"2024-09-20T22:53:53.362628+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:25.223611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:47:22.881109+0000","last_clean_scrub_stamp":"2024-09-20T22:53:25.223611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:53.455503+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":1412,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-20T22:53:53.314970+0000","last_change":"2024-09-20T22:53:06.644045+0000","last_active":"2024-09-20T22:53:53.314970+0000","last_peered":"2024-09-20T22:53:53.314970+0000","last_clean":"2024-09-20T22:53:53.314970+0000","last_became_active":"2024-09-20T22:41:44.830515+0000","last_became_peered":"2024-09-20T22:41:44.830515+0000","last_unstale":"2024-09-20T22:53:53.314970+0000","last_undegraded":"2024-09-20T22:53:53.314970+0000","last_fullsized":"2024-09-20T22:53:53.314970+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:06.643947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:53:06.643947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:16.583688+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"752'393","reported_seq":3743,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-20T22:53:53.314882+0000","last_change":"2024-09-20T22:53:52.462588+0000","last_active":"2024-09-20T22:53:53.314882+0000","last_peered":"2024-09-20T22:53:53.314882+0000","last_clean":"2024-09-20T22:53:53.314882+0000","last_became_active":"2024-09-20T22:48:57.236417+0000","last_became_peered":"2024-09-20T22:48:57.236417+0000","last_unstale":"2024-09-20T22:53:53.314882+0000","last_undegraded":"2024-09-20T22:53:53.314882+0000","last_fullsized":"2024-09-20T22:53:53.314882+0000","mapping_epoch":391,"log_start":"450'190","ondisk_log_start":"450'190","created":94,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"674'316","last_scrub_stamp":"2024-09-20T22:52:35.685017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:52:35.685017+0000","objects_scrubbed":8,"log_size":203,"log_dups_size":240,"ondisk_log_size":203,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.0055678439999999997,"stat_sum":{"num_bytes":14965811,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":99,"num_read_kb":89344,"num_write":98,"num_write_kb":40424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":19673726,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"10e","length":"4"},{"start":"113","length":"c"},{"start":"120","length":"1"},{"start":"122","length":"2"},{"start":"125","length":"2"},{"start":"129","length":"1"},{"start":"12b","length":"3"},{"start":"131","length":"1"},{"start":"133","length":"6"},{"start":"13c","length":"1"},{"start":"13f","length":"2"},{"start":"143","length":"2"},{"start":"146","length":"2"},{"start":"14b","length":"1"},{"start":"14d","length":"1"},{"start":"150","length":"1"}]},{"pgid":"1.3","version":"77'128","reported_seq":630,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-20T22:53:53.455498+0000","last_change":"2024-09-20T22:53:05.866046+0000","last_active":"2024-09-20T22:53:53.455498+0000","last_peered":"2024-09-20T22:53:53.455498+0000","last_clean":"2024-09-20T22:53:53.455498+0000","last_became_active":"2024-09-20T22:51:23.266810+0000","last_became_peered":"2024-09-20T22:51:23.266810+0000","last_unstale":"2024-09-20T22:53:53.455498+0000","last_undegraded":"2024-09-20T22:53:53.455498+0000","last_fullsized":"2024-09-20T22:53:53.455498+0000","mapping_epoch":602,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":603,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:53:05.865942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:53:05.865942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:28.578041+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.6","version":"625'24","reported_seq":809,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-20T22:53:53.303843+0000","last_change":"2024-09-20T22:52:48.829973+0000","last_active":"2024-09-20T22:53:53.303843+0000","last_peered":"2024-09-20T22:53:53.303843+0000","last_clean":"2024-09-20T22:53:53.303843+0000","last_became_active":"2024-09-20T22:51:20.283112+0000","last_became_peered":"2024-09-20T22:51:20.283112+0000","last_unstale":"2024-09-20T22:53:53.303843+0000","last_undegraded":"2024-09-20T22:53:53.303843+0000","last_fullsized":"2024-09-20T22:53:53.303843+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'24","last_scrub_stamp":"2024-09-20T22:52:48.829882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:52:48.829882+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:17.822454+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":771,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-20T22:53:53.302552+0000","last_change":"2024-09-20T22:53:25.172560+0000","last_active":"2024-09-20T22:53:53.302552+0000","last_peered":"2024-09-20T22:53:53.302552+0000","last_clean":"2024-09-20T22:53:53.302552+0000","last_became_active":"2024-09-20T22:48:05.233093+0000","last_became_peered":"2024-09-20T22:48:05.233093+0000","last_unstale":"2024-09-20T22:53:53.302552+0000","last_undegraded":"2024-09-20T22:53:53.302552+0000","last_fullsized":"2024-09-20T22:53:53.302552+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:25.172487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:44:19.834641+0000","last_clean_scrub_stamp":"2024-09-20T22:53:25.172487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:39.242526+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"75'5","reported_seq":2193,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-20T22:53:53.314422+0000","last_change":"2024-09-20T22:52:44.568232+0000","last_active":"2024-09-20T22:53:53.314422+0000","last_peered":"2024-09-20T22:53:53.314422+0000","last_clean":"2024-09-20T22:53:53.314422+0000","last_became_active":"2024-09-20T22:51:20.281329+0000","last_became_peered":"2024-09-20T22:51:20.281329+0000","last_unstale":"2024-09-20T22:53:53.314422+0000","last_undegraded":"2024-09-20T22:53:53.314422+0000","last_fullsized":"2024-09-20T22:53:53.314422+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-09-20T22:52:44.568129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:36.130684+0000","last_clean_scrub_stamp":"2024-09-20T22:52:44.568129+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:05.662332+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00078631700000000003,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":725,"num_read_kb":720,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1167,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":1436,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-20T22:53:53.453710+0000","last_change":"2024-09-20T22:52:48.082097+0000","last_active":"2024-09-20T22:53:53.453710+0000","last_peered":"2024-09-20T22:53:53.453710+0000","last_clean":"2024-09-20T22:53:53.453710+0000","last_became_active":"2024-09-20T22:51:20.274522+0000","last_became_peered":"2024-09-20T22:51:20.274522+0000","last_unstale":"2024-09-20T22:53:53.453710+0000","last_undegraded":"2024-09-20T22:53:53.453710+0000","last_fullsized":"2024-09-20T22:53:53.453710+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:48.082027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:52:48.082027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":327,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-20T22:53:53.453430+0000","last_change":"2024-09-20T22:53:49.031995+0000","last_active":"2024-09-20T22:53:53.453430+0000","last_peered":"2024-09-20T22:53:53.453430+0000","last_clean":"2024-09-20T22:53:53.453430+0000","last_became_active":"2024-09-20T22:51:20.287750+0000","last_became_peered":"2024-09-20T22:51:20.287750+0000","last_unstale":"2024-09-20T22:53:53.453430+0000","last_undegraded":"2024-09-20T22:53:53.453430+0000","last_fullsized":"2024-09-20T22:53:53.453430+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:49.031912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:53:49.031912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:01.922210+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.6","version":"752'512","reported_seq":3786,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-20T22:53:53.303644+0000","last_change":"2024-09-20T22:53:52.460829+0000","last_active":"2024-09-20T22:53:53.303644+0000","last_peered":"2024-09-20T22:53:53.303644+0000","last_clean":"2024-09-20T22:53:53.303644+0000","last_became_active":"2024-09-20T22:48:47.234531+0000","last_became_peered":"2024-09-20T22:48:47.234531+0000","last_unstale":"2024-09-20T22:53:53.303644+0000","last_undegraded":"2024-09-20T22:53:53.303644+0000","last_fullsized":"2024-09-20T22:53:53.303644+0000","mapping_epoch":391,"log_start":"443'181","ondisk_log_start":"443'181","created":94,"last_epoch_clean":447,"parent":"0.0","parent_split_bits":0,"last_scrub":"654'392","last_scrub_stamp":"2024-09-20T22:52:14.706591+0000","last_deep_scrub":"440'177","last_deep_scrub_stamp":"2024-09-20T22:48:40.602342+0000","last_clean_scrub_stamp":"2024-09-20T22:52:14.706591+0000","objects_scrubbed":10,"log_size":331,"log_dups_size":314,"ondisk_log_size":331,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.0061309889999999999,"stat_sum":{"num_bytes":22318112,"num_objects":11,"num_object_clones":9,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":1,"num_read":199,"num_read_kb":209855,"num_write":141,"num_write_kb":55093,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":18662529,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"10e","length":"4"},{"start":"113","length":"c"},{"start":"120","length":"1"},{"start":"122","length":"2"},{"start":"125","length":"2"},{"start":"129","length":"1"},{"start":"12b","length":"3"},{"start":"131","length":"1"},{"start":"133","length":"6"},{"start":"13c","length":"1"},{"start":"13f","length":"2"},{"start":"143","length":"2"},{"start":"146","length":"2"},{"start":"14b","length":"1"},{"start":"14d","length":"1"},{"start":"150","length":"1"}]},{"pgid":"4.5","version":"625'32","reported_seq":1713,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-20T22:53:53.455957+0000","last_change":"2024-09-20T22:53:07.867174+0000","last_active":"2024-09-20T22:53:53.455957+0000","last_peered":"2024-09-20T22:53:53.455957+0000","last_clean":"2024-09-20T22:53:53.455957+0000","last_became_active":"2024-09-20T22:48:05.243523+0000","last_became_peered":"2024-09-20T22:48:05.243523+0000","last_unstale":"2024-09-20T22:53:53.455957+0000","last_undegraded":"2024-09-20T22:53:53.455957+0000","last_fullsized":"2024-09-20T22:53:53.455957+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'32","last_scrub_stamp":"2024-09-20T22:53:07.867088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:53:07.867088+0000","objects_scrubbed":8,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:23.661414+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"67'1","reported_seq":1415,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-20T22:53:53.365706+0000","last_change":"2024-09-20T22:51:35.440516+0000","last_active":"2024-09-20T22:53:53.365706+0000","last_peered":"2024-09-20T22:53:53.365706+0000","last_clean":"2024-09-20T22:53:53.365706+0000","last_became_active":"2024-09-20T22:41:38.158917+0000","last_became_peered":"2024-09-20T22:41:38.158917+0000","last_unstale":"2024-09-20T22:53:53.365706+0000","last_undegraded":"2024-09-20T22:53:53.365706+0000","last_fullsized":"2024-09-20T22:53:53.365706+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-09-20T22:51:35.440416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:51:35.440416+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"77'128","reported_seq":1766,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-20T22:53:53.314492+0000","last_change":"2024-09-20T22:52:18.688292+0000","last_active":"2024-09-20T22:53:53.314492+0000","last_peered":"2024-09-20T22:53:53.314492+0000","last_clean":"2024-09-20T22:53:53.314492+0000","last_became_active":"2024-09-20T22:47:52.206169+0000","last_became_peered":"2024-09-20T22:47:52.206169+0000","last_unstale":"2024-09-20T22:53:53.314492+0000","last_undegraded":"2024-09-20T22:53:53.314492+0000","last_fullsized":"2024-09-20T22:53:53.314492+0000","mapping_epoch":391,"log_start":"77'128","ondisk_log_start":"77'128","created":19,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:52:18.688135+0000","last_deep_scrub":"77'128","last_deep_scrub_stamp":"2024-09-20T22:47:22.417099+0000","last_clean_scrub_stamp":"2024-09-20T22:52:18.688135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":1424,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-20T22:53:53.314969+0000","last_change":"2024-09-20T22:53:03.564792+0000","last_active":"2024-09-20T22:53:53.314969+0000","last_peered":"2024-09-20T22:53:53.314969+0000","last_clean":"2024-09-20T22:53:53.314969+0000","last_became_active":"2024-09-20T22:41:42.184387+0000","last_became_peered":"2024-09-20T22:41:42.184387+0000","last_unstale":"2024-09-20T22:53:53.314969+0000","last_undegraded":"2024-09-20T22:53:53.314969+0000","last_fullsized":"2024-09-20T22:53:53.314969+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:03.564726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:53:03.564726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:25.313703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":1426,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-20T22:53:53.453451+0000","last_change":"2024-09-20T22:53:09.957459+0000","last_active":"2024-09-20T22:53:53.453451+0000","last_peered":"2024-09-20T22:53:53.453451+0000","last_clean":"2024-09-20T22:53:53.453451+0000","last_became_active":"2024-09-20T22:48:05.241602+0000","last_became_peered":"2024-09-20T22:48:05.241602+0000","last_unstale":"2024-09-20T22:53:53.453451+0000","last_undegraded":"2024-09-20T22:53:53.453451+0000","last_fullsized":"2024-09-20T22:53:53.453451+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:09.957374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:53:09.957374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:12.536331+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"752'603","reported_seq":4000,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-20T22:53:53.455878+0000","last_change":"2024-09-20T22:53:52.464181+0000","last_active":"2024-09-20T22:53:53.455878+0000","last_peered":"2024-09-20T22:53:53.455878+0000","last_clean":"2024-09-20T22:53:53.455878+0000","last_became_active":"2024-09-20T22:51:23.269216+0000","last_became_peered":"2024-09-20T22:51:23.269216+0000","last_unstale":"2024-09-20T22:53:53.455878+0000","last_undegraded":"2024-09-20T22:53:53.455878+0000","last_fullsized":"2024-09-20T22:53:53.455878+0000","mapping_epoch":602,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":603,"parent":"0.0","parent_split_bits":0,"last_scrub":"702'544","last_scrub_stamp":"2024-09-20T22:53:03.918130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:53:03.918130+0000","objects_scrubbed":10,"log_size":603,"log_dups_size":0,"ondisk_log_size":603,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:06.361345+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.0055691100000000004,"stat_sum":{"num_bytes":22554409,"num_objects":9,"num_object_clones":7,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":197,"num_read_kb":205211,"num_write":108,"num_write_kb":45319,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":52,"num_bytes_recovered":30669453,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"10e","length":"4"},{"start":"113","length":"c"},{"start":"120","length":"1"},{"start":"122","length":"2"},{"start":"125","length":"2"},{"start":"129","length":"1"},{"start":"12b","length":"3"},{"start":"131","length":"1"},{"start":"133","length":"6"},{"start":"13c","length":"1"},{"start":"13f","length":"2"},{"start":"143","length":"2"},{"start":"146","length":"2"},{"start":"14b","length":"1"},{"start":"14d","length":"1"},{"start":"150","length":"1"}]},{"pgid":"4.e","version":"625'19","reported_seq":766,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-20T22:53:53.302200+0000","last_change":"2024-09-20T22:52:44.783087+0000","last_active":"2024-09-20T22:53:53.302200+0000","last_peered":"2024-09-20T22:53:53.302200+0000","last_clean":"2024-09-20T22:53:53.302200+0000","last_became_active":"2024-09-20T22:51:20.288958+0000","last_became_peered":"2024-09-20T22:51:20.288958+0000","last_unstale":"2024-09-20T22:53:53.302200+0000","last_undegraded":"2024-09-20T22:53:53.302200+0000","last_fullsized":"2024-09-20T22:53:53.302200+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'19","last_scrub_stamp":"2024-09-20T22:52:44.782975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:52:44.782975+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":724,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-20T22:53:53.300992+0000","last_change":"2024-09-20T22:51:29.757136+0000","last_active":"2024-09-20T22:53:53.300992+0000","last_peered":"2024-09-20T22:53:53.300992+0000","last_clean":"2024-09-20T22:53:53.300992+0000","last_became_active":"2024-09-20T22:48:05.254688+0000","last_became_peered":"2024-09-20T22:48:05.254688+0000","last_unstale":"2024-09-20T22:53:53.300992+0000","last_undegraded":"2024-09-20T22:53:53.300992+0000","last_fullsized":"2024-09-20T22:53:53.300992+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:29.757030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:51:29.757030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":1422,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-20T22:53:53.453582+0000","last_change":"2024-09-20T22:51:51.002353+0000","last_active":"2024-09-20T22:53:53.453582+0000","last_peered":"2024-09-20T22:53:53.453582+0000","last_clean":"2024-09-20T22:53:53.453582+0000","last_became_active":"2024-09-20T22:48:05.233778+0000","last_became_peered":"2024-09-20T22:48:05.233778+0000","last_unstale":"2024-09-20T22:53:53.453582+0000","last_undegraded":"2024-09-20T22:53:53.453582+0000","last_fullsized":"2024-09-20T22:53:53.453582+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:51.002226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:51:51.002226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":1405,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-20T22:53:53.364959+0000","last_change":"2024-09-20T22:53:10.647075+0000","last_active":"2024-09-20T22:53:53.364959+0000","last_peered":"2024-09-20T22:53:53.364959+0000","last_clean":"2024-09-20T22:53:53.364959+0000","last_became_active":"2024-09-20T22:41:44.832009+0000","last_became_peered":"2024-09-20T22:41:44.832009+0000","last_unstale":"2024-09-20T22:53:53.364959+0000","last_undegraded":"2024-09-20T22:53:53.364959+0000","last_fullsized":"2024-09-20T22:53:53.364959+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:10.647005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:48:40.127123+0000","last_clean_scrub_stamp":"2024-09-20T22:53:10.647005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:32.128560+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.e","version":"752'759","reported_seq":4088,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-20T22:53:53.362598+0000","last_change":"2024-09-20T22:53:52.469962+0000","last_active":"2024-09-20T22:53:53.362598+0000","last_peered":"2024-09-20T22:53:53.362598+0000","last_clean":"2024-09-20T22:53:53.362598+0000","last_became_active":"2024-09-20T22:47:52.209452+0000","last_became_peered":"2024-09-20T22:47:52.209452+0000","last_unstale":"2024-09-20T22:53:53.362598+0000","last_undegraded":"2024-09-20T22:53:53.362598+0000","last_fullsized":"2024-09-20T22:53:53.362598+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"681'618","last_scrub_stamp":"2024-09-20T22:52:45.935130+0000","last_deep_scrub":"174'60","last_deep_scrub_stamp":"2024-09-20T22:44:18.301776+0000","last_clean_scrub_stamp":"2024-09-20T22:52:45.935130+0000","objects_scrubbed":13,"log_size":759,"log_dups_size":0,"ondisk_log_size":759,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:09.945098+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.007625153,"stat_sum":{"num_bytes":27683579,"num_objects":12,"num_object_clones":10,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":254,"num_read_kb":256137,"num_write":183,"num_write_kb":65479,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":23110741,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"10e","length":"4"},{"start":"113","length":"c"},{"start":"120","length":"1"},{"start":"122","length":"2"},{"start":"125","length":"2"},{"start":"129","length":"1"},{"start":"12b","length":"3"},{"start":"131","length":"1"},{"start":"133","length":"6"},{"start":"13c","length":"1"},{"start":"13f","length":"2"},{"start":"143","length":"2"},{"start":"146","length":"2"},{"start":"14b","length":"1"},{"start":"14d","length":"1"},{"start":"150","length":"1"}]},{"pgid":"1.8","version":"77'128","reported_seq":1686,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-20T22:53:53.453265+0000","last_change":"2024-09-20T22:51:56.108038+0000","last_active":"2024-09-20T22:53:53.453265+0000","last_peered":"2024-09-20T22:53:53.453265+0000","last_clean":"2024-09-20T22:53:53.453265+0000","last_became_active":"2024-09-20T22:42:52.405594+0000","last_became_peered":"2024-09-20T22:42:52.405594+0000","last_unstale":"2024-09-20T22:53:53.453265+0000","last_undegraded":"2024-09-20T22:53:53.453265+0000","last_fullsized":"2024-09-20T22:53:53.453265+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:51:56.107944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:51:56.107944+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"625'31","reported_seq":795,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-20T22:53:53.301970+0000","last_change":"2024-09-20T22:52:13.620897+0000","last_active":"2024-09-20T22:53:53.301970+0000","last_peered":"2024-09-20T22:53:53.301970+0000","last_clean":"2024-09-20T22:53:53.301970+0000","last_became_active":"2024-09-20T22:51:20.288846+0000","last_became_peered":"2024-09-20T22:51:20.288846+0000","last_unstale":"2024-09-20T22:53:53.301970+0000","last_undegraded":"2024-09-20T22:53:53.301970+0000","last_fullsized":"2024-09-20T22:53:53.301970+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'31","last_scrub_stamp":"2024-09-20T22:52:13.620754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:52:13.620754+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":49,"num_read_kb":39,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":980,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-20T22:53:53.361687+0000","last_change":"2024-09-20T22:53:08.105116+0000","last_active":"2024-09-20T22:53:53.361687+0000","last_peered":"2024-09-20T22:53:53.361687+0000","last_clean":"2024-09-20T22:53:53.361687+0000","last_became_active":"2024-09-20T22:51:20.294073+0000","last_became_peered":"2024-09-20T22:51:20.294073+0000","last_unstale":"2024-09-20T22:53:53.361687+0000","last_undegraded":"2024-09-20T22:53:53.361687+0000","last_fullsized":"2024-09-20T22:53:53.361687+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:08.105010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:35.492562+0000","last_clean_scrub_stamp":"2024-09-20T22:53:08.105010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:35.105470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":1037,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-20T22:53:53.301426+0000","last_change":"2024-09-20T22:51:25.738132+0000","last_active":"2024-09-20T22:53:53.301426+0000","last_peered":"2024-09-20T22:53:53.301426+0000","last_clean":"2024-09-20T22:53:53.301426+0000","last_became_active":"2024-09-20T22:51:20.281724+0000","last_became_peered":"2024-09-20T22:51:20.281724+0000","last_unstale":"2024-09-20T22:53:53.301426+0000","last_undegraded":"2024-09-20T22:53:53.301426+0000","last_fullsized":"2024-09-20T22:53:53.301426+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:25.738061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:51:25.738061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":1441,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-20T22:53:53.362488+0000","last_change":"2024-09-20T22:53:05.106323+0000","last_active":"2024-09-20T22:53:53.362488+0000","last_peered":"2024-09-20T22:53:53.362488+0000","last_clean":"2024-09-20T22:53:53.362488+0000","last_became_active":"2024-09-20T22:48:05.239006+0000","last_became_peered":"2024-09-20T22:48:05.239006+0000","last_unstale":"2024-09-20T22:53:53.362488+0000","last_undegraded":"2024-09-20T22:53:53.362488+0000","last_fullsized":"2024-09-20T22:53:53.362488+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:05.106224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:53:05.106224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:05.128923+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"752'834","reported_seq":4140,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-20T22:53:53.303547+0000","last_change":"2024-09-20T22:53:52.468157+0000","last_active":"2024-09-20T22:53:53.303547+0000","last_peered":"2024-09-20T22:53:53.303547+0000","last_clean":"2024-09-20T22:53:53.303547+0000","last_became_active":"2024-09-20T22:47:52.216198+0000","last_became_peered":"2024-09-20T22:47:52.216198+0000","last_unstale":"2024-09-20T22:53:53.303547+0000","last_undegraded":"2024-09-20T22:53:53.303547+0000","last_fullsized":"2024-09-20T22:53:53.303547+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"630'603","last_scrub_stamp":"2024-09-20T22:51:50.769050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:51:50.769050+0000","objects_scrubbed":8,"log_size":834,"log_dups_size":0,"ondisk_log_size":834,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.013213449,"stat_sum":{"num_bytes":32101893,"num_objects":17,"num_object_clones":14,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":1,"num_read":334,"num_read_kb":329186,"num_write":216,"num_write_kb":70806,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":53,"num_bytes_recovered":14983556,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"10e","length":"4"},{"start":"113","length":"c"},{"start":"120","length":"1"},{"start":"122","length":"2"},{"start":"125","length":"2"},{"start":"129","length":"1"},{"start":"12b","length":"3"},{"start":"131","length":"1"},{"start":"133","length":"6"},{"start":"13c","length":"1"},{"start":"13f","length":"2"},{"start":"143","length":"2"},{"start":"146","length":"2"},{"start":"14b","length":"1"},{"start":"14d","length":"1"},{"start":"150","length":"1"}]},{"pgid":"1.9","version":"77'128","reported_seq":1742,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-20T22:53:53.314307+0000","last_change":"2024-09-20T22:53:50.381120+0000","last_active":"2024-09-20T22:53:53.314307+0000","last_peered":"2024-09-20T22:53:53.314307+0000","last_clean":"2024-09-20T22:53:53.314307+0000","last_became_active":"2024-09-20T22:50:42.262862+0000","last_became_peered":"2024-09-20T22:50:42.262862+0000","last_unstale":"2024-09-20T22:53:53.314307+0000","last_undegraded":"2024-09-20T22:53:53.314307+0000","last_fullsized":"2024-09-20T22:53:53.314307+0000","mapping_epoch":561,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":562,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:53:50.381052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:53:50.381052+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:18.159814+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"625'20","reported_seq":751,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-20T22:53:53.301771+0000","last_change":"2024-09-20T22:52:22.646547+0000","last_active":"2024-09-20T22:53:53.301771+0000","last_peered":"2024-09-20T22:53:53.301771+0000","last_clean":"2024-09-20T22:53:53.301771+0000","last_became_active":"2024-09-20T22:48:05.242811+0000","last_became_peered":"2024-09-20T22:48:05.242811+0000","last_unstale":"2024-09-20T22:53:53.301771+0000","last_undegraded":"2024-09-20T22:53:53.301771+0000","last_fullsized":"2024-09-20T22:53:53.301771+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'20","last_scrub_stamp":"2024-09-20T22:52:22.646468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:52:22.646468+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":1441,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-20T22:53:53.365262+0000","last_change":"2024-09-20T22:52:44.527004+0000","last_active":"2024-09-20T22:53:53.365262+0000","last_peered":"2024-09-20T22:53:53.365262+0000","last_clean":"2024-09-20T22:53:53.365262+0000","last_became_active":"2024-09-20T22:48:05.233999+0000","last_became_peered":"2024-09-20T22:48:05.233999+0000","last_unstale":"2024-09-20T22:53:53.365262+0000","last_undegraded":"2024-09-20T22:53:53.365262+0000","last_fullsized":"2024-09-20T22:53:53.365262+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:44.526935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:52:44.526935+0000","last_clean_scrub_stamp":"2024-09-20T22:52:44.526935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"261'9","reported_seq":1582,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-20T22:53:53.362438+0000","last_change":"2024-09-20T22:52:18.218722+0000","last_active":"2024-09-20T22:53:53.362438+0000","last_peered":"2024-09-20T22:53:53.362438+0000","last_clean":"2024-09-20T22:53:53.362438+0000","last_became_active":"2024-09-20T22:41:42.185863+0000","last_became_peered":"2024-09-20T22:41:42.185863+0000","last_unstale":"2024-09-20T22:53:53.362438+0000","last_undegraded":"2024-09-20T22:53:53.362438+0000","last_fullsized":"2024-09-20T22:53:53.362438+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'9","last_scrub_stamp":"2024-09-20T22:52:18.218644+0000","last_deep_scrub":"261'9","last_deep_scrub_stamp":"2024-09-20T22:46:50.974735+0000","last_clean_scrub_stamp":"2024-09-20T22:52:18.218644+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":738,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-20T22:53:53.301935+0000","last_change":"2024-09-20T22:53:10.650499+0000","last_active":"2024-09-20T22:53:53.301935+0000","last_peered":"2024-09-20T22:53:53.301935+0000","last_clean":"2024-09-20T22:53:53.301935+0000","last_became_active":"2024-09-20T22:51:20.296204+0000","last_became_peered":"2024-09-20T22:51:20.296204+0000","last_unstale":"2024-09-20T22:53:53.301935+0000","last_undegraded":"2024-09-20T22:53:53.301935+0000","last_fullsized":"2024-09-20T22:53:53.301935+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:10.650428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:53:10.650428+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:28.093442+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.8","version":"752'944","reported_seq":4094,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-20T22:53:53.494534+0000","last_change":"2024-09-20T22:53:52.470369+0000","last_active":"2024-09-20T22:53:53.494534+0000","last_peered":"2024-09-20T22:53:53.494534+0000","last_clean":"2024-09-20T22:53:53.494534+0000","last_became_active":"2024-09-20T22:48:31.224359+0000","last_became_peered":"2024-09-20T22:48:31.224359+0000","last_unstale":"2024-09-20T22:53:53.494534+0000","last_undegraded":"2024-09-20T22:53:53.494534+0000","last_fullsized":"2024-09-20T22:53:53.494534+0000","mapping_epoch":407,"log_start":"426'268","ondisk_log_start":"426'268","created":94,"last_epoch_clean":431,"parent":"0.0","parent_split_bits":0,"last_scrub":"674'746","last_scrub_stamp":"2024-09-20T22:52:35.065256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:52:35.065256+0000","objects_scrubbed":20,"log_size":676,"log_dups_size":445,"ondisk_log_size":676,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:04.805230+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.0085650220000000003,"stat_sum":{"num_bytes":34060348,"num_objects":16,"num_object_clones":13,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":1,"num_read":303,"num_read_kb":310536,"num_write":237,"num_write_kb":78277,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":26027409,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"10e","length":"4"},{"start":"113","length":"c"},{"start":"120","length":"1"},{"start":"122","length":"2"},{"start":"125","length":"2"},{"start":"129","length":"1"},{"start":"12b","length":"3"},{"start":"131","length":"1"},{"start":"133","length":"6"},{"start":"13c","length":"1"},{"start":"13f","length":"2"},{"start":"143","length":"2"},{"start":"146","length":"2"},{"start":"14b","length":"1"},{"start":"14d","length":"1"},{"start":"150","length":"1"}]},{"pgid":"4.b","version":"625'15","reported_seq":816,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-20T22:53:53.302896+0000","last_change":"2024-09-20T22:53:30.976691+0000","last_active":"2024-09-20T22:53:53.302896+0000","last_peered":"2024-09-20T22:53:53.302896+0000","last_clean":"2024-09-20T22:53:53.302896+0000","last_became_active":"2024-09-20T22:51:20.273094+0000","last_became_peered":"2024-09-20T22:51:20.273094+0000","last_unstale":"2024-09-20T22:53:53.302896+0000","last_undegraded":"2024-09-20T22:53:53.302896+0000","last_fullsized":"2024-09-20T22:53:53.302896+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'15","last_scrub_stamp":"2024-09-20T22:53:30.976566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:53:30.976566+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:51.015430+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":1418,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-20T22:53:53.453331+0000","last_change":"2024-09-20T22:51:12.755628+0000","last_active":"2024-09-20T22:53:53.453331+0000","last_peered":"2024-09-20T22:53:53.453331+0000","last_clean":"2024-09-20T22:53:53.453331+0000","last_became_active":"2024-09-20T22:41:38.155953+0000","last_became_peered":"2024-09-20T22:41:38.155953+0000","last_unstale":"2024-09-20T22:53:53.453331+0000","last_undegraded":"2024-09-20T22:53:53.453331+0000","last_fullsized":"2024-09-20T22:53:53.453331+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:12.755526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:51:12.755526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":1449,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-20T22:53:53.361807+0000","last_change":"2024-09-20T22:52:42.998852+0000","last_active":"2024-09-20T22:53:53.361807+0000","last_peered":"2024-09-20T22:53:53.361807+0000","last_clean":"2024-09-20T22:53:53.361807+0000","last_became_active":"2024-09-20T22:48:05.240613+0000","last_became_peered":"2024-09-20T22:48:05.240613+0000","last_unstale":"2024-09-20T22:53:53.361807+0000","last_undegraded":"2024-09-20T22:53:53.361807+0000","last_fullsized":"2024-09-20T22:53:53.361807+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:42.998756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:52:42.998756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:11.946882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":757,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-20T22:53:53.302642+0000","last_change":"2024-09-20T22:52:16.706497+0000","last_active":"2024-09-20T22:53:53.302642+0000","last_peered":"2024-09-20T22:53:53.302642+0000","last_clean":"2024-09-20T22:53:53.302642+0000","last_became_active":"2024-09-20T22:47:49.219486+0000","last_became_peered":"2024-09-20T22:47:49.219486+0000","last_unstale":"2024-09-20T22:53:53.302642+0000","last_undegraded":"2024-09-20T22:53:53.302642+0000","last_fullsized":"2024-09-20T22:53:53.302642+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:16.706429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:52:16.706429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"753'1274","reported_seq":4637,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-20T22:53:53.312200+0000","last_change":"2024-09-20T22:53:52.457582+0000","last_active":"2024-09-20T22:53:53.312200+0000","last_peered":"2024-09-20T22:53:53.312200+0000","last_clean":"2024-09-20T22:53:53.312200+0000","last_became_active":"2024-09-20T22:51:23.270962+0000","last_became_peered":"2024-09-20T22:51:23.270962+0000","last_unstale":"2024-09-20T22:53:53.312200+0000","last_undegraded":"2024-09-20T22:53:53.312200+0000","last_fullsized":"2024-09-20T22:53:53.312200+0000","mapping_epoch":602,"log_start":"423'280","ondisk_log_start":"423'280","created":94,"last_epoch_clean":603,"parent":"0.0","parent_split_bits":0,"last_scrub":"674'1040","last_scrub_stamp":"2024-09-20T22:52:34.647394+0000","last_deep_scrub":"590'828","last_deep_scrub_stamp":"2024-09-20T22:51:10.909364+0000","last_clean_scrub_stamp":"2024-09-20T22:52:34.647394+0000","objects_scrubbed":29,"log_size":994,"log_dups_size":649,"ondisk_log_size":994,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:54.329404+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0040687750000000002,"stat_sum":{"num_bytes":53224142,"num_objects":21,"num_object_clones":16,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":2,"num_read":466,"num_read_kb":478342,"num_write":337,"num_write_kb":130862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":91020358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"10e","length":"4"},{"start":"113","length":"c"},{"start":"120","length":"1"},{"start":"122","length":"2"},{"start":"125","length":"2"},{"start":"129","length":"1"},{"start":"12b","length":"3"},{"start":"131","length":"1"},{"start":"133","length":"6"},{"start":"13c","length":"1"},{"start":"13f","length":"2"},{"start":"143","length":"2"},{"start":"146","length":"2"},{"start":"14b","length":"1"},{"start":"14d","length":"1"},{"start":"150","length":"1"}]},{"pgid":"4.a","version":"625'44","reported_seq":1527,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-20T22:53:53.455224+0000","last_change":"2024-09-20T22:53:50.910522+0000","last_active":"2024-09-20T22:53:53.455224+0000","last_peered":"2024-09-20T22:53:53.455224+0000","last_clean":"2024-09-20T22:53:53.455224+0000","last_became_active":"2024-09-20T22:51:20.287620+0000","last_became_peered":"2024-09-20T22:51:20.287620+0000","last_unstale":"2024-09-20T22:53:53.455224+0000","last_undegraded":"2024-09-20T22:53:53.455224+0000","last_fullsized":"2024-09-20T22:53:53.455224+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'44","last_scrub_stamp":"2024-09-20T22:53:50.910441+0000","last_deep_scrub":"113'28","last_deep_scrub_stamp":"2024-09-20T22:43:12.725900+0000","last_clean_scrub_stamp":"2024-09-20T22:53:50.910441+0000","objects_scrubbed":9,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:12.635234+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":1424,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-20T22:53:53.315243+0000","last_change":"2024-09-20T22:53:27.585334+0000","last_active":"2024-09-20T22:53:53.315243+0000","last_peered":"2024-09-20T22:53:53.315243+0000","last_clean":"2024-09-20T22:53:53.315243+0000","last_became_active":"2024-09-20T22:41:38.161524+0000","last_became_peered":"2024-09-20T22:41:38.161524+0000","last_unstale":"2024-09-20T22:53:53.315243+0000","last_undegraded":"2024-09-20T22:53:53.315243+0000","last_fullsized":"2024-09-20T22:53:53.315243+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:27.585262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:53:27.585262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:52.375797+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":1437,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-20T22:53:53.362050+0000","last_change":"2024-09-20T22:52:40.004730+0000","last_active":"2024-09-20T22:53:53.362050+0000","last_peered":"2024-09-20T22:53:53.362050+0000","last_clean":"2024-09-20T22:53:53.362050+0000","last_became_active":"2024-09-20T22:47:49.223220+0000","last_became_peered":"2024-09-20T22:47:49.223220+0000","last_unstale":"2024-09-20T22:53:53.362050+0000","last_undegraded":"2024-09-20T22:53:53.362050+0000","last_fullsized":"2024-09-20T22:53:53.362050+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:40.004661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:52:40.004661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:04.999859+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":1420,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-20T22:53:53.365843+0000","last_change":"2024-09-20T22:53:53.365864+0000","last_active":"2024-09-20T22:53:53.365843+0000","last_peered":"2024-09-20T22:53:53.365843+0000","last_clean":"2024-09-20T22:53:53.365843+0000","last_became_active":"2024-09-20T22:41:44.199395+0000","last_became_peered":"2024-09-20T22:41:44.199395+0000","last_unstale":"2024-09-20T22:53:53.365843+0000","last_undegraded":"2024-09-20T22:53:53.365843+0000","last_fullsized":"2024-09-20T22:53:53.365843+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:53.365789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:47:18.329209+0000","last_clean_scrub_stamp":"2024-09-20T22:53:53.365789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:06.119698+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.a","version":"751'902","reported_seq":4108,"reported_epoch":753,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T22:53:53.365720+0000","last_change":"2024-09-20T22:53:52.417594+0000","last_active":"2024-09-20T22:53:53.365720+0000","last_peered":"2024-09-20T22:53:53.365720+0000","last_clean":"2024-09-20T22:53:53.365720+0000","last_became_active":"2024-09-20T22:42:52.405085+0000","last_became_peered":"2024-09-20T22:42:52.405085+0000","last_unstale":"2024-09-20T22:53:53.365720+0000","last_undegraded":"2024-09-20T22:53:53.365720+0000","last_fullsized":"2024-09-20T22:53:53.365720+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"505'466","last_scrub_stamp":"2024-09-20T22:49:47.088165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:49:47.088165+0000","objects_scrubbed":13,"log_size":902,"log_dups_size":0,"ondisk_log_size":902,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":9,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.96227981399999996,"stat_sum":{"num_bytes":34688697,"num_objects":15,"num_object_clones":12,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":364,"num_read_kb":380456,"num_write":280,"num_write_kb":109585,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"10e","length":"4"},{"start":"113","length":"c"},{"start":"120","length":"1"},{"start":"122","length":"2"},{"start":"125","length":"2"},{"start":"129","length":"1"},{"start":"12b","length":"1"},{"start":"12d","length":"1"},{"start":"131","length":"1"},{"start":"133","length":"6"},{"start":"13f","length":"2"}]},{"pgid":"4.9","version":"686'23","reported_seq":786,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-20T22:53:53.301069+0000","last_change":"2024-09-20T22:52:46.768898+0000","last_active":"2024-09-20T22:53:53.301069+0000","last_peered":"2024-09-20T22:53:53.301069+0000","last_clean":"2024-09-20T22:53:53.301069+0000","last_became_active":"2024-09-20T22:52:11.286549+0000","last_became_peered":"2024-09-20T22:52:11.286549+0000","last_unstale":"2024-09-20T22:53:53.301069+0000","last_undegraded":"2024-09-20T22:53:53.301069+0000","last_fullsized":"2024-09-20T22:53:53.301069+0000","mapping_epoch":650,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":651,"parent":"0.0","parent_split_bits":0,"last_scrub":"686'23","last_scrub_stamp":"2024-09-20T22:52:46.768812+0000","last_deep_scrub":"686'23","last_deep_scrub_stamp":"2024-09-20T22:52:46.768812+0000","last_clean_scrub_stamp":"2024-09-20T22:52:46.768812+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":21,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":1462,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-20T22:53:53.314887+0000","last_change":"2024-09-20T22:53:35.478160+0000","last_active":"2024-09-20T22:53:53.314887+0000","last_peered":"2024-09-20T22:53:53.314887+0000","last_clean":"2024-09-20T22:53:53.314887+0000","last_became_active":"2024-09-20T22:51:20.293701+0000","last_became_peered":"2024-09-20T22:51:20.293701+0000","last_unstale":"2024-09-20T22:53:53.314887+0000","last_undegraded":"2024-09-20T22:53:53.314887+0000","last_fullsized":"2024-09-20T22:53:53.314887+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:35.478053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:46:52.465940+0000","last_clean_scrub_stamp":"2024-09-20T22:53:35.478053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:46.753808+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":1451,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-20T22:53:53.362209+0000","last_change":"2024-09-20T22:53:07.104977+0000","last_active":"2024-09-20T22:53:53.362209+0000","last_peered":"2024-09-20T22:53:53.362209+0000","last_clean":"2024-09-20T22:53:53.362209+0000","last_became_active":"2024-09-20T22:51:20.286626+0000","last_became_peered":"2024-09-20T22:51:20.286626+0000","last_unstale":"2024-09-20T22:53:53.362209+0000","last_undegraded":"2024-09-20T22:53:53.362209+0000","last_fullsized":"2024-09-20T22:53:53.362209+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:07.104891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:53:07.104891+0000","last_clean_scrub_stamp":"2024-09-20T22:53:07.104891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:17.124586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":1411,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-20T22:53:53.365362+0000","last_change":"2024-09-20T22:53:53.242276+0000","last_active":"2024-09-20T22:53:53.365362+0000","last_peered":"2024-09-20T22:53:53.365362+0000","last_clean":"2024-09-20T22:53:53.365362+0000","last_became_active":"2024-09-20T22:51:20.279088+0000","last_became_peered":"2024-09-20T22:51:20.279088+0000","last_unstale":"2024-09-20T22:53:53.365362+0000","last_undegraded":"2024-09-20T22:53:53.365362+0000","last_fullsized":"2024-09-20T22:53:53.365362+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:53.242204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:55.170379+0000","last_clean_scrub_stamp":"2024-09-20T22:53:53.242204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:14.854197+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"753'1178","reported_seq":4544,"reported_epoch":753,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T22:53:53.453992+0000","last_change":"2024-09-20T22:53:52.399543+0000","last_active":"2024-09-20T22:53:53.453992+0000","last_peered":"2024-09-20T22:53:53.453992+0000","last_clean":"2024-09-20T22:53:53.453992+0000","last_became_active":"2024-09-20T22:51:23.271244+0000","last_became_peered":"2024-09-20T22:51:23.271244+0000","last_unstale":"2024-09-20T22:53:53.453992+0000","last_undegraded":"2024-09-20T22:53:53.453992+0000","last_fullsized":"2024-09-20T22:53:53.453992+0000","mapping_epoch":602,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":603,"parent":"0.0","parent_split_bits":0,"last_scrub":"661'1000","last_scrub_stamp":"2024-09-20T22:52:23.164028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:52:23.164028+0000","objects_scrubbed":23,"log_size":1178,"log_dups_size":0,"ondisk_log_size":1178,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":11,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.93860152900000005,"stat_sum":{"num_bytes":40098527,"num_objects":18,"num_object_clones":13,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":3,"num_read":516,"num_read_kb":586632,"num_write":354,"num_write_kb":123866,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":76,"num_bytes_recovered":39288293,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"10e","length":"4"},{"start":"113","length":"c"},{"start":"120","length":"1"},{"start":"122","length":"2"},{"start":"125","length":"2"},{"start":"129","length":"1"},{"start":"12b","length":"1"},{"start":"12d","length":"1"},{"start":"131","length":"1"},{"start":"133","length":"6"}]},{"pgid":"4.8","version":"625'27","reported_seq":1457,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-20T22:53:53.453453+0000","last_change":"2024-09-20T22:51:53.020721+0000","last_active":"2024-09-20T22:53:53.453453+0000","last_peered":"2024-09-20T22:53:53.453453+0000","last_clean":"2024-09-20T22:53:53.453453+0000","last_became_active":"2024-09-20T22:41:40.252113+0000","last_became_peered":"2024-09-20T22:41:40.252113+0000","last_unstale":"2024-09-20T22:53:53.453453+0000","last_undegraded":"2024-09-20T22:53:53.453453+0000","last_fullsized":"2024-09-20T22:53:53.453453+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'27","last_scrub_stamp":"2024-09-20T22:51:53.020628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:51:53.020628+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'2","reported_seq":1487,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-20T22:53:53.315383+0000","last_change":"2024-09-20T22:53:33.526131+0000","last_active":"2024-09-20T22:53:53.315383+0000","last_peered":"2024-09-20T22:53:53.315383+0000","last_clean":"2024-09-20T22:53:53.315383+0000","last_became_active":"2024-09-20T22:48:05.238098+0000","last_became_peered":"2024-09-20T22:48:05.238098+0000","last_unstale":"2024-09-20T22:53:53.315383+0000","last_undegraded":"2024-09-20T22:53:53.315383+0000","last_fullsized":"2024-09-20T22:53:53.315383+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-09-20T22:53:33.526049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:53:33.526049+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:58.400288+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"408'10","reported_seq":1600,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-20T22:53:53.314553+0000","last_change":"2024-09-20T22:53:24.628363+0000","last_active":"2024-09-20T22:53:53.314553+0000","last_peered":"2024-09-20T22:53:53.314553+0000","last_clean":"2024-09-20T22:53:53.314553+0000","last_became_active":"2024-09-20T22:48:05.239344+0000","last_became_peered":"2024-09-20T22:48:05.239344+0000","last_unstale":"2024-09-20T22:53:53.314553+0000","last_undegraded":"2024-09-20T22:53:53.314553+0000","last_fullsized":"2024-09-20T22:53:53.314553+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"408'10","last_scrub_stamp":"2024-09-20T22:53:24.628269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:53:24.628269+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:33.393650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":1418,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-20T22:53:53.452946+0000","last_change":"2024-09-20T22:51:54.066995+0000","last_active":"2024-09-20T22:53:53.452946+0000","last_peered":"2024-09-20T22:53:53.452946+0000","last_clean":"2024-09-20T22:53:53.452946+0000","last_became_active":"2024-09-20T22:47:49.227569+0000","last_became_peered":"2024-09-20T22:47:49.227569+0000","last_unstale":"2024-09-20T22:53:53.452946+0000","last_undegraded":"2024-09-20T22:53:53.452946+0000","last_fullsized":"2024-09-20T22:53:53.452946+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:54.066913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:51:54.066913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":1453,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-20T22:53:53.455095+0000","last_change":"2024-09-20T22:53:11.873951+0000","last_active":"2024-09-20T22:53:53.455095+0000","last_peered":"2024-09-20T22:53:53.455095+0000","last_clean":"2024-09-20T22:53:53.455095+0000","last_became_active":"2024-09-20T22:47:49.213292+0000","last_became_peered":"2024-09-20T22:47:49.213292+0000","last_unstale":"2024-09-20T22:53:53.455095+0000","last_undegraded":"2024-09-20T22:53:53.455095+0000","last_fullsized":"2024-09-20T22:53:53.455095+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:11.873813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:53:11.873813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:18.510382+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"625'12","reported_seq":1453,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-20T22:53:53.365519+0000","last_change":"2024-09-20T22:53:26.319275+0000","last_active":"2024-09-20T22:53:53.365519+0000","last_peered":"2024-09-20T22:53:53.365519+0000","last_clean":"2024-09-20T22:53:53.365519+0000","last_became_active":"2024-09-20T22:43:09.116887+0000","last_became_peered":"2024-09-20T22:43:09.116887+0000","last_unstale":"2024-09-20T22:53:53.365519+0000","last_undegraded":"2024-09-20T22:53:53.365519+0000","last_fullsized":"2024-09-20T22:53:53.365519+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'12","last_scrub_stamp":"2024-09-20T22:53:26.319144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:53:26.319144+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:37.907176+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":1424,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-20T22:53:53.454148+0000","last_change":"2024-09-20T22:53:03.888115+0000","last_active":"2024-09-20T22:53:53.454148+0000","last_peered":"2024-09-20T22:53:53.454148+0000","last_clean":"2024-09-20T22:53:53.454148+0000","last_became_active":"2024-09-20T22:51:20.293232+0000","last_became_peered":"2024-09-20T22:51:20.293232+0000","last_unstale":"2024-09-20T22:53:53.454148+0000","last_undegraded":"2024-09-20T22:53:53.454148+0000","last_fullsized":"2024-09-20T22:53:53.454148+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:03.888040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:09.705632+0000","last_clean_scrub_stamp":"2024-09-20T22:53:03.888040+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:19.352989+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"74'1","reported_seq":1433,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-20T22:53:53.314765+0000","last_change":"2024-09-20T22:52:49.575819+0000","last_active":"2024-09-20T22:53:53.314765+0000","last_peered":"2024-09-20T22:53:53.314765+0000","last_clean":"2024-09-20T22:53:53.314765+0000","last_became_active":"2024-09-20T22:48:05.237882+0000","last_became_peered":"2024-09-20T22:48:05.237882+0000","last_unstale":"2024-09-20T22:53:53.314765+0000","last_undegraded":"2024-09-20T22:53:53.314765+0000","last_fullsized":"2024-09-20T22:53:53.314765+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-09-20T22:52:49.575702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:52:49.575702+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:06.855774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"625'15","reported_seq":784,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-20T22:53:53.303675+0000","last_change":"2024-09-20T22:53:50.889818+0000","last_active":"2024-09-20T22:53:53.303675+0000","last_peered":"2024-09-20T22:53:53.303675+0000","last_clean":"2024-09-20T22:53:53.303675+0000","last_became_active":"2024-09-20T22:47:49.231315+0000","last_became_peered":"2024-09-20T22:47:49.231315+0000","last_unstale":"2024-09-20T22:53:53.303675+0000","last_undegraded":"2024-09-20T22:53:53.303675+0000","last_fullsized":"2024-09-20T22:53:53.303675+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'15","last_scrub_stamp":"2024-09-20T22:53:50.889717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:53:50.889717+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:51.999731+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":1448,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-20T22:53:53.314197+0000","last_change":"2024-09-20T22:53:10.778908+0000","last_active":"2024-09-20T22:53:53.314197+0000","last_peered":"2024-09-20T22:53:53.314197+0000","last_clean":"2024-09-20T22:53:53.314197+0000","last_became_active":"2024-09-20T22:48:05.238801+0000","last_became_peered":"2024-09-20T22:48:05.238801+0000","last_unstale":"2024-09-20T22:53:53.314197+0000","last_undegraded":"2024-09-20T22:53:53.314197+0000","last_fullsized":"2024-09-20T22:53:53.314197+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:10.778836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:53:10.778836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:25.023394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":1415,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-20T22:53:53.365318+0000","last_change":"2024-09-20T22:52:20.410227+0000","last_active":"2024-09-20T22:53:53.365318+0000","last_peered":"2024-09-20T22:53:53.365318+0000","last_clean":"2024-09-20T22:53:53.365318+0000","last_became_active":"2024-09-20T22:51:20.279397+0000","last_became_peered":"2024-09-20T22:51:20.279397+0000","last_unstale":"2024-09-20T22:53:53.365318+0000","last_undegraded":"2024-09-20T22:53:53.365318+0000","last_fullsized":"2024-09-20T22:53:53.365318+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:20.410164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:52:20.410164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":1445,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-20T22:53:53.361886+0000","last_change":"2024-09-20T22:52:19.253411+0000","last_active":"2024-09-20T22:53:53.361886+0000","last_peered":"2024-09-20T22:53:53.361886+0000","last_clean":"2024-09-20T22:53:53.361886+0000","last_became_active":"2024-09-20T22:48:05.239710+0000","last_became_peered":"2024-09-20T22:48:05.239710+0000","last_unstale":"2024-09-20T22:53:53.361886+0000","last_undegraded":"2024-09-20T22:53:53.361886+0000","last_fullsized":"2024-09-20T22:53:53.361886+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:19.253343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:52:19.253343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"625'15","reported_seq":1447,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-20T22:53:53.454047+0000","last_change":"2024-09-20T22:53:53.070301+0000","last_active":"2024-09-20T22:53:53.454047+0000","last_peered":"2024-09-20T22:53:53.454047+0000","last_clean":"2024-09-20T22:53:53.454047+0000","last_became_active":"2024-09-20T22:41:40.252486+0000","last_became_peered":"2024-09-20T22:41:40.252486+0000","last_unstale":"2024-09-20T22:53:53.454047+0000","last_undegraded":"2024-09-20T22:53:53.454047+0000","last_fullsized":"2024-09-20T22:53:53.454047+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'15","last_scrub_stamp":"2024-09-20T22:53:53.070214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:53:53.070214+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:11.603607+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":765,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-20T22:53:53.302858+0000","last_change":"2024-09-20T22:52:13.735451+0000","last_active":"2024-09-20T22:53:53.302858+0000","last_peered":"2024-09-20T22:53:53.302858+0000","last_clean":"2024-09-20T22:53:53.302858+0000","last_became_active":"2024-09-20T22:47:49.229936+0000","last_became_peered":"2024-09-20T22:47:49.229936+0000","last_unstale":"2024-09-20T22:53:53.302858+0000","last_undegraded":"2024-09-20T22:53:53.302858+0000","last_fullsized":"2024-09-20T22:53:53.302858+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:13.735382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:52:13.735382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"74'8","reported_seq":1560,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-20T22:53:53.364810+0000","last_change":"2024-09-20T22:50:38.269118+0000","last_active":"2024-09-20T22:53:53.364810+0000","last_peered":"2024-09-20T22:53:53.364810+0000","last_clean":"2024-09-20T22:53:53.364810+0000","last_became_active":"2024-09-20T22:41:42.186352+0000","last_became_peered":"2024-09-20T22:41:42.186352+0000","last_unstale":"2024-09-20T22:53:53.364810+0000","last_undegraded":"2024-09-20T22:53:53.364810+0000","last_fullsized":"2024-09-20T22:53:53.364810+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-09-20T22:50:38.269010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:50:38.269010+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":724,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-20T22:53:53.302109+0000","last_change":"2024-09-20T22:52:16.623463+0000","last_active":"2024-09-20T22:53:53.302109+0000","last_peered":"2024-09-20T22:53:53.302109+0000","last_clean":"2024-09-20T22:53:53.302109+0000","last_became_active":"2024-09-20T22:48:05.249656+0000","last_became_peered":"2024-09-20T22:48:05.249656+0000","last_unstale":"2024-09-20T22:53:53.302109+0000","last_undegraded":"2024-09-20T22:53:53.302109+0000","last_fullsized":"2024-09-20T22:53:53.302109+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:16.623368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:52:16.623368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"625'20","reported_seq":1461,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-20T22:53:53.364541+0000","last_change":"2024-09-20T22:53:02.491402+0000","last_active":"2024-09-20T22:53:53.364541+0000","last_peered":"2024-09-20T22:53:53.364541+0000","last_clean":"2024-09-20T22:53:53.364541+0000","last_became_active":"2024-09-20T22:51:20.281859+0000","last_became_peered":"2024-09-20T22:51:20.281859+0000","last_unstale":"2024-09-20T22:53:53.364541+0000","last_undegraded":"2024-09-20T22:53:53.364541+0000","last_fullsized":"2024-09-20T22:53:53.364541+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'20","last_scrub_stamp":"2024-09-20T22:53:02.491320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:53:02.491320+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:10.839410+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":1436,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-20T22:53:53.314890+0000","last_change":"2024-09-20T22:52:20.621799+0000","last_active":"2024-09-20T22:53:53.314890+0000","last_peered":"2024-09-20T22:53:53.314890+0000","last_clean":"2024-09-20T22:53:53.314890+0000","last_became_active":"2024-09-20T22:48:05.240231+0000","last_became_peered":"2024-09-20T22:48:05.240231+0000","last_unstale":"2024-09-20T22:53:53.314890+0000","last_undegraded":"2024-09-20T22:53:53.314890+0000","last_fullsized":"2024-09-20T22:53:53.314890+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:20.621718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:52:20.621718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"600'11","reported_seq":1608,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-20T22:53:53.453257+0000","last_change":"2024-09-20T22:52:36.264515+0000","last_active":"2024-09-20T22:53:53.453257+0000","last_peered":"2024-09-20T22:53:53.453257+0000","last_clean":"2024-09-20T22:53:53.453257+0000","last_became_active":"2024-09-20T22:51:20.283393+0000","last_became_peered":"2024-09-20T22:51:20.283393+0000","last_unstale":"2024-09-20T22:53:53.453257+0000","last_undegraded":"2024-09-20T22:53:53.453257+0000","last_fullsized":"2024-09-20T22:53:53.453257+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"600'11","last_scrub_stamp":"2024-09-20T22:52:36.264410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:52:36.264410+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":757,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-20T22:53:53.303397+0000","last_change":"2024-09-20T22:52:44.834418+0000","last_active":"2024-09-20T22:53:53.303397+0000","last_peered":"2024-09-20T22:53:53.303397+0000","last_clean":"2024-09-20T22:53:53.303397+0000","last_became_active":"2024-09-20T22:47:49.230674+0000","last_became_peered":"2024-09-20T22:47:49.230674+0000","last_unstale":"2024-09-20T22:53:53.303397+0000","last_undegraded":"2024-09-20T22:53:53.303397+0000","last_fullsized":"2024-09-20T22:53:53.303397+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:44.834352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:50:21.853070+0000","last_clean_scrub_stamp":"2024-09-20T22:52:44.834352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:57.888226+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"625'19","reported_seq":774,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-20T22:53:53.301694+0000","last_change":"2024-09-20T22:53:49.547691+0000","last_active":"2024-09-20T22:53:53.301694+0000","last_peered":"2024-09-20T22:53:53.301694+0000","last_clean":"2024-09-20T22:53:53.301694+0000","last_became_active":"2024-09-20T22:51:20.289584+0000","last_became_peered":"2024-09-20T22:51:20.289584+0000","last_unstale":"2024-09-20T22:53:53.301694+0000","last_undegraded":"2024-09-20T22:53:53.301694+0000","last_fullsized":"2024-09-20T22:53:53.301694+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'19","last_scrub_stamp":"2024-09-20T22:53:49.547607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:53:49.547607+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:14.243559+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":732,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-20T22:53:53.301628+0000","last_change":"2024-09-20T22:52:47.717598+0000","last_active":"2024-09-20T22:53:53.301628+0000","last_peered":"2024-09-20T22:53:53.301628+0000","last_clean":"2024-09-20T22:53:53.301628+0000","last_became_active":"2024-09-20T22:48:05.252026+0000","last_became_peered":"2024-09-20T22:48:05.252026+0000","last_unstale":"2024-09-20T22:53:53.301628+0000","last_undegraded":"2024-09-20T22:53:53.301628+0000","last_fullsized":"2024-09-20T22:53:53.301628+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:47.717490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:52:47.717490+0000","last_clean_scrub_stamp":"2024-09-20T22:52:47.717490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:59.413281+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":319,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-20T22:53:53.453096+0000","last_change":"2024-09-20T22:51:20.291857+0000","last_active":"2024-09-20T22:53:53.453096+0000","last_peered":"2024-09-20T22:53:53.453096+0000","last_clean":"2024-09-20T22:53:53.453096+0000","last_became_active":"2024-09-20T22:51:20.291264+0000","last_became_peered":"2024-09-20T22:51:20.291264+0000","last_unstale":"2024-09-20T22:53:53.453096+0000","last_undegraded":"2024-09-20T22:53:53.453096+0000","last_fullsized":"2024-09-20T22:53:53.453096+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:06.651167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:51:06.651167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":1417,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-20T22:53:53.363971+0000","last_change":"2024-09-20T22:52:42.582474+0000","last_active":"2024-09-20T22:53:53.363971+0000","last_peered":"2024-09-20T22:53:53.363971+0000","last_clean":"2024-09-20T22:53:53.363971+0000","last_became_active":"2024-09-20T22:47:49.216141+0000","last_became_peered":"2024-09-20T22:47:49.216141+0000","last_unstale":"2024-09-20T22:53:53.363971+0000","last_undegraded":"2024-09-20T22:53:53.363971+0000","last_fullsized":"2024-09-20T22:53:53.363971+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:42.582398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:50:20.206984+0000","last_clean_scrub_stamp":"2024-09-20T22:52:42.582398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:57.745331+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"625'19","reported_seq":1071,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-20T22:53:53.455321+0000","last_change":"2024-09-20T22:52:40.792541+0000","last_active":"2024-09-20T22:53:53.455321+0000","last_peered":"2024-09-20T22:53:53.455321+0000","last_clean":"2024-09-20T22:53:53.455321+0000","last_became_active":"2024-09-20T22:51:20.283868+0000","last_became_peered":"2024-09-20T22:51:20.283868+0000","last_unstale":"2024-09-20T22:53:53.455321+0000","last_undegraded":"2024-09-20T22:53:53.455321+0000","last_fullsized":"2024-09-20T22:53:53.455321+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'19","last_scrub_stamp":"2024-09-20T22:52:40.792457+0000","last_deep_scrub":"404'13","last_deep_scrub_stamp":"2024-09-20T22:48:04.276282+0000","last_clean_scrub_stamp":"2024-09-20T22:52:40.792457+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:53:55.252741+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,0],"acting":[6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":1451,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-20T22:53:53.315140+0000","last_change":"2024-09-20T22:52:42.505280+0000","last_active":"2024-09-20T22:53:53.315140+0000","last_peered":"2024-09-20T22:53:53.315140+0000","last_clean":"2024-09-20T22:53:53.315140+0000","last_became_active":"2024-09-20T22:48:05.238812+0000","last_became_peered":"2024-09-20T22:48:05.238812+0000","last_unstale":"2024-09-20T22:53:53.315140+0000","last_undegraded":"2024-09-20T22:53:53.315140+0000","last_fullsized":"2024-09-20T22:53:53.315140+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:42.505211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:31.138067+0000","last_clean_scrub_stamp":"2024-09-20T22:52:42.505211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":1427,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-20T22:53:53.364622+0000","last_change":"2024-09-20T22:53:34.345146+0000","last_active":"2024-09-20T22:53:53.364622+0000","last_peered":"2024-09-20T22:53:53.364622+0000","last_clean":"2024-09-20T22:53:53.364622+0000","last_became_active":"2024-09-20T22:51:20.288000+0000","last_became_peered":"2024-09-20T22:51:20.288000+0000","last_unstale":"2024-09-20T22:53:53.364622+0000","last_undegraded":"2024-09-20T22:53:53.364622+0000","last_fullsized":"2024-09-20T22:53:53.364622+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:34.345077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:53:34.345077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:58.604576+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":763,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-20T22:53:53.302794+0000","last_change":"2024-09-20T22:53:09.890579+0000","last_active":"2024-09-20T22:53:53.302794+0000","last_peered":"2024-09-20T22:53:53.302794+0000","last_clean":"2024-09-20T22:53:53.302794+0000","last_became_active":"2024-09-20T22:51:20.278147+0000","last_became_peered":"2024-09-20T22:51:20.278147+0000","last_unstale":"2024-09-20T22:53:53.302794+0000","last_undegraded":"2024-09-20T22:53:53.302794+0000","last_fullsized":"2024-09-20T22:53:53.302794+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:09.890501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:53:09.890501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:36.773749+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"625'24","reported_seq":1248,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-20T22:53:53.452782+0000","last_change":"2024-09-20T22:53:33.089688+0000","last_active":"2024-09-20T22:53:53.452782+0000","last_peered":"2024-09-20T22:53:53.452782+0000","last_clean":"2024-09-20T22:53:53.452782+0000","last_became_active":"2024-09-20T22:44:09.170785+0000","last_became_peered":"2024-09-20T22:44:09.170785+0000","last_unstale":"2024-09-20T22:53:53.452782+0000","last_undegraded":"2024-09-20T22:53:53.452782+0000","last_fullsized":"2024-09-20T22:53:53.452782+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'24","last_scrub_stamp":"2024-09-20T22:53:33.089580+0000","last_deep_scrub":"470'16","last_deep_scrub_stamp":"2024-09-20T22:49:10.659868+0000","last_clean_scrub_stamp":"2024-09-20T22:53:33.089580+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:46.655555+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":1432,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-20T22:53:53.453417+0000","last_change":"2024-09-20T22:53:07.945728+0000","last_active":"2024-09-20T22:53:53.453417+0000","last_peered":"2024-09-20T22:53:53.453417+0000","last_clean":"2024-09-20T22:53:53.453417+0000","last_became_active":"2024-09-20T22:51:20.293645+0000","last_became_peered":"2024-09-20T22:51:20.293645+0000","last_unstale":"2024-09-20T22:53:53.453417+0000","last_undegraded":"2024-09-20T22:53:53.453417+0000","last_fullsized":"2024-09-20T22:53:53.453417+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:07.945664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:45:24.765414+0000","last_clean_scrub_stamp":"2024-09-20T22:53:07.945664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:21.914259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":1440,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-20T22:53:53.314010+0000","last_change":"2024-09-20T22:53:09.740214+0000","last_active":"2024-09-20T22:53:53.314010+0000","last_peered":"2024-09-20T22:53:53.314010+0000","last_clean":"2024-09-20T22:53:53.314010+0000","last_became_active":"2024-09-20T22:48:05.238334+0000","last_became_peered":"2024-09-20T22:48:05.238334+0000","last_unstale":"2024-09-20T22:53:53.314010+0000","last_undegraded":"2024-09-20T22:53:53.314010+0000","last_fullsized":"2024-09-20T22:53:53.314010+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:09.740145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:50.174861+0000","last_clean_scrub_stamp":"2024-09-20T22:53:09.740145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:33.910113+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":1409,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-20T22:53:53.365082+0000","last_change":"2024-09-20T22:51:04.379662+0000","last_active":"2024-09-20T22:53:53.365082+0000","last_peered":"2024-09-20T22:53:53.365082+0000","last_clean":"2024-09-20T22:53:53.365082+0000","last_became_active":"2024-09-20T22:47:49.214487+0000","last_became_peered":"2024-09-20T22:47:49.214487+0000","last_unstale":"2024-09-20T22:53:53.365082+0000","last_undegraded":"2024-09-20T22:53:53.365082+0000","last_fullsized":"2024-09-20T22:53:53.365082+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:04.379558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:51:04.379558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"625'8","reported_seq":1437,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-20T22:53:53.364060+0000","last_change":"2024-09-20T22:53:05.489357+0000","last_active":"2024-09-20T22:53:53.364060+0000","last_peered":"2024-09-20T22:53:53.364060+0000","last_clean":"2024-09-20T22:53:53.364060+0000","last_became_active":"2024-09-20T22:51:20.290269+0000","last_became_peered":"2024-09-20T22:51:20.290269+0000","last_unstale":"2024-09-20T22:53:53.364060+0000","last_undegraded":"2024-09-20T22:53:53.364060+0000","last_fullsized":"2024-09-20T22:53:53.364060+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'8","last_scrub_stamp":"2024-09-20T22:53:05.489279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:53:05.489279+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:30.438834+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":765,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-20T22:53:53.303308+0000","last_change":"2024-09-20T22:53:07.918178+0000","last_active":"2024-09-20T22:53:53.303308+0000","last_peered":"2024-09-20T22:53:53.303308+0000","last_clean":"2024-09-20T22:53:53.303308+0000","last_became_active":"2024-09-20T22:47:49.231964+0000","last_became_peered":"2024-09-20T22:47:49.231964+0000","last_unstale":"2024-09-20T22:53:53.303308+0000","last_undegraded":"2024-09-20T22:53:53.303308+0000","last_fullsized":"2024-09-20T22:53:53.303308+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:07.918108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:53:07.918108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:20.150259+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":1457,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-20T22:53:53.455297+0000","last_change":"2024-09-20T22:53:51.937667+0000","last_active":"2024-09-20T22:53:53.455297+0000","last_peered":"2024-09-20T22:53:53.455297+0000","last_clean":"2024-09-20T22:53:53.455297+0000","last_became_active":"2024-09-20T22:48:05.244659+0000","last_became_peered":"2024-09-20T22:48:05.244659+0000","last_unstale":"2024-09-20T22:53:53.455297+0000","last_undegraded":"2024-09-20T22:53:53.455297+0000","last_fullsized":"2024-09-20T22:53:53.455297+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:51.937584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:53:51.937584+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:07.795334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":1432,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-20T22:53:53.314248+0000","last_change":"2024-09-20T22:53:05.612142+0000","last_active":"2024-09-20T22:53:53.314248+0000","last_peered":"2024-09-20T22:53:53.314248+0000","last_clean":"2024-09-20T22:53:53.314248+0000","last_became_active":"2024-09-20T22:48:05.237765+0000","last_became_peered":"2024-09-20T22:48:05.237765+0000","last_unstale":"2024-09-20T22:53:53.314248+0000","last_undegraded":"2024-09-20T22:53:53.314248+0000","last_fullsized":"2024-09-20T22:53:53.314248+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:05.612030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:10.233487+0000","last_clean_scrub_stamp":"2024-09-20T22:53:05.612030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:09.643188+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":1022,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-20T22:53:53.453949+0000","last_change":"2024-09-20T22:51:29.904430+0000","last_active":"2024-09-20T22:53:53.453949+0000","last_peered":"2024-09-20T22:53:53.453949+0000","last_clean":"2024-09-20T22:53:53.453949+0000","last_became_active":"2024-09-20T22:51:20.297739+0000","last_became_peered":"2024-09-20T22:51:20.297739+0000","last_unstale":"2024-09-20T22:53:53.453949+0000","last_undegraded":"2024-09-20T22:53:53.453949+0000","last_fullsized":"2024-09-20T22:53:53.453949+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:29.904355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:51:29.904355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":1449,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-20T22:53:53.363884+0000","last_change":"2024-09-20T22:53:31.274457+0000","last_active":"2024-09-20T22:53:53.363884+0000","last_peered":"2024-09-20T22:53:53.363884+0000","last_clean":"2024-09-20T22:53:53.363884+0000","last_became_active":"2024-09-20T22:51:20.274891+0000","last_became_peered":"2024-09-20T22:51:20.274891+0000","last_unstale":"2024-09-20T22:53:53.363884+0000","last_undegraded":"2024-09-20T22:53:53.363884+0000","last_fullsized":"2024-09-20T22:53:53.363884+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:31.274390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:53:31.274390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:36.584363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"625'19","reported_seq":1489,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-20T22:53:53.456206+0000","last_change":"2024-09-20T22:53:30.953158+0000","last_active":"2024-09-20T22:53:53.456206+0000","last_peered":"2024-09-20T22:53:53.456206+0000","last_clean":"2024-09-20T22:53:53.456206+0000","last_became_active":"2024-09-20T22:51:20.284937+0000","last_became_peered":"2024-09-20T22:51:20.284937+0000","last_unstale":"2024-09-20T22:53:53.456206+0000","last_undegraded":"2024-09-20T22:53:53.456206+0000","last_fullsized":"2024-09-20T22:53:53.456206+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'19","last_scrub_stamp":"2024-09-20T22:53:30.953067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:53:30.953067+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:59.104815+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":758,"reported_epoch":753,"state":"active+clean","last_fresh":"2024-09-20T22:53:53.303460+0000","last_change":"2024-09-20T22:52:18.698154+0000","last_active":"2024-09-20T22:53:53.303460+0000","last_peered":"2024-09-20T22:53:53.303460+0000","last_clean":"2024-09-20T22:53:53.303460+0000","last_became_active":"2024-09-20T22:47:49.220971+0000","last_became_peered":"2024-09-20T22:47:49.220971+0000","last_unstale":"2024-09-20T22:53:53.303460+0000","last_undegraded":"2024-09-20T22:53:53.303460+0000","last_fullsized":"2024-09-20T22:53:53.303460+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:18.698087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:51.667341+0000","last_clean_scrub_stamp":"2024-09-20T22:52:18.698087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":567427465,"num_objects":255,"num_object_clones":206,"num_object_copies":765,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":255,"num_whiteouts":14,"num_read":5195,"num_read_kb":5423943,"num_write":3762,"num_write_kb":1400104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":790,"num_bytes_recovered":519843158,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1195835392,"data_stored":1747202346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":11319,"ondisk_log_size":11319,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86,"ondisk_log_size":86,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1109,"num_read_kb":866,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":312,"num_bytes_recovered":7464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":499712,"data_stored":13224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":744,"ondisk_log_size":744,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":1034,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":4931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":725,"num_read_kb":720,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1186,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":384,"ondisk_log_size":384,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070615,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":246888,"kb_used_data":193488,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93498264,"statfs":{"total":95995035648,"available":95742222336,"internally_reserved":0,"allocated":198131712,"data_stored":276097899,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":54655434},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430365,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":220656,"kb_used_data":167256,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93524496,"statfs":{"total":95995035648,"available":95769083904,"internally_reserved":0,"allocated":171270144,"data_stored":232875645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798692002,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":176356,"kb_used_data":118792,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":93568796,"statfs":{"total":95995035648,"available":95814447104,"internally_reserved":0,"allocated":121643008,"data_stored":175287761,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986343,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":127132,"kb_used_data":95944,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93618020,"statfs":{"total":95995035648,"available":95864852480,"internally_reserved":0,"allocated":98246656,"data_stored":139124463,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182573,"num_pgs":76,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":342716,"kb_used_data":267112,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93402436,"statfs":{"total":95995035648,"available":95644094464,"internally_reserved":0,"allocated":273522688,"data_stored":394054948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":77396423},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411506,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":245524,"kb_used_data":196288,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":93499628,"statfs":{"total":95995035648,"available":95743619072,"internally_reserved":0,"allocated":200998912,"data_stored":299620527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":55834575031,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738556,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":227888,"kb_used_data":178652,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":93517264,"statfs":{"total":95995035648,"available":95761678336,"internally_reserved":0,"allocated":182939648,"data_stored":256443297,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":175915008,"data_stored":253966032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":193740800,"data_stored":295087504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":265904128,"data_stored":390986621,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":91222016,"data_stored":134846773,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":114044928,"data_stored":170425854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":164241408,"data_stored":230397999,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":190767104,"data_stored":271491563,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-20T22:53:58.573 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-09-20T22:53:58.573 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-20T22:53:58.573 INFO:tasks.thrashosds.thrasher:inject_pause on osd.4 2024-09-20T22:53:58.573 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-20T22:53:58.573 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-20T22:53:58.573 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.4 config set filestore_inject_stall 3 2024-09-20T22:53:58.574 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:53:58.590 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:53:58.594 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:53:58.616 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:58.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:58.617 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:53:58.718 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:53:58.739 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:58.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:58.740 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:53:58.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:58 smithi154 ceph-mon[33264]: 3.1d scrub starts 2024-09-20T22:53:58.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:58 smithi154 ceph-mon[33264]: 3.1d scrub ok 2024-09-20T22:53:58.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:58 smithi154 ceph-mon[33264]: 4.4 scrub starts 2024-09-20T22:53:58.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:58 smithi154 ceph-mon[33264]: 4.4 scrub ok 2024-09-20T22:53:58.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:58 smithi154 ceph-mon[33264]: 5.c scrub starts 2024-09-20T22:53:58.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:58 smithi154 ceph-mon[33264]: 5.c scrub ok 2024-09-20T22:53:58.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:58 smithi154 ceph-mon[33264]: osdmap e757: 8 total, 8 up, 7 in 2024-09-20T22:53:58.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:58 smithi154 ceph-mon[33264]: 6.0 scrub starts 2024-09-20T22:53:58.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:58 smithi154 ceph-mon[33264]: 6.0 scrub ok 2024-09-20T22:53:58.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:58 smithi154 ceph-mon[33264]: 5.2 scrub starts 2024-09-20T22:53:58.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:58 smithi154 ceph-mon[33264]: 5.2 scrub ok 2024-09-20T22:53:58.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:58 smithi154 ceph-mon[33264]: osdmap e758: 8 total, 8 up, 7 in 2024-09-20T22:53:58.832 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:58 smithi084 ceph-mon[40024]: 3.1d scrub starts 2024-09-20T22:53:58.832 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:58 smithi084 ceph-mon[40024]: 3.1d scrub ok 2024-09-20T22:53:58.832 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:58 smithi084 ceph-mon[40024]: 4.4 scrub starts 2024-09-20T22:53:58.832 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:58 smithi084 ceph-mon[40024]: 4.4 scrub ok 2024-09-20T22:53:58.832 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:58 smithi084 ceph-mon[40024]: 5.c scrub starts 2024-09-20T22:53:58.832 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:58 smithi084 ceph-mon[40024]: 5.c scrub ok 2024-09-20T22:53:58.832 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:58 smithi084 ceph-mon[40024]: osdmap e757: 8 total, 8 up, 7 in 2024-09-20T22:53:58.833 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:58 smithi084 ceph-mon[40024]: 6.0 scrub starts 2024-09-20T22:53:58.833 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:58 smithi084 ceph-mon[40024]: 6.0 scrub ok 2024-09-20T22:53:58.833 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:58 smithi084 ceph-mon[40024]: 5.2 scrub starts 2024-09-20T22:53:58.833 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:58 smithi084 ceph-mon[40024]: 5.2 scrub ok 2024-09-20T22:53:58.833 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:58 smithi084 ceph-mon[40024]: osdmap e758: 8 total, 8 up, 7 in 2024-09-20T22:53:58.833 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:58 smithi084 ceph-mon[29774]: 3.1d scrub starts 2024-09-20T22:53:58.833 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:58 smithi084 ceph-mon[29774]: 3.1d scrub ok 2024-09-20T22:53:58.833 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:58 smithi084 ceph-mon[29774]: 4.4 scrub starts 2024-09-20T22:53:58.833 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:58 smithi084 ceph-mon[29774]: 4.4 scrub ok 2024-09-20T22:53:58.833 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:58 smithi084 ceph-mon[29774]: 5.c scrub starts 2024-09-20T22:53:58.833 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:58 smithi084 ceph-mon[29774]: 5.c scrub ok 2024-09-20T22:53:58.833 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:58 smithi084 ceph-mon[29774]: osdmap e757: 8 total, 8 up, 7 in 2024-09-20T22:53:58.834 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:58 smithi084 ceph-mon[29774]: 6.0 scrub starts 2024-09-20T22:53:58.834 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:58 smithi084 ceph-mon[29774]: 6.0 scrub ok 2024-09-20T22:53:58.834 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:58 smithi084 ceph-mon[29774]: 5.2 scrub starts 2024-09-20T22:53:58.834 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:58 smithi084 ceph-mon[29774]: 5.2 scrub ok 2024-09-20T22:53:58.834 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:58 smithi084 ceph-mon[29774]: osdmap e758: 8 total, 8 up, 7 in 2024-09-20T22:53:58.841 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:53:58.863 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:58.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:58.864 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:53:58.866 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:53:58.964 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:53:58.985 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:58.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:58.986 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:53:59.087 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:53:59.108 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:59.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:59.109 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:53:59.209 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:53:59.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:59.232 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:59.233 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:53:59.305 INFO:tasks.rados.rados.0.smithi084.stdout:3016: done (3 left) 2024-09-20T22:53:59.305 INFO:tasks.rados.rados.0.smithi084.stdout:3017: done (2 left) 2024-09-20T22:53:59.305 INFO:tasks.rados.rados.0.smithi084.stdout:3018: done (1 left) 2024-09-20T22:53:59.306 INFO:tasks.rados.rados.0.smithi084.stdout:3019: done (0 left) 2024-09-20T22:53:59.306 INFO:tasks.rados.rados.0.smithi084.stdout:3020: delete oid 44 current snap is 342 2024-09-20T22:53:59.312 INFO:tasks.rados.rados.0.smithi084.stdout:3020: done (0 left) 2024-09-20T22:53:59.312 INFO:tasks.rados.rados.0.smithi084.stdout:3021: delete oid 27 current snap is 342 2024-09-20T22:53:59.312 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-20T22:53:59.312 INFO:teuthology.orchestra.run.smithi154.stdout: "success": "" 2024-09-20T22:53:59.312 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-20T22:53:59.314 INFO:tasks.rados.rados.0.smithi084.stdout:3021: done (0 left) 2024-09-20T22:53:59.314 INFO:tasks.rados.rados.0.smithi084.stdout:3022: copy_from oid 33 from oid 41 current snap is 342 2024-09-20T22:53:59.314 INFO:tasks.rados.rados.0.smithi084.stdout:3023: snap_remove snap 315 2024-09-20T22:53:59.333 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:53:59.355 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:59.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:59.355 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:53:59.356 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:53:59.356 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:53:59.372 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:53:59.372 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:53:59.387 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:53:59.388 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:53:59.403 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-20T22:53:59.404 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T22:53:59.420 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:53:59.420 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:53:59.437 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:53:59.438 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:53:59.454 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:53:59.454 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:53:59.456 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:53:59.472 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:53:59.472 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:53:59.479 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:59.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:59.479 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:53:59.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:59 smithi154 ceph-mon[33264]: pgmap v1038: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 542 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail 2024-09-20T22:53:59.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:59 smithi154 ceph-mon[33264]: 6.1d scrub starts 2024-09-20T22:53:59.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:59 smithi154 ceph-mon[33264]: 6.1d scrub ok 2024-09-20T22:53:59.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:59 smithi154 ceph-mon[33264]: 6.14 deep-scrub starts 2024-09-20T22:53:59.534 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:59 smithi154 ceph-mon[33264]: 6.14 deep-scrub ok 2024-09-20T22:53:59.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:59 smithi154 ceph-mon[33264]: 6.1a scrub starts 2024-09-20T22:53:59.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:59 smithi154 ceph-mon[33264]: 6.1a scrub ok 2024-09-20T22:53:59.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:59 smithi154 ceph-mon[33264]: 1.2 scrub starts 2024-09-20T22:53:59.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:59 smithi154 ceph-mon[33264]: 1.2 scrub ok 2024-09-20T22:53:59.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:59 smithi154 ceph-mon[33264]: 3.12 scrub starts 2024-09-20T22:53:59.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:59 smithi154 ceph-mon[33264]: 3.12 scrub ok 2024-09-20T22:53:59.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:59 smithi154 ceph-mon[33264]: 4.12 scrub starts 2024-09-20T22:53:59.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:59 smithi154 ceph-mon[33264]: 4.12 scrub ok 2024-09-20T22:53:59.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:59 smithi154 ceph-mon[33264]: osdmap e759: 8 total, 8 up, 7 in 2024-09-20T22:53:59.535 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:53:59 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/260843860' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:53:59.581 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:53:59.603 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:53:59.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:59.604 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:53:59.614 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:59 smithi084 ceph-mon[40024]: pgmap v1038: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 542 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail 2024-09-20T22:53:59.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:59 smithi084 ceph-mon[40024]: 6.1d scrub starts 2024-09-20T22:53:59.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:59 smithi084 ceph-mon[40024]: 6.1d scrub ok 2024-09-20T22:53:59.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:59 smithi084 ceph-mon[40024]: 6.14 deep-scrub starts 2024-09-20T22:53:59.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:59 smithi084 ceph-mon[40024]: 6.14 deep-scrub ok 2024-09-20T22:53:59.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:59 smithi084 ceph-mon[40024]: 6.1a scrub starts 2024-09-20T22:53:59.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:59 smithi084 ceph-mon[40024]: 6.1a scrub ok 2024-09-20T22:53:59.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:59 smithi084 ceph-mon[40024]: 1.2 scrub starts 2024-09-20T22:53:59.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:59 smithi084 ceph-mon[40024]: 1.2 scrub ok 2024-09-20T22:53:59.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:59 smithi084 ceph-mon[40024]: 3.12 scrub starts 2024-09-20T22:53:59.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:59 smithi084 ceph-mon[40024]: 3.12 scrub ok 2024-09-20T22:53:59.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:59 smithi084 ceph-mon[40024]: 4.12 scrub starts 2024-09-20T22:53:59.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:59 smithi084 ceph-mon[40024]: 4.12 scrub ok 2024-09-20T22:53:59.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:59 smithi084 ceph-mon[40024]: osdmap e759: 8 total, 8 up, 7 in 2024-09-20T22:53:59.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:53:59 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/260843860' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:53:59.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:59 smithi084 ceph-mon[29774]: pgmap v1038: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 542 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail 2024-09-20T22:53:59.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:59 smithi084 ceph-mon[29774]: 6.1d scrub starts 2024-09-20T22:53:59.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:59 smithi084 ceph-mon[29774]: 6.1d scrub ok 2024-09-20T22:53:59.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:59 smithi084 ceph-mon[29774]: 6.14 deep-scrub starts 2024-09-20T22:53:59.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:59 smithi084 ceph-mon[29774]: 6.14 deep-scrub ok 2024-09-20T22:53:59.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:59 smithi084 ceph-mon[29774]: 6.1a scrub starts 2024-09-20T22:53:59.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:59 smithi084 ceph-mon[29774]: 6.1a scrub ok 2024-09-20T22:53:59.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:59 smithi084 ceph-mon[29774]: 1.2 scrub starts 2024-09-20T22:53:59.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:59 smithi084 ceph-mon[29774]: 1.2 scrub ok 2024-09-20T22:53:59.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:59 smithi084 ceph-mon[29774]: 3.12 scrub starts 2024-09-20T22:53:59.617 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:59 smithi084 ceph-mon[29774]: 3.12 scrub ok 2024-09-20T22:53:59.617 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:59 smithi084 ceph-mon[29774]: 4.12 scrub starts 2024-09-20T22:53:59.617 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:59 smithi084 ceph-mon[29774]: 4.12 scrub ok 2024-09-20T22:53:59.617 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:59 smithi084 ceph-mon[29774]: osdmap e759: 8 total, 8 up, 7 in 2024-09-20T22:53:59.617 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:53:59 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/260843860' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:53:59.694 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-20T22:53:59.694 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:53:59.705 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:53:59.726 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:59.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:59.727 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:53:59.816 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd set noscrub 2024-09-20T22:53:59.827 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:53:59.882 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:53:59.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:53:59.883 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:53:59.983 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:53:59.995 INFO:journalctl@ceph.iscsi.iscsi.a.smithi154.stdout:Sep 20 22:53:59 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-iscsi-iscsi-a[59730]: debug there is no tcmu-runner data available 2024-09-20T22:54:00.006 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:00.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:00.007 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:54:00.108 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:54:00.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:00.129 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:00.129 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:54:00.230 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:54:00.252 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:00.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:00.253 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:54:00.310 INFO:tasks.rados.rados.0.smithi084.stdout:3023: done (1 left) 2024-09-20T22:54:00.310 INFO:tasks.rados.rados.0.smithi084.stdout:3024: read oid 39 snap -1 2024-09-20T22:54:00.311 INFO:tasks.rados.rados.0.smithi084.stdout:3024: expect deleted 2024-09-20T22:54:00.311 INFO:tasks.rados.rados.0.smithi084.stdout:3025: snap_remove snap 313 2024-09-20T22:54:00.345 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-20T22:54:00.354 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:54:00.375 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:00.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:00.376 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:54:00.476 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:54:00.498 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:00.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:00.499 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:54:00.600 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:54:00.603 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:54:00.614 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:00 smithi084 ceph-mon[29774]: 5.14 scrub starts 2024-09-20T22:54:00.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:00 smithi084 ceph-mon[29774]: 5.14 scrub ok 2024-09-20T22:54:00.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:00 smithi084 ceph-mon[29774]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:54:00.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:00 smithi084 ceph-mon[29774]: purged_snaps scrub starts 2024-09-20T22:54:00.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:00 smithi084 ceph-mon[29774]: purged_snaps scrub ok 2024-09-20T22:54:00.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:00 smithi084 ceph-mon[29774]: osdmap e760: 8 total, 8 up, 7 in 2024-09-20T22:54:00.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:00 smithi084 ceph-mon[40024]: 5.14 scrub starts 2024-09-20T22:54:00.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:00 smithi084 ceph-mon[40024]: 5.14 scrub ok 2024-09-20T22:54:00.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:00 smithi084 ceph-mon[40024]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:54:00.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:00 smithi084 ceph-mon[40024]: purged_snaps scrub starts 2024-09-20T22:54:00.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:00 smithi084 ceph-mon[40024]: purged_snaps scrub ok 2024-09-20T22:54:00.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:00 smithi084 ceph-mon[40024]: osdmap e760: 8 total, 8 up, 7 in 2024-09-20T22:54:00.623 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:00.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:00.624 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:54:00.724 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:00 smithi154 ceph-mon[33264]: 5.14 scrub starts 2024-09-20T22:54:00.724 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:00 smithi154 ceph-mon[33264]: 5.14 scrub ok 2024-09-20T22:54:00.724 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:00 smithi154 ceph-mon[33264]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:54:00.724 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:00 smithi154 ceph-mon[33264]: purged_snaps scrub starts 2024-09-20T22:54:00.724 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:00 smithi154 ceph-mon[33264]: purged_snaps scrub ok 2024-09-20T22:54:00.725 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:00 smithi154 ceph-mon[33264]: osdmap e760: 8 total, 8 up, 7 in 2024-09-20T22:54:00.725 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:54:00.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:00.747 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:00.747 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:54:00.848 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:54:00.869 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:00.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:00.870 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:54:00.971 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:54:00.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:00.993 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:00.993 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:54:01.094 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:54:01.115 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:01.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:01.116 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:54:01.121 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-20T22:54:01.121 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:54:01.217 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:54:01.234 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:54:01.239 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:01.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:01.240 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:54:01.308 INFO:tasks.rados.rados.0.smithi084.stdout:3025: done (2 left) 2024-09-20T22:54:01.309 INFO:tasks.rados.rados.0.smithi084.stdout:3026: write oid 46 current snap is 342 2024-09-20T22:54:01.309 INFO:tasks.rados.rados.0.smithi084.stdout:3022: finishing copy_from racing read to smithi08477203-33 2024-09-20T22:54:01.309 INFO:tasks.rados.rados.0.smithi084.stdout:3026: seq_num 1046 ranges {780122=650815,2126165=799355} 2024-09-20T22:54:01.317 INFO:tasks.rados.rados.0.smithi084.stdout:3026: writing smithi08477203-46 from 780122 to 1430937 tid 1 2024-09-20T22:54:01.327 INFO:tasks.rados.rados.0.smithi084.stdout:3026: writing smithi08477203-46 from 2126165 to 2925520 tid 2 2024-09-20T22:54:01.327 INFO:tasks.rados.rados.0.smithi084.stdout:3027: write oid 20 current snap is 342 2024-09-20T22:54:01.327 INFO:tasks.rados.rados.0.smithi084.stdout:3022: finishing copy_from to smithi08477203-33 2024-09-20T22:54:01.327 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 33 v 1829 (ObjNum 984 snap 319 seq_num 984) dirty exists 2024-09-20T22:54:01.327 INFO:tasks.rados.rados.0.smithi084.stdout:3027: seq_num 1047 ranges {796929=726064,2184647=659366,3302344=294585} 2024-09-20T22:54:01.335 INFO:tasks.rados.rados.0.smithi084.stdout:3027: writing smithi08477203-20 from 796929 to 1522993 tid 1 2024-09-20T22:54:01.340 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:54:01.343 INFO:tasks.rados.rados.0.smithi084.stdout:3027: writing smithi08477203-20 from 2184647 to 2844013 tid 2 2024-09-20T22:54:01.347 INFO:tasks.rados.rados.0.smithi084.stdout:3027: writing smithi08477203-20 from 3302344 to 3596929 tid 3 2024-09-20T22:54:01.347 INFO:tasks.rados.rados.0.smithi084.stdout:3022: done (3 left) 2024-09-20T22:54:01.347 INFO:tasks.rados.rados.0.smithi084.stdout:3024: done (2 left) 2024-09-20T22:54:01.347 INFO:tasks.rados.rados.0.smithi084.stdout:3028: read oid 27 snap -1 2024-09-20T22:54:01.347 INFO:tasks.rados.rados.0.smithi084.stdout:3028: expect deleted 2024-09-20T22:54:01.347 INFO:tasks.rados.rados.0.smithi084.stdout:3029: read oid 11 snap -1 2024-09-20T22:54:01.347 INFO:tasks.rados.rados.0.smithi084.stdout:3029: expect (ObjNum 1043 snap 341 seq_num 1043) 2024-09-20T22:54:01.348 INFO:tasks.rados.rados.0.smithi084.stdout:3030: read oid 1 snap 327 2024-09-20T22:54:01.348 INFO:tasks.rados.rados.0.smithi084.stdout:3030: expect (ObjNum 738 snap 235 seq_num 738) 2024-09-20T22:54:01.348 INFO:tasks.rados.rados.0.smithi084.stdout:3031: read oid 28 snap -1 2024-09-20T22:54:01.348 INFO:tasks.rados.rados.0.smithi084.stdout:3031: expect deleted 2024-09-20T22:54:01.348 INFO:tasks.rados.rados.0.smithi084.stdout:3026: finishing write tid 1 to smithi08477203-46 2024-09-20T22:54:01.348 INFO:tasks.rados.rados.0.smithi084.stdout:3026: finishing write tid 2 to smithi08477203-46 2024-09-20T22:54:01.348 INFO:tasks.rados.rados.0.smithi084.stdout:3026: finishing write tid 3 to smithi08477203-46 2024-09-20T22:54:01.348 INFO:tasks.rados.rados.0.smithi084.stdout:3026: finishing write tid 4 to smithi08477203-46 2024-09-20T22:54:01.348 INFO:tasks.rados.rados.0.smithi084.stdout:3026: oid 46 updating version 0 to 1857 2024-09-20T22:54:01.348 INFO:tasks.rados.rados.0.smithi084.stdout:3026: oid 46 version 1857 is already newer than 1856 2024-09-20T22:54:01.348 INFO:tasks.rados.rados.0.smithi084.stdout:3026: oid 46 version 1857 is already newer than 1855 2024-09-20T22:54:01.348 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 46 v 1857 (ObjNum 1046 snap 342 seq_num 1046) dirty exists 2024-09-20T22:54:01.348 INFO:tasks.rados.rados.0.smithi084.stdout:3026: left oid 46 (ObjNum 1046 snap 342 seq_num 1046) 2024-09-20T22:54:01.349 INFO:tasks.rados.rados.0.smithi084.stdout:3027: finishing write tid 1 to smithi08477203-20 2024-09-20T22:54:01.349 INFO:tasks.rados.rados.0.smithi084.stdout:3027: finishing write tid 2 to smithi08477203-20 2024-09-20T22:54:01.349 INFO:tasks.rados.rados.0.smithi084.stdout:3026: done (5 left) 2024-09-20T22:54:01.349 INFO:tasks.rados.rados.0.smithi084.stdout:3032: copy_from oid 42 from oid 23 current snap is 342 2024-09-20T22:54:01.349 INFO:tasks.rados.rados.0.smithi084.stdout:3033: snap_create 2024-09-20T22:54:01.351 INFO:tasks.rados.rados.0.smithi084.stdout:3027: finishing write tid 3 to smithi08477203-20 2024-09-20T22:54:01.351 INFO:tasks.rados.rados.0.smithi084.stdout:3027: finishing write tid 4 to smithi08477203-20 2024-09-20T22:54:01.351 INFO:tasks.rados.rados.0.smithi084.stdout:3027: finishing write tid 5 to smithi08477203-20 2024-09-20T22:54:01.351 INFO:tasks.rados.rados.0.smithi084.stdout:3027: oid 20 updating version 0 to 1910 2024-09-20T22:54:01.351 INFO:tasks.rados.rados.0.smithi084.stdout:3027: oid 20 version 1910 is already newer than 1909 2024-09-20T22:54:01.351 INFO:tasks.rados.rados.0.smithi084.stdout:3027: oid 20 version 1910 is already newer than 1908 2024-09-20T22:54:01.351 INFO:tasks.rados.rados.0.smithi084.stdout:3027: oid 20 version 1910 is already newer than 1907 2024-09-20T22:54:01.351 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 20 v 1910 (ObjNum 1047 snap 342 seq_num 1047) dirty exists 2024-09-20T22:54:01.352 INFO:tasks.rados.rados.0.smithi084.stdout:3027: left oid 20 (ObjNum 1047 snap 342 seq_num 1047) 2024-09-20T22:54:01.356 INFO:tasks.rados.rados.0.smithi084.stdout:3029: expect (ObjNum 1043 snap 341 seq_num 1043) 2024-09-20T22:54:01.366 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:01.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:01.367 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:54:01.468 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:54:01.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:01.490 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:01.491 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:54:01.591 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:54:01.612 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:01.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:01.613 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:54:01.637 INFO:tasks.rados.rados.0.smithi084.stdout:3032: finishing copy_from racing read to smithi08477203-42 2024-09-20T22:54:01.637 INFO:tasks.rados.rados.0.smithi084.stdout:3030: expect (ObjNum 738 snap 235 seq_num 738) 2024-09-20T22:54:01.709 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:01 smithi154 ceph-mon[33264]: pgmap v1041: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 544 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 2.5 MiB/s rd, 2.3 MiB/s wr, 9 op/s 2024-09-20T22:54:01.709 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:01 smithi154 ceph-mon[33264]: osdmap e761: 8 total, 8 up, 7 in 2024-09-20T22:54:01.713 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:54:01.720 INFO:tasks.rados.rados.0.smithi084.stdout:3032: finishing copy_from to smithi08477203-42 2024-09-20T22:54:01.720 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 42 v 1904 (ObjNum 957 snap 313 seq_num 957) dirty exists 2024-09-20T22:54:01.735 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:01.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:01.736 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:54:01.738 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.4 dump_blocked_ops 2024-09-20T22:54:01.837 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:54:01.857 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:01.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:01.858 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:54:01.864 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:01 smithi084 ceph-mon[29774]: pgmap v1041: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 544 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 2.5 MiB/s rd, 2.3 MiB/s wr, 9 op/s 2024-09-20T22:54:01.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:01 smithi084 ceph-mon[29774]: osdmap e761: 8 total, 8 up, 7 in 2024-09-20T22:54:01.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:01 smithi084 ceph-mon[40024]: pgmap v1041: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 544 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 2.5 MiB/s rd, 2.3 MiB/s wr, 9 op/s 2024-09-20T22:54:01.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:01 smithi084 ceph-mon[40024]: osdmap e761: 8 total, 8 up, 7 in 2024-09-20T22:54:01.960 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:54:01.981 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:01.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:01.982 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:54:02.023 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:54:02.083 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:54:02.105 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:02.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:02.106 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:54:02.206 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:54:02.230 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:02.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:02.231 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:54:02.310 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is set 2024-09-20T22:54:02.310 INFO:tasks.rados.rados.0.smithi084.stdout:3027: done (6 left) 2024-09-20T22:54:02.311 INFO:tasks.rados.rados.0.smithi084.stdout:3028: done (5 left) 2024-09-20T22:54:02.311 INFO:tasks.rados.rados.0.smithi084.stdout:3029: done (4 left) 2024-09-20T22:54:02.311 INFO:tasks.rados.rados.0.smithi084.stdout:3030: done (3 left) 2024-09-20T22:54:02.311 INFO:tasks.rados.rados.0.smithi084.stdout:3031: done (2 left) 2024-09-20T22:54:02.311 INFO:tasks.rados.rados.0.smithi084.stdout:3032: done (1 left) 2024-09-20T22:54:02.311 INFO:tasks.rados.rados.0.smithi084.stdout:3033: done (0 left) 2024-09-20T22:54:02.311 INFO:tasks.rados.rados.0.smithi084.stdout:3034: rollback oid 28 current snap is 343 2024-09-20T22:54:02.311 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 28 to 312 2024-09-20T22:54:02.311 INFO:tasks.rados.rados.0.smithi084.stdout:3035: snap_remove snap 336 2024-09-20T22:54:02.331 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:54:02.353 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:02.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:02.354 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:54:02.454 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:54:02.462 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-20T22:54:02.462 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:54:02.477 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:02.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:02.478 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:54:02.579 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:54:02.600 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:02.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:02.601 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:54:02.614 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:02 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/4139565865' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-20T22:54:02.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:02 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-20T22:54:02.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:02 smithi084 ceph-mon[29774]: 6.1f scrub starts 2024-09-20T22:54:02.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:02 smithi084 ceph-mon[29774]: 6.1f scrub ok 2024-09-20T22:54:02.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:02 smithi084 ceph-mon[29774]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:54:02.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:02 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T22:54:02.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:02 smithi084 ceph-mon[29774]: osdmap e762: 8 total, 8 up, 7 in 2024-09-20T22:54:02.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:02 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/4139565865' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-20T22:54:02.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:02 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-20T22:54:02.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:02 smithi084 ceph-mon[40024]: 6.1f scrub starts 2024-09-20T22:54:02.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:02 smithi084 ceph-mon[40024]: 6.1f scrub ok 2024-09-20T22:54:02.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:02 smithi084 ceph-mon[40024]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:54:02.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:02 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T22:54:02.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:02 smithi084 ceph-mon[40024]: osdmap e762: 8 total, 8 up, 7 in 2024-09-20T22:54:02.702 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:54:02.725 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:02.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:02.726 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:54:02.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:02 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/4139565865' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-20T22:54:02.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:02 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-20T22:54:02.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:02 smithi154 ceph-mon[33264]: 6.1f scrub starts 2024-09-20T22:54:02.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:02 smithi154 ceph-mon[33264]: 6.1f scrub ok 2024-09-20T22:54:02.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:02 smithi154 ceph-mon[33264]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:54:02.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:02 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T22:54:02.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:02 smithi154 ceph-mon[33264]: osdmap e762: 8 total, 8 up, 7 in 2024-09-20T22:54:02.827 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:54:02.849 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:02.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:02.850 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:54:02.950 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:54:02.972 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:02.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:02.972 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:54:03.073 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:54:03.095 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:03.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:03.096 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:54:03.157 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-20T22:54:03.159 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.4 dump_historic_ops 2024-09-20T22:54:03.196 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:54:03.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:03.223 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:03.223 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:54:03.306 INFO:tasks.rados.rados.0.smithi084.stdout:3035: done (1 left) 2024-09-20T22:54:03.306 INFO:tasks.rados.rados.0.smithi084.stdout:3036: write oid 24 current snap is 343 2024-09-20T22:54:03.306 INFO:tasks.rados.rados.0.smithi084.stdout:3036: seq_num 1048 ranges {413736=401313,1326776=403024,2413414=457893,3438716=175020} 2024-09-20T22:54:03.310 INFO:tasks.rados.rados.0.smithi084.stdout:3036: writing smithi08477203-24 from 413736 to 815049 tid 1 2024-09-20T22:54:03.316 INFO:tasks.rados.rados.0.smithi084.stdout:3036: writing smithi08477203-24 from 1326776 to 1729800 tid 2 2024-09-20T22:54:03.324 INFO:tasks.rados.rados.0.smithi084.stdout:3036: writing smithi08477203-24 from 2413414 to 2871307 tid 3 2024-09-20T22:54:03.324 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:54:03.329 INFO:tasks.rados.rados.0.smithi084.stdout:3036: writing smithi08477203-24 from 3438716 to 3613736 tid 4 2024-09-20T22:54:03.330 INFO:tasks.rados.rados.0.smithi084.stdout:3037: snap_remove snap 293 2024-09-20T22:54:03.348 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:03.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:03.349 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:54:03.415 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:54:03.427 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:54:03.450 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:54:03.471 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:03.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:03.472 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:54:03.572 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:54:03.593 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:03.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:03.594 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:54:03.695 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:54:03.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:03.717 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:03.717 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:54:03.795 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:03 smithi154 ceph-mon[33264]: pgmap v1044: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 544 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 2.5 MiB/s rd, 2.3 MiB/s wr, 9 op/s 2024-09-20T22:54:03.795 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:03 smithi154 ceph-mon[33264]: osdmap e763: 8 total, 8 up, 7 in 2024-09-20T22:54:03.816 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:03 smithi084 ceph-mon[40024]: pgmap v1044: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 544 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 2.5 MiB/s rd, 2.3 MiB/s wr, 9 op/s 2024-09-20T22:54:03.816 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:03 smithi084 ceph-mon[40024]: osdmap e763: 8 total, 8 up, 7 in 2024-09-20T22:54:03.816 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:03 smithi084 ceph-mon[29774]: pgmap v1044: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 544 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 2.5 MiB/s rd, 2.3 MiB/s wr, 9 op/s 2024-09-20T22:54:03.816 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:03 smithi084 ceph-mon[29774]: osdmap e763: 8 total, 8 up, 7 in 2024-09-20T22:54:03.817 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:54:03.839 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:03.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:03.840 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:54:03.914 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:54:03.914 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:54:03.930 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:54:03.930 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:54:03.941 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:54:03.946 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:54:03.946 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:54:03.962 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-20T22:54:03.962 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T22:54:03.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:03.963 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:03.964 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:54:03.978 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:54:03.979 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:54:03.995 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:54:03.995 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:54:04.011 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:54:04.012 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:54:04.028 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:54:04.028 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:54:04.036 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-20T22:54:04.036 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:54:04.065 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:54:04.086 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:04.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:04.087 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:54:04.188 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:54:04.209 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:04.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:04.211 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:54:04.311 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:54:04.320 INFO:tasks.rados.rados.0.smithi084.stdout:3037: done (2 left) 2024-09-20T22:54:04.320 INFO:tasks.rados.rados.0.smithi084.stdout:3038: read oid 35 snap -1 2024-09-20T22:54:04.320 INFO:tasks.rados.rados.0.smithi084.stdout:3038: expect deleted 2024-09-20T22:54:04.321 INFO:tasks.rados.rados.0.smithi084.stdout:3039: snap_remove snap 319 2024-09-20T22:54:04.339 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:04.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:04.340 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:54:04.441 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:54:04.462 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:04.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:04.463 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:54:04.564 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:54:04.586 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:04.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:04.587 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:54:04.614 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:04 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/2482885527' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:54:04.614 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:04 smithi084 ceph-mon[29774]: osdmap e764: 8 total, 8 up, 7 in 2024-09-20T22:54:04.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:04 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/2482885527' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:54:04.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:04 smithi084 ceph-mon[40024]: osdmap e764: 8 total, 8 up, 7 in 2024-09-20T22:54:04.687 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:54:04.699 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:04 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/2482885527' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:54:04.699 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:04 smithi154 ceph-mon[33264]: osdmap e764: 8 total, 8 up, 7 in 2024-09-20T22:54:04.709 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:04.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:04.710 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:54:04.725 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-20T22:54:04.811 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:54:04.832 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:04.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:04.833 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:54:04.934 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:54:04.956 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:04.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:04.957 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:54:04.972 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:54:05.054 INFO:tasks.thrashosds.thrasher:in_osds: [2, 3, 5, 6, 7, 0, 4] out_osds: [1] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-20T22:54:05.055 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-20T22:54:05.055 INFO:tasks.thrashosds.thrasher:fixing pg num pool .mgr 2024-09-20T22:54:05.055 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph pg dump --format=json 2024-09-20T22:54:05.057 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:54:05.078 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:05.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:05.079 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:54:05.110 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-20T22:54:05.181 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:54:05.202 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:05.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:05.203 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:54:05.304 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:54:05.320 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:54:05.322 INFO:tasks.rados.rados.0.smithi084.stdout:3039: done (3 left) 2024-09-20T22:54:05.322 INFO:tasks.rados.rados.0.smithi084.stdout:3040: snap_create 2024-09-20T22:54:05.322 INFO:tasks.rados.rados.0.smithi084.stdout:3034: finishing rollback tid 1 to smithi08477203-28 2024-09-20T22:54:05.322 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 28 v 2189 (ObjNum 33 snap 0 seq_num 2091956688) dirty dne 2024-09-20T22:54:05.322 INFO:tasks.rados.rados.0.smithi084.stdout:3036: finishing write tid 1 to smithi08477203-24 2024-09-20T22:54:05.322 INFO:tasks.rados.rados.0.smithi084.stdout:3036: finishing write tid 2 to smithi08477203-24 2024-09-20T22:54:05.323 INFO:tasks.rados.rados.0.smithi084.stdout:3036: finishing write tid 3 to smithi08477203-24 2024-09-20T22:54:05.323 INFO:tasks.rados.rados.0.smithi084.stdout:3036: finishing write tid 4 to smithi08477203-24 2024-09-20T22:54:05.323 INFO:tasks.rados.rados.0.smithi084.stdout:3036: finishing write tid 5 to smithi08477203-24 2024-09-20T22:54:05.323 INFO:tasks.rados.rados.0.smithi084.stdout:3036: finishing write tid 6 to smithi08477203-24 2024-09-20T22:54:05.323 INFO:tasks.rados.rados.0.smithi084.stdout:3036: oid 24 updating version 0 to 1873 2024-09-20T22:54:05.323 INFO:tasks.rados.rados.0.smithi084.stdout:3036: oid 24 version 1873 is already newer than 1870 2024-09-20T22:54:05.323 INFO:tasks.rados.rados.0.smithi084.stdout:3036: oid 24 version 1873 is already newer than 1871 2024-09-20T22:54:05.323 INFO:tasks.rados.rados.0.smithi084.stdout:3036: oid 24 version 1873 is already newer than 1869 2024-09-20T22:54:05.323 INFO:tasks.rados.rados.0.smithi084.stdout:3036: oid 24 version 1873 is already newer than 1872 2024-09-20T22:54:05.323 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 24 v 1873 (ObjNum 1048 snap 343 seq_num 1048) dirty exists 2024-09-20T22:54:05.323 INFO:tasks.rados.rados.0.smithi084.stdout:3036: left oid 24 (ObjNum 1048 snap 343 seq_num 1048) 2024-09-20T22:54:05.338 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:05.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:05.339 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:54:05.440 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:54:05.444 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-20T22:54:05.444 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [], 2024-09-20T22:54:05.444 INFO:teuthology.orchestra.run.smithi154.stdout: "num_ops": 0 2024-09-20T22:54:05.444 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-20T22:54:05.464 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:05.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:05.465 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:54:05.566 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:54:05.587 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:05.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:05.588 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:54:05.689 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:54:05.710 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:05.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:05.711 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:54:05.741 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:05 smithi084 ceph-mon[40024]: pgmap v1047: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 553 MiB data, 1.6 GiB used, 624 GiB / 626 GiB avail; 7.3 MiB/s rd, 2.2 MiB/s wr, 11 op/s 2024-09-20T22:54:05.741 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:05 smithi084 ceph-mon[40024]: osdmap e765: 8 total, 8 up, 7 in 2024-09-20T22:54:05.741 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:05 smithi084 ceph-mon[29774]: pgmap v1047: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 553 MiB data, 1.6 GiB used, 624 GiB / 626 GiB avail; 7.3 MiB/s rd, 2.2 MiB/s wr, 11 op/s 2024-09-20T22:54:05.741 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:05 smithi084 ceph-mon[29774]: osdmap e765: 8 total, 8 up, 7 in 2024-09-20T22:54:05.747 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:54:05.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:05 smithi154 ceph-mon[33264]: pgmap v1047: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 553 MiB data, 1.6 GiB used, 624 GiB / 626 GiB avail; 7.3 MiB/s rd, 2.2 MiB/s wr, 11 op/s 2024-09-20T22:54:05.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:05 smithi154 ceph-mon[33264]: osdmap e765: 8 total, 8 up, 7 in 2024-09-20T22:54:05.812 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:54:05.833 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:05.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:05.834 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:54:05.935 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:54:05.946 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-20T22:54:05.948 INFO:teuthology.orchestra.run.smithi084.stderr:dumped all 2024-09-20T22:54:05.955 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:05.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:05.956 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:54:06.057 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:54:06.078 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:06.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:06.079 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:54:06.160 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.5 dump_blocked_ops 2024-09-20T22:54:06.180 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:54:06.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:06.202 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:06.202 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:54:06.303 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:54:06.317 INFO:tasks.rados.rados.0.smithi084.stdout:3034: done (3 left) 2024-09-20T22:54:06.317 INFO:tasks.rados.rados.0.smithi084.stdout:3036: done (2 left) 2024-09-20T22:54:06.317 INFO:tasks.rados.rados.0.smithi084.stdout:3038: done (1 left) 2024-09-20T22:54:06.317 INFO:tasks.rados.rados.0.smithi084.stdout:3040: done (0 left) 2024-09-20T22:54:06.317 INFO:tasks.rados.rados.0.smithi084.stdout:3041: write oid 36 current snap is 344 2024-09-20T22:54:06.318 INFO:tasks.rados.rados.0.smithi084.stdout:3041: seq_num 1049 ranges {430543=476562,1385258=663035,2840837=669165} 2024-09-20T22:54:06.322 INFO:tasks.rados.rados.0.smithi084.stdout:3041: writing smithi08477203-36 from 430543 to 907105 tid 1 2024-09-20T22:54:06.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:06.327 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:06.327 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:54:06.328 INFO:tasks.rados.rados.0.smithi084.stdout:3041: writing smithi08477203-36 from 1385258 to 2048293 tid 2 2024-09-20T22:54:06.337 INFO:tasks.rados.rados.0.smithi084.stdout:3041: writing smithi08477203-36 from 2840837 to 3510002 tid 3 2024-09-20T22:54:06.337 INFO:tasks.rados.rados.0.smithi084.stdout:3042: rollback oid 14 current snap is 344 2024-09-20T22:54:06.337 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 14 to 332 2024-09-20T22:54:06.337 INFO:tasks.rados.rados.0.smithi084.stdout:3041: finishing write tid 1 to smithi08477203-36 2024-09-20T22:54:06.337 INFO:tasks.rados.rados.0.smithi084.stdout:3041: finishing write tid 2 to smithi08477203-36 2024-09-20T22:54:06.337 INFO:tasks.rados.rados.0.smithi084.stdout:3043: copy_from oid 42 from oid 39 current snap is 344 2024-09-20T22:54:06.337 INFO:tasks.rados.rados.0.smithi084.stdout:3044: copy_from oid 17 from oid 4 current snap is 344 2024-09-20T22:54:06.338 INFO:tasks.rados.rados.0.smithi084.stdout:3045: snap_create 2024-09-20T22:54:06.341 INFO:tasks.rados.rados.0.smithi084.stdout:3042: finishing rollback tid 0 to smithi08477203-14 2024-09-20T22:54:06.342 INFO:tasks.rados.rados.0.smithi084.stdout:3043: finishing copy_from to smithi08477203-42 2024-09-20T22:54:06.342 INFO:tasks.rados.rados.0.smithi084.stdout:3043: got expected ENOENT (src dne) 2024-09-20T22:54:06.342 INFO:tasks.rados.rados.0.smithi084.stdout:3043: finishing copy_from racing read to smithi08477203-42 2024-09-20T22:54:06.342 INFO:tasks.rados.rados.0.smithi084.stdout:3042: finishing rollback tid 1 to smithi08477203-14 2024-09-20T22:54:06.342 INFO:tasks.rados.rados.0.smithi084.stdout:3041: finishing write tid 3 to smithi08477203-36 2024-09-20T22:54:06.342 INFO:tasks.rados.rados.0.smithi084.stdout:3042: finishing rollback tid 2 to smithi08477203-14 2024-09-20T22:54:06.342 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 14 v 1913 (ObjNum 879 snap 285 seq_num 879) dirty exists 2024-09-20T22:54:06.342 INFO:tasks.rados.rados.0.smithi084.stdout:3041: finishing write tid 4 to smithi08477203-36 2024-09-20T22:54:06.342 INFO:tasks.rados.rados.0.smithi084.stdout:3041: finishing write tid 5 to smithi08477203-36 2024-09-20T22:54:06.342 INFO:tasks.rados.rados.0.smithi084.stdout:3041: oid 36 updating version 0 to 1907 2024-09-20T22:54:06.343 INFO:tasks.rados.rados.0.smithi084.stdout:3041: oid 36 version 1907 is already newer than 1905 2024-09-20T22:54:06.343 INFO:tasks.rados.rados.0.smithi084.stdout:3041: oid 36 updating version 1907 to 1908 2024-09-20T22:54:06.343 INFO:tasks.rados.rados.0.smithi084.stdout:3041: oid 36 version 1908 is already newer than 1906 2024-09-20T22:54:06.343 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 36 v 1908 (ObjNum 1049 snap 344 seq_num 1049) dirty exists 2024-09-20T22:54:06.343 INFO:tasks.rados.rados.0.smithi084.stdout:3041: left oid 36 (ObjNum 1049 snap 344 seq_num 1049) 2024-09-20T22:54:06.348 INFO:tasks.rados.rados.0.smithi084.stdout:3044: finishing copy_from racing read to smithi08477203-17 2024-09-20T22:54:06.351 INFO:tasks.rados.rados.0.smithi084.stdout:3044: finishing copy_from to smithi08477203-17 2024-09-20T22:54:06.352 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 17 v 1253 (ObjNum 964 snap 314 seq_num 964) dirty exists 2024-09-20T22:54:06.413 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:54:06.428 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:54:06.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:06.451 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:06.451 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:54:06.552 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:54:06.573 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:06.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:06.574 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:54:06.675 INFO:teuthology.orchestra.run.smithi084.stdout:{"pg_ready":true,"pg_map":{"version":1050,"stamp":"2024-09-20T22:54:05.810100+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":579875362,"num_objects":458,"num_object_clones":210,"num_object_copies":1374,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":458,"num_whiteouts":15,"num_read":7102,"num_read_kb":5465781,"num_write":4522,"num_write_kb":1418548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1142,"num_bytes_recovered":520180548,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":12884,"ondisk_log_size":12884,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":474,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":1630512,"kb_used_data":1252560,"kb_used_omap":10,"kb_used_meta":377781,"kb_avail":654585552,"statfs":{"total":671965249536,"available":670295605248,"internally_reserved":0,"allocated":1282621440,"data_stored":1808897354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11157,"internal_metadata":386847851},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":9103224,"num_objects":4,"num_object_clones":4,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":2,"num_read":29,"num_read_kb":30090,"num_write":19,"num_write_kb":8979,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001047"},"pg_stats":[{"pgid":"3.1f","version":"86'2","reported_seq":782,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-20T22:54:03.306816+0000","last_change":"2024-09-20T22:52:21.630419+0000","last_active":"2024-09-20T22:54:03.306816+0000","last_peered":"2024-09-20T22:54:03.306816+0000","last_clean":"2024-09-20T22:54:03.306816+0000","last_became_active":"2024-09-20T22:47:49.222114+0000","last_became_peered":"2024-09-20T22:47:49.222114+0000","last_unstale":"2024-09-20T22:54:03.306816+0000","last_undegraded":"2024-09-20T22:54:03.306816+0000","last_fullsized":"2024-09-20T22:54:03.306816+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'2","last_scrub_stamp":"2024-09-20T22:52:21.630304+0000","last_deep_scrub":"86'2","last_deep_scrub_stamp":"2024-09-20T22:42:39.664421+0000","last_clean_scrub_stamp":"2024-09-20T22:52:21.630304+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:07.874019+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":436,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"625'15","reported_seq":779,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-20T22:54:03.302865+0000","last_change":"2024-09-20T22:51:51.813358+0000","last_active":"2024-09-20T22:54:03.302865+0000","last_peered":"2024-09-20T22:54:03.302865+0000","last_clean":"2024-09-20T22:54:03.302865+0000","last_became_active":"2024-09-20T22:51:20.286651+0000","last_became_peered":"2024-09-20T22:51:20.286651+0000","last_unstale":"2024-09-20T22:54:03.302865+0000","last_undegraded":"2024-09-20T22:54:03.302865+0000","last_fullsized":"2024-09-20T22:54:03.302865+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'15","last_scrub_stamp":"2024-09-20T22:51:51.813269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:51:51.813269+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":347,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-20T22:54:03.314664+0000","last_change":"2024-09-20T22:52:41.184388+0000","last_active":"2024-09-20T22:54:03.314664+0000","last_peered":"2024-09-20T22:54:03.314664+0000","last_clean":"2024-09-20T22:54:03.314664+0000","last_became_active":"2024-09-20T22:51:20.291272+0000","last_became_peered":"2024-09-20T22:51:20.291272+0000","last_unstale":"2024-09-20T22:54:03.314664+0000","last_undegraded":"2024-09-20T22:54:03.314664+0000","last_fullsized":"2024-09-20T22:54:03.314664+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:41.184318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:52:41.184318+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:04.632649+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":758,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-20T22:54:03.303410+0000","last_change":"2024-09-20T22:53:58.481454+0000","last_active":"2024-09-20T22:54:03.303410+0000","last_peered":"2024-09-20T22:54:03.303410+0000","last_clean":"2024-09-20T22:54:03.303410+0000","last_became_active":"2024-09-20T22:51:20.295608+0000","last_became_peered":"2024-09-20T22:51:20.295608+0000","last_unstale":"2024-09-20T22:54:03.303410+0000","last_undegraded":"2024-09-20T22:54:03.303410+0000","last_fullsized":"2024-09-20T22:54:03.303410+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:58.481327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:51:36.730015+0000","last_clean_scrub_stamp":"2024-09-20T22:53:58.481327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:11.441168+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":1424,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-20T22:54:03.304228+0000","last_change":"2024-09-20T22:51:01.404127+0000","last_active":"2024-09-20T22:54:03.304228+0000","last_peered":"2024-09-20T22:54:03.304228+0000","last_clean":"2024-09-20T22:54:03.304228+0000","last_became_active":"2024-09-20T22:41:44.832099+0000","last_became_peered":"2024-09-20T22:41:44.832099+0000","last_unstale":"2024-09-20T22:54:03.304228+0000","last_undegraded":"2024-09-20T22:54:03.304228+0000","last_fullsized":"2024-09-20T22:54:03.304228+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:01.404063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:51:01.404063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:08.164450+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":1430,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-20T22:54:03.305686+0000","last_change":"2024-09-20T22:51:12.418041+0000","last_active":"2024-09-20T22:54:03.305686+0000","last_peered":"2024-09-20T22:54:03.305686+0000","last_clean":"2024-09-20T22:54:03.305686+0000","last_became_active":"2024-09-20T22:41:38.157298+0000","last_became_peered":"2024-09-20T22:41:38.157298+0000","last_unstale":"2024-09-20T22:54:03.305686+0000","last_undegraded":"2024-09-20T22:54:03.305686+0000","last_fullsized":"2024-09-20T22:54:03.305686+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:12.417920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:51:12.417920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:08.164514+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"625'27","reported_seq":1512,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-20T22:54:03.303903+0000","last_change":"2024-09-20T22:51:08.441927+0000","last_active":"2024-09-20T22:54:03.303903+0000","last_peered":"2024-09-20T22:54:03.303903+0000","last_clean":"2024-09-20T22:54:03.303903+0000","last_became_active":"2024-09-20T22:47:49.218798+0000","last_became_peered":"2024-09-20T22:47:49.218798+0000","last_unstale":"2024-09-20T22:54:03.303903+0000","last_undegraded":"2024-09-20T22:54:03.303903+0000","last_fullsized":"2024-09-20T22:54:03.303903+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-09-20T22:51:08.441788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:51:08.441788+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:08.164482+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":750,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-20T22:54:03.302755+0000","last_change":"2024-09-20T22:52:14.594895+0000","last_active":"2024-09-20T22:54:03.302755+0000","last_peered":"2024-09-20T22:54:03.302755+0000","last_clean":"2024-09-20T22:54:03.302755+0000","last_became_active":"2024-09-20T22:51:20.287060+0000","last_became_peered":"2024-09-20T22:51:20.287060+0000","last_unstale":"2024-09-20T22:54:03.302755+0000","last_undegraded":"2024-09-20T22:54:03.302755+0000","last_fullsized":"2024-09-20T22:54:03.302755+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:14.594823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:52:14.594823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":1457,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-20T22:54:03.316673+0000","last_change":"2024-09-20T22:53:56.980951+0000","last_active":"2024-09-20T22:54:03.316673+0000","last_peered":"2024-09-20T22:54:03.316673+0000","last_clean":"2024-09-20T22:54:03.316673+0000","last_became_active":"2024-09-20T22:48:05.242059+0000","last_became_peered":"2024-09-20T22:48:05.242059+0000","last_unstale":"2024-09-20T22:54:03.316673+0000","last_undegraded":"2024-09-20T22:54:03.316673+0000","last_fullsized":"2024-09-20T22:54:03.316673+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:56.980885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:46:00.999939+0000","last_clean_scrub_stamp":"2024-09-20T22:53:56.980885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:13.013721+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"625'15","reported_seq":771,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-20T22:54:03.303121+0000","last_change":"2024-09-20T22:53:51.486324+0000","last_active":"2024-09-20T22:54:03.303121+0000","last_peered":"2024-09-20T22:54:03.303121+0000","last_clean":"2024-09-20T22:54:03.303121+0000","last_became_active":"2024-09-20T22:48:05.233429+0000","last_became_peered":"2024-09-20T22:48:05.233429+0000","last_unstale":"2024-09-20T22:54:03.303121+0000","last_undegraded":"2024-09-20T22:54:03.303121+0000","last_fullsized":"2024-09-20T22:54:03.303121+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'15","last_scrub_stamp":"2024-09-20T22:53:51.486202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:53:51.486202+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:15.973503+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":1441,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-20T22:54:03.315484+0000","last_change":"2024-09-20T22:51:33.977328+0000","last_active":"2024-09-20T22:54:03.315484+0000","last_peered":"2024-09-20T22:54:03.315484+0000","last_clean":"2024-09-20T22:54:03.315484+0000","last_became_active":"2024-09-20T22:47:49.227379+0000","last_became_peered":"2024-09-20T22:47:49.227379+0000","last_unstale":"2024-09-20T22:54:03.315484+0000","last_undegraded":"2024-09-20T22:54:03.315484+0000","last_fullsized":"2024-09-20T22:54:03.315484+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:33.977257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:51:33.977257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:08.101697+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":791,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-20T22:54:03.302629+0000","last_change":"2024-09-20T22:53:35.921037+0000","last_active":"2024-09-20T22:54:03.302629+0000","last_peered":"2024-09-20T22:54:03.302629+0000","last_clean":"2024-09-20T22:54:03.302629+0000","last_became_active":"2024-09-20T22:51:20.277465+0000","last_became_peered":"2024-09-20T22:51:20.277465+0000","last_unstale":"2024-09-20T22:54:03.302629+0000","last_undegraded":"2024-09-20T22:54:03.302629+0000","last_fullsized":"2024-09-20T22:54:03.302629+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:35.920968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:49:08.664407+0000","last_clean_scrub_stamp":"2024-09-20T22:53:35.920968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:03.583550+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":1463,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-20T22:54:03.314982+0000","last_change":"2024-09-20T22:53:31.092991+0000","last_active":"2024-09-20T22:54:03.314982+0000","last_peered":"2024-09-20T22:54:03.314982+0000","last_clean":"2024-09-20T22:54:03.314982+0000","last_became_active":"2024-09-20T22:51:20.297787+0000","last_became_peered":"2024-09-20T22:51:20.297787+0000","last_unstale":"2024-09-20T22:54:03.314982+0000","last_undegraded":"2024-09-20T22:54:03.314982+0000","last_fullsized":"2024-09-20T22:54:03.314982+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:31.092915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:53:31.092915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:41.403638+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"625'9","reported_seq":773,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-20T22:54:03.303421+0000","last_change":"2024-09-20T22:53:09.623541+0000","last_active":"2024-09-20T22:54:03.303421+0000","last_peered":"2024-09-20T22:54:03.303421+0000","last_clean":"2024-09-20T22:54:03.303421+0000","last_became_active":"2024-09-20T22:51:20.285795+0000","last_became_peered":"2024-09-20T22:51:20.285795+0000","last_unstale":"2024-09-20T22:54:03.303421+0000","last_undegraded":"2024-09-20T22:54:03.303421+0000","last_fullsized":"2024-09-20T22:54:03.303421+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'9","last_scrub_stamp":"2024-09-20T22:53:09.623458+0000","last_deep_scrub":"181'7","last_deep_scrub_stamp":"2024-09-20T22:48:41.049381+0000","last_clean_scrub_stamp":"2024-09-20T22:53:09.623458+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:34.232945+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":9,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":1468,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-20T22:54:03.308616+0000","last_change":"2024-09-20T22:52:16.705733+0000","last_active":"2024-09-20T22:54:03.308616+0000","last_peered":"2024-09-20T22:54:03.308616+0000","last_clean":"2024-09-20T22:54:03.308616+0000","last_became_active":"2024-09-20T22:51:20.285368+0000","last_became_peered":"2024-09-20T22:51:20.285368+0000","last_unstale":"2024-09-20T22:54:03.308616+0000","last_undegraded":"2024-09-20T22:54:03.308616+0000","last_fullsized":"2024-09-20T22:54:03.308616+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:16.705645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:47:40.361958+0000","last_clean_scrub_stamp":"2024-09-20T22:52:16.705645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":1443,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-20T22:54:03.314724+0000","last_change":"2024-09-20T22:53:52.116314+0000","last_active":"2024-09-20T22:54:03.314724+0000","last_peered":"2024-09-20T22:54:03.314724+0000","last_clean":"2024-09-20T22:54:03.314724+0000","last_became_active":"2024-09-20T22:51:20.279307+0000","last_became_peered":"2024-09-20T22:51:20.279307+0000","last_unstale":"2024-09-20T22:54:03.314724+0000","last_undegraded":"2024-09-20T22:54:03.314724+0000","last_fullsized":"2024-09-20T22:54:03.314724+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:52.116243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:53:52.116243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:12.673739+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":353,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-20T22:54:03.303499+0000","last_change":"2024-09-20T22:53:48.558689+0000","last_active":"2024-09-20T22:54:03.303499+0000","last_peered":"2024-09-20T22:54:03.303499+0000","last_clean":"2024-09-20T22:54:03.303499+0000","last_became_active":"2024-09-20T22:51:20.282393+0000","last_became_peered":"2024-09-20T22:51:20.282393+0000","last_unstale":"2024-09-20T22:54:03.303499+0000","last_undegraded":"2024-09-20T22:54:03.303499+0000","last_fullsized":"2024-09-20T22:54:03.303499+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:48.558619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:53:48.558619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:02.645042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":783,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-20T22:54:03.303178+0000","last_change":"2024-09-20T22:52:42.745105+0000","last_active":"2024-09-20T22:54:03.303178+0000","last_peered":"2024-09-20T22:54:03.303178+0000","last_clean":"2024-09-20T22:54:03.303178+0000","last_became_active":"2024-09-20T22:48:05.233687+0000","last_became_peered":"2024-09-20T22:48:05.233687+0000","last_unstale":"2024-09-20T22:54:03.303178+0000","last_undegraded":"2024-09-20T22:54:03.303178+0000","last_fullsized":"2024-09-20T22:54:03.303178+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:42.745032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:49:53.933601+0000","last_clean_scrub_stamp":"2024-09-20T22:52:42.745032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:07.874145+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"625'34","reported_seq":1528,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-20T22:54:03.319307+0000","last_change":"2024-09-20T22:53:09.119883+0000","last_active":"2024-09-20T22:54:03.319307+0000","last_peered":"2024-09-20T22:54:03.319307+0000","last_clean":"2024-09-20T22:54:03.319307+0000","last_became_active":"2024-09-20T22:51:20.276338+0000","last_became_peered":"2024-09-20T22:51:20.276338+0000","last_unstale":"2024-09-20T22:54:03.319307+0000","last_undegraded":"2024-09-20T22:54:03.319307+0000","last_fullsized":"2024-09-20T22:54:03.319307+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'34","last_scrub_stamp":"2024-09-20T22:53:09.119791+0000","last_deep_scrub":"113'22","last_deep_scrub_stamp":"2024-09-20T22:43:12.191878+0000","last_clean_scrub_stamp":"2024-09-20T22:53:09.119791+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:36.423122+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":1057,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-20T22:54:03.303516+0000","last_change":"2024-09-20T22:52:43.758236+0000","last_active":"2024-09-20T22:54:03.303516+0000","last_peered":"2024-09-20T22:54:03.303516+0000","last_clean":"2024-09-20T22:54:03.303516+0000","last_became_active":"2024-09-20T22:51:20.295564+0000","last_became_peered":"2024-09-20T22:51:20.295564+0000","last_unstale":"2024-09-20T22:54:03.303516+0000","last_undegraded":"2024-09-20T22:54:03.303516+0000","last_fullsized":"2024-09-20T22:54:03.303516+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:43.758162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:50:40.784154+0000","last_clean_scrub_stamp":"2024-09-20T22:52:43.758162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:10.526451+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":1425,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-20T22:54:03.306255+0000","last_change":"2024-09-20T22:54:02.131730+0000","last_active":"2024-09-20T22:54:03.306255+0000","last_peered":"2024-09-20T22:54:03.306255+0000","last_clean":"2024-09-20T22:54:03.306255+0000","last_became_active":"2024-09-20T22:41:44.832131+0000","last_became_peered":"2024-09-20T22:41:44.832131+0000","last_unstale":"2024-09-20T22:54:03.306255+0000","last_undegraded":"2024-09-20T22:54:03.306255+0000","last_fullsized":"2024-09-20T22:54:03.306255+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:54:02.131659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:45:25.002199+0000","last_clean_scrub_stamp":"2024-09-20T22:54:02.131659+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:08.048458+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":758,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-20T22:54:03.303323+0000","last_change":"2024-09-20T22:53:04.583753+0000","last_active":"2024-09-20T22:54:03.303323+0000","last_peered":"2024-09-20T22:54:03.303323+0000","last_clean":"2024-09-20T22:54:03.303323+0000","last_became_active":"2024-09-20T22:51:20.284621+0000","last_became_peered":"2024-09-20T22:51:20.284621+0000","last_unstale":"2024-09-20T22:54:03.303323+0000","last_undegraded":"2024-09-20T22:54:03.303323+0000","last_fullsized":"2024-09-20T22:54:03.303323+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:04.583683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:50:41.885844+0000","last_clean_scrub_stamp":"2024-09-20T22:53:04.583683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:27.888402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"625'18","reported_seq":1528,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-20T22:54:03.305877+0000","last_change":"2024-09-20T22:51:20.300828+0000","last_active":"2024-09-20T22:54:03.305877+0000","last_peered":"2024-09-20T22:54:03.305877+0000","last_clean":"2024-09-20T22:54:03.305877+0000","last_became_active":"2024-09-20T22:51:20.278886+0000","last_became_peered":"2024-09-20T22:51:20.278886+0000","last_unstale":"2024-09-20T22:54:03.305877+0000","last_undegraded":"2024-09-20T22:54:03.305877+0000","last_fullsized":"2024-09-20T22:54:03.305877+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-09-20T22:51:03.380934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:51:03.380934+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:08.164498+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":752,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-20T22:54:03.303343+0000","last_change":"2024-09-20T22:52:45.738376+0000","last_active":"2024-09-20T22:54:03.303343+0000","last_peered":"2024-09-20T22:54:03.303343+0000","last_clean":"2024-09-20T22:54:03.303343+0000","last_became_active":"2024-09-20T22:48:05.250319+0000","last_became_peered":"2024-09-20T22:48:05.250319+0000","last_unstale":"2024-09-20T22:54:03.303343+0000","last_undegraded":"2024-09-20T22:54:03.303343+0000","last_fullsized":"2024-09-20T22:54:03.303343+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:45.738275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:52:45.738275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"154'2","reported_seq":1433,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-20T22:54:03.308839+0000","last_change":"2024-09-20T22:52:45.578519+0000","last_active":"2024-09-20T22:54:03.308839+0000","last_peered":"2024-09-20T22:54:03.308839+0000","last_clean":"2024-09-20T22:54:03.308839+0000","last_became_active":"2024-09-20T22:41:44.199822+0000","last_became_peered":"2024-09-20T22:41:44.199822+0000","last_unstale":"2024-09-20T22:54:03.308839+0000","last_undegraded":"2024-09-20T22:54:03.308839+0000","last_fullsized":"2024-09-20T22:54:03.308839+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"154'2","last_scrub_stamp":"2024-09-20T22:52:45.578416+0000","last_deep_scrub":"154'2","last_deep_scrub_stamp":"2024-09-20T22:43:54.134066+0000","last_clean_scrub_stamp":"2024-09-20T22:52:45.578416+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"67'1","reported_seq":358,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-20T22:54:03.303755+0000","last_change":"2024-09-20T22:53:08.612473+0000","last_active":"2024-09-20T22:54:03.303755+0000","last_peered":"2024-09-20T22:54:03.303755+0000","last_clean":"2024-09-20T22:54:03.303755+0000","last_became_active":"2024-09-20T22:51:20.279978+0000","last_became_peered":"2024-09-20T22:51:20.279978+0000","last_unstale":"2024-09-20T22:54:03.303755+0000","last_undegraded":"2024-09-20T22:54:03.303755+0000","last_fullsized":"2024-09-20T22:54:03.303755+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-09-20T22:53:08.612395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:53:08.612395+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:13.547026+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"625'25","reported_seq":813,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-20T22:54:03.308028+0000","last_change":"2024-09-20T22:53:26.146547+0000","last_active":"2024-09-20T22:54:03.308028+0000","last_peered":"2024-09-20T22:54:03.308028+0000","last_clean":"2024-09-20T22:54:03.308028+0000","last_became_active":"2024-09-20T22:47:49.232181+0000","last_became_peered":"2024-09-20T22:47:49.232181+0000","last_unstale":"2024-09-20T22:54:03.308028+0000","last_undegraded":"2024-09-20T22:54:03.308028+0000","last_fullsized":"2024-09-20T22:54:03.308028+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'25","last_scrub_stamp":"2024-09-20T22:53:26.146466+0000","last_deep_scrub":"151'15","last_deep_scrub_stamp":"2024-09-20T22:43:50.715761+0000","last_clean_scrub_stamp":"2024-09-20T22:53:26.146466+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:51.749507+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"661'11","reported_seq":1638,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-20T22:54:03.312332+0000","last_change":"2024-09-20T22:53:34.944681+0000","last_active":"2024-09-20T22:54:03.312332+0000","last_peered":"2024-09-20T22:54:03.312332+0000","last_clean":"2024-09-20T22:54:03.312332+0000","last_became_active":"2024-09-20T22:48:05.244270+0000","last_became_peered":"2024-09-20T22:48:05.244270+0000","last_unstale":"2024-09-20T22:54:03.312332+0000","last_undegraded":"2024-09-20T22:54:03.312332+0000","last_fullsized":"2024-09-20T22:54:03.312332+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"661'11","last_scrub_stamp":"2024-09-20T22:53:34.944591+0000","last_deep_scrub":"661'11","last_deep_scrub_stamp":"2024-09-20T22:52:21.907943+0000","last_clean_scrub_stamp":"2024-09-20T22:53:34.944591+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:38.596007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.c","version":"763'2431","reported_seq":5658,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-20T22:54:03.324726+0000","last_change":"2024-09-20T22:54:03.325057+0000","last_active":"2024-09-20T22:54:03.324726+0000","last_peered":"2024-09-20T22:54:03.324726+0000","last_clean":"2024-09-20T22:54:03.324726+0000","last_became_active":"2024-09-20T22:42:52.404027+0000","last_became_peered":"2024-09-20T22:42:52.404027+0000","last_unstale":"2024-09-20T22:54:03.324726+0000","last_undegraded":"2024-09-20T22:54:03.324726+0000","last_fullsized":"2024-09-20T22:54:03.324726+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"754'2388","last_scrub_stamp":"2024-09-20T22:53:54.900766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:53:54.900766+0000","objects_scrubbed":32,"log_size":2431,"log_dups_size":0,"ondisk_log_size":2431,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:07.663541+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0.012598821,"stat_sum":{"num_bytes":65593036,"num_objects":33,"num_object_clones":26,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":5,"num_read":915,"num_read_kb":974438,"num_write":603,"num_write_kb":234876,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"118","length":"9"},{"start":"122","length":"2"},{"start":"125","length":"2"},{"start":"128","length":"2"},{"start":"12b","length":"3"},{"start":"131","length":"1"},{"start":"133","length":"6"},{"start":"13b","length":"3"},{"start":"13f","length":"2"},{"start":"143","length":"2"},{"start":"146","length":"2"},{"start":"14b","length":"1"},{"start":"14d","length":"1"},{"start":"150","length":"1"},{"start":"152","length":"1"},{"start":"154","length":"1"}]},{"pgid":"1.a","version":"77'128","reported_seq":1706,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-20T22:54:03.314892+0000","last_change":"2024-09-20T22:53:28.095343+0000","last_active":"2024-09-20T22:54:03.314892+0000","last_peered":"2024-09-20T22:54:03.314892+0000","last_clean":"2024-09-20T22:54:03.314892+0000","last_became_active":"2024-09-20T22:42:52.407791+0000","last_became_peered":"2024-09-20T22:42:52.407791+0000","last_unstale":"2024-09-20T22:54:03.314892+0000","last_undegraded":"2024-09-20T22:54:03.314892+0000","last_fullsized":"2024-09-20T22:54:03.314892+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:53:28.095266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:53:28.095266+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:45.776903+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"625'27","reported_seq":424,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-20T22:54:03.306331+0000","last_change":"2024-09-20T22:52:15.358218+0000","last_active":"2024-09-20T22:54:03.306331+0000","last_peered":"2024-09-20T22:54:03.306331+0000","last_clean":"2024-09-20T22:54:03.306331+0000","last_became_active":"2024-09-20T22:51:20.287083+0000","last_became_peered":"2024-09-20T22:51:20.287083+0000","last_unstale":"2024-09-20T22:54:03.306331+0000","last_undegraded":"2024-09-20T22:54:03.306331+0000","last_fullsized":"2024-09-20T22:54:03.306331+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'27","last_scrub_stamp":"2024-09-20T22:52:15.358100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:52:15.358100+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:08.164612+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":1435,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-20T22:54:03.302479+0000","last_change":"2024-09-20T22:51:29.394206+0000","last_active":"2024-09-20T22:54:03.302479+0000","last_peered":"2024-09-20T22:54:03.302479+0000","last_clean":"2024-09-20T22:54:03.302479+0000","last_became_active":"2024-09-20T22:51:20.279439+0000","last_became_peered":"2024-09-20T22:51:20.279439+0000","last_unstale":"2024-09-20T22:54:03.302479+0000","last_undegraded":"2024-09-20T22:54:03.302479+0000","last_fullsized":"2024-09-20T22:54:03.302479+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:29.394097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:49:12.070675+0000","last_clean_scrub_stamp":"2024-09-20T22:51:29.394097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:08.164579+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"408'12","reported_seq":820,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-20T22:54:03.303613+0000","last_change":"2024-09-20T22:51:50.785575+0000","last_active":"2024-09-20T22:54:03.303613+0000","last_peered":"2024-09-20T22:54:03.303613+0000","last_clean":"2024-09-20T22:54:03.303613+0000","last_became_active":"2024-09-20T22:48:05.242914+0000","last_became_peered":"2024-09-20T22:48:05.242914+0000","last_unstale":"2024-09-20T22:54:03.303613+0000","last_undegraded":"2024-09-20T22:54:03.303613+0000","last_fullsized":"2024-09-20T22:54:03.303613+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"408'12","last_scrub_stamp":"2024-09-20T22:51:50.785478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:51:50.785478+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":1444,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-20T22:54:03.316284+0000","last_change":"2024-09-20T22:51:31.960415+0000","last_active":"2024-09-20T22:54:03.316284+0000","last_peered":"2024-09-20T22:54:03.316284+0000","last_clean":"2024-09-20T22:54:03.316284+0000","last_became_active":"2024-09-20T22:51:20.280145+0000","last_became_peered":"2024-09-20T22:51:20.280145+0000","last_unstale":"2024-09-20T22:54:03.316284+0000","last_undegraded":"2024-09-20T22:54:03.316284+0000","last_fullsized":"2024-09-20T22:54:03.316284+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:31.960331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:51:31.960331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:08.101712+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3","version":"762'362","reported_seq":3538,"reported_epoch":763,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T22:54:03.304380+0000","last_change":"2024-09-20T22:54:02.411560+0000","last_active":"2024-09-20T22:54:03.304380+0000","last_peered":"2024-09-20T22:54:03.304380+0000","last_clean":"2024-09-20T22:54:03.304380+0000","last_became_active":"2024-09-20T22:49:05.221402+0000","last_became_peered":"2024-09-20T22:49:05.221402+0000","last_unstale":"2024-09-20T22:54:03.304380+0000","last_undegraded":"2024-09-20T22:54:03.304380+0000","last_fullsized":"2024-09-20T22:54:03.304380+0000","mapping_epoch":404,"log_start":"456'129","ondisk_log_start":"456'129","created":94,"last_epoch_clean":465,"parent":"0.0","parent_split_bits":0,"last_scrub":"609'243","last_scrub_stamp":"2024-09-20T22:51:31.402013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:51:31.402013+0000","objects_scrubbed":6,"log_size":233,"log_dups_size":149,"ondisk_log_size":233,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":8,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.113937575,"stat_sum":{"num_bytes":16659450,"num_objects":8,"num_object_clones":7,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":105,"num_read_kb":128303,"num_write":83,"num_write_kb":28717,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":20196357,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"118","length":"9"},{"start":"122","length":"2"},{"start":"125","length":"2"},{"start":"128","length":"2"},{"start":"12b","length":"3"},{"start":"131","length":"1"},{"start":"133","length":"6"},{"start":"13c","length":"1"},{"start":"13f","length":"2"},{"start":"143","length":"2"},{"start":"146","length":"1"}]},{"pgid":"1.5","version":"77'128","reported_seq":1611,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-20T22:54:03.304017+0000","last_change":"2024-09-20T22:53:35.470320+0000","last_active":"2024-09-20T22:54:03.304017+0000","last_peered":"2024-09-20T22:54:03.304017+0000","last_clean":"2024-09-20T22:54:03.304017+0000","last_became_active":"2024-09-20T22:48:08.207018+0000","last_became_peered":"2024-09-20T22:48:08.207018+0000","last_unstale":"2024-09-20T22:54:03.304017+0000","last_undegraded":"2024-09-20T22:54:03.304017+0000","last_fullsized":"2024-09-20T22:54:03.304017+0000","mapping_epoch":407,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":408,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:53:35.470248+0000","last_deep_scrub":"77'128","last_deep_scrub_stamp":"2024-09-20T22:45:41.608075+0000","last_clean_scrub_stamp":"2024-09-20T22:53:35.470248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:57.007268+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"625'30","reported_seq":1532,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-20T22:54:03.302756+0000","last_change":"2024-09-20T22:53:48.346424+0000","last_active":"2024-09-20T22:54:03.302756+0000","last_peered":"2024-09-20T22:54:03.302756+0000","last_clean":"2024-09-20T22:54:03.302756+0000","last_became_active":"2024-09-20T22:47:49.217605+0000","last_became_peered":"2024-09-20T22:47:49.217605+0000","last_unstale":"2024-09-20T22:54:03.302756+0000","last_undegraded":"2024-09-20T22:54:03.302756+0000","last_fullsized":"2024-09-20T22:54:03.302756+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'30","last_scrub_stamp":"2024-09-20T22:53:48.346341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:53:48.346341+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:02.270695+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":1449,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-20T22:54:03.306479+0000","last_change":"2024-09-20T22:52:45.537243+0000","last_active":"2024-09-20T22:54:03.306479+0000","last_peered":"2024-09-20T22:54:03.306479+0000","last_clean":"2024-09-20T22:54:03.306479+0000","last_became_active":"2024-09-20T22:47:49.227069+0000","last_became_peered":"2024-09-20T22:47:49.227069+0000","last_unstale":"2024-09-20T22:54:03.306479+0000","last_undegraded":"2024-09-20T22:54:03.306479+0000","last_fullsized":"2024-09-20T22:54:03.306479+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:45.537173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:52:45.537173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:08.164673+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":752,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-20T22:54:03.303984+0000","last_change":"2024-09-20T22:53:26.623524+0000","last_active":"2024-09-20T22:54:03.303984+0000","last_peered":"2024-09-20T22:54:03.303984+0000","last_clean":"2024-09-20T22:54:03.303984+0000","last_became_active":"2024-09-20T22:48:05.230003+0000","last_became_peered":"2024-09-20T22:48:05.230003+0000","last_unstale":"2024-09-20T22:54:03.303984+0000","last_undegraded":"2024-09-20T22:54:03.303984+0000","last_fullsized":"2024-09-20T22:54:03.303984+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:26.623455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:07.947945+0000","last_clean_scrub_stamp":"2024-09-20T22:53:26.623455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:36.274129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":744,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-20T22:54:03.303230+0000","last_change":"2024-09-20T22:51:55.748527+0000","last_active":"2024-09-20T22:54:03.303230+0000","last_peered":"2024-09-20T22:54:03.303230+0000","last_clean":"2024-09-20T22:54:03.303230+0000","last_became_active":"2024-09-20T22:48:05.239917+0000","last_became_peered":"2024-09-20T22:48:05.239917+0000","last_unstale":"2024-09-20T22:54:03.303230+0000","last_undegraded":"2024-09-20T22:54:03.303230+0000","last_fullsized":"2024-09-20T22:54:03.303230+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:55.748425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:53.036601+0000","last_clean_scrub_stamp":"2024-09-20T22:51:55.748425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2","version":"760'809","reported_seq":3341,"reported_epoch":763,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-20T22:54:03.304759+0000","last_change":"2024-09-20T22:54:02.428099+0000","last_active":"2024-09-20T22:54:03.304759+0000","last_peered":"2024-09-20T22:54:03.304759+0000","last_clean":"2024-09-20T22:54:03.304759+0000","last_became_active":"2024-09-20T22:52:23.286461+0000","last_became_peered":"2024-09-20T22:52:23.286461+0000","last_unstale":"2024-09-20T22:54:03.304759+0000","last_undegraded":"2024-09-20T22:54:03.304759+0000","last_fullsized":"2024-09-20T22:54:03.304759+0000","mapping_epoch":662,"log_start":"471'464","ondisk_log_start":"471'464","created":94,"last_epoch_clean":663,"parent":"0.0","parent_split_bits":0,"last_scrub":"282'120","last_scrub_stamp":"2024-09-20T22:46:22.099642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:46:22.099642+0000","objects_scrubbed":1,"log_size":345,"log_dups_size":687,"ondisk_log_size":345,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":37,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.11327907399999999,"stat_sum":{"num_bytes":51331449,"num_objects":20,"num_object_clones":16,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":1,"num_read":414,"num_read_kb":423819,"num_write":266,"num_write_kb":104890,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":144,"num_bytes_recovered":106353268,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"118","length":"2"}]},{"pgid":"1.4","version":"77'128","reported_seq":1730,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-20T22:54:03.305579+0000","last_change":"2024-09-20T22:52:02.474080+0000","last_active":"2024-09-20T22:54:03.305579+0000","last_peered":"2024-09-20T22:54:03.305579+0000","last_clean":"2024-09-20T22:54:03.305579+0000","last_became_active":"2024-09-20T22:46:15.181305+0000","last_became_peered":"2024-09-20T22:46:15.181305+0000","last_unstale":"2024-09-20T22:54:03.305579+0000","last_undegraded":"2024-09-20T22:54:03.305579+0000","last_fullsized":"2024-09-20T22:54:03.305579+0000","mapping_epoch":94,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:52:02.473979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:52:02.473979+0000","objects_scrubbed":1,"log_size":0,"log_dups_size":59,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:08.164596+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"625'28","reported_seq":781,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-20T22:54:03.303707+0000","last_change":"2024-09-20T22:51:58.712602+0000","last_active":"2024-09-20T22:54:03.303707+0000","last_peered":"2024-09-20T22:54:03.303707+0000","last_clean":"2024-09-20T22:54:03.303707+0000","last_became_active":"2024-09-20T22:48:05.235812+0000","last_became_peered":"2024-09-20T22:48:05.235812+0000","last_unstale":"2024-09-20T22:54:03.303707+0000","last_undegraded":"2024-09-20T22:54:03.303707+0000","last_fullsized":"2024-09-20T22:54:03.303707+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'28","last_scrub_stamp":"2024-09-20T22:51:58.712488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:51:58.712488+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":41,"num_read_kb":34,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"67'1","reported_seq":809,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-20T22:54:03.307759+0000","last_change":"2024-09-20T22:53:32.995298+0000","last_active":"2024-09-20T22:54:03.307759+0000","last_peered":"2024-09-20T22:54:03.307759+0000","last_clean":"2024-09-20T22:54:03.307759+0000","last_became_active":"2024-09-20T22:51:20.283011+0000","last_became_peered":"2024-09-20T22:51:20.283011+0000","last_unstale":"2024-09-20T22:54:03.307759+0000","last_undegraded":"2024-09-20T22:54:03.307759+0000","last_fullsized":"2024-09-20T22:54:03.307759+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-09-20T22:53:32.995211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:53:32.995211+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:59.820181+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"600'13","reported_seq":1631,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-20T22:54:03.303258+0000","last_change":"2024-09-20T22:51:32.448600+0000","last_active":"2024-09-20T22:54:03.303258+0000","last_peered":"2024-09-20T22:54:03.303258+0000","last_clean":"2024-09-20T22:54:03.303258+0000","last_became_active":"2024-09-20T22:51:20.289873+0000","last_became_peered":"2024-09-20T22:51:20.289873+0000","last_unstale":"2024-09-20T22:54:03.303258+0000","last_undegraded":"2024-09-20T22:54:03.303258+0000","last_fullsized":"2024-09-20T22:54:03.303258+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"600'13","last_scrub_stamp":"2024-09-20T22:51:32.448512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:51:32.448512+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:08.164546+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":1432,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-20T22:54:03.307496+0000","last_change":"2024-09-20T22:53:02.610452+0000","last_active":"2024-09-20T22:54:03.307496+0000","last_peered":"2024-09-20T22:54:03.307496+0000","last_clean":"2024-09-20T22:54:03.307496+0000","last_became_active":"2024-09-20T22:41:44.831901+0000","last_became_peered":"2024-09-20T22:41:44.831901+0000","last_unstale":"2024-09-20T22:54:03.307496+0000","last_undegraded":"2024-09-20T22:54:03.307496+0000","last_fullsized":"2024-09-20T22:54:03.307496+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:02.610359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:53:02.610359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:32.529803+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1","version":"761'462","reported_seq":3746,"reported_epoch":763,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T22:54:03.303589+0000","last_change":"2024-09-20T22:54:02.317768+0000","last_active":"2024-09-20T22:54:03.303589+0000","last_peered":"2024-09-20T22:54:03.303589+0000","last_clean":"2024-09-20T22:54:03.303589+0000","last_became_active":"2024-09-20T22:49:21.232750+0000","last_became_peered":"2024-09-20T22:49:21.232750+0000","last_unstale":"2024-09-20T22:54:03.303589+0000","last_undegraded":"2024-09-20T22:54:03.303589+0000","last_fullsized":"2024-09-20T22:54:03.303589+0000","mapping_epoch":407,"log_start":"474'259","ondisk_log_start":"474'259","created":94,"last_epoch_clean":481,"parent":"0.0","parent_split_bits":0,"last_scrub":"703'407","last_scrub_stamp":"2024-09-20T22:53:06.625437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:53:06.625437+0000","objects_scrubbed":6,"log_size":203,"log_dups_size":338,"ondisk_log_size":203,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:17.457526+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":1.010978081,"stat_sum":{"num_bytes":11851465,"num_objects":9,"num_object_clones":8,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":1,"num_read":124,"num_read_kb":142534,"num_write":97,"num_write_kb":50330,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":8732217,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"118","length":"9"},{"start":"122","length":"2"},{"start":"125","length":"2"},{"start":"128","length":"2"},{"start":"12b","length":"3"},{"start":"131","length":"1"},{"start":"133","length":"6"},{"start":"13c","length":"2"},{"start":"13f","length":"2"},{"start":"143","length":"2"},{"start":"146","length":"2"},{"start":"14b","length":"1"},{"start":"14d","length":"1"},{"start":"150","length":"1"},{"start":"154","length":"1"}]},{"pgid":"1.7","version":"77'128","reported_seq":1751,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-20T22:54:03.311835+0000","last_change":"2024-09-20T22:53:56.869885+0000","last_active":"2024-09-20T22:54:03.311835+0000","last_peered":"2024-09-20T22:54:03.311835+0000","last_clean":"2024-09-20T22:54:03.311835+0000","last_became_active":"2024-09-20T22:46:03.169596+0000","last_became_peered":"2024-09-20T22:46:03.169596+0000","last_unstale":"2024-09-20T22:54:03.311835+0000","last_undegraded":"2024-09-20T22:54:03.311835+0000","last_fullsized":"2024-09-20T22:54:03.311835+0000","mapping_epoch":94,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:53:56.869735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:53:56.869735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:04.429626+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"625'20","reported_seq":408,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-20T22:54:03.315299+0000","last_change":"2024-09-20T22:52:19.125548+0000","last_active":"2024-09-20T22:54:03.315299+0000","last_peered":"2024-09-20T22:54:03.315299+0000","last_clean":"2024-09-20T22:54:03.315299+0000","last_became_active":"2024-09-20T22:51:20.297933+0000","last_became_peered":"2024-09-20T22:51:20.297933+0000","last_unstale":"2024-09-20T22:54:03.315299+0000","last_undegraded":"2024-09-20T22:54:03.315299+0000","last_fullsized":"2024-09-20T22:54:03.315299+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'20","last_scrub_stamp":"2024-09-20T22:52:19.125462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:52:19.125462+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:08.101847+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":1446,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-20T22:54:03.316708+0000","last_change":"2024-09-20T22:52:46.062448+0000","last_active":"2024-09-20T22:54:03.316708+0000","last_peered":"2024-09-20T22:54:03.316708+0000","last_clean":"2024-09-20T22:54:03.316708+0000","last_became_active":"2024-09-20T22:41:38.155629+0000","last_became_peered":"2024-09-20T22:41:38.155629+0000","last_unstale":"2024-09-20T22:54:03.316708+0000","last_undegraded":"2024-09-20T22:54:03.316708+0000","last_fullsized":"2024-09-20T22:54:03.316708+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:46.062352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:52:46.062352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:15.728943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"733'12","reported_seq":867,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-20T22:54:03.339917+0000","last_change":"2024-09-20T22:53:33.928576+0000","last_active":"2024-09-20T22:54:03.339917+0000","last_peered":"2024-09-20T22:54:03.339917+0000","last_clean":"2024-09-20T22:54:03.339917+0000","last_became_active":"2024-09-20T22:47:49.227264+0000","last_became_peered":"2024-09-20T22:47:49.227264+0000","last_unstale":"2024-09-20T22:54:03.339917+0000","last_undegraded":"2024-09-20T22:54:03.339917+0000","last_fullsized":"2024-09-20T22:54:03.339917+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"733'12","last_scrub_stamp":"2024-09-20T22:53:33.928499+0000","last_deep_scrub":"733'12","last_deep_scrub_stamp":"2024-09-20T22:53:33.928499+0000","last_clean_scrub_stamp":"2024-09-20T22:53:33.928499+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:02.678336+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":786,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-20T22:54:03.302480+0000","last_change":"2024-09-20T22:53:57.868607+0000","last_active":"2024-09-20T22:54:03.302480+0000","last_peered":"2024-09-20T22:54:03.302480+0000","last_clean":"2024-09-20T22:54:03.302480+0000","last_became_active":"2024-09-20T22:47:49.232380+0000","last_became_peered":"2024-09-20T22:47:49.232380+0000","last_unstale":"2024-09-20T22:54:03.302480+0000","last_undegraded":"2024-09-20T22:54:03.302480+0000","last_fullsized":"2024-09-20T22:54:03.302480+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:57.868534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:53:57.868534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:14.269112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.0","version":"763'730","reported_seq":4069,"reported_epoch":763,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T22:54:03.479433+0000","last_change":"2024-09-20T22:54:03.479433+0000","last_active":"2024-09-20T22:54:03.479433+0000","last_peered":"2024-09-20T22:54:03.479433+0000","last_clean":"2024-09-20T22:54:03.479433+0000","last_became_active":"2024-09-20T22:51:53.366096+0000","last_became_peered":"2024-09-20T22:51:53.366096+0000","last_unstale":"2024-09-20T22:54:03.479433+0000","last_undegraded":"2024-09-20T22:54:03.479433+0000","last_fullsized":"2024-09-20T22:54:03.479433+0000","mapping_epoch":632,"log_start":"480'253","ondisk_log_start":"480'253","created":94,"last_epoch_clean":633,"parent":"0.0","parent_split_bits":0,"last_scrub":"562'368","last_scrub_stamp":"2024-09-20T22:50:44.669910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:50:44.669910+0000","objects_scrubbed":15,"log_size":477,"log_dups_size":416,"ondisk_log_size":477,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":13,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.16448991399999999,"stat_sum":{"num_bytes":58303436,"num_objects":27,"num_object_clones":24,"num_object_copies":81,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":264,"num_read_kb":267350,"num_write":225,"num_write_kb":82820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":42943203,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"118","length":"9"},{"start":"122","length":"2"},{"start":"125","length":"2"},{"start":"128","length":"2"},{"start":"12b","length":"1"},{"start":"12d","length":"1"},{"start":"131","length":"1"},{"start":"133","length":"6"},{"start":"13f","length":"2"}]},{"pgid":"1.6","version":"77'128","reported_seq":1765,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-20T22:54:03.312049+0000","last_change":"2024-09-20T22:52:38.843686+0000","last_active":"2024-09-20T22:54:03.312049+0000","last_peered":"2024-09-20T22:54:03.312049+0000","last_clean":"2024-09-20T22:54:03.312049+0000","last_became_active":"2024-09-20T22:47:52.212190+0000","last_became_peered":"2024-09-20T22:47:52.212190+0000","last_unstale":"2024-09-20T22:54:03.312049+0000","last_undegraded":"2024-09-20T22:54:03.312049+0000","last_fullsized":"2024-09-20T22:54:03.312049+0000","mapping_epoch":391,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:52:38.843586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:52:38.843586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:07.967280+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"625'37","reported_seq":819,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-20T22:54:03.303029+0000","last_change":"2024-09-20T22:52:39.703172+0000","last_active":"2024-09-20T22:54:03.303029+0000","last_peered":"2024-09-20T22:54:03.303029+0000","last_clean":"2024-09-20T22:54:03.303029+0000","last_became_active":"2024-09-20T22:47:49.222009+0000","last_became_peered":"2024-09-20T22:47:49.222009+0000","last_unstale":"2024-09-20T22:54:03.303029+0000","last_undegraded":"2024-09-20T22:54:03.303029+0000","last_fullsized":"2024-09-20T22:54:03.303029+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'37","last_scrub_stamp":"2024-09-20T22:52:39.703093+0000","last_deep_scrub":"515'27","last_deep_scrub_stamp":"2024-09-20T22:49:55.964153+0000","last_clean_scrub_stamp":"2024-09-20T22:52:39.703093+0000","objects_scrubbed":8,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:07.874112+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":53,"num_read_kb":39,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":355,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-20T22:54:03.319396+0000","last_change":"2024-09-20T22:53:55.227335+0000","last_active":"2024-09-20T22:54:03.319396+0000","last_peered":"2024-09-20T22:54:03.319396+0000","last_clean":"2024-09-20T22:54:03.319396+0000","last_became_active":"2024-09-20T22:51:20.286646+0000","last_became_peered":"2024-09-20T22:51:20.286646+0000","last_unstale":"2024-09-20T22:54:03.319396+0000","last_undegraded":"2024-09-20T22:54:03.319396+0000","last_fullsized":"2024-09-20T22:54:03.319396+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:55.227211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:48:23.320031+0000","last_clean_scrub_stamp":"2024-09-20T22:53:55.227211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:21.328227+0000","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":1465,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-20T22:54:03.310650+0000","last_change":"2024-09-20T22:53:57.882313+0000","last_active":"2024-09-20T22:54:03.310650+0000","last_peered":"2024-09-20T22:54:03.310650+0000","last_clean":"2024-09-20T22:54:03.310650+0000","last_became_active":"2024-09-20T22:47:49.214055+0000","last_became_peered":"2024-09-20T22:47:49.214055+0000","last_unstale":"2024-09-20T22:54:03.310650+0000","last_undegraded":"2024-09-20T22:54:03.310650+0000","last_fullsized":"2024-09-20T22:54:03.310650+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:57.882212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:53:57.882212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:12.731589+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":355,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-20T22:54:03.310580+0000","last_change":"2024-09-20T22:53:02.916523+0000","last_active":"2024-09-20T22:54:03.310580+0000","last_peered":"2024-09-20T22:54:03.310580+0000","last_clean":"2024-09-20T22:54:03.310580+0000","last_became_active":"2024-09-20T22:51:20.284874+0000","last_became_peered":"2024-09-20T22:51:20.284874+0000","last_unstale":"2024-09-20T22:54:03.310580+0000","last_undegraded":"2024-09-20T22:54:03.310580+0000","last_fullsized":"2024-09-20T22:54:03.310580+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:02.916449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:48:41.208853+0000","last_clean_scrub_stamp":"2024-09-20T22:53:02.916449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:15.012347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.7","version":"763'1405","reported_seq":4788,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-20T22:54:03.319265+0000","last_change":"2024-09-20T22:54:03.319532+0000","last_active":"2024-09-20T22:54:03.319265+0000","last_peered":"2024-09-20T22:54:03.319265+0000","last_clean":"2024-09-20T22:54:03.319265+0000","last_became_active":"2024-09-20T22:48:43.224386+0000","last_became_peered":"2024-09-20T22:48:43.224386+0000","last_unstale":"2024-09-20T22:54:03.319265+0000","last_undegraded":"2024-09-20T22:54:03.319265+0000","last_fullsized":"2024-09-20T22:54:03.319265+0000","mapping_epoch":391,"log_start":"436'456","ondisk_log_start":"436'456","created":94,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"704'1235","last_scrub_stamp":"2024-09-20T22:53:06.830408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:53:06.830408+0000","objects_scrubbed":31,"log_size":949,"log_dups_size":886,"ondisk_log_size":949,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:28.984893+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0072233899999999997,"stat_sum":{"num_bytes":54632926,"num_objects":23,"num_object_clones":18,"num_object_copies":69,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":1,"num_read":474,"num_read_kb":486374,"num_write":399,"num_write_kb":134673,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":115,"num_bytes_recovered":51615167,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"118","length":"9"},{"start":"122","length":"2"},{"start":"125","length":"2"},{"start":"128","length":"2"},{"start":"12b","length":"3"},{"start":"131","length":"1"},{"start":"133","length":"6"},{"start":"13b","length":"3"},{"start":"13f","length":"2"},{"start":"143","length":"2"},{"start":"146","length":"2"},{"start":"14b","length":"1"},{"start":"14d","length":"1"},{"start":"150","length":"1"},{"start":"152","length":"1"},{"start":"154","length":"1"}]},{"pgid":"1.1","version":"77'128","reported_seq":1754,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-20T22:54:03.314583+0000","last_change":"2024-09-20T22:52:42.146147+0000","last_active":"2024-09-20T22:54:03.314583+0000","last_peered":"2024-09-20T22:54:03.314583+0000","last_clean":"2024-09-20T22:54:03.314583+0000","last_became_active":"2024-09-20T22:47:52.208102+0000","last_became_peered":"2024-09-20T22:47:52.208102+0000","last_unstale":"2024-09-20T22:54:03.314583+0000","last_undegraded":"2024-09-20T22:54:03.314583+0000","last_fullsized":"2024-09-20T22:54:03.314583+0000","mapping_epoch":391,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:52:42.146069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:52:42.146069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:08.101863+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"625'51","reported_seq":539,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-20T22:54:03.319254+0000","last_change":"2024-09-20T22:53:57.184257+0000","last_active":"2024-09-20T22:54:03.319254+0000","last_peered":"2024-09-20T22:54:03.319254+0000","last_clean":"2024-09-20T22:54:03.319254+0000","last_became_active":"2024-09-20T22:51:20.289376+0000","last_became_peered":"2024-09-20T22:51:20.289376+0000","last_unstale":"2024-09-20T22:54:03.319254+0000","last_undegraded":"2024-09-20T22:54:03.319254+0000","last_fullsized":"2024-09-20T22:54:03.319254+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'51","last_scrub_stamp":"2024-09-20T22:53:57.184165+0000","last_deep_scrub":"86'38","last_deep_scrub_stamp":"2024-09-20T22:42:43.935677+0000","last_clean_scrub_stamp":"2024-09-20T22:53:57.184165+0000","objects_scrubbed":10,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:16.652330+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":744,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-20T22:54:03.303226+0000","last_change":"2024-09-20T22:52:36.695537+0000","last_active":"2024-09-20T22:54:03.303226+0000","last_peered":"2024-09-20T22:54:03.303226+0000","last_clean":"2024-09-20T22:54:03.303226+0000","last_became_active":"2024-09-20T22:48:05.237756+0000","last_became_peered":"2024-09-20T22:48:05.237756+0000","last_unstale":"2024-09-20T22:54:03.303226+0000","last_undegraded":"2024-09-20T22:54:03.303226+0000","last_fullsized":"2024-09-20T22:54:03.303226+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:36.695446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:52:36.695446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"69'2","reported_seq":1466,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-20T22:54:03.315756+0000","last_change":"2024-09-20T22:51:37.070187+0000","last_active":"2024-09-20T22:54:03.315756+0000","last_peered":"2024-09-20T22:54:03.315756+0000","last_clean":"2024-09-20T22:54:03.315756+0000","last_became_active":"2024-09-20T22:51:20.286952+0000","last_became_peered":"2024-09-20T22:51:20.286952+0000","last_unstale":"2024-09-20T22:54:03.315756+0000","last_undegraded":"2024-09-20T22:54:03.315756+0000","last_fullsized":"2024-09-20T22:54:03.315756+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2024-09-20T22:51:37.070111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:36.130684+0000","last_clean_scrub_stamp":"2024-09-20T22:51:37.070111+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:08.101729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087280599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":798,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-20T22:54:03.303719+0000","last_change":"2024-09-20T22:53:51.849903+0000","last_active":"2024-09-20T22:54:03.303719+0000","last_peered":"2024-09-20T22:54:03.303719+0000","last_clean":"2024-09-20T22:54:03.303719+0000","last_became_active":"2024-09-20T22:51:20.278427+0000","last_became_peered":"2024-09-20T22:51:20.278427+0000","last_unstale":"2024-09-20T22:54:03.303719+0000","last_undegraded":"2024-09-20T22:54:03.303719+0000","last_fullsized":"2024-09-20T22:54:03.303719+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:51.849772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:53:51.849772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:20.262243+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":1437,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-20T22:54:03.305445+0000","last_change":"2024-09-20T22:52:21.363432+0000","last_active":"2024-09-20T22:54:03.305445+0000","last_peered":"2024-09-20T22:54:03.305445+0000","last_clean":"2024-09-20T22:54:03.305445+0000","last_became_active":"2024-09-20T22:48:05.234193+0000","last_became_peered":"2024-09-20T22:48:05.234193+0000","last_unstale":"2024-09-20T22:54:03.305445+0000","last_undegraded":"2024-09-20T22:54:03.305445+0000","last_fullsized":"2024-09-20T22:54:03.305445+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:21.363359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:52:21.363359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:08.164627+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"763'526","reported_seq":2358,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-20T22:54:03.310342+0000","last_change":"2024-09-20T22:54:03.310342+0000","last_active":"2024-09-20T22:54:03.310342+0000","last_peered":"2024-09-20T22:54:03.310342+0000","last_clean":"2024-09-20T22:54:03.310342+0000","last_became_active":"2024-09-20T22:51:23.267613+0000","last_became_peered":"2024-09-20T22:51:23.267613+0000","last_unstale":"2024-09-20T22:54:03.310342+0000","last_undegraded":"2024-09-20T22:54:03.310342+0000","last_fullsized":"2024-09-20T22:54:03.310342+0000","mapping_epoch":602,"log_start":"456'176","ondisk_log_start":"456'176","created":94,"last_epoch_clean":603,"parent":"0.0","parent_split_bits":0,"last_scrub":"744'500","last_scrub_stamp":"2024-09-20T22:53:49.903129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:53:49.903129+0000","objects_scrubbed":10,"log_size":350,"log_dups_size":318,"ondisk_log_size":350,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:13.172957+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0056519400000000003,"stat_sum":{"num_bytes":26104196,"num_objects":10,"num_object_clones":8,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":173,"num_read_kb":155428,"num_write":150,"num_write_kb":51652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":26566881,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"118","length":"9"},{"start":"122","length":"2"},{"start":"125","length":"2"},{"start":"128","length":"2"},{"start":"12b","length":"3"},{"start":"131","length":"1"},{"start":"133","length":"6"},{"start":"13b","length":"3"},{"start":"13f","length":"2"},{"start":"143","length":"2"},{"start":"146","length":"2"},{"start":"14b","length":"1"},{"start":"14d","length":"1"},{"start":"150","length":"1"},{"start":"152","length":"1"},{"start":"154","length":"1"}]},{"pgid":"1.2","version":"77'128","reported_seq":1822,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-20T22:54:03.307552+0000","last_change":"2024-09-20T22:53:58.495293+0000","last_active":"2024-09-20T22:54:03.307552+0000","last_peered":"2024-09-20T22:54:03.307552+0000","last_clean":"2024-09-20T22:54:03.307552+0000","last_became_active":"2024-09-20T22:48:08.203028+0000","last_became_peered":"2024-09-20T22:48:08.203028+0000","last_unstale":"2024-09-20T22:54:03.307552+0000","last_undegraded":"2024-09-20T22:54:03.307552+0000","last_fullsized":"2024-09-20T22:54:03.307552+0000","mapping_epoch":407,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":408,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:53:58.495191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:53:58.495191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:00.289077+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"625'22","reported_seq":1208,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-20T22:54:03.316196+0000","last_change":"2024-09-20T22:53:35.117496+0000","last_active":"2024-09-20T22:54:03.316196+0000","last_peered":"2024-09-20T22:54:03.316196+0000","last_clean":"2024-09-20T22:54:03.316196+0000","last_became_active":"2024-09-20T22:51:20.293610+0000","last_became_peered":"2024-09-20T22:51:20.293610+0000","last_unstale":"2024-09-20T22:54:03.316196+0000","last_undegraded":"2024-09-20T22:54:03.316196+0000","last_fullsized":"2024-09-20T22:54:03.316196+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'22","last_scrub_stamp":"2024-09-20T22:53:35.117387+0000","last_deep_scrub":"443'18","last_deep_scrub_stamp":"2024-09-20T22:48:44.182517+0000","last_clean_scrub_stamp":"2024-09-20T22:53:35.117387+0000","objects_scrubbed":5,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:45.532207+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":990,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":355,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-20T22:54:03.320401+0000","last_change":"2024-09-20T22:53:30.210048+0000","last_active":"2024-09-20T22:54:03.320401+0000","last_peered":"2024-09-20T22:54:03.320401+0000","last_clean":"2024-09-20T22:54:03.320401+0000","last_became_active":"2024-09-20T22:51:20.278609+0000","last_became_peered":"2024-09-20T22:51:20.278609+0000","last_unstale":"2024-09-20T22:54:03.320401+0000","last_undegraded":"2024-09-20T22:54:03.320401+0000","last_fullsized":"2024-09-20T22:54:03.320401+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:30.209986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:53:30.209986+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:35.918431+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"380'2","reported_seq":1492,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-20T22:54:03.318217+0000","last_change":"2024-09-20T22:52:37.033309+0000","last_active":"2024-09-20T22:54:03.318217+0000","last_peered":"2024-09-20T22:54:03.318217+0000","last_clean":"2024-09-20T22:54:03.318217+0000","last_became_active":"2024-09-20T22:47:49.222812+0000","last_became_peered":"2024-09-20T22:47:49.222812+0000","last_unstale":"2024-09-20T22:54:03.318217+0000","last_undegraded":"2024-09-20T22:54:03.318217+0000","last_fullsized":"2024-09-20T22:54:03.318217+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"380'2","last_scrub_stamp":"2024-09-20T22:52:37.033222+0000","last_deep_scrub":"380'2","last_deep_scrub_stamp":"2024-09-20T22:47:40.983039+0000","last_clean_scrub_stamp":"2024-09-20T22:52:37.033222+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:08.257133+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001301459,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":1445,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-20T22:54:03.320235+0000","last_change":"2024-09-20T22:53:25.223722+0000","last_active":"2024-09-20T22:54:03.320235+0000","last_peered":"2024-09-20T22:54:03.320235+0000","last_clean":"2024-09-20T22:54:03.320235+0000","last_became_active":"2024-09-20T22:41:42.185112+0000","last_became_peered":"2024-09-20T22:41:42.185112+0000","last_unstale":"2024-09-20T22:54:03.320235+0000","last_undegraded":"2024-09-20T22:54:03.320235+0000","last_fullsized":"2024-09-20T22:54:03.320235+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:25.223611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:47:22.881109+0000","last_clean_scrub_stamp":"2024-09-20T22:53:25.223611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:53.455503+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":1432,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-20T22:54:03.308151+0000","last_change":"2024-09-20T22:53:06.644045+0000","last_active":"2024-09-20T22:54:03.308151+0000","last_peered":"2024-09-20T22:54:03.308151+0000","last_clean":"2024-09-20T22:54:03.308151+0000","last_became_active":"2024-09-20T22:41:44.830515+0000","last_became_peered":"2024-09-20T22:41:44.830515+0000","last_unstale":"2024-09-20T22:54:03.308151+0000","last_undegraded":"2024-09-20T22:54:03.308151+0000","last_fullsized":"2024-09-20T22:54:03.308151+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:06.643947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:53:06.643947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:16.583688+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"761'401","reported_seq":3805,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-20T22:54:03.308231+0000","last_change":"2024-09-20T22:54:03.308987+0000","last_active":"2024-09-20T22:54:03.308231+0000","last_peered":"2024-09-20T22:54:03.308231+0000","last_clean":"2024-09-20T22:54:03.308231+0000","last_became_active":"2024-09-20T22:48:57.236417+0000","last_became_peered":"2024-09-20T22:48:57.236417+0000","last_unstale":"2024-09-20T22:54:03.308231+0000","last_undegraded":"2024-09-20T22:54:03.308231+0000","last_fullsized":"2024-09-20T22:54:03.308231+0000","mapping_epoch":391,"log_start":"450'190","ondisk_log_start":"450'190","created":94,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"674'316","last_scrub_stamp":"2024-09-20T22:52:35.685017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:52:35.685017+0000","objects_scrubbed":8,"log_size":211,"log_dups_size":240,"ondisk_log_size":211,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.0011022289999999999,"stat_sum":{"num_bytes":14965811,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":99,"num_read_kb":89344,"num_write":98,"num_write_kb":40424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":19673726,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"118","length":"9"},{"start":"122","length":"2"},{"start":"125","length":"2"},{"start":"128","length":"2"},{"start":"12b","length":"3"},{"start":"131","length":"1"},{"start":"133","length":"6"},{"start":"13b","length":"3"},{"start":"13f","length":"2"},{"start":"143","length":"2"},{"start":"146","length":"2"},{"start":"14b","length":"1"},{"start":"14d","length":"1"},{"start":"150","length":"1"},{"start":"152","length":"1"},{"start":"154","length":"1"}]},{"pgid":"1.3","version":"77'128","reported_seq":650,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-20T22:54:03.310784+0000","last_change":"2024-09-20T22:53:05.866046+0000","last_active":"2024-09-20T22:54:03.310784+0000","last_peered":"2024-09-20T22:54:03.310784+0000","last_clean":"2024-09-20T22:54:03.310784+0000","last_became_active":"2024-09-20T22:51:23.266810+0000","last_became_peered":"2024-09-20T22:51:23.266810+0000","last_unstale":"2024-09-20T22:54:03.310784+0000","last_undegraded":"2024-09-20T22:54:03.310784+0000","last_fullsized":"2024-09-20T22:54:03.310784+0000","mapping_epoch":602,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":603,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:53:05.865942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:53:05.865942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:28.578041+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.6","version":"625'24","reported_seq":829,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-20T22:54:03.308120+0000","last_change":"2024-09-20T22:52:48.829973+0000","last_active":"2024-09-20T22:54:03.308120+0000","last_peered":"2024-09-20T22:54:03.308120+0000","last_clean":"2024-09-20T22:54:03.308120+0000","last_became_active":"2024-09-20T22:51:20.283112+0000","last_became_peered":"2024-09-20T22:51:20.283112+0000","last_unstale":"2024-09-20T22:54:03.308120+0000","last_undegraded":"2024-09-20T22:54:03.308120+0000","last_fullsized":"2024-09-20T22:54:03.308120+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'24","last_scrub_stamp":"2024-09-20T22:52:48.829882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:52:48.829882+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:17.822454+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":792,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-20T22:54:03.304012+0000","last_change":"2024-09-20T22:53:25.172560+0000","last_active":"2024-09-20T22:54:03.304012+0000","last_peered":"2024-09-20T22:54:03.304012+0000","last_clean":"2024-09-20T22:54:03.304012+0000","last_became_active":"2024-09-20T22:48:05.233093+0000","last_became_peered":"2024-09-20T22:48:05.233093+0000","last_unstale":"2024-09-20T22:54:03.304012+0000","last_undegraded":"2024-09-20T22:54:03.304012+0000","last_fullsized":"2024-09-20T22:54:03.304012+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:25.172487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:44:19.834641+0000","last_clean_scrub_stamp":"2024-09-20T22:53:25.172487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:39.242526+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"75'5","reported_seq":2223,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-20T22:54:03.308098+0000","last_change":"2024-09-20T22:52:44.568232+0000","last_active":"2024-09-20T22:54:03.308098+0000","last_peered":"2024-09-20T22:54:03.308098+0000","last_clean":"2024-09-20T22:54:03.308098+0000","last_became_active":"2024-09-20T22:51:20.281329+0000","last_became_peered":"2024-09-20T22:51:20.281329+0000","last_unstale":"2024-09-20T22:54:03.308098+0000","last_undegraded":"2024-09-20T22:54:03.308098+0000","last_fullsized":"2024-09-20T22:54:03.308098+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-09-20T22:52:44.568129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:36.130684+0000","last_clean_scrub_stamp":"2024-09-20T22:52:44.568129+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:05.662332+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00078631700000000003,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":735,"num_read_kb":730,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1167,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":1456,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-20T22:54:03.316255+0000","last_change":"2024-09-20T22:52:48.082097+0000","last_active":"2024-09-20T22:54:03.316255+0000","last_peered":"2024-09-20T22:54:03.316255+0000","last_clean":"2024-09-20T22:54:03.316255+0000","last_became_active":"2024-09-20T22:51:20.274522+0000","last_became_peered":"2024-09-20T22:51:20.274522+0000","last_unstale":"2024-09-20T22:54:03.316255+0000","last_undegraded":"2024-09-20T22:54:03.316255+0000","last_fullsized":"2024-09-20T22:54:03.316255+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:48.082027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:52:48.082027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:08.101878+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":347,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-20T22:54:03.315068+0000","last_change":"2024-09-20T22:53:49.031995+0000","last_active":"2024-09-20T22:54:03.315068+0000","last_peered":"2024-09-20T22:54:03.315068+0000","last_clean":"2024-09-20T22:54:03.315068+0000","last_became_active":"2024-09-20T22:51:20.287750+0000","last_became_peered":"2024-09-20T22:51:20.287750+0000","last_unstale":"2024-09-20T22:54:03.315068+0000","last_undegraded":"2024-09-20T22:54:03.315068+0000","last_fullsized":"2024-09-20T22:54:03.315068+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:49.031912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:53:49.031912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:01.922210+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.6","version":"763'533","reported_seq":3862,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-20T22:54:03.320749+0000","last_change":"2024-09-20T22:54:03.321002+0000","last_active":"2024-09-20T22:54:03.320749+0000","last_peered":"2024-09-20T22:54:03.320749+0000","last_clean":"2024-09-20T22:54:03.320749+0000","last_became_active":"2024-09-20T22:48:47.234531+0000","last_became_peered":"2024-09-20T22:48:47.234531+0000","last_unstale":"2024-09-20T22:54:03.320749+0000","last_undegraded":"2024-09-20T22:54:03.320749+0000","last_fullsized":"2024-09-20T22:54:03.320749+0000","mapping_epoch":391,"log_start":"443'181","ondisk_log_start":"443'181","created":94,"last_epoch_clean":447,"parent":"0.0","parent_split_bits":0,"last_scrub":"654'392","last_scrub_stamp":"2024-09-20T22:52:14.706591+0000","last_deep_scrub":"440'177","last_deep_scrub_stamp":"2024-09-20T22:48:40.602342+0000","last_clean_scrub_stamp":"2024-09-20T22:52:14.706591+0000","objects_scrubbed":10,"log_size":352,"log_dups_size":314,"ondisk_log_size":352,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.013261538999999999,"stat_sum":{"num_bytes":27969352,"num_objects":12,"num_object_clones":10,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":201,"num_read_kb":209857,"num_write":151,"num_write_kb":58468,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":18662529,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"118","length":"9"},{"start":"122","length":"2"},{"start":"125","length":"2"},{"start":"128","length":"2"},{"start":"12b","length":"3"},{"start":"131","length":"1"},{"start":"133","length":"6"},{"start":"13b","length":"3"},{"start":"13f","length":"2"},{"start":"143","length":"2"},{"start":"146","length":"2"},{"start":"14b","length":"1"},{"start":"14d","length":"1"},{"start":"150","length":"1"},{"start":"152","length":"1"},{"start":"154","length":"1"}]},{"pgid":"4.5","version":"625'32","reported_seq":1735,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-20T22:54:03.311861+0000","last_change":"2024-09-20T22:53:07.867174+0000","last_active":"2024-09-20T22:54:03.311861+0000","last_peered":"2024-09-20T22:54:03.311861+0000","last_clean":"2024-09-20T22:54:03.311861+0000","last_became_active":"2024-09-20T22:48:05.243523+0000","last_became_peered":"2024-09-20T22:48:05.243523+0000","last_unstale":"2024-09-20T22:54:03.311861+0000","last_undegraded":"2024-09-20T22:54:03.311861+0000","last_fullsized":"2024-09-20T22:54:03.311861+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'32","last_scrub_stamp":"2024-09-20T22:53:07.867088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:53:07.867088+0000","objects_scrubbed":8,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:23.661414+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"67'1","reported_seq":1435,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-20T22:54:03.304972+0000","last_change":"2024-09-20T22:51:35.440516+0000","last_active":"2024-09-20T22:54:03.304972+0000","last_peered":"2024-09-20T22:54:03.304972+0000","last_clean":"2024-09-20T22:54:03.304972+0000","last_became_active":"2024-09-20T22:41:38.158917+0000","last_became_peered":"2024-09-20T22:41:38.158917+0000","last_unstale":"2024-09-20T22:54:03.304972+0000","last_undegraded":"2024-09-20T22:54:03.304972+0000","last_fullsized":"2024-09-20T22:54:03.304972+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-09-20T22:51:35.440416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:51:35.440416+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:08.164564+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"77'128","reported_seq":1786,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-20T22:54:03.308252+0000","last_change":"2024-09-20T22:52:18.688292+0000","last_active":"2024-09-20T22:54:03.308252+0000","last_peered":"2024-09-20T22:54:03.308252+0000","last_clean":"2024-09-20T22:54:03.308252+0000","last_became_active":"2024-09-20T22:47:52.206169+0000","last_became_peered":"2024-09-20T22:47:52.206169+0000","last_unstale":"2024-09-20T22:54:03.308252+0000","last_undegraded":"2024-09-20T22:54:03.308252+0000","last_fullsized":"2024-09-20T22:54:03.308252+0000","mapping_epoch":391,"log_start":"77'128","ondisk_log_start":"77'128","created":19,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:52:18.688135+0000","last_deep_scrub":"77'128","last_deep_scrub_stamp":"2024-09-20T22:47:22.417099+0000","last_clean_scrub_stamp":"2024-09-20T22:52:18.688135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":1444,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-20T22:54:03.308934+0000","last_change":"2024-09-20T22:53:03.564792+0000","last_active":"2024-09-20T22:54:03.308934+0000","last_peered":"2024-09-20T22:54:03.308934+0000","last_clean":"2024-09-20T22:54:03.308934+0000","last_became_active":"2024-09-20T22:41:42.184387+0000","last_became_peered":"2024-09-20T22:41:42.184387+0000","last_unstale":"2024-09-20T22:54:03.308934+0000","last_undegraded":"2024-09-20T22:54:03.308934+0000","last_fullsized":"2024-09-20T22:54:03.308934+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:03.564726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:53:03.564726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:25.313703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":1446,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-20T22:54:03.315701+0000","last_change":"2024-09-20T22:53:09.957459+0000","last_active":"2024-09-20T22:54:03.315701+0000","last_peered":"2024-09-20T22:54:03.315701+0000","last_clean":"2024-09-20T22:54:03.315701+0000","last_became_active":"2024-09-20T22:48:05.241602+0000","last_became_peered":"2024-09-20T22:48:05.241602+0000","last_unstale":"2024-09-20T22:54:03.315701+0000","last_undegraded":"2024-09-20T22:54:03.315701+0000","last_fullsized":"2024-09-20T22:54:03.315701+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:09.957374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:53:09.957374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:12.536331+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"763'615","reported_seq":4066,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-20T22:54:03.316040+0000","last_change":"2024-09-20T22:54:03.316328+0000","last_active":"2024-09-20T22:54:03.316040+0000","last_peered":"2024-09-20T22:54:03.316040+0000","last_clean":"2024-09-20T22:54:03.316040+0000","last_became_active":"2024-09-20T22:51:23.269216+0000","last_became_peered":"2024-09-20T22:51:23.269216+0000","last_unstale":"2024-09-20T22:54:03.316040+0000","last_undegraded":"2024-09-20T22:54:03.316040+0000","last_fullsized":"2024-09-20T22:54:03.316040+0000","mapping_epoch":602,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":603,"parent":"0.0","parent_split_bits":0,"last_scrub":"702'544","last_scrub_stamp":"2024-09-20T22:53:03.918130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:53:03.918130+0000","objects_scrubbed":10,"log_size":615,"log_dups_size":0,"ondisk_log_size":615,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:20.023532+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.0045340060000000001,"stat_sum":{"num_bytes":24638708,"num_objects":10,"num_object_clones":8,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":198,"num_read_kb":205211,"num_write":109,"num_write_kb":47355,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":52,"num_bytes_recovered":30669453,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"118","length":"9"},{"start":"122","length":"2"},{"start":"125","length":"2"},{"start":"128","length":"2"},{"start":"12b","length":"3"},{"start":"131","length":"1"},{"start":"133","length":"6"},{"start":"13b","length":"3"},{"start":"13f","length":"2"},{"start":"143","length":"2"},{"start":"146","length":"2"},{"start":"14b","length":"1"},{"start":"14d","length":"1"},{"start":"150","length":"1"},{"start":"152","length":"1"},{"start":"154","length":"1"}]},{"pgid":"4.e","version":"625'19","reported_seq":786,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-20T22:54:03.303663+0000","last_change":"2024-09-20T22:52:44.783087+0000","last_active":"2024-09-20T22:54:03.303663+0000","last_peered":"2024-09-20T22:54:03.303663+0000","last_clean":"2024-09-20T22:54:03.303663+0000","last_became_active":"2024-09-20T22:51:20.288958+0000","last_became_peered":"2024-09-20T22:51:20.288958+0000","last_unstale":"2024-09-20T22:54:03.303663+0000","last_undegraded":"2024-09-20T22:54:03.303663+0000","last_fullsized":"2024-09-20T22:54:03.303663+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'19","last_scrub_stamp":"2024-09-20T22:52:44.782975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:52:44.782975+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":744,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-20T22:54:03.302827+0000","last_change":"2024-09-20T22:51:29.757136+0000","last_active":"2024-09-20T22:54:03.302827+0000","last_peered":"2024-09-20T22:54:03.302827+0000","last_clean":"2024-09-20T22:54:03.302827+0000","last_became_active":"2024-09-20T22:48:05.254688+0000","last_became_peered":"2024-09-20T22:48:05.254688+0000","last_unstale":"2024-09-20T22:54:03.302827+0000","last_undegraded":"2024-09-20T22:54:03.302827+0000","last_fullsized":"2024-09-20T22:54:03.302827+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:29.757030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:51:29.757030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":1442,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-20T22:54:03.315206+0000","last_change":"2024-09-20T22:51:51.002353+0000","last_active":"2024-09-20T22:54:03.315206+0000","last_peered":"2024-09-20T22:54:03.315206+0000","last_clean":"2024-09-20T22:54:03.315206+0000","last_became_active":"2024-09-20T22:48:05.233778+0000","last_became_peered":"2024-09-20T22:48:05.233778+0000","last_unstale":"2024-09-20T22:54:03.315206+0000","last_undegraded":"2024-09-20T22:54:03.315206+0000","last_fullsized":"2024-09-20T22:54:03.315206+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:51.002226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:51:51.002226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:08.101789+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":1425,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-20T22:54:03.305561+0000","last_change":"2024-09-20T22:53:10.647075+0000","last_active":"2024-09-20T22:54:03.305561+0000","last_peered":"2024-09-20T22:54:03.305561+0000","last_clean":"2024-09-20T22:54:03.305561+0000","last_became_active":"2024-09-20T22:41:44.832009+0000","last_became_peered":"2024-09-20T22:41:44.832009+0000","last_unstale":"2024-09-20T22:54:03.305561+0000","last_undegraded":"2024-09-20T22:54:03.305561+0000","last_fullsized":"2024-09-20T22:54:03.305561+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:10.647005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:48:40.127123+0000","last_clean_scrub_stamp":"2024-09-20T22:53:10.647005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:32.128560+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.e","version":"763'781","reported_seq":4159,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-20T22:54:03.325723+0000","last_change":"2024-09-20T22:54:03.326024+0000","last_active":"2024-09-20T22:54:03.325723+0000","last_peered":"2024-09-20T22:54:03.325723+0000","last_clean":"2024-09-20T22:54:03.325723+0000","last_became_active":"2024-09-20T22:47:52.209452+0000","last_became_peered":"2024-09-20T22:47:52.209452+0000","last_unstale":"2024-09-20T22:54:03.325723+0000","last_undegraded":"2024-09-20T22:54:03.325723+0000","last_fullsized":"2024-09-20T22:54:03.325723+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"681'618","last_scrub_stamp":"2024-09-20T22:52:45.935130+0000","last_deep_scrub":"174'60","last_deep_scrub_stamp":"2024-09-20T22:44:18.301776+0000","last_clean_scrub_stamp":"2024-09-20T22:52:45.935130+0000","objects_scrubbed":13,"log_size":781,"log_dups_size":0,"ondisk_log_size":781,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.006522936,"stat_sum":{"num_bytes":27683579,"num_objects":13,"num_object_clones":11,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":254,"num_read_kb":256137,"num_write":185,"num_write_kb":65479,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":23110741,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"118","length":"9"},{"start":"122","length":"2"},{"start":"125","length":"2"},{"start":"128","length":"2"},{"start":"12b","length":"3"},{"start":"131","length":"1"},{"start":"133","length":"6"},{"start":"13b","length":"3"},{"start":"13f","length":"2"},{"start":"143","length":"2"},{"start":"146","length":"2"},{"start":"14b","length":"1"},{"start":"14d","length":"1"},{"start":"150","length":"1"},{"start":"152","length":"1"},{"start":"154","length":"1"}]},{"pgid":"1.8","version":"77'128","reported_seq":1706,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-20T22:54:03.315033+0000","last_change":"2024-09-20T22:51:56.108038+0000","last_active":"2024-09-20T22:54:03.315033+0000","last_peered":"2024-09-20T22:54:03.315033+0000","last_clean":"2024-09-20T22:54:03.315033+0000","last_became_active":"2024-09-20T22:42:52.405594+0000","last_became_peered":"2024-09-20T22:42:52.405594+0000","last_unstale":"2024-09-20T22:54:03.315033+0000","last_undegraded":"2024-09-20T22:54:03.315033+0000","last_fullsized":"2024-09-20T22:54:03.315033+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:51:56.107944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:51:56.107944+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:08.101775+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"625'31","reported_seq":815,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-20T22:54:03.303605+0000","last_change":"2024-09-20T22:52:13.620897+0000","last_active":"2024-09-20T22:54:03.303605+0000","last_peered":"2024-09-20T22:54:03.303605+0000","last_clean":"2024-09-20T22:54:03.303605+0000","last_became_active":"2024-09-20T22:51:20.288846+0000","last_became_peered":"2024-09-20T22:51:20.288846+0000","last_unstale":"2024-09-20T22:54:03.303605+0000","last_undegraded":"2024-09-20T22:54:03.303605+0000","last_fullsized":"2024-09-20T22:54:03.303605+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'31","last_scrub_stamp":"2024-09-20T22:52:13.620754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:52:13.620754+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":49,"num_read_kb":39,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":1000,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-20T22:54:03.317963+0000","last_change":"2024-09-20T22:53:08.105116+0000","last_active":"2024-09-20T22:54:03.317963+0000","last_peered":"2024-09-20T22:54:03.317963+0000","last_clean":"2024-09-20T22:54:03.317963+0000","last_became_active":"2024-09-20T22:51:20.294073+0000","last_became_peered":"2024-09-20T22:51:20.294073+0000","last_unstale":"2024-09-20T22:54:03.317963+0000","last_undegraded":"2024-09-20T22:54:03.317963+0000","last_fullsized":"2024-09-20T22:54:03.317963+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:08.105010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:35.492562+0000","last_clean_scrub_stamp":"2024-09-20T22:53:08.105010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:35.105470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":1065,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-20T22:54:03.302943+0000","last_change":"2024-09-20T22:53:57.442849+0000","last_active":"2024-09-20T22:54:03.302943+0000","last_peered":"2024-09-20T22:54:03.302943+0000","last_clean":"2024-09-20T22:54:03.302943+0000","last_became_active":"2024-09-20T22:51:20.281724+0000","last_became_peered":"2024-09-20T22:51:20.281724+0000","last_unstale":"2024-09-20T22:54:03.302943+0000","last_undegraded":"2024-09-20T22:54:03.302943+0000","last_fullsized":"2024-09-20T22:54:03.302943+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:57.442770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:53:57.442770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:03.011749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":1461,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-20T22:54:03.318828+0000","last_change":"2024-09-20T22:53:05.106323+0000","last_active":"2024-09-20T22:54:03.318828+0000","last_peered":"2024-09-20T22:54:03.318828+0000","last_clean":"2024-09-20T22:54:03.318828+0000","last_became_active":"2024-09-20T22:48:05.239006+0000","last_became_peered":"2024-09-20T22:48:05.239006+0000","last_unstale":"2024-09-20T22:54:03.318828+0000","last_undegraded":"2024-09-20T22:54:03.318828+0000","last_fullsized":"2024-09-20T22:54:03.318828+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:05.106224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:53:05.106224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:05.128923+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"763'858","reported_seq":4221,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-20T22:54:03.320628+0000","last_change":"2024-09-20T22:54:03.320946+0000","last_active":"2024-09-20T22:54:03.320628+0000","last_peered":"2024-09-20T22:54:03.320628+0000","last_clean":"2024-09-20T22:54:03.320628+0000","last_became_active":"2024-09-20T22:47:52.216198+0000","last_became_peered":"2024-09-20T22:47:52.216198+0000","last_unstale":"2024-09-20T22:54:03.320628+0000","last_undegraded":"2024-09-20T22:54:03.320628+0000","last_fullsized":"2024-09-20T22:54:03.320628+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"754'838","last_scrub_stamp":"2024-09-20T22:53:54.908987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:53:54.908987+0000","objects_scrubbed":17,"log_size":858,"log_dups_size":0,"ondisk_log_size":858,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:22.391414+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.013938703,"stat_sum":{"num_bytes":31248663,"num_objects":16,"num_object_clones":13,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":1,"num_read":334,"num_read_kb":329186,"num_write":216,"num_write_kb":70806,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":53,"num_bytes_recovered":14983556,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"118","length":"9"},{"start":"122","length":"2"},{"start":"125","length":"2"},{"start":"128","length":"2"},{"start":"12b","length":"3"},{"start":"131","length":"1"},{"start":"133","length":"6"},{"start":"13b","length":"3"},{"start":"13f","length":"2"},{"start":"143","length":"2"},{"start":"146","length":"2"},{"start":"14b","length":"1"},{"start":"14d","length":"1"},{"start":"150","length":"1"},{"start":"152","length":"1"},{"start":"154","length":"1"}]},{"pgid":"1.9","version":"77'128","reported_seq":1762,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-20T22:54:03.308884+0000","last_change":"2024-09-20T22:53:50.381120+0000","last_active":"2024-09-20T22:54:03.308884+0000","last_peered":"2024-09-20T22:54:03.308884+0000","last_clean":"2024-09-20T22:54:03.308884+0000","last_became_active":"2024-09-20T22:50:42.262862+0000","last_became_peered":"2024-09-20T22:50:42.262862+0000","last_unstale":"2024-09-20T22:54:03.308884+0000","last_undegraded":"2024-09-20T22:54:03.308884+0000","last_fullsized":"2024-09-20T22:54:03.308884+0000","mapping_epoch":561,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":562,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:53:50.381052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:53:50.381052+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:18.159814+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"625'20","reported_seq":771,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-20T22:54:03.303496+0000","last_change":"2024-09-20T22:52:22.646547+0000","last_active":"2024-09-20T22:54:03.303496+0000","last_peered":"2024-09-20T22:54:03.303496+0000","last_clean":"2024-09-20T22:54:03.303496+0000","last_became_active":"2024-09-20T22:48:05.242811+0000","last_became_peered":"2024-09-20T22:48:05.242811+0000","last_unstale":"2024-09-20T22:54:03.303496+0000","last_undegraded":"2024-09-20T22:54:03.303496+0000","last_fullsized":"2024-09-20T22:54:03.303496+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'20","last_scrub_stamp":"2024-09-20T22:52:22.646468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:52:22.646468+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":1461,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-20T22:54:03.305499+0000","last_change":"2024-09-20T22:52:44.527004+0000","last_active":"2024-09-20T22:54:03.305499+0000","last_peered":"2024-09-20T22:54:03.305499+0000","last_clean":"2024-09-20T22:54:03.305499+0000","last_became_active":"2024-09-20T22:48:05.233999+0000","last_became_peered":"2024-09-20T22:48:05.233999+0000","last_unstale":"2024-09-20T22:54:03.305499+0000","last_undegraded":"2024-09-20T22:54:03.305499+0000","last_fullsized":"2024-09-20T22:54:03.305499+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:44.526935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:52:44.526935+0000","last_clean_scrub_stamp":"2024-09-20T22:52:44.526935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:08.164642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"261'9","reported_seq":1612,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-20T22:54:03.319711+0000","last_change":"2024-09-20T22:53:54.190905+0000","last_active":"2024-09-20T22:54:03.319711+0000","last_peered":"2024-09-20T22:54:03.319711+0000","last_clean":"2024-09-20T22:54:03.319711+0000","last_became_active":"2024-09-20T22:41:42.185863+0000","last_became_peered":"2024-09-20T22:41:42.185863+0000","last_unstale":"2024-09-20T22:54:03.319711+0000","last_undegraded":"2024-09-20T22:54:03.319711+0000","last_fullsized":"2024-09-20T22:54:03.319711+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'9","last_scrub_stamp":"2024-09-20T22:53:54.190823+0000","last_deep_scrub":"261'9","last_deep_scrub_stamp":"2024-09-20T22:46:50.974735+0000","last_clean_scrub_stamp":"2024-09-20T22:53:54.190823+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:22.929855+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":758,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-20T22:54:03.303479+0000","last_change":"2024-09-20T22:53:10.650499+0000","last_active":"2024-09-20T22:54:03.303479+0000","last_peered":"2024-09-20T22:54:03.303479+0000","last_clean":"2024-09-20T22:54:03.303479+0000","last_became_active":"2024-09-20T22:51:20.296204+0000","last_became_peered":"2024-09-20T22:51:20.296204+0000","last_unstale":"2024-09-20T22:54:03.303479+0000","last_undegraded":"2024-09-20T22:54:03.303479+0000","last_fullsized":"2024-09-20T22:54:03.303479+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:10.650428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:53:10.650428+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:28.093442+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.8","version":"763'973","reported_seq":4176,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-20T22:54:03.324760+0000","last_change":"2024-09-20T22:54:03.325033+0000","last_active":"2024-09-20T22:54:03.324760+0000","last_peered":"2024-09-20T22:54:03.324760+0000","last_clean":"2024-09-20T22:54:03.324760+0000","last_became_active":"2024-09-20T22:48:31.224359+0000","last_became_peered":"2024-09-20T22:48:31.224359+0000","last_unstale":"2024-09-20T22:54:03.324760+0000","last_undegraded":"2024-09-20T22:54:03.324760+0000","last_fullsized":"2024-09-20T22:54:03.324760+0000","mapping_epoch":407,"log_start":"426'268","ondisk_log_start":"426'268","created":94,"last_epoch_clean":431,"parent":"0.0","parent_split_bits":0,"last_scrub":"674'746","last_scrub_stamp":"2024-09-20T22:52:35.065256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:52:35.065256+0000","objects_scrubbed":20,"log_size":705,"log_dups_size":445,"ondisk_log_size":705,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.0051746020000000004,"stat_sum":{"num_bytes":36011408,"num_objects":16,"num_object_clones":13,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":1,"num_read":315,"num_read_kb":331223,"num_write":242,"num_write_kb":79804,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":26027409,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"118","length":"9"},{"start":"122","length":"2"},{"start":"125","length":"2"},{"start":"128","length":"2"},{"start":"12b","length":"3"},{"start":"131","length":"1"},{"start":"133","length":"6"},{"start":"13b","length":"3"},{"start":"13f","length":"2"},{"start":"143","length":"2"},{"start":"146","length":"2"},{"start":"14b","length":"1"},{"start":"14d","length":"1"},{"start":"150","length":"1"},{"start":"152","length":"1"},{"start":"154","length":"1"}]},{"pgid":"4.b","version":"625'15","reported_seq":836,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-20T22:54:03.303085+0000","last_change":"2024-09-20T22:53:30.976691+0000","last_active":"2024-09-20T22:54:03.303085+0000","last_peered":"2024-09-20T22:54:03.303085+0000","last_clean":"2024-09-20T22:54:03.303085+0000","last_became_active":"2024-09-20T22:51:20.273094+0000","last_became_peered":"2024-09-20T22:51:20.273094+0000","last_unstale":"2024-09-20T22:54:03.303085+0000","last_undegraded":"2024-09-20T22:54:03.303085+0000","last_fullsized":"2024-09-20T22:54:03.303085+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'15","last_scrub_stamp":"2024-09-20T22:53:30.976566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:53:30.976566+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:51.015430+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":1446,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-20T22:54:03.315119+0000","last_change":"2024-09-20T22:53:55.019427+0000","last_active":"2024-09-20T22:54:03.315119+0000","last_peered":"2024-09-20T22:54:03.315119+0000","last_clean":"2024-09-20T22:54:03.315119+0000","last_became_active":"2024-09-20T22:41:38.155953+0000","last_became_peered":"2024-09-20T22:41:38.155953+0000","last_unstale":"2024-09-20T22:54:03.315119+0000","last_undegraded":"2024-09-20T22:54:03.315119+0000","last_fullsized":"2024-09-20T22:54:03.315119+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:55.019362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:53:55.019362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:17.821354+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":1469,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-20T22:54:03.318154+0000","last_change":"2024-09-20T22:52:42.998852+0000","last_active":"2024-09-20T22:54:03.318154+0000","last_peered":"2024-09-20T22:54:03.318154+0000","last_clean":"2024-09-20T22:54:03.318154+0000","last_became_active":"2024-09-20T22:48:05.240613+0000","last_became_peered":"2024-09-20T22:48:05.240613+0000","last_unstale":"2024-09-20T22:54:03.318154+0000","last_undegraded":"2024-09-20T22:54:03.318154+0000","last_fullsized":"2024-09-20T22:54:03.318154+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:42.998756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:52:42.998756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:11.946882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":777,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-20T22:54:03.304057+0000","last_change":"2024-09-20T22:52:16.706497+0000","last_active":"2024-09-20T22:54:03.304057+0000","last_peered":"2024-09-20T22:54:03.304057+0000","last_clean":"2024-09-20T22:54:03.304057+0000","last_became_active":"2024-09-20T22:47:49.219486+0000","last_became_peered":"2024-09-20T22:47:49.219486+0000","last_unstale":"2024-09-20T22:54:03.304057+0000","last_undegraded":"2024-09-20T22:54:03.304057+0000","last_fullsized":"2024-09-20T22:54:03.304057+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:16.706429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:52:16.706429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:07.873999+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"763'1306","reported_seq":4721,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-20T22:54:03.313593+0000","last_change":"2024-09-20T22:54:03.313990+0000","last_active":"2024-09-20T22:54:03.313593+0000","last_peered":"2024-09-20T22:54:03.313593+0000","last_clean":"2024-09-20T22:54:03.313593+0000","last_became_active":"2024-09-20T22:51:23.270962+0000","last_became_peered":"2024-09-20T22:51:23.270962+0000","last_unstale":"2024-09-20T22:54:03.313593+0000","last_undegraded":"2024-09-20T22:54:03.313593+0000","last_fullsized":"2024-09-20T22:54:03.313593+0000","mapping_epoch":602,"log_start":"423'280","ondisk_log_start":"423'280","created":94,"last_epoch_clean":603,"parent":"0.0","parent_split_bits":0,"last_scrub":"674'1040","last_scrub_stamp":"2024-09-20T22:52:34.647394+0000","last_deep_scrub":"590'828","last_deep_scrub_stamp":"2024-09-20T22:51:10.909364+0000","last_clean_scrub_stamp":"2024-09-20T22:52:34.647394+0000","objects_scrubbed":29,"log_size":1026,"log_dups_size":649,"ondisk_log_size":1026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0094932980000000007,"stat_sum":{"num_bytes":54960064,"num_objects":22,"num_object_clones":17,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":2,"num_read":475,"num_read_kb":488473,"num_write":341,"num_write_kb":132279,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":91020358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"118","length":"9"},{"start":"122","length":"2"},{"start":"125","length":"2"},{"start":"128","length":"2"},{"start":"12b","length":"3"},{"start":"131","length":"1"},{"start":"133","length":"6"},{"start":"13b","length":"3"},{"start":"13f","length":"2"},{"start":"143","length":"2"},{"start":"146","length":"2"},{"start":"14b","length":"1"},{"start":"14d","length":"1"},{"start":"150","length":"1"},{"start":"152","length":"1"},{"start":"154","length":"1"}]},{"pgid":"4.a","version":"625'44","reported_seq":1547,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-20T22:54:03.310737+0000","last_change":"2024-09-20T22:53:50.910522+0000","last_active":"2024-09-20T22:54:03.310737+0000","last_peered":"2024-09-20T22:54:03.310737+0000","last_clean":"2024-09-20T22:54:03.310737+0000","last_became_active":"2024-09-20T22:51:20.287620+0000","last_became_peered":"2024-09-20T22:51:20.287620+0000","last_unstale":"2024-09-20T22:54:03.310737+0000","last_undegraded":"2024-09-20T22:54:03.310737+0000","last_fullsized":"2024-09-20T22:54:03.310737+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'44","last_scrub_stamp":"2024-09-20T22:53:50.910441+0000","last_deep_scrub":"113'28","last_deep_scrub_stamp":"2024-09-20T22:43:12.725900+0000","last_clean_scrub_stamp":"2024-09-20T22:53:50.910441+0000","objects_scrubbed":9,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:12.635234+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":1444,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-20T22:54:03.308696+0000","last_change":"2024-09-20T22:53:27.585334+0000","last_active":"2024-09-20T22:54:03.308696+0000","last_peered":"2024-09-20T22:54:03.308696+0000","last_clean":"2024-09-20T22:54:03.308696+0000","last_became_active":"2024-09-20T22:41:38.161524+0000","last_became_peered":"2024-09-20T22:41:38.161524+0000","last_unstale":"2024-09-20T22:54:03.308696+0000","last_undegraded":"2024-09-20T22:54:03.308696+0000","last_fullsized":"2024-09-20T22:54:03.308696+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:27.585262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:53:27.585262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:52.375797+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":1457,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-20T22:54:03.320032+0000","last_change":"2024-09-20T22:52:40.004730+0000","last_active":"2024-09-20T22:54:03.320032+0000","last_peered":"2024-09-20T22:54:03.320032+0000","last_clean":"2024-09-20T22:54:03.320032+0000","last_became_active":"2024-09-20T22:47:49.223220+0000","last_became_peered":"2024-09-20T22:47:49.223220+0000","last_unstale":"2024-09-20T22:54:03.320032+0000","last_undegraded":"2024-09-20T22:54:03.320032+0000","last_fullsized":"2024-09-20T22:54:03.320032+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:40.004661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:52:40.004661+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:04.999859+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":1440,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-20T22:54:03.308151+0000","last_change":"2024-09-20T22:53:53.365864+0000","last_active":"2024-09-20T22:54:03.308151+0000","last_peered":"2024-09-20T22:54:03.308151+0000","last_clean":"2024-09-20T22:54:03.308151+0000","last_became_active":"2024-09-20T22:41:44.199395+0000","last_became_peered":"2024-09-20T22:41:44.199395+0000","last_unstale":"2024-09-20T22:54:03.308151+0000","last_undegraded":"2024-09-20T22:54:03.308151+0000","last_fullsized":"2024-09-20T22:54:03.308151+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:53.365789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:47:18.329209+0000","last_clean_scrub_stamp":"2024-09-20T22:53:53.365789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:06.119698+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.a","version":"762'920","reported_seq":4169,"reported_epoch":763,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T22:54:03.304506+0000","last_change":"2024-09-20T22:54:02.428164+0000","last_active":"2024-09-20T22:54:03.304506+0000","last_peered":"2024-09-20T22:54:03.304506+0000","last_clean":"2024-09-20T22:54:03.304506+0000","last_became_active":"2024-09-20T22:42:52.405085+0000","last_became_peered":"2024-09-20T22:42:52.405085+0000","last_unstale":"2024-09-20T22:54:03.304506+0000","last_undegraded":"2024-09-20T22:54:03.304506+0000","last_fullsized":"2024-09-20T22:54:03.304506+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"505'466","last_scrub_stamp":"2024-09-20T22:49:47.088165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:49:47.088165+0000","objects_scrubbed":13,"log_size":920,"log_dups_size":0,"ondisk_log_size":920,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":11,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.11283753000000001,"stat_sum":{"num_bytes":37226785,"num_objects":16,"num_object_clones":13,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":365,"num_read_kb":380456,"num_write":281,"num_write_kb":112064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"118","length":"9"},{"start":"122","length":"2"},{"start":"125","length":"2"},{"start":"128","length":"2"},{"start":"12b","length":"3"},{"start":"131","length":"1"},{"start":"133","length":"6"},{"start":"13c","length":"1"},{"start":"13f","length":"2"}]},{"pgid":"4.9","version":"686'23","reported_seq":806,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-20T22:54:03.302897+0000","last_change":"2024-09-20T22:52:46.768898+0000","last_active":"2024-09-20T22:54:03.302897+0000","last_peered":"2024-09-20T22:54:03.302897+0000","last_clean":"2024-09-20T22:54:03.302897+0000","last_became_active":"2024-09-20T22:52:11.286549+0000","last_became_peered":"2024-09-20T22:52:11.286549+0000","last_unstale":"2024-09-20T22:54:03.302897+0000","last_undegraded":"2024-09-20T22:54:03.302897+0000","last_fullsized":"2024-09-20T22:54:03.302897+0000","mapping_epoch":650,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":651,"parent":"0.0","parent_split_bits":0,"last_scrub":"686'23","last_scrub_stamp":"2024-09-20T22:52:46.768812+0000","last_deep_scrub":"686'23","last_deep_scrub_stamp":"2024-09-20T22:52:46.768812+0000","last_clean_scrub_stamp":"2024-09-20T22:52:46.768812+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":21,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":1482,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-20T22:54:03.307754+0000","last_change":"2024-09-20T22:53:35.478160+0000","last_active":"2024-09-20T22:54:03.307754+0000","last_peered":"2024-09-20T22:54:03.307754+0000","last_clean":"2024-09-20T22:54:03.307754+0000","last_became_active":"2024-09-20T22:51:20.293701+0000","last_became_peered":"2024-09-20T22:51:20.293701+0000","last_unstale":"2024-09-20T22:54:03.307754+0000","last_undegraded":"2024-09-20T22:54:03.307754+0000","last_fullsized":"2024-09-20T22:54:03.307754+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:35.478053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:46:52.465940+0000","last_clean_scrub_stamp":"2024-09-20T22:53:35.478053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:46.753808+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":1471,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-20T22:54:03.320281+0000","last_change":"2024-09-20T22:53:07.104977+0000","last_active":"2024-09-20T22:54:03.320281+0000","last_peered":"2024-09-20T22:54:03.320281+0000","last_clean":"2024-09-20T22:54:03.320281+0000","last_became_active":"2024-09-20T22:51:20.286626+0000","last_became_peered":"2024-09-20T22:51:20.286626+0000","last_unstale":"2024-09-20T22:54:03.320281+0000","last_undegraded":"2024-09-20T22:54:03.320281+0000","last_fullsized":"2024-09-20T22:54:03.320281+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:07.104891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:53:07.104891+0000","last_clean_scrub_stamp":"2024-09-20T22:53:07.104891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:17.124586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":1431,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-20T22:54:03.304322+0000","last_change":"2024-09-20T22:53:53.242276+0000","last_active":"2024-09-20T22:54:03.304322+0000","last_peered":"2024-09-20T22:54:03.304322+0000","last_clean":"2024-09-20T22:54:03.304322+0000","last_became_active":"2024-09-20T22:51:20.279088+0000","last_became_peered":"2024-09-20T22:51:20.279088+0000","last_unstale":"2024-09-20T22:54:03.304322+0000","last_undegraded":"2024-09-20T22:54:03.304322+0000","last_fullsized":"2024-09-20T22:54:03.304322+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:53.242204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:55.170379+0000","last_clean_scrub_stamp":"2024-09-20T22:53:53.242204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:14.854197+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"762'1195","reported_seq":4599,"reported_epoch":763,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T22:54:03.315241+0000","last_change":"2024-09-20T22:54:00.409307+0000","last_active":"2024-09-20T22:54:03.315241+0000","last_peered":"2024-09-20T22:54:03.315241+0000","last_clean":"2024-09-20T22:54:03.315241+0000","last_became_active":"2024-09-20T22:51:23.271244+0000","last_became_peered":"2024-09-20T22:51:23.271244+0000","last_unstale":"2024-09-20T22:54:03.315241+0000","last_undegraded":"2024-09-20T22:54:03.315241+0000","last_fullsized":"2024-09-20T22:54:03.315241+0000","mapping_epoch":602,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":603,"parent":"0.0","parent_split_bits":0,"last_scrub":"661'1000","last_scrub_stamp":"2024-09-20T22:52:23.164028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:52:23.164028+0000","objects_scrubbed":23,"log_size":1195,"log_dups_size":0,"ondisk_log_size":1195,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":15,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.095040789000000001,"stat_sum":{"num_bytes":40098527,"num_objects":18,"num_object_clones":13,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":3,"num_read":524,"num_read_kb":596028,"num_write":354,"num_write_kb":123866,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":76,"num_bytes_recovered":39288293,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"118","length":"7"},{"start":"120","length":"1"},{"start":"122","length":"2"},{"start":"125","length":"2"},{"start":"128","length":"2"},{"start":"12b","length":"3"},{"start":"131","length":"1"},{"start":"133","length":"6"}]},{"pgid":"4.8","version":"625'27","reported_seq":1477,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-20T22:54:03.315184+0000","last_change":"2024-09-20T22:51:53.020721+0000","last_active":"2024-09-20T22:54:03.315184+0000","last_peered":"2024-09-20T22:54:03.315184+0000","last_clean":"2024-09-20T22:54:03.315184+0000","last_became_active":"2024-09-20T22:41:40.252113+0000","last_became_peered":"2024-09-20T22:41:40.252113+0000","last_unstale":"2024-09-20T22:54:03.315184+0000","last_undegraded":"2024-09-20T22:54:03.315184+0000","last_fullsized":"2024-09-20T22:54:03.315184+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'27","last_scrub_stamp":"2024-09-20T22:51:53.020628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:51:53.020628+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:08.101745+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'2","reported_seq":1507,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-20T22:54:03.308210+0000","last_change":"2024-09-20T22:53:33.526131+0000","last_active":"2024-09-20T22:54:03.308210+0000","last_peered":"2024-09-20T22:54:03.308210+0000","last_clean":"2024-09-20T22:54:03.308210+0000","last_became_active":"2024-09-20T22:48:05.238098+0000","last_became_peered":"2024-09-20T22:48:05.238098+0000","last_unstale":"2024-09-20T22:54:03.308210+0000","last_undegraded":"2024-09-20T22:54:03.308210+0000","last_fullsized":"2024-09-20T22:54:03.308210+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-09-20T22:53:33.526049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:53:33.526049+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:58.400288+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"408'10","reported_seq":1622,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-20T22:54:03.308674+0000","last_change":"2024-09-20T22:53:24.628363+0000","last_active":"2024-09-20T22:54:03.308674+0000","last_peered":"2024-09-20T22:54:03.308674+0000","last_clean":"2024-09-20T22:54:03.308674+0000","last_became_active":"2024-09-20T22:48:05.239344+0000","last_became_peered":"2024-09-20T22:48:05.239344+0000","last_unstale":"2024-09-20T22:54:03.308674+0000","last_undegraded":"2024-09-20T22:54:03.308674+0000","last_fullsized":"2024-09-20T22:54:03.308674+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"408'10","last_scrub_stamp":"2024-09-20T22:53:24.628269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:53:24.628269+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:33.393650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":1438,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-20T22:54:03.317116+0000","last_change":"2024-09-20T22:51:54.066995+0000","last_active":"2024-09-20T22:54:03.317116+0000","last_peered":"2024-09-20T22:54:03.317116+0000","last_clean":"2024-09-20T22:54:03.317116+0000","last_became_active":"2024-09-20T22:47:49.227569+0000","last_became_peered":"2024-09-20T22:47:49.227569+0000","last_unstale":"2024-09-20T22:54:03.317116+0000","last_undegraded":"2024-09-20T22:54:03.317116+0000","last_fullsized":"2024-09-20T22:54:03.317116+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:54.066913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:51:54.066913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:08.101760+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":1473,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-20T22:54:03.310331+0000","last_change":"2024-09-20T22:53:11.873951+0000","last_active":"2024-09-20T22:54:03.310331+0000","last_peered":"2024-09-20T22:54:03.310331+0000","last_clean":"2024-09-20T22:54:03.310331+0000","last_became_active":"2024-09-20T22:47:49.213292+0000","last_became_peered":"2024-09-20T22:47:49.213292+0000","last_unstale":"2024-09-20T22:54:03.310331+0000","last_undegraded":"2024-09-20T22:54:03.310331+0000","last_fullsized":"2024-09-20T22:54:03.310331+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:11.873813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:53:11.873813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:18.510382+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"625'12","reported_seq":1473,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-20T22:54:03.306408+0000","last_change":"2024-09-20T22:53:26.319275+0000","last_active":"2024-09-20T22:54:03.306408+0000","last_peered":"2024-09-20T22:54:03.306408+0000","last_clean":"2024-09-20T22:54:03.306408+0000","last_became_active":"2024-09-20T22:43:09.116887+0000","last_became_peered":"2024-09-20T22:43:09.116887+0000","last_unstale":"2024-09-20T22:54:03.306408+0000","last_undegraded":"2024-09-20T22:54:03.306408+0000","last_fullsized":"2024-09-20T22:54:03.306408+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'12","last_scrub_stamp":"2024-09-20T22:53:26.319144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:53:26.319144+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:37.907176+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":1444,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-20T22:54:03.315558+0000","last_change":"2024-09-20T22:53:03.888115+0000","last_active":"2024-09-20T22:54:03.315558+0000","last_peered":"2024-09-20T22:54:03.315558+0000","last_clean":"2024-09-20T22:54:03.315558+0000","last_became_active":"2024-09-20T22:51:20.293232+0000","last_became_peered":"2024-09-20T22:51:20.293232+0000","last_unstale":"2024-09-20T22:54:03.315558+0000","last_undegraded":"2024-09-20T22:54:03.315558+0000","last_fullsized":"2024-09-20T22:54:03.315558+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:03.888040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:09.705632+0000","last_clean_scrub_stamp":"2024-09-20T22:53:03.888040+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:19.352989+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"74'1","reported_seq":1453,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-20T22:54:03.308036+0000","last_change":"2024-09-20T22:52:49.575819+0000","last_active":"2024-09-20T22:54:03.308036+0000","last_peered":"2024-09-20T22:54:03.308036+0000","last_clean":"2024-09-20T22:54:03.308036+0000","last_became_active":"2024-09-20T22:48:05.237882+0000","last_became_peered":"2024-09-20T22:48:05.237882+0000","last_unstale":"2024-09-20T22:54:03.308036+0000","last_undegraded":"2024-09-20T22:54:03.308036+0000","last_fullsized":"2024-09-20T22:54:03.308036+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-09-20T22:52:49.575702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:52:49.575702+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:06.855774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"625'15","reported_seq":804,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-20T22:54:03.307862+0000","last_change":"2024-09-20T22:53:50.889818+0000","last_active":"2024-09-20T22:54:03.307862+0000","last_peered":"2024-09-20T22:54:03.307862+0000","last_clean":"2024-09-20T22:54:03.307862+0000","last_became_active":"2024-09-20T22:47:49.231315+0000","last_became_peered":"2024-09-20T22:47:49.231315+0000","last_unstale":"2024-09-20T22:54:03.307862+0000","last_undegraded":"2024-09-20T22:54:03.307862+0000","last_fullsized":"2024-09-20T22:54:03.307862+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'15","last_scrub_stamp":"2024-09-20T22:53:50.889717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:53:50.889717+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:51.999731+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":1468,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-20T22:54:03.308117+0000","last_change":"2024-09-20T22:53:10.778908+0000","last_active":"2024-09-20T22:54:03.308117+0000","last_peered":"2024-09-20T22:54:03.308117+0000","last_clean":"2024-09-20T22:54:03.308117+0000","last_became_active":"2024-09-20T22:48:05.238801+0000","last_became_peered":"2024-09-20T22:48:05.238801+0000","last_unstale":"2024-09-20T22:54:03.308117+0000","last_undegraded":"2024-09-20T22:54:03.308117+0000","last_fullsized":"2024-09-20T22:54:03.308117+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:10.778836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:53:10.778836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:25.023394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":1435,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-20T22:54:03.306238+0000","last_change":"2024-09-20T22:52:20.410227+0000","last_active":"2024-09-20T22:54:03.306238+0000","last_peered":"2024-09-20T22:54:03.306238+0000","last_clean":"2024-09-20T22:54:03.306238+0000","last_became_active":"2024-09-20T22:51:20.279397+0000","last_became_peered":"2024-09-20T22:51:20.279397+0000","last_unstale":"2024-09-20T22:54:03.306238+0000","last_undegraded":"2024-09-20T22:54:03.306238+0000","last_fullsized":"2024-09-20T22:54:03.306238+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:20.410164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:52:20.410164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:08.164658+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":1473,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-20T22:54:03.318488+0000","last_change":"2024-09-20T22:53:58.169978+0000","last_active":"2024-09-20T22:54:03.318488+0000","last_peered":"2024-09-20T22:54:03.318488+0000","last_clean":"2024-09-20T22:54:03.318488+0000","last_became_active":"2024-09-20T22:48:05.239710+0000","last_became_peered":"2024-09-20T22:48:05.239710+0000","last_unstale":"2024-09-20T22:54:03.318488+0000","last_undegraded":"2024-09-20T22:54:03.318488+0000","last_fullsized":"2024-09-20T22:54:03.318488+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:58.169894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:53:58.169894+0000","last_clean_scrub_stamp":"2024-09-20T22:53:58.169894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:04.779145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"625'15","reported_seq":1467,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-20T22:54:03.316611+0000","last_change":"2024-09-20T22:53:53.070301+0000","last_active":"2024-09-20T22:54:03.316611+0000","last_peered":"2024-09-20T22:54:03.316611+0000","last_clean":"2024-09-20T22:54:03.316611+0000","last_became_active":"2024-09-20T22:41:40.252486+0000","last_became_peered":"2024-09-20T22:41:40.252486+0000","last_unstale":"2024-09-20T22:54:03.316611+0000","last_undegraded":"2024-09-20T22:54:03.316611+0000","last_fullsized":"2024-09-20T22:54:03.316611+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'15","last_scrub_stamp":"2024-09-20T22:53:53.070214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:53:53.070214+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:11.603607+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":793,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-20T22:54:03.302973+0000","last_change":"2024-09-20T22:53:58.829885+0000","last_active":"2024-09-20T22:54:03.302973+0000","last_peered":"2024-09-20T22:54:03.302973+0000","last_clean":"2024-09-20T22:54:03.302973+0000","last_became_active":"2024-09-20T22:47:49.229936+0000","last_became_peered":"2024-09-20T22:47:49.229936+0000","last_unstale":"2024-09-20T22:54:03.302973+0000","last_undegraded":"2024-09-20T22:54:03.302973+0000","last_fullsized":"2024-09-20T22:54:03.302973+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:58.829817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:53:58.829817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:21.096752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"74'8","reported_seq":1590,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-20T22:54:03.305455+0000","last_change":"2024-09-20T22:53:59.165210+0000","last_active":"2024-09-20T22:54:03.305455+0000","last_peered":"2024-09-20T22:54:03.305455+0000","last_clean":"2024-09-20T22:54:03.305455+0000","last_became_active":"2024-09-20T22:41:42.186352+0000","last_became_peered":"2024-09-20T22:41:42.186352+0000","last_unstale":"2024-09-20T22:54:03.305455+0000","last_undegraded":"2024-09-20T22:54:03.305455+0000","last_fullsized":"2024-09-20T22:54:03.305455+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-09-20T22:53:59.165134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:53:59.165134+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:12.296267+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":744,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-20T22:54:03.304225+0000","last_change":"2024-09-20T22:52:16.623463+0000","last_active":"2024-09-20T22:54:03.304225+0000","last_peered":"2024-09-20T22:54:03.304225+0000","last_clean":"2024-09-20T22:54:03.304225+0000","last_became_active":"2024-09-20T22:48:05.249656+0000","last_became_peered":"2024-09-20T22:48:05.249656+0000","last_unstale":"2024-09-20T22:54:03.304225+0000","last_undegraded":"2024-09-20T22:54:03.304225+0000","last_fullsized":"2024-09-20T22:54:03.304225+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:16.623368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:52:16.623368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"625'20","reported_seq":1481,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-20T22:54:03.305221+0000","last_change":"2024-09-20T22:53:02.491402+0000","last_active":"2024-09-20T22:54:03.305221+0000","last_peered":"2024-09-20T22:54:03.305221+0000","last_clean":"2024-09-20T22:54:03.305221+0000","last_became_active":"2024-09-20T22:51:20.281859+0000","last_became_peered":"2024-09-20T22:51:20.281859+0000","last_unstale":"2024-09-20T22:54:03.305221+0000","last_undegraded":"2024-09-20T22:54:03.305221+0000","last_fullsized":"2024-09-20T22:54:03.305221+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'20","last_scrub_stamp":"2024-09-20T22:53:02.491320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:53:02.491320+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:10.839410+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":1464,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-20T22:54:03.307836+0000","last_change":"2024-09-20T22:53:56.396136+0000","last_active":"2024-09-20T22:54:03.307836+0000","last_peered":"2024-09-20T22:54:03.307836+0000","last_clean":"2024-09-20T22:54:03.307836+0000","last_became_active":"2024-09-20T22:48:05.240231+0000","last_became_peered":"2024-09-20T22:48:05.240231+0000","last_unstale":"2024-09-20T22:54:03.307836+0000","last_undegraded":"2024-09-20T22:54:03.307836+0000","last_fullsized":"2024-09-20T22:54:03.307836+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:56.396065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:53:56.396065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:04.372880+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"600'11","reported_seq":1630,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-20T22:54:03.316520+0000","last_change":"2024-09-20T22:52:36.264515+0000","last_active":"2024-09-20T22:54:03.316520+0000","last_peered":"2024-09-20T22:54:03.316520+0000","last_clean":"2024-09-20T22:54:03.316520+0000","last_became_active":"2024-09-20T22:51:20.283393+0000","last_became_peered":"2024-09-20T22:51:20.283393+0000","last_unstale":"2024-09-20T22:54:03.316520+0000","last_undegraded":"2024-09-20T22:54:03.316520+0000","last_fullsized":"2024-09-20T22:54:03.316520+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"600'11","last_scrub_stamp":"2024-09-20T22:52:36.264410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:52:36.264410+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:08.101896+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":777,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-20T22:54:03.307077+0000","last_change":"2024-09-20T22:52:44.834418+0000","last_active":"2024-09-20T22:54:03.307077+0000","last_peered":"2024-09-20T22:54:03.307077+0000","last_clean":"2024-09-20T22:54:03.307077+0000","last_became_active":"2024-09-20T22:47:49.230674+0000","last_became_peered":"2024-09-20T22:47:49.230674+0000","last_unstale":"2024-09-20T22:54:03.307077+0000","last_undegraded":"2024-09-20T22:54:03.307077+0000","last_fullsized":"2024-09-20T22:54:03.307077+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:44.834352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:50:21.853070+0000","last_clean_scrub_stamp":"2024-09-20T22:52:44.834352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:07.874129+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"625'19","reported_seq":794,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-20T22:54:03.303321+0000","last_change":"2024-09-20T22:53:49.547691+0000","last_active":"2024-09-20T22:54:03.303321+0000","last_peered":"2024-09-20T22:54:03.303321+0000","last_clean":"2024-09-20T22:54:03.303321+0000","last_became_active":"2024-09-20T22:51:20.289584+0000","last_became_peered":"2024-09-20T22:51:20.289584+0000","last_unstale":"2024-09-20T22:54:03.303321+0000","last_undegraded":"2024-09-20T22:54:03.303321+0000","last_fullsized":"2024-09-20T22:54:03.303321+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'19","last_scrub_stamp":"2024-09-20T22:53:49.547607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:53:49.547607+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:14.243559+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":752,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-20T22:54:03.303233+0000","last_change":"2024-09-20T22:52:47.717598+0000","last_active":"2024-09-20T22:54:03.303233+0000","last_peered":"2024-09-20T22:54:03.303233+0000","last_clean":"2024-09-20T22:54:03.303233+0000","last_became_active":"2024-09-20T22:48:05.252026+0000","last_became_peered":"2024-09-20T22:48:05.252026+0000","last_unstale":"2024-09-20T22:54:03.303233+0000","last_undegraded":"2024-09-20T22:54:03.303233+0000","last_fullsized":"2024-09-20T22:54:03.303233+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:47.717490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:52:47.717490+0000","last_clean_scrub_stamp":"2024-09-20T22:52:47.717490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":347,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-20T22:54:03.315196+0000","last_change":"2024-09-20T22:53:56.000276+0000","last_active":"2024-09-20T22:54:03.315196+0000","last_peered":"2024-09-20T22:54:03.315196+0000","last_clean":"2024-09-20T22:54:03.315196+0000","last_became_active":"2024-09-20T22:51:20.291264+0000","last_became_peered":"2024-09-20T22:51:20.291264+0000","last_unstale":"2024-09-20T22:54:03.315196+0000","last_undegraded":"2024-09-20T22:54:03.315196+0000","last_fullsized":"2024-09-20T22:54:03.315196+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:56.000202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:53:56.000202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:01.678853+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":1437,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-20T22:54:03.304411+0000","last_change":"2024-09-20T22:52:42.582474+0000","last_active":"2024-09-20T22:54:03.304411+0000","last_peered":"2024-09-20T22:54:03.304411+0000","last_clean":"2024-09-20T22:54:03.304411+0000","last_became_active":"2024-09-20T22:47:49.216141+0000","last_became_peered":"2024-09-20T22:47:49.216141+0000","last_unstale":"2024-09-20T22:54:03.304411+0000","last_undegraded":"2024-09-20T22:54:03.304411+0000","last_fullsized":"2024-09-20T22:54:03.304411+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:42.582398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:50:20.206984+0000","last_clean_scrub_stamp":"2024-09-20T22:52:42.582398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:08.164689+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"625'19","reported_seq":1099,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-20T22:54:03.310948+0000","last_change":"2024-09-20T22:53:58.933469+0000","last_active":"2024-09-20T22:54:03.310948+0000","last_peered":"2024-09-20T22:54:03.310948+0000","last_clean":"2024-09-20T22:54:03.310948+0000","last_became_active":"2024-09-20T22:51:20.283868+0000","last_became_peered":"2024-09-20T22:51:20.283868+0000","last_unstale":"2024-09-20T22:54:03.310948+0000","last_undegraded":"2024-09-20T22:54:03.310948+0000","last_fullsized":"2024-09-20T22:54:03.310948+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'19","last_scrub_stamp":"2024-09-20T22:53:58.933385+0000","last_deep_scrub":"404'13","last_deep_scrub_stamp":"2024-09-20T22:48:04.276282+0000","last_clean_scrub_stamp":"2024-09-20T22:53:58.933385+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:13.199828+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,0],"acting":[6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":1471,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-20T22:54:03.308552+0000","last_change":"2024-09-20T22:52:42.505280+0000","last_active":"2024-09-20T22:54:03.308552+0000","last_peered":"2024-09-20T22:54:03.308552+0000","last_clean":"2024-09-20T22:54:03.308552+0000","last_became_active":"2024-09-20T22:48:05.238812+0000","last_became_peered":"2024-09-20T22:48:05.238812+0000","last_unstale":"2024-09-20T22:54:03.308552+0000","last_undegraded":"2024-09-20T22:54:03.308552+0000","last_fullsized":"2024-09-20T22:54:03.308552+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:42.505211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:31.138067+0000","last_clean_scrub_stamp":"2024-09-20T22:52:42.505211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":1447,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-20T22:54:03.304406+0000","last_change":"2024-09-20T22:53:34.345146+0000","last_active":"2024-09-20T22:54:03.304406+0000","last_peered":"2024-09-20T22:54:03.304406+0000","last_clean":"2024-09-20T22:54:03.304406+0000","last_became_active":"2024-09-20T22:51:20.288000+0000","last_became_peered":"2024-09-20T22:51:20.288000+0000","last_unstale":"2024-09-20T22:54:03.304406+0000","last_undegraded":"2024-09-20T22:54:03.304406+0000","last_fullsized":"2024-09-20T22:54:03.304406+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:34.345077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:53:34.345077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:58.604576+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":783,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-20T22:54:03.302706+0000","last_change":"2024-09-20T22:53:09.890579+0000","last_active":"2024-09-20T22:54:03.302706+0000","last_peered":"2024-09-20T22:54:03.302706+0000","last_clean":"2024-09-20T22:54:03.302706+0000","last_became_active":"2024-09-20T22:51:20.278147+0000","last_became_peered":"2024-09-20T22:51:20.278147+0000","last_unstale":"2024-09-20T22:54:03.302706+0000","last_undegraded":"2024-09-20T22:54:03.302706+0000","last_fullsized":"2024-09-20T22:54:03.302706+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:09.890501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:53:09.890501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:36.773749+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"625'24","reported_seq":1268,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-20T22:54:03.314434+0000","last_change":"2024-09-20T22:53:33.089688+0000","last_active":"2024-09-20T22:54:03.314434+0000","last_peered":"2024-09-20T22:54:03.314434+0000","last_clean":"2024-09-20T22:54:03.314434+0000","last_became_active":"2024-09-20T22:44:09.170785+0000","last_became_peered":"2024-09-20T22:44:09.170785+0000","last_unstale":"2024-09-20T22:54:03.314434+0000","last_undegraded":"2024-09-20T22:54:03.314434+0000","last_fullsized":"2024-09-20T22:54:03.314434+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'24","last_scrub_stamp":"2024-09-20T22:53:33.089580+0000","last_deep_scrub":"470'16","last_deep_scrub_stamp":"2024-09-20T22:49:10.659868+0000","last_clean_scrub_stamp":"2024-09-20T22:53:33.089580+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:46.655555+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":1452,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-20T22:54:03.315722+0000","last_change":"2024-09-20T22:53:07.945728+0000","last_active":"2024-09-20T22:54:03.315722+0000","last_peered":"2024-09-20T22:54:03.315722+0000","last_clean":"2024-09-20T22:54:03.315722+0000","last_became_active":"2024-09-20T22:51:20.293645+0000","last_became_peered":"2024-09-20T22:51:20.293645+0000","last_unstale":"2024-09-20T22:54:03.315722+0000","last_undegraded":"2024-09-20T22:54:03.315722+0000","last_fullsized":"2024-09-20T22:54:03.315722+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:07.945664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:45:24.765414+0000","last_clean_scrub_stamp":"2024-09-20T22:53:07.945664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:21.914259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":1460,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-20T22:54:03.307258+0000","last_change":"2024-09-20T22:53:09.740214+0000","last_active":"2024-09-20T22:54:03.307258+0000","last_peered":"2024-09-20T22:54:03.307258+0000","last_clean":"2024-09-20T22:54:03.307258+0000","last_became_active":"2024-09-20T22:48:05.238334+0000","last_became_peered":"2024-09-20T22:48:05.238334+0000","last_unstale":"2024-09-20T22:54:03.307258+0000","last_undegraded":"2024-09-20T22:54:03.307258+0000","last_fullsized":"2024-09-20T22:54:03.307258+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:09.740145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:50.174861+0000","last_clean_scrub_stamp":"2024-09-20T22:53:09.740145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:33.910113+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":1429,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-20T22:54:03.304820+0000","last_change":"2024-09-20T22:51:04.379662+0000","last_active":"2024-09-20T22:54:03.304820+0000","last_peered":"2024-09-20T22:54:03.304820+0000","last_clean":"2024-09-20T22:54:03.304820+0000","last_became_active":"2024-09-20T22:47:49.214487+0000","last_became_peered":"2024-09-20T22:47:49.214487+0000","last_unstale":"2024-09-20T22:54:03.304820+0000","last_undegraded":"2024-09-20T22:54:03.304820+0000","last_fullsized":"2024-09-20T22:54:03.304820+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:04.379558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:51:04.379558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:08.164465+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"625'8","reported_seq":1457,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-20T22:54:03.302941+0000","last_change":"2024-09-20T22:53:05.489357+0000","last_active":"2024-09-20T22:54:03.302941+0000","last_peered":"2024-09-20T22:54:03.302941+0000","last_clean":"2024-09-20T22:54:03.302941+0000","last_became_active":"2024-09-20T22:51:20.290269+0000","last_became_peered":"2024-09-20T22:51:20.290269+0000","last_unstale":"2024-09-20T22:54:03.302941+0000","last_undegraded":"2024-09-20T22:54:03.302941+0000","last_fullsized":"2024-09-20T22:54:03.302941+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'8","last_scrub_stamp":"2024-09-20T22:53:05.489279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:53:05.489279+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:30.438834+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":785,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-20T22:54:03.306627+0000","last_change":"2024-09-20T22:53:07.918178+0000","last_active":"2024-09-20T22:54:03.306627+0000","last_peered":"2024-09-20T22:54:03.306627+0000","last_clean":"2024-09-20T22:54:03.306627+0000","last_became_active":"2024-09-20T22:47:49.231964+0000","last_became_peered":"2024-09-20T22:47:49.231964+0000","last_unstale":"2024-09-20T22:54:03.306627+0000","last_undegraded":"2024-09-20T22:54:03.306627+0000","last_fullsized":"2024-09-20T22:54:03.306627+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:07.918108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:53:07.918108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:20.150259+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":1477,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-20T22:54:03.310583+0000","last_change":"2024-09-20T22:53:51.937667+0000","last_active":"2024-09-20T22:54:03.310583+0000","last_peered":"2024-09-20T22:54:03.310583+0000","last_clean":"2024-09-20T22:54:03.310583+0000","last_became_active":"2024-09-20T22:48:05.244659+0000","last_became_peered":"2024-09-20T22:48:05.244659+0000","last_unstale":"2024-09-20T22:54:03.310583+0000","last_undegraded":"2024-09-20T22:54:03.310583+0000","last_fullsized":"2024-09-20T22:54:03.310583+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:51.937584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:53:51.937584+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:07.795334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":1452,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-20T22:54:03.308572+0000","last_change":"2024-09-20T22:53:05.612142+0000","last_active":"2024-09-20T22:54:03.308572+0000","last_peered":"2024-09-20T22:54:03.308572+0000","last_clean":"2024-09-20T22:54:03.308572+0000","last_became_active":"2024-09-20T22:48:05.237765+0000","last_became_peered":"2024-09-20T22:48:05.237765+0000","last_unstale":"2024-09-20T22:54:03.308572+0000","last_undegraded":"2024-09-20T22:54:03.308572+0000","last_fullsized":"2024-09-20T22:54:03.308572+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:05.612030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:10.233487+0000","last_clean_scrub_stamp":"2024-09-20T22:53:05.612030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:09.643188+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":1050,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-20T22:54:03.316460+0000","last_change":"2024-09-20T22:53:58.003345+0000","last_active":"2024-09-20T22:54:03.316460+0000","last_peered":"2024-09-20T22:54:03.316460+0000","last_clean":"2024-09-20T22:54:03.316460+0000","last_became_active":"2024-09-20T22:51:20.297739+0000","last_became_peered":"2024-09-20T22:51:20.297739+0000","last_unstale":"2024-09-20T22:54:03.316460+0000","last_undegraded":"2024-09-20T22:54:03.316460+0000","last_fullsized":"2024-09-20T22:54:03.316460+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:58.003274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:53:58.003274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:11.432000+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":1469,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-20T22:54:03.302570+0000","last_change":"2024-09-20T22:53:31.274457+0000","last_active":"2024-09-20T22:54:03.302570+0000","last_peered":"2024-09-20T22:54:03.302570+0000","last_clean":"2024-09-20T22:54:03.302570+0000","last_became_active":"2024-09-20T22:51:20.274891+0000","last_became_peered":"2024-09-20T22:51:20.274891+0000","last_unstale":"2024-09-20T22:54:03.302570+0000","last_undegraded":"2024-09-20T22:54:03.302570+0000","last_fullsized":"2024-09-20T22:54:03.302570+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:31.274390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:53:31.274390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:36.584363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"625'19","reported_seq":1509,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-20T22:54:03.312384+0000","last_change":"2024-09-20T22:53:30.953158+0000","last_active":"2024-09-20T22:54:03.312384+0000","last_peered":"2024-09-20T22:54:03.312384+0000","last_clean":"2024-09-20T22:54:03.312384+0000","last_became_active":"2024-09-20T22:51:20.284937+0000","last_became_peered":"2024-09-20T22:51:20.284937+0000","last_unstale":"2024-09-20T22:54:03.312384+0000","last_undegraded":"2024-09-20T22:54:03.312384+0000","last_fullsized":"2024-09-20T22:54:03.312384+0000","mapping_epoch":599,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":600,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'19","last_scrub_stamp":"2024-09-20T22:53:30.953067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:53:30.953067+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:59.104815+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":778,"reported_epoch":763,"state":"active+clean","last_fresh":"2024-09-20T22:54:03.307460+0000","last_change":"2024-09-20T22:52:18.698154+0000","last_active":"2024-09-20T22:54:03.307460+0000","last_peered":"2024-09-20T22:54:03.307460+0000","last_clean":"2024-09-20T22:54:03.307460+0000","last_became_active":"2024-09-20T22:47:49.220971+0000","last_became_peered":"2024-09-20T22:47:49.220971+0000","last_unstale":"2024-09-20T22:54:03.307460+0000","last_undegraded":"2024-09-20T22:54:03.307460+0000","last_fullsized":"2024-09-20T22:54:03.307460+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:18.698087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:51.667341+0000","last_clean_scrub_stamp":"2024-09-20T22:52:18.698087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:07.874095+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":579278855,"num_objects":259,"num_object_clones":210,"num_object_copies":777,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":259,"num_whiteouts":15,"num_read":5234,"num_read_kb":5464161,"num_write":3800,"num_write_kb":1418503,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":790,"num_bytes_recovered":519843158,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1235206144,"data_stored":1786201299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":11651,"ondisk_log_size":11651,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86,"ondisk_log_size":86,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1109,"num_read_kb":866,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":312,"num_bytes_recovered":7464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":499712,"data_stored":13224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":744,"ondisk_log_size":744,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":1034,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":4931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":735,"num_read_kb":730,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1186,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":384,"ondisk_log_size":384,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070617,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":254040,"kb_used_data":200640,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93491112,"statfs":{"total":95995035648,"available":95734898688,"internally_reserved":0,"allocated":205455360,"data_stored":282734614,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":54655434},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430367,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":222648,"kb_used_data":169248,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93522504,"statfs":{"total":95995035648,"available":95767044096,"internally_reserved":0,"allocated":173309952,"data_stored":230891914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798692004,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":178948,"kb_used_data":121392,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":93566204,"statfs":{"total":95995035648,"available":95811792896,"internally_reserved":0,"allocated":124305408,"data_stored":178795565,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986345,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":136768,"kb_used_data":101416,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93608384,"statfs":{"total":95995035648,"available":95854985216,"internally_reserved":0,"allocated":103849984,"data_stored":146621889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182575,"num_pgs":76,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":350424,"kb_used_data":274816,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93394728,"statfs":{"total":95995035648,"available":95636201472,"internally_reserved":0,"allocated":281411584,"data_stored":399287204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":77396423},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411508,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":251196,"kb_used_data":201960,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":93493956,"statfs":{"total":95995035648,"available":95737810944,"internally_reserved":0,"allocated":206807040,"data_stored":306360914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":55834575033,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738558,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":236488,"kb_used_data":183088,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93508664,"statfs":{"total":95995035648,"available":95752871936,"internally_reserved":0,"allocated":187482112,"data_stored":264205254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":180334592,"data_stored":259858166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":200880128,"data_stored":304925139,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":275124224,"data_stored":397522390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":96702464,"data_stored":142268087,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":118038528,"data_stored":177030906,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":166158336,"data_stored":226544445,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":197967872,"data_stored":278052166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-20T22:54:06.682 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd pool set .mgr pgp_num 11 2024-09-20T22:54:06.682 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:54:06.703 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:06.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:06.703 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:54:06.740 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-20T22:54:06.804 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:54:06.828 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:06.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:06.829 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:54:06.860 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-20T22:54:06.860 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [], 2024-09-20T22:54:06.860 INFO:teuthology.orchestra.run.smithi154.stdout: "complaint_time": 30, 2024-09-20T22:54:06.860 INFO:teuthology.orchestra.run.smithi154.stdout: "num_blocked_ops": 0 2024-09-20T22:54:06.861 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-20T22:54:06.929 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:54:06.952 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:06.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:06.953 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:54:07.004 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:54:07.028 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:54:07.053 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:54:07.073 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:07.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:07.074 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:54:07.175 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:54:07.197 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:07.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:07.198 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:54:07.298 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:54:07.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:07.323 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:07.323 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:54:07.323 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is set 2024-09-20T22:54:07.326 INFO:tasks.rados.rados.0.smithi084.stdout:3041: done (4 left) 2024-09-20T22:54:07.326 INFO:tasks.rados.rados.0.smithi084.stdout:3042: done (3 left) 2024-09-20T22:54:07.326 INFO:tasks.rados.rados.0.smithi084.stdout:3043: done (2 left) 2024-09-20T22:54:07.326 INFO:tasks.rados.rados.0.smithi084.stdout:3044: done (1 left) 2024-09-20T22:54:07.326 INFO:tasks.rados.rados.0.smithi084.stdout:3045: done (0 left) 2024-09-20T22:54:07.326 INFO:tasks.rados.rados.0.smithi084.stdout:3046: read oid 34 snap -1 2024-09-20T22:54:07.326 INFO:tasks.rados.rados.0.smithi084.stdout:3046: expect deleted 2024-09-20T22:54:07.326 INFO:tasks.rados.rados.0.smithi084.stdout:3047: write oid 40 current snap is 345 2024-09-20T22:54:07.326 INFO:tasks.rados.rados.0.smithi084.stdout:3047: seq_num 1050 ranges {447350=668164,1560093=406693,2584613=480437} 2024-09-20T22:54:07.331 INFO:tasks.rados.rados.0.smithi084.stdout:3047: writing smithi08477203-40 from 447350 to 1115514 tid 1 2024-09-20T22:54:07.337 INFO:tasks.rados.rados.0.smithi084.stdout:3047: writing smithi08477203-40 from 1560093 to 1966786 tid 2 2024-09-20T22:54:07.343 INFO:tasks.rados.rados.0.smithi084.stdout:3047: writing smithi08477203-40 from 2584613 to 3065050 tid 3 2024-09-20T22:54:07.343 INFO:tasks.rados.rados.0.smithi084.stdout:3048: write oid 44 current snap is 345 2024-09-20T22:54:07.344 INFO:tasks.rados.rados.0.smithi084.stdout:3048: seq_num 1051 ranges {464157=743413,1618575=666704,3012036=408062} 2024-09-20T22:54:07.351 INFO:tasks.rados.rados.0.smithi084.stdout:3048: writing smithi08477203-44 from 464157 to 1207570 tid 1 2024-09-20T22:54:07.357 INFO:tasks.rados.rados.0.smithi084.stdout:3048: writing smithi08477203-44 from 1618575 to 2285279 tid 2 2024-09-20T22:54:07.363 INFO:tasks.rados.rados.0.smithi084.stdout:3048: writing smithi08477203-44 from 3012036 to 3420098 tid 3 2024-09-20T22:54:07.363 INFO:tasks.rados.rados.0.smithi084.stdout:3049: read oid 7 snap -1 2024-09-20T22:54:07.363 INFO:tasks.rados.rados.0.smithi084.stdout:3049: expect (ObjNum 964 snap 314 seq_num 964) 2024-09-20T22:54:07.363 INFO:tasks.rados.rados.0.smithi084.stdout:3050: snap_remove snap 257 2024-09-20T22:54:07.389 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:07 smithi084 ceph-mon[29774]: pgmap v1050: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 553 MiB data, 1.6 GiB used, 624 GiB / 626 GiB avail; 7.3 MiB/s rd, 2.2 MiB/s wr, 11 op/s 2024-09-20T22:54:07.389 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:07 smithi084 ceph-mon[29774]: from='client.16947 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T22:54:07.389 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:07 smithi084 ceph-mon[29774]: osdmap e766: 8 total, 8 up, 7 in 2024-09-20T22:54:07.389 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:07 smithi084 ceph-mon[29774]: purged_snaps scrub starts 2024-09-20T22:54:07.389 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:07 smithi084 ceph-mon[29774]: purged_snaps scrub ok 2024-09-20T22:54:07.389 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:07 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/4285773149' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:54:07.389 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:07 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:54:07.389 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:07 smithi084 ceph-mon[40024]: pgmap v1050: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 553 MiB data, 1.6 GiB used, 624 GiB / 626 GiB avail; 7.3 MiB/s rd, 2.2 MiB/s wr, 11 op/s 2024-09-20T22:54:07.389 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:07 smithi084 ceph-mon[40024]: from='client.16947 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T22:54:07.389 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:07 smithi084 ceph-mon[40024]: osdmap e766: 8 total, 8 up, 7 in 2024-09-20T22:54:07.389 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:07 smithi084 ceph-mon[40024]: purged_snaps scrub starts 2024-09-20T22:54:07.390 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:07 smithi084 ceph-mon[40024]: purged_snaps scrub ok 2024-09-20T22:54:07.390 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:07 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/4285773149' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:54:07.390 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:07 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:54:07.424 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:54:07.446 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:07.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:07.447 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:54:07.475 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.5 dump_historic_ops 2024-09-20T22:54:07.548 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:54:07.580 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:07.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:07.581 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:54:07.682 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:54:07.703 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:07.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:07.704 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:54:07.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:07 smithi154 ceph-mon[33264]: pgmap v1050: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 553 MiB data, 1.6 GiB used, 624 GiB / 626 GiB avail; 7.3 MiB/s rd, 2.2 MiB/s wr, 11 op/s 2024-09-20T22:54:07.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:07 smithi154 ceph-mon[33264]: from='client.16947 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T22:54:07.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:07 smithi154 ceph-mon[33264]: osdmap e766: 8 total, 8 up, 7 in 2024-09-20T22:54:07.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:07 smithi154 ceph-mon[33264]: purged_snaps scrub starts 2024-09-20T22:54:07.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:07 smithi154 ceph-mon[33264]: purged_snaps scrub ok 2024-09-20T22:54:07.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:07 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/4285773149' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:54:07.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:07 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:54:07.766 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:54:07.805 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:54:07.827 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:07.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:07.828 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:54:07.928 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:54:07.950 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:07.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:07.951 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:54:08.052 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:54:08.075 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:08.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:08.076 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:54:08.155 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:54:08.155 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:54:08.172 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:54:08.172 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:54:08.176 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:54:08.188 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:54:08.188 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:54:08.197 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:08.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:08.198 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:54:08.204 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-20T22:54:08.205 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T22:54:08.221 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:54:08.221 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:54:08.237 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:54:08.238 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:54:08.255 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:54:08.256 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:54:08.272 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:54:08.272 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:54:08.298 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:54:08.311 INFO:tasks.rados.rados.0.smithi084.stdout:3050: done (4 left) 2024-09-20T22:54:08.311 INFO:tasks.rados.rados.0.smithi084.stdout:3051: write oid 35 current snap is 345 2024-09-20T22:54:08.311 INFO:tasks.rados.rados.0.smithi084.stdout:3051: seq_num 1052 ranges {480964=418662,1560704=410362,2523106=619334} 2024-09-20T22:54:08.316 INFO:tasks.rados.rados.0.smithi084.stdout:3051: writing smithi08477203-35 from 480964 to 899626 tid 1 2024-09-20T22:54:08.323 INFO:tasks.rados.rados.0.smithi084.stdout:3051: writing smithi08477203-35 from 1560704 to 1971066 tid 2 2024-09-20T22:54:08.326 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:08.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:08.327 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:54:08.331 INFO:tasks.rados.rados.0.smithi084.stdout:3051: writing smithi08477203-35 from 2523106 to 3142440 tid 3 2024-09-20T22:54:08.331 INFO:tasks.rados.rados.0.smithi084.stdout:3052: read oid 39 snap 311 2024-09-20T22:54:08.331 INFO:tasks.rados.rados.0.smithi084.stdout:3052: expect deleted 2024-09-20T22:54:08.331 INFO:tasks.rados.rados.0.smithi084.stdout:3053: read oid 43 snap -1 2024-09-20T22:54:08.331 INFO:tasks.rados.rados.0.smithi084.stdout:3053: expect (ObjNum 1019 snap 333 seq_num 1019) 2024-09-20T22:54:08.332 INFO:tasks.rados.rados.0.smithi084.stdout:3047: finishing write tid 1 to smithi08477203-40 2024-09-20T22:54:08.332 INFO:tasks.rados.rados.0.smithi084.stdout:3047: finishing write tid 2 to smithi08477203-40 2024-09-20T22:54:08.332 INFO:tasks.rados.rados.0.smithi084.stdout:3047: finishing write tid 3 to smithi08477203-40 2024-09-20T22:54:08.332 INFO:tasks.rados.rados.0.smithi084.stdout:3047: finishing write tid 4 to smithi08477203-40 2024-09-20T22:54:08.332 INFO:tasks.rados.rados.0.smithi084.stdout:3047: finishing write tid 5 to smithi08477203-40 2024-09-20T22:54:08.332 INFO:tasks.rados.rados.0.smithi084.stdout:3047: oid 40 updating version 0 to 1906 2024-09-20T22:54:08.332 INFO:tasks.rados.rados.0.smithi084.stdout:3047: oid 40 version 1906 is already newer than 1905 2024-09-20T22:54:08.332 INFO:tasks.rados.rados.0.smithi084.stdout:3047: oid 40 updating version 1906 to 1908 2024-09-20T22:54:08.332 INFO:tasks.rados.rados.0.smithi084.stdout:3047: oid 40 version 1908 is already newer than 1907 2024-09-20T22:54:08.332 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 40 v 1908 (ObjNum 1050 snap 345 seq_num 1050) dirty exists 2024-09-20T22:54:08.332 INFO:tasks.rados.rados.0.smithi084.stdout:3047: left oid 40 (ObjNum 1050 snap 345 seq_num 1050) 2024-09-20T22:54:08.332 INFO:tasks.rados.rados.0.smithi084.stdout:3048: finishing write tid 1 to smithi08477203-44 2024-09-20T22:54:08.333 INFO:tasks.rados.rados.0.smithi084.stdout:3048: finishing write tid 2 to smithi08477203-44 2024-09-20T22:54:08.333 INFO:tasks.rados.rados.0.smithi084.stdout:3049: expect (ObjNum 964 snap 314 seq_num 964) 2024-09-20T22:54:08.345 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-20T22:54:08.345 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:54:08.428 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:54:08.450 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:08.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:08.451 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:54:08.457 INFO:tasks.rados.rados.0.smithi084.stdout:3048: finishing write tid 3 to smithi08477203-44 2024-09-20T22:54:08.458 INFO:tasks.rados.rados.0.smithi084.stdout:3048: finishing write tid 4 to smithi08477203-44 2024-09-20T22:54:08.458 INFO:tasks.rados.rados.0.smithi084.stdout:3048: finishing write tid 5 to smithi08477203-44 2024-09-20T22:54:08.458 INFO:tasks.rados.rados.0.smithi084.stdout:3048: oid 44 updating version 0 to 2453 2024-09-20T22:54:08.458 INFO:tasks.rados.rados.0.smithi084.stdout:3048: oid 44 version 2453 is already newer than 2452 2024-09-20T22:54:08.458 INFO:tasks.rados.rados.0.smithi084.stdout:3048: oid 44 version 2453 is already newer than 2450 2024-09-20T22:54:08.458 INFO:tasks.rados.rados.0.smithi084.stdout:3048: oid 44 version 2453 is already newer than 2451 2024-09-20T22:54:08.458 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 44 v 2453 (ObjNum 1051 snap 345 seq_num 1051) dirty exists 2024-09-20T22:54:08.458 INFO:tasks.rados.rados.0.smithi084.stdout:3048: left oid 44 (ObjNum 1051 snap 345 seq_num 1051) 2024-09-20T22:54:08.458 INFO:tasks.rados.rados.0.smithi084.stdout:3054: read oid 20 snap 342 2024-09-20T22:54:08.458 INFO:tasks.rados.rados.0.smithi084.stdout:3054: expect (ObjNum 1047 snap 342 seq_num 1047) 2024-09-20T22:54:08.458 INFO:tasks.rados.rados.0.smithi084.stdout:3051: finishing write tid 1 to smithi08477203-35 2024-09-20T22:54:08.458 INFO:tasks.rados.rados.0.smithi084.stdout:3053: expect (ObjNum 1019 snap 333 seq_num 1019) 2024-09-20T22:54:08.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:08 smithi154 ceph-mon[33264]: purged_snaps scrub starts 2024-09-20T22:54:08.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:08 smithi154 ceph-mon[33264]: purged_snaps scrub ok 2024-09-20T22:54:08.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:08 smithi154 ceph-mon[33264]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:54:08.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:08 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T22:54:08.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:08 smithi154 ceph-mon[33264]: osdmap e767: 8 total, 8 up, 7 in 2024-09-20T22:54:08.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:08 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/4036408009' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:54:08.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:08 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:54:08.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:08 smithi154 ceph-mon[33264]: osdmap e768: 8 total, 8 up, 7 in 2024-09-20T22:54:08.552 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:54:08.553 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:08 smithi084 ceph-mon[40024]: purged_snaps scrub starts 2024-09-20T22:54:08.553 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:08 smithi084 ceph-mon[40024]: purged_snaps scrub ok 2024-09-20T22:54:08.553 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:08 smithi084 ceph-mon[40024]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:54:08.553 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:08 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T22:54:08.554 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:08 smithi084 ceph-mon[40024]: osdmap e767: 8 total, 8 up, 7 in 2024-09-20T22:54:08.554 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:08 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/4036408009' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:54:08.554 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:08 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:54:08.554 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:08 smithi084 ceph-mon[40024]: osdmap e768: 8 total, 8 up, 7 in 2024-09-20T22:54:08.555 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:08 smithi084 ceph-mon[29774]: purged_snaps scrub starts 2024-09-20T22:54:08.555 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:08 smithi084 ceph-mon[29774]: purged_snaps scrub ok 2024-09-20T22:54:08.555 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:08 smithi084 ceph-mon[29774]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:54:08.555 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:08 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T22:54:08.555 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:08 smithi084 ceph-mon[29774]: osdmap e767: 8 total, 8 up, 7 in 2024-09-20T22:54:08.555 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:08 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/4036408009' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:54:08.556 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:08 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:54:08.556 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:08 smithi084 ceph-mon[29774]: osdmap e768: 8 total, 8 up, 7 in 2024-09-20T22:54:08.556 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:54:08 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: ::ffff:172.21.15.154 - - [20/Sep/2024:22:54:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-20T22:54:08.560 INFO:tasks.rados.rados.0.smithi084.stdout:3051: finishing write tid 2 to smithi08477203-35 2024-09-20T22:54:08.560 INFO:tasks.rados.rados.0.smithi084.stdout:3051: finishing write tid 3 to smithi08477203-35 2024-09-20T22:54:08.560 INFO:tasks.rados.rados.0.smithi084.stdout:3046: done (7 left) 2024-09-20T22:54:08.561 INFO:tasks.rados.rados.0.smithi084.stdout:3047: done (6 left) 2024-09-20T22:54:08.561 INFO:tasks.rados.rados.0.smithi084.stdout:3048: done (5 left) 2024-09-20T22:54:08.561 INFO:tasks.rados.rados.0.smithi084.stdout:3049: done (4 left) 2024-09-20T22:54:08.561 INFO:tasks.rados.rados.0.smithi084.stdout:3052: done (3 left) 2024-09-20T22:54:08.561 INFO:tasks.rados.rados.0.smithi084.stdout:3053: done (2 left) 2024-09-20T22:54:08.561 INFO:tasks.rados.rados.0.smithi084.stdout:3055: read oid 12 snap -1 2024-09-20T22:54:08.561 INFO:tasks.rados.rados.0.smithi084.stdout:3055: expect (ObjNum 795 snap 254 seq_num 795) 2024-09-20T22:54:08.561 INFO:tasks.rados.rados.0.smithi084.stdout:3056: snap_remove snap 302 2024-09-20T22:54:08.576 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:08.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:08.577 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:54:08.678 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:54:08.699 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:08.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:08.700 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:54:08.801 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:54:08.822 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:08.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:08.823 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:54:08.924 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:54:08.941 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-20T22:54:08.945 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:08.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:08.946 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:54:09.047 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:54:09.068 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:09.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:09.069 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:54:09.169 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:54:09.187 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:54:09.190 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:09.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:09.191 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:54:09.291 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:54:09.313 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:09.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:09.314 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:54:09.343 INFO:teuthology.orchestra.run.smithi084.stderr:set pool 1 pgp_num to 11 2024-09-20T22:54:09.349 INFO:tasks.rados.rados.0.smithi084.stdout:3056: done (3 left) 2024-09-20T22:54:09.349 INFO:tasks.rados.rados.0.smithi084.stdout:3057: read oid 43 snap -1 2024-09-20T22:54:09.349 INFO:tasks.rados.rados.0.smithi084.stdout:3057: expect (ObjNum 1019 snap 333 seq_num 1019) 2024-09-20T22:54:09.349 INFO:tasks.rados.rados.0.smithi084.stdout:3058: copy_from oid 18 from oid 9 current snap is 345 2024-09-20T22:54:09.349 INFO:tasks.rados.rados.0.smithi084.stdout:3059: copy_from oid 14 from oid 24 current snap is 345 2024-09-20T22:54:09.349 INFO:tasks.rados.rados.0.smithi084.stdout:3060: write oid 3 current snap is 345 2024-09-20T22:54:09.349 INFO:tasks.rados.rados.0.smithi084.stdout:3060: seq_num 1053 ranges {497771=493911,1619186=554020,2950529=430606} 2024-09-20T22:54:09.354 INFO:tasks.rados.rados.0.smithi084.stdout:3060: writing smithi08477203-3 from 497771 to 991682 tid 1 2024-09-20T22:54:09.362 INFO:tasks.rados.rados.0.smithi084.stdout:3060: writing smithi08477203-3 from 1619186 to 2173206 tid 2 2024-09-20T22:54:09.369 INFO:tasks.rados.rados.0.smithi084.stdout:3060: writing smithi08477203-3 from 2950529 to 3381135 tid 3 2024-09-20T22:54:09.369 INFO:tasks.rados.rados.0.smithi084.stdout:3061: snap_remove snap 287 2024-09-20T22:54:09.414 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:54:09.435 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:09.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:09.436 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:54:09.537 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:54:09.558 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:09.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:09.559 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:54:09.560 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:09 smithi154 ceph-mon[33264]: pgmap v1053: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 553 MiB data, 1.6 GiB used, 624 GiB / 626 GiB avail 2024-09-20T22:54:09.560 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:09 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/3791833957' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]: dispatch 2024-09-20T22:54:09.614 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:09 smithi084 ceph-mon[40024]: pgmap v1053: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 553 MiB data, 1.6 GiB used, 624 GiB / 626 GiB avail 2024-09-20T22:54:09.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:09 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/3791833957' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]: dispatch 2024-09-20T22:54:09.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:09 smithi084 ceph-mon[29774]: pgmap v1053: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 553 MiB data, 1.6 GiB used, 624 GiB / 626 GiB avail 2024-09-20T22:54:09.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:09 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/3791833957' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]: dispatch 2024-09-20T22:54:09.660 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:54:09.661 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-20T22:54:09.661 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:54:09.683 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:09.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:09.684 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:54:09.784 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:54:09.806 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:09.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:09.807 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:54:09.908 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:54:09.930 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:09.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:09.931 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:54:10.004 INFO:journalctl@ceph.iscsi.iscsi.a.smithi154.stdout:Sep 20 22:54:09 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-iscsi-iscsi-a[59730]: debug there is no tcmu-runner data available 2024-09-20T22:54:10.031 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:54:10.052 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:10.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:10.053 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:54:10.154 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:54:10.174 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:10.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:10.176 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:54:10.276 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:54:10.298 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:10.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:10.299 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:54:10.358 INFO:tasks.rados.rados.0.smithi084.stdout:3061: done (7 left) 2024-09-20T22:54:10.358 INFO:tasks.rados.rados.0.smithi084.stdout:3062: read oid 25 snap -1 2024-09-20T22:54:10.358 INFO:tasks.rados.rados.0.smithi084.stdout:3062: expect deleted 2024-09-20T22:54:10.358 INFO:tasks.rados.rados.0.smithi084.stdout:3063: rollback oid 50 current snap is 345 2024-09-20T22:54:10.359 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 50 to 304 2024-09-20T22:54:10.359 INFO:tasks.rados.rados.0.smithi084.stdout:3051: finishing write tid 4 to smithi08477203-35 2024-09-20T22:54:10.359 INFO:tasks.rados.rados.0.smithi084.stdout:3051: finishing write tid 5 to smithi08477203-35 2024-09-20T22:54:10.359 INFO:tasks.rados.rados.0.smithi084.stdout:3051: oid 35 updating version 0 to 982 2024-09-20T22:54:10.359 INFO:tasks.rados.rados.0.smithi084.stdout:3051: oid 35 updating version 982 to 988 2024-09-20T22:54:10.359 INFO:tasks.rados.rados.0.smithi084.stdout:3051: oid 35 updating version 988 to 989 2024-09-20T22:54:10.359 INFO:tasks.rados.rados.0.smithi084.stdout:3051: oid 35 version 989 is already newer than 987 2024-09-20T22:54:10.359 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 35 v 989 (ObjNum 1052 snap 345 seq_num 1052) dirty exists 2024-09-20T22:54:10.359 INFO:tasks.rados.rados.0.smithi084.stdout:3051: left oid 35 (ObjNum 1052 snap 345 seq_num 1052) 2024-09-20T22:54:10.359 INFO:tasks.rados.rados.0.smithi084.stdout:3051: done (8 left) 2024-09-20T22:54:10.359 INFO:tasks.rados.rados.0.smithi084.stdout:3064: snap_create 2024-09-20T22:54:10.359 INFO:tasks.rados.rados.0.smithi084.stdout:3054: expect (ObjNum 1047 snap 342 seq_num 1047) 2024-09-20T22:54:10.400 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:54:10.421 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:10.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:10.422 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:54:10.429 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.6 dump_blocked_ops 2024-09-20T22:54:10.523 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:54:10.544 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:10.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:10.545 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:54:10.547 INFO:tasks.rados.rados.0.smithi084.stdout:3055: expect (ObjNum 795 snap 254 seq_num 795) 2024-09-20T22:54:10.586 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd unset noscrub 2024-09-20T22:54:10.602 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:10 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/3791833957' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]': finished 2024-09-20T22:54:10.602 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:10 smithi154 ceph-mon[33264]: osdmap e769: 8 total, 8 up, 7 in 2024-09-20T22:54:10.602 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:10 smithi154 ceph-mon[33264]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:54:10.607 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:10 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/3791833957' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]': finished 2024-09-20T22:54:10.607 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:10 smithi084 ceph-mon[40024]: osdmap e769: 8 total, 8 up, 7 in 2024-09-20T22:54:10.607 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:10 smithi084 ceph-mon[40024]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:54:10.607 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:10 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/3791833957' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]': finished 2024-09-20T22:54:10.607 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:10 smithi084 ceph-mon[29774]: osdmap e769: 8 total, 8 up, 7 in 2024-09-20T22:54:10.607 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:10 smithi084 ceph-mon[29774]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:54:10.646 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:54:10.656 INFO:tasks.rados.rados.0.smithi084.stdout:3058: finishing copy_from to smithi08477203-18 2024-09-20T22:54:10.656 INFO:tasks.rados.rados.0.smithi084.stdout:3058: got expected ENOENT (src dne) 2024-09-20T22:54:10.656 INFO:tasks.rados.rados.0.smithi084.stdout:3058: finishing copy_from racing read to smithi08477203-18 2024-09-20T22:54:10.656 INFO:tasks.rados.rados.0.smithi084.stdout:3057: expect (ObjNum 1019 snap 333 seq_num 1019) 2024-09-20T22:54:10.667 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:10.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:10.668 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:54:10.721 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:54:10.758 INFO:tasks.rados.rados.0.smithi084.stdout:3060: finishing write tid 1 to smithi08477203-3 2024-09-20T22:54:10.758 INFO:tasks.rados.rados.0.smithi084.stdout:3060: finishing write tid 2 to smithi08477203-3 2024-09-20T22:54:10.758 INFO:tasks.rados.rados.0.smithi084.stdout:3059: finishing copy_from racing read to smithi08477203-14 2024-09-20T22:54:10.758 INFO:tasks.rados.rados.0.smithi084.stdout:3060: finishing write tid 3 to smithi08477203-3 2024-09-20T22:54:10.758 INFO:tasks.rados.rados.0.smithi084.stdout:3060: finishing write tid 4 to smithi08477203-3 2024-09-20T22:54:10.758 INFO:tasks.rados.rados.0.smithi084.stdout:3059: finishing copy_from to smithi08477203-14 2024-09-20T22:54:10.758 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 14 v 1873 (ObjNum 1048 snap 343 seq_num 1048) dirty exists 2024-09-20T22:54:10.758 INFO:tasks.rados.rados.0.smithi084.stdout:3060: finishing write tid 5 to smithi08477203-3 2024-09-20T22:54:10.758 INFO:tasks.rados.rados.0.smithi084.stdout:3060: oid 3 updating version 0 to 1861 2024-09-20T22:54:10.758 INFO:tasks.rados.rados.0.smithi084.stdout:3060: oid 3 version 1861 is already newer than 1860 2024-09-20T22:54:10.758 INFO:tasks.rados.rados.0.smithi084.stdout:3060: oid 3 version 1861 is already newer than 1858 2024-09-20T22:54:10.758 INFO:tasks.rados.rados.0.smithi084.stdout:3060: oid 3 version 1861 is already newer than 1859 2024-09-20T22:54:10.758 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 3 v 1861 (ObjNum 1053 snap 345 seq_num 1053) dirty exists 2024-09-20T22:54:10.759 INFO:tasks.rados.rados.0.smithi084.stdout:3060: left oid 3 (ObjNum 1053 snap 345 seq_num 1053) 2024-09-20T22:54:10.759 INFO:tasks.rados.rados.0.smithi084.stdout:3063: finishing rollback tid 0 to smithi08477203-50 2024-09-20T22:54:10.759 INFO:tasks.rados.rados.0.smithi084.stdout:3063: finishing rollback tid 1 to smithi08477203-50 2024-09-20T22:54:10.759 INFO:tasks.rados.rados.0.smithi084.stdout:3063: finishing rollback tid 2 to smithi08477203-50 2024-09-20T22:54:10.759 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 50 v 1876 (ObjNum 515 snap 148 seq_num 515) dirty exists 2024-09-20T22:54:10.769 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:54:10.790 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:10.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:10.791 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:54:10.833 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:54:10.892 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:54:10.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:10.914 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:10.914 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:54:11.014 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:54:11.036 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:11.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:11.037 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:54:11.137 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:54:11.160 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:11.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:11.161 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:54:11.232 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-20T22:54:11.232 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:54:11.262 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:54:11.283 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:11.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:11.284 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:54:11.365 INFO:tasks.rados.rados.0.smithi084.stdout:3054: done (8 left) 2024-09-20T22:54:11.365 INFO:tasks.rados.rados.0.smithi084.stdout:3055: done (7 left) 2024-09-20T22:54:11.365 INFO:tasks.rados.rados.0.smithi084.stdout:3057: done (6 left) 2024-09-20T22:54:11.365 INFO:tasks.rados.rados.0.smithi084.stdout:3058: done (5 left) 2024-09-20T22:54:11.365 INFO:tasks.rados.rados.0.smithi084.stdout:3059: done (4 left) 2024-09-20T22:54:11.365 INFO:tasks.rados.rados.0.smithi084.stdout:3060: done (3 left) 2024-09-20T22:54:11.365 INFO:tasks.rados.rados.0.smithi084.stdout:3062: done (2 left) 2024-09-20T22:54:11.366 INFO:tasks.rados.rados.0.smithi084.stdout:3063: done (1 left) 2024-09-20T22:54:11.366 INFO:tasks.rados.rados.0.smithi084.stdout:3064: done (0 left) 2024-09-20T22:54:11.366 INFO:tasks.rados.rados.0.smithi084.stdout:3065: snap_create 2024-09-20T22:54:11.385 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:54:11.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:11.406 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:11.407 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:54:11.419 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:11 smithi154 ceph-mon[33264]: pgmap v1056: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 514 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 11 MiB/s rd, 2.1 MiB/s wr, 16 op/s 2024-09-20T22:54:11.419 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:11 smithi154 ceph-mon[33264]: osdmap e770: 8 total, 8 up, 7 in 2024-09-20T22:54:11.461 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-20T22:54:11.507 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:54:11.530 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:11.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:11.531 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:54:11.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:11 smithi084 ceph-mon[40024]: pgmap v1056: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 514 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 11 MiB/s rd, 2.1 MiB/s wr, 16 op/s 2024-09-20T22:54:11.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:11 smithi084 ceph-mon[40024]: osdmap e770: 8 total, 8 up, 7 in 2024-09-20T22:54:11.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:11 smithi084 ceph-mon[29774]: pgmap v1056: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 514 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 11 MiB/s rd, 2.1 MiB/s wr, 16 op/s 2024-09-20T22:54:11.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:11 smithi084 ceph-mon[29774]: osdmap e770: 8 total, 8 up, 7 in 2024-09-20T22:54:11.632 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:54:11.655 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:11.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:11.656 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:54:11.710 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:54:11.757 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:54:11.778 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:11.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:11.779 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:54:11.879 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:54:11.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:11.901 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:11.902 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:54:11.915 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.6 dump_historic_ops 2024-09-20T22:54:12.002 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:54:12.023 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:12.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:12.024 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:54:12.125 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:54:12.147 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:12.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:12.148 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:54:12.175 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:54:12.184 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:54:12.184 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:54:12.200 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:54:12.200 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:54:12.216 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:54:12.216 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:54:12.232 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-20T22:54:12.232 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T22:54:12.248 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:54:12.248 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:54:12.248 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:54:12.264 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:54:12.265 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:54:12.269 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:12.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:12.270 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:54:12.281 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:54:12.281 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:54:12.298 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:54:12.298 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:54:12.366 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is unset 2024-09-20T22:54:12.370 INFO:tasks.rados.rados.0.smithi084.stdout:3065: done (0 left) 2024-09-20T22:54:12.370 INFO:tasks.rados.rados.0.smithi084.stdout:3066: delete oid 6 current snap is 347 2024-09-20T22:54:12.371 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:54:12.373 INFO:tasks.rados.rados.0.smithi084.stdout:3066: done (0 left) 2024-09-20T22:54:12.373 INFO:tasks.rados.rados.0.smithi084.stdout:3067: write oid 19 current snap is 347 2024-09-20T22:54:12.373 INFO:tasks.rados.rados.0.smithi084.stdout:3067: seq_num 1055 ranges {531385=644409,1736150=557689,3005375=569503} 2024-09-20T22:54:12.379 INFO:tasks.rados.rados.0.smithi084.stdout:3067: writing smithi08477203-19 from 531385 to 1175794 tid 1 2024-09-20T22:54:12.386 INFO:tasks.rados.rados.0.smithi084.stdout:3067: writing smithi08477203-19 from 1736150 to 2293839 tid 2 2024-09-20T22:54:12.393 INFO:tasks.rados.rados.0.smithi084.stdout:3067: writing smithi08477203-19 from 3005375 to 3574878 tid 3 2024-09-20T22:54:12.393 INFO:tasks.rados.rados.0.smithi084.stdout:3068: write (excl) oid 26 current snap is 347 2024-09-20T22:54:12.393 INFO:tasks.rados.rados.0.smithi084.stdout:3067: finishing write tid 1 to smithi08477203-19 2024-09-20T22:54:12.394 INFO:tasks.rados.rados.0.smithi084.stdout:3067: finishing write tid 2 to smithi08477203-19 2024-09-20T22:54:12.394 INFO:tasks.rados.rados.0.smithi084.stdout:3068: seq_num 1056 ranges {548192=719658,1678279=417700,2516445=780775} 2024-09-20T22:54:12.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:12.395 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:12.396 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:54:12.400 INFO:tasks.rados.rados.0.smithi084.stdout:3068: writing smithi08477203-26 from 548192 to 1267850 tid 1 2024-09-20T22:54:12.405 INFO:tasks.rados.rados.0.smithi084.stdout:3068: writing smithi08477203-26 from 1678279 to 2095979 tid 2 2024-09-20T22:54:12.412 INFO:tasks.rados.rados.0.smithi084.stdout:3068: writing smithi08477203-26 from 2516445 to 3297220 tid 3 2024-09-20T22:54:12.412 INFO:tasks.rados.rados.0.smithi084.stdout:3069: write (excl) oid 38 current snap is 347 2024-09-20T22:54:12.412 INFO:tasks.rados.rados.0.smithi084.stdout:3069: seq_num 1057 ranges {564999=511260,1853114=561358,3060221=592047} 2024-09-20T22:54:12.418 INFO:tasks.rados.rados.0.smithi084.stdout:3069: writing smithi08477203-38 from 564999 to 1076259 tid 1 2024-09-20T22:54:12.426 INFO:tasks.rados.rados.0.smithi084.stdout:3069: writing smithi08477203-38 from 1853114 to 2414472 tid 2 2024-09-20T22:54:12.433 INFO:tasks.rados.rados.0.smithi084.stdout:3069: writing smithi08477203-38 from 3060221 to 3652268 tid 3 2024-09-20T22:54:12.433 INFO:tasks.rados.rados.0.smithi084.stdout:3070: copy_from oid 40 from oid 30 current snap is 347 2024-09-20T22:54:12.433 INFO:tasks.rados.rados.0.smithi084.stdout:3067: finishing write tid 3 to smithi08477203-19 2024-09-20T22:54:12.433 INFO:tasks.rados.rados.0.smithi084.stdout:3067: finishing write tid 4 to smithi08477203-19 2024-09-20T22:54:12.433 INFO:tasks.rados.rados.0.smithi084.stdout:3067: finishing write tid 5 to smithi08477203-19 2024-09-20T22:54:12.433 INFO:tasks.rados.rados.0.smithi084.stdout:3067: oid 19 updating version 0 to 2487 2024-09-20T22:54:12.434 INFO:tasks.rados.rados.0.smithi084.stdout:3067: oid 19 updating version 2487 to 2490 2024-09-20T22:54:12.434 INFO:tasks.rados.rados.0.smithi084.stdout:3067: oid 19 version 2490 is already newer than 2488 2024-09-20T22:54:12.434 INFO:tasks.rados.rados.0.smithi084.stdout:3067: oid 19 version 2490 is already newer than 2489 2024-09-20T22:54:12.434 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 19 v 2490 (ObjNum 1055 snap 347 seq_num 1055) dirty exists 2024-09-20T22:54:12.434 INFO:tasks.rados.rados.0.smithi084.stdout:3067: left oid 19 (ObjNum 1055 snap 347 seq_num 1055) 2024-09-20T22:54:12.434 INFO:tasks.rados.rados.0.smithi084.stdout:3067: done (3 left) 2024-09-20T22:54:12.434 INFO:tasks.rados.rados.0.smithi084.stdout:3071: write oid 31 current snap is 347 2024-09-20T22:54:12.434 INFO:tasks.rados.rados.0.smithi084.stdout:3068: finishing write tid 1 to smithi08477203-26 2024-09-20T22:54:12.434 INFO:tasks.rados.rados.0.smithi084.stdout:3068: finishing write tid 2 to smithi08477203-26 2024-09-20T22:54:12.434 INFO:tasks.rados.rados.0.smithi084.stdout:3068: finishing write tid 3 to smithi08477203-26 2024-09-20T22:54:12.434 INFO:tasks.rados.rados.0.smithi084.stdout:3068: finishing write tid 4 to smithi08477203-26 2024-09-20T22:54:12.434 INFO:tasks.rados.rados.0.smithi084.stdout:3068: finishing write tid 5 to smithi08477203-26 2024-09-20T22:54:12.434 INFO:tasks.rados.rados.0.smithi084.stdout:3068: oid 26 updating version 0 to 1824 2024-09-20T22:54:12.434 INFO:tasks.rados.rados.0.smithi084.stdout:3068: oid 26 version 1824 is already newer than 1823 2024-09-20T22:54:12.435 INFO:tasks.rados.rados.0.smithi084.stdout:3068: oid 26 updating version 1824 to 1825 2024-09-20T22:54:12.435 INFO:tasks.rados.rados.0.smithi084.stdout:3068: oid 26 updating version 1825 to 1826 2024-09-20T22:54:12.435 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 26 v 1826 (ObjNum 1056 snap 347 seq_num 1056) dirty exists 2024-09-20T22:54:12.435 INFO:tasks.rados.rados.0.smithi084.stdout:3068: left oid 26 (ObjNum 1056 snap 347 seq_num 1056) 2024-09-20T22:54:12.435 INFO:tasks.rados.rados.0.smithi084.stdout:3069: finishing write tid 1 to smithi08477203-38 2024-09-20T22:54:12.435 INFO:tasks.rados.rados.0.smithi084.stdout:3069: finishing write tid 2 to smithi08477203-38 2024-09-20T22:54:12.435 INFO:tasks.rados.rados.0.smithi084.stdout:3071: seq_num 1058 ranges {581806=586509,1911596=421369,3087644=403319} 2024-09-20T22:54:12.440 INFO:tasks.rados.rados.0.smithi084.stdout:3071: writing smithi08477203-31 from 581806 to 1168315 tid 1 2024-09-20T22:54:12.446 INFO:tasks.rados.rados.0.smithi084.stdout:3071: writing smithi08477203-31 from 1911596 to 2332965 tid 2 2024-09-20T22:54:12.454 INFO:tasks.rados.rados.0.smithi084.stdout:3071: writing smithi08477203-31 from 3087644 to 3490963 tid 3 2024-09-20T22:54:12.454 INFO:tasks.rados.rados.0.smithi084.stdout:3068: done (3 left) 2024-09-20T22:54:12.454 INFO:tasks.rados.rados.0.smithi084.stdout:3072: read oid 6 snap -1 2024-09-20T22:54:12.454 INFO:tasks.rados.rados.0.smithi084.stdout:3072: expect deleted 2024-09-20T22:54:12.454 INFO:tasks.rados.rados.0.smithi084.stdout:3073: delete oid 24 current snap is 347 2024-09-20T22:54:12.454 INFO:tasks.rados.rados.0.smithi084.stdout:3069: finishing write tid 3 to smithi08477203-38 2024-09-20T22:54:12.454 INFO:tasks.rados.rados.0.smithi084.stdout:3069: finishing write tid 4 to smithi08477203-38 2024-09-20T22:54:12.454 INFO:tasks.rados.rados.0.smithi084.stdout:3069: finishing write tid 5 to smithi08477203-38 2024-09-20T22:54:12.454 INFO:tasks.rados.rados.0.smithi084.stdout:3069: oid 38 updating version 0 to 1830 2024-09-20T22:54:12.454 INFO:tasks.rados.rados.0.smithi084.stdout:3069: oid 38 version 1830 is already newer than 1829 2024-09-20T22:54:12.454 INFO:tasks.rados.rados.0.smithi084.stdout:3069: oid 38 version 1830 is already newer than 1828 2024-09-20T22:54:12.454 INFO:tasks.rados.rados.0.smithi084.stdout:3069: oid 38 version 1830 is already newer than 1827 2024-09-20T22:54:12.454 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 38 v 1830 (ObjNum 1057 snap 347 seq_num 1057) dirty exists 2024-09-20T22:54:12.455 INFO:tasks.rados.rados.0.smithi084.stdout:3069: left oid 38 (ObjNum 1057 snap 347 seq_num 1057) 2024-09-20T22:54:12.455 INFO:tasks.rados.rados.0.smithi084.stdout:3070: finishing copy_from racing read to smithi08477203-40 2024-09-20T22:54:12.455 INFO:tasks.rados.rados.0.smithi084.stdout:3071: finishing write tid 1 to smithi08477203-31 2024-09-20T22:54:12.455 INFO:tasks.rados.rados.0.smithi084.stdout:3071: finishing write tid 2 to smithi08477203-31 2024-09-20T22:54:12.455 INFO:tasks.rados.rados.0.smithi084.stdout:3070: finishing copy_from to smithi08477203-40 2024-09-20T22:54:12.455 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 40 v 1852 (ObjNum 787 snap 254 seq_num 787) dirty exists 2024-09-20T22:54:12.455 INFO:tasks.rados.rados.0.smithi084.stdout:3069: done (4 left) 2024-09-20T22:54:12.456 INFO:tasks.rados.rados.0.smithi084.stdout:3070: done (3 left) 2024-09-20T22:54:12.456 INFO:tasks.rados.rados.0.smithi084.stdout:3072: done (2 left) 2024-09-20T22:54:12.456 INFO:tasks.rados.rados.0.smithi084.stdout:3073: done (1 left) 2024-09-20T22:54:12.456 INFO:tasks.rados.rados.0.smithi084.stdout:3074: copy_from oid 23 from oid 33 current snap is 347 2024-09-20T22:54:12.456 INFO:tasks.rados.rados.0.smithi084.stdout:3075: copy_from oid 44 from oid 29 current snap is 347 2024-09-20T22:54:12.456 INFO:tasks.rados.rados.0.smithi084.stdout:3076: snap_create 2024-09-20T22:54:12.457 INFO:tasks.rados.rados.0.smithi084.stdout:3071: finishing write tid 3 to smithi08477203-31 2024-09-20T22:54:12.457 INFO:tasks.rados.rados.0.smithi084.stdout:3075: finishing copy_from to smithi08477203-44 2024-09-20T22:54:12.457 INFO:tasks.rados.rados.0.smithi084.stdout:3075: got expected ENOENT (src dne) 2024-09-20T22:54:12.457 INFO:tasks.rados.rados.0.smithi084.stdout:3071: finishing write tid 4 to smithi08477203-31 2024-09-20T22:54:12.457 INFO:tasks.rados.rados.0.smithi084.stdout:3075: finishing copy_from racing read to smithi08477203-44 2024-09-20T22:54:12.457 INFO:tasks.rados.rados.0.smithi084.stdout:3071: finishing write tid 5 to smithi08477203-31 2024-09-20T22:54:12.457 INFO:tasks.rados.rados.0.smithi084.stdout:3071: oid 31 updating version 0 to 1833 2024-09-20T22:54:12.457 INFO:tasks.rados.rados.0.smithi084.stdout:3071: oid 31 version 1833 is already newer than 1831 2024-09-20T22:54:12.457 INFO:tasks.rados.rados.0.smithi084.stdout:3071: oid 31 updating version 1833 to 1834 2024-09-20T22:54:12.457 INFO:tasks.rados.rados.0.smithi084.stdout:3071: oid 31 version 1834 is already newer than 1832 2024-09-20T22:54:12.457 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 31 v 1834 (ObjNum 1058 snap 347 seq_num 1058) dirty exists 2024-09-20T22:54:12.457 INFO:tasks.rados.rados.0.smithi084.stdout:3071: left oid 31 (ObjNum 1058 snap 347 seq_num 1058) 2024-09-20T22:54:12.463 INFO:tasks.rados.rados.0.smithi084.stdout:3074: finishing copy_from racing read to smithi08477203-23 2024-09-20T22:54:12.467 INFO:tasks.rados.rados.0.smithi084.stdout:3074: finishing copy_from to smithi08477203-23 2024-09-20T22:54:12.467 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 23 v 1829 (ObjNum 984 snap 319 seq_num 984) dirty exists 2024-09-20T22:54:12.496 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:54:12.518 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:12.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:12.519 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:54:12.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:12 smithi084 ceph-mon[29774]: osdmap e771: 8 total, 8 up, 7 in 2024-09-20T22:54:12.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:12 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/2150253749' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:54:12.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:12 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/3459485685' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:54:12.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:12 smithi084 ceph-mon[40024]: osdmap e771: 8 total, 8 up, 7 in 2024-09-20T22:54:12.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:12 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/2150253749' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:54:12.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:12 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/3459485685' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:54:12.620 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:12 smithi154 ceph-mon[33264]: osdmap e771: 8 total, 8 up, 7 in 2024-09-20T22:54:12.620 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:12 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/2150253749' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:54:12.620 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:12 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/3459485685' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:54:12.620 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:54:12.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:12.643 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:12.643 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:54:12.728 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-20T22:54:12.728 INFO:teuthology.orchestra.run.smithi154.stdout: "size": 20, 2024-09-20T22:54:12.728 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 600, 2024-09-20T22:54:12.728 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [ 2024-09-20T22:54:12.728 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.729 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:1525 7.c 7:3f865ef5:::smithi08477203-28:head [copy-from ver 127] snapc 33=[33] ondisk+write+known_if_redirected+supports_pool_eio e197)", 2024-09-20T22:54:12.729 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:44:36.198921+0000", 2024-09-20T22:54:12.729 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 576.52239948500005, 2024-09-20T22:54:12.729 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.028621220999999999, 2024-09-20T22:54:12.729 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:54:12.729 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:54:12.729 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:54:12.730 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:54:12.730 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:54:12.730 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:54:12.730 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 1525 2024-09-20T22:54:12.730 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.730 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:54:12.730 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.730 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:54:12.730 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.198921+0000", 2024-09-20T22:54:12.730 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:54:12.730 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.730 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.731 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:54:12.731 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.198921+0000", 2024-09-20T22:54:12.731 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:54:12.731 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.731 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.731 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:54:12.731 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.198923+0000", 2024-09-20T22:54:12.731 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.1090000000000001e-06 2024-09-20T22:54:12.731 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.731 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.731 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:54:12.732 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.198934+0000", 2024-09-20T22:54:12.732 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0837e-05 2024-09-20T22:54:12.732 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.732 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.732 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:54:12.732 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.198955+0000", 2024-09-20T22:54:12.732 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.0789e-05 2024-09-20T22:54:12.732 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.732 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.732 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:54:12.732 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.198966+0000", 2024-09-20T22:54:12.732 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1778e-05 2024-09-20T22:54:12.733 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.733 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.733 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:54:12.733 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.199046+0000", 2024-09-20T22:54:12.733 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.9567999999999994e-05 2024-09-20T22:54:12.733 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.733 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.733 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op must wait for map", 2024-09-20T22:54:12.733 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.199054+0000", 2024-09-20T22:54:12.733 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.9349999999999994e-06 2024-09-20T22:54:12.733 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.733 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.733 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:54:12.734 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.201428+0000", 2024-09-20T22:54:12.734 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0023744389999999999 2024-09-20T22:54:12.734 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.734 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.734 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:54:12.734 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.201527+0000", 2024-09-20T22:54:12.734 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.8554000000000005e-05 2024-09-20T22:54:12.734 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.734 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.734 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 3,7", 2024-09-20T22:54:12.734 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.215277+0000", 2024-09-20T22:54:12.734 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.013750326 2024-09-20T22:54:12.735 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.735 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.735 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:54:12.735 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.226629+0000", 2024-09-20T22:54:12.735 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.011351781999999999 2024-09-20T22:54:12.735 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.735 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.735 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:54:12.735 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.227366+0000", 2024-09-20T22:54:12.735 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00073709199999999996 2024-09-20T22:54:12.735 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.736 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.736 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:54:12.736 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.227450+0000", 2024-09-20T22:54:12.736 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.3902999999999999e-05 2024-09-20T22:54:12.736 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.736 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.736 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:54:12.736 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.227489+0000", 2024-09-20T22:54:12.736 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.9103000000000002e-05 2024-09-20T22:54:12.736 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.736 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.737 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:54:12.737 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:36.227542+0000", 2024-09-20T22:54:12.737 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.3006e-05 2024-09-20T22:54:12.737 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:54:12.737 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:54:12.737 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:54:12.737 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.737 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.737 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:1652 7.c 7:31d0211f:::smithi08477203-25:head [copy-from ver 161] snapc 38=[38,37] ondisk+write+known_if_redirected+supports_pool_eio e206)", 2024-09-20T22:54:12.737 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:44:45.159981+0000", 2024-09-20T22:54:12.737 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 567.56133908499999, 2024-09-20T22:54:12.738 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.029481621, 2024-09-20T22:54:12.738 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:54:12.738 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:54:12.738 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:54:12.738 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:54:12.738 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:54:12.738 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:54:12.738 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 1652 2024-09-20T22:54:12.738 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.738 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:54:12.738 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.738 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:54:12.739 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:45.159981+0000", 2024-09-20T22:54:12.739 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:54:12.739 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.739 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.739 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:54:12.739 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:45.159981+0000", 2024-09-20T22:54:12.739 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:54:12.739 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.739 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.739 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:54:12.739 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:45.159982+0000", 2024-09-20T22:54:12.739 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.8999999999999995e-07 2024-09-20T22:54:12.740 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.740 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.740 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:54:12.740 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:45.159986+0000", 2024-09-20T22:54:12.740 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.732e-06 2024-09-20T22:54:12.740 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.740 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.740 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:54:12.740 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:45.159996+0000", 2024-09-20T22:54:12.740 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.9639999999999998e-06 2024-09-20T22:54:12.740 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.741 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.741 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:54:12.741 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:45.160001+0000", 2024-09-20T22:54:12.741 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.8010000000000003e-06 2024-09-20T22:54:12.741 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.741 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.741 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:54:12.741 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:45.165664+0000", 2024-09-20T22:54:12.741 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0056630869999999998 2024-09-20T22:54:12.741 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.741 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.741 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:54:12.741 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:45.165710+0000", 2024-09-20T22:54:12.742 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.6371e-05 2024-09-20T22:54:12.742 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.742 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.742 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 3,7", 2024-09-20T22:54:12.742 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:45.175729+0000", 2024-09-20T22:54:12.742 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.010018750999999999 2024-09-20T22:54:12.742 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.742 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.742 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:54:12.742 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:45.186279+0000", 2024-09-20T22:54:12.742 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.010550241 2024-09-20T22:54:12.743 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.743 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.743 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:54:12.743 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:45.186520+0000", 2024-09-20T22:54:12.743 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000241394 2024-09-20T22:54:12.743 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.743 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.743 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:54:12.743 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:45.189408+0000", 2024-09-20T22:54:12.743 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0028872849999999999 2024-09-20T22:54:12.743 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.743 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.744 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:54:12.744 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:45.189440+0000", 2024-09-20T22:54:12.744 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.2341000000000001e-05 2024-09-20T22:54:12.744 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.744 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.744 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:54:12.744 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:44:45.189463+0000", 2024-09-20T22:54:12.744 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.2764000000000001e-05 2024-09-20T22:54:12.744 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:54:12.744 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:54:12.744 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:54:12.744 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.744 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.745 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:3132 7.19 7:9eac4560:::smithi08477203-30:head [copy-from ver 389] snapc 77=[77,76,75,74,73,70,6b] ondisk+write+known_if_redirected+supports_pool_eio e328)", 2024-09-20T22:54:12.745 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:46:48.209876+0000", 2024-09-20T22:54:12.745 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 444.51144434499997, 2024-09-20T22:54:12.745 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.030505467000000001, 2024-09-20T22:54:12.745 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:54:12.745 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:54:12.745 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:54:12.745 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:54:12.745 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:54:12.745 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:54:12.746 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 3132 2024-09-20T22:54:12.746 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.746 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:54:12.746 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.746 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:54:12.746 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.209876+0000", 2024-09-20T22:54:12.746 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:54:12.746 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.746 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.746 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:54:12.746 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.209876+0000", 2024-09-20T22:54:12.747 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:54:12.747 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.747 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.747 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:54:12.747 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.209876+0000", 2024-09-20T22:54:12.747 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.7e-07 2024-09-20T22:54:12.747 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.747 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.747 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:54:12.747 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.209880+0000", 2024-09-20T22:54:12.747 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.4149999999999999e-06 2024-09-20T22:54:12.748 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.748 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.748 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:54:12.748 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.209886+0000", 2024-09-20T22:54:12.748 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.5819999999999998e-06 2024-09-20T22:54:12.748 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.748 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.748 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:54:12.748 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.209893+0000", 2024-09-20T22:54:12.748 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.1820000000000003e-06 2024-09-20T22:54:12.749 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.749 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.749 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:54:12.749 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.209945+0000", 2024-09-20T22:54:12.749 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.2528999999999998e-05 2024-09-20T22:54:12.749 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.749 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.749 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:54:12.749 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.210028+0000", 2024-09-20T22:54:12.749 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.3067999999999998e-05 2024-09-20T22:54:12.749 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.749 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.749 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 2,7", 2024-09-20T22:54:12.750 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.223189+0000", 2024-09-20T22:54:12.750 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.013160579 2024-09-20T22:54:12.750 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.750 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.750 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:54:12.750 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.232498+0000", 2024-09-20T22:54:12.750 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0093088760000000007 2024-09-20T22:54:12.750 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.750 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.750 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:54:12.750 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.240058+0000", 2024-09-20T22:54:12.750 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0075599930000000001 2024-09-20T22:54:12.750 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.751 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.751 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:54:12.751 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.240300+0000", 2024-09-20T22:54:12.751 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00024190999999999999 2024-09-20T22:54:12.751 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.751 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.751 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:54:12.751 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.240343+0000", 2024-09-20T22:54:12.751 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.2972000000000001e-05 2024-09-20T22:54:12.751 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.751 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.751 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:54:12.752 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.240381+0000", 2024-09-20T22:54:12.752 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.8890999999999997e-05 2024-09-20T22:54:12.752 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:54:12.752 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:54:12.752 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:54:12.752 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.752 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.752 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:3747 7.c 7:3f865ef5:::smithi08477203-28:head [copy-from ver 108] snapc 98=[98,97,96,95,94,93,92,8e,8d,8c,8b,8a,88,87,86,80,7f,73] ondisk+write+known_if_redirected+supports_pool_eio e383)", 2024-09-20T22:54:12.752 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:47:43.253429+0000", 2024-09-20T22:54:12.752 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 389.46789119300001, 2024-09-20T22:54:12.752 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.028706249999999999, 2024-09-20T22:54:12.752 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:54:12.753 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:54:12.753 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:54:12.753 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:54:12.753 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:54:12.753 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:54:12.753 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 3747 2024-09-20T22:54:12.753 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.753 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:54:12.753 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.753 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:54:12.753 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.253429+0000", 2024-09-20T22:54:12.753 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:54:12.753 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.754 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.754 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:54:12.754 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.253429+0000", 2024-09-20T22:54:12.754 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:54:12.754 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.754 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.754 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:54:12.754 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.253431+0000", 2024-09-20T22:54:12.754 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.829e-06 2024-09-20T22:54:12.754 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.754 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.754 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:54:12.754 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.253438+0000", 2024-09-20T22:54:12.755 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.418e-06 2024-09-20T22:54:12.755 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.755 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.755 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:54:12.755 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.253451+0000", 2024-09-20T22:54:12.755 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2317000000000001e-05 2024-09-20T22:54:12.755 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.755 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.755 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:54:12.755 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.253457+0000", 2024-09-20T22:54:12.755 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.4710000000000004e-06 2024-09-20T22:54:12.755 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.755 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.755 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:54:12.756 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.253530+0000", 2024-09-20T22:54:12.756 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.3047999999999996e-05 2024-09-20T22:54:12.756 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.756 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.756 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op must wait for map", 2024-09-20T22:54:12.756 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.253536+0000", 2024-09-20T22:54:12.756 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.5140000000000001e-06 2024-09-20T22:54:12.756 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.756 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.756 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:54:12.756 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.257582+0000", 2024-09-20T22:54:12.756 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0040464849999999998 2024-09-20T22:54:12.757 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.757 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.757 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:54:12.757 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.257697+0000", 2024-09-20T22:54:12.757 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00011471 2024-09-20T22:54:12.757 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.757 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.757 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 3,7", 2024-09-20T22:54:12.757 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.267067+0000", 2024-09-20T22:54:12.757 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0093696560000000005 2024-09-20T22:54:12.757 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.757 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.757 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:54:12.758 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.276328+0000", 2024-09-20T22:54:12.758 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0092619669999999994 2024-09-20T22:54:12.758 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.758 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.758 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:54:12.758 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.278462+0000", 2024-09-20T22:54:12.758 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0021334430000000001 2024-09-20T22:54:12.758 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.758 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.758 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:54:12.759 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.282036+0000", 2024-09-20T22:54:12.759 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0035745249999999998 2024-09-20T22:54:12.759 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.759 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.759 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:54:12.759 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.282094+0000", 2024-09-20T22:54:12.759 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.7312999999999998e-05 2024-09-20T22:54:12.759 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.759 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.759 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:54:12.759 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.282135+0000", 2024-09-20T22:54:12.759 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.1554000000000002e-05 2024-09-20T22:54:12.759 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:54:12.760 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:54:12.760 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:54:12.760 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.760 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.760 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:3856 7.12 7:4fa4b2b3:::smithi08477203-21:head [copy-from ver 170] snapc 9e=[9e,9d,9c,9b,9a,98,96,95,94,93,92,8e,8d,8c,8b,8a,88,87,86,80,7f,73] ondisk+write+known_if_redirected+supports_pool_eio e391)", 2024-09-20T22:54:12.760 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:47:51.204384+0000", 2024-09-20T22:54:12.760 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 381.516935875, 2024-09-20T22:54:12.760 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0068049349999999, 2024-09-20T22:54:12.760 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:54:12.760 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:54:12.760 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:54:12.761 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:54:12.761 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:54:12.761 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:54:12.761 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 3856 2024-09-20T22:54:12.761 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.761 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:54:12.761 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.761 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:54:12.761 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:51.204384+0000", 2024-09-20T22:54:12.762 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:54:12.762 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.762 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.762 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:54:12.762 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:51.204384+0000", 2024-09-20T22:54:12.762 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:54:12.762 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.762 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.762 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:54:12.762 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:51.204386+0000", 2024-09-20T22:54:12.762 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.57e-06 2024-09-20T22:54:12.762 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.763 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.763 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:54:12.763 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:51.204391+0000", 2024-09-20T22:54:12.763 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.6009999999999997e-06 2024-09-20T22:54:12.763 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.763 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.763 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:54:12.763 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:51.204400+0000", 2024-09-20T22:54:12.763 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.0499999999999997e-06 2024-09-20T22:54:12.763 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.763 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.763 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:54:12.764 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:51.204410+0000", 2024-09-20T22:54:12.764 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.01e-05 2024-09-20T22:54:12.764 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.764 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.764 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:54:12.764 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:51.204944+0000", 2024-09-20T22:54:12.764 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00053440599999999997 2024-09-20T22:54:12.764 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.764 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.764 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-20T22:54:12.764 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:51.204954+0000", 2024-09-20T22:54:12.765 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0162e-05 2024-09-20T22:54:12.765 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.765 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.765 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:54:12.765 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:52.210122+0000", 2024-09-20T22:54:12.765 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0051679019999999 2024-09-20T22:54:12.765 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.765 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.765 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:54:12.765 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:52.210316+0000", 2024-09-20T22:54:12.765 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000193704 2024-09-20T22:54:12.766 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.766 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.766 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:54:12.766 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:52.211189+0000", 2024-09-20T22:54:12.766 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00087343999999999996 2024-09-20T22:54:12.766 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:54:12.766 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:54:12.766 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:54:12.766 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.766 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.766 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:3857 7.12 7:4fa4b2b3:::smithi08477203-21:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e391)", 2024-09-20T22:54:12.766 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:47:51.204423+0000", 2024-09-20T22:54:12.767 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 381.51689743499998, 2024-09-20T22:54:12.767 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0069932960000001, 2024-09-20T22:54:12.767 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:54:12.767 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:54:12.767 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:54:12.767 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:54:12.767 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:54:12.767 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:54:12.767 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 3857 2024-09-20T22:54:12.767 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.767 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:54:12.767 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.767 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:54:12.768 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:51.204423+0000", 2024-09-20T22:54:12.768 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:54:12.768 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.768 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.768 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:54:12.768 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:51.204423+0000", 2024-09-20T22:54:12.768 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:54:12.768 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.768 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.768 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:54:12.768 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:51.204423+0000", 2024-09-20T22:54:12.768 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.3099999999999998e-07 2024-09-20T22:54:12.769 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.769 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.769 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:54:12.769 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:51.204426+0000", 2024-09-20T22:54:12.769 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.4030000000000001e-06 2024-09-20T22:54:12.769 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.769 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.769 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:54:12.769 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:51.204432+0000", 2024-09-20T22:54:12.769 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.4099999999999996e-06 2024-09-20T22:54:12.769 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.769 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.769 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:54:12.770 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:51.204438+0000", 2024-09-20T22:54:12.770 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.4600000000000002e-06 2024-09-20T22:54:12.770 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.770 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.770 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:54:12.770 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:51.204980+0000", 2024-09-20T22:54:12.770 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00054220600000000005 2024-09-20T22:54:12.770 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.770 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.770 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-20T22:54:12.770 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:51.204987+0000", 2024-09-20T22:54:12.770 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.3320000000000003e-06 2024-09-20T22:54:12.771 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.771 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.771 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:54:12.771 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:52.211064+0000", 2024-09-20T22:54:12.771 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0060768899999999 2024-09-20T22:54:12.771 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.771 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.771 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for blocked object", 2024-09-20T22:54:12.771 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:52.211095+0000", 2024-09-20T22:54:12.771 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.1399999999999998e-05 2024-09-20T22:54:12.771 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.771 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.772 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:54:12.772 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:52.211260+0000", 2024-09-20T22:54:12.772 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00016436099999999999 2024-09-20T22:54:12.772 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.772 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.772 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:54:12.772 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:52.211310+0000", 2024-09-20T22:54:12.772 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.0417999999999997e-05 2024-09-20T22:54:12.772 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.772 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.772 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:54:12.773 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:52.211416+0000", 2024-09-20T22:54:12.773 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00010588499999999999 2024-09-20T22:54:12.773 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:54:12.773 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:54:12.773 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:54:12.773 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.773 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.773 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "MOSDPGPush(7.17 434/433 [PushOp(7:ed54065f:::smithi08477203-12:90, version: 429'392, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:ed54065f:::smithi08477203-12:90@429'392, size: 2033746, copy_subset: [(0, 2033746)], clone_subset: {7:ed54065f:::smithi08477203-12:88: [], 7:ed54065f:::smithi08477203-12:a5: []}, snapset: b1=[]:{86: [86], 88: [87], 90: [8e], 9b: [9a], a5: [a5, 9f, 9e, 9d, 9c], b1: [b1, b0, af, ae, ad, ab, a9, a7, a6]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-20T22:54:12.773 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:48:34.294270+0000", 2024-09-20T22:54:12.773 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 338.42705030500002, 2024-09-20T22:54:12.773 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.044145694999999999, 2024-09-20T22:54:12.773 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:54:12.773 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:54:12.774 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:54:12.774 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:54:12.774 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.774 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:54:12.774 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:34.294270+0000", 2024-09-20T22:54:12.774 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:54:12.774 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.774 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.774 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:54:12.774 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:34.294270+0000", 2024-09-20T22:54:12.774 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:54:12.774 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.775 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.775 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:54:12.775 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:34.294272+0000", 2024-09-20T22:54:12.775 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.576e-06 2024-09-20T22:54:12.775 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.775 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.775 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:54:12.775 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:34.337875+0000", 2024-09-20T22:54:12.775 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.043603555000000002 2024-09-20T22:54:12.775 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.775 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.775 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:54:12.776 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:54:12.777 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:34.337895+0000", 2024-09-20T22:54:12.777 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.0075999999999999e-05 2024-09-20T22:54:12.777 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.777 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.778 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:54:12.778 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:34.337909+0000", 2024-09-20T22:54:12.778 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4070000000000001e-05 2024-09-20T22:54:12.778 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.778 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.778 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:54:12.778 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:34.338016+0000", 2024-09-20T22:54:12.778 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000106969 2024-09-20T22:54:12.778 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.778 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.778 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:54:12.778 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:34.338037+0000", 2024-09-20T22:54:12.779 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.0409999999999999e-05 2024-09-20T22:54:12.779 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.779 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.779 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:54:12.779 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:34.338416+0000", 2024-09-20T22:54:12.779 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00037903900000000001 2024-09-20T22:54:12.779 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:54:12.779 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:54:12.779 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:54:12.779 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.779 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.779 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:4373 7.7 7:ea720b85:::smithi08477203-26:head [rollback af] snapc b5=[b5,b4,b3,b2,b1,b0,af,ae,ab,a9,a7,a6,a5,9f,9e,9c,9a,96,95,87,86] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e442)", 2024-09-20T22:54:12.780 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:48:42.215127+0000", 2024-09-20T22:54:12.780 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 330.50619367500002, 2024-09-20T22:54:12.780 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0149912029999999, 2024-09-20T22:54:12.780 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:54:12.780 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:54:12.780 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:54:12.780 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:54:12.780 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:54:12.780 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:54:12.780 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 4373 2024-09-20T22:54:12.780 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.780 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:54:12.781 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.781 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:54:12.781 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:42.215127+0000", 2024-09-20T22:54:12.781 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:54:12.781 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.781 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.781 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:54:12.781 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:42.215127+0000", 2024-09-20T22:54:12.781 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:54:12.781 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.781 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.781 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:54:12.781 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:42.215128+0000", 2024-09-20T22:54:12.782 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.291e-06 2024-09-20T22:54:12.782 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.782 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.782 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:54:12.782 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:42.215133+0000", 2024-09-20T22:54:12.782 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.5350000000000004e-06 2024-09-20T22:54:12.782 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.782 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.782 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:54:12.782 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:42.215146+0000", 2024-09-20T22:54:12.782 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2099999999999999e-05 2024-09-20T22:54:12.783 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.783 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.783 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:54:12.783 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:42.215154+0000", 2024-09-20T22:54:12.783 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.9190000000000005e-06 2024-09-20T22:54:12.783 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.783 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.783 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:54:12.783 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:42.215229+0000", 2024-09-20T22:54:12.783 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.4233000000000006e-05 2024-09-20T22:54:12.783 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.783 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.783 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op must wait for map", 2024-09-20T22:54:12.784 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:42.215238+0000", 2024-09-20T22:54:12.784 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.5950000000000005e-06 2024-09-20T22:54:12.784 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.784 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.784 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:54:12.784 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:42.227011+0000", 2024-09-20T22:54:12.784 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.011772553999999999 2024-09-20T22:54:12.784 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.784 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.784 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-20T22:54:12.784 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:42.227026+0000", 2024-09-20T22:54:12.784 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.5119999999999999e-05 2024-09-20T22:54:12.785 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.785 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.785 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:54:12.785 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:43.224744+0000", 2024-09-20T22:54:12.785 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.99771833399999998 2024-09-20T22:54:12.785 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.785 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.785 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:54:12.785 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:43.224967+0000", 2024-09-20T22:54:12.785 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00022256000000000001 2024-09-20T22:54:12.785 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.786 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.786 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:54:12.786 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:43.230118+0000", 2024-09-20T22:54:12.786 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0051509620000000002 2024-09-20T22:54:12.786 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:54:12.786 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:54:12.786 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:54:12.786 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.786 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.786 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:4638 7.d 7:b30e4612:::smithi08477203-42:head [copy-from ver 560] snapc c2=[c2,c0,bf,be,bc,bb,ba,b9,b8,b7,b4,b2,b1,a7,a5,9e,9c,95,87,86] ondisk+write+known_if_redirected+supports_pool_eio e468)", 2024-09-20T22:54:12.786 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:49:08.237769+0000", 2024-09-20T22:54:12.786 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 304.483551218, 2024-09-20T22:54:12.787 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.035118309, 2024-09-20T22:54:12.787 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:54:12.787 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:54:12.787 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:54:12.787 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:54:12.787 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:54:12.787 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:54:12.787 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 4638 2024-09-20T22:54:12.787 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.787 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:54:12.787 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.787 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:54:12.788 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:08.237769+0000", 2024-09-20T22:54:12.788 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:54:12.788 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.788 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.788 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:54:12.788 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:08.237769+0000", 2024-09-20T22:54:12.788 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:54:12.788 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.788 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.788 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:54:12.788 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:08.237772+0000", 2024-09-20T22:54:12.788 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.8629999999999999e-06 2024-09-20T22:54:12.788 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.789 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.789 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:54:12.789 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:08.237785+0000", 2024-09-20T22:54:12.789 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3039e-05 2024-09-20T22:54:12.789 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.789 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.789 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:54:12.789 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:08.237809+0000", 2024-09-20T22:54:12.789 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.4094e-05 2024-09-20T22:54:12.789 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.789 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.789 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:54:12.790 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:08.237828+0000", 2024-09-20T22:54:12.790 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.8839e-05 2024-09-20T22:54:12.790 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.790 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.790 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:54:12.790 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:08.237921+0000", 2024-09-20T22:54:12.790 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.2727999999999997e-05 2024-09-20T22:54:12.790 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.790 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.790 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op must wait for map", 2024-09-20T22:54:12.790 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:08.237929+0000", 2024-09-20T22:54:12.790 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.5609999999999999e-06 2024-09-20T22:54:12.790 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.791 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.791 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:54:12.791 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:08.245301+0000", 2024-09-20T22:54:12.791 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0073717670000000004 2024-09-20T22:54:12.791 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.791 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.791 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:54:12.791 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:08.245405+0000", 2024-09-20T22:54:12.791 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00010370999999999999 2024-09-20T22:54:12.791 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.791 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.792 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 0,1", 2024-09-20T22:54:12.792 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:08.256435+0000", 2024-09-20T22:54:12.792 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.011030659 2024-09-20T22:54:12.792 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.792 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.792 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:54:12.792 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:08.262369+0000", 2024-09-20T22:54:12.792 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0059332029999999997 2024-09-20T22:54:12.792 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.792 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.792 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:54:12.792 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:08.272100+0000", 2024-09-20T22:54:12.792 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0097312560000000006 2024-09-20T22:54:12.793 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.793 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.793 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:54:12.793 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:08.272791+0000", 2024-09-20T22:54:12.793 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00069164699999999999 2024-09-20T22:54:12.793 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.793 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.793 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:54:12.793 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:08.272858+0000", 2024-09-20T22:54:12.793 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.6532999999999995e-05 2024-09-20T22:54:12.793 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.793 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.794 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:54:12.794 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:08.272887+0000", 2024-09-20T22:54:12.794 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.9410000000000001e-05 2024-09-20T22:54:12.794 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:54:12.794 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:54:12.794 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:54:12.794 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.794 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.794 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:6018 7.4 e606/602 7:24b1a612:::smithi08477203-43:head v 606'331, mlcod=603'327)", 2024-09-20T22:54:12.794 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:51:26.433797+0000", 2024-09-20T22:54:12.794 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 166.28752328600001, 2024-09-20T22:54:12.795 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.029417520999999999, 2024-09-20T22:54:12.795 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:54:12.795 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:54:12.795 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:54:12.795 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:54:12.795 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.795 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:54:12.795 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:26.433797+0000", 2024-09-20T22:54:12.795 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:54:12.795 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.795 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.795 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:54:12.796 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:26.433797+0000", 2024-09-20T22:54:12.796 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:54:12.796 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.796 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.796 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:54:12.796 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:26.433797+0000", 2024-09-20T22:54:12.796 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.29e-07 2024-09-20T22:54:12.796 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.796 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.796 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:54:12.796 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:26.460482+0000", 2024-09-20T22:54:12.796 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.026684802000000001 2024-09-20T22:54:12.797 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.797 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.797 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:54:12.797 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:26.460502+0000", 2024-09-20T22:54:12.797 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9828999999999999e-05 2024-09-20T22:54:12.797 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.797 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.797 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:54:12.797 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:26.460528+0000", 2024-09-20T22:54:12.797 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.5709999999999999e-05 2024-09-20T22:54:12.797 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.797 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.798 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:54:12.798 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:26.460592+0000", 2024-09-20T22:54:12.798 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.4706e-05 2024-09-20T22:54:12.798 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.798 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.798 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:54:12.798 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:26.460615+0000", 2024-09-20T22:54:12.798 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.2297e-05 2024-09-20T22:54:12.798 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.798 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.799 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:54:12.799 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:26.463172+0000", 2024-09-20T22:54:12.799 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.002557541 2024-09-20T22:54:12.799 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.799 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.799 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:54:12.799 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:26.463215+0000", 2024-09-20T22:54:12.799 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.2407000000000002e-05 2024-09-20T22:54:12.799 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:54:12.799 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:54:12.799 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:54:12.799 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.799 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.800 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:6356 7.c 7:3f0fe232:::smithi08477203-4:head [copy-from ver 1803] snapc 119=[119,118,117,116,115,114,113,112,111,110,10e,10c,107,105,104,103,fd,fb,fa,f8,f5,f2,ed,d7,c2,bf] ondisk+write+known_if_redirected+supports_pool_eio e638)", 2024-09-20T22:54:12.800 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:51:58.274309+0000", 2024-09-20T22:54:12.800 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 134.44701090999999, 2024-09-20T22:54:12.800 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.038014514999999999, 2024-09-20T22:54:12.800 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:54:12.800 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:54:12.800 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:54:12.800 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:54:12.800 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:54:12.800 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:54:12.800 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 6356 2024-09-20T22:54:12.800 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.801 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:54:12.801 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.801 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:54:12.801 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:58.274309+0000", 2024-09-20T22:54:12.801 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:54:12.801 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.801 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.801 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:54:12.801 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:58.274309+0000", 2024-09-20T22:54:12.801 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:54:12.801 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.801 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.801 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:54:12.802 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:58.274312+0000", 2024-09-20T22:54:12.802 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.9050000000000001e-06 2024-09-20T22:54:12.802 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.802 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.802 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:54:12.802 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:58.274324+0000", 2024-09-20T22:54:12.802 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1338999999999999e-05 2024-09-20T22:54:12.802 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.802 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.802 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:54:12.802 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:58.274348+0000", 2024-09-20T22:54:12.803 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.4876999999999999e-05 2024-09-20T22:54:12.803 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.803 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.803 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:54:12.803 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:58.274360+0000", 2024-09-20T22:54:12.803 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1301000000000001e-05 2024-09-20T22:54:12.803 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.803 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.803 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:54:12.803 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:58.274489+0000", 2024-09-20T22:54:12.803 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00012894299999999999 2024-09-20T22:54:12.804 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.804 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.804 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op must wait for map", 2024-09-20T22:54:12.804 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:58.274496+0000", 2024-09-20T22:54:12.804 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.401e-06 2024-09-20T22:54:12.804 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.804 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.804 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:54:12.804 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:58.277772+0000", 2024-09-20T22:54:12.804 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0032760409999999999 2024-09-20T22:54:12.804 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.804 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.805 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:54:12.805 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:58.277882+0000", 2024-09-20T22:54:12.805 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000109598 2024-09-20T22:54:12.805 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.805 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.805 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 3,7", 2024-09-20T22:54:12.805 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:58.302919+0000", 2024-09-20T22:54:12.805 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.025037197000000001 2024-09-20T22:54:12.805 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.805 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.805 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:54:12.805 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:58.311526+0000", 2024-09-20T22:54:12.805 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0086071370000000008 2024-09-20T22:54:12.806 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.806 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.806 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:54:12.806 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:58.311824+0000", 2024-09-20T22:54:12.806 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00029813600000000001 2024-09-20T22:54:12.806 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.806 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.806 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:54:12.806 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:58.312234+0000", 2024-09-20T22:54:12.806 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00040939699999999998 2024-09-20T22:54:12.806 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.806 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.807 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:54:12.807 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:58.312270+0000", 2024-09-20T22:54:12.807 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.6797e-05 2024-09-20T22:54:12.807 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.807 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.807 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:54:12.807 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:58.312324+0000", 2024-09-20T22:54:12.807 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.3446000000000003e-05 2024-09-20T22:54:12.807 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:54:12.807 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:54:12.807 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:54:12.807 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.807 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.808 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:6357 7.c 7:3f0fe232:::smithi08477203-4:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e638)", 2024-09-20T22:54:12.808 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:51:58.274400+0000", 2024-09-20T22:54:12.808 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 134.44692013700001, 2024-09-20T22:54:12.808 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.033818943999999997, 2024-09-20T22:54:12.808 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:54:12.808 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:54:12.808 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:54:12.808 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:54:12.808 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:54:12.808 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:54:12.808 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 6357 2024-09-20T22:54:12.808 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.808 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:54:12.809 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.809 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:54:12.809 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:58.274400+0000", 2024-09-20T22:54:12.809 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:54:12.809 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.809 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.809 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:54:12.809 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:58.274400+0000", 2024-09-20T22:54:12.809 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:54:12.809 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.809 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.809 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:54:12.809 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:58.274401+0000", 2024-09-20T22:54:12.810 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.4799999999999997e-07 2024-09-20T22:54:12.810 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.810 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.810 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:54:12.810 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:58.274405+0000", 2024-09-20T22:54:12.810 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.8310000000000003e-06 2024-09-20T22:54:12.810 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.810 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.810 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:54:12.810 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:58.274414+0000", 2024-09-20T22:54:12.810 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.0790000000000007e-06 2024-09-20T22:54:12.810 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.811 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.811 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:54:12.811 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:58.274420+0000", 2024-09-20T22:54:12.811 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.6059999999999999e-06 2024-09-20T22:54:12.811 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.811 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.811 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:54:12.811 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:58.274557+0000", 2024-09-20T22:54:12.811 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00013666700000000001 2024-09-20T22:54:12.811 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.811 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.812 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting_for_map not empty", 2024-09-20T22:54:12.812 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:58.274562+0000", 2024-09-20T22:54:12.812 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.4079999999999997e-06 2024-09-20T22:54:12.812 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.812 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.812 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:54:12.812 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:58.278047+0000", 2024-09-20T22:54:12.812 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.003484647 2024-09-20T22:54:12.812 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.812 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.812 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for blocked object", 2024-09-20T22:54:12.812 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:58.278096+0000", 2024-09-20T22:54:12.813 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.9298000000000001e-05 2024-09-20T22:54:12.813 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.813 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.813 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:54:12.813 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:58.308056+0000", 2024-09-20T22:54:12.813 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.029959835000000001 2024-09-20T22:54:12.813 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.813 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.813 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:54:12.813 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:58.308131+0000", 2024-09-20T22:54:12.813 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.5168999999999998e-05 2024-09-20T22:54:12.814 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.814 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.814 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:54:12.814 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:58.308219+0000", 2024-09-20T22:54:12.814 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.7656000000000002e-05 2024-09-20T22:54:12.814 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:54:12.814 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:54:12.814 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:54:12.814 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.814 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.815 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:6396 7.c 7:318c7512:::smithi08477203-44:head [delete] snapc 11d=[11d,11c,11b,11a,119,118,117,116,115,114,113,112,111,110,10e,10c,107,105,103,fd,fb,fa,f8,f5,f2,ed,d7,c2,bf] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e643)", 2024-09-20T22:54:12.815 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:52:03.276860+0000", 2024-09-20T22:54:12.815 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 129.44446062099999, 2024-09-20T22:54:12.815 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.029783397, 2024-09-20T22:54:12.815 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:54:12.815 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:54:12.815 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:54:12.815 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:54:12.815 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:54:12.815 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:54:12.815 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 6396 2024-09-20T22:54:12.816 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.816 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:54:12.816 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.816 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:54:12.816 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:03.276860+0000", 2024-09-20T22:54:12.816 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:54:12.816 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.816 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.816 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:54:12.816 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:03.276860+0000", 2024-09-20T22:54:12.817 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:54:12.817 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.817 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.817 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:54:12.817 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:03.276860+0000", 2024-09-20T22:54:12.817 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.5600000000000001e-07 2024-09-20T22:54:12.817 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.817 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.817 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:54:12.817 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:03.276862+0000", 2024-09-20T22:54:12.817 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.3489999999999999e-06 2024-09-20T22:54:12.817 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.818 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.818 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:54:12.818 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:03.276869+0000", 2024-09-20T22:54:12.818 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.3749999999999999e-06 2024-09-20T22:54:12.818 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.818 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.818 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:54:12.818 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:03.276874+0000", 2024-09-20T22:54:12.818 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.463e-06 2024-09-20T22:54:12.818 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.819 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.819 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:54:12.819 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:03.276944+0000", 2024-09-20T22:54:12.819 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.9689000000000003e-05 2024-09-20T22:54:12.819 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.819 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.819 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op must wait for map", 2024-09-20T22:54:12.819 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:03.276950+0000", 2024-09-20T22:54:12.819 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.5300000000000004e-06 2024-09-20T22:54:12.819 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.819 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.820 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:54:12.820 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:03.282451+0000", 2024-09-20T22:54:12.820 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0055015079999999996 2024-09-20T22:54:12.820 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.820 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.820 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:54:12.820 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:03.282562+0000", 2024-09-20T22:54:12.820 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000110517 2024-09-20T22:54:12.820 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.820 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.820 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:54:12.821 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:03.306643+0000", 2024-09-20T22:54:12.821 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.02408151 2024-09-20T22:54:12.821 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:54:12.821 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:54:12.821 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:54:12.821 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.821 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.821 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:6840 7.7 7:ed54065f:::smithi08477203-12:head [copy-from ver 788] snapc 136=[136,135,134,132,131,130,12f,12e,12d,12c,12b,12a,128,127,126,124,122,121,11f,11b,119,112,10e,10c,107,103,fb,fa,f8,ed,d7,c2,bf] ondisk+write+known_if_redirected+supports_pool_eio e689)", 2024-09-20T22:54:12.821 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:52:49.294097+0000", 2024-09-20T22:54:12.821 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 83.427222857999993, 2024-09-20T22:54:12.821 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.029501171, 2024-09-20T22:54:12.821 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:54:12.822 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:54:12.822 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:54:12.822 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:54:12.822 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:54:12.822 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:54:12.822 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 6840 2024-09-20T22:54:12.822 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.822 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:54:12.822 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.822 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:54:12.822 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:49.294097+0000", 2024-09-20T22:54:12.823 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:54:12.823 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.823 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.823 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:54:12.823 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:49.294097+0000", 2024-09-20T22:54:12.823 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:54:12.823 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.823 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.823 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:54:12.823 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:49.294099+0000", 2024-09-20T22:54:12.823 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.604e-06 2024-09-20T22:54:12.823 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.823 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.824 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:54:12.824 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:49.294106+0000", 2024-09-20T22:54:12.824 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.1980000000000004e-06 2024-09-20T22:54:12.824 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.824 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.824 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:54:12.824 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:49.294123+0000", 2024-09-20T22:54:12.824 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.6626e-05 2024-09-20T22:54:12.824 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.824 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.824 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:54:12.824 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:49.294131+0000", 2024-09-20T22:54:12.825 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.0309999999999998e-06 2024-09-20T22:54:12.825 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.825 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.825 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:54:12.825 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:49.294308+0000", 2024-09-20T22:54:12.825 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00017662900000000001 2024-09-20T22:54:12.825 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.825 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.825 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:54:12.825 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:49.294462+0000", 2024-09-20T22:54:12.825 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00015426400000000001 2024-09-20T22:54:12.826 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.826 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.826 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 0,3", 2024-09-20T22:54:12.826 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:49.308555+0000", 2024-09-20T22:54:12.826 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.014093207999999999 2024-09-20T22:54:12.826 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.826 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.826 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:54:12.826 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:49.314307+0000", 2024-09-20T22:54:12.826 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0057518789999999997 2024-09-20T22:54:12.826 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.826 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.827 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:54:12.827 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:49.323332+0000", 2024-09-20T22:54:12.827 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0090252010000000001 2024-09-20T22:54:12.827 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.827 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.827 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:54:12.827 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:49.323500+0000", 2024-09-20T22:54:12.827 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00016768100000000001 2024-09-20T22:54:12.827 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.827 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.827 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:54:12.828 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:49.323551+0000", 2024-09-20T22:54:12.828 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.0991e-05 2024-09-20T22:54:12.828 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.828 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.828 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:54:12.828 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:49.323599+0000", 2024-09-20T22:54:12.828 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.7858999999999997e-05 2024-09-20T22:54:12.828 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:54:12.828 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:54:12.828 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:54:12.828 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.829 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.829 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:7255 7.7 7:ed54065f:::smithi08477203-12:head [write 423807~566492,stat] snapc 149=[149,148,147,146,145,144,142,141,140,13e,13d,13c,13b,13a,139,138,136,135,132,130,12f,12e,12d,12c,12a,128,127,124,121,11f,112,103,ed] ondisk+write+known_if_redirected+supports_pool_eio e728)", 2024-09-20T22:54:12.829 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:53:28.303251+0000", 2024-09-20T22:54:12.829 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 44.418069037000002, 2024-09-20T22:54:12.829 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.039791188999999998, 2024-09-20T22:54:12.829 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:54:12.829 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:54:12.829 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:54:12.829 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:54:12.829 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:54:12.829 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:54:12.829 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 7255 2024-09-20T22:54:12.830 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.830 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:54:12.830 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.830 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:54:12.830 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:28.303251+0000", 2024-09-20T22:54:12.830 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:54:12.830 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.830 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.830 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:54:12.830 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:28.303251+0000", 2024-09-20T22:54:12.830 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:54:12.831 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.831 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.831 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:54:12.831 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:28.303253+0000", 2024-09-20T22:54:12.831 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4479999999999999e-06 2024-09-20T22:54:12.831 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.831 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.831 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:54:12.831 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:28.303922+0000", 2024-09-20T22:54:12.831 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00066900300000000005 2024-09-20T22:54:12.831 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.832 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.832 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:54:12.832 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:28.303935+0000", 2024-09-20T22:54:12.832 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3081000000000001e-05 2024-09-20T22:54:12.832 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.832 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.832 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:54:12.832 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:28.303942+0000", 2024-09-20T22:54:12.832 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.9369999999999998e-06 2024-09-20T22:54:12.832 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.832 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.832 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:54:12.833 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:28.303999+0000", 2024-09-20T22:54:12.833 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.7717999999999999e-05 2024-09-20T22:54:12.833 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.833 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.833 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op must wait for map", 2024-09-20T22:54:12.833 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:28.304006+0000", 2024-09-20T22:54:12.833 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.6150000000000002e-06 2024-09-20T22:54:12.833 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.833 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.833 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:54:12.833 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:28.307027+0000", 2024-09-20T22:54:12.833 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.003020474 2024-09-20T22:54:12.834 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.834 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.834 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:54:12.834 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:28.307136+0000", 2024-09-20T22:54:12.834 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000109522 2024-09-20T22:54:12.834 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.834 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.834 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 0,3", 2024-09-20T22:54:12.834 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:28.307288+0000", 2024-09-20T22:54:12.834 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00015164099999999999 2024-09-20T22:54:12.834 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.834 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.834 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:54:12.835 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:28.308664+0000", 2024-09-20T22:54:12.835 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001375923 2024-09-20T22:54:12.835 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.835 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.835 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:54:12.835 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:28.314469+0000", 2024-09-20T22:54:12.835 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0058058850000000002 2024-09-20T22:54:12.835 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.835 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.835 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:54:12.835 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:28.342877+0000", 2024-09-20T22:54:12.835 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.028407874999999999 2024-09-20T22:54:12.836 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.836 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.836 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:54:12.836 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:28.342967+0000", 2024-09-20T22:54:12.836 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.9480000000000004e-05 2024-09-20T22:54:12.836 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.836 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.836 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:54:12.836 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:28.343042+0000", 2024-09-20T22:54:12.836 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.5587000000000007e-05 2024-09-20T22:54:12.836 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:54:12.837 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:54:12.837 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:54:12.837 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.837 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.837 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:7256 7.7 7:ed54065f:::smithi08477203-12:head [write 1676240~407685] snapc 149=[149,148,147,146,145,144,142,141,140,13e,13d,13c,13b,13a,139,138,136,135,132,130,12f,12e,12d,12c,12a,128,127,124,121,11f,112,103,ed] ondisk+write+known_if_redirected+supports_pool_eio e728)", 2024-09-20T22:54:12.837 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:53:28.310420+0000", 2024-09-20T22:54:12.837 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 44.410899780999998, 2024-09-20T22:54:12.837 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.032753075, 2024-09-20T22:54:12.837 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:54:12.837 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:54:12.837 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:54:12.837 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:54:12.838 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:54:12.838 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:54:12.838 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 7256 2024-09-20T22:54:12.838 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.838 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:54:12.838 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.838 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:54:12.838 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:28.310420+0000", 2024-09-20T22:54:12.838 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:54:12.838 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.838 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.838 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:54:12.839 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:28.310420+0000", 2024-09-20T22:54:12.839 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:54:12.839 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.839 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.839 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:54:12.839 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:28.310423+0000", 2024-09-20T22:54:12.839 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.6060000000000001e-06 2024-09-20T22:54:12.839 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.839 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.839 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:54:12.839 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:28.310795+0000", 2024-09-20T22:54:12.839 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000371849 2024-09-20T22:54:12.840 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.840 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.840 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:54:12.840 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:28.310832+0000", 2024-09-20T22:54:12.840 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.6704e-05 2024-09-20T22:54:12.840 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.840 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.840 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:54:12.840 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:28.310846+0000", 2024-09-20T22:54:12.840 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4015999999999999e-05 2024-09-20T22:54:12.840 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.840 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.841 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:54:12.841 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:28.310916+0000", 2024-09-20T22:54:12.841 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.0037999999999996e-05 2024-09-20T22:54:12.841 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.841 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.841 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:54:12.841 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:28.311004+0000", 2024-09-20T22:54:12.841 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.8494999999999999e-05 2024-09-20T22:54:12.841 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.841 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.841 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 0,3", 2024-09-20T22:54:12.842 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:28.311132+0000", 2024-09-20T22:54:12.842 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000128219 2024-09-20T22:54:12.842 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.842 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.842 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:54:12.842 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:28.312266+0000", 2024-09-20T22:54:12.842 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001133081 2024-09-20T22:54:12.842 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.842 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.842 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:54:12.842 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:28.317154+0000", 2024-09-20T22:54:12.842 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0048885040000000001 2024-09-20T22:54:12.843 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.843 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.843 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:54:12.843 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:28.343095+0000", 2024-09-20T22:54:12.843 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.025940595 2024-09-20T22:54:12.843 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.843 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.843 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:54:12.843 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:28.343137+0000", 2024-09-20T22:54:12.843 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.2815000000000003e-05 2024-09-20T22:54:12.843 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.843 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.844 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:54:12.844 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:28.343174+0000", 2024-09-20T22:54:12.844 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.6152999999999998e-05 2024-09-20T22:54:12.844 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:54:12.844 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:54:12.844 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:54:12.844 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.844 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.844 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:7259 7.7 7:ed54065f:::smithi08477203-12:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e728)", 2024-09-20T22:54:12.844 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:53:28.314947+0000", 2024-09-20T22:54:12.844 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 44.406373295999998, 2024-09-20T22:54:12.845 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.028624334000000001, 2024-09-20T22:54:12.845 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:54:12.845 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:54:12.845 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:54:12.845 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:54:12.845 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:54:12.845 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:54:12.845 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 7259 2024-09-20T22:54:12.845 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.845 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:54:12.845 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.846 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:54:12.846 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:28.314947+0000", 2024-09-20T22:54:12.846 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:54:12.846 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.846 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.846 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:54:12.846 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:28.314947+0000", 2024-09-20T22:54:12.846 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:54:12.846 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.846 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.846 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:54:12.846 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:28.314947+0000", 2024-09-20T22:54:12.847 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.8099999999999999e-07 2024-09-20T22:54:12.847 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.847 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.847 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:54:12.847 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:28.314949+0000", 2024-09-20T22:54:12.847 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.689e-06 2024-09-20T22:54:12.847 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.847 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.847 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:54:12.847 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:28.314954+0000", 2024-09-20T22:54:12.847 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.1739999999999999e-06 2024-09-20T22:54:12.847 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.848 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.848 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:54:12.848 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:28.314958+0000", 2024-09-20T22:54:12.848 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.0570000000000003e-06 2024-09-20T22:54:12.848 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.848 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.848 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:54:12.848 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:28.317268+0000", 2024-09-20T22:54:12.848 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0023099230000000002 2024-09-20T22:54:12.848 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.848 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.849 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for rw locks", 2024-09-20T22:54:12.849 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:28.317387+0000", 2024-09-20T22:54:12.849 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000118591 2024-09-20T22:54:12.849 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.849 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.849 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:54:12.849 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:28.343433+0000", 2024-09-20T22:54:12.849 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.026046430999999998 2024-09-20T22:54:12.849 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.849 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.849 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:54:12.849 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:28.343493+0000", 2024-09-20T22:54:12.849 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.0262e-05 2024-09-20T22:54:12.850 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.850 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.850 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:54:12.850 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:28.343571+0000", 2024-09-20T22:54:12.850 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.7925999999999999e-05 2024-09-20T22:54:12.850 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:54:12.850 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:54:12.850 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:54:12.850 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.850 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.850 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:7418 7.c 7:3f0fe232:::smithi08477203-4:head [copy-from ver 2292] snapc 14e=[14e,14c,14a,149,148,145,142,141,13e,13d,13c,13b,13a,139,135,132,130,12f,12e,12c,12a,128,127,124,121,11f,112,103] ondisk+write+known_if_redirected+supports_pool_eio e743)", 2024-09-20T22:54:12.851 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:53:43.499506+0000", 2024-09-20T22:54:12.851 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 29.221814556999998, 2024-09-20T22:54:12.851 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.014878801000000001, 2024-09-20T22:54:12.851 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:54:12.851 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:54:12.851 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:54:12.851 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:54:12.851 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:54:12.851 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:54:12.851 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 7418 2024-09-20T22:54:12.851 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.852 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:54:12.852 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.852 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:54:12.852 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:43.499506+0000", 2024-09-20T22:54:12.852 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:54:12.852 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.852 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.852 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:54:12.852 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:43.499506+0000", 2024-09-20T22:54:12.852 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:54:12.853 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.853 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.853 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:54:12.853 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:43.499507+0000", 2024-09-20T22:54:12.853 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.153e-06 2024-09-20T22:54:12.853 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.853 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.853 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:54:12.853 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:43.499513+0000", 2024-09-20T22:54:12.853 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.9699999999999996e-06 2024-09-20T22:54:12.853 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.853 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.854 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:54:12.854 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:43.499524+0000", 2024-09-20T22:54:12.854 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.136e-05 2024-09-20T22:54:12.854 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.854 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.854 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:54:12.854 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:43.499530+0000", 2024-09-20T22:54:12.854 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.339e-06 2024-09-20T22:54:12.854 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.854 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.854 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:54:12.854 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:43.499614+0000", 2024-09-20T22:54:12.854 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.4133000000000002e-05 2024-09-20T22:54:12.855 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.855 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.855 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:54:12.855 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:43.499692+0000", 2024-09-20T22:54:12.855 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.8404000000000004e-05 2024-09-20T22:54:12.855 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.855 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.855 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 3,7", 2024-09-20T22:54:12.855 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:43.503888+0000", 2024-09-20T22:54:12.855 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0041963520000000004 2024-09-20T22:54:12.855 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.855 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.855 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:54:12.856 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:43.511389+0000", 2024-09-20T22:54:12.856 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.007500423 2024-09-20T22:54:12.856 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.856 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.856 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:54:12.856 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:43.514158+0000", 2024-09-20T22:54:12.856 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00276866 2024-09-20T22:54:12.856 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.856 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.856 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:54:12.856 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:43.514308+0000", 2024-09-20T22:54:12.856 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00015024 2024-09-20T22:54:12.856 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.856 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.857 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:54:12.857 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:43.514348+0000", 2024-09-20T22:54:12.857 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.9975000000000003e-05 2024-09-20T22:54:12.857 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.857 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.857 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:54:12.857 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:43.514385+0000", 2024-09-20T22:54:12.857 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.6792000000000003e-05 2024-09-20T22:54:12.857 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:54:12.857 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:54:12.857 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:54:12.857 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.857 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.858 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:7437 7.7 7:ed54065f:::smithi08477203-12:head [rollback 124] snapc 14e=[14e,14c,14a,149,148,145,142,141,13e,13d,13c,13b,13a,139,132,130,12f,12e,12c,12a,128,127,124,121,11f,112,103] ondisk+write+known_if_redirected+supports_pool_eio e744)", 2024-09-20T22:54:12.858 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:53:44.435304+0000", 2024-09-20T22:54:12.858 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 28.286016371999999, 2024-09-20T22:54:12.858 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.021637786999999999, 2024-09-20T22:54:12.858 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:54:12.858 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:54:12.858 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:54:12.858 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:54:12.858 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:54:12.858 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:54:12.858 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 7437 2024-09-20T22:54:12.858 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.858 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:54:12.859 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.859 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:54:12.859 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:44.435304+0000", 2024-09-20T22:54:12.859 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:54:12.859 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.859 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.859 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:54:12.859 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:44.435304+0000", 2024-09-20T22:54:12.859 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:54:12.859 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.859 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.859 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:54:12.860 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:44.435304+0000", 2024-09-20T22:54:12.860 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.7300000000000002e-07 2024-09-20T22:54:12.860 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.860 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.860 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:54:12.860 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:44.435306+0000", 2024-09-20T22:54:12.860 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.0360000000000001e-06 2024-09-20T22:54:12.860 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.860 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.860 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:54:12.860 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:44.435311+0000", 2024-09-20T22:54:12.861 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.4519999999999999e-06 2024-09-20T22:54:12.861 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.861 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.861 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:54:12.861 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:44.435314+0000", 2024-09-20T22:54:12.861 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.3409999999999999e-06 2024-09-20T22:54:12.861 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.861 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.861 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:54:12.861 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:44.436703+0000", 2024-09-20T22:54:12.861 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0013886300000000001 2024-09-20T22:54:12.861 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.862 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.862 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:54:12.862 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:44.436796+0000", 2024-09-20T22:54:12.862 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.3012999999999999e-05 2024-09-20T22:54:12.862 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.862 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.862 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 0,3", 2024-09-20T22:54:12.862 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:44.436981+0000", 2024-09-20T22:54:12.862 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00018479100000000001 2024-09-20T22:54:12.862 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.863 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.863 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:54:12.863 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:44.438490+0000", 2024-09-20T22:54:12.863 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001509774 2024-09-20T22:54:12.863 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.863 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.863 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:54:12.863 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:44.438899+0000", 2024-09-20T22:54:12.863 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00040900099999999998 2024-09-20T22:54:12.863 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.863 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.863 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:54:12.863 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:44.456854+0000", 2024-09-20T22:54:12.864 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.017954365999999999 2024-09-20T22:54:12.864 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.864 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.864 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:54:12.864 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:44.456901+0000", 2024-09-20T22:54:12.864 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.7068999999999999e-05 2024-09-20T22:54:12.864 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.864 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.864 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:54:12.864 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:44.456942+0000", 2024-09-20T22:54:12.864 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.0941e-05 2024-09-20T22:54:12.865 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:54:12.865 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:54:12.865 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:54:12.865 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.865 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.865 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:7438 7.7 7:ed54065f:::smithi08477203-12:head [append 0~0] snapc 14e=[14e,14c,14a,149,148,145,142,141,13e,13d,13c,13b,13a,139,132,130,12f,12e,12c,12a,128,127,124,121,11f,112,103] ondisk+write+known_if_redirected+supports_pool_eio e744)", 2024-09-20T22:54:12.865 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:53:44.435340+0000", 2024-09-20T22:54:12.865 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 28.285980296000002, 2024-09-20T22:54:12.865 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.037994317999999999, 2024-09-20T22:54:12.865 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:54:12.865 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:54:12.865 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:54:12.865 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:54:12.865 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:54:12.866 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:54:12.866 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 7438 2024-09-20T22:54:12.866 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.866 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:54:12.866 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.866 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:54:12.866 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:44.435340+0000", 2024-09-20T22:54:12.866 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:54:12.866 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.866 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.866 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:54:12.866 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:44.435340+0000", 2024-09-20T22:54:12.867 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:54:12.867 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.867 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.867 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:54:12.867 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:44.435340+0000", 2024-09-20T22:54:12.867 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.5100000000000001e-07 2024-09-20T22:54:12.867 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.867 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.867 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:54:12.867 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:44.435342+0000", 2024-09-20T22:54:12.867 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.7349999999999999e-06 2024-09-20T22:54:12.867 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.868 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.868 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:54:12.868 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:44.435350+0000", 2024-09-20T22:54:12.868 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.0560000000000005e-06 2024-09-20T22:54:12.868 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.868 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.868 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:54:12.868 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:44.435354+0000", 2024-09-20T22:54:12.868 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.0069999999999999e-06 2024-09-20T22:54:12.868 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.868 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.868 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:54:12.869 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:44.437843+0000", 2024-09-20T22:54:12.869 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.002489018 2024-09-20T22:54:12.869 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.869 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.869 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:54:12.869 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:44.437924+0000", 2024-09-20T22:54:12.869 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.0477999999999998e-05 2024-09-20T22:54:12.869 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.869 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.869 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 0,3", 2024-09-20T22:54:12.869 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:44.438050+0000", 2024-09-20T22:54:12.869 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00012694599999999999 2024-09-20T22:54:12.869 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.870 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.870 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:54:12.870 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:44.438583+0000", 2024-09-20T22:54:12.870 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00053302799999999995 2024-09-20T22:54:12.870 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.870 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.870 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:54:12.870 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:44.439219+0000", 2024-09-20T22:54:12.870 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00063534599999999996 2024-09-20T22:54:12.870 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.870 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.871 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:54:12.871 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:44.473187+0000", 2024-09-20T22:54:12.871 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.033968241000000003 2024-09-20T22:54:12.871 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.871 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.871 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:54:12.871 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:44.473269+0000", 2024-09-20T22:54:12.871 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.2150999999999993e-05 2024-09-20T22:54:12.871 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:54:12.871 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:54:12.871 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:54:12.872 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:44.473334+0000", 2024-09-20T22:54:12.872 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.5061000000000006e-05 2024-09-20T22:54:12.872 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:54:12.872 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:54:12.872 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:54:12.872 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:54:12.872 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:54:12.872 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-20T22:54:12.894 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:12.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:12.895 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:54:12.996 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:54:13.017 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:13.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:13.018 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:54:13.119 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:54:13.141 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:13.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:13.142 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:54:13.243 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:54:13.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:13.266 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:13.266 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:54:13.310 INFO:tasks.rados.rados.0.smithi084.stdout:3071: done (3 left) 2024-09-20T22:54:13.310 INFO:tasks.rados.rados.0.smithi084.stdout:3074: done (2 left) 2024-09-20T22:54:13.310 INFO:tasks.rados.rados.0.smithi084.stdout:3075: done (1 left) 2024-09-20T22:54:13.310 INFO:tasks.rados.rados.0.smithi084.stdout:3076: done (0 left) 2024-09-20T22:54:13.310 INFO:tasks.rados.rados.0.smithi084.stdout:3077: copy_from oid 3 from oid 28 current snap is 348 2024-09-20T22:54:13.310 INFO:tasks.rados.rados.0.smithi084.stdout:3078: write (excl) oid 18 current snap is 348 2024-09-20T22:54:13.310 INFO:tasks.rados.rados.0.smithi084.stdout:3078: seq_num 1060 ranges {615420=737007,1912207=425038,3026137=542216} 2024-09-20T22:54:13.317 INFO:tasks.rados.rados.0.smithi084.stdout:3078: writing smithi08477203-18 from 615420 to 1352427 tid 1 2024-09-20T22:54:13.323 INFO:tasks.rados.rados.0.smithi084.stdout:3078: writing smithi08477203-18 from 1912207 to 2337245 tid 2 2024-09-20T22:54:13.330 INFO:tasks.rados.rados.0.smithi084.stdout:3078: writing smithi08477203-18 from 3026137 to 3568353 tid 3 2024-09-20T22:54:13.330 INFO:tasks.rados.rados.0.smithi084.stdout:3079: read oid 39 snap -1 2024-09-20T22:54:13.330 INFO:tasks.rados.rados.0.smithi084.stdout:3079: expect deleted 2024-09-20T22:54:13.330 INFO:tasks.rados.rados.0.smithi084.stdout:3080: copy_from oid 48 from oid 33 current snap is 348 2024-09-20T22:54:13.330 INFO:tasks.rados.rados.0.smithi084.stdout:3077: finishing copy_from to smithi08477203-3 2024-09-20T22:54:13.330 INFO:tasks.rados.rados.0.smithi084.stdout:3077: got expected ENOENT (src dne) 2024-09-20T22:54:13.330 INFO:tasks.rados.rados.0.smithi084.stdout:3077: finishing copy_from racing read to smithi08477203-3 2024-09-20T22:54:13.331 INFO:tasks.rados.rados.0.smithi084.stdout:3078: finishing write tid 1 to smithi08477203-18 2024-09-20T22:54:13.331 INFO:tasks.rados.rados.0.smithi084.stdout:3078: finishing write tid 2 to smithi08477203-18 2024-09-20T22:54:13.331 INFO:tasks.rados.rados.0.smithi084.stdout:3077: done (3 left) 2024-09-20T22:54:13.331 INFO:tasks.rados.rados.0.smithi084.stdout:3081: write oid 6 current snap is 348 2024-09-20T22:54:13.331 INFO:tasks.rados.rados.0.smithi084.stdout:3081: seq_num 1061 ranges {632227=412256,1570689=568696,2653560=753488} 2024-09-20T22:54:13.335 INFO:tasks.rados.rados.0.smithi084.stdout:3081: writing smithi08477203-6 from 632227 to 1044483 tid 1 2024-09-20T22:54:13.342 INFO:tasks.rados.rados.0.smithi084.stdout:3081: writing smithi08477203-6 from 1570689 to 2139385 tid 2 2024-09-20T22:54:13.350 INFO:tasks.rados.rados.0.smithi084.stdout:3081: writing smithi08477203-6 from 2653560 to 3407048 tid 3 2024-09-20T22:54:13.350 INFO:tasks.rados.rados.0.smithi084.stdout:3082: snap_remove snap 296 2024-09-20T22:54:13.367 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:54:13.387 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:13.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:13.388 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:54:13.489 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:54:13.512 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:13.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:13.513 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:54:13.613 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:54:13.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:13 smithi084 ceph-mon[40024]: pgmap v1059: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 514 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 11 MiB/s rd, 2.1 MiB/s wr, 16 op/s 2024-09-20T22:54:13.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:13 smithi084 ceph-mon[40024]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:54:13.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:13 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/2150253749' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T22:54:13.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:13 smithi084 ceph-mon[40024]: osdmap e772: 8 total, 8 up, 7 in 2024-09-20T22:54:13.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:13 smithi084 ceph-mon[40024]: 7.d scrub starts 2024-09-20T22:54:13.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:13 smithi084 ceph-mon[40024]: 7.d scrub ok 2024-09-20T22:54:13.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:13 smithi084 ceph-mon[40024]: osdmap e773: 8 total, 8 up, 7 in 2024-09-20T22:54:13.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:13 smithi084 ceph-mon[29774]: pgmap v1059: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 514 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 11 MiB/s rd, 2.1 MiB/s wr, 16 op/s 2024-09-20T22:54:13.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:13 smithi084 ceph-mon[29774]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:54:13.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:13 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/2150253749' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T22:54:13.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:13 smithi084 ceph-mon[29774]: osdmap e772: 8 total, 8 up, 7 in 2024-09-20T22:54:13.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:13 smithi084 ceph-mon[29774]: 7.d scrub starts 2024-09-20T22:54:13.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:13 smithi084 ceph-mon[29774]: 7.d scrub ok 2024-09-20T22:54:13.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:13 smithi084 ceph-mon[29774]: osdmap e773: 8 total, 8 up, 7 in 2024-09-20T22:54:13.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:13.636 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:13.636 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:54:13.718 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:13 smithi154 ceph-mon[33264]: pgmap v1059: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 514 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 11 MiB/s rd, 2.1 MiB/s wr, 16 op/s 2024-09-20T22:54:13.718 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:13 smithi154 ceph-mon[33264]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:54:13.718 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:13 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/2150253749' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T22:54:13.718 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:13 smithi154 ceph-mon[33264]: osdmap e772: 8 total, 8 up, 7 in 2024-09-20T22:54:13.718 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:13 smithi154 ceph-mon[33264]: 7.d scrub starts 2024-09-20T22:54:13.719 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:13 smithi154 ceph-mon[33264]: 7.d scrub ok 2024-09-20T22:54:13.719 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:13 smithi154 ceph-mon[33264]: osdmap e773: 8 total, 8 up, 7 in 2024-09-20T22:54:13.737 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:54:13.746 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-20T22:54:13.757 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:13.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:13.758 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:54:13.860 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:54:13.881 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:13.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:13.882 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:54:13.983 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:54:14.002 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:54:14.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:14.090 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:14.091 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:54:14.191 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:54:14.213 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:14.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:14.214 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:54:14.313 INFO:tasks.rados.rados.0.smithi084.stdout:3082: done (4 left) 2024-09-20T22:54:14.313 INFO:tasks.rados.rados.0.smithi084.stdout:3083: snap_create 2024-09-20T22:54:14.313 INFO:tasks.rados.rados.0.smithi084.stdout:3078: finishing write tid 3 to smithi08477203-18 2024-09-20T22:54:14.313 INFO:tasks.rados.rados.0.smithi084.stdout:3078: finishing write tid 4 to smithi08477203-18 2024-09-20T22:54:14.313 INFO:tasks.rados.rados.0.smithi084.stdout:3078: finishing write tid 5 to smithi08477203-18 2024-09-20T22:54:14.313 INFO:tasks.rados.rados.0.smithi084.stdout:3078: oid 18 updating version 0 to 1827 2024-09-20T22:54:14.314 INFO:tasks.rados.rados.0.smithi084.stdout:3078: oid 18 updating version 1827 to 1829 2024-09-20T22:54:14.314 INFO:tasks.rados.rados.0.smithi084.stdout:3078: oid 18 version 1829 is already newer than 1828 2024-09-20T22:54:14.314 INFO:tasks.rados.rados.0.smithi084.stdout:3078: oid 18 updating version 1829 to 1830 2024-09-20T22:54:14.314 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 18 v 1830 (ObjNum 1060 snap 348 seq_num 1060) dirty exists 2024-09-20T22:54:14.314 INFO:tasks.rados.rados.0.smithi084.stdout:3078: left oid 18 (ObjNum 1060 snap 348 seq_num 1060) 2024-09-20T22:54:14.314 INFO:tasks.rados.rados.0.smithi084.stdout:3081: finishing write tid 1 to smithi08477203-6 2024-09-20T22:54:14.314 INFO:tasks.rados.rados.0.smithi084.stdout:3080: finishing copy_from racing read to smithi08477203-48 2024-09-20T22:54:14.314 INFO:tasks.rados.rados.0.smithi084.stdout:3080: finishing copy_from to smithi08477203-48 2024-09-20T22:54:14.314 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 48 v 1829 (ObjNum 984 snap 319 seq_num 984) dirty exists 2024-09-20T22:54:14.314 INFO:tasks.rados.rados.0.smithi084.stdout:3081: finishing write tid 2 to smithi08477203-6 2024-09-20T22:54:14.314 INFO:tasks.rados.rados.0.smithi084.stdout:3081: finishing write tid 3 to smithi08477203-6 2024-09-20T22:54:14.314 INFO:tasks.rados.rados.0.smithi084.stdout:3081: finishing write tid 4 to smithi08477203-6 2024-09-20T22:54:14.314 INFO:tasks.rados.rados.0.smithi084.stdout:3081: finishing write tid 5 to smithi08477203-6 2024-09-20T22:54:14.314 INFO:tasks.rados.rados.0.smithi084.stdout:3081: oid 6 updating version 0 to 1255 2024-09-20T22:54:14.315 INFO:tasks.rados.rados.0.smithi084.stdout:3081: oid 6 updating version 1255 to 1257 2024-09-20T22:54:14.315 INFO:tasks.rados.rados.0.smithi084.stdout:3081: oid 6 updating version 1257 to 1258 2024-09-20T22:54:14.315 INFO:tasks.rados.rados.0.smithi084.stdout:3081: oid 6 version 1258 is already newer than 1256 2024-09-20T22:54:14.315 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 6 v 1258 (ObjNum 1061 snap 348 seq_num 1061) dirty exists 2024-09-20T22:54:14.315 INFO:tasks.rados.rados.0.smithi084.stdout:3081: left oid 6 (ObjNum 1061 snap 348 seq_num 1061) 2024-09-20T22:54:14.315 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:54:14.339 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:14.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:14.340 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:54:14.386 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:14 smithi154 ceph-mon[33264]: 7.7 scrub starts 2024-09-20T22:54:14.386 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:14 smithi154 ceph-mon[33264]: 7.7 scrub ok 2024-09-20T22:54:14.387 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:14 smithi154 ceph-mon[33264]: osdmap e774: 8 total, 8 up, 7 in 2024-09-20T22:54:14.441 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:54:14.462 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:14.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:14.463 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:54:14.564 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:54:14.585 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:14.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:14.586 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:54:14.687 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:54:14.689 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-20T22:54:14.689 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [], 2024-09-20T22:54:14.689 INFO:teuthology.orchestra.run.smithi154.stdout: "num_ops": 0 2024-09-20T22:54:14.689 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-20T22:54:14.708 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:14.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:14.709 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:54:14.810 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:54:14.832 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:14.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:14.833 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:54:14.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:14 smithi084 ceph-mon[29774]: 7.7 scrub starts 2024-09-20T22:54:14.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:14 smithi084 ceph-mon[29774]: 7.7 scrub ok 2024-09-20T22:54:14.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:14 smithi084 ceph-mon[29774]: osdmap e774: 8 total, 8 up, 7 in 2024-09-20T22:54:14.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:14 smithi084 ceph-mon[40024]: 7.7 scrub starts 2024-09-20T22:54:14.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:14 smithi084 ceph-mon[40024]: 7.7 scrub ok 2024-09-20T22:54:14.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:14 smithi084 ceph-mon[40024]: osdmap e774: 8 total, 8 up, 7 in 2024-09-20T22:54:14.934 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:54:14.956 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:14.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:14.957 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:54:15.048 INFO:tasks.thrashosds.thrasher:in_osds: [2, 3, 5, 6, 7, 0, 4] out_osds: [1] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-20T22:54:15.048 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-20T22:54:15.048 INFO:tasks.thrashosds.thrasher:Adding osd 1 2024-09-20T22:54:15.048 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd in 1 2024-09-20T22:54:15.058 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:54:15.080 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:15.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:15.081 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:54:15.182 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:54:15.203 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:15.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:15.204 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:54:15.281 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-20T22:54:15.293 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:54:15.305 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:54:15.316 INFO:tasks.rados.rados.0.smithi084.stdout:3078: done (4 left) 2024-09-20T22:54:15.316 INFO:tasks.rados.rados.0.smithi084.stdout:3079: done (3 left) 2024-09-20T22:54:15.316 INFO:tasks.rados.rados.0.smithi084.stdout:3080: done (2 left) 2024-09-20T22:54:15.316 INFO:tasks.rados.rados.0.smithi084.stdout:3081: done (1 left) 2024-09-20T22:54:15.316 INFO:tasks.rados.rados.0.smithi084.stdout:3083: done (0 left) 2024-09-20T22:54:15.316 INFO:tasks.rados.rados.0.smithi084.stdout:3084: read oid 32 snap 316 2024-09-20T22:54:15.317 INFO:tasks.rados.rados.0.smithi084.stdout:3084: expect (ObjNum 466 snap 128 seq_num 466) 2024-09-20T22:54:15.317 INFO:tasks.rados.rados.0.smithi084.stdout:3085: read oid 27 snap -1 2024-09-20T22:54:15.317 INFO:tasks.rados.rados.0.smithi084.stdout:3085: expect deleted 2024-09-20T22:54:15.317 INFO:tasks.rados.rados.0.smithi084.stdout:3086: read oid 31 snap -1 2024-09-20T22:54:15.317 INFO:tasks.rados.rados.0.smithi084.stdout:3086: expect (ObjNum 1058 snap 347 seq_num 1058) 2024-09-20T22:54:15.317 INFO:tasks.rados.rados.0.smithi084.stdout:3087: delete oid 18 current snap is 349 2024-09-20T22:54:15.322 INFO:tasks.rados.rados.0.smithi084.stdout:3085: done (3 left) 2024-09-20T22:54:15.322 INFO:tasks.rados.rados.0.smithi084.stdout:3087: done (2 left) 2024-09-20T22:54:15.322 INFO:tasks.rados.rados.0.smithi084.stdout:3088: snap_remove snap 343 2024-09-20T22:54:15.332 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:15.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:15.333 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:54:15.434 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:54:15.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:15.456 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:15.456 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:54:15.553 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:15 smithi154 ceph-mon[33264]: pgmap v1062: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 529 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 4.5 MiB/s rd, 5.6 MiB/s wr, 18 op/s 2024-09-20T22:54:15.553 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:15 smithi154 ceph-mon[33264]: 7.6 scrub starts 2024-09-20T22:54:15.553 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:15 smithi154 ceph-mon[33264]: 7.6 scrub ok 2024-09-20T22:54:15.553 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:15 smithi154 ceph-mon[33264]: 4.14 scrub starts 2024-09-20T22:54:15.553 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:15 smithi154 ceph-mon[33264]: 4.14 scrub ok 2024-09-20T22:54:15.554 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:15 smithi154 ceph-mon[33264]: 6.1 scrub starts 2024-09-20T22:54:15.554 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:15 smithi154 ceph-mon[33264]: 6.1 scrub ok 2024-09-20T22:54:15.554 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:15 smithi154 ceph-mon[33264]: osdmap e775: 8 total, 8 up, 7 in 2024-09-20T22:54:15.555 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.7 dump_blocked_ops 2024-09-20T22:54:15.557 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:54:15.577 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:15.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:15.578 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:54:15.664 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:15 smithi084 ceph-mon[40024]: pgmap v1062: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 529 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 4.5 MiB/s rd, 5.6 MiB/s wr, 18 op/s 2024-09-20T22:54:15.664 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:15 smithi084 ceph-mon[40024]: 7.6 scrub starts 2024-09-20T22:54:15.664 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:15 smithi084 ceph-mon[40024]: 7.6 scrub ok 2024-09-20T22:54:15.664 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:15 smithi084 ceph-mon[40024]: 4.14 scrub starts 2024-09-20T22:54:15.664 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:15 smithi084 ceph-mon[40024]: 4.14 scrub ok 2024-09-20T22:54:15.664 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:15 smithi084 ceph-mon[40024]: 6.1 scrub starts 2024-09-20T22:54:15.664 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:15 smithi084 ceph-mon[40024]: 6.1 scrub ok 2024-09-20T22:54:15.664 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:15 smithi084 ceph-mon[40024]: osdmap e775: 8 total, 8 up, 7 in 2024-09-20T22:54:15.665 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:15 smithi084 ceph-mon[29774]: pgmap v1062: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 529 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 4.5 MiB/s rd, 5.6 MiB/s wr, 18 op/s 2024-09-20T22:54:15.665 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:15 smithi084 ceph-mon[29774]: 7.6 scrub starts 2024-09-20T22:54:15.665 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:15 smithi084 ceph-mon[29774]: 7.6 scrub ok 2024-09-20T22:54:15.665 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:15 smithi084 ceph-mon[29774]: 4.14 scrub starts 2024-09-20T22:54:15.665 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:15 smithi084 ceph-mon[29774]: 4.14 scrub ok 2024-09-20T22:54:15.665 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:15 smithi084 ceph-mon[29774]: 6.1 scrub starts 2024-09-20T22:54:15.665 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:15 smithi084 ceph-mon[29774]: 6.1 scrub ok 2024-09-20T22:54:15.665 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:15 smithi084 ceph-mon[29774]: osdmap e775: 8 total, 8 up, 7 in 2024-09-20T22:54:15.679 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:54:15.700 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:15.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:15.701 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:54:15.723 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-20T22:54:15.801 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:54:15.801 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:54:15.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:15.823 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:15.823 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:54:15.924 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:54:15.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:15.949 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:15.950 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:54:15.970 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:54:16.008 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:54:16.051 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:54:16.074 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:16.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:16.075 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:54:16.176 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:54:16.197 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:16.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:16.198 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:54:16.228 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:16 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mon-a[29770]: 2024-09-20T22:54:16.106+0000 7f26ad5e8640 -1 mon.a@0(leader).osd e775 definitely_dead 0 2024-09-20T22:54:16.291 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-20T22:54:16.291 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [], 2024-09-20T22:54:16.291 INFO:teuthology.orchestra.run.smithi154.stdout: "complaint_time": 30, 2024-09-20T22:54:16.291 INFO:teuthology.orchestra.run.smithi154.stdout: "num_blocked_ops": 0 2024-09-20T22:54:16.291 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-20T22:54:16.299 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:54:16.309 INFO:teuthology.orchestra.run.smithi084.stderr:marked in osd.1. 2024-09-20T22:54:16.322 INFO:tasks.rados.rados.0.smithi084.stdout:3088: done (2 left) 2024-09-20T22:54:16.322 INFO:tasks.rados.rados.0.smithi084.stdout:3089: write oid 47 current snap is 349 2024-09-20T22:54:16.323 INFO:tasks.rados.rados.0.smithi084.stdout:3089: seq_num 1063 ranges {665841=562754,1687653=572365,2708406=776032} 2024-09-20T22:54:16.328 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:16.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:16.329 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:54:16.329 INFO:tasks.rados.rados.0.smithi084.stdout:3089: writing smithi08477203-47 from 665841 to 1228595 tid 1 2024-09-20T22:54:16.336 INFO:tasks.rados.rados.0.smithi084.stdout:3089: writing smithi08477203-47 from 1687653 to 2260018 tid 2 2024-09-20T22:54:16.344 INFO:tasks.rados.rados.0.smithi084.stdout:3089: writing smithi08477203-47 from 2708406 to 3484438 tid 3 2024-09-20T22:54:16.345 INFO:tasks.rados.rados.0.smithi084.stdout:3084: expect (ObjNum 466 snap 128 seq_num 466) 2024-09-20T22:54:16.430 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:54:16.453 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:16.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:16.454 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:54:16.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:16 smithi154 ceph-mon[33264]: 7.9 scrub starts 2024-09-20T22:54:16.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:16 smithi154 ceph-mon[33264]: 7.9 scrub ok 2024-09-20T22:54:16.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:16 smithi154 ceph-mon[33264]: 6.1b scrub starts 2024-09-20T22:54:16.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:16 smithi154 ceph-mon[33264]: 6.1b scrub ok 2024-09-20T22:54:16.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:16 smithi154 ceph-mon[33264]: 7.8 scrub starts 2024-09-20T22:54:16.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:16 smithi154 ceph-mon[33264]: 7.8 scrub ok 2024-09-20T22:54:16.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:16 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/4156641079' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["1"]}]: dispatch 2024-09-20T22:54:16.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:16 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/4156641079' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["1"]}]': finished 2024-09-20T22:54:16.506 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:16 smithi154 ceph-mon[33264]: osdmap e776: 8 total, 8 up, 8 in 2024-09-20T22:54:16.554 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:54:16.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:16.577 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:16.577 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:54:16.602 INFO:tasks.rados.rados.0.smithi084.stdout:3086: expect (ObjNum 1058 snap 347 seq_num 1058) 2024-09-20T22:54:16.614 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:16 smithi084 ceph-mon[29774]: 7.9 scrub starts 2024-09-20T22:54:16.614 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:16 smithi084 ceph-mon[29774]: 7.9 scrub ok 2024-09-20T22:54:16.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:16 smithi084 ceph-mon[29774]: 6.1b scrub starts 2024-09-20T22:54:16.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:16 smithi084 ceph-mon[29774]: 6.1b scrub ok 2024-09-20T22:54:16.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:16 smithi084 ceph-mon[29774]: 7.8 scrub starts 2024-09-20T22:54:16.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:16 smithi084 ceph-mon[29774]: 7.8 scrub ok 2024-09-20T22:54:16.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:16 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/4156641079' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["1"]}]: dispatch 2024-09-20T22:54:16.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:16 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/4156641079' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["1"]}]': finished 2024-09-20T22:54:16.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:16 smithi084 ceph-mon[29774]: osdmap e776: 8 total, 8 up, 8 in 2024-09-20T22:54:16.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:16 smithi084 ceph-mon[40024]: 7.9 scrub starts 2024-09-20T22:54:16.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:16 smithi084 ceph-mon[40024]: 7.9 scrub ok 2024-09-20T22:54:16.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:16 smithi084 ceph-mon[40024]: 6.1b scrub starts 2024-09-20T22:54:16.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:16 smithi084 ceph-mon[40024]: 6.1b scrub ok 2024-09-20T22:54:16.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:16 smithi084 ceph-mon[40024]: 7.8 scrub starts 2024-09-20T22:54:16.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:16 smithi084 ceph-mon[40024]: 7.8 scrub ok 2024-09-20T22:54:16.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:16 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/4156641079' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["1"]}]: dispatch 2024-09-20T22:54:16.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:16 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/4156641079' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["1"]}]': finished 2024-09-20T22:54:16.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:16 smithi084 ceph-mon[40024]: osdmap e776: 8 total, 8 up, 8 in 2024-09-20T22:54:16.678 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:54:16.700 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:16.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:16.701 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:54:16.801 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:54:16.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:16.823 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:16.824 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:54:16.846 INFO:tasks.rados.rados.0.smithi084.stdout:3089: finishing write tid 1 to smithi08477203-47 2024-09-20T22:54:16.846 INFO:tasks.rados.rados.0.smithi084.stdout:3089: finishing write tid 2 to smithi08477203-47 2024-09-20T22:54:16.846 INFO:tasks.rados.rados.0.smithi084.stdout:3089: finishing write tid 3 to smithi08477203-47 2024-09-20T22:54:16.846 INFO:tasks.rados.rados.0.smithi084.stdout:3089: finishing write tid 4 to smithi08477203-47 2024-09-20T22:54:16.846 INFO:tasks.rados.rados.0.smithi084.stdout:3089: finishing write tid 5 to smithi08477203-47 2024-09-20T22:54:16.846 INFO:tasks.rados.rados.0.smithi084.stdout:3089: oid 47 updating version 0 to 1835 2024-09-20T22:54:16.846 INFO:tasks.rados.rados.0.smithi084.stdout:3089: oid 47 updating version 1835 to 1837 2024-09-20T22:54:16.846 INFO:tasks.rados.rados.0.smithi084.stdout:3089: oid 47 updating version 1837 to 1838 2024-09-20T22:54:16.846 INFO:tasks.rados.rados.0.smithi084.stdout:3089: oid 47 version 1838 is already newer than 1836 2024-09-20T22:54:16.847 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 47 v 1838 (ObjNum 1063 snap 349 seq_num 1063) dirty exists 2024-09-20T22:54:16.847 INFO:tasks.rados.rados.0.smithi084.stdout:3089: left oid 47 (ObjNum 1063 snap 349 seq_num 1063) 2024-09-20T22:54:16.847 INFO:tasks.rados.rados.0.smithi084.stdout:3084: done (2 left) 2024-09-20T22:54:16.847 INFO:tasks.rados.rados.0.smithi084.stdout:3086: done (1 left) 2024-09-20T22:54:16.847 INFO:tasks.rados.rados.0.smithi084.stdout:3089: done (0 left) 2024-09-20T22:54:16.847 INFO:tasks.rados.rados.0.smithi084.stdout:3090: snap_create 2024-09-20T22:54:16.925 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:54:16.946 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:16.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:16.947 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:54:17.047 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:54:17.070 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:17.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:17.071 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:54:17.082 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.7 dump_historic_ops 2024-09-20T22:54:17.135 INFO:tasks.thrashosds.thrasher:Added osd 1 2024-09-20T22:54:17.172 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:54:17.193 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:17.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:17.194 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:54:17.294 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:54:17.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:17.320 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:17.320 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:54:17.321 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is unset 2024-09-20T22:54:17.321 INFO:tasks.rados.rados.0.smithi084.stdout:3090: done (0 left) 2024-09-20T22:54:17.321 INFO:tasks.rados.rados.0.smithi084.stdout:3091: read oid 7 snap -1 2024-09-20T22:54:17.321 INFO:tasks.rados.rados.0.smithi084.stdout:3091: expect (ObjNum 964 snap 314 seq_num 964) 2024-09-20T22:54:17.321 INFO:tasks.rados.rados.0.smithi084.stdout:3092: copy_from oid 28 from oid 21 current snap is 350 2024-09-20T22:54:17.321 INFO:tasks.rados.rados.0.smithi084.stdout:3093: snap_remove snap 304 2024-09-20T22:54:17.395 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:54:17.421 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:54:17.443 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:17.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:17.444 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:54:17.544 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:54:17.567 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:17.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:17.568 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:54:17.669 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:54:17.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:17.691 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:17.691 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:54:17.742 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:17 smithi154 ceph-mon[33264]: 7.5 scrub starts 2024-09-20T22:54:17.742 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:17 smithi154 ceph-mon[33264]: 7.5 scrub ok 2024-09-20T22:54:17.742 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:17 smithi154 ceph-mon[33264]: pgmap v1065: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 529 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 4.5 MiB/s rd, 5.6 MiB/s wr, 18 op/s 2024-09-20T22:54:17.742 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:17 smithi154 ceph-mon[33264]: 5.19 scrub starts 2024-09-20T22:54:17.742 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:17 smithi154 ceph-mon[33264]: 5.19 scrub ok 2024-09-20T22:54:17.742 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:17 smithi154 ceph-mon[33264]: 2.1 scrub starts 2024-09-20T22:54:17.742 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:17 smithi154 ceph-mon[33264]: 2.1 scrub ok 2024-09-20T22:54:17.742 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:17 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/296817853' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:54:17.743 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:17 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:54:17.743 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:17 smithi154 ceph-mon[33264]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T22:54:17.743 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:17 smithi154 ceph-mon[33264]: Cluster is now healthy 2024-09-20T22:54:17.743 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:17 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T22:54:17.743 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:17 smithi154 ceph-mon[33264]: osdmap e777: 8 total, 8 up, 8 in 2024-09-20T22:54:17.773 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:17 smithi084 ceph-mon[40024]: 7.5 scrub starts 2024-09-20T22:54:17.773 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:17 smithi084 ceph-mon[40024]: 7.5 scrub ok 2024-09-20T22:54:17.773 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:17 smithi084 ceph-mon[40024]: pgmap v1065: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 529 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 4.5 MiB/s rd, 5.6 MiB/s wr, 18 op/s 2024-09-20T22:54:17.773 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:17 smithi084 ceph-mon[40024]: 5.19 scrub starts 2024-09-20T22:54:17.773 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:17 smithi084 ceph-mon[40024]: 5.19 scrub ok 2024-09-20T22:54:17.773 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:17 smithi084 ceph-mon[40024]: 2.1 scrub starts 2024-09-20T22:54:17.773 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:17 smithi084 ceph-mon[40024]: 2.1 scrub ok 2024-09-20T22:54:17.774 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:17 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/296817853' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:54:17.774 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:17 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:54:17.774 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:17 smithi084 ceph-mon[40024]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T22:54:17.774 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:17 smithi084 ceph-mon[40024]: Cluster is now healthy 2024-09-20T22:54:17.774 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:17 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T22:54:17.774 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:17 smithi084 ceph-mon[40024]: osdmap e777: 8 total, 8 up, 8 in 2024-09-20T22:54:17.774 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:17 smithi084 ceph-mon[29774]: 7.5 scrub starts 2024-09-20T22:54:17.774 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:17 smithi084 ceph-mon[29774]: 7.5 scrub ok 2024-09-20T22:54:17.774 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:17 smithi084 ceph-mon[29774]: pgmap v1065: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 529 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 4.5 MiB/s rd, 5.6 MiB/s wr, 18 op/s 2024-09-20T22:54:17.774 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:17 smithi084 ceph-mon[29774]: 5.19 scrub starts 2024-09-20T22:54:17.774 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:17 smithi084 ceph-mon[29774]: 5.19 scrub ok 2024-09-20T22:54:17.775 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:17 smithi084 ceph-mon[29774]: 2.1 scrub starts 2024-09-20T22:54:17.775 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:17 smithi084 ceph-mon[29774]: 2.1 scrub ok 2024-09-20T22:54:17.775 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:17 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/296817853' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:54:17.775 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:17 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:54:17.775 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:17 smithi084 ceph-mon[29774]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T22:54:17.775 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:17 smithi084 ceph-mon[29774]: Cluster is now healthy 2024-09-20T22:54:17.775 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:17 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T22:54:17.775 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:17 smithi084 ceph-mon[29774]: osdmap e777: 8 total, 8 up, 8 in 2024-09-20T22:54:17.792 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:54:17.813 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:17.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:17.814 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:54:17.870 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:54:17.870 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:54:17.886 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:54:17.886 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:54:17.902 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:54:17.903 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:54:17.915 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:54:17.919 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-20T22:54:17.920 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T22:54:17.937 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:17.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:17.938 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:54:17.939 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:54:17.939 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:54:17.955 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:54:17.955 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:54:17.971 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:54:17.971 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:54:17.988 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:54:17.988 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:54:18.015 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-20T22:54:18.015 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:54:18.039 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:54:18.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:18.061 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:18.061 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:54:18.162 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:54:18.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:18.185 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:18.185 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:54:18.286 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:54:18.306 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:18.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:18.307 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:54:18.310 INFO:tasks.rados.rados.0.smithi084.stdout:3093: done (2 left) 2024-09-20T22:54:18.310 INFO:tasks.rados.rados.0.smithi084.stdout:3094: delete oid 26 current snap is 350 2024-09-20T22:54:18.310 INFO:tasks.rados.rados.0.smithi084.stdout:3091: expect (ObjNum 964 snap 314 seq_num 964) 2024-09-20T22:54:18.343 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:54:18 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: ::ffff:172.21.15.154 - - [20/Sep/2024:22:54:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-20T22:54:18.408 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:54:18.429 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:18.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:18.430 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:54:18.462 INFO:tasks.rados.rados.0.smithi084.stdout:3091: done (2 left) 2024-09-20T22:54:18.462 INFO:tasks.rados.rados.0.smithi084.stdout:3094: done (1 left) 2024-09-20T22:54:18.462 INFO:tasks.rados.rados.0.smithi084.stdout:3095: snap_create 2024-09-20T22:54:18.463 INFO:tasks.rados.rados.0.smithi084.stdout:3092: finishing copy_from racing read to smithi08477203-28 2024-09-20T22:54:18.463 INFO:tasks.rados.rados.0.smithi084.stdout:3092: finishing copy_from to smithi08477203-28 2024-09-20T22:54:18.463 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 28 v 1829 (ObjNum 984 snap 319 seq_num 984) dirty exists 2024-09-20T22:54:18.530 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:54:18.552 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:18.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:18.553 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:54:18.614 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:18 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/3443025897' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:54:18.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:18 smithi084 ceph-mon[40024]: osdmap e778: 8 total, 8 up, 8 in 2024-09-20T22:54:18.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:18 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/3443025897' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:54:18.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:18 smithi084 ceph-mon[29774]: osdmap e778: 8 total, 8 up, 8 in 2024-09-20T22:54:18.615 INFO:journalctl@ceph.rgw.foo.a.smithi084.stdout:Sep 20 22:54:18 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-rgw-foo-a[66081]: 2024-09-20T22:54:18.341+0000 7f94f1bf5640 -1 rgw watcher librados: RGWWatcher::handle_error cookie 94843898684416 err (110) Connection timed out 2024-09-20T22:54:18.654 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:54:18.675 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:18.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:18.676 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:54:18.745 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:18 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/3443025897' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:54:18.745 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:18 smithi154 ceph-mon[33264]: osdmap e778: 8 total, 8 up, 8 in 2024-09-20T22:54:18.776 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:54:18.797 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:18.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:18.799 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:54:18.899 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:54:18.906 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-20T22:54:18.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:18.921 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:18.921 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:54:19.022 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:54:19.043 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:19.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:19.044 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:54:19.144 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:54:19.166 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:19.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:19.167 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:54:19.167 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:54:19.268 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:54:19.289 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:19.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:19.289 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:54:19.390 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:54:19.406 INFO:tasks.rados.rados.0.smithi084.stdout:3092: done (1 left) 2024-09-20T22:54:19.407 INFO:tasks.rados.rados.0.smithi084.stdout:3095: done (0 left) 2024-09-20T22:54:19.407 INFO:tasks.rados.rados.0.smithi084.stdout:3096: read oid 42 snap -1 2024-09-20T22:54:19.407 INFO:tasks.rados.rados.0.smithi084.stdout:3096: expect (ObjNum 957 snap 313 seq_num 957) 2024-09-20T22:54:19.407 INFO:tasks.rados.rados.0.smithi084.stdout:3097: read oid 10 snap -1 2024-09-20T22:54:19.407 INFO:tasks.rados.rados.0.smithi084.stdout:3097: expect deleted 2024-09-20T22:54:19.407 INFO:tasks.rados.rados.0.smithi084.stdout:3098: write oid 19 current snap is 351 2024-09-20T22:54:19.407 INFO:tasks.rados.rados.0.smithi084.stdout:3098: seq_num 1065 ranges {699455=429605,1804617=576034,3163252=514929} 2024-09-20T22:54:19.413 INFO:tasks.rados.rados.0.smithi084.stdout:3098: writing smithi08477203-19 from 699455 to 1129060 tid 1 2024-09-20T22:54:19.417 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:19.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:19.418 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:54:19.420 INFO:tasks.rados.rados.0.smithi084.stdout:3098: writing smithi08477203-19 from 1804617 to 2380651 tid 2 2024-09-20T22:54:19.428 INFO:tasks.rados.rados.0.smithi084.stdout:3098: writing smithi08477203-19 from 3163252 to 3678181 tid 3 2024-09-20T22:54:19.428 INFO:tasks.rados.rados.0.smithi084.stdout:3099: rollback oid 1 current snap is 351 2024-09-20T22:54:19.428 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 1 to 347 2024-09-20T22:54:19.428 INFO:tasks.rados.rados.0.smithi084.stdout:3098: finishing write tid 1 to smithi08477203-19 2024-09-20T22:54:19.428 INFO:tasks.rados.rados.0.smithi084.stdout:3100: read oid 3 snap -1 2024-09-20T22:54:19.428 INFO:tasks.rados.rados.0.smithi084.stdout:3100: expect (ObjNum 1053 snap 345 seq_num 1053) 2024-09-20T22:54:19.428 INFO:tasks.rados.rados.0.smithi084.stdout:3101: read oid 45 snap -1 2024-09-20T22:54:19.428 INFO:tasks.rados.rados.0.smithi084.stdout:3101: expect (ObjNum 1025 snap 333 seq_num 1025) 2024-09-20T22:54:19.429 INFO:tasks.rados.rados.0.smithi084.stdout:3102: snap_create 2024-09-20T22:54:19.429 INFO:tasks.rados.rados.0.smithi084.stdout:3098: finishing write tid 2 to smithi08477203-19 2024-09-20T22:54:19.431 INFO:tasks.rados.rados.0.smithi084.stdout:3098: finishing write tid 3 to smithi08477203-19 2024-09-20T22:54:19.431 INFO:tasks.rados.rados.0.smithi084.stdout:3098: finishing write tid 4 to smithi08477203-19 2024-09-20T22:54:19.432 INFO:tasks.rados.rados.0.smithi084.stdout:3098: finishing write tid 5 to smithi08477203-19 2024-09-20T22:54:19.432 INFO:tasks.rados.rados.0.smithi084.stdout:3098: oid 19 updating version 0 to 2518 2024-09-20T22:54:19.432 INFO:tasks.rados.rados.0.smithi084.stdout:3098: oid 19 version 2518 is already newer than 2515 2024-09-20T22:54:19.432 INFO:tasks.rados.rados.0.smithi084.stdout:3098: oid 19 version 2518 is already newer than 2517 2024-09-20T22:54:19.432 INFO:tasks.rados.rados.0.smithi084.stdout:3098: oid 19 version 2518 is already newer than 2516 2024-09-20T22:54:19.432 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 19 v 2518 (ObjNum 1065 snap 351 seq_num 1065) dirty exists 2024-09-20T22:54:19.432 INFO:tasks.rados.rados.0.smithi084.stdout:3098: left oid 19 (ObjNum 1065 snap 351 seq_num 1065) 2024-09-20T22:54:19.519 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:54:19.541 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:19.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:19.542 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:54:19.567 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:19 smithi084 ceph-mon[29774]: pgmap v1068: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 529 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail 2024-09-20T22:54:19.567 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:19 smithi084 ceph-mon[29774]: 4.1d scrub starts 2024-09-20T22:54:19.567 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:19 smithi084 ceph-mon[29774]: 4.1d scrub ok 2024-09-20T22:54:19.568 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:19 smithi084 ceph-mon[29774]: 1.3 starting backfill to osd.1 from (0'0,0'0] MAX to 77'128 2024-09-20T22:54:19.568 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:19 smithi084 ceph-mon[29774]: 7.0 starting backfill to osd.1 from (0'0,0'0] MAX to 775'756 2024-09-20T22:54:19.568 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:19 smithi084 ceph-mon[29774]: 7.9 starting backfill to osd.1 from (0'0,0'0] MAX to 774'1348 2024-09-20T22:54:19.568 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:19 smithi084 ceph-mon[29774]: 7.4 starting backfill to osd.1 from (0'0,0'0] MAX to 774'550 2024-09-20T22:54:19.568 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:19 smithi084 ceph-mon[29774]: 7.2 starting backfill to osd.1 from (0'0,0'0] MAX to 774'819 2024-09-20T22:54:19.568 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:19 smithi084 ceph-mon[40024]: pgmap v1068: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 529 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail 2024-09-20T22:54:19.568 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:19 smithi084 ceph-mon[40024]: 4.1d scrub starts 2024-09-20T22:54:19.568 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:19 smithi084 ceph-mon[40024]: 4.1d scrub ok 2024-09-20T22:54:19.568 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:19 smithi084 ceph-mon[40024]: 1.3 starting backfill to osd.1 from (0'0,0'0] MAX to 77'128 2024-09-20T22:54:19.568 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:19 smithi084 ceph-mon[40024]: 7.0 starting backfill to osd.1 from (0'0,0'0] MAX to 775'756 2024-09-20T22:54:19.568 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:19 smithi084 ceph-mon[40024]: 7.9 starting backfill to osd.1 from (0'0,0'0] MAX to 774'1348 2024-09-20T22:54:19.569 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:19 smithi084 ceph-mon[40024]: 7.4 starting backfill to osd.1 from (0'0,0'0] MAX to 774'550 2024-09-20T22:54:19.569 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:19 smithi084 ceph-mon[40024]: 7.2 starting backfill to osd.1 from (0'0,0'0] MAX to 774'819 2024-09-20T22:54:19.642 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:54:19.663 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:19.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:19.664 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:54:19.667 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-20T22:54:19.667 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:54:19.754 INFO:journalctl@ceph.iscsi.iscsi.a.smithi154.stdout:Sep 20 22:54:19 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-iscsi-iscsi-a[59730]: debug there is no tcmu-runner data available 2024-09-20T22:54:19.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:19 smithi154 ceph-mon[33264]: pgmap v1068: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 529 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail 2024-09-20T22:54:19.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:19 smithi154 ceph-mon[33264]: 4.1d scrub starts 2024-09-20T22:54:19.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:19 smithi154 ceph-mon[33264]: 4.1d scrub ok 2024-09-20T22:54:19.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:19 smithi154 ceph-mon[33264]: 1.3 starting backfill to osd.1 from (0'0,0'0] MAX to 77'128 2024-09-20T22:54:19.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:19 smithi154 ceph-mon[33264]: 7.0 starting backfill to osd.1 from (0'0,0'0] MAX to 775'756 2024-09-20T22:54:19.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:19 smithi154 ceph-mon[33264]: 7.9 starting backfill to osd.1 from (0'0,0'0] MAX to 774'1348 2024-09-20T22:54:19.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:19 smithi154 ceph-mon[33264]: 7.4 starting backfill to osd.1 from (0'0,0'0] MAX to 774'550 2024-09-20T22:54:19.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:19 smithi154 ceph-mon[33264]: 7.2 starting backfill to osd.1 from (0'0,0'0] MAX to 774'819 2024-09-20T22:54:19.765 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:54:19.787 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:19.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:19.788 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:54:19.889 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:54:19.911 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:19.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:19.912 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:54:20.013 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:54:20.034 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:20.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:20.035 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:54:20.136 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:54:20.158 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:20.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:20.159 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:54:20.260 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:54:20.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:20.283 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:20.283 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:54:20.384 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:54:20.406 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:20.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:20.407 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:54:20.413 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.0 dump_blocked_ops 2024-09-20T22:54:20.416 INFO:tasks.rados.rados.0.smithi084.stdout:3098: done (6 left) 2024-09-20T22:54:20.417 INFO:tasks.rados.rados.0.smithi084.stdout:3102: done (5 left) 2024-09-20T22:54:20.417 INFO:tasks.rados.rados.0.smithi084.stdout:3103: snap_create 2024-09-20T22:54:20.423 INFO:tasks.rados.rados.0.smithi084.stdout:3100: expect (ObjNum 1053 snap 345 seq_num 1053) 2024-09-20T22:54:20.458 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd set noscrub 2024-09-20T22:54:20.508 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:54:20.528 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:20.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:20.529 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:54:20.552 INFO:tasks.rados.rados.0.smithi084.stdout:3096: expect (ObjNum 957 snap 313 seq_num 957) 2024-09-20T22:54:20.630 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:54:20.640 INFO:tasks.rados.rados.0.smithi084.stdout:3099: finishing rollback tid 0 to smithi08477203-1 2024-09-20T22:54:20.640 INFO:tasks.rados.rados.0.smithi084.stdout:3099: finishing rollback tid 1 to smithi08477203-1 2024-09-20T22:54:20.640 INFO:tasks.rados.rados.0.smithi084.stdout:3099: finishing rollback tid 2 to smithi08477203-1 2024-09-20T22:54:20.640 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 1 v 1834 (ObjNum 738 snap 235 seq_num 738) dirty exists 2024-09-20T22:54:20.640 INFO:tasks.rados.rados.0.smithi084.stdout:3101: expect (ObjNum 1025 snap 333 seq_num 1025) 2024-09-20T22:54:20.651 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:20.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:20.652 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:54:20.679 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:54:20.752 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:54:20.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:20 smithi154 ceph-mon[33264]: osdmap e779: 8 total, 8 up, 8 in 2024-09-20T22:54:20.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:20 smithi154 ceph-mon[33264]: 7.e scrub starts 2024-09-20T22:54:20.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:20 smithi154 ceph-mon[33264]: 7.e scrub ok 2024-09-20T22:54:20.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:20 smithi154 ceph-mon[33264]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:54:20.773 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:20.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:20.774 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:54:20.802 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:20 smithi084 ceph-mon[40024]: osdmap e779: 8 total, 8 up, 8 in 2024-09-20T22:54:20.802 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:20 smithi084 ceph-mon[40024]: 7.e scrub starts 2024-09-20T22:54:20.803 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:20 smithi084 ceph-mon[40024]: 7.e scrub ok 2024-09-20T22:54:20.803 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:20 smithi084 ceph-mon[40024]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:54:20.803 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:20 smithi084 ceph-mon[29774]: osdmap e779: 8 total, 8 up, 8 in 2024-09-20T22:54:20.803 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:20 smithi084 ceph-mon[29774]: 7.e scrub starts 2024-09-20T22:54:20.803 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:20 smithi084 ceph-mon[29774]: 7.e scrub ok 2024-09-20T22:54:20.803 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:20 smithi084 ceph-mon[29774]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:54:20.875 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:54:20.897 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:20.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:20.898 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:54:20.951 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:54:20.999 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:54:21.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:21.021 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:21.021 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:54:21.069 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-20T22:54:21.122 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:54:21.144 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:21.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:21.145 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:54:21.175 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-20T22:54:21.176 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:54:21.246 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:54:21.267 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:21.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:21.268 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:54:21.369 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:54:21.390 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:21.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:21.391 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:54:21.411 INFO:tasks.rados.rados.0.smithi084.stdout:3096: done (5 left) 2024-09-20T22:54:21.412 INFO:tasks.rados.rados.0.smithi084.stdout:3097: done (4 left) 2024-09-20T22:54:21.412 INFO:tasks.rados.rados.0.smithi084.stdout:3099: done (3 left) 2024-09-20T22:54:21.412 INFO:tasks.rados.rados.0.smithi084.stdout:3100: done (2 left) 2024-09-20T22:54:21.412 INFO:tasks.rados.rados.0.smithi084.stdout:3101: done (1 left) 2024-09-20T22:54:21.412 INFO:tasks.rados.rados.0.smithi084.stdout:3103: done (0 left) 2024-09-20T22:54:21.412 INFO:tasks.rados.rados.0.smithi084.stdout:3104: write (excl) oid 34 current snap is 353 2024-09-20T22:54:21.412 INFO:tasks.rados.rados.0.smithi084.stdout:3104: seq_num 1066 ranges {716262=504854,1863099=719692,3074322=726201} 2024-09-20T22:54:21.418 INFO:tasks.rados.rados.0.smithi084.stdout:3104: writing smithi08477203-34 from 716262 to 1221116 tid 1 2024-09-20T22:54:21.426 INFO:tasks.rados.rados.0.smithi084.stdout:3104: writing smithi08477203-34 from 1863099 to 2582791 tid 2 2024-09-20T22:54:21.433 INFO:tasks.rados.rados.0.smithi084.stdout:3104: writing smithi08477203-34 from 3074322 to 3800523 tid 3 2024-09-20T22:54:21.433 INFO:tasks.rados.rados.0.smithi084.stdout:3105: snap_create 2024-09-20T22:54:21.433 INFO:tasks.rados.rados.0.smithi084.stdout:3104: finishing write tid 1 to smithi08477203-34 2024-09-20T22:54:21.433 INFO:tasks.rados.rados.0.smithi084.stdout:3104: finishing write tid 2 to smithi08477203-34 2024-09-20T22:54:21.435 INFO:tasks.rados.rados.0.smithi084.stdout:3104: finishing write tid 3 to smithi08477203-34 2024-09-20T22:54:21.436 INFO:tasks.rados.rados.0.smithi084.stdout:3104: finishing write tid 4 to smithi08477203-34 2024-09-20T22:54:21.436 INFO:tasks.rados.rados.0.smithi084.stdout:3104: finishing write tid 5 to smithi08477203-34 2024-09-20T22:54:21.436 INFO:tasks.rados.rados.0.smithi084.stdout:3104: oid 34 updating version 0 to 1837 2024-09-20T22:54:21.436 INFO:tasks.rados.rados.0.smithi084.stdout:3104: oid 34 updating version 1837 to 1839 2024-09-20T22:54:21.436 INFO:tasks.rados.rados.0.smithi084.stdout:3104: oid 34 version 1839 is already newer than 1838 2024-09-20T22:54:21.436 INFO:tasks.rados.rados.0.smithi084.stdout:3104: oid 34 version 1839 is already newer than 1836 2024-09-20T22:54:21.436 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 34 v 1839 (ObjNum 1066 snap 353 seq_num 1066) dirty exists 2024-09-20T22:54:21.436 INFO:tasks.rados.rados.0.smithi084.stdout:3104: left oid 34 (ObjNum 1066 snap 353 seq_num 1066) 2024-09-20T22:54:21.492 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:54:21.513 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:21.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:21.514 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:54:21.614 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:54:21.635 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:21.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:21.636 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:54:21.737 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:54:21.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:21 smithi154 ceph-mon[33264]: 5.1a scrub starts 2024-09-20T22:54:21.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:21 smithi154 ceph-mon[33264]: 5.1a scrub ok 2024-09-20T22:54:21.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:21 smithi154 ceph-mon[33264]: pgmap v1071: 158 pgs: 1 active+undersized+remapped, 1 active+recovering+remapped, 1 active+clean+remapped, 1 active+remapped+backfill_wait, 2 active+remapped+backfilling, 3 active+clean+snaptrim, 1 active+remapped+backfill_toofull, 148 active+clean; 528 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 9.2 MiB/s rd, 1.1 MiB/s wr, 8 op/s; 28/1311 objects misplaced (2.136%); 34 MiB/s, 70 objects/s recovering 2024-09-20T22:54:21.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:21 smithi154 ceph-mon[33264]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-09-20T22:54:21.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:21 smithi154 ceph-mon[33264]: osdmap e780: 8 total, 8 up, 8 in 2024-09-20T22:54:21.758 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:21.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:21.759 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:54:21.763 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:21 smithi084 ceph-mon[40024]: 5.1a scrub starts 2024-09-20T22:54:21.763 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:21 smithi084 ceph-mon[40024]: 5.1a scrub ok 2024-09-20T22:54:21.763 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:21 smithi084 ceph-mon[40024]: pgmap v1071: 158 pgs: 1 active+undersized+remapped, 1 active+recovering+remapped, 1 active+clean+remapped, 1 active+remapped+backfill_wait, 2 active+remapped+backfilling, 3 active+clean+snaptrim, 1 active+remapped+backfill_toofull, 148 active+clean; 528 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 9.2 MiB/s rd, 1.1 MiB/s wr, 8 op/s; 28/1311 objects misplaced (2.136%); 34 MiB/s, 70 objects/s recovering 2024-09-20T22:54:21.763 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:21 smithi084 ceph-mon[40024]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-09-20T22:54:21.763 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:21 smithi084 ceph-mon[40024]: osdmap e780: 8 total, 8 up, 8 in 2024-09-20T22:54:21.764 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:21 smithi084 ceph-mon[29774]: 5.1a scrub starts 2024-09-20T22:54:21.764 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:21 smithi084 ceph-mon[29774]: 5.1a scrub ok 2024-09-20T22:54:21.764 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:21 smithi084 ceph-mon[29774]: pgmap v1071: 158 pgs: 1 active+undersized+remapped, 1 active+recovering+remapped, 1 active+clean+remapped, 1 active+remapped+backfill_wait, 2 active+remapped+backfilling, 3 active+clean+snaptrim, 1 active+remapped+backfill_toofull, 148 active+clean; 528 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 9.2 MiB/s rd, 1.1 MiB/s wr, 8 op/s; 28/1311 objects misplaced (2.136%); 34 MiB/s, 70 objects/s recovering 2024-09-20T22:54:21.764 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:21 smithi084 ceph-mon[29774]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-09-20T22:54:21.764 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:21 smithi084 ceph-mon[29774]: osdmap e780: 8 total, 8 up, 8 in 2024-09-20T22:54:21.789 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.0 dump_historic_ops 2024-09-20T22:54:21.860 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:54:21.882 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:21.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:21.883 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:54:21.984 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:54:22.005 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:22.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:22.006 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:54:22.107 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:54:22.128 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:22.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:22.129 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:54:22.136 INFO:tasks.thrashosds.thrasher:in_osds: [2, 3, 5, 6, 7, 0, 4, 1] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-20T22:54:22.136 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-20T22:54:22.136 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-09-20T22:54:22.137 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd dump -f json-pretty 2024-09-20T22:54:22.230 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:54:22.251 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:22.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:22.252 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:54:22.352 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:54:22.374 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:22.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:22.375 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:54:22.418 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:22 smithi084 ceph-mon[29774]: 5.1b scrub starts 2024-09-20T22:54:22.418 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:22 smithi084 ceph-mon[29774]: 5.1b scrub ok 2024-09-20T22:54:22.418 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:22 smithi084 ceph-mon[29774]: osdmap e781: 8 total, 8 up, 8 in 2024-09-20T22:54:22.427 INFO:tasks.rados.rados.0.smithi084.stdout:3104: done (1 left) 2024-09-20T22:54:22.427 INFO:tasks.rados.rados.0.smithi084.stdout:3105: done (0 left) 2024-09-20T22:54:22.427 INFO:tasks.rados.rados.0.smithi084.stdout:3106: read oid 21 snap 337 2024-09-20T22:54:22.427 INFO:tasks.rados.rados.0.smithi084.stdout:3106: expect (ObjNum 984 snap 319 seq_num 984) 2024-09-20T22:54:22.427 INFO:tasks.rados.rados.0.smithi084.stdout:3107: write oid 14 current snap is 354 2024-09-20T22:54:22.427 INFO:tasks.rados.rados.0.smithi084.stdout:3107: seq_num 1067 ranges {733069=580103,1921581=579703,3218098=537473} 2024-09-20T22:54:22.434 INFO:tasks.rados.rados.0.smithi084.stdout:3107: writing smithi08477203-14 from 733069 to 1313172 tid 1 2024-09-20T22:54:22.441 INFO:tasks.rados.rados.0.smithi084.stdout:3107: writing smithi08477203-14 from 1921581 to 2501284 tid 2 2024-09-20T22:54:22.448 INFO:tasks.rados.rados.0.smithi084.stdout:3107: writing smithi08477203-14 from 3218098 to 3755571 tid 3 2024-09-20T22:54:22.448 INFO:tasks.rados.rados.0.smithi084.stdout:3108: rollback oid 9 current snap is 354 2024-09-20T22:54:22.448 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 9 to 341 2024-09-20T22:54:22.448 INFO:tasks.rados.rados.0.smithi084.stdout:3106: expect (ObjNum 984 snap 319 seq_num 984) 2024-09-20T22:54:22.476 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:54:22.497 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:22.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:22.498 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:54:22.538 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:54:22.546 INFO:tasks.rados.rados.0.smithi084.stdout:3107: finishing write tid 1 to smithi08477203-14 2024-09-20T22:54:22.546 INFO:tasks.rados.rados.0.smithi084.stdout:3107: finishing write tid 2 to smithi08477203-14 2024-09-20T22:54:22.546 INFO:tasks.rados.rados.0.smithi084.stdout:3106: done (2 left) 2024-09-20T22:54:22.546 INFO:tasks.rados.rados.0.smithi084.stdout:3109: rollback oid 47 current snap is 354 2024-09-20T22:54:22.546 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 47 to 352 2024-09-20T22:54:22.546 INFO:tasks.rados.rados.0.smithi084.stdout:3108: finishing rollback tid 1 to smithi08477203-9 2024-09-20T22:54:22.546 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 9 v 0 (ObjNum 129 snap 0 seq_num 2091283552) dirty dne 2024-09-20T22:54:22.546 INFO:tasks.rados.rados.0.smithi084.stdout:3107: finishing write tid 3 to smithi08477203-14 2024-09-20T22:54:22.546 INFO:tasks.rados.rados.0.smithi084.stdout:3107: finishing write tid 4 to smithi08477203-14 2024-09-20T22:54:22.547 INFO:tasks.rados.rados.0.smithi084.stdout:3107: finishing write tid 5 to smithi08477203-14 2024-09-20T22:54:22.547 INFO:tasks.rados.rados.0.smithi084.stdout:3107: oid 14 updating version 0 to 1914 2024-09-20T22:54:22.547 INFO:tasks.rados.rados.0.smithi084.stdout:3107: oid 14 updating version 1914 to 1917 2024-09-20T22:54:22.547 INFO:tasks.rados.rados.0.smithi084.stdout:3107: oid 14 version 1917 is already newer than 1915 2024-09-20T22:54:22.547 INFO:tasks.rados.rados.0.smithi084.stdout:3107: oid 14 version 1917 is already newer than 1916 2024-09-20T22:54:22.547 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 14 v 1917 (ObjNum 1067 snap 354 seq_num 1067) dirty exists 2024-09-20T22:54:22.547 INFO:tasks.rados.rados.0.smithi084.stdout:3107: left oid 14 (ObjNum 1067 snap 354 seq_num 1067) 2024-09-20T22:54:22.547 INFO:tasks.rados.rados.0.smithi084.stdout:3107: done (2 left) 2024-09-20T22:54:22.547 INFO:tasks.rados.rados.0.smithi084.stdout:3108: done (1 left) 2024-09-20T22:54:22.547 INFO:tasks.rados.rados.0.smithi084.stdout:3110: write oid 31 current snap is 354 2024-09-20T22:54:22.547 INFO:tasks.rados.rados.0.smithi084.stdout:3110: seq_num 1068 ranges {749876=655352,1863710=723361,3012815=748745} 2024-09-20T22:54:22.554 INFO:tasks.rados.rados.0.smithi084.stdout:3110: writing smithi08477203-31 from 749876 to 1405228 tid 1 2024-09-20T22:54:22.561 INFO:tasks.rados.rados.0.smithi084.stdout:3110: writing smithi08477203-31 from 1863710 to 2587071 tid 2 2024-09-20T22:54:22.569 INFO:tasks.rados.rados.0.smithi084.stdout:3110: writing smithi08477203-31 from 3012815 to 3761560 tid 3 2024-09-20T22:54:22.570 INFO:tasks.rados.rados.0.smithi084.stdout:3111: rollback oid 8 current snap is 354 2024-09-20T22:54:22.570 INFO:tasks.rados.rados.0.smithi084.stdout:3109: finishing rollback tid 0 to smithi08477203-47 2024-09-20T22:54:22.570 INFO:tasks.rados.rados.0.smithi084.stdout:3109: finishing rollback tid 1 to smithi08477203-47 2024-09-20T22:54:22.570 INFO:tasks.rados.rados.0.smithi084.stdout:3109: finishing rollback tid 2 to smithi08477203-47 2024-09-20T22:54:22.570 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 47 v 1842 (ObjNum 1063 snap 349 seq_num 1063) dirty exists 2024-09-20T22:54:22.570 INFO:tasks.rados.rados.0.smithi084.stdout:3110: finishing write tid 1 to smithi08477203-31 2024-09-20T22:54:22.570 INFO:tasks.rados.rados.0.smithi084.stdout:3110: finishing write tid 2 to smithi08477203-31 2024-09-20T22:54:22.571 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 8 to 349 2024-09-20T22:54:22.571 INFO:tasks.rados.rados.0.smithi084.stdout:3109: done (2 left) 2024-09-20T22:54:22.571 INFO:tasks.rados.rados.0.smithi084.stdout:3112: read oid 23 snap -1 2024-09-20T22:54:22.571 INFO:tasks.rados.rados.0.smithi084.stdout:3112: expect (ObjNum 984 snap 319 seq_num 984) 2024-09-20T22:54:22.571 INFO:tasks.rados.rados.0.smithi084.stdout:3113: snap_create 2024-09-20T22:54:22.576 INFO:tasks.rados.rados.0.smithi084.stdout:3110: finishing write tid 3 to smithi08477203-31 2024-09-20T22:54:22.576 INFO:tasks.rados.rados.0.smithi084.stdout:3111: finishing rollback tid 1 to smithi08477203-8 2024-09-20T22:54:22.576 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 8 v 1853 (ObjNum 145 snap 0 seq_num 2091900144) dirty dne 2024-09-20T22:54:22.576 INFO:tasks.rados.rados.0.smithi084.stdout:3110: finishing write tid 4 to smithi08477203-31 2024-09-20T22:54:22.576 INFO:tasks.rados.rados.0.smithi084.stdout:3110: finishing write tid 5 to smithi08477203-31 2024-09-20T22:54:22.576 INFO:tasks.rados.rados.0.smithi084.stdout:3110: oid 31 updating version 0 to 1844 2024-09-20T22:54:22.576 INFO:tasks.rados.rados.0.smithi084.stdout:3110: oid 31 updating version 1844 to 1845 2024-09-20T22:54:22.576 INFO:tasks.rados.rados.0.smithi084.stdout:3110: oid 31 version 1845 is already newer than 1843 2024-09-20T22:54:22.576 INFO:tasks.rados.rados.0.smithi084.stdout:3110: oid 31 updating version 1845 to 1846 2024-09-20T22:54:22.576 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 31 v 1846 (ObjNum 1068 snap 354 seq_num 1068) dirty exists 2024-09-20T22:54:22.576 INFO:tasks.rados.rados.0.smithi084.stdout:3110: left oid 31 (ObjNum 1068 snap 354 seq_num 1068) 2024-09-20T22:54:22.577 INFO:tasks.rados.rados.0.smithi084.stdout:3112: expect (ObjNum 984 snap 319 seq_num 984) 2024-09-20T22:54:22.599 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:54:22.621 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:22.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:22.622 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:54:22.685 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:22 smithi084 ceph-mon[29774]: 3.9 scrub starts 2024-09-20T22:54:22.685 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:22 smithi084 ceph-mon[29774]: 3.9 scrub ok 2024-09-20T22:54:22.685 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:22 smithi084 ceph-mon[29774]: 5.b deep-scrub starts 2024-09-20T22:54:22.685 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:22 smithi084 ceph-mon[29774]: 5.b deep-scrub ok 2024-09-20T22:54:22.685 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:22 smithi084 ceph-mon[29774]: 1.6 scrub starts 2024-09-20T22:54:22.685 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:22 smithi084 ceph-mon[29774]: 1.6 scrub ok 2024-09-20T22:54:22.685 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:22 smithi084 ceph-mon[40024]: 5.1b scrub starts 2024-09-20T22:54:22.685 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:22 smithi084 ceph-mon[40024]: 5.1b scrub ok 2024-09-20T22:54:22.685 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:22 smithi084 ceph-mon[40024]: osdmap e781: 8 total, 8 up, 8 in 2024-09-20T22:54:22.685 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:22 smithi084 ceph-mon[40024]: 3.9 scrub starts 2024-09-20T22:54:22.685 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:22 smithi084 ceph-mon[40024]: 3.9 scrub ok 2024-09-20T22:54:22.686 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:22 smithi084 ceph-mon[40024]: 5.b deep-scrub starts 2024-09-20T22:54:22.686 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:22 smithi084 ceph-mon[40024]: 5.b deep-scrub ok 2024-09-20T22:54:22.686 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:22 smithi084 ceph-mon[40024]: 1.6 scrub starts 2024-09-20T22:54:22.686 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:22 smithi084 ceph-mon[40024]: 1.6 scrub ok 2024-09-20T22:54:22.722 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:54:22.743 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:22.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:22.744 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:54:22.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:22 smithi154 ceph-mon[33264]: 5.1b scrub starts 2024-09-20T22:54:22.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:22 smithi154 ceph-mon[33264]: 5.1b scrub ok 2024-09-20T22:54:22.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:22 smithi154 ceph-mon[33264]: osdmap e781: 8 total, 8 up, 8 in 2024-09-20T22:54:22.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:22 smithi154 ceph-mon[33264]: 3.9 scrub starts 2024-09-20T22:54:22.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:22 smithi154 ceph-mon[33264]: 3.9 scrub ok 2024-09-20T22:54:22.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:22 smithi154 ceph-mon[33264]: 5.b deep-scrub starts 2024-09-20T22:54:22.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:22 smithi154 ceph-mon[33264]: 5.b deep-scrub ok 2024-09-20T22:54:22.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:22 smithi154 ceph-mon[33264]: 1.6 scrub starts 2024-09-20T22:54:22.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:22 smithi154 ceph-mon[33264]: 1.6 scrub ok 2024-09-20T22:54:22.826 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:54:22.845 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:54:22.866 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:22.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:22.867 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:54:22.968 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:54:22.989 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:22.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:22.990 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:54:23.091 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:54:23.113 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:23.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:23.114 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:54:23.171 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:54:23.171 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:54:23.171 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:54:23.171 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-20T22:54:23.171 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:54:23.172 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:54:23.172 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:54:23.172 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:54:23.172 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T22:54:23.172 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:54:23.172 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:54:23.172 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:54:23.172 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:54:23.173 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:54:23.173 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:54:23.173 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:54:23.214 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:54:23.235 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:23.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:23.236 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:54:23.312 INFO:tasks.rados.rados.0.smithi084.stdout:3110: done (3 left) 2024-09-20T22:54:23.312 INFO:tasks.rados.rados.0.smithi084.stdout:3111: done (2 left) 2024-09-20T22:54:23.312 INFO:tasks.rados.rados.0.smithi084.stdout:3112: done (1 left) 2024-09-20T22:54:23.312 INFO:tasks.rados.rados.0.smithi084.stdout:3113: done (0 left) 2024-09-20T22:54:23.313 INFO:tasks.rados.rados.0.smithi084.stdout:3114: rollback oid 5 current snap is 355 2024-09-20T22:54:23.313 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 5 to 344 2024-09-20T22:54:23.313 INFO:tasks.rados.rados.0.smithi084.stdout:3115: write (excl) oid 11 current snap is 355 2024-09-20T22:54:23.313 INFO:tasks.rados.rados.0.smithi084.stdout:3115: seq_num 1069 ranges {766683=730601,1922192=583372,3156591=676370} 2024-09-20T22:54:23.313 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is set 2024-09-20T22:54:23.320 INFO:tasks.rados.rados.0.smithi084.stdout:3115: writing smithi08477203-11 from 766683 to 1497284 tid 1 2024-09-20T22:54:23.326 INFO:tasks.rados.rados.0.smithi084.stdout:3115: writing smithi08477203-11 from 1922192 to 2505564 tid 2 2024-09-20T22:54:23.334 INFO:tasks.rados.rados.0.smithi084.stdout:3115: writing smithi08477203-11 from 3156591 to 3832961 tid 3 2024-09-20T22:54:23.334 INFO:tasks.rados.rados.0.smithi084.stdout:3116: rollback oid 40 current snap is 355 2024-09-20T22:54:23.334 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 40 to 326 2024-09-20T22:54:23.334 INFO:tasks.rados.rados.0.smithi084.stdout:3114: finishing rollback tid 0 to smithi08477203-5 2024-09-20T22:54:23.334 INFO:tasks.rados.rados.0.smithi084.stdout:3114: finishing rollback tid 1 to smithi08477203-5 2024-09-20T22:54:23.334 INFO:tasks.rados.rados.0.smithi084.stdout:3114: finishing rollback tid 2 to smithi08477203-5 2024-09-20T22:54:23.334 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 5 v 1013 (ObjNum 1012 snap 333 seq_num 1012) dirty exists 2024-09-20T22:54:23.334 INFO:tasks.rados.rados.0.smithi084.stdout:3115: finishing write tid 1 to smithi08477203-11 2024-09-20T22:54:23.334 INFO:tasks.rados.rados.0.smithi084.stdout:3115: finishing write tid 2 to smithi08477203-11 2024-09-20T22:54:23.334 INFO:tasks.rados.rados.0.smithi084.stdout:3114: done (2 left) 2024-09-20T22:54:23.334 INFO:tasks.rados.rados.0.smithi084.stdout:3117: write oid 25 current snap is 355 2024-09-20T22:54:23.334 INFO:tasks.rados.rados.0.smithi084.stdout:3117: seq_num 1070 ranges {783490=405850,1980674=727030,3184014=487642} 2024-09-20T22:54:23.337 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:54:23.340 INFO:tasks.rados.rados.0.smithi084.stdout:3117: writing smithi08477203-25 from 783490 to 1189340 tid 1 2024-09-20T22:54:23.349 INFO:tasks.rados.rados.0.smithi084.stdout:3117: writing smithi08477203-25 from 1980674 to 2707704 tid 2 2024-09-20T22:54:23.355 INFO:tasks.rados.rados.0.smithi084.stdout:3117: writing smithi08477203-25 from 3184014 to 3671656 tid 3 2024-09-20T22:54:23.355 INFO:tasks.rados.rados.0.smithi084.stdout:3118: snap_create 2024-09-20T22:54:23.355 INFO:tasks.rados.rados.0.smithi084.stdout:3115: finishing write tid 3 to smithi08477203-11 2024-09-20T22:54:23.355 INFO:tasks.rados.rados.0.smithi084.stdout:3115: finishing write tid 4 to smithi08477203-11 2024-09-20T22:54:23.355 INFO:tasks.rados.rados.0.smithi084.stdout:3116: finishing rollback tid 0 to smithi08477203-40 2024-09-20T22:54:23.355 INFO:tasks.rados.rados.0.smithi084.stdout:3115: finishing write tid 5 to smithi08477203-11 2024-09-20T22:54:23.355 INFO:tasks.rados.rados.0.smithi084.stdout:3115: oid 11 updating version 0 to 1015 2024-09-20T22:54:23.355 INFO:tasks.rados.rados.0.smithi084.stdout:3115: oid 11 updating version 1015 to 1016 2024-09-20T22:54:23.355 INFO:tasks.rados.rados.0.smithi084.stdout:3115: oid 11 updating version 1016 to 1017 2024-09-20T22:54:23.356 INFO:tasks.rados.rados.0.smithi084.stdout:3115: oid 11 updating version 1017 to 1018 2024-09-20T22:54:23.356 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 11 v 1018 (ObjNum 1069 snap 355 seq_num 1069) dirty exists 2024-09-20T22:54:23.356 INFO:tasks.rados.rados.0.smithi084.stdout:3115: left oid 11 (ObjNum 1069 snap 355 seq_num 1069) 2024-09-20T22:54:23.356 INFO:tasks.rados.rados.0.smithi084.stdout:3116: finishing rollback tid 1 to smithi08477203-40 2024-09-20T22:54:23.356 INFO:tasks.rados.rados.0.smithi084.stdout:3116: finishing rollback tid 2 to smithi08477203-40 2024-09-20T22:54:23.356 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 40 v 1911 (ObjNum 970 snap 315 seq_num 970) dirty exists 2024-09-20T22:54:23.356 INFO:tasks.rados.rados.0.smithi084.stdout:3117: finishing write tid 1 to smithi08477203-25 2024-09-20T22:54:23.356 INFO:tasks.rados.rados.0.smithi084.stdout:3117: finishing write tid 2 to smithi08477203-25 2024-09-20T22:54:23.358 INFO:tasks.rados.rados.0.smithi084.stdout:3117: finishing write tid 3 to smithi08477203-25 2024-09-20T22:54:23.359 INFO:tasks.rados.rados.0.smithi084.stdout:3117: finishing write tid 4 to smithi08477203-25 2024-09-20T22:54:23.359 INFO:tasks.rados.rados.0.smithi084.stdout:3117: finishing write tid 5 to smithi08477203-25 2024-09-20T22:54:23.359 INFO:tasks.rados.rados.0.smithi084.stdout:3117: oid 25 updating version 0 to 2520 2024-09-20T22:54:23.359 INFO:tasks.rados.rados.0.smithi084.stdout:3117: oid 25 updating version 2520 to 2522 2024-09-20T22:54:23.359 INFO:tasks.rados.rados.0.smithi084.stdout:3117: oid 25 version 2522 is already newer than 2519 2024-09-20T22:54:23.359 INFO:tasks.rados.rados.0.smithi084.stdout:3117: oid 25 version 2522 is already newer than 2521 2024-09-20T22:54:23.359 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 25 v 2522 (ObjNum 1070 snap 355 seq_num 1070) dirty exists 2024-09-20T22:54:23.359 INFO:tasks.rados.rados.0.smithi084.stdout:3117: left oid 25 (ObjNum 1070 snap 355 seq_num 1070) 2024-09-20T22:54:23.360 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:23.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:23.361 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:54:23.461 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:54:23.482 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:23.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:23.483 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:54:23.567 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:23 smithi084 ceph-mon[40024]: 1.0 scrub starts 2024-09-20T22:54:23.568 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:23 smithi084 ceph-mon[40024]: 1.0 scrub ok 2024-09-20T22:54:23.568 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:23 smithi084 ceph-mon[40024]: pgmap v1074: 158 pgs: 1 active+undersized+remapped, 1 active+recovering+remapped, 1 active+clean+remapped, 1 active+remapped+backfill_wait, 2 active+remapped+backfilling, 3 active+clean+snaptrim, 1 active+remapped+backfill_toofull, 148 active+clean; 528 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 9.2 MiB/s rd, 1.1 MiB/s wr, 8 op/s; 28/1311 objects misplaced (2.136%); 34 MiB/s, 70 objects/s recovering 2024-09-20T22:54:23.568 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:23 smithi084 ceph-mon[40024]: 6.d scrub starts 2024-09-20T22:54:23.568 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:23 smithi084 ceph-mon[40024]: 6.d scrub ok 2024-09-20T22:54:23.568 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:23 smithi084 ceph-mon[40024]: 6.13 scrub starts 2024-09-20T22:54:23.568 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:23 smithi084 ceph-mon[40024]: 6.13 scrub ok 2024-09-20T22:54:23.568 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:23 smithi084 ceph-mon[40024]: osdmap e782: 8 total, 8 up, 8 in 2024-09-20T22:54:23.568 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:23 smithi084 ceph-mon[40024]: 5.e scrub starts 2024-09-20T22:54:23.568 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:23 smithi084 ceph-mon[40024]: 5.e scrub ok 2024-09-20T22:54:23.568 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:23 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/3439024040' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-20T22:54:23.568 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:23 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-20T22:54:23.568 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:23 smithi084 ceph-mon[40024]: 4.f scrub starts 2024-09-20T22:54:23.568 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:23 smithi084 ceph-mon[40024]: 4.f scrub ok 2024-09-20T22:54:23.569 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:23 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/2243256074' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:54:23.569 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:23 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:54:23.569 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:23 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:54:23.569 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:23 smithi084 ceph-mon[40024]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:54:23.569 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:23 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T22:54:23.569 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:23 smithi084 ceph-mon[40024]: osdmap e783: 8 total, 8 up, 8 in 2024-09-20T22:54:23.569 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:23 smithi084 ceph-mon[29774]: 1.0 scrub starts 2024-09-20T22:54:23.569 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:23 smithi084 ceph-mon[29774]: 1.0 scrub ok 2024-09-20T22:54:23.569 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:23 smithi084 ceph-mon[29774]: pgmap v1074: 158 pgs: 1 active+undersized+remapped, 1 active+recovering+remapped, 1 active+clean+remapped, 1 active+remapped+backfill_wait, 2 active+remapped+backfilling, 3 active+clean+snaptrim, 1 active+remapped+backfill_toofull, 148 active+clean; 528 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 9.2 MiB/s rd, 1.1 MiB/s wr, 8 op/s; 28/1311 objects misplaced (2.136%); 34 MiB/s, 70 objects/s recovering 2024-09-20T22:54:23.569 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:23 smithi084 ceph-mon[29774]: 6.d scrub starts 2024-09-20T22:54:23.569 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:23 smithi084 ceph-mon[29774]: 6.d scrub ok 2024-09-20T22:54:23.569 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:23 smithi084 ceph-mon[29774]: 6.13 scrub starts 2024-09-20T22:54:23.570 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:23 smithi084 ceph-mon[29774]: 6.13 scrub ok 2024-09-20T22:54:23.570 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:23 smithi084 ceph-mon[29774]: osdmap e782: 8 total, 8 up, 8 in 2024-09-20T22:54:23.570 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:23 smithi084 ceph-mon[29774]: 5.e scrub starts 2024-09-20T22:54:23.570 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:23 smithi084 ceph-mon[29774]: 5.e scrub ok 2024-09-20T22:54:23.570 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:23 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/3439024040' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-20T22:54:23.570 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:23 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-20T22:54:23.570 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:23 smithi084 ceph-mon[29774]: 4.f scrub starts 2024-09-20T22:54:23.570 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:23 smithi084 ceph-mon[29774]: 4.f scrub ok 2024-09-20T22:54:23.570 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:23 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/2243256074' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:54:23.570 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:23 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:54:23.570 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:23 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:54:23.570 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:23 smithi084 ceph-mon[29774]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:54:23.570 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:23 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T22:54:23.570 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:23 smithi084 ceph-mon[29774]: osdmap e783: 8 total, 8 up, 8 in 2024-09-20T22:54:23.584 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:54:23.604 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:23.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:23.605 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:54:23.705 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:54:23.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:23.727 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:23.728 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:54:23.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:23 smithi154 ceph-mon[33264]: 1.0 scrub starts 2024-09-20T22:54:23.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:23 smithi154 ceph-mon[33264]: 1.0 scrub ok 2024-09-20T22:54:23.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:23 smithi154 ceph-mon[33264]: pgmap v1074: 158 pgs: 1 active+undersized+remapped, 1 active+recovering+remapped, 1 active+clean+remapped, 1 active+remapped+backfill_wait, 2 active+remapped+backfilling, 3 active+clean+snaptrim, 1 active+remapped+backfill_toofull, 148 active+clean; 528 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 9.2 MiB/s rd, 1.1 MiB/s wr, 8 op/s; 28/1311 objects misplaced (2.136%); 34 MiB/s, 70 objects/s recovering 2024-09-20T22:54:23.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:23 smithi154 ceph-mon[33264]: 6.d scrub starts 2024-09-20T22:54:23.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:23 smithi154 ceph-mon[33264]: 6.d scrub ok 2024-09-20T22:54:23.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:23 smithi154 ceph-mon[33264]: 6.13 scrub starts 2024-09-20T22:54:23.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:23 smithi154 ceph-mon[33264]: 6.13 scrub ok 2024-09-20T22:54:23.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:23 smithi154 ceph-mon[33264]: osdmap e782: 8 total, 8 up, 8 in 2024-09-20T22:54:23.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:23 smithi154 ceph-mon[33264]: 5.e scrub starts 2024-09-20T22:54:23.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:23 smithi154 ceph-mon[33264]: 5.e scrub ok 2024-09-20T22:54:23.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:23 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/3439024040' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-20T22:54:23.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:23 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-20T22:54:23.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:23 smithi154 ceph-mon[33264]: 4.f scrub starts 2024-09-20T22:54:23.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:23 smithi154 ceph-mon[33264]: 4.f scrub ok 2024-09-20T22:54:23.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:23 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/2243256074' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:54:23.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:23 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:54:23.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:23 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:54:23.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:23 smithi154 ceph-mon[33264]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:54:23.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:23 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T22:54:23.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:23 smithi154 ceph-mon[33264]: osdmap e783: 8 total, 8 up, 8 in 2024-09-20T22:54:23.828 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:54:23.849 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:23.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:23.850 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:54:23.951 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:54:23.973 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:23.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:23.974 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:54:24.075 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:54:24.098 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:24.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:24.099 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:54:24.200 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:54:24.221 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:24.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:24.222 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:54:24.313 INFO:tasks.rados.rados.0.smithi084.stdout:3115: done (3 left) 2024-09-20T22:54:24.313 INFO:tasks.rados.rados.0.smithi084.stdout:3116: done (2 left) 2024-09-20T22:54:24.313 INFO:tasks.rados.rados.0.smithi084.stdout:3117: done (1 left) 2024-09-20T22:54:24.313 INFO:tasks.rados.rados.0.smithi084.stdout:3118: done (0 left) 2024-09-20T22:54:24.313 INFO:tasks.rados.rados.0.smithi084.stdout:3119: snap_remove snap 335 2024-09-20T22:54:24.322 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:54:24.343 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:24.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:24.344 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:54:24.445 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:54:24.466 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:24.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:24.467 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:54:24.568 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:54:24.582 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:24 smithi084 ceph-mon[40024]: 2.2 scrub starts 2024-09-20T22:54:24.582 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:24 smithi084 ceph-mon[40024]: 2.2 scrub ok 2024-09-20T22:54:24.582 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:24 smithi084 ceph-mon[40024]: osdmap e784: 8 total, 8 up, 8 in 2024-09-20T22:54:24.583 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:24 smithi084 ceph-mon[29774]: 2.2 scrub starts 2024-09-20T22:54:24.583 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:24 smithi084 ceph-mon[29774]: 2.2 scrub ok 2024-09-20T22:54:24.583 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:24 smithi084 ceph-mon[29774]: osdmap e784: 8 total, 8 up, 8 in 2024-09-20T22:54:24.583 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-20T22:54:24.583 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-20T22:54:24.583 INFO:teuthology.orchestra.run.smithi084.stdout: "epoch": 784, 2024-09-20T22:54:24.583 INFO:teuthology.orchestra.run.smithi084.stdout: "fsid": "a9c0bc70-77a0-11ef-baf5-efdc52797490", 2024-09-20T22:54:24.583 INFO:teuthology.orchestra.run.smithi084.stdout: "created": "2024-09-20T22:36:30.240626+0000", 2024-09-20T22:54:24.583 INFO:teuthology.orchestra.run.smithi084.stdout: "modified": "2024-09-20T22:54:24.306583+0000", 2024-09-20T22:54:24.584 INFO:teuthology.orchestra.run.smithi084.stdout: "last_up_change": "2024-09-20T22:41:27.098565+0000", 2024-09-20T22:54:24.584 INFO:teuthology.orchestra.run.smithi084.stdout: "last_in_change": "2024-09-20T22:54:16.306377+0000", 2024-09-20T22:54:24.584 INFO:teuthology.orchestra.run.smithi084.stdout: "flags": "noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-20T22:54:24.584 INFO:teuthology.orchestra.run.smithi084.stdout: "flags_num": 5801984, 2024-09-20T22:54:24.584 INFO:teuthology.orchestra.run.smithi084.stdout: "flags_set": [ 2024-09-20T22:54:24.584 INFO:teuthology.orchestra.run.smithi084.stdout: "noscrub", 2024-09-20T22:54:24.584 INFO:teuthology.orchestra.run.smithi084.stdout: "pglog_hardlimit", 2024-09-20T22:54:24.584 INFO:teuthology.orchestra.run.smithi084.stdout: "purged_snapdirs", 2024-09-20T22:54:24.584 INFO:teuthology.orchestra.run.smithi084.stdout: "recovery_deletes", 2024-09-20T22:54:24.584 INFO:teuthology.orchestra.run.smithi084.stdout: "sortbitwise" 2024-09-20T22:54:24.584 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2024-09-20T22:54:24.585 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_version": 18, 2024-09-20T22:54:24.585 INFO:teuthology.orchestra.run.smithi084.stdout: "full_ratio": 0.94999998807907104, 2024-09-20T22:54:24.585 INFO:teuthology.orchestra.run.smithi084.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-20T22:54:24.585 INFO:teuthology.orchestra.run.smithi084.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-20T22:54:24.585 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_snapshot": "", 2024-09-20T22:54:24.585 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_max": 7, 2024-09-20T22:54:24.585 INFO:teuthology.orchestra.run.smithi084.stdout: "max_osd": 8, 2024-09-20T22:54:24.585 INFO:teuthology.orchestra.run.smithi084.stdout: "require_min_compat_client": "luminous", 2024-09-20T22:54:24.585 INFO:teuthology.orchestra.run.smithi084.stdout: "min_compat_client": "luminous", 2024-09-20T22:54:24.585 INFO:teuthology.orchestra.run.smithi084.stdout: "require_osd_release": "squid", 2024-09-20T22:54:24.585 INFO:teuthology.orchestra.run.smithi084.stdout: "allow_crimson": false, 2024-09-20T22:54:24.586 INFO:teuthology.orchestra.run.smithi084.stdout: "pools": [ 2024-09-20T22:54:24.586 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:24.586 INFO:teuthology.orchestra.run.smithi084.stdout: "pool": 1, 2024-09-20T22:54:24.586 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_name": ".mgr", 2024-09-20T22:54:24.586 INFO:teuthology.orchestra.run.smithi084.stdout: "create_time": "2024-09-20T22:39:21.984381+0000", 2024-09-20T22:54:24.586 INFO:teuthology.orchestra.run.smithi084.stdout: "flags": 1, 2024-09-20T22:54:24.586 INFO:teuthology.orchestra.run.smithi084.stdout: "flags_names": "hashpspool", 2024-09-20T22:54:24.586 INFO:teuthology.orchestra.run.smithi084.stdout: "type": 1, 2024-09-20T22:54:24.586 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 3, 2024-09-20T22:54:24.586 INFO:teuthology.orchestra.run.smithi084.stdout: "min_size": 2, 2024-09-20T22:54:24.586 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_rule": 0, 2024-09-20T22:54:24.587 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_count": 0, 2024-09-20T22:54:24.587 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_target": 0, 2024-09-20T22:54:24.587 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_barrier": 0, 2024-09-20T22:54:24.587 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-20T22:54:24.587 INFO:teuthology.orchestra.run.smithi084.stdout: "is_stretch_pool": false, 2024-09-20T22:54:24.587 INFO:teuthology.orchestra.run.smithi084.stdout: "object_hash": 2, 2024-09-20T22:54:24.587 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_autoscale_mode": "off", 2024-09-20T22:54:24.587 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num": 11, 2024-09-20T22:54:24.587 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num": 11, 2024-09-20T22:54:24.587 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num_target": 11, 2024-09-20T22:54:24.587 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_target": 11, 2024-09-20T22:54:24.588 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_pending": 11, 2024-09-20T22:54:24.588 INFO:teuthology.orchestra.run.smithi084.stdout: "last_pg_merge_meta": { 2024-09-20T22:54:24.588 INFO:teuthology.orchestra.run.smithi084.stdout: "source_pgid": "1.b", 2024-09-20T22:54:24.588 INFO:teuthology.orchestra.run.smithi084.stdout: "ready_epoch": 299, 2024-09-20T22:54:24.588 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_started": 298, 2024-09-20T22:54:24.588 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_clean": 298, 2024-09-20T22:54:24.588 INFO:teuthology.orchestra.run.smithi084.stdout: "source_version": "77'128", 2024-09-20T22:54:24.588 INFO:teuthology.orchestra.run.smithi084.stdout: "target_version": "77'128" 2024-09-20T22:54:24.588 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:24.588 INFO:teuthology.orchestra.run.smithi084.stdout: "last_change": "769", 2024-09-20T22:54:24.588 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend": "0", 2024-09-20T22:54:24.589 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_prenautilus": "299", 2024-09-20T22:54:24.589 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_preluminous": "297", 2024-09-20T22:54:24.589 INFO:teuthology.orchestra.run.smithi084.stdout: "auid": 0, 2024-09-20T22:54:24.589 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_mode": "selfmanaged", 2024-09-20T22:54:24.589 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_seq": 0, 2024-09-20T22:54:24.589 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_epoch": 0, 2024-09-20T22:54:24.589 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_snaps": [], 2024-09-20T22:54:24.589 INFO:teuthology.orchestra.run.smithi084.stdout: "removed_snaps": "[]", 2024-09-20T22:54:24.589 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_bytes": 0, 2024-09-20T22:54:24.589 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_objects": 0, 2024-09-20T22:54:24.589 INFO:teuthology.orchestra.run.smithi084.stdout: "tiers": [], 2024-09-20T22:54:24.589 INFO:teuthology.orchestra.run.smithi084.stdout: "tier_of": -1, 2024-09-20T22:54:24.590 INFO:teuthology.orchestra.run.smithi084.stdout: "read_tier": -1, 2024-09-20T22:54:24.590 INFO:teuthology.orchestra.run.smithi084.stdout: "write_tier": -1, 2024-09-20T22:54:24.590 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_mode": "none", 2024-09-20T22:54:24.590 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_bytes": 0, 2024-09-20T22:54:24.590 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_objects": 0, 2024-09-20T22:54:24.590 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-20T22:54:24.590 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-20T22:54:24.590 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-20T22:54:24.590 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_flush_age": 0, 2024-09-20T22:54:24.590 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_evict_age": 0, 2024-09-20T22:54:24.590 INFO:teuthology.orchestra.run.smithi084.stdout: "erasure_code_profile": "", 2024-09-20T22:54:24.591 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_params": { 2024-09-20T22:54:24.591 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "none" 2024-09-20T22:54:24.591 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:24.591 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_period": 0, 2024-09-20T22:54:24.591 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_count": 0, 2024-09-20T22:54:24.591 INFO:teuthology.orchestra.run.smithi084.stdout: "use_gmt_hitset": true, 2024-09-20T22:54:24.591 INFO:teuthology.orchestra.run.smithi084.stdout: "min_read_recency_for_promote": 0, 2024-09-20T22:54:24.591 INFO:teuthology.orchestra.run.smithi084.stdout: "min_write_recency_for_promote": 0, 2024-09-20T22:54:24.591 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_grade_decay_rate": 0, 2024-09-20T22:54:24.591 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_search_last_n": 0, 2024-09-20T22:54:24.591 INFO:teuthology.orchestra.run.smithi084.stdout: "grade_table": [], 2024-09-20T22:54:24.592 INFO:teuthology.orchestra.run.smithi084.stdout: "stripe_width": 0, 2024-09-20T22:54:24.592 INFO:teuthology.orchestra.run.smithi084.stdout: "expected_num_objects": 0, 2024-09-20T22:54:24.592 INFO:teuthology.orchestra.run.smithi084.stdout: "fast_read": false, 2024-09-20T22:54:24.592 INFO:teuthology.orchestra.run.smithi084.stdout: "options": { 2024-09-20T22:54:24.592 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_max": 32, 2024-09-20T22:54:24.592 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_min": 1 2024-09-20T22:54:24.592 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:24.592 INFO:teuthology.orchestra.run.smithi084.stdout: "application_metadata": { 2024-09-20T22:54:24.592 INFO:teuthology.orchestra.run.smithi084.stdout: "mgr": {} 2024-09-20T22:54:24.592 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:24.592 INFO:teuthology.orchestra.run.smithi084.stdout: "read_balance": { 2024-09-20T22:54:24.592 INFO:teuthology.orchestra.run.smithi084.stdout: "score_type": "Fair distribution", 2024-09-20T22:54:24.593 INFO:teuthology.orchestra.run.smithi084.stdout: "score_acting": 2.1800000667572021, 2024-09-20T22:54:24.593 INFO:teuthology.orchestra.run.smithi084.stdout: "score_stable": 2.1800000667572021, 2024-09-20T22:54:24.593 INFO:teuthology.orchestra.run.smithi084.stdout: "optimal_score": 1, 2024-09-20T22:54:24.593 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_acting": 2.1800000667572021, 2024-09-20T22:54:24.593 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_stable": 2.1800000667572021, 2024-09-20T22:54:24.593 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity_weighted": 1, 2024-09-20T22:54:24.593 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity": 1, 2024-09-20T22:54:24.593 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity_weighted": 1 2024-09-20T22:54:24.593 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:24.593 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:24.593 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:24.594 INFO:teuthology.orchestra.run.smithi084.stdout: "pool": 2, 2024-09-20T22:54:24.594 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_name": "datapool", 2024-09-20T22:54:24.594 INFO:teuthology.orchestra.run.smithi084.stdout: "create_time": "2024-09-20T22:41:35.820081+0000", 2024-09-20T22:54:24.594 INFO:teuthology.orchestra.run.smithi084.stdout: "flags": 8193, 2024-09-20T22:54:24.594 INFO:teuthology.orchestra.run.smithi084.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-20T22:54:24.594 INFO:teuthology.orchestra.run.smithi084.stdout: "type": 1, 2024-09-20T22:54:24.594 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 3, 2024-09-20T22:54:24.594 INFO:teuthology.orchestra.run.smithi084.stdout: "min_size": 2, 2024-09-20T22:54:24.594 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_rule": 0, 2024-09-20T22:54:24.594 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_count": 0, 2024-09-20T22:54:24.594 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_target": 0, 2024-09-20T22:54:24.595 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_barrier": 0, 2024-09-20T22:54:24.595 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-20T22:54:24.595 INFO:teuthology.orchestra.run.smithi084.stdout: "is_stretch_pool": false, 2024-09-20T22:54:24.595 INFO:teuthology.orchestra.run.smithi084.stdout: "object_hash": 2, 2024-09-20T22:54:24.595 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_autoscale_mode": "off", 2024-09-20T22:54:24.595 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num": 3, 2024-09-20T22:54:24.595 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num": 3, 2024-09-20T22:54:24.595 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num_target": 3, 2024-09-20T22:54:24.595 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_target": 3, 2024-09-20T22:54:24.595 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_pending": 3, 2024-09-20T22:54:24.595 INFO:teuthology.orchestra.run.smithi084.stdout: "last_pg_merge_meta": { 2024-09-20T22:54:24.595 INFO:teuthology.orchestra.run.smithi084.stdout: "source_pgid": "0.0", 2024-09-20T22:54:24.595 INFO:teuthology.orchestra.run.smithi084.stdout: "ready_epoch": 0, 2024-09-20T22:54:24.596 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_started": 0, 2024-09-20T22:54:24.596 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_clean": 0, 2024-09-20T22:54:24.596 INFO:teuthology.orchestra.run.smithi084.stdout: "source_version": "0'0", 2024-09-20T22:54:24.596 INFO:teuthology.orchestra.run.smithi084.stdout: "target_version": "0'0" 2024-09-20T22:54:24.596 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:24.596 INFO:teuthology.orchestra.run.smithi084.stdout: "last_change": "69", 2024-09-20T22:54:24.596 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend": "0", 2024-09-20T22:54:24.596 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-20T22:54:24.596 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_preluminous": "0", 2024-09-20T22:54:24.596 INFO:teuthology.orchestra.run.smithi084.stdout: "auid": 0, 2024-09-20T22:54:24.596 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_mode": "selfmanaged", 2024-09-20T22:54:24.596 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_seq": 2, 2024-09-20T22:54:24.597 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_epoch": 69, 2024-09-20T22:54:24.597 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_snaps": [], 2024-09-20T22:54:24.597 INFO:teuthology.orchestra.run.smithi084.stdout: "removed_snaps": "[]", 2024-09-20T22:54:24.597 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_bytes": 0, 2024-09-20T22:54:24.597 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_objects": 0, 2024-09-20T22:54:24.597 INFO:teuthology.orchestra.run.smithi084.stdout: "tiers": [], 2024-09-20T22:54:24.597 INFO:teuthology.orchestra.run.smithi084.stdout: "tier_of": -1, 2024-09-20T22:54:24.597 INFO:teuthology.orchestra.run.smithi084.stdout: "read_tier": -1, 2024-09-20T22:54:24.597 INFO:teuthology.orchestra.run.smithi084.stdout: "write_tier": -1, 2024-09-20T22:54:24.597 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_mode": "none", 2024-09-20T22:54:24.597 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_bytes": 0, 2024-09-20T22:54:24.597 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_objects": 0, 2024-09-20T22:54:24.598 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-20T22:54:24.598 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-20T22:54:24.598 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-20T22:54:24.598 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_flush_age": 0, 2024-09-20T22:54:24.598 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_evict_age": 0, 2024-09-20T22:54:24.598 INFO:teuthology.orchestra.run.smithi084.stdout: "erasure_code_profile": "", 2024-09-20T22:54:24.598 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_params": { 2024-09-20T22:54:24.598 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "none" 2024-09-20T22:54:24.598 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:24.598 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_period": 0, 2024-09-20T22:54:24.598 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_count": 0, 2024-09-20T22:54:24.598 INFO:teuthology.orchestra.run.smithi084.stdout: "use_gmt_hitset": true, 2024-09-20T22:54:24.599 INFO:teuthology.orchestra.run.smithi084.stdout: "min_read_recency_for_promote": 0, 2024-09-20T22:54:24.599 INFO:teuthology.orchestra.run.smithi084.stdout: "min_write_recency_for_promote": 0, 2024-09-20T22:54:24.599 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_grade_decay_rate": 0, 2024-09-20T22:54:24.599 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_search_last_n": 0, 2024-09-20T22:54:24.599 INFO:teuthology.orchestra.run.smithi084.stdout: "grade_table": [], 2024-09-20T22:54:24.599 INFO:teuthology.orchestra.run.smithi084.stdout: "stripe_width": 0, 2024-09-20T22:54:24.599 INFO:teuthology.orchestra.run.smithi084.stdout: "expected_num_objects": 0, 2024-09-20T22:54:24.599 INFO:teuthology.orchestra.run.smithi084.stdout: "fast_read": false, 2024-09-20T22:54:24.599 INFO:teuthology.orchestra.run.smithi084.stdout: "options": {}, 2024-09-20T22:54:24.599 INFO:teuthology.orchestra.run.smithi084.stdout: "application_metadata": { 2024-09-20T22:54:24.599 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd": {} 2024-09-20T22:54:24.599 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:24.600 INFO:teuthology.orchestra.run.smithi084.stdout: "read_balance": { 2024-09-20T22:54:24.600 INFO:teuthology.orchestra.run.smithi084.stdout: "score_type": "Fair distribution", 2024-09-20T22:54:24.600 INFO:teuthology.orchestra.run.smithi084.stdout: "score_acting": 2.6500000953674316, 2024-09-20T22:54:24.600 INFO:teuthology.orchestra.run.smithi084.stdout: "score_stable": 2.6500000953674316, 2024-09-20T22:54:24.600 INFO:teuthology.orchestra.run.smithi084.stdout: "optimal_score": 0.87999999523162842, 2024-09-20T22:54:24.600 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_acting": 2.3299999237060547, 2024-09-20T22:54:24.600 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_stable": 2.3299999237060547, 2024-09-20T22:54:24.600 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity_weighted": 1, 2024-09-20T22:54:24.600 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity": 1, 2024-09-20T22:54:24.600 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity_weighted": 1 2024-09-20T22:54:24.600 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:24.601 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:24.601 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:24.601 INFO:teuthology.orchestra.run.smithi084.stdout: "pool": 3, 2024-09-20T22:54:24.601 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_name": ".rgw.root", 2024-09-20T22:54:24.601 INFO:teuthology.orchestra.run.smithi084.stdout: "create_time": "2024-09-20T22:41:36.546389+0000", 2024-09-20T22:54:24.601 INFO:teuthology.orchestra.run.smithi084.stdout: "flags": 1, 2024-09-20T22:54:24.601 INFO:teuthology.orchestra.run.smithi084.stdout: "flags_names": "hashpspool", 2024-09-20T22:54:24.601 INFO:teuthology.orchestra.run.smithi084.stdout: "type": 1, 2024-09-20T22:54:24.601 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 3, 2024-09-20T22:54:24.601 INFO:teuthology.orchestra.run.smithi084.stdout: "min_size": 2, 2024-09-20T22:54:24.601 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_rule": 0, 2024-09-20T22:54:24.601 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_count": 0, 2024-09-20T22:54:24.601 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_target": 0, 2024-09-20T22:54:24.602 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_barrier": 0, 2024-09-20T22:54:24.602 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-20T22:54:24.602 INFO:teuthology.orchestra.run.smithi084.stdout: "is_stretch_pool": false, 2024-09-20T22:54:24.602 INFO:teuthology.orchestra.run.smithi084.stdout: "object_hash": 2, 2024-09-20T22:54:24.602 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_autoscale_mode": "off", 2024-09-20T22:54:24.602 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num": 32, 2024-09-20T22:54:24.602 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num": 32, 2024-09-20T22:54:24.602 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num_target": 32, 2024-09-20T22:54:24.602 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_target": 32, 2024-09-20T22:54:24.602 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_pending": 32, 2024-09-20T22:54:24.602 INFO:teuthology.orchestra.run.smithi084.stdout: "last_pg_merge_meta": { 2024-09-20T22:54:24.603 INFO:teuthology.orchestra.run.smithi084.stdout: "source_pgid": "0.0", 2024-09-20T22:54:24.603 INFO:teuthology.orchestra.run.smithi084.stdout: "ready_epoch": 0, 2024-09-20T22:54:24.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:24.604 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_started": 0, 2024-09-20T22:54:24.604 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_clean": 0, 2024-09-20T22:54:24.604 INFO:teuthology.orchestra.run.smithi084.stdout: "source_version": "0'0", 2024-09-20T22:54:24.604 INFO:teuthology.orchestra.run.smithi084.stdout: "target_version": "0'0" 2024-09-20T22:54:24.604 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:24.605 INFO:teuthology.orchestra.run.smithi084.stdout: "last_change": "68", 2024-09-20T22:54:24.605 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend": "0", 2024-09-20T22:54:24.605 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-20T22:54:24.605 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_preluminous": "0", 2024-09-20T22:54:24.605 INFO:teuthology.orchestra.run.smithi084.stdout: "auid": 0, 2024-09-20T22:54:24.605 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_mode": "selfmanaged", 2024-09-20T22:54:24.605 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_seq": 0, 2024-09-20T22:54:24.605 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_epoch": 0, 2024-09-20T22:54:24.605 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_snaps": [], 2024-09-20T22:54:24.605 INFO:teuthology.orchestra.run.smithi084.stdout: "removed_snaps": "[]", 2024-09-20T22:54:24.605 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_bytes": 0, 2024-09-20T22:54:24.605 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_objects": 0, 2024-09-20T22:54:24.605 INFO:teuthology.orchestra.run.smithi084.stdout: "tiers": [], 2024-09-20T22:54:24.606 INFO:teuthology.orchestra.run.smithi084.stdout: "tier_of": -1, 2024-09-20T22:54:24.606 INFO:teuthology.orchestra.run.smithi084.stdout: "read_tier": -1, 2024-09-20T22:54:24.606 INFO:teuthology.orchestra.run.smithi084.stdout: "write_tier": -1, 2024-09-20T22:54:24.606 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_mode": "none", 2024-09-20T22:54:24.606 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_bytes": 0, 2024-09-20T22:54:24.606 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_objects": 0, 2024-09-20T22:54:24.606 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-20T22:54:24.606 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-20T22:54:24.606 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-20T22:54:24.606 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_flush_age": 0, 2024-09-20T22:54:24.606 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_evict_age": 0, 2024-09-20T22:54:24.606 INFO:teuthology.orchestra.run.smithi084.stdout: "erasure_code_profile": "", 2024-09-20T22:54:24.607 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_params": { 2024-09-20T22:54:24.607 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "none" 2024-09-20T22:54:24.607 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:24.607 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_period": 0, 2024-09-20T22:54:24.607 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_count": 0, 2024-09-20T22:54:24.607 INFO:teuthology.orchestra.run.smithi084.stdout: "use_gmt_hitset": true, 2024-09-20T22:54:24.607 INFO:teuthology.orchestra.run.smithi084.stdout: "min_read_recency_for_promote": 0, 2024-09-20T22:54:24.607 INFO:teuthology.orchestra.run.smithi084.stdout: "min_write_recency_for_promote": 0, 2024-09-20T22:54:24.607 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_grade_decay_rate": 0, 2024-09-20T22:54:24.607 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_search_last_n": 0, 2024-09-20T22:54:24.607 INFO:teuthology.orchestra.run.smithi084.stdout: "grade_table": [], 2024-09-20T22:54:24.607 INFO:teuthology.orchestra.run.smithi084.stdout: "stripe_width": 0, 2024-09-20T22:54:24.607 INFO:teuthology.orchestra.run.smithi084.stdout: "expected_num_objects": 0, 2024-09-20T22:54:24.608 INFO:teuthology.orchestra.run.smithi084.stdout: "fast_read": false, 2024-09-20T22:54:24.608 INFO:teuthology.orchestra.run.smithi084.stdout: "options": {}, 2024-09-20T22:54:24.608 INFO:teuthology.orchestra.run.smithi084.stdout: "application_metadata": { 2024-09-20T22:54:24.608 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw": {} 2024-09-20T22:54:24.608 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:24.608 INFO:teuthology.orchestra.run.smithi084.stdout: "read_balance": { 2024-09-20T22:54:24.608 INFO:teuthology.orchestra.run.smithi084.stdout: "score_type": "Fair distribution", 2024-09-20T22:54:24.608 INFO:teuthology.orchestra.run.smithi084.stdout: "score_acting": 1.5, 2024-09-20T22:54:24.608 INFO:teuthology.orchestra.run.smithi084.stdout: "score_stable": 1.5, 2024-09-20T22:54:24.608 INFO:teuthology.orchestra.run.smithi084.stdout: "optimal_score": 1, 2024-09-20T22:54:24.608 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_acting": 1.5, 2024-09-20T22:54:24.609 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_stable": 1.5, 2024-09-20T22:54:24.609 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity_weighted": 1, 2024-09-20T22:54:24.609 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity": 1, 2024-09-20T22:54:24.609 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity_weighted": 1 2024-09-20T22:54:24.609 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:24.609 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:24.609 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:24.609 INFO:teuthology.orchestra.run.smithi084.stdout: "pool": 4, 2024-09-20T22:54:24.609 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_name": "default.rgw.log", 2024-09-20T22:54:24.609 INFO:teuthology.orchestra.run.smithi084.stdout: "create_time": "2024-09-20T22:41:38.454640+0000", 2024-09-20T22:54:24.609 INFO:teuthology.orchestra.run.smithi084.stdout: "flags": 1, 2024-09-20T22:54:24.610 INFO:teuthology.orchestra.run.smithi084.stdout: "flags_names": "hashpspool", 2024-09-20T22:54:24.610 INFO:teuthology.orchestra.run.smithi084.stdout: "type": 1, 2024-09-20T22:54:24.610 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 3, 2024-09-20T22:54:24.610 INFO:teuthology.orchestra.run.smithi084.stdout: "min_size": 2, 2024-09-20T22:54:24.610 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_rule": 0, 2024-09-20T22:54:24.610 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_count": 0, 2024-09-20T22:54:24.610 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_target": 0, 2024-09-20T22:54:24.610 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_barrier": 0, 2024-09-20T22:54:24.610 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-20T22:54:24.610 INFO:teuthology.orchestra.run.smithi084.stdout: "is_stretch_pool": false, 2024-09-20T22:54:24.610 INFO:teuthology.orchestra.run.smithi084.stdout: "object_hash": 2, 2024-09-20T22:54:24.611 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_autoscale_mode": "off", 2024-09-20T22:54:24.611 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num": 32, 2024-09-20T22:54:24.611 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num": 32, 2024-09-20T22:54:24.611 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num_target": 32, 2024-09-20T22:54:24.611 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_target": 32, 2024-09-20T22:54:24.611 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_pending": 32, 2024-09-20T22:54:24.611 INFO:teuthology.orchestra.run.smithi084.stdout: "last_pg_merge_meta": { 2024-09-20T22:54:24.611 INFO:teuthology.orchestra.run.smithi084.stdout: "source_pgid": "0.0", 2024-09-20T22:54:24.611 INFO:teuthology.orchestra.run.smithi084.stdout: "ready_epoch": 0, 2024-09-20T22:54:24.611 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_started": 0, 2024-09-20T22:54:24.611 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_clean": 0, 2024-09-20T22:54:24.611 INFO:teuthology.orchestra.run.smithi084.stdout: "source_version": "0'0", 2024-09-20T22:54:24.612 INFO:teuthology.orchestra.run.smithi084.stdout: "target_version": "0'0" 2024-09-20T22:54:24.612 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:24.612 INFO:teuthology.orchestra.run.smithi084.stdout: "last_change": "70", 2024-09-20T22:54:24.612 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend": "0", 2024-09-20T22:54:24.612 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-20T22:54:24.612 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_preluminous": "0", 2024-09-20T22:54:24.612 INFO:teuthology.orchestra.run.smithi084.stdout: "auid": 0, 2024-09-20T22:54:24.612 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_mode": "selfmanaged", 2024-09-20T22:54:24.612 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_seq": 0, 2024-09-20T22:54:24.612 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_epoch": 0, 2024-09-20T22:54:24.612 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_snaps": [], 2024-09-20T22:54:24.612 INFO:teuthology.orchestra.run.smithi084.stdout: "removed_snaps": "[]", 2024-09-20T22:54:24.613 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_bytes": 0, 2024-09-20T22:54:24.613 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_objects": 0, 2024-09-20T22:54:24.613 INFO:teuthology.orchestra.run.smithi084.stdout: "tiers": [], 2024-09-20T22:54:24.613 INFO:teuthology.orchestra.run.smithi084.stdout: "tier_of": -1, 2024-09-20T22:54:24.613 INFO:teuthology.orchestra.run.smithi084.stdout: "read_tier": -1, 2024-09-20T22:54:24.613 INFO:teuthology.orchestra.run.smithi084.stdout: "write_tier": -1, 2024-09-20T22:54:24.613 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_mode": "none", 2024-09-20T22:54:24.613 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_bytes": 0, 2024-09-20T22:54:24.613 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_objects": 0, 2024-09-20T22:54:24.613 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-20T22:54:24.613 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-20T22:54:24.613 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-20T22:54:24.614 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_flush_age": 0, 2024-09-20T22:54:24.614 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_evict_age": 0, 2024-09-20T22:54:24.615 INFO:teuthology.orchestra.run.smithi084.stdout: "erasure_code_profile": "", 2024-09-20T22:54:24.615 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_params": { 2024-09-20T22:54:24.615 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "none" 2024-09-20T22:54:24.615 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:24.615 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_period": 0, 2024-09-20T22:54:24.615 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_count": 0, 2024-09-20T22:54:24.615 INFO:teuthology.orchestra.run.smithi084.stdout: "use_gmt_hitset": true, 2024-09-20T22:54:24.615 INFO:teuthology.orchestra.run.smithi084.stdout: "min_read_recency_for_promote": 0, 2024-09-20T22:54:24.615 INFO:teuthology.orchestra.run.smithi084.stdout: "min_write_recency_for_promote": 0, 2024-09-20T22:54:24.615 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_grade_decay_rate": 0, 2024-09-20T22:54:24.615 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_search_last_n": 0, 2024-09-20T22:54:24.616 INFO:teuthology.orchestra.run.smithi084.stdout: "grade_table": [], 2024-09-20T22:54:24.616 INFO:teuthology.orchestra.run.smithi084.stdout: "stripe_width": 0, 2024-09-20T22:54:24.616 INFO:teuthology.orchestra.run.smithi084.stdout: "expected_num_objects": 0, 2024-09-20T22:54:24.616 INFO:teuthology.orchestra.run.smithi084.stdout: "fast_read": false, 2024-09-20T22:54:24.616 INFO:teuthology.orchestra.run.smithi084.stdout: "options": {}, 2024-09-20T22:54:24.616 INFO:teuthology.orchestra.run.smithi084.stdout: "application_metadata": { 2024-09-20T22:54:24.616 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw": {} 2024-09-20T22:54:24.616 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:24.616 INFO:teuthology.orchestra.run.smithi084.stdout: "read_balance": { 2024-09-20T22:54:24.616 INFO:teuthology.orchestra.run.smithi084.stdout: "score_type": "Fair distribution", 2024-09-20T22:54:24.616 INFO:teuthology.orchestra.run.smithi084.stdout: "score_acting": 2.25, 2024-09-20T22:54:24.616 INFO:teuthology.orchestra.run.smithi084.stdout: "score_stable": 2.25, 2024-09-20T22:54:24.616 INFO:teuthology.orchestra.run.smithi084.stdout: "optimal_score": 1, 2024-09-20T22:54:24.617 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_acting": 2.25, 2024-09-20T22:54:24.617 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_stable": 2.25, 2024-09-20T22:54:24.617 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity_weighted": 1, 2024-09-20T22:54:24.617 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity": 1, 2024-09-20T22:54:24.617 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity_weighted": 1 2024-09-20T22:54:24.617 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:24.617 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:24.617 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:24.617 INFO:teuthology.orchestra.run.smithi084.stdout: "pool": 5, 2024-09-20T22:54:24.617 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_name": "default.rgw.control", 2024-09-20T22:54:24.617 INFO:teuthology.orchestra.run.smithi084.stdout: "create_time": "2024-09-20T22:41:40.407129+0000", 2024-09-20T22:54:24.617 INFO:teuthology.orchestra.run.smithi084.stdout: "flags": 1, 2024-09-20T22:54:24.618 INFO:teuthology.orchestra.run.smithi084.stdout: "flags_names": "hashpspool", 2024-09-20T22:54:24.618 INFO:teuthology.orchestra.run.smithi084.stdout: "type": 1, 2024-09-20T22:54:24.618 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 3, 2024-09-20T22:54:24.618 INFO:teuthology.orchestra.run.smithi084.stdout: "min_size": 2, 2024-09-20T22:54:24.618 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_rule": 0, 2024-09-20T22:54:24.618 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_count": 0, 2024-09-20T22:54:24.618 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_target": 0, 2024-09-20T22:54:24.618 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_barrier": 0, 2024-09-20T22:54:24.618 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-20T22:54:24.618 INFO:teuthology.orchestra.run.smithi084.stdout: "is_stretch_pool": false, 2024-09-20T22:54:24.618 INFO:teuthology.orchestra.run.smithi084.stdout: "object_hash": 2, 2024-09-20T22:54:24.618 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_autoscale_mode": "off", 2024-09-20T22:54:24.618 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num": 32, 2024-09-20T22:54:24.619 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num": 32, 2024-09-20T22:54:24.619 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num_target": 32, 2024-09-20T22:54:24.619 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_target": 32, 2024-09-20T22:54:24.619 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_pending": 32, 2024-09-20T22:54:24.619 INFO:teuthology.orchestra.run.smithi084.stdout: "last_pg_merge_meta": { 2024-09-20T22:54:24.619 INFO:teuthology.orchestra.run.smithi084.stdout: "source_pgid": "0.0", 2024-09-20T22:54:24.619 INFO:teuthology.orchestra.run.smithi084.stdout: "ready_epoch": 0, 2024-09-20T22:54:24.619 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_started": 0, 2024-09-20T22:54:24.619 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_clean": 0, 2024-09-20T22:54:24.619 INFO:teuthology.orchestra.run.smithi084.stdout: "source_version": "0'0", 2024-09-20T22:54:24.619 INFO:teuthology.orchestra.run.smithi084.stdout: "target_version": "0'0" 2024-09-20T22:54:24.620 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:24.620 INFO:teuthology.orchestra.run.smithi084.stdout: "last_change": "72", 2024-09-20T22:54:24.620 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend": "0", 2024-09-20T22:54:24.620 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-20T22:54:24.620 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_preluminous": "0", 2024-09-20T22:54:24.620 INFO:teuthology.orchestra.run.smithi084.stdout: "auid": 0, 2024-09-20T22:54:24.620 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_mode": "selfmanaged", 2024-09-20T22:54:24.620 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_seq": 0, 2024-09-20T22:54:24.620 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_epoch": 0, 2024-09-20T22:54:24.620 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_snaps": [], 2024-09-20T22:54:24.620 INFO:teuthology.orchestra.run.smithi084.stdout: "removed_snaps": "[]", 2024-09-20T22:54:24.620 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_bytes": 0, 2024-09-20T22:54:24.620 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_objects": 0, 2024-09-20T22:54:24.620 INFO:teuthology.orchestra.run.smithi084.stdout: "tiers": [], 2024-09-20T22:54:24.621 INFO:teuthology.orchestra.run.smithi084.stdout: "tier_of": -1, 2024-09-20T22:54:24.621 INFO:teuthology.orchestra.run.smithi084.stdout: "read_tier": -1, 2024-09-20T22:54:24.621 INFO:teuthology.orchestra.run.smithi084.stdout: "write_tier": -1, 2024-09-20T22:54:24.621 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_mode": "none", 2024-09-20T22:54:24.621 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_bytes": 0, 2024-09-20T22:54:24.621 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_objects": 0, 2024-09-20T22:54:24.621 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-20T22:54:24.621 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-20T22:54:24.621 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-20T22:54:24.621 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_flush_age": 0, 2024-09-20T22:54:24.621 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_evict_age": 0, 2024-09-20T22:54:24.621 INFO:teuthology.orchestra.run.smithi084.stdout: "erasure_code_profile": "", 2024-09-20T22:54:24.621 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_params": { 2024-09-20T22:54:24.621 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "none" 2024-09-20T22:54:24.622 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:24.622 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_period": 0, 2024-09-20T22:54:24.622 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_count": 0, 2024-09-20T22:54:24.622 INFO:teuthology.orchestra.run.smithi084.stdout: "use_gmt_hitset": true, 2024-09-20T22:54:24.622 INFO:teuthology.orchestra.run.smithi084.stdout: "min_read_recency_for_promote": 0, 2024-09-20T22:54:24.622 INFO:teuthology.orchestra.run.smithi084.stdout: "min_write_recency_for_promote": 0, 2024-09-20T22:54:24.622 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_grade_decay_rate": 0, 2024-09-20T22:54:24.622 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_search_last_n": 0, 2024-09-20T22:54:24.622 INFO:teuthology.orchestra.run.smithi084.stdout: "grade_table": [], 2024-09-20T22:54:24.622 INFO:teuthology.orchestra.run.smithi084.stdout: "stripe_width": 0, 2024-09-20T22:54:24.622 INFO:teuthology.orchestra.run.smithi084.stdout: "expected_num_objects": 0, 2024-09-20T22:54:24.622 INFO:teuthology.orchestra.run.smithi084.stdout: "fast_read": false, 2024-09-20T22:54:24.623 INFO:teuthology.orchestra.run.smithi084.stdout: "options": {}, 2024-09-20T22:54:24.623 INFO:teuthology.orchestra.run.smithi084.stdout: "application_metadata": { 2024-09-20T22:54:24.623 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw": {} 2024-09-20T22:54:24.623 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:24.623 INFO:teuthology.orchestra.run.smithi084.stdout: "read_balance": { 2024-09-20T22:54:24.623 INFO:teuthology.orchestra.run.smithi084.stdout: "score_type": "Fair distribution", 2024-09-20T22:54:24.623 INFO:teuthology.orchestra.run.smithi084.stdout: "score_acting": 1.25, 2024-09-20T22:54:24.623 INFO:teuthology.orchestra.run.smithi084.stdout: "score_stable": 1.25, 2024-09-20T22:54:24.623 INFO:teuthology.orchestra.run.smithi084.stdout: "optimal_score": 1, 2024-09-20T22:54:24.623 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_acting": 1.25, 2024-09-20T22:54:24.624 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_stable": 1.25, 2024-09-20T22:54:24.624 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity_weighted": 1, 2024-09-20T22:54:24.624 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity": 1, 2024-09-20T22:54:24.624 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity_weighted": 1 2024-09-20T22:54:24.624 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:24.624 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:24.624 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:24.626 INFO:teuthology.orchestra.run.smithi084.stdout: "pool": 6, 2024-09-20T22:54:24.626 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_name": "default.rgw.meta", 2024-09-20T22:54:24.626 INFO:teuthology.orchestra.run.smithi084.stdout: "create_time": "2024-09-20T22:41:42.476716+0000", 2024-09-20T22:54:24.626 INFO:teuthology.orchestra.run.smithi084.stdout: "flags": 1, 2024-09-20T22:54:24.627 INFO:teuthology.orchestra.run.smithi084.stdout: "flags_names": "hashpspool", 2024-09-20T22:54:24.627 INFO:teuthology.orchestra.run.smithi084.stdout: "type": 1, 2024-09-20T22:54:24.627 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 3, 2024-09-20T22:54:24.627 INFO:teuthology.orchestra.run.smithi084.stdout: "min_size": 2, 2024-09-20T22:54:24.627 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_rule": 0, 2024-09-20T22:54:24.627 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_count": 0, 2024-09-20T22:54:24.627 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_target": 0, 2024-09-20T22:54:24.627 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_barrier": 0, 2024-09-20T22:54:24.627 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-20T22:54:24.627 INFO:teuthology.orchestra.run.smithi084.stdout: "is_stretch_pool": false, 2024-09-20T22:54:24.627 INFO:teuthology.orchestra.run.smithi084.stdout: "object_hash": 2, 2024-09-20T22:54:24.628 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_autoscale_mode": "off", 2024-09-20T22:54:24.628 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num": 32, 2024-09-20T22:54:24.628 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num": 32, 2024-09-20T22:54:24.628 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num_target": 32, 2024-09-20T22:54:24.628 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_target": 32, 2024-09-20T22:54:24.628 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_pending": 32, 2024-09-20T22:54:24.628 INFO:teuthology.orchestra.run.smithi084.stdout: "last_pg_merge_meta": { 2024-09-20T22:54:24.628 INFO:teuthology.orchestra.run.smithi084.stdout: "source_pgid": "0.0", 2024-09-20T22:54:24.628 INFO:teuthology.orchestra.run.smithi084.stdout: "ready_epoch": 0, 2024-09-20T22:54:24.628 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_started": 0, 2024-09-20T22:54:24.628 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_clean": 0, 2024-09-20T22:54:24.628 INFO:teuthology.orchestra.run.smithi084.stdout: "source_version": "0'0", 2024-09-20T22:54:24.629 INFO:teuthology.orchestra.run.smithi084.stdout: "target_version": "0'0" 2024-09-20T22:54:24.629 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:24.629 INFO:teuthology.orchestra.run.smithi084.stdout: "last_change": "74", 2024-09-20T22:54:24.629 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend": "0", 2024-09-20T22:54:24.629 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-20T22:54:24.629 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_preluminous": "0", 2024-09-20T22:54:24.629 INFO:teuthology.orchestra.run.smithi084.stdout: "auid": 0, 2024-09-20T22:54:24.629 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_mode": "selfmanaged", 2024-09-20T22:54:24.629 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_seq": 0, 2024-09-20T22:54:24.629 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_epoch": 0, 2024-09-20T22:54:24.629 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_snaps": [], 2024-09-20T22:54:24.629 INFO:teuthology.orchestra.run.smithi084.stdout: "removed_snaps": "[]", 2024-09-20T22:54:24.629 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_bytes": 0, 2024-09-20T22:54:24.629 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_objects": 0, 2024-09-20T22:54:24.630 INFO:teuthology.orchestra.run.smithi084.stdout: "tiers": [], 2024-09-20T22:54:24.630 INFO:teuthology.orchestra.run.smithi084.stdout: "tier_of": -1, 2024-09-20T22:54:24.630 INFO:teuthology.orchestra.run.smithi084.stdout: "read_tier": -1, 2024-09-20T22:54:24.630 INFO:teuthology.orchestra.run.smithi084.stdout: "write_tier": -1, 2024-09-20T22:54:24.630 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_mode": "none", 2024-09-20T22:54:24.630 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_bytes": 0, 2024-09-20T22:54:24.630 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_objects": 0, 2024-09-20T22:54:24.630 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-20T22:54:24.630 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-20T22:54:24.630 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-20T22:54:24.630 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_flush_age": 0, 2024-09-20T22:54:24.631 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_evict_age": 0, 2024-09-20T22:54:24.631 INFO:teuthology.orchestra.run.smithi084.stdout: "erasure_code_profile": "", 2024-09-20T22:54:24.631 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_params": { 2024-09-20T22:54:24.631 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "none" 2024-09-20T22:54:24.631 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:24.631 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_period": 0, 2024-09-20T22:54:24.631 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_count": 0, 2024-09-20T22:54:24.631 INFO:teuthology.orchestra.run.smithi084.stdout: "use_gmt_hitset": true, 2024-09-20T22:54:24.631 INFO:teuthology.orchestra.run.smithi084.stdout: "min_read_recency_for_promote": 0, 2024-09-20T22:54:24.631 INFO:teuthology.orchestra.run.smithi084.stdout: "min_write_recency_for_promote": 0, 2024-09-20T22:54:24.631 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_grade_decay_rate": 0, 2024-09-20T22:54:24.631 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_search_last_n": 0, 2024-09-20T22:54:24.632 INFO:teuthology.orchestra.run.smithi084.stdout: "grade_table": [], 2024-09-20T22:54:24.632 INFO:teuthology.orchestra.run.smithi084.stdout: "stripe_width": 0, 2024-09-20T22:54:24.632 INFO:teuthology.orchestra.run.smithi084.stdout: "expected_num_objects": 0, 2024-09-20T22:54:24.632 INFO:teuthology.orchestra.run.smithi084.stdout: "fast_read": false, 2024-09-20T22:54:24.632 INFO:teuthology.orchestra.run.smithi084.stdout: "options": { 2024-09-20T22:54:24.632 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_autoscale_bias": 4 2024-09-20T22:54:24.632 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:24.632 INFO:teuthology.orchestra.run.smithi084.stdout: "application_metadata": { 2024-09-20T22:54:24.633 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw": {} 2024-09-20T22:54:24.633 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:24.633 INFO:teuthology.orchestra.run.smithi084.stdout: "read_balance": { 2024-09-20T22:54:24.633 INFO:teuthology.orchestra.run.smithi084.stdout: "score_type": "Fair distribution", 2024-09-20T22:54:24.633 INFO:teuthology.orchestra.run.smithi084.stdout: "score_acting": 1.75, 2024-09-20T22:54:24.633 INFO:teuthology.orchestra.run.smithi084.stdout: "score_stable": 1.75, 2024-09-20T22:54:24.633 INFO:teuthology.orchestra.run.smithi084.stdout: "optimal_score": 1, 2024-09-20T22:54:24.633 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_acting": 1.75, 2024-09-20T22:54:24.634 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_stable": 1.75, 2024-09-20T22:54:24.634 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity_weighted": 1, 2024-09-20T22:54:24.634 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity": 1, 2024-09-20T22:54:24.634 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity_weighted": 1 2024-09-20T22:54:24.634 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:24.634 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:24.634 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:24.634 INFO:teuthology.orchestra.run.smithi084.stdout: "pool": 7, 2024-09-20T22:54:24.634 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_name": "unique_pool_0", 2024-09-20T22:54:24.635 INFO:teuthology.orchestra.run.smithi084.stdout: "create_time": "2024-09-20T22:42:51.354358+0000", 2024-09-20T22:54:24.635 INFO:teuthology.orchestra.run.smithi084.stdout: "flags": 8193, 2024-09-20T22:54:24.635 INFO:teuthology.orchestra.run.smithi084.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-20T22:54:24.635 INFO:teuthology.orchestra.run.smithi084.stdout: "type": 1, 2024-09-20T22:54:24.635 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 3, 2024-09-20T22:54:24.635 INFO:teuthology.orchestra.run.smithi084.stdout: "min_size": 2, 2024-09-20T22:54:24.635 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_rule": 0, 2024-09-20T22:54:24.635 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_count": 0, 2024-09-20T22:54:24.635 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_target": 0, 2024-09-20T22:54:24.635 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_barrier": 0, 2024-09-20T22:54:24.636 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-20T22:54:24.636 INFO:teuthology.orchestra.run.smithi084.stdout: "is_stretch_pool": false, 2024-09-20T22:54:24.636 INFO:teuthology.orchestra.run.smithi084.stdout: "object_hash": 2, 2024-09-20T22:54:24.636 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_autoscale_mode": "off", 2024-09-20T22:54:24.636 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num": 16, 2024-09-20T22:54:24.636 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num": 16, 2024-09-20T22:54:24.636 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num_target": 16, 2024-09-20T22:54:24.636 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_target": 16, 2024-09-20T22:54:24.636 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_pending": 16, 2024-09-20T22:54:24.636 INFO:teuthology.orchestra.run.smithi084.stdout: "last_pg_merge_meta": { 2024-09-20T22:54:24.636 INFO:teuthology.orchestra.run.smithi084.stdout: "source_pgid": "7.10", 2024-09-20T22:54:24.636 INFO:teuthology.orchestra.run.smithi084.stdout: "ready_epoch": 484, 2024-09-20T22:54:24.636 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_started": 483, 2024-09-20T22:54:24.636 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_clean": 483, 2024-09-20T22:54:24.637 INFO:teuthology.orchestra.run.smithi084.stdout: "source_version": "480'172", 2024-09-20T22:54:24.637 INFO:teuthology.orchestra.run.smithi084.stdout: "target_version": "480'253" 2024-09-20T22:54:24.637 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:24.637 INFO:teuthology.orchestra.run.smithi084.stdout: "last_change": "784", 2024-09-20T22:54:24.637 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend": "0", 2024-09-20T22:54:24.637 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_prenautilus": "484", 2024-09-20T22:54:24.637 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_preluminous": "482", 2024-09-20T22:54:24.637 INFO:teuthology.orchestra.run.smithi084.stdout: "auid": 0, 2024-09-20T22:54:24.637 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_mode": "selfmanaged", 2024-09-20T22:54:24.637 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_seq": 357, 2024-09-20T22:54:24.637 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_epoch": 784, 2024-09-20T22:54:24.637 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_snaps": [], 2024-09-20T22:54:24.637 INFO:teuthology.orchestra.run.smithi084.stdout: "removed_snaps": "[]", 2024-09-20T22:54:24.637 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_bytes": 0, 2024-09-20T22:54:24.637 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_objects": 0, 2024-09-20T22:54:24.638 INFO:teuthology.orchestra.run.smithi084.stdout: "tiers": [], 2024-09-20T22:54:24.638 INFO:teuthology.orchestra.run.smithi084.stdout: "tier_of": -1, 2024-09-20T22:54:24.638 INFO:teuthology.orchestra.run.smithi084.stdout: "read_tier": -1, 2024-09-20T22:54:24.638 INFO:teuthology.orchestra.run.smithi084.stdout: "write_tier": -1, 2024-09-20T22:54:24.638 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_mode": "none", 2024-09-20T22:54:24.638 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_bytes": 0, 2024-09-20T22:54:24.638 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_objects": 0, 2024-09-20T22:54:24.638 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-20T22:54:24.638 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-20T22:54:24.638 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-20T22:54:24.638 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_flush_age": 0, 2024-09-20T22:54:24.638 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_evict_age": 0, 2024-09-20T22:54:24.639 INFO:teuthology.orchestra.run.smithi084.stdout: "erasure_code_profile": "", 2024-09-20T22:54:24.639 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_params": { 2024-09-20T22:54:24.639 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "none" 2024-09-20T22:54:24.639 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:24.639 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_period": 0, 2024-09-20T22:54:24.639 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_count": 0, 2024-09-20T22:54:24.639 INFO:teuthology.orchestra.run.smithi084.stdout: "use_gmt_hitset": true, 2024-09-20T22:54:24.639 INFO:teuthology.orchestra.run.smithi084.stdout: "min_read_recency_for_promote": 0, 2024-09-20T22:54:24.639 INFO:teuthology.orchestra.run.smithi084.stdout: "min_write_recency_for_promote": 0, 2024-09-20T22:54:24.639 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_grade_decay_rate": 0, 2024-09-20T22:54:24.639 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_search_last_n": 0, 2024-09-20T22:54:24.639 INFO:teuthology.orchestra.run.smithi084.stdout: "grade_table": [], 2024-09-20T22:54:24.639 INFO:teuthology.orchestra.run.smithi084.stdout: "stripe_width": 0, 2024-09-20T22:54:24.640 INFO:teuthology.orchestra.run.smithi084.stdout: "expected_num_objects": 0, 2024-09-20T22:54:24.640 INFO:teuthology.orchestra.run.smithi084.stdout: "fast_read": false, 2024-09-20T22:54:24.640 INFO:teuthology.orchestra.run.smithi084.stdout: "options": {}, 2024-09-20T22:54:24.640 INFO:teuthology.orchestra.run.smithi084.stdout: "application_metadata": { 2024-09-20T22:54:24.640 INFO:teuthology.orchestra.run.smithi084.stdout: "rados": {} 2024-09-20T22:54:24.640 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:24.640 INFO:teuthology.orchestra.run.smithi084.stdout: "read_balance": { 2024-09-20T22:54:24.640 INFO:teuthology.orchestra.run.smithi084.stdout: "score_type": "Fair distribution", 2024-09-20T22:54:24.640 INFO:teuthology.orchestra.run.smithi084.stdout: "score_acting": 2, 2024-09-20T22:54:24.640 INFO:teuthology.orchestra.run.smithi084.stdout: "score_stable": 2, 2024-09-20T22:54:24.640 INFO:teuthology.orchestra.run.smithi084.stdout: "optimal_score": 1, 2024-09-20T22:54:24.641 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_acting": 2, 2024-09-20T22:54:24.641 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_stable": 2, 2024-09-20T22:54:24.641 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity_weighted": 1, 2024-09-20T22:54:24.641 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity": 1, 2024-09-20T22:54:24.641 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity_weighted": 1 2024-09-20T22:54:24.641 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:24.641 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:24.641 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2024-09-20T22:54:24.641 INFO:teuthology.orchestra.run.smithi084.stdout: "osds": [ 2024-09-20T22:54:24.641 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:24.641 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 0, 2024-09-20T22:54:24.641 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "ea36a489-6f91-4b2f-b784-b90506b9805d", 2024-09-20T22:54:24.641 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-09-20T22:54:24.641 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-09-20T22:54:24.642 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-09-20T22:54:24.642 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-09-20T22:54:24.642 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-09-20T22:54:24.642 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-09-20T22:54:24.642 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 8, 2024-09-20T22:54:24.642 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 779, 2024-09-20T22:54:24.642 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-09-20T22:54:24.642 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-09-20T22:54:24.642 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-09-20T22:54:24.642 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:54:24.642 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:24.643 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:54:24.643 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6801", 2024-09-20T22:54:24.643 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 268084626 2024-09-20T22:54:24.643 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:24.643 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:24.643 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:24.643 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-09-20T22:54:24.643 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:54:24.643 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:24.643 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:54:24.643 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6802", 2024-09-20T22:54:24.644 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 268084626 2024-09-20T22:54:24.644 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:24.644 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:24.644 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:24.644 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-09-20T22:54:24.644 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:54:24.644 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:24.644 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:54:24.644 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6804", 2024-09-20T22:54:24.644 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 268084626 2024-09-20T22:54:24.644 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:24.644 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:24.645 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:24.645 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-09-20T22:54:24.645 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:54:24.645 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:24.645 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:54:24.645 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6803", 2024-09-20T22:54:24.645 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 268084626 2024-09-20T22:54:24.645 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:24.645 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:24.645 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:24.645 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.84:6801/268084626", 2024-09-20T22:54:24.645 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.84:6802/268084626", 2024-09-20T22:54:24.645 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.84:6804/268084626", 2024-09-20T22:54:24.646 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.84:6803/268084626", 2024-09-20T22:54:24.646 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-09-20T22:54:24.646 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-09-20T22:54:24.646 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-09-20T22:54:24.646 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:24.646 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:24.646 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:24.646 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 1, 2024-09-20T22:54:24.646 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "c8649d2f-27ef-47f8-bc94-f1cd9e62bdb8", 2024-09-20T22:54:24.646 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-09-20T22:54:24.646 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-09-20T22:54:24.646 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-09-20T22:54:24.646 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-09-20T22:54:24.647 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-09-20T22:54:24.647 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-09-20T22:54:24.647 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 13, 2024-09-20T22:54:24.647 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 779, 2024-09-20T22:54:24.647 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-09-20T22:54:24.647 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-09-20T22:54:24.647 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-09-20T22:54:24.647 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:54:24.647 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:24.647 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:54:24.647 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6805", 2024-09-20T22:54:24.647 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 418374136 2024-09-20T22:54:24.647 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:24.647 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:24.647 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:24.648 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-09-20T22:54:24.648 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:54:24.648 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:24.648 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:54:24.648 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6806", 2024-09-20T22:54:24.648 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 418374136 2024-09-20T22:54:24.648 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:24.648 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:24.648 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:24.648 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-09-20T22:54:24.648 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:54:24.648 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:24.648 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:54:24.649 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6808", 2024-09-20T22:54:24.649 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 418374136 2024-09-20T22:54:24.649 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:24.649 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:24.649 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:24.649 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-09-20T22:54:24.649 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:54:24.649 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:24.649 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:54:24.649 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6807", 2024-09-20T22:54:24.649 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 418374136 2024-09-20T22:54:24.649 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:24.649 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:24.650 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:24.650 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.84:6805/418374136", 2024-09-20T22:54:24.650 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.84:6806/418374136", 2024-09-20T22:54:24.650 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.84:6808/418374136", 2024-09-20T22:54:24.650 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.84:6807/418374136", 2024-09-20T22:54:24.650 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-09-20T22:54:24.650 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-09-20T22:54:24.651 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-09-20T22:54:24.651 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:24.651 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:24.651 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:24.651 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 2, 2024-09-20T22:54:24.651 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "c728eb48-a0f0-4de4-a651-7f3bec867280", 2024-09-20T22:54:24.651 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-09-20T22:54:24.651 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-09-20T22:54:24.651 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-09-20T22:54:24.651 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-09-20T22:54:24.652 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-09-20T22:54:24.652 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-09-20T22:54:24.652 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 18, 2024-09-20T22:54:24.652 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 778, 2024-09-20T22:54:24.652 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-09-20T22:54:24.652 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-09-20T22:54:24.653 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-09-20T22:54:24.653 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:54:24.653 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:24.653 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:54:24.653 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6809", 2024-09-20T22:54:24.653 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3977168611 2024-09-20T22:54:24.653 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:24.653 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:24.653 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:24.654 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-09-20T22:54:24.654 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:54:24.654 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:24.654 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:54:24.654 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6810", 2024-09-20T22:54:24.654 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3977168611 2024-09-20T22:54:24.654 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:24.654 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:24.654 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:24.654 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-09-20T22:54:24.654 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:54:24.654 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:24.655 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:54:24.655 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6812", 2024-09-20T22:54:24.655 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3977168611 2024-09-20T22:54:24.655 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:24.655 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:24.655 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:24.655 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-09-20T22:54:24.655 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:54:24.655 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:24.655 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:54:24.655 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6811", 2024-09-20T22:54:24.655 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3977168611 2024-09-20T22:54:24.655 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:24.656 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:24.656 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:24.656 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.84:6809/3977168611", 2024-09-20T22:54:24.656 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.84:6810/3977168611", 2024-09-20T22:54:24.656 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.84:6812/3977168611", 2024-09-20T22:54:24.656 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.84:6811/3977168611", 2024-09-20T22:54:24.656 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-09-20T22:54:24.656 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-09-20T22:54:24.656 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-09-20T22:54:24.656 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:24.656 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:24.656 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:24.656 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 3, 2024-09-20T22:54:24.657 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "573f44eb-0853-4630-a7d8-9caaeb256f9b", 2024-09-20T22:54:24.657 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-09-20T22:54:24.657 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-09-20T22:54:24.657 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-09-20T22:54:24.657 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-09-20T22:54:24.657 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-09-20T22:54:24.657 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-09-20T22:54:24.657 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 25, 2024-09-20T22:54:24.657 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 777, 2024-09-20T22:54:24.657 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-09-20T22:54:24.657 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-09-20T22:54:24.658 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-09-20T22:54:24.658 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:54:24.658 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:24.658 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:54:24.658 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6813", 2024-09-20T22:54:24.658 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2192239945 2024-09-20T22:54:24.658 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:24.658 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:24.658 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:24.658 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-09-20T22:54:24.658 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:54:24.658 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:24.659 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:54:24.659 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6814", 2024-09-20T22:54:24.659 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2192239945 2024-09-20T22:54:24.659 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:24.659 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:24.659 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:24.659 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-09-20T22:54:24.659 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:54:24.659 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:24.659 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:54:24.659 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6816", 2024-09-20T22:54:24.659 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2192239945 2024-09-20T22:54:24.659 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:24.660 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:24.660 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:24.660 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-09-20T22:54:24.660 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:54:24.660 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:24.660 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:54:24.660 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6815", 2024-09-20T22:54:24.660 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2192239945 2024-09-20T22:54:24.660 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:24.660 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:24.660 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:24.660 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.84:6813/2192239945", 2024-09-20T22:54:24.660 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.84:6814/2192239945", 2024-09-20T22:54:24.661 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.84:6816/2192239945", 2024-09-20T22:54:24.661 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.84:6815/2192239945", 2024-09-20T22:54:24.661 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-09-20T22:54:24.661 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-09-20T22:54:24.661 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-09-20T22:54:24.661 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:24.661 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:24.661 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:24.661 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 4, 2024-09-20T22:54:24.661 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "a52c3b94-c5f3-490f-8ac8-09061b9b2abb", 2024-09-20T22:54:24.661 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-09-20T22:54:24.661 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-09-20T22:54:24.662 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-09-20T22:54:24.662 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-09-20T22:54:24.662 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-09-20T22:54:24.662 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-09-20T22:54:24.662 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 31, 2024-09-20T22:54:24.662 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 778, 2024-09-20T22:54:24.662 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-09-20T22:54:24.662 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-09-20T22:54:24.662 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-09-20T22:54:24.662 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:54:24.662 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:24.662 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:54:24.663 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.154:6800", 2024-09-20T22:54:24.663 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1793136002 2024-09-20T22:54:24.663 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:24.663 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:24.663 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:24.663 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-09-20T22:54:24.663 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:54:24.663 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:24.663 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:54:24.663 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.154:6801", 2024-09-20T22:54:24.663 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1793136002 2024-09-20T22:54:24.663 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:24.663 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:24.663 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:24.664 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-09-20T22:54:24.664 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:54:24.664 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:24.664 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:54:24.664 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.154:6803", 2024-09-20T22:54:24.664 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1793136002 2024-09-20T22:54:24.664 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:24.664 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:24.664 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:24.664 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-09-20T22:54:24.664 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:54:24.664 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:24.665 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:54:24.665 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.154:6802", 2024-09-20T22:54:24.665 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1793136002 2024-09-20T22:54:24.665 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:24.665 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:24.665 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:24.665 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.154:6800/1793136002", 2024-09-20T22:54:24.665 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.154:6801/1793136002", 2024-09-20T22:54:24.665 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.154:6803/1793136002", 2024-09-20T22:54:24.665 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.154:6802/1793136002", 2024-09-20T22:54:24.665 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-09-20T22:54:24.666 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-09-20T22:54:24.666 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-09-20T22:54:24.666 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:24.666 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:24.666 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:24.666 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 5, 2024-09-20T22:54:24.666 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "4fa0dc8e-25cc-46cd-96ff-33f20b6faec3", 2024-09-20T22:54:24.666 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-09-20T22:54:24.666 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-09-20T22:54:24.666 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-09-20T22:54:24.666 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-09-20T22:54:24.666 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-09-20T22:54:24.667 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-09-20T22:54:24.667 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 40, 2024-09-20T22:54:24.667 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 779, 2024-09-20T22:54:24.667 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-09-20T22:54:24.667 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-09-20T22:54:24.667 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-09-20T22:54:24.667 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:54:24.667 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:24.667 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:54:24.667 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.154:6804", 2024-09-20T22:54:24.667 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 493899927 2024-09-20T22:54:24.667 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:24.667 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:24.667 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:24.667 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-09-20T22:54:24.668 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:54:24.668 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:24.668 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:54:24.668 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.154:6805", 2024-09-20T22:54:24.668 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 493899927 2024-09-20T22:54:24.668 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:24.668 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:24.668 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:24.668 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-09-20T22:54:24.668 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:54:24.668 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:24.668 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:54:24.668 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.154:6807", 2024-09-20T22:54:24.668 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 493899927 2024-09-20T22:54:24.669 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:24.669 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:24.669 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:24.669 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-09-20T22:54:24.669 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:54:24.669 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:24.669 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:54:24.669 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.154:6806", 2024-09-20T22:54:24.669 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 493899927 2024-09-20T22:54:24.669 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:24.669 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:24.669 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:24.670 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.154:6804/493899927", 2024-09-20T22:54:24.670 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.154:6805/493899927", 2024-09-20T22:54:24.670 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.154:6807/493899927", 2024-09-20T22:54:24.670 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.154:6806/493899927", 2024-09-20T22:54:24.670 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-09-20T22:54:24.670 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-09-20T22:54:24.670 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-09-20T22:54:24.670 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:24.670 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:24.670 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:24.670 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 6, 2024-09-20T22:54:24.671 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "56788d74-810e-4827-a615-958c12f8a368", 2024-09-20T22:54:24.671 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-09-20T22:54:24.671 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-09-20T22:54:24.671 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-09-20T22:54:24.671 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-09-20T22:54:24.671 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-09-20T22:54:24.671 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-09-20T22:54:24.671 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 48, 2024-09-20T22:54:24.671 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 779, 2024-09-20T22:54:24.671 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-09-20T22:54:24.671 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-09-20T22:54:24.671 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-09-20T22:54:24.671 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:54:24.671 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:24.672 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:54:24.672 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.154:6808", 2024-09-20T22:54:24.672 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2375091033 2024-09-20T22:54:24.672 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:24.672 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:24.672 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:24.672 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-09-20T22:54:24.672 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:54:24.672 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:24.672 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:54:24.673 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.154:6809", 2024-09-20T22:54:24.673 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2375091033 2024-09-20T22:54:24.673 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:24.673 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:24.673 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:24.673 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-09-20T22:54:24.673 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:54:24.673 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:24.673 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:54:24.673 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.154:6811", 2024-09-20T22:54:24.673 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2375091033 2024-09-20T22:54:24.673 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:24.673 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:24.673 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:24.674 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-09-20T22:54:24.674 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:54:24.674 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:24.674 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:54:24.674 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.154:6810", 2024-09-20T22:54:24.674 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2375091033 2024-09-20T22:54:24.674 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:24.674 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:24.674 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:24.674 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.154:6808/2375091033", 2024-09-20T22:54:24.674 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.154:6809/2375091033", 2024-09-20T22:54:24.675 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.154:6811/2375091033", 2024-09-20T22:54:24.675 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.154:6810/2375091033", 2024-09-20T22:54:24.675 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-09-20T22:54:24.675 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-09-20T22:54:24.675 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-09-20T22:54:24.675 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:24.675 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:24.675 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:24.675 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 7, 2024-09-20T22:54:24.675 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "83503727-afc1-4b95-8b96-ecb63f3338c2", 2024-09-20T22:54:24.675 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-09-20T22:54:24.675 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-09-20T22:54:24.675 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-09-20T22:54:24.675 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-09-20T22:54:24.675 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-09-20T22:54:24.675 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-09-20T22:54:24.676 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 59, 2024-09-20T22:54:24.676 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 776, 2024-09-20T22:54:24.676 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-09-20T22:54:24.676 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-09-20T22:54:24.676 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-09-20T22:54:24.676 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:54:24.676 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:24.676 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:54:24.676 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.154:6812", 2024-09-20T22:54:24.676 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3717704491 2024-09-20T22:54:24.676 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:24.676 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:24.676 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:24.676 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-09-20T22:54:24.676 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:54:24.677 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:24.677 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:54:24.677 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.154:6813", 2024-09-20T22:54:24.677 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3717704491 2024-09-20T22:54:24.677 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:24.677 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:24.677 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:24.677 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-09-20T22:54:24.677 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:54:24.677 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:24.677 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:54:24.677 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.154:6815", 2024-09-20T22:54:24.677 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3717704491 2024-09-20T22:54:24.677 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:24.677 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:24.678 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:24.678 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-09-20T22:54:24.678 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:54:24.678 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:24.678 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:54:24.678 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.154:6814", 2024-09-20T22:54:24.678 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3717704491 2024-09-20T22:54:24.678 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:24.678 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:24.678 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:24.678 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.154:6812/3717704491", 2024-09-20T22:54:24.678 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.154:6813/3717704491", 2024-09-20T22:54:24.678 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.154:6815/3717704491", 2024-09-20T22:54:24.679 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.154:6814/3717704491", 2024-09-20T22:54:24.679 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-09-20T22:54:24.679 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-09-20T22:54:24.679 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-09-20T22:54:24.679 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:24.679 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:24.679 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2024-09-20T22:54:24.679 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_xinfo": [ 2024-09-20T22:54:24.679 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:24.679 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 0, 2024-09-20T22:54:24.679 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-09-20T22:54:24.679 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-09-20T22:54:24.679 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-09-20T22:54:24.679 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540701547738038271, 2024-09-20T22:54:24.680 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-09-20T22:54:24.680 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-09-20T22:54:06.520163+0000", 2024-09-20T22:54:24.680 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-09-20T22:54:24.680 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:24.680 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:24.680 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 1, 2024-09-20T22:54:24.680 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-09-20T22:54:24.680 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-09-20T22:54:24.680 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-09-20T22:54:24.680 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540701547738038271, 2024-09-20T22:54:24.680 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-09-20T22:54:24.680 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-09-20T22:53:42.234428+0000", 2024-09-20T22:54:24.680 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-09-20T22:54:24.680 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:24.681 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:24.681 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 2, 2024-09-20T22:54:24.681 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-09-20T22:54:24.681 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-09-20T22:54:24.681 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-09-20T22:54:24.681 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540701547738038271, 2024-09-20T22:54:24.681 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-09-20T22:54:24.681 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-09-20T22:53:33.270476+0000", 2024-09-20T22:54:24.681 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-09-20T22:54:24.681 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:24.681 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:24.681 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 3, 2024-09-20T22:54:24.681 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-09-20T22:54:24.681 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-09-20T22:54:24.682 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-09-20T22:54:24.682 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540701547738038271, 2024-09-20T22:54:24.682 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-09-20T22:54:24.682 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-09-20T22:53:12.550674+0000", 2024-09-20T22:54:24.682 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-09-20T22:54:24.682 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:24.682 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:24.682 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 4, 2024-09-20T22:54:24.682 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-09-20T22:54:24.682 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-09-20T22:54:24.682 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-09-20T22:54:24.682 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540701547738038271, 2024-09-20T22:54:24.682 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-09-20T22:54:24.683 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-09-20T22:54:06.316620+0000", 2024-09-20T22:54:24.683 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-09-20T22:54:24.683 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:24.683 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:24.683 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 5, 2024-09-20T22:54:24.683 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-09-20T22:54:24.683 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-09-20T22:54:24.683 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-09-20T22:54:24.683 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540701547738038271, 2024-09-20T22:54:24.683 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-09-20T22:54:24.683 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-09-20T22:53:26.920706+0000", 2024-09-20T22:54:24.683 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-09-20T22:54:24.684 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:24.684 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:24.684 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 6, 2024-09-20T22:54:24.684 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-09-20T22:54:24.684 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-09-20T22:54:24.684 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-09-20T22:54:24.684 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540701547738038271, 2024-09-20T22:54:24.684 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-09-20T22:54:24.684 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-09-20T22:53:20.439087+0000", 2024-09-20T22:54:24.684 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-09-20T22:54:24.684 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:24.684 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:24.684 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 7, 2024-09-20T22:54:24.684 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-09-20T22:54:24.685 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-09-20T22:54:24.685 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-09-20T22:54:24.685 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540701547738038271, 2024-09-20T22:54:24.685 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-09-20T22:54:24.685 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-09-20T22:54:00.021655+0000", 2024-09-20T22:54:24.685 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-09-20T22:54:24.685 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:24.685 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2024-09-20T22:54:24.685 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_upmap": [ 2024-09-20T22:54:24.685 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:24.685 INFO:teuthology.orchestra.run.smithi084.stdout: "pgid": "1.9", 2024-09-20T22:54:24.685 INFO:teuthology.orchestra.run.smithi084.stdout: "osds": [ 2024-09-20T22:54:24.685 INFO:teuthology.orchestra.run.smithi084.stdout: 7, 2024-09-20T22:54:24.685 INFO:teuthology.orchestra.run.smithi084.stdout: 3, 2024-09-20T22:54:24.685 INFO:teuthology.orchestra.run.smithi084.stdout: 5 2024-09-20T22:54:24.686 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:24.686 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:24.686 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:24.686 INFO:teuthology.orchestra.run.smithi084.stdout: "pgid": "6.e", 2024-09-20T22:54:24.686 INFO:teuthology.orchestra.run.smithi084.stdout: "osds": [ 2024-09-20T22:54:24.686 INFO:teuthology.orchestra.run.smithi084.stdout: 1, 2024-09-20T22:54:24.686 INFO:teuthology.orchestra.run.smithi084.stdout: 2, 2024-09-20T22:54:24.686 INFO:teuthology.orchestra.run.smithi084.stdout: 5 2024-09-20T22:54:24.686 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:24.686 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:24.686 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:24.686 INFO:teuthology.orchestra.run.smithi084.stdout: "pgid": "6.1e", 2024-09-20T22:54:24.686 INFO:teuthology.orchestra.run.smithi084.stdout: "osds": [ 2024-09-20T22:54:24.686 INFO:teuthology.orchestra.run.smithi084.stdout: 0, 2024-09-20T22:54:24.686 INFO:teuthology.orchestra.run.smithi084.stdout: 1, 2024-09-20T22:54:24.687 INFO:teuthology.orchestra.run.smithi084.stdout: 3 2024-09-20T22:54:24.687 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:24.687 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:24.687 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2024-09-20T22:54:24.687 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_upmap_items": [ 2024-09-20T22:54:24.687 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:24.687 INFO:teuthology.orchestra.run.smithi084.stdout: "pgid": "3.a", 2024-09-20T22:54:24.687 INFO:teuthology.orchestra.run.smithi084.stdout: "mappings": [ 2024-09-20T22:54:24.687 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:24.687 INFO:teuthology.orchestra.run.smithi084.stdout: "from": 6, 2024-09-20T22:54:24.687 INFO:teuthology.orchestra.run.smithi084.stdout: "to": 2 2024-09-20T22:54:24.687 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:24.687 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:24.687 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:24.688 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:24.688 INFO:teuthology.orchestra.run.smithi084.stdout: "pgid": "4.9", 2024-09-20T22:54:24.688 INFO:teuthology.orchestra.run.smithi084.stdout: "mappings": [ 2024-09-20T22:54:24.688 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:24.688 INFO:teuthology.orchestra.run.smithi084.stdout: "from": 3, 2024-09-20T22:54:24.688 INFO:teuthology.orchestra.run.smithi084.stdout: "to": 2 2024-09-20T22:54:24.688 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:24.688 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:24.688 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:24.688 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:24.688 INFO:teuthology.orchestra.run.smithi084.stdout: "pgid": "4.11", 2024-09-20T22:54:24.688 INFO:teuthology.orchestra.run.smithi084.stdout: "mappings": [ 2024-09-20T22:54:24.688 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:24.688 INFO:teuthology.orchestra.run.smithi084.stdout: "from": 3, 2024-09-20T22:54:24.689 INFO:teuthology.orchestra.run.smithi084.stdout: "to": 5 2024-09-20T22:54:24.689 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:24.689 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:24.689 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:24.689 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2024-09-20T22:54:24.689 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_upmap_primaries": [], 2024-09-20T22:54:24.689 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_temp": [ 2024-09-20T22:54:24.689 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:24.689 INFO:teuthology.orchestra.run.smithi084.stdout: "pgid": "7.2", 2024-09-20T22:54:24.689 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:24.690 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:54:24.690 INFO:teuthology.orchestra.run.smithi084.stdout: "osds": [ 2024-09-20T22:54:24.690 INFO:teuthology.orchestra.run.smithi084.stdout: 3, 2024-09-20T22:54:24.690 INFO:teuthology.orchestra.run.smithi084.stdout: 2, 2024-09-20T22:54:24.690 INFO:teuthology.orchestra.run.smithi084.stdout: 7 2024-09-20T22:54:24.690 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:24.690 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:24.690 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2024-09-20T22:54:24.691 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_temp": [], 2024-09-20T22:54:24.691 INFO:teuthology.orchestra.run.smithi084.stdout: "blocklist": { 2024-09-20T22:54:24.691 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/542020545": "2024-09-21T22:42:07.596045+0000", 2024-09-20T22:54:24.691 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/25873236": "2024-09-21T22:36:54.917812+0000", 2024-09-20T22:54:24.691 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:6800/820908470": "2024-09-21T22:36:54.917812+0000", 2024-09-20T22:54:24.691 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/567614908": "2024-09-21T22:36:54.917812+0000", 2024-09-20T22:54:24.691 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/2963622459": "2024-09-21T22:42:07.596045+0000", 2024-09-20T22:54:24.691 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/1176849276": "2024-09-21T22:37:17.690104+0000", 2024-09-20T22:54:24.691 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:6800/2932420050": "2024-09-21T22:37:17.690104+0000", 2024-09-20T22:54:24.691 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/3879263635": "2024-09-21T22:37:17.690104+0000", 2024-09-20T22:54:24.691 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/969472279": "2024-09-21T22:36:54.917812+0000", 2024-09-20T22:54:24.691 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/1361143214": "2024-09-21T22:37:17.690104+0000", 2024-09-20T22:54:24.691 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/3353126955": "2024-09-21T22:42:07.596045+0000", 2024-09-20T22:54:24.692 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:6800/4024278029": "2024-09-21T22:42:07.596045+0000", 2024-09-20T22:54:24.692 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/547902262": "2024-09-21T22:42:07.596045+0000", 2024-09-20T22:54:24.692 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/1242929317": "2024-09-21T22:42:07.596045+0000" 2024-09-20T22:54:24.692 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:24.692 INFO:teuthology.orchestra.run.smithi084.stdout: "range_blocklist": {}, 2024-09-20T22:54:24.692 INFO:teuthology.orchestra.run.smithi084.stdout: "erasure_code_profiles": { 2024-09-20T22:54:24.692 INFO:teuthology.orchestra.run.smithi084.stdout: "default": { 2024-09-20T22:54:24.692 INFO:teuthology.orchestra.run.smithi084.stdout: "crush-failure-domain": "osd", 2024-09-20T22:54:24.692 INFO:teuthology.orchestra.run.smithi084.stdout: "k": "2", 2024-09-20T22:54:24.692 INFO:teuthology.orchestra.run.smithi084.stdout: "m": "1", 2024-09-20T22:54:24.692 INFO:teuthology.orchestra.run.smithi084.stdout: "plugin": "jerasure", 2024-09-20T22:54:24.692 INFO:teuthology.orchestra.run.smithi084.stdout: "technique": "reed_sol_van" 2024-09-20T22:54:24.692 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:24.692 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:24.693 INFO:teuthology.orchestra.run.smithi084.stdout: "removed_snaps_queue": [ 2024-09-20T22:54:24.693 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:24.693 INFO:teuthology.orchestra.run.smithi084.stdout: "pool": 7, 2024-09-20T22:54:24.693 INFO:teuthology.orchestra.run.smithi084.stdout: "snaps": [ 2024-09-20T22:54:24.693 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:24.693 INFO:teuthology.orchestra.run.smithi084.stdout: "begin": 287, 2024-09-20T22:54:24.693 INFO:teuthology.orchestra.run.smithi084.stdout: "length": 5 2024-09-20T22:54:24.693 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:24.693 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:24.693 INFO:teuthology.orchestra.run.smithi084.stdout: "begin": 293, 2024-09-20T22:54:24.693 INFO:teuthology.orchestra.run.smithi084.stdout: "length": 9 2024-09-20T22:54:24.693 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:24.693 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:24.694 INFO:teuthology.orchestra.run.smithi084.stdout: "begin": 304, 2024-09-20T22:54:24.694 INFO:teuthology.orchestra.run.smithi084.stdout: "length": 9 2024-09-20T22:54:24.694 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:24.694 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:24.694 INFO:teuthology.orchestra.run.smithi084.stdout: "begin": 315, 2024-09-20T22:54:24.694 INFO:teuthology.orchestra.run.smithi084.stdout: "length": 3 2024-09-20T22:54:24.694 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:24.694 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:24.694 INFO:teuthology.orchestra.run.smithi084.stdout: "begin": 319, 2024-09-20T22:54:24.694 INFO:teuthology.orchestra.run.smithi084.stdout: "length": 3 2024-09-20T22:54:24.694 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:24.694 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:24.694 INFO:teuthology.orchestra.run.smithi084.stdout: "begin": 323, 2024-09-20T22:54:24.694 INFO:teuthology.orchestra.run.smithi084.stdout: "length": 2 2024-09-20T22:54:24.695 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:24.695 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:24.695 INFO:teuthology.orchestra.run.smithi084.stdout: "begin": 326, 2024-09-20T22:54:24.695 INFO:teuthology.orchestra.run.smithi084.stdout: "length": 2 2024-09-20T22:54:24.695 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:24.695 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:24.695 INFO:teuthology.orchestra.run.smithi084.stdout: "begin": 331, 2024-09-20T22:54:24.695 INFO:teuthology.orchestra.run.smithi084.stdout: "length": 1 2024-09-20T22:54:24.695 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:24.695 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:24.695 INFO:teuthology.orchestra.run.smithi084.stdout: "begin": 333, 2024-09-20T22:54:24.695 INFO:teuthology.orchestra.run.smithi084.stdout: "length": 1 2024-09-20T22:54:24.695 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:24.695 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:24.696 INFO:teuthology.orchestra.run.smithi084.stdout: "begin": 336, 2024-09-20T22:54:24.696 INFO:teuthology.orchestra.run.smithi084.stdout: "length": 1 2024-09-20T22:54:24.696 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:24.696 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:24.696 INFO:teuthology.orchestra.run.smithi084.stdout: "begin": 338, 2024-09-20T22:54:24.696 INFO:teuthology.orchestra.run.smithi084.stdout: "length": 1 2024-09-20T22:54:24.696 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:24.696 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:24.696 INFO:teuthology.orchestra.run.smithi084.stdout: "begin": 340, 2024-09-20T22:54:24.696 INFO:teuthology.orchestra.run.smithi084.stdout: "length": 1 2024-09-20T22:54:24.696 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:24.696 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:24.696 INFO:teuthology.orchestra.run.smithi084.stdout: "begin": 345, 2024-09-20T22:54:24.696 INFO:teuthology.orchestra.run.smithi084.stdout: "length": 1 2024-09-20T22:54:24.697 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:24.697 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:24.697 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:24.697 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2024-09-20T22:54:24.697 INFO:teuthology.orchestra.run.smithi084.stdout: "new_removed_snaps": [], 2024-09-20T22:54:24.697 INFO:teuthology.orchestra.run.smithi084.stdout: "new_purged_snaps": [], 2024-09-20T22:54:24.697 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_node_flags": {}, 2024-09-20T22:54:24.697 INFO:teuthology.orchestra.run.smithi084.stdout: "device_class_flags": {}, 2024-09-20T22:54:24.697 INFO:teuthology.orchestra.run.smithi084.stdout: "stretch_mode": { 2024-09-20T22:54:24.697 INFO:teuthology.orchestra.run.smithi084.stdout: "stretch_mode_enabled": false, 2024-09-20T22:54:24.697 INFO:teuthology.orchestra.run.smithi084.stdout: "stretch_bucket_count": 0, 2024-09-20T22:54:24.697 INFO:teuthology.orchestra.run.smithi084.stdout: "degraded_stretch_mode": 0, 2024-09-20T22:54:24.698 INFO:teuthology.orchestra.run.smithi084.stdout: "recovering_stretch_mode": 0, 2024-09-20T22:54:24.698 INFO:teuthology.orchestra.run.smithi084.stdout: "stretch_mode_bucket": 0 2024-09-20T22:54:24.698 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:24.698 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-20T22:54:24.698 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:54:24.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:24 smithi154 ceph-mon[33264]: 2.2 scrub starts 2024-09-20T22:54:24.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:24 smithi154 ceph-mon[33264]: 2.2 scrub ok 2024-09-20T22:54:24.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:24 smithi154 ceph-mon[33264]: osdmap e784: 8 total, 8 up, 8 in 2024-09-20T22:54:24.791 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:54:24.817 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:24.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:24.818 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:54:24.918 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:54:24.940 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:24.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:24.941 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:54:25.041 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:54:25.062 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:25.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:25.063 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:54:25.164 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:54:25.190 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:25.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:25.191 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:54:25.292 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:54:25.315 INFO:tasks.rados.rados.0.smithi084.stdout:3119: done (0 left) 2024-09-20T22:54:25.315 INFO:tasks.rados.rados.0.smithi084.stdout:3120: write (excl) oid 35 current snap is 356 2024-09-20T22:54:25.316 INFO:tasks.rados.rados.0.smithi084.stdout:3120: seq_num 1071 ranges {400297=481099,1639156=361141} 2024-09-20T22:54:25.320 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:25.321 INFO:tasks.rados.rados.0.smithi084.stdout:3120: writing smithi08477203-35 from 400297 to 881396 tid 1 2024-09-20T22:54:25.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:25.322 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:54:25.328 INFO:tasks.rados.rados.0.smithi084.stdout:3120: writing smithi08477203-35 from 1639156 to 2000297 tid 2 2024-09-20T22:54:25.328 INFO:tasks.rados.rados.0.smithi084.stdout:3121: rollback oid 9 current snap is 356 2024-09-20T22:54:25.328 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 9 to 348 2024-09-20T22:54:25.328 INFO:tasks.rados.rados.0.smithi084.stdout:3120: finishing write tid 1 to smithi08477203-35 2024-09-20T22:54:25.329 INFO:tasks.rados.rados.0.smithi084.stdout:3122: snap_remove snap 353 2024-09-20T22:54:25.423 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:54:25.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:25.447 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:25.447 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:54:25.547 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:54:25.568 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:25.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:25.569 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:54:25.670 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:54:25.691 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:25.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:25.692 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:54:25.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:25 smithi154 ceph-mon[33264]: pgmap v1077: 158 pgs: 3 active+clean+snaptrim, 1 active+remapped+backfill_toofull, 154 active+clean; 538 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 13 MiB/s rd, 2.6 MiB/s wr, 23 op/s; 15/1326 objects misplaced (1.131%); 7.0 MiB/s, 2 objects/s recovering 2024-09-20T22:54:25.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:25 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/3692172328' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-20T22:54:25.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:25 smithi154 ceph-mon[33264]: osdmap e785: 8 total, 8 up, 8 in 2024-09-20T22:54:25.793 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:54:25.819 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:25.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:25.820 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:54:25.847 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:25 smithi084 ceph-mon[40024]: pgmap v1077: 158 pgs: 3 active+clean+snaptrim, 1 active+remapped+backfill_toofull, 154 active+clean; 538 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 13 MiB/s rd, 2.6 MiB/s wr, 23 op/s; 15/1326 objects misplaced (1.131%); 7.0 MiB/s, 2 objects/s recovering 2024-09-20T22:54:25.847 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:25 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/3692172328' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-20T22:54:25.847 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:25 smithi084 ceph-mon[40024]: osdmap e785: 8 total, 8 up, 8 in 2024-09-20T22:54:25.847 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:25 smithi084 ceph-mon[29774]: pgmap v1077: 158 pgs: 3 active+clean+snaptrim, 1 active+remapped+backfill_toofull, 154 active+clean; 538 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 13 MiB/s rd, 2.6 MiB/s wr, 23 op/s; 15/1326 objects misplaced (1.131%); 7.0 MiB/s, 2 objects/s recovering 2024-09-20T22:54:25.847 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:25 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/3692172328' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-20T22:54:25.847 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:25 smithi084 ceph-mon[29774]: osdmap e785: 8 total, 8 up, 8 in 2024-09-20T22:54:25.921 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:54:25.947 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:25.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:25.948 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:54:25.996 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-20T22:54:26.049 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:54:26.108 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:26.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:26.109 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:54:26.210 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:54:26.231 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:26.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:26.232 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:54:26.317 INFO:tasks.rados.rados.0.smithi084.stdout:3122: done (2 left) 2024-09-20T22:54:26.317 INFO:tasks.rados.rados.0.smithi084.stdout:3123: copy_from oid 4 from oid 2 current snap is 356 2024-09-20T22:54:26.317 INFO:tasks.rados.rados.0.smithi084.stdout:3124: write (excl) oid 28 current snap is 356 2024-09-20T22:54:26.317 INFO:tasks.rados.rados.0.smithi084.stdout:3124: seq_num 1072 ranges {417104=672701,1813991=203113} 2024-09-20T22:54:26.323 INFO:tasks.rados.rados.0.smithi084.stdout:3124: writing smithi08477203-28 from 417104 to 1089805 tid 1 2024-09-20T22:54:26.329 INFO:tasks.rados.rados.0.smithi084.stdout:3124: writing smithi08477203-28 from 1813991 to 2017104 tid 2 2024-09-20T22:54:26.329 INFO:tasks.rados.rados.0.smithi084.stdout:3125: delete oid 41 current snap is 356 2024-09-20T22:54:26.329 INFO:tasks.rados.rados.0.smithi084.stdout:3121: finishing rollback tid 1 to smithi08477203-9 2024-09-20T22:54:26.329 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 9 v 0 (ObjNum 129 snap 0 seq_num 2091705744) dirty dne 2024-09-20T22:54:26.329 INFO:tasks.rados.rados.0.smithi084.stdout:3120: finishing write tid 2 to smithi08477203-35 2024-09-20T22:54:26.329 INFO:tasks.rados.rados.0.smithi084.stdout:3120: finishing write tid 3 to smithi08477203-35 2024-09-20T22:54:26.329 INFO:tasks.rados.rados.0.smithi084.stdout:3120: finishing write tid 4 to smithi08477203-35 2024-09-20T22:54:26.329 INFO:tasks.rados.rados.0.smithi084.stdout:3120: oid 35 updating version 0 to 1025 2024-09-20T22:54:26.329 INFO:tasks.rados.rados.0.smithi084.stdout:3120: oid 35 version 1025 is already newer than 1024 2024-09-20T22:54:26.329 INFO:tasks.rados.rados.0.smithi084.stdout:3120: oid 35 updating version 1025 to 1026 2024-09-20T22:54:26.329 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 35 v 1026 (ObjNum 1071 snap 356 seq_num 1071) dirty exists 2024-09-20T22:54:26.329 INFO:tasks.rados.rados.0.smithi084.stdout:3120: left oid 35 (ObjNum 1071 snap 356 seq_num 1071) 2024-09-20T22:54:26.329 INFO:tasks.rados.rados.0.smithi084.stdout:3124: finishing write tid 1 to smithi08477203-28 2024-09-20T22:54:26.330 INFO:tasks.rados.rados.0.smithi084.stdout:3120: done (4 left) 2024-09-20T22:54:26.330 INFO:tasks.rados.rados.0.smithi084.stdout:3121: done (3 left) 2024-09-20T22:54:26.330 INFO:tasks.rados.rados.0.smithi084.stdout:3125: done (2 left) 2024-09-20T22:54:26.331 INFO:tasks.rados.rados.0.smithi084.stdout:3126: snap_create 2024-09-20T22:54:26.333 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:54:26.339 INFO:tasks.rados.rados.0.smithi084.stdout:3123: finishing copy_from racing read to smithi08477203-4 2024-09-20T22:54:26.339 INFO:tasks.rados.rados.0.smithi084.stdout:3124: finishing write tid 2 to smithi08477203-28 2024-09-20T22:54:26.339 INFO:tasks.rados.rados.0.smithi084.stdout:3124: finishing write tid 3 to smithi08477203-28 2024-09-20T22:54:26.342 INFO:tasks.rados.rados.0.smithi084.stdout:3124: finishing write tid 4 to smithi08477203-28 2024-09-20T22:54:26.342 INFO:tasks.rados.rados.0.smithi084.stdout:3124: oid 28 updating version 0 to 2531 2024-09-20T22:54:26.342 INFO:tasks.rados.rados.0.smithi084.stdout:3124: oid 28 updating version 2531 to 2532 2024-09-20T22:54:26.342 INFO:tasks.rados.rados.0.smithi084.stdout:3124: oid 28 version 2532 is already newer than 2530 2024-09-20T22:54:26.342 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 28 v 2532 (ObjNum 1072 snap 356 seq_num 1072) dirty exists 2024-09-20T22:54:26.342 INFO:tasks.rados.rados.0.smithi084.stdout:3124: left oid 28 (ObjNum 1072 snap 356 seq_num 1072) 2024-09-20T22:54:26.343 INFO:tasks.rados.rados.0.smithi084.stdout:3123: finishing copy_from to smithi08477203-4 2024-09-20T22:54:26.343 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 4 v 1835 (ObjNum 918 snap 302 seq_num 918) dirty exists 2024-09-20T22:54:26.357 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:26.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:26.358 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:54:26.459 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:54:26.470 INFO:tasks.thrashosds.thrasher:j is {'epoch': 784, 'fsid': 'a9c0bc70-77a0-11ef-baf5-efdc52797490', 'created': '2024-09-20T22:36:30.240626+0000', 'modified': '2024-09-20T22:54:24.306583+0000', 'last_up_change': '2024-09-20T22:41:27.098565+0000', 'last_in_change': '2024-09-20T22:54:16.306377+0000', 'flags': 'noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5801984, 'flags_set': ['noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 18, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 7, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-20T22:39:21.984381+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 11, 'pg_placement_num': 11, 'pg_placement_num_target': 11, 'pg_num_target': 11, 'pg_num_pending': 11, 'last_pg_merge_meta': {'source_pgid': '1.b', 'ready_epoch': 299, 'last_epoch_started': 298, 'last_epoch_clean': 298, 'source_version': "77'128", 'target_version': "77'128"}, 'last_change': '769', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '299', 'last_force_op_resend_preluminous': '297', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.180000066757202, 'score_stable': 2.180000066757202, 'optimal_score': 1, 'raw_score_acting': 2.180000066757202, 'raw_score_stable': 2.180000066757202, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-09-20T22:41:35.820081+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '69', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 69, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.6500000953674316, 'score_stable': 2.6500000953674316, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 2.3299999237060547, 'raw_score_stable': 2.3299999237060547, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-09-20T22:41:36.546389+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-09-20T22:41:38.454640+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '70', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-09-20T22:41:40.407129+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '72', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-09-20T22:41:42.476716+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '74', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'unique_pool_0', 'create_time': '2024-09-20T22:42:51.354358+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '7.10', 'ready_epoch': 484, 'last_epoch_started': 483, 'last_epoch_clean': 483, 'source_version': "480'172", 'target_version': "480'253"}, 'last_change': '784', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '484', 'last_force_op_resend_preluminous': '482', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 357, 'snap_epoch': 784, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': 'ea36a489-6f91-4b2f-b784-b90506b9805d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 779, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6801', 'nonce': 268084626}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6802', 'nonce': 268084626}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6804', 'nonce': 268084626}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6803', 'nonce': 268084626}]}, 'public_addr': '172.21.15.84:6801/268084626', 'cluster_addr': '172.21.15.84:6802/268084626', 'heartbeat_back_addr': '172.21.15.84:6804/268084626', 'heartbeat_front_addr': '172.21.15.84:6803/268084626', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'c8649d2f-27ef-47f8-bc94-f1cd9e62bdb8', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 779, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6805', 'nonce': 418374136}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6806', 'nonce': 418374136}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6808', 'nonce': 418374136}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6807', 'nonce': 418374136}]}, 'public_addr': '172.21.15.84:6805/418374136', 'cluster_addr': '172.21.15.84:6806/418374136', 'heartbeat_back_addr': '172.21.15.84:6808/418374136', 'heartbeat_front_addr': '172.21.15.84:6807/418374136', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'c728eb48-a0f0-4de4-a651-7f3bec867280', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 778, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6809', 'nonce': 3977168611}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6810', 'nonce': 3977168611}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6812', 'nonce': 3977168611}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6811', 'nonce': 3977168611}]}, 'public_addr': '172.21.15.84:6809/3977168611', 'cluster_addr': '172.21.15.84:6810/3977168611', 'heartbeat_back_addr': '172.21.15.84:6812/3977168611', 'heartbeat_front_addr': '172.21.15.84:6811/3977168611', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '573f44eb-0853-4630-a7d8-9caaeb256f9b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 25, 'up_thru': 777, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6813', 'nonce': 2192239945}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6814', 'nonce': 2192239945}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6816', 'nonce': 2192239945}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6815', 'nonce': 2192239945}]}, 'public_addr': '172.21.15.84:6813/2192239945', 'cluster_addr': '172.21.15.84:6814/2192239945', 'heartbeat_back_addr': '172.21.15.84:6816/2192239945', 'heartbeat_front_addr': '172.21.15.84:6815/2192239945', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'a52c3b94-c5f3-490f-8ac8-09061b9b2abb', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 778, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6800', 'nonce': 1793136002}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6801', 'nonce': 1793136002}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6803', 'nonce': 1793136002}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6802', 'nonce': 1793136002}]}, 'public_addr': '172.21.15.154:6800/1793136002', 'cluster_addr': '172.21.15.154:6801/1793136002', 'heartbeat_back_addr': '172.21.15.154:6803/1793136002', 'heartbeat_front_addr': '172.21.15.154:6802/1793136002', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '4fa0dc8e-25cc-46cd-96ff-33f20b6faec3', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 40, 'up_thru': 779, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6804', 'nonce': 493899927}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6805', 'nonce': 493899927}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6807', 'nonce': 493899927}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6806', 'nonce': 493899927}]}, 'public_addr': '172.21.15.154:6804/493899927', 'cluster_addr': '172.21.15.154:6805/493899927', 'heartbeat_back_addr': '172.21.15.154:6807/493899927', 'heartbeat_front_addr': '172.21.15.154:6806/493899927', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '56788d74-810e-4827-a615-958c12f8a368', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 48, 'up_thru': 779, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6808', 'nonce': 2375091033}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6809', 'nonce': 2375091033}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6811', 'nonce': 2375091033}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6810', 'nonce': 2375091033}]}, 'public_addr': '172.21.15.154:6808/2375091033', 'cluster_addr': '172.21.15.154:6809/2375091033', 'heartbeat_back_addr': '172.21.15.154:6811/2375091033', 'heartbeat_front_addr': '172.21.15.154:6810/2375091033', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '83503727-afc1-4b95-8b96-ecb63f3338c2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 59, 'up_thru': 776, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6812', 'nonce': 3717704491}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6813', 'nonce': 3717704491}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6815', 'nonce': 3717704491}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6814', 'nonce': 3717704491}]}, 'public_addr': '172.21.15.154:6812/3717704491', 'cluster_addr': '172.21.15.154:6813/3717704491', 'heartbeat_back_addr': '172.21.15.154:6815/3717704491', 'heartbeat_front_addr': '172.21.15.154:6814/3717704491', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-20T22:54:06.520163+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-20T22:53:42.234428+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-20T22:53:33.270476+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-20T22:53:12.550674+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-20T22:54:06.316620+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-20T22:53:26.920706+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-20T22:53:20.439087+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-20T22:54:00.021655+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '1.9', 'osds': [7, 3, 5]}, {'pgid': '6.e', 'osds': [1, 2, 5]}, {'pgid': '6.1e', 'osds': [0, 1, 3]}], 'pg_upmap_items': [{'pgid': '3.a', 'mappings': [{'from': 6, 'to': 2}]}, {'pgid': '4.9', 'mappings': [{'from': 3, 'to': 2}]}, {'pgid': '4.11', 'mappings': [{'from': 3, 'to': 5}]}], 'pg_upmap_primaries': [], 'pg_temp': [{'pgid': '7.2', 'osds': [3, 2, 7]}], 'primary_temp': [], 'blocklist': {'172.21.15.84:0/542020545': '2024-09-21T22:42:07.596045+0000', '172.21.15.84:0/25873236': '2024-09-21T22:36:54.917812+0000', '172.21.15.84:6800/820908470': '2024-09-21T22:36:54.917812+0000', '172.21.15.84:0/567614908': '2024-09-21T22:36:54.917812+0000', '172.21.15.84:0/2963622459': '2024-09-21T22:42:07.596045+0000', '172.21.15.84:0/1176849276': '2024-09-21T22:37:17.690104+0000', '172.21.15.84:6800/2932420050': '2024-09-21T22:37:17.690104+0000', '172.21.15.84:0/3879263635': '2024-09-21T22:37:17.690104+0000', '172.21.15.84:0/969472279': '2024-09-21T22:36:54.917812+0000', '172.21.15.84:0/1361143214': '2024-09-21T22:37:17.690104+0000', '172.21.15.84:0/3353126955': '2024-09-21T22:42:07.596045+0000', '172.21.15.84:6800/4024278029': '2024-09-21T22:42:07.596045+0000', '172.21.15.84:0/547902262': '2024-09-21T22:42:07.596045+0000', '172.21.15.84:0/1242929317': '2024-09-21T22:42:07.596045+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [{'pool': 7, 'snaps': [{'begin': 287, 'length': 5}, {'begin': 293, 'length': 9}, {'begin': 304, 'length': 9}, {'begin': 315, 'length': 3}, {'begin': 319, 'length': 3}, {'begin': 323, 'length': 2}, {'begin': 326, 'length': 2}, {'begin': 331, 'length': 1}, {'begin': 333, 'length': 1}, {'begin': 336, 'length': 1}, {'begin': 338, 'length': 1}, {'begin': 340, 'length': 1}, {'begin': 345, 'length': 1}]}], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-20T22:54:26.471 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph pg dump --format=json 2024-09-20T22:54:26.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:26.481 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:26.481 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:54:26.582 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:54:26.607 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:26.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:26.608 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:54:26.680 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-20T22:54:26.680 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2024-09-20T22:54:26.680 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2024-09-20T22:54:26.680 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [ 2024-09-20T22:54:26.680 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.680 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:1583 7.18 7:1e9e2524:::smithi08477203-5:head [copy-from ver 154] snapc 35=[35] ondisk+write+known_if_redirected+supports_pool_eio e201)", 2024-09-20T22:54:26.680 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:44:40.229595+0000", 2024-09-20T22:54:26.680 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 586.44214934000001, 2024-09-20T22:54:26.680 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.026717541000000001, 2024-09-20T22:54:26.680 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:54:26.680 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:54:26.681 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:54:26.681 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:54:26.681 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:54:26.681 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:54:26.681 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 1583 2024-09-20T22:54:26.681 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.681 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:54:26.681 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.681 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:54:26.681 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:40.229595+0000", 2024-09-20T22:54:26.681 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:54:26.681 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.682 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.682 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:54:26.682 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:40.229595+0000", 2024-09-20T22:54:26.682 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:54:26.682 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.682 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.682 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:54:26.682 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:40.229596+0000", 2024-09-20T22:54:26.682 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.68e-07 2024-09-20T22:54:26.682 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.682 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.682 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:54:26.682 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:40.229600+0000", 2024-09-20T22:54:26.683 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.805e-06 2024-09-20T22:54:26.683 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.683 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.683 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:54:26.683 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:40.229606+0000", 2024-09-20T22:54:26.683 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.4300000000000003e-06 2024-09-20T22:54:26.683 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.683 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.683 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:54:26.683 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:40.229611+0000", 2024-09-20T22:54:26.683 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.0250000000000002e-06 2024-09-20T22:54:26.683 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.684 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.684 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:54:26.684 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:40.229661+0000", 2024-09-20T22:54:26.684 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.0343999999999998e-05 2024-09-20T22:54:26.684 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.684 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.684 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:54:26.684 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:40.229737+0000", 2024-09-20T22:54:26.684 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.5444000000000005e-05 2024-09-20T22:54:26.684 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.684 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.684 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 6,7", 2024-09-20T22:54:26.684 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:40.244365+0000", 2024-09-20T22:54:26.684 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014628238999999999 2024-09-20T22:54:26.685 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.685 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.685 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:54:26.685 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:40.249487+0000", 2024-09-20T22:54:26.685 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0051221799999999996 2024-09-20T22:54:26.685 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.685 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.685 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:54:26.685 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:40.252864+0000", 2024-09-20T22:54:26.685 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0033763830000000002 2024-09-20T22:54:26.685 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.685 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.685 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:54:26.685 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:40.256237+0000", 2024-09-20T22:54:26.685 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0033731550000000001 2024-09-20T22:54:26.686 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.686 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.686 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:54:26.686 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:40.256277+0000", 2024-09-20T22:54:26.686 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.0092999999999997e-05 2024-09-20T22:54:26.686 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.686 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.686 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:54:26.686 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:44:40.256313+0000", 2024-09-20T22:54:26.686 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.5874999999999998e-05 2024-09-20T22:54:26.686 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:26.686 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:26.686 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:26.686 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.686 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.687 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:4311 7.17 7:ed54065f:::smithi08477203-12:head [append 0~0] snapc b1=[b1,b0,af,ae,ad,ab,a9,a7,a6,a5,9f,9e,9d,9c,9a,96,95,8e,87,86] ondisk+write+known_if_redirected+supports_pool_eio e433)", 2024-09-20T22:54:26.687 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:48:33.213556+0000", 2024-09-20T22:54:26.687 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 353.45818817000003, 2024-09-20T22:54:26.687 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0280906110000001, 2024-09-20T22:54:26.687 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:54:26.687 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:54:26.687 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:54:26.687 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:54:26.687 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:54:26.687 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:54:26.687 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 4311 2024-09-20T22:54:26.687 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.687 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:54:26.687 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.687 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:54:26.688 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:33.213556+0000", 2024-09-20T22:54:26.688 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:54:26.688 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.688 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.688 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:54:26.688 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:33.213556+0000", 2024-09-20T22:54:26.688 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:54:26.688 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.688 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.688 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:54:26.688 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:33.213558+0000", 2024-09-20T22:54:26.688 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.4470000000000001e-06 2024-09-20T22:54:26.688 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.688 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.689 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:54:26.689 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:33.213563+0000", 2024-09-20T22:54:26.689 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.6840000000000004e-06 2024-09-20T22:54:26.689 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.689 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.689 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:54:26.689 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:33.213573+0000", 2024-09-20T22:54:26.689 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0873000000000001e-05 2024-09-20T22:54:26.689 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.689 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.689 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:54:26.689 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:33.213584+0000", 2024-09-20T22:54:26.689 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0852e-05 2024-09-20T22:54:26.689 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.689 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.690 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:54:26.690 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:33.213633+0000", 2024-09-20T22:54:26.690 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.8683999999999997e-05 2024-09-20T22:54:26.690 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.690 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.690 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-09-20T22:54:26.690 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:33.213646+0000", 2024-09-20T22:54:26.690 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3205e-05 2024-09-20T22:54:26.690 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.690 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.690 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:54:26.690 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:34.226325+0000", 2024-09-20T22:54:26.690 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0126793300000001 2024-09-20T22:54:26.690 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.690 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.691 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for degraded object", 2024-09-20T22:54:26.691 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:34.228377+0000", 2024-09-20T22:54:26.691 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0020512540000000002 2024-09-20T22:54:26.691 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.691 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.691 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:54:26.691 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:34.239166+0000", 2024-09-20T22:54:26.691 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010789029 2024-09-20T22:54:26.691 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.691 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.691 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:54:26.692 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:34.239275+0000", 2024-09-20T22:54:26.692 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00010938900000000001 2024-09-20T22:54:26.692 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.692 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.692 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 3,6", 2024-09-20T22:54:26.692 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:34.239569+0000", 2024-09-20T22:54:26.692 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00029410699999999999 2024-09-20T22:54:26.692 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.692 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.692 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:54:26.692 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:34.241371+0000", 2024-09-20T22:54:26.692 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00180171 2024-09-20T22:54:26.692 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.693 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.693 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:54:26.693 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:34.241393+0000", 2024-09-20T22:54:26.693 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.1928000000000001e-05 2024-09-20T22:54:26.693 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.693 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.693 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:54:26.693 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:34.241465+0000", 2024-09-20T22:54:26.693 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.1799999999999997e-05 2024-09-20T22:54:26.693 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.693 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.693 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:54:26.693 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:34.241528+0000", 2024-09-20T22:54:26.693 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.2946000000000003e-05 2024-09-20T22:54:26.694 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.694 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.694 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:54:26.694 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:34.241647+0000", 2024-09-20T22:54:26.694 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000119373 2024-09-20T22:54:26.694 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:26.694 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:26.694 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:26.694 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.694 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.694 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:4312 7.17 7:ed54065f:::smithi08477203-12:head [rollback 96] snapc b1=[b1,b0,af,ae,ad,ab,a9,a7,a6,a5,9f,9e,9d,9c,9a,96,95,8e,87,86] ondisk+write+known_if_redirected+supports_pool_eio e433)", 2024-09-20T22:54:26.694 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:48:33.213603+0000", 2024-09-20T22:54:26.695 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 353.458141748, 2024-09-20T22:54:26.695 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0286283940000001, 2024-09-20T22:54:26.695 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:54:26.695 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:54:26.695 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:54:26.695 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:54:26.695 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:54:26.695 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:54:26.695 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 4312 2024-09-20T22:54:26.695 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.695 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:54:26.695 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.695 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:54:26.695 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:33.213603+0000", 2024-09-20T22:54:26.695 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:54:26.695 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.696 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.696 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:54:26.696 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:33.213603+0000", 2024-09-20T22:54:26.696 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:54:26.696 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.696 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.696 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:54:26.696 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:33.213603+0000", 2024-09-20T22:54:26.696 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.9200000000000001e-07 2024-09-20T22:54:26.696 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.696 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.696 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:54:26.696 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:33.213606+0000", 2024-09-20T22:54:26.696 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.6390000000000001e-06 2024-09-20T22:54:26.696 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.697 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.697 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:54:26.697 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:33.213612+0000", 2024-09-20T22:54:26.697 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.5429999999999999e-06 2024-09-20T22:54:26.697 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.697 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.697 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:54:26.697 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:33.213618+0000", 2024-09-20T22:54:26.697 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.4249999999999997e-06 2024-09-20T22:54:26.697 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.697 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.697 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:54:26.697 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:33.213674+0000", 2024-09-20T22:54:26.698 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.6341000000000001e-05 2024-09-20T22:54:26.698 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.698 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.698 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-09-20T22:54:26.698 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:33.214118+0000", 2024-09-20T22:54:26.698 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00044406600000000001 2024-09-20T22:54:26.698 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.698 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.698 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:54:26.698 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:34.228938+0000", 2024-09-20T22:54:26.698 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0148198470000001 2024-09-20T22:54:26.698 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.698 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.698 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for degraded object", 2024-09-20T22:54:26.699 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:34.229016+0000", 2024-09-20T22:54:26.699 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.7383000000000001e-05 2024-09-20T22:54:26.699 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.699 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.699 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:54:26.699 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:34.240634+0000", 2024-09-20T22:54:26.699 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011618277999999999 2024-09-20T22:54:26.699 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.699 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.699 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:54:26.699 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:34.240749+0000", 2024-09-20T22:54:26.700 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00011473500000000001 2024-09-20T22:54:26.700 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.700 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.700 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 3,6", 2024-09-20T22:54:26.700 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:34.241041+0000", 2024-09-20T22:54:26.700 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00029246699999999998 2024-09-20T22:54:26.700 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.700 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.700 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:54:26.700 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:34.242022+0000", 2024-09-20T22:54:26.700 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00098068999999999999 2024-09-20T22:54:26.700 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.700 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.700 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:54:26.701 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:34.242068+0000", 2024-09-20T22:54:26.701 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.6581e-05 2024-09-20T22:54:26.701 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.701 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.701 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:54:26.701 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:34.242081+0000", 2024-09-20T22:54:26.701 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2709e-05 2024-09-20T22:54:26.701 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.701 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.701 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:54:26.701 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:34.242116+0000", 2024-09-20T22:54:26.701 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.4579000000000002e-05 2024-09-20T22:54:26.701 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.702 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.702 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:54:26.702 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:34.242231+0000", 2024-09-20T22:54:26.702 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00011561900000000001 2024-09-20T22:54:26.702 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:26.702 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:26.702 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:26.702 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.702 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.702 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:4415 7.16 7.fa65a116 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e444)", 2024-09-20T22:54:26.702 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:48:44.266352+0000", 2024-09-20T22:54:26.702 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 342.40539243199999, 2024-09-20T22:54:26.702 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.96293146200000002, 2024-09-20T22:54:26.703 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:54:26.703 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:54:26.703 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "reached pg", 2024-09-20T22:54:26.703 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:54:26.703 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:54:26.703 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:54:26.703 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 4415 2024-09-20T22:54:26.703 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.703 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:54:26.703 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.703 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:54:26.703 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:44.266352+0000", 2024-09-20T22:54:26.703 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:54:26.703 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.704 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.704 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:54:26.704 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:44.266352+0000", 2024-09-20T22:54:26.704 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:54:26.704 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.704 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.704 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:54:26.704 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:44.266354+0000", 2024-09-20T22:54:26.704 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.733e-06 2024-09-20T22:54:26.704 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.704 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.704 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:54:26.704 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:44.266610+0000", 2024-09-20T22:54:26.704 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000256352 2024-09-20T22:54:26.705 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.705 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.705 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:54:26.705 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:44.266626+0000", 2024-09-20T22:54:26.705 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.5347000000000001e-05 2024-09-20T22:54:26.705 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.705 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.705 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:54:26.705 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:44.266634+0000", 2024-09-20T22:54:26.705 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.5390000000000007e-06 2024-09-20T22:54:26.705 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.705 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.705 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:54:26.705 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:44.266701+0000", 2024-09-20T22:54:26.706 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.6908000000000004e-05 2024-09-20T22:54:26.706 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.706 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.706 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-09-20T22:54:26.706 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:44.266710+0000", 2024-09-20T22:54:26.706 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.2920000000000001e-06 2024-09-20T22:54:26.706 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.706 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.706 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:54:26.706 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:45.229222+0000", 2024-09-20T22:54:26.706 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.96251129899999999 2024-09-20T22:54:26.706 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.706 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.706 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:54:26.706 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:45.229284+0000", 2024-09-20T22:54:26.707 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.1991999999999999e-05 2024-09-20T22:54:26.707 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:26.707 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:26.707 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:26.707 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.707 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.707 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:4416 7.16 7.fa65a116 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e444)", 2024-09-20T22:54:26.707 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:48:44.273462+0000", 2024-09-20T22:54:26.707 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 342.39828300200003, 2024-09-20T22:54:26.707 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.95612304000000004, 2024-09-20T22:54:26.707 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:54:26.707 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:54:26.707 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "reached pg", 2024-09-20T22:54:26.707 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:54:26.708 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:54:26.708 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:54:26.708 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 4416 2024-09-20T22:54:26.708 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.708 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:54:26.708 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.708 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:54:26.708 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:44.273462+0000", 2024-09-20T22:54:26.708 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:54:26.708 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.708 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.708 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:54:26.708 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:44.273462+0000", 2024-09-20T22:54:26.709 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:54:26.709 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.709 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.709 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:54:26.709 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:44.273464+0000", 2024-09-20T22:54:26.709 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.3609999999999999e-06 2024-09-20T22:54:26.709 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.709 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.709 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:54:26.709 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:44.273675+0000", 2024-09-20T22:54:26.709 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00021111400000000001 2024-09-20T22:54:26.709 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.709 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.709 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:54:26.710 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:44.273697+0000", 2024-09-20T22:54:26.710 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.2305000000000001e-05 2024-09-20T22:54:26.710 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.710 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.710 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:54:26.710 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:44.273708+0000", 2024-09-20T22:54:26.710 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0669999999999999e-05 2024-09-20T22:54:26.710 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.710 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.710 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:54:26.710 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:44.273865+0000", 2024-09-20T22:54:26.710 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000156766 2024-09-20T22:54:26.711 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.711 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.711 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-09-20T22:54:26.711 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:44.273878+0000", 2024-09-20T22:54:26.711 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3178e-05 2024-09-20T22:54:26.711 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.711 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.711 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:54:26.711 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:45.229577+0000", 2024-09-20T22:54:26.711 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.95569872600000005 2024-09-20T22:54:26.711 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.711 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.711 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:54:26.711 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:45.229585+0000", 2024-09-20T22:54:26.712 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.9200000000000004e-06 2024-09-20T22:54:26.712 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:26.712 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:26.712 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:26.712 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.712 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.712 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:4417 7.16 7.fa65a116 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e444)", 2024-09-20T22:54:26.712 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:48:44.273740+0000", 2024-09-20T22:54:26.712 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 342.39800432200002, 2024-09-20T22:54:26.712 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.95591940399999997, 2024-09-20T22:54:26.712 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:54:26.712 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:54:26.712 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "reached pg", 2024-09-20T22:54:26.713 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:54:26.713 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:54:26.713 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:54:26.713 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 4417 2024-09-20T22:54:26.713 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.713 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:54:26.713 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.713 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:54:26.713 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:44.273740+0000", 2024-09-20T22:54:26.713 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:54:26.713 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.713 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.713 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:54:26.714 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:44.273740+0000", 2024-09-20T22:54:26.714 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:54:26.714 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.714 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.714 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:54:26.714 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:44.273741+0000", 2024-09-20T22:54:26.714 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.06e-07 2024-09-20T22:54:26.714 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.714 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.714 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:54:26.714 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:44.273746+0000", 2024-09-20T22:54:26.714 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.7129999999999996e-06 2024-09-20T22:54:26.715 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.715 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.715 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:54:26.715 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:44.273751+0000", 2024-09-20T22:54:26.715 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.8309999999999997e-06 2024-09-20T22:54:26.715 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.715 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.715 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:54:26.715 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:44.273756+0000", 2024-09-20T22:54:26.715 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.6859999999999997e-06 2024-09-20T22:54:26.715 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.715 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.715 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:54:26.715 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:44.274433+0000", 2024-09-20T22:54:26.715 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00067743000000000002 2024-09-20T22:54:26.716 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.716 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.716 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-09-20T22:54:26.716 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:44.274446+0000", 2024-09-20T22:54:26.716 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2737e-05 2024-09-20T22:54:26.716 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.716 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.716 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:54:26.716 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:45.229650+0000", 2024-09-20T22:54:26.716 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.95520395499999999 2024-09-20T22:54:26.716 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.716 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.716 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:54:26.716 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:45.229660+0000", 2024-09-20T22:54:26.716 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.4460000000000007e-06 2024-09-20T22:54:26.717 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:26.717 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:26.717 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:26.717 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.717 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.717 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:4418 7.16 7.fa65a116 (undecoded) ondisk+read+rwordered+known_if_redirected+supports_pool_eio e444)", 2024-09-20T22:54:26.717 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:48:44.273764+0000", 2024-09-20T22:54:26.717 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 342.397980668, 2024-09-20T22:54:26.717 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.95592935400000001, 2024-09-20T22:54:26.717 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:54:26.717 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:54:26.717 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "reached pg", 2024-09-20T22:54:26.717 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:54:26.717 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:54:26.718 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:54:26.718 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 4418 2024-09-20T22:54:26.718 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.718 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:54:26.718 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.718 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:54:26.718 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:44.273764+0000", 2024-09-20T22:54:26.718 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:54:26.718 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.718 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.718 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:54:26.718 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:44.273764+0000", 2024-09-20T22:54:26.718 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:54:26.718 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.719 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.719 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:54:26.719 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:44.273764+0000", 2024-09-20T22:54:26.719 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.2e-07 2024-09-20T22:54:26.719 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.719 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.719 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:54:26.719 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:44.273773+0000", 2024-09-20T22:54:26.719 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.0489999999999993e-06 2024-09-20T22:54:26.719 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.719 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.719 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:54:26.719 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:44.273780+0000", 2024-09-20T22:54:26.719 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.195e-06 2024-09-20T22:54:26.720 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.720 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.720 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:54:26.720 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:44.273784+0000", 2024-09-20T22:54:26.720 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.245e-06 2024-09-20T22:54:26.720 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.720 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.720 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:54:26.720 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:44.274475+0000", 2024-09-20T22:54:26.720 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00069082200000000001 2024-09-20T22:54:26.720 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.720 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.720 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-09-20T22:54:26.720 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:44.274484+0000", 2024-09-20T22:54:26.721 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.6940000000000007e-06 2024-09-20T22:54:26.721 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.721 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.721 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:54:26.721 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:45.229685+0000", 2024-09-20T22:54:26.721 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.95520094 2024-09-20T22:54:26.721 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.721 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.721 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:54:26.721 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:45.229693+0000", 2024-09-20T22:54:26.721 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.9889999999999992e-06 2024-09-20T22:54:26.721 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:26.721 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:26.721 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:26.722 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.722 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.722 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:4415 7.6 7:6885a65f:::smithi08477203-14:head [write 685884~560628] snapc b7=[b7,b6,b5,b4,b3,b2,b1,b0,af,ae,a9,a7,a6,a5,9e,9c,9a,96,95,87,86] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e446)", 2024-09-20T22:54:26.722 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:48:46.223519+0000", 2024-09-20T22:54:26.722 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:54:26.724 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 340.44822561299998, 2024-09-20T22:54:26.724 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0194105410000001, 2024-09-20T22:54:26.724 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:54:26.724 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:54:26.724 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:54:26.724 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:54:26.724 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:54:26.724 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:54:26.724 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 4415 2024-09-20T22:54:26.724 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.724 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:54:26.724 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.724 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:54:26.724 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:46.223519+0000", 2024-09-20T22:54:26.725 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:54:26.725 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.725 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.725 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:54:26.725 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:46.223519+0000", 2024-09-20T22:54:26.725 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:54:26.725 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.725 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.725 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:54:26.725 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:46.223521+0000", 2024-09-20T22:54:26.725 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.7260000000000001e-06 2024-09-20T22:54:26.725 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.725 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.725 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:54:26.726 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:46.223861+0000", 2024-09-20T22:54:26.726 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00034058200000000003 2024-09-20T22:54:26.726 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.726 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.726 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:54:26.726 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:46.223874+0000", 2024-09-20T22:54:26.726 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2777e-05 2024-09-20T22:54:26.726 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.726 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.726 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:54:26.726 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:46.223884+0000", 2024-09-20T22:54:26.726 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0326000000000001e-05 2024-09-20T22:54:26.727 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.727 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.727 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:54:26.727 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:46.228721+0000", 2024-09-20T22:54:26.727 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0048363470000000004 2024-09-20T22:54:26.727 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.727 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.727 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-09-20T22:54:26.727 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:46.228734+0000", 2024-09-20T22:54:26.727 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3219e-05 2024-09-20T22:54:26.727 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.727 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.727 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:54:26.727 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:47.235776+0000", 2024-09-20T22:54:26.728 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0070420579999999 2024-09-20T22:54:26.728 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.728 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.728 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:54:26.728 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:47.235914+0000", 2024-09-20T22:54:26.728 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013761599999999999 2024-09-20T22:54:26.728 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.728 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.728 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 3,7", 2024-09-20T22:54:26.728 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:47.236144+0000", 2024-09-20T22:54:26.728 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000230165 2024-09-20T22:54:26.728 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.728 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.729 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:54:26.729 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:47.242485+0000", 2024-09-20T22:54:26.729 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0063409929999999996 2024-09-20T22:54:26.729 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.729 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.729 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:54:26.729 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:47.242500+0000", 2024-09-20T22:54:26.729 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.5676e-05 2024-09-20T22:54:26.729 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.729 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.729 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:54:26.730 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:47.242790+0000", 2024-09-20T22:54:26.730 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00028984699999999998 2024-09-20T22:54:26.730 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.730 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.730 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:54:26.730 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:47.242860+0000", 2024-09-20T22:54:26.730 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.9765999999999996e-05 2024-09-20T22:54:26.730 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.730 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.730 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:54:26.730 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:47.242929+0000", 2024-09-20T22:54:26.730 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.9443000000000006e-05 2024-09-20T22:54:26.730 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:26.730 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:26.731 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:26.731 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.731 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.731 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:4416 7.6 7:6885a65f:::smithi08477203-14:head [write 2046274~239610] snapc b7=[b7,b6,b5,b4,b3,b2,b1,b0,af,ae,a9,a7,a6,a5,9e,9c,9a,96,95,87,86] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e446)", 2024-09-20T22:54:26.731 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:48:46.225256+0000", 2024-09-20T22:54:26.731 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 340.44648832000001, 2024-09-20T22:54:26.731 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0189633840000001, 2024-09-20T22:54:26.731 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:54:26.731 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:54:26.731 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:54:26.731 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:54:26.731 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:54:26.731 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:54:26.731 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 4416 2024-09-20T22:54:26.732 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.732 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:54:26.732 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.732 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:54:26.732 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:46.225256+0000", 2024-09-20T22:54:26.732 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:54:26.732 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.732 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.732 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:54:26.732 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:46.225256+0000", 2024-09-20T22:54:26.732 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:54:26.732 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.732 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.732 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:54:26.732 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:46.225258+0000", 2024-09-20T22:54:26.733 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.279e-06 2024-09-20T22:54:26.733 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.733 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.733 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:54:26.733 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:46.225882+0000", 2024-09-20T22:54:26.733 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00062460200000000003 2024-09-20T22:54:26.733 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.733 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.733 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:54:26.733 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:46.225894+0000", 2024-09-20T22:54:26.733 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2306999999999999e-05 2024-09-20T22:54:26.733 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.733 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.733 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:54:26.733 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:46.225908+0000", 2024-09-20T22:54:26.734 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3643e-05 2024-09-20T22:54:26.734 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.734 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.734 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:54:26.734 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:46.228769+0000", 2024-09-20T22:54:26.734 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002860656 2024-09-20T22:54:26.734 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.734 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.734 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-09-20T22:54:26.734 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:46.228778+0000", 2024-09-20T22:54:26.734 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.8370000000000003e-06 2024-09-20T22:54:26.734 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.734 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.734 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:54:26.735 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:47.237586+0000", 2024-09-20T22:54:26.735 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.008808207 2024-09-20T22:54:26.735 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.735 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.735 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:54:26.735 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:47.237766+0000", 2024-09-20T22:54:26.735 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00018071600000000001 2024-09-20T22:54:26.735 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.735 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.735 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 3,7", 2024-09-20T22:54:26.735 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:47.238007+0000", 2024-09-20T22:54:26.735 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00024016400000000001 2024-09-20T22:54:26.735 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.736 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.736 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:54:26.736 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:47.242507+0000", 2024-09-20T22:54:26.736 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0045000539999999999 2024-09-20T22:54:26.736 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.736 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.736 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:54:26.736 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:47.243911+0000", 2024-09-20T22:54:26.736 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001403856 2024-09-20T22:54:26.736 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.736 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.736 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:54:26.736 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:47.244094+0000", 2024-09-20T22:54:26.736 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000183239 2024-09-20T22:54:26.736 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.736 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.737 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:54:26.737 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:47.244161+0000", 2024-09-20T22:54:26.737 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.7688000000000007e-05 2024-09-20T22:54:26.737 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.737 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.737 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:54:26.737 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:47.244220+0000", 2024-09-20T22:54:26.737 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.8136e-05 2024-09-20T22:54:26.737 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:26.737 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:26.737 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:26.737 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.738 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.738 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:4417 7.6 7:6885a65f:::smithi08477203-14:head [setxattr _header (58) in=7b,truncate 2285884] snapc b7=[b7,b6,b5,b4,b3,b2,b1,b0,af,ae,a9,a7,a6,a5,9e,9c,9a,96,95,87,86] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e446)", 2024-09-20T22:54:26.738 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:48:46.225923+0000", 2024-09-20T22:54:26.738 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 340.44582170799998, 2024-09-20T22:54:26.738 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.021878477, 2024-09-20T22:54:26.738 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:54:26.738 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:54:26.738 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:54:26.738 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:54:26.738 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:54:26.738 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:54:26.738 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 4417 2024-09-20T22:54:26.738 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.738 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:54:26.739 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.739 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:54:26.739 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:46.225923+0000", 2024-09-20T22:54:26.739 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:54:26.739 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.739 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.739 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:54:26.739 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:46.225923+0000", 2024-09-20T22:54:26.739 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:54:26.739 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.739 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.739 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:54:26.739 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:46.225924+0000", 2024-09-20T22:54:26.739 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.8199999999999999e-07 2024-09-20T22:54:26.739 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.740 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.740 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:54:26.740 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:46.225928+0000", 2024-09-20T22:54:26.740 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.2409999999999997e-06 2024-09-20T22:54:26.740 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.740 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.740 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:54:26.740 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:46.225934+0000", 2024-09-20T22:54:26.740 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.353e-06 2024-09-20T22:54:26.740 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.740 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.740 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:54:26.740 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:46.225939+0000", 2024-09-20T22:54:26.740 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.3299999999999998e-06 2024-09-20T22:54:26.741 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.741 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.741 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:54:26.741 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:46.228811+0000", 2024-09-20T22:54:26.741 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0028720199999999999 2024-09-20T22:54:26.741 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.741 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.741 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-09-20T22:54:26.741 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:46.228819+0000", 2024-09-20T22:54:26.741 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.875e-06 2024-09-20T22:54:26.741 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.741 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.741 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:54:26.741 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:47.241348+0000", 2024-09-20T22:54:26.741 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0125285070000001 2024-09-20T22:54:26.741 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.741 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.742 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:54:26.742 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:47.241459+0000", 2024-09-20T22:54:26.742 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000111622 2024-09-20T22:54:26.742 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.742 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.742 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 3,7", 2024-09-20T22:54:26.742 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:47.241661+0000", 2024-09-20T22:54:26.742 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00020146699999999999 2024-09-20T22:54:26.742 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.742 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.742 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:54:26.742 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:47.242513+0000", 2024-09-20T22:54:26.742 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00085188600000000005 2024-09-20T22:54:26.742 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.743 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.743 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:54:26.743 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:47.243955+0000", 2024-09-20T22:54:26.743 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001442313 2024-09-20T22:54:26.743 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.743 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.743 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:54:26.743 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:47.247555+0000", 2024-09-20T22:54:26.743 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0036002529999999999 2024-09-20T22:54:26.743 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.743 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.743 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:54:26.743 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:47.247628+0000", 2024-09-20T22:54:26.743 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.2878000000000003e-05 2024-09-20T22:54:26.743 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.744 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.744 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:54:26.744 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:47.247801+0000", 2024-09-20T22:54:26.744 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00017305000000000001 2024-09-20T22:54:26.744 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:26.744 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:26.744 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:26.744 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.744 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.744 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:4418 7.6 7:6885a65f:::smithi08477203-14:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e446)", 2024-09-20T22:54:26.744 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:48:46.225948+0000", 2024-09-20T22:54:26.744 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 340.44579682900002, 2024-09-20T22:54:26.744 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.022323377, 2024-09-20T22:54:26.744 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:54:26.744 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:54:26.744 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-20T22:54:26.745 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:54:26.745 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:54:26.745 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:54:26.745 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 4418 2024-09-20T22:54:26.745 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.745 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:54:26.745 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.745 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:54:26.745 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:46.225948+0000", 2024-09-20T22:54:26.745 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:54:26.745 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.745 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.745 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:54:26.745 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:46.225948+0000", 2024-09-20T22:54:26.745 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:54:26.745 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.746 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.746 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:54:26.746 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:46.225948+0000", 2024-09-20T22:54:26.746 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.9999999999999998e-07 2024-09-20T22:54:26.746 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.746 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.746 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:54:26.746 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:46.225951+0000", 2024-09-20T22:54:26.746 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.4499999999999998e-06 2024-09-20T22:54:26.746 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.746 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.746 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:54:26.746 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:46.225956+0000", 2024-09-20T22:54:26.746 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.0309999999999998e-06 2024-09-20T22:54:26.746 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.746 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.747 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:54:26.747 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:46.225961+0000", 2024-09-20T22:54:26.747 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.8359999999999999e-06 2024-09-20T22:54:26.747 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.747 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.747 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:54:26.747 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:46.228848+0000", 2024-09-20T22:54:26.747 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0028873670000000001 2024-09-20T22:54:26.747 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.747 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.747 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-09-20T22:54:26.747 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:46.228857+0000", 2024-09-20T22:54:26.747 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.6999999999999997e-06 2024-09-20T22:54:26.747 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.747 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.748 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:54:26.748 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:47.242965+0000", 2024-09-20T22:54:26.748 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0141083310000001 2024-09-20T22:54:26.748 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.748 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.748 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for rw locks", 2024-09-20T22:54:26.748 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:47.243076+0000", 2024-09-20T22:54:26.748 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000110945 2024-09-20T22:54:26.748 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.748 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.748 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:54:26.748 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:47.247935+0000", 2024-09-20T22:54:26.748 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0048591049999999998 2024-09-20T22:54:26.748 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.748 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.749 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:54:26.749 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:47.248116+0000", 2024-09-20T22:54:26.749 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000180936 2024-09-20T22:54:26.749 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.749 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.749 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:54:26.749 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:47.248271+0000", 2024-09-20T22:54:26.749 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00015517600000000001 2024-09-20T22:54:26.749 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:26.749 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:26.749 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:26.749 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.749 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.749 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:5057 7.9 7:9eac4560:::smithi08477203-30:head [write 548093~787174,stat] snapc da=[da,d9,d8,d7,d6,d5,d4,d3,d1,d0,cf,ce,cd,cc,ca,c7,c6,c5,c2,bf,ba,b4,b2,a7,9e,9c,95,87] ondisk+write+known_if_redirected+supports_pool_eio e508)", 2024-09-20T22:54:26.750 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:49:48.244197+0000", 2024-09-20T22:54:26.750 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 278.42754726499999, 2024-09-20T22:54:26.750 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.037773875999999998, 2024-09-20T22:54:26.750 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:54:26.750 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:54:26.750 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:54:26.750 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:54:26.750 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:54:26.750 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:54:26.750 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 5057 2024-09-20T22:54:26.750 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.750 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:54:26.750 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.750 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:54:26.751 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:49:48.244197+0000", 2024-09-20T22:54:26.751 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:54:26.751 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.751 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.751 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:54:26.751 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:49:48.244197+0000", 2024-09-20T22:54:26.751 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:54:26.751 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.751 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.751 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:54:26.751 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:49:48.244199+0000", 2024-09-20T22:54:26.751 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.4750000000000001e-06 2024-09-20T22:54:26.751 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.751 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.751 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:54:26.751 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:49:48.244695+0000", 2024-09-20T22:54:26.752 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00049580299999999996 2024-09-20T22:54:26.752 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.752 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.752 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:54:26.752 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:49:48.244712+0000", 2024-09-20T22:54:26.752 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.7e-05 2024-09-20T22:54:26.752 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.752 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.752 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:54:26.752 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:49:48.244720+0000", 2024-09-20T22:54:26.752 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.2099999999999993e-06 2024-09-20T22:54:26.752 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.753 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.753 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:54:26.753 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:49:48.245032+0000", 2024-09-20T22:54:26.753 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00031212699999999999 2024-09-20T22:54:26.753 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.753 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.753 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:54:26.753 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:49:48.245187+0000", 2024-09-20T22:54:26.753 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00015518699999999999 2024-09-20T22:54:26.753 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.753 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.753 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 1,4", 2024-09-20T22:54:26.753 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:49:48.245562+0000", 2024-09-20T22:54:26.754 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00037534899999999997 2024-09-20T22:54:26.754 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.754 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.754 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:54:26.754 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:49:48.250266+0000", 2024-09-20T22:54:26.754 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0047034989999999999 2024-09-20T22:54:26.754 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.754 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.754 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:54:26.754 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:49:48.253360+0000", 2024-09-20T22:54:26.754 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0030940400000000002 2024-09-20T22:54:26.754 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.754 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.755 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:54:26.755 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:49:48.281847+0000", 2024-09-20T22:54:26.755 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.028487350000000002 2024-09-20T22:54:26.755 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.755 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.755 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:54:26.755 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:49:48.281906+0000", 2024-09-20T22:54:26.755 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.8289999999999999e-05 2024-09-20T22:54:26.755 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.755 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.755 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:54:26.755 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:49:48.281971+0000", 2024-09-20T22:54:26.755 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.5545999999999999e-05 2024-09-20T22:54:26.756 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:26.756 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:26.756 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:26.756 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.756 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.756 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:5058 7.9 7:9eac4560:::smithi08477203-30:head [write 1970678~449902] snapc da=[da,d9,d8,d7,d6,d5,d4,d3,d1,d0,cf,ce,cd,cc,ca,c7,c6,c5,c2,bf,ba,b4,b2,a7,9e,9c,95,87] ondisk+write+known_if_redirected+supports_pool_eio e508)", 2024-09-20T22:54:26.756 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:49:48.250601+0000", 2024-09-20T22:54:26.756 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 278.421143218, 2024-09-20T22:54:26.756 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.031506248000000001, 2024-09-20T22:54:26.756 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:54:26.756 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:54:26.756 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:54:26.756 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:54:26.756 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:54:26.756 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:54:26.757 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 5058 2024-09-20T22:54:26.757 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.757 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:54:26.757 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.757 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:54:26.757 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:49:48.250601+0000", 2024-09-20T22:54:26.757 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:54:26.757 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.757 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.757 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:54:26.757 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:49:48.250601+0000", 2024-09-20T22:54:26.757 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:54:26.757 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.757 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.757 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:54:26.758 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:49:48.250604+0000", 2024-09-20T22:54:26.758 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.5270000000000001e-06 2024-09-20T22:54:26.758 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.758 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.758 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:54:26.758 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:49:48.250924+0000", 2024-09-20T22:54:26.758 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000320391 2024-09-20T22:54:26.758 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.758 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.758 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:54:26.758 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:49:48.250941+0000", 2024-09-20T22:54:26.758 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.6481999999999999e-05 2024-09-20T22:54:26.759 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.759 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.759 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:54:26.759 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:49:48.250951+0000", 2024-09-20T22:54:26.759 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.9280000000000005e-06 2024-09-20T22:54:26.759 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.759 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.759 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:54:26.759 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:49:48.251037+0000", 2024-09-20T22:54:26.759 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.6310999999999994e-05 2024-09-20T22:54:26.759 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.759 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.759 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:54:26.759 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:49:48.251176+0000", 2024-09-20T22:54:26.760 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000139505 2024-09-20T22:54:26.760 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.760 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.760 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 1,4", 2024-09-20T22:54:26.760 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:49:48.251397+0000", 2024-09-20T22:54:26.760 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00022060299999999999 2024-09-20T22:54:26.760 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.760 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.760 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:54:26.760 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:49:48.254904+0000", 2024-09-20T22:54:26.760 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0035071989999999999 2024-09-20T22:54:26.760 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.760 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.761 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:54:26.761 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:49:48.255361+0000", 2024-09-20T22:54:26.761 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00045642300000000002 2024-09-20T22:54:26.761 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.761 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.761 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:54:26.761 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:49:48.282027+0000", 2024-09-20T22:54:26.761 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.026665852 2024-09-20T22:54:26.761 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.761 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.761 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:54:26.761 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:49:48.282069+0000", 2024-09-20T22:54:26.762 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.2778000000000003e-05 2024-09-20T22:54:26.762 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.762 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.762 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:54:26.762 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:49:48.282108+0000", 2024-09-20T22:54:26.762 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.8248999999999999e-05 2024-09-20T22:54:26.762 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:26.762 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:26.762 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:26.762 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.762 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.762 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:5412 7.6 7:64bcd2fc:::smithi08477203-20:head [copy-from ver 794] snapc ef=[ef,ed,ec,eb,ea,e8,e7,e6,e5,e4,e2,df,db,da,d7,d5,d4,d3,d0,cf,ce,ca,c5,c2,bf,b4,b2,a7,95] ondisk+write+known_if_redirected+supports_pool_eio e549)", 2024-09-20T22:54:26.762 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:50:29.255763+0000", 2024-09-20T22:54:26.763 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 237.41598160300001, 2024-09-20T22:54:26.763 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.032860671000000001, 2024-09-20T22:54:26.763 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:54:26.763 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:54:26.763 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:54:26.763 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:54:26.763 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:54:26.763 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:54:26.763 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 5412 2024-09-20T22:54:26.763 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.763 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:54:26.763 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.763 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:54:26.764 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:29.255763+0000", 2024-09-20T22:54:26.764 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:54:26.764 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.764 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.764 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:54:26.764 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:29.255763+0000", 2024-09-20T22:54:26.764 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:54:26.764 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.764 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.764 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:54:26.764 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:29.255764+0000", 2024-09-20T22:54:26.764 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.11e-06 2024-09-20T22:54:26.765 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.765 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.765 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:54:26.765 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:29.255768+0000", 2024-09-20T22:54:26.765 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.019e-06 2024-09-20T22:54:26.765 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.765 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.765 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:54:26.765 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:29.255777+0000", 2024-09-20T22:54:26.765 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.5539999999999998e-06 2024-09-20T22:54:26.765 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.765 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.765 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:54:26.765 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:29.255782+0000", 2024-09-20T22:54:26.766 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.9490000000000004e-06 2024-09-20T22:54:26.766 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.766 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.766 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:54:26.766 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:29.255859+0000", 2024-09-20T22:54:26.766 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.7751999999999997e-05 2024-09-20T22:54:26.766 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.766 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.766 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:54:26.766 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:29.256005+0000", 2024-09-20T22:54:26.766 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014578799999999999 2024-09-20T22:54:26.766 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.767 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.767 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 3,7", 2024-09-20T22:54:26.767 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:29.273220+0000", 2024-09-20T22:54:26.767 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.017214808000000002 2024-09-20T22:54:26.767 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.767 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.767 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:54:26.767 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:29.286455+0000", 2024-09-20T22:54:26.767 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013235232 2024-09-20T22:54:26.767 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.767 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.767 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:54:26.768 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:29.286879+0000", 2024-09-20T22:54:26.768 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00042428099999999999 2024-09-20T22:54:26.768 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.768 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.768 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:54:26.768 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:29.288446+0000", 2024-09-20T22:54:26.768 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015662810000000001 2024-09-20T22:54:26.768 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.768 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.768 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:54:26.768 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:29.288516+0000", 2024-09-20T22:54:26.768 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.0321000000000007e-05 2024-09-20T22:54:26.769 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.769 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.769 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:54:26.769 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:29.288624+0000", 2024-09-20T22:54:26.769 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000107576 2024-09-20T22:54:26.769 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:26.769 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:26.769 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:26.769 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.769 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.769 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:5413 7.6 7:64bcd2fc:::smithi08477203-20:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e549)", 2024-09-20T22:54:26.769 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:50:29.255802+0000", 2024-09-20T22:54:26.769 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 237.415942261, 2024-09-20T22:54:26.769 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.024182717999999999, 2024-09-20T22:54:26.770 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:54:26.770 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:54:26.770 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-20T22:54:26.770 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:54:26.770 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:54:26.770 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:54:26.770 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 5413 2024-09-20T22:54:26.770 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.770 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:54:26.770 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.770 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:54:26.770 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:29.255802+0000", 2024-09-20T22:54:26.770 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:54:26.771 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.771 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.771 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:54:26.771 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:29.255802+0000", 2024-09-20T22:54:26.771 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:54:26.771 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.771 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.771 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:54:26.771 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:29.255803+0000", 2024-09-20T22:54:26.771 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.8800000000000003e-07 2024-09-20T22:54:26.771 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.771 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.771 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:54:26.771 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:29.255805+0000", 2024-09-20T22:54:26.772 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.5950000000000001e-06 2024-09-20T22:54:26.772 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.772 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.772 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:54:26.772 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:29.255814+0000", 2024-09-20T22:54:26.772 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.0480000000000004e-06 2024-09-20T22:54:26.772 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.772 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.772 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:54:26.772 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:29.255820+0000", 2024-09-20T22:54:26.772 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.57e-06 2024-09-20T22:54:26.772 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.773 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.773 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:54:26.773 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:29.256223+0000", 2024-09-20T22:54:26.773 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00040313099999999999 2024-09-20T22:54:26.773 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.773 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.773 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for blocked object", 2024-09-20T22:54:26.773 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:29.256281+0000", 2024-09-20T22:54:26.773 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.7679000000000001e-05 2024-09-20T22:54:26.773 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.773 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.773 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:54:26.774 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:29.279736+0000", 2024-09-20T22:54:26.774 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.023454900000000001 2024-09-20T22:54:26.774 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.774 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.774 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:54:26.774 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:29.279857+0000", 2024-09-20T22:54:26.774 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000121401 2024-09-20T22:54:26.774 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.774 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.774 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:54:26.774 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:29.279985+0000", 2024-09-20T22:54:26.774 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000127906 2024-09-20T22:54:26.774 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:26.774 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:26.774 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:26.775 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.775 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.775 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:6018 7.4 7:24b1a612:::smithi08477203-43:head [copy-from ver 846] snapc 108=[108,107,105,104,103,102,101,ff,fe,fd,fc,fb,fa,f8,f5,f2,ed,ec,ea,df,da,d7,c2,bf] ondisk+write+known_if_redirected+supports_pool_eio e606)", 2024-09-20T22:54:26.775 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:51:26.429397+0000", 2024-09-20T22:54:26.775 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 180.24234706999999, 2024-09-20T22:54:26.775 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.034058235999999999, 2024-09-20T22:54:26.775 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:54:26.775 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:54:26.775 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:54:26.775 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:54:26.775 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:54:26.775 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:54:26.775 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 6018 2024-09-20T22:54:26.776 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.776 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:54:26.776 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.776 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:54:26.776 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:26.429397+0000", 2024-09-20T22:54:26.776 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:54:26.776 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.776 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.776 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:54:26.776 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:26.429397+0000", 2024-09-20T22:54:26.776 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:54:26.776 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.776 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.776 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:54:26.777 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:26.429399+0000", 2024-09-20T22:54:26.777 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.824e-06 2024-09-20T22:54:26.777 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.777 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.777 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:54:26.777 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:26.429408+0000", 2024-09-20T22:54:26.777 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.1770000000000004e-06 2024-09-20T22:54:26.777 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.777 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.777 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:54:26.777 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:26.429422+0000", 2024-09-20T22:54:26.777 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3574000000000001e-05 2024-09-20T22:54:26.777 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.777 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.778 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:54:26.778 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:26.429430+0000", 2024-09-20T22:54:26.778 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.8550000000000002e-06 2024-09-20T22:54:26.778 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.778 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.778 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:54:26.778 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:26.429492+0000", 2024-09-20T22:54:26.778 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.2433000000000003e-05 2024-09-20T22:54:26.778 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.778 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.778 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:54:26.778 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:26.429577+0000", 2024-09-20T22:54:26.778 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.5018999999999994e-05 2024-09-20T22:54:26.778 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.779 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.779 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 4,6", 2024-09-20T22:54:26.779 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:26.433580+0000", 2024-09-20T22:54:26.779 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00400264 2024-09-20T22:54:26.779 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.779 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.779 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:54:26.779 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:26.437548+0000", 2024-09-20T22:54:26.779 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0039683740000000002 2024-09-20T22:54:26.779 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.779 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.779 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:54:26.779 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:26.440702+0000", 2024-09-20T22:54:26.779 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0031540990000000001 2024-09-20T22:54:26.780 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.780 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.780 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:54:26.780 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:26.463373+0000", 2024-09-20T22:54:26.780 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.022670613999999999 2024-09-20T22:54:26.780 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.780 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.780 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:54:26.780 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:26.463418+0000", 2024-09-20T22:54:26.780 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.5046999999999997e-05 2024-09-20T22:54:26.780 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.780 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.780 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:54:26.780 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:26.463456+0000", 2024-09-20T22:54:26.781 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.7580000000000003e-05 2024-09-20T22:54:26.781 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:26.781 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:26.781 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:26.781 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.781 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.781 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:6253 7.9 7:91f2d47c:::smithi08477203-8:head [copy-from ver 1797] snapc 114=[114,113,112,111,110,10e,10c,10b,108,107,105,104,103,101,fd,fb,fa,f8,f5,f2,ed,ec,d7,c2,bf] ondisk+write+known_if_redirected+supports_pool_eio e629)", 2024-09-20T22:54:26.781 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:51:49.618540+0000", 2024-09-20T22:54:26.781 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 157.053204421, 2024-09-20T22:54:26.781 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.024931412999999999, 2024-09-20T22:54:26.781 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:54:26.781 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:54:26.781 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:54:26.781 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:54:26.781 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:54:26.782 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:54:26.782 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 6253 2024-09-20T22:54:26.782 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.782 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:54:26.782 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.782 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:54:26.782 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:49.618540+0000", 2024-09-20T22:54:26.782 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:54:26.782 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.782 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.782 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:54:26.782 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:49.618540+0000", 2024-09-20T22:54:26.782 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:54:26.783 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.783 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.783 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:54:26.783 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:49.618542+0000", 2024-09-20T22:54:26.783 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.637e-06 2024-09-20T22:54:26.783 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.783 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.783 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:54:26.783 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:49.618548+0000", 2024-09-20T22:54:26.783 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.088e-06 2024-09-20T22:54:26.783 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.783 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.783 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:54:26.783 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:49.618559+0000", 2024-09-20T22:54:26.784 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1422e-05 2024-09-20T22:54:26.784 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.784 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.784 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:54:26.784 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:49.618565+0000", 2024-09-20T22:54:26.784 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.3909999999999997e-06 2024-09-20T22:54:26.784 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.784 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.784 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:54:26.784 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:49.618642+0000", 2024-09-20T22:54:26.784 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.737e-05 2024-09-20T22:54:26.784 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.784 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.784 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:54:26.784 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:49.618716+0000", 2024-09-20T22:54:26.785 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.4103000000000005e-05 2024-09-20T22:54:26.785 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.785 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.785 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 4,7", 2024-09-20T22:54:26.785 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:49.625475+0000", 2024-09-20T22:54:26.785 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0067586299999999998 2024-09-20T22:54:26.785 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.785 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.785 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:54:26.785 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:49.633603+0000", 2024-09-20T22:54:26.785 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0081277929999999995 2024-09-20T22:54:26.785 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.785 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.785 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:54:26.786 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:49.643271+0000", 2024-09-20T22:54:26.786 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0096679839999999993 2024-09-20T22:54:26.786 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.786 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.786 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:54:26.786 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:49.643392+0000", 2024-09-20T22:54:26.786 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000121734 2024-09-20T22:54:26.786 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.786 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.786 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:54:26.786 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:49.643427+0000", 2024-09-20T22:54:26.786 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.5138999999999997e-05 2024-09-20T22:54:26.786 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.786 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.787 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:54:26.787 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:49.643472+0000", 2024-09-20T22:54:26.787 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.4122000000000002e-05 2024-09-20T22:54:26.787 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:26.787 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:26.787 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:26.787 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.787 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.787 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:7387 7.9 7:907aec0e:::smithi08477203-3:head [write 692719~403182,stat] snapc 14e=[14e,14c,14a,149,148,145,142,141,13e,13d,13c,13b,13a,139,135,132,130,12f,12e,12c,12a,128,127,124,121,11f,112,103] ondisk+write+known_if_redirected+supports_pool_eio e743)", 2024-09-20T22:54:26.787 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:53:43.308708+0000", 2024-09-20T22:54:26.787 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 43.363036141999999, 2024-09-20T22:54:26.787 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.03359815, 2024-09-20T22:54:26.787 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:54:26.788 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:54:26.788 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:54:26.788 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:54:26.788 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:54:26.788 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:54:26.788 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 7387 2024-09-20T22:54:26.788 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.788 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:54:26.788 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.788 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:54:26.788 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:43.308708+0000", 2024-09-20T22:54:26.788 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:54:26.788 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.788 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.788 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:54:26.789 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:43.308708+0000", 2024-09-20T22:54:26.789 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:54:26.789 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.789 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.789 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:54:26.789 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:43.308710+0000", 2024-09-20T22:54:26.789 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.68e-06 2024-09-20T22:54:26.789 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.789 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.789 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:54:26.789 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:43.308941+0000", 2024-09-20T22:54:26.789 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000231277 2024-09-20T22:54:26.789 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.789 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.789 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:54:26.790 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:43.308954+0000", 2024-09-20T22:54:26.790 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.29e-05 2024-09-20T22:54:26.790 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.790 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.790 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:54:26.790 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:43.308961+0000", 2024-09-20T22:54:26.790 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.7490000000000001e-06 2024-09-20T22:54:26.790 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.790 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.790 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:54:26.790 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:43.309015+0000", 2024-09-20T22:54:26.790 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.3869999999999998e-05 2024-09-20T22:54:26.790 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.790 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.790 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:54:26.791 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:43.309117+0000", 2024-09-20T22:54:26.791 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00010163800000000001 2024-09-20T22:54:26.791 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.791 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.791 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 4,7", 2024-09-20T22:54:26.791 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:43.309296+0000", 2024-09-20T22:54:26.791 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00017921199999999999 2024-09-20T22:54:26.791 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.791 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.791 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:54:26.791 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:43.312537+0000", 2024-09-20T22:54:26.791 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0032413870000000001 2024-09-20T22:54:26.791 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.792 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.792 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:54:26.792 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:43.340218+0000", 2024-09-20T22:54:26.792 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.027680986000000001 2024-09-20T22:54:26.792 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.792 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.792 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:54:26.792 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:43.342189+0000", 2024-09-20T22:54:26.792 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0019712670000000001 2024-09-20T22:54:26.792 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.792 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.792 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:54:26.792 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:43.342233+0000", 2024-09-20T22:54:26.792 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.3676e-05 2024-09-20T22:54:26.793 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.793 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.793 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:54:26.793 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:43.342307+0000", 2024-09-20T22:54:26.793 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.3508000000000002e-05 2024-09-20T22:54:26.793 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:26.793 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:26.793 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:26.793 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.793 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.793 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:7388 7.9 7:907aec0e:::smithi08477203-3:head [write 1579246~720684] snapc 14e=[14e,14c,14a,149,148,145,142,141,13e,13d,13c,13b,13a,139,135,132,130,12f,12e,12c,12a,128,127,124,121,11f,112,103] ondisk+write+known_if_redirected+supports_pool_eio e743)", 2024-09-20T22:54:26.793 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:53:43.316061+0000", 2024-09-20T22:54:26.794 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 43.355683085000003, 2024-09-20T22:54:26.794 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.026368321, 2024-09-20T22:54:26.794 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:54:26.794 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:54:26.794 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:54:26.794 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:54:26.794 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:54:26.794 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:54:26.794 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 7388 2024-09-20T22:54:26.794 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.794 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:54:26.794 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.794 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:54:26.794 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:43.316061+0000", 2024-09-20T22:54:26.795 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:54:26.795 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.795 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.795 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:54:26.795 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:43.316061+0000", 2024-09-20T22:54:26.795 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:54:26.795 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.795 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.795 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:54:26.795 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:43.316063+0000", 2024-09-20T22:54:26.795 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.181e-06 2024-09-20T22:54:26.795 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.795 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.795 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:54:26.795 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:43.316639+0000", 2024-09-20T22:54:26.795 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00057640100000000004 2024-09-20T22:54:26.796 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.796 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.796 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:54:26.796 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:43.316649+0000", 2024-09-20T22:54:26.796 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.8479999999999996e-06 2024-09-20T22:54:26.796 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.796 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.796 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:54:26.796 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:43.316654+0000", 2024-09-20T22:54:26.796 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.3260000000000003e-06 2024-09-20T22:54:26.796 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.796 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.796 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:54:26.796 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:43.316698+0000", 2024-09-20T22:54:26.796 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.3643999999999998e-05 2024-09-20T22:54:26.797 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.797 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.797 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:54:26.797 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:43.316771+0000", 2024-09-20T22:54:26.797 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.2750000000000007e-05 2024-09-20T22:54:26.797 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.797 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.797 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 4,7", 2024-09-20T22:54:26.797 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:43.316869+0000", 2024-09-20T22:54:26.797 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.8845999999999995e-05 2024-09-20T22:54:26.797 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.797 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.797 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:54:26.797 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:43.319492+0000", 2024-09-20T22:54:26.797 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026225520000000002 2024-09-20T22:54:26.797 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.798 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.798 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:54:26.798 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:43.341085+0000", 2024-09-20T22:54:26.798 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.021593131000000002 2024-09-20T22:54:26.798 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.798 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.798 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:54:26.798 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:43.342350+0000", 2024-09-20T22:54:26.798 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0012649219999999999 2024-09-20T22:54:26.798 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.798 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.798 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:54:26.798 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:43.342389+0000", 2024-09-20T22:54:26.798 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.9257e-05 2024-09-20T22:54:26.799 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.799 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.799 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:54:26.799 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:43.342430+0000", 2024-09-20T22:54:26.799 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.0463000000000002e-05 2024-09-20T22:54:26.799 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:26.799 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:26.799 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:26.799 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.799 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.799 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:7726 7.f 7:faee1812:::smithi08477203-48:head [copy-from ver 1829] snapc 15d=[15d,15c,15b,15a,159,158,157,156,155,153,151,14f,14e,14c,14a,149,148,145,142,13e,13a,139,132,12f,12e,12a,124,112] ondisk+write+known_if_redirected+supports_pool_eio e773)", 2024-09-20T22:54:26.799 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:54:13.329331+0000", 2024-09-20T22:54:26.800 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 13.3424137, 2024-09-20T22:54:26.800 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013843645999999999, 2024-09-20T22:54:26.800 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:54:26.800 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:54:26.800 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:54:26.800 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:54:26.800 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:54:26.800 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:54:26.800 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 7726 2024-09-20T22:54:26.800 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.800 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:54:26.800 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.800 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:54:26.800 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:54:13.329331+0000", 2024-09-20T22:54:26.800 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:54:26.801 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.801 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.801 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:54:26.801 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:54:13.329331+0000", 2024-09-20T22:54:26.801 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:54:26.801 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.801 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.801 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:54:26.801 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:54:13.329332+0000", 2024-09-20T22:54:26.801 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.5009999999999999e-06 2024-09-20T22:54:26.801 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.801 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.801 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:54:26.801 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:54:13.329339+0000", 2024-09-20T22:54:26.802 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.0350000000000004e-06 2024-09-20T22:54:26.802 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.802 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.802 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:54:26.802 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:54:13.329352+0000", 2024-09-20T22:54:26.802 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2356e-05 2024-09-20T22:54:26.802 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.802 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.802 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:54:26.802 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:54:13.329359+0000", 2024-09-20T22:54:26.802 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.8750000000000002e-06 2024-09-20T22:54:26.802 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.803 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.803 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:54:26.803 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:54:13.329415+0000", 2024-09-20T22:54:26.803 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.6839000000000002e-05 2024-09-20T22:54:26.803 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.803 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.803 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:54:26.803 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:54:13.329502+0000", 2024-09-20T22:54:26.803 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.6669999999999995e-05 2024-09-20T22:54:26.803 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.803 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.803 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 3,7", 2024-09-20T22:54:26.803 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:54:13.335406+0000", 2024-09-20T22:54:26.804 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0059036540000000004 2024-09-20T22:54:26.804 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.804 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.804 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:54:26.804 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:54:13.341601+0000", 2024-09-20T22:54:26.804 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0061955049999999996 2024-09-20T22:54:26.804 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.804 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.804 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:54:26.804 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:54:13.342592+0000", 2024-09-20T22:54:26.804 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00099027899999999994 2024-09-20T22:54:26.804 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.804 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.804 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:54:26.804 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:54:13.343026+0000", 2024-09-20T22:54:26.804 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000434518 2024-09-20T22:54:26.805 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.805 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.805 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:54:26.805 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:54:13.343105+0000", 2024-09-20T22:54:26.805 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.8955999999999994e-05 2024-09-20T22:54:26.805 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:26.805 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:26.805 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:54:26.805 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:54:13.343174+0000", 2024-09-20T22:54:26.805 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.9457999999999998e-05 2024-09-20T22:54:26.805 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:26.805 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:26.805 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:26.805 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:26.806 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:26.806 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-20T22:54:26.830 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:26.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:26.831 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:54:26.932 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:54:26.956 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:26.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:26.957 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:54:27.058 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:54:27.059 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:54:27.082 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:27.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:27.083 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:54:27.184 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:54:27.208 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:27.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:27.209 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:54:27.310 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:54:27.320 INFO:tasks.rados.rados.0.smithi084.stdout:3123: done (2 left) 2024-09-20T22:54:27.320 INFO:tasks.rados.rados.0.smithi084.stdout:3124: done (1 left) 2024-09-20T22:54:27.320 INFO:tasks.rados.rados.0.smithi084.stdout:3126: done (0 left) 2024-09-20T22:54:27.321 INFO:tasks.rados.rados.0.smithi084.stdout:3127: snap_create 2024-09-20T22:54:27.335 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:27.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:27.336 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:54:27.436 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:54:27.460 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-20T22:54:27.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:27.462 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:27.462 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:54:27.482 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:27 smithi084 ceph-mon[40024]: pgmap v1080: 158 pgs: 3 active+clean+snaptrim, 1 active+remapped+backfill_toofull, 154 active+clean; 538 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 13 MiB/s rd, 2.6 MiB/s wr, 23 op/s; 15/1326 objects misplaced (1.131%); 7.0 MiB/s, 2 objects/s recovering 2024-09-20T22:54:27.482 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:27 smithi084 ceph-mon[40024]: osdmap e786: 8 total, 8 up, 8 in 2024-09-20T22:54:27.482 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:27 smithi084 ceph-mon[29774]: pgmap v1080: 158 pgs: 3 active+clean+snaptrim, 1 active+remapped+backfill_toofull, 154 active+clean; 538 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 13 MiB/s rd, 2.6 MiB/s wr, 23 op/s; 15/1326 objects misplaced (1.131%); 7.0 MiB/s, 2 objects/s recovering 2024-09-20T22:54:27.482 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:27 smithi084 ceph-mon[29774]: osdmap e786: 8 total, 8 up, 8 in 2024-09-20T22:54:27.547 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-20T22:54:27.563 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:54:27.608 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:27.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:27.609 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:54:27.629 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:54:27.709 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:54:27.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:27.732 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:27.732 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:54:27.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:27 smithi154 ceph-mon[33264]: pgmap v1080: 158 pgs: 3 active+clean+snaptrim, 1 active+remapped+backfill_toofull, 154 active+clean; 538 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 13 MiB/s rd, 2.6 MiB/s wr, 23 op/s; 15/1326 objects misplaced (1.131%); 7.0 MiB/s, 2 objects/s recovering 2024-09-20T22:54:27.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:27 smithi154 ceph-mon[33264]: osdmap e786: 8 total, 8 up, 8 in 2024-09-20T22:54:27.833 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:54:27.856 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:27.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:27.857 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:54:27.958 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:54:27.979 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:27.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:27.980 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:54:28.081 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:54:28.102 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:28.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:28.103 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:54:28.165 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:54:28.165 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:54:28.165 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:54:28.166 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:54:28.166 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:54:28.166 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:54:28.177 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-20T22:54:28.177 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T22:54:28.194 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:54:28.195 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:54:28.204 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:54:28.211 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:54:28.211 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:54:28.230 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:54:28.230 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:54:28.231 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:28.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:28.232 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:54:28.247 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:54:28.247 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:54:28.313 INFO:tasks.rados.rados.0.smithi084.stdout:3127: done (0 left) 2024-09-20T22:54:28.314 INFO:tasks.rados.rados.0.smithi084.stdout:3128: write oid 13 current snap is 358 2024-09-20T22:54:28.314 INFO:tasks.rados.rados.0.smithi084.stdout:3128: seq_num 1074 ranges {450718=423199,1414602=636116} 2024-09-20T22:54:28.318 INFO:tasks.rados.rados.0.smithi084.stdout:3128: writing smithi08477203-13 from 450718 to 873917 tid 1 2024-09-20T22:54:28.325 INFO:tasks.rados.rados.0.smithi084.stdout:3128: writing smithi08477203-13 from 1414602 to 2050718 tid 2 2024-09-20T22:54:28.326 INFO:tasks.rados.rados.0.smithi084.stdout:3129: write (excl) oid 20 current snap is 358 2024-09-20T22:54:28.326 INFO:tasks.rados.rados.0.smithi084.stdout:3128: finishing write tid 1 to smithi08477203-13 2024-09-20T22:54:28.326 INFO:tasks.rados.rados.0.smithi084.stdout:3129: seq_num 1075 ranges {467525=498448,1473084=594379} 2024-09-20T22:54:28.326 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:54:28 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: ::ffff:172.21.15.154 - - [20/Sep/2024:22:54:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-20T22:54:28.331 INFO:tasks.rados.rados.0.smithi084.stdout:3129: writing smithi08477203-20 from 467525 to 965973 tid 1 2024-09-20T22:54:28.332 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:54:28.338 INFO:tasks.rados.rados.0.smithi084.stdout:3129: writing smithi08477203-20 from 1473084 to 2067463 tid 2 2024-09-20T22:54:28.338 INFO:tasks.rados.rados.0.smithi084.stdout:3130: copy_from oid 47 from oid 44 current snap is 358 2024-09-20T22:54:28.338 INFO:tasks.rados.rados.0.smithi084.stdout:3128: finishing write tid 2 to smithi08477203-13 2024-09-20T22:54:28.338 INFO:tasks.rados.rados.0.smithi084.stdout:3128: finishing write tid 3 to smithi08477203-13 2024-09-20T22:54:28.338 INFO:tasks.rados.rados.0.smithi084.stdout:3128: finishing write tid 4 to smithi08477203-13 2024-09-20T22:54:28.338 INFO:tasks.rados.rados.0.smithi084.stdout:3128: oid 13 updating version 0 to 2536 2024-09-20T22:54:28.339 INFO:tasks.rados.rados.0.smithi084.stdout:3128: oid 13 updating version 2536 to 2537 2024-09-20T22:54:28.339 INFO:tasks.rados.rados.0.smithi084.stdout:3128: oid 13 version 2537 is already newer than 2535 2024-09-20T22:54:28.339 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 13 v 2537 (ObjNum 1074 snap 358 seq_num 1074) dirty exists 2024-09-20T22:54:28.339 INFO:tasks.rados.rados.0.smithi084.stdout:3128: left oid 13 (ObjNum 1074 snap 358 seq_num 1074) 2024-09-20T22:54:28.339 INFO:tasks.rados.rados.0.smithi084.stdout:3129: finishing write tid 1 to smithi08477203-20 2024-09-20T22:54:28.339 INFO:tasks.rados.rados.0.smithi084.stdout:3128: done (2 left) 2024-09-20T22:54:28.339 INFO:tasks.rados.rados.0.smithi084.stdout:3131: delete oid 42 current snap is 358 2024-09-20T22:54:28.349 INFO:tasks.rados.rados.0.smithi084.stdout:3130: finishing copy_from racing read to smithi08477203-47 2024-09-20T22:54:28.358 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:28.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:28.359 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:54:28.362 INFO:tasks.rados.rados.0.smithi084.stdout:3131: done (2 left) 2024-09-20T22:54:28.362 INFO:tasks.rados.rados.0.smithi084.stdout:3132: write oid 27 current snap is 358 2024-09-20T22:54:28.362 INFO:tasks.rados.rados.0.smithi084.stdout:3132: seq_num 1077 ranges {501139=648946,1873695=227444} 2024-09-20T22:54:28.368 INFO:tasks.rados.rados.0.smithi084.stdout:3132: writing smithi08477203-27 from 501139 to 1150085 tid 1 2024-09-20T22:54:28.374 INFO:tasks.rados.rados.0.smithi084.stdout:3132: writing smithi08477203-27 from 1873695 to 2101139 tid 2 2024-09-20T22:54:28.374 INFO:tasks.rados.rados.0.smithi084.stdout:3133: copy_from oid 46 from oid 38 current snap is 358 2024-09-20T22:54:28.374 INFO:tasks.rados.rados.0.smithi084.stdout:3129: finishing write tid 2 to smithi08477203-20 2024-09-20T22:54:28.375 INFO:tasks.rados.rados.0.smithi084.stdout:3129: finishing write tid 3 to smithi08477203-20 2024-09-20T22:54:28.375 INFO:tasks.rados.rados.0.smithi084.stdout:3129: finishing write tid 4 to smithi08477203-20 2024-09-20T22:54:28.375 INFO:tasks.rados.rados.0.smithi084.stdout:3129: oid 20 updating version 0 to 1920 2024-09-20T22:54:28.375 INFO:tasks.rados.rados.0.smithi084.stdout:3129: oid 20 version 1920 is already newer than 1919 2024-09-20T22:54:28.375 INFO:tasks.rados.rados.0.smithi084.stdout:3129: oid 20 version 1920 is already newer than 1918 2024-09-20T22:54:28.375 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 20 v 1920 (ObjNum 1075 snap 358 seq_num 1075) dirty exists 2024-09-20T22:54:28.376 INFO:tasks.rados.rados.0.smithi084.stdout:3129: left oid 20 (ObjNum 1075 snap 358 seq_num 1075) 2024-09-20T22:54:28.376 INFO:tasks.rados.rados.0.smithi084.stdout:3130: finishing copy_from to smithi08477203-47 2024-09-20T22:54:28.376 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 47 v 2453 (ObjNum 1051 snap 345 seq_num 1051) dirty exists 2024-09-20T22:54:28.376 INFO:tasks.rados.rados.0.smithi084.stdout:3132: finishing write tid 1 to smithi08477203-27 2024-09-20T22:54:28.376 INFO:tasks.rados.rados.0.smithi084.stdout:3129: done (3 left) 2024-09-20T22:54:28.376 INFO:tasks.rados.rados.0.smithi084.stdout:3130: done (2 left) 2024-09-20T22:54:28.376 INFO:tasks.rados.rados.0.smithi084.stdout:3134: read oid 44 snap -1 2024-09-20T22:54:28.376 INFO:tasks.rados.rados.0.smithi084.stdout:3134: expect (ObjNum 1051 snap 345 seq_num 1051) 2024-09-20T22:54:28.377 INFO:tasks.rados.rados.0.smithi084.stdout:3135: write oid 32 current snap is 358 2024-09-20T22:54:28.377 INFO:tasks.rados.rados.0.smithi084.stdout:3135: seq_num 1078 ranges {517946=724195,1932177=185769} 2024-09-20T22:54:28.380 INFO:tasks.rados.rados.0.smithi084.stdout:3135: writing smithi08477203-32 from 517946 to 1242141 tid 1 2024-09-20T22:54:28.387 INFO:tasks.rados.rados.0.smithi084.stdout:3135: writing smithi08477203-32 from 1932177 to 2117946 tid 2 2024-09-20T22:54:28.387 INFO:tasks.rados.rados.0.smithi084.stdout:3136: copy_from oid 21 from oid 18 current snap is 358 2024-09-20T22:54:28.387 INFO:tasks.rados.rados.0.smithi084.stdout:3132: finishing write tid 2 to smithi08477203-27 2024-09-20T22:54:28.387 INFO:tasks.rados.rados.0.smithi084.stdout:3132: finishing write tid 3 to smithi08477203-27 2024-09-20T22:54:28.387 INFO:tasks.rados.rados.0.smithi084.stdout:3132: finishing write tid 4 to smithi08477203-27 2024-09-20T22:54:28.387 INFO:tasks.rados.rados.0.smithi084.stdout:3132: oid 27 updating version 0 to 1811 2024-09-20T22:54:28.387 INFO:tasks.rados.rados.0.smithi084.stdout:3132: oid 27 updating version 1811 to 1813 2024-09-20T22:54:28.387 INFO:tasks.rados.rados.0.smithi084.stdout:3132: oid 27 version 1813 is already newer than 1812 2024-09-20T22:54:28.387 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 27 v 1813 (ObjNum 1077 snap 358 seq_num 1077) dirty exists 2024-09-20T22:54:28.387 INFO:tasks.rados.rados.0.smithi084.stdout:3132: left oid 27 (ObjNum 1077 snap 358 seq_num 1077) 2024-09-20T22:54:28.387 INFO:tasks.rados.rados.0.smithi084.stdout:3134: expect (ObjNum 1051 snap 345 seq_num 1051) 2024-09-20T22:54:28.460 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:54:28.482 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:28.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:28.482 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:54:28.518 INFO:tasks.rados.rados.0.smithi084.stdout:3135: finishing write tid 1 to smithi08477203-32 2024-09-20T22:54:28.518 INFO:tasks.rados.rados.0.smithi084.stdout:3132: done (4 left) 2024-09-20T22:54:28.518 INFO:tasks.rados.rados.0.smithi084.stdout:3134: done (3 left) 2024-09-20T22:54:28.518 INFO:tasks.rados.rados.0.smithi084.stdout:3137: write (excl) oid 15 current snap is 358 2024-09-20T22:54:28.518 INFO:tasks.rados.rados.0.smithi084.stdout:3136: finishing copy_from to smithi08477203-21 2024-09-20T22:54:28.518 INFO:tasks.rados.rados.0.smithi084.stdout:3136: got expected ENOENT (src dne) 2024-09-20T22:54:28.518 INFO:tasks.rados.rados.0.smithi084.stdout:3136: finishing copy_from racing read to smithi08477203-21 2024-09-20T22:54:28.519 INFO:tasks.rados.rados.0.smithi084.stdout:3135: finishing write tid 2 to smithi08477203-32 2024-09-20T22:54:28.519 INFO:tasks.rados.rados.0.smithi084.stdout:3135: finishing write tid 3 to smithi08477203-32 2024-09-20T22:54:28.519 INFO:tasks.rados.rados.0.smithi084.stdout:3135: finishing write tid 4 to smithi08477203-32 2024-09-20T22:54:28.519 INFO:tasks.rados.rados.0.smithi084.stdout:3135: oid 32 updating version 0 to 1840 2024-09-20T22:54:28.519 INFO:tasks.rados.rados.0.smithi084.stdout:3135: oid 32 version 1840 is already newer than 1838 2024-09-20T22:54:28.519 INFO:tasks.rados.rados.0.smithi084.stdout:3135: oid 32 version 1840 is already newer than 1839 2024-09-20T22:54:28.519 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 32 v 1840 (ObjNum 1078 snap 358 seq_num 1078) dirty exists 2024-09-20T22:54:28.519 INFO:tasks.rados.rados.0.smithi084.stdout:3135: left oid 32 (ObjNum 1078 snap 358 seq_num 1078) 2024-09-20T22:54:28.519 INFO:tasks.rados.rados.0.smithi084.stdout:3133: finishing copy_from racing read to smithi08477203-46 2024-09-20T22:54:28.519 INFO:tasks.rados.rados.0.smithi084.stdout:3133: finishing copy_from to smithi08477203-46 2024-09-20T22:54:28.519 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 46 v 1830 (ObjNum 1057 snap 347 seq_num 1057) dirty exists 2024-09-20T22:54:28.519 INFO:tasks.rados.rados.0.smithi084.stdout:3137: seq_num 1079 ranges {534753=799444,1990659=144094} 2024-09-20T22:54:28.525 INFO:tasks.rados.rados.0.smithi084.stdout:3137: writing smithi08477203-15 from 534753 to 1334197 tid 1 2024-09-20T22:54:28.529 INFO:tasks.rados.rados.0.smithi084.stdout:3137: writing smithi08477203-15 from 1990659 to 2134753 tid 2 2024-09-20T22:54:28.530 INFO:tasks.rados.rados.0.smithi084.stdout:3133: done (3 left) 2024-09-20T22:54:28.530 INFO:tasks.rados.rados.0.smithi084.stdout:3135: done (2 left) 2024-09-20T22:54:28.530 INFO:tasks.rados.rados.0.smithi084.stdout:3136: done (1 left) 2024-09-20T22:54:28.530 INFO:tasks.rados.rados.0.smithi084.stdout:3138: snap_create 2024-09-20T22:54:28.530 INFO:tasks.rados.rados.0.smithi084.stdout:3137: finishing write tid 1 to smithi08477203-15 2024-09-20T22:54:28.531 INFO:tasks.rados.rados.0.smithi084.stdout:3137: finishing write tid 2 to smithi08477203-15 2024-09-20T22:54:28.532 INFO:tasks.rados.rados.0.smithi084.stdout:3137: finishing write tid 3 to smithi08477203-15 2024-09-20T22:54:28.532 INFO:tasks.rados.rados.0.smithi084.stdout:3137: finishing write tid 4 to smithi08477203-15 2024-09-20T22:54:28.532 INFO:tasks.rados.rados.0.smithi084.stdout:3137: oid 15 updating version 0 to 1862 2024-09-20T22:54:28.532 INFO:tasks.rados.rados.0.smithi084.stdout:3137: oid 15 updating version 1862 to 1864 2024-09-20T22:54:28.532 INFO:tasks.rados.rados.0.smithi084.stdout:3137: oid 15 version 1864 is already newer than 1863 2024-09-20T22:54:28.532 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 15 v 1864 (ObjNum 1079 snap 358 seq_num 1079) dirty exists 2024-09-20T22:54:28.532 INFO:tasks.rados.rados.0.smithi084.stdout:3137: left oid 15 (ObjNum 1079 snap 358 seq_num 1079) 2024-09-20T22:54:28.583 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:54:28.605 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:28.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:28.606 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:54:28.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:28 smithi084 ceph-mon[29774]: osdmap e787: 8 total, 8 up, 8 in 2024-09-20T22:54:28.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:28 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/228908383' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:54:28.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:28 smithi084 ceph-mon[29774]: osdmap e788: 8 total, 8 up, 8 in 2024-09-20T22:54:28.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:28 smithi084 ceph-mon[40024]: osdmap e787: 8 total, 8 up, 8 in 2024-09-20T22:54:28.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:28 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/228908383' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:54:28.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:28 smithi084 ceph-mon[40024]: osdmap e788: 8 total, 8 up, 8 in 2024-09-20T22:54:28.707 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:54:28.728 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:28.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:28.729 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:54:28.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:28 smithi154 ceph-mon[33264]: osdmap e787: 8 total, 8 up, 8 in 2024-09-20T22:54:28.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:28 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/228908383' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:54:28.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:28 smithi154 ceph-mon[33264]: osdmap e788: 8 total, 8 up, 8 in 2024-09-20T22:54:28.830 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:54:28.852 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:28.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:28.853 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:54:28.954 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:54:28.975 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:28.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:28.976 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:54:29.077 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:54:29.099 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:29.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:29.100 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:54:29.201 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:54:29.224 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:29.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:29.225 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:54:29.326 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:54:29.328 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:54:29.340 INFO:tasks.rados.rados.0.smithi084.stdout:3137: done (1 left) 2024-09-20T22:54:29.340 INFO:tasks.rados.rados.0.smithi084.stdout:3138: done (0 left) 2024-09-20T22:54:29.340 INFO:tasks.rados.rados.0.smithi084.stdout:3139: read oid 25 snap 346 2024-09-20T22:54:29.341 INFO:tasks.rados.rados.0.smithi084.stdout:3139: expect deleted 2024-09-20T22:54:29.341 INFO:tasks.rados.rados.0.smithi084.stdout:3140: delete oid 15 current snap is 359 2024-09-20T22:54:29.344 INFO:tasks.rados.rados.0.smithi084.stdout:3139: done (1 left) 2024-09-20T22:54:29.344 INFO:tasks.rados.rados.0.smithi084.stdout:3140: done (0 left) 2024-09-20T22:54:29.345 INFO:tasks.rados.rados.0.smithi084.stdout:3141: rollback oid 46 current snap is 359 2024-09-20T22:54:29.345 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 46 to 358 2024-09-20T22:54:29.345 INFO:tasks.rados.rados.0.smithi084.stdout:3142: copy_from oid 36 from oid 10 current snap is 359 2024-09-20T22:54:29.345 INFO:tasks.rados.rados.0.smithi084.stdout:3143: snap_create 2024-09-20T22:54:29.345 INFO:tasks.rados.rados.0.smithi084.stdout:3142: finishing copy_from to smithi08477203-36 2024-09-20T22:54:29.345 INFO:tasks.rados.rados.0.smithi084.stdout:3142: got expected ENOENT (src dne) 2024-09-20T22:54:29.346 INFO:tasks.rados.rados.0.smithi084.stdout:3142: finishing copy_from racing read to smithi08477203-36 2024-09-20T22:54:29.350 INFO:tasks.rados.rados.0.smithi084.stdout:3141: finishing rollback tid 0 to smithi08477203-46 2024-09-20T22:54:29.350 INFO:tasks.rados.rados.0.smithi084.stdout:3141: finishing rollback tid 1 to smithi08477203-46 2024-09-20T22:54:29.350 INFO:tasks.rados.rados.0.smithi084.stdout:3141: finishing rollback tid 2 to smithi08477203-46 2024-09-20T22:54:29.350 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 46 v 1868 (ObjNum 1057 snap 347 seq_num 1057) dirty exists 2024-09-20T22:54:29.357 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:29.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:29.358 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:54:29.372 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:54:29.459 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:54:29.481 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:29.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:29.482 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:54:29.509 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-20T22:54:29.511 INFO:teuthology.orchestra.run.smithi084.stderr:dumped all 2024-09-20T22:54:29.573 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:29 smithi084 ceph-mon[40024]: pgmap v1083: 158 pgs: 3 active+clean+snaptrim, 1 active+remapped+backfill_toofull, 154 active+clean; 538 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 15/1326 objects misplaced (1.131%) 2024-09-20T22:54:29.573 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:29 smithi084 ceph-mon[29774]: pgmap v1083: 158 pgs: 3 active+clean+snaptrim, 1 active+remapped+backfill_toofull, 154 active+clean; 538 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 15/1326 objects misplaced (1.131%) 2024-09-20T22:54:29.584 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:54:29.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:29.606 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:29.607 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:54:29.616 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:29 smithi154 ceph-mon[33264]: pgmap v1083: 158 pgs: 3 active+clean+snaptrim, 1 active+remapped+backfill_toofull, 154 active+clean; 538 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 15/1326 objects misplaced (1.131%) 2024-09-20T22:54:29.707 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:54:29.729 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:29.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:29.730 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:54:29.831 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:54:29.854 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:29.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:29.855 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:54:29.955 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:54:29.976 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:29.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:29.977 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:54:30.004 INFO:journalctl@ceph.iscsi.iscsi.a.smithi154.stdout:Sep 20 22:54:29 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-iscsi-iscsi-a[59730]: debug there is no tcmu-runner data available 2024-09-20T22:54:30.078 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:54:30.099 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:30.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:30.100 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:54:30.201 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:54:30.222 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:30.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:30.223 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:54:30.324 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:54:30.344 INFO:tasks.rados.rados.0.smithi084.stdout:3141: done (2 left) 2024-09-20T22:54:30.344 INFO:tasks.rados.rados.0.smithi084.stdout:3142: done (1 left) 2024-09-20T22:54:30.344 INFO:tasks.rados.rados.0.smithi084.stdout:3143: done (0 left) 2024-09-20T22:54:30.345 INFO:tasks.rados.rados.0.smithi084.stdout:3144: read oid 1 snap -1 2024-09-20T22:54:30.345 INFO:tasks.rados.rados.0.smithi084.stdout:3144: expect (ObjNum 738 snap 235 seq_num 738) 2024-09-20T22:54:30.345 INFO:tasks.rados.rados.0.smithi084.stdout:3145: read oid 3 snap -1 2024-09-20T22:54:30.345 INFO:tasks.rados.rados.0.smithi084.stdout:3145: expect (ObjNum 1053 snap 345 seq_num 1053) 2024-09-20T22:54:30.345 INFO:tasks.rados.rados.0.smithi084.stdout:3146: copy_from oid 39 from oid 44 current snap is 360 2024-09-20T22:54:30.345 INFO:tasks.rados.rados.0.smithi084.stdout:3147: rollback oid 10 current snap is 360 2024-09-20T22:54:30.345 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 10 to 300 2024-09-20T22:54:30.345 INFO:tasks.rados.rados.0.smithi084.stdout:3148: read oid 48 snap 328 2024-09-20T22:54:30.345 INFO:tasks.rados.rados.0.smithi084.stdout:3148: expect (ObjNum 985 snap 323 seq_num 985) 2024-09-20T22:54:30.345 INFO:tasks.rados.rados.0.smithi084.stdout:3149: snap_remove snap 358 2024-09-20T22:54:30.350 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:30.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:30.351 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:54:30.452 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:54:30.456 INFO:teuthology.orchestra.run.smithi084.stdout:{"pg_ready":true,"pg_map":{"version":1085,"stamp":"2024-09-20T22:54:29.333042+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":579202706,"num_objects":450,"num_object_clones":203,"num_object_copies":1350,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":15,"num_objects_unfound":0,"num_objects_dirty":450,"num_whiteouts":9,"num_read":7289,"num_read_kb":5623474,"num_write":4667,"num_write_kb":1482303,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1438,"num_bytes_recovered":693312186,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":13601,"ondisk_log_size":13601,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":475,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":1706596,"kb_used_data":1291856,"kb_used_omap":12,"kb_used_meta":414515,"kb_avail":748254620,"statfs":{"total":767960285184,"available":766212730880,"internally_reserved":0,"allocated":1322860544,"data_stored":1778089506,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":424463920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":14611815,"num_objects":8,"num_object_clones":8,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":27,"num_read_kb":5344,"num_write":33,"num_write_kb":17057,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.023086"},"pg_stats":[{"pgid":"3.1f","version":"86'2","reported_seq":832,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-09-20T22:54:28.310912+0000","last_change":"2024-09-20T22:52:21.630419+0000","last_active":"2024-09-20T22:54:28.310912+0000","last_peered":"2024-09-20T22:54:28.310912+0000","last_clean":"2024-09-20T22:54:28.310912+0000","last_became_active":"2024-09-20T22:47:49.222114+0000","last_became_peered":"2024-09-20T22:47:49.222114+0000","last_unstale":"2024-09-20T22:54:28.310912+0000","last_undegraded":"2024-09-20T22:54:28.310912+0000","last_fullsized":"2024-09-20T22:54:28.310912+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'2","last_scrub_stamp":"2024-09-20T22:52:21.630304+0000","last_deep_scrub":"86'2","last_deep_scrub_stamp":"2024-09-20T22:42:39.664421+0000","last_clean_scrub_stamp":"2024-09-20T22:52:21.630304+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:29.935852+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":436,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"625'15","reported_seq":845,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-09-20T22:54:28.310053+0000","last_change":"2024-09-20T22:54:17.395366+0000","last_active":"2024-09-20T22:54:28.310053+0000","last_peered":"2024-09-20T22:54:28.310053+0000","last_clean":"2024-09-20T22:54:28.310053+0000","last_became_active":"2024-09-20T22:54:17.354145+0000","last_became_peered":"2024-09-20T22:54:17.354145+0000","last_unstale":"2024-09-20T22:54:28.310053+0000","last_undegraded":"2024-09-20T22:54:28.310053+0000","last_fullsized":"2024-09-20T22:54:28.310053+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'15","last_scrub_stamp":"2024-09-20T22:51:51.813269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:51:51.813269+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:31.370191+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":33,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-09-20T22:54:28.362269+0000","last_change":"2024-09-20T22:54:17.338438+0000","last_active":"2024-09-20T22:54:28.362269+0000","last_peered":"2024-09-20T22:54:28.362269+0000","last_clean":"2024-09-20T22:54:28.362269+0000","last_became_active":"2024-09-20T22:54:17.337980+0000","last_became_peered":"2024-09-20T22:54:17.337980+0000","last_unstale":"2024-09-20T22:54:28.362269+0000","last_undegraded":"2024-09-20T22:54:28.362269+0000","last_fullsized":"2024-09-20T22:54:28.362269+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:54:15.974498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:54:15.974498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:29.491929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":814,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-09-20T22:54:28.310466+0000","last_change":"2024-09-20T22:54:17.351246+0000","last_active":"2024-09-20T22:54:28.310466+0000","last_peered":"2024-09-20T22:54:28.310466+0000","last_clean":"2024-09-20T22:54:28.310466+0000","last_became_active":"2024-09-20T22:54:17.350829+0000","last_became_peered":"2024-09-20T22:54:17.350829+0000","last_unstale":"2024-09-20T22:54:28.310466+0000","last_undegraded":"2024-09-20T22:54:28.310466+0000","last_fullsized":"2024-09-20T22:54:28.310466+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:58.481327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:51:36.730015+0000","last_clean_scrub_stamp":"2024-09-20T22:53:58.481327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:16.493810+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":1483,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-09-20T22:54:28.310318+0000","last_change":"2024-09-20T22:54:15.238419+0000","last_active":"2024-09-20T22:54:28.310318+0000","last_peered":"2024-09-20T22:54:28.310318+0000","last_clean":"2024-09-20T22:54:28.310318+0000","last_became_active":"2024-09-20T22:41:44.832099+0000","last_became_peered":"2024-09-20T22:41:44.832099+0000","last_unstale":"2024-09-20T22:54:28.310318+0000","last_undegraded":"2024-09-20T22:54:28.310318+0000","last_fullsized":"2024-09-20T22:54:28.310318+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:54:15.238344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:54:15.238344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:29.894427+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":1480,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-09-20T22:54:28.311049+0000","last_change":"2024-09-20T22:51:12.418041+0000","last_active":"2024-09-20T22:54:28.311049+0000","last_peered":"2024-09-20T22:54:28.311049+0000","last_clean":"2024-09-20T22:54:28.311049+0000","last_became_active":"2024-09-20T22:41:38.157298+0000","last_became_peered":"2024-09-20T22:41:38.157298+0000","last_unstale":"2024-09-20T22:54:28.311049+0000","last_undegraded":"2024-09-20T22:54:28.311049+0000","last_fullsized":"2024-09-20T22:54:28.311049+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:12.417920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:51:12.417920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:31.292465+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"625'27","reported_seq":1562,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-09-20T22:54:28.310295+0000","last_change":"2024-09-20T22:51:08.441927+0000","last_active":"2024-09-20T22:54:28.310295+0000","last_peered":"2024-09-20T22:54:28.310295+0000","last_clean":"2024-09-20T22:54:28.310295+0000","last_became_active":"2024-09-20T22:47:49.218798+0000","last_became_peered":"2024-09-20T22:47:49.218798+0000","last_unstale":"2024-09-20T22:54:28.310295+0000","last_undegraded":"2024-09-20T22:54:28.310295+0000","last_fullsized":"2024-09-20T22:54:28.310295+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-09-20T22:51:08.441788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:51:08.441788+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:31.292449+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":806,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-09-20T22:54:28.309978+0000","last_change":"2024-09-20T22:54:17.351386+0000","last_active":"2024-09-20T22:54:28.309978+0000","last_peered":"2024-09-20T22:54:28.309978+0000","last_clean":"2024-09-20T22:54:28.309978+0000","last_became_active":"2024-09-20T22:54:17.351020+0000","last_became_peered":"2024-09-20T22:54:17.351020+0000","last_unstale":"2024-09-20T22:54:28.309978+0000","last_undegraded":"2024-09-20T22:54:28.309978+0000","last_fullsized":"2024-09-20T22:54:28.309978+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:14.594823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:52:14.594823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:31.370295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":1508,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-09-20T22:54:28.369755+0000","last_change":"2024-09-20T22:53:56.980951+0000","last_active":"2024-09-20T22:54:28.369755+0000","last_peered":"2024-09-20T22:54:28.369755+0000","last_clean":"2024-09-20T22:54:28.369755+0000","last_became_active":"2024-09-20T22:48:05.242059+0000","last_became_peered":"2024-09-20T22:48:05.242059+0000","last_unstale":"2024-09-20T22:54:28.369755+0000","last_undegraded":"2024-09-20T22:54:28.369755+0000","last_fullsized":"2024-09-20T22:54:28.369755+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:56.980885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:46:00.999939+0000","last_clean_scrub_stamp":"2024-09-20T22:53:56.980885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:13.013721+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"625'15","reported_seq":821,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-09-20T22:54:28.310724+0000","last_change":"2024-09-20T22:53:51.486324+0000","last_active":"2024-09-20T22:54:28.310724+0000","last_peered":"2024-09-20T22:54:28.310724+0000","last_clean":"2024-09-20T22:54:28.310724+0000","last_became_active":"2024-09-20T22:48:05.233429+0000","last_became_peered":"2024-09-20T22:48:05.233429+0000","last_unstale":"2024-09-20T22:54:28.310724+0000","last_undegraded":"2024-09-20T22:54:28.310724+0000","last_fullsized":"2024-09-20T22:54:28.310724+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'15","last_scrub_stamp":"2024-09-20T22:53:51.486202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:53:51.486202+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:15.973503+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":1500,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-09-20T22:54:28.369105+0000","last_change":"2024-09-20T22:54:21.027796+0000","last_active":"2024-09-20T22:54:28.369105+0000","last_peered":"2024-09-20T22:54:28.369105+0000","last_clean":"2024-09-20T22:54:28.369105+0000","last_became_active":"2024-09-20T22:47:49.227379+0000","last_became_peered":"2024-09-20T22:47:49.227379+0000","last_unstale":"2024-09-20T22:54:28.369105+0000","last_undegraded":"2024-09-20T22:54:28.369105+0000","last_fullsized":"2024-09-20T22:54:28.369105+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:54:21.027723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:54:21.027723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:28.806253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":847,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-09-20T22:54:28.309821+0000","last_change":"2024-09-20T22:54:17.340772+0000","last_active":"2024-09-20T22:54:28.309821+0000","last_peered":"2024-09-20T22:54:28.309821+0000","last_clean":"2024-09-20T22:54:28.309821+0000","last_became_active":"2024-09-20T22:54:17.340467+0000","last_became_peered":"2024-09-20T22:54:17.340467+0000","last_unstale":"2024-09-20T22:54:28.309821+0000","last_undegraded":"2024-09-20T22:54:28.309821+0000","last_fullsized":"2024-09-20T22:54:28.309821+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:35.920968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:49:08.664407+0000","last_clean_scrub_stamp":"2024-09-20T22:53:35.920968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:02.922026+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":1520,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-09-20T22:54:28.369345+0000","last_change":"2024-09-20T22:54:17.342948+0000","last_active":"2024-09-20T22:54:28.369345+0000","last_peered":"2024-09-20T22:54:28.369345+0000","last_clean":"2024-09-20T22:54:28.369345+0000","last_became_active":"2024-09-20T22:54:17.342619+0000","last_became_peered":"2024-09-20T22:54:17.342619+0000","last_unstale":"2024-09-20T22:54:28.369345+0000","last_undegraded":"2024-09-20T22:54:28.369345+0000","last_fullsized":"2024-09-20T22:54:28.369345+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:31.092915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:53:31.092915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:54.763182+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"625'9","reported_seq":835,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-09-20T22:54:28.310874+0000","last_change":"2024-09-20T22:54:17.383703+0000","last_active":"2024-09-20T22:54:28.310874+0000","last_peered":"2024-09-20T22:54:28.310874+0000","last_clean":"2024-09-20T22:54:28.310874+0000","last_became_active":"2024-09-20T22:54:17.352032+0000","last_became_peered":"2024-09-20T22:54:17.352032+0000","last_unstale":"2024-09-20T22:54:28.310874+0000","last_undegraded":"2024-09-20T22:54:28.310874+0000","last_fullsized":"2024-09-20T22:54:28.310874+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'9","last_scrub_stamp":"2024-09-20T22:53:09.623458+0000","last_deep_scrub":"181'7","last_deep_scrub_stamp":"2024-09-20T22:48:41.049381+0000","last_clean_scrub_stamp":"2024-09-20T22:53:09.623458+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:31.370412+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":9,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":1533,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-09-20T22:54:28.321683+0000","last_change":"2024-09-20T22:54:19.572825+0000","last_active":"2024-09-20T22:54:28.321683+0000","last_peered":"2024-09-20T22:54:28.321683+0000","last_clean":"2024-09-20T22:54:28.321683+0000","last_became_active":"2024-09-20T22:54:17.331217+0000","last_became_peered":"2024-09-20T22:54:17.331217+0000","last_unstale":"2024-09-20T22:54:28.321683+0000","last_undegraded":"2024-09-20T22:54:28.321683+0000","last_fullsized":"2024-09-20T22:54:28.321683+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:54:19.572737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:47:40.361958+0000","last_clean_scrub_stamp":"2024-09-20T22:54:19.572737+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:20.397333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":1499,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-09-20T22:54:28.368981+0000","last_change":"2024-09-20T22:54:17.340038+0000","last_active":"2024-09-20T22:54:28.368981+0000","last_peered":"2024-09-20T22:54:28.368981+0000","last_clean":"2024-09-20T22:54:28.368981+0000","last_became_active":"2024-09-20T22:54:17.339658+0000","last_became_peered":"2024-09-20T22:54:17.339658+0000","last_unstale":"2024-09-20T22:54:28.368981+0000","last_undegraded":"2024-09-20T22:54:28.368981+0000","last_fullsized":"2024-09-20T22:54:28.368981+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:52.116243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:53:52.116243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:56.071404+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":33,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-09-20T22:54:28.310034+0000","last_change":"2024-09-20T22:54:17.339874+0000","last_active":"2024-09-20T22:54:28.310034+0000","last_peered":"2024-09-20T22:54:28.310034+0000","last_clean":"2024-09-20T22:54:28.310034+0000","last_became_active":"2024-09-20T22:54:17.339529+0000","last_became_peered":"2024-09-20T22:54:17.339529+0000","last_unstale":"2024-09-20T22:54:28.310034+0000","last_undegraded":"2024-09-20T22:54:28.310034+0000","last_fullsized":"2024-09-20T22:54:28.310034+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:48.558619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:53:48.558619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:57.888187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":833,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-09-20T22:54:28.310429+0000","last_change":"2024-09-20T22:52:42.745105+0000","last_active":"2024-09-20T22:54:28.310429+0000","last_peered":"2024-09-20T22:54:28.310429+0000","last_clean":"2024-09-20T22:54:28.310429+0000","last_became_active":"2024-09-20T22:48:05.233687+0000","last_became_peered":"2024-09-20T22:48:05.233687+0000","last_unstale":"2024-09-20T22:54:28.310429+0000","last_undegraded":"2024-09-20T22:54:28.310429+0000","last_fullsized":"2024-09-20T22:54:28.310429+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:42.745032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:49:53.933601+0000","last_clean_scrub_stamp":"2024-09-20T22:52:42.745032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:29.935972+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"625'34","reported_seq":1601,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-09-20T22:54:28.369318+0000","last_change":"2024-09-20T22:54:17.364170+0000","last_active":"2024-09-20T22:54:28.369318+0000","last_peered":"2024-09-20T22:54:28.369318+0000","last_clean":"2024-09-20T22:54:28.369318+0000","last_became_active":"2024-09-20T22:54:17.331250+0000","last_became_peered":"2024-09-20T22:54:17.331250+0000","last_unstale":"2024-09-20T22:54:28.369318+0000","last_undegraded":"2024-09-20T22:54:28.369318+0000","last_fullsized":"2024-09-20T22:54:28.369318+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'34","last_scrub_stamp":"2024-09-20T22:53:09.119791+0000","last_deep_scrub":"113'22","last_deep_scrub_stamp":"2024-09-20T22:43:12.191878+0000","last_clean_scrub_stamp":"2024-09-20T22:53:09.119791+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:30.212381+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":33,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-09-20T22:54:28.363153+0000","last_change":"2024-09-20T22:54:17.349933+0000","last_active":"2024-09-20T22:54:28.363153+0000","last_peered":"2024-09-20T22:54:28.363153+0000","last_clean":"2024-09-20T22:54:28.363153+0000","last_became_active":"2024-09-20T22:54:17.349501+0000","last_became_peered":"2024-09-20T22:54:17.349501+0000","last_unstale":"2024-09-20T22:54:28.363153+0000","last_undegraded":"2024-09-20T22:54:28.363153+0000","last_fullsized":"2024-09-20T22:54:28.363153+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:43.758162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:50:40.784154+0000","last_clean_scrub_stamp":"2024-09-20T22:52:43.758162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":1476,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-09-20T22:54:28.310876+0000","last_change":"2024-09-20T22:54:02.131730+0000","last_active":"2024-09-20T22:54:28.310876+0000","last_peered":"2024-09-20T22:54:28.310876+0000","last_clean":"2024-09-20T22:54:28.310876+0000","last_became_active":"2024-09-20T22:41:44.832131+0000","last_became_peered":"2024-09-20T22:41:44.832131+0000","last_unstale":"2024-09-20T22:54:28.310876+0000","last_undegraded":"2024-09-20T22:54:28.310876+0000","last_fullsized":"2024-09-20T22:54:28.310876+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:54:02.131659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:45:25.002199+0000","last_clean_scrub_stamp":"2024-09-20T22:54:02.131659+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:08.048458+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":814,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-09-20T22:54:28.310709+0000","last_change":"2024-09-20T22:54:17.340036+0000","last_active":"2024-09-20T22:54:28.310709+0000","last_peered":"2024-09-20T22:54:28.310709+0000","last_clean":"2024-09-20T22:54:28.310709+0000","last_became_active":"2024-09-20T22:54:17.339702+0000","last_became_peered":"2024-09-20T22:54:17.339702+0000","last_unstale":"2024-09-20T22:54:28.310709+0000","last_undegraded":"2024-09-20T22:54:28.310709+0000","last_fullsized":"2024-09-20T22:54:28.310709+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:04.583683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:50:41.885844+0000","last_clean_scrub_stamp":"2024-09-20T22:53:04.583683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:31.002293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"625'18","reported_seq":1605,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-09-20T22:54:28.310342+0000","last_change":"2024-09-20T22:54:18.284372+0000","last_active":"2024-09-20T22:54:28.310342+0000","last_peered":"2024-09-20T22:54:28.310342+0000","last_clean":"2024-09-20T22:54:28.310342+0000","last_became_active":"2024-09-20T22:54:17.333427+0000","last_became_peered":"2024-09-20T22:54:17.333427+0000","last_unstale":"2024-09-20T22:54:28.310342+0000","last_undegraded":"2024-09-20T22:54:28.310342+0000","last_fullsized":"2024-09-20T22:54:28.310342+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'18","last_scrub_stamp":"2024-09-20T22:54:18.284289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:54:18.284289+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:22.283251+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":1100,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":802,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-09-20T22:54:28.310928+0000","last_change":"2024-09-20T22:52:45.738376+0000","last_active":"2024-09-20T22:54:28.310928+0000","last_peered":"2024-09-20T22:54:28.310928+0000","last_clean":"2024-09-20T22:54:28.310928+0000","last_became_active":"2024-09-20T22:48:05.250319+0000","last_became_peered":"2024-09-20T22:48:05.250319+0000","last_unstale":"2024-09-20T22:54:28.310928+0000","last_undegraded":"2024-09-20T22:54:28.310928+0000","last_fullsized":"2024-09-20T22:54:28.310928+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:45.738275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:52:45.738275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:31.370361+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"154'2","reported_seq":1484,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-09-20T22:54:28.322154+0000","last_change":"2024-09-20T22:52:45.578519+0000","last_active":"2024-09-20T22:54:28.322154+0000","last_peered":"2024-09-20T22:54:28.322154+0000","last_clean":"2024-09-20T22:54:28.322154+0000","last_became_active":"2024-09-20T22:41:44.199822+0000","last_became_peered":"2024-09-20T22:41:44.199822+0000","last_unstale":"2024-09-20T22:54:28.322154+0000","last_undegraded":"2024-09-20T22:54:28.322154+0000","last_fullsized":"2024-09-20T22:54:28.322154+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"154'2","last_scrub_stamp":"2024-09-20T22:52:45.578416+0000","last_deep_scrub":"154'2","last_deep_scrub_stamp":"2024-09-20T22:43:54.134066+0000","last_clean_scrub_stamp":"2024-09-20T22:52:45.578416+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:28.667419+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"67'1","reported_seq":37,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-09-20T22:54:28.362265+0000","last_change":"2024-09-20T22:54:17.366841+0000","last_active":"2024-09-20T22:54:28.362265+0000","last_peered":"2024-09-20T22:54:28.362265+0000","last_clean":"2024-09-20T22:54:28.362265+0000","last_became_active":"2024-09-20T22:54:17.348571+0000","last_became_peered":"2024-09-20T22:54:17.348571+0000","last_unstale":"2024-09-20T22:54:28.362265+0000","last_undegraded":"2024-09-20T22:54:28.362265+0000","last_fullsized":"2024-09-20T22:54:28.362265+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-09-20T22:53:08.612395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:53:08.612395+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:33.257497+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"625'25","reported_seq":863,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-09-20T22:54:28.312407+0000","last_change":"2024-09-20T22:53:26.146547+0000","last_active":"2024-09-20T22:54:28.312407+0000","last_peered":"2024-09-20T22:54:28.312407+0000","last_clean":"2024-09-20T22:54:28.312407+0000","last_became_active":"2024-09-20T22:47:49.232181+0000","last_became_peered":"2024-09-20T22:47:49.232181+0000","last_unstale":"2024-09-20T22:54:28.312407+0000","last_undegraded":"2024-09-20T22:54:28.312407+0000","last_fullsized":"2024-09-20T22:54:28.312407+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'25","last_scrub_stamp":"2024-09-20T22:53:26.146466+0000","last_deep_scrub":"151'15","last_deep_scrub_stamp":"2024-09-20T22:43:50.715761+0000","last_clean_scrub_stamp":"2024-09-20T22:53:26.146466+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:51.749507+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"661'11","reported_seq":1694,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-09-20T22:54:28.314022+0000","last_change":"2024-09-20T22:53:34.944681+0000","last_active":"2024-09-20T22:54:28.314022+0000","last_peered":"2024-09-20T22:54:28.314022+0000","last_clean":"2024-09-20T22:54:28.314022+0000","last_became_active":"2024-09-20T22:48:05.244270+0000","last_became_peered":"2024-09-20T22:48:05.244270+0000","last_unstale":"2024-09-20T22:54:28.314022+0000","last_undegraded":"2024-09-20T22:54:28.314022+0000","last_fullsized":"2024-09-20T22:54:28.314022+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"661'11","last_scrub_stamp":"2024-09-20T22:53:34.944591+0000","last_deep_scrub":"661'11","last_deep_scrub_stamp":"2024-09-20T22:52:21.907943+0000","last_clean_scrub_stamp":"2024-09-20T22:53:34.944591+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:38.596007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.c","version":"788'2537","reported_seq":5872,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-09-20T22:54:28.374221+0000","last_change":"2024-09-20T22:54:26.320316+0000","last_active":"2024-09-20T22:54:28.374221+0000","last_peered":"2024-09-20T22:54:28.374221+0000","last_clean":"2024-09-20T22:54:28.374221+0000","last_became_active":"2024-09-20T22:42:52.404027+0000","last_became_peered":"2024-09-20T22:42:52.404027+0000","last_unstale":"2024-09-20T22:54:28.374221+0000","last_undegraded":"2024-09-20T22:54:28.374221+0000","last_fullsized":"2024-09-20T22:54:28.374221+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"754'2388","last_scrub_stamp":"2024-09-20T22:53:54.900766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:53:54.900766+0000","objects_scrubbed":32,"log_size":2537,"log_dups_size":0,"ondisk_log_size":2537,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:05.992598+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0.0010156950000000001,"stat_sum":{"num_bytes":70991522,"num_objects":30,"num_object_clones":23,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":1,"num_read":932,"num_read_kb":979776,"num_write":633,"num_write_kb":249175,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"11f","length":"5"},{"start":"125","length":"9"},{"start":"130","length":"9"},{"start":"13b","length":"3"},{"start":"13f","length":"3"},{"start":"143","length":"2"},{"start":"146","length":"2"},{"start":"14b","length":"1"},{"start":"14d","length":"1"},{"start":"150","length":"3"},{"start":"154","length":"1"},{"start":"159","length":"1"},{"start":"163","length":"1"}]},{"pgid":"1.a","version":"77'128","reported_seq":1757,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-09-20T22:54:28.369192+0000","last_change":"2024-09-20T22:53:28.095343+0000","last_active":"2024-09-20T22:54:28.369192+0000","last_peered":"2024-09-20T22:54:28.369192+0000","last_clean":"2024-09-20T22:54:28.369192+0000","last_became_active":"2024-09-20T22:42:52.407791+0000","last_became_peered":"2024-09-20T22:42:52.407791+0000","last_unstale":"2024-09-20T22:54:28.369192+0000","last_undegraded":"2024-09-20T22:54:28.369192+0000","last_fullsized":"2024-09-20T22:54:28.369192+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:53:28.095266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:53:28.095266+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:41.330892+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"625'27","reported_seq":57,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-09-20T22:54:28.363199+0000","last_change":"2024-09-20T22:54:22.820985+0000","last_active":"2024-09-20T22:54:28.363199+0000","last_peered":"2024-09-20T22:54:28.363199+0000","last_clean":"2024-09-20T22:54:28.363199+0000","last_became_active":"2024-09-20T22:54:17.347544+0000","last_became_peered":"2024-09-20T22:54:17.347544+0000","last_unstale":"2024-09-20T22:54:28.363199+0000","last_undegraded":"2024-09-20T22:54:28.363199+0000","last_fullsized":"2024-09-20T22:54:28.363199+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'27","last_scrub_stamp":"2024-09-20T22:54:22.820824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:54:22.820824+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:36.004359+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":1491,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-09-20T22:54:28.310724+0000","last_change":"2024-09-20T22:54:17.342130+0000","last_active":"2024-09-20T22:54:28.310724+0000","last_peered":"2024-09-20T22:54:28.310724+0000","last_clean":"2024-09-20T22:54:28.310724+0000","last_became_active":"2024-09-20T22:54:17.341270+0000","last_became_peered":"2024-09-20T22:54:17.341270+0000","last_unstale":"2024-09-20T22:54:28.310724+0000","last_undegraded":"2024-09-20T22:54:28.310724+0000","last_fullsized":"2024-09-20T22:54:28.310724+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:29.394097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:49:12.070675+0000","last_clean_scrub_stamp":"2024-09-20T22:51:29.394097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:31.292480+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"408'12","reported_seq":883,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-09-20T22:54:28.311163+0000","last_change":"2024-09-20T22:54:22.471855+0000","last_active":"2024-09-20T22:54:28.311163+0000","last_peered":"2024-09-20T22:54:28.311163+0000","last_clean":"2024-09-20T22:54:28.311163+0000","last_became_active":"2024-09-20T22:48:05.242914+0000","last_became_peered":"2024-09-20T22:48:05.242914+0000","last_unstale":"2024-09-20T22:54:28.311163+0000","last_undegraded":"2024-09-20T22:54:28.311163+0000","last_fullsized":"2024-09-20T22:54:28.311163+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"408'12","last_scrub_stamp":"2024-09-20T22:54:22.471772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:54:22.471772+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:33.991199+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":1508,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-09-20T22:54:28.369301+0000","last_change":"2024-09-20T22:54:22.078071+0000","last_active":"2024-09-20T22:54:28.369301+0000","last_peered":"2024-09-20T22:54:28.369301+0000","last_clean":"2024-09-20T22:54:28.369301+0000","last_became_active":"2024-09-20T22:54:17.340327+0000","last_became_peered":"2024-09-20T22:54:17.340327+0000","last_unstale":"2024-09-20T22:54:28.369301+0000","last_undegraded":"2024-09-20T22:54:28.369301+0000","last_fullsized":"2024-09-20T22:54:28.369301+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:54:22.077994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:54:22.077994+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:29.125180+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3","version":"788'382","reported_seq":3706,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-09-20T22:54:28.388777+0000","last_change":"2024-09-20T22:54:27.323389+0000","last_active":"2024-09-20T22:54:28.388777+0000","last_peered":"2024-09-20T22:54:28.388777+0000","last_clean":"2024-09-20T22:54:28.388777+0000","last_became_active":"2024-09-20T22:49:05.221402+0000","last_became_peered":"2024-09-20T22:49:05.221402+0000","last_unstale":"2024-09-20T22:54:28.388777+0000","last_undegraded":"2024-09-20T22:54:28.388777+0000","last_fullsized":"2024-09-20T22:54:28.388777+0000","mapping_epoch":404,"log_start":"456'129","ondisk_log_start":"456'129","created":94,"last_epoch_clean":465,"parent":"0.0","parent_split_bits":0,"last_scrub":"609'243","last_scrub_stamp":"2024-09-20T22:51:31.402013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:51:31.402013+0000","objects_scrubbed":6,"log_size":253,"log_dups_size":149,"ondisk_log_size":253,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":1.006768785,"stat_sum":{"num_bytes":17569414,"num_objects":9,"num_object_clones":8,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":115,"num_read_kb":147023,"num_write":87,"num_write_kb":29607,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":20196357,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"11f","length":"5"},{"start":"125","length":"9"},{"start":"130","length":"9"},{"start":"13b","length":"3"},{"start":"13f","length":"3"},{"start":"143","length":"2"},{"start":"146","length":"2"},{"start":"14b","length":"1"},{"start":"14d","length":"1"},{"start":"150","length":"3"},{"start":"154","length":"1"},{"start":"159","length":"1"},{"start":"163","length":"1"}]},{"pgid":"1.5","version":"77'128","reported_seq":1663,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-09-20T22:54:28.311079+0000","last_change":"2024-09-20T22:53:35.470320+0000","last_active":"2024-09-20T22:54:28.311079+0000","last_peered":"2024-09-20T22:54:28.311079+0000","last_clean":"2024-09-20T22:54:28.311079+0000","last_became_active":"2024-09-20T22:48:08.207018+0000","last_became_peered":"2024-09-20T22:48:08.207018+0000","last_unstale":"2024-09-20T22:54:28.311079+0000","last_undegraded":"2024-09-20T22:54:28.311079+0000","last_fullsized":"2024-09-20T22:54:28.311079+0000","mapping_epoch":407,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":408,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:53:35.470248+0000","last_deep_scrub":"77'128","last_deep_scrub_stamp":"2024-09-20T22:45:41.608075+0000","last_clean_scrub_stamp":"2024-09-20T22:53:35.470248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:57.578065+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"625'30","reported_seq":1582,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-09-20T22:54:28.310986+0000","last_change":"2024-09-20T22:53:48.346424+0000","last_active":"2024-09-20T22:54:28.310986+0000","last_peered":"2024-09-20T22:54:28.310986+0000","last_clean":"2024-09-20T22:54:28.310986+0000","last_became_active":"2024-09-20T22:47:49.217605+0000","last_became_peered":"2024-09-20T22:47:49.217605+0000","last_unstale":"2024-09-20T22:54:28.310986+0000","last_undegraded":"2024-09-20T22:54:28.310986+0000","last_fullsized":"2024-09-20T22:54:28.310986+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'30","last_scrub_stamp":"2024-09-20T22:53:48.346341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:53:48.346341+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:02.270695+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":1499,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-09-20T22:54:28.311795+0000","last_change":"2024-09-20T22:52:45.537243+0000","last_active":"2024-09-20T22:54:28.311795+0000","last_peered":"2024-09-20T22:54:28.311795+0000","last_clean":"2024-09-20T22:54:28.311795+0000","last_became_active":"2024-09-20T22:47:49.227069+0000","last_became_peered":"2024-09-20T22:47:49.227069+0000","last_unstale":"2024-09-20T22:54:28.311795+0000","last_undegraded":"2024-09-20T22:54:28.311795+0000","last_fullsized":"2024-09-20T22:54:28.311795+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:45.537173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:52:45.537173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:31.292600+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":803,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-09-20T22:54:28.310619+0000","last_change":"2024-09-20T22:53:26.623524+0000","last_active":"2024-09-20T22:54:28.310619+0000","last_peered":"2024-09-20T22:54:28.310619+0000","last_clean":"2024-09-20T22:54:28.310619+0000","last_became_active":"2024-09-20T22:48:05.230003+0000","last_became_peered":"2024-09-20T22:48:05.230003+0000","last_unstale":"2024-09-20T22:54:28.310619+0000","last_undegraded":"2024-09-20T22:54:28.310619+0000","last_fullsized":"2024-09-20T22:54:28.310619+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:26.623455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:07.947945+0000","last_clean_scrub_stamp":"2024-09-20T22:53:26.623455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:36.274129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":795,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-09-20T22:54:28.310588+0000","last_change":"2024-09-20T22:51:55.748527+0000","last_active":"2024-09-20T22:54:28.310588+0000","last_peered":"2024-09-20T22:54:28.310588+0000","last_clean":"2024-09-20T22:54:28.310588+0000","last_became_active":"2024-09-20T22:48:05.239917+0000","last_became_peered":"2024-09-20T22:48:05.239917+0000","last_unstale":"2024-09-20T22:54:28.310588+0000","last_undegraded":"2024-09-20T22:54:28.310588+0000","last_fullsized":"2024-09-20T22:54:28.310588+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:55.748425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:53.036601+0000","last_clean_scrub_stamp":"2024-09-20T22:51:55.748425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:31.370227+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2","version":"781'823","reported_seq":3471,"reported_epoch":788,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-20T22:54:28.311821+0000","last_change":"2024-09-20T22:54:18.384334+0000","last_active":"2024-09-20T22:54:28.311821+0000","last_peered":"2024-09-20T22:54:28.311821+0000","last_clean":"2024-09-20T22:54:15.316680+0000","last_became_active":"2024-09-20T22:54:18.366594+0000","last_became_peered":"2024-09-20T22:54:18.366594+0000","last_unstale":"2024-09-20T22:54:28.311821+0000","last_undegraded":"2024-09-20T22:54:28.311821+0000","last_fullsized":"2024-09-20T22:54:28.311821+0000","mapping_epoch":777,"log_start":"471'464","ondisk_log_start":"471'464","created":94,"last_epoch_clean":663,"parent":"0.0","parent_split_bits":0,"last_scrub":"282'120","last_scrub_stamp":"2024-09-20T22:46:22.099642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:46:22.099642+0000","objects_scrubbed":1,"log_size":359,"log_dups_size":687,"ondisk_log_size":359,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":41,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.10959926,"stat_sum":{"num_bytes":41768978,"num_objects":15,"num_object_clones":11,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":15,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":426,"num_read_kb":435000,"num_write":271,"num_write_kb":106797,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":145,"num_bytes_recovered":108958518,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,2,7],"avail_no_missing":["3","2","7"],"object_location_counts":[{"shards":"2,3,7","objects":15}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"77'128","reported_seq":1781,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-09-20T22:54:28.311945+0000","last_change":"2024-09-20T22:52:02.474080+0000","last_active":"2024-09-20T22:54:28.311945+0000","last_peered":"2024-09-20T22:54:28.311945+0000","last_clean":"2024-09-20T22:54:28.311945+0000","last_became_active":"2024-09-20T22:46:15.181305+0000","last_became_peered":"2024-09-20T22:46:15.181305+0000","last_unstale":"2024-09-20T22:54:28.311945+0000","last_undegraded":"2024-09-20T22:54:28.311945+0000","last_fullsized":"2024-09-20T22:54:28.311945+0000","mapping_epoch":94,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:52:02.473979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:52:02.473979+0000","objects_scrubbed":1,"log_size":0,"log_dups_size":59,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:31.292542+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"625'28","reported_seq":832,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-09-20T22:54:28.310490+0000","last_change":"2024-09-20T22:51:58.712602+0000","last_active":"2024-09-20T22:54:28.310490+0000","last_peered":"2024-09-20T22:54:28.310490+0000","last_clean":"2024-09-20T22:54:28.310490+0000","last_became_active":"2024-09-20T22:48:05.235812+0000","last_became_peered":"2024-09-20T22:48:05.235812+0000","last_unstale":"2024-09-20T22:54:28.310490+0000","last_undegraded":"2024-09-20T22:54:28.310490+0000","last_fullsized":"2024-09-20T22:54:28.310490+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'28","last_scrub_stamp":"2024-09-20T22:51:58.712488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:51:58.712488+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:31.370210+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":41,"num_read_kb":34,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"67'1","reported_seq":870,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-09-20T22:54:28.312385+0000","last_change":"2024-09-20T22:54:17.366948+0000","last_active":"2024-09-20T22:54:28.312385+0000","last_peered":"2024-09-20T22:54:28.312385+0000","last_clean":"2024-09-20T22:54:28.312385+0000","last_became_active":"2024-09-20T22:54:17.340423+0000","last_became_peered":"2024-09-20T22:54:17.340423+0000","last_unstale":"2024-09-20T22:54:28.312385+0000","last_undegraded":"2024-09-20T22:54:28.312385+0000","last_fullsized":"2024-09-20T22:54:28.312385+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-09-20T22:53:32.995211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:53:32.995211+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:56.795754+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":230,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"778'16","reported_seq":1698,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-09-20T22:54:28.311214+0000","last_change":"2024-09-20T22:54:17.375877+0000","last_active":"2024-09-20T22:54:28.311214+0000","last_peered":"2024-09-20T22:54:28.311214+0000","last_clean":"2024-09-20T22:54:28.311214+0000","last_became_active":"2024-09-20T22:54:17.341871+0000","last_became_peered":"2024-09-20T22:54:17.341871+0000","last_unstale":"2024-09-20T22:54:28.311214+0000","last_undegraded":"2024-09-20T22:54:28.311214+0000","last_fullsized":"2024-09-20T22:54:28.311214+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"600'13","last_scrub_stamp":"2024-09-20T22:51:32.448512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:51:32.448512+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:31.292510+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":1483,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-09-20T22:54:28.320919+0000","last_change":"2024-09-20T22:53:02.610452+0000","last_active":"2024-09-20T22:54:28.320919+0000","last_peered":"2024-09-20T22:54:28.320919+0000","last_clean":"2024-09-20T22:54:28.320919+0000","last_became_active":"2024-09-20T22:41:44.831901+0000","last_became_peered":"2024-09-20T22:41:44.831901+0000","last_unstale":"2024-09-20T22:54:28.320919+0000","last_undegraded":"2024-09-20T22:54:28.320919+0000","last_fullsized":"2024-09-20T22:54:28.320919+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:02.610359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:53:02.610359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:32.529803+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1","version":"788'483","reported_seq":3890,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-09-20T22:54:28.376807+0000","last_change":"2024-09-20T22:54:27.323194+0000","last_active":"2024-09-20T22:54:28.376807+0000","last_peered":"2024-09-20T22:54:28.376807+0000","last_clean":"2024-09-20T22:54:28.376807+0000","last_became_active":"2024-09-20T22:49:21.232750+0000","last_became_peered":"2024-09-20T22:49:21.232750+0000","last_unstale":"2024-09-20T22:54:28.376807+0000","last_undegraded":"2024-09-20T22:54:28.376807+0000","last_fullsized":"2024-09-20T22:54:28.376807+0000","mapping_epoch":407,"log_start":"474'259","ondisk_log_start":"474'259","created":94,"last_epoch_clean":481,"parent":"0.0","parent_split_bits":0,"last_scrub":"703'407","last_scrub_stamp":"2024-09-20T22:53:06.625437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:53:06.625437+0000","objects_scrubbed":6,"log_size":224,"log_dups_size":338,"ondisk_log_size":224,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:32.386077+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.0050803259999999996,"stat_sum":{"num_bytes":13116175,"num_objects":8,"num_object_clones":7,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":125,"num_read_kb":142535,"num_write":101,"num_write_kb":51187,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":8732217,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"11f","length":"5"},{"start":"125","length":"9"},{"start":"130","length":"9"},{"start":"13b","length":"3"},{"start":"13f","length":"3"},{"start":"143","length":"2"},{"start":"146","length":"2"},{"start":"14b","length":"1"},{"start":"14d","length":"1"},{"start":"150","length":"3"},{"start":"154","length":"1"},{"start":"159","length":"1"},{"start":"163","length":"1"}]},{"pgid":"1.7","version":"77'128","reported_seq":1802,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-09-20T22:54:28.313824+0000","last_change":"2024-09-20T22:53:56.869885+0000","last_active":"2024-09-20T22:54:28.313824+0000","last_peered":"2024-09-20T22:54:28.313824+0000","last_clean":"2024-09-20T22:54:28.313824+0000","last_became_active":"2024-09-20T22:46:03.169596+0000","last_became_peered":"2024-09-20T22:46:03.169596+0000","last_unstale":"2024-09-20T22:54:28.313824+0000","last_undegraded":"2024-09-20T22:54:28.313824+0000","last_fullsized":"2024-09-20T22:54:28.313824+0000","mapping_epoch":94,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:53:56.869735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:53:56.869735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:09.756086+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"625'20","reported_seq":45,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-09-20T22:54:28.362452+0000","last_change":"2024-09-20T22:54:17.376198+0000","last_active":"2024-09-20T22:54:28.362452+0000","last_peered":"2024-09-20T22:54:28.362452+0000","last_clean":"2024-09-20T22:54:28.362452+0000","last_became_active":"2024-09-20T22:54:17.349113+0000","last_became_peered":"2024-09-20T22:54:17.349113+0000","last_unstale":"2024-09-20T22:54:28.362452+0000","last_undegraded":"2024-09-20T22:54:28.362452+0000","last_fullsized":"2024-09-20T22:54:28.362452+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'20","last_scrub_stamp":"2024-09-20T22:52:19.125462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:52:19.125462+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":28,"num_read_kb":23,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":1497,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-09-20T22:54:28.369760+0000","last_change":"2024-09-20T22:52:46.062448+0000","last_active":"2024-09-20T22:54:28.369760+0000","last_peered":"2024-09-20T22:54:28.369760+0000","last_clean":"2024-09-20T22:54:28.369760+0000","last_became_active":"2024-09-20T22:41:38.155629+0000","last_became_peered":"2024-09-20T22:41:38.155629+0000","last_unstale":"2024-09-20T22:54:28.369760+0000","last_undegraded":"2024-09-20T22:54:28.369760+0000","last_fullsized":"2024-09-20T22:54:28.369760+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:46.062352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:52:46.062352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:30.065592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"733'12","reported_seq":922,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-09-20T22:54:28.310384+0000","last_change":"2024-09-20T22:53:33.928576+0000","last_active":"2024-09-20T22:54:28.310384+0000","last_peered":"2024-09-20T22:54:28.310384+0000","last_clean":"2024-09-20T22:54:28.310384+0000","last_became_active":"2024-09-20T22:47:49.227264+0000","last_became_peered":"2024-09-20T22:47:49.227264+0000","last_unstale":"2024-09-20T22:54:28.310384+0000","last_undegraded":"2024-09-20T22:54:28.310384+0000","last_fullsized":"2024-09-20T22:54:28.310384+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"733'12","last_scrub_stamp":"2024-09-20T22:53:33.928499+0000","last_deep_scrub":"733'12","last_deep_scrub_stamp":"2024-09-20T22:53:33.928499+0000","last_clean_scrub_stamp":"2024-09-20T22:53:33.928499+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:02.678336+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":836,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-09-20T22:54:28.309724+0000","last_change":"2024-09-20T22:53:57.868607+0000","last_active":"2024-09-20T22:54:28.309724+0000","last_peered":"2024-09-20T22:54:28.309724+0000","last_clean":"2024-09-20T22:54:28.309724+0000","last_became_active":"2024-09-20T22:47:49.232380+0000","last_became_peered":"2024-09-20T22:47:49.232380+0000","last_unstale":"2024-09-20T22:54:28.309724+0000","last_undegraded":"2024-09-20T22:54:28.309724+0000","last_fullsized":"2024-09-20T22:54:28.309724+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:57.868534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:53:57.868534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:14.269112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.0","version":"788'772","reported_seq":4291,"reported_epoch":788,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T22:54:28.437403+0000","last_change":"2024-09-20T22:54:26.435234+0000","last_active":"2024-09-20T22:54:28.437403+0000","last_peered":"2024-09-20T22:54:28.437403+0000","last_clean":"2024-09-20T22:54:28.437403+0000","last_became_active":"2024-09-20T22:54:20.418799+0000","last_became_peered":"2024-09-20T22:54:20.418799+0000","last_unstale":"2024-09-20T22:54:28.437403+0000","last_undegraded":"2024-09-20T22:54:28.437403+0000","last_fullsized":"2024-09-20T22:54:28.437403+0000","mapping_epoch":779,"log_start":"480'253","ondisk_log_start":"480'253","created":94,"last_epoch_clean":780,"parent":"0.0","parent_split_bits":0,"last_scrub":"562'368","last_scrub_stamp":"2024-09-20T22:50:44.669910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:50:44.669910+0000","objects_scrubbed":15,"log_size":519,"log_dups_size":416,"ondisk_log_size":519,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":14,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.069375678999999996,"stat_sum":{"num_bytes":50717870,"num_objects":25,"num_object_clones":22,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":1,"num_read":273,"num_read_kb":282737,"num_write":228,"num_write_kb":82820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":54,"num_bytes_recovered":106304666,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"11f","length":"5"},{"start":"125","length":"9"},{"start":"130","length":"9"},{"start":"13b","length":"2"},{"start":"13f","length":"2"}]},{"pgid":"1.6","version":"77'128","reported_seq":1824,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-09-20T22:54:28.313712+0000","last_change":"2024-09-20T22:54:22.148323+0000","last_active":"2024-09-20T22:54:28.313712+0000","last_peered":"2024-09-20T22:54:28.313712+0000","last_clean":"2024-09-20T22:54:28.313712+0000","last_became_active":"2024-09-20T22:47:52.212190+0000","last_became_peered":"2024-09-20T22:47:52.212190+0000","last_unstale":"2024-09-20T22:54:28.313712+0000","last_undegraded":"2024-09-20T22:54:28.313712+0000","last_fullsized":"2024-09-20T22:54:28.313712+0000","mapping_epoch":391,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:54:22.148244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:54:22.148244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:46.700798+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"625'37","reported_seq":869,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-09-20T22:54:28.310083+0000","last_change":"2024-09-20T22:52:39.703172+0000","last_active":"2024-09-20T22:54:28.310083+0000","last_peered":"2024-09-20T22:54:28.310083+0000","last_clean":"2024-09-20T22:54:28.310083+0000","last_became_active":"2024-09-20T22:47:49.222009+0000","last_became_peered":"2024-09-20T22:47:49.222009+0000","last_unstale":"2024-09-20T22:54:28.310083+0000","last_undegraded":"2024-09-20T22:54:28.310083+0000","last_fullsized":"2024-09-20T22:54:28.310083+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'37","last_scrub_stamp":"2024-09-20T22:52:39.703093+0000","last_deep_scrub":"515'27","last_deep_scrub_stamp":"2024-09-20T22:49:55.964153+0000","last_clean_scrub_stamp":"2024-09-20T22:52:39.703093+0000","objects_scrubbed":8,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:29.935918+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":53,"num_read_kb":39,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":33,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-09-20T22:54:28.363037+0000","last_change":"2024-09-20T22:54:17.342966+0000","last_active":"2024-09-20T22:54:28.363037+0000","last_peered":"2024-09-20T22:54:28.363037+0000","last_clean":"2024-09-20T22:54:28.363037+0000","last_became_active":"2024-09-20T22:54:17.341248+0000","last_became_peered":"2024-09-20T22:54:17.341248+0000","last_unstale":"2024-09-20T22:54:28.363037+0000","last_undegraded":"2024-09-20T22:54:28.363037+0000","last_fullsized":"2024-09-20T22:54:28.363037+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:55.227211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:48:23.320031+0000","last_clean_scrub_stamp":"2024-09-20T22:53:55.227211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:23.492541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":1516,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-09-20T22:54:28.313239+0000","last_change":"2024-09-20T22:53:57.882313+0000","last_active":"2024-09-20T22:54:28.313239+0000","last_peered":"2024-09-20T22:54:28.313239+0000","last_clean":"2024-09-20T22:54:28.313239+0000","last_became_active":"2024-09-20T22:47:49.214055+0000","last_became_peered":"2024-09-20T22:47:49.214055+0000","last_unstale":"2024-09-20T22:54:28.313239+0000","last_undegraded":"2024-09-20T22:54:28.313239+0000","last_fullsized":"2024-09-20T22:54:28.313239+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:57.882212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:53:57.882212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:12.731589+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":33,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-09-20T22:54:28.362365+0000","last_change":"2024-09-20T22:54:17.340872+0000","last_active":"2024-09-20T22:54:28.362365+0000","last_peered":"2024-09-20T22:54:28.362365+0000","last_clean":"2024-09-20T22:54:28.362365+0000","last_became_active":"2024-09-20T22:54:17.340458+0000","last_became_peered":"2024-09-20T22:54:17.340458+0000","last_unstale":"2024-09-20T22:54:28.362365+0000","last_undegraded":"2024-09-20T22:54:28.362365+0000","last_fullsized":"2024-09-20T22:54:28.362365+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:54:15.141588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:48:41.208853+0000","last_clean_scrub_stamp":"2024-09-20T22:54:15.141588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:41.084042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.7","version":"788'1499","reported_seq":4998,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-09-20T22:54:28.362093+0000","last_change":"2024-09-20T22:54:26.322675+0000","last_active":"2024-09-20T22:54:28.362093+0000","last_peered":"2024-09-20T22:54:28.362093+0000","last_clean":"2024-09-20T22:54:28.362093+0000","last_became_active":"2024-09-20T22:48:43.224386+0000","last_became_peered":"2024-09-20T22:48:43.224386+0000","last_unstale":"2024-09-20T22:54:28.362093+0000","last_undegraded":"2024-09-20T22:54:28.362093+0000","last_fullsized":"2024-09-20T22:54:28.362093+0000","mapping_epoch":391,"log_start":"436'456","ondisk_log_start":"436'456","created":94,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"772'1452","last_scrub_stamp":"2024-09-20T22:54:14.120176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:54:14.120176+0000","objects_scrubbed":23,"log_size":1043,"log_dups_size":886,"ondisk_log_size":1043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:32.699347+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.003306879,"stat_sum":{"num_bytes":60761273,"num_objects":25,"num_object_clones":20,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":1,"num_read":498,"num_read_kb":506237,"num_write":428,"num_write_kb":147079,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":115,"num_bytes_recovered":51615167,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"11f","length":"5"},{"start":"125","length":"9"},{"start":"130","length":"9"},{"start":"13b","length":"3"},{"start":"13f","length":"3"},{"start":"143","length":"2"},{"start":"146","length":"2"},{"start":"14b","length":"1"},{"start":"14d","length":"1"},{"start":"150","length":"3"},{"start":"154","length":"1"},{"start":"159","length":"1"},{"start":"163","length":"1"}]},{"pgid":"1.1","version":"77'128","reported_seq":1805,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-09-20T22:54:28.368805+0000","last_change":"2024-09-20T22:52:42.146147+0000","last_active":"2024-09-20T22:54:28.368805+0000","last_peered":"2024-09-20T22:54:28.368805+0000","last_clean":"2024-09-20T22:54:28.368805+0000","last_became_active":"2024-09-20T22:47:52.208102+0000","last_became_peered":"2024-09-20T22:47:52.208102+0000","last_unstale":"2024-09-20T22:54:28.368805+0000","last_undegraded":"2024-09-20T22:54:28.368805+0000","last_fullsized":"2024-09-20T22:54:28.368805+0000","mapping_epoch":391,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:52:42.146069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:52:42.146069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:30.065493+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"778'52","reported_seq":58,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-09-20T22:54:28.363097+0000","last_change":"2024-09-20T22:54:17.373012+0000","last_active":"2024-09-20T22:54:28.363097+0000","last_peered":"2024-09-20T22:54:28.363097+0000","last_clean":"2024-09-20T22:54:28.363097+0000","last_became_active":"2024-09-20T22:54:17.342586+0000","last_became_peered":"2024-09-20T22:54:17.342586+0000","last_unstale":"2024-09-20T22:54:28.363097+0000","last_undegraded":"2024-09-20T22:54:28.363097+0000","last_fullsized":"2024-09-20T22:54:28.363097+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'51","last_scrub_stamp":"2024-09-20T22:53:57.184165+0000","last_deep_scrub":"86'38","last_deep_scrub_stamp":"2024-09-20T22:42:43.935677+0000","last_clean_scrub_stamp":"2024-09-20T22:53:57.184165+0000","objects_scrubbed":10,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:59.706724+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":795,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-09-20T22:54:28.310685+0000","last_change":"2024-09-20T22:52:36.695537+0000","last_active":"2024-09-20T22:54:28.310685+0000","last_peered":"2024-09-20T22:54:28.310685+0000","last_clean":"2024-09-20T22:54:28.310685+0000","last_became_active":"2024-09-20T22:48:05.237756+0000","last_became_peered":"2024-09-20T22:48:05.237756+0000","last_unstale":"2024-09-20T22:54:28.310685+0000","last_undegraded":"2024-09-20T22:54:28.310685+0000","last_fullsized":"2024-09-20T22:54:28.310685+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:36.695446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:52:36.695446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:31.370278+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"69'2","reported_seq":1535,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-09-20T22:54:28.369581+0000","last_change":"2024-09-20T22:54:23.097050+0000","last_active":"2024-09-20T22:54:28.369581+0000","last_peered":"2024-09-20T22:54:28.369581+0000","last_clean":"2024-09-20T22:54:28.369581+0000","last_became_active":"2024-09-20T22:54:17.341059+0000","last_became_peered":"2024-09-20T22:54:17.341059+0000","last_unstale":"2024-09-20T22:54:28.369581+0000","last_undegraded":"2024-09-20T22:54:28.369581+0000","last_fullsized":"2024-09-20T22:54:28.369581+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2024-09-20T22:54:23.096974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:36.130684+0000","last_clean_scrub_stamp":"2024-09-20T22:54:23.096974+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:40.124528+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00087280599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":854,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-09-20T22:54:28.310309+0000","last_change":"2024-09-20T22:54:17.340261+0000","last_active":"2024-09-20T22:54:28.310309+0000","last_peered":"2024-09-20T22:54:28.310309+0000","last_clean":"2024-09-20T22:54:28.310309+0000","last_became_active":"2024-09-20T22:54:17.339860+0000","last_became_peered":"2024-09-20T22:54:17.339860+0000","last_unstale":"2024-09-20T22:54:28.310309+0000","last_undegraded":"2024-09-20T22:54:28.310309+0000","last_fullsized":"2024-09-20T22:54:28.310309+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:51.849772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:53:51.849772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:18.462456+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":1487,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-09-20T22:54:28.310783+0000","last_change":"2024-09-20T22:52:21.363432+0000","last_active":"2024-09-20T22:54:28.310783+0000","last_peered":"2024-09-20T22:54:28.310783+0000","last_clean":"2024-09-20T22:54:28.310783+0000","last_became_active":"2024-09-20T22:48:05.234193+0000","last_became_peered":"2024-09-20T22:48:05.234193+0000","last_unstale":"2024-09-20T22:54:28.310783+0000","last_undegraded":"2024-09-20T22:54:28.310783+0000","last_fullsized":"2024-09-20T22:54:28.310783+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:21.363359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:52:21.363359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:31.292557+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"786'558","reported_seq":2531,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-09-20T22:54:28.310410+0000","last_change":"2024-09-20T22:54:26.313630+0000","last_active":"2024-09-20T22:54:28.310410+0000","last_peered":"2024-09-20T22:54:28.310410+0000","last_clean":"2024-09-20T22:54:28.310410+0000","last_became_active":"2024-09-20T22:54:20.412859+0000","last_became_peered":"2024-09-20T22:54:20.412859+0000","last_unstale":"2024-09-20T22:54:28.310410+0000","last_undegraded":"2024-09-20T22:54:28.310410+0000","last_fullsized":"2024-09-20T22:54:28.310410+0000","mapping_epoch":779,"log_start":"456'176","ondisk_log_start":"456'176","created":94,"last_epoch_clean":780,"parent":"0.0","parent_split_bits":0,"last_scrub":"744'500","last_scrub_stamp":"2024-09-20T22:53:49.903129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:53:49.903129+0000","objects_scrubbed":10,"log_size":382,"log_dups_size":318,"ondisk_log_size":382,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:14.705715+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00052385599999999995,"stat_sum":{"num_bytes":22624982,"num_objects":10,"num_object_clones":8,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":1,"num_read":192,"num_read_kb":187338,"num_write":151,"num_write_kb":51652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":41680310,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"11f","length":"5"},{"start":"125","length":"9"},{"start":"130","length":"9"},{"start":"13b","length":"3"},{"start":"13f","length":"3"},{"start":"143","length":"2"},{"start":"146","length":"2"},{"start":"14b","length":"1"},{"start":"14d","length":"1"},{"start":"150","length":"3"},{"start":"154","length":"1"},{"start":"159","length":"1"},{"start":"163","length":"1"}]},{"pgid":"1.2","version":"77'128","reported_seq":1873,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-09-20T22:54:28.320667+0000","last_change":"2024-09-20T22:53:58.495293+0000","last_active":"2024-09-20T22:54:28.320667+0000","last_peered":"2024-09-20T22:54:28.320667+0000","last_clean":"2024-09-20T22:54:28.320667+0000","last_became_active":"2024-09-20T22:48:08.203028+0000","last_became_peered":"2024-09-20T22:48:08.203028+0000","last_unstale":"2024-09-20T22:54:28.320667+0000","last_undegraded":"2024-09-20T22:54:28.320667+0000","last_fullsized":"2024-09-20T22:54:28.320667+0000","mapping_epoch":407,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":408,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:53:58.495191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:53:58.495191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:23.083909+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"625'22","reported_seq":45,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-09-20T22:54:28.363290+0000","last_change":"2024-09-20T22:54:17.379582+0000","last_active":"2024-09-20T22:54:28.363290+0000","last_peered":"2024-09-20T22:54:28.363290+0000","last_clean":"2024-09-20T22:54:28.363290+0000","last_became_active":"2024-09-20T22:54:17.349415+0000","last_became_peered":"2024-09-20T22:54:17.349415+0000","last_unstale":"2024-09-20T22:54:28.363290+0000","last_undegraded":"2024-09-20T22:54:28.363290+0000","last_fullsized":"2024-09-20T22:54:28.363290+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'22","last_scrub_stamp":"2024-09-20T22:53:35.117387+0000","last_deep_scrub":"443'18","last_deep_scrub_stamp":"2024-09-20T22:48:44.182517+0000","last_clean_scrub_stamp":"2024-09-20T22:53:35.117387+0000","objects_scrubbed":5,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:43.508688+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":24,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":1320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":33,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-09-20T22:54:28.362639+0000","last_change":"2024-09-20T22:54:17.344534+0000","last_active":"2024-09-20T22:54:28.362639+0000","last_peered":"2024-09-20T22:54:28.362639+0000","last_clean":"2024-09-20T22:54:28.362639+0000","last_became_active":"2024-09-20T22:54:17.343806+0000","last_became_peered":"2024-09-20T22:54:17.343806+0000","last_unstale":"2024-09-20T22:54:28.362639+0000","last_undegraded":"2024-09-20T22:54:28.362639+0000","last_fullsized":"2024-09-20T22:54:28.362639+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:30.209986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:53:30.209986+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:37.881854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"380'2","reported_seq":1550,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-09-20T22:54:28.369105+0000","last_change":"2024-09-20T22:54:16.438102+0000","last_active":"2024-09-20T22:54:28.369105+0000","last_peered":"2024-09-20T22:54:28.369105+0000","last_clean":"2024-09-20T22:54:28.369105+0000","last_became_active":"2024-09-20T22:47:49.222812+0000","last_became_peered":"2024-09-20T22:47:49.222812+0000","last_unstale":"2024-09-20T22:54:28.369105+0000","last_undegraded":"2024-09-20T22:54:28.369105+0000","last_fullsized":"2024-09-20T22:54:28.369105+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"380'2","last_scrub_stamp":"2024-09-20T22:54:16.437989+0000","last_deep_scrub":"380'2","last_deep_scrub_stamp":"2024-09-20T22:47:40.983039+0000","last_clean_scrub_stamp":"2024-09-20T22:54:16.437989+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:30.798230+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.001301459,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":1495,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-09-20T22:54:28.369666+0000","last_change":"2024-09-20T22:53:25.223722+0000","last_active":"2024-09-20T22:54:28.369666+0000","last_peered":"2024-09-20T22:54:28.369666+0000","last_clean":"2024-09-20T22:54:28.369666+0000","last_became_active":"2024-09-20T22:41:42.185112+0000","last_became_peered":"2024-09-20T22:41:42.185112+0000","last_unstale":"2024-09-20T22:54:28.369666+0000","last_undegraded":"2024-09-20T22:54:28.369666+0000","last_fullsized":"2024-09-20T22:54:28.369666+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:25.223611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:47:22.881109+0000","last_clean_scrub_stamp":"2024-09-20T22:53:25.223611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:53.455503+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":1483,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-09-20T22:54:28.321573+0000","last_change":"2024-09-20T22:53:06.644045+0000","last_active":"2024-09-20T22:54:28.321573+0000","last_peered":"2024-09-20T22:54:28.321573+0000","last_clean":"2024-09-20T22:54:28.321573+0000","last_became_active":"2024-09-20T22:41:44.830515+0000","last_became_peered":"2024-09-20T22:41:44.830515+0000","last_unstale":"2024-09-20T22:54:28.321573+0000","last_undegraded":"2024-09-20T22:54:28.321573+0000","last_fullsized":"2024-09-20T22:54:28.321573+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:06.643947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:53:06.643947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:28.667502+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"778'415","reported_seq":3942,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-09-20T22:54:28.321293+0000","last_change":"2024-09-20T22:54:26.318044+0000","last_active":"2024-09-20T22:54:28.321293+0000","last_peered":"2024-09-20T22:54:28.321293+0000","last_clean":"2024-09-20T22:54:28.321293+0000","last_became_active":"2024-09-20T22:48:57.236417+0000","last_became_peered":"2024-09-20T22:48:57.236417+0000","last_unstale":"2024-09-20T22:54:28.321293+0000","last_undegraded":"2024-09-20T22:54:28.321293+0000","last_fullsized":"2024-09-20T22:54:28.321293+0000","mapping_epoch":391,"log_start":"450'190","ondisk_log_start":"450'190","created":94,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"774'413","last_scrub_stamp":"2024-09-20T22:54:15.568174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:54:15.568174+0000","objects_scrubbed":5,"log_size":225,"log_dups_size":240,"ondisk_log_size":225,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:25.736472+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0010065199999999999,"stat_sum":{"num_bytes":12497960,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":99,"num_read_kb":89344,"num_write":98,"num_write_kb":40424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":19673726,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"11f","length":"5"},{"start":"125","length":"9"},{"start":"130","length":"9"},{"start":"13b","length":"3"},{"start":"13f","length":"3"},{"start":"143","length":"2"},{"start":"146","length":"2"},{"start":"14b","length":"1"},{"start":"14d","length":"1"},{"start":"150","length":"3"},{"start":"154","length":"1"},{"start":"159","length":"1"},{"start":"163","length":"1"}]},{"pgid":"1.3","version":"77'128","reported_seq":1052,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-09-20T22:54:28.362710+0000","last_change":"2024-09-20T22:54:20.413560+0000","last_active":"2024-09-20T22:54:28.362710+0000","last_peered":"2024-09-20T22:54:28.362710+0000","last_clean":"2024-09-20T22:54:28.362710+0000","last_became_active":"2024-09-20T22:54:20.413191+0000","last_became_peered":"2024-09-20T22:54:20.413191+0000","last_unstale":"2024-09-20T22:54:28.362710+0000","last_undegraded":"2024-09-20T22:54:28.362710+0000","last_fullsized":"2024-09-20T22:54:28.362710+0000","mapping_epoch":779,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":780,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:53:05.865942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:53:05.865942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"625'24","reported_seq":900,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-09-20T22:54:28.312504+0000","last_change":"2024-09-20T22:54:17.378683+0000","last_active":"2024-09-20T22:54:28.312504+0000","last_peered":"2024-09-20T22:54:28.312504+0000","last_clean":"2024-09-20T22:54:28.312504+0000","last_became_active":"2024-09-20T22:54:17.341751+0000","last_became_peered":"2024-09-20T22:54:17.341751+0000","last_unstale":"2024-09-20T22:54:28.312504+0000","last_undegraded":"2024-09-20T22:54:28.312504+0000","last_fullsized":"2024-09-20T22:54:28.312504+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'24","last_scrub_stamp":"2024-09-20T22:52:48.829882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:52:48.829882+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:29.936004+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":842,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-09-20T22:54:28.311343+0000","last_change":"2024-09-20T22:53:25.172560+0000","last_active":"2024-09-20T22:54:28.311343+0000","last_peered":"2024-09-20T22:54:28.311343+0000","last_clean":"2024-09-20T22:54:28.311343+0000","last_became_active":"2024-09-20T22:48:05.233093+0000","last_became_peered":"2024-09-20T22:48:05.233093+0000","last_unstale":"2024-09-20T22:54:28.311343+0000","last_undegraded":"2024-09-20T22:54:28.311343+0000","last_fullsized":"2024-09-20T22:54:28.311343+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:25.172487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:44:19.834641+0000","last_clean_scrub_stamp":"2024-09-20T22:53:25.172487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:39.242526+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"75'5","reported_seq":2307,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-09-20T22:54:28.320923+0000","last_change":"2024-09-20T22:54:17.346609+0000","last_active":"2024-09-20T22:54:28.320923+0000","last_peered":"2024-09-20T22:54:28.320923+0000","last_clean":"2024-09-20T22:54:28.320923+0000","last_became_active":"2024-09-20T22:54:17.337205+0000","last_became_peered":"2024-09-20T22:54:17.337205+0000","last_unstale":"2024-09-20T22:54:28.320923+0000","last_undegraded":"2024-09-20T22:54:28.320923+0000","last_fullsized":"2024-09-20T22:54:28.320923+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-09-20T22:52:44.568129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:36.130684+0000","last_clean_scrub_stamp":"2024-09-20T22:52:44.568129+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:28.667386+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00078631700000000003,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":759,"num_read_kb":754,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1556,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":1512,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-09-20T22:54:28.369812+0000","last_change":"2024-09-20T22:54:17.340134+0000","last_active":"2024-09-20T22:54:28.369812+0000","last_peered":"2024-09-20T22:54:28.369812+0000","last_clean":"2024-09-20T22:54:28.369812+0000","last_became_active":"2024-09-20T22:54:17.339805+0000","last_became_peered":"2024-09-20T22:54:17.339805+0000","last_unstale":"2024-09-20T22:54:28.369812+0000","last_undegraded":"2024-09-20T22:54:28.369812+0000","last_fullsized":"2024-09-20T22:54:28.369812+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:48.082027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:52:48.082027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:30.065423+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":33,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-09-20T22:54:28.362953+0000","last_change":"2024-09-20T22:54:17.339943+0000","last_active":"2024-09-20T22:54:28.362953+0000","last_peered":"2024-09-20T22:54:28.362953+0000","last_clean":"2024-09-20T22:54:28.362953+0000","last_became_active":"2024-09-20T22:54:17.339564+0000","last_became_peered":"2024-09-20T22:54:17.339564+0000","last_unstale":"2024-09-20T22:54:28.362953+0000","last_undegraded":"2024-09-20T22:54:28.362953+0000","last_fullsized":"2024-09-20T22:54:28.362953+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:49.031912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:53:49.031912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:06.466960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.6","version":"788'574","reported_seq":4026,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-09-20T22:54:28.361479+0000","last_change":"2024-09-20T22:54:26.317654+0000","last_active":"2024-09-20T22:54:28.361479+0000","last_peered":"2024-09-20T22:54:28.361479+0000","last_clean":"2024-09-20T22:54:28.361479+0000","last_became_active":"2024-09-20T22:48:47.234531+0000","last_became_peered":"2024-09-20T22:48:47.234531+0000","last_unstale":"2024-09-20T22:54:28.361479+0000","last_undegraded":"2024-09-20T22:54:28.361479+0000","last_fullsized":"2024-09-20T22:54:28.361479+0000","mapping_epoch":391,"log_start":"443'181","ondisk_log_start":"443'181","created":94,"last_epoch_clean":447,"parent":"0.0","parent_split_bits":0,"last_scrub":"770'553","last_scrub_stamp":"2024-09-20T22:54:13.963191+0000","last_deep_scrub":"440'177","last_deep_scrub_stamp":"2024-09-20T22:48:40.602342+0000","last_clean_scrub_stamp":"2024-09-20T22:54:13.963191+0000","objects_scrubbed":13,"log_size":393,"log_dups_size":314,"ondisk_log_size":393,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:41.260738+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.004190498,"stat_sum":{"num_bytes":35876992,"num_objects":15,"num_object_clones":13,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":213,"num_read_kb":223021,"num_write":163,"num_write_kb":64725,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":18662529,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"11f","length":"5"},{"start":"125","length":"9"},{"start":"130","length":"9"},{"start":"13b","length":"3"},{"start":"13f","length":"3"},{"start":"143","length":"2"},{"start":"146","length":"2"},{"start":"14b","length":"1"},{"start":"14d","length":"1"},{"start":"150","length":"3"},{"start":"154","length":"1"},{"start":"159","length":"1"},{"start":"163","length":"1"}]},{"pgid":"4.5","version":"625'32","reported_seq":1791,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-09-20T22:54:28.313890+0000","last_change":"2024-09-20T22:53:07.867174+0000","last_active":"2024-09-20T22:54:28.313890+0000","last_peered":"2024-09-20T22:54:28.313890+0000","last_clean":"2024-09-20T22:54:28.313890+0000","last_became_active":"2024-09-20T22:48:05.243523+0000","last_became_peered":"2024-09-20T22:48:05.243523+0000","last_unstale":"2024-09-20T22:54:28.313890+0000","last_undegraded":"2024-09-20T22:54:28.313890+0000","last_fullsized":"2024-09-20T22:54:28.313890+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'32","last_scrub_stamp":"2024-09-20T22:53:07.867088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:53:07.867088+0000","objects_scrubbed":8,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:31.983588+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"67'1","reported_seq":1486,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-09-20T22:54:28.312290+0000","last_change":"2024-09-20T22:51:35.440516+0000","last_active":"2024-09-20T22:54:28.312290+0000","last_peered":"2024-09-20T22:54:28.312290+0000","last_clean":"2024-09-20T22:54:28.312290+0000","last_became_active":"2024-09-20T22:41:38.158917+0000","last_became_peered":"2024-09-20T22:41:38.158917+0000","last_unstale":"2024-09-20T22:54:28.312290+0000","last_undegraded":"2024-09-20T22:54:28.312290+0000","last_fullsized":"2024-09-20T22:54:28.312290+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-09-20T22:51:35.440416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:51:35.440416+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:31.292526+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"77'128","reported_seq":1845,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-09-20T22:54:28.321074+0000","last_change":"2024-09-20T22:54:21.616304+0000","last_active":"2024-09-20T22:54:28.321074+0000","last_peered":"2024-09-20T22:54:28.321074+0000","last_clean":"2024-09-20T22:54:28.321074+0000","last_became_active":"2024-09-20T22:47:52.206169+0000","last_became_peered":"2024-09-20T22:47:52.206169+0000","last_unstale":"2024-09-20T22:54:28.321074+0000","last_undegraded":"2024-09-20T22:54:28.321074+0000","last_fullsized":"2024-09-20T22:54:28.321074+0000","mapping_epoch":391,"log_start":"77'128","ondisk_log_start":"77'128","created":19,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:54:21.616231+0000","last_deep_scrub":"77'128","last_deep_scrub_stamp":"2024-09-20T22:47:22.417099+0000","last_clean_scrub_stamp":"2024-09-20T22:54:21.616231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:23.588140+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":1495,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-09-20T22:54:28.322259+0000","last_change":"2024-09-20T22:53:03.564792+0000","last_active":"2024-09-20T22:54:28.322259+0000","last_peered":"2024-09-20T22:54:28.322259+0000","last_clean":"2024-09-20T22:54:28.322259+0000","last_became_active":"2024-09-20T22:41:42.184387+0000","last_became_peered":"2024-09-20T22:41:42.184387+0000","last_unstale":"2024-09-20T22:54:28.322259+0000","last_undegraded":"2024-09-20T22:54:28.322259+0000","last_fullsized":"2024-09-20T22:54:28.322259+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:03.564726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:53:03.564726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:31.705537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":1496,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-09-20T22:54:28.369695+0000","last_change":"2024-09-20T22:53:09.957459+0000","last_active":"2024-09-20T22:54:28.369695+0000","last_peered":"2024-09-20T22:54:28.369695+0000","last_clean":"2024-09-20T22:54:28.369695+0000","last_became_active":"2024-09-20T22:48:05.241602+0000","last_became_peered":"2024-09-20T22:48:05.241602+0000","last_unstale":"2024-09-20T22:54:28.369695+0000","last_undegraded":"2024-09-20T22:54:28.369695+0000","last_fullsized":"2024-09-20T22:54:28.369695+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:09.957374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:53:09.957374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:30.065566+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"788'646","reported_seq":4299,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-09-20T22:54:28.313839+0000","last_change":"2024-09-20T22:54:26.319624+0000","last_active":"2024-09-20T22:54:28.313839+0000","last_peered":"2024-09-20T22:54:28.313839+0000","last_clean":"2024-09-20T22:54:28.313839+0000","last_became_active":"2024-09-20T22:54:20.414528+0000","last_became_peered":"2024-09-20T22:54:20.414528+0000","last_unstale":"2024-09-20T22:54:28.313839+0000","last_undegraded":"2024-09-20T22:54:28.313839+0000","last_fullsized":"2024-09-20T22:54:28.313839+0000","mapping_epoch":779,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":780,"parent":"0.0","parent_split_bits":0,"last_scrub":"770'632","last_scrub_stamp":"2024-09-20T22:54:13.158459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:54:13.158459+0000","objects_scrubbed":10,"log_size":646,"log_dups_size":0,"ondisk_log_size":646,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:39.662118+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00059662599999999995,"stat_sum":{"num_bytes":21620290,"num_objects":10,"num_object_clones":8,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":1,"num_read":208,"num_read_kb":213360,"num_write":115,"num_write_kb":49123,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":88,"num_bytes_recovered":51113242,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"11f","length":"5"},{"start":"125","length":"9"},{"start":"130","length":"9"},{"start":"13b","length":"3"},{"start":"13f","length":"3"},{"start":"143","length":"2"},{"start":"146","length":"2"},{"start":"14b","length":"1"},{"start":"14d","length":"1"},{"start":"150","length":"3"},{"start":"154","length":"1"},{"start":"159","length":"1"},{"start":"163","length":"1"}]},{"pgid":"4.e","version":"625'19","reported_seq":854,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-09-20T22:54:28.311143+0000","last_change":"2024-09-20T22:54:17.386152+0000","last_active":"2024-09-20T22:54:28.311143+0000","last_peered":"2024-09-20T22:54:28.311143+0000","last_clean":"2024-09-20T22:54:28.311143+0000","last_became_active":"2024-09-20T22:54:17.352687+0000","last_became_peered":"2024-09-20T22:54:17.352687+0000","last_unstale":"2024-09-20T22:54:28.311143+0000","last_undegraded":"2024-09-20T22:54:28.311143+0000","last_fullsized":"2024-09-20T22:54:28.311143+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'19","last_scrub_stamp":"2024-09-20T22:52:44.782975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:52:44.782975+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:31.370313+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":802,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-09-20T22:54:28.310160+0000","last_change":"2024-09-20T22:54:21.423843+0000","last_active":"2024-09-20T22:54:28.310160+0000","last_peered":"2024-09-20T22:54:28.310160+0000","last_clean":"2024-09-20T22:54:28.310160+0000","last_became_active":"2024-09-20T22:48:05.254688+0000","last_became_peered":"2024-09-20T22:48:05.254688+0000","last_unstale":"2024-09-20T22:54:28.310160+0000","last_undegraded":"2024-09-20T22:54:28.310160+0000","last_fullsized":"2024-09-20T22:54:28.310160+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:54:21.423766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:54:21.423766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:38.441270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":1492,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-09-20T22:54:28.369610+0000","last_change":"2024-09-20T22:51:51.002353+0000","last_active":"2024-09-20T22:54:28.369610+0000","last_peered":"2024-09-20T22:54:28.369610+0000","last_clean":"2024-09-20T22:54:28.369610+0000","last_became_active":"2024-09-20T22:48:05.233778+0000","last_became_peered":"2024-09-20T22:48:05.233778+0000","last_unstale":"2024-09-20T22:54:28.369610+0000","last_undegraded":"2024-09-20T22:54:28.369610+0000","last_fullsized":"2024-09-20T22:54:28.369610+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:51.002226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:51:51.002226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:30.065395+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":1475,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-09-20T22:54:28.311885+0000","last_change":"2024-09-20T22:53:10.647075+0000","last_active":"2024-09-20T22:54:28.311885+0000","last_peered":"2024-09-20T22:54:28.311885+0000","last_clean":"2024-09-20T22:54:28.311885+0000","last_became_active":"2024-09-20T22:41:44.832009+0000","last_became_peered":"2024-09-20T22:41:44.832009+0000","last_unstale":"2024-09-20T22:54:28.311885+0000","last_undegraded":"2024-09-20T22:54:28.311885+0000","last_fullsized":"2024-09-20T22:54:28.311885+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:10.647005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:48:40.127123+0000","last_clean_scrub_stamp":"2024-09-20T22:53:10.647005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:32.128560+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.e","version":"785'825","reported_seq":4318,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-09-20T22:54:28.369513+0000","last_change":"2024-09-20T22:54:26.366688+0000","last_active":"2024-09-20T22:54:28.369513+0000","last_peered":"2024-09-20T22:54:28.369513+0000","last_clean":"2024-09-20T22:54:28.369513+0000","last_became_active":"2024-09-20T22:47:52.209452+0000","last_became_peered":"2024-09-20T22:47:52.209452+0000","last_unstale":"2024-09-20T22:54:28.369513+0000","last_undegraded":"2024-09-20T22:54:28.369513+0000","last_fullsized":"2024-09-20T22:54:28.369513+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"778'821","last_scrub_stamp":"2024-09-20T22:54:19.513024+0000","last_deep_scrub":"174'60","last_deep_scrub_stamp":"2024-09-20T22:44:18.301776+0000","last_clean_scrub_stamp":"2024-09-20T22:54:19.513024+0000","objects_scrubbed":13,"log_size":825,"log_dups_size":0,"ondisk_log_size":825,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:30.032740+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0.00048923300000000005,"stat_sum":{"num_bytes":29724860,"num_objects":13,"num_object_clones":11,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":257,"num_read_kb":256138,"num_write":192,"num_write_kb":69325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":23110741,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"11f","length":"5"},{"start":"125","length":"9"},{"start":"130","length":"9"},{"start":"13b","length":"3"},{"start":"13f","length":"3"},{"start":"143","length":"2"},{"start":"146","length":"2"},{"start":"14b","length":"1"},{"start":"14d","length":"1"},{"start":"150","length":"3"},{"start":"154","length":"1"},{"start":"159","length":"1"},{"start":"163","length":"1"}]},{"pgid":"1.8","version":"77'128","reported_seq":1757,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-09-20T22:54:28.368705+0000","last_change":"2024-09-20T22:51:56.108038+0000","last_active":"2024-09-20T22:54:28.368705+0000","last_peered":"2024-09-20T22:54:28.368705+0000","last_clean":"2024-09-20T22:54:28.368705+0000","last_became_active":"2024-09-20T22:42:52.405594+0000","last_became_peered":"2024-09-20T22:42:52.405594+0000","last_unstale":"2024-09-20T22:54:28.368705+0000","last_undegraded":"2024-09-20T22:54:28.368705+0000","last_fullsized":"2024-09-20T22:54:28.368705+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:51:56.107944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:51:56.107944+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:30.065366+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"625'31","reported_seq":889,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-09-20T22:54:28.310928+0000","last_change":"2024-09-20T22:54:17.396085+0000","last_active":"2024-09-20T22:54:28.310928+0000","last_peered":"2024-09-20T22:54:28.310928+0000","last_clean":"2024-09-20T22:54:28.310928+0000","last_became_active":"2024-09-20T22:54:17.353293+0000","last_became_peered":"2024-09-20T22:54:17.353293+0000","last_unstale":"2024-09-20T22:54:28.310928+0000","last_undegraded":"2024-09-20T22:54:28.310928+0000","last_fullsized":"2024-09-20T22:54:28.310928+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'31","last_scrub_stamp":"2024-09-20T22:52:13.620754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:52:13.620754+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:31.370263+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":49,"num_read_kb":39,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":1056,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-09-20T22:54:28.368859+0000","last_change":"2024-09-20T22:54:17.332953+0000","last_active":"2024-09-20T22:54:28.368859+0000","last_peered":"2024-09-20T22:54:28.368859+0000","last_clean":"2024-09-20T22:54:28.368859+0000","last_became_active":"2024-09-20T22:54:17.332636+0000","last_became_peered":"2024-09-20T22:54:17.332636+0000","last_unstale":"2024-09-20T22:54:28.368859+0000","last_undegraded":"2024-09-20T22:54:28.368859+0000","last_fullsized":"2024-09-20T22:54:28.368859+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:08.105010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:35.492562+0000","last_clean_scrub_stamp":"2024-09-20T22:53:08.105010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:29.538072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":33,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-09-20T22:54:28.363010+0000","last_change":"2024-09-20T22:54:17.349436+0000","last_active":"2024-09-20T22:54:28.363010+0000","last_peered":"2024-09-20T22:54:28.363010+0000","last_clean":"2024-09-20T22:54:28.363010+0000","last_became_active":"2024-09-20T22:54:17.349144+0000","last_became_peered":"2024-09-20T22:54:17.349144+0000","last_unstale":"2024-09-20T22:54:28.363010+0000","last_undegraded":"2024-09-20T22:54:28.363010+0000","last_fullsized":"2024-09-20T22:54:28.363010+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:57.442770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:53:57.442770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:58.266258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":1511,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-09-20T22:54:28.369451+0000","last_change":"2024-09-20T22:53:05.106323+0000","last_active":"2024-09-20T22:54:28.369451+0000","last_peered":"2024-09-20T22:54:28.369451+0000","last_clean":"2024-09-20T22:54:28.369451+0000","last_became_active":"2024-09-20T22:48:05.239006+0000","last_became_peered":"2024-09-20T22:48:05.239006+0000","last_unstale":"2024-09-20T22:54:28.369451+0000","last_undegraded":"2024-09-20T22:54:28.369451+0000","last_fullsized":"2024-09-20T22:54:28.369451+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:05.106224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:53:05.106224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:29.538025+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"785'889","reported_seq":4368,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-09-20T22:54:28.311169+0000","last_change":"2024-09-20T22:54:26.315373+0000","last_active":"2024-09-20T22:54:28.311169+0000","last_peered":"2024-09-20T22:54:28.311169+0000","last_clean":"2024-09-20T22:54:28.311169+0000","last_became_active":"2024-09-20T22:47:52.216198+0000","last_became_peered":"2024-09-20T22:47:52.216198+0000","last_unstale":"2024-09-20T22:54:28.311169+0000","last_undegraded":"2024-09-20T22:54:28.311169+0000","last_fullsized":"2024-09-20T22:54:28.311169+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"754'838","last_scrub_stamp":"2024-09-20T22:53:54.908987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:53:54.908987+0000","objects_scrubbed":17,"log_size":889,"log_dups_size":0,"ondisk_log_size":889,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:06.641904+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.00050714499999999997,"stat_sum":{"num_bytes":32417533,"num_objects":15,"num_object_clones":12,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":351,"num_read_kb":346402,"num_write":217,"num_write_kb":73285,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":53,"num_bytes_recovered":14983556,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"11f","length":"5"},{"start":"125","length":"9"},{"start":"130","length":"9"},{"start":"13b","length":"3"},{"start":"13f","length":"3"},{"start":"143","length":"2"},{"start":"146","length":"2"},{"start":"14b","length":"1"},{"start":"14d","length":"1"},{"start":"150","length":"3"},{"start":"154","length":"1"},{"start":"159","length":"1"},{"start":"163","length":"1"}]},{"pgid":"1.9","version":"77'128","reported_seq":1813,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-09-20T22:54:28.321166+0000","last_change":"2024-09-20T22:53:50.381120+0000","last_active":"2024-09-20T22:54:28.321166+0000","last_peered":"2024-09-20T22:54:28.321166+0000","last_clean":"2024-09-20T22:54:28.321166+0000","last_became_active":"2024-09-20T22:50:42.262862+0000","last_became_peered":"2024-09-20T22:50:42.262862+0000","last_unstale":"2024-09-20T22:54:28.321166+0000","last_undegraded":"2024-09-20T22:54:28.321166+0000","last_fullsized":"2024-09-20T22:54:28.321166+0000","mapping_epoch":561,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":562,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:53:50.381052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:53:50.381052+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:53.001645+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"625'20","reported_seq":821,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-09-20T22:54:28.310932+0000","last_change":"2024-09-20T22:52:22.646547+0000","last_active":"2024-09-20T22:54:28.310932+0000","last_peered":"2024-09-20T22:54:28.310932+0000","last_clean":"2024-09-20T22:54:28.310932+0000","last_became_active":"2024-09-20T22:48:05.242811+0000","last_became_peered":"2024-09-20T22:48:05.242811+0000","last_unstale":"2024-09-20T22:54:28.310932+0000","last_undegraded":"2024-09-20T22:54:28.310932+0000","last_fullsized":"2024-09-20T22:54:28.310932+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'20","last_scrub_stamp":"2024-09-20T22:52:22.646468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:52:22.646468+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:31.370330+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":1511,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-09-20T22:54:28.311090+0000","last_change":"2024-09-20T22:52:44.527004+0000","last_active":"2024-09-20T22:54:28.311090+0000","last_peered":"2024-09-20T22:54:28.311090+0000","last_clean":"2024-09-20T22:54:28.311090+0000","last_became_active":"2024-09-20T22:48:05.233999+0000","last_became_peered":"2024-09-20T22:48:05.233999+0000","last_unstale":"2024-09-20T22:54:28.311090+0000","last_undegraded":"2024-09-20T22:54:28.311090+0000","last_fullsized":"2024-09-20T22:54:28.311090+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:44.526935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:52:44.526935+0000","last_clean_scrub_stamp":"2024-09-20T22:52:44.526935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:31.292585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"261'9","reported_seq":1667,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-09-20T22:54:28.369368+0000","last_change":"2024-09-20T22:53:54.190905+0000","last_active":"2024-09-20T22:54:28.369368+0000","last_peered":"2024-09-20T22:54:28.369368+0000","last_clean":"2024-09-20T22:54:28.369368+0000","last_became_active":"2024-09-20T22:41:42.185863+0000","last_became_peered":"2024-09-20T22:41:42.185863+0000","last_unstale":"2024-09-20T22:54:28.369368+0000","last_undegraded":"2024-09-20T22:54:28.369368+0000","last_fullsized":"2024-09-20T22:54:28.369368+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'9","last_scrub_stamp":"2024-09-20T22:53:54.190823+0000","last_deep_scrub":"261'9","last_deep_scrub_stamp":"2024-09-20T22:46:50.974735+0000","last_clean_scrub_stamp":"2024-09-20T22:53:54.190823+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:22.929855+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":33,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-09-20T22:54:28.362779+0000","last_change":"2024-09-20T22:54:17.341373+0000","last_active":"2024-09-20T22:54:28.362779+0000","last_peered":"2024-09-20T22:54:28.362779+0000","last_clean":"2024-09-20T22:54:28.362779+0000","last_became_active":"2024-09-20T22:54:17.340684+0000","last_became_peered":"2024-09-20T22:54:17.340684+0000","last_unstale":"2024-09-20T22:54:28.362779+0000","last_undegraded":"2024-09-20T22:54:28.362779+0000","last_fullsized":"2024-09-20T22:54:28.362779+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:10.650428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:53:10.650428+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.8","version":"786'1032","reported_seq":4349,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-09-20T22:54:28.368799+0000","last_change":"2024-09-20T22:54:26.369621+0000","last_active":"2024-09-20T22:54:28.368799+0000","last_peered":"2024-09-20T22:54:28.368799+0000","last_clean":"2024-09-20T22:54:28.368799+0000","last_became_active":"2024-09-20T22:48:31.224359+0000","last_became_peered":"2024-09-20T22:48:31.224359+0000","last_unstale":"2024-09-20T22:54:28.368799+0000","last_undegraded":"2024-09-20T22:54:28.368799+0000","last_fullsized":"2024-09-20T22:54:28.368799+0000","mapping_epoch":407,"log_start":"426'268","ondisk_log_start":"426'268","created":94,"last_epoch_clean":431,"parent":"0.0","parent_split_bits":0,"last_scrub":"774'1005","last_scrub_stamp":"2024-09-20T22:54:15.466690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:54:15.466690+0000","objects_scrubbed":14,"log_size":764,"log_dups_size":445,"ondisk_log_size":764,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:19.624480+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.0039980550000000004,"stat_sum":{"num_bytes":40371665,"num_objects":17,"num_object_clones":14,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":320,"num_read_kb":331226,"num_write":258,"num_write_kb":83987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":26027409,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"11f","length":"5"},{"start":"125","length":"9"},{"start":"130","length":"9"},{"start":"13b","length":"3"},{"start":"13f","length":"3"},{"start":"143","length":"2"},{"start":"146","length":"2"},{"start":"14b","length":"1"},{"start":"14d","length":"1"},{"start":"150","length":"3"},{"start":"154","length":"1"},{"start":"159","length":"1"},{"start":"163","length":"1"}]},{"pgid":"4.b","version":"625'15","reported_seq":902,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-09-20T22:54:28.310178+0000","last_change":"2024-09-20T22:54:17.376019+0000","last_active":"2024-09-20T22:54:28.310178+0000","last_peered":"2024-09-20T22:54:28.310178+0000","last_clean":"2024-09-20T22:54:28.310178+0000","last_became_active":"2024-09-20T22:54:17.340998+0000","last_became_peered":"2024-09-20T22:54:17.340998+0000","last_unstale":"2024-09-20T22:54:28.310178+0000","last_undegraded":"2024-09-20T22:54:28.310178+0000","last_fullsized":"2024-09-20T22:54:28.310178+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'15","last_scrub_stamp":"2024-09-20T22:53:30.976566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:53:30.976566+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:52.443242+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":1496,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-09-20T22:54:28.369405+0000","last_change":"2024-09-20T22:53:55.019427+0000","last_active":"2024-09-20T22:54:28.369405+0000","last_peered":"2024-09-20T22:54:28.369405+0000","last_clean":"2024-09-20T22:54:28.369405+0000","last_became_active":"2024-09-20T22:41:38.155953+0000","last_became_peered":"2024-09-20T22:41:38.155953+0000","last_unstale":"2024-09-20T22:54:28.369405+0000","last_undegraded":"2024-09-20T22:54:28.369405+0000","last_fullsized":"2024-09-20T22:54:28.369405+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:55.019362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:53:55.019362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:17.821354+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":1519,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-09-20T22:54:28.369130+0000","last_change":"2024-09-20T22:52:42.998852+0000","last_active":"2024-09-20T22:54:28.369130+0000","last_peered":"2024-09-20T22:54:28.369130+0000","last_clean":"2024-09-20T22:54:28.369130+0000","last_became_active":"2024-09-20T22:48:05.240613+0000","last_became_peered":"2024-09-20T22:48:05.240613+0000","last_unstale":"2024-09-20T22:54:28.369130+0000","last_undegraded":"2024-09-20T22:54:28.369130+0000","last_fullsized":"2024-09-20T22:54:28.369130+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:42.998756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:52:42.998756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:29.538041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":828,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-09-20T22:54:28.311565+0000","last_change":"2024-09-20T22:52:16.706497+0000","last_active":"2024-09-20T22:54:28.311565+0000","last_peered":"2024-09-20T22:54:28.311565+0000","last_clean":"2024-09-20T22:54:28.311565+0000","last_became_active":"2024-09-20T22:47:49.219486+0000","last_became_peered":"2024-09-20T22:47:49.219486+0000","last_unstale":"2024-09-20T22:54:28.311565+0000","last_undegraded":"2024-09-20T22:54:28.311565+0000","last_fullsized":"2024-09-20T22:54:28.311565+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:16.706429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:52:16.706429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:29.935817+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"788'1369","reported_seq":4951,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-09-20T22:54:28.531268+0000","last_change":"2024-09-20T22:54:26.313871+0000","last_active":"2024-09-20T22:54:28.531268+0000","last_peered":"2024-09-20T22:54:28.531268+0000","last_clean":"2024-09-20T22:54:28.531268+0000","last_became_active":"2024-09-20T22:54:20.414945+0000","last_became_peered":"2024-09-20T22:54:20.414945+0000","last_unstale":"2024-09-20T22:54:28.531268+0000","last_undegraded":"2024-09-20T22:54:28.531268+0000","last_fullsized":"2024-09-20T22:54:28.531268+0000","mapping_epoch":779,"log_start":"423'280","ondisk_log_start":"423'280","created":94,"last_epoch_clean":780,"parent":"0.0","parent_split_bits":0,"last_scrub":"774'1348","last_scrub_stamp":"2024-09-20T22:54:15.010110+0000","last_deep_scrub":"590'828","last_deep_scrub_stamp":"2024-09-20T22:51:10.909364+0000","last_clean_scrub_stamp":"2024-09-20T22:54:15.010110+0000","objects_scrubbed":19,"log_size":1089,"log_dups_size":649,"ondisk_log_size":1089,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:41.490762+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.001132163,"stat_sum":{"num_bytes":54441327,"num_objects":21,"num_object_clones":16,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":1,"num_read":487,"num_read_kb":492811,"num_write":351,"num_write_kb":138324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":109,"num_bytes_recovered":133262200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"11f","length":"5"},{"start":"125","length":"9"},{"start":"130","length":"9"},{"start":"13b","length":"3"},{"start":"13f","length":"3"},{"start":"143","length":"2"},{"start":"146","length":"2"},{"start":"14b","length":"1"},{"start":"14d","length":"1"},{"start":"150","length":"3"},{"start":"154","length":"1"},{"start":"159","length":"1"},{"start":"163","length":"1"}]},{"pgid":"4.a","version":"625'44","reported_seq":1623,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-09-20T22:54:28.313216+0000","last_change":"2024-09-20T22:54:17.368002+0000","last_active":"2024-09-20T22:54:28.313216+0000","last_peered":"2024-09-20T22:54:28.313216+0000","last_clean":"2024-09-20T22:54:28.313216+0000","last_became_active":"2024-09-20T22:54:17.341124+0000","last_became_peered":"2024-09-20T22:54:17.341124+0000","last_unstale":"2024-09-20T22:54:28.313216+0000","last_undegraded":"2024-09-20T22:54:28.313216+0000","last_fullsized":"2024-09-20T22:54:28.313216+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'44","last_scrub_stamp":"2024-09-20T22:53:50.910441+0000","last_deep_scrub":"113'28","last_deep_scrub_stamp":"2024-09-20T22:43:12.725900+0000","last_clean_scrub_stamp":"2024-09-20T22:53:50.910441+0000","objects_scrubbed":9,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:01.136304+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":1494,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-09-20T22:54:28.322059+0000","last_change":"2024-09-20T22:53:27.585334+0000","last_active":"2024-09-20T22:54:28.322059+0000","last_peered":"2024-09-20T22:54:28.322059+0000","last_clean":"2024-09-20T22:54:28.322059+0000","last_became_active":"2024-09-20T22:41:38.161524+0000","last_became_peered":"2024-09-20T22:41:38.161524+0000","last_unstale":"2024-09-20T22:54:28.322059+0000","last_undegraded":"2024-09-20T22:54:28.322059+0000","last_fullsized":"2024-09-20T22:54:28.322059+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:27.585262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:53:27.585262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:52.375797+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":1515,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-09-20T22:54:28.369091+0000","last_change":"2024-09-20T22:54:21.496172+0000","last_active":"2024-09-20T22:54:28.369091+0000","last_peered":"2024-09-20T22:54:28.369091+0000","last_clean":"2024-09-20T22:54:28.369091+0000","last_became_active":"2024-09-20T22:47:49.223220+0000","last_became_peered":"2024-09-20T22:47:49.223220+0000","last_unstale":"2024-09-20T22:54:28.369091+0000","last_undegraded":"2024-09-20T22:54:28.369091+0000","last_fullsized":"2024-09-20T22:54:28.369091+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:54:21.496068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:54:21.496068+0000","last_clean_scrub_stamp":"2024-09-20T22:54:21.496068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:23.538970+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":1490,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-09-20T22:54:28.320978+0000","last_change":"2024-09-20T22:53:53.365864+0000","last_active":"2024-09-20T22:54:28.320978+0000","last_peered":"2024-09-20T22:54:28.320978+0000","last_clean":"2024-09-20T22:54:28.320978+0000","last_became_active":"2024-09-20T22:41:44.199395+0000","last_became_peered":"2024-09-20T22:41:44.199395+0000","last_unstale":"2024-09-20T22:54:28.320978+0000","last_undegraded":"2024-09-20T22:54:28.320978+0000","last_fullsized":"2024-09-20T22:54:28.320978+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:53.365789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:47:18.329209+0000","last_clean_scrub_stamp":"2024-09-20T22:53:53.365789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:06.119698+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.a","version":"788'967","reported_seq":4357,"reported_epoch":788,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T22:54:28.367265+0000","last_change":"2024-09-20T22:54:26.364297+0000","last_active":"2024-09-20T22:54:28.367265+0000","last_peered":"2024-09-20T22:54:28.367265+0000","last_clean":"2024-09-20T22:54:28.367265+0000","last_became_active":"2024-09-20T22:42:52.405085+0000","last_became_peered":"2024-09-20T22:42:52.405085+0000","last_unstale":"2024-09-20T22:54:28.367265+0000","last_undegraded":"2024-09-20T22:54:28.367265+0000","last_fullsized":"2024-09-20T22:54:28.367265+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"505'466","last_scrub_stamp":"2024-09-20T22:49:47.088165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:49:47.088165+0000","objects_scrubbed":13,"log_size":967,"log_dups_size":0,"ondisk_log_size":967,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.047879103999999999,"stat_sum":{"num_bytes":42648159,"num_objects":18,"num_object_clones":15,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":377,"num_read_kb":392856,"num_write":290,"num_write_kb":119216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"11f","length":"5"},{"start":"125","length":"9"},{"start":"130","length":"9"},{"start":"13b","length":"3"},{"start":"13f","length":"3"},{"start":"143","length":"2"},{"start":"146","length":"2"},{"start":"14b","length":"1"},{"start":"14d","length":"1"},{"start":"150","length":"1"},{"start":"152","length":"1"}]},{"pgid":"4.9","version":"686'23","reported_seq":874,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-09-20T22:54:28.310279+0000","last_change":"2024-09-20T22:54:17.392658+0000","last_active":"2024-09-20T22:54:28.310279+0000","last_peered":"2024-09-20T22:54:28.310279+0000","last_clean":"2024-09-20T22:54:28.310279+0000","last_became_active":"2024-09-20T22:54:17.352939+0000","last_became_peered":"2024-09-20T22:54:17.352939+0000","last_unstale":"2024-09-20T22:54:28.310279+0000","last_undegraded":"2024-09-20T22:54:28.310279+0000","last_fullsized":"2024-09-20T22:54:28.310279+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"686'23","last_scrub_stamp":"2024-09-20T22:52:46.768812+0000","last_deep_scrub":"686'23","last_deep_scrub_stamp":"2024-09-20T22:52:46.768812+0000","last_clean_scrub_stamp":"2024-09-20T22:52:46.768812+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:31.370378+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":21,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":1100,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":1538,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-09-20T22:54:28.321097+0000","last_change":"2024-09-20T22:54:17.333114+0000","last_active":"2024-09-20T22:54:28.321097+0000","last_peered":"2024-09-20T22:54:28.321097+0000","last_clean":"2024-09-20T22:54:28.321097+0000","last_became_active":"2024-09-20T22:54:17.332759+0000","last_became_peered":"2024-09-20T22:54:17.332759+0000","last_unstale":"2024-09-20T22:54:28.321097+0000","last_undegraded":"2024-09-20T22:54:28.321097+0000","last_fullsized":"2024-09-20T22:54:28.321097+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:35.478053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:46:52.465940+0000","last_clean_scrub_stamp":"2024-09-20T22:53:35.478053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:51.053621+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":1527,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-09-20T22:54:28.368957+0000","last_change":"2024-09-20T22:54:17.340588+0000","last_active":"2024-09-20T22:54:28.368957+0000","last_peered":"2024-09-20T22:54:28.368957+0000","last_clean":"2024-09-20T22:54:28.368957+0000","last_became_active":"2024-09-20T22:54:17.340269+0000","last_became_peered":"2024-09-20T22:54:17.340269+0000","last_unstale":"2024-09-20T22:54:28.368957+0000","last_undegraded":"2024-09-20T22:54:28.368957+0000","last_fullsized":"2024-09-20T22:54:28.368957+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:07.104891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:53:07.104891+0000","last_clean_scrub_stamp":"2024-09-20T22:53:07.104891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:29.538058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":1487,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-09-20T22:54:28.310402+0000","last_change":"2024-09-20T22:54:17.331117+0000","last_active":"2024-09-20T22:54:28.310402+0000","last_peered":"2024-09-20T22:54:28.310402+0000","last_clean":"2024-09-20T22:54:28.310402+0000","last_became_active":"2024-09-20T22:54:17.329277+0000","last_became_peered":"2024-09-20T22:54:17.329277+0000","last_unstale":"2024-09-20T22:54:28.310402+0000","last_undegraded":"2024-09-20T22:54:28.310402+0000","last_fullsized":"2024-09-20T22:54:28.310402+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:53.242204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:55.170379+0000","last_clean_scrub_stamp":"2024-09-20T22:53:53.242204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:01.634636+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"788'1248","reported_seq":4797,"reported_epoch":788,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T22:54:28.444457+0000","last_change":"2024-09-20T22:54:28.442667+0000","last_active":"2024-09-20T22:54:28.444457+0000","last_peered":"2024-09-20T22:54:28.444457+0000","last_clean":"2024-09-20T22:54:28.444457+0000","last_became_active":"2024-09-20T22:54:20.415706+0000","last_became_peered":"2024-09-20T22:54:20.415706+0000","last_unstale":"2024-09-20T22:54:28.444457+0000","last_undegraded":"2024-09-20T22:54:28.444457+0000","last_fullsized":"2024-09-20T22:54:28.444457+0000","mapping_epoch":779,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":780,"parent":"0.0","parent_split_bits":0,"last_scrub":"661'1000","last_scrub_stamp":"2024-09-20T22:52:23.164028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:52:23.164028+0000","objects_scrubbed":23,"log_size":1248,"log_dups_size":0,"ondisk_log_size":1248,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":5,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.079634404000000006,"stat_sum":{"num_bytes":31457199,"num_objects":15,"num_object_clones":11,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":3,"num_read":527,"num_read_kb":596029,"num_write":362,"num_write_kb":125532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":168,"num_bytes_recovered":68651320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"11f","length":"5"},{"start":"125","length":"9"},{"start":"130","length":"9"},{"start":"13b","length":"3"},{"start":"13f","length":"3"},{"start":"143","length":"2"},{"start":"146","length":"2"},{"start":"14b","length":"1"},{"start":"14d","length":"1"},{"start":"150","length":"1"}]},{"pgid":"4.8","version":"625'27","reported_seq":1527,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-09-20T22:54:28.368895+0000","last_change":"2024-09-20T22:51:53.020721+0000","last_active":"2024-09-20T22:54:28.368895+0000","last_peered":"2024-09-20T22:54:28.368895+0000","last_clean":"2024-09-20T22:54:28.368895+0000","last_became_active":"2024-09-20T22:41:40.252113+0000","last_became_peered":"2024-09-20T22:41:40.252113+0000","last_unstale":"2024-09-20T22:54:28.368895+0000","last_undegraded":"2024-09-20T22:54:28.368895+0000","last_fullsized":"2024-09-20T22:54:28.368895+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'27","last_scrub_stamp":"2024-09-20T22:51:53.020628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:51:53.020628+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:30.065298+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'2","reported_seq":1557,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-09-20T22:54:28.322118+0000","last_change":"2024-09-20T22:53:33.526131+0000","last_active":"2024-09-20T22:54:28.322118+0000","last_peered":"2024-09-20T22:54:28.322118+0000","last_clean":"2024-09-20T22:54:28.322118+0000","last_became_active":"2024-09-20T22:48:05.238098+0000","last_became_peered":"2024-09-20T22:48:05.238098+0000","last_unstale":"2024-09-20T22:54:28.322118+0000","last_undegraded":"2024-09-20T22:54:28.322118+0000","last_fullsized":"2024-09-20T22:54:28.322118+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-09-20T22:53:33.526049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:53:33.526049+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:58.400288+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"408'10","reported_seq":1677,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-09-20T22:54:28.321509+0000","last_change":"2024-09-20T22:53:24.628363+0000","last_active":"2024-09-20T22:54:28.321509+0000","last_peered":"2024-09-20T22:54:28.321509+0000","last_clean":"2024-09-20T22:54:28.321509+0000","last_became_active":"2024-09-20T22:48:05.239344+0000","last_became_peered":"2024-09-20T22:48:05.239344+0000","last_unstale":"2024-09-20T22:54:28.321509+0000","last_undegraded":"2024-09-20T22:54:28.321509+0000","last_fullsized":"2024-09-20T22:54:28.321509+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"408'10","last_scrub_stamp":"2024-09-20T22:53:24.628269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:53:24.628269+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:33.393650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":1488,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-09-20T22:54:28.369105+0000","last_change":"2024-09-20T22:51:54.066995+0000","last_active":"2024-09-20T22:54:28.369105+0000","last_peered":"2024-09-20T22:54:28.369105+0000","last_clean":"2024-09-20T22:54:28.369105+0000","last_became_active":"2024-09-20T22:47:49.227569+0000","last_became_peered":"2024-09-20T22:47:49.227569+0000","last_unstale":"2024-09-20T22:54:28.369105+0000","last_undegraded":"2024-09-20T22:54:28.369105+0000","last_fullsized":"2024-09-20T22:54:28.369105+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:54.066913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:51:54.066913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:30.065331+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":1523,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-09-20T22:54:28.313083+0000","last_change":"2024-09-20T22:53:11.873951+0000","last_active":"2024-09-20T22:54:28.313083+0000","last_peered":"2024-09-20T22:54:28.313083+0000","last_clean":"2024-09-20T22:54:28.313083+0000","last_became_active":"2024-09-20T22:47:49.213292+0000","last_became_peered":"2024-09-20T22:47:49.213292+0000","last_unstale":"2024-09-20T22:54:28.313083+0000","last_undegraded":"2024-09-20T22:54:28.313083+0000","last_fullsized":"2024-09-20T22:54:28.313083+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:11.873813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:53:11.873813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:31.983568+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"625'12","reported_seq":1537,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-09-20T22:54:28.311646+0000","last_change":"2024-09-20T22:54:17.387599+0000","last_active":"2024-09-20T22:54:28.311646+0000","last_peered":"2024-09-20T22:54:28.311646+0000","last_clean":"2024-09-20T22:54:28.311646+0000","last_became_active":"2024-09-20T22:54:17.342098+0000","last_became_peered":"2024-09-20T22:54:17.342098+0000","last_unstale":"2024-09-20T22:54:28.311646+0000","last_undegraded":"2024-09-20T22:54:28.311646+0000","last_fullsized":"2024-09-20T22:54:28.311646+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'12","last_scrub_stamp":"2024-09-20T22:53:26.319144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:53:26.319144+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:48.496082+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":1500,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-09-20T22:54:28.369581+0000","last_change":"2024-09-20T22:54:17.340916+0000","last_active":"2024-09-20T22:54:28.369581+0000","last_peered":"2024-09-20T22:54:28.369581+0000","last_clean":"2024-09-20T22:54:28.369581+0000","last_became_active":"2024-09-20T22:54:17.339912+0000","last_became_peered":"2024-09-20T22:54:17.339912+0000","last_unstale":"2024-09-20T22:54:28.369581+0000","last_undegraded":"2024-09-20T22:54:28.369581+0000","last_fullsized":"2024-09-20T22:54:28.369581+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:03.888040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:09.705632+0000","last_clean_scrub_stamp":"2024-09-20T22:53:03.888040+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:30.065521+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"74'1","reported_seq":1503,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-09-20T22:54:28.321453+0000","last_change":"2024-09-20T22:52:49.575819+0000","last_active":"2024-09-20T22:54:28.321453+0000","last_peered":"2024-09-20T22:54:28.321453+0000","last_clean":"2024-09-20T22:54:28.321453+0000","last_became_active":"2024-09-20T22:48:05.237882+0000","last_became_peered":"2024-09-20T22:48:05.237882+0000","last_unstale":"2024-09-20T22:54:28.321453+0000","last_undegraded":"2024-09-20T22:54:28.321453+0000","last_fullsized":"2024-09-20T22:54:28.321453+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-09-20T22:52:49.575702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:52:49.575702+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:28.667449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"625'15","reported_seq":854,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-09-20T22:54:28.312312+0000","last_change":"2024-09-20T22:53:50.889818+0000","last_active":"2024-09-20T22:54:28.312312+0000","last_peered":"2024-09-20T22:54:28.312312+0000","last_clean":"2024-09-20T22:54:28.312312+0000","last_became_active":"2024-09-20T22:47:49.231315+0000","last_became_peered":"2024-09-20T22:47:49.231315+0000","last_unstale":"2024-09-20T22:54:28.312312+0000","last_undegraded":"2024-09-20T22:54:28.312312+0000","last_fullsized":"2024-09-20T22:54:28.312312+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'15","last_scrub_stamp":"2024-09-20T22:53:50.889717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:53:50.889717+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:51.999731+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":1518,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-09-20T22:54:28.320657+0000","last_change":"2024-09-20T22:53:10.778908+0000","last_active":"2024-09-20T22:54:28.320657+0000","last_peered":"2024-09-20T22:54:28.320657+0000","last_clean":"2024-09-20T22:54:28.320657+0000","last_became_active":"2024-09-20T22:48:05.238801+0000","last_became_peered":"2024-09-20T22:48:05.238801+0000","last_unstale":"2024-09-20T22:54:28.320657+0000","last_undegraded":"2024-09-20T22:54:28.320657+0000","last_fullsized":"2024-09-20T22:54:28.320657+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:10.778836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:53:10.778836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:31.705519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":1491,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-09-20T22:54:28.311128+0000","last_change":"2024-09-20T22:54:17.342665+0000","last_active":"2024-09-20T22:54:28.311128+0000","last_peered":"2024-09-20T22:54:28.311128+0000","last_clean":"2024-09-20T22:54:28.311128+0000","last_became_active":"2024-09-20T22:54:17.341617+0000","last_became_peered":"2024-09-20T22:54:17.341617+0000","last_unstale":"2024-09-20T22:54:28.311128+0000","last_undegraded":"2024-09-20T22:54:28.311128+0000","last_fullsized":"2024-09-20T22:54:28.311128+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:20.410164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:52:20.410164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:31.292571+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":1523,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-09-20T22:54:28.368953+0000","last_change":"2024-09-20T22:53:58.169978+0000","last_active":"2024-09-20T22:54:28.368953+0000","last_peered":"2024-09-20T22:54:28.368953+0000","last_clean":"2024-09-20T22:54:28.368953+0000","last_became_active":"2024-09-20T22:48:05.239710+0000","last_became_peered":"2024-09-20T22:48:05.239710+0000","last_unstale":"2024-09-20T22:54:28.368953+0000","last_undegraded":"2024-09-20T22:54:28.368953+0000","last_fullsized":"2024-09-20T22:54:28.368953+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:58.169894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:53:58.169894+0000","last_clean_scrub_stamp":"2024-09-20T22:53:58.169894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:04.779145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"625'15","reported_seq":1517,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-09-20T22:54:28.369597+0000","last_change":"2024-09-20T22:53:53.070301+0000","last_active":"2024-09-20T22:54:28.369597+0000","last_peered":"2024-09-20T22:54:28.369597+0000","last_clean":"2024-09-20T22:54:28.369597+0000","last_became_active":"2024-09-20T22:41:40.252486+0000","last_became_peered":"2024-09-20T22:41:40.252486+0000","last_unstale":"2024-09-20T22:54:28.369597+0000","last_undegraded":"2024-09-20T22:54:28.369597+0000","last_fullsized":"2024-09-20T22:54:28.369597+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'15","last_scrub_stamp":"2024-09-20T22:53:53.070214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:53:53.070214+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:11.603607+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":843,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-09-20T22:54:28.310726+0000","last_change":"2024-09-20T22:53:58.829885+0000","last_active":"2024-09-20T22:54:28.310726+0000","last_peered":"2024-09-20T22:54:28.310726+0000","last_clean":"2024-09-20T22:54:28.310726+0000","last_became_active":"2024-09-20T22:47:49.229936+0000","last_became_peered":"2024-09-20T22:47:49.229936+0000","last_unstale":"2024-09-20T22:54:28.310726+0000","last_undegraded":"2024-09-20T22:54:28.310726+0000","last_fullsized":"2024-09-20T22:54:28.310726+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:58.829817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:53:58.829817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:21.096752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"74'8","reported_seq":1645,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-09-20T22:54:28.311770+0000","last_change":"2024-09-20T22:53:59.165210+0000","last_active":"2024-09-20T22:54:28.311770+0000","last_peered":"2024-09-20T22:54:28.311770+0000","last_clean":"2024-09-20T22:54:28.311770+0000","last_became_active":"2024-09-20T22:41:42.186352+0000","last_became_peered":"2024-09-20T22:41:42.186352+0000","last_unstale":"2024-09-20T22:54:28.311770+0000","last_undegraded":"2024-09-20T22:54:28.311770+0000","last_fullsized":"2024-09-20T22:54:28.311770+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-09-20T22:53:59.165134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:53:59.165134+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:12.296267+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":794,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-09-20T22:54:28.311239+0000","last_change":"2024-09-20T22:52:16.623463+0000","last_active":"2024-09-20T22:54:28.311239+0000","last_peered":"2024-09-20T22:54:28.311239+0000","last_clean":"2024-09-20T22:54:28.311239+0000","last_became_active":"2024-09-20T22:48:05.249656+0000","last_became_peered":"2024-09-20T22:48:05.249656+0000","last_unstale":"2024-09-20T22:54:28.311239+0000","last_undegraded":"2024-09-20T22:54:28.311239+0000","last_fullsized":"2024-09-20T22:54:28.311239+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:16.623368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:52:16.623368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:31.370245+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"625'20","reported_seq":1557,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-09-20T22:54:28.311298+0000","last_change":"2024-09-20T22:54:17.390484+0000","last_active":"2024-09-20T22:54:28.311298+0000","last_peered":"2024-09-20T22:54:28.311298+0000","last_clean":"2024-09-20T22:54:28.311298+0000","last_became_active":"2024-09-20T22:54:17.343066+0000","last_became_peered":"2024-09-20T22:54:17.343066+0000","last_unstale":"2024-09-20T22:54:28.311298+0000","last_undegraded":"2024-09-20T22:54:28.311298+0000","last_fullsized":"2024-09-20T22:54:28.311298+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'20","last_scrub_stamp":"2024-09-20T22:54:14.200839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:54:14.200839+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:15.294970+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":1514,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-09-20T22:54:28.321203+0000","last_change":"2024-09-20T22:53:56.396136+0000","last_active":"2024-09-20T22:54:28.321203+0000","last_peered":"2024-09-20T22:54:28.321203+0000","last_clean":"2024-09-20T22:54:28.321203+0000","last_became_active":"2024-09-20T22:48:05.240231+0000","last_became_peered":"2024-09-20T22:48:05.240231+0000","last_unstale":"2024-09-20T22:54:28.321203+0000","last_undegraded":"2024-09-20T22:54:28.321203+0000","last_fullsized":"2024-09-20T22:54:28.321203+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:56.396065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:53:56.396065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:04.372880+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"778'12","reported_seq":1695,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-09-20T22:54:28.369432+0000","last_change":"2024-09-20T22:54:17.379970+0000","last_active":"2024-09-20T22:54:28.369432+0000","last_peered":"2024-09-20T22:54:28.369432+0000","last_clean":"2024-09-20T22:54:28.369432+0000","last_became_active":"2024-09-20T22:54:17.341014+0000","last_became_peered":"2024-09-20T22:54:17.341014+0000","last_unstale":"2024-09-20T22:54:28.369432+0000","last_undegraded":"2024-09-20T22:54:28.369432+0000","last_fullsized":"2024-09-20T22:54:28.369432+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"600'11","last_scrub_stamp":"2024-09-20T22:52:36.264410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:52:36.264410+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:30.065458+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":827,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-09-20T22:54:28.310179+0000","last_change":"2024-09-20T22:52:44.834418+0000","last_active":"2024-09-20T22:54:28.310179+0000","last_peered":"2024-09-20T22:54:28.310179+0000","last_clean":"2024-09-20T22:54:28.310179+0000","last_became_active":"2024-09-20T22:47:49.230674+0000","last_became_peered":"2024-09-20T22:47:49.230674+0000","last_unstale":"2024-09-20T22:54:28.310179+0000","last_undegraded":"2024-09-20T22:54:28.310179+0000","last_fullsized":"2024-09-20T22:54:28.310179+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:44.834352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:50:21.853070+0000","last_clean_scrub_stamp":"2024-09-20T22:52:44.834352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:29.935946+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"625'19","reported_seq":862,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-09-20T22:54:28.310736+0000","last_change":"2024-09-20T22:54:17.385330+0000","last_active":"2024-09-20T22:54:28.310736+0000","last_peered":"2024-09-20T22:54:28.310736+0000","last_clean":"2024-09-20T22:54:28.310736+0000","last_became_active":"2024-09-20T22:54:17.351129+0000","last_became_peered":"2024-09-20T22:54:17.351129+0000","last_unstale":"2024-09-20T22:54:28.310736+0000","last_undegraded":"2024-09-20T22:54:28.310736+0000","last_fullsized":"2024-09-20T22:54:28.310736+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'19","last_scrub_stamp":"2024-09-20T22:53:49.547607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:53:49.547607+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:50.519930+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":802,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-09-20T22:54:28.310667+0000","last_change":"2024-09-20T22:52:47.717598+0000","last_active":"2024-09-20T22:54:28.310667+0000","last_peered":"2024-09-20T22:54:28.310667+0000","last_clean":"2024-09-20T22:54:28.310667+0000","last_became_active":"2024-09-20T22:48:05.252026+0000","last_became_peered":"2024-09-20T22:48:05.252026+0000","last_unstale":"2024-09-20T22:54:28.310667+0000","last_undegraded":"2024-09-20T22:54:28.310667+0000","last_fullsized":"2024-09-20T22:54:28.310667+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:47.717490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:52:47.717490+0000","last_clean_scrub_stamp":"2024-09-20T22:52:47.717490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:31.370346+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":33,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-09-20T22:54:28.362443+0000","last_change":"2024-09-20T22:54:17.337178+0000","last_active":"2024-09-20T22:54:28.362443+0000","last_peered":"2024-09-20T22:54:28.362443+0000","last_clean":"2024-09-20T22:54:28.362443+0000","last_became_active":"2024-09-20T22:54:17.336853+0000","last_became_peered":"2024-09-20T22:54:17.336853+0000","last_unstale":"2024-09-20T22:54:28.362443+0000","last_undegraded":"2024-09-20T22:54:28.362443+0000","last_fullsized":"2024-09-20T22:54:28.362443+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:56.000202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:53:56.000202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:03.068366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":1487,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-09-20T22:54:28.310454+0000","last_change":"2024-09-20T22:52:42.582474+0000","last_active":"2024-09-20T22:54:28.310454+0000","last_peered":"2024-09-20T22:54:28.310454+0000","last_clean":"2024-09-20T22:54:28.310454+0000","last_became_active":"2024-09-20T22:47:49.216141+0000","last_became_peered":"2024-09-20T22:47:49.216141+0000","last_unstale":"2024-09-20T22:54:28.310454+0000","last_undegraded":"2024-09-20T22:54:28.310454+0000","last_fullsized":"2024-09-20T22:54:28.310454+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:42.582398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:50:20.206984+0000","last_clean_scrub_stamp":"2024-09-20T22:52:42.582398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:31.292613+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"625'19","reported_seq":43,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-09-20T22:54:28.362554+0000","last_change":"2024-09-20T22:54:17.360066+0000","last_active":"2024-09-20T22:54:28.362554+0000","last_peered":"2024-09-20T22:54:28.362554+0000","last_clean":"2024-09-20T22:54:28.362554+0000","last_became_active":"2024-09-20T22:54:17.340539+0000","last_became_peered":"2024-09-20T22:54:17.340539+0000","last_unstale":"2024-09-20T22:54:28.362554+0000","last_undegraded":"2024-09-20T22:54:28.362554+0000","last_fullsized":"2024-09-20T22:54:28.362554+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'19","last_scrub_stamp":"2024-09-20T22:53:58.933385+0000","last_deep_scrub":"404'13","last_deep_scrub_stamp":"2024-09-20T22:48:04.276282+0000","last_clean_scrub_stamp":"2024-09-20T22:53:58.933385+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:27.507376+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":1521,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-09-20T22:54:28.321945+0000","last_change":"2024-09-20T22:52:42.505280+0000","last_active":"2024-09-20T22:54:28.321945+0000","last_peered":"2024-09-20T22:54:28.321945+0000","last_clean":"2024-09-20T22:54:28.321945+0000","last_became_active":"2024-09-20T22:48:05.238812+0000","last_became_peered":"2024-09-20T22:48:05.238812+0000","last_unstale":"2024-09-20T22:54:28.321945+0000","last_undegraded":"2024-09-20T22:54:28.321945+0000","last_fullsized":"2024-09-20T22:54:28.321945+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:42.505211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:31.138067+0000","last_clean_scrub_stamp":"2024-09-20T22:52:42.505211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:28.667350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":1503,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-09-20T22:54:28.310486+0000","last_change":"2024-09-20T22:54:17.337400+0000","last_active":"2024-09-20T22:54:28.310486+0000","last_peered":"2024-09-20T22:54:28.310486+0000","last_clean":"2024-09-20T22:54:28.310486+0000","last_became_active":"2024-09-20T22:54:17.337102+0000","last_became_peered":"2024-09-20T22:54:17.337102+0000","last_unstale":"2024-09-20T22:54:28.310486+0000","last_undegraded":"2024-09-20T22:54:28.310486+0000","last_fullsized":"2024-09-20T22:54:28.310486+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:34.345077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:53:34.345077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:03.047569+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":839,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-09-20T22:54:28.309844+0000","last_change":"2024-09-20T22:54:17.344363+0000","last_active":"2024-09-20T22:54:28.309844+0000","last_peered":"2024-09-20T22:54:28.309844+0000","last_clean":"2024-09-20T22:54:28.309844+0000","last_became_active":"2024-09-20T22:54:17.344021+0000","last_became_peered":"2024-09-20T22:54:17.344021+0000","last_unstale":"2024-09-20T22:54:28.309844+0000","last_undegraded":"2024-09-20T22:54:28.309844+0000","last_fullsized":"2024-09-20T22:54:28.309844+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:09.890501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:53:09.890501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:29.936034+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"625'24","reported_seq":1318,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-09-20T22:54:28.368771+0000","last_change":"2024-09-20T22:53:33.089688+0000","last_active":"2024-09-20T22:54:28.368771+0000","last_peered":"2024-09-20T22:54:28.368771+0000","last_clean":"2024-09-20T22:54:28.368771+0000","last_became_active":"2024-09-20T22:44:09.170785+0000","last_became_peered":"2024-09-20T22:44:09.170785+0000","last_unstale":"2024-09-20T22:54:28.368771+0000","last_undegraded":"2024-09-20T22:54:28.368771+0000","last_fullsized":"2024-09-20T22:54:28.368771+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'24","last_scrub_stamp":"2024-09-20T22:53:33.089580+0000","last_deep_scrub":"470'16","last_deep_scrub_stamp":"2024-09-20T22:49:10.659868+0000","last_clean_scrub_stamp":"2024-09-20T22:53:33.089580+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:46.655555+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":1508,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-09-20T22:54:28.369663+0000","last_change":"2024-09-20T22:54:17.341390+0000","last_active":"2024-09-20T22:54:28.369663+0000","last_peered":"2024-09-20T22:54:28.369663+0000","last_clean":"2024-09-20T22:54:28.369663+0000","last_became_active":"2024-09-20T22:54:17.341069+0000","last_became_peered":"2024-09-20T22:54:17.341069+0000","last_unstale":"2024-09-20T22:54:28.369663+0000","last_undegraded":"2024-09-20T22:54:28.369663+0000","last_fullsized":"2024-09-20T22:54:28.369663+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:07.945664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:45:24.765414+0000","last_clean_scrub_stamp":"2024-09-20T22:53:07.945664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:30.065619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":1510,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-09-20T22:54:28.320527+0000","last_change":"2024-09-20T22:53:09.740214+0000","last_active":"2024-09-20T22:54:28.320527+0000","last_peered":"2024-09-20T22:54:28.320527+0000","last_clean":"2024-09-20T22:54:28.320527+0000","last_became_active":"2024-09-20T22:48:05.238334+0000","last_became_peered":"2024-09-20T22:48:05.238334+0000","last_unstale":"2024-09-20T22:54:28.320527+0000","last_undegraded":"2024-09-20T22:54:28.320527+0000","last_fullsized":"2024-09-20T22:54:28.320527+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:09.740145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:50.174861+0000","last_clean_scrub_stamp":"2024-09-20T22:53:09.740145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:33.910113+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":1487,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-09-20T22:54:28.311005+0000","last_change":"2024-09-20T22:54:22.276389+0000","last_active":"2024-09-20T22:54:28.311005+0000","last_peered":"2024-09-20T22:54:28.311005+0000","last_clean":"2024-09-20T22:54:28.311005+0000","last_became_active":"2024-09-20T22:47:49.214487+0000","last_became_peered":"2024-09-20T22:47:49.214487+0000","last_unstale":"2024-09-20T22:54:28.311005+0000","last_undegraded":"2024-09-20T22:54:28.311005+0000","last_fullsized":"2024-09-20T22:54:28.311005+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:54:22.276321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:54:22.276321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:27.586502+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"625'8","reported_seq":1519,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-09-20T22:54:28.311076+0000","last_change":"2024-09-20T22:54:17.373557+0000","last_active":"2024-09-20T22:54:28.311076+0000","last_peered":"2024-09-20T22:54:28.311076+0000","last_clean":"2024-09-20T22:54:28.311076+0000","last_became_active":"2024-09-20T22:54:17.341776+0000","last_became_peered":"2024-09-20T22:54:17.341776+0000","last_unstale":"2024-09-20T22:54:28.311076+0000","last_undegraded":"2024-09-20T22:54:28.311076+0000","last_fullsized":"2024-09-20T22:54:28.311076+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'8","last_scrub_stamp":"2024-09-20T22:53:05.489279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:53:05.489279+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:31.292629+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":835,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-09-20T22:54:28.310785+0000","last_change":"2024-09-20T22:53:07.918178+0000","last_active":"2024-09-20T22:54:28.310785+0000","last_peered":"2024-09-20T22:54:28.310785+0000","last_clean":"2024-09-20T22:54:28.310785+0000","last_became_active":"2024-09-20T22:47:49.231964+0000","last_became_peered":"2024-09-20T22:47:49.231964+0000","last_unstale":"2024-09-20T22:54:28.310785+0000","last_undegraded":"2024-09-20T22:54:28.310785+0000","last_fullsized":"2024-09-20T22:54:28.310785+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:07.918108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:53:07.918108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:29.936060+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":1527,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-09-20T22:54:28.313104+0000","last_change":"2024-09-20T22:53:51.937667+0000","last_active":"2024-09-20T22:54:28.313104+0000","last_peered":"2024-09-20T22:54:28.313104+0000","last_clean":"2024-09-20T22:54:28.313104+0000","last_became_active":"2024-09-20T22:48:05.244659+0000","last_became_peered":"2024-09-20T22:48:05.244659+0000","last_unstale":"2024-09-20T22:54:28.313104+0000","last_undegraded":"2024-09-20T22:54:28.313104+0000","last_fullsized":"2024-09-20T22:54:28.313104+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:51.937584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:53:51.937584+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:07.795334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":1502,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-09-20T22:54:28.320794+0000","last_change":"2024-09-20T22:53:05.612142+0000","last_active":"2024-09-20T22:54:28.320794+0000","last_peered":"2024-09-20T22:54:28.320794+0000","last_clean":"2024-09-20T22:54:28.320794+0000","last_became_active":"2024-09-20T22:48:05.237765+0000","last_became_peered":"2024-09-20T22:48:05.237765+0000","last_unstale":"2024-09-20T22:54:28.320794+0000","last_undegraded":"2024-09-20T22:54:28.320794+0000","last_fullsized":"2024-09-20T22:54:28.320794+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:05.612030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:10.233487+0000","last_clean_scrub_stamp":"2024-09-20T22:53:05.612030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:28.667476+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":33,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-09-20T22:54:28.363150+0000","last_change":"2024-09-20T22:54:17.348523+0000","last_active":"2024-09-20T22:54:28.363150+0000","last_peered":"2024-09-20T22:54:28.363150+0000","last_clean":"2024-09-20T22:54:28.363150+0000","last_became_active":"2024-09-20T22:54:17.348206+0000","last_became_peered":"2024-09-20T22:54:17.348206+0000","last_unstale":"2024-09-20T22:54:28.363150+0000","last_undegraded":"2024-09-20T22:54:28.363150+0000","last_fullsized":"2024-09-20T22:54:28.363150+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:58.003274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:53:58.003274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:07.487217+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":1525,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-09-20T22:54:28.310806+0000","last_change":"2024-09-20T22:54:17.342220+0000","last_active":"2024-09-20T22:54:28.310806+0000","last_peered":"2024-09-20T22:54:28.310806+0000","last_clean":"2024-09-20T22:54:28.310806+0000","last_became_active":"2024-09-20T22:54:17.341405+0000","last_became_peered":"2024-09-20T22:54:17.341405+0000","last_unstale":"2024-09-20T22:54:28.310806+0000","last_undegraded":"2024-09-20T22:54:28.310806+0000","last_fullsized":"2024-09-20T22:54:28.310806+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:31.274390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:53:31.274390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:59.504104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"625'19","reported_seq":1578,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-09-20T22:54:28.314055+0000","last_change":"2024-09-20T22:54:17.370392+0000","last_active":"2024-09-20T22:54:28.314055+0000","last_peered":"2024-09-20T22:54:28.314055+0000","last_clean":"2024-09-20T22:54:28.314055+0000","last_became_active":"2024-09-20T22:54:17.342236+0000","last_became_peered":"2024-09-20T22:54:17.342236+0000","last_unstale":"2024-09-20T22:54:28.314055+0000","last_undegraded":"2024-09-20T22:54:28.314055+0000","last_fullsized":"2024-09-20T22:54:28.314055+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'19","last_scrub_stamp":"2024-09-20T22:53:30.953067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:53:30.953067+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:33.510823+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":828,"reported_epoch":788,"state":"active+clean","last_fresh":"2024-09-20T22:54:28.311449+0000","last_change":"2024-09-20T22:52:18.698154+0000","last_active":"2024-09-20T22:54:28.311449+0000","last_peered":"2024-09-20T22:54:28.311449+0000","last_clean":"2024-09-20T22:54:28.311449+0000","last_became_active":"2024-09-20T22:47:49.220971+0000","last_became_peered":"2024-09-20T22:47:49.220971+0000","last_unstale":"2024-09-20T22:54:28.311449+0000","last_undegraded":"2024-09-20T22:54:28.311449+0000","last_fullsized":"2024-09-20T22:54:28.311449+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:18.698087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:51.667341+0000","last_clean_scrub_stamp":"2024-09-20T22:52:18.698087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:54:29.935879+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":578606199,"num_objects":251,"num_object_clones":203,"num_object_copies":753,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":15,"num_objects_unfound":0,"num_objects_dirty":251,"num_whiteouts":9,"num_read":5400,"num_read_kb":5621833,"num_write":3945,"num_write_kb":1482258,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":974,"num_bytes_recovered":692971958,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1289981952,"data_stored":1775055119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":12363,"ondisk_log_size":12363,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":90,"ondisk_log_size":90,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1106,"num_read_kb":863,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":418,"num_bytes_recovered":9802,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":745,"ondisk_log_size":745,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":1126,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":759,"num_read_kb":754,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":1594,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":384,"ondisk_log_size":384,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070622,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":221920,"kb_used_data":168516,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93523232,"statfs":{"total":95995035648,"available":95767789568,"internally_reserved":0,"allocated":172560384,"data_stored":238735538,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":54655434},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430372,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":204732,"kb_used_data":151328,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93540420,"statfs":{"total":95995035648,"available":95785390080,"internally_reserved":0,"allocated":154959872,"data_stored":202164563,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798692009,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":165016,"kb_used_data":119940,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":93580136,"statfs":{"total":95995035648,"available":95826059264,"internally_reserved":0,"allocated":122818560,"data_stored":172985603,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986350,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":137024,"kb_used_data":101668,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93608128,"statfs":{"total":95995035648,"available":95854723072,"internally_reserved":0,"allocated":104108032,"data_stored":146677015,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182580,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":341588,"kb_used_data":261816,"kb_used_omap":1,"kb_used_meta":79742,"kb_avail":93403564,"statfs":{"total":95995035648,"available":95645249536,"internally_reserved":0,"allocated":268099584,"data_stored":380345550,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":81656263},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411513,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":181344,"kb_used_data":132100,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":93563808,"statfs":{"total":95995035648,"available":95809339392,"internally_reserved":0,"allocated":135270400,"data_stored":197752701,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575038,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":215568,"kb_used_data":170488,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":93529584,"statfs":{"total":95995035648,"available":95774294016,"internally_reserved":0,"allocated":174579712,"data_stored":180506327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738563,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":239404,"kb_used_data":186000,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93505748,"statfs":{"total":95995035648,"available":95749885952,"internally_reserved":0,"allocated":190464000,"data_stored":258922209,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":192507904,"data_stored":266014360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":171835392,"data_stored":181873789,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":127574016,"data_stored":192943608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":267698176,"data_stored":385000908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":97603584,"data_stored":144265549,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":114810880,"data_stored":167862832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":153165824,"data_stored":203232914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":164786176,"data_stored":233861159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-20T22:54:30.463 INFO:tasks.thrashosds.thrasher:Setting 1.0 to [0, 7, 4, 5, 2, 6] 2024-09-20T22:54:30.463 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '1.0', '0', '7', '4', '5', '2', '6'] 2024-09-20T22:54:30.463 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd pg-upmap-items 1.0 0 7 4 5 2 6 2024-09-20T22:54:30.474 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:30.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:30.475 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:54:30.576 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:54:30.597 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:30.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:30.598 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:54:30.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:30 smithi084 ceph-mon[29774]: osdmap e789: 8 total, 8 up, 8 in 2024-09-20T22:54:30.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:30 smithi084 ceph-mon[29774]: from='client.17025 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T22:54:30.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:30 smithi084 ceph-mon[29774]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:54:30.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:30 smithi084 ceph-mon[40024]: osdmap e789: 8 total, 8 up, 8 in 2024-09-20T22:54:30.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:30 smithi084 ceph-mon[40024]: from='client.17025 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T22:54:30.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:30 smithi084 ceph-mon[40024]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:54:30.663 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-20T22:54:30.664 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:54:30.699 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:54:30.720 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:30.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:30.721 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:54:30.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:30 smithi154 ceph-mon[33264]: osdmap e789: 8 total, 8 up, 8 in 2024-09-20T22:54:30.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:30 smithi154 ceph-mon[33264]: from='client.17025 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T22:54:30.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:30 smithi154 ceph-mon[33264]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:54:30.822 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:54:30.842 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:30.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:30.843 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:54:30.944 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:54:30.970 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:30.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:30.971 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:54:31.059 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-20T22:54:31.071 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:54:31.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:31.097 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:31.097 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:54:31.198 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:54:31.223 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:31.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:31.224 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:54:31.325 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:54:31.349 INFO:tasks.rados.rados.0.smithi084.stdout:3149: done (5 left) 2024-09-20T22:54:31.350 INFO:tasks.rados.rados.0.smithi084.stdout:3150: delete oid 31 current snap is 360 2024-09-20T22:54:31.350 INFO:tasks.rados.rados.0.smithi084.stdout:3145: expect (ObjNum 1053 snap 345 seq_num 1053) 2024-09-20T22:54:31.352 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:31.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:31.354 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:54:31.454 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:54:31.477 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:31.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:31.478 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:54:31.478 INFO:tasks.rados.rados.0.smithi084.stdout:3144: expect (ObjNum 738 snap 235 seq_num 738) 2024-09-20T22:54:31.559 INFO:tasks.rados.rados.0.smithi084.stdout:3148: expect (ObjNum 985 snap 323 seq_num 985) 2024-09-20T22:54:31.579 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:54:31.604 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:31.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:31.605 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:54:31.614 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:31 smithi084 ceph-mon[29774]: purged_snaps scrub starts 2024-09-20T22:54:31.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:31 smithi084 ceph-mon[29774]: purged_snaps scrub ok 2024-09-20T22:54:31.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:31 smithi084 ceph-mon[29774]: pgmap v1086: 158 pgs: 3 active+clean+snaptrim, 1 active+remapped+backfill_toofull, 154 active+clean; 552 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 1.3 MiB/s rd, 4.2 MiB/s wr, 14 op/s; 15/1350 objects misplaced (1.111%) 2024-09-20T22:54:31.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:31 smithi084 ceph-mon[29774]: osdmap e790: 8 total, 8 up, 8 in 2024-09-20T22:54:31.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:31 smithi084 ceph-mon[40024]: purged_snaps scrub starts 2024-09-20T22:54:31.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:31 smithi084 ceph-mon[40024]: purged_snaps scrub ok 2024-09-20T22:54:31.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:31 smithi084 ceph-mon[40024]: pgmap v1086: 158 pgs: 3 active+clean+snaptrim, 1 active+remapped+backfill_toofull, 154 active+clean; 552 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 1.3 MiB/s rd, 4.2 MiB/s wr, 14 op/s; 15/1350 objects misplaced (1.111%) 2024-09-20T22:54:31.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:31 smithi084 ceph-mon[40024]: osdmap e790: 8 total, 8 up, 8 in 2024-09-20T22:54:31.677 INFO:tasks.rados.rados.0.smithi084.stdout:3147: finishing rollback tid 1 to smithi08477203-10 2024-09-20T22:54:31.677 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 10 v 1826 (ObjNum 129 snap 0 seq_num 2091922320) dirty dne 2024-09-20T22:54:31.677 INFO:tasks.rados.rados.0.smithi084.stdout:3146: finishing copy_from racing read to smithi08477203-39 2024-09-20T22:54:31.677 INFO:tasks.rados.rados.0.smithi084.stdout:3146: finishing copy_from to smithi08477203-39 2024-09-20T22:54:31.677 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 39 v 2453 (ObjNum 1051 snap 345 seq_num 1051) dirty exists 2024-09-20T22:54:31.677 INFO:tasks.rados.rados.0.smithi084.stdout:3144: done (5 left) 2024-09-20T22:54:31.677 INFO:tasks.rados.rados.0.smithi084.stdout:3145: done (4 left) 2024-09-20T22:54:31.677 INFO:tasks.rados.rados.0.smithi084.stdout:3146: done (3 left) 2024-09-20T22:54:31.678 INFO:tasks.rados.rados.0.smithi084.stdout:3147: done (2 left) 2024-09-20T22:54:31.678 INFO:tasks.rados.rados.0.smithi084.stdout:3148: done (1 left) 2024-09-20T22:54:31.678 INFO:tasks.rados.rados.0.smithi084.stdout:3150: done (0 left) 2024-09-20T22:54:31.678 INFO:tasks.rados.rados.0.smithi084.stdout:3151: read oid 3 snap 312 2024-09-20T22:54:31.678 INFO:tasks.rados.rados.0.smithi084.stdout:3151: expect deleted 2024-09-20T22:54:31.678 INFO:tasks.rados.rados.0.smithi084.stdout:3152: copy_from oid 24 from oid 2 current snap is 360 2024-09-20T22:54:31.678 INFO:tasks.rados.rados.0.smithi084.stdout:3153: rollback oid 21 current snap is 360 2024-09-20T22:54:31.678 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 21 to 326 2024-09-20T22:54:31.678 INFO:tasks.rados.rados.0.smithi084.stdout:3154: copy_from oid 18 from oid 38 current snap is 360 2024-09-20T22:54:31.678 INFO:tasks.rados.rados.0.smithi084.stdout:3155: rollback oid 29 current snap is 360 2024-09-20T22:54:31.678 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 29 to 344 2024-09-20T22:54:31.678 INFO:tasks.rados.rados.0.smithi084.stdout:3156: rollback oid 25 current snap is 360 2024-09-20T22:54:31.678 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 25 to 357 2024-09-20T22:54:31.679 INFO:tasks.rados.rados.0.smithi084.stdout:3157: delete oid 40 current snap is 360 2024-09-20T22:54:31.680 INFO:tasks.rados.rados.0.smithi084.stdout:3153: finishing rollback tid 0 to smithi08477203-21 2024-09-20T22:54:31.680 INFO:tasks.rados.rados.0.smithi084.stdout:3153: finishing rollback tid 1 to smithi08477203-21 2024-09-20T22:54:31.681 INFO:tasks.rados.rados.0.smithi084.stdout:3153: finishing rollback tid 2 to smithi08477203-21 2024-09-20T22:54:31.681 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 21 v 1842 (ObjNum 984 snap 319 seq_num 984) dirty exists 2024-09-20T22:54:31.683 INFO:tasks.rados.rados.0.smithi084.stdout:3151: done (6 left) 2024-09-20T22:54:31.683 INFO:tasks.rados.rados.0.smithi084.stdout:3153: done (5 left) 2024-09-20T22:54:31.683 INFO:tasks.rados.rados.0.smithi084.stdout:3157: done (4 left) 2024-09-20T22:54:31.683 INFO:tasks.rados.rados.0.smithi084.stdout:3158: copy_from oid 27 from oid 22 current snap is 360 2024-09-20T22:54:31.684 INFO:tasks.rados.rados.0.smithi084.stdout:3159: snap_create 2024-09-20T22:54:31.684 INFO:tasks.rados.rados.0.smithi084.stdout:3155: finishing rollback tid 1 to smithi08477203-29 2024-09-20T22:54:31.684 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 29 v 2342 (ObjNum 289 snap 0 seq_num 0) dirty dne 2024-09-20T22:54:31.684 INFO:tasks.rados.rados.0.smithi084.stdout:3156: finishing rollback tid 0 to smithi08477203-25 2024-09-20T22:54:31.685 INFO:tasks.rados.rados.0.smithi084.stdout:3156: finishing rollback tid 1 to smithi08477203-25 2024-09-20T22:54:31.685 INFO:tasks.rados.rados.0.smithi084.stdout:3156: finishing rollback tid 2 to smithi08477203-25 2024-09-20T22:54:31.685 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 25 v 2542 (ObjNum 1070 snap 355 seq_num 1070) dirty exists 2024-09-20T22:54:31.685 INFO:tasks.rados.rados.0.smithi084.stdout:3158: finishing copy_from to smithi08477203-27 2024-09-20T22:54:31.685 INFO:tasks.rados.rados.0.smithi084.stdout:3158: got expected ENOENT (src dne) 2024-09-20T22:54:31.685 INFO:tasks.rados.rados.0.smithi084.stdout:3158: finishing copy_from racing read to smithi08477203-27 2024-09-20T22:54:31.688 INFO:tasks.rados.rados.0.smithi084.stdout:3154: finishing copy_from racing read to smithi08477203-18 2024-09-20T22:54:31.693 INFO:tasks.rados.rados.0.smithi084.stdout:3152: finishing copy_from racing read to smithi08477203-24 2024-09-20T22:54:31.702 INFO:tasks.rados.rados.0.smithi084.stdout:3154: finishing copy_from to smithi08477203-18 2024-09-20T22:54:31.702 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 18 v 1830 (ObjNum 1057 snap 347 seq_num 1057) dirty exists 2024-09-20T22:54:31.703 INFO:tasks.rados.rados.0.smithi084.stdout:3152: finishing copy_from to smithi08477203-24 2024-09-20T22:54:31.703 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 24 v 1835 (ObjNum 918 snap 302 seq_num 918) dirty exists 2024-09-20T22:54:31.706 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:54:31.727 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:31.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:31.728 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:54:31.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:31 smithi154 ceph-mon[33264]: purged_snaps scrub starts 2024-09-20T22:54:31.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:31 smithi154 ceph-mon[33264]: purged_snaps scrub ok 2024-09-20T22:54:31.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:31 smithi154 ceph-mon[33264]: pgmap v1086: 158 pgs: 3 active+clean+snaptrim, 1 active+remapped+backfill_toofull, 154 active+clean; 552 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 1.3 MiB/s rd, 4.2 MiB/s wr, 14 op/s; 15/1350 objects misplaced (1.111%) 2024-09-20T22:54:31.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:31 smithi154 ceph-mon[33264]: osdmap e790: 8 total, 8 up, 8 in 2024-09-20T22:54:31.826 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:54:31.828 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:54:31.841 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.1 dump_blocked_ops 2024-09-20T22:54:31.851 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:31.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:31.852 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:54:31.953 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:54:31.978 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:31.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:31.979 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:54:32.080 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:54:32.104 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:32.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:32.105 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:54:32.206 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:54:32.228 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:32.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:32.229 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:54:32.309 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:54:32.309 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:54:32.315 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:54:32.326 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:54:32.326 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:54:32.330 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:54:32.345 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:54:32.345 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:54:32.351 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:54:32.354 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:32.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:32.355 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:54:32.360 INFO:tasks.rados.rados.0.smithi084.stdout:3152: done (5 left) 2024-09-20T22:54:32.360 INFO:tasks.rados.rados.0.smithi084.stdout:3154: done (4 left) 2024-09-20T22:54:32.360 INFO:tasks.rados.rados.0.smithi084.stdout:3155: done (3 left) 2024-09-20T22:54:32.360 INFO:tasks.rados.rados.0.smithi084.stdout:3156: done (2 left) 2024-09-20T22:54:32.360 INFO:tasks.rados.rados.0.smithi084.stdout:3158: done (1 left) 2024-09-20T22:54:32.360 INFO:tasks.rados.rados.0.smithi084.stdout:3159: done (0 left) 2024-09-20T22:54:32.360 INFO:tasks.rados.rados.0.smithi084.stdout:3160: copy_from oid 36 from oid 27 current snap is 361 2024-09-20T22:54:32.360 INFO:tasks.rados.rados.0.smithi084.stdout:3161: read oid 7 snap -1 2024-09-20T22:54:32.360 INFO:tasks.rados.rados.0.smithi084.stdout:3161: expect (ObjNum 964 snap 314 seq_num 964) 2024-09-20T22:54:32.360 INFO:tasks.rados.rados.0.smithi084.stdout:3162: copy_from oid 32 from oid 41 current snap is 361 2024-09-20T22:54:32.360 INFO:tasks.rados.rados.0.smithi084.stdout:3163: snap_remove snap 360 2024-09-20T22:54:32.361 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is set 2024-09-20T22:54:32.373 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-20T22:54:32.373 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T22:54:32.391 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:54:32.391 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:54:32.408 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:54:32.409 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:54:32.425 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:54:32.426 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:54:32.442 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:54:32.442 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:54:32.456 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:54:32.479 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:32.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:32.480 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:54:32.581 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:54:32.604 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:32.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:32.605 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:54:32.610 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:32 smithi084 ceph-mon[29774]: osdmap e791: 8 total, 8 up, 8 in 2024-09-20T22:54:32.610 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:32 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/618375752' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:54:32.610 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:32 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:54:32.610 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:32 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/3603386224' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:54:32.610 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:32 smithi084 ceph-mon[40024]: osdmap e791: 8 total, 8 up, 8 in 2024-09-20T22:54:32.611 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:32 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/618375752' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:54:32.611 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:32 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:54:32.611 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:32 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/3603386224' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:54:32.706 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:54:32.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:32.729 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:32.729 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:54:32.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:32 smithi154 ceph-mon[33264]: osdmap e791: 8 total, 8 up, 8 in 2024-09-20T22:54:32.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:32 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/618375752' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:54:32.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:32 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:54:32.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:32 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/3603386224' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:54:32.830 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:54:32.853 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:32.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:32.854 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:54:32.955 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:54:32.979 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:32.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:32.981 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:54:33.081 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:54:33.102 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:33.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:33.103 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:54:33.204 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:54:33.229 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:33.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:33.230 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:54:33.331 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:54:33.355 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:33.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:33.356 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:54:33.364 INFO:tasks.rados.rados.0.smithi084.stdout:3163: done (3 left) 2024-09-20T22:54:33.365 INFO:tasks.rados.rados.0.smithi084.stdout:3164: rollback oid 11 current snap is 361 2024-09-20T22:54:33.365 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 11 to 328 2024-09-20T22:54:33.365 INFO:tasks.rados.rados.0.smithi084.stdout:3161: expect (ObjNum 964 snap 314 seq_num 964) 2024-09-20T22:54:33.458 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:54:33.481 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:33.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:33.482 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:54:33.488 INFO:tasks.rados.rados.0.smithi084.stdout:3162: finishing copy_from to smithi08477203-32 2024-09-20T22:54:33.489 INFO:tasks.rados.rados.0.smithi084.stdout:3162: got expected ENOENT (src dne) 2024-09-20T22:54:33.489 INFO:tasks.rados.rados.0.smithi084.stdout:3162: finishing copy_from racing read to smithi08477203-32 2024-09-20T22:54:33.489 INFO:tasks.rados.rados.0.smithi084.stdout:3160: finishing copy_from racing read to smithi08477203-36 2024-09-20T22:54:33.489 INFO:tasks.rados.rados.0.smithi084.stdout:3160: finishing copy_from to smithi08477203-36 2024-09-20T22:54:33.489 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 36 v 1813 (ObjNum 1077 snap 358 seq_num 1077) dirty exists 2024-09-20T22:54:33.489 INFO:tasks.rados.rados.0.smithi084.stdout:3160: done (3 left) 2024-09-20T22:54:33.489 INFO:tasks.rados.rados.0.smithi084.stdout:3161: done (2 left) 2024-09-20T22:54:33.489 INFO:tasks.rados.rados.0.smithi084.stdout:3162: done (1 left) 2024-09-20T22:54:33.489 INFO:tasks.rados.rados.0.smithi084.stdout:3165: snap_remove snap 330 2024-09-20T22:54:33.583 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:54:33.606 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:33.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:33.607 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:54:33.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:33 smithi084 ceph-mon[29774]: pgmap v1089: 158 pgs: 3 active+clean+snaptrim, 1 active+remapped+backfill_toofull, 154 active+clean; 552 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 1.3 MiB/s rd, 4.2 MiB/s wr, 14 op/s; 15/1350 objects misplaced (1.111%) 2024-09-20T22:54:33.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:33 smithi084 ceph-mon[29774]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:54:33.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:33 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T22:54:33.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:33 smithi084 ceph-mon[29774]: osdmap e792: 8 total, 8 up, 8 in 2024-09-20T22:54:33.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:33 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/1222419926' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "1.0", "id": [0, 7, 4, 5, 2, 6]}]: dispatch 2024-09-20T22:54:33.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:33 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "1.0", "id": [0, 7, 4, 5, 2, 6]}]: dispatch 2024-09-20T22:54:33.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:33 smithi084 ceph-mon[40024]: pgmap v1089: 158 pgs: 3 active+clean+snaptrim, 1 active+remapped+backfill_toofull, 154 active+clean; 552 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 1.3 MiB/s rd, 4.2 MiB/s wr, 14 op/s; 15/1350 objects misplaced (1.111%) 2024-09-20T22:54:33.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:33 smithi084 ceph-mon[40024]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:54:33.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:33 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T22:54:33.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:33 smithi084 ceph-mon[40024]: osdmap e792: 8 total, 8 up, 8 in 2024-09-20T22:54:33.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:33 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/1222419926' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "1.0", "id": [0, 7, 4, 5, 2, 6]}]: dispatch 2024-09-20T22:54:33.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:33 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "1.0", "id": [0, 7, 4, 5, 2, 6]}]: dispatch 2024-09-20T22:54:33.707 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:54:33.729 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:33.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:33.730 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:54:33.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:33 smithi154 ceph-mon[33264]: pgmap v1089: 158 pgs: 3 active+clean+snaptrim, 1 active+remapped+backfill_toofull, 154 active+clean; 552 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 1.3 MiB/s rd, 4.2 MiB/s wr, 14 op/s; 15/1350 objects misplaced (1.111%) 2024-09-20T22:54:33.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:33 smithi154 ceph-mon[33264]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:54:33.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:33 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T22:54:33.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:33 smithi154 ceph-mon[33264]: osdmap e792: 8 total, 8 up, 8 in 2024-09-20T22:54:33.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:33 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/1222419926' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "1.0", "id": [0, 7, 4, 5, 2, 6]}]: dispatch 2024-09-20T22:54:33.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:33 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "1.0", "id": [0, 7, 4, 5, 2, 6]}]: dispatch 2024-09-20T22:54:33.831 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:54:33.854 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:33.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:33.855 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:54:33.955 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:54:33.976 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:33.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:33.977 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:54:34.078 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:54:34.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:34.103 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:34.103 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:54:34.204 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:54:34.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:34.229 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:34.229 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:54:34.308 INFO:teuthology.orchestra.run.smithi084.stderr:set 1.0 pg_upmap_items mapping to [0->7,4->5,2->6] 2024-09-20T22:54:34.330 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:54:34.351 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:34.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:34.352 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:54:34.373 INFO:tasks.rados.rados.0.smithi084.stdout:3165: done (1 left) 2024-09-20T22:54:34.374 INFO:tasks.rados.rados.0.smithi084.stdout:3166: snap_create 2024-09-20T22:54:34.374 INFO:tasks.rados.rados.0.smithi084.stdout:3164: finishing rollback tid 0 to smithi08477203-11 2024-09-20T22:54:34.374 INFO:tasks.rados.rados.0.smithi084.stdout:3164: finishing rollback tid 1 to smithi08477203-11 2024-09-20T22:54:34.374 INFO:tasks.rados.rados.0.smithi084.stdout:3164: finishing rollback tid 2 to smithi08477203-11 2024-09-20T22:54:34.374 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 11 v 1036 (ObjNum 991 snap 325 seq_num 991) dirty exists 2024-09-20T22:54:34.453 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:54:34.476 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:34.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:34.477 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:54:34.578 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:54:34.599 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:34.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:34.600 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:54:34.614 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:34 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "1.0", "id": [0, 7, 4, 5, 2, 6]}]': finished 2024-09-20T22:54:34.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:34 smithi084 ceph-mon[40024]: osdmap e793: 8 total, 8 up, 8 in 2024-09-20T22:54:34.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:34 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "1.0", "id": [0, 7, 4, 5, 2, 6]}]': finished 2024-09-20T22:54:34.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:34 smithi084 ceph-mon[29774]: osdmap e793: 8 total, 8 up, 8 in 2024-09-20T22:54:34.701 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:54:34.722 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:34.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:34.723 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:54:34.735 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-20T22:54:34.735 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2024-09-20T22:54:34.736 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2024-09-20T22:54:34.736 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2024-09-20T22:54:34.736 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-20T22:54:34.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:34 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "1.0", "id": [0, 7, 4, 5, 2, 6]}]': finished 2024-09-20T22:54:34.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:34 smithi154 ceph-mon[33264]: osdmap e793: 8 total, 8 up, 8 in 2024-09-20T22:54:34.823 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:54:34.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:34.846 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:34.846 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:54:34.946 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:54:34.972 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:34.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:34.973 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:54:35.074 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:54:35.100 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:35.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:35.101 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:54:35.202 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:54:35.227 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:35.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:35.228 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:54:35.329 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:54:35.351 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:35.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:35.352 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:54:35.370 INFO:tasks.rados.rados.0.smithi084.stdout:3164: done (1 left) 2024-09-20T22:54:35.370 INFO:tasks.rados.rados.0.smithi084.stdout:3166: done (0 left) 2024-09-20T22:54:35.370 INFO:tasks.rados.rados.0.smithi084.stdout:3167: snap_create 2024-09-20T22:54:35.454 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:54:35.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:35.476 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:35.476 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:54:35.577 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:54:35.598 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:35.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:35.599 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:54:35.614 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:35 smithi084 ceph-mon[40024]: pgmap v1092: 158 pgs: 1 active+clean+snaptrim, 1 active+remapped+backfill_toofull, 156 active+clean; 565 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 3.1 MiB/s wr, 15 op/s; 16/1368 objects misplaced (1.170%) 2024-09-20T22:54:35.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:35 smithi084 ceph-mon[40024]: osdmap e794: 8 total, 8 up, 8 in 2024-09-20T22:54:35.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:35 smithi084 ceph-mon[29774]: pgmap v1092: 158 pgs: 1 active+clean+snaptrim, 1 active+remapped+backfill_toofull, 156 active+clean; 565 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 3.1 MiB/s wr, 15 op/s; 16/1368 objects misplaced (1.170%) 2024-09-20T22:54:35.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:35 smithi084 ceph-mon[29774]: osdmap e794: 8 total, 8 up, 8 in 2024-09-20T22:54:35.701 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:54:35.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:35.722 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:35.722 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:54:35.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:35 smithi154 ceph-mon[33264]: pgmap v1092: 158 pgs: 1 active+clean+snaptrim, 1 active+remapped+backfill_toofull, 156 active+clean; 565 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 3.1 MiB/s wr, 15 op/s; 16/1368 objects misplaced (1.170%) 2024-09-20T22:54:35.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:35 smithi154 ceph-mon[33264]: osdmap e794: 8 total, 8 up, 8 in 2024-09-20T22:54:35.823 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:54:35.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:35.845 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:35.845 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:54:35.947 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:54:35.968 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:35.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:35.970 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:54:35.976 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd unset noscrub 2024-09-20T22:54:36.049 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.1 dump_historic_ops 2024-09-20T22:54:36.071 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:54:36.097 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:36.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:36.098 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:54:36.199 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:54:36.224 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:36.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:36.225 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:54:36.270 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:54:36.325 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:54:36.350 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:36.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:36.350 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:54:36.374 INFO:tasks.rados.rados.0.smithi084.stdout:3167: done (0 left) 2024-09-20T22:54:36.374 INFO:tasks.rados.rados.0.smithi084.stdout:3168: snap_remove snap 354 2024-09-20T22:54:36.451 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:54:36.472 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:36.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:36.473 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:54:36.574 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:54:36.598 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:36.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:36.599 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:54:36.628 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:36 smithi084 ceph-mon[40024]: osdmap e795: 8 total, 8 up, 8 in 2024-09-20T22:54:36.628 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:36 smithi084 ceph-mon[29774]: osdmap e795: 8 total, 8 up, 8 in 2024-09-20T22:54:36.700 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:54:36.723 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:36.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:36.724 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:54:36.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:36 smithi154 ceph-mon[33264]: osdmap e795: 8 total, 8 up, 8 in 2024-09-20T22:54:36.825 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:54:36.848 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:36.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:36.849 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:54:36.922 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:54:36.949 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:54:36.972 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:36.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:36.973 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:54:37.074 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:54:37.097 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:37.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:37.097 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:54:37.198 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:54:37.220 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:37.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:37.221 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:54:37.322 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:54:37.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:37.343 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:37.343 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:54:37.360 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-20T22:54:37.360 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2024-09-20T22:54:37.360 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2024-09-20T22:54:37.360 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [ 2024-09-20T22:54:37.360 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.360 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:2351 7.13 7:cae5792a:::smithi08477203-32:head [copy-from ver 376] snapc 56=[56,55,53,52,51,50,47,3e] ondisk+write+known_if_redirected+supports_pool_eio e260)", 2024-09-20T22:54:37.360 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:45:39.362090+0000", 2024-09-20T22:54:37.361 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 537.98599546699995, 2024-09-20T22:54:37.361 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.88649603300000002, 2024-09-20T22:54:37.361 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:54:37.361 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:54:37.361 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:54:37.361 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:54:37.361 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:54:37.361 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:54:37.361 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 2351 2024-09-20T22:54:37.361 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.361 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:54:37.361 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.361 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:54:37.361 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.362090+0000", 2024-09-20T22:54:37.361 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:54:37.362 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.362 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.362 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:54:37.362 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.362090+0000", 2024-09-20T22:54:37.362 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:54:37.362 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.362 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.362 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:54:37.362 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.362091+0000", 2024-09-20T22:54:37.362 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.5570000000000001e-06 2024-09-20T22:54:37.362 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.362 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.362 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:54:37.362 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.362099+0000", 2024-09-20T22:54:37.363 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.7030000000000006e-06 2024-09-20T22:54:37.363 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.363 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.363 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:54:37.363 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.362113+0000", 2024-09-20T22:54:37.363 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3492999999999999e-05 2024-09-20T22:54:37.363 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.363 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.363 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:54:37.363 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.362119+0000", 2024-09-20T22:54:37.363 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.7700000000000004e-06 2024-09-20T22:54:37.363 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.363 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.363 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:54:37.364 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.362190+0000", 2024-09-20T22:54:37.364 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.0576999999999999e-05 2024-09-20T22:54:37.364 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.364 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.364 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-09-20T22:54:37.364 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.362203+0000", 2024-09-20T22:54:37.364 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3484e-05 2024-09-20T22:54:37.364 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.364 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.364 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:54:37.364 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:40.226702+0000", 2024-09-20T22:54:37.364 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.86449815799999996 2024-09-20T22:54:37.364 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.364 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.365 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for missing object", 2024-09-20T22:54:37.365 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:40.227275+0000", 2024-09-20T22:54:37.365 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00057360899999999999 2024-09-20T22:54:37.365 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.365 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.365 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:54:37.365 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:40.232018+0000", 2024-09-20T22:54:37.365 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0047427839999999999 2024-09-20T22:54:37.365 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.365 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.365 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:54:37.365 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:40.232136+0000", 2024-09-20T22:54:37.365 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000118068 2024-09-20T22:54:37.365 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.365 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.366 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 2,5", 2024-09-20T22:54:37.366 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:40.240483+0000", 2024-09-20T22:54:37.366 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0083470390000000005 2024-09-20T22:54:37.366 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.366 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.366 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:54:37.366 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:40.245573+0000", 2024-09-20T22:54:37.366 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00508998 2024-09-20T22:54:37.366 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.366 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.366 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:54:37.366 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:40.248086+0000", 2024-09-20T22:54:37.366 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0025128870000000001 2024-09-20T22:54:37.367 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.367 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.367 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:54:37.367 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:40.248398+0000", 2024-09-20T22:54:37.367 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00031257600000000001 2024-09-20T22:54:37.367 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.367 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.367 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:54:37.367 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:40.248476+0000", 2024-09-20T22:54:37.367 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.7491999999999996e-05 2024-09-20T22:54:37.367 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.367 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.367 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:54:37.367 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:40.248586+0000", 2024-09-20T22:54:37.367 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000109856 2024-09-20T22:54:37.368 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:37.368 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:37.368 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:37.368 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.368 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.368 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:2352 7.13 7:cae5792a:::smithi08477203-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e260)", 2024-09-20T22:54:37.368 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:45:39.362137+0000", 2024-09-20T22:54:37.368 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 537.98594860000003, 2024-09-20T22:54:37.368 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.883015891, 2024-09-20T22:54:37.368 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:54:37.368 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:54:37.368 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-20T22:54:37.368 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:54:37.368 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:54:37.368 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:54:37.369 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 2352 2024-09-20T22:54:37.369 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.369 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:54:37.369 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.369 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:54:37.369 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.362137+0000", 2024-09-20T22:54:37.369 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:54:37.369 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.369 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.369 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:54:37.369 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.362137+0000", 2024-09-20T22:54:37.369 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:54:37.369 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.369 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.369 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:54:37.370 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.362137+0000", 2024-09-20T22:54:37.370 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.6300000000000001e-07 2024-09-20T22:54:37.370 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.370 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.370 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:54:37.370 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.362139+0000", 2024-09-20T22:54:37.370 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.7460000000000001e-06 2024-09-20T22:54:37.370 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.370 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.370 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:54:37.370 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.362144+0000", 2024-09-20T22:54:37.370 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.1080000000000001e-06 2024-09-20T22:54:37.370 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.370 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.371 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:54:37.371 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.362147+0000", 2024-09-20T22:54:37.371 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.2109999999999998e-06 2024-09-20T22:54:37.371 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.371 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.371 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:54:37.371 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.362233+0000", 2024-09-20T22:54:37.371 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.5736999999999996e-05 2024-09-20T22:54:37.371 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.371 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.371 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-09-20T22:54:37.371 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.362242+0000", 2024-09-20T22:54:37.371 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.8179999999999993e-06 2024-09-20T22:54:37.372 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.372 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.372 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:54:37.372 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:40.227702+0000", 2024-09-20T22:54:37.372 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.86546061299999999 2024-09-20T22:54:37.372 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.372 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.372 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for missing object", 2024-09-20T22:54:37.372 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:40.227751+0000", 2024-09-20T22:54:37.372 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.9128000000000001e-05 2024-09-20T22:54:37.372 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.372 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.372 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:54:37.372 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:40.232300+0000", 2024-09-20T22:54:37.372 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0045490260000000003 2024-09-20T22:54:37.373 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.373 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.373 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for blocked object", 2024-09-20T22:54:37.373 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:40.232354+0000", 2024-09-20T22:54:37.373 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.3183000000000002e-05 2024-09-20T22:54:37.373 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.373 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.373 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:54:37.373 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:40.244838+0000", 2024-09-20T22:54:37.373 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.012484004 2024-09-20T22:54:37.373 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.373 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.373 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:54:37.373 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:40.244974+0000", 2024-09-20T22:54:37.374 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013651399999999999 2024-09-20T22:54:37.374 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.374 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.374 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:54:37.374 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:40.245153+0000", 2024-09-20T22:54:37.374 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00017844000000000001 2024-09-20T22:54:37.374 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:37.374 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:37.374 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:37.374 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.374 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.374 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(osd.5.0:136 7.2 e269/169 7:45b54b6b:::smithi08477203-2:54 v 269'114, mlcod=265'111)", 2024-09-20T22:54:37.374 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:45:48.170083+0000", 2024-09-20T22:54:37.375 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 529.178002687, 2024-09-20T22:54:37.375 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.017111619000000002, 2024-09-20T22:54:37.375 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:54:37.375 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:54:37.375 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:54:37.375 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:54:37.375 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.375 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:54:37.375 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.170083+0000", 2024-09-20T22:54:37.375 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:54:37.375 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.375 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.375 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:54:37.375 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.170083+0000", 2024-09-20T22:54:37.375 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:54:37.376 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.376 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.376 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:54:37.376 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.170083+0000", 2024-09-20T22:54:37.376 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.2500000000000001e-07 2024-09-20T22:54:37.376 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.376 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.376 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:54:37.376 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.170096+0000", 2024-09-20T22:54:37.376 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3101999999999999e-05 2024-09-20T22:54:37.376 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.376 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.376 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:54:37.376 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.170101+0000", 2024-09-20T22:54:37.377 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.1379999999999998e-06 2024-09-20T22:54:37.377 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.377 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.377 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:54:37.377 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.170111+0000", 2024-09-20T22:54:37.377 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0305e-05 2024-09-20T22:54:37.377 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.377 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.377 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:54:37.377 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.173555+0000", 2024-09-20T22:54:37.377 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0034440389999999999 2024-09-20T22:54:37.377 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.377 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.377 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:54:37.378 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.173580+0000", 2024-09-20T22:54:37.378 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.4142e-05 2024-09-20T22:54:37.378 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.378 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.378 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:54:37.378 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.187133+0000", 2024-09-20T22:54:37.378 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013553170999999999 2024-09-20T22:54:37.378 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.378 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.378 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:54:37.378 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.187194+0000", 2024-09-20T22:54:37.378 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.1396999999999995e-05 2024-09-20T22:54:37.378 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:37.378 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:37.379 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:37.379 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.379 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.379 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:3124 7.b 7:df7b2b73:::smithi08477203-10:head [write 695922~709528] snapc 77=[77,76,75,74,73,70,6b] ondisk+write+known_if_redirected+supports_pool_eio e328)", 2024-09-20T22:54:37.379 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:46:48.190758+0000", 2024-09-20T22:54:37.379 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 469.15732749300003, 2024-09-20T22:54:37.379 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.030625260000000001, 2024-09-20T22:54:37.379 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:54:37.379 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:54:37.379 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:54:37.379 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:54:37.379 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:54:37.379 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:54:37.380 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 3124 2024-09-20T22:54:37.380 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.380 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:54:37.380 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.380 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:54:37.380 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.190758+0000", 2024-09-20T22:54:37.380 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:54:37.380 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.380 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.380 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:54:37.380 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.190758+0000", 2024-09-20T22:54:37.380 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:54:37.380 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.380 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.380 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:54:37.381 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.190760+0000", 2024-09-20T22:54:37.381 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.0109999999999999e-06 2024-09-20T22:54:37.381 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.381 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.381 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:54:37.381 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.191218+0000", 2024-09-20T22:54:37.381 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00045814 2024-09-20T22:54:37.381 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.381 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.381 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:54:37.381 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.191242+0000", 2024-09-20T22:54:37.381 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.4264e-05 2024-09-20T22:54:37.381 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.381 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.381 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:54:37.381 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.191256+0000", 2024-09-20T22:54:37.382 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.345e-05 2024-09-20T22:54:37.382 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.382 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.382 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:54:37.382 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.191362+0000", 2024-09-20T22:54:37.382 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00010649699999999999 2024-09-20T22:54:37.382 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.382 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.382 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op must wait for map", 2024-09-20T22:54:37.382 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.191370+0000", 2024-09-20T22:54:37.382 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.9049999999999997e-06 2024-09-20T22:54:37.382 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.382 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.382 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:54:37.383 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.202536+0000", 2024-09-20T22:54:37.383 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011165552 2024-09-20T22:54:37.383 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.383 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.383 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:54:37.383 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.202637+0000", 2024-09-20T22:54:37.383 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000101648 2024-09-20T22:54:37.383 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.383 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.383 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5,6", 2024-09-20T22:54:37.383 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.202824+0000", 2024-09-20T22:54:37.383 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00018637699999999999 2024-09-20T22:54:37.383 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.383 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.384 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:54:37.384 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.211036+0000", 2024-09-20T22:54:37.384 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0082124829999999996 2024-09-20T22:54:37.384 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.384 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.384 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:54:37.384 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.217521+0000", 2024-09-20T22:54:37.384 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0064849080000000002 2024-09-20T22:54:37.384 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.384 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.384 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:54:37.384 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.221251+0000", 2024-09-20T22:54:37.384 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0037300200000000001 2024-09-20T22:54:37.384 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.384 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.385 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:54:37.385 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.221325+0000", 2024-09-20T22:54:37.385 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.3737000000000002e-05 2024-09-20T22:54:37.385 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.385 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.385 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:54:37.385 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.221383+0000", 2024-09-20T22:54:37.385 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.8267999999999998e-05 2024-09-20T22:54:37.385 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:37.385 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:37.385 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:37.385 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.385 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.385 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:3125 7.b 7:df7b2b73:::smithi08477203-10:head [write 2032616~607480] snapc 77=[77,76,75,74,73,70,6b] ondisk+write+known_if_redirected+supports_pool_eio e328)", 2024-09-20T22:54:37.386 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:46:48.204698+0000", 2024-09-20T22:54:37.386 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 469.14338772299999, 2024-09-20T22:54:37.386 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.016777378999999999, 2024-09-20T22:54:37.386 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:54:37.386 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:54:37.386 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:54:37.386 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:54:37.386 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:54:37.386 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:54:37.386 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 3125 2024-09-20T22:54:37.386 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.386 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:54:37.386 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.386 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:54:37.387 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.204698+0000", 2024-09-20T22:54:37.387 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:54:37.387 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.387 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.387 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:54:37.387 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.204698+0000", 2024-09-20T22:54:37.387 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:54:37.387 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.387 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.387 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:54:37.387 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.204700+0000", 2024-09-20T22:54:37.387 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.1890000000000001e-06 2024-09-20T22:54:37.387 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.387 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.388 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:54:37.388 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.204872+0000", 2024-09-20T22:54:37.388 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00017232799999999999 2024-09-20T22:54:37.388 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.388 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.388 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:54:37.388 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.204885+0000", 2024-09-20T22:54:37.388 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3135e-05 2024-09-20T22:54:37.388 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.388 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.388 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:54:37.388 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.204896+0000", 2024-09-20T22:54:37.388 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1048e-05 2024-09-20T22:54:37.388 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.389 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.389 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:54:37.389 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.205292+0000", 2024-09-20T22:54:37.389 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00039570499999999999 2024-09-20T22:54:37.389 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.389 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.389 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:54:37.389 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.205404+0000", 2024-09-20T22:54:37.389 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000112224 2024-09-20T22:54:37.389 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.389 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.389 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5,6", 2024-09-20T22:54:37.389 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.206058+0000", 2024-09-20T22:54:37.389 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00065375100000000005 2024-09-20T22:54:37.390 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.390 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.390 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:54:37.390 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.211942+0000", 2024-09-20T22:54:37.390 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0058842690000000001 2024-09-20T22:54:37.390 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.390 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.390 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:54:37.390 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.217575+0000", 2024-09-20T22:54:37.390 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.005633201 2024-09-20T22:54:37.390 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.390 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.390 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:54:37.390 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.221393+0000", 2024-09-20T22:54:37.391 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00381797 2024-09-20T22:54:37.391 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.391 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.391 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:54:37.391 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.221438+0000", 2024-09-20T22:54:37.391 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.4347000000000001e-05 2024-09-20T22:54:37.391 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.391 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.391 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:54:37.391 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.221475+0000", 2024-09-20T22:54:37.391 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.7212000000000002e-05 2024-09-20T22:54:37.391 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:37.391 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:37.392 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:37.392 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.392 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.392 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:3237 7.b 7:d960cad4:::smithi08477203-18:head [copy-from ver 252] snapc 7e=[7e,7d,7c,7b,7a,79,78,75,74,73,6b] ondisk+write+known_if_redirected+supports_pool_eio e338)", 2024-09-20T22:54:37.392 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:46:58.180925+0000", 2024-09-20T22:54:37.392 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 459.16716016399999, 2024-09-20T22:54:37.392 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.029255882, 2024-09-20T22:54:37.392 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:54:37.392 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:54:37.392 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:54:37.392 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:54:37.392 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:54:37.392 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:54:37.392 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 3237 2024-09-20T22:54:37.393 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.393 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:54:37.393 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.393 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:54:37.393 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.180925+0000", 2024-09-20T22:54:37.393 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:54:37.393 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.393 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.393 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:54:37.393 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.180925+0000", 2024-09-20T22:54:37.393 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:54:37.393 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.393 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.393 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:54:37.393 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.180927+0000", 2024-09-20T22:54:37.393 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.945e-06 2024-09-20T22:54:37.394 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.394 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.394 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:54:37.394 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.180933+0000", 2024-09-20T22:54:37.394 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.7400000000000001e-06 2024-09-20T22:54:37.394 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.394 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.394 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:54:37.394 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.180947+0000", 2024-09-20T22:54:37.394 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3947999999999999e-05 2024-09-20T22:54:37.394 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.394 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.394 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:54:37.394 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.180954+0000", 2024-09-20T22:54:37.395 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.25e-06 2024-09-20T22:54:37.395 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.395 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.395 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:54:37.395 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.181002+0000", 2024-09-20T22:54:37.395 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.8331000000000002e-05 2024-09-20T22:54:37.395 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.395 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.395 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:54:37.395 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.181101+0000", 2024-09-20T22:54:37.395 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.8553000000000003e-05 2024-09-20T22:54:37.395 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.395 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.396 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5,6", 2024-09-20T22:54:37.396 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.196353+0000", 2024-09-20T22:54:37.396 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.015251789 2024-09-20T22:54:37.396 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.396 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.396 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:54:37.396 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.202094+0000", 2024-09-20T22:54:37.396 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0057409829999999999 2024-09-20T22:54:37.396 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.396 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.396 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:54:37.396 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.202682+0000", 2024-09-20T22:54:37.396 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000587898 2024-09-20T22:54:37.396 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.396 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.397 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:54:37.397 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.210104+0000", 2024-09-20T22:54:37.397 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0074227900000000003 2024-09-20T22:54:37.397 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.397 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.397 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:54:37.397 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.210145+0000", 2024-09-20T22:54:37.397 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.1050999999999997e-05 2024-09-20T22:54:37.397 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.397 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.397 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:54:37.397 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.210181+0000", 2024-09-20T22:54:37.397 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.5604e-05 2024-09-20T22:54:37.397 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:37.398 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:37.398 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:37.398 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.398 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.398 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:3238 7.b 7:d960cad4:::smithi08477203-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e338)", 2024-09-20T22:54:37.398 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:46:58.180973+0000", 2024-09-20T22:54:37.398 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 459.16711247900002, 2024-09-20T22:54:37.398 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.020749782000000001, 2024-09-20T22:54:37.398 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:54:37.398 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:54:37.398 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-20T22:54:37.398 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:54:37.398 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:54:37.398 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:54:37.399 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 3238 2024-09-20T22:54:37.399 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.399 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:54:37.399 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.399 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:54:37.399 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.180973+0000", 2024-09-20T22:54:37.399 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:54:37.399 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.399 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.399 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:54:37.399 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.180973+0000", 2024-09-20T22:54:37.399 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:54:37.399 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.399 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.399 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:54:37.399 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.180973+0000", 2024-09-20T22:54:37.400 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.4000000000000002e-07 2024-09-20T22:54:37.400 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.400 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.400 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:54:37.400 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.180976+0000", 2024-09-20T22:54:37.400 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.576e-06 2024-09-20T22:54:37.400 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.400 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.400 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:54:37.400 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.180982+0000", 2024-09-20T22:54:37.400 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.0700000000000003e-06 2024-09-20T22:54:37.400 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.400 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.400 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:54:37.401 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.180988+0000", 2024-09-20T22:54:37.401 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.5929999999999996e-06 2024-09-20T22:54:37.401 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.401 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.401 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:54:37.401 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.182094+0000", 2024-09-20T22:54:37.401 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001106604 2024-09-20T22:54:37.401 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.401 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.401 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for blocked object", 2024-09-20T22:54:37.401 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.182223+0000", 2024-09-20T22:54:37.401 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000129152 2024-09-20T22:54:37.401 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.401 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.401 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:54:37.401 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.201400+0000", 2024-09-20T22:54:37.402 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.019176192000000002 2024-09-20T22:54:37.402 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.402 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.402 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:54:37.402 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.201563+0000", 2024-09-20T22:54:37.402 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00016328999999999999 2024-09-20T22:54:37.402 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.402 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.402 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:54:37.402 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.201723+0000", 2024-09-20T22:54:37.402 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000159765 2024-09-20T22:54:37.402 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:37.402 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:37.402 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:37.402 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.403 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.403 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24652.0:3235 7.1 e338/157 7:84a866f5:::smithi08477203-27:head v 338'167, mlcod=320'164)", 2024-09-20T22:54:37.403 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:46:58.205374+0000", 2024-09-20T22:54:37.403 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 459.14271160700002, 2024-09-20T22:54:37.403 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.019862464999999999, 2024-09-20T22:54:37.403 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:54:37.403 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:54:37.403 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:54:37.403 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:54:37.403 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.403 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:54:37.403 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.205374+0000", 2024-09-20T22:54:37.403 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:54:37.403 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.404 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.404 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:54:37.404 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.205374+0000", 2024-09-20T22:54:37.404 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:54:37.404 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.404 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.404 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:54:37.404 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.205374+0000", 2024-09-20T22:54:37.404 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.3200000000000001e-07 2024-09-20T22:54:37.404 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.404 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.404 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:54:37.404 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.208223+0000", 2024-09-20T22:54:37.404 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0028490820000000002 2024-09-20T22:54:37.404 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.404 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.405 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:54:37.405 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.208249+0000", 2024-09-20T22:54:37.405 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.584e-05 2024-09-20T22:54:37.405 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.405 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.405 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:54:37.405 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.208262+0000", 2024-09-20T22:54:37.405 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3166999999999999e-05 2024-09-20T22:54:37.405 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.405 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.405 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:54:37.405 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.208344+0000", 2024-09-20T22:54:37.405 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.1567e-05 2024-09-20T22:54:37.405 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.405 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.406 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:54:37.406 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.208370+0000", 2024-09-20T22:54:37.406 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.5973999999999999e-05 2024-09-20T22:54:37.406 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.406 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.406 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:54:37.406 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.225185+0000", 2024-09-20T22:54:37.406 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.016814976999999998 2024-09-20T22:54:37.406 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.406 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.406 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:54:37.406 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.225236+0000", 2024-09-20T22:54:37.406 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.1526000000000001e-05 2024-09-20T22:54:37.406 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:37.406 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:37.407 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:37.407 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.407 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.407 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:3589 7.b 7:dc95255e:::smithi08477203-1:head [copy-from ver 516] snapc 8d=[8d,8c,8b,8a,89,88,87,86,85,84,80,7f,7a,74,73,6b] ondisk+write+known_if_redirected+supports_pool_eio e363)", 2024-09-20T22:54:37.407 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:47:23.191965+0000", 2024-09-20T22:54:37.407 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 434.156120198, 2024-09-20T22:54:37.407 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.027894147000000001, 2024-09-20T22:54:37.407 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:54:37.407 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:54:37.407 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:54:37.407 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:54:37.407 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:54:37.407 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:54:37.407 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 3589 2024-09-20T22:54:37.407 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.408 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:54:37.408 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.408 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:54:37.408 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:23.191965+0000", 2024-09-20T22:54:37.408 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:54:37.408 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.408 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.408 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:54:37.408 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:23.191965+0000", 2024-09-20T22:54:37.408 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:54:37.408 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.408 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.408 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:54:37.408 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:23.191967+0000", 2024-09-20T22:54:37.409 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.6920000000000001e-06 2024-09-20T22:54:37.409 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.409 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.409 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:54:37.409 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:23.191972+0000", 2024-09-20T22:54:37.409 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.3770000000000004e-06 2024-09-20T22:54:37.409 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.409 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.409 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:54:37.409 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:23.191985+0000", 2024-09-20T22:54:37.409 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2509e-05 2024-09-20T22:54:37.409 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.409 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.409 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:54:37.410 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:23.191993+0000", 2024-09-20T22:54:37.410 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.14e-06 2024-09-20T22:54:37.410 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.410 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.410 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:54:37.410 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:23.192151+0000", 2024-09-20T22:54:37.410 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00015796699999999999 2024-09-20T22:54:37.410 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.410 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.410 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:54:37.410 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:23.192343+0000", 2024-09-20T22:54:37.410 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00019251000000000001 2024-09-20T22:54:37.410 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.410 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.411 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5,6", 2024-09-20T22:54:37.411 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:23.205044+0000", 2024-09-20T22:54:37.411 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.012700751 2024-09-20T22:54:37.411 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.411 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.411 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:54:37.411 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:23.209060+0000", 2024-09-20T22:54:37.411 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0040161959999999997 2024-09-20T22:54:37.411 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.411 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.411 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:54:37.411 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:23.219722+0000", 2024-09-20T22:54:37.411 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010661976 2024-09-20T22:54:37.411 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.411 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.411 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:54:37.412 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:23.219767+0000", 2024-09-20T22:54:37.412 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.4768999999999998e-05 2024-09-20T22:54:37.412 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.412 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.412 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:54:37.412 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:23.219810+0000", 2024-09-20T22:54:37.412 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.3544000000000002e-05 2024-09-20T22:54:37.412 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.412 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.412 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:54:37.412 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:23.219859+0000", 2024-09-20T22:54:37.412 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.8715999999999999e-05 2024-09-20T22:54:37.412 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:37.412 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:37.412 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:37.412 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.413 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.413 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:3590 7.b 7:dc95255e:::smithi08477203-1:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e363)", 2024-09-20T22:54:37.413 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:47:23.192017+0000", 2024-09-20T22:54:37.413 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 434.15606816000002, 2024-09-20T22:54:37.413 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.016178512999999999, 2024-09-20T22:54:37.413 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:54:37.413 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:54:37.413 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-20T22:54:37.413 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:54:37.413 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:54:37.413 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:54:37.414 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-20T22:54:37.415 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 3590 2024-09-20T22:54:37.415 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.415 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:54:37.415 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.415 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:54:37.415 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:23.192017+0000", 2024-09-20T22:54:37.415 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:54:37.415 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.415 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.416 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:54:37.416 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:23.192017+0000", 2024-09-20T22:54:37.416 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:54:37.416 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.416 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.416 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:54:37.416 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:23.192018+0000", 2024-09-20T22:54:37.416 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.4300000000000003e-07 2024-09-20T22:54:37.416 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.416 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.416 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:54:37.416 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:23.192020+0000", 2024-09-20T22:54:37.416 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.553e-06 2024-09-20T22:54:37.416 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.416 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.417 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:54:37.417 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:23.192027+0000", 2024-09-20T22:54:37.417 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.3990000000000001e-06 2024-09-20T22:54:37.417 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.417 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.417 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:54:37.417 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:23.192032+0000", 2024-09-20T22:54:37.417 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.3709999999999999e-06 2024-09-20T22:54:37.417 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.417 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.417 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:54:37.417 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:23.192511+0000", 2024-09-20T22:54:37.417 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00047946700000000002 2024-09-20T22:54:37.418 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.418 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.418 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for blocked object", 2024-09-20T22:54:37.418 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:23.192563+0000", 2024-09-20T22:54:37.418 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.1158e-05 2024-09-20T22:54:37.418 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.418 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.418 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:54:37.418 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:23.207970+0000", 2024-09-20T22:54:37.418 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.015407522999999999 2024-09-20T22:54:37.418 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.418 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.418 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:54:37.419 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:23.208078+0000", 2024-09-20T22:54:37.419 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00010755999999999999 2024-09-20T22:54:37.419 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.419 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.419 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:54:37.419 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:23.208196+0000", 2024-09-20T22:54:37.419 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000117939 2024-09-20T22:54:37.419 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:37.419 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:37.419 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:37.419 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.419 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.419 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:3760 7.13 7:cae5792a:::smithi08477203-32:head [copy-from ver 131] snapc 98=[98,97,96,95,94,93,92,8e,8d,8c,8b,8a,88,87,86,80,7f,73] ondisk+write+known_if_redirected+supports_pool_eio e383)", 2024-09-20T22:54:37.420 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:47:43.282985+0000", 2024-09-20T22:54:37.420 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 414.06509979999998, 2024-09-20T22:54:37.420 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.021555008000000001, 2024-09-20T22:54:37.420 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:54:37.420 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:54:37.420 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:54:37.420 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:54:37.420 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:54:37.420 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:54:37.420 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 3760 2024-09-20T22:54:37.420 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.420 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:54:37.420 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.420 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:54:37.420 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:43.282985+0000", 2024-09-20T22:54:37.421 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:54:37.421 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.421 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.421 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:54:37.421 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:43.282985+0000", 2024-09-20T22:54:37.421 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:54:37.421 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.421 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.421 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:54:37.421 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:43.282989+0000", 2024-09-20T22:54:37.421 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.4570000000000001e-06 2024-09-20T22:54:37.421 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.421 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.421 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:54:37.421 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:43.283000+0000", 2024-09-20T22:54:37.422 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1012999999999999e-05 2024-09-20T22:54:37.422 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.422 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.422 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:54:37.422 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:43.283026+0000", 2024-09-20T22:54:37.422 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.6313e-05 2024-09-20T22:54:37.422 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.422 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.422 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:54:37.422 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:43.283038+0000", 2024-09-20T22:54:37.422 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1976e-05 2024-09-20T22:54:37.422 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.422 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.422 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:54:37.423 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:43.283136+0000", 2024-09-20T22:54:37.423 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.7569e-05 2024-09-20T22:54:37.423 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.423 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.423 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:54:37.423 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:43.283257+0000", 2024-09-20T22:54:37.423 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000121408 2024-09-20T22:54:37.423 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.423 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.423 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 2,5", 2024-09-20T22:54:37.423 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:43.291446+0000", 2024-09-20T22:54:37.423 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0081890850000000005 2024-09-20T22:54:37.423 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.423 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.424 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:54:37.424 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:43.303163+0000", 2024-09-20T22:54:37.424 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011716809 2024-09-20T22:54:37.424 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.424 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.424 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:54:37.424 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:43.304043+0000", 2024-09-20T22:54:37.424 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00087947299999999997 2024-09-20T22:54:37.424 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.424 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.424 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:54:37.424 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:43.304457+0000", 2024-09-20T22:54:37.424 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00041483299999999998 2024-09-20T22:54:37.424 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.425 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.425 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:54:37.425 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:43.304500+0000", 2024-09-20T22:54:37.425 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.2484000000000002e-05 2024-09-20T22:54:37.425 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.425 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.425 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:54:37.425 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:43.304540+0000", 2024-09-20T22:54:37.425 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.0587999999999998e-05 2024-09-20T22:54:37.425 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:37.425 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:37.425 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:37.425 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.425 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.425 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(osd.5.0:213 7.2 e404/169 7:45b54b6b:::smithi08477203-2:a5 v 404'178, mlcod=402'175)", 2024-09-20T22:54:37.426 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:48:04.218296+0000", 2024-09-20T22:54:37.426 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 393.12978907299998, 2024-09-20T22:54:37.426 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.016469849000000002, 2024-09-20T22:54:37.426 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:54:37.426 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:54:37.426 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:54:37.426 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:54:37.426 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.426 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:54:37.426 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:04.218296+0000", 2024-09-20T22:54:37.426 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:54:37.426 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.426 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.426 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:54:37.427 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:04.218296+0000", 2024-09-20T22:54:37.427 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:54:37.427 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.427 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.427 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:54:37.427 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:04.218297+0000", 2024-09-20T22:54:37.427 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.3799999999999998e-07 2024-09-20T22:54:37.427 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.427 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.427 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:54:37.427 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:04.218306+0000", 2024-09-20T22:54:37.427 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.9429999999999995e-06 2024-09-20T22:54:37.427 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.427 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.427 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:54:37.428 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:04.218312+0000", 2024-09-20T22:54:37.428 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.7679999999999997e-06 2024-09-20T22:54:37.428 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.428 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.428 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:54:37.428 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:04.218321+0000", 2024-09-20T22:54:37.428 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.6619999999999994e-06 2024-09-20T22:54:37.428 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.428 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.428 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:54:37.428 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:04.233566+0000", 2024-09-20T22:54:37.428 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.015245491999999999 2024-09-20T22:54:37.428 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.428 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.428 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:54:37.428 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:04.233589+0000", 2024-09-20T22:54:37.429 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.2317e-05 2024-09-20T22:54:37.429 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.429 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.429 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:54:37.429 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:04.234725+0000", 2024-09-20T22:54:37.429 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0011366460000000001 2024-09-20T22:54:37.429 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.429 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.429 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:54:37.429 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:04.234766+0000", 2024-09-20T22:54:37.429 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.0683000000000003e-05 2024-09-20T22:54:37.429 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:37.429 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:37.429 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:37.430 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.430 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.430 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24652.0:4444 7.0 e447/137 7:05bb442d:::smithi08477203-24:head v 447'216, mlcod=444'212)", 2024-09-20T22:54:37.430 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:48:47.242826+0000", 2024-09-20T22:54:37.430 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 350.10525895199999, 2024-09-20T22:54:37.430 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.017527779, 2024-09-20T22:54:37.430 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:54:37.430 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:54:37.430 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:54:37.430 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:54:37.430 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.430 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:54:37.430 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:47.242826+0000", 2024-09-20T22:54:37.431 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:54:37.431 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.431 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.431 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:54:37.431 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:47.242826+0000", 2024-09-20T22:54:37.431 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:54:37.431 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.431 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.431 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:54:37.431 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:47.242827+0000", 2024-09-20T22:54:37.431 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.9999999999999998e-07 2024-09-20T22:54:37.431 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.431 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.431 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:54:37.431 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:47.251760+0000", 2024-09-20T22:54:37.432 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0089333339999999994 2024-09-20T22:54:37.432 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.432 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.432 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:54:37.432 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:47.251788+0000", 2024-09-20T22:54:37.432 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.8343e-05 2024-09-20T22:54:37.432 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.432 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.432 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:54:37.432 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:47.251806+0000", 2024-09-20T22:54:37.432 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.7909000000000001e-05 2024-09-20T22:54:37.432 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.432 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.432 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:54:37.432 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:47.251920+0000", 2024-09-20T22:54:37.433 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000113165 2024-09-20T22:54:37.433 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.433 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.433 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:54:37.433 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:47.251944+0000", 2024-09-20T22:54:37.433 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.4507000000000001e-05 2024-09-20T22:54:37.433 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.433 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.433 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:54:37.433 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:47.260296+0000", 2024-09-20T22:54:37.433 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0083519419999999994 2024-09-20T22:54:37.433 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.433 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.433 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:54:37.433 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:47.260354+0000", 2024-09-20T22:54:37.433 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.8078999999999997e-05 2024-09-20T22:54:37.434 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:37.434 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:37.434 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:37.434 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.434 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.434 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24652.0:4640 7.9 e468/426 7:907aec0e:::smithi08477203-3:head v 468'434, mlcod=468'431)", 2024-09-20T22:54:37.434 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:49:08.245712+0000", 2024-09-20T22:54:37.434 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 329.10237282200001, 2024-09-20T22:54:37.434 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.01546574, 2024-09-20T22:54:37.434 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:54:37.434 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:54:37.434 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:54:37.434 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:54:37.434 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.434 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:54:37.435 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:49:08.245712+0000", 2024-09-20T22:54:37.435 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:54:37.435 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.435 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.435 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:54:37.435 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:49:08.245712+0000", 2024-09-20T22:54:37.435 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:54:37.435 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.435 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.435 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:54:37.435 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:49:08.245713+0000", 2024-09-20T22:54:37.435 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.6300000000000004e-07 2024-09-20T22:54:37.435 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.435 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.435 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:54:37.436 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:49:08.246012+0000", 2024-09-20T22:54:37.436 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000298521 2024-09-20T22:54:37.436 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.436 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.436 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:54:37.436 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:49:08.246024+0000", 2024-09-20T22:54:37.436 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2014999999999999e-05 2024-09-20T22:54:37.436 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.436 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.436 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:54:37.436 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:49:08.246034+0000", 2024-09-20T22:54:37.436 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0147e-05 2024-09-20T22:54:37.436 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.436 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.436 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:54:37.436 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:49:08.246097+0000", 2024-09-20T22:54:37.437 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.2874999999999997e-05 2024-09-20T22:54:37.437 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.437 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.437 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:54:37.437 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:49:08.246123+0000", 2024-09-20T22:54:37.437 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.6211999999999999e-05 2024-09-20T22:54:37.437 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.437 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.437 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:54:37.437 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:49:08.261118+0000", 2024-09-20T22:54:37.437 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014995517999999999 2024-09-20T22:54:37.437 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.437 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.437 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:54:37.437 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:49:08.261178+0000", 2024-09-20T22:54:37.437 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.9688999999999997e-05 2024-09-20T22:54:37.438 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:37.438 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:37.438 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:37.438 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.438 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.438 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24652.0:4638 7.d e468/391 7:b30e4612:::smithi08477203-42:head v 468'257, mlcod=466'252)", 2024-09-20T22:54:37.438 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:49:08.256560+0000", 2024-09-20T22:54:37.438 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 329.09152561899998, 2024-09-20T22:54:37.438 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.015962423999999999, 2024-09-20T22:54:37.438 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:54:37.438 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:54:37.438 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:54:37.438 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:54:37.438 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.438 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:54:37.439 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:49:08.256560+0000", 2024-09-20T22:54:37.439 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:54:37.439 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.439 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.439 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:54:37.439 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:49:08.256560+0000", 2024-09-20T22:54:37.439 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:54:37.439 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.439 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.439 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:54:37.439 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:49:08.256560+0000", 2024-09-20T22:54:37.439 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.7100000000000001e-07 2024-09-20T22:54:37.439 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.439 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.439 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:54:37.439 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:49:08.263534+0000", 2024-09-20T22:54:37.440 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0069739069999999997 2024-09-20T22:54:37.440 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.440 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.440 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:54:37.440 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:49:08.263550+0000", 2024-09-20T22:54:37.440 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.5974999999999999e-05 2024-09-20T22:54:37.440 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.440 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.440 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:54:37.440 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:49:08.263560+0000", 2024-09-20T22:54:37.440 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.5889999999999992e-06 2024-09-20T22:54:37.440 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.440 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.440 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:54:37.440 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:49:08.263643+0000", 2024-09-20T22:54:37.440 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.3369999999999996e-05 2024-09-20T22:54:37.440 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.441 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.441 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:54:37.441 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:49:08.263666+0000", 2024-09-20T22:54:37.441 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.3224e-05 2024-09-20T22:54:37.441 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.441 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.441 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:54:37.441 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:49:08.272477+0000", 2024-09-20T22:54:37.441 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0088104729999999992 2024-09-20T22:54:37.441 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.441 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.441 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:54:37.441 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:49:08.272522+0000", 2024-09-20T22:54:37.441 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.5114999999999997e-05 2024-09-20T22:54:37.441 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:37.441 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:37.442 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:37.442 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.442 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.442 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:5011 7.b 7:df7b2b73:::smithi08477203-10:head [copy-from ver 1207] snapc d8=[d8,d7,d6,d5,d4,d3,d2,d1,d0,cf,ce,cd,cc,ca,c7,c6,c5,c2,bf,ba,b4,b2,a7,9e,9c,95,87] ondisk+write+known_if_redirected+supports_pool_eio e505)", 2024-09-20T22:54:37.442 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:49:45.297952+0000", 2024-09-20T22:54:37.442 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 292.050132905, 2024-09-20T22:54:37.442 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.022999847, 2024-09-20T22:54:37.442 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:54:37.442 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:54:37.442 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:54:37.442 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:54:37.442 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:54:37.442 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:54:37.442 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 5011 2024-09-20T22:54:37.442 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.443 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:54:37.443 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.443 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:54:37.443 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:49:45.297952+0000", 2024-09-20T22:54:37.443 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:54:37.443 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.443 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.443 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:54:37.443 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:49:45.297952+0000", 2024-09-20T22:54:37.443 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:54:37.443 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.443 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.443 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:54:37.443 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:49:45.297954+0000", 2024-09-20T22:54:37.443 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.9190000000000002e-06 2024-09-20T22:54:37.443 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.444 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.444 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:54:37.444 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:49:45.297967+0000", 2024-09-20T22:54:37.444 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.242e-05 2024-09-20T22:54:37.444 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.444 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.444 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:54:37.444 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:49:45.297982+0000", 2024-09-20T22:54:37.444 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.5183e-05 2024-09-20T22:54:37.444 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.444 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.444 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:54:37.444 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:49:45.297990+0000", 2024-09-20T22:54:37.444 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.0020000000000006e-06 2024-09-20T22:54:37.444 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.444 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.445 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:54:37.445 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:49:45.298075+0000", 2024-09-20T22:54:37.445 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.4852999999999995e-05 2024-09-20T22:54:37.445 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.445 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.445 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:54:37.445 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:49:45.298178+0000", 2024-09-20T22:54:37.445 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000103553 2024-09-20T22:54:37.445 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.445 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.445 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5,6", 2024-09-20T22:54:37.445 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:49:45.305815+0000", 2024-09-20T22:54:37.445 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0076365349999999999 2024-09-20T22:54:37.445 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.445 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.446 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:54:37.446 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:49:45.310882+0000", 2024-09-20T22:54:37.446 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.005066966 2024-09-20T22:54:37.446 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.446 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.446 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:54:37.446 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:49:45.320429+0000", 2024-09-20T22:54:37.446 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0095475709999999995 2024-09-20T22:54:37.446 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.446 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.446 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:54:37.446 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:49:45.320842+0000", 2024-09-20T22:54:37.446 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00041306300000000003 2024-09-20T22:54:37.446 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.447 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.447 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:54:37.447 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:49:45.320886+0000", 2024-09-20T22:54:37.447 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.3204999999999997e-05 2024-09-20T22:54:37.447 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.447 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.447 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:54:37.447 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:49:45.320952+0000", 2024-09-20T22:54:37.447 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.6576999999999996e-05 2024-09-20T22:54:37.447 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:37.447 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:37.447 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:37.447 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.447 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.448 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:5514 7.b 7:d3ca3619:::smithi08477203-39:head [copy-from ver 729] snapc f4=[f4,f3,f2,f1,f0,ef,ed,ec,ea,e8,e6,e4,e2,df,db,da,d7,d5,d4,d3,cf,ce,c5,c2,bf,b4,b2,a7] ondisk+write+known_if_redirected+supports_pool_eio e560)", 2024-09-20T22:54:37.448 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:50:40.272011+0000", 2024-09-20T22:54:37.448 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 237.07607413900001, 2024-09-20T22:54:37.448 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.028606645, 2024-09-20T22:54:37.448 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:54:37.448 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:54:37.448 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:54:37.448 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:54:37.448 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:54:37.448 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:54:37.448 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 5514 2024-09-20T22:54:37.448 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.448 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:54:37.448 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.448 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:54:37.449 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:40.272011+0000", 2024-09-20T22:54:37.449 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:54:37.449 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.449 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.449 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:54:37.449 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:40.272011+0000", 2024-09-20T22:54:37.449 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:54:37.449 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.449 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.449 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:54:37.449 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:40.272013+0000", 2024-09-20T22:54:37.449 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.3159999999999999e-06 2024-09-20T22:54:37.449 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.449 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.449 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:54:37.450 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:40.272023+0000", 2024-09-20T22:54:37.450 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.9229999999999997e-06 2024-09-20T22:54:37.450 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.450 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.450 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:54:37.450 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:40.272044+0000", 2024-09-20T22:54:37.450 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.0590000000000001e-05 2024-09-20T22:54:37.450 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.450 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.450 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:54:37.450 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:40.272054+0000", 2024-09-20T22:54:37.450 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.9760000000000007e-06 2024-09-20T22:54:37.450 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.450 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.451 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:54:37.451 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:40.272138+0000", 2024-09-20T22:54:37.451 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.4122999999999994e-05 2024-09-20T22:54:37.451 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.451 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.451 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op must wait for map", 2024-09-20T22:54:37.451 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:40.272147+0000", 2024-09-20T22:54:37.451 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.1919999999999994e-06 2024-09-20T22:54:37.451 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.451 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.451 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:54:37.451 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:40.277061+0000", 2024-09-20T22:54:37.451 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0049135070000000001 2024-09-20T22:54:37.451 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.451 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.451 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:54:37.452 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:40.277135+0000", 2024-09-20T22:54:37.452 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.4307999999999994e-05 2024-09-20T22:54:37.452 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.452 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.452 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5,6", 2024-09-20T22:54:37.452 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:40.288318+0000", 2024-09-20T22:54:37.452 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011183324 2024-09-20T22:54:37.452 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.452 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.452 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:54:37.452 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:40.296101+0000", 2024-09-20T22:54:37.452 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0077823500000000004 2024-09-20T22:54:37.452 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.452 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.452 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:54:37.452 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:40.299263+0000", 2024-09-20T22:54:37.453 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0031626279999999998 2024-09-20T22:54:37.453 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.453 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.453 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:54:37.453 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:40.300471+0000", 2024-09-20T22:54:37.453 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00120729 2024-09-20T22:54:37.453 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.453 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.453 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:54:37.453 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:40.300554+0000", 2024-09-20T22:54:37.453 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.3728999999999997e-05 2024-09-20T22:54:37.453 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.453 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.453 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:54:37.453 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:40.300618+0000", 2024-09-20T22:54:37.453 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.3388999999999999e-05 2024-09-20T22:54:37.454 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:37.454 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:37.454 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:37.454 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.454 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.454 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:5515 7.b 7:d3ca3619:::smithi08477203-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e560)", 2024-09-20T22:54:37.454 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:50:40.272081+0000", 2024-09-20T22:54:37.454 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 237.07600432199999, 2024-09-20T22:54:37.454 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.020994938000000001, 2024-09-20T22:54:37.454 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:54:37.454 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:54:37.454 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-20T22:54:37.454 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:54:37.454 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:54:37.455 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:54:37.455 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 5515 2024-09-20T22:54:37.455 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.455 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:54:37.455 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.455 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:54:37.455 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:40.272081+0000", 2024-09-20T22:54:37.455 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:54:37.455 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.455 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.455 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:54:37.455 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:40.272081+0000", 2024-09-20T22:54:37.455 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:54:37.455 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.455 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.456 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:54:37.456 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:40.272082+0000", 2024-09-20T22:54:37.456 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.7700000000000004e-07 2024-09-20T22:54:37.456 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.456 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.456 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:54:37.456 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:40.272085+0000", 2024-09-20T22:54:37.456 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.5240000000000001e-06 2024-09-20T22:54:37.456 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.456 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.456 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:54:37.456 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:40.272093+0000", 2024-09-20T22:54:37.456 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.8790000000000003e-06 2024-09-20T22:54:37.456 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.456 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.456 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:54:37.457 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:40.272099+0000", 2024-09-20T22:54:37.457 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.879e-06 2024-09-20T22:54:37.457 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.457 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.457 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:54:37.457 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:40.272222+0000", 2024-09-20T22:54:37.457 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00012285 2024-09-20T22:54:37.457 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.457 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.457 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting_for_map not empty", 2024-09-20T22:54:37.457 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:40.272235+0000", 2024-09-20T22:54:37.457 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3788000000000001e-05 2024-09-20T22:54:37.457 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.457 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.457 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:54:37.457 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:40.277248+0000", 2024-09-20T22:54:37.458 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0050127210000000004 2024-09-20T22:54:37.458 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.458 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.458 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for blocked object", 2024-09-20T22:54:37.458 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:40.277277+0000", 2024-09-20T22:54:37.458 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.8498000000000001e-05 2024-09-20T22:54:37.458 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.458 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.458 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:54:37.458 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:40.292787+0000", 2024-09-20T22:54:37.458 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.015510636 2024-09-20T22:54:37.458 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.458 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.458 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:54:37.458 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:40.292928+0000", 2024-09-20T22:54:37.459 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014066400000000001 2024-09-20T22:54:37.459 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.459 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.459 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:54:37.459 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:40.293076+0000", 2024-09-20T22:54:37.459 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014792200000000001 2024-09-20T22:54:37.459 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:37.459 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:37.459 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:37.459 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.459 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.459 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:5659 7.b 7:dc95255e:::smithi08477203-1:head [copy-from ver 731] snapc f9=[f9,f8,f7,f6,f5,f4,f2,f1,ef,ed,ec,ea,e6,e4,e2,df,db,da,d7,d4,d3,ce,c5,c2,bf,b4,b2,a7] ondisk+write+known_if_redirected+supports_pool_eio e570)", 2024-09-20T22:54:37.459 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:50:51.277671+0000", 2024-09-20T22:54:37.460 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 226.070413758, 2024-09-20T22:54:37.460 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.024917069999999999, 2024-09-20T22:54:37.460 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:54:37.460 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:54:37.460 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:54:37.460 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:54:37.460 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:54:37.460 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:54:37.460 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 5659 2024-09-20T22:54:37.460 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.460 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:54:37.460 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.460 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:54:37.460 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:51.277671+0000", 2024-09-20T22:54:37.461 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:54:37.461 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.461 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.461 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:54:37.461 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:51.277671+0000", 2024-09-20T22:54:37.461 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:54:37.461 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.461 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.461 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:54:37.461 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:51.277672+0000", 2024-09-20T22:54:37.461 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.2399999999999999e-07 2024-09-20T22:54:37.461 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.461 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.461 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:54:37.461 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:51.277676+0000", 2024-09-20T22:54:37.461 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.0879999999999997e-06 2024-09-20T22:54:37.462 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.462 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.462 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:54:37.462 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:51.277682+0000", 2024-09-20T22:54:37.462 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.8130000000000001e-06 2024-09-20T22:54:37.462 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.462 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.462 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:54:37.462 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:51.277688+0000", 2024-09-20T22:54:37.462 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.9699999999999996e-06 2024-09-20T22:54:37.462 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.462 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.462 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:54:37.462 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:51.278165+0000", 2024-09-20T22:54:37.462 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00047736300000000002 2024-09-20T22:54:37.462 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.463 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.463 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:54:37.463 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:51.278262+0000", 2024-09-20T22:54:37.463 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.6638000000000005e-05 2024-09-20T22:54:37.463 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.463 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.463 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 5,6", 2024-09-20T22:54:37.463 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:51.285554+0000", 2024-09-20T22:54:37.463 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0072918219999999999 2024-09-20T22:54:37.463 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.463 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.463 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:54:37.463 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:51.294109+0000", 2024-09-20T22:54:37.463 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0085554729999999992 2024-09-20T22:54:37.463 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.464 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.464 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:54:37.464 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:51.302364+0000", 2024-09-20T22:54:37.464 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0082549540000000001 2024-09-20T22:54:37.464 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.464 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.464 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:54:37.464 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:51.302426+0000", 2024-09-20T22:54:37.464 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.224e-05 2024-09-20T22:54:37.464 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.464 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.464 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:54:37.464 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:51.302507+0000", 2024-09-20T22:54:37.464 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.0797999999999994e-05 2024-09-20T22:54:37.464 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.464 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.465 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:54:37.465 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:51.302589+0000", 2024-09-20T22:54:37.465 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.1487000000000001e-05 2024-09-20T22:54:37.465 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:37.465 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:37.465 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:37.465 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.465 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.465 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24652.0:6378 7.2 e639/600 7:4d20232a:::smithi08477203-34:head v 639'665, mlcod=629'662)", 2024-09-20T22:54:37.465 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:51:59.278699+0000", 2024-09-20T22:54:37.465 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 158.06938628500001, 2024-09-20T22:54:37.465 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.015272501000000001, 2024-09-20T22:54:37.465 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:54:37.465 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:54:37.465 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:54:37.465 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:54:37.465 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.466 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:54:37.466 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:59.278699+0000", 2024-09-20T22:54:37.466 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:54:37.466 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.466 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.466 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:54:37.466 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:59.278699+0000", 2024-09-20T22:54:37.466 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:54:37.466 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.466 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.466 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:54:37.466 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:59.278699+0000", 2024-09-20T22:54:37.466 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.6399999999999998e-07 2024-09-20T22:54:37.466 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.466 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.467 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:54:37.467 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:59.281173+0000", 2024-09-20T22:54:37.467 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002473523 2024-09-20T22:54:37.467 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.467 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.467 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:54:37.467 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:59.281190+0000", 2024-09-20T22:54:37.467 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.7391e-05 2024-09-20T22:54:37.467 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.467 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.467 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:54:37.467 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:59.281199+0000", 2024-09-20T22:54:37.467 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.9900000000000003e-06 2024-09-20T22:54:37.467 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.467 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.467 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:54:37.467 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:59.281278+0000", 2024-09-20T22:54:37.468 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.8770999999999995e-05 2024-09-20T22:54:37.468 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.468 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.468 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:54:37.468 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:59.281313+0000", 2024-09-20T22:54:37.468 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.4573000000000003e-05 2024-09-20T22:54:37.468 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.468 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.468 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:54:37.468 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:59.293921+0000", 2024-09-20T22:54:37.468 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.012608263 2024-09-20T22:54:37.468 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:37.468 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:37.468 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:54:37.468 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:59.293971+0000", 2024-09-20T22:54:37.468 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.0625999999999999e-05 2024-09-20T22:54:37.469 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:37.469 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:37.469 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:37.469 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:37.469 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:37.469 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-20T22:54:37.469 INFO:tasks.rados.rados.0.smithi084.stdout:3168: done (0 left) 2024-09-20T22:54:37.469 INFO:tasks.rados.rados.0.smithi084.stdout:3169: read oid 2 snap -1 2024-09-20T22:54:37.469 INFO:tasks.rados.rados.0.smithi084.stdout:3169: expect (ObjNum 918 snap 302 seq_num 918) 2024-09-20T22:54:37.469 INFO:tasks.rados.rados.0.smithi084.stdout:3170: rollback oid 46 current snap is 363 2024-09-20T22:54:37.470 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 46 to 352 2024-09-20T22:54:37.470 INFO:tasks.rados.rados.0.smithi084.stdout:3171: delete oid 15 current snap is 363 2024-09-20T22:54:37.470 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is unset 2024-09-20T22:54:37.470 INFO:tasks.rados.rados.0.smithi084.stdout:3169: expect (ObjNum 918 snap 302 seq_num 918) 2024-09-20T22:54:37.471 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:54:37.492 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:37.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:37.493 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:54:37.561 INFO:tasks.rados.rados.0.smithi084.stdout:3170: finishing rollback tid 0 to smithi08477203-46 2024-09-20T22:54:37.561 INFO:tasks.rados.rados.0.smithi084.stdout:3170: finishing rollback tid 1 to smithi08477203-46 2024-09-20T22:54:37.561 INFO:tasks.rados.rados.0.smithi084.stdout:3170: finishing rollback tid 2 to smithi08477203-46 2024-09-20T22:54:37.561 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 46 v 1871 (ObjNum 1046 snap 342 seq_num 1046) dirty exists 2024-09-20T22:54:37.561 INFO:tasks.rados.rados.0.smithi084.stdout:3169: done (2 left) 2024-09-20T22:54:37.561 INFO:tasks.rados.rados.0.smithi084.stdout:3170: done (1 left) 2024-09-20T22:54:37.561 INFO:tasks.rados.rados.0.smithi084.stdout:3171: done (0 left) 2024-09-20T22:54:37.561 INFO:tasks.rados.rados.0.smithi084.stdout:3172: read oid 38 snap -1 2024-09-20T22:54:37.562 INFO:tasks.rados.rados.0.smithi084.stdout:3172: expect (ObjNum 1057 snap 347 seq_num 1057) 2024-09-20T22:54:37.562 INFO:tasks.rados.rados.0.smithi084.stdout:3173: snap_remove snap 326 2024-09-20T22:54:37.589 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:37 smithi084 ceph-mon[40024]: pgmap v1095: 158 pgs: 1 active+clean+snaptrim, 1 active+remapped+backfill_toofull, 156 active+clean; 565 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 3.1 MiB/s wr, 15 op/s; 16/1368 objects misplaced (1.170%) 2024-09-20T22:54:37.589 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:37 smithi084 ceph-mon[40024]: osdmap e796: 8 total, 8 up, 8 in 2024-09-20T22:54:37.589 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:37 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/378422596' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:54:37.589 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:37 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:54:37.589 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:37 smithi084 ceph-mon[29774]: pgmap v1095: 158 pgs: 1 active+clean+snaptrim, 1 active+remapped+backfill_toofull, 156 active+clean; 565 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 3.1 MiB/s wr, 15 op/s; 16/1368 objects misplaced (1.170%) 2024-09-20T22:54:37.589 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:37 smithi084 ceph-mon[29774]: osdmap e796: 8 total, 8 up, 8 in 2024-09-20T22:54:37.589 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:37 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/378422596' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:54:37.590 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:37 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:54:37.594 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:54:37.616 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:37.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:37.617 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:54:37.717 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:54:37.738 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:37.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:37.739 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:54:37.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:37 smithi154 ceph-mon[33264]: pgmap v1095: 158 pgs: 1 active+clean+snaptrim, 1 active+remapped+backfill_toofull, 156 active+clean; 565 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 3.1 MiB/s wr, 15 op/s; 16/1368 objects misplaced (1.170%) 2024-09-20T22:54:37.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:37 smithi154 ceph-mon[33264]: osdmap e796: 8 total, 8 up, 8 in 2024-09-20T22:54:37.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:37 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/378422596' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:54:37.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:37 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:54:37.840 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:54:37.860 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:37.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:37.861 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:54:37.962 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:54:37.986 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:37.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:37.986 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:54:38.087 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:54:38.111 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:38.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:38.112 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:54:38.213 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:54:38.236 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:38.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:38.237 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:54:38.321 INFO:tasks.rados.rados.0.smithi084.stdout:3173: done (1 left) 2024-09-20T22:54:38.321 INFO:tasks.rados.rados.0.smithi084.stdout:3174: snap_remove snap 351 2024-09-20T22:54:38.337 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:54:38.360 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:38.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:38.361 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:54:38.462 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:54:38.483 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:38.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:38.484 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:54:38.584 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:54:38.606 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:38.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:38.607 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:54:38.614 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:54:38 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: ::ffff:172.21.15.154 - - [20/Sep/2024:22:54:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-20T22:54:38.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:38 smithi084 ceph-mon[40024]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:54:38.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:38 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T22:54:38.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:38 smithi084 ceph-mon[40024]: osdmap e797: 8 total, 8 up, 8 in 2024-09-20T22:54:38.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:38 smithi084 ceph-mon[40024]: 7.b scrub starts 2024-09-20T22:54:38.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:38 smithi084 ceph-mon[40024]: 7.b scrub ok 2024-09-20T22:54:38.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:38 smithi084 ceph-mon[40024]: 3.10 scrub starts 2024-09-20T22:54:38.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:38 smithi084 ceph-mon[40024]: 3.10 scrub ok 2024-09-20T22:54:38.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:38 smithi084 ceph-mon[40024]: 7.1 scrub starts 2024-09-20T22:54:38.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:38 smithi084 ceph-mon[40024]: 7.1 scrub ok 2024-09-20T22:54:38.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:38 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:54:38.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:38 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:54:38.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:38 smithi084 ceph-mon[40024]: osdmap e798: 8 total, 8 up, 8 in 2024-09-20T22:54:38.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:38 smithi084 ceph-mon[29774]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:54:38.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:38 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T22:54:38.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:38 smithi084 ceph-mon[29774]: osdmap e797: 8 total, 8 up, 8 in 2024-09-20T22:54:38.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:38 smithi084 ceph-mon[29774]: 7.b scrub starts 2024-09-20T22:54:38.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:38 smithi084 ceph-mon[29774]: 7.b scrub ok 2024-09-20T22:54:38.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:38 smithi084 ceph-mon[29774]: 3.10 scrub starts 2024-09-20T22:54:38.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:38 smithi084 ceph-mon[29774]: 3.10 scrub ok 2024-09-20T22:54:38.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:38 smithi084 ceph-mon[29774]: 7.1 scrub starts 2024-09-20T22:54:38.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:38 smithi084 ceph-mon[29774]: 7.1 scrub ok 2024-09-20T22:54:38.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:38 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:54:38.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:38 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:54:38.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:38 smithi084 ceph-mon[29774]: osdmap e798: 8 total, 8 up, 8 in 2024-09-20T22:54:38.707 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:54:38.728 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:38.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:38.729 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:54:38.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:38 smithi154 ceph-mon[33264]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:54:38.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:38 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T22:54:38.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:38 smithi154 ceph-mon[33264]: osdmap e797: 8 total, 8 up, 8 in 2024-09-20T22:54:38.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:38 smithi154 ceph-mon[33264]: 7.b scrub starts 2024-09-20T22:54:38.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:38 smithi154 ceph-mon[33264]: 7.b scrub ok 2024-09-20T22:54:38.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:38 smithi154 ceph-mon[33264]: 3.10 scrub starts 2024-09-20T22:54:38.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:38 smithi154 ceph-mon[33264]: 3.10 scrub ok 2024-09-20T22:54:38.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:38 smithi154 ceph-mon[33264]: 7.1 scrub starts 2024-09-20T22:54:38.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:38 smithi154 ceph-mon[33264]: 7.1 scrub ok 2024-09-20T22:54:38.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:38 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:54:38.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:38 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:54:38.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:38 smithi154 ceph-mon[33264]: osdmap e798: 8 total, 8 up, 8 in 2024-09-20T22:54:38.830 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:54:38.855 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:38.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:38.856 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:54:38.957 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:54:38.981 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:38.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:38.982 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:54:39.082 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:54:39.104 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:39.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:39.105 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:54:39.123 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-20T22:54:39.159 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:54:39.206 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:54:39.227 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:39.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:39.228 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:54:39.322 INFO:tasks.rados.rados.0.smithi084.stdout:3174: done (1 left) 2024-09-20T22:54:39.322 INFO:tasks.rados.rados.0.smithi084.stdout:3175: write (excl) oid 50 current snap is 363 2024-09-20T22:54:39.323 INFO:tasks.rados.rados.0.smithi084.stdout:3175: seq_num 1084 ranges {618788=492042,1883069=335719} 2024-09-20T22:54:39.329 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:54:39.330 INFO:tasks.rados.rados.0.smithi084.stdout:3175: writing smithi08477203-50 from 618788 to 1110830 tid 1 2024-09-20T22:54:39.337 INFO:tasks.rados.rados.0.smithi084.stdout:3175: writing smithi08477203-50 from 1883069 to 2218788 tid 2 2024-09-20T22:54:39.337 INFO:tasks.rados.rados.0.smithi084.stdout:3176: delete oid 48 current snap is 363 2024-09-20T22:54:39.337 INFO:tasks.rados.rados.0.smithi084.stdout:3172: expect (ObjNum 1057 snap 347 seq_num 1057) 2024-09-20T22:54:39.354 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:39.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:39.355 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:54:39.456 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:54:39.479 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:39.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:39.480 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:54:39.571 INFO:tasks.rados.rados.0.smithi084.stdout:3175: finishing write tid 1 to smithi08477203-50 2024-09-20T22:54:39.571 INFO:tasks.rados.rados.0.smithi084.stdout:3175: finishing write tid 2 to smithi08477203-50 2024-09-20T22:54:39.572 INFO:tasks.rados.rados.0.smithi084.stdout:3175: finishing write tid 3 to smithi08477203-50 2024-09-20T22:54:39.572 INFO:tasks.rados.rados.0.smithi084.stdout:3175: finishing write tid 4 to smithi08477203-50 2024-09-20T22:54:39.572 INFO:tasks.rados.rados.0.smithi084.stdout:3175: oid 50 updating version 0 to 1880 2024-09-20T22:54:39.572 INFO:tasks.rados.rados.0.smithi084.stdout:3175: oid 50 version 1880 is already newer than 1879 2024-09-20T22:54:39.572 INFO:tasks.rados.rados.0.smithi084.stdout:3175: oid 50 version 1880 is already newer than 1878 2024-09-20T22:54:39.572 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 50 v 1880 (ObjNum 1084 snap 363 seq_num 1084) dirty exists 2024-09-20T22:54:39.572 INFO:tasks.rados.rados.0.smithi084.stdout:3175: left oid 50 (ObjNum 1084 snap 363 seq_num 1084) 2024-09-20T22:54:39.572 INFO:tasks.rados.rados.0.smithi084.stdout:3172: done (2 left) 2024-09-20T22:54:39.572 INFO:tasks.rados.rados.0.smithi084.stdout:3175: done (1 left) 2024-09-20T22:54:39.572 INFO:tasks.rados.rados.0.smithi084.stdout:3176: done (0 left) 2024-09-20T22:54:39.572 INFO:tasks.rados.rados.0.smithi084.stdout:3177: read oid 3 snap -1 2024-09-20T22:54:39.572 INFO:tasks.rados.rados.0.smithi084.stdout:3177: expect (ObjNum 1053 snap 345 seq_num 1053) 2024-09-20T22:54:39.572 INFO:tasks.rados.rados.0.smithi084.stdout:3178: write (excl) oid 36 current snap is 363 2024-09-20T22:54:39.573 INFO:tasks.rados.rados.0.smithi084.stdout:3178: seq_num 1086 ranges {652402=642540,1883680=368722} 2024-09-20T22:54:39.578 INFO:tasks.rados.rados.0.smithi084.stdout:3178: writing smithi08477203-36 from 652402 to 1294942 tid 1 2024-09-20T22:54:39.580 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:54:39.584 INFO:tasks.rados.rados.0.smithi084.stdout:3178: writing smithi08477203-36 from 1883680 to 2252402 tid 2 2024-09-20T22:54:39.584 INFO:tasks.rados.rados.0.smithi084.stdout:3179: read oid 30 snap -1 2024-09-20T22:54:39.584 INFO:tasks.rados.rados.0.smithi084.stdout:3179: expect (ObjNum 787 snap 254 seq_num 787) 2024-09-20T22:54:39.584 INFO:tasks.rados.rados.0.smithi084.stdout:3180: copy_from oid 18 from oid 45 current snap is 363 2024-09-20T22:54:39.584 INFO:tasks.rados.rados.0.smithi084.stdout:3177: expect (ObjNum 1053 snap 345 seq_num 1053) 2024-09-20T22:54:39.605 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:39.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:39.606 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:54:39.687 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:39 smithi084 ceph-mon[40024]: pgmap v1098: 158 pgs: 1 active+clean+snaptrim, 1 active+remapped+backfill_toofull, 156 active+clean; 565 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 16/1368 objects misplaced (1.170%) 2024-09-20T22:54:39.687 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:39 smithi084 ceph-mon[40024]: osdmap e799: 8 total, 8 up, 8 in 2024-09-20T22:54:39.687 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:39 smithi084 ceph-mon[29774]: pgmap v1098: 158 pgs: 1 active+clean+snaptrim, 1 active+remapped+backfill_toofull, 156 active+clean; 565 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 16/1368 objects misplaced (1.170%) 2024-09-20T22:54:39.687 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:39 smithi084 ceph-mon[29774]: osdmap e799: 8 total, 8 up, 8 in 2024-09-20T22:54:39.691 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:54:39.692 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:54:39.704 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:54:39.704 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:54:39.707 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:54:39.713 INFO:tasks.rados.rados.0.smithi084.stdout:3178: finishing write tid 1 to smithi08477203-36 2024-09-20T22:54:39.713 INFO:tasks.rados.rados.0.smithi084.stdout:3177: done (3 left) 2024-09-20T22:54:39.713 INFO:tasks.rados.rados.0.smithi084.stdout:3181: delete oid 6 current snap is 363 2024-09-20T22:54:39.713 INFO:tasks.rados.rados.0.smithi084.stdout:3178: finishing write tid 2 to smithi08477203-36 2024-09-20T22:54:39.714 INFO:tasks.rados.rados.0.smithi084.stdout:3178: finishing write tid 3 to smithi08477203-36 2024-09-20T22:54:39.714 INFO:tasks.rados.rados.0.smithi084.stdout:3178: finishing write tid 4 to smithi08477203-36 2024-09-20T22:54:39.714 INFO:tasks.rados.rados.0.smithi084.stdout:3178: oid 36 updating version 0 to 1911 2024-09-20T22:54:39.714 INFO:tasks.rados.rados.0.smithi084.stdout:3178: oid 36 updating version 1911 to 1912 2024-09-20T22:54:39.714 INFO:tasks.rados.rados.0.smithi084.stdout:3178: oid 36 version 1912 is already newer than 1910 2024-09-20T22:54:39.714 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 36 v 1912 (ObjNum 1086 snap 363 seq_num 1086) dirty exists 2024-09-20T22:54:39.714 INFO:tasks.rados.rados.0.smithi084.stdout:3178: left oid 36 (ObjNum 1086 snap 363 seq_num 1086) 2024-09-20T22:54:39.714 INFO:tasks.rados.rados.0.smithi084.stdout:3179: expect (ObjNum 787 snap 254 seq_num 787) 2024-09-20T22:54:39.721 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:54:39.722 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:54:39.729 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:39.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:39.730 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:54:39.737 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:54:39.738 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-20T22:54:39.739 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T22:54:39.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:39 smithi154 ceph-mon[33264]: pgmap v1098: 158 pgs: 1 active+clean+snaptrim, 1 active+remapped+backfill_toofull, 156 active+clean; 565 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 16/1368 objects misplaced (1.170%) 2024-09-20T22:54:39.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:39 smithi154 ceph-mon[33264]: osdmap e799: 8 total, 8 up, 8 in 2024-09-20T22:54:39.754 INFO:journalctl@ceph.iscsi.iscsi.a.smithi154.stdout:Sep 20 22:54:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-iscsi-iscsi-a[59730]: debug there is no tcmu-runner data available 2024-09-20T22:54:39.756 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:54:39.756 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:54:39.781 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:54:39.781 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:54:39.798 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:54:39.798 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:54:39.814 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:54:39.814 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:54:39.830 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:54:39.853 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:39.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:39.854 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:54:39.882 INFO:tasks.rados.rados.0.smithi084.stdout:3180: finishing copy_from racing read to smithi08477203-18 2024-09-20T22:54:39.883 INFO:tasks.rados.rados.0.smithi084.stdout:3178: done (3 left) 2024-09-20T22:54:39.883 INFO:tasks.rados.rados.0.smithi084.stdout:3179: done (2 left) 2024-09-20T22:54:39.883 INFO:tasks.rados.rados.0.smithi084.stdout:3181: done (1 left) 2024-09-20T22:54:39.883 INFO:tasks.rados.rados.0.smithi084.stdout:3182: read oid 30 snap -1 2024-09-20T22:54:39.883 INFO:tasks.rados.rados.0.smithi084.stdout:3182: expect (ObjNum 787 snap 254 seq_num 787) 2024-09-20T22:54:39.883 INFO:tasks.rados.rados.0.smithi084.stdout:3183: copy_from oid 15 from oid 29 current snap is 363 2024-09-20T22:54:39.883 INFO:tasks.rados.rados.0.smithi084.stdout:3180: finishing copy_from to smithi08477203-18 2024-09-20T22:54:39.883 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 18 v 1861 (ObjNum 1025 snap 333 seq_num 1025) dirty exists 2024-09-20T22:54:39.883 INFO:tasks.rados.rados.0.smithi084.stdout:3180: done (2 left) 2024-09-20T22:54:39.884 INFO:tasks.rados.rados.0.smithi084.stdout:3184: write oid 46 current snap is 363 2024-09-20T22:54:39.884 INFO:tasks.rados.rados.0.smithi084.stdout:3184: seq_num 1088 ranges {686016=509391,1716997=569019} 2024-09-20T22:54:39.888 INFO:tasks.rados.rados.0.smithi084.stdout:3184: writing smithi08477203-46 from 686016 to 1195407 tid 1 2024-09-20T22:54:39.895 INFO:tasks.rados.rados.0.smithi084.stdout:3184: writing smithi08477203-46 from 1716997 to 2286016 tid 2 2024-09-20T22:54:39.895 INFO:tasks.rados.rados.0.smithi084.stdout:3185: rollback oid 16 current snap is 363 2024-09-20T22:54:39.895 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 16 to 344 2024-09-20T22:54:39.895 INFO:tasks.rados.rados.0.smithi084.stdout:3182: expect (ObjNum 787 snap 254 seq_num 787) 2024-09-20T22:54:39.955 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:54:39.976 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:39.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:39.977 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:54:40.053 INFO:tasks.rados.rados.0.smithi084.stdout:3183: finishing copy_from to smithi08477203-15 2024-09-20T22:54:40.053 INFO:tasks.rados.rados.0.smithi084.stdout:3183: got expected ENOENT (src dne) 2024-09-20T22:54:40.053 INFO:tasks.rados.rados.0.smithi084.stdout:3183: finishing copy_from racing read to smithi08477203-15 2024-09-20T22:54:40.053 INFO:tasks.rados.rados.0.smithi084.stdout:3184: finishing write tid 1 to smithi08477203-46 2024-09-20T22:54:40.053 INFO:tasks.rados.rados.0.smithi084.stdout:3184: finishing write tid 2 to smithi08477203-46 2024-09-20T22:54:40.053 INFO:tasks.rados.rados.0.smithi084.stdout:3185: finishing rollback tid 0 to smithi08477203-16 2024-09-20T22:54:40.054 INFO:tasks.rados.rados.0.smithi084.stdout:3184: finishing write tid 3 to smithi08477203-46 2024-09-20T22:54:40.054 INFO:tasks.rados.rados.0.smithi084.stdout:3185: finishing rollback tid 1 to smithi08477203-16 2024-09-20T22:54:40.054 INFO:tasks.rados.rados.0.smithi084.stdout:3185: finishing rollback tid 2 to smithi08477203-16 2024-09-20T22:54:40.054 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 16 v 1820 (ObjNum 419 snap 105 seq_num 419) dirty exists 2024-09-20T22:54:40.054 INFO:tasks.rados.rados.0.smithi084.stdout:3184: finishing write tid 4 to smithi08477203-46 2024-09-20T22:54:40.054 INFO:tasks.rados.rados.0.smithi084.stdout:3184: oid 46 updating version 0 to 1874 2024-09-20T22:54:40.054 INFO:tasks.rados.rados.0.smithi084.stdout:3184: oid 46 version 1874 is already newer than 1873 2024-09-20T22:54:40.054 INFO:tasks.rados.rados.0.smithi084.stdout:3184: oid 46 version 1874 is already newer than 1872 2024-09-20T22:54:40.054 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 46 v 1874 (ObjNum 1088 snap 363 seq_num 1088) dirty exists 2024-09-20T22:54:40.054 INFO:tasks.rados.rados.0.smithi084.stdout:3184: left oid 46 (ObjNum 1088 snap 363 seq_num 1088) 2024-09-20T22:54:40.054 INFO:tasks.rados.rados.0.smithi084.stdout:3182: done (3 left) 2024-09-20T22:54:40.054 INFO:tasks.rados.rados.0.smithi084.stdout:3183: done (2 left) 2024-09-20T22:54:40.054 INFO:tasks.rados.rados.0.smithi084.stdout:3184: done (1 left) 2024-09-20T22:54:40.054 INFO:tasks.rados.rados.0.smithi084.stdout:3185: done (0 left) 2024-09-20T22:54:40.055 INFO:tasks.rados.rados.0.smithi084.stdout:3186: rollback oid 20 current snap is 363 2024-09-20T22:54:40.055 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 20 to 349 2024-09-20T22:54:40.055 INFO:tasks.rados.rados.0.smithi084.stdout:3187: read oid 44 snap -1 2024-09-20T22:54:40.055 INFO:tasks.rados.rados.0.smithi084.stdout:3187: expect (ObjNum 1051 snap 345 seq_num 1051) 2024-09-20T22:54:40.055 INFO:tasks.rados.rados.0.smithi084.stdout:3188: snap_create 2024-09-20T22:54:40.056 INFO:tasks.rados.rados.0.smithi084.stdout:3186: finishing rollback tid 0 to smithi08477203-20 2024-09-20T22:54:40.056 INFO:tasks.rados.rados.0.smithi084.stdout:3186: finishing rollback tid 1 to smithi08477203-20 2024-09-20T22:54:40.056 INFO:tasks.rados.rados.0.smithi084.stdout:3186: finishing rollback tid 2 to smithi08477203-20 2024-09-20T22:54:40.056 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 20 v 1923 (ObjNum 1047 snap 342 seq_num 1047) dirty exists 2024-09-20T22:54:40.060 INFO:tasks.rados.rados.0.smithi084.stdout:3187: expect (ObjNum 1051 snap 345 seq_num 1051) 2024-09-20T22:54:40.078 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:54:40.101 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:40.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:40.102 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:54:40.202 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:54:40.227 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:40.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:40.228 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:54:40.259 INFO:tasks.thrashosds.thrasher:in_osds: [2, 3, 5, 6, 7, 0, 4, 1] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-20T22:54:40.259 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-20T22:54:40.259 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-09-20T22:54:40.259 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd dump -f json-pretty 2024-09-20T22:54:40.328 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:54:40.350 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:40.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:40.351 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:54:40.395 INFO:tasks.rados.rados.0.smithi084.stdout:3186: done (2 left) 2024-09-20T22:54:40.395 INFO:tasks.rados.rados.0.smithi084.stdout:3187: done (1 left) 2024-09-20T22:54:40.395 INFO:tasks.rados.rados.0.smithi084.stdout:3188: done (0 left) 2024-09-20T22:54:40.395 INFO:tasks.rados.rados.0.smithi084.stdout:3189: delete oid 15 current snap is 364 2024-09-20T22:54:40.398 INFO:tasks.rados.rados.0.smithi084.stdout:3189: done (0 left) 2024-09-20T22:54:40.398 INFO:tasks.rados.rados.0.smithi084.stdout:3190: snap_create 2024-09-20T22:54:40.452 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:54:40.475 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:40.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:40.476 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:54:40.494 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:40 smithi084 ceph-mon[29774]: 6.9 scrub starts 2024-09-20T22:54:40.494 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:40 smithi084 ceph-mon[29774]: 6.9 scrub ok 2024-09-20T22:54:40.495 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:40 smithi084 ceph-mon[29774]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:54:40.495 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:40 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/1455754297' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:54:40.495 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:40 smithi084 ceph-mon[29774]: 4.2 scrub starts 2024-09-20T22:54:40.495 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:40 smithi084 ceph-mon[29774]: 4.2 scrub ok 2024-09-20T22:54:40.495 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:40 smithi084 ceph-mon[40024]: 6.9 scrub starts 2024-09-20T22:54:40.495 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:40 smithi084 ceph-mon[40024]: 6.9 scrub ok 2024-09-20T22:54:40.495 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:40 smithi084 ceph-mon[40024]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:54:40.495 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:40 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/1455754297' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:54:40.495 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:40 smithi084 ceph-mon[40024]: 4.2 scrub starts 2024-09-20T22:54:40.496 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:40 smithi084 ceph-mon[40024]: 4.2 scrub ok 2024-09-20T22:54:40.496 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-20T22:54:40.496 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:54:40.577 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:54:40.598 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:40.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:40.599 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:54:40.655 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:40 smithi154 ceph-mon[33264]: 6.9 scrub starts 2024-09-20T22:54:40.656 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:40 smithi154 ceph-mon[33264]: 6.9 scrub ok 2024-09-20T22:54:40.656 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:40 smithi154 ceph-mon[33264]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:54:40.656 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:40 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/1455754297' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:54:40.656 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:40 smithi154 ceph-mon[33264]: 4.2 scrub starts 2024-09-20T22:54:40.656 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:40 smithi154 ceph-mon[33264]: 4.2 scrub ok 2024-09-20T22:54:40.679 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-20T22:54:40.700 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:54:40.724 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:40.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:40.725 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:54:40.826 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:54:40.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:40.849 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:40.849 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:54:40.950 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:54:40.971 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:40.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:40.972 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:54:41.073 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:54:41.094 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:41.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:41.095 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:54:41.196 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:54:41.206 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.2 dump_blocked_ops 2024-09-20T22:54:41.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:41.217 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:41.217 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:54:41.318 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:54:41.341 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:41.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:41.342 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:54:41.398 INFO:tasks.rados.rados.0.smithi084.stdout:3190: done (0 left) 2024-09-20T22:54:41.398 INFO:tasks.rados.rados.0.smithi084.stdout:3191: rollback oid 42 current snap is 365 2024-09-20T22:54:41.398 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 42 to 312 2024-09-20T22:54:41.398 INFO:tasks.rados.rados.0.smithi084.stdout:3192: rollback oid 4 current snap is 365 2024-09-20T22:54:41.398 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 4 to 328 2024-09-20T22:54:41.398 INFO:tasks.rados.rados.0.smithi084.stdout:3193: snap_create 2024-09-20T22:54:41.405 INFO:tasks.rados.rados.0.smithi084.stdout:3192: finishing rollback tid 0 to smithi08477203-4 2024-09-20T22:54:41.405 INFO:tasks.rados.rados.0.smithi084.stdout:3192: finishing rollback tid 1 to smithi08477203-4 2024-09-20T22:54:41.405 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 4 v 2565 (ObjNum 129 snap 0 seq_num 2091323944) dirty dne 2024-09-20T22:54:41.405 INFO:tasks.rados.rados.0.smithi084.stdout:3191: finishing rollback tid 1 to smithi08477203-42 2024-09-20T22:54:41.405 INFO:tasks.rados.rados.0.smithi084.stdout:3191: finishing rollback tid 2 to smithi08477203-42 2024-09-20T22:54:41.405 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 42 v 1914 (ObjNum 625 snap 186 seq_num 625) dirty exists 2024-09-20T22:54:41.443 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:54:41.464 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:41.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:41.465 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:54:41.565 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:54:41.586 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:41.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:41.587 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:54:41.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:41 smithi084 ceph-mon[29774]: pgmap v1101: 158 pgs: 5 active+clean+snaptrim, 1 active+remapped+backfill_toofull, 152 active+clean; 569 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 6.6 MiB/s rd, 0 B/s wr, 5 op/s; 16/1371 objects misplaced (1.167%) 2024-09-20T22:54:41.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:41 smithi084 ceph-mon[29774]: osdmap e800: 8 total, 8 up, 8 in 2024-09-20T22:54:41.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:41 smithi084 ceph-mon[40024]: pgmap v1101: 158 pgs: 5 active+clean+snaptrim, 1 active+remapped+backfill_toofull, 152 active+clean; 569 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 6.6 MiB/s rd, 0 B/s wr, 5 op/s; 16/1371 objects misplaced (1.167%) 2024-09-20T22:54:41.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:41 smithi084 ceph-mon[40024]: osdmap e800: 8 total, 8 up, 8 in 2024-09-20T22:54:41.688 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:54:41.710 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:41.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:41.711 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:54:41.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:41 smithi154 ceph-mon[33264]: pgmap v1101: 158 pgs: 5 active+clean+snaptrim, 1 active+remapped+backfill_toofull, 152 active+clean; 569 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 6.6 MiB/s rd, 0 B/s wr, 5 op/s; 16/1371 objects misplaced (1.167%) 2024-09-20T22:54:41.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:41 smithi154 ceph-mon[33264]: osdmap e800: 8 total, 8 up, 8 in 2024-09-20T22:54:41.783 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:54:41.784 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:54:41.812 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:54:41.815 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:54:41.833 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:41.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:41.834 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:54:41.935 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:54:41.956 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:41.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:41.957 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:54:42.058 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:54:42.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:42.080 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:42.080 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:54:42.181 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:54:42.202 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:42.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:42.203 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:54:42.304 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:54:42.324 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:42.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:42.325 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:54:42.400 INFO:tasks.rados.rados.0.smithi084.stdout:3191: done (2 left) 2024-09-20T22:54:42.401 INFO:tasks.rados.rados.0.smithi084.stdout:3192: done (1 left) 2024-09-20T22:54:42.401 INFO:tasks.rados.rados.0.smithi084.stdout:3193: done (0 left) 2024-09-20T22:54:42.401 INFO:tasks.rados.rados.0.smithi084.stdout:3194: rollback oid 9 current snap is 366 2024-09-20T22:54:42.401 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 9 to 342 2024-09-20T22:54:42.401 INFO:tasks.rados.rados.0.smithi084.stdout:3195: delete oid 23 current snap is 366 2024-09-20T22:54:42.407 INFO:tasks.rados.rados.0.smithi084.stdout:3195: done (1 left) 2024-09-20T22:54:42.407 INFO:tasks.rados.rados.0.smithi084.stdout:3196: read oid 1 snap -1 2024-09-20T22:54:42.407 INFO:tasks.rados.rados.0.smithi084.stdout:3196: expect (ObjNum 738 snap 235 seq_num 738) 2024-09-20T22:54:42.407 INFO:tasks.rados.rados.0.smithi084.stdout:3197: read oid 33 snap -1 2024-09-20T22:54:42.407 INFO:tasks.rados.rados.0.smithi084.stdout:3197: expect (ObjNum 984 snap 319 seq_num 984) 2024-09-20T22:54:42.407 INFO:tasks.rados.rados.0.smithi084.stdout:3198: delete oid 5 current snap is 366 2024-09-20T22:54:42.412 INFO:tasks.rados.rados.0.smithi084.stdout:3197: expect (ObjNum 984 snap 319 seq_num 984) 2024-09-20T22:54:42.426 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:54:42.447 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:42.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:42.448 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:54:42.497 INFO:tasks.rados.rados.0.smithi084.stdout:3196: expect (ObjNum 738 snap 235 seq_num 738) 2024-09-20T22:54:42.549 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:54:42.571 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:42.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:42.572 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:54:42.578 INFO:tasks.rados.rados.0.smithi084.stdout:3194: finishing rollback tid 1 to smithi08477203-9 2024-09-20T22:54:42.578 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 9 v 0 (ObjNum 289 snap 0 seq_num 0) dirty dne 2024-09-20T22:54:42.578 INFO:tasks.rados.rados.0.smithi084.stdout:3194: done (3 left) 2024-09-20T22:54:42.579 INFO:tasks.rados.rados.0.smithi084.stdout:3196: done (2 left) 2024-09-20T22:54:42.579 INFO:tasks.rados.rados.0.smithi084.stdout:3197: done (1 left) 2024-09-20T22:54:42.579 INFO:tasks.rados.rados.0.smithi084.stdout:3198: done (0 left) 2024-09-20T22:54:42.579 INFO:tasks.rados.rados.0.smithi084.stdout:3199: rollback oid 24 current snap is 366 2024-09-20T22:54:42.579 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 24 to 332 2024-09-20T22:54:42.579 INFO:tasks.rados.rados.0.smithi084.stdout:3200: snap_remove snap 345 2024-09-20T22:54:42.612 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:42 smithi084 ceph-mon[29774]: 3.1f scrub starts 2024-09-20T22:54:42.612 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:42 smithi084 ceph-mon[29774]: 3.1f scrub ok 2024-09-20T22:54:42.612 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:42 smithi084 ceph-mon[29774]: 4.8 scrub starts 2024-09-20T22:54:42.612 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:42 smithi084 ceph-mon[29774]: 4.8 scrub ok 2024-09-20T22:54:42.612 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:42 smithi084 ceph-mon[29774]: osdmap e801: 8 total, 8 up, 8 in 2024-09-20T22:54:42.612 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:42 smithi084 ceph-mon[29774]: 6.f scrub starts 2024-09-20T22:54:42.613 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:42 smithi084 ceph-mon[29774]: 6.f scrub ok 2024-09-20T22:54:42.613 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:42 smithi084 ceph-mon[29774]: 5.1d scrub starts 2024-09-20T22:54:42.613 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:42 smithi084 ceph-mon[29774]: 5.1d scrub ok 2024-09-20T22:54:42.613 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:42 smithi084 ceph-mon[29774]: 4.5 scrub starts 2024-09-20T22:54:42.613 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:42 smithi084 ceph-mon[29774]: 4.5 scrub ok 2024-09-20T22:54:42.613 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:42 smithi084 ceph-mon[40024]: 3.1f scrub starts 2024-09-20T22:54:42.613 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:42 smithi084 ceph-mon[40024]: 3.1f scrub ok 2024-09-20T22:54:42.613 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:42 smithi084 ceph-mon[40024]: 4.8 scrub starts 2024-09-20T22:54:42.613 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:42 smithi084 ceph-mon[40024]: 4.8 scrub ok 2024-09-20T22:54:42.613 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:42 smithi084 ceph-mon[40024]: osdmap e801: 8 total, 8 up, 8 in 2024-09-20T22:54:42.613 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:42 smithi084 ceph-mon[40024]: 6.f scrub starts 2024-09-20T22:54:42.613 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:42 smithi084 ceph-mon[40024]: 6.f scrub ok 2024-09-20T22:54:42.613 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:42 smithi084 ceph-mon[40024]: 5.1d scrub starts 2024-09-20T22:54:42.614 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:42 smithi084 ceph-mon[40024]: 5.1d scrub ok 2024-09-20T22:54:42.614 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:42 smithi084 ceph-mon[40024]: 4.5 scrub starts 2024-09-20T22:54:42.614 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:42 smithi084 ceph-mon[40024]: 4.5 scrub ok 2024-09-20T22:54:42.672 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:54:42.694 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:42.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:42.696 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:54:42.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:42 smithi154 ceph-mon[33264]: 3.1f scrub starts 2024-09-20T22:54:42.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:42 smithi154 ceph-mon[33264]: 3.1f scrub ok 2024-09-20T22:54:42.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:42 smithi154 ceph-mon[33264]: 4.8 scrub starts 2024-09-20T22:54:42.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:42 smithi154 ceph-mon[33264]: 4.8 scrub ok 2024-09-20T22:54:42.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:42 smithi154 ceph-mon[33264]: osdmap e801: 8 total, 8 up, 8 in 2024-09-20T22:54:42.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:42 smithi154 ceph-mon[33264]: 6.f scrub starts 2024-09-20T22:54:42.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:42 smithi154 ceph-mon[33264]: 6.f scrub ok 2024-09-20T22:54:42.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:42 smithi154 ceph-mon[33264]: 5.1d scrub starts 2024-09-20T22:54:42.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:42 smithi154 ceph-mon[33264]: 5.1d scrub ok 2024-09-20T22:54:42.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:42 smithi154 ceph-mon[33264]: 4.5 scrub starts 2024-09-20T22:54:42.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:42 smithi154 ceph-mon[33264]: 4.5 scrub ok 2024-09-20T22:54:42.796 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:54:42.818 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:42.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:42.819 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:54:42.920 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:54:42.942 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:42.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:42.943 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:54:43.044 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:54:43.067 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:43.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:43.068 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:54:43.077 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-20T22:54:43.077 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:54:43.099 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-20T22:54:43.099 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-20T22:54:43.099 INFO:teuthology.orchestra.run.smithi084.stdout: "epoch": 802, 2024-09-20T22:54:43.099 INFO:teuthology.orchestra.run.smithi084.stdout: "fsid": "a9c0bc70-77a0-11ef-baf5-efdc52797490", 2024-09-20T22:54:43.099 INFO:teuthology.orchestra.run.smithi084.stdout: "created": "2024-09-20T22:36:30.240626+0000", 2024-09-20T22:54:43.099 INFO:teuthology.orchestra.run.smithi084.stdout: "modified": "2024-09-20T22:54:42.393575+0000", 2024-09-20T22:54:43.099 INFO:teuthology.orchestra.run.smithi084.stdout: "last_up_change": "2024-09-20T22:41:27.098565+0000", 2024-09-20T22:54:43.099 INFO:teuthology.orchestra.run.smithi084.stdout: "last_in_change": "2024-09-20T22:54:16.306377+0000", 2024-09-20T22:54:43.099 INFO:teuthology.orchestra.run.smithi084.stdout: "flags": "nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-20T22:54:43.100 INFO:teuthology.orchestra.run.smithi084.stdout: "flags_num": 5804032, 2024-09-20T22:54:43.100 INFO:teuthology.orchestra.run.smithi084.stdout: "flags_set": [ 2024-09-20T22:54:43.100 INFO:teuthology.orchestra.run.smithi084.stdout: "nodeep-scrub", 2024-09-20T22:54:43.100 INFO:teuthology.orchestra.run.smithi084.stdout: "pglog_hardlimit", 2024-09-20T22:54:43.100 INFO:teuthology.orchestra.run.smithi084.stdout: "purged_snapdirs", 2024-09-20T22:54:43.100 INFO:teuthology.orchestra.run.smithi084.stdout: "recovery_deletes", 2024-09-20T22:54:43.100 INFO:teuthology.orchestra.run.smithi084.stdout: "sortbitwise" 2024-09-20T22:54:43.100 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2024-09-20T22:54:43.100 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_version": 18, 2024-09-20T22:54:43.100 INFO:teuthology.orchestra.run.smithi084.stdout: "full_ratio": 0.94999998807907104, 2024-09-20T22:54:43.100 INFO:teuthology.orchestra.run.smithi084.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-20T22:54:43.100 INFO:teuthology.orchestra.run.smithi084.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-20T22:54:43.100 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_snapshot": "", 2024-09-20T22:54:43.100 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_max": 7, 2024-09-20T22:54:43.101 INFO:teuthology.orchestra.run.smithi084.stdout: "max_osd": 8, 2024-09-20T22:54:43.101 INFO:teuthology.orchestra.run.smithi084.stdout: "require_min_compat_client": "luminous", 2024-09-20T22:54:43.101 INFO:teuthology.orchestra.run.smithi084.stdout: "min_compat_client": "luminous", 2024-09-20T22:54:43.101 INFO:teuthology.orchestra.run.smithi084.stdout: "require_osd_release": "squid", 2024-09-20T22:54:43.101 INFO:teuthology.orchestra.run.smithi084.stdout: "allow_crimson": false, 2024-09-20T22:54:43.101 INFO:teuthology.orchestra.run.smithi084.stdout: "pools": [ 2024-09-20T22:54:43.101 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:43.101 INFO:teuthology.orchestra.run.smithi084.stdout: "pool": 1, 2024-09-20T22:54:43.101 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_name": ".mgr", 2024-09-20T22:54:43.101 INFO:teuthology.orchestra.run.smithi084.stdout: "create_time": "2024-09-20T22:39:21.984381+0000", 2024-09-20T22:54:43.101 INFO:teuthology.orchestra.run.smithi084.stdout: "flags": 1, 2024-09-20T22:54:43.101 INFO:teuthology.orchestra.run.smithi084.stdout: "flags_names": "hashpspool", 2024-09-20T22:54:43.101 INFO:teuthology.orchestra.run.smithi084.stdout: "type": 1, 2024-09-20T22:54:43.101 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 3, 2024-09-20T22:54:43.101 INFO:teuthology.orchestra.run.smithi084.stdout: "min_size": 2, 2024-09-20T22:54:43.102 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_rule": 0, 2024-09-20T22:54:43.102 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_count": 0, 2024-09-20T22:54:43.102 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_target": 0, 2024-09-20T22:54:43.102 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_barrier": 0, 2024-09-20T22:54:43.102 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-20T22:54:43.102 INFO:teuthology.orchestra.run.smithi084.stdout: "is_stretch_pool": false, 2024-09-20T22:54:43.102 INFO:teuthology.orchestra.run.smithi084.stdout: "object_hash": 2, 2024-09-20T22:54:43.102 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_autoscale_mode": "off", 2024-09-20T22:54:43.102 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num": 11, 2024-09-20T22:54:43.102 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num": 11, 2024-09-20T22:54:43.102 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num_target": 11, 2024-09-20T22:54:43.102 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_target": 11, 2024-09-20T22:54:43.102 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_pending": 11, 2024-09-20T22:54:43.102 INFO:teuthology.orchestra.run.smithi084.stdout: "last_pg_merge_meta": { 2024-09-20T22:54:43.103 INFO:teuthology.orchestra.run.smithi084.stdout: "source_pgid": "1.b", 2024-09-20T22:54:43.103 INFO:teuthology.orchestra.run.smithi084.stdout: "ready_epoch": 299, 2024-09-20T22:54:43.103 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_started": 298, 2024-09-20T22:54:43.103 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_clean": 298, 2024-09-20T22:54:43.103 INFO:teuthology.orchestra.run.smithi084.stdout: "source_version": "77'128", 2024-09-20T22:54:43.103 INFO:teuthology.orchestra.run.smithi084.stdout: "target_version": "77'128" 2024-09-20T22:54:43.103 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:43.103 INFO:teuthology.orchestra.run.smithi084.stdout: "last_change": "769", 2024-09-20T22:54:43.103 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend": "0", 2024-09-20T22:54:43.103 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_prenautilus": "299", 2024-09-20T22:54:43.103 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_preluminous": "297", 2024-09-20T22:54:43.103 INFO:teuthology.orchestra.run.smithi084.stdout: "auid": 0, 2024-09-20T22:54:43.103 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_mode": "selfmanaged", 2024-09-20T22:54:43.103 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_seq": 0, 2024-09-20T22:54:43.104 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_epoch": 0, 2024-09-20T22:54:43.104 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_snaps": [], 2024-09-20T22:54:43.104 INFO:teuthology.orchestra.run.smithi084.stdout: "removed_snaps": "[]", 2024-09-20T22:54:43.104 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_bytes": 0, 2024-09-20T22:54:43.104 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_objects": 0, 2024-09-20T22:54:43.104 INFO:teuthology.orchestra.run.smithi084.stdout: "tiers": [], 2024-09-20T22:54:43.104 INFO:teuthology.orchestra.run.smithi084.stdout: "tier_of": -1, 2024-09-20T22:54:43.104 INFO:teuthology.orchestra.run.smithi084.stdout: "read_tier": -1, 2024-09-20T22:54:43.104 INFO:teuthology.orchestra.run.smithi084.stdout: "write_tier": -1, 2024-09-20T22:54:43.104 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_mode": "none", 2024-09-20T22:54:43.104 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_bytes": 0, 2024-09-20T22:54:43.104 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_objects": 0, 2024-09-20T22:54:43.104 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-20T22:54:43.104 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-20T22:54:43.104 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-20T22:54:43.105 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_flush_age": 0, 2024-09-20T22:54:43.105 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_evict_age": 0, 2024-09-20T22:54:43.105 INFO:teuthology.orchestra.run.smithi084.stdout: "erasure_code_profile": "", 2024-09-20T22:54:43.105 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_params": { 2024-09-20T22:54:43.105 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "none" 2024-09-20T22:54:43.105 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:43.105 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_period": 0, 2024-09-20T22:54:43.105 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_count": 0, 2024-09-20T22:54:43.105 INFO:teuthology.orchestra.run.smithi084.stdout: "use_gmt_hitset": true, 2024-09-20T22:54:43.105 INFO:teuthology.orchestra.run.smithi084.stdout: "min_read_recency_for_promote": 0, 2024-09-20T22:54:43.105 INFO:teuthology.orchestra.run.smithi084.stdout: "min_write_recency_for_promote": 0, 2024-09-20T22:54:43.105 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_grade_decay_rate": 0, 2024-09-20T22:54:43.105 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_search_last_n": 0, 2024-09-20T22:54:43.105 INFO:teuthology.orchestra.run.smithi084.stdout: "grade_table": [], 2024-09-20T22:54:43.106 INFO:teuthology.orchestra.run.smithi084.stdout: "stripe_width": 0, 2024-09-20T22:54:43.106 INFO:teuthology.orchestra.run.smithi084.stdout: "expected_num_objects": 0, 2024-09-20T22:54:43.106 INFO:teuthology.orchestra.run.smithi084.stdout: "fast_read": false, 2024-09-20T22:54:43.106 INFO:teuthology.orchestra.run.smithi084.stdout: "options": { 2024-09-20T22:54:43.106 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_max": 32, 2024-09-20T22:54:43.106 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_min": 1 2024-09-20T22:54:43.106 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:43.106 INFO:teuthology.orchestra.run.smithi084.stdout: "application_metadata": { 2024-09-20T22:54:43.106 INFO:teuthology.orchestra.run.smithi084.stdout: "mgr": {} 2024-09-20T22:54:43.106 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:43.106 INFO:teuthology.orchestra.run.smithi084.stdout: "read_balance": { 2024-09-20T22:54:43.106 INFO:teuthology.orchestra.run.smithi084.stdout: "score_type": "Fair distribution", 2024-09-20T22:54:43.106 INFO:teuthology.orchestra.run.smithi084.stdout: "score_acting": 2.1800000667572021, 2024-09-20T22:54:43.106 INFO:teuthology.orchestra.run.smithi084.stdout: "score_stable": 2.1800000667572021, 2024-09-20T22:54:43.107 INFO:teuthology.orchestra.run.smithi084.stdout: "optimal_score": 1, 2024-09-20T22:54:43.107 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_acting": 2.1800000667572021, 2024-09-20T22:54:43.107 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_stable": 2.1800000667572021, 2024-09-20T22:54:43.107 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity_weighted": 1, 2024-09-20T22:54:43.107 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity": 1, 2024-09-20T22:54:43.107 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity_weighted": 1 2024-09-20T22:54:43.107 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:43.107 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:43.107 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:43.107 INFO:teuthology.orchestra.run.smithi084.stdout: "pool": 2, 2024-09-20T22:54:43.107 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_name": "datapool", 2024-09-20T22:54:43.107 INFO:teuthology.orchestra.run.smithi084.stdout: "create_time": "2024-09-20T22:41:35.820081+0000", 2024-09-20T22:54:43.107 INFO:teuthology.orchestra.run.smithi084.stdout: "flags": 8193, 2024-09-20T22:54:43.107 INFO:teuthology.orchestra.run.smithi084.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-20T22:54:43.108 INFO:teuthology.orchestra.run.smithi084.stdout: "type": 1, 2024-09-20T22:54:43.108 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 3, 2024-09-20T22:54:43.108 INFO:teuthology.orchestra.run.smithi084.stdout: "min_size": 2, 2024-09-20T22:54:43.108 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_rule": 0, 2024-09-20T22:54:43.108 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_count": 0, 2024-09-20T22:54:43.108 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_target": 0, 2024-09-20T22:54:43.108 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_barrier": 0, 2024-09-20T22:54:43.108 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-20T22:54:43.108 INFO:teuthology.orchestra.run.smithi084.stdout: "is_stretch_pool": false, 2024-09-20T22:54:43.108 INFO:teuthology.orchestra.run.smithi084.stdout: "object_hash": 2, 2024-09-20T22:54:43.108 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_autoscale_mode": "off", 2024-09-20T22:54:43.108 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num": 3, 2024-09-20T22:54:43.108 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num": 3, 2024-09-20T22:54:43.109 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num_target": 3, 2024-09-20T22:54:43.109 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_target": 3, 2024-09-20T22:54:43.109 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_pending": 3, 2024-09-20T22:54:43.109 INFO:teuthology.orchestra.run.smithi084.stdout: "last_pg_merge_meta": { 2024-09-20T22:54:43.109 INFO:teuthology.orchestra.run.smithi084.stdout: "source_pgid": "0.0", 2024-09-20T22:54:43.109 INFO:teuthology.orchestra.run.smithi084.stdout: "ready_epoch": 0, 2024-09-20T22:54:43.109 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_started": 0, 2024-09-20T22:54:43.109 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_clean": 0, 2024-09-20T22:54:43.109 INFO:teuthology.orchestra.run.smithi084.stdout: "source_version": "0'0", 2024-09-20T22:54:43.109 INFO:teuthology.orchestra.run.smithi084.stdout: "target_version": "0'0" 2024-09-20T22:54:43.109 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:43.109 INFO:teuthology.orchestra.run.smithi084.stdout: "last_change": "69", 2024-09-20T22:54:43.109 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend": "0", 2024-09-20T22:54:43.109 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-20T22:54:43.109 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_preluminous": "0", 2024-09-20T22:54:43.110 INFO:teuthology.orchestra.run.smithi084.stdout: "auid": 0, 2024-09-20T22:54:43.110 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_mode": "selfmanaged", 2024-09-20T22:54:43.110 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_seq": 2, 2024-09-20T22:54:43.110 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_epoch": 69, 2024-09-20T22:54:43.110 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_snaps": [], 2024-09-20T22:54:43.110 INFO:teuthology.orchestra.run.smithi084.stdout: "removed_snaps": "[]", 2024-09-20T22:54:43.110 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_bytes": 0, 2024-09-20T22:54:43.110 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_objects": 0, 2024-09-20T22:54:43.110 INFO:teuthology.orchestra.run.smithi084.stdout: "tiers": [], 2024-09-20T22:54:43.110 INFO:teuthology.orchestra.run.smithi084.stdout: "tier_of": -1, 2024-09-20T22:54:43.110 INFO:teuthology.orchestra.run.smithi084.stdout: "read_tier": -1, 2024-09-20T22:54:43.110 INFO:teuthology.orchestra.run.smithi084.stdout: "write_tier": -1, 2024-09-20T22:54:43.110 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_mode": "none", 2024-09-20T22:54:43.110 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_bytes": 0, 2024-09-20T22:54:43.111 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_objects": 0, 2024-09-20T22:54:43.111 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-20T22:54:43.111 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-20T22:54:43.111 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-20T22:54:43.111 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_flush_age": 0, 2024-09-20T22:54:43.111 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_evict_age": 0, 2024-09-20T22:54:43.111 INFO:teuthology.orchestra.run.smithi084.stdout: "erasure_code_profile": "", 2024-09-20T22:54:43.111 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_params": { 2024-09-20T22:54:43.111 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "none" 2024-09-20T22:54:43.111 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:43.111 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_period": 0, 2024-09-20T22:54:43.111 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_count": 0, 2024-09-20T22:54:43.111 INFO:teuthology.orchestra.run.smithi084.stdout: "use_gmt_hitset": true, 2024-09-20T22:54:43.111 INFO:teuthology.orchestra.run.smithi084.stdout: "min_read_recency_for_promote": 0, 2024-09-20T22:54:43.111 INFO:teuthology.orchestra.run.smithi084.stdout: "min_write_recency_for_promote": 0, 2024-09-20T22:54:43.112 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_grade_decay_rate": 0, 2024-09-20T22:54:43.112 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_search_last_n": 0, 2024-09-20T22:54:43.112 INFO:teuthology.orchestra.run.smithi084.stdout: "grade_table": [], 2024-09-20T22:54:43.112 INFO:teuthology.orchestra.run.smithi084.stdout: "stripe_width": 0, 2024-09-20T22:54:43.112 INFO:teuthology.orchestra.run.smithi084.stdout: "expected_num_objects": 0, 2024-09-20T22:54:43.112 INFO:teuthology.orchestra.run.smithi084.stdout: "fast_read": false, 2024-09-20T22:54:43.112 INFO:teuthology.orchestra.run.smithi084.stdout: "options": {}, 2024-09-20T22:54:43.112 INFO:teuthology.orchestra.run.smithi084.stdout: "application_metadata": { 2024-09-20T22:54:43.112 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd": {} 2024-09-20T22:54:43.112 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:43.112 INFO:teuthology.orchestra.run.smithi084.stdout: "read_balance": { 2024-09-20T22:54:43.112 INFO:teuthology.orchestra.run.smithi084.stdout: "score_type": "Fair distribution", 2024-09-20T22:54:43.112 INFO:teuthology.orchestra.run.smithi084.stdout: "score_acting": 2.6500000953674316, 2024-09-20T22:54:43.112 INFO:teuthology.orchestra.run.smithi084.stdout: "score_stable": 2.6500000953674316, 2024-09-20T22:54:43.112 INFO:teuthology.orchestra.run.smithi084.stdout: "optimal_score": 0.87999999523162842, 2024-09-20T22:54:43.113 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_acting": 2.3299999237060547, 2024-09-20T22:54:43.113 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_stable": 2.3299999237060547, 2024-09-20T22:54:43.113 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity_weighted": 1, 2024-09-20T22:54:43.113 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity": 1, 2024-09-20T22:54:43.113 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity_weighted": 1 2024-09-20T22:54:43.113 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:43.113 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:43.113 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:43.113 INFO:teuthology.orchestra.run.smithi084.stdout: "pool": 3, 2024-09-20T22:54:43.113 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_name": ".rgw.root", 2024-09-20T22:54:43.113 INFO:teuthology.orchestra.run.smithi084.stdout: "create_time": "2024-09-20T22:41:36.546389+0000", 2024-09-20T22:54:43.113 INFO:teuthology.orchestra.run.smithi084.stdout: "flags": 1, 2024-09-20T22:54:43.113 INFO:teuthology.orchestra.run.smithi084.stdout: "flags_names": "hashpspool", 2024-09-20T22:54:43.113 INFO:teuthology.orchestra.run.smithi084.stdout: "type": 1, 2024-09-20T22:54:43.114 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 3, 2024-09-20T22:54:43.114 INFO:teuthology.orchestra.run.smithi084.stdout: "min_size": 2, 2024-09-20T22:54:43.114 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_rule": 0, 2024-09-20T22:54:43.114 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_count": 0, 2024-09-20T22:54:43.114 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_target": 0, 2024-09-20T22:54:43.114 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_barrier": 0, 2024-09-20T22:54:43.114 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-20T22:54:43.114 INFO:teuthology.orchestra.run.smithi084.stdout: "is_stretch_pool": false, 2024-09-20T22:54:43.114 INFO:teuthology.orchestra.run.smithi084.stdout: "object_hash": 2, 2024-09-20T22:54:43.114 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_autoscale_mode": "off", 2024-09-20T22:54:43.114 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num": 32, 2024-09-20T22:54:43.114 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num": 32, 2024-09-20T22:54:43.114 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num_target": 32, 2024-09-20T22:54:43.114 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_target": 32, 2024-09-20T22:54:43.115 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_pending": 32, 2024-09-20T22:54:43.115 INFO:teuthology.orchestra.run.smithi084.stdout: "last_pg_merge_meta": { 2024-09-20T22:54:43.115 INFO:teuthology.orchestra.run.smithi084.stdout: "source_pgid": "0.0", 2024-09-20T22:54:43.115 INFO:teuthology.orchestra.run.smithi084.stdout: "ready_epoch": 0, 2024-09-20T22:54:43.116 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_started": 0, 2024-09-20T22:54:43.116 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_clean": 0, 2024-09-20T22:54:43.116 INFO:teuthology.orchestra.run.smithi084.stdout: "source_version": "0'0", 2024-09-20T22:54:43.116 INFO:teuthology.orchestra.run.smithi084.stdout: "target_version": "0'0" 2024-09-20T22:54:43.116 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:43.116 INFO:teuthology.orchestra.run.smithi084.stdout: "last_change": "68", 2024-09-20T22:54:43.116 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend": "0", 2024-09-20T22:54:43.116 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-20T22:54:43.116 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_preluminous": "0", 2024-09-20T22:54:43.116 INFO:teuthology.orchestra.run.smithi084.stdout: "auid": 0, 2024-09-20T22:54:43.116 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_mode": "selfmanaged", 2024-09-20T22:54:43.116 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_seq": 0, 2024-09-20T22:54:43.116 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_epoch": 0, 2024-09-20T22:54:43.117 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_snaps": [], 2024-09-20T22:54:43.117 INFO:teuthology.orchestra.run.smithi084.stdout: "removed_snaps": "[]", 2024-09-20T22:54:43.117 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_bytes": 0, 2024-09-20T22:54:43.117 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_objects": 0, 2024-09-20T22:54:43.117 INFO:teuthology.orchestra.run.smithi084.stdout: "tiers": [], 2024-09-20T22:54:43.117 INFO:teuthology.orchestra.run.smithi084.stdout: "tier_of": -1, 2024-09-20T22:54:43.117 INFO:teuthology.orchestra.run.smithi084.stdout: "read_tier": -1, 2024-09-20T22:54:43.117 INFO:teuthology.orchestra.run.smithi084.stdout: "write_tier": -1, 2024-09-20T22:54:43.117 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_mode": "none", 2024-09-20T22:54:43.117 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_bytes": 0, 2024-09-20T22:54:43.117 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_objects": 0, 2024-09-20T22:54:43.117 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-20T22:54:43.117 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-20T22:54:43.117 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-20T22:54:43.118 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_flush_age": 0, 2024-09-20T22:54:43.118 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_evict_age": 0, 2024-09-20T22:54:43.118 INFO:teuthology.orchestra.run.smithi084.stdout: "erasure_code_profile": "", 2024-09-20T22:54:43.118 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_params": { 2024-09-20T22:54:43.118 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "none" 2024-09-20T22:54:43.118 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:43.118 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_period": 0, 2024-09-20T22:54:43.118 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_count": 0, 2024-09-20T22:54:43.118 INFO:teuthology.orchestra.run.smithi084.stdout: "use_gmt_hitset": true, 2024-09-20T22:54:43.118 INFO:teuthology.orchestra.run.smithi084.stdout: "min_read_recency_for_promote": 0, 2024-09-20T22:54:43.118 INFO:teuthology.orchestra.run.smithi084.stdout: "min_write_recency_for_promote": 0, 2024-09-20T22:54:43.118 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_grade_decay_rate": 0, 2024-09-20T22:54:43.118 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_search_last_n": 0, 2024-09-20T22:54:43.118 INFO:teuthology.orchestra.run.smithi084.stdout: "grade_table": [], 2024-09-20T22:54:43.119 INFO:teuthology.orchestra.run.smithi084.stdout: "stripe_width": 0, 2024-09-20T22:54:43.119 INFO:teuthology.orchestra.run.smithi084.stdout: "expected_num_objects": 0, 2024-09-20T22:54:43.119 INFO:teuthology.orchestra.run.smithi084.stdout: "fast_read": false, 2024-09-20T22:54:43.119 INFO:teuthology.orchestra.run.smithi084.stdout: "options": {}, 2024-09-20T22:54:43.119 INFO:teuthology.orchestra.run.smithi084.stdout: "application_metadata": { 2024-09-20T22:54:43.119 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw": {} 2024-09-20T22:54:43.119 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:43.119 INFO:teuthology.orchestra.run.smithi084.stdout: "read_balance": { 2024-09-20T22:54:43.119 INFO:teuthology.orchestra.run.smithi084.stdout: "score_type": "Fair distribution", 2024-09-20T22:54:43.119 INFO:teuthology.orchestra.run.smithi084.stdout: "score_acting": 1.5, 2024-09-20T22:54:43.119 INFO:teuthology.orchestra.run.smithi084.stdout: "score_stable": 1.5, 2024-09-20T22:54:43.119 INFO:teuthology.orchestra.run.smithi084.stdout: "optimal_score": 1, 2024-09-20T22:54:43.120 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_acting": 1.5, 2024-09-20T22:54:43.120 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_stable": 1.5, 2024-09-20T22:54:43.120 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity_weighted": 1, 2024-09-20T22:54:43.120 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity": 1, 2024-09-20T22:54:43.120 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity_weighted": 1 2024-09-20T22:54:43.120 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:43.121 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:43.121 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:43.121 INFO:teuthology.orchestra.run.smithi084.stdout: "pool": 4, 2024-09-20T22:54:43.122 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_name": "default.rgw.log", 2024-09-20T22:54:43.122 INFO:teuthology.orchestra.run.smithi084.stdout: "create_time": "2024-09-20T22:41:38.454640+0000", 2024-09-20T22:54:43.122 INFO:teuthology.orchestra.run.smithi084.stdout: "flags": 1, 2024-09-20T22:54:43.122 INFO:teuthology.orchestra.run.smithi084.stdout: "flags_names": "hashpspool", 2024-09-20T22:54:43.122 INFO:teuthology.orchestra.run.smithi084.stdout: "type": 1, 2024-09-20T22:54:43.122 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 3, 2024-09-20T22:54:43.122 INFO:teuthology.orchestra.run.smithi084.stdout: "min_size": 2, 2024-09-20T22:54:43.122 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_rule": 0, 2024-09-20T22:54:43.122 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_count": 0, 2024-09-20T22:54:43.122 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_target": 0, 2024-09-20T22:54:43.122 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_barrier": 0, 2024-09-20T22:54:43.122 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-20T22:54:43.122 INFO:teuthology.orchestra.run.smithi084.stdout: "is_stretch_pool": false, 2024-09-20T22:54:43.123 INFO:teuthology.orchestra.run.smithi084.stdout: "object_hash": 2, 2024-09-20T22:54:43.123 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_autoscale_mode": "off", 2024-09-20T22:54:43.123 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num": 32, 2024-09-20T22:54:43.123 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num": 32, 2024-09-20T22:54:43.123 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num_target": 32, 2024-09-20T22:54:43.123 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_target": 32, 2024-09-20T22:54:43.123 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_pending": 32, 2024-09-20T22:54:43.123 INFO:teuthology.orchestra.run.smithi084.stdout: "last_pg_merge_meta": { 2024-09-20T22:54:43.123 INFO:teuthology.orchestra.run.smithi084.stdout: "source_pgid": "0.0", 2024-09-20T22:54:43.123 INFO:teuthology.orchestra.run.smithi084.stdout: "ready_epoch": 0, 2024-09-20T22:54:43.123 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_started": 0, 2024-09-20T22:54:43.123 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_clean": 0, 2024-09-20T22:54:43.123 INFO:teuthology.orchestra.run.smithi084.stdout: "source_version": "0'0", 2024-09-20T22:54:43.124 INFO:teuthology.orchestra.run.smithi084.stdout: "target_version": "0'0" 2024-09-20T22:54:43.124 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:43.124 INFO:teuthology.orchestra.run.smithi084.stdout: "last_change": "70", 2024-09-20T22:54:43.124 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend": "0", 2024-09-20T22:54:43.124 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-20T22:54:43.124 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_preluminous": "0", 2024-09-20T22:54:43.124 INFO:teuthology.orchestra.run.smithi084.stdout: "auid": 0, 2024-09-20T22:54:43.124 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_mode": "selfmanaged", 2024-09-20T22:54:43.124 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_seq": 0, 2024-09-20T22:54:43.124 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_epoch": 0, 2024-09-20T22:54:43.124 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_snaps": [], 2024-09-20T22:54:43.124 INFO:teuthology.orchestra.run.smithi084.stdout: "removed_snaps": "[]", 2024-09-20T22:54:43.124 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_bytes": 0, 2024-09-20T22:54:43.124 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_objects": 0, 2024-09-20T22:54:43.125 INFO:teuthology.orchestra.run.smithi084.stdout: "tiers": [], 2024-09-20T22:54:43.125 INFO:teuthology.orchestra.run.smithi084.stdout: "tier_of": -1, 2024-09-20T22:54:43.125 INFO:teuthology.orchestra.run.smithi084.stdout: "read_tier": -1, 2024-09-20T22:54:43.125 INFO:teuthology.orchestra.run.smithi084.stdout: "write_tier": -1, 2024-09-20T22:54:43.125 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_mode": "none", 2024-09-20T22:54:43.125 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_bytes": 0, 2024-09-20T22:54:43.125 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_objects": 0, 2024-09-20T22:54:43.125 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-20T22:54:43.125 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-20T22:54:43.125 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-20T22:54:43.125 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_flush_age": 0, 2024-09-20T22:54:43.125 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_evict_age": 0, 2024-09-20T22:54:43.125 INFO:teuthology.orchestra.run.smithi084.stdout: "erasure_code_profile": "", 2024-09-20T22:54:43.125 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_params": { 2024-09-20T22:54:43.126 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "none" 2024-09-20T22:54:43.126 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:43.126 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_period": 0, 2024-09-20T22:54:43.126 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_count": 0, 2024-09-20T22:54:43.126 INFO:teuthology.orchestra.run.smithi084.stdout: "use_gmt_hitset": true, 2024-09-20T22:54:43.126 INFO:teuthology.orchestra.run.smithi084.stdout: "min_read_recency_for_promote": 0, 2024-09-20T22:54:43.126 INFO:teuthology.orchestra.run.smithi084.stdout: "min_write_recency_for_promote": 0, 2024-09-20T22:54:43.126 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_grade_decay_rate": 0, 2024-09-20T22:54:43.126 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_search_last_n": 0, 2024-09-20T22:54:43.126 INFO:teuthology.orchestra.run.smithi084.stdout: "grade_table": [], 2024-09-20T22:54:43.126 INFO:teuthology.orchestra.run.smithi084.stdout: "stripe_width": 0, 2024-09-20T22:54:43.126 INFO:teuthology.orchestra.run.smithi084.stdout: "expected_num_objects": 0, 2024-09-20T22:54:43.126 INFO:teuthology.orchestra.run.smithi084.stdout: "fast_read": false, 2024-09-20T22:54:43.126 INFO:teuthology.orchestra.run.smithi084.stdout: "options": {}, 2024-09-20T22:54:43.127 INFO:teuthology.orchestra.run.smithi084.stdout: "application_metadata": { 2024-09-20T22:54:43.127 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw": {} 2024-09-20T22:54:43.127 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:43.127 INFO:teuthology.orchestra.run.smithi084.stdout: "read_balance": { 2024-09-20T22:54:43.127 INFO:teuthology.orchestra.run.smithi084.stdout: "score_type": "Fair distribution", 2024-09-20T22:54:43.127 INFO:teuthology.orchestra.run.smithi084.stdout: "score_acting": 2.25, 2024-09-20T22:54:43.127 INFO:teuthology.orchestra.run.smithi084.stdout: "score_stable": 2.25, 2024-09-20T22:54:43.127 INFO:teuthology.orchestra.run.smithi084.stdout: "optimal_score": 1, 2024-09-20T22:54:43.127 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_acting": 2.25, 2024-09-20T22:54:43.127 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_stable": 2.25, 2024-09-20T22:54:43.127 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity_weighted": 1, 2024-09-20T22:54:43.127 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity": 1, 2024-09-20T22:54:43.127 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity_weighted": 1 2024-09-20T22:54:43.127 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:43.128 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:43.128 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:43.128 INFO:teuthology.orchestra.run.smithi084.stdout: "pool": 5, 2024-09-20T22:54:43.128 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_name": "default.rgw.control", 2024-09-20T22:54:43.128 INFO:teuthology.orchestra.run.smithi084.stdout: "create_time": "2024-09-20T22:41:40.407129+0000", 2024-09-20T22:54:43.128 INFO:teuthology.orchestra.run.smithi084.stdout: "flags": 1, 2024-09-20T22:54:43.128 INFO:teuthology.orchestra.run.smithi084.stdout: "flags_names": "hashpspool", 2024-09-20T22:54:43.128 INFO:teuthology.orchestra.run.smithi084.stdout: "type": 1, 2024-09-20T22:54:43.128 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 3, 2024-09-20T22:54:43.128 INFO:teuthology.orchestra.run.smithi084.stdout: "min_size": 2, 2024-09-20T22:54:43.128 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_rule": 0, 2024-09-20T22:54:43.128 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_count": 0, 2024-09-20T22:54:43.128 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_target": 0, 2024-09-20T22:54:43.128 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_barrier": 0, 2024-09-20T22:54:43.129 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-20T22:54:43.129 INFO:teuthology.orchestra.run.smithi084.stdout: "is_stretch_pool": false, 2024-09-20T22:54:43.129 INFO:teuthology.orchestra.run.smithi084.stdout: "object_hash": 2, 2024-09-20T22:54:43.129 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_autoscale_mode": "off", 2024-09-20T22:54:43.129 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num": 32, 2024-09-20T22:54:43.129 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num": 32, 2024-09-20T22:54:43.129 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num_target": 32, 2024-09-20T22:54:43.129 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_target": 32, 2024-09-20T22:54:43.129 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_pending": 32, 2024-09-20T22:54:43.129 INFO:teuthology.orchestra.run.smithi084.stdout: "last_pg_merge_meta": { 2024-09-20T22:54:43.129 INFO:teuthology.orchestra.run.smithi084.stdout: "source_pgid": "0.0", 2024-09-20T22:54:43.129 INFO:teuthology.orchestra.run.smithi084.stdout: "ready_epoch": 0, 2024-09-20T22:54:43.129 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_started": 0, 2024-09-20T22:54:43.129 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_clean": 0, 2024-09-20T22:54:43.130 INFO:teuthology.orchestra.run.smithi084.stdout: "source_version": "0'0", 2024-09-20T22:54:43.130 INFO:teuthology.orchestra.run.smithi084.stdout: "target_version": "0'0" 2024-09-20T22:54:43.130 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:43.130 INFO:teuthology.orchestra.run.smithi084.stdout: "last_change": "72", 2024-09-20T22:54:43.130 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend": "0", 2024-09-20T22:54:43.130 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-20T22:54:43.130 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_preluminous": "0", 2024-09-20T22:54:43.130 INFO:teuthology.orchestra.run.smithi084.stdout: "auid": 0, 2024-09-20T22:54:43.130 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_mode": "selfmanaged", 2024-09-20T22:54:43.130 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_seq": 0, 2024-09-20T22:54:43.130 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_epoch": 0, 2024-09-20T22:54:43.130 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_snaps": [], 2024-09-20T22:54:43.130 INFO:teuthology.orchestra.run.smithi084.stdout: "removed_snaps": "[]", 2024-09-20T22:54:43.130 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_bytes": 0, 2024-09-20T22:54:43.130 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_objects": 0, 2024-09-20T22:54:43.131 INFO:teuthology.orchestra.run.smithi084.stdout: "tiers": [], 2024-09-20T22:54:43.131 INFO:teuthology.orchestra.run.smithi084.stdout: "tier_of": -1, 2024-09-20T22:54:43.131 INFO:teuthology.orchestra.run.smithi084.stdout: "read_tier": -1, 2024-09-20T22:54:43.131 INFO:teuthology.orchestra.run.smithi084.stdout: "write_tier": -1, 2024-09-20T22:54:43.131 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_mode": "none", 2024-09-20T22:54:43.131 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_bytes": 0, 2024-09-20T22:54:43.131 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_objects": 0, 2024-09-20T22:54:43.131 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-20T22:54:43.131 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-20T22:54:43.131 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-20T22:54:43.131 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_flush_age": 0, 2024-09-20T22:54:43.131 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_evict_age": 0, 2024-09-20T22:54:43.131 INFO:teuthology.orchestra.run.smithi084.stdout: "erasure_code_profile": "", 2024-09-20T22:54:43.132 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_params": { 2024-09-20T22:54:43.132 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "none" 2024-09-20T22:54:43.132 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:43.132 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_period": 0, 2024-09-20T22:54:43.132 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_count": 0, 2024-09-20T22:54:43.132 INFO:teuthology.orchestra.run.smithi084.stdout: "use_gmt_hitset": true, 2024-09-20T22:54:43.132 INFO:teuthology.orchestra.run.smithi084.stdout: "min_read_recency_for_promote": 0, 2024-09-20T22:54:43.132 INFO:teuthology.orchestra.run.smithi084.stdout: "min_write_recency_for_promote": 0, 2024-09-20T22:54:43.132 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_grade_decay_rate": 0, 2024-09-20T22:54:43.132 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_search_last_n": 0, 2024-09-20T22:54:43.132 INFO:teuthology.orchestra.run.smithi084.stdout: "grade_table": [], 2024-09-20T22:54:43.132 INFO:teuthology.orchestra.run.smithi084.stdout: "stripe_width": 0, 2024-09-20T22:54:43.132 INFO:teuthology.orchestra.run.smithi084.stdout: "expected_num_objects": 0, 2024-09-20T22:54:43.132 INFO:teuthology.orchestra.run.smithi084.stdout: "fast_read": false, 2024-09-20T22:54:43.133 INFO:teuthology.orchestra.run.smithi084.stdout: "options": {}, 2024-09-20T22:54:43.133 INFO:teuthology.orchestra.run.smithi084.stdout: "application_metadata": { 2024-09-20T22:54:43.133 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw": {} 2024-09-20T22:54:43.133 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:43.133 INFO:teuthology.orchestra.run.smithi084.stdout: "read_balance": { 2024-09-20T22:54:43.133 INFO:teuthology.orchestra.run.smithi084.stdout: "score_type": "Fair distribution", 2024-09-20T22:54:43.133 INFO:teuthology.orchestra.run.smithi084.stdout: "score_acting": 1.25, 2024-09-20T22:54:43.133 INFO:teuthology.orchestra.run.smithi084.stdout: "score_stable": 1.25, 2024-09-20T22:54:43.133 INFO:teuthology.orchestra.run.smithi084.stdout: "optimal_score": 1, 2024-09-20T22:54:43.133 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_acting": 1.25, 2024-09-20T22:54:43.133 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_stable": 1.25, 2024-09-20T22:54:43.133 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity_weighted": 1, 2024-09-20T22:54:43.133 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity": 1, 2024-09-20T22:54:43.134 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity_weighted": 1 2024-09-20T22:54:43.134 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:43.134 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:43.134 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:43.134 INFO:teuthology.orchestra.run.smithi084.stdout: "pool": 6, 2024-09-20T22:54:43.134 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_name": "default.rgw.meta", 2024-09-20T22:54:43.134 INFO:teuthology.orchestra.run.smithi084.stdout: "create_time": "2024-09-20T22:41:42.476716+0000", 2024-09-20T22:54:43.134 INFO:teuthology.orchestra.run.smithi084.stdout: "flags": 1, 2024-09-20T22:54:43.134 INFO:teuthology.orchestra.run.smithi084.stdout: "flags_names": "hashpspool", 2024-09-20T22:54:43.134 INFO:teuthology.orchestra.run.smithi084.stdout: "type": 1, 2024-09-20T22:54:43.134 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 3, 2024-09-20T22:54:43.134 INFO:teuthology.orchestra.run.smithi084.stdout: "min_size": 2, 2024-09-20T22:54:43.134 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_rule": 0, 2024-09-20T22:54:43.135 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_count": 0, 2024-09-20T22:54:43.135 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_target": 0, 2024-09-20T22:54:43.135 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_barrier": 0, 2024-09-20T22:54:43.135 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-20T22:54:43.135 INFO:teuthology.orchestra.run.smithi084.stdout: "is_stretch_pool": false, 2024-09-20T22:54:43.135 INFO:teuthology.orchestra.run.smithi084.stdout: "object_hash": 2, 2024-09-20T22:54:43.135 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_autoscale_mode": "off", 2024-09-20T22:54:43.135 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num": 32, 2024-09-20T22:54:43.135 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num": 32, 2024-09-20T22:54:43.135 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num_target": 32, 2024-09-20T22:54:43.135 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_target": 32, 2024-09-20T22:54:43.135 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_pending": 32, 2024-09-20T22:54:43.135 INFO:teuthology.orchestra.run.smithi084.stdout: "last_pg_merge_meta": { 2024-09-20T22:54:43.135 INFO:teuthology.orchestra.run.smithi084.stdout: "source_pgid": "0.0", 2024-09-20T22:54:43.135 INFO:teuthology.orchestra.run.smithi084.stdout: "ready_epoch": 0, 2024-09-20T22:54:43.136 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_started": 0, 2024-09-20T22:54:43.136 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_clean": 0, 2024-09-20T22:54:43.136 INFO:teuthology.orchestra.run.smithi084.stdout: "source_version": "0'0", 2024-09-20T22:54:43.136 INFO:teuthology.orchestra.run.smithi084.stdout: "target_version": "0'0" 2024-09-20T22:54:43.136 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:43.136 INFO:teuthology.orchestra.run.smithi084.stdout: "last_change": "74", 2024-09-20T22:54:43.136 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend": "0", 2024-09-20T22:54:43.136 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-20T22:54:43.136 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_preluminous": "0", 2024-09-20T22:54:43.136 INFO:teuthology.orchestra.run.smithi084.stdout: "auid": 0, 2024-09-20T22:54:43.136 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_mode": "selfmanaged", 2024-09-20T22:54:43.136 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_seq": 0, 2024-09-20T22:54:43.136 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_epoch": 0, 2024-09-20T22:54:43.136 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_snaps": [], 2024-09-20T22:54:43.136 INFO:teuthology.orchestra.run.smithi084.stdout: "removed_snaps": "[]", 2024-09-20T22:54:43.137 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_bytes": 0, 2024-09-20T22:54:43.137 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_objects": 0, 2024-09-20T22:54:43.137 INFO:teuthology.orchestra.run.smithi084.stdout: "tiers": [], 2024-09-20T22:54:43.137 INFO:teuthology.orchestra.run.smithi084.stdout: "tier_of": -1, 2024-09-20T22:54:43.137 INFO:teuthology.orchestra.run.smithi084.stdout: "read_tier": -1, 2024-09-20T22:54:43.137 INFO:teuthology.orchestra.run.smithi084.stdout: "write_tier": -1, 2024-09-20T22:54:43.137 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_mode": "none", 2024-09-20T22:54:43.137 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_bytes": 0, 2024-09-20T22:54:43.137 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_objects": 0, 2024-09-20T22:54:43.137 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-20T22:54:43.137 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-20T22:54:43.137 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-20T22:54:43.137 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_flush_age": 0, 2024-09-20T22:54:43.137 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_evict_age": 0, 2024-09-20T22:54:43.138 INFO:teuthology.orchestra.run.smithi084.stdout: "erasure_code_profile": "", 2024-09-20T22:54:43.138 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_params": { 2024-09-20T22:54:43.138 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "none" 2024-09-20T22:54:43.138 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:43.138 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_period": 0, 2024-09-20T22:54:43.138 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_count": 0, 2024-09-20T22:54:43.138 INFO:teuthology.orchestra.run.smithi084.stdout: "use_gmt_hitset": true, 2024-09-20T22:54:43.138 INFO:teuthology.orchestra.run.smithi084.stdout: "min_read_recency_for_promote": 0, 2024-09-20T22:54:43.138 INFO:teuthology.orchestra.run.smithi084.stdout: "min_write_recency_for_promote": 0, 2024-09-20T22:54:43.138 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_grade_decay_rate": 0, 2024-09-20T22:54:43.138 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_search_last_n": 0, 2024-09-20T22:54:43.138 INFO:teuthology.orchestra.run.smithi084.stdout: "grade_table": [], 2024-09-20T22:54:43.138 INFO:teuthology.orchestra.run.smithi084.stdout: "stripe_width": 0, 2024-09-20T22:54:43.138 INFO:teuthology.orchestra.run.smithi084.stdout: "expected_num_objects": 0, 2024-09-20T22:54:43.139 INFO:teuthology.orchestra.run.smithi084.stdout: "fast_read": false, 2024-09-20T22:54:43.139 INFO:teuthology.orchestra.run.smithi084.stdout: "options": { 2024-09-20T22:54:43.139 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_autoscale_bias": 4 2024-09-20T22:54:43.139 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:43.139 INFO:teuthology.orchestra.run.smithi084.stdout: "application_metadata": { 2024-09-20T22:54:43.139 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw": {} 2024-09-20T22:54:43.139 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:43.139 INFO:teuthology.orchestra.run.smithi084.stdout: "read_balance": { 2024-09-20T22:54:43.139 INFO:teuthology.orchestra.run.smithi084.stdout: "score_type": "Fair distribution", 2024-09-20T22:54:43.139 INFO:teuthology.orchestra.run.smithi084.stdout: "score_acting": 1.75, 2024-09-20T22:54:43.139 INFO:teuthology.orchestra.run.smithi084.stdout: "score_stable": 1.75, 2024-09-20T22:54:43.139 INFO:teuthology.orchestra.run.smithi084.stdout: "optimal_score": 1, 2024-09-20T22:54:43.139 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_acting": 1.75, 2024-09-20T22:54:43.139 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_stable": 1.75, 2024-09-20T22:54:43.140 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity_weighted": 1, 2024-09-20T22:54:43.140 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity": 1, 2024-09-20T22:54:43.140 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity_weighted": 1 2024-09-20T22:54:43.140 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:43.140 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:43.140 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:43.140 INFO:teuthology.orchestra.run.smithi084.stdout: "pool": 7, 2024-09-20T22:54:43.140 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_name": "unique_pool_0", 2024-09-20T22:54:43.140 INFO:teuthology.orchestra.run.smithi084.stdout: "create_time": "2024-09-20T22:42:51.354358+0000", 2024-09-20T22:54:43.140 INFO:teuthology.orchestra.run.smithi084.stdout: "flags": 8193, 2024-09-20T22:54:43.140 INFO:teuthology.orchestra.run.smithi084.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-20T22:54:43.140 INFO:teuthology.orchestra.run.smithi084.stdout: "type": 1, 2024-09-20T22:54:43.140 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 3, 2024-09-20T22:54:43.140 INFO:teuthology.orchestra.run.smithi084.stdout: "min_size": 2, 2024-09-20T22:54:43.141 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_rule": 0, 2024-09-20T22:54:43.141 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_count": 0, 2024-09-20T22:54:43.141 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_target": 0, 2024-09-20T22:54:43.141 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_barrier": 0, 2024-09-20T22:54:43.141 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-20T22:54:43.141 INFO:teuthology.orchestra.run.smithi084.stdout: "is_stretch_pool": false, 2024-09-20T22:54:43.141 INFO:teuthology.orchestra.run.smithi084.stdout: "object_hash": 2, 2024-09-20T22:54:43.141 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_autoscale_mode": "off", 2024-09-20T22:54:43.141 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num": 16, 2024-09-20T22:54:43.141 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num": 16, 2024-09-20T22:54:43.141 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num_target": 16, 2024-09-20T22:54:43.141 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_target": 16, 2024-09-20T22:54:43.141 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_pending": 16, 2024-09-20T22:54:43.142 INFO:teuthology.orchestra.run.smithi084.stdout: "last_pg_merge_meta": { 2024-09-20T22:54:43.142 INFO:teuthology.orchestra.run.smithi084.stdout: "source_pgid": "7.10", 2024-09-20T22:54:43.142 INFO:teuthology.orchestra.run.smithi084.stdout: "ready_epoch": 484, 2024-09-20T22:54:43.142 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_started": 483, 2024-09-20T22:54:43.142 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_clean": 483, 2024-09-20T22:54:43.142 INFO:teuthology.orchestra.run.smithi084.stdout: "source_version": "480'172", 2024-09-20T22:54:43.142 INFO:teuthology.orchestra.run.smithi084.stdout: "target_version": "480'253" 2024-09-20T22:54:43.142 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:43.142 INFO:teuthology.orchestra.run.smithi084.stdout: "last_change": "802", 2024-09-20T22:54:43.142 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend": "0", 2024-09-20T22:54:43.142 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_prenautilus": "484", 2024-09-20T22:54:43.142 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_preluminous": "482", 2024-09-20T22:54:43.142 INFO:teuthology.orchestra.run.smithi084.stdout: "auid": 0, 2024-09-20T22:54:43.143 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_mode": "selfmanaged", 2024-09-20T22:54:43.143 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_seq": 367, 2024-09-20T22:54:43.143 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_epoch": 802, 2024-09-20T22:54:43.143 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_snaps": [], 2024-09-20T22:54:43.143 INFO:teuthology.orchestra.run.smithi084.stdout: "removed_snaps": "[]", 2024-09-20T22:54:43.143 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_bytes": 0, 2024-09-20T22:54:43.143 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_objects": 0, 2024-09-20T22:54:43.143 INFO:teuthology.orchestra.run.smithi084.stdout: "tiers": [], 2024-09-20T22:54:43.143 INFO:teuthology.orchestra.run.smithi084.stdout: "tier_of": -1, 2024-09-20T22:54:43.143 INFO:teuthology.orchestra.run.smithi084.stdout: "read_tier": -1, 2024-09-20T22:54:43.143 INFO:teuthology.orchestra.run.smithi084.stdout: "write_tier": -1, 2024-09-20T22:54:43.143 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_mode": "none", 2024-09-20T22:54:43.143 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_bytes": 0, 2024-09-20T22:54:43.144 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_objects": 0, 2024-09-20T22:54:43.144 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-20T22:54:43.144 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-20T22:54:43.144 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-20T22:54:43.144 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_flush_age": 0, 2024-09-20T22:54:43.144 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_evict_age": 0, 2024-09-20T22:54:43.144 INFO:teuthology.orchestra.run.smithi084.stdout: "erasure_code_profile": "", 2024-09-20T22:54:43.144 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_params": { 2024-09-20T22:54:43.144 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "none" 2024-09-20T22:54:43.144 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:43.144 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_period": 0, 2024-09-20T22:54:43.144 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_count": 0, 2024-09-20T22:54:43.144 INFO:teuthology.orchestra.run.smithi084.stdout: "use_gmt_hitset": true, 2024-09-20T22:54:43.144 INFO:teuthology.orchestra.run.smithi084.stdout: "min_read_recency_for_promote": 0, 2024-09-20T22:54:43.144 INFO:teuthology.orchestra.run.smithi084.stdout: "min_write_recency_for_promote": 0, 2024-09-20T22:54:43.145 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_grade_decay_rate": 0, 2024-09-20T22:54:43.145 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_search_last_n": 0, 2024-09-20T22:54:43.145 INFO:teuthology.orchestra.run.smithi084.stdout: "grade_table": [], 2024-09-20T22:54:43.145 INFO:teuthology.orchestra.run.smithi084.stdout: "stripe_width": 0, 2024-09-20T22:54:43.145 INFO:teuthology.orchestra.run.smithi084.stdout: "expected_num_objects": 0, 2024-09-20T22:54:43.145 INFO:teuthology.orchestra.run.smithi084.stdout: "fast_read": false, 2024-09-20T22:54:43.145 INFO:teuthology.orchestra.run.smithi084.stdout: "options": {}, 2024-09-20T22:54:43.145 INFO:teuthology.orchestra.run.smithi084.stdout: "application_metadata": { 2024-09-20T22:54:43.145 INFO:teuthology.orchestra.run.smithi084.stdout: "rados": {} 2024-09-20T22:54:43.145 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:43.145 INFO:teuthology.orchestra.run.smithi084.stdout: "read_balance": { 2024-09-20T22:54:43.145 INFO:teuthology.orchestra.run.smithi084.stdout: "score_type": "Fair distribution", 2024-09-20T22:54:43.145 INFO:teuthology.orchestra.run.smithi084.stdout: "score_acting": 2, 2024-09-20T22:54:43.145 INFO:teuthology.orchestra.run.smithi084.stdout: "score_stable": 2, 2024-09-20T22:54:43.146 INFO:teuthology.orchestra.run.smithi084.stdout: "optimal_score": 1, 2024-09-20T22:54:43.146 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_acting": 2, 2024-09-20T22:54:43.146 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_stable": 2, 2024-09-20T22:54:43.146 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity_weighted": 1, 2024-09-20T22:54:43.146 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity": 1, 2024-09-20T22:54:43.146 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity_weighted": 1 2024-09-20T22:54:43.146 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:43.146 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:43.146 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2024-09-20T22:54:43.146 INFO:teuthology.orchestra.run.smithi084.stdout: "osds": [ 2024-09-20T22:54:43.146 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:43.146 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 0, 2024-09-20T22:54:43.146 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "ea36a489-6f91-4b2f-b784-b90506b9805d", 2024-09-20T22:54:43.147 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-09-20T22:54:43.147 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-09-20T22:54:43.147 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-09-20T22:54:43.147 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-09-20T22:54:43.147 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-09-20T22:54:43.147 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-09-20T22:54:43.147 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 8, 2024-09-20T22:54:43.147 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 779, 2024-09-20T22:54:43.147 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-09-20T22:54:43.147 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-09-20T22:54:43.147 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-09-20T22:54:43.147 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:54:43.147 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:43.147 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:54:43.147 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6801", 2024-09-20T22:54:43.148 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 268084626 2024-09-20T22:54:43.148 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:43.148 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:43.148 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:43.148 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-09-20T22:54:43.148 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:54:43.148 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:43.148 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:54:43.148 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6802", 2024-09-20T22:54:43.148 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 268084626 2024-09-20T22:54:43.148 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:43.148 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:43.148 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:43.148 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-09-20T22:54:43.148 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:54:43.149 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:43.149 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:54:43.149 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6804", 2024-09-20T22:54:43.149 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 268084626 2024-09-20T22:54:43.149 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:43.149 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:43.149 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:43.149 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-09-20T22:54:43.149 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:54:43.149 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:43.149 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:54:43.149 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6803", 2024-09-20T22:54:43.149 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 268084626 2024-09-20T22:54:43.150 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:43.150 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:43.150 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:43.150 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.84:6801/268084626", 2024-09-20T22:54:43.150 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.84:6802/268084626", 2024-09-20T22:54:43.150 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.84:6804/268084626", 2024-09-20T22:54:43.150 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.84:6803/268084626", 2024-09-20T22:54:43.150 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-09-20T22:54:43.150 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-09-20T22:54:43.150 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-09-20T22:54:43.150 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:43.150 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:43.150 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:43.150 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 1, 2024-09-20T22:54:43.150 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "c8649d2f-27ef-47f8-bc94-f1cd9e62bdb8", 2024-09-20T22:54:43.151 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-09-20T22:54:43.151 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-09-20T22:54:43.151 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-09-20T22:54:43.151 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-09-20T22:54:43.151 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-09-20T22:54:43.151 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-09-20T22:54:43.151 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 13, 2024-09-20T22:54:43.151 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 779, 2024-09-20T22:54:43.151 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-09-20T22:54:43.151 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-09-20T22:54:43.151 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-09-20T22:54:43.152 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:54:43.152 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:43.152 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:54:43.152 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6805", 2024-09-20T22:54:43.152 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 418374136 2024-09-20T22:54:43.152 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:43.152 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:43.152 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:43.152 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-09-20T22:54:43.152 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:54:43.152 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:43.152 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:54:43.152 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6806", 2024-09-20T22:54:43.152 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 418374136 2024-09-20T22:54:43.153 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:43.153 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:43.153 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:43.153 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-09-20T22:54:43.153 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:54:43.153 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:43.153 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:54:43.153 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6808", 2024-09-20T22:54:43.153 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 418374136 2024-09-20T22:54:43.153 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:43.153 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:43.153 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:43.154 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-09-20T22:54:43.154 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:54:43.154 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:43.154 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:54:43.154 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6807", 2024-09-20T22:54:43.154 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 418374136 2024-09-20T22:54:43.154 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:43.154 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:43.154 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:43.154 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.84:6805/418374136", 2024-09-20T22:54:43.154 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.84:6806/418374136", 2024-09-20T22:54:43.154 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.84:6808/418374136", 2024-09-20T22:54:43.154 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.84:6807/418374136", 2024-09-20T22:54:43.155 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-09-20T22:54:43.155 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-09-20T22:54:43.155 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-09-20T22:54:43.155 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:43.155 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:43.155 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:43.155 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 2, 2024-09-20T22:54:43.155 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "c728eb48-a0f0-4de4-a651-7f3bec867280", 2024-09-20T22:54:43.155 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-09-20T22:54:43.155 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-09-20T22:54:43.155 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-09-20T22:54:43.155 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-09-20T22:54:43.155 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-09-20T22:54:43.155 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-09-20T22:54:43.156 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 18, 2024-09-20T22:54:43.156 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 778, 2024-09-20T22:54:43.156 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-09-20T22:54:43.156 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-09-20T22:54:43.156 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-09-20T22:54:43.156 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:54:43.156 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:43.156 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:54:43.156 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6809", 2024-09-20T22:54:43.156 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3977168611 2024-09-20T22:54:43.156 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:43.156 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:43.156 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:43.156 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-09-20T22:54:43.157 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:54:43.157 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:43.157 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:54:43.157 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6810", 2024-09-20T22:54:43.157 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3977168611 2024-09-20T22:54:43.157 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:43.157 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:43.157 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:43.157 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-09-20T22:54:43.157 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:54:43.157 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:43.157 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:54:43.157 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6812", 2024-09-20T22:54:43.157 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3977168611 2024-09-20T22:54:43.158 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:43.158 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:43.158 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:43.158 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-09-20T22:54:43.158 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:54:43.158 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:43.158 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:54:43.158 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6811", 2024-09-20T22:54:43.158 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3977168611 2024-09-20T22:54:43.158 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:43.158 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:43.158 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:43.158 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.84:6809/3977168611", 2024-09-20T22:54:43.158 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.84:6810/3977168611", 2024-09-20T22:54:43.159 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.84:6812/3977168611", 2024-09-20T22:54:43.159 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.84:6811/3977168611", 2024-09-20T22:54:43.159 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-09-20T22:54:43.159 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-09-20T22:54:43.159 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-09-20T22:54:43.159 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:43.159 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:43.159 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:43.159 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 3, 2024-09-20T22:54:43.159 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "573f44eb-0853-4630-a7d8-9caaeb256f9b", 2024-09-20T22:54:43.159 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-09-20T22:54:43.159 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-09-20T22:54:43.159 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-09-20T22:54:43.159 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-09-20T22:54:43.160 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-09-20T22:54:43.160 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-09-20T22:54:43.160 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 25, 2024-09-20T22:54:43.160 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 777, 2024-09-20T22:54:43.160 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-09-20T22:54:43.160 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-09-20T22:54:43.160 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-09-20T22:54:43.160 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:54:43.160 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:43.160 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:54:43.160 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6813", 2024-09-20T22:54:43.160 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2192239945 2024-09-20T22:54:43.160 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:43.160 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:43.160 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:43.161 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-09-20T22:54:43.161 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:54:43.161 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:43.161 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:54:43.161 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6814", 2024-09-20T22:54:43.161 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2192239945 2024-09-20T22:54:43.161 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:43.161 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:43.161 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:43.161 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-09-20T22:54:43.161 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:54:43.161 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:43.161 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:54:43.161 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6816", 2024-09-20T22:54:43.161 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2192239945 2024-09-20T22:54:43.162 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:43.162 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:43.162 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:43.162 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-09-20T22:54:43.162 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:54:43.162 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:43.162 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:54:43.162 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6815", 2024-09-20T22:54:43.162 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2192239945 2024-09-20T22:54:43.162 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:43.162 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:43.162 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:43.162 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.84:6813/2192239945", 2024-09-20T22:54:43.162 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.84:6814/2192239945", 2024-09-20T22:54:43.163 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.84:6816/2192239945", 2024-09-20T22:54:43.163 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.84:6815/2192239945", 2024-09-20T22:54:43.163 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-09-20T22:54:43.163 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-09-20T22:54:43.163 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-09-20T22:54:43.163 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:43.163 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:43.163 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:43.163 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 4, 2024-09-20T22:54:43.163 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "a52c3b94-c5f3-490f-8ac8-09061b9b2abb", 2024-09-20T22:54:43.163 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-09-20T22:54:43.163 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-09-20T22:54:43.163 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-09-20T22:54:43.163 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-09-20T22:54:43.164 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-09-20T22:54:43.164 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-09-20T22:54:43.164 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 31, 2024-09-20T22:54:43.164 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 778, 2024-09-20T22:54:43.164 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-09-20T22:54:43.164 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-09-20T22:54:43.164 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-09-20T22:54:43.164 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:54:43.164 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:43.164 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:54:43.164 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.154:6800", 2024-09-20T22:54:43.164 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1793136002 2024-09-20T22:54:43.164 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:43.164 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:43.164 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:43.165 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-09-20T22:54:43.165 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:54:43.165 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:43.165 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:54:43.165 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.154:6801", 2024-09-20T22:54:43.165 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1793136002 2024-09-20T22:54:43.165 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:43.165 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:43.165 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:43.165 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-09-20T22:54:43.165 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:54:43.165 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:43.165 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:54:43.165 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.154:6803", 2024-09-20T22:54:43.166 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1793136002 2024-09-20T22:54:43.166 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:43.166 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:43.166 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:43.166 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-09-20T22:54:43.166 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:54:43.166 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:43.166 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:54:43.166 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.154:6802", 2024-09-20T22:54:43.166 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1793136002 2024-09-20T22:54:43.166 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:43.166 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:43.166 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:43.166 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.154:6800/1793136002", 2024-09-20T22:54:43.166 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.154:6801/1793136002", 2024-09-20T22:54:43.167 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.154:6803/1793136002", 2024-09-20T22:54:43.167 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.154:6802/1793136002", 2024-09-20T22:54:43.167 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-09-20T22:54:43.167 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-09-20T22:54:43.167 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-09-20T22:54:43.167 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:43.167 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:43.167 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:43.167 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 5, 2024-09-20T22:54:43.167 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "4fa0dc8e-25cc-46cd-96ff-33f20b6faec3", 2024-09-20T22:54:43.167 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-09-20T22:54:43.167 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-09-20T22:54:43.167 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-09-20T22:54:43.167 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-09-20T22:54:43.167 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-09-20T22:54:43.168 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-09-20T22:54:43.168 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 40, 2024-09-20T22:54:43.168 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 779, 2024-09-20T22:54:43.168 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-09-20T22:54:43.168 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-09-20T22:54:43.168 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-09-20T22:54:43.168 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:54:43.168 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:43.168 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:54:43.168 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.154:6804", 2024-09-20T22:54:43.168 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 493899927 2024-09-20T22:54:43.168 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:43.168 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:43.168 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:43.168 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-09-20T22:54:43.169 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:54:43.169 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:43.169 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:54:43.169 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.154:6805", 2024-09-20T22:54:43.169 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 493899927 2024-09-20T22:54:43.169 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:43.169 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:43.169 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:43.169 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-09-20T22:54:43.169 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:54:43.169 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:43.169 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:54:43.170 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:54:43.171 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.154:6807", 2024-09-20T22:54:43.171 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 493899927 2024-09-20T22:54:43.171 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:43.171 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:43.171 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:43.171 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-09-20T22:54:43.171 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:54:43.171 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:43.171 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:54:43.171 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.154:6806", 2024-09-20T22:54:43.171 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 493899927 2024-09-20T22:54:43.171 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:43.171 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:43.172 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:43.172 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.154:6804/493899927", 2024-09-20T22:54:43.172 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.154:6805/493899927", 2024-09-20T22:54:43.172 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.154:6807/493899927", 2024-09-20T22:54:43.172 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.154:6806/493899927", 2024-09-20T22:54:43.172 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-09-20T22:54:43.172 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-09-20T22:54:43.172 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-09-20T22:54:43.172 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:43.172 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:43.172 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:43.172 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 6, 2024-09-20T22:54:43.172 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "56788d74-810e-4827-a615-958c12f8a368", 2024-09-20T22:54:43.173 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-09-20T22:54:43.173 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-09-20T22:54:43.173 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-09-20T22:54:43.173 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-09-20T22:54:43.173 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-09-20T22:54:43.173 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-09-20T22:54:43.173 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 48, 2024-09-20T22:54:43.173 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 779, 2024-09-20T22:54:43.173 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-09-20T22:54:43.173 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-09-20T22:54:43.173 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-09-20T22:54:43.173 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:54:43.173 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:43.173 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:54:43.174 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.154:6808", 2024-09-20T22:54:43.174 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2375091033 2024-09-20T22:54:43.174 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:43.174 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:43.174 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:43.174 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-09-20T22:54:43.174 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:54:43.174 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:43.174 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:54:43.174 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.154:6809", 2024-09-20T22:54:43.174 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2375091033 2024-09-20T22:54:43.174 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:43.174 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:43.174 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:43.175 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-09-20T22:54:43.175 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:54:43.175 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:43.175 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:54:43.175 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.154:6811", 2024-09-20T22:54:43.175 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2375091033 2024-09-20T22:54:43.175 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:43.175 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:43.175 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:43.175 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-09-20T22:54:43.175 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:54:43.175 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:43.176 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:54:43.176 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.154:6810", 2024-09-20T22:54:43.176 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2375091033 2024-09-20T22:54:43.176 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:43.176 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:43.176 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:43.176 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.154:6808/2375091033", 2024-09-20T22:54:43.176 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.154:6809/2375091033", 2024-09-20T22:54:43.176 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.154:6811/2375091033", 2024-09-20T22:54:43.176 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.154:6810/2375091033", 2024-09-20T22:54:43.176 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-09-20T22:54:43.176 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-09-20T22:54:43.176 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-09-20T22:54:43.176 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:43.177 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:43.177 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:43.177 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 7, 2024-09-20T22:54:43.177 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "83503727-afc1-4b95-8b96-ecb63f3338c2", 2024-09-20T22:54:43.177 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-09-20T22:54:43.177 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-09-20T22:54:43.177 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-09-20T22:54:43.177 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-09-20T22:54:43.177 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-09-20T22:54:43.177 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-09-20T22:54:43.177 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 59, 2024-09-20T22:54:43.177 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 776, 2024-09-20T22:54:43.177 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-09-20T22:54:43.177 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-09-20T22:54:43.178 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-09-20T22:54:43.178 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:54:43.178 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:43.178 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:54:43.178 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.154:6812", 2024-09-20T22:54:43.178 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3717704491 2024-09-20T22:54:43.178 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:43.178 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:43.178 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:43.178 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-09-20T22:54:43.178 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:54:43.178 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:43.178 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:54:43.178 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.154:6813", 2024-09-20T22:54:43.179 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3717704491 2024-09-20T22:54:43.179 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:43.179 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:43.179 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:43.179 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-09-20T22:54:43.179 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:54:43.179 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:43.179 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:54:43.179 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.154:6815", 2024-09-20T22:54:43.179 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3717704491 2024-09-20T22:54:43.179 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:43.179 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:43.179 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:43.180 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-09-20T22:54:43.180 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:54:43.180 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:43.180 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:54:43.180 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.154:6814", 2024-09-20T22:54:43.180 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3717704491 2024-09-20T22:54:43.180 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:43.180 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:43.180 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:43.180 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.154:6812/3717704491", 2024-09-20T22:54:43.180 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.154:6813/3717704491", 2024-09-20T22:54:43.180 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.154:6815/3717704491", 2024-09-20T22:54:43.181 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.154:6814/3717704491", 2024-09-20T22:54:43.181 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-09-20T22:54:43.181 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-09-20T22:54:43.181 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-09-20T22:54:43.181 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:43.181 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:43.181 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2024-09-20T22:54:43.181 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_xinfo": [ 2024-09-20T22:54:43.181 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:43.181 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 0, 2024-09-20T22:54:43.181 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-09-20T22:54:43.181 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-09-20T22:54:43.181 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-09-20T22:54:43.181 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540701547738038271, 2024-09-20T22:54:43.182 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-09-20T22:54:43.182 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-09-20T22:54:06.520163+0000", 2024-09-20T22:54:43.182 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-09-20T22:54:43.182 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:43.182 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:43.182 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 1, 2024-09-20T22:54:43.182 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-09-20T22:54:43.182 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-09-20T22:54:43.182 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-09-20T22:54:43.182 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540701547738038271, 2024-09-20T22:54:43.182 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-09-20T22:54:43.182 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-09-20T22:53:42.234428+0000", 2024-09-20T22:54:43.182 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-09-20T22:54:43.182 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:43.183 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:43.183 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 2, 2024-09-20T22:54:43.183 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-09-20T22:54:43.183 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-09-20T22:54:43.183 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-09-20T22:54:43.183 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540701547738038271, 2024-09-20T22:54:43.183 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-09-20T22:54:43.183 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-09-20T22:53:33.270476+0000", 2024-09-20T22:54:43.183 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-09-20T22:54:43.183 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:43.183 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:43.183 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 3, 2024-09-20T22:54:43.183 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-09-20T22:54:43.183 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-09-20T22:54:43.183 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-09-20T22:54:43.184 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540701547738038271, 2024-09-20T22:54:43.184 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-09-20T22:54:43.184 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-09-20T22:54:29.696694+0000", 2024-09-20T22:54:43.184 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-09-20T22:54:43.184 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:43.184 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:43.184 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 4, 2024-09-20T22:54:43.184 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-09-20T22:54:43.184 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-09-20T22:54:43.184 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-09-20T22:54:43.184 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540701547738038271, 2024-09-20T22:54:43.184 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-09-20T22:54:43.184 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-09-20T22:54:06.316620+0000", 2024-09-20T22:54:43.184 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-09-20T22:54:43.185 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:43.185 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:43.185 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 5, 2024-09-20T22:54:43.185 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-09-20T22:54:43.185 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-09-20T22:54:43.185 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-09-20T22:54:43.185 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540701547738038271, 2024-09-20T22:54:43.185 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-09-20T22:54:43.185 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-09-20T22:53:26.920706+0000", 2024-09-20T22:54:43.185 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-09-20T22:54:43.185 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:43.185 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:43.185 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 6, 2024-09-20T22:54:43.186 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-09-20T22:54:43.186 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-09-20T22:54:43.186 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-09-20T22:54:43.186 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540701547738038271, 2024-09-20T22:54:43.186 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-09-20T22:54:43.186 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-09-20T22:53:20.439087+0000", 2024-09-20T22:54:43.186 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-09-20T22:54:43.186 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:43.186 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:43.186 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 7, 2024-09-20T22:54:43.186 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-09-20T22:54:43.186 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-09-20T22:54:43.186 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-09-20T22:54:43.186 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540701547738038271, 2024-09-20T22:54:43.187 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-09-20T22:54:43.187 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-09-20T22:54:00.021655+0000", 2024-09-20T22:54:43.187 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-09-20T22:54:43.187 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:43.187 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2024-09-20T22:54:43.187 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_upmap": [ 2024-09-20T22:54:43.187 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:43.187 INFO:teuthology.orchestra.run.smithi084.stdout: "pgid": "1.9", 2024-09-20T22:54:43.187 INFO:teuthology.orchestra.run.smithi084.stdout: "osds": [ 2024-09-20T22:54:43.187 INFO:teuthology.orchestra.run.smithi084.stdout: 7, 2024-09-20T22:54:43.187 INFO:teuthology.orchestra.run.smithi084.stdout: 3, 2024-09-20T22:54:43.187 INFO:teuthology.orchestra.run.smithi084.stdout: 5 2024-09-20T22:54:43.187 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:43.187 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:43.188 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:43.188 INFO:teuthology.orchestra.run.smithi084.stdout: "pgid": "6.e", 2024-09-20T22:54:43.188 INFO:teuthology.orchestra.run.smithi084.stdout: "osds": [ 2024-09-20T22:54:43.188 INFO:teuthology.orchestra.run.smithi084.stdout: 1, 2024-09-20T22:54:43.188 INFO:teuthology.orchestra.run.smithi084.stdout: 2, 2024-09-20T22:54:43.188 INFO:teuthology.orchestra.run.smithi084.stdout: 5 2024-09-20T22:54:43.188 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:43.188 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:43.188 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:43.188 INFO:teuthology.orchestra.run.smithi084.stdout: "pgid": "6.1e", 2024-09-20T22:54:43.188 INFO:teuthology.orchestra.run.smithi084.stdout: "osds": [ 2024-09-20T22:54:43.188 INFO:teuthology.orchestra.run.smithi084.stdout: 0, 2024-09-20T22:54:43.188 INFO:teuthology.orchestra.run.smithi084.stdout: 1, 2024-09-20T22:54:43.188 INFO:teuthology.orchestra.run.smithi084.stdout: 3 2024-09-20T22:54:43.188 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:43.189 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:43.189 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2024-09-20T22:54:43.189 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_upmap_items": [ 2024-09-20T22:54:43.189 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:43.189 INFO:teuthology.orchestra.run.smithi084.stdout: "pgid": "1.0", 2024-09-20T22:54:43.189 INFO:teuthology.orchestra.run.smithi084.stdout: "mappings": [ 2024-09-20T22:54:43.189 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:43.189 INFO:teuthology.orchestra.run.smithi084.stdout: "from": 0, 2024-09-20T22:54:43.189 INFO:teuthology.orchestra.run.smithi084.stdout: "to": 7 2024-09-20T22:54:43.189 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:43.189 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:43.189 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:43.189 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:43.189 INFO:teuthology.orchestra.run.smithi084.stdout: "pgid": "3.a", 2024-09-20T22:54:43.190 INFO:teuthology.orchestra.run.smithi084.stdout: "mappings": [ 2024-09-20T22:54:43.190 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:43.190 INFO:teuthology.orchestra.run.smithi084.stdout: "from": 6, 2024-09-20T22:54:43.190 INFO:teuthology.orchestra.run.smithi084.stdout: "to": 2 2024-09-20T22:54:43.190 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:43.190 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:43.190 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:43.190 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:43.190 INFO:teuthology.orchestra.run.smithi084.stdout: "pgid": "4.9", 2024-09-20T22:54:43.190 INFO:teuthology.orchestra.run.smithi084.stdout: "mappings": [ 2024-09-20T22:54:43.190 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:43.190 INFO:teuthology.orchestra.run.smithi084.stdout: "from": 3, 2024-09-20T22:54:43.190 INFO:teuthology.orchestra.run.smithi084.stdout: "to": 2 2024-09-20T22:54:43.191 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:43.191 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:43.191 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:43.191 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:43.191 INFO:teuthology.orchestra.run.smithi084.stdout: "pgid": "4.11", 2024-09-20T22:54:43.191 INFO:teuthology.orchestra.run.smithi084.stdout: "mappings": [ 2024-09-20T22:54:43.191 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:43.191 INFO:teuthology.orchestra.run.smithi084.stdout: "from": 3, 2024-09-20T22:54:43.191 INFO:teuthology.orchestra.run.smithi084.stdout: "to": 5 2024-09-20T22:54:43.191 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:43.191 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:43.191 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:43.191 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2024-09-20T22:54:43.191 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_upmap_primaries": [], 2024-09-20T22:54:43.192 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_temp": [ 2024-09-20T22:54:43.192 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:43.192 INFO:teuthology.orchestra.run.smithi084.stdout: "pgid": "7.2", 2024-09-20T22:54:43.192 INFO:teuthology.orchestra.run.smithi084.stdout: "osds": [ 2024-09-20T22:54:43.192 INFO:teuthology.orchestra.run.smithi084.stdout: 3, 2024-09-20T22:54:43.192 INFO:teuthology.orchestra.run.smithi084.stdout: 2, 2024-09-20T22:54:43.192 INFO:teuthology.orchestra.run.smithi084.stdout: 7 2024-09-20T22:54:43.192 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:43.192 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:43.192 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2024-09-20T22:54:43.192 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_temp": [], 2024-09-20T22:54:43.192 INFO:teuthology.orchestra.run.smithi084.stdout: "blocklist": { 2024-09-20T22:54:43.192 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/542020545": "2024-09-21T22:42:07.596045+0000", 2024-09-20T22:54:43.193 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/25873236": "2024-09-21T22:36:54.917812+0000", 2024-09-20T22:54:43.193 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:6800/820908470": "2024-09-21T22:36:54.917812+0000", 2024-09-20T22:54:43.193 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/567614908": "2024-09-21T22:36:54.917812+0000", 2024-09-20T22:54:43.193 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/2963622459": "2024-09-21T22:42:07.596045+0000", 2024-09-20T22:54:43.193 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/1176849276": "2024-09-21T22:37:17.690104+0000", 2024-09-20T22:54:43.193 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:6800/2932420050": "2024-09-21T22:37:17.690104+0000", 2024-09-20T22:54:43.193 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/3879263635": "2024-09-21T22:37:17.690104+0000", 2024-09-20T22:54:43.193 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/969472279": "2024-09-21T22:36:54.917812+0000", 2024-09-20T22:54:43.193 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/1361143214": "2024-09-21T22:37:17.690104+0000", 2024-09-20T22:54:43.193 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/3353126955": "2024-09-21T22:42:07.596045+0000", 2024-09-20T22:54:43.193 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:6800/4024278029": "2024-09-21T22:42:07.596045+0000", 2024-09-20T22:54:43.193 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/547902262": "2024-09-21T22:42:07.596045+0000", 2024-09-20T22:54:43.193 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/1242929317": "2024-09-21T22:42:07.596045+0000" 2024-09-20T22:54:43.194 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:43.194 INFO:teuthology.orchestra.run.smithi084.stdout: "range_blocklist": {}, 2024-09-20T22:54:43.194 INFO:teuthology.orchestra.run.smithi084.stdout: "erasure_code_profiles": { 2024-09-20T22:54:43.194 INFO:teuthology.orchestra.run.smithi084.stdout: "default": { 2024-09-20T22:54:43.194 INFO:teuthology.orchestra.run.smithi084.stdout: "crush-failure-domain": "osd", 2024-09-20T22:54:43.194 INFO:teuthology.orchestra.run.smithi084.stdout: "k": "2", 2024-09-20T22:54:43.194 INFO:teuthology.orchestra.run.smithi084.stdout: "m": "1", 2024-09-20T22:54:43.194 INFO:teuthology.orchestra.run.smithi084.stdout: "plugin": "jerasure", 2024-09-20T22:54:43.194 INFO:teuthology.orchestra.run.smithi084.stdout: "technique": "reed_sol_van" 2024-09-20T22:54:43.194 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:43.194 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:43.194 INFO:teuthology.orchestra.run.smithi084.stdout: "removed_snaps_queue": [ 2024-09-20T22:54:43.194 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:43.194 INFO:teuthology.orchestra.run.smithi084.stdout: "pool": 7, 2024-09-20T22:54:43.195 INFO:teuthology.orchestra.run.smithi084.stdout: "snaps": [ 2024-09-20T22:54:43.195 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:43.195 INFO:teuthology.orchestra.run.smithi084.stdout: "begin": 287, 2024-09-20T22:54:43.195 INFO:teuthology.orchestra.run.smithi084.stdout: "length": 5 2024-09-20T22:54:43.195 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:43.195 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:43.195 INFO:teuthology.orchestra.run.smithi084.stdout: "begin": 293, 2024-09-20T22:54:43.195 INFO:teuthology.orchestra.run.smithi084.stdout: "length": 9 2024-09-20T22:54:43.195 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:43.195 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:43.195 INFO:teuthology.orchestra.run.smithi084.stdout: "begin": 304, 2024-09-20T22:54:43.195 INFO:teuthology.orchestra.run.smithi084.stdout: "length": 9 2024-09-20T22:54:43.195 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:43.195 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:43.196 INFO:teuthology.orchestra.run.smithi084.stdout: "begin": 315, 2024-09-20T22:54:43.196 INFO:teuthology.orchestra.run.smithi084.stdout: "length": 3 2024-09-20T22:54:43.196 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:43.196 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:43.196 INFO:teuthology.orchestra.run.smithi084.stdout: "begin": 319, 2024-09-20T22:54:43.196 INFO:teuthology.orchestra.run.smithi084.stdout: "length": 3 2024-09-20T22:54:43.196 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:43.196 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:43.196 INFO:teuthology.orchestra.run.smithi084.stdout: "begin": 323, 2024-09-20T22:54:43.196 INFO:teuthology.orchestra.run.smithi084.stdout: "length": 2 2024-09-20T22:54:43.196 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:43.196 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:43.196 INFO:teuthology.orchestra.run.smithi084.stdout: "begin": 326, 2024-09-20T22:54:43.196 INFO:teuthology.orchestra.run.smithi084.stdout: "length": 3 2024-09-20T22:54:43.197 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:43.197 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:43.197 INFO:teuthology.orchestra.run.smithi084.stdout: "begin": 331, 2024-09-20T22:54:43.197 INFO:teuthology.orchestra.run.smithi084.stdout: "length": 3 2024-09-20T22:54:43.197 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:43.197 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:43.197 INFO:teuthology.orchestra.run.smithi084.stdout: "begin": 336, 2024-09-20T22:54:43.197 INFO:teuthology.orchestra.run.smithi084.stdout: "length": 3 2024-09-20T22:54:43.197 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:43.197 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:43.197 INFO:teuthology.orchestra.run.smithi084.stdout: "begin": 340, 2024-09-20T22:54:43.197 INFO:teuthology.orchestra.run.smithi084.stdout: "length": 1 2024-09-20T22:54:43.197 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:43.197 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:43.197 INFO:teuthology.orchestra.run.smithi084.stdout: "begin": 345, 2024-09-20T22:54:43.198 INFO:teuthology.orchestra.run.smithi084.stdout: "length": 1 2024-09-20T22:54:43.198 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:43.198 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:43.198 INFO:teuthology.orchestra.run.smithi084.stdout: "begin": 353, 2024-09-20T22:54:43.198 INFO:teuthology.orchestra.run.smithi084.stdout: "length": 1 2024-09-20T22:54:43.198 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:43.198 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:43.198 INFO:teuthology.orchestra.run.smithi084.stdout: "begin": 355, 2024-09-20T22:54:43.198 INFO:teuthology.orchestra.run.smithi084.stdout: "length": 2 2024-09-20T22:54:43.198 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:43.198 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:43.198 INFO:teuthology.orchestra.run.smithi084.stdout: "begin": 360, 2024-09-20T22:54:43.198 INFO:teuthology.orchestra.run.smithi084.stdout: "length": 1 2024-09-20T22:54:43.199 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:43.199 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:43.199 INFO:teuthology.orchestra.run.smithi084.stdout: "begin": 362, 2024-09-20T22:54:43.199 INFO:teuthology.orchestra.run.smithi084.stdout: "length": 1 2024-09-20T22:54:43.199 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:43.199 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:43.199 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:43.199 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2024-09-20T22:54:43.199 INFO:teuthology.orchestra.run.smithi084.stdout: "new_removed_snaps": [], 2024-09-20T22:54:43.199 INFO:teuthology.orchestra.run.smithi084.stdout: "new_purged_snaps": [], 2024-09-20T22:54:43.199 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_node_flags": {}, 2024-09-20T22:54:43.199 INFO:teuthology.orchestra.run.smithi084.stdout: "device_class_flags": {}, 2024-09-20T22:54:43.199 INFO:teuthology.orchestra.run.smithi084.stdout: "stretch_mode": { 2024-09-20T22:54:43.199 INFO:teuthology.orchestra.run.smithi084.stdout: "stretch_mode_enabled": false, 2024-09-20T22:54:43.199 INFO:teuthology.orchestra.run.smithi084.stdout: "stretch_bucket_count": 0, 2024-09-20T22:54:43.200 INFO:teuthology.orchestra.run.smithi084.stdout: "degraded_stretch_mode": 0, 2024-09-20T22:54:43.200 INFO:teuthology.orchestra.run.smithi084.stdout: "recovering_stretch_mode": 0, 2024-09-20T22:54:43.200 INFO:teuthology.orchestra.run.smithi084.stdout: "stretch_mode_bucket": 0 2024-09-20T22:54:43.200 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:43.200 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-20T22:54:43.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:43.221 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:43.221 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:54:43.322 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:54:43.344 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:43.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:43.345 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:54:43.419 INFO:tasks.rados.rados.0.smithi084.stdout:3200: done (1 left) 2024-09-20T22:54:43.419 INFO:tasks.rados.rados.0.smithi084.stdout:3201: read oid 12 snap 320 2024-09-20T22:54:43.419 INFO:tasks.rados.rados.0.smithi084.stdout:3201: expect deleted 2024-09-20T22:54:43.419 INFO:tasks.rados.rados.0.smithi084.stdout:3202: copy_from oid 36 from oid 27 current snap is 366 2024-09-20T22:54:43.420 INFO:tasks.rados.rados.0.smithi084.stdout:3203: read oid 39 snap 339 2024-09-20T22:54:43.420 INFO:tasks.rados.rados.0.smithi084.stdout:3203: expect deleted 2024-09-20T22:54:43.420 INFO:tasks.rados.rados.0.smithi084.stdout:3204: snap_create 2024-09-20T22:54:43.420 INFO:tasks.rados.rados.0.smithi084.stdout:3199: finishing rollback tid 0 to smithi08477203-24 2024-09-20T22:54:43.420 INFO:tasks.rados.rados.0.smithi084.stdout:3199: finishing rollback tid 1 to smithi08477203-24 2024-09-20T22:54:43.420 INFO:tasks.rados.rados.0.smithi084.stdout:3199: finishing rollback tid 2 to smithi08477203-24 2024-09-20T22:54:43.420 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 24 v 1883 (ObjNum 780 snap 252 seq_num 780) dirty exists 2024-09-20T22:54:43.421 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is unset 2024-09-20T22:54:43.430 INFO:tasks.rados.rados.0.smithi084.stdout:3202: finishing copy_from racing read to smithi08477203-36 2024-09-20T22:54:43.435 INFO:tasks.rados.rados.0.smithi084.stdout:3202: finishing copy_from to smithi08477203-36 2024-09-20T22:54:43.435 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 36 v 1813 (ObjNum 1077 snap 358 seq_num 1077) dirty exists 2024-09-20T22:54:43.446 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:54:43.467 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:43.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:43.469 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:54:43.570 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:54:43.591 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:43.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:43.592 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:54:43.693 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:54:43.706 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-20T22:54:43.715 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:43.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:43.715 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:54:43.728 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:43 smithi084 ceph-mon[40024]: 3.11 scrub starts 2024-09-20T22:54:43.728 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:43 smithi084 ceph-mon[40024]: 3.11 scrub ok 2024-09-20T22:54:43.728 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:43 smithi084 ceph-mon[40024]: pgmap v1104: 158 pgs: 5 active+clean+snaptrim, 1 active+remapped+backfill_toofull, 152 active+clean; 569 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 6.6 MiB/s rd, 0 B/s wr, 5 op/s; 16/1371 objects misplaced (1.167%) 2024-09-20T22:54:43.728 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:43 smithi084 ceph-mon[40024]: osdmap e802: 8 total, 8 up, 8 in 2024-09-20T22:54:43.729 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:43 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/3189773203' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:54:43.729 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:43 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:54:43.729 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:43 smithi084 ceph-mon[40024]: 6.e scrub starts 2024-09-20T22:54:43.729 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:43 smithi084 ceph-mon[40024]: 6.e scrub ok 2024-09-20T22:54:43.729 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:43 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/2369875134' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-20T22:54:43.729 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:43 smithi084 ceph-mon[40024]: 4.18 scrub starts 2024-09-20T22:54:43.729 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:43 smithi084 ceph-mon[40024]: 4.18 scrub ok 2024-09-20T22:54:43.729 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:43 smithi084 ceph-mon[29774]: 3.11 scrub starts 2024-09-20T22:54:43.729 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:43 smithi084 ceph-mon[29774]: 3.11 scrub ok 2024-09-20T22:54:43.729 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:43 smithi084 ceph-mon[29774]: pgmap v1104: 158 pgs: 5 active+clean+snaptrim, 1 active+remapped+backfill_toofull, 152 active+clean; 569 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 6.6 MiB/s rd, 0 B/s wr, 5 op/s; 16/1371 objects misplaced (1.167%) 2024-09-20T22:54:43.729 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:43 smithi084 ceph-mon[29774]: osdmap e802: 8 total, 8 up, 8 in 2024-09-20T22:54:43.729 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:43 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/3189773203' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:54:43.730 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:43 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:54:43.730 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:43 smithi084 ceph-mon[29774]: 6.e scrub starts 2024-09-20T22:54:43.730 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:43 smithi084 ceph-mon[29774]: 6.e scrub ok 2024-09-20T22:54:43.730 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:43 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/2369875134' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-20T22:54:43.730 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:43 smithi084 ceph-mon[29774]: 4.18 scrub starts 2024-09-20T22:54:43.730 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:43 smithi084 ceph-mon[29774]: 4.18 scrub ok 2024-09-20T22:54:43.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:43 smithi154 ceph-mon[33264]: 3.11 scrub starts 2024-09-20T22:54:43.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:43 smithi154 ceph-mon[33264]: 3.11 scrub ok 2024-09-20T22:54:43.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:43 smithi154 ceph-mon[33264]: pgmap v1104: 158 pgs: 5 active+clean+snaptrim, 1 active+remapped+backfill_toofull, 152 active+clean; 569 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 6.6 MiB/s rd, 0 B/s wr, 5 op/s; 16/1371 objects misplaced (1.167%) 2024-09-20T22:54:43.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:43 smithi154 ceph-mon[33264]: osdmap e802: 8 total, 8 up, 8 in 2024-09-20T22:54:43.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:43 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/3189773203' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:54:43.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:43 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:54:43.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:43 smithi154 ceph-mon[33264]: 6.e scrub starts 2024-09-20T22:54:43.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:43 smithi154 ceph-mon[33264]: 6.e scrub ok 2024-09-20T22:54:43.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:43 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/2369875134' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-20T22:54:43.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:43 smithi154 ceph-mon[33264]: 4.18 scrub starts 2024-09-20T22:54:43.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:43 smithi154 ceph-mon[33264]: 4.18 scrub ok 2024-09-20T22:54:43.816 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:54:43.837 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:43.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:43.838 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:54:43.939 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:54:43.960 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:43.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:43.961 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:54:44.062 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:54:44.084 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:44.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:44.085 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:54:44.167 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.2 dump_historic_ops 2024-09-20T22:54:44.186 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:54:44.231 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:44.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:44.232 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:54:44.333 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:54:44.354 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:44.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:44.355 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:54:44.424 INFO:tasks.rados.rados.0.smithi084.stdout:3199: done (4 left) 2024-09-20T22:54:44.424 INFO:tasks.rados.rados.0.smithi084.stdout:3201: done (3 left) 2024-09-20T22:54:44.424 INFO:tasks.rados.rados.0.smithi084.stdout:3202: done (2 left) 2024-09-20T22:54:44.424 INFO:tasks.rados.rados.0.smithi084.stdout:3203: done (1 left) 2024-09-20T22:54:44.424 INFO:tasks.rados.rados.0.smithi084.stdout:3204: done (0 left) 2024-09-20T22:54:44.424 INFO:tasks.rados.rados.0.smithi084.stdout:3205: snap_remove snap 347 2024-09-20T22:54:44.457 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:54:44.478 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:44.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:44.479 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:54:44.580 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:54:44.601 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:44.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:44.602 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:54:44.703 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:54:44.724 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:44.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:44.725 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:54:44.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:44 smithi154 ceph-mon[33264]: 6.a scrub starts 2024-09-20T22:54:44.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:44 smithi154 ceph-mon[33264]: 6.a scrub ok 2024-09-20T22:54:44.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:44 smithi154 ceph-mon[33264]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T22:54:44.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:44 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T22:54:44.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:44 smithi154 ceph-mon[33264]: osdmap e803: 8 total, 8 up, 8 in 2024-09-20T22:54:44.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:44 smithi154 ceph-mon[33264]: 5.1e deep-scrub starts 2024-09-20T22:54:44.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:44 smithi154 ceph-mon[33264]: 5.1e deep-scrub ok 2024-09-20T22:54:44.770 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:44 smithi084 ceph-mon[40024]: 6.a scrub starts 2024-09-20T22:54:44.770 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:44 smithi084 ceph-mon[40024]: 6.a scrub ok 2024-09-20T22:54:44.770 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:44 smithi084 ceph-mon[40024]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T22:54:44.770 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:44 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T22:54:44.770 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:44 smithi084 ceph-mon[40024]: osdmap e803: 8 total, 8 up, 8 in 2024-09-20T22:54:44.770 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:44 smithi084 ceph-mon[40024]: 5.1e deep-scrub starts 2024-09-20T22:54:44.770 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:44 smithi084 ceph-mon[40024]: 5.1e deep-scrub ok 2024-09-20T22:54:44.770 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:44 smithi084 ceph-mon[29774]: 6.a scrub starts 2024-09-20T22:54:44.770 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:44 smithi084 ceph-mon[29774]: 6.a scrub ok 2024-09-20T22:54:44.770 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:44 smithi084 ceph-mon[29774]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T22:54:44.771 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:44 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T22:54:44.771 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:44 smithi084 ceph-mon[29774]: osdmap e803: 8 total, 8 up, 8 in 2024-09-20T22:54:44.771 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:44 smithi084 ceph-mon[29774]: 5.1e deep-scrub starts 2024-09-20T22:54:44.771 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:44 smithi084 ceph-mon[29774]: 5.1e deep-scrub ok 2024-09-20T22:54:44.799 INFO:tasks.thrashosds.thrasher:j is {'epoch': 802, 'fsid': 'a9c0bc70-77a0-11ef-baf5-efdc52797490', 'created': '2024-09-20T22:36:30.240626+0000', 'modified': '2024-09-20T22:54:42.393575+0000', 'last_up_change': '2024-09-20T22:41:27.098565+0000', 'last_in_change': '2024-09-20T22:54:16.306377+0000', 'flags': 'nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5804032, 'flags_set': ['nodeep-scrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 18, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 7, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-20T22:39:21.984381+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 11, 'pg_placement_num': 11, 'pg_placement_num_target': 11, 'pg_num_target': 11, 'pg_num_pending': 11, 'last_pg_merge_meta': {'source_pgid': '1.b', 'ready_epoch': 299, 'last_epoch_started': 298, 'last_epoch_clean': 298, 'source_version': "77'128", 'target_version': "77'128"}, 'last_change': '769', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '299', 'last_force_op_resend_preluminous': '297', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.180000066757202, 'score_stable': 2.180000066757202, 'optimal_score': 1, 'raw_score_acting': 2.180000066757202, 'raw_score_stable': 2.180000066757202, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-09-20T22:41:35.820081+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '69', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 69, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.6500000953674316, 'score_stable': 2.6500000953674316, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 2.3299999237060547, 'raw_score_stable': 2.3299999237060547, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-09-20T22:41:36.546389+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-09-20T22:41:38.454640+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '70', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-09-20T22:41:40.407129+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '72', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-09-20T22:41:42.476716+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '74', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'unique_pool_0', 'create_time': '2024-09-20T22:42:51.354358+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '7.10', 'ready_epoch': 484, 'last_epoch_started': 483, 'last_epoch_clean': 483, 'source_version': "480'172", 'target_version': "480'253"}, 'last_change': '802', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '484', 'last_force_op_resend_preluminous': '482', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 367, 'snap_epoch': 802, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': 'ea36a489-6f91-4b2f-b784-b90506b9805d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 779, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6801', 'nonce': 268084626}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6802', 'nonce': 268084626}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6804', 'nonce': 268084626}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6803', 'nonce': 268084626}]}, 'public_addr': '172.21.15.84:6801/268084626', 'cluster_addr': '172.21.15.84:6802/268084626', 'heartbeat_back_addr': '172.21.15.84:6804/268084626', 'heartbeat_front_addr': '172.21.15.84:6803/268084626', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'c8649d2f-27ef-47f8-bc94-f1cd9e62bdb8', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 779, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6805', 'nonce': 418374136}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6806', 'nonce': 418374136}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6808', 'nonce': 418374136}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6807', 'nonce': 418374136}]}, 'public_addr': '172.21.15.84:6805/418374136', 'cluster_addr': '172.21.15.84:6806/418374136', 'heartbeat_back_addr': '172.21.15.84:6808/418374136', 'heartbeat_front_addr': '172.21.15.84:6807/418374136', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'c728eb48-a0f0-4de4-a651-7f3bec867280', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 778, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6809', 'nonce': 3977168611}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6810', 'nonce': 3977168611}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6812', 'nonce': 3977168611}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6811', 'nonce': 3977168611}]}, 'public_addr': '172.21.15.84:6809/3977168611', 'cluster_addr': '172.21.15.84:6810/3977168611', 'heartbeat_back_addr': '172.21.15.84:6812/3977168611', 'heartbeat_front_addr': '172.21.15.84:6811/3977168611', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '573f44eb-0853-4630-a7d8-9caaeb256f9b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 25, 'up_thru': 777, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6813', 'nonce': 2192239945}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6814', 'nonce': 2192239945}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6816', 'nonce': 2192239945}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6815', 'nonce': 2192239945}]}, 'public_addr': '172.21.15.84:6813/2192239945', 'cluster_addr': '172.21.15.84:6814/2192239945', 'heartbeat_back_addr': '172.21.15.84:6816/2192239945', 'heartbeat_front_addr': '172.21.15.84:6815/2192239945', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'a52c3b94-c5f3-490f-8ac8-09061b9b2abb', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 778, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6800', 'nonce': 1793136002}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6801', 'nonce': 1793136002}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6803', 'nonce': 1793136002}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6802', 'nonce': 1793136002}]}, 'public_addr': '172.21.15.154:6800/1793136002', 'cluster_addr': '172.21.15.154:6801/1793136002', 'heartbeat_back_addr': '172.21.15.154:6803/1793136002', 'heartbeat_front_addr': '172.21.15.154:6802/1793136002', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '4fa0dc8e-25cc-46cd-96ff-33f20b6faec3', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 40, 'up_thru': 779, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6804', 'nonce': 493899927}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6805', 'nonce': 493899927}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6807', 'nonce': 493899927}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6806', 'nonce': 493899927}]}, 'public_addr': '172.21.15.154:6804/493899927', 'cluster_addr': '172.21.15.154:6805/493899927', 'heartbeat_back_addr': '172.21.15.154:6807/493899927', 'heartbeat_front_addr': '172.21.15.154:6806/493899927', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '56788d74-810e-4827-a615-958c12f8a368', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 48, 'up_thru': 779, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6808', 'nonce': 2375091033}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6809', 'nonce': 2375091033}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6811', 'nonce': 2375091033}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6810', 'nonce': 2375091033}]}, 'public_addr': '172.21.15.154:6808/2375091033', 'cluster_addr': '172.21.15.154:6809/2375091033', 'heartbeat_back_addr': '172.21.15.154:6811/2375091033', 'heartbeat_front_addr': '172.21.15.154:6810/2375091033', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '83503727-afc1-4b95-8b96-ecb63f3338c2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 59, 'up_thru': 776, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6812', 'nonce': 3717704491}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6813', 'nonce': 3717704491}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6815', 'nonce': 3717704491}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6814', 'nonce': 3717704491}]}, 'public_addr': '172.21.15.154:6812/3717704491', 'cluster_addr': '172.21.15.154:6813/3717704491', 'heartbeat_back_addr': '172.21.15.154:6815/3717704491', 'heartbeat_front_addr': '172.21.15.154:6814/3717704491', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-20T22:54:06.520163+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-20T22:53:42.234428+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-20T22:53:33.270476+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-20T22:54:29.696694+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-20T22:54:06.316620+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-20T22:53:26.920706+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-20T22:53:20.439087+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-20T22:54:00.021655+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '1.9', 'osds': [7, 3, 5]}, {'pgid': '6.e', 'osds': [1, 2, 5]}, {'pgid': '6.1e', 'osds': [0, 1, 3]}], 'pg_upmap_items': [{'pgid': '1.0', 'mappings': [{'from': 0, 'to': 7}]}, {'pgid': '3.a', 'mappings': [{'from': 6, 'to': 2}]}, {'pgid': '4.9', 'mappings': [{'from': 3, 'to': 2}]}, {'pgid': '4.11', 'mappings': [{'from': 3, 'to': 5}]}], 'pg_upmap_primaries': [], 'pg_temp': [{'pgid': '7.2', 'osds': [3, 2, 7]}], 'primary_temp': [], 'blocklist': {'172.21.15.84:0/542020545': '2024-09-21T22:42:07.596045+0000', '172.21.15.84:0/25873236': '2024-09-21T22:36:54.917812+0000', '172.21.15.84:6800/820908470': '2024-09-21T22:36:54.917812+0000', '172.21.15.84:0/567614908': '2024-09-21T22:36:54.917812+0000', '172.21.15.84:0/2963622459': '2024-09-21T22:42:07.596045+0000', '172.21.15.84:0/1176849276': '2024-09-21T22:37:17.690104+0000', '172.21.15.84:6800/2932420050': '2024-09-21T22:37:17.690104+0000', '172.21.15.84:0/3879263635': '2024-09-21T22:37:17.690104+0000', '172.21.15.84:0/969472279': '2024-09-21T22:36:54.917812+0000', '172.21.15.84:0/1361143214': '2024-09-21T22:37:17.690104+0000', '172.21.15.84:0/3353126955': '2024-09-21T22:42:07.596045+0000', '172.21.15.84:6800/4024278029': '2024-09-21T22:42:07.596045+0000', '172.21.15.84:0/547902262': '2024-09-21T22:42:07.596045+0000', '172.21.15.84:0/1242929317': '2024-09-21T22:42:07.596045+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [{'pool': 7, 'snaps': [{'begin': 287, 'length': 5}, {'begin': 293, 'length': 9}, {'begin': 304, 'length': 9}, {'begin': 315, 'length': 3}, {'begin': 319, 'length': 3}, {'begin': 323, 'length': 2}, {'begin': 326, 'length': 3}, {'begin': 331, 'length': 3}, {'begin': 336, 'length': 3}, {'begin': 340, 'length': 1}, {'begin': 345, 'length': 1}, {'begin': 353, 'length': 1}, {'begin': 355, 'length': 2}, {'begin': 360, 'length': 1}, {'begin': 362, 'length': 1}]}], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-20T22:54:44.799 INFO:tasks.thrashosds.thrasher:Clearing pg_upmap on 6.1e 2024-09-20T22:54:44.800 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd rm-pg-upmap 6.1e 2024-09-20T22:54:44.826 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:54:44.847 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:44.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:44.849 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:54:44.949 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:54:44.970 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:44.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:44.971 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:54:45.072 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:54:45.094 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:45.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:45.095 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:54:45.195 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:54:45.217 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:45.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:45.218 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:54:45.319 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:54:45.340 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:45.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:45.341 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:54:45.426 INFO:tasks.rados.rados.0.smithi084.stdout:3205: done (0 left) 2024-09-20T22:54:45.426 INFO:tasks.rados.rados.0.smithi084.stdout:3206: snap_remove snap 359 2024-09-20T22:54:45.442 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:54:45.463 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:45.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:45.464 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:54:45.565 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:54:45.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:45.587 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:45.587 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:54:45.635 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:54:45.636 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:54:45.670 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:54:45.688 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:54:45.709 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:45.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:45.710 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:54:45.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:45 smithi154 ceph-mon[33264]: pgmap v1107: 158 pgs: 4 active+clean+snaptrim, 1 active+remapped+backfill_toofull, 153 active+clean; 582 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 19 MiB/s rd, 2.0 MiB/s wr, 23 op/s; 16/1407 objects misplaced (1.137%) 2024-09-20T22:54:45.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:45 smithi154 ceph-mon[33264]: osdmap e804: 8 total, 8 up, 8 in 2024-09-20T22:54:45.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:45 smithi154 ceph-mon[33264]: 1.3 scrub starts 2024-09-20T22:54:45.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:45 smithi154 ceph-mon[33264]: 1.3 scrub ok 2024-09-20T22:54:45.759 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:45 smithi084 ceph-mon[29774]: pgmap v1107: 158 pgs: 4 active+clean+snaptrim, 1 active+remapped+backfill_toofull, 153 active+clean; 582 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 19 MiB/s rd, 2.0 MiB/s wr, 23 op/s; 16/1407 objects misplaced (1.137%) 2024-09-20T22:54:45.759 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:45 smithi084 ceph-mon[29774]: osdmap e804: 8 total, 8 up, 8 in 2024-09-20T22:54:45.759 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:45 smithi084 ceph-mon[29774]: 1.3 scrub starts 2024-09-20T22:54:45.760 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:45 smithi084 ceph-mon[29774]: 1.3 scrub ok 2024-09-20T22:54:45.760 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:45 smithi084 ceph-mon[40024]: pgmap v1107: 158 pgs: 4 active+clean+snaptrim, 1 active+remapped+backfill_toofull, 153 active+clean; 582 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 19 MiB/s rd, 2.0 MiB/s wr, 23 op/s; 16/1407 objects misplaced (1.137%) 2024-09-20T22:54:45.760 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:45 smithi084 ceph-mon[40024]: osdmap e804: 8 total, 8 up, 8 in 2024-09-20T22:54:45.760 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:45 smithi084 ceph-mon[40024]: 1.3 scrub starts 2024-09-20T22:54:45.760 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:45 smithi084 ceph-mon[40024]: 1.3 scrub ok 2024-09-20T22:54:45.810 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:54:45.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:45.833 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:45.833 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:54:45.934 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:54:45.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:45.955 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:45.955 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:54:46.056 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:54:46.077 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:46.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:46.078 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:54:46.179 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:54:46.199 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:46.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:46.200 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:54:46.301 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:54:46.324 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:46.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:46.325 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:54:46.426 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:54:46.438 INFO:tasks.rados.rados.0.smithi084.stdout:3206: done (0 left) 2024-09-20T22:54:46.438 INFO:tasks.rados.rados.0.smithi084.stdout:3207: rollback oid 13 current snap is 367 2024-09-20T22:54:46.439 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 13 to 327 2024-09-20T22:54:46.439 INFO:tasks.rados.rados.0.smithi084.stdout:3208: copy_from oid 40 from oid 12 current snap is 367 2024-09-20T22:54:46.439 INFO:tasks.rados.rados.0.smithi084.stdout:3209: snap_remove snap 352 2024-09-20T22:54:46.454 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:46.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:46.455 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:54:46.549 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-20T22:54:46.549 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2024-09-20T22:54:46.549 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2024-09-20T22:54:46.549 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [ 2024-09-20T22:54:46.549 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.549 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24652.0:1799 7.17 e229/139 7:ea720b85:::smithi08477203-26:head v 229'168, mlcod=222'164)", 2024-09-20T22:54:46.550 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:45:08.171867+0000", 2024-09-20T22:54:46.550 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 578.37459280400003, 2024-09-20T22:54:46.550 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014913454, 2024-09-20T22:54:46.550 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:54:46.550 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:54:46.550 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:54:46.550 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:54:46.550 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.550 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:54:46.550 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.171867+0000", 2024-09-20T22:54:46.550 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:54:46.550 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.550 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.551 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:54:46.551 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.171867+0000", 2024-09-20T22:54:46.551 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:54:46.551 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.551 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.551 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:54:46.551 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.171867+0000", 2024-09-20T22:54:46.551 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.3799999999999999e-07 2024-09-20T22:54:46.551 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.551 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.551 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:54:46.551 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.173016+0000", 2024-09-20T22:54:46.551 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0011485740000000001 2024-09-20T22:54:46.551 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.552 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.552 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:54:46.552 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.173024+0000", 2024-09-20T22:54:46.552 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.4780000000000008e-06 2024-09-20T22:54:46.552 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.552 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.552 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:54:46.552 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.173034+0000", 2024-09-20T22:54:46.552 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.7860000000000008e-06 2024-09-20T22:54:46.552 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.552 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.552 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:54:46.552 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.173098+0000", 2024-09-20T22:54:46.552 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.4405000000000004e-05 2024-09-20T22:54:46.553 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.553 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.553 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:54:46.553 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.173125+0000", 2024-09-20T22:54:46.553 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.6950000000000001e-05 2024-09-20T22:54:46.553 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.553 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.553 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:54:46.553 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.186711+0000", 2024-09-20T22:54:46.553 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.01358608 2024-09-20T22:54:46.553 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.553 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.553 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:54:46.553 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:08.186780+0000", 2024-09-20T22:54:46.553 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.8942999999999994e-05 2024-09-20T22:54:46.554 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:46.554 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:46.554 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:46.554 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.554 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.554 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24652.0:2318 7.5 e258/137 7:a160b5d4:::smithi08477203-16:head v 258'43, mlcod=239'39)", 2024-09-20T22:54:46.554 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:45:37.192419+0000", 2024-09-20T22:54:46.554 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 549.35404071699998, 2024-09-20T22:54:46.554 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013974723, 2024-09-20T22:54:46.554 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:54:46.554 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:54:46.554 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:54:46.554 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:54:46.554 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.555 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:54:46.555 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.192419+0000", 2024-09-20T22:54:46.555 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:54:46.555 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.555 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.555 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:54:46.555 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.192419+0000", 2024-09-20T22:54:46.555 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:54:46.555 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.555 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.555 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:54:46.555 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.192420+0000", 2024-09-20T22:54:46.555 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.3799999999999997e-07 2024-09-20T22:54:46.556 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.556 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.556 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:54:46.556 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.204088+0000", 2024-09-20T22:54:46.556 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011668883 2024-09-20T22:54:46.556 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.556 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.556 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:54:46.556 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.204104+0000", 2024-09-20T22:54:46.556 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.5437000000000001e-05 2024-09-20T22:54:46.556 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.556 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.556 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:54:46.556 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.204112+0000", 2024-09-20T22:54:46.557 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.2519999999999999e-06 2024-09-20T22:54:46.557 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.557 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.557 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:54:46.557 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.204168+0000", 2024-09-20T22:54:46.557 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.5977999999999998e-05 2024-09-20T22:54:46.557 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.557 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.557 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:54:46.557 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.204188+0000", 2024-09-20T22:54:46.557 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.9400999999999999e-05 2024-09-20T22:54:46.557 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.557 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.557 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:54:46.557 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.206340+0000", 2024-09-20T22:54:46.558 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0021527159999999998 2024-09-20T22:54:46.558 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.558 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.558 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:54:46.558 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:37.206394+0000", 2024-09-20T22:54:46.558 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.3418000000000003e-05 2024-09-20T22:54:46.558 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:46.558 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:46.558 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:46.558 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.558 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.558 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(osd.6.0:726 7.19 e260/139 7:9eac4560:::smithi08477203-30:4c v 260'159, mlcod=259'156)", 2024-09-20T22:54:46.558 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:45:39.197025+0000", 2024-09-20T22:54:46.559 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 547.34943480699997, 2024-09-20T22:54:46.559 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.018150454, 2024-09-20T22:54:46.559 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:54:46.559 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:54:46.559 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:54:46.559 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:54:46.559 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.559 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:54:46.559 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.197025+0000", 2024-09-20T22:54:46.559 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:54:46.559 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.559 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.559 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:54:46.559 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.197025+0000", 2024-09-20T22:54:46.560 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:54:46.560 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.560 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.560 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:54:46.560 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.197025+0000", 2024-09-20T22:54:46.560 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.7599999999999998e-07 2024-09-20T22:54:46.560 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.560 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.560 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:54:46.560 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.197033+0000", 2024-09-20T22:54:46.560 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.9119999999999998e-06 2024-09-20T22:54:46.560 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.560 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.560 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:54:46.560 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.197042+0000", 2024-09-20T22:54:46.561 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.3349999999999996e-06 2024-09-20T22:54:46.561 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.561 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.561 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:54:46.561 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.197050+0000", 2024-09-20T22:54:46.561 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.7230000000000004e-06 2024-09-20T22:54:46.561 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.561 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.561 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:54:46.561 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.200756+0000", 2024-09-20T22:54:46.561 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0037059749999999998 2024-09-20T22:54:46.561 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.561 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.561 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:54:46.562 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.200778+0000", 2024-09-20T22:54:46.562 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.2140000000000001e-05 2024-09-20T22:54:46.562 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.562 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.562 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:54:46.562 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.215131+0000", 2024-09-20T22:54:46.562 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014353054000000001 2024-09-20T22:54:46.562 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.562 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.562 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:54:46.562 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:39.215175+0000", 2024-09-20T22:54:46.562 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.4039000000000003e-05 2024-09-20T22:54:46.562 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:46.562 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:46.562 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:46.563 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.563 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.563 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(osd.7.0:152 7.5 e269/260 7:a160b5d4:::smithi08477203-16:55 v 269'55, mlcod=263'51)", 2024-09-20T22:54:46.563 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:45:48.173893+0000", 2024-09-20T22:54:46.563 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 538.37256706599999, 2024-09-20T22:54:46.563 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014936935, 2024-09-20T22:54:46.563 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:54:46.563 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:54:46.563 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:54:46.564 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:54:46.565 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:54:46.565 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.565 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:54:46.565 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.173893+0000", 2024-09-20T22:54:46.565 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:54:46.565 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.565 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.565 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:54:46.565 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.173893+0000", 2024-09-20T22:54:46.565 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:54:46.565 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.565 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.565 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:54:46.566 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.173893+0000", 2024-09-20T22:54:46.566 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.8800000000000002e-07 2024-09-20T22:54:46.566 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.566 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.566 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:54:46.566 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.173916+0000", 2024-09-20T22:54:46.566 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.2889999999999999e-05 2024-09-20T22:54:46.566 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.566 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.566 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:54:46.566 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.173924+0000", 2024-09-20T22:54:46.566 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.4420000000000001e-06 2024-09-20T22:54:46.566 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.566 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.567 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:54:46.567 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.173932+0000", 2024-09-20T22:54:46.567 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.5299999999999996e-06 2024-09-20T22:54:46.567 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.567 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.567 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:54:46.567 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.174011+0000", 2024-09-20T22:54:46.567 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.9342e-05 2024-09-20T22:54:46.567 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.567 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.567 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:54:46.567 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.174047+0000", 2024-09-20T22:54:46.567 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.5237999999999997e-05 2024-09-20T22:54:46.567 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.568 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.568 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:54:46.568 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.188782+0000", 2024-09-20T22:54:46.568 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014735599 2024-09-20T22:54:46.568 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.568 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.568 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:54:46.568 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.188830+0000", 2024-09-20T22:54:46.568 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.7206000000000002e-05 2024-09-20T22:54:46.568 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:46.568 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:46.568 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:46.568 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.568 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.568 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(osd.6.0:767 7.12 e269/263 7:4fa4b2b3:::smithi08477203-21:53 v 269'262, mlcod=265'259)", 2024-09-20T22:54:46.569 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:45:48.175130+0000", 2024-09-20T22:54:46.569 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 538.37132939599996, 2024-09-20T22:54:46.569 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013842727000000001, 2024-09-20T22:54:46.569 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:54:46.569 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:54:46.569 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:54:46.569 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:54:46.569 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.569 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:54:46.569 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.175130+0000", 2024-09-20T22:54:46.569 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:54:46.569 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.569 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.569 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:54:46.570 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.175130+0000", 2024-09-20T22:54:46.570 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:54:46.570 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.570 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.570 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:54:46.570 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.175130+0000", 2024-09-20T22:54:46.570 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.98e-07 2024-09-20T22:54:46.570 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.570 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.570 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:54:46.570 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.175139+0000", 2024-09-20T22:54:46.570 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.5909999999999996e-06 2024-09-20T22:54:46.570 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.570 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.571 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:54:46.571 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.175144+0000", 2024-09-20T22:54:46.571 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.361e-06 2024-09-20T22:54:46.571 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.571 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.571 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:54:46.571 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.175156+0000", 2024-09-20T22:54:46.571 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1236999999999999e-05 2024-09-20T22:54:46.571 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.571 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.571 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:54:46.571 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.176359+0000", 2024-09-20T22:54:46.571 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0012036060000000001 2024-09-20T22:54:46.571 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.572 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.572 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:54:46.572 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.176380+0000", 2024-09-20T22:54:46.572 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.0497999999999999e-05 2024-09-20T22:54:46.572 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.572 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.572 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:54:46.572 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.188936+0000", 2024-09-20T22:54:46.572 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.012556286999999999 2024-09-20T22:54:46.572 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.572 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.572 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:54:46.572 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.188973+0000", 2024-09-20T22:54:46.572 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.6949000000000001e-05 2024-09-20T22:54:46.573 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:46.573 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:46.573 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:46.573 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.573 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.573 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:3030 7.e 7:71b5c915:::smithi08477203-6:head [copy-from ver 216] snapc 72=[72,71,70,6e,6c,6b,6a,68] ondisk+write+known_if_redirected+supports_pool_eio e317)", 2024-09-20T22:54:46.573 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:46:37.293029+0000", 2024-09-20T22:54:46.573 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 489.25343032400002, 2024-09-20T22:54:46.573 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.019929916999999998, 2024-09-20T22:54:46.573 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:54:46.573 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:54:46.573 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:54:46.573 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:54:46.573 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:54:46.574 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:54:46.574 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 3030 2024-09-20T22:54:46.574 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.574 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:54:46.574 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.574 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:54:46.574 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.293029+0000", 2024-09-20T22:54:46.574 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:54:46.574 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.574 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.574 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:54:46.574 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.293029+0000", 2024-09-20T22:54:46.574 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:54:46.574 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.575 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.575 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:54:46.575 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.293031+0000", 2024-09-20T22:54:46.575 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.4190000000000001e-06 2024-09-20T22:54:46.575 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.575 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.575 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:54:46.575 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.293040+0000", 2024-09-20T22:54:46.575 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.6039999999999999e-06 2024-09-20T22:54:46.575 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.575 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.575 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:54:46.575 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.293054+0000", 2024-09-20T22:54:46.576 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.4014e-05 2024-09-20T22:54:46.576 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.576 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.576 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:54:46.576 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.293062+0000", 2024-09-20T22:54:46.576 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.8820000000000001e-06 2024-09-20T22:54:46.576 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.576 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.576 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:54:46.576 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.293121+0000", 2024-09-20T22:54:46.576 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.8977999999999997e-05 2024-09-20T22:54:46.576 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.576 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.576 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op must wait for map", 2024-09-20T22:54:46.576 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.293125+0000", 2024-09-20T22:54:46.577 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.2089999999999999e-06 2024-09-20T22:54:46.577 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.577 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.577 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:54:46.577 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.298805+0000", 2024-09-20T22:54:46.577 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0056790870000000002 2024-09-20T22:54:46.577 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.577 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.577 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:54:46.577 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.298886+0000", 2024-09-20T22:54:46.577 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.1363e-05 2024-09-20T22:54:46.577 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.577 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.577 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 3,5", 2024-09-20T22:54:46.578 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.305883+0000", 2024-09-20T22:54:46.578 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0069966159999999998 2024-09-20T22:54:46.578 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.578 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.578 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:54:46.578 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.312556+0000", 2024-09-20T22:54:46.578 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0066737009999999998 2024-09-20T22:54:46.578 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.578 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.578 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:54:46.578 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.312587+0000", 2024-09-20T22:54:46.578 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.0286000000000001e-05 2024-09-20T22:54:46.578 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.578 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.579 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:54:46.579 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.312890+0000", 2024-09-20T22:54:46.579 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00030374800000000001 2024-09-20T22:54:46.579 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.579 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.579 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:54:46.579 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.312932+0000", 2024-09-20T22:54:46.579 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.1621e-05 2024-09-20T22:54:46.579 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.579 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.579 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:54:46.579 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.312959+0000", 2024-09-20T22:54:46.579 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.7389000000000002e-05 2024-09-20T22:54:46.579 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:46.580 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:46.580 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:46.580 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.580 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.580 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:3031 7.e 7:71b5c915:::smithi08477203-6:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e317)", 2024-09-20T22:54:46.580 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:46:37.293082+0000", 2024-09-20T22:54:46.580 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 489.253377946, 2024-09-20T22:54:46.580 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.016134757, 2024-09-20T22:54:46.580 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:54:46.580 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:54:46.580 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-20T22:54:46.580 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:54:46.580 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:54:46.580 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:54:46.581 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 3031 2024-09-20T22:54:46.581 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.581 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:54:46.581 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.581 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:54:46.581 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.293082+0000", 2024-09-20T22:54:46.581 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:54:46.581 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.581 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.581 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:54:46.581 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.293082+0000", 2024-09-20T22:54:46.581 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:54:46.581 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.581 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.581 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:54:46.582 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.293082+0000", 2024-09-20T22:54:46.582 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.1799999999999995e-07 2024-09-20T22:54:46.582 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.582 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.582 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:54:46.582 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.293085+0000", 2024-09-20T22:54:46.582 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.937e-06 2024-09-20T22:54:46.582 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.582 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.582 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:54:46.582 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.293093+0000", 2024-09-20T22:54:46.582 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.554e-06 2024-09-20T22:54:46.582 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.583 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.583 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:54:46.583 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.293098+0000", 2024-09-20T22:54:46.583 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.643e-06 2024-09-20T22:54:46.583 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.583 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.583 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:54:46.583 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.293164+0000", 2024-09-20T22:54:46.583 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.5806e-05 2024-09-20T22:54:46.583 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.583 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.583 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting_for_map not empty", 2024-09-20T22:54:46.583 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.293168+0000", 2024-09-20T22:54:46.583 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.2459999999999999e-06 2024-09-20T22:54:46.583 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.584 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.584 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:54:46.584 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.299033+0000", 2024-09-20T22:54:46.584 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0058656699999999999 2024-09-20T22:54:46.584 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.584 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.584 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for blocked object", 2024-09-20T22:54:46.584 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.299069+0000", 2024-09-20T22:54:46.584 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.5392999999999998e-05 2024-09-20T22:54:46.584 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.584 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.584 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:54:46.584 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.309088+0000", 2024-09-20T22:54:46.584 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010019287999999999 2024-09-20T22:54:46.584 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.585 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.585 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:54:46.585 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.309150+0000", 2024-09-20T22:54:46.585 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.2008000000000007e-05 2024-09-20T22:54:46.585 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.585 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.585 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:54:46.585 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.309216+0000", 2024-09-20T22:54:46.585 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.6594000000000006e-05 2024-09-20T22:54:46.585 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:46.585 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:46.585 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:46.585 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.585 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.586 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:3290 7.8 7:12e9c8db:::smithi08477203-11:head [copy-from ver 675] snapc 7f=[7f,7e,7d,7c,7b,7a,79,78,74,73,6b] ondisk+write+known_if_redirected+supports_pool_eio e340)", 2024-09-20T22:54:46.586 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:47:00.706846+0000", 2024-09-20T22:54:46.586 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 465.83961408699997, 2024-09-20T22:54:46.586 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.018408746, 2024-09-20T22:54:46.586 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:54:46.586 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:54:46.586 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:54:46.586 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:54:46.586 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:54:46.586 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:54:46.586 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 3290 2024-09-20T22:54:46.586 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.586 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:54:46.586 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.587 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:54:46.587 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.706846+0000", 2024-09-20T22:54:46.587 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:54:46.587 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.587 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.587 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:54:46.587 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.706846+0000", 2024-09-20T22:54:46.587 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:54:46.587 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.587 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.587 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:54:46.587 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.706848+0000", 2024-09-20T22:54:46.587 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.2340000000000001e-06 2024-09-20T22:54:46.587 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.587 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.588 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:54:46.588 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.706859+0000", 2024-09-20T22:54:46.588 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1001e-05 2024-09-20T22:54:46.588 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.588 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.588 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:54:46.588 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.706879+0000", 2024-09-20T22:54:46.588 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.0556000000000001e-05 2024-09-20T22:54:46.588 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.588 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.588 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:54:46.588 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.706890+0000", 2024-09-20T22:54:46.588 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0328e-05 2024-09-20T22:54:46.588 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.589 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.589 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:54:46.589 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.706973+0000", 2024-09-20T22:54:46.589 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.3574999999999999e-05 2024-09-20T22:54:46.589 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.589 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.589 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:54:46.589 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.707042+0000", 2024-09-20T22:54:46.589 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.8987999999999997e-05 2024-09-20T22:54:46.589 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.589 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.589 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 3,7", 2024-09-20T22:54:46.589 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.717491+0000", 2024-09-20T22:54:46.589 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010448618 2024-09-20T22:54:46.589 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.590 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.590 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:54:46.590 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.722875+0000", 2024-09-20T22:54:46.590 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0053846190000000002 2024-09-20T22:54:46.590 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.590 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.590 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:54:46.590 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.724841+0000", 2024-09-20T22:54:46.590 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0019651579999999998 2024-09-20T22:54:46.590 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.590 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.590 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:54:46.590 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.725168+0000", 2024-09-20T22:54:46.590 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00032734499999999999 2024-09-20T22:54:46.591 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.591 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.591 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:54:46.591 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.725218+0000", 2024-09-20T22:54:46.591 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.0123000000000001e-05 2024-09-20T22:54:46.591 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.591 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.591 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:54:46.591 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.725254+0000", 2024-09-20T22:54:46.591 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.6201000000000001e-05 2024-09-20T22:54:46.591 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:46.591 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:46.591 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:46.591 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.592 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.592 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:3291 7.8 7:12e9c8db:::smithi08477203-11:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e340)", 2024-09-20T22:54:46.592 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:47:00.706927+0000", 2024-09-20T22:54:46.592 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 465.839532448, 2024-09-20T22:54:46.592 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014159255000000001, 2024-09-20T22:54:46.592 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:54:46.592 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:54:46.592 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-20T22:54:46.592 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:54:46.592 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:54:46.592 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:54:46.592 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 3291 2024-09-20T22:54:46.592 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.593 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:54:46.593 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.593 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:54:46.593 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.706927+0000", 2024-09-20T22:54:46.593 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:54:46.593 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.593 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.593 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:54:46.593 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.706927+0000", 2024-09-20T22:54:46.593 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:54:46.593 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.593 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.593 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:54:46.593 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.706928+0000", 2024-09-20T22:54:46.594 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.4099999999999999e-07 2024-09-20T22:54:46.594 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.594 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.594 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:54:46.594 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.706941+0000", 2024-09-20T22:54:46.594 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3335e-05 2024-09-20T22:54:46.594 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.594 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.594 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:54:46.594 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.706948+0000", 2024-09-20T22:54:46.594 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.3350000000000001e-06 2024-09-20T22:54:46.594 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.594 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.594 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:54:46.595 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.706954+0000", 2024-09-20T22:54:46.595 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.9190000000000004e-06 2024-09-20T22:54:46.595 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.595 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.595 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:54:46.595 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.707166+0000", 2024-09-20T22:54:46.595 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00021159 2024-09-20T22:54:46.595 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.595 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.595 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for blocked object", 2024-09-20T22:54:46.595 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.707217+0000", 2024-09-20T22:54:46.595 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.1167e-05 2024-09-20T22:54:46.595 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.595 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.596 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:54:46.596 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.720939+0000", 2024-09-20T22:54:46.596 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013721458000000001 2024-09-20T22:54:46.596 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.596 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.596 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:54:46.596 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.721010+0000", 2024-09-20T22:54:46.596 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.1570999999999996e-05 2024-09-20T22:54:46.596 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.596 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.596 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:54:46.596 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.721086+0000", 2024-09-20T22:54:46.596 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.6339000000000002e-05 2024-09-20T22:54:46.596 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:46.597 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:46.597 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:46.597 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.597 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.597 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24652.0:6378 7.2 e639/600 7:4d20232a:::smithi08477203-34:head v 639'665, mlcod=629'662)", 2024-09-20T22:54:46.597 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:51:59.278714+0000", 2024-09-20T22:54:46.597 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 167.267745792, 2024-09-20T22:54:46.597 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.015218334, 2024-09-20T22:54:46.597 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:54:46.597 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:54:46.597 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:54:46.597 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:54:46.597 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.597 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:54:46.598 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:59.278714+0000", 2024-09-20T22:54:46.598 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:54:46.598 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.598 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.598 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:54:46.598 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:59.278714+0000", 2024-09-20T22:54:46.598 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:54:46.598 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.598 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.598 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:54:46.598 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:59.278714+0000", 2024-09-20T22:54:46.598 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.6e-07 2024-09-20T22:54:46.598 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.598 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.599 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:54:46.599 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:59.281091+0000", 2024-09-20T22:54:46.599 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0023769239999999999 2024-09-20T22:54:46.599 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.599 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.599 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:54:46.599 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:59.281110+0000", 2024-09-20T22:54:46.599 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.9230000000000001e-05 2024-09-20T22:54:46.599 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.599 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.599 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:54:46.599 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:59.281122+0000", 2024-09-20T22:54:46.599 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1997999999999999e-05 2024-09-20T22:54:46.599 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.599 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.600 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:54:46.600 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:59.281186+0000", 2024-09-20T22:54:46.600 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.3998999999999995e-05 2024-09-20T22:54:46.600 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.600 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.600 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:54:46.600 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:59.281209+0000", 2024-09-20T22:54:46.600 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.2654999999999999e-05 2024-09-20T22:54:46.600 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.600 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.600 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:54:46.600 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:59.293866+0000", 2024-09-20T22:54:46.600 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.012656745 2024-09-20T22:54:46.600 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.601 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.601 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:54:46.601 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:59.293932+0000", 2024-09-20T22:54:46.601 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.6523e-05 2024-09-20T22:54:46.601 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:46.601 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:46.601 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:46.601 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.601 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.601 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(osd.5.0:520 7.b e643/602 7:dc95255e:::smithi08477203-1:107 v 643'959, mlcod=637'955)", 2024-09-20T22:54:46.601 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:52:03.289043+0000", 2024-09-20T22:54:46.601 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 163.257416683, 2024-09-20T22:54:46.601 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.015367396, 2024-09-20T22:54:46.601 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:54:46.602 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:54:46.602 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:54:46.602 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:54:46.602 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.602 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:54:46.602 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:52:03.289043+0000", 2024-09-20T22:54:46.602 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:54:46.602 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.602 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.602 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:54:46.602 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:52:03.289043+0000", 2024-09-20T22:54:46.602 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:54:46.602 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.603 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.603 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:54:46.603 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:52:03.289044+0000", 2024-09-20T22:54:46.603 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.8100000000000002e-07 2024-09-20T22:54:46.603 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.603 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.603 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:54:46.603 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:52:03.289056+0000", 2024-09-20T22:54:46.603 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2408999999999999e-05 2024-09-20T22:54:46.603 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.603 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.603 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:54:46.603 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:52:03.289071+0000", 2024-09-20T22:54:46.603 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.4331000000000001e-05 2024-09-20T22:54:46.603 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.604 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.604 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:54:46.604 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:52:03.289079+0000", 2024-09-20T22:54:46.604 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.1009999999999992e-06 2024-09-20T22:54:46.604 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.604 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.604 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:54:46.604 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:52:03.289168+0000", 2024-09-20T22:54:46.604 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.9614e-05 2024-09-20T22:54:46.604 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.604 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.604 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:54:46.604 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:52:03.289189+0000", 2024-09-20T22:54:46.604 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.0701999999999999e-05 2024-09-20T22:54:46.605 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.605 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.605 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:54:46.605 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:52:03.304364+0000", 2024-09-20T22:54:46.605 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.015175414 2024-09-20T22:54:46.605 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.605 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.605 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:54:46.605 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:52:03.304410+0000", 2024-09-20T22:54:46.605 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.6044000000000002e-05 2024-09-20T22:54:46.605 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:46.605 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:46.605 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:46.605 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.606 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.606 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(osd.5.0:521 7.b e643/602 7:d960cad4:::smithi08477203-18:107 v 643'961, mlcod=637'955)", 2024-09-20T22:54:46.606 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:52:03.289097+0000", 2024-09-20T22:54:46.606 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 163.25736303599999, 2024-09-20T22:54:46.606 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.016179973, 2024-09-20T22:54:46.606 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:54:46.606 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:54:46.606 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:54:46.606 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:54:46.606 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.606 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:54:46.606 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:52:03.289097+0000", 2024-09-20T22:54:46.606 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:54:46.606 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.607 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.607 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:54:46.607 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:52:03.289097+0000", 2024-09-20T22:54:46.607 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:54:46.607 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.607 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.607 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:54:46.607 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:52:03.289097+0000", 2024-09-20T22:54:46.607 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.3599999999999999e-07 2024-09-20T22:54:46.607 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.607 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.607 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:54:46.607 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:52:03.289105+0000", 2024-09-20T22:54:46.607 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.8869999999999992e-06 2024-09-20T22:54:46.608 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.608 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.608 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:54:46.608 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:52:03.289111+0000", 2024-09-20T22:54:46.608 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.2419999999999997e-06 2024-09-20T22:54:46.608 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.608 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.608 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:54:46.608 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:52:03.289116+0000", 2024-09-20T22:54:46.608 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.6569999999999996e-06 2024-09-20T22:54:46.608 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.608 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.608 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:54:46.608 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:52:03.291170+0000", 2024-09-20T22:54:46.608 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0020537989999999998 2024-09-20T22:54:46.609 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.609 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.609 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:54:46.609 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:52:03.291192+0000", 2024-09-20T22:54:46.609 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.2548000000000002e-05 2024-09-20T22:54:46.609 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.609 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.609 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:54:46.609 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:52:03.305241+0000", 2024-09-20T22:54:46.609 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.01404832 2024-09-20T22:54:46.609 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.609 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.609 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:54:46.609 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:52:03.305277+0000", 2024-09-20T22:54:46.609 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.6084000000000002e-05 2024-09-20T22:54:46.610 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:46.610 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:46.610 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:46.610 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.610 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.610 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24652.0:6790 7.0 e687/632 7:064f66ab:::smithi08477203-45:head v 687'535, mlcod=687'532)", 2024-09-20T22:54:46.610 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:52:47.711923+0000", 2024-09-20T22:54:46.610 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 118.834536222, 2024-09-20T22:54:46.610 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014321054999999999, 2024-09-20T22:54:46.610 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:54:46.610 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:54:46.610 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:54:46.610 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:54:46.610 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.611 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:54:46.611 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:52:47.711923+0000", 2024-09-20T22:54:46.611 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:54:46.611 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.611 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.611 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:54:46.611 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:52:47.711923+0000", 2024-09-20T22:54:46.611 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:54:46.611 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.611 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.611 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:54:46.611 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:52:47.711924+0000", 2024-09-20T22:54:46.611 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.33e-07 2024-09-20T22:54:46.611 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.611 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.612 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:54:46.612 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:52:47.720461+0000", 2024-09-20T22:54:46.612 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0085366850000000005 2024-09-20T22:54:46.612 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.612 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.612 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:54:46.612 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:52:47.720487+0000", 2024-09-20T22:54:46.612 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.5463e-05 2024-09-20T22:54:46.612 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.612 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.612 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:54:46.612 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:52:47.720540+0000", 2024-09-20T22:54:46.612 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.3356999999999998e-05 2024-09-20T22:54:46.612 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.613 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.613 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:54:46.613 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:52:47.720631+0000", 2024-09-20T22:54:46.613 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.1305999999999996e-05 2024-09-20T22:54:46.613 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.613 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.613 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:54:46.613 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:52:47.720662+0000", 2024-09-20T22:54:46.613 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.1149000000000003e-05 2024-09-20T22:54:46.613 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.613 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.613 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:54:46.613 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:52:47.726196+0000", 2024-09-20T22:54:46.613 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0055332430000000002 2024-09-20T22:54:46.614 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.614 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.614 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:54:46.614 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:52:47.726244+0000", 2024-09-20T22:54:46.614 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.8918999999999997e-05 2024-09-20T22:54:46.614 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:46.614 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:46.614 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:46.614 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.614 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.614 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24652.0:6829 7.5 e688/456 7:a160b5d4:::smithi08477203-16:head v 688'328, mlcod=684'324)", 2024-09-20T22:54:46.614 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:52:48.309561+0000", 2024-09-20T22:54:46.614 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 118.23689843699999, 2024-09-20T22:54:46.614 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.017015579999999999, 2024-09-20T22:54:46.615 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:54:46.615 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:54:46.615 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:54:46.615 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:54:46.615 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.615 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:54:46.615 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:52:48.309561+0000", 2024-09-20T22:54:46.615 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:54:46.615 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.615 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.615 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:54:46.615 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:52:48.309561+0000", 2024-09-20T22:54:46.615 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:54:46.615 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.615 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.616 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:54:46.616 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:52:48.309562+0000", 2024-09-20T22:54:46.616 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.7599999999999996e-07 2024-09-20T22:54:46.616 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.616 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.616 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:54:46.616 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:52:48.310336+0000", 2024-09-20T22:54:46.616 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00077435700000000004 2024-09-20T22:54:46.616 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.616 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.616 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:54:46.616 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:52:48.310355+0000", 2024-09-20T22:54:46.616 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.8763e-05 2024-09-20T22:54:46.616 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.616 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.617 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:54:46.617 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:52:48.310365+0000", 2024-09-20T22:54:46.617 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.4420000000000004e-06 2024-09-20T22:54:46.617 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.617 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.617 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:54:46.617 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:52:48.310463+0000", 2024-09-20T22:54:46.617 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.8177999999999994e-05 2024-09-20T22:54:46.617 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.617 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.617 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:54:46.617 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:52:48.310494+0000", 2024-09-20T22:54:46.617 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.1284000000000001e-05 2024-09-20T22:54:46.617 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.617 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.618 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:54:46.618 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:52:48.326521+0000", 2024-09-20T22:54:46.618 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.016026757999999999 2024-09-20T22:54:46.618 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.618 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.618 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:54:46.618 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:52:48.326577+0000", 2024-09-20T22:54:46.618 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.5921999999999998e-05 2024-09-20T22:54:46.618 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:46.618 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:46.618 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:46.618 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.618 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.618 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24652.0:7143 7.0 e719/632 7:0f524219:::smithi08477203-50:head v 719'618, mlcod=714'614)", 2024-09-20T22:54:46.619 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:53:19.300601+0000", 2024-09-20T22:54:46.619 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 87.245858557999995, 2024-09-20T22:54:46.619 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.016058294000000001, 2024-09-20T22:54:46.619 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:54:46.619 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:54:46.619 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:54:46.619 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:54:46.619 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.619 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:54:46.619 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:19.300601+0000", 2024-09-20T22:54:46.619 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:54:46.619 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.620 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.620 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:54:46.620 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:19.300601+0000", 2024-09-20T22:54:46.620 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:54:46.621 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.621 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.621 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:54:46.621 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:19.300601+0000", 2024-09-20T22:54:46.621 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.6400000000000001e-07 2024-09-20T22:54:46.621 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.621 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.621 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:54:46.621 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:19.307294+0000", 2024-09-20T22:54:46.621 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0066925200000000004 2024-09-20T22:54:46.621 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.621 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.621 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:54:46.621 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:19.307307+0000", 2024-09-20T22:54:46.622 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3431999999999999e-05 2024-09-20T22:54:46.622 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.622 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.622 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:54:46.622 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:19.307316+0000", 2024-09-20T22:54:46.622 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.5099999999999998e-06 2024-09-20T22:54:46.622 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.622 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.622 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:54:46.622 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:19.307385+0000", 2024-09-20T22:54:46.622 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.8771999999999998e-05 2024-09-20T22:54:46.622 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.622 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.623 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:54:46.623 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:19.307413+0000", 2024-09-20T22:54:46.623 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.8373000000000001e-05 2024-09-20T22:54:46.623 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.623 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.623 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:54:46.623 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:19.316602+0000", 2024-09-20T22:54:46.623 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0091889950000000001 2024-09-20T22:54:46.623 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.623 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.623 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:54:46.623 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:19.316659+0000", 2024-09-20T22:54:46.623 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.7528000000000002e-05 2024-09-20T22:54:46.623 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:46.624 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:46.624 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:46.624 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.624 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.624 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24652.0:7356 7.a e738/94 7:5aaef466:::smithi08477203-40:head v 738'878, mlcod=736'875)", 2024-09-20T22:54:46.624 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:53:38.313061+0000", 2024-09-20T22:54:46.624 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 68.233398953999995, 2024-09-20T22:54:46.624 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.027472555999999999, 2024-09-20T22:54:46.624 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:54:46.624 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:54:46.624 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:54:46.624 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:54:46.624 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.624 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:54:46.625 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:38.313061+0000", 2024-09-20T22:54:46.625 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:54:46.625 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.625 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.625 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:54:46.625 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:38.313061+0000", 2024-09-20T22:54:46.625 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:54:46.625 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.625 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.625 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:54:46.625 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:38.313061+0000", 2024-09-20T22:54:46.625 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.0399999999999996e-07 2024-09-20T22:54:46.625 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.625 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.626 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:54:46.626 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:38.316261+0000", 2024-09-20T22:54:46.626 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0031994129999999999 2024-09-20T22:54:46.626 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.626 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.626 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:54:46.626 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:38.316282+0000", 2024-09-20T22:54:46.626 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.1090999999999998e-05 2024-09-20T22:54:46.626 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.626 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.626 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:54:46.626 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:38.316295+0000", 2024-09-20T22:54:46.626 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3226e-05 2024-09-20T22:54:46.626 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.627 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.627 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:54:46.627 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:38.316385+0000", 2024-09-20T22:54:46.627 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.0048000000000002e-05 2024-09-20T22:54:46.627 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.627 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.627 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:54:46.627 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:38.316411+0000", 2024-09-20T22:54:46.627 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.6401e-05 2024-09-20T22:54:46.627 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.627 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.627 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:54:46.627 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:38.340477+0000", 2024-09-20T22:54:46.627 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.024065468999999999 2024-09-20T22:54:46.628 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.628 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.628 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:54:46.628 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:38.340533+0000", 2024-09-20T22:54:46.628 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.6403999999999997e-05 2024-09-20T22:54:46.628 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:46.628 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:46.628 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:46.628 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.628 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.628 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24652.0:7432 7.0 e744/632 7:064f66ab:::smithi08477203-45:head v 744'684, mlcod=743'681)", 2024-09-20T22:54:46.628 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:53:44.436593+0000", 2024-09-20T22:54:46.628 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 62.109867051999998, 2024-09-20T22:54:46.629 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.040892389000000001, 2024-09-20T22:54:46.629 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:54:46.629 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:54:46.629 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:54:46.629 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:54:46.629 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.629 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:54:46.629 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:44.436593+0000", 2024-09-20T22:54:46.629 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:54:46.629 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.629 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.629 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:54:46.629 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:44.436593+0000", 2024-09-20T22:54:46.629 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:54:46.630 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.630 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.630 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:54:46.630 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:44.436593+0000", 2024-09-20T22:54:46.630 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.6699999999999999e-07 2024-09-20T22:54:46.630 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.630 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.630 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:54:46.630 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:44.475632+0000", 2024-09-20T22:54:46.630 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.039039010999999998 2024-09-20T22:54:46.630 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.630 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.630 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:54:46.630 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:44.475645+0000", 2024-09-20T22:54:46.631 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3084999999999999e-05 2024-09-20T22:54:46.631 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.631 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.631 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:54:46.631 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:44.475651+0000", 2024-09-20T22:54:46.631 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.1190000000000002e-06 2024-09-20T22:54:46.631 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.631 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.631 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:54:46.631 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:44.475714+0000", 2024-09-20T22:54:46.631 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.3239999999999998e-05 2024-09-20T22:54:46.631 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.631 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.631 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:54:46.631 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:44.475741+0000", 2024-09-20T22:54:46.632 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.6911e-05 2024-09-20T22:54:46.632 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.632 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.632 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:54:46.632 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:44.477457+0000", 2024-09-20T22:54:46.632 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0017154959999999999 2024-09-20T22:54:46.632 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.632 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.632 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:54:46.632 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:44.477485+0000", 2024-09-20T22:54:46.632 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.8160000000000001e-05 2024-09-20T22:54:46.632 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:46.632 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:46.632 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:46.632 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.633 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.633 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:7625 7.e 7:7c62ced4:::smithi08477203-17:head [copy-from ver 1253] snapc 159=[159,158,157,156,155,153,151,14f,14e,14c,14a,149,148,145,142,13e,13a,139,132,130,12f,12e,12a,124,121,112,103] ondisk+write+known_if_redirected+supports_pool_eio e766)", 2024-09-20T22:54:46.633 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:54:06.336464+0000", 2024-09-20T22:54:46.633 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 40.209995313999997, 2024-09-20T22:54:46.633 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014856506, 2024-09-20T22:54:46.633 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:54:46.633 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:54:46.633 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:54:46.633 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:54:46.633 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:54:46.633 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:54:46.633 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 7625 2024-09-20T22:54:46.633 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.633 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:54:46.634 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.634 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:54:46.634 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:54:06.336464+0000", 2024-09-20T22:54:46.634 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:54:46.634 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.634 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.634 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:54:46.634 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:54:06.336464+0000", 2024-09-20T22:54:46.634 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:54:46.634 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.634 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.634 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:54:46.634 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:54:06.336466+0000", 2024-09-20T22:54:46.634 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.7770000000000001e-06 2024-09-20T22:54:46.635 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.635 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.635 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:54:46.635 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:54:06.336475+0000", 2024-09-20T22:54:46.635 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.9579999999999996e-06 2024-09-20T22:54:46.635 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.635 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.635 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:54:46.635 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:54:06.336491+0000", 2024-09-20T22:54:46.635 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.6079999999999999e-05 2024-09-20T22:54:46.635 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.635 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.635 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:54:46.635 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:54:06.336501+0000", 2024-09-20T22:54:46.635 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.8239999999999995e-06 2024-09-20T22:54:46.636 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.636 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.636 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:54:46.636 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:54:06.336571+0000", 2024-09-20T22:54:46.636 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.9746999999999995e-05 2024-09-20T22:54:46.636 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.636 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.636 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op must wait for map", 2024-09-20T22:54:46.636 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:54:06.336580+0000", 2024-09-20T22:54:46.636 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.2900000000000008e-06 2024-09-20T22:54:46.636 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.636 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.636 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:54:46.636 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:54:06.338593+0000", 2024-09-20T22:54:46.636 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0020125939999999999 2024-09-20T22:54:46.637 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.637 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.637 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:54:46.637 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:54:06.338748+0000", 2024-09-20T22:54:46.637 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00015523199999999999 2024-09-20T22:54:46.637 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.637 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.637 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 0,5", 2024-09-20T22:54:46.637 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:54:06.344247+0000", 2024-09-20T22:54:46.637 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0054991079999999999 2024-09-20T22:54:46.637 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.637 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.637 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:54:46.637 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:54:06.349881+0000", 2024-09-20T22:54:46.638 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0056344689999999996 2024-09-20T22:54:46.638 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.638 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.638 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:54:46.638 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:54:06.351077+0000", 2024-09-20T22:54:46.638 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001196016 2024-09-20T22:54:46.638 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.638 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.638 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:54:46.638 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:54:06.351225+0000", 2024-09-20T22:54:46.638 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014721100000000001 2024-09-20T22:54:46.638 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.638 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.638 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:54:46.639 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:54:06.351265+0000", 2024-09-20T22:54:46.639 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.985e-05 2024-09-20T22:54:46.639 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.639 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.639 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:54:46.639 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:54:06.351321+0000", 2024-09-20T22:54:46.639 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.6350000000000001e-05 2024-09-20T22:54:46.639 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:46.639 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:46.639 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:46.639 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.639 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.639 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:7626 7.e 7:7c62ced4:::smithi08477203-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e766)", 2024-09-20T22:54:46.639 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:54:06.336525+0000", 2024-09-20T22:54:46.639 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 40.209934820000001, 2024-09-20T22:54:46.640 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011108501999999999, 2024-09-20T22:54:46.640 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:54:46.640 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:54:46.640 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-20T22:54:46.640 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:54:46.640 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:54:46.640 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:54:46.640 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 7626 2024-09-20T22:54:46.640 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.640 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:54:46.640 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.640 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:54:46.640 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:54:06.336525+0000", 2024-09-20T22:54:46.640 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:54:46.641 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.641 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.641 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:54:46.641 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:54:06.336525+0000", 2024-09-20T22:54:46.641 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:54:46.641 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.641 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.641 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:54:46.641 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:54:06.336525+0000", 2024-09-20T22:54:46.641 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.7000000000000005e-07 2024-09-20T22:54:46.641 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.641 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.641 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:54:46.641 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:54:06.336528+0000", 2024-09-20T22:54:46.641 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.8109999999999999e-06 2024-09-20T22:54:46.642 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.642 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.642 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:54:46.642 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:54:06.336542+0000", 2024-09-20T22:54:46.642 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.4263e-05 2024-09-20T22:54:46.642 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.642 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.642 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:54:46.642 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:54:06.336550+0000", 2024-09-20T22:54:46.642 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.3509999999999996e-06 2024-09-20T22:54:46.642 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.642 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.642 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:54:46.643 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:54:06.336636+0000", 2024-09-20T22:54:46.643 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.6652999999999999e-05 2024-09-20T22:54:46.643 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.643 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.643 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting_for_map not empty", 2024-09-20T22:54:46.643 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:54:06.336645+0000", 2024-09-20T22:54:46.643 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.6540000000000005e-06 2024-09-20T22:54:46.643 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.643 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.643 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:54:46.643 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:54:06.338929+0000", 2024-09-20T22:54:46.643 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0022836699999999998 2024-09-20T22:54:46.644 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.644 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.644 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for blocked object", 2024-09-20T22:54:46.644 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:54:06.338984+0000", 2024-09-20T22:54:46.644 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.4892000000000002e-05 2024-09-20T22:54:46.644 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.644 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.644 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:54:46.644 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:54:06.347468+0000", 2024-09-20T22:54:46.644 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0084845530000000006 2024-09-20T22:54:46.644 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.644 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.644 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:54:46.644 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:54:06.347547+0000", 2024-09-20T22:54:46.645 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.9144e-05 2024-09-20T22:54:46.645 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.645 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.645 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:54:46.645 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:54:06.347633+0000", 2024-09-20T22:54:46.645 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.5940999999999996e-05 2024-09-20T22:54:46.645 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:46.645 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:46.645 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:46.645 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.645 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.645 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:7926 7.8 7:12e9c8db:::smithi08477203-11:head [append 0~0] snapc 16a=[169,167,166,165,164,162,161,160,15f,15e,15d,15c,15b,15a,158,157,156,155,153,14f,14e,14c,14a,149,148,145,142,13e,13a,139,12f,12e,124,112] ondisk+write+known_if_redirected+supports_pool_eio e793)", 2024-09-20T22:54:46.645 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:54:33.364268+0000", 2024-09-20T22:54:46.645 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 13.182192073, 2024-09-20T22:54:46.646 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0061548269999999999, 2024-09-20T22:54:46.646 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:54:46.646 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:54:46.646 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:54:46.646 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:54:46.646 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:54:46.646 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:54:46.646 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 7926 2024-09-20T22:54:46.646 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.646 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:54:46.646 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.646 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:54:46.646 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:54:33.364268+0000", 2024-09-20T22:54:46.646 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:54:46.647 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.647 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.647 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:54:46.647 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:54:33.364268+0000", 2024-09-20T22:54:46.647 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:54:46.647 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.647 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.647 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:54:46.647 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:54:33.364268+0000", 2024-09-20T22:54:46.647 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.7099999999999998e-07 2024-09-20T22:54:46.647 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.647 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.647 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:54:46.647 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:54:33.364270+0000", 2024-09-20T22:54:46.647 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.2299999999999998e-06 2024-09-20T22:54:46.648 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.648 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.648 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:54:46.648 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:54:33.364274+0000", 2024-09-20T22:54:46.648 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.1629999999999998e-06 2024-09-20T22:54:46.648 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.648 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.648 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:54:46.648 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:54:33.364277+0000", 2024-09-20T22:54:46.648 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.9569999999999998e-06 2024-09-20T22:54:46.648 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.648 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.648 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:54:46.648 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:54:33.364417+0000", 2024-09-20T22:54:46.648 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000139297 2024-09-20T22:54:46.649 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.649 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.649 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting_for_map not empty", 2024-09-20T22:54:46.649 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:54:33.364425+0000", 2024-09-20T22:54:46.649 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.1009999999999992e-06 2024-09-20T22:54:46.649 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.649 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.649 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:54:46.649 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:54:33.368060+0000", 2024-09-20T22:54:46.649 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0036349339999999998 2024-09-20T22:54:46.649 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.649 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.649 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:54:46.649 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:54:33.368129+0000", 2024-09-20T22:54:46.649 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.9119e-05 2024-09-20T22:54:46.650 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.650 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.650 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 4,7", 2024-09-20T22:54:46.650 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:54:33.368266+0000", 2024-09-20T22:54:46.650 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000137293 2024-09-20T22:54:46.650 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.650 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.650 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:54:46.650 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:54:33.368763+0000", 2024-09-20T22:54:46.650 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00049738899999999999 2024-09-20T22:54:46.650 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.650 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.650 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:54:46.650 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:54:33.369863+0000", 2024-09-20T22:54:46.650 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010996420000000001 2024-09-20T22:54:46.651 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.651 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.651 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:54:46.651 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:54:33.370298+0000", 2024-09-20T22:54:46.651 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00043452399999999999 2024-09-20T22:54:46.651 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.651 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.651 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:54:46.651 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:54:33.370366+0000", 2024-09-20T22:54:46.651 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.8827999999999999e-05 2024-09-20T22:54:46.651 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:46.651 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:46.651 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:54:46.651 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:54:33.370422+0000", 2024-09-20T22:54:46.651 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.6079000000000003e-05 2024-09-20T22:54:46.652 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:46.652 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:46.652 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:46.652 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:46.652 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:46.652 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-20T22:54:46.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:46.653 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:46.653 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:54:46.685 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:46 smithi084 ceph-mon[40024]: 4.19 scrub starts 2024-09-20T22:54:46.685 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:46 smithi084 ceph-mon[40024]: 4.19 scrub ok 2024-09-20T22:54:46.685 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:46 smithi084 ceph-mon[40024]: osdmap e805: 8 total, 8 up, 8 in 2024-09-20T22:54:46.685 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:46 smithi084 ceph-mon[40024]: 4.1 scrub starts 2024-09-20T22:54:46.685 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:46 smithi084 ceph-mon[40024]: 4.1 scrub ok 2024-09-20T22:54:46.685 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:46 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/416556682' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:54:46.685 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:46 smithi084 ceph-mon[29774]: 4.19 scrub starts 2024-09-20T22:54:46.686 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:46 smithi084 ceph-mon[29774]: 4.19 scrub ok 2024-09-20T22:54:46.686 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:46 smithi084 ceph-mon[29774]: osdmap e805: 8 total, 8 up, 8 in 2024-09-20T22:54:46.686 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:46 smithi084 ceph-mon[29774]: 4.1 scrub starts 2024-09-20T22:54:46.686 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:46 smithi084 ceph-mon[29774]: 4.1 scrub ok 2024-09-20T22:54:46.686 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:46 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/416556682' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:54:46.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:46 smithi154 ceph-mon[33264]: 4.19 scrub starts 2024-09-20T22:54:46.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:46 smithi154 ceph-mon[33264]: 4.19 scrub ok 2024-09-20T22:54:46.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:46 smithi154 ceph-mon[33264]: osdmap e805: 8 total, 8 up, 8 in 2024-09-20T22:54:46.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:46 smithi154 ceph-mon[33264]: 4.1 scrub starts 2024-09-20T22:54:46.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:46 smithi154 ceph-mon[33264]: 4.1 scrub ok 2024-09-20T22:54:46.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:46 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/416556682' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:54:46.755 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:54:46.777 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:46.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:46.778 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:54:46.879 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:54:46.900 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:46.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:46.901 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:54:47.002 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:54:47.023 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:47.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:47.024 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:54:47.125 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:54:47.145 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:54:47.145 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:54:47.145 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:54:47.145 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-20T22:54:47.146 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:54:47.146 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:54:47.146 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:54:47.146 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T22:54:47.146 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:54:47.146 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:54:47.146 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:54:47.146 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:54:47.147 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:54:47.147 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:54:47.147 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:54:47.147 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:54:47.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:47.148 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:47.148 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:54:47.249 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:54:47.271 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:47.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:47.272 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:54:47.373 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:54:47.395 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:47.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:47.396 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:54:47.448 INFO:tasks.rados.rados.0.smithi084.stdout:3209: done (2 left) 2024-09-20T22:54:47.449 INFO:tasks.rados.rados.0.smithi084.stdout:3210: snap_remove snap 272 2024-09-20T22:54:47.496 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:54:47.518 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:47.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:47.519 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:54:47.552 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd set noscrub 2024-09-20T22:54:47.620 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:54:47.642 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:47.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:47.643 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:54:47.743 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:54:47.750 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:47 smithi084 ceph-mon[29774]: pgmap v1110: 158 pgs: 4 active+clean+snaptrim, 1 active+remapped+backfill_toofull, 153 active+clean; 582 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 19 MiB/s rd, 2.0 MiB/s wr, 23 op/s; 16/1407 objects misplaced (1.137%) 2024-09-20T22:54:47.750 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:47 smithi084 ceph-mon[29774]: 1.8 scrub starts 2024-09-20T22:54:47.750 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:47 smithi084 ceph-mon[29774]: 1.8 scrub ok 2024-09-20T22:54:47.750 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:47 smithi084 ceph-mon[29774]: osdmap e806: 8 total, 8 up, 8 in 2024-09-20T22:54:47.750 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:47 smithi084 ceph-mon[29774]: 4.3 scrub starts 2024-09-20T22:54:47.751 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:47 smithi084 ceph-mon[29774]: 4.3 scrub ok 2024-09-20T22:54:47.751 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:47 smithi084 ceph-mon[29774]: purged_snaps scrub starts 2024-09-20T22:54:47.751 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:47 smithi084 ceph-mon[29774]: purged_snaps scrub ok 2024-09-20T22:54:47.751 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:47 smithi084 ceph-mon[40024]: pgmap v1110: 158 pgs: 4 active+clean+snaptrim, 1 active+remapped+backfill_toofull, 153 active+clean; 582 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 19 MiB/s rd, 2.0 MiB/s wr, 23 op/s; 16/1407 objects misplaced (1.137%) 2024-09-20T22:54:47.751 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:47 smithi084 ceph-mon[40024]: 1.8 scrub starts 2024-09-20T22:54:47.751 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:47 smithi084 ceph-mon[40024]: 1.8 scrub ok 2024-09-20T22:54:47.751 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:47 smithi084 ceph-mon[40024]: osdmap e806: 8 total, 8 up, 8 in 2024-09-20T22:54:47.751 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:47 smithi084 ceph-mon[40024]: 4.3 scrub starts 2024-09-20T22:54:47.751 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:47 smithi084 ceph-mon[40024]: 4.3 scrub ok 2024-09-20T22:54:47.751 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:47 smithi084 ceph-mon[40024]: purged_snaps scrub starts 2024-09-20T22:54:47.751 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:47 smithi084 ceph-mon[40024]: purged_snaps scrub ok 2024-09-20T22:54:47.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:47 smithi154 ceph-mon[33264]: pgmap v1110: 158 pgs: 4 active+clean+snaptrim, 1 active+remapped+backfill_toofull, 153 active+clean; 582 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 19 MiB/s rd, 2.0 MiB/s wr, 23 op/s; 16/1407 objects misplaced (1.137%) 2024-09-20T22:54:47.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:47 smithi154 ceph-mon[33264]: 1.8 scrub starts 2024-09-20T22:54:47.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:47 smithi154 ceph-mon[33264]: 1.8 scrub ok 2024-09-20T22:54:47.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:47 smithi154 ceph-mon[33264]: osdmap e806: 8 total, 8 up, 8 in 2024-09-20T22:54:47.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:47 smithi154 ceph-mon[33264]: 4.3 scrub starts 2024-09-20T22:54:47.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:47 smithi154 ceph-mon[33264]: 4.3 scrub ok 2024-09-20T22:54:47.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:47 smithi154 ceph-mon[33264]: purged_snaps scrub starts 2024-09-20T22:54:47.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:47 smithi154 ceph-mon[33264]: purged_snaps scrub ok 2024-09-20T22:54:47.767 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:47.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:47.768 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:54:47.869 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:54:47.890 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:47.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:47.891 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:54:47.991 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:54:48.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:48.016 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:48.016 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:54:48.117 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:54:48.139 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:48.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:48.140 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:54:48.240 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:54:48.263 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:48.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:48.264 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:54:48.274 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-20T22:54:48.324 INFO:teuthology.orchestra.run.smithi084.stderr:clear 6.1e pg_upmap mapping 2024-09-20T22:54:48.326 INFO:tasks.rados.rados.0.smithi084.stdout:3210: done (2 left) 2024-09-20T22:54:48.326 INFO:tasks.rados.rados.0.smithi084.stdout:3211: snap_remove snap 366 2024-09-20T22:54:48.364 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:54:48.385 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:48.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:48.386 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:54:48.487 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:54:48.510 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:48.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:48.511 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:54:48.611 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:54:48.615 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:54:48 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: ::ffff:172.21.15.154 - - [20/Sep/2024:22:54:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-20T22:54:48.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:48 smithi084 ceph-mon[29774]: 3.15 scrub starts 2024-09-20T22:54:48.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:48 smithi084 ceph-mon[29774]: 3.15 scrub ok 2024-09-20T22:54:48.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:48 smithi084 ceph-mon[29774]: 5.f deep-scrub starts 2024-09-20T22:54:48.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:48 smithi084 ceph-mon[29774]: 5.f deep-scrub ok 2024-09-20T22:54:48.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:48 smithi084 ceph-mon[29774]: 5.a scrub starts 2024-09-20T22:54:48.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:48 smithi084 ceph-mon[29774]: 5.a scrub ok 2024-09-20T22:54:48.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:48 smithi084 ceph-mon[29774]: 3.1e scrub starts 2024-09-20T22:54:48.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:48 smithi084 ceph-mon[29774]: 3.1e scrub ok 2024-09-20T22:54:48.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:48 smithi084 ceph-mon[29774]: osdmap e807: 8 total, 8 up, 8 in 2024-09-20T22:54:48.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:48 smithi084 ceph-mon[29774]: 6.16 scrub starts 2024-09-20T22:54:48.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:48 smithi084 ceph-mon[29774]: 6.16 scrub ok 2024-09-20T22:54:48.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:48 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/3694269842' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap", "pgid": "6.1e"}]: dispatch 2024-09-20T22:54:48.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:48 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap", "pgid": "6.1e"}]: dispatch 2024-09-20T22:54:48.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:48 smithi084 ceph-mon[29774]: purged_snaps scrub starts 2024-09-20T22:54:48.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:48 smithi084 ceph-mon[29774]: purged_snaps scrub ok 2024-09-20T22:54:48.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:48 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap", "pgid": "6.1e"}]': finished 2024-09-20T22:54:48.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:48 smithi084 ceph-mon[29774]: osdmap e808: 8 total, 8 up, 8 in 2024-09-20T22:54:48.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:48 smithi084 ceph-mon[40024]: 3.15 scrub starts 2024-09-20T22:54:48.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:48 smithi084 ceph-mon[40024]: 3.15 scrub ok 2024-09-20T22:54:48.618 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:48 smithi084 ceph-mon[40024]: 5.f deep-scrub starts 2024-09-20T22:54:48.618 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:48 smithi084 ceph-mon[40024]: 5.f deep-scrub ok 2024-09-20T22:54:48.618 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:48 smithi084 ceph-mon[40024]: 5.a scrub starts 2024-09-20T22:54:48.618 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:48 smithi084 ceph-mon[40024]: 5.a scrub ok 2024-09-20T22:54:48.618 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:48 smithi084 ceph-mon[40024]: 3.1e scrub starts 2024-09-20T22:54:48.618 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:48 smithi084 ceph-mon[40024]: 3.1e scrub ok 2024-09-20T22:54:48.619 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:48 smithi084 ceph-mon[40024]: osdmap e807: 8 total, 8 up, 8 in 2024-09-20T22:54:48.619 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:48 smithi084 ceph-mon[40024]: 6.16 scrub starts 2024-09-20T22:54:48.619 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:48 smithi084 ceph-mon[40024]: 6.16 scrub ok 2024-09-20T22:54:48.619 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:48 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/3694269842' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap", "pgid": "6.1e"}]: dispatch 2024-09-20T22:54:48.619 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:48 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap", "pgid": "6.1e"}]: dispatch 2024-09-20T22:54:48.619 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:48 smithi084 ceph-mon[40024]: purged_snaps scrub starts 2024-09-20T22:54:48.619 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:48 smithi084 ceph-mon[40024]: purged_snaps scrub ok 2024-09-20T22:54:48.619 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:48 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap", "pgid": "6.1e"}]': finished 2024-09-20T22:54:48.619 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:48 smithi084 ceph-mon[40024]: osdmap e808: 8 total, 8 up, 8 in 2024-09-20T22:54:48.633 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:48.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:48.634 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:54:48.735 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:54:48.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:48 smithi154 ceph-mon[33264]: 3.15 scrub starts 2024-09-20T22:54:48.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:48 smithi154 ceph-mon[33264]: 3.15 scrub ok 2024-09-20T22:54:48.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:48 smithi154 ceph-mon[33264]: 5.f deep-scrub starts 2024-09-20T22:54:48.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:48 smithi154 ceph-mon[33264]: 5.f deep-scrub ok 2024-09-20T22:54:48.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:48 smithi154 ceph-mon[33264]: 5.a scrub starts 2024-09-20T22:54:48.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:48 smithi154 ceph-mon[33264]: 5.a scrub ok 2024-09-20T22:54:48.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:48 smithi154 ceph-mon[33264]: 3.1e scrub starts 2024-09-20T22:54:48.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:48 smithi154 ceph-mon[33264]: 3.1e scrub ok 2024-09-20T22:54:48.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:48 smithi154 ceph-mon[33264]: osdmap e807: 8 total, 8 up, 8 in 2024-09-20T22:54:48.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:48 smithi154 ceph-mon[33264]: 6.16 scrub starts 2024-09-20T22:54:48.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:48 smithi154 ceph-mon[33264]: 6.16 scrub ok 2024-09-20T22:54:48.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:48 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/3694269842' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap", "pgid": "6.1e"}]: dispatch 2024-09-20T22:54:48.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:48 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap", "pgid": "6.1e"}]: dispatch 2024-09-20T22:54:48.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:48 smithi154 ceph-mon[33264]: purged_snaps scrub starts 2024-09-20T22:54:48.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:48 smithi154 ceph-mon[33264]: purged_snaps scrub ok 2024-09-20T22:54:48.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:48 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap", "pgid": "6.1e"}]': finished 2024-09-20T22:54:48.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:48 smithi154 ceph-mon[33264]: osdmap e808: 8 total, 8 up, 8 in 2024-09-20T22:54:48.756 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:48.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:48.757 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:54:48.857 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:54:48.881 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:48.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:48.882 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:54:48.983 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:54:49.005 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:49.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:49.006 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:54:49.107 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:54:49.129 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:49.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:49.130 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:54:49.230 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:54:49.252 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:49.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:49.253 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:54:49.335 INFO:tasks.rados.rados.0.smithi084.stdout:3211: done (2 left) 2024-09-20T22:54:49.335 INFO:tasks.rados.rados.0.smithi084.stdout:3212: write oid 50 current snap is 367 2024-09-20T22:54:49.335 INFO:tasks.rados.rados.0.smithi084.stdout:3212: seq_num 1092 ranges {753244=410387,1834572=518672} 2024-09-20T22:54:49.341 INFO:tasks.rados.rados.0.smithi084.stdout:3212: writing smithi08477203-50 from 753244 to 1163631 tid 1 2024-09-20T22:54:49.349 INFO:tasks.rados.rados.0.smithi084.stdout:3212: writing smithi08477203-50 from 1834572 to 2353244 tid 2 2024-09-20T22:54:49.349 INFO:tasks.rados.rados.0.smithi084.stdout:3213: write (excl) oid 14 current snap is 367 2024-09-20T22:54:49.349 INFO:tasks.rados.rados.0.smithi084.stdout:3207: finishing rollback tid 0 to smithi08477203-13 2024-09-20T22:54:49.349 INFO:tasks.rados.rados.0.smithi084.stdout:3207: finishing rollback tid 1 to smithi08477203-13 2024-09-20T22:54:49.349 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 13 v 2576 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-09-20T22:54:49.349 INFO:tasks.rados.rados.0.smithi084.stdout:3208: finishing copy_from racing read to smithi08477203-40 2024-09-20T22:54:49.349 INFO:tasks.rados.rados.0.smithi084.stdout:3208: finishing copy_from to smithi08477203-40 2024-09-20T22:54:49.349 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 40 v 1813 (ObjNum 795 snap 254 seq_num 795) dirty exists 2024-09-20T22:54:49.349 INFO:tasks.rados.rados.0.smithi084.stdout:3212: finishing write tid 1 to smithi08477203-50 2024-09-20T22:54:49.350 INFO:tasks.rados.rados.0.smithi084.stdout:3213: seq_num 1093 ranges {770051=485636,1776701=511047} 2024-09-20T22:54:49.354 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:54:49.356 INFO:tasks.rados.rados.0.smithi084.stdout:3213: writing smithi08477203-14 from 770051 to 1255687 tid 1 2024-09-20T22:54:49.362 INFO:tasks.rados.rados.0.smithi084.stdout:3213: writing smithi08477203-14 from 1776701 to 2287748 tid 2 2024-09-20T22:54:49.362 INFO:tasks.rados.rados.0.smithi084.stdout:3207: done (3 left) 2024-09-20T22:54:49.362 INFO:tasks.rados.rados.0.smithi084.stdout:3208: done (2 left) 2024-09-20T22:54:49.362 INFO:tasks.rados.rados.0.smithi084.stdout:3214: read oid 15 snap -1 2024-09-20T22:54:49.362 INFO:tasks.rados.rados.0.smithi084.stdout:3214: expect deleted 2024-09-20T22:54:49.362 INFO:tasks.rados.rados.0.smithi084.stdout:3215: write oid 9 current snap is 367 2024-09-20T22:54:49.362 INFO:tasks.rados.rados.0.smithi084.stdout:3212: finishing write tid 2 to smithi08477203-50 2024-09-20T22:54:49.362 INFO:tasks.rados.rados.0.smithi084.stdout:3215: seq_num 1094 ranges {786858=560885,1835183=551675} 2024-09-20T22:54:49.369 INFO:tasks.rados.rados.0.smithi084.stdout:3215: writing smithi08477203-9 from 786858 to 1347743 tid 1 2024-09-20T22:54:49.376 INFO:tasks.rados.rados.0.smithi084.stdout:3215: writing smithi08477203-9 from 1835183 to 2386858 tid 2 2024-09-20T22:54:49.376 INFO:tasks.rados.rados.0.smithi084.stdout:3216: write (excl) oid 35 current snap is 367 2024-09-20T22:54:49.376 INFO:tasks.rados.rados.0.smithi084.stdout:3212: finishing write tid 3 to smithi08477203-50 2024-09-20T22:54:49.376 INFO:tasks.rados.rados.0.smithi084.stdout:3212: finishing write tid 4 to smithi08477203-50 2024-09-20T22:54:49.376 INFO:tasks.rados.rados.0.smithi084.stdout:3212: oid 50 updating version 0 to 1886 2024-09-20T22:54:49.376 INFO:tasks.rados.rados.0.smithi084.stdout:3212: oid 50 version 1886 is already newer than 1885 2024-09-20T22:54:49.376 INFO:tasks.rados.rados.0.smithi084.stdout:3212: oid 50 version 1886 is already newer than 1884 2024-09-20T22:54:49.376 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 50 v 1886 (ObjNum 1092 snap 367 seq_num 1092) dirty exists 2024-09-20T22:54:49.376 INFO:tasks.rados.rados.0.smithi084.stdout:3212: left oid 50 (ObjNum 1092 snap 367 seq_num 1092) 2024-09-20T22:54:49.376 INFO:tasks.rados.rados.0.smithi084.stdout:3213: finishing write tid 1 to smithi08477203-14 2024-09-20T22:54:49.376 INFO:tasks.rados.rados.0.smithi084.stdout:3216: seq_num 1095 ranges {403665=752487,1610018=514716} 2024-09-20T22:54:49.377 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:49.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:49.378 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:54:49.382 INFO:tasks.rados.rados.0.smithi084.stdout:3216: writing smithi08477203-35 from 403665 to 1156152 tid 1 2024-09-20T22:54:49.388 INFO:tasks.rados.rados.0.smithi084.stdout:3216: writing smithi08477203-35 from 1610018 to 2124734 tid 2 2024-09-20T22:54:49.388 INFO:tasks.rados.rados.0.smithi084.stdout:3212: done (4 left) 2024-09-20T22:54:49.388 INFO:tasks.rados.rados.0.smithi084.stdout:3217: snap_create 2024-09-20T22:54:49.388 INFO:tasks.rados.rados.0.smithi084.stdout:3213: finishing write tid 2 to smithi08477203-14 2024-09-20T22:54:49.388 INFO:tasks.rados.rados.0.smithi084.stdout:3213: finishing write tid 3 to smithi08477203-14 2024-09-20T22:54:49.388 INFO:tasks.rados.rados.0.smithi084.stdout:3213: finishing write tid 4 to smithi08477203-14 2024-09-20T22:54:49.388 INFO:tasks.rados.rados.0.smithi084.stdout:3213: oid 14 updating version 0 to 1925 2024-09-20T22:54:49.388 INFO:tasks.rados.rados.0.smithi084.stdout:3213: oid 14 updating version 1925 to 1926 2024-09-20T22:54:49.388 INFO:tasks.rados.rados.0.smithi084.stdout:3213: oid 14 version 1926 is already newer than 1924 2024-09-20T22:54:49.388 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 14 v 1926 (ObjNum 1093 snap 367 seq_num 1093) dirty exists 2024-09-20T22:54:49.388 INFO:tasks.rados.rados.0.smithi084.stdout:3213: left oid 14 (ObjNum 1093 snap 367 seq_num 1093) 2024-09-20T22:54:49.388 INFO:tasks.rados.rados.0.smithi084.stdout:3215: finishing write tid 1 to smithi08477203-9 2024-09-20T22:54:49.389 INFO:tasks.rados.rados.0.smithi084.stdout:3215: finishing write tid 2 to smithi08477203-9 2024-09-20T22:54:49.389 INFO:tasks.rados.rados.0.smithi084.stdout:3215: finishing write tid 3 to smithi08477203-9 2024-09-20T22:54:49.389 INFO:tasks.rados.rados.0.smithi084.stdout:3215: finishing write tid 4 to smithi08477203-9 2024-09-20T22:54:49.389 INFO:tasks.rados.rados.0.smithi084.stdout:3215: oid 9 updating version 0 to 2455 2024-09-20T22:54:49.389 INFO:tasks.rados.rados.0.smithi084.stdout:3215: oid 9 version 2455 is already newer than 2454 2024-09-20T22:54:49.389 INFO:tasks.rados.rados.0.smithi084.stdout:3215: oid 9 updating version 2455 to 2456 2024-09-20T22:54:49.389 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 9 v 2456 (ObjNum 1094 snap 367 seq_num 1094) dirty exists 2024-09-20T22:54:49.389 INFO:tasks.rados.rados.0.smithi084.stdout:3215: left oid 9 (ObjNum 1094 snap 367 seq_num 1094) 2024-09-20T22:54:49.389 INFO:tasks.rados.rados.0.smithi084.stdout:3216: finishing write tid 1 to smithi08477203-35 2024-09-20T22:54:49.390 INFO:tasks.rados.rados.0.smithi084.stdout:3216: finishing write tid 2 to smithi08477203-35 2024-09-20T22:54:49.390 INFO:tasks.rados.rados.0.smithi084.stdout:3216: finishing write tid 3 to smithi08477203-35 2024-09-20T22:54:49.391 INFO:tasks.rados.rados.0.smithi084.stdout:3216: finishing write tid 4 to smithi08477203-35 2024-09-20T22:54:49.391 INFO:tasks.rados.rados.0.smithi084.stdout:3216: oid 35 updating version 0 to 1088 2024-09-20T22:54:49.391 INFO:tasks.rados.rados.0.smithi084.stdout:3216: oid 35 updating version 1088 to 1089 2024-09-20T22:54:49.391 INFO:tasks.rados.rados.0.smithi084.stdout:3216: oid 35 updating version 1089 to 1090 2024-09-20T22:54:49.391 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 35 v 1090 (ObjNum 1095 snap 367 seq_num 1095) dirty exists 2024-09-20T22:54:49.391 INFO:tasks.rados.rados.0.smithi084.stdout:3216: left oid 35 (ObjNum 1095 snap 367 seq_num 1095) 2024-09-20T22:54:49.479 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:54:49.501 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:49.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:49.503 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:54:49.588 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:49 smithi084 ceph-mon[29774]: pgmap v1113: 158 pgs: 4 active+clean+snaptrim, 1 active+remapped+backfill_toofull, 153 active+clean; 582 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 16/1407 objects misplaced (1.137%) 2024-09-20T22:54:49.588 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:49 smithi084 ceph-mon[29774]: 5.8 scrub starts 2024-09-20T22:54:49.588 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:49 smithi084 ceph-mon[29774]: 5.8 scrub ok 2024-09-20T22:54:49.588 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:49 smithi084 ceph-mon[29774]: 2.0 scrub starts 2024-09-20T22:54:49.588 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:49 smithi084 ceph-mon[29774]: 2.0 scrub ok 2024-09-20T22:54:49.588 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:49 smithi084 ceph-mon[29774]: 3.19 scrub starts 2024-09-20T22:54:49.588 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:49 smithi084 ceph-mon[29774]: 3.19 scrub ok 2024-09-20T22:54:49.588 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:49 smithi084 ceph-mon[29774]: osdmap e809: 8 total, 8 up, 8 in 2024-09-20T22:54:49.588 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:49 smithi084 ceph-mon[40024]: pgmap v1113: 158 pgs: 4 active+clean+snaptrim, 1 active+remapped+backfill_toofull, 153 active+clean; 582 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 16/1407 objects misplaced (1.137%) 2024-09-20T22:54:49.589 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:49 smithi084 ceph-mon[40024]: 5.8 scrub starts 2024-09-20T22:54:49.589 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:49 smithi084 ceph-mon[40024]: 5.8 scrub ok 2024-09-20T22:54:49.589 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:49 smithi084 ceph-mon[40024]: 2.0 scrub starts 2024-09-20T22:54:49.589 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:49 smithi084 ceph-mon[40024]: 2.0 scrub ok 2024-09-20T22:54:49.589 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:49 smithi084 ceph-mon[40024]: 3.19 scrub starts 2024-09-20T22:54:49.589 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:49 smithi084 ceph-mon[40024]: 3.19 scrub ok 2024-09-20T22:54:49.589 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:49 smithi084 ceph-mon[40024]: osdmap e809: 8 total, 8 up, 8 in 2024-09-20T22:54:49.603 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:54:49.625 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:49.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:49.626 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:54:49.691 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:54:49.695 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:54:49.727 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:54:49.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:49.749 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:49.749 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:54:49.754 INFO:journalctl@ceph.iscsi.iscsi.a.smithi154.stdout:Sep 20 22:54:49 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-iscsi-iscsi-a[59730]: debug there is no tcmu-runner data available 2024-09-20T22:54:49.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:49 smithi154 ceph-mon[33264]: pgmap v1113: 158 pgs: 4 active+clean+snaptrim, 1 active+remapped+backfill_toofull, 153 active+clean; 582 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 16/1407 objects misplaced (1.137%) 2024-09-20T22:54:49.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:49 smithi154 ceph-mon[33264]: 5.8 scrub starts 2024-09-20T22:54:49.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:49 smithi154 ceph-mon[33264]: 5.8 scrub ok 2024-09-20T22:54:49.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:49 smithi154 ceph-mon[33264]: 2.0 scrub starts 2024-09-20T22:54:49.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:49 smithi154 ceph-mon[33264]: 2.0 scrub ok 2024-09-20T22:54:49.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:49 smithi154 ceph-mon[33264]: 3.19 scrub starts 2024-09-20T22:54:49.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:49 smithi154 ceph-mon[33264]: 3.19 scrub ok 2024-09-20T22:54:49.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:49 smithi154 ceph-mon[33264]: osdmap e809: 8 total, 8 up, 8 in 2024-09-20T22:54:49.850 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:54:49.872 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:49.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:49.873 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:54:49.973 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:54:49.995 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:49.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:49.996 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:54:50.096 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:54:50.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:50.118 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:50.119 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:54:50.219 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:54:50.241 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:50.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:50.242 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:54:50.340 INFO:tasks.rados.rados.0.smithi084.stdout:3213: done (4 left) 2024-09-20T22:54:50.340 INFO:tasks.rados.rados.0.smithi084.stdout:3214: done (3 left) 2024-09-20T22:54:50.340 INFO:tasks.rados.rados.0.smithi084.stdout:3215: done (2 left) 2024-09-20T22:54:50.341 INFO:tasks.rados.rados.0.smithi084.stdout:3216: done (1 left) 2024-09-20T22:54:50.341 INFO:tasks.rados.rados.0.smithi084.stdout:3217: done (0 left) 2024-09-20T22:54:50.341 INFO:tasks.rados.rados.0.smithi084.stdout:3218: read oid 1 snap -1 2024-09-20T22:54:50.341 INFO:tasks.rados.rados.0.smithi084.stdout:3218: expect (ObjNum 738 snap 235 seq_num 738) 2024-09-20T22:54:50.341 INFO:tasks.rados.rados.0.smithi084.stdout:3219: read oid 40 snap -1 2024-09-20T22:54:50.341 INFO:tasks.rados.rados.0.smithi084.stdout:3219: expect (ObjNum 795 snap 254 seq_num 795) 2024-09-20T22:54:50.341 INFO:tasks.rados.rados.0.smithi084.stdout:3220: copy_from oid 18 from oid 47 current snap is 368 2024-09-20T22:54:50.341 INFO:tasks.rados.rados.0.smithi084.stdout:3221: read oid 27 snap -1 2024-09-20T22:54:50.341 INFO:tasks.rados.rados.0.smithi084.stdout:3221: expect (ObjNum 1077 snap 358 seq_num 1077) 2024-09-20T22:54:50.341 INFO:tasks.rados.rados.0.smithi084.stdout:3222: copy_from oid 3 from oid 25 current snap is 368 2024-09-20T22:54:50.341 INFO:tasks.rados.rados.0.smithi084.stdout:3223: write (excl) oid 5 current snap is 368 2024-09-20T22:54:50.341 INFO:tasks.rados.rados.0.smithi084.stdout:3223: seq_num 1096 ranges {420472=427736,1268500=658374,2347953=72519} 2024-09-20T22:54:50.343 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:54:50.345 INFO:tasks.rados.rados.0.smithi084.stdout:3223: writing smithi08477203-5 from 420472 to 848208 tid 1 2024-09-20T22:54:50.352 INFO:tasks.rados.rados.0.smithi084.stdout:3223: writing smithi08477203-5 from 1268500 to 1926874 tid 2 2024-09-20T22:54:50.355 INFO:tasks.rados.rados.0.smithi084.stdout:3223: writing smithi08477203-5 from 2347953 to 2420472 tid 3 2024-09-20T22:54:50.355 INFO:tasks.rados.rados.0.smithi084.stdout:3224: write (excl) oid 15 current snap is 368 2024-09-20T22:54:50.355 INFO:tasks.rados.rados.0.smithi084.stdout:3224: seq_num 1097 ranges {437279=502985,1610629=518385} 2024-09-20T22:54:50.361 INFO:tasks.rados.rados.0.smithi084.stdout:3224: writing smithi08477203-15 from 437279 to 940264 tid 1 2024-09-20T22:54:50.369 INFO:tasks.rados.rados.0.smithi084.stdout:3224: writing smithi08477203-15 from 1610629 to 2129014 tid 2 2024-09-20T22:54:50.369 INFO:tasks.rados.rados.0.smithi084.stdout:3225: copy_from oid 14 from oid 2 current snap is 368 2024-09-20T22:54:50.369 INFO:tasks.rados.rados.0.smithi084.stdout:3218: expect (ObjNum 738 snap 235 seq_num 738) 2024-09-20T22:54:50.370 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:50.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:50.371 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:54:50.454 INFO:tasks.rados.rados.0.smithi084.stdout:3221: expect (ObjNum 1077 snap 358 seq_num 1077) 2024-09-20T22:54:50.472 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:54:50.494 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:50.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:50.495 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:54:50.525 INFO:tasks.rados.rados.0.smithi084.stdout:3219: expect (ObjNum 795 snap 254 seq_num 795) 2024-09-20T22:54:50.579 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:50 smithi084 ceph-mon[40024]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:54:50.579 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:50 smithi084 ceph-mon[40024]: 3.0 scrub starts 2024-09-20T22:54:50.579 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:50 smithi084 ceph-mon[40024]: 3.0 scrub ok 2024-09-20T22:54:50.579 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:50 smithi084 ceph-mon[40024]: 4.1f scrub starts 2024-09-20T22:54:50.579 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:50 smithi084 ceph-mon[40024]: 4.1f scrub ok 2024-09-20T22:54:50.579 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:50 smithi084 ceph-mon[40024]: purged_snaps scrub starts 2024-09-20T22:54:50.579 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:50 smithi084 ceph-mon[40024]: purged_snaps scrub ok 2024-09-20T22:54:50.580 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:50 smithi084 ceph-mon[40024]: osdmap e810: 8 total, 8 up, 8 in 2024-09-20T22:54:50.580 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:50 smithi084 ceph-mon[29774]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:54:50.580 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:50 smithi084 ceph-mon[29774]: 3.0 scrub starts 2024-09-20T22:54:50.580 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:50 smithi084 ceph-mon[29774]: 3.0 scrub ok 2024-09-20T22:54:50.580 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:50 smithi084 ceph-mon[29774]: 4.1f scrub starts 2024-09-20T22:54:50.580 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:50 smithi084 ceph-mon[29774]: 4.1f scrub ok 2024-09-20T22:54:50.580 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:50 smithi084 ceph-mon[29774]: purged_snaps scrub starts 2024-09-20T22:54:50.580 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:50 smithi084 ceph-mon[29774]: purged_snaps scrub ok 2024-09-20T22:54:50.580 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:50 smithi084 ceph-mon[29774]: osdmap e810: 8 total, 8 up, 8 in 2024-09-20T22:54:50.586 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-20T22:54:50.586 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2024-09-20T22:54:50.586 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2024-09-20T22:54:50.586 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-20T22:54:50.596 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:54:50.619 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:50.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:50.620 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:54:50.639 INFO:tasks.rados.rados.0.smithi084.stdout:3223: finishing write tid 1 to smithi08477203-5 2024-09-20T22:54:50.639 INFO:tasks.rados.rados.0.smithi084.stdout:3223: finishing write tid 2 to smithi08477203-5 2024-09-20T22:54:50.639 INFO:tasks.rados.rados.0.smithi084.stdout:3218: done (7 left) 2024-09-20T22:54:50.639 INFO:tasks.rados.rados.0.smithi084.stdout:3219: done (6 left) 2024-09-20T22:54:50.639 INFO:tasks.rados.rados.0.smithi084.stdout:3221: done (5 left) 2024-09-20T22:54:50.639 INFO:tasks.rados.rados.0.smithi084.stdout:3226: read oid 40 snap -1 2024-09-20T22:54:50.639 INFO:tasks.rados.rados.0.smithi084.stdout:3226: expect (ObjNum 795 snap 254 seq_num 795) 2024-09-20T22:54:50.640 INFO:tasks.rados.rados.0.smithi084.stdout:3227: snap_create 2024-09-20T22:54:50.640 INFO:tasks.rados.rados.0.smithi084.stdout:3223: finishing write tid 3 to smithi08477203-5 2024-09-20T22:54:50.640 INFO:tasks.rados.rados.0.smithi084.stdout:3223: finishing write tid 4 to smithi08477203-5 2024-09-20T22:54:50.640 INFO:tasks.rados.rados.0.smithi084.stdout:3223: finishing write tid 5 to smithi08477203-5 2024-09-20T22:54:50.640 INFO:tasks.rados.rados.0.smithi084.stdout:3223: oid 5 updating version 0 to 1093 2024-09-20T22:54:50.640 INFO:tasks.rados.rados.0.smithi084.stdout:3223: oid 5 updating version 1093 to 1094 2024-09-20T22:54:50.640 INFO:tasks.rados.rados.0.smithi084.stdout:3223: oid 5 version 1094 is already newer than 1091 2024-09-20T22:54:50.640 INFO:tasks.rados.rados.0.smithi084.stdout:3223: oid 5 version 1094 is already newer than 1092 2024-09-20T22:54:50.640 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 5 v 1094 (ObjNum 1096 snap 368 seq_num 1096) dirty exists 2024-09-20T22:54:50.640 INFO:tasks.rados.rados.0.smithi084.stdout:3223: left oid 5 (ObjNum 1096 snap 368 seq_num 1096) 2024-09-20T22:54:50.640 INFO:tasks.rados.rados.0.smithi084.stdout:3222: finishing copy_from racing read to smithi08477203-3 2024-09-20T22:54:50.640 INFO:tasks.rados.rados.0.smithi084.stdout:3220: finishing copy_from racing read to smithi08477203-18 2024-09-20T22:54:50.640 INFO:tasks.rados.rados.0.smithi084.stdout:3222: finishing copy_from to smithi08477203-3 2024-09-20T22:54:50.640 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 3 v 2542 (ObjNum 1070 snap 355 seq_num 1070) dirty exists 2024-09-20T22:54:50.641 INFO:tasks.rados.rados.0.smithi084.stdout:3224: finishing write tid 1 to smithi08477203-15 2024-09-20T22:54:50.641 INFO:tasks.rados.rados.0.smithi084.stdout:3220: finishing copy_from to smithi08477203-18 2024-09-20T22:54:50.641 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 18 v 2453 (ObjNum 1051 snap 345 seq_num 1051) dirty exists 2024-09-20T22:54:50.641 INFO:tasks.rados.rados.0.smithi084.stdout:3224: finishing write tid 2 to smithi08477203-15 2024-09-20T22:54:50.641 INFO:tasks.rados.rados.0.smithi084.stdout:3224: finishing write tid 3 to smithi08477203-15 2024-09-20T22:54:50.641 INFO:tasks.rados.rados.0.smithi084.stdout:3224: finishing write tid 4 to smithi08477203-15 2024-09-20T22:54:50.641 INFO:tasks.rados.rados.0.smithi084.stdout:3224: oid 15 updating version 0 to 2545 2024-09-20T22:54:50.641 INFO:tasks.rados.rados.0.smithi084.stdout:3224: oid 15 version 2545 is already newer than 2544 2024-09-20T22:54:50.641 INFO:tasks.rados.rados.0.smithi084.stdout:3224: oid 15 version 2545 is already newer than 2543 2024-09-20T22:54:50.641 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 15 v 2545 (ObjNum 1097 snap 368 seq_num 1097) dirty exists 2024-09-20T22:54:50.641 INFO:tasks.rados.rados.0.smithi084.stdout:3224: left oid 15 (ObjNum 1097 snap 368 seq_num 1097) 2024-09-20T22:54:50.641 INFO:tasks.rados.rados.0.smithi084.stdout:3225: finishing copy_from racing read to smithi08477203-14 2024-09-20T22:54:50.641 INFO:tasks.rados.rados.0.smithi084.stdout:3225: finishing copy_from to smithi08477203-14 2024-09-20T22:54:50.641 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 14 v 1835 (ObjNum 918 snap 302 seq_num 918) dirty exists 2024-09-20T22:54:50.642 INFO:tasks.rados.rados.0.smithi084.stdout:3226: expect (ObjNum 795 snap 254 seq_num 795) 2024-09-20T22:54:50.720 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:54:50.741 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:50.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:50.742 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:54:50.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:50 smithi154 ceph-mon[33264]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:54:50.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:50 smithi154 ceph-mon[33264]: 3.0 scrub starts 2024-09-20T22:54:50.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:50 smithi154 ceph-mon[33264]: 3.0 scrub ok 2024-09-20T22:54:50.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:50 smithi154 ceph-mon[33264]: 4.1f scrub starts 2024-09-20T22:54:50.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:50 smithi154 ceph-mon[33264]: 4.1f scrub ok 2024-09-20T22:54:50.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:50 smithi154 ceph-mon[33264]: purged_snaps scrub starts 2024-09-20T22:54:50.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:50 smithi154 ceph-mon[33264]: purged_snaps scrub ok 2024-09-20T22:54:50.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:50 smithi154 ceph-mon[33264]: osdmap e810: 8 total, 8 up, 8 in 2024-09-20T22:54:50.843 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:54:50.859 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-20T22:54:50.864 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:50.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:50.865 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:54:50.966 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:54:50.987 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:50.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:50.988 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:54:51.088 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:54:51.109 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:51.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:51.110 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:54:51.211 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:54:51.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:51.233 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:51.233 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:54:51.327 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.3 dump_blocked_ops 2024-09-20T22:54:51.334 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:54:51.392 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:51.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:51.393 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:54:51.442 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:54:51.461 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is set 2024-09-20T22:54:51.466 INFO:tasks.rados.rados.0.smithi084.stdout:3220: done (6 left) 2024-09-20T22:54:51.467 INFO:tasks.rados.rados.0.smithi084.stdout:3222: done (5 left) 2024-09-20T22:54:51.467 INFO:tasks.rados.rados.0.smithi084.stdout:3223: done (4 left) 2024-09-20T22:54:51.467 INFO:tasks.rados.rados.0.smithi084.stdout:3224: done (3 left) 2024-09-20T22:54:51.467 INFO:tasks.rados.rados.0.smithi084.stdout:3225: done (2 left) 2024-09-20T22:54:51.467 INFO:tasks.rados.rados.0.smithi084.stdout:3226: done (1 left) 2024-09-20T22:54:51.467 INFO:tasks.rados.rados.0.smithi084.stdout:3227: done (0 left) 2024-09-20T22:54:51.467 INFO:tasks.rados.rados.0.smithi084.stdout:3228: snap_create 2024-09-20T22:54:51.493 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:54:51.517 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:51.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:51.518 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:54:51.569 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:51 smithi084 ceph-mon[29774]: pgmap v1116: 158 pgs: 1 active+clean+remapped+snaptrim_wait, 1 peering, 5 active+clean+snaptrim, 151 active+clean; 523 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 821 KiB/s wr, 2 op/s; 8.9 MiB/s, 3 objects/s recovering 2024-09-20T22:54:51.569 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:51 smithi084 ceph-mon[29774]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-09-20T22:54:51.569 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:51 smithi084 ceph-mon[29774]: Cluster is now healthy 2024-09-20T22:54:51.569 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:51 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/3273513586' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-20T22:54:51.569 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:51 smithi084 ceph-mon[40024]: pgmap v1116: 158 pgs: 1 active+clean+remapped+snaptrim_wait, 1 peering, 5 active+clean+snaptrim, 151 active+clean; 523 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 821 KiB/s wr, 2 op/s; 8.9 MiB/s, 3 objects/s recovering 2024-09-20T22:54:51.570 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:51 smithi084 ceph-mon[40024]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-09-20T22:54:51.570 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:51 smithi084 ceph-mon[40024]: Cluster is now healthy 2024-09-20T22:54:51.570 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:51 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/3273513586' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-20T22:54:51.619 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:54:51.641 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:51.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:51.642 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:54:51.743 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:54:51.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:51 smithi154 ceph-mon[33264]: pgmap v1116: 158 pgs: 1 active+clean+remapped+snaptrim_wait, 1 peering, 5 active+clean+snaptrim, 151 active+clean; 523 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 821 KiB/s wr, 2 op/s; 8.9 MiB/s, 3 objects/s recovering 2024-09-20T22:54:51.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:51 smithi154 ceph-mon[33264]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-09-20T22:54:51.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:51 smithi154 ceph-mon[33264]: Cluster is now healthy 2024-09-20T22:54:51.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:51 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/3273513586' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-20T22:54:51.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:51.765 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:51.765 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:54:51.866 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:54:51.886 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:51.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:51.887 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:54:51.988 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:54:52.009 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:52.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:52.010 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:54:52.111 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:54:52.132 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:52.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:52.133 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:54:52.234 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:54:52.255 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:52.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:52.256 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:54:52.321 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:54:52.322 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:54:52.339 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:54:52.340 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:54:52.356 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:54:52.356 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:54:52.356 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:54:52.372 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-20T22:54:52.372 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T22:54:52.377 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:52.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:52.377 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:54:52.389 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:54:52.390 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:54:52.406 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:54:52.407 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:54:52.423 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:54:52.423 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:54:52.441 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:54:52.441 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:54:52.473 INFO:tasks.rados.rados.0.smithi084.stdout:3228: done (0 left) 2024-09-20T22:54:52.474 INFO:tasks.rados.rados.0.smithi084.stdout:3229: write oid 32 current snap is 370 2024-09-20T22:54:52.474 INFO:tasks.rados.rados.0.smithi084.stdout:3229: seq_num 1098 ranges {454086=578234,1669111=662043} 2024-09-20T22:54:52.478 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:54:52.479 INFO:tasks.rados.rados.0.smithi084.stdout:3229: writing smithi08477203-32 from 454086 to 1032320 tid 1 2024-09-20T22:54:52.486 INFO:tasks.rados.rados.0.smithi084.stdout:3229: writing smithi08477203-32 from 1669111 to 2331154 tid 2 2024-09-20T22:54:52.486 INFO:tasks.rados.rados.0.smithi084.stdout:3230: read oid 28 snap -1 2024-09-20T22:54:52.486 INFO:tasks.rados.rados.0.smithi084.stdout:3230: expect (ObjNum 1072 snap 356 seq_num 1072) 2024-09-20T22:54:52.486 INFO:tasks.rados.rados.0.smithi084.stdout:3231: snap_create 2024-09-20T22:54:52.486 INFO:tasks.rados.rados.0.smithi084.stdout:3229: finishing write tid 1 to smithi08477203-32 2024-09-20T22:54:52.490 INFO:tasks.rados.rados.0.smithi084.stdout:3229: finishing write tid 2 to smithi08477203-32 2024-09-20T22:54:52.490 INFO:tasks.rados.rados.0.smithi084.stdout:3229: finishing write tid 3 to smithi08477203-32 2024-09-20T22:54:52.490 INFO:tasks.rados.rados.0.smithi084.stdout:3229: finishing write tid 4 to smithi08477203-32 2024-09-20T22:54:52.490 INFO:tasks.rados.rados.0.smithi084.stdout:3229: oid 32 updating version 0 to 1841 2024-09-20T22:54:52.490 INFO:tasks.rados.rados.0.smithi084.stdout:3229: oid 32 updating version 1841 to 1842 2024-09-20T22:54:52.490 INFO:tasks.rados.rados.0.smithi084.stdout:3229: oid 32 updating version 1842 to 1843 2024-09-20T22:54:52.490 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 32 v 1843 (ObjNum 1098 snap 370 seq_num 1098) dirty exists 2024-09-20T22:54:52.490 INFO:tasks.rados.rados.0.smithi084.stdout:3229: left oid 32 (ObjNum 1098 snap 370 seq_num 1098) 2024-09-20T22:54:52.497 INFO:tasks.rados.rados.0.smithi084.stdout:3230: expect (ObjNum 1072 snap 356 seq_num 1072) 2024-09-20T22:54:52.501 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:52.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:52.502 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:54:52.602 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:54:52.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:52 smithi084 ceph-mon[29774]: 5.7 scrub starts 2024-09-20T22:54:52.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:52 smithi084 ceph-mon[29774]: 5.7 scrub ok 2024-09-20T22:54:52.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:52 smithi084 ceph-mon[29774]: 3.a scrub starts 2024-09-20T22:54:52.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:52 smithi084 ceph-mon[29774]: 3.a scrub ok 2024-09-20T22:54:52.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:52 smithi084 ceph-mon[29774]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:54:52.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:52 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/3273513586' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T22:54:52.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:52 smithi084 ceph-mon[29774]: osdmap e811: 8 total, 8 up, 8 in 2024-09-20T22:54:52.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:52 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:54:52.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:52 smithi084 ceph-mon[29774]: 4.7 scrub starts 2024-09-20T22:54:52.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:52 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/442378614' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:54:52.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:52 smithi084 ceph-mon[40024]: 5.7 scrub starts 2024-09-20T22:54:52.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:52 smithi084 ceph-mon[40024]: 5.7 scrub ok 2024-09-20T22:54:52.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:52 smithi084 ceph-mon[40024]: 3.a scrub starts 2024-09-20T22:54:52.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:52 smithi084 ceph-mon[40024]: 3.a scrub ok 2024-09-20T22:54:52.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:52 smithi084 ceph-mon[40024]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:54:52.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:52 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/3273513586' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T22:54:52.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:52 smithi084 ceph-mon[40024]: osdmap e811: 8 total, 8 up, 8 in 2024-09-20T22:54:52.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:52 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:54:52.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:52 smithi084 ceph-mon[40024]: 4.7 scrub starts 2024-09-20T22:54:52.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:52 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/442378614' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:54:52.624 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:52.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:52.625 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:54:52.691 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:52 smithi154 ceph-mon[33264]: 5.7 scrub starts 2024-09-20T22:54:52.691 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:52 smithi154 ceph-mon[33264]: 5.7 scrub ok 2024-09-20T22:54:52.692 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:52 smithi154 ceph-mon[33264]: 3.a scrub starts 2024-09-20T22:54:52.692 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:52 smithi154 ceph-mon[33264]: 3.a scrub ok 2024-09-20T22:54:52.692 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:52 smithi154 ceph-mon[33264]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:54:52.692 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:52 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/3273513586' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T22:54:52.692 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:52 smithi154 ceph-mon[33264]: osdmap e811: 8 total, 8 up, 8 in 2024-09-20T22:54:52.692 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:52 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:54:52.692 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:52 smithi154 ceph-mon[33264]: 4.7 scrub starts 2024-09-20T22:54:52.692 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:52 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/442378614' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:54:52.726 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:54:52.747 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:52.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:52.748 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:54:52.847 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:54:52.849 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:54:52.870 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:52.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:52.871 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:54:52.972 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:54:52.995 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:52.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:52.996 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:54:53.096 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:54:53.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:53.120 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:53.120 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:54:53.221 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:54:53.243 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:53.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:53.244 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:54:53.344 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:54:53.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:53.366 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:53.366 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:54:53.467 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:54:53.480 INFO:tasks.rados.rados.0.smithi084.stdout:3229: done (2 left) 2024-09-20T22:54:53.480 INFO:tasks.rados.rados.0.smithi084.stdout:3230: done (1 left) 2024-09-20T22:54:53.480 INFO:tasks.rados.rados.0.smithi084.stdout:3231: done (0 left) 2024-09-20T22:54:53.480 INFO:tasks.rados.rados.0.smithi084.stdout:3232: copy_from oid 44 from oid 48 current snap is 371 2024-09-20T22:54:53.480 INFO:tasks.rados.rados.0.smithi084.stdout:3233: copy_from oid 19 from oid 13 current snap is 371 2024-09-20T22:54:53.480 INFO:tasks.rados.rados.0.smithi084.stdout:3234: read oid 32 snap -1 2024-09-20T22:54:53.480 INFO:tasks.rados.rados.0.smithi084.stdout:3234: expect (ObjNum 1098 snap 370 seq_num 1098) 2024-09-20T22:54:53.480 INFO:tasks.rados.rados.0.smithi084.stdout:3235: snap_create 2024-09-20T22:54:53.482 INFO:tasks.rados.rados.0.smithi084.stdout:3232: finishing copy_from to smithi08477203-44 2024-09-20T22:54:53.482 INFO:tasks.rados.rados.0.smithi084.stdout:3232: got expected ENOENT (src dne) 2024-09-20T22:54:53.483 INFO:tasks.rados.rados.0.smithi084.stdout:3233: finishing copy_from to smithi08477203-19 2024-09-20T22:54:53.483 INFO:tasks.rados.rados.0.smithi084.stdout:3233: got expected ENOENT (src dne) 2024-09-20T22:54:53.483 INFO:tasks.rados.rados.0.smithi084.stdout:3233: finishing copy_from racing read to smithi08477203-19 2024-09-20T22:54:53.483 INFO:tasks.rados.rados.0.smithi084.stdout:3232: finishing copy_from racing read to smithi08477203-44 2024-09-20T22:54:53.484 INFO:tasks.rados.rados.0.smithi084.stdout:3234: expect (ObjNum 1098 snap 370 seq_num 1098) 2024-09-20T22:54:53.490 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:53.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:53.491 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:54:53.500 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-20T22:54:53.500 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:54:53.592 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:54:53.613 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:53.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:53.614 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:54:53.678 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:53 smithi154 ceph-mon[33264]: pgmap v1119: 158 pgs: 1 active+clean+remapped+snaptrim_wait, 1 peering, 5 active+clean+snaptrim, 151 active+clean; 523 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 821 KiB/s wr, 2 op/s; 8.9 MiB/s, 3 objects/s recovering 2024-09-20T22:54:53.678 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:53 smithi154 ceph-mon[33264]: osdmap e812: 8 total, 8 up, 8 in 2024-09-20T22:54:53.678 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:53 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:54:53.678 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:53 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:54:53.715 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:54:53.736 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:53.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:53.737 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:54:53.838 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:54:53.859 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:53.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:53.860 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:54:53.864 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:53 smithi084 ceph-mon[29774]: pgmap v1119: 158 pgs: 1 active+clean+remapped+snaptrim_wait, 1 peering, 5 active+clean+snaptrim, 151 active+clean; 523 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 821 KiB/s wr, 2 op/s; 8.9 MiB/s, 3 objects/s recovering 2024-09-20T22:54:53.864 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:53 smithi084 ceph-mon[29774]: osdmap e812: 8 total, 8 up, 8 in 2024-09-20T22:54:53.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:53 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:54:53.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:53 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:54:53.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:53 smithi084 ceph-mon[40024]: pgmap v1119: 158 pgs: 1 active+clean+remapped+snaptrim_wait, 1 peering, 5 active+clean+snaptrim, 151 active+clean; 523 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 821 KiB/s wr, 2 op/s; 8.9 MiB/s, 3 objects/s recovering 2024-09-20T22:54:53.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:53 smithi084 ceph-mon[40024]: osdmap e812: 8 total, 8 up, 8 in 2024-09-20T22:54:53.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:53 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:54:53.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:53 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:54:53.961 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:54:53.982 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:53.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:53.983 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:54:54.084 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:54:54.105 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:54.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:54.106 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:54:54.207 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:54:54.228 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:54.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:54.229 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:54:54.311 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.3 dump_historic_ops 2024-09-20T22:54:54.330 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:54:54.351 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:54.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:54.352 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:54:54.453 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:54:54.475 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:54.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:54.476 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:54:54.479 INFO:tasks.rados.rados.0.smithi084.stdout:3232: done (3 left) 2024-09-20T22:54:54.479 INFO:tasks.rados.rados.0.smithi084.stdout:3233: done (2 left) 2024-09-20T22:54:54.480 INFO:tasks.rados.rados.0.smithi084.stdout:3234: done (1 left) 2024-09-20T22:54:54.480 INFO:tasks.rados.rados.0.smithi084.stdout:3235: done (0 left) 2024-09-20T22:54:54.480 INFO:tasks.rados.rados.0.smithi084.stdout:3236: snap_remove snap 320 2024-09-20T22:54:54.576 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:54:54.598 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:54.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:54.599 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:54:54.614 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:54 smithi084 ceph-mon[29774]: osdmap e813: 8 total, 8 up, 8 in 2024-09-20T22:54:54.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:54 smithi084 ceph-mon[40024]: osdmap e813: 8 total, 8 up, 8 in 2024-09-20T22:54:54.616 INFO:tasks.thrashosds.thrasher:in_osds: [2, 3, 5, 6, 7, 0, 4, 1] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-20T22:54:54.616 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-20T22:54:54.616 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-09-20T22:54:54.616 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd dump -f json-pretty 2024-09-20T22:54:54.699 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:54:54.720 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:54.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:54.721 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:54:54.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:54 smithi154 ceph-mon[33264]: osdmap e813: 8 total, 8 up, 8 in 2024-09-20T22:54:54.822 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:54:54.843 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:54.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:54.844 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:54:54.855 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-20T22:54:54.944 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:54:54.965 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:54.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:54.966 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:54:55.067 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:54:55.089 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:55.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:55.090 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:54:55.138 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:54:55.190 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:54:55.212 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:55.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:55.213 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:54:55.314 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:54:55.335 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:55.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:55.335 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:54:55.436 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:54:55.458 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:55.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:55.459 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:54:55.500 INFO:tasks.rados.rados.0.smithi084.stdout:3236: done (0 left) 2024-09-20T22:54:55.500 INFO:tasks.rados.rados.0.smithi084.stdout:3237: snap_remove snap 355 2024-09-20T22:54:55.559 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:54:55.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:55.581 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:55.582 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:54:55.610 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:55 smithi084 ceph-mon[40024]: pgmap v1122: 158 pgs: 3 active+clean+snaptrim, 155 active+clean; 531 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 15 MiB/s rd, 4.5 MiB/s wr, 25 op/s 2024-09-20T22:54:55.610 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:55 smithi084 ceph-mon[40024]: osdmap e814: 8 total, 8 up, 8 in 2024-09-20T22:54:55.610 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:55 smithi084 ceph-mon[40024]: purged_snaps scrub starts 2024-09-20T22:54:55.610 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:55 smithi084 ceph-mon[40024]: purged_snaps scrub ok 2024-09-20T22:54:55.610 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:55 smithi084 ceph-mon[29774]: pgmap v1122: 158 pgs: 3 active+clean+snaptrim, 155 active+clean; 531 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 15 MiB/s rd, 4.5 MiB/s wr, 25 op/s 2024-09-20T22:54:55.610 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:55 smithi084 ceph-mon[29774]: osdmap e814: 8 total, 8 up, 8 in 2024-09-20T22:54:55.610 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:55 smithi084 ceph-mon[29774]: purged_snaps scrub starts 2024-09-20T22:54:55.611 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:55 smithi084 ceph-mon[29774]: purged_snaps scrub ok 2024-09-20T22:54:55.615 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:54:55.683 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:54:55.704 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:55.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:55.705 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:54:55.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:55 smithi154 ceph-mon[33264]: pgmap v1122: 158 pgs: 3 active+clean+snaptrim, 155 active+clean; 531 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 15 MiB/s rd, 4.5 MiB/s wr, 25 op/s 2024-09-20T22:54:55.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:55 smithi154 ceph-mon[33264]: osdmap e814: 8 total, 8 up, 8 in 2024-09-20T22:54:55.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:55 smithi154 ceph-mon[33264]: purged_snaps scrub starts 2024-09-20T22:54:55.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:55 smithi154 ceph-mon[33264]: purged_snaps scrub ok 2024-09-20T22:54:55.805 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:54:55.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:55.829 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:55.829 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:54:55.930 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:54:55.951 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:55.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:55.952 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:54:56.052 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:54:56.073 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:56.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:56.074 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:54:56.174 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:54:56.197 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:56.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:56.198 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:54:56.299 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:54:56.310 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-20T22:54:56.311 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-20T22:54:56.311 INFO:teuthology.orchestra.run.smithi084.stdout: "epoch": 815, 2024-09-20T22:54:56.311 INFO:teuthology.orchestra.run.smithi084.stdout: "fsid": "a9c0bc70-77a0-11ef-baf5-efdc52797490", 2024-09-20T22:54:56.311 INFO:teuthology.orchestra.run.smithi084.stdout: "created": "2024-09-20T22:36:30.240626+0000", 2024-09-20T22:54:56.311 INFO:teuthology.orchestra.run.smithi084.stdout: "modified": "2024-09-20T22:54:55.485925+0000", 2024-09-20T22:54:56.311 INFO:teuthology.orchestra.run.smithi084.stdout: "last_up_change": "2024-09-20T22:41:27.098565+0000", 2024-09-20T22:54:56.311 INFO:teuthology.orchestra.run.smithi084.stdout: "last_in_change": "2024-09-20T22:54:16.306377+0000", 2024-09-20T22:54:56.311 INFO:teuthology.orchestra.run.smithi084.stdout: "flags": "noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-20T22:54:56.311 INFO:teuthology.orchestra.run.smithi084.stdout: "flags_num": 5801984, 2024-09-20T22:54:56.311 INFO:teuthology.orchestra.run.smithi084.stdout: "flags_set": [ 2024-09-20T22:54:56.311 INFO:teuthology.orchestra.run.smithi084.stdout: "noscrub", 2024-09-20T22:54:56.312 INFO:teuthology.orchestra.run.smithi084.stdout: "pglog_hardlimit", 2024-09-20T22:54:56.312 INFO:teuthology.orchestra.run.smithi084.stdout: "purged_snapdirs", 2024-09-20T22:54:56.312 INFO:teuthology.orchestra.run.smithi084.stdout: "recovery_deletes", 2024-09-20T22:54:56.312 INFO:teuthology.orchestra.run.smithi084.stdout: "sortbitwise" 2024-09-20T22:54:56.312 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2024-09-20T22:54:56.312 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_version": 18, 2024-09-20T22:54:56.312 INFO:teuthology.orchestra.run.smithi084.stdout: "full_ratio": 0.94999998807907104, 2024-09-20T22:54:56.312 INFO:teuthology.orchestra.run.smithi084.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-20T22:54:56.312 INFO:teuthology.orchestra.run.smithi084.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-20T22:54:56.312 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_snapshot": "", 2024-09-20T22:54:56.312 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_max": 7, 2024-09-20T22:54:56.312 INFO:teuthology.orchestra.run.smithi084.stdout: "max_osd": 8, 2024-09-20T22:54:56.312 INFO:teuthology.orchestra.run.smithi084.stdout: "require_min_compat_client": "luminous", 2024-09-20T22:54:56.312 INFO:teuthology.orchestra.run.smithi084.stdout: "min_compat_client": "luminous", 2024-09-20T22:54:56.313 INFO:teuthology.orchestra.run.smithi084.stdout: "require_osd_release": "squid", 2024-09-20T22:54:56.313 INFO:teuthology.orchestra.run.smithi084.stdout: "allow_crimson": false, 2024-09-20T22:54:56.313 INFO:teuthology.orchestra.run.smithi084.stdout: "pools": [ 2024-09-20T22:54:56.313 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:56.313 INFO:teuthology.orchestra.run.smithi084.stdout: "pool": 1, 2024-09-20T22:54:56.313 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_name": ".mgr", 2024-09-20T22:54:56.313 INFO:teuthology.orchestra.run.smithi084.stdout: "create_time": "2024-09-20T22:39:21.984381+0000", 2024-09-20T22:54:56.313 INFO:teuthology.orchestra.run.smithi084.stdout: "flags": 1, 2024-09-20T22:54:56.313 INFO:teuthology.orchestra.run.smithi084.stdout: "flags_names": "hashpspool", 2024-09-20T22:54:56.313 INFO:teuthology.orchestra.run.smithi084.stdout: "type": 1, 2024-09-20T22:54:56.313 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 3, 2024-09-20T22:54:56.313 INFO:teuthology.orchestra.run.smithi084.stdout: "min_size": 2, 2024-09-20T22:54:56.313 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_rule": 0, 2024-09-20T22:54:56.314 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_count": 0, 2024-09-20T22:54:56.314 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_target": 0, 2024-09-20T22:54:56.314 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_barrier": 0, 2024-09-20T22:54:56.314 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-20T22:54:56.314 INFO:teuthology.orchestra.run.smithi084.stdout: "is_stretch_pool": false, 2024-09-20T22:54:56.314 INFO:teuthology.orchestra.run.smithi084.stdout: "object_hash": 2, 2024-09-20T22:54:56.314 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_autoscale_mode": "off", 2024-09-20T22:54:56.314 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num": 11, 2024-09-20T22:54:56.314 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num": 11, 2024-09-20T22:54:56.314 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num_target": 11, 2024-09-20T22:54:56.314 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_target": 11, 2024-09-20T22:54:56.314 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_pending": 11, 2024-09-20T22:54:56.314 INFO:teuthology.orchestra.run.smithi084.stdout: "last_pg_merge_meta": { 2024-09-20T22:54:56.315 INFO:teuthology.orchestra.run.smithi084.stdout: "source_pgid": "1.b", 2024-09-20T22:54:56.315 INFO:teuthology.orchestra.run.smithi084.stdout: "ready_epoch": 299, 2024-09-20T22:54:56.315 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_started": 298, 2024-09-20T22:54:56.315 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_clean": 298, 2024-09-20T22:54:56.315 INFO:teuthology.orchestra.run.smithi084.stdout: "source_version": "77'128", 2024-09-20T22:54:56.315 INFO:teuthology.orchestra.run.smithi084.stdout: "target_version": "77'128" 2024-09-20T22:54:56.315 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:56.315 INFO:teuthology.orchestra.run.smithi084.stdout: "last_change": "769", 2024-09-20T22:54:56.315 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend": "0", 2024-09-20T22:54:56.315 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_prenautilus": "299", 2024-09-20T22:54:56.315 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_preluminous": "297", 2024-09-20T22:54:56.315 INFO:teuthology.orchestra.run.smithi084.stdout: "auid": 0, 2024-09-20T22:54:56.315 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_mode": "selfmanaged", 2024-09-20T22:54:56.315 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_seq": 0, 2024-09-20T22:54:56.316 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_epoch": 0, 2024-09-20T22:54:56.316 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_snaps": [], 2024-09-20T22:54:56.316 INFO:teuthology.orchestra.run.smithi084.stdout: "removed_snaps": "[]", 2024-09-20T22:54:56.316 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_bytes": 0, 2024-09-20T22:54:56.316 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_objects": 0, 2024-09-20T22:54:56.316 INFO:teuthology.orchestra.run.smithi084.stdout: "tiers": [], 2024-09-20T22:54:56.316 INFO:teuthology.orchestra.run.smithi084.stdout: "tier_of": -1, 2024-09-20T22:54:56.316 INFO:teuthology.orchestra.run.smithi084.stdout: "read_tier": -1, 2024-09-20T22:54:56.316 INFO:teuthology.orchestra.run.smithi084.stdout: "write_tier": -1, 2024-09-20T22:54:56.316 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_mode": "none", 2024-09-20T22:54:56.316 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_bytes": 0, 2024-09-20T22:54:56.316 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_objects": 0, 2024-09-20T22:54:56.316 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-20T22:54:56.316 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-20T22:54:56.317 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-20T22:54:56.317 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_flush_age": 0, 2024-09-20T22:54:56.317 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_evict_age": 0, 2024-09-20T22:54:56.317 INFO:teuthology.orchestra.run.smithi084.stdout: "erasure_code_profile": "", 2024-09-20T22:54:56.317 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_params": { 2024-09-20T22:54:56.317 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "none" 2024-09-20T22:54:56.317 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:56.317 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_period": 0, 2024-09-20T22:54:56.317 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_count": 0, 2024-09-20T22:54:56.317 INFO:teuthology.orchestra.run.smithi084.stdout: "use_gmt_hitset": true, 2024-09-20T22:54:56.317 INFO:teuthology.orchestra.run.smithi084.stdout: "min_read_recency_for_promote": 0, 2024-09-20T22:54:56.317 INFO:teuthology.orchestra.run.smithi084.stdout: "min_write_recency_for_promote": 0, 2024-09-20T22:54:56.317 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_grade_decay_rate": 0, 2024-09-20T22:54:56.318 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_search_last_n": 0, 2024-09-20T22:54:56.318 INFO:teuthology.orchestra.run.smithi084.stdout: "grade_table": [], 2024-09-20T22:54:56.318 INFO:teuthology.orchestra.run.smithi084.stdout: "stripe_width": 0, 2024-09-20T22:54:56.318 INFO:teuthology.orchestra.run.smithi084.stdout: "expected_num_objects": 0, 2024-09-20T22:54:56.318 INFO:teuthology.orchestra.run.smithi084.stdout: "fast_read": false, 2024-09-20T22:54:56.318 INFO:teuthology.orchestra.run.smithi084.stdout: "options": { 2024-09-20T22:54:56.318 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_max": 32, 2024-09-20T22:54:56.318 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_min": 1 2024-09-20T22:54:56.318 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:56.318 INFO:teuthology.orchestra.run.smithi084.stdout: "application_metadata": { 2024-09-20T22:54:56.318 INFO:teuthology.orchestra.run.smithi084.stdout: "mgr": {} 2024-09-20T22:54:56.318 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:56.318 INFO:teuthology.orchestra.run.smithi084.stdout: "read_balance": { 2024-09-20T22:54:56.318 INFO:teuthology.orchestra.run.smithi084.stdout: "score_type": "Fair distribution", 2024-09-20T22:54:56.318 INFO:teuthology.orchestra.run.smithi084.stdout: "score_acting": 2.1800000667572021, 2024-09-20T22:54:56.319 INFO:teuthology.orchestra.run.smithi084.stdout: "score_stable": 2.1800000667572021, 2024-09-20T22:54:56.319 INFO:teuthology.orchestra.run.smithi084.stdout: "optimal_score": 1, 2024-09-20T22:54:56.319 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_acting": 2.1800000667572021, 2024-09-20T22:54:56.319 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_stable": 2.1800000667572021, 2024-09-20T22:54:56.319 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity_weighted": 1, 2024-09-20T22:54:56.319 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity": 1, 2024-09-20T22:54:56.319 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity_weighted": 1 2024-09-20T22:54:56.319 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:56.319 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:56.319 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:56.319 INFO:teuthology.orchestra.run.smithi084.stdout: "pool": 2, 2024-09-20T22:54:56.319 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_name": "datapool", 2024-09-20T22:54:56.319 INFO:teuthology.orchestra.run.smithi084.stdout: "create_time": "2024-09-20T22:41:35.820081+0000", 2024-09-20T22:54:56.319 INFO:teuthology.orchestra.run.smithi084.stdout: "flags": 8193, 2024-09-20T22:54:56.320 INFO:teuthology.orchestra.run.smithi084.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-20T22:54:56.320 INFO:teuthology.orchestra.run.smithi084.stdout: "type": 1, 2024-09-20T22:54:56.320 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 3, 2024-09-20T22:54:56.320 INFO:teuthology.orchestra.run.smithi084.stdout: "min_size": 2, 2024-09-20T22:54:56.320 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_rule": 0, 2024-09-20T22:54:56.320 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_count": 0, 2024-09-20T22:54:56.320 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_target": 0, 2024-09-20T22:54:56.320 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_barrier": 0, 2024-09-20T22:54:56.320 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-20T22:54:56.320 INFO:teuthology.orchestra.run.smithi084.stdout: "is_stretch_pool": false, 2024-09-20T22:54:56.320 INFO:teuthology.orchestra.run.smithi084.stdout: "object_hash": 2, 2024-09-20T22:54:56.320 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_autoscale_mode": "off", 2024-09-20T22:54:56.320 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num": 3, 2024-09-20T22:54:56.320 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num": 3, 2024-09-20T22:54:56.320 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num_target": 3, 2024-09-20T22:54:56.320 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_target": 3, 2024-09-20T22:54:56.321 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_pending": 3, 2024-09-20T22:54:56.321 INFO:teuthology.orchestra.run.smithi084.stdout: "last_pg_merge_meta": { 2024-09-20T22:54:56.321 INFO:teuthology.orchestra.run.smithi084.stdout: "source_pgid": "0.0", 2024-09-20T22:54:56.321 INFO:teuthology.orchestra.run.smithi084.stdout: "ready_epoch": 0, 2024-09-20T22:54:56.321 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_started": 0, 2024-09-20T22:54:56.321 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_clean": 0, 2024-09-20T22:54:56.321 INFO:teuthology.orchestra.run.smithi084.stdout: "source_version": "0'0", 2024-09-20T22:54:56.321 INFO:teuthology.orchestra.run.smithi084.stdout: "target_version": "0'0" 2024-09-20T22:54:56.321 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:56.321 INFO:teuthology.orchestra.run.smithi084.stdout: "last_change": "69", 2024-09-20T22:54:56.321 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend": "0", 2024-09-20T22:54:56.321 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-20T22:54:56.321 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_preluminous": "0", 2024-09-20T22:54:56.321 INFO:teuthology.orchestra.run.smithi084.stdout: "auid": 0, 2024-09-20T22:54:56.321 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_mode": "selfmanaged", 2024-09-20T22:54:56.321 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_seq": 2, 2024-09-20T22:54:56.322 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_epoch": 69, 2024-09-20T22:54:56.322 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_snaps": [], 2024-09-20T22:54:56.322 INFO:teuthology.orchestra.run.smithi084.stdout: "removed_snaps": "[]", 2024-09-20T22:54:56.322 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_bytes": 0, 2024-09-20T22:54:56.322 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_objects": 0, 2024-09-20T22:54:56.322 INFO:teuthology.orchestra.run.smithi084.stdout: "tiers": [], 2024-09-20T22:54:56.322 INFO:teuthology.orchestra.run.smithi084.stdout: "tier_of": -1, 2024-09-20T22:54:56.322 INFO:teuthology.orchestra.run.smithi084.stdout: "read_tier": -1, 2024-09-20T22:54:56.322 INFO:teuthology.orchestra.run.smithi084.stdout: "write_tier": -1, 2024-09-20T22:54:56.322 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_mode": "none", 2024-09-20T22:54:56.322 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_bytes": 0, 2024-09-20T22:54:56.322 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_objects": 0, 2024-09-20T22:54:56.322 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-20T22:54:56.322 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-20T22:54:56.322 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-20T22:54:56.323 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_flush_age": 0, 2024-09-20T22:54:56.323 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_evict_age": 0, 2024-09-20T22:54:56.323 INFO:teuthology.orchestra.run.smithi084.stdout: "erasure_code_profile": "", 2024-09-20T22:54:56.323 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_params": { 2024-09-20T22:54:56.323 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "none" 2024-09-20T22:54:56.323 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:56.323 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_period": 0, 2024-09-20T22:54:56.323 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_count": 0, 2024-09-20T22:54:56.323 INFO:teuthology.orchestra.run.smithi084.stdout: "use_gmt_hitset": true, 2024-09-20T22:54:56.323 INFO:teuthology.orchestra.run.smithi084.stdout: "min_read_recency_for_promote": 0, 2024-09-20T22:54:56.323 INFO:teuthology.orchestra.run.smithi084.stdout: "min_write_recency_for_promote": 0, 2024-09-20T22:54:56.323 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_grade_decay_rate": 0, 2024-09-20T22:54:56.324 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_search_last_n": 0, 2024-09-20T22:54:56.324 INFO:teuthology.orchestra.run.smithi084.stdout: "grade_table": [], 2024-09-20T22:54:56.324 INFO:teuthology.orchestra.run.smithi084.stdout: "stripe_width": 0, 2024-09-20T22:54:56.324 INFO:teuthology.orchestra.run.smithi084.stdout: "expected_num_objects": 0, 2024-09-20T22:54:56.324 INFO:teuthology.orchestra.run.smithi084.stdout: "fast_read": false, 2024-09-20T22:54:56.324 INFO:teuthology.orchestra.run.smithi084.stdout: "options": {}, 2024-09-20T22:54:56.324 INFO:teuthology.orchestra.run.smithi084.stdout: "application_metadata": { 2024-09-20T22:54:56.324 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd": {} 2024-09-20T22:54:56.324 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:56.324 INFO:teuthology.orchestra.run.smithi084.stdout: "read_balance": { 2024-09-20T22:54:56.324 INFO:teuthology.orchestra.run.smithi084.stdout: "score_type": "Fair distribution", 2024-09-20T22:54:56.324 INFO:teuthology.orchestra.run.smithi084.stdout: "score_acting": 2.6500000953674316, 2024-09-20T22:54:56.324 INFO:teuthology.orchestra.run.smithi084.stdout: "score_stable": 2.6500000953674316, 2024-09-20T22:54:56.324 INFO:teuthology.orchestra.run.smithi084.stdout: "optimal_score": 0.87999999523162842, 2024-09-20T22:54:56.325 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_acting": 2.3299999237060547, 2024-09-20T22:54:56.325 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_stable": 2.3299999237060547, 2024-09-20T22:54:56.325 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity_weighted": 1, 2024-09-20T22:54:56.325 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity": 1, 2024-09-20T22:54:56.325 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity_weighted": 1 2024-09-20T22:54:56.325 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:56.325 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:56.325 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:56.325 INFO:teuthology.orchestra.run.smithi084.stdout: "pool": 3, 2024-09-20T22:54:56.325 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_name": ".rgw.root", 2024-09-20T22:54:56.325 INFO:teuthology.orchestra.run.smithi084.stdout: "create_time": "2024-09-20T22:41:36.546389+0000", 2024-09-20T22:54:56.325 INFO:teuthology.orchestra.run.smithi084.stdout: "flags": 1, 2024-09-20T22:54:56.325 INFO:teuthology.orchestra.run.smithi084.stdout: "flags_names": "hashpspool", 2024-09-20T22:54:56.325 INFO:teuthology.orchestra.run.smithi084.stdout: "type": 1, 2024-09-20T22:54:56.325 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 3, 2024-09-20T22:54:56.326 INFO:teuthology.orchestra.run.smithi084.stdout: "min_size": 2, 2024-09-20T22:54:56.326 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_rule": 0, 2024-09-20T22:54:56.326 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_count": 0, 2024-09-20T22:54:56.326 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_target": 0, 2024-09-20T22:54:56.326 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_barrier": 0, 2024-09-20T22:54:56.326 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-20T22:54:56.326 INFO:teuthology.orchestra.run.smithi084.stdout: "is_stretch_pool": false, 2024-09-20T22:54:56.326 INFO:teuthology.orchestra.run.smithi084.stdout: "object_hash": 2, 2024-09-20T22:54:56.326 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_autoscale_mode": "off", 2024-09-20T22:54:56.326 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num": 32, 2024-09-20T22:54:56.326 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num": 32, 2024-09-20T22:54:56.326 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num_target": 32, 2024-09-20T22:54:56.326 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_target": 32, 2024-09-20T22:54:56.326 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_pending": 32, 2024-09-20T22:54:56.326 INFO:teuthology.orchestra.run.smithi084.stdout: "last_pg_merge_meta": { 2024-09-20T22:54:56.327 INFO:teuthology.orchestra.run.smithi084.stdout: "source_pgid": "0.0", 2024-09-20T22:54:56.327 INFO:teuthology.orchestra.run.smithi084.stdout: "ready_epoch": 0, 2024-09-20T22:54:56.327 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_started": 0, 2024-09-20T22:54:56.327 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_clean": 0, 2024-09-20T22:54:56.327 INFO:teuthology.orchestra.run.smithi084.stdout: "source_version": "0'0", 2024-09-20T22:54:56.327 INFO:teuthology.orchestra.run.smithi084.stdout: "target_version": "0'0" 2024-09-20T22:54:56.328 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:56.328 INFO:teuthology.orchestra.run.smithi084.stdout: "last_change": "68", 2024-09-20T22:54:56.328 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend": "0", 2024-09-20T22:54:56.328 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-20T22:54:56.328 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_preluminous": "0", 2024-09-20T22:54:56.328 INFO:teuthology.orchestra.run.smithi084.stdout: "auid": 0, 2024-09-20T22:54:56.328 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_mode": "selfmanaged", 2024-09-20T22:54:56.328 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_seq": 0, 2024-09-20T22:54:56.328 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_epoch": 0, 2024-09-20T22:54:56.328 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_snaps": [], 2024-09-20T22:54:56.328 INFO:teuthology.orchestra.run.smithi084.stdout: "removed_snaps": "[]", 2024-09-20T22:54:56.328 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_bytes": 0, 2024-09-20T22:54:56.328 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_objects": 0, 2024-09-20T22:54:56.329 INFO:teuthology.orchestra.run.smithi084.stdout: "tiers": [], 2024-09-20T22:54:56.329 INFO:teuthology.orchestra.run.smithi084.stdout: "tier_of": -1, 2024-09-20T22:54:56.329 INFO:teuthology.orchestra.run.smithi084.stdout: "read_tier": -1, 2024-09-20T22:54:56.329 INFO:teuthology.orchestra.run.smithi084.stdout: "write_tier": -1, 2024-09-20T22:54:56.329 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_mode": "none", 2024-09-20T22:54:56.329 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_bytes": 0, 2024-09-20T22:54:56.329 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_objects": 0, 2024-09-20T22:54:56.329 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-20T22:54:56.329 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-20T22:54:56.329 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-20T22:54:56.329 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_flush_age": 0, 2024-09-20T22:54:56.329 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_evict_age": 0, 2024-09-20T22:54:56.329 INFO:teuthology.orchestra.run.smithi084.stdout: "erasure_code_profile": "", 2024-09-20T22:54:56.329 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_params": { 2024-09-20T22:54:56.330 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "none" 2024-09-20T22:54:56.330 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:56.330 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_period": 0, 2024-09-20T22:54:56.330 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_count": 0, 2024-09-20T22:54:56.330 INFO:teuthology.orchestra.run.smithi084.stdout: "use_gmt_hitset": true, 2024-09-20T22:54:56.330 INFO:teuthology.orchestra.run.smithi084.stdout: "min_read_recency_for_promote": 0, 2024-09-20T22:54:56.330 INFO:teuthology.orchestra.run.smithi084.stdout: "min_write_recency_for_promote": 0, 2024-09-20T22:54:56.330 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_grade_decay_rate": 0, 2024-09-20T22:54:56.330 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_search_last_n": 0, 2024-09-20T22:54:56.330 INFO:teuthology.orchestra.run.smithi084.stdout: "grade_table": [], 2024-09-20T22:54:56.330 INFO:teuthology.orchestra.run.smithi084.stdout: "stripe_width": 0, 2024-09-20T22:54:56.330 INFO:teuthology.orchestra.run.smithi084.stdout: "expected_num_objects": 0, 2024-09-20T22:54:56.330 INFO:teuthology.orchestra.run.smithi084.stdout: "fast_read": false, 2024-09-20T22:54:56.330 INFO:teuthology.orchestra.run.smithi084.stdout: "options": {}, 2024-09-20T22:54:56.331 INFO:teuthology.orchestra.run.smithi084.stdout: "application_metadata": { 2024-09-20T22:54:56.331 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw": {} 2024-09-20T22:54:56.331 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:56.331 INFO:teuthology.orchestra.run.smithi084.stdout: "read_balance": { 2024-09-20T22:54:56.331 INFO:teuthology.orchestra.run.smithi084.stdout: "score_type": "Fair distribution", 2024-09-20T22:54:56.331 INFO:teuthology.orchestra.run.smithi084.stdout: "score_acting": 1.5, 2024-09-20T22:54:56.331 INFO:teuthology.orchestra.run.smithi084.stdout: "score_stable": 1.5, 2024-09-20T22:54:56.331 INFO:teuthology.orchestra.run.smithi084.stdout: "optimal_score": 1, 2024-09-20T22:54:56.331 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_acting": 1.5, 2024-09-20T22:54:56.331 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_stable": 1.5, 2024-09-20T22:54:56.332 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity_weighted": 1, 2024-09-20T22:54:56.332 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity": 1, 2024-09-20T22:54:56.332 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity_weighted": 1 2024-09-20T22:54:56.332 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:56.332 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:56.332 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:56.332 INFO:teuthology.orchestra.run.smithi084.stdout: "pool": 4, 2024-09-20T22:54:56.332 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_name": "default.rgw.log", 2024-09-20T22:54:56.332 INFO:teuthology.orchestra.run.smithi084.stdout: "create_time": "2024-09-20T22:41:38.454640+0000", 2024-09-20T22:54:56.332 INFO:teuthology.orchestra.run.smithi084.stdout: "flags": 1, 2024-09-20T22:54:56.332 INFO:teuthology.orchestra.run.smithi084.stdout: "flags_names": "hashpspool", 2024-09-20T22:54:56.332 INFO:teuthology.orchestra.run.smithi084.stdout: "type": 1, 2024-09-20T22:54:56.332 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 3, 2024-09-20T22:54:56.332 INFO:teuthology.orchestra.run.smithi084.stdout: "min_size": 2, 2024-09-20T22:54:56.333 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_rule": 0, 2024-09-20T22:54:56.333 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_count": 0, 2024-09-20T22:54:56.333 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_target": 0, 2024-09-20T22:54:56.333 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_barrier": 0, 2024-09-20T22:54:56.333 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-20T22:54:56.333 INFO:teuthology.orchestra.run.smithi084.stdout: "is_stretch_pool": false, 2024-09-20T22:54:56.333 INFO:teuthology.orchestra.run.smithi084.stdout: "object_hash": 2, 2024-09-20T22:54:56.333 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_autoscale_mode": "off", 2024-09-20T22:54:56.333 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num": 32, 2024-09-20T22:54:56.333 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num": 32, 2024-09-20T22:54:56.333 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num_target": 32, 2024-09-20T22:54:56.333 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_target": 32, 2024-09-20T22:54:56.334 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_pending": 32, 2024-09-20T22:54:56.334 INFO:teuthology.orchestra.run.smithi084.stdout: "last_pg_merge_meta": { 2024-09-20T22:54:56.334 INFO:teuthology.orchestra.run.smithi084.stdout: "source_pgid": "0.0", 2024-09-20T22:54:56.334 INFO:teuthology.orchestra.run.smithi084.stdout: "ready_epoch": 0, 2024-09-20T22:54:56.334 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_started": 0, 2024-09-20T22:54:56.334 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_clean": 0, 2024-09-20T22:54:56.334 INFO:teuthology.orchestra.run.smithi084.stdout: "source_version": "0'0", 2024-09-20T22:54:56.334 INFO:teuthology.orchestra.run.smithi084.stdout: "target_version": "0'0" 2024-09-20T22:54:56.334 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:56.334 INFO:teuthology.orchestra.run.smithi084.stdout: "last_change": "70", 2024-09-20T22:54:56.334 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend": "0", 2024-09-20T22:54:56.334 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-20T22:54:56.334 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_preluminous": "0", 2024-09-20T22:54:56.335 INFO:teuthology.orchestra.run.smithi084.stdout: "auid": 0, 2024-09-20T22:54:56.335 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_mode": "selfmanaged", 2024-09-20T22:54:56.335 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_seq": 0, 2024-09-20T22:54:56.335 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_epoch": 0, 2024-09-20T22:54:56.335 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_snaps": [], 2024-09-20T22:54:56.335 INFO:teuthology.orchestra.run.smithi084.stdout: "removed_snaps": "[]", 2024-09-20T22:54:56.335 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_bytes": 0, 2024-09-20T22:54:56.335 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_objects": 0, 2024-09-20T22:54:56.335 INFO:teuthology.orchestra.run.smithi084.stdout: "tiers": [], 2024-09-20T22:54:56.335 INFO:teuthology.orchestra.run.smithi084.stdout: "tier_of": -1, 2024-09-20T22:54:56.335 INFO:teuthology.orchestra.run.smithi084.stdout: "read_tier": -1, 2024-09-20T22:54:56.335 INFO:teuthology.orchestra.run.smithi084.stdout: "write_tier": -1, 2024-09-20T22:54:56.335 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_mode": "none", 2024-09-20T22:54:56.336 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_bytes": 0, 2024-09-20T22:54:56.336 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_objects": 0, 2024-09-20T22:54:56.336 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-20T22:54:56.336 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-20T22:54:56.336 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-20T22:54:56.336 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_flush_age": 0, 2024-09-20T22:54:56.336 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_evict_age": 0, 2024-09-20T22:54:56.336 INFO:teuthology.orchestra.run.smithi084.stdout: "erasure_code_profile": "", 2024-09-20T22:54:56.336 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_params": { 2024-09-20T22:54:56.336 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "none" 2024-09-20T22:54:56.336 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:56.336 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_period": 0, 2024-09-20T22:54:56.336 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_count": 0, 2024-09-20T22:54:56.336 INFO:teuthology.orchestra.run.smithi084.stdout: "use_gmt_hitset": true, 2024-09-20T22:54:56.336 INFO:teuthology.orchestra.run.smithi084.stdout: "min_read_recency_for_promote": 0, 2024-09-20T22:54:56.337 INFO:teuthology.orchestra.run.smithi084.stdout: "min_write_recency_for_promote": 0, 2024-09-20T22:54:56.337 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_grade_decay_rate": 0, 2024-09-20T22:54:56.337 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_search_last_n": 0, 2024-09-20T22:54:56.337 INFO:teuthology.orchestra.run.smithi084.stdout: "grade_table": [], 2024-09-20T22:54:56.337 INFO:teuthology.orchestra.run.smithi084.stdout: "stripe_width": 0, 2024-09-20T22:54:56.337 INFO:teuthology.orchestra.run.smithi084.stdout: "expected_num_objects": 0, 2024-09-20T22:54:56.337 INFO:teuthology.orchestra.run.smithi084.stdout: "fast_read": false, 2024-09-20T22:54:56.337 INFO:teuthology.orchestra.run.smithi084.stdout: "options": {}, 2024-09-20T22:54:56.337 INFO:teuthology.orchestra.run.smithi084.stdout: "application_metadata": { 2024-09-20T22:54:56.337 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw": {} 2024-09-20T22:54:56.337 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:56.337 INFO:teuthology.orchestra.run.smithi084.stdout: "read_balance": { 2024-09-20T22:54:56.337 INFO:teuthology.orchestra.run.smithi084.stdout: "score_type": "Fair distribution", 2024-09-20T22:54:56.337 INFO:teuthology.orchestra.run.smithi084.stdout: "score_acting": 2.25, 2024-09-20T22:54:56.338 INFO:teuthology.orchestra.run.smithi084.stdout: "score_stable": 2.25, 2024-09-20T22:54:56.338 INFO:teuthology.orchestra.run.smithi084.stdout: "optimal_score": 1, 2024-09-20T22:54:56.338 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_acting": 2.25, 2024-09-20T22:54:56.338 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_stable": 2.25, 2024-09-20T22:54:56.338 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity_weighted": 1, 2024-09-20T22:54:56.338 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity": 1, 2024-09-20T22:54:56.338 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity_weighted": 1 2024-09-20T22:54:56.338 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:56.338 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:56.338 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:56.338 INFO:teuthology.orchestra.run.smithi084.stdout: "pool": 5, 2024-09-20T22:54:56.338 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_name": "default.rgw.control", 2024-09-20T22:54:56.338 INFO:teuthology.orchestra.run.smithi084.stdout: "create_time": "2024-09-20T22:41:40.407129+0000", 2024-09-20T22:54:56.338 INFO:teuthology.orchestra.run.smithi084.stdout: "flags": 1, 2024-09-20T22:54:56.338 INFO:teuthology.orchestra.run.smithi084.stdout: "flags_names": "hashpspool", 2024-09-20T22:54:56.339 INFO:teuthology.orchestra.run.smithi084.stdout: "type": 1, 2024-09-20T22:54:56.339 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 3, 2024-09-20T22:54:56.339 INFO:teuthology.orchestra.run.smithi084.stdout: "min_size": 2, 2024-09-20T22:54:56.339 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_rule": 0, 2024-09-20T22:54:56.339 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_count": 0, 2024-09-20T22:54:56.339 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_target": 0, 2024-09-20T22:54:56.339 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_barrier": 0, 2024-09-20T22:54:56.339 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-20T22:54:56.339 INFO:teuthology.orchestra.run.smithi084.stdout: "is_stretch_pool": false, 2024-09-20T22:54:56.339 INFO:teuthology.orchestra.run.smithi084.stdout: "object_hash": 2, 2024-09-20T22:54:56.339 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_autoscale_mode": "off", 2024-09-20T22:54:56.339 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num": 32, 2024-09-20T22:54:56.339 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num": 32, 2024-09-20T22:54:56.339 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num_target": 32, 2024-09-20T22:54:56.339 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_target": 32, 2024-09-20T22:54:56.340 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_pending": 32, 2024-09-20T22:54:56.340 INFO:teuthology.orchestra.run.smithi084.stdout: "last_pg_merge_meta": { 2024-09-20T22:54:56.340 INFO:teuthology.orchestra.run.smithi084.stdout: "source_pgid": "0.0", 2024-09-20T22:54:56.340 INFO:teuthology.orchestra.run.smithi084.stdout: "ready_epoch": 0, 2024-09-20T22:54:56.340 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_started": 0, 2024-09-20T22:54:56.340 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_clean": 0, 2024-09-20T22:54:56.340 INFO:teuthology.orchestra.run.smithi084.stdout: "source_version": "0'0", 2024-09-20T22:54:56.340 INFO:teuthology.orchestra.run.smithi084.stdout: "target_version": "0'0" 2024-09-20T22:54:56.340 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:56.340 INFO:teuthology.orchestra.run.smithi084.stdout: "last_change": "72", 2024-09-20T22:54:56.340 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend": "0", 2024-09-20T22:54:56.340 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-20T22:54:56.340 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_preluminous": "0", 2024-09-20T22:54:56.340 INFO:teuthology.orchestra.run.smithi084.stdout: "auid": 0, 2024-09-20T22:54:56.341 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_mode": "selfmanaged", 2024-09-20T22:54:56.341 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_seq": 0, 2024-09-20T22:54:56.341 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_epoch": 0, 2024-09-20T22:54:56.341 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_snaps": [], 2024-09-20T22:54:56.341 INFO:teuthology.orchestra.run.smithi084.stdout: "removed_snaps": "[]", 2024-09-20T22:54:56.341 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_bytes": 0, 2024-09-20T22:54:56.341 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_objects": 0, 2024-09-20T22:54:56.341 INFO:teuthology.orchestra.run.smithi084.stdout: "tiers": [], 2024-09-20T22:54:56.341 INFO:teuthology.orchestra.run.smithi084.stdout: "tier_of": -1, 2024-09-20T22:54:56.341 INFO:teuthology.orchestra.run.smithi084.stdout: "read_tier": -1, 2024-09-20T22:54:56.341 INFO:teuthology.orchestra.run.smithi084.stdout: "write_tier": -1, 2024-09-20T22:54:56.341 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_mode": "none", 2024-09-20T22:54:56.341 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_bytes": 0, 2024-09-20T22:54:56.341 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_objects": 0, 2024-09-20T22:54:56.342 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-20T22:54:56.342 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-20T22:54:56.342 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-20T22:54:56.342 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_flush_age": 0, 2024-09-20T22:54:56.342 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_evict_age": 0, 2024-09-20T22:54:56.342 INFO:teuthology.orchestra.run.smithi084.stdout: "erasure_code_profile": "", 2024-09-20T22:54:56.342 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_params": { 2024-09-20T22:54:56.342 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "none" 2024-09-20T22:54:56.342 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:56.342 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_period": 0, 2024-09-20T22:54:56.342 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_count": 0, 2024-09-20T22:54:56.342 INFO:teuthology.orchestra.run.smithi084.stdout: "use_gmt_hitset": true, 2024-09-20T22:54:56.342 INFO:teuthology.orchestra.run.smithi084.stdout: "min_read_recency_for_promote": 0, 2024-09-20T22:54:56.343 INFO:teuthology.orchestra.run.smithi084.stdout: "min_write_recency_for_promote": 0, 2024-09-20T22:54:56.343 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_grade_decay_rate": 0, 2024-09-20T22:54:56.343 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_search_last_n": 0, 2024-09-20T22:54:56.343 INFO:teuthology.orchestra.run.smithi084.stdout: "grade_table": [], 2024-09-20T22:54:56.343 INFO:teuthology.orchestra.run.smithi084.stdout: "stripe_width": 0, 2024-09-20T22:54:56.343 INFO:teuthology.orchestra.run.smithi084.stdout: "expected_num_objects": 0, 2024-09-20T22:54:56.343 INFO:teuthology.orchestra.run.smithi084.stdout: "fast_read": false, 2024-09-20T22:54:56.343 INFO:teuthology.orchestra.run.smithi084.stdout: "options": {}, 2024-09-20T22:54:56.343 INFO:teuthology.orchestra.run.smithi084.stdout: "application_metadata": { 2024-09-20T22:54:56.343 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw": {} 2024-09-20T22:54:56.343 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:56.343 INFO:teuthology.orchestra.run.smithi084.stdout: "read_balance": { 2024-09-20T22:54:56.343 INFO:teuthology.orchestra.run.smithi084.stdout: "score_type": "Fair distribution", 2024-09-20T22:54:56.343 INFO:teuthology.orchestra.run.smithi084.stdout: "score_acting": 1.25, 2024-09-20T22:54:56.344 INFO:teuthology.orchestra.run.smithi084.stdout: "score_stable": 1.25, 2024-09-20T22:54:56.344 INFO:teuthology.orchestra.run.smithi084.stdout: "optimal_score": 1, 2024-09-20T22:54:56.344 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_acting": 1.25, 2024-09-20T22:54:56.344 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_stable": 1.25, 2024-09-20T22:54:56.344 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity_weighted": 1, 2024-09-20T22:54:56.344 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity": 1, 2024-09-20T22:54:56.344 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity_weighted": 1 2024-09-20T22:54:56.344 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:56.344 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:56.344 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:56.344 INFO:teuthology.orchestra.run.smithi084.stdout: "pool": 6, 2024-09-20T22:54:56.344 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_name": "default.rgw.meta", 2024-09-20T22:54:56.344 INFO:teuthology.orchestra.run.smithi084.stdout: "create_time": "2024-09-20T22:41:42.476716+0000", 2024-09-20T22:54:56.344 INFO:teuthology.orchestra.run.smithi084.stdout: "flags": 1, 2024-09-20T22:54:56.344 INFO:teuthology.orchestra.run.smithi084.stdout: "flags_names": "hashpspool", 2024-09-20T22:54:56.345 INFO:teuthology.orchestra.run.smithi084.stdout: "type": 1, 2024-09-20T22:54:56.345 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 3, 2024-09-20T22:54:56.345 INFO:teuthology.orchestra.run.smithi084.stdout: "min_size": 2, 2024-09-20T22:54:56.345 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_rule": 0, 2024-09-20T22:54:56.345 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_count": 0, 2024-09-20T22:54:56.345 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_target": 0, 2024-09-20T22:54:56.345 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_barrier": 0, 2024-09-20T22:54:56.345 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-20T22:54:56.345 INFO:teuthology.orchestra.run.smithi084.stdout: "is_stretch_pool": false, 2024-09-20T22:54:56.345 INFO:teuthology.orchestra.run.smithi084.stdout: "object_hash": 2, 2024-09-20T22:54:56.345 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_autoscale_mode": "off", 2024-09-20T22:54:56.345 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num": 32, 2024-09-20T22:54:56.345 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num": 32, 2024-09-20T22:54:56.345 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num_target": 32, 2024-09-20T22:54:56.345 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_target": 32, 2024-09-20T22:54:56.345 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_pending": 32, 2024-09-20T22:54:56.346 INFO:teuthology.orchestra.run.smithi084.stdout: "last_pg_merge_meta": { 2024-09-20T22:54:56.346 INFO:teuthology.orchestra.run.smithi084.stdout: "source_pgid": "0.0", 2024-09-20T22:54:56.346 INFO:teuthology.orchestra.run.smithi084.stdout: "ready_epoch": 0, 2024-09-20T22:54:56.346 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_started": 0, 2024-09-20T22:54:56.346 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_clean": 0, 2024-09-20T22:54:56.346 INFO:teuthology.orchestra.run.smithi084.stdout: "source_version": "0'0", 2024-09-20T22:54:56.346 INFO:teuthology.orchestra.run.smithi084.stdout: "target_version": "0'0" 2024-09-20T22:54:56.346 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:56.346 INFO:teuthology.orchestra.run.smithi084.stdout: "last_change": "74", 2024-09-20T22:54:56.346 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend": "0", 2024-09-20T22:54:56.346 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-20T22:54:56.346 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_preluminous": "0", 2024-09-20T22:54:56.346 INFO:teuthology.orchestra.run.smithi084.stdout: "auid": 0, 2024-09-20T22:54:56.346 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_mode": "selfmanaged", 2024-09-20T22:54:56.346 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_seq": 0, 2024-09-20T22:54:56.347 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_epoch": 0, 2024-09-20T22:54:56.347 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_snaps": [], 2024-09-20T22:54:56.347 INFO:teuthology.orchestra.run.smithi084.stdout: "removed_snaps": "[]", 2024-09-20T22:54:56.347 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_bytes": 0, 2024-09-20T22:54:56.347 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_objects": 0, 2024-09-20T22:54:56.347 INFO:teuthology.orchestra.run.smithi084.stdout: "tiers": [], 2024-09-20T22:54:56.347 INFO:teuthology.orchestra.run.smithi084.stdout: "tier_of": -1, 2024-09-20T22:54:56.347 INFO:teuthology.orchestra.run.smithi084.stdout: "read_tier": -1, 2024-09-20T22:54:56.347 INFO:teuthology.orchestra.run.smithi084.stdout: "write_tier": -1, 2024-09-20T22:54:56.347 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_mode": "none", 2024-09-20T22:54:56.347 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_bytes": 0, 2024-09-20T22:54:56.347 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_objects": 0, 2024-09-20T22:54:56.347 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-20T22:54:56.347 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-20T22:54:56.348 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-20T22:54:56.348 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_flush_age": 0, 2024-09-20T22:54:56.348 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_evict_age": 0, 2024-09-20T22:54:56.348 INFO:teuthology.orchestra.run.smithi084.stdout: "erasure_code_profile": "", 2024-09-20T22:54:56.348 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_params": { 2024-09-20T22:54:56.348 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "none" 2024-09-20T22:54:56.348 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:56.348 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_period": 0, 2024-09-20T22:54:56.348 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_count": 0, 2024-09-20T22:54:56.348 INFO:teuthology.orchestra.run.smithi084.stdout: "use_gmt_hitset": true, 2024-09-20T22:54:56.348 INFO:teuthology.orchestra.run.smithi084.stdout: "min_read_recency_for_promote": 0, 2024-09-20T22:54:56.348 INFO:teuthology.orchestra.run.smithi084.stdout: "min_write_recency_for_promote": 0, 2024-09-20T22:54:56.348 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_grade_decay_rate": 0, 2024-09-20T22:54:56.348 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_search_last_n": 0, 2024-09-20T22:54:56.348 INFO:teuthology.orchestra.run.smithi084.stdout: "grade_table": [], 2024-09-20T22:54:56.349 INFO:teuthology.orchestra.run.smithi084.stdout: "stripe_width": 0, 2024-09-20T22:54:56.349 INFO:teuthology.orchestra.run.smithi084.stdout: "expected_num_objects": 0, 2024-09-20T22:54:56.349 INFO:teuthology.orchestra.run.smithi084.stdout: "fast_read": false, 2024-09-20T22:54:56.349 INFO:teuthology.orchestra.run.smithi084.stdout: "options": { 2024-09-20T22:54:56.349 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_autoscale_bias": 4 2024-09-20T22:54:56.349 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:56.349 INFO:teuthology.orchestra.run.smithi084.stdout: "application_metadata": { 2024-09-20T22:54:56.349 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw": {} 2024-09-20T22:54:56.349 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:56.349 INFO:teuthology.orchestra.run.smithi084.stdout: "read_balance": { 2024-09-20T22:54:56.349 INFO:teuthology.orchestra.run.smithi084.stdout: "score_type": "Fair distribution", 2024-09-20T22:54:56.350 INFO:teuthology.orchestra.run.smithi084.stdout: "score_acting": 1.75, 2024-09-20T22:54:56.350 INFO:teuthology.orchestra.run.smithi084.stdout: "score_stable": 1.75, 2024-09-20T22:54:56.350 INFO:teuthology.orchestra.run.smithi084.stdout: "optimal_score": 1, 2024-09-20T22:54:56.350 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_acting": 1.75, 2024-09-20T22:54:56.350 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_stable": 1.75, 2024-09-20T22:54:56.350 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity_weighted": 1, 2024-09-20T22:54:56.350 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity": 1, 2024-09-20T22:54:56.350 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity_weighted": 1 2024-09-20T22:54:56.350 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:56.350 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:56.350 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:56.350 INFO:teuthology.orchestra.run.smithi084.stdout: "pool": 7, 2024-09-20T22:54:56.351 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_name": "unique_pool_0", 2024-09-20T22:54:56.351 INFO:teuthology.orchestra.run.smithi084.stdout: "create_time": "2024-09-20T22:42:51.354358+0000", 2024-09-20T22:54:56.351 INFO:teuthology.orchestra.run.smithi084.stdout: "flags": 8193, 2024-09-20T22:54:56.351 INFO:teuthology.orchestra.run.smithi084.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-20T22:54:56.351 INFO:teuthology.orchestra.run.smithi084.stdout: "type": 1, 2024-09-20T22:54:56.351 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 3, 2024-09-20T22:54:56.351 INFO:teuthology.orchestra.run.smithi084.stdout: "min_size": 2, 2024-09-20T22:54:56.351 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_rule": 0, 2024-09-20T22:54:56.351 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_count": 0, 2024-09-20T22:54:56.351 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_target": 0, 2024-09-20T22:54:56.351 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_barrier": 0, 2024-09-20T22:54:56.351 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-20T22:54:56.351 INFO:teuthology.orchestra.run.smithi084.stdout: "is_stretch_pool": false, 2024-09-20T22:54:56.352 INFO:teuthology.orchestra.run.smithi084.stdout: "object_hash": 2, 2024-09-20T22:54:56.352 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_autoscale_mode": "off", 2024-09-20T22:54:56.352 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num": 16, 2024-09-20T22:54:56.352 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num": 16, 2024-09-20T22:54:56.352 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num_target": 16, 2024-09-20T22:54:56.352 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_target": 16, 2024-09-20T22:54:56.352 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_pending": 16, 2024-09-20T22:54:56.352 INFO:teuthology.orchestra.run.smithi084.stdout: "last_pg_merge_meta": { 2024-09-20T22:54:56.352 INFO:teuthology.orchestra.run.smithi084.stdout: "source_pgid": "7.10", 2024-09-20T22:54:56.352 INFO:teuthology.orchestra.run.smithi084.stdout: "ready_epoch": 484, 2024-09-20T22:54:56.352 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_started": 483, 2024-09-20T22:54:56.352 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_clean": 483, 2024-09-20T22:54:56.352 INFO:teuthology.orchestra.run.smithi084.stdout: "source_version": "480'172", 2024-09-20T22:54:56.353 INFO:teuthology.orchestra.run.smithi084.stdout: "target_version": "480'253" 2024-09-20T22:54:56.353 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:56.353 INFO:teuthology.orchestra.run.smithi084.stdout: "last_change": "815", 2024-09-20T22:54:56.353 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend": "0", 2024-09-20T22:54:56.353 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_prenautilus": "484", 2024-09-20T22:54:56.353 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_preluminous": "482", 2024-09-20T22:54:56.353 INFO:teuthology.orchestra.run.smithi084.stdout: "auid": 0, 2024-09-20T22:54:56.353 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_mode": "selfmanaged", 2024-09-20T22:54:56.353 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_seq": 373, 2024-09-20T22:54:56.353 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_epoch": 815, 2024-09-20T22:54:56.353 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_snaps": [], 2024-09-20T22:54:56.354 INFO:teuthology.orchestra.run.smithi084.stdout: "removed_snaps": "[]", 2024-09-20T22:54:56.354 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_bytes": 0, 2024-09-20T22:54:56.354 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_objects": 0, 2024-09-20T22:54:56.354 INFO:teuthology.orchestra.run.smithi084.stdout: "tiers": [], 2024-09-20T22:54:56.354 INFO:teuthology.orchestra.run.smithi084.stdout: "tier_of": -1, 2024-09-20T22:54:56.354 INFO:teuthology.orchestra.run.smithi084.stdout: "read_tier": -1, 2024-09-20T22:54:56.354 INFO:teuthology.orchestra.run.smithi084.stdout: "write_tier": -1, 2024-09-20T22:54:56.354 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_mode": "none", 2024-09-20T22:54:56.354 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_bytes": 0, 2024-09-20T22:54:56.354 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_objects": 0, 2024-09-20T22:54:56.354 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-20T22:54:56.354 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-20T22:54:56.354 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-20T22:54:56.355 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_flush_age": 0, 2024-09-20T22:54:56.355 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_evict_age": 0, 2024-09-20T22:54:56.355 INFO:teuthology.orchestra.run.smithi084.stdout: "erasure_code_profile": "", 2024-09-20T22:54:56.355 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_params": { 2024-09-20T22:54:56.355 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "none" 2024-09-20T22:54:56.355 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:56.355 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_period": 0, 2024-09-20T22:54:56.355 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_count": 0, 2024-09-20T22:54:56.355 INFO:teuthology.orchestra.run.smithi084.stdout: "use_gmt_hitset": true, 2024-09-20T22:54:56.355 INFO:teuthology.orchestra.run.smithi084.stdout: "min_read_recency_for_promote": 0, 2024-09-20T22:54:56.355 INFO:teuthology.orchestra.run.smithi084.stdout: "min_write_recency_for_promote": 0, 2024-09-20T22:54:56.355 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_grade_decay_rate": 0, 2024-09-20T22:54:56.355 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_search_last_n": 0, 2024-09-20T22:54:56.355 INFO:teuthology.orchestra.run.smithi084.stdout: "grade_table": [], 2024-09-20T22:54:56.356 INFO:teuthology.orchestra.run.smithi084.stdout: "stripe_width": 0, 2024-09-20T22:54:56.356 INFO:teuthology.orchestra.run.smithi084.stdout: "expected_num_objects": 0, 2024-09-20T22:54:56.356 INFO:teuthology.orchestra.run.smithi084.stdout: "fast_read": false, 2024-09-20T22:54:56.356 INFO:teuthology.orchestra.run.smithi084.stdout: "options": {}, 2024-09-20T22:54:56.356 INFO:teuthology.orchestra.run.smithi084.stdout: "application_metadata": { 2024-09-20T22:54:56.356 INFO:teuthology.orchestra.run.smithi084.stdout: "rados": {} 2024-09-20T22:54:56.356 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:56.356 INFO:teuthology.orchestra.run.smithi084.stdout: "read_balance": { 2024-09-20T22:54:56.356 INFO:teuthology.orchestra.run.smithi084.stdout: "score_type": "Fair distribution", 2024-09-20T22:54:56.356 INFO:teuthology.orchestra.run.smithi084.stdout: "score_acting": 2, 2024-09-20T22:54:56.356 INFO:teuthology.orchestra.run.smithi084.stdout: "score_stable": 2, 2024-09-20T22:54:56.356 INFO:teuthology.orchestra.run.smithi084.stdout: "optimal_score": 1, 2024-09-20T22:54:56.356 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_acting": 2, 2024-09-20T22:54:56.357 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_stable": 2, 2024-09-20T22:54:56.357 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity_weighted": 1, 2024-09-20T22:54:56.357 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity": 1, 2024-09-20T22:54:56.357 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity_weighted": 1 2024-09-20T22:54:56.357 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:56.357 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:56.357 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2024-09-20T22:54:56.357 INFO:teuthology.orchestra.run.smithi084.stdout: "osds": [ 2024-09-20T22:54:56.357 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:56.357 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 0, 2024-09-20T22:54:56.357 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "ea36a489-6f91-4b2f-b784-b90506b9805d", 2024-09-20T22:54:56.357 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-09-20T22:54:56.358 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-09-20T22:54:56.358 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-09-20T22:54:56.358 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-09-20T22:54:56.358 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-09-20T22:54:56.358 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-09-20T22:54:56.358 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 8, 2024-09-20T22:54:56.358 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 779, 2024-09-20T22:54:56.358 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-09-20T22:54:56.358 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-09-20T22:54:56.358 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-09-20T22:54:56.358 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:54:56.358 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:56.358 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:54:56.358 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6801", 2024-09-20T22:54:56.359 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 268084626 2024-09-20T22:54:56.359 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:56.359 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:56.359 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:56.359 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-09-20T22:54:56.359 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:54:56.359 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:56.359 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:54:56.359 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6802", 2024-09-20T22:54:56.359 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 268084626 2024-09-20T22:54:56.359 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:56.359 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:56.359 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:56.359 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-09-20T22:54:56.360 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:54:56.360 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:56.360 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:54:56.360 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6804", 2024-09-20T22:54:56.360 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 268084626 2024-09-20T22:54:56.360 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:56.360 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:56.360 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:56.360 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-09-20T22:54:56.360 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:54:56.360 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:56.360 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:54:56.360 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6803", 2024-09-20T22:54:56.360 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 268084626 2024-09-20T22:54:56.361 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:56.361 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:56.361 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:56.361 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.84:6801/268084626", 2024-09-20T22:54:56.361 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.84:6802/268084626", 2024-09-20T22:54:56.361 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.84:6804/268084626", 2024-09-20T22:54:56.361 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.84:6803/268084626", 2024-09-20T22:54:56.361 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-09-20T22:54:56.361 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-09-20T22:54:56.361 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-09-20T22:54:56.361 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:56.361 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:56.361 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:56.361 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 1, 2024-09-20T22:54:56.361 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "c8649d2f-27ef-47f8-bc94-f1cd9e62bdb8", 2024-09-20T22:54:56.361 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-09-20T22:54:56.362 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-09-20T22:54:56.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:56.362 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:56.363 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:54:56.363 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-09-20T22:54:56.363 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-09-20T22:54:56.363 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-09-20T22:54:56.363 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-09-20T22:54:56.363 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 13, 2024-09-20T22:54:56.364 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 779, 2024-09-20T22:54:56.364 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-09-20T22:54:56.364 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-09-20T22:54:56.364 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-09-20T22:54:56.364 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:54:56.364 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:56.364 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:54:56.364 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6805", 2024-09-20T22:54:56.364 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 418374136 2024-09-20T22:54:56.364 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:56.364 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:56.364 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:56.364 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-09-20T22:54:56.365 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:54:56.365 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:56.365 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:54:56.365 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6806", 2024-09-20T22:54:56.365 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 418374136 2024-09-20T22:54:56.365 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:56.365 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:56.365 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:56.365 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-09-20T22:54:56.365 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:54:56.365 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:56.365 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:54:56.365 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6808", 2024-09-20T22:54:56.365 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 418374136 2024-09-20T22:54:56.365 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:56.366 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:56.366 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:56.366 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-09-20T22:54:56.366 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:54:56.366 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:56.366 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:54:56.366 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6807", 2024-09-20T22:54:56.366 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 418374136 2024-09-20T22:54:56.366 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:56.366 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:56.366 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:56.366 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.84:6805/418374136", 2024-09-20T22:54:56.366 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.84:6806/418374136", 2024-09-20T22:54:56.366 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.84:6808/418374136", 2024-09-20T22:54:56.367 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.84:6807/418374136", 2024-09-20T22:54:56.367 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-09-20T22:54:56.367 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-09-20T22:54:56.367 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-09-20T22:54:56.367 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:56.367 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:56.367 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:56.367 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 2, 2024-09-20T22:54:56.367 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "c728eb48-a0f0-4de4-a651-7f3bec867280", 2024-09-20T22:54:56.367 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-09-20T22:54:56.367 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-09-20T22:54:56.367 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-09-20T22:54:56.367 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-09-20T22:54:56.368 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-09-20T22:54:56.368 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-09-20T22:54:56.368 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 18, 2024-09-20T22:54:56.368 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 778, 2024-09-20T22:54:56.368 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-09-20T22:54:56.368 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-09-20T22:54:56.368 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-09-20T22:54:56.368 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:54:56.368 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:56.368 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:54:56.368 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6809", 2024-09-20T22:54:56.368 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3977168611 2024-09-20T22:54:56.368 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:56.368 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:56.369 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:56.369 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-09-20T22:54:56.369 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:54:56.369 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:56.369 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:54:56.369 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6810", 2024-09-20T22:54:56.369 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3977168611 2024-09-20T22:54:56.369 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:56.369 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:56.369 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:56.369 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-09-20T22:54:56.369 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:54:56.369 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:56.369 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:54:56.370 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6812", 2024-09-20T22:54:56.370 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3977168611 2024-09-20T22:54:56.370 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:56.370 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:56.370 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:56.370 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-09-20T22:54:56.370 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:54:56.370 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:56.370 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:54:56.370 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6811", 2024-09-20T22:54:56.370 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3977168611 2024-09-20T22:54:56.370 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:56.370 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:56.370 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:56.371 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.84:6809/3977168611", 2024-09-20T22:54:56.371 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.84:6810/3977168611", 2024-09-20T22:54:56.371 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.84:6812/3977168611", 2024-09-20T22:54:56.371 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.84:6811/3977168611", 2024-09-20T22:54:56.371 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-09-20T22:54:56.371 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-09-20T22:54:56.371 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-09-20T22:54:56.371 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:56.371 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:56.371 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:56.371 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 3, 2024-09-20T22:54:56.371 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "573f44eb-0853-4630-a7d8-9caaeb256f9b", 2024-09-20T22:54:56.371 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-09-20T22:54:56.372 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-09-20T22:54:56.372 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-09-20T22:54:56.372 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-09-20T22:54:56.372 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-09-20T22:54:56.372 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-09-20T22:54:56.372 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 25, 2024-09-20T22:54:56.372 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 809, 2024-09-20T22:54:56.372 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-09-20T22:54:56.372 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-09-20T22:54:56.372 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-09-20T22:54:56.372 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:54:56.372 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:56.372 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:54:56.372 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6813", 2024-09-20T22:54:56.373 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2192239945 2024-09-20T22:54:56.373 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:56.373 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:56.373 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:56.373 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-09-20T22:54:56.373 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:54:56.373 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:56.373 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:54:56.373 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6814", 2024-09-20T22:54:56.373 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2192239945 2024-09-20T22:54:56.373 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:56.373 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:56.373 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:56.373 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-09-20T22:54:56.374 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:54:56.374 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:56.374 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:54:56.374 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6816", 2024-09-20T22:54:56.374 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2192239945 2024-09-20T22:54:56.374 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:56.374 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:56.374 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:56.374 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-09-20T22:54:56.374 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:54:56.375 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:56.375 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:54:56.375 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6815", 2024-09-20T22:54:56.375 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2192239945 2024-09-20T22:54:56.375 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:56.375 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:56.375 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:56.375 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.84:6813/2192239945", 2024-09-20T22:54:56.375 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.84:6814/2192239945", 2024-09-20T22:54:56.375 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.84:6816/2192239945", 2024-09-20T22:54:56.375 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.84:6815/2192239945", 2024-09-20T22:54:56.375 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-09-20T22:54:56.376 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-09-20T22:54:56.376 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-09-20T22:54:56.376 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:56.376 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:56.376 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:56.376 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 4, 2024-09-20T22:54:56.376 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "a52c3b94-c5f3-490f-8ac8-09061b9b2abb", 2024-09-20T22:54:56.376 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-09-20T22:54:56.376 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-09-20T22:54:56.376 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-09-20T22:54:56.376 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-09-20T22:54:56.376 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-09-20T22:54:56.376 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-09-20T22:54:56.376 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 31, 2024-09-20T22:54:56.377 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 808, 2024-09-20T22:54:56.377 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-09-20T22:54:56.377 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-09-20T22:54:56.377 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-09-20T22:54:56.377 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:54:56.377 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:56.377 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:54:56.377 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.154:6800", 2024-09-20T22:54:56.377 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1793136002 2024-09-20T22:54:56.377 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:56.377 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:56.377 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:56.377 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-09-20T22:54:56.377 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:54:56.377 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:56.378 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:54:56.378 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.154:6801", 2024-09-20T22:54:56.378 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1793136002 2024-09-20T22:54:56.378 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:56.378 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:56.378 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:56.378 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-09-20T22:54:56.378 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:54:56.378 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:56.378 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:54:56.378 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.154:6803", 2024-09-20T22:54:56.378 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1793136002 2024-09-20T22:54:56.378 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:56.378 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:56.379 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:56.379 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-09-20T22:54:56.379 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:54:56.379 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:56.379 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:54:56.379 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.154:6802", 2024-09-20T22:54:56.379 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1793136002 2024-09-20T22:54:56.379 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:56.379 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:56.379 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:56.379 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.154:6800/1793136002", 2024-09-20T22:54:56.379 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.154:6801/1793136002", 2024-09-20T22:54:56.379 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.154:6803/1793136002", 2024-09-20T22:54:56.379 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.154:6802/1793136002", 2024-09-20T22:54:56.379 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-09-20T22:54:56.379 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-09-20T22:54:56.380 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-09-20T22:54:56.380 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:56.380 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:56.380 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:56.380 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 5, 2024-09-20T22:54:56.380 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "4fa0dc8e-25cc-46cd-96ff-33f20b6faec3", 2024-09-20T22:54:56.380 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-09-20T22:54:56.380 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-09-20T22:54:56.380 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-09-20T22:54:56.380 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-09-20T22:54:56.380 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-09-20T22:54:56.380 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-09-20T22:54:56.380 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 40, 2024-09-20T22:54:56.380 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 779, 2024-09-20T22:54:56.380 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-09-20T22:54:56.381 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-09-20T22:54:56.381 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-09-20T22:54:56.381 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:54:56.381 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:56.381 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:54:56.381 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.154:6804", 2024-09-20T22:54:56.381 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 493899927 2024-09-20T22:54:56.381 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:56.381 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:56.381 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:56.381 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-09-20T22:54:56.381 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:54:56.381 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:56.381 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:54:56.381 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.154:6805", 2024-09-20T22:54:56.382 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 493899927 2024-09-20T22:54:56.382 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:56.382 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:56.382 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:56.382 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-09-20T22:54:56.382 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:54:56.382 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:56.382 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:54:56.382 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.154:6807", 2024-09-20T22:54:56.382 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 493899927 2024-09-20T22:54:56.382 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:56.382 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:56.382 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:56.382 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-09-20T22:54:56.383 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:54:56.383 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:56.383 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:54:56.383 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.154:6806", 2024-09-20T22:54:56.383 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 493899927 2024-09-20T22:54:56.383 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:56.383 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:56.383 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:56.383 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.154:6804/493899927", 2024-09-20T22:54:56.383 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.154:6805/493899927", 2024-09-20T22:54:56.383 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.154:6807/493899927", 2024-09-20T22:54:56.383 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.154:6806/493899927", 2024-09-20T22:54:56.383 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-09-20T22:54:56.383 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-09-20T22:54:56.384 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-09-20T22:54:56.384 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:56.384 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:56.384 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:56.384 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 6, 2024-09-20T22:54:56.384 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "56788d74-810e-4827-a615-958c12f8a368", 2024-09-20T22:54:56.384 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-09-20T22:54:56.384 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-09-20T22:54:56.384 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-09-20T22:54:56.384 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-09-20T22:54:56.384 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-09-20T22:54:56.384 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-09-20T22:54:56.384 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 48, 2024-09-20T22:54:56.385 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 779, 2024-09-20T22:54:56.385 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-09-20T22:54:56.385 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-09-20T22:54:56.385 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-09-20T22:54:56.385 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:54:56.385 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:56.385 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:54:56.385 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.154:6808", 2024-09-20T22:54:56.385 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2375091033 2024-09-20T22:54:56.385 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:56.385 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:56.385 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:56.385 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-09-20T22:54:56.385 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:54:56.386 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:56.386 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:54:56.386 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.154:6809", 2024-09-20T22:54:56.386 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2375091033 2024-09-20T22:54:56.386 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:56.386 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:56.386 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:56.386 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-09-20T22:54:56.386 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:54:56.386 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:56.386 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:54:56.386 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.154:6811", 2024-09-20T22:54:56.386 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2375091033 2024-09-20T22:54:56.387 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:56.387 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:56.387 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:56.387 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-09-20T22:54:56.387 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:54:56.387 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:56.387 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:54:56.387 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.154:6810", 2024-09-20T22:54:56.387 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2375091033 2024-09-20T22:54:56.387 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:56.387 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:56.388 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:56.388 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.154:6808/2375091033", 2024-09-20T22:54:56.388 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.154:6809/2375091033", 2024-09-20T22:54:56.388 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.154:6811/2375091033", 2024-09-20T22:54:56.388 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.154:6810/2375091033", 2024-09-20T22:54:56.388 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-09-20T22:54:56.388 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-09-20T22:54:56.388 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-09-20T22:54:56.388 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:56.388 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:56.388 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:56.388 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 7, 2024-09-20T22:54:56.388 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "83503727-afc1-4b95-8b96-ecb63f3338c2", 2024-09-20T22:54:56.388 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-09-20T22:54:56.389 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-09-20T22:54:56.389 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-09-20T22:54:56.389 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-09-20T22:54:56.389 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-09-20T22:54:56.390 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-09-20T22:54:56.390 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 59, 2024-09-20T22:54:56.390 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 776, 2024-09-20T22:54:56.390 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-09-20T22:54:56.390 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-09-20T22:54:56.390 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-09-20T22:54:56.390 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:54:56.390 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:56.390 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:54:56.390 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.154:6812", 2024-09-20T22:54:56.390 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3717704491 2024-09-20T22:54:56.390 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:56.391 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:56.391 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:56.391 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-09-20T22:54:56.391 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:54:56.391 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:56.391 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:54:56.391 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.154:6813", 2024-09-20T22:54:56.391 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3717704491 2024-09-20T22:54:56.391 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:56.391 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:56.391 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:56.391 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-09-20T22:54:56.391 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:54:56.392 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:56.392 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:54:56.392 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.154:6815", 2024-09-20T22:54:56.392 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3717704491 2024-09-20T22:54:56.392 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:56.392 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:56.392 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:56.392 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-09-20T22:54:56.392 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:54:56.392 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:56.392 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:54:56.392 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.154:6814", 2024-09-20T22:54:56.392 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3717704491 2024-09-20T22:54:56.393 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:56.393 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:56.393 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:56.393 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.154:6812/3717704491", 2024-09-20T22:54:56.393 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.154:6813/3717704491", 2024-09-20T22:54:56.393 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.154:6815/3717704491", 2024-09-20T22:54:56.393 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.154:6814/3717704491", 2024-09-20T22:54:56.393 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-09-20T22:54:56.393 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-09-20T22:54:56.393 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-09-20T22:54:56.393 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:56.393 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:56.393 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2024-09-20T22:54:56.393 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_xinfo": [ 2024-09-20T22:54:56.393 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:56.394 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 0, 2024-09-20T22:54:56.394 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-09-20T22:54:56.394 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-09-20T22:54:56.394 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-09-20T22:54:56.394 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540701547738038271, 2024-09-20T22:54:56.394 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-09-20T22:54:56.394 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-09-20T22:54:06.520163+0000", 2024-09-20T22:54:56.394 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-09-20T22:54:56.394 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:56.394 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:56.394 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 1, 2024-09-20T22:54:56.394 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-09-20T22:54:56.394 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-09-20T22:54:56.394 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-09-20T22:54:56.394 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540701547738038271, 2024-09-20T22:54:56.395 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-09-20T22:54:56.395 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-09-20T22:54:48.239990+0000", 2024-09-20T22:54:56.395 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-09-20T22:54:56.395 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:56.395 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:56.395 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 2, 2024-09-20T22:54:56.395 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-09-20T22:54:56.395 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-09-20T22:54:56.395 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-09-20T22:54:56.395 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540701547738038271, 2024-09-20T22:54:56.395 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-09-20T22:54:56.395 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-09-20T22:54:46.929979+0000", 2024-09-20T22:54:56.396 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-09-20T22:54:56.396 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:56.396 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:56.396 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 3, 2024-09-20T22:54:56.396 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-09-20T22:54:56.396 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-09-20T22:54:56.396 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-09-20T22:54:56.396 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540701547738038271, 2024-09-20T22:54:56.396 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-09-20T22:54:56.396 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-09-20T22:54:29.696694+0000", 2024-09-20T22:54:56.396 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-09-20T22:54:56.396 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:56.396 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:56.397 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 4, 2024-09-20T22:54:56.397 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-09-20T22:54:56.397 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-09-20T22:54:56.397 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-09-20T22:54:56.397 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540701547738038271, 2024-09-20T22:54:56.397 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-09-20T22:54:56.397 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-09-20T22:54:06.316620+0000", 2024-09-20T22:54:56.397 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-09-20T22:54:56.397 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:56.397 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:56.397 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 5, 2024-09-20T22:54:56.397 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-09-20T22:54:56.397 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-09-20T22:54:56.398 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-09-20T22:54:56.398 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540701547738038271, 2024-09-20T22:54:56.398 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-09-20T22:54:56.398 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-09-20T22:54:54.642519+0000", 2024-09-20T22:54:56.398 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-09-20T22:54:56.398 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:56.398 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:56.398 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 6, 2024-09-20T22:54:56.398 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-09-20T22:54:56.398 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-09-20T22:54:56.398 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-09-20T22:54:56.399 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540701547738038271, 2024-09-20T22:54:56.399 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-09-20T22:54:56.399 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-09-20T22:54:50.271376+0000", 2024-09-20T22:54:56.399 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-09-20T22:54:56.399 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:56.399 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:56.399 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 7, 2024-09-20T22:54:56.399 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-09-20T22:54:56.399 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-09-20T22:54:56.399 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-09-20T22:54:56.399 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540701547738038271, 2024-09-20T22:54:56.399 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-09-20T22:54:56.399 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-09-20T22:54:00.021655+0000", 2024-09-20T22:54:56.399 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-09-20T22:54:56.399 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:56.400 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2024-09-20T22:54:56.400 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_upmap": [ 2024-09-20T22:54:56.400 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:56.400 INFO:teuthology.orchestra.run.smithi084.stdout: "pgid": "1.9", 2024-09-20T22:54:56.400 INFO:teuthology.orchestra.run.smithi084.stdout: "osds": [ 2024-09-20T22:54:56.400 INFO:teuthology.orchestra.run.smithi084.stdout: 7, 2024-09-20T22:54:56.400 INFO:teuthology.orchestra.run.smithi084.stdout: 3, 2024-09-20T22:54:56.400 INFO:teuthology.orchestra.run.smithi084.stdout: 5 2024-09-20T22:54:56.400 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:56.400 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:56.400 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:56.400 INFO:teuthology.orchestra.run.smithi084.stdout: "pgid": "6.e", 2024-09-20T22:54:56.400 INFO:teuthology.orchestra.run.smithi084.stdout: "osds": [ 2024-09-20T22:54:56.400 INFO:teuthology.orchestra.run.smithi084.stdout: 1, 2024-09-20T22:54:56.400 INFO:teuthology.orchestra.run.smithi084.stdout: 2, 2024-09-20T22:54:56.400 INFO:teuthology.orchestra.run.smithi084.stdout: 5 2024-09-20T22:54:56.401 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:56.401 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:56.401 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2024-09-20T22:54:56.401 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_upmap_items": [ 2024-09-20T22:54:56.401 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:56.401 INFO:teuthology.orchestra.run.smithi084.stdout: "pgid": "1.0", 2024-09-20T22:54:56.401 INFO:teuthology.orchestra.run.smithi084.stdout: "mappings": [ 2024-09-20T22:54:56.401 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:56.401 INFO:teuthology.orchestra.run.smithi084.stdout: "from": 0, 2024-09-20T22:54:56.401 INFO:teuthology.orchestra.run.smithi084.stdout: "to": 7 2024-09-20T22:54:56.401 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:56.401 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:56.401 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:56.401 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:56.401 INFO:teuthology.orchestra.run.smithi084.stdout: "pgid": "3.a", 2024-09-20T22:54:56.402 INFO:teuthology.orchestra.run.smithi084.stdout: "mappings": [ 2024-09-20T22:54:56.402 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:56.402 INFO:teuthology.orchestra.run.smithi084.stdout: "from": 6, 2024-09-20T22:54:56.402 INFO:teuthology.orchestra.run.smithi084.stdout: "to": 2 2024-09-20T22:54:56.402 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:56.402 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:56.402 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:56.402 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:56.402 INFO:teuthology.orchestra.run.smithi084.stdout: "pgid": "4.9", 2024-09-20T22:54:56.402 INFO:teuthology.orchestra.run.smithi084.stdout: "mappings": [ 2024-09-20T22:54:56.402 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:56.402 INFO:teuthology.orchestra.run.smithi084.stdout: "from": 3, 2024-09-20T22:54:56.402 INFO:teuthology.orchestra.run.smithi084.stdout: "to": 2 2024-09-20T22:54:56.403 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:56.403 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:56.403 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:56.403 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:56.403 INFO:teuthology.orchestra.run.smithi084.stdout: "pgid": "4.11", 2024-09-20T22:54:56.403 INFO:teuthology.orchestra.run.smithi084.stdout: "mappings": [ 2024-09-20T22:54:56.403 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:56.403 INFO:teuthology.orchestra.run.smithi084.stdout: "from": 3, 2024-09-20T22:54:56.403 INFO:teuthology.orchestra.run.smithi084.stdout: "to": 5 2024-09-20T22:54:56.403 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:56.403 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:56.403 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:56.403 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2024-09-20T22:54:56.403 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_upmap_primaries": [], 2024-09-20T22:54:56.404 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_temp": [], 2024-09-20T22:54:56.404 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_temp": [], 2024-09-20T22:54:56.404 INFO:teuthology.orchestra.run.smithi084.stdout: "blocklist": { 2024-09-20T22:54:56.404 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/542020545": "2024-09-21T22:42:07.596045+0000", 2024-09-20T22:54:56.404 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/25873236": "2024-09-21T22:36:54.917812+0000", 2024-09-20T22:54:56.404 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:6800/820908470": "2024-09-21T22:36:54.917812+0000", 2024-09-20T22:54:56.404 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/567614908": "2024-09-21T22:36:54.917812+0000", 2024-09-20T22:54:56.404 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/2963622459": "2024-09-21T22:42:07.596045+0000", 2024-09-20T22:54:56.404 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/1176849276": "2024-09-21T22:37:17.690104+0000", 2024-09-20T22:54:56.404 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:6800/2932420050": "2024-09-21T22:37:17.690104+0000", 2024-09-20T22:54:56.404 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/3879263635": "2024-09-21T22:37:17.690104+0000", 2024-09-20T22:54:56.404 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/969472279": "2024-09-21T22:36:54.917812+0000", 2024-09-20T22:54:56.404 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/1361143214": "2024-09-21T22:37:17.690104+0000", 2024-09-20T22:54:56.404 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/3353126955": "2024-09-21T22:42:07.596045+0000", 2024-09-20T22:54:56.405 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:6800/4024278029": "2024-09-21T22:42:07.596045+0000", 2024-09-20T22:54:56.405 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/547902262": "2024-09-21T22:42:07.596045+0000", 2024-09-20T22:54:56.405 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/1242929317": "2024-09-21T22:42:07.596045+0000" 2024-09-20T22:54:56.405 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:56.405 INFO:teuthology.orchestra.run.smithi084.stdout: "range_blocklist": {}, 2024-09-20T22:54:56.405 INFO:teuthology.orchestra.run.smithi084.stdout: "erasure_code_profiles": { 2024-09-20T22:54:56.405 INFO:teuthology.orchestra.run.smithi084.stdout: "default": { 2024-09-20T22:54:56.405 INFO:teuthology.orchestra.run.smithi084.stdout: "crush-failure-domain": "osd", 2024-09-20T22:54:56.405 INFO:teuthology.orchestra.run.smithi084.stdout: "k": "2", 2024-09-20T22:54:56.405 INFO:teuthology.orchestra.run.smithi084.stdout: "m": "1", 2024-09-20T22:54:56.405 INFO:teuthology.orchestra.run.smithi084.stdout: "plugin": "jerasure", 2024-09-20T22:54:56.405 INFO:teuthology.orchestra.run.smithi084.stdout: "technique": "reed_sol_van" 2024-09-20T22:54:56.405 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:56.405 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:56.405 INFO:teuthology.orchestra.run.smithi084.stdout: "removed_snaps_queue": [ 2024-09-20T22:54:56.406 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:56.406 INFO:teuthology.orchestra.run.smithi084.stdout: "pool": 7, 2024-09-20T22:54:56.406 INFO:teuthology.orchestra.run.smithi084.stdout: "snaps": [ 2024-09-20T22:54:56.406 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:56.406 INFO:teuthology.orchestra.run.smithi084.stdout: "begin": 274, 2024-09-20T22:54:56.406 INFO:teuthology.orchestra.run.smithi084.stdout: "length": 1 2024-09-20T22:54:56.406 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:56.406 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:56.406 INFO:teuthology.orchestra.run.smithi084.stdout: "begin": 287, 2024-09-20T22:54:56.406 INFO:teuthology.orchestra.run.smithi084.stdout: "length": 5 2024-09-20T22:54:56.406 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:56.406 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:56.406 INFO:teuthology.orchestra.run.smithi084.stdout: "begin": 293, 2024-09-20T22:54:56.406 INFO:teuthology.orchestra.run.smithi084.stdout: "length": 9 2024-09-20T22:54:56.407 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:56.407 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:56.407 INFO:teuthology.orchestra.run.smithi084.stdout: "begin": 304, 2024-09-20T22:54:56.407 INFO:teuthology.orchestra.run.smithi084.stdout: "length": 9 2024-09-20T22:54:56.407 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:56.407 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:56.407 INFO:teuthology.orchestra.run.smithi084.stdout: "begin": 315, 2024-09-20T22:54:56.407 INFO:teuthology.orchestra.run.smithi084.stdout: "length": 3 2024-09-20T22:54:56.407 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:56.407 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:56.407 INFO:teuthology.orchestra.run.smithi084.stdout: "begin": 319, 2024-09-20T22:54:56.407 INFO:teuthology.orchestra.run.smithi084.stdout: "length": 6 2024-09-20T22:54:56.407 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:56.408 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:56.408 INFO:teuthology.orchestra.run.smithi084.stdout: "begin": 326, 2024-09-20T22:54:56.408 INFO:teuthology.orchestra.run.smithi084.stdout: "length": 3 2024-09-20T22:54:56.408 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:56.408 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:56.408 INFO:teuthology.orchestra.run.smithi084.stdout: "begin": 331, 2024-09-20T22:54:56.408 INFO:teuthology.orchestra.run.smithi084.stdout: "length": 3 2024-09-20T22:54:56.408 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:56.408 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:56.408 INFO:teuthology.orchestra.run.smithi084.stdout: "begin": 336, 2024-09-20T22:54:56.408 INFO:teuthology.orchestra.run.smithi084.stdout: "length": 3 2024-09-20T22:54:56.408 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:56.408 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:56.408 INFO:teuthology.orchestra.run.smithi084.stdout: "begin": 340, 2024-09-20T22:54:56.408 INFO:teuthology.orchestra.run.smithi084.stdout: "length": 1 2024-09-20T22:54:56.409 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:56.409 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:56.409 INFO:teuthology.orchestra.run.smithi084.stdout: "begin": 345, 2024-09-20T22:54:56.409 INFO:teuthology.orchestra.run.smithi084.stdout: "length": 1 2024-09-20T22:54:56.409 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:56.409 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:56.409 INFO:teuthology.orchestra.run.smithi084.stdout: "begin": 347, 2024-09-20T22:54:56.409 INFO:teuthology.orchestra.run.smithi084.stdout: "length": 1 2024-09-20T22:54:56.409 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:56.409 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:56.409 INFO:teuthology.orchestra.run.smithi084.stdout: "begin": 349, 2024-09-20T22:54:56.409 INFO:teuthology.orchestra.run.smithi084.stdout: "length": 1 2024-09-20T22:54:56.409 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:56.409 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:56.410 INFO:teuthology.orchestra.run.smithi084.stdout: "begin": 353, 2024-09-20T22:54:56.410 INFO:teuthology.orchestra.run.smithi084.stdout: "length": 4 2024-09-20T22:54:56.410 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:56.410 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:56.410 INFO:teuthology.orchestra.run.smithi084.stdout: "begin": 360, 2024-09-20T22:54:56.410 INFO:teuthology.orchestra.run.smithi084.stdout: "length": 3 2024-09-20T22:54:56.410 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:54:56.410 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:56.410 INFO:teuthology.orchestra.run.smithi084.stdout: "begin": 368, 2024-09-20T22:54:56.410 INFO:teuthology.orchestra.run.smithi084.stdout: "length": 1 2024-09-20T22:54:56.410 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:56.410 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:56.410 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:56.410 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2024-09-20T22:54:56.411 INFO:teuthology.orchestra.run.smithi084.stdout: "new_removed_snaps": [ 2024-09-20T22:54:56.411 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:56.411 INFO:teuthology.orchestra.run.smithi084.stdout: "pool": 7, 2024-09-20T22:54:56.411 INFO:teuthology.orchestra.run.smithi084.stdout: "snaps": [ 2024-09-20T22:54:56.411 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:54:56.411 INFO:teuthology.orchestra.run.smithi084.stdout: "begin": 322, 2024-09-20T22:54:56.411 INFO:teuthology.orchestra.run.smithi084.stdout: "length": 1 2024-09-20T22:54:56.411 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:56.411 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:54:56.411 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:56.411 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2024-09-20T22:54:56.411 INFO:teuthology.orchestra.run.smithi084.stdout: "new_purged_snaps": [], 2024-09-20T22:54:56.411 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_node_flags": {}, 2024-09-20T22:54:56.411 INFO:teuthology.orchestra.run.smithi084.stdout: "device_class_flags": {}, 2024-09-20T22:54:56.411 INFO:teuthology.orchestra.run.smithi084.stdout: "stretch_mode": { 2024-09-20T22:54:56.412 INFO:teuthology.orchestra.run.smithi084.stdout: "stretch_mode_enabled": false, 2024-09-20T22:54:56.412 INFO:teuthology.orchestra.run.smithi084.stdout: "stretch_bucket_count": 0, 2024-09-20T22:54:56.412 INFO:teuthology.orchestra.run.smithi084.stdout: "degraded_stretch_mode": 0, 2024-09-20T22:54:56.412 INFO:teuthology.orchestra.run.smithi084.stdout: "recovering_stretch_mode": 0, 2024-09-20T22:54:56.412 INFO:teuthology.orchestra.run.smithi084.stdout: "stretch_mode_bucket": 0 2024-09-20T22:54:56.412 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:54:56.412 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-20T22:54:56.463 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:54:56.484 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:56.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:56.485 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:54:56.500 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is set 2024-09-20T22:54:56.507 INFO:tasks.rados.rados.0.smithi084.stdout:3237: done (0 left) 2024-09-20T22:54:56.507 INFO:tasks.rados.rados.0.smithi084.stdout:3238: delete oid 3 current snap is 372 2024-09-20T22:54:56.515 INFO:tasks.rados.rados.0.smithi084.stdout:3238: done (0 left) 2024-09-20T22:54:56.515 INFO:tasks.rados.rados.0.smithi084.stdout:3239: read oid 10 snap -1 2024-09-20T22:54:56.515 INFO:tasks.rados.rados.0.smithi084.stdout:3239: expect deleted 2024-09-20T22:54:56.515 INFO:tasks.rados.rados.0.smithi084.stdout:3240: read oid 43 snap -1 2024-09-20T22:54:56.515 INFO:tasks.rados.rados.0.smithi084.stdout:3240: expect (ObjNum 1019 snap 333 seq_num 1019) 2024-09-20T22:54:56.515 INFO:tasks.rados.rados.0.smithi084.stdout:3241: write (excl) oid 19 current snap is 372 2024-09-20T22:54:56.515 INFO:tasks.rados.rados.0.smithi084.stdout:3241: seq_num 1100 ranges {487700=728732,1786075=665712} 2024-09-20T22:54:56.521 INFO:tasks.rados.rados.0.smithi084.stdout:3241: writing smithi08477203-19 from 487700 to 1216432 tid 1 2024-09-20T22:54:56.528 INFO:tasks.rados.rados.0.smithi084.stdout:3241: writing smithi08477203-19 from 1786075 to 2451787 tid 2 2024-09-20T22:54:56.528 INFO:tasks.rados.rados.0.smithi084.stdout:3242: write oid 31 current snap is 372 2024-09-20T22:54:56.528 INFO:tasks.rados.rados.0.smithi084.stdout:3242: seq_num 1101 ranges {504507=403981,1328204=409370,2252362=252145} 2024-09-20T22:54:56.533 INFO:tasks.rados.rados.0.smithi084.stdout:3242: writing smithi08477203-31 from 504507 to 908488 tid 1 2024-09-20T22:54:56.538 INFO:tasks.rados.rados.0.smithi084.stdout:3242: writing smithi08477203-31 from 1328204 to 1737574 tid 2 2024-09-20T22:54:56.543 INFO:tasks.rados.rados.0.smithi084.stdout:3242: writing smithi08477203-31 from 2252362 to 2504507 tid 3 2024-09-20T22:54:56.543 INFO:tasks.rados.rados.0.smithi084.stdout:3243: rollback oid 12 current snap is 372 2024-09-20T22:54:56.543 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 12 to 327 2024-09-20T22:54:56.543 INFO:tasks.rados.rados.0.smithi084.stdout:3240: expect (ObjNum 1019 snap 333 seq_num 1019) 2024-09-20T22:54:56.586 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:54:56.607 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:56.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:56.608 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:54:56.614 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:56 smithi084 ceph-mon[29774]: osdmap e815: 8 total, 8 up, 8 in 2024-09-20T22:54:56.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:56 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/103349085' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:54:56.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:56 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:54:56.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:56 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:54:56.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:56 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/6414291' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-20T22:54:56.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:56 smithi084 ceph-mon[40024]: osdmap e815: 8 total, 8 up, 8 in 2024-09-20T22:54:56.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:56 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/103349085' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:54:56.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:56 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:54:56.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:56 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:54:56.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:56 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/6414291' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-20T22:54:56.644 INFO:tasks.rados.rados.0.smithi084.stdout:3241: finishing write tid 1 to smithi08477203-19 2024-09-20T22:54:56.645 INFO:tasks.rados.rados.0.smithi084.stdout:3241: finishing write tid 2 to smithi08477203-19 2024-09-20T22:54:56.645 INFO:tasks.rados.rados.0.smithi084.stdout:3239: done (4 left) 2024-09-20T22:54:56.645 INFO:tasks.rados.rados.0.smithi084.stdout:3240: done (3 left) 2024-09-20T22:54:56.645 INFO:tasks.rados.rados.0.smithi084.stdout:3244: snap_remove snap 357 2024-09-20T22:54:56.708 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:54:56.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:56.731 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:56.731 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:54:56.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:56 smithi154 ceph-mon[33264]: osdmap e815: 8 total, 8 up, 8 in 2024-09-20T22:54:56.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:56 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/103349085' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:54:56.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:56 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:54:56.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:56 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:54:56.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:56 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/6414291' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-20T22:54:56.832 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:54:56.853 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:56.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:56.854 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:54:56.955 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:54:56.977 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:56.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:56.978 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:54:57.079 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:54:57.082 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-20T22:54:57.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:57.103 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:57.103 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:54:57.205 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:54:57.228 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:57.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:57.229 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:54:57.330 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:54:57.353 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:57.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:57.354 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:54:57.455 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:54:57.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:57.478 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:57.478 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:54:57.514 INFO:tasks.rados.rados.0.smithi084.stdout:3244: done (3 left) 2024-09-20T22:54:57.514 INFO:tasks.rados.rados.0.smithi084.stdout:3245: snap_create 2024-09-20T22:54:57.514 INFO:tasks.rados.rados.0.smithi084.stdout:3241: finishing write tid 3 to smithi08477203-19 2024-09-20T22:54:57.514 INFO:tasks.rados.rados.0.smithi084.stdout:3241: finishing write tid 4 to smithi08477203-19 2024-09-20T22:54:57.514 INFO:tasks.rados.rados.0.smithi084.stdout:3241: oid 19 updating version 0 to 2609 2024-09-20T22:54:57.514 INFO:tasks.rados.rados.0.smithi084.stdout:3241: oid 19 updating version 2609 to 2610 2024-09-20T22:54:57.514 INFO:tasks.rados.rados.0.smithi084.stdout:3241: oid 19 version 2610 is already newer than 2604 2024-09-20T22:54:57.514 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 19 v 2610 (ObjNum 1100 snap 372 seq_num 1100) dirty exists 2024-09-20T22:54:57.514 INFO:tasks.rados.rados.0.smithi084.stdout:3241: left oid 19 (ObjNum 1100 snap 372 seq_num 1100) 2024-09-20T22:54:57.515 INFO:tasks.rados.rados.0.smithi084.stdout:3242: finishing write tid 1 to smithi08477203-31 2024-09-20T22:54:57.515 INFO:tasks.rados.rados.0.smithi084.stdout:3242: finishing write tid 2 to smithi08477203-31 2024-09-20T22:54:57.515 INFO:tasks.rados.rados.0.smithi084.stdout:3242: finishing write tid 3 to smithi08477203-31 2024-09-20T22:54:57.515 INFO:tasks.rados.rados.0.smithi084.stdout:3242: finishing write tid 4 to smithi08477203-31 2024-09-20T22:54:57.515 INFO:tasks.rados.rados.0.smithi084.stdout:3242: finishing write tid 5 to smithi08477203-31 2024-09-20T22:54:57.515 INFO:tasks.rados.rados.0.smithi084.stdout:3242: oid 31 updating version 0 to 2457 2024-09-20T22:54:57.515 INFO:tasks.rados.rados.0.smithi084.stdout:3242: oid 31 version 2457 is already newer than 2455 2024-09-20T22:54:57.515 INFO:tasks.rados.rados.0.smithi084.stdout:3242: oid 31 version 2457 is already newer than 2456 2024-09-20T22:54:57.515 INFO:tasks.rados.rados.0.smithi084.stdout:3242: oid 31 updating version 2457 to 2458 2024-09-20T22:54:57.515 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 31 v 2458 (ObjNum 1101 snap 372 seq_num 1101) dirty exists 2024-09-20T22:54:57.515 INFO:tasks.rados.rados.0.smithi084.stdout:3242: left oid 31 (ObjNum 1101 snap 372 seq_num 1101) 2024-09-20T22:54:57.515 INFO:tasks.rados.rados.0.smithi084.stdout:3243: finishing rollback tid 0 to smithi08477203-12 2024-09-20T22:54:57.515 INFO:tasks.rados.rados.0.smithi084.stdout:3243: finishing rollback tid 1 to smithi08477203-12 2024-09-20T22:54:57.515 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 12 v 2460 (ObjNum 113 snap 0 seq_num 2091793136) dirty dne 2024-09-20T22:54:57.579 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:54:57.600 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:57.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:57.601 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:54:57.702 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:54:57.723 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:57.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:57.724 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:54:57.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:57 smithi154 ceph-mon[33264]: pgmap v1125: 158 pgs: 3 active+clean+snaptrim, 155 active+clean; 531 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 15 MiB/s rd, 4.5 MiB/s wr, 25 op/s 2024-09-20T22:54:57.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:57 smithi154 ceph-mon[33264]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:54:57.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:57 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/103349085' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T22:54:57.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:57 smithi154 ceph-mon[33264]: osdmap e816: 8 total, 8 up, 8 in 2024-09-20T22:54:57.807 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:57 smithi084 ceph-mon[40024]: pgmap v1125: 158 pgs: 3 active+clean+snaptrim, 155 active+clean; 531 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 15 MiB/s rd, 4.5 MiB/s wr, 25 op/s 2024-09-20T22:54:57.807 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:57 smithi084 ceph-mon[40024]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:54:57.807 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:57 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/103349085' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T22:54:57.807 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:57 smithi084 ceph-mon[40024]: osdmap e816: 8 total, 8 up, 8 in 2024-09-20T22:54:57.807 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:57 smithi084 ceph-mon[29774]: pgmap v1125: 158 pgs: 3 active+clean+snaptrim, 155 active+clean; 531 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 15 MiB/s rd, 4.5 MiB/s wr, 25 op/s 2024-09-20T22:54:57.807 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:57 smithi084 ceph-mon[29774]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:54:57.807 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:57 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/103349085' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T22:54:57.808 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:57 smithi084 ceph-mon[29774]: osdmap e816: 8 total, 8 up, 8 in 2024-09-20T22:54:57.825 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:54:57.846 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:57.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:57.847 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:54:57.947 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:54:57.969 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:57.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:57.970 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:54:58.071 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:54:58.092 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:58.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:58.093 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:54:58.194 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:54:58.215 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:58.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:58.215 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:54:58.217 INFO:tasks.thrashosds.thrasher:j is {'epoch': 815, 'fsid': 'a9c0bc70-77a0-11ef-baf5-efdc52797490', 'created': '2024-09-20T22:36:30.240626+0000', 'modified': '2024-09-20T22:54:55.485925+0000', 'last_up_change': '2024-09-20T22:41:27.098565+0000', 'last_in_change': '2024-09-20T22:54:16.306377+0000', 'flags': 'noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5801984, 'flags_set': ['noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 18, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 7, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-20T22:39:21.984381+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 11, 'pg_placement_num': 11, 'pg_placement_num_target': 11, 'pg_num_target': 11, 'pg_num_pending': 11, 'last_pg_merge_meta': {'source_pgid': '1.b', 'ready_epoch': 299, 'last_epoch_started': 298, 'last_epoch_clean': 298, 'source_version': "77'128", 'target_version': "77'128"}, 'last_change': '769', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '299', 'last_force_op_resend_preluminous': '297', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.180000066757202, 'score_stable': 2.180000066757202, 'optimal_score': 1, 'raw_score_acting': 2.180000066757202, 'raw_score_stable': 2.180000066757202, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-09-20T22:41:35.820081+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '69', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 69, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.6500000953674316, 'score_stable': 2.6500000953674316, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 2.3299999237060547, 'raw_score_stable': 2.3299999237060547, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-09-20T22:41:36.546389+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-09-20T22:41:38.454640+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '70', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-09-20T22:41:40.407129+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '72', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-09-20T22:41:42.476716+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '74', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'unique_pool_0', 'create_time': '2024-09-20T22:42:51.354358+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '7.10', 'ready_epoch': 484, 'last_epoch_started': 483, 'last_epoch_clean': 483, 'source_version': "480'172", 'target_version': "480'253"}, 'last_change': '815', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '484', 'last_force_op_resend_preluminous': '482', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 373, 'snap_epoch': 815, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': 'ea36a489-6f91-4b2f-b784-b90506b9805d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 779, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6801', 'nonce': 268084626}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6802', 'nonce': 268084626}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6804', 'nonce': 268084626}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6803', 'nonce': 268084626}]}, 'public_addr': '172.21.15.84:6801/268084626', 'cluster_addr': '172.21.15.84:6802/268084626', 'heartbeat_back_addr': '172.21.15.84:6804/268084626', 'heartbeat_front_addr': '172.21.15.84:6803/268084626', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'c8649d2f-27ef-47f8-bc94-f1cd9e62bdb8', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 779, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6805', 'nonce': 418374136}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6806', 'nonce': 418374136}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6808', 'nonce': 418374136}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6807', 'nonce': 418374136}]}, 'public_addr': '172.21.15.84:6805/418374136', 'cluster_addr': '172.21.15.84:6806/418374136', 'heartbeat_back_addr': '172.21.15.84:6808/418374136', 'heartbeat_front_addr': '172.21.15.84:6807/418374136', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'c728eb48-a0f0-4de4-a651-7f3bec867280', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 778, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6809', 'nonce': 3977168611}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6810', 'nonce': 3977168611}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6812', 'nonce': 3977168611}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6811', 'nonce': 3977168611}]}, 'public_addr': '172.21.15.84:6809/3977168611', 'cluster_addr': '172.21.15.84:6810/3977168611', 'heartbeat_back_addr': '172.21.15.84:6812/3977168611', 'heartbeat_front_addr': '172.21.15.84:6811/3977168611', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '573f44eb-0853-4630-a7d8-9caaeb256f9b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 25, 'up_thru': 809, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6813', 'nonce': 2192239945}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6814', 'nonce': 2192239945}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6816', 'nonce': 2192239945}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6815', 'nonce': 2192239945}]}, 'public_addr': '172.21.15.84:6813/2192239945', 'cluster_addr': '172.21.15.84:6814/2192239945', 'heartbeat_back_addr': '172.21.15.84:6816/2192239945', 'heartbeat_front_addr': '172.21.15.84:6815/2192239945', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'a52c3b94-c5f3-490f-8ac8-09061b9b2abb', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 808, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6800', 'nonce': 1793136002}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6801', 'nonce': 1793136002}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6803', 'nonce': 1793136002}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6802', 'nonce': 1793136002}]}, 'public_addr': '172.21.15.154:6800/1793136002', 'cluster_addr': '172.21.15.154:6801/1793136002', 'heartbeat_back_addr': '172.21.15.154:6803/1793136002', 'heartbeat_front_addr': '172.21.15.154:6802/1793136002', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '4fa0dc8e-25cc-46cd-96ff-33f20b6faec3', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 40, 'up_thru': 779, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6804', 'nonce': 493899927}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6805', 'nonce': 493899927}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6807', 'nonce': 493899927}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6806', 'nonce': 493899927}]}, 'public_addr': '172.21.15.154:6804/493899927', 'cluster_addr': '172.21.15.154:6805/493899927', 'heartbeat_back_addr': '172.21.15.154:6807/493899927', 'heartbeat_front_addr': '172.21.15.154:6806/493899927', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '56788d74-810e-4827-a615-958c12f8a368', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 48, 'up_thru': 779, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6808', 'nonce': 2375091033}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6809', 'nonce': 2375091033}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6811', 'nonce': 2375091033}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6810', 'nonce': 2375091033}]}, 'public_addr': '172.21.15.154:6808/2375091033', 'cluster_addr': '172.21.15.154:6809/2375091033', 'heartbeat_back_addr': '172.21.15.154:6811/2375091033', 'heartbeat_front_addr': '172.21.15.154:6810/2375091033', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '83503727-afc1-4b95-8b96-ecb63f3338c2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 59, 'up_thru': 776, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6812', 'nonce': 3717704491}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6813', 'nonce': 3717704491}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6815', 'nonce': 3717704491}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6814', 'nonce': 3717704491}]}, 'public_addr': '172.21.15.154:6812/3717704491', 'cluster_addr': '172.21.15.154:6813/3717704491', 'heartbeat_back_addr': '172.21.15.154:6815/3717704491', 'heartbeat_front_addr': '172.21.15.154:6814/3717704491', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-20T22:54:06.520163+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-20T22:54:48.239990+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-20T22:54:46.929979+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-20T22:54:29.696694+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-20T22:54:06.316620+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-20T22:54:54.642519+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-20T22:54:50.271376+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-20T22:54:00.021655+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '1.9', 'osds': [7, 3, 5]}, {'pgid': '6.e', 'osds': [1, 2, 5]}], 'pg_upmap_items': [{'pgid': '1.0', 'mappings': [{'from': 0, 'to': 7}]}, {'pgid': '3.a', 'mappings': [{'from': 6, 'to': 2}]}, {'pgid': '4.9', 'mappings': [{'from': 3, 'to': 2}]}, {'pgid': '4.11', 'mappings': [{'from': 3, 'to': 5}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.84:0/542020545': '2024-09-21T22:42:07.596045+0000', '172.21.15.84:0/25873236': '2024-09-21T22:36:54.917812+0000', '172.21.15.84:6800/820908470': '2024-09-21T22:36:54.917812+0000', '172.21.15.84:0/567614908': '2024-09-21T22:36:54.917812+0000', '172.21.15.84:0/2963622459': '2024-09-21T22:42:07.596045+0000', '172.21.15.84:0/1176849276': '2024-09-21T22:37:17.690104+0000', '172.21.15.84:6800/2932420050': '2024-09-21T22:37:17.690104+0000', '172.21.15.84:0/3879263635': '2024-09-21T22:37:17.690104+0000', '172.21.15.84:0/969472279': '2024-09-21T22:36:54.917812+0000', '172.21.15.84:0/1361143214': '2024-09-21T22:37:17.690104+0000', '172.21.15.84:0/3353126955': '2024-09-21T22:42:07.596045+0000', '172.21.15.84:6800/4024278029': '2024-09-21T22:42:07.596045+0000', '172.21.15.84:0/547902262': '2024-09-21T22:42:07.596045+0000', '172.21.15.84:0/1242929317': '2024-09-21T22:42:07.596045+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [{'pool': 7, 'snaps': [{'begin': 274, 'length': 1}, {'begin': 287, 'length': 5}, {'begin': 293, 'length': 9}, {'begin': 304, 'length': 9}, {'begin': 315, 'length': 3}, {'begin': 319, 'length': 6}, {'begin': 326, 'length': 3}, {'begin': 331, 'length': 3}, {'begin': 336, 'length': 3}, {'begin': 340, 'length': 1}, {'begin': 345, 'length': 1}, {'begin': 347, 'length': 1}, {'begin': 349, 'length': 1}, {'begin': 353, 'length': 4}, {'begin': 360, 'length': 3}, {'begin': 368, 'length': 1}]}], 'new_removed_snaps': [{'pool': 7, 'snaps': [{'begin': 322, 'length': 1}]}], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-20T22:54:58.217 INFO:tasks.thrashosds.thrasher:Clearing pg_upmap on 1.9 2024-09-20T22:54:58.217 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd rm-pg-upmap 1.9 2024-09-20T22:54:58.316 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:54:58.322 INFO:tasks.rados.rados.0.smithi084.stdout:3241: done (3 left) 2024-09-20T22:54:58.322 INFO:tasks.rados.rados.0.smithi084.stdout:3242: done (2 left) 2024-09-20T22:54:58.322 INFO:tasks.rados.rados.0.smithi084.stdout:3243: done (1 left) 2024-09-20T22:54:58.322 INFO:tasks.rados.rados.0.smithi084.stdout:3245: done (0 left) 2024-09-20T22:54:58.323 INFO:tasks.rados.rados.0.smithi084.stdout:3246: snap_remove snap 290 2024-09-20T22:54:58.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:58.338 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:58.338 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:54:58.439 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:54:58.461 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:58.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:58.462 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:54:58.510 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:54:58 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: ::ffff:172.21.15.154 - - [20/Sep/2024:22:54:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-20T22:54:58.551 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:54:58.563 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:54:58.584 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:58.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:58.585 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:54:58.686 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:54:58.707 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:58.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:58.708 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:54:58.795 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:58 smithi084 ceph-mon[40024]: osdmap e817: 8 total, 8 up, 8 in 2024-09-20T22:54:58.795 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:58 smithi084 ceph-mon[40024]: osdmap e818: 8 total, 8 up, 8 in 2024-09-20T22:54:58.795 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:58 smithi084 ceph-mon[29774]: osdmap e817: 8 total, 8 up, 8 in 2024-09-20T22:54:58.795 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:58 smithi084 ceph-mon[29774]: osdmap e818: 8 total, 8 up, 8 in 2024-09-20T22:54:58.809 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:54:58.831 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:58.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:58.831 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:54:58.932 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:54:58.953 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:58.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:58.954 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:54:59.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:58 smithi154 ceph-mon[33264]: osdmap e817: 8 total, 8 up, 8 in 2024-09-20T22:54:59.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:58 smithi154 ceph-mon[33264]: osdmap e818: 8 total, 8 up, 8 in 2024-09-20T22:54:59.032 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:54:59.032 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:54:59.037 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-20T22:54:59.037 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:54:59.054 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:54:59.076 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:59.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:59.077 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:54:59.178 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:54:59.201 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:59.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:59.202 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:54:59.303 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:54:59.327 INFO:tasks.rados.rados.0.smithi084.stdout:3246: done (0 left) 2024-09-20T22:54:59.327 INFO:tasks.rados.rados.0.smithi084.stdout:3247: delete oid 33 current snap is 373 2024-09-20T22:54:59.328 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:59.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:59.329 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:54:59.334 INFO:tasks.rados.rados.0.smithi084.stdout:3247: done (0 left) 2024-09-20T22:54:59.334 INFO:tasks.rados.rados.0.smithi084.stdout:3248: snap_remove snap 349 2024-09-20T22:54:59.430 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:54:59.451 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:59.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:59.452 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:54:59.553 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:54:59.574 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:59.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:59.575 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:54:59.653 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd unset noscrub 2024-09-20T22:54:59.674 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:59 smithi084 ceph-mon[40024]: pgmap v1128: 158 pgs: 3 active+clean+snaptrim, 155 active+clean; 531 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail 2024-09-20T22:54:59.675 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:54:59 smithi084 ceph-mon[40024]: osdmap e819: 8 total, 8 up, 8 in 2024-09-20T22:54:59.675 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:59 smithi084 ceph-mon[29774]: pgmap v1128: 158 pgs: 3 active+clean+snaptrim, 155 active+clean; 531 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail 2024-09-20T22:54:59.675 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:54:59 smithi084 ceph-mon[29774]: osdmap e819: 8 total, 8 up, 8 in 2024-09-20T22:54:59.676 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:54:59.697 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:54:59.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:59.698 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:54:59.799 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:54:59.821 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:59.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:59.822 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:54:59.922 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:54:59.944 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:54:59.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:54:59.945 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:55:00.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:59 smithi154 ceph-mon[33264]: pgmap v1128: 158 pgs: 3 active+clean+snaptrim, 155 active+clean; 531 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail 2024-09-20T22:55:00.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:54:59 smithi154 ceph-mon[33264]: osdmap e819: 8 total, 8 up, 8 in 2024-09-20T22:55:00.004 INFO:journalctl@ceph.iscsi.iscsi.a.smithi154.stdout:Sep 20 22:54:59 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-iscsi-iscsi-a[59730]: debug there is no tcmu-runner data available 2024-09-20T22:55:00.046 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:55:00.067 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:00.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:00.068 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:55:00.168 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:55:00.190 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:00.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:00.191 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:55:00.193 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:55:00.193 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:55:00.210 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:55:00.210 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:55:00.226 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:55:00.226 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:55:00.244 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-20T22:55:00.244 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T22:55:00.260 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:55:00.261 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:55:00.278 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:55:00.278 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:55:00.292 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:55:00.295 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:55:00.295 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:55:00.312 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:55:00.312 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:55:00.315 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:00.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:00.316 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:55:00.319 INFO:teuthology.orchestra.run.smithi084.stderr:clear 1.9 pg_upmap mapping 2024-09-20T22:55:00.325 INFO:tasks.rados.rados.0.smithi084.stdout:3248: done (0 left) 2024-09-20T22:55:00.325 INFO:tasks.rados.rados.0.smithi084.stdout:3249: read oid 9 snap 372 2024-09-20T22:55:00.325 INFO:tasks.rados.rados.0.smithi084.stdout:3249: expect (ObjNum 1094 snap 367 seq_num 1094) 2024-09-20T22:55:00.325 INFO:tasks.rados.rados.0.smithi084.stdout:3250: delete oid 21 current snap is 373 2024-09-20T22:55:00.332 INFO:tasks.rados.rados.0.smithi084.stdout:3250: done (1 left) 2024-09-20T22:55:00.332 INFO:tasks.rados.rados.0.smithi084.stdout:3251: rollback oid 12 current snap is 373 2024-09-20T22:55:00.332 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 12 to 365 2024-09-20T22:55:00.332 INFO:tasks.rados.rados.0.smithi084.stdout:3252: read oid 49 snap -1 2024-09-20T22:55:00.332 INFO:tasks.rados.rados.0.smithi084.stdout:3252: expect (ObjNum 928 snap 308 seq_num 928) 2024-09-20T22:55:00.332 INFO:tasks.rados.rados.0.smithi084.stdout:3253: read oid 7 snap -1 2024-09-20T22:55:00.332 INFO:tasks.rados.rados.0.smithi084.stdout:3253: expect (ObjNum 964 snap 314 seq_num 964) 2024-09-20T22:55:00.333 INFO:tasks.rados.rados.0.smithi084.stdout:3254: read oid 24 snap -1 2024-09-20T22:55:00.333 INFO:tasks.rados.rados.0.smithi084.stdout:3254: expect (ObjNum 780 snap 252 seq_num 780) 2024-09-20T22:55:00.333 INFO:tasks.rados.rados.0.smithi084.stdout:3255: read oid 34 snap 341 2024-09-20T22:55:00.333 INFO:tasks.rados.rados.0.smithi084.stdout:3255: expect deleted 2024-09-20T22:55:00.333 INFO:tasks.rados.rados.0.smithi084.stdout:3256: read oid 5 snap 341 2024-09-20T22:55:00.333 INFO:tasks.rados.rados.0.smithi084.stdout:3256: expect (ObjNum 1012 snap 333 seq_num 1012) 2024-09-20T22:55:00.333 INFO:tasks.rados.rados.0.smithi084.stdout:3257: read oid 10 snap 327 2024-09-20T22:55:00.333 INFO:tasks.rados.rados.0.smithi084.stdout:3257: expect deleted 2024-09-20T22:55:00.333 INFO:tasks.rados.rados.0.smithi084.stdout:3258: snap_create 2024-09-20T22:55:00.343 INFO:tasks.rados.rados.0.smithi084.stdout:3253: expect (ObjNum 964 snap 314 seq_num 964) 2024-09-20T22:55:00.416 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:55:00.437 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:00.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:00.438 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:55:00.468 INFO:tasks.rados.rados.0.smithi084.stdout:3249: expect (ObjNum 1094 snap 367 seq_num 1094) 2024-09-20T22:55:00.539 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:55:00.546 INFO:tasks.rados.rados.0.smithi084.stdout:3256: expect (ObjNum 1012 snap 333 seq_num 1012) 2024-09-20T22:55:00.560 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:00.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:00.561 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:55:00.662 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:55:00.683 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:00.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:00.684 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:55:00.753 INFO:tasks.rados.rados.0.smithi084.stdout:3254: expect (ObjNum 780 snap 252 seq_num 780) 2024-09-20T22:55:00.784 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:55:00.799 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:00 smithi084 ceph-mon[40024]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:55:00.799 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:00 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/2292466919' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:55:00.800 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:00 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/1998314848' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap", "pgid": "1.9"}]: dispatch 2024-09-20T22:55:00.800 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:00 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/1998314848' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap", "pgid": "1.9"}]': finished 2024-09-20T22:55:00.800 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:00 smithi084 ceph-mon[40024]: osdmap e820: 8 total, 8 up, 8 in 2024-09-20T22:55:00.800 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:00 smithi084 ceph-mon[29774]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:55:00.800 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:00 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/2292466919' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:55:00.800 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:00 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/1998314848' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap", "pgid": "1.9"}]: dispatch 2024-09-20T22:55:00.800 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:00 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/1998314848' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap", "pgid": "1.9"}]': finished 2024-09-20T22:55:00.800 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:00 smithi084 ceph-mon[29774]: osdmap e820: 8 total, 8 up, 8 in 2024-09-20T22:55:00.805 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:00.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:00.806 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:55:00.875 INFO:tasks.rados.rados.0.smithi084.stdout:3252: expect (ObjNum 928 snap 308 seq_num 928) 2024-09-20T22:55:00.906 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:55:00.927 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:00.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:00.928 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:55:01.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:00 smithi154 ceph-mon[33264]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:55:01.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:00 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/2292466919' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:55:01.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:00 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/1998314848' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap", "pgid": "1.9"}]: dispatch 2024-09-20T22:55:01.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:00 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/1998314848' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap", "pgid": "1.9"}]': finished 2024-09-20T22:55:01.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:00 smithi154 ceph-mon[33264]: osdmap e820: 8 total, 8 up, 8 in 2024-09-20T22:55:01.029 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:55:01.049 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:01.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:01.050 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:55:01.118 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-20T22:55:01.151 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:55:01.169 INFO:tasks.rados.rados.0.smithi084.stdout:3251: finishing rollback tid 1 to smithi08477203-12 2024-09-20T22:55:01.169 INFO:tasks.rados.rados.0.smithi084.stdout:3251: finishing rollback tid 2 to smithi08477203-12 2024-09-20T22:55:01.169 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 12 v 2462 (ObjNum 795 snap 254 seq_num 795) dirty exists 2024-09-20T22:55:01.172 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:01.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:01.173 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:55:01.273 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:55:01.294 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:01.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:01.295 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:55:01.325 INFO:tasks.rados.rados.0.smithi084.stdout:3249: done (8 left) 2024-09-20T22:55:01.325 INFO:tasks.rados.rados.0.smithi084.stdout:3251: done (7 left) 2024-09-20T22:55:01.325 INFO:tasks.rados.rados.0.smithi084.stdout:3252: done (6 left) 2024-09-20T22:55:01.325 INFO:tasks.rados.rados.0.smithi084.stdout:3253: done (5 left) 2024-09-20T22:55:01.325 INFO:tasks.rados.rados.0.smithi084.stdout:3254: done (4 left) 2024-09-20T22:55:01.325 INFO:tasks.rados.rados.0.smithi084.stdout:3255: done (3 left) 2024-09-20T22:55:01.325 INFO:tasks.rados.rados.0.smithi084.stdout:3256: done (2 left) 2024-09-20T22:55:01.325 INFO:tasks.rados.rados.0.smithi084.stdout:3257: done (1 left) 2024-09-20T22:55:01.325 INFO:tasks.rados.rados.0.smithi084.stdout:3258: done (0 left) 2024-09-20T22:55:01.326 INFO:tasks.rados.rados.0.smithi084.stdout:3259: read oid 31 snap 344 2024-09-20T22:55:01.326 INFO:tasks.rados.rados.0.smithi084.stdout:3259: expect (ObjNum 910 snap 297 seq_num 910) 2024-09-20T22:55:01.326 INFO:tasks.rados.rados.0.smithi084.stdout:3260: write (excl) oid 42 current snap is 374 2024-09-20T22:55:01.326 INFO:tasks.rados.rados.0.smithi084.stdout:3260: seq_num 1104 ranges {554928=746081,2020003=534925} 2024-09-20T22:55:01.332 INFO:tasks.rados.rados.0.smithi084.stdout:3260: writing smithi08477203-42 from 554928 to 1301009 tid 1 2024-09-20T22:55:01.340 INFO:tasks.rados.rados.0.smithi084.stdout:3260: writing smithi08477203-42 from 2020003 to 2554928 tid 2 2024-09-20T22:55:01.340 INFO:tasks.rados.rados.0.smithi084.stdout:3261: read oid 1 snap 362 2024-09-20T22:55:01.340 INFO:tasks.rados.rados.0.smithi084.stdout:3261: expect (ObjNum 738 snap 235 seq_num 738) 2024-09-20T22:55:01.341 INFO:tasks.rados.rados.0.smithi084.stdout:3262: write (excl) oid 21 current snap is 374 2024-09-20T22:55:01.341 INFO:tasks.rados.rados.0.smithi084.stdout:3262: seq_num 1105 ranges {571735=421330,1562132=416708} 2024-09-20T22:55:01.346 INFO:tasks.rados.rados.0.smithi084.stdout:3262: writing smithi08477203-21 from 571735 to 993065 tid 1 2024-09-20T22:55:01.351 INFO:tasks.rados.rados.0.smithi084.stdout:3262: writing smithi08477203-21 from 1562132 to 1978840 tid 2 2024-09-20T22:55:01.351 INFO:tasks.rados.rados.0.smithi084.stdout:3263: snap_create 2024-09-20T22:55:01.351 INFO:tasks.rados.rados.0.smithi084.stdout:3259: expect (ObjNum 910 snap 297 seq_num 910) 2024-09-20T22:55:01.372 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:55:01.396 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:55:01.417 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:01.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:01.418 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:55:01.518 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:55:01.526 INFO:tasks.rados.rados.0.smithi084.stdout:3261: expect (ObjNum 738 snap 235 seq_num 738) 2024-09-20T22:55:01.539 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:01.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:01.540 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:55:01.607 INFO:tasks.rados.rados.0.smithi084.stdout:3260: finishing write tid 1 to smithi08477203-42 2024-09-20T22:55:01.608 INFO:tasks.rados.rados.0.smithi084.stdout:3260: finishing write tid 2 to smithi08477203-42 2024-09-20T22:55:01.608 INFO:tasks.rados.rados.0.smithi084.stdout:3260: finishing write tid 3 to smithi08477203-42 2024-09-20T22:55:01.608 INFO:tasks.rados.rados.0.smithi084.stdout:3260: finishing write tid 4 to smithi08477203-42 2024-09-20T22:55:01.608 INFO:tasks.rados.rados.0.smithi084.stdout:3260: oid 42 updating version 0 to 1916 2024-09-20T22:55:01.608 INFO:tasks.rados.rados.0.smithi084.stdout:3260: oid 42 updating version 1916 to 1917 2024-09-20T22:55:01.608 INFO:tasks.rados.rados.0.smithi084.stdout:3260: oid 42 version 1917 is already newer than 1915 2024-09-20T22:55:01.608 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 42 v 1917 (ObjNum 1104 snap 374 seq_num 1104) dirty exists 2024-09-20T22:55:01.608 INFO:tasks.rados.rados.0.smithi084.stdout:3260: left oid 42 (ObjNum 1104 snap 374 seq_num 1104) 2024-09-20T22:55:01.608 INFO:tasks.rados.rados.0.smithi084.stdout:3262: finishing write tid 1 to smithi08477203-21 2024-09-20T22:55:01.608 INFO:tasks.rados.rados.0.smithi084.stdout:3262: finishing write tid 2 to smithi08477203-21 2024-09-20T22:55:01.608 INFO:tasks.rados.rados.0.smithi084.stdout:3262: finishing write tid 3 to smithi08477203-21 2024-09-20T22:55:01.608 INFO:tasks.rados.rados.0.smithi084.stdout:3262: finishing write tid 4 to smithi08477203-21 2024-09-20T22:55:01.608 INFO:tasks.rados.rados.0.smithi084.stdout:3262: oid 21 updating version 0 to 1845 2024-09-20T22:55:01.608 INFO:tasks.rados.rados.0.smithi084.stdout:3262: oid 21 updating version 1845 to 1846 2024-09-20T22:55:01.609 INFO:tasks.rados.rados.0.smithi084.stdout:3262: oid 21 version 1846 is already newer than 1844 2024-09-20T22:55:01.609 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 21 v 1846 (ObjNum 1105 snap 374 seq_num 1105) dirty exists 2024-09-20T22:55:01.609 INFO:tasks.rados.rados.0.smithi084.stdout:3262: left oid 21 (ObjNum 1105 snap 374 seq_num 1105) 2024-09-20T22:55:01.641 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:55:01.664 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:01.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:01.665 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:55:01.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:01 smithi154 ceph-mon[33264]: pgmap v1131: 158 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 153 active+clean; 535 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 3.3 MiB/s rd, 601 KiB/s wr, 7 op/s 2024-09-20T22:55:01.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:01 smithi154 ceph-mon[33264]: osdmap e821: 8 total, 8 up, 8 in 2024-09-20T22:55:01.766 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:55:01.787 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:01.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:01.788 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:55:01.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:01 smithi084 ceph-mon[40024]: pgmap v1131: 158 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 153 active+clean; 535 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 3.3 MiB/s rd, 601 KiB/s wr, 7 op/s 2024-09-20T22:55:01.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:01 smithi084 ceph-mon[40024]: osdmap e821: 8 total, 8 up, 8 in 2024-09-20T22:55:01.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:01 smithi084 ceph-mon[29774]: pgmap v1131: 158 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 153 active+clean; 535 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 3.3 MiB/s rd, 601 KiB/s wr, 7 op/s 2024-09-20T22:55:01.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:01 smithi084 ceph-mon[29774]: osdmap e821: 8 total, 8 up, 8 in 2024-09-20T22:55:01.889 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:55:01.894 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-20T22:55:01.894 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [], 2024-09-20T22:55:01.894 INFO:teuthology.orchestra.run.smithi154.stdout: "num_ops": 0 2024-09-20T22:55:01.894 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-20T22:55:01.910 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:01.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:01.911 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:55:02.012 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:55:02.035 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:02.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:02.036 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:55:02.137 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:55:02.158 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:02.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:02.159 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:55:02.260 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:55:02.281 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:02.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:02.282 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:55:02.328 INFO:tasks.rados.rados.0.smithi084.stdout:3259: done (4 left) 2024-09-20T22:55:02.328 INFO:tasks.rados.rados.0.smithi084.stdout:3260: done (3 left) 2024-09-20T22:55:02.328 INFO:tasks.rados.rados.0.smithi084.stdout:3261: done (2 left) 2024-09-20T22:55:02.328 INFO:tasks.rados.rados.0.smithi084.stdout:3262: done (1 left) 2024-09-20T22:55:02.328 INFO:tasks.rados.rados.0.smithi084.stdout:3263: done (0 left) 2024-09-20T22:55:02.328 INFO:tasks.rados.rados.0.smithi084.stdout:3264: write (excl) oid 47 current snap is 375 2024-09-20T22:55:02.328 INFO:tasks.rados.rados.0.smithi084.stdout:3264: seq_num 1106 ranges {588542=496579,1620614=676719} 2024-09-20T22:55:02.333 INFO:tasks.rados.rados.0.smithi084.stdout:3264: writing smithi08477203-47 from 588542 to 1085121 tid 1 2024-09-20T22:55:02.340 INFO:tasks.rados.rados.0.smithi084.stdout:3264: writing smithi08477203-47 from 1620614 to 2297333 tid 2 2024-09-20T22:55:02.341 INFO:tasks.rados.rados.0.smithi084.stdout:3265: read oid 31 snap -1 2024-09-20T22:55:02.341 INFO:tasks.rados.rados.0.smithi084.stdout:3265: expect (ObjNum 1101 snap 372 seq_num 1101) 2024-09-20T22:55:02.341 INFO:tasks.rados.rados.0.smithi084.stdout:3266: snap_create 2024-09-20T22:55:02.341 INFO:tasks.rados.rados.0.smithi084.stdout:3264: finishing write tid 1 to smithi08477203-47 2024-09-20T22:55:02.351 INFO:tasks.rados.rados.0.smithi084.stdout:3264: finishing write tid 2 to smithi08477203-47 2024-09-20T22:55:02.351 INFO:tasks.rados.rados.0.smithi084.stdout:3264: finishing write tid 3 to smithi08477203-47 2024-09-20T22:55:02.352 INFO:tasks.rados.rados.0.smithi084.stdout:3264: finishing write tid 4 to smithi08477203-47 2024-09-20T22:55:02.352 INFO:tasks.rados.rados.0.smithi084.stdout:3264: oid 47 updating version 0 to 2463 2024-09-20T22:55:02.352 INFO:tasks.rados.rados.0.smithi084.stdout:3264: oid 47 updating version 2463 to 2465 2024-09-20T22:55:02.352 INFO:tasks.rados.rados.0.smithi084.stdout:3264: oid 47 version 2465 is already newer than 2464 2024-09-20T22:55:02.352 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 47 v 2465 (ObjNum 1106 snap 375 seq_num 1106) dirty exists 2024-09-20T22:55:02.352 INFO:tasks.rados.rados.0.smithi084.stdout:3264: left oid 47 (ObjNum 1106 snap 375 seq_num 1106) 2024-09-20T22:55:02.353 INFO:tasks.rados.rados.0.smithi084.stdout:3265: expect (ObjNum 1101 snap 372 seq_num 1101) 2024-09-20T22:55:02.382 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:55:02.398 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:55:02.404 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:02.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:02.405 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:55:02.506 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:55:02.527 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:02.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:02.528 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:55:02.628 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:55:02.652 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:02.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:02.652 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:55:02.669 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.4 dump_blocked_ops 2024-09-20T22:55:02.754 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:55:02.775 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:02.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:02.776 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:55:02.877 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:55:02.900 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:02.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:02.901 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:55:02.923 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:55:03.002 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:55:03.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:03.023 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:03.023 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:55:03.124 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:55:03.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:03.146 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:03.147 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:55:03.248 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:55:03.269 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:03.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:03.270 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:55:03.318 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is unset 2024-09-20T22:55:03.325 INFO:tasks.rados.rados.0.smithi084.stdout:3264: done (2 left) 2024-09-20T22:55:03.325 INFO:tasks.rados.rados.0.smithi084.stdout:3265: done (1 left) 2024-09-20T22:55:03.326 INFO:tasks.rados.rados.0.smithi084.stdout:3266: done (0 left) 2024-09-20T22:55:03.326 INFO:tasks.rados.rados.0.smithi084.stdout:3267: read oid 44 snap -1 2024-09-20T22:55:03.326 INFO:tasks.rados.rados.0.smithi084.stdout:3267: expect (ObjNum 1051 snap 345 seq_num 1051) 2024-09-20T22:55:03.326 INFO:tasks.rados.rados.0.smithi084.stdout:3268: delete oid 23 current snap is 376 2024-09-20T22:55:03.331 INFO:tasks.rados.rados.0.smithi084.stdout:3268: done (1 left) 2024-09-20T22:55:03.332 INFO:tasks.rados.rados.0.smithi084.stdout:3269: snap_create 2024-09-20T22:55:03.341 INFO:tasks.rados.rados.0.smithi084.stdout:3267: expect (ObjNum 1051 snap 345 seq_num 1051) 2024-09-20T22:55:03.371 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:55:03.392 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:03.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:03.393 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:55:03.428 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-20T22:55:03.428 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [], 2024-09-20T22:55:03.428 INFO:teuthology.orchestra.run.smithi154.stdout: "complaint_time": 30, 2024-09-20T22:55:03.428 INFO:teuthology.orchestra.run.smithi154.stdout: "num_blocked_ops": 0 2024-09-20T22:55:03.428 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-20T22:55:03.494 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:55:03.515 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:03.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:03.516 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:55:03.587 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-20T22:55:03.609 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:03 smithi084 ceph-mon[40024]: pgmap v1134: 158 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 153 active+clean; 535 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 3.3 MiB/s rd, 601 KiB/s wr, 7 op/s 2024-09-20T22:55:03.609 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:03 smithi084 ceph-mon[40024]: osdmap e822: 8 total, 8 up, 8 in 2024-09-20T22:55:03.609 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:03 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/2548948350' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:55:03.609 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:03 smithi084 ceph-mon[40024]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:55:03.610 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:03 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/2548948350' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T22:55:03.610 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:03 smithi084 ceph-mon[29774]: pgmap v1134: 158 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 153 active+clean; 535 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 3.3 MiB/s rd, 601 KiB/s wr, 7 op/s 2024-09-20T22:55:03.610 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:03 smithi084 ceph-mon[29774]: osdmap e822: 8 total, 8 up, 8 in 2024-09-20T22:55:03.610 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:03 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/2548948350' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:55:03.610 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:03 smithi084 ceph-mon[29774]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:55:03.610 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:03 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/2548948350' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T22:55:03.617 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:55:03.638 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:03.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:03.639 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:55:03.739 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:55:03.749 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:03 smithi154 ceph-mon[33264]: pgmap v1134: 158 pgs: 1 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 153 active+clean; 535 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 3.3 MiB/s rd, 601 KiB/s wr, 7 op/s 2024-09-20T22:55:03.749 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:03 smithi154 ceph-mon[33264]: osdmap e822: 8 total, 8 up, 8 in 2024-09-20T22:55:03.749 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:03 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/2548948350' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:55:03.749 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:03 smithi154 ceph-mon[33264]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:55:03.749 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:03 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/2548948350' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T22:55:03.760 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:03.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:03.761 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:55:03.862 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:55:03.883 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:03.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:03.884 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:55:03.985 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:55:04.006 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:04.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:04.007 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:55:04.041 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.4 dump_historic_ops 2024-09-20T22:55:04.107 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:55:04.146 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:04.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:04.147 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:55:04.247 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:55:04.269 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:04.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:04.270 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:55:04.332 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:55:04.342 INFO:tasks.rados.rados.0.smithi084.stdout:3267: done (1 left) 2024-09-20T22:55:04.342 INFO:tasks.rados.rados.0.smithi084.stdout:3269: done (0 left) 2024-09-20T22:55:04.342 INFO:tasks.rados.rados.0.smithi084.stdout:3270: delete oid 28 current snap is 377 2024-09-20T22:55:04.348 INFO:tasks.rados.rados.0.smithi084.stdout:3270: done (0 left) 2024-09-20T22:55:04.349 INFO:tasks.rados.rados.0.smithi084.stdout:3271: read oid 6 snap -1 2024-09-20T22:55:04.349 INFO:tasks.rados.rados.0.smithi084.stdout:3271: expect deleted 2024-09-20T22:55:04.349 INFO:tasks.rados.rados.0.smithi084.stdout:3272: copy_from oid 7 from oid 40 current snap is 377 2024-09-20T22:55:04.349 INFO:tasks.rados.rados.0.smithi084.stdout:3273: write (excl) oid 3 current snap is 377 2024-09-20T22:55:04.349 INFO:tasks.rados.rados.0.smithi084.stdout:3273: seq_num 1109 ranges {638963=722326,2079707=424046} 2024-09-20T22:55:04.356 INFO:tasks.rados.rados.0.smithi084.stdout:3273: writing smithi08477203-3 from 638963 to 1361289 tid 1 2024-09-20T22:55:04.363 INFO:tasks.rados.rados.0.smithi084.stdout:3273: writing smithi08477203-3 from 2079707 to 2503753 tid 2 2024-09-20T22:55:04.363 INFO:tasks.rados.rados.0.smithi084.stdout:3274: rollback oid 8 current snap is 377 2024-09-20T22:55:04.363 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 8 to 371 2024-09-20T22:55:04.363 INFO:tasks.rados.rados.0.smithi084.stdout:3272: finishing copy_from racing read to smithi08477203-7 2024-09-20T22:55:04.363 INFO:tasks.rados.rados.0.smithi084.stdout:3272: finishing copy_from to smithi08477203-7 2024-09-20T22:55:04.363 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 7 v 1813 (ObjNum 795 snap 254 seq_num 795) dirty exists 2024-09-20T22:55:04.363 INFO:tasks.rados.rados.0.smithi084.stdout:3273: finishing write tid 1 to smithi08477203-3 2024-09-20T22:55:04.363 INFO:tasks.rados.rados.0.smithi084.stdout:3271: done (3 left) 2024-09-20T22:55:04.363 INFO:tasks.rados.rados.0.smithi084.stdout:3272: done (2 left) 2024-09-20T22:55:04.364 INFO:tasks.rados.rados.0.smithi084.stdout:3275: write oid 30 current snap is 377 2024-09-20T22:55:04.364 INFO:tasks.rados.rados.0.smithi084.stdout:3275: seq_num 1110 ranges {655770=513928,1854542=684057} 2024-09-20T22:55:04.369 INFO:tasks.rados.rados.0.smithi084.stdout:3275: writing smithi08477203-30 from 655770 to 1169698 tid 1 2024-09-20T22:55:04.371 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:55:04.377 INFO:tasks.rados.rados.0.smithi084.stdout:3275: writing smithi08477203-30 from 1854542 to 2538599 tid 2 2024-09-20T22:55:04.377 INFO:tasks.rados.rados.0.smithi084.stdout:3276: snap_create 2024-09-20T22:55:04.377 INFO:tasks.rados.rados.0.smithi084.stdout:3273: finishing write tid 2 to smithi08477203-3 2024-09-20T22:55:04.377 INFO:tasks.rados.rados.0.smithi084.stdout:3273: finishing write tid 3 to smithi08477203-3 2024-09-20T22:55:04.377 INFO:tasks.rados.rados.0.smithi084.stdout:3273: finishing write tid 4 to smithi08477203-3 2024-09-20T22:55:04.377 INFO:tasks.rados.rados.0.smithi084.stdout:3273: oid 3 updating version 0 to 2548 2024-09-20T22:55:04.377 INFO:tasks.rados.rados.0.smithi084.stdout:3273: oid 3 updating version 2548 to 2549 2024-09-20T22:55:04.377 INFO:tasks.rados.rados.0.smithi084.stdout:3273: oid 3 version 2549 is already newer than 2547 2024-09-20T22:55:04.377 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 3 v 2549 (ObjNum 1109 snap 377 seq_num 1109) dirty exists 2024-09-20T22:55:04.377 INFO:tasks.rados.rados.0.smithi084.stdout:3273: left oid 3 (ObjNum 1109 snap 377 seq_num 1109) 2024-09-20T22:55:04.377 INFO:tasks.rados.rados.0.smithi084.stdout:3274: finishing rollback tid 1 to smithi08477203-8 2024-09-20T22:55:04.378 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 8 v 1853 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-09-20T22:55:04.378 INFO:tasks.rados.rados.0.smithi084.stdout:3275: finishing write tid 1 to smithi08477203-30 2024-09-20T22:55:04.380 INFO:tasks.rados.rados.0.smithi084.stdout:3275: finishing write tid 2 to smithi08477203-30 2024-09-20T22:55:04.380 INFO:tasks.rados.rados.0.smithi084.stdout:3275: finishing write tid 3 to smithi08477203-30 2024-09-20T22:55:04.380 INFO:tasks.rados.rados.0.smithi084.stdout:3275: finishing write tid 4 to smithi08477203-30 2024-09-20T22:55:04.380 INFO:tasks.rados.rados.0.smithi084.stdout:3275: oid 30 updating version 0 to 2551 2024-09-20T22:55:04.381 INFO:tasks.rados.rados.0.smithi084.stdout:3275: oid 30 version 2551 is already newer than 2550 2024-09-20T22:55:04.381 INFO:tasks.rados.rados.0.smithi084.stdout:3275: oid 30 updating version 2551 to 2552 2024-09-20T22:55:04.381 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 30 v 2552 (ObjNum 1110 snap 377 seq_num 1110) dirty exists 2024-09-20T22:55:04.381 INFO:tasks.rados.rados.0.smithi084.stdout:3275: left oid 30 (ObjNum 1110 snap 377 seq_num 1110) 2024-09-20T22:55:04.392 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:04.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:04.393 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:55:04.494 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:55:04.515 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:04.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:04.516 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:55:04.605 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:04 smithi154 ceph-mon[33264]: osdmap e823: 8 total, 8 up, 8 in 2024-09-20T22:55:04.605 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:04 smithi154 ceph-mon[33264]: 7.c scrub starts 2024-09-20T22:55:04.605 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:04 smithi154 ceph-mon[33264]: 7.c scrub ok 2024-09-20T22:55:04.605 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:04 smithi154 ceph-mon[33264]: 5.12 scrub starts 2024-09-20T22:55:04.605 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:04 smithi154 ceph-mon[33264]: 5.12 scrub ok 2024-09-20T22:55:04.614 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:04 smithi084 ceph-mon[29774]: osdmap e823: 8 total, 8 up, 8 in 2024-09-20T22:55:04.614 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:04 smithi084 ceph-mon[29774]: 7.c scrub starts 2024-09-20T22:55:04.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:04 smithi084 ceph-mon[29774]: 7.c scrub ok 2024-09-20T22:55:04.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:04 smithi084 ceph-mon[29774]: 5.12 scrub starts 2024-09-20T22:55:04.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:04 smithi084 ceph-mon[29774]: 5.12 scrub ok 2024-09-20T22:55:04.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:04 smithi084 ceph-mon[40024]: osdmap e823: 8 total, 8 up, 8 in 2024-09-20T22:55:04.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:04 smithi084 ceph-mon[40024]: 7.c scrub starts 2024-09-20T22:55:04.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:04 smithi084 ceph-mon[40024]: 7.c scrub ok 2024-09-20T22:55:04.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:04 smithi084 ceph-mon[40024]: 5.12 scrub starts 2024-09-20T22:55:04.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:04 smithi084 ceph-mon[40024]: 5.12 scrub ok 2024-09-20T22:55:04.617 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:55:04.637 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:04.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:04.638 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:55:04.739 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:55:04.762 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:04.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:04.763 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:55:04.863 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:55:04.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:04.885 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:04.885 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:55:04.948 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-20T22:55:04.949 INFO:teuthology.orchestra.run.smithi154.stdout: "size": 20, 2024-09-20T22:55:04.949 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 600, 2024-09-20T22:55:04.949 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [ 2024-09-20T22:55:04.949 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:04.949 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:4229 7.14 e423/421 7:2f8647a2:::smithi08477203-41:head v 423'133, mlcod=422'129)", 2024-09-20T22:55:04.949 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:48:23.296695+0000", 2024-09-20T22:55:04.949 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 401.52875363200002, 2024-09-20T22:55:04.949 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.011591664, 2024-09-20T22:55:04.949 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:55:04.949 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:55:04.949 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:55:04.949 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:55:04.949 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:04.950 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:55:04.950 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:23.296695+0000", 2024-09-20T22:55:04.950 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:55:04.950 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:04.950 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:04.950 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:55:04.950 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:23.296695+0000", 2024-09-20T22:55:04.950 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:55:04.950 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:04.950 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:04.950 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:55:04.950 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:23.296697+0000", 2024-09-20T22:55:04.950 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.8509999999999999e-06 2024-09-20T22:55:04.950 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:04.951 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:04.951 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:55:04.951 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:23.302610+0000", 2024-09-20T22:55:04.951 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0059131959999999999 2024-09-20T22:55:04.951 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:04.951 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:04.951 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:55:04.951 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:23.302632+0000", 2024-09-20T22:55:04.951 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.2146000000000001e-05 2024-09-20T22:55:04.951 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:04.951 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:04.951 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:55:04.951 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:23.302641+0000", 2024-09-20T22:55:04.952 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.9069999999999996e-06 2024-09-20T22:55:04.952 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:04.952 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:04.952 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:55:04.952 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:23.302711+0000", 2024-09-20T22:55:04.952 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.0038999999999998e-05 2024-09-20T22:55:04.952 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:04.952 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:04.952 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:55:04.952 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:23.302732+0000", 2024-09-20T22:55:04.952 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.0922000000000001e-05 2024-09-20T22:55:04.952 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:04.952 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:04.953 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:55:04.953 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:23.308245+0000", 2024-09-20T22:55:04.953 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0055130889999999997 2024-09-20T22:55:04.953 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:04.953 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:04.953 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:55:04.953 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:23.308287+0000", 2024-09-20T22:55:04.953 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.1514000000000003e-05 2024-09-20T22:55:04.953 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:55:04.953 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:55:04.953 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:55:04.953 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:04.953 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:04.953 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:5057 7.9 e508/426 7:9eac4560:::smithi08477203-30:head v 508'554, mlcod=505'548)", 2024-09-20T22:55:04.954 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:49:48.245783+0000", 2024-09-20T22:55:04.954 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 316.57966567800003, 2024-09-20T22:55:04.954 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.035976553000000001, 2024-09-20T22:55:04.954 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:55:04.954 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:55:04.954 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:55:04.954 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:55:04.954 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:04.954 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:55:04.954 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:48.245783+0000", 2024-09-20T22:55:04.954 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:55:04.954 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:04.955 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:04.955 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:55:04.955 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:48.245783+0000", 2024-09-20T22:55:04.955 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:55:04.955 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:04.955 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:04.955 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:55:04.955 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:48.245784+0000", 2024-09-20T22:55:04.955 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.08e-06 2024-09-20T22:55:04.955 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:04.955 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:04.955 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:55:04.955 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:48.279966+0000", 2024-09-20T22:55:04.956 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.034181493 2024-09-20T22:55:04.956 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:04.956 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:04.956 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:55:04.956 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:48.279981+0000", 2024-09-20T22:55:04.956 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.5092000000000001e-05 2024-09-20T22:55:04.956 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:04.956 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:04.956 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:55:04.956 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:48.280000+0000", 2024-09-20T22:55:04.956 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9493000000000001e-05 2024-09-20T22:55:04.956 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:04.956 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:04.957 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:55:04.957 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:48.280054+0000", 2024-09-20T22:55:04.957 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.3687000000000003e-05 2024-09-20T22:55:04.957 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:04.957 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:04.957 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:55:04.957 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:48.280072+0000", 2024-09-20T22:55:04.957 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.8581e-05 2024-09-20T22:55:04.957 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:04.957 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:04.957 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:55:04.957 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:48.281731+0000", 2024-09-20T22:55:04.957 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0016588569999999999 2024-09-20T22:55:04.957 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:04.958 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:04.958 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:55:04.958 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:48.281760+0000", 2024-09-20T22:55:04.958 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.8269999999999999e-05 2024-09-20T22:55:04.958 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:55:04.958 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:55:04.958 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:55:04.958 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:04.958 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:04.958 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:5414 7.8 e549/430 7:1cd232b2:::smithi08477203-35:head v 549'467, mlcod=548'459)", 2024-09-20T22:55:04.958 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:50:29.260963+0000", 2024-09-20T22:55:04.958 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 275.56448617900003, 2024-09-20T22:55:04.958 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0085861400000000008, 2024-09-20T22:55:04.959 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:55:04.959 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:55:04.959 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:55:04.959 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:55:04.959 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:04.959 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:55:04.959 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:29.260963+0000", 2024-09-20T22:55:04.959 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:55:04.959 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:04.959 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:04.959 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:55:04.959 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:29.260963+0000", 2024-09-20T22:55:04.959 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:55:04.960 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:04.960 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:04.960 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:55:04.960 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:29.260963+0000", 2024-09-20T22:55:04.960 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.0900000000000002e-07 2024-09-20T22:55:04.960 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:04.960 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:04.960 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:55:04.960 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:29.266981+0000", 2024-09-20T22:55:04.960 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0060184360000000003 2024-09-20T22:55:04.960 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:04.960 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:04.960 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:55:04.961 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:29.266992+0000", 2024-09-20T22:55:04.961 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0254e-05 2024-09-20T22:55:04.961 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:04.961 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:04.961 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:55:04.961 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:29.266999+0000", 2024-09-20T22:55:04.961 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.0369999999999997e-06 2024-09-20T22:55:04.961 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:04.961 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:04.961 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:55:04.961 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:29.267052+0000", 2024-09-20T22:55:04.961 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.3377000000000001e-05 2024-09-20T22:55:04.961 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:04.961 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:04.961 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:55:04.962 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:29.267073+0000", 2024-09-20T22:55:04.962 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.0604000000000001e-05 2024-09-20T22:55:04.962 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:04.962 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:04.962 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:55:04.962 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:29.269512+0000", 2024-09-20T22:55:04.962 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0024397749999999999 2024-09-20T22:55:04.962 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:04.962 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:04.962 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:55:04.962 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:29.269549+0000", 2024-09-20T22:55:04.962 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.6248000000000003e-05 2024-09-20T22:55:04.962 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:55:04.962 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:55:04.963 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:55:04.963 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:04.963 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:04.963 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:5418 7.1 7:8422f82d:::smithi08477203-22:head [copy-from ver 726] snapc ef=[ef,ed,ec,eb,ea,e8,e7,e6,e5,e4,e2,df,db,da,d7,d5,d4,d3,d0,cf,ce,ca,c5,c2,bf,b4,b2,a7,95] ondisk+write+known_if_redirected+supports_pool_eio e549)", 2024-09-20T22:55:04.963 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:50:29.266643+0000", 2024-09-20T22:55:04.963 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 275.558805511, 2024-09-20T22:55:04.963 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.022362705, 2024-09-20T22:55:04.963 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:55:04.963 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:55:04.963 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:55:04.963 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:55:04.963 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:55:04.963 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:55:04.963 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 5418 2024-09-20T22:55:04.964 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:04.964 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:55:04.964 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:04.964 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:55:04.964 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:29.266643+0000", 2024-09-20T22:55:04.964 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:55:04.964 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:04.964 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:04.964 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:55:04.964 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:29.266643+0000", 2024-09-20T22:55:04.964 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:55:04.964 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:04.964 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:04.964 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:55:04.964 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:29.266646+0000", 2024-09-20T22:55:04.965 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.835e-06 2024-09-20T22:55:04.965 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:04.965 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:04.965 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:55:04.965 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:29.266654+0000", 2024-09-20T22:55:04.965 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.7540000000000006e-06 2024-09-20T22:55:04.965 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:04.965 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:04.965 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:55:04.965 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:29.266672+0000", 2024-09-20T22:55:04.965 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.8187000000000001e-05 2024-09-20T22:55:04.965 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:04.965 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:04.965 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:55:04.966 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:29.266681+0000", 2024-09-20T22:55:04.966 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.5199999999999997e-06 2024-09-20T22:55:04.966 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:04.966 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:04.966 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:55:04.966 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:29.266788+0000", 2024-09-20T22:55:04.966 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00010746 2024-09-20T22:55:04.966 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:04.966 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:04.966 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:55:04.966 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:29.266936+0000", 2024-09-20T22:55:04.966 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000148238 2024-09-20T22:55:04.966 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:04.967 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:04.967 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 3,6", 2024-09-20T22:55:04.967 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:29.274529+0000", 2024-09-20T22:55:04.967 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0075932170000000002 2024-09-20T22:55:04.967 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:04.967 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:04.967 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:55:04.967 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:29.279305+0000", 2024-09-20T22:55:04.967 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0047758990000000001 2024-09-20T22:55:04.967 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:04.967 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:04.967 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:55:04.967 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:29.281410+0000", 2024-09-20T22:55:04.967 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.002104341 2024-09-20T22:55:04.968 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:04.968 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:04.968 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:55:04.968 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:29.288915+0000", 2024-09-20T22:55:04.968 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0075048900000000002 2024-09-20T22:55:04.968 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:04.968 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:04.968 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:55:04.968 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:29.288966+0000", 2024-09-20T22:55:04.968 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.1669000000000003e-05 2024-09-20T22:55:04.968 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:04.968 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:04.968 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:55:04.968 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:29.289006+0000", 2024-09-20T22:55:04.969 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.9694999999999999e-05 2024-09-20T22:55:04.969 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:55:04.969 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:55:04.969 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:55:04.969 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:04.969 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:04.969 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:5419 7.1 7:8422f82d:::smithi08477203-22:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e549)", 2024-09-20T22:55:04.969 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:50:29.266705+0000", 2024-09-20T22:55:04.969 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 275.55874417799998, 2024-09-20T22:55:04.969 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.012031030999999999, 2024-09-20T22:55:04.969 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:55:04.969 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:55:04.969 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:55:04.969 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:55:04.970 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:55:04.970 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:55:04.970 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 5419 2024-09-20T22:55:04.970 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:04.970 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:55:04.970 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:04.970 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:55:04.970 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:29.266705+0000", 2024-09-20T22:55:04.970 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:55:04.970 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:04.970 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:04.970 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:55:04.970 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:29.266705+0000", 2024-09-20T22:55:04.971 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:55:04.971 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:04.971 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:04.971 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:55:04.971 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:29.266705+0000", 2024-09-20T22:55:04.971 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.5000000000000002e-07 2024-09-20T22:55:04.971 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:04.971 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:04.971 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:55:04.971 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:29.266708+0000", 2024-09-20T22:55:04.971 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.242e-06 2024-09-20T22:55:04.971 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:04.971 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:04.971 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:55:04.971 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:29.266717+0000", 2024-09-20T22:55:04.972 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.9390000000000003e-06 2024-09-20T22:55:04.972 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:04.972 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:04.972 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:55:04.972 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:29.266723+0000", 2024-09-20T22:55:04.972 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.7289999999999997e-06 2024-09-20T22:55:04.972 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:04.972 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:04.972 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:55:04.972 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:29.267079+0000", 2024-09-20T22:55:04.972 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00035592600000000001 2024-09-20T22:55:04.972 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:04.972 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:04.972 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for blocked object", 2024-09-20T22:55:04.972 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:29.267125+0000", 2024-09-20T22:55:04.973 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.5859000000000003e-05 2024-09-20T22:55:04.973 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:04.973 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:04.973 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:55:04.973 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:29.278512+0000", 2024-09-20T22:55:04.973 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.011386871999999999 2024-09-20T22:55:04.973 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:04.973 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:04.973 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:55:04.973 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:29.278620+0000", 2024-09-20T22:55:04.973 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000108729 2024-09-20T22:55:04.973 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:04.973 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:04.973 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:55:04.973 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:29.278736+0000", 2024-09-20T22:55:04.974 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000115085 2024-09-20T22:55:04.974 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:55:04.974 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:55:04.974 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:55:04.974 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:04.974 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:04.974 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(osd.0.0:1215 7.9 e588/426 7:91f2d47c:::smithi08477203-8:ef v 588'801, mlcod=588'794)", 2024-09-20T22:55:04.974 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:51:08.278858+0000", 2024-09-20T22:55:04.974 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 236.54659087900001, 2024-09-20T22:55:04.974 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0079378230000000001, 2024-09-20T22:55:04.974 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:55:04.974 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:55:04.974 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:55:04.974 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:55:04.975 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:04.975 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:55:04.975 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:08.278858+0000", 2024-09-20T22:55:04.975 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:55:04.975 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:04.975 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:04.975 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:55:04.975 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:08.278858+0000", 2024-09-20T22:55:04.975 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:55:04.975 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:04.975 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:04.975 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:55:04.975 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:08.278859+0000", 2024-09-20T22:55:04.975 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4249999999999999e-06 2024-09-20T22:55:04.975 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:04.976 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:04.976 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:55:04.976 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:08.278875+0000", 2024-09-20T22:55:04.976 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.5698000000000001e-05 2024-09-20T22:55:04.976 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:04.976 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:04.976 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:55:04.976 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:08.278888+0000", 2024-09-20T22:55:04.976 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3377000000000001e-05 2024-09-20T22:55:04.976 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:04.976 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:04.976 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:55:04.976 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:08.278895+0000", 2024-09-20T22:55:04.977 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.1990000000000003e-06 2024-09-20T22:55:04.977 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:04.977 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:04.977 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:55:04.977 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:08.278963+0000", 2024-09-20T22:55:04.977 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.8042999999999999e-05 2024-09-20T22:55:04.977 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:04.977 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:04.977 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:55:04.977 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:08.278994+0000", 2024-09-20T22:55:04.977 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.1343999999999997e-05 2024-09-20T22:55:04.977 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:04.977 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:04.977 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:55:04.977 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:08.286741+0000", 2024-09-20T22:55:04.978 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0077470780000000001 2024-09-20T22:55:04.978 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:04.978 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:04.978 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:55:04.978 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:08.286796+0000", 2024-09-20T22:55:04.978 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.4659e-05 2024-09-20T22:55:04.978 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:55:04.978 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:55:04.978 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:55:04.978 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:04.978 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:04.978 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(osd.0.0:1265 7.9 e596/426 7:907aec0e:::smithi08477203-3:aa v 596'852, mlcod=593'848)", 2024-09-20T22:55:04.979 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:51:16.390167+0000", 2024-09-20T22:55:04.979 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 228.43528161200001, 2024-09-20T22:55:04.979 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0085014630000000008, 2024-09-20T22:55:04.979 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:55:04.979 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:55:04.979 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:55:04.979 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:55:04.979 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:04.979 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:55:04.979 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:16.390167+0000", 2024-09-20T22:55:04.979 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:55:04.979 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:04.979 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:04.979 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:55:04.979 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:16.390167+0000", 2024-09-20T22:55:04.980 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:55:04.980 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:04.980 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:04.980 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:55:04.980 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:16.390167+0000", 2024-09-20T22:55:04.980 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9999999999999999e-07 2024-09-20T22:55:04.980 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:04.980 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:04.980 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:55:04.980 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:16.390172+0000", 2024-09-20T22:55:04.980 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.7480000000000001e-06 2024-09-20T22:55:04.980 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:04.980 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:04.980 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:55:04.980 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:16.390188+0000", 2024-09-20T22:55:04.981 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.6246999999999999e-05 2024-09-20T22:55:04.981 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:04.981 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:04.981 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:55:04.981 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:16.390196+0000", 2024-09-20T22:55:04.981 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.4549999999999998e-06 2024-09-20T22:55:04.981 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:04.981 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:04.981 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:55:04.981 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:16.390261+0000", 2024-09-20T22:55:04.981 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.4888000000000006e-05 2024-09-20T22:55:04.981 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:04.981 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:04.982 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:55:04.982 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:16.390279+0000", 2024-09-20T22:55:04.982 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.8553999999999999e-05 2024-09-20T22:55:04.982 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:04.982 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:04.982 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:55:04.982 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:16.398617+0000", 2024-09-20T22:55:04.982 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0083380550000000005 2024-09-20T22:55:04.982 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:04.982 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:04.982 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:55:04.982 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:16.398669+0000", 2024-09-20T22:55:04.982 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.1316000000000001e-05 2024-09-20T22:55:04.982 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:55:04.983 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:55:04.983 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:55:04.983 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:04.983 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:04.983 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(osd.0.0:1266 7.9 e596/426 7:9eac4560:::smithi08477203-30:ae v 596'854, mlcod=593'848)", 2024-09-20T22:55:04.983 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:51:16.392066+0000", 2024-09-20T22:55:04.983 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 228.433382921, 2024-09-20T22:55:04.983 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0099900849999999992, 2024-09-20T22:55:04.983 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:55:04.983 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:55:04.983 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:55:04.983 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:55:04.983 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:04.983 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:55:04.984 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:16.392066+0000", 2024-09-20T22:55:04.984 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:55:04.984 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:04.984 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:04.984 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:55:04.984 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:16.392066+0000", 2024-09-20T22:55:04.984 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:55:04.984 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:04.984 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:04.984 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:55:04.984 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:16.392067+0000", 2024-09-20T22:55:04.984 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.2600000000000001e-07 2024-09-20T22:55:04.984 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:04.984 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:04.984 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:55:04.984 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:16.392073+0000", 2024-09-20T22:55:04.985 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.6989999999999997e-06 2024-09-20T22:55:04.985 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:04.985 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:04.985 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:55:04.985 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:16.392082+0000", 2024-09-20T22:55:04.985 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.8200000000000003e-06 2024-09-20T22:55:04.985 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:04.985 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:04.985 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:55:04.985 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:16.392090+0000", 2024-09-20T22:55:04.985 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.7160000000000003e-06 2024-09-20T22:55:04.985 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:04.985 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:04.985 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:55:04.985 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:16.392637+0000", 2024-09-20T22:55:04.985 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00054682600000000004 2024-09-20T22:55:04.986 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:04.986 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:04.986 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:55:04.986 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:16.392657+0000", 2024-09-20T22:55:04.986 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9956e-05 2024-09-20T22:55:04.986 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:04.986 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:04.986 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:55:04.986 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:16.401999+0000", 2024-09-20T22:55:04.986 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0093422560000000002 2024-09-20T22:55:04.986 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:04.986 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:04.986 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:55:04.986 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:16.402056+0000", 2024-09-20T22:55:04.986 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.6986e-05 2024-09-20T22:55:04.987 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:55:04.987 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:55:04.987 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:55:04.987 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:04.987 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:04.987 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(osd.0.0:1267 7.4 e596/460 7:24b1a612:::smithi08477203-43:aa v 596'321, mlcod=592'317)", 2024-09-20T22:55:04.987 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:51:16.392654+0000", 2024-09-20T22:55:04.987 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 228.43279518400001, 2024-09-20T22:55:04.987 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0093429540000000005, 2024-09-20T22:55:04.987 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:55:04.987 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:55:04.987 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:55:04.987 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:55:04.988 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:04.988 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:55:04.988 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:16.392654+0000", 2024-09-20T22:55:04.988 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:55:04.988 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:04.988 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:04.988 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:55:04.988 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:16.392654+0000", 2024-09-20T22:55:04.988 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:55:04.988 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:04.988 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:04.988 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:55:04.988 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:16.392654+0000", 2024-09-20T22:55:04.988 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.4300000000000002e-07 2024-09-20T22:55:04.989 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:04.989 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:04.989 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:55:04.989 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:16.392664+0000", 2024-09-20T22:55:04.989 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.2830000000000007e-06 2024-09-20T22:55:04.989 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:04.989 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:04.989 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:55:04.989 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:16.392671+0000", 2024-09-20T22:55:04.989 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.3499999999999999e-06 2024-09-20T22:55:04.989 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:04.989 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:04.989 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:55:04.989 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:16.392677+0000", 2024-09-20T22:55:04.989 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.4590000000000003e-06 2024-09-20T22:55:04.989 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:04.990 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:04.990 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:55:04.990 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:16.392753+0000", 2024-09-20T22:55:04.990 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.5387000000000002e-05 2024-09-20T22:55:04.990 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:04.990 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:04.990 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:55:04.990 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:16.392776+0000", 2024-09-20T22:55:04.990 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.3703e-05 2024-09-20T22:55:04.990 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:04.990 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:04.990 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:55:04.990 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:16.401944+0000", 2024-09-20T22:55:04.991 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0091675100000000002 2024-09-20T22:55:04.991 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:04.991 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:04.991 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:55:04.991 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:16.401996+0000", 2024-09-20T22:55:04.991 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.2519000000000003e-05 2024-09-20T22:55:04.991 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:55:04.991 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:55:04.991 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:55:04.991 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:04.991 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:04.991 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(osd.0.0:1268 7.4 e596/460 7:2f8647a2:::smithi08477203-41:aa v 596'323, mlcod=592'317)", 2024-09-20T22:55:04.991 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:51:16.392986+0000", 2024-09-20T22:55:04.991 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 228.432463079, 2024-09-20T22:55:04.991 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0091905690000000009, 2024-09-20T22:55:04.992 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:55:04.992 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:55:04.992 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:55:04.992 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:55:04.992 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:04.992 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:55:04.992 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:16.392986+0000", 2024-09-20T22:55:04.992 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:55:04.992 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:04.992 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:04.992 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:55:04.992 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:16.392986+0000", 2024-09-20T22:55:04.992 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:55:04.992 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:04.992 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:04.993 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:55:04.993 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:16.392986+0000", 2024-09-20T22:55:04.993 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.1099999999999995e-07 2024-09-20T22:55:04.993 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:04.993 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:04.993 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:55:04.993 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:16.392992+0000", 2024-09-20T22:55:04.993 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.7089999999999999e-06 2024-09-20T22:55:04.993 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:04.993 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:04.993 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:55:04.993 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:16.393000+0000", 2024-09-20T22:55:04.993 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.5229999999999995e-06 2024-09-20T22:55:04.993 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:04.993 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:04.994 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:55:04.994 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:16.393007+0000", 2024-09-20T22:55:04.994 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.5169999999999996e-06 2024-09-20T22:55:04.994 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:04.994 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:04.994 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:55:04.994 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:16.393223+0000", 2024-09-20T22:55:04.994 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00021552600000000001 2024-09-20T22:55:04.994 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:04.994 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:04.994 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:55:04.994 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:16.393245+0000", 2024-09-20T22:55:04.994 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.2172e-05 2024-09-20T22:55:04.995 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:04.995 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:04.995 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:55:04.995 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:16.402143+0000", 2024-09-20T22:55:04.995 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.008898418 2024-09-20T22:55:04.995 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:04.995 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:04.995 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:55:04.995 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:16.402176+0000", 2024-09-20T22:55:04.995 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.3093000000000003e-05 2024-09-20T22:55:04.995 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:55:04.995 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:55:04.995 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:55:04.995 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:04.995 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:04.996 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:6000 7.9 e604/602 7:9eac4560:::smithi08477203-30:head v 604'886, mlcod=604'882)", 2024-09-20T22:55:04.996 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:51:24.304331+0000", 2024-09-20T22:55:04.996 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 220.52111804699999, 2024-09-20T22:55:04.996 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.014326934, 2024-09-20T22:55:04.996 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:55:04.996 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:55:04.996 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:55:04.996 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:55:04.996 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:04.996 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:55:04.996 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:24.304331+0000", 2024-09-20T22:55:04.996 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:55:04.996 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:04.997 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:04.997 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:55:04.997 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:24.304331+0000", 2024-09-20T22:55:04.997 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:55:04.997 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:04.997 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:04.997 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:55:04.997 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:24.304332+0000", 2024-09-20T22:55:04.997 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.8400000000000002e-07 2024-09-20T22:55:04.997 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:04.997 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:04.997 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:55:04.997 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:24.310538+0000", 2024-09-20T22:55:04.998 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0062066710000000004 2024-09-20T22:55:04.998 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:04.998 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:04.998 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:55:04.998 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:24.310568+0000", 2024-09-20T22:55:04.998 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.9283000000000001e-05 2024-09-20T22:55:04.998 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:04.998 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:04.998 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:55:04.998 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:24.310576+0000", 2024-09-20T22:55:04.998 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.3639999999999999e-06 2024-09-20T22:55:04.998 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:04.998 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:04.998 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:55:04.999 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:24.310641+0000", 2024-09-20T22:55:04.999 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.4690000000000006e-05 2024-09-20T22:55:04.999 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:04.999 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:04.999 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:55:04.999 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:24.310665+0000", 2024-09-20T22:55:04.999 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.4635999999999999e-05 2024-09-20T22:55:04.999 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:04.999 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:04.999 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:55:04.999 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:24.318613+0000", 2024-09-20T22:55:04.999 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0079475429999999996 2024-09-20T22:55:04.999 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:04.999 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:04.999 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:55:04.999 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:24.318658+0000", 2024-09-20T22:55:05.000 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.4762999999999998e-05 2024-09-20T22:55:05.000 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:55:05.000 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:55:05.000 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:55:05.000 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:05.000 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:05.000 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:6253 7.9 e629/602 7:91f2d47c:::smithi08477203-8:head v 629'948, mlcod=627'945)", 2024-09-20T22:55:05.000 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:51:49.625685+0000", 2024-09-20T22:55:05.000 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 195.199763819, 2024-09-20T22:55:05.000 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.017501778999999999, 2024-09-20T22:55:05.000 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:55:05.000 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:55:05.000 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:55:05.000 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:55:05.001 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:05.001 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:55:05.001 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:49.625685+0000", 2024-09-20T22:55:05.001 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:55:05.001 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:05.001 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:05.001 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:55:05.001 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:49.625685+0000", 2024-09-20T22:55:05.001 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:55:05.001 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:05.001 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:05.001 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:55:05.001 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:49.625686+0000", 2024-09-20T22:55:05.002 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.457e-06 2024-09-20T22:55:05.002 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:05.002 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:05.002 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:55:05.002 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:49.632417+0000", 2024-09-20T22:55:05.002 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.006730453 2024-09-20T22:55:05.002 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:05.002 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:05.002 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:55:05.002 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:49.632440+0000", 2024-09-20T22:55:05.002 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.2999000000000001e-05 2024-09-20T22:55:05.002 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:05.002 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:05.002 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:55:05.003 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:49.632450+0000", 2024-09-20T22:55:05.003 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0210999999999999e-05 2024-09-20T22:55:05.003 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:05.003 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:05.003 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:55:05.003 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:49.632524+0000", 2024-09-20T22:55:05.003 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.4234999999999997e-05 2024-09-20T22:55:05.003 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:05.003 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:05.003 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:55:05.003 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:49.632549+0000", 2024-09-20T22:55:05.003 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.4572000000000001e-05 2024-09-20T22:55:05.003 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:05.003 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:05.004 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:55:05.004 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:49.643131+0000", 2024-09-20T22:55:05.004 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.010582446000000001 2024-09-20T22:55:05.004 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:05.004 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:05.004 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:55:05.004 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:49.643187+0000", 2024-09-20T22:55:05.004 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.5405999999999998e-05 2024-09-20T22:55:05.004 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:55:05.004 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:55:05.004 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:55:05.004 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:05.004 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:05.005 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:6360 7.8 e638/430 7:1e9e2524:::smithi08477203-5:head v 638'681, mlcod=637'677)", 2024-09-20T22:55:05.005 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:51:58.286846+0000", 2024-09-20T22:55:05.005 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 186.53860240200001, 2024-09-20T22:55:05.005 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.012344572999999999, 2024-09-20T22:55:05.005 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:55:05.005 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:55:05.005 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:55:05.005 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:55:05.005 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:05.005 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:55:05.005 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:58.286846+0000", 2024-09-20T22:55:05.005 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:55:05.005 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:05.005 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:05.005 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:55:05.006 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:58.286846+0000", 2024-09-20T22:55:05.006 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:55:05.006 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:05.006 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:05.006 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:55:05.006 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:58.286847+0000", 2024-09-20T22:55:05.006 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.3700000000000001e-07 2024-09-20T22:55:05.006 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:05.006 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:05.006 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:55:05.006 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:58.297345+0000", 2024-09-20T22:55:05.006 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.010498535 2024-09-20T22:55:05.006 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:05.007 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:05.007 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:55:05.007 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:58.297359+0000", 2024-09-20T22:55:05.007 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3631999999999999e-05 2024-09-20T22:55:05.007 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:05.007 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:05.007 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:55:05.007 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:58.297365+0000", 2024-09-20T22:55:05.007 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.4489999999999996e-06 2024-09-20T22:55:05.007 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:05.007 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:05.007 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:55:05.007 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:58.297433+0000", 2024-09-20T22:55:05.007 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.7206000000000007e-05 2024-09-20T22:55:05.008 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:05.008 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:05.008 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:55:05.008 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:58.297455+0000", 2024-09-20T22:55:05.008 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.2711999999999999e-05 2024-09-20T22:55:05.008 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:05.008 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:05.008 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:55:05.008 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:58.299091+0000", 2024-09-20T22:55:05.008 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0016354010000000001 2024-09-20T22:55:05.008 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:05.008 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:05.008 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:55:05.008 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:58.299191+0000", 2024-09-20T22:55:05.008 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000100201 2024-09-20T22:55:05.008 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:55:05.009 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:55:05.009 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:55:05.009 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:05.009 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:05.009 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:7267 7.1 7:84a866f5:::smithi08477203-27:head [copy-from ver 1805] snapc 149=[149,148,147,146,145,144,142,141,140,13e,13d,13c,13b,13a,139,138,136,135,132,130,12f,12e,12d,12c,12a,128,127,124,121,11f,112,103,ed] ondisk+write+known_if_redirected+supports_pool_eio e728)", 2024-09-20T22:55:05.009 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:53:28.329140+0000", 2024-09-20T22:55:05.009 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 96.496308690000006, 2024-09-20T22:55:05.009 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.016910652000000002, 2024-09-20T22:55:05.009 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:55:05.009 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:55:05.009 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:55:05.009 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:55:05.009 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:55:05.009 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:55:05.009 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 7267 2024-09-20T22:55:05.010 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:05.010 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:55:05.010 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:05.010 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:55:05.010 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:28.329140+0000", 2024-09-20T22:55:05.010 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:55:05.010 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:05.010 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:05.010 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:55:05.010 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:28.329140+0000", 2024-09-20T22:55:05.010 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:55:05.010 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:05.010 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:05.010 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:55:05.010 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:28.329142+0000", 2024-09-20T22:55:05.011 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.4260000000000002e-06 2024-09-20T22:55:05.011 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:05.011 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:05.011 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:55:05.011 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:28.329152+0000", 2024-09-20T22:55:05.011 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.3519999999999996e-06 2024-09-20T22:55:05.011 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:05.011 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:05.011 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:55:05.011 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:28.329170+0000", 2024-09-20T22:55:05.011 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.7748e-05 2024-09-20T22:55:05.011 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:05.011 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:05.011 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:55:05.012 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:28.329179+0000", 2024-09-20T22:55:05.012 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.6979999999999993e-06 2024-09-20T22:55:05.012 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:05.012 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:05.012 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:55:05.012 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:28.329253+0000", 2024-09-20T22:55:05.012 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.3708000000000006e-05 2024-09-20T22:55:05.012 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:05.012 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:05.012 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:55:05.012 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:28.329354+0000", 2024-09-20T22:55:05.012 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00010073099999999999 2024-09-20T22:55:05.012 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:05.012 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:05.013 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 3,6", 2024-09-20T22:55:05.013 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:28.336766+0000", 2024-09-20T22:55:05.013 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0074122010000000002 2024-09-20T22:55:05.013 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:05.013 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:05.013 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:55:05.013 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:28.345430+0000", 2024-09-20T22:55:05.013 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0086638119999999999 2024-09-20T22:55:05.013 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:05.013 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:05.013 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:55:05.013 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:28.345800+0000", 2024-09-20T22:55:05.013 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00037052199999999997 2024-09-20T22:55:05.013 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:05.013 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:05.014 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:55:05.014 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:28.345963+0000", 2024-09-20T22:55:05.014 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000163077 2024-09-20T22:55:05.014 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:05.014 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:05.014 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:55:05.014 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:28.346010+0000", 2024-09-20T22:55:05.014 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.6338999999999998e-05 2024-09-20T22:55:05.014 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:05.014 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:05.014 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:55:05.014 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:28.346051+0000", 2024-09-20T22:55:05.015 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.1038000000000002e-05 2024-09-20T22:55:05.015 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:55:05.015 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:55:05.015 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:55:05.015 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:05.015 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:05.015 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:7268 7.1 7:84a866f5:::smithi08477203-27:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e728)", 2024-09-20T22:55:05.015 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:53:28.329202+0000", 2024-09-20T22:55:05.015 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 96.496246927000001, 2024-09-20T22:55:05.015 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.012525870999999999, 2024-09-20T22:55:05.015 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:55:05.015 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:55:05.015 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:55:05.015 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:55:05.015 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:55:05.016 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:55:05.016 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 7268 2024-09-20T22:55:05.016 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:05.016 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:55:05.016 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:05.016 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:55:05.016 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:28.329202+0000", 2024-09-20T22:55:05.016 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:55:05.016 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:05.016 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:05.016 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:55:05.016 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:28.329202+0000", 2024-09-20T22:55:05.016 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:55:05.016 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:05.016 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:05.017 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:55:05.017 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:28.329202+0000", 2024-09-20T22:55:05.017 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.7400000000000003e-07 2024-09-20T22:55:05.017 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:05.017 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:05.017 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:55:05.017 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:28.329206+0000", 2024-09-20T22:55:05.017 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.1e-06 2024-09-20T22:55:05.017 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:05.017 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:05.017 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:55:05.017 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:28.329213+0000", 2024-09-20T22:55:05.017 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.3520000000000001e-06 2024-09-20T22:55:05.017 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:05.017 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:05.018 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:55:05.018 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:28.329229+0000", 2024-09-20T22:55:05.018 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.6169999999999999e-05 2024-09-20T22:55:05.018 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:05.018 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:05.018 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:55:05.018 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:28.329486+0000", 2024-09-20T22:55:05.018 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000256703 2024-09-20T22:55:05.018 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:05.018 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:05.018 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for blocked object", 2024-09-20T22:55:05.018 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:28.329541+0000", 2024-09-20T22:55:05.018 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.5158000000000003e-05 2024-09-20T22:55:05.018 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:05.018 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:05.019 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:55:05.019 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:28.341541+0000", 2024-09-20T22:55:05.019 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.011999774 2024-09-20T22:55:05.019 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:05.019 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:05.019 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:55:05.019 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:28.341634+0000", 2024-09-20T22:55:05.019 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.3271999999999997e-05 2024-09-20T22:55:05.019 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:05.019 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:05.019 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:55:05.019 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:28.341728+0000", 2024-09-20T22:55:05.019 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.3667999999999998e-05 2024-09-20T22:55:05.019 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:55:05.019 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:55:05.020 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:55:05.020 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:05.020 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:05.020 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:7387 7.9 e743/602 7:907aec0e:::smithi08477203-3:head v 743'1242, mlcod=741'1234)", 2024-09-20T22:55:05.020 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:53:43.309715+0000", 2024-09-20T22:55:05.020 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 81.515733932000003, 2024-09-20T22:55:05.020 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.030438433000000001, 2024-09-20T22:55:05.020 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:55:05.020 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:55:05.020 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:55:05.020 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:55:05.020 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:05.020 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:55:05.020 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:43.309715+0000", 2024-09-20T22:55:05.020 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:55:05.021 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:05.021 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:05.021 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:55:05.021 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:43.309715+0000", 2024-09-20T22:55:05.021 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:55:05.021 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:05.021 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:05.021 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:55:05.021 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:43.309716+0000", 2024-09-20T22:55:05.021 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.023e-06 2024-09-20T22:55:05.021 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:05.021 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:05.021 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:55:05.022 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:43.336825+0000", 2024-09-20T22:55:05.022 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.027108795000000002 2024-09-20T22:55:05.022 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:05.022 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:05.022 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:55:05.022 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:43.336837+0000", 2024-09-20T22:55:05.022 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2751e-05 2024-09-20T22:55:05.022 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:05.022 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:05.022 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:55:05.022 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:43.336843+0000", 2024-09-20T22:55:05.022 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.8969999999999996e-06 2024-09-20T22:55:05.022 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:05.022 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:05.022 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:55:05.023 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:43.336894+0000", 2024-09-20T22:55:05.023 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.1201e-05 2024-09-20T22:55:05.023 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:05.023 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:05.023 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:55:05.023 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:43.336911+0000", 2024-09-20T22:55:05.023 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.6062e-05 2024-09-20T22:55:05.023 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:05.023 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:05.023 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:55:05.023 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:43.340108+0000", 2024-09-20T22:55:05.023 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0031979109999999999 2024-09-20T22:55:05.023 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:05.024 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:05.024 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:55:05.024 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:43.340153+0000", 2024-09-20T22:55:05.024 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.4793000000000003e-05 2024-09-20T22:55:05.024 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:55:05.024 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:55:05.024 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:55:05.024 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:05.024 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:05.024 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:7392 7.3 e743/464 7:cae5792a:::smithi08477203-32:head v 743'338, mlcod=738'334)", 2024-09-20T22:55:05.024 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:53:43.328452+0000", 2024-09-20T22:55:05.024 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 81.496997113999996, 2024-09-20T22:55:05.024 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.013073611000000001, 2024-09-20T22:55:05.024 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:55:05.024 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:55:05.025 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:55:05.025 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:55:05.025 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:05.025 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:55:05.025 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:43.328452+0000", 2024-09-20T22:55:05.025 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:55:05.025 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:05.025 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:05.025 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:55:05.025 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:43.328452+0000", 2024-09-20T22:55:05.025 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:55:05.025 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:05.025 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:05.025 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:55:05.026 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:43.328452+0000", 2024-09-20T22:55:05.026 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.6199999999999999e-07 2024-09-20T22:55:05.026 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:05.026 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:05.026 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:55:05.026 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:43.339025+0000", 2024-09-20T22:55:05.026 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.010572936999999999 2024-09-20T22:55:05.026 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:05.026 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:05.026 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:55:05.026 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:43.339079+0000", 2024-09-20T22:55:05.026 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.3888999999999999e-05 2024-09-20T22:55:05.026 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:05.026 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:05.026 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:55:05.027 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:43.339087+0000", 2024-09-20T22:55:05.027 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.2330000000000006e-06 2024-09-20T22:55:05.027 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:05.027 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:05.027 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:55:05.027 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:43.339211+0000", 2024-09-20T22:55:05.027 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00012425199999999999 2024-09-20T22:55:05.027 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:05.027 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:05.027 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:55:05.027 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:43.339229+0000", 2024-09-20T22:55:05.027 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.8403999999999999e-05 2024-09-20T22:55:05.027 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:05.028 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:05.028 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:55:05.028 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:43.341475+0000", 2024-09-20T22:55:05.028 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0022457390000000001 2024-09-20T22:55:05.028 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:05.028 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:05.028 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:55:05.028 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:43.341525+0000", 2024-09-20T22:55:05.028 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.9994999999999998e-05 2024-09-20T22:55:05.028 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:55:05.028 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:55:05.028 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:55:05.028 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:05.028 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:05.028 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:7549 7.4 e755/602 7:2f8647a2:::smithi08477203-41:head v 755'514, mlcod=754'510)", 2024-09-20T22:55:05.029 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:53:55.515416+0000", 2024-09-20T22:55:05.029 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 69.310032747999998, 2024-09-20T22:55:05.029 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0096741410000000007, 2024-09-20T22:55:05.029 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:55:05.029 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:55:05.029 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:55:05.029 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:55:05.029 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:05.029 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:55:05.029 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:55.515416+0000", 2024-09-20T22:55:05.029 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:55:05.029 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:05.029 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:05.029 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:55:05.030 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:55.515416+0000", 2024-09-20T22:55:05.030 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:55:05.030 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:05.030 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:05.030 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:55:05.030 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:55.515418+0000", 2024-09-20T22:55:05.030 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.1169999999999998e-06 2024-09-20T22:55:05.030 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:05.030 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:05.030 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:55:05.030 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:55.520055+0000", 2024-09-20T22:55:05.030 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0046364709999999996 2024-09-20T22:55:05.031 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:05.031 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:05.031 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:55:05.031 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:55.520074+0000", 2024-09-20T22:55:05.031 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9219e-05 2024-09-20T22:55:05.031 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:05.031 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:05.031 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:55:05.031 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:55.520084+0000", 2024-09-20T22:55:05.031 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0064000000000001e-05 2024-09-20T22:55:05.031 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:05.031 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:05.031 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:55:05.031 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:55.520140+0000", 2024-09-20T22:55:05.032 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.6444999999999999e-05 2024-09-20T22:55:05.032 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:05.032 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:05.032 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:55:05.032 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:55.520158+0000", 2024-09-20T22:55:05.032 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.7720999999999999e-05 2024-09-20T22:55:05.032 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:05.032 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:05.032 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:55:05.032 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:55.525049+0000", 2024-09-20T22:55:05.032 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0048913890000000003 2024-09-20T22:55:05.032 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:05.032 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:05.033 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:55:05.033 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:55.525090+0000", 2024-09-20T22:55:05.033 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.0714999999999999e-05 2024-09-20T22:55:05.033 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:55:05.033 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:55:05.033 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:55:05.033 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:05.033 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:05.033 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:8049 7.9 e810/779 7:907aec0e:::smithi08477203-3:head v 810'1428, mlcod=808'1424)", 2024-09-20T22:55:05.033 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:54:50.354916+0000", 2024-09-20T22:55:05.033 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 14.470533093, 2024-09-20T22:55:05.033 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.018163367999999999, 2024-09-20T22:55:05.033 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:55:05.033 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:55:05.034 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:55:05.034 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:55:05.034 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:05.034 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:55:05.034 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:54:50.354916+0000", 2024-09-20T22:55:05.034 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:55:05.034 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:05.034 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:05.034 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:55:05.034 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:54:50.354916+0000", 2024-09-20T22:55:05.034 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:55:05.034 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:05.034 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:05.035 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:55:05.035 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:54:50.354916+0000", 2024-09-20T22:55:05.035 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.7099999999999998e-07 2024-09-20T22:55:05.035 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:05.035 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:05.035 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:55:05.035 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:54:50.359771+0000", 2024-09-20T22:55:05.035 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0048552090000000001 2024-09-20T22:55:05.035 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:05.035 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:05.035 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:55:05.035 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:54:50.359786+0000", 2024-09-20T22:55:05.035 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4430000000000001e-05 2024-09-20T22:55:05.035 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:05.036 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:05.036 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:55:05.036 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:54:50.359796+0000", 2024-09-20T22:55:05.036 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0539e-05 2024-09-20T22:55:05.036 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:05.036 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:05.036 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:55:05.036 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:54:50.359882+0000", 2024-09-20T22:55:05.036 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.5795000000000001e-05 2024-09-20T22:55:05.036 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:05.036 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:05.036 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:55:05.036 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:54:50.359906+0000", 2024-09-20T22:55:05.036 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.4528e-05 2024-09-20T22:55:05.037 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:05.037 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:05.037 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:55:05.037 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:54:50.372996+0000", 2024-09-20T22:55:05.037 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.013089987000000001 2024-09-20T22:55:05.037 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:05.037 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:05.037 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:55:05.037 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:54:50.373079+0000", 2024-09-20T22:55:05.037 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.2608999999999995e-05 2024-09-20T22:55:05.037 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:55:05.037 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:55:05.037 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:55:05.037 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:05.037 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:05.038 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:8056 7.9 e810/779 7:9970b578:::smithi08477203-15:head v 810'1429, mlcod=808'1424)", 2024-09-20T22:55:05.038 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:54:50.364589+0000", 2024-09-20T22:55:05.038 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 14.460859362000001, 2024-09-20T22:55:05.038 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0085585869999999994, 2024-09-20T22:55:05.038 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:55:05.038 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:55:05.038 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:55:05.038 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:55:05.038 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:05.038 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:55:05.038 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:54:50.364589+0000", 2024-09-20T22:55:05.038 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:55:05.038 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:05.039 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:05.039 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:55:05.039 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:54:50.364589+0000", 2024-09-20T22:55:05.039 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:55:05.039 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:05.039 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:05.039 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:55:05.039 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:54:50.364599+0000", 2024-09-20T22:55:05.039 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.2639999999999997e-06 2024-09-20T22:55:05.039 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:05.039 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:05.039 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:55:05.039 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:54:50.365506+0000", 2024-09-20T22:55:05.039 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00090696699999999995 2024-09-20T22:55:05.040 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:05.040 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:05.040 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:55:05.040 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:54:50.365528+0000", 2024-09-20T22:55:05.040 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.2705999999999999e-05 2024-09-20T22:55:05.040 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:05.040 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:05.040 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:55:05.040 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:54:50.365538+0000", 2024-09-20T22:55:05.040 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.9709999999999999e-06 2024-09-20T22:55:05.040 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:05.040 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:05.040 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:55:05.040 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:54:50.365607+0000", 2024-09-20T22:55:05.040 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.8423000000000005e-05 2024-09-20T22:55:05.041 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:05.041 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:05.041 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:55:05.041 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:54:50.365626+0000", 2024-09-20T22:55:05.041 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9028000000000001e-05 2024-09-20T22:55:05.041 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:05.041 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:05.041 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:55:05.041 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:54:50.373126+0000", 2024-09-20T22:55:05.041 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00749996 2024-09-20T22:55:05.041 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:05.041 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:05.041 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:55:05.041 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:54:50.373148+0000", 2024-09-20T22:55:05.042 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.2268000000000001e-05 2024-09-20T22:55:05.042 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:55:05.042 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:55:05.042 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:55:05.042 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:55:05.042 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:55:05.042 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-20T22:55:05.042 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:55:05.064 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:05.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:05.065 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:55:05.166 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:55:05.186 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:05.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:05.187 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:55:05.225 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:55:05.287 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:55:05.308 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:05.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:05.309 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:55:05.347 INFO:tasks.rados.rados.0.smithi084.stdout:3273: done (3 left) 2024-09-20T22:55:05.347 INFO:tasks.rados.rados.0.smithi084.stdout:3274: done (2 left) 2024-09-20T22:55:05.348 INFO:tasks.rados.rados.0.smithi084.stdout:3275: done (1 left) 2024-09-20T22:55:05.348 INFO:tasks.rados.rados.0.smithi084.stdout:3276: done (0 left) 2024-09-20T22:55:05.348 INFO:tasks.rados.rados.0.smithi084.stdout:3277: read oid 15 snap 341 2024-09-20T22:55:05.348 INFO:tasks.rados.rados.0.smithi084.stdout:3277: expect (ObjNum 1012 snap 333 seq_num 1012) 2024-09-20T22:55:05.348 INFO:tasks.rados.rados.0.smithi084.stdout:3278: write oid 4 current snap is 378 2024-09-20T22:55:05.348 INFO:tasks.rados.rados.0.smithi084.stdout:3278: seq_num 1111 ranges {672577=589177,1913024=427715} 2024-09-20T22:55:05.355 INFO:tasks.rados.rados.0.smithi084.stdout:3278: writing smithi08477203-4 from 672577 to 1261754 tid 1 2024-09-20T22:55:05.361 INFO:tasks.rados.rados.0.smithi084.stdout:3278: writing smithi08477203-4 from 1913024 to 2340739 tid 2 2024-09-20T22:55:05.361 INFO:tasks.rados.rados.0.smithi084.stdout:3279: rollback oid 33 current snap is 378 2024-09-20T22:55:05.361 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 33 to 316 2024-09-20T22:55:05.361 INFO:tasks.rados.rados.0.smithi084.stdout:3277: expect (ObjNum 1012 snap 333 seq_num 1012) 2024-09-20T22:55:05.410 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:55:05.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:05.432 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:05.432 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:55:05.533 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:55:05.554 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:05.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:05.555 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:55:05.576 INFO:tasks.rados.rados.0.smithi084.stdout:3278: finishing write tid 1 to smithi08477203-4 2024-09-20T22:55:05.577 INFO:tasks.rados.rados.0.smithi084.stdout:3277: done (2 left) 2024-09-20T22:55:05.577 INFO:tasks.rados.rados.0.smithi084.stdout:3280: read oid 11 snap -1 2024-09-20T22:55:05.577 INFO:tasks.rados.rados.0.smithi084.stdout:3280: expect (ObjNum 991 snap 325 seq_num 991) 2024-09-20T22:55:05.577 INFO:tasks.rados.rados.0.smithi084.stdout:3281: copy_from oid 44 from oid 50 current snap is 378 2024-09-20T22:55:05.577 INFO:tasks.rados.rados.0.smithi084.stdout:3279: finishing rollback tid 1 to smithi08477203-33 2024-09-20T22:55:05.577 INFO:tasks.rados.rados.0.smithi084.stdout:3278: finishing write tid 2 to smithi08477203-4 2024-09-20T22:55:05.577 INFO:tasks.rados.rados.0.smithi084.stdout:3278: finishing write tid 3 to smithi08477203-4 2024-09-20T22:55:05.577 INFO:tasks.rados.rados.0.smithi084.stdout:3279: finishing rollback tid 2 to smithi08477203-33 2024-09-20T22:55:05.577 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 33 v 1916 (ObjNum 980 snap 316 seq_num 980) dirty exists 2024-09-20T22:55:05.577 INFO:tasks.rados.rados.0.smithi084.stdout:3278: finishing write tid 4 to smithi08477203-4 2024-09-20T22:55:05.577 INFO:tasks.rados.rados.0.smithi084.stdout:3278: oid 4 updating version 0 to 2637 2024-09-20T22:55:05.577 INFO:tasks.rados.rados.0.smithi084.stdout:3278: oid 4 updating version 2637 to 2639 2024-09-20T22:55:05.577 INFO:tasks.rados.rados.0.smithi084.stdout:3278: oid 4 version 2639 is already newer than 2638 2024-09-20T22:55:05.577 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 4 v 2639 (ObjNum 1111 snap 378 seq_num 1111) dirty exists 2024-09-20T22:55:05.578 INFO:tasks.rados.rados.0.smithi084.stdout:3278: left oid 4 (ObjNum 1111 snap 378 seq_num 1111) 2024-09-20T22:55:05.578 INFO:tasks.rados.rados.0.smithi084.stdout:3278: done (3 left) 2024-09-20T22:55:05.578 INFO:tasks.rados.rados.0.smithi084.stdout:3279: done (2 left) 2024-09-20T22:55:05.578 INFO:tasks.rados.rados.0.smithi084.stdout:3282: write (excl) oid 22 current snap is 378 2024-09-20T22:55:05.578 INFO:tasks.rados.rados.0.smithi084.stdout:3282: seq_num 1112 ranges {689384=664426,1971506=687726} 2024-09-20T22:55:05.584 INFO:tasks.rados.rados.0.smithi084.stdout:3282: writing smithi08477203-22 from 689384 to 1353810 tid 1 2024-09-20T22:55:05.591 INFO:tasks.rados.rados.0.smithi084.stdout:3282: writing smithi08477203-22 from 1971506 to 2659232 tid 2 2024-09-20T22:55:05.592 INFO:tasks.rados.rados.0.smithi084.stdout:3283: write oid 20 current snap is 378 2024-09-20T22:55:05.592 INFO:tasks.rados.rados.0.smithi084.stdout:3283: seq_num 1113 ranges {706191=739675,1913635=431384} 2024-09-20T22:55:05.599 INFO:tasks.rados.rados.0.smithi084.stdout:3283: writing smithi08477203-20 from 706191 to 1445866 tid 1 2024-09-20T22:55:05.605 INFO:tasks.rados.rados.0.smithi084.stdout:3283: writing smithi08477203-20 from 1913635 to 2345019 tid 2 2024-09-20T22:55:05.605 INFO:tasks.rados.rados.0.smithi084.stdout:3284: read oid 7 snap 337 2024-09-20T22:55:05.605 INFO:tasks.rados.rados.0.smithi084.stdout:3284: expect (ObjNum 996 snap 326 seq_num 996) 2024-09-20T22:55:05.605 INFO:tasks.rados.rados.0.smithi084.stdout:3285: write oid 31 current snap is 378 2024-09-20T22:55:05.605 INFO:tasks.rados.rados.0.smithi084.stdout:3285: seq_num 1114 ranges {722998=414924,1572117=575042} 2024-09-20T22:55:05.611 INFO:tasks.rados.rados.0.smithi084.stdout:3285: writing smithi08477203-31 from 722998 to 1137922 tid 1 2024-09-20T22:55:05.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:05 smithi084 ceph-mon[29774]: pgmap v1137: 158 pgs: 1 active+clean+snaptrim_wait, 1 peering, 4 active+clean+snaptrim, 152 active+clean; 506 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 24 MiB/s rd, 804 KiB/s wr, 25 op/s 2024-09-20T22:55:05.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:05 smithi084 ceph-mon[29774]: 4.1c scrub starts 2024-09-20T22:55:05.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:05 smithi084 ceph-mon[29774]: 4.1c scrub ok 2024-09-20T22:55:05.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:05 smithi084 ceph-mon[29774]: osdmap e824: 8 total, 8 up, 8 in 2024-09-20T22:55:05.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:05 smithi084 ceph-mon[29774]: 1.5 scrub starts 2024-09-20T22:55:05.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:05 smithi084 ceph-mon[29774]: 1.5 scrub ok 2024-09-20T22:55:05.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:05 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:55:05.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:05 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:55:05.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:05 smithi084 ceph-mon[40024]: pgmap v1137: 158 pgs: 1 active+clean+snaptrim_wait, 1 peering, 4 active+clean+snaptrim, 152 active+clean; 506 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 24 MiB/s rd, 804 KiB/s wr, 25 op/s 2024-09-20T22:55:05.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:05 smithi084 ceph-mon[40024]: 4.1c scrub starts 2024-09-20T22:55:05.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:05 smithi084 ceph-mon[40024]: 4.1c scrub ok 2024-09-20T22:55:05.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:05 smithi084 ceph-mon[40024]: osdmap e824: 8 total, 8 up, 8 in 2024-09-20T22:55:05.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:05 smithi084 ceph-mon[40024]: 1.5 scrub starts 2024-09-20T22:55:05.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:05 smithi084 ceph-mon[40024]: 1.5 scrub ok 2024-09-20T22:55:05.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:05 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:55:05.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:05 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:55:05.617 INFO:tasks.rados.rados.0.smithi084.stdout:3285: writing smithi08477203-31 from 1572117 to 2147159 tid 2 2024-09-20T22:55:05.617 INFO:tasks.rados.rados.0.smithi084.stdout:3286: copy_from oid 26 from oid 10 current snap is 378 2024-09-20T22:55:05.617 INFO:tasks.rados.rados.0.smithi084.stdout:3280: expect (ObjNum 991 snap 325 seq_num 991) 2024-09-20T22:55:05.657 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:55:05.678 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:05.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:05.679 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:55:05.744 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:55:05.744 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:55:05.749 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:05 smithi154 ceph-mon[33264]: pgmap v1137: 158 pgs: 1 active+clean+snaptrim_wait, 1 peering, 4 active+clean+snaptrim, 152 active+clean; 506 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 24 MiB/s rd, 804 KiB/s wr, 25 op/s 2024-09-20T22:55:05.749 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:05 smithi154 ceph-mon[33264]: 4.1c scrub starts 2024-09-20T22:55:05.749 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:05 smithi154 ceph-mon[33264]: 4.1c scrub ok 2024-09-20T22:55:05.749 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:05 smithi154 ceph-mon[33264]: osdmap e824: 8 total, 8 up, 8 in 2024-09-20T22:55:05.749 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:05 smithi154 ceph-mon[33264]: 1.5 scrub starts 2024-09-20T22:55:05.749 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:05 smithi154 ceph-mon[33264]: 1.5 scrub ok 2024-09-20T22:55:05.749 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:05 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:55:05.749 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:05 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:55:05.761 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:55:05.761 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:55:05.776 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-20T22:55:05.776 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:55:05.776 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:55:05.779 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:55:05.793 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-20T22:55:05.793 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T22:55:05.801 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:05.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:05.801 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:55:05.810 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:55:05.810 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:55:05.827 INFO:tasks.rados.rados.0.smithi084.stdout:3281: finishing copy_from racing read to smithi08477203-44 2024-09-20T22:55:05.827 INFO:tasks.rados.rados.0.smithi084.stdout:3282: finishing write tid 1 to smithi08477203-22 2024-09-20T22:55:05.827 INFO:tasks.rados.rados.0.smithi084.stdout:3281: finishing copy_from to smithi08477203-44 2024-09-20T22:55:05.827 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 44 v 1886 (ObjNum 1092 snap 367 seq_num 1092) dirty exists 2024-09-20T22:55:05.827 INFO:tasks.rados.rados.0.smithi084.stdout:3280: done (6 left) 2024-09-20T22:55:05.827 INFO:tasks.rados.rados.0.smithi084.stdout:3281: done (5 left) 2024-09-20T22:55:05.827 INFO:tasks.rados.rados.0.smithi084.stdout:3287: read oid 29 snap -1 2024-09-20T22:55:05.827 INFO:tasks.rados.rados.0.smithi084.stdout:3287: expect deleted 2024-09-20T22:55:05.827 INFO:tasks.rados.rados.0.smithi084.stdout:3288: copy_from oid 47 from oid 13 current snap is 378 2024-09-20T22:55:05.827 INFO:tasks.rados.rados.0.smithi084.stdout:3282: finishing write tid 2 to smithi08477203-22 2024-09-20T22:55:05.827 INFO:tasks.rados.rados.0.smithi084.stdout:3282: finishing write tid 3 to smithi08477203-22 2024-09-20T22:55:05.827 INFO:tasks.rados.rados.0.smithi084.stdout:3282: finishing write tid 4 to smithi08477203-22 2024-09-20T22:55:05.828 INFO:tasks.rados.rados.0.smithi084.stdout:3282: oid 22 updating version 0 to 1814 2024-09-20T22:55:05.828 INFO:tasks.rados.rados.0.smithi084.stdout:3282: oid 22 updating version 1814 to 1815 2024-09-20T22:55:05.828 INFO:tasks.rados.rados.0.smithi084.stdout:3282: oid 22 updating version 1815 to 1816 2024-09-20T22:55:05.828 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 22 v 1816 (ObjNum 1112 snap 378 seq_num 1112) dirty exists 2024-09-20T22:55:05.828 INFO:tasks.rados.rados.0.smithi084.stdout:3282: left oid 22 (ObjNum 1112 snap 378 seq_num 1112) 2024-09-20T22:55:05.828 INFO:tasks.rados.rados.0.smithi084.stdout:3283: finishing write tid 1 to smithi08477203-20 2024-09-20T22:55:05.828 INFO:tasks.rados.rados.0.smithi084.stdout:3283: finishing write tid 2 to smithi08477203-20 2024-09-20T22:55:05.828 INFO:tasks.rados.rados.0.smithi084.stdout:3283: finishing write tid 3 to smithi08477203-20 2024-09-20T22:55:05.828 INFO:tasks.rados.rados.0.smithi084.stdout:3283: finishing write tid 4 to smithi08477203-20 2024-09-20T22:55:05.828 INFO:tasks.rados.rados.0.smithi084.stdout:3283: oid 20 updating version 0 to 1928 2024-09-20T22:55:05.828 INFO:tasks.rados.rados.0.smithi084.stdout:3283: oid 20 updating version 1928 to 1929 2024-09-20T22:55:05.828 INFO:tasks.rados.rados.0.smithi084.stdout:3283: oid 20 version 1929 is already newer than 1927 2024-09-20T22:55:05.829 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 20 v 1929 (ObjNum 1113 snap 378 seq_num 1113) dirty exists 2024-09-20T22:55:05.829 INFO:tasks.rados.rados.0.smithi084.stdout:3283: left oid 20 (ObjNum 1113 snap 378 seq_num 1113) 2024-09-20T22:55:05.829 INFO:tasks.rados.rados.0.smithi084.stdout:3284: expect (ObjNum 996 snap 326 seq_num 996) 2024-09-20T22:55:05.829 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:55:05.829 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:55:05.843 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:55:05.843 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:55:05.859 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:55:05.859 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:55:05.903 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:55:05.924 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:05.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:05.924 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:55:06.023 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:55:06.026 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:55:06.046 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:06.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:06.047 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:55:06.086 INFO:tasks.rados.rados.0.smithi084.stdout:3285: finishing write tid 1 to smithi08477203-31 2024-09-20T22:55:06.086 INFO:tasks.rados.rados.0.smithi084.stdout:3286: finishing copy_from to smithi08477203-26 2024-09-20T22:55:06.086 INFO:tasks.rados.rados.0.smithi084.stdout:3286: got expected ENOENT (src dne) 2024-09-20T22:55:06.086 INFO:tasks.rados.rados.0.smithi084.stdout:3286: finishing copy_from racing read to smithi08477203-26 2024-09-20T22:55:06.086 INFO:tasks.rados.rados.0.smithi084.stdout:3285: finishing write tid 2 to smithi08477203-31 2024-09-20T22:55:06.086 INFO:tasks.rados.rados.0.smithi084.stdout:3285: finishing write tid 3 to smithi08477203-31 2024-09-20T22:55:06.086 INFO:tasks.rados.rados.0.smithi084.stdout:3285: finishing write tid 4 to smithi08477203-31 2024-09-20T22:55:06.086 INFO:tasks.rados.rados.0.smithi084.stdout:3285: oid 31 updating version 0 to 2467 2024-09-20T22:55:06.086 INFO:tasks.rados.rados.0.smithi084.stdout:3285: oid 31 updating version 2467 to 2468 2024-09-20T22:55:06.086 INFO:tasks.rados.rados.0.smithi084.stdout:3285: oid 31 version 2468 is already newer than 2466 2024-09-20T22:55:06.086 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 31 v 2468 (ObjNum 1114 snap 378 seq_num 1114) dirty exists 2024-09-20T22:55:06.086 INFO:tasks.rados.rados.0.smithi084.stdout:3285: left oid 31 (ObjNum 1114 snap 378 seq_num 1114) 2024-09-20T22:55:06.087 INFO:tasks.rados.rados.0.smithi084.stdout:3282: done (6 left) 2024-09-20T22:55:06.087 INFO:tasks.rados.rados.0.smithi084.stdout:3283: done (5 left) 2024-09-20T22:55:06.087 INFO:tasks.rados.rados.0.smithi084.stdout:3284: done (4 left) 2024-09-20T22:55:06.087 INFO:tasks.rados.rados.0.smithi084.stdout:3285: done (3 left) 2024-09-20T22:55:06.087 INFO:tasks.rados.rados.0.smithi084.stdout:3286: done (2 left) 2024-09-20T22:55:06.087 INFO:tasks.rados.rados.0.smithi084.stdout:3289: snap_create 2024-09-20T22:55:06.087 INFO:tasks.rados.rados.0.smithi084.stdout:3288: finishing copy_from to smithi08477203-47 2024-09-20T22:55:06.087 INFO:tasks.rados.rados.0.smithi084.stdout:3288: got expected ENOENT (src dne) 2024-09-20T22:55:06.087 INFO:tasks.rados.rados.0.smithi084.stdout:3288: finishing copy_from racing read to smithi08477203-47 2024-09-20T22:55:06.149 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:55:06.169 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:06.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:06.170 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:55:06.271 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:55:06.292 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:06.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:06.293 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:55:06.394 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:55:06.415 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:06.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:06.416 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:55:06.455 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:06 smithi154 ceph-mon[33264]: osdmap e825: 8 total, 8 up, 8 in 2024-09-20T22:55:06.455 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:06 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:55:06.455 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:06 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:55:06.455 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:06 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:55:06.455 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:06 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/2381011994' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:55:06.518 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:55:06.540 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:06.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:06.541 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:55:06.553 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-20T22:55:06.553 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:55:06.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:06 smithi084 ceph-mon[40024]: osdmap e825: 8 total, 8 up, 8 in 2024-09-20T22:55:06.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:06 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:55:06.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:06 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:55:06.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:06 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:55:06.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:06 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/2381011994' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:55:06.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:06 smithi084 ceph-mon[29774]: osdmap e825: 8 total, 8 up, 8 in 2024-09-20T22:55:06.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:06 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:55:06.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:06 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:55:06.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:06 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:55:06.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:06 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/2381011994' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:55:06.642 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:55:06.663 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:06.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:06.664 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:55:06.679 INFO:tasks.rados.rados.0.smithi084.stdout:3287: done (2 left) 2024-09-20T22:55:06.679 INFO:tasks.rados.rados.0.smithi084.stdout:3288: done (1 left) 2024-09-20T22:55:06.679 INFO:tasks.rados.rados.0.smithi084.stdout:3289: done (0 left) 2024-09-20T22:55:06.679 INFO:tasks.rados.rados.0.smithi084.stdout:3290: delete oid 39 current snap is 379 2024-09-20T22:55:06.685 INFO:tasks.rados.rados.0.smithi084.stdout:3290: done (0 left) 2024-09-20T22:55:06.685 INFO:tasks.rados.rados.0.smithi084.stdout:3291: read oid 2 snap -1 2024-09-20T22:55:06.686 INFO:tasks.rados.rados.0.smithi084.stdout:3291: expect (ObjNum 918 snap 302 seq_num 918) 2024-09-20T22:55:06.686 INFO:tasks.rados.rados.0.smithi084.stdout:3292: write (excl) oid 14 current snap is 379 2024-09-20T22:55:06.686 INFO:tasks.rados.rados.0.smithi084.stdout:3292: seq_num 1116 ranges {756612=565422,2089081=578711} 2024-09-20T22:55:06.692 INFO:tasks.rados.rados.0.smithi084.stdout:3292: writing smithi08477203-14 from 756612 to 1322034 tid 1 2024-09-20T22:55:06.700 INFO:tasks.rados.rados.0.smithi084.stdout:3292: writing smithi08477203-14 from 2089081 to 2667792 tid 2 2024-09-20T22:55:06.700 INFO:tasks.rados.rados.0.smithi084.stdout:3293: delete oid 36 current snap is 379 2024-09-20T22:55:06.700 INFO:tasks.rados.rados.0.smithi084.stdout:3291: expect (ObjNum 918 snap 302 seq_num 918) 2024-09-20T22:55:06.765 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:55:06.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:06.788 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:06.788 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:55:06.856 INFO:tasks.rados.rados.0.smithi084.stdout:3292: finishing write tid 1 to smithi08477203-14 2024-09-20T22:55:06.856 INFO:tasks.rados.rados.0.smithi084.stdout:3292: finishing write tid 2 to smithi08477203-14 2024-09-20T22:55:06.856 INFO:tasks.rados.rados.0.smithi084.stdout:3292: finishing write tid 3 to smithi08477203-14 2024-09-20T22:55:06.856 INFO:tasks.rados.rados.0.smithi084.stdout:3292: finishing write tid 4 to smithi08477203-14 2024-09-20T22:55:06.856 INFO:tasks.rados.rados.0.smithi084.stdout:3292: oid 14 updating version 0 to 1932 2024-09-20T22:55:06.856 INFO:tasks.rados.rados.0.smithi084.stdout:3292: oid 14 version 1932 is already newer than 1930 2024-09-20T22:55:06.856 INFO:tasks.rados.rados.0.smithi084.stdout:3292: oid 14 version 1932 is already newer than 1931 2024-09-20T22:55:06.856 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 14 v 1932 (ObjNum 1116 snap 379 seq_num 1116) dirty exists 2024-09-20T22:55:06.856 INFO:tasks.rados.rados.0.smithi084.stdout:3292: left oid 14 (ObjNum 1116 snap 379 seq_num 1116) 2024-09-20T22:55:06.857 INFO:tasks.rados.rados.0.smithi084.stdout:3291: done (2 left) 2024-09-20T22:55:06.857 INFO:tasks.rados.rados.0.smithi084.stdout:3292: done (1 left) 2024-09-20T22:55:06.857 INFO:tasks.rados.rados.0.smithi084.stdout:3293: done (0 left) 2024-09-20T22:55:06.857 INFO:tasks.rados.rados.0.smithi084.stdout:3294: read oid 40 snap 363 2024-09-20T22:55:06.857 INFO:tasks.rados.rados.0.smithi084.stdout:3294: expect deleted 2024-09-20T22:55:06.857 INFO:tasks.rados.rados.0.smithi084.stdout:3295: snap_create 2024-09-20T22:55:06.889 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:55:06.892 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-20T22:55:06.910 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:06.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:06.911 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:55:07.012 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:55:07.035 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:07.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:07.036 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:55:07.136 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:55:07.153 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:55:07.158 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:07.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:07.159 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:55:07.187 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.5 dump_blocked_ops 2024-09-20T22:55:07.260 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:55:07.281 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:07.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:07.282 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:55:07.317 INFO:tasks.thrashosds.thrasher:in_osds: [2, 3, 5, 6, 7, 0, 4, 1] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-20T22:55:07.318 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-20T22:55:07.318 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph pg dump --format=json 2024-09-20T22:55:07.383 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:55:07.405 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:07.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:07.406 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:55:07.473 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:55:07.507 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:55:07.528 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:07.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:07.529 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:55:07.629 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:55:07.651 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:07.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:07.651 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:55:07.685 INFO:tasks.rados.rados.0.smithi084.stdout:3294: done (1 left) 2024-09-20T22:55:07.686 INFO:tasks.rados.rados.0.smithi084.stdout:3295: done (0 left) 2024-09-20T22:55:07.686 INFO:tasks.rados.rados.0.smithi084.stdout:3296: copy_from oid 37 from oid 20 current snap is 380 2024-09-20T22:55:07.686 INFO:tasks.rados.rados.0.smithi084.stdout:3297: write (excl) oid 38 current snap is 380 2024-09-20T22:55:07.686 INFO:tasks.rados.rados.0.smithi084.stdout:3297: seq_num 1118 ranges {790226=432273,1806045=582380} 2024-09-20T22:55:07.693 INFO:tasks.rados.rados.0.smithi084.stdout:3297: writing smithi08477203-38 from 790226 to 1222499 tid 1 2024-09-20T22:55:07.700 INFO:tasks.rados.rados.0.smithi084.stdout:3297: writing smithi08477203-38 from 1806045 to 2388425 tid 2 2024-09-20T22:55:07.700 INFO:tasks.rados.rados.0.smithi084.stdout:3296: finishing copy_from racing read to smithi08477203-37 2024-09-20T22:55:07.700 INFO:tasks.rados.rados.0.smithi084.stdout:3298: snap_remove snap 337 2024-09-20T22:55:07.752 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:55:07.759 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:55:07.774 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:07.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:07.775 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:55:07.855 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:07 smithi154 ceph-mon[33264]: pgmap v1140: 158 pgs: 1 active+clean+snaptrim_wait, 1 peering, 4 active+clean+snaptrim, 152 active+clean; 506 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 24 MiB/s rd, 804 KiB/s wr, 25 op/s 2024-09-20T22:55:07.855 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:07 smithi154 ceph-mon[33264]: 3.1b scrub starts 2024-09-20T22:55:07.855 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:07 smithi154 ceph-mon[33264]: 3.1b scrub ok 2024-09-20T22:55:07.855 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:07 smithi154 ceph-mon[33264]: 6.1c scrub starts 2024-09-20T22:55:07.855 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:07 smithi154 ceph-mon[33264]: 6.1c scrub ok 2024-09-20T22:55:07.856 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:07 smithi154 ceph-mon[33264]: osdmap e826: 8 total, 8 up, 8 in 2024-09-20T22:55:07.856 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:07 smithi154 ceph-mon[33264]: 5.1f scrub starts 2024-09-20T22:55:07.856 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:07 smithi154 ceph-mon[33264]: 5.1f scrub ok 2024-09-20T22:55:07.856 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:07 smithi154 ceph-mon[33264]: 7.0 scrub starts 2024-09-20T22:55:07.856 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:07 smithi154 ceph-mon[33264]: 7.0 scrub ok 2024-09-20T22:55:07.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:07 smithi084 ceph-mon[29774]: pgmap v1140: 158 pgs: 1 active+clean+snaptrim_wait, 1 peering, 4 active+clean+snaptrim, 152 active+clean; 506 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 24 MiB/s rd, 804 KiB/s wr, 25 op/s 2024-09-20T22:55:07.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:07 smithi084 ceph-mon[29774]: 3.1b scrub starts 2024-09-20T22:55:07.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:07 smithi084 ceph-mon[29774]: 3.1b scrub ok 2024-09-20T22:55:07.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:07 smithi084 ceph-mon[29774]: 6.1c scrub starts 2024-09-20T22:55:07.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:07 smithi084 ceph-mon[29774]: 6.1c scrub ok 2024-09-20T22:55:07.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:07 smithi084 ceph-mon[29774]: osdmap e826: 8 total, 8 up, 8 in 2024-09-20T22:55:07.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:07 smithi084 ceph-mon[29774]: 5.1f scrub starts 2024-09-20T22:55:07.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:07 smithi084 ceph-mon[29774]: 5.1f scrub ok 2024-09-20T22:55:07.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:07 smithi084 ceph-mon[29774]: 7.0 scrub starts 2024-09-20T22:55:07.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:07 smithi084 ceph-mon[29774]: 7.0 scrub ok 2024-09-20T22:55:07.866 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:07 smithi084 ceph-mon[40024]: pgmap v1140: 158 pgs: 1 active+clean+snaptrim_wait, 1 peering, 4 active+clean+snaptrim, 152 active+clean; 506 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 24 MiB/s rd, 804 KiB/s wr, 25 op/s 2024-09-20T22:55:07.866 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:07 smithi084 ceph-mon[40024]: 3.1b scrub starts 2024-09-20T22:55:07.866 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:07 smithi084 ceph-mon[40024]: 3.1b scrub ok 2024-09-20T22:55:07.866 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:07 smithi084 ceph-mon[40024]: 6.1c scrub starts 2024-09-20T22:55:07.866 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:07 smithi084 ceph-mon[40024]: 6.1c scrub ok 2024-09-20T22:55:07.866 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:07 smithi084 ceph-mon[40024]: osdmap e826: 8 total, 8 up, 8 in 2024-09-20T22:55:07.866 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:07 smithi084 ceph-mon[40024]: 5.1f scrub starts 2024-09-20T22:55:07.866 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:07 smithi084 ceph-mon[40024]: 5.1f scrub ok 2024-09-20T22:55:07.866 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:07 smithi084 ceph-mon[40024]: 7.0 scrub starts 2024-09-20T22:55:07.866 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:07 smithi084 ceph-mon[40024]: 7.0 scrub ok 2024-09-20T22:55:07.876 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:55:07.897 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:07.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:07.898 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:55:07.954 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-20T22:55:07.954 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:55:07.998 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:55:08.020 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:08.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:08.021 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:55:08.121 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:55:08.143 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:08.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:08.143 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:55:08.244 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:55:08.266 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:08.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:08.267 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:55:08.319 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is unset 2024-09-20T22:55:08.327 INFO:tasks.rados.rados.0.smithi084.stdout:3298: done (2 left) 2024-09-20T22:55:08.327 INFO:tasks.rados.rados.0.smithi084.stdout:3299: delete oid 6 current snap is 380 2024-09-20T22:55:08.327 INFO:tasks.rados.rados.0.smithi084.stdout:3296: finishing copy_from to smithi08477203-37 2024-09-20T22:55:08.327 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 37 v 1929 (ObjNum 1113 snap 378 seq_num 1113) dirty exists 2024-09-20T22:55:08.327 INFO:tasks.rados.rados.0.smithi084.stdout:3297: finishing write tid 1 to smithi08477203-38 2024-09-20T22:55:08.328 INFO:tasks.rados.rados.0.smithi084.stdout:3297: finishing write tid 2 to smithi08477203-38 2024-09-20T22:55:08.328 INFO:tasks.rados.rados.0.smithi084.stdout:3297: finishing write tid 3 to smithi08477203-38 2024-09-20T22:55:08.328 INFO:tasks.rados.rados.0.smithi084.stdout:3297: finishing write tid 4 to smithi08477203-38 2024-09-20T22:55:08.328 INFO:tasks.rados.rados.0.smithi084.stdout:3297: oid 38 updating version 0 to 2469 2024-09-20T22:55:08.328 INFO:tasks.rados.rados.0.smithi084.stdout:3297: oid 38 updating version 2469 to 2471 2024-09-20T22:55:08.328 INFO:tasks.rados.rados.0.smithi084.stdout:3297: oid 38 version 2471 is already newer than 2470 2024-09-20T22:55:08.328 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 38 v 2471 (ObjNum 1118 snap 380 seq_num 1118) dirty exists 2024-09-20T22:55:08.328 INFO:tasks.rados.rados.0.smithi084.stdout:3297: left oid 38 (ObjNum 1118 snap 380 seq_num 1118) 2024-09-20T22:55:08.347 INFO:tasks.rados.rados.0.smithi084.stdout:3296: done (2 left) 2024-09-20T22:55:08.348 INFO:tasks.rados.rados.0.smithi084.stdout:3297: done (1 left) 2024-09-20T22:55:08.348 INFO:tasks.rados.rados.0.smithi084.stdout:3299: done (0 left) 2024-09-20T22:55:08.348 INFO:tasks.rados.rados.0.smithi084.stdout:3300: read oid 38 snap -1 2024-09-20T22:55:08.348 INFO:tasks.rados.rados.0.smithi084.stdout:3300: expect (ObjNum 1118 snap 380 seq_num 1118) 2024-09-20T22:55:08.348 INFO:tasks.rados.rados.0.smithi084.stdout:3301: write oid 39 current snap is 380 2024-09-20T22:55:08.348 INFO:tasks.rados.rados.0.smithi084.stdout:3301: seq_num 1120 ranges {423840=582771,1523009=586049,2657046=166794} 2024-09-20T22:55:08.353 INFO:tasks.rados.rados.0.smithi084.stdout:3301: writing smithi08477203-39 from 423840 to 1006611 tid 1 2024-09-20T22:55:08.360 INFO:tasks.rados.rados.0.smithi084.stdout:3301: writing smithi08477203-39 from 1523009 to 2109058 tid 2 2024-09-20T22:55:08.364 INFO:tasks.rados.rados.0.smithi084.stdout:3301: writing smithi08477203-39 from 2657046 to 2823840 tid 3 2024-09-20T22:55:08.364 INFO:tasks.rados.rados.0.smithi084.stdout:3302: write oid 35 current snap is 380 2024-09-20T22:55:08.364 INFO:tasks.rados.rados.0.smithi084.stdout:3302: seq_num 1121 ranges {440647=658020,1581491=446060,2800822=39825} 2024-09-20T22:55:08.368 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:55:08.370 INFO:tasks.rados.rados.0.smithi084.stdout:3302: writing smithi08477203-35 from 440647 to 1098667 tid 1 2024-09-20T22:55:08.376 INFO:tasks.rados.rados.0.smithi084.stdout:3302: writing smithi08477203-35 from 1581491 to 2027551 tid 2 2024-09-20T22:55:08.380 INFO:tasks.rados.rados.0.smithi084.stdout:3302: writing smithi08477203-35 from 2800822 to 2840647 tid 3 2024-09-20T22:55:08.380 INFO:tasks.rados.rados.0.smithi084.stdout:3303: write oid 3 current snap is 380 2024-09-20T22:55:08.381 INFO:tasks.rados.rados.0.smithi084.stdout:3303: seq_num 1122 ranges {457454=733269,1923620=589718} 2024-09-20T22:55:08.387 INFO:tasks.rados.rados.0.smithi084.stdout:3303: writing smithi08477203-3 from 457454 to 1190723 tid 1 2024-09-20T22:55:08.389 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:08.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:08.390 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:55:08.393 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:55:08 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: ::ffff:172.21.15.154 - - [20/Sep/2024:22:55:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-20T22:55:08.395 INFO:tasks.rados.rados.0.smithi084.stdout:3303: writing smithi08477203-3 from 1923620 to 2513338 tid 2 2024-09-20T22:55:08.395 INFO:tasks.rados.rados.0.smithi084.stdout:3304: delete oid 36 current snap is 380 2024-09-20T22:55:08.395 INFO:tasks.rados.rados.0.smithi084.stdout:3300: expect (ObjNum 1118 snap 380 seq_num 1118) 2024-09-20T22:55:08.441 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-20T22:55:08.443 INFO:teuthology.orchestra.run.smithi084.stderr:dumped all 2024-09-20T22:55:08.491 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:55:08.512 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:08.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:08.513 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:55:08.584 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-20T22:55:08.613 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:55:08.627 INFO:tasks.rados.rados.0.smithi084.stdout:3301: finishing write tid 1 to smithi08477203-39 2024-09-20T22:55:08.628 INFO:tasks.rados.rados.0.smithi084.stdout:3300: done (4 left) 2024-09-20T22:55:08.628 INFO:tasks.rados.rados.0.smithi084.stdout:3304: done (3 left) 2024-09-20T22:55:08.628 INFO:tasks.rados.rados.0.smithi084.stdout:3305: snap_create 2024-09-20T22:55:08.628 INFO:tasks.rados.rados.0.smithi084.stdout:3302: finishing write tid 1 to smithi08477203-35 2024-09-20T22:55:08.628 INFO:tasks.rados.rados.0.smithi084.stdout:3301: finishing write tid 2 to smithi08477203-39 2024-09-20T22:55:08.628 INFO:tasks.rados.rados.0.smithi084.stdout:3302: finishing write tid 2 to smithi08477203-35 2024-09-20T22:55:08.628 INFO:tasks.rados.rados.0.smithi084.stdout:3301: finishing write tid 3 to smithi08477203-39 2024-09-20T22:55:08.628 INFO:tasks.rados.rados.0.smithi084.stdout:3301: finishing write tid 4 to smithi08477203-39 2024-09-20T22:55:08.628 INFO:tasks.rados.rados.0.smithi084.stdout:3301: finishing write tid 5 to smithi08477203-39 2024-09-20T22:55:08.628 INFO:tasks.rados.rados.0.smithi084.stdout:3301: oid 39 updating version 0 to 2458 2024-09-20T22:55:08.628 INFO:tasks.rados.rados.0.smithi084.stdout:3301: oid 39 updating version 2458 to 2460 2024-09-20T22:55:08.628 INFO:tasks.rados.rados.0.smithi084.stdout:3301: oid 39 version 2460 is already newer than 2459 2024-09-20T22:55:08.628 INFO:tasks.rados.rados.0.smithi084.stdout:3301: oid 39 updating version 2460 to 2461 2024-09-20T22:55:08.628 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 39 v 2461 (ObjNum 1120 snap 380 seq_num 1120) dirty exists 2024-09-20T22:55:08.629 INFO:tasks.rados.rados.0.smithi084.stdout:3301: left oid 39 (ObjNum 1120 snap 380 seq_num 1120) 2024-09-20T22:55:08.629 INFO:tasks.rados.rados.0.smithi084.stdout:3302: finishing write tid 3 to smithi08477203-35 2024-09-20T22:55:08.629 INFO:tasks.rados.rados.0.smithi084.stdout:3302: finishing write tid 4 to smithi08477203-35 2024-09-20T22:55:08.629 INFO:tasks.rados.rados.0.smithi084.stdout:3302: finishing write tid 5 to smithi08477203-35 2024-09-20T22:55:08.629 INFO:tasks.rados.rados.0.smithi084.stdout:3302: oid 35 updating version 0 to 1126 2024-09-20T22:55:08.629 INFO:tasks.rados.rados.0.smithi084.stdout:3302: oid 35 updating version 1126 to 1127 2024-09-20T22:55:08.629 INFO:tasks.rados.rados.0.smithi084.stdout:3302: oid 35 updating version 1127 to 1129 2024-09-20T22:55:08.629 INFO:tasks.rados.rados.0.smithi084.stdout:3302: oid 35 version 1129 is already newer than 1128 2024-09-20T22:55:08.629 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 35 v 1129 (ObjNum 1121 snap 380 seq_num 1121) dirty exists 2024-09-20T22:55:08.629 INFO:tasks.rados.rados.0.smithi084.stdout:3302: left oid 35 (ObjNum 1121 snap 380 seq_num 1121) 2024-09-20T22:55:08.629 INFO:tasks.rados.rados.0.smithi084.stdout:3303: finishing write tid 1 to smithi08477203-3 2024-09-20T22:55:08.629 INFO:tasks.rados.rados.0.smithi084.stdout:3303: finishing write tid 2 to smithi08477203-3 2024-09-20T22:55:08.629 INFO:tasks.rados.rados.0.smithi084.stdout:3303: finishing write tid 3 to smithi08477203-3 2024-09-20T22:55:08.629 INFO:tasks.rados.rados.0.smithi084.stdout:3303: finishing write tid 4 to smithi08477203-3 2024-09-20T22:55:08.630 INFO:tasks.rados.rados.0.smithi084.stdout:3303: oid 3 updating version 0 to 2555 2024-09-20T22:55:08.630 INFO:tasks.rados.rados.0.smithi084.stdout:3303: oid 3 version 2555 is already newer than 2554 2024-09-20T22:55:08.630 INFO:tasks.rados.rados.0.smithi084.stdout:3303: oid 3 version 2555 is already newer than 2553 2024-09-20T22:55:08.630 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 3 v 2555 (ObjNum 1122 snap 380 seq_num 1122) dirty exists 2024-09-20T22:55:08.630 INFO:tasks.rados.rados.0.smithi084.stdout:3303: left oid 3 (ObjNum 1122 snap 380 seq_num 1122) 2024-09-20T22:55:08.634 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:08.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:08.635 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:55:08.678 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.5 dump_historic_ops 2024-09-20T22:55:08.736 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:55:08.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:08.759 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:08.759 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:55:08.860 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:55:08.882 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:08.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:08.883 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:55:08.927 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:55:08.983 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:55:09.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:08 smithi154 ceph-mon[33264]: 6.15 scrub starts 2024-09-20T22:55:09.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:08 smithi154 ceph-mon[33264]: 6.15 scrub ok 2024-09-20T22:55:09.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:08 smithi154 ceph-mon[33264]: osdmap e827: 8 total, 8 up, 8 in 2024-09-20T22:55:09.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:08 smithi154 ceph-mon[33264]: pgmap v1143: 158 pgs: 1 active+clean+snaptrim_wait, 1 peering, 4 active+clean+snaptrim, 152 active+clean; 506 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail 2024-09-20T22:55:09.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:08 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/870205038' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:55:09.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:08 smithi154 ceph-mon[33264]: 4.7 scrub starts 2024-09-20T22:55:09.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:08 smithi154 ceph-mon[33264]: 4.7 scrub ok 2024-09-20T22:55:09.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:08 smithi154 ceph-mon[33264]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T22:55:09.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:08 smithi154 ceph-mon[33264]: Cluster is now healthy 2024-09-20T22:55:09.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:08 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/870205038' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T22:55:09.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:08 smithi154 ceph-mon[33264]: osdmap e828: 8 total, 8 up, 8 in 2024-09-20T22:55:09.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:08 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:55:09.006 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:08 smithi154 ceph-mon[33264]: from='client.17139 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T22:55:09.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:09.007 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:09.007 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:55:09.108 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:55:09.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:08 smithi084 ceph-mon[29774]: 6.15 scrub starts 2024-09-20T22:55:09.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:08 smithi084 ceph-mon[29774]: 6.15 scrub ok 2024-09-20T22:55:09.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:08 smithi084 ceph-mon[29774]: osdmap e827: 8 total, 8 up, 8 in 2024-09-20T22:55:09.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:08 smithi084 ceph-mon[29774]: pgmap v1143: 158 pgs: 1 active+clean+snaptrim_wait, 1 peering, 4 active+clean+snaptrim, 152 active+clean; 506 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail 2024-09-20T22:55:09.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:08 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/870205038' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:55:09.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:08 smithi084 ceph-mon[29774]: 4.7 scrub starts 2024-09-20T22:55:09.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:08 smithi084 ceph-mon[29774]: 4.7 scrub ok 2024-09-20T22:55:09.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:08 smithi084 ceph-mon[29774]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T22:55:09.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:08 smithi084 ceph-mon[29774]: Cluster is now healthy 2024-09-20T22:55:09.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:08 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/870205038' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T22:55:09.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:08 smithi084 ceph-mon[29774]: osdmap e828: 8 total, 8 up, 8 in 2024-09-20T22:55:09.116 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:08 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:55:09.116 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:08 smithi084 ceph-mon[29774]: from='client.17139 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T22:55:09.116 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:08 smithi084 ceph-mon[40024]: 6.15 scrub starts 2024-09-20T22:55:09.116 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:08 smithi084 ceph-mon[40024]: 6.15 scrub ok 2024-09-20T22:55:09.116 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:08 smithi084 ceph-mon[40024]: osdmap e827: 8 total, 8 up, 8 in 2024-09-20T22:55:09.116 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:08 smithi084 ceph-mon[40024]: pgmap v1143: 158 pgs: 1 active+clean+snaptrim_wait, 1 peering, 4 active+clean+snaptrim, 152 active+clean; 506 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail 2024-09-20T22:55:09.116 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:08 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/870205038' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:55:09.116 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:08 smithi084 ceph-mon[40024]: 4.7 scrub starts 2024-09-20T22:55:09.116 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:08 smithi084 ceph-mon[40024]: 4.7 scrub ok 2024-09-20T22:55:09.116 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:08 smithi084 ceph-mon[40024]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T22:55:09.116 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:08 smithi084 ceph-mon[40024]: Cluster is now healthy 2024-09-20T22:55:09.117 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:08 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/870205038' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T22:55:09.117 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:08 smithi084 ceph-mon[40024]: osdmap e828: 8 total, 8 up, 8 in 2024-09-20T22:55:09.117 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:08 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:55:09.117 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:08 smithi084 ceph-mon[40024]: from='client.17139 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T22:55:09.130 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:09.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:09.131 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:55:09.231 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:55:09.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:09.255 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:09.255 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:55:09.332 INFO:tasks.rados.rados.0.smithi084.stdout:3301: done (3 left) 2024-09-20T22:55:09.332 INFO:tasks.rados.rados.0.smithi084.stdout:3302: done (2 left) 2024-09-20T22:55:09.332 INFO:tasks.rados.rados.0.smithi084.stdout:3303: done (1 left) 2024-09-20T22:55:09.332 INFO:tasks.rados.rados.0.smithi084.stdout:3305: done (0 left) 2024-09-20T22:55:09.332 INFO:tasks.rados.rados.0.smithi084.stdout:3306: write (excl) oid 33 current snap is 381 2024-09-20T22:55:09.332 INFO:tasks.rados.rados.0.smithi084.stdout:3306: seq_num 1124 ranges {491068=483767,1640584=593387,2650385=240683} 2024-09-20T22:55:09.337 INFO:tasks.rados.rados.0.smithi084.stdout:3306: writing smithi08477203-33 from 491068 to 974835 tid 1 2024-09-20T22:55:09.344 INFO:tasks.rados.rados.0.smithi084.stdout:3306: writing smithi08477203-33 from 1640584 to 2233971 tid 2 2024-09-20T22:55:09.348 INFO:tasks.rados.rados.0.smithi084.stdout:3306: writing smithi08477203-33 from 2650385 to 2891068 tid 3 2024-09-20T22:55:09.348 INFO:tasks.rados.rados.0.smithi084.stdout:3307: read oid 41 snap -1 2024-09-20T22:55:09.348 INFO:tasks.rados.rados.0.smithi084.stdout:3307: expect deleted 2024-09-20T22:55:09.348 INFO:tasks.rados.rados.0.smithi084.stdout:3308: delete oid 45 current snap is 381 2024-09-20T22:55:09.348 INFO:tasks.rados.rados.0.smithi084.stdout:3306: finishing write tid 1 to smithi08477203-33 2024-09-20T22:55:09.348 INFO:tasks.rados.rados.0.smithi084.stdout:3306: finishing write tid 2 to smithi08477203-33 2024-09-20T22:55:09.352 INFO:tasks.rados.rados.0.smithi084.stdout:3307: done (2 left) 2024-09-20T22:55:09.352 INFO:tasks.rados.rados.0.smithi084.stdout:3308: done (1 left) 2024-09-20T22:55:09.352 INFO:tasks.rados.rados.0.smithi084.stdout:3309: read oid 21 snap 341 2024-09-20T22:55:09.352 INFO:tasks.rados.rados.0.smithi084.stdout:3309: expect (ObjNum 984 snap 319 seq_num 984) 2024-09-20T22:55:09.352 INFO:tasks.rados.rados.0.smithi084.stdout:3310: snap_remove snap 316 2024-09-20T22:55:09.356 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:55:09.365 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-20T22:55:09.365 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:55:09.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:09.380 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:09.380 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:55:09.481 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:55:09.503 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:09.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:09.504 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:55:09.605 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:55:09.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:09.627 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:09.627 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:55:09.728 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:55:09.750 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:09.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:09.751 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:55:09.852 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:55:09.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:09 smithi084 ceph-mon[29774]: 5.6 scrub starts 2024-09-20T22:55:09.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:09 smithi084 ceph-mon[29774]: 5.6 scrub ok 2024-09-20T22:55:09.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:09 smithi084 ceph-mon[29774]: 6.12 deep-scrub starts 2024-09-20T22:55:09.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:09 smithi084 ceph-mon[29774]: 6.12 deep-scrub ok 2024-09-20T22:55:09.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:09 smithi084 ceph-mon[29774]: 4.6 deep-scrub starts 2024-09-20T22:55:09.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:09 smithi084 ceph-mon[29774]: 4.6 deep-scrub ok 2024-09-20T22:55:09.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:09 smithi084 ceph-mon[29774]: 4.a scrub starts 2024-09-20T22:55:09.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:09 smithi084 ceph-mon[29774]: 4.a scrub ok 2024-09-20T22:55:09.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:09 smithi084 ceph-mon[29774]: 5.c scrub starts 2024-09-20T22:55:09.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:09 smithi084 ceph-mon[29774]: 5.c scrub ok 2024-09-20T22:55:09.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:09 smithi084 ceph-mon[29774]: osdmap e829: 8 total, 8 up, 8 in 2024-09-20T22:55:09.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:09 smithi084 ceph-mon[29774]: 3.8 scrub starts 2024-09-20T22:55:09.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:09 smithi084 ceph-mon[29774]: 3.8 scrub ok 2024-09-20T22:55:09.866 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:09 smithi084 ceph-mon[29774]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:55:09.866 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:09 smithi084 ceph-mon[40024]: 5.6 scrub starts 2024-09-20T22:55:09.866 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:09 smithi084 ceph-mon[40024]: 5.6 scrub ok 2024-09-20T22:55:09.866 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:09 smithi084 ceph-mon[40024]: 6.12 deep-scrub starts 2024-09-20T22:55:09.866 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:09 smithi084 ceph-mon[40024]: 6.12 deep-scrub ok 2024-09-20T22:55:09.866 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:09 smithi084 ceph-mon[40024]: 4.6 deep-scrub starts 2024-09-20T22:55:09.866 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:09 smithi084 ceph-mon[40024]: 4.6 deep-scrub ok 2024-09-20T22:55:09.866 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:09 smithi084 ceph-mon[40024]: 4.a scrub starts 2024-09-20T22:55:09.866 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:09 smithi084 ceph-mon[40024]: 4.a scrub ok 2024-09-20T22:55:09.866 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:09 smithi084 ceph-mon[40024]: 5.c scrub starts 2024-09-20T22:55:09.866 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:09 smithi084 ceph-mon[40024]: 5.c scrub ok 2024-09-20T22:55:09.867 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:09 smithi084 ceph-mon[40024]: osdmap e829: 8 total, 8 up, 8 in 2024-09-20T22:55:09.867 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:09 smithi084 ceph-mon[40024]: 3.8 scrub starts 2024-09-20T22:55:09.867 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:09 smithi084 ceph-mon[40024]: 3.8 scrub ok 2024-09-20T22:55:09.867 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:09 smithi084 ceph-mon[40024]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:55:09.875 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:09.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:09.876 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:55:09.893 INFO:teuthology.orchestra.run.smithi084.stdout:{"pg_ready":true,"pg_map":{"version":1144,"stamp":"2024-09-20T22:55:08.319466+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":530830427,"num_objects":440,"num_object_clones":192,"num_object_copies":1320,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":440,"num_whiteouts":12,"num_read":7605,"num_read_kb":5948891,"num_write":4773,"num_write_kb":1530291,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1453,"num_bytes_recovered":730756519,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":14450,"ondisk_log_size":14450,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":474,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":1685772,"kb_used_data":1257168,"kb_used_omap":12,"kb_used_meta":428403,"kb_avail":748275444,"statfs":{"total":767960285184,"available":766234054656,"internally_reserved":0,"allocated":1287340032,"data_stored":1697799109,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":438685232},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.981535"},"pg_stats":[{"pgid":"3.1f","version":"86'2","reported_seq":910,"reported_epoch":823,"state":"active+clean","last_fresh":"2024-09-20T22:55:03.323641+0000","last_change":"2024-09-20T22:54:40.830995+0000","last_active":"2024-09-20T22:55:03.323641+0000","last_peered":"2024-09-20T22:55:03.323641+0000","last_clean":"2024-09-20T22:55:03.323641+0000","last_became_active":"2024-09-20T22:47:49.222114+0000","last_became_peered":"2024-09-20T22:47:49.222114+0000","last_unstale":"2024-09-20T22:55:03.323641+0000","last_undegraded":"2024-09-20T22:55:03.323641+0000","last_fullsized":"2024-09-20T22:55:03.323641+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'2","last_scrub_stamp":"2024-09-20T22:54:40.830920+0000","last_deep_scrub":"86'2","last_deep_scrub_stamp":"2024-09-20T22:42:39.664421+0000","last_clean_scrub_stamp":"2024-09-20T22:54:40.830920+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:47.276232+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":436,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"625'15","reported_seq":922,"reported_epoch":823,"state":"active+clean","last_fresh":"2024-09-20T22:55:03.318291+0000","last_change":"2024-09-20T22:54:43.123303+0000","last_active":"2024-09-20T22:55:03.318291+0000","last_peered":"2024-09-20T22:55:03.318291+0000","last_clean":"2024-09-20T22:55:03.318291+0000","last_became_active":"2024-09-20T22:54:17.354145+0000","last_became_peered":"2024-09-20T22:54:17.354145+0000","last_unstale":"2024-09-20T22:55:03.318291+0000","last_undegraded":"2024-09-20T22:55:03.318291+0000","last_fullsized":"2024-09-20T22:55:03.318291+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'15","last_scrub_stamp":"2024-09-20T22:54:43.123216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:54:43.123216+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:53.380617+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":103,"reported_epoch":823,"state":"active+clean","last_fresh":"2024-09-20T22:55:03.473664+0000","last_change":"2024-09-20T22:54:17.338438+0000","last_active":"2024-09-20T22:55:03.473664+0000","last_peered":"2024-09-20T22:55:03.473664+0000","last_clean":"2024-09-20T22:55:03.473664+0000","last_became_active":"2024-09-20T22:54:17.337980+0000","last_became_peered":"2024-09-20T22:54:17.337980+0000","last_unstale":"2024-09-20T22:55:03.473664+0000","last_undegraded":"2024-09-20T22:55:03.473664+0000","last_fullsized":"2024-09-20T22:55:03.473664+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:54:15.974498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:54:15.974498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:29.491929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":883,"reported_epoch":823,"state":"active+clean","last_fresh":"2024-09-20T22:55:03.318523+0000","last_change":"2024-09-20T22:54:17.351246+0000","last_active":"2024-09-20T22:55:03.318523+0000","last_peered":"2024-09-20T22:55:03.318523+0000","last_clean":"2024-09-20T22:55:03.318523+0000","last_became_active":"2024-09-20T22:54:17.350829+0000","last_became_peered":"2024-09-20T22:54:17.350829+0000","last_unstale":"2024-09-20T22:55:03.318523+0000","last_undegraded":"2024-09-20T22:55:03.318523+0000","last_fullsized":"2024-09-20T22:55:03.318523+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:58.481327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:51:36.730015+0000","last_clean_scrub_stamp":"2024-09-20T22:53:58.481327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:16.493810+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":1553,"reported_epoch":823,"state":"active+clean","last_fresh":"2024-09-20T22:55:03.322370+0000","last_change":"2024-09-20T22:54:15.238419+0000","last_active":"2024-09-20T22:55:03.322370+0000","last_peered":"2024-09-20T22:55:03.322370+0000","last_clean":"2024-09-20T22:55:03.322370+0000","last_became_active":"2024-09-20T22:41:44.832099+0000","last_became_peered":"2024-09-20T22:41:44.832099+0000","last_unstale":"2024-09-20T22:55:03.322370+0000","last_undegraded":"2024-09-20T22:55:03.322370+0000","last_fullsized":"2024-09-20T22:55:03.322370+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:54:15.238344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:54:15.238344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:29.894427+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":1558,"reported_epoch":823,"state":"active+clean","last_fresh":"2024-09-20T22:55:03.322323+0000","last_change":"2024-09-20T22:54:47.427999+0000","last_active":"2024-09-20T22:55:03.322323+0000","last_peered":"2024-09-20T22:55:03.322323+0000","last_clean":"2024-09-20T22:55:03.322323+0000","last_became_active":"2024-09-20T22:41:38.157298+0000","last_became_peered":"2024-09-20T22:41:38.157298+0000","last_unstale":"2024-09-20T22:55:03.322323+0000","last_undegraded":"2024-09-20T22:55:03.322323+0000","last_fullsized":"2024-09-20T22:55:03.322323+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:54:47.427927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:54:47.427927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:51.157529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"625'27","reported_seq":1640,"reported_epoch":823,"state":"active+clean","last_fresh":"2024-09-20T22:55:03.323327+0000","last_change":"2024-09-20T22:54:45.390249+0000","last_active":"2024-09-20T22:55:03.323327+0000","last_peered":"2024-09-20T22:55:03.323327+0000","last_clean":"2024-09-20T22:55:03.323327+0000","last_became_active":"2024-09-20T22:47:49.218798+0000","last_became_peered":"2024-09-20T22:47:49.218798+0000","last_unstale":"2024-09-20T22:55:03.323327+0000","last_undegraded":"2024-09-20T22:55:03.323327+0000","last_fullsized":"2024-09-20T22:55:03.323327+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'27","last_scrub_stamp":"2024-09-20T22:54:45.390169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:54:45.390169+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:51.072485+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":875,"reported_epoch":823,"state":"active+clean","last_fresh":"2024-09-20T22:55:03.318193+0000","last_change":"2024-09-20T22:54:17.351386+0000","last_active":"2024-09-20T22:55:03.318193+0000","last_peered":"2024-09-20T22:55:03.318193+0000","last_clean":"2024-09-20T22:55:03.318193+0000","last_became_active":"2024-09-20T22:54:17.351020+0000","last_became_peered":"2024-09-20T22:54:17.351020+0000","last_unstale":"2024-09-20T22:55:03.318193+0000","last_undegraded":"2024-09-20T22:55:03.318193+0000","last_fullsized":"2024-09-20T22:55:03.318193+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:14.594823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:52:14.594823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:05.026573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":1577,"reported_epoch":823,"state":"active+clean","last_fresh":"2024-09-20T22:55:03.329252+0000","last_change":"2024-09-20T22:53:56.980951+0000","last_active":"2024-09-20T22:55:03.329252+0000","last_peered":"2024-09-20T22:55:03.329252+0000","last_clean":"2024-09-20T22:55:03.329252+0000","last_became_active":"2024-09-20T22:48:05.242059+0000","last_became_peered":"2024-09-20T22:48:05.242059+0000","last_unstale":"2024-09-20T22:55:03.329252+0000","last_undegraded":"2024-09-20T22:55:03.329252+0000","last_fullsized":"2024-09-20T22:55:03.329252+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:56.980885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:46:00.999939+0000","last_clean_scrub_stamp":"2024-09-20T22:53:56.980885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:13.013721+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"625'15","reported_seq":890,"reported_epoch":823,"state":"active+clean","last_fresh":"2024-09-20T22:55:03.318547+0000","last_change":"2024-09-20T22:53:51.486324+0000","last_active":"2024-09-20T22:55:03.318547+0000","last_peered":"2024-09-20T22:55:03.318547+0000","last_clean":"2024-09-20T22:55:03.318547+0000","last_became_active":"2024-09-20T22:48:05.233429+0000","last_became_peered":"2024-09-20T22:48:05.233429+0000","last_unstale":"2024-09-20T22:55:03.318547+0000","last_undegraded":"2024-09-20T22:55:03.318547+0000","last_fullsized":"2024-09-20T22:55:03.318547+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'15","last_scrub_stamp":"2024-09-20T22:53:51.486202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:53:51.486202+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:15.973503+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":1569,"reported_epoch":823,"state":"active+clean","last_fresh":"2024-09-20T22:55:03.328381+0000","last_change":"2024-09-20T22:54:21.027796+0000","last_active":"2024-09-20T22:55:03.328381+0000","last_peered":"2024-09-20T22:55:03.328381+0000","last_clean":"2024-09-20T22:55:03.328381+0000","last_became_active":"2024-09-20T22:47:49.227379+0000","last_became_peered":"2024-09-20T22:47:49.227379+0000","last_unstale":"2024-09-20T22:55:03.328381+0000","last_undegraded":"2024-09-20T22:55:03.328381+0000","last_fullsized":"2024-09-20T22:55:03.328381+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:54:21.027723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:54:21.027723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:28.806253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":918,"reported_epoch":823,"state":"active+clean","last_fresh":"2024-09-20T22:55:03.321064+0000","last_change":"2024-09-20T22:54:17.340772+0000","last_active":"2024-09-20T22:55:03.321064+0000","last_peered":"2024-09-20T22:55:03.321064+0000","last_clean":"2024-09-20T22:55:03.321064+0000","last_became_active":"2024-09-20T22:54:17.340467+0000","last_became_peered":"2024-09-20T22:54:17.340467+0000","last_unstale":"2024-09-20T22:55:03.321064+0000","last_undegraded":"2024-09-20T22:55:03.321064+0000","last_fullsized":"2024-09-20T22:55:03.321064+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:35.920968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:49:08.664407+0000","last_clean_scrub_stamp":"2024-09-20T22:53:35.920968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":1589,"reported_epoch":823,"state":"active+clean","last_fresh":"2024-09-20T22:55:03.328287+0000","last_change":"2024-09-20T22:54:17.342948+0000","last_active":"2024-09-20T22:55:03.328287+0000","last_peered":"2024-09-20T22:55:03.328287+0000","last_clean":"2024-09-20T22:55:03.328287+0000","last_became_active":"2024-09-20T22:54:17.342619+0000","last_became_peered":"2024-09-20T22:54:17.342619+0000","last_unstale":"2024-09-20T22:55:03.328287+0000","last_undegraded":"2024-09-20T22:55:03.328287+0000","last_fullsized":"2024-09-20T22:55:03.328287+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:31.092915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:53:31.092915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:07.223074+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"625'9","reported_seq":904,"reported_epoch":823,"state":"active+clean","last_fresh":"2024-09-20T22:55:03.319014+0000","last_change":"2024-09-20T22:54:17.383703+0000","last_active":"2024-09-20T22:55:03.319014+0000","last_peered":"2024-09-20T22:55:03.319014+0000","last_clean":"2024-09-20T22:55:03.319014+0000","last_became_active":"2024-09-20T22:54:17.352032+0000","last_became_peered":"2024-09-20T22:54:17.352032+0000","last_unstale":"2024-09-20T22:55:03.319014+0000","last_undegraded":"2024-09-20T22:55:03.319014+0000","last_fullsized":"2024-09-20T22:55:03.319014+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'9","last_scrub_stamp":"2024-09-20T22:53:09.623458+0000","last_deep_scrub":"181'7","last_deep_scrub_stamp":"2024-09-20T22:48:41.049381+0000","last_clean_scrub_stamp":"2024-09-20T22:53:09.623458+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:05.026664+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":9,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":1602,"reported_epoch":823,"state":"active+clean","last_fresh":"2024-09-20T22:55:03.325378+0000","last_change":"2024-09-20T22:54:19.572825+0000","last_active":"2024-09-20T22:55:03.325378+0000","last_peered":"2024-09-20T22:55:03.325378+0000","last_clean":"2024-09-20T22:55:03.325378+0000","last_became_active":"2024-09-20T22:54:17.331217+0000","last_became_peered":"2024-09-20T22:54:17.331217+0000","last_unstale":"2024-09-20T22:55:03.325378+0000","last_undegraded":"2024-09-20T22:55:03.325378+0000","last_fullsized":"2024-09-20T22:55:03.325378+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:54:19.572737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:47:40.361958+0000","last_clean_scrub_stamp":"2024-09-20T22:54:19.572737+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:20.397333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":1568,"reported_epoch":823,"state":"active+clean","last_fresh":"2024-09-20T22:55:03.328830+0000","last_change":"2024-09-20T22:54:17.340038+0000","last_active":"2024-09-20T22:55:03.328830+0000","last_peered":"2024-09-20T22:55:03.328830+0000","last_clean":"2024-09-20T22:55:03.328830+0000","last_became_active":"2024-09-20T22:54:17.339658+0000","last_became_peered":"2024-09-20T22:54:17.339658+0000","last_unstale":"2024-09-20T22:55:03.328830+0000","last_undegraded":"2024-09-20T22:55:03.328830+0000","last_fullsized":"2024-09-20T22:55:03.328830+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:52.116243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:53:52.116243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":38,"reported_epoch":823,"state":"active+clean","last_fresh":"2024-09-20T22:55:03.319903+0000","last_change":"2024-09-20T22:54:49.373154+0000","last_active":"2024-09-20T22:55:03.319903+0000","last_peered":"2024-09-20T22:55:03.319903+0000","last_clean":"2024-09-20T22:55:03.319903+0000","last_became_active":"2024-09-20T22:54:49.372771+0000","last_became_peered":"2024-09-20T22:54:49.372771+0000","last_unstale":"2024-09-20T22:55:03.319903+0000","last_undegraded":"2024-09-20T22:55:03.319903+0000","last_fullsized":"2024-09-20T22:55:03.319903+0000","mapping_epoch":808,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":809,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:48.558619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:53:48.558619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:07.033376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":903,"reported_epoch":823,"state":"active+clean","last_fresh":"2024-09-20T22:55:03.321955+0000","last_change":"2024-09-20T22:52:42.745105+0000","last_active":"2024-09-20T22:55:03.321955+0000","last_peered":"2024-09-20T22:55:03.321955+0000","last_clean":"2024-09-20T22:55:03.321955+0000","last_became_active":"2024-09-20T22:48:05.233687+0000","last_became_peered":"2024-09-20T22:48:05.233687+0000","last_unstale":"2024-09-20T22:55:03.321955+0000","last_undegraded":"2024-09-20T22:55:03.321955+0000","last_fullsized":"2024-09-20T22:55:03.321955+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:42.745032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:49:53.933601+0000","last_clean_scrub_stamp":"2024-09-20T22:52:42.745032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:03.693781+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"625'34","reported_seq":1671,"reported_epoch":823,"state":"active+clean","last_fresh":"2024-09-20T22:55:03.326985+0000","last_change":"2024-09-20T22:54:17.364170+0000","last_active":"2024-09-20T22:55:03.326985+0000","last_peered":"2024-09-20T22:55:03.326985+0000","last_clean":"2024-09-20T22:55:03.326985+0000","last_became_active":"2024-09-20T22:54:17.331250+0000","last_became_peered":"2024-09-20T22:54:17.331250+0000","last_unstale":"2024-09-20T22:55:03.326985+0000","last_undegraded":"2024-09-20T22:55:03.326985+0000","last_fullsized":"2024-09-20T22:55:03.326985+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'34","last_scrub_stamp":"2024-09-20T22:53:09.119791+0000","last_deep_scrub":"113'22","last_deep_scrub_stamp":"2024-09-20T22:43:12.191878+0000","last_clean_scrub_stamp":"2024-09-20T22:53:09.119791+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:04.232976+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":111,"reported_epoch":823,"state":"active+clean","last_fresh":"2024-09-20T22:55:03.474266+0000","last_change":"2024-09-20T22:54:41.984008+0000","last_active":"2024-09-20T22:55:03.474266+0000","last_peered":"2024-09-20T22:55:03.474266+0000","last_clean":"2024-09-20T22:55:03.474266+0000","last_became_active":"2024-09-20T22:54:17.349501+0000","last_became_peered":"2024-09-20T22:54:17.349501+0000","last_unstale":"2024-09-20T22:55:03.474266+0000","last_undegraded":"2024-09-20T22:55:03.474266+0000","last_fullsized":"2024-09-20T22:55:03.474266+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:54:41.983930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:50:40.784154+0000","last_clean_scrub_stamp":"2024-09-20T22:54:41.983930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:43.474654+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":1546,"reported_epoch":823,"state":"active+clean","last_fresh":"2024-09-20T22:55:03.324388+0000","last_change":"2024-09-20T22:54:02.131730+0000","last_active":"2024-09-20T22:55:03.324388+0000","last_peered":"2024-09-20T22:55:03.324388+0000","last_clean":"2024-09-20T22:55:03.324388+0000","last_became_active":"2024-09-20T22:41:44.832131+0000","last_became_peered":"2024-09-20T22:41:44.832131+0000","last_unstale":"2024-09-20T22:55:03.324388+0000","last_undegraded":"2024-09-20T22:55:03.324388+0000","last_fullsized":"2024-09-20T22:55:03.324388+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:54:02.131659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:45:25.002199+0000","last_clean_scrub_stamp":"2024-09-20T22:54:02.131659+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:08.048458+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":883,"reported_epoch":823,"state":"active+clean","last_fresh":"2024-09-20T22:55:03.318644+0000","last_change":"2024-09-20T22:54:17.340036+0000","last_active":"2024-09-20T22:55:03.318644+0000","last_peered":"2024-09-20T22:55:03.318644+0000","last_clean":"2024-09-20T22:55:03.318644+0000","last_became_active":"2024-09-20T22:54:17.339702+0000","last_became_peered":"2024-09-20T22:54:17.339702+0000","last_unstale":"2024-09-20T22:55:03.318644+0000","last_undegraded":"2024-09-20T22:55:03.318644+0000","last_fullsized":"2024-09-20T22:55:03.318644+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:04.583683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:50:41.885844+0000","last_clean_scrub_stamp":"2024-09-20T22:53:04.583683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:05.026677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"625'18","reported_seq":1675,"reported_epoch":823,"state":"active+clean","last_fresh":"2024-09-20T22:55:03.321815+0000","last_change":"2024-09-20T22:54:18.284372+0000","last_active":"2024-09-20T22:55:03.321815+0000","last_peered":"2024-09-20T22:55:03.321815+0000","last_clean":"2024-09-20T22:55:03.321815+0000","last_became_active":"2024-09-20T22:54:17.333427+0000","last_became_peered":"2024-09-20T22:54:17.333427+0000","last_unstale":"2024-09-20T22:55:03.321815+0000","last_undegraded":"2024-09-20T22:55:03.321815+0000","last_fullsized":"2024-09-20T22:55:03.321815+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'18","last_scrub_stamp":"2024-09-20T22:54:18.284289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:54:18.284289+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:22.283251+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":1100,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":871,"reported_epoch":823,"state":"active+clean","last_fresh":"2024-09-20T22:55:03.319657+0000","last_change":"2024-09-20T22:52:45.738376+0000","last_active":"2024-09-20T22:55:03.319657+0000","last_peered":"2024-09-20T22:55:03.319657+0000","last_clean":"2024-09-20T22:55:03.319657+0000","last_became_active":"2024-09-20T22:48:05.250319+0000","last_became_peered":"2024-09-20T22:48:05.250319+0000","last_unstale":"2024-09-20T22:55:03.319657+0000","last_undegraded":"2024-09-20T22:55:03.319657+0000","last_fullsized":"2024-09-20T22:55:03.319657+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:45.738275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:52:45.738275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:05.026632+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"154'2","reported_seq":1553,"reported_epoch":823,"state":"active+clean","last_fresh":"2024-09-20T22:55:03.326019+0000","last_change":"2024-09-20T22:52:45.578519+0000","last_active":"2024-09-20T22:55:03.326019+0000","last_peered":"2024-09-20T22:55:03.326019+0000","last_clean":"2024-09-20T22:55:03.326019+0000","last_became_active":"2024-09-20T22:41:44.199822+0000","last_became_peered":"2024-09-20T22:41:44.199822+0000","last_unstale":"2024-09-20T22:55:03.326019+0000","last_undegraded":"2024-09-20T22:55:03.326019+0000","last_fullsized":"2024-09-20T22:55:03.326019+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"154'2","last_scrub_stamp":"2024-09-20T22:52:45.578416+0000","last_deep_scrub":"154'2","last_deep_scrub_stamp":"2024-09-20T22:43:54.134066+0000","last_clean_scrub_stamp":"2024-09-20T22:52:45.578416+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:05.703495+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"67'1","reported_seq":115,"reported_epoch":823,"state":"active+clean","last_fresh":"2024-09-20T22:55:03.473643+0000","last_change":"2024-09-20T22:54:49.003707+0000","last_active":"2024-09-20T22:55:03.473643+0000","last_peered":"2024-09-20T22:55:03.473643+0000","last_clean":"2024-09-20T22:55:03.473643+0000","last_became_active":"2024-09-20T22:54:17.348571+0000","last_became_peered":"2024-09-20T22:54:17.348571+0000","last_unstale":"2024-09-20T22:55:03.473643+0000","last_undegraded":"2024-09-20T22:55:03.473643+0000","last_fullsized":"2024-09-20T22:55:03.473643+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-09-20T22:54:49.003633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:54:49.003633+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:58.739129+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"625'25","reported_seq":933,"reported_epoch":823,"state":"active+clean","last_fresh":"2024-09-20T22:55:03.323025+0000","last_change":"2024-09-20T22:53:26.146547+0000","last_active":"2024-09-20T22:55:03.323025+0000","last_peered":"2024-09-20T22:55:03.323025+0000","last_clean":"2024-09-20T22:55:03.323025+0000","last_became_active":"2024-09-20T22:47:49.232181+0000","last_became_peered":"2024-09-20T22:47:49.232181+0000","last_unstale":"2024-09-20T22:55:03.323025+0000","last_undegraded":"2024-09-20T22:55:03.323025+0000","last_fullsized":"2024-09-20T22:55:03.323025+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'25","last_scrub_stamp":"2024-09-20T22:53:26.146466+0000","last_deep_scrub":"151'15","last_deep_scrub_stamp":"2024-09-20T22:43:50.715761+0000","last_clean_scrub_stamp":"2024-09-20T22:53:26.146466+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:03.693859+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"661'11","reported_seq":1771,"reported_epoch":823,"state":"active+clean","last_fresh":"2024-09-20T22:55:03.332270+0000","last_change":"2024-09-20T22:53:34.944681+0000","last_active":"2024-09-20T22:55:03.332270+0000","last_peered":"2024-09-20T22:55:03.332270+0000","last_clean":"2024-09-20T22:55:03.332270+0000","last_became_active":"2024-09-20T22:48:05.244270+0000","last_became_peered":"2024-09-20T22:48:05.244270+0000","last_unstale":"2024-09-20T22:55:03.332270+0000","last_undegraded":"2024-09-20T22:55:03.332270+0000","last_fullsized":"2024-09-20T22:55:03.332270+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"661'11","last_scrub_stamp":"2024-09-20T22:53:34.944591+0000","last_deep_scrub":"661'11","last_deep_scrub_stamp":"2024-09-20T22:52:21.907943+0000","last_clean_scrub_stamp":"2024-09-20T22:53:34.944591+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:05.885306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.c","version":"820'2634","reported_seq":6134,"reported_epoch":823,"state":"active+clean","last_fresh":"2024-09-20T22:55:03.331701+0000","last_change":"2024-09-20T22:55:00.340647+0000","last_active":"2024-09-20T22:55:03.331701+0000","last_peered":"2024-09-20T22:55:03.331701+0000","last_clean":"2024-09-20T22:55:03.331701+0000","last_became_active":"2024-09-20T22:42:52.404027+0000","last_became_peered":"2024-09-20T22:42:52.404027+0000","last_unstale":"2024-09-20T22:55:03.331701+0000","last_undegraded":"2024-09-20T22:55:03.331701+0000","last_fullsized":"2024-09-20T22:55:03.331701+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"754'2388","last_scrub_stamp":"2024-09-20T22:53:54.900766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:53:54.900766+0000","objects_scrubbed":32,"log_size":2634,"log_dups_size":0,"ondisk_log_size":2634,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0.0068420069999999998,"stat_sum":{"num_bytes":54090374,"num_objects":27,"num_object_clones":20,"num_object_copies":81,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":3,"num_read":963,"num_read_kb":1018411,"num_write":643,"num_write_kb":250538,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"123","length":"b"},{"start":"130","length":"9"},{"start":"13b","length":"3"},{"start":"13f","length":"6"},{"start":"146","length":"3"},{"start":"14b","length":"3"},{"start":"150","length":"3"},{"start":"154","length":"1"},{"start":"159","length":"1"},{"start":"15b","length":"1"},{"start":"15d","length":"1"},{"start":"15f","length":"1"},{"start":"161","length":"5"},{"start":"167","length":"4"},{"start":"170","length":"1"}]},{"pgid":"1.a","version":"77'128","reported_seq":1826,"reported_epoch":823,"state":"active+clean","last_fresh":"2024-09-20T22:55:03.329491+0000","last_change":"2024-09-20T22:53:28.095343+0000","last_active":"2024-09-20T22:55:03.329491+0000","last_peered":"2024-09-20T22:55:03.329491+0000","last_clean":"2024-09-20T22:55:03.329491+0000","last_became_active":"2024-09-20T22:42:52.407791+0000","last_became_peered":"2024-09-20T22:42:52.407791+0000","last_unstale":"2024-09-20T22:55:03.329491+0000","last_undegraded":"2024-09-20T22:55:03.329491+0000","last_fullsized":"2024-09-20T22:55:03.329491+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:53:28.095266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:53:28.095266+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:07.223016+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"625'27","reported_seq":127,"reported_epoch":823,"state":"active+clean","last_fresh":"2024-09-20T22:55:03.474450+0000","last_change":"2024-09-20T22:54:22.820985+0000","last_active":"2024-09-20T22:55:03.474450+0000","last_peered":"2024-09-20T22:55:03.474450+0000","last_clean":"2024-09-20T22:55:03.474450+0000","last_became_active":"2024-09-20T22:54:17.347544+0000","last_became_peered":"2024-09-20T22:54:17.347544+0000","last_unstale":"2024-09-20T22:55:03.474450+0000","last_undegraded":"2024-09-20T22:55:03.474450+0000","last_fullsized":"2024-09-20T22:55:03.474450+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'27","last_scrub_stamp":"2024-09-20T22:54:22.820824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:54:22.820824+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:36.004359+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":1562,"reported_epoch":823,"state":"active+clean","last_fresh":"2024-09-20T22:55:03.321039+0000","last_change":"2024-09-20T22:54:17.342130+0000","last_active":"2024-09-20T22:55:03.321039+0000","last_peered":"2024-09-20T22:55:03.321039+0000","last_clean":"2024-09-20T22:55:03.321039+0000","last_became_active":"2024-09-20T22:54:17.341270+0000","last_became_peered":"2024-09-20T22:54:17.341270+0000","last_unstale":"2024-09-20T22:55:03.321039+0000","last_undegraded":"2024-09-20T22:55:03.321039+0000","last_fullsized":"2024-09-20T22:55:03.321039+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:29.394097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:49:12.070675+0000","last_clean_scrub_stamp":"2024-09-20T22:51:29.394097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:06.328989+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"408'12","reported_seq":960,"reported_epoch":823,"state":"active+clean","last_fresh":"2024-09-20T22:55:03.320041+0000","last_change":"2024-09-20T22:54:22.471855+0000","last_active":"2024-09-20T22:55:03.320041+0000","last_peered":"2024-09-20T22:55:03.320041+0000","last_clean":"2024-09-20T22:55:03.320041+0000","last_became_active":"2024-09-20T22:48:05.242914+0000","last_became_peered":"2024-09-20T22:48:05.242914+0000","last_unstale":"2024-09-20T22:55:03.320041+0000","last_undegraded":"2024-09-20T22:55:03.320041+0000","last_fullsized":"2024-09-20T22:55:03.320041+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"408'12","last_scrub_stamp":"2024-09-20T22:54:22.471772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:54:22.471772+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:33.991199+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":1577,"reported_epoch":823,"state":"active+clean","last_fresh":"2024-09-20T22:55:03.328788+0000","last_change":"2024-09-20T22:54:22.078071+0000","last_active":"2024-09-20T22:55:03.328788+0000","last_peered":"2024-09-20T22:55:03.328788+0000","last_clean":"2024-09-20T22:55:03.328788+0000","last_became_active":"2024-09-20T22:54:17.340327+0000","last_became_peered":"2024-09-20T22:54:17.340327+0000","last_unstale":"2024-09-20T22:55:03.328788+0000","last_undegraded":"2024-09-20T22:55:03.328788+0000","last_fullsized":"2024-09-20T22:55:03.328788+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:54:22.077994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:54:22.077994+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:29.125180+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3","version":"820'408","reported_seq":3902,"reported_epoch":823,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-20T22:55:03.322161+0000","last_change":"2024-09-20T22:55:02.416161+0000","last_active":"2024-09-20T22:55:03.322161+0000","last_peered":"2024-09-20T22:55:03.322161+0000","last_clean":"2024-09-20T22:55:03.322161+0000","last_became_active":"2024-09-20T22:49:05.221402+0000","last_became_peered":"2024-09-20T22:49:05.221402+0000","last_unstale":"2024-09-20T22:55:03.322161+0000","last_undegraded":"2024-09-20T22:55:03.322161+0000","last_fullsized":"2024-09-20T22:55:03.322161+0000","mapping_epoch":404,"log_start":"456'129","ondisk_log_start":"456'129","created":94,"last_epoch_clean":465,"parent":"0.0","parent_split_bits":0,"last_scrub":"609'243","last_scrub_stamp":"2024-09-20T22:51:31.402013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:51:31.402013+0000","objects_scrubbed":6,"log_size":279,"log_dups_size":149,"ondisk_log_size":279,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.086175030999999999,"stat_sum":{"num_bytes":15025132,"num_objects":8,"num_object_clones":7,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":124,"num_read_kb":153859,"num_write":91,"num_write_kb":30819,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":20196357,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"123","length":"b"},{"start":"130","length":"9"},{"start":"13b","length":"3"},{"start":"13f","length":"6"},{"start":"146","length":"3"},{"start":"14b","length":"3"},{"start":"150","length":"3"},{"start":"154","length":"1"},{"start":"159","length":"1"},{"start":"15b","length":"1"},{"start":"15d","length":"1"},{"start":"161","length":"5"},{"start":"168","length":"3"},{"start":"170","length":"1"}]},{"pgid":"1.5","version":"77'128","reported_seq":1732,"reported_epoch":823,"state":"active+clean","last_fresh":"2024-09-20T22:55:03.320007+0000","last_change":"2024-09-20T22:53:35.470320+0000","last_active":"2024-09-20T22:55:03.320007+0000","last_peered":"2024-09-20T22:55:03.320007+0000","last_clean":"2024-09-20T22:55:03.320007+0000","last_became_active":"2024-09-20T22:48:08.207018+0000","last_became_peered":"2024-09-20T22:48:08.207018+0000","last_unstale":"2024-09-20T22:55:03.320007+0000","last_undegraded":"2024-09-20T22:55:03.320007+0000","last_fullsized":"2024-09-20T22:55:03.320007+0000","mapping_epoch":407,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":408,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:53:35.470248+0000","last_deep_scrub":"77'128","last_deep_scrub_stamp":"2024-09-20T22:45:41.608075+0000","last_clean_scrub_stamp":"2024-09-20T22:53:35.470248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"625'30","reported_seq":1652,"reported_epoch":823,"state":"active+clean","last_fresh":"2024-09-20T22:55:03.321451+0000","last_change":"2024-09-20T22:53:48.346424+0000","last_active":"2024-09-20T22:55:03.321451+0000","last_peered":"2024-09-20T22:55:03.321451+0000","last_clean":"2024-09-20T22:55:03.321451+0000","last_became_active":"2024-09-20T22:47:49.217605+0000","last_became_peered":"2024-09-20T22:47:49.217605+0000","last_unstale":"2024-09-20T22:55:03.321451+0000","last_undegraded":"2024-09-20T22:55:03.321451+0000","last_fullsized":"2024-09-20T22:55:03.321451+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'30","last_scrub_stamp":"2024-09-20T22:53:48.346341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:53:48.346341+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":1570,"reported_epoch":823,"state":"active+clean","last_fresh":"2024-09-20T22:55:03.324731+0000","last_change":"2024-09-20T22:52:45.537243+0000","last_active":"2024-09-20T22:55:03.324731+0000","last_peered":"2024-09-20T22:55:03.324731+0000","last_clean":"2024-09-20T22:55:03.324731+0000","last_became_active":"2024-09-20T22:47:49.227069+0000","last_became_peered":"2024-09-20T22:47:49.227069+0000","last_unstale":"2024-09-20T22:55:03.324731+0000","last_undegraded":"2024-09-20T22:55:03.324731+0000","last_fullsized":"2024-09-20T22:55:03.324731+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:45.537173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:52:45.537173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:06.329195+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":872,"reported_epoch":823,"state":"active+clean","last_fresh":"2024-09-20T22:55:03.319331+0000","last_change":"2024-09-20T22:53:26.623524+0000","last_active":"2024-09-20T22:55:03.319331+0000","last_peered":"2024-09-20T22:55:03.319331+0000","last_clean":"2024-09-20T22:55:03.319331+0000","last_became_active":"2024-09-20T22:48:05.230003+0000","last_became_peered":"2024-09-20T22:48:05.230003+0000","last_unstale":"2024-09-20T22:55:03.319331+0000","last_undegraded":"2024-09-20T22:55:03.319331+0000","last_fullsized":"2024-09-20T22:55:03.319331+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:26.623455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:07.947945+0000","last_clean_scrub_stamp":"2024-09-20T22:53:26.623455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:05.026691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":864,"reported_epoch":823,"state":"active+clean","last_fresh":"2024-09-20T22:55:03.318606+0000","last_change":"2024-09-20T22:51:55.748527+0000","last_active":"2024-09-20T22:55:03.318606+0000","last_peered":"2024-09-20T22:55:03.318606+0000","last_clean":"2024-09-20T22:55:03.318606+0000","last_became_active":"2024-09-20T22:48:05.239917+0000","last_became_peered":"2024-09-20T22:48:05.239917+0000","last_unstale":"2024-09-20T22:55:03.318606+0000","last_undegraded":"2024-09-20T22:55:03.318606+0000","last_fullsized":"2024-09-20T22:55:03.318606+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:55.748425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:53.036601+0000","last_clean_scrub_stamp":"2024-09-20T22:51:55.748425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:05.026512+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2","version":"823'848","reported_seq":3674,"reported_epoch":823,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T22:55:03.471876+0000","last_change":"2024-09-20T22:55:01.469939+0000","last_active":"2024-09-20T22:55:03.471876+0000","last_peered":"2024-09-20T22:55:03.471876+0000","last_clean":"2024-09-20T22:55:03.471876+0000","last_became_active":"2024-09-20T22:54:50.348406+0000","last_became_peered":"2024-09-20T22:54:50.348406+0000","last_unstale":"2024-09-20T22:55:03.471876+0000","last_undegraded":"2024-09-20T22:55:03.471876+0000","last_fullsized":"2024-09-20T22:55:03.471876+0000","mapping_epoch":809,"log_start":"471'464","ondisk_log_start":"471'464","created":94,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":0,"last_scrub":"282'120","last_scrub_stamp":"2024-09-20T22:46:22.099642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:46:22.099642+0000","objects_scrubbed":1,"log_size":384,"log_dups_size":687,"ondisk_log_size":384,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":52,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.088870548999999993,"stat_sum":{"num_bytes":37768846,"num_objects":15,"num_object_clones":11,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":444,"num_read_kb":457671,"num_write":278,"num_write_kb":107616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":160,"num_bytes_recovered":146402851,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"123","length":"1"}]},{"pgid":"1.4","version":"77'128","reported_seq":1851,"reported_epoch":823,"state":"active+clean","last_fresh":"2024-09-20T22:55:03.324139+0000","last_change":"2024-09-20T22:52:02.474080+0000","last_active":"2024-09-20T22:55:03.324139+0000","last_peered":"2024-09-20T22:55:03.324139+0000","last_clean":"2024-09-20T22:55:03.324139+0000","last_became_active":"2024-09-20T22:46:15.181305+0000","last_became_peered":"2024-09-20T22:46:15.181305+0000","last_unstale":"2024-09-20T22:55:03.324139+0000","last_undegraded":"2024-09-20T22:55:03.324139+0000","last_fullsized":"2024-09-20T22:55:03.324139+0000","mapping_epoch":94,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:52:02.473979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:52:02.473979+0000","objects_scrubbed":1,"log_size":0,"log_dups_size":59,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:06.329096+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"625'28","reported_seq":909,"reported_epoch":823,"state":"active+clean","last_fresh":"2024-09-20T22:55:03.319246+0000","last_change":"2024-09-20T22:54:46.173616+0000","last_active":"2024-09-20T22:55:03.319246+0000","last_peered":"2024-09-20T22:55:03.319246+0000","last_clean":"2024-09-20T22:55:03.319246+0000","last_became_active":"2024-09-20T22:48:05.235812+0000","last_became_peered":"2024-09-20T22:48:05.235812+0000","last_unstale":"2024-09-20T22:55:03.319246+0000","last_undegraded":"2024-09-20T22:55:03.319246+0000","last_fullsized":"2024-09-20T22:55:03.319246+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'28","last_scrub_stamp":"2024-09-20T22:54:46.173524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:54:46.173524+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:57.634041+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":41,"num_read_kb":34,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"67'1","reported_seq":940,"reported_epoch":823,"state":"active+clean","last_fresh":"2024-09-20T22:55:03.322851+0000","last_change":"2024-09-20T22:54:17.366948+0000","last_active":"2024-09-20T22:55:03.322851+0000","last_peered":"2024-09-20T22:55:03.322851+0000","last_clean":"2024-09-20T22:55:03.322851+0000","last_became_active":"2024-09-20T22:54:17.340423+0000","last_became_peered":"2024-09-20T22:54:17.340423+0000","last_unstale":"2024-09-20T22:55:03.322851+0000","last_undegraded":"2024-09-20T22:55:03.322851+0000","last_fullsized":"2024-09-20T22:55:03.322851+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-09-20T22:53:32.995211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:53:32.995211+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:03.693914+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":230,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"778'16","reported_seq":1775,"reported_epoch":823,"state":"active+clean","last_fresh":"2024-09-20T22:55:03.321770+0000","last_change":"2024-09-20T22:54:17.375877+0000","last_active":"2024-09-20T22:55:03.321770+0000","last_peered":"2024-09-20T22:55:03.321770+0000","last_clean":"2024-09-20T22:55:03.321770+0000","last_became_active":"2024-09-20T22:54:17.341871+0000","last_became_peered":"2024-09-20T22:54:17.341871+0000","last_unstale":"2024-09-20T22:55:03.321770+0000","last_undegraded":"2024-09-20T22:55:03.321770+0000","last_fullsized":"2024-09-20T22:55:03.321770+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"600'13","last_scrub_stamp":"2024-09-20T22:51:32.448512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:51:32.448512+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:06.329019+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":1552,"reported_epoch":823,"state":"active+clean","last_fresh":"2024-09-20T22:55:03.325515+0000","last_change":"2024-09-20T22:53:02.610452+0000","last_active":"2024-09-20T22:55:03.325515+0000","last_peered":"2024-09-20T22:55:03.325515+0000","last_clean":"2024-09-20T22:55:03.325515+0000","last_became_active":"2024-09-20T22:41:44.831901+0000","last_became_peered":"2024-09-20T22:41:44.831901+0000","last_unstale":"2024-09-20T22:55:03.325515+0000","last_undegraded":"2024-09-20T22:55:03.325515+0000","last_fullsized":"2024-09-20T22:55:03.325515+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:02.610359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:53:02.610359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:05.703630+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1","version":"819'493","reported_seq":4085,"reported_epoch":823,"state":"active+clean","last_fresh":"2024-09-20T22:55:03.319005+0000","last_change":"2024-09-20T22:55:01.328916+0000","last_active":"2024-09-20T22:55:03.319005+0000","last_peered":"2024-09-20T22:55:03.319005+0000","last_clean":"2024-09-20T22:55:03.319005+0000","last_became_active":"2024-09-20T22:49:21.232750+0000","last_became_peered":"2024-09-20T22:49:21.232750+0000","last_unstale":"2024-09-20T22:55:03.319005+0000","last_undegraded":"2024-09-20T22:55:03.319005+0000","last_fullsized":"2024-09-20T22:55:03.319005+0000","mapping_epoch":407,"log_start":"474'259","ondisk_log_start":"474'259","created":94,"last_epoch_clean":481,"parent":"0.0","parent_split_bits":0,"last_scrub":"794'485","last_scrub_stamp":"2024-09-20T22:54:38.213249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:54:38.213249+0000","objects_scrubbed":8,"log_size":234,"log_dups_size":338,"ondisk_log_size":234,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:56:02.188183+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":1.0007689120000001,"stat_sum":{"num_bytes":10085879,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":135,"num_read_kb":151603,"num_write":101,"num_write_kb":51187,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":8732217,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"123","length":"b"},{"start":"130","length":"9"},{"start":"13b","length":"3"},{"start":"13f","length":"6"},{"start":"146","length":"3"},{"start":"14b","length":"3"},{"start":"150","length":"3"},{"start":"154","length":"1"},{"start":"159","length":"1"},{"start":"15b","length":"1"},{"start":"15d","length":"1"},{"start":"15f","length":"1"},{"start":"161","length":"5"},{"start":"167","length":"4"},{"start":"170","length":"1"}]},{"pgid":"1.7","version":"77'128","reported_seq":1872,"reported_epoch":823,"state":"active+clean","last_fresh":"2024-09-20T22:55:03.327323+0000","last_change":"2024-09-20T22:53:56.869885+0000","last_active":"2024-09-20T22:55:03.327323+0000","last_peered":"2024-09-20T22:55:03.327323+0000","last_clean":"2024-09-20T22:55:03.327323+0000","last_became_active":"2024-09-20T22:46:03.169596+0000","last_became_peered":"2024-09-20T22:46:03.169596+0000","last_unstale":"2024-09-20T22:55:03.327323+0000","last_undegraded":"2024-09-20T22:55:03.327323+0000","last_fullsized":"2024-09-20T22:55:03.327323+0000","mapping_epoch":94,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:53:56.869735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:53:56.869735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:09.756086+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"625'20","reported_seq":123,"reported_epoch":823,"state":"active+clean","last_fresh":"2024-09-20T22:55:03.473915+0000","last_change":"2024-09-20T22:54:39.933063+0000","last_active":"2024-09-20T22:55:03.473915+0000","last_peered":"2024-09-20T22:55:03.473915+0000","last_clean":"2024-09-20T22:55:03.473915+0000","last_became_active":"2024-09-20T22:54:17.349113+0000","last_became_peered":"2024-09-20T22:54:17.349113+0000","last_unstale":"2024-09-20T22:55:03.473915+0000","last_undegraded":"2024-09-20T22:55:03.473915+0000","last_fullsized":"2024-09-20T22:55:03.473915+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'20","last_scrub_stamp":"2024-09-20T22:54:39.932960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:54:39.932960+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:56:07.875833+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":28,"num_read_kb":23,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":1566,"reported_epoch":823,"state":"active+clean","last_fresh":"2024-09-20T22:55:03.329378+0000","last_change":"2024-09-20T22:52:46.062448+0000","last_active":"2024-09-20T22:55:03.329378+0000","last_peered":"2024-09-20T22:55:03.329378+0000","last_clean":"2024-09-20T22:55:03.329378+0000","last_became_active":"2024-09-20T22:41:38.155629+0000","last_became_peered":"2024-09-20T22:41:38.155629+0000","last_unstale":"2024-09-20T22:55:03.329378+0000","last_undegraded":"2024-09-20T22:55:03.329378+0000","last_fullsized":"2024-09-20T22:55:03.329378+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:46.062352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:52:46.062352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:07.222959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"733'12","reported_seq":999,"reported_epoch":823,"state":"active+clean","last_fresh":"2024-09-20T22:55:03.321906+0000","last_change":"2024-09-20T22:53:33.928576+0000","last_active":"2024-09-20T22:55:03.321906+0000","last_peered":"2024-09-20T22:55:03.321906+0000","last_clean":"2024-09-20T22:55:03.321906+0000","last_became_active":"2024-09-20T22:47:49.227264+0000","last_became_peered":"2024-09-20T22:47:49.227264+0000","last_unstale":"2024-09-20T22:55:03.321906+0000","last_undegraded":"2024-09-20T22:55:03.321906+0000","last_fullsized":"2024-09-20T22:55:03.321906+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"733'12","last_scrub_stamp":"2024-09-20T22:53:33.928499+0000","last_deep_scrub":"733'12","last_deep_scrub_stamp":"2024-09-20T22:53:33.928499+0000","last_clean_scrub_stamp":"2024-09-20T22:53:33.928499+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":906,"reported_epoch":823,"state":"active+clean","last_fresh":"2024-09-20T22:55:03.320841+0000","last_change":"2024-09-20T22:53:57.868607+0000","last_active":"2024-09-20T22:55:03.320841+0000","last_peered":"2024-09-20T22:55:03.320841+0000","last_clean":"2024-09-20T22:55:03.320841+0000","last_became_active":"2024-09-20T22:47:49.232380+0000","last_became_peered":"2024-09-20T22:47:49.232380+0000","last_unstale":"2024-09-20T22:55:03.320841+0000","last_undegraded":"2024-09-20T22:55:03.320841+0000","last_fullsized":"2024-09-20T22:55:03.320841+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:57.868534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:53:57.868534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:14.269112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.0","version":"823'833","reported_seq":4565,"reported_epoch":823,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T22:55:03.522420+0000","last_change":"2024-09-20T22:55:03.522420+0000","last_active":"2024-09-20T22:55:03.522420+0000","last_peered":"2024-09-20T22:55:03.522420+0000","last_clean":"2024-09-20T22:55:03.522420+0000","last_became_active":"2024-09-20T22:54:20.418799+0000","last_became_peered":"2024-09-20T22:54:20.418799+0000","last_unstale":"2024-09-20T22:55:03.522420+0000","last_undegraded":"2024-09-20T22:55:03.522420+0000","last_fullsized":"2024-09-20T22:55:03.522420+0000","mapping_epoch":779,"log_start":"480'253","ondisk_log_start":"480'253","created":94,"last_epoch_clean":780,"parent":"0.0","parent_split_bits":0,"last_scrub":"562'368","last_scrub_stamp":"2024-09-20T22:50:44.669910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:50:44.669910+0000","objects_scrubbed":15,"log_size":580,"log_dups_size":416,"ondisk_log_size":580,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.19412088599999999,"stat_sum":{"num_bytes":53078827,"num_objects":25,"num_object_clones":22,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":284,"num_read_kb":287300,"num_write":239,"num_write_kb":87886,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":54,"num_bytes_recovered":106304666,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"123","length":"b"},{"start":"130","length":"9"},{"start":"13b","length":"3"},{"start":"13f","length":"6"},{"start":"146","length":"3"},{"start":"14b","length":"3"},{"start":"150","length":"3"},{"start":"154","length":"1"},{"start":"159","length":"1"},{"start":"15b","length":"1"},{"start":"15d","length":"1"},{"start":"161","length":"4"},{"start":"168","length":"3"},{"start":"170","length":"1"}]},{"pgid":"1.6","version":"77'128","reported_seq":1894,"reported_epoch":823,"state":"active+clean","last_fresh":"2024-09-20T22:55:03.327658+0000","last_change":"2024-09-20T22:54:22.148323+0000","last_active":"2024-09-20T22:55:03.327658+0000","last_peered":"2024-09-20T22:55:03.327658+0000","last_clean":"2024-09-20T22:55:03.327658+0000","last_became_active":"2024-09-20T22:47:52.212190+0000","last_became_peered":"2024-09-20T22:47:52.212190+0000","last_unstale":"2024-09-20T22:55:03.327658+0000","last_undegraded":"2024-09-20T22:55:03.327658+0000","last_fullsized":"2024-09-20T22:55:03.327658+0000","mapping_epoch":391,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:54:22.148244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:54:22.148244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:46.700798+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"625'37","reported_seq":947,"reported_epoch":823,"state":"active+clean","last_fresh":"2024-09-20T22:55:03.321623+0000","last_change":"2024-09-20T22:54:46.870917+0000","last_active":"2024-09-20T22:55:03.321623+0000","last_peered":"2024-09-20T22:55:03.321623+0000","last_clean":"2024-09-20T22:55:03.321623+0000","last_became_active":"2024-09-20T22:47:49.222009+0000","last_became_peered":"2024-09-20T22:47:49.222009+0000","last_unstale":"2024-09-20T22:55:03.321623+0000","last_undegraded":"2024-09-20T22:55:03.321623+0000","last_fullsized":"2024-09-20T22:55:03.321623+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'37","last_scrub_stamp":"2024-09-20T22:54:46.870818+0000","last_deep_scrub":"515'27","last_deep_scrub_stamp":"2024-09-20T22:49:55.964153+0000","last_clean_scrub_stamp":"2024-09-20T22:54:46.870818+0000","objects_scrubbed":8,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:56:09.839466+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":53,"num_read_kb":39,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":103,"reported_epoch":823,"state":"active+clean","last_fresh":"2024-09-20T22:55:03.474173+0000","last_change":"2024-09-20T22:54:17.342966+0000","last_active":"2024-09-20T22:55:03.474173+0000","last_peered":"2024-09-20T22:55:03.474173+0000","last_clean":"2024-09-20T22:55:03.474173+0000","last_became_active":"2024-09-20T22:54:17.341248+0000","last_became_peered":"2024-09-20T22:54:17.341248+0000","last_unstale":"2024-09-20T22:55:03.474173+0000","last_undegraded":"2024-09-20T22:55:03.474173+0000","last_fullsized":"2024-09-20T22:55:03.474173+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:55.227211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:48:23.320031+0000","last_clean_scrub_stamp":"2024-09-20T22:53:55.227211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:23.492541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":1586,"reported_epoch":823,"state":"active+clean","last_fresh":"2024-09-20T22:55:03.327203+0000","last_change":"2024-09-20T22:53:57.882313+0000","last_active":"2024-09-20T22:55:03.327203+0000","last_peered":"2024-09-20T22:55:03.327203+0000","last_clean":"2024-09-20T22:55:03.327203+0000","last_became_active":"2024-09-20T22:47:49.214055+0000","last_became_peered":"2024-09-20T22:47:49.214055+0000","last_unstale":"2024-09-20T22:55:03.327203+0000","last_undegraded":"2024-09-20T22:55:03.327203+0000","last_fullsized":"2024-09-20T22:55:03.327203+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:57.882212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:53:57.882212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:12.731589+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":103,"reported_epoch":823,"state":"active+clean","last_fresh":"2024-09-20T22:55:03.473732+0000","last_change":"2024-09-20T22:54:17.340872+0000","last_active":"2024-09-20T22:55:03.473732+0000","last_peered":"2024-09-20T22:55:03.473732+0000","last_clean":"2024-09-20T22:55:03.473732+0000","last_became_active":"2024-09-20T22:54:17.340458+0000","last_became_peered":"2024-09-20T22:54:17.340458+0000","last_unstale":"2024-09-20T22:55:03.473732+0000","last_undegraded":"2024-09-20T22:55:03.473732+0000","last_fullsized":"2024-09-20T22:55:03.473732+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:54:15.141588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:48:41.208853+0000","last_clean_scrub_stamp":"2024-09-20T22:54:15.141588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:41.084042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.7","version":"822'1590","reported_seq":5257,"reported_epoch":823,"state":"active+clean","last_fresh":"2024-09-20T22:55:03.327759+0000","last_change":"2024-09-20T22:55:00.356243+0000","last_active":"2024-09-20T22:55:03.327759+0000","last_peered":"2024-09-20T22:55:03.327759+0000","last_clean":"2024-09-20T22:55:03.327759+0000","last_became_active":"2024-09-20T22:48:43.224386+0000","last_became_peered":"2024-09-20T22:48:43.224386+0000","last_unstale":"2024-09-20T22:55:03.327759+0000","last_undegraded":"2024-09-20T22:55:03.327759+0000","last_fullsized":"2024-09-20T22:55:03.327759+0000","mapping_epoch":391,"log_start":"436'456","ondisk_log_start":"436'456","created":94,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"772'1452","last_scrub_stamp":"2024-09-20T22:54:14.120176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:54:14.120176+0000","objects_scrubbed":23,"log_size":1134,"log_dups_size":886,"ondisk_log_size":1134,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:19.168499+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.022104342999999999,"stat_sum":{"num_bytes":57451132,"num_objects":24,"num_object_clones":19,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":1,"num_read":528,"num_read_kb":551441,"num_write":441,"num_write_kb":149267,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":115,"num_bytes_recovered":51615167,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"123","length":"b"},{"start":"130","length":"9"},{"start":"13b","length":"3"},{"start":"13f","length":"6"},{"start":"146","length":"3"},{"start":"14b","length":"3"},{"start":"150","length":"3"},{"start":"154","length":"1"},{"start":"159","length":"1"},{"start":"15b","length":"1"},{"start":"15d","length":"1"},{"start":"15f","length":"1"},{"start":"161","length":"5"},{"start":"167","length":"4"},{"start":"170","length":"1"}]},{"pgid":"1.1","version":"77'128","reported_seq":1874,"reported_epoch":823,"state":"active+clean","last_fresh":"2024-09-20T22:55:03.328664+0000","last_change":"2024-09-20T22:52:42.146147+0000","last_active":"2024-09-20T22:55:03.328664+0000","last_peered":"2024-09-20T22:55:03.328664+0000","last_clean":"2024-09-20T22:55:03.328664+0000","last_became_active":"2024-09-20T22:47:52.208102+0000","last_became_peered":"2024-09-20T22:47:52.208102+0000","last_unstale":"2024-09-20T22:55:03.328664+0000","last_undegraded":"2024-09-20T22:55:03.328664+0000","last_fullsized":"2024-09-20T22:55:03.328664+0000","mapping_epoch":391,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:52:42.146069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:52:42.146069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:07.222881+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"778'52","reported_seq":135,"reported_epoch":823,"state":"active+clean","last_fresh":"2024-09-20T22:55:03.474219+0000","last_change":"2024-09-20T22:54:17.373012+0000","last_active":"2024-09-20T22:55:03.474219+0000","last_peered":"2024-09-20T22:55:03.474219+0000","last_clean":"2024-09-20T22:55:03.474219+0000","last_became_active":"2024-09-20T22:54:17.342586+0000","last_became_peered":"2024-09-20T22:54:17.342586+0000","last_unstale":"2024-09-20T22:55:03.474219+0000","last_undegraded":"2024-09-20T22:55:03.474219+0000","last_fullsized":"2024-09-20T22:55:03.474219+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'51","last_scrub_stamp":"2024-09-20T22:53:57.184165+0000","last_deep_scrub":"86'38","last_deep_scrub_stamp":"2024-09-20T22:42:43.935677+0000","last_clean_scrub_stamp":"2024-09-20T22:53:57.184165+0000","objects_scrubbed":10,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:04.925929+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":864,"reported_epoch":823,"state":"active+clean","last_fresh":"2024-09-20T22:55:03.318931+0000","last_change":"2024-09-20T22:52:36.695537+0000","last_active":"2024-09-20T22:55:03.318931+0000","last_peered":"2024-09-20T22:55:03.318931+0000","last_clean":"2024-09-20T22:55:03.318931+0000","last_became_active":"2024-09-20T22:48:05.237756+0000","last_became_peered":"2024-09-20T22:48:05.237756+0000","last_unstale":"2024-09-20T22:55:03.318931+0000","last_undegraded":"2024-09-20T22:55:03.318931+0000","last_fullsized":"2024-09-20T22:55:03.318931+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:36.695446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:52:36.695446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:05.026558+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"69'2","reported_seq":1604,"reported_epoch":823,"state":"active+clean","last_fresh":"2024-09-20T22:55:03.329933+0000","last_change":"2024-09-20T22:54:23.097050+0000","last_active":"2024-09-20T22:55:03.329933+0000","last_peered":"2024-09-20T22:55:03.329933+0000","last_clean":"2024-09-20T22:55:03.329933+0000","last_became_active":"2024-09-20T22:54:17.341059+0000","last_became_peered":"2024-09-20T22:54:17.341059+0000","last_unstale":"2024-09-20T22:55:03.329933+0000","last_undegraded":"2024-09-20T22:55:03.329933+0000","last_fullsized":"2024-09-20T22:55:03.329933+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2024-09-20T22:54:23.096974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:36.130684+0000","last_clean_scrub_stamp":"2024-09-20T22:54:23.096974+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:40.124528+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00087280599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":924,"reported_epoch":823,"state":"active+clean","last_fresh":"2024-09-20T22:55:03.322181+0000","last_change":"2024-09-20T22:54:17.340261+0000","last_active":"2024-09-20T22:55:03.322181+0000","last_peered":"2024-09-20T22:55:03.322181+0000","last_clean":"2024-09-20T22:55:03.322181+0000","last_became_active":"2024-09-20T22:54:17.339860+0000","last_became_peered":"2024-09-20T22:54:17.339860+0000","last_unstale":"2024-09-20T22:55:03.322181+0000","last_undegraded":"2024-09-20T22:55:03.322181+0000","last_fullsized":"2024-09-20T22:55:03.322181+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:51.849772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:53:51.849772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:18.462456+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":1558,"reported_epoch":823,"state":"active+clean","last_fresh":"2024-09-20T22:55:03.322069+0000","last_change":"2024-09-20T22:52:21.363432+0000","last_active":"2024-09-20T22:55:03.322069+0000","last_peered":"2024-09-20T22:55:03.322069+0000","last_clean":"2024-09-20T22:55:03.322069+0000","last_became_active":"2024-09-20T22:48:05.234193+0000","last_became_peered":"2024-09-20T22:48:05.234193+0000","last_unstale":"2024-09-20T22:55:03.322069+0000","last_undegraded":"2024-09-20T22:55:03.322069+0000","last_fullsized":"2024-09-20T22:55:03.322069+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:21.363359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:52:21.363359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:06.329121+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"820'588","reported_seq":2734,"reported_epoch":823,"state":"active+clean","last_fresh":"2024-09-20T22:55:03.321027+0000","last_change":"2024-09-20T22:55:00.336888+0000","last_active":"2024-09-20T22:55:03.321027+0000","last_peered":"2024-09-20T22:55:03.321027+0000","last_clean":"2024-09-20T22:55:03.321027+0000","last_became_active":"2024-09-20T22:54:20.412859+0000","last_became_peered":"2024-09-20T22:54:20.412859+0000","last_unstale":"2024-09-20T22:55:03.321027+0000","last_undegraded":"2024-09-20T22:55:03.321027+0000","last_fullsized":"2024-09-20T22:55:03.321027+0000","mapping_epoch":779,"log_start":"456'176","ondisk_log_start":"456'176","created":94,"last_epoch_clean":780,"parent":"0.0","parent_split_bits":0,"last_scrub":"744'500","last_scrub_stamp":"2024-09-20T22:53:49.903129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:53:49.903129+0000","objects_scrubbed":10,"log_size":412,"log_dups_size":318,"ondisk_log_size":412,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:11.698411+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0068001909999999997,"stat_sum":{"num_bytes":17868981,"num_objects":8,"num_object_clones":6,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":1,"num_read":201,"num_read_kb":200920,"num_write":151,"num_write_kb":51652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":41680310,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"123","length":"b"},{"start":"130","length":"9"},{"start":"13b","length":"3"},{"start":"13f","length":"6"},{"start":"146","length":"3"},{"start":"14b","length":"3"},{"start":"150","length":"3"},{"start":"154","length":"1"},{"start":"159","length":"1"},{"start":"15b","length":"1"},{"start":"15d","length":"1"},{"start":"15f","length":"1"},{"start":"161","length":"5"},{"start":"167","length":"4"},{"start":"170","length":"1"}]},{"pgid":"1.2","version":"77'128","reported_seq":1942,"reported_epoch":823,"state":"active+clean","last_fresh":"2024-09-20T22:55:03.324953+0000","last_change":"2024-09-20T22:53:58.495293+0000","last_active":"2024-09-20T22:55:03.324953+0000","last_peered":"2024-09-20T22:55:03.324953+0000","last_clean":"2024-09-20T22:55:03.324953+0000","last_became_active":"2024-09-20T22:48:08.203028+0000","last_became_peered":"2024-09-20T22:48:08.203028+0000","last_unstale":"2024-09-20T22:55:03.324953+0000","last_undegraded":"2024-09-20T22:55:03.324953+0000","last_fullsized":"2024-09-20T22:55:03.324953+0000","mapping_epoch":407,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":408,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:53:58.495191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:53:58.495191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:23.083909+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"625'22","reported_seq":118,"reported_epoch":823,"state":"active+clean","last_fresh":"2024-09-20T22:55:03.474455+0000","last_change":"2024-09-20T22:54:53.462271+0000","last_active":"2024-09-20T22:55:03.474455+0000","last_peered":"2024-09-20T22:55:03.474455+0000","last_clean":"2024-09-20T22:55:03.474455+0000","last_became_active":"2024-09-20T22:54:17.349415+0000","last_became_peered":"2024-09-20T22:54:17.349415+0000","last_unstale":"2024-09-20T22:55:03.474455+0000","last_undegraded":"2024-09-20T22:55:03.474455+0000","last_fullsized":"2024-09-20T22:55:03.474455+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'22","last_scrub_stamp":"2024-09-20T22:53:35.117387+0000","last_deep_scrub":"443'18","last_deep_scrub_stamp":"2024-09-20T22:48:44.182517+0000","last_clean_scrub_stamp":"2024-09-20T22:53:35.117387+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:04.925851+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":24,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":1320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":111,"reported_epoch":823,"state":"active+clean","last_fresh":"2024-09-20T22:55:03.473862+0000","last_change":"2024-09-20T22:54:49.964373+0000","last_active":"2024-09-20T22:55:03.473862+0000","last_peered":"2024-09-20T22:55:03.473862+0000","last_clean":"2024-09-20T22:55:03.473862+0000","last_became_active":"2024-09-20T22:54:17.343806+0000","last_became_peered":"2024-09-20T22:54:17.343806+0000","last_unstale":"2024-09-20T22:55:03.473862+0000","last_undegraded":"2024-09-20T22:55:03.473862+0000","last_fullsized":"2024-09-20T22:55:03.473862+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:54:49.964301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:54:49.964301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:55.059202+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"380'2","reported_seq":1620,"reported_epoch":823,"state":"active+clean","last_fresh":"2024-09-20T22:55:03.326175+0000","last_change":"2024-09-20T22:54:16.438102+0000","last_active":"2024-09-20T22:55:03.326175+0000","last_peered":"2024-09-20T22:55:03.326175+0000","last_clean":"2024-09-20T22:55:03.326175+0000","last_became_active":"2024-09-20T22:47:49.222812+0000","last_became_peered":"2024-09-20T22:47:49.222812+0000","last_unstale":"2024-09-20T22:55:03.326175+0000","last_undegraded":"2024-09-20T22:55:03.326175+0000","last_fullsized":"2024-09-20T22:55:03.326175+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"380'2","last_scrub_stamp":"2024-09-20T22:54:16.437989+0000","last_deep_scrub":"380'2","last_deep_scrub_stamp":"2024-09-20T22:47:40.983039+0000","last_clean_scrub_stamp":"2024-09-20T22:54:16.437989+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:30.798230+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.001301459,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":1566,"reported_epoch":823,"state":"active+clean","last_fresh":"2024-09-20T22:55:03.327601+0000","last_change":"2024-09-20T22:53:25.223722+0000","last_active":"2024-09-20T22:55:03.327601+0000","last_peered":"2024-09-20T22:55:03.327601+0000","last_clean":"2024-09-20T22:55:03.327601+0000","last_became_active":"2024-09-20T22:41:42.185112+0000","last_became_peered":"2024-09-20T22:41:42.185112+0000","last_unstale":"2024-09-20T22:55:03.327601+0000","last_undegraded":"2024-09-20T22:55:03.327601+0000","last_fullsized":"2024-09-20T22:55:03.327601+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:25.223611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:47:22.881109+0000","last_clean_scrub_stamp":"2024-09-20T22:53:25.223611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:04.232994+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":1552,"reported_epoch":823,"state":"active+clean","last_fresh":"2024-09-20T22:55:03.325926+0000","last_change":"2024-09-20T22:53:06.644045+0000","last_active":"2024-09-20T22:55:03.325926+0000","last_peered":"2024-09-20T22:55:03.325926+0000","last_clean":"2024-09-20T22:55:03.325926+0000","last_became_active":"2024-09-20T22:41:44.830515+0000","last_became_peered":"2024-09-20T22:41:44.830515+0000","last_unstale":"2024-09-20T22:55:03.325926+0000","last_undegraded":"2024-09-20T22:55:03.325926+0000","last_fullsized":"2024-09-20T22:55:03.325926+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:06.643947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:53:06.643947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:05.703581+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"820'443","reported_seq":4140,"reported_epoch":823,"state":"active+clean","last_fresh":"2024-09-20T22:55:03.325752+0000","last_change":"2024-09-20T22:55:00.330467+0000","last_active":"2024-09-20T22:55:03.325752+0000","last_peered":"2024-09-20T22:55:03.325752+0000","last_clean":"2024-09-20T22:55:03.325752+0000","last_became_active":"2024-09-20T22:48:57.236417+0000","last_became_peered":"2024-09-20T22:48:57.236417+0000","last_unstale":"2024-09-20T22:55:03.325752+0000","last_undegraded":"2024-09-20T22:55:03.325752+0000","last_fullsized":"2024-09-20T22:55:03.325752+0000","mapping_epoch":391,"log_start":"450'190","ondisk_log_start":"450'190","created":94,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"774'413","last_scrub_stamp":"2024-09-20T22:54:15.568174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:54:15.568174+0000","objects_scrubbed":5,"log_size":253,"log_dups_size":240,"ondisk_log_size":253,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:24.868626+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.009936739,"stat_sum":{"num_bytes":10464115,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":99,"num_read_kb":89344,"num_write":100,"num_write_kb":40424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":19673726,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"123","length":"b"},{"start":"130","length":"9"},{"start":"13b","length":"3"},{"start":"13f","length":"6"},{"start":"146","length":"3"},{"start":"14b","length":"3"},{"start":"150","length":"3"},{"start":"154","length":"1"},{"start":"159","length":"1"},{"start":"15b","length":"1"},{"start":"15d","length":"1"},{"start":"15f","length":"1"},{"start":"161","length":"5"},{"start":"167","length":"4"},{"start":"170","length":"1"}]},{"pgid":"1.3","version":"77'128","reported_seq":1130,"reported_epoch":823,"state":"active+clean","last_fresh":"2024-09-20T22:55:03.474063+0000","last_change":"2024-09-20T22:54:44.971544+0000","last_active":"2024-09-20T22:55:03.474063+0000","last_peered":"2024-09-20T22:55:03.474063+0000","last_clean":"2024-09-20T22:55:03.474063+0000","last_became_active":"2024-09-20T22:54:20.413191+0000","last_became_peered":"2024-09-20T22:54:20.413191+0000","last_unstale":"2024-09-20T22:55:03.474063+0000","last_undegraded":"2024-09-20T22:55:03.474063+0000","last_fullsized":"2024-09-20T22:55:03.474063+0000","mapping_epoch":779,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":780,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:54:44.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:54:44.971463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:56:04.555885+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"625'24","reported_seq":970,"reported_epoch":823,"state":"active+clean","last_fresh":"2024-09-20T22:55:03.323106+0000","last_change":"2024-09-20T22:54:17.378683+0000","last_active":"2024-09-20T22:55:03.323106+0000","last_peered":"2024-09-20T22:55:03.323106+0000","last_clean":"2024-09-20T22:55:03.323106+0000","last_became_active":"2024-09-20T22:54:17.341751+0000","last_became_peered":"2024-09-20T22:54:17.341751+0000","last_unstale":"2024-09-20T22:55:03.323106+0000","last_undegraded":"2024-09-20T22:55:03.323106+0000","last_fullsized":"2024-09-20T22:55:03.323106+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'24","last_scrub_stamp":"2024-09-20T22:52:48.829882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:52:48.829882+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:03.693800+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":912,"reported_epoch":823,"state":"active+clean","last_fresh":"2024-09-20T22:55:03.321450+0000","last_change":"2024-09-20T22:53:25.172560+0000","last_active":"2024-09-20T22:55:03.321450+0000","last_peered":"2024-09-20T22:55:03.321450+0000","last_clean":"2024-09-20T22:55:03.321450+0000","last_became_active":"2024-09-20T22:48:05.233093+0000","last_became_peered":"2024-09-20T22:48:05.233093+0000","last_unstale":"2024-09-20T22:55:03.321450+0000","last_undegraded":"2024-09-20T22:55:03.321450+0000","last_fullsized":"2024-09-20T22:55:03.321450+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:25.172487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:44:19.834641+0000","last_clean_scrub_stamp":"2024-09-20T22:53:25.172487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:03.693843+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"75'5","reported_seq":2419,"reported_epoch":823,"state":"active+clean","last_fresh":"2024-09-20T22:55:03.326088+0000","last_change":"2024-09-20T22:54:48.577002+0000","last_active":"2024-09-20T22:55:03.326088+0000","last_peered":"2024-09-20T22:55:03.326088+0000","last_clean":"2024-09-20T22:55:03.326088+0000","last_became_active":"2024-09-20T22:54:17.337205+0000","last_became_peered":"2024-09-20T22:54:17.337205+0000","last_unstale":"2024-09-20T22:55:03.326088+0000","last_undegraded":"2024-09-20T22:55:03.326088+0000","last_fullsized":"2024-09-20T22:55:03.326088+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-09-20T22:54:48.576929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:36.130684+0000","last_clean_scrub_stamp":"2024-09-20T22:54:48.576929+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:56:05.805706+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00078631700000000003,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":794,"num_read_kb":789,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1556,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":1590,"reported_epoch":823,"state":"active+clean","last_fresh":"2024-09-20T22:55:03.329549+0000","last_change":"2024-09-20T22:54:51.175204+0000","last_active":"2024-09-20T22:55:03.329549+0000","last_peered":"2024-09-20T22:55:03.329549+0000","last_clean":"2024-09-20T22:55:03.329549+0000","last_became_active":"2024-09-20T22:54:17.339805+0000","last_became_peered":"2024-09-20T22:54:17.339805+0000","last_unstale":"2024-09-20T22:55:03.329549+0000","last_undegraded":"2024-09-20T22:55:03.329549+0000","last_fullsized":"2024-09-20T22:55:03.329549+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:54:51.175120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:54:51.175120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:56:09.146766+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":103,"reported_epoch":823,"state":"active+clean","last_fresh":"2024-09-20T22:55:03.474080+0000","last_change":"2024-09-20T22:54:17.339943+0000","last_active":"2024-09-20T22:55:03.474080+0000","last_peered":"2024-09-20T22:55:03.474080+0000","last_clean":"2024-09-20T22:55:03.474080+0000","last_became_active":"2024-09-20T22:54:17.339564+0000","last_became_peered":"2024-09-20T22:54:17.339564+0000","last_unstale":"2024-09-20T22:55:03.474080+0000","last_undegraded":"2024-09-20T22:55:03.474080+0000","last_fullsized":"2024-09-20T22:55:03.474080+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:49.031912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:53:49.031912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:06.466960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.6","version":"820'634","reported_seq":4247,"reported_epoch":823,"state":"active+clean","last_fresh":"2024-09-20T22:55:03.323156+0000","last_change":"2024-09-20T22:55:00.352444+0000","last_active":"2024-09-20T22:55:03.323156+0000","last_peered":"2024-09-20T22:55:03.323156+0000","last_clean":"2024-09-20T22:55:03.323156+0000","last_became_active":"2024-09-20T22:48:47.234531+0000","last_became_peered":"2024-09-20T22:48:47.234531+0000","last_unstale":"2024-09-20T22:55:03.323156+0000","last_undegraded":"2024-09-20T22:55:03.323156+0000","last_fullsized":"2024-09-20T22:55:03.323156+0000","mapping_epoch":391,"log_start":"443'181","ondisk_log_start":"443'181","created":94,"last_epoch_clean":447,"parent":"0.0","parent_split_bits":0,"last_scrub":"770'553","last_scrub_stamp":"2024-09-20T22:54:13.963191+0000","last_deep_scrub":"440'177","last_deep_scrub_stamp":"2024-09-20T22:48:40.602342+0000","last_clean_scrub_stamp":"2024-09-20T22:54:13.963191+0000","objects_scrubbed":13,"log_size":453,"log_dups_size":314,"ondisk_log_size":453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:22.037299+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.021575460000000001,"stat_sum":{"num_bytes":37038059,"num_objects":16,"num_object_clones":14,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":216,"num_read_kb":223022,"num_write":170,"num_write_kb":69049,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":18662529,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"123","length":"b"},{"start":"130","length":"9"},{"start":"13b","length":"3"},{"start":"13f","length":"6"},{"start":"146","length":"3"},{"start":"14b","length":"3"},{"start":"150","length":"3"},{"start":"154","length":"1"},{"start":"159","length":"1"},{"start":"15b","length":"1"},{"start":"15d","length":"1"},{"start":"15f","length":"1"},{"start":"161","length":"5"},{"start":"167","length":"4"},{"start":"170","length":"1"}]},{"pgid":"4.5","version":"625'32","reported_seq":1876,"reported_epoch":823,"state":"active+clean","last_fresh":"2024-09-20T22:55:03.327901+0000","last_change":"2024-09-20T22:54:41.994229+0000","last_active":"2024-09-20T22:55:03.327901+0000","last_peered":"2024-09-20T22:55:03.327901+0000","last_clean":"2024-09-20T22:55:03.327901+0000","last_became_active":"2024-09-20T22:48:05.243523+0000","last_became_peered":"2024-09-20T22:48:05.243523+0000","last_unstale":"2024-09-20T22:55:03.327901+0000","last_undegraded":"2024-09-20T22:55:03.327901+0000","last_fullsized":"2024-09-20T22:55:03.327901+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'32","last_scrub_stamp":"2024-09-20T22:54:41.994141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:54:41.994141+0000","objects_scrubbed":8,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:56:06.164224+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"67'1","reported_seq":1556,"reported_epoch":823,"state":"active+clean","last_fresh":"2024-09-20T22:55:03.323590+0000","last_change":"2024-09-20T22:51:35.440516+0000","last_active":"2024-09-20T22:55:03.323590+0000","last_peered":"2024-09-20T22:55:03.323590+0000","last_clean":"2024-09-20T22:55:03.323590+0000","last_became_active":"2024-09-20T22:41:38.158917+0000","last_became_peered":"2024-09-20T22:41:38.158917+0000","last_unstale":"2024-09-20T22:55:03.323590+0000","last_undegraded":"2024-09-20T22:55:03.323590+0000","last_fullsized":"2024-09-20T22:55:03.323590+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-09-20T22:51:35.440416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:51:35.440416+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:06.329068+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"77'128","reported_seq":1914,"reported_epoch":823,"state":"active+clean","last_fresh":"2024-09-20T22:55:03.326177+0000","last_change":"2024-09-20T22:54:21.616304+0000","last_active":"2024-09-20T22:55:03.326177+0000","last_peered":"2024-09-20T22:55:03.326177+0000","last_clean":"2024-09-20T22:55:03.326177+0000","last_became_active":"2024-09-20T22:47:52.206169+0000","last_became_peered":"2024-09-20T22:47:52.206169+0000","last_unstale":"2024-09-20T22:55:03.326177+0000","last_undegraded":"2024-09-20T22:55:03.326177+0000","last_fullsized":"2024-09-20T22:55:03.326177+0000","mapping_epoch":391,"log_start":"77'128","ondisk_log_start":"77'128","created":19,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:54:21.616231+0000","last_deep_scrub":"77'128","last_deep_scrub_stamp":"2024-09-20T22:47:22.417099+0000","last_clean_scrub_stamp":"2024-09-20T22:54:21.616231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:23.588140+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":1564,"reported_epoch":823,"state":"active+clean","last_fresh":"2024-09-20T22:55:03.326082+0000","last_change":"2024-09-20T22:53:03.564792+0000","last_active":"2024-09-20T22:55:03.326082+0000","last_peered":"2024-09-20T22:55:03.326082+0000","last_clean":"2024-09-20T22:55:03.326082+0000","last_became_active":"2024-09-20T22:41:42.184387+0000","last_became_peered":"2024-09-20T22:41:42.184387+0000","last_unstale":"2024-09-20T22:55:03.326082+0000","last_undegraded":"2024-09-20T22:55:03.326082+0000","last_fullsized":"2024-09-20T22:55:03.326082+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:03.564726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:53:03.564726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:05.703605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":1566,"reported_epoch":823,"state":"active+clean","last_fresh":"2024-09-20T22:55:03.329382+0000","last_change":"2024-09-20T22:53:09.957459+0000","last_active":"2024-09-20T22:55:03.329382+0000","last_peered":"2024-09-20T22:55:03.329382+0000","last_clean":"2024-09-20T22:55:03.329382+0000","last_became_active":"2024-09-20T22:48:05.241602+0000","last_became_peered":"2024-09-20T22:48:05.241602+0000","last_unstale":"2024-09-20T22:55:03.329382+0000","last_undegraded":"2024-09-20T22:55:03.329382+0000","last_fullsized":"2024-09-20T22:55:03.329382+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:09.957374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:53:09.957374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:07.222932+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"821'714","reported_seq":4529,"reported_epoch":823,"state":"active+clean","last_fresh":"2024-09-20T22:55:03.327551+0000","last_change":"2024-09-20T22:55:00.350699+0000","last_active":"2024-09-20T22:55:03.327551+0000","last_peered":"2024-09-20T22:55:03.327551+0000","last_clean":"2024-09-20T22:55:03.327551+0000","last_became_active":"2024-09-20T22:54:20.414528+0000","last_became_peered":"2024-09-20T22:54:20.414528+0000","last_unstale":"2024-09-20T22:55:03.327551+0000","last_undegraded":"2024-09-20T22:55:03.327551+0000","last_fullsized":"2024-09-20T22:55:03.327551+0000","mapping_epoch":779,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":780,"parent":"0.0","parent_split_bits":0,"last_scrub":"770'632","last_scrub_stamp":"2024-09-20T22:54:13.158459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:54:13.158459+0000","objects_scrubbed":10,"log_size":714,"log_dups_size":0,"ondisk_log_size":714,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:40.831814+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.016733624999999999,"stat_sum":{"num_bytes":25009709,"num_objects":12,"num_object_clones":10,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":215,"num_read_kb":213362,"num_write":126,"num_write_kb":55468,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":88,"num_bytes_recovered":51113242,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"123","length":"b"},{"start":"130","length":"9"},{"start":"13b","length":"3"},{"start":"13f","length":"6"},{"start":"146","length":"3"},{"start":"14b","length":"3"},{"start":"150","length":"3"},{"start":"154","length":"1"},{"start":"159","length":"1"},{"start":"15b","length":"1"},{"start":"15d","length":"1"},{"start":"15f","length":"1"},{"start":"161","length":"5"},{"start":"167","length":"4"},{"start":"170","length":"1"}]},{"pgid":"4.e","version":"625'19","reported_seq":924,"reported_epoch":823,"state":"active+clean","last_fresh":"2024-09-20T22:55:03.320053+0000","last_change":"2024-09-20T22:54:17.386152+0000","last_active":"2024-09-20T22:55:03.320053+0000","last_peered":"2024-09-20T22:55:03.320053+0000","last_clean":"2024-09-20T22:55:03.320053+0000","last_became_active":"2024-09-20T22:54:17.352687+0000","last_became_peered":"2024-09-20T22:54:17.352687+0000","last_unstale":"2024-09-20T22:55:03.320053+0000","last_undegraded":"2024-09-20T22:55:03.320053+0000","last_fullsized":"2024-09-20T22:55:03.320053+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'19","last_scrub_stamp":"2024-09-20T22:52:44.782975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:52:44.782975+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:05.026588+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":872,"reported_epoch":823,"state":"active+clean","last_fresh":"2024-09-20T22:55:03.318342+0000","last_change":"2024-09-20T22:54:21.423843+0000","last_active":"2024-09-20T22:55:03.318342+0000","last_peered":"2024-09-20T22:55:03.318342+0000","last_clean":"2024-09-20T22:55:03.318342+0000","last_became_active":"2024-09-20T22:48:05.254688+0000","last_became_peered":"2024-09-20T22:48:05.254688+0000","last_unstale":"2024-09-20T22:55:03.318342+0000","last_undegraded":"2024-09-20T22:55:03.318342+0000","last_fullsized":"2024-09-20T22:55:03.318342+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:54:21.423766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:54:21.423766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:38.441270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":1569,"reported_epoch":823,"state":"active+clean","last_fresh":"2024-09-20T22:55:03.329286+0000","last_change":"2024-09-20T22:54:47.166950+0000","last_active":"2024-09-20T22:55:03.329286+0000","last_peered":"2024-09-20T22:55:03.329286+0000","last_clean":"2024-09-20T22:55:03.329286+0000","last_became_active":"2024-09-20T22:48:05.233778+0000","last_became_peered":"2024-09-20T22:48:05.233778+0000","last_unstale":"2024-09-20T22:55:03.329286+0000","last_undegraded":"2024-09-20T22:55:03.329286+0000","last_fullsized":"2024-09-20T22:55:03.329286+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:54:47.166836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:54:47.166836+0000","last_clean_scrub_stamp":"2024-09-20T22:54:47.166836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:56:01.866772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":1545,"reported_epoch":823,"state":"active+clean","last_fresh":"2024-09-20T22:55:03.324103+0000","last_change":"2024-09-20T22:53:10.647075+0000","last_active":"2024-09-20T22:55:03.324103+0000","last_peered":"2024-09-20T22:55:03.324103+0000","last_clean":"2024-09-20T22:55:03.324103+0000","last_became_active":"2024-09-20T22:41:44.832009+0000","last_became_peered":"2024-09-20T22:41:44.832009+0000","last_unstale":"2024-09-20T22:55:03.324103+0000","last_undegraded":"2024-09-20T22:55:03.324103+0000","last_fullsized":"2024-09-20T22:55:03.324103+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:10.647005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:48:40.127123+0000","last_clean_scrub_stamp":"2024-09-20T22:53:10.647005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:06.329273+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.e","version":"820'857","reported_seq":4518,"reported_epoch":823,"state":"active+clean","last_fresh":"2024-09-20T22:55:03.327539+0000","last_change":"2024-09-20T22:55:00.332373+0000","last_active":"2024-09-20T22:55:03.327539+0000","last_peered":"2024-09-20T22:55:03.327539+0000","last_clean":"2024-09-20T22:55:03.327539+0000","last_became_active":"2024-09-20T22:47:52.209452+0000","last_became_peered":"2024-09-20T22:47:52.209452+0000","last_unstale":"2024-09-20T22:55:03.327539+0000","last_undegraded":"2024-09-20T22:55:03.327539+0000","last_fullsized":"2024-09-20T22:55:03.327539+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"778'821","last_scrub_stamp":"2024-09-20T22:54:19.513024+0000","last_deep_scrub":"174'60","last_deep_scrub_stamp":"2024-09-20T22:44:18.301776+0000","last_clean_scrub_stamp":"2024-09-20T22:54:19.513024+0000","objects_scrubbed":13,"log_size":857,"log_dups_size":0,"ondisk_log_size":857,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:36.839334+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0.0092817999999999998,"stat_sum":{"num_bytes":25021295,"num_objects":12,"num_object_clones":10,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":1,"num_read":257,"num_read_kb":256138,"num_write":193,"num_write_kb":69325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":23110741,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"123","length":"b"},{"start":"130","length":"9"},{"start":"13b","length":"3"},{"start":"13f","length":"6"},{"start":"146","length":"3"},{"start":"14b","length":"3"},{"start":"150","length":"3"},{"start":"154","length":"1"},{"start":"159","length":"1"},{"start":"15b","length":"1"},{"start":"15d","length":"1"},{"start":"15f","length":"1"},{"start":"161","length":"5"},{"start":"167","length":"4"},{"start":"170","length":"1"}]},{"pgid":"1.8","version":"77'128","reported_seq":1834,"reported_epoch":823,"state":"active+clean","last_fresh":"2024-09-20T22:55:03.328295+0000","last_change":"2024-09-20T22:54:46.128026+0000","last_active":"2024-09-20T22:55:03.328295+0000","last_peered":"2024-09-20T22:55:03.328295+0000","last_clean":"2024-09-20T22:55:03.328295+0000","last_became_active":"2024-09-20T22:42:52.405594+0000","last_became_peered":"2024-09-20T22:42:52.405594+0000","last_unstale":"2024-09-20T22:55:03.328295+0000","last_undegraded":"2024-09-20T22:55:03.328295+0000","last_fullsized":"2024-09-20T22:55:03.328295+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:54:46.127943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:54:46.127943+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:49.849638+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"625'31","reported_seq":959,"reported_epoch":823,"state":"active+clean","last_fresh":"2024-09-20T22:55:03.319941+0000","last_change":"2024-09-20T22:54:17.396085+0000","last_active":"2024-09-20T22:55:03.319941+0000","last_peered":"2024-09-20T22:55:03.319941+0000","last_clean":"2024-09-20T22:55:03.319941+0000","last_became_active":"2024-09-20T22:54:17.353293+0000","last_became_peered":"2024-09-20T22:54:17.353293+0000","last_unstale":"2024-09-20T22:55:03.319941+0000","last_undegraded":"2024-09-20T22:55:03.319941+0000","last_fullsized":"2024-09-20T22:55:03.319941+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'31","last_scrub_stamp":"2024-09-20T22:52:13.620754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:52:13.620754+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:05.026544+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":49,"num_read_kb":39,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":1135,"reported_epoch":823,"state":"active+clean","last_fresh":"2024-09-20T22:55:03.326023+0000","last_change":"2024-09-20T22:54:51.323936+0000","last_active":"2024-09-20T22:55:03.326023+0000","last_peered":"2024-09-20T22:55:03.326023+0000","last_clean":"2024-09-20T22:55:03.326023+0000","last_became_active":"2024-09-20T22:54:17.332636+0000","last_became_peered":"2024-09-20T22:54:17.332636+0000","last_unstale":"2024-09-20T22:55:03.326023+0000","last_undegraded":"2024-09-20T22:55:03.326023+0000","last_fullsized":"2024-09-20T22:55:03.326023+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:54:51.323855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:35.492562+0000","last_clean_scrub_stamp":"2024-09-20T22:54:51.323855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:58.729458+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":103,"reported_epoch":823,"state":"active+clean","last_fresh":"2024-09-20T22:55:03.474107+0000","last_change":"2024-09-20T22:54:17.349436+0000","last_active":"2024-09-20T22:55:03.474107+0000","last_peered":"2024-09-20T22:55:03.474107+0000","last_clean":"2024-09-20T22:55:03.474107+0000","last_became_active":"2024-09-20T22:54:17.349144+0000","last_became_peered":"2024-09-20T22:54:17.349144+0000","last_unstale":"2024-09-20T22:55:03.474107+0000","last_undegraded":"2024-09-20T22:55:03.474107+0000","last_fullsized":"2024-09-20T22:55:03.474107+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:57.442770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:53:57.442770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:04.925884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":1589,"reported_epoch":823,"state":"active+clean","last_fresh":"2024-09-20T22:55:03.327198+0000","last_change":"2024-09-20T22:54:41.501615+0000","last_active":"2024-09-20T22:55:03.327198+0000","last_peered":"2024-09-20T22:55:03.327198+0000","last_clean":"2024-09-20T22:55:03.327198+0000","last_became_active":"2024-09-20T22:48:05.239006+0000","last_became_peered":"2024-09-20T22:48:05.239006+0000","last_unstale":"2024-09-20T22:55:03.327198+0000","last_undegraded":"2024-09-20T22:55:03.327198+0000","last_fullsized":"2024-09-20T22:55:03.327198+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:54:41.501549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:54:41.501549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:54.722407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"820'925","reported_seq":4581,"reported_epoch":823,"state":"active+clean","last_fresh":"2024-09-20T22:55:03.323939+0000","last_change":"2024-09-20T22:55:00.354138+0000","last_active":"2024-09-20T22:55:03.323939+0000","last_peered":"2024-09-20T22:55:03.323939+0000","last_clean":"2024-09-20T22:55:03.323939+0000","last_became_active":"2024-09-20T22:47:52.216198+0000","last_became_peered":"2024-09-20T22:47:52.216198+0000","last_unstale":"2024-09-20T22:55:03.323939+0000","last_undegraded":"2024-09-20T22:55:03.323939+0000","last_fullsized":"2024-09-20T22:55:03.323939+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"754'838","last_scrub_stamp":"2024-09-20T22:53:54.908987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:53:54.908987+0000","objects_scrubbed":17,"log_size":925,"log_dups_size":0,"ondisk_log_size":925,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.022448103000000001,"stat_sum":{"num_bytes":22615031,"num_objects":12,"num_object_clones":9,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":1,"num_read":375,"num_read_kb":369621,"num_write":218,"num_write_kb":73285,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":53,"num_bytes_recovered":14983556,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"123","length":"b"},{"start":"130","length":"9"},{"start":"13b","length":"3"},{"start":"13f","length":"6"},{"start":"146","length":"3"},{"start":"14b","length":"3"},{"start":"150","length":"3"},{"start":"154","length":"1"},{"start":"159","length":"1"},{"start":"15b","length":"1"},{"start":"15d","length":"1"},{"start":"15f","length":"1"},{"start":"161","length":"5"},{"start":"167","length":"4"},{"start":"170","length":"1"}]},{"pgid":"1.9","version":"77'128","reported_seq":1884,"reported_epoch":822,"state":"peering","last_fresh":"2024-09-20T22:55:02.419508+0000","last_change":"2024-09-20T22:55:03.325425+0000","last_active":"2024-09-20T22:55:03.325425+0000","last_peered":"2024-09-20T22:55:02.419508+0000","last_clean":"2024-09-20T22:54:59.328145+0000","last_became_active":"2024-09-20T22:55:02.419216+0000","last_became_peered":"2024-09-20T22:55:02.419216+0000","last_unstale":"2024-09-20T22:55:02.419508+0000","last_undegraded":"2024-09-20T22:55:02.419508+0000","last_fullsized":"2024-09-20T22:55:02.327822+0000","mapping_epoch":823,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":562,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:53:50.381052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:53:50.381052+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,3,5],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"625'20","reported_seq":891,"reported_epoch":823,"state":"active+clean","last_fresh":"2024-09-20T22:55:03.319717+0000","last_change":"2024-09-20T22:52:22.646547+0000","last_active":"2024-09-20T22:55:03.319717+0000","last_peered":"2024-09-20T22:55:03.319717+0000","last_clean":"2024-09-20T22:55:03.319717+0000","last_became_active":"2024-09-20T22:48:05.242811+0000","last_became_peered":"2024-09-20T22:48:05.242811+0000","last_unstale":"2024-09-20T22:55:03.319717+0000","last_undegraded":"2024-09-20T22:55:03.319717+0000","last_fullsized":"2024-09-20T22:55:03.319717+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'20","last_scrub_stamp":"2024-09-20T22:52:22.646468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:52:22.646468+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:05.026604+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":1581,"reported_epoch":823,"state":"active+clean","last_fresh":"2024-09-20T22:55:03.324181+0000","last_change":"2024-09-20T22:52:44.527004+0000","last_active":"2024-09-20T22:55:03.324181+0000","last_peered":"2024-09-20T22:55:03.324181+0000","last_clean":"2024-09-20T22:55:03.324181+0000","last_became_active":"2024-09-20T22:48:05.233999+0000","last_became_peered":"2024-09-20T22:48:05.233999+0000","last_unstale":"2024-09-20T22:55:03.324181+0000","last_undegraded":"2024-09-20T22:55:03.324181+0000","last_fullsized":"2024-09-20T22:55:03.324181+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:44.526935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:52:44.526935+0000","last_clean_scrub_stamp":"2024-09-20T22:52:44.526935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:06.329170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"261'9","reported_seq":1744,"reported_epoch":823,"state":"active+clean","last_fresh":"2024-09-20T22:55:03.327099+0000","last_change":"2024-09-20T22:53:54.190905+0000","last_active":"2024-09-20T22:55:03.327099+0000","last_peered":"2024-09-20T22:55:03.327099+0000","last_clean":"2024-09-20T22:55:03.327099+0000","last_became_active":"2024-09-20T22:41:42.185863+0000","last_became_peered":"2024-09-20T22:41:42.185863+0000","last_unstale":"2024-09-20T22:55:03.327099+0000","last_undegraded":"2024-09-20T22:55:03.327099+0000","last_fullsized":"2024-09-20T22:55:03.327099+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'9","last_scrub_stamp":"2024-09-20T22:53:54.190823+0000","last_deep_scrub":"261'9","last_deep_scrub_stamp":"2024-09-20T22:46:50.974735+0000","last_clean_scrub_stamp":"2024-09-20T22:53:54.190823+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:22.929855+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":111,"reported_epoch":823,"state":"active+clean","last_fresh":"2024-09-20T22:55:03.474318+0000","last_change":"2024-09-20T22:54:43.022989+0000","last_active":"2024-09-20T22:55:03.474318+0000","last_peered":"2024-09-20T22:55:03.474318+0000","last_clean":"2024-09-20T22:55:03.474318+0000","last_became_active":"2024-09-20T22:54:17.340684+0000","last_became_peered":"2024-09-20T22:54:17.340684+0000","last_unstale":"2024-09-20T22:55:03.474318+0000","last_undegraded":"2024-09-20T22:55:03.474318+0000","last_fullsized":"2024-09-20T22:55:03.474318+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:54:43.022914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:54:43.022914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:56:01.405842+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.8","version":"820'1120","reported_seq":4598,"reported_epoch":823,"state":"active+clean","last_fresh":"2024-09-20T22:55:03.326544+0000","last_change":"2024-09-20T22:55:00.331437+0000","last_active":"2024-09-20T22:55:03.326544+0000","last_peered":"2024-09-20T22:55:03.326544+0000","last_clean":"2024-09-20T22:55:03.326544+0000","last_became_active":"2024-09-20T22:48:31.224359+0000","last_became_peered":"2024-09-20T22:48:31.224359+0000","last_unstale":"2024-09-20T22:55:03.326544+0000","last_undegraded":"2024-09-20T22:55:03.326544+0000","last_fullsized":"2024-09-20T22:55:03.326544+0000","mapping_epoch":407,"log_start":"426'268","ondisk_log_start":"426'268","created":94,"last_epoch_clean":431,"parent":"0.0","parent_split_bits":0,"last_scrub":"774'1005","last_scrub_stamp":"2024-09-20T22:54:15.466690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:54:15.466690+0000","objects_scrubbed":14,"log_size":852,"log_dups_size":445,"ondisk_log_size":852,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:33.502639+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.0095105309999999992,"stat_sum":{"num_bytes":35842844,"num_objects":17,"num_object_clones":14,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":334,"num_read_kb":345014,"num_write":270,"num_write_kb":86357,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":26027409,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"123","length":"b"},{"start":"130","length":"9"},{"start":"13b","length":"3"},{"start":"13f","length":"6"},{"start":"146","length":"3"},{"start":"14b","length":"3"},{"start":"150","length":"3"},{"start":"154","length":"1"},{"start":"159","length":"1"},{"start":"15b","length":"1"},{"start":"15d","length":"1"},{"start":"15f","length":"1"},{"start":"161","length":"5"},{"start":"167","length":"4"},{"start":"170","length":"1"}]},{"pgid":"4.b","version":"625'15","reported_seq":973,"reported_epoch":823,"state":"active+clean","last_fresh":"2024-09-20T22:55:03.321717+0000","last_change":"2024-09-20T22:54:17.376019+0000","last_active":"2024-09-20T22:55:03.321717+0000","last_peered":"2024-09-20T22:55:03.321717+0000","last_clean":"2024-09-20T22:55:03.321717+0000","last_became_active":"2024-09-20T22:54:17.340998+0000","last_became_peered":"2024-09-20T22:54:17.340998+0000","last_unstale":"2024-09-20T22:55:03.321717+0000","last_undegraded":"2024-09-20T22:55:03.321717+0000","last_fullsized":"2024-09-20T22:55:03.321717+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'15","last_scrub_stamp":"2024-09-20T22:53:30.976566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:53:30.976566+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:03.693891+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":1565,"reported_epoch":823,"state":"active+clean","last_fresh":"2024-09-20T22:55:03.328509+0000","last_change":"2024-09-20T22:53:55.019427+0000","last_active":"2024-09-20T22:55:03.328509+0000","last_peered":"2024-09-20T22:55:03.328509+0000","last_clean":"2024-09-20T22:55:03.328509+0000","last_became_active":"2024-09-20T22:41:38.155953+0000","last_became_peered":"2024-09-20T22:41:38.155953+0000","last_unstale":"2024-09-20T22:55:03.328509+0000","last_undegraded":"2024-09-20T22:55:03.328509+0000","last_fullsized":"2024-09-20T22:55:03.328509+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:55.019362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:53:55.019362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:17.821354+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":1598,"reported_epoch":823,"state":"active+clean","last_fresh":"2024-09-20T22:55:03.326219+0000","last_change":"2024-09-20T22:54:47.368607+0000","last_active":"2024-09-20T22:55:03.326219+0000","last_peered":"2024-09-20T22:55:03.326219+0000","last_clean":"2024-09-20T22:55:03.326219+0000","last_became_active":"2024-09-20T22:48:05.240613+0000","last_became_peered":"2024-09-20T22:48:05.240613+0000","last_unstale":"2024-09-20T22:55:03.326219+0000","last_undegraded":"2024-09-20T22:55:03.326219+0000","last_fullsized":"2024-09-20T22:55:03.326219+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:54:47.368529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:54:47.368529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:58.934561+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":906,"reported_epoch":823,"state":"active+clean","last_fresh":"2024-09-20T22:55:03.321529+0000","last_change":"2024-09-20T22:54:38.856800+0000","last_active":"2024-09-20T22:55:03.321529+0000","last_peered":"2024-09-20T22:55:03.321529+0000","last_clean":"2024-09-20T22:55:03.321529+0000","last_became_active":"2024-09-20T22:47:49.219486+0000","last_became_peered":"2024-09-20T22:47:49.219486+0000","last_unstale":"2024-09-20T22:55:03.321529+0000","last_undegraded":"2024-09-20T22:55:03.321529+0000","last_fullsized":"2024-09-20T22:55:03.321529+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:54:38.856726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:54:38.856726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:56.536952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"820'1465","reported_seq":5215,"reported_epoch":823,"state":"active+clean","last_fresh":"2024-09-20T22:55:03.321785+0000","last_change":"2024-09-20T22:55:00.351583+0000","last_active":"2024-09-20T22:55:03.321785+0000","last_peered":"2024-09-20T22:55:03.321785+0000","last_clean":"2024-09-20T22:55:03.321785+0000","last_became_active":"2024-09-20T22:54:20.414945+0000","last_became_peered":"2024-09-20T22:54:20.414945+0000","last_unstale":"2024-09-20T22:55:03.321785+0000","last_undegraded":"2024-09-20T22:55:03.321785+0000","last_fullsized":"2024-09-20T22:55:03.321785+0000","mapping_epoch":779,"log_start":"423'280","ondisk_log_start":"423'280","created":94,"last_epoch_clean":780,"parent":"0.0","parent_split_bits":0,"last_scrub":"774'1348","last_scrub_stamp":"2024-09-20T22:54:15.010110+0000","last_deep_scrub":"590'828","last_deep_scrub_stamp":"2024-09-20T22:51:10.909364+0000","last_clean_scrub_stamp":"2024-09-20T22:54:15.010110+0000","objects_scrubbed":19,"log_size":1185,"log_dups_size":649,"ondisk_log_size":1185,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:17.805034+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.023386704000000001,"stat_sum":{"num_bytes":51464505,"num_objects":21,"num_object_clones":16,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":2,"num_read":526,"num_read_kb":550353,"num_write":366,"num_write_kb":144268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":109,"num_bytes_recovered":133262200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"123","length":"b"},{"start":"130","length":"9"},{"start":"13b","length":"3"},{"start":"13f","length":"6"},{"start":"146","length":"3"},{"start":"14b","length":"3"},{"start":"150","length":"3"},{"start":"154","length":"1"},{"start":"159","length":"1"},{"start":"15b","length":"1"},{"start":"15d","length":"1"},{"start":"15f","length":"1"},{"start":"161","length":"5"},{"start":"167","length":"4"},{"start":"170","length":"1"}]},{"pgid":"4.a","version":"625'44","reported_seq":1694,"reported_epoch":823,"state":"active+clean","last_fresh":"2024-09-20T22:55:03.327292+0000","last_change":"2024-09-20T22:54:17.368002+0000","last_active":"2024-09-20T22:55:03.327292+0000","last_peered":"2024-09-20T22:55:03.327292+0000","last_clean":"2024-09-20T22:55:03.327292+0000","last_became_active":"2024-09-20T22:54:17.341124+0000","last_became_peered":"2024-09-20T22:54:17.341124+0000","last_unstale":"2024-09-20T22:55:03.327292+0000","last_undegraded":"2024-09-20T22:55:03.327292+0000","last_fullsized":"2024-09-20T22:55:03.327292+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'44","last_scrub_stamp":"2024-09-20T22:53:50.910441+0000","last_deep_scrub":"113'28","last_deep_scrub_stamp":"2024-09-20T22:43:12.725900+0000","last_clean_scrub_stamp":"2024-09-20T22:53:50.910441+0000","objects_scrubbed":9,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:06.851263+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":1563,"reported_epoch":823,"state":"active+clean","last_fresh":"2024-09-20T22:55:03.326281+0000","last_change":"2024-09-20T22:53:27.585334+0000","last_active":"2024-09-20T22:55:03.326281+0000","last_peered":"2024-09-20T22:55:03.326281+0000","last_clean":"2024-09-20T22:55:03.326281+0000","last_became_active":"2024-09-20T22:41:38.161524+0000","last_became_peered":"2024-09-20T22:41:38.161524+0000","last_unstale":"2024-09-20T22:55:03.326281+0000","last_undegraded":"2024-09-20T22:55:03.326281+0000","last_fullsized":"2024-09-20T22:55:03.326281+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:27.585262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:53:27.585262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:05.703740+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":1585,"reported_epoch":823,"state":"active+clean","last_fresh":"2024-09-20T22:55:03.326503+0000","last_change":"2024-09-20T22:54:21.496172+0000","last_active":"2024-09-20T22:55:03.326503+0000","last_peered":"2024-09-20T22:55:03.326503+0000","last_clean":"2024-09-20T22:55:03.326503+0000","last_became_active":"2024-09-20T22:47:49.223220+0000","last_became_peered":"2024-09-20T22:47:49.223220+0000","last_unstale":"2024-09-20T22:55:03.326503+0000","last_undegraded":"2024-09-20T22:55:03.326503+0000","last_fullsized":"2024-09-20T22:55:03.326503+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:54:21.496068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:54:21.496068+0000","last_clean_scrub_stamp":"2024-09-20T22:54:21.496068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:23.538970+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":1560,"reported_epoch":823,"state":"active+clean","last_fresh":"2024-09-20T22:55:03.326143+0000","last_change":"2024-09-20T22:53:53.365864+0000","last_active":"2024-09-20T22:55:03.326143+0000","last_peered":"2024-09-20T22:55:03.326143+0000","last_clean":"2024-09-20T22:55:03.326143+0000","last_became_active":"2024-09-20T22:41:44.199395+0000","last_became_peered":"2024-09-20T22:41:44.199395+0000","last_unstale":"2024-09-20T22:55:03.326143+0000","last_undegraded":"2024-09-20T22:55:03.326143+0000","last_fullsized":"2024-09-20T22:55:03.326143+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:53.365789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:47:18.329209+0000","last_clean_scrub_stamp":"2024-09-20T22:53:53.365789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:06.119698+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.a","version":"823'1021","reported_seq":4579,"reported_epoch":823,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T22:55:03.323227+0000","last_change":"2024-09-20T22:55:02.416215+0000","last_active":"2024-09-20T22:55:03.323227+0000","last_peered":"2024-09-20T22:55:03.323227+0000","last_clean":"2024-09-20T22:55:03.323227+0000","last_became_active":"2024-09-20T22:42:52.405085+0000","last_became_peered":"2024-09-20T22:42:52.405085+0000","last_unstale":"2024-09-20T22:55:03.323227+0000","last_undegraded":"2024-09-20T22:55:03.323227+0000","last_fullsized":"2024-09-20T22:55:03.323227+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"505'466","last_scrub_stamp":"2024-09-20T22:49:47.088165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:49:47.088165+0000","objects_scrubbed":13,"log_size":1021,"log_dups_size":0,"ondisk_log_size":1021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":5,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.14351615500000001,"stat_sum":{"num_bytes":38214208,"num_objects":18,"num_object_clones":15,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":2,"num_read":406,"num_read_kb":434006,"num_write":294,"num_write_kb":122499,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"123","length":"1"},{"start":"125","length":"9"},{"start":"130","length":"9"},{"start":"13b","length":"3"},{"start":"13f","length":"6"},{"start":"146","length":"3"},{"start":"14b","length":"3"},{"start":"150","length":"3"},{"start":"154","length":"1"},{"start":"159","length":"1"},{"start":"15b","length":"1"},{"start":"15d","length":"1"},{"start":"161","length":"4"},{"start":"168","length":"3"}]},{"pgid":"4.9","version":"686'23","reported_seq":944,"reported_epoch":823,"state":"active+clean","last_fresh":"2024-09-20T22:55:03.318721+0000","last_change":"2024-09-20T22:54:17.392658+0000","last_active":"2024-09-20T22:55:03.318721+0000","last_peered":"2024-09-20T22:55:03.318721+0000","last_clean":"2024-09-20T22:55:03.318721+0000","last_became_active":"2024-09-20T22:54:17.352939+0000","last_became_peered":"2024-09-20T22:54:17.352939+0000","last_unstale":"2024-09-20T22:55:03.318721+0000","last_undegraded":"2024-09-20T22:55:03.318721+0000","last_fullsized":"2024-09-20T22:55:03.318721+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"686'23","last_scrub_stamp":"2024-09-20T22:52:46.768812+0000","last_deep_scrub":"686'23","last_deep_scrub_stamp":"2024-09-20T22:52:46.768812+0000","last_clean_scrub_stamp":"2024-09-20T22:52:46.768812+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:05.026648+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":21,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":1100,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":1607,"reported_epoch":823,"state":"active+clean","last_fresh":"2024-09-20T22:55:03.325636+0000","last_change":"2024-09-20T22:54:17.333114+0000","last_active":"2024-09-20T22:55:03.325636+0000","last_peered":"2024-09-20T22:55:03.325636+0000","last_clean":"2024-09-20T22:55:03.325636+0000","last_became_active":"2024-09-20T22:54:17.332759+0000","last_became_peered":"2024-09-20T22:54:17.332759+0000","last_unstale":"2024-09-20T22:55:03.325636+0000","last_undegraded":"2024-09-20T22:55:03.325636+0000","last_fullsized":"2024-09-20T22:55:03.325636+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:35.478053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:46:52.465940+0000","last_clean_scrub_stamp":"2024-09-20T22:53:35.478053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:05.703715+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":1606,"reported_epoch":823,"state":"active+clean","last_fresh":"2024-09-20T22:55:03.326771+0000","last_change":"2024-09-20T22:54:48.350732+0000","last_active":"2024-09-20T22:55:03.326771+0000","last_peered":"2024-09-20T22:55:03.326771+0000","last_clean":"2024-09-20T22:55:03.326771+0000","last_became_active":"2024-09-20T22:54:17.340269+0000","last_became_peered":"2024-09-20T22:54:17.340269+0000","last_unstale":"2024-09-20T22:55:03.326771+0000","last_undegraded":"2024-09-20T22:55:03.326771+0000","last_fullsized":"2024-09-20T22:55:03.326771+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:54:48.350593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:53:07.104891+0000","last_clean_scrub_stamp":"2024-09-20T22:54:48.350593+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:56:05.996922+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":1557,"reported_epoch":823,"state":"active+clean","last_fresh":"2024-09-20T22:55:03.323520+0000","last_change":"2024-09-20T22:54:17.331117+0000","last_active":"2024-09-20T22:55:03.323520+0000","last_peered":"2024-09-20T22:55:03.323520+0000","last_clean":"2024-09-20T22:55:03.323520+0000","last_became_active":"2024-09-20T22:54:17.329277+0000","last_became_peered":"2024-09-20T22:54:17.329277+0000","last_unstale":"2024-09-20T22:55:03.323520+0000","last_undegraded":"2024-09-20T22:55:03.323520+0000","last_fullsized":"2024-09-20T22:55:03.323520+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:53.242204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:55.170379+0000","last_clean_scrub_stamp":"2024-09-20T22:53:53.242204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"823'1295","reported_seq":5044,"reported_epoch":823,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T22:55:03.522299+0000","last_change":"2024-09-20T22:55:03.522299+0000","last_active":"2024-09-20T22:55:03.522299+0000","last_peered":"2024-09-20T22:55:03.522299+0000","last_clean":"2024-09-20T22:55:03.522299+0000","last_became_active":"2024-09-20T22:54:20.415706+0000","last_became_peered":"2024-09-20T22:54:20.415706+0000","last_unstale":"2024-09-20T22:55:03.522299+0000","last_undegraded":"2024-09-20T22:55:03.522299+0000","last_fullsized":"2024-09-20T22:55:03.522299+0000","mapping_epoch":779,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":780,"parent":"0.0","parent_split_bits":0,"last_scrub":"794'1255","last_scrub_stamp":"2024-09-20T22:54:37.866509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:54:37.866509+0000","objects_scrubbed":15,"log_size":1295,"log_dups_size":0,"ondisk_log_size":1295,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:48.906342+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.047747645999999998,"stat_sum":{"num_bytes":39194983,"num_objects":15,"num_object_clones":10,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":1,"num_read":574,"num_read_kb":645150,"num_write":370,"num_write_kb":140606,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":168,"num_bytes_recovered":68651320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"123","length":"b"},{"start":"130","length":"9"},{"start":"13b","length":"3"},{"start":"13f","length":"6"},{"start":"146","length":"3"},{"start":"14b","length":"3"},{"start":"150","length":"3"},{"start":"154","length":"1"},{"start":"159","length":"1"},{"start":"15b","length":"1"},{"start":"15d","length":"1"},{"start":"161","length":"4"},{"start":"168","length":"3"},{"start":"170","length":"1"}]},{"pgid":"4.8","version":"625'27","reported_seq":1605,"reported_epoch":823,"state":"active+clean","last_fresh":"2024-09-20T22:55:03.328782+0000","last_change":"2024-09-20T22:54:41.095070+0000","last_active":"2024-09-20T22:55:03.328782+0000","last_peered":"2024-09-20T22:55:03.328782+0000","last_clean":"2024-09-20T22:55:03.328782+0000","last_became_active":"2024-09-20T22:41:40.252113+0000","last_became_peered":"2024-09-20T22:41:40.252113+0000","last_unstale":"2024-09-20T22:55:03.328782+0000","last_undegraded":"2024-09-20T22:55:03.328782+0000","last_fullsized":"2024-09-20T22:55:03.328782+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'27","last_scrub_stamp":"2024-09-20T22:54:41.094989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:54:41.094989+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:57.543047+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'2","reported_seq":1626,"reported_epoch":823,"state":"active+clean","last_fresh":"2024-09-20T22:55:03.326901+0000","last_change":"2024-09-20T22:53:33.526131+0000","last_active":"2024-09-20T22:55:03.326901+0000","last_peered":"2024-09-20T22:55:03.326901+0000","last_clean":"2024-09-20T22:55:03.326901+0000","last_became_active":"2024-09-20T22:48:05.238098+0000","last_became_peered":"2024-09-20T22:48:05.238098+0000","last_unstale":"2024-09-20T22:55:03.326901+0000","last_undegraded":"2024-09-20T22:55:03.326901+0000","last_fullsized":"2024-09-20T22:55:03.326901+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-09-20T22:53:33.526049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:53:33.526049+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:05.703769+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"408'10","reported_seq":1754,"reported_epoch":823,"state":"active+clean","last_fresh":"2024-09-20T22:55:03.331760+0000","last_change":"2024-09-20T22:53:24.628363+0000","last_active":"2024-09-20T22:55:03.331760+0000","last_peered":"2024-09-20T22:55:03.331760+0000","last_clean":"2024-09-20T22:55:03.331760+0000","last_became_active":"2024-09-20T22:48:05.239344+0000","last_became_peered":"2024-09-20T22:48:05.239344+0000","last_unstale":"2024-09-20T22:55:03.331760+0000","last_undegraded":"2024-09-20T22:55:03.331760+0000","last_fullsized":"2024-09-20T22:55:03.331760+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"408'10","last_scrub_stamp":"2024-09-20T22:53:24.628269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:53:24.628269+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:05.703661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":1566,"reported_epoch":823,"state":"active+clean","last_fresh":"2024-09-20T22:55:03.329635+0000","last_change":"2024-09-20T22:54:43.129941+0000","last_active":"2024-09-20T22:55:03.329635+0000","last_peered":"2024-09-20T22:55:03.329635+0000","last_clean":"2024-09-20T22:55:03.329635+0000","last_became_active":"2024-09-20T22:47:49.227569+0000","last_became_peered":"2024-09-20T22:47:49.227569+0000","last_unstale":"2024-09-20T22:55:03.329635+0000","last_undegraded":"2024-09-20T22:55:03.329635+0000","last_fullsized":"2024-09-20T22:55:03.329635+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:54:43.129868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:54:43.129868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:56:10.190377+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":1601,"reported_epoch":823,"state":"active+clean","last_fresh":"2024-09-20T22:55:03.327164+0000","last_change":"2024-09-20T22:54:38.024885+0000","last_active":"2024-09-20T22:55:03.327164+0000","last_peered":"2024-09-20T22:55:03.327164+0000","last_clean":"2024-09-20T22:55:03.327164+0000","last_became_active":"2024-09-20T22:47:49.213292+0000","last_became_peered":"2024-09-20T22:47:49.213292+0000","last_unstale":"2024-09-20T22:55:03.327164+0000","last_undegraded":"2024-09-20T22:55:03.327164+0000","last_fullsized":"2024-09-20T22:55:03.327164+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:54:38.024794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:54:38.024794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:55.383474+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"625'12","reported_seq":1607,"reported_epoch":823,"state":"active+clean","last_fresh":"2024-09-20T22:55:03.324588+0000","last_change":"2024-09-20T22:54:17.387599+0000","last_active":"2024-09-20T22:55:03.324588+0000","last_peered":"2024-09-20T22:55:03.324588+0000","last_clean":"2024-09-20T22:55:03.324588+0000","last_became_active":"2024-09-20T22:54:17.342098+0000","last_became_peered":"2024-09-20T22:54:17.342098+0000","last_unstale":"2024-09-20T22:55:03.324588+0000","last_undegraded":"2024-09-20T22:55:03.324588+0000","last_fullsized":"2024-09-20T22:55:03.324588+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'12","last_scrub_stamp":"2024-09-20T22:53:26.319144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:53:26.319144+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:06.329303+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":1569,"reported_epoch":823,"state":"active+clean","last_fresh":"2024-09-20T22:55:03.329700+0000","last_change":"2024-09-20T22:54:17.340916+0000","last_active":"2024-09-20T22:55:03.329700+0000","last_peered":"2024-09-20T22:55:03.329700+0000","last_clean":"2024-09-20T22:55:03.329700+0000","last_became_active":"2024-09-20T22:54:17.339912+0000","last_became_peered":"2024-09-20T22:54:17.339912+0000","last_unstale":"2024-09-20T22:55:03.329700+0000","last_undegraded":"2024-09-20T22:55:03.329700+0000","last_fullsized":"2024-09-20T22:55:03.329700+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:03.888040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:09.705632+0000","last_clean_scrub_stamp":"2024-09-20T22:53:03.888040+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:07.222909+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"74'1","reported_seq":1572,"reported_epoch":823,"state":"active+clean","last_fresh":"2024-09-20T22:55:03.325778+0000","last_change":"2024-09-20T22:52:49.575819+0000","last_active":"2024-09-20T22:55:03.325778+0000","last_peered":"2024-09-20T22:55:03.325778+0000","last_clean":"2024-09-20T22:55:03.325778+0000","last_became_active":"2024-09-20T22:48:05.237882+0000","last_became_peered":"2024-09-20T22:48:05.237882+0000","last_unstale":"2024-09-20T22:55:03.325778+0000","last_undegraded":"2024-09-20T22:55:03.325778+0000","last_fullsized":"2024-09-20T22:55:03.325778+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-09-20T22:52:49.575702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:52:49.575702+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:05.703529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"625'15","reported_seq":925,"reported_epoch":823,"state":"active+clean","last_fresh":"2024-09-20T22:55:03.322978+0000","last_change":"2024-09-20T22:53:50.889818+0000","last_active":"2024-09-20T22:55:03.322978+0000","last_peered":"2024-09-20T22:55:03.322978+0000","last_clean":"2024-09-20T22:55:03.322978+0000","last_became_active":"2024-09-20T22:47:49.231315+0000","last_became_peered":"2024-09-20T22:47:49.231315+0000","last_unstale":"2024-09-20T22:55:03.322978+0000","last_undegraded":"2024-09-20T22:55:03.322978+0000","last_fullsized":"2024-09-20T22:55:03.322978+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'15","last_scrub_stamp":"2024-09-20T22:53:50.889717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:53:50.889717+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:03.693875+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":1595,"reported_epoch":823,"state":"active+clean","last_fresh":"2024-09-20T22:55:03.325129+0000","last_change":"2024-09-20T22:54:41.628923+0000","last_active":"2024-09-20T22:55:03.325129+0000","last_peered":"2024-09-20T22:55:03.325129+0000","last_clean":"2024-09-20T22:55:03.325129+0000","last_became_active":"2024-09-20T22:48:05.238801+0000","last_became_peered":"2024-09-20T22:48:05.238801+0000","last_unstale":"2024-09-20T22:55:03.325129+0000","last_undegraded":"2024-09-20T22:55:03.325129+0000","last_fullsized":"2024-09-20T22:55:03.325129+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:54:41.628846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:54:41.628846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:57.050553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":1561,"reported_epoch":823,"state":"active+clean","last_fresh":"2024-09-20T22:55:03.324436+0000","last_change":"2024-09-20T22:54:17.342665+0000","last_active":"2024-09-20T22:55:03.324436+0000","last_peered":"2024-09-20T22:55:03.324436+0000","last_clean":"2024-09-20T22:55:03.324436+0000","last_became_active":"2024-09-20T22:54:17.341617+0000","last_became_peered":"2024-09-20T22:54:17.341617+0000","last_unstale":"2024-09-20T22:55:03.324436+0000","last_undegraded":"2024-09-20T22:55:03.324436+0000","last_fullsized":"2024-09-20T22:55:03.324436+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:20.410164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:52:20.410164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:06.329145+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":1593,"reported_epoch":823,"state":"active+clean","last_fresh":"2024-09-20T22:55:03.326626+0000","last_change":"2024-09-20T22:53:58.169978+0000","last_active":"2024-09-20T22:55:03.326626+0000","last_peered":"2024-09-20T22:55:03.326626+0000","last_clean":"2024-09-20T22:55:03.326626+0000","last_became_active":"2024-09-20T22:48:05.239710+0000","last_became_peered":"2024-09-20T22:48:05.239710+0000","last_unstale":"2024-09-20T22:55:03.326626+0000","last_undegraded":"2024-09-20T22:55:03.326626+0000","last_fullsized":"2024-09-20T22:55:03.326626+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:58.169894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:53:58.169894+0000","last_clean_scrub_stamp":"2024-09-20T22:53:58.169894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:04.779145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"625'15","reported_seq":1586,"reported_epoch":823,"state":"active+clean","last_fresh":"2024-09-20T22:55:03.329162+0000","last_change":"2024-09-20T22:53:53.070301+0000","last_active":"2024-09-20T22:55:03.329162+0000","last_peered":"2024-09-20T22:55:03.329162+0000","last_clean":"2024-09-20T22:55:03.329162+0000","last_became_active":"2024-09-20T22:41:40.252486+0000","last_became_peered":"2024-09-20T22:41:40.252486+0000","last_unstale":"2024-09-20T22:55:03.329162+0000","last_undegraded":"2024-09-20T22:55:03.329162+0000","last_fullsized":"2024-09-20T22:55:03.329162+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'15","last_scrub_stamp":"2024-09-20T22:53:53.070214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:53:53.070214+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:11.603607+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":914,"reported_epoch":823,"state":"active+clean","last_fresh":"2024-09-20T22:55:03.321324+0000","last_change":"2024-09-20T22:53:58.829885+0000","last_active":"2024-09-20T22:55:03.321324+0000","last_peered":"2024-09-20T22:55:03.321324+0000","last_clean":"2024-09-20T22:55:03.321324+0000","last_became_active":"2024-09-20T22:47:49.229936+0000","last_became_peered":"2024-09-20T22:47:49.229936+0000","last_unstale":"2024-09-20T22:55:03.321324+0000","last_undegraded":"2024-09-20T22:55:03.321324+0000","last_fullsized":"2024-09-20T22:55:03.321324+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:58.829817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:53:58.829817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:21.096752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"74'8","reported_seq":1722,"reported_epoch":823,"state":"active+clean","last_fresh":"2024-09-20T22:55:03.324006+0000","last_change":"2024-09-20T22:53:59.165210+0000","last_active":"2024-09-20T22:55:03.324006+0000","last_peered":"2024-09-20T22:55:03.324006+0000","last_clean":"2024-09-20T22:55:03.324006+0000","last_became_active":"2024-09-20T22:41:42.186352+0000","last_became_peered":"2024-09-20T22:41:42.186352+0000","last_unstale":"2024-09-20T22:55:03.324006+0000","last_undegraded":"2024-09-20T22:55:03.324006+0000","last_fullsized":"2024-09-20T22:55:03.324006+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-09-20T22:53:59.165134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:53:59.165134+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:12.296267+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":863,"reported_epoch":823,"state":"active+clean","last_fresh":"2024-09-20T22:55:03.320089+0000","last_change":"2024-09-20T22:52:16.623463+0000","last_active":"2024-09-20T22:55:03.320089+0000","last_peered":"2024-09-20T22:55:03.320089+0000","last_clean":"2024-09-20T22:55:03.320089+0000","last_became_active":"2024-09-20T22:48:05.249656+0000","last_became_peered":"2024-09-20T22:48:05.249656+0000","last_unstale":"2024-09-20T22:55:03.320089+0000","last_undegraded":"2024-09-20T22:55:03.320089+0000","last_fullsized":"2024-09-20T22:55:03.320089+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:16.623368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:52:16.623368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:05.026528+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"625'20","reported_seq":1627,"reported_epoch":823,"state":"active+clean","last_fresh":"2024-09-20T22:55:03.322441+0000","last_change":"2024-09-20T22:54:17.390484+0000","last_active":"2024-09-20T22:55:03.322441+0000","last_peered":"2024-09-20T22:55:03.322441+0000","last_clean":"2024-09-20T22:55:03.322441+0000","last_became_active":"2024-09-20T22:54:17.343066+0000","last_became_peered":"2024-09-20T22:54:17.343066+0000","last_unstale":"2024-09-20T22:55:03.322441+0000","last_undegraded":"2024-09-20T22:55:03.322441+0000","last_fullsized":"2024-09-20T22:55:03.322441+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'20","last_scrub_stamp":"2024-09-20T22:54:14.200839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:54:14.200839+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:15.294970+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":1583,"reported_epoch":823,"state":"active+clean","last_fresh":"2024-09-20T22:55:03.325918+0000","last_change":"2024-09-20T22:53:56.396136+0000","last_active":"2024-09-20T22:55:03.325918+0000","last_peered":"2024-09-20T22:55:03.325918+0000","last_clean":"2024-09-20T22:55:03.325918+0000","last_became_active":"2024-09-20T22:48:05.240231+0000","last_became_peered":"2024-09-20T22:48:05.240231+0000","last_unstale":"2024-09-20T22:55:03.325918+0000","last_undegraded":"2024-09-20T22:55:03.325918+0000","last_fullsized":"2024-09-20T22:55:03.325918+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:56.396065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:53:56.396065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:04.372880+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"778'12","reported_seq":1771,"reported_epoch":823,"state":"active+clean","last_fresh":"2024-09-20T22:55:03.329007+0000","last_change":"2024-09-20T22:54:17.379970+0000","last_active":"2024-09-20T22:55:03.329007+0000","last_peered":"2024-09-20T22:55:03.329007+0000","last_clean":"2024-09-20T22:55:03.329007+0000","last_became_active":"2024-09-20T22:54:17.341014+0000","last_became_peered":"2024-09-20T22:54:17.341014+0000","last_unstale":"2024-09-20T22:55:03.329007+0000","last_undegraded":"2024-09-20T22:55:03.329007+0000","last_fullsized":"2024-09-20T22:55:03.329007+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"600'11","last_scrub_stamp":"2024-09-20T22:52:36.264410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:52:36.264410+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:07.222854+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":906,"reported_epoch":823,"state":"active+clean","last_fresh":"2024-09-20T22:55:03.322715+0000","last_change":"2024-09-20T22:54:47.822738+0000","last_active":"2024-09-20T22:55:03.322715+0000","last_peered":"2024-09-20T22:55:03.322715+0000","last_clean":"2024-09-20T22:55:03.322715+0000","last_became_active":"2024-09-20T22:47:49.230674+0000","last_became_peered":"2024-09-20T22:47:49.230674+0000","last_unstale":"2024-09-20T22:55:03.322715+0000","last_undegraded":"2024-09-20T22:55:03.322715+0000","last_fullsized":"2024-09-20T22:55:03.322715+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:54:47.822663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:50:21.853070+0000","last_clean_scrub_stamp":"2024-09-20T22:54:47.822663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:55.206157+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"625'19","reported_seq":931,"reported_epoch":823,"state":"active+clean","last_fresh":"2024-09-20T22:55:03.318957+0000","last_change":"2024-09-20T22:54:17.385330+0000","last_active":"2024-09-20T22:55:03.318957+0000","last_peered":"2024-09-20T22:55:03.318957+0000","last_clean":"2024-09-20T22:55:03.318957+0000","last_became_active":"2024-09-20T22:54:17.351129+0000","last_became_peered":"2024-09-20T22:54:17.351129+0000","last_unstale":"2024-09-20T22:55:03.318957+0000","last_undegraded":"2024-09-20T22:55:03.318957+0000","last_fullsized":"2024-09-20T22:55:03.318957+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'19","last_scrub_stamp":"2024-09-20T22:53:49.547607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:53:49.547607+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:05.026705+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":871,"reported_epoch":823,"state":"active+clean","last_fresh":"2024-09-20T22:55:03.318593+0000","last_change":"2024-09-20T22:52:47.717598+0000","last_active":"2024-09-20T22:55:03.318593+0000","last_peered":"2024-09-20T22:55:03.318593+0000","last_clean":"2024-09-20T22:55:03.318593+0000","last_became_active":"2024-09-20T22:48:05.252026+0000","last_became_peered":"2024-09-20T22:48:05.252026+0000","last_unstale":"2024-09-20T22:55:03.318593+0000","last_undegraded":"2024-09-20T22:55:03.318593+0000","last_fullsized":"2024-09-20T22:55:03.318593+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:47.717490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:52:47.717490+0000","last_clean_scrub_stamp":"2024-09-20T22:52:47.717490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:05.026618+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":103,"reported_epoch":823,"state":"active+clean","last_fresh":"2024-09-20T22:55:03.473867+0000","last_change":"2024-09-20T22:54:17.337178+0000","last_active":"2024-09-20T22:55:03.473867+0000","last_peered":"2024-09-20T22:55:03.473867+0000","last_clean":"2024-09-20T22:55:03.473867+0000","last_became_active":"2024-09-20T22:54:17.336853+0000","last_became_peered":"2024-09-20T22:54:17.336853+0000","last_unstale":"2024-09-20T22:55:03.473867+0000","last_undegraded":"2024-09-20T22:55:03.473867+0000","last_fullsized":"2024-09-20T22:55:03.473867+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:56.000202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:53:56.000202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":1557,"reported_epoch":823,"state":"active+clean","last_fresh":"2024-09-20T22:55:03.322955+0000","last_change":"2024-09-20T22:52:42.582474+0000","last_active":"2024-09-20T22:55:03.322955+0000","last_peered":"2024-09-20T22:55:03.322955+0000","last_clean":"2024-09-20T22:55:03.322955+0000","last_became_active":"2024-09-20T22:47:49.216141+0000","last_became_peered":"2024-09-20T22:47:49.216141+0000","last_unstale":"2024-09-20T22:55:03.322955+0000","last_undegraded":"2024-09-20T22:55:03.322955+0000","last_fullsized":"2024-09-20T22:55:03.322955+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:42.582398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:50:20.206984+0000","last_clean_scrub_stamp":"2024-09-20T22:52:42.582398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:06.329219+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"625'19","reported_seq":113,"reported_epoch":823,"state":"active+clean","last_fresh":"2024-09-20T22:55:03.474017+0000","last_change":"2024-09-20T22:54:17.360066+0000","last_active":"2024-09-20T22:55:03.474017+0000","last_peered":"2024-09-20T22:55:03.474017+0000","last_clean":"2024-09-20T22:55:03.474017+0000","last_became_active":"2024-09-20T22:54:17.340539+0000","last_became_peered":"2024-09-20T22:54:17.340539+0000","last_unstale":"2024-09-20T22:55:03.474017+0000","last_undegraded":"2024-09-20T22:55:03.474017+0000","last_fullsized":"2024-09-20T22:55:03.474017+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'19","last_scrub_stamp":"2024-09-20T22:53:58.933385+0000","last_deep_scrub":"404'13","last_deep_scrub_stamp":"2024-09-20T22:48:04.276282+0000","last_clean_scrub_stamp":"2024-09-20T22:53:58.933385+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:27.507376+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":1598,"reported_epoch":823,"state":"active+clean","last_fresh":"2024-09-20T22:55:03.326137+0000","last_change":"2024-09-20T22:54:46.541182+0000","last_active":"2024-09-20T22:55:03.326137+0000","last_peered":"2024-09-20T22:55:03.326137+0000","last_clean":"2024-09-20T22:55:03.326137+0000","last_became_active":"2024-09-20T22:48:05.238812+0000","last_became_peered":"2024-09-20T22:48:05.238812+0000","last_unstale":"2024-09-20T22:55:03.326137+0000","last_undegraded":"2024-09-20T22:55:03.326137+0000","last_fullsized":"2024-09-20T22:55:03.326137+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:54:46.541109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:31.138067+0000","last_clean_scrub_stamp":"2024-09-20T22:54:46.541109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:58.391916+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":1573,"reported_epoch":823,"state":"active+clean","last_fresh":"2024-09-20T22:55:03.323643+0000","last_change":"2024-09-20T22:54:17.337400+0000","last_active":"2024-09-20T22:55:03.323643+0000","last_peered":"2024-09-20T22:55:03.323643+0000","last_clean":"2024-09-20T22:55:03.323643+0000","last_became_active":"2024-09-20T22:54:17.337102+0000","last_became_peered":"2024-09-20T22:54:17.337102+0000","last_unstale":"2024-09-20T22:55:03.323643+0000","last_undegraded":"2024-09-20T22:55:03.323643+0000","last_fullsized":"2024-09-20T22:55:03.323643+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:34.345077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:53:34.345077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":910,"reported_epoch":823,"state":"active+clean","last_fresh":"2024-09-20T22:55:03.321110+0000","last_change":"2024-09-20T22:54:17.344363+0000","last_active":"2024-09-20T22:55:03.321110+0000","last_peered":"2024-09-20T22:55:03.321110+0000","last_clean":"2024-09-20T22:55:03.321110+0000","last_became_active":"2024-09-20T22:54:17.344021+0000","last_became_peered":"2024-09-20T22:54:17.344021+0000","last_unstale":"2024-09-20T22:55:03.321110+0000","last_undegraded":"2024-09-20T22:55:03.321110+0000","last_fullsized":"2024-09-20T22:55:03.321110+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:09.890501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:53:09.890501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:03.693815+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"625'24","reported_seq":1387,"reported_epoch":823,"state":"active+clean","last_fresh":"2024-09-20T22:55:03.328656+0000","last_change":"2024-09-20T22:53:33.089688+0000","last_active":"2024-09-20T22:55:03.328656+0000","last_peered":"2024-09-20T22:55:03.328656+0000","last_clean":"2024-09-20T22:55:03.328656+0000","last_became_active":"2024-09-20T22:44:09.170785+0000","last_became_peered":"2024-09-20T22:44:09.170785+0000","last_unstale":"2024-09-20T22:55:03.328656+0000","last_undegraded":"2024-09-20T22:55:03.328656+0000","last_fullsized":"2024-09-20T22:55:03.328656+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'24","last_scrub_stamp":"2024-09-20T22:53:33.089580+0000","last_deep_scrub":"470'16","last_deep_scrub_stamp":"2024-09-20T22:49:10.659868+0000","last_clean_scrub_stamp":"2024-09-20T22:53:33.089580+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:07.223047+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":1577,"reported_epoch":823,"state":"active+clean","last_fresh":"2024-09-20T22:55:03.329235+0000","last_change":"2024-09-20T22:54:17.341390+0000","last_active":"2024-09-20T22:55:03.329235+0000","last_peered":"2024-09-20T22:55:03.329235+0000","last_clean":"2024-09-20T22:55:03.329235+0000","last_became_active":"2024-09-20T22:54:17.341069+0000","last_became_peered":"2024-09-20T22:54:17.341069+0000","last_unstale":"2024-09-20T22:55:03.329235+0000","last_undegraded":"2024-09-20T22:55:03.329235+0000","last_fullsized":"2024-09-20T22:55:03.329235+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:07.945664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:45:24.765414+0000","last_clean_scrub_stamp":"2024-09-20T22:53:07.945664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:07.222984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":1579,"reported_epoch":823,"state":"active+clean","last_fresh":"2024-09-20T22:55:03.324996+0000","last_change":"2024-09-20T22:53:09.740214+0000","last_active":"2024-09-20T22:55:03.324996+0000","last_peered":"2024-09-20T22:55:03.324996+0000","last_clean":"2024-09-20T22:55:03.324996+0000","last_became_active":"2024-09-20T22:48:05.238334+0000","last_became_peered":"2024-09-20T22:48:05.238334+0000","last_unstale":"2024-09-20T22:55:03.324996+0000","last_undegraded":"2024-09-20T22:55:03.324996+0000","last_fullsized":"2024-09-20T22:55:03.324996+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:09.740145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:50.174861+0000","last_clean_scrub_stamp":"2024-09-20T22:53:09.740145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:05.703689+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":1557,"reported_epoch":823,"state":"active+clean","last_fresh":"2024-09-20T22:55:03.324090+0000","last_change":"2024-09-20T22:54:22.276389+0000","last_active":"2024-09-20T22:55:03.324090+0000","last_peered":"2024-09-20T22:55:03.324090+0000","last_clean":"2024-09-20T22:55:03.324090+0000","last_became_active":"2024-09-20T22:47:49.214487+0000","last_became_peered":"2024-09-20T22:47:49.214487+0000","last_unstale":"2024-09-20T22:55:03.324090+0000","last_undegraded":"2024-09-20T22:55:03.324090+0000","last_fullsized":"2024-09-20T22:55:03.324090+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:54:22.276321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:54:22.276321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:27.586502+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"625'8","reported_seq":1589,"reported_epoch":823,"state":"active+clean","last_fresh":"2024-09-20T22:55:03.321542+0000","last_change":"2024-09-20T22:54:17.373557+0000","last_active":"2024-09-20T22:55:03.321542+0000","last_peered":"2024-09-20T22:55:03.321542+0000","last_clean":"2024-09-20T22:55:03.321542+0000","last_became_active":"2024-09-20T22:54:17.341776+0000","last_became_peered":"2024-09-20T22:54:17.341776+0000","last_unstale":"2024-09-20T22:55:03.321542+0000","last_undegraded":"2024-09-20T22:55:03.321542+0000","last_fullsized":"2024-09-20T22:55:03.321542+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'8","last_scrub_stamp":"2024-09-20T22:53:05.489279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:53:05.489279+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:06.329247+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":906,"reported_epoch":823,"state":"active+clean","last_fresh":"2024-09-20T22:55:03.323522+0000","last_change":"2024-09-20T22:53:07.918178+0000","last_active":"2024-09-20T22:55:03.323522+0000","last_peered":"2024-09-20T22:55:03.323522+0000","last_clean":"2024-09-20T22:55:03.323522+0000","last_became_active":"2024-09-20T22:47:49.231964+0000","last_became_peered":"2024-09-20T22:47:49.231964+0000","last_unstale":"2024-09-20T22:55:03.323522+0000","last_undegraded":"2024-09-20T22:55:03.323522+0000","last_fullsized":"2024-09-20T22:55:03.323522+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:07.918108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:53:07.918108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:03.693829+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":1597,"reported_epoch":823,"state":"active+clean","last_fresh":"2024-09-20T22:55:03.327033+0000","last_change":"2024-09-20T22:53:51.937667+0000","last_active":"2024-09-20T22:55:03.327033+0000","last_peered":"2024-09-20T22:55:03.327033+0000","last_clean":"2024-09-20T22:55:03.327033+0000","last_became_active":"2024-09-20T22:48:05.244659+0000","last_became_peered":"2024-09-20T22:48:05.244659+0000","last_unstale":"2024-09-20T22:55:03.327033+0000","last_undegraded":"2024-09-20T22:55:03.327033+0000","last_fullsized":"2024-09-20T22:55:03.327033+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:51.937584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:53:51.937584+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:07.795334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":1571,"reported_epoch":823,"state":"active+clean","last_fresh":"2024-09-20T22:55:03.325106+0000","last_change":"2024-09-20T22:53:05.612142+0000","last_active":"2024-09-20T22:55:03.325106+0000","last_peered":"2024-09-20T22:55:03.325106+0000","last_clean":"2024-09-20T22:55:03.325106+0000","last_became_active":"2024-09-20T22:48:05.237765+0000","last_became_peered":"2024-09-20T22:48:05.237765+0000","last_unstale":"2024-09-20T22:55:03.325106+0000","last_undegraded":"2024-09-20T22:55:03.325106+0000","last_fullsized":"2024-09-20T22:55:03.325106+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:05.612030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:10.233487+0000","last_clean_scrub_stamp":"2024-09-20T22:53:05.612030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:05.703554+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":103,"reported_epoch":823,"state":"active+clean","last_fresh":"2024-09-20T22:55:03.474319+0000","last_change":"2024-09-20T22:54:17.348523+0000","last_active":"2024-09-20T22:55:03.474319+0000","last_peered":"2024-09-20T22:55:03.474319+0000","last_clean":"2024-09-20T22:55:03.474319+0000","last_became_active":"2024-09-20T22:54:17.348206+0000","last_became_peered":"2024-09-20T22:54:17.348206+0000","last_unstale":"2024-09-20T22:55:03.474319+0000","last_undegraded":"2024-09-20T22:55:03.474319+0000","last_fullsized":"2024-09-20T22:55:03.474319+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:58.003274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:53:58.003274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:07.487217+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":1595,"reported_epoch":823,"state":"active+clean","last_fresh":"2024-09-20T22:55:03.321130+0000","last_change":"2024-09-20T22:54:17.342220+0000","last_active":"2024-09-20T22:55:03.321130+0000","last_peered":"2024-09-20T22:55:03.321130+0000","last_clean":"2024-09-20T22:55:03.321130+0000","last_became_active":"2024-09-20T22:54:17.341405+0000","last_became_peered":"2024-09-20T22:54:17.341405+0000","last_unstale":"2024-09-20T22:55:03.321130+0000","last_undegraded":"2024-09-20T22:55:03.321130+0000","last_fullsized":"2024-09-20T22:55:03.321130+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:31.274390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:53:31.274390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:06.329330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"625'19","reported_seq":1656,"reported_epoch":823,"state":"active+clean","last_fresh":"2024-09-20T22:55:03.332227+0000","last_change":"2024-09-20T22:54:50.039084+0000","last_active":"2024-09-20T22:55:03.332227+0000","last_peered":"2024-09-20T22:55:03.332227+0000","last_clean":"2024-09-20T22:55:03.332227+0000","last_became_active":"2024-09-20T22:54:17.342236+0000","last_became_peered":"2024-09-20T22:54:17.342236+0000","last_unstale":"2024-09-20T22:55:03.332227+0000","last_undegraded":"2024-09-20T22:55:03.332227+0000","last_fullsized":"2024-09-20T22:55:03.332227+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'19","last_scrub_stamp":"2024-09-20T22:54:50.039001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:54:50.039001+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:55.885068+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":906,"reported_epoch":823,"state":"active+clean","last_fresh":"2024-09-20T22:55:03.322766+0000","last_change":"2024-09-20T22:54:43.813615+0000","last_active":"2024-09-20T22:55:03.322766+0000","last_peered":"2024-09-20T22:55:03.322766+0000","last_clean":"2024-09-20T22:55:03.322766+0000","last_became_active":"2024-09-20T22:47:49.220971+0000","last_became_peered":"2024-09-20T22:47:49.220971+0000","last_unstale":"2024-09-20T22:55:03.322766+0000","last_undegraded":"2024-09-20T22:55:03.322766+0000","last_fullsized":"2024-09-20T22:55:03.322766+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:54:43.813540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:54:43.813540+0000","last_clean_scrub_stamp":"2024-09-20T22:54:43.813540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:55.503018+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":530233920,"num_objects":241,"num_object_clones":192,"num_object_copies":723,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":241,"num_whiteouts":12,"num_read":5681,"num_read_kb":5947215,"num_write":4051,"num_write_kb":1530246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":989,"num_bytes_recovered":730416291,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1222590464,"data_stored":1657556955,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":13212,"ondisk_log_size":13212,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":90,"ondisk_log_size":90,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1106,"num_read_kb":863,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":418,"num_bytes_recovered":9802,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":745,"ondisk_log_size":745,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":1126,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":794,"num_read_kb":789,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":1594,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":384,"ondisk_log_size":384,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070629,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":180252,"kb_used_data":122692,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":93564900,"statfs":{"total":95995035648,"available":95810457600,"internally_reserved":0,"allocated":125636608,"data_stored":178268396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":58915274},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430379,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":208696,"kb_used_data":151136,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":93536456,"statfs":{"total":95995035648,"available":95781330944,"internally_reserved":0,"allocated":154763264,"data_stored":198176996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798692016,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":170120,"kb_used_data":125044,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":93575032,"statfs":{"total":95995035648,"available":95820832768,"internally_reserved":0,"allocated":128045056,"data_stored":174130402,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986357,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":149652,"kb_used_data":96256,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93595500,"statfs":{"total":95995035648,"available":95841792000,"internally_reserved":0,"allocated":98566144,"data_stored":129638854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182587,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":311212,"kb_used_data":248080,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":93433940,"statfs":{"total":95995035648,"available":95676354560,"internally_reserved":0,"allocated":254033920,"data_stored":356502637,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":64616903},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411520,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":172100,"kb_used_data":118704,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93573052,"statfs":{"total":95995035648,"available":95818805248,"internally_reserved":0,"allocated":121552896,"data_stored":177856795,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575045,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":254036,"kb_used_data":208956,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":93491116,"statfs":{"total":95995035648,"available":95734902784,"internally_reserved":0,"allocated":213970944,"data_stored":228141153,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738570,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":239704,"kb_used_data":186300,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93505448,"statfs":{"total":95995035648,"available":95749578752,"internally_reserved":0,"allocated":190771200,"data_stored":255083876,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":182890496,"data_stored":250270247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":206065664,"data_stored":223334596,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":113438720,"data_stored":172788531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":245559296,"data_stored":351098316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":90693632,"data_stored":124833464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":119595008,"data_stored":168733364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":146915328,"data_stored":193371218,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":117432320,"data_stored":173127219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-20T22:55:09.976 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:55:09.998 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:09.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:09.999 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:55:09.999 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:55:10.004 INFO:journalctl@ceph.iscsi.iscsi.a.smithi154.stdout:Sep 20 22:55:09 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-iscsi-iscsi-a[59730]: debug there is no tcmu-runner data available 2024-09-20T22:55:10.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:09 smithi154 ceph-mon[33264]: 5.6 scrub starts 2024-09-20T22:55:10.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:09 smithi154 ceph-mon[33264]: 5.6 scrub ok 2024-09-20T22:55:10.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:09 smithi154 ceph-mon[33264]: 6.12 deep-scrub starts 2024-09-20T22:55:10.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:09 smithi154 ceph-mon[33264]: 6.12 deep-scrub ok 2024-09-20T22:55:10.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:09 smithi154 ceph-mon[33264]: 4.6 deep-scrub starts 2024-09-20T22:55:10.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:09 smithi154 ceph-mon[33264]: 4.6 deep-scrub ok 2024-09-20T22:55:10.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:09 smithi154 ceph-mon[33264]: 4.a scrub starts 2024-09-20T22:55:10.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:09 smithi154 ceph-mon[33264]: 4.a scrub ok 2024-09-20T22:55:10.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:09 smithi154 ceph-mon[33264]: 5.c scrub starts 2024-09-20T22:55:10.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:09 smithi154 ceph-mon[33264]: 5.c scrub ok 2024-09-20T22:55:10.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:09 smithi154 ceph-mon[33264]: osdmap e829: 8 total, 8 up, 8 in 2024-09-20T22:55:10.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:09 smithi154 ceph-mon[33264]: 3.8 scrub starts 2024-09-20T22:55:10.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:09 smithi154 ceph-mon[33264]: 3.8 scrub ok 2024-09-20T22:55:10.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:09 smithi154 ceph-mon[33264]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:55:10.099 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:55:10.122 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:10.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:10.123 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:55:10.224 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:55:10.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:10.249 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:10.249 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:55:10.290 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-20T22:55:10.338 INFO:tasks.rados.rados.0.smithi084.stdout:3310: done (2 left) 2024-09-20T22:55:10.338 INFO:tasks.rados.rados.0.smithi084.stdout:3311: copy_from oid 43 from oid 20 current snap is 381 2024-09-20T22:55:10.338 INFO:tasks.rados.rados.0.smithi084.stdout:3312: write (excl) oid 17 current snap is 381 2024-09-20T22:55:10.338 INFO:tasks.rados.rados.0.smithi084.stdout:3312: seq_num 1126 ranges {524682=750618,1757548=597056} 2024-09-20T22:55:10.346 INFO:tasks.rados.rados.0.smithi084.stdout:3312: writing smithi08477203-17 from 524682 to 1275300 tid 1 2024-09-20T22:55:10.350 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:55:10.354 INFO:tasks.rados.rados.0.smithi084.stdout:3312: writing smithi08477203-17 from 1757548 to 2354604 tid 2 2024-09-20T22:55:10.355 INFO:tasks.rados.rados.0.smithi084.stdout:3313: rollback oid 10 current snap is 381 2024-09-20T22:55:10.355 INFO:tasks.rados.rados.0.smithi084.stdout:3306: finishing write tid 3 to smithi08477203-33 2024-09-20T22:55:10.355 INFO:tasks.rados.rados.0.smithi084.stdout:3306: finishing write tid 4 to smithi08477203-33 2024-09-20T22:55:10.355 INFO:tasks.rados.rados.0.smithi084.stdout:3306: finishing write tid 5 to smithi08477203-33 2024-09-20T22:55:10.355 INFO:tasks.rados.rados.0.smithi084.stdout:3306: oid 33 updating version 0 to 1918 2024-09-20T22:55:10.356 INFO:tasks.rados.rados.0.smithi084.stdout:3306: oid 33 updating version 1918 to 1920 2024-09-20T22:55:10.356 INFO:tasks.rados.rados.0.smithi084.stdout:3306: oid 33 version 1920 is already newer than 1919 2024-09-20T22:55:10.356 INFO:tasks.rados.rados.0.smithi084.stdout:3306: oid 33 version 1920 is already newer than 1917 2024-09-20T22:55:10.356 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 33 v 1920 (ObjNum 1124 snap 381 seq_num 1124) dirty exists 2024-09-20T22:55:10.356 INFO:tasks.rados.rados.0.smithi084.stdout:3306: left oid 33 (ObjNum 1124 snap 381 seq_num 1124) 2024-09-20T22:55:10.356 INFO:tasks.rados.rados.0.smithi084.stdout:3309: expect (ObjNum 984 snap 319 seq_num 984) 2024-09-20T22:55:10.396 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:10.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:10.397 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:55:10.440 INFO:tasks.rados.rados.0.smithi084.stdout:3311: finishing copy_from racing read to smithi08477203-43 2024-09-20T22:55:10.440 INFO:tasks.rados.rados.0.smithi084.stdout:3311: finishing copy_from to smithi08477203-43 2024-09-20T22:55:10.440 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 43 v 1929 (ObjNum 1113 snap 378 seq_num 1113) dirty exists 2024-09-20T22:55:10.440 INFO:tasks.rados.rados.0.smithi084.stdout:3312: finishing write tid 1 to smithi08477203-17 2024-09-20T22:55:10.440 INFO:tasks.rados.rados.0.smithi084.stdout:3312: finishing write tid 2 to smithi08477203-17 2024-09-20T22:55:10.440 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 10 to 379 2024-09-20T22:55:10.441 INFO:tasks.rados.rados.0.smithi084.stdout:3312: finishing write tid 3 to smithi08477203-17 2024-09-20T22:55:10.441 INFO:tasks.rados.rados.0.smithi084.stdout:3312: finishing write tid 4 to smithi08477203-17 2024-09-20T22:55:10.441 INFO:tasks.rados.rados.0.smithi084.stdout:3312: oid 17 updating version 0 to 1262 2024-09-20T22:55:10.441 INFO:tasks.rados.rados.0.smithi084.stdout:3312: oid 17 version 1262 is already newer than 1260 2024-09-20T22:55:10.441 INFO:tasks.rados.rados.0.smithi084.stdout:3312: oid 17 version 1262 is already newer than 1261 2024-09-20T22:55:10.441 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 17 v 1262 (ObjNum 1126 snap 381 seq_num 1126) dirty exists 2024-09-20T22:55:10.441 INFO:tasks.rados.rados.0.smithi084.stdout:3312: left oid 17 (ObjNum 1126 snap 381 seq_num 1126) 2024-09-20T22:55:10.441 INFO:tasks.rados.rados.0.smithi084.stdout:3306: done (4 left) 2024-09-20T22:55:10.441 INFO:tasks.rados.rados.0.smithi084.stdout:3309: done (3 left) 2024-09-20T22:55:10.441 INFO:tasks.rados.rados.0.smithi084.stdout:3311: done (2 left) 2024-09-20T22:55:10.441 INFO:tasks.rados.rados.0.smithi084.stdout:3312: done (1 left) 2024-09-20T22:55:10.441 INFO:tasks.rados.rados.0.smithi084.stdout:3314: write (excl) oid 34 current snap is 381 2024-09-20T22:55:10.442 INFO:tasks.rados.rados.0.smithi084.stdout:3314: seq_num 1127 ranges {541489=425867,1416030=740714,2732654=208835} 2024-09-20T22:55:10.445 INFO:tasks.rados.rados.0.smithi084.stdout:3314: writing smithi08477203-34 from 541489 to 967356 tid 1 2024-09-20T22:55:10.451 INFO:tasks.rados.rados.0.smithi084.stdout:3314: writing smithi08477203-34 from 1416030 to 2156744 tid 2 2024-09-20T22:55:10.456 INFO:tasks.rados.rados.0.smithi084.stdout:3314: writing smithi08477203-34 from 2732654 to 2941489 tid 3 2024-09-20T22:55:10.456 INFO:tasks.rados.rados.0.smithi084.stdout:3315: rollback oid 29 current snap is 381 2024-09-20T22:55:10.457 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 29 to 328 2024-09-20T22:55:10.457 INFO:tasks.rados.rados.0.smithi084.stdout:3313: finishing rollback tid 1 to smithi08477203-10 2024-09-20T22:55:10.457 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 10 v 1826 (ObjNum 49 snap 0 seq_num 1953066355) dirty dne 2024-09-20T22:55:10.457 INFO:tasks.rados.rados.0.smithi084.stdout:3314: finishing write tid 1 to smithi08477203-34 2024-09-20T22:55:10.457 INFO:tasks.rados.rados.0.smithi084.stdout:3314: finishing write tid 2 to smithi08477203-34 2024-09-20T22:55:10.457 INFO:tasks.rados.rados.0.smithi084.stdout:3313: done (2 left) 2024-09-20T22:55:10.457 INFO:tasks.rados.rados.0.smithi084.stdout:3316: rollback oid 9 current snap is 381 2024-09-20T22:55:10.457 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 9 to 300 2024-09-20T22:55:10.457 INFO:tasks.rados.rados.0.smithi084.stdout:3317: rollback oid 26 current snap is 381 2024-09-20T22:55:10.457 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 26 to 312 2024-09-20T22:55:10.457 INFO:tasks.rados.rados.0.smithi084.stdout:3318: write (excl) oid 50 current snap is 381 2024-09-20T22:55:10.457 INFO:tasks.rados.rados.0.smithi084.stdout:3318: seq_num 1128 ranges {558296=501116,1474512=600725,2760077=198219} 2024-09-20T22:55:10.463 INFO:tasks.rados.rados.0.smithi084.stdout:3318: writing smithi08477203-50 from 558296 to 1059412 tid 1 2024-09-20T22:55:10.468 INFO:tasks.rados.rados.0.smithi084.stdout:3318: writing smithi08477203-50 from 1474512 to 2075237 tid 2 2024-09-20T22:55:10.474 INFO:tasks.rados.rados.0.smithi084.stdout:3318: writing smithi08477203-50 from 2760077 to 2958296 tid 3 2024-09-20T22:55:10.474 INFO:tasks.rados.rados.0.smithi084.stdout:3319: delete oid 43 current snap is 381 2024-09-20T22:55:10.474 INFO:tasks.rados.rados.0.smithi084.stdout:3314: finishing write tid 3 to smithi08477203-34 2024-09-20T22:55:10.475 INFO:tasks.rados.rados.0.smithi084.stdout:3317: finishing rollback tid 1 to smithi08477203-26 2024-09-20T22:55:10.475 INFO:tasks.rados.rados.0.smithi084.stdout:3316: finishing rollback tid 0 to smithi08477203-9 2024-09-20T22:55:10.475 INFO:tasks.rados.rados.0.smithi084.stdout:3315: finishing rollback tid 1 to smithi08477203-29 2024-09-20T22:55:10.475 INFO:tasks.rados.rados.0.smithi084.stdout:3315: finishing rollback tid 2 to smithi08477203-29 2024-09-20T22:55:10.475 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 29 v 2659 (ObjNum 949 snap 311 seq_num 949) dirty exists 2024-09-20T22:55:10.475 INFO:tasks.rados.rados.0.smithi084.stdout:3314: finishing write tid 4 to smithi08477203-34 2024-09-20T22:55:10.475 INFO:tasks.rados.rados.0.smithi084.stdout:3317: finishing rollback tid 2 to smithi08477203-26 2024-09-20T22:55:10.475 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 26 v 2473 (ObjNum 906 snap 297 seq_num 906) dirty exists 2024-09-20T22:55:10.475 INFO:tasks.rados.rados.0.smithi084.stdout:3314: finishing write tid 5 to smithi08477203-34 2024-09-20T22:55:10.475 INFO:tasks.rados.rados.0.smithi084.stdout:3314: oid 34 updating version 0 to 1850 2024-09-20T22:55:10.475 INFO:tasks.rados.rados.0.smithi084.stdout:3314: oid 34 version 1850 is already newer than 1847 2024-09-20T22:55:10.475 INFO:tasks.rados.rados.0.smithi084.stdout:3314: oid 34 version 1850 is already newer than 1848 2024-09-20T22:55:10.475 INFO:tasks.rados.rados.0.smithi084.stdout:3314: oid 34 version 1850 is already newer than 1849 2024-09-20T22:55:10.475 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 34 v 1850 (ObjNum 1127 snap 381 seq_num 1127) dirty exists 2024-09-20T22:55:10.476 INFO:tasks.rados.rados.0.smithi084.stdout:3314: left oid 34 (ObjNum 1127 snap 381 seq_num 1127) 2024-09-20T22:55:10.476 INFO:tasks.rados.rados.0.smithi084.stdout:3316: finishing rollback tid 1 to smithi08477203-9 2024-09-20T22:55:10.476 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 9 v 2463 (ObjNum 49 snap 0 seq_num 1953066355) dirty dne 2024-09-20T22:55:10.476 INFO:tasks.rados.rados.0.smithi084.stdout:3318: finishing write tid 1 to smithi08477203-50 2024-09-20T22:55:10.476 INFO:tasks.rados.rados.0.smithi084.stdout:3318: finishing write tid 2 to smithi08477203-50 2024-09-20T22:55:10.476 INFO:tasks.rados.rados.0.smithi084.stdout:3314: done (5 left) 2024-09-20T22:55:10.476 INFO:tasks.rados.rados.0.smithi084.stdout:3315: done (4 left) 2024-09-20T22:55:10.476 INFO:tasks.rados.rados.0.smithi084.stdout:3316: done (3 left) 2024-09-20T22:55:10.476 INFO:tasks.rados.rados.0.smithi084.stdout:3317: done (2 left) 2024-09-20T22:55:10.476 INFO:tasks.rados.rados.0.smithi084.stdout:3319: done (1 left) 2024-09-20T22:55:10.477 INFO:tasks.rados.rados.0.smithi084.stdout:3320: snap_remove snap 340 2024-09-20T22:55:10.495 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:55:10.495 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:55:10.497 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:55:10.510 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:55:10.510 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:55:10.519 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:10.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:10.519 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:55:10.526 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:55:10.526 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:55:10.542 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-20T22:55:10.542 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T22:55:10.558 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:55:10.559 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:55:10.575 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:55:10.576 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:55:10.593 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:55:10.593 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:55:10.593 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:55:10.609 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:55:10.610 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:55:10.620 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:55:10.641 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:10.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:10.642 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:55:10.743 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:55:10.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:10.766 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:10.766 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:55:10.867 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:55:10.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:10.889 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:10.889 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:55:10.947 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:10 smithi154 ceph-mon[33264]: 6.10 deep-scrub starts 2024-09-20T22:55:10.947 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:10 smithi154 ceph-mon[33264]: 6.10 deep-scrub ok 2024-09-20T22:55:10.947 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:10 smithi154 ceph-mon[33264]: 6.5 scrub starts 2024-09-20T22:55:10.947 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:10 smithi154 ceph-mon[33264]: 6.5 scrub ok 2024-09-20T22:55:10.947 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:10 smithi154 ceph-mon[33264]: pgmap v1146: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 535 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 16 MiB/s rd, 5.1 MiB/s wr, 30 op/s 2024-09-20T22:55:10.947 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:10 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]: dispatch 2024-09-20T22:55:10.948 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:10 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]: dispatch 2024-09-20T22:55:10.948 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:10 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]: dispatch 2024-09-20T22:55:10.948 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:10 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 2]}]: dispatch 2024-09-20T22:55:10.948 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:10 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]: dispatch 2024-09-20T22:55:10.948 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:10 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 2]}]: dispatch 2024-09-20T22:55:10.948 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:10 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]': finished 2024-09-20T22:55:10.948 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:10 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]': finished 2024-09-20T22:55:10.948 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:10 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 2]}]': finished 2024-09-20T22:55:10.948 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:10 smithi154 ceph-mon[33264]: osdmap e830: 8 total, 8 up, 8 in 2024-09-20T22:55:10.948 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:10 smithi154 ceph-mon[33264]: 5.0 scrub starts 2024-09-20T22:55:10.948 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:10 smithi154 ceph-mon[33264]: 5.0 scrub ok 2024-09-20T22:55:10.948 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:10 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/3664741221' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:55:10.974 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:10 smithi084 ceph-mon[40024]: 6.10 deep-scrub starts 2024-09-20T22:55:10.974 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:10 smithi084 ceph-mon[40024]: 6.10 deep-scrub ok 2024-09-20T22:55:10.974 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:10 smithi084 ceph-mon[40024]: 6.5 scrub starts 2024-09-20T22:55:10.974 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:10 smithi084 ceph-mon[40024]: 6.5 scrub ok 2024-09-20T22:55:10.974 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:10 smithi084 ceph-mon[40024]: pgmap v1146: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 535 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 16 MiB/s rd, 5.1 MiB/s wr, 30 op/s 2024-09-20T22:55:10.974 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:10 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]: dispatch 2024-09-20T22:55:10.974 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:10 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]: dispatch 2024-09-20T22:55:10.975 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:10 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]: dispatch 2024-09-20T22:55:10.975 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:10 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 2]}]: dispatch 2024-09-20T22:55:10.975 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:10 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]: dispatch 2024-09-20T22:55:10.975 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:10 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 2]}]: dispatch 2024-09-20T22:55:10.975 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:10 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]': finished 2024-09-20T22:55:10.975 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:10 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]': finished 2024-09-20T22:55:10.975 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:10 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 2]}]': finished 2024-09-20T22:55:10.975 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:10 smithi084 ceph-mon[40024]: osdmap e830: 8 total, 8 up, 8 in 2024-09-20T22:55:10.975 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:10 smithi084 ceph-mon[40024]: 5.0 scrub starts 2024-09-20T22:55:10.975 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:10 smithi084 ceph-mon[40024]: 5.0 scrub ok 2024-09-20T22:55:10.975 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:10 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/3664741221' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:55:10.975 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:10 smithi084 ceph-mon[29774]: 6.10 deep-scrub starts 2024-09-20T22:55:10.976 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:10 smithi084 ceph-mon[29774]: 6.10 deep-scrub ok 2024-09-20T22:55:10.976 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:10 smithi084 ceph-mon[29774]: 6.5 scrub starts 2024-09-20T22:55:10.976 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:10 smithi084 ceph-mon[29774]: 6.5 scrub ok 2024-09-20T22:55:10.976 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:10 smithi084 ceph-mon[29774]: pgmap v1146: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 535 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 16 MiB/s rd, 5.1 MiB/s wr, 30 op/s 2024-09-20T22:55:10.976 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:10 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]: dispatch 2024-09-20T22:55:10.976 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:10 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]: dispatch 2024-09-20T22:55:10.976 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:10 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]: dispatch 2024-09-20T22:55:10.976 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:10 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 2]}]: dispatch 2024-09-20T22:55:10.976 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:10 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]: dispatch 2024-09-20T22:55:10.976 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:10 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 2]}]: dispatch 2024-09-20T22:55:10.976 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:10 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]': finished 2024-09-20T22:55:10.976 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:10 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]': finished 2024-09-20T22:55:10.976 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:10 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 2]}]': finished 2024-09-20T22:55:10.976 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:10 smithi084 ceph-mon[29774]: osdmap e830: 8 total, 8 up, 8 in 2024-09-20T22:55:10.976 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:10 smithi084 ceph-mon[29774]: 5.0 scrub starts 2024-09-20T22:55:10.977 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:10 smithi084 ceph-mon[29774]: 5.0 scrub ok 2024-09-20T22:55:10.977 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:10 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/3664741221' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:55:10.990 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:55:11.011 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:11.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:11.012 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:55:11.047 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-20T22:55:11.047 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [], 2024-09-20T22:55:11.047 INFO:teuthology.orchestra.run.smithi154.stdout: "num_ops": 0 2024-09-20T22:55:11.047 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-20T22:55:11.113 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:55:11.136 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:11.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:11.137 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:55:11.238 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:55:11.259 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:11.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:11.260 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:55:11.285 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd set noscrub 2024-09-20T22:55:11.338 INFO:tasks.rados.rados.0.smithi084.stdout:3320: done (1 left) 2024-09-20T22:55:11.338 INFO:tasks.rados.rados.0.smithi084.stdout:3321: read oid 16 snap -1 2024-09-20T22:55:11.338 INFO:tasks.rados.rados.0.smithi084.stdout:3321: expect (ObjNum 419 snap 105 seq_num 419) 2024-09-20T22:55:11.338 INFO:tasks.rados.rados.0.smithi084.stdout:3322: read oid 12 snap -1 2024-09-20T22:55:11.339 INFO:tasks.rados.rados.0.smithi084.stdout:3322: expect (ObjNum 795 snap 254 seq_num 795) 2024-09-20T22:55:11.339 INFO:tasks.rados.rados.0.smithi084.stdout:3323: copy_from oid 21 from oid 45 current snap is 381 2024-09-20T22:55:11.339 INFO:tasks.rados.rados.0.smithi084.stdout:3324: snap_remove snap 367 2024-09-20T22:55:11.361 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:55:11.401 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:11.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:11.402 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:55:11.503 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:55:11.525 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:11.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:11.526 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:55:11.564 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:55:11.626 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:55:11.648 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:11.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:11.649 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:55:11.750 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:55:11.771 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:11.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:11.772 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:55:11.873 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:55:11.880 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.6 dump_blocked_ops 2024-09-20T22:55:11.895 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:11.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:11.895 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:55:11.996 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:11 smithi084 ceph-mon[40024]: 5.4 scrub starts 2024-09-20T22:55:11.996 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:11 smithi084 ceph-mon[40024]: 5.4 scrub ok 2024-09-20T22:55:11.996 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:11 smithi084 ceph-mon[40024]: 6.4 scrub starts 2024-09-20T22:55:11.996 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:11 smithi084 ceph-mon[40024]: 6.4 scrub ok 2024-09-20T22:55:11.996 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:11 smithi084 ceph-mon[40024]: osdmap e831: 8 total, 8 up, 8 in 2024-09-20T22:55:11.996 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:11 smithi084 ceph-mon[29774]: 5.4 scrub starts 2024-09-20T22:55:11.996 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:11 smithi084 ceph-mon[29774]: 5.4 scrub ok 2024-09-20T22:55:11.996 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:11 smithi084 ceph-mon[29774]: 6.4 scrub starts 2024-09-20T22:55:11.996 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:11 smithi084 ceph-mon[29774]: 6.4 scrub ok 2024-09-20T22:55:11.996 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:11 smithi084 ceph-mon[29774]: osdmap e831: 8 total, 8 up, 8 in 2024-09-20T22:55:11.997 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:55:12.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:11 smithi154 ceph-mon[33264]: 5.4 scrub starts 2024-09-20T22:55:12.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:11 smithi154 ceph-mon[33264]: 5.4 scrub ok 2024-09-20T22:55:12.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:11 smithi154 ceph-mon[33264]: 6.4 scrub starts 2024-09-20T22:55:12.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:11 smithi154 ceph-mon[33264]: 6.4 scrub ok 2024-09-20T22:55:12.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:11 smithi154 ceph-mon[33264]: osdmap e831: 8 total, 8 up, 8 in 2024-09-20T22:55:12.017 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:12.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:12.018 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:55:12.119 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:55:12.124 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:55:12.141 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:12.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:12.142 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:55:12.243 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:55:12.263 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:12.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:12.264 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:55:12.332 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is set 2024-09-20T22:55:12.338 INFO:tasks.rados.rados.0.smithi084.stdout:3324: done (4 left) 2024-09-20T22:55:12.338 INFO:tasks.rados.rados.0.smithi084.stdout:3325: read oid 36 snap -1 2024-09-20T22:55:12.338 INFO:tasks.rados.rados.0.smithi084.stdout:3325: expect deleted 2024-09-20T22:55:12.338 INFO:tasks.rados.rados.0.smithi084.stdout:3326: rollback oid 13 current snap is 381 2024-09-20T22:55:12.338 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 13 to 356 2024-09-20T22:55:12.338 INFO:tasks.rados.rados.0.smithi084.stdout:3318: finishing write tid 3 to smithi08477203-50 2024-09-20T22:55:12.338 INFO:tasks.rados.rados.0.smithi084.stdout:3318: finishing write tid 4 to smithi08477203-50 2024-09-20T22:55:12.338 INFO:tasks.rados.rados.0.smithi084.stdout:3318: finishing write tid 5 to smithi08477203-50 2024-09-20T22:55:12.338 INFO:tasks.rados.rados.0.smithi084.stdout:3318: oid 50 updating version 0 to 1888 2024-09-20T22:55:12.338 INFO:tasks.rados.rados.0.smithi084.stdout:3318: oid 50 updating version 1888 to 1891 2024-09-20T22:55:12.339 INFO:tasks.rados.rados.0.smithi084.stdout:3318: oid 50 version 1891 is already newer than 1890 2024-09-20T22:55:12.339 INFO:tasks.rados.rados.0.smithi084.stdout:3318: oid 50 version 1891 is already newer than 1889 2024-09-20T22:55:12.339 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 50 v 1891 (ObjNum 1128 snap 381 seq_num 1128) dirty exists 2024-09-20T22:55:12.339 INFO:tasks.rados.rados.0.smithi084.stdout:3318: left oid 50 (ObjNum 1128 snap 381 seq_num 1128) 2024-09-20T22:55:12.339 INFO:tasks.rados.rados.0.smithi084.stdout:3323: finishing copy_from to smithi08477203-21 2024-09-20T22:55:12.339 INFO:tasks.rados.rados.0.smithi084.stdout:3323: got expected ENOENT (src dne) 2024-09-20T22:55:12.339 INFO:tasks.rados.rados.0.smithi084.stdout:3323: finishing copy_from racing read to smithi08477203-21 2024-09-20T22:55:12.339 INFO:tasks.rados.rados.0.smithi084.stdout:3318: done (5 left) 2024-09-20T22:55:12.339 INFO:tasks.rados.rados.0.smithi084.stdout:3323: done (4 left) 2024-09-20T22:55:12.339 INFO:tasks.rados.rados.0.smithi084.stdout:3327: delete oid 9 current snap is 381 2024-09-20T22:55:12.339 INFO:tasks.rados.rados.0.smithi084.stdout:3321: expect (ObjNum 419 snap 105 seq_num 419) 2024-09-20T22:55:12.366 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:55:12.388 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:12.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:12.389 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:55:12.489 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:55:12.511 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:12.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:12.512 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:55:12.512 INFO:tasks.rados.rados.0.smithi084.stdout:3322: expect (ObjNum 795 snap 254 seq_num 795) 2024-09-20T22:55:12.613 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:55:12.621 INFO:tasks.rados.rados.0.smithi084.stdout:3326: finishing rollback tid 1 to smithi08477203-13 2024-09-20T22:55:12.621 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 13 v 2576 (ObjNum 129 snap 0 seq_num 712582594) dirty dne 2024-09-20T22:55:12.621 INFO:tasks.rados.rados.0.smithi084.stdout:3321: done (4 left) 2024-09-20T22:55:12.621 INFO:tasks.rados.rados.0.smithi084.stdout:3322: done (3 left) 2024-09-20T22:55:12.621 INFO:tasks.rados.rados.0.smithi084.stdout:3325: done (2 left) 2024-09-20T22:55:12.621 INFO:tasks.rados.rados.0.smithi084.stdout:3326: done (1 left) 2024-09-20T22:55:12.621 INFO:tasks.rados.rados.0.smithi084.stdout:3327: done (0 left) 2024-09-20T22:55:12.622 INFO:tasks.rados.rados.0.smithi084.stdout:3328: write oid 34 current snap is 381 2024-09-20T22:55:12.622 INFO:tasks.rados.rados.0.smithi084.stdout:3328: seq_num 1131 ranges {608717=726863,1933605=748052} 2024-09-20T22:55:12.627 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-20T22:55:12.627 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [], 2024-09-20T22:55:12.628 INFO:teuthology.orchestra.run.smithi154.stdout: "complaint_time": 30, 2024-09-20T22:55:12.628 INFO:teuthology.orchestra.run.smithi154.stdout: "num_blocked_ops": 0 2024-09-20T22:55:12.628 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-20T22:55:12.628 INFO:tasks.rados.rados.0.smithi084.stdout:3328: writing smithi08477203-34 from 608717 to 1335580 tid 1 2024-09-20T22:55:12.634 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:12.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:12.635 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:55:12.636 INFO:tasks.rados.rados.0.smithi084.stdout:3328: writing smithi08477203-34 from 1933605 to 2681657 tid 2 2024-09-20T22:55:12.636 INFO:tasks.rados.rados.0.smithi084.stdout:3329: delete oid 45 current snap is 381 2024-09-20T22:55:12.636 INFO:tasks.rados.rados.0.smithi084.stdout:3328: finishing write tid 1 to smithi08477203-34 2024-09-20T22:55:12.638 INFO:tasks.rados.rados.0.smithi084.stdout:3329: done (1 left) 2024-09-20T22:55:12.638 INFO:tasks.rados.rados.0.smithi084.stdout:3330: write (excl) oid 48 current snap is 381 2024-09-20T22:55:12.638 INFO:tasks.rados.rados.0.smithi084.stdout:3330: seq_num 1133 ranges {642331=593714,1766922=751721} 2024-09-20T22:55:12.645 INFO:tasks.rados.rados.0.smithi084.stdout:3330: writing smithi08477203-48 from 642331 to 1236045 tid 1 2024-09-20T22:55:12.652 INFO:tasks.rados.rados.0.smithi084.stdout:3330: writing smithi08477203-48 from 1766922 to 2518643 tid 2 2024-09-20T22:55:12.652 INFO:tasks.rados.rados.0.smithi084.stdout:3331: snap_create 2024-09-20T22:55:12.652 INFO:tasks.rados.rados.0.smithi084.stdout:3328: finishing write tid 2 to smithi08477203-34 2024-09-20T22:55:12.652 INFO:tasks.rados.rados.0.smithi084.stdout:3328: finishing write tid 3 to smithi08477203-34 2024-09-20T22:55:12.652 INFO:tasks.rados.rados.0.smithi084.stdout:3328: finishing write tid 4 to smithi08477203-34 2024-09-20T22:55:12.653 INFO:tasks.rados.rados.0.smithi084.stdout:3328: oid 34 updating version 0 to 1851 2024-09-20T22:55:12.653 INFO:tasks.rados.rados.0.smithi084.stdout:3328: oid 34 updating version 1851 to 1852 2024-09-20T22:55:12.653 INFO:tasks.rados.rados.0.smithi084.stdout:3328: oid 34 updating version 1852 to 1853 2024-09-20T22:55:12.653 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 34 v 1853 (ObjNum 1131 snap 381 seq_num 1131) dirty exists 2024-09-20T22:55:12.653 INFO:tasks.rados.rados.0.smithi084.stdout:3328: left oid 34 (ObjNum 1131 snap 381 seq_num 1131) 2024-09-20T22:55:12.653 INFO:tasks.rados.rados.0.smithi084.stdout:3330: finishing write tid 1 to smithi08477203-48 2024-09-20T22:55:12.656 INFO:tasks.rados.rados.0.smithi084.stdout:3330: finishing write tid 2 to smithi08477203-48 2024-09-20T22:55:12.656 INFO:tasks.rados.rados.0.smithi084.stdout:3330: finishing write tid 3 to smithi08477203-48 2024-09-20T22:55:12.656 INFO:tasks.rados.rados.0.smithi084.stdout:3330: finishing write tid 4 to smithi08477203-48 2024-09-20T22:55:12.656 INFO:tasks.rados.rados.0.smithi084.stdout:3330: oid 48 updating version 0 to 1932 2024-09-20T22:55:12.656 INFO:tasks.rados.rados.0.smithi084.stdout:3330: oid 48 version 1932 is already newer than 1931 2024-09-20T22:55:12.656 INFO:tasks.rados.rados.0.smithi084.stdout:3330: oid 48 version 1932 is already newer than 1930 2024-09-20T22:55:12.656 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 48 v 1932 (ObjNum 1133 snap 381 seq_num 1133) dirty exists 2024-09-20T22:55:12.657 INFO:tasks.rados.rados.0.smithi084.stdout:3330: left oid 48 (ObjNum 1133 snap 381 seq_num 1133) 2024-09-20T22:55:12.736 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:55:12.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:12.759 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:12.759 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:55:12.860 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:55:12.881 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:12.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:12.882 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:55:12.969 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:12 smithi154 ceph-mon[33264]: 3.17 scrub starts 2024-09-20T22:55:12.970 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:12 smithi154 ceph-mon[33264]: 3.17 scrub ok 2024-09-20T22:55:12.970 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:12 smithi154 ceph-mon[33264]: pgmap v1149: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 535 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 16 MiB/s rd, 5.1 MiB/s wr, 30 op/s 2024-09-20T22:55:12.970 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:12 smithi154 ceph-mon[33264]: 5.11 scrub starts 2024-09-20T22:55:12.970 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:12 smithi154 ceph-mon[33264]: 5.11 scrub ok 2024-09-20T22:55:12.970 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:12 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/4280056068' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-20T22:55:12.970 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:12 smithi154 ceph-mon[33264]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:55:12.970 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:12 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/4280056068' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T22:55:12.970 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:12 smithi154 ceph-mon[33264]: osdmap e832: 8 total, 8 up, 8 in 2024-09-20T22:55:12.983 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:55:12.984 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:12 smithi084 ceph-mon[29774]: 3.17 scrub starts 2024-09-20T22:55:12.984 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:12 smithi084 ceph-mon[29774]: 3.17 scrub ok 2024-09-20T22:55:12.984 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:12 smithi084 ceph-mon[29774]: pgmap v1149: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 535 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 16 MiB/s rd, 5.1 MiB/s wr, 30 op/s 2024-09-20T22:55:12.984 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:12 smithi084 ceph-mon[29774]: 5.11 scrub starts 2024-09-20T22:55:12.984 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:12 smithi084 ceph-mon[29774]: 5.11 scrub ok 2024-09-20T22:55:12.984 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:12 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/4280056068' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-20T22:55:12.984 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:12 smithi084 ceph-mon[29774]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:55:12.984 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:12 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/4280056068' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T22:55:12.984 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:12 smithi084 ceph-mon[29774]: osdmap e832: 8 total, 8 up, 8 in 2024-09-20T22:55:12.985 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:12 smithi084 ceph-mon[40024]: 3.17 scrub starts 2024-09-20T22:55:12.985 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:12 smithi084 ceph-mon[40024]: 3.17 scrub ok 2024-09-20T22:55:12.985 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:12 smithi084 ceph-mon[40024]: pgmap v1149: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 535 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 16 MiB/s rd, 5.1 MiB/s wr, 30 op/s 2024-09-20T22:55:12.985 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:12 smithi084 ceph-mon[40024]: 5.11 scrub starts 2024-09-20T22:55:12.985 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:12 smithi084 ceph-mon[40024]: 5.11 scrub ok 2024-09-20T22:55:12.985 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:12 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/4280056068' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-20T22:55:12.985 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:12 smithi084 ceph-mon[40024]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:55:12.985 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:12 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/4280056068' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T22:55:12.985 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:12 smithi084 ceph-mon[40024]: osdmap e832: 8 total, 8 up, 8 in 2024-09-20T22:55:13.006 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:13.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:13.007 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:55:13.107 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:55:13.128 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:13.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:13.129 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:55:13.230 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:55:13.251 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:13.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:13.253 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:55:13.324 INFO:tasks.rados.rados.0.smithi084.stdout:3328: done (2 left) 2024-09-20T22:55:13.324 INFO:tasks.rados.rados.0.smithi084.stdout:3330: done (1 left) 2024-09-20T22:55:13.325 INFO:tasks.rados.rados.0.smithi084.stdout:3331: done (0 left) 2024-09-20T22:55:13.325 INFO:tasks.rados.rados.0.smithi084.stdout:3332: delete oid 3 current snap is 382 2024-09-20T22:55:13.326 INFO:tasks.rados.rados.0.smithi084.stdout:3332: done (0 left) 2024-09-20T22:55:13.326 INFO:tasks.rados.rados.0.smithi084.stdout:3333: write (excl) oid 23 current snap is 382 2024-09-20T22:55:13.327 INFO:tasks.rados.rados.0.smithi084.stdout:3333: seq_num 1135 ranges {675945=744212,2167533=755390} 2024-09-20T22:55:13.334 INFO:tasks.rados.rados.0.smithi084.stdout:3333: writing smithi08477203-23 from 675945 to 1420157 tid 1 2024-09-20T22:55:13.341 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-20T22:55:13.344 INFO:tasks.rados.rados.0.smithi084.stdout:3333: writing smithi08477203-23 from 2167533 to 2922923 tid 2 2024-09-20T22:55:13.344 INFO:tasks.rados.rados.0.smithi084.stdout:3334: snap_remove snap 323 2024-09-20T22:55:13.353 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:55:13.362 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.6 dump_historic_ops 2024-09-20T22:55:13.405 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:13.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:13.407 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:55:13.507 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:55:13.529 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:13.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:13.530 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:55:13.591 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:55:13.611 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:55:13.631 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:55:13.653 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:13.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:13.654 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:55:13.755 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:55:13.776 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:13.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:13.777 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:55:13.878 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:55:13.900 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:13.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:13.901 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:55:13.987 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:13 smithi084 ceph-mon[40024]: 6.14 scrub starts 2024-09-20T22:55:13.987 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:13 smithi084 ceph-mon[40024]: 6.14 scrub ok 2024-09-20T22:55:13.987 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:13 smithi084 ceph-mon[40024]: osdmap e833: 8 total, 8 up, 8 in 2024-09-20T22:55:13.987 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:13 smithi084 ceph-mon[29774]: 6.14 scrub starts 2024-09-20T22:55:13.987 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:13 smithi084 ceph-mon[29774]: 6.14 scrub ok 2024-09-20T22:55:13.987 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:13 smithi084 ceph-mon[29774]: osdmap e833: 8 total, 8 up, 8 in 2024-09-20T22:55:14.002 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:55:14.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:13 smithi154 ceph-mon[33264]: 6.14 scrub starts 2024-09-20T22:55:14.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:13 smithi154 ceph-mon[33264]: 6.14 scrub ok 2024-09-20T22:55:14.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:13 smithi154 ceph-mon[33264]: osdmap e833: 8 total, 8 up, 8 in 2024-09-20T22:55:14.023 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:14.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:14.023 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:55:14.095 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:55:14.095 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:55:14.103 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:55:14.103 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:55:14.116 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-20T22:55:14.117 INFO:teuthology.orchestra.run.smithi154.stdout: "size": 20, 2024-09-20T22:55:14.117 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 600, 2024-09-20T22:55:14.117 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [ 2024-09-20T22:55:14.117 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.117 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:3132 7.19 7:9eac4560:::smithi08477203-30:head [copy-from ver 389] snapc 77=[77,76,75,74,73,70,6b] ondisk+write+known_if_redirected+supports_pool_eio e328)", 2024-09-20T22:55:14.117 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:46:48.209876+0000", 2024-09-20T22:55:14.117 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 505.90405084600002, 2024-09-20T22:55:14.117 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.030505467000000001, 2024-09-20T22:55:14.117 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:55:14.117 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:55:14.117 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:55:14.117 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:55:14.117 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:55:14.118 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:55:14.118 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 3132 2024-09-20T22:55:14.118 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.118 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:55:14.118 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.118 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:55:14.118 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.209876+0000", 2024-09-20T22:55:14.118 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:55:14.118 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.118 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.118 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:55:14.118 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.209876+0000", 2024-09-20T22:55:14.118 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:55:14.118 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.119 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.119 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:55:14.119 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.209876+0000", 2024-09-20T22:55:14.119 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.7e-07 2024-09-20T22:55:14.119 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.119 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.119 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:55:14.119 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.209880+0000", 2024-09-20T22:55:14.119 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.4149999999999999e-06 2024-09-20T22:55:14.119 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.119 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.119 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:55:14.119 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.209886+0000", 2024-09-20T22:55:14.120 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.5819999999999998e-06 2024-09-20T22:55:14.120 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.120 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.120 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:55:14.120 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.209893+0000", 2024-09-20T22:55:14.120 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.1820000000000003e-06 2024-09-20T22:55:14.120 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.120 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.120 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:55:14.120 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.209945+0000", 2024-09-20T22:55:14.120 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.2528999999999998e-05 2024-09-20T22:55:14.120 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.120 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.120 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:55:14.120 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.210028+0000", 2024-09-20T22:55:14.121 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.3067999999999998e-05 2024-09-20T22:55:14.121 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.121 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.121 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 2,7", 2024-09-20T22:55:14.121 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.223189+0000", 2024-09-20T22:55:14.121 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.013160579 2024-09-20T22:55:14.121 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.121 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.121 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:55:14.121 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.232498+0000", 2024-09-20T22:55:14.121 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0093088760000000007 2024-09-20T22:55:14.121 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.121 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.121 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:55:14.122 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.240058+0000", 2024-09-20T22:55:14.122 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0075599930000000001 2024-09-20T22:55:14.122 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.122 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.122 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:55:14.122 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.240300+0000", 2024-09-20T22:55:14.122 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00024190999999999999 2024-09-20T22:55:14.122 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.122 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.122 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:55:14.122 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.240343+0000", 2024-09-20T22:55:14.122 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.2972000000000001e-05 2024-09-20T22:55:14.122 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.122 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.123 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:55:14.123 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.240381+0000", 2024-09-20T22:55:14.123 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.8890999999999997e-05 2024-09-20T22:55:14.123 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:55:14.123 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:55:14.123 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:55:14.123 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.123 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.123 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:3747 7.c 7:3f865ef5:::smithi08477203-28:head [copy-from ver 108] snapc 98=[98,97,96,95,94,93,92,8e,8d,8c,8b,8a,88,87,86,80,7f,73] ondisk+write+known_if_redirected+supports_pool_eio e383)", 2024-09-20T22:55:14.123 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:47:43.253429+0000", 2024-09-20T22:55:14.123 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 450.860497694, 2024-09-20T22:55:14.123 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.028706249999999999, 2024-09-20T22:55:14.123 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:55:14.123 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:55:14.124 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:55:14.124 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:55:14.124 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:55:14.124 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:55:14.124 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 3747 2024-09-20T22:55:14.124 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.124 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:55:14.124 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.124 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:55:14.124 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.253429+0000", 2024-09-20T22:55:14.124 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:55:14.124 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.124 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.124 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:55:14.124 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.253429+0000", 2024-09-20T22:55:14.125 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:55:14.125 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.125 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.125 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:55:14.125 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.253431+0000", 2024-09-20T22:55:14.125 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.829e-06 2024-09-20T22:55:14.125 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.125 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.125 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:55:14.125 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.253438+0000", 2024-09-20T22:55:14.125 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.418e-06 2024-09-20T22:55:14.125 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.125 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.125 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:55:14.125 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.253451+0000", 2024-09-20T22:55:14.126 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2317000000000001e-05 2024-09-20T22:55:14.126 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.126 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.126 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:55:14.126 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.253457+0000", 2024-09-20T22:55:14.126 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.4710000000000004e-06 2024-09-20T22:55:14.126 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.126 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.126 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:55:14.126 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.253530+0000", 2024-09-20T22:55:14.126 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.3047999999999996e-05 2024-09-20T22:55:14.126 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.126 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.126 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op must wait for map", 2024-09-20T22:55:14.127 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.253536+0000", 2024-09-20T22:55:14.127 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.5140000000000001e-06 2024-09-20T22:55:14.127 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.127 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.127 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:55:14.127 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.257582+0000", 2024-09-20T22:55:14.127 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0040464849999999998 2024-09-20T22:55:14.127 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.127 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.127 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:55:14.127 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.257697+0000", 2024-09-20T22:55:14.127 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00011471 2024-09-20T22:55:14.127 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.127 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.128 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 3,7", 2024-09-20T22:55:14.128 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.267067+0000", 2024-09-20T22:55:14.128 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0093696560000000005 2024-09-20T22:55:14.128 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.128 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.128 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:55:14.128 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.276328+0000", 2024-09-20T22:55:14.128 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0092619669999999994 2024-09-20T22:55:14.128 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.128 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.128 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:55:14.128 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.278462+0000", 2024-09-20T22:55:14.128 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0021334430000000001 2024-09-20T22:55:14.128 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.129 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.129 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:55:14.129 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.282036+0000", 2024-09-20T22:55:14.129 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0035745249999999998 2024-09-20T22:55:14.129 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.129 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.129 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:55:14.129 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.282094+0000", 2024-09-20T22:55:14.129 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.7312999999999998e-05 2024-09-20T22:55:14.129 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.129 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.129 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:55:14.129 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.282135+0000", 2024-09-20T22:55:14.129 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.1554000000000002e-05 2024-09-20T22:55:14.129 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:55:14.130 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:55:14.130 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:55:14.130 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.130 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.130 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:55:14.131 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:55:14.131 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:55:14.132 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:3856 7.12 7:4fa4b2b3:::smithi08477203-21:head [copy-from ver 170] snapc 9e=[9e,9d,9c,9b,9a,98,96,95,94,93,92,8e,8d,8c,8b,8a,88,87,86,80,7f,73] ondisk+write+known_if_redirected+supports_pool_eio e391)", 2024-09-20T22:55:14.132 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:47:51.204384+0000", 2024-09-20T22:55:14.132 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 442.90954237599999, 2024-09-20T22:55:14.132 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0068049349999999, 2024-09-20T22:55:14.132 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:55:14.132 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:55:14.132 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:55:14.132 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:55:14.132 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:55:14.133 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:55:14.133 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 3856 2024-09-20T22:55:14.133 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.133 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:55:14.133 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.133 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:55:14.133 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:51.204384+0000", 2024-09-20T22:55:14.133 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:55:14.133 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.133 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.133 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:55:14.133 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:51.204384+0000", 2024-09-20T22:55:14.133 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:55:14.133 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.133 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.134 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:55:14.134 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:51.204386+0000", 2024-09-20T22:55:14.134 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.57e-06 2024-09-20T22:55:14.134 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.134 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.134 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:55:14.134 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:51.204391+0000", 2024-09-20T22:55:14.134 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.6009999999999997e-06 2024-09-20T22:55:14.134 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.134 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.134 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:55:14.134 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:51.204400+0000", 2024-09-20T22:55:14.134 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.0499999999999997e-06 2024-09-20T22:55:14.134 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.135 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.135 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:55:14.135 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:51.204410+0000", 2024-09-20T22:55:14.135 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.01e-05 2024-09-20T22:55:14.135 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.135 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.135 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:55:14.135 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:51.204944+0000", 2024-09-20T22:55:14.135 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00053440599999999997 2024-09-20T22:55:14.135 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.135 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.135 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-20T22:55:14.135 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:51.204954+0000", 2024-09-20T22:55:14.135 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0162e-05 2024-09-20T22:55:14.136 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.136 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.136 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:55:14.136 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:52.210122+0000", 2024-09-20T22:55:14.136 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0051679019999999 2024-09-20T22:55:14.136 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.136 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.136 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:55:14.136 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:52.210316+0000", 2024-09-20T22:55:14.136 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000193704 2024-09-20T22:55:14.136 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.136 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.136 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:55:14.136 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:52.211189+0000", 2024-09-20T22:55:14.137 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00087343999999999996 2024-09-20T22:55:14.137 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:55:14.137 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:55:14.137 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:55:14.137 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.137 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.137 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:3857 7.12 7:4fa4b2b3:::smithi08477203-21:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e391)", 2024-09-20T22:55:14.137 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:47:51.204423+0000", 2024-09-20T22:55:14.137 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 442.90950393600002, 2024-09-20T22:55:14.137 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0069932960000001, 2024-09-20T22:55:14.137 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:55:14.137 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:55:14.137 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:55:14.137 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:55:14.138 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:55:14.138 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:55:14.138 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 3857 2024-09-20T22:55:14.138 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.138 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:55:14.138 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.138 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:55:14.138 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:51.204423+0000", 2024-09-20T22:55:14.138 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:55:14.138 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.138 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.138 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:55:14.138 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:51.204423+0000", 2024-09-20T22:55:14.138 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:55:14.139 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.139 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.139 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:55:14.139 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:51.204423+0000", 2024-09-20T22:55:14.139 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.3099999999999998e-07 2024-09-20T22:55:14.139 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.139 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.139 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:55:14.139 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:51.204426+0000", 2024-09-20T22:55:14.139 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.4030000000000001e-06 2024-09-20T22:55:14.139 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.139 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.139 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:55:14.139 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:51.204432+0000", 2024-09-20T22:55:14.140 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.4099999999999996e-06 2024-09-20T22:55:14.140 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.140 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.140 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:55:14.140 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:51.204438+0000", 2024-09-20T22:55:14.140 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.4600000000000002e-06 2024-09-20T22:55:14.140 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.140 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.140 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:55:14.140 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:51.204980+0000", 2024-09-20T22:55:14.140 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00054220600000000005 2024-09-20T22:55:14.140 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.140 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.140 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-20T22:55:14.140 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:51.204987+0000", 2024-09-20T22:55:14.141 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.3320000000000003e-06 2024-09-20T22:55:14.141 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.141 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.141 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:55:14.141 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:52.211064+0000", 2024-09-20T22:55:14.141 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0060768899999999 2024-09-20T22:55:14.141 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.141 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.141 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for blocked object", 2024-09-20T22:55:14.141 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:52.211095+0000", 2024-09-20T22:55:14.141 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.1399999999999998e-05 2024-09-20T22:55:14.141 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.141 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.141 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:55:14.142 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:52.211260+0000", 2024-09-20T22:55:14.142 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00016436099999999999 2024-09-20T22:55:14.142 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.142 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.142 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:55:14.142 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:52.211310+0000", 2024-09-20T22:55:14.142 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.0417999999999997e-05 2024-09-20T22:55:14.142 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.142 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.142 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:55:14.142 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:52.211416+0000", 2024-09-20T22:55:14.142 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00010588499999999999 2024-09-20T22:55:14.142 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:55:14.142 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:55:14.143 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:55:14.143 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.143 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.143 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "MOSDPGPush(7.17 434/433 [PushOp(7:ed54065f:::smithi08477203-12:90, version: 429'392, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:ed54065f:::smithi08477203-12:90@429'392, size: 2033746, copy_subset: [(0, 2033746)], clone_subset: {7:ed54065f:::smithi08477203-12:88: [], 7:ed54065f:::smithi08477203-12:a5: []}, snapset: b1=[]:{86: [86], 88: [87], 90: [8e], 9b: [9a], a5: [a5, 9f, 9e, 9d, 9c], b1: [b1, b0, af, ae, ad, ab, a9, a7, a6]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-20T22:55:14.143 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:48:34.294270+0000", 2024-09-20T22:55:14.143 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 399.81965680600001, 2024-09-20T22:55:14.143 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.044145694999999999, 2024-09-20T22:55:14.143 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:55:14.143 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:55:14.143 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:55:14.143 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:55:14.143 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.143 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:55:14.143 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:34.294270+0000", 2024-09-20T22:55:14.143 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:55:14.144 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.144 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.144 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:55:14.144 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:34.294270+0000", 2024-09-20T22:55:14.144 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:55:14.144 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.144 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.144 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:55:14.144 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:34.294272+0000", 2024-09-20T22:55:14.144 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.576e-06 2024-09-20T22:55:14.144 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.144 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.144 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:55:14.144 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:34.337875+0000", 2024-09-20T22:55:14.145 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.043603555000000002 2024-09-20T22:55:14.145 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.145 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.145 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:55:14.145 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:34.337895+0000", 2024-09-20T22:55:14.145 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.0075999999999999e-05 2024-09-20T22:55:14.145 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.145 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.145 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:55:14.145 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:34.337909+0000", 2024-09-20T22:55:14.145 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4070000000000001e-05 2024-09-20T22:55:14.145 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.145 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.145 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:55:14.146 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:34.338016+0000", 2024-09-20T22:55:14.146 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000106969 2024-09-20T22:55:14.146 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.146 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.146 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:55:14.146 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:34.338037+0000", 2024-09-20T22:55:14.146 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.0409999999999999e-05 2024-09-20T22:55:14.146 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.146 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.146 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:55:14.146 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:34.338416+0000", 2024-09-20T22:55:14.146 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00037903900000000001 2024-09-20T22:55:14.146 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:55:14.146 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:55:14.147 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:55:14.147 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.147 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.147 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:4373 7.7 7:ea720b85:::smithi08477203-26:head [rollback af] snapc b5=[b5,b4,b3,b2,b1,b0,af,ae,ab,a9,a7,a6,a5,9f,9e,9c,9a,96,95,87,86] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e442)", 2024-09-20T22:55:14.147 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:48:42.215127+0000", 2024-09-20T22:55:14.147 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 391.89880017600001, 2024-09-20T22:55:14.147 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0149912029999999, 2024-09-20T22:55:14.147 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:55:14.147 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:55:14.147 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:55:14.147 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:55:14.147 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:55:14.147 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:55:14.147 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 4373 2024-09-20T22:55:14.148 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.148 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:55:14.148 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.148 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:55:14.148 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:42.215127+0000", 2024-09-20T22:55:14.148 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:55:14.148 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.148 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.148 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:55:14.148 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:42.215127+0000", 2024-09-20T22:55:14.148 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:55:14.148 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.148 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.148 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:55:14.149 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:42.215128+0000", 2024-09-20T22:55:14.149 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.291e-06 2024-09-20T22:55:14.149 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.149 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.149 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:55:14.149 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:42.215133+0000", 2024-09-20T22:55:14.149 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.5350000000000004e-06 2024-09-20T22:55:14.149 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.149 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.149 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:55:14.149 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:42.215146+0000", 2024-09-20T22:55:14.149 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2099999999999999e-05 2024-09-20T22:55:14.149 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.149 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.150 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:55:14.150 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:42.215154+0000", 2024-09-20T22:55:14.150 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.9190000000000005e-06 2024-09-20T22:55:14.150 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.150 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.150 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:55:14.150 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:42.215229+0000", 2024-09-20T22:55:14.150 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.4233000000000006e-05 2024-09-20T22:55:14.150 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.150 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.150 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op must wait for map", 2024-09-20T22:55:14.150 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:42.215238+0000", 2024-09-20T22:55:14.150 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.5950000000000005e-06 2024-09-20T22:55:14.150 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.151 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.151 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:55:14.151 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:42.227011+0000", 2024-09-20T22:55:14.151 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.011772553999999999 2024-09-20T22:55:14.151 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.151 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.151 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-20T22:55:14.151 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:42.227026+0000", 2024-09-20T22:55:14.151 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.5119999999999999e-05 2024-09-20T22:55:14.151 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.151 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.151 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:55:14.151 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:43.224744+0000", 2024-09-20T22:55:14.151 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.99771833399999998 2024-09-20T22:55:14.151 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.152 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.152 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:55:14.152 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:43.224967+0000", 2024-09-20T22:55:14.152 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00022256000000000001 2024-09-20T22:55:14.152 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.152 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.152 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:55:14.152 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:43.230118+0000", 2024-09-20T22:55:14.152 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0051509620000000002 2024-09-20T22:55:14.152 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:55:14.152 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:55:14.152 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:55:14.152 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.152 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.153 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:4638 7.d 7:b30e4612:::smithi08477203-42:head [copy-from ver 560] snapc c2=[c2,c0,bf,be,bc,bb,ba,b9,b8,b7,b4,b2,b1,a7,a5,9e,9c,95,87,86] ondisk+write+known_if_redirected+supports_pool_eio e468)", 2024-09-20T22:55:14.153 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:49:08.237769+0000", 2024-09-20T22:55:14.153 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 365.87615771899999, 2024-09-20T22:55:14.153 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.035118309, 2024-09-20T22:55:14.153 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:55:14.153 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:55:14.153 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:55:14.153 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:55:14.153 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:55:14.153 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:55:14.153 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 4638 2024-09-20T22:55:14.154 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.154 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:55:14.154 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.154 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:55:14.154 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:08.237769+0000", 2024-09-20T22:55:14.154 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:55:14.154 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.154 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.154 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:55:14.154 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:08.237769+0000", 2024-09-20T22:55:14.154 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:55:14.154 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.154 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.154 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:55:14.155 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:08.237772+0000", 2024-09-20T22:55:14.155 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.8629999999999999e-06 2024-09-20T22:55:14.155 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.155 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.155 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:55:14.155 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:08.237785+0000", 2024-09-20T22:55:14.155 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3039e-05 2024-09-20T22:55:14.155 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.155 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.155 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:55:14.155 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:08.237809+0000", 2024-09-20T22:55:14.156 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.4094e-05 2024-09-20T22:55:14.156 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.156 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.156 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:55:14.156 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:08.237828+0000", 2024-09-20T22:55:14.156 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.8839e-05 2024-09-20T22:55:14.156 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.156 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.156 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:55:14.156 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:08.237921+0000", 2024-09-20T22:55:14.156 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.2727999999999997e-05 2024-09-20T22:55:14.156 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.156 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.156 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op must wait for map", 2024-09-20T22:55:14.157 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:08.237929+0000", 2024-09-20T22:55:14.157 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.5609999999999999e-06 2024-09-20T22:55:14.157 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.157 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.157 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:55:14.157 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:08.245301+0000", 2024-09-20T22:55:14.157 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0073717670000000004 2024-09-20T22:55:14.157 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.157 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.157 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:55:14.157 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:08.245405+0000", 2024-09-20T22:55:14.157 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00010370999999999999 2024-09-20T22:55:14.157 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.157 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.158 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 0,1", 2024-09-20T22:55:14.158 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:08.256435+0000", 2024-09-20T22:55:14.158 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.011030659 2024-09-20T22:55:14.158 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.158 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.158 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:55:14.158 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:08.262369+0000", 2024-09-20T22:55:14.158 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0059332029999999997 2024-09-20T22:55:14.158 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.158 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.158 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:55:14.158 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:08.272100+0000", 2024-09-20T22:55:14.158 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0097312560000000006 2024-09-20T22:55:14.158 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.159 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.159 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:55:14.159 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:08.272791+0000", 2024-09-20T22:55:14.159 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00069164699999999999 2024-09-20T22:55:14.159 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.159 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.159 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:55:14.159 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:08.272858+0000", 2024-09-20T22:55:14.159 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.6532999999999995e-05 2024-09-20T22:55:14.159 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.159 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.159 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:55:14.159 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:08.272887+0000", 2024-09-20T22:55:14.160 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.9410000000000001e-05 2024-09-20T22:55:14.160 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:55:14.160 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:55:14.160 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:55:14.160 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.160 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.160 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:6018 7.4 e606/602 7:24b1a612:::smithi08477203-43:head v 606'331, mlcod=603'327)", 2024-09-20T22:55:14.160 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:51:26.433797+0000", 2024-09-20T22:55:14.160 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 227.680129787, 2024-09-20T22:55:14.160 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.029417520999999999, 2024-09-20T22:55:14.160 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:55:14.160 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:55:14.160 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:55:14.160 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:55:14.161 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.161 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:55:14.161 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:26.433797+0000", 2024-09-20T22:55:14.161 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:55:14.161 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.161 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.161 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:55:14.161 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:26.433797+0000", 2024-09-20T22:55:14.161 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:55:14.161 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.161 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.161 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:55:14.161 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:26.433797+0000", 2024-09-20T22:55:14.161 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.29e-07 2024-09-20T22:55:14.162 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.162 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.162 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:55:14.162 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:26.460482+0000", 2024-09-20T22:55:14.162 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.026684802000000001 2024-09-20T22:55:14.162 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.162 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.162 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:55:14.162 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:26.460502+0000", 2024-09-20T22:55:14.162 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9828999999999999e-05 2024-09-20T22:55:14.162 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.162 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.162 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:55:14.162 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:26.460528+0000", 2024-09-20T22:55:14.162 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.5709999999999999e-05 2024-09-20T22:55:14.163 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.163 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.163 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:55:14.163 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:26.460592+0000", 2024-09-20T22:55:14.163 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.4706e-05 2024-09-20T22:55:14.163 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.163 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.163 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:55:14.163 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:26.460615+0000", 2024-09-20T22:55:14.163 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.2297e-05 2024-09-20T22:55:14.163 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.163 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.163 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:55:14.163 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:26.463172+0000", 2024-09-20T22:55:14.164 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.002557541 2024-09-20T22:55:14.164 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.164 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.164 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:55:14.164 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:26.463215+0000", 2024-09-20T22:55:14.164 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.2407000000000002e-05 2024-09-20T22:55:14.164 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:55:14.164 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:55:14.164 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:55:14.164 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.164 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.164 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:6356 7.c 7:3f0fe232:::smithi08477203-4:head [copy-from ver 1803] snapc 119=[119,118,117,116,115,114,113,112,111,110,10e,10c,107,105,104,103,fd,fb,fa,f8,f5,f2,ed,d7,c2,bf] ondisk+write+known_if_redirected+supports_pool_eio e638)", 2024-09-20T22:55:14.164 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:51:58.274309+0000", 2024-09-20T22:55:14.164 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 195.83961741100001, 2024-09-20T22:55:14.165 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.038014514999999999, 2024-09-20T22:55:14.165 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:55:14.165 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:55:14.165 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:55:14.165 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:55:14.165 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:55:14.165 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:55:14.165 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 6356 2024-09-20T22:55:14.165 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.165 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:55:14.165 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.165 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:55:14.165 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:58.274309+0000", 2024-09-20T22:55:14.165 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:55:14.166 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.166 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.166 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:55:14.166 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:58.274309+0000", 2024-09-20T22:55:14.166 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:55:14.166 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.166 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.166 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:55:14.166 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:58.274312+0000", 2024-09-20T22:55:14.166 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.9050000000000001e-06 2024-09-20T22:55:14.166 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.166 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.166 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:55:14.166 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:58.274324+0000", 2024-09-20T22:55:14.167 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1338999999999999e-05 2024-09-20T22:55:14.167 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.167 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.167 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:55:14.167 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:58.274348+0000", 2024-09-20T22:55:14.167 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.4876999999999999e-05 2024-09-20T22:55:14.167 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.167 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.167 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:55:14.167 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:58.274360+0000", 2024-09-20T22:55:14.167 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1301000000000001e-05 2024-09-20T22:55:14.167 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.167 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.167 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:55:14.168 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:58.274489+0000", 2024-09-20T22:55:14.168 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00012894299999999999 2024-09-20T22:55:14.168 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.168 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.168 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op must wait for map", 2024-09-20T22:55:14.168 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:58.274496+0000", 2024-09-20T22:55:14.168 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.401e-06 2024-09-20T22:55:14.168 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.168 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.168 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:55:14.168 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:58.277772+0000", 2024-09-20T22:55:14.168 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0032760409999999999 2024-09-20T22:55:14.168 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.168 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.168 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:55:14.169 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:58.277882+0000", 2024-09-20T22:55:14.169 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000109598 2024-09-20T22:55:14.169 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.169 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.169 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 3,7", 2024-09-20T22:55:14.169 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:58.302919+0000", 2024-09-20T22:55:14.169 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.025037197000000001 2024-09-20T22:55:14.169 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.169 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.169 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:55:14.169 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:58.311526+0000", 2024-09-20T22:55:14.169 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0086071370000000008 2024-09-20T22:55:14.169 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.169 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.170 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:55:14.170 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:58.311824+0000", 2024-09-20T22:55:14.170 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00029813600000000001 2024-09-20T22:55:14.170 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.170 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.170 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:55:14.170 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:58.312234+0000", 2024-09-20T22:55:14.170 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00040939699999999998 2024-09-20T22:55:14.170 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.170 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.170 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:55:14.170 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:58.312270+0000", 2024-09-20T22:55:14.170 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.6797e-05 2024-09-20T22:55:14.170 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.171 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.171 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:55:14.171 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:58.312324+0000", 2024-09-20T22:55:14.171 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.3446000000000003e-05 2024-09-20T22:55:14.171 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:55:14.171 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:55:14.171 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:55:14.171 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.171 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.171 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:6357 7.c 7:3f0fe232:::smithi08477203-4:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e638)", 2024-09-20T22:55:14.171 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:51:58.274400+0000", 2024-09-20T22:55:14.171 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 195.839526638, 2024-09-20T22:55:14.171 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.033818943999999997, 2024-09-20T22:55:14.171 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:55:14.171 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:55:14.172 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:55:14.172 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:55:14.172 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:55:14.172 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:55:14.172 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 6357 2024-09-20T22:55:14.172 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.172 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:55:14.172 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.172 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:55:14.172 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:58.274400+0000", 2024-09-20T22:55:14.172 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:55:14.172 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.172 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.172 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:55:14.173 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:58.274400+0000", 2024-09-20T22:55:14.173 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:55:14.173 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.173 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.173 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:55:14.173 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:58.274401+0000", 2024-09-20T22:55:14.173 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.4799999999999997e-07 2024-09-20T22:55:14.173 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.173 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.173 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:55:14.173 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:58.274405+0000", 2024-09-20T22:55:14.173 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.8310000000000003e-06 2024-09-20T22:55:14.173 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.173 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.173 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:55:14.174 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:58.274414+0000", 2024-09-20T22:55:14.174 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.0790000000000007e-06 2024-09-20T22:55:14.174 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.174 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.174 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:55:14.174 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:58.274420+0000", 2024-09-20T22:55:14.174 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.6059999999999999e-06 2024-09-20T22:55:14.174 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.174 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.174 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:55:14.174 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:58.274557+0000", 2024-09-20T22:55:14.174 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00013666700000000001 2024-09-20T22:55:14.174 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.175 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.175 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting_for_map not empty", 2024-09-20T22:55:14.175 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:58.274562+0000", 2024-09-20T22:55:14.175 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.4079999999999997e-06 2024-09-20T22:55:14.175 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.175 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.175 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:55:14.175 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:58.278047+0000", 2024-09-20T22:55:14.175 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.003484647 2024-09-20T22:55:14.175 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.175 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.175 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for blocked object", 2024-09-20T22:55:14.175 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:58.278096+0000", 2024-09-20T22:55:14.176 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.9298000000000001e-05 2024-09-20T22:55:14.176 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.176 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.176 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:55:14.176 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:58.308056+0000", 2024-09-20T22:55:14.176 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.029959835000000001 2024-09-20T22:55:14.176 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.176 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.176 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:55:14.176 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:58.308131+0000", 2024-09-20T22:55:14.176 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.5168999999999998e-05 2024-09-20T22:55:14.176 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.176 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.176 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:55:14.177 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:58.308219+0000", 2024-09-20T22:55:14.177 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.7656000000000002e-05 2024-09-20T22:55:14.177 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:55:14.177 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:55:14.177 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:55:14.177 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.177 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.177 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:6396 7.c 7:318c7512:::smithi08477203-44:head [delete] snapc 11d=[11d,11c,11b,11a,119,118,117,116,115,114,113,112,111,110,10e,10c,107,105,103,fd,fb,fa,f8,f5,f2,ed,d7,c2,bf] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e643)", 2024-09-20T22:55:14.177 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:52:03.276860+0000", 2024-09-20T22:55:14.177 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 190.83706712200001, 2024-09-20T22:55:14.177 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.029783397, 2024-09-20T22:55:14.177 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:55:14.177 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:55:14.177 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:55:14.178 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:55:14.178 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:55:14.178 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:55:14.178 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 6396 2024-09-20T22:55:14.178 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.178 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:55:14.178 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.178 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:55:14.178 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:03.276860+0000", 2024-09-20T22:55:14.178 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:55:14.178 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.178 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.178 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:55:14.178 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:03.276860+0000", 2024-09-20T22:55:14.179 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:55:14.179 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.179 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.179 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:55:14.179 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:03.276860+0000", 2024-09-20T22:55:14.179 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.5600000000000001e-07 2024-09-20T22:55:14.179 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.179 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.179 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:55:14.179 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:03.276862+0000", 2024-09-20T22:55:14.179 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.3489999999999999e-06 2024-09-20T22:55:14.179 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.179 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.179 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:55:14.180 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:03.276869+0000", 2024-09-20T22:55:14.180 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.3749999999999999e-06 2024-09-20T22:55:14.180 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.180 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.180 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:55:14.180 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:03.276874+0000", 2024-09-20T22:55:14.180 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.463e-06 2024-09-20T22:55:14.180 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.180 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.180 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:55:14.180 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:03.276944+0000", 2024-09-20T22:55:14.180 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.9689000000000003e-05 2024-09-20T22:55:14.180 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.180 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.180 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op must wait for map", 2024-09-20T22:55:14.181 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:03.276950+0000", 2024-09-20T22:55:14.181 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.5300000000000004e-06 2024-09-20T22:55:14.181 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.181 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.181 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:55:14.181 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:03.282451+0000", 2024-09-20T22:55:14.181 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0055015079999999996 2024-09-20T22:55:14.181 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.181 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.181 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:55:14.181 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:03.282562+0000", 2024-09-20T22:55:14.181 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000110517 2024-09-20T22:55:14.181 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.181 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.181 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:55:14.182 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:03.306643+0000", 2024-09-20T22:55:14.182 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.02408151 2024-09-20T22:55:14.182 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:55:14.182 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:55:14.182 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:55:14.182 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.182 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.182 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:6840 7.7 7:ed54065f:::smithi08477203-12:head [copy-from ver 788] snapc 136=[136,135,134,132,131,130,12f,12e,12d,12c,12b,12a,128,127,126,124,122,121,11f,11b,119,112,10e,10c,107,103,fb,fa,f8,ed,d7,c2,bf] ondisk+write+known_if_redirected+supports_pool_eio e689)", 2024-09-20T22:55:14.182 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:52:49.294097+0000", 2024-09-20T22:55:14.182 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 144.81982935900001, 2024-09-20T22:55:14.182 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.029501171, 2024-09-20T22:55:14.182 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:55:14.182 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:55:14.182 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:55:14.183 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:55:14.183 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:55:14.183 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:55:14.183 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 6840 2024-09-20T22:55:14.183 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.183 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:55:14.183 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.183 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:55:14.183 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:49.294097+0000", 2024-09-20T22:55:14.183 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:55:14.183 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.183 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.183 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:55:14.183 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:49.294097+0000", 2024-09-20T22:55:14.184 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:55:14.184 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.184 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.184 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:55:14.184 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:49.294099+0000", 2024-09-20T22:55:14.184 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.604e-06 2024-09-20T22:55:14.184 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.184 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.184 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:55:14.184 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:49.294106+0000", 2024-09-20T22:55:14.184 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.1980000000000004e-06 2024-09-20T22:55:14.184 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.185 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.185 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:55:14.185 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:49.294123+0000", 2024-09-20T22:55:14.185 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.6626e-05 2024-09-20T22:55:14.185 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.186 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.186 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:55:14.186 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:49.294131+0000", 2024-09-20T22:55:14.186 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.0309999999999998e-06 2024-09-20T22:55:14.187 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-20T22:55:14.188 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T22:55:14.188 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.188 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.188 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:55:14.188 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:49.294308+0000", 2024-09-20T22:55:14.188 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00017662900000000001 2024-09-20T22:55:14.189 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.189 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.189 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:55:14.189 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:49.294462+0000", 2024-09-20T22:55:14.189 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00015426400000000001 2024-09-20T22:55:14.189 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.189 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.189 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 0,3", 2024-09-20T22:55:14.189 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:49.308555+0000", 2024-09-20T22:55:14.189 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.014093207999999999 2024-09-20T22:55:14.189 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.189 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.189 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:55:14.189 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:49.314307+0000", 2024-09-20T22:55:14.190 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0057518789999999997 2024-09-20T22:55:14.190 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.190 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.190 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:55:14.190 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:49.323332+0000", 2024-09-20T22:55:14.190 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0090252010000000001 2024-09-20T22:55:14.190 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.190 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.190 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:55:14.190 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:49.323500+0000", 2024-09-20T22:55:14.190 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00016768100000000001 2024-09-20T22:55:14.190 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.190 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.190 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:55:14.191 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:49.323551+0000", 2024-09-20T22:55:14.191 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.0991e-05 2024-09-20T22:55:14.191 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.191 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.191 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:55:14.191 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:49.323599+0000", 2024-09-20T22:55:14.191 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.7858999999999997e-05 2024-09-20T22:55:14.191 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:55:14.191 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:55:14.191 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:55:14.191 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.191 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.191 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:7255 7.7 7:ed54065f:::smithi08477203-12:head [write 423807~566492,stat] snapc 149=[149,148,147,146,145,144,142,141,140,13e,13d,13c,13b,13a,139,138,136,135,132,130,12f,12e,12d,12c,12a,128,127,124,121,11f,112,103,ed] ondisk+write+known_if_redirected+supports_pool_eio e728)", 2024-09-20T22:55:14.191 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:53:28.303251+0000", 2024-09-20T22:55:14.192 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 105.810675538, 2024-09-20T22:55:14.192 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.039791188999999998, 2024-09-20T22:55:14.192 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:55:14.192 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:55:14.192 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:55:14.192 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:55:14.192 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:55:14.192 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:55:14.192 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 7255 2024-09-20T22:55:14.192 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.192 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:55:14.192 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.192 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:55:14.192 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:28.303251+0000", 2024-09-20T22:55:14.193 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:55:14.193 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.193 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.193 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:55:14.193 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:28.303251+0000", 2024-09-20T22:55:14.193 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:55:14.193 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.193 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.193 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:55:14.193 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:28.303253+0000", 2024-09-20T22:55:14.193 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4479999999999999e-06 2024-09-20T22:55:14.193 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.193 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.193 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:55:14.193 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:28.303922+0000", 2024-09-20T22:55:14.194 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00066900300000000005 2024-09-20T22:55:14.194 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.194 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.194 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:55:14.194 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:28.303935+0000", 2024-09-20T22:55:14.194 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3081000000000001e-05 2024-09-20T22:55:14.194 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.194 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.194 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:55:14.194 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:28.303942+0000", 2024-09-20T22:55:14.194 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.9369999999999998e-06 2024-09-20T22:55:14.194 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.194 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.194 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:55:14.195 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:28.303999+0000", 2024-09-20T22:55:14.195 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.7717999999999999e-05 2024-09-20T22:55:14.195 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.195 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.195 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op must wait for map", 2024-09-20T22:55:14.195 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:28.304006+0000", 2024-09-20T22:55:14.195 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.6150000000000002e-06 2024-09-20T22:55:14.195 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.195 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.195 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:55:14.195 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:28.307027+0000", 2024-09-20T22:55:14.195 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.003020474 2024-09-20T22:55:14.195 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.196 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.196 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:55:14.196 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:28.307136+0000", 2024-09-20T22:55:14.196 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000109522 2024-09-20T22:55:14.196 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.196 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.196 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 0,3", 2024-09-20T22:55:14.196 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:28.307288+0000", 2024-09-20T22:55:14.196 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00015164099999999999 2024-09-20T22:55:14.196 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.196 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.196 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:55:14.196 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:28.308664+0000", 2024-09-20T22:55:14.196 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001375923 2024-09-20T22:55:14.196 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.197 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.197 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:55:14.197 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:28.314469+0000", 2024-09-20T22:55:14.197 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0058058850000000002 2024-09-20T22:55:14.197 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.197 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.197 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:55:14.197 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:28.342877+0000", 2024-09-20T22:55:14.197 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.028407874999999999 2024-09-20T22:55:14.197 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.197 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.197 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:55:14.197 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:28.342967+0000", 2024-09-20T22:55:14.197 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.9480000000000004e-05 2024-09-20T22:55:14.198 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.198 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.198 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:55:14.198 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:28.343042+0000", 2024-09-20T22:55:14.198 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.5587000000000007e-05 2024-09-20T22:55:14.198 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:55:14.198 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:55:14.198 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:55:14.198 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.198 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.198 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:7256 7.7 7:ed54065f:::smithi08477203-12:head [write 1676240~407685] snapc 149=[149,148,147,146,145,144,142,141,140,13e,13d,13c,13b,13a,139,138,136,135,132,130,12f,12e,12d,12c,12a,128,127,124,121,11f,112,103,ed] ondisk+write+known_if_redirected+supports_pool_eio e728)", 2024-09-20T22:55:14.198 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:53:28.310420+0000", 2024-09-20T22:55:14.198 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 105.803506282, 2024-09-20T22:55:14.198 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.032753075, 2024-09-20T22:55:14.199 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:55:14.199 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:55:14.199 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:55:14.199 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:55:14.199 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:55:14.199 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:55:14.199 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 7256 2024-09-20T22:55:14.199 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.199 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:55:14.199 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.199 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:55:14.199 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:28.310420+0000", 2024-09-20T22:55:14.199 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:55:14.199 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.200 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.200 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:55:14.200 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:28.310420+0000", 2024-09-20T22:55:14.200 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:55:14.200 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.200 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.200 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:55:14.200 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:28.310423+0000", 2024-09-20T22:55:14.200 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.6060000000000001e-06 2024-09-20T22:55:14.200 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.200 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.200 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:55:14.200 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:28.310795+0000", 2024-09-20T22:55:14.200 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000371849 2024-09-20T22:55:14.201 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.201 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.201 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:55:14.201 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:28.310832+0000", 2024-09-20T22:55:14.201 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.6704e-05 2024-09-20T22:55:14.201 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.201 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.201 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:55:14.201 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:28.310846+0000", 2024-09-20T22:55:14.201 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4015999999999999e-05 2024-09-20T22:55:14.201 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.201 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.201 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:55:14.202 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:28.310916+0000", 2024-09-20T22:55:14.202 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.0037999999999996e-05 2024-09-20T22:55:14.202 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.202 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.202 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:55:14.202 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:28.311004+0000", 2024-09-20T22:55:14.202 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.8494999999999999e-05 2024-09-20T22:55:14.202 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.202 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.202 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 0,3", 2024-09-20T22:55:14.202 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:28.311132+0000", 2024-09-20T22:55:14.202 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000128219 2024-09-20T22:55:14.202 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.202 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.202 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:55:14.203 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:28.312266+0000", 2024-09-20T22:55:14.203 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001133081 2024-09-20T22:55:14.203 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.203 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.203 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:55:14.203 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:28.317154+0000", 2024-09-20T22:55:14.203 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0048885040000000001 2024-09-20T22:55:14.203 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.203 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.203 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:55:14.203 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:28.343095+0000", 2024-09-20T22:55:14.203 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.025940595 2024-09-20T22:55:14.203 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.204 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.204 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:55:14.204 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:28.343137+0000", 2024-09-20T22:55:14.204 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.2815000000000003e-05 2024-09-20T22:55:14.204 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.204 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.204 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:55:14.204 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:28.343174+0000", 2024-09-20T22:55:14.204 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.6152999999999998e-05 2024-09-20T22:55:14.204 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:55:14.204 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:55:14.204 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:55:14.204 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.204 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.204 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:7259 7.7 7:ed54065f:::smithi08477203-12:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e728)", 2024-09-20T22:55:14.205 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:53:28.314947+0000", 2024-09-20T22:55:14.205 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 105.798979797, 2024-09-20T22:55:14.205 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.028624334000000001, 2024-09-20T22:55:14.205 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:55:14.205 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:55:14.205 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:55:14.205 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:55:14.205 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:55:14.205 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:55:14.205 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 7259 2024-09-20T22:55:14.205 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.205 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:55:14.205 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.205 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:55:14.206 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:28.314947+0000", 2024-09-20T22:55:14.206 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:55:14.206 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.206 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.206 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:55:14.206 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:28.314947+0000", 2024-09-20T22:55:14.206 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:55:14.206 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.206 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.206 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:55:14.206 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:28.314947+0000", 2024-09-20T22:55:14.206 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.8099999999999999e-07 2024-09-20T22:55:14.206 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.206 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.207 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:55:14.207 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:28.314949+0000", 2024-09-20T22:55:14.207 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.689e-06 2024-09-20T22:55:14.207 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.207 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.207 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:55:14.207 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:28.314954+0000", 2024-09-20T22:55:14.207 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.1739999999999999e-06 2024-09-20T22:55:14.207 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.207 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.207 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:55:14.207 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:28.314958+0000", 2024-09-20T22:55:14.207 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.0570000000000003e-06 2024-09-20T22:55:14.207 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.208 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.208 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:55:14.208 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:28.317268+0000", 2024-09-20T22:55:14.208 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0023099230000000002 2024-09-20T22:55:14.208 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.208 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.208 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for rw locks", 2024-09-20T22:55:14.208 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:28.317387+0000", 2024-09-20T22:55:14.208 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000118591 2024-09-20T22:55:14.208 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.208 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.208 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:55:14.208 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:28.343433+0000", 2024-09-20T22:55:14.208 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.026046430999999998 2024-09-20T22:55:14.208 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.209 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.209 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:55:14.209 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:28.343493+0000", 2024-09-20T22:55:14.209 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.0262e-05 2024-09-20T22:55:14.209 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.209 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.209 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:55:14.209 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:28.343571+0000", 2024-09-20T22:55:14.209 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.7925999999999999e-05 2024-09-20T22:55:14.209 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:55:14.209 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:55:14.209 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:55:14.209 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.209 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.210 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:7438 7.7 7:ed54065f:::smithi08477203-12:head [append 0~0] snapc 14e=[14e,14c,14a,149,148,145,142,141,13e,13d,13c,13b,13a,139,132,130,12f,12e,12c,12a,128,127,124,121,11f,112,103] ondisk+write+known_if_redirected+supports_pool_eio e744)", 2024-09-20T22:55:14.210 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:53:44.435340+0000", 2024-09-20T22:55:14.210 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 89.678586796999994, 2024-09-20T22:55:14.210 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.037994317999999999, 2024-09-20T22:55:14.210 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:55:14.210 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:55:14.210 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:55:14.210 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:55:14.210 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:55:14.210 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:55:14.210 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 7438 2024-09-20T22:55:14.210 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.210 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:55:14.210 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.211 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:55:14.211 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:44.435340+0000", 2024-09-20T22:55:14.211 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:55:14.211 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.211 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.211 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:55:14.211 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:44.435340+0000", 2024-09-20T22:55:14.211 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:55:14.211 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.211 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.211 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:55:14.211 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:44.435340+0000", 2024-09-20T22:55:14.211 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.5100000000000001e-07 2024-09-20T22:55:14.211 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.212 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.212 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:55:14.212 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:44.435342+0000", 2024-09-20T22:55:14.212 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.7349999999999999e-06 2024-09-20T22:55:14.212 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.212 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.212 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:55:14.212 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:44.435350+0000", 2024-09-20T22:55:14.212 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.0560000000000005e-06 2024-09-20T22:55:14.212 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.212 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.212 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:55:14.212 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:44.435354+0000", 2024-09-20T22:55:14.212 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.0069999999999999e-06 2024-09-20T22:55:14.213 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.213 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.213 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:55:14.213 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:44.437843+0000", 2024-09-20T22:55:14.213 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.002489018 2024-09-20T22:55:14.213 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.213 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.213 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:55:14.213 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:44.437924+0000", 2024-09-20T22:55:14.213 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.0477999999999998e-05 2024-09-20T22:55:14.213 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.213 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.213 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 0,3", 2024-09-20T22:55:14.213 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:44.438050+0000", 2024-09-20T22:55:14.213 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00012694599999999999 2024-09-20T22:55:14.214 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.214 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.214 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:55:14.214 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:44.438583+0000", 2024-09-20T22:55:14.214 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00053302799999999995 2024-09-20T22:55:14.214 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.214 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.214 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:55:14.214 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:44.439219+0000", 2024-09-20T22:55:14.214 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00063534599999999996 2024-09-20T22:55:14.214 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.214 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.214 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:55:14.214 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:44.473187+0000", 2024-09-20T22:55:14.215 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.033968241000000003 2024-09-20T22:55:14.215 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.215 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.215 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:55:14.215 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:44.473269+0000", 2024-09-20T22:55:14.215 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.2150999999999993e-05 2024-09-20T22:55:14.215 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.215 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.215 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:55:14.215 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:44.473334+0000", 2024-09-20T22:55:14.215 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.5061000000000006e-05 2024-09-20T22:55:14.215 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:55:14.215 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:55:14.215 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:55:14.216 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.216 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.216 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:7834 7.c 7:3f0fe232:::smithi08477203-4:head [copy-from ver 1835] snapc 165=[165,164,162,161,160,15f,15e,15d,15c,15b,15a,158,157,156,155,153,14f,14e,14c,14a,149,148,145,142,13e,13a,139,12f,12e,124,112] ondisk+write+known_if_redirected+supports_pool_eio e786)", 2024-09-20T22:55:14.216 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:54:26.316663+0000", 2024-09-20T22:55:14.216 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 47.797264259999999, 2024-09-20T22:55:14.216 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.026458226000000001, 2024-09-20T22:55:14.216 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:55:14.216 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:55:14.216 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:55:14.216 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:55:14.216 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:55:14.216 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:55:14.216 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 7834 2024-09-20T22:55:14.216 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.217 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:55:14.217 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.217 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:55:14.217 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:54:26.316663+0000", 2024-09-20T22:55:14.217 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:55:14.217 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.217 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.217 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:55:14.217 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:54:26.316663+0000", 2024-09-20T22:55:14.217 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:55:14.217 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.217 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.217 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:55:14.217 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:54:26.316665+0000", 2024-09-20T22:55:14.218 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.6290000000000002e-06 2024-09-20T22:55:14.218 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.218 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.218 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:55:14.218 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:54:26.316678+0000", 2024-09-20T22:55:14.218 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2472e-05 2024-09-20T22:55:14.218 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.218 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.218 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:55:14.218 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:54:26.316704+0000", 2024-09-20T22:55:14.218 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.6455e-05 2024-09-20T22:55:14.218 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.218 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.218 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:55:14.219 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:54:26.316717+0000", 2024-09-20T22:55:14.219 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2867999999999999e-05 2024-09-20T22:55:14.219 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.219 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.219 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:55:14.219 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:54:26.316857+0000", 2024-09-20T22:55:14.219 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00013957999999999999 2024-09-20T22:55:14.219 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.219 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.219 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op must wait for map", 2024-09-20T22:55:14.219 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:54:26.316866+0000", 2024-09-20T22:55:14.219 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.0929999999999992e-06 2024-09-20T22:55:14.219 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.219 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.219 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:55:14.220 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:54:26.319639+0000", 2024-09-20T22:55:14.220 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.002773298 2024-09-20T22:55:14.220 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.220 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.220 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:55:14.220 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:54:26.319790+0000", 2024-09-20T22:55:14.220 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00015067100000000001 2024-09-20T22:55:14.220 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.220 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.220 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 3,7", 2024-09-20T22:55:14.220 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:54:26.332633+0000", 2024-09-20T22:55:14.220 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.012843795 2024-09-20T22:55:14.220 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.220 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.220 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:55:14.221 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:54:26.342061+0000", 2024-09-20T22:55:14.221 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0094278690000000002 2024-09-20T22:55:14.221 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.221 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.221 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:55:14.221 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:54:26.342345+0000", 2024-09-20T22:55:14.221 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000283572 2024-09-20T22:55:14.221 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.221 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.221 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:55:14.221 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:54:26.343030+0000", 2024-09-20T22:55:14.221 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00068502099999999996 2024-09-20T22:55:14.221 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.221 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.222 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:55:14.222 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:54:26.343082+0000", 2024-09-20T22:55:14.222 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.2373999999999997e-05 2024-09-20T22:55:14.222 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.222 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.222 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:55:14.222 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:54:26.343121+0000", 2024-09-20T22:55:14.222 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.8529000000000003e-05 2024-09-20T22:55:14.222 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:55:14.222 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:55:14.222 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:55:14.222 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.222 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.222 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:7835 7.c 7:3f0fe232:::smithi08477203-4:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e786)", 2024-09-20T22:55:14.223 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:54:26.316754+0000", 2024-09-20T22:55:14.223 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 47.797173010999998, 2024-09-20T22:55:14.223 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.021899195, 2024-09-20T22:55:14.223 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:55:14.223 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:55:14.223 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:55:14.223 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:55:14.223 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:55:14.223 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:55:14.223 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 7835 2024-09-20T22:55:14.223 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.223 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:55:14.223 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.223 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:55:14.223 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:54:26.316754+0000", 2024-09-20T22:55:14.224 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:55:14.224 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.224 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.224 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:55:14.224 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:54:26.316754+0000", 2024-09-20T22:55:14.224 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:55:14.224 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.224 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.224 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:55:14.224 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:54:26.316755+0000", 2024-09-20T22:55:14.224 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.6400000000000001e-07 2024-09-20T22:55:14.224 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.224 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.224 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:55:14.225 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:54:26.316759+0000", 2024-09-20T22:55:14.225 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.0559999999999998e-06 2024-09-20T22:55:14.225 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.225 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.225 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:55:14.225 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:54:26.316768+0000", 2024-09-20T22:55:14.225 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.9400000000000008e-06 2024-09-20T22:55:14.225 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.225 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.225 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:55:14.225 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:54:26.316775+0000", 2024-09-20T22:55:14.225 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.4449999999999999e-06 2024-09-20T22:55:14.225 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.225 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.225 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:55:14.226 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:54:26.316924+0000", 2024-09-20T22:55:14.226 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000149507 2024-09-20T22:55:14.226 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.226 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.226 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting_for_map not empty", 2024-09-20T22:55:14.226 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:54:26.316931+0000", 2024-09-20T22:55:14.226 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.8979999999999999e-06 2024-09-20T22:55:14.226 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.226 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.226 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:55:14.226 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:54:26.320015+0000", 2024-09-20T22:55:14.226 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0030839180000000002 2024-09-20T22:55:14.226 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.226 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.227 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for blocked object", 2024-09-20T22:55:14.227 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:54:26.320053+0000", 2024-09-20T22:55:14.227 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.8157000000000001e-05 2024-09-20T22:55:14.227 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.227 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.227 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:55:14.227 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:54:26.338445+0000", 2024-09-20T22:55:14.227 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.01839174 2024-09-20T22:55:14.227 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.227 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.227 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:55:14.227 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:54:26.338534+0000", 2024-09-20T22:55:14.227 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.8542999999999996e-05 2024-09-20T22:55:14.227 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.227 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.228 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:55:14.228 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:54:26.338653+0000", 2024-09-20T22:55:14.228 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000119227 2024-09-20T22:55:14.228 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:55:14.228 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:55:14.228 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:55:14.228 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.228 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.228 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:8112 7.7 7:ed54065f:::smithi08477203-12:head [rollback 16f] snapc 176=[176,175,174,173,172,171,16f,16e,16d,16c,16b,166,160,15e,15c,15a,158,157,156,155,153,14f,14e,14a,149,145,13e,13a,139,12f,12e] ondisk+write+known_if_redirected+supports_pool_eio e820)", 2024-09-20T22:55:14.228 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:55:00.331928+0000", 2024-09-20T22:55:14.228 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 13.781998494, 2024-09-20T22:55:14.228 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.021454840999999999, 2024-09-20T22:55:14.228 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:55:14.228 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:55:14.229 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:55:14.229 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:55:14.229 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:55:14.229 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:55:14.229 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 8112 2024-09-20T22:55:14.229 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.229 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:55:14.229 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.229 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:55:14.229 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:00.331928+0000", 2024-09-20T22:55:14.229 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:55:14.229 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.229 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.229 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:55:14.230 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:00.331928+0000", 2024-09-20T22:55:14.230 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:55:14.230 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.230 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.230 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:55:14.230 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:00.331930+0000", 2024-09-20T22:55:14.230 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.5910000000000001e-06 2024-09-20T22:55:14.230 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.230 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.230 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:55:14.230 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:00.331939+0000", 2024-09-20T22:55:14.230 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.234e-06 2024-09-20T22:55:14.230 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.231 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.231 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:55:14.231 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:00.331957+0000", 2024-09-20T22:55:14.231 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.7897000000000002e-05 2024-09-20T22:55:14.231 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.231 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.231 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:55:14.231 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:00.331968+0000", 2024-09-20T22:55:14.231 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1110999999999999e-05 2024-09-20T22:55:14.231 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.231 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.231 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:55:14.231 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:00.332081+0000", 2024-09-20T22:55:14.231 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000113331 2024-09-20T22:55:14.232 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.232 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.232 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op must wait for map", 2024-09-20T22:55:14.232 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:00.332089+0000", 2024-09-20T22:55:14.232 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.3309999999999998e-06 2024-09-20T22:55:14.232 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.232 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.232 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:55:14.232 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:00.334319+0000", 2024-09-20T22:55:14.232 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.002230227 2024-09-20T22:55:14.232 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.232 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.232 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:55:14.232 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:00.334437+0000", 2024-09-20T22:55:14.232 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000117506 2024-09-20T22:55:14.233 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.233 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.233 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 0,3", 2024-09-20T22:55:14.233 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:00.334708+0000", 2024-09-20T22:55:14.233 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000271597 2024-09-20T22:55:14.233 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.233 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.233 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:55:14.233 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:00.336741+0000", 2024-09-20T22:55:14.233 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0020328579999999998 2024-09-20T22:55:14.233 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.233 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.233 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:55:14.233 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:00.340010+0000", 2024-09-20T22:55:14.234 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0032693349999999999 2024-09-20T22:55:14.234 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.234 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.234 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:55:14.234 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:00.353302+0000", 2024-09-20T22:55:14.234 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.013291261 2024-09-20T22:55:14.234 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.234 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.234 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:55:14.234 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:00.353345+0000", 2024-09-20T22:55:14.234 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.3736999999999998e-05 2024-09-20T22:55:14.234 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.234 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.234 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:55:14.235 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:00.353383+0000", 2024-09-20T22:55:14.235 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.7824999999999998e-05 2024-09-20T22:55:14.235 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:55:14.235 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:55:14.235 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:55:14.235 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.235 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.235 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:8113 7.7 7:ed54065f:::smithi08477203-12:head [append 0~0] snapc 176=[176,175,174,173,172,171,16f,16e,16d,16c,16b,166,160,15e,15c,15a,158,157,156,155,153,14f,14e,14a,149,145,13e,13a,139,12f,12e] ondisk+write+known_if_redirected+supports_pool_eio e820)", 2024-09-20T22:55:14.235 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:55:00.331993+0000", 2024-09-20T22:55:14.235 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 13.781933701, 2024-09-20T22:55:14.235 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.021570592, 2024-09-20T22:55:14.235 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:55:14.235 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:55:14.235 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:55:14.236 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:55:14.236 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:55:14.236 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:55:14.236 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 8113 2024-09-20T22:55:14.236 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.236 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:55:14.236 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.236 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:55:14.236 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:00.331993+0000", 2024-09-20T22:55:14.236 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:55:14.236 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.236 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.236 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:55:14.236 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:00.331993+0000", 2024-09-20T22:55:14.236 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:55:14.237 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.237 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.237 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:55:14.237 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:00.331994+0000", 2024-09-20T22:55:14.237 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.4499999999999997e-07 2024-09-20T22:55:14.237 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.237 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.237 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:55:14.237 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:00.331997+0000", 2024-09-20T22:55:14.237 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.191e-06 2024-09-20T22:55:14.237 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.237 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.237 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:55:14.237 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:00.332008+0000", 2024-09-20T22:55:14.237 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1056999999999999e-05 2024-09-20T22:55:14.238 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.238 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.238 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:55:14.238 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:00.332014+0000", 2024-09-20T22:55:14.238 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.195e-06 2024-09-20T22:55:14.238 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.238 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.238 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:55:14.238 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:00.332244+0000", 2024-09-20T22:55:14.238 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00023003499999999999 2024-09-20T22:55:14.238 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.238 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.238 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting_for_map not empty", 2024-09-20T22:55:14.238 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:00.332254+0000", 2024-09-20T22:55:14.239 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.4380000000000001e-06 2024-09-20T22:55:14.239 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.239 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.239 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:55:14.239 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:00.335571+0000", 2024-09-20T22:55:14.239 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0033172660000000001 2024-09-20T22:55:14.239 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.239 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.239 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:55:14.239 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:00.335696+0000", 2024-09-20T22:55:14.239 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00012541599999999999 2024-09-20T22:55:14.239 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.239 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.239 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 0,3", 2024-09-20T22:55:14.240 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:00.335938+0000", 2024-09-20T22:55:14.240 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00024170800000000001 2024-09-20T22:55:14.240 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.240 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.240 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:55:14.240 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:00.336891+0000", 2024-09-20T22:55:14.240 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00095290900000000005 2024-09-20T22:55:14.240 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.240 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.240 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:55:14.240 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:00.340132+0000", 2024-09-20T22:55:14.240 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0032409420000000001 2024-09-20T22:55:14.240 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.240 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.241 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:55:14.241 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:00.353433+0000", 2024-09-20T22:55:14.241 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.013301445 2024-09-20T22:55:14.241 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.241 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.241 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:55:14.241 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:00.353515+0000", 2024-09-20T22:55:14.241 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.1452999999999994e-05 2024-09-20T22:55:14.241 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:14.241 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:14.241 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:55:14.241 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:00.353564+0000", 2024-09-20T22:55:14.241 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.8992000000000001e-05 2024-09-20T22:55:14.241 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:55:14.242 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:55:14.242 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:55:14.242 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:55:14.242 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:55:14.242 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-20T22:55:14.242 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:55:14.242 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:55:14.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:14.243 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:14.243 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:55:14.244 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:55:14.244 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:55:14.244 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:55:14.244 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:55:14.245 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:55:14.245 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:55:14.327 INFO:tasks.rados.rados.0.smithi084.stdout:3334: done (1 left) 2024-09-20T22:55:14.327 INFO:tasks.rados.rados.0.smithi084.stdout:3335: read oid 18 snap -1 2024-09-20T22:55:14.327 INFO:tasks.rados.rados.0.smithi084.stdout:3335: expect (ObjNum 1051 snap 345 seq_num 1051) 2024-09-20T22:55:14.327 INFO:tasks.rados.rados.0.smithi084.stdout:3336: read oid 43 snap -1 2024-09-20T22:55:14.327 INFO:tasks.rados.rados.0.smithi084.stdout:3336: expect deleted 2024-09-20T22:55:14.327 INFO:tasks.rados.rados.0.smithi084.stdout:3337: copy_from oid 36 from oid 8 current snap is 382 2024-09-20T22:55:14.327 INFO:tasks.rados.rados.0.smithi084.stdout:3338: write (excl) oid 24 current snap is 382 2024-09-20T22:55:14.327 INFO:tasks.rados.rados.0.smithi084.stdout:3338: seq_num 1136 ranges {692752=419461,1826015=615401,2979461=113291} 2024-09-20T22:55:14.333 INFO:tasks.rados.rados.0.smithi084.stdout:3338: writing smithi08477203-24 from 692752 to 1112213 tid 1 2024-09-20T22:55:14.342 INFO:tasks.rados.rados.0.smithi084.stdout:3338: writing smithi08477203-24 from 1826015 to 2441416 tid 2 2024-09-20T22:55:14.344 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:55:14.346 INFO:tasks.rados.rados.0.smithi084.stdout:3338: writing smithi08477203-24 from 2979461 to 3092752 tid 3 2024-09-20T22:55:14.347 INFO:tasks.rados.rados.0.smithi084.stdout:3339: snap_create 2024-09-20T22:55:14.347 INFO:tasks.rados.rados.0.smithi084.stdout:3333: finishing write tid 1 to smithi08477203-23 2024-09-20T22:55:14.347 INFO:tasks.rados.rados.0.smithi084.stdout:3333: finishing write tid 2 to smithi08477203-23 2024-09-20T22:55:14.347 INFO:tasks.rados.rados.0.smithi084.stdout:3333: finishing write tid 3 to smithi08477203-23 2024-09-20T22:55:14.347 INFO:tasks.rados.rados.0.smithi084.stdout:3333: finishing write tid 4 to smithi08477203-23 2024-09-20T22:55:14.347 INFO:tasks.rados.rados.0.smithi084.stdout:3333: oid 23 updating version 0 to 1922 2024-09-20T22:55:14.347 INFO:tasks.rados.rados.0.smithi084.stdout:3333: oid 23 updating version 1922 to 1923 2024-09-20T22:55:14.347 INFO:tasks.rados.rados.0.smithi084.stdout:3333: oid 23 version 1923 is already newer than 1921 2024-09-20T22:55:14.347 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 23 v 1923 (ObjNum 1135 snap 382 seq_num 1135) dirty exists 2024-09-20T22:55:14.347 INFO:tasks.rados.rados.0.smithi084.stdout:3333: left oid 23 (ObjNum 1135 snap 382 seq_num 1135) 2024-09-20T22:55:14.347 INFO:tasks.rados.rados.0.smithi084.stdout:3337: finishing copy_from to smithi08477203-36 2024-09-20T22:55:14.347 INFO:tasks.rados.rados.0.smithi084.stdout:3337: got expected ENOENT (src dne) 2024-09-20T22:55:14.347 INFO:tasks.rados.rados.0.smithi084.stdout:3337: finishing copy_from racing read to smithi08477203-36 2024-09-20T22:55:14.347 INFO:tasks.rados.rados.0.smithi084.stdout:3335: expect (ObjNum 1051 snap 345 seq_num 1051) 2024-09-20T22:55:14.366 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:14.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:14.367 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:55:14.464 INFO:tasks.rados.rados.0.smithi084.stdout:3338: finishing write tid 1 to smithi08477203-24 2024-09-20T22:55:14.464 INFO:tasks.rados.rados.0.smithi084.stdout:3338: finishing write tid 2 to smithi08477203-24 2024-09-20T22:55:14.464 INFO:tasks.rados.rados.0.smithi084.stdout:3338: finishing write tid 3 to smithi08477203-24 2024-09-20T22:55:14.464 INFO:tasks.rados.rados.0.smithi084.stdout:3338: finishing write tid 4 to smithi08477203-24 2024-09-20T22:55:14.464 INFO:tasks.rados.rados.0.smithi084.stdout:3338: finishing write tid 5 to smithi08477203-24 2024-09-20T22:55:14.464 INFO:tasks.rados.rados.0.smithi084.stdout:3338: oid 24 updating version 0 to 1895 2024-09-20T22:55:14.464 INFO:tasks.rados.rados.0.smithi084.stdout:3338: oid 24 version 1895 is already newer than 1894 2024-09-20T22:55:14.465 INFO:tasks.rados.rados.0.smithi084.stdout:3338: oid 24 version 1895 is already newer than 1892 2024-09-20T22:55:14.465 INFO:tasks.rados.rados.0.smithi084.stdout:3338: oid 24 version 1895 is already newer than 1893 2024-09-20T22:55:14.465 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 24 v 1895 (ObjNum 1136 snap 382 seq_num 1136) dirty exists 2024-09-20T22:55:14.465 INFO:tasks.rados.rados.0.smithi084.stdout:3338: left oid 24 (ObjNum 1136 snap 382 seq_num 1136) 2024-09-20T22:55:14.467 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:55:14.489 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:14.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:14.489 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:55:14.590 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:55:14.611 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:14.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:14.612 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:55:14.712 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:55:14.733 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:14.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:14.734 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:55:14.834 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:55:14.856 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:14.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:14.857 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:55:14.900 INFO:tasks.thrashosds.thrasher:in_osds: [2, 3, 5, 6, 7, 0, 4, 1] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-20T22:55:14.900 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-20T22:55:14.900 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-20T22:55:14.900 INFO:tasks.thrashosds.thrasher:Setting osd 4 primary_affinity to 0.000000 2024-09-20T22:55:14.901 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd primary-affinity 4 0 2024-09-20T22:55:14.957 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:55:14.978 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:14.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:14.979 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:55:15.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:14 smithi154 ceph-mon[33264]: pgmap v1152: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 550 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 10 MiB/s rd, 2.9 MiB/s wr, 21 op/s; 116 B/s, 4 objects/s recovering 2024-09-20T22:55:15.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:14 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/2838106062' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:55:15.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:14 smithi154 ceph-mon[33264]: osdmap e834: 8 total, 8 up, 8 in 2024-09-20T22:55:15.011 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-20T22:55:15.041 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:14 smithi084 ceph-mon[29774]: pgmap v1152: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 550 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 10 MiB/s rd, 2.9 MiB/s wr, 21 op/s; 116 B/s, 4 objects/s recovering 2024-09-20T22:55:15.041 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:14 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/2838106062' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:55:15.041 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:14 smithi084 ceph-mon[29774]: osdmap e834: 8 total, 8 up, 8 in 2024-09-20T22:55:15.041 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:14 smithi084 ceph-mon[40024]: pgmap v1152: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 550 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 10 MiB/s rd, 2.9 MiB/s wr, 21 op/s; 116 B/s, 4 objects/s recovering 2024-09-20T22:55:15.041 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:14 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/2838106062' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:55:15.041 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:14 smithi084 ceph-mon[40024]: osdmap e834: 8 total, 8 up, 8 in 2024-09-20T22:55:15.080 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:55:15.093 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-20T22:55:15.102 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:15.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:15.103 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:55:15.162 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:55:15.203 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:55:15.224 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:15.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:15.225 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:55:15.326 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:55:15.329 INFO:tasks.rados.rados.0.smithi084.stdout:3333: done (5 left) 2024-09-20T22:55:15.329 INFO:tasks.rados.rados.0.smithi084.stdout:3335: done (4 left) 2024-09-20T22:55:15.329 INFO:tasks.rados.rados.0.smithi084.stdout:3336: done (3 left) 2024-09-20T22:55:15.329 INFO:tasks.rados.rados.0.smithi084.stdout:3337: done (2 left) 2024-09-20T22:55:15.329 INFO:tasks.rados.rados.0.smithi084.stdout:3338: done (1 left) 2024-09-20T22:55:15.329 INFO:tasks.rados.rados.0.smithi084.stdout:3339: done (0 left) 2024-09-20T22:55:15.329 INFO:tasks.rados.rados.0.smithi084.stdout:3340: copy_from oid 44 from oid 25 current snap is 383 2024-09-20T22:55:15.329 INFO:tasks.rados.rados.0.smithi084.stdout:3341: delete oid 33 current snap is 383 2024-09-20T22:55:15.335 INFO:tasks.rados.rados.0.smithi084.stdout:3341: done (1 left) 2024-09-20T22:55:15.335 INFO:tasks.rados.rados.0.smithi084.stdout:3342: read oid 43 snap 380 2024-09-20T22:55:15.335 INFO:tasks.rados.rados.0.smithi084.stdout:3342: expect (ObjNum 1019 snap 333 seq_num 1019) 2024-09-20T22:55:15.335 INFO:tasks.rados.rados.0.smithi084.stdout:3343: rollback oid 30 current snap is 383 2024-09-20T22:55:15.335 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 30 to 382 2024-09-20T22:55:15.335 INFO:tasks.rados.rados.0.smithi084.stdout:3344: delete oid 21 current snap is 383 2024-09-20T22:55:15.337 INFO:tasks.rados.rados.0.smithi084.stdout:3344: done (3 left) 2024-09-20T22:55:15.337 INFO:tasks.rados.rados.0.smithi084.stdout:3345: snap_create 2024-09-20T22:55:15.340 INFO:tasks.rados.rados.0.smithi084.stdout:3343: finishing rollback tid 0 to smithi08477203-30 2024-09-20T22:55:15.341 INFO:tasks.rados.rados.0.smithi084.stdout:3342: expect (ObjNum 1019 snap 333 seq_num 1019) 2024-09-20T22:55:15.349 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:55:15.354 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:15.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:15.355 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:55:15.448 INFO:tasks.rados.rados.0.smithi084.stdout:3343: finishing rollback tid 1 to smithi08477203-30 2024-09-20T22:55:15.587 INFO:tasks.rados.rados.0.smithi084.stdout:3343: finishing rollback tid 2 to smithi08477203-30 2024-09-20T22:55:15.587 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 30 v 2559 (ObjNum 1110 snap 377 seq_num 1110) dirty exists 2024-09-20T22:55:15.587 INFO:tasks.rados.rados.0.smithi084.stdout:3340: finishing copy_from racing read to smithi08477203-44 2024-09-20T22:55:15.588 INFO:tasks.rados.rados.0.smithi084.stdout:3340: finishing copy_from to smithi08477203-44 2024-09-20T22:55:15.588 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 44 v 2542 (ObjNum 1070 snap 355 seq_num 1070) dirty exists 2024-09-20T22:55:15.588 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:55:15.611 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:15.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:15.612 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:55:15.699 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:55:15.713 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:55:15.734 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:15.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:15.735 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:55:15.814 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-20T22:55:15.814 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:55:15.836 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:55:15.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:15.858 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:15.858 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:55:15.959 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:55:15.980 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:15.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:15.981 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:55:16.082 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:55:16.103 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:16.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:16.104 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:55:16.205 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:55:16.226 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:16.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:16.227 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:55:16.327 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:55:16.334 INFO:teuthology.orchestra.run.smithi084.stderr:set osd.4 primary-affinity to 0 (802) 2024-09-20T22:55:16.337 INFO:tasks.rados.rados.0.smithi084.stdout:3340: done (3 left) 2024-09-20T22:55:16.337 INFO:tasks.rados.rados.0.smithi084.stdout:3342: done (2 left) 2024-09-20T22:55:16.337 INFO:tasks.rados.rados.0.smithi084.stdout:3343: done (1 left) 2024-09-20T22:55:16.337 INFO:tasks.rados.rados.0.smithi084.stdout:3345: done (0 left) 2024-09-20T22:55:16.337 INFO:tasks.rados.rados.0.smithi084.stdout:3346: snap_create 2024-09-20T22:55:16.357 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:16.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:16.358 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:55:16.459 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:55:16.480 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:16.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:16.481 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:55:16.523 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.7 dump_blocked_ops 2024-09-20T22:55:16.582 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:55:16.584 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:16 smithi154 ceph-mon[33264]: osdmap e835: 8 total, 8 up, 8 in 2024-09-20T22:55:16.584 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:16 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/4004898672' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.0}]: dispatch 2024-09-20T22:55:16.584 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:16 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.0}]: dispatch 2024-09-20T22:55:16.603 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:16.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:16.604 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:55:16.614 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:16 smithi084 ceph-mon[29774]: osdmap e835: 8 total, 8 up, 8 in 2024-09-20T22:55:16.614 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:16 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/4004898672' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.0}]: dispatch 2024-09-20T22:55:16.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:16 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.0}]: dispatch 2024-09-20T22:55:16.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:16 smithi084 ceph-mon[40024]: osdmap e835: 8 total, 8 up, 8 in 2024-09-20T22:55:16.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:16 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/4004898672' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.0}]: dispatch 2024-09-20T22:55:16.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:16 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.0}]: dispatch 2024-09-20T22:55:16.705 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:55:16.726 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:16.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:16.727 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:55:16.815 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:55:16.828 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:55:16.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:16.850 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:16.850 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:55:16.951 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:55:16.972 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:16.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:16.973 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:55:17.074 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:55:17.085 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-20T22:55:17.095 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:17.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:17.096 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:55:17.197 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:55:17.218 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:17.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:17.219 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:55:17.250 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-20T22:55:17.250 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:55:17.320 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:55:17.338 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is set 2024-09-20T22:55:17.338 INFO:tasks.rados.rados.0.smithi084.stdout:3346: done (0 left) 2024-09-20T22:55:17.338 INFO:tasks.rados.rados.0.smithi084.stdout:3347: write oid 42 current snap is 385 2024-09-20T22:55:17.339 INFO:tasks.rados.rados.0.smithi084.stdout:3347: seq_num 1139 ranges {743173=645208,1885108=762728} 2024-09-20T22:55:17.348 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:17.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:17.349 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:55:17.351 INFO:tasks.rados.rados.0.smithi084.stdout:3347: writing smithi08477203-42 from 743173 to 1388381 tid 1 2024-09-20T22:55:17.353 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:55:17.359 INFO:tasks.rados.rados.0.smithi084.stdout:3347: writing smithi08477203-42 from 1885108 to 2647836 tid 2 2024-09-20T22:55:17.359 INFO:tasks.rados.rados.0.smithi084.stdout:3348: write (excl) oid 18 current snap is 385 2024-09-20T22:55:17.359 INFO:tasks.rados.rados.0.smithi084.stdout:3347: finishing write tid 1 to smithi08477203-42 2024-09-20T22:55:17.359 INFO:tasks.rados.rados.0.smithi084.stdout:3348: seq_num 1140 ranges {759980=720457,1943590=506386,2856447=303533} 2024-09-20T22:55:17.367 INFO:tasks.rados.rados.0.smithi084.stdout:3348: writing smithi08477203-18 from 759980 to 1480437 tid 1 2024-09-20T22:55:17.373 INFO:tasks.rados.rados.0.smithi084.stdout:3348: writing smithi08477203-18 from 1943590 to 2449976 tid 2 2024-09-20T22:55:17.377 INFO:tasks.rados.rados.0.smithi084.stdout:3348: writing smithi08477203-18 from 2856447 to 3159980 tid 3 2024-09-20T22:55:17.378 INFO:tasks.rados.rados.0.smithi084.stdout:3349: write (excl) oid 48 current snap is 385 2024-09-20T22:55:17.378 INFO:tasks.rados.rados.0.smithi084.stdout:3347: finishing write tid 2 to smithi08477203-42 2024-09-20T22:55:17.378 INFO:tasks.rados.rados.0.smithi084.stdout:3349: seq_num 1141 ranges {776787=512059,1718425=766397,3000223=176564} 2024-09-20T22:55:17.385 INFO:tasks.rados.rados.0.smithi084.stdout:3349: writing smithi08477203-48 from 776787 to 1288846 tid 1 2024-09-20T22:55:17.391 INFO:tasks.rados.rados.0.smithi084.stdout:3349: writing smithi08477203-48 from 1718425 to 2484822 tid 2 2024-09-20T22:55:17.398 INFO:tasks.rados.rados.0.smithi084.stdout:3349: writing smithi08477203-48 from 3000223 to 3176787 tid 3 2024-09-20T22:55:17.398 INFO:tasks.rados.rados.0.smithi084.stdout:3350: write (excl) oid 14 current snap is 385 2024-09-20T22:55:17.398 INFO:tasks.rados.rados.0.smithi084.stdout:3347: finishing write tid 3 to smithi08477203-42 2024-09-20T22:55:17.398 INFO:tasks.rados.rados.0.smithi084.stdout:3347: finishing write tid 4 to smithi08477203-42 2024-09-20T22:55:17.398 INFO:tasks.rados.rados.0.smithi084.stdout:3347: oid 42 updating version 0 to 1921 2024-09-20T22:55:17.398 INFO:tasks.rados.rados.0.smithi084.stdout:3347: oid 42 version 1921 is already newer than 1919 2024-09-20T22:55:17.398 INFO:tasks.rados.rados.0.smithi084.stdout:3347: oid 42 version 1921 is already newer than 1920 2024-09-20T22:55:17.399 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 42 v 1921 (ObjNum 1139 snap 385 seq_num 1139) dirty exists 2024-09-20T22:55:17.399 INFO:tasks.rados.rados.0.smithi084.stdout:3347: left oid 42 (ObjNum 1139 snap 385 seq_num 1139) 2024-09-20T22:55:17.399 INFO:tasks.rados.rados.0.smithi084.stdout:3348: finishing write tid 1 to smithi08477203-18 2024-09-20T22:55:17.399 INFO:tasks.rados.rados.0.smithi084.stdout:3348: finishing write tid 2 to smithi08477203-18 2024-09-20T22:55:17.399 INFO:tasks.rados.rados.0.smithi084.stdout:3350: seq_num 1142 ranges {793594=587308,2060554=510055} 2024-09-20T22:55:17.403 INFO:tasks.rados.rados.0.smithi084.stdout:3350: writing smithi08477203-14 from 793594 to 1380902 tid 1 2024-09-20T22:55:17.410 INFO:tasks.rados.rados.0.smithi084.stdout:3350: writing smithi08477203-14 from 2060554 to 2570609 tid 2 2024-09-20T22:55:17.410 INFO:tasks.rados.rados.0.smithi084.stdout:3347: done (3 left) 2024-09-20T22:55:17.410 INFO:tasks.rados.rados.0.smithi084.stdout:3351: read oid 19 snap -1 2024-09-20T22:55:17.410 INFO:tasks.rados.rados.0.smithi084.stdout:3351: expect (ObjNum 1100 snap 372 seq_num 1100) 2024-09-20T22:55:17.410 INFO:tasks.rados.rados.0.smithi084.stdout:3352: snap_create 2024-09-20T22:55:17.410 INFO:tasks.rados.rados.0.smithi084.stdout:3348: finishing write tid 3 to smithi08477203-18 2024-09-20T22:55:17.410 INFO:tasks.rados.rados.0.smithi084.stdout:3348: finishing write tid 4 to smithi08477203-18 2024-09-20T22:55:17.410 INFO:tasks.rados.rados.0.smithi084.stdout:3348: finishing write tid 5 to smithi08477203-18 2024-09-20T22:55:17.410 INFO:tasks.rados.rados.0.smithi084.stdout:3348: oid 18 updating version 0 to 2464 2024-09-20T22:55:17.410 INFO:tasks.rados.rados.0.smithi084.stdout:3348: oid 18 updating version 2464 to 2465 2024-09-20T22:55:17.410 INFO:tasks.rados.rados.0.smithi084.stdout:3348: oid 18 updating version 2465 to 2467 2024-09-20T22:55:17.411 INFO:tasks.rados.rados.0.smithi084.stdout:3348: oid 18 version 2467 is already newer than 2466 2024-09-20T22:55:17.411 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 18 v 2467 (ObjNum 1140 snap 385 seq_num 1140) dirty exists 2024-09-20T22:55:17.411 INFO:tasks.rados.rados.0.smithi084.stdout:3348: left oid 18 (ObjNum 1140 snap 385 seq_num 1140) 2024-09-20T22:55:17.411 INFO:tasks.rados.rados.0.smithi084.stdout:3349: finishing write tid 1 to smithi08477203-48 2024-09-20T22:55:17.411 INFO:tasks.rados.rados.0.smithi084.stdout:3349: finishing write tid 2 to smithi08477203-48 2024-09-20T22:55:17.411 INFO:tasks.rados.rados.0.smithi084.stdout:3349: finishing write tid 3 to smithi08477203-48 2024-09-20T22:55:17.411 INFO:tasks.rados.rados.0.smithi084.stdout:3349: finishing write tid 4 to smithi08477203-48 2024-09-20T22:55:17.411 INFO:tasks.rados.rados.0.smithi084.stdout:3349: finishing write tid 5 to smithi08477203-48 2024-09-20T22:55:17.411 INFO:tasks.rados.rados.0.smithi084.stdout:3349: oid 48 updating version 0 to 1935 2024-09-20T22:55:17.411 INFO:tasks.rados.rados.0.smithi084.stdout:3349: oid 48 version 1935 is already newer than 1933 2024-09-20T22:55:17.411 INFO:tasks.rados.rados.0.smithi084.stdout:3349: oid 48 version 1935 is already newer than 1934 2024-09-20T22:55:17.411 INFO:tasks.rados.rados.0.smithi084.stdout:3349: oid 48 updating version 1935 to 1936 2024-09-20T22:55:17.411 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 48 v 1936 (ObjNum 1141 snap 385 seq_num 1141) dirty exists 2024-09-20T22:55:17.411 INFO:tasks.rados.rados.0.smithi084.stdout:3349: left oid 48 (ObjNum 1141 snap 385 seq_num 1141) 2024-09-20T22:55:17.412 INFO:tasks.rados.rados.0.smithi084.stdout:3350: finishing write tid 1 to smithi08477203-14 2024-09-20T22:55:17.412 INFO:tasks.rados.rados.0.smithi084.stdout:3350: finishing write tid 2 to smithi08477203-14 2024-09-20T22:55:17.412 INFO:tasks.rados.rados.0.smithi084.stdout:3350: finishing write tid 3 to smithi08477203-14 2024-09-20T22:55:17.412 INFO:tasks.rados.rados.0.smithi084.stdout:3350: finishing write tid 4 to smithi08477203-14 2024-09-20T22:55:17.412 INFO:tasks.rados.rados.0.smithi084.stdout:3350: oid 14 updating version 0 to 1933 2024-09-20T22:55:17.412 INFO:tasks.rados.rados.0.smithi084.stdout:3350: oid 14 updating version 1933 to 1935 2024-09-20T22:55:17.412 INFO:tasks.rados.rados.0.smithi084.stdout:3350: oid 14 version 1935 is already newer than 1934 2024-09-20T22:55:17.413 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 14 v 1935 (ObjNum 1142 snap 385 seq_num 1142) dirty exists 2024-09-20T22:55:17.413 INFO:tasks.rados.rados.0.smithi084.stdout:3350: left oid 14 (ObjNum 1142 snap 385 seq_num 1142) 2024-09-20T22:55:17.420 INFO:tasks.rados.rados.0.smithi084.stdout:3351: expect (ObjNum 1100 snap 372 seq_num 1100) 2024-09-20T22:55:17.450 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:55:17.471 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:17.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:17.472 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:55:17.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:17 smithi154 ceph-mon[33264]: pgmap v1155: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 550 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 10 MiB/s rd, 2.9 MiB/s wr, 21 op/s; 116 B/s, 4 objects/s recovering 2024-09-20T22:55:17.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:17 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.0}]': finished 2024-09-20T22:55:17.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:17 smithi154 ceph-mon[33264]: osdmap e836: 8 total, 8 up, 8 in 2024-09-20T22:55:17.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:17 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/876333410' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:55:17.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:17 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:55:17.573 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:55:17.595 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:17.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:17.595 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:55:17.614 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:17 smithi084 ceph-mon[40024]: pgmap v1155: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 550 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 10 MiB/s rd, 2.9 MiB/s wr, 21 op/s; 116 B/s, 4 objects/s recovering 2024-09-20T22:55:17.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:17 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.0}]': finished 2024-09-20T22:55:17.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:17 smithi084 ceph-mon[40024]: osdmap e836: 8 total, 8 up, 8 in 2024-09-20T22:55:17.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:17 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/876333410' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:55:17.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:17 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:55:17.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:17 smithi084 ceph-mon[29774]: pgmap v1155: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 550 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 10 MiB/s rd, 2.9 MiB/s wr, 21 op/s; 116 B/s, 4 objects/s recovering 2024-09-20T22:55:17.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:17 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 0.0}]': finished 2024-09-20T22:55:17.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:17 smithi084 ceph-mon[29774]: osdmap e836: 8 total, 8 up, 8 in 2024-09-20T22:55:17.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:17 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/876333410' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:55:17.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:17 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:55:17.696 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:55:17.717 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:17.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:17.718 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:55:17.819 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:55:17.840 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:17.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:17.841 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:55:17.941 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:55:17.963 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:17.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:17.964 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:55:17.990 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.7 dump_historic_ops 2024-09-20T22:55:18.064 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:55:18.085 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:18.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:18.086 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:55:18.187 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:55:18.208 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:18.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:18.209 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:55:18.237 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:55:18.310 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:55:18.326 INFO:tasks.rados.rados.0.smithi084.stdout:3348: done (4 left) 2024-09-20T22:55:18.326 INFO:tasks.rados.rados.0.smithi084.stdout:3349: done (3 left) 2024-09-20T22:55:18.326 INFO:tasks.rados.rados.0.smithi084.stdout:3350: done (2 left) 2024-09-20T22:55:18.326 INFO:tasks.rados.rados.0.smithi084.stdout:3351: done (1 left) 2024-09-20T22:55:18.326 INFO:tasks.rados.rados.0.smithi084.stdout:3352: done (0 left) 2024-09-20T22:55:18.326 INFO:tasks.rados.rados.0.smithi084.stdout:3353: rollback oid 31 current snap is 386 2024-09-20T22:55:18.326 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 31 to 376 2024-09-20T22:55:18.326 INFO:tasks.rados.rados.0.smithi084.stdout:3354: copy_from oid 47 from oid 2 current snap is 386 2024-09-20T22:55:18.326 INFO:tasks.rados.rados.0.smithi084.stdout:3355: snap_create 2024-09-20T22:55:18.331 INFO:tasks.rados.rados.0.smithi084.stdout:3353: finishing rollback tid 0 to smithi08477203-31 2024-09-20T22:55:18.332 INFO:tasks.rados.rados.0.smithi084.stdout:3353: finishing rollback tid 1 to smithi08477203-31 2024-09-20T22:55:18.332 INFO:tasks.rados.rados.0.smithi084.stdout:3353: finishing rollback tid 2 to smithi08477203-31 2024-09-20T22:55:18.332 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 31 v 2476 (ObjNum 1101 snap 372 seq_num 1101) dirty exists 2024-09-20T22:55:18.334 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:18.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:18.335 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:55:18.344 INFO:tasks.rados.rados.0.smithi084.stdout:3354: finishing copy_from racing read to smithi08477203-47 2024-09-20T22:55:18.353 INFO:tasks.rados.rados.0.smithi084.stdout:3354: finishing copy_from to smithi08477203-47 2024-09-20T22:55:18.353 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 47 v 1835 (ObjNum 918 snap 302 seq_num 918) dirty exists 2024-09-20T22:55:18.435 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:55:18.456 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:18.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:18.457 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:55:18.518 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:55:18 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: ::ffff:172.21.15.154 - - [20/Sep/2024:22:55:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-20T22:55:18.519 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:18 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T22:55:18.519 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:18 smithi084 ceph-mon[29774]: osdmap e837: 8 total, 8 up, 8 in 2024-09-20T22:55:18.519 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:18 smithi084 ceph-mon[29774]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:55:18.519 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:18 smithi084 ceph-mon[29774]: osdmap e838: 8 total, 8 up, 8 in 2024-09-20T22:55:18.519 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:18 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T22:55:18.519 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:18 smithi084 ceph-mon[40024]: osdmap e837: 8 total, 8 up, 8 in 2024-09-20T22:55:18.519 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:18 smithi084 ceph-mon[40024]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:55:18.519 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:18 smithi084 ceph-mon[40024]: osdmap e838: 8 total, 8 up, 8 in 2024-09-20T22:55:18.520 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:55:18.520 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:55:18.535 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:55:18.535 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:55:18.550 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:55:18.551 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:55:18.558 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:55:18.566 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-20T22:55:18.567 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T22:55:18.579 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:18.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:18.580 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:55:18.582 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:55:18.583 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:55:18.599 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:55:18.599 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:55:18.615 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:55:18.616 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:55:18.632 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:55:18.632 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:55:18.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:18 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T22:55:18.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:18 smithi154 ceph-mon[33264]: osdmap e837: 8 total, 8 up, 8 in 2024-09-20T22:55:18.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:18 smithi154 ceph-mon[33264]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:55:18.642 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:18 smithi154 ceph-mon[33264]: osdmap e838: 8 total, 8 up, 8 in 2024-09-20T22:55:18.681 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:55:18.703 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:18.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:18.704 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:55:18.744 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-20T22:55:18.745 INFO:teuthology.orchestra.run.smithi154.stdout: "size": 20, 2024-09-20T22:55:18.745 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 600, 2024-09-20T22:55:18.745 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [ 2024-09-20T22:55:18.745 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.745 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:2318 7.5 7:a160b5d4:::smithi08477203-16:head [copy-from ver 168] snapc 55=[55,53,52,51,50,4b,47,3e] ondisk+write+known_if_redirected+supports_pool_eio e258)", 2024-09-20T22:55:18.745 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:45:37.175863+0000", 2024-09-20T22:55:18.745 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 581.56577464700001, 2024-09-20T22:55:18.745 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.030719883999999999, 2024-09-20T22:55:18.745 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:55:18.745 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:55:18.745 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:55:18.745 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:55:18.746 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:55:18.746 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:55:18.746 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 2318 2024-09-20T22:55:18.746 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.746 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:55:18.746 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.746 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:55:18.746 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.175863+0000", 2024-09-20T22:55:18.746 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:55:18.746 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.746 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.746 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:55:18.746 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.175863+0000", 2024-09-20T22:55:18.746 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:55:18.747 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.747 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.747 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:55:18.747 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.175865+0000", 2024-09-20T22:55:18.747 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.488e-06 2024-09-20T22:55:18.747 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.747 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.747 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:55:18.747 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.175871+0000", 2024-09-20T22:55:18.747 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.438e-06 2024-09-20T22:55:18.747 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.747 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.747 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:55:18.747 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.175881+0000", 2024-09-20T22:55:18.748 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.6430000000000007e-06 2024-09-20T22:55:18.748 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.748 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.748 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:55:18.748 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.175886+0000", 2024-09-20T22:55:18.748 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.3059999999999997e-06 2024-09-20T22:55:18.748 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.748 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.748 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:55:18.748 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.175944+0000", 2024-09-20T22:55:18.748 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.7982999999999997e-05 2024-09-20T22:55:18.748 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.748 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.748 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op must wait for map", 2024-09-20T22:55:18.748 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.175951+0000", 2024-09-20T22:55:18.749 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.9280000000000004e-06 2024-09-20T22:55:18.749 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.749 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.749 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:55:18.749 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.179794+0000", 2024-09-20T22:55:18.749 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0038435209999999999 2024-09-20T22:55:18.749 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.749 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.749 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:55:18.749 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.179940+0000", 2024-09-20T22:55:18.749 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00014550000000000001 2024-09-20T22:55:18.749 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.749 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.750 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 0,2", 2024-09-20T22:55:18.750 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.190992+0000", 2024-09-20T22:55:18.750 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.011051587999999999 2024-09-20T22:55:18.750 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.750 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.750 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:55:18.750 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.195070+0000", 2024-09-20T22:55:18.750 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0040784929999999999 2024-09-20T22:55:18.750 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.750 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.750 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:55:18.750 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.202908+0000", 2024-09-20T22:55:18.750 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0078376690000000002 2024-09-20T22:55:18.750 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.751 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.751 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:55:18.751 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.206517+0000", 2024-09-20T22:55:18.751 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0036088880000000002 2024-09-20T22:55:18.751 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.751 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.751 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:55:18.751 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.206553+0000", 2024-09-20T22:55:18.751 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.6363999999999999e-05 2024-09-20T22:55:18.751 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.751 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.751 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:55:18.751 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.206583+0000", 2024-09-20T22:55:18.752 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.0074999999999999e-05 2024-09-20T22:55:18.752 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:55:18.752 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:55:18.752 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:55:18.752 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.752 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.752 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:2319 7.5 7:a160b5d4:::smithi08477203-16:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e258)", 2024-09-20T22:55:18.752 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:45:37.175983+0000", 2024-09-20T22:55:18.752 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 581.565654892, 2024-09-20T22:55:18.752 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.018323547999999999, 2024-09-20T22:55:18.752 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:55:18.752 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:55:18.752 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:55:18.753 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:55:18.753 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:55:18.753 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:55:18.753 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 2319 2024-09-20T22:55:18.753 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.753 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:55:18.753 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.753 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:55:18.753 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.175983+0000", 2024-09-20T22:55:18.753 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:55:18.753 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.753 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.753 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:55:18.753 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.175983+0000", 2024-09-20T22:55:18.754 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:55:18.754 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.754 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.754 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:55:18.754 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.175984+0000", 2024-09-20T22:55:18.754 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.1099999999999995e-07 2024-09-20T22:55:18.754 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.754 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.754 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:55:18.754 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.175996+0000", 2024-09-20T22:55:18.754 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2121e-05 2024-09-20T22:55:18.754 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.754 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.754 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:55:18.755 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.176019+0000", 2024-09-20T22:55:18.755 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.3465999999999999e-05 2024-09-20T22:55:18.755 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.755 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.755 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:55:18.755 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.176026+0000", 2024-09-20T22:55:18.755 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.8190000000000003e-06 2024-09-20T22:55:18.755 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.755 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.755 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:55:18.755 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.176110+0000", 2024-09-20T22:55:18.755 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.3967000000000004e-05 2024-09-20T22:55:18.755 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.756 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.756 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting_for_map not empty", 2024-09-20T22:55:18.756 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.176116+0000", 2024-09-20T22:55:18.756 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.0249999999999999e-06 2024-09-20T22:55:18.756 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.756 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.756 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:55:18.756 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.180106+0000", 2024-09-20T22:55:18.756 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0039898980000000004 2024-09-20T22:55:18.756 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.756 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.756 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for blocked object", 2024-09-20T22:55:18.756 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.180135+0000", 2024-09-20T22:55:18.756 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.9595999999999999e-05 2024-09-20T22:55:18.756 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.756 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.757 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:55:18.757 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.194106+0000", 2024-09-20T22:55:18.757 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.013970761999999999 2024-09-20T22:55:18.757 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.757 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.757 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:55:18.757 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.194203+0000", 2024-09-20T22:55:18.757 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.7211000000000002e-05 2024-09-20T22:55:18.757 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.757 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.757 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:55:18.757 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:37.194306+0000", 2024-09-20T22:55:18.757 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00010307200000000001 2024-09-20T22:55:18.757 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:55:18.757 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:55:18.757 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:55:18.758 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.758 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.758 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24415.0:230 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch (1) in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e259)", 2024-09-20T22:55:18.758 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:45:39.207410+0000", 2024-09-20T22:55:18.758 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 579.53422781999996, 2024-09-20T22:55:18.758 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0072298289999999, 2024-09-20T22:55:18.758 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:55:18.758 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:55:18.758 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:55:18.758 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:55:18.758 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24415", 2024-09-20T22:55:18.758 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/2056244102", 2024-09-20T22:55:18.758 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 230 2024-09-20T22:55:18.758 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.759 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:55:18.759 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.759 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:55:18.759 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.207410+0000", 2024-09-20T22:55:18.759 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:55:18.759 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.759 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.759 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:55:18.759 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.207410+0000", 2024-09-20T22:55:18.759 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:55:18.759 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.759 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.759 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:55:18.759 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.207412+0000", 2024-09-20T22:55:18.759 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.7519999999999999e-06 2024-09-20T22:55:18.760 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.760 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.760 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:55:18.760 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.207419+0000", 2024-09-20T22:55:18.760 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.097e-06 2024-09-20T22:55:18.760 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.760 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.760 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:55:18.760 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.207432+0000", 2024-09-20T22:55:18.760 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2719e-05 2024-09-20T22:55:18.760 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.760 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.760 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:55:18.760 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.207464+0000", 2024-09-20T22:55:18.760 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.2820000000000001e-05 2024-09-20T22:55:18.761 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.761 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.761 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:55:18.761 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.207670+0000", 2024-09-20T22:55:18.761 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000205486 2024-09-20T22:55:18.761 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.761 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.761 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-20T22:55:18.761 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.207687+0000", 2024-09-20T22:55:18.761 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.7107000000000001e-05 2024-09-20T22:55:18.761 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.761 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.761 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:55:18.761 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:40.212672+0000", 2024-09-20T22:55:18.761 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.004984562 2024-09-20T22:55:18.761 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.762 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.762 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:55:18.762 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:40.213576+0000", 2024-09-20T22:55:18.762 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00090404799999999996 2024-09-20T22:55:18.762 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.762 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.762 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:55:18.762 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:40.214640+0000", 2024-09-20T22:55:18.762 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0010642379999999999 2024-09-20T22:55:18.762 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:55:18.762 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:55:18.762 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:55:18.762 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.762 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.763 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:2367 7.5 7:a160b5d4:::smithi08477203-16:head [stat,delete] snapc 56=[56,55,53,52,51,50,47,3e] ondisk+write+known_if_redirected+supports_pool_eio e260)", 2024-09-20T22:55:18.763 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:45:39.601299+0000", 2024-09-20T22:55:18.763 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 579.14033839900003, 2024-09-20T22:55:18.763 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.64714523300000004, 2024-09-20T22:55:18.763 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:55:18.763 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:55:18.763 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:55:18.763 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:55:18.763 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:55:18.763 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:55:18.763 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 2367 2024-09-20T22:55:18.763 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.763 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:55:18.763 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.763 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:55:18.764 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.601299+0000", 2024-09-20T22:55:18.764 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:55:18.764 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.764 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.764 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:55:18.764 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.601299+0000", 2024-09-20T22:55:18.764 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:55:18.764 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.764 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.764 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:55:18.764 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.601301+0000", 2024-09-20T22:55:18.764 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3850000000000001e-06 2024-09-20T22:55:18.764 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.764 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.764 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:55:18.764 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.601306+0000", 2024-09-20T22:55:18.765 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.2730000000000002e-06 2024-09-20T22:55:18.765 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.765 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.765 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:55:18.765 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.601318+0000", 2024-09-20T22:55:18.765 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2148e-05 2024-09-20T22:55:18.765 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.765 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.765 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:55:18.765 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.601324+0000", 2024-09-20T22:55:18.765 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.852e-06 2024-09-20T22:55:18.765 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.765 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.765 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:55:18.765 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.601383+0000", 2024-09-20T22:55:18.765 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.8570000000000003e-05 2024-09-20T22:55:18.766 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.766 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.766 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-20T22:55:18.766 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:39.601391+0000", 2024-09-20T22:55:18.766 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.5290000000000008e-06 2024-09-20T22:55:18.766 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.766 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.766 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:55:18.766 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:40.221227+0000", 2024-09-20T22:55:18.766 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.619835515 2024-09-20T22:55:18.766 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.766 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.766 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for degraded object", 2024-09-20T22:55:18.766 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:40.223919+0000", 2024-09-20T22:55:18.767 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.002692168 2024-09-20T22:55:18.767 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.767 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.767 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:55:18.767 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:40.244536+0000", 2024-09-20T22:55:18.767 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.020617613 2024-09-20T22:55:18.767 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.767 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.767 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:55:18.767 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:40.244653+0000", 2024-09-20T22:55:18.767 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000116796 2024-09-20T22:55:18.767 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.767 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.768 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 1,2", 2024-09-20T22:55:18.768 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:40.244832+0000", 2024-09-20T22:55:18.768 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00017833500000000001 2024-09-20T22:55:18.768 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.768 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.768 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:55:18.768 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:40.245494+0000", 2024-09-20T22:55:18.768 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00066241399999999995 2024-09-20T22:55:18.768 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.768 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.768 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:55:18.768 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:40.246501+0000", 2024-09-20T22:55:18.768 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0010066299999999999 2024-09-20T22:55:18.768 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.769 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.769 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:55:18.769 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:40.248326+0000", 2024-09-20T22:55:18.769 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001824895 2024-09-20T22:55:18.769 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.769 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.769 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:55:18.769 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:40.248370+0000", 2024-09-20T22:55:18.769 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.4298999999999997e-05 2024-09-20T22:55:18.769 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.769 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.769 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:55:18.769 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:45:40.248445+0000", 2024-09-20T22:55:18.769 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.4810999999999999e-05 2024-09-20T22:55:18.770 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:55:18.770 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:55:18.770 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:55:18.770 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.770 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.770 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:3132 7.19 e328/139 7:9eac4560:::smithi08477203-30:head v 328'198, mlcod=322'194)", 2024-09-20T22:55:18.770 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:46:48.223406+0000", 2024-09-20T22:55:18.770 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 510.51823196300001, 2024-09-20T22:55:18.770 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.016765688000000001, 2024-09-20T22:55:18.770 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:55:18.770 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:55:18.770 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:55:18.770 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:55:18.771 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.771 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:55:18.771 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.223406+0000", 2024-09-20T22:55:18.771 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:55:18.771 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.771 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.771 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:55:18.771 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.223406+0000", 2024-09-20T22:55:18.771 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:55:18.771 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.771 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.771 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:55:18.771 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.223406+0000", 2024-09-20T22:55:18.773 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.6699999999999999e-07 2024-09-20T22:55:18.773 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.773 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.773 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:55:18.773 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.226227+0000", 2024-09-20T22:55:18.773 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0028205690000000002 2024-09-20T22:55:18.773 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.773 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.773 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:55:18.773 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.226238+0000", 2024-09-20T22:55:18.773 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0879999999999999e-05 2024-09-20T22:55:18.773 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.773 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.774 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:55:18.774 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.226249+0000", 2024-09-20T22:55:18.774 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0912e-05 2024-09-20T22:55:18.774 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.774 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.774 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:55:18.774 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.227960+0000", 2024-09-20T22:55:18.774 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0017109619999999999 2024-09-20T22:55:18.774 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.774 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.774 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:55:18.774 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.227987+0000", 2024-09-20T22:55:18.774 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.7659999999999999e-05 2024-09-20T22:55:18.774 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.775 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.775 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:55:18.775 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.240119+0000", 2024-09-20T22:55:18.775 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.012131885 2024-09-20T22:55:18.775 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.775 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.775 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:55:18.775 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.240172+0000", 2024-09-20T22:55:18.775 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.2453000000000001e-05 2024-09-20T22:55:18.775 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:55:18.775 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:55:18.775 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:55:18.775 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.775 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.776 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:3232 7.16 7:6885a65f:::smithi08477203-14:head [copy-from ver 187] snapc 7e=[7e,7d,7c,7b,7a,79,78,75,74,73,70,6b] ondisk+write+known_if_redirected+supports_pool_eio e337)", 2024-09-20T22:55:18.776 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:46:57.213186+0000", 2024-09-20T22:55:18.776 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 501.528452292, 2024-09-20T22:55:18.776 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.028083105000000001, 2024-09-20T22:55:18.776 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:55:18.776 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:55:18.776 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:55:18.776 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:55:18.776 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:55:18.776 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:55:18.776 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 3232 2024-09-20T22:55:18.776 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.776 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:55:18.776 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.776 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:55:18.777 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.213186+0000", 2024-09-20T22:55:18.777 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:55:18.777 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.777 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.777 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:55:18.777 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.213186+0000", 2024-09-20T22:55:18.777 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:55:18.777 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.777 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.777 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:55:18.777 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.213188+0000", 2024-09-20T22:55:18.777 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.5129999999999999e-06 2024-09-20T22:55:18.777 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.777 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.777 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:55:18.778 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.213194+0000", 2024-09-20T22:55:18.778 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.4810000000000003e-06 2024-09-20T22:55:18.778 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.778 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.778 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:55:18.778 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.213215+0000", 2024-09-20T22:55:18.778 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.0455999999999999e-05 2024-09-20T22:55:18.778 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.778 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.778 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:55:18.778 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.213225+0000", 2024-09-20T22:55:18.778 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0212e-05 2024-09-20T22:55:18.778 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.779 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.779 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:55:18.779 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.213315+0000", 2024-09-20T22:55:18.779 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.9524000000000005e-05 2024-09-20T22:55:18.779 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.779 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.779 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op must wait for map", 2024-09-20T22:55:18.779 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.213324+0000", 2024-09-20T22:55:18.779 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.8890000000000008e-06 2024-09-20T22:55:18.779 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.779 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.779 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:55:18.779 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.220517+0000", 2024-09-20T22:55:18.779 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0071931219999999997 2024-09-20T22:55:18.780 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.780 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.780 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:55:18.780 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.220636+0000", 2024-09-20T22:55:18.780 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000118875 2024-09-20T22:55:18.780 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.780 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.780 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 2,5", 2024-09-20T22:55:18.780 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.230308+0000", 2024-09-20T22:55:18.780 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0096726299999999998 2024-09-20T22:55:18.780 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.780 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.780 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:55:18.781 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.238007+0000", 2024-09-20T22:55:18.781 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.007699287 2024-09-20T22:55:18.781 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.781 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.781 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:55:18.781 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.240863+0000", 2024-09-20T22:55:18.781 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0028554880000000002 2024-09-20T22:55:18.781 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.781 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.781 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:55:18.781 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.241194+0000", 2024-09-20T22:55:18.781 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00033087399999999999 2024-09-20T22:55:18.781 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.781 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.781 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:55:18.782 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.241233+0000", 2024-09-20T22:55:18.782 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.9338000000000002e-05 2024-09-20T22:55:18.782 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.782 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.782 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:55:18.782 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.241269+0000", 2024-09-20T22:55:18.782 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.5416000000000001e-05 2024-09-20T22:55:18.782 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:55:18.782 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:55:18.782 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:55:18.782 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.782 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.782 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:3233 7.16 7:6885a65f:::smithi08477203-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e337)", 2024-09-20T22:55:18.783 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:46:57.213253+0000", 2024-09-20T22:55:18.783 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 501.52838486799999, 2024-09-20T22:55:18.783 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.023594825, 2024-09-20T22:55:18.783 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:55:18.783 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:55:18.783 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:55:18.783 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:55:18.783 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:55:18.783 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:55:18.783 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 3233 2024-09-20T22:55:18.783 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.783 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:55:18.783 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.783 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:55:18.784 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.213253+0000", 2024-09-20T22:55:18.784 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:55:18.784 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.784 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.784 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:55:18.784 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.213253+0000", 2024-09-20T22:55:18.784 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:55:18.784 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.784 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.784 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:55:18.784 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.213254+0000", 2024-09-20T22:55:18.784 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.4799999999999998e-07 2024-09-20T22:55:18.784 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.784 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.785 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:55:18.785 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.213257+0000", 2024-09-20T22:55:18.785 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.3780000000000001e-06 2024-09-20T22:55:18.785 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.785 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.785 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:55:18.785 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.213267+0000", 2024-09-20T22:55:18.785 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.9480000000000003e-06 2024-09-20T22:55:18.785 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.785 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.785 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:55:18.785 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.213273+0000", 2024-09-20T22:55:18.785 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.1240000000000002e-06 2024-09-20T22:55:18.785 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.786 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.786 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:55:18.786 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.213394+0000", 2024-09-20T22:55:18.786 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000120717 2024-09-20T22:55:18.786 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.786 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.786 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting_for_map not empty", 2024-09-20T22:55:18.786 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.213400+0000", 2024-09-20T22:55:18.786 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.0769999999999996e-06 2024-09-20T22:55:18.786 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.786 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.786 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:55:18.787 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.220851+0000", 2024-09-20T22:55:18.787 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0074511309999999997 2024-09-20T22:55:18.787 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.787 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.787 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for blocked object", 2024-09-20T22:55:18.787 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.220903+0000", 2024-09-20T22:55:18.787 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.215e-05 2024-09-20T22:55:18.787 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.787 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.787 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:55:18.787 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.236698+0000", 2024-09-20T22:55:18.787 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.015794823 2024-09-20T22:55:18.787 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.787 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.788 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:55:18.788 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.236766+0000", 2024-09-20T22:55:18.788 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.8126999999999993e-05 2024-09-20T22:55:18.788 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.788 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.788 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:55:18.788 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.236848+0000", 2024-09-20T22:55:18.788 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.1701999999999998e-05 2024-09-20T22:55:18.788 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:55:18.788 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:55:18.788 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:55:18.788 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.788 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.788 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:3671 7.5 7:a160b5d4:::smithi08477203-16:head [copy-from ver 236] snapc 91=[91,8e,8d,8c,8b,8a,88,87,86,85,84,80,7f,7a,74,73,6b] ondisk+write+known_if_redirected+supports_pool_eio e370)", 2024-09-20T22:55:18.788 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:47:30.237235+0000", 2024-09-20T22:55:18.789 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 468.50440306899998, 2024-09-20T22:55:18.789 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.019719474000000001, 2024-09-20T22:55:18.789 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:55:18.789 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:55:18.789 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:55:18.789 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:55:18.789 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:55:18.789 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:55:18.789 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 3671 2024-09-20T22:55:18.789 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.789 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:55:18.789 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.789 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:55:18.789 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:30.237235+0000", 2024-09-20T22:55:18.789 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:55:18.790 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.790 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.790 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:55:18.790 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:30.237235+0000", 2024-09-20T22:55:18.790 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:55:18.790 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.790 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.790 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:55:18.790 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:30.237236+0000", 2024-09-20T22:55:18.790 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.5570000000000001e-06 2024-09-20T22:55:18.790 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.790 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.790 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:55:18.790 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:30.237241+0000", 2024-09-20T22:55:18.791 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.3279999999999999e-06 2024-09-20T22:55:18.791 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.791 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.791 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:55:18.791 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:30.237252+0000", 2024-09-20T22:55:18.791 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1004e-05 2024-09-20T22:55:18.791 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.791 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.791 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:55:18.791 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:30.237263+0000", 2024-09-20T22:55:18.791 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1341e-05 2024-09-20T22:55:18.791 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.791 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.791 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:55:18.792 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:30.237319+0000", 2024-09-20T22:55:18.792 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.5939000000000001e-05 2024-09-20T22:55:18.792 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.792 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.792 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:55:18.792 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:30.237436+0000", 2024-09-20T22:55:18.792 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000116866 2024-09-20T22:55:18.792 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.792 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.792 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 1,2", 2024-09-20T22:55:18.792 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:30.244330+0000", 2024-09-20T22:55:18.792 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0068945619999999999 2024-09-20T22:55:18.792 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.792 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.792 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:55:18.792 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:30.249103+0000", 2024-09-20T22:55:18.793 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0047726319999999997 2024-09-20T22:55:18.793 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.793 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.793 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:55:18.793 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:30.255771+0000", 2024-09-20T22:55:18.793 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.006667666 2024-09-20T22:55:18.793 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.793 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.793 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:55:18.793 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:30.256879+0000", 2024-09-20T22:55:18.793 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0011083110000000001 2024-09-20T22:55:18.793 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.793 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.793 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:55:18.793 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:30.256919+0000", 2024-09-20T22:55:18.794 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.9589000000000003e-05 2024-09-20T22:55:18.794 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.794 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.794 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:55:18.794 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:30.256954+0000", 2024-09-20T22:55:18.794 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.5679000000000002e-05 2024-09-20T22:55:18.794 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:55:18.794 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:55:18.794 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:55:18.794 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.794 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.795 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:3747 7.c e383/94 7:3f865ef5:::smithi08477203-28:head v 383'822, mlcod=382'818)", 2024-09-20T22:55:18.795 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:47:43.267267+0000", 2024-09-20T22:55:18.795 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 455.47437060499999, 2024-09-20T22:55:18.795 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.014405072, 2024-09-20T22:55:18.795 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:55:18.795 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:55:18.795 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:55:18.795 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:55:18.795 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.795 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:55:18.795 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.267267+0000", 2024-09-20T22:55:18.795 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:55:18.795 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.795 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.796 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:55:18.796 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.267267+0000", 2024-09-20T22:55:18.796 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:55:18.796 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.796 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.796 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:55:18.796 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.267267+0000", 2024-09-20T22:55:18.796 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.73e-07 2024-09-20T22:55:18.796 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.796 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.796 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:55:18.796 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.269537+0000", 2024-09-20T22:55:18.796 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0022696650000000001 2024-09-20T22:55:18.796 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.797 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.797 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:55:18.797 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.269564+0000", 2024-09-20T22:55:18.797 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.6653999999999999e-05 2024-09-20T22:55:18.797 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.797 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.797 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:55:18.797 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.269575+0000", 2024-09-20T22:55:18.797 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1661999999999999e-05 2024-09-20T22:55:18.797 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.797 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.797 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:55:18.797 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.269672+0000", 2024-09-20T22:55:18.797 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.6430999999999999e-05 2024-09-20T22:55:18.797 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.798 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.798 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:55:18.798 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.269716+0000", 2024-09-20T22:55:18.798 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.4518000000000003e-05 2024-09-20T22:55:18.798 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.798 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.798 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:55:18.798 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.281609+0000", 2024-09-20T22:55:18.798 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.011892982999999999 2024-09-20T22:55:18.798 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.798 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.798 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:55:18.798 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.281672+0000", 2024-09-20T22:55:18.798 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.2985999999999996e-05 2024-09-20T22:55:18.799 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:55:18.799 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:55:18.799 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:55:18.799 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.799 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.799 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:6253 7.9 e629/602 7:91f2d47c:::smithi08477203-8:head v 629'948, mlcod=627'945)", 2024-09-20T22:55:18.799 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:51:49.626110+0000", 2024-09-20T22:55:18.799 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 209.11552771500001, 2024-09-20T22:55:18.799 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.017238303, 2024-09-20T22:55:18.799 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:55:18.799 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:55:18.799 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:55:18.799 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:55:18.799 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.799 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:55:18.800 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:49.626110+0000", 2024-09-20T22:55:18.800 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:55:18.800 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.800 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.800 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:55:18.800 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:49.626110+0000", 2024-09-20T22:55:18.800 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:55:18.800 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.800 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.800 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:55:18.800 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:49.626111+0000", 2024-09-20T22:55:18.800 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.33e-06 2024-09-20T22:55:18.800 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.800 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.801 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:55:18.801 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:49.633470+0000", 2024-09-20T22:55:18.801 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0073589700000000003 2024-09-20T22:55:18.801 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.801 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.801 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:55:18.801 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:49.633492+0000", 2024-09-20T22:55:18.801 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.1443000000000001e-05 2024-09-20T22:55:18.801 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.801 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.801 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:55:18.801 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:49.633503+0000", 2024-09-20T22:55:18.801 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1610000000000001e-05 2024-09-20T22:55:18.801 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.802 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.802 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:55:18.802 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:49.633575+0000", 2024-09-20T22:55:18.802 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.1433999999999994e-05 2024-09-20T22:55:18.802 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.802 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.802 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:55:18.802 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:49.633600+0000", 2024-09-20T22:55:18.802 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.4960000000000002e-05 2024-09-20T22:55:18.802 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.802 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.802 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:55:18.802 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:49.643306+0000", 2024-09-20T22:55:18.802 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0097065109999999993 2024-09-20T22:55:18.803 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.803 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.803 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:55:18.803 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:49.643348+0000", 2024-09-20T22:55:18.803 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.2045000000000001e-05 2024-09-20T22:55:18.803 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:55:18.803 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:55:18.803 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:55:18.803 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.803 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.803 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:6659 7.5 7:a160b5d4:::smithi08477203-16:head [copy-from ver 1807] snapc 12d=[12d,12c,12b,12a,128,127,126,124,122,121,120,11f,11e,11b,119,118,115,112,10e,10c,107,103,fd,fb,fa,f8,ed,d7,c2,bf] ondisk+write+known_if_redirected+supports_pool_eio e674)", 2024-09-20T22:55:18.803 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:52:34.370352+0000", 2024-09-20T22:55:18.803 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 164.37128622099999, 2024-09-20T22:55:18.803 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.018083765000000002, 2024-09-20T22:55:18.804 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:55:18.804 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:55:18.804 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:55:18.804 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:55:18.804 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:55:18.804 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:55:18.804 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 6659 2024-09-20T22:55:18.804 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.804 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:55:18.804 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.804 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:55:18.804 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:34.370352+0000", 2024-09-20T22:55:18.804 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:55:18.804 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.804 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.805 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:55:18.805 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:34.370352+0000", 2024-09-20T22:55:18.805 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:55:18.805 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.805 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.805 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:55:18.805 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:34.370353+0000", 2024-09-20T22:55:18.805 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.61e-06 2024-09-20T22:55:18.805 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.805 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.805 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:55:18.805 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:34.370360+0000", 2024-09-20T22:55:18.805 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.6760000000000001e-06 2024-09-20T22:55:18.805 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.806 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.806 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:55:18.806 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:34.370371+0000", 2024-09-20T22:55:18.806 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1155000000000001e-05 2024-09-20T22:55:18.806 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.806 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.806 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:55:18.806 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:34.370380+0000", 2024-09-20T22:55:18.806 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.9039999999999998e-06 2024-09-20T22:55:18.806 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.806 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.806 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:55:18.807 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:34.370461+0000", 2024-09-20T22:55:18.807 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.0667000000000005e-05 2024-09-20T22:55:18.807 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.807 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.807 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:55:18.807 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:34.370571+0000", 2024-09-20T22:55:18.807 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000110129 2024-09-20T22:55:18.807 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.807 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.807 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 0,2", 2024-09-20T22:55:18.807 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:34.378118+0000", 2024-09-20T22:55:18.807 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0075469489999999998 2024-09-20T22:55:18.807 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.807 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.807 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:55:18.808 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:34.382656+0000", 2024-09-20T22:55:18.808 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0045383239999999998 2024-09-20T22:55:18.808 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.808 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.808 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:55:18.808 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:34.385168+0000", 2024-09-20T22:55:18.808 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0025123770000000001 2024-09-20T22:55:18.808 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.808 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.808 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:55:18.808 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:34.388343+0000", 2024-09-20T22:55:18.808 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.003174713 2024-09-20T22:55:18.808 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.808 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.809 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:55:18.809 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:34.388382+0000", 2024-09-20T22:55:18.809 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.8550000000000002e-05 2024-09-20T22:55:18.809 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.809 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.809 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:55:18.809 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:34.388435+0000", 2024-09-20T22:55:18.809 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.3711000000000001e-05 2024-09-20T22:55:18.809 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:55:18.809 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:55:18.809 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:55:18.809 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.809 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.809 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:6829 7.5 7:a160b5d4:::smithi08477203-16:head [write 548159~419732,stat] snapc 136=[136,135,134,133,132,131,130,12f,12e,12d,12c,12b,12a,128,127,126,124,122,121,11f,11b,119,112,10e,10c,107,103,fb,fa,f8,ed,d7,c2,bf] ondisk+write+known_if_redirected+supports_pool_eio e688)", 2024-09-20T22:55:18.810 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:52:48.308503+0000", 2024-09-20T22:55:18.810 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 150.43313489600001, 2024-09-20T22:55:18.810 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.020227210999999998, 2024-09-20T22:55:18.810 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:55:18.810 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:55:18.810 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:55:18.810 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:55:18.810 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:55:18.810 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:55:18.810 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 6829 2024-09-20T22:55:18.810 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.810 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:55:18.810 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.810 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:55:18.811 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.308503+0000", 2024-09-20T22:55:18.811 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:55:18.811 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.811 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.811 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:55:18.811 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.308503+0000", 2024-09-20T22:55:18.811 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:55:18.811 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.811 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.811 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:55:18.811 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.308505+0000", 2024-09-20T22:55:18.811 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.765e-06 2024-09-20T22:55:18.811 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.811 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.812 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:55:18.812 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.308957+0000", 2024-09-20T22:55:18.812 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00045254399999999998 2024-09-20T22:55:18.812 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.812 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.812 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:55:18.812 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.308975+0000", 2024-09-20T22:55:18.812 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.7489000000000001e-05 2024-09-20T22:55:18.812 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.812 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.812 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:55:18.812 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.308983+0000", 2024-09-20T22:55:18.812 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.4500000000000004e-06 2024-09-20T22:55:18.812 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.813 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.813 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:55:18.813 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.309063+0000", 2024-09-20T22:55:18.813 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.0264000000000003e-05 2024-09-20T22:55:18.813 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.813 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.813 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:55:18.813 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.309200+0000", 2024-09-20T22:55:18.813 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00013697500000000001 2024-09-20T22:55:18.813 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.813 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.813 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 0,2", 2024-09-20T22:55:18.813 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.309374+0000", 2024-09-20T22:55:18.813 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000173597 2024-09-20T22:55:18.814 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.814 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.814 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:55:18.814 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.318972+0000", 2024-09-20T22:55:18.814 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0095980430000000005 2024-09-20T22:55:18.814 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.814 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.814 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:55:18.814 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.326825+0000", 2024-09-20T22:55:18.814 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0078531720000000003 2024-09-20T22:55:18.814 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.814 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.814 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:55:18.814 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.328638+0000", 2024-09-20T22:55:18.815 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0018131969999999999 2024-09-20T22:55:18.815 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.815 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.815 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:55:18.815 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.328685+0000", 2024-09-20T22:55:18.815 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.6536999999999998e-05 2024-09-20T22:55:18.815 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.815 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.815 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:55:18.815 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.328730+0000", 2024-09-20T22:55:18.815 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.5178e-05 2024-09-20T22:55:18.815 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:55:18.815 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:55:18.815 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:55:18.816 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.816 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.816 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:7173 7.8 e721/430 7:1e9e2524:::smithi08477203-5:head v 721'865, mlcod=721'861)", 2024-09-20T22:55:18.816 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:53:21.336783+0000", 2024-09-20T22:55:18.816 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 117.40485502600001, 2024-09-20T22:55:18.816 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.015521446, 2024-09-20T22:55:18.816 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:55:18.816 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:55:18.816 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:55:18.816 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:55:18.816 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.816 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:55:18.816 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:21.336783+0000", 2024-09-20T22:55:18.816 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:55:18.816 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.817 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.817 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:55:18.817 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:21.336783+0000", 2024-09-20T22:55:18.817 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:55:18.817 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.817 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.817 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:55:18.817 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:21.336784+0000", 2024-09-20T22:55:18.817 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.7899999999999997e-07 2024-09-20T22:55:18.817 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.817 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.817 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:55:18.817 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:21.344323+0000", 2024-09-20T22:55:18.817 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0075393090000000001 2024-09-20T22:55:18.818 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.818 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.818 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:55:18.818 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:21.344349+0000", 2024-09-20T22:55:18.818 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.5712e-05 2024-09-20T22:55:18.818 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.818 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.818 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:55:18.818 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:21.344362+0000", 2024-09-20T22:55:18.818 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3131e-05 2024-09-20T22:55:18.818 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.818 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.818 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:55:18.818 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:21.344434+0000", 2024-09-20T22:55:18.819 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.1837999999999999e-05 2024-09-20T22:55:18.819 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.819 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.819 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:55:18.819 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:21.344454+0000", 2024-09-20T22:55:18.819 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.0404999999999999e-05 2024-09-20T22:55:18.819 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.819 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.819 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:55:18.819 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:21.352250+0000", 2024-09-20T22:55:18.819 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0077961489999999996 2024-09-20T22:55:18.819 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.819 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.819 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:55:18.819 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:21.352304+0000", 2024-09-20T22:55:18.820 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.4122999999999997e-05 2024-09-20T22:55:18.820 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:55:18.820 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:55:18.820 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:55:18.820 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.820 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.820 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:7387 7.9 e743/602 7:907aec0e:::smithi08477203-3:head v 743'1242, mlcod=741'1234)", 2024-09-20T22:55:18.820 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:53:43.309636+0000", 2024-09-20T22:55:18.820 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 95.432001854000006, 2024-09-20T22:55:18.820 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.032548296999999997, 2024-09-20T22:55:18.820 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:55:18.820 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:55:18.820 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:55:18.820 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:55:18.820 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.821 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:55:18.821 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:43.309636+0000", 2024-09-20T22:55:18.821 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:55:18.821 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.821 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.821 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:55:18.821 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:43.309636+0000", 2024-09-20T22:55:18.821 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:55:18.821 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.821 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.821 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:55:18.821 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:43.309637+0000", 2024-09-20T22:55:18.821 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.1699999999999997e-07 2024-09-20T22:55:18.821 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.822 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.822 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:55:18.822 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:43.338457+0000", 2024-09-20T22:55:18.822 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.028819827999999999 2024-09-20T22:55:18.822 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.822 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.822 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:55:18.822 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:43.338482+0000", 2024-09-20T22:55:18.822 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.55e-05 2024-09-20T22:55:18.822 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.822 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.822 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:55:18.822 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:43.338493+0000", 2024-09-20T22:55:18.823 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1308000000000001e-05 2024-09-20T22:55:18.823 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.823 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.823 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:55:18.823 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:43.338561+0000", 2024-09-20T22:55:18.823 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.7535999999999998e-05 2024-09-20T22:55:18.823 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.823 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.823 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:55:18.823 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:43.338585+0000", 2024-09-20T22:55:18.823 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.3649e-05 2024-09-20T22:55:18.823 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.823 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.823 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:55:18.824 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:43.342155+0000", 2024-09-20T22:55:18.824 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0035700509999999999 2024-09-20T22:55:18.824 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.824 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.824 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:55:18.824 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:43.342184+0000", 2024-09-20T22:55:18.824 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.9607999999999998e-05 2024-09-20T22:55:18.824 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:55:18.824 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:55:18.824 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:55:18.824 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.824 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.824 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24415.0:744 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch (1) in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e775)", 2024-09-20T22:55:18.824 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:54:16.547604+0000", 2024-09-20T22:55:18.824 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 62.194033890999997, 2024-09-20T22:55:18.825 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.79000451299999996, 2024-09-20T22:55:18.825 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:55:18.825 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:55:18.825 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:55:18.825 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:55:18.825 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24415", 2024-09-20T22:55:18.825 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/2056244102", 2024-09-20T22:55:18.825 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 744 2024-09-20T22:55:18.825 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.825 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:55:18.825 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.825 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:55:18.825 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:54:16.547604+0000", 2024-09-20T22:55:18.825 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:55:18.826 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.826 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.826 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:55:18.826 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:54:16.547604+0000", 2024-09-20T22:55:18.826 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:55:18.826 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.826 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.826 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:55:18.826 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:54:16.547607+0000", 2024-09-20T22:55:18.826 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.7460000000000001e-06 2024-09-20T22:55:18.826 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.826 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.826 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:55:18.826 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:54:16.547620+0000", 2024-09-20T22:55:18.827 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3164e-05 2024-09-20T22:55:18.827 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.827 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.827 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:55:18.827 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:54:16.547647+0000", 2024-09-20T22:55:18.827 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.7679e-05 2024-09-20T22:55:18.827 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.827 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.827 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:55:18.827 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:54:16.547663+0000", 2024-09-20T22:55:18.827 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.523e-05 2024-09-20T22:55:18.827 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.827 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.827 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:55:18.828 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:54:16.547788+0000", 2024-09-20T22:55:18.828 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00012490999999999999 2024-09-20T22:55:18.828 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.828 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.828 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-20T22:55:18.828 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:54:16.547801+0000", 2024-09-20T22:55:18.828 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3158e-05 2024-09-20T22:55:18.828 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.828 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.828 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:55:18.828 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:54:17.337429+0000", 2024-09-20T22:55:18.828 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.78962777299999998 2024-09-20T22:55:18.828 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.828 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.829 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:55:18.829 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:54:17.337522+0000", 2024-09-20T22:55:18.829 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.3556999999999999e-05 2024-09-20T22:55:18.829 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.829 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.829 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:55:18.829 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:54:17.337608+0000", 2024-09-20T22:55:18.829 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.6296000000000002e-05 2024-09-20T22:55:18.829 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:55:18.829 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:55:18.829 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:55:18.829 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.829 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.829 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:7834 7.c e786/94 7:3f0fe232:::smithi08477203-4:head v 786'2534, mlcod=785'2528)", 2024-09-20T22:55:18.829 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:54:26.332738+0000", 2024-09-20T22:55:18.830 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 52.408899896999998, 2024-09-20T22:55:18.830 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.010142463000000001, 2024-09-20T22:55:18.830 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:55:18.830 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:55:18.830 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:55:18.830 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:55:18.830 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.830 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:55:18.830 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:54:26.332738+0000", 2024-09-20T22:55:18.830 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:55:18.830 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.830 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.830 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:55:18.830 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:54:26.332738+0000", 2024-09-20T22:55:18.831 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:55:18.831 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.831 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.831 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:55:18.831 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:54:26.332739+0000", 2024-09-20T22:55:18.831 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2130000000000001e-06 2024-09-20T22:55:18.831 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.831 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.831 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:55:18.831 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:54:26.335206+0000", 2024-09-20T22:55:18.831 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0024668260000000001 2024-09-20T22:55:18.831 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.831 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.831 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:55:18.831 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:54:26.335237+0000", 2024-09-20T22:55:18.832 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.1106999999999999e-05 2024-09-20T22:55:18.832 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.832 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.832 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:55:18.832 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:54:26.335252+0000", 2024-09-20T22:55:18.832 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.5288e-05 2024-09-20T22:55:18.832 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.832 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.832 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:55:18.832 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:54:26.335348+0000", 2024-09-20T22:55:18.832 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.5351000000000002e-05 2024-09-20T22:55:18.832 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.832 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.832 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:55:18.832 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:54:26.335379+0000", 2024-09-20T22:55:18.833 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.1807000000000002e-05 2024-09-20T22:55:18.833 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.833 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.833 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:55:18.833 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:54:26.342798+0000", 2024-09-20T22:55:18.833 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0074187660000000002 2024-09-20T22:55:18.833 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.833 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.833 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:55:18.833 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:54:26.342880+0000", 2024-09-20T22:55:18.833 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.2105000000000001e-05 2024-09-20T22:55:18.833 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:55:18.833 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:55:18.833 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:55:18.833 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.834 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.834 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:8165 7.f e824/391 7:fa55e5ae:::smithi08477203-7:head v 824'927, mlcod=819'923)", 2024-09-20T22:55:18.834 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:55:04.351014+0000", 2024-09-20T22:55:18.834 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 14.390623731, 2024-09-20T22:55:18.834 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0096140429999999992, 2024-09-20T22:55:18.834 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:55:18.834 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:55:18.834 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:55:18.834 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:55:18.834 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.834 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:55:18.834 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:04.351014+0000", 2024-09-20T22:55:18.834 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:55:18.834 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.835 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.835 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:55:18.835 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:04.351014+0000", 2024-09-20T22:55:18.835 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:55:18.835 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.835 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.835 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:55:18.835 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:04.351014+0000", 2024-09-20T22:55:18.835 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9999999999999999e-07 2024-09-20T22:55:18.835 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.835 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.835 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:55:18.835 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:04.354512+0000", 2024-09-20T22:55:18.835 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0034980990000000002 2024-09-20T22:55:18.835 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.836 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.836 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:55:18.836 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:04.354544+0000", 2024-09-20T22:55:18.836 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.1294999999999998e-05 2024-09-20T22:55:18.836 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.836 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.836 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:55:18.836 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:04.354557+0000", 2024-09-20T22:55:18.836 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3825999999999999e-05 2024-09-20T22:55:18.836 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.836 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.836 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:55:18.836 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:04.354685+0000", 2024-09-20T22:55:18.836 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00012776000000000001 2024-09-20T22:55:18.837 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.837 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.837 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:55:18.837 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:04.354717+0000", 2024-09-20T22:55:18.837 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.2054000000000002e-05 2024-09-20T22:55:18.837 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.837 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.837 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:55:18.837 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:04.360567+0000", 2024-09-20T22:55:18.837 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0058492709999999996 2024-09-20T22:55:18.837 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.837 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.837 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:55:18.837 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:04.360628+0000", 2024-09-20T22:55:18.837 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.1538000000000006e-05 2024-09-20T22:55:18.838 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:55:18.838 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:55:18.838 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:55:18.838 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.838 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.838 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:8189 7.c e825/94 7:318c7512:::smithi08477203-44:head v 825'2641, mlcod=825'2638)", 2024-09-20T22:55:18.838 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:55:05.581130+0000", 2024-09-20T22:55:18.838 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 13.160507957, 2024-09-20T22:55:18.838 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0077411670000000002, 2024-09-20T22:55:18.838 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:55:18.838 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:55:18.838 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:55:18.838 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:55:18.839 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.839 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:55:18.839 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:05.581130+0000", 2024-09-20T22:55:18.839 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:55:18.839 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.839 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.839 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:55:18.839 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:05.581130+0000", 2024-09-20T22:55:18.839 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:55:18.839 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.839 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.839 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:55:18.839 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:05.581133+0000", 2024-09-20T22:55:18.840 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.2040000000000002e-06 2024-09-20T22:55:18.840 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.840 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.840 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:55:18.840 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:05.582999+0000", 2024-09-20T22:55:18.840 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0018663919999999999 2024-09-20T22:55:18.840 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.840 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.840 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:55:18.840 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:05.583026+0000", 2024-09-20T22:55:18.840 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.6503999999999999e-05 2024-09-20T22:55:18.840 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.840 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.840 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:55:18.840 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:05.583040+0000", 2024-09-20T22:55:18.841 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4430000000000001e-05 2024-09-20T22:55:18.841 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.841 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.841 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:55:18.841 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:05.583132+0000", 2024-09-20T22:55:18.841 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.1939000000000002e-05 2024-09-20T22:55:18.841 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.841 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.841 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:55:18.841 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:05.583163+0000", 2024-09-20T22:55:18.841 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.0341999999999999e-05 2024-09-20T22:55:18.841 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.841 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.841 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:55:18.841 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:05.588810+0000", 2024-09-20T22:55:18.842 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0056476629999999998 2024-09-20T22:55:18.842 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.842 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.842 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:55:18.842 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:05.588871+0000", 2024-09-20T22:55:18.842 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.0693000000000003e-05 2024-09-20T22:55:18.842 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:55:18.842 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:55:18.842 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:55:18.842 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.842 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.842 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:8293 7.5 7:a160b5d4:::smithi08477203-16:head [sparse-read 0~3042133] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e831)", 2024-09-20T22:55:18.842 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:55:11.340443+0000", 2024-09-20T22:55:18.842 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 7.4011944600000001, 2024-09-20T22:55:18.842 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0070335900000000002, 2024-09-20T22:55:18.843 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:55:18.843 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:55:18.843 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:55:18.843 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:55:18.843 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:55:18.843 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:55:18.843 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 8293 2024-09-20T22:55:18.843 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.843 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:55:18.843 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.843 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:55:18.843 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:11.340443+0000", 2024-09-20T22:55:18.843 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:55:18.844 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.844 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.844 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:55:18.844 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:11.340443+0000", 2024-09-20T22:55:18.844 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:55:18.844 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.844 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.844 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:55:18.844 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:11.340444+0000", 2024-09-20T22:55:18.844 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.6599999999999996e-07 2024-09-20T22:55:18.844 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.844 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.844 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:55:18.844 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:11.340447+0000", 2024-09-20T22:55:18.845 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.9730000000000002e-06 2024-09-20T22:55:18.845 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.845 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.845 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:55:18.845 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:11.340460+0000", 2024-09-20T22:55:18.845 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2864000000000001e-05 2024-09-20T22:55:18.845 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.845 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.845 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:55:18.845 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:11.340466+0000", 2024-09-20T22:55:18.845 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.015e-06 2024-09-20T22:55:18.845 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.845 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.845 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:55:18.846 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:11.340660+0000", 2024-09-20T22:55:18.846 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00019474299999999999 2024-09-20T22:55:18.846 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.846 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.846 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for rw locks", 2024-09-20T22:55:18.846 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:11.340750+0000", 2024-09-20T22:55:18.846 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.9091999999999993e-05 2024-09-20T22:55:18.846 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.846 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.846 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:55:18.846 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:11.347192+0000", 2024-09-20T22:55:18.846 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0064426759999999996 2024-09-20T22:55:18.846 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.847 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.847 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:55:18.847 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:11.347290+0000", 2024-09-20T22:55:18.847 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.7893999999999995e-05 2024-09-20T22:55:18.847 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.847 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.847 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:55:18.847 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:11.347477+0000", 2024-09-20T22:55:18.847 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00018676700000000001 2024-09-20T22:55:18.847 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:55:18.847 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:55:18.847 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:55:18.847 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.847 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.848 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:8294 7.5 7:a160b5d4:::smithi08477203-16:head [read 0~3042133,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e831)", 2024-09-20T22:55:18.848 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:55:11.340475+0000", 2024-09-20T22:55:18.848 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 7.4011631099999997, 2024-09-20T22:55:18.848 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0075077099999999999, 2024-09-20T22:55:18.848 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:55:18.848 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:55:18.848 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:55:18.848 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:55:18.848 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:55:18.848 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:55:18.848 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 8294 2024-09-20T22:55:18.848 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.848 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:55:18.848 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.849 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:55:18.849 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:11.340475+0000", 2024-09-20T22:55:18.849 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:55:18.849 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.849 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.849 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:55:18.849 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:11.340475+0000", 2024-09-20T22:55:18.849 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:55:18.849 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.849 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.849 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:55:18.849 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:11.340475+0000", 2024-09-20T22:55:18.849 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.1099999999999996e-07 2024-09-20T22:55:18.849 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.849 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.850 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:55:18.850 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:11.340479+0000", 2024-09-20T22:55:18.850 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.4170000000000001e-06 2024-09-20T22:55:18.850 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.850 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.850 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:55:18.850 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:11.340484+0000", 2024-09-20T22:55:18.850 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.7570000000000001e-06 2024-09-20T22:55:18.850 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.850 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.850 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:55:18.850 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:11.340489+0000", 2024-09-20T22:55:18.850 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.3819999999999997e-06 2024-09-20T22:55:18.850 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.851 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.851 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:55:18.851 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:11.340800+0000", 2024-09-20T22:55:18.851 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00031126299999999999 2024-09-20T22:55:18.851 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.851 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.851 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for rw locks", 2024-09-20T22:55:18.851 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:11.340908+0000", 2024-09-20T22:55:18.851 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000108088 2024-09-20T22:55:18.851 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.851 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.851 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:55:18.851 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:11.347505+0000", 2024-09-20T22:55:18.851 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0065973050000000004 2024-09-20T22:55:18.851 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.852 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.852 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:55:18.852 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:11.347565+0000", 2024-09-20T22:55:18.852 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.9567000000000001e-05 2024-09-20T22:55:18.852 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:55:18.852 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:55:18.852 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:55:18.852 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:11.347982+0000", 2024-09-20T22:55:18.852 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00041742 2024-09-20T22:55:18.852 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:55:18.852 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:55:18.852 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:55:18.852 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:55:18.852 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:55:18.853 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-20T22:55:18.853 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:55:18.875 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:18.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:18.875 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:55:18.977 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:55:18.999 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:19.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:19.001 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:55:19.102 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:55:19.123 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:19.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:19.124 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:55:19.225 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:55:19.247 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:19.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:19.248 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:55:19.327 INFO:tasks.rados.rados.0.smithi084.stdout:3353: done (2 left) 2024-09-20T22:55:19.327 INFO:tasks.rados.rados.0.smithi084.stdout:3354: done (1 left) 2024-09-20T22:55:19.327 INFO:tasks.rados.rados.0.smithi084.stdout:3355: done (0 left) 2024-09-20T22:55:19.327 INFO:tasks.rados.rados.0.smithi084.stdout:3356: snap_create 2024-09-20T22:55:19.348 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:55:19.370 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:19.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:19.371 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:55:19.471 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:55:19.493 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:19.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:19.494 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:55:19.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:19 smithi154 ceph-mon[33264]: pgmap v1158: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 550 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail 2024-09-20T22:55:19.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:19 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/2906106817' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:55:19.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:19 smithi154 ceph-mon[33264]: osdmap e839: 8 total, 8 up, 8 in 2024-09-20T22:55:19.595 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:55:19.614 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:19 smithi084 ceph-mon[40024]: pgmap v1158: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 550 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail 2024-09-20T22:55:19.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:19 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/2906106817' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:55:19.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:19 smithi084 ceph-mon[40024]: osdmap e839: 8 total, 8 up, 8 in 2024-09-20T22:55:19.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:19 smithi084 ceph-mon[29774]: pgmap v1158: 158 pgs: 1 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 152 active+clean; 550 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail 2024-09-20T22:55:19.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:19 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/2906106817' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:55:19.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:19 smithi084 ceph-mon[29774]: osdmap e839: 8 total, 8 up, 8 in 2024-09-20T22:55:19.616 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:19.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:19.617 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:55:19.718 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:55:19.739 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:19.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:19.740 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:55:19.815 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-20T22:55:19.841 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:55:19.879 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:19.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:19.880 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:55:19.980 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:55:20.004 INFO:journalctl@ceph.iscsi.iscsi.a.smithi154.stdout:Sep 20 22:55:19 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-iscsi-iscsi-a[59730]: debug there is no tcmu-runner data available 2024-09-20T22:55:20.004 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:20.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:20.005 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:55:20.061 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:55:20.106 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:55:20.127 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:20.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:20.128 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:55:20.229 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:55:20.237 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd unset noscrub 2024-09-20T22:55:20.254 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:20.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:20.255 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:55:20.330 INFO:tasks.rados.rados.0.smithi084.stdout:3356: done (0 left) 2024-09-20T22:55:20.331 INFO:tasks.rados.rados.0.smithi084.stdout:3357: read oid 21 snap 333 2024-09-20T22:55:20.331 INFO:tasks.rados.rados.0.smithi084.stdout:3357: expect (ObjNum 984 snap 319 seq_num 984) 2024-09-20T22:55:20.331 INFO:tasks.rados.rados.0.smithi084.stdout:3358: snap_remove snap 363 2024-09-20T22:55:20.355 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:55:20.377 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:20.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:20.378 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:55:20.453 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:20 smithi084 ceph-mon[40024]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:55:20.454 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:20 smithi084 ceph-mon[40024]: osdmap e840: 8 total, 8 up, 8 in 2024-09-20T22:55:20.454 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:20 smithi084 ceph-mon[29774]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:55:20.454 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:20 smithi084 ceph-mon[29774]: osdmap e840: 8 total, 8 up, 8 in 2024-09-20T22:55:20.479 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:55:20.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:20.503 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:20.503 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:55:20.556 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-20T22:55:20.556 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2024-09-20T22:55:20.556 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2024-09-20T22:55:20.556 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-20T22:55:20.604 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:55:20.631 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:20.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:20.632 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:55:20.733 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:55:20.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:20 smithi154 ceph-mon[33264]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:55:20.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:20 smithi154 ceph-mon[33264]: osdmap e840: 8 total, 8 up, 8 in 2024-09-20T22:55:20.758 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:20.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:20.759 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:55:20.860 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:55:20.884 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:20.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:20.884 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:55:20.985 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:55:21.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:21.013 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:21.013 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:55:21.115 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:55:21.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:21.142 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:21.142 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:55:21.243 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:55:21.266 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:21.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:21.267 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:55:21.293 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.0 dump_blocked_ops 2024-09-20T22:55:21.334 INFO:tasks.rados.rados.0.smithi084.stdout:3358: done (1 left) 2024-09-20T22:55:21.334 INFO:tasks.rados.rados.0.smithi084.stdout:3359: snap_remove snap 374 2024-09-20T22:55:21.341 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:55:21.367 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:55:21.378 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-20T22:55:21.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:21.390 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:21.390 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:55:21.464 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:21 smithi084 ceph-mon[29774]: purged_snaps scrub starts 2024-09-20T22:55:21.464 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:21 smithi084 ceph-mon[29774]: purged_snaps scrub ok 2024-09-20T22:55:21.464 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:21 smithi084 ceph-mon[29774]: pgmap v1161: 158 pgs: 2 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 152 active+clean; 546 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 9.2 MiB/s rd, 3.4 MiB/s wr, 16 op/s 2024-09-20T22:55:21.465 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:21 smithi084 ceph-mon[29774]: osdmap e841: 8 total, 8 up, 8 in 2024-09-20T22:55:21.465 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:21 smithi084 ceph-mon[40024]: purged_snaps scrub starts 2024-09-20T22:55:21.465 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:21 smithi084 ceph-mon[40024]: purged_snaps scrub ok 2024-09-20T22:55:21.465 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:21 smithi084 ceph-mon[40024]: pgmap v1161: 158 pgs: 2 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 152 active+clean; 546 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 9.2 MiB/s rd, 3.4 MiB/s wr, 16 op/s 2024-09-20T22:55:21.465 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:21 smithi084 ceph-mon[40024]: osdmap e841: 8 total, 8 up, 8 in 2024-09-20T22:55:21.492 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:55:21.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:21 smithi154 ceph-mon[33264]: purged_snaps scrub starts 2024-09-20T22:55:21.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:21 smithi154 ceph-mon[33264]: purged_snaps scrub ok 2024-09-20T22:55:21.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:21 smithi154 ceph-mon[33264]: pgmap v1161: 158 pgs: 2 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 152 active+clean; 546 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 9.2 MiB/s rd, 3.4 MiB/s wr, 16 op/s 2024-09-20T22:55:21.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:21 smithi154 ceph-mon[33264]: osdmap e841: 8 total, 8 up, 8 in 2024-09-20T22:55:21.514 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:21.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:21.516 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:55:21.617 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:55:21.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:21.639 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:21.639 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:55:21.740 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:55:21.763 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:21.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:21.764 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:55:21.865 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:55:21.886 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:21.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:21.887 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:55:21.894 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:55:21.895 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:55:21.988 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:55:22.010 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:22.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:22.011 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:55:22.032 INFO:tasks.thrashosds.thrasher:in_osds: [2, 3, 5, 6, 7, 0, 4, 1] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-20T22:55:22.032 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-20T22:55:22.032 INFO:tasks.thrashosds.thrasher:inject_pause on osd.1 2024-09-20T22:55:22.033 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-20T22:55:22.033 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-20T22:55:22.033 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.1 config set filestore_inject_stall 3 2024-09-20T22:55:22.111 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:55:22.133 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:22.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:22.134 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:55:22.235 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:55:22.256 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:22.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:22.257 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:55:22.327 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is unset 2024-09-20T22:55:22.332 INFO:tasks.rados.rados.0.smithi084.stdout:3359: done (1 left) 2024-09-20T22:55:22.332 INFO:tasks.rados.rados.0.smithi084.stdout:3360: snap_create 2024-09-20T22:55:22.332 INFO:tasks.rados.rados.0.smithi084.stdout:3357: expect (ObjNum 984 snap 319 seq_num 984) 2024-09-20T22:55:22.358 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:55:22.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:22.387 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:22.387 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:55:22.488 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:55:22.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:22.510 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:22.510 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:55:22.589 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:22 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/3042583976' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:55:22.589 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:22 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/3042583976' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T22:55:22.589 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:22 smithi084 ceph-mon[40024]: osdmap e842: 8 total, 8 up, 8 in 2024-09-20T22:55:22.589 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:22 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/3042583976' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:55:22.589 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:22 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/3042583976' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T22:55:22.589 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:22 smithi084 ceph-mon[29774]: osdmap e842: 8 total, 8 up, 8 in 2024-09-20T22:55:22.612 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:55:22.623 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:55:22.623 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:55:22.623 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:55:22.623 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:55:22.624 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:55:22.624 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:55:22.633 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:22.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:22.634 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:55:22.639 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-20T22:55:22.640 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T22:55:22.656 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:55:22.657 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:55:22.673 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:55:22.673 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:55:22.690 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:55:22.690 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:55:22.706 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:55:22.706 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:55:22.735 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:55:22.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:22 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/3042583976' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:55:22.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:22 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/3042583976' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T22:55:22.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:22 smithi154 ceph-mon[33264]: osdmap e842: 8 total, 8 up, 8 in 2024-09-20T22:55:22.755 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:22.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:22.756 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:55:22.857 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:55:22.874 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-20T22:55:22.874 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:55:22.880 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:22.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:22.881 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:55:22.981 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:55:23.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:23.004 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:23.005 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:55:23.106 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:55:23.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:23.128 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:23.128 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:55:23.229 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:55:23.249 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:23.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:23.250 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:55:23.330 INFO:tasks.rados.rados.0.smithi084.stdout:3357: done (1 left) 2024-09-20T22:55:23.330 INFO:tasks.rados.rados.0.smithi084.stdout:3360: done (0 left) 2024-09-20T22:55:23.330 INFO:tasks.rados.rados.0.smithi084.stdout:3361: read oid 39 snap -1 2024-09-20T22:55:23.330 INFO:tasks.rados.rados.0.smithi084.stdout:3361: expect (ObjNum 1120 snap 380 seq_num 1120) 2024-09-20T22:55:23.330 INFO:tasks.rados.rados.0.smithi084.stdout:3362: copy_from oid 42 from oid 4 current snap is 389 2024-09-20T22:55:23.330 INFO:tasks.rados.rados.0.smithi084.stdout:3363: write (excl) oid 16 current snap is 389 2024-09-20T22:55:23.330 INFO:tasks.rados.rados.0.smithi084.stdout:3363: seq_num 1143 ranges {410401=662557,1719036=770066,2938716=271685} 2024-09-20T22:55:23.336 INFO:tasks.rados.rados.0.smithi084.stdout:3363: writing smithi08477203-16 from 410401 to 1072958 tid 1 2024-09-20T22:55:23.345 INFO:tasks.rados.rados.0.smithi084.stdout:3363: writing smithi08477203-16 from 1719036 to 2489102 tid 2 2024-09-20T22:55:23.350 INFO:tasks.rados.rados.0.smithi084.stdout:3363: writing smithi08477203-16 from 2938716 to 3210401 tid 3 2024-09-20T22:55:23.350 INFO:tasks.rados.rados.0.smithi084.stdout:3364: snap_create 2024-09-20T22:55:23.351 INFO:tasks.rados.rados.0.smithi084.stdout:3361: expect (ObjNum 1120 snap 380 seq_num 1120) 2024-09-20T22:55:23.351 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:55:23.373 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:23.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:23.374 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:55:23.446 INFO:tasks.rados.rados.0.smithi084.stdout:3362: finishing copy_from racing read to smithi08477203-42 2024-09-20T22:55:23.446 INFO:tasks.rados.rados.0.smithi084.stdout:3363: finishing write tid 1 to smithi08477203-16 2024-09-20T22:55:23.446 INFO:tasks.rados.rados.0.smithi084.stdout:3363: finishing write tid 2 to smithi08477203-16 2024-09-20T22:55:23.446 INFO:tasks.rados.rados.0.smithi084.stdout:3362: finishing copy_from to smithi08477203-42 2024-09-20T22:55:23.446 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 42 v 2639 (ObjNum 1111 snap 378 seq_num 1111) dirty exists 2024-09-20T22:55:23.446 INFO:tasks.rados.rados.0.smithi084.stdout:3363: finishing write tid 3 to smithi08477203-16 2024-09-20T22:55:23.446 INFO:tasks.rados.rados.0.smithi084.stdout:3363: finishing write tid 4 to smithi08477203-16 2024-09-20T22:55:23.446 INFO:tasks.rados.rados.0.smithi084.stdout:3363: finishing write tid 5 to smithi08477203-16 2024-09-20T22:55:23.446 INFO:tasks.rados.rados.0.smithi084.stdout:3363: oid 16 updating version 0 to 1823 2024-09-20T22:55:23.446 INFO:tasks.rados.rados.0.smithi084.stdout:3363: oid 16 version 1823 is already newer than 1822 2024-09-20T22:55:23.446 INFO:tasks.rados.rados.0.smithi084.stdout:3363: oid 16 updating version 1823 to 1824 2024-09-20T22:55:23.447 INFO:tasks.rados.rados.0.smithi084.stdout:3363: oid 16 version 1824 is already newer than 1821 2024-09-20T22:55:23.447 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 16 v 1824 (ObjNum 1143 snap 389 seq_num 1143) dirty exists 2024-09-20T22:55:23.447 INFO:tasks.rados.rados.0.smithi084.stdout:3363: left oid 16 (ObjNum 1143 snap 389 seq_num 1143) 2024-09-20T22:55:23.474 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:55:23.496 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:23.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:23.497 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:55:23.540 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:23 smithi084 ceph-mon[40024]: pgmap v1164: 158 pgs: 2 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 152 active+clean; 546 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 9.2 MiB/s rd, 3.4 MiB/s wr, 16 op/s 2024-09-20T22:55:23.540 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:23 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/182344111' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:55:23.540 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:23 smithi084 ceph-mon[40024]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:55:23.540 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:23 smithi084 ceph-mon[40024]: osdmap e843: 8 total, 8 up, 8 in 2024-09-20T22:55:23.540 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:23 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:55:23.540 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:23 smithi084 ceph-mon[29774]: pgmap v1164: 158 pgs: 2 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 152 active+clean; 546 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 9.2 MiB/s rd, 3.4 MiB/s wr, 16 op/s 2024-09-20T22:55:23.540 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:23 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/182344111' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:55:23.540 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:23 smithi084 ceph-mon[29774]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:55:23.541 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:23 smithi084 ceph-mon[29774]: osdmap e843: 8 total, 8 up, 8 in 2024-09-20T22:55:23.541 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:23 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:55:23.597 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:55:23.618 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:23.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:23.619 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:55:23.720 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:55:23.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:23.742 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:23.742 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:55:23.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:23 smithi154 ceph-mon[33264]: pgmap v1164: 158 pgs: 2 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 152 active+clean; 546 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 9.2 MiB/s rd, 3.4 MiB/s wr, 16 op/s 2024-09-20T22:55:23.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:23 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/182344111' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:55:23.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:23 smithi154 ceph-mon[33264]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:55:23.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:23 smithi154 ceph-mon[33264]: osdmap e843: 8 total, 8 up, 8 in 2024-09-20T22:55:23.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:23 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:55:23.844 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:55:23.865 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:23.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:23.866 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:55:23.967 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:55:23.988 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:23.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:23.989 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:55:24.090 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:55:24.112 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:24.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:24.113 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:55:24.214 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:55:24.236 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:24.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:24.237 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:55:24.338 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:55:24.340 INFO:tasks.rados.rados.0.smithi084.stdout:3361: done (3 left) 2024-09-20T22:55:24.340 INFO:tasks.rados.rados.0.smithi084.stdout:3362: done (2 left) 2024-09-20T22:55:24.340 INFO:tasks.rados.rados.0.smithi084.stdout:3363: done (1 left) 2024-09-20T22:55:24.340 INFO:tasks.rados.rados.0.smithi084.stdout:3364: done (0 left) 2024-09-20T22:55:24.340 INFO:tasks.rados.rados.0.smithi084.stdout:3365: snap_remove snap 348 2024-09-20T22:55:24.360 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:24.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:24.361 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:55:24.461 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:55:24.485 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:24.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:24.485 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:55:24.586 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:55:24.608 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:24.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:24.609 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:55:24.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:24 smithi084 ceph-mon[40024]: 3.14 scrub starts 2024-09-20T22:55:24.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:24 smithi084 ceph-mon[40024]: 3.14 scrub ok 2024-09-20T22:55:24.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:24 smithi084 ceph-mon[40024]: 4.14 scrub starts 2024-09-20T22:55:24.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:24 smithi084 ceph-mon[40024]: 4.14 scrub ok 2024-09-20T22:55:24.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:24 smithi084 ceph-mon[40024]: 6.1d scrub starts 2024-09-20T22:55:24.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:24 smithi084 ceph-mon[40024]: 6.1d scrub ok 2024-09-20T22:55:24.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:24 smithi084 ceph-mon[40024]: osdmap e844: 8 total, 8 up, 8 in 2024-09-20T22:55:24.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:24 smithi084 ceph-mon[29774]: 3.14 scrub starts 2024-09-20T22:55:24.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:24 smithi084 ceph-mon[29774]: 3.14 scrub ok 2024-09-20T22:55:24.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:24 smithi084 ceph-mon[29774]: 4.14 scrub starts 2024-09-20T22:55:24.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:24 smithi084 ceph-mon[29774]: 4.14 scrub ok 2024-09-20T22:55:24.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:24 smithi084 ceph-mon[29774]: 6.1d scrub starts 2024-09-20T22:55:24.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:24 smithi084 ceph-mon[29774]: 6.1d scrub ok 2024-09-20T22:55:24.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:24 smithi084 ceph-mon[29774]: osdmap e844: 8 total, 8 up, 8 in 2024-09-20T22:55:24.710 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:55:24.731 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:24.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:24.732 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:55:24.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:24 smithi154 ceph-mon[33264]: 3.14 scrub starts 2024-09-20T22:55:24.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:24 smithi154 ceph-mon[33264]: 3.14 scrub ok 2024-09-20T22:55:24.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:24 smithi154 ceph-mon[33264]: 4.14 scrub starts 2024-09-20T22:55:24.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:24 smithi154 ceph-mon[33264]: 4.14 scrub ok 2024-09-20T22:55:24.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:24 smithi154 ceph-mon[33264]: 6.1d scrub starts 2024-09-20T22:55:24.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:24 smithi154 ceph-mon[33264]: 6.1d scrub ok 2024-09-20T22:55:24.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:24 smithi154 ceph-mon[33264]: osdmap e844: 8 total, 8 up, 8 in 2024-09-20T22:55:24.833 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:55:24.856 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:24.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:24.857 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:55:24.952 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.0 dump_historic_ops 2024-09-20T22:55:24.957 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:55:24.979 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:55:25.000 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:25.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:25.001 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:55:25.102 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:55:25.124 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:25.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:25.125 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:55:25.226 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:55:25.248 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:25.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:25.249 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:55:25.339 INFO:tasks.rados.rados.0.smithi084.stdout:3365: done (0 left) 2024-09-20T22:55:25.340 INFO:tasks.rados.rados.0.smithi084.stdout:3366: delete oid 12 current snap is 390 2024-09-20T22:55:25.349 INFO:tasks.rados.rados.0.smithi084.stdout:3366: done (0 left) 2024-09-20T22:55:25.349 INFO:tasks.rados.rados.0.smithi084.stdout:3367: write (excl) oid 40 current snap is 390 2024-09-20T22:55:25.349 INFO:tasks.rados.rados.0.smithi084.stdout:3367: seq_num 1145 ranges {444015=413055,1436000=773735,2709915=534100} 2024-09-20T22:55:25.350 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:55:25.354 INFO:tasks.rados.rados.0.smithi084.stdout:3367: writing smithi08477203-40 from 444015 to 857070 tid 1 2024-09-20T22:55:25.362 INFO:tasks.rados.rados.0.smithi084.stdout:3367: writing smithi08477203-40 from 1436000 to 2209735 tid 2 2024-09-20T22:55:25.368 INFO:tasks.rados.rados.0.smithi084.stdout:3367: writing smithi08477203-40 from 2709915 to 3244015 tid 3 2024-09-20T22:55:25.368 INFO:tasks.rados.rados.0.smithi084.stdout:3368: read oid 42 snap 378 2024-09-20T22:55:25.368 INFO:tasks.rados.rados.0.smithi084.stdout:3368: expect (ObjNum 1104 snap 374 seq_num 1104) 2024-09-20T22:55:25.369 INFO:tasks.rados.rados.0.smithi084.stdout:3369: read oid 35 snap -1 2024-09-20T22:55:25.369 INFO:tasks.rados.rados.0.smithi084.stdout:3369: expect (ObjNum 1121 snap 380 seq_num 1121) 2024-09-20T22:55:25.369 INFO:tasks.rados.rados.0.smithi084.stdout:3370: copy_from oid 31 from oid 16 current snap is 390 2024-09-20T22:55:25.369 INFO:tasks.rados.rados.0.smithi084.stdout:3367: finishing write tid 1 to smithi08477203-40 2024-09-20T22:55:25.369 INFO:tasks.rados.rados.0.smithi084.stdout:3367: finishing write tid 2 to smithi08477203-40 2024-09-20T22:55:25.369 INFO:tasks.rados.rados.0.smithi084.stdout:3371: rollback oid 10 current snap is 390 2024-09-20T22:55:25.369 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 10 to 350 2024-09-20T22:55:25.369 INFO:tasks.rados.rados.0.smithi084.stdout:3372: copy_from oid 37 from oid 9 current snap is 390 2024-09-20T22:55:25.369 INFO:tasks.rados.rados.0.smithi084.stdout:3373: copy_from oid 3 from oid 1 current snap is 390 2024-09-20T22:55:25.369 INFO:tasks.rados.rados.0.smithi084.stdout:3374: snap_create 2024-09-20T22:55:25.370 INFO:tasks.rados.rados.0.smithi084.stdout:3372: finishing copy_from to smithi08477203-37 2024-09-20T22:55:25.370 INFO:tasks.rados.rados.0.smithi084.stdout:3372: got expected ENOENT (src dne) 2024-09-20T22:55:25.371 INFO:tasks.rados.rados.0.smithi084.stdout:3372: finishing copy_from racing read to smithi08477203-37 2024-09-20T22:55:25.371 INFO:tasks.rados.rados.0.smithi084.stdout:3367: finishing write tid 3 to smithi08477203-40 2024-09-20T22:55:25.372 INFO:tasks.rados.rados.0.smithi084.stdout:3367: finishing write tid 4 to smithi08477203-40 2024-09-20T22:55:25.372 INFO:tasks.rados.rados.0.smithi084.stdout:3367: finishing write tid 5 to smithi08477203-40 2024-09-20T22:55:25.372 INFO:tasks.rados.rados.0.smithi084.stdout:3367: oid 40 updating version 0 to 1927 2024-09-20T22:55:25.372 INFO:tasks.rados.rados.0.smithi084.stdout:3367: oid 40 version 1927 is already newer than 1926 2024-09-20T22:55:25.372 INFO:tasks.rados.rados.0.smithi084.stdout:3367: oid 40 version 1927 is already newer than 1925 2024-09-20T22:55:25.372 INFO:tasks.rados.rados.0.smithi084.stdout:3367: oid 40 updating version 1927 to 1928 2024-09-20T22:55:25.372 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 40 v 1928 (ObjNum 1145 snap 390 seq_num 1145) dirty exists 2024-09-20T22:55:25.372 INFO:tasks.rados.rados.0.smithi084.stdout:3367: left oid 40 (ObjNum 1145 snap 390 seq_num 1145) 2024-09-20T22:55:25.373 INFO:tasks.rados.rados.0.smithi084.stdout:3371: finishing rollback tid 1 to smithi08477203-10 2024-09-20T22:55:25.374 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 10 v 1826 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-09-20T22:55:25.374 INFO:tasks.rados.rados.0.smithi084.stdout:3369: expect (ObjNum 1121 snap 380 seq_num 1121) 2024-09-20T22:55:25.375 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:25.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:25.377 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:55:25.478 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:55:25.485 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-20T22:55:25.485 INFO:teuthology.orchestra.run.smithi084.stdout: "success": "" 2024-09-20T22:55:25.485 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-20T22:55:25.504 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:25.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:25.505 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:55:25.568 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-20T22:55:25.573 INFO:tasks.rados.rados.0.smithi084.stdout:3373: finishing copy_from racing read to smithi08477203-3 2024-09-20T22:55:25.573 INFO:tasks.rados.rados.0.smithi084.stdout:3373: finishing copy_from to smithi08477203-3 2024-09-20T22:55:25.573 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 3 v 1834 (ObjNum 738 snap 235 seq_num 738) dirty exists 2024-09-20T22:55:25.573 INFO:tasks.rados.rados.0.smithi084.stdout:3368: expect (ObjNum 1104 snap 374 seq_num 1104) 2024-09-20T22:55:25.590 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:25 smithi084 ceph-mon[40024]: pgmap v1167: 158 pgs: 4 active+clean+snaptrim, 154 active+clean; 548 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 5.9 MiB/s rd, 1.0 MiB/s wr, 7 op/s 2024-09-20T22:55:25.590 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:25 smithi084 ceph-mon[40024]: 6.3 scrub starts 2024-09-20T22:55:25.590 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:25 smithi084 ceph-mon[40024]: 6.3 scrub ok 2024-09-20T22:55:25.591 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:25 smithi084 ceph-mon[40024]: osdmap e845: 8 total, 8 up, 8 in 2024-09-20T22:55:25.591 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:25 smithi084 ceph-mon[29774]: pgmap v1167: 158 pgs: 4 active+clean+snaptrim, 154 active+clean; 548 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 5.9 MiB/s rd, 1.0 MiB/s wr, 7 op/s 2024-09-20T22:55:25.591 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:25 smithi084 ceph-mon[29774]: 6.3 scrub starts 2024-09-20T22:55:25.591 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:25 smithi084 ceph-mon[29774]: 6.3 scrub ok 2024-09-20T22:55:25.591 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:25 smithi084 ceph-mon[29774]: osdmap e845: 8 total, 8 up, 8 in 2024-09-20T22:55:25.606 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:55:25.628 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:25.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:25.629 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:55:25.710 INFO:tasks.rados.rados.0.smithi084.stdout:3370: finishing copy_from racing read to smithi08477203-31 2024-09-20T22:55:25.710 INFO:tasks.rados.rados.0.smithi084.stdout:3370: finishing copy_from to smithi08477203-31 2024-09-20T22:55:25.710 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 31 v 1824 (ObjNum 1143 snap 389 seq_num 1143) dirty exists 2024-09-20T22:55:25.730 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:55:25.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:25 smithi154 ceph-mon[33264]: pgmap v1167: 158 pgs: 4 active+clean+snaptrim, 154 active+clean; 548 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 5.9 MiB/s rd, 1.0 MiB/s wr, 7 op/s 2024-09-20T22:55:25.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:25 smithi154 ceph-mon[33264]: 6.3 scrub starts 2024-09-20T22:55:25.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:25 smithi154 ceph-mon[33264]: 6.3 scrub ok 2024-09-20T22:55:25.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:25 smithi154 ceph-mon[33264]: osdmap e845: 8 total, 8 up, 8 in 2024-09-20T22:55:25.755 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:25.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:25.756 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:55:25.856 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:55:25.880 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:25.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:25.881 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:55:25.982 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:55:26.004 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:26.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:26.005 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:55:26.105 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:55:26.126 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:26.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:26.127 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:55:26.156 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:55:26.229 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:55:26.254 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:26.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:26.255 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:55:26.337 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-20T22:55:26.340 INFO:tasks.rados.rados.0.smithi084.stdout:3367: done (7 left) 2024-09-20T22:55:26.340 INFO:tasks.rados.rados.0.smithi084.stdout:3368: done (6 left) 2024-09-20T22:55:26.341 INFO:tasks.rados.rados.0.smithi084.stdout:3369: done (5 left) 2024-09-20T22:55:26.341 INFO:tasks.rados.rados.0.smithi084.stdout:3370: done (4 left) 2024-09-20T22:55:26.341 INFO:tasks.rados.rados.0.smithi084.stdout:3371: done (3 left) 2024-09-20T22:55:26.341 INFO:tasks.rados.rados.0.smithi084.stdout:3372: done (2 left) 2024-09-20T22:55:26.341 INFO:tasks.rados.rados.0.smithi084.stdout:3373: done (1 left) 2024-09-20T22:55:26.341 INFO:tasks.rados.rados.0.smithi084.stdout:3374: done (0 left) 2024-09-20T22:55:26.341 INFO:tasks.rados.rados.0.smithi084.stdout:3375: snap_create 2024-09-20T22:55:26.356 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:55:26.404 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:26.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:26.405 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:55:26.427 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:26 smithi084 ceph-mon[29774]: 3.3 scrub starts 2024-09-20T22:55:26.427 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:26 smithi084 ceph-mon[29774]: 3.3 scrub ok 2024-09-20T22:55:26.427 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:26 smithi084 ceph-mon[29774]: 5.18 scrub starts 2024-09-20T22:55:26.427 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:26 smithi084 ceph-mon[29774]: 5.18 scrub ok 2024-09-20T22:55:26.427 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:26 smithi084 ceph-mon[29774]: 4.9 scrub starts 2024-09-20T22:55:26.427 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:26 smithi084 ceph-mon[29774]: 4.9 scrub ok 2024-09-20T22:55:26.428 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:26 smithi084 ceph-mon[29774]: osdmap e846: 8 total, 8 up, 8 in 2024-09-20T22:55:26.428 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:26 smithi084 ceph-mon[40024]: 3.3 scrub starts 2024-09-20T22:55:26.428 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:26 smithi084 ceph-mon[40024]: 3.3 scrub ok 2024-09-20T22:55:26.428 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:26 smithi084 ceph-mon[40024]: 5.18 scrub starts 2024-09-20T22:55:26.428 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:26 smithi084 ceph-mon[40024]: 5.18 scrub ok 2024-09-20T22:55:26.428 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:26 smithi084 ceph-mon[40024]: 4.9 scrub starts 2024-09-20T22:55:26.428 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:26 smithi084 ceph-mon[40024]: 4.9 scrub ok 2024-09-20T22:55:26.428 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:26 smithi084 ceph-mon[40024]: osdmap e846: 8 total, 8 up, 8 in 2024-09-20T22:55:26.506 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:55:26.528 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:26.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:26.529 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:55:26.630 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:55:26.651 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:26.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:26.652 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:55:26.686 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-20T22:55:26.686 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:55:26.753 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:55:26.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:26 smithi154 ceph-mon[33264]: 3.3 scrub starts 2024-09-20T22:55:26.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:26 smithi154 ceph-mon[33264]: 3.3 scrub ok 2024-09-20T22:55:26.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:26 smithi154 ceph-mon[33264]: 5.18 scrub starts 2024-09-20T22:55:26.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:26 smithi154 ceph-mon[33264]: 5.18 scrub ok 2024-09-20T22:55:26.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:26 smithi154 ceph-mon[33264]: 4.9 scrub starts 2024-09-20T22:55:26.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:26 smithi154 ceph-mon[33264]: 4.9 scrub ok 2024-09-20T22:55:26.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:26 smithi154 ceph-mon[33264]: osdmap e846: 8 total, 8 up, 8 in 2024-09-20T22:55:26.776 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:26.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:26.777 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:55:26.877 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:55:26.898 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:26.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:26.899 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:55:27.000 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:55:27.026 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:27.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:27.027 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:55:27.127 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:55:27.152 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:27.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:27.153 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:55:27.254 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:55:27.279 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:27.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:27.279 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:55:27.341 INFO:tasks.rados.rados.0.smithi084.stdout:3375: done (0 left) 2024-09-20T22:55:27.341 INFO:tasks.rados.rados.0.smithi084.stdout:3376: write (excl) oid 47 current snap is 392 2024-09-20T22:55:27.341 INFO:tasks.rados.rados.0.smithi084.stdout:3376: seq_num 1146 ranges {460822=488304,1378129=517393,2504632=405962} 2024-09-20T22:55:27.346 INFO:tasks.rados.rados.0.smithi084.stdout:3376: writing smithi08477203-47 from 460822 to 949126 tid 1 2024-09-20T22:55:27.352 INFO:tasks.rados.rados.0.smithi084.stdout:3376: writing smithi08477203-47 from 1378129 to 1895522 tid 2 2024-09-20T22:55:27.358 INFO:tasks.rados.rados.0.smithi084.stdout:3376: writing smithi08477203-47 from 2504632 to 2910594 tid 3 2024-09-20T22:55:27.358 INFO:tasks.rados.rados.0.smithi084.stdout:3377: copy_from oid 1 from oid 42 current snap is 392 2024-09-20T22:55:27.358 INFO:tasks.rados.rados.0.smithi084.stdout:3376: finishing write tid 1 to smithi08477203-47 2024-09-20T22:55:27.359 INFO:tasks.rados.rados.0.smithi084.stdout:3376: finishing write tid 2 to smithi08477203-47 2024-09-20T22:55:27.359 INFO:tasks.rados.rados.0.smithi084.stdout:3378: write (excl) oid 17 current snap is 392 2024-09-20T22:55:27.359 INFO:tasks.rados.rados.0.smithi084.stdout:3378: seq_num 1147 ranges {477629=563553,1836611=777404,3048408=229221} 2024-09-20T22:55:27.364 INFO:tasks.rados.rados.0.smithi084.stdout:3378: writing smithi08477203-17 from 477629 to 1041182 tid 1 2024-09-20T22:55:27.375 INFO:tasks.rados.rados.0.smithi084.stdout:3378: writing smithi08477203-17 from 1836611 to 2614015 tid 2 2024-09-20T22:55:27.380 INFO:tasks.rados.rados.0.smithi084.stdout:3378: writing smithi08477203-17 from 3048408 to 3277629 tid 3 2024-09-20T22:55:27.380 INFO:tasks.rados.rados.0.smithi084.stdout:3379: snap_create 2024-09-20T22:55:27.380 INFO:tasks.rados.rados.0.smithi084.stdout:3376: finishing write tid 3 to smithi08477203-47 2024-09-20T22:55:27.380 INFO:tasks.rados.rados.0.smithi084.stdout:3376: finishing write tid 4 to smithi08477203-47 2024-09-20T22:55:27.380 INFO:tasks.rados.rados.0.smithi084.stdout:3376: finishing write tid 5 to smithi08477203-47 2024-09-20T22:55:27.380 INFO:tasks.rados.rados.0.smithi084.stdout:3376: oid 47 updating version 0 to 2481 2024-09-20T22:55:27.380 INFO:tasks.rados.rados.0.smithi084.stdout:3376: oid 47 version 2481 is already newer than 2480 2024-09-20T22:55:27.380 INFO:tasks.rados.rados.0.smithi084.stdout:3376: oid 47 version 2481 is already newer than 2479 2024-09-20T22:55:27.380 INFO:tasks.rados.rados.0.smithi084.stdout:3376: oid 47 version 2481 is already newer than 2478 2024-09-20T22:55:27.380 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 47 v 2481 (ObjNum 1146 snap 392 seq_num 1146) dirty exists 2024-09-20T22:55:27.381 INFO:tasks.rados.rados.0.smithi084.stdout:3376: left oid 47 (ObjNum 1146 snap 392 seq_num 1146) 2024-09-20T22:55:27.381 INFO:tasks.rados.rados.0.smithi084.stdout:3377: finishing copy_from racing read to smithi08477203-1 2024-09-20T22:55:27.381 INFO:tasks.rados.rados.0.smithi084.stdout:3378: finishing write tid 1 to smithi08477203-17 2024-09-20T22:55:27.381 INFO:tasks.rados.rados.0.smithi084.stdout:3377: finishing copy_from to smithi08477203-1 2024-09-20T22:55:27.381 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 1 v 2639 (ObjNum 1111 snap 378 seq_num 1111) dirty exists 2024-09-20T22:55:27.381 INFO:tasks.rados.rados.0.smithi084.stdout:3378: finishing write tid 2 to smithi08477203-17 2024-09-20T22:55:27.381 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:55:27.382 INFO:tasks.rados.rados.0.smithi084.stdout:3378: finishing write tid 3 to smithi08477203-17 2024-09-20T22:55:27.384 INFO:tasks.rados.rados.0.smithi084.stdout:3378: finishing write tid 4 to smithi08477203-17 2024-09-20T22:55:27.384 INFO:tasks.rados.rados.0.smithi084.stdout:3378: finishing write tid 5 to smithi08477203-17 2024-09-20T22:55:27.384 INFO:tasks.rados.rados.0.smithi084.stdout:3378: oid 17 updating version 0 to 1264 2024-09-20T22:55:27.384 INFO:tasks.rados.rados.0.smithi084.stdout:3378: oid 17 updating version 1264 to 1266 2024-09-20T22:55:27.384 INFO:tasks.rados.rados.0.smithi084.stdout:3378: oid 17 version 1266 is already newer than 1263 2024-09-20T22:55:27.384 INFO:tasks.rados.rados.0.smithi084.stdout:3378: oid 17 version 1266 is already newer than 1265 2024-09-20T22:55:27.384 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 17 v 1266 (ObjNum 1147 snap 392 seq_num 1147) dirty exists 2024-09-20T22:55:27.384 INFO:tasks.rados.rados.0.smithi084.stdout:3378: left oid 17 (ObjNum 1147 snap 392 seq_num 1147) 2024-09-20T22:55:27.398 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-20T22:55:27.403 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:27.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:27.404 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:55:27.461 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:55:27.462 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:55:27.505 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:55:27.525 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:27.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:27.526 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:55:27.614 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:27 smithi084 ceph-mon[40024]: 3.1 scrub starts 2024-09-20T22:55:27.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:27 smithi084 ceph-mon[40024]: 3.1 scrub ok 2024-09-20T22:55:27.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:27 smithi084 ceph-mon[40024]: pgmap v1170: 158 pgs: 4 active+clean+snaptrim, 154 active+clean; 548 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 5.9 MiB/s rd, 1.0 MiB/s wr, 7 op/s 2024-09-20T22:55:27.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:27 smithi084 ceph-mon[40024]: 4.d scrub starts 2024-09-20T22:55:27.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:27 smithi084 ceph-mon[40024]: 4.d scrub ok 2024-09-20T22:55:27.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:27 smithi084 ceph-mon[40024]: 5.15 scrub starts 2024-09-20T22:55:27.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:27 smithi084 ceph-mon[40024]: 5.15 scrub ok 2024-09-20T22:55:27.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:27 smithi084 ceph-mon[40024]: osdmap e847: 8 total, 8 up, 8 in 2024-09-20T22:55:27.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:27 smithi084 ceph-mon[29774]: 3.1 scrub starts 2024-09-20T22:55:27.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:27 smithi084 ceph-mon[29774]: 3.1 scrub ok 2024-09-20T22:55:27.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:27 smithi084 ceph-mon[29774]: pgmap v1170: 158 pgs: 4 active+clean+snaptrim, 154 active+clean; 548 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 5.9 MiB/s rd, 1.0 MiB/s wr, 7 op/s 2024-09-20T22:55:27.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:27 smithi084 ceph-mon[29774]: 4.d scrub starts 2024-09-20T22:55:27.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:27 smithi084 ceph-mon[29774]: 4.d scrub ok 2024-09-20T22:55:27.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:27 smithi084 ceph-mon[29774]: 5.15 scrub starts 2024-09-20T22:55:27.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:27 smithi084 ceph-mon[29774]: 5.15 scrub ok 2024-09-20T22:55:27.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:27 smithi084 ceph-mon[29774]: osdmap e847: 8 total, 8 up, 8 in 2024-09-20T22:55:27.627 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:55:27.647 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:27.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:27.648 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:55:27.749 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:55:27.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:27 smithi154 ceph-mon[33264]: 3.1 scrub starts 2024-09-20T22:55:27.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:27 smithi154 ceph-mon[33264]: 3.1 scrub ok 2024-09-20T22:55:27.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:27 smithi154 ceph-mon[33264]: pgmap v1170: 158 pgs: 4 active+clean+snaptrim, 154 active+clean; 548 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 5.9 MiB/s rd, 1.0 MiB/s wr, 7 op/s 2024-09-20T22:55:27.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:27 smithi154 ceph-mon[33264]: 4.d scrub starts 2024-09-20T22:55:27.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:27 smithi154 ceph-mon[33264]: 4.d scrub ok 2024-09-20T22:55:27.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:27 smithi154 ceph-mon[33264]: 5.15 scrub starts 2024-09-20T22:55:27.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:27 smithi154 ceph-mon[33264]: 5.15 scrub ok 2024-09-20T22:55:27.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:27 smithi154 ceph-mon[33264]: osdmap e847: 8 total, 8 up, 8 in 2024-09-20T22:55:27.789 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:27.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:27.790 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:55:27.890 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:55:27.912 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:27.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:27.913 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:55:28.014 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:55:28.040 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:28.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:28.041 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:55:28.141 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:55:28.163 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:28.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:28.164 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:55:28.177 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:55:28.177 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:55:28.194 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:55:28.194 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:55:28.210 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:55:28.210 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:55:28.226 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-20T22:55:28.226 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T22:55:28.243 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:55:28.243 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:55:28.260 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:55:28.261 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:55:28.265 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:55:28.277 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:55:28.277 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:55:28.285 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:28.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:28.286 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:55:28.295 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:55:28.295 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:55:28.332 INFO:tasks.rados.rados.0.smithi084.stdout:3376: done (3 left) 2024-09-20T22:55:28.332 INFO:tasks.rados.rados.0.smithi084.stdout:3377: done (2 left) 2024-09-20T22:55:28.332 INFO:tasks.rados.rados.0.smithi084.stdout:3378: done (1 left) 2024-09-20T22:55:28.332 INFO:tasks.rados.rados.0.smithi084.stdout:3379: done (0 left) 2024-09-20T22:55:28.332 INFO:tasks.rados.rados.0.smithi084.stdout:3380: rollback oid 33 current snap is 393 2024-09-20T22:55:28.332 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 33 to 376 2024-09-20T22:55:28.333 INFO:tasks.rados.rados.0.smithi084.stdout:3381: copy_from oid 39 from oid 45 current snap is 393 2024-09-20T22:55:28.333 INFO:tasks.rados.rados.0.smithi084.stdout:3382: read oid 36 snap 378 2024-09-20T22:55:28.333 INFO:tasks.rados.rados.0.smithi084.stdout:3382: expect (ObjNum 1077 snap 358 seq_num 1077) 2024-09-20T22:55:28.333 INFO:tasks.rados.rados.0.smithi084.stdout:3383: snap_create 2024-09-20T22:55:28.333 INFO:tasks.rados.rados.0.smithi084.stdout:3381: finishing copy_from to smithi08477203-39 2024-09-20T22:55:28.333 INFO:tasks.rados.rados.0.smithi084.stdout:3381: got expected ENOENT (src dne) 2024-09-20T22:55:28.333 INFO:tasks.rados.rados.0.smithi084.stdout:3381: finishing copy_from racing read to smithi08477203-39 2024-09-20T22:55:28.339 INFO:tasks.rados.rados.0.smithi084.stdout:3380: finishing rollback tid 1 to smithi08477203-33 2024-09-20T22:55:28.340 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 33 v 1924 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-09-20T22:55:28.346 INFO:tasks.rados.rados.0.smithi084.stdout:3382: expect (ObjNum 1077 snap 358 seq_num 1077) 2024-09-20T22:55:28.387 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:55:28.412 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:28.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:28.413 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:55:28.514 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:55:28.538 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:28.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:28.539 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:55:28.615 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:55:28 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: ::ffff:172.21.15.154 - - [20/Sep/2024:22:55:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-20T22:55:28.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:28 smithi084 ceph-mon[29774]: 4.1e scrub starts 2024-09-20T22:55:28.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:28 smithi084 ceph-mon[29774]: 4.1e scrub ok 2024-09-20T22:55:28.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:28 smithi084 ceph-mon[29774]: 5.9 scrub starts 2024-09-20T22:55:28.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:28 smithi084 ceph-mon[29774]: 5.9 scrub ok 2024-09-20T22:55:28.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:28 smithi084 ceph-mon[29774]: 1.1 scrub starts 2024-09-20T22:55:28.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:28 smithi084 ceph-mon[29774]: 1.1 scrub ok 2024-09-20T22:55:28.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:28 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/23488491' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:55:28.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:28 smithi084 ceph-mon[29774]: osdmap e848: 8 total, 8 up, 8 in 2024-09-20T22:55:28.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:28 smithi084 ceph-mon[40024]: 4.1e scrub starts 2024-09-20T22:55:28.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:28 smithi084 ceph-mon[40024]: 4.1e scrub ok 2024-09-20T22:55:28.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:28 smithi084 ceph-mon[40024]: 5.9 scrub starts 2024-09-20T22:55:28.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:28 smithi084 ceph-mon[40024]: 5.9 scrub ok 2024-09-20T22:55:28.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:28 smithi084 ceph-mon[40024]: 1.1 scrub starts 2024-09-20T22:55:28.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:28 smithi084 ceph-mon[40024]: 1.1 scrub ok 2024-09-20T22:55:28.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:28 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/23488491' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:55:28.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:28 smithi084 ceph-mon[40024]: osdmap e848: 8 total, 8 up, 8 in 2024-09-20T22:55:28.639 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:55:28.661 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:28.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:28.661 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:55:28.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:28 smithi154 ceph-mon[33264]: 4.1e scrub starts 2024-09-20T22:55:28.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:28 smithi154 ceph-mon[33264]: 4.1e scrub ok 2024-09-20T22:55:28.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:28 smithi154 ceph-mon[33264]: 5.9 scrub starts 2024-09-20T22:55:28.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:28 smithi154 ceph-mon[33264]: 5.9 scrub ok 2024-09-20T22:55:28.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:28 smithi154 ceph-mon[33264]: 1.1 scrub starts 2024-09-20T22:55:28.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:28 smithi154 ceph-mon[33264]: 1.1 scrub ok 2024-09-20T22:55:28.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:28 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/23488491' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:55:28.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:28 smithi154 ceph-mon[33264]: osdmap e848: 8 total, 8 up, 8 in 2024-09-20T22:55:28.762 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:55:28.783 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:28.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:28.784 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:55:28.885 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:55:28.911 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:28.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:28.912 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:55:29.012 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:55:29.035 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:29.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:29.036 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:55:29.133 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:55:29.137 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:55:29.158 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:29.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:29.159 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:55:29.259 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:55:29.281 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:29.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:29.282 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:55:29.335 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is unset 2024-09-20T22:55:29.340 INFO:tasks.rados.rados.0.smithi084.stdout:3380: done (3 left) 2024-09-20T22:55:29.340 INFO:tasks.rados.rados.0.smithi084.stdout:3381: done (2 left) 2024-09-20T22:55:29.340 INFO:tasks.rados.rados.0.smithi084.stdout:3382: done (1 left) 2024-09-20T22:55:29.341 INFO:tasks.rados.rados.0.smithi084.stdout:3383: done (0 left) 2024-09-20T22:55:29.341 INFO:tasks.rados.rados.0.smithi084.stdout:3384: delete oid 32 current snap is 394 2024-09-20T22:55:29.345 INFO:tasks.rados.rados.0.smithi084.stdout:3384: done (0 left) 2024-09-20T22:55:29.345 INFO:tasks.rados.rados.0.smithi084.stdout:3385: write (excl) oid 30 current snap is 394 2024-09-20T22:55:29.345 INFO:tasks.rados.rados.0.smithi084.stdout:3385: seq_num 1149 ranges {511243=430404,1669928=664720,3103254=207989} 2024-09-20T22:55:29.351 INFO:tasks.rados.rados.0.smithi084.stdout:3385: writing smithi08477203-30 from 511243 to 941647 tid 1 2024-09-20T22:55:29.359 INFO:tasks.rados.rados.0.smithi084.stdout:3385: writing smithi08477203-30 from 1669928 to 2334648 tid 2 2024-09-20T22:55:29.365 INFO:tasks.rados.rados.0.smithi084.stdout:3385: writing smithi08477203-30 from 3103254 to 3311243 tid 3 2024-09-20T22:55:29.365 INFO:tasks.rados.rados.0.smithi084.stdout:3386: copy_from oid 25 from oid 50 current snap is 394 2024-09-20T22:55:29.365 INFO:tasks.rados.rados.0.smithi084.stdout:3385: finishing write tid 1 to smithi08477203-30 2024-09-20T22:55:29.365 INFO:tasks.rados.rados.0.smithi084.stdout:3385: finishing write tid 2 to smithi08477203-30 2024-09-20T22:55:29.365 INFO:tasks.rados.rados.0.smithi084.stdout:3387: snap_create 2024-09-20T22:55:29.367 INFO:tasks.rados.rados.0.smithi084.stdout:3385: finishing write tid 3 to smithi08477203-30 2024-09-20T22:55:29.367 INFO:tasks.rados.rados.0.smithi084.stdout:3385: finishing write tid 4 to smithi08477203-30 2024-09-20T22:55:29.368 INFO:tasks.rados.rados.0.smithi084.stdout:3385: finishing write tid 5 to smithi08477203-30 2024-09-20T22:55:29.368 INFO:tasks.rados.rados.0.smithi084.stdout:3385: oid 30 updating version 0 to 2561 2024-09-20T22:55:29.368 INFO:tasks.rados.rados.0.smithi084.stdout:3385: oid 30 updating version 2561 to 2562 2024-09-20T22:55:29.368 INFO:tasks.rados.rados.0.smithi084.stdout:3385: oid 30 updating version 2562 to 2563 2024-09-20T22:55:29.368 INFO:tasks.rados.rados.0.smithi084.stdout:3385: oid 30 version 2563 is already newer than 2560 2024-09-20T22:55:29.368 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 30 v 2563 (ObjNum 1149 snap 394 seq_num 1149) dirty exists 2024-09-20T22:55:29.368 INFO:tasks.rados.rados.0.smithi084.stdout:3385: left oid 30 (ObjNum 1149 snap 394 seq_num 1149) 2024-09-20T22:55:29.381 INFO:tasks.rados.rados.0.smithi084.stdout:3386: finishing copy_from racing read to smithi08477203-25 2024-09-20T22:55:29.382 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:55:29.387 INFO:tasks.rados.rados.0.smithi084.stdout:3386: finishing copy_from to smithi08477203-25 2024-09-20T22:55:29.387 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 25 v 1891 (ObjNum 1128 snap 381 seq_num 1128) dirty exists 2024-09-20T22:55:29.405 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:29.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:29.406 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:55:29.507 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:55:29.528 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:29.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:29.529 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:55:29.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:29 smithi084 ceph-mon[29774]: 4.16 scrub starts 2024-09-20T22:55:29.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:29 smithi084 ceph-mon[29774]: 4.16 scrub ok 2024-09-20T22:55:29.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:29 smithi084 ceph-mon[29774]: pgmap v1173: 158 pgs: 4 active+clean+snaptrim, 154 active+clean; 548 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail 2024-09-20T22:55:29.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:29 smithi084 ceph-mon[29774]: 5.10 scrub starts 2024-09-20T22:55:29.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:29 smithi084 ceph-mon[29774]: 5.10 scrub ok 2024-09-20T22:55:29.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:29 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/610100510' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:55:29.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:29 smithi084 ceph-mon[29774]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T22:55:29.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:29 smithi084 ceph-mon[29774]: Cluster is now healthy 2024-09-20T22:55:29.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:29 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/610100510' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T22:55:29.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:29 smithi084 ceph-mon[29774]: osdmap e849: 8 total, 8 up, 8 in 2024-09-20T22:55:29.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:29 smithi084 ceph-mon[40024]: 4.16 scrub starts 2024-09-20T22:55:29.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:29 smithi084 ceph-mon[40024]: 4.16 scrub ok 2024-09-20T22:55:29.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:29 smithi084 ceph-mon[40024]: pgmap v1173: 158 pgs: 4 active+clean+snaptrim, 154 active+clean; 548 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail 2024-09-20T22:55:29.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:29 smithi084 ceph-mon[40024]: 5.10 scrub starts 2024-09-20T22:55:29.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:29 smithi084 ceph-mon[40024]: 5.10 scrub ok 2024-09-20T22:55:29.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:29 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/610100510' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:55:29.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:29 smithi084 ceph-mon[40024]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T22:55:29.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:29 smithi084 ceph-mon[40024]: Cluster is now healthy 2024-09-20T22:55:29.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:29 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/610100510' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T22:55:29.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:29 smithi084 ceph-mon[40024]: osdmap e849: 8 total, 8 up, 8 in 2024-09-20T22:55:29.630 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:55:29.651 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:29.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:29.653 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:55:29.669 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:29 smithi154 ceph-mon[33264]: 4.16 scrub starts 2024-09-20T22:55:29.669 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:29 smithi154 ceph-mon[33264]: 4.16 scrub ok 2024-09-20T22:55:29.669 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:29 smithi154 ceph-mon[33264]: pgmap v1173: 158 pgs: 4 active+clean+snaptrim, 154 active+clean; 548 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail 2024-09-20T22:55:29.669 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:29 smithi154 ceph-mon[33264]: 5.10 scrub starts 2024-09-20T22:55:29.669 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:29 smithi154 ceph-mon[33264]: 5.10 scrub ok 2024-09-20T22:55:29.669 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:29 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/610100510' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:55:29.669 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:29 smithi154 ceph-mon[33264]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T22:55:29.670 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:29 smithi154 ceph-mon[33264]: Cluster is now healthy 2024-09-20T22:55:29.670 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:29 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/610100510' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T22:55:29.670 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:29 smithi154 ceph-mon[33264]: osdmap e849: 8 total, 8 up, 8 in 2024-09-20T22:55:29.753 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:55:29.775 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:29.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:29.776 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:55:29.877 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:55:29.898 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:29.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:29.899 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:55:30.000 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:55:30.004 INFO:journalctl@ceph.iscsi.iscsi.a.smithi154.stdout:Sep 20 22:55:29 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-iscsi-iscsi-a[59730]: debug there is no tcmu-runner data available 2024-09-20T22:55:30.020 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:30.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:30.021 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:55:30.122 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:55:30.148 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:30.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:30.149 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:55:30.249 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:55:30.270 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:30.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:30.271 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:55:30.341 INFO:tasks.rados.rados.0.smithi084.stdout:3385: done (2 left) 2024-09-20T22:55:30.341 INFO:tasks.rados.rados.0.smithi084.stdout:3386: done (1 left) 2024-09-20T22:55:30.341 INFO:tasks.rados.rados.0.smithi084.stdout:3387: done (0 left) 2024-09-20T22:55:30.341 INFO:tasks.rados.rados.0.smithi084.stdout:3388: delete oid 24 current snap is 395 2024-09-20T22:55:30.345 INFO:tasks.rados.rados.0.smithi084.stdout:3388: done (0 left) 2024-09-20T22:55:30.346 INFO:tasks.rados.rados.0.smithi084.stdout:3389: delete oid 3 current snap is 395 2024-09-20T22:55:30.348 INFO:tasks.rados.rados.0.smithi084.stdout:3389: done (0 left) 2024-09-20T22:55:30.348 INFO:tasks.rados.rados.0.smithi084.stdout:3390: snap_create 2024-09-20T22:55:30.372 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:55:30.384 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-20T22:55:30.384 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2024-09-20T22:55:30.384 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2024-09-20T22:55:30.384 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-20T22:55:30.393 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:30.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:30.394 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:55:30.495 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:55:30.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:30.522 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:30.523 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:55:30.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:30 smithi084 ceph-mon[29774]: 3.e scrub starts 2024-09-20T22:55:30.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:30 smithi084 ceph-mon[29774]: 3.e scrub ok 2024-09-20T22:55:30.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:30 smithi084 ceph-mon[29774]: 6.17 scrub starts 2024-09-20T22:55:30.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:30 smithi084 ceph-mon[29774]: 6.17 scrub ok 2024-09-20T22:55:30.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:30 smithi084 ceph-mon[29774]: purged_snaps scrub starts 2024-09-20T22:55:30.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:30 smithi084 ceph-mon[29774]: purged_snaps scrub ok 2024-09-20T22:55:30.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:30 smithi084 ceph-mon[29774]: 5.16 scrub starts 2024-09-20T22:55:30.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:30 smithi084 ceph-mon[29774]: 5.16 scrub ok 2024-09-20T22:55:30.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:30 smithi084 ceph-mon[29774]: 7.3 scrub starts 2024-09-20T22:55:30.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:30 smithi084 ceph-mon[29774]: 7.3 scrub ok 2024-09-20T22:55:30.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:30 smithi084 ceph-mon[29774]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:55:30.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:30 smithi084 ceph-mon[29774]: 3.1a scrub starts 2024-09-20T22:55:30.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:30 smithi084 ceph-mon[29774]: 3.1a scrub ok 2024-09-20T22:55:30.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:30 smithi084 ceph-mon[29774]: osdmap e850: 8 total, 8 up, 8 in 2024-09-20T22:55:30.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:30 smithi084 ceph-mon[40024]: 3.e scrub starts 2024-09-20T22:55:30.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:30 smithi084 ceph-mon[40024]: 3.e scrub ok 2024-09-20T22:55:30.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:30 smithi084 ceph-mon[40024]: 6.17 scrub starts 2024-09-20T22:55:30.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:30 smithi084 ceph-mon[40024]: 6.17 scrub ok 2024-09-20T22:55:30.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:30 smithi084 ceph-mon[40024]: purged_snaps scrub starts 2024-09-20T22:55:30.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:30 smithi084 ceph-mon[40024]: purged_snaps scrub ok 2024-09-20T22:55:30.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:30 smithi084 ceph-mon[40024]: 5.16 scrub starts 2024-09-20T22:55:30.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:30 smithi084 ceph-mon[40024]: 5.16 scrub ok 2024-09-20T22:55:30.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:30 smithi084 ceph-mon[40024]: 7.3 scrub starts 2024-09-20T22:55:30.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:30 smithi084 ceph-mon[40024]: 7.3 scrub ok 2024-09-20T22:55:30.617 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:30 smithi084 ceph-mon[40024]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:55:30.617 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:30 smithi084 ceph-mon[40024]: 3.1a scrub starts 2024-09-20T22:55:30.617 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:30 smithi084 ceph-mon[40024]: 3.1a scrub ok 2024-09-20T22:55:30.617 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:30 smithi084 ceph-mon[40024]: osdmap e850: 8 total, 8 up, 8 in 2024-09-20T22:55:30.623 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:55:30.644 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:30.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:30.645 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:55:30.746 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:55:30.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:30 smithi154 ceph-mon[33264]: 3.e scrub starts 2024-09-20T22:55:30.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:30 smithi154 ceph-mon[33264]: 3.e scrub ok 2024-09-20T22:55:30.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:30 smithi154 ceph-mon[33264]: 6.17 scrub starts 2024-09-20T22:55:30.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:30 smithi154 ceph-mon[33264]: 6.17 scrub ok 2024-09-20T22:55:30.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:30 smithi154 ceph-mon[33264]: purged_snaps scrub starts 2024-09-20T22:55:30.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:30 smithi154 ceph-mon[33264]: purged_snaps scrub ok 2024-09-20T22:55:30.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:30 smithi154 ceph-mon[33264]: 5.16 scrub starts 2024-09-20T22:55:30.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:30 smithi154 ceph-mon[33264]: 5.16 scrub ok 2024-09-20T22:55:30.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:30 smithi154 ceph-mon[33264]: 7.3 scrub starts 2024-09-20T22:55:30.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:30 smithi154 ceph-mon[33264]: 7.3 scrub ok 2024-09-20T22:55:30.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:30 smithi154 ceph-mon[33264]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:55:30.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:30 smithi154 ceph-mon[33264]: 3.1a scrub starts 2024-09-20T22:55:30.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:30 smithi154 ceph-mon[33264]: 3.1a scrub ok 2024-09-20T22:55:30.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:30 smithi154 ceph-mon[33264]: osdmap e850: 8 total, 8 up, 8 in 2024-09-20T22:55:30.772 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:30.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:30.773 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:55:30.874 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:55:30.899 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:30.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:30.900 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:55:30.974 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.1 dump_blocked_ops 2024-09-20T22:55:31.002 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:55:31.037 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:31.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:31.038 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:55:31.047 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-20T22:55:31.139 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:55:31.165 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:31.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:31.166 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:55:31.182 INFO:tasks.thrashosds.thrasher:in_osds: [2, 3, 5, 6, 7, 0, 4, 1] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-20T22:55:31.182 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-20T22:55:31.182 INFO:tasks.thrashosds.thrasher:fixing pg num pool unique_pool_0 2024-09-20T22:55:31.183 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph pg dump --format=json 2024-09-20T22:55:31.240 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:55:31.266 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:55:31.291 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:31.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:31.292 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:55:31.344 INFO:tasks.rados.rados.0.smithi084.stdout:3390: done (0 left) 2024-09-20T22:55:31.345 INFO:tasks.rados.rados.0.smithi084.stdout:3391: write oid 23 current snap is 396 2024-09-20T22:55:31.345 INFO:tasks.rados.rados.0.smithi084.stdout:3391: seq_num 1152 ranges {561664=656151,1729021=528400,2669170=692494} 2024-09-20T22:55:31.351 INFO:tasks.rados.rados.0.smithi084.stdout:3391: writing smithi08477203-23 from 561664 to 1217815 tid 1 2024-09-20T22:55:31.357 INFO:tasks.rados.rados.0.smithi084.stdout:3391: writing smithi08477203-23 from 1729021 to 2257421 tid 2 2024-09-20T22:55:31.363 INFO:tasks.rados.rados.0.smithi084.stdout:3391: writing smithi08477203-23 from 2669170 to 3361664 tid 3 2024-09-20T22:55:31.363 INFO:tasks.rados.rados.0.smithi084.stdout:3392: write (excl) oid 11 current snap is 396 2024-09-20T22:55:31.364 INFO:tasks.rados.rados.0.smithi084.stdout:3391: finishing write tid 1 to smithi08477203-23 2024-09-20T22:55:31.364 INFO:tasks.rados.rados.0.smithi084.stdout:3391: finishing write tid 2 to smithi08477203-23 2024-09-20T22:55:31.364 INFO:tasks.rados.rados.0.smithi084.stdout:3392: seq_num 1153 ranges {578471=731400,1787503=672058,3096593=281878} 2024-09-20T22:55:31.370 INFO:tasks.rados.rados.0.smithi084.stdout:3392: writing smithi08477203-11 from 578471 to 1309871 tid 1 2024-09-20T22:55:31.377 INFO:tasks.rados.rados.0.smithi084.stdout:3392: writing smithi08477203-11 from 1787503 to 2459561 tid 2 2024-09-20T22:55:31.383 INFO:tasks.rados.rados.0.smithi084.stdout:3392: writing smithi08477203-11 from 3096593 to 3378471 tid 3 2024-09-20T22:55:31.383 INFO:tasks.rados.rados.0.smithi084.stdout:3393: copy_from oid 17 from oid 4 current snap is 396 2024-09-20T22:55:31.383 INFO:tasks.rados.rados.0.smithi084.stdout:3391: finishing write tid 3 to smithi08477203-23 2024-09-20T22:55:31.383 INFO:tasks.rados.rados.0.smithi084.stdout:3391: finishing write tid 4 to smithi08477203-23 2024-09-20T22:55:31.383 INFO:tasks.rados.rados.0.smithi084.stdout:3391: finishing write tid 5 to smithi08477203-23 2024-09-20T22:55:31.383 INFO:tasks.rados.rados.0.smithi084.stdout:3391: oid 23 updating version 0 to 1932 2024-09-20T22:55:31.383 INFO:tasks.rados.rados.0.smithi084.stdout:3391: oid 23 version 1932 is already newer than 1929 2024-09-20T22:55:31.383 INFO:tasks.rados.rados.0.smithi084.stdout:3391: oid 23 version 1932 is already newer than 1931 2024-09-20T22:55:31.383 INFO:tasks.rados.rados.0.smithi084.stdout:3391: oid 23 version 1932 is already newer than 1930 2024-09-20T22:55:31.383 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 23 v 1932 (ObjNum 1152 snap 396 seq_num 1152) dirty exists 2024-09-20T22:55:31.383 INFO:tasks.rados.rados.0.smithi084.stdout:3391: left oid 23 (ObjNum 1152 snap 396 seq_num 1152) 2024-09-20T22:55:31.383 INFO:tasks.rados.rados.0.smithi084.stdout:3392: finishing write tid 1 to smithi08477203-11 2024-09-20T22:55:31.383 INFO:tasks.rados.rados.0.smithi084.stdout:3392: finishing write tid 2 to smithi08477203-11 2024-09-20T22:55:31.383 INFO:tasks.rados.rados.0.smithi084.stdout:3391: done (2 left) 2024-09-20T22:55:31.384 INFO:tasks.rados.rados.0.smithi084.stdout:3394: write oid 14 current snap is 396 2024-09-20T22:55:31.384 INFO:tasks.rados.rados.0.smithi084.stdout:3394: seq_num 1154 ranges {595278=406649,1729632=532069,3007663=387615} 2024-09-20T22:55:31.388 INFO:tasks.rados.rados.0.smithi084.stdout:3394: writing smithi08477203-14 from 595278 to 1001927 tid 1 2024-09-20T22:55:31.393 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:55:31.396 INFO:tasks.rados.rados.0.smithi084.stdout:3394: writing smithi08477203-14 from 1729632 to 2261701 tid 2 2024-09-20T22:55:31.403 INFO:tasks.rados.rados.0.smithi084.stdout:3394: writing smithi08477203-14 from 3007663 to 3395278 tid 3 2024-09-20T22:55:31.403 INFO:tasks.rados.rados.0.smithi084.stdout:3395: read oid 28 snap -1 2024-09-20T22:55:31.403 INFO:tasks.rados.rados.0.smithi084.stdout:3395: expect deleted 2024-09-20T22:55:31.403 INFO:tasks.rados.rados.0.smithi084.stdout:3392: finishing write tid 3 to smithi08477203-11 2024-09-20T22:55:31.403 INFO:tasks.rados.rados.0.smithi084.stdout:3392: finishing write tid 4 to smithi08477203-11 2024-09-20T22:55:31.403 INFO:tasks.rados.rados.0.smithi084.stdout:3392: finishing write tid 5 to smithi08477203-11 2024-09-20T22:55:31.403 INFO:tasks.rados.rados.0.smithi084.stdout:3392: oid 11 updating version 0 to 1159 2024-09-20T22:55:31.403 INFO:tasks.rados.rados.0.smithi084.stdout:3392: oid 11 updating version 1159 to 1160 2024-09-20T22:55:31.403 INFO:tasks.rados.rados.0.smithi084.stdout:3392: oid 11 version 1160 is already newer than 1158 2024-09-20T22:55:31.404 INFO:tasks.rados.rados.0.smithi084.stdout:3392: oid 11 version 1160 is already newer than 1157 2024-09-20T22:55:31.404 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 11 v 1160 (ObjNum 1153 snap 396 seq_num 1153) dirty exists 2024-09-20T22:55:31.404 INFO:tasks.rados.rados.0.smithi084.stdout:3392: left oid 11 (ObjNum 1153 snap 396 seq_num 1153) 2024-09-20T22:55:31.404 INFO:tasks.rados.rados.0.smithi084.stdout:3394: finishing write tid 1 to smithi08477203-14 2024-09-20T22:55:31.404 INFO:tasks.rados.rados.0.smithi084.stdout:3393: finishing copy_from racing read to smithi08477203-17 2024-09-20T22:55:31.404 INFO:tasks.rados.rados.0.smithi084.stdout:3394: finishing write tid 2 to smithi08477203-14 2024-09-20T22:55:31.404 INFO:tasks.rados.rados.0.smithi084.stdout:3393: finishing copy_from to smithi08477203-17 2024-09-20T22:55:31.404 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 17 v 2639 (ObjNum 1111 snap 378 seq_num 1111) dirty exists 2024-09-20T22:55:31.404 INFO:tasks.rados.rados.0.smithi084.stdout:3392: done (3 left) 2024-09-20T22:55:31.404 INFO:tasks.rados.rados.0.smithi084.stdout:3393: done (2 left) 2024-09-20T22:55:31.404 INFO:tasks.rados.rados.0.smithi084.stdout:3396: rollback oid 36 current snap is 396 2024-09-20T22:55:31.404 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 36 to 328 2024-09-20T22:55:31.405 INFO:tasks.rados.rados.0.smithi084.stdout:3397: write oid 35 current snap is 396 2024-09-20T22:55:31.405 INFO:tasks.rados.rados.0.smithi084.stdout:3397: seq_num 1155 ranges {612085=481898,1788114=675727,3035086=376999} 2024-09-20T22:55:31.409 INFO:tasks.rados.rados.0.smithi084.stdout:3397: writing smithi08477203-35 from 612085 to 1093983 tid 1 2024-09-20T22:55:31.417 INFO:tasks.rados.rados.0.smithi084.stdout:3397: writing smithi08477203-35 from 1788114 to 2463841 tid 2 2024-09-20T22:55:31.421 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:31.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:31.422 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:55:31.423 INFO:tasks.rados.rados.0.smithi084.stdout:3397: writing smithi08477203-35 from 3035086 to 3412085 tid 3 2024-09-20T22:55:31.423 INFO:tasks.rados.rados.0.smithi084.stdout:3398: snap_create 2024-09-20T22:55:31.423 INFO:tasks.rados.rados.0.smithi084.stdout:3394: finishing write tid 3 to smithi08477203-14 2024-09-20T22:55:31.423 INFO:tasks.rados.rados.0.smithi084.stdout:3394: finishing write tid 4 to smithi08477203-14 2024-09-20T22:55:31.423 INFO:tasks.rados.rados.0.smithi084.stdout:3394: finishing write tid 5 to smithi08477203-14 2024-09-20T22:55:31.423 INFO:tasks.rados.rados.0.smithi084.stdout:3394: oid 14 updating version 0 to 1936 2024-09-20T22:55:31.424 INFO:tasks.rados.rados.0.smithi084.stdout:3394: oid 14 updating version 1936 to 1937 2024-09-20T22:55:31.424 INFO:tasks.rados.rados.0.smithi084.stdout:3394: oid 14 updating version 1937 to 1939 2024-09-20T22:55:31.424 INFO:tasks.rados.rados.0.smithi084.stdout:3394: oid 14 version 1939 is already newer than 1938 2024-09-20T22:55:31.424 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 14 v 1939 (ObjNum 1154 snap 396 seq_num 1154) dirty exists 2024-09-20T22:55:31.424 INFO:tasks.rados.rados.0.smithi084.stdout:3394: left oid 14 (ObjNum 1154 snap 396 seq_num 1154) 2024-09-20T22:55:31.424 INFO:tasks.rados.rados.0.smithi084.stdout:3396: finishing rollback tid 1 to smithi08477203-36 2024-09-20T22:55:31.424 INFO:tasks.rados.rados.0.smithi084.stdout:3396: finishing rollback tid 2 to smithi08477203-36 2024-09-20T22:55:31.424 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 36 v 2641 (ObjNum 945 snap 311 seq_num 945) dirty exists 2024-09-20T22:55:31.424 INFO:tasks.rados.rados.0.smithi084.stdout:3397: finishing write tid 1 to smithi08477203-35 2024-09-20T22:55:31.424 INFO:tasks.rados.rados.0.smithi084.stdout:3397: finishing write tid 2 to smithi08477203-35 2024-09-20T22:55:31.426 INFO:tasks.rados.rados.0.smithi084.stdout:3397: finishing write tid 3 to smithi08477203-35 2024-09-20T22:55:31.426 INFO:tasks.rados.rados.0.smithi084.stdout:3397: finishing write tid 4 to smithi08477203-35 2024-09-20T22:55:31.426 INFO:tasks.rados.rados.0.smithi084.stdout:3397: finishing write tid 5 to smithi08477203-35 2024-09-20T22:55:31.427 INFO:tasks.rados.rados.0.smithi084.stdout:3397: oid 35 updating version 0 to 1162 2024-09-20T22:55:31.427 INFO:tasks.rados.rados.0.smithi084.stdout:3397: oid 35 updating version 1162 to 1163 2024-09-20T22:55:31.427 INFO:tasks.rados.rados.0.smithi084.stdout:3397: oid 35 updating version 1163 to 1165 2024-09-20T22:55:31.427 INFO:tasks.rados.rados.0.smithi084.stdout:3397: oid 35 version 1165 is already newer than 1164 2024-09-20T22:55:31.427 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 35 v 1165 (ObjNum 1155 snap 396 seq_num 1155) dirty exists 2024-09-20T22:55:31.427 INFO:tasks.rados.rados.0.smithi084.stdout:3397: left oid 35 (ObjNum 1155 snap 396 seq_num 1155) 2024-09-20T22:55:31.512 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:31 smithi084 ceph-mon[29774]: 3.d scrub starts 2024-09-20T22:55:31.512 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:31 smithi084 ceph-mon[29774]: 3.d scrub ok 2024-09-20T22:55:31.512 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:31 smithi084 ceph-mon[29774]: pgmap v1176: 158 pgs: 4 active+clean+snaptrim, 154 active+clean; 556 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 7.6 MiB/s rd, 3.1 MiB/s wr, 15 op/s 2024-09-20T22:55:31.512 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:31 smithi084 ceph-mon[29774]: purged_snaps scrub starts 2024-09-20T22:55:31.513 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:31 smithi084 ceph-mon[29774]: purged_snaps scrub ok 2024-09-20T22:55:31.513 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:31 smithi084 ceph-mon[29774]: 4.e deep-scrub starts 2024-09-20T22:55:31.513 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:31 smithi084 ceph-mon[29774]: 4.e deep-scrub ok 2024-09-20T22:55:31.513 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:31 smithi084 ceph-mon[29774]: 3.4 scrub starts 2024-09-20T22:55:31.513 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:31 smithi084 ceph-mon[29774]: 3.4 scrub ok 2024-09-20T22:55:31.513 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:31 smithi084 ceph-mon[29774]: osdmap e851: 8 total, 8 up, 8 in 2024-09-20T22:55:31.513 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:31 smithi084 ceph-mon[40024]: 3.d scrub starts 2024-09-20T22:55:31.513 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:31 smithi084 ceph-mon[40024]: 3.d scrub ok 2024-09-20T22:55:31.513 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:31 smithi084 ceph-mon[40024]: pgmap v1176: 158 pgs: 4 active+clean+snaptrim, 154 active+clean; 556 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 7.6 MiB/s rd, 3.1 MiB/s wr, 15 op/s 2024-09-20T22:55:31.513 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:31 smithi084 ceph-mon[40024]: purged_snaps scrub starts 2024-09-20T22:55:31.514 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:31 smithi084 ceph-mon[40024]: purged_snaps scrub ok 2024-09-20T22:55:31.514 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:31 smithi084 ceph-mon[40024]: 4.e deep-scrub starts 2024-09-20T22:55:31.514 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:31 smithi084 ceph-mon[40024]: 4.e deep-scrub ok 2024-09-20T22:55:31.514 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:31 smithi084 ceph-mon[40024]: 3.4 scrub starts 2024-09-20T22:55:31.514 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:31 smithi084 ceph-mon[40024]: 3.4 scrub ok 2024-09-20T22:55:31.514 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:31 smithi084 ceph-mon[40024]: osdmap e851: 8 total, 8 up, 8 in 2024-09-20T22:55:31.522 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:55:31.545 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:31.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:31.546 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:55:31.647 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:55:31.670 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:31.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:31.671 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:55:31.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:31 smithi154 ceph-mon[33264]: 3.d scrub starts 2024-09-20T22:55:31.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:31 smithi154 ceph-mon[33264]: 3.d scrub ok 2024-09-20T22:55:31.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:31 smithi154 ceph-mon[33264]: pgmap v1176: 158 pgs: 4 active+clean+snaptrim, 154 active+clean; 556 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 7.6 MiB/s rd, 3.1 MiB/s wr, 15 op/s 2024-09-20T22:55:31.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:31 smithi154 ceph-mon[33264]: purged_snaps scrub starts 2024-09-20T22:55:31.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:31 smithi154 ceph-mon[33264]: purged_snaps scrub ok 2024-09-20T22:55:31.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:31 smithi154 ceph-mon[33264]: 4.e deep-scrub starts 2024-09-20T22:55:31.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:31 smithi154 ceph-mon[33264]: 4.e deep-scrub ok 2024-09-20T22:55:31.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:31 smithi154 ceph-mon[33264]: 3.4 scrub starts 2024-09-20T22:55:31.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:31 smithi154 ceph-mon[33264]: 3.4 scrub ok 2024-09-20T22:55:31.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:31 smithi154 ceph-mon[33264]: osdmap e851: 8 total, 8 up, 8 in 2024-09-20T22:55:31.772 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:55:31.775 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-20T22:55:31.775 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2024-09-20T22:55:31.775 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2024-09-20T22:55:31.775 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2024-09-20T22:55:31.776 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-20T22:55:31.794 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:31.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:31.795 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:55:31.896 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:55:31.918 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:31.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:31.919 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:55:32.019 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:55:32.027 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:55:32.028 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:55:32.042 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:32.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:32.042 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:55:32.143 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:55:32.166 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:32.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:32.167 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:55:32.237 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd set noscrub 2024-09-20T22:55:32.268 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:55:32.290 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:32.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:32.291 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:55:32.391 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:55:32.413 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:32.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:32.414 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:55:32.418 INFO:tasks.rados.rados.0.smithi084.stdout:3394: done (4 left) 2024-09-20T22:55:32.418 INFO:tasks.rados.rados.0.smithi084.stdout:3395: done (3 left) 2024-09-20T22:55:32.418 INFO:tasks.rados.rados.0.smithi084.stdout:3396: done (2 left) 2024-09-20T22:55:32.418 INFO:tasks.rados.rados.0.smithi084.stdout:3397: done (1 left) 2024-09-20T22:55:32.418 INFO:tasks.rados.rados.0.smithi084.stdout:3398: done (0 left) 2024-09-20T22:55:32.419 INFO:tasks.rados.rados.0.smithi084.stdout:3399: delete oid 2 current snap is 397 2024-09-20T22:55:32.424 INFO:tasks.rados.rados.0.smithi084.stdout:3399: done (0 left) 2024-09-20T22:55:32.424 INFO:tasks.rados.rados.0.smithi084.stdout:3400: read oid 36 snap -1 2024-09-20T22:55:32.424 INFO:tasks.rados.rados.0.smithi084.stdout:3400: expect (ObjNum 945 snap 311 seq_num 945) 2024-09-20T22:55:32.424 INFO:tasks.rados.rados.0.smithi084.stdout:3401: write oid 43 current snap is 397 2024-09-20T22:55:32.424 INFO:tasks.rados.rados.0.smithi084.stdout:3401: seq_num 1157 ranges {645699=748749,2021431=679396,3206285=239414} 2024-09-20T22:55:32.432 INFO:tasks.rados.rados.0.smithi084.stdout:3401: writing smithi08477203-43 from 645699 to 1394448 tid 1 2024-09-20T22:55:32.439 INFO:tasks.rados.rados.0.smithi084.stdout:3401: writing smithi08477203-43 from 2021431 to 2700827 tid 2 2024-09-20T22:55:32.444 INFO:tasks.rados.rados.0.smithi084.stdout:3401: writing smithi08477203-43 from 3206285 to 3445699 tid 3 2024-09-20T22:55:32.444 INFO:tasks.rados.rados.0.smithi084.stdout:3402: read oid 11 snap -1 2024-09-20T22:55:32.444 INFO:tasks.rados.rados.0.smithi084.stdout:3402: expect (ObjNum 1153 snap 396 seq_num 1153) 2024-09-20T22:55:32.444 INFO:tasks.rados.rados.0.smithi084.stdout:3403: write oid 46 current snap is 397 2024-09-20T22:55:32.444 INFO:tasks.rados.rados.0.smithi084.stdout:3403: seq_num 1158 ranges {662506=423998,1563560=539407,2717355=490285} 2024-09-20T22:55:32.449 INFO:tasks.rados.rados.0.smithi084.stdout:3403: writing smithi08477203-46 from 662506 to 1086504 tid 1 2024-09-20T22:55:32.455 INFO:tasks.rados.rados.0.smithi084.stdout:3403: writing smithi08477203-46 from 1563560 to 2102967 tid 2 2024-09-20T22:55:32.461 INFO:tasks.rados.rados.0.smithi084.stdout:3403: writing smithi08477203-46 from 2717355 to 3207640 tid 3 2024-09-20T22:55:32.461 INFO:tasks.rados.rados.0.smithi084.stdout:3404: snap_create 2024-09-20T22:55:32.461 INFO:tasks.rados.rados.0.smithi084.stdout:3400: expect (ObjNum 945 snap 311 seq_num 945) 2024-09-20T22:55:32.516 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:55:32.537 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:32.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:32.538 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:55:32.567 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.1 dump_historic_ops 2024-09-20T22:55:32.599 INFO:tasks.rados.rados.0.smithi084.stdout:3401: finishing write tid 1 to smithi08477203-43 2024-09-20T22:55:32.599 INFO:tasks.rados.rados.0.smithi084.stdout:3401: finishing write tid 2 to smithi08477203-43 2024-09-20T22:55:32.599 INFO:tasks.rados.rados.0.smithi084.stdout:3401: finishing write tid 3 to smithi08477203-43 2024-09-20T22:55:32.599 INFO:tasks.rados.rados.0.smithi084.stdout:3401: finishing write tid 4 to smithi08477203-43 2024-09-20T22:55:32.599 INFO:tasks.rados.rados.0.smithi084.stdout:3401: finishing write tid 5 to smithi08477203-43 2024-09-20T22:55:32.599 INFO:tasks.rados.rados.0.smithi084.stdout:3401: oid 43 updating version 0 to 1933 2024-09-20T22:55:32.599 INFO:tasks.rados.rados.0.smithi084.stdout:3401: oid 43 version 1933 is already newer than 1932 2024-09-20T22:55:32.599 INFO:tasks.rados.rados.0.smithi084.stdout:3401: oid 43 updating version 1933 to 1934 2024-09-20T22:55:32.599 INFO:tasks.rados.rados.0.smithi084.stdout:3401: oid 43 version 1934 is already newer than 1931 2024-09-20T22:55:32.600 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 43 v 1934 (ObjNum 1157 snap 397 seq_num 1157) dirty exists 2024-09-20T22:55:32.600 INFO:tasks.rados.rados.0.smithi084.stdout:3401: left oid 43 (ObjNum 1157 snap 397 seq_num 1157) 2024-09-20T22:55:32.600 INFO:tasks.rados.rados.0.smithi084.stdout:3402: expect (ObjNum 1153 snap 396 seq_num 1153) 2024-09-20T22:55:32.639 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:55:32.661 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:32.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:32.662 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:55:32.763 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:55:32.784 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:32.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:32.785 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:55:32.865 INFO:tasks.rados.rados.0.smithi084.stdout:3403: finishing write tid 1 to smithi08477203-46 2024-09-20T22:55:32.865 INFO:tasks.rados.rados.0.smithi084.stdout:3403: finishing write tid 2 to smithi08477203-46 2024-09-20T22:55:32.865 INFO:tasks.rados.rados.0.smithi084.stdout:3403: finishing write tid 3 to smithi08477203-46 2024-09-20T22:55:32.865 INFO:tasks.rados.rados.0.smithi084.stdout:3403: finishing write tid 4 to smithi08477203-46 2024-09-20T22:55:32.865 INFO:tasks.rados.rados.0.smithi084.stdout:3403: finishing write tid 5 to smithi08477203-46 2024-09-20T22:55:32.865 INFO:tasks.rados.rados.0.smithi084.stdout:3403: oid 46 updating version 0 to 2566 2024-09-20T22:55:32.865 INFO:tasks.rados.rados.0.smithi084.stdout:3403: oid 46 updating version 2566 to 2568 2024-09-20T22:55:32.865 INFO:tasks.rados.rados.0.smithi084.stdout:3403: oid 46 version 2568 is already newer than 2565 2024-09-20T22:55:32.865 INFO:tasks.rados.rados.0.smithi084.stdout:3403: oid 46 version 2568 is already newer than 2567 2024-09-20T22:55:32.865 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 46 v 2568 (ObjNum 1158 snap 397 seq_num 1158) dirty exists 2024-09-20T22:55:32.865 INFO:tasks.rados.rados.0.smithi084.stdout:3403: left oid 46 (ObjNum 1158 snap 397 seq_num 1158) 2024-09-20T22:55:32.886 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:55:32.907 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:32.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:32.908 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:55:33.008 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:55:33.029 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:33.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:33.030 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:55:33.107 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:55:33.108 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:55:33.128 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:55:33.128 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:55:33.131 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:55:33.145 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:55:33.145 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:55:33.154 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:33.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:33.155 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:55:33.161 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-20T22:55:33.161 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T22:55:33.178 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:55:33.178 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:55:33.195 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:55:33.195 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:55:33.212 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:55:33.212 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:55:33.230 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:55:33.231 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:55:33.256 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:55:33.279 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:33.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:33.280 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:55:33.335 INFO:tasks.rados.rados.0.smithi084.stdout:3400: done (4 left) 2024-09-20T22:55:33.335 INFO:tasks.rados.rados.0.smithi084.stdout:3401: done (3 left) 2024-09-20T22:55:33.336 INFO:tasks.rados.rados.0.smithi084.stdout:3402: done (2 left) 2024-09-20T22:55:33.336 INFO:tasks.rados.rados.0.smithi084.stdout:3403: done (1 left) 2024-09-20T22:55:33.336 INFO:tasks.rados.rados.0.smithi084.stdout:3404: done (0 left) 2024-09-20T22:55:33.336 INFO:tasks.rados.rados.0.smithi084.stdout:3405: rollback oid 41 current snap is 398 2024-09-20T22:55:33.336 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 41 to 388 2024-09-20T22:55:33.336 INFO:tasks.rados.rados.0.smithi084.stdout:3406: write (excl) oid 23 current snap is 398 2024-09-20T22:55:33.336 INFO:tasks.rados.rados.0.smithi084.stdout:3406: seq_num 1159 ranges {679313=499247,1622042=683065,2744778=701557} 2024-09-20T22:55:33.342 INFO:tasks.rados.rados.0.smithi084.stdout:3406: writing smithi08477203-23 from 679313 to 1178560 tid 1 2024-09-20T22:55:33.350 INFO:tasks.rados.rados.0.smithi084.stdout:3406: writing smithi08477203-23 from 1622042 to 2305107 tid 2 2024-09-20T22:55:33.357 INFO:tasks.rados.rados.0.smithi084.stdout:3406: writing smithi08477203-23 from 2744778 to 3446335 tid 3 2024-09-20T22:55:33.357 INFO:tasks.rados.rados.0.smithi084.stdout:3407: copy_from oid 8 from oid 20 current snap is 398 2024-09-20T22:55:33.357 INFO:tasks.rados.rados.0.smithi084.stdout:3405: finishing rollback tid 1 to smithi08477203-41 2024-09-20T22:55:33.357 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 41 v 1830 (ObjNum 129 snap 0 seq_num 2091411360) dirty dne 2024-09-20T22:55:33.357 INFO:tasks.rados.rados.0.smithi084.stdout:3406: finishing write tid 1 to smithi08477203-23 2024-09-20T22:55:33.358 INFO:tasks.rados.rados.0.smithi084.stdout:3406: finishing write tid 2 to smithi08477203-23 2024-09-20T22:55:33.358 INFO:tasks.rados.rados.0.smithi084.stdout:3405: done (2 left) 2024-09-20T22:55:33.358 INFO:tasks.rados.rados.0.smithi084.stdout:3408: write oid 28 current snap is 398 2024-09-20T22:55:33.358 INFO:tasks.rados.rados.0.smithi084.stdout:3408: seq_num 1160 ranges {696120=574496,1680524=543076,2888554=512829} 2024-09-20T22:55:33.364 INFO:tasks.rados.rados.0.smithi084.stdout:3408: writing smithi08477203-28 from 696120 to 1270616 tid 1 2024-09-20T22:55:33.370 INFO:tasks.rados.rados.0.smithi084.stdout:3408: writing smithi08477203-28 from 1680524 to 2223600 tid 2 2024-09-20T22:55:33.377 INFO:tasks.rados.rados.0.smithi084.stdout:3408: writing smithi08477203-28 from 2888554 to 3401383 tid 3 2024-09-20T22:55:33.377 INFO:tasks.rados.rados.0.smithi084.stdout:3409: snap_remove snap 381 2024-09-20T22:55:33.380 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:55:33.401 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:33.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:33.402 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:55:33.503 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:55:33.525 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:33.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:33.526 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:55:33.627 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:55:33.648 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:33.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:33.649 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:55:33.749 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:55:33.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:33 smithi154 ceph-mon[33264]: pgmap v1179: 158 pgs: 4 active+clean+snaptrim, 154 active+clean; 556 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 7.6 MiB/s rd, 3.1 MiB/s wr, 15 op/s 2024-09-20T22:55:33.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:33 smithi154 ceph-mon[33264]: 7.8 scrub starts 2024-09-20T22:55:33.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:33 smithi154 ceph-mon[33264]: 7.8 scrub ok 2024-09-20T22:55:33.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:33 smithi154 ceph-mon[33264]: osdmap e852: 8 total, 8 up, 8 in 2024-09-20T22:55:33.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:33 smithi154 ceph-mon[33264]: 4.b scrub starts 2024-09-20T22:55:33.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:33 smithi154 ceph-mon[33264]: 4.b scrub ok 2024-09-20T22:55:33.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:33 smithi154 ceph-mon[33264]: 6.1e scrub starts 2024-09-20T22:55:33.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:33 smithi154 ceph-mon[33264]: 6.1e scrub ok 2024-09-20T22:55:33.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:33 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/2913599167' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:55:33.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:33 smithi154 ceph-mon[33264]: osdmap e853: 8 total, 8 up, 8 in 2024-09-20T22:55:33.770 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:33.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:33.771 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:55:33.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:33 smithi084 ceph-mon[29774]: pgmap v1179: 158 pgs: 4 active+clean+snaptrim, 154 active+clean; 556 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 7.6 MiB/s rd, 3.1 MiB/s wr, 15 op/s 2024-09-20T22:55:33.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:33 smithi084 ceph-mon[29774]: 7.8 scrub starts 2024-09-20T22:55:33.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:33 smithi084 ceph-mon[29774]: 7.8 scrub ok 2024-09-20T22:55:33.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:33 smithi084 ceph-mon[29774]: osdmap e852: 8 total, 8 up, 8 in 2024-09-20T22:55:33.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:33 smithi084 ceph-mon[29774]: 4.b scrub starts 2024-09-20T22:55:33.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:33 smithi084 ceph-mon[29774]: 4.b scrub ok 2024-09-20T22:55:33.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:33 smithi084 ceph-mon[29774]: 6.1e scrub starts 2024-09-20T22:55:33.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:33 smithi084 ceph-mon[29774]: 6.1e scrub ok 2024-09-20T22:55:33.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:33 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/2913599167' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:55:33.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:33 smithi084 ceph-mon[29774]: osdmap e853: 8 total, 8 up, 8 in 2024-09-20T22:55:33.866 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:33 smithi084 ceph-mon[40024]: pgmap v1179: 158 pgs: 4 active+clean+snaptrim, 154 active+clean; 556 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail; 7.6 MiB/s rd, 3.1 MiB/s wr, 15 op/s 2024-09-20T22:55:33.866 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:33 smithi084 ceph-mon[40024]: 7.8 scrub starts 2024-09-20T22:55:33.866 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:33 smithi084 ceph-mon[40024]: 7.8 scrub ok 2024-09-20T22:55:33.866 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:33 smithi084 ceph-mon[40024]: osdmap e852: 8 total, 8 up, 8 in 2024-09-20T22:55:33.866 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:33 smithi084 ceph-mon[40024]: 4.b scrub starts 2024-09-20T22:55:33.866 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:33 smithi084 ceph-mon[40024]: 4.b scrub ok 2024-09-20T22:55:33.866 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:33 smithi084 ceph-mon[40024]: 6.1e scrub starts 2024-09-20T22:55:33.866 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:33 smithi084 ceph-mon[40024]: 6.1e scrub ok 2024-09-20T22:55:33.866 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:33 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/2913599167' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:55:33.866 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:33 smithi084 ceph-mon[40024]: osdmap e853: 8 total, 8 up, 8 in 2024-09-20T22:55:33.872 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:55:33.893 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:33.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:33.894 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:55:33.995 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:55:34.016 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:34.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:34.017 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:55:34.118 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:55:34.139 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:34.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:34.140 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:55:34.241 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:55:34.262 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:34.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:34.263 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:55:34.338 INFO:tasks.rados.rados.0.smithi084.stdout:3409: done (3 left) 2024-09-20T22:55:34.339 INFO:tasks.rados.rados.0.smithi084.stdout:3410: write oid 44 current snap is 398 2024-09-20T22:55:34.339 INFO:tasks.rados.rados.0.smithi084.stdout:3410: seq_num 1161 ranges {712927=649745,2139006=686734} 2024-09-20T22:55:34.346 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:55:34.346 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:55:34.347 INFO:tasks.rados.rados.0.smithi084.stdout:3410: writing smithi08477203-44 from 712927 to 1362672 tid 1 2024-09-20T22:55:34.356 INFO:tasks.rados.rados.0.smithi084.stdout:3410: writing smithi08477203-44 from 2139006 to 2825740 tid 2 2024-09-20T22:55:34.356 INFO:tasks.rados.rados.0.smithi084.stdout:3411: copy_from oid 27 from oid 47 current snap is 398 2024-09-20T22:55:34.356 INFO:tasks.rados.rados.0.smithi084.stdout:3406: finishing write tid 3 to smithi08477203-23 2024-09-20T22:55:34.357 INFO:tasks.rados.rados.0.smithi084.stdout:3412: snap_create 2024-09-20T22:55:34.357 INFO:tasks.rados.rados.0.smithi084.stdout:3406: finishing write tid 4 to smithi08477203-23 2024-09-20T22:55:34.357 INFO:tasks.rados.rados.0.smithi084.stdout:3406: finishing write tid 5 to smithi08477203-23 2024-09-20T22:55:34.357 INFO:tasks.rados.rados.0.smithi084.stdout:3406: oid 23 updating version 0 to 1933 2024-09-20T22:55:34.357 INFO:tasks.rados.rados.0.smithi084.stdout:3406: oid 23 updating version 1933 to 1934 2024-09-20T22:55:34.357 INFO:tasks.rados.rados.0.smithi084.stdout:3406: oid 23 updating version 1934 to 1935 2024-09-20T22:55:34.357 INFO:tasks.rados.rados.0.smithi084.stdout:3406: oid 23 updating version 1935 to 1936 2024-09-20T22:55:34.357 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 23 v 1936 (ObjNum 1159 snap 398 seq_num 1159) dirty exists 2024-09-20T22:55:34.357 INFO:tasks.rados.rados.0.smithi084.stdout:3406: left oid 23 (ObjNum 1159 snap 398 seq_num 1159) 2024-09-20T22:55:34.357 INFO:tasks.rados.rados.0.smithi084.stdout:3407: finishing copy_from racing read to smithi08477203-8 2024-09-20T22:55:34.357 INFO:tasks.rados.rados.0.smithi084.stdout:3407: finishing copy_from to smithi08477203-8 2024-09-20T22:55:34.357 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 8 v 1929 (ObjNum 1113 snap 378 seq_num 1113) dirty exists 2024-09-20T22:55:34.357 INFO:tasks.rados.rados.0.smithi084.stdout:3408: finishing write tid 1 to smithi08477203-28 2024-09-20T22:55:34.358 INFO:tasks.rados.rados.0.smithi084.stdout:3408: finishing write tid 2 to smithi08477203-28 2024-09-20T22:55:34.358 INFO:tasks.rados.rados.0.smithi084.stdout:3408: finishing write tid 3 to smithi08477203-28 2024-09-20T22:55:34.358 INFO:tasks.rados.rados.0.smithi084.stdout:3408: finishing write tid 4 to smithi08477203-28 2024-09-20T22:55:34.358 INFO:tasks.rados.rados.0.smithi084.stdout:3408: finishing write tid 5 to smithi08477203-28 2024-09-20T22:55:34.358 INFO:tasks.rados.rados.0.smithi084.stdout:3408: oid 28 updating version 0 to 2707 2024-09-20T22:55:34.358 INFO:tasks.rados.rados.0.smithi084.stdout:3408: oid 28 updating version 2707 to 2708 2024-09-20T22:55:34.358 INFO:tasks.rados.rados.0.smithi084.stdout:3408: oid 28 version 2708 is already newer than 2705 2024-09-20T22:55:34.358 INFO:tasks.rados.rados.0.smithi084.stdout:3408: oid 28 version 2708 is already newer than 2706 2024-09-20T22:55:34.358 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 28 v 2708 (ObjNum 1160 snap 398 seq_num 1160) dirty exists 2024-09-20T22:55:34.358 INFO:tasks.rados.rados.0.smithi084.stdout:3408: left oid 28 (ObjNum 1160 snap 398 seq_num 1160) 2024-09-20T22:55:34.358 INFO:tasks.rados.rados.0.smithi084.stdout:3410: finishing write tid 1 to smithi08477203-44 2024-09-20T22:55:34.361 INFO:tasks.rados.rados.0.smithi084.stdout:3410: finishing write tid 2 to smithi08477203-44 2024-09-20T22:55:34.361 INFO:tasks.rados.rados.0.smithi084.stdout:3410: finishing write tid 3 to smithi08477203-44 2024-09-20T22:55:34.361 INFO:tasks.rados.rados.0.smithi084.stdout:3410: finishing write tid 4 to smithi08477203-44 2024-09-20T22:55:34.361 INFO:tasks.rados.rados.0.smithi084.stdout:3410: oid 44 updating version 0 to 2716 2024-09-20T22:55:34.361 INFO:tasks.rados.rados.0.smithi084.stdout:3410: oid 44 version 2716 is already newer than 2714 2024-09-20T22:55:34.361 INFO:tasks.rados.rados.0.smithi084.stdout:3410: oid 44 version 2716 is already newer than 2715 2024-09-20T22:55:34.362 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 44 v 2716 (ObjNum 1161 snap 398 seq_num 1161) dirty exists 2024-09-20T22:55:34.362 INFO:tasks.rados.rados.0.smithi084.stdout:3410: left oid 44 (ObjNum 1161 snap 398 seq_num 1161) 2024-09-20T22:55:34.364 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:55:34.371 INFO:tasks.rados.rados.0.smithi084.stdout:3411: finishing copy_from racing read to smithi08477203-27 2024-09-20T22:55:34.379 INFO:tasks.rados.rados.0.smithi084.stdout:3411: finishing copy_from to smithi08477203-27 2024-09-20T22:55:34.379 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 27 v 2481 (ObjNum 1146 snap 392 seq_num 1146) dirty exists 2024-09-20T22:55:34.386 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:34.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:34.387 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:55:34.488 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:55:34.510 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:34.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:34.511 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:55:34.532 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-20T22:55:34.534 INFO:teuthology.orchestra.run.smithi084.stderr:dumped all 2024-09-20T22:55:34.612 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:55:34.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:34 smithi084 ceph-mon[29774]: 7.a scrub starts 2024-09-20T22:55:34.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:34 smithi084 ceph-mon[29774]: 7.a scrub ok 2024-09-20T22:55:34.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:34 smithi084 ceph-mon[29774]: 4.12 scrub starts 2024-09-20T22:55:34.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:34 smithi084 ceph-mon[29774]: 4.12 scrub ok 2024-09-20T22:55:34.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:34 smithi084 ceph-mon[29774]: osdmap e854: 8 total, 8 up, 8 in 2024-09-20T22:55:34.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:34 smithi084 ceph-mon[40024]: 7.a scrub starts 2024-09-20T22:55:34.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:34 smithi084 ceph-mon[40024]: 7.a scrub ok 2024-09-20T22:55:34.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:34 smithi084 ceph-mon[40024]: 4.12 scrub starts 2024-09-20T22:55:34.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:34 smithi084 ceph-mon[40024]: 4.12 scrub ok 2024-09-20T22:55:34.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:34 smithi084 ceph-mon[40024]: osdmap e854: 8 total, 8 up, 8 in 2024-09-20T22:55:34.633 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:34.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:34.634 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:55:34.735 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:55:34.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:34 smithi154 ceph-mon[33264]: 7.a scrub starts 2024-09-20T22:55:34.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:34 smithi154 ceph-mon[33264]: 7.a scrub ok 2024-09-20T22:55:34.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:34 smithi154 ceph-mon[33264]: 4.12 scrub starts 2024-09-20T22:55:34.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:34 smithi154 ceph-mon[33264]: 4.12 scrub ok 2024-09-20T22:55:34.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:34 smithi154 ceph-mon[33264]: osdmap e854: 8 total, 8 up, 8 in 2024-09-20T22:55:34.756 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:34.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:34.757 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:55:34.858 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:55:34.880 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:34.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:34.881 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:55:34.982 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:55:35.003 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:35.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:35.004 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:55:35.105 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:55:35.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:35.127 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:35.127 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:55:35.228 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:55:35.249 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:35.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:35.251 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:55:35.335 INFO:tasks.rados.rados.0.smithi084.stdout:3406: done (5 left) 2024-09-20T22:55:35.335 INFO:tasks.rados.rados.0.smithi084.stdout:3407: done (4 left) 2024-09-20T22:55:35.335 INFO:tasks.rados.rados.0.smithi084.stdout:3408: done (3 left) 2024-09-20T22:55:35.335 INFO:tasks.rados.rados.0.smithi084.stdout:3410: done (2 left) 2024-09-20T22:55:35.335 INFO:tasks.rados.rados.0.smithi084.stdout:3411: done (1 left) 2024-09-20T22:55:35.335 INFO:tasks.rados.rados.0.smithi084.stdout:3412: done (0 left) 2024-09-20T22:55:35.335 INFO:tasks.rados.rados.0.smithi084.stdout:3413: write (excl) oid 12 current snap is 399 2024-09-20T22:55:35.336 INFO:tasks.rados.rados.0.smithi084.stdout:3413: seq_num 1162 ranges {729734=724994,2197488=430392,3227047=302687} 2024-09-20T22:55:35.343 INFO:tasks.rados.rados.0.smithi084.stdout:3413: writing smithi08477203-12 from 729734 to 1454728 tid 1 2024-09-20T22:55:35.349 INFO:tasks.rados.rados.0.smithi084.stdout:3413: writing smithi08477203-12 from 2197488 to 2627880 tid 2 2024-09-20T22:55:35.352 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:55:35.355 INFO:tasks.rados.rados.0.smithi084.stdout:3413: writing smithi08477203-12 from 3227047 to 3529734 tid 3 2024-09-20T22:55:35.355 INFO:tasks.rados.rados.0.smithi084.stdout:3414: copy_from oid 5 from oid 1 current snap is 399 2024-09-20T22:55:35.355 INFO:tasks.rados.rados.0.smithi084.stdout:3413: finishing write tid 1 to smithi08477203-12 2024-09-20T22:55:35.355 INFO:tasks.rados.rados.0.smithi084.stdout:3413: finishing write tid 2 to smithi08477203-12 2024-09-20T22:55:35.355 INFO:tasks.rados.rados.0.smithi084.stdout:3415: snap_remove snap 386 2024-09-20T22:55:35.376 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:35.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:35.377 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:55:35.478 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:55:35.500 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:35.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:35.500 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:55:35.601 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:55:35.614 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:35 smithi084 ceph-mon[40024]: 3.2 scrub starts 2024-09-20T22:55:35.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:35 smithi084 ceph-mon[40024]: 3.2 scrub ok 2024-09-20T22:55:35.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:35 smithi084 ceph-mon[40024]: 3.f scrub starts 2024-09-20T22:55:35.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:35 smithi084 ceph-mon[40024]: 3.f scrub ok 2024-09-20T22:55:35.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:35 smithi084 ceph-mon[40024]: pgmap v1182: 158 pgs: 1 active+clean+snaptrim, 157 active+clean; 586 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 5.6 MiB/s rd, 5.4 MiB/s wr, 22 op/s 2024-09-20T22:55:35.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:35 smithi084 ceph-mon[40024]: from='client.17211 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T22:55:35.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:35 smithi084 ceph-mon[40024]: 7.4 scrub starts 2024-09-20T22:55:35.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:35 smithi084 ceph-mon[40024]: 7.4 scrub ok 2024-09-20T22:55:35.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:35 smithi084 ceph-mon[40024]: 5.19 scrub starts 2024-09-20T22:55:35.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:35 smithi084 ceph-mon[40024]: 5.19 scrub ok 2024-09-20T22:55:35.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:35 smithi084 ceph-mon[40024]: osdmap e855: 8 total, 8 up, 8 in 2024-09-20T22:55:35.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:35 smithi084 ceph-mon[29774]: 3.2 scrub starts 2024-09-20T22:55:35.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:35 smithi084 ceph-mon[29774]: 3.2 scrub ok 2024-09-20T22:55:35.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:35 smithi084 ceph-mon[29774]: 3.f scrub starts 2024-09-20T22:55:35.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:35 smithi084 ceph-mon[29774]: 3.f scrub ok 2024-09-20T22:55:35.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:35 smithi084 ceph-mon[29774]: pgmap v1182: 158 pgs: 1 active+clean+snaptrim, 157 active+clean; 586 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 5.6 MiB/s rd, 5.4 MiB/s wr, 22 op/s 2024-09-20T22:55:35.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:35 smithi084 ceph-mon[29774]: from='client.17211 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T22:55:35.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:35 smithi084 ceph-mon[29774]: 7.4 scrub starts 2024-09-20T22:55:35.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:35 smithi084 ceph-mon[29774]: 7.4 scrub ok 2024-09-20T22:55:35.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:35 smithi084 ceph-mon[29774]: 5.19 scrub starts 2024-09-20T22:55:35.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:35 smithi084 ceph-mon[29774]: 5.19 scrub ok 2024-09-20T22:55:35.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:35 smithi084 ceph-mon[29774]: osdmap e855: 8 total, 8 up, 8 in 2024-09-20T22:55:35.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:35.624 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:35.624 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:55:35.725 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:55:35.746 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:35.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:35.747 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:55:35.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:35 smithi154 ceph-mon[33264]: 3.2 scrub starts 2024-09-20T22:55:35.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:35 smithi154 ceph-mon[33264]: 3.2 scrub ok 2024-09-20T22:55:35.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:35 smithi154 ceph-mon[33264]: 3.f scrub starts 2024-09-20T22:55:35.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:35 smithi154 ceph-mon[33264]: 3.f scrub ok 2024-09-20T22:55:35.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:35 smithi154 ceph-mon[33264]: pgmap v1182: 158 pgs: 1 active+clean+snaptrim, 157 active+clean; 586 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 5.6 MiB/s rd, 5.4 MiB/s wr, 22 op/s 2024-09-20T22:55:35.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:35 smithi154 ceph-mon[33264]: from='client.17211 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T22:55:35.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:35 smithi154 ceph-mon[33264]: 7.4 scrub starts 2024-09-20T22:55:35.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:35 smithi154 ceph-mon[33264]: 7.4 scrub ok 2024-09-20T22:55:35.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:35 smithi154 ceph-mon[33264]: 5.19 scrub starts 2024-09-20T22:55:35.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:35 smithi154 ceph-mon[33264]: 5.19 scrub ok 2024-09-20T22:55:35.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:35 smithi154 ceph-mon[33264]: osdmap e855: 8 total, 8 up, 8 in 2024-09-20T22:55:35.803 INFO:teuthology.orchestra.run.smithi084.stdout:{"pg_ready":true,"pg_map":{"version":1183,"stamp":"2024-09-20T22:55:34.330803+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":614647829,"num_objects":476,"num_object_clones":227,"num_object_copies":1428,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":476,"num_whiteouts":13,"num_read":7864,"num_read_kb":6174047,"num_write":4974,"num_write_kb":1616032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1440,"num_bytes_recovered":730756107,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":15179,"ondisk_log_size":15179,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":474,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":1884572,"kb_used_data":1425508,"kb_used_omap":12,"kb_used_meta":458867,"kb_avail":748076644,"statfs":{"total":767960285184,"available":766030483456,"internally_reserved":0,"allocated":1459720192,"data_stored":1960484910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":469880368},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":32027470,"num_objects":13,"num_object_clones":13,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":39,"num_read_kb":22876,"num_write":53,"num_write_kb":22184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.993977"},"pg_stats":[{"pgid":"3.1f","version":"86'2","reported_seq":970,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-09-20T22:55:33.333563+0000","last_change":"2024-09-20T22:54:40.830995+0000","last_active":"2024-09-20T22:55:33.333563+0000","last_peered":"2024-09-20T22:55:33.333563+0000","last_clean":"2024-09-20T22:55:33.333563+0000","last_became_active":"2024-09-20T22:47:49.222114+0000","last_became_peered":"2024-09-20T22:47:49.222114+0000","last_unstale":"2024-09-20T22:55:33.333563+0000","last_undegraded":"2024-09-20T22:55:33.333563+0000","last_fullsized":"2024-09-20T22:55:33.333563+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'2","last_scrub_stamp":"2024-09-20T22:54:40.830920+0000","last_deep_scrub":"86'2","last_deep_scrub_stamp":"2024-09-20T22:42:39.664421+0000","last_clean_scrub_stamp":"2024-09-20T22:54:40.830920+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:47.276232+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":436,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"625'15","reported_seq":537,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-09-20T22:55:33.366837+0000","last_change":"2024-09-20T22:55:17.341879+0000","last_active":"2024-09-20T22:55:33.366837+0000","last_peered":"2024-09-20T22:55:33.366837+0000","last_clean":"2024-09-20T22:55:33.366837+0000","last_became_active":"2024-09-20T22:55:17.341556+0000","last_became_peered":"2024-09-20T22:55:17.341556+0000","last_unstale":"2024-09-20T22:55:33.366837+0000","last_undegraded":"2024-09-20T22:55:33.366837+0000","last_fullsized":"2024-09-20T22:55:33.366837+0000","mapping_epoch":836,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":837,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'15","last_scrub_stamp":"2024-09-20T22:54:43.123216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:54:43.123216+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:51.877398+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":163,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-09-20T22:55:33.332325+0000","last_change":"2024-09-20T22:54:17.338438+0000","last_active":"2024-09-20T22:55:33.332325+0000","last_peered":"2024-09-20T22:55:33.332325+0000","last_clean":"2024-09-20T22:55:33.332325+0000","last_became_active":"2024-09-20T22:54:17.337980+0000","last_became_peered":"2024-09-20T22:54:17.337980+0000","last_unstale":"2024-09-20T22:55:33.332325+0000","last_undegraded":"2024-09-20T22:55:33.332325+0000","last_fullsized":"2024-09-20T22:55:33.332325+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:54:15.974498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:54:15.974498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":827,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-09-20T22:55:33.349263+0000","last_change":"2024-09-20T22:55:17.342349+0000","last_active":"2024-09-20T22:55:33.349263+0000","last_peered":"2024-09-20T22:55:33.349263+0000","last_clean":"2024-09-20T22:55:33.349263+0000","last_became_active":"2024-09-20T22:55:17.342068+0000","last_became_peered":"2024-09-20T22:55:17.342068+0000","last_unstale":"2024-09-20T22:55:33.349263+0000","last_undegraded":"2024-09-20T22:55:33.349263+0000","last_fullsized":"2024-09-20T22:55:33.349263+0000","mapping_epoch":836,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":837,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:58.481327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:51:36.730015+0000","last_clean_scrub_stamp":"2024-09-20T22:53:58.481327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":1613,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-09-20T22:55:33.344858+0000","last_change":"2024-09-20T22:54:15.238419+0000","last_active":"2024-09-20T22:55:33.344858+0000","last_peered":"2024-09-20T22:55:33.344858+0000","last_clean":"2024-09-20T22:55:33.344858+0000","last_became_active":"2024-09-20T22:41:44.832099+0000","last_became_peered":"2024-09-20T22:41:44.832099+0000","last_unstale":"2024-09-20T22:55:33.344858+0000","last_undegraded":"2024-09-20T22:55:33.344858+0000","last_fullsized":"2024-09-20T22:55:33.344858+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:54:15.238344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:54:15.238344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":1618,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-09-20T22:55:33.346562+0000","last_change":"2024-09-20T22:54:47.427999+0000","last_active":"2024-09-20T22:55:33.346562+0000","last_peered":"2024-09-20T22:55:33.346562+0000","last_clean":"2024-09-20T22:55:33.346562+0000","last_became_active":"2024-09-20T22:41:38.157298+0000","last_became_peered":"2024-09-20T22:41:38.157298+0000","last_unstale":"2024-09-20T22:55:33.346562+0000","last_undegraded":"2024-09-20T22:55:33.346562+0000","last_fullsized":"2024-09-20T22:55:33.346562+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:54:47.427927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:54:47.427927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:51.157529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"625'27","reported_seq":1700,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-09-20T22:55:33.344614+0000","last_change":"2024-09-20T22:54:45.390249+0000","last_active":"2024-09-20T22:55:33.344614+0000","last_peered":"2024-09-20T22:55:33.344614+0000","last_clean":"2024-09-20T22:55:33.344614+0000","last_became_active":"2024-09-20T22:47:49.218798+0000","last_became_peered":"2024-09-20T22:47:49.218798+0000","last_unstale":"2024-09-20T22:55:33.344614+0000","last_undegraded":"2024-09-20T22:55:33.344614+0000","last_fullsized":"2024-09-20T22:55:33.344614+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'27","last_scrub_stamp":"2024-09-20T22:54:45.390169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:54:45.390169+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:51.072485+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":827,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-09-20T22:55:33.366679+0000","last_change":"2024-09-20T22:55:25.730415+0000","last_active":"2024-09-20T22:55:33.366679+0000","last_peered":"2024-09-20T22:55:33.366679+0000","last_clean":"2024-09-20T22:55:33.366679+0000","last_became_active":"2024-09-20T22:55:17.341160+0000","last_became_peered":"2024-09-20T22:55:17.341160+0000","last_unstale":"2024-09-20T22:55:33.366679+0000","last_undegraded":"2024-09-20T22:55:33.366679+0000","last_fullsized":"2024-09-20T22:55:33.366679+0000","mapping_epoch":836,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":837,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:55:25.730335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:55:25.730335+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:56:43.672450+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":1637,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-09-20T22:55:33.348982+0000","last_change":"2024-09-20T22:53:56.980951+0000","last_active":"2024-09-20T22:55:33.348982+0000","last_peered":"2024-09-20T22:55:33.348982+0000","last_clean":"2024-09-20T22:55:33.348982+0000","last_became_active":"2024-09-20T22:48:05.242059+0000","last_became_peered":"2024-09-20T22:48:05.242059+0000","last_unstale":"2024-09-20T22:55:33.348982+0000","last_undegraded":"2024-09-20T22:55:33.348982+0000","last_fullsized":"2024-09-20T22:55:33.348982+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:56.980885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:46:00.999939+0000","last_clean_scrub_stamp":"2024-09-20T22:53:56.980885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"625'15","reported_seq":529,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-09-20T22:55:33.346303+0000","last_change":"2024-09-20T22:55:17.348852+0000","last_active":"2024-09-20T22:55:33.346303+0000","last_peered":"2024-09-20T22:55:33.346303+0000","last_clean":"2024-09-20T22:55:33.346303+0000","last_became_active":"2024-09-20T22:55:17.348531+0000","last_became_peered":"2024-09-20T22:55:17.348531+0000","last_unstale":"2024-09-20T22:55:33.346303+0000","last_undegraded":"2024-09-20T22:55:33.346303+0000","last_fullsized":"2024-09-20T22:55:33.346303+0000","mapping_epoch":836,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":837,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'15","last_scrub_stamp":"2024-09-20T22:53:51.486202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:53:51.486202+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":1629,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-09-20T22:55:33.348734+0000","last_change":"2024-09-20T22:54:21.027796+0000","last_active":"2024-09-20T22:55:33.348734+0000","last_peered":"2024-09-20T22:55:33.348734+0000","last_clean":"2024-09-20T22:55:33.348734+0000","last_became_active":"2024-09-20T22:47:49.227379+0000","last_became_peered":"2024-09-20T22:47:49.227379+0000","last_unstale":"2024-09-20T22:55:33.348734+0000","last_undegraded":"2024-09-20T22:55:33.348734+0000","last_fullsized":"2024-09-20T22:55:33.348734+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:54:21.027723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:54:21.027723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":978,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-09-20T22:55:33.332291+0000","last_change":"2024-09-20T22:54:17.340772+0000","last_active":"2024-09-20T22:55:33.332291+0000","last_peered":"2024-09-20T22:55:33.332291+0000","last_clean":"2024-09-20T22:55:33.332291+0000","last_became_active":"2024-09-20T22:54:17.340467+0000","last_became_peered":"2024-09-20T22:54:17.340467+0000","last_unstale":"2024-09-20T22:55:33.332291+0000","last_undegraded":"2024-09-20T22:55:33.332291+0000","last_fullsized":"2024-09-20T22:55:33.332291+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:35.920968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:49:08.664407+0000","last_clean_scrub_stamp":"2024-09-20T22:53:35.920968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":1649,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-09-20T22:55:33.348878+0000","last_change":"2024-09-20T22:54:17.342948+0000","last_active":"2024-09-20T22:55:33.348878+0000","last_peered":"2024-09-20T22:55:33.348878+0000","last_clean":"2024-09-20T22:55:33.348878+0000","last_became_active":"2024-09-20T22:54:17.342619+0000","last_became_peered":"2024-09-20T22:54:17.342619+0000","last_unstale":"2024-09-20T22:55:33.348878+0000","last_undegraded":"2024-09-20T22:55:33.348878+0000","last_fullsized":"2024-09-20T22:55:33.348878+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:31.092915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:53:31.092915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"625'9","reported_seq":531,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-09-20T22:55:33.345083+0000","last_change":"2024-09-20T22:55:17.350631+0000","last_active":"2024-09-20T22:55:33.345083+0000","last_peered":"2024-09-20T22:55:33.345083+0000","last_clean":"2024-09-20T22:55:33.345083+0000","last_became_active":"2024-09-20T22:55:17.350145+0000","last_became_peered":"2024-09-20T22:55:17.350145+0000","last_unstale":"2024-09-20T22:55:33.345083+0000","last_undegraded":"2024-09-20T22:55:33.345083+0000","last_fullsized":"2024-09-20T22:55:33.345083+0000","mapping_epoch":836,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":837,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'9","last_scrub_stamp":"2024-09-20T22:53:09.623458+0000","last_deep_scrub":"181'7","last_deep_scrub_stamp":"2024-09-20T22:48:41.049381+0000","last_clean_scrub_stamp":"2024-09-20T22:53:09.623458+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":8,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":1661,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-09-20T22:55:33.369305+0000","last_change":"2024-09-20T22:54:19.572825+0000","last_active":"2024-09-20T22:55:33.369305+0000","last_peered":"2024-09-20T22:55:33.369305+0000","last_clean":"2024-09-20T22:55:33.369305+0000","last_became_active":"2024-09-20T22:54:17.331217+0000","last_became_peered":"2024-09-20T22:54:17.331217+0000","last_unstale":"2024-09-20T22:55:33.369305+0000","last_undegraded":"2024-09-20T22:55:33.369305+0000","last_fullsized":"2024-09-20T22:55:33.369305+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:54:19.572737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:47:40.361958+0000","last_clean_scrub_stamp":"2024-09-20T22:54:19.572737+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":1628,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-09-20T22:55:33.348544+0000","last_change":"2024-09-20T22:54:17.340038+0000","last_active":"2024-09-20T22:55:33.348544+0000","last_peered":"2024-09-20T22:55:33.348544+0000","last_clean":"2024-09-20T22:55:33.348544+0000","last_became_active":"2024-09-20T22:54:17.339658+0000","last_became_peered":"2024-09-20T22:54:17.339658+0000","last_unstale":"2024-09-20T22:55:33.348544+0000","last_undegraded":"2024-09-20T22:55:33.348544+0000","last_fullsized":"2024-09-20T22:55:33.348544+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:52.116243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:53:52.116243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":54,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-09-20T22:55:33.366905+0000","last_change":"2024-09-20T22:55:32.840041+0000","last_active":"2024-09-20T22:55:33.366905+0000","last_peered":"2024-09-20T22:55:33.366905+0000","last_clean":"2024-09-20T22:55:33.366905+0000","last_became_active":"2024-09-20T22:55:17.343025+0000","last_became_peered":"2024-09-20T22:55:17.343025+0000","last_unstale":"2024-09-20T22:55:33.366905+0000","last_undegraded":"2024-09-20T22:55:33.366905+0000","last_fullsized":"2024-09-20T22:55:33.366905+0000","mapping_epoch":836,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":837,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:55:32.839935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:55:32.839935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:56:42.578949+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":972,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-09-20T22:55:33.333167+0000","last_change":"2024-09-20T22:55:06.636634+0000","last_active":"2024-09-20T22:55:33.333167+0000","last_peered":"2024-09-20T22:55:33.333167+0000","last_clean":"2024-09-20T22:55:33.333167+0000","last_became_active":"2024-09-20T22:48:05.233687+0000","last_became_peered":"2024-09-20T22:48:05.233687+0000","last_unstale":"2024-09-20T22:55:33.333167+0000","last_undegraded":"2024-09-20T22:55:33.333167+0000","last_fullsized":"2024-09-20T22:55:33.333167+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:55:06.636534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:49:53.933601+0000","last_clean_scrub_stamp":"2024-09-20T22:55:06.636534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:56:19.432625+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"625'34","reported_seq":1738,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-09-20T22:55:33.349563+0000","last_change":"2024-09-20T22:55:04.302057+0000","last_active":"2024-09-20T22:55:33.349563+0000","last_peered":"2024-09-20T22:55:33.349563+0000","last_clean":"2024-09-20T22:55:33.349563+0000","last_became_active":"2024-09-20T22:54:17.331250+0000","last_became_peered":"2024-09-20T22:54:17.331250+0000","last_unstale":"2024-09-20T22:55:33.349563+0000","last_undegraded":"2024-09-20T22:55:33.349563+0000","last_fullsized":"2024-09-20T22:55:33.349563+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'34","last_scrub_stamp":"2024-09-20T22:55:04.301955+0000","last_deep_scrub":"113'22","last_deep_scrub_stamp":"2024-09-20T22:43:12.191878+0000","last_clean_scrub_stamp":"2024-09-20T22:55:04.301955+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:56:29.317286+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":171,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-09-20T22:55:33.332698+0000","last_change":"2024-09-20T22:54:41.984008+0000","last_active":"2024-09-20T22:55:33.332698+0000","last_peered":"2024-09-20T22:55:33.332698+0000","last_clean":"2024-09-20T22:55:33.332698+0000","last_became_active":"2024-09-20T22:54:17.349501+0000","last_became_peered":"2024-09-20T22:54:17.349501+0000","last_unstale":"2024-09-20T22:55:33.332698+0000","last_undegraded":"2024-09-20T22:55:33.332698+0000","last_fullsized":"2024-09-20T22:55:33.332698+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:54:41.983930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:50:40.784154+0000","last_clean_scrub_stamp":"2024-09-20T22:54:41.983930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:43.474654+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":1606,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-09-20T22:55:33.347015+0000","last_change":"2024-09-20T22:54:02.131730+0000","last_active":"2024-09-20T22:55:33.347015+0000","last_peered":"2024-09-20T22:55:33.347015+0000","last_clean":"2024-09-20T22:55:33.347015+0000","last_became_active":"2024-09-20T22:41:44.832131+0000","last_became_peered":"2024-09-20T22:41:44.832131+0000","last_unstale":"2024-09-20T22:55:33.347015+0000","last_undegraded":"2024-09-20T22:55:33.347015+0000","last_fullsized":"2024-09-20T22:55:33.347015+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:54:02.131659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:45:25.002199+0000","last_clean_scrub_stamp":"2024-09-20T22:54:02.131659+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":835,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-09-20T22:55:33.333568+0000","last_change":"2024-09-20T22:55:29.864586+0000","last_active":"2024-09-20T22:55:33.333568+0000","last_peered":"2024-09-20T22:55:33.333568+0000","last_clean":"2024-09-20T22:55:33.333568+0000","last_became_active":"2024-09-20T22:55:17.346400+0000","last_became_peered":"2024-09-20T22:55:17.346400+0000","last_unstale":"2024-09-20T22:55:33.333568+0000","last_undegraded":"2024-09-20T22:55:33.333568+0000","last_fullsized":"2024-09-20T22:55:33.333568+0000","mapping_epoch":836,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":837,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:55:29.864513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:50:41.885844+0000","last_clean_scrub_stamp":"2024-09-20T22:55:29.864513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:56:51.327377+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"625'18","reported_seq":1735,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-09-20T22:55:33.345553+0000","last_change":"2024-09-20T22:54:18.284372+0000","last_active":"2024-09-20T22:55:33.345553+0000","last_peered":"2024-09-20T22:55:33.345553+0000","last_clean":"2024-09-20T22:55:33.345553+0000","last_became_active":"2024-09-20T22:54:17.333427+0000","last_became_peered":"2024-09-20T22:54:17.333427+0000","last_unstale":"2024-09-20T22:55:33.345553+0000","last_undegraded":"2024-09-20T22:55:33.345553+0000","last_fullsized":"2024-09-20T22:55:33.345553+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'18","last_scrub_stamp":"2024-09-20T22:54:18.284289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:54:18.284289+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":1100,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":553,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-09-20T22:55:33.345884+0000","last_change":"2024-09-20T22:55:17.349454+0000","last_active":"2024-09-20T22:55:33.345884+0000","last_peered":"2024-09-20T22:55:33.345884+0000","last_clean":"2024-09-20T22:55:33.345884+0000","last_became_active":"2024-09-20T22:55:17.349132+0000","last_became_peered":"2024-09-20T22:55:17.349132+0000","last_unstale":"2024-09-20T22:55:33.345884+0000","last_undegraded":"2024-09-20T22:55:33.345884+0000","last_fullsized":"2024-09-20T22:55:33.345884+0000","mapping_epoch":836,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":837,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:45.738275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:52:45.738275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"154'2","reported_seq":1620,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-09-20T22:55:33.370780+0000","last_change":"2024-09-20T22:55:06.675422+0000","last_active":"2024-09-20T22:55:33.370780+0000","last_peered":"2024-09-20T22:55:33.370780+0000","last_clean":"2024-09-20T22:55:33.370780+0000","last_became_active":"2024-09-20T22:41:44.199822+0000","last_became_peered":"2024-09-20T22:41:44.199822+0000","last_unstale":"2024-09-20T22:55:33.370780+0000","last_undegraded":"2024-09-20T22:55:33.370780+0000","last_fullsized":"2024-09-20T22:55:33.370780+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"154'2","last_scrub_stamp":"2024-09-20T22:55:06.675346+0000","last_deep_scrub":"154'2","last_deep_scrub_stamp":"2024-09-20T22:43:54.134066+0000","last_clean_scrub_stamp":"2024-09-20T22:55:06.675346+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:56:25.687805+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"67'1","reported_seq":175,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-09-20T22:55:33.332236+0000","last_change":"2024-09-20T22:54:49.003707+0000","last_active":"2024-09-20T22:55:33.332236+0000","last_peered":"2024-09-20T22:55:33.332236+0000","last_clean":"2024-09-20T22:55:33.332236+0000","last_became_active":"2024-09-20T22:54:17.348571+0000","last_became_peered":"2024-09-20T22:54:17.348571+0000","last_unstale":"2024-09-20T22:55:33.332236+0000","last_undegraded":"2024-09-20T22:55:33.332236+0000","last_fullsized":"2024-09-20T22:55:33.332236+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-09-20T22:54:49.003633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:54:49.003633+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:58.739129+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"625'25","reported_seq":1001,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-09-20T22:55:33.333974+0000","last_change":"2024-09-20T22:55:26.646306+0000","last_active":"2024-09-20T22:55:33.333974+0000","last_peered":"2024-09-20T22:55:33.333974+0000","last_clean":"2024-09-20T22:55:33.333974+0000","last_became_active":"2024-09-20T22:47:49.232181+0000","last_became_peered":"2024-09-20T22:47:49.232181+0000","last_unstale":"2024-09-20T22:55:33.333974+0000","last_undegraded":"2024-09-20T22:55:33.333974+0000","last_fullsized":"2024-09-20T22:55:33.333974+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'25","last_scrub_stamp":"2024-09-20T22:55:26.646181+0000","last_deep_scrub":"151'15","last_deep_scrub_stamp":"2024-09-20T22:43:50.715761+0000","last_clean_scrub_stamp":"2024-09-20T22:55:26.646181+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:56:35.462665+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"661'11","reported_seq":1845,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-09-20T22:55:33.367788+0000","last_change":"2024-09-20T22:55:06.848372+0000","last_active":"2024-09-20T22:55:33.367788+0000","last_peered":"2024-09-20T22:55:33.367788+0000","last_clean":"2024-09-20T22:55:33.367788+0000","last_became_active":"2024-09-20T22:48:05.244270+0000","last_became_peered":"2024-09-20T22:48:05.244270+0000","last_unstale":"2024-09-20T22:55:33.367788+0000","last_undegraded":"2024-09-20T22:55:33.367788+0000","last_fullsized":"2024-09-20T22:55:33.367788+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"661'11","last_scrub_stamp":"2024-09-20T22:55:06.848298+0000","last_deep_scrub":"661'11","last_deep_scrub_stamp":"2024-09-20T22:52:21.907943+0000","last_clean_scrub_stamp":"2024-09-20T22:55:06.848298+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:56:12.891978+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.c","version":"853'2708","reported_seq":6334,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-09-20T22:55:33.395385+0000","last_change":"2024-09-20T22:55:25.350795+0000","last_active":"2024-09-20T22:55:33.395385+0000","last_peered":"2024-09-20T22:55:33.395385+0000","last_clean":"2024-09-20T22:55:33.395385+0000","last_became_active":"2024-09-20T22:42:52.404027+0000","last_became_peered":"2024-09-20T22:42:52.404027+0000","last_unstale":"2024-09-20T22:55:33.395385+0000","last_undegraded":"2024-09-20T22:55:33.395385+0000","last_fullsized":"2024-09-20T22:55:33.395385+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"820'2634","last_scrub_stamp":"2024-09-20T22:55:03.827253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:55:03.827253+0000","objects_scrubbed":27,"log_size":2708,"log_dups_size":0,"ondisk_log_size":2708,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:56:27.386351+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0.0095433570000000006,"stat_sum":{"num_bytes":71335610,"num_objects":30,"num_object_clones":23,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":1,"num_read":976,"num_read_kb":1028137,"num_write":657,"num_write_kb":262205,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"13e","length":"b"},{"start":"14b","length":"3"},{"start":"150","length":"5"},{"start":"156","length":"1"},{"start":"159","length":"1"},{"start":"15b","length":"1"},{"start":"15d","length":"3"},{"start":"161","length":"5"},{"start":"167","length":"4"},{"start":"16d","length":"1"},{"start":"170","length":"2"},{"start":"178","length":"1"}]},{"pgid":"1.a","version":"77'128","reported_seq":1886,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-09-20T22:55:33.349500+0000","last_change":"2024-09-20T22:53:28.095343+0000","last_active":"2024-09-20T22:55:33.349500+0000","last_peered":"2024-09-20T22:55:33.349500+0000","last_clean":"2024-09-20T22:55:33.349500+0000","last_became_active":"2024-09-20T22:42:52.407791+0000","last_became_peered":"2024-09-20T22:42:52.407791+0000","last_unstale":"2024-09-20T22:55:33.349500+0000","last_undegraded":"2024-09-20T22:55:33.349500+0000","last_fullsized":"2024-09-20T22:55:33.349500+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:53:28.095266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:53:28.095266+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"625'27","reported_seq":187,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-09-20T22:55:33.333119+0000","last_change":"2024-09-20T22:54:22.820985+0000","last_active":"2024-09-20T22:55:33.333119+0000","last_peered":"2024-09-20T22:55:33.333119+0000","last_clean":"2024-09-20T22:55:33.333119+0000","last_became_active":"2024-09-20T22:54:17.347544+0000","last_became_peered":"2024-09-20T22:54:17.347544+0000","last_unstale":"2024-09-20T22:55:33.333119+0000","last_undegraded":"2024-09-20T22:55:33.333119+0000","last_fullsized":"2024-09-20T22:55:33.333119+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'27","last_scrub_stamp":"2024-09-20T22:54:22.820824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:54:22.820824+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:36.004359+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":1630,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-09-20T22:55:33.343894+0000","last_change":"2024-09-20T22:55:09.369982+0000","last_active":"2024-09-20T22:55:33.343894+0000","last_peered":"2024-09-20T22:55:33.343894+0000","last_clean":"2024-09-20T22:55:33.343894+0000","last_became_active":"2024-09-20T22:54:17.341270+0000","last_became_peered":"2024-09-20T22:54:17.341270+0000","last_unstale":"2024-09-20T22:55:33.343894+0000","last_undegraded":"2024-09-20T22:55:33.343894+0000","last_fullsized":"2024-09-20T22:55:33.343894+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:55:09.369892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:49:12.070675+0000","last_clean_scrub_stamp":"2024-09-20T22:55:09.369892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:56:31.694873+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"838'13","reported_seq":665,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-09-20T22:55:33.348305+0000","last_change":"2024-09-20T22:55:17.343867+0000","last_active":"2024-09-20T22:55:33.348305+0000","last_peered":"2024-09-20T22:55:33.348305+0000","last_clean":"2024-09-20T22:55:33.348305+0000","last_became_active":"2024-09-20T22:55:17.343505+0000","last_became_peered":"2024-09-20T22:55:17.343505+0000","last_unstale":"2024-09-20T22:55:33.348305+0000","last_undegraded":"2024-09-20T22:55:33.348305+0000","last_fullsized":"2024-09-20T22:55:33.348305+0000","mapping_epoch":836,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":837,"parent":"0.0","parent_split_bits":0,"last_scrub":"408'12","last_scrub_stamp":"2024-09-20T22:54:22.471772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:54:22.471772+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:39.928604+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":1638,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-09-20T22:55:33.348685+0000","last_change":"2024-09-20T22:54:22.078071+0000","last_active":"2024-09-20T22:55:33.348685+0000","last_peered":"2024-09-20T22:55:33.348685+0000","last_clean":"2024-09-20T22:55:33.348685+0000","last_became_active":"2024-09-20T22:54:17.340327+0000","last_became_peered":"2024-09-20T22:54:17.340327+0000","last_unstale":"2024-09-20T22:55:33.348685+0000","last_undegraded":"2024-09-20T22:55:33.348685+0000","last_fullsized":"2024-09-20T22:55:33.348685+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:54:22.077994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:54:22.077994+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3","version":"849'426","reported_seq":4057,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-09-20T22:55:33.344698+0000","last_change":"2024-09-20T22:55:29.475492+0000","last_active":"2024-09-20T22:55:33.344698+0000","last_peered":"2024-09-20T22:55:33.344698+0000","last_clean":"2024-09-20T22:55:33.344698+0000","last_became_active":"2024-09-20T22:49:05.221402+0000","last_became_peered":"2024-09-20T22:49:05.221402+0000","last_unstale":"2024-09-20T22:55:33.344698+0000","last_undegraded":"2024-09-20T22:55:33.344698+0000","last_fullsized":"2024-09-20T22:55:33.344698+0000","mapping_epoch":404,"log_start":"456'129","ondisk_log_start":"456'129","created":94,"last_epoch_clean":465,"parent":"0.0","parent_split_bits":0,"last_scrub":"849'426","last_scrub_stamp":"2024-09-20T22:55:29.475387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:55:29.475387+0000","objects_scrubbed":9,"log_size":297,"log_dups_size":149,"ondisk_log_size":297,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:56:51.001077+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.15372964,"stat_sum":{"num_bytes":15025132,"num_objects":9,"num_object_clones":8,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":1,"num_read":125,"num_read_kb":153859,"num_write":92,"num_write_kb":30819,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":20196357,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"13e","length":"b"},{"start":"14b","length":"3"},{"start":"150","length":"5"},{"start":"156","length":"1"},{"start":"159","length":"1"},{"start":"15b","length":"1"},{"start":"15d","length":"3"},{"start":"161","length":"5"},{"start":"167","length":"4"},{"start":"16d","length":"1"},{"start":"170","length":"2"},{"start":"178","length":"1"}]},{"pgid":"1.5","version":"77'128","reported_seq":910,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-09-20T22:55:33.348342+0000","last_change":"2024-09-20T22:55:17.345787+0000","last_active":"2024-09-20T22:55:33.348342+0000","last_peered":"2024-09-20T22:55:33.348342+0000","last_clean":"2024-09-20T22:55:33.348342+0000","last_became_active":"2024-09-20T22:55:17.345460+0000","last_became_peered":"2024-09-20T22:55:17.345460+0000","last_unstale":"2024-09-20T22:55:33.348342+0000","last_undegraded":"2024-09-20T22:55:33.348342+0000","last_fullsized":"2024-09-20T22:55:33.348342+0000","mapping_epoch":836,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":837,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:55:04.973719+0000","last_deep_scrub":"77'128","last_deep_scrub_stamp":"2024-09-20T22:45:41.608075+0000","last_clean_scrub_stamp":"2024-09-20T22:55:04.973719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:56:25.040136+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"625'30","reported_seq":1712,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-09-20T22:55:33.344109+0000","last_change":"2024-09-20T22:53:48.346424+0000","last_active":"2024-09-20T22:55:33.344109+0000","last_peered":"2024-09-20T22:55:33.344109+0000","last_clean":"2024-09-20T22:55:33.344109+0000","last_became_active":"2024-09-20T22:47:49.217605+0000","last_became_peered":"2024-09-20T22:47:49.217605+0000","last_unstale":"2024-09-20T22:55:33.344109+0000","last_undegraded":"2024-09-20T22:55:33.344109+0000","last_fullsized":"2024-09-20T22:55:33.344109+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'30","last_scrub_stamp":"2024-09-20T22:53:48.346341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:53:48.346341+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":1630,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-09-20T22:55:33.347178+0000","last_change":"2024-09-20T22:52:45.537243+0000","last_active":"2024-09-20T22:55:33.347178+0000","last_peered":"2024-09-20T22:55:33.347178+0000","last_clean":"2024-09-20T22:55:33.347178+0000","last_became_active":"2024-09-20T22:47:49.227069+0000","last_became_peered":"2024-09-20T22:47:49.227069+0000","last_unstale":"2024-09-20T22:55:33.347178+0000","last_undegraded":"2024-09-20T22:55:33.347178+0000","last_fullsized":"2024-09-20T22:55:33.347178+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:45.537173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:52:45.537173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":565,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-09-20T22:55:33.344409+0000","last_change":"2024-09-20T22:55:17.349016+0000","last_active":"2024-09-20T22:55:33.344409+0000","last_peered":"2024-09-20T22:55:33.344409+0000","last_clean":"2024-09-20T22:55:33.344409+0000","last_became_active":"2024-09-20T22:55:17.348022+0000","last_became_peered":"2024-09-20T22:55:17.348022+0000","last_unstale":"2024-09-20T22:55:33.344409+0000","last_undegraded":"2024-09-20T22:55:33.344409+0000","last_fullsized":"2024-09-20T22:55:33.344409+0000","mapping_epoch":836,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":837,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:26.623455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:07.947945+0000","last_clean_scrub_stamp":"2024-09-20T22:53:26.623455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":554,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-09-20T22:55:33.345892+0000","last_change":"2024-09-20T22:55:17.351633+0000","last_active":"2024-09-20T22:55:33.345892+0000","last_peered":"2024-09-20T22:55:33.345892+0000","last_clean":"2024-09-20T22:55:33.345892+0000","last_became_active":"2024-09-20T22:55:17.351296+0000","last_became_peered":"2024-09-20T22:55:17.351296+0000","last_unstale":"2024-09-20T22:55:33.345892+0000","last_undegraded":"2024-09-20T22:55:33.345892+0000","last_fullsized":"2024-09-20T22:55:33.345892+0000","mapping_epoch":836,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":837,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:51:55.748425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:53.036601+0000","last_clean_scrub_stamp":"2024-09-20T22:51:55.748425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2","version":"853'880","reported_seq":4042,"reported_epoch":853,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T22:55:33.522607+0000","last_change":"2024-09-20T22:55:33.522607+0000","last_active":"2024-09-20T22:55:33.522607+0000","last_peered":"2024-09-20T22:55:33.522607+0000","last_clean":"2024-09-20T22:55:33.522607+0000","last_became_active":"2024-09-20T22:54:50.348406+0000","last_became_peered":"2024-09-20T22:54:50.348406+0000","last_unstale":"2024-09-20T22:55:33.522607+0000","last_undegraded":"2024-09-20T22:55:33.522607+0000","last_fullsized":"2024-09-20T22:55:33.522607+0000","mapping_epoch":809,"log_start":"471'464","ondisk_log_start":"471'464","created":94,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":0,"last_scrub":"282'120","last_scrub_stamp":"2024-09-20T22:46:22.099642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:46:22.099642+0000","objects_scrubbed":1,"log_size":416,"log_dups_size":687,"ondisk_log_size":416,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.176968923,"stat_sum":{"num_bytes":33801672,"num_objects":15,"num_object_clones":11,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":2,"num_read":475,"num_read_kb":488907,"num_write":289,"num_write_kb":110401,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":160,"num_bytes_recovered":146402851,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"13e","length":"b"},{"start":"14b","length":"3"},{"start":"150","length":"5"},{"start":"156","length":"1"},{"start":"159","length":"1"},{"start":"15b","length":"1"},{"start":"15d","length":"3"},{"start":"161","length":"5"},{"start":"167","length":"4"},{"start":"16d","length":"1"},{"start":"170","length":"2"}]},{"pgid":"1.4","version":"77'128","reported_seq":1912,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-09-20T22:55:33.345988+0000","last_change":"2024-09-20T22:52:02.474080+0000","last_active":"2024-09-20T22:55:33.345988+0000","last_peered":"2024-09-20T22:55:33.345988+0000","last_clean":"2024-09-20T22:55:33.345988+0000","last_became_active":"2024-09-20T22:46:15.181305+0000","last_became_peered":"2024-09-20T22:46:15.181305+0000","last_unstale":"2024-09-20T22:55:33.345988+0000","last_undegraded":"2024-09-20T22:55:33.345988+0000","last_fullsized":"2024-09-20T22:55:33.345988+0000","mapping_epoch":94,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:52:02.473979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:52:02.473979+0000","objects_scrubbed":1,"log_size":0,"log_dups_size":59,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"625'28","reported_seq":539,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-09-20T22:55:33.348202+0000","last_change":"2024-09-20T22:55:17.344950+0000","last_active":"2024-09-20T22:55:33.348202+0000","last_peered":"2024-09-20T22:55:33.348202+0000","last_clean":"2024-09-20T22:55:33.348202+0000","last_became_active":"2024-09-20T22:55:17.344611+0000","last_became_peered":"2024-09-20T22:55:17.344611+0000","last_unstale":"2024-09-20T22:55:33.348202+0000","last_undegraded":"2024-09-20T22:55:33.348202+0000","last_fullsized":"2024-09-20T22:55:33.348202+0000","mapping_epoch":836,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":837,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'28","last_scrub_stamp":"2024-09-20T22:54:46.173524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:54:46.173524+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:56.898792+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":40,"num_read_kb":33,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.6","version":"67'1","reported_seq":1000,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-09-20T22:55:33.333776+0000","last_change":"2024-09-20T22:54:17.366948+0000","last_active":"2024-09-20T22:55:33.333776+0000","last_peered":"2024-09-20T22:55:33.333776+0000","last_clean":"2024-09-20T22:55:33.333776+0000","last_became_active":"2024-09-20T22:54:17.340423+0000","last_became_peered":"2024-09-20T22:54:17.340423+0000","last_unstale":"2024-09-20T22:55:33.333776+0000","last_undegraded":"2024-09-20T22:55:33.333776+0000","last_fullsized":"2024-09-20T22:55:33.333776+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-09-20T22:53:32.995211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:53:32.995211+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":230,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"778'16","reported_seq":1849,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-09-20T22:55:33.344352+0000","last_change":"2024-09-20T22:55:10.378111+0000","last_active":"2024-09-20T22:55:33.344352+0000","last_peered":"2024-09-20T22:55:33.344352+0000","last_clean":"2024-09-20T22:55:33.344352+0000","last_became_active":"2024-09-20T22:54:17.341871+0000","last_became_peered":"2024-09-20T22:54:17.341871+0000","last_unstale":"2024-09-20T22:55:33.344352+0000","last_undegraded":"2024-09-20T22:55:33.344352+0000","last_fullsized":"2024-09-20T22:55:33.344352+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"778'16","last_scrub_stamp":"2024-09-20T22:55:10.378046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:55:10.378046+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:56:19.394443+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":1619,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-09-20T22:55:33.369434+0000","last_change":"2024-09-20T22:55:23.858755+0000","last_active":"2024-09-20T22:55:33.369434+0000","last_peered":"2024-09-20T22:55:33.369434+0000","last_clean":"2024-09-20T22:55:33.369434+0000","last_became_active":"2024-09-20T22:41:44.831901+0000","last_became_peered":"2024-09-20T22:41:44.831901+0000","last_unstale":"2024-09-20T22:55:33.369434+0000","last_undegraded":"2024-09-20T22:55:33.369434+0000","last_fullsized":"2024-09-20T22:55:33.369434+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:55:23.858671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:55:23.858671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:56:29.596094+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1","version":"834'504","reported_seq":4222,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-09-20T22:55:33.344368+0000","last_change":"2024-09-20T22:55:26.476384+0000","last_active":"2024-09-20T22:55:33.344368+0000","last_peered":"2024-09-20T22:55:33.344368+0000","last_clean":"2024-09-20T22:55:33.344368+0000","last_became_active":"2024-09-20T22:55:17.348786+0000","last_became_peered":"2024-09-20T22:55:17.348786+0000","last_unstale":"2024-09-20T22:55:33.344368+0000","last_undegraded":"2024-09-20T22:55:33.344368+0000","last_fullsized":"2024-09-20T22:55:33.344368+0000","mapping_epoch":836,"log_start":"474'259","ondisk_log_start":"474'259","created":94,"last_epoch_clean":837,"parent":"0.0","parent_split_bits":0,"last_scrub":"794'485","last_scrub_stamp":"2024-09-20T22:54:38.213249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:54:38.213249+0000","objects_scrubbed":8,"log_size":245,"log_dups_size":338,"ondisk_log_size":245,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:51.054374+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.10207846700000001,"stat_sum":{"num_bytes":12775263,"num_objects":7,"num_object_clones":5,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":137,"num_read_kb":151604,"num_write":105,"num_write_kb":52508,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":8732217,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"13e","length":"b"},{"start":"14b","length":"3"},{"start":"150","length":"5"},{"start":"156","length":"1"},{"start":"159","length":"1"},{"start":"15b","length":"1"},{"start":"15d","length":"3"},{"start":"161","length":"5"},{"start":"167","length":"4"},{"start":"16d","length":"1"},{"start":"170","length":"2"},{"start":"178","length":"1"}]},{"pgid":"1.7","version":"77'128","reported_seq":1932,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-09-20T22:55:33.367614+0000","last_change":"2024-09-20T22:53:56.869885+0000","last_active":"2024-09-20T22:55:33.367614+0000","last_peered":"2024-09-20T22:55:33.367614+0000","last_clean":"2024-09-20T22:55:33.367614+0000","last_became_active":"2024-09-20T22:46:03.169596+0000","last_became_peered":"2024-09-20T22:46:03.169596+0000","last_unstale":"2024-09-20T22:55:33.367614+0000","last_undegraded":"2024-09-20T22:55:33.367614+0000","last_fullsized":"2024-09-20T22:55:33.367614+0000","mapping_epoch":94,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":284,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:53:56.869735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:53:56.869735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"625'20","reported_seq":183,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-09-20T22:55:33.333868+0000","last_change":"2024-09-20T22:54:39.933063+0000","last_active":"2024-09-20T22:55:33.333868+0000","last_peered":"2024-09-20T22:55:33.333868+0000","last_clean":"2024-09-20T22:55:33.333868+0000","last_became_active":"2024-09-20T22:54:17.349113+0000","last_became_peered":"2024-09-20T22:54:17.349113+0000","last_unstale":"2024-09-20T22:55:33.333868+0000","last_undegraded":"2024-09-20T22:55:33.333868+0000","last_fullsized":"2024-09-20T22:55:33.333868+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'20","last_scrub_stamp":"2024-09-20T22:54:39.932960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:54:39.932960+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:56:07.875833+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":28,"num_read_kb":23,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":1626,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-09-20T22:55:33.349047+0000","last_change":"2024-09-20T22:52:46.062448+0000","last_active":"2024-09-20T22:55:33.349047+0000","last_peered":"2024-09-20T22:55:33.349047+0000","last_clean":"2024-09-20T22:55:33.349047+0000","last_became_active":"2024-09-20T22:41:38.155629+0000","last_became_peered":"2024-09-20T22:41:38.155629+0000","last_unstale":"2024-09-20T22:55:33.349047+0000","last_undegraded":"2024-09-20T22:55:33.349047+0000","last_fullsized":"2024-09-20T22:55:33.349047+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:46.062352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:52:46.062352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"733'12","reported_seq":1065,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-09-20T22:55:33.333172+0000","last_change":"2024-09-20T22:53:33.928576+0000","last_active":"2024-09-20T22:55:33.333172+0000","last_peered":"2024-09-20T22:55:33.333172+0000","last_clean":"2024-09-20T22:55:33.333172+0000","last_became_active":"2024-09-20T22:47:49.227264+0000","last_became_peered":"2024-09-20T22:47:49.227264+0000","last_unstale":"2024-09-20T22:55:33.333172+0000","last_undegraded":"2024-09-20T22:55:33.333172+0000","last_fullsized":"2024-09-20T22:55:33.333172+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"733'12","last_scrub_stamp":"2024-09-20T22:53:33.928499+0000","last_deep_scrub":"733'12","last_deep_scrub_stamp":"2024-09-20T22:53:33.928499+0000","last_clean_scrub_stamp":"2024-09-20T22:53:33.928499+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":966,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-09-20T22:55:33.332142+0000","last_change":"2024-09-20T22:53:57.868607+0000","last_active":"2024-09-20T22:55:33.332142+0000","last_peered":"2024-09-20T22:55:33.332142+0000","last_clean":"2024-09-20T22:55:33.332142+0000","last_became_active":"2024-09-20T22:47:49.232380+0000","last_became_peered":"2024-09-20T22:47:49.232380+0000","last_unstale":"2024-09-20T22:55:33.332142+0000","last_undegraded":"2024-09-20T22:55:33.332142+0000","last_fullsized":"2024-09-20T22:55:33.332142+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:57.868534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:53:57.868534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.0","version":"850'884","reported_seq":4750,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-09-20T22:55:33.348507+0000","last_change":"2024-09-20T22:55:32.382832+0000","last_active":"2024-09-20T22:55:33.348507+0000","last_peered":"2024-09-20T22:55:33.348507+0000","last_clean":"2024-09-20T22:55:33.348507+0000","last_became_active":"2024-09-20T22:54:20.418799+0000","last_became_peered":"2024-09-20T22:54:20.418799+0000","last_unstale":"2024-09-20T22:55:33.348507+0000","last_undegraded":"2024-09-20T22:55:33.348507+0000","last_fullsized":"2024-09-20T22:55:33.348507+0000","mapping_epoch":779,"log_start":"480'253","ondisk_log_start":"480'253","created":94,"last_epoch_clean":780,"parent":"0.0","parent_split_bits":0,"last_scrub":"823'833","last_scrub_stamp":"2024-09-20T22:55:07.264241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:55:07.264241+0000","objects_scrubbed":25,"log_size":631,"log_dups_size":416,"ondisk_log_size":631,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:56:09.941007+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":1.026984865,"stat_sum":{"num_bytes":50049195,"num_objects":26,"num_object_clones":23,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":2,"num_read":288,"num_read_kb":287302,"num_write":251,"num_write_kb":90279,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":54,"num_bytes_recovered":106304666,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"13e","length":"b"},{"start":"14b","length":"3"},{"start":"150","length":"5"},{"start":"156","length":"1"},{"start":"159","length":"1"},{"start":"15b","length":"1"},{"start":"15d","length":"3"},{"start":"161","length":"5"},{"start":"167","length":"4"},{"start":"16d","length":"1"},{"start":"170","length":"2"},{"start":"178","length":"1"}]},{"pgid":"1.6","version":"77'128","reported_seq":1954,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-09-20T22:55:33.367506+0000","last_change":"2024-09-20T22:54:22.148323+0000","last_active":"2024-09-20T22:55:33.367506+0000","last_peered":"2024-09-20T22:55:33.367506+0000","last_clean":"2024-09-20T22:55:33.367506+0000","last_became_active":"2024-09-20T22:47:52.212190+0000","last_became_peered":"2024-09-20T22:47:52.212190+0000","last_unstale":"2024-09-20T22:55:33.367506+0000","last_undegraded":"2024-09-20T22:55:33.367506+0000","last_fullsized":"2024-09-20T22:55:33.367506+0000","mapping_epoch":391,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:54:22.148244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:54:22.148244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:46.700798+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"625'37","reported_seq":1007,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-09-20T22:55:33.332912+0000","last_change":"2024-09-20T22:54:46.870917+0000","last_active":"2024-09-20T22:55:33.332912+0000","last_peered":"2024-09-20T22:55:33.332912+0000","last_clean":"2024-09-20T22:55:33.332912+0000","last_became_active":"2024-09-20T22:47:49.222009+0000","last_became_peered":"2024-09-20T22:47:49.222009+0000","last_unstale":"2024-09-20T22:55:33.332912+0000","last_undegraded":"2024-09-20T22:55:33.332912+0000","last_fullsized":"2024-09-20T22:55:33.332912+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'37","last_scrub_stamp":"2024-09-20T22:54:46.870818+0000","last_deep_scrub":"515'27","last_deep_scrub_stamp":"2024-09-20T22:49:55.964153+0000","last_clean_scrub_stamp":"2024-09-20T22:54:46.870818+0000","objects_scrubbed":8,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:56:09.839466+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":53,"num_read_kb":39,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":171,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-09-20T22:55:33.333323+0000","last_change":"2024-09-20T22:55:30.876432+0000","last_active":"2024-09-20T22:55:33.333323+0000","last_peered":"2024-09-20T22:55:33.333323+0000","last_clean":"2024-09-20T22:55:33.333323+0000","last_became_active":"2024-09-20T22:54:17.341248+0000","last_became_peered":"2024-09-20T22:54:17.341248+0000","last_unstale":"2024-09-20T22:55:33.333323+0000","last_undegraded":"2024-09-20T22:55:33.333323+0000","last_fullsized":"2024-09-20T22:55:33.333323+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:55:30.876361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:48:23.320031+0000","last_clean_scrub_stamp":"2024-09-20T22:55:30.876361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:56:48.575021+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":1646,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-09-20T22:55:33.367013+0000","last_change":"2024-09-20T22:53:57.882313+0000","last_active":"2024-09-20T22:55:33.367013+0000","last_peered":"2024-09-20T22:55:33.367013+0000","last_clean":"2024-09-20T22:55:33.367013+0000","last_became_active":"2024-09-20T22:47:49.214055+0000","last_became_peered":"2024-09-20T22:47:49.214055+0000","last_unstale":"2024-09-20T22:55:33.367013+0000","last_undegraded":"2024-09-20T22:55:33.367013+0000","last_fullsized":"2024-09-20T22:55:33.367013+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:57.882212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:53:57.882212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":163,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-09-20T22:55:33.332410+0000","last_change":"2024-09-20T22:54:17.340872+0000","last_active":"2024-09-20T22:55:33.332410+0000","last_peered":"2024-09-20T22:55:33.332410+0000","last_clean":"2024-09-20T22:55:33.332410+0000","last_became_active":"2024-09-20T22:54:17.340458+0000","last_became_peered":"2024-09-20T22:54:17.340458+0000","last_unstale":"2024-09-20T22:55:33.332410+0000","last_undegraded":"2024-09-20T22:55:33.332410+0000","last_fullsized":"2024-09-20T22:55:33.332410+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:54:15.141588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:48:41.208853+0000","last_clean_scrub_stamp":"2024-09-20T22:54:15.141588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:41.084042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.7","version":"847'1667","reported_seq":5451,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-09-20T22:55:33.367943+0000","last_change":"2024-09-20T22:55:25.351941+0000","last_active":"2024-09-20T22:55:33.367943+0000","last_peered":"2024-09-20T22:55:33.367943+0000","last_clean":"2024-09-20T22:55:33.367943+0000","last_became_active":"2024-09-20T22:48:43.224386+0000","last_became_peered":"2024-09-20T22:48:43.224386+0000","last_unstale":"2024-09-20T22:55:33.367943+0000","last_undegraded":"2024-09-20T22:55:33.367943+0000","last_fullsized":"2024-09-20T22:55:33.367943+0000","mapping_epoch":391,"log_start":"436'456","ondisk_log_start":"436'456","created":94,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"772'1452","last_scrub_stamp":"2024-09-20T22:54:14.120176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:54:14.120176+0000","objects_scrubbed":23,"log_size":1211,"log_dups_size":886,"ondisk_log_size":1211,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:38.984776+0000","scrub_duration":7,"objects_trimmed":1,"snaptrim_duration":0.0097162440000000006,"stat_sum":{"num_bytes":66065335,"num_objects":28,"num_object_clones":23,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":1,"num_read":556,"num_read_kb":584775,"num_write":460,"num_write_kb":159092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":115,"num_bytes_recovered":51615167,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"13e","length":"b"},{"start":"14b","length":"3"},{"start":"150","length":"5"},{"start":"156","length":"1"},{"start":"159","length":"1"},{"start":"15b","length":"1"},{"start":"15d","length":"3"},{"start":"161","length":"5"},{"start":"167","length":"4"},{"start":"16d","length":"1"},{"start":"170","length":"2"},{"start":"178","length":"1"}]},{"pgid":"1.1","version":"77'128","reported_seq":1943,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-09-20T22:55:33.348408+0000","last_change":"2024-09-20T22:55:27.327609+0000","last_active":"2024-09-20T22:55:33.348408+0000","last_peered":"2024-09-20T22:55:33.348408+0000","last_clean":"2024-09-20T22:55:33.348408+0000","last_became_active":"2024-09-20T22:47:52.208102+0000","last_became_peered":"2024-09-20T22:47:52.208102+0000","last_unstale":"2024-09-20T22:55:33.348408+0000","last_undegraded":"2024-09-20T22:55:33.348408+0000","last_fullsized":"2024-09-20T22:55:33.348408+0000","mapping_epoch":391,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:55:27.327480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:55:27.327480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:56:53.124422+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"833'53","reported_seq":82,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-09-20T22:55:33.348664+0000","last_change":"2024-09-20T22:55:11.365933+0000","last_active":"2024-09-20T22:55:33.348664+0000","last_peered":"2024-09-20T22:55:33.348664+0000","last_clean":"2024-09-20T22:55:33.348664+0000","last_became_active":"2024-09-20T22:55:11.346099+0000","last_became_peered":"2024-09-20T22:55:11.346099+0000","last_unstale":"2024-09-20T22:55:33.348664+0000","last_undegraded":"2024-09-20T22:55:33.348664+0000","last_fullsized":"2024-09-20T22:55:33.348664+0000","mapping_epoch":830,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":831,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'51","last_scrub_stamp":"2024-09-20T22:53:57.184165+0000","last_deep_scrub":"86'38","last_deep_scrub_stamp":"2024-09-20T22:42:43.935677+0000","last_clean_scrub_stamp":"2024-09-20T22:53:57.184165+0000","objects_scrubbed":10,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":1074,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":90,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-09-20T22:55:33.333088+0000","last_change":"2024-09-20T22:55:24.603577+0000","last_active":"2024-09-20T22:55:33.333088+0000","last_peered":"2024-09-20T22:55:33.333088+0000","last_clean":"2024-09-20T22:55:33.333088+0000","last_became_active":"2024-09-20T22:55:17.340038+0000","last_became_peered":"2024-09-20T22:55:17.340038+0000","last_unstale":"2024-09-20T22:55:33.333088+0000","last_undegraded":"2024-09-20T22:55:33.333088+0000","last_fullsized":"2024-09-20T22:55:33.333088+0000","mapping_epoch":836,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":837,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:55:24.603480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:55:24.603480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:56:26.224035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"69'2","reported_seq":1664,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-09-20T22:55:33.349739+0000","last_change":"2024-09-20T22:54:23.097050+0000","last_active":"2024-09-20T22:55:33.349739+0000","last_peered":"2024-09-20T22:55:33.349739+0000","last_clean":"2024-09-20T22:55:33.349739+0000","last_became_active":"2024-09-20T22:54:17.341059+0000","last_became_peered":"2024-09-20T22:54:17.341059+0000","last_unstale":"2024-09-20T22:55:33.349739+0000","last_undegraded":"2024-09-20T22:55:33.349739+0000","last_fullsized":"2024-09-20T22:55:33.349739+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2024-09-20T22:54:23.096974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:36.130684+0000","last_clean_scrub_stamp":"2024-09-20T22:54:23.096974+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:40.124528+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00087280599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":984,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-09-20T22:55:33.333523+0000","last_change":"2024-09-20T22:54:17.340261+0000","last_active":"2024-09-20T22:55:33.333523+0000","last_peered":"2024-09-20T22:55:33.333523+0000","last_clean":"2024-09-20T22:55:33.333523+0000","last_became_active":"2024-09-20T22:54:17.339860+0000","last_became_peered":"2024-09-20T22:54:17.339860+0000","last_unstale":"2024-09-20T22:55:33.333523+0000","last_undegraded":"2024-09-20T22:55:33.333523+0000","last_fullsized":"2024-09-20T22:55:33.333523+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:51.849772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:53:51.849772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":1618,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-09-20T22:55:33.346328+0000","last_change":"2024-09-20T22:52:21.363432+0000","last_active":"2024-09-20T22:55:33.346328+0000","last_peered":"2024-09-20T22:55:33.346328+0000","last_clean":"2024-09-20T22:55:33.346328+0000","last_became_active":"2024-09-20T22:48:05.234193+0000","last_became_peered":"2024-09-20T22:48:05.234193+0000","last_unstale":"2024-09-20T22:55:33.346328+0000","last_undegraded":"2024-09-20T22:55:33.346328+0000","last_fullsized":"2024-09-20T22:55:33.346328+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:21.363359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:52:21.363359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"853'620","reported_seq":2888,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-09-20T22:55:33.333041+0000","last_change":"2024-09-20T22:55:25.338485+0000","last_active":"2024-09-20T22:55:33.333041+0000","last_peered":"2024-09-20T22:55:33.333041+0000","last_clean":"2024-09-20T22:55:33.333041+0000","last_became_active":"2024-09-20T22:54:20.412859+0000","last_became_peered":"2024-09-20T22:54:20.412859+0000","last_unstale":"2024-09-20T22:55:33.333041+0000","last_undegraded":"2024-09-20T22:55:33.333041+0000","last_fullsized":"2024-09-20T22:55:33.333041+0000","mapping_epoch":779,"log_start":"456'176","ondisk_log_start":"456'176","created":94,"last_epoch_clean":780,"parent":"0.0","parent_split_bits":0,"last_scrub":"744'500","last_scrub_stamp":"2024-09-20T22:53:49.903129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:53:49.903129+0000","objects_scrubbed":10,"log_size":444,"log_dups_size":318,"ondisk_log_size":444,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0039636769999999997,"stat_sum":{"num_bytes":19129539,"num_objects":8,"num_object_clones":6,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":1,"num_read":212,"num_read_kb":214503,"num_write":158,"num_write_kb":55925,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":41680310,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"13e","length":"b"},{"start":"14b","length":"3"},{"start":"150","length":"5"},{"start":"156","length":"1"},{"start":"159","length":"1"},{"start":"15b","length":"1"},{"start":"15d","length":"3"},{"start":"161","length":"5"},{"start":"167","length":"4"},{"start":"16d","length":"1"},{"start":"170","length":"2"},{"start":"178","length":"1"}]},{"pgid":"1.2","version":"77'128","reported_seq":2002,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-09-20T22:55:33.369147+0000","last_change":"2024-09-20T22:53:58.495293+0000","last_active":"2024-09-20T22:55:33.369147+0000","last_peered":"2024-09-20T22:55:33.369147+0000","last_clean":"2024-09-20T22:55:33.369147+0000","last_became_active":"2024-09-20T22:48:08.203028+0000","last_became_peered":"2024-09-20T22:48:08.203028+0000","last_unstale":"2024-09-20T22:55:33.369147+0000","last_undegraded":"2024-09-20T22:55:33.369147+0000","last_fullsized":"2024-09-20T22:55:33.369147+0000","mapping_epoch":407,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":408,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:53:58.495191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:53:58.495191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"625'22","reported_seq":186,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-09-20T22:55:33.333210+0000","last_change":"2024-09-20T22:55:07.954993+0000","last_active":"2024-09-20T22:55:33.333210+0000","last_peered":"2024-09-20T22:55:33.333210+0000","last_clean":"2024-09-20T22:55:33.333210+0000","last_became_active":"2024-09-20T22:54:17.349415+0000","last_became_peered":"2024-09-20T22:54:17.349415+0000","last_unstale":"2024-09-20T22:55:33.333210+0000","last_undegraded":"2024-09-20T22:55:33.333210+0000","last_fullsized":"2024-09-20T22:55:33.333210+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'22","last_scrub_stamp":"2024-09-20T22:55:07.954868+0000","last_deep_scrub":"443'18","last_deep_scrub_stamp":"2024-09-20T22:48:44.182517+0000","last_clean_scrub_stamp":"2024-09-20T22:55:07.954868+0000","objects_scrubbed":5,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:56:26.192346+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":24,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":1320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":171,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-09-20T22:55:33.333123+0000","last_change":"2024-09-20T22:54:49.964373+0000","last_active":"2024-09-20T22:55:33.333123+0000","last_peered":"2024-09-20T22:55:33.333123+0000","last_clean":"2024-09-20T22:55:33.333123+0000","last_became_active":"2024-09-20T22:54:17.343806+0000","last_became_peered":"2024-09-20T22:54:17.343806+0000","last_unstale":"2024-09-20T22:55:33.333123+0000","last_undegraded":"2024-09-20T22:55:33.333123+0000","last_fullsized":"2024-09-20T22:55:33.333123+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:54:49.964301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:54:49.964301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:55.059202+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"380'2","reported_seq":1679,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-09-20T22:55:33.349065+0000","last_change":"2024-09-20T22:54:16.438102+0000","last_active":"2024-09-20T22:55:33.349065+0000","last_peered":"2024-09-20T22:55:33.349065+0000","last_clean":"2024-09-20T22:55:33.349065+0000","last_became_active":"2024-09-20T22:47:49.222812+0000","last_became_peered":"2024-09-20T22:47:49.222812+0000","last_unstale":"2024-09-20T22:55:33.349065+0000","last_undegraded":"2024-09-20T22:55:33.349065+0000","last_fullsized":"2024-09-20T22:55:33.349065+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"380'2","last_scrub_stamp":"2024-09-20T22:54:16.437989+0000","last_deep_scrub":"380'2","last_deep_scrub_stamp":"2024-09-20T22:47:40.983039+0000","last_clean_scrub_stamp":"2024-09-20T22:54:16.437989+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.001301459,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":1633,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-09-20T22:55:33.349701+0000","last_change":"2024-09-20T22:55:08.384425+0000","last_active":"2024-09-20T22:55:33.349701+0000","last_peered":"2024-09-20T22:55:33.349701+0000","last_clean":"2024-09-20T22:55:33.349701+0000","last_became_active":"2024-09-20T22:41:42.185112+0000","last_became_peered":"2024-09-20T22:41:42.185112+0000","last_unstale":"2024-09-20T22:55:33.349701+0000","last_undegraded":"2024-09-20T22:55:33.349701+0000","last_fullsized":"2024-09-20T22:55:33.349701+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:55:08.384321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:47:22.881109+0000","last_clean_scrub_stamp":"2024-09-20T22:55:08.384321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:56:15.686670+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":1619,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-09-20T22:55:33.369689+0000","last_change":"2024-09-20T22:55:09.670832+0000","last_active":"2024-09-20T22:55:33.369689+0000","last_peered":"2024-09-20T22:55:33.369689+0000","last_clean":"2024-09-20T22:55:33.369689+0000","last_became_active":"2024-09-20T22:41:44.830515+0000","last_became_peered":"2024-09-20T22:41:44.830515+0000","last_unstale":"2024-09-20T22:55:33.369689+0000","last_undegraded":"2024-09-20T22:55:33.369689+0000","last_fullsized":"2024-09-20T22:55:33.369689+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:55:09.670757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:55:09.670757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:56:35.616925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"845'458","reported_seq":4280,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-09-20T22:55:33.369709+0000","last_change":"2024-09-20T22:55:25.342297+0000","last_active":"2024-09-20T22:55:33.369709+0000","last_peered":"2024-09-20T22:55:33.369709+0000","last_clean":"2024-09-20T22:55:33.369709+0000","last_became_active":"2024-09-20T22:48:57.236417+0000","last_became_peered":"2024-09-20T22:48:57.236417+0000","last_unstale":"2024-09-20T22:55:33.369709+0000","last_undegraded":"2024-09-20T22:55:33.369709+0000","last_fullsized":"2024-09-20T22:55:33.369709+0000","mapping_epoch":391,"log_start":"450'190","ondisk_log_start":"450'190","created":94,"last_epoch_clean":457,"parent":"0.0","parent_split_bits":0,"last_scrub":"774'413","last_scrub_stamp":"2024-09-20T22:54:15.568174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:54:15.568174+0000","objects_scrubbed":5,"log_size":268,"log_dups_size":240,"ondisk_log_size":268,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:35.845842+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0074078420000000004,"stat_sum":{"num_bytes":12168423,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":110,"num_read_kb":104204,"num_write":105,"num_write_kb":42091,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":19673726,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"13e","length":"b"},{"start":"14b","length":"3"},{"start":"150","length":"5"},{"start":"156","length":"1"},{"start":"159","length":"1"},{"start":"15b","length":"1"},{"start":"15d","length":"3"},{"start":"161","length":"5"},{"start":"167","length":"4"},{"start":"16d","length":"1"},{"start":"170","length":"2"},{"start":"178","length":"1"}]},{"pgid":"1.3","version":"77'128","reported_seq":1190,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-09-20T22:55:33.334234+0000","last_change":"2024-09-20T22:54:44.971544+0000","last_active":"2024-09-20T22:55:33.334234+0000","last_peered":"2024-09-20T22:55:33.334234+0000","last_clean":"2024-09-20T22:55:33.334234+0000","last_became_active":"2024-09-20T22:54:20.413191+0000","last_became_peered":"2024-09-20T22:54:20.413191+0000","last_unstale":"2024-09-20T22:55:33.334234+0000","last_undegraded":"2024-09-20T22:55:33.334234+0000","last_fullsized":"2024-09-20T22:55:33.334234+0000","mapping_epoch":779,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":780,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:54:44.971463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:54:44.971463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:56:04.555885+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"828'30","reported_seq":1064,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-09-20T22:55:33.334054+0000","last_change":"2024-09-20T22:55:11.360221+0000","last_active":"2024-09-20T22:55:33.334054+0000","last_peered":"2024-09-20T22:55:33.334054+0000","last_clean":"2024-09-20T22:55:33.334054+0000","last_became_active":"2024-09-20T22:55:11.346018+0000","last_became_peered":"2024-09-20T22:55:11.346018+0000","last_unstale":"2024-09-20T22:55:33.334054+0000","last_undegraded":"2024-09-20T22:55:33.334054+0000","last_fullsized":"2024-09-20T22:55:33.334054+0000","mapping_epoch":830,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":831,"parent":"0.0","parent_split_bits":0,"last_scrub":"828'30","last_scrub_stamp":"2024-09-20T22:55:08.705571+0000","last_deep_scrub":"828'30","last_deep_scrub_stamp":"2024-09-20T22:55:08.705571+0000","last_clean_scrub_stamp":"2024-09-20T22:55:08.705571+0000","objects_scrubbed":6,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:56:17.575678+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":980,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-09-20T22:55:33.332061+0000","last_change":"2024-09-20T22:55:25.640612+0000","last_active":"2024-09-20T22:55:33.332061+0000","last_peered":"2024-09-20T22:55:33.332061+0000","last_clean":"2024-09-20T22:55:33.332061+0000","last_became_active":"2024-09-20T22:48:05.233093+0000","last_became_peered":"2024-09-20T22:48:05.233093+0000","last_unstale":"2024-09-20T22:55:33.332061+0000","last_undegraded":"2024-09-20T22:55:33.332061+0000","last_fullsized":"2024-09-20T22:55:33.332061+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:55:25.640520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:44:19.834641+0000","last_clean_scrub_stamp":"2024-09-20T22:55:25.640520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:56:39.009398+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"75'5","reported_seq":2508,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-09-20T22:55:33.369308+0000","last_change":"2024-09-20T22:54:48.577002+0000","last_active":"2024-09-20T22:55:33.369308+0000","last_peered":"2024-09-20T22:55:33.369308+0000","last_clean":"2024-09-20T22:55:33.369308+0000","last_became_active":"2024-09-20T22:54:17.337205+0000","last_became_peered":"2024-09-20T22:54:17.337205+0000","last_unstale":"2024-09-20T22:55:33.369308+0000","last_undegraded":"2024-09-20T22:55:33.369308+0000","last_fullsized":"2024-09-20T22:55:33.369308+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-09-20T22:54:48.576929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:36.130684+0000","last_clean_scrub_stamp":"2024-09-20T22:54:48.576929+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:56:05.805706+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00078631700000000003,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":824,"num_read_kb":819,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1556,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":1650,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-09-20T22:55:33.349459+0000","last_change":"2024-09-20T22:54:51.175204+0000","last_active":"2024-09-20T22:55:33.349459+0000","last_peered":"2024-09-20T22:55:33.349459+0000","last_clean":"2024-09-20T22:55:33.349459+0000","last_became_active":"2024-09-20T22:54:17.339805+0000","last_became_peered":"2024-09-20T22:54:17.339805+0000","last_unstale":"2024-09-20T22:55:33.349459+0000","last_undegraded":"2024-09-20T22:55:33.349459+0000","last_fullsized":"2024-09-20T22:55:33.349459+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:54:51.175120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:54:51.175120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:56:09.146766+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":171,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-09-20T22:55:33.333170+0000","last_change":"2024-09-20T22:55:11.066946+0000","last_active":"2024-09-20T22:55:33.333170+0000","last_peered":"2024-09-20T22:55:33.333170+0000","last_clean":"2024-09-20T22:55:33.333170+0000","last_became_active":"2024-09-20T22:54:17.339564+0000","last_became_peered":"2024-09-20T22:54:17.339564+0000","last_unstale":"2024-09-20T22:55:33.333170+0000","last_undegraded":"2024-09-20T22:55:33.333170+0000","last_fullsized":"2024-09-20T22:55:33.333170+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:55:11.066868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:55:11.066868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:56:34.564475+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.6","version":"851'681","reported_seq":4412,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-09-20T22:55:33.333912+0000","last_change":"2024-09-20T22:55:25.344077+0000","last_active":"2024-09-20T22:55:33.333912+0000","last_peered":"2024-09-20T22:55:33.333912+0000","last_clean":"2024-09-20T22:55:33.333912+0000","last_became_active":"2024-09-20T22:48:47.234531+0000","last_became_peered":"2024-09-20T22:48:47.234531+0000","last_unstale":"2024-09-20T22:55:33.333912+0000","last_undegraded":"2024-09-20T22:55:33.333912+0000","last_fullsized":"2024-09-20T22:55:33.333912+0000","mapping_epoch":391,"log_start":"443'181","ondisk_log_start":"443'181","created":94,"last_epoch_clean":447,"parent":"0.0","parent_split_bits":0,"last_scrub":"770'553","last_scrub_stamp":"2024-09-20T22:54:13.963191+0000","last_deep_scrub":"440'177","last_deep_scrub_stamp":"2024-09-20T22:48:40.602342+0000","last_clean_scrub_stamp":"2024-09-20T22:54:13.963191+0000","objects_scrubbed":13,"log_size":500,"log_dups_size":314,"ondisk_log_size":500,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:43.787756+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.0089261960000000008,"stat_sum":{"num_bytes":40057730,"num_objects":18,"num_object_clones":16,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":222,"num_read_kb":223026,"num_write":187,"num_write_kb":73683,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":18662529,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"13e","length":"b"},{"start":"14b","length":"3"},{"start":"150","length":"5"},{"start":"156","length":"1"},{"start":"159","length":"1"},{"start":"15b","length":"1"},{"start":"15d","length":"3"},{"start":"161","length":"5"},{"start":"167","length":"4"},{"start":"16d","length":"1"},{"start":"170","length":"2"},{"start":"178","length":"1"}]},{"pgid":"4.5","version":"625'32","reported_seq":1942,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-09-20T22:55:33.367716+0000","last_change":"2024-09-20T22:54:41.994229+0000","last_active":"2024-09-20T22:55:33.367716+0000","last_peered":"2024-09-20T22:55:33.367716+0000","last_clean":"2024-09-20T22:55:33.367716+0000","last_became_active":"2024-09-20T22:48:05.243523+0000","last_became_peered":"2024-09-20T22:48:05.243523+0000","last_unstale":"2024-09-20T22:55:33.367716+0000","last_undegraded":"2024-09-20T22:55:33.367716+0000","last_fullsized":"2024-09-20T22:55:33.367716+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'32","last_scrub_stamp":"2024-09-20T22:54:41.994141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:54:41.994141+0000","objects_scrubbed":8,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:56:06.164224+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"67'1","reported_seq":1624,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-09-20T22:55:33.544268+0000","last_change":"2024-09-20T22:55:33.544290+0000","last_active":"2024-09-20T22:55:33.544268+0000","last_peered":"2024-09-20T22:55:33.544268+0000","last_clean":"2024-09-20T22:55:33.544268+0000","last_became_active":"2024-09-20T22:41:38.158917+0000","last_became_peered":"2024-09-20T22:41:38.158917+0000","last_unstale":"2024-09-20T22:55:33.544268+0000","last_undegraded":"2024-09-20T22:55:33.544268+0000","last_fullsized":"2024-09-20T22:55:33.544268+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-09-20T22:55:33.544218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:55:33.544218+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:00.545241+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"77'128","reported_seq":1974,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-09-20T22:55:33.369414+0000","last_change":"2024-09-20T22:54:21.616304+0000","last_active":"2024-09-20T22:55:33.369414+0000","last_peered":"2024-09-20T22:55:33.369414+0000","last_clean":"2024-09-20T22:55:33.369414+0000","last_became_active":"2024-09-20T22:47:52.206169+0000","last_became_peered":"2024-09-20T22:47:52.206169+0000","last_unstale":"2024-09-20T22:55:33.369414+0000","last_undegraded":"2024-09-20T22:55:33.369414+0000","last_fullsized":"2024-09-20T22:55:33.369414+0000","mapping_epoch":391,"log_start":"77'128","ondisk_log_start":"77'128","created":19,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:54:21.616231+0000","last_deep_scrub":"77'128","last_deep_scrub_stamp":"2024-09-20T22:47:22.417099+0000","last_clean_scrub_stamp":"2024-09-20T22:54:21.616231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":1631,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-09-20T22:55:33.370886+0000","last_change":"2024-09-20T22:55:10.707884+0000","last_active":"2024-09-20T22:55:33.370886+0000","last_peered":"2024-09-20T22:55:33.370886+0000","last_clean":"2024-09-20T22:55:33.370886+0000","last_became_active":"2024-09-20T22:41:42.184387+0000","last_became_peered":"2024-09-20T22:41:42.184387+0000","last_unstale":"2024-09-20T22:55:33.370886+0000","last_undegraded":"2024-09-20T22:55:33.370886+0000","last_fullsized":"2024-09-20T22:55:33.370886+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:55:10.707805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:55:10.707805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:56:23.981708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":1626,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-09-20T22:55:33.349139+0000","last_change":"2024-09-20T22:53:09.957459+0000","last_active":"2024-09-20T22:55:33.349139+0000","last_peered":"2024-09-20T22:55:33.349139+0000","last_clean":"2024-09-20T22:55:33.349139+0000","last_became_active":"2024-09-20T22:48:05.241602+0000","last_became_peered":"2024-09-20T22:48:05.241602+0000","last_unstale":"2024-09-20T22:55:33.349139+0000","last_undegraded":"2024-09-20T22:55:33.349139+0000","last_fullsized":"2024-09-20T22:55:33.349139+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:09.957374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:53:09.957374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"851'755","reported_seq":4694,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-09-20T22:55:33.367563+0000","last_change":"2024-09-20T22:55:25.353286+0000","last_active":"2024-09-20T22:55:33.367563+0000","last_peered":"2024-09-20T22:55:33.367563+0000","last_clean":"2024-09-20T22:55:33.367563+0000","last_became_active":"2024-09-20T22:54:20.414528+0000","last_became_peered":"2024-09-20T22:54:20.414528+0000","last_unstale":"2024-09-20T22:55:33.367563+0000","last_undegraded":"2024-09-20T22:55:33.367563+0000","last_fullsized":"2024-09-20T22:55:33.367563+0000","mapping_epoch":779,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":780,"parent":"0.0","parent_split_bits":0,"last_scrub":"770'632","last_scrub_stamp":"2024-09-20T22:54:13.158459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:54:13.158459+0000","objects_scrubbed":10,"log_size":755,"log_dups_size":0,"ondisk_log_size":755,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:40.013847+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.011553653000000001,"stat_sum":{"num_bytes":32040425,"num_objects":14,"num_object_clones":12,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":242,"num_read_kb":242909,"num_write":133,"num_write_kb":59454,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":88,"num_bytes_recovered":51113242,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"13e","length":"b"},{"start":"14b","length":"3"},{"start":"150","length":"5"},{"start":"156","length":"1"},{"start":"159","length":"1"},{"start":"15b","length":"1"},{"start":"15d","length":"3"},{"start":"161","length":"5"},{"start":"167","length":"4"},{"start":"16d","length":"1"},{"start":"170","length":"2"},{"start":"178","length":"1"}]},{"pgid":"4.e","version":"850'24","reported_seq":557,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-09-20T22:55:33.366754+0000","last_change":"2024-09-20T22:55:30.779124+0000","last_active":"2024-09-20T22:55:33.366754+0000","last_peered":"2024-09-20T22:55:33.366754+0000","last_clean":"2024-09-20T22:55:33.366754+0000","last_became_active":"2024-09-20T22:55:17.342390+0000","last_became_peered":"2024-09-20T22:55:17.342390+0000","last_unstale":"2024-09-20T22:55:33.366754+0000","last_undegraded":"2024-09-20T22:55:33.366754+0000","last_fullsized":"2024-09-20T22:55:33.366754+0000","mapping_epoch":836,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":837,"parent":"0.0","parent_split_bits":0,"last_scrub":"850'24","last_scrub_stamp":"2024-09-20T22:55:30.779018+0000","last_deep_scrub":"850'24","last_deep_scrub_stamp":"2024-09-20T22:55:30.779018+0000","last_clean_scrub_stamp":"2024-09-20T22:55:30.779018+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:56:58.207882+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":23,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":562,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-09-20T22:55:33.348065+0000","last_change":"2024-09-20T22:55:17.344028+0000","last_active":"2024-09-20T22:55:33.348065+0000","last_peered":"2024-09-20T22:55:33.348065+0000","last_clean":"2024-09-20T22:55:33.348065+0000","last_became_active":"2024-09-20T22:55:17.343742+0000","last_became_peered":"2024-09-20T22:55:17.343742+0000","last_unstale":"2024-09-20T22:55:33.348065+0000","last_undegraded":"2024-09-20T22:55:33.348065+0000","last_fullsized":"2024-09-20T22:55:33.348065+0000","mapping_epoch":836,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":837,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:54:21.423766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:54:21.423766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:47.521577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":1630,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-09-20T22:55:33.349068+0000","last_change":"2024-09-20T22:54:47.166950+0000","last_active":"2024-09-20T22:55:33.349068+0000","last_peered":"2024-09-20T22:55:33.349068+0000","last_clean":"2024-09-20T22:55:33.349068+0000","last_became_active":"2024-09-20T22:48:05.233778+0000","last_became_peered":"2024-09-20T22:48:05.233778+0000","last_unstale":"2024-09-20T22:55:33.349068+0000","last_undegraded":"2024-09-20T22:55:33.349068+0000","last_fullsized":"2024-09-20T22:55:33.349068+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:54:47.166836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:54:47.166836+0000","last_clean_scrub_stamp":"2024-09-20T22:54:47.166836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:56:01.866772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":1606,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-09-20T22:55:33.345926+0000","last_change":"2024-09-20T22:53:10.647075+0000","last_active":"2024-09-20T22:55:33.345926+0000","last_peered":"2024-09-20T22:55:33.345926+0000","last_clean":"2024-09-20T22:55:33.345926+0000","last_became_active":"2024-09-20T22:41:44.832009+0000","last_became_peered":"2024-09-20T22:41:44.832009+0000","last_unstale":"2024-09-20T22:55:33.345926+0000","last_undegraded":"2024-09-20T22:55:33.345926+0000","last_fullsized":"2024-09-20T22:55:33.345926+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:10.647005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:48:40.127123+0000","last_clean_scrub_stamp":"2024-09-20T22:53:10.647005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.e","version":"851'895","reported_seq":4671,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-09-20T22:55:33.349681+0000","last_change":"2024-09-20T22:55:25.346803+0000","last_active":"2024-09-20T22:55:33.349681+0000","last_peered":"2024-09-20T22:55:33.349681+0000","last_clean":"2024-09-20T22:55:33.349681+0000","last_became_active":"2024-09-20T22:47:52.209452+0000","last_became_peered":"2024-09-20T22:47:52.209452+0000","last_unstale":"2024-09-20T22:55:33.349681+0000","last_undegraded":"2024-09-20T22:55:33.349681+0000","last_fullsized":"2024-09-20T22:55:33.349681+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"778'821","last_scrub_stamp":"2024-09-20T22:54:19.513024+0000","last_deep_scrub":"174'60","last_deep_scrub_stamp":"2024-09-20T22:44:18.301776+0000","last_clean_scrub_stamp":"2024-09-20T22:54:19.513024+0000","objects_scrubbed":13,"log_size":895,"log_dups_size":0,"ondisk_log_size":895,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:34.127781+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0.0070843269999999996,"stat_sum":{"num_bytes":31305528,"num_objects":15,"num_object_clones":13,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":1,"num_read":262,"num_read_kb":256140,"num_write":203,"num_write_kb":74788,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":23110741,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"13e","length":"b"},{"start":"14b","length":"3"},{"start":"150","length":"5"},{"start":"156","length":"1"},{"start":"159","length":"1"},{"start":"15b","length":"1"},{"start":"15d","length":"3"},{"start":"161","length":"5"},{"start":"167","length":"4"},{"start":"16d","length":"1"},{"start":"170","length":"2"},{"start":"178","length":"1"}]},{"pgid":"1.8","version":"77'128","reported_seq":1894,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-09-20T22:55:33.348350+0000","last_change":"2024-09-20T22:54:46.128026+0000","last_active":"2024-09-20T22:55:33.348350+0000","last_peered":"2024-09-20T22:55:33.348350+0000","last_clean":"2024-09-20T22:55:33.348350+0000","last_became_active":"2024-09-20T22:42:52.405594+0000","last_became_peered":"2024-09-20T22:42:52.405594+0000","last_unstale":"2024-09-20T22:55:33.348350+0000","last_undegraded":"2024-09-20T22:55:33.348350+0000","last_fullsized":"2024-09-20T22:55:33.348350+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:54:46.127943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:54:46.127943+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:49.849638+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"625'31","reported_seq":581,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-09-20T22:55:33.348200+0000","last_change":"2024-09-20T22:55:26.193125+0000","last_active":"2024-09-20T22:55:33.348200+0000","last_peered":"2024-09-20T22:55:33.348200+0000","last_clean":"2024-09-20T22:55:33.348200+0000","last_became_active":"2024-09-20T22:55:17.342262+0000","last_became_peered":"2024-09-20T22:55:17.342262+0000","last_unstale":"2024-09-20T22:55:33.348200+0000","last_undegraded":"2024-09-20T22:55:33.348200+0000","last_fullsized":"2024-09-20T22:55:33.348200+0000","mapping_epoch":836,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":837,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'31","last_scrub_stamp":"2024-09-20T22:55:26.193034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:55:26.193034+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:56:27.335087+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":48,"num_read_kb":38,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":1194,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-09-20T22:55:33.350367+0000","last_change":"2024-09-20T22:54:51.323936+0000","last_active":"2024-09-20T22:55:33.350367+0000","last_peered":"2024-09-20T22:55:33.350367+0000","last_clean":"2024-09-20T22:55:33.350367+0000","last_became_active":"2024-09-20T22:54:17.332636+0000","last_became_peered":"2024-09-20T22:54:17.332636+0000","last_unstale":"2024-09-20T22:55:33.350367+0000","last_undegraded":"2024-09-20T22:55:33.350367+0000","last_fullsized":"2024-09-20T22:55:33.350367+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:54:51.323855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:35.492562+0000","last_clean_scrub_stamp":"2024-09-20T22:54:51.323855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:58.729458+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":171,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-09-20T22:55:33.333251+0000","last_change":"2024-09-20T22:55:08.995095+0000","last_active":"2024-09-20T22:55:33.333251+0000","last_peered":"2024-09-20T22:55:33.333251+0000","last_clean":"2024-09-20T22:55:33.333251+0000","last_became_active":"2024-09-20T22:54:17.349144+0000","last_became_peered":"2024-09-20T22:54:17.349144+0000","last_unstale":"2024-09-20T22:55:33.333251+0000","last_undegraded":"2024-09-20T22:55:33.333251+0000","last_fullsized":"2024-09-20T22:55:33.333251+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:55:08.995021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:55:08.995021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:56:09.366460+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":1649,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-09-20T22:55:33.349735+0000","last_change":"2024-09-20T22:54:41.501615+0000","last_active":"2024-09-20T22:55:33.349735+0000","last_peered":"2024-09-20T22:55:33.349735+0000","last_clean":"2024-09-20T22:55:33.349735+0000","last_became_active":"2024-09-20T22:48:05.239006+0000","last_became_peered":"2024-09-20T22:48:05.239006+0000","last_unstale":"2024-09-20T22:55:33.349735+0000","last_undegraded":"2024-09-20T22:55:33.349735+0000","last_fullsized":"2024-09-20T22:55:33.349735+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:54:41.501549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:54:41.501549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:54.722407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"845'973","reported_seq":4748,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-09-20T22:55:33.333831+0000","last_change":"2024-09-20T22:55:25.351059+0000","last_active":"2024-09-20T22:55:33.333831+0000","last_peered":"2024-09-20T22:55:33.333831+0000","last_clean":"2024-09-20T22:55:33.333831+0000","last_became_active":"2024-09-20T22:47:52.216198+0000","last_became_peered":"2024-09-20T22:47:52.216198+0000","last_unstale":"2024-09-20T22:55:33.333831+0000","last_undegraded":"2024-09-20T22:55:33.333831+0000","last_fullsized":"2024-09-20T22:55:33.333831+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"754'838","last_scrub_stamp":"2024-09-20T22:53:54.908987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:53:54.908987+0000","objects_scrubbed":17,"log_size":973,"log_dups_size":0,"ondisk_log_size":973,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.015584246,"stat_sum":{"num_bytes":33180138,"num_objects":15,"num_object_clones":12,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":390,"num_read_kb":379761,"num_write":229,"num_write_kb":81949,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":53,"num_bytes_recovered":14983556,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"13e","length":"b"},{"start":"14b","length":"3"},{"start":"150","length":"5"},{"start":"156","length":"1"},{"start":"159","length":"1"},{"start":"15b","length":"1"},{"start":"15d","length":"3"},{"start":"161","length":"5"},{"start":"167","length":"4"},{"start":"16d","length":"1"},{"start":"170","length":"2"},{"start":"178","length":"1"}]},{"pgid":"1.9","version":"77'128","reported_seq":1950,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-09-20T22:55:33.369233+0000","last_change":"2024-09-20T22:55:04.347145+0000","last_active":"2024-09-20T22:55:33.369233+0000","last_peered":"2024-09-20T22:55:33.369233+0000","last_clean":"2024-09-20T22:55:33.369233+0000","last_became_active":"2024-09-20T22:55:04.346893+0000","last_became_peered":"2024-09-20T22:55:04.346893+0000","last_unstale":"2024-09-20T22:55:33.369233+0000","last_undegraded":"2024-09-20T22:55:33.369233+0000","last_fullsized":"2024-09-20T22:55:33.369233+0000","mapping_epoch":823,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":824,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:53:50.381052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:53:50.381052+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"625'20","reported_seq":529,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-09-20T22:55:33.345629+0000","last_change":"2024-09-20T22:55:17.349857+0000","last_active":"2024-09-20T22:55:33.345629+0000","last_peered":"2024-09-20T22:55:33.345629+0000","last_clean":"2024-09-20T22:55:33.345629+0000","last_became_active":"2024-09-20T22:55:17.349462+0000","last_became_peered":"2024-09-20T22:55:17.349462+0000","last_unstale":"2024-09-20T22:55:33.345629+0000","last_undegraded":"2024-09-20T22:55:33.345629+0000","last_fullsized":"2024-09-20T22:55:33.345629+0000","mapping_epoch":836,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":837,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'20","last_scrub_stamp":"2024-09-20T22:52:22.646468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:52:22.646468+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":28,"num_read_kb":23,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":1642,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-09-20T22:55:33.345240+0000","last_change":"2024-09-20T22:52:44.527004+0000","last_active":"2024-09-20T22:55:33.345240+0000","last_peered":"2024-09-20T22:55:33.345240+0000","last_clean":"2024-09-20T22:55:33.345240+0000","last_became_active":"2024-09-20T22:48:05.233999+0000","last_became_peered":"2024-09-20T22:48:05.233999+0000","last_unstale":"2024-09-20T22:55:33.345240+0000","last_undegraded":"2024-09-20T22:55:33.345240+0000","last_fullsized":"2024-09-20T22:55:33.345240+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:44.526935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:52:44.526935+0000","last_clean_scrub_stamp":"2024-09-20T22:52:44.526935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"261'9","reported_seq":1810,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-09-20T22:55:33.349983+0000","last_change":"2024-09-20T22:53:54.190905+0000","last_active":"2024-09-20T22:55:33.349983+0000","last_peered":"2024-09-20T22:55:33.349983+0000","last_clean":"2024-09-20T22:55:33.349983+0000","last_became_active":"2024-09-20T22:41:42.185863+0000","last_became_peered":"2024-09-20T22:41:42.185863+0000","last_unstale":"2024-09-20T22:55:33.349983+0000","last_undegraded":"2024-09-20T22:55:33.349983+0000","last_fullsized":"2024-09-20T22:55:33.349983+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'9","last_scrub_stamp":"2024-09-20T22:53:54.190823+0000","last_deep_scrub":"261'9","last_deep_scrub_stamp":"2024-09-20T22:46:50.974735+0000","last_clean_scrub_stamp":"2024-09-20T22:53:54.190823+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":171,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-09-20T22:55:33.332762+0000","last_change":"2024-09-20T22:54:43.022989+0000","last_active":"2024-09-20T22:55:33.332762+0000","last_peered":"2024-09-20T22:55:33.332762+0000","last_clean":"2024-09-20T22:55:33.332762+0000","last_became_active":"2024-09-20T22:54:17.340684+0000","last_became_peered":"2024-09-20T22:54:17.340684+0000","last_unstale":"2024-09-20T22:55:33.332762+0000","last_undegraded":"2024-09-20T22:55:33.332762+0000","last_fullsized":"2024-09-20T22:55:33.332762+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:54:43.022914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:54:43.022914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:56:01.405842+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.8","version":"851'1165","reported_seq":4774,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-09-20T22:55:33.349731+0000","last_change":"2024-09-20T22:55:32.164431+0000","last_active":"2024-09-20T22:55:33.349731+0000","last_peered":"2024-09-20T22:55:33.349731+0000","last_clean":"2024-09-20T22:55:33.349731+0000","last_became_active":"2024-09-20T22:48:31.224359+0000","last_became_peered":"2024-09-20T22:48:31.224359+0000","last_unstale":"2024-09-20T22:55:33.349731+0000","last_undegraded":"2024-09-20T22:55:33.349731+0000","last_fullsized":"2024-09-20T22:55:33.349731+0000","mapping_epoch":407,"log_start":"426'268","ondisk_log_start":"426'268","created":94,"last_epoch_clean":431,"parent":"0.0","parent_split_bits":0,"last_scrub":"851'1165","last_scrub_stamp":"2024-09-20T22:55:32.164332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:55:32.164332+0000","objects_scrubbed":19,"log_size":897,"log_dups_size":445,"ondisk_log_size":897,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:56:38.835737+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.011805184999999999,"stat_sum":{"num_bytes":40600732,"num_objects":19,"num_object_clones":16,"num_object_copies":57,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":364,"num_read_kb":382130,"num_write":285,"num_write_kb":90623,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":26027409,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"13e","length":"b"},{"start":"14b","length":"3"},{"start":"150","length":"5"},{"start":"156","length":"1"},{"start":"159","length":"1"},{"start":"15b","length":"1"},{"start":"15d","length":"3"},{"start":"161","length":"5"},{"start":"167","length":"4"},{"start":"16d","length":"1"},{"start":"170","length":"2"},{"start":"178","length":"1"}]},{"pgid":"4.b","version":"625'15","reported_seq":1041,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-09-20T22:55:33.333005+0000","last_change":"2024-09-20T22:55:32.601025+0000","last_active":"2024-09-20T22:55:33.333005+0000","last_peered":"2024-09-20T22:55:33.333005+0000","last_clean":"2024-09-20T22:55:33.333005+0000","last_became_active":"2024-09-20T22:54:17.340998+0000","last_became_peered":"2024-09-20T22:54:17.340998+0000","last_unstale":"2024-09-20T22:55:33.333005+0000","last_undegraded":"2024-09-20T22:55:33.333005+0000","last_fullsized":"2024-09-20T22:55:33.333005+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'15","last_scrub_stamp":"2024-09-20T22:55:32.600891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:55:32.600891+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:56:47.449824+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":1626,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-09-20T22:55:33.349248+0000","last_change":"2024-09-20T22:53:55.019427+0000","last_active":"2024-09-20T22:55:33.349248+0000","last_peered":"2024-09-20T22:55:33.349248+0000","last_clean":"2024-09-20T22:55:33.349248+0000","last_became_active":"2024-09-20T22:41:38.155953+0000","last_became_peered":"2024-09-20T22:41:38.155953+0000","last_unstale":"2024-09-20T22:55:33.349248+0000","last_undegraded":"2024-09-20T22:55:33.349248+0000","last_fullsized":"2024-09-20T22:55:33.349248+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:55.019362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:53:55.019362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":1657,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-09-20T22:55:33.349950+0000","last_change":"2024-09-20T22:54:47.368607+0000","last_active":"2024-09-20T22:55:33.349950+0000","last_peered":"2024-09-20T22:55:33.349950+0000","last_clean":"2024-09-20T22:55:33.349950+0000","last_became_active":"2024-09-20T22:48:05.240613+0000","last_became_peered":"2024-09-20T22:48:05.240613+0000","last_unstale":"2024-09-20T22:55:33.349950+0000","last_undegraded":"2024-09-20T22:55:33.349950+0000","last_fullsized":"2024-09-20T22:55:33.349950+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:54:47.368529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:54:47.368529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:58.934561+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":966,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-09-20T22:55:33.332101+0000","last_change":"2024-09-20T22:54:38.856800+0000","last_active":"2024-09-20T22:55:33.332101+0000","last_peered":"2024-09-20T22:55:33.332101+0000","last_clean":"2024-09-20T22:55:33.332101+0000","last_became_active":"2024-09-20T22:47:49.219486+0000","last_became_peered":"2024-09-20T22:47:49.219486+0000","last_unstale":"2024-09-20T22:55:33.332101+0000","last_undegraded":"2024-09-20T22:55:33.332101+0000","last_fullsized":"2024-09-20T22:55:33.332101+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:54:38.856726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:54:38.856726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:56.536952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"853'1539","reported_seq":5406,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-09-20T22:55:33.363962+0000","last_change":"2024-09-20T22:55:25.346433+0000","last_active":"2024-09-20T22:55:33.363962+0000","last_peered":"2024-09-20T22:55:33.363962+0000","last_clean":"2024-09-20T22:55:33.363962+0000","last_became_active":"2024-09-20T22:54:20.414945+0000","last_became_peered":"2024-09-20T22:54:20.414945+0000","last_unstale":"2024-09-20T22:55:33.363962+0000","last_undegraded":"2024-09-20T22:55:33.363962+0000","last_fullsized":"2024-09-20T22:55:33.363962+0000","mapping_epoch":779,"log_start":"423'280","ondisk_log_start":"423'280","created":94,"last_epoch_clean":780,"parent":"0.0","parent_split_bits":0,"last_scrub":"774'1348","last_scrub_stamp":"2024-09-20T22:54:15.010110+0000","last_deep_scrub":"590'828","last_deep_scrub_stamp":"2024-09-20T22:51:10.909364+0000","last_clean_scrub_stamp":"2024-09-20T22:54:15.010110+0000","objects_scrubbed":19,"log_size":1259,"log_dups_size":649,"ondisk_log_size":1259,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:35.517391+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.012100260999999999,"stat_sum":{"num_bytes":63389127,"num_objects":27,"num_object_clones":22,"num_object_copies":81,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":1,"num_read":546,"num_read_kb":565057,"num_write":394,"num_write_kb":155540,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":109,"num_bytes_recovered":133262200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"13e","length":"b"},{"start":"14b","length":"3"},{"start":"150","length":"5"},{"start":"156","length":"1"},{"start":"159","length":"1"},{"start":"15b","length":"1"},{"start":"15d","length":"3"},{"start":"161","length":"5"},{"start":"167","length":"4"},{"start":"16d","length":"1"},{"start":"170","length":"2"},{"start":"178","length":"1"}]},{"pgid":"4.a","version":"625'44","reported_seq":1762,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-09-20T22:55:33.367061+0000","last_change":"2024-09-20T22:55:08.851508+0000","last_active":"2024-09-20T22:55:33.367061+0000","last_peered":"2024-09-20T22:55:33.367061+0000","last_clean":"2024-09-20T22:55:33.367061+0000","last_became_active":"2024-09-20T22:54:17.341124+0000","last_became_peered":"2024-09-20T22:54:17.341124+0000","last_unstale":"2024-09-20T22:55:33.367061+0000","last_undegraded":"2024-09-20T22:55:33.367061+0000","last_fullsized":"2024-09-20T22:55:33.367061+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'44","last_scrub_stamp":"2024-09-20T22:55:08.851376+0000","last_deep_scrub":"113'28","last_deep_scrub_stamp":"2024-09-20T22:43:12.725900+0000","last_clean_scrub_stamp":"2024-09-20T22:55:08.851376+0000","objects_scrubbed":9,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:56:10.587942+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":1631,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-09-20T22:55:33.369968+0000","last_change":"2024-09-20T22:55:29.772681+0000","last_active":"2024-09-20T22:55:33.369968+0000","last_peered":"2024-09-20T22:55:33.369968+0000","last_clean":"2024-09-20T22:55:33.369968+0000","last_became_active":"2024-09-20T22:41:38.161524+0000","last_became_peered":"2024-09-20T22:41:38.161524+0000","last_unstale":"2024-09-20T22:55:33.369968+0000","last_undegraded":"2024-09-20T22:55:33.369968+0000","last_fullsized":"2024-09-20T22:55:33.369968+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:55:29.772589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:55:29.772589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:56:54.387744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":1645,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-09-20T22:55:33.349172+0000","last_change":"2024-09-20T22:54:21.496172+0000","last_active":"2024-09-20T22:55:33.349172+0000","last_peered":"2024-09-20T22:55:33.349172+0000","last_clean":"2024-09-20T22:55:33.349172+0000","last_became_active":"2024-09-20T22:47:49.223220+0000","last_became_peered":"2024-09-20T22:47:49.223220+0000","last_unstale":"2024-09-20T22:55:33.349172+0000","last_undegraded":"2024-09-20T22:55:33.349172+0000","last_fullsized":"2024-09-20T22:55:33.349172+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:54:21.496068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:54:21.496068+0000","last_clean_scrub_stamp":"2024-09-20T22:54:21.496068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":1619,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-09-20T22:55:33.369275+0000","last_change":"2024-09-20T22:53:53.365864+0000","last_active":"2024-09-20T22:55:33.369275+0000","last_peered":"2024-09-20T22:55:33.369275+0000","last_clean":"2024-09-20T22:55:33.369275+0000","last_became_active":"2024-09-20T22:41:44.199395+0000","last_became_peered":"2024-09-20T22:41:44.199395+0000","last_unstale":"2024-09-20T22:55:33.369275+0000","last_undegraded":"2024-09-20T22:55:33.369275+0000","last_fullsized":"2024-09-20T22:55:33.369275+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:53.365789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:47:18.329209+0000","last_clean_scrub_stamp":"2024-09-20T22:53:53.365789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.a","version":"853'1081","reported_seq":4783,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-09-20T22:55:33.361414+0000","last_change":"2024-09-20T22:55:32.567243+0000","last_active":"2024-09-20T22:55:33.361414+0000","last_peered":"2024-09-20T22:55:33.361414+0000","last_clean":"2024-09-20T22:55:33.361414+0000","last_became_active":"2024-09-20T22:42:52.405085+0000","last_became_peered":"2024-09-20T22:42:52.405085+0000","last_unstale":"2024-09-20T22:55:33.361414+0000","last_undegraded":"2024-09-20T22:55:33.361414+0000","last_fullsized":"2024-09-20T22:55:33.361414+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"851'1076","last_scrub_stamp":"2024-09-20T22:55:32.567038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:55:32.567038+0000","objects_scrubbed":21,"log_size":1081,"log_dups_size":0,"ondisk_log_size":1081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:56:32.927520+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0.13898474199999999,"stat_sum":{"num_bytes":49545917,"num_objects":22,"num_object_clones":19,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":1,"num_read":416,"num_read_kb":434011,"num_write":320,"num_write_kb":130612,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"13e","length":"b"},{"start":"14b","length":"3"},{"start":"150","length":"5"},{"start":"156","length":"1"},{"start":"159","length":"1"},{"start":"15b","length":"1"},{"start":"15d","length":"3"},{"start":"161","length":"5"},{"start":"167","length":"4"},{"start":"16d","length":"1"},{"start":"170","length":"2"},{"start":"178","length":"1"}]},{"pgid":"4.9","version":"686'23","reported_seq":883,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-09-20T22:55:33.332135+0000","last_change":"2024-09-20T22:55:25.836827+0000","last_active":"2024-09-20T22:55:33.332135+0000","last_peered":"2024-09-20T22:55:33.332135+0000","last_clean":"2024-09-20T22:55:33.332135+0000","last_became_active":"2024-09-20T22:55:17.346757+0000","last_became_peered":"2024-09-20T22:55:17.346757+0000","last_unstale":"2024-09-20T22:55:33.332135+0000","last_undegraded":"2024-09-20T22:55:33.332135+0000","last_fullsized":"2024-09-20T22:55:33.332135+0000","mapping_epoch":836,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":837,"parent":"0.0","parent_split_bits":0,"last_scrub":"686'23","last_scrub_stamp":"2024-09-20T22:55:25.836747+0000","last_deep_scrub":"686'23","last_deep_scrub_stamp":"2024-09-20T22:52:46.768812+0000","last_clean_scrub_stamp":"2024-09-20T22:55:25.836747+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:56:53.576369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":21,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":1675,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-09-20T22:55:33.369924+0000","last_change":"2024-09-20T22:55:28.819125+0000","last_active":"2024-09-20T22:55:33.369924+0000","last_peered":"2024-09-20T22:55:33.369924+0000","last_clean":"2024-09-20T22:55:33.369924+0000","last_became_active":"2024-09-20T22:54:17.332759+0000","last_became_peered":"2024-09-20T22:54:17.332759+0000","last_unstale":"2024-09-20T22:55:33.369924+0000","last_undegraded":"2024-09-20T22:55:33.369924+0000","last_fullsized":"2024-09-20T22:55:33.369924+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:55:28.819002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:46:52.465940+0000","last_clean_scrub_stamp":"2024-09-20T22:55:28.819002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:56:35.940594+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":1665,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-09-20T22:55:33.350093+0000","last_change":"2024-09-20T22:54:48.350732+0000","last_active":"2024-09-20T22:55:33.350093+0000","last_peered":"2024-09-20T22:55:33.350093+0000","last_clean":"2024-09-20T22:55:33.350093+0000","last_became_active":"2024-09-20T22:54:17.340269+0000","last_became_peered":"2024-09-20T22:54:17.340269+0000","last_unstale":"2024-09-20T22:55:33.350093+0000","last_undegraded":"2024-09-20T22:55:33.350093+0000","last_fullsized":"2024-09-20T22:55:33.350093+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:54:48.350593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:53:07.104891+0000","last_clean_scrub_stamp":"2024-09-20T22:54:48.350593+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:56:05.996922+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":1618,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-09-20T22:55:33.344959+0000","last_change":"2024-09-20T22:54:17.331117+0000","last_active":"2024-09-20T22:55:33.344959+0000","last_peered":"2024-09-20T22:55:33.344959+0000","last_clean":"2024-09-20T22:55:33.344959+0000","last_became_active":"2024-09-20T22:54:17.329277+0000","last_became_peered":"2024-09-20T22:54:17.329277+0000","last_unstale":"2024-09-20T22:55:33.344959+0000","last_undegraded":"2024-09-20T22:55:33.344959+0000","last_fullsized":"2024-09-20T22:55:33.344959+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:53.242204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:55.170379+0000","last_clean_scrub_stamp":"2024-09-20T22:53:53.242204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"848'1348","reported_seq":5230,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-09-20T22:55:33.334401+0000","last_change":"2024-09-20T22:55:30.380223+0000","last_active":"2024-09-20T22:55:33.334401+0000","last_peered":"2024-09-20T22:55:33.334401+0000","last_clean":"2024-09-20T22:55:33.334401+0000","last_became_active":"2024-09-20T22:54:20.415706+0000","last_became_peered":"2024-09-20T22:54:20.415706+0000","last_unstale":"2024-09-20T22:55:33.334401+0000","last_undegraded":"2024-09-20T22:55:33.334401+0000","last_fullsized":"2024-09-20T22:55:33.334401+0000","mapping_epoch":779,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":780,"parent":"0.0","parent_split_bits":0,"last_scrub":"794'1255","last_scrub_stamp":"2024-09-20T22:54:37.866509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:54:37.866509+0000","objects_scrubbed":15,"log_size":1348,"log_dups_size":0,"ondisk_log_size":1348,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:50.935517+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.040075363000000003,"stat_sum":{"num_bytes":43581556,"num_objects":18,"num_object_clones":13,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":2,"num_read":597,"num_read_kb":676024,"num_write":384,"num_write_kb":146018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":168,"num_bytes_recovered":68651320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"13e","length":"b"},{"start":"14b","length":"3"},{"start":"150","length":"5"},{"start":"156","length":"1"},{"start":"159","length":"1"},{"start":"15b","length":"1"},{"start":"15d","length":"3"},{"start":"161","length":"5"},{"start":"167","length":"4"},{"start":"16d","length":"1"},{"start":"170","length":"2"},{"start":"178","length":"1"}]},{"pgid":"4.8","version":"625'27","reported_seq":1665,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-09-20T22:55:33.348523+0000","last_change":"2024-09-20T22:54:41.095070+0000","last_active":"2024-09-20T22:55:33.348523+0000","last_peered":"2024-09-20T22:55:33.348523+0000","last_clean":"2024-09-20T22:55:33.348523+0000","last_became_active":"2024-09-20T22:41:40.252113+0000","last_became_peered":"2024-09-20T22:41:40.252113+0000","last_unstale":"2024-09-20T22:55:33.348523+0000","last_undegraded":"2024-09-20T22:55:33.348523+0000","last_fullsized":"2024-09-20T22:55:33.348523+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'27","last_scrub_stamp":"2024-09-20T22:54:41.094989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:54:41.094989+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:57.543047+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'2","reported_seq":1686,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-09-20T22:55:33.370425+0000","last_change":"2024-09-20T22:53:33.526131+0000","last_active":"2024-09-20T22:55:33.370425+0000","last_peered":"2024-09-20T22:55:33.370425+0000","last_clean":"2024-09-20T22:55:33.370425+0000","last_became_active":"2024-09-20T22:48:05.238098+0000","last_became_peered":"2024-09-20T22:48:05.238098+0000","last_unstale":"2024-09-20T22:55:33.370425+0000","last_undegraded":"2024-09-20T22:55:33.370425+0000","last_fullsized":"2024-09-20T22:55:33.370425+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-09-20T22:53:33.526049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:53:33.526049+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"408'10","reported_seq":1827,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-09-20T22:55:33.369423+0000","last_change":"2024-09-20T22:55:26.889319+0000","last_active":"2024-09-20T22:55:33.369423+0000","last_peered":"2024-09-20T22:55:33.369423+0000","last_clean":"2024-09-20T22:55:33.369423+0000","last_became_active":"2024-09-20T22:48:05.239344+0000","last_became_peered":"2024-09-20T22:48:05.239344+0000","last_unstale":"2024-09-20T22:55:33.369423+0000","last_undegraded":"2024-09-20T22:55:33.369423+0000","last_fullsized":"2024-09-20T22:55:33.369423+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"408'10","last_scrub_stamp":"2024-09-20T22:55:26.889218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:55:26.889218+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:56:51.948935+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":1626,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-09-20T22:55:33.349538+0000","last_change":"2024-09-20T22:54:43.129941+0000","last_active":"2024-09-20T22:55:33.349538+0000","last_peered":"2024-09-20T22:55:33.349538+0000","last_clean":"2024-09-20T22:55:33.349538+0000","last_became_active":"2024-09-20T22:47:49.227569+0000","last_became_peered":"2024-09-20T22:47:49.227569+0000","last_unstale":"2024-09-20T22:55:33.349538+0000","last_undegraded":"2024-09-20T22:55:33.349538+0000","last_fullsized":"2024-09-20T22:55:33.349538+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:54:43.129868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:54:43.129868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:56:10.190377+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":1661,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-09-20T22:55:33.367180+0000","last_change":"2024-09-20T22:54:38.024885+0000","last_active":"2024-09-20T22:55:33.367180+0000","last_peered":"2024-09-20T22:55:33.367180+0000","last_clean":"2024-09-20T22:55:33.367180+0000","last_became_active":"2024-09-20T22:47:49.213292+0000","last_became_peered":"2024-09-20T22:47:49.213292+0000","last_unstale":"2024-09-20T22:55:33.367180+0000","last_undegraded":"2024-09-20T22:55:33.367180+0000","last_fullsized":"2024-09-20T22:55:33.367180+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:54:38.024794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:54:38.024794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:55.383474+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"625'12","reported_seq":1667,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-09-20T22:55:33.347120+0000","last_change":"2024-09-20T22:54:17.387599+0000","last_active":"2024-09-20T22:55:33.347120+0000","last_peered":"2024-09-20T22:55:33.347120+0000","last_clean":"2024-09-20T22:55:33.347120+0000","last_became_active":"2024-09-20T22:54:17.342098+0000","last_became_peered":"2024-09-20T22:54:17.342098+0000","last_unstale":"2024-09-20T22:55:33.347120+0000","last_undegraded":"2024-09-20T22:55:33.347120+0000","last_fullsized":"2024-09-20T22:55:33.347120+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'12","last_scrub_stamp":"2024-09-20T22:53:26.319144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:53:26.319144+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":1637,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-09-20T22:55:33.349093+0000","last_change":"2024-09-20T22:55:29.324367+0000","last_active":"2024-09-20T22:55:33.349093+0000","last_peered":"2024-09-20T22:55:33.349093+0000","last_clean":"2024-09-20T22:55:33.349093+0000","last_became_active":"2024-09-20T22:54:17.339912+0000","last_became_peered":"2024-09-20T22:54:17.339912+0000","last_unstale":"2024-09-20T22:55:33.349093+0000","last_undegraded":"2024-09-20T22:55:33.349093+0000","last_fullsized":"2024-09-20T22:55:33.349093+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:55:29.324286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:09.705632+0000","last_clean_scrub_stamp":"2024-09-20T22:55:29.324286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:56:41.944927+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"74'1","reported_seq":1639,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-09-20T22:55:33.369589+0000","last_change":"2024-09-20T22:55:07.676092+0000","last_active":"2024-09-20T22:55:33.369589+0000","last_peered":"2024-09-20T22:55:33.369589+0000","last_clean":"2024-09-20T22:55:33.369589+0000","last_became_active":"2024-09-20T22:48:05.237882+0000","last_became_peered":"2024-09-20T22:48:05.237882+0000","last_unstale":"2024-09-20T22:55:33.369589+0000","last_undegraded":"2024-09-20T22:55:33.369589+0000","last_fullsized":"2024-09-20T22:55:33.369589+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-09-20T22:55:07.676011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:55:07.676011+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:56:31.650759+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"625'15","reported_seq":993,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-09-20T22:55:33.333874+0000","last_change":"2024-09-20T22:55:27.638028+0000","last_active":"2024-09-20T22:55:33.333874+0000","last_peered":"2024-09-20T22:55:33.333874+0000","last_clean":"2024-09-20T22:55:33.333874+0000","last_became_active":"2024-09-20T22:47:49.231315+0000","last_became_peered":"2024-09-20T22:47:49.231315+0000","last_unstale":"2024-09-20T22:55:33.333874+0000","last_undegraded":"2024-09-20T22:55:33.333874+0000","last_fullsized":"2024-09-20T22:55:33.333874+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'15","last_scrub_stamp":"2024-09-20T22:55:27.637946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:55:27.637946+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:56:30.491674+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":1654,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-09-20T22:55:33.369068+0000","last_change":"2024-09-20T22:54:41.628923+0000","last_active":"2024-09-20T22:55:33.369068+0000","last_peered":"2024-09-20T22:55:33.369068+0000","last_clean":"2024-09-20T22:55:33.369068+0000","last_became_active":"2024-09-20T22:48:05.238801+0000","last_became_peered":"2024-09-20T22:48:05.238801+0000","last_unstale":"2024-09-20T22:55:33.369068+0000","last_undegraded":"2024-09-20T22:55:33.369068+0000","last_fullsized":"2024-09-20T22:55:33.369068+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:54:41.628846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:54:41.628846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:57.050553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":1621,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-09-20T22:55:33.346890+0000","last_change":"2024-09-20T22:54:17.342665+0000","last_active":"2024-09-20T22:55:33.346890+0000","last_peered":"2024-09-20T22:55:33.346890+0000","last_clean":"2024-09-20T22:55:33.346890+0000","last_became_active":"2024-09-20T22:54:17.341617+0000","last_became_peered":"2024-09-20T22:54:17.341617+0000","last_unstale":"2024-09-20T22:55:33.346890+0000","last_undegraded":"2024-09-20T22:55:33.346890+0000","last_fullsized":"2024-09-20T22:55:33.346890+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:20.410164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:52:20.410164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":1660,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-09-20T22:55:33.349045+0000","last_change":"2024-09-20T22:55:12.331222+0000","last_active":"2024-09-20T22:55:33.349045+0000","last_peered":"2024-09-20T22:55:33.349045+0000","last_clean":"2024-09-20T22:55:33.349045+0000","last_became_active":"2024-09-20T22:48:05.239710+0000","last_became_peered":"2024-09-20T22:48:05.239710+0000","last_unstale":"2024-09-20T22:55:33.349045+0000","last_undegraded":"2024-09-20T22:55:33.349045+0000","last_fullsized":"2024-09-20T22:55:33.349045+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:55:12.331111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:53:58.169894+0000","last_clean_scrub_stamp":"2024-09-20T22:55:12.331111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:56:14.829529+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"625'15","reported_seq":1646,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-09-20T22:55:33.348950+0000","last_change":"2024-09-20T22:53:53.070301+0000","last_active":"2024-09-20T22:55:33.348950+0000","last_peered":"2024-09-20T22:55:33.348950+0000","last_clean":"2024-09-20T22:55:33.348950+0000","last_became_active":"2024-09-20T22:41:40.252486+0000","last_became_peered":"2024-09-20T22:41:40.252486+0000","last_unstale":"2024-09-20T22:55:33.348950+0000","last_undegraded":"2024-09-20T22:55:33.348950+0000","last_fullsized":"2024-09-20T22:55:33.348950+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'15","last_scrub_stamp":"2024-09-20T22:53:53.070214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:53:53.070214+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":974,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-09-20T22:55:33.332631+0000","last_change":"2024-09-20T22:53:58.829885+0000","last_active":"2024-09-20T22:55:33.332631+0000","last_peered":"2024-09-20T22:55:33.332631+0000","last_clean":"2024-09-20T22:55:33.332631+0000","last_became_active":"2024-09-20T22:47:49.229936+0000","last_became_peered":"2024-09-20T22:47:49.229936+0000","last_unstale":"2024-09-20T22:55:33.332631+0000","last_undegraded":"2024-09-20T22:55:33.332631+0000","last_fullsized":"2024-09-20T22:55:33.332631+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:58.829817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:53:58.829817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"74'8","reported_seq":1788,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-09-20T22:55:33.345867+0000","last_change":"2024-09-20T22:53:59.165210+0000","last_active":"2024-09-20T22:55:33.345867+0000","last_peered":"2024-09-20T22:55:33.345867+0000","last_clean":"2024-09-20T22:55:33.345867+0000","last_became_active":"2024-09-20T22:41:42.186352+0000","last_became_peered":"2024-09-20T22:41:42.186352+0000","last_unstale":"2024-09-20T22:55:33.345867+0000","last_undegraded":"2024-09-20T22:55:33.345867+0000","last_fullsized":"2024-09-20T22:55:33.345867+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-09-20T22:53:59.165134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:53:59.165134+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":591,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-09-20T22:55:33.348486+0000","last_change":"2024-09-20T22:55:29.161667+0000","last_active":"2024-09-20T22:55:33.348486+0000","last_peered":"2024-09-20T22:55:33.348486+0000","last_clean":"2024-09-20T22:55:33.348486+0000","last_became_active":"2024-09-20T22:55:17.346270+0000","last_became_peered":"2024-09-20T22:55:17.346270+0000","last_unstale":"2024-09-20T22:55:33.348486+0000","last_undegraded":"2024-09-20T22:55:33.348486+0000","last_fullsized":"2024-09-20T22:55:33.348486+0000","mapping_epoch":836,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":837,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:55:29.161595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:55:29.161595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:56:56.713856+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.14","version":"625'20","reported_seq":1695,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-09-20T22:55:33.345454+0000","last_change":"2024-09-20T22:55:23.474445+0000","last_active":"2024-09-20T22:55:33.345454+0000","last_peered":"2024-09-20T22:55:33.345454+0000","last_clean":"2024-09-20T22:55:33.345454+0000","last_became_active":"2024-09-20T22:54:17.343066+0000","last_became_peered":"2024-09-20T22:54:17.343066+0000","last_unstale":"2024-09-20T22:55:33.345454+0000","last_undegraded":"2024-09-20T22:55:33.345454+0000","last_fullsized":"2024-09-20T22:55:33.345454+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'20","last_scrub_stamp":"2024-09-20T22:55:23.474362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:55:23.474362+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:56:50.888876+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":1642,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-09-20T22:55:33.369603+0000","last_change":"2024-09-20T22:53:56.396136+0000","last_active":"2024-09-20T22:55:33.369603+0000","last_peered":"2024-09-20T22:55:33.369603+0000","last_clean":"2024-09-20T22:55:33.369603+0000","last_became_active":"2024-09-20T22:48:05.240231+0000","last_became_peered":"2024-09-20T22:48:05.240231+0000","last_unstale":"2024-09-20T22:55:33.369603+0000","last_undegraded":"2024-09-20T22:55:33.369603+0000","last_fullsized":"2024-09-20T22:55:33.369603+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:56.396065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:53:56.396065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"778'12","reported_seq":1845,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-09-20T22:55:33.348866+0000","last_change":"2024-09-20T22:55:26.312258+0000","last_active":"2024-09-20T22:55:33.348866+0000","last_peered":"2024-09-20T22:55:33.348866+0000","last_clean":"2024-09-20T22:55:33.348866+0000","last_became_active":"2024-09-20T22:54:17.341014+0000","last_became_peered":"2024-09-20T22:54:17.341014+0000","last_unstale":"2024-09-20T22:55:33.348866+0000","last_undegraded":"2024-09-20T22:55:33.348866+0000","last_fullsized":"2024-09-20T22:55:33.348866+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"778'12","last_scrub_stamp":"2024-09-20T22:55:26.312166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:55:26.312166+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:56:35.792028+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":966,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-09-20T22:55:33.333578+0000","last_change":"2024-09-20T22:54:47.822738+0000","last_active":"2024-09-20T22:55:33.333578+0000","last_peered":"2024-09-20T22:55:33.333578+0000","last_clean":"2024-09-20T22:55:33.333578+0000","last_became_active":"2024-09-20T22:47:49.230674+0000","last_became_peered":"2024-09-20T22:47:49.230674+0000","last_unstale":"2024-09-20T22:55:33.333578+0000","last_undegraded":"2024-09-20T22:55:33.333578+0000","last_fullsized":"2024-09-20T22:55:33.333578+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:54:47.822663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:50:21.853070+0000","last_clean_scrub_stamp":"2024-09-20T22:54:47.822663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:55.206157+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"625'19","reported_seq":552,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-09-20T22:55:33.366746+0000","last_change":"2024-09-20T22:55:17.342681+0000","last_active":"2024-09-20T22:55:33.366746+0000","last_peered":"2024-09-20T22:55:33.366746+0000","last_clean":"2024-09-20T22:55:33.366746+0000","last_became_active":"2024-09-20T22:55:17.342351+0000","last_became_peered":"2024-09-20T22:55:17.342351+0000","last_unstale":"2024-09-20T22:55:33.366746+0000","last_undegraded":"2024-09-20T22:55:33.366746+0000","last_fullsized":"2024-09-20T22:55:33.366746+0000","mapping_epoch":836,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":837,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'19","last_scrub_stamp":"2024-09-20T22:53:49.547607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:53:49.547607+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":23,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":560,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-09-20T22:55:33.368940+0000","last_change":"2024-09-20T22:55:22.876777+0000","last_active":"2024-09-20T22:55:33.368940+0000","last_peered":"2024-09-20T22:55:33.368940+0000","last_clean":"2024-09-20T22:55:33.368940+0000","last_became_active":"2024-09-20T22:55:17.342174+0000","last_became_peered":"2024-09-20T22:55:17.342174+0000","last_unstale":"2024-09-20T22:55:33.368940+0000","last_undegraded":"2024-09-20T22:55:33.368940+0000","last_fullsized":"2024-09-20T22:55:33.368940+0000","mapping_epoch":836,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":837,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:55:22.876700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:52:47.717490+0000","last_clean_scrub_stamp":"2024-09-20T22:55:22.876700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:56:49.776969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":171,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-09-20T22:55:33.333789+0000","last_change":"2024-09-20T22:55:03.853796+0000","last_active":"2024-09-20T22:55:33.333789+0000","last_peered":"2024-09-20T22:55:33.333789+0000","last_clean":"2024-09-20T22:55:33.333789+0000","last_became_active":"2024-09-20T22:54:17.336853+0000","last_became_peered":"2024-09-20T22:54:17.336853+0000","last_unstale":"2024-09-20T22:55:33.333789+0000","last_undegraded":"2024-09-20T22:55:33.333789+0000","last_fullsized":"2024-09-20T22:55:33.333789+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:55:03.853725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:55:03.853725+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:56:28.863210+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":1617,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-09-20T22:55:33.344330+0000","last_change":"2024-09-20T22:52:42.582474+0000","last_active":"2024-09-20T22:55:33.344330+0000","last_peered":"2024-09-20T22:55:33.344330+0000","last_clean":"2024-09-20T22:55:33.344330+0000","last_became_active":"2024-09-20T22:47:49.216141+0000","last_became_peered":"2024-09-20T22:47:49.216141+0000","last_unstale":"2024-09-20T22:55:33.344330+0000","last_undegraded":"2024-09-20T22:55:33.344330+0000","last_fullsized":"2024-09-20T22:55:33.344330+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:52:42.582398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:50:20.206984+0000","last_clean_scrub_stamp":"2024-09-20T22:52:42.582398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"625'19","reported_seq":173,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-09-20T22:55:33.334008+0000","last_change":"2024-09-20T22:54:17.360066+0000","last_active":"2024-09-20T22:55:33.334008+0000","last_peered":"2024-09-20T22:55:33.334008+0000","last_clean":"2024-09-20T22:55:33.334008+0000","last_became_active":"2024-09-20T22:54:17.340539+0000","last_became_peered":"2024-09-20T22:54:17.340539+0000","last_unstale":"2024-09-20T22:55:33.334008+0000","last_undegraded":"2024-09-20T22:55:33.334008+0000","last_fullsized":"2024-09-20T22:55:33.334008+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'19","last_scrub_stamp":"2024-09-20T22:53:58.933385+0000","last_deep_scrub":"404'13","last_deep_scrub_stamp":"2024-09-20T22:48:04.276282+0000","last_clean_scrub_stamp":"2024-09-20T22:53:58.933385+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":1657,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-09-20T22:55:33.369984+0000","last_change":"2024-09-20T22:54:46.541182+0000","last_active":"2024-09-20T22:55:33.369984+0000","last_peered":"2024-09-20T22:55:33.369984+0000","last_clean":"2024-09-20T22:55:33.369984+0000","last_became_active":"2024-09-20T22:48:05.238812+0000","last_became_peered":"2024-09-20T22:48:05.238812+0000","last_unstale":"2024-09-20T22:55:33.369984+0000","last_undegraded":"2024-09-20T22:55:33.369984+0000","last_fullsized":"2024-09-20T22:55:33.369984+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:54:46.541109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:31.138067+0000","last_clean_scrub_stamp":"2024-09-20T22:54:46.541109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:58.391916+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":1633,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-09-20T22:55:33.345013+0000","last_change":"2024-09-20T22:54:17.337400+0000","last_active":"2024-09-20T22:55:33.345013+0000","last_peered":"2024-09-20T22:55:33.345013+0000","last_clean":"2024-09-20T22:55:33.345013+0000","last_became_active":"2024-09-20T22:54:17.337102+0000","last_became_peered":"2024-09-20T22:54:17.337102+0000","last_unstale":"2024-09-20T22:55:33.345013+0000","last_undegraded":"2024-09-20T22:55:33.345013+0000","last_fullsized":"2024-09-20T22:55:33.345013+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:34.345077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:53:34.345077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":978,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-09-20T22:55:33.332378+0000","last_change":"2024-09-20T22:55:09.664923+0000","last_active":"2024-09-20T22:55:33.332378+0000","last_peered":"2024-09-20T22:55:33.332378+0000","last_clean":"2024-09-20T22:55:33.332378+0000","last_became_active":"2024-09-20T22:54:17.344021+0000","last_became_peered":"2024-09-20T22:54:17.344021+0000","last_unstale":"2024-09-20T22:55:33.332378+0000","last_undegraded":"2024-09-20T22:55:33.332378+0000","last_fullsized":"2024-09-20T22:55:33.332378+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:55:09.664839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:55:09.664839+0000","last_clean_scrub_stamp":"2024-09-20T22:55:09.664839+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:56:23.744894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"625'24","reported_seq":1447,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-09-20T22:55:33.348392+0000","last_change":"2024-09-20T22:53:33.089688+0000","last_active":"2024-09-20T22:55:33.348392+0000","last_peered":"2024-09-20T22:55:33.348392+0000","last_clean":"2024-09-20T22:55:33.348392+0000","last_became_active":"2024-09-20T22:44:09.170785+0000","last_became_peered":"2024-09-20T22:44:09.170785+0000","last_unstale":"2024-09-20T22:55:33.348392+0000","last_undegraded":"2024-09-20T22:55:33.348392+0000","last_fullsized":"2024-09-20T22:55:33.348392+0000","mapping_epoch":169,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":170,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'24","last_scrub_stamp":"2024-09-20T22:53:33.089580+0000","last_deep_scrub":"470'16","last_deep_scrub_stamp":"2024-09-20T22:49:10.659868+0000","last_clean_scrub_stamp":"2024-09-20T22:53:33.089580+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":1637,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-09-20T22:55:33.349160+0000","last_change":"2024-09-20T22:54:17.341390+0000","last_active":"2024-09-20T22:55:33.349160+0000","last_peered":"2024-09-20T22:55:33.349160+0000","last_clean":"2024-09-20T22:55:33.349160+0000","last_became_active":"2024-09-20T22:54:17.341069+0000","last_became_peered":"2024-09-20T22:54:17.341069+0000","last_unstale":"2024-09-20T22:55:33.349160+0000","last_undegraded":"2024-09-20T22:55:33.349160+0000","last_fullsized":"2024-09-20T22:55:33.349160+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:07.945664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:45:24.765414+0000","last_clean_scrub_stamp":"2024-09-20T22:53:07.945664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":1646,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-09-20T22:55:33.369056+0000","last_change":"2024-09-20T22:55:27.853174+0000","last_active":"2024-09-20T22:55:33.369056+0000","last_peered":"2024-09-20T22:55:33.369056+0000","last_clean":"2024-09-20T22:55:33.369056+0000","last_became_active":"2024-09-20T22:48:05.238334+0000","last_became_peered":"2024-09-20T22:48:05.238334+0000","last_unstale":"2024-09-20T22:55:33.369056+0000","last_undegraded":"2024-09-20T22:55:33.369056+0000","last_fullsized":"2024-09-20T22:55:33.369056+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:55:27.853089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:50.174861+0000","last_clean_scrub_stamp":"2024-09-20T22:55:27.853089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:56:35.115083+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":1617,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-09-20T22:55:33.345165+0000","last_change":"2024-09-20T22:54:22.276389+0000","last_active":"2024-09-20T22:55:33.345165+0000","last_peered":"2024-09-20T22:55:33.345165+0000","last_clean":"2024-09-20T22:55:33.345165+0000","last_became_active":"2024-09-20T22:47:49.214487+0000","last_became_peered":"2024-09-20T22:47:49.214487+0000","last_unstale":"2024-09-20T22:55:33.345165+0000","last_undegraded":"2024-09-20T22:55:33.345165+0000","last_fullsized":"2024-09-20T22:55:33.345165+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:54:22.276321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:54:22.276321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"625'8","reported_seq":1649,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-09-20T22:55:33.344231+0000","last_change":"2024-09-20T22:54:17.373557+0000","last_active":"2024-09-20T22:55:33.344231+0000","last_peered":"2024-09-20T22:55:33.344231+0000","last_clean":"2024-09-20T22:55:33.344231+0000","last_became_active":"2024-09-20T22:54:17.341776+0000","last_became_peered":"2024-09-20T22:54:17.341776+0000","last_unstale":"2024-09-20T22:55:33.344231+0000","last_undegraded":"2024-09-20T22:55:33.344231+0000","last_fullsized":"2024-09-20T22:55:33.344231+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'8","last_scrub_stamp":"2024-09-20T22:53:05.489279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:53:05.489279+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":974,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-09-20T22:55:33.333458+0000","last_change":"2024-09-20T22:55:11.688577+0000","last_active":"2024-09-20T22:55:33.333458+0000","last_peered":"2024-09-20T22:55:33.333458+0000","last_clean":"2024-09-20T22:55:33.333458+0000","last_became_active":"2024-09-20T22:47:49.231964+0000","last_became_peered":"2024-09-20T22:47:49.231964+0000","last_unstale":"2024-09-20T22:55:33.333458+0000","last_undegraded":"2024-09-20T22:55:33.333458+0000","last_fullsized":"2024-09-20T22:55:33.333458+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:55:11.688508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:55:11.688508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:56:35.315737+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":1665,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-09-20T22:55:33.367065+0000","last_change":"2024-09-20T22:55:11.880548+0000","last_active":"2024-09-20T22:55:33.367065+0000","last_peered":"2024-09-20T22:55:33.367065+0000","last_clean":"2024-09-20T22:55:33.367065+0000","last_became_active":"2024-09-20T22:48:05.244659+0000","last_became_peered":"2024-09-20T22:48:05.244659+0000","last_unstale":"2024-09-20T22:55:33.367065+0000","last_undegraded":"2024-09-20T22:55:33.367065+0000","last_fullsized":"2024-09-20T22:55:33.367065+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:55:11.880448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:55:11.880448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:56:22.309018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":1638,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-09-20T22:55:33.369200+0000","last_change":"2024-09-20T22:55:08.681806+0000","last_active":"2024-09-20T22:55:33.369200+0000","last_peered":"2024-09-20T22:55:33.369200+0000","last_clean":"2024-09-20T22:55:33.369200+0000","last_became_active":"2024-09-20T22:48:05.237765+0000","last_became_peered":"2024-09-20T22:48:05.237765+0000","last_unstale":"2024-09-20T22:55:33.369200+0000","last_undegraded":"2024-09-20T22:55:33.369200+0000","last_fullsized":"2024-09-20T22:55:33.369200+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:55:08.681729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:55:08.681729+0000","last_clean_scrub_stamp":"2024-09-20T22:55:08.681729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:56:31.069500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":171,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-09-20T22:55:33.332788+0000","last_change":"2024-09-20T22:55:23.899481+0000","last_active":"2024-09-20T22:55:33.332788+0000","last_peered":"2024-09-20T22:55:33.332788+0000","last_clean":"2024-09-20T22:55:33.332788+0000","last_became_active":"2024-09-20T22:54:17.348206+0000","last_became_peered":"2024-09-20T22:54:17.348206+0000","last_unstale":"2024-09-20T22:55:33.332788+0000","last_undegraded":"2024-09-20T22:55:33.332788+0000","last_fullsized":"2024-09-20T22:55:33.332788+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:55:23.899398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:55:23.899398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:56:29.704828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":1655,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-09-20T22:55:33.343999+0000","last_change":"2024-09-20T22:54:17.342220+0000","last_active":"2024-09-20T22:55:33.343999+0000","last_peered":"2024-09-20T22:55:33.343999+0000","last_clean":"2024-09-20T22:55:33.343999+0000","last_became_active":"2024-09-20T22:54:17.341405+0000","last_became_peered":"2024-09-20T22:54:17.341405+0000","last_unstale":"2024-09-20T22:55:33.343999+0000","last_undegraded":"2024-09-20T22:55:33.343999+0000","last_fullsized":"2024-09-20T22:55:33.343999+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:31.274390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:53:31.274390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"625'19","reported_seq":1716,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-09-20T22:55:33.367873+0000","last_change":"2024-09-20T22:54:50.039084+0000","last_active":"2024-09-20T22:55:33.367873+0000","last_peered":"2024-09-20T22:55:33.367873+0000","last_clean":"2024-09-20T22:55:33.367873+0000","last_became_active":"2024-09-20T22:54:17.342236+0000","last_became_peered":"2024-09-20T22:54:17.342236+0000","last_unstale":"2024-09-20T22:55:33.367873+0000","last_undegraded":"2024-09-20T22:55:33.367873+0000","last_fullsized":"2024-09-20T22:55:33.367873+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'19","last_scrub_stamp":"2024-09-20T22:54:50.039001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:54:50.039001+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:55.885068+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":966,"reported_epoch":853,"state":"active+clean","last_fresh":"2024-09-20T22:55:33.333681+0000","last_change":"2024-09-20T22:54:43.813615+0000","last_active":"2024-09-20T22:55:33.333681+0000","last_peered":"2024-09-20T22:55:33.333681+0000","last_clean":"2024-09-20T22:55:33.333681+0000","last_became_active":"2024-09-20T22:47:49.220971+0000","last_became_peered":"2024-09-20T22:47:49.220971+0000","last_unstale":"2024-09-20T22:55:33.333681+0000","last_undegraded":"2024-09-20T22:55:33.333681+0000","last_fullsized":"2024-09-20T22:55:33.333681+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:54:43.813540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:54:43.813540+0000","last_clean_scrub_stamp":"2024-09-20T22:54:43.813540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:55:55.503018+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":614051322,"num_objects":277,"num_object_clones":227,"num_object_copies":831,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":277,"num_whiteouts":13,"num_read":5918,"num_read_kb":6172349,"num_write":4252,"num_write_kb":1615987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":989,"num_bytes_recovered":730416291,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1402613760,"data_stored":1929777744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":13928,"ondisk_log_size":13928,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":91,"ondisk_log_size":91,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1098,"num_read_kb":855,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":405,"num_bytes_recovered":9390,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":757,"ondisk_log_size":757,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":1126,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":824,"num_read_kb":819,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":1594,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":384,"ondisk_log_size":384,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070635,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":206304,"kb_used_data":148748,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":93538848,"statfs":{"total":95995035648,"available":95783780352,"internally_reserved":0,"allocated":152317952,"data_stored":214464823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":58915274},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430385,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":230404,"kb_used_data":172840,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":93514748,"statfs":{"total":95995035648,"available":95759101952,"internally_reserved":0,"allocated":176988160,"data_stored":234387187,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798692022,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":186728,"kb_used_data":141648,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":93558424,"statfs":{"total":95995035648,"available":95803826176,"internally_reserved":0,"allocated":145047552,"data_stored":196279388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986363,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":163240,"kb_used_data":109848,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93581912,"statfs":{"total":95995035648,"available":95827877888,"internally_reserved":0,"allocated":112484352,"data_stored":150700337,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182593,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":348676,"kb_used_data":281388,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":93396476,"statfs":{"total":95995035648,"available":95637991424,"internally_reserved":0,"allocated":288141312,"data_stored":405215780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":68876743},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411526,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":187748,"kb_used_data":134352,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93557404,"statfs":{"total":95995035648,"available":95802781696,"internally_reserved":0,"allocated":137576448,"data_stored":197480513,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575051,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":262908,"kb_used_data":213668,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":93482244,"statfs":{"total":95995035648,"available":95725817856,"internally_reserved":0,"allocated":218796032,"data_stored":244447276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738576,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298564,"kb_used_data":223016,"kb_used_omap":1,"kb_used_meta":75518,"kb_avail":93446588,"statfs":{"total":95995035648,"available":95689306112,"internally_reserved":0,"allocated":228368384,"data_stored":317509606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77330885},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":220119040,"data_stored":312468368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":213250048,"data_stored":242119747,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":130981888,"data_stored":194867294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":279298048,"data_stored":399583850,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":106950656,"data_stored":148373507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":136200192,"data_stored":190646756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":170414080,"data_stored":230984179,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":145399808,"data_stored":210734043,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-20T22:55:35.810 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd pool set unique_pool_0 pgp_num 16 2024-09-20T22:55:35.848 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:55:35.873 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:35.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:35.874 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:55:35.975 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:55:35.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:35.997 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:35.998 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:55:36.004 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-20T22:55:36.004 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:55:36.086 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-20T22:55:36.098 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:55:36.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:36.152 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:36.152 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:55:36.253 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:55:36.277 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:36.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:36.278 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:55:36.331 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is set 2024-09-20T22:55:36.339 INFO:tasks.rados.rados.0.smithi084.stdout:3415: done (2 left) 2024-09-20T22:55:36.339 INFO:tasks.rados.rados.0.smithi084.stdout:3416: write oid 3 current snap is 399 2024-09-20T22:55:36.339 INFO:tasks.rados.rados.0.smithi084.stdout:3413: finishing write tid 3 to smithi08477203-12 2024-09-20T22:55:36.339 INFO:tasks.rados.rados.0.smithi084.stdout:3413: finishing write tid 4 to smithi08477203-12 2024-09-20T22:55:36.339 INFO:tasks.rados.rados.0.smithi084.stdout:3413: finishing write tid 5 to smithi08477203-12 2024-09-20T22:55:36.339 INFO:tasks.rados.rados.0.smithi084.stdout:3413: oid 12 updating version 0 to 2485 2024-09-20T22:55:36.339 INFO:tasks.rados.rados.0.smithi084.stdout:3413: oid 12 version 2485 is already newer than 2483 2024-09-20T22:55:36.340 INFO:tasks.rados.rados.0.smithi084.stdout:3413: oid 12 version 2485 is already newer than 2484 2024-09-20T22:55:36.340 INFO:tasks.rados.rados.0.smithi084.stdout:3413: oid 12 version 2485 is already newer than 2482 2024-09-20T22:55:36.340 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 12 v 2485 (ObjNum 1162 snap 399 seq_num 1162) dirty exists 2024-09-20T22:55:36.340 INFO:tasks.rados.rados.0.smithi084.stdout:3413: left oid 12 (ObjNum 1162 snap 399 seq_num 1162) 2024-09-20T22:55:36.340 INFO:tasks.rados.rados.0.smithi084.stdout:3414: finishing copy_from racing read to smithi08477203-5 2024-09-20T22:55:36.340 INFO:tasks.rados.rados.0.smithi084.stdout:3414: finishing copy_from to smithi08477203-5 2024-09-20T22:55:36.340 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 5 v 2639 (ObjNum 1111 snap 378 seq_num 1111) dirty exists 2024-09-20T22:55:36.340 INFO:tasks.rados.rados.0.smithi084.stdout:3416: seq_num 1163 ranges {746541=400243,1739617=690403,3138117=408424} 2024-09-20T22:55:36.346 INFO:tasks.rados.rados.0.smithi084.stdout:3416: writing smithi08477203-3 from 746541 to 1146784 tid 1 2024-09-20T22:55:36.354 INFO:tasks.rados.rados.0.smithi084.stdout:3416: writing smithi08477203-3 from 1739617 to 2430020 tid 2 2024-09-20T22:55:36.361 INFO:tasks.rados.rados.0.smithi084.stdout:3416: writing smithi08477203-3 from 3138117 to 3546541 tid 3 2024-09-20T22:55:36.361 INFO:tasks.rados.rados.0.smithi084.stdout:3413: done (2 left) 2024-09-20T22:55:36.362 INFO:tasks.rados.rados.0.smithi084.stdout:3414: done (1 left) 2024-09-20T22:55:36.362 INFO:tasks.rados.rados.0.smithi084.stdout:3417: read oid 23 snap -1 2024-09-20T22:55:36.362 INFO:tasks.rados.rados.0.smithi084.stdout:3417: expect (ObjNum 1159 snap 398 seq_num 1159) 2024-09-20T22:55:36.362 INFO:tasks.rados.rados.0.smithi084.stdout:3418: rollback oid 14 current snap is 399 2024-09-20T22:55:36.362 INFO:tasks.rados.rados.0.smithi084.stdout:3416: finishing write tid 1 to smithi08477203-3 2024-09-20T22:55:36.362 INFO:tasks.rados.rados.0.smithi084.stdout:3416: finishing write tid 2 to smithi08477203-3 2024-09-20T22:55:36.362 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 14 to 383 2024-09-20T22:55:36.362 INFO:tasks.rados.rados.0.smithi084.stdout:3419: rollback oid 44 current snap is 399 2024-09-20T22:55:36.362 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 44 to 332 2024-09-20T22:55:36.362 INFO:tasks.rados.rados.0.smithi084.stdout:3420: read oid 35 snap -1 2024-09-20T22:55:36.362 INFO:tasks.rados.rados.0.smithi084.stdout:3420: expect (ObjNum 1155 snap 396 seq_num 1155) 2024-09-20T22:55:36.362 INFO:tasks.rados.rados.0.smithi084.stdout:3421: read oid 42 snap -1 2024-09-20T22:55:36.362 INFO:tasks.rados.rados.0.smithi084.stdout:3421: expect (ObjNum 1111 snap 378 seq_num 1111) 2024-09-20T22:55:36.363 INFO:tasks.rados.rados.0.smithi084.stdout:3422: rollback oid 2 current snap is 399 2024-09-20T22:55:36.363 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 2 to 376 2024-09-20T22:55:36.363 INFO:tasks.rados.rados.0.smithi084.stdout:3423: snap_create 2024-09-20T22:55:36.366 INFO:tasks.rados.rados.0.smithi084.stdout:3416: finishing write tid 3 to smithi08477203-3 2024-09-20T22:55:36.366 INFO:tasks.rados.rados.0.smithi084.stdout:3418: finishing rollback tid 0 to smithi08477203-14 2024-09-20T22:55:36.366 INFO:tasks.rados.rados.0.smithi084.stdout:3418: finishing rollback tid 1 to smithi08477203-14 2024-09-20T22:55:36.366 INFO:tasks.rados.rados.0.smithi084.stdout:3418: finishing rollback tid 2 to smithi08477203-14 2024-09-20T22:55:36.366 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 14 v 1942 (ObjNum 1116 snap 379 seq_num 1116) dirty exists 2024-09-20T22:55:36.366 INFO:tasks.rados.rados.0.smithi084.stdout:3416: finishing write tid 4 to smithi08477203-3 2024-09-20T22:55:36.366 INFO:tasks.rados.rados.0.smithi084.stdout:3416: finishing write tid 5 to smithi08477203-3 2024-09-20T22:55:36.367 INFO:tasks.rados.rados.0.smithi084.stdout:3416: oid 3 updating version 0 to 2570 2024-09-20T22:55:36.367 INFO:tasks.rados.rados.0.smithi084.stdout:3416: oid 3 version 2570 is already newer than 2569 2024-09-20T22:55:36.367 INFO:tasks.rados.rados.0.smithi084.stdout:3416: oid 3 updating version 2570 to 2571 2024-09-20T22:55:36.367 INFO:tasks.rados.rados.0.smithi084.stdout:3416: oid 3 updating version 2571 to 2572 2024-09-20T22:55:36.367 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 3 v 2572 (ObjNum 1163 snap 399 seq_num 1163) dirty exists 2024-09-20T22:55:36.367 INFO:tasks.rados.rados.0.smithi084.stdout:3416: left oid 3 (ObjNum 1163 snap 399 seq_num 1163) 2024-09-20T22:55:36.370 INFO:tasks.rados.rados.0.smithi084.stdout:3420: expect (ObjNum 1155 snap 396 seq_num 1155) 2024-09-20T22:55:36.379 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:55:36.400 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:36.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:36.401 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:55:36.502 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:55:36.523 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:36.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:36.524 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:55:36.590 INFO:tasks.rados.rados.0.smithi084.stdout:3421: expect (ObjNum 1111 snap 378 seq_num 1111) 2024-09-20T22:55:36.625 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:55:36.645 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:36.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:36.646 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:55:36.677 INFO:tasks.rados.rados.0.smithi084.stdout:3419: finishing rollback tid 0 to smithi08477203-44 2024-09-20T22:55:36.677 INFO:tasks.rados.rados.0.smithi084.stdout:3419: finishing rollback tid 1 to smithi08477203-44 2024-09-20T22:55:36.677 INFO:tasks.rados.rados.0.smithi084.stdout:3419: finishing rollback tid 2 to smithi08477203-44 2024-09-20T22:55:36.677 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 44 v 2724 (ObjNum 879 snap 285 seq_num 879) dirty exists 2024-09-20T22:55:36.677 INFO:tasks.rados.rados.0.smithi084.stdout:3417: expect (ObjNum 1159 snap 398 seq_num 1159) 2024-09-20T22:55:36.747 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:55:36.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:36 smithi154 ceph-mon[33264]: 7.e scrub starts 2024-09-20T22:55:36.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:36 smithi154 ceph-mon[33264]: 7.e scrub ok 2024-09-20T22:55:36.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:36 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/2794726904' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-20T22:55:36.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:36 smithi154 ceph-mon[33264]: 3.6 deep-scrub starts 2024-09-20T22:55:36.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:36 smithi154 ceph-mon[33264]: 3.6 deep-scrub ok 2024-09-20T22:55:36.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:36 smithi154 ceph-mon[33264]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:55:36.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:36 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/2794726904' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T22:55:36.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:36 smithi154 ceph-mon[33264]: osdmap e856: 8 total, 8 up, 8 in 2024-09-20T22:55:36.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:36.769 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:36.769 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:55:36.783 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:36 smithi084 ceph-mon[40024]: 7.e scrub starts 2024-09-20T22:55:36.783 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:36 smithi084 ceph-mon[40024]: 7.e scrub ok 2024-09-20T22:55:36.784 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:36 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/2794726904' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-20T22:55:36.784 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:36 smithi084 ceph-mon[40024]: 3.6 deep-scrub starts 2024-09-20T22:55:36.784 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:36 smithi084 ceph-mon[40024]: 3.6 deep-scrub ok 2024-09-20T22:55:36.784 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:36 smithi084 ceph-mon[40024]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:55:36.784 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:36 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/2794726904' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T22:55:36.784 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:36 smithi084 ceph-mon[40024]: osdmap e856: 8 total, 8 up, 8 in 2024-09-20T22:55:36.784 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:36 smithi084 ceph-mon[29774]: 7.e scrub starts 2024-09-20T22:55:36.784 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:36 smithi084 ceph-mon[29774]: 7.e scrub ok 2024-09-20T22:55:36.784 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:36 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/2794726904' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-20T22:55:36.784 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:36 smithi084 ceph-mon[29774]: 3.6 deep-scrub starts 2024-09-20T22:55:36.785 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:36 smithi084 ceph-mon[29774]: 3.6 deep-scrub ok 2024-09-20T22:55:36.785 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:36 smithi084 ceph-mon[29774]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:55:36.785 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:36 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/2794726904' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T22:55:36.785 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:36 smithi084 ceph-mon[29774]: osdmap e856: 8 total, 8 up, 8 in 2024-09-20T22:55:36.810 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-20T22:55:36.870 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:55:36.875 INFO:tasks.rados.rados.0.smithi084.stdout:3422: finishing rollback tid 1 to smithi08477203-2 2024-09-20T22:55:36.875 INFO:tasks.rados.rados.0.smithi084.stdout:3422: finishing rollback tid 2 to smithi08477203-2 2024-09-20T22:55:36.875 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 2 v 1857 (ObjNum 918 snap 302 seq_num 918) dirty exists 2024-09-20T22:55:36.891 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:36.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:36.892 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:55:36.992 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:55:37.013 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:37.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:37.014 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:55:37.048 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:55:37.115 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:55:37.136 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:37.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:37.137 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:55:37.238 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:55:37.259 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:37.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:37.260 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:55:37.338 INFO:tasks.rados.rados.0.smithi084.stdout:3416: done (7 left) 2024-09-20T22:55:37.338 INFO:tasks.rados.rados.0.smithi084.stdout:3417: done (6 left) 2024-09-20T22:55:37.338 INFO:tasks.rados.rados.0.smithi084.stdout:3418: done (5 left) 2024-09-20T22:55:37.338 INFO:tasks.rados.rados.0.smithi084.stdout:3419: done (4 left) 2024-09-20T22:55:37.338 INFO:tasks.rados.rados.0.smithi084.stdout:3420: done (3 left) 2024-09-20T22:55:37.338 INFO:tasks.rados.rados.0.smithi084.stdout:3421: done (2 left) 2024-09-20T22:55:37.338 INFO:tasks.rados.rados.0.smithi084.stdout:3422: done (1 left) 2024-09-20T22:55:37.338 INFO:tasks.rados.rados.0.smithi084.stdout:3423: done (0 left) 2024-09-20T22:55:37.338 INFO:tasks.rados.rados.0.smithi084.stdout:3424: read oid 49 snap -1 2024-09-20T22:55:37.338 INFO:tasks.rados.rados.0.smithi084.stdout:3424: expect (ObjNum 928 snap 308 seq_num 928) 2024-09-20T22:55:37.338 INFO:tasks.rados.rados.0.smithi084.stdout:3425: write oid 15 current snap is 400 2024-09-20T22:55:37.338 INFO:tasks.rados.rados.0.smithi084.stdout:3425: seq_num 1164 ranges {763348=591845,1914452=434061,2881893=674270} 2024-09-20T22:55:37.346 INFO:tasks.rados.rados.0.smithi084.stdout:3425: writing smithi08477203-15 from 763348 to 1355193 tid 1 2024-09-20T22:55:37.351 INFO:tasks.rados.rados.0.smithi084.stdout:3425: writing smithi08477203-15 from 1914452 to 2348513 tid 2 2024-09-20T22:55:37.358 INFO:tasks.rados.rados.0.smithi084.stdout:3425: writing smithi08477203-15 from 2881893 to 3556163 tid 3 2024-09-20T22:55:37.358 INFO:tasks.rados.rados.0.smithi084.stdout:3426: copy_from oid 23 from oid 44 current snap is 400 2024-09-20T22:55:37.359 INFO:tasks.rados.rados.0.smithi084.stdout:3424: expect (ObjNum 928 snap 308 seq_num 928) 2024-09-20T22:55:37.360 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:55:37.383 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:37.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:37.384 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:55:37.485 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:55:37.506 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:37.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:37.507 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:55:37.608 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:55:37.612 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:55:37.633 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:37.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:37.634 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:55:37.659 INFO:tasks.rados.rados.0.smithi084.stdout:3425: finishing write tid 1 to smithi08477203-15 2024-09-20T22:55:37.659 INFO:tasks.rados.rados.0.smithi084.stdout:3425: finishing write tid 2 to smithi08477203-15 2024-09-20T22:55:37.659 INFO:tasks.rados.rados.0.smithi084.stdout:3424: done (2 left) 2024-09-20T22:55:37.659 INFO:tasks.rados.rados.0.smithi084.stdout:3427: rollback oid 8 current snap is 400 2024-09-20T22:55:37.659 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 8 to 341 2024-09-20T22:55:37.660 INFO:tasks.rados.rados.0.smithi084.stdout:3425: finishing write tid 3 to smithi08477203-15 2024-09-20T22:55:37.660 INFO:tasks.rados.rados.0.smithi084.stdout:3425: finishing write tid 4 to smithi08477203-15 2024-09-20T22:55:37.660 INFO:tasks.rados.rados.0.smithi084.stdout:3425: finishing write tid 5 to smithi08477203-15 2024-09-20T22:55:37.660 INFO:tasks.rados.rados.0.smithi084.stdout:3425: oid 15 updating version 0 to 2576 2024-09-20T22:55:37.660 INFO:tasks.rados.rados.0.smithi084.stdout:3425: oid 15 version 2576 is already newer than 2574 2024-09-20T22:55:37.660 INFO:tasks.rados.rados.0.smithi084.stdout:3425: oid 15 version 2576 is already newer than 2575 2024-09-20T22:55:37.660 INFO:tasks.rados.rados.0.smithi084.stdout:3425: oid 15 version 2576 is already newer than 2573 2024-09-20T22:55:37.660 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 15 v 2576 (ObjNum 1164 snap 400 seq_num 1164) dirty exists 2024-09-20T22:55:37.660 INFO:tasks.rados.rados.0.smithi084.stdout:3425: left oid 15 (ObjNum 1164 snap 400 seq_num 1164) 2024-09-20T22:55:37.660 INFO:tasks.rados.rados.0.smithi084.stdout:3426: finishing copy_from racing read to smithi08477203-23 2024-09-20T22:55:37.660 INFO:tasks.rados.rados.0.smithi084.stdout:3426: finishing copy_from to smithi08477203-23 2024-09-20T22:55:37.660 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 23 v 2724 (ObjNum 879 snap 285 seq_num 879) dirty exists 2024-09-20T22:55:37.660 INFO:tasks.rados.rados.0.smithi084.stdout:3425: done (2 left) 2024-09-20T22:55:37.660 INFO:tasks.rados.rados.0.smithi084.stdout:3426: done (1 left) 2024-09-20T22:55:37.660 INFO:tasks.rados.rados.0.smithi084.stdout:3428: read oid 23 snap 371 2024-09-20T22:55:37.661 INFO:tasks.rados.rados.0.smithi084.stdout:3428: expect deleted 2024-09-20T22:55:37.661 INFO:tasks.rados.rados.0.smithi084.stdout:3429: snap_remove snap 344 2024-09-20T22:55:37.702 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:37 smithi084 ceph-mon[40024]: 1.4 deep-scrub starts 2024-09-20T22:55:37.702 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:37 smithi084 ceph-mon[40024]: 1.4 deep-scrub ok 2024-09-20T22:55:37.702 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:37 smithi084 ceph-mon[40024]: pgmap v1185: 158 pgs: 1 active+clean+snaptrim, 157 active+clean; 586 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 5.6 MiB/s rd, 5.4 MiB/s wr, 22 op/s 2024-09-20T22:55:37.702 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:37 smithi084 ceph-mon[40024]: 5.d scrub starts 2024-09-20T22:55:37.702 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:37 smithi084 ceph-mon[40024]: 5.d scrub ok 2024-09-20T22:55:37.702 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:37 smithi084 ceph-mon[40024]: osdmap e857: 8 total, 8 up, 8 in 2024-09-20T22:55:37.703 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:37 smithi084 ceph-mon[29774]: 1.4 deep-scrub starts 2024-09-20T22:55:37.703 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:37 smithi084 ceph-mon[29774]: 1.4 deep-scrub ok 2024-09-20T22:55:37.703 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:37 smithi084 ceph-mon[29774]: pgmap v1185: 158 pgs: 1 active+clean+snaptrim, 157 active+clean; 586 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 5.6 MiB/s rd, 5.4 MiB/s wr, 22 op/s 2024-09-20T22:55:37.703 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:37 smithi084 ceph-mon[29774]: 5.d scrub starts 2024-09-20T22:55:37.703 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:37 smithi084 ceph-mon[29774]: 5.d scrub ok 2024-09-20T22:55:37.703 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:37 smithi084 ceph-mon[29774]: osdmap e857: 8 total, 8 up, 8 in 2024-09-20T22:55:37.735 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:55:37.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:37 smithi154 ceph-mon[33264]: 1.4 deep-scrub starts 2024-09-20T22:55:37.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:37 smithi154 ceph-mon[33264]: 1.4 deep-scrub ok 2024-09-20T22:55:37.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:37 smithi154 ceph-mon[33264]: pgmap v1185: 158 pgs: 1 active+clean+snaptrim, 157 active+clean; 586 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 5.6 MiB/s rd, 5.4 MiB/s wr, 22 op/s 2024-09-20T22:55:37.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:37 smithi154 ceph-mon[33264]: 5.d scrub starts 2024-09-20T22:55:37.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:37 smithi154 ceph-mon[33264]: 5.d scrub ok 2024-09-20T22:55:37.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:37 smithi154 ceph-mon[33264]: osdmap e857: 8 total, 8 up, 8 in 2024-09-20T22:55:37.756 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:37.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:37.757 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:55:37.858 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:55:37.879 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:37.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:37.880 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:55:37.981 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:55:38.003 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:38.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:38.004 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:55:38.104 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:55:38.126 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:38.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:38.127 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:55:38.227 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:55:38.231 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:55:38.249 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:38.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:38.250 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:55:38.333 INFO:tasks.rados.rados.0.smithi084.stdout:3429: done (2 left) 2024-09-20T22:55:38.333 INFO:tasks.rados.rados.0.smithi084.stdout:3430: copy_from oid 21 from oid 13 current snap is 400 2024-09-20T22:55:38.333 INFO:tasks.rados.rados.0.smithi084.stdout:3427: finishing rollback tid 0 to smithi08477203-8 2024-09-20T22:55:38.333 INFO:tasks.rados.rados.0.smithi084.stdout:3427: finishing rollback tid 1 to smithi08477203-8 2024-09-20T22:55:38.333 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 8 v 2578 (ObjNum 0 snap 0 seq_num 304) dirty dne 2024-09-20T22:55:38.334 INFO:tasks.rados.rados.0.smithi084.stdout:3427: done (2 left) 2024-09-20T22:55:38.334 INFO:tasks.rados.rados.0.smithi084.stdout:3428: done (1 left) 2024-09-20T22:55:38.334 INFO:tasks.rados.rados.0.smithi084.stdout:3431: read oid 28 snap -1 2024-09-20T22:55:38.334 INFO:tasks.rados.rados.0.smithi084.stdout:3431: expect (ObjNum 1160 snap 398 seq_num 1160) 2024-09-20T22:55:38.334 INFO:tasks.rados.rados.0.smithi084.stdout:3432: snap_create 2024-09-20T22:55:38.347 INFO:tasks.rados.rados.0.smithi084.stdout:3431: expect (ObjNum 1160 snap 398 seq_num 1160) 2024-09-20T22:55:38.350 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:55:38.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:38.386 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:38.386 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:55:38.477 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:55:38 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: ::ffff:172.21.15.154 - - [20/Sep/2024:22:55:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-20T22:55:38.477 INFO:tasks.rados.rados.0.smithi084.stdout:3430: finishing copy_from to smithi08477203-21 2024-09-20T22:55:38.477 INFO:tasks.rados.rados.0.smithi084.stdout:3430: got expected ENOENT (src dne) 2024-09-20T22:55:38.477 INFO:tasks.rados.rados.0.smithi084.stdout:3430: finishing copy_from racing read to smithi08477203-21 2024-09-20T22:55:38.486 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:55:38.518 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:38.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:38.519 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:55:38.619 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:55:38.640 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:38.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:38.641 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:55:38.703 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-20T22:55:38.704 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:55:38.742 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:55:38.763 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:38.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:38.765 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:55:38.865 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:55:38.886 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:38.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:38.887 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:55:38.988 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:55:39.009 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:39.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:39.010 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:55:39.110 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:55:39.133 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:39.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:39.134 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:55:39.235 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:55:39.256 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:39.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:39.257 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:55:39.330 INFO:teuthology.orchestra.run.smithi084.stderr:set pool 7 pgp_num to 16 2024-09-20T22:55:39.337 INFO:tasks.rados.rados.0.smithi084.stdout:3430: done (2 left) 2024-09-20T22:55:39.337 INFO:tasks.rados.rados.0.smithi084.stdout:3431: done (1 left) 2024-09-20T22:55:39.337 INFO:tasks.rados.rados.0.smithi084.stdout:3432: done (0 left) 2024-09-20T22:55:39.337 INFO:tasks.rados.rados.0.smithi084.stdout:3433: copy_from oid 38 from oid 16 current snap is 401 2024-09-20T22:55:39.337 INFO:tasks.rados.rados.0.smithi084.stdout:3434: write (excl) oid 49 current snap is 401 2024-09-20T22:55:39.337 INFO:tasks.rados.rados.0.smithi084.stdout:3434: seq_num 1165 ranges {780155=667094,1972934=694072,3309316=270839} 2024-09-20T22:55:39.345 INFO:tasks.rados.rados.0.smithi084.stdout:3434: writing smithi08477203-49 from 780155 to 1447249 tid 1 2024-09-20T22:55:39.354 INFO:tasks.rados.rados.0.smithi084.stdout:3434: writing smithi08477203-49 from 1972934 to 2667006 tid 2 2024-09-20T22:55:39.358 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:55:39.359 INFO:tasks.rados.rados.0.smithi084.stdout:3434: writing smithi08477203-49 from 3309316 to 3580155 tid 3 2024-09-20T22:55:39.359 INFO:tasks.rados.rados.0.smithi084.stdout:3435: delete oid 33 current snap is 401 2024-09-20T22:55:39.359 INFO:tasks.rados.rados.0.smithi084.stdout:3433: finishing copy_from racing read to smithi08477203-38 2024-09-20T22:55:39.359 INFO:tasks.rados.rados.0.smithi084.stdout:3434: finishing write tid 1 to smithi08477203-49 2024-09-20T22:55:39.364 INFO:tasks.rados.rados.0.smithi084.stdout:3434: finishing write tid 2 to smithi08477203-49 2024-09-20T22:55:39.364 INFO:tasks.rados.rados.0.smithi084.stdout:3435: done (2 left) 2024-09-20T22:55:39.364 INFO:tasks.rados.rados.0.smithi084.stdout:3436: rollback oid 43 current snap is 401 2024-09-20T22:55:39.364 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 43 to 362 2024-09-20T22:55:39.364 INFO:tasks.rados.rados.0.smithi084.stdout:3434: finishing write tid 3 to smithi08477203-49 2024-09-20T22:55:39.364 INFO:tasks.rados.rados.0.smithi084.stdout:3437: delete oid 33 current snap is 401 2024-09-20T22:55:39.364 INFO:tasks.rados.rados.0.smithi084.stdout:3434: finishing write tid 4 to smithi08477203-49 2024-09-20T22:55:39.365 INFO:tasks.rados.rados.0.smithi084.stdout:3433: finishing copy_from to smithi08477203-38 2024-09-20T22:55:39.365 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 38 v 1824 (ObjNum 1143 snap 389 seq_num 1143) dirty exists 2024-09-20T22:55:39.365 INFO:tasks.rados.rados.0.smithi084.stdout:3434: finishing write tid 5 to smithi08477203-49 2024-09-20T22:55:39.365 INFO:tasks.rados.rados.0.smithi084.stdout:3434: oid 49 updating version 0 to 1858 2024-09-20T22:55:39.365 INFO:tasks.rados.rados.0.smithi084.stdout:3434: oid 49 updating version 1858 to 1861 2024-09-20T22:55:39.365 INFO:tasks.rados.rados.0.smithi084.stdout:3434: oid 49 version 1861 is already newer than 1859 2024-09-20T22:55:39.365 INFO:tasks.rados.rados.0.smithi084.stdout:3434: oid 49 version 1861 is already newer than 1860 2024-09-20T22:55:39.365 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 49 v 1861 (ObjNum 1165 snap 401 seq_num 1165) dirty exists 2024-09-20T22:55:39.365 INFO:tasks.rados.rados.0.smithi084.stdout:3434: left oid 49 (ObjNum 1165 snap 401 seq_num 1165) 2024-09-20T22:55:39.366 INFO:tasks.rados.rados.0.smithi084.stdout:3433: done (3 left) 2024-09-20T22:55:39.366 INFO:tasks.rados.rados.0.smithi084.stdout:3434: done (2 left) 2024-09-20T22:55:39.366 INFO:tasks.rados.rados.0.smithi084.stdout:3437: done (1 left) 2024-09-20T22:55:39.366 INFO:tasks.rados.rados.0.smithi084.stdout:3438: snap_remove snap 390 2024-09-20T22:55:39.381 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:39.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:39.382 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:55:39.477 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.2 dump_blocked_ops 2024-09-20T22:55:39.483 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:55:39.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:39.542 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:39.542 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:55:39.547 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-20T22:55:39.604 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:39 smithi084 ceph-mon[40024]: pgmap v1188: 158 pgs: 1 active+clean+snaptrim, 157 active+clean; 586 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail 2024-09-20T22:55:39.604 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:39 smithi084 ceph-mon[40024]: osdmap e858: 8 total, 8 up, 8 in 2024-09-20T22:55:39.604 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:39 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:55:39.604 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:39 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/3021357779' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num", "val": "16"}]: dispatch 2024-09-20T22:55:39.605 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:39 smithi084 ceph-mon[29774]: pgmap v1188: 158 pgs: 1 active+clean+snaptrim, 157 active+clean; 586 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail 2024-09-20T22:55:39.605 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:39 smithi084 ceph-mon[29774]: osdmap e858: 8 total, 8 up, 8 in 2024-09-20T22:55:39.605 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:39 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:55:39.605 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:39 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/3021357779' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num", "val": "16"}]: dispatch 2024-09-20T22:55:39.644 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:55:39.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:39.668 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:39.669 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:55:39.680 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:39 smithi154 ceph-mon[33264]: pgmap v1188: 158 pgs: 1 active+clean+snaptrim, 157 active+clean; 586 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail 2024-09-20T22:55:39.680 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:39 smithi154 ceph-mon[33264]: osdmap e858: 8 total, 8 up, 8 in 2024-09-20T22:55:39.680 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:39 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:55:39.680 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:39 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/3021357779' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num", "val": "16"}]: dispatch 2024-09-20T22:55:39.770 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:55:39.792 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:39.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:39.793 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:55:39.894 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:55:39.916 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:39.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:39.917 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:55:40.004 INFO:journalctl@ceph.iscsi.iscsi.a.smithi154.stdout:Sep 20 22:55:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-iscsi-iscsi-a[59730]: debug there is no tcmu-runner data available 2024-09-20T22:55:40.017 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:55:40.041 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:40.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:40.042 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:55:40.143 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:55:40.166 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:40.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:40.168 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:55:40.269 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:55:40.292 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:40.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:40.293 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:55:40.343 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:40 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/3021357779' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num", "val": "16"}]': finished 2024-09-20T22:55:40.343 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:40 smithi084 ceph-mon[29774]: osdmap e859: 8 total, 8 up, 8 in 2024-09-20T22:55:40.347 INFO:tasks.rados.rados.0.smithi084.stdout:3438: done (1 left) 2024-09-20T22:55:40.347 INFO:tasks.rados.rados.0.smithi084.stdout:3439: read oid 22 snap 392 2024-09-20T22:55:40.347 INFO:tasks.rados.rados.0.smithi084.stdout:3439: expect (ObjNum 1112 snap 378 seq_num 1112) 2024-09-20T22:55:40.347 INFO:tasks.rados.rados.0.smithi084.stdout:3440: snap_remove snap 356 2024-09-20T22:55:40.393 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:55:40.417 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:40.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:40.419 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:55:40.520 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:55:40.540 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:40.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:40.541 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:55:40.614 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:40 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/3021357779' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num", "val": "16"}]': finished 2024-09-20T22:55:40.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:40 smithi084 ceph-mon[40024]: osdmap e859: 8 total, 8 up, 8 in 2024-09-20T22:55:40.642 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:55:40.663 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:40.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:40.664 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:55:40.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:40 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/3021357779' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num", "val": "16"}]': finished 2024-09-20T22:55:40.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:40 smithi154 ceph-mon[33264]: osdmap e859: 8 total, 8 up, 8 in 2024-09-20T22:55:40.765 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:55:40.766 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:55:40.766 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:55:40.783 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:55:40.783 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:55:40.788 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:40.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:40.789 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:55:40.800 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:55:40.801 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:55:40.803 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:55:40.817 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-20T22:55:40.817 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T22:55:40.835 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:55:40.835 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:55:40.852 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:55:40.852 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:55:40.868 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:55:40.869 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:55:40.885 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:55:40.886 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:55:40.890 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:55:40.913 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:40.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:40.914 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:55:41.014 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:55:41.036 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:41.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:41.037 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:55:41.138 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:55:41.163 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:41.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:41.164 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:55:41.265 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:55:41.286 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:41.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:41.287 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:55:41.354 INFO:tasks.rados.rados.0.smithi084.stdout:3440: done (2 left) 2024-09-20T22:55:41.354 INFO:tasks.rados.rados.0.smithi084.stdout:3441: snap_create 2024-09-20T22:55:41.355 INFO:tasks.rados.rados.0.smithi084.stdout:3436: finishing rollback tid 0 to smithi08477203-43 2024-09-20T22:55:41.355 INFO:tasks.rados.rados.0.smithi084.stdout:3436: finishing rollback tid 1 to smithi08477203-43 2024-09-20T22:55:41.355 INFO:tasks.rados.rados.0.smithi084.stdout:3436: finishing rollback tid 2 to smithi08477203-43 2024-09-20T22:55:41.355 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 43 v 1937 (ObjNum 1019 snap 333 seq_num 1019) dirty exists 2024-09-20T22:55:41.355 INFO:tasks.rados.rados.0.smithi084.stdout:3439: expect (ObjNum 1112 snap 378 seq_num 1112) 2024-09-20T22:55:41.388 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:55:41.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:41.410 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:41.410 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:55:41.511 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:55:41.536 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:41.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:41.537 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:55:41.614 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:41 smithi084 ceph-mon[29774]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:55:41.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:41 smithi084 ceph-mon[29774]: pgmap v1191: 158 pgs: 2 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 152 active+clean; 609 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 14 MiB/s rd, 3.5 MiB/s wr, 20 op/s 2024-09-20T22:55:41.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:41 smithi084 ceph-mon[29774]: osdmap e860: 8 total, 8 up, 8 in 2024-09-20T22:55:41.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:41 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/323461225' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:55:41.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:41 smithi084 ceph-mon[40024]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:55:41.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:41 smithi084 ceph-mon[40024]: pgmap v1191: 158 pgs: 2 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 152 active+clean; 609 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 14 MiB/s rd, 3.5 MiB/s wr, 20 op/s 2024-09-20T22:55:41.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:41 smithi084 ceph-mon[40024]: osdmap e860: 8 total, 8 up, 8 in 2024-09-20T22:55:41.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:41 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/323461225' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:55:41.638 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:55:41.659 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:41.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:41.660 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:55:41.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:41 smithi154 ceph-mon[33264]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:55:41.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:41 smithi154 ceph-mon[33264]: pgmap v1191: 158 pgs: 2 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 152 active+clean; 609 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 14 MiB/s rd, 3.5 MiB/s wr, 20 op/s 2024-09-20T22:55:41.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:41 smithi154 ceph-mon[33264]: osdmap e860: 8 total, 8 up, 8 in 2024-09-20T22:55:41.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:41 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/323461225' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:55:41.761 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:55:41.782 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:41.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:41.783 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:55:41.883 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:55:41.904 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:41.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:41.906 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:55:42.006 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:55:42.032 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:42.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:42.033 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:55:42.035 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:55:42.134 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:55:42.156 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:42.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:42.157 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:55:42.258 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:55:42.284 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:42.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:42.285 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:55:42.356 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is set 2024-09-20T22:55:42.365 INFO:tasks.rados.rados.0.smithi084.stdout:3436: done (2 left) 2024-09-20T22:55:42.365 INFO:tasks.rados.rados.0.smithi084.stdout:3439: done (1 left) 2024-09-20T22:55:42.366 INFO:tasks.rados.rados.0.smithi084.stdout:3441: done (0 left) 2024-09-20T22:55:42.366 INFO:tasks.rados.rados.0.smithi084.stdout:3442: snap_create 2024-09-20T22:55:42.385 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:55:42.412 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:42.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:42.413 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:55:42.514 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:55:42.534 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-20T22:55:42.535 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2024-09-20T22:55:42.535 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2024-09-20T22:55:42.535 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2024-09-20T22:55:42.535 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-20T22:55:42.538 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:42.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:42.539 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:55:42.543 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:42 smithi084 ceph-mon[40024]: osdmap e861: 8 total, 8 up, 8 in 2024-09-20T22:55:42.543 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:42 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/704664985' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:55:42.543 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:42 smithi084 ceph-mon[29774]: osdmap e861: 8 total, 8 up, 8 in 2024-09-20T22:55:42.543 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:42 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/704664985' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:55:42.640 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:55:42.661 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:42.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:42.662 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:55:42.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:42 smithi154 ceph-mon[33264]: osdmap e861: 8 total, 8 up, 8 in 2024-09-20T22:55:42.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:42 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/704664985' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:55:42.762 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:55:42.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:42.784 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:42.784 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:55:42.885 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:55:42.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:42.907 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:42.907 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:55:43.007 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:55:43.033 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:43.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:43.034 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:55:43.135 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:55:43.156 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:43.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:43.157 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:55:43.192 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.2 dump_historic_ops 2024-09-20T22:55:43.259 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:55:43.296 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:43.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:43.297 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:55:43.343 INFO:tasks.rados.rados.0.smithi084.stdout:3442: done (0 left) 2024-09-20T22:55:43.343 INFO:tasks.rados.rados.0.smithi084.stdout:3443: write (excl) oid 13 current snap is 403 2024-09-20T22:55:43.343 INFO:tasks.rados.rados.0.smithi084.stdout:3443: seq_num 1168 ranges {430576=492841,1632027=441399,2475232=435711,3365397=265179} 2024-09-20T22:55:43.348 INFO:tasks.rados.rados.0.smithi084.stdout:3443: writing smithi08477203-13 from 430576 to 923417 tid 1 2024-09-20T22:55:43.354 INFO:tasks.rados.rados.0.smithi084.stdout:3443: writing smithi08477203-13 from 1632027 to 2073426 tid 2 2024-09-20T22:55:43.359 INFO:tasks.rados.rados.0.smithi084.stdout:3443: writing smithi08477203-13 from 2475232 to 2910943 tid 3 2024-09-20T22:55:43.364 INFO:tasks.rados.rados.0.smithi084.stdout:3443: writing smithi08477203-13 from 3365397 to 3630576 tid 4 2024-09-20T22:55:43.364 INFO:tasks.rados.rados.0.smithi084.stdout:3444: write oid 29 current snap is 403 2024-09-20T22:55:43.364 INFO:tasks.rados.rados.0.smithi084.stdout:3443: finishing write tid 1 to smithi08477203-13 2024-09-20T22:55:43.364 INFO:tasks.rados.rados.0.smithi084.stdout:3444: seq_num 1169 ranges {447383=568090,1690509=701410,2902655=646983} 2024-09-20T22:55:43.369 INFO:tasks.rados.rados.0.smithi084.stdout:3444: writing smithi08477203-29 from 447383 to 1015473 tid 1 2024-09-20T22:55:43.376 INFO:tasks.rados.rados.0.smithi084.stdout:3444: writing smithi08477203-29 from 1690509 to 2391919 tid 2 2024-09-20T22:55:43.383 INFO:tasks.rados.rados.0.smithi084.stdout:3444: writing smithi08477203-29 from 2902655 to 3549638 tid 3 2024-09-20T22:55:43.383 INFO:tasks.rados.rados.0.smithi084.stdout:3445: read oid 46 snap -1 2024-09-20T22:55:43.383 INFO:tasks.rados.rados.0.smithi084.stdout:3445: expect (ObjNum 1158 snap 397 seq_num 1158) 2024-09-20T22:55:43.383 INFO:tasks.rados.rados.0.smithi084.stdout:3446: snap_remove snap 341 2024-09-20T22:55:43.399 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:55:43.420 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:43.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:43.421 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:55:43.522 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:55:43.528 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:43 smithi084 ceph-mon[40024]: pgmap v1194: 158 pgs: 2 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 152 active+clean; 609 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 14 MiB/s rd, 3.5 MiB/s wr, 20 op/s 2024-09-20T22:55:43.528 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:43 smithi084 ceph-mon[40024]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:55:43.528 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:43 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/704664985' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T22:55:43.528 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:43 smithi084 ceph-mon[40024]: osdmap e862: 8 total, 8 up, 8 in 2024-09-20T22:55:43.528 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:43 smithi084 ceph-mon[40024]: osdmap e863: 8 total, 8 up, 8 in 2024-09-20T22:55:43.528 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:43 smithi084 ceph-mon[29774]: pgmap v1194: 158 pgs: 2 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 152 active+clean; 609 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 14 MiB/s rd, 3.5 MiB/s wr, 20 op/s 2024-09-20T22:55:43.528 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:43 smithi084 ceph-mon[29774]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:55:43.529 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:43 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/704664985' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T22:55:43.529 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:43 smithi084 ceph-mon[29774]: osdmap e862: 8 total, 8 up, 8 in 2024-09-20T22:55:43.529 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:43 smithi084 ceph-mon[29774]: osdmap e863: 8 total, 8 up, 8 in 2024-09-20T22:55:43.545 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:43.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:43.546 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:55:43.647 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:55:43.668 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:43.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:43.669 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:55:43.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:43 smithi154 ceph-mon[33264]: pgmap v1194: 158 pgs: 2 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 152 active+clean; 609 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 14 MiB/s rd, 3.5 MiB/s wr, 20 op/s 2024-09-20T22:55:43.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:43 smithi154 ceph-mon[33264]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:55:43.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:43 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/704664985' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T22:55:43.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:43 smithi154 ceph-mon[33264]: osdmap e862: 8 total, 8 up, 8 in 2024-09-20T22:55:43.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:43 smithi154 ceph-mon[33264]: osdmap e863: 8 total, 8 up, 8 in 2024-09-20T22:55:43.769 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:55:43.790 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:43.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:43.791 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:55:43.892 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:55:43.918 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:43.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:43.919 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:55:44.000 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-20T22:55:44.020 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:55:44.026 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:55:44.048 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:44.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:44.049 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:55:44.149 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:55:44.175 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:44.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:44.176 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:55:44.276 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:55:44.298 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:44.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:44.299 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:55:44.365 INFO:tasks.rados.rados.0.smithi084.stdout:3446: done (3 left) 2024-09-20T22:55:44.365 INFO:tasks.rados.rados.0.smithi084.stdout:3447: snap_remove snap 393 2024-09-20T22:55:44.400 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:55:44.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:44.423 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:44.424 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:55:44.524 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:55:44.549 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:44.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:44.550 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:55:44.565 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-20T22:55:44.565 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2024-09-20T22:55:44.565 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2024-09-20T22:55:44.565 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [ 2024-09-20T22:55:44.565 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.565 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(osd.7.0:152 7.5 e269/260 7:a160b5d4:::smithi08477203-16:55 v 269'55, mlcod=263'51)", 2024-09-20T22:55:44.565 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:45:48.173893+0000", 2024-09-20T22:55:44.565 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 596.38775663900003, 2024-09-20T22:55:44.565 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014936935, 2024-09-20T22:55:44.566 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:55:44.566 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:55:44.566 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:55:44.566 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:55:44.566 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.566 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:55:44.566 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.173893+0000", 2024-09-20T22:55:44.566 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:55:44.566 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.566 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.566 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:55:44.566 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.173893+0000", 2024-09-20T22:55:44.566 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:55:44.567 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.567 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.567 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:55:44.567 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.173893+0000", 2024-09-20T22:55:44.567 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.8800000000000002e-07 2024-09-20T22:55:44.567 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.567 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.567 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:55:44.567 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.173916+0000", 2024-09-20T22:55:44.567 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.2889999999999999e-05 2024-09-20T22:55:44.567 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.567 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.567 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:55:44.567 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.173924+0000", 2024-09-20T22:55:44.568 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.4420000000000001e-06 2024-09-20T22:55:44.568 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.568 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.568 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:55:44.568 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.173932+0000", 2024-09-20T22:55:44.568 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.5299999999999996e-06 2024-09-20T22:55:44.568 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.568 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.568 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:55:44.568 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.174011+0000", 2024-09-20T22:55:44.568 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.9342e-05 2024-09-20T22:55:44.568 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.568 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.568 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:55:44.569 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.174047+0000", 2024-09-20T22:55:44.569 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.5237999999999997e-05 2024-09-20T22:55:44.569 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.569 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.569 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:55:44.569 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.188782+0000", 2024-09-20T22:55:44.569 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014735599 2024-09-20T22:55:44.569 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.569 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.569 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:55:44.569 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.188830+0000", 2024-09-20T22:55:44.569 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.7206000000000002e-05 2024-09-20T22:55:44.569 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:55:44.569 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:55:44.570 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:55:44.570 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.570 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.570 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(osd.6.0:767 7.12 e269/263 7:4fa4b2b3:::smithi08477203-21:53 v 269'262, mlcod=265'259)", 2024-09-20T22:55:44.570 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:45:48.175130+0000", 2024-09-20T22:55:44.570 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 596.38651896900001, 2024-09-20T22:55:44.570 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013842727000000001, 2024-09-20T22:55:44.570 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:55:44.570 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:55:44.570 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:55:44.570 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:55:44.570 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.570 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:55:44.570 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.175130+0000", 2024-09-20T22:55:44.571 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:55:44.571 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.571 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.571 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:55:44.571 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.175130+0000", 2024-09-20T22:55:44.571 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:55:44.571 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.571 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.571 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:55:44.571 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.175130+0000", 2024-09-20T22:55:44.571 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.98e-07 2024-09-20T22:55:44.571 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.571 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.571 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:55:44.572 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.175139+0000", 2024-09-20T22:55:44.572 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.5909999999999996e-06 2024-09-20T22:55:44.572 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.572 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.572 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:55:44.572 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.175144+0000", 2024-09-20T22:55:44.572 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.361e-06 2024-09-20T22:55:44.572 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.572 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.572 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:55:44.572 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.175156+0000", 2024-09-20T22:55:44.572 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1236999999999999e-05 2024-09-20T22:55:44.572 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.572 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.573 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:55:44.573 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.176359+0000", 2024-09-20T22:55:44.573 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0012036060000000001 2024-09-20T22:55:44.573 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.573 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.573 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:55:44.573 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.176380+0000", 2024-09-20T22:55:44.573 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.0497999999999999e-05 2024-09-20T22:55:44.573 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.573 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.573 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:55:44.573 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.188936+0000", 2024-09-20T22:55:44.573 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.012556286999999999 2024-09-20T22:55:44.573 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.574 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.574 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:55:44.574 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:48.188973+0000", 2024-09-20T22:55:44.574 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.6949000000000001e-05 2024-09-20T22:55:44.574 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:55:44.574 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:55:44.574 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:55:44.574 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.574 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.574 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:3030 7.e 7:71b5c915:::smithi08477203-6:head [copy-from ver 216] snapc 72=[72,71,70,6e,6c,6b,6a,68] ondisk+write+known_if_redirected+supports_pool_eio e317)", 2024-09-20T22:55:44.574 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:46:37.293029+0000", 2024-09-20T22:55:44.574 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 547.26861989700001, 2024-09-20T22:55:44.574 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.019929916999999998, 2024-09-20T22:55:44.574 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:55:44.575 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:55:44.575 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:55:44.575 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:55:44.575 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:55:44.575 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:55:44.575 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 3030 2024-09-20T22:55:44.575 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.575 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:55:44.575 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.575 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:55:44.575 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.293029+0000", 2024-09-20T22:55:44.575 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:55:44.575 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.575 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.576 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:55:44.576 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.293029+0000", 2024-09-20T22:55:44.576 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:55:44.576 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.576 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.576 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:55:44.576 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.293031+0000", 2024-09-20T22:55:44.576 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.4190000000000001e-06 2024-09-20T22:55:44.576 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.576 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.576 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:55:44.576 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.293040+0000", 2024-09-20T22:55:44.576 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.6039999999999999e-06 2024-09-20T22:55:44.576 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.577 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.577 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:55:44.577 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.293054+0000", 2024-09-20T22:55:44.577 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.4014e-05 2024-09-20T22:55:44.577 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.577 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.577 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:55:44.577 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.293062+0000", 2024-09-20T22:55:44.577 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.8820000000000001e-06 2024-09-20T22:55:44.577 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.577 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.577 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:55:44.577 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.293121+0000", 2024-09-20T22:55:44.577 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.8977999999999997e-05 2024-09-20T22:55:44.577 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.577 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.578 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op must wait for map", 2024-09-20T22:55:44.578 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.293125+0000", 2024-09-20T22:55:44.578 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.2089999999999999e-06 2024-09-20T22:55:44.578 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.578 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.578 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:55:44.578 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.298805+0000", 2024-09-20T22:55:44.578 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0056790870000000002 2024-09-20T22:55:44.578 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.578 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.578 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:55:44.578 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.298886+0000", 2024-09-20T22:55:44.578 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.1363e-05 2024-09-20T22:55:44.578 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.579 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.579 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 3,5", 2024-09-20T22:55:44.579 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.305883+0000", 2024-09-20T22:55:44.579 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0069966159999999998 2024-09-20T22:55:44.579 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.579 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.579 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:55:44.579 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.312556+0000", 2024-09-20T22:55:44.579 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0066737009999999998 2024-09-20T22:55:44.579 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.579 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.579 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:55:44.579 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.312587+0000", 2024-09-20T22:55:44.580 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.0286000000000001e-05 2024-09-20T22:55:44.580 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.580 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.580 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:55:44.580 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.312890+0000", 2024-09-20T22:55:44.580 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00030374800000000001 2024-09-20T22:55:44.580 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.580 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.580 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:55:44.580 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.312932+0000", 2024-09-20T22:55:44.580 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.1621e-05 2024-09-20T22:55:44.580 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.580 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.580 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:55:44.581 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.312959+0000", 2024-09-20T22:55:44.581 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.7389000000000002e-05 2024-09-20T22:55:44.581 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:55:44.581 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:55:44.581 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:55:44.581 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.581 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.581 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:3031 7.e 7:71b5c915:::smithi08477203-6:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e317)", 2024-09-20T22:55:44.581 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:46:37.293082+0000", 2024-09-20T22:55:44.581 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 547.26856751900004, 2024-09-20T22:55:44.581 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.016134757, 2024-09-20T22:55:44.581 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:55:44.581 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:55:44.581 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-20T22:55:44.582 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:55:44.582 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:55:44.582 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:55:44.582 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 3031 2024-09-20T22:55:44.582 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.582 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:55:44.582 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.582 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:55:44.582 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.293082+0000", 2024-09-20T22:55:44.582 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:55:44.582 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.582 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.582 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:55:44.583 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.293082+0000", 2024-09-20T22:55:44.583 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:55:44.583 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.583 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.583 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:55:44.583 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.293082+0000", 2024-09-20T22:55:44.583 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.1799999999999995e-07 2024-09-20T22:55:44.583 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.583 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.583 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:55:44.584 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.293085+0000", 2024-09-20T22:55:44.584 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.937e-06 2024-09-20T22:55:44.584 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.584 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.584 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:55:44.584 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.293093+0000", 2024-09-20T22:55:44.584 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.554e-06 2024-09-20T22:55:44.584 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.584 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.584 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:55:44.584 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.293098+0000", 2024-09-20T22:55:44.584 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.643e-06 2024-09-20T22:55:44.584 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.584 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.585 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:55:44.585 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.293164+0000", 2024-09-20T22:55:44.585 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.5806e-05 2024-09-20T22:55:44.585 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.585 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.585 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting_for_map not empty", 2024-09-20T22:55:44.585 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.293168+0000", 2024-09-20T22:55:44.585 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.2459999999999999e-06 2024-09-20T22:55:44.585 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.585 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.585 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:55:44.585 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.299033+0000", 2024-09-20T22:55:44.585 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0058656699999999999 2024-09-20T22:55:44.585 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.586 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.586 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for blocked object", 2024-09-20T22:55:44.586 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.299069+0000", 2024-09-20T22:55:44.586 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.5392999999999998e-05 2024-09-20T22:55:44.586 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.586 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.586 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:55:44.586 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.309088+0000", 2024-09-20T22:55:44.586 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010019287999999999 2024-09-20T22:55:44.586 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.586 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.586 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:55:44.586 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.309150+0000", 2024-09-20T22:55:44.587 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.2008000000000007e-05 2024-09-20T22:55:44.587 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.587 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.587 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:55:44.587 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:37.309216+0000", 2024-09-20T22:55:44.587 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.6594000000000006e-05 2024-09-20T22:55:44.587 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:55:44.587 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:55:44.587 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:55:44.587 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.587 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.587 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:3290 7.8 7:12e9c8db:::smithi08477203-11:head [copy-from ver 675] snapc 7f=[7f,7e,7d,7c,7b,7a,79,78,74,73,6b] ondisk+write+known_if_redirected+supports_pool_eio e340)", 2024-09-20T22:55:44.587 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:47:00.706846+0000", 2024-09-20T22:55:44.587 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 523.85480366000002, 2024-09-20T22:55:44.588 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.018408746, 2024-09-20T22:55:44.588 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:55:44.588 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:55:44.588 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:55:44.588 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:55:44.588 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:55:44.588 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:55:44.588 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 3290 2024-09-20T22:55:44.588 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.588 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:55:44.588 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.588 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:55:44.588 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.706846+0000", 2024-09-20T22:55:44.588 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:55:44.589 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.589 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.589 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:55:44.589 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.706846+0000", 2024-09-20T22:55:44.589 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:55:44.589 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.589 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.589 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:55:44.589 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.706848+0000", 2024-09-20T22:55:44.589 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.2340000000000001e-06 2024-09-20T22:55:44.589 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.589 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.589 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:55:44.589 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.706859+0000", 2024-09-20T22:55:44.590 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1001e-05 2024-09-20T22:55:44.590 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.590 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.590 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:55:44.590 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.706879+0000", 2024-09-20T22:55:44.590 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.0556000000000001e-05 2024-09-20T22:55:44.590 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.590 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.590 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:55:44.590 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.706890+0000", 2024-09-20T22:55:44.591 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0328e-05 2024-09-20T22:55:44.591 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.591 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.591 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:55:44.591 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.706973+0000", 2024-09-20T22:55:44.591 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.3574999999999999e-05 2024-09-20T22:55:44.591 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.591 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.591 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:55:44.591 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.707042+0000", 2024-09-20T22:55:44.591 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.8987999999999997e-05 2024-09-20T22:55:44.591 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.591 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.592 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 3,7", 2024-09-20T22:55:44.592 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.717491+0000", 2024-09-20T22:55:44.592 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010448618 2024-09-20T22:55:44.592 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.592 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.592 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:55:44.592 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.722875+0000", 2024-09-20T22:55:44.592 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0053846190000000002 2024-09-20T22:55:44.592 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.592 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.593 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:55:44.593 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.724841+0000", 2024-09-20T22:55:44.593 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0019651579999999998 2024-09-20T22:55:44.593 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.593 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.593 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:55:44.593 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.725168+0000", 2024-09-20T22:55:44.593 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00032734499999999999 2024-09-20T22:55:44.593 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.593 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.593 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:55:44.594 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.725218+0000", 2024-09-20T22:55:44.594 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.0123000000000001e-05 2024-09-20T22:55:44.594 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.594 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.595 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:55:44.595 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.725254+0000", 2024-09-20T22:55:44.595 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.6201000000000001e-05 2024-09-20T22:55:44.595 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:55:44.595 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:55:44.595 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:55:44.596 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.596 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.596 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:3291 7.8 7:12e9c8db:::smithi08477203-11:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e340)", 2024-09-20T22:55:44.596 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:47:00.706927+0000", 2024-09-20T22:55:44.596 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 523.85472202100004, 2024-09-20T22:55:44.596 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014159255000000001, 2024-09-20T22:55:44.596 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:55:44.596 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:55:44.596 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-20T22:55:44.597 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:55:44.597 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:55:44.597 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:55:44.597 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 3291 2024-09-20T22:55:44.597 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.597 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:55:44.597 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.597 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:55:44.597 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.706927+0000", 2024-09-20T22:55:44.598 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:55:44.598 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.598 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.598 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:55:44.598 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.706927+0000", 2024-09-20T22:55:44.598 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:55:44.598 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.598 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.598 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:55:44.598 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.706928+0000", 2024-09-20T22:55:44.598 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.4099999999999999e-07 2024-09-20T22:55:44.598 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.599 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.599 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:55:44.599 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.706941+0000", 2024-09-20T22:55:44.599 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3335e-05 2024-09-20T22:55:44.599 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.599 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.599 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:55:44.599 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.706948+0000", 2024-09-20T22:55:44.599 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.3350000000000001e-06 2024-09-20T22:55:44.599 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.599 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.599 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:55:44.599 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.706954+0000", 2024-09-20T22:55:44.599 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.9190000000000004e-06 2024-09-20T22:55:44.600 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.600 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.600 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:55:44.600 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.707166+0000", 2024-09-20T22:55:44.600 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00021159 2024-09-20T22:55:44.600 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.600 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.600 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for blocked object", 2024-09-20T22:55:44.600 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.707217+0000", 2024-09-20T22:55:44.600 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.1167e-05 2024-09-20T22:55:44.600 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.600 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.600 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:55:44.600 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.720939+0000", 2024-09-20T22:55:44.601 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013721458000000001 2024-09-20T22:55:44.601 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.601 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.601 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:55:44.601 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.721010+0000", 2024-09-20T22:55:44.601 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.1570999999999996e-05 2024-09-20T22:55:44.601 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.601 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.601 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:55:44.601 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:47:00.721086+0000", 2024-09-20T22:55:44.601 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.6339000000000002e-05 2024-09-20T22:55:44.601 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:55:44.601 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:55:44.602 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:55:44.602 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.602 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.602 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24652.0:6378 7.2 e639/600 7:4d20232a:::smithi08477203-34:head v 639'665, mlcod=629'662)", 2024-09-20T22:55:44.602 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:51:59.278714+0000", 2024-09-20T22:55:44.602 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 225.28293536499999, 2024-09-20T22:55:44.602 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.015218334, 2024-09-20T22:55:44.602 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:55:44.602 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:55:44.602 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:55:44.602 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:55:44.602 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.602 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:55:44.602 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:59.278714+0000", 2024-09-20T22:55:44.603 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:55:44.603 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.603 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.603 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:55:44.603 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:59.278714+0000", 2024-09-20T22:55:44.603 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:55:44.603 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.603 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.603 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:55:44.603 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:59.278714+0000", 2024-09-20T22:55:44.603 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.6e-07 2024-09-20T22:55:44.603 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.603 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.603 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:55:44.603 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:59.281091+0000", 2024-09-20T22:55:44.604 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0023769239999999999 2024-09-20T22:55:44.604 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.604 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.604 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:55:44.604 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:59.281110+0000", 2024-09-20T22:55:44.604 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.9230000000000001e-05 2024-09-20T22:55:44.604 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.604 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.604 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:55:44.604 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:59.281122+0000", 2024-09-20T22:55:44.604 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1997999999999999e-05 2024-09-20T22:55:44.604 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.604 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.604 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:55:44.605 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:59.281186+0000", 2024-09-20T22:55:44.605 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.3998999999999995e-05 2024-09-20T22:55:44.605 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.605 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.605 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:55:44.605 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:59.281209+0000", 2024-09-20T22:55:44.605 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.2654999999999999e-05 2024-09-20T22:55:44.605 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.605 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.605 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:55:44.605 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:59.293866+0000", 2024-09-20T22:55:44.605 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.012656745 2024-09-20T22:55:44.605 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.605 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.605 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:55:44.606 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:59.293932+0000", 2024-09-20T22:55:44.606 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.6523e-05 2024-09-20T22:55:44.606 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:55:44.606 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:55:44.606 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:55:44.606 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.606 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.606 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(osd.5.0:520 7.b e643/602 7:dc95255e:::smithi08477203-1:107 v 643'959, mlcod=637'955)", 2024-09-20T22:55:44.606 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:52:03.289043+0000", 2024-09-20T22:55:44.606 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 221.27260625599999, 2024-09-20T22:55:44.606 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.015367396, 2024-09-20T22:55:44.606 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:55:44.606 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:55:44.606 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:55:44.607 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:55:44.607 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.607 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:55:44.607 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:52:03.289043+0000", 2024-09-20T22:55:44.607 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:55:44.607 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.607 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.607 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:55:44.607 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:52:03.289043+0000", 2024-09-20T22:55:44.607 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:55:44.607 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.607 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.607 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:55:44.607 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:52:03.289044+0000", 2024-09-20T22:55:44.608 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.8100000000000002e-07 2024-09-20T22:55:44.608 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.608 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.608 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:55:44.608 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:52:03.289056+0000", 2024-09-20T22:55:44.608 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2408999999999999e-05 2024-09-20T22:55:44.608 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.608 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.608 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:55:44.608 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:52:03.289071+0000", 2024-09-20T22:55:44.608 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.4331000000000001e-05 2024-09-20T22:55:44.608 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.608 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.608 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:55:44.609 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:52:03.289079+0000", 2024-09-20T22:55:44.609 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.1009999999999992e-06 2024-09-20T22:55:44.609 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.609 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.609 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:55:44.609 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:52:03.289168+0000", 2024-09-20T22:55:44.609 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.9614e-05 2024-09-20T22:55:44.609 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.609 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.609 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:55:44.609 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:52:03.289189+0000", 2024-09-20T22:55:44.609 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.0701999999999999e-05 2024-09-20T22:55:44.609 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.609 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.610 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:55:44.610 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:52:03.304364+0000", 2024-09-20T22:55:44.610 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.015175414 2024-09-20T22:55:44.610 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.610 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.610 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:55:44.610 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:52:03.304410+0000", 2024-09-20T22:55:44.610 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.6044000000000002e-05 2024-09-20T22:55:44.610 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:55:44.610 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:55:44.610 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:55:44.610 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.610 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.610 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(osd.5.0:521 7.b e643/602 7:d960cad4:::smithi08477203-18:107 v 643'961, mlcod=637'955)", 2024-09-20T22:55:44.610 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:52:03.289097+0000", 2024-09-20T22:55:44.611 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 221.272552609, 2024-09-20T22:55:44.611 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.016179973, 2024-09-20T22:55:44.611 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:55:44.611 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:55:44.611 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:55:44.611 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:55:44.611 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.611 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:55:44.611 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:52:03.289097+0000", 2024-09-20T22:55:44.611 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:55:44.611 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.611 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.611 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:55:44.611 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:52:03.289097+0000", 2024-09-20T22:55:44.612 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:55:44.612 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.612 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.612 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:55:44.612 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:52:03.289097+0000", 2024-09-20T22:55:44.612 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.3599999999999999e-07 2024-09-20T22:55:44.612 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.612 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.612 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:55:44.612 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:52:03.289105+0000", 2024-09-20T22:55:44.612 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.8869999999999992e-06 2024-09-20T22:55:44.612 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.612 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.612 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:55:44.612 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:52:03.289111+0000", 2024-09-20T22:55:44.613 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.2419999999999997e-06 2024-09-20T22:55:44.613 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.613 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.613 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:55:44.613 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:52:03.289116+0000", 2024-09-20T22:55:44.613 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.6569999999999996e-06 2024-09-20T22:55:44.613 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.613 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.613 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:55:44.613 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:52:03.291170+0000", 2024-09-20T22:55:44.613 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0020537989999999998 2024-09-20T22:55:44.613 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.613 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.613 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:55:44.613 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:52:03.291192+0000", 2024-09-20T22:55:44.614 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.2548000000000002e-05 2024-09-20T22:55:44.614 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.614 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.614 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:55:44.614 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:52:03.305241+0000", 2024-09-20T22:55:44.614 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.01404832 2024-09-20T22:55:44.614 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.614 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.614 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:55:44.614 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:52:03.305277+0000", 2024-09-20T22:55:44.614 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.6084000000000002e-05 2024-09-20T22:55:44.614 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:55:44.614 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:55:44.614 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:55:44.615 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.615 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.615 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24652.0:6790 7.0 e687/632 7:064f66ab:::smithi08477203-45:head v 687'535, mlcod=687'532)", 2024-09-20T22:55:44.615 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:52:47.711923+0000", 2024-09-20T22:55:44.615 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 176.84972579500001, 2024-09-20T22:55:44.615 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014321054999999999, 2024-09-20T22:55:44.615 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:55:44.615 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:55:44.615 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:55:44.615 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:55:44.615 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.615 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:55:44.615 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:52:47.711923+0000", 2024-09-20T22:55:44.615 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:55:44.615 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.616 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.616 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:55:44.616 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:52:47.711923+0000", 2024-09-20T22:55:44.616 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:55:44.616 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.616 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.616 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:55:44.616 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:52:47.711924+0000", 2024-09-20T22:55:44.616 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.33e-07 2024-09-20T22:55:44.616 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.616 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.616 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:55:44.616 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:52:47.720461+0000", 2024-09-20T22:55:44.616 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0085366850000000005 2024-09-20T22:55:44.616 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.617 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.617 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:55:44.617 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:52:47.720487+0000", 2024-09-20T22:55:44.617 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.5463e-05 2024-09-20T22:55:44.617 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.617 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.617 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:55:44.617 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:52:47.720540+0000", 2024-09-20T22:55:44.617 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.3356999999999998e-05 2024-09-20T22:55:44.617 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.617 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.617 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:55:44.617 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:52:47.720631+0000", 2024-09-20T22:55:44.617 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.1305999999999996e-05 2024-09-20T22:55:44.617 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.618 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.618 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:55:44.618 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:52:47.720662+0000", 2024-09-20T22:55:44.618 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.1149000000000003e-05 2024-09-20T22:55:44.618 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.618 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.618 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:55:44.618 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:52:47.726196+0000", 2024-09-20T22:55:44.619 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0055332430000000002 2024-09-20T22:55:44.619 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.619 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.619 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:55:44.619 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:52:47.726244+0000", 2024-09-20T22:55:44.619 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.8918999999999997e-05 2024-09-20T22:55:44.619 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:55:44.619 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:55:44.619 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:55:44.619 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.619 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.619 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24652.0:6829 7.5 e688/456 7:a160b5d4:::smithi08477203-16:head v 688'328, mlcod=684'324)", 2024-09-20T22:55:44.619 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:52:48.309561+0000", 2024-09-20T22:55:44.619 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 176.25208800999999, 2024-09-20T22:55:44.620 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.017015579999999999, 2024-09-20T22:55:44.620 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:55:44.620 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:55:44.620 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:55:44.620 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:55:44.620 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.620 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:55:44.620 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:52:48.309561+0000", 2024-09-20T22:55:44.620 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:55:44.620 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.621 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.621 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:55:44.621 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:52:48.309561+0000", 2024-09-20T22:55:44.621 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:55:44.621 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.621 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.621 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:55:44.621 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:52:48.309562+0000", 2024-09-20T22:55:44.621 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.7599999999999996e-07 2024-09-20T22:55:44.621 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.621 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.621 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:55:44.621 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:52:48.310336+0000", 2024-09-20T22:55:44.621 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00077435700000000004 2024-09-20T22:55:44.622 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.622 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.622 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:55:44.622 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:52:48.310355+0000", 2024-09-20T22:55:44.622 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.8763e-05 2024-09-20T22:55:44.622 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.623 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.623 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:55:44.623 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:52:48.310365+0000", 2024-09-20T22:55:44.623 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.4420000000000004e-06 2024-09-20T22:55:44.623 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.623 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.623 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:55:44.623 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:52:48.310463+0000", 2024-09-20T22:55:44.623 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.8177999999999994e-05 2024-09-20T22:55:44.623 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.623 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.623 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:55:44.623 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:52:48.310494+0000", 2024-09-20T22:55:44.624 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.1284000000000001e-05 2024-09-20T22:55:44.624 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.624 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.624 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:55:44.624 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:52:48.326521+0000", 2024-09-20T22:55:44.624 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.016026757999999999 2024-09-20T22:55:44.624 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.624 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.624 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:55:44.624 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:52:48.326577+0000", 2024-09-20T22:55:44.625 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.5921999999999998e-05 2024-09-20T22:55:44.625 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:55:44.625 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:55:44.625 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:55:44.625 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.625 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.625 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24652.0:7143 7.0 e719/632 7:0f524219:::smithi08477203-50:head v 719'618, mlcod=714'614)", 2024-09-20T22:55:44.625 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:53:19.300601+0000", 2024-09-20T22:55:44.625 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 145.261048131, 2024-09-20T22:55:44.625 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.016058294000000001, 2024-09-20T22:55:44.625 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:55:44.626 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:55:44.626 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:55:44.626 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:55:44.626 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.626 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:55:44.626 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:19.300601+0000", 2024-09-20T22:55:44.626 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:55:44.626 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.626 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.626 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:55:44.627 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:19.300601+0000", 2024-09-20T22:55:44.627 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:55:44.627 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.627 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.627 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:55:44.627 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:19.300601+0000", 2024-09-20T22:55:44.627 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.6400000000000001e-07 2024-09-20T22:55:44.627 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.627 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.627 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:55:44.627 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:19.307294+0000", 2024-09-20T22:55:44.627 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0066925200000000004 2024-09-20T22:55:44.628 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.628 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.628 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:55:44.628 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:19.307307+0000", 2024-09-20T22:55:44.628 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3431999999999999e-05 2024-09-20T22:55:44.628 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.628 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.628 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:55:44.628 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:19.307316+0000", 2024-09-20T22:55:44.628 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.5099999999999998e-06 2024-09-20T22:55:44.629 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.629 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.629 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:55:44.629 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:19.307385+0000", 2024-09-20T22:55:44.629 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.8771999999999998e-05 2024-09-20T22:55:44.629 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.629 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.629 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:55:44.629 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:19.307413+0000", 2024-09-20T22:55:44.630 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.8373000000000001e-05 2024-09-20T22:55:44.630 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.630 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.630 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:55:44.630 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:19.316602+0000", 2024-09-20T22:55:44.630 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0091889950000000001 2024-09-20T22:55:44.630 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.630 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.630 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:55:44.630 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:19.316659+0000", 2024-09-20T22:55:44.630 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.7528000000000002e-05 2024-09-20T22:55:44.630 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:55:44.630 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:55:44.631 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:55:44.631 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.631 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.631 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24652.0:7356 7.a e738/94 7:5aaef466:::smithi08477203-40:head v 738'878, mlcod=736'875)", 2024-09-20T22:55:44.631 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:53:38.313061+0000", 2024-09-20T22:55:44.631 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 126.248588527, 2024-09-20T22:55:44.631 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.027472555999999999, 2024-09-20T22:55:44.631 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:55:44.631 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:55:44.631 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:55:44.631 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:55:44.631 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.631 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:55:44.631 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:38.313061+0000", 2024-09-20T22:55:44.632 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:55:44.632 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.632 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.632 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:55:44.632 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:38.313061+0000", 2024-09-20T22:55:44.632 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:55:44.632 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.632 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.632 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:55:44.632 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:38.313061+0000", 2024-09-20T22:55:44.632 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.0399999999999996e-07 2024-09-20T22:55:44.632 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.632 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.632 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:55:44.633 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:38.316261+0000", 2024-09-20T22:55:44.633 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0031994129999999999 2024-09-20T22:55:44.633 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.633 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.633 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:55:44.633 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:38.316282+0000", 2024-09-20T22:55:44.633 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.1090999999999998e-05 2024-09-20T22:55:44.633 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.633 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.633 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:55:44.633 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:38.316295+0000", 2024-09-20T22:55:44.633 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3226e-05 2024-09-20T22:55:44.633 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.633 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.634 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:55:44.634 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:38.316385+0000", 2024-09-20T22:55:44.634 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.0048000000000002e-05 2024-09-20T22:55:44.634 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.634 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.634 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:55:44.634 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:38.316411+0000", 2024-09-20T22:55:44.634 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.6401e-05 2024-09-20T22:55:44.634 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.634 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.634 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:55:44.634 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:38.340477+0000", 2024-09-20T22:55:44.634 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.024065468999999999 2024-09-20T22:55:44.634 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.635 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.635 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:55:44.635 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:38.340533+0000", 2024-09-20T22:55:44.635 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.6403999999999997e-05 2024-09-20T22:55:44.635 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:55:44.635 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:55:44.635 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:55:44.635 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.635 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.635 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24652.0:7432 7.0 e744/632 7:064f66ab:::smithi08477203-45:head v 744'684, mlcod=743'681)", 2024-09-20T22:55:44.635 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:53:44.436593+0000", 2024-09-20T22:55:44.635 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 120.125056625, 2024-09-20T22:55:44.635 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.040892389000000001, 2024-09-20T22:55:44.635 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:55:44.636 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:55:44.636 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:55:44.636 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:55:44.636 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.636 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:55:44.636 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:44.436593+0000", 2024-09-20T22:55:44.636 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:55:44.636 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.636 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.636 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:55:44.636 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:44.436593+0000", 2024-09-20T22:55:44.636 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:55:44.636 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.636 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.637 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:55:44.637 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:44.436593+0000", 2024-09-20T22:55:44.637 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.6699999999999999e-07 2024-09-20T22:55:44.637 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.637 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.637 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:55:44.637 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:44.475632+0000", 2024-09-20T22:55:44.637 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.039039010999999998 2024-09-20T22:55:44.637 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.637 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.637 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:55:44.637 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:44.475645+0000", 2024-09-20T22:55:44.637 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3084999999999999e-05 2024-09-20T22:55:44.638 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.638 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.638 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:55:44.638 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:44.475651+0000", 2024-09-20T22:55:44.638 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.1190000000000002e-06 2024-09-20T22:55:44.638 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.638 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.638 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:55:44.638 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:44.475714+0000", 2024-09-20T22:55:44.638 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.3239999999999998e-05 2024-09-20T22:55:44.638 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.638 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.638 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:55:44.639 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:44.475741+0000", 2024-09-20T22:55:44.639 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.6911e-05 2024-09-20T22:55:44.639 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.639 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.639 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:55:44.639 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:44.477457+0000", 2024-09-20T22:55:44.639 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0017154959999999999 2024-09-20T22:55:44.639 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.639 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.639 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:55:44.639 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:44.477485+0000", 2024-09-20T22:55:44.639 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.8160000000000001e-05 2024-09-20T22:55:44.639 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:55:44.639 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:55:44.639 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:55:44.640 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.640 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.640 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:7625 7.e 7:7c62ced4:::smithi08477203-17:head [copy-from ver 1253] snapc 159=[159,158,157,156,155,153,151,14f,14e,14c,14a,149,148,145,142,13e,13a,139,132,130,12f,12e,12a,124,121,112,103] ondisk+write+known_if_redirected+supports_pool_eio e766)", 2024-09-20T22:55:44.640 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:54:06.336464+0000", 2024-09-20T22:55:44.640 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 98.225184886999998, 2024-09-20T22:55:44.640 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014856506, 2024-09-20T22:55:44.640 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:55:44.640 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:55:44.640 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:55:44.640 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:55:44.640 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:55:44.640 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:55:44.640 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 7625 2024-09-20T22:55:44.640 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.641 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:55:44.641 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.641 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:55:44.641 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:54:06.336464+0000", 2024-09-20T22:55:44.641 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:55:44.641 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.641 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.641 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:55:44.641 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:54:06.336464+0000", 2024-09-20T22:55:44.641 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:55:44.641 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.641 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.641 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:55:44.641 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:54:06.336466+0000", 2024-09-20T22:55:44.642 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.7770000000000001e-06 2024-09-20T22:55:44.642 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.642 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.642 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:55:44.642 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:54:06.336475+0000", 2024-09-20T22:55:44.642 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.9579999999999996e-06 2024-09-20T22:55:44.642 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.642 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.642 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:55:44.642 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:54:06.336491+0000", 2024-09-20T22:55:44.642 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.6079999999999999e-05 2024-09-20T22:55:44.642 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.642 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.643 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:55:44.643 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:54:06.336501+0000", 2024-09-20T22:55:44.643 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.8239999999999995e-06 2024-09-20T22:55:44.643 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.643 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.643 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:55:44.643 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:54:06.336571+0000", 2024-09-20T22:55:44.643 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.9746999999999995e-05 2024-09-20T22:55:44.643 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.643 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.643 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op must wait for map", 2024-09-20T22:55:44.643 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:54:06.336580+0000", 2024-09-20T22:55:44.643 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.2900000000000008e-06 2024-09-20T22:55:44.643 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.644 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.644 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:55:44.644 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:54:06.338593+0000", 2024-09-20T22:55:44.644 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0020125939999999999 2024-09-20T22:55:44.644 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.644 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.644 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:55:44.644 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:54:06.338748+0000", 2024-09-20T22:55:44.644 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00015523199999999999 2024-09-20T22:55:44.644 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.644 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.644 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 0,5", 2024-09-20T22:55:44.644 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:54:06.344247+0000", 2024-09-20T22:55:44.644 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0054991079999999999 2024-09-20T22:55:44.645 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.645 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.645 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:55:44.645 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:54:06.349881+0000", 2024-09-20T22:55:44.645 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0056344689999999996 2024-09-20T22:55:44.645 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.645 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.645 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:55:44.645 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:54:06.351077+0000", 2024-09-20T22:55:44.645 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001196016 2024-09-20T22:55:44.645 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.645 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.645 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:55:44.645 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:54:06.351225+0000", 2024-09-20T22:55:44.645 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014721100000000001 2024-09-20T22:55:44.646 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.646 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.646 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:55:44.646 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:54:06.351265+0000", 2024-09-20T22:55:44.646 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.985e-05 2024-09-20T22:55:44.646 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.646 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.646 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:55:44.646 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:54:06.351321+0000", 2024-09-20T22:55:44.646 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.6350000000000001e-05 2024-09-20T22:55:44.646 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:55:44.646 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:55:44.646 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:55:44.646 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.647 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.647 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:8051 7.8 7:1e9e2524:::smithi08477203-5:head [write 420472~427736,stat] snapc 171=[171,16f,16e,16d,16c,16b,167,166,165,160,15f,15e,15c,15a,158,157,156,155,153,14f,14e,14a,149,145,142,13e,13a,139,12f,12e,124] ondisk+write+known_if_redirected+supports_pool_eio e810)", 2024-09-20T22:55:44.647 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:54:50.344852+0000", 2024-09-20T22:55:44.647 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 54.216797290000002, 2024-09-20T22:55:44.647 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013666950000000001, 2024-09-20T22:55:44.647 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:55:44.647 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:55:44.647 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:55:44.647 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:55:44.647 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:55:44.647 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:55:44.647 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 8051 2024-09-20T22:55:44.647 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.647 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:55:44.647 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.648 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:55:44.648 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:54:50.344852+0000", 2024-09-20T22:55:44.648 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:55:44.648 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.648 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.648 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:55:44.648 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:54:50.344852+0000", 2024-09-20T22:55:44.648 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:55:44.648 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.648 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.648 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:55:44.648 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:54:50.344854+0000", 2024-09-20T22:55:44.648 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.21e-06 2024-09-20T22:55:44.648 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.648 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.649 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:55:44.649 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:54:50.345132+0000", 2024-09-20T22:55:44.649 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00027765999999999999 2024-09-20T22:55:44.649 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.649 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.649 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:55:44.649 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:54:50.345150+0000", 2024-09-20T22:55:44.649 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.7876e-05 2024-09-20T22:55:44.649 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.649 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.649 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:55:44.649 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:54:50.345160+0000", 2024-09-20T22:55:44.650 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0349e-05 2024-09-20T22:55:44.650 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.650 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.650 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:55:44.650 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:54:50.345261+0000", 2024-09-20T22:55:44.650 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000100621 2024-09-20T22:55:44.650 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.650 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.650 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op must wait for map", 2024-09-20T22:55:44.650 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:54:50.345268+0000", 2024-09-20T22:55:44.651 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.5320000000000001e-06 2024-09-20T22:55:44.651 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.651 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.651 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:55:44.651 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:54:50.350192+0000", 2024-09-20T22:55:44.651 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0049241659999999998 2024-09-20T22:55:44.651 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.651 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.651 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:55:44.651 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:54:50.350315+0000", 2024-09-20T22:55:44.651 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000122948 2024-09-20T22:55:44.651 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.651 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.651 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 4,7", 2024-09-20T22:55:44.652 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:54:50.350506+0000", 2024-09-20T22:55:44.652 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000190609 2024-09-20T22:55:44.652 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.652 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.652 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:55:44.652 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:54:50.354160+0000", 2024-09-20T22:55:44.652 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.003654058 2024-09-20T22:55:44.652 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.652 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.652 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:55:44.653 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:54:50.354235+0000", 2024-09-20T22:55:44.653 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.4894000000000005e-05 2024-09-20T22:55:44.653 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.653 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.653 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:55:44.653 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:54:50.358382+0000", 2024-09-20T22:55:44.653 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0041471839999999999 2024-09-20T22:55:44.653 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.653 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.653 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:55:44.653 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:54:50.358448+0000", 2024-09-20T22:55:44.653 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.5857000000000003e-05 2024-09-20T22:55:44.653 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.654 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.654 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:55:44.654 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:54:50.358519+0000", 2024-09-20T22:55:44.654 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.0986000000000001e-05 2024-09-20T22:55:44.654 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:55:44.654 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:55:44.654 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:55:44.654 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.654 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.654 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:55:44.655 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:8458 7.e 7:7c62ced4:::smithi08477203-17:head [copy-from ver 2639] snapc 18d=[18d,18c,18b,18a,189,188,187,186,185,184,183,182,181,180,17f,17e,17d,17c,17b,17a,179,177,176,175,174,173,172,16f,16e,16c,16b,166,160,15c,15a,158,157,155,14f,14e,14a,149,13a,139,12f,12e] ondisk+write+known_if_redirected+supports_pool_eio e851)", 2024-09-20T22:55:44.655 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:55:31.382098+0000", 2024-09-20T22:55:44.655 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 13.17955102, 2024-09-20T22:55:44.655 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.019106850000000002, 2024-09-20T22:55:44.655 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:55:44.655 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:55:44.655 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:55:44.655 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:55:44.655 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:55:44.656 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:55:44.656 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 8458 2024-09-20T22:55:44.656 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.656 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:55:44.656 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.656 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:55:44.656 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:55:31.382098+0000", 2024-09-20T22:55:44.656 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:55:44.656 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.656 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.656 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:55:44.656 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:55:31.382098+0000", 2024-09-20T22:55:44.657 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:55:44.657 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.657 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.657 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:55:44.657 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:55:31.382098+0000", 2024-09-20T22:55:44.657 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.5400000000000002e-07 2024-09-20T22:55:44.657 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.657 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.657 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:55:44.657 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:55:31.382100+0000", 2024-09-20T22:55:44.658 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.905e-06 2024-09-20T22:55:44.658 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.658 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.658 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:55:44.658 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:55:31.382103+0000", 2024-09-20T22:55:44.658 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.9450000000000002e-06 2024-09-20T22:55:44.658 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.658 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.658 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:55:44.658 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:55:31.382106+0000", 2024-09-20T22:55:44.658 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.7810000000000001e-06 2024-09-20T22:55:44.658 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.658 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.659 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:55:44.659 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:55:31.382166+0000", 2024-09-20T22:55:44.659 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.9876000000000001e-05 2024-09-20T22:55:44.659 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.659 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.659 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:55:44.659 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:55:31.382296+0000", 2024-09-20T22:55:44.659 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00012992 2024-09-20T22:55:44.659 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.659 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.660 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 0,5", 2024-09-20T22:55:44.660 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:55:31.389774+0000", 2024-09-20T22:55:44.660 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0074777969999999996 2024-09-20T22:55:44.660 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.660 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.660 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:55:44.660 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:55:31.396494+0000", 2024-09-20T22:55:44.660 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0067207869999999998 2024-09-20T22:55:44.660 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.660 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.661 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:55:44.661 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:55:31.397877+0000", 2024-09-20T22:55:44.661 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001382306 2024-09-20T22:55:44.661 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.661 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.661 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:55:44.661 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:55:31.401048+0000", 2024-09-20T22:55:44.661 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0031707739999999999 2024-09-20T22:55:44.661 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.661 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.662 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:55:44.662 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:55:31.401130+0000", 2024-09-20T22:55:44.662 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.2281999999999996e-05 2024-09-20T22:55:44.662 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.662 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.662 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:55:44.662 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:55:31.401205+0000", 2024-09-20T22:55:44.662 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.5222999999999995e-05 2024-09-20T22:55:44.662 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:55:44.663 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:55:44.663 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:55:44.663 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.663 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.663 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:8459 7.e 7:7c62ced4:::smithi08477203-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e851)", 2024-09-20T22:55:44.663 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:55:31.382114+0000", 2024-09-20T22:55:44.663 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 13.179534777000001, 2024-09-20T22:55:44.663 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.012195655999999999, 2024-09-20T22:55:44.663 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:55:44.663 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:55:44.663 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-20T22:55:44.664 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:55:44.664 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:55:44.664 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:55:44.664 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 8459 2024-09-20T22:55:44.664 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.664 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:55:44.664 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.664 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:55:44.665 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:55:31.382114+0000", 2024-09-20T22:55:44.665 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:55:44.665 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.665 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.665 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:55:44.665 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:55:31.382114+0000", 2024-09-20T22:55:44.665 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:55:44.665 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.665 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.665 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:55:44.665 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:55:31.382115+0000", 2024-09-20T22:55:44.665 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.1e-07 2024-09-20T22:55:44.665 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.665 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.666 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:55:44.666 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:55:31.382116+0000", 2024-09-20T22:55:44.666 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1790000000000001e-06 2024-09-20T22:55:44.666 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.666 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.666 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:55:44.666 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:55:31.382120+0000", 2024-09-20T22:55:44.666 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.2699999999999998e-06 2024-09-20T22:55:44.666 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.666 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.666 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:55:44.666 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:55:31.382123+0000", 2024-09-20T22:55:44.666 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.6800000000000002e-06 2024-09-20T22:55:44.667 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.667 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.667 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:55:44.667 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:55:31.382514+0000", 2024-09-20T22:55:44.667 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00039134700000000001 2024-09-20T22:55:44.667 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.667 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.667 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for blocked object", 2024-09-20T22:55:44.667 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:55:31.382576+0000", 2024-09-20T22:55:44.667 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.1954999999999999e-05 2024-09-20T22:55:44.667 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.667 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.667 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:55:44.667 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:55:31.394061+0000", 2024-09-20T22:55:44.667 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011484604000000001 2024-09-20T22:55:44.668 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.668 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.668 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:55:44.668 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:55:31.394187+0000", 2024-09-20T22:55:44.668 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000126513 2024-09-20T22:55:44.668 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.668 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.668 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:55:44.668 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:55:31.394310+0000", 2024-09-20T22:55:44.668 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000122798 2024-09-20T22:55:44.668 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:55:44.668 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:55:44.668 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:55:44.668 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.669 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.669 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(osd.7.0:554 7.5 e861/456 7:a160b5d4:::smithi08477203-16:16c v 861'466, mlcod=856'462)", 2024-09-20T22:55:44.669 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:55:41.351562+0000", 2024-09-20T22:55:44.669 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 3.2100870270000001, 2024-09-20T22:55:44.669 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0024407209999999999, 2024-09-20T22:55:44.669 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:55:44.669 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:55:44.669 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:55:44.669 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:55:44.669 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.669 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:55:44.669 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:55:41.351562+0000", 2024-09-20T22:55:44.669 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:55:44.669 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.669 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.670 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:55:44.670 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:55:41.351562+0000", 2024-09-20T22:55:44.670 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:55:44.670 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.670 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.670 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:55:44.670 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:55:41.351563+0000", 2024-09-20T22:55:44.670 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.2200000000000004e-07 2024-09-20T22:55:44.670 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.670 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.670 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:55:44.670 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:55:41.351569+0000", 2024-09-20T22:55:44.670 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.4489999999999996e-06 2024-09-20T22:55:44.671 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.671 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.671 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:55:44.671 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:55:41.351581+0000", 2024-09-20T22:55:44.671 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2221999999999999e-05 2024-09-20T22:55:44.671 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.671 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.671 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:55:44.671 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:55:41.351589+0000", 2024-09-20T22:55:44.671 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.9440000000000005e-06 2024-09-20T22:55:44.671 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.671 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.671 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:55:44.671 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:55:41.351660+0000", 2024-09-20T22:55:44.672 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.0650999999999998e-05 2024-09-20T22:55:44.672 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.672 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.672 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:55:44.672 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:55:41.351693+0000", 2024-09-20T22:55:44.672 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.3212e-05 2024-09-20T22:55:44.672 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.672 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.672 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:55:44.672 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:55:41.353961+0000", 2024-09-20T22:55:44.672 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0022674280000000002 2024-09-20T22:55:44.672 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.672 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.672 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:55:44.672 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:55:41.354003+0000", 2024-09-20T22:55:44.673 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.2293000000000003e-05 2024-09-20T22:55:44.673 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:55:44.673 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:55:44.673 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:55:44.673 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.673 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.673 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(osd.3.0:1649 7.2 e863/809 7:4d20232a:::smithi08477203-34:17e v 863'899, mlcod=860'893)", 2024-09-20T22:55:44.673 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:55:43.534512+0000", 2024-09-20T22:55:44.673 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 1.02713677, 2024-09-20T22:55:44.673 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001234099, 2024-09-20T22:55:44.673 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:55:44.673 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:55:44.673 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:55:44.673 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:55:44.673 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.674 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:55:44.674 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:55:43.534512+0000", 2024-09-20T22:55:44.674 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:55:44.674 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.674 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.674 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:55:44.674 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:55:43.534512+0000", 2024-09-20T22:55:44.674 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:55:44.674 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.674 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.674 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:55:44.674 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:55:43.534513+0000", 2024-09-20T22:55:44.674 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.5099999999999999e-07 2024-09-20T22:55:44.674 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.675 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.675 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:55:44.675 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:55:43.534520+0000", 2024-09-20T22:55:44.675 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.5570000000000001e-06 2024-09-20T22:55:44.675 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.675 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.675 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:55:44.675 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:55:43.534530+0000", 2024-09-20T22:55:44.675 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.8770000000000005e-06 2024-09-20T22:55:44.675 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.675 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.675 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:55:44.675 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:55:43.534536+0000", 2024-09-20T22:55:44.675 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.7719999999999997e-06 2024-09-20T22:55:44.676 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.676 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.676 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:55:44.676 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:55:43.534653+0000", 2024-09-20T22:55:44.676 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00011636100000000001 2024-09-20T22:55:44.676 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.676 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.676 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:55:44.676 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:55:43.534672+0000", 2024-09-20T22:55:44.676 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.9154e-05 2024-09-20T22:55:44.676 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.676 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.676 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:55:44.676 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:55:43.535713+0000", 2024-09-20T22:55:44.676 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010413550000000001 2024-09-20T22:55:44.677 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:44.677 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:44.677 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:55:44.677 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:55:43.535747+0000", 2024-09-20T22:55:44.677 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.3272000000000003e-05 2024-09-20T22:55:44.677 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:55:44.677 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:55:44.677 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:55:44.677 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:55:44.677 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:55:44.677 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-20T22:55:44.677 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:55:44.703 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:44.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:44.704 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:55:44.805 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:55:44.827 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:44.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:44.829 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:55:44.929 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:55:44.955 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:44.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:44.956 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:55:45.057 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:55:45.078 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:45.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:45.080 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:55:45.180 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:55:45.202 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:45.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:45.203 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:55:45.303 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:55:45.324 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:45.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:45.325 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:55:45.375 INFO:tasks.rados.rados.0.smithi084.stdout:3447: done (3 left) 2024-09-20T22:55:45.375 INFO:tasks.rados.rados.0.smithi084.stdout:3448: snap_remove snap 402 2024-09-20T22:55:45.426 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:55:45.451 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:45.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:45.452 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:55:45.474 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:55:45.474 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:55:45.474 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:55:45.474 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-20T22:55:45.475 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:55:45.475 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:55:45.475 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:55:45.475 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T22:55:45.477 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:55:45.477 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:55:45.493 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:55:45.494 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:55:45.510 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:55:45.511 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:55:45.527 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:55:45.528 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:55:45.552 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:55:45.576 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:45.576 INFO:tasks.thrashosds.thrasher:in_osds: [2, 3, 5, 6, 7, 0, 4, 1] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-20T22:55:45.576 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-20T22:55:45.577 INFO:tasks.thrashosds.thrasher:inject_pause on osd.1 2024-09-20T22:55:45.577 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-20T22:55:45.577 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-20T22:55:45.577 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.1 config set filestore_inject_stall 3 2024-09-20T22:55:45.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:45.578 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:55:45.599 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:45 smithi084 ceph-mon[40024]: pgmap v1197: 158 pgs: 2 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 152 active+clean; 618 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 6.9 MiB/s rd, 2.0 MiB/s wr, 11 op/s 2024-09-20T22:55:45.599 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:45 smithi084 ceph-mon[40024]: osdmap e864: 8 total, 8 up, 8 in 2024-09-20T22:55:45.599 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:45 smithi084 ceph-mon[29774]: pgmap v1197: 158 pgs: 2 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 152 active+clean; 618 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 6.9 MiB/s rd, 2.0 MiB/s wr, 11 op/s 2024-09-20T22:55:45.599 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:45 smithi084 ceph-mon[29774]: osdmap e864: 8 total, 8 up, 8 in 2024-09-20T22:55:45.679 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:55:45.703 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:45.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:45.703 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:55:45.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:45 smithi154 ceph-mon[33264]: pgmap v1197: 158 pgs: 2 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 152 active+clean; 618 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 6.9 MiB/s rd, 2.0 MiB/s wr, 11 op/s 2024-09-20T22:55:45.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:45 smithi154 ceph-mon[33264]: osdmap e864: 8 total, 8 up, 8 in 2024-09-20T22:55:45.805 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:55:45.828 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:45.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:45.829 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:55:45.930 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:55:45.952 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:45.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:45.952 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:55:45.986 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd unset noscrub 2024-09-20T22:55:46.053 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:55:46.077 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:46.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:46.078 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:55:46.179 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:55:46.202 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:46.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:46.203 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:55:46.285 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-20T22:55:46.304 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:55:46.326 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:46.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:46.327 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:55:46.377 INFO:tasks.rados.rados.0.smithi084.stdout:3448: done (3 left) 2024-09-20T22:55:46.377 INFO:tasks.rados.rados.0.smithi084.stdout:3449: write (excl) oid 34 current snap is 403 2024-09-20T22:55:46.377 INFO:tasks.rados.rados.0.smithi084.stdout:3449: seq_num 1170 ranges {464190=643339,1748991=445068,2930078=574608} 2024-09-20T22:55:46.382 INFO:tasks.rados.rados.0.smithi084.stdout:3449: writing smithi08477203-34 from 464190 to 1107529 tid 1 2024-09-20T22:55:46.390 INFO:tasks.rados.rados.0.smithi084.stdout:3449: writing smithi08477203-34 from 1748991 to 2194059 tid 2 2024-09-20T22:55:46.397 INFO:tasks.rados.rados.0.smithi084.stdout:3449: writing smithi08477203-34 from 2930078 to 3504686 tid 3 2024-09-20T22:55:46.398 INFO:tasks.rados.rados.0.smithi084.stdout:3450: rollback oid 9 current snap is 403 2024-09-20T22:55:46.398 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 9 to 377 2024-09-20T22:55:46.398 INFO:tasks.rados.rados.0.smithi084.stdout:3443: finishing write tid 2 to smithi08477203-13 2024-09-20T22:55:46.398 INFO:tasks.rados.rados.0.smithi084.stdout:3443: finishing write tid 3 to smithi08477203-13 2024-09-20T22:55:46.398 INFO:tasks.rados.rados.0.smithi084.stdout:3443: finishing write tid 4 to smithi08477203-13 2024-09-20T22:55:46.398 INFO:tasks.rados.rados.0.smithi084.stdout:3443: finishing write tid 5 to smithi08477203-13 2024-09-20T22:55:46.398 INFO:tasks.rados.rados.0.smithi084.stdout:3443: finishing write tid 6 to smithi08477203-13 2024-09-20T22:55:46.398 INFO:tasks.rados.rados.0.smithi084.stdout:3443: oid 13 updating version 0 to 2746 2024-09-20T22:55:46.398 INFO:tasks.rados.rados.0.smithi084.stdout:3443: oid 13 version 2746 is already newer than 2745 2024-09-20T22:55:46.398 INFO:tasks.rados.rados.0.smithi084.stdout:3443: oid 13 updating version 2746 to 2747 2024-09-20T22:55:46.398 INFO:tasks.rados.rados.0.smithi084.stdout:3443: oid 13 version 2747 is already newer than 2744 2024-09-20T22:55:46.398 INFO:tasks.rados.rados.0.smithi084.stdout:3443: oid 13 version 2747 is already newer than 2743 2024-09-20T22:55:46.399 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 13 v 2747 (ObjNum 1168 snap 403 seq_num 1168) dirty exists 2024-09-20T22:55:46.399 INFO:tasks.rados.rados.0.smithi084.stdout:3443: left oid 13 (ObjNum 1168 snap 403 seq_num 1168) 2024-09-20T22:55:46.399 INFO:tasks.rados.rados.0.smithi084.stdout:3444: finishing write tid 1 to smithi08477203-29 2024-09-20T22:55:46.399 INFO:tasks.rados.rados.0.smithi084.stdout:3444: finishing write tid 2 to smithi08477203-29 2024-09-20T22:55:46.399 INFO:tasks.rados.rados.0.smithi084.stdout:3443: done (4 left) 2024-09-20T22:55:46.399 INFO:tasks.rados.rados.0.smithi084.stdout:3451: write (excl) oid 6 current snap is 403 2024-09-20T22:55:46.399 INFO:tasks.rados.rados.0.smithi084.stdout:3444: finishing write tid 3 to smithi08477203-29 2024-09-20T22:55:46.399 INFO:tasks.rados.rados.0.smithi084.stdout:3451: seq_num 1171 ranges {480997=434941,1407473=705079,2557501=785880} 2024-09-20T22:55:46.402 INFO:tasks.rados.rados.0.smithi084.stdout:3451: writing smithi08477203-6 from 480997 to 915938 tid 1 2024-09-20T22:55:46.410 INFO:tasks.rados.rados.0.smithi084.stdout:3451: writing smithi08477203-6 from 1407473 to 2112552 tid 2 2024-09-20T22:55:46.417 INFO:tasks.rados.rados.0.smithi084.stdout:3451: writing smithi08477203-6 from 2557501 to 3343381 tid 3 2024-09-20T22:55:46.417 INFO:tasks.rados.rados.0.smithi084.stdout:3452: delete oid 21 current snap is 403 2024-09-20T22:55:46.418 INFO:tasks.rados.rados.0.smithi084.stdout:3444: finishing write tid 4 to smithi08477203-29 2024-09-20T22:55:46.418 INFO:tasks.rados.rados.0.smithi084.stdout:3444: finishing write tid 5 to smithi08477203-29 2024-09-20T22:55:46.418 INFO:tasks.rados.rados.0.smithi084.stdout:3444: oid 29 updating version 0 to 2751 2024-09-20T22:55:46.418 INFO:tasks.rados.rados.0.smithi084.stdout:3444: oid 29 updating version 2751 to 2752 2024-09-20T22:55:46.418 INFO:tasks.rados.rados.0.smithi084.stdout:3444: oid 29 version 2752 is already newer than 2750 2024-09-20T22:55:46.418 INFO:tasks.rados.rados.0.smithi084.stdout:3444: oid 29 version 2752 is already newer than 2749 2024-09-20T22:55:46.418 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 29 v 2752 (ObjNum 1169 snap 403 seq_num 1169) dirty exists 2024-09-20T22:55:46.418 INFO:tasks.rados.rados.0.smithi084.stdout:3444: left oid 29 (ObjNum 1169 snap 403 seq_num 1169) 2024-09-20T22:55:46.418 INFO:tasks.rados.rados.0.smithi084.stdout:3445: expect (ObjNum 1158 snap 397 seq_num 1158) 2024-09-20T22:55:46.428 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:55:46.517 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:46.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:46.518 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:55:46.614 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:46 smithi084 ceph-mon[40024]: osdmap e865: 8 total, 8 up, 8 in 2024-09-20T22:55:46.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:46 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/3921562371' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:55:46.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:46 smithi084 ceph-mon[29774]: osdmap e865: 8 total, 8 up, 8 in 2024-09-20T22:55:46.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:46 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/3921562371' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:55:46.620 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:55:46.633 INFO:tasks.rados.rados.0.smithi084.stdout:3449: finishing write tid 1 to smithi08477203-34 2024-09-20T22:55:46.633 INFO:tasks.rados.rados.0.smithi084.stdout:3449: finishing write tid 2 to smithi08477203-34 2024-09-20T22:55:46.633 INFO:tasks.rados.rados.0.smithi084.stdout:3449: finishing write tid 3 to smithi08477203-34 2024-09-20T22:55:46.633 INFO:tasks.rados.rados.0.smithi084.stdout:3449: finishing write tid 4 to smithi08477203-34 2024-09-20T22:55:46.633 INFO:tasks.rados.rados.0.smithi084.stdout:3449: finishing write tid 5 to smithi08477203-34 2024-09-20T22:55:46.633 INFO:tasks.rados.rados.0.smithi084.stdout:3449: oid 34 updating version 0 to 1865 2024-09-20T22:55:46.633 INFO:tasks.rados.rados.0.smithi084.stdout:3449: oid 34 version 1865 is already newer than 1863 2024-09-20T22:55:46.633 INFO:tasks.rados.rados.0.smithi084.stdout:3449: oid 34 version 1865 is already newer than 1862 2024-09-20T22:55:46.633 INFO:tasks.rados.rados.0.smithi084.stdout:3449: oid 34 version 1865 is already newer than 1864 2024-09-20T22:55:46.633 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 34 v 1865 (ObjNum 1170 snap 403 seq_num 1170) dirty exists 2024-09-20T22:55:46.633 INFO:tasks.rados.rados.0.smithi084.stdout:3449: left oid 34 (ObjNum 1170 snap 403 seq_num 1170) 2024-09-20T22:55:46.634 INFO:tasks.rados.rados.0.smithi084.stdout:3450: finishing rollback tid 1 to smithi08477203-9 2024-09-20T22:55:46.634 INFO:tasks.rados.rados.0.smithi084.stdout:3450: finishing rollback tid 2 to smithi08477203-9 2024-09-20T22:55:46.634 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 9 v 2641 (ObjNum 1094 snap 367 seq_num 1094) dirty exists 2024-09-20T22:55:46.634 INFO:tasks.rados.rados.0.smithi084.stdout:3451: finishing write tid 1 to smithi08477203-6 2024-09-20T22:55:46.634 INFO:tasks.rados.rados.0.smithi084.stdout:3444: done (5 left) 2024-09-20T22:55:46.634 INFO:tasks.rados.rados.0.smithi084.stdout:3445: done (4 left) 2024-09-20T22:55:46.634 INFO:tasks.rados.rados.0.smithi084.stdout:3449: done (3 left) 2024-09-20T22:55:46.634 INFO:tasks.rados.rados.0.smithi084.stdout:3450: done (2 left) 2024-09-20T22:55:46.634 INFO:tasks.rados.rados.0.smithi084.stdout:3452: done (1 left) 2024-09-20T22:55:46.634 INFO:tasks.rados.rados.0.smithi084.stdout:3453: delete oid 44 current snap is 403 2024-09-20T22:55:46.634 INFO:tasks.rados.rados.0.smithi084.stdout:3451: finishing write tid 2 to smithi08477203-6 2024-09-20T22:55:46.634 INFO:tasks.rados.rados.0.smithi084.stdout:3451: finishing write tid 3 to smithi08477203-6 2024-09-20T22:55:46.634 INFO:tasks.rados.rados.0.smithi084.stdout:3451: finishing write tid 4 to smithi08477203-6 2024-09-20T22:55:46.634 INFO:tasks.rados.rados.0.smithi084.stdout:3451: finishing write tid 5 to smithi08477203-6 2024-09-20T22:55:46.635 INFO:tasks.rados.rados.0.smithi084.stdout:3451: oid 6 updating version 0 to 2643 2024-09-20T22:55:46.635 INFO:tasks.rados.rados.0.smithi084.stdout:3451: oid 6 version 2643 is already newer than 2640 2024-09-20T22:55:46.635 INFO:tasks.rados.rados.0.smithi084.stdout:3451: oid 6 version 2643 is already newer than 2641 2024-09-20T22:55:46.635 INFO:tasks.rados.rados.0.smithi084.stdout:3451: oid 6 version 2643 is already newer than 2642 2024-09-20T22:55:46.635 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 6 v 2643 (ObjNum 1171 snap 403 seq_num 1171) dirty exists 2024-09-20T22:55:46.635 INFO:tasks.rados.rados.0.smithi084.stdout:3451: left oid 6 (ObjNum 1171 snap 403 seq_num 1171) 2024-09-20T22:55:46.636 INFO:tasks.rados.rados.0.smithi084.stdout:3451: done (1 left) 2024-09-20T22:55:46.636 INFO:tasks.rados.rados.0.smithi084.stdout:3453: done (0 left) 2024-09-20T22:55:46.636 INFO:tasks.rados.rados.0.smithi084.stdout:3454: write oid 19 current snap is 403 2024-09-20T22:55:46.636 INFO:tasks.rados.rados.0.smithi084.stdout:3454: seq_num 1174 ranges {531418=660688,1982919=452406,3039770=691648} 2024-09-20T22:55:46.642 INFO:tasks.rados.rados.0.smithi084.stdout:3454: writing smithi08477203-19 from 531418 to 1192106 tid 1 2024-09-20T22:55:46.646 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:46.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:46.647 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:55:46.649 INFO:tasks.rados.rados.0.smithi084.stdout:3454: writing smithi08477203-19 from 1982919 to 2435325 tid 2 2024-09-20T22:55:46.656 INFO:tasks.rados.rados.0.smithi084.stdout:3454: writing smithi08477203-19 from 3039770 to 3731418 tid 3 2024-09-20T22:55:46.656 INFO:tasks.rados.rados.0.smithi084.stdout:3455: write oid 33 current snap is 403 2024-09-20T22:55:46.657 INFO:tasks.rados.rados.0.smithi084.stdout:3454: finishing write tid 1 to smithi08477203-19 2024-09-20T22:55:46.657 INFO:tasks.rados.rados.0.smithi084.stdout:3454: finishing write tid 2 to smithi08477203-19 2024-09-20T22:55:46.657 INFO:tasks.rados.rados.0.smithi084.stdout:3455: seq_num 1175 ranges {548225=735937,1925048=596064,2950840=547321} 2024-09-20T22:55:46.663 INFO:tasks.rados.rados.0.smithi084.stdout:3455: writing smithi08477203-33 from 548225 to 1284162 tid 1 2024-09-20T22:55:46.670 INFO:tasks.rados.rados.0.smithi084.stdout:3455: writing smithi08477203-33 from 1925048 to 2521112 tid 2 2024-09-20T22:55:46.676 INFO:tasks.rados.rados.0.smithi084.stdout:3455: writing smithi08477203-33 from 2950840 to 3498161 tid 3 2024-09-20T22:55:46.676 INFO:tasks.rados.rados.0.smithi084.stdout:3456: rollback oid 13 current snap is 403 2024-09-20T22:55:46.676 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 13 to 399 2024-09-20T22:55:46.676 INFO:tasks.rados.rados.0.smithi084.stdout:3454: finishing write tid 3 to smithi08477203-19 2024-09-20T22:55:46.676 INFO:tasks.rados.rados.0.smithi084.stdout:3457: snap_create 2024-09-20T22:55:46.676 INFO:tasks.rados.rados.0.smithi084.stdout:3454: finishing write tid 4 to smithi08477203-19 2024-09-20T22:55:46.676 INFO:tasks.rados.rados.0.smithi084.stdout:3454: finishing write tid 5 to smithi08477203-19 2024-09-20T22:55:46.676 INFO:tasks.rados.rados.0.smithi084.stdout:3454: oid 19 updating version 0 to 2770 2024-09-20T22:55:46.677 INFO:tasks.rados.rados.0.smithi084.stdout:3454: oid 19 updating version 2770 to 2771 2024-09-20T22:55:46.677 INFO:tasks.rados.rados.0.smithi084.stdout:3454: oid 19 updating version 2771 to 2773 2024-09-20T22:55:46.677 INFO:tasks.rados.rados.0.smithi084.stdout:3454: oid 19 version 2773 is already newer than 2772 2024-09-20T22:55:46.677 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 19 v 2773 (ObjNum 1174 snap 403 seq_num 1174) dirty exists 2024-09-20T22:55:46.677 INFO:tasks.rados.rados.0.smithi084.stdout:3454: left oid 19 (ObjNum 1174 snap 403 seq_num 1174) 2024-09-20T22:55:46.677 INFO:tasks.rados.rados.0.smithi084.stdout:3455: finishing write tid 1 to smithi08477203-33 2024-09-20T22:55:46.677 INFO:tasks.rados.rados.0.smithi084.stdout:3455: finishing write tid 2 to smithi08477203-33 2024-09-20T22:55:46.679 INFO:tasks.rados.rados.0.smithi084.stdout:3455: finishing write tid 3 to smithi08477203-33 2024-09-20T22:55:46.679 INFO:tasks.rados.rados.0.smithi084.stdout:3455: finishing write tid 4 to smithi08477203-33 2024-09-20T22:55:46.679 INFO:tasks.rados.rados.0.smithi084.stdout:3455: finishing write tid 5 to smithi08477203-33 2024-09-20T22:55:46.679 INFO:tasks.rados.rados.0.smithi084.stdout:3455: oid 33 updating version 0 to 2728 2024-09-20T22:55:46.679 INFO:tasks.rados.rados.0.smithi084.stdout:3455: oid 33 version 2728 is already newer than 2727 2024-09-20T22:55:46.679 INFO:tasks.rados.rados.0.smithi084.stdout:3455: oid 33 version 2728 is already newer than 2726 2024-09-20T22:55:46.679 INFO:tasks.rados.rados.0.smithi084.stdout:3455: oid 33 version 2728 is already newer than 2725 2024-09-20T22:55:46.679 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 33 v 2728 (ObjNum 1175 snap 403 seq_num 1175) dirty exists 2024-09-20T22:55:46.679 INFO:tasks.rados.rados.0.smithi084.stdout:3455: left oid 33 (ObjNum 1175 snap 403 seq_num 1175) 2024-09-20T22:55:46.680 INFO:tasks.rados.rados.0.smithi084.stdout:3456: finishing rollback tid 0 to smithi08477203-13 2024-09-20T22:55:46.680 INFO:tasks.rados.rados.0.smithi084.stdout:3456: finishing rollback tid 1 to smithi08477203-13 2024-09-20T22:55:46.680 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 13 v 2775 (ObjNum 305 snap 0 seq_num 1) dirty dne 2024-09-20T22:55:46.747 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:55:46.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:46 smithi154 ceph-mon[33264]: osdmap e865: 8 total, 8 up, 8 in 2024-09-20T22:55:46.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:46 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/3921562371' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:55:46.770 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:46.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:46.771 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:55:46.871 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:55:46.893 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:46.893 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:55:46.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:46.895 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:55:46.895 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:55:46.996 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:55:47.019 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:47.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:47.020 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:55:47.120 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:55:47.141 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:47.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:47.142 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:55:47.243 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:55:47.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:47.267 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:47.267 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:55:47.367 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:55:47.382 INFO:tasks.rados.rados.0.smithi084.stdout:3454: done (3 left) 2024-09-20T22:55:47.382 INFO:tasks.rados.rados.0.smithi084.stdout:3455: done (2 left) 2024-09-20T22:55:47.382 INFO:tasks.rados.rados.0.smithi084.stdout:3456: done (1 left) 2024-09-20T22:55:47.383 INFO:tasks.rados.rados.0.smithi084.stdout:3457: done (0 left) 2024-09-20T22:55:47.383 INFO:tasks.rados.rados.0.smithi084.stdout:3458: delete oid 6 current snap is 404 2024-09-20T22:55:47.387 INFO:tasks.rados.rados.0.smithi084.stdout:3458: done (0 left) 2024-09-20T22:55:47.387 INFO:tasks.rados.rados.0.smithi084.stdout:3459: rollback oid 35 current snap is 404 2024-09-20T22:55:47.387 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 35 to 385 2024-09-20T22:55:47.387 INFO:tasks.rados.rados.0.smithi084.stdout:3460: snap_remove snap 378 2024-09-20T22:55:47.392 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:47.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:47.394 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:55:47.494 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:55:47.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:47.519 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:47.519 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:55:47.620 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:55:47.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:47.643 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:47.643 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:55:47.660 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:47 smithi084 ceph-mon[29774]: pgmap v1200: 158 pgs: 2 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 152 active+clean; 618 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 6.9 MiB/s rd, 2.0 MiB/s wr, 11 op/s 2024-09-20T22:55:47.660 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:47 smithi084 ceph-mon[29774]: osdmap e866: 8 total, 8 up, 8 in 2024-09-20T22:55:47.660 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:47 smithi084 ceph-mon[40024]: pgmap v1200: 158 pgs: 2 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 152 active+clean; 618 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 6.9 MiB/s rd, 2.0 MiB/s wr, 11 op/s 2024-09-20T22:55:47.661 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:47 smithi084 ceph-mon[40024]: osdmap e866: 8 total, 8 up, 8 in 2024-09-20T22:55:47.744 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:55:47.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:47 smithi154 ceph-mon[33264]: pgmap v1200: 158 pgs: 2 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 152 active+clean; 618 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 6.9 MiB/s rd, 2.0 MiB/s wr, 11 op/s 2024-09-20T22:55:47.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:47 smithi154 ceph-mon[33264]: osdmap e866: 8 total, 8 up, 8 in 2024-09-20T22:55:47.760 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-20T22:55:47.760 INFO:teuthology.orchestra.run.smithi084.stdout: "success": "" 2024-09-20T22:55:47.760 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-20T22:55:47.769 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:47.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:47.770 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:55:47.871 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:55:47.895 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:47.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:47.896 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:55:47.997 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:55:48.013 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:55:48.020 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:48.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:48.020 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:55:48.121 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:55:48.142 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:48.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:48.143 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:55:48.244 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:55:48.270 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:48.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:48.271 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:55:48.331 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is unset 2024-09-20T22:55:48.336 INFO:tasks.rados.rados.0.smithi084.stdout:3460: done (1 left) 2024-09-20T22:55:48.336 INFO:tasks.rados.rados.0.smithi084.stdout:3461: snap_remove snap 389 2024-09-20T22:55:48.372 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:55:48.395 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:48.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:48.396 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:55:48.497 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:55:48.518 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:48.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:48.519 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:55:48.549 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:55:48 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: ::ffff:172.21.15.154 - - [20/Sep/2024:22:55:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-20T22:55:48.550 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:48 smithi084 ceph-mon[40024]: purged_snaps scrub starts 2024-09-20T22:55:48.550 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:48 smithi084 ceph-mon[40024]: purged_snaps scrub ok 2024-09-20T22:55:48.550 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:48 smithi084 ceph-mon[40024]: osdmap e867: 8 total, 8 up, 8 in 2024-09-20T22:55:48.550 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:48 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/1663634611' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:55:48.550 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:48 smithi084 ceph-mon[40024]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:55:48.550 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:48 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/1663634611' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T22:55:48.550 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:48 smithi084 ceph-mon[40024]: osdmap e868: 8 total, 8 up, 8 in 2024-09-20T22:55:48.550 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:48 smithi084 ceph-mon[29774]: purged_snaps scrub starts 2024-09-20T22:55:48.551 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:48 smithi084 ceph-mon[29774]: purged_snaps scrub ok 2024-09-20T22:55:48.551 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:48 smithi084 ceph-mon[29774]: osdmap e867: 8 total, 8 up, 8 in 2024-09-20T22:55:48.551 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:48 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/1663634611' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:55:48.551 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:48 smithi084 ceph-mon[29774]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:55:48.551 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:48 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/1663634611' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T22:55:48.551 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:48 smithi084 ceph-mon[29774]: osdmap e868: 8 total, 8 up, 8 in 2024-09-20T22:55:48.620 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:55:48.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:48.642 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:48.643 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:55:48.744 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:55:48.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:48 smithi154 ceph-mon[33264]: purged_snaps scrub starts 2024-09-20T22:55:48.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:48 smithi154 ceph-mon[33264]: purged_snaps scrub ok 2024-09-20T22:55:48.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:48 smithi154 ceph-mon[33264]: osdmap e867: 8 total, 8 up, 8 in 2024-09-20T22:55:48.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:48 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/1663634611' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:55:48.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:48 smithi154 ceph-mon[33264]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:55:48.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:48 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/1663634611' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T22:55:48.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:48 smithi154 ceph-mon[33264]: osdmap e868: 8 total, 8 up, 8 in 2024-09-20T22:55:48.765 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:48.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:48.766 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:55:48.849 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-20T22:55:48.866 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:55:48.913 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:48.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:48.914 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:55:49.014 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:55:49.036 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:49.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:49.037 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:55:49.138 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:55:49.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:49.165 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:49.166 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:55:49.266 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:55:49.292 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:49.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:49.293 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:55:49.345 INFO:tasks.rados.rados.0.smithi084.stdout:3461: done (1 left) 2024-09-20T22:55:49.345 INFO:tasks.rados.rados.0.smithi084.stdout:3462: rollback oid 14 current snap is 404 2024-09-20T22:55:49.345 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 14 to 397 2024-09-20T22:55:49.345 INFO:tasks.rados.rados.0.smithi084.stdout:3463: write oid 3 current snap is 404 2024-09-20T22:55:49.345 INFO:tasks.rados.rados.0.smithi084.stdout:3463: seq_num 1177 ranges {581839=486435,1642012=599733,3005686=569865} 2024-09-20T22:55:49.351 INFO:tasks.rados.rados.0.smithi084.stdout:3463: writing smithi08477203-3 from 581839 to 1068274 tid 1 2024-09-20T22:55:49.357 INFO:tasks.rados.rados.0.smithi084.stdout:3463: writing smithi08477203-3 from 1642012 to 2241745 tid 2 2024-09-20T22:55:49.365 INFO:tasks.rados.rados.0.smithi084.stdout:3463: writing smithi08477203-3 from 3005686 to 3575551 tid 3 2024-09-20T22:55:49.365 INFO:tasks.rados.rados.0.smithi084.stdout:3464: snap_create 2024-09-20T22:55:49.365 INFO:tasks.rados.rados.0.smithi084.stdout:3459: finishing rollback tid 0 to smithi08477203-35 2024-09-20T22:55:49.365 INFO:tasks.rados.rados.0.smithi084.stdout:3459: finishing rollback tid 1 to smithi08477203-35 2024-09-20T22:55:49.365 INFO:tasks.rados.rados.0.smithi084.stdout:3459: finishing rollback tid 2 to smithi08477203-35 2024-09-20T22:55:49.365 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 35 v 2642 (ObjNum 1121 snap 380 seq_num 1121) dirty exists 2024-09-20T22:55:49.365 INFO:tasks.rados.rados.0.smithi084.stdout:3462: finishing rollback tid 0 to smithi08477203-14 2024-09-20T22:55:49.365 INFO:tasks.rados.rados.0.smithi084.stdout:3462: finishing rollback tid 1 to smithi08477203-14 2024-09-20T22:55:49.365 INFO:tasks.rados.rados.0.smithi084.stdout:3462: finishing rollback tid 2 to smithi08477203-14 2024-09-20T22:55:49.365 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 14 v 1945 (ObjNum 1154 snap 396 seq_num 1154) dirty exists 2024-09-20T22:55:49.365 INFO:tasks.rados.rados.0.smithi084.stdout:3463: finishing write tid 1 to smithi08477203-3 2024-09-20T22:55:49.366 INFO:tasks.rados.rados.0.smithi084.stdout:3463: finishing write tid 2 to smithi08477203-3 2024-09-20T22:55:49.367 INFO:tasks.rados.rados.0.smithi084.stdout:3463: finishing write tid 3 to smithi08477203-3 2024-09-20T22:55:49.367 INFO:tasks.rados.rados.0.smithi084.stdout:3463: finishing write tid 4 to smithi08477203-3 2024-09-20T22:55:49.368 INFO:tasks.rados.rados.0.smithi084.stdout:3463: finishing write tid 5 to smithi08477203-3 2024-09-20T22:55:49.368 INFO:tasks.rados.rados.0.smithi084.stdout:3463: oid 3 updating version 0 to 2581 2024-09-20T22:55:49.368 INFO:tasks.rados.rados.0.smithi084.stdout:3463: oid 3 version 2581 is already newer than 2579 2024-09-20T22:55:49.368 INFO:tasks.rados.rados.0.smithi084.stdout:3463: oid 3 updating version 2581 to 2582 2024-09-20T22:55:49.368 INFO:tasks.rados.rados.0.smithi084.stdout:3463: oid 3 version 2582 is already newer than 2580 2024-09-20T22:55:49.368 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 3 v 2582 (ObjNum 1177 snap 404 seq_num 1177) dirty exists 2024-09-20T22:55:49.368 INFO:tasks.rados.rados.0.smithi084.stdout:3463: left oid 3 (ObjNum 1177 snap 404 seq_num 1177) 2024-09-20T22:55:49.394 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:55:49.415 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:49.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:49.417 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:55:49.518 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:55:49.539 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:49.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:49.540 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:55:49.641 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:55:49.662 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:49.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:49.664 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:55:49.674 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:49 smithi084 ceph-mon[40024]: pgmap v1203: 158 pgs: 2 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 152 active+clean; 618 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail 2024-09-20T22:55:49.674 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:49 smithi084 ceph-mon[40024]: 6.2 scrub starts 2024-09-20T22:55:49.674 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:49 smithi084 ceph-mon[40024]: 6.2 scrub ok 2024-09-20T22:55:49.674 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:49 smithi084 ceph-mon[40024]: osdmap e869: 8 total, 8 up, 8 in 2024-09-20T22:55:49.674 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:49 smithi084 ceph-mon[29774]: pgmap v1203: 158 pgs: 2 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 152 active+clean; 618 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail 2024-09-20T22:55:49.674 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:49 smithi084 ceph-mon[29774]: 6.2 scrub starts 2024-09-20T22:55:49.674 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:49 smithi084 ceph-mon[29774]: 6.2 scrub ok 2024-09-20T22:55:49.675 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:49 smithi084 ceph-mon[29774]: osdmap e869: 8 total, 8 up, 8 in 2024-09-20T22:55:49.691 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:49 smithi154 ceph-mon[33264]: pgmap v1203: 158 pgs: 2 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 152 active+clean; 618 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail 2024-09-20T22:55:49.691 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:49 smithi154 ceph-mon[33264]: 6.2 scrub starts 2024-09-20T22:55:49.691 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:49 smithi154 ceph-mon[33264]: 6.2 scrub ok 2024-09-20T22:55:49.691 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:49 smithi154 ceph-mon[33264]: osdmap e869: 8 total, 8 up, 8 in 2024-09-20T22:55:49.764 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:55:49.782 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-20T22:55:49.782 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:55:49.791 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:49.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:49.792 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:55:49.892 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:55:49.918 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:49.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:49.919 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:55:50.004 INFO:journalctl@ceph.iscsi.iscsi.a.smithi154.stdout:Sep 20 22:55:49 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-iscsi-iscsi-a[59730]: debug there is no tcmu-runner data available 2024-09-20T22:55:50.019 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:55:50.020 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:55:50.046 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:50.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:50.047 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:55:50.148 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:55:50.171 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:50.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:50.172 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:55:50.273 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:55:50.299 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:50.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:50.300 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:55:50.348 INFO:tasks.rados.rados.0.smithi084.stdout:3459: done (3 left) 2024-09-20T22:55:50.348 INFO:tasks.rados.rados.0.smithi084.stdout:3462: done (2 left) 2024-09-20T22:55:50.348 INFO:tasks.rados.rados.0.smithi084.stdout:3463: done (1 left) 2024-09-20T22:55:50.348 INFO:tasks.rados.rados.0.smithi084.stdout:3464: done (0 left) 2024-09-20T22:55:50.348 INFO:tasks.rados.rados.0.smithi084.stdout:3465: delete oid 1 current snap is 405 2024-09-20T22:55:50.354 INFO:tasks.rados.rados.0.smithi084.stdout:3465: done (0 left) 2024-09-20T22:55:50.354 INFO:tasks.rados.rados.0.smithi084.stdout:3466: write (excl) oid 47 current snap is 405 2024-09-20T22:55:50.354 INFO:tasks.rados.rados.0.smithi084.stdout:3466: seq_num 1179 ranges {615453=753286,2158976=603402,3460532=354921} 2024-09-20T22:55:50.361 INFO:tasks.rados.rados.0.smithi084.stdout:3466: writing smithi08477203-47 from 615453 to 1368739 tid 1 2024-09-20T22:55:50.370 INFO:tasks.rados.rados.0.smithi084.stdout:3466: writing smithi08477203-47 from 2158976 to 2762378 tid 2 2024-09-20T22:55:50.376 INFO:tasks.rados.rados.0.smithi084.stdout:3466: writing smithi08477203-47 from 3460532 to 3815453 tid 3 2024-09-20T22:55:50.376 INFO:tasks.rados.rados.0.smithi084.stdout:3467: write (excl) oid 32 current snap is 405 2024-09-20T22:55:50.376 INFO:tasks.rados.rados.0.smithi084.stdout:3466: finishing write tid 1 to smithi08477203-47 2024-09-20T22:55:50.376 INFO:tasks.rados.rados.0.smithi084.stdout:3467: seq_num 1180 ranges {632260=428535,1817458=463413,2687955=520034,3709324=122936} 2024-09-20T22:55:50.382 INFO:tasks.rados.rados.0.smithi084.stdout:3467: writing smithi08477203-32 from 632260 to 1060795 tid 1 2024-09-20T22:55:50.388 INFO:tasks.rados.rados.0.smithi084.stdout:3467: writing smithi08477203-32 from 1817458 to 2280871 tid 2 2024-09-20T22:55:50.394 INFO:tasks.rados.rados.0.smithi084.stdout:3467: writing smithi08477203-32 from 2687955 to 3207989 tid 3 2024-09-20T22:55:50.398 INFO:tasks.rados.rados.0.smithi084.stdout:3467: writing smithi08477203-32 from 3709324 to 3832260 tid 4 2024-09-20T22:55:50.398 INFO:tasks.rados.rados.0.smithi084.stdout:3468: snap_create 2024-09-20T22:55:50.398 INFO:tasks.rados.rados.0.smithi084.stdout:3466: finishing write tid 2 to smithi08477203-47 2024-09-20T22:55:50.398 INFO:tasks.rados.rados.0.smithi084.stdout:3466: finishing write tid 3 to smithi08477203-47 2024-09-20T22:55:50.399 INFO:tasks.rados.rados.0.smithi084.stdout:3466: finishing write tid 4 to smithi08477203-47 2024-09-20T22:55:50.399 INFO:tasks.rados.rados.0.smithi084.stdout:3466: finishing write tid 5 to smithi08477203-47 2024-09-20T22:55:50.399 INFO:tasks.rados.rados.0.smithi084.stdout:3466: oid 47 updating version 0 to 2489 2024-09-20T22:55:50.399 INFO:tasks.rados.rados.0.smithi084.stdout:3466: oid 47 version 2489 is already newer than 2488 2024-09-20T22:55:50.399 INFO:tasks.rados.rados.0.smithi084.stdout:3466: oid 47 version 2489 is already newer than 2487 2024-09-20T22:55:50.399 INFO:tasks.rados.rados.0.smithi084.stdout:3466: oid 47 version 2489 is already newer than 2486 2024-09-20T22:55:50.399 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 47 v 2489 (ObjNum 1179 snap 405 seq_num 1179) dirty exists 2024-09-20T22:55:50.399 INFO:tasks.rados.rados.0.smithi084.stdout:3466: left oid 47 (ObjNum 1179 snap 405 seq_num 1179) 2024-09-20T22:55:50.399 INFO:tasks.rados.rados.0.smithi084.stdout:3467: finishing write tid 1 to smithi08477203-32 2024-09-20T22:55:50.399 INFO:tasks.rados.rados.0.smithi084.stdout:3467: finishing write tid 2 to smithi08477203-32 2024-09-20T22:55:50.399 INFO:tasks.rados.rados.0.smithi084.stdout:3467: finishing write tid 3 to smithi08477203-32 2024-09-20T22:55:50.400 INFO:tasks.rados.rados.0.smithi084.stdout:3467: finishing write tid 4 to smithi08477203-32 2024-09-20T22:55:50.400 INFO:tasks.rados.rados.0.smithi084.stdout:3467: finishing write tid 5 to smithi08477203-32 2024-09-20T22:55:50.400 INFO:tasks.rados.rados.0.smithi084.stdout:3467: finishing write tid 6 to smithi08477203-32 2024-09-20T22:55:50.400 INFO:tasks.rados.rados.0.smithi084.stdout:3467: oid 32 updating version 0 to 1847 2024-09-20T22:55:50.400 INFO:tasks.rados.rados.0.smithi084.stdout:3467: oid 32 version 1847 is already newer than 1845 2024-09-20T22:55:50.400 INFO:tasks.rados.rados.0.smithi084.stdout:3467: oid 32 updating version 1847 to 1849 2024-09-20T22:55:50.401 INFO:tasks.rados.rados.0.smithi084.stdout:3467: oid 32 version 1849 is already newer than 1848 2024-09-20T22:55:50.401 INFO:tasks.rados.rados.0.smithi084.stdout:3467: oid 32 version 1849 is already newer than 1846 2024-09-20T22:55:50.401 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 32 v 1849 (ObjNum 1180 snap 405 seq_num 1180) dirty exists 2024-09-20T22:55:50.401 INFO:tasks.rados.rados.0.smithi084.stdout:3467: left oid 32 (ObjNum 1180 snap 405 seq_num 1180) 2024-09-20T22:55:50.401 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:55:50.424 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:50.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:50.425 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:55:50.526 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:55:50.548 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:50.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:50.549 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:55:50.572 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.3 dump_blocked_ops 2024-09-20T22:55:50.649 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:55:50.672 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:50.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:50.673 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:55:50.696 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:50 smithi084 ceph-mon[40024]: 4.f scrub starts 2024-09-20T22:55:50.696 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:50 smithi084 ceph-mon[40024]: 4.f scrub ok 2024-09-20T22:55:50.697 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:50 smithi084 ceph-mon[40024]: osdmap e870: 8 total, 8 up, 8 in 2024-09-20T22:55:50.697 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:50 smithi084 ceph-mon[29774]: 4.f scrub starts 2024-09-20T22:55:50.697 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:50 smithi084 ceph-mon[29774]: 4.f scrub ok 2024-09-20T22:55:50.697 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:50 smithi084 ceph-mon[29774]: osdmap e870: 8 total, 8 up, 8 in 2024-09-20T22:55:50.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:50 smithi154 ceph-mon[33264]: 4.f scrub starts 2024-09-20T22:55:50.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:50 smithi154 ceph-mon[33264]: 4.f scrub ok 2024-09-20T22:55:50.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:50 smithi154 ceph-mon[33264]: osdmap e870: 8 total, 8 up, 8 in 2024-09-20T22:55:50.774 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:55:50.797 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:50.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:50.798 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:55:50.898 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:55:50.921 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:50.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:50.922 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:55:51.023 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:55:51.024 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:55:51.024 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:55:51.039 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:55:51.040 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:55:51.046 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:51.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:51.047 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:55:51.063 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:55:51.063 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:55:51.065 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:55:51.079 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-20T22:55:51.080 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T22:55:51.101 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:55:51.102 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:55:51.118 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:55:51.118 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:55:51.135 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:55:51.135 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:55:51.148 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:55:51.151 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:55:51.151 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:55:51.169 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:51.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:51.171 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:55:51.271 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:55:51.293 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:51.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:51.294 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:55:51.395 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:55:51.404 INFO:tasks.rados.rados.0.smithi084.stdout:3466: done (2 left) 2024-09-20T22:55:51.404 INFO:tasks.rados.rados.0.smithi084.stdout:3467: done (1 left) 2024-09-20T22:55:51.404 INFO:tasks.rados.rados.0.smithi084.stdout:3468: done (0 left) 2024-09-20T22:55:51.404 INFO:tasks.rados.rados.0.smithi084.stdout:3469: rollback oid 15 current snap is 406 2024-09-20T22:55:51.404 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 15 to 388 2024-09-20T22:55:51.404 INFO:tasks.rados.rados.0.smithi084.stdout:3470: snap_remove snap 327 2024-09-20T22:55:51.421 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:51.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:51.422 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:55:51.495 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-20T22:55:51.523 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:55:51.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:51.560 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:51.560 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:55:51.660 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:55:51.683 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:51.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:51.684 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:55:51.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:51 smithi154 ceph-mon[33264]: 7.f scrub starts 2024-09-20T22:55:51.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:51 smithi154 ceph-mon[33264]: 7.f scrub ok 2024-09-20T22:55:51.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:51 smithi154 ceph-mon[33264]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:55:51.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:51 smithi154 ceph-mon[33264]: pgmap v1206: 158 pgs: 2 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 152 active+clean; 629 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 2.2 KiB/s rd, 1.7 MiB/s wr, 9 op/s 2024-09-20T22:55:51.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:51 smithi154 ceph-mon[33264]: 3.13 scrub starts 2024-09-20T22:55:51.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:51 smithi154 ceph-mon[33264]: 3.13 scrub ok 2024-09-20T22:55:51.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:51 smithi154 ceph-mon[33264]: 6.7 scrub starts 2024-09-20T22:55:51.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:51 smithi154 ceph-mon[33264]: 6.7 scrub ok 2024-09-20T22:55:51.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:51 smithi154 ceph-mon[33264]: 1.7 scrub starts 2024-09-20T22:55:51.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:51 smithi154 ceph-mon[33264]: 1.7 scrub ok 2024-09-20T22:55:51.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:51 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/1464171579' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:55:51.786 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:55:51.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:51.823 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:51.823 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:55:51.823 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:51 smithi084 ceph-mon[29774]: 7.f scrub starts 2024-09-20T22:55:51.823 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:51 smithi084 ceph-mon[29774]: 7.f scrub ok 2024-09-20T22:55:51.823 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:51 smithi084 ceph-mon[29774]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:55:51.824 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:51 smithi084 ceph-mon[29774]: pgmap v1206: 158 pgs: 2 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 152 active+clean; 629 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 2.2 KiB/s rd, 1.7 MiB/s wr, 9 op/s 2024-09-20T22:55:51.824 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:51 smithi084 ceph-mon[29774]: 3.13 scrub starts 2024-09-20T22:55:51.824 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:51 smithi084 ceph-mon[29774]: 3.13 scrub ok 2024-09-20T22:55:51.824 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:51 smithi084 ceph-mon[29774]: 6.7 scrub starts 2024-09-20T22:55:51.824 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:51 smithi084 ceph-mon[29774]: 6.7 scrub ok 2024-09-20T22:55:51.824 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:51 smithi084 ceph-mon[29774]: 1.7 scrub starts 2024-09-20T22:55:51.824 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:51 smithi084 ceph-mon[29774]: 1.7 scrub ok 2024-09-20T22:55:51.824 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:51 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/1464171579' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:55:51.824 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:51 smithi084 ceph-mon[40024]: 7.f scrub starts 2024-09-20T22:55:51.824 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:51 smithi084 ceph-mon[40024]: 7.f scrub ok 2024-09-20T22:55:51.824 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:51 smithi084 ceph-mon[40024]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:55:51.824 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:51 smithi084 ceph-mon[40024]: pgmap v1206: 158 pgs: 2 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 152 active+clean; 629 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 2.2 KiB/s rd, 1.7 MiB/s wr, 9 op/s 2024-09-20T22:55:51.825 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:51 smithi084 ceph-mon[40024]: 3.13 scrub starts 2024-09-20T22:55:51.825 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:51 smithi084 ceph-mon[40024]: 3.13 scrub ok 2024-09-20T22:55:51.825 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:51 smithi084 ceph-mon[40024]: 6.7 scrub starts 2024-09-20T22:55:51.825 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:51 smithi084 ceph-mon[40024]: 6.7 scrub ok 2024-09-20T22:55:51.825 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:51 smithi084 ceph-mon[40024]: 1.7 scrub starts 2024-09-20T22:55:51.825 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:51 smithi084 ceph-mon[40024]: 1.7 scrub ok 2024-09-20T22:55:51.825 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:51 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/1464171579' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:55:51.923 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:55:51.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:51.959 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:51.959 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:55:52.062 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:55:52.097 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:52.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:52.098 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:55:52.198 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:55:52.227 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:52.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:52.228 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:55:52.329 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:55:52.356 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:52.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:52.358 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:55:52.424 INFO:tasks.rados.rados.0.smithi084.stdout:3470: done (1 left) 2024-09-20T22:55:52.424 INFO:tasks.rados.rados.0.smithi084.stdout:3471: snap_create 2024-09-20T22:55:52.424 INFO:tasks.rados.rados.0.smithi084.stdout:3469: finishing rollback tid 0 to smithi08477203-15 2024-09-20T22:55:52.424 INFO:tasks.rados.rados.0.smithi084.stdout:3469: finishing rollback tid 1 to smithi08477203-15 2024-09-20T22:55:52.424 INFO:tasks.rados.rados.0.smithi084.stdout:3469: finishing rollback tid 2 to smithi08477203-15 2024-09-20T22:55:52.425 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 15 v 2585 (ObjNum 1097 snap 368 seq_num 1097) dirty exists 2024-09-20T22:55:52.458 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:55:52.473 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:55:52.483 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-20T22:55:52.483 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2024-09-20T22:55:52.483 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2024-09-20T22:55:52.484 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2024-09-20T22:55:52.484 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-20T22:55:52.484 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:52 smithi084 ceph-mon[40024]: 5.3 scrub starts 2024-09-20T22:55:52.484 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:52 smithi084 ceph-mon[40024]: 5.3 scrub ok 2024-09-20T22:55:52.484 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:52 smithi084 ceph-mon[40024]: osdmap e871: 8 total, 8 up, 8 in 2024-09-20T22:55:52.484 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:52 smithi084 ceph-mon[40024]: 6.6 scrub starts 2024-09-20T22:55:52.485 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:52 smithi084 ceph-mon[40024]: 6.6 scrub ok 2024-09-20T22:55:52.485 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:52 smithi084 ceph-mon[29774]: 5.3 scrub starts 2024-09-20T22:55:52.485 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:52 smithi084 ceph-mon[29774]: 5.3 scrub ok 2024-09-20T22:55:52.485 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:52 smithi084 ceph-mon[29774]: osdmap e871: 8 total, 8 up, 8 in 2024-09-20T22:55:52.485 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:52 smithi084 ceph-mon[29774]: 6.6 scrub starts 2024-09-20T22:55:52.485 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:52 smithi084 ceph-mon[29774]: 6.6 scrub ok 2024-09-20T22:55:52.488 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:52.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:52.490 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:55:52.590 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:55:52.617 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:52.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:52.618 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:55:52.719 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:55:52.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:52.747 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:52.748 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:55:52.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:52 smithi154 ceph-mon[33264]: 5.3 scrub starts 2024-09-20T22:55:52.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:52 smithi154 ceph-mon[33264]: 5.3 scrub ok 2024-09-20T22:55:52.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:52 smithi154 ceph-mon[33264]: osdmap e871: 8 total, 8 up, 8 in 2024-09-20T22:55:52.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:52 smithi154 ceph-mon[33264]: 6.6 scrub starts 2024-09-20T22:55:52.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:52 smithi154 ceph-mon[33264]: 6.6 scrub ok 2024-09-20T22:55:52.848 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:55:52.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:52.875 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:52.875 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:55:52.976 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:55:53.002 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:53.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:53.003 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:55:53.089 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.3 dump_historic_ops 2024-09-20T22:55:53.104 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:55:53.156 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:53.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:53.157 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:55:53.258 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:55:53.285 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:53.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:53.286 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:55:53.387 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:55:53.407 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:53.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:53.408 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:55:53.420 INFO:tasks.rados.rados.0.smithi084.stdout:3469: done (1 left) 2024-09-20T22:55:53.421 INFO:tasks.rados.rados.0.smithi084.stdout:3471: done (0 left) 2024-09-20T22:55:53.421 INFO:tasks.rados.rados.0.smithi084.stdout:3472: write (excl) oid 40 current snap is 407 2024-09-20T22:55:53.421 INFO:tasks.rados.rados.0.smithi084.stdout:3472: seq_num 1181 ranges {649067=503784,1875940=607071,3115378=731306} 2024-09-20T22:55:53.426 INFO:tasks.rados.rados.0.smithi084.stdout:3472: writing smithi08477203-40 from 649067 to 1152851 tid 1 2024-09-20T22:55:53.434 INFO:tasks.rados.rados.0.smithi084.stdout:3472: writing smithi08477203-40 from 1875940 to 2483011 tid 2 2024-09-20T22:55:53.442 INFO:tasks.rados.rados.0.smithi084.stdout:3472: writing smithi08477203-40 from 3115378 to 3846684 tid 3 2024-09-20T22:55:53.442 INFO:tasks.rados.rados.0.smithi084.stdout:3473: delete oid 47 current snap is 407 2024-09-20T22:55:53.446 INFO:tasks.rados.rados.0.smithi084.stdout:3472: finishing write tid 1 to smithi08477203-40 2024-09-20T22:55:53.446 INFO:tasks.rados.rados.0.smithi084.stdout:3472: finishing write tid 2 to smithi08477203-40 2024-09-20T22:55:53.446 INFO:tasks.rados.rados.0.smithi084.stdout:3472: finishing write tid 3 to smithi08477203-40 2024-09-20T22:55:53.447 INFO:tasks.rados.rados.0.smithi084.stdout:3472: finishing write tid 4 to smithi08477203-40 2024-09-20T22:55:53.447 INFO:tasks.rados.rados.0.smithi084.stdout:3472: finishing write tid 5 to smithi08477203-40 2024-09-20T22:55:53.447 INFO:tasks.rados.rados.0.smithi084.stdout:3472: oid 40 updating version 0 to 2729 2024-09-20T22:55:53.447 INFO:tasks.rados.rados.0.smithi084.stdout:3472: oid 40 updating version 2729 to 2730 2024-09-20T22:55:53.448 INFO:tasks.rados.rados.0.smithi084.stdout:3472: oid 40 updating version 2730 to 2731 2024-09-20T22:55:53.448 INFO:tasks.rados.rados.0.smithi084.stdout:3472: oid 40 updating version 2731 to 2732 2024-09-20T22:55:53.448 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 40 v 2732 (ObjNum 1181 snap 407 seq_num 1181) dirty exists 2024-09-20T22:55:53.448 INFO:tasks.rados.rados.0.smithi084.stdout:3472: left oid 40 (ObjNum 1181 snap 407 seq_num 1181) 2024-09-20T22:55:53.448 INFO:tasks.rados.rados.0.smithi084.stdout:3472: done (1 left) 2024-09-20T22:55:53.448 INFO:tasks.rados.rados.0.smithi084.stdout:3473: done (0 left) 2024-09-20T22:55:53.448 INFO:tasks.rados.rados.0.smithi084.stdout:3474: read oid 16 snap -1 2024-09-20T22:55:53.448 INFO:tasks.rados.rados.0.smithi084.stdout:3474: expect (ObjNum 1143 snap 389 seq_num 1143) 2024-09-20T22:55:53.448 INFO:tasks.rados.rados.0.smithi084.stdout:3475: delete oid 13 current snap is 407 2024-09-20T22:55:53.452 INFO:tasks.rados.rados.0.smithi084.stdout:3475: done (1 left) 2024-09-20T22:55:53.452 INFO:tasks.rados.rados.0.smithi084.stdout:3476: snap_remove snap 400 2024-09-20T22:55:53.510 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:55:53.532 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:53.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:53.533 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:55:53.579 INFO:tasks.thrashosds.thrasher:in_osds: [2, 3, 5, 6, 7, 0, 4, 1] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-20T22:55:53.579 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-20T22:55:53.579 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-09-20T22:55:53.579 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd dump -f json-pretty 2024-09-20T22:55:53.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:53 smithi084 ceph-mon[40024]: pgmap v1209: 158 pgs: 2 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 152 active+clean; 629 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 2.2 KiB/s rd, 1.7 MiB/s wr, 9 op/s 2024-09-20T22:55:53.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:53 smithi084 ceph-mon[40024]: purged_snaps scrub starts 2024-09-20T22:55:53.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:53 smithi084 ceph-mon[40024]: purged_snaps scrub ok 2024-09-20T22:55:53.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:53 smithi084 ceph-mon[40024]: osdmap e872: 8 total, 8 up, 8 in 2024-09-20T22:55:53.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:53 smithi084 ceph-mon[40024]: 5.17 scrub starts 2024-09-20T22:55:53.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:53 smithi084 ceph-mon[40024]: 5.17 scrub ok 2024-09-20T22:55:53.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:53 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:55:53.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:53 smithi084 ceph-mon[29774]: pgmap v1209: 158 pgs: 2 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 152 active+clean; 629 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 2.2 KiB/s rd, 1.7 MiB/s wr, 9 op/s 2024-09-20T22:55:53.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:53 smithi084 ceph-mon[29774]: purged_snaps scrub starts 2024-09-20T22:55:53.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:53 smithi084 ceph-mon[29774]: purged_snaps scrub ok 2024-09-20T22:55:53.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:53 smithi084 ceph-mon[29774]: osdmap e872: 8 total, 8 up, 8 in 2024-09-20T22:55:53.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:53 smithi084 ceph-mon[29774]: 5.17 scrub starts 2024-09-20T22:55:53.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:53 smithi084 ceph-mon[29774]: 5.17 scrub ok 2024-09-20T22:55:53.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:53 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:55:53.634 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:55:53.656 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:53.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:53.657 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:55:53.686 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:55:53.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:53 smithi154 ceph-mon[33264]: pgmap v1209: 158 pgs: 2 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 152 active+clean; 629 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 2.2 KiB/s rd, 1.7 MiB/s wr, 9 op/s 2024-09-20T22:55:53.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:53 smithi154 ceph-mon[33264]: purged_snaps scrub starts 2024-09-20T22:55:53.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:53 smithi154 ceph-mon[33264]: purged_snaps scrub ok 2024-09-20T22:55:53.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:53 smithi154 ceph-mon[33264]: osdmap e872: 8 total, 8 up, 8 in 2024-09-20T22:55:53.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:53 smithi154 ceph-mon[33264]: 5.17 scrub starts 2024-09-20T22:55:53.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:53 smithi154 ceph-mon[33264]: 5.17 scrub ok 2024-09-20T22:55:53.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:53 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:55:53.758 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:55:53.782 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:53.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:53.783 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:55:53.884 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:55:53.908 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:53.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:53.909 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:55:54.010 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:55:54.031 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:54.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:54.032 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:55:54.133 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:55:54.144 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-20T22:55:54.144 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2024-09-20T22:55:54.144 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2024-09-20T22:55:54.144 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [ 2024-09-20T22:55:54.144 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.144 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:2525 7.1 7:8422f82d:::smithi08477203-22:head [copy-from ver 174] snapc 5d=[5d,5c,5a,50,47] ondisk+write+known_if_redirected+supports_pool_eio e278)", 2024-09-20T22:55:54.144 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:45:57.170348+0000", 2024-09-20T22:55:54.145 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 596.969397577, 2024-09-20T22:55:54.145 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.023333486000000001, 2024-09-20T22:55:54.145 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:55:54.145 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:55:54.145 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:55:54.145 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:55:54.145 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:55:54.145 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:55:54.145 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 2525 2024-09-20T22:55:54.145 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.145 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:55:54.145 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.145 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:55:54.146 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:57.170348+0000", 2024-09-20T22:55:54.146 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:55:54.146 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.146 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.146 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:55:54.146 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:57.170348+0000", 2024-09-20T22:55:54.146 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:55:54.146 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.146 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.146 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:55:54.146 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:57.170350+0000", 2024-09-20T22:55:54.146 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.1509999999999998e-06 2024-09-20T22:55:54.147 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.147 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.147 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:55:54.147 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:57.170361+0000", 2024-09-20T22:55:54.147 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0492999999999999e-05 2024-09-20T22:55:54.147 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.147 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.147 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:55:54.147 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:57.170381+0000", 2024-09-20T22:55:54.147 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.0509999999999998e-05 2024-09-20T22:55:54.147 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.147 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.147 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:55:54.147 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:57.170393+0000", 2024-09-20T22:55:54.148 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2009e-05 2024-09-20T22:55:54.148 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.148 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.148 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:55:54.148 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:57.170465+0000", 2024-09-20T22:55:54.148 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.1927000000000004e-05 2024-09-20T22:55:54.148 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.148 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.148 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op must wait for map", 2024-09-20T22:55:54.148 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:57.170473+0000", 2024-09-20T22:55:54.148 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.1890000000000007e-06 2024-09-20T22:55:54.148 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.148 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.148 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:55:54.149 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:57.172855+0000", 2024-09-20T22:55:54.149 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002382168 2024-09-20T22:55:54.149 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.149 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.149 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:55:54.149 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:57.172967+0000", 2024-09-20T22:55:54.149 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00011108299999999999 2024-09-20T22:55:54.149 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.149 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.149 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 1,6", 2024-09-20T22:55:54.149 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:57.181533+0000", 2024-09-20T22:55:54.149 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0085667499999999997 2024-09-20T22:55:54.150 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.150 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.150 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:55:54.150 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:57.191599+0000", 2024-09-20T22:55:54.150 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010065795000000001 2024-09-20T22:55:54.150 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.150 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.150 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:55:54.150 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:57.193469+0000", 2024-09-20T22:55:54.150 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0018695199999999999 2024-09-20T22:55:54.150 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.150 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.150 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:55:54.150 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:57.193550+0000", 2024-09-20T22:55:54.151 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.1891000000000006e-05 2024-09-20T22:55:54.151 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.151 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.151 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:55:54.151 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:57.193625+0000", 2024-09-20T22:55:54.151 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.4679999999999996e-05 2024-09-20T22:55:54.151 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.151 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.151 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:55:54.151 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:45:57.193681+0000", 2024-09-20T22:55:54.151 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.6320000000000003e-05 2024-09-20T22:55:54.151 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:55:54.151 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:55:54.151 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:55:54.152 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.152 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.152 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:2853 7.10 7:0f524219:::smithi08477203-50:head [copy-from ver 114] snapc 68=[68,67] ondisk+write+known_if_redirected+supports_pool_eio e303)", 2024-09-20T22:55:54.152 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:46:23.285507+0000", 2024-09-20T22:55:54.152 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 570.85423884800002, 2024-09-20T22:55:54.152 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.023440631, 2024-09-20T22:55:54.152 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:55:54.152 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:55:54.152 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:55:54.152 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:55:54.152 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:55:54.152 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:55:54.152 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 2853 2024-09-20T22:55:54.152 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.152 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:55:54.153 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.153 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:55:54.153 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.285507+0000", 2024-09-20T22:55:54.153 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:55:54.153 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.153 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.153 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:55:54.153 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.285507+0000", 2024-09-20T22:55:54.153 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:55:54.153 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.153 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.153 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:55:54.153 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.285508+0000", 2024-09-20T22:55:54.153 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.491e-06 2024-09-20T22:55:54.154 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.154 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.154 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:55:54.154 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.285514+0000", 2024-09-20T22:55:54.154 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.502e-06 2024-09-20T22:55:54.154 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.154 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.154 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:55:54.154 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.285526+0000", 2024-09-20T22:55:54.154 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2639e-05 2024-09-20T22:55:54.154 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.154 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.154 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:55:54.154 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.285535+0000", 2024-09-20T22:55:54.155 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.4689999999999997e-06 2024-09-20T22:55:54.155 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.155 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.155 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:55:54.155 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.285591+0000", 2024-09-20T22:55:54.156 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.6407999999999999e-05 2024-09-20T22:55:54.156 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.156 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.156 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:55:54.156 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.285681+0000", 2024-09-20T22:55:54.156 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.9845000000000005e-05 2024-09-20T22:55:54.156 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.156 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.156 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 1,2", 2024-09-20T22:55:54.156 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.293719+0000", 2024-09-20T22:55:54.156 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0080383950000000003 2024-09-20T22:55:54.156 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.156 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.156 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:55:54.156 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.307398+0000", 2024-09-20T22:55:54.157 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013678074 2024-09-20T22:55:54.157 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.157 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.157 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:55:54.157 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.308451+0000", 2024-09-20T22:55:54.157 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001052975 2024-09-20T22:55:54.157 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.157 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.157 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:55:54.157 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.308811+0000", 2024-09-20T22:55:54.157 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00036041399999999998 2024-09-20T22:55:54.157 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.157 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.158 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:55:54.158 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.308878+0000", 2024-09-20T22:55:54.158 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.7114999999999997e-05 2024-09-20T22:55:54.158 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.158 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.158 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:55:54.158 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:23.308947+0000", 2024-09-20T22:55:54.158 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.9303999999999999e-05 2024-09-20T22:55:54.158 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:55:54.158 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:55:54.158 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:55:54.158 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.158 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.159 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:3136 7.a 7:5122292a:::smithi08477203-33:head [copy-from ver 385] snapc 77=[77,76,75,74,73,70,6b] ondisk+write+known_if_redirected+supports_pool_eio e328)", 2024-09-20T22:55:54.159 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:46:48.210350+0000", 2024-09-20T22:55:54.159 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 545.92939571700003, 2024-09-20T22:55:54.159 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.030150626999999999, 2024-09-20T22:55:54.159 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:55:54.159 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:55:54.159 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:55:54.159 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:55:54.159 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:55:54.159 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:55:54.159 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 3136 2024-09-20T22:55:54.159 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.159 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:55:54.159 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.160 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:55:54.160 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.210350+0000", 2024-09-20T22:55:54.160 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:55:54.160 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.160 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.160 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:55:54.160 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.210350+0000", 2024-09-20T22:55:54.160 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:55:54.160 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.160 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.160 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:55:54.160 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.210352+0000", 2024-09-20T22:55:54.160 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.889e-06 2024-09-20T22:55:54.160 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.160 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.161 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:55:54.161 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.210360+0000", 2024-09-20T22:55:54.161 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.8890000000000002e-06 2024-09-20T22:55:54.161 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.161 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.161 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:55:54.161 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.210372+0000", 2024-09-20T22:55:54.161 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2662e-05 2024-09-20T22:55:54.161 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.161 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.161 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:55:54.161 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.210381+0000", 2024-09-20T22:55:54.161 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.8589999999999994e-06 2024-09-20T22:55:54.161 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.162 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.162 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:55:54.162 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.210470+0000", 2024-09-20T22:55:54.162 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.9034000000000002e-05 2024-09-20T22:55:54.162 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.162 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.162 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:55:54.162 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.210568+0000", 2024-09-20T22:55:54.162 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.7919999999999998e-05 2024-09-20T22:55:54.162 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.162 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.162 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 2,5", 2024-09-20T22:55:54.162 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.218491+0000", 2024-09-20T22:55:54.162 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0079229490000000003 2024-09-20T22:55:54.163 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.163 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.163 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:55:54.163 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.224053+0000", 2024-09-20T22:55:54.163 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0055624330000000003 2024-09-20T22:55:54.163 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.163 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.163 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:55:54.163 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.227143+0000", 2024-09-20T22:55:54.163 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0030896360000000002 2024-09-20T22:55:54.163 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.163 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.163 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:55:54.164 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.240355+0000", 2024-09-20T22:55:54.164 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013211624999999999 2024-09-20T22:55:54.164 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.164 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.164 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:55:54.164 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.240439+0000", 2024-09-20T22:55:54.164 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.4009000000000002e-05 2024-09-20T22:55:54.164 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.164 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.164 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:55:54.164 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:48.240500+0000", 2024-09-20T22:55:54.164 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.1722000000000003e-05 2024-09-20T22:55:54.164 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:55:54.164 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:55:54.165 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:55:54.165 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.165 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.165 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:3235 7.1 7:84a866f5:::smithi08477203-27:head [copy-from ver 114] snapc 7e=[7e,7d,7c,7b,7a,79,78,75,74,73,6b] ondisk+write+known_if_redirected+supports_pool_eio e338)", 2024-09-20T22:55:54.165 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:46:58.180892+0000", 2024-09-20T22:55:54.165 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 535.95885349399998, 2024-09-20T22:55:54.165 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.044574093000000002, 2024-09-20T22:55:54.165 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:55:54.165 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:55:54.165 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:55:54.165 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:55:54.165 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:55:54.165 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:55:54.165 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 3235 2024-09-20T22:55:54.165 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.166 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:55:54.166 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.166 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:55:54.166 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.180892+0000", 2024-09-20T22:55:54.166 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:55:54.166 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.166 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.166 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:55:54.166 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.180892+0000", 2024-09-20T22:55:54.166 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:55:54.166 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.166 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.166 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:55:54.166 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.180894+0000", 2024-09-20T22:55:54.167 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.1789999999999998e-06 2024-09-20T22:55:54.167 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.167 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.167 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:55:54.167 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.180915+0000", 2024-09-20T22:55:54.167 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.0859000000000001e-05 2024-09-20T22:55:54.167 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.167 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.167 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:55:54.167 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.180937+0000", 2024-09-20T22:55:54.167 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.2357999999999998e-05 2024-09-20T22:55:54.167 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.167 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.167 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:55:54.167 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.180947+0000", 2024-09-20T22:55:54.168 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.8090000000000004e-06 2024-09-20T22:55:54.168 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.168 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.168 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:55:54.168 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.181042+0000", 2024-09-20T22:55:54.168 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.4327999999999995e-05 2024-09-20T22:55:54.168 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.168 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.168 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op must wait for map", 2024-09-20T22:55:54.168 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.181049+0000", 2024-09-20T22:55:54.168 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.1640000000000004e-06 2024-09-20T22:55:54.168 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.168 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.168 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:55:54.168 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.185933+0000", 2024-09-20T22:55:54.169 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.004884645 2024-09-20T22:55:54.169 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.169 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.169 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:55:54.169 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.186177+0000", 2024-09-20T22:55:54.169 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000243107 2024-09-20T22:55:54.169 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.169 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.169 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 1,6", 2024-09-20T22:55:54.169 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.204218+0000", 2024-09-20T22:55:54.169 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.018041805000000001 2024-09-20T22:55:54.169 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.169 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.169 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:55:54.170 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.218275+0000", 2024-09-20T22:55:54.170 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014056528 2024-09-20T22:55:54.170 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.170 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.170 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:55:54.170 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.224283+0000", 2024-09-20T22:55:54.170 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0060084860000000004 2024-09-20T22:55:54.170 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.170 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.170 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:55:54.170 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.225379+0000", 2024-09-20T22:55:54.170 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010960830000000001 2024-09-20T22:55:54.170 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.170 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.171 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:55:54.171 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.225429+0000", 2024-09-20T22:55:54.171 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.9440000000000001e-05 2024-09-20T22:55:54.171 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.171 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.171 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:55:54.171 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.225466+0000", 2024-09-20T22:55:54.171 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.7302000000000003e-05 2024-09-20T22:55:54.171 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:55:54.171 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:55:54.171 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:55:54.171 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.171 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.171 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:3236 7.1 7:84a866f5:::smithi08477203-27:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e338)", 2024-09-20T22:55:54.171 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:46:58.180973+0000", 2024-09-20T22:55:54.172 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 535.95877287200005, 2024-09-20T22:55:54.172 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.028358540000000002, 2024-09-20T22:55:54.172 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:55:54.172 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:55:54.172 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-20T22:55:54.172 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:55:54.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:54.174 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:54.174 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:55:54.174 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:55:54.174 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 3236 2024-09-20T22:55:54.174 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.174 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:55:54.174 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.175 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:55:54.175 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.180973+0000", 2024-09-20T22:55:54.175 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:55:54.175 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.175 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.175 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:55:54.175 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.180973+0000", 2024-09-20T22:55:54.175 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:55:54.175 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.175 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.175 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:55:54.176 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.180973+0000", 2024-09-20T22:55:54.176 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.9100000000000004e-07 2024-09-20T22:55:54.176 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.176 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.176 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:55:54.176 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.180976+0000", 2024-09-20T22:55:54.176 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.118e-06 2024-09-20T22:55:54.176 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.176 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.176 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:55:54.176 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.180984+0000", 2024-09-20T22:55:54.176 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.43e-06 2024-09-20T22:55:54.176 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.177 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.177 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:55:54.177 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.180990+0000", 2024-09-20T22:55:54.177 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.1109999999999996e-06 2024-09-20T22:55:54.177 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.177 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.177 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:55:54.177 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.181102+0000", 2024-09-20T22:55:54.177 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000112193 2024-09-20T22:55:54.177 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.177 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.177 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting_for_map not empty", 2024-09-20T22:55:54.178 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.181107+0000", 2024-09-20T22:55:54.178 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.0490000000000003e-06 2024-09-20T22:55:54.178 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.178 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.178 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:55:54.178 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.186302+0000", 2024-09-20T22:55:54.178 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0051950240000000003 2024-09-20T22:55:54.178 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.178 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.178 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for blocked object", 2024-09-20T22:55:54.178 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.186344+0000", 2024-09-20T22:55:54.178 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.2168e-05 2024-09-20T22:55:54.178 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.179 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.179 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:55:54.179 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.209115+0000", 2024-09-20T22:55:54.179 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.022770291000000002 2024-09-20T22:55:54.179 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.179 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.179 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:55:54.179 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.209217+0000", 2024-09-20T22:55:54.179 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000102192 2024-09-20T22:55:54.179 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.179 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.179 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:55:54.179 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:46:58.209331+0000", 2024-09-20T22:55:54.179 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000114373 2024-09-20T22:55:54.180 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:55:54.180 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:55:54.180 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:55:54.180 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.180 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.180 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:4534 7.12 7:4b06cbed:::smithi08477203-49:head [copy-from ver 722] snapc bb=[bb,ba,b9,b8,b7,b5,b4,b2,b1,a7,a6,a5,9e,9c,95,87,86] ondisk+write+known_if_redirected+supports_pool_eio e456)", 2024-09-20T22:55:54.180 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:48:56.230481+0000", 2024-09-20T22:55:54.180 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 417.90926482600003, 2024-09-20T22:55:54.180 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.028199295999999999, 2024-09-20T22:55:54.180 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:55:54.180 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:55:54.180 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:55:54.180 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:55:54.180 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:55:54.180 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:55:54.181 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 4534 2024-09-20T22:55:54.181 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.181 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:55:54.181 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.181 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:55:54.181 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:56.230481+0000", 2024-09-20T22:55:54.181 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:55:54.181 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.181 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.181 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:55:54.181 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:56.230481+0000", 2024-09-20T22:55:54.181 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:55:54.181 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.181 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.181 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:55:54.182 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:56.230482+0000", 2024-09-20T22:55:54.182 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.674e-06 2024-09-20T22:55:54.182 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.182 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.182 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:55:54.182 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:56.230497+0000", 2024-09-20T22:55:54.182 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.4178e-05 2024-09-20T22:55:54.182 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.182 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.182 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:55:54.182 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:56.230517+0000", 2024-09-20T22:55:54.182 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.0336e-05 2024-09-20T22:55:54.182 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.182 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.183 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:55:54.183 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:56.230527+0000", 2024-09-20T22:55:54.183 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.6039999999999999e-06 2024-09-20T22:55:54.183 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.183 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.183 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:55:54.183 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:56.230631+0000", 2024-09-20T22:55:54.183 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00010445 2024-09-20T22:55:54.183 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.183 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.183 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op must wait for map", 2024-09-20T22:55:54.183 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:56.230639+0000", 2024-09-20T22:55:54.183 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.7759999999999997e-06 2024-09-20T22:55:54.183 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.183 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.184 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:55:54.184 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:56.237931+0000", 2024-09-20T22:55:54.184 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0072925630000000002 2024-09-20T22:55:54.184 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.184 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.184 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:55:54.184 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:56.238030+0000", 2024-09-20T22:55:54.184 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.8592999999999996e-05 2024-09-20T22:55:54.184 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.184 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.184 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 1,2", 2024-09-20T22:55:54.184 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:56.244691+0000", 2024-09-20T22:55:54.184 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0066614680000000003 2024-09-20T22:55:54.184 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.184 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.185 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:55:54.185 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:56.258211+0000", 2024-09-20T22:55:54.185 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013519721 2024-09-20T22:55:54.185 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.185 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.185 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:55:54.185 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:56.258404+0000", 2024-09-20T22:55:54.185 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00019257000000000001 2024-09-20T22:55:54.185 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.185 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.185 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:55:54.185 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:56.258595+0000", 2024-09-20T22:55:54.185 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000191387 2024-09-20T22:55:54.185 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.185 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.186 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:55:54.186 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:56.258635+0000", 2024-09-20T22:55:54.186 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.9918e-05 2024-09-20T22:55:54.186 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.186 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.186 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:55:54.186 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:56.258680+0000", 2024-09-20T22:55:54.186 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.5058000000000001e-05 2024-09-20T22:55:54.186 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:55:54.186 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:55:54.186 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:55:54.186 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.186 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.186 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:5677 7.2 7:4fa4b2b3:::smithi08477203-21:head [copy-from ver 1213] snapc fa=[fa,f9,f8,f7,f5,f4,f2,f1,ef,ed,ec,ea,e6,e4,e2,df,db,da,d7,d4,d3,ce,c5,c2,bf,b4,b2,a7] ondisk+write+known_if_redirected+supports_pool_eio e572)", 2024-09-20T22:55:54.187 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:50:52.384255+0000", 2024-09-20T22:55:54.187 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 301.75549007799998, 2024-09-20T22:55:54.187 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.027330337, 2024-09-20T22:55:54.187 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:55:54.187 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:55:54.187 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:55:54.187 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:55:54.187 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:55:54.187 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:55:54.187 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 5677 2024-09-20T22:55:54.187 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.187 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:55:54.187 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.187 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:55:54.188 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:52.384255+0000", 2024-09-20T22:55:54.188 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:55:54.188 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.188 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.188 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:55:54.188 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:52.384255+0000", 2024-09-20T22:55:54.188 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:55:54.188 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.188 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.188 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:55:54.188 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:52.384258+0000", 2024-09-20T22:55:54.188 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.0600000000000002e-06 2024-09-20T22:55:54.188 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.188 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.189 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:55:54.189 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:52.384267+0000", 2024-09-20T22:55:54.189 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.3519999999999996e-06 2024-09-20T22:55:54.189 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.189 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.189 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:55:54.189 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:52.384288+0000", 2024-09-20T22:55:54.189 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.1039e-05 2024-09-20T22:55:54.189 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.189 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.189 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:55:54.189 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:52.384298+0000", 2024-09-20T22:55:54.189 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.6630000000000005e-06 2024-09-20T22:55:54.189 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.189 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.190 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:55:54.190 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:52.384381+0000", 2024-09-20T22:55:54.190 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.3372000000000001e-05 2024-09-20T22:55:54.190 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.190 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.190 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:55:54.190 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:52.384455+0000", 2024-09-20T22:55:54.190 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.4414999999999998e-05 2024-09-20T22:55:54.190 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.190 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.190 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 1,2", 2024-09-20T22:55:54.190 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:52.398437+0000", 2024-09-20T22:55:54.190 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013981959 2024-09-20T22:55:54.191 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.191 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.191 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:55:54.191 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:52.410532+0000", 2024-09-20T22:55:54.191 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.012094898 2024-09-20T22:55:54.191 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.191 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.191 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:55:54.191 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:52.411064+0000", 2024-09-20T22:55:54.191 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00053131999999999995 2024-09-20T22:55:54.191 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.191 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.191 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:55:54.192 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:52.411446+0000", 2024-09-20T22:55:54.192 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00038282300000000002 2024-09-20T22:55:54.192 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.192 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.192 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:55:54.192 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:52.411525+0000", 2024-09-20T22:55:54.192 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.8339999999999999e-05 2024-09-20T22:55:54.192 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.192 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.192 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:55:54.192 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:52.411586+0000", 2024-09-20T22:55:54.192 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.1095999999999999e-05 2024-09-20T22:55:54.192 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:55:54.192 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:55:54.193 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:55:54.193 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.193 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.193 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:6378 7.2 7:4d20232a:::smithi08477203-34:head [copy-from ver 1800] snapc 11a=[11a,119,118,117,116,115,114,113,112,111,110,10e,10c,107,105,104,103,fd,fb,fa,f8,f5,f2,ed,d7,c2,bf] ondisk+write+known_if_redirected+supports_pool_eio e639)", 2024-09-20T22:55:54.193 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:51:59.270245+0000", 2024-09-20T22:55:54.193 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 234.86950021300001, 2024-09-20T22:55:54.193 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.024049938999999999, 2024-09-20T22:55:54.193 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:55:54.193 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:55:54.193 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:55:54.193 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:55:54.193 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:55:54.193 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:55:54.193 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 6378 2024-09-20T22:55:54.194 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.194 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:55:54.194 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.194 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:55:54.194 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:59.270245+0000", 2024-09-20T22:55:54.194 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:55:54.194 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.194 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.194 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:55:54.194 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:59.270245+0000", 2024-09-20T22:55:54.194 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:55:54.194 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.194 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.195 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:55:54.195 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:59.270248+0000", 2024-09-20T22:55:54.195 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.8159999999999998e-06 2024-09-20T22:55:54.195 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.195 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.195 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:55:54.195 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:59.270262+0000", 2024-09-20T22:55:54.195 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3429e-05 2024-09-20T22:55:54.195 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.195 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.195 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:55:54.195 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:59.270279+0000", 2024-09-20T22:55:54.195 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.7269e-05 2024-09-20T22:55:54.195 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.196 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.196 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:55:54.196 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:59.270288+0000", 2024-09-20T22:55:54.196 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.3610000000000007e-06 2024-09-20T22:55:54.196 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.196 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.196 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:55:54.196 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:59.270383+0000", 2024-09-20T22:55:54.196 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.5199999999999997e-05 2024-09-20T22:55:54.196 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.196 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.196 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op must wait for map", 2024-09-20T22:55:54.196 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:59.270393+0000", 2024-09-20T22:55:54.196 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.4190000000000008e-06 2024-09-20T22:55:54.197 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.197 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.197 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:55:54.197 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:59.272509+0000", 2024-09-20T22:55:54.197 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0021163839999999998 2024-09-20T22:55:54.197 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.197 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.197 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:55:54.197 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:59.272677+0000", 2024-09-20T22:55:54.197 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00016751 2024-09-20T22:55:54.197 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.197 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.197 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 1,2,7", 2024-09-20T22:55:54.198 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:59.278535+0000", 2024-09-20T22:55:54.198 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0058579849999999996 2024-09-20T22:55:54.198 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.198 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.198 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:55:54.198 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:59.285175+0000", 2024-09-20T22:55:54.198 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0066406829999999997 2024-09-20T22:55:54.198 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.198 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.198 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:55:54.198 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:59.293955+0000", 2024-09-20T22:55:54.198 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0087799990000000001 2024-09-20T22:55:54.198 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.199 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.199 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:55:54.199 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:59.294123+0000", 2024-09-20T22:55:54.199 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00016759699999999999 2024-09-20T22:55:54.199 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.199 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.199 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:55:54.199 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:59.294156+0000", 2024-09-20T22:55:54.199 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.2756000000000003e-05 2024-09-20T22:55:54.199 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.199 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.199 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:55:54.199 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:59.294197+0000", 2024-09-20T22:55:54.200 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.1712000000000003e-05 2024-09-20T22:55:54.200 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.200 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.200 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:55:54.200 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:59.294295+0000", 2024-09-20T22:55:54.200 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.7819000000000007e-05 2024-09-20T22:55:54.200 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:55:54.200 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:55:54.200 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:55:54.200 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.200 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.200 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24652.0:7255 7.7 e728/442 7:ed54065f:::smithi08477203-12:head v 728'1294, mlcod=726'1291)", 2024-09-20T22:55:54.201 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:53:28.307390+0000", 2024-09-20T22:55:54.201 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 145.83235547199999, 2024-09-20T22:55:54.201 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.035000903999999999, 2024-09-20T22:55:54.201 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:55:54.201 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:55:54.201 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:55:54.201 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:55:54.201 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.201 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:55:54.201 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:28.307390+0000", 2024-09-20T22:55:54.201 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:55:54.201 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.201 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.202 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:55:54.202 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:28.307390+0000", 2024-09-20T22:55:54.202 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:55:54.202 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.202 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.202 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:55:54.202 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:28.307390+0000", 2024-09-20T22:55:54.202 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.28e-07 2024-09-20T22:55:54.202 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.202 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.202 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:55:54.202 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:28.320604+0000", 2024-09-20T22:55:54.202 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013213582999999999 2024-09-20T22:55:54.203 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.203 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.203 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:55:54.203 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:28.320632+0000", 2024-09-20T22:55:54.203 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.8354e-05 2024-09-20T22:55:54.203 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.203 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.203 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:55:54.203 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:28.320645+0000", 2024-09-20T22:55:54.203 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2989999999999999e-05 2024-09-20T22:55:54.203 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.203 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.203 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:55:54.203 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:28.320733+0000", 2024-09-20T22:55:54.204 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.8101000000000002e-05 2024-09-20T22:55:54.204 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.204 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.204 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:55:54.204 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:28.320761+0000", 2024-09-20T22:55:54.204 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.7608000000000001e-05 2024-09-20T22:55:54.204 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.204 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.204 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:55:54.204 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:28.342351+0000", 2024-09-20T22:55:54.204 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.021589856000000001 2024-09-20T22:55:54.204 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.204 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.204 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:55:54.204 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:28.342391+0000", 2024-09-20T22:55:54.205 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.0184e-05 2024-09-20T22:55:54.205 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:55:54.205 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:55:54.205 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:55:54.205 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.205 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.205 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:7356 7.a 7:5aaef466:::smithi08477203-40:head [copy-from ver 1901] snapc 14e=[14e,14d,14c,14a,149,148,146,145,142,141,140,13e,13d,13c,13b,13a,139,138,136,135,132,130,12f,12e,12c,12a,128,127,124,121,11f,112,103] ondisk+write+known_if_redirected+supports_pool_eio e738)", 2024-09-20T22:55:54.205 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:53:38.297845+0000", 2024-09-20T22:55:54.205 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 135.84190023400001, 2024-09-20T22:55:54.205 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.042948833999999998, 2024-09-20T22:55:54.205 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:55:54.205 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:55:54.205 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:55:54.206 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:55:54.206 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:55:54.206 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:55:54.206 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 7356 2024-09-20T22:55:54.206 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.206 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:55:54.206 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.206 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:55:54.206 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:38.297845+0000", 2024-09-20T22:55:54.206 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:55:54.206 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.206 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.206 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:55:54.207 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:38.297845+0000", 2024-09-20T22:55:54.207 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:55:54.207 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.207 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.207 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:55:54.207 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:38.297847+0000", 2024-09-20T22:55:54.207 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.7859999999999999e-06 2024-09-20T22:55:54.207 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.207 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.207 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:55:54.207 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:38.297863+0000", 2024-09-20T22:55:54.207 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.6184e-05 2024-09-20T22:55:54.207 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.207 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.208 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:55:54.208 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:38.297880+0000", 2024-09-20T22:55:54.208 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.6756e-05 2024-09-20T22:55:54.208 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.208 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.208 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:55:54.208 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:38.297891+0000", 2024-09-20T22:55:54.208 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0828e-05 2024-09-20T22:55:54.208 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.208 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.208 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:55:54.208 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:38.297996+0000", 2024-09-20T22:55:54.208 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00010518099999999999 2024-09-20T22:55:54.208 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.208 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.209 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op must wait for map", 2024-09-20T22:55:54.209 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:38.298007+0000", 2024-09-20T22:55:54.209 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0448999999999999e-05 2024-09-20T22:55:54.209 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.209 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.209 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:55:54.209 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:38.303629+0000", 2024-09-20T22:55:54.209 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0056224789999999997 2024-09-20T22:55:54.209 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.209 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.209 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:55:54.209 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:38.303768+0000", 2024-09-20T22:55:54.209 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013854699999999999 2024-09-20T22:55:54.209 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.209 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.210 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 2,5", 2024-09-20T22:55:54.210 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:38.312851+0000", 2024-09-20T22:55:54.210 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0090839030000000008 2024-09-20T22:55:54.210 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.210 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.210 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:55:54.210 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:38.321464+0000", 2024-09-20T22:55:54.210 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0086123620000000001 2024-09-20T22:55:54.210 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.210 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.210 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:55:54.210 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:38.322469+0000", 2024-09-20T22:55:54.210 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0010051630000000001 2024-09-20T22:55:54.210 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.210 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.211 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:55:54.211 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:38.340676+0000", 2024-09-20T22:55:54.211 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.018207042999999999 2024-09-20T22:55:54.211 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.211 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.211 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:55:54.211 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:38.340727+0000", 2024-09-20T22:55:54.211 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.1479e-05 2024-09-20T22:55:54.211 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.211 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.211 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:55:54.211 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:38.340794+0000", 2024-09-20T22:55:54.211 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.6674000000000006e-05 2024-09-20T22:55:54.211 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:55:54.212 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:55:54.212 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:55:54.212 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.212 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.212 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:7357 7.a 7:5aaef466:::smithi08477203-40:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e738)", 2024-09-20T22:55:54.212 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:53:38.297921+0000", 2024-09-20T22:55:54.212 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 135.84182463499999, 2024-09-20T22:55:54.212 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.021166364, 2024-09-20T22:55:54.212 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:55:54.212 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:55:54.212 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-20T22:55:54.212 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:55:54.212 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:55:54.212 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:55:54.212 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 7357 2024-09-20T22:55:54.212 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.213 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:55:54.213 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.213 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:55:54.213 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:38.297921+0000", 2024-09-20T22:55:54.213 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:55:54.213 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.213 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.213 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:55:54.213 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:38.297921+0000", 2024-09-20T22:55:54.213 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:55:54.213 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.213 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.213 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:55:54.213 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:38.297922+0000", 2024-09-20T22:55:54.214 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.9400000000000005e-07 2024-09-20T22:55:54.214 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.214 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.214 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:55:54.214 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:38.297925+0000", 2024-09-20T22:55:54.214 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.3900000000000002e-06 2024-09-20T22:55:54.214 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.214 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.214 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:55:54.214 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:38.297933+0000", 2024-09-20T22:55:54.214 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.9449999999999993e-06 2024-09-20T22:55:54.214 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.214 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.214 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:55:54.215 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:38.297939+0000", 2024-09-20T22:55:54.215 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.9089999999999997e-06 2024-09-20T22:55:54.215 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.215 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.215 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:55:54.215 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:38.298073+0000", 2024-09-20T22:55:54.215 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013442 2024-09-20T22:55:54.215 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.215 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.215 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting_for_map not empty", 2024-09-20T22:55:54.215 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:38.298082+0000", 2024-09-20T22:55:54.215 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.3960000000000006e-06 2024-09-20T22:55:54.215 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.215 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.215 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:55:54.216 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:38.304064+0000", 2024-09-20T22:55:54.216 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0059820530000000002 2024-09-20T22:55:54.216 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.216 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.216 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for blocked object", 2024-09-20T22:55:54.216 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:38.304119+0000", 2024-09-20T22:55:54.216 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.5637000000000003e-05 2024-09-20T22:55:54.216 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.216 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.216 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:55:54.216 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:38.318815+0000", 2024-09-20T22:55:54.216 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014695692999999999 2024-09-20T22:55:54.216 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.216 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.216 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:55:54.216 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:38.318984+0000", 2024-09-20T22:55:54.217 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00016855900000000001 2024-09-20T22:55:54.217 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.217 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.217 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:55:54.217 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:38.319087+0000", 2024-09-20T22:55:54.217 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000103768 2024-09-20T22:55:54.217 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:55:54.217 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:55:54.217 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:55:54.217 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.217 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.217 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(osd.6.0:2993 7.c e738/94 7:35bf785a:::smithi08477203-29:f9 v 738'2309, mlcod=738'2305)", 2024-09-20T22:55:54.217 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:53:38.323270+0000", 2024-09-20T22:55:54.217 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 135.81647564900001, 2024-09-20T22:55:54.217 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.015000573999999999, 2024-09-20T22:55:54.218 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:55:54.218 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:55:54.218 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:55:54.218 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:55:54.218 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.218 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:55:54.218 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:38.323270+0000", 2024-09-20T22:55:54.218 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:55:54.218 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.218 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.218 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:55:54.218 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:38.323270+0000", 2024-09-20T22:55:54.218 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:55:54.218 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.218 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.219 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:55:54.219 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:38.323271+0000", 2024-09-20T22:55:54.219 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.206e-06 2024-09-20T22:55:54.219 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.219 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.219 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:55:54.219 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:38.323283+0000", 2024-09-20T22:55:54.219 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1552999999999999e-05 2024-09-20T22:55:54.219 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.219 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.219 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:55:54.219 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:38.323292+0000", 2024-09-20T22:55:54.219 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.6940000000000007e-06 2024-09-20T22:55:54.219 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.219 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.220 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:55:54.220 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:38.323300+0000", 2024-09-20T22:55:54.220 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.6809999999999998e-06 2024-09-20T22:55:54.220 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.220 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.220 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:55:54.220 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:38.323360+0000", 2024-09-20T22:55:54.220 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.9794999999999999e-05 2024-09-20T22:55:54.220 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.220 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.220 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:55:54.220 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:38.323387+0000", 2024-09-20T22:55:54.220 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.6869999999999999e-05 2024-09-20T22:55:54.220 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.220 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.221 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:55:54.221 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:38.338224+0000", 2024-09-20T22:55:54.221 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.01483743 2024-09-20T22:55:54.221 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.221 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.221 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:55:54.221 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:38.338270+0000", 2024-09-20T22:55:54.221 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.6344999999999998e-05 2024-09-20T22:55:54.221 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:55:54.221 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:55:54.221 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:55:54.221 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.221 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.221 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:7392 7.3 7:cae5792a:::smithi08477203-32:head [write 709526~478431] snapc 14e=[14e,14c,14a,149,148,145,142,141,13e,13d,13c,13b,13a,139,135,132,130,12f,12e,12c,12a,128,127,124,121,11f,112,103] ondisk+write+known_if_redirected+supports_pool_eio e743)", 2024-09-20T22:55:54.221 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:53:43.327315+0000", 2024-09-20T22:55:54.221 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 130.81243065500001, 2024-09-20T22:55:54.222 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014351166, 2024-09-20T22:55:54.222 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:55:54.222 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:55:54.222 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:55:54.222 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:55:54.222 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:55:54.222 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:55:54.222 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 7392 2024-09-20T22:55:54.222 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.222 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:55:54.222 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.222 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:55:54.222 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:43.327315+0000", 2024-09-20T22:55:54.223 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:55:54.223 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.223 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.223 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:55:54.223 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:43.327315+0000", 2024-09-20T22:55:54.223 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:55:54.223 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.223 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.223 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:55:54.223 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:43.327316+0000", 2024-09-20T22:55:54.223 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.4729999999999999e-06 2024-09-20T22:55:54.223 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.223 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.224 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:55:54.224 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:43.327699+0000", 2024-09-20T22:55:54.224 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00038287299999999999 2024-09-20T22:55:54.224 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.224 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.224 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:55:54.224 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:43.327710+0000", 2024-09-20T22:55:54.224 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1072e-05 2024-09-20T22:55:54.224 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.224 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.224 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:55:54.224 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:43.327717+0000", 2024-09-20T22:55:54.224 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.1579999999999999e-06 2024-09-20T22:55:54.225 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.225 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.225 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:55:54.225 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:43.327776+0000", 2024-09-20T22:55:54.225 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.8757e-05 2024-09-20T22:55:54.225 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.225 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.225 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:55:54.225 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:43.327887+0000", 2024-09-20T22:55:54.225 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000110779 2024-09-20T22:55:54.225 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.225 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.225 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 2,4", 2024-09-20T22:55:54.226 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:43.328109+0000", 2024-09-20T22:55:54.226 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00022187899999999999 2024-09-20T22:55:54.226 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.226 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.226 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:55:54.226 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:43.329528+0000", 2024-09-20T22:55:54.226 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0014191780000000001 2024-09-20T22:55:54.226 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.226 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.226 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:55:54.226 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:43.329822+0000", 2024-09-20T22:55:54.226 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00029383700000000003 2024-09-20T22:55:54.226 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.226 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.227 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:55:54.227 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:43.341568+0000", 2024-09-20T22:55:54.227 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.01174619 2024-09-20T22:55:54.227 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.227 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.227 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:55:54.227 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:43.341620+0000", 2024-09-20T22:55:54.227 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.215e-05 2024-09-20T22:55:54.227 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.227 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.227 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:55:54.227 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:43.341666+0000", 2024-09-20T22:55:54.227 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.5819999999999998e-05 2024-09-20T22:55:54.227 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:55:54.227 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:55:54.228 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:55:54.228 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.228 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.228 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:7570 7.a 7:5122292a:::smithi08477203-33:head [copy-from ver 1829] snapc 157=[157,156,155,153,152,151,14f,14e,14c,14a,149,148,145,142,141,13e,13d,13b,13a,139,132,130,12f,12e,12a,127,124,121,112,103] ondisk+write+known_if_redirected+supports_pool_eio e759)", 2024-09-20T22:55:54.228 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:53:59.313723+0000", 2024-09-20T22:55:54.228 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 114.82602247299999, 2024-09-20T22:55:54.228 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.012092524, 2024-09-20T22:55:54.228 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:55:54.228 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:55:54.228 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:55:54.228 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:55:54.228 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:55:54.228 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:55:54.228 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 7570 2024-09-20T22:55:54.228 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.229 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:55:54.229 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.229 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:55:54.229 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:59.313723+0000", 2024-09-20T22:55:54.229 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:55:54.229 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.229 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.229 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:55:54.229 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:59.313723+0000", 2024-09-20T22:55:54.229 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:55:54.229 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.229 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.229 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:55:54.230 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:59.313724+0000", 2024-09-20T22:55:54.230 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3909999999999999e-06 2024-09-20T22:55:54.230 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.230 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.230 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:55:54.230 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:59.313729+0000", 2024-09-20T22:55:54.230 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.2300000000000002e-06 2024-09-20T22:55:54.230 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.230 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.230 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:55:54.230 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:59.313739+0000", 2024-09-20T22:55:54.230 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0039e-05 2024-09-20T22:55:54.230 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.230 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.231 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:55:54.231 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:59.313744+0000", 2024-09-20T22:55:54.231 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.147e-06 2024-09-20T22:55:54.231 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.231 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.231 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:55:54.231 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:59.313786+0000", 2024-09-20T22:55:54.231 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.2135999999999998e-05 2024-09-20T22:55:54.231 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.231 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.231 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:55:54.231 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:59.313871+0000", 2024-09-20T22:55:54.231 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.4515e-05 2024-09-20T22:55:54.231 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.232 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.232 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 2,5", 2024-09-20T22:55:54.232 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:59.317919+0000", 2024-09-20T22:55:54.232 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0040488859999999998 2024-09-20T22:55:54.232 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.232 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.232 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:55:54.232 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:59.322547+0000", 2024-09-20T22:55:54.232 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0046278810000000004 2024-09-20T22:55:54.232 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.232 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.232 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:55:54.232 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:59.325486+0000", 2024-09-20T22:55:54.232 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0029383629999999998 2024-09-20T22:55:54.233 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.233 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.233 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:55:54.233 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:59.325706+0000", 2024-09-20T22:55:54.233 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00022070100000000001 2024-09-20T22:55:54.233 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.233 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.233 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:55:54.233 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:59.325754+0000", 2024-09-20T22:55:54.233 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.8099000000000001e-05 2024-09-20T22:55:54.233 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.233 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.233 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:55:54.233 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:59.325816+0000", 2024-09-20T22:55:54.234 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.1136000000000006e-05 2024-09-20T22:55:54.234 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:55:54.234 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:55:54.234 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:55:54.234 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.234 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.234 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:7700 7.a 7:5aaef466:::smithi08477203-40:head [copy-from ver 1852] snapc 15c=[15c,15b,15a,159,158,157,156,155,153,151,14f,14e,14c,14a,149,148,145,142,13e,13a,139,132,12f,12e,12a,124,112] ondisk+write+known_if_redirected+supports_pool_eio e772)", 2024-09-20T22:55:54.234 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:54:12.432372+0000", 2024-09-20T22:55:54.234 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 101.707373096, 2024-09-20T22:55:54.234 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.017236896000000002, 2024-09-20T22:55:54.234 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:55:54.234 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:55:54.234 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:55:54.234 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:55:54.235 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:55:54.235 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:55:54.235 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 7700 2024-09-20T22:55:54.235 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.235 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:55:54.235 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.235 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:55:54.235 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:54:12.432372+0000", 2024-09-20T22:55:54.235 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:55:54.235 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.235 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.236 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:55:54.236 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:54:12.432372+0000", 2024-09-20T22:55:54.236 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:55:54.236 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.236 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.236 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:55:54.236 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:54:12.432374+0000", 2024-09-20T22:55:54.236 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.719e-06 2024-09-20T22:55:54.236 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.236 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.236 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:55:54.236 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:54:12.432379+0000", 2024-09-20T22:55:54.236 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.8749999999999999e-06 2024-09-20T22:55:54.237 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.237 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.237 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:55:54.237 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:54:12.432390+0000", 2024-09-20T22:55:54.237 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1182999999999999e-05 2024-09-20T22:55:54.237 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.237 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.237 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:55:54.237 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:54:12.432396+0000", 2024-09-20T22:55:54.237 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.361e-06 2024-09-20T22:55:54.237 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.237 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.237 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:55:54.237 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:54:12.432455+0000", 2024-09-20T22:55:54.238 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.9335e-05 2024-09-20T22:55:54.238 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.238 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.238 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:55:54.238 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:54:12.432581+0000", 2024-09-20T22:55:54.238 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00012652799999999999 2024-09-20T22:55:54.238 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.238 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.238 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 2,5", 2024-09-20T22:55:54.238 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:54:12.437761+0000", 2024-09-20T22:55:54.238 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0051796780000000001 2024-09-20T22:55:54.238 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.238 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.239 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:55:54.239 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:54:12.443924+0000", 2024-09-20T22:55:54.239 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0061626739999999999 2024-09-20T22:55:54.239 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.239 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.239 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:55:54.239 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:54:12.449146+0000", 2024-09-20T22:55:54.239 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0052219049999999998 2024-09-20T22:55:54.239 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.239 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.239 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:55:54.239 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:54:12.449458+0000", 2024-09-20T22:55:54.239 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00031229700000000001 2024-09-20T22:55:54.239 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.240 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.240 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:55:54.240 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:54:12.449538+0000", 2024-09-20T22:55:54.240 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.9763000000000002e-05 2024-09-20T22:55:54.240 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.240 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.240 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:55:54.240 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:54:12.449609+0000", 2024-09-20T22:55:54.240 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.1577999999999998e-05 2024-09-20T22:55:54.240 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:55:54.240 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:55:54.240 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:55:54.240 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.241 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.241 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:8125 7.2 7:4d20232a:::smithi08477203-34:157 [read 0~0,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e820)", 2024-09-20T22:55:54.241 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:55:00.332139+0000", 2024-09-20T22:55:54.241 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 53.807606294999999, 2024-09-20T22:55:54.241 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010368334, 2024-09-20T22:55:54.241 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:55:54.241 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:55:54.241 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "reached pg", 2024-09-20T22:55:54.241 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:55:54.241 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:55:54.241 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:55:54.241 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 8125 2024-09-20T22:55:54.241 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.242 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:55:54.242 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.242 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:55:54.242 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:55:00.332139+0000", 2024-09-20T22:55:54.242 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:55:54.242 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.242 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.242 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:55:54.242 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:55:00.332139+0000", 2024-09-20T22:55:54.242 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:55:54.242 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.242 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.242 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:55:54.242 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:55:00.332140+0000", 2024-09-20T22:55:54.243 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.8999999999999997e-07 2024-09-20T22:55:54.243 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.243 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.243 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:55:54.243 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:55:00.332143+0000", 2024-09-20T22:55:54.243 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.4800000000000001e-06 2024-09-20T22:55:54.243 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.243 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.243 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:55:54.243 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:55:00.332148+0000", 2024-09-20T22:55:54.243 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.8169999999999998e-06 2024-09-20T22:55:54.243 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.243 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.243 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:55:54.244 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:55:00.332153+0000", 2024-09-20T22:55:54.244 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.4519999999999999e-06 2024-09-20T22:55:54.244 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.244 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.244 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:55:54.244 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:55:00.342418+0000", 2024-09-20T22:55:54.244 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010265501999999999 2024-09-20T22:55:54.244 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.244 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.244 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:55:54.244 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:55:00.342508+0000", 2024-09-20T22:55:54.244 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.9592999999999994e-05 2024-09-20T22:55:54.244 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:55:54.244 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:55:54.245 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:55:54.245 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.245 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.245 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(osd.0.0:2360 7.f e864/391 7:fa55e5ae:::smithi08477203-7:17a v 864'987, mlcod=861'983)", 2024-09-20T22:55:54.245 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:55:44.367564+0000", 2024-09-20T22:55:54.245 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 9.7721811550000002, 2024-09-20T22:55:54.245 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.01015689, 2024-09-20T22:55:54.245 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:55:54.245 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:55:54.245 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:55:54.245 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:55:54.246 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.246 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:55:54.246 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:55:44.367564+0000", 2024-09-20T22:55:54.246 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:55:54.246 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.246 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.246 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:55:54.246 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:55:44.367564+0000", 2024-09-20T22:55:54.246 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:55:54.246 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.246 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.246 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:55:54.246 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:55:44.367565+0000", 2024-09-20T22:55:54.247 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.3200000000000004e-07 2024-09-20T22:55:54.247 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.247 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.247 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:55:54.247 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:55:44.367571+0000", 2024-09-20T22:55:54.247 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.0449999999999997e-06 2024-09-20T22:55:54.247 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.247 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.247 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:55:54.247 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:55:44.375106+0000", 2024-09-20T22:55:54.247 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0075349090000000002 2024-09-20T22:55:54.247 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.247 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.247 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:55:54.247 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:55:44.375227+0000", 2024-09-20T22:55:54.248 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000121004 2024-09-20T22:55:54.248 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.248 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.248 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:55:54.248 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:55:44.376008+0000", 2024-09-20T22:55:54.248 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00078093499999999996 2024-09-20T22:55:54.248 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.248 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.248 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:55:54.248 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:55:44.376030+0000", 2024-09-20T22:55:54.248 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.1535e-05 2024-09-20T22:55:54.248 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.248 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.248 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:55:54.249 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:55:44.377683+0000", 2024-09-20T22:55:54.249 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001652862 2024-09-20T22:55:54.249 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.249 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.249 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:55:54.249 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:55:44.377721+0000", 2024-09-20T22:55:54.249 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.8767999999999999e-05 2024-09-20T22:55:54.249 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:55:54.249 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:55:54.249 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:55:54.249 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.249 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.249 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(osd.0.0:2390 7.f e872/391 7:fa55e5ae:::smithi08477203-7:153 v 872'995, mlcod=864'989)", 2024-09-20T22:55:54.249 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:55:52.416170+0000", 2024-09-20T22:55:54.249 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 1.723575946, 2024-09-20T22:55:54.250 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011422157, 2024-09-20T22:55:54.250 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:55:54.250 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:55:54.250 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:55:54.250 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:55:54.250 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.250 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:55:54.250 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:55:52.416170+0000", 2024-09-20T22:55:54.250 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:55:54.250 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.250 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.250 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:55:54.250 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:55:52.416170+0000", 2024-09-20T22:55:54.250 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:55:54.251 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.251 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.251 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:55:54.251 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:55:52.416170+0000", 2024-09-20T22:55:54.251 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.9700000000000002e-07 2024-09-20T22:55:54.251 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.251 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.251 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:55:54.251 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:55:52.416179+0000", 2024-09-20T22:55:54.251 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.2969999999999992e-06 2024-09-20T22:55:54.251 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.251 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.251 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:55:54.251 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:55:52.416191+0000", 2024-09-20T22:55:54.252 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1192999999999999e-05 2024-09-20T22:55:54.252 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.252 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.252 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:55:54.252 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:55:52.416197+0000", 2024-09-20T22:55:54.252 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.2879999999999998e-06 2024-09-20T22:55:54.252 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.252 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.252 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:55:54.252 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:55:52.416981+0000", 2024-09-20T22:55:54.252 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00078445499999999998 2024-09-20T22:55:54.252 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.252 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.252 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:55:54.253 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:55:52.417001+0000", 2024-09-20T22:55:54.253 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.0112e-05 2024-09-20T22:55:54.253 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.253 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.253 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:55:54.253 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:55:52.427529+0000", 2024-09-20T22:55:54.253 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010527643999999999 2024-09-20T22:55:54.253 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.253 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.253 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:55:54.253 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:55:52.427592+0000", 2024-09-20T22:55:54.253 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.2770999999999999e-05 2024-09-20T22:55:54.253 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:55:54.253 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:55:54.254 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:55:54.254 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.254 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.254 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:8664 7.a 7:5aaef466:::smithi08477203-40:head [write 649067~503784,stat] snapc 198=[198,197,196,195,193,192,191,190,18f,18e,18d,18c,18a,189,186,185,183,182,181,180,17e,17d,17b,17a,179,177,176,175,174,173,172,16f,16e,16c,16b,160,15c,158,155,14f,14e,14a,13a,139,12f,12e] ondisk+write+known_if_redirected+supports_pool_eio e873)", 2024-09-20T22:55:54.254 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:55:53.426425+0000", 2024-09-20T22:55:54.254 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 0.71332050800000002, 2024-09-20T22:55:54.254 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.019000435, 2024-09-20T22:55:54.254 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:55:54.254 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:55:54.254 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:55:54.254 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:55:54.254 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:55:54.254 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:55:54.254 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 8664 2024-09-20T22:55:54.254 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.255 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:55:54.255 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.255 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:55:54.255 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:55:53.426425+0000", 2024-09-20T22:55:54.255 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:55:54.255 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.255 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.255 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:55:54.255 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:55:53.426425+0000", 2024-09-20T22:55:54.255 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:55:54.255 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.255 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.255 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:55:54.255 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:55:53.426427+0000", 2024-09-20T22:55:54.256 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.5999999999999999e-06 2024-09-20T22:55:54.256 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.256 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.256 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:55:54.256 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:55:53.426644+0000", 2024-09-20T22:55:54.256 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000217772 2024-09-20T22:55:54.256 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.256 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.256 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:55:54.256 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:55:54.257 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:55:53.436248+0000", 2024-09-20T22:55:54.257 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0096033019999999993 2024-09-20T22:55:54.257 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.257 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.257 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:55:54.257 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:55:53.436261+0000", 2024-09-20T22:55:54.257 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3740000000000001e-05 2024-09-20T22:55:54.257 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.257 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.257 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:55:54.257 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:55:53.436447+0000", 2024-09-20T22:55:54.258 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00018589400000000001 2024-09-20T22:55:54.258 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.258 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.258 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:55:54.258 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:55:53.436580+0000", 2024-09-20T22:55:54.258 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000132774 2024-09-20T22:55:54.258 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.258 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.258 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 2,5", 2024-09-20T22:55:54.258 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:55:53.436846+0000", 2024-09-20T22:55:54.258 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00026542699999999999 2024-09-20T22:55:54.258 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.258 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.258 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:55:54.259 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:55:53.440332+0000", 2024-09-20T22:55:54.259 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0034859840000000001 2024-09-20T22:55:54.259 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.259 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.259 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:55:54.259 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:55:53.442140+0000", 2024-09-20T22:55:54.259 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001808065 2024-09-20T22:55:54.259 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.259 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.259 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:55:54.259 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:55:53.445313+0000", 2024-09-20T22:55:54.259 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0031738140000000001 2024-09-20T22:55:54.259 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.260 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.260 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:55:54.260 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:55:53.445376+0000", 2024-09-20T22:55:54.260 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.2872000000000004e-05 2024-09-20T22:55:54.260 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.260 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.260 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:55:54.260 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:55:53.445425+0000", 2024-09-20T22:55:54.260 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.9190999999999997e-05 2024-09-20T22:55:54.260 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:55:54.260 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:55:54.260 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:55:54.260 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.261 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.261 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:8665 7.a 7:5aaef466:::smithi08477203-40:head [write 1875940~607071] snapc 198=[198,197,196,195,193,192,191,190,18f,18e,18d,18c,18a,189,186,185,183,182,181,180,17e,17d,17b,17a,179,177,176,175,174,173,172,16f,16e,16c,16b,160,15c,158,155,14f,14e,14a,13a,139,12f,12e] ondisk+write+known_if_redirected+supports_pool_eio e873)", 2024-09-20T22:55:54.261 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:55:53.436294+0000", 2024-09-20T22:55:54.261 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 0.70345181499999998, 2024-09-20T22:55:54.261 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0093140710000000002, 2024-09-20T22:55:54.261 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:55:54.261 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:55:54.261 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:55:54.261 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:55:54.261 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:55:54.261 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:55:54.261 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 8665 2024-09-20T22:55:54.261 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.261 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:55:54.262 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.262 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:55:54.262 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:55:53.436294+0000", 2024-09-20T22:55:54.262 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:55:54.262 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.262 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.262 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:55:54.262 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:55:53.436294+0000", 2024-09-20T22:55:54.262 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:55:54.262 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.262 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.262 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:55:54.262 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:55:53.436295+0000", 2024-09-20T22:55:54.262 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.0999999999999997e-07 2024-09-20T22:55:54.262 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.263 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.263 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:55:54.263 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:55:53.436616+0000", 2024-09-20T22:55:54.263 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00032161000000000001 2024-09-20T22:55:54.263 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.263 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.263 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:55:54.263 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:55:53.436625+0000", 2024-09-20T22:55:54.263 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.6249999999999996e-06 2024-09-20T22:55:54.263 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.263 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.263 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:55:54.263 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:55:53.436631+0000", 2024-09-20T22:55:54.263 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.2400000000000004e-06 2024-09-20T22:55:54.264 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.264 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.264 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:55:54.264 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:55:53.438724+0000", 2024-09-20T22:55:54.264 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0020933840000000002 2024-09-20T22:55:54.264 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.264 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.264 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:55:54.264 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:55:53.438846+0000", 2024-09-20T22:55:54.264 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000121236 2024-09-20T22:55:54.264 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.264 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.264 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 2,5", 2024-09-20T22:55:54.264 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:55:53.439049+0000", 2024-09-20T22:55:54.264 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000203721 2024-09-20T22:55:54.265 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.265 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.265 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:55:54.265 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:55:53.440652+0000", 2024-09-20T22:55:54.265 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001602359 2024-09-20T22:55:54.265 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.265 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.265 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:55:54.265 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:55:53.442384+0000", 2024-09-20T22:55:54.265 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001732234 2024-09-20T22:55:54.265 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.265 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.265 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:55:54.265 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:55:53.445481+0000", 2024-09-20T22:55:54.265 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.003096723 2024-09-20T22:55:54.266 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.266 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.266 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:55:54.266 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:55:53.445557+0000", 2024-09-20T22:55:54.266 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.6207999999999999e-05 2024-09-20T22:55:54.266 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:54.266 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:54.266 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:55:54.266 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:55:53.445608+0000", 2024-09-20T22:55:54.266 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.0921000000000003e-05 2024-09-20T22:55:54.266 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:55:54.266 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:55:54.266 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:55:54.266 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:55:54.267 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:55:54.267 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-20T22:55:54.267 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-20T22:55:54.357 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:55:54.383 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:54.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:54.384 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:55:54.422 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is unset 2024-09-20T22:55:54.428 INFO:tasks.rados.rados.0.smithi084.stdout:3476: done (1 left) 2024-09-20T22:55:54.428 INFO:tasks.rados.rados.0.smithi084.stdout:3477: snap_create 2024-09-20T22:55:54.428 INFO:tasks.rados.rados.0.smithi084.stdout:3474: expect (ObjNum 1143 snap 389 seq_num 1143) 2024-09-20T22:55:54.440 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:54 smithi084 ceph-mon[29774]: osdmap e873: 8 total, 8 up, 8 in 2024-09-20T22:55:54.440 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:54 smithi084 ceph-mon[29774]: 3.5 scrub starts 2024-09-20T22:55:54.440 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:54 smithi084 ceph-mon[29774]: 3.5 scrub ok 2024-09-20T22:55:54.440 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:54 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/2956014594' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:55:54.440 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:54 smithi084 ceph-mon[29774]: 5.2 scrub starts 2024-09-20T22:55:54.440 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:54 smithi084 ceph-mon[29774]: 5.2 scrub ok 2024-09-20T22:55:54.440 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:54 smithi084 ceph-mon[29774]: 6.1 scrub starts 2024-09-20T22:55:54.441 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:54 smithi084 ceph-mon[29774]: 6.1 scrub ok 2024-09-20T22:55:54.441 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:54 smithi084 ceph-mon[40024]: osdmap e873: 8 total, 8 up, 8 in 2024-09-20T22:55:54.441 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:54 smithi084 ceph-mon[40024]: 3.5 scrub starts 2024-09-20T22:55:54.441 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:54 smithi084 ceph-mon[40024]: 3.5 scrub ok 2024-09-20T22:55:54.441 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:54 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/2956014594' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:55:54.441 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:54 smithi084 ceph-mon[40024]: 5.2 scrub starts 2024-09-20T22:55:54.441 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:54 smithi084 ceph-mon[40024]: 5.2 scrub ok 2024-09-20T22:55:54.441 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:54 smithi084 ceph-mon[40024]: 6.1 scrub starts 2024-09-20T22:55:54.441 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:54 smithi084 ceph-mon[40024]: 6.1 scrub ok 2024-09-20T22:55:54.485 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:55:54.506 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:54.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:54.507 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:55:54.608 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:55:54.634 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:54.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:54.635 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:55:54.736 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:55:54.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:54 smithi154 ceph-mon[33264]: osdmap e873: 8 total, 8 up, 8 in 2024-09-20T22:55:54.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:54 smithi154 ceph-mon[33264]: 3.5 scrub starts 2024-09-20T22:55:54.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:54 smithi154 ceph-mon[33264]: 3.5 scrub ok 2024-09-20T22:55:54.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:54 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/2956014594' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:55:54.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:54 smithi154 ceph-mon[33264]: 5.2 scrub starts 2024-09-20T22:55:54.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:54 smithi154 ceph-mon[33264]: 5.2 scrub ok 2024-09-20T22:55:54.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:54 smithi154 ceph-mon[33264]: 6.1 scrub starts 2024-09-20T22:55:54.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:54 smithi154 ceph-mon[33264]: 6.1 scrub ok 2024-09-20T22:55:54.762 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:54.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:54.763 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:55:54.863 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:55:54.889 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:54.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:54.890 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:55:54.991 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:55:55.017 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:55.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:55.018 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:55:55.118 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:55:55.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:55.141 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:55.142 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:55:55.242 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:55:55.268 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:55.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:55.269 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:55:55.370 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:55:55.393 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:55.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:55.394 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:55:55.433 INFO:tasks.rados.rados.0.smithi084.stdout:3474: done (1 left) 2024-09-20T22:55:55.433 INFO:tasks.rados.rados.0.smithi084.stdout:3477: done (0 left) 2024-09-20T22:55:55.433 INFO:tasks.rados.rados.0.smithi084.stdout:3478: write (excl) oid 22 current snap is 408 2024-09-20T22:55:55.433 INFO:tasks.rados.rados.0.smithi084.stdout:3478: seq_num 1184 ranges {699488=729531,1935033=470751,3081294=681475} 2024-09-20T22:55:55.440 INFO:tasks.rados.rados.0.smithi084.stdout:3478: writing smithi08477203-22 from 699488 to 1429019 tid 1 2024-09-20T22:55:55.446 INFO:tasks.rados.rados.0.smithi084.stdout:3478: writing smithi08477203-22 from 1935033 to 2405784 tid 2 2024-09-20T22:55:55.455 INFO:tasks.rados.rados.0.smithi084.stdout:3478: writing smithi08477203-22 from 3081294 to 3762769 tid 3 2024-09-20T22:55:55.455 INFO:tasks.rados.rados.0.smithi084.stdout:3479: write oid 41 current snap is 408 2024-09-20T22:55:55.455 INFO:tasks.rados.rados.0.smithi084.stdout:3478: finishing write tid 1 to smithi08477203-22 2024-09-20T22:55:55.455 INFO:tasks.rados.rados.0.smithi084.stdout:3479: seq_num 1185 ranges {716295=404780,1593515=614409,2708717=492747,3640519=275776} 2024-09-20T22:55:55.460 INFO:tasks.rados.rados.0.smithi084.stdout:3479: writing smithi08477203-41 from 716295 to 1121075 tid 1 2024-09-20T22:55:55.466 INFO:tasks.rados.rados.0.smithi084.stdout:3479: writing smithi08477203-41 from 1593515 to 2207924 tid 2 2024-09-20T22:55:55.472 INFO:tasks.rados.rados.0.smithi084.stdout:3479: writing smithi08477203-41 from 2708717 to 3201464 tid 3 2024-09-20T22:55:55.476 INFO:tasks.rados.rados.0.smithi084.stdout:3479: writing smithi08477203-41 from 3640519 to 3916295 tid 4 2024-09-20T22:55:55.476 INFO:tasks.rados.rados.0.smithi084.stdout:3480: write oid 10 current snap is 408 2024-09-20T22:55:55.476 INFO:tasks.rados.rados.0.smithi084.stdout:3478: finishing write tid 2 to smithi08477203-22 2024-09-20T22:55:55.476 INFO:tasks.rados.rados.0.smithi084.stdout:3480: seq_num 1186 ranges {733102=596382,1768350=758067,3136140=704019} 2024-09-20T22:55:55.483 INFO:tasks.rados.rados.0.smithi084.stdout:3480: writing smithi08477203-10 from 733102 to 1329484 tid 1 2024-09-20T22:55:55.490 INFO:tasks.rados.rados.0.smithi084.stdout:3480: writing smithi08477203-10 from 1768350 to 2526417 tid 2 2024-09-20T22:55:55.494 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:55:55.497 INFO:tasks.rados.rados.0.smithi084.stdout:3480: writing smithi08477203-10 from 3136140 to 3840159 tid 3 2024-09-20T22:55:55.498 INFO:tasks.rados.rados.0.smithi084.stdout:3481: write oid 2 current snap is 408 2024-09-20T22:55:55.498 INFO:tasks.rados.rados.0.smithi084.stdout:3478: finishing write tid 3 to smithi08477203-22 2024-09-20T22:55:55.498 INFO:tasks.rados.rados.0.smithi084.stdout:3478: finishing write tid 4 to smithi08477203-22 2024-09-20T22:55:55.498 INFO:tasks.rados.rados.0.smithi084.stdout:3478: finishing write tid 5 to smithi08477203-22 2024-09-20T22:55:55.498 INFO:tasks.rados.rados.0.smithi084.stdout:3478: oid 22 updating version 0 to 2485 2024-09-20T22:55:55.498 INFO:tasks.rados.rados.0.smithi084.stdout:3478: oid 22 version 2485 is already newer than 2483 2024-09-20T22:55:55.498 INFO:tasks.rados.rados.0.smithi084.stdout:3478: oid 22 version 2485 is already newer than 2484 2024-09-20T22:55:55.498 INFO:tasks.rados.rados.0.smithi084.stdout:3478: oid 22 version 2485 is already newer than 2482 2024-09-20T22:55:55.498 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 22 v 2485 (ObjNum 1184 snap 408 seq_num 1184) dirty exists 2024-09-20T22:55:55.498 INFO:tasks.rados.rados.0.smithi084.stdout:3478: left oid 22 (ObjNum 1184 snap 408 seq_num 1184) 2024-09-20T22:55:55.498 INFO:tasks.rados.rados.0.smithi084.stdout:3479: finishing write tid 1 to smithi08477203-41 2024-09-20T22:55:55.498 INFO:tasks.rados.rados.0.smithi084.stdout:3479: finishing write tid 2 to smithi08477203-41 2024-09-20T22:55:55.498 INFO:tasks.rados.rados.0.smithi084.stdout:3479: finishing write tid 3 to smithi08477203-41 2024-09-20T22:55:55.499 INFO:tasks.rados.rados.0.smithi084.stdout:3481: seq_num 1187 ranges {749909=671631,2110479=618078,3163563=515291} 2024-09-20T22:55:55.505 INFO:tasks.rados.rados.0.smithi084.stdout:3481: writing smithi08477203-2 from 749909 to 1421540 tid 1 2024-09-20T22:55:55.512 INFO:tasks.rados.rados.0.smithi084.stdout:3481: writing smithi08477203-2 from 2110479 to 2728557 tid 2 2024-09-20T22:55:55.517 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:55.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:55.518 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:55:55.518 INFO:tasks.rados.rados.0.smithi084.stdout:3481: writing smithi08477203-2 from 3163563 to 3678854 tid 3 2024-09-20T22:55:55.518 INFO:tasks.rados.rados.0.smithi084.stdout:3478: done (3 left) 2024-09-20T22:55:55.519 INFO:tasks.rados.rados.0.smithi084.stdout:3482: snap_remove snap 300 2024-09-20T22:55:55.584 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:55:55.618 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:55:55.639 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:55.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:55.640 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:55:55.700 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:55 smithi084 ceph-mon[40024]: pgmap v1212: 158 pgs: 2 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 152 active+clean; 639 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 3.1 MiB/s rd, 1.6 MiB/s wr, 11 op/s 2024-09-20T22:55:55.700 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:55 smithi084 ceph-mon[40024]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T22:55:55.700 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:55 smithi084 ceph-mon[40024]: Cluster is now healthy 2024-09-20T22:55:55.700 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:55 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/2956014594' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T22:55:55.700 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:55 smithi084 ceph-mon[40024]: osdmap e874: 8 total, 8 up, 8 in 2024-09-20T22:55:55.700 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:55 smithi084 ceph-mon[40024]: 3.16 scrub starts 2024-09-20T22:55:55.700 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:55 smithi084 ceph-mon[40024]: 3.16 scrub ok 2024-09-20T22:55:55.700 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:55 smithi084 ceph-mon[29774]: pgmap v1212: 158 pgs: 2 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 152 active+clean; 639 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 3.1 MiB/s rd, 1.6 MiB/s wr, 11 op/s 2024-09-20T22:55:55.701 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:55 smithi084 ceph-mon[29774]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T22:55:55.701 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:55 smithi084 ceph-mon[29774]: Cluster is now healthy 2024-09-20T22:55:55.701 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:55 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/2956014594' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T22:55:55.701 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:55 smithi084 ceph-mon[29774]: osdmap e874: 8 total, 8 up, 8 in 2024-09-20T22:55:55.701 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:55 smithi084 ceph-mon[29774]: 3.16 scrub starts 2024-09-20T22:55:55.701 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:55 smithi084 ceph-mon[29774]: 3.16 scrub ok 2024-09-20T22:55:55.741 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:55:55.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:55 smithi154 ceph-mon[33264]: pgmap v1212: 158 pgs: 2 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 152 active+clean; 639 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 3.1 MiB/s rd, 1.6 MiB/s wr, 11 op/s 2024-09-20T22:55:55.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:55 smithi154 ceph-mon[33264]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T22:55:55.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:55 smithi154 ceph-mon[33264]: Cluster is now healthy 2024-09-20T22:55:55.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:55 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/2956014594' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T22:55:55.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:55 smithi154 ceph-mon[33264]: osdmap e874: 8 total, 8 up, 8 in 2024-09-20T22:55:55.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:55 smithi154 ceph-mon[33264]: 3.16 scrub starts 2024-09-20T22:55:55.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:55 smithi154 ceph-mon[33264]: 3.16 scrub ok 2024-09-20T22:55:55.765 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:55.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:55.766 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:55:55.867 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:55:55.888 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:55.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:55.889 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:55:55.990 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:55:56.000 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-20T22:55:56.011 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:56.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:56.012 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:55:56.047 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:55:56.113 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:55:56.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:56.135 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:56.135 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:55:56.236 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:55:56.255 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:55:56.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:56.258 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:56.258 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:55:56.359 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:55:56.380 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:56.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:56.381 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:55:56.444 INFO:tasks.rados.rados.0.smithi084.stdout:3482: done (3 left) 2024-09-20T22:55:56.444 INFO:tasks.rados.rados.0.smithi084.stdout:3483: delete oid 23 current snap is 408 2024-09-20T22:55:56.444 INFO:tasks.rados.rados.0.smithi084.stdout:3479: finishing write tid 4 to smithi08477203-41 2024-09-20T22:55:56.444 INFO:tasks.rados.rados.0.smithi084.stdout:3479: finishing write tid 5 to smithi08477203-41 2024-09-20T22:55:56.444 INFO:tasks.rados.rados.0.smithi084.stdout:3479: finishing write tid 6 to smithi08477203-41 2024-09-20T22:55:56.444 INFO:tasks.rados.rados.0.smithi084.stdout:3479: oid 41 updating version 0 to 1942 2024-09-20T22:55:56.444 INFO:tasks.rados.rados.0.smithi084.stdout:3479: oid 41 version 1942 is already newer than 1939 2024-09-20T22:55:56.445 INFO:tasks.rados.rados.0.smithi084.stdout:3479: oid 41 version 1942 is already newer than 1938 2024-09-20T22:55:56.445 INFO:tasks.rados.rados.0.smithi084.stdout:3479: oid 41 version 1942 is already newer than 1940 2024-09-20T22:55:56.445 INFO:tasks.rados.rados.0.smithi084.stdout:3479: oid 41 version 1942 is already newer than 1941 2024-09-20T22:55:56.445 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 41 v 1942 (ObjNum 1185 snap 408 seq_num 1185) dirty exists 2024-09-20T22:55:56.445 INFO:tasks.rados.rados.0.smithi084.stdout:3479: left oid 41 (ObjNum 1185 snap 408 seq_num 1185) 2024-09-20T22:55:56.445 INFO:tasks.rados.rados.0.smithi084.stdout:3480: finishing write tid 1 to smithi08477203-10 2024-09-20T22:55:56.445 INFO:tasks.rados.rados.0.smithi084.stdout:3480: finishing write tid 2 to smithi08477203-10 2024-09-20T22:55:56.445 INFO:tasks.rados.rados.0.smithi084.stdout:3480: finishing write tid 3 to smithi08477203-10 2024-09-20T22:55:56.445 INFO:tasks.rados.rados.0.smithi084.stdout:3480: finishing write tid 4 to smithi08477203-10 2024-09-20T22:55:56.445 INFO:tasks.rados.rados.0.smithi084.stdout:3480: finishing write tid 5 to smithi08477203-10 2024-09-20T22:55:56.445 INFO:tasks.rados.rados.0.smithi084.stdout:3480: oid 10 updating version 0 to 2646 2024-09-20T22:55:56.445 INFO:tasks.rados.rados.0.smithi084.stdout:3480: oid 10 version 2646 is already newer than 2645 2024-09-20T22:55:56.445 INFO:tasks.rados.rados.0.smithi084.stdout:3480: oid 10 version 2646 is already newer than 2644 2024-09-20T22:55:56.445 INFO:tasks.rados.rados.0.smithi084.stdout:3480: oid 10 version 2646 is already newer than 2643 2024-09-20T22:55:56.446 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 10 v 2646 (ObjNum 1186 snap 408 seq_num 1186) dirty exists 2024-09-20T22:55:56.446 INFO:tasks.rados.rados.0.smithi084.stdout:3480: left oid 10 (ObjNum 1186 snap 408 seq_num 1186) 2024-09-20T22:55:56.446 INFO:tasks.rados.rados.0.smithi084.stdout:3481: finishing write tid 1 to smithi08477203-2 2024-09-20T22:55:56.446 INFO:tasks.rados.rados.0.smithi084.stdout:3481: finishing write tid 2 to smithi08477203-2 2024-09-20T22:55:56.446 INFO:tasks.rados.rados.0.smithi084.stdout:3481: finishing write tid 3 to smithi08477203-2 2024-09-20T22:55:56.446 INFO:tasks.rados.rados.0.smithi084.stdout:3481: finishing write tid 4 to smithi08477203-2 2024-09-20T22:55:56.446 INFO:tasks.rados.rados.0.smithi084.stdout:3481: finishing write tid 5 to smithi08477203-2 2024-09-20T22:55:56.446 INFO:tasks.rados.rados.0.smithi084.stdout:3481: oid 2 updating version 0 to 1866 2024-09-20T22:55:56.446 INFO:tasks.rados.rados.0.smithi084.stdout:3481: oid 2 updating version 1866 to 1867 2024-09-20T22:55:56.446 INFO:tasks.rados.rados.0.smithi084.stdout:3481: oid 2 updating version 1867 to 1869 2024-09-20T22:55:56.446 INFO:tasks.rados.rados.0.smithi084.stdout:3481: oid 2 version 1869 is already newer than 1868 2024-09-20T22:55:56.446 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 2 v 1869 (ObjNum 1187 snap 408 seq_num 1187) dirty exists 2024-09-20T22:55:56.446 INFO:tasks.rados.rados.0.smithi084.stdout:3481: left oid 2 (ObjNum 1187 snap 408 seq_num 1187) 2024-09-20T22:55:56.447 INFO:tasks.rados.rados.0.smithi084.stdout:3479: done (3 left) 2024-09-20T22:55:56.447 INFO:tasks.rados.rados.0.smithi084.stdout:3480: done (2 left) 2024-09-20T22:55:56.447 INFO:tasks.rados.rados.0.smithi084.stdout:3481: done (1 left) 2024-09-20T22:55:56.447 INFO:tasks.rados.rados.0.smithi084.stdout:3483: done (0 left) 2024-09-20T22:55:56.448 INFO:tasks.rados.rados.0.smithi084.stdout:3484: read oid 1 snap -1 2024-09-20T22:55:56.448 INFO:tasks.rados.rados.0.smithi084.stdout:3484: expect deleted 2024-09-20T22:55:56.448 INFO:tasks.rados.rados.0.smithi084.stdout:3485: snap_remove snap 372 2024-09-20T22:55:56.483 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:55:56.504 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:56.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:56.505 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:55:56.606 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:55:56.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:56 smithi084 ceph-mon[29774]: 6.8 deep-scrub starts 2024-09-20T22:55:56.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:56 smithi084 ceph-mon[29774]: 6.8 deep-scrub ok 2024-09-20T22:55:56.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:56 smithi084 ceph-mon[29774]: osdmap e875: 8 total, 8 up, 8 in 2024-09-20T22:55:56.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:56 smithi084 ceph-mon[29774]: 6.18 scrub starts 2024-09-20T22:55:56.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:56 smithi084 ceph-mon[29774]: 6.18 scrub ok 2024-09-20T22:55:56.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:56 smithi084 ceph-mon[29774]: 4.13 scrub starts 2024-09-20T22:55:56.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:56 smithi084 ceph-mon[29774]: 4.13 scrub ok 2024-09-20T22:55:56.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:56 smithi084 ceph-mon[40024]: 6.8 deep-scrub starts 2024-09-20T22:55:56.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:56 smithi084 ceph-mon[40024]: 6.8 deep-scrub ok 2024-09-20T22:55:56.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:56 smithi084 ceph-mon[40024]: osdmap e875: 8 total, 8 up, 8 in 2024-09-20T22:55:56.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:56 smithi084 ceph-mon[40024]: 6.18 scrub starts 2024-09-20T22:55:56.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:56 smithi084 ceph-mon[40024]: 6.18 scrub ok 2024-09-20T22:55:56.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:56 smithi084 ceph-mon[40024]: 4.13 scrub starts 2024-09-20T22:55:56.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:56 smithi084 ceph-mon[40024]: 4.13 scrub ok 2024-09-20T22:55:56.626 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:56.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:56.627 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:55:56.628 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:56 smithi154 ceph-mon[33264]: 6.8 deep-scrub starts 2024-09-20T22:55:56.628 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:56 smithi154 ceph-mon[33264]: 6.8 deep-scrub ok 2024-09-20T22:55:56.628 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:56 smithi154 ceph-mon[33264]: osdmap e875: 8 total, 8 up, 8 in 2024-09-20T22:55:56.628 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:56 smithi154 ceph-mon[33264]: 6.18 scrub starts 2024-09-20T22:55:56.628 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:56 smithi154 ceph-mon[33264]: 6.18 scrub ok 2024-09-20T22:55:56.628 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:56 smithi154 ceph-mon[33264]: 4.13 scrub starts 2024-09-20T22:55:56.628 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:56 smithi154 ceph-mon[33264]: 4.13 scrub ok 2024-09-20T22:55:56.728 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:55:56.729 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-20T22:55:56.729 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [], 2024-09-20T22:55:56.729 INFO:teuthology.orchestra.run.smithi154.stdout: "num_ops": 0 2024-09-20T22:55:56.729 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-20T22:55:56.743 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-20T22:55:56.743 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-20T22:55:56.744 INFO:teuthology.orchestra.run.smithi084.stdout: "epoch": 876, 2024-09-20T22:55:56.744 INFO:teuthology.orchestra.run.smithi084.stdout: "fsid": "a9c0bc70-77a0-11ef-baf5-efdc52797490", 2024-09-20T22:55:56.744 INFO:teuthology.orchestra.run.smithi084.stdout: "created": "2024-09-20T22:36:30.240626+0000", 2024-09-20T22:55:56.744 INFO:teuthology.orchestra.run.smithi084.stdout: "modified": "2024-09-20T22:55:56.427021+0000", 2024-09-20T22:55:56.744 INFO:teuthology.orchestra.run.smithi084.stdout: "last_up_change": "2024-09-20T22:41:27.098565+0000", 2024-09-20T22:55:56.744 INFO:teuthology.orchestra.run.smithi084.stdout: "last_in_change": "2024-09-20T22:54:16.306377+0000", 2024-09-20T22:55:56.744 INFO:teuthology.orchestra.run.smithi084.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-20T22:55:56.744 INFO:teuthology.orchestra.run.smithi084.stdout: "flags_num": 5799936, 2024-09-20T22:55:56.744 INFO:teuthology.orchestra.run.smithi084.stdout: "flags_set": [ 2024-09-20T22:55:56.744 INFO:teuthology.orchestra.run.smithi084.stdout: "pglog_hardlimit", 2024-09-20T22:55:56.744 INFO:teuthology.orchestra.run.smithi084.stdout: "purged_snapdirs", 2024-09-20T22:55:56.744 INFO:teuthology.orchestra.run.smithi084.stdout: "recovery_deletes", 2024-09-20T22:55:56.744 INFO:teuthology.orchestra.run.smithi084.stdout: "sortbitwise" 2024-09-20T22:55:56.745 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2024-09-20T22:55:56.745 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_version": 18, 2024-09-20T22:55:56.745 INFO:teuthology.orchestra.run.smithi084.stdout: "full_ratio": 0.94999998807907104, 2024-09-20T22:55:56.745 INFO:teuthology.orchestra.run.smithi084.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-20T22:55:56.745 INFO:teuthology.orchestra.run.smithi084.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-20T22:55:56.745 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_snapshot": "", 2024-09-20T22:55:56.745 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_max": 7, 2024-09-20T22:55:56.745 INFO:teuthology.orchestra.run.smithi084.stdout: "max_osd": 8, 2024-09-20T22:55:56.745 INFO:teuthology.orchestra.run.smithi084.stdout: "require_min_compat_client": "luminous", 2024-09-20T22:55:56.745 INFO:teuthology.orchestra.run.smithi084.stdout: "min_compat_client": "luminous", 2024-09-20T22:55:56.745 INFO:teuthology.orchestra.run.smithi084.stdout: "require_osd_release": "squid", 2024-09-20T22:55:56.746 INFO:teuthology.orchestra.run.smithi084.stdout: "allow_crimson": false, 2024-09-20T22:55:56.746 INFO:teuthology.orchestra.run.smithi084.stdout: "pools": [ 2024-09-20T22:55:56.746 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:56.746 INFO:teuthology.orchestra.run.smithi084.stdout: "pool": 1, 2024-09-20T22:55:56.746 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_name": ".mgr", 2024-09-20T22:55:56.746 INFO:teuthology.orchestra.run.smithi084.stdout: "create_time": "2024-09-20T22:39:21.984381+0000", 2024-09-20T22:55:56.746 INFO:teuthology.orchestra.run.smithi084.stdout: "flags": 1, 2024-09-20T22:55:56.746 INFO:teuthology.orchestra.run.smithi084.stdout: "flags_names": "hashpspool", 2024-09-20T22:55:56.746 INFO:teuthology.orchestra.run.smithi084.stdout: "type": 1, 2024-09-20T22:55:56.746 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 3, 2024-09-20T22:55:56.746 INFO:teuthology.orchestra.run.smithi084.stdout: "min_size": 2, 2024-09-20T22:55:56.746 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_rule": 0, 2024-09-20T22:55:56.747 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_count": 0, 2024-09-20T22:55:56.747 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_target": 0, 2024-09-20T22:55:56.747 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_barrier": 0, 2024-09-20T22:55:56.747 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-20T22:55:56.747 INFO:teuthology.orchestra.run.smithi084.stdout: "is_stretch_pool": false, 2024-09-20T22:55:56.747 INFO:teuthology.orchestra.run.smithi084.stdout: "object_hash": 2, 2024-09-20T22:55:56.747 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_autoscale_mode": "off", 2024-09-20T22:55:56.747 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num": 11, 2024-09-20T22:55:56.747 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num": 11, 2024-09-20T22:55:56.747 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num_target": 11, 2024-09-20T22:55:56.747 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_target": 11, 2024-09-20T22:55:56.747 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_pending": 11, 2024-09-20T22:55:56.748 INFO:teuthology.orchestra.run.smithi084.stdout: "last_pg_merge_meta": { 2024-09-20T22:55:56.748 INFO:teuthology.orchestra.run.smithi084.stdout: "source_pgid": "1.b", 2024-09-20T22:55:56.748 INFO:teuthology.orchestra.run.smithi084.stdout: "ready_epoch": 299, 2024-09-20T22:55:56.748 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_started": 298, 2024-09-20T22:55:56.748 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_clean": 298, 2024-09-20T22:55:56.748 INFO:teuthology.orchestra.run.smithi084.stdout: "source_version": "77'128", 2024-09-20T22:55:56.748 INFO:teuthology.orchestra.run.smithi084.stdout: "target_version": "77'128" 2024-09-20T22:55:56.748 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:56.748 INFO:teuthology.orchestra.run.smithi084.stdout: "last_change": "769", 2024-09-20T22:55:56.748 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend": "0", 2024-09-20T22:55:56.748 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_prenautilus": "299", 2024-09-20T22:55:56.748 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_preluminous": "297", 2024-09-20T22:55:56.749 INFO:teuthology.orchestra.run.smithi084.stdout: "auid": 0, 2024-09-20T22:55:56.749 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_mode": "selfmanaged", 2024-09-20T22:55:56.749 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_seq": 0, 2024-09-20T22:55:56.749 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_epoch": 0, 2024-09-20T22:55:56.749 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_snaps": [], 2024-09-20T22:55:56.749 INFO:teuthology.orchestra.run.smithi084.stdout: "removed_snaps": "[]", 2024-09-20T22:55:56.749 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_bytes": 0, 2024-09-20T22:55:56.749 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_objects": 0, 2024-09-20T22:55:56.749 INFO:teuthology.orchestra.run.smithi084.stdout: "tiers": [], 2024-09-20T22:55:56.749 INFO:teuthology.orchestra.run.smithi084.stdout: "tier_of": -1, 2024-09-20T22:55:56.749 INFO:teuthology.orchestra.run.smithi084.stdout: "read_tier": -1, 2024-09-20T22:55:56.749 INFO:teuthology.orchestra.run.smithi084.stdout: "write_tier": -1, 2024-09-20T22:55:56.750 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_mode": "none", 2024-09-20T22:55:56.750 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_bytes": 0, 2024-09-20T22:55:56.750 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_objects": 0, 2024-09-20T22:55:56.750 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-20T22:55:56.750 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-20T22:55:56.750 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-20T22:55:56.750 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_flush_age": 0, 2024-09-20T22:55:56.750 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_evict_age": 0, 2024-09-20T22:55:56.750 INFO:teuthology.orchestra.run.smithi084.stdout: "erasure_code_profile": "", 2024-09-20T22:55:56.750 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_params": { 2024-09-20T22:55:56.751 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "none" 2024-09-20T22:55:56.751 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:56.751 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_period": 0, 2024-09-20T22:55:56.751 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_count": 0, 2024-09-20T22:55:56.751 INFO:teuthology.orchestra.run.smithi084.stdout: "use_gmt_hitset": true, 2024-09-20T22:55:56.751 INFO:teuthology.orchestra.run.smithi084.stdout: "min_read_recency_for_promote": 0, 2024-09-20T22:55:56.751 INFO:teuthology.orchestra.run.smithi084.stdout: "min_write_recency_for_promote": 0, 2024-09-20T22:55:56.751 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_grade_decay_rate": 0, 2024-09-20T22:55:56.751 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_search_last_n": 0, 2024-09-20T22:55:56.751 INFO:teuthology.orchestra.run.smithi084.stdout: "grade_table": [], 2024-09-20T22:55:56.751 INFO:teuthology.orchestra.run.smithi084.stdout: "stripe_width": 0, 2024-09-20T22:55:56.752 INFO:teuthology.orchestra.run.smithi084.stdout: "expected_num_objects": 0, 2024-09-20T22:55:56.752 INFO:teuthology.orchestra.run.smithi084.stdout: "fast_read": false, 2024-09-20T22:55:56.752 INFO:teuthology.orchestra.run.smithi084.stdout: "options": { 2024-09-20T22:55:56.752 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_max": 32, 2024-09-20T22:55:56.752 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_min": 1 2024-09-20T22:55:56.752 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:56.752 INFO:teuthology.orchestra.run.smithi084.stdout: "application_metadata": { 2024-09-20T22:55:56.752 INFO:teuthology.orchestra.run.smithi084.stdout: "mgr": {} 2024-09-20T22:55:56.753 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:56.753 INFO:teuthology.orchestra.run.smithi084.stdout: "read_balance": { 2024-09-20T22:55:56.753 INFO:teuthology.orchestra.run.smithi084.stdout: "score_type": "Fair distribution", 2024-09-20T22:55:56.753 INFO:teuthology.orchestra.run.smithi084.stdout: "score_acting": 1.9099999666213989, 2024-09-20T22:55:56.753 INFO:teuthology.orchestra.run.smithi084.stdout: "score_stable": 1.9099999666213989, 2024-09-20T22:55:56.753 INFO:teuthology.orchestra.run.smithi084.stdout: "optimal_score": 1.1399999856948853, 2024-09-20T22:55:56.753 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_acting": 2.1800000667572021, 2024-09-20T22:55:56.753 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_stable": 2.1800000667572021, 2024-09-20T22:55:56.753 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity_weighted": 0.875, 2024-09-20T22:55:56.753 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity": 1, 2024-09-20T22:55:56.754 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-09-20T22:55:56.754 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:55:56.754 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:56.754 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:56.754 INFO:teuthology.orchestra.run.smithi084.stdout: "pool": 2, 2024-09-20T22:55:56.754 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_name": "datapool", 2024-09-20T22:55:56.754 INFO:teuthology.orchestra.run.smithi084.stdout: "create_time": "2024-09-20T22:41:35.820081+0000", 2024-09-20T22:55:56.754 INFO:teuthology.orchestra.run.smithi084.stdout: "flags": 8193, 2024-09-20T22:55:56.755 INFO:teuthology.orchestra.run.smithi084.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-20T22:55:56.755 INFO:teuthology.orchestra.run.smithi084.stdout: "type": 1, 2024-09-20T22:55:56.755 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 3, 2024-09-20T22:55:56.755 INFO:teuthology.orchestra.run.smithi084.stdout: "min_size": 2, 2024-09-20T22:55:56.755 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_rule": 0, 2024-09-20T22:55:56.755 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_count": 0, 2024-09-20T22:55:56.755 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_target": 0, 2024-09-20T22:55:56.755 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_barrier": 0, 2024-09-20T22:55:56.755 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-20T22:55:56.755 INFO:teuthology.orchestra.run.smithi084.stdout: "is_stretch_pool": false, 2024-09-20T22:55:56.755 INFO:teuthology.orchestra.run.smithi084.stdout: "object_hash": 2, 2024-09-20T22:55:56.755 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_autoscale_mode": "off", 2024-09-20T22:55:56.756 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num": 3, 2024-09-20T22:55:56.756 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num": 3, 2024-09-20T22:55:56.756 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num_target": 3, 2024-09-20T22:55:56.756 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_target": 3, 2024-09-20T22:55:56.756 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_pending": 3, 2024-09-20T22:55:56.756 INFO:teuthology.orchestra.run.smithi084.stdout: "last_pg_merge_meta": { 2024-09-20T22:55:56.756 INFO:teuthology.orchestra.run.smithi084.stdout: "source_pgid": "0.0", 2024-09-20T22:55:56.756 INFO:teuthology.orchestra.run.smithi084.stdout: "ready_epoch": 0, 2024-09-20T22:55:56.756 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_started": 0, 2024-09-20T22:55:56.756 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_clean": 0, 2024-09-20T22:55:56.756 INFO:teuthology.orchestra.run.smithi084.stdout: "source_version": "0'0", 2024-09-20T22:55:56.756 INFO:teuthology.orchestra.run.smithi084.stdout: "target_version": "0'0" 2024-09-20T22:55:56.756 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:56.757 INFO:teuthology.orchestra.run.smithi084.stdout: "last_change": "69", 2024-09-20T22:55:56.757 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend": "0", 2024-09-20T22:55:56.757 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-20T22:55:56.757 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_preluminous": "0", 2024-09-20T22:55:56.757 INFO:teuthology.orchestra.run.smithi084.stdout: "auid": 0, 2024-09-20T22:55:56.757 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_mode": "selfmanaged", 2024-09-20T22:55:56.757 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_seq": 2, 2024-09-20T22:55:56.757 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_epoch": 69, 2024-09-20T22:55:56.757 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_snaps": [], 2024-09-20T22:55:56.757 INFO:teuthology.orchestra.run.smithi084.stdout: "removed_snaps": "[]", 2024-09-20T22:55:56.757 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_bytes": 0, 2024-09-20T22:55:56.757 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_objects": 0, 2024-09-20T22:55:56.757 INFO:teuthology.orchestra.run.smithi084.stdout: "tiers": [], 2024-09-20T22:55:56.757 INFO:teuthology.orchestra.run.smithi084.stdout: "tier_of": -1, 2024-09-20T22:55:56.758 INFO:teuthology.orchestra.run.smithi084.stdout: "read_tier": -1, 2024-09-20T22:55:56.758 INFO:teuthology.orchestra.run.smithi084.stdout: "write_tier": -1, 2024-09-20T22:55:56.758 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_mode": "none", 2024-09-20T22:55:56.758 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_bytes": 0, 2024-09-20T22:55:56.758 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_objects": 0, 2024-09-20T22:55:56.758 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-20T22:55:56.758 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-20T22:55:56.758 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-20T22:55:56.758 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_flush_age": 0, 2024-09-20T22:55:56.758 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_evict_age": 0, 2024-09-20T22:55:56.758 INFO:teuthology.orchestra.run.smithi084.stdout: "erasure_code_profile": "", 2024-09-20T22:55:56.758 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_params": { 2024-09-20T22:55:56.758 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "none" 2024-09-20T22:55:56.759 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:56.759 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_period": 0, 2024-09-20T22:55:56.759 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_count": 0, 2024-09-20T22:55:56.759 INFO:teuthology.orchestra.run.smithi084.stdout: "use_gmt_hitset": true, 2024-09-20T22:55:56.759 INFO:teuthology.orchestra.run.smithi084.stdout: "min_read_recency_for_promote": 0, 2024-09-20T22:55:56.759 INFO:teuthology.orchestra.run.smithi084.stdout: "min_write_recency_for_promote": 0, 2024-09-20T22:55:56.759 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_grade_decay_rate": 0, 2024-09-20T22:55:56.759 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_search_last_n": 0, 2024-09-20T22:55:56.759 INFO:teuthology.orchestra.run.smithi084.stdout: "grade_table": [], 2024-09-20T22:55:56.759 INFO:teuthology.orchestra.run.smithi084.stdout: "stripe_width": 0, 2024-09-20T22:55:56.759 INFO:teuthology.orchestra.run.smithi084.stdout: "expected_num_objects": 0, 2024-09-20T22:55:56.759 INFO:teuthology.orchestra.run.smithi084.stdout: "fast_read": false, 2024-09-20T22:55:56.759 INFO:teuthology.orchestra.run.smithi084.stdout: "options": {}, 2024-09-20T22:55:56.759 INFO:teuthology.orchestra.run.smithi084.stdout: "application_metadata": { 2024-09-20T22:55:56.759 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd": {} 2024-09-20T22:55:56.760 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:56.760 INFO:teuthology.orchestra.run.smithi084.stdout: "read_balance": { 2024-09-20T22:55:56.760 INFO:teuthology.orchestra.run.smithi084.stdout: "score_type": "Fair distribution", 2024-09-20T22:55:56.760 INFO:teuthology.orchestra.run.smithi084.stdout: "score_acting": 2.3299999237060547, 2024-09-20T22:55:56.760 INFO:teuthology.orchestra.run.smithi084.stdout: "score_stable": 2.3299999237060547, 2024-09-20T22:55:56.760 INFO:teuthology.orchestra.run.smithi084.stdout: "optimal_score": 1, 2024-09-20T22:55:56.760 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_acting": 2.3299999237060547, 2024-09-20T22:55:56.760 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_stable": 2.3299999237060547, 2024-09-20T22:55:56.760 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity_weighted": 0.875, 2024-09-20T22:55:56.760 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity": 1, 2024-09-20T22:55:56.760 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-09-20T22:55:56.760 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:55:56.760 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:56.760 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:56.760 INFO:teuthology.orchestra.run.smithi084.stdout: "pool": 3, 2024-09-20T22:55:56.761 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_name": ".rgw.root", 2024-09-20T22:55:56.761 INFO:teuthology.orchestra.run.smithi084.stdout: "create_time": "2024-09-20T22:41:36.546389+0000", 2024-09-20T22:55:56.761 INFO:teuthology.orchestra.run.smithi084.stdout: "flags": 1, 2024-09-20T22:55:56.761 INFO:teuthology.orchestra.run.smithi084.stdout: "flags_names": "hashpspool", 2024-09-20T22:55:56.761 INFO:teuthology.orchestra.run.smithi084.stdout: "type": 1, 2024-09-20T22:55:56.761 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 3, 2024-09-20T22:55:56.761 INFO:teuthology.orchestra.run.smithi084.stdout: "min_size": 2, 2024-09-20T22:55:56.761 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_rule": 0, 2024-09-20T22:55:56.761 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_count": 0, 2024-09-20T22:55:56.761 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_target": 0, 2024-09-20T22:55:56.761 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_barrier": 0, 2024-09-20T22:55:56.761 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-20T22:55:56.761 INFO:teuthology.orchestra.run.smithi084.stdout: "is_stretch_pool": false, 2024-09-20T22:55:56.761 INFO:teuthology.orchestra.run.smithi084.stdout: "object_hash": 2, 2024-09-20T22:55:56.761 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_autoscale_mode": "off", 2024-09-20T22:55:56.762 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num": 32, 2024-09-20T22:55:56.762 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num": 32, 2024-09-20T22:55:56.762 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num_target": 32, 2024-09-20T22:55:56.762 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_target": 32, 2024-09-20T22:55:56.762 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_pending": 32, 2024-09-20T22:55:56.762 INFO:teuthology.orchestra.run.smithi084.stdout: "last_pg_merge_meta": { 2024-09-20T22:55:56.762 INFO:teuthology.orchestra.run.smithi084.stdout: "source_pgid": "0.0", 2024-09-20T22:55:56.762 INFO:teuthology.orchestra.run.smithi084.stdout: "ready_epoch": 0, 2024-09-20T22:55:56.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:56.763 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_started": 0, 2024-09-20T22:55:56.764 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_clean": 0, 2024-09-20T22:55:56.764 INFO:teuthology.orchestra.run.smithi084.stdout: "source_version": "0'0", 2024-09-20T22:55:56.764 INFO:teuthology.orchestra.run.smithi084.stdout: "target_version": "0'0" 2024-09-20T22:55:56.764 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:56.764 INFO:teuthology.orchestra.run.smithi084.stdout: "last_change": "68", 2024-09-20T22:55:56.764 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend": "0", 2024-09-20T22:55:56.764 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-20T22:55:56.764 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_preluminous": "0", 2024-09-20T22:55:56.764 INFO:teuthology.orchestra.run.smithi084.stdout: "auid": 0, 2024-09-20T22:55:56.764 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_mode": "selfmanaged", 2024-09-20T22:55:56.764 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_seq": 0, 2024-09-20T22:55:56.764 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_epoch": 0, 2024-09-20T22:55:56.764 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_snaps": [], 2024-09-20T22:55:56.764 INFO:teuthology.orchestra.run.smithi084.stdout: "removed_snaps": "[]", 2024-09-20T22:55:56.765 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_bytes": 0, 2024-09-20T22:55:56.765 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_objects": 0, 2024-09-20T22:55:56.765 INFO:teuthology.orchestra.run.smithi084.stdout: "tiers": [], 2024-09-20T22:55:56.765 INFO:teuthology.orchestra.run.smithi084.stdout: "tier_of": -1, 2024-09-20T22:55:56.765 INFO:teuthology.orchestra.run.smithi084.stdout: "read_tier": -1, 2024-09-20T22:55:56.765 INFO:teuthology.orchestra.run.smithi084.stdout: "write_tier": -1, 2024-09-20T22:55:56.765 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_mode": "none", 2024-09-20T22:55:56.765 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_bytes": 0, 2024-09-20T22:55:56.765 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_objects": 0, 2024-09-20T22:55:56.765 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-20T22:55:56.765 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-20T22:55:56.765 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-20T22:55:56.765 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_flush_age": 0, 2024-09-20T22:55:56.765 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_evict_age": 0, 2024-09-20T22:55:56.765 INFO:teuthology.orchestra.run.smithi084.stdout: "erasure_code_profile": "", 2024-09-20T22:55:56.766 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_params": { 2024-09-20T22:55:56.766 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "none" 2024-09-20T22:55:56.766 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:56.766 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_period": 0, 2024-09-20T22:55:56.766 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_count": 0, 2024-09-20T22:55:56.766 INFO:teuthology.orchestra.run.smithi084.stdout: "use_gmt_hitset": true, 2024-09-20T22:55:56.766 INFO:teuthology.orchestra.run.smithi084.stdout: "min_read_recency_for_promote": 0, 2024-09-20T22:55:56.766 INFO:teuthology.orchestra.run.smithi084.stdout: "min_write_recency_for_promote": 0, 2024-09-20T22:55:56.766 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_grade_decay_rate": 0, 2024-09-20T22:55:56.766 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_search_last_n": 0, 2024-09-20T22:55:56.766 INFO:teuthology.orchestra.run.smithi084.stdout: "grade_table": [], 2024-09-20T22:55:56.766 INFO:teuthology.orchestra.run.smithi084.stdout: "stripe_width": 0, 2024-09-20T22:55:56.766 INFO:teuthology.orchestra.run.smithi084.stdout: "expected_num_objects": 0, 2024-09-20T22:55:56.766 INFO:teuthology.orchestra.run.smithi084.stdout: "fast_read": false, 2024-09-20T22:55:56.767 INFO:teuthology.orchestra.run.smithi084.stdout: "options": {}, 2024-09-20T22:55:56.767 INFO:teuthology.orchestra.run.smithi084.stdout: "application_metadata": { 2024-09-20T22:55:56.767 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw": {} 2024-09-20T22:55:56.767 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:56.767 INFO:teuthology.orchestra.run.smithi084.stdout: "read_balance": { 2024-09-20T22:55:56.767 INFO:teuthology.orchestra.run.smithi084.stdout: "score_type": "Fair distribution", 2024-09-20T22:55:56.767 INFO:teuthology.orchestra.run.smithi084.stdout: "score_acting": 1.5399999618530273, 2024-09-20T22:55:56.767 INFO:teuthology.orchestra.run.smithi084.stdout: "score_stable": 1.5399999618530273, 2024-09-20T22:55:56.767 INFO:teuthology.orchestra.run.smithi084.stdout: "optimal_score": 1.1399999856948853, 2024-09-20T22:55:56.767 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_acting": 1.75, 2024-09-20T22:55:56.767 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_stable": 1.75, 2024-09-20T22:55:56.767 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity_weighted": 0.875, 2024-09-20T22:55:56.767 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity": 1, 2024-09-20T22:55:56.767 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-09-20T22:55:56.767 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:55:56.768 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:56.768 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:56.768 INFO:teuthology.orchestra.run.smithi084.stdout: "pool": 4, 2024-09-20T22:55:56.768 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_name": "default.rgw.log", 2024-09-20T22:55:56.768 INFO:teuthology.orchestra.run.smithi084.stdout: "create_time": "2024-09-20T22:41:38.454640+0000", 2024-09-20T22:55:56.768 INFO:teuthology.orchestra.run.smithi084.stdout: "flags": 1, 2024-09-20T22:55:56.768 INFO:teuthology.orchestra.run.smithi084.stdout: "flags_names": "hashpspool", 2024-09-20T22:55:56.768 INFO:teuthology.orchestra.run.smithi084.stdout: "type": 1, 2024-09-20T22:55:56.768 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 3, 2024-09-20T22:55:56.768 INFO:teuthology.orchestra.run.smithi084.stdout: "min_size": 2, 2024-09-20T22:55:56.768 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_rule": 0, 2024-09-20T22:55:56.768 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_count": 0, 2024-09-20T22:55:56.768 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_target": 0, 2024-09-20T22:55:56.768 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_barrier": 0, 2024-09-20T22:55:56.768 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-20T22:55:56.769 INFO:teuthology.orchestra.run.smithi084.stdout: "is_stretch_pool": false, 2024-09-20T22:55:56.769 INFO:teuthology.orchestra.run.smithi084.stdout: "object_hash": 2, 2024-09-20T22:55:56.769 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_autoscale_mode": "off", 2024-09-20T22:55:56.769 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num": 32, 2024-09-20T22:55:56.769 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num": 32, 2024-09-20T22:55:56.769 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num_target": 32, 2024-09-20T22:55:56.769 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_target": 32, 2024-09-20T22:55:56.769 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_pending": 32, 2024-09-20T22:55:56.769 INFO:teuthology.orchestra.run.smithi084.stdout: "last_pg_merge_meta": { 2024-09-20T22:55:56.769 INFO:teuthology.orchestra.run.smithi084.stdout: "source_pgid": "0.0", 2024-09-20T22:55:56.769 INFO:teuthology.orchestra.run.smithi084.stdout: "ready_epoch": 0, 2024-09-20T22:55:56.769 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_started": 0, 2024-09-20T22:55:56.769 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_clean": 0, 2024-09-20T22:55:56.769 INFO:teuthology.orchestra.run.smithi084.stdout: "source_version": "0'0", 2024-09-20T22:55:56.770 INFO:teuthology.orchestra.run.smithi084.stdout: "target_version": "0'0" 2024-09-20T22:55:56.770 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:56.770 INFO:teuthology.orchestra.run.smithi084.stdout: "last_change": "70", 2024-09-20T22:55:56.770 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend": "0", 2024-09-20T22:55:56.770 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-20T22:55:56.770 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_preluminous": "0", 2024-09-20T22:55:56.770 INFO:teuthology.orchestra.run.smithi084.stdout: "auid": 0, 2024-09-20T22:55:56.770 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_mode": "selfmanaged", 2024-09-20T22:55:56.770 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_seq": 0, 2024-09-20T22:55:56.770 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_epoch": 0, 2024-09-20T22:55:56.770 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_snaps": [], 2024-09-20T22:55:56.770 INFO:teuthology.orchestra.run.smithi084.stdout: "removed_snaps": "[]", 2024-09-20T22:55:56.770 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_bytes": 0, 2024-09-20T22:55:56.771 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_objects": 0, 2024-09-20T22:55:56.771 INFO:teuthology.orchestra.run.smithi084.stdout: "tiers": [], 2024-09-20T22:55:56.771 INFO:teuthology.orchestra.run.smithi084.stdout: "tier_of": -1, 2024-09-20T22:55:56.771 INFO:teuthology.orchestra.run.smithi084.stdout: "read_tier": -1, 2024-09-20T22:55:56.771 INFO:teuthology.orchestra.run.smithi084.stdout: "write_tier": -1, 2024-09-20T22:55:56.771 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_mode": "none", 2024-09-20T22:55:56.771 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_bytes": 0, 2024-09-20T22:55:56.771 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_objects": 0, 2024-09-20T22:55:56.771 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-20T22:55:56.771 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-20T22:55:56.771 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-20T22:55:56.771 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_flush_age": 0, 2024-09-20T22:55:56.771 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_evict_age": 0, 2024-09-20T22:55:56.772 INFO:teuthology.orchestra.run.smithi084.stdout: "erasure_code_profile": "", 2024-09-20T22:55:56.772 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_params": { 2024-09-20T22:55:56.772 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "none" 2024-09-20T22:55:56.772 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:56.772 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_period": 0, 2024-09-20T22:55:56.772 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_count": 0, 2024-09-20T22:55:56.772 INFO:teuthology.orchestra.run.smithi084.stdout: "use_gmt_hitset": true, 2024-09-20T22:55:56.772 INFO:teuthology.orchestra.run.smithi084.stdout: "min_read_recency_for_promote": 0, 2024-09-20T22:55:56.772 INFO:teuthology.orchestra.run.smithi084.stdout: "min_write_recency_for_promote": 0, 2024-09-20T22:55:56.772 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_grade_decay_rate": 0, 2024-09-20T22:55:56.772 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_search_last_n": 0, 2024-09-20T22:55:56.772 INFO:teuthology.orchestra.run.smithi084.stdout: "grade_table": [], 2024-09-20T22:55:56.772 INFO:teuthology.orchestra.run.smithi084.stdout: "stripe_width": 0, 2024-09-20T22:55:56.772 INFO:teuthology.orchestra.run.smithi084.stdout: "expected_num_objects": 0, 2024-09-20T22:55:56.773 INFO:teuthology.orchestra.run.smithi084.stdout: "fast_read": false, 2024-09-20T22:55:56.773 INFO:teuthology.orchestra.run.smithi084.stdout: "options": {}, 2024-09-20T22:55:56.773 INFO:teuthology.orchestra.run.smithi084.stdout: "application_metadata": { 2024-09-20T22:55:56.773 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw": {} 2024-09-20T22:55:56.773 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:56.773 INFO:teuthology.orchestra.run.smithi084.stdout: "read_balance": { 2024-09-20T22:55:56.773 INFO:teuthology.orchestra.run.smithi084.stdout: "score_type": "Fair distribution", 2024-09-20T22:55:56.773 INFO:teuthology.orchestra.run.smithi084.stdout: "score_acting": 1.9700000286102295, 2024-09-20T22:55:56.773 INFO:teuthology.orchestra.run.smithi084.stdout: "score_stable": 1.9700000286102295, 2024-09-20T22:55:56.773 INFO:teuthology.orchestra.run.smithi084.stdout: "optimal_score": 1.1399999856948853, 2024-09-20T22:55:56.773 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_acting": 2.25, 2024-09-20T22:55:56.773 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_stable": 2.25, 2024-09-20T22:55:56.773 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity_weighted": 0.875, 2024-09-20T22:55:56.773 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity": 1, 2024-09-20T22:55:56.773 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-09-20T22:55:56.774 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:55:56.774 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:56.774 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:56.774 INFO:teuthology.orchestra.run.smithi084.stdout: "pool": 5, 2024-09-20T22:55:56.774 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_name": "default.rgw.control", 2024-09-20T22:55:56.774 INFO:teuthology.orchestra.run.smithi084.stdout: "create_time": "2024-09-20T22:41:40.407129+0000", 2024-09-20T22:55:56.774 INFO:teuthology.orchestra.run.smithi084.stdout: "flags": 1, 2024-09-20T22:55:56.774 INFO:teuthology.orchestra.run.smithi084.stdout: "flags_names": "hashpspool", 2024-09-20T22:55:56.774 INFO:teuthology.orchestra.run.smithi084.stdout: "type": 1, 2024-09-20T22:55:56.775 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 3, 2024-09-20T22:55:56.775 INFO:teuthology.orchestra.run.smithi084.stdout: "min_size": 2, 2024-09-20T22:55:56.775 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_rule": 0, 2024-09-20T22:55:56.775 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_count": 0, 2024-09-20T22:55:56.775 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_target": 0, 2024-09-20T22:55:56.775 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_barrier": 0, 2024-09-20T22:55:56.775 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-20T22:55:56.775 INFO:teuthology.orchestra.run.smithi084.stdout: "is_stretch_pool": false, 2024-09-20T22:55:56.775 INFO:teuthology.orchestra.run.smithi084.stdout: "object_hash": 2, 2024-09-20T22:55:56.775 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_autoscale_mode": "off", 2024-09-20T22:55:56.775 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num": 32, 2024-09-20T22:55:56.775 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num": 32, 2024-09-20T22:55:56.775 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num_target": 32, 2024-09-20T22:55:56.775 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_target": 32, 2024-09-20T22:55:56.775 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_pending": 32, 2024-09-20T22:55:56.776 INFO:teuthology.orchestra.run.smithi084.stdout: "last_pg_merge_meta": { 2024-09-20T22:55:56.776 INFO:teuthology.orchestra.run.smithi084.stdout: "source_pgid": "0.0", 2024-09-20T22:55:56.776 INFO:teuthology.orchestra.run.smithi084.stdout: "ready_epoch": 0, 2024-09-20T22:55:56.776 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_started": 0, 2024-09-20T22:55:56.776 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_clean": 0, 2024-09-20T22:55:56.776 INFO:teuthology.orchestra.run.smithi084.stdout: "source_version": "0'0", 2024-09-20T22:55:56.776 INFO:teuthology.orchestra.run.smithi084.stdout: "target_version": "0'0" 2024-09-20T22:55:56.776 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:56.776 INFO:teuthology.orchestra.run.smithi084.stdout: "last_change": "72", 2024-09-20T22:55:56.776 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend": "0", 2024-09-20T22:55:56.776 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-20T22:55:56.776 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_preluminous": "0", 2024-09-20T22:55:56.776 INFO:teuthology.orchestra.run.smithi084.stdout: "auid": 0, 2024-09-20T22:55:56.776 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_mode": "selfmanaged", 2024-09-20T22:55:56.777 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_seq": 0, 2024-09-20T22:55:56.777 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_epoch": 0, 2024-09-20T22:55:56.777 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_snaps": [], 2024-09-20T22:55:56.777 INFO:teuthology.orchestra.run.smithi084.stdout: "removed_snaps": "[]", 2024-09-20T22:55:56.777 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_bytes": 0, 2024-09-20T22:55:56.777 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_objects": 0, 2024-09-20T22:55:56.777 INFO:teuthology.orchestra.run.smithi084.stdout: "tiers": [], 2024-09-20T22:55:56.777 INFO:teuthology.orchestra.run.smithi084.stdout: "tier_of": -1, 2024-09-20T22:55:56.777 INFO:teuthology.orchestra.run.smithi084.stdout: "read_tier": -1, 2024-09-20T22:55:56.777 INFO:teuthology.orchestra.run.smithi084.stdout: "write_tier": -1, 2024-09-20T22:55:56.777 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_mode": "none", 2024-09-20T22:55:56.777 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_bytes": 0, 2024-09-20T22:55:56.777 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_objects": 0, 2024-09-20T22:55:56.777 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-20T22:55:56.777 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-20T22:55:56.778 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-20T22:55:56.778 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_flush_age": 0, 2024-09-20T22:55:56.778 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_evict_age": 0, 2024-09-20T22:55:56.778 INFO:teuthology.orchestra.run.smithi084.stdout: "erasure_code_profile": "", 2024-09-20T22:55:56.778 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_params": { 2024-09-20T22:55:56.778 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "none" 2024-09-20T22:55:56.778 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:56.778 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_period": 0, 2024-09-20T22:55:56.778 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_count": 0, 2024-09-20T22:55:56.778 INFO:teuthology.orchestra.run.smithi084.stdout: "use_gmt_hitset": true, 2024-09-20T22:55:56.778 INFO:teuthology.orchestra.run.smithi084.stdout: "min_read_recency_for_promote": 0, 2024-09-20T22:55:56.778 INFO:teuthology.orchestra.run.smithi084.stdout: "min_write_recency_for_promote": 0, 2024-09-20T22:55:56.779 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_grade_decay_rate": 0, 2024-09-20T22:55:56.779 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_search_last_n": 0, 2024-09-20T22:55:56.779 INFO:teuthology.orchestra.run.smithi084.stdout: "grade_table": [], 2024-09-20T22:55:56.779 INFO:teuthology.orchestra.run.smithi084.stdout: "stripe_width": 0, 2024-09-20T22:55:56.779 INFO:teuthology.orchestra.run.smithi084.stdout: "expected_num_objects": 0, 2024-09-20T22:55:56.779 INFO:teuthology.orchestra.run.smithi084.stdout: "fast_read": false, 2024-09-20T22:55:56.779 INFO:teuthology.orchestra.run.smithi084.stdout: "options": {}, 2024-09-20T22:55:56.779 INFO:teuthology.orchestra.run.smithi084.stdout: "application_metadata": { 2024-09-20T22:55:56.779 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw": {} 2024-09-20T22:55:56.779 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:56.779 INFO:teuthology.orchestra.run.smithi084.stdout: "read_balance": { 2024-09-20T22:55:56.780 INFO:teuthology.orchestra.run.smithi084.stdout: "score_type": "Fair distribution", 2024-09-20T22:55:56.780 INFO:teuthology.orchestra.run.smithi084.stdout: "score_acting": 1.3200000524520874, 2024-09-20T22:55:56.780 INFO:teuthology.orchestra.run.smithi084.stdout: "score_stable": 1.3200000524520874, 2024-09-20T22:55:56.780 INFO:teuthology.orchestra.run.smithi084.stdout: "optimal_score": 1.1399999856948853, 2024-09-20T22:55:56.780 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_acting": 1.5, 2024-09-20T22:55:56.780 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_stable": 1.5, 2024-09-20T22:55:56.780 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity_weighted": 0.875, 2024-09-20T22:55:56.780 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity": 1, 2024-09-20T22:55:56.780 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-09-20T22:55:56.780 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:55:56.780 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:56.780 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:56.780 INFO:teuthology.orchestra.run.smithi084.stdout: "pool": 6, 2024-09-20T22:55:56.781 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_name": "default.rgw.meta", 2024-09-20T22:55:56.781 INFO:teuthology.orchestra.run.smithi084.stdout: "create_time": "2024-09-20T22:41:42.476716+0000", 2024-09-20T22:55:56.781 INFO:teuthology.orchestra.run.smithi084.stdout: "flags": 1, 2024-09-20T22:55:56.781 INFO:teuthology.orchestra.run.smithi084.stdout: "flags_names": "hashpspool", 2024-09-20T22:55:56.781 INFO:teuthology.orchestra.run.smithi084.stdout: "type": 1, 2024-09-20T22:55:56.781 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 3, 2024-09-20T22:55:56.781 INFO:teuthology.orchestra.run.smithi084.stdout: "min_size": 2, 2024-09-20T22:55:56.781 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_rule": 0, 2024-09-20T22:55:56.781 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_count": 0, 2024-09-20T22:55:56.781 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_target": 0, 2024-09-20T22:55:56.781 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_barrier": 0, 2024-09-20T22:55:56.782 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-20T22:55:56.782 INFO:teuthology.orchestra.run.smithi084.stdout: "is_stretch_pool": false, 2024-09-20T22:55:56.782 INFO:teuthology.orchestra.run.smithi084.stdout: "object_hash": 2, 2024-09-20T22:55:56.782 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_autoscale_mode": "off", 2024-09-20T22:55:56.782 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num": 32, 2024-09-20T22:55:56.782 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num": 32, 2024-09-20T22:55:56.782 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num_target": 32, 2024-09-20T22:55:56.782 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_target": 32, 2024-09-20T22:55:56.782 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_pending": 32, 2024-09-20T22:55:56.782 INFO:teuthology.orchestra.run.smithi084.stdout: "last_pg_merge_meta": { 2024-09-20T22:55:56.782 INFO:teuthology.orchestra.run.smithi084.stdout: "source_pgid": "0.0", 2024-09-20T22:55:56.782 INFO:teuthology.orchestra.run.smithi084.stdout: "ready_epoch": 0, 2024-09-20T22:55:56.783 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_started": 0, 2024-09-20T22:55:56.783 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_clean": 0, 2024-09-20T22:55:56.783 INFO:teuthology.orchestra.run.smithi084.stdout: "source_version": "0'0", 2024-09-20T22:55:56.783 INFO:teuthology.orchestra.run.smithi084.stdout: "target_version": "0'0" 2024-09-20T22:55:56.783 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:56.783 INFO:teuthology.orchestra.run.smithi084.stdout: "last_change": "74", 2024-09-20T22:55:56.783 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend": "0", 2024-09-20T22:55:56.783 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-20T22:55:56.783 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_preluminous": "0", 2024-09-20T22:55:56.783 INFO:teuthology.orchestra.run.smithi084.stdout: "auid": 0, 2024-09-20T22:55:56.783 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_mode": "selfmanaged", 2024-09-20T22:55:56.783 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_seq": 0, 2024-09-20T22:55:56.784 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_epoch": 0, 2024-09-20T22:55:56.784 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_snaps": [], 2024-09-20T22:55:56.784 INFO:teuthology.orchestra.run.smithi084.stdout: "removed_snaps": "[]", 2024-09-20T22:55:56.784 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_bytes": 0, 2024-09-20T22:55:56.784 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_objects": 0, 2024-09-20T22:55:56.784 INFO:teuthology.orchestra.run.smithi084.stdout: "tiers": [], 2024-09-20T22:55:56.784 INFO:teuthology.orchestra.run.smithi084.stdout: "tier_of": -1, 2024-09-20T22:55:56.784 INFO:teuthology.orchestra.run.smithi084.stdout: "read_tier": -1, 2024-09-20T22:55:56.784 INFO:teuthology.orchestra.run.smithi084.stdout: "write_tier": -1, 2024-09-20T22:55:56.784 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_mode": "none", 2024-09-20T22:55:56.785 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_bytes": 0, 2024-09-20T22:55:56.785 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_objects": 0, 2024-09-20T22:55:56.785 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-20T22:55:56.785 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-20T22:55:56.785 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-20T22:55:56.785 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_flush_age": 0, 2024-09-20T22:55:56.785 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_evict_age": 0, 2024-09-20T22:55:56.785 INFO:teuthology.orchestra.run.smithi084.stdout: "erasure_code_profile": "", 2024-09-20T22:55:56.785 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_params": { 2024-09-20T22:55:56.785 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "none" 2024-09-20T22:55:56.785 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:56.785 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_period": 0, 2024-09-20T22:55:56.786 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_count": 0, 2024-09-20T22:55:56.786 INFO:teuthology.orchestra.run.smithi084.stdout: "use_gmt_hitset": true, 2024-09-20T22:55:56.786 INFO:teuthology.orchestra.run.smithi084.stdout: "min_read_recency_for_promote": 0, 2024-09-20T22:55:56.786 INFO:teuthology.orchestra.run.smithi084.stdout: "min_write_recency_for_promote": 0, 2024-09-20T22:55:56.786 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_grade_decay_rate": 0, 2024-09-20T22:55:56.786 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_search_last_n": 0, 2024-09-20T22:55:56.786 INFO:teuthology.orchestra.run.smithi084.stdout: "grade_table": [], 2024-09-20T22:55:56.786 INFO:teuthology.orchestra.run.smithi084.stdout: "stripe_width": 0, 2024-09-20T22:55:56.786 INFO:teuthology.orchestra.run.smithi084.stdout: "expected_num_objects": 0, 2024-09-20T22:55:56.787 INFO:teuthology.orchestra.run.smithi084.stdout: "fast_read": false, 2024-09-20T22:55:56.787 INFO:teuthology.orchestra.run.smithi084.stdout: "options": { 2024-09-20T22:55:56.787 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_autoscale_bias": 4 2024-09-20T22:55:56.787 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:56.787 INFO:teuthology.orchestra.run.smithi084.stdout: "application_metadata": { 2024-09-20T22:55:56.787 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw": {} 2024-09-20T22:55:56.787 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:56.787 INFO:teuthology.orchestra.run.smithi084.stdout: "read_balance": { 2024-09-20T22:55:56.787 INFO:teuthology.orchestra.run.smithi084.stdout: "score_type": "Fair distribution", 2024-09-20T22:55:56.787 INFO:teuthology.orchestra.run.smithi084.stdout: "score_acting": 1.75, 2024-09-20T22:55:56.787 INFO:teuthology.orchestra.run.smithi084.stdout: "score_stable": 1.75, 2024-09-20T22:55:56.788 INFO:teuthology.orchestra.run.smithi084.stdout: "optimal_score": 1.1399999856948853, 2024-09-20T22:55:56.788 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_acting": 2, 2024-09-20T22:55:56.788 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_stable": 2, 2024-09-20T22:55:56.788 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity_weighted": 0.875, 2024-09-20T22:55:56.788 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity": 1, 2024-09-20T22:55:56.788 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-09-20T22:55:56.788 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:55:56.788 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:56.788 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:56.788 INFO:teuthology.orchestra.run.smithi084.stdout: "pool": 7, 2024-09-20T22:55:56.789 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_name": "unique_pool_0", 2024-09-20T22:55:56.789 INFO:teuthology.orchestra.run.smithi084.stdout: "create_time": "2024-09-20T22:42:51.354358+0000", 2024-09-20T22:55:56.789 INFO:teuthology.orchestra.run.smithi084.stdout: "flags": 8193, 2024-09-20T22:55:56.789 INFO:teuthology.orchestra.run.smithi084.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-20T22:55:56.789 INFO:teuthology.orchestra.run.smithi084.stdout: "type": 1, 2024-09-20T22:55:56.789 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 3, 2024-09-20T22:55:56.789 INFO:teuthology.orchestra.run.smithi084.stdout: "min_size": 2, 2024-09-20T22:55:56.789 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_rule": 0, 2024-09-20T22:55:56.789 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_count": 0, 2024-09-20T22:55:56.789 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_target": 0, 2024-09-20T22:55:56.789 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_barrier": 0, 2024-09-20T22:55:56.789 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-20T22:55:56.789 INFO:teuthology.orchestra.run.smithi084.stdout: "is_stretch_pool": false, 2024-09-20T22:55:56.789 INFO:teuthology.orchestra.run.smithi084.stdout: "object_hash": 2, 2024-09-20T22:55:56.790 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_autoscale_mode": "off", 2024-09-20T22:55:56.790 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num": 16, 2024-09-20T22:55:56.790 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num": 16, 2024-09-20T22:55:56.790 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num_target": 16, 2024-09-20T22:55:56.790 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_target": 16, 2024-09-20T22:55:56.790 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_pending": 16, 2024-09-20T22:55:56.790 INFO:teuthology.orchestra.run.smithi084.stdout: "last_pg_merge_meta": { 2024-09-20T22:55:56.790 INFO:teuthology.orchestra.run.smithi084.stdout: "source_pgid": "7.10", 2024-09-20T22:55:56.790 INFO:teuthology.orchestra.run.smithi084.stdout: "ready_epoch": 484, 2024-09-20T22:55:56.790 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_started": 483, 2024-09-20T22:55:56.790 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_clean": 483, 2024-09-20T22:55:56.790 INFO:teuthology.orchestra.run.smithi084.stdout: "source_version": "480'172", 2024-09-20T22:55:56.790 INFO:teuthology.orchestra.run.smithi084.stdout: "target_version": "480'253" 2024-09-20T22:55:56.791 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:56.791 INFO:teuthology.orchestra.run.smithi084.stdout: "last_change": "876", 2024-09-20T22:55:56.791 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend": "0", 2024-09-20T22:55:56.791 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_prenautilus": "484", 2024-09-20T22:55:56.791 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_preluminous": "482", 2024-09-20T22:55:56.791 INFO:teuthology.orchestra.run.smithi084.stdout: "auid": 0, 2024-09-20T22:55:56.791 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_mode": "selfmanaged", 2024-09-20T22:55:56.791 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_seq": 409, 2024-09-20T22:55:56.791 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_epoch": 876, 2024-09-20T22:55:56.791 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_snaps": [], 2024-09-20T22:55:56.791 INFO:teuthology.orchestra.run.smithi084.stdout: "removed_snaps": "[]", 2024-09-20T22:55:56.791 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_bytes": 0, 2024-09-20T22:55:56.791 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_objects": 0, 2024-09-20T22:55:56.791 INFO:teuthology.orchestra.run.smithi084.stdout: "tiers": [], 2024-09-20T22:55:56.791 INFO:teuthology.orchestra.run.smithi084.stdout: "tier_of": -1, 2024-09-20T22:55:56.792 INFO:teuthology.orchestra.run.smithi084.stdout: "read_tier": -1, 2024-09-20T22:55:56.792 INFO:teuthology.orchestra.run.smithi084.stdout: "write_tier": -1, 2024-09-20T22:55:56.792 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_mode": "none", 2024-09-20T22:55:56.792 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_bytes": 0, 2024-09-20T22:55:56.792 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_objects": 0, 2024-09-20T22:55:56.792 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-20T22:55:56.792 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-20T22:55:56.792 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-20T22:55:56.792 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_flush_age": 0, 2024-09-20T22:55:56.792 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_evict_age": 0, 2024-09-20T22:55:56.792 INFO:teuthology.orchestra.run.smithi084.stdout: "erasure_code_profile": "", 2024-09-20T22:55:56.792 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_params": { 2024-09-20T22:55:56.793 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "none" 2024-09-20T22:55:56.793 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:56.793 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_period": 0, 2024-09-20T22:55:56.793 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_count": 0, 2024-09-20T22:55:56.793 INFO:teuthology.orchestra.run.smithi084.stdout: "use_gmt_hitset": true, 2024-09-20T22:55:56.793 INFO:teuthology.orchestra.run.smithi084.stdout: "min_read_recency_for_promote": 0, 2024-09-20T22:55:56.793 INFO:teuthology.orchestra.run.smithi084.stdout: "min_write_recency_for_promote": 0, 2024-09-20T22:55:56.793 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_grade_decay_rate": 0, 2024-09-20T22:55:56.793 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_search_last_n": 0, 2024-09-20T22:55:56.793 INFO:teuthology.orchestra.run.smithi084.stdout: "grade_table": [], 2024-09-20T22:55:56.793 INFO:teuthology.orchestra.run.smithi084.stdout: "stripe_width": 0, 2024-09-20T22:55:56.793 INFO:teuthology.orchestra.run.smithi084.stdout: "expected_num_objects": 0, 2024-09-20T22:55:56.793 INFO:teuthology.orchestra.run.smithi084.stdout: "fast_read": false, 2024-09-20T22:55:56.793 INFO:teuthology.orchestra.run.smithi084.stdout: "options": {}, 2024-09-20T22:55:56.793 INFO:teuthology.orchestra.run.smithi084.stdout: "application_metadata": { 2024-09-20T22:55:56.794 INFO:teuthology.orchestra.run.smithi084.stdout: "rados": {} 2024-09-20T22:55:56.794 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:56.794 INFO:teuthology.orchestra.run.smithi084.stdout: "read_balance": { 2024-09-20T22:55:56.794 INFO:teuthology.orchestra.run.smithi084.stdout: "score_type": "Fair distribution", 2024-09-20T22:55:56.794 INFO:teuthology.orchestra.run.smithi084.stdout: "score_acting": 1.75, 2024-09-20T22:55:56.794 INFO:teuthology.orchestra.run.smithi084.stdout: "score_stable": 1.75, 2024-09-20T22:55:56.794 INFO:teuthology.orchestra.run.smithi084.stdout: "optimal_score": 1.1399999856948853, 2024-09-20T22:55:56.794 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_acting": 2, 2024-09-20T22:55:56.794 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_stable": 2, 2024-09-20T22:55:56.794 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity_weighted": 0.875, 2024-09-20T22:55:56.794 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity": 1, 2024-09-20T22:55:56.794 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-09-20T22:55:56.795 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:55:56.795 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:55:56.795 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2024-09-20T22:55:56.795 INFO:teuthology.orchestra.run.smithi084.stdout: "osds": [ 2024-09-20T22:55:56.795 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:56.795 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 0, 2024-09-20T22:55:56.795 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "ea36a489-6f91-4b2f-b784-b90506b9805d", 2024-09-20T22:55:56.795 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-09-20T22:55:56.795 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-09-20T22:55:56.795 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-09-20T22:55:56.795 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-09-20T22:55:56.795 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-09-20T22:55:56.795 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-09-20T22:55:56.796 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 8, 2024-09-20T22:55:56.796 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 836, 2024-09-20T22:55:56.796 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-09-20T22:55:56.796 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-09-20T22:55:56.796 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-09-20T22:55:56.796 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:55:56.796 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:56.796 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:55:56.796 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6801", 2024-09-20T22:55:56.796 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 268084626 2024-09-20T22:55:56.796 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:55:56.796 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:55:56.796 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:56.796 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-09-20T22:55:56.797 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:55:56.797 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:56.797 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:55:56.797 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6802", 2024-09-20T22:55:56.797 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 268084626 2024-09-20T22:55:56.797 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:55:56.797 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:55:56.797 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:56.797 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-09-20T22:55:56.797 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:55:56.797 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:56.797 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:55:56.797 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6804", 2024-09-20T22:55:56.797 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 268084626 2024-09-20T22:55:56.797 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:55:56.798 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:55:56.798 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:56.798 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-09-20T22:55:56.798 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:55:56.798 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:56.798 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:55:56.798 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6803", 2024-09-20T22:55:56.798 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 268084626 2024-09-20T22:55:56.798 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:55:56.798 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:55:56.798 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:56.799 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.84:6801/268084626", 2024-09-20T22:55:56.799 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.84:6802/268084626", 2024-09-20T22:55:56.799 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.84:6804/268084626", 2024-09-20T22:55:56.799 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.84:6803/268084626", 2024-09-20T22:55:56.799 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-09-20T22:55:56.799 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-09-20T22:55:56.799 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-09-20T22:55:56.799 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:55:56.799 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:56.799 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:56.799 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 1, 2024-09-20T22:55:56.800 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "c8649d2f-27ef-47f8-bc94-f1cd9e62bdb8", 2024-09-20T22:55:56.800 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-09-20T22:55:56.800 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-09-20T22:55:56.800 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-09-20T22:55:56.800 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-09-20T22:55:56.800 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-09-20T22:55:56.800 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-09-20T22:55:56.800 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 13, 2024-09-20T22:55:56.800 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 836, 2024-09-20T22:55:56.800 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-09-20T22:55:56.800 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-09-20T22:55:56.800 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-09-20T22:55:56.801 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:55:56.801 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:56.801 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:55:56.801 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6805", 2024-09-20T22:55:56.801 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 418374136 2024-09-20T22:55:56.801 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:55:56.801 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:55:56.801 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:56.801 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-09-20T22:55:56.801 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:55:56.801 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:56.801 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:55:56.801 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6806", 2024-09-20T22:55:56.801 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 418374136 2024-09-20T22:55:56.802 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:55:56.802 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:55:56.802 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:56.802 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-09-20T22:55:56.802 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:55:56.802 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:56.802 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:55:56.802 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6808", 2024-09-20T22:55:56.802 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 418374136 2024-09-20T22:55:56.802 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:55:56.802 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:55:56.803 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:56.803 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-09-20T22:55:56.803 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:55:56.803 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:56.803 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:55:56.803 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6807", 2024-09-20T22:55:56.803 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 418374136 2024-09-20T22:55:56.803 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:55:56.803 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:55:56.803 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:56.803 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.84:6805/418374136", 2024-09-20T22:55:56.803 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.84:6806/418374136", 2024-09-20T22:55:56.803 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.84:6808/418374136", 2024-09-20T22:55:56.803 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.84:6807/418374136", 2024-09-20T22:55:56.804 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-09-20T22:55:56.804 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-09-20T22:55:56.804 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-09-20T22:55:56.804 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:55:56.804 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:56.804 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:56.804 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 2, 2024-09-20T22:55:56.804 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "c728eb48-a0f0-4de4-a651-7f3bec867280", 2024-09-20T22:55:56.804 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-09-20T22:55:56.804 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-09-20T22:55:56.804 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-09-20T22:55:56.804 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-09-20T22:55:56.804 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-09-20T22:55:56.805 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-09-20T22:55:56.805 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 18, 2024-09-20T22:55:56.805 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 836, 2024-09-20T22:55:56.805 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-09-20T22:55:56.805 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-09-20T22:55:56.805 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-09-20T22:55:56.805 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:55:56.805 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:56.805 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:55:56.805 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6809", 2024-09-20T22:55:56.805 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3977168611 2024-09-20T22:55:56.805 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:55:56.805 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:55:56.806 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:56.806 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-09-20T22:55:56.806 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:55:56.806 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:56.806 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:55:56.806 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6810", 2024-09-20T22:55:56.806 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3977168611 2024-09-20T22:55:56.806 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:55:56.806 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:55:56.806 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:56.806 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-09-20T22:55:56.806 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:55:56.806 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:56.807 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:55:56.807 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6812", 2024-09-20T22:55:56.807 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3977168611 2024-09-20T22:55:56.807 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:55:56.807 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:55:56.807 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:56.807 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-09-20T22:55:56.807 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:55:56.807 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:56.807 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:55:56.807 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6811", 2024-09-20T22:55:56.807 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3977168611 2024-09-20T22:55:56.807 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:55:56.808 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:55:56.808 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:56.808 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.84:6809/3977168611", 2024-09-20T22:55:56.808 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.84:6810/3977168611", 2024-09-20T22:55:56.808 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.84:6812/3977168611", 2024-09-20T22:55:56.808 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.84:6811/3977168611", 2024-09-20T22:55:56.808 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-09-20T22:55:56.808 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-09-20T22:55:56.808 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-09-20T22:55:56.808 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:55:56.808 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:56.809 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:56.809 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 3, 2024-09-20T22:55:56.809 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "573f44eb-0853-4630-a7d8-9caaeb256f9b", 2024-09-20T22:55:56.809 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-09-20T22:55:56.809 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-09-20T22:55:56.809 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-09-20T22:55:56.809 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-09-20T22:55:56.809 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-09-20T22:55:56.809 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-09-20T22:55:56.809 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 25, 2024-09-20T22:55:56.809 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 836, 2024-09-20T22:55:56.809 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-09-20T22:55:56.810 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-09-20T22:55:56.810 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-09-20T22:55:56.810 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:55:56.810 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:56.810 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:55:56.810 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6813", 2024-09-20T22:55:56.810 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2192239945 2024-09-20T22:55:56.810 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:55:56.810 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:55:56.810 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:56.810 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-09-20T22:55:56.811 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:55:56.811 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:56.811 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:55:56.811 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6814", 2024-09-20T22:55:56.811 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2192239945 2024-09-20T22:55:56.811 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:55:56.811 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:55:56.811 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:56.811 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-09-20T22:55:56.811 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:55:56.811 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:56.812 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:55:56.812 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6816", 2024-09-20T22:55:56.812 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2192239945 2024-09-20T22:55:56.812 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:55:56.812 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:55:56.812 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:56.812 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-09-20T22:55:56.812 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:55:56.812 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:56.812 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:55:56.812 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6815", 2024-09-20T22:55:56.812 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2192239945 2024-09-20T22:55:56.813 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:55:56.813 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:55:56.813 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:56.813 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.84:6813/2192239945", 2024-09-20T22:55:56.813 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.84:6814/2192239945", 2024-09-20T22:55:56.813 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.84:6816/2192239945", 2024-09-20T22:55:56.813 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.84:6815/2192239945", 2024-09-20T22:55:56.813 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-09-20T22:55:56.813 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-09-20T22:55:56.813 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-09-20T22:55:56.814 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:55:56.814 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:56.814 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:56.814 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 4, 2024-09-20T22:55:56.814 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "a52c3b94-c5f3-490f-8ac8-09061b9b2abb", 2024-09-20T22:55:56.814 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-09-20T22:55:56.814 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-09-20T22:55:56.814 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-09-20T22:55:56.814 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 0, 2024-09-20T22:55:56.814 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-09-20T22:55:56.814 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-09-20T22:55:56.815 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 31, 2024-09-20T22:55:56.815 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 830, 2024-09-20T22:55:56.815 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-09-20T22:55:56.815 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-09-20T22:55:56.815 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-09-20T22:55:56.815 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:55:56.815 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:56.815 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:55:56.815 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.154:6800", 2024-09-20T22:55:56.815 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1793136002 2024-09-20T22:55:56.815 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:55:56.815 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:55:56.816 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:56.816 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-09-20T22:55:56.816 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:55:56.816 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:56.816 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:55:56.816 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.154:6801", 2024-09-20T22:55:56.816 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1793136002 2024-09-20T22:55:56.816 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:55:56.816 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:55:56.816 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:56.816 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-09-20T22:55:56.817 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:55:56.817 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:56.817 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:55:56.817 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.154:6803", 2024-09-20T22:55:56.817 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1793136002 2024-09-20T22:55:56.817 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:55:56.817 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:55:56.818 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:56.818 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-09-20T22:55:56.818 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:55:56.818 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:56.818 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:55:56.818 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.154:6802", 2024-09-20T22:55:56.818 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1793136002 2024-09-20T22:55:56.818 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:55:56.818 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:55:56.818 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:56.818 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.154:6800/1793136002", 2024-09-20T22:55:56.819 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.154:6801/1793136002", 2024-09-20T22:55:56.819 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.154:6803/1793136002", 2024-09-20T22:55:56.819 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.154:6802/1793136002", 2024-09-20T22:55:56.819 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-09-20T22:55:56.819 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-09-20T22:55:56.819 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-09-20T22:55:56.819 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:55:56.819 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:56.819 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:56.819 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 5, 2024-09-20T22:55:56.820 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "4fa0dc8e-25cc-46cd-96ff-33f20b6faec3", 2024-09-20T22:55:56.820 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-09-20T22:55:56.820 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-09-20T22:55:56.820 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-09-20T22:55:56.820 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-09-20T22:55:56.820 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-09-20T22:55:56.820 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-09-20T22:55:56.820 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 40, 2024-09-20T22:55:56.820 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 836, 2024-09-20T22:55:56.821 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-09-20T22:55:56.821 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-09-20T22:55:56.821 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-09-20T22:55:56.821 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:55:56.821 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:56.821 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:55:56.821 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.154:6804", 2024-09-20T22:55:56.821 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 493899927 2024-09-20T22:55:56.821 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:55:56.821 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:55:56.821 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:56.821 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-09-20T22:55:56.822 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:55:56.822 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:56.822 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:55:56.822 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.154:6805", 2024-09-20T22:55:56.822 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 493899927 2024-09-20T22:55:56.822 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:55:56.822 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:55:56.822 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:56.822 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-09-20T22:55:56.822 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:55:56.822 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:56.822 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:55:56.822 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.154:6807", 2024-09-20T22:55:56.822 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 493899927 2024-09-20T22:55:56.823 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:55:56.823 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:55:56.823 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:56.823 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-09-20T22:55:56.823 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:55:56.823 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:56.823 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:55:56.823 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.154:6806", 2024-09-20T22:55:56.823 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 493899927 2024-09-20T22:55:56.823 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:55:56.823 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:55:56.823 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:56.823 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.154:6804/493899927", 2024-09-20T22:55:56.823 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.154:6805/493899927", 2024-09-20T22:55:56.824 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.154:6807/493899927", 2024-09-20T22:55:56.824 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.154:6806/493899927", 2024-09-20T22:55:56.824 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-09-20T22:55:56.824 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-09-20T22:55:56.824 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-09-20T22:55:56.824 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:55:56.824 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:56.824 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:56.824 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 6, 2024-09-20T22:55:56.824 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "56788d74-810e-4827-a615-958c12f8a368", 2024-09-20T22:55:56.824 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-09-20T22:55:56.824 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-09-20T22:55:56.824 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-09-20T22:55:56.824 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-09-20T22:55:56.825 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-09-20T22:55:56.825 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-09-20T22:55:56.825 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 48, 2024-09-20T22:55:56.825 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 836, 2024-09-20T22:55:56.825 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-09-20T22:55:56.825 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-09-20T22:55:56.825 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-09-20T22:55:56.825 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:55:56.825 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:56.825 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:55:56.825 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.154:6808", 2024-09-20T22:55:56.825 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2375091033 2024-09-20T22:55:56.826 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:55:56.826 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:55:56.826 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:56.826 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-09-20T22:55:56.826 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:55:56.826 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:56.826 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:55:56.826 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.154:6809", 2024-09-20T22:55:56.826 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2375091033 2024-09-20T22:55:56.826 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:55:56.826 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:55:56.826 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:56.826 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-09-20T22:55:56.826 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:55:56.827 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:56.827 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:55:56.827 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.154:6811", 2024-09-20T22:55:56.827 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2375091033 2024-09-20T22:55:56.827 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:55:56.827 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:55:56.827 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:56.827 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-09-20T22:55:56.827 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:55:56.827 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:56.827 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:55:56.827 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.154:6810", 2024-09-20T22:55:56.827 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2375091033 2024-09-20T22:55:56.827 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:55:56.828 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:55:56.828 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:56.828 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.154:6808/2375091033", 2024-09-20T22:55:56.828 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.154:6809/2375091033", 2024-09-20T22:55:56.828 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.154:6811/2375091033", 2024-09-20T22:55:56.828 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.154:6810/2375091033", 2024-09-20T22:55:56.828 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-09-20T22:55:56.828 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-09-20T22:55:56.828 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-09-20T22:55:56.828 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:55:56.828 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:56.828 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:56.828 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 7, 2024-09-20T22:55:56.828 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "83503727-afc1-4b95-8b96-ecb63f3338c2", 2024-09-20T22:55:56.829 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-09-20T22:55:56.829 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-09-20T22:55:56.829 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-09-20T22:55:56.829 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-09-20T22:55:56.829 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-09-20T22:55:56.829 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-09-20T22:55:56.829 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 59, 2024-09-20T22:55:56.829 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 836, 2024-09-20T22:55:56.829 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-09-20T22:55:56.829 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-09-20T22:55:56.829 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-09-20T22:55:56.829 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:55:56.829 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:56.829 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:55:56.830 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.154:6812", 2024-09-20T22:55:56.830 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3717704491 2024-09-20T22:55:56.830 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:55:56.830 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:55:56.830 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:56.830 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-09-20T22:55:56.830 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:55:56.830 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:56.830 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:55:56.830 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.154:6813", 2024-09-20T22:55:56.830 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3717704491 2024-09-20T22:55:56.830 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:55:56.830 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:55:56.830 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:56.831 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-09-20T22:55:56.831 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:55:56.831 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:56.831 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:55:56.831 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.154:6815", 2024-09-20T22:55:56.831 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3717704491 2024-09-20T22:55:56.831 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:55:56.831 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:55:56.831 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:56.831 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-09-20T22:55:56.831 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:55:56.831 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:56.831 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:55:56.831 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.154:6814", 2024-09-20T22:55:56.832 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3717704491 2024-09-20T22:55:56.832 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:55:56.832 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:55:56.832 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:56.832 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.154:6812/3717704491", 2024-09-20T22:55:56.832 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.154:6813/3717704491", 2024-09-20T22:55:56.832 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.154:6815/3717704491", 2024-09-20T22:55:56.832 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.154:6814/3717704491", 2024-09-20T22:55:56.832 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-09-20T22:55:56.832 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-09-20T22:55:56.832 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-09-20T22:55:56.832 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:55:56.832 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:55:56.832 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2024-09-20T22:55:56.832 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_xinfo": [ 2024-09-20T22:55:56.833 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:56.833 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 0, 2024-09-20T22:55:56.833 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-09-20T22:55:56.833 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-09-20T22:55:56.833 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-09-20T22:55:56.833 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540701547738038271, 2024-09-20T22:55:56.833 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-09-20T22:55:56.833 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-09-20T22:55:19.615519+0000", 2024-09-20T22:55:56.833 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-09-20T22:55:56.833 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:56.833 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:56.833 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 1, 2024-09-20T22:55:56.833 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-09-20T22:55:56.833 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-09-20T22:55:56.833 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-09-20T22:55:56.834 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540701547738038271, 2024-09-20T22:55:56.834 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-09-20T22:55:56.834 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-09-20T22:54:48.239990+0000", 2024-09-20T22:55:56.834 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-09-20T22:55:56.834 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:56.834 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:56.834 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 2, 2024-09-20T22:55:56.834 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-09-20T22:55:56.834 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-09-20T22:55:56.834 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-09-20T22:55:56.834 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540701547738038271, 2024-09-20T22:55:56.834 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-09-20T22:55:56.835 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-09-20T22:55:52.250049+0000", 2024-09-20T22:55:56.835 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-09-20T22:55:56.835 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:56.835 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:56.835 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 3, 2024-09-20T22:55:56.835 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-09-20T22:55:56.835 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-09-20T22:55:56.835 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-09-20T22:55:56.835 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540701547738038271, 2024-09-20T22:55:56.835 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-09-20T22:55:56.835 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-09-20T22:55:46.853367+0000", 2024-09-20T22:55:56.835 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-09-20T22:55:56.835 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:56.835 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:56.835 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 4, 2024-09-20T22:55:56.836 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-09-20T22:55:56.836 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-09-20T22:55:56.836 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-09-20T22:55:56.836 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540701547738038271, 2024-09-20T22:55:56.836 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-09-20T22:55:56.836 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-09-20T22:55:29.301253+0000", 2024-09-20T22:55:56.836 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-09-20T22:55:56.836 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:56.836 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:56.836 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 5, 2024-09-20T22:55:56.836 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-09-20T22:55:56.836 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-09-20T22:55:56.836 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-09-20T22:55:56.836 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540701547738038271, 2024-09-20T22:55:56.837 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-09-20T22:55:56.837 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-09-20T22:54:54.642519+0000", 2024-09-20T22:55:56.837 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-09-20T22:55:56.837 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:56.837 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:56.837 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 6, 2024-09-20T22:55:56.837 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-09-20T22:55:56.837 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-09-20T22:55:56.837 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-09-20T22:55:56.837 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540701547738038271, 2024-09-20T22:55:56.837 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-09-20T22:55:56.837 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-09-20T22:54:50.271376+0000", 2024-09-20T22:55:56.837 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-09-20T22:55:56.837 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:56.838 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:56.838 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 7, 2024-09-20T22:55:56.838 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-09-20T22:55:56.838 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-09-20T22:55:56.838 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-09-20T22:55:56.838 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540701547738038271, 2024-09-20T22:55:56.838 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-09-20T22:55:56.838 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-09-20T22:55:29.961705+0000", 2024-09-20T22:55:56.838 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-09-20T22:55:56.838 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:55:56.838 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2024-09-20T22:55:56.838 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_upmap": [ 2024-09-20T22:55:56.838 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:56.839 INFO:teuthology.orchestra.run.smithi084.stdout: "pgid": "6.e", 2024-09-20T22:55:56.839 INFO:teuthology.orchestra.run.smithi084.stdout: "osds": [ 2024-09-20T22:55:56.839 INFO:teuthology.orchestra.run.smithi084.stdout: 1, 2024-09-20T22:55:56.839 INFO:teuthology.orchestra.run.smithi084.stdout: 2, 2024-09-20T22:55:56.839 INFO:teuthology.orchestra.run.smithi084.stdout: 5 2024-09-20T22:55:56.839 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:55:56.839 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:55:56.839 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2024-09-20T22:55:56.839 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_upmap_items": [ 2024-09-20T22:55:56.839 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:56.839 INFO:teuthology.orchestra.run.smithi084.stdout: "pgid": "1.0", 2024-09-20T22:55:56.839 INFO:teuthology.orchestra.run.smithi084.stdout: "mappings": [ 2024-09-20T22:55:56.839 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:56.840 INFO:teuthology.orchestra.run.smithi084.stdout: "from": 0, 2024-09-20T22:55:56.840 INFO:teuthology.orchestra.run.smithi084.stdout: "to": 7 2024-09-20T22:55:56.840 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:55:56.840 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:55:56.840 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:56.840 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:56.840 INFO:teuthology.orchestra.run.smithi084.stdout: "pgid": "3.a", 2024-09-20T22:55:56.840 INFO:teuthology.orchestra.run.smithi084.stdout: "mappings": [ 2024-09-20T22:55:56.840 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:56.840 INFO:teuthology.orchestra.run.smithi084.stdout: "from": 6, 2024-09-20T22:55:56.840 INFO:teuthology.orchestra.run.smithi084.stdout: "to": 2 2024-09-20T22:55:56.840 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:55:56.840 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:55:56.840 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:56.841 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:56.841 INFO:teuthology.orchestra.run.smithi084.stdout: "pgid": "4.4", 2024-09-20T22:55:56.841 INFO:teuthology.orchestra.run.smithi084.stdout: "mappings": [ 2024-09-20T22:55:56.841 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:56.841 INFO:teuthology.orchestra.run.smithi084.stdout: "from": 1, 2024-09-20T22:55:56.841 INFO:teuthology.orchestra.run.smithi084.stdout: "to": 5 2024-09-20T22:55:56.841 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:55:56.841 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:55:56.841 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:56.841 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:56.842 INFO:teuthology.orchestra.run.smithi084.stdout: "pgid": "4.6", 2024-09-20T22:55:56.842 INFO:teuthology.orchestra.run.smithi084.stdout: "mappings": [ 2024-09-20T22:55:56.842 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:56.842 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:56.842 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:55:56.843 INFO:teuthology.orchestra.run.smithi084.stdout: "from": 1, 2024-09-20T22:55:56.843 INFO:teuthology.orchestra.run.smithi084.stdout: "to": 5 2024-09-20T22:55:56.843 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:55:56.843 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:55:56.843 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:56.843 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:56.843 INFO:teuthology.orchestra.run.smithi084.stdout: "pgid": "4.9", 2024-09-20T22:55:56.844 INFO:teuthology.orchestra.run.smithi084.stdout: "mappings": [ 2024-09-20T22:55:56.844 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:56.844 INFO:teuthology.orchestra.run.smithi084.stdout: "from": 3, 2024-09-20T22:55:56.844 INFO:teuthology.orchestra.run.smithi084.stdout: "to": 2 2024-09-20T22:55:56.844 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:55:56.844 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:55:56.844 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:56.844 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:56.844 INFO:teuthology.orchestra.run.smithi084.stdout: "pgid": "4.11", 2024-09-20T22:55:56.844 INFO:teuthology.orchestra.run.smithi084.stdout: "mappings": [ 2024-09-20T22:55:56.844 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:56.844 INFO:teuthology.orchestra.run.smithi084.stdout: "from": 3, 2024-09-20T22:55:56.844 INFO:teuthology.orchestra.run.smithi084.stdout: "to": 5 2024-09-20T22:55:56.844 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:55:56.845 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:55:56.845 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:56.845 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:56.845 INFO:teuthology.orchestra.run.smithi084.stdout: "pgid": "4.1b", 2024-09-20T22:55:56.845 INFO:teuthology.orchestra.run.smithi084.stdout: "mappings": [ 2024-09-20T22:55:56.845 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:56.845 INFO:teuthology.orchestra.run.smithi084.stdout: "from": 1, 2024-09-20T22:55:56.845 INFO:teuthology.orchestra.run.smithi084.stdout: "to": 2 2024-09-20T22:55:56.845 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:55:56.845 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:55:56.845 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:55:56.846 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2024-09-20T22:55:56.846 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_upmap_primaries": [], 2024-09-20T22:55:56.846 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_temp": [], 2024-09-20T22:55:56.846 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_temp": [], 2024-09-20T22:55:56.846 INFO:teuthology.orchestra.run.smithi084.stdout: "blocklist": { 2024-09-20T22:55:56.846 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/542020545": "2024-09-21T22:42:07.596045+0000", 2024-09-20T22:55:56.846 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/25873236": "2024-09-21T22:36:54.917812+0000", 2024-09-20T22:55:56.846 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:6800/820908470": "2024-09-21T22:36:54.917812+0000", 2024-09-20T22:55:56.846 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/567614908": "2024-09-21T22:36:54.917812+0000", 2024-09-20T22:55:56.846 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/2963622459": "2024-09-21T22:42:07.596045+0000", 2024-09-20T22:55:56.847 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/1176849276": "2024-09-21T22:37:17.690104+0000", 2024-09-20T22:55:56.847 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:6800/2932420050": "2024-09-21T22:37:17.690104+0000", 2024-09-20T22:55:56.847 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/3879263635": "2024-09-21T22:37:17.690104+0000", 2024-09-20T22:55:56.847 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/969472279": "2024-09-21T22:36:54.917812+0000", 2024-09-20T22:55:56.847 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/1361143214": "2024-09-21T22:37:17.690104+0000", 2024-09-20T22:55:56.847 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/3353126955": "2024-09-21T22:42:07.596045+0000", 2024-09-20T22:55:56.847 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:6800/4024278029": "2024-09-21T22:42:07.596045+0000", 2024-09-20T22:55:56.847 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/547902262": "2024-09-21T22:42:07.596045+0000", 2024-09-20T22:55:56.847 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/1242929317": "2024-09-21T22:42:07.596045+0000" 2024-09-20T22:55:56.847 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:56.847 INFO:teuthology.orchestra.run.smithi084.stdout: "range_blocklist": {}, 2024-09-20T22:55:56.847 INFO:teuthology.orchestra.run.smithi084.stdout: "erasure_code_profiles": { 2024-09-20T22:55:56.847 INFO:teuthology.orchestra.run.smithi084.stdout: "default": { 2024-09-20T22:55:56.847 INFO:teuthology.orchestra.run.smithi084.stdout: "crush-failure-domain": "osd", 2024-09-20T22:55:56.847 INFO:teuthology.orchestra.run.smithi084.stdout: "k": "2", 2024-09-20T22:55:56.848 INFO:teuthology.orchestra.run.smithi084.stdout: "m": "1", 2024-09-20T22:55:56.848 INFO:teuthology.orchestra.run.smithi084.stdout: "plugin": "jerasure", 2024-09-20T22:55:56.848 INFO:teuthology.orchestra.run.smithi084.stdout: "technique": "reed_sol_van" 2024-09-20T22:55:56.848 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:55:56.848 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:56.848 INFO:teuthology.orchestra.run.smithi084.stdout: "removed_snaps_queue": [ 2024-09-20T22:55:56.848 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:56.848 INFO:teuthology.orchestra.run.smithi084.stdout: "pool": 7, 2024-09-20T22:55:56.848 INFO:teuthology.orchestra.run.smithi084.stdout: "snaps": [ 2024-09-20T22:55:56.848 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:56.848 INFO:teuthology.orchestra.run.smithi084.stdout: "begin": 302, 2024-09-20T22:55:56.848 INFO:teuthology.orchestra.run.smithi084.stdout: "length": 1 2024-09-20T22:55:56.848 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:56.848 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:56.849 INFO:teuthology.orchestra.run.smithi084.stdout: "begin": 329, 2024-09-20T22:55:56.849 INFO:teuthology.orchestra.run.smithi084.stdout: "length": 1 2024-09-20T22:55:56.849 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:56.849 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:56.849 INFO:teuthology.orchestra.run.smithi084.stdout: "begin": 343, 2024-09-20T22:55:56.849 INFO:teuthology.orchestra.run.smithi084.stdout: "length": 1 2024-09-20T22:55:56.849 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:56.849 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:56.849 INFO:teuthology.orchestra.run.smithi084.stdout: "begin": 358, 2024-09-20T22:55:56.849 INFO:teuthology.orchestra.run.smithi084.stdout: "length": 1 2024-09-20T22:55:56.849 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:56.849 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:56.849 INFO:teuthology.orchestra.run.smithi084.stdout: "begin": 380, 2024-09-20T22:55:56.850 INFO:teuthology.orchestra.run.smithi084.stdout: "length": 1 2024-09-20T22:55:56.850 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:56.850 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:56.850 INFO:teuthology.orchestra.run.smithi084.stdout: "begin": 388, 2024-09-20T22:55:56.850 INFO:teuthology.orchestra.run.smithi084.stdout: "length": 1 2024-09-20T22:55:56.850 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:56.850 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:56.850 INFO:teuthology.orchestra.run.smithi084.stdout: "begin": 391, 2024-09-20T22:55:56.850 INFO:teuthology.orchestra.run.smithi084.stdout: "length": 2 2024-09-20T22:55:56.851 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:56.851 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:56.851 INFO:teuthology.orchestra.run.smithi084.stdout: "begin": 395, 2024-09-20T22:55:56.851 INFO:teuthology.orchestra.run.smithi084.stdout: "length": 1 2024-09-20T22:55:56.851 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:56.851 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:56.851 INFO:teuthology.orchestra.run.smithi084.stdout: "begin": 402, 2024-09-20T22:55:56.851 INFO:teuthology.orchestra.run.smithi084.stdout: "length": 1 2024-09-20T22:55:56.851 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:55:56.851 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:56.851 INFO:teuthology.orchestra.run.smithi084.stdout: "begin": 404, 2024-09-20T22:55:56.851 INFO:teuthology.orchestra.run.smithi084.stdout: "length": 1 2024-09-20T22:55:56.852 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:55:56.852 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:55:56.852 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:55:56.852 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2024-09-20T22:55:56.852 INFO:teuthology.orchestra.run.smithi084.stdout: "new_removed_snaps": [ 2024-09-20T22:55:56.852 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:56.852 INFO:teuthology.orchestra.run.smithi084.stdout: "pool": 7, 2024-09-20T22:55:56.852 INFO:teuthology.orchestra.run.smithi084.stdout: "snaps": [ 2024-09-20T22:55:56.852 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:55:56.852 INFO:teuthology.orchestra.run.smithi084.stdout: "begin": 302, 2024-09-20T22:55:56.853 INFO:teuthology.orchestra.run.smithi084.stdout: "length": 1 2024-09-20T22:55:56.853 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:55:56.853 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:55:56.853 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:55:56.853 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2024-09-20T22:55:56.853 INFO:teuthology.orchestra.run.smithi084.stdout: "new_purged_snaps": [], 2024-09-20T22:55:56.853 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_node_flags": {}, 2024-09-20T22:55:56.853 INFO:teuthology.orchestra.run.smithi084.stdout: "device_class_flags": {}, 2024-09-20T22:55:56.854 INFO:teuthology.orchestra.run.smithi084.stdout: "stretch_mode": { 2024-09-20T22:55:56.854 INFO:teuthology.orchestra.run.smithi084.stdout: "stretch_mode_enabled": false, 2024-09-20T22:55:56.854 INFO:teuthology.orchestra.run.smithi084.stdout: "stretch_bucket_count": 0, 2024-09-20T22:55:56.854 INFO:teuthology.orchestra.run.smithi084.stdout: "degraded_stretch_mode": 0, 2024-09-20T22:55:56.854 INFO:teuthology.orchestra.run.smithi084.stdout: "recovering_stretch_mode": 0, 2024-09-20T22:55:56.854 INFO:teuthology.orchestra.run.smithi084.stdout: "stretch_mode_bucket": 0 2024-09-20T22:55:56.854 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:55:56.854 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-20T22:55:56.943 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:55:56.964 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:56.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:56.965 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:55:57.066 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:55:57.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:57.088 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:57.089 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:55:57.189 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:55:57.210 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:57.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:57.211 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:55:57.312 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:55:57.334 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:57.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:57.335 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:55:57.436 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:55:57.445 INFO:tasks.rados.rados.0.smithi084.stdout:3485: done (1 left) 2024-09-20T22:55:57.445 INFO:tasks.rados.rados.0.smithi084.stdout:3486: write oid 37 current snap is 408 2024-09-20T22:55:57.445 INFO:tasks.rados.rados.0.smithi084.stdout:3486: seq_num 1189 ranges {783523=422129,1827443=621747,3218409=654188} 2024-09-20T22:55:57.451 INFO:tasks.rados.rados.0.smithi084.stdout:3486: writing smithi08477203-37 from 783523 to 1205652 tid 1 2024-09-20T22:55:57.458 INFO:tasks.rados.rados.0.smithi084.stdout:3486: writing smithi08477203-37 from 1827443 to 2449190 tid 2 2024-09-20T22:55:57.460 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:57.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:57.461 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:55:57.466 INFO:tasks.rados.rados.0.smithi084.stdout:3486: writing smithi08477203-37 from 3218409 to 3872597 tid 3 2024-09-20T22:55:57.466 INFO:tasks.rados.rados.0.smithi084.stdout:3487: rollback oid 27 current snap is 408 2024-09-20T22:55:57.466 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 27 to 369 2024-09-20T22:55:57.466 INFO:tasks.rados.rados.0.smithi084.stdout:3486: finishing write tid 1 to smithi08477203-37 2024-09-20T22:55:57.467 INFO:tasks.rados.rados.0.smithi084.stdout:3486: finishing write tid 2 to smithi08477203-37 2024-09-20T22:55:57.467 INFO:tasks.rados.rados.0.smithi084.stdout:3484: done (2 left) 2024-09-20T22:55:57.467 INFO:tasks.rados.rados.0.smithi084.stdout:3488: snap_create 2024-09-20T22:55:57.470 INFO:tasks.rados.rados.0.smithi084.stdout:3487: finishing rollback tid 0 to smithi08477203-27 2024-09-20T22:55:57.470 INFO:tasks.rados.rados.0.smithi084.stdout:3486: finishing write tid 3 to smithi08477203-37 2024-09-20T22:55:57.471 INFO:tasks.rados.rados.0.smithi084.stdout:3487: finishing rollback tid 1 to smithi08477203-27 2024-09-20T22:55:57.471 INFO:tasks.rados.rados.0.smithi084.stdout:3486: finishing write tid 4 to smithi08477203-37 2024-09-20T22:55:57.471 INFO:tasks.rados.rados.0.smithi084.stdout:3487: finishing rollback tid 2 to smithi08477203-27 2024-09-20T22:55:57.471 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 27 v 2488 (ObjNum 1077 snap 358 seq_num 1077) dirty exists 2024-09-20T22:55:57.471 INFO:tasks.rados.rados.0.smithi084.stdout:3486: finishing write tid 5 to smithi08477203-37 2024-09-20T22:55:57.471 INFO:tasks.rados.rados.0.smithi084.stdout:3486: oid 37 updating version 0 to 1940 2024-09-20T22:55:57.471 INFO:tasks.rados.rados.0.smithi084.stdout:3486: oid 37 version 1940 is already newer than 1937 2024-09-20T22:55:57.472 INFO:tasks.rados.rados.0.smithi084.stdout:3486: oid 37 version 1940 is already newer than 1938 2024-09-20T22:55:57.472 INFO:tasks.rados.rados.0.smithi084.stdout:3486: oid 37 version 1940 is already newer than 1939 2024-09-20T22:55:57.472 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 37 v 1940 (ObjNum 1189 snap 408 seq_num 1189) dirty exists 2024-09-20T22:55:57.472 INFO:tasks.rados.rados.0.smithi084.stdout:3486: left oid 37 (ObjNum 1189 snap 408 seq_num 1189) 2024-09-20T22:55:57.490 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.4 dump_blocked_ops 2024-09-20T22:55:57.510 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd set noscrub 2024-09-20T22:55:57.562 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:55:57.583 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:57.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:57.585 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:55:57.685 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:55:57.700 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:57 smithi084 ceph-mon[40024]: pgmap v1215: 158 pgs: 2 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 152 active+clean; 639 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 3.1 MiB/s rd, 1.6 MiB/s wr, 11 op/s 2024-09-20T22:55:57.700 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:57 smithi084 ceph-mon[40024]: 5.b scrub starts 2024-09-20T22:55:57.700 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:57 smithi084 ceph-mon[40024]: 5.b scrub ok 2024-09-20T22:55:57.700 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:57 smithi084 ceph-mon[40024]: osdmap e876: 8 total, 8 up, 8 in 2024-09-20T22:55:57.700 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:57 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/137786807' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-20T22:55:57.700 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:57 smithi084 ceph-mon[40024]: 2.1 scrub starts 2024-09-20T22:55:57.701 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:57 smithi084 ceph-mon[40024]: 2.1 scrub ok 2024-09-20T22:55:57.701 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:57 smithi084 ceph-mon[40024]: 5.1d deep-scrub starts 2024-09-20T22:55:57.701 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:57 smithi084 ceph-mon[40024]: 5.1d deep-scrub ok 2024-09-20T22:55:57.701 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:57 smithi084 ceph-mon[40024]: 6.0 scrub starts 2024-09-20T22:55:57.701 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:57 smithi084 ceph-mon[40024]: 6.0 scrub ok 2024-09-20T22:55:57.701 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:57 smithi084 ceph-mon[29774]: pgmap v1215: 158 pgs: 2 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 152 active+clean; 639 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 3.1 MiB/s rd, 1.6 MiB/s wr, 11 op/s 2024-09-20T22:55:57.701 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:57 smithi084 ceph-mon[29774]: 5.b scrub starts 2024-09-20T22:55:57.701 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:57 smithi084 ceph-mon[29774]: 5.b scrub ok 2024-09-20T22:55:57.701 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:57 smithi084 ceph-mon[29774]: osdmap e876: 8 total, 8 up, 8 in 2024-09-20T22:55:57.701 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:57 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/137786807' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-20T22:55:57.701 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:57 smithi084 ceph-mon[29774]: 2.1 scrub starts 2024-09-20T22:55:57.702 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:57 smithi084 ceph-mon[29774]: 2.1 scrub ok 2024-09-20T22:55:57.702 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:57 smithi084 ceph-mon[29774]: 5.1d deep-scrub starts 2024-09-20T22:55:57.702 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:57 smithi084 ceph-mon[29774]: 5.1d deep-scrub ok 2024-09-20T22:55:57.702 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:57 smithi084 ceph-mon[29774]: 6.0 scrub starts 2024-09-20T22:55:57.702 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:57 smithi084 ceph-mon[29774]: 6.0 scrub ok 2024-09-20T22:55:57.707 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:57.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:57.708 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:55:57.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:57 smithi154 ceph-mon[33264]: pgmap v1215: 158 pgs: 2 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 152 active+clean; 639 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 3.1 MiB/s rd, 1.6 MiB/s wr, 11 op/s 2024-09-20T22:55:57.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:57 smithi154 ceph-mon[33264]: 5.b scrub starts 2024-09-20T22:55:57.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:57 smithi154 ceph-mon[33264]: 5.b scrub ok 2024-09-20T22:55:57.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:57 smithi154 ceph-mon[33264]: osdmap e876: 8 total, 8 up, 8 in 2024-09-20T22:55:57.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:57 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/137786807' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-20T22:55:57.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:57 smithi154 ceph-mon[33264]: 2.1 scrub starts 2024-09-20T22:55:57.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:57 smithi154 ceph-mon[33264]: 2.1 scrub ok 2024-09-20T22:55:57.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:57 smithi154 ceph-mon[33264]: 5.1d deep-scrub starts 2024-09-20T22:55:57.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:57 smithi154 ceph-mon[33264]: 5.1d deep-scrub ok 2024-09-20T22:55:57.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:57 smithi154 ceph-mon[33264]: 6.0 scrub starts 2024-09-20T22:55:57.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:57 smithi154 ceph-mon[33264]: 6.0 scrub ok 2024-09-20T22:55:57.775 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:55:57.777 INFO:tasks.thrashosds.thrasher:j is {'epoch': 876, 'fsid': 'a9c0bc70-77a0-11ef-baf5-efdc52797490', 'created': '2024-09-20T22:36:30.240626+0000', 'modified': '2024-09-20T22:55:56.427021+0000', 'last_up_change': '2024-09-20T22:41:27.098565+0000', 'last_in_change': '2024-09-20T22:54:16.306377+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 18, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 7, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-20T22:39:21.984381+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 11, 'pg_placement_num': 11, 'pg_placement_num_target': 11, 'pg_num_target': 11, 'pg_num_pending': 11, 'last_pg_merge_meta': {'source_pgid': '1.b', 'ready_epoch': 299, 'last_epoch_started': 298, 'last_epoch_clean': 298, 'source_version': "77'128", 'target_version': "77'128"}, 'last_change': '769', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '299', 'last_force_op_resend_preluminous': '297', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.909999966621399, 'score_stable': 1.909999966621399, 'optimal_score': 1.1399999856948853, 'raw_score_acting': 2.180000066757202, 'raw_score_stable': 2.180000066757202, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-09-20T22:41:35.820081+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '69', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 69, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.3299999237060547, 'score_stable': 2.3299999237060547, 'optimal_score': 1, 'raw_score_acting': 2.3299999237060547, 'raw_score_stable': 2.3299999237060547, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-09-20T22:41:36.546389+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5399999618530273, 'score_stable': 1.5399999618530273, 'optimal_score': 1.1399999856948853, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-09-20T22:41:38.454640+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '70', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.9700000286102295, 'score_stable': 1.9700000286102295, 'optimal_score': 1.1399999856948853, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-09-20T22:41:40.407129+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '72', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3200000524520874, 'score_stable': 1.3200000524520874, 'optimal_score': 1.1399999856948853, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-09-20T22:41:42.476716+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '74', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1.1399999856948853, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 7, 'pool_name': 'unique_pool_0', 'create_time': '2024-09-20T22:42:51.354358+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '7.10', 'ready_epoch': 484, 'last_epoch_started': 483, 'last_epoch_clean': 483, 'source_version': "480'172", 'target_version': "480'253"}, 'last_change': '876', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '484', 'last_force_op_resend_preluminous': '482', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 409, 'snap_epoch': 876, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1.1399999856948853, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}], 'osds': [{'osd': 0, 'uuid': 'ea36a489-6f91-4b2f-b784-b90506b9805d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 836, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6801', 'nonce': 268084626}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6802', 'nonce': 268084626}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6804', 'nonce': 268084626}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6803', 'nonce': 268084626}]}, 'public_addr': '172.21.15.84:6801/268084626', 'cluster_addr': '172.21.15.84:6802/268084626', 'heartbeat_back_addr': '172.21.15.84:6804/268084626', 'heartbeat_front_addr': '172.21.15.84:6803/268084626', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'c8649d2f-27ef-47f8-bc94-f1cd9e62bdb8', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 836, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6805', 'nonce': 418374136}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6806', 'nonce': 418374136}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6808', 'nonce': 418374136}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6807', 'nonce': 418374136}]}, 'public_addr': '172.21.15.84:6805/418374136', 'cluster_addr': '172.21.15.84:6806/418374136', 'heartbeat_back_addr': '172.21.15.84:6808/418374136', 'heartbeat_front_addr': '172.21.15.84:6807/418374136', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'c728eb48-a0f0-4de4-a651-7f3bec867280', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 836, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6809', 'nonce': 3977168611}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6810', 'nonce': 3977168611}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6812', 'nonce': 3977168611}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6811', 'nonce': 3977168611}]}, 'public_addr': '172.21.15.84:6809/3977168611', 'cluster_addr': '172.21.15.84:6810/3977168611', 'heartbeat_back_addr': '172.21.15.84:6812/3977168611', 'heartbeat_front_addr': '172.21.15.84:6811/3977168611', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '573f44eb-0853-4630-a7d8-9caaeb256f9b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 25, 'up_thru': 836, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6813', 'nonce': 2192239945}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6814', 'nonce': 2192239945}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6816', 'nonce': 2192239945}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6815', 'nonce': 2192239945}]}, 'public_addr': '172.21.15.84:6813/2192239945', 'cluster_addr': '172.21.15.84:6814/2192239945', 'heartbeat_back_addr': '172.21.15.84:6816/2192239945', 'heartbeat_front_addr': '172.21.15.84:6815/2192239945', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'a52c3b94-c5f3-490f-8ac8-09061b9b2abb', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 830, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6800', 'nonce': 1793136002}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6801', 'nonce': 1793136002}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6803', 'nonce': 1793136002}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6802', 'nonce': 1793136002}]}, 'public_addr': '172.21.15.154:6800/1793136002', 'cluster_addr': '172.21.15.154:6801/1793136002', 'heartbeat_back_addr': '172.21.15.154:6803/1793136002', 'heartbeat_front_addr': '172.21.15.154:6802/1793136002', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '4fa0dc8e-25cc-46cd-96ff-33f20b6faec3', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 40, 'up_thru': 836, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6804', 'nonce': 493899927}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6805', 'nonce': 493899927}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6807', 'nonce': 493899927}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6806', 'nonce': 493899927}]}, 'public_addr': '172.21.15.154:6804/493899927', 'cluster_addr': '172.21.15.154:6805/493899927', 'heartbeat_back_addr': '172.21.15.154:6807/493899927', 'heartbeat_front_addr': '172.21.15.154:6806/493899927', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '56788d74-810e-4827-a615-958c12f8a368', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 48, 'up_thru': 836, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6808', 'nonce': 2375091033}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6809', 'nonce': 2375091033}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6811', 'nonce': 2375091033}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6810', 'nonce': 2375091033}]}, 'public_addr': '172.21.15.154:6808/2375091033', 'cluster_addr': '172.21.15.154:6809/2375091033', 'heartbeat_back_addr': '172.21.15.154:6811/2375091033', 'heartbeat_front_addr': '172.21.15.154:6810/2375091033', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '83503727-afc1-4b95-8b96-ecb63f3338c2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 59, 'up_thru': 836, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6812', 'nonce': 3717704491}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6813', 'nonce': 3717704491}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6815', 'nonce': 3717704491}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6814', 'nonce': 3717704491}]}, 'public_addr': '172.21.15.154:6812/3717704491', 'cluster_addr': '172.21.15.154:6813/3717704491', 'heartbeat_back_addr': '172.21.15.154:6815/3717704491', 'heartbeat_front_addr': '172.21.15.154:6814/3717704491', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-20T22:55:19.615519+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-20T22:54:48.239990+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-20T22:55:52.250049+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-20T22:55:46.853367+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-20T22:55:29.301253+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-20T22:54:54.642519+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-20T22:54:50.271376+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-20T22:55:29.961705+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '6.e', 'osds': [1, 2, 5]}], 'pg_upmap_items': [{'pgid': '1.0', 'mappings': [{'from': 0, 'to': 7}]}, {'pgid': '3.a', 'mappings': [{'from': 6, 'to': 2}]}, {'pgid': '4.4', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.6', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.9', 'mappings': [{'from': 3, 'to': 2}]}, {'pgid': '4.11', 'mappings': [{'from': 3, 'to': 5}]}, {'pgid': '4.1b', 'mappings': [{'from': 1, 'to': 2}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.84:0/542020545': '2024-09-21T22:42:07.596045+0000', '172.21.15.84:0/25873236': '2024-09-21T22:36:54.917812+0000', '172.21.15.84:6800/820908470': '2024-09-21T22:36:54.917812+0000', '172.21.15.84:0/567614908': '2024-09-21T22:36:54.917812+0000', '172.21.15.84:0/2963622459': '2024-09-21T22:42:07.596045+0000', '172.21.15.84:0/1176849276': '2024-09-21T22:37:17.690104+0000', '172.21.15.84:6800/2932420050': '2024-09-21T22:37:17.690104+0000', '172.21.15.84:0/3879263635': '2024-09-21T22:37:17.690104+0000', '172.21.15.84:0/969472279': '2024-09-21T22:36:54.917812+0000', '172.21.15.84:0/1361143214': '2024-09-21T22:37:17.690104+0000', '172.21.15.84:0/3353126955': '2024-09-21T22:42:07.596045+0000', '172.21.15.84:6800/4024278029': '2024-09-21T22:42:07.596045+0000', '172.21.15.84:0/547902262': '2024-09-21T22:42:07.596045+0000', '172.21.15.84:0/1242929317': '2024-09-21T22:42:07.596045+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [{'pool': 7, 'snaps': [{'begin': 302, 'length': 1}, {'begin': 329, 'length': 1}, {'begin': 343, 'length': 1}, {'begin': 358, 'length': 1}, {'begin': 380, 'length': 1}, {'begin': 388, 'length': 1}, {'begin': 391, 'length': 2}, {'begin': 395, 'length': 1}, {'begin': 402, 'length': 1}, {'begin': 404, 'length': 1}]}], 'new_removed_snaps': [{'pool': 7, 'snaps': [{'begin': 302, 'length': 1}]}], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-20T22:55:57.777 INFO:tasks.thrashosds.thrasher:Clearing pg_upmap on 6.e 2024-09-20T22:55:57.777 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd rm-pg-upmap 6.e 2024-09-20T22:55:57.809 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:55:57.842 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:57.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:57.843 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:55:57.944 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:55:57.965 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:57.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:57.967 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:55:58.029 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:55:58.029 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:55:58.052 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:55:58.052 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:55:58.052 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:55:58.068 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:55:58.069 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:55:58.070 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:55:58.089 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:58.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:58.090 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:55:58.091 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-20T22:55:58.091 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T22:55:58.093 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:55:58.108 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:55:58.109 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:55:58.126 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:55:58.127 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:55:58.143 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:55:58.143 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:55:58.159 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:55:58.160 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:55:58.191 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:55:58.213 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:58.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:58.214 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:55:58.246 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-20T22:55:58.246 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:55:58.314 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:55:58.323 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:55:58 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: ::ffff:172.21.15.154 - - [20/Sep/2024:22:55:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-20T22:55:58.337 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:58.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:58.338 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:55:58.339 INFO:tasks.rados.rados.0.smithi084.stdout:3486: done (2 left) 2024-09-20T22:55:58.339 INFO:tasks.rados.rados.0.smithi084.stdout:3487: done (1 left) 2024-09-20T22:55:58.339 INFO:tasks.rados.rados.0.smithi084.stdout:3488: done (0 left) 2024-09-20T22:55:58.339 INFO:tasks.rados.rados.0.smithi084.stdout:3489: read oid 25 snap -1 2024-09-20T22:55:58.339 INFO:tasks.rados.rados.0.smithi084.stdout:3489: expect (ObjNum 1128 snap 381 seq_num 1128) 2024-09-20T22:55:58.339 INFO:tasks.rados.rados.0.smithi084.stdout:3490: write oid 40 current snap is 409 2024-09-20T22:55:58.340 INFO:tasks.rados.rados.0.smithi084.stdout:3490: seq_num 1190 ranges {400330=497378,1485925=514405} 2024-09-20T22:55:58.344 INFO:tasks.rados.rados.0.smithi084.stdout:3490: writing smithi08477203-40 from 400330 to 897708 tid 1 2024-09-20T22:55:58.351 INFO:tasks.rados.rados.0.smithi084.stdout:3490: writing smithi08477203-40 from 1485925 to 2000330 tid 2 2024-09-20T22:55:58.351 INFO:tasks.rados.rados.0.smithi084.stdout:3491: snap_create 2024-09-20T22:55:58.356 INFO:tasks.rados.rados.0.smithi084.stdout:3489: expect (ObjNum 1128 snap 381 seq_num 1128) 2024-09-20T22:55:58.439 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:55:58.462 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:58.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:58.463 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:55:58.530 INFO:tasks.rados.rados.0.smithi084.stdout:3490: finishing write tid 1 to smithi08477203-40 2024-09-20T22:55:58.530 INFO:tasks.rados.rados.0.smithi084.stdout:3490: finishing write tid 2 to smithi08477203-40 2024-09-20T22:55:58.530 INFO:tasks.rados.rados.0.smithi084.stdout:3490: finishing write tid 3 to smithi08477203-40 2024-09-20T22:55:58.530 INFO:tasks.rados.rados.0.smithi084.stdout:3490: finishing write tid 4 to smithi08477203-40 2024-09-20T22:55:58.531 INFO:tasks.rados.rados.0.smithi084.stdout:3490: oid 40 updating version 0 to 2736 2024-09-20T22:55:58.531 INFO:tasks.rados.rados.0.smithi084.stdout:3490: oid 40 version 2736 is already newer than 2735 2024-09-20T22:55:58.531 INFO:tasks.rados.rados.0.smithi084.stdout:3490: oid 40 version 2736 is already newer than 2734 2024-09-20T22:55:58.531 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 40 v 2736 (ObjNum 1190 snap 409 seq_num 1190) dirty exists 2024-09-20T22:55:58.531 INFO:tasks.rados.rados.0.smithi084.stdout:3490: left oid 40 (ObjNum 1190 snap 409 seq_num 1190) 2024-09-20T22:55:58.563 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:55:58.585 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:58.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:58.585 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:55:58.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:58 smithi084 ceph-mon[29774]: 1.9 scrub starts 2024-09-20T22:55:58.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:58 smithi084 ceph-mon[29774]: 1.9 scrub ok 2024-09-20T22:55:58.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:58 smithi084 ceph-mon[29774]: osdmap e877: 8 total, 8 up, 8 in 2024-09-20T22:55:58.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:58 smithi084 ceph-mon[29774]: 4.c deep-scrub starts 2024-09-20T22:55:58.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:58 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/1235646073' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:55:58.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:58 smithi084 ceph-mon[29774]: osdmap e878: 8 total, 8 up, 8 in 2024-09-20T22:55:58.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:58 smithi084 ceph-mon[40024]: 1.9 scrub starts 2024-09-20T22:55:58.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:58 smithi084 ceph-mon[40024]: 1.9 scrub ok 2024-09-20T22:55:58.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:58 smithi084 ceph-mon[40024]: osdmap e877: 8 total, 8 up, 8 in 2024-09-20T22:55:58.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:58 smithi084 ceph-mon[40024]: 4.c deep-scrub starts 2024-09-20T22:55:58.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:58 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/1235646073' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:55:58.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:58 smithi084 ceph-mon[40024]: osdmap e878: 8 total, 8 up, 8 in 2024-09-20T22:55:58.686 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:55:58.707 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:58.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:58.708 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:55:58.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:58 smithi154 ceph-mon[33264]: 1.9 scrub starts 2024-09-20T22:55:58.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:58 smithi154 ceph-mon[33264]: 1.9 scrub ok 2024-09-20T22:55:58.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:58 smithi154 ceph-mon[33264]: osdmap e877: 8 total, 8 up, 8 in 2024-09-20T22:55:58.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:58 smithi154 ceph-mon[33264]: 4.c deep-scrub starts 2024-09-20T22:55:58.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:58 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/1235646073' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:55:58.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:58 smithi154 ceph-mon[33264]: osdmap e878: 8 total, 8 up, 8 in 2024-09-20T22:55:58.809 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:55:58.831 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:58.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:58.832 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:55:58.853 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.4 dump_historic_ops 2024-09-20T22:55:58.933 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:55:58.954 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:58.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:58.955 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:55:59.056 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:55:59.078 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:59.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:59.079 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:55:59.137 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:55:59.180 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:55:59.200 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:59.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:59.201 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:55:59.302 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:55:59.323 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:59.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:59.324 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:55:59.341 INFO:tasks.rados.rados.0.smithi084.stdout:3489: done (2 left) 2024-09-20T22:55:59.341 INFO:tasks.rados.rados.0.smithi084.stdout:3490: done (1 left) 2024-09-20T22:55:59.341 INFO:tasks.rados.rados.0.smithi084.stdout:3491: done (0 left) 2024-09-20T22:55:59.341 INFO:tasks.rados.rados.0.smithi084.stdout:3492: read oid 6 snap 384 2024-09-20T22:55:59.342 INFO:tasks.rados.rados.0.smithi084.stdout:3492: expect deleted 2024-09-20T22:55:59.342 INFO:tasks.rados.rados.0.smithi084.stdout:3493: read oid 2 snap -1 2024-09-20T22:55:59.342 INFO:tasks.rados.rados.0.smithi084.stdout:3493: expect (ObjNum 1187 snap 408 seq_num 1187) 2024-09-20T22:55:59.342 INFO:tasks.rados.rados.0.smithi084.stdout:3494: delete oid 46 current snap is 410 2024-09-20T22:55:59.346 INFO:tasks.rados.rados.0.smithi084.stdout:3492: done (2 left) 2024-09-20T22:55:59.346 INFO:tasks.rados.rados.0.smithi084.stdout:3494: done (1 left) 2024-09-20T22:55:59.346 INFO:tasks.rados.rados.0.smithi084.stdout:3495: rollback oid 11 current snap is 410 2024-09-20T22:55:59.346 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 11 to 396 2024-09-20T22:55:59.347 INFO:tasks.rados.rados.0.smithi084.stdout:3496: read oid 6 snap -1 2024-09-20T22:55:59.347 INFO:tasks.rados.rados.0.smithi084.stdout:3496: expect deleted 2024-09-20T22:55:59.347 INFO:tasks.rados.rados.0.smithi084.stdout:3497: write oid 43 current snap is 410 2024-09-20T22:55:59.347 INFO:tasks.rados.rados.0.smithi084.stdout:3497: seq_num 1192 ranges {433944=647876,1486536=547408} 2024-09-20T22:55:59.352 INFO:tasks.rados.rados.0.smithi084.stdout:3497: writing smithi08477203-43 from 433944 to 1081820 tid 1 2024-09-20T22:55:59.358 INFO:tasks.rados.rados.0.smithi084.stdout:3497: writing smithi08477203-43 from 1486536 to 2033944 tid 2 2024-09-20T22:55:59.358 INFO:tasks.rados.rados.0.smithi084.stdout:3498: copy_from oid 49 from oid 3 current snap is 410 2024-09-20T22:55:59.358 INFO:tasks.rados.rados.0.smithi084.stdout:3493: expect (ObjNum 1187 snap 408 seq_num 1187) 2024-09-20T22:55:59.424 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:55:59.445 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:59.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:59.445 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:55:59.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:59 smithi154 ceph-mon[33264]: 4.c deep-scrub ok 2024-09-20T22:55:59.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:59 smithi154 ceph-mon[33264]: pgmap v1218: 158 pgs: 2 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 152 active+clean; 639 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-09-20T22:55:59.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:59 smithi154 ceph-mon[33264]: 7.d scrub starts 2024-09-20T22:55:59.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:59 smithi154 ceph-mon[33264]: 7.d scrub ok 2024-09-20T22:55:59.528 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:55:59 smithi154 ceph-mon[33264]: osdmap e879: 8 total, 8 up, 8 in 2024-09-20T22:55:59.546 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:55:59.568 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:55:59.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:59.569 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:55:59.583 INFO:tasks.rados.rados.0.smithi084.stdout:3495: finishing rollback tid 0 to smithi08477203-11 2024-09-20T22:55:59.583 INFO:tasks.rados.rados.0.smithi084.stdout:3495: finishing rollback tid 1 to smithi08477203-11 2024-09-20T22:55:59.583 INFO:tasks.rados.rados.0.smithi084.stdout:3493: done (4 left) 2024-09-20T22:55:59.583 INFO:tasks.rados.rados.0.smithi084.stdout:3496: done (3 left) 2024-09-20T22:55:59.583 INFO:tasks.rados.rados.0.smithi084.stdout:3499: delete oid 42 current snap is 410 2024-09-20T22:55:59.584 INFO:tasks.rados.rados.0.smithi084.stdout:3495: finishing rollback tid 2 to smithi08477203-11 2024-09-20T22:55:59.584 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 11 v 2645 (ObjNum 1153 snap 396 seq_num 1153) dirty exists 2024-09-20T22:55:59.584 INFO:tasks.rados.rados.0.smithi084.stdout:3497: finishing write tid 1 to smithi08477203-43 2024-09-20T22:55:59.584 INFO:tasks.rados.rados.0.smithi084.stdout:3497: finishing write tid 2 to smithi08477203-43 2024-09-20T22:55:59.584 INFO:tasks.rados.rados.0.smithi084.stdout:3497: finishing write tid 3 to smithi08477203-43 2024-09-20T22:55:59.584 INFO:tasks.rados.rados.0.smithi084.stdout:3497: finishing write tid 4 to smithi08477203-43 2024-09-20T22:55:59.584 INFO:tasks.rados.rados.0.smithi084.stdout:3497: oid 43 updating version 0 to 1945 2024-09-20T22:55:59.584 INFO:tasks.rados.rados.0.smithi084.stdout:3497: oid 43 version 1945 is already newer than 1943 2024-09-20T22:55:59.584 INFO:tasks.rados.rados.0.smithi084.stdout:3497: oid 43 version 1945 is already newer than 1944 2024-09-20T22:55:59.584 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 43 v 1945 (ObjNum 1192 snap 410 seq_num 1192) dirty exists 2024-09-20T22:55:59.584 INFO:tasks.rados.rados.0.smithi084.stdout:3497: left oid 43 (ObjNum 1192 snap 410 seq_num 1192) 2024-09-20T22:55:59.584 INFO:tasks.rados.rados.0.smithi084.stdout:3498: finishing copy_from racing read to smithi08477203-49 2024-09-20T22:55:59.584 INFO:tasks.rados.rados.0.smithi084.stdout:3498: finishing copy_from to smithi08477203-49 2024-09-20T22:55:59.584 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 49 v 2582 (ObjNum 1177 snap 404 seq_num 1177) dirty exists 2024-09-20T22:55:59.585 INFO:tasks.rados.rados.0.smithi084.stdout:3495: done (3 left) 2024-09-20T22:55:59.585 INFO:tasks.rados.rados.0.smithi084.stdout:3497: done (2 left) 2024-09-20T22:55:59.585 INFO:tasks.rados.rados.0.smithi084.stdout:3498: done (1 left) 2024-09-20T22:55:59.586 INFO:tasks.rados.rados.0.smithi084.stdout:3499: done (0 left) 2024-09-20T22:55:59.586 INFO:tasks.rados.rados.0.smithi084.stdout:3500: read oid 15 snap -1 2024-09-20T22:55:59.586 INFO:tasks.rados.rados.0.smithi084.stdout:3500: expect (ObjNum 1097 snap 368 seq_num 1097) 2024-09-20T22:55:59.586 INFO:tasks.rados.rados.0.smithi084.stdout:3501: write (excl) oid 43 current snap is 410 2024-09-20T22:55:59.586 INFO:tasks.rados.rados.0.smithi084.stdout:3501: seq_num 1194 ranges {467558=514727,1719853=347705} 2024-09-20T22:55:59.591 INFO:tasks.rados.rados.0.smithi084.stdout:3501: writing smithi08477203-43 from 467558 to 982285 tid 1 2024-09-20T22:55:59.597 INFO:tasks.rados.rados.0.smithi084.stdout:3501: writing smithi08477203-43 from 1719853 to 2067558 tid 2 2024-09-20T22:55:59.597 INFO:tasks.rados.rados.0.smithi084.stdout:3502: read oid 49 snap -1 2024-09-20T22:55:59.597 INFO:tasks.rados.rados.0.smithi084.stdout:3502: expect (ObjNum 1177 snap 404 seq_num 1177) 2024-09-20T22:55:59.597 INFO:tasks.rados.rados.0.smithi084.stdout:3503: delete oid 20 current snap is 410 2024-09-20T22:55:59.597 INFO:tasks.rados.rados.0.smithi084.stdout:3500: expect (ObjNum 1097 snap 368 seq_num 1097) 2024-09-20T22:55:59.670 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:55:59.670 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-20T22:55:59.670 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:55:59.684 INFO:tasks.rados.rados.0.smithi084.stdout:3501: finishing write tid 1 to smithi08477203-43 2024-09-20T22:55:59.684 INFO:tasks.rados.rados.0.smithi084.stdout:3500: done (3 left) 2024-09-20T22:55:59.684 INFO:tasks.rados.rados.0.smithi084.stdout:3503: done (2 left) 2024-09-20T22:55:59.684 INFO:tasks.rados.rados.0.smithi084.stdout:3504: snap_create 2024-09-20T22:55:59.685 INFO:tasks.rados.rados.0.smithi084.stdout:3501: finishing write tid 2 to smithi08477203-43 2024-09-20T22:55:59.685 INFO:tasks.rados.rados.0.smithi084.stdout:3501: finishing write tid 3 to smithi08477203-43 2024-09-20T22:55:59.685 INFO:tasks.rados.rados.0.smithi084.stdout:3501: finishing write tid 4 to smithi08477203-43 2024-09-20T22:55:59.685 INFO:tasks.rados.rados.0.smithi084.stdout:3501: oid 43 updating version 0 to 1947 2024-09-20T22:55:59.685 INFO:tasks.rados.rados.0.smithi084.stdout:3501: oid 43 updating version 1947 to 1948 2024-09-20T22:55:59.685 INFO:tasks.rados.rados.0.smithi084.stdout:3501: oid 43 version 1948 is already newer than 1946 2024-09-20T22:55:59.685 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 43 v 1948 (ObjNum 1194 snap 410 seq_num 1194) dirty exists 2024-09-20T22:55:59.685 INFO:tasks.rados.rados.0.smithi084.stdout:3501: left oid 43 (ObjNum 1194 snap 410 seq_num 1194) 2024-09-20T22:55:59.685 INFO:tasks.rados.rados.0.smithi084.stdout:3502: expect (ObjNum 1177 snap 404 seq_num 1177) 2024-09-20T22:55:59.692 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:59.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:59.693 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:55:59.749 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:59 smithi084 ceph-mon[40024]: 4.c deep-scrub ok 2024-09-20T22:55:59.749 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:59 smithi084 ceph-mon[40024]: pgmap v1218: 158 pgs: 2 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 152 active+clean; 639 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-09-20T22:55:59.749 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:59 smithi084 ceph-mon[40024]: 7.d scrub starts 2024-09-20T22:55:59.749 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:59 smithi084 ceph-mon[40024]: 7.d scrub ok 2024-09-20T22:55:59.749 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:55:59 smithi084 ceph-mon[40024]: osdmap e879: 8 total, 8 up, 8 in 2024-09-20T22:55:59.749 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:59 smithi084 ceph-mon[29774]: 4.c deep-scrub ok 2024-09-20T22:55:59.749 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:59 smithi084 ceph-mon[29774]: pgmap v1218: 158 pgs: 2 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 152 active+clean; 639 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-09-20T22:55:59.749 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:59 smithi084 ceph-mon[29774]: 7.d scrub starts 2024-09-20T22:55:59.750 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:59 smithi084 ceph-mon[29774]: 7.d scrub ok 2024-09-20T22:55:59.750 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:55:59 smithi084 ceph-mon[29774]: osdmap e879: 8 total, 8 up, 8 in 2024-09-20T22:55:59.793 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:55:59.814 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:59.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:59.815 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:55:59.916 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:55:59.940 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:55:59.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:55:59.941 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:55:59.991 INFO:journalctl@ceph.iscsi.iscsi.a.smithi154.stdout:Sep 20 22:55:59 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-iscsi-iscsi-a[59730]: debug there is no tcmu-runner data available 2024-09-20T22:56:00.042 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:56:00.063 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:00.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:00.064 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:56:00.164 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:56:00.186 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:00.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:00.187 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:56:00.268 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-20T22:56:00.288 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:56:00.333 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:00.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:00.334 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:56:00.434 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:56:00.449 INFO:teuthology.orchestra.run.smithi084.stderr:clear 6.e pg_upmap mapping 2024-09-20T22:56:00.455 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is set 2024-09-20T22:56:00.456 INFO:tasks.rados.rados.0.smithi084.stdout:3501: done (2 left) 2024-09-20T22:56:00.456 INFO:tasks.rados.rados.0.smithi084.stdout:3502: done (1 left) 2024-09-20T22:56:00.456 INFO:tasks.rados.rados.0.smithi084.stdout:3504: done (0 left) 2024-09-20T22:56:00.456 INFO:tasks.rados.rados.0.smithi084.stdout:3505: rollback oid 25 current snap is 411 2024-09-20T22:56:00.456 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 25 to 346 2024-09-20T22:56:00.457 INFO:tasks.rados.rados.0.smithi084.stdout:3506: write (excl) oid 39 current snap is 411 2024-09-20T22:56:00.457 INFO:tasks.rados.rados.0.smithi084.stdout:3506: seq_num 1196 ranges {501172=665225,1720464=380708} 2024-09-20T22:56:00.462 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:00.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:00.463 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:56:00.464 INFO:tasks.rados.rados.0.smithi084.stdout:3506: writing smithi08477203-39 from 501172 to 1166397 tid 1 2024-09-20T22:56:00.471 INFO:tasks.rados.rados.0.smithi084.stdout:3506: writing smithi08477203-39 from 1720464 to 2101172 tid 2 2024-09-20T22:56:00.471 INFO:tasks.rados.rados.0.smithi084.stdout:3507: write (excl) oid 8 current snap is 411 2024-09-20T22:56:00.471 INFO:tasks.rados.rados.0.smithi084.stdout:3505: finishing rollback tid 0 to smithi08477203-25 2024-09-20T22:56:00.471 INFO:tasks.rados.rados.0.smithi084.stdout:3506: finishing write tid 1 to smithi08477203-39 2024-09-20T22:56:00.471 INFO:tasks.rados.rados.0.smithi084.stdout:3505: finishing rollback tid 1 to smithi08477203-25 2024-09-20T22:56:00.472 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 25 v 2821 (ObjNum 289 snap 0 seq_num 0) dirty dne 2024-09-20T22:56:00.472 INFO:tasks.rados.rados.0.smithi084.stdout:3507: seq_num 1197 ranges {517979=740474,1778946=339033} 2024-09-20T22:56:00.478 INFO:tasks.rados.rados.0.smithi084.stdout:3507: writing smithi08477203-8 from 517979 to 1258453 tid 1 2024-09-20T22:56:00.484 INFO:tasks.rados.rados.0.smithi084.stdout:3507: writing smithi08477203-8 from 1778946 to 2117979 tid 2 2024-09-20T22:56:00.484 INFO:tasks.rados.rados.0.smithi084.stdout:3505: done (2 left) 2024-09-20T22:56:00.484 INFO:tasks.rados.rados.0.smithi084.stdout:3508: read oid 19 snap -1 2024-09-20T22:56:00.484 INFO:tasks.rados.rados.0.smithi084.stdout:3508: expect (ObjNum 1174 snap 403 seq_num 1174) 2024-09-20T22:56:00.484 INFO:tasks.rados.rados.0.smithi084.stdout:3509: rollback oid 14 current snap is 411 2024-09-20T22:56:00.484 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 14 to 301 2024-09-20T22:56:00.484 INFO:tasks.rados.rados.0.smithi084.stdout:3506: finishing write tid 2 to smithi08477203-39 2024-09-20T22:56:00.484 INFO:tasks.rados.rados.0.smithi084.stdout:3506: finishing write tid 3 to smithi08477203-39 2024-09-20T22:56:00.484 INFO:tasks.rados.rados.0.smithi084.stdout:3506: finishing write tid 4 to smithi08477203-39 2024-09-20T22:56:00.484 INFO:tasks.rados.rados.0.smithi084.stdout:3506: oid 39 updating version 0 to 2648 2024-09-20T22:56:00.484 INFO:tasks.rados.rados.0.smithi084.stdout:3506: oid 39 updating version 2648 to 2649 2024-09-20T22:56:00.484 INFO:tasks.rados.rados.0.smithi084.stdout:3506: oid 39 version 2649 is already newer than 2647 2024-09-20T22:56:00.484 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 39 v 2649 (ObjNum 1196 snap 411 seq_num 1196) dirty exists 2024-09-20T22:56:00.485 INFO:tasks.rados.rados.0.smithi084.stdout:3506: left oid 39 (ObjNum 1196 snap 411 seq_num 1196) 2024-09-20T22:56:00.485 INFO:tasks.rados.rados.0.smithi084.stdout:3507: finishing write tid 1 to smithi08477203-8 2024-09-20T22:56:00.485 INFO:tasks.rados.rados.0.smithi084.stdout:3506: done (3 left) 2024-09-20T22:56:00.485 INFO:tasks.rados.rados.0.smithi084.stdout:3510: read oid 6 snap -1 2024-09-20T22:56:00.485 INFO:tasks.rados.rados.0.smithi084.stdout:3510: expect deleted 2024-09-20T22:56:00.485 INFO:tasks.rados.rados.0.smithi084.stdout:3511: read oid 26 snap 342 2024-09-20T22:56:00.485 INFO:tasks.rados.rados.0.smithi084.stdout:3511: expect (ObjNum 762 snap 243 seq_num 762) 2024-09-20T22:56:00.485 INFO:tasks.rados.rados.0.smithi084.stdout:3512: snap_remove snap 311 2024-09-20T22:56:00.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:00 smithi154 ceph-mon[33264]: 3.b deep-scrub starts 2024-09-20T22:56:00.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:00 smithi154 ceph-mon[33264]: 3.b deep-scrub ok 2024-09-20T22:56:00.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:00 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/3999613298' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-20T22:56:00.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:00 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-20T22:56:00.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:00 smithi154 ceph-mon[33264]: 7.7 deep-scrub starts 2024-09-20T22:56:00.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:00 smithi154 ceph-mon[33264]: 7.7 deep-scrub ok 2024-09-20T22:56:00.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:00 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/1825561953' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap", "pgid": "6.e"}]: dispatch 2024-09-20T22:56:00.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:00 smithi154 ceph-mon[33264]: 3.0 deep-scrub starts 2024-09-20T22:56:00.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:00 smithi154 ceph-mon[33264]: 3.0 deep-scrub ok 2024-09-20T22:56:00.522 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:56:00.564 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:56:00.585 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:00.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:00.586 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:56:00.687 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:56:00.708 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:00.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:00.709 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:56:00.778 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:00 smithi084 ceph-mon[40024]: 3.b deep-scrub starts 2024-09-20T22:56:00.778 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:00 smithi084 ceph-mon[40024]: 3.b deep-scrub ok 2024-09-20T22:56:00.778 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:00 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/3999613298' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-20T22:56:00.778 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:00 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-20T22:56:00.778 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:00 smithi084 ceph-mon[40024]: 7.7 deep-scrub starts 2024-09-20T22:56:00.778 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:00 smithi084 ceph-mon[40024]: 7.7 deep-scrub ok 2024-09-20T22:56:00.778 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:00 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/1825561953' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap", "pgid": "6.e"}]: dispatch 2024-09-20T22:56:00.779 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:00 smithi084 ceph-mon[40024]: 3.0 deep-scrub starts 2024-09-20T22:56:00.779 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:00 smithi084 ceph-mon[40024]: 3.0 deep-scrub ok 2024-09-20T22:56:00.779 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:00 smithi084 ceph-mon[29774]: 3.b deep-scrub starts 2024-09-20T22:56:00.779 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:00 smithi084 ceph-mon[29774]: 3.b deep-scrub ok 2024-09-20T22:56:00.779 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:00 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/3999613298' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-20T22:56:00.779 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:00 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-20T22:56:00.779 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:00 smithi084 ceph-mon[29774]: 7.7 deep-scrub starts 2024-09-20T22:56:00.779 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:00 smithi084 ceph-mon[29774]: 7.7 deep-scrub ok 2024-09-20T22:56:00.779 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:00 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/1825561953' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap", "pgid": "6.e"}]: dispatch 2024-09-20T22:56:00.779 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:00 smithi084 ceph-mon[29774]: 3.0 deep-scrub starts 2024-09-20T22:56:00.779 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:00 smithi084 ceph-mon[29774]: 3.0 deep-scrub ok 2024-09-20T22:56:00.810 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:56:00.831 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:00.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:00.832 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:56:00.933 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:56:00.954 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:00.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:00.955 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:56:00.983 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-20T22:56:00.984 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:56:01.056 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:56:01.078 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:01.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:01.079 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:56:01.180 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:56:01.198 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-20T22:56:01.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:01.202 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:01.202 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:56:01.304 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:56:01.325 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:01.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:01.326 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:56:01.426 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:56:01.448 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:01.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:01.449 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:56:01.469 INFO:tasks.rados.rados.0.smithi084.stdout:3512: done (5 left) 2024-09-20T22:56:01.469 INFO:tasks.rados.rados.0.smithi084.stdout:3513: rollback oid 17 current snap is 411 2024-09-20T22:56:01.469 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 17 to 382 2024-09-20T22:56:01.469 INFO:tasks.rados.rados.0.smithi084.stdout:3507: finishing write tid 2 to smithi08477203-8 2024-09-20T22:56:01.469 INFO:tasks.rados.rados.0.smithi084.stdout:3507: finishing write tid 3 to smithi08477203-8 2024-09-20T22:56:01.469 INFO:tasks.rados.rados.0.smithi084.stdout:3507: finishing write tid 4 to smithi08477203-8 2024-09-20T22:56:01.470 INFO:tasks.rados.rados.0.smithi084.stdout:3507: oid 8 updating version 0 to 2588 2024-09-20T22:56:01.470 INFO:tasks.rados.rados.0.smithi084.stdout:3507: oid 8 version 2588 is already newer than 2587 2024-09-20T22:56:01.470 INFO:tasks.rados.rados.0.smithi084.stdout:3507: oid 8 updating version 2588 to 2589 2024-09-20T22:56:01.470 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 8 v 2589 (ObjNum 1197 snap 411 seq_num 1197) dirty exists 2024-09-20T22:56:01.470 INFO:tasks.rados.rados.0.smithi084.stdout:3507: left oid 8 (ObjNum 1197 snap 411 seq_num 1197) 2024-09-20T22:56:01.470 INFO:tasks.rados.rados.0.smithi084.stdout:3509: finishing rollback tid 0 to smithi08477203-14 2024-09-20T22:56:01.470 INFO:tasks.rados.rados.0.smithi084.stdout:3509: finishing rollback tid 1 to smithi08477203-14 2024-09-20T22:56:01.470 INFO:tasks.rados.rados.0.smithi084.stdout:3509: finishing rollback tid 2 to smithi08477203-14 2024-09-20T22:56:01.470 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 14 v 1949 (ObjNum 618 snap 184 seq_num 618) dirty exists 2024-09-20T22:56:01.470 INFO:tasks.rados.rados.0.smithi084.stdout:3508: expect (ObjNum 1174 snap 403 seq_num 1174) 2024-09-20T22:56:01.550 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:56:01.573 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:01.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:01.574 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:56:01.580 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.5 dump_blocked_ops 2024-09-20T22:56:01.674 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:56:01.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:01.696 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:01.696 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:56:01.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:01 smithi154 ceph-mon[33264]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:56:01.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:01 smithi154 ceph-mon[33264]: pgmap v1221: 158 pgs: 2 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 152 active+clean; 653 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 4.2 MiB/s rd, 2.4 MiB/s wr, 13 op/s 2024-09-20T22:56:01.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:01 smithi154 ceph-mon[33264]: 5.1a scrub starts 2024-09-20T22:56:01.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:01 smithi154 ceph-mon[33264]: 5.1a scrub ok 2024-09-20T22:56:01.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:01 smithi154 ceph-mon[33264]: 1.a scrub starts 2024-09-20T22:56:01.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:01 smithi154 ceph-mon[33264]: 1.a scrub ok 2024-09-20T22:56:01.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:01 smithi154 ceph-mon[33264]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:56:01.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:01 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T22:56:01.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:01 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/1825561953' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap", "pgid": "6.e"}]': finished 2024-09-20T22:56:01.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:01 smithi154 ceph-mon[33264]: osdmap e880: 8 total, 8 up, 8 in 2024-09-20T22:56:01.766 INFO:tasks.rados.rados.0.smithi084.stdout:3511: expect (ObjNum 762 snap 243 seq_num 762) 2024-09-20T22:56:01.796 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:56:01.820 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:01.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:01.821 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:56:01.827 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:56:01.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:01 smithi084 ceph-mon[29774]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:56:01.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:01 smithi084 ceph-mon[29774]: pgmap v1221: 158 pgs: 2 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 152 active+clean; 653 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 4.2 MiB/s rd, 2.4 MiB/s wr, 13 op/s 2024-09-20T22:56:01.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:01 smithi084 ceph-mon[29774]: 5.1a scrub starts 2024-09-20T22:56:01.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:01 smithi084 ceph-mon[29774]: 5.1a scrub ok 2024-09-20T22:56:01.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:01 smithi084 ceph-mon[29774]: 1.a scrub starts 2024-09-20T22:56:01.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:01 smithi084 ceph-mon[29774]: 1.a scrub ok 2024-09-20T22:56:01.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:01 smithi084 ceph-mon[29774]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:56:01.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:01 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T22:56:01.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:01 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/1825561953' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap", "pgid": "6.e"}]': finished 2024-09-20T22:56:01.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:01 smithi084 ceph-mon[29774]: osdmap e880: 8 total, 8 up, 8 in 2024-09-20T22:56:01.866 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:01 smithi084 ceph-mon[40024]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:56:01.866 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:01 smithi084 ceph-mon[40024]: pgmap v1221: 158 pgs: 2 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 152 active+clean; 653 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 4.2 MiB/s rd, 2.4 MiB/s wr, 13 op/s 2024-09-20T22:56:01.866 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:01 smithi084 ceph-mon[40024]: 5.1a scrub starts 2024-09-20T22:56:01.866 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:01 smithi084 ceph-mon[40024]: 5.1a scrub ok 2024-09-20T22:56:01.866 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:01 smithi084 ceph-mon[40024]: 1.a scrub starts 2024-09-20T22:56:01.866 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:01 smithi084 ceph-mon[40024]: 1.a scrub ok 2024-09-20T22:56:01.866 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:01 smithi084 ceph-mon[40024]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:56:01.866 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:01 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T22:56:01.866 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:01 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/1825561953' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap", "pgid": "6.e"}]': finished 2024-09-20T22:56:01.866 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:01 smithi084 ceph-mon[40024]: osdmap e880: 8 total, 8 up, 8 in 2024-09-20T22:56:01.890 INFO:tasks.rados.rados.0.smithi084.stdout:3513: finishing rollback tid 0 to smithi08477203-17 2024-09-20T22:56:01.890 INFO:tasks.rados.rados.0.smithi084.stdout:3513: finishing rollback tid 1 to smithi08477203-17 2024-09-20T22:56:01.890 INFO:tasks.rados.rados.0.smithi084.stdout:3507: done (5 left) 2024-09-20T22:56:01.890 INFO:tasks.rados.rados.0.smithi084.stdout:3508: done (4 left) 2024-09-20T22:56:01.890 INFO:tasks.rados.rados.0.smithi084.stdout:3509: done (3 left) 2024-09-20T22:56:01.890 INFO:tasks.rados.rados.0.smithi084.stdout:3510: done (2 left) 2024-09-20T22:56:01.890 INFO:tasks.rados.rados.0.smithi084.stdout:3511: done (1 left) 2024-09-20T22:56:01.890 INFO:tasks.rados.rados.0.smithi084.stdout:3514: write oid 28 current snap is 411 2024-09-20T22:56:01.890 INFO:tasks.rados.rados.0.smithi084.stdout:3513: finishing rollback tid 2 to smithi08477203-17 2024-09-20T22:56:01.890 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 17 v 2647 (ObjNum 1126 snap 381 seq_num 1126) dirty exists 2024-09-20T22:56:01.891 INFO:tasks.rados.rados.0.smithi084.stdout:3514: seq_num 1198 ranges {534786=415723,1437428=697358} 2024-09-20T22:56:01.894 INFO:tasks.rados.rados.0.smithi084.stdout:3514: writing smithi08477203-28 from 534786 to 950509 tid 1 2024-09-20T22:56:01.901 INFO:tasks.rados.rados.0.smithi084.stdout:3514: writing smithi08477203-28 from 1437428 to 2134786 tid 2 2024-09-20T22:56:01.902 INFO:tasks.rados.rados.0.smithi084.stdout:3513: done (1 left) 2024-09-20T22:56:01.902 INFO:tasks.rados.rados.0.smithi084.stdout:3515: read oid 23 snap 385 2024-09-20T22:56:01.902 INFO:tasks.rados.rados.0.smithi084.stdout:3515: expect (ObjNum 1135 snap 382 seq_num 1135) 2024-09-20T22:56:01.902 INFO:tasks.rados.rados.0.smithi084.stdout:3516: snap_remove snap 388 2024-09-20T22:56:01.902 INFO:tasks.rados.rados.0.smithi084.stdout:3514: finishing write tid 1 to smithi08477203-28 2024-09-20T22:56:01.921 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:56:01.943 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:01.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:01.944 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:56:02.045 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:56:02.066 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:02.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:02.067 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:56:02.083 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:56:02.168 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:56:02.189 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:02.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:02.190 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:56:02.291 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:56:02.306 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-20T22:56:02.306 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:56:02.312 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:02.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:02.313 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:56:02.414 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:56:02.435 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:02.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:02.436 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:56:02.476 INFO:tasks.rados.rados.0.smithi084.stdout:3516: done (2 left) 2024-09-20T22:56:02.476 INFO:tasks.rados.rados.0.smithi084.stdout:3517: write oid 34 current snap is 411 2024-09-20T22:56:02.476 INFO:tasks.rados.rados.0.smithi084.stdout:3517: seq_num 1199 ranges {551593=490972,1779557=372036} 2024-09-20T22:56:02.482 INFO:tasks.rados.rados.0.smithi084.stdout:3517: writing smithi08477203-34 from 551593 to 1042565 tid 1 2024-09-20T22:56:02.489 INFO:tasks.rados.rados.0.smithi084.stdout:3517: writing smithi08477203-34 from 1779557 to 2151593 tid 2 2024-09-20T22:56:02.489 INFO:tasks.rados.rados.0.smithi084.stdout:3518: write oid 16 current snap is 411 2024-09-20T22:56:02.489 INFO:tasks.rados.rados.0.smithi084.stdout:3518: seq_num 1200 ranges {568400=566221,1838039=330361} 2024-09-20T22:56:02.495 INFO:tasks.rados.rados.0.smithi084.stdout:3518: writing smithi08477203-16 from 568400 to 1134621 tid 1 2024-09-20T22:56:02.501 INFO:tasks.rados.rados.0.smithi084.stdout:3518: writing smithi08477203-16 from 1838039 to 2168400 tid 2 2024-09-20T22:56:02.501 INFO:tasks.rados.rados.0.smithi084.stdout:3519: write oid 33 current snap is 411 2024-09-20T22:56:02.501 INFO:tasks.rados.rados.0.smithi084.stdout:3514: finishing write tid 2 to smithi08477203-28 2024-09-20T22:56:02.501 INFO:tasks.rados.rados.0.smithi084.stdout:3519: seq_num 1201 ranges {585207=641470,1896521=288686} 2024-09-20T22:56:02.507 INFO:tasks.rados.rados.0.smithi084.stdout:3519: writing smithi08477203-33 from 585207 to 1226677 tid 1 2024-09-20T22:56:02.513 INFO:tasks.rados.rados.0.smithi084.stdout:3519: writing smithi08477203-33 from 1896521 to 2185207 tid 2 2024-09-20T22:56:02.513 INFO:tasks.rados.rados.0.smithi084.stdout:3520: snap_remove snap 383 2024-09-20T22:56:02.537 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:56:02.559 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:02.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:02.560 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:56:02.570 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:02 smithi084 ceph-mon[29774]: osdmap e881: 8 total, 8 up, 8 in 2024-09-20T22:56:02.570 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:02 smithi084 ceph-mon[40024]: osdmap e881: 8 total, 8 up, 8 in 2024-09-20T22:56:02.572 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:56:02.572 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:56:02.589 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:56:02.589 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:56:02.605 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:56:02.605 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:56:02.620 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-20T22:56:02.621 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T22:56:02.637 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:56:02.637 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:56:02.654 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:56:02.654 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:56:02.661 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:56:02.670 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:56:02.670 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:56:02.682 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:02.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:02.683 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:56:02.687 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:56:02.687 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:56:02.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:02 smithi154 ceph-mon[33264]: osdmap e881: 8 total, 8 up, 8 in 2024-09-20T22:56:02.784 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:56:02.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:02.807 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:02.808 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:56:02.908 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:56:02.925 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.5 dump_historic_ops 2024-09-20T22:56:02.929 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:02.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:02.930 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:56:03.031 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:56:03.052 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:03.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:03.053 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:56:03.154 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:56:03.169 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:56:03.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:03.177 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:03.178 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:56:03.278 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:56:03.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:03.302 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:03.302 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:56:03.343 INFO:tasks.rados.rados.0.smithi084.stdout:3520: done (5 left) 2024-09-20T22:56:03.343 INFO:tasks.rados.rados.0.smithi084.stdout:3521: read oid 48 snap -1 2024-09-20T22:56:03.343 INFO:tasks.rados.rados.0.smithi084.stdout:3521: expect (ObjNum 1141 snap 385 seq_num 1141) 2024-09-20T22:56:03.343 INFO:tasks.rados.rados.0.smithi084.stdout:3522: write (excl) oid 44 current snap is 411 2024-09-20T22:56:03.343 INFO:tasks.rados.rados.0.smithi084.stdout:3522: seq_num 1202 ranges {602014=433072,1671356=530658} 2024-09-20T22:56:03.354 INFO:tasks.rados.rados.0.smithi084.stdout:3522: writing smithi08477203-44 from 602014 to 1035086 tid 1 2024-09-20T22:56:03.362 INFO:tasks.rados.rados.0.smithi084.stdout:3522: writing smithi08477203-44 from 1671356 to 2202014 tid 2 2024-09-20T22:56:03.362 INFO:tasks.rados.rados.0.smithi084.stdout:3523: read oid 4 snap -1 2024-09-20T22:56:03.362 INFO:tasks.rados.rados.0.smithi084.stdout:3523: expect (ObjNum 1111 snap 378 seq_num 1111) 2024-09-20T22:56:03.363 INFO:tasks.rados.rados.0.smithi084.stdout:3524: rollback oid 27 current snap is 411 2024-09-20T22:56:03.363 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 27 to 333 2024-09-20T22:56:03.364 INFO:tasks.rados.rados.0.smithi084.stdout:3514: finishing write tid 3 to smithi08477203-28 2024-09-20T22:56:03.364 INFO:tasks.rados.rados.0.smithi084.stdout:3514: finishing write tid 4 to smithi08477203-28 2024-09-20T22:56:03.364 INFO:tasks.rados.rados.0.smithi084.stdout:3514: oid 28 updating version 0 to 2833 2024-09-20T22:56:03.364 INFO:tasks.rados.rados.0.smithi084.stdout:3514: oid 28 updating version 2833 to 2834 2024-09-20T22:56:03.364 INFO:tasks.rados.rados.0.smithi084.stdout:3514: oid 28 updating version 2834 to 2835 2024-09-20T22:56:03.364 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 28 v 2835 (ObjNum 1198 snap 411 seq_num 1198) dirty exists 2024-09-20T22:56:03.365 INFO:tasks.rados.rados.0.smithi084.stdout:3514: left oid 28 (ObjNum 1198 snap 411 seq_num 1198) 2024-09-20T22:56:03.365 INFO:tasks.rados.rados.0.smithi084.stdout:3515: expect (ObjNum 1135 snap 382 seq_num 1135) 2024-09-20T22:56:03.403 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:56:03.426 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:03.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:03.427 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:56:03.479 INFO:tasks.rados.rados.0.smithi084.stdout:3517: finishing write tid 1 to smithi08477203-34 2024-09-20T22:56:03.480 INFO:tasks.rados.rados.0.smithi084.stdout:3517: finishing write tid 2 to smithi08477203-34 2024-09-20T22:56:03.480 INFO:tasks.rados.rados.0.smithi084.stdout:3517: finishing write tid 3 to smithi08477203-34 2024-09-20T22:56:03.480 INFO:tasks.rados.rados.0.smithi084.stdout:3517: finishing write tid 4 to smithi08477203-34 2024-09-20T22:56:03.480 INFO:tasks.rados.rados.0.smithi084.stdout:3517: oid 34 updating version 0 to 2583 2024-09-20T22:56:03.480 INFO:tasks.rados.rados.0.smithi084.stdout:3517: oid 34 updating version 2583 to 2585 2024-09-20T22:56:03.480 INFO:tasks.rados.rados.0.smithi084.stdout:3517: oid 34 version 2585 is already newer than 2584 2024-09-20T22:56:03.480 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 34 v 2585 (ObjNum 1199 snap 411 seq_num 1199) dirty exists 2024-09-20T22:56:03.480 INFO:tasks.rados.rados.0.smithi084.stdout:3517: left oid 34 (ObjNum 1199 snap 411 seq_num 1199) 2024-09-20T22:56:03.480 INFO:tasks.rados.rados.0.smithi084.stdout:3518: finishing write tid 1 to smithi08477203-16 2024-09-20T22:56:03.480 INFO:tasks.rados.rados.0.smithi084.stdout:3518: finishing write tid 2 to smithi08477203-16 2024-09-20T22:56:03.480 INFO:tasks.rados.rados.0.smithi084.stdout:3518: finishing write tid 3 to smithi08477203-16 2024-09-20T22:56:03.480 INFO:tasks.rados.rados.0.smithi084.stdout:3514: done (8 left) 2024-09-20T22:56:03.480 INFO:tasks.rados.rados.0.smithi084.stdout:3515: done (7 left) 2024-09-20T22:56:03.480 INFO:tasks.rados.rados.0.smithi084.stdout:3517: done (6 left) 2024-09-20T22:56:03.481 INFO:tasks.rados.rados.0.smithi084.stdout:3525: read oid 32 snap -1 2024-09-20T22:56:03.481 INFO:tasks.rados.rados.0.smithi084.stdout:3525: expect (ObjNum 1180 snap 405 seq_num 1180) 2024-09-20T22:56:03.481 INFO:tasks.rados.rados.0.smithi084.stdout:3526: delete oid 11 current snap is 411 2024-09-20T22:56:03.481 INFO:tasks.rados.rados.0.smithi084.stdout:3518: finishing write tid 4 to smithi08477203-16 2024-09-20T22:56:03.481 INFO:tasks.rados.rados.0.smithi084.stdout:3518: oid 16 updating version 0 to 1827 2024-09-20T22:56:03.481 INFO:tasks.rados.rados.0.smithi084.stdout:3518: oid 16 version 1827 is already newer than 1826 2024-09-20T22:56:03.481 INFO:tasks.rados.rados.0.smithi084.stdout:3518: oid 16 version 1827 is already newer than 1825 2024-09-20T22:56:03.481 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 16 v 1827 (ObjNum 1200 snap 411 seq_num 1200) dirty exists 2024-09-20T22:56:03.481 INFO:tasks.rados.rados.0.smithi084.stdout:3518: left oid 16 (ObjNum 1200 snap 411 seq_num 1200) 2024-09-20T22:56:03.481 INFO:tasks.rados.rados.0.smithi084.stdout:3519: finishing write tid 1 to smithi08477203-33 2024-09-20T22:56:03.482 INFO:tasks.rados.rados.0.smithi084.stdout:3519: finishing write tid 2 to smithi08477203-33 2024-09-20T22:56:03.482 INFO:tasks.rados.rados.0.smithi084.stdout:3519: finishing write tid 3 to smithi08477203-33 2024-09-20T22:56:03.482 INFO:tasks.rados.rados.0.smithi084.stdout:3519: finishing write tid 4 to smithi08477203-33 2024-09-20T22:56:03.482 INFO:tasks.rados.rados.0.smithi084.stdout:3519: oid 33 updating version 0 to 2739 2024-09-20T22:56:03.482 INFO:tasks.rados.rados.0.smithi084.stdout:3519: oid 33 version 2739 is already newer than 2737 2024-09-20T22:56:03.482 INFO:tasks.rados.rados.0.smithi084.stdout:3519: oid 33 version 2739 is already newer than 2738 2024-09-20T22:56:03.482 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 33 v 2739 (ObjNum 1201 snap 411 seq_num 1201) dirty exists 2024-09-20T22:56:03.482 INFO:tasks.rados.rados.0.smithi084.stdout:3519: left oid 33 (ObjNum 1201 snap 411 seq_num 1201) 2024-09-20T22:56:03.482 INFO:tasks.rados.rados.0.smithi084.stdout:3521: expect (ObjNum 1141 snap 385 seq_num 1141) 2024-09-20T22:56:03.528 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:56:03.549 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:03.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:03.550 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:56:03.560 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:03 smithi154 ceph-mon[33264]: pgmap v1224: 158 pgs: 2 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 152 active+clean; 653 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 4.2 MiB/s rd, 2.4 MiB/s wr, 13 op/s 2024-09-20T22:56:03.560 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:03 smithi154 ceph-mon[33264]: osdmap e882: 8 total, 8 up, 8 in 2024-09-20T22:56:03.561 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:03 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/224383180' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:56:03.561 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:03 smithi154 ceph-mon[33264]: osdmap e883: 8 total, 8 up, 8 in 2024-09-20T22:56:03.619 INFO:tasks.rados.rados.0.smithi084.stdout:3522: finishing write tid 1 to smithi08477203-44 2024-09-20T22:56:03.619 INFO:tasks.rados.rados.0.smithi084.stdout:3524: finishing rollback tid 0 to smithi08477203-27 2024-09-20T22:56:03.619 INFO:tasks.rados.rados.0.smithi084.stdout:3524: finishing rollback tid 1 to smithi08477203-27 2024-09-20T22:56:03.619 INFO:tasks.rados.rados.0.smithi084.stdout:3524: finishing rollback tid 2 to smithi08477203-27 2024-09-20T22:56:03.619 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 27 v 2491 (ObjNum 1016 snap 333 seq_num 1016) dirty exists 2024-09-20T22:56:03.619 INFO:tasks.rados.rados.0.smithi084.stdout:3522: finishing write tid 2 to smithi08477203-44 2024-09-20T22:56:03.619 INFO:tasks.rados.rados.0.smithi084.stdout:3523: expect (ObjNum 1111 snap 378 seq_num 1111) 2024-09-20T22:56:03.651 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:56:03.672 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:03.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:03.673 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:56:03.708 INFO:tasks.rados.rados.0.smithi084.stdout:3522: finishing write tid 3 to smithi08477203-44 2024-09-20T22:56:03.708 INFO:tasks.rados.rados.0.smithi084.stdout:3522: finishing write tid 4 to smithi08477203-44 2024-09-20T22:56:03.708 INFO:tasks.rados.rados.0.smithi084.stdout:3522: oid 44 updating version 0 to 2842 2024-09-20T22:56:03.708 INFO:tasks.rados.rados.0.smithi084.stdout:3522: oid 44 version 2842 is already newer than 2841 2024-09-20T22:56:03.708 INFO:tasks.rados.rados.0.smithi084.stdout:3522: oid 44 version 2842 is already newer than 2836 2024-09-20T22:56:03.708 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 44 v 2842 (ObjNum 1202 snap 411 seq_num 1202) dirty exists 2024-09-20T22:56:03.708 INFO:tasks.rados.rados.0.smithi084.stdout:3522: left oid 44 (ObjNum 1202 snap 411 seq_num 1202) 2024-09-20T22:56:03.708 INFO:tasks.rados.rados.0.smithi084.stdout:3518: done (7 left) 2024-09-20T22:56:03.708 INFO:tasks.rados.rados.0.smithi084.stdout:3519: done (6 left) 2024-09-20T22:56:03.708 INFO:tasks.rados.rados.0.smithi084.stdout:3521: done (5 left) 2024-09-20T22:56:03.709 INFO:tasks.rados.rados.0.smithi084.stdout:3522: done (4 left) 2024-09-20T22:56:03.709 INFO:tasks.rados.rados.0.smithi084.stdout:3523: done (3 left) 2024-09-20T22:56:03.709 INFO:tasks.rados.rados.0.smithi084.stdout:3524: done (2 left) 2024-09-20T22:56:03.709 INFO:tasks.rados.rados.0.smithi084.stdout:3526: done (1 left) 2024-09-20T22:56:03.709 INFO:tasks.rados.rados.0.smithi084.stdout:3527: write (excl) oid 11 current snap is 411 2024-09-20T22:56:03.709 INFO:tasks.rados.rados.0.smithi084.stdout:3525: expect (ObjNum 1180 snap 405 seq_num 1180) 2024-09-20T22:56:03.714 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-20T22:56:03.714 INFO:teuthology.orchestra.run.smithi154.stdout: "size": 20, 2024-09-20T22:56:03.714 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 600, 2024-09-20T22:56:03.714 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [ 2024-09-20T22:56:03.715 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.715 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:3134 7.2 7:45b54b6b:::smithi08477203-2:head [copy-from ver 151] snapc 77=[77,76,75,74,73,70,6b] ondisk+write+known_if_redirected+supports_pool_eio e328)", 2024-09-20T22:56:03.715 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:46:48.211103+0000", 2024-09-20T22:56:03.715 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 555.44889901399995, 2024-09-20T22:56:03.715 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.030812519, 2024-09-20T22:56:03.715 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:56:03.715 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:56:03.715 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:56:03.715 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:56:03.715 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:56:03.715 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:56:03.715 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 3134 2024-09-20T22:56:03.715 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.715 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:56:03.715 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.716 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:56:03.716 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.211103+0000", 2024-09-20T22:56:03.716 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:56:03.716 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.716 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.716 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:56:03.716 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.211103+0000", 2024-09-20T22:56:03.716 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:56:03.716 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.716 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.716 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:56:03.716 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.211105+0000", 2024-09-20T22:56:03.716 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.3369999999999998e-06 2024-09-20T22:56:03.716 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.717 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.717 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:56:03.717 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.211111+0000", 2024-09-20T22:56:03.717 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.2759999999999997e-06 2024-09-20T22:56:03.717 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.717 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.717 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:56:03.717 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.211123+0000", 2024-09-20T22:56:03.717 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2119999999999999e-05 2024-09-20T22:56:03.717 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.717 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.717 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:56:03.717 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.211131+0000", 2024-09-20T22:56:03.717 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.2899999999999997e-06 2024-09-20T22:56:03.717 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.718 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.718 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:56:03.718 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.211213+0000", 2024-09-20T22:56:03.718 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.2756999999999994e-05 2024-09-20T22:56:03.718 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.718 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.718 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:56:03.718 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.211310+0000", 2024-09-20T22:56:03.718 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.6644999999999994e-05 2024-09-20T22:56:03.718 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.718 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.718 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 1,2", 2024-09-20T22:56:03.718 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.229109+0000", 2024-09-20T22:56:03.718 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.017798587000000001 2024-09-20T22:56:03.719 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.719 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.719 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:56:03.719 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.240265+0000", 2024-09-20T22:56:03.719 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.011156716000000001 2024-09-20T22:56:03.719 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.719 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.719 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:56:03.719 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.241608+0000", 2024-09-20T22:56:03.719 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001343033 2024-09-20T22:56:03.719 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.719 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.719 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:56:03.719 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.241831+0000", 2024-09-20T22:56:03.720 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000223148 2024-09-20T22:56:03.720 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.720 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.720 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:56:03.720 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.241885+0000", 2024-09-20T22:56:03.720 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.3161000000000001e-05 2024-09-20T22:56:03.720 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.720 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.720 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:56:03.720 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.241915+0000", 2024-09-20T22:56:03.720 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.0448999999999999e-05 2024-09-20T22:56:03.720 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:56:03.721 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:56:03.721 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:56:03.721 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.721 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.721 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:3135 7.2 7:45b54b6b:::smithi08477203-2:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e328)", 2024-09-20T22:56:03.721 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:46:48.211153+0000", 2024-09-20T22:56:03.721 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 555.44884904200001, 2024-09-20T22:56:03.721 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.022731534000000001, 2024-09-20T22:56:03.721 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:56:03.721 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:56:03.721 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:56:03.721 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:56:03.721 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:56:03.721 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:56:03.721 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 3135 2024-09-20T22:56:03.722 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.722 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:56:03.722 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.722 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:56:03.722 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.211153+0000", 2024-09-20T22:56:03.722 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:56:03.722 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.722 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.722 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:56:03.722 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.211153+0000", 2024-09-20T22:56:03.722 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:56:03.722 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.722 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.722 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:56:03.723 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.211153+0000", 2024-09-20T22:56:03.723 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.1600000000000001e-07 2024-09-20T22:56:03.723 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.723 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.723 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:56:03.723 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.211157+0000", 2024-09-20T22:56:03.723 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.5470000000000001e-06 2024-09-20T22:56:03.723 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.723 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.723 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:56:03.723 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.211164+0000", 2024-09-20T22:56:03.723 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.2799999999999998e-06 2024-09-20T22:56:03.723 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.723 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.723 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:56:03.724 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.211170+0000", 2024-09-20T22:56:03.724 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.0730000000000001e-06 2024-09-20T22:56:03.724 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.724 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.724 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:56:03.724 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.211634+0000", 2024-09-20T22:56:03.724 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00046401800000000002 2024-09-20T22:56:03.724 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.724 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.724 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for blocked object", 2024-09-20T22:56:03.724 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.211690+0000", 2024-09-20T22:56:03.724 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.5587999999999997e-05 2024-09-20T22:56:03.724 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.724 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.725 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:56:03.725 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.233669+0000", 2024-09-20T22:56:03.725 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.021979240000000001 2024-09-20T22:56:03.725 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.725 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.725 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:56:03.725 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.233762+0000", 2024-09-20T22:56:03.725 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.2658999999999995e-05 2024-09-20T22:56:03.725 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.725 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.725 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:56:03.725 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.233884+0000", 2024-09-20T22:56:03.725 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00012241300000000001 2024-09-20T22:56:03.725 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:56:03.725 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:56:03.726 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:56:03.726 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.726 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.726 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:4444 7.0 7:05bb442d:::smithi08477203-24:head [copy-from ver 421] snapc b8=[b8,b7,b6,b5,b4,b3,b2,b1,b0,ae,a9,a7,a6,a5,9e,9c,96,95,87,86] ondisk+write+known_if_redirected+supports_pool_eio e447)", 2024-09-20T22:56:03.726 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:48:47.227411+0000", 2024-09-20T22:56:03.726 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 436.43259090599997, 2024-09-20T22:56:03.726 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.033566952999999997, 2024-09-20T22:56:03.726 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:56:03.726 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:56:03.726 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:56:03.726 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:56:03.726 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:56:03.726 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:56:03.726 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 4444 2024-09-20T22:56:03.727 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.727 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:56:03.727 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.727 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:56:03.727 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.227411+0000", 2024-09-20T22:56:03.727 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:56:03.727 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.727 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.727 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:56:03.727 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.227411+0000", 2024-09-20T22:56:03.727 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:56:03.727 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.727 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.727 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:56:03.728 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.227413+0000", 2024-09-20T22:56:03.728 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.057e-06 2024-09-20T22:56:03.728 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.728 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.728 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:56:03.728 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.227422+0000", 2024-09-20T22:56:03.728 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.3219999999999999e-06 2024-09-20T22:56:03.728 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.728 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.728 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:56:03.728 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.227442+0000", 2024-09-20T22:56:03.728 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9449999999999998e-05 2024-09-20T22:56:03.728 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.728 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.729 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:56:03.729 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.227452+0000", 2024-09-20T22:56:03.729 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0699999999999999e-05 2024-09-20T22:56:03.729 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.729 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.729 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:56:03.729 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.227531+0000", 2024-09-20T22:56:03.729 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.8993999999999996e-05 2024-09-20T22:56:03.729 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.729 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.729 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op must wait for map", 2024-09-20T22:56:03.729 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.227542+0000", 2024-09-20T22:56:03.729 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0797e-05 2024-09-20T22:56:03.729 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.730 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.730 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:56:03.730 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.229501+0000", 2024-09-20T22:56:03.730 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0019594080000000002 2024-09-20T22:56:03.730 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.730 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.730 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:56:03.730 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.229616+0000", 2024-09-20T22:56:03.730 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000114577 2024-09-20T22:56:03.730 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.730 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.730 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 1,3", 2024-09-20T22:56:03.730 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.241752+0000", 2024-09-20T22:56:03.730 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.012135920999999999 2024-09-20T22:56:03.731 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.731 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.731 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:56:03.731 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.249601+0000", 2024-09-20T22:56:03.731 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0078493120000000006 2024-09-20T22:56:03.731 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.731 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.731 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:56:03.731 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.260731+0000", 2024-09-20T22:56:03.731 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.011129781 2024-09-20T22:56:03.731 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.731 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.731 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:56:03.731 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.260861+0000", 2024-09-20T22:56:03.731 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00013027899999999999 2024-09-20T22:56:03.732 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.732 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.732 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:56:03.732 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.260913+0000", 2024-09-20T22:56:03.732 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.2172999999999997e-05 2024-09-20T22:56:03.732 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.732 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.732 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:56:03.732 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.260978+0000", 2024-09-20T22:56:03.732 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.4182000000000003e-05 2024-09-20T22:56:03.732 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:56:03.732 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:56:03.732 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:56:03.732 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.733 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.733 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:4445 7.0 7:05bb442d:::smithi08477203-24:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e447)", 2024-09-20T22:56:03.733 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:48:47.227478+0000", 2024-09-20T22:56:03.733 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 436.43252396899999, 2024-09-20T22:56:03.733 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.019465758999999999, 2024-09-20T22:56:03.733 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:56:03.733 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:56:03.733 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:56:03.733 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:56:03.733 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:56:03.733 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:56:03.733 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 4445 2024-09-20T22:56:03.733 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.733 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:56:03.733 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.734 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:56:03.734 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.227478+0000", 2024-09-20T22:56:03.734 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:56:03.734 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.734 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.734 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:56:03.734 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.227478+0000", 2024-09-20T22:56:03.734 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:56:03.734 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.734 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.734 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:56:03.734 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.227478+0000", 2024-09-20T22:56:03.734 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.2200000000000004e-07 2024-09-20T22:56:03.734 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.735 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.735 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:56:03.735 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.227481+0000", 2024-09-20T22:56:03.735 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.0060000000000001e-06 2024-09-20T22:56:03.735 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.735 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.735 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:56:03.735 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.227500+0000", 2024-09-20T22:56:03.735 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9071000000000001e-05 2024-09-20T22:56:03.735 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.735 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.735 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:56:03.735 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.227507+0000", 2024-09-20T22:56:03.735 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.1779999999999997e-06 2024-09-20T22:56:03.735 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.736 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.736 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:56:03.736 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.227628+0000", 2024-09-20T22:56:03.736 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000120627 2024-09-20T22:56:03.736 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.736 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.736 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting_for_map not empty", 2024-09-20T22:56:03.736 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.227633+0000", 2024-09-20T22:56:03.736 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.4730000000000002e-06 2024-09-20T22:56:03.736 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.736 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.736 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:56:03.736 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.229766+0000", 2024-09-20T22:56:03.736 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0021332339999999999 2024-09-20T22:56:03.736 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.737 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.737 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for blocked object", 2024-09-20T22:56:03.737 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.229814+0000", 2024-09-20T22:56:03.737 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.7746e-05 2024-09-20T22:56:03.737 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.737 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.737 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:56:03.737 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.246707+0000", 2024-09-20T22:56:03.737 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.016893542000000001 2024-09-20T22:56:03.737 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.737 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.737 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:56:03.737 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.246863+0000", 2024-09-20T22:56:03.737 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000156105 2024-09-20T22:56:03.738 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.738 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.738 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:56:03.738 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.246943+0000", 2024-09-20T22:56:03.738 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.0154999999999994e-05 2024-09-20T22:56:03.738 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:56:03.738 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:56:03.738 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:56:03.738 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.738 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.738 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:6487 7.b 7:d3ca3619:::smithi08477203-39:head [copy-from ver 1231] snapc 121=[121,120,11f,11e,11d,11b,119,118,116,115,114,112,110,10e,10c,107,105,103,fd,fb,fa,f8,f5,ed,d7,c2,bf] ondisk+write+known_if_redirected+supports_pool_eio e653)", 2024-09-20T22:56:03.738 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:52:13.479898+0000", 2024-09-20T22:56:03.738 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 230.18010339200001, 2024-09-20T22:56:03.738 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.023339255, 2024-09-20T22:56:03.739 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:56:03.739 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:56:03.739 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:56:03.739 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:56:03.739 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:56:03.739 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:56:03.739 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 6487 2024-09-20T22:56:03.739 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.739 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:56:03.739 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.739 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:56:03.739 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:13.479898+0000", 2024-09-20T22:56:03.739 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:56:03.739 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.739 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.740 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:56:03.740 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:13.479898+0000", 2024-09-20T22:56:03.740 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:56:03.740 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.740 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.740 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:56:03.740 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:13.479900+0000", 2024-09-20T22:56:03.740 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.415e-06 2024-09-20T22:56:03.740 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.740 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.740 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:56:03.740 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:13.479907+0000", 2024-09-20T22:56:03.740 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.0740000000000004e-06 2024-09-20T22:56:03.740 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.740 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.741 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:56:03.741 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:13.479916+0000", 2024-09-20T22:56:03.741 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.6420000000000002e-06 2024-09-20T22:56:03.741 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.741 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.741 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:56:03.741 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:13.479922+0000", 2024-09-20T22:56:03.741 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.1070000000000001e-06 2024-09-20T22:56:03.741 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.741 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.741 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:56:03.741 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:13.479981+0000", 2024-09-20T22:56:03.741 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.8613000000000003e-05 2024-09-20T22:56:03.742 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.742 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.742 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:56:03.742 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:13.480095+0000", 2024-09-20T22:56:03.742 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00011437799999999999 2024-09-20T22:56:03.742 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.742 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.742 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 2,6", 2024-09-20T22:56:03.742 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:13.496711+0000", 2024-09-20T22:56:03.742 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.016615938 2024-09-20T22:56:03.742 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.742 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.742 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:56:03.742 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:13.502323+0000", 2024-09-20T22:56:03.742 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0056115799999999997 2024-09-20T22:56:03.743 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.743 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.743 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:56:03.743 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:13.502878+0000", 2024-09-20T22:56:03.743 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00055555000000000003 2024-09-20T22:56:03.743 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.743 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.743 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:56:03.743 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:13.503163+0000", 2024-09-20T22:56:03.743 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000284037 2024-09-20T22:56:03.743 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.743 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.743 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:56:03.743 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:13.503201+0000", 2024-09-20T22:56:03.744 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.8569000000000003e-05 2024-09-20T22:56:03.744 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.744 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.744 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:56:03.744 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:13.503237+0000", 2024-09-20T22:56:03.744 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.6352e-05 2024-09-20T22:56:03.744 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:56:03.744 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:56:03.744 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:56:03.744 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.744 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.744 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:6488 7.b 7:d3ca3619:::smithi08477203-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e653)", 2024-09-20T22:56:03.744 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:52:13.479939+0000", 2024-09-20T22:56:03.744 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 230.18006271499999, 2024-09-20T22:56:03.745 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.019688051000000002, 2024-09-20T22:56:03.745 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:56:03.745 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:56:03.745 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:56:03.745 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:56:03.745 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:56:03.745 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:56:03.745 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 6488 2024-09-20T22:56:03.745 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.745 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:56:03.745 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.745 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:56:03.745 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:13.479939+0000", 2024-09-20T22:56:03.745 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:56:03.745 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.746 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.746 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:56:03.746 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:13.479939+0000", 2024-09-20T22:56:03.746 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:56:03.746 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.746 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.746 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:56:03.746 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:13.479940+0000", 2024-09-20T22:56:03.746 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.7199999999999998e-07 2024-09-20T22:56:03.746 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.746 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.746 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:56:03.746 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:13.479942+0000", 2024-09-20T22:56:03.746 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.853e-06 2024-09-20T22:56:03.746 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.747 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.747 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:56:03.747 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:13.479951+0000", 2024-09-20T22:56:03.747 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.1100000000000003e-06 2024-09-20T22:56:03.747 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.747 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.747 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:56:03.747 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:13.479956+0000", 2024-09-20T22:56:03.747 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.4190000000000001e-06 2024-09-20T22:56:03.747 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.747 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.747 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:56:03.747 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:13.480258+0000", 2024-09-20T22:56:03.747 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00030201400000000003 2024-09-20T22:56:03.747 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.748 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.748 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for blocked object", 2024-09-20T22:56:03.748 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:13.480300+0000", 2024-09-20T22:56:03.748 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.1816000000000001e-05 2024-09-20T22:56:03.748 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.748 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.748 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:56:03.748 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:13.499455+0000", 2024-09-20T22:56:03.748 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.019155583 2024-09-20T22:56:03.748 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.748 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.748 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:56:03.748 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:13.499533+0000", 2024-09-20T22:56:03.748 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.7972000000000005e-05 2024-09-20T22:56:03.748 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.749 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.749 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:56:03.749 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:13.499627+0000", 2024-09-20T22:56:03.749 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.3611999999999998e-05 2024-09-20T22:56:03.749 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:56:03.749 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:56:03.749 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:56:03.749 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.749 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.749 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:6770 7.0 7:064f66ab:::smithi08477203-45:head [copy-from ver 1338] snapc 135=[135,134,133,132,131,130,12f,12e,12d,12c,12b,12a,128,127,126,124,122,121,11f,11b,119,112,10e,10c,107,103,fb,fa,f8,ed,d7,c2,bf] ondisk+write+known_if_redirected+supports_pool_eio e687)", 2024-09-20T22:56:03.749 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:52:47.286358+0000", 2024-09-20T22:56:03.749 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 196.373644063, 2024-09-20T22:56:03.749 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.024891383, 2024-09-20T22:56:03.749 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:56:03.749 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:56:03.750 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:56:03.750 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:56:03.750 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:56:03.750 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:56:03.750 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 6770 2024-09-20T22:56:03.750 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.750 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:56:03.750 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.750 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:56:03.750 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:47.286358+0000", 2024-09-20T22:56:03.750 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:56:03.750 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.750 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.751 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:56:03.751 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:47.286358+0000", 2024-09-20T22:56:03.751 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:56:03.751 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.751 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.751 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:56:03.751 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:47.286360+0000", 2024-09-20T22:56:03.751 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.1260000000000001e-06 2024-09-20T22:56:03.751 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.751 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.751 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:56:03.751 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:47.286369+0000", 2024-09-20T22:56:03.751 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.431e-06 2024-09-20T22:56:03.751 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.752 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.752 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:56:03.752 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:47.286391+0000", 2024-09-20T22:56:03.752 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.1824e-05 2024-09-20T22:56:03.752 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.752 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.752 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:56:03.752 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:47.286402+0000", 2024-09-20T22:56:03.752 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0801e-05 2024-09-20T22:56:03.752 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.752 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.752 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:56:03.752 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:47.286608+0000", 2024-09-20T22:56:03.752 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00020635400000000001 2024-09-20T22:56:03.753 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.753 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.753 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op must wait for map", 2024-09-20T22:56:03.753 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:47.286616+0000", 2024-09-20T22:56:03.753 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.7389999999999999e-06 2024-09-20T22:56:03.753 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.753 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.753 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:56:03.753 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:47.289915+0000", 2024-09-20T22:56:03.753 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.003299022 2024-09-20T22:56:03.753 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.753 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.753 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:56:03.753 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:47.290021+0000", 2024-09-20T22:56:03.753 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000106267 2024-09-20T22:56:03.754 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.754 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.754 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 2,3", 2024-09-20T22:56:03.754 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:47.299972+0000", 2024-09-20T22:56:03.754 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0099504399999999996 2024-09-20T22:56:03.754 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.754 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.754 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:56:03.754 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:47.304349+0000", 2024-09-20T22:56:03.754 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0043777549999999997 2024-09-20T22:56:03.754 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.754 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.754 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:56:03.755 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:47.310757+0000", 2024-09-20T22:56:03.755 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0064077650000000002 2024-09-20T22:56:03.755 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.755 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.755 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:56:03.755 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:47.311157+0000", 2024-09-20T22:56:03.755 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00039990000000000002 2024-09-20T22:56:03.755 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.755 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.755 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:56:03.755 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:47.311206+0000", 2024-09-20T22:56:03.755 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.8906000000000002e-05 2024-09-20T22:56:03.755 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.755 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.756 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:56:03.756 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:47.311249+0000", 2024-09-20T22:56:03.756 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.3053000000000003e-05 2024-09-20T22:56:03.756 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:56:03.756 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:56:03.756 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:56:03.756 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.756 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.756 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:6771 7.0 7:064f66ab:::smithi08477203-45:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e687)", 2024-09-20T22:56:03.756 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:52:47.286426+0000", 2024-09-20T22:56:03.756 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 196.37357520899999, 2024-09-20T22:56:03.756 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.017336328000000002, 2024-09-20T22:56:03.756 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:56:03.756 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:56:03.756 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:56:03.757 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:56:03.757 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:56:03.757 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:56:03.757 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 6771 2024-09-20T22:56:03.757 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.757 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:56:03.757 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.757 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:56:03.757 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:47.286426+0000", 2024-09-20T22:56:03.757 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:56:03.757 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.757 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.757 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:56:03.757 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:47.286426+0000", 2024-09-20T22:56:03.757 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:56:03.758 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.758 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.758 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:56:03.758 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:47.286427+0000", 2024-09-20T22:56:03.758 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.2099999999999996e-07 2024-09-20T22:56:03.758 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.758 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.758 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:56:03.758 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:47.286432+0000", 2024-09-20T22:56:03.758 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.1030000000000001e-06 2024-09-20T22:56:03.758 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.758 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.758 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:56:03.758 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:47.286440+0000", 2024-09-20T22:56:03.759 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.8990000000000001e-06 2024-09-20T22:56:03.759 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.759 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.759 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:56:03.759 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:47.286446+0000", 2024-09-20T22:56:03.759 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.8089999999999998e-06 2024-09-20T22:56:03.759 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.759 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.759 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:56:03.759 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:47.286640+0000", 2024-09-20T22:56:03.759 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00019450899999999999 2024-09-20T22:56:03.759 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.759 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.759 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting_for_map not empty", 2024-09-20T22:56:03.760 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:47.286648+0000", 2024-09-20T22:56:03.760 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.8150000000000006e-06 2024-09-20T22:56:03.760 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.760 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.760 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:56:03.760 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:47.290185+0000", 2024-09-20T22:56:03.760 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0035371030000000002 2024-09-20T22:56:03.760 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.760 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.760 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for blocked object", 2024-09-20T22:56:03.760 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:47.290239+0000", 2024-09-20T22:56:03.760 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.3671000000000002e-05 2024-09-20T22:56:03.760 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.760 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.760 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:56:03.761 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:47.303495+0000", 2024-09-20T22:56:03.761 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.013256396 2024-09-20T22:56:03.761 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.761 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.761 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:56:03.761 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:47.303636+0000", 2024-09-20T22:56:03.761 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00014082600000000001 2024-09-20T22:56:03.761 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.761 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.761 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:56:03.761 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:47.303763+0000", 2024-09-20T22:56:03.761 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00012657599999999999 2024-09-20T22:56:03.761 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:56:03.761 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:56:03.761 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:56:03.762 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.762 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.762 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:6790 7.0 7:064f66ab:::smithi08477203-45:head [copy-from ver 1815] snapc 135=[135,134,133,132,131,130,12f,12e,12d,12c,12b,12a,128,127,126,124,122,121,11f,11b,119,112,10e,10c,107,103,fb,fa,f8,ed,d7,c2,bf] ondisk+write+known_if_redirected+supports_pool_eio e687)", 2024-09-20T22:56:03.762 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:52:47.698638+0000", 2024-09-20T22:56:03.762 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 195.96136401699999, 2024-09-20T22:56:03.762 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.029923735, 2024-09-20T22:56:03.762 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:56:03.762 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:56:03.762 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:56:03.762 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:56:03.762 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:56:03.762 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:56:03.762 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 6790 2024-09-20T22:56:03.762 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.763 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:56:03.763 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.763 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:56:03.763 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:47.698638+0000", 2024-09-20T22:56:03.763 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:56:03.763 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.763 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.763 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:56:03.763 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:47.698638+0000", 2024-09-20T22:56:03.763 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:56:03.763 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.763 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.763 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:56:03.763 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:47.698639+0000", 2024-09-20T22:56:03.763 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.5880000000000001e-06 2024-09-20T22:56:03.764 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.764 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.764 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:56:03.764 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:47.698647+0000", 2024-09-20T22:56:03.764 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.6040000000000004e-06 2024-09-20T22:56:03.764 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.764 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.764 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:56:03.764 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:47.698662+0000", 2024-09-20T22:56:03.764 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4997999999999999e-05 2024-09-20T22:56:03.764 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.764 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.764 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:56:03.764 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:47.698669+0000", 2024-09-20T22:56:03.765 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.0400000000000004e-06 2024-09-20T22:56:03.765 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.765 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.765 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:56:03.765 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:47.698744+0000", 2024-09-20T22:56:03.765 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.5018999999999994e-05 2024-09-20T22:56:03.765 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.765 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.765 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:56:03.765 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:47.698841+0000", 2024-09-20T22:56:03.765 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.7229e-05 2024-09-20T22:56:03.765 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.765 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.765 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 2,3", 2024-09-20T22:56:03.766 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:47.711667+0000", 2024-09-20T22:56:03.766 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.012825731999999999 2024-09-20T22:56:03.766 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.766 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.766 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:56:03.766 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:47.717229+0000", 2024-09-20T22:56:03.766 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0055624200000000002 2024-09-20T22:56:03.766 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.766 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.766 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:56:03.766 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:47.726665+0000", 2024-09-20T22:56:03.766 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0094358689999999995 2024-09-20T22:56:03.766 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.766 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.767 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:56:03.767 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:47.728457+0000", 2024-09-20T22:56:03.767 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0017914789999999999 2024-09-20T22:56:03.767 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.767 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.767 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:56:03.767 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:47.728511+0000", 2024-09-20T22:56:03.767 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.3953000000000003e-05 2024-09-20T22:56:03.767 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.767 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.767 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:56:03.767 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:47.728561+0000", 2024-09-20T22:56:03.767 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.0803999999999997e-05 2024-09-20T22:56:03.767 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:56:03.767 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:56:03.768 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:56:03.768 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.768 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.768 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:6791 7.0 7:064f66ab:::smithi08477203-45:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e687)", 2024-09-20T22:56:03.768 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:52:47.698687+0000", 2024-09-20T22:56:03.768 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 195.961314727, 2024-09-20T22:56:03.768 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.017803079999999999, 2024-09-20T22:56:03.768 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:56:03.768 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:56:03.768 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:56:03.768 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:56:03.768 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:56:03.768 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:56:03.768 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 6791 2024-09-20T22:56:03.769 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.769 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:56:03.769 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.769 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:56:03.769 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:47.698687+0000", 2024-09-20T22:56:03.769 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:56:03.769 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.769 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.769 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:56:03.769 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:47.698687+0000", 2024-09-20T22:56:03.769 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:56:03.769 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.769 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.769 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:56:03.769 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:47.698687+0000", 2024-09-20T22:56:03.770 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.2099999999999997e-07 2024-09-20T22:56:03.770 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.770 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.770 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:56:03.770 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:47.698690+0000", 2024-09-20T22:56:03.770 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.067e-06 2024-09-20T22:56:03.770 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.770 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.770 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:56:03.770 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:47.698702+0000", 2024-09-20T22:56:03.770 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1222e-05 2024-09-20T22:56:03.770 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.770 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.770 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:56:03.771 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:47.698708+0000", 2024-09-20T22:56:03.771 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.8479999999999997e-06 2024-09-20T22:56:03.771 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.771 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.771 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:56:03.771 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:47.699009+0000", 2024-09-20T22:56:03.771 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00030181000000000001 2024-09-20T22:56:03.771 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.771 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.771 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for blocked object", 2024-09-20T22:56:03.771 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:47.699075+0000", 2024-09-20T22:56:03.771 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.6021999999999999e-05 2024-09-20T22:56:03.771 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.771 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.771 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:56:03.772 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:47.716226+0000", 2024-09-20T22:56:03.772 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.017150657 2024-09-20T22:56:03.772 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.772 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.772 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:56:03.772 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:47.716352+0000", 2024-09-20T22:56:03.772 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00012609599999999999 2024-09-20T22:56:03.772 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.772 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.772 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:56:03.772 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:47.716490+0000", 2024-09-20T22:56:03.772 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000137837 2024-09-20T22:56:03.772 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:56:03.772 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:56:03.772 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:56:03.773 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.773 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.773 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:6816 7.b 7:dc95255e:::smithi08477203-1:head [copy-from ver 751] snapc 136=[136,135,134,133,132,131,130,12f,12e,12d,12c,12b,12a,128,127,126,124,122,121,11f,11b,119,112,10e,10c,107,103,fb,fa,f8,ed,d7,c2,bf] ondisk+write+known_if_redirected+supports_pool_eio e688)", 2024-09-20T22:56:03.773 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:52:48.282781+0000", 2024-09-20T22:56:03.773 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 195.377220421, 2024-09-20T22:56:03.773 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.036288688999999999, 2024-09-20T22:56:03.773 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:56:03.773 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:56:03.773 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:56:03.773 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:56:03.773 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:56:03.773 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:56:03.773 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 6816 2024-09-20T22:56:03.773 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.774 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:56:03.774 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.774 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:56:03.774 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.282781+0000", 2024-09-20T22:56:03.774 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:56:03.774 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.774 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.774 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:56:03.774 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.282781+0000", 2024-09-20T22:56:03.774 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:56:03.774 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.774 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.774 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:56:03.774 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.282783+0000", 2024-09-20T22:56:03.775 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.381e-06 2024-09-20T22:56:03.775 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.775 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.775 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:56:03.775 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.282789+0000", 2024-09-20T22:56:03.775 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.2210000000000002e-06 2024-09-20T22:56:03.775 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.775 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.775 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:56:03.775 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.282800+0000", 2024-09-20T22:56:03.775 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1581e-05 2024-09-20T22:56:03.775 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.775 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.775 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:56:03.775 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.282812+0000", 2024-09-20T22:56:03.776 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1797000000000001e-05 2024-09-20T22:56:03.776 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.776 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.776 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:56:03.776 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.282886+0000", 2024-09-20T22:56:03.776 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.4327999999999997e-05 2024-09-20T22:56:03.776 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.776 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.776 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op must wait for map", 2024-09-20T22:56:03.776 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.282896+0000", 2024-09-20T22:56:03.776 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.7410000000000004e-06 2024-09-20T22:56:03.776 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.776 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.776 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:56:03.776 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.285446+0000", 2024-09-20T22:56:03.777 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0025500319999999998 2024-09-20T22:56:03.777 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.777 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.777 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:56:03.777 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.285598+0000", 2024-09-20T22:56:03.777 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000151302 2024-09-20T22:56:03.777 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.777 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.777 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 2,6", 2024-09-20T22:56:03.777 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.300505+0000", 2024-09-20T22:56:03.777 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.014907204 2024-09-20T22:56:03.777 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.777 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.777 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:56:03.777 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.309549+0000", 2024-09-20T22:56:03.778 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0090438819999999996 2024-09-20T22:56:03.778 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.778 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.778 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:56:03.778 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.318842+0000", 2024-09-20T22:56:03.778 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0092930649999999997 2024-09-20T22:56:03.778 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.778 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.778 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:56:03.778 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.318947+0000", 2024-09-20T22:56:03.778 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00010507500000000001 2024-09-20T22:56:03.778 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.778 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.778 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:56:03.779 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.319009+0000", 2024-09-20T22:56:03.779 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.2607000000000006e-05 2024-09-20T22:56:03.779 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.779 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.779 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:56:03.779 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.319070+0000", 2024-09-20T22:56:03.779 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.0473000000000002e-05 2024-09-20T22:56:03.779 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:56:03.779 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:56:03.779 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:56:03.779 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.779 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.779 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:6817 7.b 7:dc95255e:::smithi08477203-1:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e688)", 2024-09-20T22:56:03.779 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:52:48.282837+0000", 2024-09-20T22:56:03.779 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 195.377164854, 2024-09-20T22:56:03.780 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.022790719000000001, 2024-09-20T22:56:03.780 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:56:03.780 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:56:03.780 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:56:03.780 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:56:03.780 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:56:03.780 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:56:03.780 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 6817 2024-09-20T22:56:03.780 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.780 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:56:03.780 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.780 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:56:03.780 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.282837+0000", 2024-09-20T22:56:03.780 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:56:03.781 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.781 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.781 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:56:03.781 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.282837+0000", 2024-09-20T22:56:03.781 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:56:03.781 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.781 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.781 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:56:03.781 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.282837+0000", 2024-09-20T22:56:03.781 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.0200000000000002e-07 2024-09-20T22:56:03.781 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.781 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.781 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:56:03.781 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.282840+0000", 2024-09-20T22:56:03.781 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.157e-06 2024-09-20T22:56:03.782 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.782 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.782 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:56:03.782 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.282847+0000", 2024-09-20T22:56:03.782 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.0159999999999997e-06 2024-09-20T22:56:03.782 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.782 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.782 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:56:03.782 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.282852+0000", 2024-09-20T22:56:03.782 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.6250000000000004e-06 2024-09-20T22:56:03.782 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.782 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.782 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:56:03.782 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.282955+0000", 2024-09-20T22:56:03.782 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000103056 2024-09-20T22:56:03.783 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.783 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.783 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting_for_map not empty", 2024-09-20T22:56:03.783 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.282963+0000", 2024-09-20T22:56:03.783 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.1650000000000006e-06 2024-09-20T22:56:03.783 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.783 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.783 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:56:03.783 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.285785+0000", 2024-09-20T22:56:03.783 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0028221259999999999 2024-09-20T22:56:03.783 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.783 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.783 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for blocked object", 2024-09-20T22:56:03.783 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.285873+0000", 2024-09-20T22:56:03.784 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.7231999999999994e-05 2024-09-20T22:56:03.784 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.784 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.784 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:56:03.784 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.305344+0000", 2024-09-20T22:56:03.784 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.019471366 2024-09-20T22:56:03.784 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.784 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.784 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:56:03.784 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.305484+0000", 2024-09-20T22:56:03.784 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00013981900000000001 2024-09-20T22:56:03.784 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.784 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.784 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:56:03.784 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.305627+0000", 2024-09-20T22:56:03.785 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00014355499999999999 2024-09-20T22:56:03.785 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:56:03.785 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:56:03.785 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:56:03.785 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.785 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.785 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:6842 7.0 7:0f524219:::smithi08477203-50:head [copy-from ver 1338] snapc 136=[136,135,134,132,131,130,12f,12e,12d,12c,12b,12a,128,127,126,124,122,121,11f,11b,119,112,10e,10c,107,103,fb,fa,f8,ed,d7,c2,bf] ondisk+write+known_if_redirected+supports_pool_eio e689)", 2024-09-20T22:56:03.785 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:52:49.294403+0000", 2024-09-20T22:56:03.785 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 194.365598285, 2024-09-20T22:56:03.785 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.020667376000000001, 2024-09-20T22:56:03.785 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:56:03.785 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:56:03.785 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:56:03.785 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:56:03.786 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:56:03.786 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:56:03.786 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 6842 2024-09-20T22:56:03.786 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.786 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:56:03.786 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.786 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:56:03.786 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:49.294403+0000", 2024-09-20T22:56:03.786 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:56:03.786 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.786 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.786 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:56:03.786 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:49.294403+0000", 2024-09-20T22:56:03.786 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:56:03.787 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.787 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.787 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:56:03.787 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:49.294406+0000", 2024-09-20T22:56:03.787 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.7199999999999998e-06 2024-09-20T22:56:03.787 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.787 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.787 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:56:03.787 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:49.294416+0000", 2024-09-20T22:56:03.787 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.7219999999999994e-06 2024-09-20T22:56:03.787 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.787 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.787 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:56:03.787 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:49.294437+0000", 2024-09-20T22:56:03.787 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.1514000000000001e-05 2024-09-20T22:56:03.788 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.788 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.788 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:56:03.788 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:49.294447+0000", 2024-09-20T22:56:03.788 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.6860000000000001e-06 2024-09-20T22:56:03.788 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.788 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.788 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:56:03.788 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:49.294546+0000", 2024-09-20T22:56:03.788 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.9258000000000004e-05 2024-09-20T22:56:03.788 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.788 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.788 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op must wait for map", 2024-09-20T22:56:03.788 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:49.294555+0000", 2024-09-20T22:56:03.788 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.7269999999999996e-06 2024-09-20T22:56:03.788 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.789 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.789 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:56:03.789 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:49.297887+0000", 2024-09-20T22:56:03.789 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0033321370000000002 2024-09-20T22:56:03.789 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.789 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.789 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:56:03.789 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:49.298017+0000", 2024-09-20T22:56:03.789 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00013003099999999999 2024-09-20T22:56:03.789 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.789 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.789 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 2,3", 2024-09-20T22:56:03.789 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:49.301096+0000", 2024-09-20T22:56:03.789 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0030791289999999999 2024-09-20T22:56:03.789 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.790 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.790 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:56:03.790 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:49.305218+0000", 2024-09-20T22:56:03.790 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0041216530000000003 2024-09-20T22:56:03.790 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.790 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.790 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:56:03.790 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:49.308776+0000", 2024-09-20T22:56:03.790 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0035582809999999999 2024-09-20T22:56:03.790 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.790 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.790 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:56:03.790 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:49.314968+0000", 2024-09-20T22:56:03.790 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.006191378 2024-09-20T22:56:03.790 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.791 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.791 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:56:03.791 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:49.315022+0000", 2024-09-20T22:56:03.791 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.4185999999999999e-05 2024-09-20T22:56:03.791 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.791 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.791 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:56:03.791 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:49.315071+0000", 2024-09-20T22:56:03.791 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.8953999999999999e-05 2024-09-20T22:56:03.791 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:56:03.791 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:56:03.791 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:56:03.791 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.791 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.792 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:7143 7.0 7:0f524219:::smithi08477203-50:head [copy-from ver 1826] snapc 145=[145,144,142,141,140,13f,13e,13d,13c,13b,13a,139,138,136,135,132,131,130,12f,12e,12d,12c,12a,128,127,124,122,121,11f,11b,112,103,fa,ed] ondisk+write+known_if_redirected+supports_pool_eio e719)", 2024-09-20T22:56:03.792 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:53:19.295040+0000", 2024-09-20T22:56:03.792 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 164.36496183, 2024-09-20T22:56:03.792 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.022201044999999999, 2024-09-20T22:56:03.792 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:56:03.792 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:56:03.792 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:56:03.792 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:56:03.792 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:56:03.792 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:56:03.792 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 7143 2024-09-20T22:56:03.792 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.792 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:56:03.792 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.792 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:56:03.793 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:19.295040+0000", 2024-09-20T22:56:03.793 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:56:03.793 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.793 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.793 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:56:03.793 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:19.295040+0000", 2024-09-20T22:56:03.793 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:56:03.793 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.793 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.793 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:56:03.793 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:19.295041+0000", 2024-09-20T22:56:03.793 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2950000000000001e-06 2024-09-20T22:56:03.793 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.793 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.794 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:56:03.794 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:19.295044+0000", 2024-09-20T22:56:03.794 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.433e-06 2024-09-20T22:56:03.794 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.794 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.794 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:56:03.794 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:19.295056+0000", 2024-09-20T22:56:03.794 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1317e-05 2024-09-20T22:56:03.794 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.794 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.794 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:56:03.794 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:19.295059+0000", 2024-09-20T22:56:03.794 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.6100000000000002e-06 2024-09-20T22:56:03.794 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.795 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.795 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:56:03.795 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:19.295124+0000", 2024-09-20T22:56:03.795 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.4338999999999995e-05 2024-09-20T22:56:03.795 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.795 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.795 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:56:03.795 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:19.295239+0000", 2024-09-20T22:56:03.795 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00011550899999999999 2024-09-20T22:56:03.795 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.795 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.795 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 2,3", 2024-09-20T22:56:03.795 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:19.300456+0000", 2024-09-20T22:56:03.795 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.005217078 2024-09-20T22:56:03.796 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.796 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.796 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:56:03.796 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:19.308370+0000", 2024-09-20T22:56:03.796 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0079131719999999996 2024-09-20T22:56:03.796 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.796 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.796 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:56:03.796 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:19.316928+0000", 2024-09-20T22:56:03.796 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0085582820000000004 2024-09-20T22:56:03.796 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.796 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.796 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:56:03.796 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:19.317133+0000", 2024-09-20T22:56:03.796 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000205064 2024-09-20T22:56:03.797 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.797 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.797 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:56:03.797 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:19.317188+0000", 2024-09-20T22:56:03.797 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.5492000000000003e-05 2024-09-20T22:56:03.797 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.797 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.797 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:56:03.797 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:19.317241+0000", 2024-09-20T22:56:03.797 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.2454000000000003e-05 2024-09-20T22:56:03.797 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:56:03.797 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:56:03.797 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:56:03.797 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.797 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.798 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:7260 7.0 7:064f66ab:::smithi08477203-45:head [write 440614~641741] snapc 149=[149,148,147,146,145,144,142,141,140,13e,13d,13c,13b,13a,139,138,136,135,132,130,12f,12e,12d,12c,12a,128,127,124,121,11f,112,103,ed] ondisk+write+known_if_redirected+supports_pool_eio e728)", 2024-09-20T22:56:03.798 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:53:28.321973+0000", 2024-09-20T22:56:03.798 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 155.338028874, 2024-09-20T22:56:03.798 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.020697989, 2024-09-20T22:56:03.798 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:56:03.798 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:56:03.798 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:56:03.798 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:56:03.798 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:56:03.798 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:56:03.798 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 7260 2024-09-20T22:56:03.798 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.798 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:56:03.798 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.799 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:56:03.799 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:28.321973+0000", 2024-09-20T22:56:03.799 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:56:03.799 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.799 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.799 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:56:03.799 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:28.321973+0000", 2024-09-20T22:56:03.799 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:56:03.799 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.799 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.799 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:56:03.799 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:28.321974+0000", 2024-09-20T22:56:03.799 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2699999999999999e-06 2024-09-20T22:56:03.799 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.799 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.800 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:56:03.800 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:28.322726+0000", 2024-09-20T22:56:03.800 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00075168700000000001 2024-09-20T22:56:03.800 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.800 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.800 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:56:03.800 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:28.322740+0000", 2024-09-20T22:56:03.800 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.453e-05 2024-09-20T22:56:03.800 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.800 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.800 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:56:03.800 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:28.322748+0000", 2024-09-20T22:56:03.800 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.6790000000000005e-06 2024-09-20T22:56:03.800 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.801 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.801 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:56:03.801 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:28.322810+0000", 2024-09-20T22:56:03.801 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.1878000000000006e-05 2024-09-20T22:56:03.801 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.801 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.801 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op must wait for map", 2024-09-20T22:56:03.801 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:28.322823+0000", 2024-09-20T22:56:03.801 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2841e-05 2024-09-20T22:56:03.801 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.801 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.801 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:56:03.801 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:28.327245+0000", 2024-09-20T22:56:03.801 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0044223400000000003 2024-09-20T22:56:03.801 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.802 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.802 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:56:03.802 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:28.327380+0000", 2024-09-20T22:56:03.802 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00013474599999999999 2024-09-20T22:56:03.802 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.802 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.802 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 2,3", 2024-09-20T22:56:03.802 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:28.327639+0000", 2024-09-20T22:56:03.802 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00025918600000000001 2024-09-20T22:56:03.802 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.802 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.802 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:56:03.802 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:28.329313+0000", 2024-09-20T22:56:03.802 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0016739750000000001 2024-09-20T22:56:03.803 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.803 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.803 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:56:03.803 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:28.340810+0000", 2024-09-20T22:56:03.803 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.011496719000000001 2024-09-20T22:56:03.803 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.803 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.803 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:56:03.803 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:28.342500+0000", 2024-09-20T22:56:03.803 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001689938 2024-09-20T22:56:03.803 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.803 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.803 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:56:03.803 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:28.342592+0000", 2024-09-20T22:56:03.803 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.2289999999999999e-05 2024-09-20T22:56:03.804 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.804 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.804 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:56:03.804 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:28.342671+0000", 2024-09-20T22:56:03.804 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.8910000000000002e-05 2024-09-20T22:56:03.804 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:56:03.804 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:56:03.804 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:56:03.804 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.804 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.804 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:7432 7.0 7:064f66ab:::smithi08477203-45:head [write 427175~721527,stat] snapc 14e=[14e,14c,14a,149,148,145,142,141,13e,13d,13c,13b,13a,139,132,130,12f,12e,12c,12a,128,127,124,121,11f,112,103] ondisk+write+known_if_redirected+supports_pool_eio e744)", 2024-09-20T22:56:03.804 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:53:44.426844+0000", 2024-09-20T22:56:03.804 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 139.23315759600001, 2024-09-20T22:56:03.804 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.051040752000000002, 2024-09-20T22:56:03.804 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:56:03.805 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:56:03.805 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:56:03.805 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:56:03.805 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:56:03.805 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:56:03.805 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 7432 2024-09-20T22:56:03.805 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.805 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:56:03.805 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.805 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:56:03.805 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:44.426844+0000", 2024-09-20T22:56:03.805 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:56:03.805 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.805 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.806 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:56:03.806 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:44.426844+0000", 2024-09-20T22:56:03.806 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:56:03.806 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.806 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.806 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:56:03.806 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:44.426846+0000", 2024-09-20T22:56:03.806 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.5379999999999999e-06 2024-09-20T22:56:03.806 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.806 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.806 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:56:03.806 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:44.427622+0000", 2024-09-20T22:56:03.806 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00077623299999999998 2024-09-20T22:56:03.806 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.807 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.807 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:56:03.807 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:44.427636+0000", 2024-09-20T22:56:03.807 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4606999999999999e-05 2024-09-20T22:56:03.807 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.807 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.807 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:56:03.807 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:44.427645+0000", 2024-09-20T22:56:03.807 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.6789999999999994e-06 2024-09-20T22:56:03.807 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.807 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.807 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:56:03.807 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:44.427707+0000", 2024-09-20T22:56:03.807 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.2383000000000002e-05 2024-09-20T22:56:03.807 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.808 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.808 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:56:03.808 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:44.427805+0000", 2024-09-20T22:56:03.808 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.7084999999999996e-05 2024-09-20T22:56:03.808 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.808 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.808 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 2,3", 2024-09-20T22:56:03.808 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:44.427962+0000", 2024-09-20T22:56:03.808 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00015762400000000001 2024-09-20T22:56:03.808 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.808 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.808 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:56:03.808 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:44.429626+0000", 2024-09-20T22:56:03.808 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00166421 2024-09-20T22:56:03.809 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.809 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.809 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:56:03.809 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:44.430495+0000", 2024-09-20T22:56:03.809 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00086868299999999998 2024-09-20T22:56:03.809 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.809 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.809 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:56:03.809 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:44.477712+0000", 2024-09-20T22:56:03.809 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.047216739000000001 2024-09-20T22:56:03.809 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.809 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.809 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:56:03.809 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:44.477805+0000", 2024-09-20T22:56:03.810 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.3424000000000005e-05 2024-09-20T22:56:03.810 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.810 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.810 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:56:03.810 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:44.477885+0000", 2024-09-20T22:56:03.810 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.9547000000000003e-05 2024-09-20T22:56:03.810 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:56:03.810 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:56:03.810 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:56:03.810 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.810 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.810 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:7433 7.0 7:064f66ab:::smithi08477203-45:head [write 1930749~735360] snapc 14e=[14e,14c,14a,149,148,145,142,141,13e,13d,13c,13b,13a,139,132,130,12f,12e,12c,12a,128,127,124,121,11f,112,103] ondisk+write+known_if_redirected+supports_pool_eio e744)", 2024-09-20T22:56:03.810 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:53:44.435104+0000", 2024-09-20T22:56:03.810 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 139.22489758099999, 2024-09-20T22:56:03.811 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.043021579999999997, 2024-09-20T22:56:03.811 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:56:03.811 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:56:03.811 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:56:03.811 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:56:03.811 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:56:03.811 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:56:03.811 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 7433 2024-09-20T22:56:03.811 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.811 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:56:03.811 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.811 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:56:03.811 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:44.435104+0000", 2024-09-20T22:56:03.811 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:56:03.811 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.812 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.812 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:56:03.812 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:44.435104+0000", 2024-09-20T22:56:03.812 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:56:03.812 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.812 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.812 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:56:03.812 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:44.435106+0000", 2024-09-20T22:56:03.812 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.5549999999999999e-06 2024-09-20T22:56:03.812 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.812 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.812 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:56:03.812 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:44.435886+0000", 2024-09-20T22:56:03.812 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00078014899999999995 2024-09-20T22:56:03.812 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.813 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.813 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:56:03.813 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:44.435900+0000", 2024-09-20T22:56:03.813 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4576000000000001e-05 2024-09-20T22:56:03.813 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.813 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.813 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:56:03.813 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:44.435908+0000", 2024-09-20T22:56:03.813 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.1380000000000007e-06 2024-09-20T22:56:03.813 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.813 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.813 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:56:03.813 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:44.435961+0000", 2024-09-20T22:56:03.813 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.2561e-05 2024-09-20T22:56:03.813 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.814 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.814 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:56:03.814 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:44.436043+0000", 2024-09-20T22:56:03.814 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.2213999999999996e-05 2024-09-20T22:56:03.814 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.814 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.814 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 2,3", 2024-09-20T22:56:03.814 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:44.436179+0000", 2024-09-20T22:56:03.814 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000135776 2024-09-20T22:56:03.814 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.814 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.814 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:56:03.814 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:44.438389+0000", 2024-09-20T22:56:03.814 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.002210203 2024-09-20T22:56:03.815 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.815 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.815 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:56:03.815 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:44.438925+0000", 2024-09-20T22:56:03.815 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00053610599999999995 2024-09-20T22:56:03.815 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.815 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.815 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:56:03.815 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:44.477956+0000", 2024-09-20T22:56:03.815 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.039030861 2024-09-20T22:56:03.815 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.815 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.815 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:56:03.815 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:44.478047+0000", 2024-09-20T22:56:03.815 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.0612999999999995e-05 2024-09-20T22:56:03.816 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.816 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.816 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:56:03.816 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:44.478126+0000", 2024-09-20T22:56:03.816 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.8827999999999998e-05 2024-09-20T22:56:03.816 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:56:03.816 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:56:03.816 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:56:03.816 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.816 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.816 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:7434 7.0 7:064f66ab:::smithi08477203-45:head [setxattr _header (58) in=7b,truncate 3227175] snapc 14e=[14e,14c,14a,149,148,145,142,141,13e,13d,13c,13b,13a,139,132,130,12f,12e,12c,12a,128,127,124,121,11f,112,103] ondisk+write+known_if_redirected+supports_pool_eio e744)", 2024-09-20T22:56:03.816 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:53:44.435927+0000", 2024-09-20T22:56:03.816 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 139.224074632, 2024-09-20T22:56:03.816 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.042348438000000002, 2024-09-20T22:56:03.817 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:56:03.817 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:56:03.817 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:56:03.817 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:56:03.817 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:56:03.817 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:56:03.817 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 7434 2024-09-20T22:56:03.817 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.817 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:56:03.817 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.817 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:56:03.817 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:44.435927+0000", 2024-09-20T22:56:03.817 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:56:03.817 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.817 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.818 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:56:03.818 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:44.435927+0000", 2024-09-20T22:56:03.818 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:56:03.818 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.818 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.818 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:56:03.818 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:44.435928+0000", 2024-09-20T22:56:03.818 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.5000000000000001e-07 2024-09-20T22:56:03.818 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.818 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.818 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:56:03.818 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:44.435933+0000", 2024-09-20T22:56:03.818 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.8790000000000002e-06 2024-09-20T22:56:03.818 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.819 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.819 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:56:03.819 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:44.435950+0000", 2024-09-20T22:56:03.819 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.7087000000000001e-05 2024-09-20T22:56:03.819 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.819 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.819 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:56:03.819 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:44.435957+0000", 2024-09-20T22:56:03.819 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.8399999999999997e-06 2024-09-20T22:56:03.819 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.819 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.819 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:56:03.819 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:44.437409+0000", 2024-09-20T22:56:03.819 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0014522350000000001 2024-09-20T22:56:03.820 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.820 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.820 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:56:03.820 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:44.437551+0000", 2024-09-20T22:56:03.820 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00014222700000000001 2024-09-20T22:56:03.820 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.820 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.820 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 2,3", 2024-09-20T22:56:03.820 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:44.437794+0000", 2024-09-20T22:56:03.820 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00024253499999999999 2024-09-20T22:56:03.820 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.820 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.820 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:56:03.820 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:44.438399+0000", 2024-09-20T22:56:03.820 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00060537799999999995 2024-09-20T22:56:03.821 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.821 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.821 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:56:03.821 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:44.439025+0000", 2024-09-20T22:56:03.821 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00062642000000000004 2024-09-20T22:56:03.821 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.821 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.821 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:56:03.821 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:44.478167+0000", 2024-09-20T22:56:03.821 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.039141148000000001 2024-09-20T22:56:03.821 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.821 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.821 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:56:03.821 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:44.478217+0000", 2024-09-20T22:56:03.821 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.0182000000000002e-05 2024-09-20T22:56:03.822 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.822 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.822 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:56:03.822 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:44.478275+0000", 2024-09-20T22:56:03.822 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.8656999999999998e-05 2024-09-20T22:56:03.822 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:56:03.822 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:56:03.822 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:56:03.822 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.822 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.822 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:7435 7.0 7:064f66ab:::smithi08477203-45:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e744)", 2024-09-20T22:56:03.822 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:53:44.435969+0000", 2024-09-20T22:56:03.822 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 139.22403291200001, 2024-09-20T22:56:03.823 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.042610406000000003, 2024-09-20T22:56:03.823 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:56:03.823 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:56:03.823 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:56:03.823 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:56:03.823 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:56:03.823 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:56:03.823 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 7435 2024-09-20T22:56:03.823 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.823 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:56:03.823 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.823 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:56:03.823 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:44.435969+0000", 2024-09-20T22:56:03.823 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:56:03.823 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.824 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.824 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:56:03.824 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:44.435969+0000", 2024-09-20T22:56:03.824 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:56:03.824 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.824 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.824 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:56:03.824 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:44.435969+0000", 2024-09-20T22:56:03.824 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.2799999999999996e-07 2024-09-20T22:56:03.824 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.824 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.824 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:56:03.824 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:44.435972+0000", 2024-09-20T22:56:03.824 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.0979999999999998e-06 2024-09-20T22:56:03.824 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.825 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.825 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:56:03.825 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:44.435981+0000", 2024-09-20T22:56:03.825 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.1300000000000001e-06 2024-09-20T22:56:03.825 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.825 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.825 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:56:03.825 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:44.435986+0000", 2024-09-20T22:56:03.825 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.9619999999999998e-06 2024-09-20T22:56:03.825 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.825 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.825 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:56:03.825 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:44.438239+0000", 2024-09-20T22:56:03.825 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0022527609999999998 2024-09-20T22:56:03.825 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.826 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.826 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for rw locks", 2024-09-20T22:56:03.826 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:44.438361+0000", 2024-09-20T22:56:03.826 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000121589 2024-09-20T22:56:03.826 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.826 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.826 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:56:03.826 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:44.478322+0000", 2024-09-20T22:56:03.826 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.03996077 2024-09-20T22:56:03.826 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.826 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.826 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:56:03.826 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:44.478438+0000", 2024-09-20T22:56:03.827 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00011638599999999999 2024-09-20T22:56:03.827 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.827 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.827 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:56:03.827 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:44.478579+0000", 2024-09-20T22:56:03.827 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000141082 2024-09-20T22:56:03.827 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:56:03.827 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:56:03.827 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:56:03.827 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.827 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.827 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:8664 7.a e873/94 7:5aaef466:::smithi08477203-40:head v 873'1111, mlcod=871'1107)", 2024-09-20T22:56:03.827 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:55:53.440649+0000", 2024-09-20T22:56:03.827 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 10.219352443, 2024-09-20T22:56:03.828 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0046148819999999998, 2024-09-20T22:56:03.828 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:56:03.828 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:56:03.828 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:56:03.828 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:56:03.828 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.828 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:56:03.828 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:53.440649+0000", 2024-09-20T22:56:03.828 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:56:03.828 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.828 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.828 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:56:03.828 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:53.440649+0000", 2024-09-20T22:56:03.828 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:56:03.828 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.829 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.829 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:56:03.829 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:53.440649+0000", 2024-09-20T22:56:03.829 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.1e-07 2024-09-20T22:56:03.829 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.829 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.829 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:56:03.829 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:53.441182+0000", 2024-09-20T22:56:03.829 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00053263799999999999 2024-09-20T22:56:03.829 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.829 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.829 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:56:03.829 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:53.441195+0000", 2024-09-20T22:56:03.829 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2649999999999999e-05 2024-09-20T22:56:03.829 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.830 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.830 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:56:03.830 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:53.441203+0000", 2024-09-20T22:56:03.830 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.1960000000000008e-06 2024-09-20T22:56:03.830 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.830 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.830 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:56:03.830 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:53.441272+0000", 2024-09-20T22:56:03.830 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.8689999999999995e-05 2024-09-20T22:56:03.830 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.830 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.830 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:56:03.830 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:53.441299+0000", 2024-09-20T22:56:03.830 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.7066000000000001e-05 2024-09-20T22:56:03.830 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.831 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.831 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:56:03.831 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:53.445230+0000", 2024-09-20T22:56:03.831 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0039313059999999999 2024-09-20T22:56:03.831 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:03.831 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:03.831 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:56:03.831 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:53.445264+0000", 2024-09-20T22:56:03.831 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.4125999999999998e-05 2024-09-20T22:56:03.831 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:56:03.831 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:56:03.831 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:56:03.831 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:56:03.831 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:56:03.831 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-20T22:56:03.832 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:56:03.833 INFO:tasks.rados.rados.0.smithi084.stdout:3527: seq_num 1204 ranges {635628=583570,1671967=563661} 2024-09-20T22:56:03.838 INFO:tasks.rados.rados.0.smithi084.stdout:3527: writing smithi08477203-11 from 635628 to 1219198 tid 1 2024-09-20T22:56:03.845 INFO:tasks.rados.rados.0.smithi084.stdout:3527: writing smithi08477203-11 from 1671967 to 2235628 tid 2 2024-09-20T22:56:03.845 INFO:tasks.rados.rados.0.smithi084.stdout:3525: done (1 left) 2024-09-20T22:56:03.845 INFO:tasks.rados.rados.0.smithi084.stdout:3528: snap_create 2024-09-20T22:56:03.845 INFO:tasks.rados.rados.0.smithi084.stdout:3527: finishing write tid 1 to smithi08477203-11 2024-09-20T22:56:03.848 INFO:tasks.rados.rados.0.smithi084.stdout:3527: finishing write tid 2 to smithi08477203-11 2024-09-20T22:56:03.848 INFO:tasks.rados.rados.0.smithi084.stdout:3527: finishing write tid 3 to smithi08477203-11 2024-09-20T22:56:03.848 INFO:tasks.rados.rados.0.smithi084.stdout:3527: finishing write tid 4 to smithi08477203-11 2024-09-20T22:56:03.848 INFO:tasks.rados.rados.0.smithi084.stdout:3527: oid 11 updating version 0 to 2647 2024-09-20T22:56:03.848 INFO:tasks.rados.rados.0.smithi084.stdout:3527: oid 11 updating version 2647 to 2649 2024-09-20T22:56:03.848 INFO:tasks.rados.rados.0.smithi084.stdout:3527: oid 11 version 2649 is already newer than 2648 2024-09-20T22:56:03.848 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 11 v 2649 (ObjNum 1204 snap 411 seq_num 1204) dirty exists 2024-09-20T22:56:03.848 INFO:tasks.rados.rados.0.smithi084.stdout:3527: left oid 11 (ObjNum 1204 snap 411 seq_num 1204) 2024-09-20T22:56:03.854 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:03.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:03.855 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:56:03.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:03 smithi084 ceph-mon[29774]: pgmap v1224: 158 pgs: 2 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 152 active+clean; 653 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 4.2 MiB/s rd, 2.4 MiB/s wr, 13 op/s 2024-09-20T22:56:03.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:03 smithi084 ceph-mon[29774]: osdmap e882: 8 total, 8 up, 8 in 2024-09-20T22:56:03.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:03 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/224383180' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:56:03.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:03 smithi084 ceph-mon[29774]: osdmap e883: 8 total, 8 up, 8 in 2024-09-20T22:56:03.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:03 smithi084 ceph-mon[40024]: pgmap v1224: 158 pgs: 2 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 152 active+clean; 653 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 4.2 MiB/s rd, 2.4 MiB/s wr, 13 op/s 2024-09-20T22:56:03.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:03 smithi084 ceph-mon[40024]: osdmap e882: 8 total, 8 up, 8 in 2024-09-20T22:56:03.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:03 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/224383180' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:56:03.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:03 smithi084 ceph-mon[40024]: osdmap e883: 8 total, 8 up, 8 in 2024-09-20T22:56:03.956 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:56:03.977 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:03.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:03.978 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:56:03.992 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-20T22:56:04.079 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:56:04.100 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:04.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:04.101 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:56:04.202 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:56:04.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:04.223 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:04.224 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:56:04.238 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:56:04.324 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:56:04.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:04.347 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:04.347 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:56:04.448 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:56:04.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:04.469 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:04.470 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:56:04.479 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-20T22:56:04.480 INFO:tasks.rados.rados.0.smithi084.stdout:3527: done (1 left) 2024-09-20T22:56:04.480 INFO:tasks.rados.rados.0.smithi084.stdout:3528: done (0 left) 2024-09-20T22:56:04.480 INFO:tasks.rados.rados.0.smithi084.stdout:3529: read oid 42 snap 371 2024-09-20T22:56:04.480 INFO:tasks.rados.rados.0.smithi084.stdout:3529: expect (ObjNum 625 snap 186 seq_num 625) 2024-09-20T22:56:04.480 INFO:tasks.rados.rados.0.smithi084.stdout:3530: write (excl) oid 48 current snap is 412 2024-09-20T22:56:04.480 INFO:tasks.rados.rados.0.smithi084.stdout:3530: seq_num 1205 ranges {652435=658819,1730449=521986} 2024-09-20T22:56:04.486 INFO:tasks.rados.rados.0.smithi084.stdout:3530: writing smithi08477203-48 from 652435 to 1311254 tid 1 2024-09-20T22:56:04.492 INFO:tasks.rados.rados.0.smithi084.stdout:3530: writing smithi08477203-48 from 1730449 to 2252435 tid 2 2024-09-20T22:56:04.492 INFO:tasks.rados.rados.0.smithi084.stdout:3531: write (excl) oid 8 current snap is 412 2024-09-20T22:56:04.492 INFO:tasks.rados.rados.0.smithi084.stdout:3531: seq_num 1206 ranges {669242=734068,2188931=80311} 2024-09-20T22:56:04.499 INFO:tasks.rados.rados.0.smithi084.stdout:3531: writing smithi08477203-8 from 669242 to 1403310 tid 1 2024-09-20T22:56:04.505 INFO:tasks.rados.rados.0.smithi084.stdout:3531: writing smithi08477203-8 from 2188931 to 2269242 tid 2 2024-09-20T22:56:04.505 INFO:tasks.rados.rados.0.smithi084.stdout:3532: copy_from oid 25 from oid 33 current snap is 412 2024-09-20T22:56:04.505 INFO:tasks.rados.rados.0.smithi084.stdout:3530: finishing write tid 1 to smithi08477203-48 2024-09-20T22:56:04.505 INFO:tasks.rados.rados.0.smithi084.stdout:3529: expect (ObjNum 625 snap 186 seq_num 625) 2024-09-20T22:56:04.570 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:56:04.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:04.593 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:04.593 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:56:04.627 INFO:tasks.rados.rados.0.smithi084.stdout:3530: finishing write tid 2 to smithi08477203-48 2024-09-20T22:56:04.627 INFO:tasks.rados.rados.0.smithi084.stdout:3530: finishing write tid 3 to smithi08477203-48 2024-09-20T22:56:04.628 INFO:tasks.rados.rados.0.smithi084.stdout:3530: finishing write tid 4 to smithi08477203-48 2024-09-20T22:56:04.628 INFO:tasks.rados.rados.0.smithi084.stdout:3530: oid 48 updating version 0 to 1943 2024-09-20T22:56:04.628 INFO:tasks.rados.rados.0.smithi084.stdout:3530: oid 48 version 1943 is already newer than 1941 2024-09-20T22:56:04.628 INFO:tasks.rados.rados.0.smithi084.stdout:3530: oid 48 version 1943 is already newer than 1942 2024-09-20T22:56:04.628 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 48 v 1943 (ObjNum 1205 snap 412 seq_num 1205) dirty exists 2024-09-20T22:56:04.628 INFO:tasks.rados.rados.0.smithi084.stdout:3530: left oid 48 (ObjNum 1205 snap 412 seq_num 1205) 2024-09-20T22:56:04.628 INFO:tasks.rados.rados.0.smithi084.stdout:3531: finishing write tid 1 to smithi08477203-8 2024-09-20T22:56:04.628 INFO:tasks.rados.rados.0.smithi084.stdout:3529: done (3 left) 2024-09-20T22:56:04.628 INFO:tasks.rados.rados.0.smithi084.stdout:3530: done (2 left) 2024-09-20T22:56:04.628 INFO:tasks.rados.rados.0.smithi084.stdout:3533: snap_remove snap 339 2024-09-20T22:56:04.694 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:56:04.715 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:04.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:04.716 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:56:04.729 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:56:04.817 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:56:04.838 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:04.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:04.839 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:56:04.940 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:56:04.961 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:04.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:04.962 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:56:05.063 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:56:05.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:05.085 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:05.085 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:56:05.186 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:56:05.194 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-20T22:56:05.194 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [], 2024-09-20T22:56:05.195 INFO:teuthology.orchestra.run.smithi154.stdout: "num_ops": 0 2024-09-20T22:56:05.195 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-20T22:56:05.209 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:05.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:05.209 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:56:05.310 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:56:05.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:05.332 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:05.332 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:56:05.433 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:56:05.449 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-20T22:56:05.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:05.456 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:05.456 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:56:05.479 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is set 2024-09-20T22:56:05.487 INFO:tasks.rados.rados.0.smithi084.stdout:3533: done (2 left) 2024-09-20T22:56:05.487 INFO:tasks.rados.rados.0.smithi084.stdout:3534: write (excl) oid 19 current snap is 412 2024-09-20T22:56:05.487 INFO:tasks.rados.rados.0.smithi084.stdout:3531: finishing write tid 2 to smithi08477203-8 2024-09-20T22:56:05.487 INFO:tasks.rados.rados.0.smithi084.stdout:3531: finishing write tid 3 to smithi08477203-8 2024-09-20T22:56:05.487 INFO:tasks.rados.rados.0.smithi084.stdout:3531: finishing write tid 4 to smithi08477203-8 2024-09-20T22:56:05.487 INFO:tasks.rados.rados.0.smithi084.stdout:3531: oid 8 updating version 0 to 2590 2024-09-20T22:56:05.487 INFO:tasks.rados.rados.0.smithi084.stdout:3531: oid 8 updating version 2590 to 2591 2024-09-20T22:56:05.488 INFO:tasks.rados.rados.0.smithi084.stdout:3531: oid 8 updating version 2591 to 2592 2024-09-20T22:56:05.488 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 8 v 2592 (ObjNum 1206 snap 412 seq_num 1206) dirty exists 2024-09-20T22:56:05.488 INFO:tasks.rados.rados.0.smithi084.stdout:3531: left oid 8 (ObjNum 1206 snap 412 seq_num 1206) 2024-09-20T22:56:05.488 INFO:tasks.rados.rados.0.smithi084.stdout:3532: finishing copy_from racing read to smithi08477203-25 2024-09-20T22:56:05.488 INFO:tasks.rados.rados.0.smithi084.stdout:3532: finishing copy_from to smithi08477203-25 2024-09-20T22:56:05.488 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 25 v 2739 (ObjNum 1201 snap 411 seq_num 1201) dirty exists 2024-09-20T22:56:05.488 INFO:tasks.rados.rados.0.smithi084.stdout:3534: seq_num 1207 ranges {686049=409317,1847413=438636} 2024-09-20T22:56:05.493 INFO:tasks.rados.rados.0.smithi084.stdout:3534: writing smithi08477203-19 from 686049 to 1095366 tid 1 2024-09-20T22:56:05.501 INFO:tasks.rados.rados.0.smithi084.stdout:3534: writing smithi08477203-19 from 1847413 to 2286049 tid 2 2024-09-20T22:56:05.502 INFO:tasks.rados.rados.0.smithi084.stdout:3531: done (2 left) 2024-09-20T22:56:05.502 INFO:tasks.rados.rados.0.smithi084.stdout:3532: done (1 left) 2024-09-20T22:56:05.502 INFO:tasks.rados.rados.0.smithi084.stdout:3535: copy_from oid 16 from oid 44 current snap is 412 2024-09-20T22:56:05.502 INFO:tasks.rados.rados.0.smithi084.stdout:3536: copy_from oid 42 from oid 48 current snap is 412 2024-09-20T22:56:05.502 INFO:tasks.rados.rados.0.smithi084.stdout:3537: read oid 31 snap -1 2024-09-20T22:56:05.502 INFO:tasks.rados.rados.0.smithi084.stdout:3537: expect (ObjNum 1143 snap 389 seq_num 1143) 2024-09-20T22:56:05.503 INFO:tasks.rados.rados.0.smithi084.stdout:3538: delete oid 49 current snap is 412 2024-09-20T22:56:05.505 INFO:tasks.rados.rados.0.smithi084.stdout:3538: done (4 left) 2024-09-20T22:56:05.505 INFO:tasks.rados.rados.0.smithi084.stdout:3539: write (excl) oid 18 current snap is 412 2024-09-20T22:56:05.505 INFO:tasks.rados.rados.0.smithi084.stdout:3539: seq_num 1209 ranges {719663=676168,1964377=355286} 2024-09-20T22:56:05.512 INFO:tasks.rados.rados.0.smithi084.stdout:3539: writing smithi08477203-18 from 719663 to 1395831 tid 1 2024-09-20T22:56:05.518 INFO:tasks.rados.rados.0.smithi084.stdout:3539: writing smithi08477203-18 from 1964377 to 2319663 tid 2 2024-09-20T22:56:05.518 INFO:tasks.rados.rados.0.smithi084.stdout:3540: snap_remove snap 346 2024-09-20T22:56:05.557 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:56:05.579 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:05.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:05.580 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:56:05.681 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:56:05.703 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:05.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:05.704 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:56:05.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:05 smithi154 ceph-mon[33264]: pgmap v1227: 158 pgs: 2 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 152 active+clean; 669 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 28 MiB/s rd, 3.0 MiB/s wr, 36 op/s 2024-09-20T22:56:05.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:05 smithi154 ceph-mon[33264]: osdmap e884: 8 total, 8 up, 8 in 2024-09-20T22:56:05.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:05 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/3524574389' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:56:05.790 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:05 smithi084 ceph-mon[29774]: pgmap v1227: 158 pgs: 2 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 152 active+clean; 669 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 28 MiB/s rd, 3.0 MiB/s wr, 36 op/s 2024-09-20T22:56:05.790 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:05 smithi084 ceph-mon[29774]: osdmap e884: 8 total, 8 up, 8 in 2024-09-20T22:56:05.790 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:05 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/3524574389' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:56:05.790 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:05 smithi084 ceph-mon[40024]: pgmap v1227: 158 pgs: 2 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 152 active+clean; 669 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 28 MiB/s rd, 3.0 MiB/s wr, 36 op/s 2024-09-20T22:56:05.790 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:05 smithi084 ceph-mon[40024]: osdmap e884: 8 total, 8 up, 8 in 2024-09-20T22:56:05.790 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:05 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/3524574389' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:56:05.805 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:56:05.826 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:05.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:05.827 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:56:05.903 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.6 dump_blocked_ops 2024-09-20T22:56:05.928 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:56:05.970 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:05.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:05.971 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:56:06.072 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:56:06.091 INFO:tasks.thrashosds.thrasher:in_osds: [2, 3, 5, 6, 7, 0, 4, 1] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-20T22:56:06.091 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-20T22:56:06.091 INFO:tasks.thrashosds.thrasher:Removing osd 7, in_osds are: [2, 3, 5, 6, 7, 0, 4, 1] 2024-09-20T22:56:06.091 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd out 7 2024-09-20T22:56:06.094 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:06.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:06.095 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:56:06.164 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:56:06.196 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:56:06.218 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:06.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:06.219 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:56:06.320 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:56:06.342 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:06.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:06.343 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:56:06.392 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:56:06.399 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:56:06.443 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:56:06.465 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:06.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:06.466 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:56:06.489 INFO:tasks.rados.rados.0.smithi084.stdout:3540: done (5 left) 2024-09-20T22:56:06.489 INFO:tasks.rados.rados.0.smithi084.stdout:3541: read oid 29 snap -1 2024-09-20T22:56:06.490 INFO:tasks.rados.rados.0.smithi084.stdout:3541: expect (ObjNum 1169 snap 403 seq_num 1169) 2024-09-20T22:56:06.490 INFO:tasks.rados.rados.0.smithi084.stdout:3542: rollback oid 34 current snap is 412 2024-09-20T22:56:06.490 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 34 to 368 2024-09-20T22:56:06.490 INFO:tasks.rados.rados.0.smithi084.stdout:3543: read oid 6 snap -1 2024-09-20T22:56:06.490 INFO:tasks.rados.rados.0.smithi084.stdout:3543: expect deleted 2024-09-20T22:56:06.490 INFO:tasks.rados.rados.0.smithi084.stdout:3544: write oid 27 current snap is 412 2024-09-20T22:56:06.490 INFO:tasks.rados.rados.0.smithi084.stdout:3544: seq_num 1210 ranges {736470=751417,2022859=313611} 2024-09-20T22:56:06.498 INFO:tasks.rados.rados.0.smithi084.stdout:3544: writing smithi08477203-27 from 736470 to 1487887 tid 1 2024-09-20T22:56:06.504 INFO:tasks.rados.rados.0.smithi084.stdout:3544: writing smithi08477203-27 from 2022859 to 2336470 tid 2 2024-09-20T22:56:06.505 INFO:tasks.rados.rados.0.smithi084.stdout:3545: write oid 1 current snap is 412 2024-09-20T22:56:06.505 INFO:tasks.rados.rados.0.smithi084.stdout:3545: seq_num 1211 ranges {753277=426666,1681341=545753} 2024-09-20T22:56:06.512 INFO:tasks.rados.rados.0.smithi084.stdout:3545: writing smithi08477203-1 from 753277 to 1179943 tid 1 2024-09-20T22:56:06.518 INFO:tasks.rados.rados.0.smithi084.stdout:3545: writing smithi08477203-1 from 1681341 to 2227094 tid 2 2024-09-20T22:56:06.518 INFO:tasks.rados.rados.0.smithi084.stdout:3546: snap_remove snap 403 2024-09-20T22:56:06.536 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:06 smithi154 ceph-mon[33264]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:56:06.537 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:06 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/3524574389' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T22:56:06.537 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:06 smithi154 ceph-mon[33264]: osdmap e885: 8 total, 8 up, 8 in 2024-09-20T22:56:06.537 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:06 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:56:06.566 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:56:06.587 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:06.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:06.588 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:56:06.641 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-20T22:56:06.641 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [], 2024-09-20T22:56:06.641 INFO:teuthology.orchestra.run.smithi154.stdout: "complaint_time": 30, 2024-09-20T22:56:06.641 INFO:teuthology.orchestra.run.smithi154.stdout: "num_blocked_ops": 0 2024-09-20T22:56:06.641 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-20T22:56:06.689 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:56:06.709 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:06.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:06.710 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:56:06.787 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:06 smithi084 ceph-mon[40024]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:56:06.787 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:06 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/3524574389' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T22:56:06.787 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:06 smithi084 ceph-mon[40024]: osdmap e885: 8 total, 8 up, 8 in 2024-09-20T22:56:06.787 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:06 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:56:06.787 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:06 smithi084 ceph-mon[29774]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:56:06.787 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:06 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/3524574389' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T22:56:06.788 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:06 smithi084 ceph-mon[29774]: osdmap e885: 8 total, 8 up, 8 in 2024-09-20T22:56:06.788 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:06 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:56:06.811 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:56:06.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:06.834 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:06.835 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:56:06.935 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:56:06.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:06.993 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:06.993 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:56:07.094 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:56:07.115 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:07.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:07.116 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:56:07.217 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:56:07.238 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:07.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:07.239 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:56:07.340 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:56:07.362 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:07.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:07.363 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:56:07.464 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:56:07.470 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:56:07.470 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:56:07.470 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:56:07.470 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:56:07.480 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:56:07.480 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:56:07.486 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:07.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:07.487 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:56:07.497 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.6 dump_historic_ops 2024-09-20T22:56:07.497 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-20T22:56:07.497 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T22:56:07.501 INFO:teuthology.orchestra.run.smithi084.stderr:marked out osd.7. 2024-09-20T22:56:07.519 INFO:tasks.rados.rados.0.smithi084.stdout:3546: done (10 left) 2024-09-20T22:56:07.520 INFO:tasks.rados.rados.0.smithi084.stdout:3547: write oid 12 current snap is 412 2024-09-20T22:56:07.520 INFO:tasks.rados.rados.0.smithi084.stdout:3547: seq_num 1212 ranges {770084=501915,2023470=346614} 2024-09-20T22:56:07.521 INFO:tasks.rados.rados.0.smithi084.stdout:3547: writing smithi08477203-12 from 770084 to 1271999 tid 1 2024-09-20T22:56:07.530 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:56:07.530 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:56:07.530 INFO:tasks.rados.rados.0.smithi084.stdout:3547: writing smithi08477203-12 from 2023470 to 2370084 tid 2 2024-09-20T22:56:07.533 INFO:tasks.rados.rados.0.smithi084.stdout:3534: finishing write tid 1 to smithi08477203-19 2024-09-20T22:56:07.533 INFO:tasks.rados.rados.0.smithi084.stdout:3535: finishing copy_from racing read to smithi08477203-16 2024-09-20T22:56:07.533 INFO:tasks.rados.rados.0.smithi084.stdout:3539: finishing write tid 1 to smithi08477203-18 2024-09-20T22:56:07.533 INFO:tasks.rados.rados.0.smithi084.stdout:3537: expect (ObjNum 1143 snap 389 seq_num 1143) 2024-09-20T22:56:07.565 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:56:07.565 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:56:07.588 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:56:07.591 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:56:07.592 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:56:07.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:07.610 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:07.610 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:56:07.619 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:56:07.619 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:56:07.711 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:56:07.732 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:07.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:07.733 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:56:07.734 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:07 smithi084 ceph-mon[40024]: pgmap v1230: 158 pgs: 2 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 152 active+clean; 669 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 28 MiB/s rd, 3.0 MiB/s wr, 36 op/s 2024-09-20T22:56:07.734 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:07 smithi084 ceph-mon[40024]: osdmap e886: 8 total, 8 up, 8 in 2024-09-20T22:56:07.735 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:07 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:56:07.735 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:07 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:56:07.735 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:07 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:56:07.735 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:07 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/4236344169' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["7"]}]: dispatch 2024-09-20T22:56:07.735 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:07 smithi084 ceph-mon[40024]: Client client.admin marked osd.7 out, while it was still marked up 2024-09-20T22:56:07.735 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:07 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/3222419873' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:56:07.735 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:07 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mon-a[29770]: 2024-09-20T22:56:07.391+0000 7f26ad5e8640 -1 mon.a@0(leader).osd e886 definitely_dead 0 2024-09-20T22:56:07.735 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:07 smithi084 ceph-mon[29774]: pgmap v1230: 158 pgs: 2 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 152 active+clean; 669 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 28 MiB/s rd, 3.0 MiB/s wr, 36 op/s 2024-09-20T22:56:07.735 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:07 smithi084 ceph-mon[29774]: osdmap e886: 8 total, 8 up, 8 in 2024-09-20T22:56:07.735 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:07 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:56:07.735 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:07 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:56:07.735 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:07 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:56:07.736 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:07 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/4236344169' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["7"]}]: dispatch 2024-09-20T22:56:07.736 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:07 smithi084 ceph-mon[29774]: Client client.admin marked osd.7 out, while it was still marked up 2024-09-20T22:56:07.736 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:07 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/3222419873' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:56:07.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:07 smithi154 ceph-mon[33264]: pgmap v1230: 158 pgs: 2 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 152 active+clean; 669 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 28 MiB/s rd, 3.0 MiB/s wr, 36 op/s 2024-09-20T22:56:07.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:07 smithi154 ceph-mon[33264]: osdmap e886: 8 total, 8 up, 8 in 2024-09-20T22:56:07.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:07 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:56:07.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:07 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:56:07.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:07 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:56:07.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:07 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/4236344169' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["7"]}]: dispatch 2024-09-20T22:56:07.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:07 smithi154 ceph-mon[33264]: Client client.admin marked osd.7 out, while it was still marked up 2024-09-20T22:56:07.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:07 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/3222419873' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:56:07.761 INFO:tasks.rados.rados.0.smithi084.stdout:3536: finishing copy_from racing read to smithi08477203-42 2024-09-20T22:56:07.761 INFO:tasks.rados.rados.0.smithi084.stdout:3534: finishing write tid 2 to smithi08477203-19 2024-09-20T22:56:07.761 INFO:tasks.rados.rados.0.smithi084.stdout:3534: finishing write tid 3 to smithi08477203-19 2024-09-20T22:56:07.762 INFO:tasks.rados.rados.0.smithi084.stdout:3534: finishing write tid 4 to smithi08477203-19 2024-09-20T22:56:07.762 INFO:tasks.rados.rados.0.smithi084.stdout:3534: oid 19 updating version 0 to 2849 2024-09-20T22:56:07.762 INFO:tasks.rados.rados.0.smithi084.stdout:3534: oid 19 updating version 2849 to 2855 2024-09-20T22:56:07.762 INFO:tasks.rados.rados.0.smithi084.stdout:3534: oid 19 version 2855 is already newer than 2854 2024-09-20T22:56:07.762 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 19 v 2855 (ObjNum 1207 snap 412 seq_num 1207) dirty exists 2024-09-20T22:56:07.762 INFO:tasks.rados.rados.0.smithi084.stdout:3534: left oid 19 (ObjNum 1207 snap 412 seq_num 1207) 2024-09-20T22:56:07.762 INFO:tasks.rados.rados.0.smithi084.stdout:3536: finishing copy_from to smithi08477203-42 2024-09-20T22:56:07.762 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 42 v 1943 (ObjNum 1205 snap 412 seq_num 1205) dirty exists 2024-09-20T22:56:07.762 INFO:tasks.rados.rados.0.smithi084.stdout:3539: finishing write tid 2 to smithi08477203-18 2024-09-20T22:56:07.762 INFO:tasks.rados.rados.0.smithi084.stdout:3539: finishing write tid 3 to smithi08477203-18 2024-09-20T22:56:07.762 INFO:tasks.rados.rados.0.smithi084.stdout:3535: finishing copy_from to smithi08477203-16 2024-09-20T22:56:07.762 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 16 v 2842 (ObjNum 1202 snap 411 seq_num 1202) dirty exists 2024-09-20T22:56:07.762 INFO:tasks.rados.rados.0.smithi084.stdout:3539: finishing write tid 4 to smithi08477203-18 2024-09-20T22:56:07.763 INFO:tasks.rados.rados.0.smithi084.stdout:3539: oid 18 updating version 0 to 2652 2024-09-20T22:56:07.763 INFO:tasks.rados.rados.0.smithi084.stdout:3539: oid 18 version 2652 is already newer than 2651 2024-09-20T22:56:07.763 INFO:tasks.rados.rados.0.smithi084.stdout:3539: oid 18 version 2652 is already newer than 2650 2024-09-20T22:56:07.763 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 18 v 2652 (ObjNum 1209 snap 412 seq_num 1209) dirty exists 2024-09-20T22:56:07.763 INFO:tasks.rados.rados.0.smithi084.stdout:3539: left oid 18 (ObjNum 1209 snap 412 seq_num 1209) 2024-09-20T22:56:07.763 INFO:tasks.rados.rados.0.smithi084.stdout:3534: done (10 left) 2024-09-20T22:56:07.763 INFO:tasks.rados.rados.0.smithi084.stdout:3535: done (9 left) 2024-09-20T22:56:07.763 INFO:tasks.rados.rados.0.smithi084.stdout:3536: done (8 left) 2024-09-20T22:56:07.763 INFO:tasks.rados.rados.0.smithi084.stdout:3537: done (7 left) 2024-09-20T22:56:07.763 INFO:tasks.rados.rados.0.smithi084.stdout:3539: done (6 left) 2024-09-20T22:56:07.763 INFO:tasks.rados.rados.0.smithi084.stdout:3548: delete oid 46 current snap is 412 2024-09-20T22:56:07.763 INFO:tasks.rados.rados.0.smithi084.stdout:3542: finishing rollback tid 0 to smithi08477203-34 2024-09-20T22:56:07.763 INFO:tasks.rados.rados.0.smithi084.stdout:3542: finishing rollback tid 1 to smithi08477203-34 2024-09-20T22:56:07.763 INFO:tasks.rados.rados.0.smithi084.stdout:3542: finishing rollback tid 2 to smithi08477203-34 2024-09-20T22:56:07.764 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 34 v 2589 (ObjNum 1066 snap 353 seq_num 1066) dirty exists 2024-09-20T22:56:07.764 INFO:tasks.rados.rados.0.smithi084.stdout:3544: finishing write tid 1 to smithi08477203-27 2024-09-20T22:56:07.764 INFO:tasks.rados.rados.0.smithi084.stdout:3544: finishing write tid 2 to smithi08477203-27 2024-09-20T22:56:07.764 INFO:tasks.rados.rados.0.smithi084.stdout:3544: finishing write tid 3 to smithi08477203-27 2024-09-20T22:56:07.764 INFO:tasks.rados.rados.0.smithi084.stdout:3544: finishing write tid 4 to smithi08477203-27 2024-09-20T22:56:07.764 INFO:tasks.rados.rados.0.smithi084.stdout:3544: oid 27 updating version 0 to 2494 2024-09-20T22:56:07.764 INFO:tasks.rados.rados.0.smithi084.stdout:3544: oid 27 version 2494 is already newer than 2493 2024-09-20T22:56:07.764 INFO:tasks.rados.rados.0.smithi084.stdout:3544: oid 27 version 2494 is already newer than 2492 2024-09-20T22:56:07.764 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 27 v 2494 (ObjNum 1210 snap 412 seq_num 1210) dirty exists 2024-09-20T22:56:07.764 INFO:tasks.rados.rados.0.smithi084.stdout:3544: left oid 27 (ObjNum 1210 snap 412 seq_num 1210) 2024-09-20T22:56:07.764 INFO:tasks.rados.rados.0.smithi084.stdout:3541: expect (ObjNum 1169 snap 403 seq_num 1169) 2024-09-20T22:56:07.783 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:56:07.833 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:56:07.854 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:07.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:07.855 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:56:07.956 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:56:07.976 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:07.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:07.977 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:56:08.062 INFO:tasks.rados.rados.0.smithi084.stdout:3545: finishing write tid 1 to smithi08477203-1 2024-09-20T22:56:08.062 INFO:tasks.rados.rados.0.smithi084.stdout:3545: finishing write tid 2 to smithi08477203-1 2024-09-20T22:56:08.062 INFO:tasks.rados.rados.0.smithi084.stdout:3545: finishing write tid 3 to smithi08477203-1 2024-09-20T22:56:08.062 INFO:tasks.rados.rados.0.smithi084.stdout:3545: finishing write tid 4 to smithi08477203-1 2024-09-20T22:56:08.062 INFO:tasks.rados.rados.0.smithi084.stdout:3545: oid 1 updating version 0 to 2655 2024-09-20T22:56:08.062 INFO:tasks.rados.rados.0.smithi084.stdout:3545: oid 1 version 2655 is already newer than 2654 2024-09-20T22:56:08.062 INFO:tasks.rados.rados.0.smithi084.stdout:3545: oid 1 version 2655 is already newer than 2653 2024-09-20T22:56:08.062 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 1 v 2655 (ObjNum 1211 snap 412 seq_num 1211) dirty exists 2024-09-20T22:56:08.062 INFO:tasks.rados.rados.0.smithi084.stdout:3545: left oid 1 (ObjNum 1211 snap 412 seq_num 1211) 2024-09-20T22:56:08.062 INFO:tasks.rados.rados.0.smithi084.stdout:3541: done (6 left) 2024-09-20T22:56:08.062 INFO:tasks.rados.rados.0.smithi084.stdout:3542: done (5 left) 2024-09-20T22:56:08.063 INFO:tasks.rados.rados.0.smithi084.stdout:3543: done (4 left) 2024-09-20T22:56:08.063 INFO:tasks.rados.rados.0.smithi084.stdout:3544: done (3 left) 2024-09-20T22:56:08.063 INFO:tasks.rados.rados.0.smithi084.stdout:3545: done (2 left) 2024-09-20T22:56:08.063 INFO:tasks.rados.rados.0.smithi084.stdout:3548: done (1 left) 2024-09-20T22:56:08.063 INFO:tasks.rados.rados.0.smithi084.stdout:3549: snap_create 2024-09-20T22:56:08.063 INFO:tasks.rados.rados.0.smithi084.stdout:3547: finishing write tid 1 to smithi08477203-12 2024-09-20T22:56:08.063 INFO:tasks.rados.rados.0.smithi084.stdout:3547: finishing write tid 2 to smithi08477203-12 2024-09-20T22:56:08.063 INFO:tasks.rados.rados.0.smithi084.stdout:3547: finishing write tid 3 to smithi08477203-12 2024-09-20T22:56:08.063 INFO:tasks.rados.rados.0.smithi084.stdout:3547: finishing write tid 4 to smithi08477203-12 2024-09-20T22:56:08.063 INFO:tasks.rados.rados.0.smithi084.stdout:3547: oid 12 updating version 0 to 2493 2024-09-20T22:56:08.063 INFO:tasks.rados.rados.0.smithi084.stdout:3547: oid 12 version 2493 is already newer than 2492 2024-09-20T22:56:08.063 INFO:tasks.rados.rados.0.smithi084.stdout:3547: oid 12 version 2493 is already newer than 2491 2024-09-20T22:56:08.063 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 12 v 2493 (ObjNum 1212 snap 412 seq_num 1212) dirty exists 2024-09-20T22:56:08.063 INFO:tasks.rados.rados.0.smithi084.stdout:3547: left oid 12 (ObjNum 1212 snap 412 seq_num 1212) 2024-09-20T22:56:08.078 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:56:08.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:08.100 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:08.100 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:56:08.201 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:56:08.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:08.223 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:08.223 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:56:08.229 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-20T22:56:08.229 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:56:08.323 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:56:08.346 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd unset noscrub 2024-09-20T22:56:08.348 INFO:tasks.rados.rados.0.smithi084.stdout:3547: done (1 left) 2024-09-20T22:56:08.348 INFO:tasks.rados.rados.0.smithi084.stdout:3549: done (0 left) 2024-09-20T22:56:08.348 INFO:tasks.rados.rados.0.smithi084.stdout:3550: delete oid 24 current snap is 413 2024-09-20T22:56:08.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:08.351 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:08.351 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:56:08.364 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:56:08 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: ::ffff:172.21.15.154 - - [20/Sep/2024:22:56:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-20T22:56:08.372 INFO:tasks.rados.rados.0.smithi084.stdout:3550: done (0 left) 2024-09-20T22:56:08.372 INFO:tasks.rados.rados.0.smithi084.stdout:3551: read oid 17 snap -1 2024-09-20T22:56:08.372 INFO:tasks.rados.rados.0.smithi084.stdout:3551: expect (ObjNum 1126 snap 381 seq_num 1126) 2024-09-20T22:56:08.372 INFO:tasks.rados.rados.0.smithi084.stdout:3552: rollback oid 5 current snap is 413 2024-09-20T22:56:08.373 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 5 to 342 2024-09-20T22:56:08.373 INFO:tasks.rados.rados.0.smithi084.stdout:3553: read oid 27 snap -1 2024-09-20T22:56:08.373 INFO:tasks.rados.rados.0.smithi084.stdout:3553: expect (ObjNum 1210 snap 412 seq_num 1210) 2024-09-20T22:56:08.373 INFO:tasks.rados.rados.0.smithi084.stdout:3554: write oid 21 current snap is 413 2024-09-20T22:56:08.373 INFO:tasks.rados.rados.0.smithi084.stdout:3554: seq_num 1215 ranges {420505=727662,1798916=553091} 2024-09-20T22:56:08.378 INFO:tasks.rados.rados.0.smithi084.stdout:3554: writing smithi08477203-21 from 420505 to 1148167 tid 1 2024-09-20T22:56:08.385 INFO:tasks.rados.rados.0.smithi084.stdout:3554: writing smithi08477203-21 from 1798916 to 2352007 tid 2 2024-09-20T22:56:08.385 INFO:tasks.rados.rados.0.smithi084.stdout:3555: copy_from oid 23 from oid 34 current snap is 413 2024-09-20T22:56:08.385 INFO:tasks.rados.rados.0.smithi084.stdout:3556: delete oid 42 current snap is 413 2024-09-20T22:56:08.387 INFO:tasks.rados.rados.0.smithi084.stdout:3551: expect (ObjNum 1126 snap 381 seq_num 1126) 2024-09-20T22:56:08.452 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:56:08.473 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:08.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:08.474 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:56:08.575 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:56:08.589 INFO:tasks.rados.rados.0.smithi084.stdout:3553: expect (ObjNum 1210 snap 412 seq_num 1210) 2024-09-20T22:56:08.597 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:08.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:08.598 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:56:08.699 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:56:08.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:08.720 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:08.721 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:56:08.732 INFO:tasks.rados.rados.0.smithi084.stdout:3555: finishing copy_from racing read to smithi08477203-23 2024-09-20T22:56:08.733 INFO:tasks.rados.rados.0.smithi084.stdout:3551: done (5 left) 2024-09-20T22:56:08.733 INFO:tasks.rados.rados.0.smithi084.stdout:3553: done (4 left) 2024-09-20T22:56:08.733 INFO:tasks.rados.rados.0.smithi084.stdout:3556: done (3 left) 2024-09-20T22:56:08.733 INFO:tasks.rados.rados.0.smithi084.stdout:3557: copy_from oid 8 from oid 48 current snap is 413 2024-09-20T22:56:08.733 INFO:tasks.rados.rados.0.smithi084.stdout:3554: finishing write tid 1 to smithi08477203-21 2024-09-20T22:56:08.733 INFO:tasks.rados.rados.0.smithi084.stdout:3554: finishing write tid 2 to smithi08477203-21 2024-09-20T22:56:08.733 INFO:tasks.rados.rados.0.smithi084.stdout:3554: finishing write tid 3 to smithi08477203-21 2024-09-20T22:56:08.733 INFO:tasks.rados.rados.0.smithi084.stdout:3554: finishing write tid 4 to smithi08477203-21 2024-09-20T22:56:08.733 INFO:tasks.rados.rados.0.smithi084.stdout:3554: oid 21 updating version 0 to 2592 2024-09-20T22:56:08.733 INFO:tasks.rados.rados.0.smithi084.stdout:3554: oid 21 version 2592 is already newer than 2590 2024-09-20T22:56:08.733 INFO:tasks.rados.rados.0.smithi084.stdout:3554: oid 21 version 2592 is already newer than 2591 2024-09-20T22:56:08.733 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 21 v 2592 (ObjNum 1215 snap 413 seq_num 1215) dirty exists 2024-09-20T22:56:08.733 INFO:tasks.rados.rados.0.smithi084.stdout:3554: left oid 21 (ObjNum 1215 snap 413 seq_num 1215) 2024-09-20T22:56:08.734 INFO:tasks.rados.rados.0.smithi084.stdout:3555: finishing copy_from to smithi08477203-23 2024-09-20T22:56:08.734 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 23 v 2589 (ObjNum 1066 snap 353 seq_num 1066) dirty exists 2024-09-20T22:56:08.734 INFO:tasks.rados.rados.0.smithi084.stdout:3554: done (3 left) 2024-09-20T22:56:08.734 INFO:tasks.rados.rados.0.smithi084.stdout:3555: done (2 left) 2024-09-20T22:56:08.734 INFO:tasks.rados.rados.0.smithi084.stdout:3558: snap_create 2024-09-20T22:56:08.821 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:56:08.843 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:08.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:08.844 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:56:08.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:08 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/4236344169' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["7"]}]': finished 2024-09-20T22:56:08.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:08 smithi084 ceph-mon[29774]: osdmap e887: 8 total, 8 up, 7 in 2024-09-20T22:56:08.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:08 smithi084 ceph-mon[29774]: osdmap e888: 8 total, 8 up, 7 in 2024-09-20T22:56:08.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:08 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:56:08.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:08 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/4236344169' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["7"]}]': finished 2024-09-20T22:56:08.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:08 smithi084 ceph-mon[40024]: osdmap e887: 8 total, 8 up, 7 in 2024-09-20T22:56:08.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:08 smithi084 ceph-mon[40024]: osdmap e888: 8 total, 8 up, 7 in 2024-09-20T22:56:08.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:08 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:56:08.882 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:08 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/4236344169' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["7"]}]': finished 2024-09-20T22:56:08.882 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:08 smithi154 ceph-mon[33264]: osdmap e887: 8 total, 8 up, 7 in 2024-09-20T22:56:08.882 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:08 smithi154 ceph-mon[33264]: osdmap e888: 8 total, 8 up, 7 in 2024-09-20T22:56:08.883 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:08 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:56:08.945 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:56:08.974 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:56:08.988 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:08.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:08.989 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:56:09.089 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:56:09.111 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:09.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:09.112 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:56:09.213 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:56:09.227 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-20T22:56:09.235 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:09.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:09.236 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:56:09.337 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:56:09.344 INFO:tasks.rados.rados.0.smithi084.stdout:3558: done (2 left) 2024-09-20T22:56:09.344 INFO:tasks.rados.rados.0.smithi084.stdout:3559: delete oid 43 current snap is 414 2024-09-20T22:56:09.359 INFO:tasks.rados.rados.0.smithi084.stdout:3559: done (2 left) 2024-09-20T22:56:09.359 INFO:tasks.rados.rados.0.smithi084.stdout:3560: snap_remove snap 399 2024-09-20T22:56:09.367 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:09.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:09.368 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:56:09.468 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:56:09.496 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:09.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:09.496 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:56:09.506 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:56:09.597 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:56:09.624 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:09.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:09.625 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:56:09.726 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:56:09.748 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:09.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:09.749 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:56:09.781 INFO:journalctl@ceph.iscsi.iscsi.a.smithi154.stdout:Sep 20 22:56:09 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-iscsi-iscsi-a[59730]: debug there is no tcmu-runner data available 2024-09-20T22:56:09.781 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:09 smithi154 ceph-mon[33264]: pgmap v1233: 158 pgs: 2 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 152 active+clean; 669 MiB data, 1.8 GiB used, 624 GiB / 626 GiB avail 2024-09-20T22:56:09.781 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:09 smithi154 ceph-mon[33264]: purged_snaps scrub starts 2024-09-20T22:56:09.781 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:09 smithi154 ceph-mon[33264]: purged_snaps scrub ok 2024-09-20T22:56:09.781 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:09 smithi154 ceph-mon[33264]: osdmap e889: 8 total, 8 up, 7 in 2024-09-20T22:56:09.781 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:09 smithi154 ceph-mon[33264]: 1.0 starting backfill to osd.1 from (0'0,0'0] MAX to 77'128 2024-09-20T22:56:09.781 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:09 smithi154 ceph-mon[33264]: 1.2 starting backfill to osd.2 from (0'0,0'0] MAX to 77'128 2024-09-20T22:56:09.782 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:09 smithi154 ceph-mon[33264]: 7.6 starting backfill to osd.5 from (0'0,0'0] MAX to 886'753 2024-09-20T22:56:09.850 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:56:09.864 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:09 smithi084 ceph-mon[40024]: pgmap v1233: 158 pgs: 2 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 152 active+clean; 669 MiB data, 1.8 GiB used, 624 GiB / 626 GiB avail 2024-09-20T22:56:09.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:09 smithi084 ceph-mon[40024]: purged_snaps scrub starts 2024-09-20T22:56:09.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:09 smithi084 ceph-mon[40024]: purged_snaps scrub ok 2024-09-20T22:56:09.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:09 smithi084 ceph-mon[40024]: osdmap e889: 8 total, 8 up, 7 in 2024-09-20T22:56:09.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:09 smithi084 ceph-mon[40024]: 1.0 starting backfill to osd.1 from (0'0,0'0] MAX to 77'128 2024-09-20T22:56:09.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:09 smithi084 ceph-mon[40024]: 1.2 starting backfill to osd.2 from (0'0,0'0] MAX to 77'128 2024-09-20T22:56:09.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:09 smithi084 ceph-mon[40024]: 7.6 starting backfill to osd.5 from (0'0,0'0] MAX to 886'753 2024-09-20T22:56:09.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:09 smithi084 ceph-mon[29774]: pgmap v1233: 158 pgs: 2 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 152 active+clean; 669 MiB data, 1.8 GiB used, 624 GiB / 626 GiB avail 2024-09-20T22:56:09.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:09 smithi084 ceph-mon[29774]: purged_snaps scrub starts 2024-09-20T22:56:09.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:09 smithi084 ceph-mon[29774]: purged_snaps scrub ok 2024-09-20T22:56:09.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:09 smithi084 ceph-mon[29774]: osdmap e889: 8 total, 8 up, 7 in 2024-09-20T22:56:09.866 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:09 smithi084 ceph-mon[29774]: 1.0 starting backfill to osd.1 from (0'0,0'0] MAX to 77'128 2024-09-20T22:56:09.866 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:09 smithi084 ceph-mon[29774]: 1.2 starting backfill to osd.2 from (0'0,0'0] MAX to 77'128 2024-09-20T22:56:09.866 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:09 smithi084 ceph-mon[29774]: 7.6 starting backfill to osd.5 from (0'0,0'0] MAX to 886'753 2024-09-20T22:56:09.873 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:09.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:09.874 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:56:09.975 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:56:09.996 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:09.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:09.997 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:56:10.037 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-20T22:56:10.037 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:56:10.097 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:56:10.119 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:10.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:10.120 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:56:10.221 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:56:10.242 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:10.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:10.243 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:56:10.342 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is unset 2024-09-20T22:56:10.344 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:56:10.348 INFO:tasks.rados.rados.0.smithi084.stdout:3560: done (2 left) 2024-09-20T22:56:10.348 INFO:tasks.rados.rados.0.smithi084.stdout:3561: snap_create 2024-09-20T22:56:10.348 INFO:tasks.rados.rados.0.smithi084.stdout:3552: finishing rollback tid 0 to smithi08477203-5 2024-09-20T22:56:10.348 INFO:tasks.rados.rados.0.smithi084.stdout:3552: finishing rollback tid 1 to smithi08477203-5 2024-09-20T22:56:10.348 INFO:tasks.rados.rados.0.smithi084.stdout:3552: finishing rollback tid 2 to smithi08477203-5 2024-09-20T22:56:10.348 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 5 v 2652 (ObjNum 1012 snap 333 seq_num 1012) dirty exists 2024-09-20T22:56:10.349 INFO:tasks.rados.rados.0.smithi084.stdout:3557: finishing copy_from racing read to smithi08477203-8 2024-09-20T22:56:10.349 INFO:tasks.rados.rados.0.smithi084.stdout:3557: finishing copy_from to smithi08477203-8 2024-09-20T22:56:10.349 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 8 v 1943 (ObjNum 1205 snap 412 seq_num 1205) dirty exists 2024-09-20T22:56:10.376 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:10.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:10.377 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:56:10.478 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:56:10.480 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-20T22:56:10.499 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:10.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:10.500 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:56:10.600 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:56:10.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:10 smithi084 ceph-mon[40024]: 1.7 starting backfill to osd.5 from (0'0,0'0] MAX to 77'128 2024-09-20T22:56:10.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:10 smithi084 ceph-mon[40024]: 7.5 starting backfill to osd.1 from (0'0,0'0] MAX to 886'492 2024-09-20T22:56:10.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:10 smithi084 ceph-mon[40024]: 7.8 starting backfill to osd.3 from (0'0,0'0] MAX to 886'1270 2024-09-20T22:56:10.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:10 smithi084 ceph-mon[40024]: 1.4 starting backfill to osd.6 from (0'0,0'0] MAX to 855'129 2024-09-20T22:56:10.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:10 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/3651933645' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:56:10.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:10 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/3651933645' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T22:56:10.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:10 smithi084 ceph-mon[40024]: osdmap e890: 8 total, 8 up, 7 in 2024-09-20T22:56:10.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:10 smithi084 ceph-mon[29774]: 1.7 starting backfill to osd.5 from (0'0,0'0] MAX to 77'128 2024-09-20T22:56:10.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:10 smithi084 ceph-mon[29774]: 7.5 starting backfill to osd.1 from (0'0,0'0] MAX to 886'492 2024-09-20T22:56:10.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:10 smithi084 ceph-mon[29774]: 7.8 starting backfill to osd.3 from (0'0,0'0] MAX to 886'1270 2024-09-20T22:56:10.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:10 smithi084 ceph-mon[29774]: 1.4 starting backfill to osd.6 from (0'0,0'0] MAX to 855'129 2024-09-20T22:56:10.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:10 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/3651933645' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:56:10.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:10 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/3651933645' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T22:56:10.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:10 smithi084 ceph-mon[29774]: osdmap e890: 8 total, 8 up, 7 in 2024-09-20T22:56:10.621 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:10.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:10.622 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:56:10.723 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:56:10.744 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:10.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:10.745 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:56:10.750 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:10 smithi154 ceph-mon[33264]: 1.7 starting backfill to osd.5 from (0'0,0'0] MAX to 77'128 2024-09-20T22:56:10.750 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:10 smithi154 ceph-mon[33264]: 7.5 starting backfill to osd.1 from (0'0,0'0] MAX to 886'492 2024-09-20T22:56:10.750 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:10 smithi154 ceph-mon[33264]: 7.8 starting backfill to osd.3 from (0'0,0'0] MAX to 886'1270 2024-09-20T22:56:10.750 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:10 smithi154 ceph-mon[33264]: 1.4 starting backfill to osd.6 from (0'0,0'0] MAX to 855'129 2024-09-20T22:56:10.750 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:10 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/3651933645' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:56:10.750 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:10 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/3651933645' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T22:56:10.750 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:10 smithi154 ceph-mon[33264]: osdmap e890: 8 total, 8 up, 7 in 2024-09-20T22:56:10.776 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.7 dump_blocked_ops 2024-09-20T22:56:10.846 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:56:10.881 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:10.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:10.882 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:56:10.982 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:56:11.005 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:11.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:11.006 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:56:11.064 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:56:11.107 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:56:11.129 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:11.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:11.130 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:56:11.162 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:56:11.231 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:56:11.253 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:11.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:11.254 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:56:11.351 INFO:tasks.rados.rados.0.smithi084.stdout:3552: done (2 left) 2024-09-20T22:56:11.351 INFO:tasks.rados.rados.0.smithi084.stdout:3557: done (1 left) 2024-09-20T22:56:11.351 INFO:tasks.rados.rados.0.smithi084.stdout:3561: done (0 left) 2024-09-20T22:56:11.351 INFO:tasks.rados.rados.0.smithi084.stdout:3562: write oid 39 current snap is 415 2024-09-20T22:56:11.352 INFO:tasks.rados.rados.0.smithi084.stdout:3562: seq_num 1218 ranges {470926=669762,1574362=700418} 2024-09-20T22:56:11.355 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:56:11.358 INFO:tasks.rados.rados.0.smithi084.stdout:3562: writing smithi08477203-39 from 470926 to 1140688 tid 1 2024-09-20T22:56:11.365 INFO:tasks.rados.rados.0.smithi084.stdout:3562: writing smithi08477203-39 from 1574362 to 2274780 tid 2 2024-09-20T22:56:11.365 INFO:tasks.rados.rados.0.smithi084.stdout:3563: write (excl) oid 31 current snap is 415 2024-09-20T22:56:11.365 INFO:tasks.rados.rados.0.smithi084.stdout:3563: seq_num 1219 ranges {487733=745011,1632844=560429} 2024-09-20T22:56:11.372 INFO:tasks.rados.rados.0.smithi084.stdout:3563: writing smithi08477203-31 from 487733 to 1232744 tid 1 2024-09-20T22:56:11.379 INFO:tasks.rados.rados.0.smithi084.stdout:3563: writing smithi08477203-31 from 1632844 to 2193273 tid 2 2024-09-20T22:56:11.379 INFO:tasks.rados.rados.0.smithi084.stdout:3564: read oid 21 snap -1 2024-09-20T22:56:11.379 INFO:tasks.rados.rados.0.smithi084.stdout:3564: expect (ObjNum 1215 snap 413 seq_num 1215) 2024-09-20T22:56:11.379 INFO:tasks.rados.rados.0.smithi084.stdout:3565: snap_remove snap 407 2024-09-20T22:56:11.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:11.384 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:11.384 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:56:11.485 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:56:11.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:11.508 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:11.508 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:56:11.538 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:11 smithi084 ceph-mon[40024]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:56:11.538 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:11 smithi084 ceph-mon[40024]: pgmap v1236: 158 pgs: 10 remapped+peering, 2 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 142 active+clean; 688 MiB data, 1.8 GiB used, 624 GiB / 626 GiB avail; 4.8 MiB/s wr, 14 op/s; 40 KiB/s, 16 objects/s recovering 2024-09-20T22:56:11.538 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:11 smithi084 ceph-mon[40024]: 7.9 scrub starts 2024-09-20T22:56:11.538 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:11 smithi084 ceph-mon[40024]: Health check failed: Reduced data availability: 1 pg inactive, 3 pgs peering (PG_AVAILABILITY) 2024-09-20T22:56:11.538 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:11 smithi084 ceph-mon[40024]: 7.9 scrub ok 2024-09-20T22:56:11.539 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:11 smithi084 ceph-mon[40024]: osdmap e891: 8 total, 8 up, 7 in 2024-09-20T22:56:11.539 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:11 smithi084 ceph-mon[29774]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:56:11.539 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:11 smithi084 ceph-mon[29774]: pgmap v1236: 158 pgs: 10 remapped+peering, 2 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 142 active+clean; 688 MiB data, 1.8 GiB used, 624 GiB / 626 GiB avail; 4.8 MiB/s wr, 14 op/s; 40 KiB/s, 16 objects/s recovering 2024-09-20T22:56:11.539 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:11 smithi084 ceph-mon[29774]: 7.9 scrub starts 2024-09-20T22:56:11.539 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:11 smithi084 ceph-mon[29774]: Health check failed: Reduced data availability: 1 pg inactive, 3 pgs peering (PG_AVAILABILITY) 2024-09-20T22:56:11.539 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:11 smithi084 ceph-mon[29774]: 7.9 scrub ok 2024-09-20T22:56:11.539 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:11 smithi084 ceph-mon[29774]: osdmap e891: 8 total, 8 up, 7 in 2024-09-20T22:56:11.546 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-20T22:56:11.546 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:56:11.609 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:56:11.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:11.631 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:11.633 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:56:11.646 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:56:11.646 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:56:11.654 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:56:11.654 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:56:11.670 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:56:11.670 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:56:11.686 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-20T22:56:11.687 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T22:56:11.703 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:56:11.703 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:56:11.720 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:56:11.720 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:56:11.734 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:56:11.736 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:56:11.736 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:56:11.753 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:56:11.753 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:56:11.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:11 smithi154 ceph-mon[33264]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:56:11.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:11 smithi154 ceph-mon[33264]: pgmap v1236: 158 pgs: 10 remapped+peering, 2 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 142 active+clean; 688 MiB data, 1.8 GiB used, 624 GiB / 626 GiB avail; 4.8 MiB/s wr, 14 op/s; 40 KiB/s, 16 objects/s recovering 2024-09-20T22:56:11.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:11 smithi154 ceph-mon[33264]: 7.9 scrub starts 2024-09-20T22:56:11.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:11 smithi154 ceph-mon[33264]: Health check failed: Reduced data availability: 1 pg inactive, 3 pgs peering (PG_AVAILABILITY) 2024-09-20T22:56:11.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:11 smithi154 ceph-mon[33264]: 7.9 scrub ok 2024-09-20T22:56:11.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:11 smithi154 ceph-mon[33264]: osdmap e891: 8 total, 8 up, 7 in 2024-09-20T22:56:11.756 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:11.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:11.757 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:56:11.857 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:56:11.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:11.879 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:11.880 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:56:11.980 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:56:12.001 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:12.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:12.002 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:56:12.103 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:56:12.125 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:12.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:12.126 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:56:12.130 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.7 dump_historic_ops 2024-09-20T22:56:12.227 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:56:12.248 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:12.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:12.249 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:56:12.349 INFO:tasks.rados.rados.0.smithi084.stdout:3565: done (3 left) 2024-09-20T22:56:12.349 INFO:tasks.rados.rados.0.smithi084.stdout:3566: snap_remove snap 376 2024-09-20T22:56:12.350 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:56:12.372 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:12.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:12.373 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:56:12.419 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:56:12.473 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:56:12.494 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:12.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:12.495 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:56:12.596 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:56:12.617 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:12.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:12.619 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:56:12.719 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:56:12.741 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:12.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:12.742 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:56:12.787 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:12 smithi154 ceph-mon[33264]: 1.9 scrub starts 2024-09-20T22:56:12.787 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:12 smithi154 ceph-mon[33264]: 1.9 scrub ok 2024-09-20T22:56:12.787 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:12 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/2636557174' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:56:12.787 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:12 smithi154 ceph-mon[33264]: 3.11 scrub starts 2024-09-20T22:56:12.787 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:12 smithi154 ceph-mon[33264]: 3.11 scrub ok 2024-09-20T22:56:12.787 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:12 smithi154 ceph-mon[33264]: osdmap e892: 8 total, 8 up, 7 in 2024-09-20T22:56:12.842 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:56:12.864 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:12.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:12 smithi084 ceph-mon[29774]: 1.9 scrub starts 2024-09-20T22:56:12.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:12 smithi084 ceph-mon[29774]: 1.9 scrub ok 2024-09-20T22:56:12.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:12 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/2636557174' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:56:12.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:12 smithi084 ceph-mon[29774]: 3.11 scrub starts 2024-09-20T22:56:12.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:12 smithi084 ceph-mon[29774]: 3.11 scrub ok 2024-09-20T22:56:12.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:12 smithi084 ceph-mon[29774]: osdmap e892: 8 total, 8 up, 7 in 2024-09-20T22:56:12.866 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:12 smithi084 ceph-mon[40024]: 1.9 scrub starts 2024-09-20T22:56:12.866 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:12 smithi084 ceph-mon[40024]: 1.9 scrub ok 2024-09-20T22:56:12.866 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:12 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/2636557174' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:56:12.866 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:12 smithi084 ceph-mon[40024]: 3.11 scrub starts 2024-09-20T22:56:12.866 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:12 smithi084 ceph-mon[40024]: 3.11 scrub ok 2024-09-20T22:56:12.866 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:12 smithi084 ceph-mon[40024]: osdmap e892: 8 total, 8 up, 7 in 2024-09-20T22:56:12.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:12.866 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:56:12.887 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-20T22:56:12.887 INFO:teuthology.orchestra.run.smithi154.stdout: "size": 20, 2024-09-20T22:56:12.888 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 600, 2024-09-20T22:56:12.888 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [ 2024-09-20T22:56:12.888 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.888 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:3132 7.19 e328/139 7:9eac4560:::smithi08477203-30:head v 328'198, mlcod=322'194)", 2024-09-20T22:56:12.888 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:46:48.223406+0000", 2024-09-20T22:56:12.888 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 564.6601369, 2024-09-20T22:56:12.888 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.016765688000000001, 2024-09-20T22:56:12.889 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:56:12.889 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:56:12.889 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:56:12.889 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:56:12.889 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.889 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:56:12.890 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.223406+0000", 2024-09-20T22:56:12.890 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:56:12.890 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.890 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.890 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:56:12.890 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.223406+0000", 2024-09-20T22:56:12.891 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:56:12.891 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.891 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.891 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:56:12.891 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.223406+0000", 2024-09-20T22:56:12.892 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.6699999999999999e-07 2024-09-20T22:56:12.892 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.892 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.892 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:56:12.892 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.226227+0000", 2024-09-20T22:56:12.892 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0028205690000000002 2024-09-20T22:56:12.893 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.893 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.893 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:56:12.893 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.226238+0000", 2024-09-20T22:56:12.893 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0879999999999999e-05 2024-09-20T22:56:12.893 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.893 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.893 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:56:12.893 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.226249+0000", 2024-09-20T22:56:12.893 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0912e-05 2024-09-20T22:56:12.893 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.893 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.894 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:56:12.894 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.227960+0000", 2024-09-20T22:56:12.894 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0017109619999999999 2024-09-20T22:56:12.894 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.894 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.894 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:56:12.894 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.227987+0000", 2024-09-20T22:56:12.894 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.7659999999999999e-05 2024-09-20T22:56:12.895 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.895 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.895 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:56:12.895 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.240119+0000", 2024-09-20T22:56:12.895 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.012131885 2024-09-20T22:56:12.895 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.895 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.895 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:56:12.895 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:48.240172+0000", 2024-09-20T22:56:12.895 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.2453000000000001e-05 2024-09-20T22:56:12.896 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:56:12.896 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:56:12.896 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:56:12.896 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.896 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.896 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:3232 7.16 7:6885a65f:::smithi08477203-14:head [copy-from ver 187] snapc 7e=[7e,7d,7c,7b,7a,79,78,75,74,73,70,6b] ondisk+write+known_if_redirected+supports_pool_eio e337)", 2024-09-20T22:56:12.896 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:46:57.213186+0000", 2024-09-20T22:56:12.896 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 555.67035722900005, 2024-09-20T22:56:12.896 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.028083105000000001, 2024-09-20T22:56:12.896 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:56:12.896 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:56:12.896 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:56:12.896 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:56:12.896 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:56:12.897 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:56:12.897 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 3232 2024-09-20T22:56:12.897 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.897 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:56:12.897 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.897 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:56:12.897 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.213186+0000", 2024-09-20T22:56:12.897 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:56:12.897 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.897 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.897 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:56:12.897 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.213186+0000", 2024-09-20T22:56:12.897 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:56:12.897 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.898 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.898 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:56:12.898 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.213188+0000", 2024-09-20T22:56:12.898 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.5129999999999999e-06 2024-09-20T22:56:12.898 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.898 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.898 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:56:12.898 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.213194+0000", 2024-09-20T22:56:12.898 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.4810000000000003e-06 2024-09-20T22:56:12.898 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.898 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.899 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:56:12.899 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.213215+0000", 2024-09-20T22:56:12.899 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.0455999999999999e-05 2024-09-20T22:56:12.899 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.899 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.899 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:56:12.899 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.213225+0000", 2024-09-20T22:56:12.899 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0212e-05 2024-09-20T22:56:12.899 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.899 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.899 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:56:12.899 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.213315+0000", 2024-09-20T22:56:12.899 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.9524000000000005e-05 2024-09-20T22:56:12.899 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.900 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.900 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op must wait for map", 2024-09-20T22:56:12.900 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.213324+0000", 2024-09-20T22:56:12.900 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.8890000000000008e-06 2024-09-20T22:56:12.900 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.900 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.900 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:56:12.900 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.220517+0000", 2024-09-20T22:56:12.900 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0071931219999999997 2024-09-20T22:56:12.901 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.901 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.901 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:56:12.901 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.220636+0000", 2024-09-20T22:56:12.901 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000118875 2024-09-20T22:56:12.901 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.902 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.902 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 2,5", 2024-09-20T22:56:12.902 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.230308+0000", 2024-09-20T22:56:12.902 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0096726299999999998 2024-09-20T22:56:12.902 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.902 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.902 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:56:12.902 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.238007+0000", 2024-09-20T22:56:12.902 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.007699287 2024-09-20T22:56:12.902 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.902 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.902 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:56:12.902 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.240863+0000", 2024-09-20T22:56:12.902 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0028554880000000002 2024-09-20T22:56:12.903 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.903 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.903 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:56:12.903 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.241194+0000", 2024-09-20T22:56:12.903 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00033087399999999999 2024-09-20T22:56:12.903 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.903 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.904 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:56:12.904 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.241233+0000", 2024-09-20T22:56:12.904 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.9338000000000002e-05 2024-09-20T22:56:12.904 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.904 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.904 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:56:12.904 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.241269+0000", 2024-09-20T22:56:12.904 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.5416000000000001e-05 2024-09-20T22:56:12.904 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:56:12.904 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:56:12.904 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:56:12.904 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.905 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.905 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:3233 7.16 7:6885a65f:::smithi08477203-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e337)", 2024-09-20T22:56:12.905 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:46:57.213253+0000", 2024-09-20T22:56:12.905 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 555.67028980500004, 2024-09-20T22:56:12.905 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.023594825, 2024-09-20T22:56:12.905 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:56:12.905 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:56:12.905 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:56:12.905 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:56:12.905 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:56:12.905 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:56:12.905 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 3233 2024-09-20T22:56:12.905 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.905 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:56:12.905 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.906 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:56:12.906 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.213253+0000", 2024-09-20T22:56:12.906 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:56:12.906 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.906 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.906 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:56:12.906 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.213253+0000", 2024-09-20T22:56:12.907 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:56:12.907 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.907 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.907 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:56:12.907 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.213254+0000", 2024-09-20T22:56:12.907 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.4799999999999998e-07 2024-09-20T22:56:12.907 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.908 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.908 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:56:12.908 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.213257+0000", 2024-09-20T22:56:12.908 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.3780000000000001e-06 2024-09-20T22:56:12.908 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.908 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.909 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:56:12.909 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.213267+0000", 2024-09-20T22:56:12.909 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.9480000000000003e-06 2024-09-20T22:56:12.909 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.909 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.909 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:56:12.909 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.213273+0000", 2024-09-20T22:56:12.910 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.1240000000000002e-06 2024-09-20T22:56:12.910 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.910 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.910 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:56:12.910 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.213394+0000", 2024-09-20T22:56:12.910 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000120717 2024-09-20T22:56:12.910 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.910 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.911 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting_for_map not empty", 2024-09-20T22:56:12.911 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.213400+0000", 2024-09-20T22:56:12.911 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.0769999999999996e-06 2024-09-20T22:56:12.911 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.911 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.911 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:56:12.911 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.220851+0000", 2024-09-20T22:56:12.911 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0074511309999999997 2024-09-20T22:56:12.912 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.912 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.912 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for blocked object", 2024-09-20T22:56:12.912 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.220903+0000", 2024-09-20T22:56:12.912 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.215e-05 2024-09-20T22:56:12.912 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.913 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.913 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:56:12.913 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.236698+0000", 2024-09-20T22:56:12.913 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.015794823 2024-09-20T22:56:12.913 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.913 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.913 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:56:12.914 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.236766+0000", 2024-09-20T22:56:12.914 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.8126999999999993e-05 2024-09-20T22:56:12.914 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.914 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.914 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:56:12.914 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:46:57.236848+0000", 2024-09-20T22:56:12.914 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.1701999999999998e-05 2024-09-20T22:56:12.914 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:56:12.915 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:56:12.915 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:56:12.915 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.915 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.915 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:3671 7.5 7:a160b5d4:::smithi08477203-16:head [copy-from ver 236] snapc 91=[91,8e,8d,8c,8b,8a,88,87,86,85,84,80,7f,7a,74,73,6b] ondisk+write+known_if_redirected+supports_pool_eio e370)", 2024-09-20T22:56:12.915 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:47:30.237235+0000", 2024-09-20T22:56:12.915 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 522.64630800600003, 2024-09-20T22:56:12.915 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.019719474000000001, 2024-09-20T22:56:12.915 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:56:12.915 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:56:12.915 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:56:12.915 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:56:12.916 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:56:12.916 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:56:12.916 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 3671 2024-09-20T22:56:12.916 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.916 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:56:12.916 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.916 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:56:12.916 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:30.237235+0000", 2024-09-20T22:56:12.916 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:56:12.916 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.916 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.916 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:56:12.916 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:30.237235+0000", 2024-09-20T22:56:12.917 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:56:12.917 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.917 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.917 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:56:12.917 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:30.237236+0000", 2024-09-20T22:56:12.917 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.5570000000000001e-06 2024-09-20T22:56:12.917 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.917 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.917 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:56:12.917 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:30.237241+0000", 2024-09-20T22:56:12.917 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.3279999999999999e-06 2024-09-20T22:56:12.917 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.917 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.917 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:56:12.918 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:30.237252+0000", 2024-09-20T22:56:12.918 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1004e-05 2024-09-20T22:56:12.918 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.918 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.918 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:56:12.918 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:30.237263+0000", 2024-09-20T22:56:12.918 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1341e-05 2024-09-20T22:56:12.918 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.918 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.918 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:56:12.918 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:30.237319+0000", 2024-09-20T22:56:12.918 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.5939000000000001e-05 2024-09-20T22:56:12.918 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.919 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.919 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:56:12.919 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:30.237436+0000", 2024-09-20T22:56:12.919 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000116866 2024-09-20T22:56:12.919 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.919 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.919 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 1,2", 2024-09-20T22:56:12.919 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:30.244330+0000", 2024-09-20T22:56:12.919 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0068945619999999999 2024-09-20T22:56:12.919 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.919 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.919 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:56:12.919 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:30.249103+0000", 2024-09-20T22:56:12.920 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0047726319999999997 2024-09-20T22:56:12.920 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.920 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.920 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:56:12.920 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:30.255771+0000", 2024-09-20T22:56:12.920 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.006667666 2024-09-20T22:56:12.920 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.920 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.920 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:56:12.920 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:30.256879+0000", 2024-09-20T22:56:12.920 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0011083110000000001 2024-09-20T22:56:12.920 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.920 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.920 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:56:12.920 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:30.256919+0000", 2024-09-20T22:56:12.921 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.9589000000000003e-05 2024-09-20T22:56:12.921 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.921 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.921 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:56:12.921 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:30.256954+0000", 2024-09-20T22:56:12.921 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.5679000000000002e-05 2024-09-20T22:56:12.921 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:56:12.921 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:56:12.921 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:56:12.921 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.921 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.921 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:3747 7.c e383/94 7:3f865ef5:::smithi08477203-28:head v 383'822, mlcod=382'818)", 2024-09-20T22:56:12.921 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:47:43.267267+0000", 2024-09-20T22:56:12.921 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 509.61627554199998, 2024-09-20T22:56:12.922 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.014405072, 2024-09-20T22:56:12.922 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:56:12.922 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:56:12.922 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:56:12.922 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:56:12.922 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.922 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:56:12.922 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.267267+0000", 2024-09-20T22:56:12.922 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:56:12.923 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.923 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.923 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:56:12.923 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.267267+0000", 2024-09-20T22:56:12.923 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:56:12.923 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.924 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.924 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:56:12.924 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.267267+0000", 2024-09-20T22:56:12.924 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.73e-07 2024-09-20T22:56:12.924 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.924 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.925 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:56:12.925 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.269537+0000", 2024-09-20T22:56:12.925 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0022696650000000001 2024-09-20T22:56:12.925 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.925 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.925 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:56:12.925 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.269564+0000", 2024-09-20T22:56:12.925 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.6653999999999999e-05 2024-09-20T22:56:12.925 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.925 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.925 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:56:12.926 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.269575+0000", 2024-09-20T22:56:12.926 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1661999999999999e-05 2024-09-20T22:56:12.926 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.926 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.926 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:56:12.926 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.269672+0000", 2024-09-20T22:56:12.926 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.6430999999999999e-05 2024-09-20T22:56:12.926 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.926 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.926 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:56:12.926 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.269716+0000", 2024-09-20T22:56:12.926 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.4518000000000003e-05 2024-09-20T22:56:12.927 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.927 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.927 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:56:12.927 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.281609+0000", 2024-09-20T22:56:12.929 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.011892982999999999 2024-09-20T22:56:12.929 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.929 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.929 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:56:12.929 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.281672+0000", 2024-09-20T22:56:12.929 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.2985999999999996e-05 2024-09-20T22:56:12.929 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:56:12.929 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:56:12.929 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:56:12.929 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.930 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.930 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:6253 7.9 e629/602 7:91f2d47c:::smithi08477203-8:head v 629'948, mlcod=627'945)", 2024-09-20T22:56:12.930 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:51:49.626110+0000", 2024-09-20T22:56:12.930 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 263.25743265199998, 2024-09-20T22:56:12.930 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.017238303, 2024-09-20T22:56:12.930 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:56:12.930 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:56:12.930 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:56:12.930 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:56:12.930 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.930 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:56:12.930 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:49.626110+0000", 2024-09-20T22:56:12.930 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:56:12.931 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.931 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.931 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:56:12.931 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:49.626110+0000", 2024-09-20T22:56:12.931 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:56:12.931 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.931 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.931 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:56:12.931 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:49.626111+0000", 2024-09-20T22:56:12.931 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.33e-06 2024-09-20T22:56:12.931 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.931 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.931 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:56:12.931 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:49.633470+0000", 2024-09-20T22:56:12.932 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0073589700000000003 2024-09-20T22:56:12.932 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.932 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.932 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:56:12.932 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:49.633492+0000", 2024-09-20T22:56:12.932 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.1443000000000001e-05 2024-09-20T22:56:12.932 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.932 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.932 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:56:12.932 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:49.633503+0000", 2024-09-20T22:56:12.932 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1610000000000001e-05 2024-09-20T22:56:12.932 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.932 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.932 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:56:12.933 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:49.633575+0000", 2024-09-20T22:56:12.933 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.1433999999999994e-05 2024-09-20T22:56:12.933 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.933 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.933 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:56:12.933 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:49.633600+0000", 2024-09-20T22:56:12.933 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.4960000000000002e-05 2024-09-20T22:56:12.933 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.933 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.933 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:56:12.933 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:49.643306+0000", 2024-09-20T22:56:12.933 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0097065109999999993 2024-09-20T22:56:12.933 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.933 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.934 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:56:12.934 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:49.643348+0000", 2024-09-20T22:56:12.934 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.2045000000000001e-05 2024-09-20T22:56:12.934 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:56:12.934 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:56:12.934 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:56:12.934 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.934 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.934 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:6659 7.5 7:a160b5d4:::smithi08477203-16:head [copy-from ver 1807] snapc 12d=[12d,12c,12b,12a,128,127,126,124,122,121,120,11f,11e,11b,119,118,115,112,10e,10c,107,103,fd,fb,fa,f8,ed,d7,c2,bf] ondisk+write+known_if_redirected+supports_pool_eio e674)", 2024-09-20T22:56:12.934 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:52:34.370352+0000", 2024-09-20T22:56:12.934 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 218.51319115800001, 2024-09-20T22:56:12.934 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.018083765000000002, 2024-09-20T22:56:12.934 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:56:12.934 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:56:12.935 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:56:12.935 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:56:12.935 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:56:12.935 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:56:12.935 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 6659 2024-09-20T22:56:12.935 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.935 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:56:12.935 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.935 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:56:12.935 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:34.370352+0000", 2024-09-20T22:56:12.935 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:56:12.935 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.935 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.935 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:56:12.936 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:34.370352+0000", 2024-09-20T22:56:12.936 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:56:12.936 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.936 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.936 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:56:12.936 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:34.370353+0000", 2024-09-20T22:56:12.936 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.61e-06 2024-09-20T22:56:12.936 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.936 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.936 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:56:12.936 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:34.370360+0000", 2024-09-20T22:56:12.936 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.6760000000000001e-06 2024-09-20T22:56:12.936 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.936 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.937 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:56:12.937 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:34.370371+0000", 2024-09-20T22:56:12.937 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1155000000000001e-05 2024-09-20T22:56:12.937 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.937 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.937 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:56:12.937 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:34.370380+0000", 2024-09-20T22:56:12.937 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.9039999999999998e-06 2024-09-20T22:56:12.937 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.937 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.937 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:56:12.937 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:34.370461+0000", 2024-09-20T22:56:12.937 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.0667000000000005e-05 2024-09-20T22:56:12.937 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.937 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.938 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:56:12.938 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:34.370571+0000", 2024-09-20T22:56:12.938 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000110129 2024-09-20T22:56:12.938 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.938 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.938 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 0,2", 2024-09-20T22:56:12.938 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:34.378118+0000", 2024-09-20T22:56:12.938 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0075469489999999998 2024-09-20T22:56:12.938 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.938 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.938 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:56:12.938 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:34.382656+0000", 2024-09-20T22:56:12.938 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0045383239999999998 2024-09-20T22:56:12.938 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.939 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.939 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:56:12.939 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:34.385168+0000", 2024-09-20T22:56:12.939 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0025123770000000001 2024-09-20T22:56:12.939 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.939 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.939 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:56:12.939 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:34.388343+0000", 2024-09-20T22:56:12.939 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.003174713 2024-09-20T22:56:12.939 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.939 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.939 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:56:12.939 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:34.388382+0000", 2024-09-20T22:56:12.939 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.8550000000000002e-05 2024-09-20T22:56:12.940 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.940 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.940 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:56:12.940 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:34.388435+0000", 2024-09-20T22:56:12.940 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.3711000000000001e-05 2024-09-20T22:56:12.940 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:56:12.940 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:56:12.940 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:56:12.940 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.940 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.940 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:6829 7.5 7:a160b5d4:::smithi08477203-16:head [write 548159~419732,stat] snapc 136=[136,135,134,133,132,131,130,12f,12e,12d,12c,12b,12a,128,127,126,124,122,121,11f,11b,119,112,10e,10c,107,103,fb,fa,f8,ed,d7,c2,bf] ondisk+write+known_if_redirected+supports_pool_eio e688)", 2024-09-20T22:56:12.940 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:52:48.308503+0000", 2024-09-20T22:56:12.940 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 204.57503983300001, 2024-09-20T22:56:12.940 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.020227210999999998, 2024-09-20T22:56:12.940 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:56:12.941 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:56:12.941 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:56:12.941 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:56:12.941 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:56:12.941 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:56:12.941 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 6829 2024-09-20T22:56:12.941 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.941 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:56:12.941 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.941 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:56:12.941 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.308503+0000", 2024-09-20T22:56:12.941 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:56:12.941 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.941 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.942 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:56:12.942 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.308503+0000", 2024-09-20T22:56:12.942 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:56:12.942 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.942 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.942 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:56:12.942 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.308505+0000", 2024-09-20T22:56:12.942 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.765e-06 2024-09-20T22:56:12.942 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.942 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.942 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:56:12.942 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.308957+0000", 2024-09-20T22:56:12.942 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00045254399999999998 2024-09-20T22:56:12.942 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.943 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.943 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:56:12.943 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.308975+0000", 2024-09-20T22:56:12.943 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.7489000000000001e-05 2024-09-20T22:56:12.943 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.943 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.943 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:56:12.943 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.308983+0000", 2024-09-20T22:56:12.943 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.4500000000000004e-06 2024-09-20T22:56:12.943 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.943 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.943 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:56:12.943 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.309063+0000", 2024-09-20T22:56:12.943 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.0264000000000003e-05 2024-09-20T22:56:12.944 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.944 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.944 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:56:12.944 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.309200+0000", 2024-09-20T22:56:12.944 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00013697500000000001 2024-09-20T22:56:12.944 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.944 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.944 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 0,2", 2024-09-20T22:56:12.944 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.309374+0000", 2024-09-20T22:56:12.944 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000173597 2024-09-20T22:56:12.944 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.944 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.944 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:56:12.944 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.318972+0000", 2024-09-20T22:56:12.944 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0095980430000000005 2024-09-20T22:56:12.945 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.945 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.945 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:56:12.945 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.326825+0000", 2024-09-20T22:56:12.945 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0078531720000000003 2024-09-20T22:56:12.945 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.945 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.945 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:56:12.945 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.328638+0000", 2024-09-20T22:56:12.945 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0018131969999999999 2024-09-20T22:56:12.945 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.945 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.945 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:56:12.945 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.328685+0000", 2024-09-20T22:56:12.946 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.6536999999999998e-05 2024-09-20T22:56:12.946 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.946 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.946 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:56:12.946 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.328730+0000", 2024-09-20T22:56:12.946 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.5178e-05 2024-09-20T22:56:12.946 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:56:12.946 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:56:12.946 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:56:12.946 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.946 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.946 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:7173 7.8 e721/430 7:1e9e2524:::smithi08477203-5:head v 721'865, mlcod=721'861)", 2024-09-20T22:56:12.946 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:53:21.336783+0000", 2024-09-20T22:56:12.946 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 171.546759963, 2024-09-20T22:56:12.947 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.015521446, 2024-09-20T22:56:12.947 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:56:12.947 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:56:12.947 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:56:12.947 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:56:12.947 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.947 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:56:12.947 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:21.336783+0000", 2024-09-20T22:56:12.947 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:56:12.947 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.947 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.947 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:56:12.947 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:21.336783+0000", 2024-09-20T22:56:12.947 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:56:12.948 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.948 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.948 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:56:12.948 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:21.336784+0000", 2024-09-20T22:56:12.948 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.7899999999999997e-07 2024-09-20T22:56:12.948 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.948 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.948 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:56:12.948 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:21.344323+0000", 2024-09-20T22:56:12.948 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0075393090000000001 2024-09-20T22:56:12.948 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.948 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.948 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:56:12.948 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:21.344349+0000", 2024-09-20T22:56:12.948 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.5712e-05 2024-09-20T22:56:12.949 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.949 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.949 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:56:12.949 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:21.344362+0000", 2024-09-20T22:56:12.949 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3131e-05 2024-09-20T22:56:12.949 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.949 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.949 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:56:12.949 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:21.344434+0000", 2024-09-20T22:56:12.949 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.1837999999999999e-05 2024-09-20T22:56:12.949 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.949 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.949 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:56:12.949 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:21.344454+0000", 2024-09-20T22:56:12.950 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.0404999999999999e-05 2024-09-20T22:56:12.950 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.950 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.950 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:56:12.950 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:21.352250+0000", 2024-09-20T22:56:12.950 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0077961489999999996 2024-09-20T22:56:12.950 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.950 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.950 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:56:12.950 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:21.352304+0000", 2024-09-20T22:56:12.950 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.4122999999999997e-05 2024-09-20T22:56:12.950 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:56:12.950 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:56:12.950 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:56:12.951 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.951 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.951 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:7387 7.9 e743/602 7:907aec0e:::smithi08477203-3:head v 743'1242, mlcod=741'1234)", 2024-09-20T22:56:12.951 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:53:43.309636+0000", 2024-09-20T22:56:12.951 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 149.57390679100001, 2024-09-20T22:56:12.951 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.032548296999999997, 2024-09-20T22:56:12.951 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:56:12.951 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:56:12.951 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:56:12.951 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:56:12.951 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.951 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:56:12.951 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:43.309636+0000", 2024-09-20T22:56:12.951 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:56:12.952 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.952 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.952 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:56:12.952 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:43.309636+0000", 2024-09-20T22:56:12.952 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:56:12.952 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.952 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.952 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:56:12.952 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:43.309637+0000", 2024-09-20T22:56:12.952 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.1699999999999997e-07 2024-09-20T22:56:12.952 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.952 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.952 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:56:12.952 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:43.338457+0000", 2024-09-20T22:56:12.953 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.028819827999999999 2024-09-20T22:56:12.953 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.953 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.953 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:56:12.953 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:43.338482+0000", 2024-09-20T22:56:12.953 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.55e-05 2024-09-20T22:56:12.953 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.953 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.953 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:56:12.953 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:43.338493+0000", 2024-09-20T22:56:12.953 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1308000000000001e-05 2024-09-20T22:56:12.953 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.953 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.953 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:56:12.953 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:43.338561+0000", 2024-09-20T22:56:12.954 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.7535999999999998e-05 2024-09-20T22:56:12.954 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.954 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.954 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:56:12.954 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:43.338585+0000", 2024-09-20T22:56:12.954 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.3649e-05 2024-09-20T22:56:12.954 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.954 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.954 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:56:12.954 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:43.342155+0000", 2024-09-20T22:56:12.954 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0035700509999999999 2024-09-20T22:56:12.954 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.954 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.954 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:56:12.955 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:43.342184+0000", 2024-09-20T22:56:12.955 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.9607999999999998e-05 2024-09-20T22:56:12.955 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:56:12.955 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:56:12.955 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:56:12.955 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.955 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.955 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24415.0:744 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch (1) in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e775)", 2024-09-20T22:56:12.955 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:54:16.547604+0000", 2024-09-20T22:56:12.955 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 116.335938828, 2024-09-20T22:56:12.955 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.79000451299999996, 2024-09-20T22:56:12.955 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:56:12.955 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:56:12.955 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:56:12.956 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:56:12.956 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24415", 2024-09-20T22:56:12.956 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/2056244102", 2024-09-20T22:56:12.956 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 744 2024-09-20T22:56:12.956 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.956 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:56:12.956 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.956 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:56:12.956 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:54:16.547604+0000", 2024-09-20T22:56:12.956 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:56:12.956 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.956 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.956 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:56:12.956 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:54:16.547604+0000", 2024-09-20T22:56:12.957 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:56:12.957 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.957 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.957 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:56:12.957 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:54:16.547607+0000", 2024-09-20T22:56:12.957 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.7460000000000001e-06 2024-09-20T22:56:12.957 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.957 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.957 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:56:12.957 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:54:16.547620+0000", 2024-09-20T22:56:12.957 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3164e-05 2024-09-20T22:56:12.957 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.957 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.957 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:56:12.958 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:54:16.547647+0000", 2024-09-20T22:56:12.958 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.7679e-05 2024-09-20T22:56:12.958 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.958 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.958 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:56:12.958 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:54:16.547663+0000", 2024-09-20T22:56:12.958 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.523e-05 2024-09-20T22:56:12.958 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.958 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.958 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:56:12.958 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:54:16.547788+0000", 2024-09-20T22:56:12.958 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00012490999999999999 2024-09-20T22:56:12.958 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.958 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.959 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-20T22:56:12.959 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:54:16.547801+0000", 2024-09-20T22:56:12.959 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3158e-05 2024-09-20T22:56:12.959 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.959 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.959 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:56:12.959 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:54:17.337429+0000", 2024-09-20T22:56:12.959 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.78962777299999998 2024-09-20T22:56:12.959 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.959 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.959 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:56:12.959 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:54:17.337522+0000", 2024-09-20T22:56:12.959 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.3556999999999999e-05 2024-09-20T22:56:12.959 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.960 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.960 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:56:12.960 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:54:17.337608+0000", 2024-09-20T22:56:12.960 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.6296000000000002e-05 2024-09-20T22:56:12.960 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:56:12.960 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:56:12.960 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:56:12.960 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.960 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.960 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:7834 7.c e786/94 7:3f0fe232:::smithi08477203-4:head v 786'2534, mlcod=785'2528)", 2024-09-20T22:56:12.960 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:54:26.332738+0000", 2024-09-20T22:56:12.960 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 106.550804834, 2024-09-20T22:56:12.960 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.010142463000000001, 2024-09-20T22:56:12.960 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:56:12.961 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:56:12.961 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:56:12.961 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:56:12.961 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.961 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:56:12.961 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:54:26.332738+0000", 2024-09-20T22:56:12.961 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:56:12.961 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.961 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.961 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:56:12.961 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:54:26.332738+0000", 2024-09-20T22:56:12.961 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:56:12.961 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.961 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.962 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:56:12.962 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:54:26.332739+0000", 2024-09-20T22:56:12.962 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2130000000000001e-06 2024-09-20T22:56:12.962 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.962 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.962 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:56:12.962 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:54:26.335206+0000", 2024-09-20T22:56:12.962 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0024668260000000001 2024-09-20T22:56:12.962 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.962 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.962 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:56:12.962 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:54:26.335237+0000", 2024-09-20T22:56:12.962 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.1106999999999999e-05 2024-09-20T22:56:12.963 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.963 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.963 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:56:12.963 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:54:26.335252+0000", 2024-09-20T22:56:12.963 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.5288e-05 2024-09-20T22:56:12.963 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.963 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.963 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:56:12.963 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:54:26.335348+0000", 2024-09-20T22:56:12.963 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.5351000000000002e-05 2024-09-20T22:56:12.963 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.963 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.963 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:56:12.963 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:54:26.335379+0000", 2024-09-20T22:56:12.964 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.1807000000000002e-05 2024-09-20T22:56:12.964 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.964 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.964 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:56:12.964 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:54:26.342798+0000", 2024-09-20T22:56:12.964 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0074187660000000002 2024-09-20T22:56:12.964 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.964 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.964 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:56:12.964 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:54:26.342880+0000", 2024-09-20T22:56:12.964 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.2105000000000001e-05 2024-09-20T22:56:12.964 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:56:12.964 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:56:12.965 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:56:12.965 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.965 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.965 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:8165 7.f e824/391 7:fa55e5ae:::smithi08477203-7:head v 824'927, mlcod=819'923)", 2024-09-20T22:56:12.965 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:55:04.351014+0000", 2024-09-20T22:56:12.965 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 68.532528667999998, 2024-09-20T22:56:12.965 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0096140429999999992, 2024-09-20T22:56:12.965 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:56:12.965 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:56:12.965 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:56:12.965 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:56:12.965 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.965 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:56:12.965 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:04.351014+0000", 2024-09-20T22:56:12.966 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:56:12.966 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.966 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.966 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:56:12.966 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:04.351014+0000", 2024-09-20T22:56:12.966 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:56:12.966 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.966 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.966 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:56:12.966 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:04.351014+0000", 2024-09-20T22:56:12.966 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9999999999999999e-07 2024-09-20T22:56:12.966 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.966 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.966 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:56:12.966 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:04.354512+0000", 2024-09-20T22:56:12.967 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0034980990000000002 2024-09-20T22:56:12.967 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.967 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.967 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:56:12.967 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:04.354544+0000", 2024-09-20T22:56:12.967 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.1294999999999998e-05 2024-09-20T22:56:12.967 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.967 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.967 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:56:12.967 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:04.354557+0000", 2024-09-20T22:56:12.967 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3825999999999999e-05 2024-09-20T22:56:12.967 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.967 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.967 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:56:12.967 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:04.354685+0000", 2024-09-20T22:56:12.968 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00012776000000000001 2024-09-20T22:56:12.968 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.968 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.968 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:56:12.968 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:04.354717+0000", 2024-09-20T22:56:12.968 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.2054000000000002e-05 2024-09-20T22:56:12.968 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.968 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.968 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:56:12.968 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:04.360567+0000", 2024-09-20T22:56:12.968 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0058492709999999996 2024-09-20T22:56:12.968 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.968 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.968 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:56:12.968 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:04.360628+0000", 2024-09-20T22:56:12.968 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.1538000000000006e-05 2024-09-20T22:56:12.969 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:56:12.969 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:56:12.969 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:56:12.969 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.969 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.969 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:8189 7.c e825/94 7:318c7512:::smithi08477203-44:head v 825'2641, mlcod=825'2638)", 2024-09-20T22:56:12.969 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:55:05.581130+0000", 2024-09-20T22:56:12.969 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 67.302412894, 2024-09-20T22:56:12.969 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0077411670000000002, 2024-09-20T22:56:12.969 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:56:12.969 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:56:12.969 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:56:12.969 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:56:12.969 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.969 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:56:12.970 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:05.581130+0000", 2024-09-20T22:56:12.970 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:56:12.970 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.970 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.970 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:56:12.970 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:05.581130+0000", 2024-09-20T22:56:12.970 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:56:12.970 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.970 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.970 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:56:12.970 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:05.581133+0000", 2024-09-20T22:56:12.970 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.2040000000000002e-06 2024-09-20T22:56:12.970 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.970 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.971 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:56:12.971 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:05.582999+0000", 2024-09-20T22:56:12.971 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0018663919999999999 2024-09-20T22:56:12.971 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.971 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.971 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:56:12.971 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:05.583026+0000", 2024-09-20T22:56:12.971 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.6503999999999999e-05 2024-09-20T22:56:12.971 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.971 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.971 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:56:12.971 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:05.583040+0000", 2024-09-20T22:56:12.971 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4430000000000001e-05 2024-09-20T22:56:12.971 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.972 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.972 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:56:12.972 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:05.583132+0000", 2024-09-20T22:56:12.972 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.1939000000000002e-05 2024-09-20T22:56:12.972 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.972 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.972 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:56:12.972 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:05.583163+0000", 2024-09-20T22:56:12.972 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.0341999999999999e-05 2024-09-20T22:56:12.972 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.972 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.972 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:56:12.972 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:05.588810+0000", 2024-09-20T22:56:12.972 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0056476629999999998 2024-09-20T22:56:12.973 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.973 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.973 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:56:12.973 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:05.588871+0000", 2024-09-20T22:56:12.973 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.0693000000000003e-05 2024-09-20T22:56:12.973 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:56:12.973 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:56:12.973 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:56:12.973 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.973 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.973 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:8294 7.5 7:a160b5d4:::smithi08477203-16:head [read 0~3042133,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e831)", 2024-09-20T22:56:12.973 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:55:11.340475+0000", 2024-09-20T22:56:12.973 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 61.543068046999998, 2024-09-20T22:56:12.973 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0075077099999999999, 2024-09-20T22:56:12.974 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:56:12.974 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:56:12.974 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:56:12.974 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:56:12.974 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:56:12.974 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:56:12.974 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 8294 2024-09-20T22:56:12.974 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.974 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:56:12.974 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.974 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:56:12.974 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:11.340475+0000", 2024-09-20T22:56:12.974 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:56:12.975 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.975 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.975 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:56:12.975 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:11.340475+0000", 2024-09-20T22:56:12.975 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:56:12.975 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.975 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.975 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:56:12.975 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:11.340475+0000", 2024-09-20T22:56:12.975 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.1099999999999996e-07 2024-09-20T22:56:12.975 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.975 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.975 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:56:12.976 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:11.340479+0000", 2024-09-20T22:56:12.976 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.4170000000000001e-06 2024-09-20T22:56:12.976 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.976 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.976 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:56:12.976 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:11.340484+0000", 2024-09-20T22:56:12.976 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.7570000000000001e-06 2024-09-20T22:56:12.976 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.976 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.976 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:56:12.976 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:11.340489+0000", 2024-09-20T22:56:12.976 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.3819999999999997e-06 2024-09-20T22:56:12.976 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.976 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.977 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:56:12.977 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:11.340800+0000", 2024-09-20T22:56:12.977 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00031126299999999999 2024-09-20T22:56:12.977 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.977 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.977 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for rw locks", 2024-09-20T22:56:12.977 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:11.340908+0000", 2024-09-20T22:56:12.977 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000108088 2024-09-20T22:56:12.977 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.977 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.977 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:56:12.977 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:11.347505+0000", 2024-09-20T22:56:12.977 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0065973050000000004 2024-09-20T22:56:12.977 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.978 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.978 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:56:12.978 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:11.347565+0000", 2024-09-20T22:56:12.978 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.9567000000000001e-05 2024-09-20T22:56:12.978 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.978 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.978 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:56:12.978 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:11.347982+0000", 2024-09-20T22:56:12.978 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00041742 2024-09-20T22:56:12.978 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:56:12.978 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:56:12.978 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:56:12.978 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.979 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.979 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:8442 7.c e849/94 7:31d0211f:::smithi08477203-25:head v 849'2704, mlcod=845'2700)", 2024-09-20T22:56:12.979 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:55:29.375191+0000", 2024-09-20T22:56:12.979 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 43.508351843, 2024-09-20T22:56:12.979 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.011331713, 2024-09-20T22:56:12.979 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:56:12.979 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:56:12.979 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:56:12.979 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:56:12.979 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.979 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:56:12.979 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:29.375191+0000", 2024-09-20T22:56:12.979 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:56:12.979 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.980 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.980 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:56:12.980 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:29.375191+0000", 2024-09-20T22:56:12.980 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:56:12.980 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.980 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.980 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:56:12.980 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:29.375191+0000", 2024-09-20T22:56:12.980 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.3200000000000001e-07 2024-09-20T22:56:12.980 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.980 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.980 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:56:12.980 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:29.377130+0000", 2024-09-20T22:56:12.981 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0019390779999999999 2024-09-20T22:56:12.981 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.981 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.981 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:56:12.981 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:29.377142+0000", 2024-09-20T22:56:12.981 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1712000000000001e-05 2024-09-20T22:56:12.981 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.981 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.981 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:56:12.981 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:29.377174+0000", 2024-09-20T22:56:12.981 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.1727000000000003e-05 2024-09-20T22:56:12.981 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.981 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.982 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:56:12.982 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:29.377522+0000", 2024-09-20T22:56:12.982 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00034803200000000002 2024-09-20T22:56:12.982 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.982 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.982 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:56:12.982 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:29.377553+0000", 2024-09-20T22:56:12.982 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.1130999999999997e-05 2024-09-20T22:56:12.982 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.982 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.982 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:56:12.982 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:29.386478+0000", 2024-09-20T22:56:12.982 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0089250430000000006 2024-09-20T22:56:12.982 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.982 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.983 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:56:12.983 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:29.386523+0000", 2024-09-20T22:56:12.983 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.4758000000000001e-05 2024-09-20T22:56:12.983 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:56:12.983 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:56:12.983 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:56:12.983 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.983 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.983 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:8804 7.c e883/94 7:318c7512:::smithi08477203-44:head v 883'2842, mlcod=881'2835)", 2024-09-20T22:56:12.983 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:56:03.367132+0000", 2024-09-20T22:56:12.983 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 9.5164103430000004, 2024-09-20T22:56:12.983 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0084714460000000005, 2024-09-20T22:56:12.983 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:56:12.984 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:56:12.984 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:56:12.984 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:56:12.984 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.984 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:56:12.984 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:56:03.367132+0000", 2024-09-20T22:56:12.984 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:56:12.984 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.984 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.984 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:56:12.984 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:56:03.367132+0000", 2024-09-20T22:56:12.984 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:56:12.984 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.984 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.985 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:56:12.985 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:56:03.367133+0000", 2024-09-20T22:56:12.985 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.3099999999999996e-07 2024-09-20T22:56:12.985 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.985 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.985 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:56:12.985 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:56:03.367141+0000", 2024-09-20T22:56:12.985 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.2540000000000003e-06 2024-09-20T22:56:12.985 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.985 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.985 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:56:12.985 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:56:03.367153+0000", 2024-09-20T22:56:12.985 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2534999999999999e-05 2024-09-20T22:56:12.986 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.986 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.986 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:56:12.986 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:56:03.367160+0000", 2024-09-20T22:56:12.986 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.2320000000000004e-06 2024-09-20T22:56:12.986 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.986 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.986 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:56:12.986 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:56:03.367454+0000", 2024-09-20T22:56:12.986 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00029333099999999997 2024-09-20T22:56:12.986 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.986 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.986 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:56:12.986 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:56:03.367480+0000", 2024-09-20T22:56:12.987 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.6781999999999999e-05 2024-09-20T22:56:12.987 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.987 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.987 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:56:12.987 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:56:03.375540+0000", 2024-09-20T22:56:12.987 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0080594840000000004 2024-09-20T22:56:12.987 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.987 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.987 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:56:12.987 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:56:03.375604+0000", 2024-09-20T22:56:12.987 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.3897000000000002e-05 2024-09-20T22:56:12.987 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:56:12.987 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:56:12.987 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:56:12.988 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.988 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.988 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:8832 7.c e884/94 7:31d0211f:::smithi08477203-25:head v 884'2847, mlcod=883'2844)", 2024-09-20T22:56:12.988 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:56:04.509874+0000", 2024-09-20T22:56:12.988 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 8.3736687730000003, 2024-09-20T22:56:12.988 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00504032, 2024-09-20T22:56:12.988 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:56:12.988 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:56:12.988 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:56:12.988 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:56:12.988 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.988 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:56:12.989 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:56:04.509874+0000", 2024-09-20T22:56:12.989 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:56:12.989 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.989 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.989 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:56:12.989 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:56:04.509874+0000", 2024-09-20T22:56:12.989 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:56:12.989 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.989 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.989 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:56:12.989 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:56:04.509874+0000", 2024-09-20T22:56:12.989 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.7300000000000002e-07 2024-09-20T22:56:12.989 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.989 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.990 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:56:12.990 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:56:04.511431+0000", 2024-09-20T22:56:12.990 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001556766 2024-09-20T22:56:12.990 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.990 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.990 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:56:12.990 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:56:04.511454+0000", 2024-09-20T22:56:12.990 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.3269e-05 2024-09-20T22:56:12.990 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.990 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.990 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:56:12.990 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:56:04.511467+0000", 2024-09-20T22:56:12.990 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2877999999999999e-05 2024-09-20T22:56:12.991 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.991 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.991 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:56:12.991 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:56:04.511554+0000", 2024-09-20T22:56:12.991 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.7087000000000002e-05 2024-09-20T22:56:12.991 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.991 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.991 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:56:12.991 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:56:04.511576+0000", 2024-09-20T22:56:12.991 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.1999000000000001e-05 2024-09-20T22:56:12.991 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.991 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.991 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:56:12.992 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:56:04.514874+0000", 2024-09-20T22:56:12.992 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.003297538 2024-09-20T22:56:12.992 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.992 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.992 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:56:12.992 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:56:04.514914+0000", 2024-09-20T22:56:12.992 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.0509999999999997e-05 2024-09-20T22:56:12.992 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:56:12.992 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:56:12.992 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:56:12.992 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.992 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.992 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:8839 7.5 7:a160b5d4:::smithi08477203-16:head [copy-from ver 2842] snapc 19d=[19d,19c,19b,19a,199,198,197,196,195,193,191,190,18f,18e,18d,18c,18a,189,185,183,182,180,17e,17d,17b,17a,179,177,175,174,173,172,16f,16e,16c,16b,160,15c,158,14f,14e,14a,13a,12f] ondisk+write+known_if_redirected+supports_pool_eio e885)", 2024-09-20T22:56:12.992 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:56:05.501335+0000", 2024-09-20T22:56:12.993 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 7.3822080970000004, 2024-09-20T22:56:12.993 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.026669249999999999, 2024-09-20T22:56:12.993 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:56:12.993 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:56:12.993 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:56:12.993 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:56:12.993 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:56:12.993 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:56:12.993 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 8839 2024-09-20T22:56:12.993 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.993 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:56:12.993 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.993 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:56:12.993 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:56:05.501335+0000", 2024-09-20T22:56:12.994 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:56:12.994 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.994 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.994 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:56:12.994 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:56:05.501335+0000", 2024-09-20T22:56:12.994 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:56:12.994 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.994 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.994 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:56:12.994 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:56:05.501336+0000", 2024-09-20T22:56:12.994 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.643e-06 2024-09-20T22:56:12.994 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.995 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.995 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:56:12.995 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:56:05.501343+0000", 2024-09-20T22:56:12.995 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.3470000000000004e-06 2024-09-20T22:56:12.995 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.995 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.995 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:56:12.995 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:56:05.501354+0000", 2024-09-20T22:56:12.995 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1561e-05 2024-09-20T22:56:12.995 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.995 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.995 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:56:12.995 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:56:05.501363+0000", 2024-09-20T22:56:12.995 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.5040000000000002e-06 2024-09-20T22:56:12.995 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.996 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.996 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:56:12.996 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:56:05.501446+0000", 2024-09-20T22:56:12.996 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.3673999999999999e-05 2024-09-20T22:56:12.996 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.996 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.996 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:56:12.996 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:56:05.501608+0000", 2024-09-20T22:56:12.996 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000161399 2024-09-20T22:56:12.996 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.996 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.996 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 0,2", 2024-09-20T22:56:12.996 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:56:05.512738+0000", 2024-09-20T22:56:12.996 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.011130509 2024-09-20T22:56:12.997 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.997 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.997 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:56:12.997 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:56:05.516390+0000", 2024-09-20T22:56:12.997 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0036521320000000002 2024-09-20T22:56:12.997 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.997 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.997 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:56:12.997 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:56:05.527650+0000", 2024-09-20T22:56:12.997 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.011259567 2024-09-20T22:56:12.997 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.997 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.997 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:56:12.998 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:56:05.527894+0000", 2024-09-20T22:56:12.998 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00024384499999999999 2024-09-20T22:56:12.998 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.998 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.998 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:56:12.998 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:56:05.527942+0000", 2024-09-20T22:56:12.998 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.8569000000000002e-05 2024-09-20T22:56:12.998 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.998 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.998 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:56:12.998 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:56:05.528004+0000", 2024-09-20T22:56:12.998 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.1500000000000004e-05 2024-09-20T22:56:12.998 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:56:12.998 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:56:12.999 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:56:12.999 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:12.999 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:12.999 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:8840 7.5 7:a160b5d4:::smithi08477203-16:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e885)", 2024-09-20T22:56:12.999 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:56:05.501431+0000", 2024-09-20T22:56:12.999 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 7.3821121590000001, 2024-09-20T22:56:12.999 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.014446565, 2024-09-20T22:56:12.999 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:56:12.999 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:56:12.999 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:56:12.999 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:56:12.999 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:56:12.999 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:56:12.999 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 8840 2024-09-20T22:56:13.000 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:13.000 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:56:13.000 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:13.000 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:56:13.000 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:56:05.501431+0000", 2024-09-20T22:56:13.000 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:56:13.000 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:13.000 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:13.000 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:56:13.000 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:56:05.501431+0000", 2024-09-20T22:56:13.000 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:56:13.000 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:13.000 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:13.000 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:56:13.001 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:56:05.501431+0000", 2024-09-20T22:56:13.001 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.1600000000000001e-07 2024-09-20T22:56:13.001 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:13.001 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:13.001 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:56:13.001 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:56:05.501434+0000", 2024-09-20T22:56:13.001 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.0699999999999998e-06 2024-09-20T22:56:13.001 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:13.001 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:13.001 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:56:13.001 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:56:05.501441+0000", 2024-09-20T22:56:13.001 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.7830000000000001e-06 2024-09-20T22:56:13.001 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:13.001 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:13.001 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:56:13.002 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:56:05.501447+0000", 2024-09-20T22:56:13.002 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.1630000000000001e-06 2024-09-20T22:56:13.002 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:13.002 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:13.002 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:56:13.002 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:56:05.501833+0000", 2024-09-20T22:56:13.002 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00038532200000000001 2024-09-20T22:56:13.002 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:13.002 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:13.002 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for blocked object", 2024-09-20T22:56:13.002 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:56:05.501902+0000", 2024-09-20T22:56:13.002 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.9587999999999998e-05 2024-09-20T22:56:13.002 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:13.002 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:13.003 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:56:13.003 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:56:05.515543+0000", 2024-09-20T22:56:13.003 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.013640664 2024-09-20T22:56:13.003 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:13.003 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:13.003 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:56:13.003 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:56:05.515736+0000", 2024-09-20T22:56:13.003 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00019273099999999999 2024-09-20T22:56:13.003 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:13.003 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:13.003 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:56:13.003 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:56:05.515877+0000", 2024-09-20T22:56:13.003 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00014162800000000001 2024-09-20T22:56:13.003 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:56:13.004 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:56:13.004 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:56:13.004 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:56:13.004 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:56:13.004 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-20T22:56:13.004 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:56:13.025 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:13.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:13.026 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:56:13.062 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-20T22:56:13.127 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:56:13.148 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:13.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:13.149 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:56:13.250 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:56:13.272 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:13.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:13.273 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:56:13.315 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:56:13.345 INFO:tasks.rados.rados.0.smithi084.stdout:3566: done (3 left) 2024-09-20T22:56:13.345 INFO:tasks.rados.rados.0.smithi084.stdout:3567: read oid 10 snap -1 2024-09-20T22:56:13.345 INFO:tasks.rados.rados.0.smithi084.stdout:3567: expect (ObjNum 1186 snap 408 seq_num 1186) 2024-09-20T22:56:13.345 INFO:tasks.rados.rados.0.smithi084.stdout:3568: read oid 15 snap 379 2024-09-20T22:56:13.345 INFO:tasks.rados.rados.0.smithi084.stdout:3568: expect (ObjNum 1097 snap 368 seq_num 1097) 2024-09-20T22:56:13.345 INFO:tasks.rados.rados.0.smithi084.stdout:3569: snap_remove snap 333 2024-09-20T22:56:13.374 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:56:13.395 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:13.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:13.396 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:56:13.497 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:56:13.518 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:13.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:13.519 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:56:13.620 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:56:13.640 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:13.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:13.641 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:56:13.734 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:13 smithi084 ceph-mon[29774]: pgmap v1239: 158 pgs: 10 remapped+peering, 2 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 142 active+clean; 688 MiB data, 1.8 GiB used, 624 GiB / 626 GiB avail; 4.8 MiB/s wr, 14 op/s; 40 KiB/s, 16 objects/s recovering 2024-09-20T22:56:13.735 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:13 smithi084 ceph-mon[29774]: 2.0 scrub starts 2024-09-20T22:56:13.735 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:13 smithi084 ceph-mon[29774]: 2.0 scrub ok 2024-09-20T22:56:13.735 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:13 smithi084 ceph-mon[29774]: 1.2 scrub starts 2024-09-20T22:56:13.735 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:13 smithi084 ceph-mon[29774]: 1.2 scrub ok 2024-09-20T22:56:13.735 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:13 smithi084 ceph-mon[29774]: 1.6 scrub starts 2024-09-20T22:56:13.735 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:13 smithi084 ceph-mon[29774]: 1.6 scrub ok 2024-09-20T22:56:13.735 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:13 smithi084 ceph-mon[29774]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:56:13.735 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:13 smithi084 ceph-mon[29774]: osdmap e893: 8 total, 8 up, 7 in 2024-09-20T22:56:13.735 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:13 smithi084 ceph-mon[40024]: pgmap v1239: 158 pgs: 10 remapped+peering, 2 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 142 active+clean; 688 MiB data, 1.8 GiB used, 624 GiB / 626 GiB avail; 4.8 MiB/s wr, 14 op/s; 40 KiB/s, 16 objects/s recovering 2024-09-20T22:56:13.735 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:13 smithi084 ceph-mon[40024]: 2.0 scrub starts 2024-09-20T22:56:13.735 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:13 smithi084 ceph-mon[40024]: 2.0 scrub ok 2024-09-20T22:56:13.735 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:13 smithi084 ceph-mon[40024]: 1.2 scrub starts 2024-09-20T22:56:13.736 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:13 smithi084 ceph-mon[40024]: 1.2 scrub ok 2024-09-20T22:56:13.736 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:13 smithi084 ceph-mon[40024]: 1.6 scrub starts 2024-09-20T22:56:13.736 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:13 smithi084 ceph-mon[40024]: 1.6 scrub ok 2024-09-20T22:56:13.736 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:13 smithi084 ceph-mon[40024]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:56:13.736 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:13 smithi084 ceph-mon[40024]: osdmap e893: 8 total, 8 up, 7 in 2024-09-20T22:56:13.742 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:56:13.764 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:13.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:13.765 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:56:13.831 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:13 smithi154 ceph-mon[33264]: pgmap v1239: 158 pgs: 10 remapped+peering, 2 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 142 active+clean; 688 MiB data, 1.8 GiB used, 624 GiB / 626 GiB avail; 4.8 MiB/s wr, 14 op/s; 40 KiB/s, 16 objects/s recovering 2024-09-20T22:56:13.831 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:13 smithi154 ceph-mon[33264]: 2.0 scrub starts 2024-09-20T22:56:13.831 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:13 smithi154 ceph-mon[33264]: 2.0 scrub ok 2024-09-20T22:56:13.831 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:13 smithi154 ceph-mon[33264]: 1.2 scrub starts 2024-09-20T22:56:13.831 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:13 smithi154 ceph-mon[33264]: 1.2 scrub ok 2024-09-20T22:56:13.832 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:13 smithi154 ceph-mon[33264]: 1.6 scrub starts 2024-09-20T22:56:13.832 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:13 smithi154 ceph-mon[33264]: 1.6 scrub ok 2024-09-20T22:56:13.832 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:13 smithi154 ceph-mon[33264]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:56:13.832 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:13 smithi154 ceph-mon[33264]: osdmap e893: 8 total, 8 up, 7 in 2024-09-20T22:56:13.859 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-20T22:56:13.866 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:56:13.888 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:13.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:13.889 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:56:13.926 INFO:tasks.thrashosds.thrasher:in_osds: [2, 3, 5, 6, 0, 4, 1] out_osds: [7] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-20T22:56:13.926 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-20T22:56:13.926 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-09-20T22:56:13.927 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph tell osd.2 'trim stale osdmaps' 2024-09-20T22:56:13.989 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:56:14.010 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:14.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:14.011 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:56:14.112 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:56:14.119 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:56:14.135 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:14.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:14.136 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:56:14.237 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:56:14.258 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:14.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:14.259 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:56:14.352 INFO:tasks.rados.rados.0.smithi084.stdout:3569: done (5 left) 2024-09-20T22:56:14.352 INFO:tasks.rados.rados.0.smithi084.stdout:3570: read oid 28 snap -1 2024-09-20T22:56:14.352 INFO:tasks.rados.rados.0.smithi084.stdout:3570: expect (ObjNum 1198 snap 411 seq_num 1198) 2024-09-20T22:56:14.352 INFO:tasks.rados.rados.0.smithi084.stdout:3571: write (excl) oid 5 current snap is 415 2024-09-20T22:56:14.352 INFO:tasks.rados.rados.0.smithi084.stdout:3571: seq_num 1220 ranges {504540=420260,1574973=704087} 2024-09-20T22:56:14.355 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is unset 2024-09-20T22:56:14.356 INFO:tasks.rados.rados.0.smithi084.stdout:3571: writing smithi08477203-5 from 504540 to 924800 tid 1 2024-09-20T22:56:14.359 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:56:14.366 INFO:tasks.rados.rados.0.smithi084.stdout:3571: writing smithi08477203-5 from 1574973 to 2279060 tid 2 2024-09-20T22:56:14.366 INFO:tasks.rados.rados.0.smithi084.stdout:3562: finishing write tid 1 to smithi08477203-39 2024-09-20T22:56:14.366 INFO:tasks.rados.rados.0.smithi084.stdout:3562: finishing write tid 2 to smithi08477203-39 2024-09-20T22:56:14.367 INFO:tasks.rados.rados.0.smithi084.stdout:3562: finishing write tid 3 to smithi08477203-39 2024-09-20T22:56:14.367 INFO:tasks.rados.rados.0.smithi084.stdout:3562: finishing write tid 4 to smithi08477203-39 2024-09-20T22:56:14.367 INFO:tasks.rados.rados.0.smithi084.stdout:3562: oid 39 updating version 0 to 2658 2024-09-20T22:56:14.367 INFO:tasks.rados.rados.0.smithi084.stdout:3562: oid 39 version 2658 is already newer than 2656 2024-09-20T22:56:14.367 INFO:tasks.rados.rados.0.smithi084.stdout:3562: oid 39 version 2658 is already newer than 2657 2024-09-20T22:56:14.367 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 39 v 2658 (ObjNum 1218 snap 415 seq_num 1218) dirty exists 2024-09-20T22:56:14.367 INFO:tasks.rados.rados.0.smithi084.stdout:3562: left oid 39 (ObjNum 1218 snap 415 seq_num 1218) 2024-09-20T22:56:14.367 INFO:tasks.rados.rados.0.smithi084.stdout:3563: finishing write tid 1 to smithi08477203-31 2024-09-20T22:56:14.367 INFO:tasks.rados.rados.0.smithi084.stdout:3563: finishing write tid 2 to smithi08477203-31 2024-09-20T22:56:14.367 INFO:tasks.rados.rados.0.smithi084.stdout:3563: finishing write tid 3 to smithi08477203-31 2024-09-20T22:56:14.367 INFO:tasks.rados.rados.0.smithi084.stdout:3564: expect (ObjNum 1215 snap 413 seq_num 1215) 2024-09-20T22:56:14.393 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:14.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:14.394 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:56:14.465 INFO:tasks.rados.rados.0.smithi084.stdout:3563: finishing write tid 4 to smithi08477203-31 2024-09-20T22:56:14.465 INFO:tasks.rados.rados.0.smithi084.stdout:3563: oid 31 updating version 0 to 2494 2024-09-20T22:56:14.465 INFO:tasks.rados.rados.0.smithi084.stdout:3563: oid 31 updating version 2494 to 2495 2024-09-20T22:56:14.465 INFO:tasks.rados.rados.0.smithi084.stdout:3563: oid 31 updating version 2495 to 2496 2024-09-20T22:56:14.465 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 31 v 2496 (ObjNum 1219 snap 415 seq_num 1219) dirty exists 2024-09-20T22:56:14.465 INFO:tasks.rados.rados.0.smithi084.stdout:3563: left oid 31 (ObjNum 1219 snap 415 seq_num 1219) 2024-09-20T22:56:14.465 INFO:tasks.rados.rados.0.smithi084.stdout:3567: expect (ObjNum 1186 snap 408 seq_num 1186) 2024-09-20T22:56:14.493 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-20T22:56:14.495 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:56:14.515 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:14.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:14.516 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:56:14.593 INFO:tasks.rados.rados.0.smithi084.stdout:3568: expect (ObjNum 1097 snap 368 seq_num 1097) 2024-09-20T22:56:14.617 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:56:14.640 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:14.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:14.641 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:56:14.678 INFO:tasks.rados.rados.0.smithi084.stdout:3570: expect (ObjNum 1198 snap 411 seq_num 1198) 2024-09-20T22:56:14.742 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:56:14.764 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:14.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:14.765 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:56:14.765 INFO:tasks.rados.rados.0.smithi084.stdout:3571: finishing write tid 1 to smithi08477203-5 2024-09-20T22:56:14.766 INFO:tasks.rados.rados.0.smithi084.stdout:3571: finishing write tid 2 to smithi08477203-5 2024-09-20T22:56:14.766 INFO:tasks.rados.rados.0.smithi084.stdout:3571: finishing write tid 3 to smithi08477203-5 2024-09-20T22:56:14.766 INFO:tasks.rados.rados.0.smithi084.stdout:3571: finishing write tid 4 to smithi08477203-5 2024-09-20T22:56:14.766 INFO:tasks.rados.rados.0.smithi084.stdout:3571: oid 5 updating version 0 to 2654 2024-09-20T22:56:14.766 INFO:tasks.rados.rados.0.smithi084.stdout:3571: oid 5 version 2654 is already newer than 2653 2024-09-20T22:56:14.766 INFO:tasks.rados.rados.0.smithi084.stdout:3571: oid 5 updating version 2654 to 2655 2024-09-20T22:56:14.766 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 5 v 2655 (ObjNum 1220 snap 415 seq_num 1220) dirty exists 2024-09-20T22:56:14.766 INFO:tasks.rados.rados.0.smithi084.stdout:3571: left oid 5 (ObjNum 1220 snap 415 seq_num 1220) 2024-09-20T22:56:14.766 INFO:tasks.rados.rados.0.smithi084.stdout:3562: done (6 left) 2024-09-20T22:56:14.766 INFO:tasks.rados.rados.0.smithi084.stdout:3563: done (5 left) 2024-09-20T22:56:14.766 INFO:tasks.rados.rados.0.smithi084.stdout:3564: done (4 left) 2024-09-20T22:56:14.766 INFO:tasks.rados.rados.0.smithi084.stdout:3567: done (3 left) 2024-09-20T22:56:14.766 INFO:tasks.rados.rados.0.smithi084.stdout:3568: done (2 left) 2024-09-20T22:56:14.766 INFO:tasks.rados.rados.0.smithi084.stdout:3570: done (1 left) 2024-09-20T22:56:14.767 INFO:tasks.rados.rados.0.smithi084.stdout:3571: done (0 left) 2024-09-20T22:56:14.767 INFO:tasks.rados.rados.0.smithi084.stdout:3572: snap_remove snap 373 2024-09-20T22:56:14.849 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:14 smithi084 ceph-mon[40024]: 5.5 scrub starts 2024-09-20T22:56:14.849 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:14 smithi084 ceph-mon[40024]: 5.5 scrub ok 2024-09-20T22:56:14.849 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:14 smithi084 ceph-mon[40024]: 3.1c scrub starts 2024-09-20T22:56:14.849 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:14 smithi084 ceph-mon[40024]: 3.1c scrub ok 2024-09-20T22:56:14.849 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:14 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/4063224411' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:56:14.850 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:14 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:56:14.850 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:14 smithi084 ceph-mon[40024]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T22:56:14.850 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:14 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T22:56:14.850 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:14 smithi084 ceph-mon[40024]: osdmap e894: 8 total, 8 up, 7 in 2024-09-20T22:56:14.850 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:14 smithi084 ceph-mon[29774]: 5.5 scrub starts 2024-09-20T22:56:14.850 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:14 smithi084 ceph-mon[29774]: 5.5 scrub ok 2024-09-20T22:56:14.850 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:14 smithi084 ceph-mon[29774]: 3.1c scrub starts 2024-09-20T22:56:14.850 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:14 smithi084 ceph-mon[29774]: 3.1c scrub ok 2024-09-20T22:56:14.850 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:14 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/4063224411' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:56:14.850 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:14 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:56:14.850 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:14 smithi084 ceph-mon[29774]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T22:56:14.851 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:14 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T22:56:14.851 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:14 smithi084 ceph-mon[29774]: osdmap e894: 8 total, 8 up, 7 in 2024-09-20T22:56:14.857 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-20T22:56:14.857 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2024-09-20T22:56:14.857 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2024-09-20T22:56:14.857 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-20T22:56:14.866 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:56:14.887 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:14.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:14.888 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:56:14.989 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:56:15.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:14 smithi154 ceph-mon[33264]: 5.5 scrub starts 2024-09-20T22:56:15.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:14 smithi154 ceph-mon[33264]: 5.5 scrub ok 2024-09-20T22:56:15.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:14 smithi154 ceph-mon[33264]: 3.1c scrub starts 2024-09-20T22:56:15.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:14 smithi154 ceph-mon[33264]: 3.1c scrub ok 2024-09-20T22:56:15.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:14 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/4063224411' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:56:15.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:14 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:56:15.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:14 smithi154 ceph-mon[33264]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T22:56:15.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:14 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T22:56:15.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:14 smithi154 ceph-mon[33264]: osdmap e894: 8 total, 8 up, 7 in 2024-09-20T22:56:15.012 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:15.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:15.013 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:56:15.114 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:56:15.136 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:15.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:15.137 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:56:15.237 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:56:15.259 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:15.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:15.260 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:56:15.361 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:56:15.384 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:15.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:15.385 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:56:15.483 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:56:15.486 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:56:15.510 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:15.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:15.511 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:56:15.540 INFO:tasks.rados.rados.0.smithi084.stdout:3572: done (0 left) 2024-09-20T22:56:15.541 INFO:tasks.rados.rados.0.smithi084.stdout:3573: read oid 17 snap 391 2024-09-20T22:56:15.541 INFO:tasks.rados.rados.0.smithi084.stdout:3573: expect (ObjNum 1126 snap 381 seq_num 1126) 2024-09-20T22:56:15.541 INFO:tasks.rados.rados.0.smithi084.stdout:3574: snap_create 2024-09-20T22:56:15.552 INFO:tasks.rados.rados.0.smithi084.stdout:3573: expect (ObjNum 1126 snap 381 seq_num 1126) 2024-09-20T22:56:15.612 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:56:15.635 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:15.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:15.636 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:56:15.737 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:56:15.757 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:15.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:15.758 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:56:15.859 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:56:15.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:15 smithi084 ceph-mon[40024]: pgmap v1242: 158 pgs: 2 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 1 active+remapped+backfill_toofull, 151 active+clean; 695 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 8.9 MiB/s rd, 4 op/s; 51 MiB/s, 72 objects/s recovering 2024-09-20T22:56:15.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:15 smithi084 ceph-mon[40024]: 6.19 deep-scrub starts 2024-09-20T22:56:15.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:15 smithi084 ceph-mon[40024]: 6.19 deep-scrub ok 2024-09-20T22:56:15.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:15 smithi084 ceph-mon[40024]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-09-20T22:56:15.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:15 smithi084 ceph-mon[40024]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 3 pgs peering) 2024-09-20T22:56:15.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:15 smithi084 ceph-mon[40024]: purged_snaps scrub starts 2024-09-20T22:56:15.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:15 smithi084 ceph-mon[40024]: purged_snaps scrub ok 2024-09-20T22:56:15.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:15 smithi084 ceph-mon[40024]: 4.18 scrub starts 2024-09-20T22:56:15.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:15 smithi084 ceph-mon[40024]: 4.18 scrub ok 2024-09-20T22:56:15.866 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:15 smithi084 ceph-mon[29774]: pgmap v1242: 158 pgs: 2 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 1 active+remapped+backfill_toofull, 151 active+clean; 695 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 8.9 MiB/s rd, 4 op/s; 51 MiB/s, 72 objects/s recovering 2024-09-20T22:56:15.866 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:15 smithi084 ceph-mon[29774]: 6.19 deep-scrub starts 2024-09-20T22:56:15.866 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:15 smithi084 ceph-mon[29774]: 6.19 deep-scrub ok 2024-09-20T22:56:15.866 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:15 smithi084 ceph-mon[29774]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-09-20T22:56:15.866 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:15 smithi084 ceph-mon[29774]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 3 pgs peering) 2024-09-20T22:56:15.866 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:15 smithi084 ceph-mon[29774]: purged_snaps scrub starts 2024-09-20T22:56:15.866 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:15 smithi084 ceph-mon[29774]: purged_snaps scrub ok 2024-09-20T22:56:15.866 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:15 smithi084 ceph-mon[29774]: 4.18 scrub starts 2024-09-20T22:56:15.867 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:15 smithi084 ceph-mon[29774]: 4.18 scrub ok 2024-09-20T22:56:15.882 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:15.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:15.883 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:56:15.950 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.0 dump_blocked_ops 2024-09-20T22:56:15.983 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:56:16.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:15 smithi154 ceph-mon[33264]: pgmap v1242: 158 pgs: 2 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 1 active+remapped+backfill_toofull, 151 active+clean; 695 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 8.9 MiB/s rd, 4 op/s; 51 MiB/s, 72 objects/s recovering 2024-09-20T22:56:16.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:15 smithi154 ceph-mon[33264]: 6.19 deep-scrub starts 2024-09-20T22:56:16.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:15 smithi154 ceph-mon[33264]: 6.19 deep-scrub ok 2024-09-20T22:56:16.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:15 smithi154 ceph-mon[33264]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-09-20T22:56:16.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:15 smithi154 ceph-mon[33264]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 3 pgs peering) 2024-09-20T22:56:16.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:15 smithi154 ceph-mon[33264]: purged_snaps scrub starts 2024-09-20T22:56:16.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:15 smithi154 ceph-mon[33264]: purged_snaps scrub ok 2024-09-20T22:56:16.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:15 smithi154 ceph-mon[33264]: 4.18 scrub starts 2024-09-20T22:56:16.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:15 smithi154 ceph-mon[33264]: 4.18 scrub ok 2024-09-20T22:56:16.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:16.006 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:16.006 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:56:16.107 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:56:16.130 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:16.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:16.131 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:56:16.164 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:56:16.232 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:56:16.254 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:16.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:16.255 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:56:16.356 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:56:16.379 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:16.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:16.380 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:56:16.481 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:56:16.504 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:16.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:16.505 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:56:16.555 INFO:tasks.rados.rados.0.smithi084.stdout:3573: done (1 left) 2024-09-20T22:56:16.555 INFO:tasks.rados.rados.0.smithi084.stdout:3574: done (0 left) 2024-09-20T22:56:16.555 INFO:tasks.rados.rados.0.smithi084.stdout:3575: delete oid 43 current snap is 416 2024-09-20T22:56:16.556 INFO:tasks.rados.rados.0.smithi084.stdout:3575: done (0 left) 2024-09-20T22:56:16.556 INFO:tasks.rados.rados.0.smithi084.stdout:3576: read oid 47 snap -1 2024-09-20T22:56:16.556 INFO:tasks.rados.rados.0.smithi084.stdout:3576: expect deleted 2024-09-20T22:56:16.556 INFO:tasks.rados.rados.0.smithi084.stdout:3577: snap_create 2024-09-20T22:56:16.606 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:56:16.628 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:16.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:16.629 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:56:16.694 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:56:16.694 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:56:16.710 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:56:16.710 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:56:16.711 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:16 smithi084 ceph-mon[40024]: 3.7 scrub starts 2024-09-20T22:56:16.711 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:16 smithi084 ceph-mon[40024]: 3.7 scrub ok 2024-09-20T22:56:16.711 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:16 smithi084 ceph-mon[40024]: 4.15 scrub starts 2024-09-20T22:56:16.711 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:16 smithi084 ceph-mon[40024]: 4.15 scrub ok 2024-09-20T22:56:16.711 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:16 smithi084 ceph-mon[40024]: 3.19 deep-scrub starts 2024-09-20T22:56:16.711 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:16 smithi084 ceph-mon[40024]: 3.19 deep-scrub ok 2024-09-20T22:56:16.711 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:16 smithi084 ceph-mon[40024]: osdmap e895: 8 total, 8 up, 7 in 2024-09-20T22:56:16.711 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:16 smithi084 ceph-mon[29774]: 3.7 scrub starts 2024-09-20T22:56:16.711 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:16 smithi084 ceph-mon[29774]: 3.7 scrub ok 2024-09-20T22:56:16.711 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:16 smithi084 ceph-mon[29774]: 4.15 scrub starts 2024-09-20T22:56:16.712 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:16 smithi084 ceph-mon[29774]: 4.15 scrub ok 2024-09-20T22:56:16.712 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:16 smithi084 ceph-mon[29774]: 3.19 deep-scrub starts 2024-09-20T22:56:16.712 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:16 smithi084 ceph-mon[29774]: 3.19 deep-scrub ok 2024-09-20T22:56:16.712 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:16 smithi084 ceph-mon[29774]: osdmap e895: 8 total, 8 up, 7 in 2024-09-20T22:56:16.726 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:56:16.726 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:56:16.729 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:56:16.743 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-20T22:56:16.743 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T22:56:16.753 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:16.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:16.754 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:56:16.764 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:56:16.764 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:56:16.781 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:56:16.781 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:56:16.798 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:56:16.798 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:56:16.814 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:56:16.814 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:56:16.855 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:56:16.878 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:16.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:16.879 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:56:16.960 INFO:teuthology.orchestra.run.smithi084.stderr:no valid command found; 10 closest matches: 2024-09-20T22:56:16.960 INFO:teuthology.orchestra.run.smithi084.stderr:0 2024-09-20T22:56:16.960 INFO:teuthology.orchestra.run.smithi084.stderr:1 2024-09-20T22:56:16.960 INFO:teuthology.orchestra.run.smithi084.stderr:2 2024-09-20T22:56:16.960 INFO:teuthology.orchestra.run.smithi084.stderr:abort 2024-09-20T22:56:16.960 INFO:teuthology.orchestra.run.smithi084.stderr:assert 2024-09-20T22:56:16.960 INFO:teuthology.orchestra.run.smithi084.stderr:bench [] [] [] [] 2024-09-20T22:56:16.961 INFO:teuthology.orchestra.run.smithi084.stderr:bluefs debug_inject_read_zeros 2024-09-20T22:56:16.961 INFO:teuthology.orchestra.run.smithi084.stderr:bluefs files list 2024-09-20T22:56:16.961 INFO:teuthology.orchestra.run.smithi084.stderr:bluefs stats 2024-09-20T22:56:16.961 INFO:teuthology.orchestra.run.smithi084.stderr:bluestore allocator dump block 2024-09-20T22:56:16.961 INFO:teuthology.orchestra.run.smithi084.stderr:Error EINVAL: invalid command 2024-09-20T22:56:16.979 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:56:17.000 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:17.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:17.001 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:56:17.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:16 smithi154 ceph-mon[33264]: 3.7 scrub starts 2024-09-20T22:56:17.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:16 smithi154 ceph-mon[33264]: 3.7 scrub ok 2024-09-20T22:56:17.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:16 smithi154 ceph-mon[33264]: 4.15 scrub starts 2024-09-20T22:56:17.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:16 smithi154 ceph-mon[33264]: 4.15 scrub ok 2024-09-20T22:56:17.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:16 smithi154 ceph-mon[33264]: 3.19 deep-scrub starts 2024-09-20T22:56:17.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:16 smithi154 ceph-mon[33264]: 3.19 deep-scrub ok 2024-09-20T22:56:17.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:16 smithi154 ceph-mon[33264]: osdmap e895: 8 total, 8 up, 7 in 2024-09-20T22:56:17.102 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:56:17.125 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:17.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:17.126 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:56:17.227 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:56:17.250 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:17.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:17.251 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:56:17.351 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:56:17.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:17.375 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:17.375 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:56:17.476 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:56:17.497 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:17.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:17.498 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:56:17.519 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd set noscrub 2024-09-20T22:56:17.556 INFO:tasks.rados.rados.0.smithi084.stdout:3576: done (1 left) 2024-09-20T22:56:17.556 INFO:tasks.rados.rados.0.smithi084.stdout:3577: done (0 left) 2024-09-20T22:56:17.556 INFO:tasks.rados.rados.0.smithi084.stdout:3578: write (excl) oid 1 current snap is 417 2024-09-20T22:56:17.556 INFO:tasks.rados.rados.0.smithi084.stdout:3578: seq_num 1222 ranges {538154=570758,1691937=707756} 2024-09-20T22:56:17.562 INFO:tasks.rados.rados.0.smithi084.stdout:3578: writing smithi08477203-1 from 538154 to 1108912 tid 1 2024-09-20T22:56:17.570 INFO:tasks.rados.rados.0.smithi084.stdout:3578: writing smithi08477203-1 from 1691937 to 2399693 tid 2 2024-09-20T22:56:17.570 INFO:tasks.rados.rados.0.smithi084.stdout:3579: write (excl) oid 29 current snap is 417 2024-09-20T22:56:17.570 INFO:tasks.rados.rados.0.smithi084.stdout:3579: seq_num 1223 ranges {554961=646007,1750419=451414} 2024-09-20T22:56:17.577 INFO:tasks.rados.rados.0.smithi084.stdout:3579: writing smithi08477203-29 from 554961 to 1200968 tid 1 2024-09-20T22:56:17.583 INFO:tasks.rados.rados.0.smithi084.stdout:3579: writing smithi08477203-29 from 1750419 to 2201833 tid 2 2024-09-20T22:56:17.584 INFO:tasks.rados.rados.0.smithi084.stdout:3580: snap_remove snap 392 2024-09-20T22:56:17.599 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:56:17.645 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:17.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:17.646 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:56:17.747 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:56:17.768 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:17.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:17.769 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:56:17.784 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:56:17.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:17 smithi084 ceph-mon[40024]: pgmap v1245: 158 pgs: 2 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 1 active+remapped+backfill_toofull, 151 active+clean; 695 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 8.9 MiB/s rd, 4 op/s; 51 MiB/s, 72 objects/s recovering 2024-09-20T22:56:17.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:17 smithi084 ceph-mon[40024]: 3.12 scrub starts 2024-09-20T22:56:17.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:17 smithi084 ceph-mon[40024]: 3.12 scrub ok 2024-09-20T22:56:17.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:17 smithi084 ceph-mon[40024]: osdmap e896: 8 total, 8 up, 7 in 2024-09-20T22:56:17.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:17 smithi084 ceph-mon[40024]: purged_snaps scrub starts 2024-09-20T22:56:17.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:17 smithi084 ceph-mon[40024]: purged_snaps scrub ok 2024-09-20T22:56:17.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:17 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/2160026843' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:56:17.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:17 smithi084 ceph-mon[29774]: pgmap v1245: 158 pgs: 2 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 1 active+remapped+backfill_toofull, 151 active+clean; 695 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 8.9 MiB/s rd, 4 op/s; 51 MiB/s, 72 objects/s recovering 2024-09-20T22:56:17.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:17 smithi084 ceph-mon[29774]: 3.12 scrub starts 2024-09-20T22:56:17.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:17 smithi084 ceph-mon[29774]: 3.12 scrub ok 2024-09-20T22:56:17.866 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:17 smithi084 ceph-mon[29774]: osdmap e896: 8 total, 8 up, 7 in 2024-09-20T22:56:17.866 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:17 smithi084 ceph-mon[29774]: purged_snaps scrub starts 2024-09-20T22:56:17.866 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:17 smithi084 ceph-mon[29774]: purged_snaps scrub ok 2024-09-20T22:56:17.866 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:17 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/2160026843' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:56:17.870 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:56:17.893 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:17.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:17.894 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:56:17.995 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:56:18.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:17 smithi154 ceph-mon[33264]: pgmap v1245: 158 pgs: 2 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 1 active+remapped+backfill_toofull, 151 active+clean; 695 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 8.9 MiB/s rd, 4 op/s; 51 MiB/s, 72 objects/s recovering 2024-09-20T22:56:18.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:17 smithi154 ceph-mon[33264]: 3.12 scrub starts 2024-09-20T22:56:18.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:17 smithi154 ceph-mon[33264]: 3.12 scrub ok 2024-09-20T22:56:18.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:17 smithi154 ceph-mon[33264]: osdmap e896: 8 total, 8 up, 7 in 2024-09-20T22:56:18.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:17 smithi154 ceph-mon[33264]: purged_snaps scrub starts 2024-09-20T22:56:18.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:17 smithi154 ceph-mon[33264]: purged_snaps scrub ok 2024-09-20T22:56:18.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:17 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/2160026843' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:56:18.016 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:18.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:18.017 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:56:18.118 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:56:18.140 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:18.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:18.141 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:56:18.191 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-20T22:56:18.191 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-20T22:56:18.192 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph tell osd.3 'trim stale osdmaps' 2024-09-20T22:56:18.242 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:56:18.263 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:18.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:18.264 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:56:18.352 INFO:tasks.rados.rados.0.smithi084.stdout:3580: done (2 left) 2024-09-20T22:56:18.352 INFO:tasks.rados.rados.0.smithi084.stdout:3581: snap_remove snap 375 2024-09-20T22:56:18.365 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:56:18.388 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:18.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:18.389 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:56:18.469 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:56:18 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: ::ffff:172.21.15.154 - - [20/Sep/2024:22:56:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-20T22:56:18.489 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:56:18.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:18.513 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:18.513 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:56:18.614 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:56:18.636 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:18.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:18.637 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:56:18.719 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:18 smithi084 ceph-mon[40024]: 3.9 scrub starts 2024-09-20T22:56:18.719 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:18 smithi084 ceph-mon[40024]: 3.9 scrub ok 2024-09-20T22:56:18.719 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:18 smithi084 ceph-mon[40024]: 4.4 scrub starts 2024-09-20T22:56:18.719 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:18 smithi084 ceph-mon[40024]: 4.4 scrub ok 2024-09-20T22:56:18.720 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:18 smithi084 ceph-mon[40024]: osdmap e897: 8 total, 8 up, 7 in 2024-09-20T22:56:18.720 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:18 smithi084 ceph-mon[40024]: 3.10 scrub starts 2024-09-20T22:56:18.720 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:18 smithi084 ceph-mon[40024]: 3.10 scrub ok 2024-09-20T22:56:18.720 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:18 smithi084 ceph-mon[40024]: osdmap e898: 8 total, 8 up, 7 in 2024-09-20T22:56:18.720 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:18 smithi084 ceph-mon[40024]: 5.1c scrub starts 2024-09-20T22:56:18.720 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:18 smithi084 ceph-mon[40024]: 5.1c scrub ok 2024-09-20T22:56:18.720 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:18 smithi084 ceph-mon[29774]: 3.9 scrub starts 2024-09-20T22:56:18.720 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:18 smithi084 ceph-mon[29774]: 3.9 scrub ok 2024-09-20T22:56:18.720 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:18 smithi084 ceph-mon[29774]: 4.4 scrub starts 2024-09-20T22:56:18.721 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:18 smithi084 ceph-mon[29774]: 4.4 scrub ok 2024-09-20T22:56:18.721 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:18 smithi084 ceph-mon[29774]: osdmap e897: 8 total, 8 up, 7 in 2024-09-20T22:56:18.721 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:18 smithi084 ceph-mon[29774]: 3.10 scrub starts 2024-09-20T22:56:18.721 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:18 smithi084 ceph-mon[29774]: 3.10 scrub ok 2024-09-20T22:56:18.721 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:18 smithi084 ceph-mon[29774]: osdmap e898: 8 total, 8 up, 7 in 2024-09-20T22:56:18.721 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:18 smithi084 ceph-mon[29774]: 5.1c scrub starts 2024-09-20T22:56:18.721 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:18 smithi084 ceph-mon[29774]: 5.1c scrub ok 2024-09-20T22:56:18.737 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:56:18.747 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-20T22:56:18.747 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:56:18.758 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:18.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:18.759 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:56:18.860 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:56:18.884 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:18.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:18.885 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:56:18.977 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:56:18.982 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:56:18.985 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:56:19.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:18 smithi154 ceph-mon[33264]: 3.9 scrub starts 2024-09-20T22:56:19.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:18 smithi154 ceph-mon[33264]: 3.9 scrub ok 2024-09-20T22:56:19.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:18 smithi154 ceph-mon[33264]: 4.4 scrub starts 2024-09-20T22:56:19.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:18 smithi154 ceph-mon[33264]: 4.4 scrub ok 2024-09-20T22:56:19.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:18 smithi154 ceph-mon[33264]: osdmap e897: 8 total, 8 up, 7 in 2024-09-20T22:56:19.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:18 smithi154 ceph-mon[33264]: 3.10 scrub starts 2024-09-20T22:56:19.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:18 smithi154 ceph-mon[33264]: 3.10 scrub ok 2024-09-20T22:56:19.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:18 smithi154 ceph-mon[33264]: osdmap e898: 8 total, 8 up, 7 in 2024-09-20T22:56:19.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:18 smithi154 ceph-mon[33264]: 5.1c scrub starts 2024-09-20T22:56:19.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:18 smithi154 ceph-mon[33264]: 5.1c scrub ok 2024-09-20T22:56:19.009 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:19.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:19.010 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:56:19.110 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:56:19.132 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:19.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:19.133 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:56:19.234 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:56:19.254 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:19.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:19.255 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:56:19.353 INFO:tasks.rados.rados.0.smithi084.stdout:3581: done (2 left) 2024-09-20T22:56:19.353 INFO:tasks.rados.rados.0.smithi084.stdout:3582: read oid 42 snap -1 2024-09-20T22:56:19.353 INFO:tasks.rados.rados.0.smithi084.stdout:3582: expect deleted 2024-09-20T22:56:19.353 INFO:tasks.rados.rados.0.smithi084.stdout:3583: write oid 32 current snap is 417 2024-09-20T22:56:19.353 INFO:tasks.rados.rados.0.smithi084.stdout:3583: seq_num 1224 ranges {571768=437609,1808901=711425} 2024-09-20T22:56:19.357 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:56:19.359 INFO:tasks.rados.rados.0.smithi084.stdout:3583: writing smithi08477203-32 from 571768 to 1009377 tid 1 2024-09-20T22:56:19.370 INFO:tasks.rados.rados.0.smithi084.stdout:3583: writing smithi08477203-32 from 1808901 to 2520326 tid 2 2024-09-20T22:56:19.370 INFO:tasks.rados.rados.0.smithi084.stdout:3584: snap_create 2024-09-20T22:56:19.370 INFO:tasks.rados.rados.0.smithi084.stdout:3578: finishing write tid 1 to smithi08477203-1 2024-09-20T22:56:19.370 INFO:tasks.rados.rados.0.smithi084.stdout:3578: finishing write tid 2 to smithi08477203-1 2024-09-20T22:56:19.370 INFO:tasks.rados.rados.0.smithi084.stdout:3578: finishing write tid 3 to smithi08477203-1 2024-09-20T22:56:19.370 INFO:tasks.rados.rados.0.smithi084.stdout:3578: finishing write tid 4 to smithi08477203-1 2024-09-20T22:56:19.370 INFO:tasks.rados.rados.0.smithi084.stdout:3578: oid 1 updating version 0 to 2659 2024-09-20T22:56:19.370 INFO:tasks.rados.rados.0.smithi084.stdout:3578: oid 1 updating version 2659 to 2661 2024-09-20T22:56:19.370 INFO:tasks.rados.rados.0.smithi084.stdout:3578: oid 1 version 2661 is already newer than 2660 2024-09-20T22:56:19.370 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 1 v 2661 (ObjNum 1222 snap 417 seq_num 1222) dirty exists 2024-09-20T22:56:19.370 INFO:tasks.rados.rados.0.smithi084.stdout:3578: left oid 1 (ObjNum 1222 snap 417 seq_num 1222) 2024-09-20T22:56:19.371 INFO:tasks.rados.rados.0.smithi084.stdout:3579: finishing write tid 1 to smithi08477203-29 2024-09-20T22:56:19.371 INFO:tasks.rados.rados.0.smithi084.stdout:3579: finishing write tid 2 to smithi08477203-29 2024-09-20T22:56:19.371 INFO:tasks.rados.rados.0.smithi084.stdout:3579: finishing write tid 3 to smithi08477203-29 2024-09-20T22:56:19.371 INFO:tasks.rados.rados.0.smithi084.stdout:3579: finishing write tid 4 to smithi08477203-29 2024-09-20T22:56:19.371 INFO:tasks.rados.rados.0.smithi084.stdout:3579: oid 29 updating version 0 to 2905 2024-09-20T22:56:19.371 INFO:tasks.rados.rados.0.smithi084.stdout:3579: oid 29 updating version 2905 to 2906 2024-09-20T22:56:19.371 INFO:tasks.rados.rados.0.smithi084.stdout:3579: oid 29 updating version 2906 to 2907 2024-09-20T22:56:19.371 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 29 v 2907 (ObjNum 1223 snap 417 seq_num 1223) dirty exists 2024-09-20T22:56:19.371 INFO:tasks.rados.rados.0.smithi084.stdout:3579: left oid 29 (ObjNum 1223 snap 417 seq_num 1223) 2024-09-20T22:56:19.371 INFO:tasks.rados.rados.0.smithi084.stdout:3583: finishing write tid 1 to smithi08477203-32 2024-09-20T22:56:19.373 INFO:tasks.rados.rados.0.smithi084.stdout:3583: finishing write tid 2 to smithi08477203-32 2024-09-20T22:56:19.373 INFO:tasks.rados.rados.0.smithi084.stdout:3583: finishing write tid 3 to smithi08477203-32 2024-09-20T22:56:19.373 INFO:tasks.rados.rados.0.smithi084.stdout:3583: finishing write tid 4 to smithi08477203-32 2024-09-20T22:56:19.373 INFO:tasks.rados.rados.0.smithi084.stdout:3583: oid 32 updating version 0 to 1852 2024-09-20T22:56:19.373 INFO:tasks.rados.rados.0.smithi084.stdout:3583: oid 32 version 1852 is already newer than 1851 2024-09-20T22:56:19.374 INFO:tasks.rados.rados.0.smithi084.stdout:3583: oid 32 version 1852 is already newer than 1850 2024-09-20T22:56:19.374 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 32 v 1852 (ObjNum 1224 snap 417 seq_num 1224) dirty exists 2024-09-20T22:56:19.374 INFO:tasks.rados.rados.0.smithi084.stdout:3583: left oid 32 (ObjNum 1224 snap 417 seq_num 1224) 2024-09-20T22:56:19.384 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:19.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:19.385 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:56:19.486 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:56:19.500 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.0 dump_historic_ops 2024-09-20T22:56:19.510 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:19.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:19.511 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:56:19.585 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-20T22:56:19.612 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:56:19.651 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:19.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:19.652 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:56:19.753 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:56:19.774 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:19.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:19.775 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:56:19.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:19 smithi084 ceph-mon[40024]: pgmap v1248: 158 pgs: 2 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 1 active+remapped+backfill_toofull, 151 active+clean; 695 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-09-20T22:56:19.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:19 smithi084 ceph-mon[40024]: 4.1f scrub starts 2024-09-20T22:56:19.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:19 smithi084 ceph-mon[40024]: 4.1f scrub ok 2024-09-20T22:56:19.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:19 smithi084 ceph-mon[40024]: osdmap e899: 8 total, 8 up, 7 in 2024-09-20T22:56:19.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:19 smithi084 ceph-mon[29774]: pgmap v1248: 158 pgs: 2 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 1 active+remapped+backfill_toofull, 151 active+clean; 695 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-09-20T22:56:19.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:19 smithi084 ceph-mon[29774]: 4.1f scrub starts 2024-09-20T22:56:19.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:19 smithi084 ceph-mon[29774]: 4.1f scrub ok 2024-09-20T22:56:19.866 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:19 smithi084 ceph-mon[29774]: osdmap e899: 8 total, 8 up, 7 in 2024-09-20T22:56:19.876 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:56:19.903 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:19.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:19.904 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:56:20.004 INFO:journalctl@ceph.iscsi.iscsi.a.smithi154.stdout:Sep 20 22:56:19 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-iscsi-iscsi-a[59730]: debug there is no tcmu-runner data available 2024-09-20T22:56:20.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:19 smithi154 ceph-mon[33264]: pgmap v1248: 158 pgs: 2 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 1 active+remapped+backfill_toofull, 151 active+clean; 695 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-09-20T22:56:20.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:19 smithi154 ceph-mon[33264]: 4.1f scrub starts 2024-09-20T22:56:20.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:19 smithi154 ceph-mon[33264]: 4.1f scrub ok 2024-09-20T22:56:20.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:19 smithi154 ceph-mon[33264]: osdmap e899: 8 total, 8 up, 7 in 2024-09-20T22:56:20.005 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:56:20.026 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:20.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:20.027 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:56:20.128 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:56:20.153 INFO:teuthology.orchestra.run.smithi084.stderr:no valid command found; 10 closest matches: 2024-09-20T22:56:20.153 INFO:teuthology.orchestra.run.smithi084.stderr:0 2024-09-20T22:56:20.153 INFO:teuthology.orchestra.run.smithi084.stderr:1 2024-09-20T22:56:20.154 INFO:teuthology.orchestra.run.smithi084.stderr:2 2024-09-20T22:56:20.154 INFO:teuthology.orchestra.run.smithi084.stderr:abort 2024-09-20T22:56:20.154 INFO:teuthology.orchestra.run.smithi084.stderr:assert 2024-09-20T22:56:20.154 INFO:teuthology.orchestra.run.smithi084.stderr:bench [] [] [] [] 2024-09-20T22:56:20.154 INFO:teuthology.orchestra.run.smithi084.stderr:bluefs debug_inject_read_zeros 2024-09-20T22:56:20.154 INFO:teuthology.orchestra.run.smithi084.stderr:bluefs files list 2024-09-20T22:56:20.154 INFO:teuthology.orchestra.run.smithi084.stderr:bluefs stats 2024-09-20T22:56:20.154 INFO:teuthology.orchestra.run.smithi084.stderr:bluestore allocator dump block 2024-09-20T22:56:20.154 INFO:teuthology.orchestra.run.smithi084.stderr:Error EINVAL: invalid command 2024-09-20T22:56:20.154 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:20.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:20.155 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:56:20.256 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:56:20.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:20.283 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:20.284 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:56:20.354 INFO:tasks.rados.rados.0.smithi084.stdout:3578: done (4 left) 2024-09-20T22:56:20.354 INFO:tasks.rados.rados.0.smithi084.stdout:3579: done (3 left) 2024-09-20T22:56:20.354 INFO:tasks.rados.rados.0.smithi084.stdout:3582: done (2 left) 2024-09-20T22:56:20.354 INFO:tasks.rados.rados.0.smithi084.stdout:3583: done (1 left) 2024-09-20T22:56:20.354 INFO:tasks.rados.rados.0.smithi084.stdout:3584: done (0 left) 2024-09-20T22:56:20.354 INFO:tasks.rados.rados.0.smithi084.stdout:3585: delete oid 20 current snap is 418 2024-09-20T22:56:20.358 INFO:tasks.rados.rados.0.smithi084.stdout:3585: done (0 left) 2024-09-20T22:56:20.358 INFO:tasks.rados.rados.0.smithi084.stdout:3586: read oid 46 snap -1 2024-09-20T22:56:20.358 INFO:tasks.rados.rados.0.smithi084.stdout:3586: expect deleted 2024-09-20T22:56:20.358 INFO:tasks.rados.rados.0.smithi084.stdout:3587: delete oid 9 current snap is 418 2024-09-20T22:56:20.363 INFO:tasks.rados.rados.0.smithi084.stdout:3586: done (1 left) 2024-09-20T22:56:20.363 INFO:tasks.rados.rados.0.smithi084.stdout:3587: done (0 left) 2024-09-20T22:56:20.363 INFO:tasks.rados.rados.0.smithi084.stdout:3588: rollback oid 34 current snap is 418 2024-09-20T22:56:20.363 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 34 to 312 2024-09-20T22:56:20.363 INFO:tasks.rados.rados.0.smithi084.stdout:3589: copy_from oid 5 from oid 3 current snap is 418 2024-09-20T22:56:20.363 INFO:tasks.rados.rados.0.smithi084.stdout:3590: snap_remove snap 377 2024-09-20T22:56:20.385 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:56:20.408 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:20.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:20.409 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:56:20.509 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:56:20.532 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:20.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:20.533 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:56:20.634 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:56:20.655 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:20.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:20.656 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:56:20.757 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:56:20.779 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:20.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:20.780 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:56:20.833 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-20T22:56:20.834 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-20T22:56:20.834 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph tell osd.5 'trim stale osdmaps' 2024-09-20T22:56:20.839 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:56:20.859 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:56:20.880 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:56:20.901 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:20.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:20.902 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:56:21.002 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:56:21.027 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:21.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:21.028 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:56:21.129 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:56:21.149 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:21.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:21.150 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:56:21.251 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:56:21.276 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:21.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:21.277 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:56:21.359 INFO:tasks.rados.rados.0.smithi084.stdout:3590: done (2 left) 2024-09-20T22:56:21.359 INFO:tasks.rados.rados.0.smithi084.stdout:3591: rollback oid 1 current snap is 418 2024-09-20T22:56:21.359 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 1 to 385 2024-09-20T22:56:21.359 INFO:tasks.rados.rados.0.smithi084.stdout:3592: copy_from oid 47 from oid 23 current snap is 418 2024-09-20T22:56:21.359 INFO:tasks.rados.rados.0.smithi084.stdout:3593: write (excl) oid 6 current snap is 418 2024-09-20T22:56:21.360 INFO:tasks.rados.rados.0.smithi084.stdout:3593: seq_num 1227 ranges {622189=663356,1984347=458752} 2024-09-20T22:56:21.367 INFO:tasks.rados.rados.0.smithi084.stdout:3593: writing smithi08477203-6 from 622189 to 1285545 tid 1 2024-09-20T22:56:21.367 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-20T22:56:21.368 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:56:21.375 INFO:tasks.rados.rados.0.smithi084.stdout:3593: writing smithi08477203-6 from 1984347 to 2443099 tid 2 2024-09-20T22:56:21.375 INFO:tasks.rados.rados.0.smithi084.stdout:3594: rollback oid 22 current snap is 418 2024-09-20T22:56:21.375 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 22 to 332 2024-09-20T22:56:21.376 INFO:tasks.rados.rados.0.smithi084.stdout:3588: finishing rollback tid 0 to smithi08477203-34 2024-09-20T22:56:21.376 INFO:tasks.rados.rados.0.smithi084.stdout:3588: finishing rollback tid 1 to smithi08477203-34 2024-09-20T22:56:21.376 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 34 v 2594 (ObjNum 129 snap 0 seq_num 2091882272) dirty dne 2024-09-20T22:56:21.376 INFO:tasks.rados.rados.0.smithi084.stdout:3589: finishing copy_from racing read to smithi08477203-5 2024-09-20T22:56:21.376 INFO:tasks.rados.rados.0.smithi084.stdout:3589: finishing copy_from to smithi08477203-5 2024-09-20T22:56:21.376 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 5 v 2582 (ObjNum 1177 snap 404 seq_num 1177) dirty exists 2024-09-20T22:56:21.376 INFO:tasks.rados.rados.0.smithi084.stdout:3591: finishing rollback tid 0 to smithi08477203-1 2024-09-20T22:56:21.376 INFO:tasks.rados.rados.0.smithi084.stdout:3591: finishing rollback tid 1 to smithi08477203-1 2024-09-20T22:56:21.376 INFO:tasks.rados.rados.0.smithi084.stdout:3591: finishing rollback tid 2 to smithi08477203-1 2024-09-20T22:56:21.376 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 1 v 2665 (ObjNum 738 snap 235 seq_num 738) dirty exists 2024-09-20T22:56:21.376 INFO:tasks.rados.rados.0.smithi084.stdout:3588: done (5 left) 2024-09-20T22:56:21.376 INFO:tasks.rados.rados.0.smithi084.stdout:3589: done (4 left) 2024-09-20T22:56:21.376 INFO:tasks.rados.rados.0.smithi084.stdout:3591: done (3 left) 2024-09-20T22:56:21.376 INFO:tasks.rados.rados.0.smithi084.stdout:3595: write (excl) oid 46 current snap is 418 2024-09-20T22:56:21.377 INFO:tasks.rados.rados.0.smithi084.stdout:3595: seq_num 1228 ranges {638996=738605,1926476=712520} 2024-09-20T22:56:21.378 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:56:21.384 INFO:tasks.rados.rados.0.smithi084.stdout:3595: writing smithi08477203-46 from 638996 to 1377601 tid 1 2024-09-20T22:56:21.392 INFO:tasks.rados.rados.0.smithi084.stdout:3595: writing smithi08477203-46 from 1926476 to 2638996 tid 2 2024-09-20T22:56:21.392 INFO:tasks.rados.rados.0.smithi084.stdout:3596: snap_remove snap 368 2024-09-20T22:56:21.404 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:21.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:21.405 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:56:21.506 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:56:21.527 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:21.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:21.528 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:56:21.595 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:21 smithi084 ceph-mon[40024]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:56:21.596 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:21 smithi084 ceph-mon[40024]: pgmap v1251: 158 pgs: 2 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 1 active+remapped+backfill_toofull, 151 active+clean; 689 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 3.4 MiB/s rd, 855 KiB/s wr, 9 op/s 2024-09-20T22:56:21.596 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:21 smithi084 ceph-mon[40024]: osdmap e900: 8 total, 8 up, 7 in 2024-09-20T22:56:21.596 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:21 smithi084 ceph-mon[40024]: 3.1d scrub starts 2024-09-20T22:56:21.596 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:21 smithi084 ceph-mon[40024]: 3.1d scrub ok 2024-09-20T22:56:21.596 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:21 smithi084 ceph-mon[29774]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:56:21.596 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:21 smithi084 ceph-mon[29774]: pgmap v1251: 158 pgs: 2 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 1 active+remapped+backfill_toofull, 151 active+clean; 689 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 3.4 MiB/s rd, 855 KiB/s wr, 9 op/s 2024-09-20T22:56:21.596 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:21 smithi084 ceph-mon[29774]: osdmap e900: 8 total, 8 up, 7 in 2024-09-20T22:56:21.596 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:21 smithi084 ceph-mon[29774]: 3.1d scrub starts 2024-09-20T22:56:21.596 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:21 smithi084 ceph-mon[29774]: 3.1d scrub ok 2024-09-20T22:56:21.629 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:56:21.650 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:21.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:21.651 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:56:21.751 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:56:21.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:21 smithi154 ceph-mon[33264]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:56:21.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:21 smithi154 ceph-mon[33264]: pgmap v1251: 158 pgs: 2 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 1 active+remapped+backfill_toofull, 151 active+clean; 689 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 3.4 MiB/s rd, 855 KiB/s wr, 9 op/s 2024-09-20T22:56:21.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:21 smithi154 ceph-mon[33264]: osdmap e900: 8 total, 8 up, 7 in 2024-09-20T22:56:21.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:21 smithi154 ceph-mon[33264]: 3.1d scrub starts 2024-09-20T22:56:21.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:21 smithi154 ceph-mon[33264]: 3.1d scrub ok 2024-09-20T22:56:21.772 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:21.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:21.773 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:56:21.874 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:56:21.896 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:21.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:21.896 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:56:21.997 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:56:22.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:22.024 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:22.024 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:56:22.125 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:56:22.151 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:22.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:22.152 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:56:22.252 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:56:22.275 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:22.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:22.276 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:56:22.372 INFO:tasks.rados.rados.0.smithi084.stdout:3596: done (4 left) 2024-09-20T22:56:22.372 INFO:tasks.rados.rados.0.smithi084.stdout:3597: rollback oid 25 current snap is 418 2024-09-20T22:56:22.372 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 25 to 405 2024-09-20T22:56:22.373 INFO:tasks.rados.rados.0.smithi084.stdout:3594: finishing rollback tid 0 to smithi08477203-22 2024-09-20T22:56:22.373 INFO:tasks.rados.rados.0.smithi084.stdout:3594: finishing rollback tid 1 to smithi08477203-22 2024-09-20T22:56:22.373 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 22 v 2496 (ObjNum 129 snap 0 seq_num 2090822304) dirty dne 2024-09-20T22:56:22.373 INFO:tasks.rados.rados.0.smithi084.stdout:3593: finishing write tid 1 to smithi08477203-6 2024-09-20T22:56:22.373 INFO:tasks.rados.rados.0.smithi084.stdout:3593: finishing write tid 2 to smithi08477203-6 2024-09-20T22:56:22.373 INFO:tasks.rados.rados.0.smithi084.stdout:3593: finishing write tid 3 to smithi08477203-6 2024-09-20T22:56:22.373 INFO:tasks.rados.rados.0.smithi084.stdout:3593: finishing write tid 4 to smithi08477203-6 2024-09-20T22:56:22.373 INFO:tasks.rados.rados.0.smithi084.stdout:3593: oid 6 updating version 0 to 2648 2024-09-20T22:56:22.373 INFO:tasks.rados.rados.0.smithi084.stdout:3593: oid 6 updating version 2648 to 2649 2024-09-20T22:56:22.373 INFO:tasks.rados.rados.0.smithi084.stdout:3593: oid 6 updating version 2649 to 2650 2024-09-20T22:56:22.373 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 6 v 2650 (ObjNum 1227 snap 418 seq_num 1227) dirty exists 2024-09-20T22:56:22.374 INFO:tasks.rados.rados.0.smithi084.stdout:3593: left oid 6 (ObjNum 1227 snap 418 seq_num 1227) 2024-09-20T22:56:22.374 INFO:tasks.rados.rados.0.smithi084.stdout:3592: finishing copy_from racing read to smithi08477203-47 2024-09-20T22:56:22.374 INFO:tasks.rados.rados.0.smithi084.stdout:3595: finishing write tid 1 to smithi08477203-46 2024-09-20T22:56:22.374 INFO:tasks.rados.rados.0.smithi084.stdout:3595: finishing write tid 2 to smithi08477203-46 2024-09-20T22:56:22.374 INFO:tasks.rados.rados.0.smithi084.stdout:3592: finishing copy_from to smithi08477203-47 2024-09-20T22:56:22.374 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 47 v 2589 (ObjNum 1066 snap 353 seq_num 1066) dirty exists 2024-09-20T22:56:22.374 INFO:tasks.rados.rados.0.smithi084.stdout:3595: finishing write tid 3 to smithi08477203-46 2024-09-20T22:56:22.374 INFO:tasks.rados.rados.0.smithi084.stdout:3595: finishing write tid 4 to smithi08477203-46 2024-09-20T22:56:22.374 INFO:tasks.rados.rados.0.smithi084.stdout:3595: oid 46 updating version 0 to 2593 2024-09-20T22:56:22.374 INFO:tasks.rados.rados.0.smithi084.stdout:3595: oid 46 updating version 2593 to 2595 2024-09-20T22:56:22.375 INFO:tasks.rados.rados.0.smithi084.stdout:3595: oid 46 version 2595 is already newer than 2594 2024-09-20T22:56:22.375 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 46 v 2595 (ObjNum 1228 snap 418 seq_num 1228) dirty exists 2024-09-20T22:56:22.375 INFO:tasks.rados.rados.0.smithi084.stdout:3595: left oid 46 (ObjNum 1228 snap 418 seq_num 1228) 2024-09-20T22:56:22.375 INFO:tasks.rados.rados.0.smithi084.stdout:3592: done (4 left) 2024-09-20T22:56:22.375 INFO:tasks.rados.rados.0.smithi084.stdout:3593: done (3 left) 2024-09-20T22:56:22.375 INFO:tasks.rados.rados.0.smithi084.stdout:3594: done (2 left) 2024-09-20T22:56:22.375 INFO:tasks.rados.rados.0.smithi084.stdout:3595: done (1 left) 2024-09-20T22:56:22.375 INFO:tasks.rados.rados.0.smithi084.stdout:3598: read oid 28 snap -1 2024-09-20T22:56:22.375 INFO:tasks.rados.rados.0.smithi084.stdout:3598: expect (ObjNum 1198 snap 411 seq_num 1198) 2024-09-20T22:56:22.375 INFO:tasks.rados.rados.0.smithi084.stdout:3599: snap_create 2024-09-20T22:56:22.377 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:56:22.388 INFO:tasks.rados.rados.0.smithi084.stdout:3598: expect (ObjNum 1198 snap 411 seq_num 1198) 2024-09-20T22:56:22.401 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:22.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:22.402 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:56:22.474 INFO:tasks.rados.rados.0.smithi084.stdout:3597: finishing rollback tid 0 to smithi08477203-25 2024-09-20T22:56:22.474 INFO:tasks.rados.rados.0.smithi084.stdout:3597: finishing rollback tid 1 to smithi08477203-25 2024-09-20T22:56:22.474 INFO:tasks.rados.rados.0.smithi084.stdout:3597: finishing rollback tid 2 to smithi08477203-25 2024-09-20T22:56:22.474 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 25 v 2933 (ObjNum 1128 snap 381 seq_num 1128) dirty exists 2024-09-20T22:56:22.503 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:56:22.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:22.526 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:22.527 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:56:22.575 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-20T22:56:22.628 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:56:22.652 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:22.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:22.653 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:56:22.670 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:22 smithi084 ceph-mon[29774]: osdmap e901: 8 total, 8 up, 7 in 2024-09-20T22:56:22.670 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:22 smithi084 ceph-mon[29774]: 1.3 scrub starts 2024-09-20T22:56:22.670 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:22 smithi084 ceph-mon[29774]: 1.3 scrub ok 2024-09-20T22:56:22.670 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:22 smithi084 ceph-mon[29774]: 4.5 scrub starts 2024-09-20T22:56:22.670 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:22 smithi084 ceph-mon[29774]: 4.5 scrub ok 2024-09-20T22:56:22.670 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:22 smithi084 ceph-mon[40024]: osdmap e901: 8 total, 8 up, 7 in 2024-09-20T22:56:22.670 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:22 smithi084 ceph-mon[40024]: 1.3 scrub starts 2024-09-20T22:56:22.670 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:22 smithi084 ceph-mon[40024]: 1.3 scrub ok 2024-09-20T22:56:22.671 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:22 smithi084 ceph-mon[40024]: 4.5 scrub starts 2024-09-20T22:56:22.671 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:22 smithi084 ceph-mon[40024]: 4.5 scrub ok 2024-09-20T22:56:22.754 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:56:22.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:22 smithi154 ceph-mon[33264]: osdmap e901: 8 total, 8 up, 7 in 2024-09-20T22:56:22.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:22 smithi154 ceph-mon[33264]: 1.3 scrub starts 2024-09-20T22:56:22.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:22 smithi154 ceph-mon[33264]: 1.3 scrub ok 2024-09-20T22:56:22.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:22 smithi154 ceph-mon[33264]: 4.5 scrub starts 2024-09-20T22:56:22.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:22 smithi154 ceph-mon[33264]: 4.5 scrub ok 2024-09-20T22:56:22.778 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:22.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:22.779 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:56:22.779 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:56:22.780 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:56:22.791 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:56:22.792 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:56:22.807 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:56:22.808 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:56:22.824 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-20T22:56:22.824 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T22:56:22.841 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:56:22.841 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:56:22.861 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:56:22.861 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:56:22.875 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:56:22.876 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:56:22.880 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:56:22.894 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:56:22.894 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:56:22.914 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:22.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:22.915 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:56:23.016 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:56:23.047 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:23.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:23.048 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:56:23.149 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:56:23.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:23.175 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:23.175 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:56:23.276 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:56:23.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:23.301 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:23.301 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:56:23.346 INFO:tasks.rados.rados.0.smithi084.stdout:3597: done (2 left) 2024-09-20T22:56:23.347 INFO:tasks.rados.rados.0.smithi084.stdout:3598: done (1 left) 2024-09-20T22:56:23.347 INFO:tasks.rados.rados.0.smithi084.stdout:3599: done (0 left) 2024-09-20T22:56:23.347 INFO:tasks.rados.rados.0.smithi084.stdout:3600: rollback oid 12 current snap is 419 2024-09-20T22:56:23.347 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 12 to 409 2024-09-20T22:56:23.347 INFO:tasks.rados.rados.0.smithi084.stdout:3601: copy_from oid 11 from oid 2 current snap is 419 2024-09-20T22:56:23.347 INFO:tasks.rados.rados.0.smithi084.stdout:3602: snap_remove snap 415 2024-09-20T22:56:23.402 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:56:23.430 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:23.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:23.431 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:56:23.532 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:56:23.573 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:23.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:23.575 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:56:23.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:23 smithi084 ceph-mon[29774]: pgmap v1254: 158 pgs: 2 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 1 active+remapped+backfill_toofull, 151 active+clean; 689 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 3.4 MiB/s rd, 855 KiB/s wr, 9 op/s 2024-09-20T22:56:23.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:23 smithi084 ceph-mon[29774]: osdmap e902: 8 total, 8 up, 7 in 2024-09-20T22:56:23.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:23 smithi084 ceph-mon[29774]: 6.11 deep-scrub starts 2024-09-20T22:56:23.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:23 smithi084 ceph-mon[29774]: 6.11 deep-scrub ok 2024-09-20T22:56:23.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:23 smithi084 ceph-mon[29774]: 6.1a deep-scrub starts 2024-09-20T22:56:23.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:23 smithi084 ceph-mon[29774]: 6.1a deep-scrub ok 2024-09-20T22:56:23.617 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:23 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/2509750420' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:56:23.617 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:23 smithi084 ceph-mon[29774]: 7.c scrub starts 2024-09-20T22:56:23.617 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:23 smithi084 ceph-mon[29774]: 7.c scrub ok 2024-09-20T22:56:23.617 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:23 smithi084 ceph-mon[29774]: 7.5 scrub starts 2024-09-20T22:56:23.617 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:23 smithi084 ceph-mon[29774]: 7.5 scrub ok 2024-09-20T22:56:23.617 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:23 smithi084 ceph-mon[29774]: osdmap e903: 8 total, 8 up, 7 in 2024-09-20T22:56:23.617 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:23 smithi084 ceph-mon[40024]: pgmap v1254: 158 pgs: 2 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 1 active+remapped+backfill_toofull, 151 active+clean; 689 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 3.4 MiB/s rd, 855 KiB/s wr, 9 op/s 2024-09-20T22:56:23.617 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:23 smithi084 ceph-mon[40024]: osdmap e902: 8 total, 8 up, 7 in 2024-09-20T22:56:23.617 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:23 smithi084 ceph-mon[40024]: 6.11 deep-scrub starts 2024-09-20T22:56:23.617 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:23 smithi084 ceph-mon[40024]: 6.11 deep-scrub ok 2024-09-20T22:56:23.617 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:23 smithi084 ceph-mon[40024]: 6.1a deep-scrub starts 2024-09-20T22:56:23.617 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:23 smithi084 ceph-mon[40024]: 6.1a deep-scrub ok 2024-09-20T22:56:23.618 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:23 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/2509750420' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:56:23.618 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:23 smithi084 ceph-mon[40024]: 7.c scrub starts 2024-09-20T22:56:23.618 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:23 smithi084 ceph-mon[40024]: 7.c scrub ok 2024-09-20T22:56:23.618 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:23 smithi084 ceph-mon[40024]: 7.5 scrub starts 2024-09-20T22:56:23.618 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:23 smithi084 ceph-mon[40024]: 7.5 scrub ok 2024-09-20T22:56:23.618 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:23 smithi084 ceph-mon[40024]: osdmap e903: 8 total, 8 up, 7 in 2024-09-20T22:56:23.675 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:56:23.700 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:23.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:23.701 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:56:23.717 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:56:23.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:23 smithi154 ceph-mon[33264]: pgmap v1254: 158 pgs: 2 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 1 active+remapped+backfill_toofull, 151 active+clean; 689 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 3.4 MiB/s rd, 855 KiB/s wr, 9 op/s 2024-09-20T22:56:23.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:23 smithi154 ceph-mon[33264]: osdmap e902: 8 total, 8 up, 7 in 2024-09-20T22:56:23.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:23 smithi154 ceph-mon[33264]: 6.11 deep-scrub starts 2024-09-20T22:56:23.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:23 smithi154 ceph-mon[33264]: 6.11 deep-scrub ok 2024-09-20T22:56:23.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:23 smithi154 ceph-mon[33264]: 6.1a deep-scrub starts 2024-09-20T22:56:23.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:23 smithi154 ceph-mon[33264]: 6.1a deep-scrub ok 2024-09-20T22:56:23.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:23 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/2509750420' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:56:23.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:23 smithi154 ceph-mon[33264]: 7.c scrub starts 2024-09-20T22:56:23.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:23 smithi154 ceph-mon[33264]: 7.c scrub ok 2024-09-20T22:56:23.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:23 smithi154 ceph-mon[33264]: 7.5 scrub starts 2024-09-20T22:56:23.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:23 smithi154 ceph-mon[33264]: 7.5 scrub ok 2024-09-20T22:56:23.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:23 smithi154 ceph-mon[33264]: osdmap e903: 8 total, 8 up, 7 in 2024-09-20T22:56:23.802 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:56:23.831 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:23.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:23.832 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:56:23.933 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:56:23.955 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:23.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:23.956 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:56:24.056 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:56:24.077 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:24.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:24.078 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:56:24.178 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:56:24.200 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:24.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:24.201 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:56:24.302 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:56:24.328 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:24.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:24.329 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:56:24.349 INFO:tasks.rados.rados.0.smithi084.stdout:3602: done (2 left) 2024-09-20T22:56:24.350 INFO:tasks.rados.rados.0.smithi084.stdout:3603: rollback oid 1 current snap is 419 2024-09-20T22:56:24.350 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 1 to 350 2024-09-20T22:56:24.350 INFO:tasks.rados.rados.0.smithi084.stdout:3600: finishing rollback tid 0 to smithi08477203-12 2024-09-20T22:56:24.350 INFO:tasks.rados.rados.0.smithi084.stdout:3600: finishing rollback tid 1 to smithi08477203-12 2024-09-20T22:56:24.350 INFO:tasks.rados.rados.0.smithi084.stdout:3600: finishing rollback tid 2 to smithi08477203-12 2024-09-20T22:56:24.350 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 12 v 2592 (ObjNum 1162 snap 399 seq_num 1162) dirty exists 2024-09-20T22:56:24.350 INFO:tasks.rados.rados.0.smithi084.stdout:3601: finishing copy_from racing read to smithi08477203-11 2024-09-20T22:56:24.350 INFO:tasks.rados.rados.0.smithi084.stdout:3601: finishing copy_from to smithi08477203-11 2024-09-20T22:56:24.350 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 11 v 1869 (ObjNum 1187 snap 408 seq_num 1187) dirty exists 2024-09-20T22:56:24.350 INFO:tasks.rados.rados.0.smithi084.stdout:3600: done (2 left) 2024-09-20T22:56:24.350 INFO:tasks.rados.rados.0.smithi084.stdout:3601: done (1 left) 2024-09-20T22:56:24.350 INFO:tasks.rados.rados.0.smithi084.stdout:3604: delete oid 31 current snap is 419 2024-09-20T22:56:24.361 INFO:tasks.rados.rados.0.smithi084.stdout:3604: done (1 left) 2024-09-20T22:56:24.362 INFO:tasks.rados.rados.0.smithi084.stdout:3605: write oid 28 current snap is 419 2024-09-20T22:56:24.362 INFO:tasks.rados.rados.0.smithi084.stdout:3605: seq_num 1230 ranges {672610=489103,1643440=722432} 2024-09-20T22:56:24.366 INFO:tasks.rados.rados.0.smithi084.stdout:3605: writing smithi08477203-28 from 672610 to 1161713 tid 1 2024-09-20T22:56:24.374 INFO:tasks.rados.rados.0.smithi084.stdout:3605: writing smithi08477203-28 from 1643440 to 2365872 tid 2 2024-09-20T22:56:24.374 INFO:tasks.rados.rados.0.smithi084.stdout:3606: write oid 5 current snap is 419 2024-09-20T22:56:24.374 INFO:tasks.rados.rados.0.smithi084.stdout:3606: seq_num 1231 ranges {689417=564352,1701922=466090} 2024-09-20T22:56:24.381 INFO:tasks.rados.rados.0.smithi084.stdout:3606: writing smithi08477203-5 from 689417 to 1253769 tid 1 2024-09-20T22:56:24.387 INFO:tasks.rados.rados.0.smithi084.stdout:3606: writing smithi08477203-5 from 1701922 to 2168012 tid 2 2024-09-20T22:56:24.387 INFO:tasks.rados.rados.0.smithi084.stdout:3607: read oid 42 snap 328 2024-09-20T22:56:24.387 INFO:tasks.rados.rados.0.smithi084.stdout:3607: expect (ObjNum 970 snap 315 seq_num 970) 2024-09-20T22:56:24.387 INFO:tasks.rados.rados.0.smithi084.stdout:3608: read oid 26 snap -1 2024-09-20T22:56:24.387 INFO:tasks.rados.rados.0.smithi084.stdout:3608: expect (ObjNum 906 snap 297 seq_num 906) 2024-09-20T22:56:24.387 INFO:tasks.rados.rados.0.smithi084.stdout:3603: finishing rollback tid 0 to smithi08477203-1 2024-09-20T22:56:24.387 INFO:tasks.rados.rados.0.smithi084.stdout:3603: finishing rollback tid 1 to smithi08477203-1 2024-09-20T22:56:24.387 INFO:tasks.rados.rados.0.smithi084.stdout:3603: finishing rollback tid 2 to smithi08477203-1 2024-09-20T22:56:24.387 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 1 v 2668 (ObjNum 738 snap 235 seq_num 738) dirty exists 2024-09-20T22:56:24.388 INFO:tasks.rados.rados.0.smithi084.stdout:3605: finishing write tid 1 to smithi08477203-28 2024-09-20T22:56:24.388 INFO:tasks.rados.rados.0.smithi084.stdout:3605: finishing write tid 2 to smithi08477203-28 2024-09-20T22:56:24.388 INFO:tasks.rados.rados.0.smithi084.stdout:3605: finishing write tid 3 to smithi08477203-28 2024-09-20T22:56:24.388 INFO:tasks.rados.rados.0.smithi084.stdout:3605: finishing write tid 4 to smithi08477203-28 2024-09-20T22:56:24.388 INFO:tasks.rados.rados.0.smithi084.stdout:3605: oid 28 updating version 0 to 2947 2024-09-20T22:56:24.388 INFO:tasks.rados.rados.0.smithi084.stdout:3605: oid 28 updating version 2947 to 2948 2024-09-20T22:56:24.388 INFO:tasks.rados.rados.0.smithi084.stdout:3605: oid 28 updating version 2948 to 2949 2024-09-20T22:56:24.388 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 28 v 2949 (ObjNum 1230 snap 419 seq_num 1230) dirty exists 2024-09-20T22:56:24.388 INFO:tasks.rados.rados.0.smithi084.stdout:3605: left oid 28 (ObjNum 1230 snap 419 seq_num 1230) 2024-09-20T22:56:24.388 INFO:tasks.rados.rados.0.smithi084.stdout:3606: finishing write tid 1 to smithi08477203-5 2024-09-20T22:56:24.388 INFO:tasks.rados.rados.0.smithi084.stdout:3603: done (4 left) 2024-09-20T22:56:24.388 INFO:tasks.rados.rados.0.smithi084.stdout:3605: done (3 left) 2024-09-20T22:56:24.388 INFO:tasks.rados.rados.0.smithi084.stdout:3609: snap_remove snap 301 2024-09-20T22:56:24.394 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:56:24.430 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:56:24.433 INFO:teuthology.orchestra.run.smithi084.stderr:no valid command found; 10 closest matches: 2024-09-20T22:56:24.433 INFO:teuthology.orchestra.run.smithi084.stderr:0 2024-09-20T22:56:24.433 INFO:teuthology.orchestra.run.smithi084.stderr:1 2024-09-20T22:56:24.434 INFO:teuthology.orchestra.run.smithi084.stderr:2 2024-09-20T22:56:24.434 INFO:teuthology.orchestra.run.smithi084.stderr:abort 2024-09-20T22:56:24.434 INFO:teuthology.orchestra.run.smithi084.stderr:assert 2024-09-20T22:56:24.434 INFO:teuthology.orchestra.run.smithi084.stderr:bench [] [] [] [] 2024-09-20T22:56:24.434 INFO:teuthology.orchestra.run.smithi084.stderr:bluefs debug_inject_read_zeros 2024-09-20T22:56:24.434 INFO:teuthology.orchestra.run.smithi084.stderr:bluefs files list 2024-09-20T22:56:24.434 INFO:teuthology.orchestra.run.smithi084.stderr:bluefs stats 2024-09-20T22:56:24.434 INFO:teuthology.orchestra.run.smithi084.stderr:bluestore allocator dump block 2024-09-20T22:56:24.434 INFO:teuthology.orchestra.run.smithi084.stderr:Error EINVAL: invalid command 2024-09-20T22:56:24.452 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:24.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:24.453 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:56:24.554 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:56:24.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:24.577 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:24.577 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:56:24.614 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:24 smithi084 ceph-mon[29774]: 7.6 deep-scrub starts 2024-09-20T22:56:24.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:24 smithi084 ceph-mon[29774]: 7.6 deep-scrub ok 2024-09-20T22:56:24.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:24 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:56:24.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:24 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:56:24.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:24 smithi084 ceph-mon[29774]: 6.e scrub starts 2024-09-20T22:56:24.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:24 smithi084 ceph-mon[29774]: 6.e scrub ok 2024-09-20T22:56:24.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:24 smithi084 ceph-mon[29774]: osdmap e904: 8 total, 8 up, 7 in 2024-09-20T22:56:24.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:24 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/3947411720' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-20T22:56:24.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:24 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-20T22:56:24.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:24 smithi084 ceph-mon[40024]: 7.6 deep-scrub starts 2024-09-20T22:56:24.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:24 smithi084 ceph-mon[40024]: 7.6 deep-scrub ok 2024-09-20T22:56:24.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:24 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:56:24.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:24 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:56:24.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:24 smithi084 ceph-mon[40024]: 6.e scrub starts 2024-09-20T22:56:24.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:24 smithi084 ceph-mon[40024]: 6.e scrub ok 2024-09-20T22:56:24.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:24 smithi084 ceph-mon[40024]: osdmap e904: 8 total, 8 up, 7 in 2024-09-20T22:56:24.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:24 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/3947411720' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-20T22:56:24.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:24 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-20T22:56:24.679 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:56:24.699 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:24.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:24.700 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:56:24.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:24 smithi154 ceph-mon[33264]: 7.6 deep-scrub starts 2024-09-20T22:56:24.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:24 smithi154 ceph-mon[33264]: 7.6 deep-scrub ok 2024-09-20T22:56:24.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:24 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:56:24.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:24 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:56:24.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:24 smithi154 ceph-mon[33264]: 6.e scrub starts 2024-09-20T22:56:24.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:24 smithi154 ceph-mon[33264]: 6.e scrub ok 2024-09-20T22:56:24.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:24 smithi154 ceph-mon[33264]: osdmap e904: 8 total, 8 up, 7 in 2024-09-20T22:56:24.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:24 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/3947411720' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-20T22:56:24.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:24 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-20T22:56:24.801 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:56:24.822 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:24.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:24.823 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:56:24.924 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:56:24.950 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:24.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:24.951 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:56:25.052 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:56:25.078 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:25.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:25.079 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:56:25.179 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:56:25.200 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:25.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:25.201 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:56:25.302 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:56:25.327 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:25.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:25.328 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:56:25.348 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is set 2024-09-20T22:56:25.350 INFO:tasks.rados.rados.0.smithi084.stdout:3609: done (3 left) 2024-09-20T22:56:25.350 INFO:tasks.rados.rados.0.smithi084.stdout:3610: write (excl) oid 16 current snap is 419 2024-09-20T22:56:25.350 INFO:tasks.rados.rados.0.smithi084.stdout:3610: seq_num 1232 ranges {706224=755954,2160404=545820} 2024-09-20T22:56:25.359 INFO:tasks.rados.rados.0.smithi084.stdout:3610: writing smithi08477203-16 from 706224 to 1462178 tid 1 2024-09-20T22:56:25.367 INFO:tasks.rados.rados.0.smithi084.stdout:3610: writing smithi08477203-16 from 2160404 to 2706224 tid 2 2024-09-20T22:56:25.368 INFO:tasks.rados.rados.0.smithi084.stdout:3606: finishing write tid 2 to smithi08477203-5 2024-09-20T22:56:25.368 INFO:tasks.rados.rados.0.smithi084.stdout:3606: finishing write tid 3 to smithi08477203-5 2024-09-20T22:56:25.368 INFO:tasks.rados.rados.0.smithi084.stdout:3606: finishing write tid 4 to smithi08477203-5 2024-09-20T22:56:25.368 INFO:tasks.rados.rados.0.smithi084.stdout:3606: oid 5 updating version 0 to 2656 2024-09-20T22:56:25.368 INFO:tasks.rados.rados.0.smithi084.stdout:3606: oid 5 updating version 2656 to 2658 2024-09-20T22:56:25.368 INFO:tasks.rados.rados.0.smithi084.stdout:3606: oid 5 version 2658 is already newer than 2657 2024-09-20T22:56:25.368 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 5 v 2658 (ObjNum 1231 snap 419 seq_num 1231) dirty exists 2024-09-20T22:56:25.368 INFO:tasks.rados.rados.0.smithi084.stdout:3606: left oid 5 (ObjNum 1231 snap 419 seq_num 1231) 2024-09-20T22:56:25.368 INFO:tasks.rados.rados.0.smithi084.stdout:3607: expect (ObjNum 970 snap 315 seq_num 970) 2024-09-20T22:56:25.429 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:56:25.446 INFO:tasks.rados.rados.0.smithi084.stdout:3608: expect (ObjNum 906 snap 297 seq_num 906) 2024-09-20T22:56:25.450 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:25.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:25.451 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:56:25.547 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-20T22:56:25.547 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-20T22:56:25.547 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph tell osd.6 'trim stale osdmaps' 2024-09-20T22:56:25.552 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:56:25.577 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:25.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:25.578 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:56:25.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:25 smithi084 ceph-mon[29774]: pgmap v1257: 158 pgs: 2 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 1 active+remapped+backfill_toofull, 151 active+clean; 710 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 2.4 MiB/s rd, 3.7 MiB/s wr, 11 op/s 2024-09-20T22:56:25.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:25 smithi084 ceph-mon[29774]: 4.1b scrub starts 2024-09-20T22:56:25.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:25 smithi084 ceph-mon[29774]: 4.1b scrub ok 2024-09-20T22:56:25.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:25 smithi084 ceph-mon[29774]: 3.c scrub starts 2024-09-20T22:56:25.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:25 smithi084 ceph-mon[29774]: 3.c scrub ok 2024-09-20T22:56:25.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:25 smithi084 ceph-mon[29774]: 4.2 deep-scrub starts 2024-09-20T22:56:25.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:25 smithi084 ceph-mon[29774]: 4.2 deep-scrub ok 2024-09-20T22:56:25.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:25 smithi084 ceph-mon[29774]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:56:25.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:25 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T22:56:25.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:25 smithi084 ceph-mon[29774]: osdmap e905: 8 total, 8 up, 7 in 2024-09-20T22:56:25.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:25 smithi084 ceph-mon[40024]: pgmap v1257: 158 pgs: 2 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 1 active+remapped+backfill_toofull, 151 active+clean; 710 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 2.4 MiB/s rd, 3.7 MiB/s wr, 11 op/s 2024-09-20T22:56:25.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:25 smithi084 ceph-mon[40024]: 4.1b scrub starts 2024-09-20T22:56:25.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:25 smithi084 ceph-mon[40024]: 4.1b scrub ok 2024-09-20T22:56:25.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:25 smithi084 ceph-mon[40024]: 3.c scrub starts 2024-09-20T22:56:25.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:25 smithi084 ceph-mon[40024]: 3.c scrub ok 2024-09-20T22:56:25.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:25 smithi084 ceph-mon[40024]: 4.2 deep-scrub starts 2024-09-20T22:56:25.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:25 smithi084 ceph-mon[40024]: 4.2 deep-scrub ok 2024-09-20T22:56:25.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:25 smithi084 ceph-mon[40024]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:56:25.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:25 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T22:56:25.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:25 smithi084 ceph-mon[40024]: osdmap e905: 8 total, 8 up, 7 in 2024-09-20T22:56:25.643 INFO:tasks.rados.rados.0.smithi084.stdout:3610: finishing write tid 1 to smithi08477203-16 2024-09-20T22:56:25.643 INFO:tasks.rados.rados.0.smithi084.stdout:3606: done (3 left) 2024-09-20T22:56:25.643 INFO:tasks.rados.rados.0.smithi084.stdout:3607: done (2 left) 2024-09-20T22:56:25.643 INFO:tasks.rados.rados.0.smithi084.stdout:3608: done (1 left) 2024-09-20T22:56:25.643 INFO:tasks.rados.rados.0.smithi084.stdout:3611: rollback oid 13 current snap is 419 2024-09-20T22:56:25.643 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 13 to 361 2024-09-20T22:56:25.643 INFO:tasks.rados.rados.0.smithi084.stdout:3610: finishing write tid 2 to smithi08477203-16 2024-09-20T22:56:25.644 INFO:tasks.rados.rados.0.smithi084.stdout:3610: finishing write tid 3 to smithi08477203-16 2024-09-20T22:56:25.644 INFO:tasks.rados.rados.0.smithi084.stdout:3610: finishing write tid 4 to smithi08477203-16 2024-09-20T22:56:25.644 INFO:tasks.rados.rados.0.smithi084.stdout:3610: oid 16 updating version 0 to 2845 2024-09-20T22:56:25.644 INFO:tasks.rados.rados.0.smithi084.stdout:3610: oid 16 version 2845 is already newer than 2844 2024-09-20T22:56:25.644 INFO:tasks.rados.rados.0.smithi084.stdout:3610: oid 16 version 2845 is already newer than 2843 2024-09-20T22:56:25.644 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 16 v 2845 (ObjNum 1232 snap 419 seq_num 1232) dirty exists 2024-09-20T22:56:25.644 INFO:tasks.rados.rados.0.smithi084.stdout:3610: left oid 16 (ObjNum 1232 snap 419 seq_num 1232) 2024-09-20T22:56:25.644 INFO:tasks.rados.rados.0.smithi084.stdout:3610: done (1 left) 2024-09-20T22:56:25.644 INFO:tasks.rados.rados.0.smithi084.stdout:3612: copy_from oid 22 from oid 2 current snap is 419 2024-09-20T22:56:25.644 INFO:tasks.rados.rados.0.smithi084.stdout:3613: copy_from oid 33 from oid 44 current snap is 419 2024-09-20T22:56:25.644 INFO:tasks.rados.rados.0.smithi084.stdout:3614: snap_remove snap 385 2024-09-20T22:56:25.679 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:56:25.702 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:25.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:25.703 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:56:25.719 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-20T22:56:25.753 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-20T22:56:25.753 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2024-09-20T22:56:25.753 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2024-09-20T22:56:25.754 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-20T22:56:25.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:25 smithi154 ceph-mon[33264]: pgmap v1257: 158 pgs: 2 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 1 active+remapped+backfill_toofull, 151 active+clean; 710 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 2.4 MiB/s rd, 3.7 MiB/s wr, 11 op/s 2024-09-20T22:56:25.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:25 smithi154 ceph-mon[33264]: 4.1b scrub starts 2024-09-20T22:56:25.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:25 smithi154 ceph-mon[33264]: 4.1b scrub ok 2024-09-20T22:56:25.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:25 smithi154 ceph-mon[33264]: 3.c scrub starts 2024-09-20T22:56:25.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:25 smithi154 ceph-mon[33264]: 3.c scrub ok 2024-09-20T22:56:25.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:25 smithi154 ceph-mon[33264]: 4.2 deep-scrub starts 2024-09-20T22:56:25.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:25 smithi154 ceph-mon[33264]: 4.2 deep-scrub ok 2024-09-20T22:56:25.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:25 smithi154 ceph-mon[33264]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:56:25.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:25 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T22:56:25.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:25 smithi154 ceph-mon[33264]: osdmap e905: 8 total, 8 up, 7 in 2024-09-20T22:56:25.804 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:56:25.827 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:25.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:25.828 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:56:25.929 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:56:25.951 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:25.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:25.952 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:56:26.053 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:56:26.079 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:26.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:26.080 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:56:26.181 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:56:26.205 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:26.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:26.206 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:56:26.307 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:56:26.328 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:26.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:26.329 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:56:26.427 INFO:tasks.rados.rados.0.smithi084.stdout:3614: done (3 left) 2024-09-20T22:56:26.427 INFO:tasks.rados.rados.0.smithi084.stdout:3615: read oid 46 snap 398 2024-09-20T22:56:26.427 INFO:tasks.rados.rados.0.smithi084.stdout:3615: expect (ObjNum 1158 snap 397 seq_num 1158) 2024-09-20T22:56:26.427 INFO:tasks.rados.rados.0.smithi084.stdout:3616: delete oid 32 current snap is 419 2024-09-20T22:56:26.427 INFO:tasks.rados.rados.0.smithi084.stdout:3612: finishing copy_from racing read to smithi08477203-22 2024-09-20T22:56:26.427 INFO:tasks.rados.rados.0.smithi084.stdout:3611: finishing rollback tid 1 to smithi08477203-13 2024-09-20T22:56:26.427 INFO:tasks.rados.rados.0.smithi084.stdout:3611: finishing rollback tid 2 to smithi08477203-13 2024-09-20T22:56:26.427 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 13 v 2951 (ObjNum 1074 snap 358 seq_num 1074) dirty exists 2024-09-20T22:56:26.427 INFO:tasks.rados.rados.0.smithi084.stdout:3613: finishing copy_from racing read to smithi08477203-33 2024-09-20T22:56:26.427 INFO:tasks.rados.rados.0.smithi084.stdout:3612: finishing copy_from to smithi08477203-22 2024-09-20T22:56:26.427 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 22 v 1869 (ObjNum 1187 snap 408 seq_num 1187) dirty exists 2024-09-20T22:56:26.427 INFO:tasks.rados.rados.0.smithi084.stdout:3613: finishing copy_from to smithi08477203-33 2024-09-20T22:56:26.428 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 33 v 2842 (ObjNum 1202 snap 411 seq_num 1202) dirty exists 2024-09-20T22:56:26.430 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:56:26.434 INFO:tasks.rados.rados.0.smithi084.stdout:3611: done (4 left) 2024-09-20T22:56:26.434 INFO:tasks.rados.rados.0.smithi084.stdout:3612: done (3 left) 2024-09-20T22:56:26.434 INFO:tasks.rados.rados.0.smithi084.stdout:3613: done (2 left) 2024-09-20T22:56:26.435 INFO:tasks.rados.rados.0.smithi084.stdout:3616: done (1 left) 2024-09-20T22:56:26.435 INFO:tasks.rados.rados.0.smithi084.stdout:3617: snap_remove snap 384 2024-09-20T22:56:26.454 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:26.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:26.456 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:56:26.556 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:56:26.582 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:26.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:26.583 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:56:26.640 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.1 dump_blocked_ops 2024-09-20T22:56:26.683 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:56:26.709 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:26.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:26.710 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:56:26.810 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:56:26.831 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:26.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:26.832 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:56:26.933 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:56:26.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:26.959 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:26.959 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:56:27.060 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:56:27.081 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:27.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:27.082 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:56:27.182 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:56:27.203 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:27.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:27.204 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:56:27.304 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:56:27.316 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:56:27.327 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:27.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:27.329 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:56:27.329 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:56:27.333 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:56:27.429 INFO:tasks.rados.rados.0.smithi084.stdout:3617: done (1 left) 2024-09-20T22:56:27.429 INFO:tasks.rados.rados.0.smithi084.stdout:3618: delete oid 25 current snap is 419 2024-09-20T22:56:27.429 INFO:tasks.rados.rados.0.smithi084.stdout:3615: expect (ObjNum 1158 snap 397 seq_num 1158) 2024-09-20T22:56:27.429 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:56:27.456 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:27.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:27.457 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:56:27.558 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:56:27.568 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:27 smithi084 ceph-mon[29774]: pgmap v1260: 158 pgs: 2 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 1 active+remapped+backfill_toofull, 151 active+clean; 710 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 2.4 MiB/s rd, 3.7 MiB/s wr, 11 op/s 2024-09-20T22:56:27.568 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:27 smithi084 ceph-mon[29774]: osdmap e906: 8 total, 8 up, 7 in 2024-09-20T22:56:27.568 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:27 smithi084 ceph-mon[40024]: pgmap v1260: 158 pgs: 2 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 1 active+remapped+backfill_toofull, 151 active+clean; 710 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 2.4 MiB/s rd, 3.7 MiB/s wr, 11 op/s 2024-09-20T22:56:27.568 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:27 smithi084 ceph-mon[40024]: osdmap e906: 8 total, 8 up, 7 in 2024-09-20T22:56:27.583 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:27.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:27.584 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:56:27.651 INFO:tasks.rados.rados.0.smithi084.stdout:3615: done (1 left) 2024-09-20T22:56:27.651 INFO:tasks.rados.rados.0.smithi084.stdout:3618: done (0 left) 2024-09-20T22:56:27.651 INFO:tasks.rados.rados.0.smithi084.stdout:3619: write (excl) oid 46 current snap is 419 2024-09-20T22:56:27.651 INFO:tasks.rados.rados.0.smithi084.stdout:3619: seq_num 1235 ranges {756645=581701,1935850=473428} 2024-09-20T22:56:27.658 INFO:tasks.rados.rados.0.smithi084.stdout:3619: writing smithi08477203-46 from 756645 to 1338346 tid 1 2024-09-20T22:56:27.664 INFO:tasks.rados.rados.0.smithi084.stdout:3619: writing smithi08477203-46 from 1935850 to 2409278 tid 2 2024-09-20T22:56:27.664 INFO:tasks.rados.rados.0.smithi084.stdout:3620: delete oid 34 current snap is 419 2024-09-20T22:56:27.664 INFO:tasks.rados.rados.0.smithi084.stdout:3619: finishing write tid 1 to smithi08477203-46 2024-09-20T22:56:27.666 INFO:tasks.rados.rados.0.smithi084.stdout:3620: done (1 left) 2024-09-20T22:56:27.666 INFO:tasks.rados.rados.0.smithi084.stdout:3621: write (excl) oid 39 current snap is 419 2024-09-20T22:56:27.667 INFO:tasks.rados.rados.0.smithi084.stdout:3621: seq_num 1237 ranges {790259=732199,1936461=477097} 2024-09-20T22:56:27.674 INFO:tasks.rados.rados.0.smithi084.stdout:3621: writing smithi08477203-39 from 790259 to 1522458 tid 1 2024-09-20T22:56:27.680 INFO:tasks.rados.rados.0.smithi084.stdout:3621: writing smithi08477203-39 from 1936461 to 2413558 tid 2 2024-09-20T22:56:27.680 INFO:tasks.rados.rados.0.smithi084.stdout:3622: snap_create 2024-09-20T22:56:27.680 INFO:tasks.rados.rados.0.smithi084.stdout:3619: finishing write tid 2 to smithi08477203-46 2024-09-20T22:56:27.680 INFO:tasks.rados.rados.0.smithi084.stdout:3619: finishing write tid 3 to smithi08477203-46 2024-09-20T22:56:27.680 INFO:tasks.rados.rados.0.smithi084.stdout:3619: finishing write tid 4 to smithi08477203-46 2024-09-20T22:56:27.680 INFO:tasks.rados.rados.0.smithi084.stdout:3619: oid 46 updating version 0 to 2596 2024-09-20T22:56:27.680 INFO:tasks.rados.rados.0.smithi084.stdout:3619: oid 46 updating version 2596 to 2598 2024-09-20T22:56:27.680 INFO:tasks.rados.rados.0.smithi084.stdout:3619: oid 46 version 2598 is already newer than 2597 2024-09-20T22:56:27.680 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 46 v 2598 (ObjNum 1235 snap 419 seq_num 1235) dirty exists 2024-09-20T22:56:27.680 INFO:tasks.rados.rados.0.smithi084.stdout:3619: left oid 46 (ObjNum 1235 snap 419 seq_num 1235) 2024-09-20T22:56:27.680 INFO:tasks.rados.rados.0.smithi084.stdout:3621: finishing write tid 1 to smithi08477203-39 2024-09-20T22:56:27.683 INFO:tasks.rados.rados.0.smithi084.stdout:3621: finishing write tid 2 to smithi08477203-39 2024-09-20T22:56:27.683 INFO:tasks.rados.rados.0.smithi084.stdout:3621: finishing write tid 3 to smithi08477203-39 2024-09-20T22:56:27.684 INFO:tasks.rados.rados.0.smithi084.stdout:3621: finishing write tid 4 to smithi08477203-39 2024-09-20T22:56:27.684 INFO:tasks.rados.rados.0.smithi084.stdout:3621: oid 39 updating version 0 to 2671 2024-09-20T22:56:27.684 INFO:tasks.rados.rados.0.smithi084.stdout:3621: oid 39 version 2671 is already newer than 2669 2024-09-20T22:56:27.684 INFO:tasks.rados.rados.0.smithi084.stdout:3621: oid 39 version 2671 is already newer than 2670 2024-09-20T22:56:27.684 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 39 v 2671 (ObjNum 1237 snap 419 seq_num 1237) dirty exists 2024-09-20T22:56:27.684 INFO:tasks.rados.rados.0.smithi084.stdout:3621: left oid 39 (ObjNum 1237 snap 419 seq_num 1237) 2024-09-20T22:56:27.684 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:56:27.709 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:27.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:27.709 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:56:27.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:27 smithi154 ceph-mon[33264]: pgmap v1260: 158 pgs: 2 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 1 active+remapped+backfill_toofull, 151 active+clean; 710 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail; 2.4 MiB/s rd, 3.7 MiB/s wr, 11 op/s 2024-09-20T22:56:27.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:27 smithi154 ceph-mon[33264]: osdmap e906: 8 total, 8 up, 7 in 2024-09-20T22:56:27.811 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:56:27.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:27.833 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:27.833 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:56:27.933 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:56:27.954 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:27.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:27.955 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:56:28.056 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:56:28.077 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:28.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:28.077 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:56:28.178 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:56:28.204 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:28.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:28.205 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:56:28.297 INFO:teuthology.orchestra.run.smithi084.stderr:no valid command found; 10 closest matches: 2024-09-20T22:56:28.297 INFO:teuthology.orchestra.run.smithi084.stderr:0 2024-09-20T22:56:28.297 INFO:teuthology.orchestra.run.smithi084.stderr:1 2024-09-20T22:56:28.298 INFO:teuthology.orchestra.run.smithi084.stderr:2 2024-09-20T22:56:28.298 INFO:teuthology.orchestra.run.smithi084.stderr:abort 2024-09-20T22:56:28.298 INFO:teuthology.orchestra.run.smithi084.stderr:assert 2024-09-20T22:56:28.298 INFO:teuthology.orchestra.run.smithi084.stderr:bench [] [] [] [] 2024-09-20T22:56:28.298 INFO:teuthology.orchestra.run.smithi084.stderr:bluefs debug_inject_read_zeros 2024-09-20T22:56:28.298 INFO:teuthology.orchestra.run.smithi084.stderr:bluefs files list 2024-09-20T22:56:28.298 INFO:teuthology.orchestra.run.smithi084.stderr:bluefs stats 2024-09-20T22:56:28.298 INFO:teuthology.orchestra.run.smithi084.stderr:bluestore allocator dump block 2024-09-20T22:56:28.298 INFO:teuthology.orchestra.run.smithi084.stderr:Error EINVAL: invalid command 2024-09-20T22:56:28.305 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:56:28.331 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:28.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:28.332 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:56:28.347 INFO:tasks.rados.rados.0.smithi084.stdout:3619: done (2 left) 2024-09-20T22:56:28.347 INFO:tasks.rados.rados.0.smithi084.stdout:3621: done (1 left) 2024-09-20T22:56:28.347 INFO:tasks.rados.rados.0.smithi084.stdout:3622: done (0 left) 2024-09-20T22:56:28.347 INFO:tasks.rados.rados.0.smithi084.stdout:3623: write (excl) oid 34 current snap is 420 2024-09-20T22:56:28.347 INFO:tasks.rados.rados.0.smithi084.stdout:3623: seq_num 1238 ranges {407066=407448,1594943=620755} 2024-09-20T22:56:28.351 INFO:tasks.rados.rados.0.smithi084.stdout:3623: writing smithi08477203-34 from 407066 to 814514 tid 1 2024-09-20T22:56:28.361 INFO:tasks.rados.rados.0.smithi084.stdout:3623: writing smithi08477203-34 from 1594943 to 2215698 tid 2 2024-09-20T22:56:28.361 INFO:tasks.rados.rados.0.smithi084.stdout:3624: read oid 31 snap 408 2024-09-20T22:56:28.361 INFO:tasks.rados.rados.0.smithi084.stdout:3624: expect (ObjNum 1143 snap 389 seq_num 1143) 2024-09-20T22:56:28.361 INFO:tasks.rados.rados.0.smithi084.stdout:3625: write oid 46 current snap is 420 2024-09-20T22:56:28.361 INFO:tasks.rados.rados.0.smithi084.stdout:3625: seq_num 1239 ranges {423873=482697,1653425=480766,2691441=132432} 2024-09-20T22:56:28.366 INFO:tasks.rados.rados.0.smithi084.stdout:3625: writing smithi08477203-46 from 423873 to 906570 tid 1 2024-09-20T22:56:28.374 INFO:tasks.rados.rados.0.smithi084.stdout:3625: writing smithi08477203-46 from 1653425 to 2134191 tid 2 2024-09-20T22:56:28.378 INFO:tasks.rados.rados.0.smithi084.stdout:3625: writing smithi08477203-46 from 2691441 to 2823873 tid 3 2024-09-20T22:56:28.378 INFO:tasks.rados.rados.0.smithi084.stdout:3626: write (excl) oid 48 current snap is 420 2024-09-20T22:56:28.379 INFO:tasks.rados.rados.0.smithi084.stdout:3623: finishing write tid 1 to smithi08477203-34 2024-09-20T22:56:28.379 INFO:tasks.rados.rados.0.smithi084.stdout:3626: seq_num 1240 ranges {440680=674299,1711907=624424} 2024-09-20T22:56:28.384 INFO:tasks.rados.rados.0.smithi084.stdout:3626: writing smithi08477203-48 from 440680 to 1114979 tid 1 2024-09-20T22:56:28.392 INFO:tasks.rados.rados.0.smithi084.stdout:3626: writing smithi08477203-48 from 1711907 to 2336331 tid 2 2024-09-20T22:56:28.392 INFO:tasks.rados.rados.0.smithi084.stdout:3627: snap_remove snap 409 2024-09-20T22:56:28.432 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:56:28.454 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:28.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:28.455 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:56:28.528 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:56:28 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: ::ffff:172.21.15.154 - - [20/Sep/2024:22:56:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-20T22:56:28.529 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:28 smithi084 ceph-mon[29774]: osdmap e907: 8 total, 8 up, 7 in 2024-09-20T22:56:28.529 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:28 smithi084 ceph-mon[29774]: osdmap e908: 8 total, 8 up, 7 in 2024-09-20T22:56:28.529 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:28 smithi084 ceph-mon[40024]: osdmap e907: 8 total, 8 up, 7 in 2024-09-20T22:56:28.529 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:28 smithi084 ceph-mon[40024]: osdmap e908: 8 total, 8 up, 7 in 2024-09-20T22:56:28.556 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:56:28.579 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:28.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:28.580 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:56:28.680 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:56:28.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:28.703 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:28.703 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:56:28.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:28 smithi154 ceph-mon[33264]: osdmap e907: 8 total, 8 up, 7 in 2024-09-20T22:56:28.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:28 smithi154 ceph-mon[33264]: osdmap e908: 8 total, 8 up, 7 in 2024-09-20T22:56:28.803 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:56:28.818 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-20T22:56:28.818 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2024-09-20T22:56:28.818 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2024-09-20T22:56:28.818 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2024-09-20T22:56:28.818 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-20T22:56:28.828 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:28.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:28.829 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:56:28.930 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:56:28.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:28.956 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:28.956 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:56:29.057 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:56:29.079 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:29.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:29.080 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:56:29.180 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:56:29.206 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:29.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:29.207 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:56:29.209 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:56:29.209 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:56:29.225 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:56:29.226 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:56:29.242 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:56:29.242 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:56:29.263 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-20T22:56:29.263 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T22:56:29.279 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:56:29.280 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:56:29.280 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-20T22:56:29.297 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:56:29.297 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:56:29.308 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:56:29.314 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:56:29.314 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:56:29.331 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:56:29.331 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:56:29.332 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:29.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:29.333 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:56:29.357 INFO:tasks.rados.rados.0.smithi084.stdout:3627: done (4 left) 2024-09-20T22:56:29.357 INFO:tasks.rados.rados.0.smithi084.stdout:3628: rollback oid 45 current snap is 420 2024-09-20T22:56:29.357 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 45 to 342 2024-09-20T22:56:29.357 INFO:tasks.rados.rados.0.smithi084.stdout:3629: snap_create 2024-09-20T22:56:29.357 INFO:tasks.rados.rados.0.smithi084.stdout:3623: finishing write tid 2 to smithi08477203-34 2024-09-20T22:56:29.357 INFO:tasks.rados.rados.0.smithi084.stdout:3623: finishing write tid 3 to smithi08477203-34 2024-09-20T22:56:29.358 INFO:tasks.rados.rados.0.smithi084.stdout:3623: finishing write tid 4 to smithi08477203-34 2024-09-20T22:56:29.358 INFO:tasks.rados.rados.0.smithi084.stdout:3623: oid 34 updating version 0 to 2597 2024-09-20T22:56:29.358 INFO:tasks.rados.rados.0.smithi084.stdout:3623: oid 34 version 2597 is already newer than 2596 2024-09-20T22:56:29.358 INFO:tasks.rados.rados.0.smithi084.stdout:3623: oid 34 version 2597 is already newer than 2595 2024-09-20T22:56:29.358 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 34 v 2597 (ObjNum 1238 snap 420 seq_num 1238) dirty exists 2024-09-20T22:56:29.358 INFO:tasks.rados.rados.0.smithi084.stdout:3623: left oid 34 (ObjNum 1238 snap 420 seq_num 1238) 2024-09-20T22:56:29.358 INFO:tasks.rados.rados.0.smithi084.stdout:3625: finishing write tid 1 to smithi08477203-46 2024-09-20T22:56:29.358 INFO:tasks.rados.rados.0.smithi084.stdout:3624: expect (ObjNum 1143 snap 389 seq_num 1143) 2024-09-20T22:56:29.433 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:56:29.457 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:29.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:29.458 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:56:29.558 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:56:29.569 INFO:tasks.rados.rados.0.smithi084.stdout:3625: finishing write tid 2 to smithi08477203-46 2024-09-20T22:56:29.569 INFO:tasks.rados.rados.0.smithi084.stdout:3625: finishing write tid 3 to smithi08477203-46 2024-09-20T22:56:29.569 INFO:tasks.rados.rados.0.smithi084.stdout:3625: finishing write tid 4 to smithi08477203-46 2024-09-20T22:56:29.569 INFO:tasks.rados.rados.0.smithi084.stdout:3625: finishing write tid 5 to smithi08477203-46 2024-09-20T22:56:29.569 INFO:tasks.rados.rados.0.smithi084.stdout:3625: oid 46 updating version 0 to 2601 2024-09-20T22:56:29.569 INFO:tasks.rados.rados.0.smithi084.stdout:3625: oid 46 updating version 2601 to 2602 2024-09-20T22:56:29.569 INFO:tasks.rados.rados.0.smithi084.stdout:3625: oid 46 version 2602 is already newer than 2599 2024-09-20T22:56:29.569 INFO:tasks.rados.rados.0.smithi084.stdout:3625: oid 46 version 2602 is already newer than 2600 2024-09-20T22:56:29.569 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 46 v 2602 (ObjNum 1239 snap 420 seq_num 1239) dirty exists 2024-09-20T22:56:29.570 INFO:tasks.rados.rados.0.smithi084.stdout:3625: left oid 46 (ObjNum 1239 snap 420 seq_num 1239) 2024-09-20T22:56:29.570 INFO:tasks.rados.rados.0.smithi084.stdout:3626: finishing write tid 1 to smithi08477203-48 2024-09-20T22:56:29.570 INFO:tasks.rados.rados.0.smithi084.stdout:3626: finishing write tid 2 to smithi08477203-48 2024-09-20T22:56:29.570 INFO:tasks.rados.rados.0.smithi084.stdout:3626: finishing write tid 3 to smithi08477203-48 2024-09-20T22:56:29.570 INFO:tasks.rados.rados.0.smithi084.stdout:3626: finishing write tid 4 to smithi08477203-48 2024-09-20T22:56:29.570 INFO:tasks.rados.rados.0.smithi084.stdout:3626: oid 48 updating version 0 to 1945 2024-09-20T22:56:29.570 INFO:tasks.rados.rados.0.smithi084.stdout:3626: oid 48 version 1945 is already newer than 1944 2024-09-20T22:56:29.570 INFO:tasks.rados.rados.0.smithi084.stdout:3626: oid 48 updating version 1945 to 1946 2024-09-20T22:56:29.570 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 48 v 1946 (ObjNum 1240 snap 420 seq_num 1240) dirty exists 2024-09-20T22:56:29.570 INFO:tasks.rados.rados.0.smithi084.stdout:3626: left oid 48 (ObjNum 1240 snap 420 seq_num 1240) 2024-09-20T22:56:29.570 INFO:tasks.rados.rados.0.smithi084.stdout:3628: finishing rollback tid 1 to smithi08477203-45 2024-09-20T22:56:29.570 INFO:tasks.rados.rados.0.smithi084.stdout:3628: finishing rollback tid 2 to smithi08477203-45 2024-09-20T22:56:29.571 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 45 v 1898 (ObjNum 1025 snap 333 seq_num 1025) dirty exists 2024-09-20T22:56:29.582 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:29.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:29.583 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:56:29.684 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:56:29.707 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:29.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:29.708 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:56:29.731 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:29 smithi154 ceph-mon[33264]: pgmap v1263: 158 pgs: 2 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 1 active+remapped+backfill_toofull, 151 active+clean; 710 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-09-20T22:56:29.731 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:29 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/613762050' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:56:29.731 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:29 smithi154 ceph-mon[33264]: osdmap e909: 8 total, 8 up, 7 in 2024-09-20T22:56:29.732 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:29 smithi084 ceph-mon[40024]: pgmap v1263: 158 pgs: 2 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 1 active+remapped+backfill_toofull, 151 active+clean; 710 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-09-20T22:56:29.732 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:29 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/613762050' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:56:29.732 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:29 smithi084 ceph-mon[40024]: osdmap e909: 8 total, 8 up, 7 in 2024-09-20T22:56:29.732 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:29 smithi084 ceph-mon[29774]: pgmap v1263: 158 pgs: 2 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 1 active+remapped+backfill_toofull, 151 active+clean; 710 MiB data, 2.1 GiB used, 624 GiB / 626 GiB avail 2024-09-20T22:56:29.732 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:29 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/613762050' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:56:29.732 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:29 smithi084 ceph-mon[29774]: osdmap e909: 8 total, 8 up, 7 in 2024-09-20T22:56:29.759 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-20T22:56:29.759 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-20T22:56:29.760 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph tell osd.0 'trim stale osdmaps' 2024-09-20T22:56:29.809 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:56:29.832 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:29.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:29.833 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:56:29.933 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:56:29.954 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:29.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:29.956 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:56:30.004 INFO:journalctl@ceph.iscsi.iscsi.a.smithi154.stdout:Sep 20 22:56:29 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-iscsi-iscsi-a[59730]: debug there is no tcmu-runner data available 2024-09-20T22:56:30.057 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:56:30.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:30.082 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:30.082 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:56:30.183 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:56:30.208 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:30.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:30.209 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:56:30.309 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:56:30.335 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:30.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:30.336 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:56:30.360 INFO:tasks.rados.rados.0.smithi084.stdout:3623: done (5 left) 2024-09-20T22:56:30.360 INFO:tasks.rados.rados.0.smithi084.stdout:3624: done (4 left) 2024-09-20T22:56:30.360 INFO:tasks.rados.rados.0.smithi084.stdout:3625: done (3 left) 2024-09-20T22:56:30.360 INFO:tasks.rados.rados.0.smithi084.stdout:3626: done (2 left) 2024-09-20T22:56:30.360 INFO:tasks.rados.rados.0.smithi084.stdout:3628: done (1 left) 2024-09-20T22:56:30.360 INFO:tasks.rados.rados.0.smithi084.stdout:3629: done (0 left) 2024-09-20T22:56:30.360 INFO:tasks.rados.rados.0.smithi084.stdout:3630: read oid 17 snap -1 2024-09-20T22:56:30.360 INFO:tasks.rados.rados.0.smithi084.stdout:3630: expect (ObjNum 1126 snap 381 seq_num 1126) 2024-09-20T22:56:30.361 INFO:tasks.rados.rados.0.smithi084.stdout:3631: write oid 5 current snap is 421 2024-09-20T22:56:30.361 INFO:tasks.rados.rados.0.smithi084.stdout:3631: seq_num 1241 ranges {457487=749548,1770389=484435,2746287=111200} 2024-09-20T22:56:30.366 INFO:tasks.rados.rados.0.smithi084.stdout:3631: writing smithi08477203-5 from 457487 to 1207035 tid 1 2024-09-20T22:56:30.372 INFO:tasks.rados.rados.0.smithi084.stdout:3631: writing smithi08477203-5 from 1770389 to 2254824 tid 2 2024-09-20T22:56:30.376 INFO:tasks.rados.rados.0.smithi084.stdout:3631: writing smithi08477203-5 from 2746287 to 2857487 tid 3 2024-09-20T22:56:30.376 INFO:tasks.rados.rados.0.smithi084.stdout:3632: rollback oid 3 current snap is 421 2024-09-20T22:56:30.376 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 3 to 361 2024-09-20T22:56:30.376 INFO:tasks.rados.rados.0.smithi084.stdout:3630: expect (ObjNum 1126 snap 381 seq_num 1126) 2024-09-20T22:56:30.437 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:56:30.462 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:30.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:30.463 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:56:30.563 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:56:30.567 INFO:tasks.rados.rados.0.smithi084.stdout:3631: finishing write tid 1 to smithi08477203-5 2024-09-20T22:56:30.567 INFO:tasks.rados.rados.0.smithi084.stdout:3631: finishing write tid 2 to smithi08477203-5 2024-09-20T22:56:30.567 INFO:tasks.rados.rados.0.smithi084.stdout:3630: done (2 left) 2024-09-20T22:56:30.567 INFO:tasks.rados.rados.0.smithi084.stdout:3633: snap_create 2024-09-20T22:56:30.567 INFO:tasks.rados.rados.0.smithi084.stdout:3631: finishing write tid 3 to smithi08477203-5 2024-09-20T22:56:30.567 INFO:tasks.rados.rados.0.smithi084.stdout:3631: finishing write tid 4 to smithi08477203-5 2024-09-20T22:56:30.567 INFO:tasks.rados.rados.0.smithi084.stdout:3631: finishing write tid 5 to smithi08477203-5 2024-09-20T22:56:30.567 INFO:tasks.rados.rados.0.smithi084.stdout:3631: oid 5 updating version 0 to 2661 2024-09-20T22:56:30.567 INFO:tasks.rados.rados.0.smithi084.stdout:3631: oid 5 version 2661 is already newer than 2659 2024-09-20T22:56:30.567 INFO:tasks.rados.rados.0.smithi084.stdout:3631: oid 5 updating version 2661 to 2662 2024-09-20T22:56:30.567 INFO:tasks.rados.rados.0.smithi084.stdout:3631: oid 5 version 2662 is already newer than 2660 2024-09-20T22:56:30.567 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 5 v 2662 (ObjNum 1241 snap 421 seq_num 1241) dirty exists 2024-09-20T22:56:30.568 INFO:tasks.rados.rados.0.smithi084.stdout:3631: left oid 5 (ObjNum 1241 snap 421 seq_num 1241) 2024-09-20T22:56:30.568 INFO:tasks.rados.rados.0.smithi084.stdout:3632: finishing rollback tid 0 to smithi08477203-3 2024-09-20T22:56:30.568 INFO:tasks.rados.rados.0.smithi084.stdout:3632: finishing rollback tid 1 to smithi08477203-3 2024-09-20T22:56:30.568 INFO:tasks.rados.rados.0.smithi084.stdout:3632: finishing rollback tid 2 to smithi08477203-3 2024-09-20T22:56:30.568 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 3 v 2605 (ObjNum 1053 snap 345 seq_num 1053) dirty exists 2024-09-20T22:56:30.588 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:30.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:30.589 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:56:30.690 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:56:30.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:30.712 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:30.712 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:56:30.813 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:56:30.834 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.1 dump_historic_ops 2024-09-20T22:56:30.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:30.835 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:30.836 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:56:30.881 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:56:30.912 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:56:30.937 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:56:30.963 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:30.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:30.964 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:56:31.064 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:56:31.085 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:31.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:31.086 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:56:31.186 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:56:31.228 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:31.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:31.229 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:56:31.329 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:56:31.355 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:31.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:31.356 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:56:31.438 INFO:tasks.rados.rados.0.smithi084.stdout:3631: done (2 left) 2024-09-20T22:56:31.439 INFO:tasks.rados.rados.0.smithi084.stdout:3632: done (1 left) 2024-09-20T22:56:31.439 INFO:tasks.rados.rados.0.smithi084.stdout:3633: done (0 left) 2024-09-20T22:56:31.439 INFO:tasks.rados.rados.0.smithi084.stdout:3634: read oid 34 snap -1 2024-09-20T22:56:31.439 INFO:tasks.rados.rados.0.smithi084.stdout:3634: expect (ObjNum 1238 snap 420 seq_num 1238) 2024-09-20T22:56:31.439 INFO:tasks.rados.rados.0.smithi084.stdout:3635: rollback oid 13 current snap is 422 2024-09-20T22:56:31.439 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 13 to 416 2024-09-20T22:56:31.439 INFO:tasks.rados.rados.0.smithi084.stdout:3636: read oid 16 snap 404 2024-09-20T22:56:31.439 INFO:tasks.rados.rados.0.smithi084.stdout:3636: expect (ObjNum 1143 snap 389 seq_num 1143) 2024-09-20T22:56:31.439 INFO:tasks.rados.rados.0.smithi084.stdout:3637: copy_from oid 23 from oid 2 current snap is 422 2024-09-20T22:56:31.439 INFO:tasks.rados.rados.0.smithi084.stdout:3638: read oid 24 snap -1 2024-09-20T22:56:31.439 INFO:tasks.rados.rados.0.smithi084.stdout:3638: expect deleted 2024-09-20T22:56:31.439 INFO:tasks.rados.rados.0.smithi084.stdout:3639: read oid 11 snap -1 2024-09-20T22:56:31.439 INFO:tasks.rados.rados.0.smithi084.stdout:3639: expect (ObjNum 1187 snap 408 seq_num 1187) 2024-09-20T22:56:31.439 INFO:tasks.rados.rados.0.smithi084.stdout:3640: read oid 46 snap -1 2024-09-20T22:56:31.440 INFO:tasks.rados.rados.0.smithi084.stdout:3640: expect (ObjNum 1239 snap 420 seq_num 1239) 2024-09-20T22:56:31.440 INFO:tasks.rados.rados.0.smithi084.stdout:3641: delete oid 12 current snap is 422 2024-09-20T22:56:31.446 INFO:tasks.rados.rados.0.smithi084.stdout:3640: expect (ObjNum 1239 snap 420 seq_num 1239) 2024-09-20T22:56:31.457 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:56:31.470 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:31 smithi084 ceph-mon[29774]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:56:31.470 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:31 smithi084 ceph-mon[29774]: pgmap v1266: 158 pgs: 2 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 1 active+remapped+backfill_toofull, 151 active+clean; 688 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 13 MiB/s rd, 3.7 MiB/s wr, 24 op/s 2024-09-20T22:56:31.470 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:31 smithi084 ceph-mon[29774]: osdmap e910: 8 total, 8 up, 7 in 2024-09-20T22:56:31.470 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:31 smithi084 ceph-mon[40024]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:56:31.471 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:31 smithi084 ceph-mon[40024]: pgmap v1266: 158 pgs: 2 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 1 active+remapped+backfill_toofull, 151 active+clean; 688 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 13 MiB/s rd, 3.7 MiB/s wr, 24 op/s 2024-09-20T22:56:31.471 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:31 smithi084 ceph-mon[40024]: osdmap e910: 8 total, 8 up, 7 in 2024-09-20T22:56:31.479 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:31.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:31.479 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:56:31.580 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:56:31.583 INFO:teuthology.orchestra.run.smithi084.stderr:no valid command found; 10 closest matches: 2024-09-20T22:56:31.583 INFO:teuthology.orchestra.run.smithi084.stderr:0 2024-09-20T22:56:31.583 INFO:teuthology.orchestra.run.smithi084.stderr:1 2024-09-20T22:56:31.583 INFO:teuthology.orchestra.run.smithi084.stderr:2 2024-09-20T22:56:31.583 INFO:teuthology.orchestra.run.smithi084.stderr:abort 2024-09-20T22:56:31.584 INFO:teuthology.orchestra.run.smithi084.stderr:assert 2024-09-20T22:56:31.584 INFO:teuthology.orchestra.run.smithi084.stderr:bench [] [] [] [] 2024-09-20T22:56:31.584 INFO:teuthology.orchestra.run.smithi084.stderr:bluefs debug_inject_read_zeros 2024-09-20T22:56:31.584 INFO:teuthology.orchestra.run.smithi084.stderr:bluefs files list 2024-09-20T22:56:31.584 INFO:teuthology.orchestra.run.smithi084.stderr:bluefs stats 2024-09-20T22:56:31.584 INFO:teuthology.orchestra.run.smithi084.stderr:bluestore allocator dump block 2024-09-20T22:56:31.584 INFO:teuthology.orchestra.run.smithi084.stderr:Error EINVAL: invalid command 2024-09-20T22:56:31.605 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:31.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:31.606 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:56:31.620 INFO:tasks.rados.rados.0.smithi084.stdout:3635: finishing rollback tid 0 to smithi08477203-13 2024-09-20T22:56:31.620 INFO:tasks.rados.rados.0.smithi084.stdout:3635: finishing rollback tid 1 to smithi08477203-13 2024-09-20T22:56:31.620 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 13 v 2988 (ObjNum 81 snap 0 seq_num 710388164) dirty dne 2024-09-20T22:56:31.620 INFO:tasks.rados.rados.0.smithi084.stdout:3636: expect (ObjNum 1143 snap 389 seq_num 1143) 2024-09-20T22:56:31.707 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:56:31.727 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:31.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:31.728 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:56:31.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:31 smithi154 ceph-mon[33264]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:56:31.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:31 smithi154 ceph-mon[33264]: pgmap v1266: 158 pgs: 2 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 1 active+remapped+backfill_toofull, 151 active+clean; 688 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 13 MiB/s rd, 3.7 MiB/s wr, 24 op/s 2024-09-20T22:56:31.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:31 smithi154 ceph-mon[33264]: osdmap e910: 8 total, 8 up, 7 in 2024-09-20T22:56:31.823 INFO:tasks.rados.rados.0.smithi084.stdout:3639: expect (ObjNum 1187 snap 408 seq_num 1187) 2024-09-20T22:56:31.830 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:56:31.856 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:31.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:31.857 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:56:31.957 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:56:31.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:31.980 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:31.980 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:56:32.055 INFO:tasks.rados.rados.0.smithi084.stdout:3634: expect (ObjNum 1238 snap 420 seq_num 1238) 2024-09-20T22:56:32.081 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:56:32.107 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:32.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:32.108 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:56:32.152 INFO:tasks.rados.rados.0.smithi084.stdout:3637: finishing copy_from racing read to smithi08477203-23 2024-09-20T22:56:32.152 INFO:tasks.rados.rados.0.smithi084.stdout:3637: finishing copy_from to smithi08477203-23 2024-09-20T22:56:32.152 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 23 v 1869 (ObjNum 1187 snap 408 seq_num 1187) dirty exists 2024-09-20T22:56:32.152 INFO:tasks.rados.rados.0.smithi084.stdout:3634: done (7 left) 2024-09-20T22:56:32.152 INFO:tasks.rados.rados.0.smithi084.stdout:3635: done (6 left) 2024-09-20T22:56:32.153 INFO:tasks.rados.rados.0.smithi084.stdout:3636: done (5 left) 2024-09-20T22:56:32.153 INFO:tasks.rados.rados.0.smithi084.stdout:3637: done (4 left) 2024-09-20T22:56:32.153 INFO:tasks.rados.rados.0.smithi084.stdout:3638: done (3 left) 2024-09-20T22:56:32.153 INFO:tasks.rados.rados.0.smithi084.stdout:3639: done (2 left) 2024-09-20T22:56:32.153 INFO:tasks.rados.rados.0.smithi084.stdout:3640: done (1 left) 2024-09-20T22:56:32.153 INFO:tasks.rados.rados.0.smithi084.stdout:3641: done (0 left) 2024-09-20T22:56:32.153 INFO:tasks.rados.rados.0.smithi084.stdout:3642: snap_remove snap 413 2024-09-20T22:56:32.208 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:56:32.233 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:32.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:32.234 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:56:32.334 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:56:32.355 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:32.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:32.356 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:56:32.420 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-20T22:56:32.438 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is set 2024-09-20T22:56:32.446 INFO:tasks.rados.rados.0.smithi084.stdout:3642: done (0 left) 2024-09-20T22:56:32.446 INFO:tasks.rados.rados.0.smithi084.stdout:3643: write (excl) oid 13 current snap is 422 2024-09-20T22:56:32.447 INFO:tasks.rados.rados.0.smithi084.stdout:3643: seq_num 1243 ranges {491101=500046,1487353=488104,2401133=489968} 2024-09-20T22:56:32.452 INFO:tasks.rados.rados.0.smithi084.stdout:3643: writing smithi08477203-13 from 491101 to 991147 tid 1 2024-09-20T22:56:32.457 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:56:32.459 INFO:tasks.rados.rados.0.smithi084.stdout:3643: writing smithi08477203-13 from 1487353 to 1975457 tid 2 2024-09-20T22:56:32.464 INFO:tasks.rados.rados.0.smithi084.stdout:3643: writing smithi08477203-13 from 2401133 to 2891101 tid 3 2024-09-20T22:56:32.464 INFO:tasks.rados.rados.0.smithi084.stdout:3644: write oid 10 current snap is 422 2024-09-20T22:56:32.464 INFO:tasks.rados.rados.0.smithi084.stdout:3643: finishing write tid 1 to smithi08477203-13 2024-09-20T22:56:32.465 INFO:tasks.rados.rados.0.smithi084.stdout:3644: seq_num 1244 ranges {507908=575295,1829482=631762} 2024-09-20T22:56:32.471 INFO:tasks.rados.rados.0.smithi084.stdout:3644: writing smithi08477203-10 from 507908 to 1083203 tid 1 2024-09-20T22:56:32.480 INFO:tasks.rados.rados.0.smithi084.stdout:3644: writing smithi08477203-10 from 1829482 to 2461244 tid 2 2024-09-20T22:56:32.480 INFO:tasks.rados.rados.0.smithi084.stdout:3645: rollback oid 39 current snap is 422 2024-09-20T22:56:32.480 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 39 to 371 2024-09-20T22:56:32.480 INFO:tasks.rados.rados.0.smithi084.stdout:3643: finishing write tid 2 to smithi08477203-13 2024-09-20T22:56:32.480 INFO:tasks.rados.rados.0.smithi084.stdout:3646: read oid 19 snap -1 2024-09-20T22:56:32.480 INFO:tasks.rados.rados.0.smithi084.stdout:3646: expect (ObjNum 1207 snap 412 seq_num 1207) 2024-09-20T22:56:32.480 INFO:tasks.rados.rados.0.smithi084.stdout:3647: read oid 7 snap -1 2024-09-20T22:56:32.480 INFO:tasks.rados.rados.0.smithi084.stdout:3647: expect (ObjNum 795 snap 254 seq_num 795) 2024-09-20T22:56:32.480 INFO:tasks.rados.rados.0.smithi084.stdout:3648: copy_from oid 38 from oid 33 current snap is 422 2024-09-20T22:56:32.480 INFO:tasks.rados.rados.0.smithi084.stdout:3643: finishing write tid 3 to smithi08477203-13 2024-09-20T22:56:32.480 INFO:tasks.rados.rados.0.smithi084.stdout:3643: finishing write tid 4 to smithi08477203-13 2024-09-20T22:56:32.480 INFO:tasks.rados.rados.0.smithi084.stdout:3643: finishing write tid 5 to smithi08477203-13 2024-09-20T22:56:32.481 INFO:tasks.rados.rados.0.smithi084.stdout:3643: oid 13 updating version 0 to 2998 2024-09-20T22:56:32.481 INFO:tasks.rados.rados.0.smithi084.stdout:3643: oid 13 version 2998 is already newer than 2996 2024-09-20T22:56:32.481 INFO:tasks.rados.rados.0.smithi084.stdout:3643: oid 13 version 2998 is already newer than 2997 2024-09-20T22:56:32.481 INFO:tasks.rados.rados.0.smithi084.stdout:3643: oid 13 version 2998 is already newer than 2995 2024-09-20T22:56:32.481 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 13 v 2998 (ObjNum 1243 snap 422 seq_num 1243) dirty exists 2024-09-20T22:56:32.481 INFO:tasks.rados.rados.0.smithi084.stdout:3643: left oid 13 (ObjNum 1243 snap 422 seq_num 1243) 2024-09-20T22:56:32.481 INFO:tasks.rados.rados.0.smithi084.stdout:3644: finishing write tid 1 to smithi08477203-10 2024-09-20T22:56:32.481 INFO:tasks.rados.rados.0.smithi084.stdout:3643: done (5 left) 2024-09-20T22:56:32.481 INFO:tasks.rados.rados.0.smithi084.stdout:3649: snap_create 2024-09-20T22:56:32.483 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:32.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:32.484 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:56:32.489 INFO:tasks.rados.rados.0.smithi084.stdout:3646: expect (ObjNum 1207 snap 412 seq_num 1207) 2024-09-20T22:56:32.584 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:56:32.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:32.608 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:32.608 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:56:32.693 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:32 smithi084 ceph-mon[29774]: purged_snaps scrub starts 2024-09-20T22:56:32.693 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:32 smithi084 ceph-mon[29774]: purged_snaps scrub ok 2024-09-20T22:56:32.693 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:32 smithi084 ceph-mon[29774]: osdmap e911: 8 total, 8 up, 7 in 2024-09-20T22:56:32.693 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:32 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/719369520' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:56:32.693 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:32 smithi084 ceph-mon[40024]: purged_snaps scrub starts 2024-09-20T22:56:32.694 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:32 smithi084 ceph-mon[40024]: purged_snaps scrub ok 2024-09-20T22:56:32.694 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:32 smithi084 ceph-mon[40024]: osdmap e911: 8 total, 8 up, 7 in 2024-09-20T22:56:32.694 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:32 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/719369520' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:56:32.709 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:56:32.732 INFO:tasks.rados.rados.0.smithi084.stdout:3648: finishing copy_from racing read to smithi08477203-38 2024-09-20T22:56:32.732 INFO:tasks.rados.rados.0.smithi084.stdout:3647: expect (ObjNum 795 snap 254 seq_num 795) 2024-09-20T22:56:32.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:32.733 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:32.733 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:56:32.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:32 smithi154 ceph-mon[33264]: purged_snaps scrub starts 2024-09-20T22:56:32.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:32 smithi154 ceph-mon[33264]: purged_snaps scrub ok 2024-09-20T22:56:32.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:32 smithi154 ceph-mon[33264]: osdmap e911: 8 total, 8 up, 7 in 2024-09-20T22:56:32.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:32 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/719369520' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:56:32.833 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:56:32.839 INFO:tasks.rados.rados.0.smithi084.stdout:3644: finishing write tid 2 to smithi08477203-10 2024-09-20T22:56:32.839 INFO:tasks.rados.rados.0.smithi084.stdout:3644: finishing write tid 3 to smithi08477203-10 2024-09-20T22:56:32.839 INFO:tasks.rados.rados.0.smithi084.stdout:3644: finishing write tid 4 to smithi08477203-10 2024-09-20T22:56:32.839 INFO:tasks.rados.rados.0.smithi084.stdout:3644: oid 10 updating version 0 to 2674 2024-09-20T22:56:32.839 INFO:tasks.rados.rados.0.smithi084.stdout:3644: oid 10 version 2674 is already newer than 2673 2024-09-20T22:56:32.839 INFO:tasks.rados.rados.0.smithi084.stdout:3644: oid 10 version 2674 is already newer than 2672 2024-09-20T22:56:32.839 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 10 v 2674 (ObjNum 1244 snap 422 seq_num 1244) dirty exists 2024-09-20T22:56:32.839 INFO:tasks.rados.rados.0.smithi084.stdout:3644: left oid 10 (ObjNum 1244 snap 422 seq_num 1244) 2024-09-20T22:56:32.839 INFO:tasks.rados.rados.0.smithi084.stdout:3645: finishing rollback tid 0 to smithi08477203-39 2024-09-20T22:56:32.839 INFO:tasks.rados.rados.0.smithi084.stdout:3645: finishing rollback tid 1 to smithi08477203-39 2024-09-20T22:56:32.840 INFO:tasks.rados.rados.0.smithi084.stdout:3645: finishing rollback tid 2 to smithi08477203-39 2024-09-20T22:56:32.840 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 39 v 2677 (ObjNum 1051 snap 345 seq_num 1051) dirty exists 2024-09-20T22:56:32.840 INFO:tasks.rados.rados.0.smithi084.stdout:3648: finishing copy_from to smithi08477203-38 2024-09-20T22:56:32.840 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 38 v 2842 (ObjNum 1202 snap 411 seq_num 1202) dirty exists 2024-09-20T22:56:32.854 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:32.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:32.855 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:56:32.931 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:56:32.955 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-20T22:56:32.955 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-20T22:56:32.955 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph tell osd.4 'trim stale osdmaps' 2024-09-20T22:56:32.956 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:56:32.976 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:32.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:32.977 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:56:33.078 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:56:33.104 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:33.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:33.105 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:56:33.206 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:56:33.227 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:33.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:33.228 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:56:33.329 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:56:33.356 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:33.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:33.357 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:56:33.447 INFO:tasks.rados.rados.0.smithi084.stdout:3644: done (5 left) 2024-09-20T22:56:33.447 INFO:tasks.rados.rados.0.smithi084.stdout:3645: done (4 left) 2024-09-20T22:56:33.447 INFO:tasks.rados.rados.0.smithi084.stdout:3646: done (3 left) 2024-09-20T22:56:33.447 INFO:tasks.rados.rados.0.smithi084.stdout:3647: done (2 left) 2024-09-20T22:56:33.447 INFO:tasks.rados.rados.0.smithi084.stdout:3648: done (1 left) 2024-09-20T22:56:33.447 INFO:tasks.rados.rados.0.smithi084.stdout:3649: done (0 left) 2024-09-20T22:56:33.447 INFO:tasks.rados.rados.0.smithi084.stdout:3650: delete oid 18 current snap is 423 2024-09-20T22:56:33.455 INFO:tasks.rados.rados.0.smithi084.stdout:3650: done (0 left) 2024-09-20T22:56:33.455 INFO:tasks.rados.rados.0.smithi084.stdout:3651: read oid 50 snap -1 2024-09-20T22:56:33.455 INFO:tasks.rados.rados.0.smithi084.stdout:3651: expect (ObjNum 1128 snap 381 seq_num 1128) 2024-09-20T22:56:33.455 INFO:tasks.rados.rados.0.smithi084.stdout:3652: read oid 4 snap 380 2024-09-20T22:56:33.455 INFO:tasks.rados.rados.0.smithi084.stdout:3652: expect (ObjNum 1111 snap 378 seq_num 1111) 2024-09-20T22:56:33.455 INFO:tasks.rados.rados.0.smithi084.stdout:3653: write oid 27 current snap is 423 2024-09-20T22:56:33.455 INFO:tasks.rados.rados.0.smithi084.stdout:3653: seq_num 1246 ranges {541522=725793,1946446=635431} 2024-09-20T22:56:33.458 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:56:33.462 INFO:tasks.rados.rados.0.smithi084.stdout:3653: writing smithi08477203-27 from 541522 to 1267315 tid 1 2024-09-20T22:56:33.471 INFO:tasks.rados.rados.0.smithi084.stdout:3653: writing smithi08477203-27 from 1946446 to 2581877 tid 2 2024-09-20T22:56:33.472 INFO:tasks.rados.rados.0.smithi084.stdout:3654: read oid 47 snap -1 2024-09-20T22:56:33.472 INFO:tasks.rados.rados.0.smithi084.stdout:3654: expect (ObjNum 1066 snap 353 seq_num 1066) 2024-09-20T22:56:33.472 INFO:tasks.rados.rados.0.smithi084.stdout:3655: write (excl) oid 7 current snap is 423 2024-09-20T22:56:33.472 INFO:tasks.rados.rados.0.smithi084.stdout:3655: seq_num 1247 ranges {558329=517395,1721281=779089} 2024-09-20T22:56:33.478 INFO:tasks.rados.rados.0.smithi084.stdout:3655: writing smithi08477203-7 from 558329 to 1075724 tid 1 2024-09-20T22:56:33.487 INFO:tasks.rados.rados.0.smithi084.stdout:3655: writing smithi08477203-7 from 1721281 to 2500370 tid 2 2024-09-20T22:56:33.487 INFO:tasks.rados.rados.0.smithi084.stdout:3656: copy_from oid 48 from oid 32 current snap is 423 2024-09-20T22:56:33.487 INFO:tasks.rados.rados.0.smithi084.stdout:3652: expect (ObjNum 1111 snap 378 seq_num 1111) 2024-09-20T22:56:33.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:33.488 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:33.488 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:56:33.579 INFO:tasks.rados.rados.0.smithi084.stdout:3651: expect (ObjNum 1128 snap 381 seq_num 1128) 2024-09-20T22:56:33.589 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:56:33.610 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:33.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:33.611 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:56:33.690 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:33 smithi084 ceph-mon[40024]: pgmap v1269: 158 pgs: 2 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 1 active+remapped+backfill_toofull, 151 active+clean; 688 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 13 MiB/s rd, 3.7 MiB/s wr, 24 op/s 2024-09-20T22:56:33.691 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:33 smithi084 ceph-mon[40024]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:56:33.691 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:33 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/719369520' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T22:56:33.691 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:33 smithi084 ceph-mon[40024]: osdmap e912: 8 total, 8 up, 7 in 2024-09-20T22:56:33.691 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:33 smithi084 ceph-mon[29774]: pgmap v1269: 158 pgs: 2 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 1 active+remapped+backfill_toofull, 151 active+clean; 688 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 13 MiB/s rd, 3.7 MiB/s wr, 24 op/s 2024-09-20T22:56:33.691 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:33 smithi084 ceph-mon[29774]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:56:33.691 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:33 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/719369520' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T22:56:33.691 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:33 smithi084 ceph-mon[29774]: osdmap e912: 8 total, 8 up, 7 in 2024-09-20T22:56:33.712 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:56:33.733 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:33.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:33.734 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:56:33.751 INFO:tasks.rados.rados.0.smithi084.stdout:3653: finishing write tid 1 to smithi08477203-27 2024-09-20T22:56:33.751 INFO:tasks.rados.rados.0.smithi084.stdout:3653: finishing write tid 2 to smithi08477203-27 2024-09-20T22:56:33.751 INFO:tasks.rados.rados.0.smithi084.stdout:3653: finishing write tid 3 to smithi08477203-27 2024-09-20T22:56:33.752 INFO:tasks.rados.rados.0.smithi084.stdout:3653: finishing write tid 4 to smithi08477203-27 2024-09-20T22:56:33.752 INFO:tasks.rados.rados.0.smithi084.stdout:3653: oid 27 updating version 0 to 2499 2024-09-20T22:56:33.752 INFO:tasks.rados.rados.0.smithi084.stdout:3653: oid 27 version 2499 is already newer than 2497 2024-09-20T22:56:33.752 INFO:tasks.rados.rados.0.smithi084.stdout:3653: oid 27 version 2499 is already newer than 2498 2024-09-20T22:56:33.752 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 27 v 2499 (ObjNum 1246 snap 423 seq_num 1246) dirty exists 2024-09-20T22:56:33.752 INFO:tasks.rados.rados.0.smithi084.stdout:3653: left oid 27 (ObjNum 1246 snap 423 seq_num 1246) 2024-09-20T22:56:33.752 INFO:tasks.rados.rados.0.smithi084.stdout:3651: done (5 left) 2024-09-20T22:56:33.752 INFO:tasks.rados.rados.0.smithi084.stdout:3652: done (4 left) 2024-09-20T22:56:33.752 INFO:tasks.rados.rados.0.smithi084.stdout:3653: done (3 left) 2024-09-20T22:56:33.752 INFO:tasks.rados.rados.0.smithi084.stdout:3657: copy_from oid 6 from oid 50 current snap is 423 2024-09-20T22:56:33.752 INFO:tasks.rados.rados.0.smithi084.stdout:3655: finishing write tid 1 to smithi08477203-7 2024-09-20T22:56:33.752 INFO:tasks.rados.rados.0.smithi084.stdout:3654: expect (ObjNum 1066 snap 353 seq_num 1066) 2024-09-20T22:56:33.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:33 smithi154 ceph-mon[33264]: pgmap v1269: 158 pgs: 2 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 1 active+remapped+backfill_toofull, 151 active+clean; 688 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 13 MiB/s rd, 3.7 MiB/s wr, 24 op/s 2024-09-20T22:56:33.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:33 smithi154 ceph-mon[33264]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:56:33.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:33 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/719369520' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T22:56:33.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:33 smithi154 ceph-mon[33264]: osdmap e912: 8 total, 8 up, 7 in 2024-09-20T22:56:33.836 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:56:33.859 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:33.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:33.860 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:56:33.882 INFO:tasks.rados.rados.0.smithi084.stdout:3655: finishing write tid 2 to smithi08477203-7 2024-09-20T22:56:33.882 INFO:tasks.rados.rados.0.smithi084.stdout:3656: finishing copy_from to smithi08477203-48 2024-09-20T22:56:33.882 INFO:tasks.rados.rados.0.smithi084.stdout:3656: got expected ENOENT (src dne) 2024-09-20T22:56:33.882 INFO:tasks.rados.rados.0.smithi084.stdout:3656: finishing copy_from racing read to smithi08477203-48 2024-09-20T22:56:33.882 INFO:tasks.rados.rados.0.smithi084.stdout:3655: finishing write tid 3 to smithi08477203-7 2024-09-20T22:56:33.882 INFO:tasks.rados.rados.0.smithi084.stdout:3655: finishing write tid 4 to smithi08477203-7 2024-09-20T22:56:33.882 INFO:tasks.rados.rados.0.smithi084.stdout:3655: oid 7 updating version 0 to 1949 2024-09-20T22:56:33.882 INFO:tasks.rados.rados.0.smithi084.stdout:3655: oid 7 version 1949 is already newer than 1948 2024-09-20T22:56:33.883 INFO:tasks.rados.rados.0.smithi084.stdout:3655: oid 7 version 1949 is already newer than 1947 2024-09-20T22:56:33.883 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 7 v 1949 (ObjNum 1247 snap 423 seq_num 1247) dirty exists 2024-09-20T22:56:33.883 INFO:tasks.rados.rados.0.smithi084.stdout:3655: left oid 7 (ObjNum 1247 snap 423 seq_num 1247) 2024-09-20T22:56:33.883 INFO:tasks.rados.rados.0.smithi084.stdout:3654: done (3 left) 2024-09-20T22:56:33.883 INFO:tasks.rados.rados.0.smithi084.stdout:3655: done (2 left) 2024-09-20T22:56:33.883 INFO:tasks.rados.rados.0.smithi084.stdout:3656: done (1 left) 2024-09-20T22:56:33.883 INFO:tasks.rados.rados.0.smithi084.stdout:3658: write oid 8 current snap is 423 2024-09-20T22:56:33.883 INFO:tasks.rados.rados.0.smithi084.stdout:3657: finishing copy_from racing read to smithi08477203-6 2024-09-20T22:56:33.883 INFO:tasks.rados.rados.0.smithi084.stdout:3657: finishing copy_from to smithi08477203-6 2024-09-20T22:56:33.883 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 6 v 1891 (ObjNum 1128 snap 381 seq_num 1128) dirty exists 2024-09-20T22:56:33.883 INFO:tasks.rados.rados.0.smithi084.stdout:3658: seq_num 1248 ranges {575136=592644,1663410=639100,2705542=269594} 2024-09-20T22:56:33.888 INFO:tasks.rados.rados.0.smithi084.stdout:3658: writing smithi08477203-8 from 575136 to 1167780 tid 1 2024-09-20T22:56:33.894 INFO:tasks.rados.rados.0.smithi084.stdout:3658: writing smithi08477203-8 from 1663410 to 2302510 tid 2 2024-09-20T22:56:33.899 INFO:tasks.rados.rados.0.smithi084.stdout:3658: writing smithi08477203-8 from 2705542 to 2975136 tid 3 2024-09-20T22:56:33.899 INFO:tasks.rados.rados.0.smithi084.stdout:3657: done (1 left) 2024-09-20T22:56:33.899 INFO:tasks.rados.rados.0.smithi084.stdout:3659: read oid 30 snap -1 2024-09-20T22:56:33.899 INFO:tasks.rados.rados.0.smithi084.stdout:3659: expect (ObjNum 1149 snap 394 seq_num 1149) 2024-09-20T22:56:33.899 INFO:tasks.rados.rados.0.smithi084.stdout:3660: write oid 20 current snap is 423 2024-09-20T22:56:33.899 INFO:tasks.rados.rados.0.smithi084.stdout:3658: finishing write tid 1 to smithi08477203-8 2024-09-20T22:56:33.899 INFO:tasks.rados.rados.0.smithi084.stdout:3660: seq_num 1249 ranges {591943=667893,1721892=782758} 2024-09-20T22:56:33.905 INFO:tasks.rados.rados.0.smithi084.stdout:3660: writing smithi08477203-20 from 591943 to 1259836 tid 1 2024-09-20T22:56:33.913 INFO:tasks.rados.rados.0.smithi084.stdout:3660: writing smithi08477203-20 from 1721892 to 2504650 tid 2 2024-09-20T22:56:33.913 INFO:tasks.rados.rados.0.smithi084.stdout:3661: snap_create 2024-09-20T22:56:33.913 INFO:tasks.rados.rados.0.smithi084.stdout:3658: finishing write tid 2 to smithi08477203-8 2024-09-20T22:56:33.913 INFO:tasks.rados.rados.0.smithi084.stdout:3658: finishing write tid 3 to smithi08477203-8 2024-09-20T22:56:33.913 INFO:tasks.rados.rados.0.smithi084.stdout:3658: finishing write tid 4 to smithi08477203-8 2024-09-20T22:56:33.913 INFO:tasks.rados.rados.0.smithi084.stdout:3658: finishing write tid 5 to smithi08477203-8 2024-09-20T22:56:33.913 INFO:tasks.rados.rados.0.smithi084.stdout:3658: oid 8 updating version 0 to 2608 2024-09-20T22:56:33.913 INFO:tasks.rados.rados.0.smithi084.stdout:3658: oid 8 version 2608 is already newer than 2607 2024-09-20T22:56:33.913 INFO:tasks.rados.rados.0.smithi084.stdout:3658: oid 8 updating version 2608 to 2609 2024-09-20T22:56:33.913 INFO:tasks.rados.rados.0.smithi084.stdout:3658: oid 8 version 2609 is already newer than 2606 2024-09-20T22:56:33.913 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 8 v 2609 (ObjNum 1248 snap 423 seq_num 1248) dirty exists 2024-09-20T22:56:33.913 INFO:tasks.rados.rados.0.smithi084.stdout:3658: left oid 8 (ObjNum 1248 snap 423 seq_num 1248) 2024-09-20T22:56:33.913 INFO:tasks.rados.rados.0.smithi084.stdout:3659: expect (ObjNum 1149 snap 394 seq_num 1149) 2024-09-20T22:56:33.961 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:56:33.982 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:33.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:33.983 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:56:34.084 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:56:34.101 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-20T22:56:34.101 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:56:34.109 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:34.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:34.110 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:56:34.162 INFO:tasks.rados.rados.0.smithi084.stdout:3660: finishing write tid 1 to smithi08477203-20 2024-09-20T22:56:34.163 INFO:tasks.rados.rados.0.smithi084.stdout:3660: finishing write tid 2 to smithi08477203-20 2024-09-20T22:56:34.163 INFO:tasks.rados.rados.0.smithi084.stdout:3660: finishing write tid 3 to smithi08477203-20 2024-09-20T22:56:34.163 INFO:tasks.rados.rados.0.smithi084.stdout:3660: finishing write tid 4 to smithi08477203-20 2024-09-20T22:56:34.163 INFO:tasks.rados.rados.0.smithi084.stdout:3660: oid 20 updating version 0 to 1950 2024-09-20T22:56:34.163 INFO:tasks.rados.rados.0.smithi084.stdout:3660: oid 20 updating version 1950 to 1951 2024-09-20T22:56:34.163 INFO:tasks.rados.rados.0.smithi084.stdout:3660: oid 20 updating version 1951 to 1952 2024-09-20T22:56:34.164 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 20 v 1952 (ObjNum 1249 snap 423 seq_num 1249) dirty exists 2024-09-20T22:56:34.164 INFO:tasks.rados.rados.0.smithi084.stdout:3660: left oid 20 (ObjNum 1249 snap 423 seq_num 1249) 2024-09-20T22:56:34.210 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:56:34.231 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:34.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:34.232 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:56:34.331 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:56:34.333 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:56:34.333 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:56:34.358 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:34.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:34.359 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:56:34.453 INFO:tasks.rados.rados.0.smithi084.stdout:3658: done (3 left) 2024-09-20T22:56:34.454 INFO:tasks.rados.rados.0.smithi084.stdout:3659: done (2 left) 2024-09-20T22:56:34.454 INFO:tasks.rados.rados.0.smithi084.stdout:3660: done (1 left) 2024-09-20T22:56:34.454 INFO:tasks.rados.rados.0.smithi084.stdout:3661: done (0 left) 2024-09-20T22:56:34.454 INFO:tasks.rados.rados.0.smithi084.stdout:3662: delete oid 40 current snap is 424 2024-09-20T22:56:34.457 INFO:tasks.rados.rados.0.smithi084.stdout:3662: done (0 left) 2024-09-20T22:56:34.457 INFO:tasks.rados.rados.0.smithi084.stdout:3663: rollback oid 15 current snap is 424 2024-09-20T22:56:34.457 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 15 to 364 2024-09-20T22:56:34.457 INFO:tasks.rados.rados.0.smithi084.stdout:3664: copy_from oid 49 from oid 21 current snap is 424 2024-09-20T22:56:34.458 INFO:tasks.rados.rados.0.smithi084.stdout:3665: read oid 26 snap 421 2024-09-20T22:56:34.458 INFO:tasks.rados.rados.0.smithi084.stdout:3665: expect (ObjNum 906 snap 297 seq_num 906) 2024-09-20T22:56:34.458 INFO:tasks.rados.rados.0.smithi084.stdout:3666: rollback oid 7 current snap is 424 2024-09-20T22:56:34.458 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 7 to 387 2024-09-20T22:56:34.458 INFO:tasks.rados.rados.0.smithi084.stdout:3667: copy_from oid 20 from oid 47 current snap is 424 2024-09-20T22:56:34.458 INFO:tasks.rados.rados.0.smithi084.stdout:3668: read oid 45 snap -1 2024-09-20T22:56:34.458 INFO:tasks.rados.rados.0.smithi084.stdout:3668: expect (ObjNum 1025 snap 333 seq_num 1025) 2024-09-20T22:56:34.458 INFO:tasks.rados.rados.0.smithi084.stdout:3669: delete oid 9 current snap is 424 2024-09-20T22:56:34.460 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:56:34.464 INFO:tasks.rados.rados.0.smithi084.stdout:3666: finishing rollback tid 0 to smithi08477203-7 2024-09-20T22:56:34.464 INFO:tasks.rados.rados.0.smithi084.stdout:3663: finishing rollback tid 0 to smithi08477203-15 2024-09-20T22:56:34.465 INFO:tasks.rados.rados.0.smithi084.stdout:3663: finishing rollback tid 1 to smithi08477203-15 2024-09-20T22:56:34.465 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 15 v 2611 (ObjNum 33 snap 0 seq_num 6) dirty dne 2024-09-20T22:56:34.465 INFO:tasks.rados.rados.0.smithi084.stdout:3666: finishing rollback tid 1 to smithi08477203-7 2024-09-20T22:56:34.465 INFO:tasks.rados.rados.0.smithi084.stdout:3663: done (6 left) 2024-09-20T22:56:34.465 INFO:tasks.rados.rados.0.smithi084.stdout:3669: done (5 left) 2024-09-20T22:56:34.465 INFO:tasks.rados.rados.0.smithi084.stdout:3670: read oid 30 snap -1 2024-09-20T22:56:34.465 INFO:tasks.rados.rados.0.smithi084.stdout:3670: expect (ObjNum 1149 snap 394 seq_num 1149) 2024-09-20T22:56:34.465 INFO:tasks.rados.rados.0.smithi084.stdout:3671: copy_from oid 50 from oid 15 current snap is 424 2024-09-20T22:56:34.465 INFO:tasks.rados.rados.0.smithi084.stdout:3672: read oid 42 snap -1 2024-09-20T22:56:34.465 INFO:tasks.rados.rados.0.smithi084.stdout:3672: expect deleted 2024-09-20T22:56:34.465 INFO:tasks.rados.rados.0.smithi084.stdout:3673: write (excl) oid 48 current snap is 424 2024-09-20T22:56:34.465 INFO:tasks.rados.rados.0.smithi084.stdout:3673: seq_num 1252 ranges {642364=493640,1897338=646438} 2024-09-20T22:56:34.471 INFO:tasks.rados.rados.0.smithi084.stdout:3673: writing smithi08477203-48 from 642364 to 1136004 tid 1 2024-09-20T22:56:34.482 INFO:tasks.rados.rados.0.smithi084.stdout:3673: writing smithi08477203-48 from 1897338 to 2543776 tid 2 2024-09-20T22:56:34.482 INFO:tasks.rados.rados.0.smithi084.stdout:3674: snap_create 2024-09-20T22:56:34.482 INFO:tasks.rados.rados.0.smithi084.stdout:3666: finishing rollback tid 2 to smithi08477203-7 2024-09-20T22:56:34.482 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 7 v 1952 (ObjNum 795 snap 254 seq_num 795) dirty exists 2024-09-20T22:56:34.482 INFO:tasks.rados.rados.0.smithi084.stdout:3664: finishing copy_from racing read to smithi08477203-49 2024-09-20T22:56:34.482 INFO:tasks.rados.rados.0.smithi084.stdout:3670: expect (ObjNum 1149 snap 394 seq_num 1149) 2024-09-20T22:56:34.493 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:34.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:34.494 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:56:34.594 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:56:34.615 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:34.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:34.616 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:56:34.717 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:56:34.723 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:34 smithi084 ceph-mon[40024]: osdmap e913: 8 total, 8 up, 7 in 2024-09-20T22:56:34.723 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:34 smithi084 ceph-mon[29774]: osdmap e913: 8 total, 8 up, 7 in 2024-09-20T22:56:34.734 INFO:tasks.rados.rados.0.smithi084.stdout:3664: finishing copy_from to smithi08477203-49 2024-09-20T22:56:34.734 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 49 v 2592 (ObjNum 1215 snap 413 seq_num 1215) dirty exists 2024-09-20T22:56:34.734 INFO:tasks.rados.rados.0.smithi084.stdout:3673: finishing write tid 1 to smithi08477203-48 2024-09-20T22:56:34.734 INFO:tasks.rados.rados.0.smithi084.stdout:3668: expect (ObjNum 1025 snap 333 seq_num 1025) 2024-09-20T22:56:34.740 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:34.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:34.741 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:56:34.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:34 smithi154 ceph-mon[33264]: osdmap e913: 8 total, 8 up, 7 in 2024-09-20T22:56:34.841 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:56:34.865 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:34.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:34.866 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:56:34.966 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:56:34.990 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:34.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:34.991 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:56:34.995 INFO:tasks.rados.rados.0.smithi084.stdout:3671: finishing copy_from to smithi08477203-50 2024-09-20T22:56:34.995 INFO:tasks.rados.rados.0.smithi084.stdout:3671: got expected ENOENT (src dne) 2024-09-20T22:56:34.995 INFO:tasks.rados.rados.0.smithi084.stdout:3671: finishing copy_from racing read to smithi08477203-50 2024-09-20T22:56:34.995 INFO:tasks.rados.rados.0.smithi084.stdout:3665: expect (ObjNum 906 snap 297 seq_num 906) 2024-09-20T22:56:35.079 INFO:teuthology.orchestra.run.smithi084.stderr:no valid command found; 10 closest matches: 2024-09-20T22:56:35.080 INFO:teuthology.orchestra.run.smithi084.stderr:0 2024-09-20T22:56:35.080 INFO:teuthology.orchestra.run.smithi084.stderr:1 2024-09-20T22:56:35.080 INFO:teuthology.orchestra.run.smithi084.stderr:2 2024-09-20T22:56:35.080 INFO:teuthology.orchestra.run.smithi084.stderr:abort 2024-09-20T22:56:35.080 INFO:teuthology.orchestra.run.smithi084.stderr:assert 2024-09-20T22:56:35.080 INFO:teuthology.orchestra.run.smithi084.stderr:bench [] [] [] [] 2024-09-20T22:56:35.080 INFO:teuthology.orchestra.run.smithi084.stderr:bluefs debug_inject_read_zeros 2024-09-20T22:56:35.080 INFO:teuthology.orchestra.run.smithi084.stderr:bluefs files list 2024-09-20T22:56:35.080 INFO:teuthology.orchestra.run.smithi084.stderr:bluefs stats 2024-09-20T22:56:35.080 INFO:teuthology.orchestra.run.smithi084.stderr:bluestore allocator dump block 2024-09-20T22:56:35.080 INFO:teuthology.orchestra.run.smithi084.stderr:Error EINVAL: invalid command 2024-09-20T22:56:35.092 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:56:35.115 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:35.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:35.116 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:56:35.193 INFO:tasks.rados.rados.0.smithi084.stdout:3667: finishing copy_from racing read to smithi08477203-20 2024-09-20T22:56:35.193 INFO:tasks.rados.rados.0.smithi084.stdout:3667: finishing copy_from to smithi08477203-20 2024-09-20T22:56:35.193 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 20 v 2589 (ObjNum 1066 snap 353 seq_num 1066) dirty exists 2024-09-20T22:56:35.193 INFO:tasks.rados.rados.0.smithi084.stdout:3673: finishing write tid 2 to smithi08477203-48 2024-09-20T22:56:35.193 INFO:tasks.rados.rados.0.smithi084.stdout:3673: finishing write tid 3 to smithi08477203-48 2024-09-20T22:56:35.193 INFO:tasks.rados.rados.0.smithi084.stdout:3673: finishing write tid 4 to smithi08477203-48 2024-09-20T22:56:35.193 INFO:tasks.rados.rados.0.smithi084.stdout:3673: oid 48 updating version 0 to 1953 2024-09-20T22:56:35.193 INFO:tasks.rados.rados.0.smithi084.stdout:3673: oid 48 updating version 1953 to 1955 2024-09-20T22:56:35.193 INFO:tasks.rados.rados.0.smithi084.stdout:3673: oid 48 version 1955 is already newer than 1954 2024-09-20T22:56:35.193 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 48 v 1955 (ObjNum 1252 snap 424 seq_num 1252) dirty exists 2024-09-20T22:56:35.193 INFO:tasks.rados.rados.0.smithi084.stdout:3673: left oid 48 (ObjNum 1252 snap 424 seq_num 1252) 2024-09-20T22:56:35.217 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:56:35.240 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:35.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:35.241 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:56:35.249 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:56:35.249 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:56:35.266 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:56:35.266 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:56:35.282 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:56:35.282 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:56:35.298 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-20T22:56:35.298 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T22:56:35.315 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:56:35.315 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:56:35.331 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:56:35.331 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:56:35.342 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:56:35.348 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:56:35.348 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:56:35.363 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:35.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:35.364 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:56:35.366 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:56:35.366 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:56:35.463 INFO:tasks.rados.rados.0.smithi084.stdout:3664: done (9 left) 2024-09-20T22:56:35.463 INFO:tasks.rados.rados.0.smithi084.stdout:3665: done (8 left) 2024-09-20T22:56:35.463 INFO:tasks.rados.rados.0.smithi084.stdout:3666: done (7 left) 2024-09-20T22:56:35.463 INFO:tasks.rados.rados.0.smithi084.stdout:3667: done (6 left) 2024-09-20T22:56:35.463 INFO:tasks.rados.rados.0.smithi084.stdout:3668: done (5 left) 2024-09-20T22:56:35.464 INFO:tasks.rados.rados.0.smithi084.stdout:3670: done (4 left) 2024-09-20T22:56:35.464 INFO:tasks.rados.rados.0.smithi084.stdout:3671: done (3 left) 2024-09-20T22:56:35.464 INFO:tasks.rados.rados.0.smithi084.stdout:3672: done (2 left) 2024-09-20T22:56:35.464 INFO:tasks.rados.rados.0.smithi084.stdout:3673: done (1 left) 2024-09-20T22:56:35.464 INFO:tasks.rados.rados.0.smithi084.stdout:3674: done (0 left) 2024-09-20T22:56:35.464 INFO:tasks.rados.rados.0.smithi084.stdout:3675: rollback oid 22 current snap is 425 2024-09-20T22:56:35.464 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 22 to 420 2024-09-20T22:56:35.464 INFO:tasks.rados.rados.0.smithi084.stdout:3676: write (excl) oid 11 current snap is 425 2024-09-20T22:56:35.464 INFO:tasks.rados.rados.0.smithi084.stdout:3676: seq_num 1253 ranges {659171=568889,1839467=790096} 2024-09-20T22:56:35.464 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:56:35.470 INFO:tasks.rados.rados.0.smithi084.stdout:3676: writing smithi08477203-11 from 659171 to 1228060 tid 1 2024-09-20T22:56:35.478 INFO:tasks.rados.rados.0.smithi084.stdout:3676: writing smithi08477203-11 from 1839467 to 2629563 tid 2 2024-09-20T22:56:35.478 INFO:tasks.rados.rados.0.smithi084.stdout:3677: copy_from oid 20 from oid 14 current snap is 425 2024-09-20T22:56:35.478 INFO:tasks.rados.rados.0.smithi084.stdout:3675: finishing rollback tid 0 to smithi08477203-22 2024-09-20T22:56:35.478 INFO:tasks.rados.rados.0.smithi084.stdout:3675: finishing rollback tid 1 to smithi08477203-22 2024-09-20T22:56:35.478 INFO:tasks.rados.rados.0.smithi084.stdout:3675: finishing rollback tid 2 to smithi08477203-22 2024-09-20T22:56:35.478 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 22 v 2502 (ObjNum 1187 snap 408 seq_num 1187) dirty exists 2024-09-20T22:56:35.478 INFO:tasks.rados.rados.0.smithi084.stdout:3676: finishing write tid 1 to smithi08477203-11 2024-09-20T22:56:35.478 INFO:tasks.rados.rados.0.smithi084.stdout:3675: done (2 left) 2024-09-20T22:56:35.478 INFO:tasks.rados.rados.0.smithi084.stdout:3678: snap_remove snap 312 2024-09-20T22:56:35.482 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd unset noscrub 2024-09-20T22:56:35.489 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:35.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:35.490 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:56:35.591 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:56:35.599 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:35 smithi084 ceph-mon[40024]: pgmap v1272: 158 pgs: 2 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 1 active+remapped+backfill_toofull, 152 active+clean; 707 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 33 MiB/s rd, 3.1 MiB/s wr, 33 op/s 2024-09-20T22:56:35.599 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:35 smithi084 ceph-mon[40024]: osdmap e914: 8 total, 8 up, 7 in 2024-09-20T22:56:35.599 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:35 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/1327577556' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:56:35.599 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:35 smithi084 ceph-mon[29774]: pgmap v1272: 158 pgs: 2 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 1 active+remapped+backfill_toofull, 152 active+clean; 707 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 33 MiB/s rd, 3.1 MiB/s wr, 33 op/s 2024-09-20T22:56:35.599 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:35 smithi084 ceph-mon[29774]: osdmap e914: 8 total, 8 up, 7 in 2024-09-20T22:56:35.599 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:35 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/1327577556' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:56:35.613 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:35.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:35.614 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:56:35.715 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:56:35.738 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:35.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:35.739 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:56:35.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:35 smithi154 ceph-mon[33264]: pgmap v1272: 158 pgs: 2 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 1 active+remapped+backfill_toofull, 152 active+clean; 707 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 33 MiB/s rd, 3.1 MiB/s wr, 33 op/s 2024-09-20T22:56:35.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:35 smithi154 ceph-mon[33264]: osdmap e914: 8 total, 8 up, 7 in 2024-09-20T22:56:35.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:35 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/1327577556' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:56:35.840 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:56:35.863 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:35.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:35.864 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:56:35.883 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-20T22:56:35.883 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-20T22:56:35.883 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph tell osd.1 'trim stale osdmaps' 2024-09-20T22:56:35.965 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:56:35.988 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:35.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:35.989 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:56:36.090 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:56:36.114 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:36.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:36.115 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:56:36.216 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:56:36.237 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:36.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:36.238 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:56:36.338 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:56:36.359 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:36.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:36.360 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:56:36.461 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:56:36.469 INFO:tasks.rados.rados.0.smithi084.stdout:3678: done (2 left) 2024-09-20T22:56:36.470 INFO:tasks.rados.rados.0.smithi084.stdout:3679: rollback oid 13 current snap is 425 2024-09-20T22:56:36.470 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 13 to 365 2024-09-20T22:56:36.470 INFO:tasks.rados.rados.0.smithi084.stdout:3680: read oid 12 snap 370 2024-09-20T22:56:36.470 INFO:tasks.rados.rados.0.smithi084.stdout:3680: expect (ObjNum 795 snap 254 seq_num 795) 2024-09-20T22:56:36.470 INFO:tasks.rados.rados.0.smithi084.stdout:3681: delete oid 8 current snap is 425 2024-09-20T22:56:36.470 INFO:tasks.rados.rados.0.smithi084.stdout:3676: finishing write tid 2 to smithi08477203-11 2024-09-20T22:56:36.470 INFO:tasks.rados.rados.0.smithi084.stdout:3676: finishing write tid 3 to smithi08477203-11 2024-09-20T22:56:36.470 INFO:tasks.rados.rados.0.smithi084.stdout:3676: finishing write tid 4 to smithi08477203-11 2024-09-20T22:56:36.470 INFO:tasks.rados.rados.0.smithi084.stdout:3676: oid 11 updating version 0 to 2665 2024-09-20T22:56:36.470 INFO:tasks.rados.rados.0.smithi084.stdout:3676: oid 11 version 2665 is already newer than 2663 2024-09-20T22:56:36.470 INFO:tasks.rados.rados.0.smithi084.stdout:3676: oid 11 version 2665 is already newer than 2664 2024-09-20T22:56:36.470 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 11 v 2665 (ObjNum 1253 snap 425 seq_num 1253) dirty exists 2024-09-20T22:56:36.470 INFO:tasks.rados.rados.0.smithi084.stdout:3676: left oid 11 (ObjNum 1253 snap 425 seq_num 1253) 2024-09-20T22:56:36.471 INFO:tasks.rados.rados.0.smithi084.stdout:3677: finishing copy_from racing read to smithi08477203-20 2024-09-20T22:56:36.471 INFO:tasks.rados.rados.0.smithi084.stdout:3677: finishing copy_from to smithi08477203-20 2024-09-20T22:56:36.471 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 20 v 1949 (ObjNum 618 snap 184 seq_num 618) dirty exists 2024-09-20T22:56:36.484 INFO:tasks.rados.rados.0.smithi084.stdout:3680: expect (ObjNum 795 snap 254 seq_num 795) 2024-09-20T22:56:36.493 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:36.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:36.494 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:56:36.595 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:56:36.596 INFO:tasks.rados.rados.0.smithi084.stdout:3679: finishing rollback tid 0 to smithi08477203-13 2024-09-20T22:56:36.596 INFO:tasks.rados.rados.0.smithi084.stdout:3679: finishing rollback tid 1 to smithi08477203-13 2024-09-20T22:56:36.596 INFO:tasks.rados.rados.0.smithi084.stdout:3676: done (4 left) 2024-09-20T22:56:36.596 INFO:tasks.rados.rados.0.smithi084.stdout:3677: done (3 left) 2024-09-20T22:56:36.596 INFO:tasks.rados.rados.0.smithi084.stdout:3680: done (2 left) 2024-09-20T22:56:36.596 INFO:tasks.rados.rados.0.smithi084.stdout:3681: done (1 left) 2024-09-20T22:56:36.597 INFO:tasks.rados.rados.0.smithi084.stdout:3682: read oid 9 snap -1 2024-09-20T22:56:36.597 INFO:tasks.rados.rados.0.smithi084.stdout:3682: expect deleted 2024-09-20T22:56:36.597 INFO:tasks.rados.rados.0.smithi084.stdout:3683: read oid 37 snap -1 2024-09-20T22:56:36.597 INFO:tasks.rados.rados.0.smithi084.stdout:3683: expect (ObjNum 1189 snap 408 seq_num 1189) 2024-09-20T22:56:36.597 INFO:tasks.rados.rados.0.smithi084.stdout:3684: read oid 34 snap 342 2024-09-20T22:56:36.597 INFO:tasks.rados.rados.0.smithi084.stdout:3684: expect deleted 2024-09-20T22:56:36.597 INFO:tasks.rados.rados.0.smithi084.stdout:3685: delete oid 20 current snap is 425 2024-09-20T22:56:36.597 INFO:tasks.rados.rados.0.smithi084.stdout:3679: finishing rollback tid 2 to smithi08477203-13 2024-09-20T22:56:36.597 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 13 v 3002 (ObjNum 1074 snap 358 seq_num 1074) dirty exists 2024-09-20T22:56:36.598 INFO:tasks.rados.rados.0.smithi084.stdout:3679: done (4 left) 2024-09-20T22:56:36.598 INFO:tasks.rados.rados.0.smithi084.stdout:3682: done (3 left) 2024-09-20T22:56:36.599 INFO:tasks.rados.rados.0.smithi084.stdout:3684: done (2 left) 2024-09-20T22:56:36.599 INFO:tasks.rados.rados.0.smithi084.stdout:3685: done (1 left) 2024-09-20T22:56:36.599 INFO:tasks.rados.rados.0.smithi084.stdout:3686: copy_from oid 21 from oid 43 current snap is 425 2024-09-20T22:56:36.599 INFO:tasks.rados.rados.0.smithi084.stdout:3687: read oid 10 snap -1 2024-09-20T22:56:36.599 INFO:tasks.rados.rados.0.smithi084.stdout:3687: expect (ObjNum 1244 snap 422 seq_num 1244) 2024-09-20T22:56:36.599 INFO:tasks.rados.rados.0.smithi084.stdout:3688: copy_from oid 19 from oid 28 current snap is 425 2024-09-20T22:56:36.599 INFO:tasks.rados.rados.0.smithi084.stdout:3689: read oid 33 snap -1 2024-09-20T22:56:36.599 INFO:tasks.rados.rados.0.smithi084.stdout:3689: expect (ObjNum 1202 snap 411 seq_num 1202) 2024-09-20T22:56:36.599 INFO:tasks.rados.rados.0.smithi084.stdout:3690: delete oid 17 current snap is 425 2024-09-20T22:56:36.600 INFO:tasks.rados.rados.0.smithi084.stdout:3686: finishing copy_from to smithi08477203-21 2024-09-20T22:56:36.600 INFO:tasks.rados.rados.0.smithi084.stdout:3686: got expected ENOENT (src dne) 2024-09-20T22:56:36.601 INFO:tasks.rados.rados.0.smithi084.stdout:3686: finishing copy_from racing read to smithi08477203-21 2024-09-20T22:56:36.602 INFO:tasks.rados.rados.0.smithi084.stdout:3689: expect (ObjNum 1202 snap 411 seq_num 1202) 2024-09-20T22:56:36.622 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:36.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:36.623 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:56:36.674 INFO:tasks.rados.rados.0.smithi084.stdout:3686: done (5 left) 2024-09-20T22:56:36.674 INFO:tasks.rados.rados.0.smithi084.stdout:3689: done (4 left) 2024-09-20T22:56:36.674 INFO:tasks.rados.rados.0.smithi084.stdout:3690: done (3 left) 2024-09-20T22:56:36.674 INFO:tasks.rados.rados.0.smithi084.stdout:3691: snap_create 2024-09-20T22:56:36.674 INFO:tasks.rados.rados.0.smithi084.stdout:3683: expect (ObjNum 1189 snap 408 seq_num 1189) 2024-09-20T22:56:36.724 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:56:36.725 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:36 smithi084 ceph-mon[29774]: osdmap e915: 8 total, 8 up, 7 in 2024-09-20T22:56:36.725 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:36 smithi084 ceph-mon[40024]: osdmap e915: 8 total, 8 up, 7 in 2024-09-20T22:56:36.727 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-20T22:56:36.747 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:36.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:36.748 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:56:36.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:36 smithi154 ceph-mon[33264]: osdmap e915: 8 total, 8 up, 7 in 2024-09-20T22:56:36.849 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:56:36.870 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:36.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:36.871 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:56:36.971 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:56:36.975 INFO:tasks.rados.rados.0.smithi084.stdout:3687: expect (ObjNum 1244 snap 422 seq_num 1244) 2024-09-20T22:56:36.992 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:36.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:36.993 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:56:37.094 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:56:37.108 INFO:tasks.rados.rados.0.smithi084.stdout:3688: finishing copy_from racing read to smithi08477203-19 2024-09-20T22:56:37.108 INFO:tasks.rados.rados.0.smithi084.stdout:3688: finishing copy_from to smithi08477203-19 2024-09-20T22:56:37.108 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 19 v 2949 (ObjNum 1230 snap 419 seq_num 1230) dirty exists 2024-09-20T22:56:37.116 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:37.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:37.117 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:56:37.143 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:56:37.218 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:56:37.239 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:37.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:37.240 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:56:37.341 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:56:37.367 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:37.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:37.368 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:56:37.468 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:56:37.469 INFO:tasks.rados.rados.0.smithi084.stdout:3683: done (3 left) 2024-09-20T22:56:37.469 INFO:tasks.rados.rados.0.smithi084.stdout:3687: done (2 left) 2024-09-20T22:56:37.469 INFO:tasks.rados.rados.0.smithi084.stdout:3688: done (1 left) 2024-09-20T22:56:37.469 INFO:tasks.rados.rados.0.smithi084.stdout:3691: done (0 left) 2024-09-20T22:56:37.469 INFO:tasks.rados.rados.0.smithi084.stdout:3692: snap_create 2024-09-20T22:56:37.494 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:37.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:37.495 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:56:37.590 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:56:37.590 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:56:37.596 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:56:37.622 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:37.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:37.623 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:56:37.715 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:37 smithi084 ceph-mon[40024]: pgmap v1275: 158 pgs: 2 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 1 active+remapped+backfill_toofull, 152 active+clean; 707 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 33 MiB/s rd, 3.1 MiB/s wr, 33 op/s 2024-09-20T22:56:37.715 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:37 smithi084 ceph-mon[40024]: osdmap e916: 8 total, 8 up, 7 in 2024-09-20T22:56:37.716 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:37 smithi084 ceph-mon[29774]: pgmap v1275: 158 pgs: 2 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 1 active+remapped+backfill_toofull, 152 active+clean; 707 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 33 MiB/s rd, 3.1 MiB/s wr, 33 op/s 2024-09-20T22:56:37.716 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:37 smithi084 ceph-mon[29774]: osdmap e916: 8 total, 8 up, 7 in 2024-09-20T22:56:37.724 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:56:37.750 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:37.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:37.751 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:56:37.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:37 smithi154 ceph-mon[33264]: pgmap v1275: 158 pgs: 2 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 1 active+remapped+backfill_toofull, 152 active+clean; 707 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 33 MiB/s rd, 3.1 MiB/s wr, 33 op/s 2024-09-20T22:56:37.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:37 smithi154 ceph-mon[33264]: osdmap e916: 8 total, 8 up, 7 in 2024-09-20T22:56:37.852 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:56:37.874 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:37.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:37.875 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:56:37.975 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:56:38.001 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:38.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:38.002 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:56:38.103 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:56:38.124 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:38.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:38.125 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:56:38.139 INFO:teuthology.orchestra.run.smithi084.stderr:no valid command found; 10 closest matches: 2024-09-20T22:56:38.139 INFO:teuthology.orchestra.run.smithi084.stderr:0 2024-09-20T22:56:38.139 INFO:teuthology.orchestra.run.smithi084.stderr:1 2024-09-20T22:56:38.139 INFO:teuthology.orchestra.run.smithi084.stderr:2 2024-09-20T22:56:38.139 INFO:teuthology.orchestra.run.smithi084.stderr:abort 2024-09-20T22:56:38.139 INFO:teuthology.orchestra.run.smithi084.stderr:assert 2024-09-20T22:56:38.139 INFO:teuthology.orchestra.run.smithi084.stderr:bench [] [] [] [] 2024-09-20T22:56:38.139 INFO:teuthology.orchestra.run.smithi084.stderr:bluefs debug_inject_read_zeros 2024-09-20T22:56:38.139 INFO:teuthology.orchestra.run.smithi084.stderr:bluefs files list 2024-09-20T22:56:38.140 INFO:teuthology.orchestra.run.smithi084.stderr:bluefs stats 2024-09-20T22:56:38.140 INFO:teuthology.orchestra.run.smithi084.stderr:bluestore allocator dump block 2024-09-20T22:56:38.140 INFO:teuthology.orchestra.run.smithi084.stderr:Error EINVAL: invalid command 2024-09-20T22:56:38.226 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:56:38.253 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:38.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:38.254 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:56:38.350 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is unset 2024-09-20T22:56:38.351 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:56:38 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: ::ffff:172.21.15.154 - - [20/Sep/2024:22:56:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-20T22:56:38.351 INFO:tasks.rados.rados.0.smithi084.stdout:3692: done (0 left) 2024-09-20T22:56:38.351 INFO:tasks.rados.rados.0.smithi084.stdout:3693: snap_remove snap 412 2024-09-20T22:56:38.354 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:56:38.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:38.378 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:38.378 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:56:38.479 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:56:38.502 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:38.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:38.503 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:56:38.604 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:56:38.614 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:38 smithi084 ceph-mon[29774]: osdmap e917: 8 total, 8 up, 7 in 2024-09-20T22:56:38.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:38 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/840381143' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:56:38.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:38 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:56:38.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:38 smithi084 ceph-mon[29774]: purged_snaps scrub starts 2024-09-20T22:56:38.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:38 smithi084 ceph-mon[29774]: purged_snaps scrub ok 2024-09-20T22:56:38.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:38 smithi084 ceph-mon[29774]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:56:38.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:38 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T22:56:38.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:38 smithi084 ceph-mon[29774]: osdmap e918: 8 total, 8 up, 7 in 2024-09-20T22:56:38.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:38 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:56:38.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:38 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:56:38.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:38 smithi084 ceph-mon[40024]: osdmap e917: 8 total, 8 up, 7 in 2024-09-20T22:56:38.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:38 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/840381143' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:56:38.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:38 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:56:38.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:38 smithi084 ceph-mon[40024]: purged_snaps scrub starts 2024-09-20T22:56:38.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:38 smithi084 ceph-mon[40024]: purged_snaps scrub ok 2024-09-20T22:56:38.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:38 smithi084 ceph-mon[40024]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:56:38.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:38 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T22:56:38.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:38 smithi084 ceph-mon[40024]: osdmap e918: 8 total, 8 up, 7 in 2024-09-20T22:56:38.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:38 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:56:38.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:38 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:56:38.629 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:38.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:38.630 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:56:38.730 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:56:38.751 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:38.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:38.752 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:56:38.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:38 smithi154 ceph-mon[33264]: osdmap e917: 8 total, 8 up, 7 in 2024-09-20T22:56:38.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:38 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/840381143' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:56:38.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:38 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:56:38.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:38 smithi154 ceph-mon[33264]: purged_snaps scrub starts 2024-09-20T22:56:38.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:38 smithi154 ceph-mon[33264]: purged_snaps scrub ok 2024-09-20T22:56:38.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:38 smithi154 ceph-mon[33264]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:56:38.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:38 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T22:56:38.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:38 smithi154 ceph-mon[33264]: osdmap e918: 8 total, 8 up, 7 in 2024-09-20T22:56:38.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:38 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:56:38.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:38 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:56:38.853 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:56:38.879 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:38.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:38.880 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:56:38.981 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:56:39.002 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:39.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:39.004 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:56:39.105 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:56:39.123 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-20T22:56:39.124 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-20T22:56:39.130 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:39.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:39.131 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:56:39.142 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-20T22:56:39.232 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:56:39.257 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:39.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:39.258 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:56:39.355 INFO:tasks.rados.rados.0.smithi084.stdout:3693: done (0 left) 2024-09-20T22:56:39.355 INFO:tasks.rados.rados.0.smithi084.stdout:3694: read oid 14 snap -1 2024-09-20T22:56:39.355 INFO:tasks.rados.rados.0.smithi084.stdout:3694: expect (ObjNum 618 snap 184 seq_num 618) 2024-09-20T22:56:39.356 INFO:tasks.rados.rados.0.smithi084.stdout:3695: snap_create 2024-09-20T22:56:39.359 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:56:39.363 INFO:tasks.rados.rados.0.smithi084.stdout:3694: expect (ObjNum 618 snap 184 seq_num 618) 2024-09-20T22:56:39.382 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:39.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:39.383 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:56:39.483 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:56:39.504 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:39.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:39.505 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:56:39.605 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:56:39.630 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:39.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:39.631 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:56:39.732 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:56:39.741 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:39 smithi154 ceph-mon[33264]: pgmap v1278: 158 pgs: 2 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 1 active+remapped+backfill_toofull, 152 active+clean; 707 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-20T22:56:39.741 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:39 smithi154 ceph-mon[33264]: 5.11 scrub starts 2024-09-20T22:56:39.741 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:39 smithi154 ceph-mon[33264]: 5.11 scrub ok 2024-09-20T22:56:39.741 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:39 smithi154 ceph-mon[33264]: osdmap e919: 8 total, 8 up, 7 in 2024-09-20T22:56:39.756 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:39.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:39.757 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:56:39.774 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:39 smithi084 ceph-mon[40024]: pgmap v1278: 158 pgs: 2 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 1 active+remapped+backfill_toofull, 152 active+clean; 707 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-20T22:56:39.774 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:39 smithi084 ceph-mon[40024]: 5.11 scrub starts 2024-09-20T22:56:39.774 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:39 smithi084 ceph-mon[40024]: 5.11 scrub ok 2024-09-20T22:56:39.774 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:39 smithi084 ceph-mon[40024]: osdmap e919: 8 total, 8 up, 7 in 2024-09-20T22:56:39.774 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:39 smithi084 ceph-mon[29774]: pgmap v1278: 158 pgs: 2 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 1 active+remapped+backfill_toofull, 152 active+clean; 707 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-20T22:56:39.774 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:39 smithi084 ceph-mon[29774]: 5.11 scrub starts 2024-09-20T22:56:39.774 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:39 smithi084 ceph-mon[29774]: 5.11 scrub ok 2024-09-20T22:56:39.774 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:39 smithi084 ceph-mon[29774]: osdmap e919: 8 total, 8 up, 7 in 2024-09-20T22:56:39.858 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:56:39.879 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:39.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:39.880 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:56:39.981 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:56:40.002 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:40.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:40.003 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:56:40.004 INFO:journalctl@ceph.iscsi.iscsi.a.smithi154.stdout:Sep 20 22:56:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-iscsi-iscsi-a[59730]: debug there is no tcmu-runner data available 2024-09-20T22:56:40.104 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:56:40.126 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-20T22:56:40.127 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2024-09-20T22:56:40.127 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2024-09-20T22:56:40.127 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-20T22:56:40.127 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:40.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:40.128 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:56:40.229 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:56:40.255 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:40.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:40.256 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:56:40.356 INFO:tasks.rados.rados.0.smithi084.stdout:3694: done (1 left) 2024-09-20T22:56:40.356 INFO:tasks.rados.rados.0.smithi084.stdout:3695: done (0 left) 2024-09-20T22:56:40.356 INFO:tasks.rados.rados.0.smithi084.stdout:3696: delete oid 9 current snap is 428 2024-09-20T22:56:40.356 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:56:40.358 INFO:tasks.rados.rados.0.smithi084.stdout:3696: done (0 left) 2024-09-20T22:56:40.359 INFO:tasks.rados.rados.0.smithi084.stdout:3697: write (excl) oid 15 current snap is 428 2024-09-20T22:56:40.359 INFO:tasks.rados.rados.0.smithi084.stdout:3697: seq_num 1258 ranges {743206=661487,2131877=541092} 2024-09-20T22:56:40.366 INFO:tasks.rados.rados.0.smithi084.stdout:3697: writing smithi08477203-15 from 743206 to 1404693 tid 1 2024-09-20T22:56:40.373 INFO:tasks.rados.rados.0.smithi084.stdout:3697: writing smithi08477203-15 from 2131877 to 2672969 tid 2 2024-09-20T22:56:40.373 INFO:tasks.rados.rados.0.smithi084.stdout:3698: snap_create 2024-09-20T22:56:40.373 INFO:tasks.rados.rados.0.smithi084.stdout:3697: finishing write tid 1 to smithi08477203-15 2024-09-20T22:56:40.377 INFO:tasks.rados.rados.0.smithi084.stdout:3697: finishing write tid 2 to smithi08477203-15 2024-09-20T22:56:40.377 INFO:tasks.rados.rados.0.smithi084.stdout:3697: finishing write tid 3 to smithi08477203-15 2024-09-20T22:56:40.377 INFO:tasks.rados.rados.0.smithi084.stdout:3697: finishing write tid 4 to smithi08477203-15 2024-09-20T22:56:40.377 INFO:tasks.rados.rados.0.smithi084.stdout:3697: oid 15 updating version 0 to 2614 2024-09-20T22:56:40.377 INFO:tasks.rados.rados.0.smithi084.stdout:3697: oid 15 version 2614 is already newer than 2613 2024-09-20T22:56:40.377 INFO:tasks.rados.rados.0.smithi084.stdout:3697: oid 15 updating version 2614 to 2615 2024-09-20T22:56:40.377 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 15 v 2615 (ObjNum 1258 snap 428 seq_num 1258) dirty exists 2024-09-20T22:56:40.377 INFO:tasks.rados.rados.0.smithi084.stdout:3697: left oid 15 (ObjNum 1258 snap 428 seq_num 1258) 2024-09-20T22:56:40.382 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:40.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:40.383 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:56:40.484 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:56:40.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:40.505 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:40.505 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:56:40.606 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:56:40.630 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:40.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:40.631 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:56:40.732 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:56:40.756 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:40.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:40.757 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:56:40.852 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.2 dump_blocked_ops 2024-09-20T22:56:40.857 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:56:40.879 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:56:40.882 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:40.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:40.883 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:56:40.984 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:56:41.004 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:41.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:41.005 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:56:41.106 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:56:41.132 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:41.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:41.133 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:56:41.234 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:56:41.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:41.256 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:41.257 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:56:41.357 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:56:41.361 INFO:tasks.rados.rados.0.smithi084.stdout:3697: done (1 left) 2024-09-20T22:56:41.361 INFO:tasks.rados.rados.0.smithi084.stdout:3698: done (0 left) 2024-09-20T22:56:41.362 INFO:tasks.rados.rados.0.smithi084.stdout:3699: snap_create 2024-09-20T22:56:41.370 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:56:41.370 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:56:41.385 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:41.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:41.386 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:56:41.389 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:56:41.390 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:56:41.390 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:41 smithi084 ceph-mon[40024]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:56:41.390 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:41 smithi084 ceph-mon[40024]: pgmap v1281: 158 pgs: 2 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 1 active+remapped+backfill_toofull, 151 active+clean; 658 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 25 MiB/s rd, 4.7 MiB/s wr, 30 op/s 2024-09-20T22:56:41.390 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:41 smithi084 ceph-mon[40024]: 6.12 scrub starts 2024-09-20T22:56:41.390 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:41 smithi084 ceph-mon[40024]: 6.12 scrub ok 2024-09-20T22:56:41.390 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:41 smithi084 ceph-mon[40024]: osdmap e920: 8 total, 8 up, 7 in 2024-09-20T22:56:41.390 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:41 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/3085795921' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:56:41.391 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:41 smithi084 ceph-mon[29774]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:56:41.391 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:41 smithi084 ceph-mon[29774]: pgmap v1281: 158 pgs: 2 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 1 active+remapped+backfill_toofull, 151 active+clean; 658 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 25 MiB/s rd, 4.7 MiB/s wr, 30 op/s 2024-09-20T22:56:41.391 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:41 smithi084 ceph-mon[29774]: 6.12 scrub starts 2024-09-20T22:56:41.391 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:41 smithi084 ceph-mon[29774]: 6.12 scrub ok 2024-09-20T22:56:41.391 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:41 smithi084 ceph-mon[29774]: osdmap e920: 8 total, 8 up, 7 in 2024-09-20T22:56:41.391 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:41 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/3085795921' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:56:41.395 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:56:41.406 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:56:41.406 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:56:41.423 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-20T22:56:41.424 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T22:56:41.440 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:56:41.441 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:56:41.457 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:56:41.457 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:56:41.475 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:56:41.475 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:56:41.486 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:56:41.491 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:56:41.492 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:56:41.508 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:41.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:41.509 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:56:41.609 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:56:41.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:41.634 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:41.634 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:56:41.667 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-20T22:56:41.735 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:56:41.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:41 smithi154 ceph-mon[33264]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:56:41.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:41 smithi154 ceph-mon[33264]: pgmap v1281: 158 pgs: 2 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 1 active+remapped+backfill_toofull, 151 active+clean; 658 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 25 MiB/s rd, 4.7 MiB/s wr, 30 op/s 2024-09-20T22:56:41.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:41 smithi154 ceph-mon[33264]: 6.12 scrub starts 2024-09-20T22:56:41.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:41 smithi154 ceph-mon[33264]: 6.12 scrub ok 2024-09-20T22:56:41.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:41 smithi154 ceph-mon[33264]: osdmap e920: 8 total, 8 up, 7 in 2024-09-20T22:56:41.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:41 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/3085795921' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:56:41.761 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:41.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:41.762 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:56:41.863 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:56:41.888 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:41.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:41.889 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:56:41.990 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:56:42.016 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:42.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:42.017 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:56:42.118 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:56:42.142 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:42.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:42.143 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:56:42.244 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:56:42.265 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:42.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:42.266 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:56:42.366 INFO:tasks.rados.rados.0.smithi084.stdout:3699: done (0 left) 2024-09-20T22:56:42.366 INFO:tasks.rados.rados.0.smithi084.stdout:3700: snap_remove snap 394 2024-09-20T22:56:42.367 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:56:42.388 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:42.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:42.389 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:56:42.490 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:56:42.511 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:42.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:42.512 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:56:42.587 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:42 smithi084 ceph-mon[29774]: osdmap e921: 8 total, 8 up, 7 in 2024-09-20T22:56:42.587 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:42 smithi084 ceph-mon[29774]: 6.d scrub starts 2024-09-20T22:56:42.587 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:42 smithi084 ceph-mon[29774]: 6.d scrub ok 2024-09-20T22:56:42.587 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:42 smithi084 ceph-mon[29774]: 6.14 scrub starts 2024-09-20T22:56:42.587 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:42 smithi084 ceph-mon[29774]: 6.14 scrub ok 2024-09-20T22:56:42.587 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:42 smithi084 ceph-mon[29774]: 5.1f scrub starts 2024-09-20T22:56:42.587 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:42 smithi084 ceph-mon[29774]: 5.1f scrub ok 2024-09-20T22:56:42.588 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:42 smithi084 ceph-mon[40024]: osdmap e921: 8 total, 8 up, 7 in 2024-09-20T22:56:42.588 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:42 smithi084 ceph-mon[40024]: 6.d scrub starts 2024-09-20T22:56:42.588 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:42 smithi084 ceph-mon[40024]: 6.d scrub ok 2024-09-20T22:56:42.588 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:42 smithi084 ceph-mon[40024]: 6.14 scrub starts 2024-09-20T22:56:42.588 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:42 smithi084 ceph-mon[40024]: 6.14 scrub ok 2024-09-20T22:56:42.588 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:42 smithi084 ceph-mon[40024]: 5.1f scrub starts 2024-09-20T22:56:42.588 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:42 smithi084 ceph-mon[40024]: 5.1f scrub ok 2024-09-20T22:56:42.593 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-20T22:56:42.593 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:56:42.612 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:56:42.637 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:42.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:42.638 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:56:42.739 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:56:42.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:42 smithi154 ceph-mon[33264]: osdmap e921: 8 total, 8 up, 7 in 2024-09-20T22:56:42.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:42 smithi154 ceph-mon[33264]: 6.d scrub starts 2024-09-20T22:56:42.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:42 smithi154 ceph-mon[33264]: 6.d scrub ok 2024-09-20T22:56:42.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:42 smithi154 ceph-mon[33264]: 6.14 scrub starts 2024-09-20T22:56:42.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:42 smithi154 ceph-mon[33264]: 6.14 scrub ok 2024-09-20T22:56:42.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:42 smithi154 ceph-mon[33264]: 5.1f scrub starts 2024-09-20T22:56:42.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:42 smithi154 ceph-mon[33264]: 5.1f scrub ok 2024-09-20T22:56:42.765 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:42.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:42.766 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:56:42.866 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:56:42.891 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:42.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:42.892 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:56:42.993 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:56:43.016 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:43.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:43.017 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:56:43.117 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:56:43.143 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:43.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:43.144 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:56:43.245 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:56:43.270 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:43.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:43.271 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:56:43.355 INFO:tasks.rados.rados.0.smithi084.stdout:3700: done (0 left) 2024-09-20T22:56:43.355 INFO:tasks.rados.rados.0.smithi084.stdout:3701: read oid 42 snap -1 2024-09-20T22:56:43.355 INFO:tasks.rados.rados.0.smithi084.stdout:3701: expect deleted 2024-09-20T22:56:43.355 INFO:tasks.rados.rados.0.smithi084.stdout:3702: read oid 39 snap -1 2024-09-20T22:56:43.355 INFO:tasks.rados.rados.0.smithi084.stdout:3702: expect (ObjNum 1051 snap 345 seq_num 1051) 2024-09-20T22:56:43.355 INFO:tasks.rados.rados.0.smithi084.stdout:3703: copy_from oid 29 from oid 14 current snap is 430 2024-09-20T22:56:43.355 INFO:tasks.rados.rados.0.smithi084.stdout:3704: read oid 33 snap 421 2024-09-20T22:56:43.355 INFO:tasks.rados.rados.0.smithi084.stdout:3704: expect (ObjNum 1202 snap 411 seq_num 1202) 2024-09-20T22:56:43.356 INFO:tasks.rados.rados.0.smithi084.stdout:3705: write (excl) oid 43 current snap is 430 2024-09-20T22:56:43.356 INFO:tasks.rados.rados.0.smithi084.stdout:3705: seq_num 1259 ranges {760013=736736,2190359=401103} 2024-09-20T22:56:43.365 INFO:tasks.rados.rados.0.smithi084.stdout:3705: writing smithi08477203-43 from 760013 to 1496749 tid 1 2024-09-20T22:56:43.371 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:56:43.373 INFO:tasks.rados.rados.0.smithi084.stdout:3705: writing smithi08477203-43 from 2190359 to 2591462 tid 2 2024-09-20T22:56:43.373 INFO:tasks.rados.rados.0.smithi084.stdout:3706: delete oid 15 current snap is 430 2024-09-20T22:56:43.376 INFO:tasks.rados.rados.0.smithi084.stdout:3704: expect (ObjNum 1202 snap 411 seq_num 1202) 2024-09-20T22:56:43.395 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:43.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:43.396 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:56:43.449 INFO:tasks.rados.rados.0.smithi084.stdout:3703: finishing copy_from racing read to smithi08477203-29 2024-09-20T22:56:43.449 INFO:tasks.rados.rados.0.smithi084.stdout:3702: expect (ObjNum 1051 snap 345 seq_num 1051) 2024-09-20T22:56:43.497 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:56:43.517 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:43.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:43.518 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:56:43.540 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:43 smithi084 ceph-mon[40024]: pgmap v1284: 158 pgs: 2 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 1 active+remapped+backfill_toofull, 151 active+clean; 658 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 25 MiB/s rd, 4.7 MiB/s wr, 30 op/s 2024-09-20T22:56:43.540 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:43 smithi084 ceph-mon[40024]: osdmap e922: 8 total, 8 up, 7 in 2024-09-20T22:56:43.540 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:43 smithi084 ceph-mon[40024]: 4.10 scrub starts 2024-09-20T22:56:43.540 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:43 smithi084 ceph-mon[40024]: 4.10 scrub ok 2024-09-20T22:56:43.540 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:43 smithi084 ceph-mon[40024]: 5.1b scrub starts 2024-09-20T22:56:43.541 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:43 smithi084 ceph-mon[40024]: 5.1b scrub ok 2024-09-20T22:56:43.541 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:43 smithi084 ceph-mon[40024]: 6.f scrub starts 2024-09-20T22:56:43.541 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:43 smithi084 ceph-mon[40024]: 6.f scrub ok 2024-09-20T22:56:43.541 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:43 smithi084 ceph-mon[40024]: osdmap e923: 8 total, 8 up, 7 in 2024-09-20T22:56:43.541 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:43 smithi084 ceph-mon[29774]: pgmap v1284: 158 pgs: 2 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 1 active+remapped+backfill_toofull, 151 active+clean; 658 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 25 MiB/s rd, 4.7 MiB/s wr, 30 op/s 2024-09-20T22:56:43.541 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:43 smithi084 ceph-mon[29774]: osdmap e922: 8 total, 8 up, 7 in 2024-09-20T22:56:43.541 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:43 smithi084 ceph-mon[29774]: 4.10 scrub starts 2024-09-20T22:56:43.541 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:43 smithi084 ceph-mon[29774]: 4.10 scrub ok 2024-09-20T22:56:43.541 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:43 smithi084 ceph-mon[29774]: 5.1b scrub starts 2024-09-20T22:56:43.541 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:43 smithi084 ceph-mon[29774]: 5.1b scrub ok 2024-09-20T22:56:43.541 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:43 smithi084 ceph-mon[29774]: 6.f scrub starts 2024-09-20T22:56:43.541 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:43 smithi084 ceph-mon[29774]: 6.f scrub ok 2024-09-20T22:56:43.541 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:43 smithi084 ceph-mon[29774]: osdmap e923: 8 total, 8 up, 7 in 2024-09-20T22:56:43.566 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.2 dump_historic_ops 2024-09-20T22:56:43.569 INFO:tasks.rados.rados.0.smithi084.stdout:3705: finishing write tid 1 to smithi08477203-43 2024-09-20T22:56:43.569 INFO:tasks.rados.rados.0.smithi084.stdout:3705: finishing write tid 2 to smithi08477203-43 2024-09-20T22:56:43.569 INFO:tasks.rados.rados.0.smithi084.stdout:3705: finishing write tid 3 to smithi08477203-43 2024-09-20T22:56:43.569 INFO:tasks.rados.rados.0.smithi084.stdout:3705: finishing write tid 4 to smithi08477203-43 2024-09-20T22:56:43.570 INFO:tasks.rados.rados.0.smithi084.stdout:3705: oid 43 updating version 0 to 1951 2024-09-20T22:56:43.570 INFO:tasks.rados.rados.0.smithi084.stdout:3705: oid 43 updating version 1951 to 1952 2024-09-20T22:56:43.570 INFO:tasks.rados.rados.0.smithi084.stdout:3705: oid 43 version 1952 is already newer than 1950 2024-09-20T22:56:43.570 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 43 v 1952 (ObjNum 1259 snap 430 seq_num 1259) dirty exists 2024-09-20T22:56:43.570 INFO:tasks.rados.rados.0.smithi084.stdout:3705: left oid 43 (ObjNum 1259 snap 430 seq_num 1259) 2024-09-20T22:56:43.570 INFO:tasks.rados.rados.0.smithi084.stdout:3703: finishing copy_from to smithi08477203-29 2024-09-20T22:56:43.570 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 29 v 1949 (ObjNum 618 snap 184 seq_num 618) dirty exists 2024-09-20T22:56:43.570 INFO:tasks.rados.rados.0.smithi084.stdout:3701: done (5 left) 2024-09-20T22:56:43.570 INFO:tasks.rados.rados.0.smithi084.stdout:3702: done (4 left) 2024-09-20T22:56:43.570 INFO:tasks.rados.rados.0.smithi084.stdout:3703: done (3 left) 2024-09-20T22:56:43.570 INFO:tasks.rados.rados.0.smithi084.stdout:3704: done (2 left) 2024-09-20T22:56:43.570 INFO:tasks.rados.rados.0.smithi084.stdout:3705: done (1 left) 2024-09-20T22:56:43.570 INFO:tasks.rados.rados.0.smithi084.stdout:3706: done (0 left) 2024-09-20T22:56:43.570 INFO:tasks.rados.rados.0.smithi084.stdout:3707: write oid 15 current snap is 430 2024-09-20T22:56:43.570 INFO:tasks.rados.rados.0.smithi084.stdout:3707: seq_num 1261 ranges {793627=487234,1790970=404772,2829335=364292} 2024-09-20T22:56:43.576 INFO:tasks.rados.rados.0.smithi084.stdout:3707: writing smithi08477203-15 from 793627 to 1280861 tid 1 2024-09-20T22:56:43.581 INFO:tasks.rados.rados.0.smithi084.stdout:3707: writing smithi08477203-15 from 1790970 to 2195742 tid 2 2024-09-20T22:56:43.587 INFO:tasks.rados.rados.0.smithi084.stdout:3707: writing smithi08477203-15 from 2829335 to 3193627 tid 3 2024-09-20T22:56:43.587 INFO:tasks.rados.rados.0.smithi084.stdout:3708: read oid 47 snap 397 2024-09-20T22:56:43.587 INFO:tasks.rados.rados.0.smithi084.stdout:3708: expect (ObjNum 1146 snap 392 seq_num 1146) 2024-09-20T22:56:43.587 INFO:tasks.rados.rados.0.smithi084.stdout:3709: delete oid 31 current snap is 430 2024-09-20T22:56:43.587 INFO:tasks.rados.rados.0.smithi084.stdout:3707: finishing write tid 1 to smithi08477203-15 2024-09-20T22:56:43.587 INFO:tasks.rados.rados.0.smithi084.stdout:3707: finishing write tid 2 to smithi08477203-15 2024-09-20T22:56:43.590 INFO:tasks.rados.rados.0.smithi084.stdout:3707: finishing write tid 3 to smithi08477203-15 2024-09-20T22:56:43.590 INFO:tasks.rados.rados.0.smithi084.stdout:3707: finishing write tid 4 to smithi08477203-15 2024-09-20T22:56:43.591 INFO:tasks.rados.rados.0.smithi084.stdout:3707: finishing write tid 5 to smithi08477203-15 2024-09-20T22:56:43.591 INFO:tasks.rados.rados.0.smithi084.stdout:3707: oid 15 updating version 0 to 2617 2024-09-20T22:56:43.591 INFO:tasks.rados.rados.0.smithi084.stdout:3707: oid 15 updating version 2617 to 2620 2024-09-20T22:56:43.591 INFO:tasks.rados.rados.0.smithi084.stdout:3707: oid 15 version 2620 is already newer than 2618 2024-09-20T22:56:43.591 INFO:tasks.rados.rados.0.smithi084.stdout:3707: oid 15 version 2620 is already newer than 2619 2024-09-20T22:56:43.591 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 15 v 2620 (ObjNum 1261 snap 430 seq_num 1261) dirty exists 2024-09-20T22:56:43.591 INFO:tasks.rados.rados.0.smithi084.stdout:3707: left oid 15 (ObjNum 1261 snap 430 seq_num 1261) 2024-09-20T22:56:43.593 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:56:43.600 INFO:tasks.rados.rados.0.smithi084.stdout:3708: expect (ObjNum 1146 snap 392 seq_num 1146) 2024-09-20T22:56:43.619 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:56:43.642 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:43.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:43.643 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:56:43.743 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:56:43.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:43 smithi154 ceph-mon[33264]: pgmap v1284: 158 pgs: 2 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 1 active+remapped+backfill_toofull, 151 active+clean; 658 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 25 MiB/s rd, 4.7 MiB/s wr, 30 op/s 2024-09-20T22:56:43.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:43 smithi154 ceph-mon[33264]: osdmap e922: 8 total, 8 up, 7 in 2024-09-20T22:56:43.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:43 smithi154 ceph-mon[33264]: 4.10 scrub starts 2024-09-20T22:56:43.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:43 smithi154 ceph-mon[33264]: 4.10 scrub ok 2024-09-20T22:56:43.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:43 smithi154 ceph-mon[33264]: 5.1b scrub starts 2024-09-20T22:56:43.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:43 smithi154 ceph-mon[33264]: 5.1b scrub ok 2024-09-20T22:56:43.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:43 smithi154 ceph-mon[33264]: 6.f scrub starts 2024-09-20T22:56:43.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:43 smithi154 ceph-mon[33264]: 6.f scrub ok 2024-09-20T22:56:43.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:43 smithi154 ceph-mon[33264]: osdmap e923: 8 total, 8 up, 7 in 2024-09-20T22:56:43.765 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:43.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:43.766 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:56:43.787 INFO:tasks.rados.rados.0.smithi084.stdout:3707: done (2 left) 2024-09-20T22:56:43.787 INFO:tasks.rados.rados.0.smithi084.stdout:3708: done (1 left) 2024-09-20T22:56:43.787 INFO:tasks.rados.rados.0.smithi084.stdout:3709: done (0 left) 2024-09-20T22:56:43.787 INFO:tasks.rados.rados.0.smithi084.stdout:3710: delete oid 10 current snap is 430 2024-09-20T22:56:43.790 INFO:tasks.rados.rados.0.smithi084.stdout:3710: done (0 left) 2024-09-20T22:56:43.790 INFO:tasks.rados.rados.0.smithi084.stdout:3711: read oid 38 snap -1 2024-09-20T22:56:43.790 INFO:tasks.rados.rados.0.smithi084.stdout:3711: expect (ObjNum 1202 snap 411 seq_num 1202) 2024-09-20T22:56:43.790 INFO:tasks.rados.rados.0.smithi084.stdout:3712: write oid 46 current snap is 430 2024-09-20T22:56:43.790 INFO:tasks.rados.rados.0.smithi084.stdout:3712: seq_num 1264 ranges {444048=429334,1282769=552099,2627957=616091} 2024-09-20T22:56:43.794 INFO:tasks.rados.rados.0.smithi084.stdout:3712: writing smithi08477203-46 from 444048 to 873382 tid 1 2024-09-20T22:56:43.800 INFO:tasks.rados.rados.0.smithi084.stdout:3712: writing smithi08477203-46 from 1282769 to 1834868 tid 2 2024-09-20T22:56:43.808 INFO:tasks.rados.rados.0.smithi084.stdout:3712: writing smithi08477203-46 from 2627957 to 3244048 tid 3 2024-09-20T22:56:43.808 INFO:tasks.rados.rados.0.smithi084.stdout:3713: delete oid 3 current snap is 430 2024-09-20T22:56:43.808 INFO:tasks.rados.rados.0.smithi084.stdout:3712: finishing write tid 1 to smithi08477203-46 2024-09-20T22:56:43.808 INFO:tasks.rados.rados.0.smithi084.stdout:3711: expect (ObjNum 1202 snap 411 seq_num 1202) 2024-09-20T22:56:43.867 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:56:43.885 INFO:tasks.rados.rados.0.smithi084.stdout:3712: finishing write tid 2 to smithi08477203-46 2024-09-20T22:56:43.886 INFO:tasks.rados.rados.0.smithi084.stdout:3712: finishing write tid 3 to smithi08477203-46 2024-09-20T22:56:43.886 INFO:tasks.rados.rados.0.smithi084.stdout:3712: finishing write tid 4 to smithi08477203-46 2024-09-20T22:56:43.894 INFO:tasks.rados.rados.0.smithi084.stdout:3712: finishing write tid 5 to smithi08477203-46 2024-09-20T22:56:43.894 INFO:tasks.rados.rados.0.smithi084.stdout:3712: oid 46 updating version 0 to 2623 2024-09-20T22:56:43.894 INFO:tasks.rados.rados.0.smithi084.stdout:3712: oid 46 version 2623 is already newer than 2621 2024-09-20T22:56:43.894 INFO:tasks.rados.rados.0.smithi084.stdout:3712: oid 46 version 2623 is already newer than 2622 2024-09-20T22:56:43.895 INFO:tasks.rados.rados.0.smithi084.stdout:3712: oid 46 updating version 2623 to 2624 2024-09-20T22:56:43.895 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 46 v 2624 (ObjNum 1264 snap 430 seq_num 1264) dirty exists 2024-09-20T22:56:43.895 INFO:tasks.rados.rados.0.smithi084.stdout:3712: left oid 46 (ObjNum 1264 snap 430 seq_num 1264) 2024-09-20T22:56:43.895 INFO:tasks.rados.rados.0.smithi084.stdout:3711: done (2 left) 2024-09-20T22:56:43.895 INFO:tasks.rados.rados.0.smithi084.stdout:3712: done (1 left) 2024-09-20T22:56:43.895 INFO:tasks.rados.rados.0.smithi084.stdout:3713: done (0 left) 2024-09-20T22:56:43.895 INFO:tasks.rados.rados.0.smithi084.stdout:3714: copy_from oid 29 from oid 14 current snap is 430 2024-09-20T22:56:43.895 INFO:tasks.rados.rados.0.smithi084.stdout:3715: snap_create 2024-09-20T22:56:43.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:43.896 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:43.896 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:56:43.898 INFO:tasks.rados.rados.0.smithi084.stdout:3714: finishing copy_from racing read to smithi08477203-29 2024-09-20T22:56:43.941 INFO:tasks.rados.rados.0.smithi084.stdout:3714: finishing copy_from to smithi08477203-29 2024-09-20T22:56:43.941 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 29 v 1949 (ObjNum 618 snap 184 seq_num 618) dirty exists 2024-09-20T22:56:43.997 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:56:44.019 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:44.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:44.020 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:56:44.121 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:56:44.124 INFO:tasks.thrashosds.thrasher:in_osds: [2, 3, 5, 6, 0, 4, 1] out_osds: [7] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-20T22:56:44.125 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-20T22:56:44.125 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_0 2024-09-20T22:56:44.125 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph pg dump --format=json 2024-09-20T22:56:44.142 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:44.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:44.143 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:56:44.150 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:56:44.243 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:56:44.263 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:44.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:44.264 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:56:44.336 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-20T22:56:44.366 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:56:44.370 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is unset 2024-09-20T22:56:44.380 INFO:tasks.rados.rados.0.smithi084.stdout:3714: done (1 left) 2024-09-20T22:56:44.380 INFO:tasks.rados.rados.0.smithi084.stdout:3715: done (0 left) 2024-09-20T22:56:44.380 INFO:tasks.rados.rados.0.smithi084.stdout:3716: delete oid 31 current snap is 431 2024-09-20T22:56:44.388 INFO:tasks.rados.rados.0.smithi084.stdout:3716: done (0 left) 2024-09-20T22:56:44.388 INFO:tasks.rados.rados.0.smithi084.stdout:3717: snap_remove snap 365 2024-09-20T22:56:44.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:44.390 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:44.390 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:56:44.402 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:44 smithi084 ceph-mon[29774]: purged_snaps scrub starts 2024-09-20T22:56:44.402 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:44 smithi084 ceph-mon[29774]: purged_snaps scrub ok 2024-09-20T22:56:44.402 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:44 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/2247193342' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:56:44.402 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:44 smithi084 ceph-mon[40024]: purged_snaps scrub starts 2024-09-20T22:56:44.402 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:44 smithi084 ceph-mon[40024]: purged_snaps scrub ok 2024-09-20T22:56:44.402 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:44 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/2247193342' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:56:44.491 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:56:44.512 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:44.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:44.513 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:56:44.614 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:56:44.634 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:44.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:44.635 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:56:44.736 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:56:44.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:44 smithi154 ceph-mon[33264]: purged_snaps scrub starts 2024-09-20T22:56:44.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:44 smithi154 ceph-mon[33264]: purged_snaps scrub ok 2024-09-20T22:56:44.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:44 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/2247193342' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:56:44.756 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:44.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:44.757 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:56:44.858 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:56:44.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:44.880 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:44.880 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:56:44.981 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:56:45.002 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:45.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:45.003 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:56:45.103 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:56:45.127 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:45.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:45.128 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:56:45.229 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:56:45.252 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:45.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:45.253 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:56:45.354 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:56:45.375 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:45.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:45.376 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:56:45.383 INFO:tasks.rados.rados.0.smithi084.stdout:3717: done (0 left) 2024-09-20T22:56:45.383 INFO:tasks.rados.rados.0.smithi084.stdout:3718: snap_remove snap 406 2024-09-20T22:56:45.426 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-20T22:56:45.426 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:56:45.438 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:56:45.477 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:56:45.500 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:45.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:45.501 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:56:45.602 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:56:45.614 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:45 smithi084 ceph-mon[40024]: 1.0 scrub starts 2024-09-20T22:56:45.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:45 smithi084 ceph-mon[40024]: 1.0 scrub ok 2024-09-20T22:56:45.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:45 smithi084 ceph-mon[40024]: pgmap v1287: 158 pgs: 2 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 152 active+clean; 650 MiB data, 1.9 GiB used, 624 GiB / 626 GiB avail; 12 MiB/s rd, 1.4 MiB/s wr, 15 op/s 2024-09-20T22:56:45.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:45 smithi084 ceph-mon[40024]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-09-20T22:56:45.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:45 smithi084 ceph-mon[40024]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T22:56:45.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:45 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/2247193342' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T22:56:45.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:45 smithi084 ceph-mon[40024]: osdmap e924: 8 total, 8 up, 7 in 2024-09-20T22:56:45.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:45 smithi084 ceph-mon[29774]: 1.0 scrub starts 2024-09-20T22:56:45.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:45 smithi084 ceph-mon[29774]: 1.0 scrub ok 2024-09-20T22:56:45.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:45 smithi084 ceph-mon[29774]: pgmap v1287: 158 pgs: 2 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 152 active+clean; 650 MiB data, 1.9 GiB used, 624 GiB / 626 GiB avail; 12 MiB/s rd, 1.4 MiB/s wr, 15 op/s 2024-09-20T22:56:45.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:45 smithi084 ceph-mon[29774]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-09-20T22:56:45.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:45 smithi084 ceph-mon[29774]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T22:56:45.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:45 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/2247193342' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T22:56:45.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:45 smithi084 ceph-mon[29774]: osdmap e924: 8 total, 8 up, 7 in 2024-09-20T22:56:45.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:45.627 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:45.628 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:56:45.730 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:56:45.750 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:45.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:45.751 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:56:45.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:45 smithi154 ceph-mon[33264]: 1.0 scrub starts 2024-09-20T22:56:45.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:45 smithi154 ceph-mon[33264]: 1.0 scrub ok 2024-09-20T22:56:45.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:45 smithi154 ceph-mon[33264]: pgmap v1287: 158 pgs: 2 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 152 active+clean; 650 MiB data, 1.9 GiB used, 624 GiB / 626 GiB avail; 12 MiB/s rd, 1.4 MiB/s wr, 15 op/s 2024-09-20T22:56:45.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:45 smithi154 ceph-mon[33264]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-09-20T22:56:45.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:45 smithi154 ceph-mon[33264]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T22:56:45.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:45 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/2247193342' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T22:56:45.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:45 smithi154 ceph-mon[33264]: osdmap e924: 8 total, 8 up, 7 in 2024-09-20T22:56:45.852 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:56:45.877 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:45.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:45.878 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:56:45.979 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:56:46.003 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:46.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:46.004 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:56:46.105 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:56:46.129 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:46.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:46.130 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:56:46.205 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-20T22:56:46.231 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:56:46.255 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:46.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:46.256 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:56:46.357 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:56:46.378 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:46.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:46.379 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:56:46.387 INFO:tasks.rados.rados.0.smithi084.stdout:3718: done (0 left) 2024-09-20T22:56:46.388 INFO:tasks.rados.rados.0.smithi084.stdout:3719: delete oid 6 current snap is 431 2024-09-20T22:56:46.395 INFO:tasks.rados.rados.0.smithi084.stdout:3719: done (0 left) 2024-09-20T22:56:46.395 INFO:tasks.rados.rados.0.smithi084.stdout:3720: read oid 29 snap -1 2024-09-20T22:56:46.395 INFO:tasks.rados.rados.0.smithi084.stdout:3720: expect (ObjNum 618 snap 184 seq_num 618) 2024-09-20T22:56:46.395 INFO:tasks.rados.rados.0.smithi084.stdout:3721: copy_from oid 17 from oid 10 current snap is 431 2024-09-20T22:56:46.395 INFO:tasks.rados.rados.0.smithi084.stdout:3722: copy_from oid 36 from oid 27 current snap is 431 2024-09-20T22:56:46.395 INFO:tasks.rados.rados.0.smithi084.stdout:3723: write oid 18 current snap is 431 2024-09-20T22:56:46.395 INFO:tasks.rados.rados.0.smithi084.stdout:3723: seq_num 1268 ranges {511276=730330,1800344=559437,3021296=289980} 2024-09-20T22:56:46.404 INFO:tasks.rados.rados.0.smithi084.stdout:3723: writing smithi08477203-18 from 511276 to 1241606 tid 1 2024-09-20T22:56:46.411 INFO:tasks.rados.rados.0.smithi084.stdout:3723: writing smithi08477203-18 from 1800344 to 2359781 tid 2 2024-09-20T22:56:46.417 INFO:tasks.rados.rados.0.smithi084.stdout:3723: writing smithi08477203-18 from 3021296 to 3311276 tid 3 2024-09-20T22:56:46.417 INFO:tasks.rados.rados.0.smithi084.stdout:3724: copy_from oid 48 from oid 31 current snap is 431 2024-09-20T22:56:46.417 INFO:tasks.rados.rados.0.smithi084.stdout:3721: finishing copy_from to smithi08477203-17 2024-09-20T22:56:46.417 INFO:tasks.rados.rados.0.smithi084.stdout:3721: got expected ENOENT (src dne) 2024-09-20T22:56:46.417 INFO:tasks.rados.rados.0.smithi084.stdout:3721: finishing copy_from racing read to smithi08477203-17 2024-09-20T22:56:46.417 INFO:tasks.rados.rados.0.smithi084.stdout:3720: expect (ObjNum 618 snap 184 seq_num 618) 2024-09-20T22:56:46.472 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:46 smithi084 ceph-mon[29774]: 6.9 scrub starts 2024-09-20T22:56:46.472 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:46 smithi084 ceph-mon[29774]: 6.9 scrub ok 2024-09-20T22:56:46.472 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:46 smithi084 ceph-mon[29774]: osdmap e925: 8 total, 8 up, 7 in 2024-09-20T22:56:46.472 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:46 smithi084 ceph-mon[40024]: 6.9 scrub starts 2024-09-20T22:56:46.472 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:46 smithi084 ceph-mon[40024]: 6.9 scrub ok 2024-09-20T22:56:46.472 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:46 smithi084 ceph-mon[40024]: osdmap e925: 8 total, 8 up, 7 in 2024-09-20T22:56:46.480 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:56:46.504 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:46.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:46.505 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:56:46.562 INFO:tasks.rados.rados.0.smithi084.stdout:3722: finishing copy_from racing read to smithi08477203-36 2024-09-20T22:56:46.562 INFO:tasks.rados.rados.0.smithi084.stdout:3720: done (4 left) 2024-09-20T22:56:46.562 INFO:tasks.rados.rados.0.smithi084.stdout:3721: done (3 left) 2024-09-20T22:56:46.562 INFO:tasks.rados.rados.0.smithi084.stdout:3725: copy_from oid 4 from oid 47 current snap is 431 2024-09-20T22:56:46.562 INFO:tasks.rados.rados.0.smithi084.stdout:3723: finishing write tid 1 to smithi08477203-18 2024-09-20T22:56:46.562 INFO:tasks.rados.rados.0.smithi084.stdout:3723: finishing write tid 2 to smithi08477203-18 2024-09-20T22:56:46.562 INFO:tasks.rados.rados.0.smithi084.stdout:3724: finishing copy_from to smithi08477203-48 2024-09-20T22:56:46.562 INFO:tasks.rados.rados.0.smithi084.stdout:3724: got expected ENOENT (src dne) 2024-09-20T22:56:46.563 INFO:tasks.rados.rados.0.smithi084.stdout:3724: finishing copy_from racing read to smithi08477203-48 2024-09-20T22:56:46.563 INFO:tasks.rados.rados.0.smithi084.stdout:3723: finishing write tid 3 to smithi08477203-18 2024-09-20T22:56:46.563 INFO:tasks.rados.rados.0.smithi084.stdout:3723: finishing write tid 4 to smithi08477203-18 2024-09-20T22:56:46.563 INFO:tasks.rados.rados.0.smithi084.stdout:3722: finishing copy_from to smithi08477203-36 2024-09-20T22:56:46.563 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 36 v 2499 (ObjNum 1246 snap 423 seq_num 1246) dirty exists 2024-09-20T22:56:46.563 INFO:tasks.rados.rados.0.smithi084.stdout:3723: finishing write tid 5 to smithi08477203-18 2024-09-20T22:56:46.563 INFO:tasks.rados.rados.0.smithi084.stdout:3723: oid 18 updating version 0 to 2681 2024-09-20T22:56:46.563 INFO:tasks.rados.rados.0.smithi084.stdout:3723: oid 18 version 2681 is already newer than 2680 2024-09-20T22:56:46.563 INFO:tasks.rados.rados.0.smithi084.stdout:3723: oid 18 updating version 2681 to 2682 2024-09-20T22:56:46.563 INFO:tasks.rados.rados.0.smithi084.stdout:3723: oid 18 updating version 2682 to 2683 2024-09-20T22:56:46.563 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 18 v 2683 (ObjNum 1268 snap 431 seq_num 1268) dirty exists 2024-09-20T22:56:46.563 INFO:tasks.rados.rados.0.smithi084.stdout:3723: left oid 18 (ObjNum 1268 snap 431 seq_num 1268) 2024-09-20T22:56:46.563 INFO:tasks.rados.rados.0.smithi084.stdout:3722: done (3 left) 2024-09-20T22:56:46.563 INFO:tasks.rados.rados.0.smithi084.stdout:3723: done (2 left) 2024-09-20T22:56:46.564 INFO:tasks.rados.rados.0.smithi084.stdout:3724: done (1 left) 2024-09-20T22:56:46.564 INFO:tasks.rados.rados.0.smithi084.stdout:3726: snap_remove snap 371 2024-09-20T22:56:46.606 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:56:46.628 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:46.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:46.629 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:56:46.721 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:56:46.722 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:56:46.728 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:56:46.730 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:56:46.739 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:56:46.740 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:56:46.752 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:46.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:46.753 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:56:46.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:46 smithi154 ceph-mon[33264]: 6.9 scrub starts 2024-09-20T22:56:46.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:46 smithi154 ceph-mon[33264]: 6.9 scrub ok 2024-09-20T22:56:46.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:46 smithi154 ceph-mon[33264]: osdmap e925: 8 total, 8 up, 7 in 2024-09-20T22:56:46.761 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:56:46.764 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:56:46.764 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:56:46.790 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-20T22:56:46.790 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T22:56:46.807 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:56:46.807 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:56:46.823 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:56:46.823 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:56:46.840 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:56:46.840 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:56:46.853 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:56:46.857 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:56:46.857 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:56:46.875 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:46.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:46.876 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:56:46.977 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:56:46.998 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:46.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:46.999 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:56:47.100 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:56:47.121 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:47.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:47.122 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:56:47.138 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd set noscrub 2024-09-20T22:56:47.223 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:56:47.244 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:47.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:47.245 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:56:47.346 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:56:47.367 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:47.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:47.369 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:56:47.389 INFO:tasks.rados.rados.0.smithi084.stdout:3726: done (1 left) 2024-09-20T22:56:47.389 INFO:tasks.rados.rados.0.smithi084.stdout:3727: rollback oid 6 current snap is 431 2024-09-20T22:56:47.389 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 6 to 405 2024-09-20T22:56:47.389 INFO:tasks.rados.rados.0.smithi084.stdout:3728: write oid 24 current snap is 431 2024-09-20T22:56:47.390 INFO:tasks.rados.rados.0.smithi084.stdout:3728: seq_num 1269 ranges {528083=405579,1342473=419448,2248719=733949} 2024-09-20T22:56:47.394 INFO:tasks.rados.rados.0.smithi084.stdout:3728: writing smithi08477203-24 from 528083 to 933662 tid 1 2024-09-20T22:56:47.400 INFO:tasks.rados.rados.0.smithi084.stdout:3728: writing smithi08477203-24 from 1342473 to 1761921 tid 2 2024-09-20T22:56:47.407 INFO:tasks.rados.rados.0.smithi084.stdout:3728: writing smithi08477203-24 from 2248719 to 2982668 tid 3 2024-09-20T22:56:47.407 INFO:tasks.rados.rados.0.smithi084.stdout:3729: write oid 32 current snap is 431 2024-09-20T22:56:47.407 INFO:tasks.rados.rados.0.smithi084.stdout:3725: finishing copy_from racing read to smithi08477203-4 2024-09-20T22:56:47.407 INFO:tasks.rados.rados.0.smithi084.stdout:3729: seq_num 1270 ranges {544890=597181,1917308=563106,3076142=268748} 2024-09-20T22:56:47.413 INFO:tasks.rados.rados.0.smithi084.stdout:3729: writing smithi08477203-32 from 544890 to 1142071 tid 1 2024-09-20T22:56:47.421 INFO:tasks.rados.rados.0.smithi084.stdout:3729: writing smithi08477203-32 from 1917308 to 2480414 tid 2 2024-09-20T22:56:47.427 INFO:tasks.rados.rados.0.smithi084.stdout:3729: writing smithi08477203-32 from 3076142 to 3344890 tid 3 2024-09-20T22:56:47.427 INFO:tasks.rados.rados.0.smithi084.stdout:3730: rollback oid 23 current snap is 431 2024-09-20T22:56:47.427 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 23 to 421 2024-09-20T22:56:47.427 INFO:tasks.rados.rados.0.smithi084.stdout:3725: finishing copy_from to smithi08477203-4 2024-09-20T22:56:47.427 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 4 v 2589 (ObjNum 1066 snap 353 seq_num 1066) dirty exists 2024-09-20T22:56:47.427 INFO:tasks.rados.rados.0.smithi084.stdout:3728: finishing write tid 1 to smithi08477203-24 2024-09-20T22:56:47.427 INFO:tasks.rados.rados.0.smithi084.stdout:3727: finishing rollback tid 1 to smithi08477203-6 2024-09-20T22:56:47.427 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 6 v 2652 (ObjNum 129 snap 0 seq_num 2091807776) dirty dne 2024-09-20T22:56:47.427 INFO:tasks.rados.rados.0.smithi084.stdout:3728: finishing write tid 2 to smithi08477203-24 2024-09-20T22:56:47.427 INFO:tasks.rados.rados.0.smithi084.stdout:3728: finishing write tid 3 to smithi08477203-24 2024-09-20T22:56:47.428 INFO:tasks.rados.rados.0.smithi084.stdout:3728: finishing write tid 4 to smithi08477203-24 2024-09-20T22:56:47.428 INFO:tasks.rados.rados.0.smithi084.stdout:3728: finishing write tid 5 to smithi08477203-24 2024-09-20T22:56:47.428 INFO:tasks.rados.rados.0.smithi084.stdout:3728: oid 24 updating version 0 to 1901 2024-09-20T22:56:47.428 INFO:tasks.rados.rados.0.smithi084.stdout:3728: oid 24 version 1901 is already newer than 1899 2024-09-20T22:56:47.428 INFO:tasks.rados.rados.0.smithi084.stdout:3728: oid 24 version 1901 is already newer than 1900 2024-09-20T22:56:47.428 INFO:tasks.rados.rados.0.smithi084.stdout:3728: oid 24 updating version 1901 to 1902 2024-09-20T22:56:47.428 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 24 v 1902 (ObjNum 1269 snap 431 seq_num 1269) dirty exists 2024-09-20T22:56:47.428 INFO:tasks.rados.rados.0.smithi084.stdout:3728: left oid 24 (ObjNum 1269 snap 431 seq_num 1269) 2024-09-20T22:56:47.428 INFO:tasks.rados.rados.0.smithi084.stdout:3729: finishing write tid 1 to smithi08477203-32 2024-09-20T22:56:47.428 INFO:tasks.rados.rados.0.smithi084.stdout:3729: finishing write tid 2 to smithi08477203-32 2024-09-20T22:56:47.428 INFO:tasks.rados.rados.0.smithi084.stdout:3725: done (4 left) 2024-09-20T22:56:47.429 INFO:tasks.rados.rados.0.smithi084.stdout:3727: done (3 left) 2024-09-20T22:56:47.429 INFO:tasks.rados.rados.0.smithi084.stdout:3728: done (2 left) 2024-09-20T22:56:47.429 INFO:tasks.rados.rados.0.smithi084.stdout:3731: rollback oid 19 current snap is 431 2024-09-20T22:56:47.429 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 19 to 370 2024-09-20T22:56:47.429 INFO:tasks.rados.rados.0.smithi084.stdout:3732: delete oid 13 current snap is 431 2024-09-20T22:56:47.430 INFO:tasks.rados.rados.0.smithi084.stdout:3729: finishing write tid 3 to smithi08477203-32 2024-09-20T22:56:47.430 INFO:tasks.rados.rados.0.smithi084.stdout:3729: finishing write tid 4 to smithi08477203-32 2024-09-20T22:56:47.430 INFO:tasks.rados.rados.0.smithi084.stdout:3729: finishing write tid 5 to smithi08477203-32 2024-09-20T22:56:47.430 INFO:tasks.rados.rados.0.smithi084.stdout:3729: oid 32 updating version 0 to 1855 2024-09-20T22:56:47.430 INFO:tasks.rados.rados.0.smithi084.stdout:3729: oid 32 updating version 1855 to 1857 2024-09-20T22:56:47.430 INFO:tasks.rados.rados.0.smithi084.stdout:3729: oid 32 version 1857 is already newer than 1854 2024-09-20T22:56:47.431 INFO:tasks.rados.rados.0.smithi084.stdout:3729: oid 32 version 1857 is already newer than 1856 2024-09-20T22:56:47.431 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 32 v 1857 (ObjNum 1270 snap 431 seq_num 1270) dirty exists 2024-09-20T22:56:47.431 INFO:tasks.rados.rados.0.smithi084.stdout:3729: left oid 32 (ObjNum 1270 snap 431 seq_num 1270) 2024-09-20T22:56:47.431 INFO:tasks.rados.rados.0.smithi084.stdout:3730: finishing rollback tid 0 to smithi08477203-23 2024-09-20T22:56:47.431 INFO:tasks.rados.rados.0.smithi084.stdout:3730: finishing rollback tid 1 to smithi08477203-23 2024-09-20T22:56:47.431 INFO:tasks.rados.rados.0.smithi084.stdout:3730: finishing rollback tid 2 to smithi08477203-23 2024-09-20T22:56:47.431 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 23 v 2846 (ObjNum 1066 snap 353 seq_num 1066) dirty exists 2024-09-20T22:56:47.433 INFO:tasks.rados.rados.0.smithi084.stdout:3731: finishing rollback tid 0 to smithi08477203-19 2024-09-20T22:56:47.435 INFO:tasks.rados.rados.0.smithi084.stdout:3731: finishing rollback tid 1 to smithi08477203-19 2024-09-20T22:56:47.437 INFO:tasks.rados.rados.0.smithi084.stdout:3731: finishing rollback tid 2 to smithi08477203-19 2024-09-20T22:56:47.437 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 19 v 3059 (ObjNum 1065 snap 351 seq_num 1065) dirty exists 2024-09-20T22:56:47.437 INFO:tasks.rados.rados.0.smithi084.stdout:3729: done (3 left) 2024-09-20T22:56:47.437 INFO:tasks.rados.rados.0.smithi084.stdout:3730: done (2 left) 2024-09-20T22:56:47.437 INFO:tasks.rados.rados.0.smithi084.stdout:3731: done (1 left) 2024-09-20T22:56:47.437 INFO:tasks.rados.rados.0.smithi084.stdout:3732: done (0 left) 2024-09-20T22:56:47.438 INFO:tasks.rados.rados.0.smithi084.stdout:3733: write oid 23 current snap is 431 2024-09-20T22:56:47.438 INFO:tasks.rados.rados.0.smithi084.stdout:3733: seq_num 1272 ranges {578504=747679,2034272=566775,3130988=247516} 2024-09-20T22:56:47.444 INFO:tasks.rados.rados.0.smithi084.stdout:3733: writing smithi08477203-23 from 578504 to 1326183 tid 1 2024-09-20T22:56:47.451 INFO:tasks.rados.rados.0.smithi084.stdout:3733: writing smithi08477203-23 from 2034272 to 2601047 tid 2 2024-09-20T22:56:47.456 INFO:tasks.rados.rados.0.smithi084.stdout:3733: writing smithi08477203-23 from 3130988 to 3378504 tid 3 2024-09-20T22:56:47.456 INFO:tasks.rados.rados.0.smithi084.stdout:3734: rollback oid 38 current snap is 431 2024-09-20T22:56:47.456 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 38 to 421 2024-09-20T22:56:47.456 INFO:tasks.rados.rados.0.smithi084.stdout:3733: finishing write tid 1 to smithi08477203-23 2024-09-20T22:56:47.456 INFO:tasks.rados.rados.0.smithi084.stdout:3733: finishing write tid 2 to smithi08477203-23 2024-09-20T22:56:47.456 INFO:tasks.rados.rados.0.smithi084.stdout:3735: read oid 44 snap -1 2024-09-20T22:56:47.456 INFO:tasks.rados.rados.0.smithi084.stdout:3735: expect (ObjNum 1202 snap 411 seq_num 1202) 2024-09-20T22:56:47.457 INFO:tasks.rados.rados.0.smithi084.stdout:3736: read oid 24 snap -1 2024-09-20T22:56:47.457 INFO:tasks.rados.rados.0.smithi084.stdout:3736: expect (ObjNum 1269 snap 431 seq_num 1269) 2024-09-20T22:56:47.457 INFO:tasks.rados.rados.0.smithi084.stdout:3737: read oid 8 snap -1 2024-09-20T22:56:47.457 INFO:tasks.rados.rados.0.smithi084.stdout:3737: expect deleted 2024-09-20T22:56:47.457 INFO:tasks.rados.rados.0.smithi084.stdout:3738: write oid 7 current snap is 431 2024-09-20T22:56:47.457 INFO:tasks.rados.rados.0.smithi084.stdout:3738: seq_num 1273 ranges {595311=422928,1576401=710433,3042058=353253} 2024-09-20T22:56:47.462 INFO:tasks.rados.rados.0.smithi084.stdout:3738: writing smithi08477203-7 from 595311 to 1018239 tid 1 2024-09-20T22:56:47.469 INFO:tasks.rados.rados.0.smithi084.stdout:3738: writing smithi08477203-7 from 1576401 to 2286834 tid 2 2024-09-20T22:56:47.469 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:56:47.476 INFO:tasks.rados.rados.0.smithi084.stdout:3738: writing smithi08477203-7 from 3042058 to 3395311 tid 3 2024-09-20T22:56:47.476 INFO:tasks.rados.rados.0.smithi084.stdout:3739: read oid 45 snap -1 2024-09-20T22:56:47.476 INFO:tasks.rados.rados.0.smithi084.stdout:3739: expect (ObjNum 1025 snap 333 seq_num 1025) 2024-09-20T22:56:47.476 INFO:tasks.rados.rados.0.smithi084.stdout:3740: copy_from oid 16 from oid 17 current snap is 431 2024-09-20T22:56:47.476 INFO:tasks.rados.rados.0.smithi084.stdout:3733: finishing write tid 3 to smithi08477203-23 2024-09-20T22:56:47.476 INFO:tasks.rados.rados.0.smithi084.stdout:3733: finishing write tid 4 to smithi08477203-23 2024-09-20T22:56:47.476 INFO:tasks.rados.rados.0.smithi084.stdout:3733: finishing write tid 5 to smithi08477203-23 2024-09-20T22:56:47.476 INFO:tasks.rados.rados.0.smithi084.stdout:3733: oid 23 updating version 0 to 2850 2024-09-20T22:56:47.476 INFO:tasks.rados.rados.0.smithi084.stdout:3733: oid 23 version 2850 is already newer than 2849 2024-09-20T22:56:47.476 INFO:tasks.rados.rados.0.smithi084.stdout:3733: oid 23 version 2850 is already newer than 2848 2024-09-20T22:56:47.476 INFO:tasks.rados.rados.0.smithi084.stdout:3733: oid 23 version 2850 is already newer than 2847 2024-09-20T22:56:47.476 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 23 v 2850 (ObjNum 1272 snap 431 seq_num 1272) dirty exists 2024-09-20T22:56:47.477 INFO:tasks.rados.rados.0.smithi084.stdout:3733: left oid 23 (ObjNum 1272 snap 431 seq_num 1272) 2024-09-20T22:56:47.477 INFO:tasks.rados.rados.0.smithi084.stdout:3735: expect (ObjNum 1202 snap 411 seq_num 1202) 2024-09-20T22:56:47.490 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:47.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:47.491 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:56:47.547 INFO:tasks.rados.rados.0.smithi084.stdout:3734: finishing rollback tid 0 to smithi08477203-38 2024-09-20T22:56:47.547 INFO:tasks.rados.rados.0.smithi084.stdout:3734: finishing rollback tid 1 to smithi08477203-38 2024-09-20T22:56:47.548 INFO:tasks.rados.rados.0.smithi084.stdout:3734: finishing rollback tid 2 to smithi08477203-38 2024-09-20T22:56:47.548 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 38 v 2845 (ObjNum 1143 snap 389 seq_num 1143) dirty exists 2024-09-20T22:56:47.548 INFO:tasks.rados.rados.0.smithi084.stdout:3738: finishing write tid 1 to smithi08477203-7 2024-09-20T22:56:47.548 INFO:tasks.rados.rados.0.smithi084.stdout:3736: expect (ObjNum 1269 snap 431 seq_num 1269) 2024-09-20T22:56:47.592 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:56:47.613 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:47.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:47.614 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:56:47.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:47 smithi084 ceph-mon[40024]: pgmap v1290: 158 pgs: 2 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 152 active+clean; 650 MiB data, 1.9 GiB used, 624 GiB / 626 GiB avail; 12 MiB/s rd, 1.4 MiB/s wr, 15 op/s 2024-09-20T22:56:47.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:47 smithi084 ceph-mon[40024]: osdmap e926: 8 total, 8 up, 7 in 2024-09-20T22:56:47.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:47 smithi084 ceph-mon[40024]: 6.c scrub starts 2024-09-20T22:56:47.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:47 smithi084 ceph-mon[40024]: 6.c scrub ok 2024-09-20T22:56:47.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:47 smithi084 ceph-mon[40024]: 5.e scrub starts 2024-09-20T22:56:47.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:47 smithi084 ceph-mon[40024]: 5.e scrub ok 2024-09-20T22:56:47.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:47 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/1914017370' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:56:47.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:47 smithi084 ceph-mon[40024]: 6.15 scrub starts 2024-09-20T22:56:47.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:47 smithi084 ceph-mon[40024]: 6.15 scrub ok 2024-09-20T22:56:47.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:47 smithi084 ceph-mon[29774]: pgmap v1290: 158 pgs: 2 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 152 active+clean; 650 MiB data, 1.9 GiB used, 624 GiB / 626 GiB avail; 12 MiB/s rd, 1.4 MiB/s wr, 15 op/s 2024-09-20T22:56:47.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:47 smithi084 ceph-mon[29774]: osdmap e926: 8 total, 8 up, 7 in 2024-09-20T22:56:47.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:47 smithi084 ceph-mon[29774]: 6.c scrub starts 2024-09-20T22:56:47.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:47 smithi084 ceph-mon[29774]: 6.c scrub ok 2024-09-20T22:56:47.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:47 smithi084 ceph-mon[29774]: 5.e scrub starts 2024-09-20T22:56:47.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:47 smithi084 ceph-mon[29774]: 5.e scrub ok 2024-09-20T22:56:47.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:47 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/1914017370' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:56:47.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:47 smithi084 ceph-mon[29774]: 6.15 scrub starts 2024-09-20T22:56:47.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:47 smithi084 ceph-mon[29774]: 6.15 scrub ok 2024-09-20T22:56:47.656 INFO:tasks.rados.rados.0.smithi084.stdout:3738: finishing write tid 2 to smithi08477203-7 2024-09-20T22:56:47.656 INFO:tasks.rados.rados.0.smithi084.stdout:3740: finishing copy_from to smithi08477203-16 2024-09-20T22:56:47.657 INFO:tasks.rados.rados.0.smithi084.stdout:3740: got expected ENOENT (src dne) 2024-09-20T22:56:47.657 INFO:tasks.rados.rados.0.smithi084.stdout:3740: finishing copy_from racing read to smithi08477203-16 2024-09-20T22:56:47.657 INFO:tasks.rados.rados.0.smithi084.stdout:3738: finishing write tid 3 to smithi08477203-7 2024-09-20T22:56:47.657 INFO:tasks.rados.rados.0.smithi084.stdout:3738: finishing write tid 4 to smithi08477203-7 2024-09-20T22:56:47.657 INFO:tasks.rados.rados.0.smithi084.stdout:3738: finishing write tid 5 to smithi08477203-7 2024-09-20T22:56:47.657 INFO:tasks.rados.rados.0.smithi084.stdout:3738: oid 7 updating version 0 to 1957 2024-09-20T22:56:47.657 INFO:tasks.rados.rados.0.smithi084.stdout:3738: oid 7 updating version 1957 to 1959 2024-09-20T22:56:47.657 INFO:tasks.rados.rados.0.smithi084.stdout:3738: oid 7 version 1959 is already newer than 1958 2024-09-20T22:56:47.657 INFO:tasks.rados.rados.0.smithi084.stdout:3738: oid 7 version 1959 is already newer than 1956 2024-09-20T22:56:47.657 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 7 v 1959 (ObjNum 1273 snap 431 seq_num 1273) dirty exists 2024-09-20T22:56:47.657 INFO:tasks.rados.rados.0.smithi084.stdout:3738: left oid 7 (ObjNum 1273 snap 431 seq_num 1273) 2024-09-20T22:56:47.657 INFO:tasks.rados.rados.0.smithi084.stdout:3739: expect (ObjNum 1025 snap 333 seq_num 1025) 2024-09-20T22:56:47.715 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:56:47.736 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:47.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:47.737 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:56:47.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:47 smithi154 ceph-mon[33264]: pgmap v1290: 158 pgs: 2 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 152 active+clean; 650 MiB data, 1.9 GiB used, 624 GiB / 626 GiB avail; 12 MiB/s rd, 1.4 MiB/s wr, 15 op/s 2024-09-20T22:56:47.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:47 smithi154 ceph-mon[33264]: osdmap e926: 8 total, 8 up, 7 in 2024-09-20T22:56:47.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:47 smithi154 ceph-mon[33264]: 6.c scrub starts 2024-09-20T22:56:47.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:47 smithi154 ceph-mon[33264]: 6.c scrub ok 2024-09-20T22:56:47.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:47 smithi154 ceph-mon[33264]: 5.e scrub starts 2024-09-20T22:56:47.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:47 smithi154 ceph-mon[33264]: 5.e scrub ok 2024-09-20T22:56:47.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:47 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/1914017370' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:56:47.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:47 smithi154 ceph-mon[33264]: 6.15 scrub starts 2024-09-20T22:56:47.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:47 smithi154 ceph-mon[33264]: 6.15 scrub ok 2024-09-20T22:56:47.838 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:56:47.859 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:47.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:47.860 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:56:47.919 INFO:tasks.rados.rados.0.smithi084.stdout:3733: done (7 left) 2024-09-20T22:56:47.919 INFO:tasks.rados.rados.0.smithi084.stdout:3734: done (6 left) 2024-09-20T22:56:47.919 INFO:tasks.rados.rados.0.smithi084.stdout:3735: done (5 left) 2024-09-20T22:56:47.919 INFO:tasks.rados.rados.0.smithi084.stdout:3736: done (4 left) 2024-09-20T22:56:47.919 INFO:tasks.rados.rados.0.smithi084.stdout:3737: done (3 left) 2024-09-20T22:56:47.919 INFO:tasks.rados.rados.0.smithi084.stdout:3738: done (2 left) 2024-09-20T22:56:47.919 INFO:tasks.rados.rados.0.smithi084.stdout:3739: done (1 left) 2024-09-20T22:56:47.919 INFO:tasks.rados.rados.0.smithi084.stdout:3740: done (0 left) 2024-09-20T22:56:47.919 INFO:tasks.rados.rados.0.smithi084.stdout:3741: copy_from oid 4 from oid 27 current snap is 431 2024-09-20T22:56:47.919 INFO:tasks.rados.rados.0.smithi084.stdout:3742: snap_create 2024-09-20T22:56:47.928 INFO:tasks.rados.rados.0.smithi084.stdout:3741: finishing copy_from racing read to smithi08477203-4 2024-09-20T22:56:47.937 INFO:tasks.rados.rados.0.smithi084.stdout:3741: finishing copy_from to smithi08477203-4 2024-09-20T22:56:47.937 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 4 v 2499 (ObjNum 1246 snap 423 seq_num 1246) dirty exists 2024-09-20T22:56:47.960 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:56:47.982 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:47.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:47.983 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:56:48.083 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:56:48.104 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:48.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:48.105 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:56:48.144 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-20T22:56:48.146 INFO:teuthology.orchestra.run.smithi084.stderr:dumped all 2024-09-20T22:56:48.206 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:56:48.227 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:48.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:48.228 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:56:48.329 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:56:48.350 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:48.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:48.351 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:56:48.353 INFO:tasks.rados.rados.0.smithi084.stdout:3741: done (1 left) 2024-09-20T22:56:48.353 INFO:tasks.rados.rados.0.smithi084.stdout:3742: done (0 left) 2024-09-20T22:56:48.353 INFO:tasks.rados.rados.0.smithi084.stdout:3743: write (excl) oid 38 current snap is 432 2024-09-20T22:56:48.354 INFO:tasks.rados.rados.0.smithi084.stdout:3743: seq_num 1274 ranges {612118=498177,1634883=570444,2669481=706662} 2024-09-20T22:56:48.359 INFO:tasks.rados.rados.0.smithi084.stdout:3743: writing smithi08477203-38 from 612118 to 1110295 tid 1 2024-09-20T22:56:48.366 INFO:tasks.rados.rados.0.smithi084.stdout:3743: writing smithi08477203-38 from 1634883 to 2205327 tid 2 2024-09-20T22:56:48.375 INFO:tasks.rados.rados.0.smithi084.stdout:3743: writing smithi08477203-38 from 2669481 to 3376143 tid 3 2024-09-20T22:56:48.398 INFO:tasks.rados.rados.0.smithi084.stdout:3744: snap_create 2024-09-20T22:56:48.398 INFO:tasks.rados.rados.0.smithi084.stdout:3743: finishing write tid 1 to smithi08477203-38 2024-09-20T22:56:48.398 INFO:tasks.rados.rados.0.smithi084.stdout:3743: finishing write tid 2 to smithi08477203-38 2024-09-20T22:56:48.399 INFO:tasks.rados.rados.0.smithi084.stdout:3743: finishing write tid 3 to smithi08477203-38 2024-09-20T22:56:48.399 INFO:tasks.rados.rados.0.smithi084.stdout:3743: finishing write tid 4 to smithi08477203-38 2024-09-20T22:56:48.399 INFO:tasks.rados.rados.0.smithi084.stdout:3743: finishing write tid 5 to smithi08477203-38 2024-09-20T22:56:48.399 INFO:tasks.rados.rados.0.smithi084.stdout:3743: oid 38 updating version 0 to 2848 2024-09-20T22:56:48.399 INFO:tasks.rados.rados.0.smithi084.stdout:3743: oid 38 version 2848 is already newer than 2847 2024-09-20T22:56:48.400 INFO:tasks.rados.rados.0.smithi084.stdout:3743: oid 38 version 2848 is already newer than 2846 2024-09-20T22:56:48.400 INFO:tasks.rados.rados.0.smithi084.stdout:3743: oid 38 updating version 2848 to 2849 2024-09-20T22:56:48.400 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 38 v 2849 (ObjNum 1274 snap 432 seq_num 1274) dirty exists 2024-09-20T22:56:48.400 INFO:tasks.rados.rados.0.smithi084.stdout:3743: left oid 38 (ObjNum 1274 snap 432 seq_num 1274) 2024-09-20T22:56:48.403 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:56:48 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: ::ffff:172.21.15.154 - - [20/Sep/2024:22:56:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-20T22:56:48.451 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:56:48.473 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:48.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:48.474 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:56:48.575 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:56:48.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:48.597 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:48.597 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:56:48.671 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:48 smithi084 ceph-mon[40024]: osdmap e927: 8 total, 8 up, 7 in 2024-09-20T22:56:48.671 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:48 smithi084 ceph-mon[40024]: 5.1 deep-scrub starts 2024-09-20T22:56:48.671 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:48 smithi084 ceph-mon[40024]: 5.1 deep-scrub ok 2024-09-20T22:56:48.671 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:48 smithi084 ceph-mon[40024]: 5.9 scrub starts 2024-09-20T22:56:48.671 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:48 smithi084 ceph-mon[40024]: 5.9 scrub ok 2024-09-20T22:56:48.671 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:48 smithi084 ceph-mon[40024]: osdmap e928: 8 total, 8 up, 7 in 2024-09-20T22:56:48.671 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:48 smithi084 ceph-mon[29774]: osdmap e927: 8 total, 8 up, 7 in 2024-09-20T22:56:48.672 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:48 smithi084 ceph-mon[29774]: 5.1 deep-scrub starts 2024-09-20T22:56:48.672 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:48 smithi084 ceph-mon[29774]: 5.1 deep-scrub ok 2024-09-20T22:56:48.672 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:48 smithi084 ceph-mon[29774]: 5.9 scrub starts 2024-09-20T22:56:48.672 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:48 smithi084 ceph-mon[29774]: 5.9 scrub ok 2024-09-20T22:56:48.672 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:48 smithi084 ceph-mon[29774]: osdmap e928: 8 total, 8 up, 7 in 2024-09-20T22:56:48.698 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:56:48.703 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-20T22:56:48.703 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2024-09-20T22:56:48.703 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2024-09-20T22:56:48.703 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-20T22:56:48.719 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:48.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:48.720 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:56:48.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:48 smithi154 ceph-mon[33264]: osdmap e927: 8 total, 8 up, 7 in 2024-09-20T22:56:48.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:48 smithi154 ceph-mon[33264]: 5.1 deep-scrub starts 2024-09-20T22:56:48.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:48 smithi154 ceph-mon[33264]: 5.1 deep-scrub ok 2024-09-20T22:56:48.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:48 smithi154 ceph-mon[33264]: 5.9 scrub starts 2024-09-20T22:56:48.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:48 smithi154 ceph-mon[33264]: 5.9 scrub ok 2024-09-20T22:56:48.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:48 smithi154 ceph-mon[33264]: osdmap e928: 8 total, 8 up, 7 in 2024-09-20T22:56:48.821 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:56:48.842 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:48.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:48.843 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:56:48.944 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:56:48.965 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:48.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:48.966 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:56:49.066 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:56:49.088 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:49.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:49.088 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:56:49.189 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:56:49.210 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:49.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:49.211 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:56:49.312 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:56:49.333 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:49.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:49.334 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:56:49.338 INFO:teuthology.orchestra.run.smithi084.stdout:{"pg_ready":true,"pg_map":{"version":1293,"stamp":"2024-09-20T22:56:47.867566+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":681926174,"num_objects":501,"num_object_clones":252,"num_object_copies":1503,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":501,"num_whiteouts":12,"num_read":8426,"num_read_kb":6854126,"num_write":5377,"num_write_kb":1770068,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1797,"num_bytes_recovered":943057011,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17564,"ondisk_log_size":17564,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":474,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":1997144,"kb_used_data":1527724,"kb_used_omap":10,"kb_used_meta":469237,"kb_avail":654218920,"statfs":{"total":671965249536,"available":669920174080,"internally_reserved":0,"allocated":1564389376,"data_stored":2085255268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11162,"internal_metadata":480498790},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001290"},"pg_stats":[{"pgid":"3.1f","version":"86'2","reported_seq":1111,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-20T22:56:43.351527+0000","last_change":"2024-09-20T22:54:40.830995+0000","last_active":"2024-09-20T22:56:43.351527+0000","last_peered":"2024-09-20T22:56:43.351527+0000","last_clean":"2024-09-20T22:56:43.351527+0000","last_became_active":"2024-09-20T22:47:49.222114+0000","last_became_peered":"2024-09-20T22:47:49.222114+0000","last_unstale":"2024-09-20T22:56:43.351527+0000","last_undegraded":"2024-09-20T22:56:43.351527+0000","last_fullsized":"2024-09-20T22:56:43.351527+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'2","last_scrub_stamp":"2024-09-20T22:54:40.830920+0000","last_deep_scrub":"86'2","last_deep_scrub_stamp":"2024-09-20T22:42:39.664421+0000","last_clean_scrub_stamp":"2024-09-20T22:54:40.830920+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":436,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"625'15","reported_seq":684,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-20T22:56:43.356155+0000","last_change":"2024-09-20T22:56:14.882214+0000","last_active":"2024-09-20T22:56:43.356155+0000","last_peered":"2024-09-20T22:56:43.356155+0000","last_clean":"2024-09-20T22:56:43.356155+0000","last_became_active":"2024-09-20T22:55:17.341556+0000","last_became_peered":"2024-09-20T22:55:17.341556+0000","last_unstale":"2024-09-20T22:56:43.356155+0000","last_undegraded":"2024-09-20T22:56:43.356155+0000","last_fullsized":"2024-09-20T22:56:43.356155+0000","mapping_epoch":836,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":837,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'15","last_scrub_stamp":"2024-09-20T22:56:14.882123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:56:14.882123+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:35.216770+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":317,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-20T22:56:43.355750+0000","last_change":"2024-09-20T22:56:08.363427+0000","last_active":"2024-09-20T22:56:43.355750+0000","last_peered":"2024-09-20T22:56:43.355750+0000","last_clean":"2024-09-20T22:56:43.355750+0000","last_became_active":"2024-09-20T22:56:08.362405+0000","last_became_peered":"2024-09-20T22:56:08.362405+0000","last_unstale":"2024-09-20T22:56:43.355750+0000","last_undegraded":"2024-09-20T22:56:43.355750+0000","last_fullsized":"2024-09-20T22:56:43.355750+0000","mapping_epoch":887,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:55:34.919340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:55:34.919340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:03.606446+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":974,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-20T22:56:43.355422+0000","last_change":"2024-09-20T22:56:22.519967+0000","last_active":"2024-09-20T22:56:43.355422+0000","last_peered":"2024-09-20T22:56:43.355422+0000","last_clean":"2024-09-20T22:56:43.355422+0000","last_became_active":"2024-09-20T22:55:17.342068+0000","last_became_peered":"2024-09-20T22:55:17.342068+0000","last_unstale":"2024-09-20T22:56:43.355422+0000","last_undegraded":"2024-09-20T22:56:43.355422+0000","last_fullsized":"2024-09-20T22:56:43.355422+0000","mapping_epoch":836,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":837,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:56:22.519884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:56:22.519884+0000","last_clean_scrub_stamp":"2024-09-20T22:56:22.519884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:38.281445+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":1759,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-20T22:56:43.352631+0000","last_change":"2024-09-20T22:56:08.393422+0000","last_active":"2024-09-20T22:56:43.352631+0000","last_peered":"2024-09-20T22:56:43.352631+0000","last_clean":"2024-09-20T22:56:43.352631+0000","last_became_active":"2024-09-20T22:56:08.393096+0000","last_became_peered":"2024-09-20T22:56:08.393096+0000","last_unstale":"2024-09-20T22:56:43.352631+0000","last_undegraded":"2024-09-20T22:56:43.352631+0000","last_fullsized":"2024-09-20T22:56:43.352631+0000","mapping_epoch":887,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:54:15.238344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:54:15.238344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":1759,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-20T22:56:43.352834+0000","last_change":"2024-09-20T22:54:47.427999+0000","last_active":"2024-09-20T22:56:43.352834+0000","last_peered":"2024-09-20T22:56:43.352834+0000","last_clean":"2024-09-20T22:56:43.352834+0000","last_became_active":"2024-09-20T22:41:38.157298+0000","last_became_peered":"2024-09-20T22:41:38.157298+0000","last_unstale":"2024-09-20T22:56:43.352834+0000","last_undegraded":"2024-09-20T22:56:43.352834+0000","last_fullsized":"2024-09-20T22:56:43.352834+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:54:47.427927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:54:47.427927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"625'27","reported_seq":1841,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-20T22:56:43.350223+0000","last_change":"2024-09-20T22:54:45.390249+0000","last_active":"2024-09-20T22:56:43.350223+0000","last_peered":"2024-09-20T22:56:43.350223+0000","last_clean":"2024-09-20T22:56:43.350223+0000","last_became_active":"2024-09-20T22:47:49.218798+0000","last_became_peered":"2024-09-20T22:47:49.218798+0000","last_unstale":"2024-09-20T22:56:43.350223+0000","last_undegraded":"2024-09-20T22:56:43.350223+0000","last_fullsized":"2024-09-20T22:56:43.350223+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'27","last_scrub_stamp":"2024-09-20T22:54:45.390169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:54:45.390169+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":966,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-20T22:56:43.356333+0000","last_change":"2024-09-20T22:55:25.730415+0000","last_active":"2024-09-20T22:56:43.356333+0000","last_peered":"2024-09-20T22:56:43.356333+0000","last_clean":"2024-09-20T22:56:43.356333+0000","last_became_active":"2024-09-20T22:55:17.341160+0000","last_became_peered":"2024-09-20T22:55:17.341160+0000","last_unstale":"2024-09-20T22:56:43.356333+0000","last_undegraded":"2024-09-20T22:56:43.356333+0000","last_fullsized":"2024-09-20T22:56:43.356333+0000","mapping_epoch":836,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":837,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:55:25.730335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:55:25.730335+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:56:43.672450+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":1785,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-20T22:56:43.356798+0000","last_change":"2024-09-20T22:56:20.497450+0000","last_active":"2024-09-20T22:56:43.356798+0000","last_peered":"2024-09-20T22:56:43.356798+0000","last_clean":"2024-09-20T22:56:43.356798+0000","last_became_active":"2024-09-20T22:48:05.242059+0000","last_became_peered":"2024-09-20T22:48:05.242059+0000","last_unstale":"2024-09-20T22:56:43.356798+0000","last_undegraded":"2024-09-20T22:56:43.356798+0000","last_fullsized":"2024-09-20T22:56:43.356798+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:56:20.497377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:46:00.999939+0000","last_clean_scrub_stamp":"2024-09-20T22:56:20.497377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:34.215098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"625'15","reported_seq":669,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-20T22:56:43.351199+0000","last_change":"2024-09-20T22:55:17.348852+0000","last_active":"2024-09-20T22:56:43.351199+0000","last_peered":"2024-09-20T22:56:43.351199+0000","last_clean":"2024-09-20T22:56:43.351199+0000","last_became_active":"2024-09-20T22:55:17.348531+0000","last_became_peered":"2024-09-20T22:55:17.348531+0000","last_unstale":"2024-09-20T22:56:43.351199+0000","last_undegraded":"2024-09-20T22:56:43.351199+0000","last_fullsized":"2024-09-20T22:56:43.351199+0000","mapping_epoch":836,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":837,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'15","last_scrub_stamp":"2024-09-20T22:53:51.486202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:53:51.486202+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":1783,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-20T22:56:43.357370+0000","last_change":"2024-09-20T22:56:42.536987+0000","last_active":"2024-09-20T22:56:43.357370+0000","last_peered":"2024-09-20T22:56:43.357370+0000","last_clean":"2024-09-20T22:56:43.357370+0000","last_became_active":"2024-09-20T22:56:08.356419+0000","last_became_peered":"2024-09-20T22:56:08.356419+0000","last_unstale":"2024-09-20T22:56:43.357370+0000","last_undegraded":"2024-09-20T22:56:43.357370+0000","last_fullsized":"2024-09-20T22:56:43.357370+0000","mapping_epoch":887,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:56:42.536863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:56:42.536863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:46.158314+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":1132,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-20T22:56:43.349138+0000","last_change":"2024-09-20T22:56:08.391304+0000","last_active":"2024-09-20T22:56:43.349138+0000","last_peered":"2024-09-20T22:56:43.349138+0000","last_clean":"2024-09-20T22:56:43.349138+0000","last_became_active":"2024-09-20T22:56:08.390640+0000","last_became_peered":"2024-09-20T22:56:08.390640+0000","last_unstale":"2024-09-20T22:56:43.349138+0000","last_undegraded":"2024-09-20T22:56:43.349138+0000","last_fullsized":"2024-09-20T22:56:43.349138+0000","mapping_epoch":887,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:55:55.454925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:49:08.664407+0000","last_clean_scrub_stamp":"2024-09-20T22:55:55.454925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:17.994006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":1797,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-20T22:56:43.356130+0000","last_change":"2024-09-20T22:56:13.490259+0000","last_active":"2024-09-20T22:56:43.356130+0000","last_peered":"2024-09-20T22:56:43.356130+0000","last_clean":"2024-09-20T22:56:43.356130+0000","last_became_active":"2024-09-20T22:54:17.342619+0000","last_became_peered":"2024-09-20T22:54:17.342619+0000","last_unstale":"2024-09-20T22:56:43.356130+0000","last_undegraded":"2024-09-20T22:56:43.356130+0000","last_fullsized":"2024-09-20T22:56:43.356130+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:56:13.490178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:56:13.490178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:19.084611+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"625'9","reported_seq":679,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-20T22:56:43.353152+0000","last_change":"2024-09-20T22:56:24.469505+0000","last_active":"2024-09-20T22:56:43.353152+0000","last_peered":"2024-09-20T22:56:43.353152+0000","last_clean":"2024-09-20T22:56:43.353152+0000","last_became_active":"2024-09-20T22:55:17.350145+0000","last_became_peered":"2024-09-20T22:55:17.350145+0000","last_unstale":"2024-09-20T22:56:43.353152+0000","last_undegraded":"2024-09-20T22:56:43.353152+0000","last_fullsized":"2024-09-20T22:56:43.353152+0000","mapping_epoch":836,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":837,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'9","last_scrub_stamp":"2024-09-20T22:56:24.469428+0000","last_deep_scrub":"181'7","last_deep_scrub_stamp":"2024-09-20T22:48:41.049381+0000","last_clean_scrub_stamp":"2024-09-20T22:56:24.469428+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:52.398957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":8,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":1575,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-20T22:56:43.356139+0000","last_change":"2024-09-20T22:56:08.357180+0000","last_active":"2024-09-20T22:56:43.356139+0000","last_peered":"2024-09-20T22:56:43.356139+0000","last_clean":"2024-09-20T22:56:43.356139+0000","last_became_active":"2024-09-20T22:56:08.356857+0000","last_became_peered":"2024-09-20T22:56:08.356857+0000","last_unstale":"2024-09-20T22:56:43.356139+0000","last_undegraded":"2024-09-20T22:56:43.356139+0000","last_fullsized":"2024-09-20T22:56:43.356139+0000","mapping_epoch":887,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:55:59.860836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:47:40.361958+0000","last_clean_scrub_stamp":"2024-09-20T22:55:59.860836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:23.952421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":1776,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-20T22:56:43.355242+0000","last_change":"2024-09-20T22:56:14.512639+0000","last_active":"2024-09-20T22:56:43.355242+0000","last_peered":"2024-09-20T22:56:43.355242+0000","last_clean":"2024-09-20T22:56:43.355242+0000","last_became_active":"2024-09-20T22:54:17.339658+0000","last_became_peered":"2024-09-20T22:54:17.339658+0000","last_unstale":"2024-09-20T22:56:43.355242+0000","last_undegraded":"2024-09-20T22:56:43.355242+0000","last_fullsized":"2024-09-20T22:56:43.355242+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:56:14.512566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:56:14.512566+0000","last_clean_scrub_stamp":"2024-09-20T22:56:14.512566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:23.668214+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":193,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-20T22:56:43.357344+0000","last_change":"2024-09-20T22:55:32.840041+0000","last_active":"2024-09-20T22:56:43.357344+0000","last_peered":"2024-09-20T22:56:43.357344+0000","last_clean":"2024-09-20T22:56:43.357344+0000","last_became_active":"2024-09-20T22:55:17.343025+0000","last_became_peered":"2024-09-20T22:55:17.343025+0000","last_unstale":"2024-09-20T22:56:43.357344+0000","last_undegraded":"2024-09-20T22:56:43.357344+0000","last_fullsized":"2024-09-20T22:56:43.357344+0000","mapping_epoch":836,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":837,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:55:32.839935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:55:32.839935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":1118,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-20T22:56:43.350470+0000","last_change":"2024-09-20T22:56:08.392264+0000","last_active":"2024-09-20T22:56:43.350470+0000","last_peered":"2024-09-20T22:56:43.350470+0000","last_clean":"2024-09-20T22:56:43.350470+0000","last_became_active":"2024-09-20T22:56:08.391965+0000","last_became_peered":"2024-09-20T22:56:08.391965+0000","last_unstale":"2024-09-20T22:56:43.350470+0000","last_undegraded":"2024-09-20T22:56:43.350470+0000","last_fullsized":"2024-09-20T22:56:43.350470+0000","mapping_epoch":887,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:55:06.636534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:49:53.933601+0000","last_clean_scrub_stamp":"2024-09-20T22:55:06.636534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"625'34","reported_seq":1879,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-20T22:56:43.372441+0000","last_change":"2024-09-20T22:55:04.302057+0000","last_active":"2024-09-20T22:56:43.372441+0000","last_peered":"2024-09-20T22:56:43.372441+0000","last_clean":"2024-09-20T22:56:43.372441+0000","last_became_active":"2024-09-20T22:54:17.331250+0000","last_became_peered":"2024-09-20T22:54:17.331250+0000","last_unstale":"2024-09-20T22:56:43.372441+0000","last_undegraded":"2024-09-20T22:56:43.372441+0000","last_fullsized":"2024-09-20T22:56:43.372441+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'34","last_scrub_stamp":"2024-09-20T22:55:04.301955+0000","last_deep_scrub":"113'22","last_deep_scrub_stamp":"2024-09-20T22:43:12.191878+0000","last_clean_scrub_stamp":"2024-09-20T22:55:04.301955+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":318,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-20T22:56:43.355375+0000","last_change":"2024-09-20T22:55:57.217769+0000","last_active":"2024-09-20T22:56:43.355375+0000","last_peered":"2024-09-20T22:56:43.355375+0000","last_clean":"2024-09-20T22:56:43.355375+0000","last_became_active":"2024-09-20T22:54:17.349501+0000","last_became_peered":"2024-09-20T22:54:17.349501+0000","last_unstale":"2024-09-20T22:56:43.355375+0000","last_undegraded":"2024-09-20T22:56:43.355375+0000","last_fullsized":"2024-09-20T22:56:43.355375+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:55:57.217692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:55:57.217692+0000","last_clean_scrub_stamp":"2024-09-20T22:55:57.217692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:56:57.648013+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":1747,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-20T22:56:43.351150+0000","last_change":"2024-09-20T22:54:02.131730+0000","last_active":"2024-09-20T22:56:43.351150+0000","last_peered":"2024-09-20T22:56:43.351150+0000","last_clean":"2024-09-20T22:56:43.351150+0000","last_became_active":"2024-09-20T22:41:44.832131+0000","last_became_peered":"2024-09-20T22:41:44.832131+0000","last_unstale":"2024-09-20T22:56:43.351150+0000","last_undegraded":"2024-09-20T22:56:43.351150+0000","last_fullsized":"2024-09-20T22:56:43.351150+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:54:02.131659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:45:25.002199+0000","last_clean_scrub_stamp":"2024-09-20T22:54:02.131659+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":974,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-20T22:56:43.356030+0000","last_change":"2024-09-20T22:55:29.864586+0000","last_active":"2024-09-20T22:56:43.356030+0000","last_peered":"2024-09-20T22:56:43.356030+0000","last_clean":"2024-09-20T22:56:43.356030+0000","last_became_active":"2024-09-20T22:55:17.346400+0000","last_became_peered":"2024-09-20T22:55:17.346400+0000","last_unstale":"2024-09-20T22:56:43.356030+0000","last_undegraded":"2024-09-20T22:56:43.356030+0000","last_fullsized":"2024-09-20T22:56:43.356030+0000","mapping_epoch":836,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":837,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:55:29.864513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:50:41.885844+0000","last_clean_scrub_stamp":"2024-09-20T22:55:29.864513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:56:51.327377+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"625'18","reported_seq":1876,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-20T22:56:43.349745+0000","last_change":"2024-09-20T22:54:18.284372+0000","last_active":"2024-09-20T22:56:43.349745+0000","last_peered":"2024-09-20T22:56:43.349745+0000","last_clean":"2024-09-20T22:56:43.349745+0000","last_became_active":"2024-09-20T22:54:17.333427+0000","last_became_peered":"2024-09-20T22:54:17.333427+0000","last_unstale":"2024-09-20T22:56:43.349745+0000","last_undegraded":"2024-09-20T22:56:43.349745+0000","last_fullsized":"2024-09-20T22:56:43.349745+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'18","last_scrub_stamp":"2024-09-20T22:54:18.284289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:54:18.284289+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":1100,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":701,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-20T22:56:43.352794+0000","last_change":"2024-09-20T22:56:18.447195+0000","last_active":"2024-09-20T22:56:43.352794+0000","last_peered":"2024-09-20T22:56:43.352794+0000","last_clean":"2024-09-20T22:56:43.352794+0000","last_became_active":"2024-09-20T22:55:17.349132+0000","last_became_peered":"2024-09-20T22:55:17.349132+0000","last_unstale":"2024-09-20T22:56:43.352794+0000","last_undegraded":"2024-09-20T22:56:43.352794+0000","last_fullsized":"2024-09-20T22:56:43.352794+0000","mapping_epoch":836,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":837,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:56:18.447131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:56:18.447131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:30.980823+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"154'2","reported_seq":262,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-20T22:56:43.355851+0000","last_change":"2024-09-20T22:56:08.415155+0000","last_active":"2024-09-20T22:56:43.355851+0000","last_peered":"2024-09-20T22:56:43.355851+0000","last_clean":"2024-09-20T22:56:43.355851+0000","last_became_active":"2024-09-20T22:56:08.374938+0000","last_became_peered":"2024-09-20T22:56:08.374938+0000","last_unstale":"2024-09-20T22:56:43.355851+0000","last_undegraded":"2024-09-20T22:56:43.355851+0000","last_fullsized":"2024-09-20T22:56:43.355851+0000","mapping_epoch":887,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":0,"last_scrub":"154'2","last_scrub_stamp":"2024-09-20T22:55:06.675346+0000","last_deep_scrub":"154'2","last_deep_scrub_stamp":"2024-09-20T22:43:54.134066+0000","last_clean_scrub_stamp":"2024-09-20T22:55:06.675346+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":403,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"894'2","reported_seq":324,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-20T22:56:43.355685+0000","last_change":"2024-09-20T22:56:15.522716+0000","last_active":"2024-09-20T22:56:43.355685+0000","last_peered":"2024-09-20T22:56:43.355685+0000","last_clean":"2024-09-20T22:56:43.355685+0000","last_became_active":"2024-09-20T22:54:17.348571+0000","last_became_peered":"2024-09-20T22:54:17.348571+0000","last_unstale":"2024-09-20T22:56:43.355685+0000","last_undegraded":"2024-09-20T22:56:43.355685+0000","last_fullsized":"2024-09-20T22:56:43.355685+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"894'2","last_scrub_stamp":"2024-09-20T22:56:15.522649+0000","last_deep_scrub":"894'2","last_deep_scrub_stamp":"2024-09-20T22:56:15.522649+0000","last_clean_scrub_stamp":"2024-09-20T22:56:15.522649+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:24.408024+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"625'25","reported_seq":1160,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-20T22:56:43.352000+0000","last_change":"2024-09-20T22:56:08.431461+0000","last_active":"2024-09-20T22:56:43.352000+0000","last_peered":"2024-09-20T22:56:43.352000+0000","last_clean":"2024-09-20T22:56:43.352000+0000","last_became_active":"2024-09-20T22:56:08.395053+0000","last_became_peered":"2024-09-20T22:56:08.395053+0000","last_unstale":"2024-09-20T22:56:43.352000+0000","last_undegraded":"2024-09-20T22:56:43.352000+0000","last_fullsized":"2024-09-20T22:56:43.352000+0000","mapping_epoch":887,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'25","last_scrub_stamp":"2024-09-20T22:55:26.646181+0000","last_deep_scrub":"151'15","last_deep_scrub_stamp":"2024-09-20T22:43:50.715761+0000","last_clean_scrub_stamp":"2024-09-20T22:55:26.646181+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"888'12","reported_seq":2017,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-20T22:56:43.357901+0000","last_change":"2024-09-20T22:56:42.107724+0000","last_active":"2024-09-20T22:56:43.357901+0000","last_peered":"2024-09-20T22:56:43.357901+0000","last_clean":"2024-09-20T22:56:43.357901+0000","last_became_active":"2024-09-20T22:56:08.368232+0000","last_became_peered":"2024-09-20T22:56:08.368232+0000","last_unstale":"2024-09-20T22:56:43.357901+0000","last_undegraded":"2024-09-20T22:56:43.357901+0000","last_fullsized":"2024-09-20T22:56:43.357901+0000","mapping_epoch":887,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":0,"last_scrub":"888'12","last_scrub_stamp":"2024-09-20T22:56:42.107635+0000","last_deep_scrub":"661'11","last_deep_scrub_stamp":"2024-09-20T22:52:21.907943+0000","last_clean_scrub_stamp":"2024-09-20T22:56:42.107635+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:09.490301+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.c","version":"923'3034","reported_seq":7358,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-20T22:56:43.384904+0000","last_change":"2024-09-20T22:56:43.385125+0000","last_active":"2024-09-20T22:56:43.384904+0000","last_peered":"2024-09-20T22:56:43.384904+0000","last_clean":"2024-09-20T22:56:43.384904+0000","last_became_active":"2024-09-20T22:56:11.368076+0000","last_became_peered":"2024-09-20T22:56:11.368076+0000","last_unstale":"2024-09-20T22:56:43.384904+0000","last_undegraded":"2024-09-20T22:56:43.384904+0000","last_fullsized":"2024-09-20T22:56:43.384904+0000","mapping_epoch":890,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":891,"parent":"0.0","parent_split_bits":0,"last_scrub":"902'2941","last_scrub_stamp":"2024-09-20T22:56:22.993088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:56:22.993088+0000","objects_scrubbed":36,"log_size":3034,"log_dups_size":0,"ondisk_log_size":3034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:45.060776+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.028548857,"stat_sum":{"num_bytes":86638052,"num_objects":37,"num_object_clones":30,"num_object_copies":111,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":1,"num_read":1068,"num_read_kb":1111924,"num_write":720,"num_write_kb":282381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":175,"num_bytes_recovered":66329111,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"13a","length":"1"},{"start":"179","length":"4"},{"start":"181","length":"4"},{"start":"186","length":"3"},{"start":"18a","length":"3"},{"start":"191","length":"2"},{"start":"194","length":"2"},{"start":"199","length":"1"},{"start":"19b","length":"1"},{"start":"19e","length":"2"},{"start":"1a1","length":"1"}]},{"pgid":"1.a","version":"77'128","reported_seq":2034,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-20T22:56:43.355323+0000","last_change":"2024-09-20T22:56:00.375695+0000","last_active":"2024-09-20T22:56:43.355323+0000","last_peered":"2024-09-20T22:56:43.355323+0000","last_clean":"2024-09-20T22:56:43.355323+0000","last_became_active":"2024-09-20T22:42:52.407791+0000","last_became_peered":"2024-09-20T22:42:52.407791+0000","last_unstale":"2024-09-20T22:56:43.355323+0000","last_undegraded":"2024-09-20T22:56:43.355323+0000","last_fullsized":"2024-09-20T22:56:43.355323+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:56:00.375623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:56:00.375623+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:08.673875+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"625'27","reported_seq":335,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-20T22:56:43.356582+0000","last_change":"2024-09-20T22:55:50.161066+0000","last_active":"2024-09-20T22:56:43.356582+0000","last_peered":"2024-09-20T22:56:43.356582+0000","last_clean":"2024-09-20T22:56:43.356582+0000","last_became_active":"2024-09-20T22:54:17.347544+0000","last_became_peered":"2024-09-20T22:54:17.347544+0000","last_unstale":"2024-09-20T22:56:43.356582+0000","last_undegraded":"2024-09-20T22:56:43.356582+0000","last_fullsized":"2024-09-20T22:56:43.356582+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'27","last_scrub_stamp":"2024-09-20T22:55:50.160922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:55:50.160922+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:12.756692+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":1776,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-20T22:56:43.349083+0000","last_change":"2024-09-20T22:56:08.394269+0000","last_active":"2024-09-20T22:56:43.349083+0000","last_peered":"2024-09-20T22:56:43.349083+0000","last_clean":"2024-09-20T22:56:43.349083+0000","last_became_active":"2024-09-20T22:56:08.393984+0000","last_became_peered":"2024-09-20T22:56:08.393984+0000","last_unstale":"2024-09-20T22:56:43.349083+0000","last_undegraded":"2024-09-20T22:56:43.349083+0000","last_fullsized":"2024-09-20T22:56:43.349083+0000","mapping_epoch":887,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:55:09.369892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:49:12.070675+0000","last_clean_scrub_stamp":"2024-09-20T22:55:09.369892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"838'13","reported_seq":818,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-20T22:56:43.356682+0000","last_change":"2024-09-20T22:55:17.343867+0000","last_active":"2024-09-20T22:56:43.356682+0000","last_peered":"2024-09-20T22:56:43.356682+0000","last_clean":"2024-09-20T22:56:43.356682+0000","last_became_active":"2024-09-20T22:55:17.343505+0000","last_became_peered":"2024-09-20T22:55:17.343505+0000","last_unstale":"2024-09-20T22:56:43.356682+0000","last_undegraded":"2024-09-20T22:56:43.356682+0000","last_fullsized":"2024-09-20T22:56:43.356682+0000","mapping_epoch":836,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":837,"parent":"0.0","parent_split_bits":0,"last_scrub":"408'12","last_scrub_stamp":"2024-09-20T22:54:22.471772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:54:22.471772+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":1785,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-20T22:56:43.356102+0000","last_change":"2024-09-20T22:56:41.547004+0000","last_active":"2024-09-20T22:56:43.356102+0000","last_peered":"2024-09-20T22:56:43.356102+0000","last_clean":"2024-09-20T22:56:43.356102+0000","last_became_active":"2024-09-20T22:54:17.340327+0000","last_became_peered":"2024-09-20T22:54:17.340327+0000","last_unstale":"2024-09-20T22:56:43.356102+0000","last_undegraded":"2024-09-20T22:56:43.356102+0000","last_fullsized":"2024-09-20T22:56:43.356102+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:56:41.546923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:56:41.546923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:04.615941+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3","version":"917'467","reported_seq":4408,"reported_epoch":923,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-20T22:56:43.614666+0000","last_change":"2024-09-20T22:56:43.614715+0000","last_active":"2024-09-20T22:56:43.614666+0000","last_peered":"2024-09-20T22:56:43.614666+0000","last_clean":"2024-09-20T22:56:43.614666+0000","last_became_active":"2024-09-20T22:49:05.221402+0000","last_became_peered":"2024-09-20T22:49:05.221402+0000","last_unstale":"2024-09-20T22:56:43.614666+0000","last_undegraded":"2024-09-20T22:56:43.614666+0000","last_fullsized":"2024-09-20T22:56:43.614666+0000","mapping_epoch":404,"log_start":"456'129","ondisk_log_start":"456'129","created":94,"last_epoch_clean":465,"parent":"0.0","parent_split_bits":0,"last_scrub":"849'426","last_scrub_stamp":"2024-09-20T22:55:29.475387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:55:29.475387+0000","objects_scrubbed":9,"log_size":338,"log_dups_size":149,"ondisk_log_size":338,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":14,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.262137173,"stat_sum":{"num_bytes":14724859,"num_objects":8,"num_object_clones":7,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":1,"num_read":138,"num_read_kb":170336,"num_write":103,"num_write_kb":33443,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":20196357,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"13a","length":"1"},{"start":"179","length":"4"},{"start":"181","length":"4"},{"start":"186","length":"1"}]},{"pgid":"1.5","version":"77'128","reported_seq":1050,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-20T22:56:43.365777+0000","last_change":"2024-09-20T22:55:17.345787+0000","last_active":"2024-09-20T22:56:43.365777+0000","last_peered":"2024-09-20T22:56:43.365777+0000","last_clean":"2024-09-20T22:56:43.365777+0000","last_became_active":"2024-09-20T22:55:17.345460+0000","last_became_peered":"2024-09-20T22:55:17.345460+0000","last_unstale":"2024-09-20T22:56:43.365777+0000","last_undegraded":"2024-09-20T22:56:43.365777+0000","last_fullsized":"2024-09-20T22:56:43.365777+0000","mapping_epoch":836,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":837,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:55:04.973719+0000","last_deep_scrub":"77'128","last_deep_scrub_stamp":"2024-09-20T22:45:41.608075+0000","last_clean_scrub_stamp":"2024-09-20T22:55:04.973719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"625'30","reported_seq":1877,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-20T22:56:43.349581+0000","last_change":"2024-09-20T22:56:08.426797+0000","last_active":"2024-09-20T22:56:43.349581+0000","last_peered":"2024-09-20T22:56:43.349581+0000","last_clean":"2024-09-20T22:56:43.349581+0000","last_became_active":"2024-09-20T22:56:08.393632+0000","last_became_peered":"2024-09-20T22:56:08.393632+0000","last_unstale":"2024-09-20T22:56:43.349581+0000","last_undegraded":"2024-09-20T22:56:43.349581+0000","last_fullsized":"2024-09-20T22:56:43.349581+0000","mapping_epoch":887,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'30","last_scrub_stamp":"2024-09-20T22:53:48.346341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:53:48.346341+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":1784,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-20T22:56:43.351753+0000","last_change":"2024-09-20T22:56:15.465303+0000","last_active":"2024-09-20T22:56:43.351753+0000","last_peered":"2024-09-20T22:56:43.351753+0000","last_clean":"2024-09-20T22:56:43.351753+0000","last_became_active":"2024-09-20T22:56:08.396456+0000","last_became_peered":"2024-09-20T22:56:08.396456+0000","last_unstale":"2024-09-20T22:56:43.351753+0000","last_undegraded":"2024-09-20T22:56:43.351753+0000","last_fullsized":"2024-09-20T22:56:43.351753+0000","mapping_epoch":887,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:56:15.465230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:56:15.465230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:30.581072+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":711,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-20T22:56:43.349131+0000","last_change":"2024-09-20T22:56:08.379658+0000","last_active":"2024-09-20T22:56:43.349131+0000","last_peered":"2024-09-20T22:56:43.349131+0000","last_clean":"2024-09-20T22:56:43.349131+0000","last_became_active":"2024-09-20T22:56:08.379036+0000","last_became_peered":"2024-09-20T22:56:08.379036+0000","last_unstale":"2024-09-20T22:56:43.349131+0000","last_undegraded":"2024-09-20T22:56:43.349131+0000","last_fullsized":"2024-09-20T22:56:43.349131+0000","mapping_epoch":887,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:26.623455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:07.947945+0000","last_clean_scrub_stamp":"2024-09-20T22:53:26.623455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":702,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-20T22:56:43.351100+0000","last_change":"2024-09-20T22:55:48.456977+0000","last_active":"2024-09-20T22:56:43.351100+0000","last_peered":"2024-09-20T22:56:43.351100+0000","last_clean":"2024-09-20T22:56:43.351100+0000","last_became_active":"2024-09-20T22:55:17.351296+0000","last_became_peered":"2024-09-20T22:55:17.351296+0000","last_unstale":"2024-09-20T22:56:43.351100+0000","last_undegraded":"2024-09-20T22:56:43.351100+0000","last_fullsized":"2024-09-20T22:56:43.351100+0000","mapping_epoch":836,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":837,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:55:48.456896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:53.036601+0000","last_clean_scrub_stamp":"2024-09-20T22:55:48.456896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:07.475090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2","version":"921'993","reported_seq":4435,"reported_epoch":923,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T22:56:43.350511+0000","last_change":"2024-09-20T22:56:42.487298+0000","last_active":"2024-09-20T22:56:43.350511+0000","last_peered":"2024-09-20T22:56:43.350511+0000","last_clean":"2024-09-20T22:56:43.350511+0000","last_became_active":"2024-09-20T22:54:50.348406+0000","last_became_peered":"2024-09-20T22:54:50.348406+0000","last_unstale":"2024-09-20T22:56:43.350511+0000","last_undegraded":"2024-09-20T22:56:43.350511+0000","last_fullsized":"2024-09-20T22:56:43.350511+0000","mapping_epoch":809,"log_start":"471'464","ondisk_log_start":"471'464","created":94,"last_epoch_clean":810,"parent":"0.0","parent_split_bits":0,"last_scrub":"282'120","last_scrub_stamp":"2024-09-20T22:46:22.099642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:46:22.099642+0000","objects_scrubbed":1,"log_size":529,"log_dups_size":687,"ondisk_log_size":529,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":23,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.059924837000000002,"stat_sum":{"num_bytes":43011538,"num_objects":17,"num_object_clones":13,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":533,"num_read_kb":566622,"num_write":324,"num_write_kb":124545,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":160,"num_bytes_recovered":146402851,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"13a","length":"1"}]},{"pgid":"1.4","version":"855'129","reported_seq":2078,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-20T22:56:43.352987+0000","last_change":"2024-09-20T22:56:11.368190+0000","last_active":"2024-09-20T22:56:43.352987+0000","last_peered":"2024-09-20T22:56:43.352987+0000","last_clean":"2024-09-20T22:56:43.352987+0000","last_became_active":"2024-09-20T22:56:11.367798+0000","last_became_peered":"2024-09-20T22:56:11.367798+0000","last_unstale":"2024-09-20T22:56:43.352987+0000","last_undegraded":"2024-09-20T22:56:43.352987+0000","last_fullsized":"2024-09-20T22:56:43.352987+0000","mapping_epoch":890,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":891,"parent":"0.0","parent_split_bits":3,"last_scrub":"855'129","last_scrub_stamp":"2024-09-20T22:55:35.536654+0000","last_deep_scrub":"855'129","last_deep_scrub_stamp":"2024-09-20T22:55:35.536654+0000","last_clean_scrub_stamp":"2024-09-20T22:55:35.536654+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":59,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":655360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"625'28","reported_seq":678,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-20T22:56:43.355306+0000","last_change":"2024-09-20T22:55:17.344950+0000","last_active":"2024-09-20T22:56:43.355306+0000","last_peered":"2024-09-20T22:56:43.355306+0000","last_clean":"2024-09-20T22:56:43.355306+0000","last_became_active":"2024-09-20T22:55:17.344611+0000","last_became_peered":"2024-09-20T22:55:17.344611+0000","last_unstale":"2024-09-20T22:56:43.355306+0000","last_undegraded":"2024-09-20T22:56:43.355306+0000","last_fullsized":"2024-09-20T22:56:43.355306+0000","mapping_epoch":836,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":837,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'28","last_scrub_stamp":"2024-09-20T22:54:46.173524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:54:46.173524+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":40,"num_read_kb":33,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.6","version":"855'2","reported_seq":1150,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-20T22:56:43.351561+0000","last_change":"2024-09-20T22:55:35.573049+0000","last_active":"2024-09-20T22:56:43.351561+0000","last_peered":"2024-09-20T22:56:43.351561+0000","last_clean":"2024-09-20T22:56:43.351561+0000","last_became_active":"2024-09-20T22:54:17.340423+0000","last_became_peered":"2024-09-20T22:54:17.340423+0000","last_unstale":"2024-09-20T22:56:43.351561+0000","last_undegraded":"2024-09-20T22:56:43.351561+0000","last_fullsized":"2024-09-20T22:56:43.351561+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"855'2","last_scrub_stamp":"2024-09-20T22:55:35.572977+0000","last_deep_scrub":"855'2","last_deep_scrub_stamp":"2024-09-20T22:55:35.572977+0000","last_clean_scrub_stamp":"2024-09-20T22:55:35.572977+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:56:55.056808+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":230,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"778'16","reported_seq":2004,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-20T22:56:43.349965+0000","last_change":"2024-09-20T22:55:10.378111+0000","last_active":"2024-09-20T22:56:43.349965+0000","last_peered":"2024-09-20T22:56:43.349965+0000","last_clean":"2024-09-20T22:56:43.349965+0000","last_became_active":"2024-09-20T22:54:17.341871+0000","last_became_peered":"2024-09-20T22:54:17.341871+0000","last_unstale":"2024-09-20T22:56:43.349965+0000","last_undegraded":"2024-09-20T22:56:43.349965+0000","last_fullsized":"2024-09-20T22:56:43.349965+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"778'16","last_scrub_stamp":"2024-09-20T22:55:10.378046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:55:10.378046+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":82,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-20T22:56:43.356938+0000","last_change":"2024-09-20T22:56:08.367482+0000","last_active":"2024-09-20T22:56:43.356938+0000","last_peered":"2024-09-20T22:56:43.356938+0000","last_clean":"2024-09-20T22:56:43.356938+0000","last_became_active":"2024-09-20T22:56:08.367182+0000","last_became_peered":"2024-09-20T22:56:08.367182+0000","last_unstale":"2024-09-20T22:56:43.356938+0000","last_undegraded":"2024-09-20T22:56:43.356938+0000","last_fullsized":"2024-09-20T22:56:43.356938+0000","mapping_epoch":887,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:55:23.858671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:55:23.858671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1","version":"920'561","reported_seq":4585,"reported_epoch":923,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-20T22:56:43.614377+0000","last_change":"2024-09-20T22:56:43.614432+0000","last_active":"2024-09-20T22:56:43.614377+0000","last_peered":"2024-09-20T22:56:43.614377+0000","last_clean":"2024-09-20T22:56:43.614377+0000","last_became_active":"2024-09-20T22:55:17.348786+0000","last_became_peered":"2024-09-20T22:55:17.348786+0000","last_unstale":"2024-09-20T22:56:43.614377+0000","last_undegraded":"2024-09-20T22:56:43.614377+0000","last_fullsized":"2024-09-20T22:56:43.614377+0000","mapping_epoch":836,"log_start":"474'259","ondisk_log_start":"474'259","created":94,"last_epoch_clean":837,"parent":"0.0","parent_split_bits":0,"last_scrub":"794'485","last_scrub_stamp":"2024-09-20T22:54:38.213249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:54:38.213249+0000","objects_scrubbed":8,"log_size":302,"log_dups_size":338,"ondisk_log_size":302,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":15,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.26540656600000001,"stat_sum":{"num_bytes":25803414,"num_objects":11,"num_object_clones":9,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":160,"num_read_kb":172572,"num_write":128,"num_write_kb":63761,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":8732217,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"13a","length":"1"},{"start":"179","length":"4"},{"start":"181","length":"4"}]},{"pgid":"1.7","version":"77'128","reported_seq":2095,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-20T22:56:43.357220+0000","last_change":"2024-09-20T22:56:11.369280+0000","last_active":"2024-09-20T22:56:43.357220+0000","last_peered":"2024-09-20T22:56:43.357220+0000","last_clean":"2024-09-20T22:56:43.357220+0000","last_became_active":"2024-09-20T22:56:11.368869+0000","last_became_peered":"2024-09-20T22:56:11.368869+0000","last_unstale":"2024-09-20T22:56:43.357220+0000","last_undegraded":"2024-09-20T22:56:43.357220+0000","last_fullsized":"2024-09-20T22:56:43.357220+0000","mapping_epoch":890,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":891,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:55:50.928296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:55:50.928296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:03.750814+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"904'25","reported_seq":336,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-20T22:56:43.356232+0000","last_change":"2024-09-20T22:56:24.575232+0000","last_active":"2024-09-20T22:56:43.356232+0000","last_peered":"2024-09-20T22:56:43.356232+0000","last_clean":"2024-09-20T22:56:43.356232+0000","last_became_active":"2024-09-20T22:54:17.349113+0000","last_became_peered":"2024-09-20T22:54:17.349113+0000","last_unstale":"2024-09-20T22:56:43.356232+0000","last_undegraded":"2024-09-20T22:56:43.356232+0000","last_fullsized":"2024-09-20T22:56:43.356232+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"904'25","last_scrub_stamp":"2024-09-20T22:56:24.575150+0000","last_deep_scrub":"904'25","last_deep_scrub_stamp":"2024-09-20T22:56:24.575150+0000","last_clean_scrub_stamp":"2024-09-20T22:56:24.575150+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:30.064695+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":28,"num_read_kb":23,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":1774,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-20T22:56:43.356927+0000","last_change":"2024-09-20T22:55:53.487471+0000","last_active":"2024-09-20T22:56:43.356927+0000","last_peered":"2024-09-20T22:56:43.356927+0000","last_clean":"2024-09-20T22:56:43.356927+0000","last_became_active":"2024-09-20T22:41:38.155629+0000","last_became_peered":"2024-09-20T22:41:38.155629+0000","last_unstale":"2024-09-20T22:56:43.356927+0000","last_undegraded":"2024-09-20T22:56:43.356927+0000","last_fullsized":"2024-09-20T22:56:43.356927+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:55:53.487366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:55:53.487366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:17.005975+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"733'12","reported_seq":1228,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-20T22:56:43.350420+0000","last_change":"2024-09-20T22:55:50.416762+0000","last_active":"2024-09-20T22:56:43.350420+0000","last_peered":"2024-09-20T22:56:43.350420+0000","last_clean":"2024-09-20T22:56:43.350420+0000","last_became_active":"2024-09-20T22:47:49.227264+0000","last_became_peered":"2024-09-20T22:47:49.227264+0000","last_unstale":"2024-09-20T22:56:43.350420+0000","last_undegraded":"2024-09-20T22:56:43.350420+0000","last_fullsized":"2024-09-20T22:56:43.350420+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"733'12","last_scrub_stamp":"2024-09-20T22:55:50.416689+0000","last_deep_scrub":"733'12","last_deep_scrub_stamp":"2024-09-20T22:53:33.928499+0000","last_clean_scrub_stamp":"2024-09-20T22:55:50.416689+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:56:57.805983+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":1115,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-20T22:56:43.348973+0000","last_change":"2024-09-20T22:55:57.400717+0000","last_active":"2024-09-20T22:56:43.348973+0000","last_peered":"2024-09-20T22:56:43.348973+0000","last_clean":"2024-09-20T22:56:43.348973+0000","last_became_active":"2024-09-20T22:47:49.232380+0000","last_became_peered":"2024-09-20T22:47:49.232380+0000","last_unstale":"2024-09-20T22:56:43.348973+0000","last_undegraded":"2024-09-20T22:56:43.348973+0000","last_fullsized":"2024-09-20T22:56:43.348973+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:55:57.400645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:55:57.400645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:20.573620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.0","version":"923'979","reported_seq":5197,"reported_epoch":923,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T22:56:43.355484+0000","last_change":"2024-09-20T22:56:43.355484+0000","last_active":"2024-09-20T22:56:43.355484+0000","last_peered":"2024-09-20T22:56:43.355484+0000","last_clean":"2024-09-20T22:56:43.355484+0000","last_became_active":"2024-09-20T22:54:20.418799+0000","last_became_peered":"2024-09-20T22:54:20.418799+0000","last_unstale":"2024-09-20T22:56:43.355484+0000","last_undegraded":"2024-09-20T22:56:43.355484+0000","last_fullsized":"2024-09-20T22:56:43.355484+0000","mapping_epoch":779,"log_start":"480'253","ondisk_log_start":"480'253","created":94,"last_epoch_clean":780,"parent":"0.0","parent_split_bits":0,"last_scrub":"823'833","last_scrub_stamp":"2024-09-20T22:55:07.264241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:55:07.264241+0000","objects_scrubbed":25,"log_size":726,"log_dups_size":416,"ondisk_log_size":726,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.122468067,"stat_sum":{"num_bytes":33599722,"num_objects":16,"num_object_clones":13,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":1,"num_read":305,"num_read_kb":310169,"num_write":252,"num_write_kb":90279,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":54,"num_bytes_recovered":106304666,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"13a","length":"1"},{"start":"179","length":"4"},{"start":"181","length":"4"},{"start":"186","length":"3"},{"start":"18a","length":"2"},{"start":"191","length":"2"},{"start":"194","length":"2"},{"start":"199","length":"1"},{"start":"19b","length":"1"},{"start":"19e","length":"2"},{"start":"1a1","length":"1"}]},{"pgid":"1.6","version":"77'128","reported_seq":2102,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-20T22:56:43.357462+0000","last_change":"2024-09-20T22:56:12.929922+0000","last_active":"2024-09-20T22:56:43.357462+0000","last_peered":"2024-09-20T22:56:43.357462+0000","last_clean":"2024-09-20T22:56:43.357462+0000","last_became_active":"2024-09-20T22:47:52.212190+0000","last_became_peered":"2024-09-20T22:47:52.212190+0000","last_unstale":"2024-09-20T22:56:43.357462+0000","last_undegraded":"2024-09-20T22:56:43.357462+0000","last_fullsized":"2024-09-20T22:56:43.357462+0000","mapping_epoch":391,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:56:12.929842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:56:12.929842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:29.962524+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"625'37","reported_seq":1172,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-20T22:56:43.350097+0000","last_change":"2024-09-20T22:56:08.427985+0000","last_active":"2024-09-20T22:56:43.350097+0000","last_peered":"2024-09-20T22:56:43.350097+0000","last_clean":"2024-09-20T22:56:43.350097+0000","last_became_active":"2024-09-20T22:56:08.367015+0000","last_became_peered":"2024-09-20T22:56:08.367015+0000","last_unstale":"2024-09-20T22:56:43.350097+0000","last_undegraded":"2024-09-20T22:56:43.350097+0000","last_fullsized":"2024-09-20T22:56:43.350097+0000","mapping_epoch":887,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'37","last_scrub_stamp":"2024-09-20T22:54:46.870818+0000","last_deep_scrub":"515'27","last_deep_scrub_stamp":"2024-09-20T22:49:55.964153+0000","last_clean_scrub_stamp":"2024-09-20T22:54:46.870818+0000","objects_scrubbed":8,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":53,"num_read_kb":39,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":311,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-20T22:56:43.355566+0000","last_change":"2024-09-20T22:55:30.876432+0000","last_active":"2024-09-20T22:56:43.355566+0000","last_peered":"2024-09-20T22:56:43.355566+0000","last_clean":"2024-09-20T22:56:43.355566+0000","last_became_active":"2024-09-20T22:54:17.341248+0000","last_became_peered":"2024-09-20T22:54:17.341248+0000","last_unstale":"2024-09-20T22:56:43.355566+0000","last_undegraded":"2024-09-20T22:56:43.355566+0000","last_fullsized":"2024-09-20T22:56:43.355566+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:55:30.876361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:48:23.320031+0000","last_clean_scrub_stamp":"2024-09-20T22:55:30.876361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:56:48.575021+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":1794,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-20T22:56:43.356409+0000","last_change":"2024-09-20T22:55:53.886918+0000","last_active":"2024-09-20T22:56:43.356409+0000","last_peered":"2024-09-20T22:56:43.356409+0000","last_clean":"2024-09-20T22:56:43.356409+0000","last_became_active":"2024-09-20T22:47:49.214055+0000","last_became_peered":"2024-09-20T22:47:49.214055+0000","last_unstale":"2024-09-20T22:56:43.356409+0000","last_undegraded":"2024-09-20T22:56:43.356409+0000","last_fullsized":"2024-09-20T22:56:43.356409+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:55:53.886833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:55:53.886833+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:08.991782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":311,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-20T22:56:43.355738+0000","last_change":"2024-09-20T22:55:54.151421+0000","last_active":"2024-09-20T22:56:43.355738+0000","last_peered":"2024-09-20T22:56:43.355738+0000","last_clean":"2024-09-20T22:56:43.355738+0000","last_became_active":"2024-09-20T22:54:17.340458+0000","last_became_peered":"2024-09-20T22:54:17.340458+0000","last_unstale":"2024-09-20T22:56:43.355738+0000","last_undegraded":"2024-09-20T22:56:43.355738+0000","last_fullsized":"2024-09-20T22:56:43.355738+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:55:54.151316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:48:41.208853+0000","last_clean_scrub_stamp":"2024-09-20T22:55:54.151316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:06.581440+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.7","version":"923'1964","reported_seq":6090,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-20T22:56:43.591406+0000","last_change":"2024-09-20T22:56:43.381944+0000","last_active":"2024-09-20T22:56:43.591406+0000","last_peered":"2024-09-20T22:56:43.591406+0000","last_clean":"2024-09-20T22:56:43.591406+0000","last_became_active":"2024-09-20T22:48:43.224386+0000","last_became_peered":"2024-09-20T22:48:43.224386+0000","last_unstale":"2024-09-20T22:56:43.591406+0000","last_undegraded":"2024-09-20T22:56:43.591406+0000","last_fullsized":"2024-09-20T22:56:43.591406+0000","mapping_epoch":391,"log_start":"436'456","ondisk_log_start":"436'456","created":94,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":0,"last_scrub":"879'1788","last_scrub_stamp":"2024-09-20T22:55:59.964830+0000","last_deep_scrub":"879'1788","last_deep_scrub_stamp":"2024-09-20T22:55:59.964830+0000","last_clean_scrub_stamp":"2024-09-20T22:55:59.964830+0000","objects_scrubbed":29,"log_size":1508,"log_dups_size":886,"ondisk_log_size":1508,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:12.177913+0000","scrub_duration":151,"objects_trimmed":0,"snaptrim_duration":0.024283328999999999,"stat_sum":{"num_bytes":70944709,"num_objects":30,"num_object_clones":25,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":2,"num_read":637,"num_read_kb":699389,"num_write":486,"num_write_kb":173409,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":115,"num_bytes_recovered":51615167,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"13a","length":"1"},{"start":"179","length":"4"},{"start":"181","length":"4"},{"start":"186","length":"3"},{"start":"18a","length":"3"},{"start":"191","length":"2"},{"start":"194","length":"2"},{"start":"199","length":"1"},{"start":"19b","length":"1"},{"start":"19e","length":"2"},{"start":"1a1","length":"1"}]},{"pgid":"1.1","version":"77'128","reported_seq":2082,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-20T22:56:43.355074+0000","last_change":"2024-09-20T22:55:27.327609+0000","last_active":"2024-09-20T22:56:43.355074+0000","last_peered":"2024-09-20T22:56:43.355074+0000","last_clean":"2024-09-20T22:56:43.355074+0000","last_became_active":"2024-09-20T22:47:52.208102+0000","last_became_peered":"2024-09-20T22:47:52.208102+0000","last_unstale":"2024-09-20T22:56:43.355074+0000","last_undegraded":"2024-09-20T22:56:43.355074+0000","last_fullsized":"2024-09-20T22:56:43.355074+0000","mapping_epoch":391,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:55:27.327480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:55:27.327480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:56:53.124422+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"833'53","reported_seq":243,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-20T22:56:43.355842+0000","last_change":"2024-09-20T22:56:17.472931+0000","last_active":"2024-09-20T22:56:43.355842+0000","last_peered":"2024-09-20T22:56:43.355842+0000","last_clean":"2024-09-20T22:56:43.355842+0000","last_became_active":"2024-09-20T22:55:11.346099+0000","last_became_peered":"2024-09-20T22:55:11.346099+0000","last_unstale":"2024-09-20T22:56:43.355842+0000","last_undegraded":"2024-09-20T22:56:43.355842+0000","last_fullsized":"2024-09-20T22:56:43.355842+0000","mapping_epoch":830,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":831,"parent":"0.0","parent_split_bits":0,"last_scrub":"833'53","last_scrub_stamp":"2024-09-20T22:56:17.472832+0000","last_deep_scrub":"86'38","last_deep_scrub_stamp":"2024-09-20T22:42:43.935677+0000","last_clean_scrub_stamp":"2024-09-20T22:56:17.472832+0000","objects_scrubbed":10,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:29.282777+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":1074,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":230,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-20T22:56:43.350283+0000","last_change":"2024-09-20T22:55:24.603577+0000","last_active":"2024-09-20T22:56:43.350283+0000","last_peered":"2024-09-20T22:56:43.350283+0000","last_clean":"2024-09-20T22:56:43.350283+0000","last_became_active":"2024-09-20T22:55:17.340038+0000","last_became_peered":"2024-09-20T22:55:17.340038+0000","last_unstale":"2024-09-20T22:56:43.350283+0000","last_undegraded":"2024-09-20T22:56:43.350283+0000","last_fullsized":"2024-09-20T22:56:43.350283+0000","mapping_epoch":836,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":837,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:55:24.603480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:55:24.603480+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"69'2","reported_seq":1804,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-20T22:56:43.355725+0000","last_change":"2024-09-20T22:54:23.097050+0000","last_active":"2024-09-20T22:56:43.355725+0000","last_peered":"2024-09-20T22:56:43.355725+0000","last_clean":"2024-09-20T22:56:43.355725+0000","last_became_active":"2024-09-20T22:54:17.341059+0000","last_became_peered":"2024-09-20T22:54:17.341059+0000","last_unstale":"2024-09-20T22:56:43.355725+0000","last_undegraded":"2024-09-20T22:56:43.355725+0000","last_fullsized":"2024-09-20T22:56:43.355725+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2024-09-20T22:54:23.096974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:36.130684+0000","last_clean_scrub_stamp":"2024-09-20T22:54:23.096974+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00087280599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":1133,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-20T22:56:43.349889+0000","last_change":"2024-09-20T22:56:13.481362+0000","last_active":"2024-09-20T22:56:43.349889+0000","last_peered":"2024-09-20T22:56:43.349889+0000","last_clean":"2024-09-20T22:56:43.349889+0000","last_became_active":"2024-09-20T22:54:17.339860+0000","last_became_peered":"2024-09-20T22:54:17.339860+0000","last_unstale":"2024-09-20T22:56:43.349889+0000","last_undegraded":"2024-09-20T22:56:43.349889+0000","last_fullsized":"2024-09-20T22:56:43.349889+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:56:13.481292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:56:13.481292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:25.515295+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":1772,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-20T22:56:43.352136+0000","last_change":"2024-09-20T22:56:08.388050+0000","last_active":"2024-09-20T22:56:43.352136+0000","last_peered":"2024-09-20T22:56:43.352136+0000","last_clean":"2024-09-20T22:56:43.352136+0000","last_became_active":"2024-09-20T22:56:08.387731+0000","last_became_peered":"2024-09-20T22:56:08.387731+0000","last_unstale":"2024-09-20T22:56:43.352136+0000","last_undegraded":"2024-09-20T22:56:43.352136+0000","last_fullsized":"2024-09-20T22:56:43.352136+0000","mapping_epoch":887,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:55:51.494144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:55:51.494144+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:21.386743+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"923'706","reported_seq":3339,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-20T22:56:43.381369+0000","last_change":"2024-09-20T22:56:43.350499+0000","last_active":"2024-09-20T22:56:43.381369+0000","last_peered":"2024-09-20T22:56:43.381369+0000","last_clean":"2024-09-20T22:56:43.381369+0000","last_became_active":"2024-09-20T22:54:20.412859+0000","last_became_peered":"2024-09-20T22:54:20.412859+0000","last_unstale":"2024-09-20T22:56:43.381369+0000","last_undegraded":"2024-09-20T22:56:43.381369+0000","last_fullsized":"2024-09-20T22:56:43.381369+0000","mapping_epoch":779,"log_start":"456'176","ondisk_log_start":"456'176","created":94,"last_epoch_clean":780,"parent":"0.0","parent_split_bits":0,"last_scrub":"853'620","last_scrub_stamp":"2024-09-20T22:55:34.565081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:55:34.565081+0000","objects_scrubbed":8,"log_size":530,"log_dups_size":318,"ondisk_log_size":530,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:01.357915+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.00085937100000000005,"stat_sum":{"num_bytes":23214203,"num_objects":8,"num_object_clones":6,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":218,"num_read_kb":214507,"num_write":179,"num_write_kb":60797,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":41680310,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"13a","length":"1"},{"start":"179","length":"4"},{"start":"181","length":"4"},{"start":"186","length":"3"},{"start":"18a","length":"3"},{"start":"191","length":"2"},{"start":"194","length":"2"},{"start":"199","length":"1"},{"start":"19b","length":"1"},{"start":"19e","length":"2"},{"start":"1a1","length":"1"}]},{"pgid":"1.2","version":"77'128","reported_seq":1117,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-20T22:56:43.348815+0000","last_change":"2024-09-20T22:56:12.520096+0000","last_active":"2024-09-20T22:56:43.348815+0000","last_peered":"2024-09-20T22:56:43.348815+0000","last_clean":"2024-09-20T22:56:43.348815+0000","last_became_active":"2024-09-20T22:56:11.355063+0000","last_became_peered":"2024-09-20T22:56:11.355063+0000","last_unstale":"2024-09-20T22:56:43.348815+0000","last_undegraded":"2024-09-20T22:56:43.348815+0000","last_fullsized":"2024-09-20T22:56:43.348815+0000","mapping_epoch":890,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":891,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:56:12.519992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:56:12.519992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:37.217577+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"625'22","reported_seq":326,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-20T22:56:43.356668+0000","last_change":"2024-09-20T22:55:07.954993+0000","last_active":"2024-09-20T22:56:43.356668+0000","last_peered":"2024-09-20T22:56:43.356668+0000","last_clean":"2024-09-20T22:56:43.356668+0000","last_became_active":"2024-09-20T22:54:17.349415+0000","last_became_peered":"2024-09-20T22:54:17.349415+0000","last_unstale":"2024-09-20T22:56:43.356668+0000","last_undegraded":"2024-09-20T22:56:43.356668+0000","last_fullsized":"2024-09-20T22:56:43.356668+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'22","last_scrub_stamp":"2024-09-20T22:55:07.954868+0000","last_deep_scrub":"443'18","last_deep_scrub_stamp":"2024-09-20T22:48:44.182517+0000","last_clean_scrub_stamp":"2024-09-20T22:55:07.954868+0000","objects_scrubbed":5,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":24,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":1320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":319,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-20T22:56:43.357176+0000","last_change":"2024-09-20T22:56:00.249870+0000","last_active":"2024-09-20T22:56:43.357176+0000","last_peered":"2024-09-20T22:56:43.357176+0000","last_clean":"2024-09-20T22:56:43.357176+0000","last_became_active":"2024-09-20T22:54:17.343806+0000","last_became_peered":"2024-09-20T22:54:17.343806+0000","last_unstale":"2024-09-20T22:56:43.357176+0000","last_undegraded":"2024-09-20T22:56:43.357176+0000","last_fullsized":"2024-09-20T22:56:43.357176+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:56:00.249796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:56:00.249796+0000","last_clean_scrub_stamp":"2024-09-20T22:56:00.249796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:28.326445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"380'2","reported_seq":1828,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-20T22:56:43.360956+0000","last_change":"2024-09-20T22:55:57.061420+0000","last_active":"2024-09-20T22:56:43.360956+0000","last_peered":"2024-09-20T22:56:43.360956+0000","last_clean":"2024-09-20T22:56:43.360956+0000","last_became_active":"2024-09-20T22:47:49.222812+0000","last_became_peered":"2024-09-20T22:47:49.222812+0000","last_unstale":"2024-09-20T22:56:43.360956+0000","last_undegraded":"2024-09-20T22:56:43.360956+0000","last_fullsized":"2024-09-20T22:56:43.360956+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"380'2","last_scrub_stamp":"2024-09-20T22:55:57.061312+0000","last_deep_scrub":"380'2","last_deep_scrub_stamp":"2024-09-20T22:47:40.983039+0000","last_clean_scrub_stamp":"2024-09-20T22:55:57.061312+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:01.125468+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001301459,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":1779,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-20T22:56:43.363676+0000","last_change":"2024-09-20T22:56:08.375995+0000","last_active":"2024-09-20T22:56:43.363676+0000","last_peered":"2024-09-20T22:56:43.363676+0000","last_clean":"2024-09-20T22:56:43.363676+0000","last_became_active":"2024-09-20T22:56:08.375555+0000","last_became_peered":"2024-09-20T22:56:08.375555+0000","last_unstale":"2024-09-20T22:56:43.363676+0000","last_undegraded":"2024-09-20T22:56:43.363676+0000","last_fullsized":"2024-09-20T22:56:43.363676+0000","mapping_epoch":887,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:55:08.384321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:47:22.881109+0000","last_clean_scrub_stamp":"2024-09-20T22:55:08.384321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":82,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-20T22:56:43.357834+0000","last_change":"2024-09-20T22:56:08.370358+0000","last_active":"2024-09-20T22:56:43.357834+0000","last_peered":"2024-09-20T22:56:43.357834+0000","last_clean":"2024-09-20T22:56:43.357834+0000","last_became_active":"2024-09-20T22:56:08.369859+0000","last_became_peered":"2024-09-20T22:56:08.369859+0000","last_unstale":"2024-09-20T22:56:43.357834+0000","last_undegraded":"2024-09-20T22:56:43.357834+0000","last_fullsized":"2024-09-20T22:56:43.357834+0000","mapping_epoch":887,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:55:09.670757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:55:09.670757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.5","version":"923'532","reported_seq":4755,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-20T22:56:43.372500+0000","last_change":"2024-09-20T22:56:43.372821+0000","last_active":"2024-09-20T22:56:43.372500+0000","last_peered":"2024-09-20T22:56:43.372500+0000","last_clean":"2024-09-20T22:56:43.372500+0000","last_became_active":"2024-09-20T22:56:11.353921+0000","last_became_peered":"2024-09-20T22:56:11.353921+0000","last_unstale":"2024-09-20T22:56:43.372500+0000","last_undegraded":"2024-09-20T22:56:43.372500+0000","last_fullsized":"2024-09-20T22:56:43.372500+0000","mapping_epoch":890,"log_start":"450'190","ondisk_log_start":"450'190","created":94,"last_epoch_clean":891,"parent":"0.0","parent_split_bits":0,"last_scrub":"902'512","last_scrub_stamp":"2024-09-20T22:56:23.005986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:56:23.005986+0000","objects_scrubbed":8,"log_size":342,"log_dups_size":240,"ondisk_log_size":342,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:26.535375+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.012823678,"stat_sum":{"num_bytes":11869743,"num_objects":7,"num_object_clones":6,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":131,"num_read_kb":132438,"num_write":114,"num_write_kb":46391,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":41118840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"13a","length":"1"},{"start":"179","length":"4"},{"start":"181","length":"4"},{"start":"186","length":"3"},{"start":"18a","length":"3"},{"start":"191","length":"2"},{"start":"194","length":"2"},{"start":"199","length":"1"},{"start":"19b","length":"1"},{"start":"19e","length":"2"},{"start":"1a1","length":"1"}]},{"pgid":"1.3","version":"77'128","reported_seq":1338,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-20T22:56:43.358673+0000","last_change":"2024-09-20T22:56:21.532248+0000","last_active":"2024-09-20T22:56:43.358673+0000","last_peered":"2024-09-20T22:56:43.358673+0000","last_clean":"2024-09-20T22:56:43.358673+0000","last_became_active":"2024-09-20T22:54:20.413191+0000","last_became_peered":"2024-09-20T22:54:20.413191+0000","last_unstale":"2024-09-20T22:56:43.358673+0000","last_undegraded":"2024-09-20T22:56:43.358673+0000","last_fullsized":"2024-09-20T22:56:43.358673+0000","mapping_epoch":779,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":780,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:56:21.532173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:56:21.532173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:45.248528+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"828'30","reported_seq":1205,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-20T22:56:43.352115+0000","last_change":"2024-09-20T22:55:11.360221+0000","last_active":"2024-09-20T22:56:43.352115+0000","last_peered":"2024-09-20T22:56:43.352115+0000","last_clean":"2024-09-20T22:56:43.352115+0000","last_became_active":"2024-09-20T22:55:11.346018+0000","last_became_peered":"2024-09-20T22:55:11.346018+0000","last_unstale":"2024-09-20T22:56:43.352115+0000","last_undegraded":"2024-09-20T22:56:43.352115+0000","last_fullsized":"2024-09-20T22:56:43.352115+0000","mapping_epoch":830,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":831,"parent":"0.0","parent_split_bits":0,"last_scrub":"828'30","last_scrub_stamp":"2024-09-20T22:55:08.705571+0000","last_deep_scrub":"828'30","last_deep_scrub_stamp":"2024-09-20T22:55:08.705571+0000","last_clean_scrub_stamp":"2024-09-20T22:55:08.705571+0000","objects_scrubbed":6,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":1121,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-20T22:56:43.349961+0000","last_change":"2024-09-20T22:55:25.640612+0000","last_active":"2024-09-20T22:56:43.349961+0000","last_peered":"2024-09-20T22:56:43.349961+0000","last_clean":"2024-09-20T22:56:43.349961+0000","last_became_active":"2024-09-20T22:48:05.233093+0000","last_became_peered":"2024-09-20T22:48:05.233093+0000","last_unstale":"2024-09-20T22:56:43.349961+0000","last_undegraded":"2024-09-20T22:56:43.349961+0000","last_fullsized":"2024-09-20T22:56:43.349961+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:55:25.640520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:44:19.834641+0000","last_clean_scrub_stamp":"2024-09-20T22:55:25.640520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"75'5","reported_seq":2389,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-20T22:56:43.356965+0000","last_change":"2024-09-20T22:56:12.471380+0000","last_active":"2024-09-20T22:56:43.356965+0000","last_peered":"2024-09-20T22:56:43.356965+0000","last_clean":"2024-09-20T22:56:43.356965+0000","last_became_active":"2024-09-20T22:56:08.391483+0000","last_became_peered":"2024-09-20T22:56:08.391483+0000","last_unstale":"2024-09-20T22:56:43.356965+0000","last_undegraded":"2024-09-20T22:56:43.356965+0000","last_fullsized":"2024-09-20T22:56:43.356965+0000","mapping_epoch":887,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-09-20T22:56:12.471215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:36.130684+0000","last_clean_scrub_stamp":"2024-09-20T22:56:12.471215+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:23.044148+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00078631700000000003,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":783,"num_read_kb":778,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1556,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":1789,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-20T22:56:43.357884+0000","last_change":"2024-09-20T22:54:51.175204+0000","last_active":"2024-09-20T22:56:43.357884+0000","last_peered":"2024-09-20T22:56:43.357884+0000","last_clean":"2024-09-20T22:56:43.357884+0000","last_became_active":"2024-09-20T22:54:17.339805+0000","last_became_peered":"2024-09-20T22:54:17.339805+0000","last_unstale":"2024-09-20T22:56:43.357884+0000","last_undegraded":"2024-09-20T22:56:43.357884+0000","last_fullsized":"2024-09-20T22:56:43.357884+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:54:51.175120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:54:51.175120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":311,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-20T22:56:43.355467+0000","last_change":"2024-09-20T22:55:11.066946+0000","last_active":"2024-09-20T22:56:43.355467+0000","last_peered":"2024-09-20T22:56:43.355467+0000","last_clean":"2024-09-20T22:56:43.355467+0000","last_became_active":"2024-09-20T22:54:17.339564+0000","last_became_peered":"2024-09-20T22:54:17.339564+0000","last_unstale":"2024-09-20T22:56:43.355467+0000","last_undegraded":"2024-09-20T22:56:43.355467+0000","last_fullsized":"2024-09-20T22:56:43.355467+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:55:11.066868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:55:11.066868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.6","version":"923'843","reported_seq":4973,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-20T22:56:43.358890+0000","last_change":"2024-09-20T22:56:43.360850+0000","last_active":"2024-09-20T22:56:43.358890+0000","last_peered":"2024-09-20T22:56:43.358890+0000","last_clean":"2024-09-20T22:56:43.358890+0000","last_became_active":"2024-09-20T22:56:11.366143+0000","last_became_peered":"2024-09-20T22:56:11.366143+0000","last_unstale":"2024-09-20T22:56:43.358890+0000","last_undegraded":"2024-09-20T22:56:43.358890+0000","last_fullsized":"2024-09-20T22:56:43.358890+0000","mapping_epoch":890,"log_start":"443'181","ondisk_log_start":"443'181","created":94,"last_epoch_clean":891,"parent":"0.0","parent_split_bits":0,"last_scrub":"902'811","last_scrub_stamp":"2024-09-20T22:56:22.658684+0000","last_deep_scrub":"902'811","last_deep_scrub_stamp":"2024-09-20T22:56:22.658684+0000","last_clean_scrub_stamp":"2024-09-20T22:56:22.658684+0000","objects_scrubbed":21,"log_size":662,"log_dups_size":314,"ondisk_log_size":662,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:35.425884+0000","scrub_duration":109,"objects_trimmed":0,"snaptrim_duration":0.0082586469999999992,"stat_sum":{"num_bytes":41621607,"num_objects":19,"num_object_clones":17,"num_object_copies":57,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":1,"num_read":235,"num_read_kb":234085,"num_write":201,"num_write_kb":81257,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":65,"num_bytes_recovered":68865510,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"13a","length":"1"},{"start":"179","length":"4"},{"start":"181","length":"4"},{"start":"186","length":"3"},{"start":"18a","length":"3"},{"start":"191","length":"2"},{"start":"194","length":"2"},{"start":"199","length":"1"},{"start":"19b","length":"1"},{"start":"19e","length":"2"},{"start":"1a1","length":"1"}]},{"pgid":"4.5","version":"888'33","reported_seq":2128,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-20T22:56:43.358509+0000","last_change":"2024-09-20T22:56:21.986738+0000","last_active":"2024-09-20T22:56:43.358509+0000","last_peered":"2024-09-20T22:56:43.358509+0000","last_clean":"2024-09-20T22:56:43.358509+0000","last_became_active":"2024-09-20T22:56:08.358532+0000","last_became_peered":"2024-09-20T22:56:08.358532+0000","last_unstale":"2024-09-20T22:56:43.358509+0000","last_undegraded":"2024-09-20T22:56:43.358509+0000","last_fullsized":"2024-09-20T22:56:43.358509+0000","mapping_epoch":887,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":0,"last_scrub":"888'33","last_scrub_stamp":"2024-09-20T22:56:21.986633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:56:21.986633+0000","objects_scrubbed":8,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:28.060278+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":770,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"67'1","reported_seq":1765,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-20T22:56:43.351349+0000","last_change":"2024-09-20T22:55:33.544290+0000","last_active":"2024-09-20T22:56:43.351349+0000","last_peered":"2024-09-20T22:56:43.351349+0000","last_clean":"2024-09-20T22:56:43.351349+0000","last_became_active":"2024-09-20T22:41:38.158917+0000","last_became_peered":"2024-09-20T22:41:38.158917+0000","last_unstale":"2024-09-20T22:56:43.351349+0000","last_undegraded":"2024-09-20T22:56:43.351349+0000","last_fullsized":"2024-09-20T22:56:43.351349+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-09-20T22:55:33.544218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:55:33.544218+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:00.545241+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"77'128","reported_seq":1108,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-20T22:56:43.348719+0000","last_change":"2024-09-20T22:56:41.368771+0000","last_active":"2024-09-20T22:56:43.348719+0000","last_peered":"2024-09-20T22:56:43.348719+0000","last_clean":"2024-09-20T22:56:43.348719+0000","last_became_active":"2024-09-20T22:56:41.368240+0000","last_became_peered":"2024-09-20T22:56:41.368240+0000","last_unstale":"2024-09-20T22:56:43.348719+0000","last_undegraded":"2024-09-20T22:56:43.348719+0000","last_fullsized":"2024-09-20T22:56:43.348719+0000","mapping_epoch":920,"log_start":"77'128","ondisk_log_start":"77'128","created":19,"last_epoch_clean":921,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:54:21.616231+0000","last_deep_scrub":"77'128","last_deep_scrub_stamp":"2024-09-20T22:47:22.417099+0000","last_clean_scrub_stamp":"2024-09-20T22:54:21.616231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":83,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-20T22:56:43.370313+0000","last_change":"2024-09-20T22:56:08.366982+0000","last_active":"2024-09-20T22:56:43.370313+0000","last_peered":"2024-09-20T22:56:43.370313+0000","last_clean":"2024-09-20T22:56:43.370313+0000","last_became_active":"2024-09-20T22:56:08.366671+0000","last_became_peered":"2024-09-20T22:56:08.366671+0000","last_unstale":"2024-09-20T22:56:43.370313+0000","last_undegraded":"2024-09-20T22:56:43.370313+0000","last_fullsized":"2024-09-20T22:56:43.370313+0000","mapping_epoch":887,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:55:10.707805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:55:10.707805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":1773,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-20T22:56:43.357585+0000","last_change":"2024-09-20T22:55:50.483137+0000","last_active":"2024-09-20T22:56:43.357585+0000","last_peered":"2024-09-20T22:56:43.357585+0000","last_clean":"2024-09-20T22:56:43.357585+0000","last_became_active":"2024-09-20T22:48:05.241602+0000","last_became_peered":"2024-09-20T22:48:05.241602+0000","last_unstale":"2024-09-20T22:56:43.357585+0000","last_undegraded":"2024-09-20T22:56:43.357585+0000","last_fullsized":"2024-09-20T22:56:43.357585+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:55:50.483056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:55:50.483056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:56:56.109615+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"923'854","reported_seq":5154,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-20T22:56:43.383662+0000","last_change":"2024-09-20T22:56:43.383892+0000","last_active":"2024-09-20T22:56:43.383662+0000","last_peered":"2024-09-20T22:56:43.383662+0000","last_clean":"2024-09-20T22:56:43.383662+0000","last_became_active":"2024-09-20T22:54:20.414528+0000","last_became_peered":"2024-09-20T22:54:20.414528+0000","last_unstale":"2024-09-20T22:56:43.383662+0000","last_undegraded":"2024-09-20T22:56:43.383662+0000","last_fullsized":"2024-09-20T22:56:43.383662+0000","mapping_epoch":779,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":780,"parent":"0.0","parent_split_bits":0,"last_scrub":"877'785","last_scrub_stamp":"2024-09-20T22:55:58.781372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:55:58.781372+0000","objects_scrubbed":14,"log_size":854,"log_dups_size":0,"ondisk_log_size":854,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:07.781840+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.025937233000000001,"stat_sum":{"num_bytes":27535852,"num_objects":13,"num_object_clones":11,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":1,"num_read":269,"num_read_kb":277000,"num_write":136,"num_write_kb":61654,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":88,"num_bytes_recovered":51113242,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"13a","length":"1"},{"start":"179","length":"4"},{"start":"181","length":"4"},{"start":"186","length":"3"},{"start":"18a","length":"3"},{"start":"191","length":"2"},{"start":"194","length":"2"},{"start":"199","length":"1"},{"start":"19b","length":"1"},{"start":"19e","length":"2"},{"start":"1a1","length":"1"}]},{"pgid":"4.e","version":"850'24","reported_seq":696,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-20T22:56:43.357472+0000","last_change":"2024-09-20T22:55:30.779124+0000","last_active":"2024-09-20T22:56:43.357472+0000","last_peered":"2024-09-20T22:56:43.357472+0000","last_clean":"2024-09-20T22:56:43.357472+0000","last_became_active":"2024-09-20T22:55:17.342390+0000","last_became_peered":"2024-09-20T22:55:17.342390+0000","last_unstale":"2024-09-20T22:56:43.357472+0000","last_undegraded":"2024-09-20T22:56:43.357472+0000","last_fullsized":"2024-09-20T22:56:43.357472+0000","mapping_epoch":836,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":837,"parent":"0.0","parent_split_bits":0,"last_scrub":"850'24","last_scrub_stamp":"2024-09-20T22:55:30.779018+0000","last_deep_scrub":"850'24","last_deep_scrub_stamp":"2024-09-20T22:55:30.779018+0000","last_clean_scrub_stamp":"2024-09-20T22:55:30.779018+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:56:58.207882+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":23,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":716,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-20T22:56:43.360803+0000","last_change":"2024-09-20T22:56:17.135450+0000","last_active":"2024-09-20T22:56:43.360803+0000","last_peered":"2024-09-20T22:56:43.360803+0000","last_clean":"2024-09-20T22:56:43.360803+0000","last_became_active":"2024-09-20T22:56:08.359647+0000","last_became_peered":"2024-09-20T22:56:08.359647+0000","last_unstale":"2024-09-20T22:56:43.360803+0000","last_undegraded":"2024-09-20T22:56:43.360803+0000","last_fullsized":"2024-09-20T22:56:43.360803+0000","mapping_epoch":887,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:56:17.135379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:56:17.135379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:23.999325+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":1769,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-20T22:56:43.357454+0000","last_change":"2024-09-20T22:54:47.166950+0000","last_active":"2024-09-20T22:56:43.357454+0000","last_peered":"2024-09-20T22:56:43.357454+0000","last_clean":"2024-09-20T22:56:43.357454+0000","last_became_active":"2024-09-20T22:48:05.233778+0000","last_became_peered":"2024-09-20T22:48:05.233778+0000","last_unstale":"2024-09-20T22:56:43.357454+0000","last_undegraded":"2024-09-20T22:56:43.357454+0000","last_fullsized":"2024-09-20T22:56:43.357454+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:54:47.166836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:54:47.166836+0000","last_clean_scrub_stamp":"2024-09-20T22:54:47.166836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":1746,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-20T22:56:43.352879+0000","last_change":"2024-09-20T22:53:10.647075+0000","last_active":"2024-09-20T22:56:43.352879+0000","last_peered":"2024-09-20T22:56:43.352879+0000","last_clean":"2024-09-20T22:56:43.352879+0000","last_became_active":"2024-09-20T22:41:44.832009+0000","last_became_peered":"2024-09-20T22:41:44.832009+0000","last_unstale":"2024-09-20T22:56:43.352879+0000","last_undegraded":"2024-09-20T22:56:43.352879+0000","last_fullsized":"2024-09-20T22:56:43.352879+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:10.647005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:48:40.127123+0000","last_clean_scrub_stamp":"2024-09-20T22:53:10.647005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.e","version":"923'1002","reported_seq":5146,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-20T22:56:43.376990+0000","last_change":"2024-09-20T22:56:43.377524+0000","last_active":"2024-09-20T22:56:43.376990+0000","last_peered":"2024-09-20T22:56:43.376990+0000","last_clean":"2024-09-20T22:56:43.376990+0000","last_became_active":"2024-09-20T22:47:52.209452+0000","last_became_peered":"2024-09-20T22:47:52.209452+0000","last_unstale":"2024-09-20T22:56:43.376990+0000","last_undegraded":"2024-09-20T22:56:43.376990+0000","last_fullsized":"2024-09-20T22:56:43.376990+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"854'897","last_scrub_stamp":"2024-09-20T22:55:35.182283+0000","last_deep_scrub":"174'60","last_deep_scrub_stamp":"2024-09-20T22:44:18.301776+0000","last_clean_scrub_stamp":"2024-09-20T22:55:35.182283+0000","objects_scrubbed":15,"log_size":1002,"log_dups_size":0,"ondisk_log_size":1002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.014221787,"stat_sum":{"num_bytes":30307240,"num_objects":13,"num_object_clones":11,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":1,"num_read":291,"num_read_kb":283682,"num_write":217,"num_write_kb":80655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":23110741,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"13a","length":"1"},{"start":"179","length":"4"},{"start":"181","length":"4"},{"start":"186","length":"3"},{"start":"18a","length":"3"},{"start":"191","length":"2"},{"start":"194","length":"2"},{"start":"199","length":"1"},{"start":"19b","length":"1"},{"start":"19e","length":"2"},{"start":"1a1","length":"1"}]},{"pgid":"1.8","version":"77'128","reported_seq":2034,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-20T22:56:43.354954+0000","last_change":"2024-09-20T22:54:46.128026+0000","last_active":"2024-09-20T22:56:43.354954+0000","last_peered":"2024-09-20T22:56:43.354954+0000","last_clean":"2024-09-20T22:56:43.354954+0000","last_became_active":"2024-09-20T22:42:52.405594+0000","last_became_peered":"2024-09-20T22:42:52.405594+0000","last_unstale":"2024-09-20T22:56:43.354954+0000","last_undegraded":"2024-09-20T22:56:43.354954+0000","last_fullsized":"2024-09-20T22:56:43.354954+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:54:46.127943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:54:46.127943+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"625'31","reported_seq":721,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-20T22:56:43.366126+0000","last_change":"2024-09-20T22:55:26.193125+0000","last_active":"2024-09-20T22:56:43.366126+0000","last_peered":"2024-09-20T22:56:43.366126+0000","last_clean":"2024-09-20T22:56:43.366126+0000","last_became_active":"2024-09-20T22:55:17.342262+0000","last_became_peered":"2024-09-20T22:55:17.342262+0000","last_unstale":"2024-09-20T22:56:43.366126+0000","last_undegraded":"2024-09-20T22:56:43.366126+0000","last_fullsized":"2024-09-20T22:56:43.366126+0000","mapping_epoch":836,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":837,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'31","last_scrub_stamp":"2024-09-20T22:55:26.193034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:55:26.193034+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":48,"num_read_kb":38,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":1334,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-20T22:56:43.361860+0000","last_change":"2024-09-20T22:54:51.323936+0000","last_active":"2024-09-20T22:56:43.361860+0000","last_peered":"2024-09-20T22:56:43.361860+0000","last_clean":"2024-09-20T22:56:43.361860+0000","last_became_active":"2024-09-20T22:54:17.332636+0000","last_became_peered":"2024-09-20T22:54:17.332636+0000","last_unstale":"2024-09-20T22:56:43.361860+0000","last_undegraded":"2024-09-20T22:56:43.361860+0000","last_fullsized":"2024-09-20T22:56:43.361860+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:54:51.323855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:35.492562+0000","last_clean_scrub_stamp":"2024-09-20T22:54:51.323855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":311,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-20T22:56:43.355526+0000","last_change":"2024-09-20T22:55:08.995095+0000","last_active":"2024-09-20T22:56:43.355526+0000","last_peered":"2024-09-20T22:56:43.355526+0000","last_clean":"2024-09-20T22:56:43.355526+0000","last_became_active":"2024-09-20T22:54:17.349144+0000","last_became_peered":"2024-09-20T22:54:17.349144+0000","last_unstale":"2024-09-20T22:56:43.355526+0000","last_undegraded":"2024-09-20T22:56:43.355526+0000","last_fullsized":"2024-09-20T22:56:43.355526+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:55:08.995021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:55:08.995021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":1797,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-20T22:56:43.364331+0000","last_change":"2024-09-20T22:56:43.023293+0000","last_active":"2024-09-20T22:56:43.364331+0000","last_peered":"2024-09-20T22:56:43.364331+0000","last_clean":"2024-09-20T22:56:43.364331+0000","last_became_active":"2024-09-20T22:48:05.239006+0000","last_became_peered":"2024-09-20T22:48:05.239006+0000","last_unstale":"2024-09-20T22:56:43.364331+0000","last_undegraded":"2024-09-20T22:56:43.364331+0000","last_fullsized":"2024-09-20T22:56:43.364331+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:56:43.023211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:56:43.023211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:08.405958+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"923'1110","reported_seq":5376,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-20T22:56:43.382983+0000","last_change":"2024-09-20T22:56:43.383439+0000","last_active":"2024-09-20T22:56:43.382983+0000","last_peered":"2024-09-20T22:56:43.382983+0000","last_clean":"2024-09-20T22:56:43.382983+0000","last_became_active":"2024-09-20T22:56:11.350506+0000","last_became_peered":"2024-09-20T22:56:11.350506+0000","last_unstale":"2024-09-20T22:56:43.382983+0000","last_undegraded":"2024-09-20T22:56:43.382983+0000","last_fullsized":"2024-09-20T22:56:43.382983+0000","mapping_epoch":890,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":891,"parent":"0.0","parent_split_bits":0,"last_scrub":"868'991","last_scrub_stamp":"2024-09-20T22:55:49.469629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:55:49.469629+0000","objects_scrubbed":15,"log_size":1110,"log_dups_size":0,"ondisk_log_size":1110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:56:57.743919+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0.032210821000000001,"stat_sum":{"num_bytes":35004973,"num_objects":16,"num_object_clones":13,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":427,"num_read_kb":429037,"num_write":252,"num_write_kb":88414,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":45612418,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"13a","length":"1"},{"start":"179","length":"4"},{"start":"181","length":"4"},{"start":"186","length":"3"},{"start":"18a","length":"3"},{"start":"191","length":"2"},{"start":"194","length":"2"},{"start":"199","length":"1"},{"start":"19b","length":"1"},{"start":"19e","length":"2"},{"start":"1a1","length":"1"}]},{"pgid":"1.9","version":"77'128","reported_seq":417,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-20T22:56:43.350033+0000","last_change":"2024-09-20T22:56:11.417186+0000","last_active":"2024-09-20T22:56:43.350033+0000","last_peered":"2024-09-20T22:56:43.350033+0000","last_clean":"2024-09-20T22:56:43.350033+0000","last_became_active":"2024-09-20T22:56:11.349599+0000","last_became_peered":"2024-09-20T22:56:11.349599+0000","last_unstale":"2024-09-20T22:56:43.350033+0000","last_undegraded":"2024-09-20T22:56:43.350033+0000","last_fullsized":"2024-09-20T22:56:43.350033+0000","mapping_epoch":890,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":891,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:56:11.417116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:56:11.417116+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:19.403923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"877'25","reported_seq":682,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-20T22:56:43.352398+0000","last_change":"2024-09-20T22:55:57.521644+0000","last_active":"2024-09-20T22:56:43.352398+0000","last_peered":"2024-09-20T22:56:43.352398+0000","last_clean":"2024-09-20T22:56:43.352398+0000","last_became_active":"2024-09-20T22:55:17.349462+0000","last_became_peered":"2024-09-20T22:55:17.349462+0000","last_unstale":"2024-09-20T22:56:43.352398+0000","last_undegraded":"2024-09-20T22:56:43.352398+0000","last_fullsized":"2024-09-20T22:56:43.352398+0000","mapping_epoch":836,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":837,"parent":"0.0","parent_split_bits":0,"last_scrub":"877'25","last_scrub_stamp":"2024-09-20T22:55:57.521557+0000","last_deep_scrub":"877'25","last_deep_scrub_stamp":"2024-09-20T22:55:57.521557+0000","last_clean_scrub_stamp":"2024-09-20T22:55:57.521557+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:00.258440+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":28,"num_read_kb":23,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":1790,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-20T22:56:43.353247+0000","last_change":"2024-09-20T22:55:58.553731+0000","last_active":"2024-09-20T22:56:43.353247+0000","last_peered":"2024-09-20T22:56:43.353247+0000","last_clean":"2024-09-20T22:56:43.353247+0000","last_became_active":"2024-09-20T22:48:05.233999+0000","last_became_peered":"2024-09-20T22:48:05.233999+0000","last_unstale":"2024-09-20T22:56:43.353247+0000","last_undegraded":"2024-09-20T22:56:43.353247+0000","last_fullsized":"2024-09-20T22:56:43.353247+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:55:58.553666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:55:58.553666+0000","last_clean_scrub_stamp":"2024-09-20T22:55:58.553666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:15.816815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"888'10","reported_seq":1982,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-20T22:56:43.366006+0000","last_change":"2024-09-20T22:56:08.398936+0000","last_active":"2024-09-20T22:56:43.366006+0000","last_peered":"2024-09-20T22:56:43.366006+0000","last_clean":"2024-09-20T22:56:43.366006+0000","last_became_active":"2024-09-20T22:56:08.374292+0000","last_became_peered":"2024-09-20T22:56:08.374292+0000","last_unstale":"2024-09-20T22:56:43.366006+0000","last_undegraded":"2024-09-20T22:56:43.366006+0000","last_fullsized":"2024-09-20T22:56:43.366006+0000","mapping_epoch":887,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'9","last_scrub_stamp":"2024-09-20T22:55:36.178528+0000","last_deep_scrub":"261'9","last_deep_scrub_stamp":"2024-09-20T22:46:50.974735+0000","last_clean_scrub_stamp":"2024-09-20T22:55:36.178528+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:00.656863+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":325,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-20T22:56:43.357190+0000","last_change":"2024-09-20T22:56:23.542870+0000","last_active":"2024-09-20T22:56:43.357190+0000","last_peered":"2024-09-20T22:56:43.357190+0000","last_clean":"2024-09-20T22:56:43.357190+0000","last_became_active":"2024-09-20T22:56:01.477943+0000","last_became_peered":"2024-09-20T22:56:01.477943+0000","last_unstale":"2024-09-20T22:56:43.357190+0000","last_undegraded":"2024-09-20T22:56:43.357190+0000","last_fullsized":"2024-09-20T22:56:43.357190+0000","mapping_epoch":880,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":881,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:56:23.542803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:56:23.542803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:45.653479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.8","version":"923'1393","reported_seq":5388,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-20T22:56:43.382584+0000","last_change":"2024-09-20T22:56:43.382911+0000","last_active":"2024-09-20T22:56:43.382584+0000","last_peered":"2024-09-20T22:56:43.382584+0000","last_clean":"2024-09-20T22:56:43.382584+0000","last_became_active":"2024-09-20T22:56:11.351077+0000","last_became_peered":"2024-09-20T22:56:11.351077+0000","last_unstale":"2024-09-20T22:56:43.382584+0000","last_undegraded":"2024-09-20T22:56:43.382584+0000","last_fullsized":"2024-09-20T22:56:43.382584+0000","mapping_epoch":890,"log_start":"426'268","ondisk_log_start":"426'268","created":94,"last_epoch_clean":891,"parent":"0.0","parent_split_bits":0,"last_scrub":"851'1165","last_scrub_stamp":"2024-09-20T22:55:32.164332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:55:32.164332+0000","objects_scrubbed":19,"log_size":1125,"log_dups_size":445,"ondisk_log_size":1125,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.022154024000000001,"stat_sum":{"num_bytes":46398147,"num_objects":24,"num_object_clones":21,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":393,"num_read_kb":416013,"num_write":316,"num_write_kb":106656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":90,"num_bytes_recovered":69393013,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"13a","length":"1"},{"start":"179","length":"4"},{"start":"181","length":"4"},{"start":"186","length":"3"},{"start":"18a","length":"3"},{"start":"191","length":"2"},{"start":"194","length":"2"},{"start":"199","length":"1"},{"start":"19b","length":"1"},{"start":"19e","length":"2"},{"start":"1a1","length":"1"}]},{"pgid":"4.b","version":"625'15","reported_seq":1181,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-20T22:56:43.350192+0000","last_change":"2024-09-20T22:55:32.601025+0000","last_active":"2024-09-20T22:56:43.350192+0000","last_peered":"2024-09-20T22:56:43.350192+0000","last_clean":"2024-09-20T22:56:43.350192+0000","last_became_active":"2024-09-20T22:54:17.340998+0000","last_became_peered":"2024-09-20T22:54:17.340998+0000","last_unstale":"2024-09-20T22:56:43.350192+0000","last_undegraded":"2024-09-20T22:56:43.350192+0000","last_fullsized":"2024-09-20T22:56:43.350192+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'15","last_scrub_stamp":"2024-09-20T22:55:32.600891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:55:32.600891+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:56:47.449824+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":1773,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-20T22:56:43.356228+0000","last_change":"2024-09-20T22:56:24.518304+0000","last_active":"2024-09-20T22:56:43.356228+0000","last_peered":"2024-09-20T22:56:43.356228+0000","last_clean":"2024-09-20T22:56:43.356228+0000","last_became_active":"2024-09-20T22:41:38.155953+0000","last_became_peered":"2024-09-20T22:41:38.155953+0000","last_unstale":"2024-09-20T22:56:43.356228+0000","last_undegraded":"2024-09-20T22:56:43.356228+0000","last_fullsized":"2024-09-20T22:56:43.356228+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:56:24.518228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:56:24.518228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:38.822179+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":1797,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-20T22:56:43.362062+0000","last_change":"2024-09-20T22:54:47.368607+0000","last_active":"2024-09-20T22:56:43.362062+0000","last_peered":"2024-09-20T22:56:43.362062+0000","last_clean":"2024-09-20T22:56:43.362062+0000","last_became_active":"2024-09-20T22:48:05.240613+0000","last_became_peered":"2024-09-20T22:48:05.240613+0000","last_unstale":"2024-09-20T22:56:43.362062+0000","last_undegraded":"2024-09-20T22:56:43.362062+0000","last_fullsized":"2024-09-20T22:56:43.362062+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:54:47.368529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:54:47.368529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":1113,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-20T22:56:43.350119+0000","last_change":"2024-09-20T22:56:08.376503+0000","last_active":"2024-09-20T22:56:43.350119+0000","last_peered":"2024-09-20T22:56:43.350119+0000","last_clean":"2024-09-20T22:56:43.350119+0000","last_became_active":"2024-09-20T22:56:08.376213+0000","last_became_peered":"2024-09-20T22:56:08.376213+0000","last_unstale":"2024-09-20T22:56:43.350119+0000","last_undegraded":"2024-09-20T22:56:43.350119+0000","last_fullsized":"2024-09-20T22:56:43.350119+0000","mapping_epoch":887,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:54:38.856726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:54:38.856726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"923'1827","reported_seq":6041,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-20T22:56:43.589776+0000","last_change":"2024-09-20T22:56:43.369828+0000","last_active":"2024-09-20T22:56:43.589776+0000","last_peered":"2024-09-20T22:56:43.589776+0000","last_clean":"2024-09-20T22:56:43.589776+0000","last_became_active":"2024-09-20T22:54:20.414945+0000","last_became_peered":"2024-09-20T22:54:20.414945+0000","last_unstale":"2024-09-20T22:56:43.589776+0000","last_undegraded":"2024-09-20T22:56:43.589776+0000","last_fullsized":"2024-09-20T22:56:43.589776+0000","mapping_epoch":779,"log_start":"423'280","ondisk_log_start":"423'280","created":94,"last_epoch_clean":780,"parent":"0.0","parent_split_bits":0,"last_scrub":"890'1700","last_scrub_stamp":"2024-09-20T22:56:10.509714+0000","last_deep_scrub":"590'828","last_deep_scrub_stamp":"2024-09-20T22:51:10.909364+0000","last_clean_scrub_stamp":"2024-09-20T22:56:10.509714+0000","objects_scrubbed":33,"log_size":1547,"log_dups_size":649,"ondisk_log_size":1547,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:38.810494+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0.019403289000000001,"stat_sum":{"num_bytes":83295037,"num_objects":34,"num_object_clones":29,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":1,"num_read":622,"num_read_kb":640821,"num_write":456,"num_write_kb":171730,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":109,"num_bytes_recovered":133262200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"13a","length":"1"},{"start":"179","length":"4"},{"start":"181","length":"4"},{"start":"186","length":"3"},{"start":"18a","length":"3"},{"start":"191","length":"2"},{"start":"194","length":"2"},{"start":"199","length":"1"},{"start":"19b","length":"1"},{"start":"19e","length":"2"},{"start":"1a1","length":"1"}]},{"pgid":"4.a","version":"625'44","reported_seq":1927,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-20T22:56:43.356503+0000","last_change":"2024-09-20T22:56:08.381130+0000","last_active":"2024-09-20T22:56:43.356503+0000","last_peered":"2024-09-20T22:56:43.356503+0000","last_clean":"2024-09-20T22:56:43.356503+0000","last_became_active":"2024-09-20T22:56:08.348928+0000","last_became_peered":"2024-09-20T22:56:08.348928+0000","last_unstale":"2024-09-20T22:56:43.356503+0000","last_undegraded":"2024-09-20T22:56:43.356503+0000","last_fullsized":"2024-09-20T22:56:43.356503+0000","mapping_epoch":887,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'44","last_scrub_stamp":"2024-09-20T22:55:08.851376+0000","last_deep_scrub":"113'28","last_deep_scrub_stamp":"2024-09-20T22:43:12.725900+0000","last_clean_scrub_stamp":"2024-09-20T22:55:08.851376+0000","objects_scrubbed":9,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":81,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-20T22:56:43.356564+0000","last_change":"2024-09-20T22:56:08.367535+0000","last_active":"2024-09-20T22:56:43.356564+0000","last_peered":"2024-09-20T22:56:43.356564+0000","last_clean":"2024-09-20T22:56:43.356564+0000","last_became_active":"2024-09-20T22:56:08.367213+0000","last_became_peered":"2024-09-20T22:56:08.367213+0000","last_unstale":"2024-09-20T22:56:43.356564+0000","last_undegraded":"2024-09-20T22:56:43.356564+0000","last_fullsized":"2024-09-20T22:56:43.356564+0000","mapping_epoch":887,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:55:29.772589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:55:29.772589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:56:46.611166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":1793,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-20T22:56:43.361397+0000","last_change":"2024-09-20T22:55:56.107908+0000","last_active":"2024-09-20T22:56:43.361397+0000","last_peered":"2024-09-20T22:56:43.361397+0000","last_clean":"2024-09-20T22:56:43.361397+0000","last_became_active":"2024-09-20T22:47:49.223220+0000","last_became_peered":"2024-09-20T22:47:49.223220+0000","last_unstale":"2024-09-20T22:56:43.361397+0000","last_undegraded":"2024-09-20T22:56:43.361397+0000","last_fullsized":"2024-09-20T22:56:43.361397+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:55:56.107827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:54:21.496068+0000","last_clean_scrub_stamp":"2024-09-20T22:55:56.107827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:04.283042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":82,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-20T22:56:43.361135+0000","last_change":"2024-09-20T22:56:08.370383+0000","last_active":"2024-09-20T22:56:43.361135+0000","last_peered":"2024-09-20T22:56:43.361135+0000","last_clean":"2024-09-20T22:56:43.361135+0000","last_became_active":"2024-09-20T22:56:08.370039+0000","last_became_peered":"2024-09-20T22:56:08.370039+0000","last_unstale":"2024-09-20T22:56:43.361135+0000","last_undegraded":"2024-09-20T22:56:43.361135+0000","last_fullsized":"2024-09-20T22:56:43.361135+0000","mapping_epoch":887,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:55:54.870627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:55:54.870627+0000","last_clean_scrub_stamp":"2024-09-20T22:55:54.870627+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:10.583725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.a","version":"923'1175","reported_seq":5170,"reported_epoch":923,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T22:56:43.614720+0000","last_change":"2024-09-20T22:56:43.614720+0000","last_active":"2024-09-20T22:56:43.614720+0000","last_peered":"2024-09-20T22:56:43.614720+0000","last_clean":"2024-09-20T22:56:43.614720+0000","last_became_active":"2024-09-20T22:42:52.405085+0000","last_became_peered":"2024-09-20T22:42:52.405085+0000","last_unstale":"2024-09-20T22:56:43.614720+0000","last_undegraded":"2024-09-20T22:56:43.614720+0000","last_fullsized":"2024-09-20T22:56:43.614720+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"851'1076","last_scrub_stamp":"2024-09-20T22:55:32.567038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:55:32.567038+0000","objects_scrubbed":21,"log_size":1175,"log_dups_size":0,"ondisk_log_size":1175,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":19,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:56:54.918789+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0.26311931300000002,"stat_sum":{"num_bytes":54164053,"num_objects":24,"num_object_clones":21,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":1,"num_read":460,"num_read_kb":478907,"num_write":344,"num_write_kb":148690,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"13a","length":"1"},{"start":"179","length":"4"}]},{"pgid":"4.9","version":"686'23","reported_seq":1022,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-20T22:56:43.355626+0000","last_change":"2024-09-20T22:55:25.836827+0000","last_active":"2024-09-20T22:56:43.355626+0000","last_peered":"2024-09-20T22:56:43.355626+0000","last_clean":"2024-09-20T22:56:43.355626+0000","last_became_active":"2024-09-20T22:55:17.346757+0000","last_became_peered":"2024-09-20T22:55:17.346757+0000","last_unstale":"2024-09-20T22:56:43.355626+0000","last_undegraded":"2024-09-20T22:56:43.355626+0000","last_fullsized":"2024-09-20T22:56:43.355626+0000","mapping_epoch":836,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":837,"parent":"0.0","parent_split_bits":0,"last_scrub":"686'23","last_scrub_stamp":"2024-09-20T22:55:25.836747+0000","last_deep_scrub":"686'23","last_deep_scrub_stamp":"2024-09-20T22:52:46.768812+0000","last_clean_scrub_stamp":"2024-09-20T22:55:25.836747+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:56:53.576369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":21,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":1588,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-20T22:56:43.356971+0000","last_change":"2024-09-20T22:56:08.359044+0000","last_active":"2024-09-20T22:56:43.356971+0000","last_peered":"2024-09-20T22:56:43.356971+0000","last_clean":"2024-09-20T22:56:43.356971+0000","last_became_active":"2024-09-20T22:56:08.358713+0000","last_became_peered":"2024-09-20T22:56:08.358713+0000","last_unstale":"2024-09-20T22:56:43.356971+0000","last_undegraded":"2024-09-20T22:56:43.356971+0000","last_fullsized":"2024-09-20T22:56:43.356971+0000","mapping_epoch":887,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:55:28.819002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:46:52.465940+0000","last_clean_scrub_stamp":"2024-09-20T22:55:28.819002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:56:44.615881+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":1805,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-20T22:56:43.361275+0000","last_change":"2024-09-20T22:54:48.350732+0000","last_active":"2024-09-20T22:56:43.361275+0000","last_peered":"2024-09-20T22:56:43.361275+0000","last_clean":"2024-09-20T22:56:43.361275+0000","last_became_active":"2024-09-20T22:54:17.340269+0000","last_became_peered":"2024-09-20T22:54:17.340269+0000","last_unstale":"2024-09-20T22:56:43.361275+0000","last_undegraded":"2024-09-20T22:56:43.361275+0000","last_fullsized":"2024-09-20T22:56:43.361275+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:54:48.350593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:53:07.104891+0000","last_clean_scrub_stamp":"2024-09-20T22:54:48.350593+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":1764,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-20T22:56:43.352743+0000","last_change":"2024-09-20T22:56:08.391579+0000","last_active":"2024-09-20T22:56:43.352743+0000","last_peered":"2024-09-20T22:56:43.352743+0000","last_clean":"2024-09-20T22:56:43.352743+0000","last_became_active":"2024-09-20T22:56:08.391268+0000","last_became_peered":"2024-09-20T22:56:08.391268+0000","last_unstale":"2024-09-20T22:56:43.352743+0000","last_undegraded":"2024-09-20T22:56:43.352743+0000","last_fullsized":"2024-09-20T22:56:43.352743+0000","mapping_epoch":887,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:53.242204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:55.170379+0000","last_clean_scrub_stamp":"2024-09-20T22:53:53.242204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"921'1511","reported_seq":5721,"reported_epoch":923,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T22:56:43.367049+0000","last_change":"2024-09-20T22:56:42.448080+0000","last_active":"2024-09-20T22:56:43.367049+0000","last_peered":"2024-09-20T22:56:43.367049+0000","last_clean":"2024-09-20T22:56:43.367049+0000","last_became_active":"2024-09-20T22:54:20.415706+0000","last_became_peered":"2024-09-20T22:54:20.415706+0000","last_unstale":"2024-09-20T22:56:43.367049+0000","last_undegraded":"2024-09-20T22:56:43.367049+0000","last_fullsized":"2024-09-20T22:56:43.367049+0000","mapping_epoch":779,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":780,"parent":"0.0","parent_split_bits":0,"last_scrub":"794'1255","last_scrub_stamp":"2024-09-20T22:54:37.866509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:54:37.866509+0000","objects_scrubbed":15,"log_size":1511,"log_dups_size":0,"ondisk_log_size":1511,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":5,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":1.0812355170000001,"stat_sum":{"num_bytes":53196518,"num_objects":25,"num_object_clones":20,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":2,"num_read":634,"num_read_kb":714967,"num_write":427,"num_write_kb":155961,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":168,"num_bytes_recovered":68651320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"13a","length":"1"},{"start":"179","length":"4"},{"start":"181","length":"4"},{"start":"186","length":"3"},{"start":"18a","length":"2"},{"start":"191","length":"2"},{"start":"194","length":"2"},{"start":"199","length":"1"}]},{"pgid":"4.8","version":"625'27","reported_seq":1826,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-20T22:56:43.355167+0000","last_change":"2024-09-20T22:56:08.394734+0000","last_active":"2024-09-20T22:56:43.355167+0000","last_peered":"2024-09-20T22:56:43.355167+0000","last_clean":"2024-09-20T22:56:43.355167+0000","last_became_active":"2024-09-20T22:56:08.360719+0000","last_became_peered":"2024-09-20T22:56:08.360719+0000","last_unstale":"2024-09-20T22:56:43.355167+0000","last_undegraded":"2024-09-20T22:56:43.355167+0000","last_fullsized":"2024-09-20T22:56:43.355167+0000","mapping_epoch":887,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'27","last_scrub_stamp":"2024-09-20T22:54:41.094989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:54:41.094989+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'2","reported_seq":793,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-20T22:56:43.350786+0000","last_change":"2024-09-20T22:56:08.422471+0000","last_active":"2024-09-20T22:56:43.350786+0000","last_peered":"2024-09-20T22:56:43.350786+0000","last_clean":"2024-09-20T22:56:43.350786+0000","last_became_active":"2024-09-20T22:56:08.392282+0000","last_became_peered":"2024-09-20T22:56:08.392282+0000","last_unstale":"2024-09-20T22:56:43.350786+0000","last_undegraded":"2024-09-20T22:56:43.350786+0000","last_fullsized":"2024-09-20T22:56:43.350786+0000","mapping_epoch":887,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-09-20T22:55:33.770279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:55:33.770279+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":276,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.9","version":"888'11","reported_seq":119,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-20T22:56:43.356913+0000","last_change":"2024-09-20T22:56:08.373687+0000","last_active":"2024-09-20T22:56:43.356913+0000","last_peered":"2024-09-20T22:56:43.356913+0000","last_clean":"2024-09-20T22:56:43.356913+0000","last_became_active":"2024-09-20T22:56:08.357520+0000","last_became_peered":"2024-09-20T22:56:08.357520+0000","last_unstale":"2024-09-20T22:56:43.356913+0000","last_undegraded":"2024-09-20T22:56:43.356913+0000","last_fullsized":"2024-09-20T22:56:43.356913+0000","mapping_epoch":887,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":0,"last_scrub":"408'10","last_scrub_stamp":"2024-09-20T22:55:26.889218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:55:26.889218+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":1765,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-20T22:56:43.355433+0000","last_change":"2024-09-20T22:54:43.129941+0000","last_active":"2024-09-20T22:56:43.355433+0000","last_peered":"2024-09-20T22:56:43.355433+0000","last_clean":"2024-09-20T22:56:43.355433+0000","last_became_active":"2024-09-20T22:47:49.227569+0000","last_became_peered":"2024-09-20T22:47:49.227569+0000","last_unstale":"2024-09-20T22:56:43.355433+0000","last_undegraded":"2024-09-20T22:56:43.355433+0000","last_fullsized":"2024-09-20T22:56:43.355433+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:54:43.129868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:54:43.129868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":1809,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-20T22:56:43.356485+0000","last_change":"2024-09-20T22:56:17.950955+0000","last_active":"2024-09-20T22:56:43.356485+0000","last_peered":"2024-09-20T22:56:43.356485+0000","last_clean":"2024-09-20T22:56:43.356485+0000","last_became_active":"2024-09-20T22:47:49.213292+0000","last_became_peered":"2024-09-20T22:47:49.213292+0000","last_unstale":"2024-09-20T22:56:43.356485+0000","last_undegraded":"2024-09-20T22:56:43.356485+0000","last_fullsized":"2024-09-20T22:56:43.356485+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:56:17.950879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:56:17.950879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:27.752643+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"625'12","reported_seq":1822,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-20T22:56:43.351421+0000","last_change":"2024-09-20T22:56:08.433560+0000","last_active":"2024-09-20T22:56:43.351421+0000","last_peered":"2024-09-20T22:56:43.351421+0000","last_clean":"2024-09-20T22:56:43.351421+0000","last_became_active":"2024-09-20T22:56:08.408706+0000","last_became_peered":"2024-09-20T22:56:08.408706+0000","last_unstale":"2024-09-20T22:56:43.351421+0000","last_undegraded":"2024-09-20T22:56:43.351421+0000","last_fullsized":"2024-09-20T22:56:43.351421+0000","mapping_epoch":887,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'12","last_scrub_stamp":"2024-09-20T22:53:26.319144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:53:26.319144+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":1777,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-20T22:56:43.356791+0000","last_change":"2024-09-20T22:55:29.324367+0000","last_active":"2024-09-20T22:56:43.356791+0000","last_peered":"2024-09-20T22:56:43.356791+0000","last_clean":"2024-09-20T22:56:43.356791+0000","last_became_active":"2024-09-20T22:54:17.339912+0000","last_became_peered":"2024-09-20T22:54:17.339912+0000","last_unstale":"2024-09-20T22:56:43.356791+0000","last_undegraded":"2024-09-20T22:56:43.356791+0000","last_fullsized":"2024-09-20T22:56:43.356791+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:55:29.324286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:09.705632+0000","last_clean_scrub_stamp":"2024-09-20T22:55:29.324286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"74'1","reported_seq":97,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-20T22:56:43.357710+0000","last_change":"2024-09-20T22:56:08.370929+0000","last_active":"2024-09-20T22:56:43.357710+0000","last_peered":"2024-09-20T22:56:43.357710+0000","last_clean":"2024-09-20T22:56:43.357710+0000","last_became_active":"2024-09-20T22:56:08.354987+0000","last_became_peered":"2024-09-20T22:56:08.354987+0000","last_unstale":"2024-09-20T22:56:43.357710+0000","last_undegraded":"2024-09-20T22:56:43.357710+0000","last_fullsized":"2024-09-20T22:56:43.357710+0000","mapping_epoch":887,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-09-20T22:55:07.676011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:55:07.676011+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.16","version":"625'15","reported_seq":1149,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-20T22:56:43.351837+0000","last_change":"2024-09-20T22:56:08.430647+0000","last_active":"2024-09-20T22:56:43.351837+0000","last_peered":"2024-09-20T22:56:43.351837+0000","last_clean":"2024-09-20T22:56:43.351837+0000","last_became_active":"2024-09-20T22:56:08.395591+0000","last_became_peered":"2024-09-20T22:56:08.395591+0000","last_unstale":"2024-09-20T22:56:43.351837+0000","last_undegraded":"2024-09-20T22:56:43.351837+0000","last_fullsized":"2024-09-20T22:56:43.351837+0000","mapping_epoch":887,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'15","last_scrub_stamp":"2024-09-20T22:55:27.637946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:55:27.637946+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":809,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-20T22:56:43.357223+0000","last_change":"2024-09-20T22:56:11.967360+0000","last_active":"2024-09-20T22:56:43.357223+0000","last_peered":"2024-09-20T22:56:43.357223+0000","last_clean":"2024-09-20T22:56:43.357223+0000","last_became_active":"2024-09-20T22:56:08.357576+0000","last_became_peered":"2024-09-20T22:56:08.357576+0000","last_unstale":"2024-09-20T22:56:43.357223+0000","last_undegraded":"2024-09-20T22:56:43.357223+0000","last_fullsized":"2024-09-20T22:56:43.357223+0000","mapping_epoch":887,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:56:11.967266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:56:11.967266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:35.229639+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":1776,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-20T22:56:43.351344+0000","last_change":"2024-09-20T22:56:08.408103+0000","last_active":"2024-09-20T22:56:43.351344+0000","last_peered":"2024-09-20T22:56:43.351344+0000","last_clean":"2024-09-20T22:56:43.351344+0000","last_became_active":"2024-09-20T22:56:08.407772+0000","last_became_peered":"2024-09-20T22:56:08.407772+0000","last_unstale":"2024-09-20T22:56:43.351344+0000","last_undegraded":"2024-09-20T22:56:43.351344+0000","last_fullsized":"2024-09-20T22:56:43.351344+0000","mapping_epoch":887,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:55:52.481737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:55:52.481737+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:02.710042+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":1815,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-20T22:56:43.369806+0000","last_change":"2024-09-20T22:56:42.057480+0000","last_active":"2024-09-20T22:56:43.369806+0000","last_peered":"2024-09-20T22:56:43.369806+0000","last_clean":"2024-09-20T22:56:43.369806+0000","last_became_active":"2024-09-20T22:56:08.369854+0000","last_became_peered":"2024-09-20T22:56:08.369854+0000","last_unstale":"2024-09-20T22:56:43.369806+0000","last_undegraded":"2024-09-20T22:56:43.369806+0000","last_fullsized":"2024-09-20T22:56:43.369806+0000","mapping_epoch":887,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:56:42.057380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:53:58.169894+0000","last_clean_scrub_stamp":"2024-09-20T22:56:42.057380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:43.185285+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"625'15","reported_seq":1810,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-20T22:56:43.356653+0000","last_change":"2024-09-20T22:56:15.488719+0000","last_active":"2024-09-20T22:56:43.356653+0000","last_peered":"2024-09-20T22:56:43.356653+0000","last_clean":"2024-09-20T22:56:43.356653+0000","last_became_active":"2024-09-20T22:56:08.359754+0000","last_became_peered":"2024-09-20T22:56:08.359754+0000","last_unstale":"2024-09-20T22:56:43.356653+0000","last_undegraded":"2024-09-20T22:56:43.356653+0000","last_fullsized":"2024-09-20T22:56:43.356653+0000","mapping_epoch":887,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'15","last_scrub_stamp":"2024-09-20T22:56:15.488641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:56:15.488641+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:35.746703+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":1128,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-20T22:56:43.349090+0000","last_change":"2024-09-20T22:56:16.523641+0000","last_active":"2024-09-20T22:56:43.349090+0000","last_peered":"2024-09-20T22:56:43.349090+0000","last_clean":"2024-09-20T22:56:43.349090+0000","last_became_active":"2024-09-20T22:56:08.390343+0000","last_became_peered":"2024-09-20T22:56:08.390343+0000","last_unstale":"2024-09-20T22:56:43.349090+0000","last_undegraded":"2024-09-20T22:56:43.349090+0000","last_fullsized":"2024-09-20T22:56:43.349090+0000","mapping_epoch":887,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:56:16.523567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:56:16.523567+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:25.089927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"888'9","reported_seq":1953,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-20T22:56:43.352742+0000","last_change":"2024-09-20T22:56:08.411549+0000","last_active":"2024-09-20T22:56:43.352742+0000","last_peered":"2024-09-20T22:56:43.352742+0000","last_clean":"2024-09-20T22:56:43.352742+0000","last_became_active":"2024-09-20T22:56:08.391526+0000","last_became_peered":"2024-09-20T22:56:08.391526+0000","last_unstale":"2024-09-20T22:56:43.352742+0000","last_undegraded":"2024-09-20T22:56:43.352742+0000","last_fullsized":"2024-09-20T22:56:43.352742+0000","mapping_epoch":887,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-09-20T22:53:59.165134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:53:59.165134+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":731,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-20T22:56:43.364233+0000","last_change":"2024-09-20T22:55:29.161667+0000","last_active":"2024-09-20T22:56:43.364233+0000","last_peered":"2024-09-20T22:56:43.364233+0000","last_clean":"2024-09-20T22:56:43.364233+0000","last_became_active":"2024-09-20T22:55:17.346270+0000","last_became_peered":"2024-09-20T22:55:17.346270+0000","last_unstale":"2024-09-20T22:56:43.364233+0000","last_undegraded":"2024-09-20T22:56:43.364233+0000","last_fullsized":"2024-09-20T22:56:43.364233+0000","mapping_epoch":836,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":837,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:55:29.161595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:55:29.161595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:56:56.713856+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.14","version":"625'20","reported_seq":1854,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-20T22:56:43.352231+0000","last_change":"2024-09-20T22:56:08.434334+0000","last_active":"2024-09-20T22:56:43.352231+0000","last_peered":"2024-09-20T22:56:43.352231+0000","last_clean":"2024-09-20T22:56:43.352231+0000","last_became_active":"2024-09-20T22:56:08.391715+0000","last_became_peered":"2024-09-20T22:56:08.391715+0000","last_unstale":"2024-09-20T22:56:43.352231+0000","last_undegraded":"2024-09-20T22:56:43.352231+0000","last_fullsized":"2024-09-20T22:56:43.352231+0000","mapping_epoch":887,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'20","last_scrub_stamp":"2024-09-20T22:55:23.474362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:55:23.474362+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":797,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-20T22:56:43.361397+0000","last_change":"2024-09-20T22:56:08.372321+0000","last_active":"2024-09-20T22:56:43.361397+0000","last_peered":"2024-09-20T22:56:43.361397+0000","last_clean":"2024-09-20T22:56:43.361397+0000","last_became_active":"2024-09-20T22:56:08.371946+0000","last_became_peered":"2024-09-20T22:56:08.371946+0000","last_unstale":"2024-09-20T22:56:43.361397+0000","last_undegraded":"2024-09-20T22:56:43.361397+0000","last_fullsized":"2024-09-20T22:56:43.361397+0000","mapping_epoch":887,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:55:49.897082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:55:49.897082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:03.022614+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.15","version":"778'12","reported_seq":1999,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-20T22:56:43.356451+0000","last_change":"2024-09-20T22:55:26.312258+0000","last_active":"2024-09-20T22:56:43.356451+0000","last_peered":"2024-09-20T22:56:43.356451+0000","last_clean":"2024-09-20T22:56:43.356451+0000","last_became_active":"2024-09-20T22:54:17.341014+0000","last_became_peered":"2024-09-20T22:54:17.341014+0000","last_unstale":"2024-09-20T22:56:43.356451+0000","last_undegraded":"2024-09-20T22:56:43.356451+0000","last_fullsized":"2024-09-20T22:56:43.356451+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"778'12","last_scrub_stamp":"2024-09-20T22:55:26.312166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:55:26.312166+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":1112,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-20T22:56:43.349620+0000","last_change":"2024-09-20T22:56:08.380765+0000","last_active":"2024-09-20T22:56:43.349620+0000","last_peered":"2024-09-20T22:56:43.349620+0000","last_clean":"2024-09-20T22:56:43.349620+0000","last_became_active":"2024-09-20T22:56:08.380461+0000","last_became_peered":"2024-09-20T22:56:08.380461+0000","last_unstale":"2024-09-20T22:56:43.349620+0000","last_undegraded":"2024-09-20T22:56:43.349620+0000","last_fullsized":"2024-09-20T22:56:43.349620+0000","mapping_epoch":887,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:54:47.822663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:50:21.853070+0000","last_clean_scrub_stamp":"2024-09-20T22:54:47.822663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"625'19","reported_seq":699,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-20T22:56:43.357290+0000","last_change":"2024-09-20T22:55:55.832120+0000","last_active":"2024-09-20T22:56:43.357290+0000","last_peered":"2024-09-20T22:56:43.357290+0000","last_clean":"2024-09-20T22:56:43.357290+0000","last_became_active":"2024-09-20T22:55:17.342351+0000","last_became_peered":"2024-09-20T22:55:17.342351+0000","last_unstale":"2024-09-20T22:56:43.357290+0000","last_undegraded":"2024-09-20T22:56:43.357290+0000","last_fullsized":"2024-09-20T22:56:43.357290+0000","mapping_epoch":836,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":837,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'19","last_scrub_stamp":"2024-09-20T22:55:55.832023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:55:55.832023+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:03.478534+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":23,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":593,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-20T22:56:43.357090+0000","last_change":"2024-09-20T22:56:08.369525+0000","last_active":"2024-09-20T22:56:43.357090+0000","last_peered":"2024-09-20T22:56:43.357090+0000","last_clean":"2024-09-20T22:56:43.357090+0000","last_became_active":"2024-09-20T22:56:08.369206+0000","last_became_peered":"2024-09-20T22:56:08.369206+0000","last_unstale":"2024-09-20T22:56:43.357090+0000","last_undegraded":"2024-09-20T22:56:43.357090+0000","last_fullsized":"2024-09-20T22:56:43.357090+0000","mapping_epoch":887,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:55:22.876700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:52:47.717490+0000","last_clean_scrub_stamp":"2024-09-20T22:55:22.876700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:56:45.495660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":311,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-20T22:56:43.357346+0000","last_change":"2024-09-20T22:55:03.853796+0000","last_active":"2024-09-20T22:56:43.357346+0000","last_peered":"2024-09-20T22:56:43.357346+0000","last_clean":"2024-09-20T22:56:43.357346+0000","last_became_active":"2024-09-20T22:54:17.336853+0000","last_became_peered":"2024-09-20T22:54:17.336853+0000","last_unstale":"2024-09-20T22:56:43.357346+0000","last_undegraded":"2024-09-20T22:56:43.357346+0000","last_fullsized":"2024-09-20T22:56:43.357346+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:55:03.853725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:55:03.853725+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":1766,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-20T22:56:43.348850+0000","last_change":"2024-09-20T22:56:22.490589+0000","last_active":"2024-09-20T22:56:43.348850+0000","last_peered":"2024-09-20T22:56:43.348850+0000","last_clean":"2024-09-20T22:56:43.348850+0000","last_became_active":"2024-09-20T22:47:49.216141+0000","last_became_peered":"2024-09-20T22:47:49.216141+0000","last_unstale":"2024-09-20T22:56:43.348850+0000","last_undegraded":"2024-09-20T22:56:43.348850+0000","last_fullsized":"2024-09-20T22:56:43.348850+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:56:22.490523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:56:22.490523+0000","last_clean_scrub_stamp":"2024-09-20T22:56:22.490523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:46.414566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"625'19","reported_seq":321,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-20T22:56:43.356386+0000","last_change":"2024-09-20T22:55:33.932286+0000","last_active":"2024-09-20T22:56:43.356386+0000","last_peered":"2024-09-20T22:56:43.356386+0000","last_clean":"2024-09-20T22:56:43.356386+0000","last_became_active":"2024-09-20T22:54:17.340539+0000","last_became_peered":"2024-09-20T22:54:17.340539+0000","last_unstale":"2024-09-20T22:56:43.356386+0000","last_undegraded":"2024-09-20T22:56:43.356386+0000","last_fullsized":"2024-09-20T22:56:43.356386+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'19","last_scrub_stamp":"2024-09-20T22:55:33.932211+0000","last_deep_scrub":"404'13","last_deep_scrub_stamp":"2024-09-20T22:48:04.276282+0000","last_clean_scrub_stamp":"2024-09-20T22:55:33.932211+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:03.430646+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":804,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-20T22:56:43.349798+0000","last_change":"2024-09-20T22:56:08.377943+0000","last_active":"2024-09-20T22:56:43.349798+0000","last_peered":"2024-09-20T22:56:43.349798+0000","last_clean":"2024-09-20T22:56:43.349798+0000","last_became_active":"2024-09-20T22:56:08.377638+0000","last_became_peered":"2024-09-20T22:56:08.377638+0000","last_unstale":"2024-09-20T22:56:43.349798+0000","last_undegraded":"2024-09-20T22:56:43.349798+0000","last_fullsized":"2024-09-20T22:56:43.349798+0000","mapping_epoch":887,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:54:46.541109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:31.138067+0000","last_clean_scrub_stamp":"2024-09-20T22:54:46.541109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":1774,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-20T22:56:43.352834+0000","last_change":"2024-09-20T22:54:17.337400+0000","last_active":"2024-09-20T22:56:43.352834+0000","last_peered":"2024-09-20T22:56:43.352834+0000","last_clean":"2024-09-20T22:56:43.352834+0000","last_became_active":"2024-09-20T22:54:17.337102+0000","last_became_peered":"2024-09-20T22:54:17.337102+0000","last_unstale":"2024-09-20T22:56:43.352834+0000","last_undegraded":"2024-09-20T22:56:43.352834+0000","last_fullsized":"2024-09-20T22:56:43.352834+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:34.345077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:53:34.345077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":1118,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-20T22:56:43.349230+0000","last_change":"2024-09-20T22:55:09.664923+0000","last_active":"2024-09-20T22:56:43.349230+0000","last_peered":"2024-09-20T22:56:43.349230+0000","last_clean":"2024-09-20T22:56:43.349230+0000","last_became_active":"2024-09-20T22:54:17.344021+0000","last_became_peered":"2024-09-20T22:54:17.344021+0000","last_unstale":"2024-09-20T22:56:43.349230+0000","last_undegraded":"2024-09-20T22:56:43.349230+0000","last_fullsized":"2024-09-20T22:56:43.349230+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:55:09.664839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:55:09.664839+0000","last_clean_scrub_stamp":"2024-09-20T22:55:09.664839+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"625'24","reported_seq":93,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-20T22:56:43.349971+0000","last_change":"2024-09-20T22:56:08.426576+0000","last_active":"2024-09-20T22:56:43.349971+0000","last_peered":"2024-09-20T22:56:43.349971+0000","last_clean":"2024-09-20T22:56:43.349971+0000","last_became_active":"2024-09-20T22:56:08.388677+0000","last_became_peered":"2024-09-20T22:56:08.388677+0000","last_unstale":"2024-09-20T22:56:43.349971+0000","last_undegraded":"2024-09-20T22:56:43.349971+0000","last_fullsized":"2024-09-20T22:56:43.349971+0000","mapping_epoch":887,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'24","last_scrub_stamp":"2024-09-20T22:53:33.089580+0000","last_deep_scrub":"470'16","last_deep_scrub_stamp":"2024-09-20T22:49:10.659868+0000","last_clean_scrub_stamp":"2024-09-20T22:53:33.089580+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":1791,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-20T22:56:43.356900+0000","last_change":"2024-09-20T22:56:08.368652+0000","last_active":"2024-09-20T22:56:43.356900+0000","last_peered":"2024-09-20T22:56:43.356900+0000","last_clean":"2024-09-20T22:56:43.356900+0000","last_became_active":"2024-09-20T22:56:08.368320+0000","last_became_peered":"2024-09-20T22:56:08.368320+0000","last_unstale":"2024-09-20T22:56:43.356900+0000","last_undegraded":"2024-09-20T22:56:43.356900+0000","last_fullsized":"2024-09-20T22:56:43.356900+0000","mapping_epoch":887,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:55:54.521725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:45:24.765414+0000","last_clean_scrub_stamp":"2024-09-20T22:55:54.521725+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:11.050782+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":793,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-20T22:56:43.355822+0000","last_change":"2024-09-20T22:56:08.356254+0000","last_active":"2024-09-20T22:56:43.355822+0000","last_peered":"2024-09-20T22:56:43.355822+0000","last_clean":"2024-09-20T22:56:43.355822+0000","last_became_active":"2024-09-20T22:56:08.355950+0000","last_became_peered":"2024-09-20T22:56:08.355950+0000","last_unstale":"2024-09-20T22:56:43.355822+0000","last_undegraded":"2024-09-20T22:56:43.355822+0000","last_fullsized":"2024-09-20T22:56:43.355822+0000","mapping_epoch":887,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:55:27.853089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:50.174861+0000","last_clean_scrub_stamp":"2024-09-20T22:55:27.853089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:56:47.902955+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":1758,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-20T22:56:43.353148+0000","last_change":"2024-09-20T22:54:22.276389+0000","last_active":"2024-09-20T22:56:43.353148+0000","last_peered":"2024-09-20T22:56:43.353148+0000","last_clean":"2024-09-20T22:56:43.353148+0000","last_became_active":"2024-09-20T22:47:49.214487+0000","last_became_peered":"2024-09-20T22:47:49.214487+0000","last_unstale":"2024-09-20T22:56:43.353148+0000","last_undegraded":"2024-09-20T22:56:43.353148+0000","last_fullsized":"2024-09-20T22:56:43.353148+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:54:22.276321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:54:22.276321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"625'8","reported_seq":1798,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-20T22:56:43.349684+0000","last_change":"2024-09-20T22:56:42.475672+0000","last_active":"2024-09-20T22:56:43.349684+0000","last_peered":"2024-09-20T22:56:43.349684+0000","last_clean":"2024-09-20T22:56:43.349684+0000","last_became_active":"2024-09-20T22:54:17.341776+0000","last_became_peered":"2024-09-20T22:54:17.341776+0000","last_unstale":"2024-09-20T22:56:43.349684+0000","last_undegraded":"2024-09-20T22:56:43.349684+0000","last_fullsized":"2024-09-20T22:56:43.349684+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'8","last_scrub_stamp":"2024-09-20T22:56:42.475598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:56:42.475598+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:43.361953+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":1114,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-20T22:56:43.350882+0000","last_change":"2024-09-20T22:55:11.688577+0000","last_active":"2024-09-20T22:56:43.350882+0000","last_peered":"2024-09-20T22:56:43.350882+0000","last_clean":"2024-09-20T22:56:43.350882+0000","last_became_active":"2024-09-20T22:47:49.231964+0000","last_became_peered":"2024-09-20T22:47:49.231964+0000","last_unstale":"2024-09-20T22:56:43.350882+0000","last_undegraded":"2024-09-20T22:56:43.350882+0000","last_fullsized":"2024-09-20T22:56:43.350882+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:55:11.688508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:55:11.688508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":1819,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-20T22:56:43.357935+0000","last_change":"2024-09-20T22:56:39.112737+0000","last_active":"2024-09-20T22:56:43.357935+0000","last_peered":"2024-09-20T22:56:43.357935+0000","last_clean":"2024-09-20T22:56:43.357935+0000","last_became_active":"2024-09-20T22:56:08.357087+0000","last_became_peered":"2024-09-20T22:56:08.357087+0000","last_unstale":"2024-09-20T22:56:43.357935+0000","last_undegraded":"2024-09-20T22:56:43.357935+0000","last_fullsized":"2024-09-20T22:56:43.357935+0000","mapping_epoch":887,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:56:39.112639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:56:39.112639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:52.233191+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":785,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-20T22:56:43.362201+0000","last_change":"2024-09-20T22:56:40.102381+0000","last_active":"2024-09-20T22:56:43.362201+0000","last_peered":"2024-09-20T22:56:43.362201+0000","last_clean":"2024-09-20T22:56:43.362201+0000","last_became_active":"2024-09-20T22:56:08.389017+0000","last_became_peered":"2024-09-20T22:56:08.389017+0000","last_unstale":"2024-09-20T22:56:43.362201+0000","last_undegraded":"2024-09-20T22:56:43.362201+0000","last_fullsized":"2024-09-20T22:56:43.362201+0000","mapping_epoch":887,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:56:40.102311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:55:08.681729+0000","last_clean_scrub_stamp":"2024-09-20T22:56:40.102311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:53.284275+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":310,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-20T22:56:43.355468+0000","last_change":"2024-09-20T22:55:23.899481+0000","last_active":"2024-09-20T22:56:43.355468+0000","last_peered":"2024-09-20T22:56:43.355468+0000","last_clean":"2024-09-20T22:56:43.355468+0000","last_became_active":"2024-09-20T22:54:17.348206+0000","last_became_peered":"2024-09-20T22:54:17.348206+0000","last_unstale":"2024-09-20T22:56:43.355468+0000","last_undegraded":"2024-09-20T22:56:43.355468+0000","last_fullsized":"2024-09-20T22:56:43.355468+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:55:23.899398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:55:23.899398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":1796,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-20T22:56:43.351915+0000","last_change":"2024-09-20T22:54:17.342220+0000","last_active":"2024-09-20T22:56:43.351915+0000","last_peered":"2024-09-20T22:56:43.351915+0000","last_clean":"2024-09-20T22:56:43.351915+0000","last_became_active":"2024-09-20T22:54:17.341405+0000","last_became_peered":"2024-09-20T22:54:17.341405+0000","last_unstale":"2024-09-20T22:56:43.351915+0000","last_undegraded":"2024-09-20T22:56:43.351915+0000","last_fullsized":"2024-09-20T22:56:43.351915+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:31.274390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:53:31.274390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"625'19","reported_seq":1864,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-20T22:56:43.358453+0000","last_change":"2024-09-20T22:56:18.981584+0000","last_active":"2024-09-20T22:56:43.358453+0000","last_peered":"2024-09-20T22:56:43.358453+0000","last_clean":"2024-09-20T22:56:43.358453+0000","last_became_active":"2024-09-20T22:54:17.342236+0000","last_became_peered":"2024-09-20T22:54:17.342236+0000","last_unstale":"2024-09-20T22:56:43.358453+0000","last_undegraded":"2024-09-20T22:56:43.358453+0000","last_fullsized":"2024-09-20T22:56:43.358453+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'19","last_scrub_stamp":"2024-09-20T22:56:18.981492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:56:18.981492+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:24.933669+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":1113,"reported_epoch":923,"state":"active+clean","last_fresh":"2024-09-20T22:56:43.352262+0000","last_change":"2024-09-20T22:56:08.391833+0000","last_active":"2024-09-20T22:56:43.352262+0000","last_peered":"2024-09-20T22:56:43.352262+0000","last_clean":"2024-09-20T22:56:43.352262+0000","last_became_active":"2024-09-20T22:56:08.391532+0000","last_became_peered":"2024-09-20T22:56:08.391532+0000","last_unstale":"2024-09-20T22:56:43.352262+0000","last_undegraded":"2024-09-20T22:56:43.352262+0000","last_fullsized":"2024-09-20T22:56:43.352262+0000","mapping_epoch":887,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:54:43.813540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:54:43.813540+0000","last_clean_scrub_stamp":"2024-09-20T22:54:43.813540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":681329667,"num_objects":302,"num_object_clones":252,"num_object_copies":906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":302,"num_whiteouts":12,"num_read":6521,"num_read_kb":6852469,"num_write":4655,"num_write_kb":1770023,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1279,"num_bytes_recovered":942387963,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1670176768,"data_stored":2295181813,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":16295,"ondisk_log_size":16295,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":95,"ondisk_log_size":95,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1098,"num_read_kb":855,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":471,"num_bytes_recovered":10644,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":462848,"data_stored":12360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":768,"ondisk_log_size":768,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":1034,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":4931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":783,"num_read_kb":778,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":1594,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":655360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":385,"ondisk_log_size":385,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":0,"seq":253403070649,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430399,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":251676,"kb_used_data":188608,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":93493476,"statfs":{"total":95995035648,"available":95737319424,"internally_reserved":0,"allocated":193134592,"data_stored":269435388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64551366},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798692036,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":228280,"kb_used_data":179044,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":93516872,"statfs":{"total":95995035648,"available":95761276928,"internally_reserved":0,"allocated":183341056,"data_stored":232158614,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986377,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":218628,"kb_used_data":161068,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":93526524,"statfs":{"total":95995035648,"available":95771160576,"internally_reserved":0,"allocated":164933632,"data_stored":217631321,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182607,"num_pgs":76,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":415676,"kb_used_data":344224,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93329476,"statfs":{"total":95995035648,"available":95569383424,"internally_reserved":0,"allocated":352485376,"data_stored":480609771,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":73136583},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411540,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":312236,"kb_used_data":218640,"kb_used_omap":1,"kb_used_meta":93566,"kb_avail":93432916,"statfs":{"total":95995035648,"available":95675305984,"internally_reserved":0,"allocated":223887360,"data_stored":300224117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":95812037},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575065,"num_pgs":75,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":257208,"kb_used_data":203812,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93487944,"statfs":{"total":95995035648,"available":95731654656,"internally_reserved":0,"allocated":208703488,"data_stored":265661038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738590,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":313440,"kb_used_data":232328,"kb_used_omap":1,"kb_used_meta":81086,"kb_avail":93431712,"statfs":{"total":95995035648,"available":95674073088,"internally_reserved":0,"allocated":237903872,"data_stored":319535019,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":83032517},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":231219200,"data_stored":316363344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":199548928,"data_stored":260087036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":216924160,"data_stored":296782703,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":342769664,"data_stored":474444904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":158244864,"data_stored":214460302,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":173625344,"data_stored":225993824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":186114048,"data_stored":265928890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":161730560,"data_stored":241120810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-20T22:56:49.345 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-09-20T22:56:49.345 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd pool set unique_pool_0 pg_num 26 2024-09-20T22:56:49.360 INFO:tasks.rados.rados.0.smithi084.stdout:3743: done (1 left) 2024-09-20T22:56:49.360 INFO:tasks.rados.rados.0.smithi084.stdout:3744: done (0 left) 2024-09-20T22:56:49.360 INFO:tasks.rados.rados.0.smithi084.stdout:3745: write (excl) oid 28 current snap is 433 2024-09-20T22:56:49.361 INFO:tasks.rados.rados.0.smithi084.stdout:3745: seq_num 1275 ranges {628925=573426,1693365=714102,3096904=332021} 2024-09-20T22:56:49.367 INFO:tasks.rados.rados.0.smithi084.stdout:3745: writing smithi08477203-28 from 628925 to 1202351 tid 1 2024-09-20T22:56:49.374 INFO:tasks.rados.rados.0.smithi084.stdout:3745: writing smithi08477203-28 from 1693365 to 2407467 tid 2 2024-09-20T22:56:49.380 INFO:tasks.rados.rados.0.smithi084.stdout:3745: writing smithi08477203-28 from 3096904 to 3428925 tid 3 2024-09-20T22:56:49.381 INFO:tasks.rados.rados.0.smithi084.stdout:3746: read oid 5 snap -1 2024-09-20T22:56:49.381 INFO:tasks.rados.rados.0.smithi084.stdout:3746: expect (ObjNum 1241 snap 421 seq_num 1241) 2024-09-20T22:56:49.381 INFO:tasks.rados.rados.0.smithi084.stdout:3747: write (excl) oid 38 current snap is 433 2024-09-20T22:56:49.381 INFO:tasks.rados.rados.0.smithi084.stdout:3745: finishing write tid 1 to smithi08477203-28 2024-09-20T22:56:49.381 INFO:tasks.rados.rados.0.smithi084.stdout:3745: finishing write tid 2 to smithi08477203-28 2024-09-20T22:56:49.381 INFO:tasks.rados.rados.0.smithi084.stdout:3747: seq_num 1276 ranges {645732=648675,1751847=574113,3124327=321405} 2024-09-20T22:56:49.387 INFO:tasks.rados.rados.0.smithi084.stdout:3747: writing smithi08477203-38 from 645732 to 1294407 tid 1 2024-09-20T22:56:49.394 INFO:tasks.rados.rados.0.smithi084.stdout:3747: writing smithi08477203-38 from 1751847 to 2325960 tid 2 2024-09-20T22:56:49.400 INFO:tasks.rados.rados.0.smithi084.stdout:3747: writing smithi08477203-38 from 3124327 to 3445732 tid 3 2024-09-20T22:56:49.400 INFO:tasks.rados.rados.0.smithi084.stdout:3748: write (excl) oid 42 current snap is 433 2024-09-20T22:56:49.400 INFO:tasks.rados.rados.0.smithi084.stdout:3745: finishing write tid 3 to smithi08477203-28 2024-09-20T22:56:49.400 INFO:tasks.rados.rados.0.smithi084.stdout:3748: seq_num 1277 ranges {662539=723924,2093976=717771,3435397=27142} 2024-09-20T22:56:49.408 INFO:tasks.rados.rados.0.smithi084.stdout:3748: writing smithi08477203-42 from 662539 to 1386463 tid 1 2024-09-20T22:56:49.416 INFO:tasks.rados.rados.0.smithi084.stdout:3748: writing smithi08477203-42 from 2093976 to 2811747 tid 2 2024-09-20T22:56:49.420 INFO:tasks.rados.rados.0.smithi084.stdout:3748: writing smithi08477203-42 from 3435397 to 3462539 tid 3 2024-09-20T22:56:49.420 INFO:tasks.rados.rados.0.smithi084.stdout:3749: read oid 11 snap -1 2024-09-20T22:56:49.421 INFO:tasks.rados.rados.0.smithi084.stdout:3749: expect (ObjNum 1253 snap 425 seq_num 1253) 2024-09-20T22:56:49.421 INFO:tasks.rados.rados.0.smithi084.stdout:3750: rollback oid 24 current snap is 433 2024-09-20T22:56:49.421 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 24 to 424 2024-09-20T22:56:49.421 INFO:tasks.rados.rados.0.smithi084.stdout:3745: finishing write tid 4 to smithi08477203-28 2024-09-20T22:56:49.421 INFO:tasks.rados.rados.0.smithi084.stdout:3745: finishing write tid 5 to smithi08477203-28 2024-09-20T22:56:49.421 INFO:tasks.rados.rados.0.smithi084.stdout:3745: oid 28 updating version 0 to 3074 2024-09-20T22:56:49.421 INFO:tasks.rados.rados.0.smithi084.stdout:3745: oid 28 updating version 3074 to 3075 2024-09-20T22:56:49.421 INFO:tasks.rados.rados.0.smithi084.stdout:3745: oid 28 version 3075 is already newer than 3073 2024-09-20T22:56:49.421 INFO:tasks.rados.rados.0.smithi084.stdout:3745: oid 28 version 3075 is already newer than 3072 2024-09-20T22:56:49.421 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 28 v 3075 (ObjNum 1275 snap 433 seq_num 1275) dirty exists 2024-09-20T22:56:49.421 INFO:tasks.rados.rados.0.smithi084.stdout:3745: left oid 28 (ObjNum 1275 snap 433 seq_num 1275) 2024-09-20T22:56:49.421 INFO:tasks.rados.rados.0.smithi084.stdout:3745: done (5 left) 2024-09-20T22:56:49.421 INFO:tasks.rados.rados.0.smithi084.stdout:3751: rollback oid 28 current snap is 433 2024-09-20T22:56:49.422 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 28 to 405 2024-09-20T22:56:49.422 INFO:tasks.rados.rados.0.smithi084.stdout:3752: copy_from oid 12 from oid 44 current snap is 433 2024-09-20T22:56:49.422 INFO:tasks.rados.rados.0.smithi084.stdout:3753: delete oid 39 current snap is 433 2024-09-20T22:56:49.422 INFO:tasks.rados.rados.0.smithi084.stdout:3746: expect (ObjNum 1241 snap 421 seq_num 1241) 2024-09-20T22:56:49.435 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:56:49.456 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:49.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:49.457 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:56:49.558 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:56:49.573 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:49 smithi084 ceph-mon[29774]: 3.1f deep-scrub starts 2024-09-20T22:56:49.573 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:49 smithi084 ceph-mon[29774]: 3.1f deep-scrub ok 2024-09-20T22:56:49.573 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:49 smithi084 ceph-mon[29774]: pgmap v1293: 158 pgs: 2 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 152 active+clean; 650 MiB data, 1.9 GiB used, 624 GiB / 626 GiB avail 2024-09-20T22:56:49.573 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:49 smithi084 ceph-mon[29774]: from='client.27332 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T22:56:49.573 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:49 smithi084 ceph-mon[29774]: osdmap e929: 8 total, 8 up, 7 in 2024-09-20T22:56:49.573 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:49 smithi084 ceph-mon[40024]: 3.1f deep-scrub starts 2024-09-20T22:56:49.573 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:49 smithi084 ceph-mon[40024]: 3.1f deep-scrub ok 2024-09-20T22:56:49.573 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:49 smithi084 ceph-mon[40024]: pgmap v1293: 158 pgs: 2 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 152 active+clean; 650 MiB data, 1.9 GiB used, 624 GiB / 626 GiB avail 2024-09-20T22:56:49.574 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:49 smithi084 ceph-mon[40024]: from='client.27332 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T22:56:49.574 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:49 smithi084 ceph-mon[40024]: osdmap e929: 8 total, 8 up, 7 in 2024-09-20T22:56:49.579 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:49.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:49.580 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:56:49.622 INFO:tasks.rados.rados.0.smithi084.stdout:3747: finishing write tid 1 to smithi08477203-38 2024-09-20T22:56:49.622 INFO:tasks.rados.rados.0.smithi084.stdout:3747: finishing write tid 2 to smithi08477203-38 2024-09-20T22:56:49.622 INFO:tasks.rados.rados.0.smithi084.stdout:3746: done (7 left) 2024-09-20T22:56:49.622 INFO:tasks.rados.rados.0.smithi084.stdout:3753: done (6 left) 2024-09-20T22:56:49.622 INFO:tasks.rados.rados.0.smithi084.stdout:3754: read oid 50 snap 420 2024-09-20T22:56:49.622 INFO:tasks.rados.rados.0.smithi084.stdout:3754: expect (ObjNum 1128 snap 381 seq_num 1128) 2024-09-20T22:56:49.622 INFO:tasks.rados.rados.0.smithi084.stdout:3755: snap_remove snap 370 2024-09-20T22:56:49.681 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:56:49.702 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:49.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:49.703 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:56:49.750 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:49 smithi154 ceph-mon[33264]: 3.1f deep-scrub starts 2024-09-20T22:56:49.750 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:49 smithi154 ceph-mon[33264]: 3.1f deep-scrub ok 2024-09-20T22:56:49.750 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:49 smithi154 ceph-mon[33264]: pgmap v1293: 158 pgs: 2 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 152 active+clean; 650 MiB data, 1.9 GiB used, 624 GiB / 626 GiB avail 2024-09-20T22:56:49.750 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:49 smithi154 ceph-mon[33264]: from='client.27332 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T22:56:49.750 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:49 smithi154 ceph-mon[33264]: osdmap e929: 8 total, 8 up, 7 in 2024-09-20T22:56:49.763 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-20T22:56:49.804 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:56:49.820 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.3 dump_blocked_ops 2024-09-20T22:56:49.826 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:49.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:49.827 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:56:49.859 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:56:49.927 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:56:49.950 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:49.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:49.950 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:56:50.004 INFO:journalctl@ceph.iscsi.iscsi.a.smithi154.stdout:Sep 20 22:56:49 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-iscsi-iscsi-a[59730]: debug there is no tcmu-runner data available 2024-09-20T22:56:50.051 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:56:50.072 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:50.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:50.073 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:56:50.174 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:56:50.196 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:50.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:50.197 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:56:50.298 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:56:50.319 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:50.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:50.320 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:56:50.418 INFO:tasks.rados.rados.0.smithi084.stdout:3755: done (7 left) 2024-09-20T22:56:50.418 INFO:tasks.rados.rados.0.smithi084.stdout:3756: snap_create 2024-09-20T22:56:50.419 INFO:tasks.rados.rados.0.smithi084.stdout:3747: finishing write tid 3 to smithi08477203-38 2024-09-20T22:56:50.419 INFO:tasks.rados.rados.0.smithi084.stdout:3747: finishing write tid 4 to smithi08477203-38 2024-09-20T22:56:50.420 INFO:tasks.rados.rados.0.smithi084.stdout:3747: finishing write tid 5 to smithi08477203-38 2024-09-20T22:56:50.420 INFO:tasks.rados.rados.0.smithi084.stdout:3747: oid 38 updating version 0 to 2853 2024-09-20T22:56:50.420 INFO:tasks.rados.rados.0.smithi084.stdout:3747: oid 38 version 2853 is already newer than 2850 2024-09-20T22:56:50.420 INFO:tasks.rados.rados.0.smithi084.stdout:3747: oid 38 version 2853 is already newer than 2851 2024-09-20T22:56:50.420 INFO:tasks.rados.rados.0.smithi084.stdout:3747: oid 38 version 2853 is already newer than 2852 2024-09-20T22:56:50.420 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 38 v 2853 (ObjNum 1276 snap 433 seq_num 1276) dirty exists 2024-09-20T22:56:50.420 INFO:tasks.rados.rados.0.smithi084.stdout:3747: left oid 38 (ObjNum 1276 snap 433 seq_num 1276) 2024-09-20T22:56:50.420 INFO:tasks.rados.rados.0.smithi084.stdout:3748: finishing write tid 1 to smithi08477203-42 2024-09-20T22:56:50.420 INFO:tasks.rados.rados.0.smithi084.stdout:3748: finishing write tid 2 to smithi08477203-42 2024-09-20T22:56:50.420 INFO:tasks.rados.rados.0.smithi084.stdout:3748: finishing write tid 3 to smithi08477203-42 2024-09-20T22:56:50.420 INFO:tasks.rados.rados.0.smithi084.stdout:3750: finishing rollback tid 0 to smithi08477203-24 2024-09-20T22:56:50.420 INFO:tasks.rados.rados.0.smithi084.stdout:3748: finishing write tid 4 to smithi08477203-42 2024-09-20T22:56:50.420 INFO:tasks.rados.rados.0.smithi084.stdout:3748: finishing write tid 5 to smithi08477203-42 2024-09-20T22:56:50.420 INFO:tasks.rados.rados.0.smithi084.stdout:3748: oid 42 updating version 0 to 2644 2024-09-20T22:56:50.420 INFO:tasks.rados.rados.0.smithi084.stdout:3748: oid 42 updating version 2644 to 2645 2024-09-20T22:56:50.421 INFO:tasks.rados.rados.0.smithi084.stdout:3748: oid 42 updating version 2645 to 2646 2024-09-20T22:56:50.421 INFO:tasks.rados.rados.0.smithi084.stdout:3748: oid 42 updating version 2646 to 2647 2024-09-20T22:56:50.421 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 42 v 2647 (ObjNum 1277 snap 433 seq_num 1277) dirty exists 2024-09-20T22:56:50.421 INFO:tasks.rados.rados.0.smithi084.stdout:3748: left oid 42 (ObjNum 1277 snap 433 seq_num 1277) 2024-09-20T22:56:50.421 INFO:tasks.rados.rados.0.smithi084.stdout:3750: finishing rollback tid 1 to smithi08477203-24 2024-09-20T22:56:50.421 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 24 v 1904 (ObjNum 33 snap 0 seq_num 709651946) dirty dne 2024-09-20T22:56:50.421 INFO:tasks.rados.rados.0.smithi084.stdout:3751: finishing rollback tid 0 to smithi08477203-28 2024-09-20T22:56:50.421 INFO:tasks.rados.rados.0.smithi084.stdout:3751: finishing rollback tid 1 to smithi08477203-28 2024-09-20T22:56:50.421 INFO:tasks.rados.rados.0.smithi084.stdout:3751: finishing rollback tid 2 to smithi08477203-28 2024-09-20T22:56:50.421 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 28 v 3078 (ObjNum 1160 snap 398 seq_num 1160) dirty exists 2024-09-20T22:56:50.421 INFO:tasks.rados.rados.0.smithi084.stdout:3752: finishing copy_from racing read to smithi08477203-12 2024-09-20T22:56:50.421 INFO:tasks.rados.rados.0.smithi084.stdout:3749: expect (ObjNum 1253 snap 425 seq_num 1253) 2024-09-20T22:56:50.422 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:56:50.423 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:56:50.442 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:50.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:50.443 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:56:50.473 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:56:50.544 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:56:50.565 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:50.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:50.567 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:56:50.664 INFO:tasks.rados.rados.0.smithi084.stdout:3752: finishing copy_from to smithi08477203-12 2024-09-20T22:56:50.664 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 12 v 2842 (ObjNum 1202 snap 411 seq_num 1202) dirty exists 2024-09-20T22:56:50.664 INFO:tasks.rados.rados.0.smithi084.stdout:3754: expect (ObjNum 1128 snap 381 seq_num 1128) 2024-09-20T22:56:50.667 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:56:50.683 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:50 smithi084 ceph-mon[40024]: 4.3 scrub starts 2024-09-20T22:56:50.683 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:50 smithi084 ceph-mon[40024]: 4.3 scrub ok 2024-09-20T22:56:50.683 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:50 smithi084 ceph-mon[40024]: 5.c scrub starts 2024-09-20T22:56:50.683 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:50 smithi084 ceph-mon[40024]: 5.c scrub ok 2024-09-20T22:56:50.683 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:50 smithi084 ceph-mon[29774]: 4.3 scrub starts 2024-09-20T22:56:50.683 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:50 smithi084 ceph-mon[29774]: 4.3 scrub ok 2024-09-20T22:56:50.683 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:50 smithi084 ceph-mon[29774]: 5.c scrub starts 2024-09-20T22:56:50.683 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:50 smithi084 ceph-mon[29774]: 5.c scrub ok 2024-09-20T22:56:50.691 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:50.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:50.692 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:56:50.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:50 smithi154 ceph-mon[33264]: 4.3 scrub starts 2024-09-20T22:56:50.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:50 smithi154 ceph-mon[33264]: 4.3 scrub ok 2024-09-20T22:56:50.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:50 smithi154 ceph-mon[33264]: 5.c scrub starts 2024-09-20T22:56:50.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:50 smithi154 ceph-mon[33264]: 5.c scrub ok 2024-09-20T22:56:50.793 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:56:50.814 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:50.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:50.815 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:56:50.916 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:56:50.924 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:56:50.937 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:50.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:50.938 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:56:51.038 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:56:51.060 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:51.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:51.061 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:56:51.161 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:56:51.182 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:51.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:51.183 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:56:51.283 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:56:51.304 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:51.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:51.305 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:56:51.406 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:56:51.411 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is set 2024-09-20T22:56:51.417 INFO:tasks.rados.rados.0.smithi084.stdout:3747: done (7 left) 2024-09-20T22:56:51.417 INFO:tasks.rados.rados.0.smithi084.stdout:3748: done (6 left) 2024-09-20T22:56:51.417 INFO:tasks.rados.rados.0.smithi084.stdout:3749: done (5 left) 2024-09-20T22:56:51.417 INFO:tasks.rados.rados.0.smithi084.stdout:3750: done (4 left) 2024-09-20T22:56:51.417 INFO:tasks.rados.rados.0.smithi084.stdout:3751: done (3 left) 2024-09-20T22:56:51.417 INFO:tasks.rados.rados.0.smithi084.stdout:3752: done (2 left) 2024-09-20T22:56:51.417 INFO:tasks.rados.rados.0.smithi084.stdout:3754: done (1 left) 2024-09-20T22:56:51.417 INFO:tasks.rados.rados.0.smithi084.stdout:3756: done (0 left) 2024-09-20T22:56:51.417 INFO:tasks.rados.rados.0.smithi084.stdout:3757: delete oid 30 current snap is 434 2024-09-20T22:56:51.417 INFO:teuthology.orchestra.run.smithi084.stderr:set pool 7 pg_num to 26 2024-09-20T22:56:51.421 INFO:tasks.rados.rados.0.smithi084.stdout:3757: done (0 left) 2024-09-20T22:56:51.421 INFO:tasks.rados.rados.0.smithi084.stdout:3758: snap_create 2024-09-20T22:56:51.428 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:51.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:51.429 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:56:51.530 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:56:51.551 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:51.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:51.552 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:56:51.577 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:56:51.578 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:56:51.578 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:51 smithi084 ceph-mon[29774]: 6.16 scrub starts 2024-09-20T22:56:51.578 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:51 smithi084 ceph-mon[29774]: 6.16 scrub ok 2024-09-20T22:56:51.578 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:51 smithi084 ceph-mon[29774]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:56:51.578 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:51 smithi084 ceph-mon[29774]: pgmap v1296: 158 pgs: 2 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 152 active+clean; 680 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 11 MiB/s rd, 5.5 MiB/s wr, 27 op/s 2024-09-20T22:56:51.578 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:51 smithi084 ceph-mon[29774]: osdmap e930: 8 total, 8 up, 7 in 2024-09-20T22:56:51.579 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:51 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/877279792' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"}]: dispatch 2024-09-20T22:56:51.579 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:51 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"}]: dispatch 2024-09-20T22:56:51.579 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:51 smithi084 ceph-mon[29774]: 6.5 scrub starts 2024-09-20T22:56:51.579 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:51 smithi084 ceph-mon[29774]: 6.5 scrub ok 2024-09-20T22:56:51.579 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:51 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/3146486030' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-20T22:56:51.579 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:51 smithi084 ceph-mon[40024]: 6.16 scrub starts 2024-09-20T22:56:51.580 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:51 smithi084 ceph-mon[40024]: 6.16 scrub ok 2024-09-20T22:56:51.580 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:51 smithi084 ceph-mon[40024]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:56:51.580 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:51 smithi084 ceph-mon[40024]: pgmap v1296: 158 pgs: 2 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 152 active+clean; 680 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 11 MiB/s rd, 5.5 MiB/s wr, 27 op/s 2024-09-20T22:56:51.580 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:51 smithi084 ceph-mon[40024]: osdmap e930: 8 total, 8 up, 7 in 2024-09-20T22:56:51.580 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:51 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/877279792' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"}]: dispatch 2024-09-20T22:56:51.580 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:51 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"}]: dispatch 2024-09-20T22:56:51.581 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:51 smithi084 ceph-mon[40024]: 6.5 scrub starts 2024-09-20T22:56:51.581 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:51 smithi084 ceph-mon[40024]: 6.5 scrub ok 2024-09-20T22:56:51.581 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:51 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/3146486030' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-20T22:56:51.582 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:56:51.582 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:56:51.600 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:56:51.600 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:56:51.616 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-20T22:56:51.616 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T22:56:51.633 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:56:51.634 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:56:51.650 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:56:51.650 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:56:51.653 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:56:51.667 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:56:51.667 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:56:51.674 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:51.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:51.675 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:56:51.683 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:56:51.684 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:56:51.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:51 smithi154 ceph-mon[33264]: 6.16 scrub starts 2024-09-20T22:56:51.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:51 smithi154 ceph-mon[33264]: 6.16 scrub ok 2024-09-20T22:56:51.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:51 smithi154 ceph-mon[33264]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:56:51.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:51 smithi154 ceph-mon[33264]: pgmap v1296: 158 pgs: 2 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 152 active+clean; 680 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 11 MiB/s rd, 5.5 MiB/s wr, 27 op/s 2024-09-20T22:56:51.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:51 smithi154 ceph-mon[33264]: osdmap e930: 8 total, 8 up, 7 in 2024-09-20T22:56:51.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:51 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/877279792' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"}]: dispatch 2024-09-20T22:56:51.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:51 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"}]: dispatch 2024-09-20T22:56:51.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:51 smithi154 ceph-mon[33264]: 6.5 scrub starts 2024-09-20T22:56:51.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:51 smithi154 ceph-mon[33264]: 6.5 scrub ok 2024-09-20T22:56:51.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:51 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/3146486030' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-20T22:56:51.777 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:56:51.798 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-20T22:56:51.798 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:56:51.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:51.799 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:51.799 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:56:51.900 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:56:51.922 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:51.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:51.923 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:56:52.024 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:56:52.045 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:52.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:52.046 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:56:52.147 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:56:52.169 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:52.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:52.170 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:56:52.271 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:56:52.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:52.293 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:52.294 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:56:52.394 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:56:52.416 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:52.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:52.417 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:56:52.437 INFO:tasks.rados.rados.0.smithi084.stdout:3758: done (0 left) 2024-09-20T22:56:52.438 INFO:tasks.rados.rados.0.smithi084.stdout:3759: delete oid 28 current snap is 435 2024-09-20T22:56:52.478 INFO:tasks.rados.rados.0.smithi084.stdout:3759: done (0 left) 2024-09-20T22:56:52.478 INFO:tasks.rados.rados.0.smithi084.stdout:3760: copy_from oid 39 from oid 1 current snap is 435 2024-09-20T22:56:52.478 INFO:tasks.rados.rados.0.smithi084.stdout:3761: copy_from oid 20 from oid 35 current snap is 435 2024-09-20T22:56:52.478 INFO:tasks.rados.rados.0.smithi084.stdout:3762: snap_create 2024-09-20T22:56:52.490 INFO:tasks.rados.rados.0.smithi084.stdout:3760: finishing copy_from racing read to smithi08477203-39 2024-09-20T22:56:52.493 INFO:tasks.rados.rados.0.smithi084.stdout:3760: finishing copy_from to smithi08477203-39 2024-09-20T22:56:52.493 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 39 v 2668 (ObjNum 738 snap 235 seq_num 738) dirty exists 2024-09-20T22:56:52.518 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:56:52.538 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:52.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:52.539 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:56:52.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:52 smithi084 ceph-mon[40024]: 3.a scrub starts 2024-09-20T22:56:52.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:52 smithi084 ceph-mon[40024]: 3.a scrub ok 2024-09-20T22:56:52.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:52 smithi084 ceph-mon[40024]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:56:52.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:52 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"}]': finished 2024-09-20T22:56:52.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:52 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/3146486030' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T22:56:52.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:52 smithi084 ceph-mon[40024]: osdmap e931: 8 total, 8 up, 7 in 2024-09-20T22:56:52.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:52 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/1451583452' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:56:52.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:52 smithi084 ceph-mon[40024]: 2.2 scrub starts 2024-09-20T22:56:52.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:52 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-09-20T22:56:52.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:52 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-09-20T22:56:52.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:52 smithi084 ceph-mon[29774]: 3.a scrub starts 2024-09-20T22:56:52.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:52 smithi084 ceph-mon[29774]: 3.a scrub ok 2024-09-20T22:56:52.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:52 smithi084 ceph-mon[29774]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:56:52.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:52 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"}]': finished 2024-09-20T22:56:52.617 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:52 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/3146486030' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T22:56:52.617 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:52 smithi084 ceph-mon[29774]: osdmap e931: 8 total, 8 up, 7 in 2024-09-20T22:56:52.617 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:52 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/1451583452' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:56:52.617 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:52 smithi084 ceph-mon[29774]: 2.2 scrub starts 2024-09-20T22:56:52.617 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:52 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-09-20T22:56:52.617 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:52 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-09-20T22:56:52.640 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:56:52.662 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:52.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:52.663 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:56:52.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:52 smithi154 ceph-mon[33264]: 3.a scrub starts 2024-09-20T22:56:52.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:52 smithi154 ceph-mon[33264]: 3.a scrub ok 2024-09-20T22:56:52.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:52 smithi154 ceph-mon[33264]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:56:52.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:52 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"}]': finished 2024-09-20T22:56:52.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:52 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/3146486030' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T22:56:52.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:52 smithi154 ceph-mon[33264]: osdmap e931: 8 total, 8 up, 7 in 2024-09-20T22:56:52.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:52 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/1451583452' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:56:52.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:52 smithi154 ceph-mon[33264]: 2.2 scrub starts 2024-09-20T22:56:52.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:52 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-09-20T22:56:52.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:52 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-09-20T22:56:52.763 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:56:52.784 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:52.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:52.785 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:56:52.886 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:56:52.907 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:52.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:52.908 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:56:53.009 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:56:53.030 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:53.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:53.031 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:56:53.132 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:56:53.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:53.157 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:53.158 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:56:53.258 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:56:53.283 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:53.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:53.284 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:56:53.360 INFO:tasks.rados.rados.0.smithi084.stdout:3760: done (2 left) 2024-09-20T22:56:53.360 INFO:tasks.rados.rados.0.smithi084.stdout:3762: done (1 left) 2024-09-20T22:56:53.361 INFO:tasks.rados.rados.0.smithi084.stdout:3763: write oid 22 current snap is 436 2024-09-20T22:56:53.361 INFO:tasks.rados.rados.0.smithi084.stdout:3763: seq_num 1281 ranges {729767=741273,1927904=725109,3145089=384678} 2024-09-20T22:56:53.365 INFO:tasks.rados.rados.0.smithi084.stdout:3763: writing smithi08477203-22 from 729767 to 1471040 tid 1 2024-09-20T22:56:53.373 INFO:tasks.rados.rados.0.smithi084.stdout:3763: writing smithi08477203-22 from 1927904 to 2653013 tid 2 2024-09-20T22:56:53.379 INFO:tasks.rados.rados.0.smithi084.stdout:3763: writing smithi08477203-22 from 3145089 to 3529767 tid 3 2024-09-20T22:56:53.379 INFO:tasks.rados.rados.0.smithi084.stdout:3764: snap_remove snap 391 2024-09-20T22:56:53.385 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:56:53.409 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:53.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:53.410 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:56:53.511 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:56:53.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:53.535 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:53.535 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:56:53.636 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:56:53.658 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:53.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:53.659 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:56:53.688 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:53 smithi084 ceph-mon[40024]: pgmap v1299: 158 pgs: 2 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 152 active+clean; 680 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 11 MiB/s rd, 5.5 MiB/s wr, 27 op/s 2024-09-20T22:56:53.688 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:53 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"}]': finished 2024-09-20T22:56:53.689 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:53 smithi084 ceph-mon[40024]: osdmap e932: 8 total, 8 up, 7 in 2024-09-20T22:56:53.689 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:53 smithi084 ceph-mon[40024]: osdmap e933: 8 total, 8 up, 7 in 2024-09-20T22:56:53.689 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:53 smithi084 ceph-mon[29774]: pgmap v1299: 158 pgs: 2 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 152 active+clean; 680 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 11 MiB/s rd, 5.5 MiB/s wr, 27 op/s 2024-09-20T22:56:53.689 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:53 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"}]': finished 2024-09-20T22:56:53.689 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:53 smithi084 ceph-mon[29774]: osdmap e932: 8 total, 8 up, 7 in 2024-09-20T22:56:53.690 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:53 smithi084 ceph-mon[29774]: osdmap e933: 8 total, 8 up, 7 in 2024-09-20T22:56:53.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:53 smithi154 ceph-mon[33264]: pgmap v1299: 158 pgs: 2 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 152 active+clean; 680 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 11 MiB/s rd, 5.5 MiB/s wr, 27 op/s 2024-09-20T22:56:53.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:53 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"}]': finished 2024-09-20T22:56:53.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:53 smithi154 ceph-mon[33264]: osdmap e932: 8 total, 8 up, 7 in 2024-09-20T22:56:53.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:53 smithi154 ceph-mon[33264]: osdmap e933: 8 total, 8 up, 7 in 2024-09-20T22:56:53.760 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:56:53.781 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:53.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:53.782 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:56:53.883 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:56:53.904 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:53.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:53.906 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:56:54.006 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:56:54.028 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:54.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:54.029 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:56:54.130 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:56:54.151 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:54.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:54.152 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:56:54.253 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:56:54.272 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.3 dump_historic_ops 2024-09-20T22:56:54.274 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:54.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:54.275 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:56:54.356 INFO:tasks.rados.rados.0.smithi084.stdout:3764: done (2 left) 2024-09-20T22:56:54.356 INFO:tasks.rados.rados.0.smithi084.stdout:3765: write oid 28 current snap is 436 2024-09-20T22:56:54.356 INFO:tasks.rados.rados.0.smithi084.stdout:3765: seq_num 1282 ranges {746574=416522,1586386=468767,2772512=629544} 2024-09-20T22:56:54.365 INFO:tasks.rados.rados.0.smithi084.stdout:3765: writing smithi08477203-28 from 746574 to 1163096 tid 1 2024-09-20T22:56:54.371 INFO:tasks.rados.rados.0.smithi084.stdout:3765: writing smithi08477203-28 from 1586386 to 2055153 tid 2 2024-09-20T22:56:54.377 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:56:54.381 INFO:tasks.rados.rados.0.smithi084.stdout:3765: writing smithi08477203-28 from 2772512 to 3402056 tid 3 2024-09-20T22:56:54.381 INFO:tasks.rados.rados.0.smithi084.stdout:3766: rollback oid 30 current snap is 436 2024-09-20T22:56:54.381 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 30 to 420 2024-09-20T22:56:54.381 INFO:tasks.rados.rados.0.smithi084.stdout:3763: finishing write tid 1 to smithi08477203-22 2024-09-20T22:56:54.381 INFO:tasks.rados.rados.0.smithi084.stdout:3763: finishing write tid 2 to smithi08477203-22 2024-09-20T22:56:54.381 INFO:tasks.rados.rados.0.smithi084.stdout:3763: finishing write tid 3 to smithi08477203-22 2024-09-20T22:56:54.381 INFO:tasks.rados.rados.0.smithi084.stdout:3763: finishing write tid 4 to smithi08477203-22 2024-09-20T22:56:54.381 INFO:tasks.rados.rados.0.smithi084.stdout:3763: finishing write tid 5 to smithi08477203-22 2024-09-20T22:56:54.382 INFO:tasks.rados.rados.0.smithi084.stdout:3763: oid 22 updating version 0 to 2503 2024-09-20T22:56:54.382 INFO:tasks.rados.rados.0.smithi084.stdout:3763: oid 22 updating version 2503 to 2504 2024-09-20T22:56:54.382 INFO:tasks.rados.rados.0.smithi084.stdout:3763: oid 22 updating version 2504 to 2506 2024-09-20T22:56:54.382 INFO:tasks.rados.rados.0.smithi084.stdout:3763: oid 22 version 2506 is already newer than 2505 2024-09-20T22:56:54.382 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 22 v 2506 (ObjNum 1281 snap 436 seq_num 1281) dirty exists 2024-09-20T22:56:54.382 INFO:tasks.rados.rados.0.smithi084.stdout:3763: left oid 22 (ObjNum 1281 snap 436 seq_num 1281) 2024-09-20T22:56:54.382 INFO:tasks.rados.rados.0.smithi084.stdout:3761: finishing copy_from racing read to smithi08477203-20 2024-09-20T22:56:54.382 INFO:tasks.rados.rados.0.smithi084.stdout:3761: finishing copy_from to smithi08477203-20 2024-09-20T22:56:54.382 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 20 v 2642 (ObjNum 1121 snap 380 seq_num 1121) dirty exists 2024-09-20T22:56:54.382 INFO:tasks.rados.rados.0.smithi084.stdout:3765: finishing write tid 1 to smithi08477203-28 2024-09-20T22:56:54.382 INFO:tasks.rados.rados.0.smithi084.stdout:3765: finishing write tid 2 to smithi08477203-28 2024-09-20T22:56:54.382 INFO:tasks.rados.rados.0.smithi084.stdout:3761: done (3 left) 2024-09-20T22:56:54.382 INFO:tasks.rados.rados.0.smithi084.stdout:3763: done (2 left) 2024-09-20T22:56:54.383 INFO:tasks.rados.rados.0.smithi084.stdout:3767: read oid 25 snap -1 2024-09-20T22:56:54.383 INFO:tasks.rados.rados.0.smithi084.stdout:3767: expect deleted 2024-09-20T22:56:54.383 INFO:tasks.rados.rados.0.smithi084.stdout:3768: write oid 20 current snap is 436 2024-09-20T22:56:54.383 INFO:tasks.rados.rados.0.smithi084.stdout:3768: seq_num 1283 ranges {763381=491771,2044868=728778,3199935=363446} 2024-09-20T22:56:54.387 INFO:tasks.rados.rados.0.smithi084.stdout:3768: writing smithi08477203-20 from 763381 to 1255152 tid 1 2024-09-20T22:56:54.395 INFO:tasks.rados.rados.0.smithi084.stdout:3768: writing smithi08477203-20 from 2044868 to 2773646 tid 2 2024-09-20T22:56:54.399 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:54.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:54.400 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:56:54.401 INFO:tasks.rados.rados.0.smithi084.stdout:3768: writing smithi08477203-20 from 3199935 to 3563381 tid 3 2024-09-20T22:56:54.401 INFO:tasks.rados.rados.0.smithi084.stdout:3769: rollback oid 31 current snap is 436 2024-09-20T22:56:54.401 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 31 to 423 2024-09-20T22:56:54.401 INFO:tasks.rados.rados.0.smithi084.stdout:3765: finishing write tid 3 to smithi08477203-28 2024-09-20T22:56:54.401 INFO:tasks.rados.rados.0.smithi084.stdout:3765: finishing write tid 4 to smithi08477203-28 2024-09-20T22:56:54.401 INFO:tasks.rados.rados.0.smithi084.stdout:3765: finishing write tid 5 to smithi08477203-28 2024-09-20T22:56:54.402 INFO:tasks.rados.rados.0.smithi084.stdout:3765: oid 28 updating version 0 to 3099 2024-09-20T22:56:54.402 INFO:tasks.rados.rados.0.smithi084.stdout:3765: oid 28 updating version 3099 to 3100 2024-09-20T22:56:54.402 INFO:tasks.rados.rados.0.smithi084.stdout:3765: oid 28 version 3100 is already newer than 3094 2024-09-20T22:56:54.402 INFO:tasks.rados.rados.0.smithi084.stdout:3765: oid 28 version 3100 is already newer than 3093 2024-09-20T22:56:54.402 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 28 v 3100 (ObjNum 1282 snap 436 seq_num 1282) dirty exists 2024-09-20T22:56:54.402 INFO:tasks.rados.rados.0.smithi084.stdout:3765: left oid 28 (ObjNum 1282 snap 436 seq_num 1282) 2024-09-20T22:56:54.402 INFO:tasks.rados.rados.0.smithi084.stdout:3765: done (4 left) 2024-09-20T22:56:54.402 INFO:tasks.rados.rados.0.smithi084.stdout:3767: done (3 left) 2024-09-20T22:56:54.402 INFO:tasks.rados.rados.0.smithi084.stdout:3770: delete oid 8 current snap is 436 2024-09-20T22:56:54.402 INFO:tasks.rados.rados.0.smithi084.stdout:3768: finishing write tid 1 to smithi08477203-20 2024-09-20T22:56:54.402 INFO:tasks.rados.rados.0.smithi084.stdout:3768: finishing write tid 2 to smithi08477203-20 2024-09-20T22:56:54.403 INFO:tasks.rados.rados.0.smithi084.stdout:3770: done (3 left) 2024-09-20T22:56:54.404 INFO:tasks.rados.rados.0.smithi084.stdout:3771: write (excl) oid 14 current snap is 436 2024-09-20T22:56:54.404 INFO:tasks.rados.rados.0.smithi084.stdout:3769: finishing rollback tid 1 to smithi08477203-31 2024-09-20T22:56:54.404 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 31 v 2593 (ObjNum 129 snap 0 seq_num 2091320056) dirty dne 2024-09-20T22:56:54.404 INFO:tasks.rados.rados.0.smithi084.stdout:3771: seq_num 1285 ranges {796995=758622,2161832=732447} 2024-09-20T22:56:54.410 INFO:tasks.rados.rados.0.smithi084.stdout:3771: writing smithi08477203-14 from 796995 to 1555617 tid 1 2024-09-20T22:56:54.419 INFO:tasks.rados.rados.0.smithi084.stdout:3771: writing smithi08477203-14 from 2161832 to 2894279 tid 2 2024-09-20T22:56:54.419 INFO:tasks.rados.rados.0.smithi084.stdout:3769: done (3 left) 2024-09-20T22:56:54.419 INFO:tasks.rados.rados.0.smithi084.stdout:3772: read oid 26 snap 405 2024-09-20T22:56:54.419 INFO:tasks.rados.rados.0.smithi084.stdout:3772: expect (ObjNum 906 snap 297 seq_num 906) 2024-09-20T22:56:54.419 INFO:tasks.rados.rados.0.smithi084.stdout:3773: copy_from oid 41 from oid 39 current snap is 436 2024-09-20T22:56:54.419 INFO:tasks.rados.rados.0.smithi084.stdout:3768: finishing write tid 3 to smithi08477203-20 2024-09-20T22:56:54.419 INFO:tasks.rados.rados.0.smithi084.stdout:3768: finishing write tid 4 to smithi08477203-20 2024-09-20T22:56:54.419 INFO:tasks.rados.rados.0.smithi084.stdout:3768: finishing write tid 5 to smithi08477203-20 2024-09-20T22:56:54.419 INFO:tasks.rados.rados.0.smithi084.stdout:3768: oid 20 updating version 0 to 2646 2024-09-20T22:56:54.419 INFO:tasks.rados.rados.0.smithi084.stdout:3768: oid 20 version 2646 is already newer than 2643 2024-09-20T22:56:54.420 INFO:tasks.rados.rados.0.smithi084.stdout:3768: oid 20 version 2646 is already newer than 2645 2024-09-20T22:56:54.420 INFO:tasks.rados.rados.0.smithi084.stdout:3768: oid 20 version 2646 is already newer than 2644 2024-09-20T22:56:54.420 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 20 v 2646 (ObjNum 1283 snap 436 seq_num 1283) dirty exists 2024-09-20T22:56:54.420 INFO:tasks.rados.rados.0.smithi084.stdout:3768: left oid 20 (ObjNum 1283 snap 436 seq_num 1283) 2024-09-20T22:56:54.420 INFO:tasks.rados.rados.0.smithi084.stdout:3768: done (4 left) 2024-09-20T22:56:54.420 INFO:tasks.rados.rados.0.smithi084.stdout:3774: write oid 35 current snap is 436 2024-09-20T22:56:54.420 INFO:tasks.rados.rados.0.smithi084.stdout:3774: seq_num 1286 ranges {413802=433871,1420314=476105,2482204=790985} 2024-09-20T22:56:54.423 INFO:tasks.rados.rados.0.smithi084.stdout:3774: writing smithi08477203-35 from 413802 to 847673 tid 1 2024-09-20T22:56:54.429 INFO:tasks.rados.rados.0.smithi084.stdout:3774: writing smithi08477203-35 from 1420314 to 1896419 tid 2 2024-09-20T22:56:54.436 INFO:tasks.rados.rados.0.smithi084.stdout:3774: writing smithi08477203-35 from 2482204 to 3273189 tid 3 2024-09-20T22:56:54.436 INFO:tasks.rados.rados.0.smithi084.stdout:3775: write (excl) oid 2 current snap is 436 2024-09-20T22:56:54.436 INFO:tasks.rados.rados.0.smithi084.stdout:3775: seq_num 1287 ranges {430609=509120,1478796=736116,2909627=602257} 2024-09-20T22:56:54.441 INFO:tasks.rados.rados.0.smithi084.stdout:3775: writing smithi08477203-2 from 430609 to 939729 tid 1 2024-09-20T22:56:54.449 INFO:tasks.rados.rados.0.smithi084.stdout:3775: writing smithi08477203-2 from 1478796 to 2214912 tid 2 2024-09-20T22:56:54.456 INFO:tasks.rados.rados.0.smithi084.stdout:3775: writing smithi08477203-2 from 2909627 to 3511884 tid 3 2024-09-20T22:56:54.457 INFO:tasks.rados.rados.0.smithi084.stdout:3776: read oid 27 snap -1 2024-09-20T22:56:54.457 INFO:tasks.rados.rados.0.smithi084.stdout:3776: expect (ObjNum 1246 snap 423 seq_num 1246) 2024-09-20T22:56:54.457 INFO:tasks.rados.rados.0.smithi084.stdout:3777: copy_from oid 3 from oid 6 current snap is 436 2024-09-20T22:56:54.457 INFO:tasks.rados.rados.0.smithi084.stdout:3775: finishing write tid 1 to smithi08477203-2 2024-09-20T22:56:54.457 INFO:tasks.rados.rados.0.smithi084.stdout:3775: finishing write tid 2 to smithi08477203-2 2024-09-20T22:56:54.457 INFO:tasks.rados.rados.0.smithi084.stdout:3778: rollback oid 46 current snap is 436 2024-09-20T22:56:54.457 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 46 to 410 2024-09-20T22:56:54.457 INFO:tasks.rados.rados.0.smithi084.stdout:3779: write (excl) oid 49 current snap is 436 2024-09-20T22:56:54.457 INFO:tasks.rados.rados.0.smithi084.stdout:3779: seq_num 1288 ranges {447416=584369,1537278=479774,2537050=529882,3524784=122632} 2024-09-20T22:56:54.462 INFO:tasks.rados.rados.0.smithi084.stdout:3779: writing smithi08477203-49 from 447416 to 1031785 tid 1 2024-09-20T22:56:54.468 INFO:tasks.rados.rados.0.smithi084.stdout:3779: writing smithi08477203-49 from 1537278 to 2017052 tid 2 2024-09-20T22:56:54.474 INFO:tasks.rados.rados.0.smithi084.stdout:3779: writing smithi08477203-49 from 2537050 to 3066932 tid 3 2024-09-20T22:56:54.477 INFO:tasks.rados.rados.0.smithi084.stdout:3779: writing smithi08477203-49 from 3524784 to 3647416 tid 4 2024-09-20T22:56:54.477 INFO:tasks.rados.rados.0.smithi084.stdout:3780: write oid 32 current snap is 436 2024-09-20T22:56:54.477 INFO:tasks.rados.rados.0.smithi084.stdout:3777: finishing copy_from to smithi08477203-3 2024-09-20T22:56:54.477 INFO:tasks.rados.rados.0.smithi084.stdout:3777: got expected ENOENT (src dne) 2024-09-20T22:56:54.477 INFO:tasks.rados.rados.0.smithi084.stdout:3780: seq_num 1289 ranges {464223=659618,1879407=739785,3248120=416103} 2024-09-20T22:56:54.483 INFO:tasks.rados.rados.0.smithi084.stdout:3780: writing smithi08477203-32 from 464223 to 1123841 tid 1 2024-09-20T22:56:54.492 INFO:tasks.rados.rados.0.smithi084.stdout:3780: writing smithi08477203-32 from 1879407 to 2619192 tid 2 2024-09-20T22:56:54.498 INFO:tasks.rados.rados.0.smithi084.stdout:3780: writing smithi08477203-32 from 3248120 to 3664223 tid 3 2024-09-20T22:56:54.498 INFO:tasks.rados.rados.0.smithi084.stdout:3781: copy_from oid 40 from oid 17 current snap is 436 2024-09-20T22:56:54.498 INFO:tasks.rados.rados.0.smithi084.stdout:3777: finishing copy_from racing read to smithi08477203-3 2024-09-20T22:56:54.498 INFO:tasks.rados.rados.0.smithi084.stdout:3776: expect (ObjNum 1246 snap 423 seq_num 1246) 2024-09-20T22:56:54.501 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:56:54.521 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:54.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:54.522 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:56:54.614 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:54 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:56:54.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:54 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:56:54.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:54 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-09-20T22:56:54.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:54 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-09-20T22:56:54.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:54 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"}]': finished 2024-09-20T22:56:54.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:54 smithi084 ceph-mon[40024]: osdmap e934: 8 total, 8 up, 7 in 2024-09-20T22:56:54.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:54 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:56:54.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:54 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:56:54.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:54 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-09-20T22:56:54.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:54 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-09-20T22:56:54.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:54 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"}]': finished 2024-09-20T22:56:54.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:54 smithi084 ceph-mon[29774]: osdmap e934: 8 total, 8 up, 7 in 2024-09-20T22:56:54.623 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:56:54.643 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:54.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:54.644 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:56:54.700 INFO:tasks.rados.rados.0.smithi084.stdout:3775: finishing write tid 3 to smithi08477203-2 2024-09-20T22:56:54.700 INFO:tasks.rados.rados.0.smithi084.stdout:3775: finishing write tid 4 to smithi08477203-2 2024-09-20T22:56:54.700 INFO:tasks.rados.rados.0.smithi084.stdout:3775: finishing write tid 5 to smithi08477203-2 2024-09-20T22:56:54.700 INFO:tasks.rados.rados.0.smithi084.stdout:3775: oid 2 updating version 0 to 2600 2024-09-20T22:56:54.700 INFO:tasks.rados.rados.0.smithi084.stdout:3775: oid 2 version 2600 is already newer than 2598 2024-09-20T22:56:54.700 INFO:tasks.rados.rados.0.smithi084.stdout:3775: oid 2 updating version 2600 to 2601 2024-09-20T22:56:54.700 INFO:tasks.rados.rados.0.smithi084.stdout:3775: oid 2 version 2601 is already newer than 2599 2024-09-20T22:56:54.700 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 2 v 2601 (ObjNum 1287 snap 436 seq_num 1287) dirty exists 2024-09-20T22:56:54.701 INFO:tasks.rados.rados.0.smithi084.stdout:3775: left oid 2 (ObjNum 1287 snap 436 seq_num 1287) 2024-09-20T22:56:54.701 INFO:tasks.rados.rados.0.smithi084.stdout:3775: done (11 left) 2024-09-20T22:56:54.701 INFO:tasks.rados.rados.0.smithi084.stdout:3776: done (10 left) 2024-09-20T22:56:54.701 INFO:tasks.rados.rados.0.smithi084.stdout:3777: done (9 left) 2024-09-20T22:56:54.701 INFO:tasks.rados.rados.0.smithi084.stdout:3782: snap_create 2024-09-20T22:56:54.701 INFO:tasks.rados.rados.0.smithi084.stdout:3781: finishing copy_from to smithi08477203-40 2024-09-20T22:56:54.701 INFO:tasks.rados.rados.0.smithi084.stdout:3781: got expected ENOENT (src dne) 2024-09-20T22:56:54.701 INFO:tasks.rados.rados.0.smithi084.stdout:3781: finishing copy_from racing read to smithi08477203-40 2024-09-20T22:56:54.745 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:56:54.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:54 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:56:54.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:54 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:56:54.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:54 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-09-20T22:56:54.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:54 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-09-20T22:56:54.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:54 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"}]': finished 2024-09-20T22:56:54.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:54 smithi154 ceph-mon[33264]: osdmap e934: 8 total, 8 up, 7 in 2024-09-20T22:56:54.767 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:54.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:54.767 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:56:54.829 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-20T22:56:54.868 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:56:54.889 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:54.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:54.890 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:56:54.978 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:56:54.991 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:56:55.013 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:55.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:55.014 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:56:55.115 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:56:55.136 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:55.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:55.137 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:56:55.238 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:56:55.260 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:55.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:55.261 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:56:55.361 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:56:55.362 INFO:tasks.rados.rados.0.smithi084.stdout:3781: done (9 left) 2024-09-20T22:56:55.362 INFO:tasks.rados.rados.0.smithi084.stdout:3782: done (8 left) 2024-09-20T22:56:55.362 INFO:tasks.rados.rados.0.smithi084.stdout:3783: write (excl) oid 28 current snap is 437 2024-09-20T22:56:55.363 INFO:tasks.rados.rados.0.smithi084.stdout:3783: seq_num 1290 ranges {481030=734867,1937889=483443,2875543=552426} 2024-09-20T22:56:55.368 INFO:tasks.rados.rados.0.smithi084.stdout:3783: writing smithi08477203-28 from 481030 to 1215897 tid 1 2024-09-20T22:56:55.376 INFO:tasks.rados.rados.0.smithi084.stdout:3783: writing smithi08477203-28 from 1937889 to 2421332 tid 2 2024-09-20T22:56:55.382 INFO:tasks.rados.rados.0.smithi084.stdout:3783: writing smithi08477203-28 from 2875543 to 3427969 tid 3 2024-09-20T22:56:55.382 INFO:tasks.rados.rados.0.smithi084.stdout:3784: rollback oid 6 current snap is 437 2024-09-20T22:56:55.382 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 6 to 435 2024-09-20T22:56:55.382 INFO:tasks.rados.rados.0.smithi084.stdout:3783: finishing write tid 1 to smithi08477203-28 2024-09-20T22:56:55.382 INFO:tasks.rados.rados.0.smithi084.stdout:3783: finishing write tid 2 to smithi08477203-28 2024-09-20T22:56:55.382 INFO:tasks.rados.rados.0.smithi084.stdout:3785: snap_remove snap 379 2024-09-20T22:56:55.384 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:55.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:55.386 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:56:55.484 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-20T22:56:55.484 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:56:55.486 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:56:55.508 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:55.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:55.510 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:56:55.611 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:56:55.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:55.633 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:55.633 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:56:55.719 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-20T22:56:55.734 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:56:55.741 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:55 smithi084 ceph-mon[40024]: pgmap v1302: 168 pgs: 5 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 159 active+clean; 696 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 9.0 MiB/s rd, 3.3 MiB/s wr, 17 op/s 2024-09-20T22:56:55.741 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:55 smithi084 ceph-mon[40024]: osdmap e935: 8 total, 8 up, 7 in 2024-09-20T22:56:55.741 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:55 smithi084 ceph-mon[29774]: pgmap v1302: 168 pgs: 5 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 159 active+clean; 696 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 9.0 MiB/s rd, 3.3 MiB/s wr, 17 op/s 2024-09-20T22:56:55.741 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:55 smithi084 ceph-mon[29774]: osdmap e935: 8 total, 8 up, 7 in 2024-09-20T22:56:55.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:55 smithi154 ceph-mon[33264]: pgmap v1302: 168 pgs: 5 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 159 active+clean; 696 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 9.0 MiB/s rd, 3.3 MiB/s wr, 17 op/s 2024-09-20T22:56:55.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:55 smithi154 ceph-mon[33264]: osdmap e935: 8 total, 8 up, 7 in 2024-09-20T22:56:55.755 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:55.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:55.756 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:56:55.857 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:56:55.878 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:55.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:55.879 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:56:55.980 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:56:56.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:56.002 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:56.002 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:56:56.103 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:56:56.108 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-20T22:56:56.124 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:56.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:56.125 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:56:56.127 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:56:56.226 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:56:56.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:56.248 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:56.248 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:56:56.349 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:56:56.353 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:56:56.362 INFO:tasks.rados.rados.0.smithi084.stdout:3785: done (10 left) 2024-09-20T22:56:56.362 INFO:tasks.rados.rados.0.smithi084.stdout:3786: copy_from oid 24 from oid 43 current snap is 437 2024-09-20T22:56:56.362 INFO:tasks.rados.rados.0.smithi084.stdout:3787: snap_create 2024-09-20T22:56:56.363 INFO:tasks.rados.rados.0.smithi084.stdout:3784: finishing rollback tid 1 to smithi08477203-6 2024-09-20T22:56:56.363 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 6 v 2652 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-09-20T22:56:56.363 INFO:tasks.rados.rados.0.smithi084.stdout:3783: finishing write tid 3 to smithi08477203-28 2024-09-20T22:56:56.363 INFO:tasks.rados.rados.0.smithi084.stdout:3783: finishing write tid 4 to smithi08477203-28 2024-09-20T22:56:56.363 INFO:tasks.rados.rados.0.smithi084.stdout:3783: finishing write tid 5 to smithi08477203-28 2024-09-20T22:56:56.363 INFO:tasks.rados.rados.0.smithi084.stdout:3783: oid 28 updating version 0 to 3104 2024-09-20T22:56:56.363 INFO:tasks.rados.rados.0.smithi084.stdout:3783: oid 28 updating version 3104 to 3105 2024-09-20T22:56:56.363 INFO:tasks.rados.rados.0.smithi084.stdout:3783: oid 28 updating version 3105 to 3107 2024-09-20T22:56:56.363 INFO:tasks.rados.rados.0.smithi084.stdout:3783: oid 28 version 3107 is already newer than 3106 2024-09-20T22:56:56.363 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 28 v 3107 (ObjNum 1290 snap 437 seq_num 1290) dirty exists 2024-09-20T22:56:56.363 INFO:tasks.rados.rados.0.smithi084.stdout:3783: left oid 28 (ObjNum 1290 snap 437 seq_num 1290) 2024-09-20T22:56:56.379 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:56.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:56.381 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:56:56.420 INFO:tasks.rados.rados.0.smithi084.stdout:3779: finishing write tid 1 to smithi08477203-49 2024-09-20T22:56:56.420 INFO:tasks.rados.rados.0.smithi084.stdout:3779: finishing write tid 2 to smithi08477203-49 2024-09-20T22:56:56.421 INFO:tasks.rados.rados.0.smithi084.stdout:3779: finishing write tid 3 to smithi08477203-49 2024-09-20T22:56:56.421 INFO:tasks.rados.rados.0.smithi084.stdout:3779: finishing write tid 4 to smithi08477203-49 2024-09-20T22:56:56.421 INFO:tasks.rados.rados.0.smithi084.stdout:3779: finishing write tid 5 to smithi08477203-49 2024-09-20T22:56:56.422 INFO:tasks.rados.rados.0.smithi084.stdout:3779: finishing write tid 6 to smithi08477203-49 2024-09-20T22:56:56.422 INFO:tasks.rados.rados.0.smithi084.stdout:3779: oid 49 updating version 0 to 2601 2024-09-20T22:56:56.422 INFO:tasks.rados.rados.0.smithi084.stdout:3779: oid 49 version 2601 is already newer than 2598 2024-09-20T22:56:56.422 INFO:tasks.rados.rados.0.smithi084.stdout:3779: oid 49 version 2601 is already newer than 2600 2024-09-20T22:56:56.422 INFO:tasks.rados.rados.0.smithi084.stdout:3779: oid 49 updating version 2601 to 2602 2024-09-20T22:56:56.422 INFO:tasks.rados.rados.0.smithi084.stdout:3779: oid 49 version 2602 is already newer than 2599 2024-09-20T22:56:56.423 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 49 v 2602 (ObjNum 1288 snap 436 seq_num 1288) dirty exists 2024-09-20T22:56:56.423 INFO:tasks.rados.rados.0.smithi084.stdout:3779: left oid 49 (ObjNum 1288 snap 436 seq_num 1288) 2024-09-20T22:56:56.425 INFO:tasks.rados.rados.0.smithi084.stdout:3786: finishing copy_from racing read to smithi08477203-24 2024-09-20T22:56:56.427 INFO:tasks.rados.rados.0.smithi084.stdout:3774: finishing write tid 1 to smithi08477203-35 2024-09-20T22:56:56.437 INFO:tasks.rados.rados.0.smithi084.stdout:3786: finishing copy_from to smithi08477203-24 2024-09-20T22:56:56.437 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 24 v 1952 (ObjNum 1259 snap 430 seq_num 1259) dirty exists 2024-09-20T22:56:56.438 INFO:tasks.rados.rados.0.smithi084.stdout:3774: finishing write tid 2 to smithi08477203-35 2024-09-20T22:56:56.438 INFO:tasks.rados.rados.0.smithi084.stdout:3774: finishing write tid 3 to smithi08477203-35 2024-09-20T22:56:56.439 INFO:tasks.rados.rados.0.smithi084.stdout:3780: finishing write tid 1 to smithi08477203-32 2024-09-20T22:56:56.440 INFO:tasks.rados.rados.0.smithi084.stdout:3780: finishing write tid 2 to smithi08477203-32 2024-09-20T22:56:56.440 INFO:tasks.rados.rados.0.smithi084.stdout:3780: finishing write tid 3 to smithi08477203-32 2024-09-20T22:56:56.440 INFO:tasks.rados.rados.0.smithi084.stdout:3774: finishing write tid 4 to smithi08477203-35 2024-09-20T22:56:56.440 INFO:tasks.rados.rados.0.smithi084.stdout:3771: finishing write tid 1 to smithi08477203-14 2024-09-20T22:56:56.440 INFO:tasks.rados.rados.0.smithi084.stdout:3774: finishing write tid 5 to smithi08477203-35 2024-09-20T22:56:56.440 INFO:tasks.rados.rados.0.smithi084.stdout:3774: oid 35 updating version 0 to 2666 2024-09-20T22:56:56.440 INFO:tasks.rados.rados.0.smithi084.stdout:3774: oid 35 updating version 2666 to 2668 2024-09-20T22:56:56.440 INFO:tasks.rados.rados.0.smithi084.stdout:3774: oid 35 version 2668 is already newer than 2667 2024-09-20T22:56:56.441 INFO:tasks.rados.rados.0.smithi084.stdout:3774: oid 35 updating version 2668 to 2669 2024-09-20T22:56:56.441 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 35 v 2669 (ObjNum 1286 snap 436 seq_num 1286) dirty exists 2024-09-20T22:56:56.441 INFO:tasks.rados.rados.0.smithi084.stdout:3774: left oid 35 (ObjNum 1286 snap 436 seq_num 1286) 2024-09-20T22:56:56.442 INFO:tasks.rados.rados.0.smithi084.stdout:3771: finishing write tid 2 to smithi08477203-14 2024-09-20T22:56:56.446 INFO:tasks.rados.rados.0.smithi084.stdout:3771: finishing write tid 3 to smithi08477203-14 2024-09-20T22:56:56.447 INFO:tasks.rados.rados.0.smithi084.stdout:3771: finishing write tid 4 to smithi08477203-14 2024-09-20T22:56:56.447 INFO:tasks.rados.rados.0.smithi084.stdout:3771: oid 14 updating version 0 to 2591 2024-09-20T22:56:56.447 INFO:tasks.rados.rados.0.smithi084.stdout:3771: oid 14 updating version 2591 to 2593 2024-09-20T22:56:56.447 INFO:tasks.rados.rados.0.smithi084.stdout:3771: oid 14 version 2593 is already newer than 2592 2024-09-20T22:56:56.447 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 14 v 2593 (ObjNum 1285 snap 436 seq_num 1285) dirty exists 2024-09-20T22:56:56.447 INFO:tasks.rados.rados.0.smithi084.stdout:3771: left oid 14 (ObjNum 1285 snap 436 seq_num 1285) 2024-09-20T22:56:56.447 INFO:tasks.rados.rados.0.smithi084.stdout:3780: finishing write tid 4 to smithi08477203-32 2024-09-20T22:56:56.448 INFO:tasks.rados.rados.0.smithi084.stdout:3780: finishing write tid 5 to smithi08477203-32 2024-09-20T22:56:56.448 INFO:tasks.rados.rados.0.smithi084.stdout:3780: oid 32 updating version 0 to 1860 2024-09-20T22:56:56.448 INFO:tasks.rados.rados.0.smithi084.stdout:3780: oid 32 version 1860 is already newer than 1858 2024-09-20T22:56:56.448 INFO:tasks.rados.rados.0.smithi084.stdout:3780: oid 32 updating version 1860 to 1861 2024-09-20T22:56:56.448 INFO:tasks.rados.rados.0.smithi084.stdout:3780: oid 32 version 1861 is already newer than 1859 2024-09-20T22:56:56.448 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 32 v 1861 (ObjNum 1289 snap 436 seq_num 1289) dirty exists 2024-09-20T22:56:56.448 INFO:tasks.rados.rados.0.smithi084.stdout:3780: left oid 32 (ObjNum 1289 snap 436 seq_num 1289) 2024-09-20T22:56:56.452 INFO:tasks.rados.rados.0.smithi084.stdout:3766: finishing rollback tid 1 to smithi08477203-30 2024-09-20T22:56:56.453 INFO:tasks.rados.rados.0.smithi084.stdout:3772: expect (ObjNum 906 snap 297 seq_num 906) 2024-09-20T22:56:56.481 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:56:56.512 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:56.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:56.513 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:56:56.614 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:56:56.640 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:56:56.643 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:56.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:56.644 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:56:56.677 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:56:56.677 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:56:56.691 INFO:tasks.rados.rados.0.smithi084.stdout:3773: finishing copy_from racing read to smithi08477203-41 2024-09-20T22:56:56.691 INFO:tasks.rados.rados.0.smithi084.stdout:3773: finishing copy_from to smithi08477203-41 2024-09-20T22:56:56.691 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 41 v 2668 (ObjNum 738 snap 235 seq_num 738) dirty exists 2024-09-20T22:56:56.691 INFO:tasks.rados.rados.0.smithi084.stdout:3766: finishing rollback tid 2 to smithi08477203-30 2024-09-20T22:56:56.691 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 30 v 2628 (ObjNum 1149 snap 394 seq_num 1149) dirty exists 2024-09-20T22:56:56.691 INFO:tasks.rados.rados.0.smithi084.stdout:3778: finishing rollback tid 0 to smithi08477203-46 2024-09-20T22:56:56.691 INFO:tasks.rados.rados.0.smithi084.stdout:3778: finishing rollback tid 1 to smithi08477203-46 2024-09-20T22:56:56.691 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 46 v 2630 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-09-20T22:56:56.701 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:56:56.701 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:56:56.718 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:56:56.718 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:56:56.738 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-20T22:56:56.738 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T22:56:56.744 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:56:56.762 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:56:56.762 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:56:56.771 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:56.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:56.772 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:56:56.784 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:56:56.785 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:56:56.802 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:56:56.803 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:56:56.822 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:56:56.822 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:56:56.874 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:56:56.894 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:56.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:56.895 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:56:56.905 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-20T22:56:56.906 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [], 2024-09-20T22:56:56.906 INFO:teuthology.orchestra.run.smithi154.stdout: "num_ops": 0 2024-09-20T22:56:56.906 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-20T22:56:56.996 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:56:57.018 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:57.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:57.019 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:56:57.119 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:56:57.140 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:57.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:57.141 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:56:57.242 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:56:57.263 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:57.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:57.264 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:56:57.362 INFO:tasks.rados.rados.0.smithi084.stdout:3766: done (11 left) 2024-09-20T22:56:57.363 INFO:tasks.rados.rados.0.smithi084.stdout:3771: done (10 left) 2024-09-20T22:56:57.363 INFO:tasks.rados.rados.0.smithi084.stdout:3772: done (9 left) 2024-09-20T22:56:57.363 INFO:tasks.rados.rados.0.smithi084.stdout:3773: done (8 left) 2024-09-20T22:56:57.363 INFO:tasks.rados.rados.0.smithi084.stdout:3774: done (7 left) 2024-09-20T22:56:57.363 INFO:tasks.rados.rados.0.smithi084.stdout:3778: done (6 left) 2024-09-20T22:56:57.363 INFO:tasks.rados.rados.0.smithi084.stdout:3779: done (5 left) 2024-09-20T22:56:57.363 INFO:tasks.rados.rados.0.smithi084.stdout:3780: done (4 left) 2024-09-20T22:56:57.363 INFO:tasks.rados.rados.0.smithi084.stdout:3783: done (3 left) 2024-09-20T22:56:57.363 INFO:tasks.rados.rados.0.smithi084.stdout:3784: done (2 left) 2024-09-20T22:56:57.363 INFO:tasks.rados.rados.0.smithi084.stdout:3786: done (1 left) 2024-09-20T22:56:57.363 INFO:tasks.rados.rados.0.smithi084.stdout:3787: done (0 left) 2024-09-20T22:56:57.363 INFO:tasks.rados.rados.0.smithi084.stdout:3788: write (excl) oid 33 current snap is 438 2024-09-20T22:56:57.364 INFO:tasks.rados.rados.0.smithi084.stdout:3788: seq_num 1291 ranges {497837=410116,1596371=743454,2902966=763698} 2024-09-20T22:56:57.365 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:56:57.368 INFO:tasks.rados.rados.0.smithi084.stdout:3788: writing smithi08477203-33 from 497837 to 907953 tid 1 2024-09-20T22:56:57.376 INFO:tasks.rados.rados.0.smithi084.stdout:3788: writing smithi08477203-33 from 1596371 to 2339825 tid 2 2024-09-20T22:56:57.384 INFO:tasks.rados.rados.0.smithi084.stdout:3788: writing smithi08477203-33 from 2902966 to 3666664 tid 3 2024-09-20T22:56:57.385 INFO:tasks.rados.rados.0.smithi084.stdout:3789: snap_create 2024-09-20T22:56:57.385 INFO:tasks.rados.rados.0.smithi084.stdout:3788: finishing write tid 1 to smithi08477203-33 2024-09-20T22:56:57.385 INFO:tasks.rados.rados.0.smithi084.stdout:3788: finishing write tid 2 to smithi08477203-33 2024-09-20T22:56:57.388 INFO:tasks.rados.rados.0.smithi084.stdout:3788: finishing write tid 3 to smithi08477203-33 2024-09-20T22:56:57.388 INFO:tasks.rados.rados.0.smithi084.stdout:3788: finishing write tid 4 to smithi08477203-33 2024-09-20T22:56:57.388 INFO:tasks.rados.rados.0.smithi084.stdout:3788: finishing write tid 5 to smithi08477203-33 2024-09-20T22:56:57.388 INFO:tasks.rados.rados.0.smithi084.stdout:3788: oid 33 updating version 0 to 2852 2024-09-20T22:56:57.389 INFO:tasks.rados.rados.0.smithi084.stdout:3788: oid 33 updating version 2852 to 2853 2024-09-20T22:56:57.389 INFO:tasks.rados.rados.0.smithi084.stdout:3788: oid 33 version 2853 is already newer than 2851 2024-09-20T22:56:57.389 INFO:tasks.rados.rados.0.smithi084.stdout:3788: oid 33 updating version 2853 to 2854 2024-09-20T22:56:57.389 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 33 v 2854 (ObjNum 1291 snap 438 seq_num 1291) dirty exists 2024-09-20T22:56:57.389 INFO:tasks.rados.rados.0.smithi084.stdout:3788: left oid 33 (ObjNum 1291 snap 438 seq_num 1291) 2024-09-20T22:56:57.394 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:57.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:57.395 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:56:57.495 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:56:57.517 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:57.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:57.518 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:56:57.618 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:56:57.620 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:57 smithi154 ceph-mon[33264]: pgmap v1305: 168 pgs: 5 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 159 active+clean; 696 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 9.0 MiB/s rd, 3.3 MiB/s wr, 17 op/s 2024-09-20T22:56:57.620 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:57 smithi154 ceph-mon[33264]: 7.18 starting backfill to osd.0 from (0'0,0'0] MAX to 930'1415 2024-09-20T22:56:57.620 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:57 smithi154 ceph-mon[33264]: 7.13 starting backfill to osd.0 from (0'0,0'0] MAX to 931'475 2024-09-20T22:56:57.620 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:57 smithi154 ceph-mon[33264]: 7.13 starting backfill to osd.5 from (0'0,0'0] MAX to 931'475 2024-09-20T22:56:57.620 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:57 smithi154 ceph-mon[33264]: osdmap e936: 8 total, 8 up, 7 in 2024-09-20T22:56:57.620 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:57 smithi154 ceph-mon[33264]: 7.12 starting backfill to osd.0 from (0'0,0'0] MAX to 930'1003 2024-09-20T22:56:57.620 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:57 smithi154 ceph-mon[33264]: 7.19 starting backfill to osd.2 from (0'0,0'0] MAX to 931'1864 2024-09-20T22:56:57.620 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:57 smithi154 ceph-mon[33264]: 7.18 starting backfill to osd.5 from (0'0,0'0] MAX to 930'1415 2024-09-20T22:56:57.621 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:57 smithi154 ceph-mon[33264]: 7.12 starting backfill to osd.6 from (0'0,0'0] MAX to 930'1003 2024-09-20T22:56:57.621 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:57 smithi154 ceph-mon[33264]: 7.14 starting backfill to osd.2 from (0'0,0'0] MAX to 930'714 2024-09-20T22:56:57.621 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:57 smithi154 ceph-mon[33264]: 7.11 starting backfill to osd.1 from (0'0,0'0] MAX to 929'565 2024-09-20T22:56:57.621 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:57 smithi154 ceph-mon[33264]: 7.18 starting backfill to osd.6 from (0'0,0'0] MAX to 930'1415 2024-09-20T22:56:57.621 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:57 smithi154 ceph-mon[33264]: 7.15 starting backfill to osd.5 from (0'0,0'0] MAX to 930'540 2024-09-20T22:56:57.621 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:57 smithi154 ceph-mon[33264]: 7.16 starting backfill to osd.2 from (0'0,0'0] MAX to 930'859 2024-09-20T22:56:57.621 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:57 smithi154 ceph-mon[33264]: 7.10 starting backfill to osd.2 from (0'0,0'0] MAX to 930'998 2024-09-20T22:56:57.621 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:57 smithi154 ceph-mon[33264]: 7.19 starting backfill to osd.5 from (0'0,0'0] MAX to 931'1864 2024-09-20T22:56:57.621 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:57 smithi154 ceph-mon[33264]: 7.17 starting backfill to osd.1 from (0'0,0'0] MAX to 930'2006 2024-09-20T22:56:57.621 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:57 smithi154 ceph-mon[33264]: 7.16 starting backfill to osd.4 from (0'0,0'0] MAX to 930'859 2024-09-20T22:56:57.621 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:57 smithi154 ceph-mon[33264]: 7.14 starting backfill to osd.3 from (0'0,0'0] MAX to 930'714 2024-09-20T22:56:57.621 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:57 smithi154 ceph-mon[33264]: 7.14 starting backfill to osd.5 from (0'0,0'0] MAX to 930'714 2024-09-20T22:56:57.621 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:57 smithi154 ceph-mon[33264]: 7.19 starting backfill to osd.6 from (0'0,0'0] MAX to 931'1864 2024-09-20T22:56:57.621 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:57 smithi154 ceph-mon[33264]: 7.17 starting backfill to osd.2 from (0'0,0'0] MAX to 930'2006 2024-09-20T22:56:57.622 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:57 smithi154 ceph-mon[33264]: 7.17 starting backfill to osd.5 from (0'0,0'0] MAX to 930'2006 2024-09-20T22:56:57.622 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:57 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/2285385944' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:56:57.641 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:57.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:57.642 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:56:57.649 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.4 dump_blocked_ops 2024-09-20T22:56:57.742 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:56:57.765 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:57.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:57.766 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:56:57.832 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:57 smithi084 ceph-mon[40024]: pgmap v1305: 168 pgs: 5 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 159 active+clean; 696 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 9.0 MiB/s rd, 3.3 MiB/s wr, 17 op/s 2024-09-20T22:56:57.832 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:57 smithi084 ceph-mon[40024]: 7.18 starting backfill to osd.0 from (0'0,0'0] MAX to 930'1415 2024-09-20T22:56:57.832 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:57 smithi084 ceph-mon[40024]: 7.13 starting backfill to osd.0 from (0'0,0'0] MAX to 931'475 2024-09-20T22:56:57.832 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:57 smithi084 ceph-mon[40024]: 7.13 starting backfill to osd.5 from (0'0,0'0] MAX to 931'475 2024-09-20T22:56:57.833 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:57 smithi084 ceph-mon[40024]: osdmap e936: 8 total, 8 up, 7 in 2024-09-20T22:56:57.833 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:57 smithi084 ceph-mon[40024]: 7.12 starting backfill to osd.0 from (0'0,0'0] MAX to 930'1003 2024-09-20T22:56:57.833 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:57 smithi084 ceph-mon[40024]: 7.19 starting backfill to osd.2 from (0'0,0'0] MAX to 931'1864 2024-09-20T22:56:57.833 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:57 smithi084 ceph-mon[40024]: 7.18 starting backfill to osd.5 from (0'0,0'0] MAX to 930'1415 2024-09-20T22:56:57.833 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:57 smithi084 ceph-mon[40024]: 7.12 starting backfill to osd.6 from (0'0,0'0] MAX to 930'1003 2024-09-20T22:56:57.833 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:57 smithi084 ceph-mon[40024]: 7.14 starting backfill to osd.2 from (0'0,0'0] MAX to 930'714 2024-09-20T22:56:57.833 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:57 smithi084 ceph-mon[40024]: 7.11 starting backfill to osd.1 from (0'0,0'0] MAX to 929'565 2024-09-20T22:56:57.833 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:57 smithi084 ceph-mon[40024]: 7.18 starting backfill to osd.6 from (0'0,0'0] MAX to 930'1415 2024-09-20T22:56:57.833 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:57 smithi084 ceph-mon[40024]: 7.15 starting backfill to osd.5 from (0'0,0'0] MAX to 930'540 2024-09-20T22:56:57.833 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:57 smithi084 ceph-mon[40024]: 7.16 starting backfill to osd.2 from (0'0,0'0] MAX to 930'859 2024-09-20T22:56:57.833 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:57 smithi084 ceph-mon[40024]: 7.10 starting backfill to osd.2 from (0'0,0'0] MAX to 930'998 2024-09-20T22:56:57.833 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:57 smithi084 ceph-mon[40024]: 7.19 starting backfill to osd.5 from (0'0,0'0] MAX to 931'1864 2024-09-20T22:56:57.833 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:57 smithi084 ceph-mon[40024]: 7.17 starting backfill to osd.1 from (0'0,0'0] MAX to 930'2006 2024-09-20T22:56:57.834 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:57 smithi084 ceph-mon[40024]: 7.16 starting backfill to osd.4 from (0'0,0'0] MAX to 930'859 2024-09-20T22:56:57.834 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:57 smithi084 ceph-mon[40024]: 7.14 starting backfill to osd.3 from (0'0,0'0] MAX to 930'714 2024-09-20T22:56:57.834 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:57 smithi084 ceph-mon[40024]: 7.14 starting backfill to osd.5 from (0'0,0'0] MAX to 930'714 2024-09-20T22:56:57.834 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:57 smithi084 ceph-mon[40024]: 7.19 starting backfill to osd.6 from (0'0,0'0] MAX to 931'1864 2024-09-20T22:56:57.834 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:57 smithi084 ceph-mon[40024]: 7.17 starting backfill to osd.2 from (0'0,0'0] MAX to 930'2006 2024-09-20T22:56:57.834 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:57 smithi084 ceph-mon[40024]: 7.17 starting backfill to osd.5 from (0'0,0'0] MAX to 930'2006 2024-09-20T22:56:57.834 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:57 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/2285385944' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:56:57.834 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:57 smithi084 ceph-mon[29774]: pgmap v1305: 168 pgs: 5 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 159 active+clean; 696 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 9.0 MiB/s rd, 3.3 MiB/s wr, 17 op/s 2024-09-20T22:56:57.834 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:57 smithi084 ceph-mon[29774]: 7.18 starting backfill to osd.0 from (0'0,0'0] MAX to 930'1415 2024-09-20T22:56:57.834 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:57 smithi084 ceph-mon[29774]: 7.13 starting backfill to osd.0 from (0'0,0'0] MAX to 931'475 2024-09-20T22:56:57.834 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:57 smithi084 ceph-mon[29774]: 7.13 starting backfill to osd.5 from (0'0,0'0] MAX to 931'475 2024-09-20T22:56:57.835 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:57 smithi084 ceph-mon[29774]: osdmap e936: 8 total, 8 up, 7 in 2024-09-20T22:56:57.835 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:57 smithi084 ceph-mon[29774]: 7.12 starting backfill to osd.0 from (0'0,0'0] MAX to 930'1003 2024-09-20T22:56:57.835 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:57 smithi084 ceph-mon[29774]: 7.19 starting backfill to osd.2 from (0'0,0'0] MAX to 931'1864 2024-09-20T22:56:57.835 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:57 smithi084 ceph-mon[29774]: 7.18 starting backfill to osd.5 from (0'0,0'0] MAX to 930'1415 2024-09-20T22:56:57.835 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:57 smithi084 ceph-mon[29774]: 7.12 starting backfill to osd.6 from (0'0,0'0] MAX to 930'1003 2024-09-20T22:56:57.835 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:57 smithi084 ceph-mon[29774]: 7.14 starting backfill to osd.2 from (0'0,0'0] MAX to 930'714 2024-09-20T22:56:57.835 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:57 smithi084 ceph-mon[29774]: 7.11 starting backfill to osd.1 from (0'0,0'0] MAX to 929'565 2024-09-20T22:56:57.835 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:57 smithi084 ceph-mon[29774]: 7.18 starting backfill to osd.6 from (0'0,0'0] MAX to 930'1415 2024-09-20T22:56:57.835 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:57 smithi084 ceph-mon[29774]: 7.15 starting backfill to osd.5 from (0'0,0'0] MAX to 930'540 2024-09-20T22:56:57.835 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:57 smithi084 ceph-mon[29774]: 7.16 starting backfill to osd.2 from (0'0,0'0] MAX to 930'859 2024-09-20T22:56:57.835 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:57 smithi084 ceph-mon[29774]: 7.10 starting backfill to osd.2 from (0'0,0'0] MAX to 930'998 2024-09-20T22:56:57.835 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:57 smithi084 ceph-mon[29774]: 7.19 starting backfill to osd.5 from (0'0,0'0] MAX to 931'1864 2024-09-20T22:56:57.835 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:57 smithi084 ceph-mon[29774]: 7.17 starting backfill to osd.1 from (0'0,0'0] MAX to 930'2006 2024-09-20T22:56:57.835 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:57 smithi084 ceph-mon[29774]: 7.16 starting backfill to osd.4 from (0'0,0'0] MAX to 930'859 2024-09-20T22:56:57.835 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:57 smithi084 ceph-mon[29774]: 7.14 starting backfill to osd.3 from (0'0,0'0] MAX to 930'714 2024-09-20T22:56:57.836 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:57 smithi084 ceph-mon[29774]: 7.14 starting backfill to osd.5 from (0'0,0'0] MAX to 930'714 2024-09-20T22:56:57.836 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:57 smithi084 ceph-mon[29774]: 7.19 starting backfill to osd.6 from (0'0,0'0] MAX to 931'1864 2024-09-20T22:56:57.836 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:57 smithi084 ceph-mon[29774]: 7.17 starting backfill to osd.2 from (0'0,0'0] MAX to 930'2006 2024-09-20T22:56:57.836 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:57 smithi084 ceph-mon[29774]: 7.17 starting backfill to osd.5 from (0'0,0'0] MAX to 930'2006 2024-09-20T22:56:57.836 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:57 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/2285385944' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:56:57.867 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:56:57.888 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:57.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:57.889 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:56:57.935 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:56:57.990 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:56:58.012 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:58.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:58.013 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:56:58.114 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:56:58.134 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:58.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:58.135 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:56:58.236 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:56:58.257 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:58.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:58.258 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:56:58.358 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:56:58.359 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:56:58 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: ::ffff:172.21.15.154 - - [20/Sep/2024:22:56:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-20T22:56:58.360 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is set 2024-09-20T22:56:58.361 INFO:tasks.rados.rados.0.smithi084.stdout:3788: done (1 left) 2024-09-20T22:56:58.361 INFO:tasks.rados.rados.0.smithi084.stdout:3789: done (0 left) 2024-09-20T22:56:58.362 INFO:tasks.rados.rados.0.smithi084.stdout:3790: copy_from oid 2 from oid 8 current snap is 439 2024-09-20T22:56:58.362 INFO:tasks.rados.rados.0.smithi084.stdout:3791: read oid 43 snap -1 2024-09-20T22:56:58.362 INFO:tasks.rados.rados.0.smithi084.stdout:3791: expect (ObjNum 1259 snap 430 seq_num 1259) 2024-09-20T22:56:58.362 INFO:tasks.rados.rados.0.smithi084.stdout:3792: write (excl) oid 11 current snap is 439 2024-09-20T22:56:58.362 INFO:tasks.rados.rados.0.smithi084.stdout:3792: seq_num 1292 ranges {514644=485365,1654853=487112,2930389=574970} 2024-09-20T22:56:58.367 INFO:tasks.rados.rados.0.smithi084.stdout:3792: writing smithi08477203-11 from 514644 to 1000009 tid 1 2024-09-20T22:56:58.375 INFO:tasks.rados.rados.0.smithi084.stdout:3792: writing smithi08477203-11 from 1654853 to 2141965 tid 2 2024-09-20T22:56:58.384 INFO:tasks.rados.rados.0.smithi084.stdout:3792: writing smithi08477203-11 from 2930389 to 3505359 tid 3 2024-09-20T22:56:58.385 INFO:tasks.rados.rados.0.smithi084.stdout:3793: read oid 37 snap -1 2024-09-20T22:56:58.385 INFO:tasks.rados.rados.0.smithi084.stdout:3793: expect (ObjNum 1189 snap 408 seq_num 1189) 2024-09-20T22:56:58.385 INFO:tasks.rados.rados.0.smithi084.stdout:3794: read oid 10 snap -1 2024-09-20T22:56:58.385 INFO:tasks.rados.rados.0.smithi084.stdout:3794: expect deleted 2024-09-20T22:56:58.385 INFO:tasks.rados.rados.0.smithi084.stdout:3795: read oid 31 snap 397 2024-09-20T22:56:58.385 INFO:tasks.rados.rados.0.smithi084.stdout:3795: expect (ObjNum 1143 snap 389 seq_num 1143) 2024-09-20T22:56:58.385 INFO:tasks.rados.rados.0.smithi084.stdout:3796: write (excl) oid 1 current snap is 439 2024-09-20T22:56:58.385 INFO:tasks.rados.rados.0.smithi084.stdout:3796: seq_num 1293 ranges {531451=676967,1713335=747123,2957812=502595} 2024-09-20T22:56:58.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:58.386 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:58.387 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:56:58.391 INFO:tasks.rados.rados.0.smithi084.stdout:3796: writing smithi08477203-1 from 531451 to 1208418 tid 1 2024-09-20T22:56:58.399 INFO:tasks.rados.rados.0.smithi084.stdout:3796: writing smithi08477203-1 from 1713335 to 2460458 tid 2 2024-09-20T22:56:58.405 INFO:tasks.rados.rados.0.smithi084.stdout:3796: writing smithi08477203-1 from 2957812 to 3460407 tid 3 2024-09-20T22:56:58.405 INFO:tasks.rados.rados.0.smithi084.stdout:3797: delete oid 50 current snap is 439 2024-09-20T22:56:58.405 INFO:tasks.rados.rados.0.smithi084.stdout:3791: expect (ObjNum 1259 snap 430 seq_num 1259) 2024-09-20T22:56:58.417 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-20T22:56:58.417 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [], 2024-09-20T22:56:58.417 INFO:teuthology.orchestra.run.smithi154.stdout: "complaint_time": 30, 2024-09-20T22:56:58.417 INFO:teuthology.orchestra.run.smithi154.stdout: "num_blocked_ops": 0 2024-09-20T22:56:58.417 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-20T22:56:58.488 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:56:58.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:58.510 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:58.510 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:56:58.511 INFO:tasks.rados.rados.0.smithi084.stdout:3790: finishing copy_from to smithi08477203-2 2024-09-20T22:56:58.511 INFO:tasks.rados.rados.0.smithi084.stdout:3790: got expected ENOENT (src dne) 2024-09-20T22:56:58.511 INFO:tasks.rados.rados.0.smithi084.stdout:3790: finishing copy_from racing read to smithi08477203-2 2024-09-20T22:56:58.511 INFO:tasks.rados.rados.0.smithi084.stdout:3792: finishing write tid 1 to smithi08477203-11 2024-09-20T22:56:58.512 INFO:tasks.rados.rados.0.smithi084.stdout:3792: finishing write tid 2 to smithi08477203-11 2024-09-20T22:56:58.512 INFO:tasks.rados.rados.0.smithi084.stdout:3790: done (7 left) 2024-09-20T22:56:58.512 INFO:tasks.rados.rados.0.smithi084.stdout:3791: done (6 left) 2024-09-20T22:56:58.512 INFO:tasks.rados.rados.0.smithi084.stdout:3797: done (5 left) 2024-09-20T22:56:58.512 INFO:tasks.rados.rados.0.smithi084.stdout:3798: write (excl) oid 25 current snap is 439 2024-09-20T22:56:58.512 INFO:tasks.rados.rados.0.smithi084.stdout:3798: seq_num 1295 ranges {565065=427465,1430299=634439,2496305=525139,3472104=292961} 2024-09-20T22:56:58.516 INFO:tasks.rados.rados.0.smithi084.stdout:3798: writing smithi08477203-25 from 565065 to 992530 tid 1 2024-09-20T22:56:58.522 INFO:tasks.rados.rados.0.smithi084.stdout:3798: writing smithi08477203-25 from 1430299 to 2064738 tid 2 2024-09-20T22:56:58.528 INFO:tasks.rados.rados.0.smithi084.stdout:3798: writing smithi08477203-25 from 2496305 to 3021444 tid 3 2024-09-20T22:56:58.533 INFO:tasks.rados.rados.0.smithi084.stdout:3798: writing smithi08477203-25 from 3472104 to 3765065 tid 4 2024-09-20T22:56:58.533 INFO:tasks.rados.rados.0.smithi084.stdout:3794: done (5 left) 2024-09-20T22:56:58.533 INFO:tasks.rados.rados.0.smithi084.stdout:3799: read oid 32 snap -1 2024-09-20T22:56:58.533 INFO:tasks.rados.rados.0.smithi084.stdout:3799: expect (ObjNum 1289 snap 436 seq_num 1289) 2024-09-20T22:56:58.533 INFO:tasks.rados.rados.0.smithi084.stdout:3800: read oid 17 snap -1 2024-09-20T22:56:58.533 INFO:tasks.rados.rados.0.smithi084.stdout:3800: expect deleted 2024-09-20T22:56:58.533 INFO:tasks.rados.rados.0.smithi084.stdout:3801: write oid 40 current snap is 439 2024-09-20T22:56:58.533 INFO:tasks.rados.rados.0.smithi084.stdout:3801: seq_num 1296 ranges {581872=502714,1488781=494450,2640081=736411} 2024-09-20T22:56:58.538 INFO:tasks.rados.rados.0.smithi084.stdout:3801: writing smithi08477203-40 from 581872 to 1084586 tid 1 2024-09-20T22:56:58.543 INFO:tasks.rados.rados.0.smithi084.stdout:3801: writing smithi08477203-40 from 1488781 to 1983231 tid 2 2024-09-20T22:56:58.551 INFO:tasks.rados.rados.0.smithi084.stdout:3801: writing smithi08477203-40 from 2640081 to 3376492 tid 3 2024-09-20T22:56:58.551 INFO:tasks.rados.rados.0.smithi084.stdout:3802: write (excl) oid 23 current snap is 439 2024-09-20T22:56:58.552 INFO:tasks.rados.rados.0.smithi084.stdout:3792: finishing write tid 3 to smithi08477203-11 2024-09-20T22:56:58.552 INFO:tasks.rados.rados.0.smithi084.stdout:3792: finishing write tid 4 to smithi08477203-11 2024-09-20T22:56:58.552 INFO:tasks.rados.rados.0.smithi084.stdout:3792: finishing write tid 5 to smithi08477203-11 2024-09-20T22:56:58.552 INFO:tasks.rados.rados.0.smithi084.stdout:3792: oid 11 updating version 0 to 2666 2024-09-20T22:56:58.552 INFO:tasks.rados.rados.0.smithi084.stdout:3792: oid 11 updating version 2666 to 2669 2024-09-20T22:56:58.552 INFO:tasks.rados.rados.0.smithi084.stdout:3792: oid 11 version 2669 is already newer than 2667 2024-09-20T22:56:58.552 INFO:tasks.rados.rados.0.smithi084.stdout:3792: oid 11 version 2669 is already newer than 2668 2024-09-20T22:56:58.552 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 11 v 2669 (ObjNum 1292 snap 439 seq_num 1292) dirty exists 2024-09-20T22:56:58.552 INFO:tasks.rados.rados.0.smithi084.stdout:3792: left oid 11 (ObjNum 1292 snap 439 seq_num 1292) 2024-09-20T22:56:58.552 INFO:tasks.rados.rados.0.smithi084.stdout:3793: expect (ObjNum 1189 snap 408 seq_num 1189) 2024-09-20T22:56:58.611 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:56:58.614 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:58 smithi084 ceph-mon[29774]: osdmap e937: 8 total, 8 up, 7 in 2024-09-20T22:56:58.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:58 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/54491355' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:56:58.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:58 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:56:58.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:58 smithi084 ceph-mon[29774]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:56:58.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:58 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T22:56:58.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:58 smithi084 ceph-mon[29774]: osdmap e938: 8 total, 8 up, 7 in 2024-09-20T22:56:58.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:58 smithi084 ceph-mon[40024]: osdmap e937: 8 total, 8 up, 7 in 2024-09-20T22:56:58.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:58 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/54491355' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:56:58.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:58 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:56:58.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:58 smithi084 ceph-mon[40024]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:56:58.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:58 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T22:56:58.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:58 smithi084 ceph-mon[40024]: osdmap e938: 8 total, 8 up, 7 in 2024-09-20T22:56:58.633 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:58.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:58.635 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:56:58.728 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:58 smithi154 ceph-mon[33264]: osdmap e937: 8 total, 8 up, 7 in 2024-09-20T22:56:58.728 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:58 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/54491355' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:56:58.728 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:58 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:56:58.728 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:58 smithi154 ceph-mon[33264]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:56:58.729 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:58 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T22:56:58.729 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:58 smithi154 ceph-mon[33264]: osdmap e938: 8 total, 8 up, 7 in 2024-09-20T22:56:58.735 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:56:58.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:58.762 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:58.762 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:56:58.849 INFO:tasks.rados.rados.0.smithi084.stdout:3795: expect (ObjNum 1143 snap 389 seq_num 1143) 2024-09-20T22:56:58.863 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:56:58.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:58.886 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:58.886 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:56:58.987 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:56:59.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:59.014 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:59.014 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:56:59.038 INFO:tasks.rados.rados.0.smithi084.stdout:3796: finishing write tid 1 to smithi08477203-1 2024-09-20T22:56:59.038 INFO:tasks.rados.rados.0.smithi084.stdout:3796: finishing write tid 2 to smithi08477203-1 2024-09-20T22:56:59.038 INFO:tasks.rados.rados.0.smithi084.stdout:3796: finishing write tid 3 to smithi08477203-1 2024-09-20T22:56:59.038 INFO:tasks.rados.rados.0.smithi084.stdout:3796: finishing write tid 4 to smithi08477203-1 2024-09-20T22:56:59.038 INFO:tasks.rados.rados.0.smithi084.stdout:3796: finishing write tid 5 to smithi08477203-1 2024-09-20T22:56:59.038 INFO:tasks.rados.rados.0.smithi084.stdout:3796: oid 1 updating version 0 to 2686 2024-09-20T22:56:59.038 INFO:tasks.rados.rados.0.smithi084.stdout:3796: oid 1 updating version 2686 to 2688 2024-09-20T22:56:59.038 INFO:tasks.rados.rados.0.smithi084.stdout:3796: oid 1 version 2688 is already newer than 2685 2024-09-20T22:56:59.039 INFO:tasks.rados.rados.0.smithi084.stdout:3796: oid 1 version 2688 is already newer than 2687 2024-09-20T22:56:59.039 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 1 v 2688 (ObjNum 1293 snap 439 seq_num 1293) dirty exists 2024-09-20T22:56:59.039 INFO:tasks.rados.rados.0.smithi084.stdout:3796: left oid 1 (ObjNum 1293 snap 439 seq_num 1293) 2024-09-20T22:56:59.039 INFO:tasks.rados.rados.0.smithi084.stdout:3802: seq_num 1297 ranges {598679=577963,1947263=638108,3351151=447528} 2024-09-20T22:56:59.046 INFO:tasks.rados.rados.0.smithi084.stdout:3802: writing smithi08477203-23 from 598679 to 1176642 tid 1 2024-09-20T22:56:59.047 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.4 dump_historic_ops 2024-09-20T22:56:59.052 INFO:tasks.rados.rados.0.smithi084.stdout:3802: writing smithi08477203-23 from 1947263 to 2585371 tid 2 2024-09-20T22:56:59.059 INFO:tasks.rados.rados.0.smithi084.stdout:3802: writing smithi08477203-23 from 3351151 to 3798679 tid 3 2024-09-20T22:56:59.059 INFO:tasks.rados.rados.0.smithi084.stdout:3792: done (8 left) 2024-09-20T22:56:59.059 INFO:tasks.rados.rados.0.smithi084.stdout:3793: done (7 left) 2024-09-20T22:56:59.059 INFO:tasks.rados.rados.0.smithi084.stdout:3795: done (6 left) 2024-09-20T22:56:59.059 INFO:tasks.rados.rados.0.smithi084.stdout:3796: done (5 left) 2024-09-20T22:56:59.059 INFO:tasks.rados.rados.0.smithi084.stdout:3803: copy_from oid 19 from oid 41 current snap is 439 2024-09-20T22:56:59.059 INFO:tasks.rados.rados.0.smithi084.stdout:3798: finishing write tid 1 to smithi08477203-25 2024-09-20T22:56:59.059 INFO:tasks.rados.rados.0.smithi084.stdout:3798: finishing write tid 2 to smithi08477203-25 2024-09-20T22:56:59.060 INFO:tasks.rados.rados.0.smithi084.stdout:3798: finishing write tid 3 to smithi08477203-25 2024-09-20T22:56:59.060 INFO:tasks.rados.rados.0.smithi084.stdout:3798: finishing write tid 4 to smithi08477203-25 2024-09-20T22:56:59.060 INFO:tasks.rados.rados.0.smithi084.stdout:3798: finishing write tid 5 to smithi08477203-25 2024-09-20T22:56:59.060 INFO:tasks.rados.rados.0.smithi084.stdout:3798: finishing write tid 6 to smithi08477203-25 2024-09-20T22:56:59.060 INFO:tasks.rados.rados.0.smithi084.stdout:3798: oid 25 updating version 0 to 3117 2024-09-20T22:56:59.060 INFO:tasks.rados.rados.0.smithi084.stdout:3798: oid 25 updating version 3117 to 3120 2024-09-20T22:56:59.060 INFO:tasks.rados.rados.0.smithi084.stdout:3798: oid 25 version 3120 is already newer than 3119 2024-09-20T22:56:59.060 INFO:tasks.rados.rados.0.smithi084.stdout:3798: oid 25 version 3120 is already newer than 3118 2024-09-20T22:56:59.060 INFO:tasks.rados.rados.0.smithi084.stdout:3798: oid 25 version 3120 is already newer than 3116 2024-09-20T22:56:59.060 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 25 v 3120 (ObjNum 1295 snap 439 seq_num 1295) dirty exists 2024-09-20T22:56:59.060 INFO:tasks.rados.rados.0.smithi084.stdout:3798: left oid 25 (ObjNum 1295 snap 439 seq_num 1295) 2024-09-20T22:56:59.060 INFO:tasks.rados.rados.0.smithi084.stdout:3799: expect (ObjNum 1289 snap 436 seq_num 1289) 2024-09-20T22:56:59.115 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:56:59.158 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:59.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:59.160 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:56:59.226 INFO:tasks.rados.rados.0.smithi084.stdout:3801: finishing write tid 1 to smithi08477203-40 2024-09-20T22:56:59.226 INFO:tasks.rados.rados.0.smithi084.stdout:3801: finishing write tid 2 to smithi08477203-40 2024-09-20T22:56:59.226 INFO:tasks.rados.rados.0.smithi084.stdout:3801: finishing write tid 3 to smithi08477203-40 2024-09-20T22:56:59.226 INFO:tasks.rados.rados.0.smithi084.stdout:3801: finishing write tid 4 to smithi08477203-40 2024-09-20T22:56:59.226 INFO:tasks.rados.rados.0.smithi084.stdout:3798: done (5 left) 2024-09-20T22:56:59.226 INFO:tasks.rados.rados.0.smithi084.stdout:3799: done (4 left) 2024-09-20T22:56:59.227 INFO:tasks.rados.rados.0.smithi084.stdout:3800: done (3 left) 2024-09-20T22:56:59.227 INFO:tasks.rados.rados.0.smithi084.stdout:3804: snap_create 2024-09-20T22:56:59.227 INFO:tasks.rados.rados.0.smithi084.stdout:3801: finishing write tid 5 to smithi08477203-40 2024-09-20T22:56:59.227 INFO:tasks.rados.rados.0.smithi084.stdout:3801: oid 40 updating version 0 to 2856 2024-09-20T22:56:59.227 INFO:tasks.rados.rados.0.smithi084.stdout:3801: oid 40 version 2856 is already newer than 2855 2024-09-20T22:56:59.227 INFO:tasks.rados.rados.0.smithi084.stdout:3801: oid 40 updating version 2856 to 2858 2024-09-20T22:56:59.227 INFO:tasks.rados.rados.0.smithi084.stdout:3801: oid 40 version 2858 is already newer than 2857 2024-09-20T22:56:59.227 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 40 v 2858 (ObjNum 1296 snap 439 seq_num 1296) dirty exists 2024-09-20T22:56:59.227 INFO:tasks.rados.rados.0.smithi084.stdout:3801: left oid 40 (ObjNum 1296 snap 439 seq_num 1296) 2024-09-20T22:56:59.227 INFO:tasks.rados.rados.0.smithi084.stdout:3802: finishing write tid 1 to smithi08477203-23 2024-09-20T22:56:59.228 INFO:tasks.rados.rados.0.smithi084.stdout:3802: finishing write tid 2 to smithi08477203-23 2024-09-20T22:56:59.228 INFO:tasks.rados.rados.0.smithi084.stdout:3802: finishing write tid 3 to smithi08477203-23 2024-09-20T22:56:59.228 INFO:tasks.rados.rados.0.smithi084.stdout:3802: finishing write tid 4 to smithi08477203-23 2024-09-20T22:56:59.228 INFO:tasks.rados.rados.0.smithi084.stdout:3802: finishing write tid 5 to smithi08477203-23 2024-09-20T22:56:59.228 INFO:tasks.rados.rados.0.smithi084.stdout:3802: oid 23 updating version 0 to 2861 2024-09-20T22:56:59.228 INFO:tasks.rados.rados.0.smithi084.stdout:3802: oid 23 updating version 2861 to 2862 2024-09-20T22:56:59.228 INFO:tasks.rados.rados.0.smithi084.stdout:3802: oid 23 version 2862 is already newer than 2860 2024-09-20T22:56:59.229 INFO:tasks.rados.rados.0.smithi084.stdout:3802: oid 23 version 2862 is already newer than 2859 2024-09-20T22:56:59.229 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 23 v 2862 (ObjNum 1297 snap 439 seq_num 1297) dirty exists 2024-09-20T22:56:59.229 INFO:tasks.rados.rados.0.smithi084.stdout:3802: left oid 23 (ObjNum 1297 snap 439 seq_num 1297) 2024-09-20T22:56:59.229 INFO:tasks.rados.rados.0.smithi084.stdout:3803: finishing copy_from racing read to smithi08477203-19 2024-09-20T22:56:59.229 INFO:tasks.rados.rados.0.smithi084.stdout:3803: finishing copy_from to smithi08477203-19 2024-09-20T22:56:59.229 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 19 v 2668 (ObjNum 738 snap 235 seq_num 738) dirty exists 2024-09-20T22:56:59.260 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:56:59.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:59.286 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:59.286 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:56:59.342 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:56:59.383 INFO:tasks.rados.rados.0.smithi084.stdout:3801: done (3 left) 2024-09-20T22:56:59.383 INFO:tasks.rados.rados.0.smithi084.stdout:3802: done (2 left) 2024-09-20T22:56:59.384 INFO:tasks.rados.rados.0.smithi084.stdout:3803: done (1 left) 2024-09-20T22:56:59.384 INFO:tasks.rados.rados.0.smithi084.stdout:3804: done (0 left) 2024-09-20T22:56:59.384 INFO:tasks.rados.rados.0.smithi084.stdout:3805: snap_create 2024-09-20T22:56:59.387 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:56:59.410 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:59.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:59.411 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:56:59.512 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:56:59.536 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:56:59.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:59.537 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:56:59.637 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:56:59.649 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:56:59 smithi154 ceph-mon[33264]: pgmap v1308: 168 pgs: 5 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 159 active+clean; 696 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-20T22:56:59.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:59.660 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:59.661 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:56:59.761 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:56:59.783 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:59.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:59.784 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:56:59.849 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-20T22:56:59.849 INFO:teuthology.orchestra.run.smithi154.stdout: "size": 20, 2024-09-20T22:56:59.850 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 600, 2024-09-20T22:56:59.850 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [ 2024-09-20T22:56:59.850 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.850 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:4229 7.14 e423/421 7:2f8647a2:::smithi08477203-41:head v 423'133, mlcod=422'129)", 2024-09-20T22:56:59.850 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:48:23.296695+0000", 2024-09-20T22:56:59.850 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 516.550371703, 2024-09-20T22:56:59.850 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.011591664, 2024-09-20T22:56:59.850 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:56:59.850 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:56:59.850 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:56:59.850 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:56:59.850 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.850 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:56:59.851 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:23.296695+0000", 2024-09-20T22:56:59.851 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:56:59.851 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.851 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.851 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:56:59.851 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:23.296695+0000", 2024-09-20T22:56:59.851 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:56:59.851 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.851 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.851 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:56:59.851 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:23.296697+0000", 2024-09-20T22:56:59.851 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.8509999999999999e-06 2024-09-20T22:56:59.851 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.851 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.852 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:56:59.852 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:23.302610+0000", 2024-09-20T22:56:59.852 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0059131959999999999 2024-09-20T22:56:59.852 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.852 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.852 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:56:59.852 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:23.302632+0000", 2024-09-20T22:56:59.852 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.2146000000000001e-05 2024-09-20T22:56:59.852 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.852 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.852 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:56:59.853 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:23.302641+0000", 2024-09-20T22:56:59.853 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.9069999999999996e-06 2024-09-20T22:56:59.853 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.853 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.853 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:56:59.853 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:23.302711+0000", 2024-09-20T22:56:59.853 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.0038999999999998e-05 2024-09-20T22:56:59.853 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.853 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.853 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:56:59.853 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:23.302732+0000", 2024-09-20T22:56:59.853 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.0922000000000001e-05 2024-09-20T22:56:59.853 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.853 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.854 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:56:59.854 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:23.308245+0000", 2024-09-20T22:56:59.854 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0055130889999999997 2024-09-20T22:56:59.854 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.854 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.854 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:56:59.854 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:23.308287+0000", 2024-09-20T22:56:59.854 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.1514000000000003e-05 2024-09-20T22:56:59.854 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:56:59.854 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:56:59.854 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:56:59.854 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.854 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.855 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:5057 7.9 e508/426 7:9eac4560:::smithi08477203-30:head v 508'554, mlcod=505'548)", 2024-09-20T22:56:59.855 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:49:48.245783+0000", 2024-09-20T22:56:59.855 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 431.601283749, 2024-09-20T22:56:59.855 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.035976553000000001, 2024-09-20T22:56:59.855 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:56:59.855 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:56:59.855 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:56:59.855 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:56:59.855 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.855 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:56:59.855 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:48.245783+0000", 2024-09-20T22:56:59.855 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:56:59.855 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.855 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.856 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:56:59.856 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:48.245783+0000", 2024-09-20T22:56:59.856 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:56:59.856 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.856 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.856 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:56:59.856 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:48.245784+0000", 2024-09-20T22:56:59.856 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.08e-06 2024-09-20T22:56:59.856 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.856 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.856 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:56:59.856 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:48.279966+0000", 2024-09-20T22:56:59.856 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.034181493 2024-09-20T22:56:59.856 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.857 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.857 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:56:59.857 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:48.279981+0000", 2024-09-20T22:56:59.857 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.5092000000000001e-05 2024-09-20T22:56:59.857 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.857 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.857 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:56:59.857 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:48.280000+0000", 2024-09-20T22:56:59.857 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9493000000000001e-05 2024-09-20T22:56:59.857 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.857 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.857 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:56:59.857 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:48.280054+0000", 2024-09-20T22:56:59.857 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.3687000000000003e-05 2024-09-20T22:56:59.858 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.858 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.858 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:56:59.858 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:48.280072+0000", 2024-09-20T22:56:59.858 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.8581e-05 2024-09-20T22:56:59.858 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.858 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.858 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:56:59.858 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:48.281731+0000", 2024-09-20T22:56:59.858 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0016588569999999999 2024-09-20T22:56:59.858 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.858 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.858 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:56:59.859 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:49:48.281760+0000", 2024-09-20T22:56:59.859 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.8269999999999999e-05 2024-09-20T22:56:59.859 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:56:59.859 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:56:59.859 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:56:59.859 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.859 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.859 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:5414 7.8 e549/430 7:1cd232b2:::smithi08477203-35:head v 549'467, mlcod=548'459)", 2024-09-20T22:56:59.859 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:50:29.260963+0000", 2024-09-20T22:56:59.859 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 390.58610425000001, 2024-09-20T22:56:59.859 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0085861400000000008, 2024-09-20T22:56:59.859 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:56:59.859 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:56:59.859 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:56:59.859 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:56:59.860 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.860 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:56:59.860 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:29.260963+0000", 2024-09-20T22:56:59.860 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:56:59.860 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.860 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.860 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:56:59.860 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:29.260963+0000", 2024-09-20T22:56:59.860 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:56:59.860 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.860 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.860 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:56:59.860 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:29.260963+0000", 2024-09-20T22:56:59.861 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.0900000000000002e-07 2024-09-20T22:56:59.861 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.861 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.861 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:56:59.862 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:29.266981+0000", 2024-09-20T22:56:59.862 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0060184360000000003 2024-09-20T22:56:59.862 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.862 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.862 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:56:59.862 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:29.266992+0000", 2024-09-20T22:56:59.862 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0254e-05 2024-09-20T22:56:59.862 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.862 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.862 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:56:59.862 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:29.266999+0000", 2024-09-20T22:56:59.862 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.0369999999999997e-06 2024-09-20T22:56:59.862 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.862 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.862 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:56:59.863 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:29.267052+0000", 2024-09-20T22:56:59.863 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.3377000000000001e-05 2024-09-20T22:56:59.863 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.863 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.863 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:56:59.863 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:29.267073+0000", 2024-09-20T22:56:59.863 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.0604000000000001e-05 2024-09-20T22:56:59.863 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.863 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.863 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:56:59.863 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:29.269512+0000", 2024-09-20T22:56:59.863 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0024397749999999999 2024-09-20T22:56:59.863 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.863 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.863 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:56:59.864 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:29.269549+0000", 2024-09-20T22:56:59.864 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.6248000000000003e-05 2024-09-20T22:56:59.864 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:56:59.864 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:56:59.864 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:56:59.864 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.864 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.864 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:5418 7.1 7:8422f82d:::smithi08477203-22:head [copy-from ver 726] snapc ef=[ef,ed,ec,eb,ea,e8,e7,e6,e5,e4,e2,df,db,da,d7,d5,d4,d3,d0,cf,ce,ca,c5,c2,bf,b4,b2,a7,95] ondisk+write+known_if_redirected+supports_pool_eio e549)", 2024-09-20T22:56:59.864 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:50:29.266643+0000", 2024-09-20T22:56:59.864 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-20T22:56:59.865 INFO:tasks.thrashosds.thrasher:in_osds: [2, 3, 5, 6, 0, 4, 1] out_osds: [7] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-20T22:56:59.865 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-20T22:56:59.865 INFO:tasks.thrashosds.thrasher:Killing osd 3, live_osds are [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-20T22:56:59.865 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-09-20T22:56:59.865 DEBUG:teuthology.orchestra.run.smithi084:> sudo systemctl stop ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490@osd.3 2024-09-20T22:56:59.866 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:56:59 smithi084 ceph-mon[29774]: pgmap v1308: 168 pgs: 5 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 159 active+clean; 696 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-20T22:56:59.866 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:56:59 smithi084 ceph-mon[40024]: pgmap v1308: 168 pgs: 5 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 159 active+clean; 696 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-20T22:56:59.867 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 390.58042358199998, 2024-09-20T22:56:59.867 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.022362705, 2024-09-20T22:56:59.867 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:56:59.867 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:56:59.867 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:56:59.867 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:56:59.867 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:56:59.867 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:56:59.868 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 5418 2024-09-20T22:56:59.868 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.868 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:56:59.868 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.868 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:56:59.868 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:29.266643+0000", 2024-09-20T22:56:59.868 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:56:59.868 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.868 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.868 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:56:59.868 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:29.266643+0000", 2024-09-20T22:56:59.868 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:56:59.868 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.868 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.869 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:56:59.869 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:29.266646+0000", 2024-09-20T22:56:59.869 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.835e-06 2024-09-20T22:56:59.869 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.869 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.869 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:56:59.869 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:29.266654+0000", 2024-09-20T22:56:59.869 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.7540000000000006e-06 2024-09-20T22:56:59.869 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.869 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.869 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:56:59.869 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:29.266672+0000", 2024-09-20T22:56:59.869 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.8187000000000001e-05 2024-09-20T22:56:59.869 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.869 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.870 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:56:59.870 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:29.266681+0000", 2024-09-20T22:56:59.870 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.5199999999999997e-06 2024-09-20T22:56:59.870 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.870 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.870 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:56:59.870 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:29.266788+0000", 2024-09-20T22:56:59.870 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00010746 2024-09-20T22:56:59.870 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.870 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.870 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:56:59.870 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:29.266936+0000", 2024-09-20T22:56:59.870 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000148238 2024-09-20T22:56:59.871 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.871 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.871 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 3,6", 2024-09-20T22:56:59.871 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:29.274529+0000", 2024-09-20T22:56:59.871 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0075932170000000002 2024-09-20T22:56:59.871 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.871 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.871 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:56:59.871 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:29.279305+0000", 2024-09-20T22:56:59.871 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0047758990000000001 2024-09-20T22:56:59.871 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.871 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.871 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:56:59.872 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:29.281410+0000", 2024-09-20T22:56:59.872 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.002104341 2024-09-20T22:56:59.872 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.872 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.872 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:56:59.872 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:29.288915+0000", 2024-09-20T22:56:59.872 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0075048900000000002 2024-09-20T22:56:59.872 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.872 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.872 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:56:59.872 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:29.288966+0000", 2024-09-20T22:56:59.872 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.1669000000000003e-05 2024-09-20T22:56:59.872 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.872 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.873 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:56:59.873 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:29.289006+0000", 2024-09-20T22:56:59.873 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.9694999999999999e-05 2024-09-20T22:56:59.873 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:56:59.873 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:56:59.873 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:56:59.873 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.873 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.873 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:5419 7.1 7:8422f82d:::smithi08477203-22:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e549)", 2024-09-20T22:56:59.873 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:50:29.266705+0000", 2024-09-20T22:56:59.873 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 390.58036224900002, 2024-09-20T22:56:59.873 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.012031030999999999, 2024-09-20T22:56:59.873 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:56:59.873 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:56:59.874 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:56:59.874 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:56:59.874 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:56:59.874 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:56:59.874 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 5419 2024-09-20T22:56:59.874 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.874 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:56:59.874 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.874 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:56:59.874 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:29.266705+0000", 2024-09-20T22:56:59.874 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:56:59.874 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.874 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.875 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:56:59.875 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:29.266705+0000", 2024-09-20T22:56:59.875 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:56:59.875 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.875 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.875 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:56:59.875 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:29.266705+0000", 2024-09-20T22:56:59.875 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.5000000000000002e-07 2024-09-20T22:56:59.875 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.875 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.875 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:56:59.875 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:29.266708+0000", 2024-09-20T22:56:59.875 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.242e-06 2024-09-20T22:56:59.875 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.876 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.876 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:56:59.876 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:29.266717+0000", 2024-09-20T22:56:59.876 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.9390000000000003e-06 2024-09-20T22:56:59.876 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.876 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.876 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:56:59.876 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:29.266723+0000", 2024-09-20T22:56:59.876 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.7289999999999997e-06 2024-09-20T22:56:59.876 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.876 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.876 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:56:59.876 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:29.267079+0000", 2024-09-20T22:56:59.877 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00035592600000000001 2024-09-20T22:56:59.877 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.877 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.877 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for blocked object", 2024-09-20T22:56:59.877 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:29.267125+0000", 2024-09-20T22:56:59.877 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.5859000000000003e-05 2024-09-20T22:56:59.877 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.877 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.877 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:56:59.877 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:29.278512+0000", 2024-09-20T22:56:59.877 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.011386871999999999 2024-09-20T22:56:59.877 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.877 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.877 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:56:59.877 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:29.278620+0000", 2024-09-20T22:56:59.878 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000108729 2024-09-20T22:56:59.878 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.878 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.878 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:56:59.878 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:50:29.278736+0000", 2024-09-20T22:56:59.878 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000115085 2024-09-20T22:56:59.878 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:56:59.878 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:56:59.878 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:56:59.878 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.878 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.878 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(osd.0.0:1266 7.9 e596/426 7:9eac4560:::smithi08477203-30:ae v 596'854, mlcod=593'848)", 2024-09-20T22:56:59.878 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:51:16.392066+0000", 2024-09-20T22:56:59.879 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 343.45500099200001, 2024-09-20T22:56:59.879 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0099900849999999992, 2024-09-20T22:56:59.879 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:56:59.879 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:56:59.879 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:56:59.879 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:56:59.879 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.879 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:56:59.879 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:16.392066+0000", 2024-09-20T22:56:59.879 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:56:59.879 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.879 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.879 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:56:59.879 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:16.392066+0000", 2024-09-20T22:56:59.879 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:56:59.880 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.880 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.880 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:56:59.880 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:16.392067+0000", 2024-09-20T22:56:59.880 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.2600000000000001e-07 2024-09-20T22:56:59.880 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.880 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.880 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:56:59.880 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:16.392073+0000", 2024-09-20T22:56:59.880 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.6989999999999997e-06 2024-09-20T22:56:59.880 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.880 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.880 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:56:59.880 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:16.392082+0000", 2024-09-20T22:56:59.881 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.8200000000000003e-06 2024-09-20T22:56:59.881 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.881 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.881 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:56:59.881 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:16.392090+0000", 2024-09-20T22:56:59.881 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.7160000000000003e-06 2024-09-20T22:56:59.881 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.881 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.881 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:56:59.881 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:16.392637+0000", 2024-09-20T22:56:59.881 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00054682600000000004 2024-09-20T22:56:59.881 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.881 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.882 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:56:59.882 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:16.392657+0000", 2024-09-20T22:56:59.882 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9956e-05 2024-09-20T22:56:59.882 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.882 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.882 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:56:59.882 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:16.401999+0000", 2024-09-20T22:56:59.882 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0093422560000000002 2024-09-20T22:56:59.882 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.882 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.882 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:56:59.882 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:16.402056+0000", 2024-09-20T22:56:59.882 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.6986e-05 2024-09-20T22:56:59.882 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:56:59.883 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:56:59.883 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:56:59.883 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.883 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.883 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(osd.0.0:1267 7.4 e596/460 7:24b1a612:::smithi08477203-43:aa v 596'321, mlcod=592'317)", 2024-09-20T22:56:59.883 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:51:16.392654+0000", 2024-09-20T22:56:59.883 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 343.45441325500002, 2024-09-20T22:56:59.883 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0093429540000000005, 2024-09-20T22:56:59.883 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:56:59.883 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:56:59.883 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:56:59.883 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:56:59.883 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.883 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:56:59.884 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:16.392654+0000", 2024-09-20T22:56:59.884 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:56:59.884 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.884 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.884 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:56:59.884 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:16.392654+0000", 2024-09-20T22:56:59.884 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:56:59.884 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.884 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.884 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:56:59.884 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:16.392654+0000", 2024-09-20T22:56:59.884 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.4300000000000002e-07 2024-09-20T22:56:59.884 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.885 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.885 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:56:59.885 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:16.392664+0000", 2024-09-20T22:56:59.885 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.2830000000000007e-06 2024-09-20T22:56:59.885 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.885 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.885 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:56:59.885 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:16.392671+0000", 2024-09-20T22:56:59.885 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.3499999999999999e-06 2024-09-20T22:56:59.885 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.885 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.885 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:56:59.885 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:16.392677+0000", 2024-09-20T22:56:59.885 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.4590000000000003e-06 2024-09-20T22:56:59.885 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.885 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.886 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:56:59.886 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:16.392753+0000", 2024-09-20T22:56:59.886 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.5387000000000002e-05 2024-09-20T22:56:59.886 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.886 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.886 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:56:59.886 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:16.392776+0000", 2024-09-20T22:56:59.886 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.3703e-05 2024-09-20T22:56:59.886 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.886 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.886 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:56:59.886 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:16.401944+0000", 2024-09-20T22:56:59.886 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0091675100000000002 2024-09-20T22:56:59.887 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.887 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.887 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:56:59.887 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:16.401996+0000", 2024-09-20T22:56:59.887 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.2519000000000003e-05 2024-09-20T22:56:59.887 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:56:59.887 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:56:59.887 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:56:59.887 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.887 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.887 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(osd.0.0:1268 7.4 e596/460 7:2f8647a2:::smithi08477203-41:aa v 596'323, mlcod=592'317)", 2024-09-20T22:56:59.887 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:51:16.392986+0000", 2024-09-20T22:56:59.887 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 343.45408114999998, 2024-09-20T22:56:59.887 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0091905690000000009, 2024-09-20T22:56:59.888 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:56:59.888 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:56:59.888 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:56:59.888 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:56:59.888 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.888 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:56:59.888 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:16.392986+0000", 2024-09-20T22:56:59.888 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:56:59.888 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.888 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.888 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:56:59.888 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:16.392986+0000", 2024-09-20T22:56:59.888 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:56:59.888 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.888 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.889 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:56:59.889 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:16.392986+0000", 2024-09-20T22:56:59.889 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.1099999999999995e-07 2024-09-20T22:56:59.889 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.889 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.889 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:56:59.889 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:16.392992+0000", 2024-09-20T22:56:59.889 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.7089999999999999e-06 2024-09-20T22:56:59.889 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.889 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.889 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:56:59.889 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:16.393000+0000", 2024-09-20T22:56:59.889 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.5229999999999995e-06 2024-09-20T22:56:59.889 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.890 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.890 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:56:59.890 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:16.393007+0000", 2024-09-20T22:56:59.890 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.5169999999999996e-06 2024-09-20T22:56:59.890 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.890 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.890 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:56:59.890 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:16.393223+0000", 2024-09-20T22:56:59.890 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00021552600000000001 2024-09-20T22:56:59.890 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.890 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.890 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:56:59.890 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:16.393245+0000", 2024-09-20T22:56:59.890 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.2172e-05 2024-09-20T22:56:59.891 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.891 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.891 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:56:59.891 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:16.402143+0000", 2024-09-20T22:56:59.891 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.008898418 2024-09-20T22:56:59.891 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.891 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.891 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:56:59.891 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:16.402176+0000", 2024-09-20T22:56:59.891 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.3093000000000003e-05 2024-09-20T22:56:59.891 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:56:59.891 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:56:59.891 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:56:59.891 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.892 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.892 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:6000 7.9 e604/602 7:9eac4560:::smithi08477203-30:head v 604'886, mlcod=604'882)", 2024-09-20T22:56:59.892 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:51:24.304331+0000", 2024-09-20T22:56:59.892 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 335.54273611799999, 2024-09-20T22:56:59.892 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.014326934, 2024-09-20T22:56:59.892 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:56:59.892 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:56:59.892 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:56:59.892 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:56:59.892 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.892 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:56:59.892 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:24.304331+0000", 2024-09-20T22:56:59.892 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:56:59.892 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.893 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.893 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:56:59.893 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:24.304331+0000", 2024-09-20T22:56:59.893 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:56:59.893 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.893 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.893 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:56:59.893 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:24.304332+0000", 2024-09-20T22:56:59.893 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.8400000000000002e-07 2024-09-20T22:56:59.893 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.893 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.893 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:56:59.893 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:24.310538+0000", 2024-09-20T22:56:59.893 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0062066710000000004 2024-09-20T22:56:59.894 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.894 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.894 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:56:59.894 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:24.310568+0000", 2024-09-20T22:56:59.894 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.9283000000000001e-05 2024-09-20T22:56:59.894 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.894 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.894 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:56:59.894 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:24.310576+0000", 2024-09-20T22:56:59.894 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.3639999999999999e-06 2024-09-20T22:56:59.894 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.894 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.894 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:56:59.894 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:24.310641+0000", 2024-09-20T22:56:59.895 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.4690000000000006e-05 2024-09-20T22:56:59.895 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.895 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.895 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:56:59.895 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:24.310665+0000", 2024-09-20T22:56:59.895 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.4635999999999999e-05 2024-09-20T22:56:59.895 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.895 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.895 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:56:59.895 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:24.318613+0000", 2024-09-20T22:56:59.895 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0079475429999999996 2024-09-20T22:56:59.895 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.895 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.896 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:56:59.896 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:24.318658+0000", 2024-09-20T22:56:59.896 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.4762999999999998e-05 2024-09-20T22:56:59.896 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:56:59.896 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:56:59.896 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:56:59.896 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.896 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.896 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:6253 7.9 e629/602 7:91f2d47c:::smithi08477203-8:head v 629'948, mlcod=627'945)", 2024-09-20T22:56:59.896 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:51:49.625685+0000", 2024-09-20T22:56:59.896 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 310.22138188999998, 2024-09-20T22:56:59.896 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.017501778999999999, 2024-09-20T22:56:59.896 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:56:59.896 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:56:59.896 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:56:59.897 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:56:59.897 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.897 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:56:59.897 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:49.625685+0000", 2024-09-20T22:56:59.897 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:56:59.897 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.897 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.897 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:56:59.897 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:49.625685+0000", 2024-09-20T22:56:59.897 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:56:59.897 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.897 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.897 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:56:59.897 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:49.625686+0000", 2024-09-20T22:56:59.898 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.457e-06 2024-09-20T22:56:59.898 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.898 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.898 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:56:59.898 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:49.632417+0000", 2024-09-20T22:56:59.898 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.006730453 2024-09-20T22:56:59.898 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.898 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.898 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:56:59.898 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:49.632440+0000", 2024-09-20T22:56:59.898 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.2999000000000001e-05 2024-09-20T22:56:59.898 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.898 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.898 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:56:59.899 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:49.632450+0000", 2024-09-20T22:56:59.899 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0210999999999999e-05 2024-09-20T22:56:59.899 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.899 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.899 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:56:59.899 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:49.632524+0000", 2024-09-20T22:56:59.899 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.4234999999999997e-05 2024-09-20T22:56:59.899 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.899 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.899 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:56:59.899 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:49.632549+0000", 2024-09-20T22:56:59.899 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.4572000000000001e-05 2024-09-20T22:56:59.899 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.899 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.900 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:56:59.900 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:49.643131+0000", 2024-09-20T22:56:59.900 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.010582446000000001 2024-09-20T22:56:59.900 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.900 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.900 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:56:59.900 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:49.643187+0000", 2024-09-20T22:56:59.900 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.5405999999999998e-05 2024-09-20T22:56:59.900 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:56:59.900 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:56:59.900 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:56:59.900 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.900 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.900 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:6360 7.8 e638/430 7:1e9e2524:::smithi08477203-5:head v 638'681, mlcod=637'677)", 2024-09-20T22:56:59.901 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:51:58.286846+0000", 2024-09-20T22:56:59.901 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 301.56022047300002, 2024-09-20T22:56:59.901 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.012344572999999999, 2024-09-20T22:56:59.901 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:56:59.901 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:56:59.901 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:56:59.901 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:56:59.901 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.901 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:56:59.901 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:58.286846+0000", 2024-09-20T22:56:59.901 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:56:59.901 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.901 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.901 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:56:59.902 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:58.286846+0000", 2024-09-20T22:56:59.902 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:56:59.902 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.902 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.902 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:56:59.902 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:58.286847+0000", 2024-09-20T22:56:59.902 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.3700000000000001e-07 2024-09-20T22:56:59.902 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.902 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.902 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:56:59.902 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:58.297345+0000", 2024-09-20T22:56:59.902 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.010498535 2024-09-20T22:56:59.902 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.903 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.903 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:56:59.903 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:58.297359+0000", 2024-09-20T22:56:59.903 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3631999999999999e-05 2024-09-20T22:56:59.903 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.903 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.903 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:56:59.903 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:58.297365+0000", 2024-09-20T22:56:59.903 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.4489999999999996e-06 2024-09-20T22:56:59.903 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.903 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.903 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:56:59.903 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:58.297433+0000", 2024-09-20T22:56:59.903 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.7206000000000007e-05 2024-09-20T22:56:59.904 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.904 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.904 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:56:59.904 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:58.297455+0000", 2024-09-20T22:56:59.904 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.2711999999999999e-05 2024-09-20T22:56:59.904 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.904 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.904 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:56:59.904 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:58.299091+0000", 2024-09-20T22:56:59.904 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0016354010000000001 2024-09-20T22:56:59.904 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.904 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.904 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:56:59.904 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:58.299191+0000", 2024-09-20T22:56:59.905 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000100201 2024-09-20T22:56:59.905 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:56:59.905 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:56:59.905 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:56:59.905 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.905 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.905 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:7267 7.1 7:84a866f5:::smithi08477203-27:head [copy-from ver 1805] snapc 149=[149,148,147,146,145,144,142,141,140,13e,13d,13c,13b,13a,139,138,136,135,132,130,12f,12e,12d,12c,12a,128,127,124,121,11f,112,103,ed] ondisk+write+known_if_redirected+supports_pool_eio e728)", 2024-09-20T22:56:59.905 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:53:28.329140+0000", 2024-09-20T22:56:59.905 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 211.51792676100001, 2024-09-20T22:56:59.905 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.016910652000000002, 2024-09-20T22:56:59.905 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:56:59.905 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:56:59.905 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:56:59.905 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:56:59.906 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:56:59.906 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:56:59.906 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 7267 2024-09-20T22:56:59.906 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.906 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:56:59.906 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.906 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:56:59.906 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:28.329140+0000", 2024-09-20T22:56:59.906 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:56:59.906 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.906 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.906 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:56:59.906 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:28.329140+0000", 2024-09-20T22:56:59.906 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:56:59.907 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.907 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.907 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:56:59.907 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:28.329142+0000", 2024-09-20T22:56:59.907 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.4260000000000002e-06 2024-09-20T22:56:59.907 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.907 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.907 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:56:59.907 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:28.329152+0000", 2024-09-20T22:56:59.907 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.3519999999999996e-06 2024-09-20T22:56:59.907 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.907 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.907 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:56:59.907 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:28.329170+0000", 2024-09-20T22:56:59.907 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.7748e-05 2024-09-20T22:56:59.908 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.908 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.908 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:56:59.908 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:28.329179+0000", 2024-09-20T22:56:59.908 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.6979999999999993e-06 2024-09-20T22:56:59.908 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.908 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.908 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:56:59.908 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:28.329253+0000", 2024-09-20T22:56:59.908 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.3708000000000006e-05 2024-09-20T22:56:59.908 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.908 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.908 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:56:59.908 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:28.329354+0000", 2024-09-20T22:56:59.909 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00010073099999999999 2024-09-20T22:56:59.909 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.909 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.909 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 3,6", 2024-09-20T22:56:59.909 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:28.336766+0000", 2024-09-20T22:56:59.909 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0074122010000000002 2024-09-20T22:56:59.909 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.909 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.909 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:56:59.909 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:28.345430+0000", 2024-09-20T22:56:59.909 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0086638119999999999 2024-09-20T22:56:59.909 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.909 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.909 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:56:59.909 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:28.345800+0000", 2024-09-20T22:56:59.910 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00037052199999999997 2024-09-20T22:56:59.910 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.910 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.910 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:56:59.910 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:28.345963+0000", 2024-09-20T22:56:59.910 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000163077 2024-09-20T22:56:59.910 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.910 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.910 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:56:59.910 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:28.346010+0000", 2024-09-20T22:56:59.910 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.6338999999999998e-05 2024-09-20T22:56:59.910 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.910 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.911 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:56:59.911 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:28.346051+0000", 2024-09-20T22:56:59.911 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.1038000000000002e-05 2024-09-20T22:56:59.911 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:56:59.911 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:56:59.911 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:56:59.911 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.911 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.911 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:7268 7.1 7:84a866f5:::smithi08477203-27:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e728)", 2024-09-20T22:56:59.911 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:53:28.329202+0000", 2024-09-20T22:56:59.911 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 211.51786499799999, 2024-09-20T22:56:59.911 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.012525870999999999, 2024-09-20T22:56:59.911 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:56:59.911 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:56:59.911 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:56:59.912 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:56:59.912 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:56:59.912 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:56:59.912 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 7268 2024-09-20T22:56:59.912 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.912 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:56:59.912 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.919 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:56:59.919 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:28.329202+0000", 2024-09-20T22:56:59.919 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:56:59.919 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.919 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.919 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:56:59.919 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:28.329202+0000", 2024-09-20T22:56:59.919 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:56:59.919 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.920 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.920 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:56:59.920 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:28.329202+0000", 2024-09-20T22:56:59.920 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.7400000000000003e-07 2024-09-20T22:56:59.920 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.920 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.920 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:56:59.920 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:28.329206+0000", 2024-09-20T22:56:59.920 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.1e-06 2024-09-20T22:56:59.920 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.920 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.920 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:56:59.920 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:28.329213+0000", 2024-09-20T22:56:59.920 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.3520000000000001e-06 2024-09-20T22:56:59.921 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.921 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.921 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:56:59.921 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:28.329229+0000", 2024-09-20T22:56:59.921 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.6169999999999999e-05 2024-09-20T22:56:59.921 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.921 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.921 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:56:59.921 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:28.329486+0000", 2024-09-20T22:56:59.921 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000256703 2024-09-20T22:56:59.921 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.921 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.921 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for blocked object", 2024-09-20T22:56:59.921 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:28.329541+0000", 2024-09-20T22:56:59.922 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.5158000000000003e-05 2024-09-20T22:56:59.922 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.922 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.922 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:56:59.922 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:28.341541+0000", 2024-09-20T22:56:59.922 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.011999774 2024-09-20T22:56:59.922 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.922 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.922 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:56:59.922 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:28.341634+0000", 2024-09-20T22:56:59.922 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.3271999999999997e-05 2024-09-20T22:56:59.922 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.922 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.922 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:56:59.923 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:28.341728+0000", 2024-09-20T22:56:59.923 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.3667999999999998e-05 2024-09-20T22:56:59.923 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:56:59.923 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:56:59.923 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:56:59.923 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.923 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.923 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:7387 7.9 e743/602 7:907aec0e:::smithi08477203-3:head v 743'1242, mlcod=741'1234)", 2024-09-20T22:56:59.923 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:53:43.309715+0000", 2024-09-20T22:56:59.923 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 196.537352003, 2024-09-20T22:56:59.923 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.030438433000000001, 2024-09-20T22:56:59.923 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:56:59.923 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:56:59.923 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:56:59.924 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:56:59.924 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.924 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:56:59.924 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:43.309715+0000", 2024-09-20T22:56:59.924 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:56:59.924 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.924 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.924 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:56:59.924 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:43.309715+0000", 2024-09-20T22:56:59.924 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:56:59.924 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.924 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.924 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:56:59.924 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:43.309716+0000", 2024-09-20T22:56:59.925 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.023e-06 2024-09-20T22:56:59.925 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.925 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.925 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:56:59.925 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:43.336825+0000", 2024-09-20T22:56:59.925 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.027108795000000002 2024-09-20T22:56:59.925 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.925 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.925 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:56:59.925 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:43.336837+0000", 2024-09-20T22:56:59.925 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2751e-05 2024-09-20T22:56:59.925 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.925 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.925 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:56:59.926 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:43.336843+0000", 2024-09-20T22:56:59.926 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.8969999999999996e-06 2024-09-20T22:56:59.926 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.926 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.926 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:56:59.926 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:43.336894+0000", 2024-09-20T22:56:59.926 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.1201e-05 2024-09-20T22:56:59.926 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.926 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.926 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:56:59.926 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:43.336911+0000", 2024-09-20T22:56:59.926 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.6062e-05 2024-09-20T22:56:59.926 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.926 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.927 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:56:59.927 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:43.340108+0000", 2024-09-20T22:56:59.927 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0031979109999999999 2024-09-20T22:56:59.927 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.927 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.927 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:56:59.927 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:43.340153+0000", 2024-09-20T22:56:59.927 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.4793000000000003e-05 2024-09-20T22:56:59.927 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:56:59.927 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:56:59.927 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:56:59.927 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.927 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.927 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:7392 7.3 e743/464 7:cae5792a:::smithi08477203-32:head v 743'338, mlcod=738'334)", 2024-09-20T22:56:59.927 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:53:43.328452+0000", 2024-09-20T22:56:59.928 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 196.51861518499999, 2024-09-20T22:56:59.928 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.013073611000000001, 2024-09-20T22:56:59.928 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:56:59.928 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:56:59.928 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:56:59.928 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:56:59.928 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.928 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:56:59.928 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:43.328452+0000", 2024-09-20T22:56:59.928 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:56:59.928 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.928 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.928 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:56:59.928 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:43.328452+0000", 2024-09-20T22:56:59.929 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:56:59.929 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.929 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.929 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:56:59.929 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:43.328452+0000", 2024-09-20T22:56:59.929 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.6199999999999999e-07 2024-09-20T22:56:59.929 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.929 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.929 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:56:59.929 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:43.339025+0000", 2024-09-20T22:56:59.929 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.010572936999999999 2024-09-20T22:56:59.929 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.929 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.930 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:56:59.931 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:56:59.931 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:43.339079+0000", 2024-09-20T22:56:59.931 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.3888999999999999e-05 2024-09-20T22:56:59.931 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.931 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.931 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:56:59.932 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:43.339087+0000", 2024-09-20T22:56:59.932 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.2330000000000006e-06 2024-09-20T22:56:59.932 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.932 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.932 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:56:59.932 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:43.339211+0000", 2024-09-20T22:56:59.932 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00012425199999999999 2024-09-20T22:56:59.932 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.932 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.932 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:56:59.932 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:43.339229+0000", 2024-09-20T22:56:59.932 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.8403999999999999e-05 2024-09-20T22:56:59.932 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.933 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.933 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:56:59.933 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:43.341475+0000", 2024-09-20T22:56:59.933 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0022457390000000001 2024-09-20T22:56:59.933 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.933 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.933 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:56:59.933 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:43.341525+0000", 2024-09-20T22:56:59.933 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.9994999999999998e-05 2024-09-20T22:56:59.933 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:56:59.933 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:56:59.933 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:56:59.933 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.933 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.933 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:7549 7.4 e755/602 7:2f8647a2:::smithi08477203-41:head v 755'514, mlcod=754'510)", 2024-09-20T22:56:59.934 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:53:55.515416+0000", 2024-09-20T22:56:59.934 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 184.331650819, 2024-09-20T22:56:59.934 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0096741410000000007, 2024-09-20T22:56:59.934 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:56:59.934 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:56:59.934 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:56:59.934 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:56:59.934 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.934 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:56:59.934 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:55.515416+0000", 2024-09-20T22:56:59.934 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:56:59.934 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.934 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.935 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:56:59.935 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:55.515416+0000", 2024-09-20T22:56:59.935 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:56:59.935 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.935 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.935 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:56:59.935 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:55.515418+0000", 2024-09-20T22:56:59.935 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.1169999999999998e-06 2024-09-20T22:56:59.935 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.935 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.935 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:56:59.935 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:55.520055+0000", 2024-09-20T22:56:59.935 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0046364709999999996 2024-09-20T22:56:59.935 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.936 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.936 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:56:59.936 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:55.520074+0000", 2024-09-20T22:56:59.936 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9219e-05 2024-09-20T22:56:59.936 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.936 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.936 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:56:59.936 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:55.520084+0000", 2024-09-20T22:56:59.936 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0064000000000001e-05 2024-09-20T22:56:59.936 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.936 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.936 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:56:59.936 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:55.520140+0000", 2024-09-20T22:56:59.936 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.6444999999999999e-05 2024-09-20T22:56:59.937 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.937 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.937 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:56:59.937 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:55.520158+0000", 2024-09-20T22:56:59.937 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.7720999999999999e-05 2024-09-20T22:56:59.937 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.937 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.937 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:56:59.937 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:55.525049+0000", 2024-09-20T22:56:59.937 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0048913890000000003 2024-09-20T22:56:59.937 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.937 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.937 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:56:59.937 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:55.525090+0000", 2024-09-20T22:56:59.938 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.0714999999999999e-05 2024-09-20T22:56:59.938 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:56:59.938 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:56:59.938 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:56:59.938 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.938 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.938 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:8049 7.9 e810/779 7:907aec0e:::smithi08477203-3:head v 810'1428, mlcod=808'1424)", 2024-09-20T22:56:59.938 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:54:50.354916+0000", 2024-09-20T22:56:59.938 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 129.49215116400001, 2024-09-20T22:56:59.938 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.018163367999999999, 2024-09-20T22:56:59.938 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:56:59.938 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:56:59.938 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:56:59.939 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:56:59.939 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.939 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:56:59.939 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:54:50.354916+0000", 2024-09-20T22:56:59.939 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:56:59.939 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.939 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.939 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:56:59.939 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:54:50.354916+0000", 2024-09-20T22:56:59.939 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:56:59.939 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.939 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.939 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:56:59.939 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:54:50.354916+0000", 2024-09-20T22:56:59.939 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.7099999999999998e-07 2024-09-20T22:56:59.940 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.940 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.940 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:56:59.940 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:54:50.359771+0000", 2024-09-20T22:56:59.940 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0048552090000000001 2024-09-20T22:56:59.940 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.940 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.940 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:56:59.940 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:54:50.359786+0000", 2024-09-20T22:56:59.940 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4430000000000001e-05 2024-09-20T22:56:59.940 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.940 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.940 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:56:59.941 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:54:50.359796+0000", 2024-09-20T22:56:59.941 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0539e-05 2024-09-20T22:56:59.941 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.941 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.941 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:56:59.941 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:54:50.359882+0000", 2024-09-20T22:56:59.941 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.5795000000000001e-05 2024-09-20T22:56:59.941 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.941 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.941 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:56:59.941 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:54:50.359906+0000", 2024-09-20T22:56:59.941 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.4528e-05 2024-09-20T22:56:59.941 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.941 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.941 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:56:59.942 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:54:50.372996+0000", 2024-09-20T22:56:59.942 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.013089987000000001 2024-09-20T22:56:59.942 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.942 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.942 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:56:59.942 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:54:50.373079+0000", 2024-09-20T22:56:59.942 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.2608999999999995e-05 2024-09-20T22:56:59.942 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:56:59.942 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:56:59.942 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:56:59.942 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.942 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.942 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:8056 7.9 e810/779 7:9970b578:::smithi08477203-15:head v 810'1429, mlcod=808'1424)", 2024-09-20T22:56:59.943 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:54:50.364589+0000", 2024-09-20T22:56:59.943 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 129.48247743300001, 2024-09-20T22:56:59.943 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0085585869999999994, 2024-09-20T22:56:59.943 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:56:59.943 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:56:59.943 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:56:59.943 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:56:59.943 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.943 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:56:59.943 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:54:50.364589+0000", 2024-09-20T22:56:59.943 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:56:59.943 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.943 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.943 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:56:59.944 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:54:50.364589+0000", 2024-09-20T22:56:59.944 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:56:59.944 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.944 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.944 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:56:59.944 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:54:50.364599+0000", 2024-09-20T22:56:59.944 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.2639999999999997e-06 2024-09-20T22:56:59.944 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.944 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.944 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:56:59.944 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:54:50.365506+0000", 2024-09-20T22:56:59.944 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00090696699999999995 2024-09-20T22:56:59.944 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.944 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.945 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:56:59.945 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:54:50.365528+0000", 2024-09-20T22:56:59.945 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.2705999999999999e-05 2024-09-20T22:56:59.945 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.945 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.945 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:56:59.945 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:54:50.365538+0000", 2024-09-20T22:56:59.945 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.9709999999999999e-06 2024-09-20T22:56:59.945 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.945 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.945 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:56:59.945 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:54:50.365607+0000", 2024-09-20T22:56:59.945 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.8423000000000005e-05 2024-09-20T22:56:59.945 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.946 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.946 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:56:59.946 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:54:50.365626+0000", 2024-09-20T22:56:59.946 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9028000000000001e-05 2024-09-20T22:56:59.946 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.946 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.946 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:56:59.946 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:54:50.373126+0000", 2024-09-20T22:56:59.946 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00749996 2024-09-20T22:56:59.946 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.946 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.946 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:56:59.946 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:54:50.373148+0000", 2024-09-20T22:56:59.946 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.2268000000000001e-05 2024-09-20T22:56:59.947 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:56:59.947 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:56:59.947 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:56:59.947 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.947 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.947 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:8990 7.8 e903/890 7:12e9c8db:::smithi08477203-11:head v 903'1336, mlcod=902'1332)", 2024-09-20T22:56:59.947 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:56:23.359570+0000", 2024-09-20T22:56:59.947 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 36.487496665999998, 2024-09-20T22:56:59.947 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.011802417000000001, 2024-09-20T22:56:59.947 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:56:59.947 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:56:59.947 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:56:59.947 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:56:59.947 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.948 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:56:59.948 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:56:23.359570+0000", 2024-09-20T22:56:59.948 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:56:59.948 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.948 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.948 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:56:59.948 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:56:23.359570+0000", 2024-09-20T22:56:59.948 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:56:59.948 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.948 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.948 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:56:59.948 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:56:23.359571+0000", 2024-09-20T22:56:59.948 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.3000000000000001e-07 2024-09-20T22:56:59.948 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.949 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.949 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:56:59.949 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:56:23.364707+0000", 2024-09-20T22:56:59.949 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0051359589999999998 2024-09-20T22:56:59.949 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.949 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.949 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:56:59.949 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:56:23.364725+0000", 2024-09-20T22:56:59.949 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.8544000000000001e-05 2024-09-20T22:56:59.949 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.949 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.949 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:56:59.949 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:56:23.364736+0000", 2024-09-20T22:56:59.949 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1257000000000001e-05 2024-09-20T22:56:59.949 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.950 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.950 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:56:59.950 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:56:23.364791+0000", 2024-09-20T22:56:59.950 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.4345999999999998e-05 2024-09-20T22:56:59.950 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.950 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.950 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:56:59.950 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:56:23.364820+0000", 2024-09-20T22:56:59.950 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.94e-05 2024-09-20T22:56:59.950 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.950 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.950 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:56:59.950 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:56:23.371316+0000", 2024-09-20T22:56:59.950 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0064956349999999996 2024-09-20T22:56:59.951 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.951 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.951 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:56:59.951 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:56:23.371373+0000", 2024-09-20T22:56:59.951 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.6746000000000002e-05 2024-09-20T22:56:59.951 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:56:59.951 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:56:59.951 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:56:59.951 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.951 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.951 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:9018 7.1 e905/836 7:8422f82d:::smithi08477203-22:head v 905'545, mlcod=901'542)", 2024-09-20T22:56:59.951 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:56:25.646733+0000", 2024-09-20T22:56:59.951 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 34.200333493000002, 2024-09-20T22:56:59.951 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.017896835999999999, 2024-09-20T22:56:59.951 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:56:59.952 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:56:59.952 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:56:59.952 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:56:59.952 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.952 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:56:59.952 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:56:25.646733+0000", 2024-09-20T22:56:59.952 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:56:59.952 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.952 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.952 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:56:59.952 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:56:25.646733+0000", 2024-09-20T22:56:59.952 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:56:59.952 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.952 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.953 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:56:59.953 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:56:25.646734+0000", 2024-09-20T22:56:59.953 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.0800000000000001e-07 2024-09-20T22:56:59.953 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.953 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.953 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:56:59.953 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:56:25.653592+0000", 2024-09-20T22:56:59.953 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0068581400000000004 2024-09-20T22:56:59.953 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.953 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.953 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:56:59.953 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:56:25.653608+0000", 2024-09-20T22:56:59.953 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.662e-05 2024-09-20T22:56:59.953 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.953 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.954 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:56:59.954 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:56:25.653619+0000", 2024-09-20T22:56:59.954 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1109e-05 2024-09-20T22:56:59.954 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.954 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.954 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:56:59.954 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:56:25.653710+0000", 2024-09-20T22:56:59.954 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.0909999999999995e-05 2024-09-20T22:56:59.954 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.954 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.954 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:56:59.954 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:56:25.653745+0000", 2024-09-20T22:56:59.954 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.4926999999999999e-05 2024-09-20T22:56:59.954 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.955 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.955 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:56:59.955 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:56:25.664567+0000", 2024-09-20T22:56:59.955 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.010821357 2024-09-20T22:56:59.955 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:56:59.955 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:56:59.955 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:56:59.955 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:56:25.664630+0000", 2024-09-20T22:56:59.955 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.3565000000000006e-05 2024-09-20T22:56:59.955 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:56:59.955 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:56:59.955 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:56:59.955 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:56:59.955 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:56:59.956 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-20T22:56:59.977 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:56:59.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:56:59.977 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:57:00.004 INFO:journalctl@ceph.iscsi.iscsi.a.smithi154.stdout:Sep 20 22:56:59 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-iscsi-iscsi-a[59730]: debug there is no tcmu-runner data available 2024-09-20T22:57:00.079 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:57:00.099 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:00.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:00.100 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:57:00.201 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:57:00.223 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:00.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:00.224 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:57:00.244 INFO:journalctl@ceph.osd.3.smithi084.stdout:Sep 20 22:56:59 smithi084 systemd[1]: Stopping Ceph osd.3 for a9c0bc70-77a0-11ef-baf5-efdc52797490... 2024-09-20T22:57:00.295 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:57:00.325 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:57:00.347 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:00.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:00.348 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:57:00.415 INFO:tasks.rados.rados.0.smithi084.stdout:3805: done (0 left) 2024-09-20T22:57:00.415 INFO:tasks.rados.rados.0.smithi084.stdout:3806: copy_from oid 19 from oid 16 current snap is 441 2024-09-20T22:57:00.415 INFO:tasks.rados.rados.0.smithi084.stdout:3807: write oid 24 current snap is 441 2024-09-20T22:57:00.415 INFO:tasks.rados.rados.0.smithi084.stdout:3807: seq_num 1298 ranges {615486=653212,1889392=498119,2978574=475308} 2024-09-20T22:57:00.416 INFO:tasks.rados.rados.0.smithi084.stdout:3807: writing smithi08477203-24 from 615486 to 1268698 tid 1 2024-09-20T22:57:00.424 INFO:tasks.rados.rados.0.smithi084.stdout:3807: writing smithi08477203-24 from 1889392 to 2387511 tid 2 2024-09-20T22:57:00.430 INFO:tasks.rados.rados.0.smithi084.stdout:3807: writing smithi08477203-24 from 2978574 to 3453882 tid 3 2024-09-20T22:57:00.430 INFO:tasks.rados.rados.0.smithi084.stdout:3808: read oid 42 snap -1 2024-09-20T22:57:00.431 INFO:tasks.rados.rados.0.smithi084.stdout:3808: expect (ObjNum 1277 snap 433 seq_num 1277) 2024-09-20T22:57:00.431 INFO:tasks.rados.rados.0.smithi084.stdout:3809: rollback oid 22 current snap is 441 2024-09-20T22:57:00.431 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 22 to 430 2024-09-20T22:57:00.431 INFO:tasks.rados.rados.0.smithi084.stdout:3810: copy_from oid 5 from oid 46 current snap is 441 2024-09-20T22:57:00.431 INFO:tasks.rados.rados.0.smithi084.stdout:3811: read oid 13 snap -1 2024-09-20T22:57:00.431 INFO:tasks.rados.rados.0.smithi084.stdout:3811: expect deleted 2024-09-20T22:57:00.431 INFO:tasks.rados.rados.0.smithi084.stdout:3812: snap_remove snap 361 2024-09-20T22:57:00.449 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:57:00.471 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:00.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:00.471 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:57:00.540 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:00 smithi084 ceph-mon[40024]: osdmap e939: 8 total, 8 up, 7 in 2024-09-20T22:57:00.540 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:00 smithi084 ceph-mon[40024]: osd.3 marked itself down and dead 2024-09-20T22:57:00.540 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:00 smithi084 ceph-mon[29774]: osdmap e939: 8 total, 8 up, 7 in 2024-09-20T22:57:00.540 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:00 smithi084 ceph-mon[29774]: osd.3 marked itself down and dead 2024-09-20T22:57:00.540 INFO:journalctl@ceph.osd.3.smithi084.stdout:Sep 20 22:57:00 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-3[61309]: 2024-09-20T22:57:00.242+0000 7f468c0e1640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-20T22:57:00.541 INFO:journalctl@ceph.osd.3.smithi084.stdout:Sep 20 22:57:00 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-3[61309]: 2024-09-20T22:57:00.242+0000 7f468c0e1640 -1 osd.3 939 *** Got signal Terminated *** 2024-09-20T22:57:00.541 INFO:journalctl@ceph.osd.3.smithi084.stdout:Sep 20 22:57:00 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-3[61309]: 2024-09-20T22:57:00.242+0000 7f468c0e1640 -1 osd.3 939 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-20T22:57:00.572 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:57:00.577 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:00 smithi154 ceph-mon[33264]: osdmap e939: 8 total, 8 up, 7 in 2024-09-20T22:57:00.577 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:00 smithi154 ceph-mon[33264]: osd.3 marked itself down and dead 2024-09-20T22:57:00.594 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:00.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:00.595 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:57:00.606 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-20T22:57:00.696 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:57:00.717 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:00.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:00.718 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:57:00.795 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:57:00.795 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:57:00.795 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:57:00.795 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:57:00.803 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:57:00.803 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:57:00.804 INFO:teuthology.orchestra.run.smithi084.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-20T22:57:00.819 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:57:00.822 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:57:00.822 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:57:00.839 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:57:00.839 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:57:00.842 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:00.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:00.843 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:57:00.856 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:57:00.856 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:57:00.872 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:57:00.872 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:57:00.894 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:57:00.944 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:57:00.966 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:00.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:00.967 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:57:01.067 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:57:01.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:01.090 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:01.090 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:57:01.161 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd unset noscrub 2024-09-20T22:57:01.191 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:57:01.224 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:01.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:01.225 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:57:01.326 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:57:01.347 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:01.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:01.348 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:57:01.400 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-20T22:57:01.400 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:57:01.405 INFO:tasks.rados.rados.0.smithi084.stdout:3812: done (6 left) 2024-09-20T22:57:01.405 INFO:tasks.rados.rados.0.smithi084.stdout:3813: delete oid 30 current snap is 441 2024-09-20T22:57:01.406 INFO:tasks.rados.rados.0.smithi084.stdout:3810: finishing copy_from to smithi08477203-5 2024-09-20T22:57:01.406 INFO:tasks.rados.rados.0.smithi084.stdout:3810: got expected ENOENT (src dne) 2024-09-20T22:57:01.406 INFO:tasks.rados.rados.0.smithi084.stdout:3810: finishing copy_from racing read to smithi08477203-5 2024-09-20T22:57:01.406 INFO:tasks.rados.rados.0.smithi084.stdout:3808: expect (ObjNum 1277 snap 433 seq_num 1277) 2024-09-20T22:57:01.449 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:57:01.472 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:01.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:01.473 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:57:01.545 INFO:tasks.rados.rados.0.smithi084.stdout:3808: done (6 left) 2024-09-20T22:57:01.545 INFO:tasks.rados.rados.0.smithi084.stdout:3810: done (5 left) 2024-09-20T22:57:01.546 INFO:tasks.rados.rados.0.smithi084.stdout:3813: done (4 left) 2024-09-20T22:57:01.546 INFO:tasks.rados.rados.0.smithi084.stdout:3814: snap_remove snap 429 2024-09-20T22:57:01.574 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:57:01.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:01.597 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:01.597 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:57:01.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:01 smithi084 ceph-mon[29774]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:57:01.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:01 smithi084 ceph-mon[29774]: pgmap v1311: 168 pgs: 3 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 5 active+remapped+backfill_toofull, 156 active+clean; 732 MiB data, 2.3 GiB used, 624 GiB / 626 GiB avail; 20 MiB/s rd, 6.7 MiB/s wr, 38 op/s; 102/1593 objects misplaced (6.403%); 31 MiB/s, 14 objects/s recovering 2024-09-20T22:57:01.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:01 smithi084 ceph-mon[29774]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-20T22:57:01.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:01 smithi084 ceph-mon[29774]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-20T22:57:01.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:01 smithi084 ceph-mon[29774]: osdmap e940: 8 total, 7 up, 7 in 2024-09-20T22:57:01.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:01 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/2973540493' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:57:01.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:01 smithi084 ceph-mon[40024]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:57:01.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:01 smithi084 ceph-mon[40024]: pgmap v1311: 168 pgs: 3 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 5 active+remapped+backfill_toofull, 156 active+clean; 732 MiB data, 2.3 GiB used, 624 GiB / 626 GiB avail; 20 MiB/s rd, 6.7 MiB/s wr, 38 op/s; 102/1593 objects misplaced (6.403%); 31 MiB/s, 14 objects/s recovering 2024-09-20T22:57:01.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:01 smithi084 ceph-mon[40024]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-20T22:57:01.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:01 smithi084 ceph-mon[40024]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-20T22:57:01.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:01 smithi084 ceph-mon[40024]: osdmap e940: 8 total, 7 up, 7 in 2024-09-20T22:57:01.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:01 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/2973540493' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:57:01.677 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-20T22:57:01.678 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-20T22:57:01.698 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:57:01.739 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:01.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:01.740 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:57:01.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:01 smithi154 ceph-mon[33264]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:57:01.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:01 smithi154 ceph-mon[33264]: pgmap v1311: 168 pgs: 3 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 5 active+remapped+backfill_toofull, 156 active+clean; 732 MiB data, 2.3 GiB used, 624 GiB / 626 GiB avail; 20 MiB/s rd, 6.7 MiB/s wr, 38 op/s; 102/1593 objects misplaced (6.403%); 31 MiB/s, 14 objects/s recovering 2024-09-20T22:57:01.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:01 smithi154 ceph-mon[33264]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-20T22:57:01.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:01 smithi154 ceph-mon[33264]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-20T22:57:01.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:01 smithi154 ceph-mon[33264]: osdmap e940: 8 total, 7 up, 7 in 2024-09-20T22:57:01.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:01 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/2973540493' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:57:01.841 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:57:01.862 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:01.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:01.863 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:57:01.955 INFO:journalctl@ceph.osd.3.smithi084.stdout:Sep 20 22:57:01 smithi084 podman[206062]: 2024-09-20 22:57:01.68816363 +0000 UTC m=+1.559059866 container died 0bc8628f3172ce9076c3fa35c14573cef56d64eaef56a29c67351149e17ec87b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-3, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=squid-9e2ac70, GIT_CLEAN=True, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True) 2024-09-20T22:57:01.964 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:57:01.986 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:01.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:01.987 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:57:02.005 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.6 dump_blocked_ops 2024-09-20T22:57:02.076 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:57:02.088 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:57:02.110 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:02.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:02.111 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:57:02.212 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:57:02.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:02.235 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:02.235 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:57:02.276 INFO:journalctl@ceph.osd.3.smithi084.stdout:Sep 20 22:57:02 smithi084 podman[206062]: 2024-09-20 22:57:02.027926055 +0000 UTC m=+1.898822279 container cleanup 0bc8628f3172ce9076c3fa35c14573cef56d64eaef56a29c67351149e17ec87b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-3, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, RELEASE=squid-9e2ac70, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-09-20T22:57:02.276 INFO:journalctl@ceph.osd.3.smithi084.stdout:Sep 20 22:57:02 smithi084 bash[206062]: ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-3 2024-09-20T22:57:02.297 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:57:02.335 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:57:02.357 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:02.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:02.358 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:57:02.420 INFO:tasks.rados.rados.0.smithi084.stdout:3814: done (4 left) 2024-09-20T22:57:02.420 INFO:tasks.rados.rados.0.smithi084.stdout:3815: copy_from oid 40 from oid 14 current snap is 441 2024-09-20T22:57:02.420 INFO:tasks.rados.rados.0.smithi084.stdout:3816: snap_create 2024-09-20T22:57:02.420 INFO:tasks.rados.rados.0.smithi084.stdout:3809: finishing rollback tid 0 to smithi08477203-22 2024-09-20T22:57:02.420 INFO:tasks.rados.rados.0.smithi084.stdout:3809: finishing rollback tid 1 to smithi08477203-22 2024-09-20T22:57:02.420 INFO:tasks.rados.rados.0.smithi084.stdout:3809: finishing rollback tid 2 to smithi08477203-22 2024-09-20T22:57:02.420 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 22 v 2509 (ObjNum 1187 snap 408 seq_num 1187) dirty exists 2024-09-20T22:57:02.420 INFO:tasks.rados.rados.0.smithi084.stdout:3807: finishing write tid 1 to smithi08477203-24 2024-09-20T22:57:02.421 INFO:tasks.rados.rados.0.smithi084.stdout:3807: finishing write tid 2 to smithi08477203-24 2024-09-20T22:57:02.421 INFO:tasks.rados.rados.0.smithi084.stdout:3807: finishing write tid 3 to smithi08477203-24 2024-09-20T22:57:02.421 INFO:tasks.rados.rados.0.smithi084.stdout:3807: finishing write tid 4 to smithi08477203-24 2024-09-20T22:57:02.421 INFO:tasks.rados.rados.0.smithi084.stdout:3807: finishing write tid 5 to smithi08477203-24 2024-09-20T22:57:02.421 INFO:tasks.rados.rados.0.smithi084.stdout:3807: oid 24 updating version 0 to 1955 2024-09-20T22:57:02.421 INFO:tasks.rados.rados.0.smithi084.stdout:3807: oid 24 updating version 1955 to 1956 2024-09-20T22:57:02.421 INFO:tasks.rados.rados.0.smithi084.stdout:3807: oid 24 version 1956 is already newer than 1954 2024-09-20T22:57:02.421 INFO:tasks.rados.rados.0.smithi084.stdout:3807: oid 24 version 1956 is already newer than 1953 2024-09-20T22:57:02.421 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 24 v 1956 (ObjNum 1298 snap 441 seq_num 1298) dirty exists 2024-09-20T22:57:02.421 INFO:tasks.rados.rados.0.smithi084.stdout:3807: left oid 24 (ObjNum 1298 snap 441 seq_num 1298) 2024-09-20T22:57:02.421 INFO:tasks.rados.rados.0.smithi084.stdout:3806: finishing copy_from racing read to smithi08477203-19 2024-09-20T22:57:02.421 INFO:tasks.rados.rados.0.smithi084.stdout:3806: finishing copy_from to smithi08477203-19 2024-09-20T22:57:02.421 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 19 v 2845 (ObjNum 1232 snap 419 seq_num 1232) dirty exists 2024-09-20T22:57:02.439 INFO:tasks.rados.rados.0.smithi084.stdout:3815: finishing copy_from racing read to smithi08477203-40 2024-09-20T22:57:02.441 INFO:tasks.rados.rados.0.smithi084.stdout:3815: finishing copy_from to smithi08477203-40 2024-09-20T22:57:02.442 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 40 v 2593 (ObjNum 1285 snap 436 seq_num 1285) dirty exists 2024-09-20T22:57:02.459 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:57:02.480 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:02.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:02.481 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:57:02.541 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:02 smithi084 ceph-mon[40024]: 7.14 continuing backfill to osd.2 from (456'176,936'716] MIN to 936'716 2024-09-20T22:57:02.541 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:02 smithi084 ceph-mon[40024]: 7.14 continuing backfill to osd.5 from (456'176,936'716] MIN to 936'716 2024-09-20T22:57:02.541 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:02 smithi084 ceph-mon[40024]: osdmap e941: 8 total, 7 up, 7 in 2024-09-20T22:57:02.541 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:02 smithi084 ceph-mon[40024]: 7.11 continuing backfill to osd.1 from (474'259,929'565] MIN to 929'565 2024-09-20T22:57:02.541 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:02 smithi084 ceph-mon[40024]: 7.17 continuing backfill to osd.1 from (436'456,930'2006] MIN to 930'2006 2024-09-20T22:57:02.541 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:02 smithi084 ceph-mon[40024]: 7.17 continuing backfill to osd.2 from (436'456,930'2006] MIN to 930'2006 2024-09-20T22:57:02.541 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:02 smithi084 ceph-mon[40024]: 7.17 continuing backfill to osd.5 from (436'456,930'2006] MIN to 930'2006 2024-09-20T22:57:02.541 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:02 smithi084 ceph-mon[40024]: 7.12 continuing backfill to osd.0 from (471'464,936'1009] MIN to 936'1009 2024-09-20T22:57:02.541 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:02 smithi084 ceph-mon[40024]: 7.12 continuing backfill to osd.6 from (471'464,936'1009] MIN to 936'1009 2024-09-20T22:57:02.541 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:02 smithi084 ceph-mon[40024]: 7.13 continuing backfill to osd.0 from (456'129,936'480] MIN to 936'480 2024-09-20T22:57:02.542 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:02 smithi084 ceph-mon[40024]: 7.13 continuing backfill to osd.5 from (456'129,936'480] MIN to 936'480 2024-09-20T22:57:02.542 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:02 smithi084 ceph-mon[29774]: 7.14 continuing backfill to osd.2 from (456'176,936'716] MIN to 936'716 2024-09-20T22:57:02.542 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:02 smithi084 ceph-mon[29774]: 7.14 continuing backfill to osd.5 from (456'176,936'716] MIN to 936'716 2024-09-20T22:57:02.542 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:02 smithi084 ceph-mon[29774]: osdmap e941: 8 total, 7 up, 7 in 2024-09-20T22:57:02.542 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:02 smithi084 ceph-mon[29774]: 7.11 continuing backfill to osd.1 from (474'259,929'565] MIN to 929'565 2024-09-20T22:57:02.542 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:02 smithi084 ceph-mon[29774]: 7.17 continuing backfill to osd.1 from (436'456,930'2006] MIN to 930'2006 2024-09-20T22:57:02.542 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:02 smithi084 ceph-mon[29774]: 7.17 continuing backfill to osd.2 from (436'456,930'2006] MIN to 930'2006 2024-09-20T22:57:02.542 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:02 smithi084 ceph-mon[29774]: 7.17 continuing backfill to osd.5 from (436'456,930'2006] MIN to 930'2006 2024-09-20T22:57:02.542 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:02 smithi084 ceph-mon[29774]: 7.12 continuing backfill to osd.0 from (471'464,936'1009] MIN to 936'1009 2024-09-20T22:57:02.542 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:02 smithi084 ceph-mon[29774]: 7.12 continuing backfill to osd.6 from (471'464,936'1009] MIN to 936'1009 2024-09-20T22:57:02.542 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:02 smithi084 ceph-mon[29774]: 7.13 continuing backfill to osd.0 from (456'129,936'480] MIN to 936'480 2024-09-20T22:57:02.542 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:02 smithi084 ceph-mon[29774]: 7.13 continuing backfill to osd.5 from (456'129,936'480] MIN to 936'480 2024-09-20T22:57:02.582 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:57:02.604 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:02.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:02.605 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:57:02.672 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:02 smithi154 ceph-mon[33264]: 7.14 continuing backfill to osd.2 from (456'176,936'716] MIN to 936'716 2024-09-20T22:57:02.672 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:02 smithi154 ceph-mon[33264]: 7.14 continuing backfill to osd.5 from (456'176,936'716] MIN to 936'716 2024-09-20T22:57:02.673 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:02 smithi154 ceph-mon[33264]: osdmap e941: 8 total, 7 up, 7 in 2024-09-20T22:57:02.673 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:02 smithi154 ceph-mon[33264]: 7.11 continuing backfill to osd.1 from (474'259,929'565] MIN to 929'565 2024-09-20T22:57:02.673 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:02 smithi154 ceph-mon[33264]: 7.17 continuing backfill to osd.1 from (436'456,930'2006] MIN to 930'2006 2024-09-20T22:57:02.673 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:02 smithi154 ceph-mon[33264]: 7.17 continuing backfill to osd.2 from (436'456,930'2006] MIN to 930'2006 2024-09-20T22:57:02.673 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:02 smithi154 ceph-mon[33264]: 7.17 continuing backfill to osd.5 from (436'456,930'2006] MIN to 930'2006 2024-09-20T22:57:02.673 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:02 smithi154 ceph-mon[33264]: 7.12 continuing backfill to osd.0 from (471'464,936'1009] MIN to 936'1009 2024-09-20T22:57:02.673 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:02 smithi154 ceph-mon[33264]: 7.12 continuing backfill to osd.6 from (471'464,936'1009] MIN to 936'1009 2024-09-20T22:57:02.673 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:02 smithi154 ceph-mon[33264]: 7.13 continuing backfill to osd.0 from (456'129,936'480] MIN to 936'480 2024-09-20T22:57:02.673 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:02 smithi154 ceph-mon[33264]: 7.13 continuing backfill to osd.5 from (456'129,936'480] MIN to 936'480 2024-09-20T22:57:02.705 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:57:02.728 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:02.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:02.729 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:57:02.773 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-20T22:57:02.773 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:57:02.826 INFO:journalctl@ceph.osd.3.smithi084.stdout:Sep 20 22:57:02 smithi084 bash[206499]: ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-3 2024-09-20T22:57:02.829 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:57:02.851 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:02.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:02.852 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:57:02.953 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:57:02.974 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:02.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:02.975 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:57:03.076 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:57:03.097 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:03.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:03.098 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:57:03.115 INFO:journalctl@ceph.osd.3.smithi084.stdout:Sep 20 22:57:02 smithi084 podman[206220]: 2024-09-20 22:57:02.824932761 +0000 UTC m=+1.863217727 container remove 0bc8628f3172ce9076c3fa35c14573cef56d64eaef56a29c67351149e17ec87b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-3, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, RELEASE=squid-9e2ac70, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731) 2024-09-20T22:57:03.115 INFO:journalctl@ceph.osd.3.smithi084.stdout:Sep 20 22:57:03 smithi084 podman[206577]: 2024-09-20 22:57:02.937079694 +0000 UTC m=+0.015806938 image pull cc9251b744a3faafca7c3095f79bfff81873afe823f31a03a11e015a34abcdbd quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5 2024-09-20T22:57:03.115 INFO:journalctl@ceph.osd.3.smithi084.stdout:Sep 20 22:57:03 smithi084 podman[206577]: 2024-09-20 22:57:03.042544409 +0000 UTC m=+0.121271642 container create 51a145006f854092ea2dd8036f31d72aef14471e335915c9cf39b8a48e70c31a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-3-deactivate, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-9e2ac70) 2024-09-20T22:57:03.198 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:57:03.220 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:03.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:03.221 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:57:03.322 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:57:03.343 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:03.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:03.344 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:57:03.416 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is unset 2024-09-20T22:57:03.423 INFO:tasks.rados.rados.0.smithi084.stdout:3806: done (5 left) 2024-09-20T22:57:03.423 INFO:tasks.rados.rados.0.smithi084.stdout:3807: done (4 left) 2024-09-20T22:57:03.423 INFO:tasks.rados.rados.0.smithi084.stdout:3809: done (3 left) 2024-09-20T22:57:03.423 INFO:tasks.rados.rados.0.smithi084.stdout:3811: done (2 left) 2024-09-20T22:57:03.424 INFO:tasks.rados.rados.0.smithi084.stdout:3815: done (1 left) 2024-09-20T22:57:03.424 INFO:tasks.rados.rados.0.smithi084.stdout:3816: done (0 left) 2024-09-20T22:57:03.424 INFO:tasks.rados.rados.0.smithi084.stdout:3817: copy_from oid 24 from oid 17 current snap is 442 2024-09-20T22:57:03.424 INFO:tasks.rados.rados.0.smithi084.stdout:3818: write (excl) oid 8 current snap is 442 2024-09-20T22:57:03.424 INFO:tasks.rados.rados.0.smithi084.stdout:3818: seq_num 1300 ranges {649100=403710,1606356=501788,2633420=497852,3636005=213095} 2024-09-20T22:57:03.429 INFO:tasks.rados.rados.0.smithi084.stdout:3818: writing smithi08477203-8 from 649100 to 1052810 tid 1 2024-09-20T22:57:03.436 INFO:tasks.rados.rados.0.smithi084.stdout:3818: writing smithi08477203-8 from 1606356 to 2108144 tid 2 2024-09-20T22:57:03.442 INFO:tasks.rados.rados.0.smithi084.stdout:3818: writing smithi08477203-8 from 2633420 to 3131272 tid 3 2024-09-20T22:57:03.445 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:57:03.447 INFO:tasks.rados.rados.0.smithi084.stdout:3818: writing smithi08477203-8 from 3636005 to 3849100 tid 4 2024-09-20T22:57:03.447 INFO:tasks.rados.rados.0.smithi084.stdout:3819: delete oid 23 current snap is 442 2024-09-20T22:57:03.447 INFO:tasks.rados.rados.0.smithi084.stdout:3817: finishing copy_from to smithi08477203-24 2024-09-20T22:57:03.447 INFO:tasks.rados.rados.0.smithi084.stdout:3817: got expected ENOENT (src dne) 2024-09-20T22:57:03.447 INFO:tasks.rados.rados.0.smithi084.stdout:3817: finishing copy_from racing read to smithi08477203-24 2024-09-20T22:57:03.447 INFO:tasks.rados.rados.0.smithi084.stdout:3818: finishing write tid 1 to smithi08477203-8 2024-09-20T22:57:03.447 INFO:tasks.rados.rados.0.smithi084.stdout:3818: finishing write tid 2 to smithi08477203-8 2024-09-20T22:57:03.447 INFO:tasks.rados.rados.0.smithi084.stdout:3818: finishing write tid 3 to smithi08477203-8 2024-09-20T22:57:03.450 INFO:tasks.rados.rados.0.smithi084.stdout:3817: done (2 left) 2024-09-20T22:57:03.450 INFO:tasks.rados.rados.0.smithi084.stdout:3819: done (1 left) 2024-09-20T22:57:03.450 INFO:tasks.rados.rados.0.smithi084.stdout:3820: write oid 17 current snap is 442 2024-09-20T22:57:03.450 INFO:tasks.rados.rados.0.smithi084.stdout:3820: seq_num 1302 ranges {682714=670561,2123320=505457,3088266=636749} 2024-09-20T22:57:03.458 INFO:tasks.rados.rados.0.smithi084.stdout:3820: writing smithi08477203-17 from 682714 to 1353275 tid 1 2024-09-20T22:57:03.463 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.6 dump_historic_ops 2024-09-20T22:57:03.465 INFO:tasks.rados.rados.0.smithi084.stdout:3820: writing smithi08477203-17 from 2123320 to 2628777 tid 2 2024-09-20T22:57:03.468 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:03.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:03.469 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:57:03.472 INFO:tasks.rados.rados.0.smithi084.stdout:3820: writing smithi08477203-17 from 3088266 to 3725015 tid 3 2024-09-20T22:57:03.472 INFO:tasks.rados.rados.0.smithi084.stdout:3821: copy_from oid 31 from oid 10 current snap is 442 2024-09-20T22:57:03.472 INFO:tasks.rados.rados.0.smithi084.stdout:3818: finishing write tid 4 to smithi08477203-8 2024-09-20T22:57:03.472 INFO:tasks.rados.rados.0.smithi084.stdout:3818: finishing write tid 5 to smithi08477203-8 2024-09-20T22:57:03.472 INFO:tasks.rados.rados.0.smithi084.stdout:3818: finishing write tid 6 to smithi08477203-8 2024-09-20T22:57:03.472 INFO:tasks.rados.rados.0.smithi084.stdout:3818: oid 8 updating version 0 to 2630 2024-09-20T22:57:03.472 INFO:tasks.rados.rados.0.smithi084.stdout:3818: oid 8 version 2630 is already newer than 2627 2024-09-20T22:57:03.472 INFO:tasks.rados.rados.0.smithi084.stdout:3818: oid 8 version 2630 is already newer than 2629 2024-09-20T22:57:03.472 INFO:tasks.rados.rados.0.smithi084.stdout:3818: oid 8 updating version 2630 to 2631 2024-09-20T22:57:03.472 INFO:tasks.rados.rados.0.smithi084.stdout:3818: oid 8 version 2631 is already newer than 2628 2024-09-20T22:57:03.472 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 8 v 2631 (ObjNum 1300 snap 442 seq_num 1300) dirty exists 2024-09-20T22:57:03.472 INFO:tasks.rados.rados.0.smithi084.stdout:3818: left oid 8 (ObjNum 1300 snap 442 seq_num 1300) 2024-09-20T22:57:03.473 INFO:tasks.rados.rados.0.smithi084.stdout:3820: finishing write tid 1 to smithi08477203-17 2024-09-20T22:57:03.473 INFO:tasks.rados.rados.0.smithi084.stdout:3818: done (2 left) 2024-09-20T22:57:03.473 INFO:tasks.rados.rados.0.smithi084.stdout:3822: rollback oid 25 current snap is 442 2024-09-20T22:57:03.473 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 25 to 411 2024-09-20T22:57:03.473 INFO:tasks.rados.rados.0.smithi084.stdout:3820: finishing write tid 2 to smithi08477203-17 2024-09-20T22:57:03.473 INFO:tasks.rados.rados.0.smithi084.stdout:3823: snap_remove snap 418 2024-09-20T22:57:03.570 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:57:03.591 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:03.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:03.592 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:57:03.676 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:03 smithi084 ceph-mon[40024]: pgmap v1314: 168 pgs: 3 stale+active+clean+snaptrim_wait, 2 stale+active+clean+snaptrim, 33 stale+active+clean, 2 active+clean+snaptrim, 5 active+remapped+backfill_toofull, 123 active+clean; 732 MiB data, 2.3 GiB used, 624 GiB / 626 GiB avail; 20 MiB/s rd, 6.7 MiB/s wr, 38 op/s; 102/1593 objects misplaced (6.403%); 31 MiB/s, 14 objects/s recovering 2024-09-20T22:57:03.676 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:03 smithi084 ceph-mon[40024]: osdmap e942: 8 total, 7 up, 7 in 2024-09-20T22:57:03.676 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:03 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/1448956831' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:57:03.677 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:03 smithi084 ceph-mon[29774]: pgmap v1314: 168 pgs: 3 stale+active+clean+snaptrim_wait, 2 stale+active+clean+snaptrim, 33 stale+active+clean, 2 active+clean+snaptrim, 5 active+remapped+backfill_toofull, 123 active+clean; 732 MiB data, 2.3 GiB used, 624 GiB / 626 GiB avail; 20 MiB/s rd, 6.7 MiB/s wr, 38 op/s; 102/1593 objects misplaced (6.403%); 31 MiB/s, 14 objects/s recovering 2024-09-20T22:57:03.677 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:03 smithi084 ceph-mon[29774]: osdmap e942: 8 total, 7 up, 7 in 2024-09-20T22:57:03.677 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:03 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/1448956831' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:57:03.677 INFO:journalctl@ceph.osd.3.smithi084.stdout:Sep 20 22:57:03 smithi084 podman[206577]: 2024-09-20 22:57:03.478752062 +0000 UTC m=+0.557479296 container init 51a145006f854092ea2dd8036f31d72aef14471e335915c9cf39b8a48e70c31a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-3-deactivate, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=squid-9e2ac70, org.label-schema.license=GPLv2) 2024-09-20T22:57:03.677 INFO:journalctl@ceph.osd.3.smithi084.stdout:Sep 20 22:57:03 smithi084 podman[206577]: 2024-09-20 22:57:03.482566442 +0000 UTC m=+0.561293673 container start 51a145006f854092ea2dd8036f31d72aef14471e335915c9cf39b8a48e70c31a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-3-deactivate, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-9e2ac70, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, org.label-schema.build-date=20240731) 2024-09-20T22:57:03.677 INFO:journalctl@ceph.osd.3.smithi084.stdout:Sep 20 22:57:03 smithi084 podman[206577]: 2024-09-20 22:57:03.637978319 +0000 UTC m=+0.716705554 container attach 51a145006f854092ea2dd8036f31d72aef14471e335915c9cf39b8a48e70c31a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-3-deactivate, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=squid-9e2ac70, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, ceph=True) 2024-09-20T22:57:03.679 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-20T22:57:03.693 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:57:03.717 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:03.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:03.719 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:57:03.719 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:57:03.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:03 smithi154 ceph-mon[33264]: pgmap v1314: 168 pgs: 3 stale+active+clean+snaptrim_wait, 2 stale+active+clean+snaptrim, 33 stale+active+clean, 2 active+clean+snaptrim, 5 active+remapped+backfill_toofull, 123 active+clean; 732 MiB data, 2.3 GiB used, 624 GiB / 626 GiB avail; 20 MiB/s rd, 6.7 MiB/s wr, 38 op/s; 102/1593 objects misplaced (6.403%); 31 MiB/s, 14 objects/s recovering 2024-09-20T22:57:03.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:03 smithi154 ceph-mon[33264]: osdmap e942: 8 total, 7 up, 7 in 2024-09-20T22:57:03.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:03 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/1448956831' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:57:03.820 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:57:03.841 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:03.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:03.842 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:57:03.942 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:57:03.964 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:03.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:03.965 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:57:04.065 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:57:04.087 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:04.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:04.088 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:57:04.188 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:57:04.209 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:04.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:04.210 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:57:04.262 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-20T22:57:04.262 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:57:04.311 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:57:04.333 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:04.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:04.334 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:57:04.434 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:57:04.446 INFO:tasks.rados.rados.0.smithi084.stdout:3823: done (3 left) 2024-09-20T22:57:04.446 INFO:tasks.rados.rados.0.smithi084.stdout:3824: snap_create 2024-09-20T22:57:04.446 INFO:tasks.rados.rados.0.smithi084.stdout:3821: finishing copy_from to smithi08477203-31 2024-09-20T22:57:04.446 INFO:tasks.rados.rados.0.smithi084.stdout:3821: got expected ENOENT (src dne) 2024-09-20T22:57:04.446 INFO:tasks.rados.rados.0.smithi084.stdout:3821: finishing copy_from racing read to smithi08477203-31 2024-09-20T22:57:04.446 INFO:tasks.rados.rados.0.smithi084.stdout:3820: finishing write tid 3 to smithi08477203-17 2024-09-20T22:57:04.446 INFO:tasks.rados.rados.0.smithi084.stdout:3820: finishing write tid 4 to smithi08477203-17 2024-09-20T22:57:04.447 INFO:tasks.rados.rados.0.smithi084.stdout:3820: finishing write tid 5 to smithi08477203-17 2024-09-20T22:57:04.447 INFO:tasks.rados.rados.0.smithi084.stdout:3820: oid 17 updating version 0 to 2656 2024-09-20T22:57:04.447 INFO:tasks.rados.rados.0.smithi084.stdout:3820: oid 17 version 2656 is already newer than 2654 2024-09-20T22:57:04.447 INFO:tasks.rados.rados.0.smithi084.stdout:3820: oid 17 version 2656 is already newer than 2655 2024-09-20T22:57:04.447 INFO:tasks.rados.rados.0.smithi084.stdout:3820: oid 17 version 2656 is already newer than 2653 2024-09-20T22:57:04.447 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 17 v 2656 (ObjNum 1302 snap 442 seq_num 1302) dirty exists 2024-09-20T22:57:04.447 INFO:tasks.rados.rados.0.smithi084.stdout:3820: left oid 17 (ObjNum 1302 snap 442 seq_num 1302) 2024-09-20T22:57:04.447 INFO:tasks.rados.rados.0.smithi084.stdout:3822: finishing rollback tid 0 to smithi08477203-25 2024-09-20T22:57:04.447 INFO:tasks.rados.rados.0.smithi084.stdout:3822: finishing rollback tid 1 to smithi08477203-25 2024-09-20T22:57:04.447 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 25 v 3127 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-09-20T22:57:04.462 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:04.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:04.463 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:57:04.564 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:57:04.585 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:04.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:04.586 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:57:04.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:04 smithi084 ceph-mon[29774]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:57:04.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:04 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/1448956831' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T22:57:04.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:04 smithi084 ceph-mon[29774]: osdmap e943: 8 total, 7 up, 7 in 2024-09-20T22:57:04.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:04 smithi084 ceph-mon[29774]: 2.2 scrub starts 2024-09-20T22:57:04.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:04 smithi084 ceph-mon[29774]: 2.2 scrub ok 2024-09-20T22:57:04.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:04 smithi084 ceph-mon[29774]: 7.18 scrub starts 2024-09-20T22:57:04.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:04 smithi084 ceph-mon[29774]: 7.18 scrub ok 2024-09-20T22:57:04.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:04 smithi084 ceph-mon[29774]: 7.19 scrub starts 2024-09-20T22:57:04.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:04 smithi084 ceph-mon[29774]: 7.19 scrub ok 2024-09-20T22:57:04.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:04 smithi084 ceph-mon[40024]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:57:04.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:04 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/1448956831' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T22:57:04.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:04 smithi084 ceph-mon[40024]: osdmap e943: 8 total, 7 up, 7 in 2024-09-20T22:57:04.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:04 smithi084 ceph-mon[40024]: 2.2 scrub starts 2024-09-20T22:57:04.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:04 smithi084 ceph-mon[40024]: 2.2 scrub ok 2024-09-20T22:57:04.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:04 smithi084 ceph-mon[40024]: 7.18 scrub starts 2024-09-20T22:57:04.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:04 smithi084 ceph-mon[40024]: 7.18 scrub ok 2024-09-20T22:57:04.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:04 smithi084 ceph-mon[40024]: 7.19 scrub starts 2024-09-20T22:57:04.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:04 smithi084 ceph-mon[40024]: 7.19 scrub ok 2024-09-20T22:57:04.616 INFO:journalctl@ceph.osd.3.smithi084.stdout:Sep 20 22:57:04 smithi084 podman[206577]: 2024-09-20 22:57:04.40484529 +0000 UTC m=+1.483572521 container died 51a145006f854092ea2dd8036f31d72aef14471e335915c9cf39b8a48e70c31a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-3-deactivate, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=squid-9e2ac70, GIT_CLEAN=True, org.label-schema.build-date=20240731) 2024-09-20T22:57:04.687 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:57:04.708 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:04.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:04.709 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:57:04.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:04 smithi154 ceph-mon[33264]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:57:04.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:04 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/1448956831' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T22:57:04.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:04 smithi154 ceph-mon[33264]: osdmap e943: 8 total, 7 up, 7 in 2024-09-20T22:57:04.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:04 smithi154 ceph-mon[33264]: 2.2 scrub starts 2024-09-20T22:57:04.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:04 smithi154 ceph-mon[33264]: 2.2 scrub ok 2024-09-20T22:57:04.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:04 smithi154 ceph-mon[33264]: 7.18 scrub starts 2024-09-20T22:57:04.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:04 smithi154 ceph-mon[33264]: 7.18 scrub ok 2024-09-20T22:57:04.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:04 smithi154 ceph-mon[33264]: 7.19 scrub starts 2024-09-20T22:57:04.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:04 smithi154 ceph-mon[33264]: 7.19 scrub ok 2024-09-20T22:57:04.811 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:57:04.832 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:04.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:04.833 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:57:04.934 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:57:04.955 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:04.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:04.956 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:57:05.029 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-20T22:57:05.057 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:57:05.073 DEBUG:teuthology.orchestra.run.smithi084:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490@osd.3.service' 2024-09-20T22:57:05.080 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:05.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:05.081 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:57:05.095 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:57:05.140 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-20T22:57:05.141 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-09-20T22:57:05.141 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.3 2024-09-20T22:57:05.141 INFO:tasks.thrashosds.thrasher:remote for osd 3 is ubuntu@smithi084.front.sepia.ceph.com 2024-09-20T22:57:05.142 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell -n osd.3 --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-3 --journal-path /var/lib/ceph/osd/ceph-3/journal --op list-pgs 2024-09-20T22:57:05.182 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:57:05.205 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:05.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:05.206 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:57:05.275 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:57:05.306 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:57:05.329 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:05.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:05.331 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:57:05.333 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/osd.3/config 2024-09-20T22:57:05.431 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:57:05.443 INFO:tasks.rados.rados.0.smithi084.stdout:3820: done (3 left) 2024-09-20T22:57:05.444 INFO:tasks.rados.rados.0.smithi084.stdout:3821: done (2 left) 2024-09-20T22:57:05.444 INFO:tasks.rados.rados.0.smithi084.stdout:3822: done (1 left) 2024-09-20T22:57:05.444 INFO:tasks.rados.rados.0.smithi084.stdout:3824: done (0 left) 2024-09-20T22:57:05.444 INFO:tasks.rados.rados.0.smithi084.stdout:3825: copy_from oid 22 from oid 49 current snap is 443 2024-09-20T22:57:05.444 INFO:tasks.rados.rados.0.smithi084.stdout:3826: write (excl) oid 25 current snap is 443 2024-09-20T22:57:05.444 INFO:tasks.rados.rados.0.smithi084.stdout:3826: seq_num 1303 ranges {699521=745810,2181802=649115,3515689=383832} 2024-09-20T22:57:05.451 INFO:tasks.rados.rados.0.smithi084.stdout:3826: writing smithi08477203-25 from 699521 to 1445331 tid 1 2024-09-20T22:57:05.456 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:05.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:05.457 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:57:05.459 INFO:tasks.rados.rados.0.smithi084.stdout:3826: writing smithi08477203-25 from 2181802 to 2830917 tid 2 2024-09-20T22:57:05.465 INFO:tasks.rados.rados.0.smithi084.stdout:3826: writing smithi08477203-25 from 3515689 to 3899521 tid 3 2024-09-20T22:57:05.465 INFO:tasks.rados.rados.0.smithi084.stdout:3827: snap_create 2024-09-20T22:57:05.465 INFO:tasks.rados.rados.0.smithi084.stdout:3826: finishing write tid 1 to smithi08477203-25 2024-09-20T22:57:05.465 INFO:tasks.rados.rados.0.smithi084.stdout:3825: finishing copy_from racing read to smithi08477203-22 2024-09-20T22:57:05.465 INFO:tasks.rados.rados.0.smithi084.stdout:3825: finishing copy_from to smithi08477203-22 2024-09-20T22:57:05.465 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 22 v 2602 (ObjNum 1288 snap 436 seq_num 1288) dirty exists 2024-09-20T22:57:05.465 INFO:tasks.rados.rados.0.smithi084.stdout:3826: finishing write tid 2 to smithi08477203-25 2024-09-20T22:57:05.468 INFO:tasks.rados.rados.0.smithi084.stdout:3826: finishing write tid 3 to smithi08477203-25 2024-09-20T22:57:05.468 INFO:tasks.rados.rados.0.smithi084.stdout:3826: finishing write tid 4 to smithi08477203-25 2024-09-20T22:57:05.469 INFO:tasks.rados.rados.0.smithi084.stdout:3826: finishing write tid 5 to smithi08477203-25 2024-09-20T22:57:05.469 INFO:tasks.rados.rados.0.smithi084.stdout:3826: oid 25 updating version 0 to 3130 2024-09-20T22:57:05.469 INFO:tasks.rados.rados.0.smithi084.stdout:3826: oid 25 version 3130 is already newer than 3129 2024-09-20T22:57:05.469 INFO:tasks.rados.rados.0.smithi084.stdout:3826: oid 25 updating version 3130 to 3131 2024-09-20T22:57:05.469 INFO:tasks.rados.rados.0.smithi084.stdout:3826: oid 25 version 3131 is already newer than 3128 2024-09-20T22:57:05.469 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 25 v 3131 (ObjNum 1303 snap 443 seq_num 1303) dirty exists 2024-09-20T22:57:05.469 INFO:tasks.rados.rados.0.smithi084.stdout:3826: left oid 25 (ObjNum 1303 snap 443 seq_num 1303) 2024-09-20T22:57:05.557 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:57:05.579 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:05.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:05.579 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:57:05.601 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:05 smithi084 ceph-mon[29774]: pgmap v1317: 168 pgs: 2 active+undersized+degraded+remapped+backfill_toofull, 1 active+clean+snaptrim, 41 active+undersized, 36 active+undersized+degraded, 1 active+remapped+backfill_toofull, 87 active+clean; 753 MiB data, 2.3 GiB used, 624 GiB / 626 GiB avail; 1.1 MiB/s rd, 3.7 MiB/s wr, 9 op/s; 285/1620 objects degraded (17.593%); 55/1620 objects misplaced (3.395%); 12 MiB/s, 0 objects/s recovering 2024-09-20T22:57:05.601 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:05 smithi084 ceph-mon[29774]: 7.5 scrub starts 2024-09-20T22:57:05.601 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:05 smithi084 ceph-mon[29774]: 7.5 scrub ok 2024-09-20T22:57:05.601 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:05 smithi084 ceph-mon[29774]: 7.15 scrub starts 2024-09-20T22:57:05.601 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:05 smithi084 ceph-mon[29774]: 7.15 scrub ok 2024-09-20T22:57:05.601 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:05 smithi084 ceph-mon[29774]: Health check failed: Degraded data redundancy: 285/1620 objects degraded (17.593%), 38 pgs degraded (PG_DEGRADED) 2024-09-20T22:57:05.601 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:05 smithi084 ceph-mon[29774]: osdmap e944: 8 total, 7 up, 7 in 2024-09-20T22:57:05.601 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:05 smithi084 ceph-mon[29774]: 7.13 scrub starts 2024-09-20T22:57:05.601 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:05 smithi084 ceph-mon[29774]: 7.13 scrub ok 2024-09-20T22:57:05.601 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:05 smithi084 ceph-mon[29774]: 7.12 scrub starts 2024-09-20T22:57:05.601 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:05 smithi084 ceph-mon[29774]: 7.12 scrub ok 2024-09-20T22:57:05.601 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:05 smithi084 ceph-mon[40024]: pgmap v1317: 168 pgs: 2 active+undersized+degraded+remapped+backfill_toofull, 1 active+clean+snaptrim, 41 active+undersized, 36 active+undersized+degraded, 1 active+remapped+backfill_toofull, 87 active+clean; 753 MiB data, 2.3 GiB used, 624 GiB / 626 GiB avail; 1.1 MiB/s rd, 3.7 MiB/s wr, 9 op/s; 285/1620 objects degraded (17.593%); 55/1620 objects misplaced (3.395%); 12 MiB/s, 0 objects/s recovering 2024-09-20T22:57:05.602 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:05 smithi084 ceph-mon[40024]: 7.5 scrub starts 2024-09-20T22:57:05.602 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:05 smithi084 ceph-mon[40024]: 7.5 scrub ok 2024-09-20T22:57:05.602 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:05 smithi084 ceph-mon[40024]: 7.15 scrub starts 2024-09-20T22:57:05.602 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:05 smithi084 ceph-mon[40024]: 7.15 scrub ok 2024-09-20T22:57:05.602 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:05 smithi084 ceph-mon[40024]: Health check failed: Degraded data redundancy: 285/1620 objects degraded (17.593%), 38 pgs degraded (PG_DEGRADED) 2024-09-20T22:57:05.602 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:05 smithi084 ceph-mon[40024]: osdmap e944: 8 total, 7 up, 7 in 2024-09-20T22:57:05.602 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:05 smithi084 ceph-mon[40024]: 7.13 scrub starts 2024-09-20T22:57:05.602 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:05 smithi084 ceph-mon[40024]: 7.13 scrub ok 2024-09-20T22:57:05.602 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:05 smithi084 ceph-mon[40024]: 7.12 scrub starts 2024-09-20T22:57:05.602 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:05 smithi084 ceph-mon[40024]: 7.12 scrub ok 2024-09-20T22:57:05.669 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:57:05.670 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:57:05.670 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:57:05.670 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:57:05.670 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:57:05.670 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:57:05.670 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:57:05.670 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:57:05.670 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:57:05.670 INFO:teuthology.orchestra.run.smithi084.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-20T22:57:05.670 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:57:05.671 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:57:05.671 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:57:05.671 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:57:05.671 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:57:05.680 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:57:05.701 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:05.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:05.702 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:57:05.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:05 smithi154 ceph-mon[33264]: pgmap v1317: 168 pgs: 2 active+undersized+degraded+remapped+backfill_toofull, 1 active+clean+snaptrim, 41 active+undersized, 36 active+undersized+degraded, 1 active+remapped+backfill_toofull, 87 active+clean; 753 MiB data, 2.3 GiB used, 624 GiB / 626 GiB avail; 1.1 MiB/s rd, 3.7 MiB/s wr, 9 op/s; 285/1620 objects degraded (17.593%); 55/1620 objects misplaced (3.395%); 12 MiB/s, 0 objects/s recovering 2024-09-20T22:57:05.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:05 smithi154 ceph-mon[33264]: 7.5 scrub starts 2024-09-20T22:57:05.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:05 smithi154 ceph-mon[33264]: 7.5 scrub ok 2024-09-20T22:57:05.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:05 smithi154 ceph-mon[33264]: 7.15 scrub starts 2024-09-20T22:57:05.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:05 smithi154 ceph-mon[33264]: 7.15 scrub ok 2024-09-20T22:57:05.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:05 smithi154 ceph-mon[33264]: Health check failed: Degraded data redundancy: 285/1620 objects degraded (17.593%), 38 pgs degraded (PG_DEGRADED) 2024-09-20T22:57:05.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:05 smithi154 ceph-mon[33264]: osdmap e944: 8 total, 7 up, 7 in 2024-09-20T22:57:05.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:05 smithi154 ceph-mon[33264]: 7.13 scrub starts 2024-09-20T22:57:05.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:05 smithi154 ceph-mon[33264]: 7.13 scrub ok 2024-09-20T22:57:05.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:05 smithi154 ceph-mon[33264]: 7.12 scrub starts 2024-09-20T22:57:05.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:05 smithi154 ceph-mon[33264]: 7.12 scrub ok 2024-09-20T22:57:05.803 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:57:05.825 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:05.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:05.826 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:57:05.869 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-20T22:57:05.869 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [], 2024-09-20T22:57:05.869 INFO:teuthology.orchestra.run.smithi154.stdout: "num_ops": 0 2024-09-20T22:57:05.869 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-20T22:57:05.927 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:57:05.948 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:05.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:05.949 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:57:06.050 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:57:06.072 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:06.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:06.072 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:57:06.173 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:57:06.194 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:06.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:06.198 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:57:06.267 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-20T22:57:06.267 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-20T22:57:06.299 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:57:06.320 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:06.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:06.321 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:57:06.355 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-20T22:57:06.422 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:57:06.446 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:06.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:06.447 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:57:06.450 INFO:tasks.rados.rados.0.smithi084.stdout:3825: done (2 left) 2024-09-20T22:57:06.451 INFO:tasks.rados.rados.0.smithi084.stdout:3826: done (1 left) 2024-09-20T22:57:06.451 INFO:tasks.rados.rados.0.smithi084.stdout:3827: done (0 left) 2024-09-20T22:57:06.451 INFO:tasks.rados.rados.0.smithi084.stdout:3828: read oid 9 snap 443 2024-09-20T22:57:06.451 INFO:tasks.rados.rados.0.smithi084.stdout:3828: expect deleted 2024-09-20T22:57:06.451 INFO:tasks.rados.rados.0.smithi084.stdout:3829: delete oid 15 current snap is 444 2024-09-20T22:57:06.456 INFO:tasks.rados.rados.0.smithi084.stdout:3828: done (1 left) 2024-09-20T22:57:06.456 INFO:tasks.rados.rados.0.smithi084.stdout:3829: done (0 left) 2024-09-20T22:57:06.456 INFO:tasks.rados.rados.0.smithi084.stdout:3830: delete oid 47 current snap is 444 2024-09-20T22:57:06.458 INFO:tasks.rados.rados.0.smithi084.stdout:3830: done (0 left) 2024-09-20T22:57:06.458 INFO:tasks.rados.rados.0.smithi084.stdout:3831: snap_create 2024-09-20T22:57:06.548 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:57:06.570 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:06.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:06.571 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:57:06.575 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:06 smithi084 ceph-mon[29774]: 7.e scrub starts 2024-09-20T22:57:06.575 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:06 smithi084 ceph-mon[29774]: 7.e scrub ok 2024-09-20T22:57:06.575 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:06 smithi084 ceph-mon[29774]: osdmap e945: 8 total, 7 up, 7 in 2024-09-20T22:57:06.575 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:06 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/131150734' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:57:06.575 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:06 smithi084 ceph-mon[29774]: 4.1 scrub starts 2024-09-20T22:57:06.575 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:06 smithi084 ceph-mon[29774]: 4.1 scrub ok 2024-09-20T22:57:06.575 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:06 smithi084 ceph-mon[29774]: 7.9 scrub starts 2024-09-20T22:57:06.575 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:06 smithi084 ceph-mon[29774]: 7.9 scrub ok 2024-09-20T22:57:06.575 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:06 smithi084 ceph-mon[29774]: 4.a scrub starts 2024-09-20T22:57:06.575 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:06 smithi084 ceph-mon[29774]: 4.a scrub ok 2024-09-20T22:57:06.576 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:06 smithi084 ceph-mon[40024]: 7.e scrub starts 2024-09-20T22:57:06.576 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:06 smithi084 ceph-mon[40024]: 7.e scrub ok 2024-09-20T22:57:06.576 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:06 smithi084 ceph-mon[40024]: osdmap e945: 8 total, 7 up, 7 in 2024-09-20T22:57:06.576 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:06 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/131150734' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:57:06.576 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:06 smithi084 ceph-mon[40024]: 4.1 scrub starts 2024-09-20T22:57:06.576 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:06 smithi084 ceph-mon[40024]: 4.1 scrub ok 2024-09-20T22:57:06.576 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:06 smithi084 ceph-mon[40024]: 7.9 scrub starts 2024-09-20T22:57:06.576 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:06 smithi084 ceph-mon[40024]: 7.9 scrub ok 2024-09-20T22:57:06.576 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:06 smithi084 ceph-mon[40024]: 4.a scrub starts 2024-09-20T22:57:06.576 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:06 smithi084 ceph-mon[40024]: 4.a scrub ok 2024-09-20T22:57:06.663 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.7 dump_blocked_ops 2024-09-20T22:57:06.672 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:57:06.694 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:06.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:06.695 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:57:06.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:06 smithi154 ceph-mon[33264]: 7.e scrub starts 2024-09-20T22:57:06.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:06 smithi154 ceph-mon[33264]: 7.e scrub ok 2024-09-20T22:57:06.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:06 smithi154 ceph-mon[33264]: osdmap e945: 8 total, 7 up, 7 in 2024-09-20T22:57:06.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:06 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/131150734' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:57:06.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:06 smithi154 ceph-mon[33264]: 4.1 scrub starts 2024-09-20T22:57:06.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:06 smithi154 ceph-mon[33264]: 4.1 scrub ok 2024-09-20T22:57:06.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:06 smithi154 ceph-mon[33264]: 7.9 scrub starts 2024-09-20T22:57:06.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:06 smithi154 ceph-mon[33264]: 7.9 scrub ok 2024-09-20T22:57:06.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:06 smithi154 ceph-mon[33264]: 4.a scrub starts 2024-09-20T22:57:06.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:06 smithi154 ceph-mon[33264]: 4.a scrub ok 2024-09-20T22:57:06.795 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:57:06.816 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:06.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:06.817 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:57:06.917 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:57:06.921 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:57:06.939 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:06.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:06.940 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:57:07.041 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:57:07.062 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:07.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:07.064 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:57:07.164 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:57:07.186 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:07.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:07.187 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:57:07.288 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:57:07.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:07.310 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:07.310 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:57:07.411 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:57:07.432 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:07.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:07.433 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:57:07.440 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-20T22:57:07.440 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [], 2024-09-20T22:57:07.440 INFO:teuthology.orchestra.run.smithi154.stdout: "complaint_time": 30, 2024-09-20T22:57:07.440 INFO:teuthology.orchestra.run.smithi154.stdout: "num_blocked_ops": 0 2024-09-20T22:57:07.440 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-20T22:57:07.450 INFO:tasks.rados.rados.0.smithi084.stdout:3831: done (0 left) 2024-09-20T22:57:07.450 INFO:tasks.rados.rados.0.smithi084.stdout:3832: copy_from oid 42 from oid 34 current snap is 445 2024-09-20T22:57:07.450 INFO:tasks.rados.rados.0.smithi084.stdout:3833: write oid 21 current snap is 445 2024-09-20T22:57:07.450 INFO:tasks.rados.rados.0.smithi084.stdout:3833: seq_num 1306 ranges {749942=571557,1840895=796442,3365252=584690} 2024-09-20T22:57:07.459 INFO:tasks.rados.rados.0.smithi084.stdout:3833: writing smithi08477203-21 from 749942 to 1321499 tid 1 2024-09-20T22:57:07.466 INFO:tasks.rados.rados.0.smithi084.stdout:3833: writing smithi08477203-21 from 1840895 to 2637337 tid 2 2024-09-20T22:57:07.474 INFO:tasks.rados.rados.0.smithi084.stdout:3833: writing smithi08477203-21 from 3365252 to 3949942 tid 3 2024-09-20T22:57:07.474 INFO:tasks.rados.rados.0.smithi084.stdout:3834: snap_create 2024-09-20T22:57:07.474 INFO:tasks.rados.rados.0.smithi084.stdout:3832: finishing copy_from racing read to smithi08477203-42 2024-09-20T22:57:07.474 INFO:tasks.rados.rados.0.smithi084.stdout:3833: finishing write tid 1 to smithi08477203-21 2024-09-20T22:57:07.475 INFO:tasks.rados.rados.0.smithi084.stdout:3832: finishing copy_from to smithi08477203-42 2024-09-20T22:57:07.475 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 42 v 2597 (ObjNum 1238 snap 420 seq_num 1238) dirty exists 2024-09-20T22:57:07.475 INFO:tasks.rados.rados.0.smithi084.stdout:3833: finishing write tid 2 to smithi08477203-21 2024-09-20T22:57:07.478 INFO:tasks.rados.rados.0.smithi084.stdout:3833: finishing write tid 3 to smithi08477203-21 2024-09-20T22:57:07.478 INFO:tasks.rados.rados.0.smithi084.stdout:3833: finishing write tid 4 to smithi08477203-21 2024-09-20T22:57:07.478 INFO:tasks.rados.rados.0.smithi084.stdout:3833: finishing write tid 5 to smithi08477203-21 2024-09-20T22:57:07.478 INFO:tasks.rados.rados.0.smithi084.stdout:3833: oid 21 updating version 0 to 2603 2024-09-20T22:57:07.478 INFO:tasks.rados.rados.0.smithi084.stdout:3833: oid 21 updating version 2603 to 2606 2024-09-20T22:57:07.478 INFO:tasks.rados.rados.0.smithi084.stdout:3833: oid 21 version 2606 is already newer than 2604 2024-09-20T22:57:07.479 INFO:tasks.rados.rados.0.smithi084.stdout:3833: oid 21 version 2606 is already newer than 2605 2024-09-20T22:57:07.479 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 21 v 2606 (ObjNum 1306 snap 445 seq_num 1306) dirty exists 2024-09-20T22:57:07.479 INFO:tasks.rados.rados.0.smithi084.stdout:3833: left oid 21 (ObjNum 1306 snap 445 seq_num 1306) 2024-09-20T22:57:07.534 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:57:07.545 INFO:teuthology.orchestra.run.smithi084.stderr:Creating an OSD daemon form without an OSD FSID value 2024-09-20T22:57:07.555 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:07.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:07.556 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:57:07.657 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:57:07.678 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:07.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:07.679 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:57:07.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:07 smithi154 ceph-mon[33264]: pgmap v1320: 168 pgs: 2 active+undersized+degraded+remapped+backfill_toofull, 1 active+clean+snaptrim, 41 active+undersized, 36 active+undersized+degraded, 1 active+remapped+backfill_toofull, 87 active+clean; 753 MiB data, 2.3 GiB used, 624 GiB / 626 GiB avail; 1.1 MiB/s rd, 3.7 MiB/s wr, 9 op/s; 285/1620 objects degraded (17.593%); 55/1620 objects misplaced (3.395%); 12 MiB/s, 0 objects/s recovering 2024-09-20T22:57:07.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:07 smithi154 ceph-mon[33264]: 4.7 scrub starts 2024-09-20T22:57:07.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:07 smithi154 ceph-mon[33264]: 4.7 scrub ok 2024-09-20T22:57:07.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:07 smithi154 ceph-mon[33264]: osdmap e946: 8 total, 7 up, 7 in 2024-09-20T22:57:07.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:07 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:57:07.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:07 smithi154 ceph-mon[33264]: 7.4 scrub starts 2024-09-20T22:57:07.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:07 smithi154 ceph-mon[33264]: 7.4 scrub ok 2024-09-20T22:57:07.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:07 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:57:07.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:07 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:57:07.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:07 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:57:07.769 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:07 smithi084 ceph-mon[40024]: pgmap v1320: 168 pgs: 2 active+undersized+degraded+remapped+backfill_toofull, 1 active+clean+snaptrim, 41 active+undersized, 36 active+undersized+degraded, 1 active+remapped+backfill_toofull, 87 active+clean; 753 MiB data, 2.3 GiB used, 624 GiB / 626 GiB avail; 1.1 MiB/s rd, 3.7 MiB/s wr, 9 op/s; 285/1620 objects degraded (17.593%); 55/1620 objects misplaced (3.395%); 12 MiB/s, 0 objects/s recovering 2024-09-20T22:57:07.769 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:07 smithi084 ceph-mon[40024]: 4.7 scrub starts 2024-09-20T22:57:07.769 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:07 smithi084 ceph-mon[40024]: 4.7 scrub ok 2024-09-20T22:57:07.769 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:07 smithi084 ceph-mon[40024]: osdmap e946: 8 total, 7 up, 7 in 2024-09-20T22:57:07.769 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:07 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:57:07.769 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:07 smithi084 ceph-mon[40024]: 7.4 scrub starts 2024-09-20T22:57:07.769 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:07 smithi084 ceph-mon[40024]: 7.4 scrub ok 2024-09-20T22:57:07.770 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:07 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:57:07.770 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:07 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:57:07.770 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:07 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:57:07.770 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:07 smithi084 ceph-mon[29774]: pgmap v1320: 168 pgs: 2 active+undersized+degraded+remapped+backfill_toofull, 1 active+clean+snaptrim, 41 active+undersized, 36 active+undersized+degraded, 1 active+remapped+backfill_toofull, 87 active+clean; 753 MiB data, 2.3 GiB used, 624 GiB / 626 GiB avail; 1.1 MiB/s rd, 3.7 MiB/s wr, 9 op/s; 285/1620 objects degraded (17.593%); 55/1620 objects misplaced (3.395%); 12 MiB/s, 0 objects/s recovering 2024-09-20T22:57:07.770 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:07 smithi084 ceph-mon[29774]: 4.7 scrub starts 2024-09-20T22:57:07.770 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:07 smithi084 ceph-mon[29774]: 4.7 scrub ok 2024-09-20T22:57:07.770 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:07 smithi084 ceph-mon[29774]: osdmap e946: 8 total, 7 up, 7 in 2024-09-20T22:57:07.770 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:07 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:57:07.770 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:07 smithi084 ceph-mon[29774]: 7.4 scrub starts 2024-09-20T22:57:07.770 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:07 smithi084 ceph-mon[29774]: 7.4 scrub ok 2024-09-20T22:57:07.770 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:07 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:57:07.770 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:07 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:57:07.771 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:07 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:57:07.780 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:57:07.801 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:07.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:07.802 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:57:07.902 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:57:07.924 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:07.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:07.925 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:57:07.991 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:57:08.026 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:57:08.048 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:08.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:08.049 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:57:08.150 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:57:08.154 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.7 dump_historic_ops 2024-09-20T22:57:08.171 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:08.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:08.172 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:57:08.268 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-20T22:57:08.273 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:57:08.299 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:08.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:08.299 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:57:08.366 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:57:08 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: ::ffff:172.21.15.154 - - [20/Sep/2024:22:57:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-20T22:57:08.369 INFO:tasks.rados.rados.0.smithi084.stdout:3832: done (2 left) 2024-09-20T22:57:08.369 INFO:tasks.rados.rados.0.smithi084.stdout:3833: done (1 left) 2024-09-20T22:57:08.369 INFO:tasks.rados.rados.0.smithi084.stdout:3834: done (0 left) 2024-09-20T22:57:08.369 INFO:tasks.rados.rados.0.smithi084.stdout:3835: read oid 29 snap -1 2024-09-20T22:57:08.369 INFO:tasks.rados.rados.0.smithi084.stdout:3835: expect (ObjNum 618 snap 184 seq_num 618) 2024-09-20T22:57:08.369 INFO:tasks.rados.rados.0.smithi084.stdout:3836: snap_create 2024-09-20T22:57:08.381 INFO:tasks.rados.rados.0.smithi084.stdout:3835: expect (ObjNum 618 snap 184 seq_num 618) 2024-09-20T22:57:08.400 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:57:08.416 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:57:08.425 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:08.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:08.426 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:57:08.527 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:57:08.548 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:08.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:08.549 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:57:08.569 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:57:08.650 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:57:08.671 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:08.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:08.672 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:57:08.690 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:08 smithi084 ceph-mon[40024]: osdmap e947: 8 total, 7 up, 7 in 2024-09-20T22:57:08.690 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:08 smithi084 ceph-mon[40024]: 7.16 scrub starts 2024-09-20T22:57:08.690 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:08 smithi084 ceph-mon[40024]: 7.16 scrub ok 2024-09-20T22:57:08.690 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:08 smithi084 ceph-mon[40024]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-20T22:57:08.691 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:08 smithi084 ceph-mon[40024]: osdmap e948: 8 total, 7 up, 7 in 2024-09-20T22:57:08.691 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:08 smithi084 ceph-mon[29774]: osdmap e947: 8 total, 7 up, 7 in 2024-09-20T22:57:08.691 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:08 smithi084 ceph-mon[29774]: 7.16 scrub starts 2024-09-20T22:57:08.691 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:08 smithi084 ceph-mon[29774]: 7.16 scrub ok 2024-09-20T22:57:08.691 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:08 smithi084 ceph-mon[29774]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-20T22:57:08.692 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:08 smithi084 ceph-mon[29774]: osdmap e948: 8 total, 7 up, 7 in 2024-09-20T22:57:08.726 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:08 smithi154 ceph-mon[33264]: osdmap e947: 8 total, 7 up, 7 in 2024-09-20T22:57:08.726 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:08 smithi154 ceph-mon[33264]: 7.16 scrub starts 2024-09-20T22:57:08.726 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:08 smithi154 ceph-mon[33264]: 7.16 scrub ok 2024-09-20T22:57:08.726 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:08 smithi154 ceph-mon[33264]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-20T22:57:08.726 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:08 smithi154 ceph-mon[33264]: osdmap e948: 8 total, 7 up, 7 in 2024-09-20T22:57:08.773 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:57:08.795 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:08.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:08.796 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:57:08.897 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:57:08.915 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-20T22:57:08.915 INFO:teuthology.orchestra.run.smithi154.stdout: "size": 20, 2024-09-20T22:57:08.915 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 600, 2024-09-20T22:57:08.915 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [ 2024-09-20T22:57:08.915 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.915 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:3671 7.5 7:a160b5d4:::smithi08477203-16:head [copy-from ver 236] snapc 91=[91,8e,8d,8c,8b,8a,88,87,86,85,84,80,7f,7a,74,73,6b] ondisk+write+known_if_redirected+supports_pool_eio e370)", 2024-09-20T22:57:08.915 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:47:30.237235+0000", 2024-09-20T22:57:08.915 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 578.675573181, 2024-09-20T22:57:08.915 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.019719474000000001, 2024-09-20T22:57:08.916 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:57:08.916 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:57:08.916 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:57:08.916 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:57:08.916 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:57:08.916 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:57:08.916 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 3671 2024-09-20T22:57:08.916 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.916 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:57:08.916 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.916 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:57:08.916 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:30.237235+0000", 2024-09-20T22:57:08.916 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:57:08.916 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.917 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.917 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:57:08.917 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:30.237235+0000", 2024-09-20T22:57:08.917 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:57:08.917 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.917 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.917 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:57:08.917 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:30.237236+0000", 2024-09-20T22:57:08.917 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.5570000000000001e-06 2024-09-20T22:57:08.917 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.917 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.917 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:57:08.917 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:30.237241+0000", 2024-09-20T22:57:08.917 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.3279999999999999e-06 2024-09-20T22:57:08.918 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.918 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.918 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:57:08.918 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:30.237252+0000", 2024-09-20T22:57:08.918 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1004e-05 2024-09-20T22:57:08.918 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.918 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.918 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:57:08.918 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:30.237263+0000", 2024-09-20T22:57:08.918 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1341e-05 2024-09-20T22:57:08.918 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.918 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.918 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:57:08.919 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:30.237319+0000", 2024-09-20T22:57:08.919 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.5939000000000001e-05 2024-09-20T22:57:08.919 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.919 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.919 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:57:08.919 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:30.237436+0000", 2024-09-20T22:57:08.919 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000116866 2024-09-20T22:57:08.919 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.919 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.919 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 1,2", 2024-09-20T22:57:08.919 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:30.244330+0000", 2024-09-20T22:57:08.919 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0068945619999999999 2024-09-20T22:57:08.920 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.920 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.920 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:57:08.920 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:30.249103+0000", 2024-09-20T22:57:08.920 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0047726319999999997 2024-09-20T22:57:08.920 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.920 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.920 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:57:08.920 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:30.255771+0000", 2024-09-20T22:57:08.920 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.006667666 2024-09-20T22:57:08.920 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.920 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.920 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:57:08.920 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:30.256879+0000", 2024-09-20T22:57:08.921 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0011083110000000001 2024-09-20T22:57:08.921 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.921 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.921 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:57:08.921 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:30.256919+0000", 2024-09-20T22:57:08.921 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.9589000000000003e-05 2024-09-20T22:57:08.921 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.921 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.921 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:57:08.921 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:30.256954+0000", 2024-09-20T22:57:08.921 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.5679000000000002e-05 2024-09-20T22:57:08.921 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:57:08.922 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:57:08.922 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:57:08.922 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.922 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.922 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:3747 7.c e383/94 7:3f865ef5:::smithi08477203-28:head v 383'822, mlcod=382'818)", 2024-09-20T22:57:08.922 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:47:43.267267+0000", 2024-09-20T22:57:08.922 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 565.64554071700002, 2024-09-20T22:57:08.922 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.014405072, 2024-09-20T22:57:08.922 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:57:08.922 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:57:08.922 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:57:08.922 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:57:08.922 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.922 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:57:08.923 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.267267+0000", 2024-09-20T22:57:08.923 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:57:08.923 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.923 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.923 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:57:08.923 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.267267+0000", 2024-09-20T22:57:08.923 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:57:08.923 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.923 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.923 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:57:08.923 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.267267+0000", 2024-09-20T22:57:08.923 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.73e-07 2024-09-20T22:57:08.923 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.924 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.924 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:57:08.924 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.269537+0000", 2024-09-20T22:57:08.924 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0022696650000000001 2024-09-20T22:57:08.924 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.924 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.924 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:57:08.924 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.269564+0000", 2024-09-20T22:57:08.924 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.6653999999999999e-05 2024-09-20T22:57:08.924 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.924 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.924 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:57:08.924 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.269575+0000", 2024-09-20T22:57:08.924 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1661999999999999e-05 2024-09-20T22:57:08.925 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.925 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.925 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:57:08.925 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.269672+0000", 2024-09-20T22:57:08.925 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.6430999999999999e-05 2024-09-20T22:57:08.925 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.925 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.925 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:57:08.925 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.269716+0000", 2024-09-20T22:57:08.925 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.4518000000000003e-05 2024-09-20T22:57:08.925 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.925 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.925 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:57:08.926 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.281609+0000", 2024-09-20T22:57:08.926 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.011892982999999999 2024-09-20T22:57:08.926 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.926 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.926 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:57:08.926 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:47:43.281672+0000", 2024-09-20T22:57:08.926 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.2985999999999996e-05 2024-09-20T22:57:08.926 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:57:08.926 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:57:08.926 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:57:08.926 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.926 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.927 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:6253 7.9 e629/602 7:91f2d47c:::smithi08477203-8:head v 629'948, mlcod=627'945)", 2024-09-20T22:57:08.927 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:51:49.626110+0000", 2024-09-20T22:57:08.927 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 319.28669782700001, 2024-09-20T22:57:08.927 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.017238303, 2024-09-20T22:57:08.927 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:57:08.927 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:57:08.927 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:57:08.927 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:57:08.927 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.927 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:57:08.927 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:49.626110+0000", 2024-09-20T22:57:08.927 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:57:08.927 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.927 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.928 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:57:08.928 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:49.626110+0000", 2024-09-20T22:57:08.928 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:57:08.928 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.928 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.928 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:57:08.928 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:49.626111+0000", 2024-09-20T22:57:08.928 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.33e-06 2024-09-20T22:57:08.928 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.928 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.928 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:57:08.928 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:49.633470+0000", 2024-09-20T22:57:08.928 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0073589700000000003 2024-09-20T22:57:08.928 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.928 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.929 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:57:08.929 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:49.633492+0000", 2024-09-20T22:57:08.929 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.1443000000000001e-05 2024-09-20T22:57:08.929 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.929 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.929 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:57:08.929 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:49.633503+0000", 2024-09-20T22:57:08.929 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1610000000000001e-05 2024-09-20T22:57:08.929 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.929 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.929 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:57:08.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:08.931 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:49.633575+0000", 2024-09-20T22:57:08.931 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.1433999999999994e-05 2024-09-20T22:57:08.931 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.931 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.931 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:57:08.931 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:49.633600+0000", 2024-09-20T22:57:08.931 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.4960000000000002e-05 2024-09-20T22:57:08.931 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.931 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.931 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:57:08.931 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:49.643306+0000", 2024-09-20T22:57:08.931 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0097065109999999993 2024-09-20T22:57:08.931 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.932 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.932 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:57:08.932 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:49.643348+0000", 2024-09-20T22:57:08.932 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.2045000000000001e-05 2024-09-20T22:57:08.932 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:57:08.932 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:57:08.932 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:57:08.932 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.932 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.932 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:6659 7.5 7:a160b5d4:::smithi08477203-16:head [copy-from ver 1807] snapc 12d=[12d,12c,12b,12a,128,127,126,124,122,121,120,11f,11e,11b,119,118,115,112,10e,10c,107,103,fd,fb,fa,f8,ed,d7,c2,bf] ondisk+write+known_if_redirected+supports_pool_eio e674)", 2024-09-20T22:57:08.932 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:52:34.370352+0000", 2024-09-20T22:57:08.933 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 274.54245633300002, 2024-09-20T22:57:08.933 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.018083765000000002, 2024-09-20T22:57:08.933 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:57:08.933 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:57:08.933 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:57:08.933 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:57:08.933 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:57:08.933 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:57:08.933 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 6659 2024-09-20T22:57:08.933 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.933 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:57:08.933 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.933 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:57:08.933 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:34.370352+0000", 2024-09-20T22:57:08.934 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:57:08.934 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.934 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.934 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:57:08.934 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:34.370352+0000", 2024-09-20T22:57:08.934 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:57:08.934 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.934 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.934 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:57:08.934 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:34.370353+0000", 2024-09-20T22:57:08.934 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.61e-06 2024-09-20T22:57:08.934 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.934 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.934 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:57:08.935 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:34.370360+0000", 2024-09-20T22:57:08.935 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.6760000000000001e-06 2024-09-20T22:57:08.935 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.935 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.935 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:57:08.935 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:34.370371+0000", 2024-09-20T22:57:08.935 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1155000000000001e-05 2024-09-20T22:57:08.935 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.935 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.935 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:57:08.935 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:34.370380+0000", 2024-09-20T22:57:08.935 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.9039999999999998e-06 2024-09-20T22:57:08.935 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.935 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.936 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:57:08.936 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:34.370461+0000", 2024-09-20T22:57:08.936 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.0667000000000005e-05 2024-09-20T22:57:08.936 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.936 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.936 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:57:08.936 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:34.370571+0000", 2024-09-20T22:57:08.936 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000110129 2024-09-20T22:57:08.936 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.936 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.936 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 0,2", 2024-09-20T22:57:08.936 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:34.378118+0000", 2024-09-20T22:57:08.937 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0075469489999999998 2024-09-20T22:57:08.937 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.937 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.937 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:57:08.937 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:34.382656+0000", 2024-09-20T22:57:08.937 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0045383239999999998 2024-09-20T22:57:08.937 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.937 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.937 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:57:08.937 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:34.385168+0000", 2024-09-20T22:57:08.937 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0025123770000000001 2024-09-20T22:57:08.937 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.937 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.937 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:57:08.938 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:34.388343+0000", 2024-09-20T22:57:08.938 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.003174713 2024-09-20T22:57:08.938 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.938 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.938 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:57:08.938 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:34.388382+0000", 2024-09-20T22:57:08.938 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.8550000000000002e-05 2024-09-20T22:57:08.938 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.938 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.938 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:57:08.938 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:34.388435+0000", 2024-09-20T22:57:08.938 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.3711000000000001e-05 2024-09-20T22:57:08.938 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:57:08.939 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:57:08.939 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:57:08.939 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.939 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.939 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:6829 7.5 7:a160b5d4:::smithi08477203-16:head [write 548159~419732,stat] snapc 136=[136,135,134,133,132,131,130,12f,12e,12d,12c,12b,12a,128,127,126,124,122,121,11f,11b,119,112,10e,10c,107,103,fb,fa,f8,ed,d7,c2,bf] ondisk+write+known_if_redirected+supports_pool_eio e688)", 2024-09-20T22:57:08.939 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:52:48.308503+0000", 2024-09-20T22:57:08.939 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 260.60430500799998, 2024-09-20T22:57:08.939 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.020227210999999998, 2024-09-20T22:57:08.939 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:57:08.939 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:57:08.939 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:57:08.939 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:57:08.939 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:57:08.939 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:57:08.940 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 6829 2024-09-20T22:57:08.940 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.940 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:57:08.940 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.940 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:57:08.940 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.308503+0000", 2024-09-20T22:57:08.940 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:57:08.940 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.940 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.940 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:57:08.940 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.308503+0000", 2024-09-20T22:57:08.940 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:57:08.940 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.941 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.941 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:57:08.941 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.308505+0000", 2024-09-20T22:57:08.941 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.765e-06 2024-09-20T22:57:08.941 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.941 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.941 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:57:08.941 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.308957+0000", 2024-09-20T22:57:08.941 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00045254399999999998 2024-09-20T22:57:08.941 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.941 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.941 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:57:08.941 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.308975+0000", 2024-09-20T22:57:08.942 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.7489000000000001e-05 2024-09-20T22:57:08.942 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.942 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.942 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:57:08.942 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.308983+0000", 2024-09-20T22:57:08.942 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.4500000000000004e-06 2024-09-20T22:57:08.942 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.942 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.942 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:57:08.942 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.309063+0000", 2024-09-20T22:57:08.942 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.0264000000000003e-05 2024-09-20T22:57:08.942 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.942 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.942 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:57:08.943 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.309200+0000", 2024-09-20T22:57:08.943 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00013697500000000001 2024-09-20T22:57:08.943 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.943 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.943 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 0,2", 2024-09-20T22:57:08.943 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.309374+0000", 2024-09-20T22:57:08.943 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000173597 2024-09-20T22:57:08.943 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.943 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.943 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:57:08.943 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.318972+0000", 2024-09-20T22:57:08.943 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0095980430000000005 2024-09-20T22:57:08.943 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.943 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.944 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:57:08.944 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.326825+0000", 2024-09-20T22:57:08.944 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0078531720000000003 2024-09-20T22:57:08.944 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.944 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.944 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:57:08.944 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.328638+0000", 2024-09-20T22:57:08.944 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0018131969999999999 2024-09-20T22:57:08.944 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.944 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.944 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:57:08.944 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.328685+0000", 2024-09-20T22:57:08.944 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.6536999999999998e-05 2024-09-20T22:57:08.944 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.944 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.945 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:57:08.945 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.328730+0000", 2024-09-20T22:57:08.945 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.5178e-05 2024-09-20T22:57:08.945 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:57:08.945 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:57:08.945 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:57:08.945 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.945 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.945 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:7173 7.8 e721/430 7:1e9e2524:::smithi08477203-5:head v 721'865, mlcod=721'861)", 2024-09-20T22:57:08.945 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:53:21.336783+0000", 2024-09-20T22:57:08.945 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 227.57602513800001, 2024-09-20T22:57:08.945 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.015521446, 2024-09-20T22:57:08.945 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:57:08.945 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:57:08.946 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:57:08.946 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:57:08.946 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.946 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:57:08.946 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:21.336783+0000", 2024-09-20T22:57:08.946 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:57:08.946 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.946 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.946 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:57:08.946 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:21.336783+0000", 2024-09-20T22:57:08.946 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:57:08.946 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.947 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.947 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:57:08.947 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:21.336784+0000", 2024-09-20T22:57:08.947 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.7899999999999997e-07 2024-09-20T22:57:08.947 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.947 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.947 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:57:08.947 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:21.344323+0000", 2024-09-20T22:57:08.947 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0075393090000000001 2024-09-20T22:57:08.947 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.948 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.948 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:57:08.948 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:21.344349+0000", 2024-09-20T22:57:08.948 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.5712e-05 2024-09-20T22:57:08.948 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.948 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.948 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:57:08.948 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:21.344362+0000", 2024-09-20T22:57:08.948 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3131e-05 2024-09-20T22:57:08.948 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.949 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.949 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:57:08.949 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:21.344434+0000", 2024-09-20T22:57:08.949 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.1837999999999999e-05 2024-09-20T22:57:08.949 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.949 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.949 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:57:08.949 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:21.344454+0000", 2024-09-20T22:57:08.949 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.0404999999999999e-05 2024-09-20T22:57:08.949 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.949 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.950 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:57:08.950 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:21.352250+0000", 2024-09-20T22:57:08.950 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0077961489999999996 2024-09-20T22:57:08.950 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.950 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.950 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:57:08.950 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:21.352304+0000", 2024-09-20T22:57:08.950 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.4122999999999997e-05 2024-09-20T22:57:08.950 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:57:08.950 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:57:08.950 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:57:08.950 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.950 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.950 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:7387 7.9 e743/602 7:907aec0e:::smithi08477203-3:head v 743'1242, mlcod=741'1234)", 2024-09-20T22:57:08.951 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:53:43.309636+0000", 2024-09-20T22:57:08.951 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 205.60317196599999, 2024-09-20T22:57:08.951 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.032548296999999997, 2024-09-20T22:57:08.951 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:57:08.951 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:57:08.951 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:57:08.951 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:57:08.951 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.951 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:57:08.951 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:43.309636+0000", 2024-09-20T22:57:08.951 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:57:08.951 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.951 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.951 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:57:08.952 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:43.309636+0000", 2024-09-20T22:57:08.952 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:57:08.952 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.952 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.952 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:57:08.952 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:43.309637+0000", 2024-09-20T22:57:08.952 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.1699999999999997e-07 2024-09-20T22:57:08.952 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.952 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.952 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:57:08.952 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:43.338457+0000", 2024-09-20T22:57:08.952 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.028819827999999999 2024-09-20T22:57:08.952 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.952 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.952 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:57:08.953 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:43.338482+0000", 2024-09-20T22:57:08.953 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.55e-05 2024-09-20T22:57:08.953 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.953 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.953 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:57:08.953 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:43.338493+0000", 2024-09-20T22:57:08.953 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1308000000000001e-05 2024-09-20T22:57:08.953 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.953 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.953 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:57:08.953 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:43.338561+0000", 2024-09-20T22:57:08.953 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.7535999999999998e-05 2024-09-20T22:57:08.953 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.953 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.954 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:57:08.954 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:43.338585+0000", 2024-09-20T22:57:08.954 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.3649e-05 2024-09-20T22:57:08.954 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.954 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.954 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:57:08.954 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:43.342155+0000", 2024-09-20T22:57:08.954 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0035700509999999999 2024-09-20T22:57:08.954 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.954 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.954 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:57:08.954 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:43.342184+0000", 2024-09-20T22:57:08.954 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.9607999999999998e-05 2024-09-20T22:57:08.954 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:57:08.955 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:57:08.955 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:57:08.955 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.955 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.955 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24415.0:744 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch (1) in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e775)", 2024-09-20T22:57:08.955 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:54:16.547604+0000", 2024-09-20T22:57:08.955 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 172.365204003, 2024-09-20T22:57:08.955 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.79000451299999996, 2024-09-20T22:57:08.955 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:57:08.955 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:57:08.955 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:57:08.955 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:57:08.955 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24415", 2024-09-20T22:57:08.955 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/2056244102", 2024-09-20T22:57:08.955 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 744 2024-09-20T22:57:08.955 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.956 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:57:08.956 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.956 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:57:08.956 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:54:16.547604+0000", 2024-09-20T22:57:08.956 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:57:08.956 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.956 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.956 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:57:08.956 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:54:16.547604+0000", 2024-09-20T22:57:08.956 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:57:08.956 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.956 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.956 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:57:08.956 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:54:16.547607+0000", 2024-09-20T22:57:08.956 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.7460000000000001e-06 2024-09-20T22:57:08.957 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.957 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.957 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:57:08.957 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:54:16.547620+0000", 2024-09-20T22:57:08.957 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3164e-05 2024-09-20T22:57:08.957 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.957 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.957 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:57:08.957 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:54:16.547647+0000", 2024-09-20T22:57:08.957 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.7679e-05 2024-09-20T22:57:08.957 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.957 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.957 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:57:08.957 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:54:16.547663+0000", 2024-09-20T22:57:08.957 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.523e-05 2024-09-20T22:57:08.957 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.957 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.958 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:57:08.958 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:54:16.547788+0000", 2024-09-20T22:57:08.958 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00012490999999999999 2024-09-20T22:57:08.958 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.958 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.958 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-20T22:57:08.958 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:54:16.547801+0000", 2024-09-20T22:57:08.958 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3158e-05 2024-09-20T22:57:08.958 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.958 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.958 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:57:08.958 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:54:17.337429+0000", 2024-09-20T22:57:08.958 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.78962777299999998 2024-09-20T22:57:08.958 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.958 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.959 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:57:08.959 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:54:17.337522+0000", 2024-09-20T22:57:08.959 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.3556999999999999e-05 2024-09-20T22:57:08.959 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.959 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.959 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:57:08.959 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:54:17.337608+0000", 2024-09-20T22:57:08.959 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.6296000000000002e-05 2024-09-20T22:57:08.959 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:57:08.959 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:57:08.959 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:57:08.959 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.959 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.959 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:7834 7.c e786/94 7:3f0fe232:::smithi08477203-4:head v 786'2534, mlcod=785'2528)", 2024-09-20T22:57:08.960 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:54:26.332738+0000", 2024-09-20T22:57:08.960 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 162.580070009, 2024-09-20T22:57:08.960 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.010142463000000001, 2024-09-20T22:57:08.960 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:57:08.960 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:57:08.960 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:57:08.960 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:57:08.960 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.960 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:57:08.960 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:54:26.332738+0000", 2024-09-20T22:57:08.960 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:57:08.960 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.960 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.960 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:57:08.961 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:54:26.332738+0000", 2024-09-20T22:57:08.961 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:57:08.961 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.961 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.961 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:57:08.961 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:54:26.332739+0000", 2024-09-20T22:57:08.961 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2130000000000001e-06 2024-09-20T22:57:08.961 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.961 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.961 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:57:08.961 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:54:26.335206+0000", 2024-09-20T22:57:08.961 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0024668260000000001 2024-09-20T22:57:08.961 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.961 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.961 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:57:08.961 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:54:26.335237+0000", 2024-09-20T22:57:08.962 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.1106999999999999e-05 2024-09-20T22:57:08.962 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.962 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.962 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:57:08.962 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:54:26.335252+0000", 2024-09-20T22:57:08.962 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.5288e-05 2024-09-20T22:57:08.962 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.962 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.962 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:57:08.962 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:54:26.335348+0000", 2024-09-20T22:57:08.962 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.5351000000000002e-05 2024-09-20T22:57:08.962 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.962 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.962 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:57:08.962 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:54:26.335379+0000", 2024-09-20T22:57:08.963 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.1807000000000002e-05 2024-09-20T22:57:08.963 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.963 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.963 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:57:08.963 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:54:26.342798+0000", 2024-09-20T22:57:08.963 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0074187660000000002 2024-09-20T22:57:08.963 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.963 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.963 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:57:08.963 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:54:26.342880+0000", 2024-09-20T22:57:08.963 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.2105000000000001e-05 2024-09-20T22:57:08.963 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:57:08.963 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:57:08.964 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:57:08.964 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.964 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.964 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:8165 7.f e824/391 7:fa55e5ae:::smithi08477203-7:head v 824'927, mlcod=819'923)", 2024-09-20T22:57:08.964 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:55:04.351014+0000", 2024-09-20T22:57:08.964 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 124.561793843, 2024-09-20T22:57:08.964 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0096140429999999992, 2024-09-20T22:57:08.964 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:57:08.964 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:57:08.964 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:57:08.964 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:57:08.964 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.964 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:57:08.964 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:04.351014+0000", 2024-09-20T22:57:08.965 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:57:08.965 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.965 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.965 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:57:08.965 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:04.351014+0000", 2024-09-20T22:57:08.965 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:57:08.965 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.965 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.965 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:57:08.965 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:04.351014+0000", 2024-09-20T22:57:08.965 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9999999999999999e-07 2024-09-20T22:57:08.965 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.965 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.965 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:57:08.965 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:04.354512+0000", 2024-09-20T22:57:08.965 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0034980990000000002 2024-09-20T22:57:08.965 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.966 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.966 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:57:08.966 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:04.354544+0000", 2024-09-20T22:57:08.966 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.1294999999999998e-05 2024-09-20T22:57:08.966 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.966 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.966 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:57:08.966 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:04.354557+0000", 2024-09-20T22:57:08.966 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3825999999999999e-05 2024-09-20T22:57:08.966 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.966 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.966 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:57:08.967 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:04.354685+0000", 2024-09-20T22:57:08.967 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00012776000000000001 2024-09-20T22:57:08.967 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.967 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.967 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:57:08.967 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:04.354717+0000", 2024-09-20T22:57:08.967 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.2054000000000002e-05 2024-09-20T22:57:08.967 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.967 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.967 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:57:08.967 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:04.360567+0000", 2024-09-20T22:57:08.967 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0058492709999999996 2024-09-20T22:57:08.967 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.967 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.967 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:57:08.968 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:04.360628+0000", 2024-09-20T22:57:08.968 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.1538000000000006e-05 2024-09-20T22:57:08.968 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:57:08.968 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:57:08.968 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:57:08.968 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.968 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.968 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:8189 7.c e825/94 7:318c7512:::smithi08477203-44:head v 825'2641, mlcod=825'2638)", 2024-09-20T22:57:08.968 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:55:05.581130+0000", 2024-09-20T22:57:08.968 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 123.33167806900001, 2024-09-20T22:57:08.968 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0077411670000000002, 2024-09-20T22:57:08.968 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:57:08.968 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:57:08.968 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:57:08.969 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:57:08.969 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.969 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:57:08.969 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:05.581130+0000", 2024-09-20T22:57:08.969 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:57:08.969 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.969 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.969 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:57:08.969 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:05.581130+0000", 2024-09-20T22:57:08.969 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:57:08.969 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.969 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.969 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:57:08.969 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:05.581133+0000", 2024-09-20T22:57:08.969 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.2040000000000002e-06 2024-09-20T22:57:08.970 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.970 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.970 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:57:08.970 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:05.582999+0000", 2024-09-20T22:57:08.970 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0018663919999999999 2024-09-20T22:57:08.970 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.970 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.970 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:57:08.970 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:05.583026+0000", 2024-09-20T22:57:08.970 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.6503999999999999e-05 2024-09-20T22:57:08.970 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.970 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.970 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:57:08.970 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:05.583040+0000", 2024-09-20T22:57:08.970 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4430000000000001e-05 2024-09-20T22:57:08.971 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.971 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.971 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:57:08.971 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:05.583132+0000", 2024-09-20T22:57:08.971 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.1939000000000002e-05 2024-09-20T22:57:08.971 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.971 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.971 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:57:08.971 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:05.583163+0000", 2024-09-20T22:57:08.971 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.0341999999999999e-05 2024-09-20T22:57:08.971 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.971 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.971 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:57:08.971 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:05.588810+0000", 2024-09-20T22:57:08.971 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0056476629999999998 2024-09-20T22:57:08.972 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.972 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.972 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:57:08.972 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:05.588871+0000", 2024-09-20T22:57:08.972 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.0693000000000003e-05 2024-09-20T22:57:08.972 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:57:08.972 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:57:08.972 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:57:08.972 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.972 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.972 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:08.973 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:57:08.973 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:8294 7.5 7:a160b5d4:::smithi08477203-16:head [read 0~3042133,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e831)", 2024-09-20T22:57:08.973 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:55:11.340475+0000", 2024-09-20T22:57:08.974 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 117.572333222, 2024-09-20T22:57:08.974 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0075077099999999999, 2024-09-20T22:57:08.974 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:57:08.974 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:57:08.974 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:57:08.974 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:57:08.974 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:57:08.974 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:57:08.974 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 8294 2024-09-20T22:57:08.974 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.974 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:57:08.974 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.974 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:57:08.974 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:11.340475+0000", 2024-09-20T22:57:08.975 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:57:08.975 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.975 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.975 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:57:08.975 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:11.340475+0000", 2024-09-20T22:57:08.975 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:57:08.975 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.975 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.975 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:57:08.975 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:11.340475+0000", 2024-09-20T22:57:08.975 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.1099999999999996e-07 2024-09-20T22:57:08.975 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.975 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.975 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:57:08.976 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:11.340479+0000", 2024-09-20T22:57:08.976 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.4170000000000001e-06 2024-09-20T22:57:08.976 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.976 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.976 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:57:08.976 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:11.340484+0000", 2024-09-20T22:57:08.976 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.7570000000000001e-06 2024-09-20T22:57:08.976 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.976 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.976 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:57:08.976 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:11.340489+0000", 2024-09-20T22:57:08.976 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.3819999999999997e-06 2024-09-20T22:57:08.976 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.976 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.977 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:57:08.977 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:11.340800+0000", 2024-09-20T22:57:08.977 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00031126299999999999 2024-09-20T22:57:08.977 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.977 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.977 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for rw locks", 2024-09-20T22:57:08.977 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:11.340908+0000", 2024-09-20T22:57:08.977 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000108088 2024-09-20T22:57:08.977 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.977 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.977 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:57:08.977 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:11.347505+0000", 2024-09-20T22:57:08.977 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0065973050000000004 2024-09-20T22:57:08.977 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.977 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.978 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:57:08.978 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:11.347565+0000", 2024-09-20T22:57:08.978 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.9567000000000001e-05 2024-09-20T22:57:08.978 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.978 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.978 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:57:08.978 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:11.347982+0000", 2024-09-20T22:57:08.978 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00041742 2024-09-20T22:57:08.978 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:57:08.978 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:57:08.978 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:57:08.978 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.979 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.979 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:8442 7.c e849/94 7:31d0211f:::smithi08477203-25:head v 849'2704, mlcod=845'2700)", 2024-09-20T22:57:08.979 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:55:29.375191+0000", 2024-09-20T22:57:08.979 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 99.537617018000006, 2024-09-20T22:57:08.979 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.011331713, 2024-09-20T22:57:08.979 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:57:08.979 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:57:08.979 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:57:08.979 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:57:08.979 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.979 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:57:08.979 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:29.375191+0000", 2024-09-20T22:57:08.979 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:57:08.979 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.979 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.980 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:57:08.980 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:29.375191+0000", 2024-09-20T22:57:08.980 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:57:08.980 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.980 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.980 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:57:08.980 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:29.375191+0000", 2024-09-20T22:57:08.980 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.3200000000000001e-07 2024-09-20T22:57:08.980 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.980 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.980 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:57:08.980 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:29.377130+0000", 2024-09-20T22:57:08.980 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0019390779999999999 2024-09-20T22:57:08.981 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.981 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.981 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:57:08.981 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:29.377142+0000", 2024-09-20T22:57:08.981 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1712000000000001e-05 2024-09-20T22:57:08.981 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.981 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.981 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:57:08.981 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:29.377174+0000", 2024-09-20T22:57:08.981 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.1727000000000003e-05 2024-09-20T22:57:08.981 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.981 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.981 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:57:08.981 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:29.377522+0000", 2024-09-20T22:57:08.981 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00034803200000000002 2024-09-20T22:57:08.982 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.982 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.982 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:57:08.982 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:29.377553+0000", 2024-09-20T22:57:08.982 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.1130999999999997e-05 2024-09-20T22:57:08.982 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.982 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.982 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:57:08.982 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:29.386478+0000", 2024-09-20T22:57:08.982 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0089250430000000006 2024-09-20T22:57:08.982 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.982 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.982 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:57:08.983 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:29.386523+0000", 2024-09-20T22:57:08.983 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.4758000000000001e-05 2024-09-20T22:57:08.983 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:57:08.983 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:57:08.983 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:57:08.983 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.983 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.983 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:8804 7.c e883/94 7:318c7512:::smithi08477203-44:head v 883'2842, mlcod=881'2835)", 2024-09-20T22:57:08.983 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:56:03.367132+0000", 2024-09-20T22:57:08.983 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 65.545675517999996, 2024-09-20T22:57:08.983 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0084714460000000005, 2024-09-20T22:57:08.983 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:57:08.983 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:57:08.983 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:57:08.984 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:57:08.984 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.984 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:57:08.984 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:56:03.367132+0000", 2024-09-20T22:57:08.984 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:57:08.984 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.984 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.984 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:57:08.984 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:56:03.367132+0000", 2024-09-20T22:57:08.984 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:57:08.984 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.984 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.984 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:57:08.984 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:56:03.367133+0000", 2024-09-20T22:57:08.985 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.3099999999999996e-07 2024-09-20T22:57:08.985 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.985 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.985 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:57:08.985 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:56:03.367141+0000", 2024-09-20T22:57:08.985 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.2540000000000003e-06 2024-09-20T22:57:08.985 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.985 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.985 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:57:08.985 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:56:03.367153+0000", 2024-09-20T22:57:08.985 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2534999999999999e-05 2024-09-20T22:57:08.985 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.986 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.986 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:57:08.986 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:56:03.367160+0000", 2024-09-20T22:57:08.986 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.2320000000000004e-06 2024-09-20T22:57:08.986 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.986 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.986 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:57:08.986 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:56:03.367454+0000", 2024-09-20T22:57:08.986 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00029333099999999997 2024-09-20T22:57:08.986 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.986 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.986 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:57:08.986 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:56:03.367480+0000", 2024-09-20T22:57:08.986 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.6781999999999999e-05 2024-09-20T22:57:08.987 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.987 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.987 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:57:08.987 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:56:03.375540+0000", 2024-09-20T22:57:08.987 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0080594840000000004 2024-09-20T22:57:08.987 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.987 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.987 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:57:08.987 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:56:03.375604+0000", 2024-09-20T22:57:08.987 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.3897000000000002e-05 2024-09-20T22:57:08.987 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:57:08.987 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:57:08.987 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:57:08.987 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.987 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.987 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:8832 7.c e884/94 7:31d0211f:::smithi08477203-25:head v 884'2847, mlcod=883'2844)", 2024-09-20T22:57:08.988 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:56:04.509874+0000", 2024-09-20T22:57:08.988 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 64.402933947999998, 2024-09-20T22:57:08.988 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00504032, 2024-09-20T22:57:08.988 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:57:08.988 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:57:08.988 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:57:08.988 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:57:08.988 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.988 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:57:08.988 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:56:04.509874+0000", 2024-09-20T22:57:08.988 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:57:08.988 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.988 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.988 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:57:08.988 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:56:04.509874+0000", 2024-09-20T22:57:08.989 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:57:08.989 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.989 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.989 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:57:08.989 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:56:04.509874+0000", 2024-09-20T22:57:08.989 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.7300000000000002e-07 2024-09-20T22:57:08.989 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.989 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.989 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:57:08.989 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:56:04.511431+0000", 2024-09-20T22:57:08.989 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001556766 2024-09-20T22:57:08.989 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.989 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.989 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:57:08.989 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:56:04.511454+0000", 2024-09-20T22:57:08.989 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.3269e-05 2024-09-20T22:57:08.990 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.990 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.990 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:57:08.990 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:56:04.511467+0000", 2024-09-20T22:57:08.990 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2877999999999999e-05 2024-09-20T22:57:08.990 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.990 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.990 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:57:08.990 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:56:04.511554+0000", 2024-09-20T22:57:08.990 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.7087000000000002e-05 2024-09-20T22:57:08.990 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.990 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.990 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:57:08.991 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:56:04.511576+0000", 2024-09-20T22:57:08.991 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.1999000000000001e-05 2024-09-20T22:57:08.991 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.991 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.991 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:57:08.991 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:56:04.514874+0000", 2024-09-20T22:57:08.991 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.003297538 2024-09-20T22:57:08.991 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.991 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.991 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:57:08.991 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:56:04.514914+0000", 2024-09-20T22:57:08.991 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.0509999999999997e-05 2024-09-20T22:57:08.991 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:57:08.992 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:57:08.992 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:57:08.992 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.992 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.992 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:8839 7.5 7:a160b5d4:::smithi08477203-16:head [copy-from ver 2842] snapc 19d=[19d,19c,19b,19a,199,198,197,196,195,193,191,190,18f,18e,18d,18c,18a,189,185,183,182,180,17e,17d,17b,17a,179,177,175,174,173,172,16f,16e,16c,16b,160,15c,158,14f,14e,14a,13a,12f] ondisk+write+known_if_redirected+supports_pool_eio e885)", 2024-09-20T22:57:08.992 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:56:05.501335+0000", 2024-09-20T22:57:08.992 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 63.411473272000002, 2024-09-20T22:57:08.992 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.026669249999999999, 2024-09-20T22:57:08.992 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:57:08.992 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:57:08.992 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:57:08.992 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:57:08.992 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:57:08.992 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:57:08.993 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 8839 2024-09-20T22:57:08.993 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.993 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:57:08.993 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.993 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:57:08.993 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:56:05.501335+0000", 2024-09-20T22:57:08.993 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:57:08.993 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.993 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.993 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:57:08.993 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:56:05.501335+0000", 2024-09-20T22:57:08.993 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:57:08.993 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.994 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.994 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:57:08.994 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:56:05.501336+0000", 2024-09-20T22:57:08.994 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.643e-06 2024-09-20T22:57:08.994 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.994 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.994 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:57:08.994 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:56:05.501343+0000", 2024-09-20T22:57:08.994 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.3470000000000004e-06 2024-09-20T22:57:08.994 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.994 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.994 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:57:08.994 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:56:05.501354+0000", 2024-09-20T22:57:08.995 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1561e-05 2024-09-20T22:57:08.995 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.995 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.995 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:57:08.995 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:56:05.501363+0000", 2024-09-20T22:57:08.995 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.5040000000000002e-06 2024-09-20T22:57:08.995 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.995 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.995 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:57:08.995 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:56:05.501446+0000", 2024-09-20T22:57:08.995 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.3673999999999999e-05 2024-09-20T22:57:08.995 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.995 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.995 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:57:08.996 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:56:05.501608+0000", 2024-09-20T22:57:08.996 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000161399 2024-09-20T22:57:08.996 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.996 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.996 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 0,2", 2024-09-20T22:57:08.996 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:56:05.512738+0000", 2024-09-20T22:57:08.996 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.011130509 2024-09-20T22:57:08.996 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.996 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.996 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:57:08.996 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:56:05.516390+0000", 2024-09-20T22:57:08.996 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0036521320000000002 2024-09-20T22:57:08.996 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.997 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.997 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:57:08.997 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:56:05.527650+0000", 2024-09-20T22:57:08.997 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.011259567 2024-09-20T22:57:08.997 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.997 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.997 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:57:08.997 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:56:05.527894+0000", 2024-09-20T22:57:08.997 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00024384499999999999 2024-09-20T22:57:08.997 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.997 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.997 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:57:08.997 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:56:05.527942+0000", 2024-09-20T22:57:08.997 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.8569000000000002e-05 2024-09-20T22:57:08.997 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.998 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.998 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:57:08.998 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:56:05.528004+0000", 2024-09-20T22:57:08.998 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.1500000000000004e-05 2024-09-20T22:57:08.998 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:57:08.998 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:57:08.998 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:57:08.998 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.998 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.998 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:8840 7.5 7:a160b5d4:::smithi08477203-16:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e885)", 2024-09-20T22:57:08.998 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:56:05.501431+0000", 2024-09-20T22:57:08.998 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 63.411377334000001, 2024-09-20T22:57:08.998 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.014446565, 2024-09-20T22:57:08.999 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:57:08.999 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:57:08.999 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:57:08.999 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:57:08.999 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:57:08.999 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:57:08.999 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 8840 2024-09-20T22:57:08.999 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:08.999 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:57:08.999 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:08.999 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:57:08.999 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:56:05.501431+0000", 2024-09-20T22:57:08.999 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:57:08.999 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:09.000 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:09.000 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:57:09.000 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:56:05.501431+0000", 2024-09-20T22:57:09.000 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:57:09.000 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:09.000 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:09.000 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:57:09.000 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:56:05.501431+0000", 2024-09-20T22:57:09.000 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.1600000000000001e-07 2024-09-20T22:57:09.000 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:09.000 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:09.000 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:57:09.000 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:56:05.501434+0000", 2024-09-20T22:57:09.000 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.0699999999999998e-06 2024-09-20T22:57:09.000 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:09.001 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:09.001 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:57:09.001 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:56:05.501441+0000", 2024-09-20T22:57:09.001 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.7830000000000001e-06 2024-09-20T22:57:09.001 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:09.001 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:09.001 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:57:09.001 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:56:05.501447+0000", 2024-09-20T22:57:09.001 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.1630000000000001e-06 2024-09-20T22:57:09.001 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:09.001 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:09.001 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:57:09.001 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:56:05.501833+0000", 2024-09-20T22:57:09.001 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00038532200000000001 2024-09-20T22:57:09.001 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:09.002 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:09.002 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for blocked object", 2024-09-20T22:57:09.002 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:56:05.501902+0000", 2024-09-20T22:57:09.002 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.9587999999999998e-05 2024-09-20T22:57:09.002 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:09.002 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:09.002 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:57:09.002 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:56:05.515543+0000", 2024-09-20T22:57:09.002 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.013640664 2024-09-20T22:57:09.002 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:09.002 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:09.002 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:57:09.002 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:56:05.515736+0000", 2024-09-20T22:57:09.002 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00019273099999999999 2024-09-20T22:57:09.003 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:09.003 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:09.003 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:57:09.003 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:56:05.515877+0000", 2024-09-20T22:57:09.003 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00014162800000000001 2024-09-20T22:57:09.003 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:57:09.003 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:57:09.003 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:57:09.003 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:57:09.003 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:57:09.003 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-20T22:57:09.049 INFO:teuthology.orchestra.run.smithi084.stdout:7.c 2024-09-20T22:57:09.049 INFO:teuthology.orchestra.run.smithi084.stdout:7.3 2024-09-20T22:57:09.049 INFO:teuthology.orchestra.run.smithi084.stdout:7.1 2024-09-20T22:57:09.049 INFO:teuthology.orchestra.run.smithi084.stdout:7.2 2024-09-20T22:57:09.049 INFO:teuthology.orchestra.run.smithi084.stdout:7.14 2024-09-20T22:57:09.049 INFO:teuthology.orchestra.run.smithi084.stdout:7.13 2024-09-20T22:57:09.049 INFO:teuthology.orchestra.run.smithi084.stdout:7.12 2024-09-20T22:57:09.049 INFO:teuthology.orchestra.run.smithi084.stdout:7.10 2024-09-20T22:57:09.050 INFO:teuthology.orchestra.run.smithi084.stdout:6.f 2024-09-20T22:57:09.050 INFO:teuthology.orchestra.run.smithi084.stdout:6.c 2024-09-20T22:57:09.050 INFO:teuthology.orchestra.run.smithi084.stdout:7.f 2024-09-20T22:57:09.050 INFO:teuthology.orchestra.run.smithi084.stdout:6.b 2024-09-20T22:57:09.050 INFO:teuthology.orchestra.run.smithi084.stdout:6.8 2024-09-20T22:57:09.050 INFO:teuthology.orchestra.run.smithi084.stdout:7.17 2024-09-20T22:57:09.050 INFO:teuthology.orchestra.run.smithi084.stdout:6.6 2024-09-20T22:57:09.050 INFO:teuthology.orchestra.run.smithi084.stdout:7.6 2024-09-20T22:57:09.050 INFO:teuthology.orchestra.run.smithi084.stdout:1.7 2024-09-20T22:57:09.050 INFO:teuthology.orchestra.run.smithi084.stdout:6.9 2024-09-20T22:57:09.050 INFO:teuthology.orchestra.run.smithi084.stdout:4.10 2024-09-20T22:57:09.050 INFO:teuthology.orchestra.run.smithi084.stdout:7.a 2024-09-20T22:57:09.050 INFO:teuthology.orchestra.run.smithi084.stdout:6.19 2024-09-20T22:57:09.050 INFO:teuthology.orchestra.run.smithi084.stdout:3.d 2024-09-20T22:57:09.050 INFO:teuthology.orchestra.run.smithi084.stdout:5.16 2024-09-20T22:57:09.051 INFO:teuthology.orchestra.run.smithi084.stdout:3.7 2024-09-20T22:57:09.051 INFO:teuthology.orchestra.run.smithi084.stdout:1.6 2024-09-20T22:57:09.051 INFO:teuthology.orchestra.run.smithi084.stdout:3.5 2024-09-20T22:57:09.051 INFO:teuthology.orchestra.run.smithi084.stdout:4.1c 2024-09-20T22:57:09.051 INFO:teuthology.orchestra.run.smithi084.stdout:6.13 2024-09-20T22:57:09.051 INFO:teuthology.orchestra.run.smithi084.stdout:1.4 2024-09-20T22:57:09.051 INFO:teuthology.orchestra.run.smithi084.stdout:7.7 2024-09-20T22:57:09.051 INFO:teuthology.orchestra.run.smithi084.stdout:2.0 2024-09-20T22:57:09.051 INFO:teuthology.orchestra.run.smithi084.stdout:5.13 2024-09-20T22:57:09.051 INFO:teuthology.orchestra.run.smithi084.stdout:3.c 2024-09-20T22:57:09.051 INFO:teuthology.orchestra.run.smithi084.stdout:4.c 2024-09-20T22:57:09.051 INFO:teuthology.orchestra.run.smithi084.stdout:3.2 2024-09-20T22:57:09.051 INFO:teuthology.orchestra.run.smithi084.stdout:4.17 2024-09-20T22:57:09.051 INFO:teuthology.orchestra.run.smithi084.stdout:5.0 2024-09-20T22:57:09.052 INFO:teuthology.orchestra.run.smithi084.stdout:1.8 2024-09-20T22:57:09.052 INFO:teuthology.orchestra.run.smithi084.stdout:5.14 2024-09-20T22:57:09.052 INFO:teuthology.orchestra.run.smithi084.stdout:3.b 2024-09-20T22:57:09.052 INFO:teuthology.orchestra.run.smithi084.stdout:6.0 2024-09-20T22:57:09.052 INFO:teuthology.orchestra.run.smithi084.stdout:2.1 2024-09-20T22:57:09.052 INFO:teuthology.orchestra.run.smithi084.stdout:3.19 2024-09-20T22:57:09.052 INFO:teuthology.orchestra.run.smithi084.stdout:1.9 2024-09-20T22:57:09.052 INFO:teuthology.orchestra.run.smithi084.stdout:3.8 2024-09-20T22:57:09.052 INFO:teuthology.orchestra.run.smithi084.stdout:7.0 2024-09-20T22:57:09.052 INFO:teuthology.orchestra.run.smithi084.stdout:4.0 2024-09-20T22:57:09.052 INFO:teuthology.orchestra.run.smithi084.stdout:4.15 2024-09-20T22:57:09.052 INFO:teuthology.orchestra.run.smithi084.stdout:3.1 2024-09-20T22:57:09.052 INFO:teuthology.orchestra.run.smithi084.stdout:5.a 2024-09-20T22:57:09.052 INFO:teuthology.orchestra.run.smithi084.stdout:4.14 2024-09-20T22:57:09.052 INFO:teuthology.orchestra.run.smithi084.stdout:3.12 2024-09-20T22:57:09.053 INFO:teuthology.orchestra.run.smithi084.stdout:3.17 2024-09-20T22:57:09.053 INFO:teuthology.orchestra.run.smithi084.stdout:4.19 2024-09-20T22:57:09.053 INFO:teuthology.orchestra.run.smithi084.stdout:4.f 2024-09-20T22:57:09.053 INFO:teuthology.orchestra.run.smithi084.stdout:4.1b 2024-09-20T22:57:09.053 INFO:teuthology.orchestra.run.smithi084.stdout:3.15 2024-09-20T22:57:09.053 INFO:teuthology.orchestra.run.smithi084.stdout:3.18 2024-09-20T22:57:09.053 INFO:teuthology.orchestra.run.smithi084.stdout:6.5 2024-09-20T22:57:09.053 INFO:teuthology.orchestra.run.smithi084.stdout:4.11 2024-09-20T22:57:09.053 INFO:teuthology.orchestra.run.smithi084.stdout:4.1d 2024-09-20T22:57:09.053 INFO:teuthology.orchestra.run.smithi084.stdout:7.11 2024-09-20T22:57:09.053 INFO:teuthology.orchestra.run.smithi084.stdout:4.1e 2024-09-20T22:57:09.053 INFO:teuthology.orchestra.run.smithi084.stdout:3.16 2024-09-20T22:57:09.053 INFO:teuthology.orchestra.run.smithi084.stdout:4.4 2024-09-20T22:57:09.053 INFO:teuthology.orchestra.run.smithi084.stdout:6.2 2024-09-20T22:57:09.053 INFO:teuthology.orchestra.run.smithi084.stdout:6.11 2024-09-20T22:57:09.054 INFO:teuthology.orchestra.run.smithi084.stdout:5.17 2024-09-20T22:57:09.054 INFO:teuthology.orchestra.run.smithi084.stdout:5.1 2024-09-20T22:57:09.054 INFO:teuthology.orchestra.run.smithi084.stdout:5.1c 2024-09-20T22:57:09.054 INFO:teuthology.orchestra.run.smithi084.stdout:6.12 2024-09-20T22:57:09.054 INFO:teuthology.orchestra.run.smithi084.stdout:6.16 2024-09-20T22:57:09.054 INFO:teuthology.orchestra.run.smithi084.stdout:6.1b 2024-09-20T22:57:09.054 INFO:teuthology.orchestra.run.smithi084.stdout:7.8 2024-09-20T22:57:09.054 INFO:teuthology.orchestra.run.smithi084.stdout:6.7 2024-09-20T22:57:09.054 INFO:teuthology.orchestra.run.smithi084.stdout:4.16 2024-09-20T22:57:09.054 INFO:teuthology.orchestra.run.smithi084.stdout:4.1a 2024-09-20T22:57:09.054 INFO:teuthology.orchestra.run.smithi084.stdout:5.12 2024-09-20T22:57:09.054 INFO:teuthology.orchestra.run.smithi084.stdout:6.1c 2024-09-20T22:57:09.054 INFO:teuthology.orchestra.run.smithi084.stdout:6.1f 2024-09-20T22:57:09.054 INFO:teuthology.orchestra.run.smithi084.stdout:3.1e 2024-09-20T22:57:09.054 INFO:teuthology.orchestra.run.smithi084.stdout:6.4 2024-09-20T22:57:09.074 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:57:09.079 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:57:09.079 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:57:09.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:09.096 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:09.096 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:57:09.097 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:57:09.097 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:57:09.112 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:57:09.112 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:57:09.112 INFO:teuthology.orchestra.run.smithi084.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-20T22:57:09.128 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:57:09.128 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:57:09.144 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:57:09.145 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:57:09.161 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:57:09.161 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:57:09.177 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:57:09.178 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:57:09.197 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:57:09.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:09.218 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:09.218 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:57:09.319 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:57:09.341 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:09.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:09.342 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:57:09.373 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is unset 2024-09-20T22:57:09.379 INFO:tasks.rados.rados.0.smithi084.stdout:3835: done (1 left) 2024-09-20T22:57:09.379 INFO:tasks.rados.rados.0.smithi084.stdout:3836: done (0 left) 2024-09-20T22:57:09.379 INFO:tasks.rados.rados.0.smithi084.stdout:3837: copy_from oid 34 from oid 18 current snap is 447 2024-09-20T22:57:09.379 INFO:tasks.rados.rados.0.smithi084.stdout:3838: snap_create 2024-09-20T22:57:09.399 INFO:tasks.rados.rados.0.smithi084.stdout:3837: finishing copy_from racing read to smithi08477203-34 2024-09-20T22:57:09.411 INFO:tasks.rados.rados.0.smithi084.stdout:3837: finishing copy_from to smithi08477203-34 2024-09-20T22:57:09.411 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 34 v 2683 (ObjNum 1268 snap 431 seq_num 1268) dirty exists 2024-09-20T22:57:09.442 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:57:09.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:09.464 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:09.464 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:57:09.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:09 smithi154 ceph-mon[33264]: pgmap v1323: 168 pgs: 2 active+undersized+degraded+remapped+backfill_toofull, 1 active+clean+snaptrim, 41 active+undersized, 36 active+undersized+degraded, 1 active+remapped+backfill_toofull, 87 active+clean; 753 MiB data, 2.3 GiB used, 624 GiB / 626 GiB avail; 285/1620 objects degraded (17.593%); 55/1620 objects misplaced (3.395%) 2024-09-20T22:57:09.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:09 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:57:09.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:09 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:57:09.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:09 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/1904538021' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:57:09.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:09 smithi154 ceph-mon[33264]: 5.1e scrub starts 2024-09-20T22:57:09.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:09 smithi154 ceph-mon[33264]: 5.1e scrub ok 2024-09-20T22:57:09.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:09 smithi154 ceph-mon[33264]: purged_snaps scrub starts 2024-09-20T22:57:09.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:09 smithi154 ceph-mon[33264]: purged_snaps scrub ok 2024-09-20T22:57:09.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:09 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/4206614546' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:57:09.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:09 smithi154 ceph-mon[33264]: 6.1d scrub starts 2024-09-20T22:57:09.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:09 smithi154 ceph-mon[33264]: 6.1d scrub ok 2024-09-20T22:57:09.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:09 smithi154 ceph-mon[33264]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T22:57:09.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:09 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/1904538021' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T22:57:09.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:09 smithi154 ceph-mon[33264]: osdmap e949: 8 total, 7 up, 7 in 2024-09-20T22:57:09.565 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:57:09.586 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:09.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:09.587 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:57:09.688 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:57:09.708 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:09.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:09.709 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:57:09.810 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:57:09.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:09.832 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:09.833 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:57:09.849 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:09 smithi084 ceph-mon[40024]: pgmap v1323: 168 pgs: 2 active+undersized+degraded+remapped+backfill_toofull, 1 active+clean+snaptrim, 41 active+undersized, 36 active+undersized+degraded, 1 active+remapped+backfill_toofull, 87 active+clean; 753 MiB data, 2.3 GiB used, 624 GiB / 626 GiB avail; 285/1620 objects degraded (17.593%); 55/1620 objects misplaced (3.395%) 2024-09-20T22:57:09.849 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:09 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:57:09.849 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:09 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:57:09.850 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:09 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/1904538021' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:57:09.850 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:09 smithi084 ceph-mon[40024]: 5.1e scrub starts 2024-09-20T22:57:09.850 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:09 smithi084 ceph-mon[40024]: 5.1e scrub ok 2024-09-20T22:57:09.850 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:09 smithi084 ceph-mon[40024]: purged_snaps scrub starts 2024-09-20T22:57:09.850 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:09 smithi084 ceph-mon[40024]: purged_snaps scrub ok 2024-09-20T22:57:09.850 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:09 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/4206614546' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:57:09.850 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:09 smithi084 ceph-mon[40024]: 6.1d scrub starts 2024-09-20T22:57:09.850 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:09 smithi084 ceph-mon[40024]: 6.1d scrub ok 2024-09-20T22:57:09.850 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:09 smithi084 ceph-mon[40024]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T22:57:09.850 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:09 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/1904538021' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T22:57:09.850 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:09 smithi084 ceph-mon[40024]: osdmap e949: 8 total, 7 up, 7 in 2024-09-20T22:57:09.850 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:09 smithi084 ceph-mon[29774]: pgmap v1323: 168 pgs: 2 active+undersized+degraded+remapped+backfill_toofull, 1 active+clean+snaptrim, 41 active+undersized, 36 active+undersized+degraded, 1 active+remapped+backfill_toofull, 87 active+clean; 753 MiB data, 2.3 GiB used, 624 GiB / 626 GiB avail; 285/1620 objects degraded (17.593%); 55/1620 objects misplaced (3.395%) 2024-09-20T22:57:09.851 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:09 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:57:09.851 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:09 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:57:09.851 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:09 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/1904538021' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:57:09.851 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:09 smithi084 ceph-mon[29774]: 5.1e scrub starts 2024-09-20T22:57:09.851 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:09 smithi084 ceph-mon[29774]: 5.1e scrub ok 2024-09-20T22:57:09.851 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:09 smithi084 ceph-mon[29774]: purged_snaps scrub starts 2024-09-20T22:57:09.851 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:09 smithi084 ceph-mon[29774]: purged_snaps scrub ok 2024-09-20T22:57:09.851 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:09 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/4206614546' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:57:09.851 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:09 smithi084 ceph-mon[29774]: 6.1d scrub starts 2024-09-20T22:57:09.851 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:09 smithi084 ceph-mon[29774]: 6.1d scrub ok 2024-09-20T22:57:09.851 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:09 smithi084 ceph-mon[29774]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T22:57:09.851 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:09 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/1904538021' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T22:57:09.851 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:09 smithi084 ceph-mon[29774]: osdmap e949: 8 total, 7 up, 7 in 2024-09-20T22:57:09.862 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-20T22:57:09.933 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:57:09.954 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:09.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:09.955 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:57:10.056 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:57:10.077 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:10.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:10.078 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:57:10.178 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:57:10.201 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:10.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:10.202 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:57:10.254 INFO:journalctl@ceph.iscsi.iscsi.a.smithi154.stdout:Sep 20 22:57:09 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-iscsi-iscsi-a[59730]: debug there is no tcmu-runner data available 2024-09-20T22:57:10.303 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:57:10.308 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell -n osd.3 --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-3 --journal-path /var/lib/ceph/osd/ceph-3/journal --op export --pgid 3.7 --file /var/log/ceph/exp.3.7.3 2024-09-20T22:57:10.324 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:10.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:10.325 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:57:10.380 INFO:tasks.rados.rados.0.smithi084.stdout:3837: done (1 left) 2024-09-20T22:57:10.381 INFO:tasks.rados.rados.0.smithi084.stdout:3838: done (0 left) 2024-09-20T22:57:10.381 INFO:tasks.rados.rados.0.smithi084.stdout:3839: copy_from oid 17 from oid 30 current snap is 448 2024-09-20T22:57:10.381 INFO:tasks.rados.rados.0.smithi084.stdout:3840: rollback oid 13 current snap is 448 2024-09-20T22:57:10.381 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 13 to 426 2024-09-20T22:57:10.381 INFO:tasks.rados.rados.0.smithi084.stdout:3841: snap_create 2024-09-20T22:57:10.386 INFO:tasks.rados.rados.0.smithi084.stdout:3839: finishing copy_from to smithi08477203-17 2024-09-20T22:57:10.386 INFO:tasks.rados.rados.0.smithi084.stdout:3839: got expected ENOENT (src dne) 2024-09-20T22:57:10.386 INFO:tasks.rados.rados.0.smithi084.stdout:3839: finishing copy_from racing read to smithi08477203-17 2024-09-20T22:57:10.388 INFO:tasks.rados.rados.0.smithi084.stdout:3840: finishing rollback tid 1 to smithi08477203-13 2024-09-20T22:57:10.388 INFO:tasks.rados.rados.0.smithi084.stdout:3840: finishing rollback tid 2 to smithi08477203-13 2024-09-20T22:57:10.388 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 13 v 3133 (ObjNum 1074 snap 358 seq_num 1074) dirty exists 2024-09-20T22:57:10.425 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:57:10.447 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:10.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:10.447 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:57:10.548 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:57:10.570 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:10.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:10.571 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:57:10.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:10 smithi084 ceph-mon[40024]: 4.6 scrub starts 2024-09-20T22:57:10.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:10 smithi084 ceph-mon[40024]: 4.6 scrub ok 2024-09-20T22:57:10.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:10 smithi084 ceph-mon[40024]: 6.3 scrub starts 2024-09-20T22:57:10.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:10 smithi084 ceph-mon[40024]: 6.3 scrub ok 2024-09-20T22:57:10.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:10 smithi084 ceph-mon[40024]: osdmap e950: 8 total, 7 up, 7 in 2024-09-20T22:57:10.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:10 smithi084 ceph-mon[29774]: 4.6 scrub starts 2024-09-20T22:57:10.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:10 smithi084 ceph-mon[29774]: 4.6 scrub ok 2024-09-20T22:57:10.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:10 smithi084 ceph-mon[29774]: 6.3 scrub starts 2024-09-20T22:57:10.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:10 smithi084 ceph-mon[29774]: 6.3 scrub ok 2024-09-20T22:57:10.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:10 smithi084 ceph-mon[29774]: osdmap e950: 8 total, 7 up, 7 in 2024-09-20T22:57:10.671 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:57:10.693 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:10.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:10.694 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:57:10.738 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-20T22:57:10.738 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-20T22:57:10.746 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/osd.3/config 2024-09-20T22:57:10.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:10 smithi154 ceph-mon[33264]: 4.6 scrub starts 2024-09-20T22:57:10.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:10 smithi154 ceph-mon[33264]: 4.6 scrub ok 2024-09-20T22:57:10.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:10 smithi154 ceph-mon[33264]: 6.3 scrub starts 2024-09-20T22:57:10.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:10 smithi154 ceph-mon[33264]: 6.3 scrub ok 2024-09-20T22:57:10.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:10 smithi154 ceph-mon[33264]: osdmap e950: 8 total, 7 up, 7 in 2024-09-20T22:57:10.795 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:57:10.816 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:10.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:10.817 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:57:10.918 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:57:10.939 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:10.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:10.940 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:57:11.041 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:57:11.062 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:11.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:11.063 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:57:11.110 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:57:11.164 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:57:11.186 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:11.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:11.187 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:57:11.288 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:57:11.309 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:11.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:11.310 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:57:11.381 INFO:tasks.rados.rados.0.smithi084.stdout:3839: done (2 left) 2024-09-20T22:57:11.382 INFO:tasks.rados.rados.0.smithi084.stdout:3840: done (1 left) 2024-09-20T22:57:11.382 INFO:tasks.rados.rados.0.smithi084.stdout:3841: done (0 left) 2024-09-20T22:57:11.382 INFO:tasks.rados.rados.0.smithi084.stdout:3842: rollback oid 38 current snap is 449 2024-09-20T22:57:11.382 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 38 to 397 2024-09-20T22:57:11.382 INFO:tasks.rados.rados.0.smithi084.stdout:3843: write (excl) oid 20 current snap is 449 2024-09-20T22:57:11.382 INFO:tasks.rados.rados.0.smithi084.stdout:3843: seq_num 1307 ranges {766749=646806,1899377=656453,3109028=609462} 2024-09-20T22:57:11.389 INFO:tasks.rados.rados.0.smithi084.stdout:3843: writing smithi08477203-20 from 766749 to 1413555 tid 1 2024-09-20T22:57:11.396 INFO:tasks.rados.rados.0.smithi084.stdout:3843: writing smithi08477203-20 from 1899377 to 2555830 tid 2 2024-09-20T22:57:11.402 INFO:tasks.rados.rados.0.smithi084.stdout:3843: writing smithi08477203-20 from 3109028 to 3718490 tid 3 2024-09-20T22:57:11.402 INFO:tasks.rados.rados.0.smithi084.stdout:3844: snap_create 2024-09-20T22:57:11.402 INFO:tasks.rados.rados.0.smithi084.stdout:3842: finishing rollback tid 0 to smithi08477203-38 2024-09-20T22:57:11.402 INFO:tasks.rados.rados.0.smithi084.stdout:3842: finishing rollback tid 1 to smithi08477203-38 2024-09-20T22:57:11.402 INFO:tasks.rados.rados.0.smithi084.stdout:3842: finishing rollback tid 2 to smithi08477203-38 2024-09-20T22:57:11.403 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 38 v 2857 (ObjNum 1118 snap 380 seq_num 1118) dirty exists 2024-09-20T22:57:11.403 INFO:tasks.rados.rados.0.smithi084.stdout:3843: finishing write tid 1 to smithi08477203-20 2024-09-20T22:57:11.403 INFO:tasks.rados.rados.0.smithi084.stdout:3843: finishing write tid 2 to smithi08477203-20 2024-09-20T22:57:11.405 INFO:tasks.rados.rados.0.smithi084.stdout:3843: finishing write tid 3 to smithi08477203-20 2024-09-20T22:57:11.405 INFO:tasks.rados.rados.0.smithi084.stdout:3843: finishing write tid 4 to smithi08477203-20 2024-09-20T22:57:11.405 INFO:tasks.rados.rados.0.smithi084.stdout:3843: finishing write tid 5 to smithi08477203-20 2024-09-20T22:57:11.405 INFO:tasks.rados.rados.0.smithi084.stdout:3843: oid 20 updating version 0 to 2649 2024-09-20T22:57:11.406 INFO:tasks.rados.rados.0.smithi084.stdout:3843: oid 20 version 2649 is already newer than 2647 2024-09-20T22:57:11.406 INFO:tasks.rados.rados.0.smithi084.stdout:3843: oid 20 updating version 2649 to 2650 2024-09-20T22:57:11.406 INFO:tasks.rados.rados.0.smithi084.stdout:3843: oid 20 version 2650 is already newer than 2648 2024-09-20T22:57:11.406 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 20 v 2650 (ObjNum 1307 snap 449 seq_num 1307) dirty exists 2024-09-20T22:57:11.406 INFO:tasks.rados.rados.0.smithi084.stdout:3843: left oid 20 (ObjNum 1307 snap 449 seq_num 1307) 2024-09-20T22:57:11.411 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:57:11.432 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:11.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:11.433 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:57:11.534 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:57:11.555 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:11.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:11.556 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:57:11.657 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:57:11.679 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:11.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:11.680 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:57:11.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:11 smithi154 ceph-mon[33264]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:57:11.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:11 smithi154 ceph-mon[33264]: pgmap v1326: 168 pgs: 1 active+clean+snaptrim, 2 active+undersized+degraded+remapped+backfill_toofull, 41 active+undersized, 1 active+remapped+backfill_toofull, 36 active+undersized+degraded, 87 active+clean; 797 MiB data, 2.3 GiB used, 624 GiB / 626 GiB avail; 287/1674 objects degraded (17.145%); 55/1674 objects misplaced (3.286%) 2024-09-20T22:57:11.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:11 smithi154 ceph-mon[33264]: 5.8 scrub starts 2024-09-20T22:57:11.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:11 smithi154 ceph-mon[33264]: 5.8 scrub ok 2024-09-20T22:57:11.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:11 smithi154 ceph-mon[33264]: Health check update: Degraded data redundancy: 287/1674 objects degraded (17.145%), 38 pgs degraded (PG_DEGRADED) 2024-09-20T22:57:11.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:11 smithi154 ceph-mon[33264]: 4.8 scrub starts 2024-09-20T22:57:11.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:11 smithi154 ceph-mon[33264]: 4.8 scrub ok 2024-09-20T22:57:11.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:11 smithi154 ceph-mon[33264]: 3.e scrub starts 2024-09-20T22:57:11.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:11 smithi154 ceph-mon[33264]: 3.e scrub ok 2024-09-20T22:57:11.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:11 smithi154 ceph-mon[33264]: osdmap e951: 8 total, 7 up, 7 in 2024-09-20T22:57:11.781 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:57:11.803 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:11.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:11.804 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:57:11.845 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:11 smithi084 ceph-mon[29774]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:57:11.845 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:11 smithi084 ceph-mon[29774]: pgmap v1326: 168 pgs: 1 active+clean+snaptrim, 2 active+undersized+degraded+remapped+backfill_toofull, 41 active+undersized, 1 active+remapped+backfill_toofull, 36 active+undersized+degraded, 87 active+clean; 797 MiB data, 2.3 GiB used, 624 GiB / 626 GiB avail; 287/1674 objects degraded (17.145%); 55/1674 objects misplaced (3.286%) 2024-09-20T22:57:11.845 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:11 smithi084 ceph-mon[29774]: 5.8 scrub starts 2024-09-20T22:57:11.845 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:11 smithi084 ceph-mon[29774]: 5.8 scrub ok 2024-09-20T22:57:11.845 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:11 smithi084 ceph-mon[29774]: Health check update: Degraded data redundancy: 287/1674 objects degraded (17.145%), 38 pgs degraded (PG_DEGRADED) 2024-09-20T22:57:11.845 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:11 smithi084 ceph-mon[29774]: 4.8 scrub starts 2024-09-20T22:57:11.845 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:11 smithi084 ceph-mon[29774]: 4.8 scrub ok 2024-09-20T22:57:11.845 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:11 smithi084 ceph-mon[29774]: 3.e scrub starts 2024-09-20T22:57:11.845 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:11 smithi084 ceph-mon[29774]: 3.e scrub ok 2024-09-20T22:57:11.845 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:11 smithi084 ceph-mon[29774]: osdmap e951: 8 total, 7 up, 7 in 2024-09-20T22:57:11.846 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:11 smithi084 ceph-mon[40024]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:57:11.846 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:11 smithi084 ceph-mon[40024]: pgmap v1326: 168 pgs: 1 active+clean+snaptrim, 2 active+undersized+degraded+remapped+backfill_toofull, 41 active+undersized, 1 active+remapped+backfill_toofull, 36 active+undersized+degraded, 87 active+clean; 797 MiB data, 2.3 GiB used, 624 GiB / 626 GiB avail; 287/1674 objects degraded (17.145%); 55/1674 objects misplaced (3.286%) 2024-09-20T22:57:11.846 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:11 smithi084 ceph-mon[40024]: 5.8 scrub starts 2024-09-20T22:57:11.846 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:11 smithi084 ceph-mon[40024]: 5.8 scrub ok 2024-09-20T22:57:11.846 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:11 smithi084 ceph-mon[40024]: Health check update: Degraded data redundancy: 287/1674 objects degraded (17.145%), 38 pgs degraded (PG_DEGRADED) 2024-09-20T22:57:11.846 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:11 smithi084 ceph-mon[40024]: 4.8 scrub starts 2024-09-20T22:57:11.846 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:11 smithi084 ceph-mon[40024]: 4.8 scrub ok 2024-09-20T22:57:11.846 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:11 smithi084 ceph-mon[40024]: 3.e scrub starts 2024-09-20T22:57:11.846 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:11 smithi084 ceph-mon[40024]: 3.e scrub ok 2024-09-20T22:57:11.846 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:11 smithi084 ceph-mon[40024]: osdmap e951: 8 total, 7 up, 7 in 2024-09-20T22:57:11.905 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:57:11.926 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:11.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:11.927 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:57:12.027 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:57:12.049 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:12.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:12.050 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:57:12.151 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:57:12.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:12.173 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:12.173 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:57:12.274 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:57:12.296 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:12.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:12.297 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:57:12.388 INFO:tasks.rados.rados.0.smithi084.stdout:3842: done (2 left) 2024-09-20T22:57:12.388 INFO:tasks.rados.rados.0.smithi084.stdout:3843: done (1 left) 2024-09-20T22:57:12.388 INFO:tasks.rados.rados.0.smithi084.stdout:3844: done (0 left) 2024-09-20T22:57:12.388 INFO:tasks.rados.rados.0.smithi084.stdout:3845: read oid 49 snap -1 2024-09-20T22:57:12.388 INFO:tasks.rados.rados.0.smithi084.stdout:3845: expect (ObjNum 1288 snap 436 seq_num 1288) 2024-09-20T22:57:12.388 INFO:tasks.rados.rados.0.smithi084.stdout:3846: write (excl) oid 15 current snap is 450 2024-09-20T22:57:12.388 INFO:tasks.rados.rados.0.smithi084.stdout:3846: seq_num 1308 ranges {783556=438408,1674212=400111,2736451=420734,3882270=101286} 2024-09-20T22:57:12.394 INFO:tasks.rados.rados.0.smithi084.stdout:3846: writing smithi08477203-15 from 783556 to 1221964 tid 1 2024-09-20T22:57:12.397 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:57:12.399 INFO:tasks.rados.rados.0.smithi084.stdout:3846: writing smithi08477203-15 from 1674212 to 2074323 tid 2 2024-09-20T22:57:12.405 INFO:tasks.rados.rados.0.smithi084.stdout:3846: writing smithi08477203-15 from 2736451 to 3157185 tid 3 2024-09-20T22:57:12.410 INFO:tasks.rados.rados.0.smithi084.stdout:3846: writing smithi08477203-15 from 3882270 to 3983556 tid 4 2024-09-20T22:57:12.410 INFO:tasks.rados.rados.0.smithi084.stdout:3847: rollback oid 20 current snap is 450 2024-09-20T22:57:12.410 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 20 to 332 2024-09-20T22:57:12.411 INFO:tasks.rados.rados.0.smithi084.stdout:3845: expect (ObjNum 1288 snap 436 seq_num 1288) 2024-09-20T22:57:12.420 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:12.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:12.421 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:57:12.522 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:57:12.543 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:12.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:12.544 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:57:12.549 INFO:teuthology.orchestra.run.smithi084.stderr:Creating an OSD daemon form without an OSD FSID value 2024-09-20T22:57:12.574 INFO:tasks.rados.rados.0.smithi084.stdout:3846: finishing write tid 1 to smithi08477203-15 2024-09-20T22:57:12.574 INFO:tasks.rados.rados.0.smithi084.stdout:3846: finishing write tid 2 to smithi08477203-15 2024-09-20T22:57:12.575 INFO:tasks.rados.rados.0.smithi084.stdout:3846: finishing write tid 3 to smithi08477203-15 2024-09-20T22:57:12.575 INFO:tasks.rados.rados.0.smithi084.stdout:3846: finishing write tid 4 to smithi08477203-15 2024-09-20T22:57:12.575 INFO:tasks.rados.rados.0.smithi084.stdout:3846: finishing write tid 5 to smithi08477203-15 2024-09-20T22:57:12.575 INFO:tasks.rados.rados.0.smithi084.stdout:3847: finishing rollback tid 0 to smithi08477203-20 2024-09-20T22:57:12.575 INFO:tasks.rados.rados.0.smithi084.stdout:3847: finishing rollback tid 1 to smithi08477203-20 2024-09-20T22:57:12.575 INFO:tasks.rados.rados.0.smithi084.stdout:3845: done (2 left) 2024-09-20T22:57:12.575 INFO:tasks.rados.rados.0.smithi084.stdout:3848: delete oid 47 current snap is 450 2024-09-20T22:57:12.575 INFO:tasks.rados.rados.0.smithi084.stdout:3846: finishing write tid 6 to smithi08477203-15 2024-09-20T22:57:12.575 INFO:tasks.rados.rados.0.smithi084.stdout:3846: oid 15 updating version 0 to 2633 2024-09-20T22:57:12.575 INFO:tasks.rados.rados.0.smithi084.stdout:3846: oid 15 updating version 2633 to 2634 2024-09-20T22:57:12.575 INFO:tasks.rados.rados.0.smithi084.stdout:3846: oid 15 updating version 2634 to 2636 2024-09-20T22:57:12.575 INFO:tasks.rados.rados.0.smithi084.stdout:3846: oid 15 version 2636 is already newer than 2635 2024-09-20T22:57:12.575 INFO:tasks.rados.rados.0.smithi084.stdout:3846: oid 15 updating version 2636 to 2637 2024-09-20T22:57:12.576 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 15 v 2637 (ObjNum 1308 snap 450 seq_num 1308) dirty exists 2024-09-20T22:57:12.576 INFO:tasks.rados.rados.0.smithi084.stdout:3846: left oid 15 (ObjNum 1308 snap 450 seq_num 1308) 2024-09-20T22:57:12.576 INFO:tasks.rados.rados.0.smithi084.stdout:3847: finishing rollback tid 2 to smithi08477203-20 2024-09-20T22:57:12.576 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 20 v 2653 (ObjNum 964 snap 314 seq_num 964) dirty exists 2024-09-20T22:57:12.576 INFO:tasks.rados.rados.0.smithi084.stdout:3846: done (2 left) 2024-09-20T22:57:12.576 INFO:tasks.rados.rados.0.smithi084.stdout:3847: done (1 left) 2024-09-20T22:57:12.576 INFO:tasks.rados.rados.0.smithi084.stdout:3848: done (0 left) 2024-09-20T22:57:12.576 INFO:tasks.rados.rados.0.smithi084.stdout:3849: snap_create 2024-09-20T22:57:12.645 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:57:12.666 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:12.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:12.667 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:57:12.734 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-20T22:57:12.735 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:57:12.735 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:12 smithi084 ceph-mon[29774]: 5.f scrub starts 2024-09-20T22:57:12.735 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:12 smithi084 ceph-mon[29774]: 5.f scrub ok 2024-09-20T22:57:12.735 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:12 smithi084 ceph-mon[29774]: 6.1e scrub starts 2024-09-20T22:57:12.735 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:12 smithi084 ceph-mon[29774]: 6.1e scrub ok 2024-09-20T22:57:12.735 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:12 smithi084 ceph-mon[29774]: osdmap e952: 8 total, 7 up, 7 in 2024-09-20T22:57:12.735 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:12 smithi084 ceph-mon[40024]: 5.f scrub starts 2024-09-20T22:57:12.736 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:12 smithi084 ceph-mon[40024]: 5.f scrub ok 2024-09-20T22:57:12.736 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:12 smithi084 ceph-mon[40024]: 6.1e scrub starts 2024-09-20T22:57:12.736 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:12 smithi084 ceph-mon[40024]: 6.1e scrub ok 2024-09-20T22:57:12.736 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:12 smithi084 ceph-mon[40024]: osdmap e952: 8 total, 7 up, 7 in 2024-09-20T22:57:12.739 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-20T22:57:12.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:12 smithi154 ceph-mon[33264]: 5.f scrub starts 2024-09-20T22:57:12.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:12 smithi154 ceph-mon[33264]: 5.f scrub ok 2024-09-20T22:57:12.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:12 smithi154 ceph-mon[33264]: 6.1e scrub starts 2024-09-20T22:57:12.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:12 smithi154 ceph-mon[33264]: 6.1e scrub ok 2024-09-20T22:57:12.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:12 smithi154 ceph-mon[33264]: osdmap e952: 8 total, 7 up, 7 in 2024-09-20T22:57:12.768 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:57:12.802 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:12.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:12.803 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:57:12.904 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:57:12.927 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:12.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:12.928 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:57:13.029 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:57:13.050 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:13.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:13.051 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:57:13.102 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd set noscrub 2024-09-20T22:57:13.152 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:57:13.173 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:13.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:13.174 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:57:13.275 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:57:13.300 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:13.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:13.301 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:57:13.402 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:57:13.425 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:13.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:13.426 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:57:13.446 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.0 dump_blocked_ops 2024-09-20T22:57:13.492 INFO:tasks.rados.rados.0.smithi084.stdout:3849: done (0 left) 2024-09-20T22:57:13.492 INFO:tasks.rados.rados.0.smithi084.stdout:3850: read oid 5 snap -1 2024-09-20T22:57:13.492 INFO:tasks.rados.rados.0.smithi084.stdout:3850: expect (ObjNum 1241 snap 421 seq_num 1241) 2024-09-20T22:57:13.492 INFO:tasks.rados.rados.0.smithi084.stdout:3851: copy_from oid 42 from oid 10 current snap is 451 2024-09-20T22:57:13.492 INFO:tasks.rados.rados.0.smithi084.stdout:3852: snap_create 2024-09-20T22:57:13.495 INFO:tasks.rados.rados.0.smithi084.stdout:3851: finishing copy_from to smithi08477203-42 2024-09-20T22:57:13.495 INFO:tasks.rados.rados.0.smithi084.stdout:3851: got expected ENOENT (src dne) 2024-09-20T22:57:13.495 INFO:tasks.rados.rados.0.smithi084.stdout:3851: finishing copy_from racing read to smithi08477203-42 2024-09-20T22:57:13.496 INFO:tasks.rados.rados.0.smithi084.stdout:3850: expect (ObjNum 1241 snap 421 seq_num 1241) 2024-09-20T22:57:13.527 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:57:13.552 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:13.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:13.553 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:57:13.654 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:57:13.677 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:13 smithi084 ceph-mon[40024]: pgmap v1329: 168 pgs: 1 active+clean+snaptrim, 2 active+undersized+degraded+remapped+backfill_toofull, 41 active+undersized, 1 active+remapped+backfill_toofull, 36 active+undersized+degraded, 87 active+clean; 797 MiB data, 2.3 GiB used, 624 GiB / 626 GiB avail; 287/1674 objects degraded (17.145%); 55/1674 objects misplaced (3.286%) 2024-09-20T22:57:13.677 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:13 smithi084 ceph-mon[40024]: 3.1b scrub starts 2024-09-20T22:57:13.677 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:13 smithi084 ceph-mon[40024]: 3.1b scrub ok 2024-09-20T22:57:13.677 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:13 smithi084 ceph-mon[40024]: 5.18 scrub starts 2024-09-20T22:57:13.677 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:13 smithi084 ceph-mon[40024]: 5.18 scrub ok 2024-09-20T22:57:13.678 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:13 smithi084 ceph-mon[29774]: pgmap v1329: 168 pgs: 1 active+clean+snaptrim, 2 active+undersized+degraded+remapped+backfill_toofull, 41 active+undersized, 1 active+remapped+backfill_toofull, 36 active+undersized+degraded, 87 active+clean; 797 MiB data, 2.3 GiB used, 624 GiB / 626 GiB avail; 287/1674 objects degraded (17.145%); 55/1674 objects misplaced (3.286%) 2024-09-20T22:57:13.678 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:13 smithi084 ceph-mon[29774]: 3.1b scrub starts 2024-09-20T22:57:13.678 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:13 smithi084 ceph-mon[29774]: 3.1b scrub ok 2024-09-20T22:57:13.678 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:13 smithi084 ceph-mon[29774]: 5.18 scrub starts 2024-09-20T22:57:13.678 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:13 smithi084 ceph-mon[29774]: 5.18 scrub ok 2024-09-20T22:57:13.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:13.679 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:13.679 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:57:13.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:13 smithi154 ceph-mon[33264]: pgmap v1329: 168 pgs: 1 active+clean+snaptrim, 2 active+undersized+degraded+remapped+backfill_toofull, 41 active+undersized, 1 active+remapped+backfill_toofull, 36 active+undersized+degraded, 87 active+clean; 797 MiB data, 2.3 GiB used, 624 GiB / 626 GiB avail; 287/1674 objects degraded (17.145%); 55/1674 objects misplaced (3.286%) 2024-09-20T22:57:13.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:13 smithi154 ceph-mon[33264]: 3.1b scrub starts 2024-09-20T22:57:13.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:13 smithi154 ceph-mon[33264]: 3.1b scrub ok 2024-09-20T22:57:13.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:13 smithi154 ceph-mon[33264]: 5.18 scrub starts 2024-09-20T22:57:13.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:13 smithi154 ceph-mon[33264]: 5.18 scrub ok 2024-09-20T22:57:13.780 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:57:13.802 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:13.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:13.803 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:57:13.899 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:57:13.903 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:57:13.903 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:57:13.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:13.927 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:13.927 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:57:14.028 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:57:14.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:14.055 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:14.055 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:57:14.156 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:57:14.181 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:14.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:14.182 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:57:14.283 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:57:14.305 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:14.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:14.306 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:57:14.406 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:57:14.432 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:14.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:14.433 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:57:14.499 INFO:tasks.rados.rados.0.smithi084.stdout:3850: done (2 left) 2024-09-20T22:57:14.500 INFO:tasks.rados.rados.0.smithi084.stdout:3851: done (1 left) 2024-09-20T22:57:14.500 INFO:tasks.rados.rados.0.smithi084.stdout:3852: done (0 left) 2024-09-20T22:57:14.500 INFO:tasks.rados.rados.0.smithi084.stdout:3853: rollback oid 8 current snap is 452 2024-09-20T22:57:14.500 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 8 to 416 2024-09-20T22:57:14.500 INFO:tasks.rados.rados.0.smithi084.stdout:3854: read oid 50 snap -1 2024-09-20T22:57:14.500 INFO:tasks.rados.rados.0.smithi084.stdout:3854: expect deleted 2024-09-20T22:57:14.500 INFO:tasks.rados.rados.0.smithi084.stdout:3855: read oid 12 snap -1 2024-09-20T22:57:14.500 INFO:tasks.rados.rados.0.smithi084.stdout:3855: expect (ObjNum 1202 snap 411 seq_num 1202) 2024-09-20T22:57:14.500 INFO:tasks.rados.rados.0.smithi084.stdout:3856: write (excl) oid 45 current snap is 452 2024-09-20T22:57:14.500 INFO:tasks.rados.rados.0.smithi084.stdout:3856: seq_num 1310 ranges {417170=588906,1674823=342347} 2024-09-20T22:57:14.505 INFO:tasks.rados.rados.0.smithi084.stdout:3856: writing smithi08477203-45 from 417170 to 1006076 tid 1 2024-09-20T22:57:14.512 INFO:tasks.rados.rados.0.smithi084.stdout:3856: writing smithi08477203-45 from 1674823 to 2017170 tid 2 2024-09-20T22:57:14.512 INFO:tasks.rados.rados.0.smithi084.stdout:3857: read oid 23 snap -1 2024-09-20T22:57:14.512 INFO:tasks.rados.rados.0.smithi084.stdout:3857: expect deleted 2024-09-20T22:57:14.512 INFO:tasks.rados.rados.0.smithi084.stdout:3858: read oid 29 snap 410 2024-09-20T22:57:14.512 INFO:tasks.rados.rados.0.smithi084.stdout:3858: expect (ObjNum 1169 snap 403 seq_num 1169) 2024-09-20T22:57:14.512 INFO:tasks.rados.rados.0.smithi084.stdout:3859: snap_create 2024-09-20T22:57:14.512 INFO:tasks.rados.rados.0.smithi084.stdout:3853: finishing rollback tid 0 to smithi08477203-8 2024-09-20T22:57:14.512 INFO:tasks.rados.rados.0.smithi084.stdout:3853: finishing rollback tid 1 to smithi08477203-8 2024-09-20T22:57:14.512 INFO:tasks.rados.rados.0.smithi084.stdout:3855: expect (ObjNum 1202 snap 411 seq_num 1202) 2024-09-20T22:57:14.533 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:57:14.557 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:14.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:14.558 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:57:14.585 INFO:tasks.rados.rados.0.smithi084.stdout:3853: finishing rollback tid 2 to smithi08477203-8 2024-09-20T22:57:14.585 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 8 v 2634 (ObjNum 1205 snap 412 seq_num 1205) dirty exists 2024-09-20T22:57:14.585 INFO:tasks.rados.rados.0.smithi084.stdout:3856: finishing write tid 1 to smithi08477203-45 2024-09-20T22:57:14.585 INFO:tasks.rados.rados.0.smithi084.stdout:3856: finishing write tid 2 to smithi08477203-45 2024-09-20T22:57:14.586 INFO:tasks.rados.rados.0.smithi084.stdout:3856: finishing write tid 3 to smithi08477203-45 2024-09-20T22:57:14.586 INFO:tasks.rados.rados.0.smithi084.stdout:3856: finishing write tid 4 to smithi08477203-45 2024-09-20T22:57:14.586 INFO:tasks.rados.rados.0.smithi084.stdout:3856: oid 45 updating version 0 to 1958 2024-09-20T22:57:14.586 INFO:tasks.rados.rados.0.smithi084.stdout:3856: oid 45 version 1958 is already newer than 1957 2024-09-20T22:57:14.586 INFO:tasks.rados.rados.0.smithi084.stdout:3856: oid 45 updating version 1958 to 1959 2024-09-20T22:57:14.586 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 45 v 1959 (ObjNum 1310 snap 452 seq_num 1310) dirty exists 2024-09-20T22:57:14.586 INFO:tasks.rados.rados.0.smithi084.stdout:3856: left oid 45 (ObjNum 1310 snap 452 seq_num 1310) 2024-09-20T22:57:14.586 INFO:tasks.rados.rados.0.smithi084.stdout:3858: expect (ObjNum 1169 snap 403 seq_num 1169) 2024-09-20T22:57:14.601 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:14 smithi084 ceph-mon[29774]: osdmap e953: 8 total, 7 up, 7 in 2024-09-20T22:57:14.601 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:14 smithi084 ceph-mon[29774]: 5.7 deep-scrub starts 2024-09-20T22:57:14.601 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:14 smithi084 ceph-mon[29774]: 5.7 deep-scrub ok 2024-09-20T22:57:14.602 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:14 smithi084 ceph-mon[40024]: osdmap e953: 8 total, 7 up, 7 in 2024-09-20T22:57:14.602 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:14 smithi084 ceph-mon[40024]: 5.7 deep-scrub starts 2024-09-20T22:57:14.602 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:14 smithi084 ceph-mon[40024]: 5.7 deep-scrub ok 2024-09-20T22:57:14.655 INFO:teuthology.orchestra.run.smithi084.stderr:Exporting 3.7 info 3.7( empty local-lis/les=887/888 n=0 ec=66/66 lis/c=887/887 les/c/f=888/888/0 sis=887) 2024-09-20T22:57:14.656 INFO:teuthology.orchestra.run.smithi084.stderr:Export successful 2024-09-20T22:57:14.659 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:57:14.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:14.681 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:14.681 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:57:14.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:14 smithi154 ceph-mon[33264]: osdmap e953: 8 total, 7 up, 7 in 2024-09-20T22:57:14.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:14 smithi154 ceph-mon[33264]: 5.7 deep-scrub starts 2024-09-20T22:57:14.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:14 smithi154 ceph-mon[33264]: 5.7 deep-scrub ok 2024-09-20T22:57:14.782 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:57:14.803 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:14.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:14.804 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:57:14.905 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:57:14.931 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:14.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:14.932 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:57:15.033 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:57:15.058 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:15.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:15.059 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:57:15.077 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-20T22:57:15.077 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:57:15.159 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:57:15.185 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:15.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:15.186 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:57:15.286 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:57:15.307 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:15.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:15.308 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:57:15.409 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:57:15.435 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:15.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:15.436 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:57:15.502 INFO:tasks.rados.rados.0.smithi084.stdout:3853: done (6 left) 2024-09-20T22:57:15.502 INFO:tasks.rados.rados.0.smithi084.stdout:3854: done (5 left) 2024-09-20T22:57:15.502 INFO:tasks.rados.rados.0.smithi084.stdout:3855: done (4 left) 2024-09-20T22:57:15.502 INFO:tasks.rados.rados.0.smithi084.stdout:3856: done (3 left) 2024-09-20T22:57:15.502 INFO:tasks.rados.rados.0.smithi084.stdout:3857: done (2 left) 2024-09-20T22:57:15.503 INFO:tasks.rados.rados.0.smithi084.stdout:3858: done (1 left) 2024-09-20T22:57:15.503 INFO:tasks.rados.rados.0.smithi084.stdout:3859: done (0 left) 2024-09-20T22:57:15.503 INFO:tasks.rados.rados.0.smithi084.stdout:3860: write oid 18 current snap is 453 2024-09-20T22:57:15.503 INFO:tasks.rados.rados.0.smithi084.stdout:3860: seq_num 1311 ranges {433977=664155,1733305=300672} 2024-09-20T22:57:15.506 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is set 2024-09-20T22:57:15.509 INFO:tasks.rados.rados.0.smithi084.stdout:3860: writing smithi08477203-18 from 433977 to 1098132 tid 1 2024-09-20T22:57:15.515 INFO:tasks.rados.rados.0.smithi084.stdout:3860: writing smithi08477203-18 from 1733305 to 2033977 tid 2 2024-09-20T22:57:15.515 INFO:tasks.rados.rados.0.smithi084.stdout:3861: write oid 41 current snap is 453 2024-09-20T22:57:15.515 INFO:tasks.rados.rados.0.smithi084.stdout:3861: seq_num 1312 ranges {450784=739404,1791787=258997} 2024-09-20T22:57:15.521 INFO:tasks.rados.rados.0.smithi084.stdout:3861: writing smithi08477203-41 from 450784 to 1190188 tid 1 2024-09-20T22:57:15.527 INFO:tasks.rados.rados.0.smithi084.stdout:3861: writing smithi08477203-41 from 1791787 to 2050784 tid 2 2024-09-20T22:57:15.527 INFO:tasks.rados.rados.0.smithi084.stdout:3862: read oid 7 snap -1 2024-09-20T22:57:15.528 INFO:tasks.rados.rados.0.smithi084.stdout:3862: expect (ObjNum 1273 snap 431 seq_num 1273) 2024-09-20T22:57:15.528 INFO:tasks.rados.rados.0.smithi084.stdout:3863: write oid 27 current snap is 453 2024-09-20T22:57:15.528 INFO:tasks.rados.rados.0.smithi084.stdout:3863: seq_num 1313 ranges {467591=414653,1450269=617322} 2024-09-20T22:57:15.533 INFO:tasks.rados.rados.0.smithi084.stdout:3863: writing smithi08477203-27 from 467591 to 882244 tid 1 2024-09-20T22:57:15.536 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:57:15.540 INFO:tasks.rados.rados.0.smithi084.stdout:3863: writing smithi08477203-27 from 1450269 to 2067591 tid 2 2024-09-20T22:57:15.540 INFO:tasks.rados.rados.0.smithi084.stdout:3860: finishing write tid 1 to smithi08477203-18 2024-09-20T22:57:15.541 INFO:tasks.rados.rados.0.smithi084.stdout:3860: finishing write tid 2 to smithi08477203-18 2024-09-20T22:57:15.541 INFO:tasks.rados.rados.0.smithi084.stdout:3860: finishing write tid 3 to smithi08477203-18 2024-09-20T22:57:15.541 INFO:tasks.rados.rados.0.smithi084.stdout:3860: finishing write tid 4 to smithi08477203-18 2024-09-20T22:57:15.541 INFO:tasks.rados.rados.0.smithi084.stdout:3860: oid 18 updating version 0 to 2690 2024-09-20T22:57:15.541 INFO:tasks.rados.rados.0.smithi084.stdout:3860: oid 18 updating version 2690 to 2691 2024-09-20T22:57:15.541 INFO:tasks.rados.rados.0.smithi084.stdout:3860: oid 18 version 2691 is already newer than 2689 2024-09-20T22:57:15.541 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 18 v 2691 (ObjNum 1311 snap 453 seq_num 1311) dirty exists 2024-09-20T22:57:15.541 INFO:tasks.rados.rados.0.smithi084.stdout:3860: left oid 18 (ObjNum 1311 snap 453 seq_num 1311) 2024-09-20T22:57:15.541 INFO:tasks.rados.rados.0.smithi084.stdout:3861: finishing write tid 1 to smithi08477203-41 2024-09-20T22:57:15.541 INFO:tasks.rados.rados.0.smithi084.stdout:3861: finishing write tid 2 to smithi08477203-41 2024-09-20T22:57:15.541 INFO:tasks.rados.rados.0.smithi084.stdout:3861: finishing write tid 3 to smithi08477203-41 2024-09-20T22:57:15.541 INFO:tasks.rados.rados.0.smithi084.stdout:3861: finishing write tid 4 to smithi08477203-41 2024-09-20T22:57:15.541 INFO:tasks.rados.rados.0.smithi084.stdout:3861: oid 41 updating version 0 to 2671 2024-09-20T22:57:15.542 INFO:tasks.rados.rados.0.smithi084.stdout:3861: oid 41 version 2671 is already newer than 2670 2024-09-20T22:57:15.542 INFO:tasks.rados.rados.0.smithi084.stdout:3861: oid 41 version 2671 is already newer than 2669 2024-09-20T22:57:15.542 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 41 v 2671 (ObjNum 1312 snap 453 seq_num 1312) dirty exists 2024-09-20T22:57:15.542 INFO:tasks.rados.rados.0.smithi084.stdout:3861: left oid 41 (ObjNum 1312 snap 453 seq_num 1312) 2024-09-20T22:57:15.542 INFO:tasks.rados.rados.0.smithi084.stdout:3862: expect (ObjNum 1273 snap 431 seq_num 1273) 2024-09-20T22:57:15.560 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:15.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:15.561 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:57:15.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:15 smithi084 ceph-mon[29774]: pgmap v1332: 168 pgs: 1 active+clean+snaptrim, 2 active+undersized+degraded+remapped+backfill_toofull, 41 active+undersized, 1 active+remapped+backfill_toofull, 36 active+undersized+degraded, 87 active+clean; 810 MiB data, 2.3 GiB used, 624 GiB / 626 GiB avail; 8.3 MiB/s rd, 1.6 MiB/s wr, 11 op/s; 290/1683 objects degraded (17.231%); 55/1683 objects misplaced (3.268%) 2024-09-20T22:57:15.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:15 smithi084 ceph-mon[29774]: 1.5 deep-scrub starts 2024-09-20T22:57:15.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:15 smithi084 ceph-mon[29774]: 1.5 deep-scrub ok 2024-09-20T22:57:15.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:15 smithi084 ceph-mon[29774]: osdmap e954: 8 total, 7 up, 7 in 2024-09-20T22:57:15.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:15 smithi084 ceph-mon[29774]: 6.a scrub starts 2024-09-20T22:57:15.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:15 smithi084 ceph-mon[29774]: 6.a scrub ok 2024-09-20T22:57:15.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:15 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/367011767' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-20T22:57:15.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:15 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-20T22:57:15.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:15 smithi084 ceph-mon[29774]: 3.14 scrub starts 2024-09-20T22:57:15.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:15 smithi084 ceph-mon[29774]: 3.14 scrub ok 2024-09-20T22:57:15.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:15 smithi084 ceph-mon[40024]: pgmap v1332: 168 pgs: 1 active+clean+snaptrim, 2 active+undersized+degraded+remapped+backfill_toofull, 41 active+undersized, 1 active+remapped+backfill_toofull, 36 active+undersized+degraded, 87 active+clean; 810 MiB data, 2.3 GiB used, 624 GiB / 626 GiB avail; 8.3 MiB/s rd, 1.6 MiB/s wr, 11 op/s; 290/1683 objects degraded (17.231%); 55/1683 objects misplaced (3.268%) 2024-09-20T22:57:15.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:15 smithi084 ceph-mon[40024]: 1.5 deep-scrub starts 2024-09-20T22:57:15.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:15 smithi084 ceph-mon[40024]: 1.5 deep-scrub ok 2024-09-20T22:57:15.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:15 smithi084 ceph-mon[40024]: osdmap e954: 8 total, 7 up, 7 in 2024-09-20T22:57:15.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:15 smithi084 ceph-mon[40024]: 6.a scrub starts 2024-09-20T22:57:15.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:15 smithi084 ceph-mon[40024]: 6.a scrub ok 2024-09-20T22:57:15.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:15 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/367011767' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-20T22:57:15.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:15 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-20T22:57:15.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:15 smithi084 ceph-mon[40024]: 3.14 scrub starts 2024-09-20T22:57:15.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:15 smithi084 ceph-mon[40024]: 3.14 scrub ok 2024-09-20T22:57:15.662 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:57:15.684 INFO:tasks.rados.rados.0.smithi084.stdout:3863: finishing write tid 1 to smithi08477203-27 2024-09-20T22:57:15.684 INFO:tasks.rados.rados.0.smithi084.stdout:3860: done (3 left) 2024-09-20T22:57:15.684 INFO:tasks.rados.rados.0.smithi084.stdout:3861: done (2 left) 2024-09-20T22:57:15.684 INFO:tasks.rados.rados.0.smithi084.stdout:3862: done (1 left) 2024-09-20T22:57:15.684 INFO:tasks.rados.rados.0.smithi084.stdout:3864: snap_remove snap 446 2024-09-20T22:57:15.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:15.685 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:15.686 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:57:15.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:15 smithi154 ceph-mon[33264]: pgmap v1332: 168 pgs: 1 active+clean+snaptrim, 2 active+undersized+degraded+remapped+backfill_toofull, 41 active+undersized, 1 active+remapped+backfill_toofull, 36 active+undersized+degraded, 87 active+clean; 810 MiB data, 2.3 GiB used, 624 GiB / 626 GiB avail; 8.3 MiB/s rd, 1.6 MiB/s wr, 11 op/s; 290/1683 objects degraded (17.231%); 55/1683 objects misplaced (3.268%) 2024-09-20T22:57:15.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:15 smithi154 ceph-mon[33264]: 1.5 deep-scrub starts 2024-09-20T22:57:15.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:15 smithi154 ceph-mon[33264]: 1.5 deep-scrub ok 2024-09-20T22:57:15.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:15 smithi154 ceph-mon[33264]: osdmap e954: 8 total, 7 up, 7 in 2024-09-20T22:57:15.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:15 smithi154 ceph-mon[33264]: 6.a scrub starts 2024-09-20T22:57:15.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:15 smithi154 ceph-mon[33264]: 6.a scrub ok 2024-09-20T22:57:15.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:15 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/367011767' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-20T22:57:15.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:15 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-20T22:57:15.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:15 smithi154 ceph-mon[33264]: 3.14 scrub starts 2024-09-20T22:57:15.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:15 smithi154 ceph-mon[33264]: 3.14 scrub ok 2024-09-20T22:57:15.786 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:57:15.809 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:15.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:15.810 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:57:15.910 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:57:15.931 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:15.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:15.932 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:57:16.033 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:57:16.057 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:16.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:16.058 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:57:16.159 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:57:16.182 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:16.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:16.183 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:57:16.234 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.0 dump_historic_ops 2024-09-20T22:57:16.284 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:57:16.305 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:16.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:16.306 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:57:16.406 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:57:16.430 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:16.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:16.431 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:57:16.507 INFO:tasks.rados.rados.0.smithi084.stdout:3864: done (1 left) 2024-09-20T22:57:16.508 INFO:tasks.rados.rados.0.smithi084.stdout:3865: read oid 38 snap -1 2024-09-20T22:57:16.508 INFO:tasks.rados.rados.0.smithi084.stdout:3865: expect (ObjNum 1118 snap 380 seq_num 1118) 2024-09-20T22:57:16.508 INFO:tasks.rados.rados.0.smithi084.stdout:3866: read oid 39 snap -1 2024-09-20T22:57:16.508 INFO:tasks.rados.rados.0.smithi084.stdout:3866: expect (ObjNum 738 snap 235 seq_num 738) 2024-09-20T22:57:16.508 INFO:tasks.rados.rados.0.smithi084.stdout:3867: read oid 23 snap -1 2024-09-20T22:57:16.508 INFO:tasks.rados.rados.0.smithi084.stdout:3867: expect deleted 2024-09-20T22:57:16.508 INFO:tasks.rados.rados.0.smithi084.stdout:3868: read oid 49 snap -1 2024-09-20T22:57:16.508 INFO:tasks.rados.rados.0.smithi084.stdout:3868: expect (ObjNum 1288 snap 436 seq_num 1288) 2024-09-20T22:57:16.508 INFO:tasks.rados.rados.0.smithi084.stdout:3869: write oid 36 current snap is 453 2024-09-20T22:57:16.508 INFO:tasks.rados.rados.0.smithi084.stdout:3869: seq_num 1314 ranges {484398=489902,1392398=411118} 2024-09-20T22:57:16.514 INFO:tasks.rados.rados.0.smithi084.stdout:3869: writing smithi08477203-36 from 484398 to 974300 tid 1 2024-09-20T22:57:16.520 INFO:tasks.rados.rados.0.smithi084.stdout:3869: writing smithi08477203-36 from 1392398 to 1803516 tid 2 2024-09-20T22:57:16.520 INFO:tasks.rados.rados.0.smithi084.stdout:3870: delete oid 30 current snap is 453 2024-09-20T22:57:16.520 INFO:tasks.rados.rados.0.smithi084.stdout:3863: finishing write tid 2 to smithi08477203-27 2024-09-20T22:57:16.521 INFO:tasks.rados.rados.0.smithi084.stdout:3863: finishing write tid 3 to smithi08477203-27 2024-09-20T22:57:16.521 INFO:tasks.rados.rados.0.smithi084.stdout:3863: finishing write tid 4 to smithi08477203-27 2024-09-20T22:57:16.521 INFO:tasks.rados.rados.0.smithi084.stdout:3863: oid 27 updating version 0 to 2605 2024-09-20T22:57:16.521 INFO:tasks.rados.rados.0.smithi084.stdout:3863: oid 27 version 2605 is already newer than 2604 2024-09-20T22:57:16.521 INFO:tasks.rados.rados.0.smithi084.stdout:3863: oid 27 version 2605 is already newer than 2603 2024-09-20T22:57:16.521 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 27 v 2605 (ObjNum 1313 snap 453 seq_num 1313) dirty exists 2024-09-20T22:57:16.521 INFO:tasks.rados.rados.0.smithi084.stdout:3863: left oid 27 (ObjNum 1313 snap 453 seq_num 1313) 2024-09-20T22:57:16.521 INFO:tasks.rados.rados.0.smithi084.stdout:3866: expect (ObjNum 738 snap 235 seq_num 738) 2024-09-20T22:57:16.531 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:57:16.553 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:16.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:16.554 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:57:16.600 INFO:tasks.rados.rados.0.smithi084.stdout:3868: expect (ObjNum 1288 snap 436 seq_num 1288) 2024-09-20T22:57:16.655 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:57:16.675 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:16.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:16.676 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:57:16.746 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:16 smithi084 ceph-mon[40024]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:57:16.746 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:16 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T22:57:16.746 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:16 smithi084 ceph-mon[40024]: osdmap e955: 8 total, 7 up, 7 in 2024-09-20T22:57:16.747 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:16 smithi084 ceph-mon[29774]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:57:16.747 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:16 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T22:57:16.747 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:16 smithi084 ceph-mon[29774]: osdmap e955: 8 total, 7 up, 7 in 2024-09-20T22:57:16.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:16 smithi154 ceph-mon[33264]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:57:16.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:16 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T22:57:16.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:16 smithi154 ceph-mon[33264]: osdmap e955: 8 total, 7 up, 7 in 2024-09-20T22:57:16.765 INFO:tasks.rados.rados.0.smithi084.stdout:3865: expect (ObjNum 1118 snap 380 seq_num 1118) 2024-09-20T22:57:16.777 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:57:16.778 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell -n osd.3 --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-3 --journal-path /var/lib/ceph/osd/ceph-3/journal --force --op remove --pgid 3.7 2024-09-20T22:57:16.800 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:16.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:16.801 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:57:16.902 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:57:16.923 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:16.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:16.924 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:57:16.998 INFO:tasks.rados.rados.0.smithi084.stdout:3869: finishing write tid 1 to smithi08477203-36 2024-09-20T22:57:16.999 INFO:tasks.rados.rados.0.smithi084.stdout:3869: finishing write tid 2 to smithi08477203-36 2024-09-20T22:57:16.999 INFO:tasks.rados.rados.0.smithi084.stdout:3869: finishing write tid 3 to smithi08477203-36 2024-09-20T22:57:16.999 INFO:tasks.rados.rados.0.smithi084.stdout:3869: finishing write tid 4 to smithi08477203-36 2024-09-20T22:57:16.999 INFO:tasks.rados.rados.0.smithi084.stdout:3869: oid 36 updating version 0 to 2648 2024-09-20T22:57:16.999 INFO:tasks.rados.rados.0.smithi084.stdout:3869: oid 36 updating version 2648 to 2650 2024-09-20T22:57:16.999 INFO:tasks.rados.rados.0.smithi084.stdout:3869: oid 36 version 2650 is already newer than 2649 2024-09-20T22:57:16.999 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 36 v 2650 (ObjNum 1314 snap 453 seq_num 1314) dirty exists 2024-09-20T22:57:16.999 INFO:tasks.rados.rados.0.smithi084.stdout:3869: left oid 36 (ObjNum 1314 snap 453 seq_num 1314) 2024-09-20T22:57:16.999 INFO:tasks.rados.rados.0.smithi084.stdout:3863: done (6 left) 2024-09-20T22:57:16.999 INFO:tasks.rados.rados.0.smithi084.stdout:3865: done (5 left) 2024-09-20T22:57:16.999 INFO:tasks.rados.rados.0.smithi084.stdout:3866: done (4 left) 2024-09-20T22:57:16.999 INFO:tasks.rados.rados.0.smithi084.stdout:3867: done (3 left) 2024-09-20T22:57:16.999 INFO:tasks.rados.rados.0.smithi084.stdout:3868: done (2 left) 2024-09-20T22:57:16.999 INFO:tasks.rados.rados.0.smithi084.stdout:3869: done (1 left) 2024-09-20T22:57:17.000 INFO:tasks.rados.rados.0.smithi084.stdout:3870: done (0 left) 2024-09-20T22:57:17.000 INFO:tasks.rados.rados.0.smithi084.stdout:3871: read oid 17 snap -1 2024-09-20T22:57:17.000 INFO:tasks.rados.rados.0.smithi084.stdout:3871: expect (ObjNum 1302 snap 442 seq_num 1302) 2024-09-20T22:57:17.000 INFO:tasks.rados.rados.0.smithi084.stdout:3872: delete oid 22 current snap is 453 2024-09-20T22:57:17.000 INFO:tasks.rados.rados.0.smithi084.stdout:3872: done (1 left) 2024-09-20T22:57:17.000 INFO:tasks.rados.rados.0.smithi084.stdout:3873: delete oid 50 current snap is 453 2024-09-20T22:57:17.002 INFO:tasks.rados.rados.0.smithi084.stdout:3873: done (1 left) 2024-09-20T22:57:17.002 INFO:tasks.rados.rados.0.smithi084.stdout:3874: snap_create 2024-09-20T22:57:17.005 INFO:tasks.rados.rados.0.smithi084.stdout:3871: expect (ObjNum 1302 snap 442 seq_num 1302) 2024-09-20T22:57:17.025 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:57:17.045 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:17.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:17.046 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:57:17.147 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:57:17.170 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:17.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:17.171 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:57:17.271 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:57:17.293 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:17.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:17.293 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:57:17.334 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:57:17.366 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:57:17.394 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:57:17.415 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:17.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:17.416 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:57:17.517 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:57:17.520 INFO:tasks.rados.rados.0.smithi084.stdout:3871: done (1 left) 2024-09-20T22:57:17.520 INFO:tasks.rados.rados.0.smithi084.stdout:3874: done (0 left) 2024-09-20T22:57:17.520 INFO:tasks.rados.rados.0.smithi084.stdout:3875: snap_create 2024-09-20T22:57:17.538 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:17.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:17.539 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:57:17.580 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:17 smithi084 ceph-mon[40024]: pgmap v1335: 168 pgs: 1 active+clean+snaptrim, 2 active+undersized+degraded+remapped+backfill_toofull, 41 active+undersized, 1 active+remapped+backfill_toofull, 36 active+undersized+degraded, 87 active+clean; 810 MiB data, 2.3 GiB used, 624 GiB / 626 GiB avail; 8.3 MiB/s rd, 1.6 MiB/s wr, 11 op/s; 290/1683 objects degraded (17.231%); 55/1683 objects misplaced (3.268%) 2024-09-20T22:57:17.580 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:17 smithi084 ceph-mon[40024]: osdmap e956: 8 total, 7 up, 7 in 2024-09-20T22:57:17.580 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:17 smithi084 ceph-mon[29774]: pgmap v1335: 168 pgs: 1 active+clean+snaptrim, 2 active+undersized+degraded+remapped+backfill_toofull, 41 active+undersized, 1 active+remapped+backfill_toofull, 36 active+undersized+degraded, 87 active+clean; 810 MiB data, 2.3 GiB used, 624 GiB / 626 GiB avail; 8.3 MiB/s rd, 1.6 MiB/s wr, 11 op/s; 290/1683 objects degraded (17.231%); 55/1683 objects misplaced (3.268%) 2024-09-20T22:57:17.581 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:17 smithi084 ceph-mon[29774]: osdmap e956: 8 total, 7 up, 7 in 2024-09-20T22:57:17.640 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:57:17.662 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:17.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:17.663 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:57:17.764 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:57:17.788 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:17.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:17.789 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:57:17.890 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:57:17.913 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:17.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:17.914 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:57:18.001 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/osd.3/config 2024-09-20T22:57:18.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:17 smithi154 ceph-mon[33264]: pgmap v1335: 168 pgs: 1 active+clean+snaptrim, 2 active+undersized+degraded+remapped+backfill_toofull, 41 active+undersized, 1 active+remapped+backfill_toofull, 36 active+undersized+degraded, 87 active+clean; 810 MiB data, 2.3 GiB used, 624 GiB / 626 GiB avail; 8.3 MiB/s rd, 1.6 MiB/s wr, 11 op/s; 290/1683 objects degraded (17.231%); 55/1683 objects misplaced (3.268%) 2024-09-20T22:57:18.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:17 smithi154 ceph-mon[33264]: osdmap e956: 8 total, 7 up, 7 in 2024-09-20T22:57:18.015 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:57:18.038 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:18.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:18.039 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:57:18.087 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:57:18.087 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:57:18.087 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:57:18.087 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:57:18.087 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:57:18.087 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:57:18.087 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:57:18.087 INFO:teuthology.orchestra.run.smithi084.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-20T22:57:18.088 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:57:18.088 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:57:18.088 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:57:18.088 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:57:18.088 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:57:18.093 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:57:18.093 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:57:18.140 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:57:18.163 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:18.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:18.164 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:57:18.265 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:57:18.286 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:18.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:18.288 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:57:18.363 INFO:tasks.rados.rados.0.smithi084.stdout:3875: done (0 left) 2024-09-20T22:57:18.364 INFO:tasks.rados.rados.0.smithi084.stdout:3876: snap_remove snap 425 2024-09-20T22:57:18.388 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:57:18.412 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:18.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:18.413 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:57:18.453 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:57:18 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: ::ffff:172.21.15.154 - - [20/Sep/2024:22:57:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-20T22:57:18.514 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:57:18.539 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:18.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:18.540 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:57:18.641 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:57:18.662 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:18.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:18.663 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:57:18.764 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:57:18.787 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:18.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:18.788 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:57:18.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:18 smithi084 ceph-mon[29774]: osdmap e957: 8 total, 7 up, 7 in 2024-09-20T22:57:18.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:18 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/455748915' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:57:18.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:18 smithi084 ceph-mon[29774]: Health check update: Degraded data redundancy: 290/1683 objects degraded (17.231%), 38 pgs degraded (PG_DEGRADED) 2024-09-20T22:57:18.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:18 smithi084 ceph-mon[29774]: osdmap e958: 8 total, 7 up, 7 in 2024-09-20T22:57:18.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:18 smithi084 ceph-mon[40024]: osdmap e957: 8 total, 7 up, 7 in 2024-09-20T22:57:18.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:18 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/455748915' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:57:18.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:18 smithi084 ceph-mon[40024]: Health check update: Degraded data redundancy: 290/1683 objects degraded (17.231%), 38 pgs degraded (PG_DEGRADED) 2024-09-20T22:57:18.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:18 smithi084 ceph-mon[40024]: osdmap e958: 8 total, 7 up, 7 in 2024-09-20T22:57:18.889 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:57:18.910 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:18.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:18.910 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:57:19.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:18 smithi154 ceph-mon[33264]: osdmap e957: 8 total, 7 up, 7 in 2024-09-20T22:57:19.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:18 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/455748915' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:57:19.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:18 smithi154 ceph-mon[33264]: Health check update: Degraded data redundancy: 290/1683 objects degraded (17.231%), 38 pgs degraded (PG_DEGRADED) 2024-09-20T22:57:19.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:18 smithi154 ceph-mon[33264]: osdmap e958: 8 total, 7 up, 7 in 2024-09-20T22:57:19.012 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:57:19.033 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:19.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:19.034 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:57:19.135 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:57:19.160 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:19.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:19.161 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:57:19.262 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:57:19.288 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:19.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:19.289 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:57:19.344 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-20T22:57:19.368 INFO:tasks.rados.rados.0.smithi084.stdout:3876: done (0 left) 2024-09-20T22:57:19.368 INFO:tasks.rados.rados.0.smithi084.stdout:3877: write (excl) oid 19 current snap is 455 2024-09-20T22:57:19.368 INFO:tasks.rados.rados.0.smithi084.stdout:3877: seq_num 1318 ranges {551626=507251,1626326=418456} 2024-09-20T22:57:19.373 INFO:tasks.rados.rados.0.smithi084.stdout:3877: writing smithi08477203-19 from 551626 to 1058877 tid 1 2024-09-20T22:57:19.379 INFO:tasks.rados.rados.0.smithi084.stdout:3877: writing smithi08477203-19 from 1626326 to 2044782 tid 2 2024-09-20T22:57:19.379 INFO:tasks.rados.rados.0.smithi084.stdout:3878: write (excl) oid 43 current snap is 455 2024-09-20T22:57:19.379 INFO:tasks.rados.rados.0.smithi084.stdout:3877: finishing write tid 1 to smithi08477203-19 2024-09-20T22:57:19.379 INFO:tasks.rados.rados.0.smithi084.stdout:3878: seq_num 1319 ranges {568433=582500,1684808=483625} 2024-09-20T22:57:19.385 INFO:tasks.rados.rados.0.smithi084.stdout:3878: writing smithi08477203-43 from 568433 to 1150933 tid 1 2024-09-20T22:57:19.389 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:57:19.391 INFO:tasks.rados.rados.0.smithi084.stdout:3878: writing smithi08477203-43 from 1684808 to 2168433 tid 2 2024-09-20T22:57:19.391 INFO:tasks.rados.rados.0.smithi084.stdout:3879: snap_remove snap 395 2024-09-20T22:57:19.413 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:19.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:19.414 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:57:19.438 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-20T22:57:19.438 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2024-09-20T22:57:19.438 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2024-09-20T22:57:19.439 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [ 2024-09-20T22:57:19.439 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.439 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:4311 7.17 7:ed54065f:::smithi08477203-12:head [append 0~0] snapc b1=[b1,b0,af,ae,ad,ab,a9,a7,a6,a5,9f,9e,9d,9c,9a,96,95,8e,87,86] ondisk+write+known_if_redirected+supports_pool_eio e433)", 2024-09-20T22:57:19.439 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:48:33.213556+0000", 2024-09-20T22:57:19.439 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 524.88288335599998, 2024-09-20T22:57:19.439 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0280906110000001, 2024-09-20T22:57:19.439 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:57:19.439 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:57:19.439 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:57:19.439 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:57:19.439 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:57:19.439 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:57:19.440 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 4311 2024-09-20T22:57:19.440 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.440 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:57:19.440 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.440 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:57:19.440 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:33.213556+0000", 2024-09-20T22:57:19.440 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:57:19.440 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.440 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.440 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:57:19.441 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:33.213556+0000", 2024-09-20T22:57:19.441 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:57:19.441 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.441 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.441 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:57:19.441 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:33.213558+0000", 2024-09-20T22:57:19.441 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.4470000000000001e-06 2024-09-20T22:57:19.441 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.441 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.441 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:57:19.441 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:33.213563+0000", 2024-09-20T22:57:19.441 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.6840000000000004e-06 2024-09-20T22:57:19.441 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.441 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.441 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:57:19.442 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:33.213573+0000", 2024-09-20T22:57:19.442 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0873000000000001e-05 2024-09-20T22:57:19.442 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.442 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.442 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:57:19.442 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:33.213584+0000", 2024-09-20T22:57:19.442 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0852e-05 2024-09-20T22:57:19.442 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.442 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.442 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:57:19.442 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:33.213633+0000", 2024-09-20T22:57:19.442 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.8683999999999997e-05 2024-09-20T22:57:19.443 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.443 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.443 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-09-20T22:57:19.443 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:33.213646+0000", 2024-09-20T22:57:19.443 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3205e-05 2024-09-20T22:57:19.443 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.443 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.443 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:57:19.443 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:34.226325+0000", 2024-09-20T22:57:19.443 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0126793300000001 2024-09-20T22:57:19.443 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.443 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.443 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for degraded object", 2024-09-20T22:57:19.443 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:34.228377+0000", 2024-09-20T22:57:19.444 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0020512540000000002 2024-09-20T22:57:19.444 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.444 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.444 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:57:19.444 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:34.239166+0000", 2024-09-20T22:57:19.444 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010789029 2024-09-20T22:57:19.444 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.444 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.444 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:57:19.444 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:34.239275+0000", 2024-09-20T22:57:19.444 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00010938900000000001 2024-09-20T22:57:19.444 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.444 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.445 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 3,6", 2024-09-20T22:57:19.445 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:34.239569+0000", 2024-09-20T22:57:19.445 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00029410699999999999 2024-09-20T22:57:19.445 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.445 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.445 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:57:19.445 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:34.241371+0000", 2024-09-20T22:57:19.445 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00180171 2024-09-20T22:57:19.445 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.445 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.445 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:57:19.445 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:34.241393+0000", 2024-09-20T22:57:19.445 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.1928000000000001e-05 2024-09-20T22:57:19.445 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.446 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.446 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:57:19.446 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:34.241465+0000", 2024-09-20T22:57:19.446 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.1799999999999997e-05 2024-09-20T22:57:19.446 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.446 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.446 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:57:19.446 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:34.241528+0000", 2024-09-20T22:57:19.446 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.2946000000000003e-05 2024-09-20T22:57:19.446 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.446 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.446 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:57:19.446 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:34.241647+0000", 2024-09-20T22:57:19.446 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000119373 2024-09-20T22:57:19.446 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:57:19.447 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:57:19.447 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:57:19.447 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.447 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.447 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:4312 7.17 7:ed54065f:::smithi08477203-12:head [rollback 96] snapc b1=[b1,b0,af,ae,ad,ab,a9,a7,a6,a5,9f,9e,9d,9c,9a,96,95,8e,87,86] ondisk+write+known_if_redirected+supports_pool_eio e433)", 2024-09-20T22:57:19.447 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:48:33.213603+0000", 2024-09-20T22:57:19.447 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 524.88283693400001, 2024-09-20T22:57:19.447 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0286283940000001, 2024-09-20T22:57:19.447 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:57:19.447 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:57:19.447 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:57:19.447 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:57:19.447 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:57:19.448 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:57:19.448 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 4312 2024-09-20T22:57:19.448 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.448 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:57:19.448 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.448 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:57:19.448 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:33.213603+0000", 2024-09-20T22:57:19.448 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:57:19.448 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.448 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.448 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:57:19.448 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:33.213603+0000", 2024-09-20T22:57:19.448 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:57:19.448 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.448 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.449 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:57:19.449 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:33.213603+0000", 2024-09-20T22:57:19.449 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.9200000000000001e-07 2024-09-20T22:57:19.449 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.449 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.449 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:57:19.449 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:33.213606+0000", 2024-09-20T22:57:19.449 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.6390000000000001e-06 2024-09-20T22:57:19.449 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.449 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.449 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:57:19.449 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:33.213612+0000", 2024-09-20T22:57:19.449 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.5429999999999999e-06 2024-09-20T22:57:19.449 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.449 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.450 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:57:19.450 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:33.213618+0000", 2024-09-20T22:57:19.450 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.4249999999999997e-06 2024-09-20T22:57:19.450 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.450 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.450 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:57:19.450 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:33.213674+0000", 2024-09-20T22:57:19.450 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.6341000000000001e-05 2024-09-20T22:57:19.450 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.450 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.450 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-09-20T22:57:19.450 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:33.214118+0000", 2024-09-20T22:57:19.450 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00044406600000000001 2024-09-20T22:57:19.451 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.451 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.451 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:57:19.451 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:34.228938+0000", 2024-09-20T22:57:19.451 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0148198470000001 2024-09-20T22:57:19.451 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.451 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.451 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for degraded object", 2024-09-20T22:57:19.451 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:34.229016+0000", 2024-09-20T22:57:19.451 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.7383000000000001e-05 2024-09-20T22:57:19.451 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.451 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.451 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:57:19.452 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:34.240634+0000", 2024-09-20T22:57:19.452 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011618277999999999 2024-09-20T22:57:19.452 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.452 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.452 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:57:19.452 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:34.240749+0000", 2024-09-20T22:57:19.452 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00011473500000000001 2024-09-20T22:57:19.452 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.452 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.452 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 3,6", 2024-09-20T22:57:19.452 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:34.241041+0000", 2024-09-20T22:57:19.452 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00029246699999999998 2024-09-20T22:57:19.452 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.452 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.453 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:57:19.453 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:34.242022+0000", 2024-09-20T22:57:19.453 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00098068999999999999 2024-09-20T22:57:19.453 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.453 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.453 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:57:19.453 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:34.242068+0000", 2024-09-20T22:57:19.453 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.6581e-05 2024-09-20T22:57:19.453 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.453 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.453 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:57:19.453 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:34.242081+0000", 2024-09-20T22:57:19.453 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2709e-05 2024-09-20T22:57:19.453 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.453 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.453 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:57:19.454 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:34.242116+0000", 2024-09-20T22:57:19.454 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.4579000000000002e-05 2024-09-20T22:57:19.454 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.454 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.454 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:57:19.454 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:34.242231+0000", 2024-09-20T22:57:19.454 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00011561900000000001 2024-09-20T22:57:19.454 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:57:19.454 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:57:19.454 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:57:19.454 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.454 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.455 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:4415 7.16 7.fa65a116 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e444)", 2024-09-20T22:57:19.455 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:48:44.266352+0000", 2024-09-20T22:57:19.455 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 513.83008761799999, 2024-09-20T22:57:19.455 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.96293146200000002, 2024-09-20T22:57:19.455 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:57:19.455 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:57:19.455 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "reached pg", 2024-09-20T22:57:19.455 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:57:19.455 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:57:19.455 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:57:19.455 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 4415 2024-09-20T22:57:19.455 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.455 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:57:19.455 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.456 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:57:19.456 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:44.266352+0000", 2024-09-20T22:57:19.456 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:57:19.456 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.456 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.456 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:57:19.456 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:44.266352+0000", 2024-09-20T22:57:19.456 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:57:19.456 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.456 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.456 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:57:19.456 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:44.266354+0000", 2024-09-20T22:57:19.456 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.733e-06 2024-09-20T22:57:19.456 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.456 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.457 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:57:19.457 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:44.266610+0000", 2024-09-20T22:57:19.457 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000256352 2024-09-20T22:57:19.457 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.457 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.457 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:57:19.457 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:44.266626+0000", 2024-09-20T22:57:19.457 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.5347000000000001e-05 2024-09-20T22:57:19.457 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.457 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.457 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:57:19.457 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:44.266634+0000", 2024-09-20T22:57:19.457 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.5390000000000007e-06 2024-09-20T22:57:19.457 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.458 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.458 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:57:19.458 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:44.266701+0000", 2024-09-20T22:57:19.458 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.6908000000000004e-05 2024-09-20T22:57:19.458 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.458 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.458 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-09-20T22:57:19.458 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:44.266710+0000", 2024-09-20T22:57:19.458 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.2920000000000001e-06 2024-09-20T22:57:19.458 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.458 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.458 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:57:19.458 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:45.229222+0000", 2024-09-20T22:57:19.458 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.96251129899999999 2024-09-20T22:57:19.458 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.459 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.459 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:57:19.459 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:45.229284+0000", 2024-09-20T22:57:19.459 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.1991999999999999e-05 2024-09-20T22:57:19.459 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:57:19.459 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:57:19.459 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:57:19.459 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.459 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.459 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:4416 7.16 7.fa65a116 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e444)", 2024-09-20T22:57:19.459 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:48:44.273462+0000", 2024-09-20T22:57:19.459 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 513.82297818799998, 2024-09-20T22:57:19.459 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.95612304000000004, 2024-09-20T22:57:19.460 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:57:19.460 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:57:19.460 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "reached pg", 2024-09-20T22:57:19.460 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:57:19.460 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:57:19.460 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:57:19.460 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 4416 2024-09-20T22:57:19.460 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.460 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:57:19.460 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.460 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:57:19.460 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:44.273462+0000", 2024-09-20T22:57:19.460 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:57:19.460 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.461 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.461 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:57:19.461 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:44.273462+0000", 2024-09-20T22:57:19.461 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:57:19.461 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.461 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.461 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:57:19.461 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:44.273464+0000", 2024-09-20T22:57:19.461 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.3609999999999999e-06 2024-09-20T22:57:19.461 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.461 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.462 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:57:19.462 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:44.273675+0000", 2024-09-20T22:57:19.462 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00021111400000000001 2024-09-20T22:57:19.462 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.462 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.462 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:57:19.462 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:44.273697+0000", 2024-09-20T22:57:19.462 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.2305000000000001e-05 2024-09-20T22:57:19.462 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.462 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.462 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:57:19.462 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:44.273708+0000", 2024-09-20T22:57:19.462 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0669999999999999e-05 2024-09-20T22:57:19.462 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.462 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.463 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:57:19.463 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:44.273865+0000", 2024-09-20T22:57:19.463 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000156766 2024-09-20T22:57:19.463 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.463 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.463 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-09-20T22:57:19.463 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:44.273878+0000", 2024-09-20T22:57:19.463 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3178e-05 2024-09-20T22:57:19.463 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.463 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.463 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:57:19.463 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:45.229577+0000", 2024-09-20T22:57:19.463 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.95569872600000005 2024-09-20T22:57:19.463 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.464 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.464 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:57:19.464 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:45.229585+0000", 2024-09-20T22:57:19.464 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.9200000000000004e-06 2024-09-20T22:57:19.464 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:57:19.464 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:57:19.464 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:57:19.464 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.464 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.464 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:4417 7.16 7.fa65a116 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e444)", 2024-09-20T22:57:19.464 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:48:44.273740+0000", 2024-09-20T22:57:19.464 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 513.82269950800003, 2024-09-20T22:57:19.464 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.95591940399999997, 2024-09-20T22:57:19.464 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:57:19.465 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:57:19.465 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "reached pg", 2024-09-20T22:57:19.465 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:57:19.465 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:57:19.465 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:57:19.465 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 4417 2024-09-20T22:57:19.465 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.465 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:57:19.465 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.465 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:57:19.465 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:44.273740+0000", 2024-09-20T22:57:19.465 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:57:19.465 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.466 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.466 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:57:19.466 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:44.273740+0000", 2024-09-20T22:57:19.466 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:57:19.466 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.466 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.466 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:57:19.466 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:44.273741+0000", 2024-09-20T22:57:19.466 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.06e-07 2024-09-20T22:57:19.466 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.466 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.466 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:57:19.466 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:44.273746+0000", 2024-09-20T22:57:19.466 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.7129999999999996e-06 2024-09-20T22:57:19.467 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.467 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.467 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:57:19.467 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:44.273751+0000", 2024-09-20T22:57:19.467 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.8309999999999997e-06 2024-09-20T22:57:19.467 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.467 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.467 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:57:19.467 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:44.273756+0000", 2024-09-20T22:57:19.467 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.6859999999999997e-06 2024-09-20T22:57:19.467 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.467 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.467 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:57:19.468 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:44.274433+0000", 2024-09-20T22:57:19.468 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00067743000000000002 2024-09-20T22:57:19.468 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.468 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.468 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-09-20T22:57:19.468 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:44.274446+0000", 2024-09-20T22:57:19.468 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2737e-05 2024-09-20T22:57:19.468 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.468 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.468 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:57:19.468 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:45.229650+0000", 2024-09-20T22:57:19.468 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.95520395499999999 2024-09-20T22:57:19.468 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.468 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.468 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:57:19.469 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:45.229660+0000", 2024-09-20T22:57:19.469 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.4460000000000007e-06 2024-09-20T22:57:19.469 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:57:19.469 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:57:19.469 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:57:19.469 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.469 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.469 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:4418 7.16 7.fa65a116 (undecoded) ondisk+read+rwordered+known_if_redirected+supports_pool_eio e444)", 2024-09-20T22:57:19.469 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:48:44.273764+0000", 2024-09-20T22:57:19.469 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 513.82267585399995, 2024-09-20T22:57:19.469 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.95592935400000001, 2024-09-20T22:57:19.469 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:57:19.469 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:57:19.469 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "reached pg", 2024-09-20T22:57:19.469 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:57:19.469 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:57:19.470 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:57:19.470 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 4418 2024-09-20T22:57:19.470 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.470 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:57:19.470 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.470 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:57:19.470 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:44.273764+0000", 2024-09-20T22:57:19.470 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:57:19.470 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.470 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.470 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:57:19.470 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:44.273764+0000", 2024-09-20T22:57:19.470 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:57:19.470 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.470 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.471 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:57:19.471 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:44.273764+0000", 2024-09-20T22:57:19.471 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.2e-07 2024-09-20T22:57:19.471 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.471 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.471 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:57:19.471 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:44.273773+0000", 2024-09-20T22:57:19.471 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.0489999999999993e-06 2024-09-20T22:57:19.471 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.471 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.471 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:57:19.472 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:44.273780+0000", 2024-09-20T22:57:19.472 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.195e-06 2024-09-20T22:57:19.472 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.472 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.472 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:57:19.472 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:44.273784+0000", 2024-09-20T22:57:19.472 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.245e-06 2024-09-20T22:57:19.472 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.472 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.472 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:57:19.472 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:44.274475+0000", 2024-09-20T22:57:19.472 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00069082200000000001 2024-09-20T22:57:19.472 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.473 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.473 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-09-20T22:57:19.473 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:44.274484+0000", 2024-09-20T22:57:19.473 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.6940000000000007e-06 2024-09-20T22:57:19.473 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.473 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.473 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:57:19.473 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:45.229685+0000", 2024-09-20T22:57:19.473 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.95520094 2024-09-20T22:57:19.473 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.473 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.473 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:57:19.473 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:45.229693+0000", 2024-09-20T22:57:19.473 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.9889999999999992e-06 2024-09-20T22:57:19.474 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:57:19.474 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:57:19.474 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:57:19.474 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.474 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.474 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:4415 7.6 7:6885a65f:::smithi08477203-14:head [write 685884~560628] snapc b7=[b7,b6,b5,b4,b3,b2,b1,b0,af,ae,a9,a7,a6,a5,9e,9c,9a,96,95,87,86] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e446)", 2024-09-20T22:57:19.474 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:48:46.223519+0000", 2024-09-20T22:57:19.474 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 511.87292079899998, 2024-09-20T22:57:19.474 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0194105410000001, 2024-09-20T22:57:19.474 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:57:19.474 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:57:19.474 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:57:19.474 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:57:19.474 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:57:19.475 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:57:19.475 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 4415 2024-09-20T22:57:19.475 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.475 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:57:19.475 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.475 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:57:19.475 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:46.223519+0000", 2024-09-20T22:57:19.475 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:57:19.475 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.475 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.475 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:57:19.475 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:46.223519+0000", 2024-09-20T22:57:19.475 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:57:19.475 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.476 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.476 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:57:19.476 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:46.223521+0000", 2024-09-20T22:57:19.476 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.7260000000000001e-06 2024-09-20T22:57:19.476 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.476 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.476 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:57:19.476 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:46.223861+0000", 2024-09-20T22:57:19.476 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00034058200000000003 2024-09-20T22:57:19.476 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.476 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.476 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:57:19.476 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:46.223874+0000", 2024-09-20T22:57:19.476 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2777e-05 2024-09-20T22:57:19.476 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.477 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.477 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:57:19.477 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:46.223884+0000", 2024-09-20T22:57:19.477 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0326000000000001e-05 2024-09-20T22:57:19.477 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.477 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.477 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:57:19.477 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:46.228721+0000", 2024-09-20T22:57:19.477 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0048363470000000004 2024-09-20T22:57:19.477 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.477 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.477 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-09-20T22:57:19.477 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:46.228734+0000", 2024-09-20T22:57:19.477 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3219e-05 2024-09-20T22:57:19.477 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.478 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.478 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:57:19.478 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:47.235776+0000", 2024-09-20T22:57:19.478 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0070420579999999 2024-09-20T22:57:19.478 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.478 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.478 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:57:19.478 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:47.235914+0000", 2024-09-20T22:57:19.478 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013761599999999999 2024-09-20T22:57:19.478 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.478 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.478 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 3,7", 2024-09-20T22:57:19.478 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:47.236144+0000", 2024-09-20T22:57:19.479 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000230165 2024-09-20T22:57:19.479 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.479 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.479 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:57:19.479 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:47.242485+0000", 2024-09-20T22:57:19.479 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0063409929999999996 2024-09-20T22:57:19.479 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.479 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.479 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:57:19.479 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:47.242500+0000", 2024-09-20T22:57:19.479 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.5676e-05 2024-09-20T22:57:19.479 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.479 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.479 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:57:19.479 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:47.242790+0000", 2024-09-20T22:57:19.480 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00028984699999999998 2024-09-20T22:57:19.480 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.480 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.480 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:57:19.480 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:47.242860+0000", 2024-09-20T22:57:19.480 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.9765999999999996e-05 2024-09-20T22:57:19.480 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.480 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.480 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:57:19.480 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:47.242929+0000", 2024-09-20T22:57:19.480 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.9443000000000006e-05 2024-09-20T22:57:19.480 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:57:19.480 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:57:19.480 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:57:19.480 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.480 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.481 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:4416 7.6 7:6885a65f:::smithi08477203-14:head [write 2046274~239610] snapc b7=[b7,b6,b5,b4,b3,b2,b1,b0,af,ae,a9,a7,a6,a5,9e,9c,9a,96,95,87,86] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e446)", 2024-09-20T22:57:19.481 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:48:46.225256+0000", 2024-09-20T22:57:19.481 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 511.87118350600002, 2024-09-20T22:57:19.481 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0189633840000001, 2024-09-20T22:57:19.481 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:57:19.481 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:57:19.481 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:57:19.481 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:57:19.481 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:57:19.481 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:57:19.482 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 4416 2024-09-20T22:57:19.482 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.482 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:57:19.482 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.482 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:57:19.482 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:46.225256+0000", 2024-09-20T22:57:19.482 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:57:19.482 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.482 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.482 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:57:19.482 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:46.225256+0000", 2024-09-20T22:57:19.482 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:57:19.482 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.483 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.483 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:57:19.483 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:46.225258+0000", 2024-09-20T22:57:19.483 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.279e-06 2024-09-20T22:57:19.483 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.483 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.483 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:57:19.483 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:46.225882+0000", 2024-09-20T22:57:19.483 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00062460200000000003 2024-09-20T22:57:19.483 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.483 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.483 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:57:19.483 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:46.225894+0000", 2024-09-20T22:57:19.483 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2306999999999999e-05 2024-09-20T22:57:19.483 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.484 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.484 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:57:19.484 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:46.225908+0000", 2024-09-20T22:57:19.484 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3643e-05 2024-09-20T22:57:19.484 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.484 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.484 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:57:19.484 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:46.228769+0000", 2024-09-20T22:57:19.484 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.002860656 2024-09-20T22:57:19.484 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.484 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.484 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-09-20T22:57:19.484 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:46.228778+0000", 2024-09-20T22:57:19.484 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.8370000000000003e-06 2024-09-20T22:57:19.485 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.485 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.485 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:57:19.485 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:47.237586+0000", 2024-09-20T22:57:19.485 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.008808207 2024-09-20T22:57:19.485 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.485 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.485 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:57:19.485 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:47.237766+0000", 2024-09-20T22:57:19.485 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00018071600000000001 2024-09-20T22:57:19.485 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.485 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.486 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 3,7", 2024-09-20T22:57:19.486 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:47.238007+0000", 2024-09-20T22:57:19.486 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00024016400000000001 2024-09-20T22:57:19.486 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.486 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.486 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:57:19.486 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:47.242507+0000", 2024-09-20T22:57:19.486 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0045000539999999999 2024-09-20T22:57:19.486 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.486 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.486 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:57:19.486 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:47.243911+0000", 2024-09-20T22:57:19.486 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001403856 2024-09-20T22:57:19.486 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.487 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.487 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:57:19.487 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:47.244094+0000", 2024-09-20T22:57:19.487 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000183239 2024-09-20T22:57:19.487 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.487 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.487 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:57:19.487 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:47.244161+0000", 2024-09-20T22:57:19.487 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.7688000000000007e-05 2024-09-20T22:57:19.487 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.487 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.487 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:57:19.487 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:47.244220+0000", 2024-09-20T22:57:19.487 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.8136e-05 2024-09-20T22:57:19.488 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:57:19.488 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:57:19.488 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:57:19.488 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.488 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.488 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:4417 7.6 7:6885a65f:::smithi08477203-14:head [setxattr _header (58) in=7b,truncate 2285884] snapc b7=[b7,b6,b5,b4,b3,b2,b1,b0,af,ae,a9,a7,a6,a5,9e,9c,9a,96,95,87,86] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e446)", 2024-09-20T22:57:19.488 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:48:46.225923+0000", 2024-09-20T22:57:19.488 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 511.87051689399999, 2024-09-20T22:57:19.488 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.021878477, 2024-09-20T22:57:19.488 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:57:19.488 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:57:19.488 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:57:19.488 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:57:19.488 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:57:19.488 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:57:19.489 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 4417 2024-09-20T22:57:19.489 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.489 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:57:19.489 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.489 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:57:19.489 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:46.225923+0000", 2024-09-20T22:57:19.489 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:57:19.489 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.489 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.489 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:57:19.489 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:46.225923+0000", 2024-09-20T22:57:19.489 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:57:19.489 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.489 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.490 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:57:19.490 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:46.225924+0000", 2024-09-20T22:57:19.490 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.8199999999999999e-07 2024-09-20T22:57:19.490 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.490 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.490 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:57:19.490 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:46.225928+0000", 2024-09-20T22:57:19.490 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.2409999999999997e-06 2024-09-20T22:57:19.490 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.490 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.490 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:57:19.490 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:46.225934+0000", 2024-09-20T22:57:19.490 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.353e-06 2024-09-20T22:57:19.490 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.491 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.491 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:57:19.491 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:46.225939+0000", 2024-09-20T22:57:19.491 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.3299999999999998e-06 2024-09-20T22:57:19.491 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.491 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.491 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:57:19.491 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:46.228811+0000", 2024-09-20T22:57:19.491 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0028720199999999999 2024-09-20T22:57:19.491 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.491 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.491 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-09-20T22:57:19.492 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:46.228819+0000", 2024-09-20T22:57:19.492 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.875e-06 2024-09-20T22:57:19.492 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.492 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.492 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:57:19.492 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:47.241348+0000", 2024-09-20T22:57:19.492 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0125285070000001 2024-09-20T22:57:19.492 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.492 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.492 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:57:19.492 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:47.241459+0000", 2024-09-20T22:57:19.492 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000111622 2024-09-20T22:57:19.492 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.493 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.493 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 3,7", 2024-09-20T22:57:19.493 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:47.241661+0000", 2024-09-20T22:57:19.493 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00020146699999999999 2024-09-20T22:57:19.493 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.493 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.493 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:57:19.493 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:47.242513+0000", 2024-09-20T22:57:19.493 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00085188600000000005 2024-09-20T22:57:19.493 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.493 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.493 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:57:19.493 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:47.243955+0000", 2024-09-20T22:57:19.493 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001442313 2024-09-20T22:57:19.494 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.494 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.494 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:57:19.494 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:47.247555+0000", 2024-09-20T22:57:19.494 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0036002529999999999 2024-09-20T22:57:19.494 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.494 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.494 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:57:19.494 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:47.247628+0000", 2024-09-20T22:57:19.494 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.2878000000000003e-05 2024-09-20T22:57:19.494 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.494 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.494 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:57:19.495 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:47.247801+0000", 2024-09-20T22:57:19.495 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00017305000000000001 2024-09-20T22:57:19.495 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:57:19.495 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:57:19.495 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:57:19.495 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.495 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.495 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:4418 7.6 7:6885a65f:::smithi08477203-14:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected+supports_pool_eio e446)", 2024-09-20T22:57:19.495 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:48:46.225948+0000", 2024-09-20T22:57:19.495 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 511.87049201500002, 2024-09-20T22:57:19.495 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.022323377, 2024-09-20T22:57:19.495 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:57:19.495 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:57:19.495 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-20T22:57:19.495 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:57:19.496 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:57:19.496 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:57:19.496 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 4418 2024-09-20T22:57:19.496 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.496 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:57:19.496 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.496 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:57:19.496 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:46.225948+0000", 2024-09-20T22:57:19.496 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:57:19.496 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.496 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.496 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:57:19.496 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:46.225948+0000", 2024-09-20T22:57:19.496 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:57:19.497 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.497 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.497 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:57:19.497 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:46.225948+0000", 2024-09-20T22:57:19.497 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.9999999999999998e-07 2024-09-20T22:57:19.497 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.497 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.497 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:57:19.497 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:46.225951+0000", 2024-09-20T22:57:19.497 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.4499999999999998e-06 2024-09-20T22:57:19.497 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.497 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.497 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:57:19.497 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:46.225956+0000", 2024-09-20T22:57:19.498 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.0309999999999998e-06 2024-09-20T22:57:19.498 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.498 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.498 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:57:19.498 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:46.225961+0000", 2024-09-20T22:57:19.498 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.8359999999999999e-06 2024-09-20T22:57:19.498 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.498 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.498 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:57:19.498 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:46.228848+0000", 2024-09-20T22:57:19.498 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0028873670000000001 2024-09-20T22:57:19.498 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.498 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.499 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for peered", 2024-09-20T22:57:19.499 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:46.228857+0000", 2024-09-20T22:57:19.499 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.6999999999999997e-06 2024-09-20T22:57:19.499 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.499 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.499 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:57:19.499 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:47.242965+0000", 2024-09-20T22:57:19.499 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0141083310000001 2024-09-20T22:57:19.499 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.499 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.499 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for rw locks", 2024-09-20T22:57:19.499 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:47.243076+0000", 2024-09-20T22:57:19.499 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000110945 2024-09-20T22:57:19.499 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.500 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.500 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:57:19.500 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:47.247935+0000", 2024-09-20T22:57:19.500 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0048591049999999998 2024-09-20T22:57:19.500 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.500 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.500 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:57:19.500 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:47.248116+0000", 2024-09-20T22:57:19.500 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000180936 2024-09-20T22:57:19.500 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.500 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.500 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:57:19.500 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:48:47.248271+0000", 2024-09-20T22:57:19.500 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00015517600000000001 2024-09-20T22:57:19.501 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:57:19.501 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:57:19.501 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:57:19.501 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.501 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.501 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:5057 7.9 7:9eac4560:::smithi08477203-30:head [write 548093~787174,stat] snapc da=[da,d9,d8,d7,d6,d5,d4,d3,d1,d0,cf,ce,cd,cc,ca,c7,c6,c5,c2,bf,ba,b4,b2,a7,9e,9c,95,87] ondisk+write+known_if_redirected+supports_pool_eio e508)", 2024-09-20T22:57:19.501 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:49:48.244197+0000", 2024-09-20T22:57:19.501 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 449.852242451, 2024-09-20T22:57:19.501 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.037773875999999998, 2024-09-20T22:57:19.501 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:57:19.501 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:57:19.501 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:57:19.501 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:57:19.501 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:57:19.502 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:57:19.502 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 5057 2024-09-20T22:57:19.502 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.502 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:57:19.502 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.502 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:57:19.502 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:49:48.244197+0000", 2024-09-20T22:57:19.502 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:57:19.502 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.502 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.503 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:57:19.503 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:49:48.244197+0000", 2024-09-20T22:57:19.503 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:57:19.503 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.503 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.503 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:57:19.503 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:49:48.244199+0000", 2024-09-20T22:57:19.503 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.4750000000000001e-06 2024-09-20T22:57:19.503 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.503 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.503 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:57:19.503 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:49:48.244695+0000", 2024-09-20T22:57:19.504 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00049580299999999996 2024-09-20T22:57:19.504 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.504 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.504 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:57:19.504 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:49:48.244712+0000", 2024-09-20T22:57:19.504 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.7e-05 2024-09-20T22:57:19.504 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.504 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.504 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:57:19.504 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:49:48.244720+0000", 2024-09-20T22:57:19.504 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.2099999999999993e-06 2024-09-20T22:57:19.504 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.504 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.504 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:57:19.505 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:49:48.245032+0000", 2024-09-20T22:57:19.505 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00031212699999999999 2024-09-20T22:57:19.505 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.505 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.505 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:57:19.505 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:49:48.245187+0000", 2024-09-20T22:57:19.505 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00015518699999999999 2024-09-20T22:57:19.505 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.505 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.505 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 1,4", 2024-09-20T22:57:19.505 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:49:48.245562+0000", 2024-09-20T22:57:19.505 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00037534899999999997 2024-09-20T22:57:19.505 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.505 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.505 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:57:19.506 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:49:48.250266+0000", 2024-09-20T22:57:19.506 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0047034989999999999 2024-09-20T22:57:19.506 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.506 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.506 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:57:19.506 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:49:48.253360+0000", 2024-09-20T22:57:19.506 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0030940400000000002 2024-09-20T22:57:19.506 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.506 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.506 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:57:19.506 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:49:48.281847+0000", 2024-09-20T22:57:19.506 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.028487350000000002 2024-09-20T22:57:19.506 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.506 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.507 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:57:19.507 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:49:48.281906+0000", 2024-09-20T22:57:19.507 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.8289999999999999e-05 2024-09-20T22:57:19.507 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.507 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.507 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:57:19.507 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:49:48.281971+0000", 2024-09-20T22:57:19.507 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.5545999999999999e-05 2024-09-20T22:57:19.507 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:57:19.507 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:57:19.507 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:57:19.507 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.507 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.507 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:5058 7.9 7:9eac4560:::smithi08477203-30:head [write 1970678~449902] snapc da=[da,d9,d8,d7,d6,d5,d4,d3,d1,d0,cf,ce,cd,cc,ca,c7,c6,c5,c2,bf,ba,b4,b2,a7,9e,9c,95,87] ondisk+write+known_if_redirected+supports_pool_eio e508)", 2024-09-20T22:57:19.508 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:49:48.250601+0000", 2024-09-20T22:57:19.508 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 449.84583840400001, 2024-09-20T22:57:19.508 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.031506248000000001, 2024-09-20T22:57:19.508 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:57:19.508 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:57:19.508 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:57:19.508 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:57:19.508 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:57:19.508 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:57:19.508 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 5058 2024-09-20T22:57:19.508 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.508 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:57:19.508 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.509 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:57:19.509 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:49:48.250601+0000", 2024-09-20T22:57:19.509 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:57:19.509 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.509 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.509 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:57:19.509 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:49:48.250601+0000", 2024-09-20T22:57:19.509 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:57:19.509 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.509 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.509 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:57:19.509 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:49:48.250604+0000", 2024-09-20T22:57:19.509 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.5270000000000001e-06 2024-09-20T22:57:19.509 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.510 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.510 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:57:19.510 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:49:48.250924+0000", 2024-09-20T22:57:19.510 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000320391 2024-09-20T22:57:19.510 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.510 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.510 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:57:19.510 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:49:48.250941+0000", 2024-09-20T22:57:19.510 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.6481999999999999e-05 2024-09-20T22:57:19.510 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.510 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.510 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:57:19.510 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:49:48.250951+0000", 2024-09-20T22:57:19.510 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.9280000000000005e-06 2024-09-20T22:57:19.511 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.511 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.511 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:57:19.511 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:49:48.251037+0000", 2024-09-20T22:57:19.511 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.6310999999999994e-05 2024-09-20T22:57:19.511 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.511 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.511 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:57:19.511 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:49:48.251176+0000", 2024-09-20T22:57:19.511 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000139505 2024-09-20T22:57:19.511 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.511 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.511 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 1,4", 2024-09-20T22:57:19.512 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:49:48.251397+0000", 2024-09-20T22:57:19.512 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00022060299999999999 2024-09-20T22:57:19.512 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.512 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.512 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:57:19.512 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:49:48.254904+0000", 2024-09-20T22:57:19.512 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0035071989999999999 2024-09-20T22:57:19.512 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.512 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.512 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:57:19.512 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:49:48.255361+0000", 2024-09-20T22:57:19.513 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00045642300000000002 2024-09-20T22:57:19.513 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.513 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.513 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:57:19.513 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:49:48.282027+0000", 2024-09-20T22:57:19.513 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.026665852 2024-09-20T22:57:19.513 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.513 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.513 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:57:19.513 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:49:48.282069+0000", 2024-09-20T22:57:19.513 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.2778000000000003e-05 2024-09-20T22:57:19.513 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.513 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.513 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:57:19.513 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:49:48.282108+0000", 2024-09-20T22:57:19.514 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.8248999999999999e-05 2024-09-20T22:57:19.514 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:57:19.514 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:57:19.514 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:57:19.514 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.514 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.514 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:5412 7.6 7:64bcd2fc:::smithi08477203-20:head [copy-from ver 794] snapc ef=[ef,ed,ec,eb,ea,e8,e7,e6,e5,e4,e2,df,db,da,d7,d5,d4,d3,d0,cf,ce,ca,c5,c2,bf,b4,b2,a7,95] ondisk+write+known_if_redirected+supports_pool_eio e549)", 2024-09-20T22:57:19.514 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:50:29.255763+0000", 2024-09-20T22:57:19.514 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 408.84067678899999, 2024-09-20T22:57:19.514 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.032860671000000001, 2024-09-20T22:57:19.514 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:57:19.514 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:57:19.514 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:57:19.515 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:57:19.515 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:57:19.515 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:57:19.515 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 5412 2024-09-20T22:57:19.515 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.515 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:57:19.515 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.515 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:57:19.515 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:29.255763+0000", 2024-09-20T22:57:19.515 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:57:19.515 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.515 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.515 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:57:19.515 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:29.255763+0000", 2024-09-20T22:57:19.515 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:57:19.516 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.516 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.516 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:57:19.516 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:29.255764+0000", 2024-09-20T22:57:19.516 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.11e-06 2024-09-20T22:57:19.516 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.516 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.516 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:57:19.516 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:29.255768+0000", 2024-09-20T22:57:19.516 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.019e-06 2024-09-20T22:57:19.516 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.516 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.517 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:57:19.517 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:29.255777+0000", 2024-09-20T22:57:19.517 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.5539999999999998e-06 2024-09-20T22:57:19.517 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.517 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.517 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:57:19.517 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:29.255782+0000", 2024-09-20T22:57:19.517 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.9490000000000004e-06 2024-09-20T22:57:19.517 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.517 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.517 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:57:19.517 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:29.255859+0000", 2024-09-20T22:57:19.517 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.7751999999999997e-05 2024-09-20T22:57:19.517 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.517 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.518 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:57:19.518 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:29.256005+0000", 2024-09-20T22:57:19.518 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014578799999999999 2024-09-20T22:57:19.518 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.518 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.518 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 3,7", 2024-09-20T22:57:19.518 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:29.273220+0000", 2024-09-20T22:57:19.518 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.017214808000000002 2024-09-20T22:57:19.518 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.518 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.518 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:57:19.518 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:29.286455+0000", 2024-09-20T22:57:19.518 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013235232 2024-09-20T22:57:19.518 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.519 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.519 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:57:19.519 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:29.286879+0000", 2024-09-20T22:57:19.519 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00042428099999999999 2024-09-20T22:57:19.519 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.519 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.519 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:57:19.519 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:29.288446+0000", 2024-09-20T22:57:19.519 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0015662810000000001 2024-09-20T22:57:19.519 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.519 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.519 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:57:19.519 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:29.288516+0000", 2024-09-20T22:57:19.519 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.0321000000000007e-05 2024-09-20T22:57:19.519 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.520 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.520 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:57:19.520 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:29.288624+0000", 2024-09-20T22:57:19.520 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000107576 2024-09-20T22:57:19.520 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:57:19.520 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:57:19.520 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:57:19.520 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.520 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.520 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:5413 7.6 7:64bcd2fc:::smithi08477203-20:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e549)", 2024-09-20T22:57:19.520 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:50:29.255802+0000", 2024-09-20T22:57:19.520 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 408.84063744700001, 2024-09-20T22:57:19.520 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.024182717999999999, 2024-09-20T22:57:19.520 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:57:19.520 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:57:19.520 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-20T22:57:19.521 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:57:19.521 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:57:19.521 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:57:19.521 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 5413 2024-09-20T22:57:19.521 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.521 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:57:19.521 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.521 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:57:19.521 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:29.255802+0000", 2024-09-20T22:57:19.521 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:57:19.521 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.521 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.521 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:57:19.521 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:29.255802+0000", 2024-09-20T22:57:19.521 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:57:19.521 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.522 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.522 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:57:19.522 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:29.255803+0000", 2024-09-20T22:57:19.522 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.8800000000000003e-07 2024-09-20T22:57:19.522 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.522 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.522 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:57:19.522 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:29.255805+0000", 2024-09-20T22:57:19.522 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.5950000000000001e-06 2024-09-20T22:57:19.522 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.522 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.522 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:57:19.522 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:29.255814+0000", 2024-09-20T22:57:19.523 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.0480000000000004e-06 2024-09-20T22:57:19.523 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.523 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.523 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:57:19.523 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:29.255820+0000", 2024-09-20T22:57:19.523 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.57e-06 2024-09-20T22:57:19.523 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.523 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.523 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:57:19.523 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:29.256223+0000", 2024-09-20T22:57:19.523 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00040313099999999999 2024-09-20T22:57:19.523 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.523 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.524 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for blocked object", 2024-09-20T22:57:19.524 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:29.256281+0000", 2024-09-20T22:57:19.524 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.7679000000000001e-05 2024-09-20T22:57:19.524 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.524 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.524 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:57:19.524 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:29.279736+0000", 2024-09-20T22:57:19.524 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.023454900000000001 2024-09-20T22:57:19.524 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.524 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.524 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:57:19.524 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:29.279857+0000", 2024-09-20T22:57:19.524 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000121401 2024-09-20T22:57:19.524 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.524 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.525 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:57:19.525 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:50:29.279985+0000", 2024-09-20T22:57:19.525 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000127906 2024-09-20T22:57:19.525 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:57:19.525 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:57:19.525 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:57:19.525 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.525 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.525 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:6018 7.4 7:24b1a612:::smithi08477203-43:head [copy-from ver 846] snapc 108=[108,107,105,104,103,102,101,ff,fe,fd,fc,fb,fa,f8,f5,f2,ed,ec,ea,df,da,d7,c2,bf] ondisk+write+known_if_redirected+supports_pool_eio e606)", 2024-09-20T22:57:19.525 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:51:26.429397+0000", 2024-09-20T22:57:19.526 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 351.667042256, 2024-09-20T22:57:19.526 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.034058235999999999, 2024-09-20T22:57:19.526 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:57:19.526 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:57:19.526 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:57:19.526 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:57:19.526 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:57:19.526 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:57:19.526 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 6018 2024-09-20T22:57:19.526 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.526 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:57:19.526 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.526 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:57:19.526 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:26.429397+0000", 2024-09-20T22:57:19.527 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:57:19.527 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.527 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.527 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:57:19.527 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:26.429397+0000", 2024-09-20T22:57:19.527 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:57:19.527 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.527 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.527 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:57:19.527 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:26.429399+0000", 2024-09-20T22:57:19.527 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.824e-06 2024-09-20T22:57:19.527 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.527 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.527 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:57:19.528 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:26.429408+0000", 2024-09-20T22:57:19.528 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.1770000000000004e-06 2024-09-20T22:57:19.528 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.528 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.528 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:57:19.528 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:26.429422+0000", 2024-09-20T22:57:19.528 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3574000000000001e-05 2024-09-20T22:57:19.528 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.528 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.528 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:57:19.528 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:26.429430+0000", 2024-09-20T22:57:19.528 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.8550000000000002e-06 2024-09-20T22:57:19.528 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.528 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.528 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:57:19.529 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:26.429492+0000", 2024-09-20T22:57:19.529 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.2433000000000003e-05 2024-09-20T22:57:19.529 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.529 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.529 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:57:19.529 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:26.429577+0000", 2024-09-20T22:57:19.529 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.5018999999999994e-05 2024-09-20T22:57:19.529 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.529 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.529 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 4,6", 2024-09-20T22:57:19.529 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:26.433580+0000", 2024-09-20T22:57:19.529 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00400264 2024-09-20T22:57:19.529 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.530 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.530 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:57:19.530 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:26.437548+0000", 2024-09-20T22:57:19.530 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0039683740000000002 2024-09-20T22:57:19.530 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.530 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.530 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:57:19.530 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:26.440702+0000", 2024-09-20T22:57:19.530 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0031540990000000001 2024-09-20T22:57:19.530 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.530 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.530 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:57:19.530 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:26.463373+0000", 2024-09-20T22:57:19.530 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.022670613999999999 2024-09-20T22:57:19.531 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.531 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.531 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:57:19.531 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:26.463418+0000", 2024-09-20T22:57:19.531 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.5046999999999997e-05 2024-09-20T22:57:19.531 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.531 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.531 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:57:19.531 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:26.463456+0000", 2024-09-20T22:57:19.531 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.7580000000000003e-05 2024-09-20T22:57:19.531 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:57:19.531 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:57:19.531 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:57:19.531 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.532 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.532 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:6253 7.9 7:91f2d47c:::smithi08477203-8:head [copy-from ver 1797] snapc 114=[114,113,112,111,110,10e,10c,10b,108,107,105,104,103,101,fd,fb,fa,f8,f5,f2,ed,ec,d7,c2,bf] ondisk+write+known_if_redirected+supports_pool_eio e629)", 2024-09-20T22:57:19.532 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:51:49.618540+0000", 2024-09-20T22:57:19.532 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 328.47789960699998, 2024-09-20T22:57:19.532 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.024931412999999999, 2024-09-20T22:57:19.532 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:57:19.532 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:57:19.532 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:57:19.532 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:57:19.532 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:57:19.532 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:57:19.532 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 6253 2024-09-20T22:57:19.533 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.533 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:57:19.533 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.533 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:57:19.533 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:49.618540+0000", 2024-09-20T22:57:19.533 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:57:19.533 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.533 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.533 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:57:19.533 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:49.618540+0000", 2024-09-20T22:57:19.533 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:57:19.534 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.534 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.534 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:57:19.534 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:49.618542+0000", 2024-09-20T22:57:19.534 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.637e-06 2024-09-20T22:57:19.534 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.534 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.534 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:57:19.534 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:49.618548+0000", 2024-09-20T22:57:19.534 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.088e-06 2024-09-20T22:57:19.534 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.534 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.534 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:57:19.534 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:49.618559+0000", 2024-09-20T22:57:19.535 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1422e-05 2024-09-20T22:57:19.535 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.535 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.535 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:57:19.535 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:49.618565+0000", 2024-09-20T22:57:19.535 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.3909999999999997e-06 2024-09-20T22:57:19.535 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.535 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.535 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:57:19.535 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:49.618642+0000", 2024-09-20T22:57:19.535 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.737e-05 2024-09-20T22:57:19.535 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.535 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.535 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:57:19.536 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:49.618716+0000", 2024-09-20T22:57:19.536 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.4103000000000005e-05 2024-09-20T22:57:19.536 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.536 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.536 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 4,7", 2024-09-20T22:57:19.536 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:49.625475+0000", 2024-09-20T22:57:19.536 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0067586299999999998 2024-09-20T22:57:19.536 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.536 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.536 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:57:19.536 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:49.633603+0000", 2024-09-20T22:57:19.536 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0081277929999999995 2024-09-20T22:57:19.536 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.536 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.536 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:57:19.537 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:49.643271+0000", 2024-09-20T22:57:19.537 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0096679839999999993 2024-09-20T22:57:19.537 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.537 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.537 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:57:19.537 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:49.643392+0000", 2024-09-20T22:57:19.537 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000121734 2024-09-20T22:57:19.537 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.537 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.537 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:57:19.537 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:49.643427+0000", 2024-09-20T22:57:19.537 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.5138999999999997e-05 2024-09-20T22:57:19.537 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.538 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.538 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:57:19.538 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:49.643472+0000", 2024-09-20T22:57:19.538 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.4122000000000002e-05 2024-09-20T22:57:19.538 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:57:19.538 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:57:19.538 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:57:19.538 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.538 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.538 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:7387 7.9 7:907aec0e:::smithi08477203-3:head [write 692719~403182,stat] snapc 14e=[14e,14c,14a,149,148,145,142,141,13e,13d,13c,13b,13a,139,135,132,130,12f,12e,12c,12a,128,127,124,121,11f,112,103] ondisk+write+known_if_redirected+supports_pool_eio e743)", 2024-09-20T22:57:19.538 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:53:43.308708+0000", 2024-09-20T22:57:19.539 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 214.78773132800001, 2024-09-20T22:57:19.539 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.03359815, 2024-09-20T22:57:19.539 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:57:19.539 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:57:19.539 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:57:19.539 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:57:19.539 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:57:19.539 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:57:19.539 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 7387 2024-09-20T22:57:19.539 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.539 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:57:19.539 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.539 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:57:19.540 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:43.308708+0000", 2024-09-20T22:57:19.540 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:57:19.540 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.540 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.540 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:57:19.540 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:43.308708+0000", 2024-09-20T22:57:19.540 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:57:19.540 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.540 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.540 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:57:19.540 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:43.308710+0000", 2024-09-20T22:57:19.540 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.68e-06 2024-09-20T22:57:19.540 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.540 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.541 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:57:19.541 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:43.308941+0000", 2024-09-20T22:57:19.541 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000231277 2024-09-20T22:57:19.541 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.541 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.541 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:57:19.541 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:43.308954+0000", 2024-09-20T22:57:19.541 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.29e-05 2024-09-20T22:57:19.541 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.541 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.541 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:57:19.541 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:43.308961+0000", 2024-09-20T22:57:19.541 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.7490000000000001e-06 2024-09-20T22:57:19.541 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.541 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.542 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:57:19.542 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:43.309015+0000", 2024-09-20T22:57:19.542 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.3869999999999998e-05 2024-09-20T22:57:19.542 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.542 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.542 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:57:19.542 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:43.309117+0000", 2024-09-20T22:57:19.542 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00010163800000000001 2024-09-20T22:57:19.542 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.542 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.542 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 4,7", 2024-09-20T22:57:19.542 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:43.309296+0000", 2024-09-20T22:57:19.542 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00017921199999999999 2024-09-20T22:57:19.543 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.543 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.543 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:57:19.543 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:43.312537+0000", 2024-09-20T22:57:19.543 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0032413870000000001 2024-09-20T22:57:19.543 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.543 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.543 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:57:19.543 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:43.340218+0000", 2024-09-20T22:57:19.543 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.027680986000000001 2024-09-20T22:57:19.543 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.543 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.543 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:57:19.544 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:43.342189+0000", 2024-09-20T22:57:19.544 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0019712670000000001 2024-09-20T22:57:19.544 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.544 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.544 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:57:19.544 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:43.342233+0000", 2024-09-20T22:57:19.544 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.3676e-05 2024-09-20T22:57:19.544 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.544 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.544 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:57:19.544 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:43.342307+0000", 2024-09-20T22:57:19.544 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.3508000000000002e-05 2024-09-20T22:57:19.544 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:57:19.544 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:57:19.545 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:57:19.545 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.545 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.545 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:7388 7.9 7:907aec0e:::smithi08477203-3:head [write 1579246~720684] snapc 14e=[14e,14c,14a,149,148,145,142,141,13e,13d,13c,13b,13a,139,135,132,130,12f,12e,12c,12a,128,127,124,121,11f,112,103] ondisk+write+known_if_redirected+supports_pool_eio e743)", 2024-09-20T22:57:19.545 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:53:43.316061+0000", 2024-09-20T22:57:19.545 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 214.78037827099999, 2024-09-20T22:57:19.545 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.026368321, 2024-09-20T22:57:19.545 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:57:19.545 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:57:19.545 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:57:19.545 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:57:19.545 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:57:19.545 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:57:19.545 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 7388 2024-09-20T22:57:19.545 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.546 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:57:19.546 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.546 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:57:19.546 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:43.316061+0000", 2024-09-20T22:57:19.546 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:57:19.546 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.546 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.546 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:57:19.546 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:43.316061+0000", 2024-09-20T22:57:19.546 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:57:19.546 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.546 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.546 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:57:19.547 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:43.316063+0000", 2024-09-20T22:57:19.547 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.181e-06 2024-09-20T22:57:19.547 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.547 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.547 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:57:19.547 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:43.316639+0000", 2024-09-20T22:57:19.547 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00057640100000000004 2024-09-20T22:57:19.547 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.547 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.547 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:57:19.547 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:43.316649+0000", 2024-09-20T22:57:19.547 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.8479999999999996e-06 2024-09-20T22:57:19.547 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.547 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.547 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:57:19.548 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:43.316654+0000", 2024-09-20T22:57:19.548 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.3260000000000003e-06 2024-09-20T22:57:19.548 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.548 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.548 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:57:19.548 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:43.316698+0000", 2024-09-20T22:57:19.548 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.3643999999999998e-05 2024-09-20T22:57:19.548 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.548 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.548 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:57:19.548 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:43.316771+0000", 2024-09-20T22:57:19.548 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.2750000000000007e-05 2024-09-20T22:57:19.548 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.548 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.548 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 4,7", 2024-09-20T22:57:19.548 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:43.316869+0000", 2024-09-20T22:57:19.549 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.8845999999999995e-05 2024-09-20T22:57:19.549 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.549 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.549 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:57:19.549 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:43.319492+0000", 2024-09-20T22:57:19.549 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0026225520000000002 2024-09-20T22:57:19.549 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.549 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.549 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:57:19.549 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:43.341085+0000", 2024-09-20T22:57:19.549 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.021593131000000002 2024-09-20T22:57:19.549 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.549 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.549 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:57:19.549 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:43.342350+0000", 2024-09-20T22:57:19.549 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0012649219999999999 2024-09-20T22:57:19.550 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.550 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.550 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:57:19.550 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:43.342389+0000", 2024-09-20T22:57:19.550 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.9257e-05 2024-09-20T22:57:19.550 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.550 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.550 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:57:19.550 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:43.342430+0000", 2024-09-20T22:57:19.550 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.0463000000000002e-05 2024-09-20T22:57:19.550 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:57:19.550 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:57:19.550 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:57:19.550 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.550 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.551 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:8049 7.9 7:907aec0e:::smithi08477203-3:head [copy-from ver 2542] snapc 171=[171,16f,16e,16d,16c,16b,167,166,165,160,15f,15e,15c,15a,158,157,156,155,153,14f,14e,14a,149,145,142,13e,13a,139,12f,12e,124] ondisk+write+known_if_redirected+supports_pool_eio e810)", 2024-09-20T22:57:19.551 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:54:50.339331+0000", 2024-09-20T22:57:19.551 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 147.75710833900001, 2024-09-20T22:57:19.551 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.033993001000000002, 2024-09-20T22:57:19.551 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:57:19.551 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:57:19.551 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:57:19.551 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:57:19.551 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:57:19.551 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:57:19.551 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 8049 2024-09-20T22:57:19.551 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.551 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:57:19.551 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.551 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:57:19.552 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:54:50.339331+0000", 2024-09-20T22:57:19.552 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:57:19.552 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.552 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.552 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:57:19.552 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:54:50.339331+0000", 2024-09-20T22:57:19.552 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:57:19.552 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.552 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.552 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:57:19.552 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:54:50.339333+0000", 2024-09-20T22:57:19.552 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.813e-06 2024-09-20T22:57:19.552 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.552 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.552 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:57:19.553 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:54:50.339342+0000", 2024-09-20T22:57:19.553 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.7839999999999992e-06 2024-09-20T22:57:19.553 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.553 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.553 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:57:19.553 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:54:50.339358+0000", 2024-09-20T22:57:19.553 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.5679999999999999e-05 2024-09-20T22:57:19.553 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.553 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.553 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:57:19.553 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:54:50.339367+0000", 2024-09-20T22:57:19.553 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.8300000000000008e-06 2024-09-20T22:57:19.553 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.553 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.553 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:57:19.554 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:54:50.339484+0000", 2024-09-20T22:57:19.554 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000116995 2024-09-20T22:57:19.554 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.554 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.554 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:57:19.554 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:54:50.339592+0000", 2024-09-20T22:57:19.554 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00010728799999999999 2024-09-20T22:57:19.554 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.554 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.554 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 1,4", 2024-09-20T22:57:19.554 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:54:50.354559+0000", 2024-09-20T22:57:19.554 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014967045999999999 2024-09-20T22:57:19.555 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.555 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.555 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:57:19.555 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:54:50.365388+0000", 2024-09-20T22:57:19.555 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010828786 2024-09-20T22:57:19.555 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.555 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.555 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:57:19.555 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:54:50.373126+0000", 2024-09-20T22:57:19.555 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0077383169999999998 2024-09-20T22:57:19.555 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.555 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.556 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:57:19.556 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:54:50.373212+0000", 2024-09-20T22:57:19.556 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.6484999999999996e-05 2024-09-20T22:57:19.556 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.556 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.556 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:57:19.556 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:54:50.373269+0000", 2024-09-20T22:57:19.557 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.6579999999999997e-05 2024-09-20T22:57:19.557 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.557 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.557 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:57:19.557 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:54:50.373324+0000", 2024-09-20T22:57:19.557 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.5396999999999998e-05 2024-09-20T22:57:19.557 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:57:19.558 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:57:19.558 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:57:19.558 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.558 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.558 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:8050 7.9 7:907aec0e:::smithi08477203-3:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e810)", 2024-09-20T22:57:19.558 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:54:50.339405+0000", 2024-09-20T22:57:19.559 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 147.75703496599999, 2024-09-20T22:57:19.559 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.023690017000000001, 2024-09-20T22:57:19.559 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:57:19.559 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:57:19.559 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-20T22:57:19.559 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:57:19.560 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:57:19.560 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:57:19.560 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 8050 2024-09-20T22:57:19.560 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.560 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:57:19.560 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.560 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:57:19.561 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:54:50.339405+0000", 2024-09-20T22:57:19.561 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:57:19.561 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.561 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.561 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:57:19.561 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:54:50.339405+0000", 2024-09-20T22:57:19.562 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:57:19.562 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.562 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.562 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:57:19.562 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:54:50.339405+0000", 2024-09-20T22:57:19.562 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.5700000000000002e-07 2024-09-20T22:57:19.562 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.563 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.563 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:57:19.563 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:54:50.339408+0000", 2024-09-20T22:57:19.563 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.072e-06 2024-09-20T22:57:19.563 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.563 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.564 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:57:19.564 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:54:50.339419+0000", 2024-09-20T22:57:19.564 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0681e-05 2024-09-20T22:57:19.564 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.564 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.564 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:57:19.565 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:54:50.339425+0000", 2024-09-20T22:57:19.565 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.2169999999999999e-06 2024-09-20T22:57:19.565 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.565 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.565 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:57:19.565 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:54:50.341917+0000", 2024-09-20T22:57:19.565 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0024919019999999998 2024-09-20T22:57:19.566 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.566 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.566 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for blocked object", 2024-09-20T22:57:19.566 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:54:50.341968+0000", 2024-09-20T22:57:19.566 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.0376e-05 2024-09-20T22:57:19.566 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.567 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.567 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:57:19.567 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:54:50.362858+0000", 2024-09-20T22:57:19.567 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.020889973999999999 2024-09-20T22:57:19.567 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.567 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.568 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:57:19.568 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:54:50.362972+0000", 2024-09-20T22:57:19.568 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000114292 2024-09-20T22:57:19.568 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:19.568 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:19.568 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:57:19.568 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:54:50.363095+0000", 2024-09-20T22:57:19.569 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000122946 2024-09-20T22:57:19.569 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:57:19.569 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:57:19.569 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:57:19.569 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:57:19.569 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:57:19.570 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-20T22:57:19.570 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:57:19.594 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:19.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:19.595 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:57:19.696 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:57:19.708 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:19 smithi084 ceph-mon[40024]: pgmap v1338: 168 pgs: 1 active+clean+snaptrim, 2 active+undersized+degraded+remapped+backfill_toofull, 41 active+undersized, 1 active+remapped+backfill_toofull, 36 active+undersized+degraded, 87 active+clean; 810 MiB data, 2.3 GiB used, 624 GiB / 626 GiB avail; 290/1683 objects degraded (17.231%); 55/1683 objects misplaced (3.268%) 2024-09-20T22:57:19.708 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:19 smithi084 ceph-mon[40024]: osdmap e959: 8 total, 7 up, 7 in 2024-09-20T22:57:19.708 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:19 smithi084 ceph-mon[29774]: pgmap v1338: 168 pgs: 1 active+clean+snaptrim, 2 active+undersized+degraded+remapped+backfill_toofull, 41 active+undersized, 1 active+remapped+backfill_toofull, 36 active+undersized+degraded, 87 active+clean; 810 MiB data, 2.3 GiB used, 624 GiB / 626 GiB avail; 290/1683 objects degraded (17.231%); 55/1683 objects misplaced (3.268%) 2024-09-20T22:57:19.708 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:19 smithi084 ceph-mon[29774]: osdmap e959: 8 total, 7 up, 7 in 2024-09-20T22:57:19.720 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:19.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:19.721 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:57:19.783 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:19 smithi154 ceph-mon[33264]: pgmap v1338: 168 pgs: 1 active+clean+snaptrim, 2 active+undersized+degraded+remapped+backfill_toofull, 41 active+undersized, 1 active+remapped+backfill_toofull, 36 active+undersized+degraded, 87 active+clean; 810 MiB data, 2.3 GiB used, 624 GiB / 626 GiB avail; 290/1683 objects degraded (17.231%); 55/1683 objects misplaced (3.268%) 2024-09-20T22:57:19.783 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:19 smithi154 ceph-mon[33264]: osdmap e959: 8 total, 7 up, 7 in 2024-09-20T22:57:19.822 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:57:19.843 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:19.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:19.844 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:57:19.945 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:57:19.971 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:19.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:19.972 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:57:20.072 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:57:20.097 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:20.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:20.098 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:57:20.154 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-20T22:57:20.198 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:57:20.224 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:20.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:20.225 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:57:20.254 INFO:journalctl@ceph.iscsi.iscsi.a.smithi154.stdout:Sep 20 22:57:19 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-iscsi-iscsi-a[59730]: debug there is no tcmu-runner data available 2024-09-20T22:57:20.327 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:57:20.352 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:20.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:20.353 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:57:20.372 INFO:tasks.rados.rados.0.smithi084.stdout:3879: done (2 left) 2024-09-20T22:57:20.372 INFO:tasks.rados.rados.0.smithi084.stdout:3880: write oid 7 current snap is 455 2024-09-20T22:57:20.372 INFO:tasks.rados.rados.0.smithi084.stdout:3880: seq_num 1320 ranges {585240=657749,1743290=422125} 2024-09-20T22:57:20.377 INFO:tasks.rados.rados.0.smithi084.stdout:3880: writing smithi08477203-7 from 585240 to 1242989 tid 1 2024-09-20T22:57:20.384 INFO:tasks.rados.rados.0.smithi084.stdout:3880: writing smithi08477203-7 from 1743290 to 2165415 tid 2 2024-09-20T22:57:20.384 INFO:tasks.rados.rados.0.smithi084.stdout:3881: read oid 36 snap -1 2024-09-20T22:57:20.384 INFO:tasks.rados.rados.0.smithi084.stdout:3881: expect (ObjNum 1314 snap 453 seq_num 1314) 2024-09-20T22:57:20.384 INFO:tasks.rados.rados.0.smithi084.stdout:3882: rollback oid 48 current snap is 455 2024-09-20T22:57:20.384 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 48 to 410 2024-09-20T22:57:20.384 INFO:tasks.rados.rados.0.smithi084.stdout:3877: finishing write tid 2 to smithi08477203-19 2024-09-20T22:57:20.385 INFO:tasks.rados.rados.0.smithi084.stdout:3877: finishing write tid 3 to smithi08477203-19 2024-09-20T22:57:20.385 INFO:tasks.rados.rados.0.smithi084.stdout:3877: finishing write tid 4 to smithi08477203-19 2024-09-20T22:57:20.385 INFO:tasks.rados.rados.0.smithi084.stdout:3877: oid 19 updating version 0 to 3137 2024-09-20T22:57:20.385 INFO:tasks.rados.rados.0.smithi084.stdout:3877: oid 19 version 3137 is already newer than 3135 2024-09-20T22:57:20.385 INFO:tasks.rados.rados.0.smithi084.stdout:3877: oid 19 version 3137 is already newer than 3136 2024-09-20T22:57:20.385 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 19 v 3137 (ObjNum 1318 snap 455 seq_num 1318) dirty exists 2024-09-20T22:57:20.385 INFO:tasks.rados.rados.0.smithi084.stdout:3877: left oid 19 (ObjNum 1318 snap 455 seq_num 1318) 2024-09-20T22:57:20.385 INFO:tasks.rados.rados.0.smithi084.stdout:3878: finishing write tid 1 to smithi08477203-43 2024-09-20T22:57:20.385 INFO:tasks.rados.rados.0.smithi084.stdout:3878: finishing write tid 2 to smithi08477203-43 2024-09-20T22:57:20.385 INFO:tasks.rados.rados.0.smithi084.stdout:3877: done (4 left) 2024-09-20T22:57:20.385 INFO:tasks.rados.rados.0.smithi084.stdout:3883: read oid 40 snap -1 2024-09-20T22:57:20.385 INFO:tasks.rados.rados.0.smithi084.stdout:3883: expect (ObjNum 1285 snap 436 seq_num 1285) 2024-09-20T22:57:20.385 INFO:tasks.rados.rados.0.smithi084.stdout:3878: finishing write tid 3 to smithi08477203-43 2024-09-20T22:57:20.385 INFO:tasks.rados.rados.0.smithi084.stdout:3878: finishing write tid 4 to smithi08477203-43 2024-09-20T22:57:20.386 INFO:tasks.rados.rados.0.smithi084.stdout:3878: oid 43 updating version 0 to 1954 2024-09-20T22:57:20.386 INFO:tasks.rados.rados.0.smithi084.stdout:3878: oid 43 updating version 1954 to 1955 2024-09-20T22:57:20.386 INFO:tasks.rados.rados.0.smithi084.stdout:3878: oid 43 version 1955 is already newer than 1953 2024-09-20T22:57:20.386 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 43 v 1955 (ObjNum 1319 snap 455 seq_num 1319) dirty exists 2024-09-20T22:57:20.386 INFO:tasks.rados.rados.0.smithi084.stdout:3878: left oid 43 (ObjNum 1319 snap 455 seq_num 1319) 2024-09-20T22:57:20.386 INFO:tasks.rados.rados.0.smithi084.stdout:3880: finishing write tid 1 to smithi08477203-7 2024-09-20T22:57:20.386 INFO:tasks.rados.rados.0.smithi084.stdout:3884: read oid 19 snap -1 2024-09-20T22:57:20.386 INFO:tasks.rados.rados.0.smithi084.stdout:3884: expect (ObjNum 1318 snap 455 seq_num 1318) 2024-09-20T22:57:20.386 INFO:tasks.rados.rados.0.smithi084.stdout:3878: done (5 left) 2024-09-20T22:57:20.386 INFO:tasks.rados.rados.0.smithi084.stdout:3885: write oid 41 current snap is 455 2024-09-20T22:57:20.386 INFO:tasks.rados.rados.0.smithi084.stdout:3885: seq_num 1321 ranges {602047=732998,1801772=400275} 2024-09-20T22:57:20.391 INFO:tasks.rados.rados.0.smithi084.stdout:3885: writing smithi08477203-41 from 602047 to 1335045 tid 1 2024-09-20T22:57:20.396 INFO:tasks.rados.rados.0.smithi084.stdout:3885: writing smithi08477203-41 from 1801772 to 2202047 tid 2 2024-09-20T22:57:20.396 INFO:tasks.rados.rados.0.smithi084.stdout:3886: read oid 30 snap -1 2024-09-20T22:57:20.396 INFO:tasks.rados.rados.0.smithi084.stdout:3886: expect deleted 2024-09-20T22:57:20.397 INFO:tasks.rados.rados.0.smithi084.stdout:3887: read oid 47 snap -1 2024-09-20T22:57:20.397 INFO:tasks.rados.rados.0.smithi084.stdout:3887: expect deleted 2024-09-20T22:57:20.397 INFO:tasks.rados.rados.0.smithi084.stdout:3888: read oid 8 snap -1 2024-09-20T22:57:20.397 INFO:tasks.rados.rados.0.smithi084.stdout:3888: expect (ObjNum 1205 snap 412 seq_num 1205) 2024-09-20T22:57:20.397 INFO:tasks.rados.rados.0.smithi084.stdout:3889: read oid 1 snap 449 2024-09-20T22:57:20.397 INFO:tasks.rados.rados.0.smithi084.stdout:3889: expect (ObjNum 1293 snap 439 seq_num 1293) 2024-09-20T22:57:20.397 INFO:tasks.rados.rados.0.smithi084.stdout:3890: rollback oid 10 current snap is 455 2024-09-20T22:57:20.397 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 10 to 434 2024-09-20T22:57:20.397 INFO:tasks.rados.rados.0.smithi084.stdout:3891: delete oid 45 current snap is 455 2024-09-20T22:57:20.397 INFO:tasks.rados.rados.0.smithi084.stdout:3880: finishing write tid 2 to smithi08477203-7 2024-09-20T22:57:20.397 INFO:tasks.rados.rados.0.smithi084.stdout:3880: finishing write tid 3 to smithi08477203-7 2024-09-20T22:57:20.397 INFO:tasks.rados.rados.0.smithi084.stdout:3880: finishing write tid 4 to smithi08477203-7 2024-09-20T22:57:20.397 INFO:tasks.rados.rados.0.smithi084.stdout:3880: oid 7 updating version 0 to 1960 2024-09-20T22:57:20.397 INFO:tasks.rados.rados.0.smithi084.stdout:3880: oid 7 updating version 1960 to 1961 2024-09-20T22:57:20.398 INFO:tasks.rados.rados.0.smithi084.stdout:3880: oid 7 updating version 1961 to 1962 2024-09-20T22:57:20.398 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 7 v 1962 (ObjNum 1320 snap 455 seq_num 1320) dirty exists 2024-09-20T22:57:20.398 INFO:tasks.rados.rados.0.smithi084.stdout:3880: left oid 7 (ObjNum 1320 snap 455 seq_num 1320) 2024-09-20T22:57:20.398 INFO:tasks.rados.rados.0.smithi084.stdout:3882: finishing rollback tid 0 to smithi08477203-48 2024-09-20T22:57:20.398 INFO:tasks.rados.rados.0.smithi084.stdout:3883: expect (ObjNum 1285 snap 436 seq_num 1285) 2024-09-20T22:57:20.454 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:57:20.478 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:20.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:20.479 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:57:20.579 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:57:20.602 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:20.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:20.603 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:57:20.616 INFO:tasks.rados.rados.0.smithi084.stdout:3882: finishing rollback tid 1 to smithi08477203-48 2024-09-20T22:57:20.616 INFO:tasks.rados.rados.0.smithi084.stdout:3882: finishing rollback tid 2 to smithi08477203-48 2024-09-20T22:57:20.616 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 48 v 1965 (ObjNum 1141 snap 385 seq_num 1141) dirty exists 2024-09-20T22:57:20.616 INFO:tasks.rados.rados.0.smithi084.stdout:3880: done (11 left) 2024-09-20T22:57:20.616 INFO:tasks.rados.rados.0.smithi084.stdout:3882: done (10 left) 2024-09-20T22:57:20.616 INFO:tasks.rados.rados.0.smithi084.stdout:3883: done (9 left) 2024-09-20T22:57:20.616 INFO:tasks.rados.rados.0.smithi084.stdout:3891: done (8 left) 2024-09-20T22:57:20.616 INFO:tasks.rados.rados.0.smithi084.stdout:3892: snap_remove snap 401 2024-09-20T22:57:20.704 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:57:20.725 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:20.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:20.726 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:57:20.775 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-20T22:57:20.775 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-20T22:57:20.827 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:57:20.848 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:20.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:20.849 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:57:20.950 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:57:20.972 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:20.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:20.973 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:57:21.074 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:57:21.095 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:21.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:21.096 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:57:21.197 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:57:21.220 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:21.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:21.221 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:57:21.221 INFO:teuthology.orchestra.run.smithi084.stderr:Creating an OSD daemon form without an OSD FSID value 2024-09-20T22:57:21.322 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:57:21.344 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:21.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:21.345 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:57:21.446 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:57:21.467 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:21 smithi084 ceph-mon[40024]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:57:21.467 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:21 smithi084 ceph-mon[40024]: pgmap v1341: 168 pgs: 1 active+clean+snaptrim, 2 active+undersized+degraded+remapped+backfill_toofull, 41 active+undersized, 1 active+remapped+backfill_toofull, 36 active+undersized+degraded, 87 active+clean; 817 MiB data, 2.3 GiB used, 624 GiB / 626 GiB avail; 19 MiB/s rd, 1.2 MiB/s wr, 22 op/s; 293/1704 objects degraded (17.195%); 58/1704 objects misplaced (3.404%) 2024-09-20T22:57:21.467 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:21 smithi084 ceph-mon[40024]: osdmap e960: 8 total, 7 up, 7 in 2024-09-20T22:57:21.467 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:21 smithi084 ceph-mon[29774]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:57:21.467 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:21 smithi084 ceph-mon[29774]: pgmap v1341: 168 pgs: 1 active+clean+snaptrim, 2 active+undersized+degraded+remapped+backfill_toofull, 41 active+undersized, 1 active+remapped+backfill_toofull, 36 active+undersized+degraded, 87 active+clean; 817 MiB data, 2.3 GiB used, 624 GiB / 626 GiB avail; 19 MiB/s rd, 1.2 MiB/s wr, 22 op/s; 293/1704 objects degraded (17.195%); 58/1704 objects misplaced (3.404%) 2024-09-20T22:57:21.467 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:21 smithi084 ceph-mon[29774]: osdmap e960: 8 total, 7 up, 7 in 2024-09-20T22:57:21.468 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:21.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:21.469 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:57:21.531 INFO:tasks.rados.rados.0.smithi084.stdout:3892: done (8 left) 2024-09-20T22:57:21.531 INFO:tasks.rados.rados.0.smithi084.stdout:3893: read oid 11 snap 433 2024-09-20T22:57:21.531 INFO:tasks.rados.rados.0.smithi084.stdout:3893: expect (ObjNum 1253 snap 425 seq_num 1253) 2024-09-20T22:57:21.531 INFO:tasks.rados.rados.0.smithi084.stdout:3894: snap_create 2024-09-20T22:57:21.531 INFO:tasks.rados.rados.0.smithi084.stdout:3885: finishing write tid 1 to smithi08477203-41 2024-09-20T22:57:21.531 INFO:tasks.rados.rados.0.smithi084.stdout:3884: expect (ObjNum 1318 snap 455 seq_num 1318) 2024-09-20T22:57:21.570 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:57:21.592 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:21.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:21.593 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:57:21.656 INFO:tasks.rados.rados.0.smithi084.stdout:3881: expect (ObjNum 1314 snap 453 seq_num 1314) 2024-09-20T22:57:21.694 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:57:21.715 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:21.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:21.716 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:57:21.732 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:57:21.733 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:57:21.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:21 smithi154 ceph-mon[33264]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:57:21.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:21 smithi154 ceph-mon[33264]: pgmap v1341: 168 pgs: 1 active+clean+snaptrim, 2 active+undersized+degraded+remapped+backfill_toofull, 41 active+undersized, 1 active+remapped+backfill_toofull, 36 active+undersized+degraded, 87 active+clean; 817 MiB data, 2.3 GiB used, 624 GiB / 626 GiB avail; 19 MiB/s rd, 1.2 MiB/s wr, 22 op/s; 293/1704 objects degraded (17.195%); 58/1704 objects misplaced (3.404%) 2024-09-20T22:57:21.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:21 smithi154 ceph-mon[33264]: osdmap e960: 8 total, 7 up, 7 in 2024-09-20T22:57:21.817 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:57:21.837 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:21.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:21.838 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:57:21.842 INFO:tasks.rados.rados.0.smithi084.stdout:3885: finishing write tid 2 to smithi08477203-41 2024-09-20T22:57:21.843 INFO:tasks.rados.rados.0.smithi084.stdout:3885: finishing write tid 3 to smithi08477203-41 2024-09-20T22:57:21.843 INFO:tasks.rados.rados.0.smithi084.stdout:3885: finishing write tid 4 to smithi08477203-41 2024-09-20T22:57:21.843 INFO:tasks.rados.rados.0.smithi084.stdout:3885: oid 41 updating version 0 to 2672 2024-09-20T22:57:21.843 INFO:tasks.rados.rados.0.smithi084.stdout:3885: oid 41 updating version 2672 to 2674 2024-09-20T22:57:21.843 INFO:tasks.rados.rados.0.smithi084.stdout:3885: oid 41 version 2674 is already newer than 2673 2024-09-20T22:57:21.843 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 41 v 2674 (ObjNum 1321 snap 455 seq_num 1321) dirty exists 2024-09-20T22:57:21.843 INFO:tasks.rados.rados.0.smithi084.stdout:3885: left oid 41 (ObjNum 1321 snap 455 seq_num 1321) 2024-09-20T22:57:21.843 INFO:tasks.rados.rados.0.smithi084.stdout:3888: expect (ObjNum 1205 snap 412 seq_num 1205) 2024-09-20T22:57:21.940 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:57:21.953 INFO:tasks.rados.rados.0.smithi084.stdout:3889: expect (ObjNum 1293 snap 439 seq_num 1293) 2024-09-20T22:57:21.962 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:21.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:21.963 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:57:22.064 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:57:22.086 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:22.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:22.087 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:57:22.121 INFO:tasks.rados.rados.0.smithi084.stdout:3890: finishing rollback tid 1 to smithi08477203-10 2024-09-20T22:57:22.121 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 10 v 2679 (ObjNum 129 snap 0 seq_num 2091208048) dirty dne 2024-09-20T22:57:22.121 INFO:tasks.rados.rados.0.smithi084.stdout:3893: expect (ObjNum 1253 snap 425 seq_num 1253) 2024-09-20T22:57:22.188 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:57:22.189 INFO:teuthology.orchestra.run.smithi084.stdout: marking collection for removal 2024-09-20T22:57:22.190 INFO:teuthology.orchestra.run.smithi084.stdout:setting '_remove' omap key 2024-09-20T22:57:22.190 INFO:teuthology.orchestra.run.smithi084.stdout:finish_remove_pgs 3.7_head removing 3.7 2024-09-20T22:57:22.190 INFO:teuthology.orchestra.run.smithi084.stdout:Remove successful 2024-09-20T22:57:22.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:22.211 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:22.211 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:57:22.312 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:57:22.335 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:22.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:22.336 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:57:22.433 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-20T22:57:22.433 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2024-09-20T22:57:22.433 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2024-09-20T22:57:22.433 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-20T22:57:22.437 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:57:22.458 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:22.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:22.459 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:57:22.536 INFO:tasks.rados.rados.0.smithi084.stdout:3881: done (9 left) 2024-09-20T22:57:22.536 INFO:tasks.rados.rados.0.smithi084.stdout:3884: done (8 left) 2024-09-20T22:57:22.536 INFO:tasks.rados.rados.0.smithi084.stdout:3885: done (7 left) 2024-09-20T22:57:22.536 INFO:tasks.rados.rados.0.smithi084.stdout:3886: done (6 left) 2024-09-20T22:57:22.536 INFO:tasks.rados.rados.0.smithi084.stdout:3887: done (5 left) 2024-09-20T22:57:22.536 INFO:tasks.rados.rados.0.smithi084.stdout:3888: done (4 left) 2024-09-20T22:57:22.536 INFO:tasks.rados.rados.0.smithi084.stdout:3889: done (3 left) 2024-09-20T22:57:22.536 INFO:tasks.rados.rados.0.smithi084.stdout:3890: done (2 left) 2024-09-20T22:57:22.536 INFO:tasks.rados.rados.0.smithi084.stdout:3893: done (1 left) 2024-09-20T22:57:22.536 INFO:tasks.rados.rados.0.smithi084.stdout:3894: done (0 left) 2024-09-20T22:57:22.537 INFO:tasks.rados.rados.0.smithi084.stdout:3895: delete oid 47 current snap is 456 2024-09-20T22:57:22.541 INFO:tasks.rados.rados.0.smithi084.stdout:3895: done (0 left) 2024-09-20T22:57:22.541 INFO:tasks.rados.rados.0.smithi084.stdout:3896: copy_from oid 22 from oid 24 current snap is 456 2024-09-20T22:57:22.541 INFO:tasks.rados.rados.0.smithi084.stdout:3897: read oid 4 snap -1 2024-09-20T22:57:22.541 INFO:tasks.rados.rados.0.smithi084.stdout:3897: expect (ObjNum 1246 snap 423 seq_num 1246) 2024-09-20T22:57:22.541 INFO:tasks.rados.rados.0.smithi084.stdout:3898: read oid 43 snap -1 2024-09-20T22:57:22.541 INFO:tasks.rados.rados.0.smithi084.stdout:3898: expect (ObjNum 1319 snap 455 seq_num 1319) 2024-09-20T22:57:22.541 INFO:tasks.rados.rados.0.smithi084.stdout:3899: delete oid 2 current snap is 456 2024-09-20T22:57:22.544 INFO:tasks.rados.rados.0.smithi084.stdout:3898: expect (ObjNum 1319 snap 455 seq_num 1319) 2024-09-20T22:57:22.560 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:57:22.582 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:22.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:22.583 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:57:22.638 INFO:tasks.rados.rados.0.smithi084.stdout:3898: done (3 left) 2024-09-20T22:57:22.638 INFO:tasks.rados.rados.0.smithi084.stdout:3899: done (2 left) 2024-09-20T22:57:22.638 INFO:tasks.rados.rados.0.smithi084.stdout:3900: write oid 23 current snap is 456 2024-09-20T22:57:22.638 INFO:tasks.rados.rados.0.smithi084.stdout:3897: expect (ObjNum 1246 snap 423 seq_num 1246) 2024-09-20T22:57:22.684 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:57:22.705 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:22.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:22.706 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:57:22.714 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:22 smithi084 ceph-mon[40024]: purged_snaps scrub starts 2024-09-20T22:57:22.714 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:22 smithi084 ceph-mon[40024]: purged_snaps scrub ok 2024-09-20T22:57:22.714 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:22 smithi084 ceph-mon[40024]: osdmap e961: 8 total, 7 up, 7 in 2024-09-20T22:57:22.714 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:22 smithi084 ceph-mon[29774]: purged_snaps scrub starts 2024-09-20T22:57:22.714 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:22 smithi084 ceph-mon[29774]: purged_snaps scrub ok 2024-09-20T22:57:22.714 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:22 smithi084 ceph-mon[29774]: osdmap e961: 8 total, 7 up, 7 in 2024-09-20T22:57:22.776 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-20T22:57:22.806 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:57:22.828 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:22.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:22.829 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:57:22.844 INFO:tasks.rados.rados.0.smithi084.stdout:3896: finishing copy_from racing read to smithi08477203-22 2024-09-20T22:57:22.844 INFO:tasks.rados.rados.0.smithi084.stdout:3896: finishing copy_from to smithi08477203-22 2024-09-20T22:57:22.844 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 22 v 1956 (ObjNum 1298 snap 441 seq_num 1298) dirty exists 2024-09-20T22:57:22.844 INFO:tasks.rados.rados.0.smithi084.stdout:3900: seq_num 1325 ranges {669275=750347,2035700=233575} 2024-09-20T22:57:22.851 INFO:tasks.rados.rados.0.smithi084.stdout:3900: writing smithi08477203-23 from 669275 to 1419622 tid 1 2024-09-20T22:57:22.857 INFO:tasks.rados.rados.0.smithi084.stdout:3900: writing smithi08477203-23 from 2035700 to 2269275 tid 2 2024-09-20T22:57:22.857 INFO:tasks.rados.rados.0.smithi084.stdout:3896: done (2 left) 2024-09-20T22:57:22.857 INFO:tasks.rados.rados.0.smithi084.stdout:3897: done (1 left) 2024-09-20T22:57:22.857 INFO:tasks.rados.rados.0.smithi084.stdout:3901: read oid 7 snap -1 2024-09-20T22:57:22.857 INFO:tasks.rados.rados.0.smithi084.stdout:3901: expect (ObjNum 1320 snap 455 seq_num 1320) 2024-09-20T22:57:22.857 INFO:tasks.rados.rados.0.smithi084.stdout:3902: read oid 18 snap -1 2024-09-20T22:57:22.857 INFO:tasks.rados.rados.0.smithi084.stdout:3902: expect (ObjNum 1311 snap 453 seq_num 1311) 2024-09-20T22:57:22.857 INFO:tasks.rados.rados.0.smithi084.stdout:3903: snap_remove snap 443 2024-09-20T22:57:22.930 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:57:22.951 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:22.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:22.952 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:57:23.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:22 smithi154 ceph-mon[33264]: purged_snaps scrub starts 2024-09-20T22:57:23.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:22 smithi154 ceph-mon[33264]: purged_snaps scrub ok 2024-09-20T22:57:23.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:22 smithi154 ceph-mon[33264]: osdmap e961: 8 total, 7 up, 7 in 2024-09-20T22:57:23.053 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:57:23.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:23.076 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:23.077 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:57:23.177 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:57:23.202 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:23.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:23.203 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:57:23.304 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:57:23.326 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:23.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:23.327 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:57:23.428 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:57:23.433 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell -n osd.3 --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph-objectstore-tool --err-to-stderr --data-path /var/lib/ceph/osd/ceph-3 --journal-path /var/lib/ceph/osd/ceph-3/journal '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --op import --file /var/log/ceph/exp.3.7.3 2024-09-20T22:57:23.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:23.450 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:23.451 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:57:23.540 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is set 2024-09-20T22:57:23.544 INFO:tasks.rados.rados.0.smithi084.stdout:3903: done (3 left) 2024-09-20T22:57:23.544 INFO:tasks.rados.rados.0.smithi084.stdout:3904: write (excl) oid 19 current snap is 456 2024-09-20T22:57:23.544 INFO:tasks.rados.rados.0.smithi084.stdout:3904: seq_num 1326 ranges {686082=425596,1577829=433132} 2024-09-20T22:57:23.550 INFO:tasks.rados.rados.0.smithi084.stdout:3904: writing smithi08477203-19 from 686082 to 1111678 tid 1 2024-09-20T22:57:23.551 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:57:23.555 INFO:tasks.rados.rados.0.smithi084.stdout:3904: writing smithi08477203-19 from 1577829 to 2010961 tid 2 2024-09-20T22:57:23.555 INFO:tasks.rados.rados.0.smithi084.stdout:3900: finishing write tid 1 to smithi08477203-23 2024-09-20T22:57:23.555 INFO:tasks.rados.rados.0.smithi084.stdout:3900: finishing write tid 2 to smithi08477203-23 2024-09-20T22:57:23.556 INFO:tasks.rados.rados.0.smithi084.stdout:3901: expect (ObjNum 1320 snap 455 seq_num 1320) 2024-09-20T22:57:23.578 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:23.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:23.579 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:57:23.680 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:57:23.682 INFO:tasks.rados.rados.0.smithi084.stdout:3900: finishing write tid 3 to smithi08477203-23 2024-09-20T22:57:23.682 INFO:tasks.rados.rados.0.smithi084.stdout:3900: finishing write tid 4 to smithi08477203-23 2024-09-20T22:57:23.682 INFO:tasks.rados.rados.0.smithi084.stdout:3900: oid 23 updating version 0 to 2866 2024-09-20T22:57:23.682 INFO:tasks.rados.rados.0.smithi084.stdout:3900: oid 23 version 2866 is already newer than 2865 2024-09-20T22:57:23.682 INFO:tasks.rados.rados.0.smithi084.stdout:3900: oid 23 version 2866 is already newer than 2864 2024-09-20T22:57:23.683 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 23 v 2866 (ObjNum 1325 snap 456 seq_num 1325) dirty exists 2024-09-20T22:57:23.683 INFO:tasks.rados.rados.0.smithi084.stdout:3900: left oid 23 (ObjNum 1325 snap 456 seq_num 1325) 2024-09-20T22:57:23.683 INFO:tasks.rados.rados.0.smithi084.stdout:3902: expect (ObjNum 1311 snap 453 seq_num 1311) 2024-09-20T22:57:23.704 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:23.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:23.705 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:57:23.765 INFO:tasks.rados.rados.0.smithi084.stdout:3904: finishing write tid 1 to smithi08477203-19 2024-09-20T22:57:23.765 INFO:tasks.rados.rados.0.smithi084.stdout:3904: finishing write tid 2 to smithi08477203-19 2024-09-20T22:57:23.766 INFO:tasks.rados.rados.0.smithi084.stdout:3904: finishing write tid 3 to smithi08477203-19 2024-09-20T22:57:23.766 INFO:tasks.rados.rados.0.smithi084.stdout:3904: finishing write tid 4 to smithi08477203-19 2024-09-20T22:57:23.766 INFO:tasks.rados.rados.0.smithi084.stdout:3904: oid 19 updating version 0 to 3139 2024-09-20T22:57:23.766 INFO:tasks.rados.rados.0.smithi084.stdout:3904: oid 19 updating version 3139 to 3140 2024-09-20T22:57:23.766 INFO:tasks.rados.rados.0.smithi084.stdout:3904: oid 19 updating version 3140 to 3141 2024-09-20T22:57:23.766 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 19 v 3141 (ObjNum 1326 snap 456 seq_num 1326) dirty exists 2024-09-20T22:57:23.766 INFO:tasks.rados.rados.0.smithi084.stdout:3904: left oid 19 (ObjNum 1326 snap 456 seq_num 1326) 2024-09-20T22:57:23.766 INFO:tasks.rados.rados.0.smithi084.stdout:3900: done (3 left) 2024-09-20T22:57:23.766 INFO:tasks.rados.rados.0.smithi084.stdout:3901: done (2 left) 2024-09-20T22:57:23.766 INFO:tasks.rados.rados.0.smithi084.stdout:3902: done (1 left) 2024-09-20T22:57:23.766 INFO:tasks.rados.rados.0.smithi084.stdout:3904: done (0 left) 2024-09-20T22:57:23.766 INFO:tasks.rados.rados.0.smithi084.stdout:3905: write (excl) oid 24 current snap is 456 2024-09-20T22:57:23.766 INFO:tasks.rados.rados.0.smithi084.stdout:3905: seq_num 1327 ranges {702889=500845,1636311=576790} 2024-09-20T22:57:23.771 INFO:tasks.rados.rados.0.smithi084.stdout:3905: writing smithi08477203-24 from 702889 to 1203734 tid 1 2024-09-20T22:57:23.777 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/osd.3/config 2024-09-20T22:57:23.777 INFO:tasks.rados.rados.0.smithi084.stdout:3905: writing smithi08477203-24 from 1636311 to 2213101 tid 2 2024-09-20T22:57:23.778 INFO:tasks.rados.rados.0.smithi084.stdout:3906: snap_remove snap 447 2024-09-20T22:57:23.806 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:57:23.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:23.828 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:23.829 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:57:23.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:23 smithi084 ceph-mon[29774]: pgmap v1344: 168 pgs: 1 active+clean+snaptrim, 2 active+undersized+degraded+remapped+backfill_toofull, 41 active+undersized, 1 active+remapped+backfill_toofull, 36 active+undersized+degraded, 87 active+clean; 817 MiB data, 2.3 GiB used, 624 GiB / 626 GiB avail; 19 MiB/s rd, 1.2 MiB/s wr, 22 op/s; 293/1704 objects degraded (17.195%); 58/1704 objects misplaced (3.404%) 2024-09-20T22:57:23.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:23 smithi084 ceph-mon[29774]: osdmap e962: 8 total, 7 up, 7 in 2024-09-20T22:57:23.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:23 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/485757437' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:57:23.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:23 smithi084 ceph-mon[29774]: Health check update: Degraded data redundancy: 293/1704 objects degraded (17.195%), 38 pgs degraded (PG_DEGRADED) 2024-09-20T22:57:23.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:23 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:57:23.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:23 smithi084 ceph-mon[40024]: pgmap v1344: 168 pgs: 1 active+clean+snaptrim, 2 active+undersized+degraded+remapped+backfill_toofull, 41 active+undersized, 1 active+remapped+backfill_toofull, 36 active+undersized+degraded, 87 active+clean; 817 MiB data, 2.3 GiB used, 624 GiB / 626 GiB avail; 19 MiB/s rd, 1.2 MiB/s wr, 22 op/s; 293/1704 objects degraded (17.195%); 58/1704 objects misplaced (3.404%) 2024-09-20T22:57:23.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:23 smithi084 ceph-mon[40024]: osdmap e962: 8 total, 7 up, 7 in 2024-09-20T22:57:23.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:23 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/485757437' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:57:23.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:23 smithi084 ceph-mon[40024]: Health check update: Degraded data redundancy: 293/1704 objects degraded (17.195%), 38 pgs degraded (PG_DEGRADED) 2024-09-20T22:57:23.866 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:23 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:57:23.929 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:57:23.954 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:23.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:23.955 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:57:24.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:23 smithi154 ceph-mon[33264]: pgmap v1344: 168 pgs: 1 active+clean+snaptrim, 2 active+undersized+degraded+remapped+backfill_toofull, 41 active+undersized, 1 active+remapped+backfill_toofull, 36 active+undersized+degraded, 87 active+clean; 817 MiB data, 2.3 GiB used, 624 GiB / 626 GiB avail; 19 MiB/s rd, 1.2 MiB/s wr, 22 op/s; 293/1704 objects degraded (17.195%); 58/1704 objects misplaced (3.404%) 2024-09-20T22:57:24.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:23 smithi154 ceph-mon[33264]: osdmap e962: 8 total, 7 up, 7 in 2024-09-20T22:57:24.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:23 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/485757437' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:57:24.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:23 smithi154 ceph-mon[33264]: Health check update: Degraded data redundancy: 293/1704 objects degraded (17.195%), 38 pgs degraded (PG_DEGRADED) 2024-09-20T22:57:24.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:23 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:57:24.056 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:57:24.078 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:24.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:24.079 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:57:24.173 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.1 dump_blocked_ops 2024-09-20T22:57:24.180 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:57:24.205 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:24.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:24.206 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:57:24.307 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:57:24.332 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:24.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:24.333 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:57:24.434 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:57:24.455 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:24.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:24.456 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:57:24.545 INFO:tasks.rados.rados.0.smithi084.stdout:3906: done (1 left) 2024-09-20T22:57:24.545 INFO:tasks.rados.rados.0.smithi084.stdout:3907: rollback oid 49 current snap is 456 2024-09-20T22:57:24.545 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 49 to 364 2024-09-20T22:57:24.545 INFO:tasks.rados.rados.0.smithi084.stdout:3908: copy_from oid 5 from oid 31 current snap is 456 2024-09-20T22:57:24.545 INFO:tasks.rados.rados.0.smithi084.stdout:3909: delete oid 16 current snap is 456 2024-09-20T22:57:24.545 INFO:tasks.rados.rados.0.smithi084.stdout:3905: finishing write tid 1 to smithi08477203-24 2024-09-20T22:57:24.545 INFO:tasks.rados.rados.0.smithi084.stdout:3905: finishing write tid 2 to smithi08477203-24 2024-09-20T22:57:24.545 INFO:tasks.rados.rados.0.smithi084.stdout:3905: finishing write tid 3 to smithi08477203-24 2024-09-20T22:57:24.546 INFO:tasks.rados.rados.0.smithi084.stdout:3905: finishing write tid 4 to smithi08477203-24 2024-09-20T22:57:24.546 INFO:tasks.rados.rados.0.smithi084.stdout:3905: oid 24 updating version 0 to 1963 2024-09-20T22:57:24.546 INFO:tasks.rados.rados.0.smithi084.stdout:3905: oid 24 version 1963 is already newer than 1962 2024-09-20T22:57:24.546 INFO:tasks.rados.rados.0.smithi084.stdout:3905: oid 24 version 1963 is already newer than 1961 2024-09-20T22:57:24.546 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 24 v 1963 (ObjNum 1327 snap 456 seq_num 1327) dirty exists 2024-09-20T22:57:24.546 INFO:tasks.rados.rados.0.smithi084.stdout:3905: left oid 24 (ObjNum 1327 snap 456 seq_num 1327) 2024-09-20T22:57:24.550 INFO:tasks.rados.rados.0.smithi084.stdout:3908: finishing copy_from to smithi08477203-5 2024-09-20T22:57:24.550 INFO:tasks.rados.rados.0.smithi084.stdout:3908: got expected ENOENT (src dne) 2024-09-20T22:57:24.550 INFO:tasks.rados.rados.0.smithi084.stdout:3908: finishing copy_from racing read to smithi08477203-5 2024-09-20T22:57:24.552 INFO:tasks.rados.rados.0.smithi084.stdout:3905: done (3 left) 2024-09-20T22:57:24.552 INFO:tasks.rados.rados.0.smithi084.stdout:3908: done (2 left) 2024-09-20T22:57:24.552 INFO:tasks.rados.rados.0.smithi084.stdout:3909: done (1 left) 2024-09-20T22:57:24.552 INFO:tasks.rados.rados.0.smithi084.stdout:3910: rollback oid 2 current snap is 456 2024-09-20T22:57:24.552 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 2 to 362 2024-09-20T22:57:24.552 INFO:tasks.rados.rados.0.smithi084.stdout:3911: rollback oid 46 current snap is 456 2024-09-20T22:57:24.552 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 46 to 444 2024-09-20T22:57:24.552 INFO:tasks.rados.rados.0.smithi084.stdout:3912: write (excl) oid 30 current snap is 456 2024-09-20T22:57:24.552 INFO:tasks.rados.rados.0.smithi084.stdout:3912: seq_num 1329 ranges {736503=651343,2153275=183228} 2024-09-20T22:57:24.557 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:57:24.559 INFO:tasks.rados.rados.0.smithi084.stdout:3912: writing smithi08477203-30 from 736503 to 1387846 tid 1 2024-09-20T22:57:24.564 INFO:tasks.rados.rados.0.smithi084.stdout:3912: writing smithi08477203-30 from 2153275 to 2336503 tid 2 2024-09-20T22:57:24.565 INFO:tasks.rados.rados.0.smithi084.stdout:3913: snap_remove snap 453 2024-09-20T22:57:24.582 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:24.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:24.582 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:57:24.683 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:57:24.709 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:24.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:24.709 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:57:24.752 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:57:24.754 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:57:24.810 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:57:24.835 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:24.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:24.836 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:57:24.839 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:24 smithi084 ceph-mon[29774]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:57:24.839 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:24 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/485757437' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T22:57:24.839 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:24 smithi084 ceph-mon[29774]: osdmap e963: 8 total, 7 up, 7 in 2024-09-20T22:57:24.839 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:24 smithi084 ceph-mon[40024]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:57:24.839 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:24 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/485757437' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T22:57:24.839 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:24 smithi084 ceph-mon[40024]: osdmap e963: 8 total, 7 up, 7 in 2024-09-20T22:57:24.937 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:57:24.961 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:24.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:24.962 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:57:25.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:24 smithi154 ceph-mon[33264]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:57:25.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:24 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/485757437' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T22:57:25.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:24 smithi154 ceph-mon[33264]: osdmap e963: 8 total, 7 up, 7 in 2024-09-20T22:57:25.063 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:57:25.087 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:25.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:25.088 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:57:25.188 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:57:25.211 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:25.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:25.212 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:57:25.313 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:57:25.334 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:25.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:25.335 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:57:25.436 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:57:25.456 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:25.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:25.457 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:57:25.553 INFO:tasks.rados.rados.0.smithi084.stdout:3913: done (4 left) 2024-09-20T22:57:25.553 INFO:tasks.rados.rados.0.smithi084.stdout:3914: copy_from oid 16 from oid 19 current snap is 456 2024-09-20T22:57:25.553 INFO:tasks.rados.rados.0.smithi084.stdout:3915: read oid 31 snap -1 2024-09-20T22:57:25.553 INFO:tasks.rados.rados.0.smithi084.stdout:3915: expect deleted 2024-09-20T22:57:25.553 INFO:tasks.rados.rados.0.smithi084.stdout:3916: write oid 4 current snap is 456 2024-09-20T22:57:25.553 INFO:tasks.rados.rados.0.smithi084.stdout:3916: seq_num 1330 ranges {753310=726592,2095404=257906} 2024-09-20T22:57:25.558 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:57:25.562 INFO:tasks.rados.rados.0.smithi084.stdout:3916: writing smithi08477203-4 from 753310 to 1479902 tid 1 2024-09-20T22:57:25.568 INFO:tasks.rados.rados.0.smithi084.stdout:3916: writing smithi08477203-4 from 2095404 to 2353310 tid 2 2024-09-20T22:57:25.568 INFO:tasks.rados.rados.0.smithi084.stdout:3917: write (excl) oid 35 current snap is 456 2024-09-20T22:57:25.568 INFO:tasks.rados.rados.0.smithi084.stdout:3907: finishing rollback tid 0 to smithi08477203-49 2024-09-20T22:57:25.568 INFO:tasks.rados.rados.0.smithi084.stdout:3917: seq_num 1331 ranges {770117=518194,1870239=499878} 2024-09-20T22:57:25.574 INFO:tasks.rados.rados.0.smithi084.stdout:3917: writing smithi08477203-35 from 770117 to 1288311 tid 1 2024-09-20T22:57:25.581 INFO:tasks.rados.rados.0.smithi084.stdout:3917: writing smithi08477203-35 from 1870239 to 2370117 tid 2 2024-09-20T22:57:25.582 INFO:tasks.rados.rados.0.smithi084.stdout:3918: read oid 34 snap -1 2024-09-20T22:57:25.582 INFO:tasks.rados.rados.0.smithi084.stdout:3918: expect (ObjNum 1268 snap 431 seq_num 1268) 2024-09-20T22:57:25.582 INFO:tasks.rados.rados.0.smithi084.stdout:3907: finishing rollback tid 1 to smithi08477203-49 2024-09-20T22:57:25.582 INFO:tasks.rados.rados.0.smithi084.stdout:3907: finishing rollback tid 2 to smithi08477203-49 2024-09-20T22:57:25.582 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 49 v 2686 (ObjNum 928 snap 308 seq_num 928) dirty exists 2024-09-20T22:57:25.582 INFO:tasks.rados.rados.0.smithi084.stdout:3910: finishing rollback tid 1 to smithi08477203-2 2024-09-20T22:57:25.582 INFO:tasks.rados.rados.0.smithi084.stdout:3919: read oid 1 snap -1 2024-09-20T22:57:25.582 INFO:tasks.rados.rados.0.smithi084.stdout:3919: expect (ObjNum 1293 snap 439 seq_num 1293) 2024-09-20T22:57:25.582 INFO:tasks.rados.rados.0.smithi084.stdout:3907: done (9 left) 2024-09-20T22:57:25.582 INFO:tasks.rados.rados.0.smithi084.stdout:3920: write oid 11 current snap is 456 2024-09-20T22:57:25.582 INFO:tasks.rados.rados.0.smithi084.stdout:3910: finishing rollback tid 2 to smithi08477203-2 2024-09-20T22:57:25.582 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 2 v 2604 (ObjNum 918 snap 302 seq_num 918) dirty exists 2024-09-20T22:57:25.582 INFO:tasks.rados.rados.0.smithi084.stdout:3911: finishing rollback tid 1 to smithi08477203-46 2024-09-20T22:57:25.582 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 46 v 2630 (ObjNum 289 snap 0 seq_num 1) dirty dne 2024-09-20T22:57:25.583 INFO:tasks.rados.rados.0.smithi084.stdout:3912: finishing write tid 1 to smithi08477203-30 2024-09-20T22:57:25.583 INFO:tasks.rados.rados.0.smithi084.stdout:3920: seq_num 1332 ranges {786924=593443,1928721=458203} 2024-09-20T22:57:25.583 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:25.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:25.584 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:57:25.589 INFO:tasks.rados.rados.0.smithi084.stdout:3920: writing smithi08477203-11 from 786924 to 1380367 tid 1 2024-09-20T22:57:25.595 INFO:tasks.rados.rados.0.smithi084.stdout:3920: writing smithi08477203-11 from 1928721 to 2386924 tid 2 2024-09-20T22:57:25.596 INFO:tasks.rados.rados.0.smithi084.stdout:3910: done (9 left) 2024-09-20T22:57:25.596 INFO:tasks.rados.rados.0.smithi084.stdout:3911: done (8 left) 2024-09-20T22:57:25.596 INFO:tasks.rados.rados.0.smithi084.stdout:3921: copy_from oid 9 from oid 39 current snap is 456 2024-09-20T22:57:25.596 INFO:tasks.rados.rados.0.smithi084.stdout:3912: finishing write tid 2 to smithi08477203-30 2024-09-20T22:57:25.596 INFO:tasks.rados.rados.0.smithi084.stdout:3922: snap_remove snap 411 2024-09-20T22:57:25.684 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:57:25.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:25.706 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:25.706 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:57:25.740 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:25 smithi084 ceph-mon[40024]: pgmap v1347: 168 pgs: 1 active+clean+snaptrim, 2 active+undersized+degraded+remapped+backfill_toofull, 41 active+undersized, 1 active+remapped+backfill_toofull, 36 active+undersized+degraded, 87 active+clean; 825 MiB data, 2.3 GiB used, 624 GiB / 626 GiB avail; 28 MiB/s rd, 2.4 MiB/s wr, 32 op/s; 299/1719 objects degraded (17.394%); 61/1719 objects misplaced (3.549%) 2024-09-20T22:57:25.740 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:25 smithi084 ceph-mon[40024]: osdmap e964: 8 total, 7 up, 7 in 2024-09-20T22:57:25.740 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:25 smithi084 ceph-mon[29774]: pgmap v1347: 168 pgs: 1 active+clean+snaptrim, 2 active+undersized+degraded+remapped+backfill_toofull, 41 active+undersized, 1 active+remapped+backfill_toofull, 36 active+undersized+degraded, 87 active+clean; 825 MiB data, 2.3 GiB used, 624 GiB / 626 GiB avail; 28 MiB/s rd, 2.4 MiB/s wr, 32 op/s; 299/1719 objects degraded (17.394%); 61/1719 objects misplaced (3.549%) 2024-09-20T22:57:25.740 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:25 smithi084 ceph-mon[29774]: osdmap e964: 8 total, 7 up, 7 in 2024-09-20T22:57:25.807 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:57:25.830 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:25.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:25.833 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:57:25.934 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:57:25.955 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:25.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:25.956 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:57:26.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:25 smithi154 ceph-mon[33264]: pgmap v1347: 168 pgs: 1 active+clean+snaptrim, 2 active+undersized+degraded+remapped+backfill_toofull, 41 active+undersized, 1 active+remapped+backfill_toofull, 36 active+undersized+degraded, 87 active+clean; 825 MiB data, 2.3 GiB used, 624 GiB / 626 GiB avail; 28 MiB/s rd, 2.4 MiB/s wr, 32 op/s; 299/1719 objects degraded (17.394%); 61/1719 objects misplaced (3.549%) 2024-09-20T22:57:26.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:25 smithi154 ceph-mon[33264]: osdmap e964: 8 total, 7 up, 7 in 2024-09-20T22:57:26.056 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:57:26.079 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:26.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:26.080 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:57:26.181 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:57:26.197 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:57:26.197 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:57:26.203 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:26.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:26.204 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:57:26.216 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:57:26.217 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:57:26.233 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:57:26.233 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:57:26.233 INFO:teuthology.orchestra.run.smithi084.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-20T22:57:26.250 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:57:26.251 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:57:26.267 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:57:26.267 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:57:26.284 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:57:26.284 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:57:26.300 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:57:26.301 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:57:26.305 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:57:26.326 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:26.326 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-20T22:57:26.327 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2024-09-20T22:57:26.327 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2024-09-20T22:57:26.327 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2024-09-20T22:57:26.327 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-20T22:57:26.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:26.328 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:57:26.428 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:57:26.453 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:26.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:26.454 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:57:26.555 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:57:26.561 INFO:tasks.rados.rados.0.smithi084.stdout:3922: done (9 left) 2024-09-20T22:57:26.561 INFO:tasks.rados.rados.0.smithi084.stdout:3923: snap_remove snap 444 2024-09-20T22:57:26.583 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:26.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:26.584 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:57:26.676 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd unset noscrub 2024-09-20T22:57:26.684 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:57:26.708 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:26.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:26.709 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:57:26.809 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:57:26.831 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:26.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:26.832 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:57:26.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:26 smithi084 ceph-mon[40024]: osdmap e965: 8 total, 7 up, 7 in 2024-09-20T22:57:26.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:26 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/3263333938' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:57:26.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:26 smithi084 ceph-mon[29774]: osdmap e965: 8 total, 7 up, 7 in 2024-09-20T22:57:26.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:26 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/3263333938' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:57:26.933 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:57:26.954 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:26.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:26.955 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:57:27.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:26 smithi154 ceph-mon[33264]: osdmap e965: 8 total, 7 up, 7 in 2024-09-20T22:57:27.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:26 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/3263333938' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:57:27.055 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:57:27.077 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:27.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:27.078 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:57:27.104 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.1 dump_historic_ops 2024-09-20T22:57:27.179 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:57:27.208 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:27.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:27.209 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:57:27.310 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:57:27.336 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:27.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:27.337 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:57:27.438 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:57:27.463 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:27.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:27.464 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:57:27.494 INFO:teuthology.orchestra.run.smithi084.stderr:Creating an OSD daemon form without an OSD FSID value 2024-09-20T22:57:27.563 INFO:tasks.rados.rados.0.smithi084.stdout:3923: done (9 left) 2024-09-20T22:57:27.563 INFO:tasks.rados.rados.0.smithi084.stdout:3924: delete oid 7 current snap is 456 2024-09-20T22:57:27.563 INFO:tasks.rados.rados.0.smithi084.stdout:3912: finishing write tid 3 to smithi08477203-30 2024-09-20T22:57:27.563 INFO:tasks.rados.rados.0.smithi084.stdout:3912: finishing write tid 4 to smithi08477203-30 2024-09-20T22:57:27.563 INFO:tasks.rados.rados.0.smithi084.stdout:3912: oid 30 updating version 0 to 2638 2024-09-20T22:57:27.563 INFO:tasks.rados.rados.0.smithi084.stdout:3912: oid 30 updating version 2638 to 2639 2024-09-20T22:57:27.564 INFO:tasks.rados.rados.0.smithi084.stdout:3912: oid 30 updating version 2639 to 2640 2024-09-20T22:57:27.564 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 30 v 2640 (ObjNum 1329 snap 456 seq_num 1329) dirty exists 2024-09-20T22:57:27.564 INFO:tasks.rados.rados.0.smithi084.stdout:3912: left oid 30 (ObjNum 1329 snap 456 seq_num 1329) 2024-09-20T22:57:27.564 INFO:tasks.rados.rados.0.smithi084.stdout:3914: finishing copy_from racing read to smithi08477203-16 2024-09-20T22:57:27.564 INFO:tasks.rados.rados.0.smithi084.stdout:3914: finishing copy_from to smithi08477203-16 2024-09-20T22:57:27.564 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 16 v 3141 (ObjNum 1326 snap 456 seq_num 1326) dirty exists 2024-09-20T22:57:27.564 INFO:tasks.rados.rados.0.smithi084.stdout:3916: finishing write tid 1 to smithi08477203-4 2024-09-20T22:57:27.564 INFO:tasks.rados.rados.0.smithi084.stdout:3916: finishing write tid 2 to smithi08477203-4 2024-09-20T22:57:27.564 INFO:tasks.rados.rados.0.smithi084.stdout:3916: finishing write tid 3 to smithi08477203-4 2024-09-20T22:57:27.564 INFO:tasks.rados.rados.0.smithi084.stdout:3916: finishing write tid 4 to smithi08477203-4 2024-09-20T22:57:27.564 INFO:tasks.rados.rados.0.smithi084.stdout:3916: oid 4 updating version 0 to 3144 2024-09-20T22:57:27.564 INFO:tasks.rados.rados.0.smithi084.stdout:3916: oid 4 version 3144 is already newer than 3143 2024-09-20T22:57:27.564 INFO:tasks.rados.rados.0.smithi084.stdout:3916: oid 4 updating version 3144 to 3145 2024-09-20T22:57:27.564 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 4 v 3145 (ObjNum 1330 snap 456 seq_num 1330) dirty exists 2024-09-20T22:57:27.564 INFO:tasks.rados.rados.0.smithi084.stdout:3916: left oid 4 (ObjNum 1330 snap 456 seq_num 1330) 2024-09-20T22:57:27.565 INFO:tasks.rados.rados.0.smithi084.stdout:3917: finishing write tid 1 to smithi08477203-35 2024-09-20T22:57:27.565 INFO:tasks.rados.rados.0.smithi084.stdout:3919: expect (ObjNum 1293 snap 439 seq_num 1293) 2024-09-20T22:57:27.565 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:57:27.589 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:27.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:27.590 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:57:27.691 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:57:27.716 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:27.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:27.717 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:57:27.731 INFO:tasks.rados.rados.0.smithi084.stdout:3917: finishing write tid 2 to smithi08477203-35 2024-09-20T22:57:27.731 INFO:tasks.rados.rados.0.smithi084.stdout:3917: finishing write tid 3 to smithi08477203-35 2024-09-20T22:57:27.731 INFO:tasks.rados.rados.0.smithi084.stdout:3917: finishing write tid 4 to smithi08477203-35 2024-09-20T22:57:27.732 INFO:tasks.rados.rados.0.smithi084.stdout:3917: oid 35 updating version 0 to 2670 2024-09-20T22:57:27.732 INFO:tasks.rados.rados.0.smithi084.stdout:3917: oid 35 updating version 2670 to 2672 2024-09-20T22:57:27.732 INFO:tasks.rados.rados.0.smithi084.stdout:3917: oid 35 version 2672 is already newer than 2671 2024-09-20T22:57:27.732 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 35 v 2672 (ObjNum 1331 snap 456 seq_num 1331) dirty exists 2024-09-20T22:57:27.732 INFO:tasks.rados.rados.0.smithi084.stdout:3917: left oid 35 (ObjNum 1331 snap 456 seq_num 1331) 2024-09-20T22:57:27.732 INFO:tasks.rados.rados.0.smithi084.stdout:3912: done (9 left) 2024-09-20T22:57:27.732 INFO:tasks.rados.rados.0.smithi084.stdout:3914: done (8 left) 2024-09-20T22:57:27.732 INFO:tasks.rados.rados.0.smithi084.stdout:3915: done (7 left) 2024-09-20T22:57:27.732 INFO:tasks.rados.rados.0.smithi084.stdout:3916: done (6 left) 2024-09-20T22:57:27.732 INFO:tasks.rados.rados.0.smithi084.stdout:3917: done (5 left) 2024-09-20T22:57:27.732 INFO:tasks.rados.rados.0.smithi084.stdout:3919: done (4 left) 2024-09-20T22:57:27.732 INFO:tasks.rados.rados.0.smithi084.stdout:3924: done (3 left) 2024-09-20T22:57:27.732 INFO:tasks.rados.rados.0.smithi084.stdout:3925: read oid 12 snap -1 2024-09-20T22:57:27.733 INFO:tasks.rados.rados.0.smithi084.stdout:3925: expect (ObjNum 1202 snap 411 seq_num 1202) 2024-09-20T22:57:27.733 INFO:tasks.rados.rados.0.smithi084.stdout:3926: snap_remove snap 424 2024-09-20T22:57:27.756 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:27 smithi084 ceph-mon[40024]: pgmap v1350: 168 pgs: 1 active+clean+snaptrim, 2 active+undersized+degraded+remapped+backfill_toofull, 41 active+undersized, 1 active+remapped+backfill_toofull, 36 active+undersized+degraded, 87 active+clean; 825 MiB data, 2.3 GiB used, 624 GiB / 626 GiB avail; 28 MiB/s rd, 2.3 MiB/s wr, 32 op/s; 299/1719 objects degraded (17.394%); 61/1719 objects misplaced (3.549%) 2024-09-20T22:57:27.756 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:27 smithi084 ceph-mon[40024]: osdmap e966: 8 total, 7 up, 7 in 2024-09-20T22:57:27.756 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:27 smithi084 ceph-mon[29774]: pgmap v1350: 168 pgs: 1 active+clean+snaptrim, 2 active+undersized+degraded+remapped+backfill_toofull, 41 active+undersized, 1 active+remapped+backfill_toofull, 36 active+undersized+degraded, 87 active+clean; 825 MiB data, 2.3 GiB used, 624 GiB / 626 GiB avail; 28 MiB/s rd, 2.3 MiB/s wr, 32 op/s; 299/1719 objects degraded (17.394%); 61/1719 objects misplaced (3.549%) 2024-09-20T22:57:27.756 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:27 smithi084 ceph-mon[29774]: osdmap e966: 8 total, 7 up, 7 in 2024-09-20T22:57:27.818 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:57:27.843 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:27.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:27.844 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:57:27.944 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:57:27.970 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:27.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:27.971 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:57:28.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:27 smithi154 ceph-mon[33264]: pgmap v1350: 168 pgs: 1 active+clean+snaptrim, 2 active+undersized+degraded+remapped+backfill_toofull, 41 active+undersized, 1 active+remapped+backfill_toofull, 36 active+undersized+degraded, 87 active+clean; 825 MiB data, 2.3 GiB used, 624 GiB / 626 GiB avail; 28 MiB/s rd, 2.3 MiB/s wr, 32 op/s; 299/1719 objects degraded (17.394%); 61/1719 objects misplaced (3.549%) 2024-09-20T22:57:28.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:27 smithi154 ceph-mon[33264]: osdmap e966: 8 total, 7 up, 7 in 2024-09-20T22:57:28.072 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:57:28.097 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:28.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:28.098 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:57:28.199 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:57:28.223 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:28.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:28.224 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:57:28.325 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:57:28.342 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:57:28 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: ::ffff:172.21.15.154 - - [20/Sep/2024:22:57:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-20T22:57:28.348 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:28.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:28.349 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:57:28.367 INFO:tasks.rados.rados.0.smithi084.stdout:3926: done (4 left) 2024-09-20T22:57:28.367 INFO:tasks.rados.rados.0.smithi084.stdout:3927: write (excl) oid 7 current snap is 456 2024-09-20T22:57:28.367 INFO:tasks.rados.rados.0.smithi084.stdout:3918: expect (ObjNum 1268 snap 431 seq_num 1268) 2024-09-20T22:57:28.449 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:57:28.473 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:28.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:28.474 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:57:28.482 INFO:teuthology.orchestra.run.smithi084.stdout:get_pg_num_history pg_num_history pg_num_history(e939 pg_nums {1={119=21,263=20,267=19,271=18,275=17,279=16,283=15,287=14,291=13,295=12,299=11},2={65=3},3={66=32},4={68=32},5={70=32},6={72=32},7={137=26,426=25,430=24,442=23,446=22,456=21,460=20,464=19,476=18,480=17,484=16,932=26}} deleted_pools ) 2024-09-20T22:57:28.483 INFO:teuthology.orchestra.run.smithi084.stdout:Importing pgid 3.7 2024-09-20T22:57:28.483 INFO:teuthology.orchestra.run.smithi084.stdout:write_pg epoch 935 info 3.7( empty local-lis/les=887/888 n=0 ec=66/66 lis/c=887/887 les/c/f=888/888/0 sis=887) 2024-09-20T22:57:28.483 INFO:teuthology.orchestra.run.smithi084.stdout:Import successful 2024-09-20T22:57:28.491 INFO:tasks.rados.rados.0.smithi084.stdout:3920: finishing write tid 1 to smithi08477203-11 2024-09-20T22:57:28.491 INFO:tasks.rados.rados.0.smithi084.stdout:3920: finishing write tid 2 to smithi08477203-11 2024-09-20T22:57:28.491 INFO:tasks.rados.rados.0.smithi084.stdout:3920: finishing write tid 3 to smithi08477203-11 2024-09-20T22:57:28.491 INFO:tasks.rados.rados.0.smithi084.stdout:3920: finishing write tid 4 to smithi08477203-11 2024-09-20T22:57:28.491 INFO:tasks.rados.rados.0.smithi084.stdout:3920: oid 11 updating version 0 to 2672 2024-09-20T22:57:28.492 INFO:tasks.rados.rados.0.smithi084.stdout:3920: oid 11 version 2672 is already newer than 2670 2024-09-20T22:57:28.492 INFO:tasks.rados.rados.0.smithi084.stdout:3920: oid 11 version 2672 is already newer than 2671 2024-09-20T22:57:28.492 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 11 v 2672 (ObjNum 1332 snap 456 seq_num 1332) dirty exists 2024-09-20T22:57:28.492 INFO:tasks.rados.rados.0.smithi084.stdout:3920: left oid 11 (ObjNum 1332 snap 456 seq_num 1332) 2024-09-20T22:57:28.492 INFO:tasks.rados.rados.0.smithi084.stdout:3921: finishing copy_from racing read to smithi08477203-9 2024-09-20T22:57:28.492 INFO:tasks.rados.rados.0.smithi084.stdout:3921: finishing copy_from to smithi08477203-9 2024-09-20T22:57:28.492 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 9 v 2668 (ObjNum 738 snap 235 seq_num 738) dirty exists 2024-09-20T22:57:28.492 INFO:tasks.rados.rados.0.smithi084.stdout:3927: seq_num 1334 ranges {420538=743941,1929332=491206} 2024-09-20T22:57:28.497 INFO:tasks.rados.rados.0.smithi084.stdout:3927: writing smithi08477203-7 from 420538 to 1164479 tid 1 2024-09-20T22:57:28.505 INFO:tasks.rados.rados.0.smithi084.stdout:3927: writing smithi08477203-7 from 1929332 to 2420538 tid 2 2024-09-20T22:57:28.505 INFO:tasks.rados.rados.0.smithi084.stdout:3918: done (4 left) 2024-09-20T22:57:28.505 INFO:tasks.rados.rados.0.smithi084.stdout:3920: done (3 left) 2024-09-20T22:57:28.505 INFO:tasks.rados.rados.0.smithi084.stdout:3921: done (2 left) 2024-09-20T22:57:28.505 INFO:tasks.rados.rados.0.smithi084.stdout:3928: copy_from oid 49 from oid 17 current snap is 456 2024-09-20T22:57:28.505 INFO:tasks.rados.rados.0.smithi084.stdout:3925: expect (ObjNum 1202 snap 411 seq_num 1202) 2024-09-20T22:57:28.574 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:57:28.582 INFO:tasks.rados.rados.0.smithi084.stdout:3927: finishing write tid 1 to smithi08477203-7 2024-09-20T22:57:28.582 INFO:tasks.rados.rados.0.smithi084.stdout:3927: finishing write tid 2 to smithi08477203-7 2024-09-20T22:57:28.582 INFO:tasks.rados.rados.0.smithi084.stdout:3927: finishing write tid 3 to smithi08477203-7 2024-09-20T22:57:28.582 INFO:tasks.rados.rados.0.smithi084.stdout:3927: finishing write tid 4 to smithi08477203-7 2024-09-20T22:57:28.582 INFO:tasks.rados.rados.0.smithi084.stdout:3927: oid 7 updating version 0 to 1967 2024-09-20T22:57:28.582 INFO:tasks.rados.rados.0.smithi084.stdout:3927: oid 7 updating version 1967 to 1968 2024-09-20T22:57:28.582 INFO:tasks.rados.rados.0.smithi084.stdout:3927: oid 7 updating version 1968 to 1969 2024-09-20T22:57:28.582 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 7 v 1969 (ObjNum 1334 snap 456 seq_num 1334) dirty exists 2024-09-20T22:57:28.582 INFO:tasks.rados.rados.0.smithi084.stdout:3927: left oid 7 (ObjNum 1334 snap 456 seq_num 1334) 2024-09-20T22:57:28.583 INFO:tasks.rados.rados.0.smithi084.stdout:3928: finishing copy_from racing read to smithi08477203-49 2024-09-20T22:57:28.583 INFO:tasks.rados.rados.0.smithi084.stdout:3925: done (2 left) 2024-09-20T22:57:28.583 INFO:tasks.rados.rados.0.smithi084.stdout:3927: done (1 left) 2024-09-20T22:57:28.583 INFO:tasks.rados.rados.0.smithi084.stdout:3929: read oid 5 snap -1 2024-09-20T22:57:28.583 INFO:tasks.rados.rados.0.smithi084.stdout:3929: expect (ObjNum 1241 snap 421 seq_num 1241) 2024-09-20T22:57:28.583 INFO:tasks.rados.rados.0.smithi084.stdout:3930: snap_create 2024-09-20T22:57:28.583 INFO:tasks.rados.rados.0.smithi084.stdout:3928: finishing copy_from to smithi08477203-49 2024-09-20T22:57:28.583 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 49 v 2656 (ObjNum 1302 snap 442 seq_num 1302) dirty exists 2024-09-20T22:57:28.584 INFO:tasks.rados.rados.0.smithi084.stdout:3929: expect (ObjNum 1241 snap 421 seq_num 1241) 2024-09-20T22:57:28.596 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:28.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:28.597 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:57:28.614 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:28 smithi084 ceph-mon[40024]: osdmap e967: 8 total, 7 up, 7 in 2024-09-20T22:57:28.614 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:28 smithi084 ceph-mon[40024]: Health check update: Degraded data redundancy: 299/1719 objects degraded (17.394%), 38 pgs degraded (PG_DEGRADED) 2024-09-20T22:57:28.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:28 smithi084 ceph-mon[40024]: osdmap e968: 8 total, 7 up, 7 in 2024-09-20T22:57:28.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:28 smithi084 ceph-mon[29774]: osdmap e967: 8 total, 7 up, 7 in 2024-09-20T22:57:28.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:28 smithi084 ceph-mon[29774]: Health check update: Degraded data redundancy: 299/1719 objects degraded (17.394%), 38 pgs degraded (PG_DEGRADED) 2024-09-20T22:57:28.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:28 smithi084 ceph-mon[29774]: osdmap e968: 8 total, 7 up, 7 in 2024-09-20T22:57:28.698 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:57:28.718 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:28.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:28.719 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:57:28.786 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-20T22:57:28.787 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-20T22:57:28.803 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:57:28.808 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:57:28.820 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:57:28.843 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:28.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:28.844 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:57:28.945 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:57:28.966 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:28.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:28.967 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:57:29.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:28 smithi154 ceph-mon[33264]: osdmap e967: 8 total, 7 up, 7 in 2024-09-20T22:57:29.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:28 smithi154 ceph-mon[33264]: Health check update: Degraded data redundancy: 299/1719 objects degraded (17.394%), 38 pgs degraded (PG_DEGRADED) 2024-09-20T22:57:29.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:28 smithi154 ceph-mon[33264]: osdmap e968: 8 total, 7 up, 7 in 2024-09-20T22:57:29.068 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:57:29.092 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:29.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:29.092 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:57:29.193 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:57:29.215 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:29.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:29.216 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:57:29.317 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:57:29.338 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:29.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:29.339 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:57:29.440 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:57:29.461 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:29.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:29.462 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:57:29.563 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:57:29.574 INFO:tasks.rados.rados.0.smithi084.stdout:3928: done (2 left) 2024-09-20T22:57:29.574 INFO:tasks.rados.rados.0.smithi084.stdout:3929: done (1 left) 2024-09-20T22:57:29.574 INFO:tasks.rados.rados.0.smithi084.stdout:3930: done (0 left) 2024-09-20T22:57:29.574 INFO:tasks.rados.rados.0.smithi084.stdout:3931: write oid 47 current snap is 457 2024-09-20T22:57:29.574 INFO:tasks.rados.rados.0.smithi084.stdout:3931: seq_num 1335 ranges {437345=419190,1587814=591466} 2024-09-20T22:57:29.578 INFO:tasks.rados.rados.0.smithi084.stdout:3931: writing smithi08477203-47 from 437345 to 856535 tid 1 2024-09-20T22:57:29.586 INFO:tasks.rados.rados.0.smithi084.stdout:3931: writing smithi08477203-47 from 1587814 to 2179280 tid 2 2024-09-20T22:57:29.586 INFO:tasks.rados.rados.0.smithi084.stdout:3932: rollback oid 50 current snap is 457 2024-09-20T22:57:29.586 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 50 to 410 2024-09-20T22:57:29.586 INFO:tasks.rados.rados.0.smithi084.stdout:3931: finishing write tid 1 to smithi08477203-47 2024-09-20T22:57:29.586 INFO:tasks.rados.rados.0.smithi084.stdout:3933: read oid 37 snap -1 2024-09-20T22:57:29.587 INFO:tasks.rados.rados.0.smithi084.stdout:3933: expect (ObjNum 1189 snap 408 seq_num 1189) 2024-09-20T22:57:29.587 INFO:tasks.rados.rados.0.smithi084.stdout:3934: snap_remove snap 420 2024-09-20T22:57:29.590 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:29.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:29.591 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:57:29.692 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:57:29.718 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:29.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:29.719 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:57:29.809 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:29 smithi084 ceph-mon[40024]: pgmap v1353: 168 pgs: 1 active+clean+snaptrim, 2 active+undersized+degraded+remapped+backfill_toofull, 41 active+undersized, 1 active+remapped+backfill_toofull, 36 active+undersized+degraded, 87 active+clean; 825 MiB data, 2.3 GiB used, 624 GiB / 626 GiB avail; 299/1719 objects degraded (17.394%); 61/1719 objects misplaced (3.549%) 2024-09-20T22:57:29.810 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:29 smithi084 ceph-mon[29774]: pgmap v1353: 168 pgs: 1 active+clean+snaptrim, 2 active+undersized+degraded+remapped+backfill_toofull, 41 active+undersized, 1 active+remapped+backfill_toofull, 36 active+undersized+degraded, 87 active+clean; 825 MiB data, 2.3 GiB used, 624 GiB / 626 GiB avail; 299/1719 objects degraded (17.394%); 61/1719 objects misplaced (3.549%) 2024-09-20T22:57:29.820 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:57:29.842 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:29.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:29.843 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:57:29.943 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:57:29.970 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:29.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:29.971 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:57:30.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:29 smithi154 ceph-mon[33264]: pgmap v1353: 168 pgs: 1 active+clean+snaptrim, 2 active+undersized+degraded+remapped+backfill_toofull, 41 active+undersized, 1 active+remapped+backfill_toofull, 36 active+undersized+degraded, 87 active+clean; 825 MiB data, 2.3 GiB used, 624 GiB / 626 GiB avail; 299/1719 objects degraded (17.394%); 61/1719 objects misplaced (3.549%) 2024-09-20T22:57:30.004 INFO:journalctl@ceph.iscsi.iscsi.a.smithi154.stdout:Sep 20 22:57:29 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-iscsi-iscsi-a[59730]: debug there is no tcmu-runner data available 2024-09-20T22:57:30.054 DEBUG:teuthology.orchestra.run.smithi084:> sudo rm -f /var/log/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/exp.3.7.3 2024-09-20T22:57:30.072 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:57:30.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:30.113 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:30.113 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:57:30.214 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:57:30.236 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:30.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:30.237 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:57:30.338 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:57:30.366 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:30.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:30.367 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:57:30.468 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:57:30.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:30.503 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:30.503 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:57:30.505 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-20T22:57:30.505 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:57:30.583 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is unset 2024-09-20T22:57:30.583 INFO:tasks.rados.rados.0.smithi084.stdout:3934: done (3 left) 2024-09-20T22:57:30.583 INFO:tasks.rados.rados.0.smithi084.stdout:3935: write (excl) oid 17 current snap is 457 2024-09-20T22:57:30.583 INFO:tasks.rados.rados.0.smithi084.stdout:3935: seq_num 1336 ranges {454152=494439,1646296=735124} 2024-09-20T22:57:30.587 INFO:tasks.rados.rados.0.smithi084.stdout:3935: writing smithi08477203-17 from 454152 to 948591 tid 1 2024-09-20T22:57:30.598 INFO:tasks.rados.rados.0.smithi084.stdout:3935: writing smithi08477203-17 from 1646296 to 2381420 tid 2 2024-09-20T22:57:30.599 INFO:tasks.rados.rados.0.smithi084.stdout:3932: finishing rollback tid 1 to smithi08477203-50 2024-09-20T22:57:30.599 INFO:tasks.rados.rados.0.smithi084.stdout:3932: finishing rollback tid 2 to smithi08477203-50 2024-09-20T22:57:30.599 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 50 v 1907 (ObjNum 1128 snap 381 seq_num 1128) dirty exists 2024-09-20T22:57:30.599 INFO:tasks.rados.rados.0.smithi084.stdout:3933: expect (ObjNum 1189 snap 408 seq_num 1189) 2024-09-20T22:57:30.604 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:57:30.633 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:30.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:30.634 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:57:30.734 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:57:30.760 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:30.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:30.761 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:57:30.788 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-20T22:57:30.810 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:30 smithi084 ceph-mon[40024]: osdmap e969: 8 total, 7 up, 7 in 2024-09-20T22:57:30.810 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:30 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/1062152280' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:57:30.810 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:30 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:57:30.811 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:30 smithi084 ceph-mon[29774]: osdmap e969: 8 total, 7 up, 7 in 2024-09-20T22:57:30.811 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:30 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/1062152280' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:57:30.811 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:30 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:57:30.862 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:57:30.888 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:30.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:30.889 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:57:30.889 INFO:tasks.rados.rados.0.smithi084.stdout:3931: finishing write tid 2 to smithi08477203-47 2024-09-20T22:57:30.889 INFO:tasks.rados.rados.0.smithi084.stdout:3931: finishing write tid 3 to smithi08477203-47 2024-09-20T22:57:30.889 INFO:tasks.rados.rados.0.smithi084.stdout:3932: done (3 left) 2024-09-20T22:57:30.890 INFO:tasks.rados.rados.0.smithi084.stdout:3933: done (2 left) 2024-09-20T22:57:30.890 INFO:tasks.rados.rados.0.smithi084.stdout:3936: snap_remove snap 410 2024-09-20T22:57:30.990 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:57:31.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:30 smithi154 ceph-mon[33264]: osdmap e969: 8 total, 7 up, 7 in 2024-09-20T22:57:31.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:30 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/1062152280' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:57:31.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:30 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:57:31.015 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:31.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:31.016 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:57:31.116 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:57:31.139 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:31.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:31.140 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:57:31.241 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:57:31.246 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-20T22:57:31.268 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:31.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:31.270 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:57:31.371 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:57:31.398 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:31.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:31.399 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:57:31.499 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:57:31.520 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:31.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:31.521 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:57:31.586 INFO:tasks.rados.rados.0.smithi084.stdout:3936: done (2 left) 2024-09-20T22:57:31.587 INFO:tasks.rados.rados.0.smithi084.stdout:3937: read oid 4 snap 440 2024-09-20T22:57:31.587 INFO:tasks.rados.rados.0.smithi084.stdout:3937: expect (ObjNum 1246 snap 423 seq_num 1246) 2024-09-20T22:57:31.587 INFO:tasks.rados.rados.0.smithi084.stdout:3938: snap_remove snap 435 2024-09-20T22:57:31.622 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:57:31.643 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:31.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:31.644 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:57:31.744 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:57:31.770 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:31.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:31.771 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:57:31.823 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:31 smithi084 ceph-mon[40024]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:57:31.823 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:31 smithi084 ceph-mon[40024]: pgmap v1356: 168 pgs: 1 active+clean+snaptrim, 2 active+undersized+degraded+remapped+backfill_toofull, 41 active+undersized, 1 active+remapped+backfill_toofull, 36 active+undersized+degraded, 87 active+clean; 842 MiB data, 2.3 GiB used, 623 GiB / 626 GiB avail; 11 MiB/s rd, 3.5 MiB/s wr, 20 op/s; 303/1731 objects degraded (17.504%); 61/1731 objects misplaced (3.524%) 2024-09-20T22:57:31.823 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:31 smithi084 ceph-mon[40024]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:57:31.823 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:31 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T22:57:31.823 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:31 smithi084 ceph-mon[40024]: osdmap e970: 8 total, 7 up, 7 in 2024-09-20T22:57:31.823 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:31 smithi084 ceph-mon[40024]: 7.d scrub starts 2024-09-20T22:57:31.823 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:31 smithi084 ceph-mon[40024]: 7.d scrub ok 2024-09-20T22:57:31.823 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:31 smithi084 ceph-mon[29774]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:57:31.823 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:31 smithi084 ceph-mon[29774]: pgmap v1356: 168 pgs: 1 active+clean+snaptrim, 2 active+undersized+degraded+remapped+backfill_toofull, 41 active+undersized, 1 active+remapped+backfill_toofull, 36 active+undersized+degraded, 87 active+clean; 842 MiB data, 2.3 GiB used, 623 GiB / 626 GiB avail; 11 MiB/s rd, 3.5 MiB/s wr, 20 op/s; 303/1731 objects degraded (17.504%); 61/1731 objects misplaced (3.524%) 2024-09-20T22:57:31.823 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:31 smithi084 ceph-mon[29774]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:57:31.824 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:31 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T22:57:31.824 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:31 smithi084 ceph-mon[29774]: osdmap e970: 8 total, 7 up, 7 in 2024-09-20T22:57:31.824 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:31 smithi084 ceph-mon[29774]: 7.d scrub starts 2024-09-20T22:57:31.824 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:31 smithi084 ceph-mon[29774]: 7.d scrub ok 2024-09-20T22:57:31.861 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:57:31.872 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:57:31.893 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:31.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:31.894 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:57:31.994 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:57:32.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:31 smithi154 ceph-mon[33264]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:57:32.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:31 smithi154 ceph-mon[33264]: pgmap v1356: 168 pgs: 1 active+clean+snaptrim, 2 active+undersized+degraded+remapped+backfill_toofull, 41 active+undersized, 1 active+remapped+backfill_toofull, 36 active+undersized+degraded, 87 active+clean; 842 MiB data, 2.3 GiB used, 623 GiB / 626 GiB avail; 11 MiB/s rd, 3.5 MiB/s wr, 20 op/s; 303/1731 objects degraded (17.504%); 61/1731 objects misplaced (3.524%) 2024-09-20T22:57:32.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:31 smithi154 ceph-mon[33264]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:57:32.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:31 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T22:57:32.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:31 smithi154 ceph-mon[33264]: osdmap e970: 8 total, 7 up, 7 in 2024-09-20T22:57:32.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:31 smithi154 ceph-mon[33264]: 7.d scrub starts 2024-09-20T22:57:32.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:31 smithi154 ceph-mon[33264]: 7.d scrub ok 2024-09-20T22:57:32.016 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:32.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:32.017 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:57:32.117 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:57:32.139 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:32.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:32.141 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:57:32.241 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:57:32.267 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:32.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:32.268 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:57:32.354 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-20T22:57:32.354 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:57:32.363 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:57:32.369 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:57:32.390 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:32.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:32.391 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:57:32.492 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:57:32.519 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:32.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:32.520 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:57:32.598 INFO:tasks.rados.rados.0.smithi084.stdout:3938: done (3 left) 2024-09-20T22:57:32.598 INFO:tasks.rados.rados.0.smithi084.stdout:3939: copy_from oid 40 from oid 25 current snap is 457 2024-09-20T22:57:32.598 INFO:tasks.rados.rados.0.smithi084.stdout:3940: delete oid 29 current snap is 457 2024-09-20T22:57:32.598 INFO:tasks.rados.rados.0.smithi084.stdout:3931: finishing write tid 4 to smithi08477203-47 2024-09-20T22:57:32.598 INFO:tasks.rados.rados.0.smithi084.stdout:3931: oid 47 updating version 0 to 2860 2024-09-20T22:57:32.598 INFO:tasks.rados.rados.0.smithi084.stdout:3931: oid 47 version 2860 is already newer than 2858 2024-09-20T22:57:32.598 INFO:tasks.rados.rados.0.smithi084.stdout:3931: oid 47 version 2860 is already newer than 2859 2024-09-20T22:57:32.598 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 47 v 2860 (ObjNum 1335 snap 457 seq_num 1335) dirty exists 2024-09-20T22:57:32.598 INFO:tasks.rados.rados.0.smithi084.stdout:3931: left oid 47 (ObjNum 1335 snap 457 seq_num 1335) 2024-09-20T22:57:32.598 INFO:tasks.rados.rados.0.smithi084.stdout:3935: finishing write tid 1 to smithi08477203-17 2024-09-20T22:57:32.598 INFO:tasks.rados.rados.0.smithi084.stdout:3935: finishing write tid 2 to smithi08477203-17 2024-09-20T22:57:32.599 INFO:tasks.rados.rados.0.smithi084.stdout:3935: finishing write tid 3 to smithi08477203-17 2024-09-20T22:57:32.599 INFO:tasks.rados.rados.0.smithi084.stdout:3935: finishing write tid 4 to smithi08477203-17 2024-09-20T22:57:32.599 INFO:tasks.rados.rados.0.smithi084.stdout:3935: oid 17 updating version 0 to 2657 2024-09-20T22:57:32.599 INFO:tasks.rados.rados.0.smithi084.stdout:3935: oid 17 updating version 2657 to 2659 2024-09-20T22:57:32.599 INFO:tasks.rados.rados.0.smithi084.stdout:3935: oid 17 version 2659 is already newer than 2658 2024-09-20T22:57:32.599 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 17 v 2659 (ObjNum 1336 snap 457 seq_num 1336) dirty exists 2024-09-20T22:57:32.599 INFO:tasks.rados.rados.0.smithi084.stdout:3935: left oid 17 (ObjNum 1336 snap 457 seq_num 1336) 2024-09-20T22:57:32.599 INFO:tasks.rados.rados.0.smithi084.stdout:3937: expect (ObjNum 1246 snap 423 seq_num 1246) 2024-09-20T22:57:32.620 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:57:32.641 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:32.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:32.642 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:57:32.743 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:57:32.763 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:32.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:32.764 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:57:32.807 INFO:tasks.rados.rados.0.smithi084.stdout:3939: finishing copy_from racing read to smithi08477203-40 2024-09-20T22:57:32.807 INFO:tasks.rados.rados.0.smithi084.stdout:3939: finishing copy_from to smithi08477203-40 2024-09-20T22:57:32.807 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 40 v 3131 (ObjNum 1303 snap 443 seq_num 1303) dirty exists 2024-09-20T22:57:32.807 INFO:tasks.rados.rados.0.smithi084.stdout:3931: done (4 left) 2024-09-20T22:57:32.807 INFO:tasks.rados.rados.0.smithi084.stdout:3935: done (3 left) 2024-09-20T22:57:32.807 INFO:tasks.rados.rados.0.smithi084.stdout:3937: done (2 left) 2024-09-20T22:57:32.808 INFO:tasks.rados.rados.0.smithi084.stdout:3939: done (1 left) 2024-09-20T22:57:32.808 INFO:tasks.rados.rados.0.smithi084.stdout:3940: done (0 left) 2024-09-20T22:57:32.808 INFO:tasks.rados.rados.0.smithi084.stdout:3941: write (excl) oid 27 current snap is 457 2024-09-20T22:57:32.808 INFO:tasks.rados.rados.0.smithi084.stdout:3941: seq_num 1338 ranges {487766=644937,1646907=738793} 2024-09-20T22:57:32.812 INFO:tasks.rados.rados.0.smithi084.stdout:3941: writing smithi08477203-27 from 487766 to 1132703 tid 1 2024-09-20T22:57:32.820 INFO:tasks.rados.rados.0.smithi084.stdout:3941: writing smithi08477203-27 from 1646907 to 2385700 tid 2 2024-09-20T22:57:32.820 INFO:tasks.rados.rados.0.smithi084.stdout:3942: delete oid 17 current snap is 457 2024-09-20T22:57:32.820 INFO:tasks.rados.rados.0.smithi084.stdout:3941: finishing write tid 1 to smithi08477203-27 2024-09-20T22:57:32.822 INFO:tasks.rados.rados.0.smithi084.stdout:3942: done (1 left) 2024-09-20T22:57:32.822 INFO:tasks.rados.rados.0.smithi084.stdout:3943: read oid 23 snap -1 2024-09-20T22:57:32.822 INFO:tasks.rados.rados.0.smithi084.stdout:3943: expect (ObjNum 1325 snap 456 seq_num 1325) 2024-09-20T22:57:32.822 INFO:tasks.rados.rados.0.smithi084.stdout:3944: snap_remove snap 369 2024-09-20T22:57:32.864 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:32 smithi084 ceph-mon[29774]: osdmap e971: 8 total, 7 up, 7 in 2024-09-20T22:57:32.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:32 smithi084 ceph-mon[40024]: osdmap e971: 8 total, 7 up, 7 in 2024-09-20T22:57:32.865 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:57:32.889 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:32.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:32.890 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:57:32.991 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:57:33.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:32 smithi154 ceph-mon[33264]: osdmap e971: 8 total, 7 up, 7 in 2024-09-20T22:57:33.015 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:33.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:33.016 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:57:33.116 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:57:33.139 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:33.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:33.140 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:57:33.236 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.2 dump_blocked_ops 2024-09-20T22:57:33.240 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:57:33.262 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:33.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:33.263 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:57:33.364 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:57:33.386 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:33.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:33.390 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:57:33.491 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:57:33.512 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:33.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:33.513 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:57:33.598 INFO:tasks.rados.rados.0.smithi084.stdout:3944: done (2 left) 2024-09-20T22:57:33.599 INFO:tasks.rados.rados.0.smithi084.stdout:3945: copy_from oid 46 from oid 36 current snap is 457 2024-09-20T22:57:33.599 INFO:tasks.rados.rados.0.smithi084.stdout:3946: snap_create 2024-09-20T22:57:33.599 INFO:tasks.rados.rados.0.smithi084.stdout:3941: finishing write tid 2 to smithi08477203-27 2024-09-20T22:57:33.599 INFO:tasks.rados.rados.0.smithi084.stdout:3941: finishing write tid 3 to smithi08477203-27 2024-09-20T22:57:33.599 INFO:tasks.rados.rados.0.smithi084.stdout:3941: finishing write tid 4 to smithi08477203-27 2024-09-20T22:57:33.599 INFO:tasks.rados.rados.0.smithi084.stdout:3941: oid 27 updating version 0 to 2608 2024-09-20T22:57:33.599 INFO:tasks.rados.rados.0.smithi084.stdout:3941: oid 27 version 2608 is already newer than 2607 2024-09-20T22:57:33.599 INFO:tasks.rados.rados.0.smithi084.stdout:3941: oid 27 updating version 2608 to 2609 2024-09-20T22:57:33.599 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 27 v 2609 (ObjNum 1338 snap 457 seq_num 1338) dirty exists 2024-09-20T22:57:33.599 INFO:tasks.rados.rados.0.smithi084.stdout:3941: left oid 27 (ObjNum 1338 snap 457 seq_num 1338) 2024-09-20T22:57:33.599 INFO:tasks.rados.rados.0.smithi084.stdout:3943: expect (ObjNum 1325 snap 456 seq_num 1325) 2024-09-20T22:57:33.614 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:57:33.635 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:33.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:33.636 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:57:33.677 INFO:tasks.rados.rados.0.smithi084.stdout:3945: finishing copy_from racing read to smithi08477203-46 2024-09-20T22:57:33.677 INFO:tasks.rados.rados.0.smithi084.stdout:3945: finishing copy_from to smithi08477203-46 2024-09-20T22:57:33.677 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 46 v 2650 (ObjNum 1314 snap 453 seq_num 1314) dirty exists 2024-09-20T22:57:33.701 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:57:33.701 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:57:33.718 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:57:33.718 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:57:33.737 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:57:33.740 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:57:33.740 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:57:33.741 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:57:33.741 INFO:teuthology.orchestra.run.smithi084.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-20T22:57:33.758 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:57:33.758 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:57:33.759 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:33.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:33.760 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:57:33.781 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:57:33.781 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:57:33.797 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:57:33.797 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:57:33.813 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:57:33.814 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:57:33.856 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-20T22:57:33.861 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:57:33.863 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:33 smithi084 ceph-mon[29774]: pgmap v1359: 168 pgs: 1 active+clean+snaptrim, 2 active+undersized+degraded+remapped+backfill_toofull, 41 active+undersized, 1 active+remapped+backfill_toofull, 36 active+undersized+degraded, 87 active+clean; 842 MiB data, 2.3 GiB used, 623 GiB / 626 GiB avail; 11 MiB/s rd, 3.5 MiB/s wr, 20 op/s; 303/1731 objects degraded (17.504%); 61/1731 objects misplaced (3.524%) 2024-09-20T22:57:33.863 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:33 smithi084 ceph-mon[29774]: osdmap e972: 8 total, 7 up, 7 in 2024-09-20T22:57:33.863 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:33 smithi084 ceph-mon[29774]: 4.2 scrub starts 2024-09-20T22:57:33.863 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:33 smithi084 ceph-mon[29774]: 4.2 scrub ok 2024-09-20T22:57:33.863 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:33 smithi084 ceph-mon[29774]: 7.11 scrub starts 2024-09-20T22:57:33.863 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:33 smithi084 ceph-mon[29774]: 7.11 scrub ok 2024-09-20T22:57:33.863 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:33 smithi084 ceph-mon[29774]: Health check update: Degraded data redundancy: 303/1731 objects degraded (17.504%), 38 pgs degraded (PG_DEGRADED) 2024-09-20T22:57:33.863 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:33 smithi084 ceph-mon[40024]: pgmap v1359: 168 pgs: 1 active+clean+snaptrim, 2 active+undersized+degraded+remapped+backfill_toofull, 41 active+undersized, 1 active+remapped+backfill_toofull, 36 active+undersized+degraded, 87 active+clean; 842 MiB data, 2.3 GiB used, 623 GiB / 626 GiB avail; 11 MiB/s rd, 3.5 MiB/s wr, 20 op/s; 303/1731 objects degraded (17.504%); 61/1731 objects misplaced (3.524%) 2024-09-20T22:57:33.864 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:33 smithi084 ceph-mon[40024]: osdmap e972: 8 total, 7 up, 7 in 2024-09-20T22:57:33.864 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:33 smithi084 ceph-mon[40024]: 4.2 scrub starts 2024-09-20T22:57:33.864 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:33 smithi084 ceph-mon[40024]: 4.2 scrub ok 2024-09-20T22:57:33.864 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:33 smithi084 ceph-mon[40024]: 7.11 scrub starts 2024-09-20T22:57:33.864 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:33 smithi084 ceph-mon[40024]: 7.11 scrub ok 2024-09-20T22:57:33.864 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:33 smithi084 ceph-mon[40024]: Health check update: Degraded data redundancy: 303/1731 objects degraded (17.504%), 38 pgs degraded (PG_DEGRADED) 2024-09-20T22:57:33.882 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:33.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:33.883 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:57:33.984 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:57:34.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:33 smithi154 ceph-mon[33264]: pgmap v1359: 168 pgs: 1 active+clean+snaptrim, 2 active+undersized+degraded+remapped+backfill_toofull, 41 active+undersized, 1 active+remapped+backfill_toofull, 36 active+undersized+degraded, 87 active+clean; 842 MiB data, 2.3 GiB used, 623 GiB / 626 GiB avail; 11 MiB/s rd, 3.5 MiB/s wr, 20 op/s; 303/1731 objects degraded (17.504%); 61/1731 objects misplaced (3.524%) 2024-09-20T22:57:34.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:33 smithi154 ceph-mon[33264]: osdmap e972: 8 total, 7 up, 7 in 2024-09-20T22:57:34.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:33 smithi154 ceph-mon[33264]: 4.2 scrub starts 2024-09-20T22:57:34.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:33 smithi154 ceph-mon[33264]: 4.2 scrub ok 2024-09-20T22:57:34.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:33 smithi154 ceph-mon[33264]: 7.11 scrub starts 2024-09-20T22:57:34.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:33 smithi154 ceph-mon[33264]: 7.11 scrub ok 2024-09-20T22:57:34.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:33 smithi154 ceph-mon[33264]: Health check update: Degraded data redundancy: 303/1731 objects degraded (17.504%), 38 pgs degraded (PG_DEGRADED) 2024-09-20T22:57:34.005 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:34.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:34.006 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:57:34.107 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:57:34.127 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:34.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:34.128 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:57:34.229 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:57:34.250 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:34.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:34.251 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:57:34.345 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-20T22:57:34.345 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2024-09-20T22:57:34.345 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2024-09-20T22:57:34.345 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2024-09-20T22:57:34.345 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-20T22:57:34.352 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:57:34.374 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:34.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:34.375 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:57:34.475 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:57:34.497 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:34.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:34.498 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:57:34.598 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:57:34.607 INFO:tasks.rados.rados.0.smithi084.stdout:3941: done (3 left) 2024-09-20T22:57:34.607 INFO:tasks.rados.rados.0.smithi084.stdout:3943: done (2 left) 2024-09-20T22:57:34.607 INFO:tasks.rados.rados.0.smithi084.stdout:3945: done (1 left) 2024-09-20T22:57:34.607 INFO:tasks.rados.rados.0.smithi084.stdout:3946: done (0 left) 2024-09-20T22:57:34.607 INFO:tasks.rados.rados.0.smithi084.stdout:3947: read oid 32 snap -1 2024-09-20T22:57:34.607 INFO:tasks.rados.rados.0.smithi084.stdout:3947: expect (ObjNum 1289 snap 436 seq_num 1289) 2024-09-20T22:57:34.607 INFO:tasks.rados.rados.0.smithi084.stdout:3948: snap_remove snap 405 2024-09-20T22:57:34.620 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:34.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:34.621 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:57:34.722 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:57:34.743 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:34.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:34.744 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:57:34.845 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:57:34.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:34 smithi084 ceph-mon[29774]: osdmap e973: 8 total, 7 up, 7 in 2024-09-20T22:57:34.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:34 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/2987551371' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:57:34.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:34 smithi084 ceph-mon[40024]: osdmap e973: 8 total, 7 up, 7 in 2024-09-20T22:57:34.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:34 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/2987551371' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:57:34.867 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:34.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:34.868 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:57:34.968 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:57:34.989 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:34.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:34.990 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:57:35.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:34 smithi154 ceph-mon[33264]: osdmap e973: 8 total, 7 up, 7 in 2024-09-20T22:57:35.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:34 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/2987551371' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:57:35.081 INFO:tasks.thrashosds.thrasher:in_osds: [2, 3, 5, 6, 0, 4, 1] out_osds: [7] dead_osds: [3] live_osds: [0, 1, 2, 4, 5, 6, 7] 2024-09-20T22:57:35.081 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-20T22:57:35.082 INFO:tasks.thrashosds.thrasher:Reviving osd 3 2024-09-20T22:57:35.082 INFO:tasks.cephadm.osd.3:Restarting osd.3 (starting--it wasn't running)... 2024-09-20T22:57:35.082 DEBUG:teuthology.orchestra.run.smithi084:osd.3> sudo journalctl -f -n 0 -u ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490@osd.3.service 2024-09-20T22:57:35.084 DEBUG:teuthology.orchestra.run.smithi084:> sudo systemctl start ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490@osd.3 2024-09-20T22:57:35.091 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:57:35.112 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:35.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:35.113 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:57:35.214 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:57:35.235 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:35.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:35.236 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:57:35.278 INFO:journalctl@ceph.osd.3.smithi084.stdout:Sep 20 22:57:35 smithi084 systemd[1]: Starting Ceph osd.3 for a9c0bc70-77a0-11ef-baf5-efdc52797490... 2024-09-20T22:57:35.279 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.2 dump_historic_ops 2024-09-20T22:57:35.337 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:57:35.358 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:35.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:35.359 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:57:35.460 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:57:35.481 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:35.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:35.482 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:57:35.583 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:57:35.604 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:35.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:35.605 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:57:35.612 INFO:tasks.rados.rados.0.smithi084.stdout:3948: done (1 left) 2024-09-20T22:57:35.612 INFO:tasks.rados.rados.0.smithi084.stdout:3949: copy_from oid 2 from oid 48 current snap is 458 2024-09-20T22:57:35.612 INFO:tasks.rados.rados.0.smithi084.stdout:3950: delete oid 30 current snap is 458 2024-09-20T22:57:35.612 INFO:tasks.rados.rados.0.smithi084.stdout:3947: expect (ObjNum 1289 snap 436 seq_num 1289) 2024-09-20T22:57:35.706 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:57:35.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:35.729 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:35.729 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:57:35.735 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-20T22:57:35.736 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-20T22:57:35.769 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:57:35.789 INFO:tasks.rados.rados.0.smithi084.stdout:3947: done (2 left) 2024-09-20T22:57:35.789 INFO:tasks.rados.rados.0.smithi084.stdout:3950: done (1 left) 2024-09-20T22:57:35.789 INFO:tasks.rados.rados.0.smithi084.stdout:3951: copy_from oid 45 from oid 29 current snap is 458 2024-09-20T22:57:35.789 INFO:tasks.rados.rados.0.smithi084.stdout:3949: finishing copy_from racing read to smithi08477203-2 2024-09-20T22:57:35.789 INFO:tasks.rados.rados.0.smithi084.stdout:3949: finishing copy_from to smithi08477203-2 2024-09-20T22:57:35.790 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 2 v 1965 (ObjNum 1141 snap 385 seq_num 1141) dirty exists 2024-09-20T22:57:35.790 INFO:tasks.rados.rados.0.smithi084.stdout:3949: done (1 left) 2024-09-20T22:57:35.790 INFO:tasks.rados.rados.0.smithi084.stdout:3952: delete oid 28 current snap is 458 2024-09-20T22:57:35.792 INFO:tasks.rados.rados.0.smithi084.stdout:3951: finishing copy_from to smithi08477203-45 2024-09-20T22:57:35.792 INFO:tasks.rados.rados.0.smithi084.stdout:3951: got expected ENOENT (src dne) 2024-09-20T22:57:35.792 INFO:tasks.rados.rados.0.smithi084.stdout:3951: finishing copy_from racing read to smithi08477203-45 2024-09-20T22:57:35.792 INFO:tasks.rados.rados.0.smithi084.stdout:3951: done (1 left) 2024-09-20T22:57:35.793 INFO:tasks.rados.rados.0.smithi084.stdout:3952: done (0 left) 2024-09-20T22:57:35.793 INFO:tasks.rados.rados.0.smithi084.stdout:3953: rollback oid 21 current snap is 458 2024-09-20T22:57:35.793 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 21 to 419 2024-09-20T22:57:35.793 INFO:tasks.rados.rados.0.smithi084.stdout:3954: read oid 13 snap 414 2024-09-20T22:57:35.793 INFO:tasks.rados.rados.0.smithi084.stdout:3954: expect deleted 2024-09-20T22:57:35.793 INFO:tasks.rados.rados.0.smithi084.stdout:3955: snap_remove snap 440 2024-09-20T22:57:35.805 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:57:35.830 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:57:35.852 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:35.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:35.853 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:57:35.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:35 smithi084 ceph-mon[29774]: pgmap v1362: 168 pgs: 1 active+clean+remapped, 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+clean+snaptrim, 41 active+undersized, 1 active+remapped+backfill_toofull, 36 active+undersized+degraded, 87 active+clean; 827 MiB data, 2.3 GiB used, 623 GiB / 626 GiB avail; 300/1704 objects degraded (17.606%); 55/1704 objects misplaced (3.228%) 2024-09-20T22:57:35.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:35 smithi084 ceph-mon[29774]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-20T22:57:35.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:35 smithi084 ceph-mon[29774]: osdmap e974: 8 total, 7 up, 7 in 2024-09-20T22:57:35.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:35 smithi084 ceph-mon[29774]: 3.4 scrub starts 2024-09-20T22:57:35.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:35 smithi084 ceph-mon[29774]: 3.4 scrub ok 2024-09-20T22:57:35.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:35 smithi084 ceph-mon[40024]: pgmap v1362: 168 pgs: 1 active+clean+remapped, 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+clean+snaptrim, 41 active+undersized, 1 active+remapped+backfill_toofull, 36 active+undersized+degraded, 87 active+clean; 827 MiB data, 2.3 GiB used, 623 GiB / 626 GiB avail; 300/1704 objects degraded (17.606%); 55/1704 objects misplaced (3.228%) 2024-09-20T22:57:35.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:35 smithi084 ceph-mon[40024]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-20T22:57:35.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:35 smithi084 ceph-mon[40024]: osdmap e974: 8 total, 7 up, 7 in 2024-09-20T22:57:35.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:35 smithi084 ceph-mon[40024]: 3.4 scrub starts 2024-09-20T22:57:35.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:35 smithi084 ceph-mon[40024]: 3.4 scrub ok 2024-09-20T22:57:35.954 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:57:35.975 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:35.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:35.976 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:57:36.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:35 smithi154 ceph-mon[33264]: pgmap v1362: 168 pgs: 1 active+clean+remapped, 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+clean+snaptrim, 41 active+undersized, 1 active+remapped+backfill_toofull, 36 active+undersized+degraded, 87 active+clean; 827 MiB data, 2.3 GiB used, 623 GiB / 626 GiB avail; 300/1704 objects degraded (17.606%); 55/1704 objects misplaced (3.228%) 2024-09-20T22:57:36.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:35 smithi154 ceph-mon[33264]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-20T22:57:36.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:35 smithi154 ceph-mon[33264]: osdmap e974: 8 total, 7 up, 7 in 2024-09-20T22:57:36.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:35 smithi154 ceph-mon[33264]: 3.4 scrub starts 2024-09-20T22:57:36.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:35 smithi154 ceph-mon[33264]: 3.4 scrub ok 2024-09-20T22:57:36.077 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:57:36.099 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:36.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:36.100 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:57:36.157 INFO:journalctl@ceph.osd.3.smithi084.stdout:Sep 20 22:57:35 smithi084 podman[212682]: 2024-09-20 22:57:35.810208523 +0000 UTC m=+0.019089415 image pull cc9251b744a3faafca7c3095f79bfff81873afe823f31a03a11e015a34abcdbd quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5 2024-09-20T22:57:36.201 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:57:36.222 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:36.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:36.223 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:57:36.323 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:57:36.346 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:36.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:36.347 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:57:36.447 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:57:36.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:36.469 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:36.469 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:57:36.570 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:57:36.592 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:36.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:36.593 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:57:36.615 INFO:tasks.rados.rados.0.smithi084.stdout:3955: done (2 left) 2024-09-20T22:57:36.615 INFO:tasks.rados.rados.0.smithi084.stdout:3956: copy_from oid 7 from oid 31 current snap is 458 2024-09-20T22:57:36.615 INFO:tasks.rados.rados.0.smithi084.stdout:3957: snap_create 2024-09-20T22:57:36.615 INFO:tasks.rados.rados.0.smithi084.stdout:3953: finishing rollback tid 0 to smithi08477203-21 2024-09-20T22:57:36.616 INFO:tasks.rados.rados.0.smithi084.stdout:3953: finishing rollback tid 1 to smithi08477203-21 2024-09-20T22:57:36.616 INFO:tasks.rados.rados.0.smithi084.stdout:3953: finishing rollback tid 2 to smithi08477203-21 2024-09-20T22:57:36.616 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 21 v 2689 (ObjNum 1215 snap 413 seq_num 1215) dirty exists 2024-09-20T22:57:36.620 INFO:tasks.rados.rados.0.smithi084.stdout:3956: finishing copy_from to smithi08477203-7 2024-09-20T22:57:36.620 INFO:tasks.rados.rados.0.smithi084.stdout:3956: got expected ENOENT (src dne) 2024-09-20T22:57:36.620 INFO:tasks.rados.rados.0.smithi084.stdout:3956: finishing copy_from racing read to smithi08477203-7 2024-09-20T22:57:36.694 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:57:36.715 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:36.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:36.716 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:57:36.721 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:36 smithi084 ceph-mon[40024]: osdmap e975: 8 total, 7 up, 7 in 2024-09-20T22:57:36.721 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:36 smithi084 ceph-mon[40024]: 5.10 scrub starts 2024-09-20T22:57:36.722 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:36 smithi084 ceph-mon[40024]: 5.10 scrub ok 2024-09-20T22:57:36.722 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:36 smithi084 ceph-mon[29774]: osdmap e975: 8 total, 7 up, 7 in 2024-09-20T22:57:36.722 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:36 smithi084 ceph-mon[29774]: 5.10 scrub starts 2024-09-20T22:57:36.722 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:36 smithi084 ceph-mon[29774]: 5.10 scrub ok 2024-09-20T22:57:36.738 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-20T22:57:36.738 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 20, 2024-09-20T22:57:36.738 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 600, 2024-09-20T22:57:36.739 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [ 2024-09-20T22:57:36.739 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.739 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24652.0:6378 7.2 e639/600 7:4d20232a:::smithi08477203-34:head v 639'665, mlcod=629'662)", 2024-09-20T22:57:36.739 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:51:59.278714+0000", 2024-09-20T22:57:36.739 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 337.45691483100001, 2024-09-20T22:57:36.739 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.015218334, 2024-09-20T22:57:36.739 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:57:36.739 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:57:36.739 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:57:36.739 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:57:36.739 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.739 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:57:36.739 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:59.278714+0000", 2024-09-20T22:57:36.739 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:57:36.740 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.740 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.740 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:57:36.740 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:59.278714+0000", 2024-09-20T22:57:36.740 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:57:36.740 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.740 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.740 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:57:36.740 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:59.278714+0000", 2024-09-20T22:57:36.740 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.6e-07 2024-09-20T22:57:36.740 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.740 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.740 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:57:36.740 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:59.281091+0000", 2024-09-20T22:57:36.741 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0023769239999999999 2024-09-20T22:57:36.741 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.741 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.741 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:57:36.741 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:59.281110+0000", 2024-09-20T22:57:36.741 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.9230000000000001e-05 2024-09-20T22:57:36.741 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.741 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.741 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:57:36.741 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:59.281122+0000", 2024-09-20T22:57:36.741 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1997999999999999e-05 2024-09-20T22:57:36.741 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.741 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.741 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:57:36.741 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:59.281186+0000", 2024-09-20T22:57:36.742 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.3998999999999995e-05 2024-09-20T22:57:36.742 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.742 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.742 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:57:36.742 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:59.281209+0000", 2024-09-20T22:57:36.742 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.2654999999999999e-05 2024-09-20T22:57:36.742 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.742 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.742 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:57:36.742 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:59.293866+0000", 2024-09-20T22:57:36.742 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.012656745 2024-09-20T22:57:36.742 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.742 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.742 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:57:36.743 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:51:59.293932+0000", 2024-09-20T22:57:36.743 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.6523e-05 2024-09-20T22:57:36.743 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:57:36.743 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:57:36.743 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:57:36.743 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.743 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.743 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(osd.5.0:520 7.b e643/602 7:dc95255e:::smithi08477203-1:107 v 643'959, mlcod=637'955)", 2024-09-20T22:57:36.743 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:52:03.289043+0000", 2024-09-20T22:57:36.743 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 333.44658572200001, 2024-09-20T22:57:36.743 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.015367396, 2024-09-20T22:57:36.743 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:57:36.743 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:57:36.743 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:57:36.744 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:57:36.744 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.744 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:57:36.744 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:52:03.289043+0000", 2024-09-20T22:57:36.744 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:57:36.744 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.744 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.744 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:57:36.744 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:52:03.289043+0000", 2024-09-20T22:57:36.744 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:57:36.744 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.744 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.744 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:57:36.745 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:52:03.289044+0000", 2024-09-20T22:57:36.745 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.8100000000000002e-07 2024-09-20T22:57:36.745 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.745 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.745 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:57:36.745 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:52:03.289056+0000", 2024-09-20T22:57:36.745 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2408999999999999e-05 2024-09-20T22:57:36.745 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.745 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.745 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:57:36.745 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:52:03.289071+0000", 2024-09-20T22:57:36.745 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.4331000000000001e-05 2024-09-20T22:57:36.745 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.746 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.746 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:57:36.746 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:52:03.289079+0000", 2024-09-20T22:57:36.746 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.1009999999999992e-06 2024-09-20T22:57:36.746 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.746 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.746 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:57:36.746 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:52:03.289168+0000", 2024-09-20T22:57:36.746 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.9614e-05 2024-09-20T22:57:36.746 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.746 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.746 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:57:36.746 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:52:03.289189+0000", 2024-09-20T22:57:36.746 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.0701999999999999e-05 2024-09-20T22:57:36.747 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.747 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.747 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:57:36.747 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:52:03.304364+0000", 2024-09-20T22:57:36.747 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.015175414 2024-09-20T22:57:36.747 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.747 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.747 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:57:36.747 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:52:03.304410+0000", 2024-09-20T22:57:36.747 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.6044000000000002e-05 2024-09-20T22:57:36.747 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:57:36.747 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:57:36.747 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:57:36.748 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.748 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.748 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(osd.5.0:521 7.b e643/602 7:d960cad4:::smithi08477203-18:107 v 643'961, mlcod=637'955)", 2024-09-20T22:57:36.748 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:52:03.289097+0000", 2024-09-20T22:57:36.748 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 333.44653207499999, 2024-09-20T22:57:36.748 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.016179973, 2024-09-20T22:57:36.748 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:57:36.748 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:57:36.748 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:57:36.748 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:57:36.748 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.748 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:57:36.748 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:52:03.289097+0000", 2024-09-20T22:57:36.748 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:57:36.748 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.749 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.749 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:57:36.749 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:52:03.289097+0000", 2024-09-20T22:57:36.749 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:57:36.749 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.749 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.749 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:57:36.749 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:52:03.289097+0000", 2024-09-20T22:57:36.749 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.3599999999999999e-07 2024-09-20T22:57:36.749 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.749 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.749 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:57:36.749 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:52:03.289105+0000", 2024-09-20T22:57:36.749 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.8869999999999992e-06 2024-09-20T22:57:36.750 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.750 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.750 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:57:36.750 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:52:03.289111+0000", 2024-09-20T22:57:36.750 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.2419999999999997e-06 2024-09-20T22:57:36.750 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.750 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.750 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:57:36.750 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:52:03.289116+0000", 2024-09-20T22:57:36.750 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.6569999999999996e-06 2024-09-20T22:57:36.750 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.750 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.751 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:57:36.751 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:52:03.291170+0000", 2024-09-20T22:57:36.751 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0020537989999999998 2024-09-20T22:57:36.751 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.751 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.751 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:57:36.751 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:52:03.291192+0000", 2024-09-20T22:57:36.751 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.2548000000000002e-05 2024-09-20T22:57:36.751 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.751 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.751 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:57:36.751 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:52:03.305241+0000", 2024-09-20T22:57:36.751 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.01404832 2024-09-20T22:57:36.751 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.752 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.752 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:57:36.752 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:52:03.305277+0000", 2024-09-20T22:57:36.752 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.6084000000000002e-05 2024-09-20T22:57:36.752 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:57:36.752 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:57:36.752 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:57:36.752 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.752 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.752 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24652.0:6790 7.0 e687/632 7:064f66ab:::smithi08477203-45:head v 687'535, mlcod=687'532)", 2024-09-20T22:57:36.752 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:52:47.711923+0000", 2024-09-20T22:57:36.752 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 289.02370526099998, 2024-09-20T22:57:36.752 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014321054999999999, 2024-09-20T22:57:36.752 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:57:36.753 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:57:36.753 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:57:36.754 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:57:36.754 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.754 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:57:36.754 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:52:47.711923+0000", 2024-09-20T22:57:36.754 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:57:36.754 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.754 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.754 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:57:36.754 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:52:47.711923+0000", 2024-09-20T22:57:36.754 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:57:36.754 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.754 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.754 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:57:36.755 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:52:47.711924+0000", 2024-09-20T22:57:36.755 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.33e-07 2024-09-20T22:57:36.755 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.755 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.755 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:57:36.755 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:52:47.720461+0000", 2024-09-20T22:57:36.755 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0085366850000000005 2024-09-20T22:57:36.755 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.755 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.755 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:57:36.755 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:52:47.720487+0000", 2024-09-20T22:57:36.755 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.5463e-05 2024-09-20T22:57:36.755 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.755 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.756 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:57:36.756 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:52:47.720540+0000", 2024-09-20T22:57:36.756 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.3356999999999998e-05 2024-09-20T22:57:36.756 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.756 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.756 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:57:36.756 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:52:47.720631+0000", 2024-09-20T22:57:36.756 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.1305999999999996e-05 2024-09-20T22:57:36.756 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.756 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.756 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:57:36.756 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:52:47.720662+0000", 2024-09-20T22:57:36.756 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.1149000000000003e-05 2024-09-20T22:57:36.756 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.757 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.757 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:57:36.757 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:52:47.726196+0000", 2024-09-20T22:57:36.757 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0055332430000000002 2024-09-20T22:57:36.757 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.757 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.757 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:57:36.757 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:52:47.726244+0000", 2024-09-20T22:57:36.757 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.8918999999999997e-05 2024-09-20T22:57:36.757 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:57:36.757 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:57:36.757 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:57:36.758 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.758 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.758 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24652.0:6829 7.5 e688/456 7:a160b5d4:::smithi08477203-16:head v 688'328, mlcod=684'324)", 2024-09-20T22:57:36.758 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:52:48.309561+0000", 2024-09-20T22:57:36.758 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 288.42606747600001, 2024-09-20T22:57:36.758 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.017015579999999999, 2024-09-20T22:57:36.758 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:57:36.758 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:57:36.758 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:57:36.758 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:57:36.758 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.758 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:57:36.758 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:52:48.309561+0000", 2024-09-20T22:57:36.759 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:57:36.759 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.759 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.759 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:57:36.759 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:52:48.309561+0000", 2024-09-20T22:57:36.759 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:57:36.759 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.759 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.759 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:57:36.759 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:52:48.309562+0000", 2024-09-20T22:57:36.759 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.7599999999999996e-07 2024-09-20T22:57:36.759 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.759 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.759 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:57:36.760 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:52:48.310336+0000", 2024-09-20T22:57:36.760 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00077435700000000004 2024-09-20T22:57:36.760 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.760 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.760 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:57:36.760 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:52:48.310355+0000", 2024-09-20T22:57:36.760 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.8763e-05 2024-09-20T22:57:36.760 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.760 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.760 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:57:36.760 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:52:48.310365+0000", 2024-09-20T22:57:36.760 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.4420000000000004e-06 2024-09-20T22:57:36.760 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.760 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.761 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:57:36.761 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:52:48.310463+0000", 2024-09-20T22:57:36.761 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.8177999999999994e-05 2024-09-20T22:57:36.761 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.761 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.761 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:57:36.761 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:52:48.310494+0000", 2024-09-20T22:57:36.761 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.1284000000000001e-05 2024-09-20T22:57:36.761 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.761 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.761 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:57:36.761 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:52:48.326521+0000", 2024-09-20T22:57:36.761 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.016026757999999999 2024-09-20T22:57:36.761 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.761 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.762 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:57:36.762 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:52:48.326577+0000", 2024-09-20T22:57:36.762 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.5921999999999998e-05 2024-09-20T22:57:36.762 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:57:36.762 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:57:36.762 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:57:36.762 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.762 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.762 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24652.0:7143 7.0 e719/632 7:0f524219:::smithi08477203-50:head v 719'618, mlcod=714'614)", 2024-09-20T22:57:36.762 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:53:19.300601+0000", 2024-09-20T22:57:36.762 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 257.43502759699999, 2024-09-20T22:57:36.762 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.016058294000000001, 2024-09-20T22:57:36.762 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:57:36.762 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:57:36.763 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:57:36.763 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:57:36.763 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.763 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:57:36.763 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:19.300601+0000", 2024-09-20T22:57:36.763 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:57:36.763 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.763 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.763 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:57:36.763 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:19.300601+0000", 2024-09-20T22:57:36.763 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:57:36.763 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.763 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.764 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:57:36.764 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:19.300601+0000", 2024-09-20T22:57:36.764 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.6400000000000001e-07 2024-09-20T22:57:36.764 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.764 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.764 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:57:36.764 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:19.307294+0000", 2024-09-20T22:57:36.764 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0066925200000000004 2024-09-20T22:57:36.764 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.764 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.764 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:57:36.764 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:19.307307+0000", 2024-09-20T22:57:36.764 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3431999999999999e-05 2024-09-20T22:57:36.764 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.765 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.765 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:57:36.765 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:19.307316+0000", 2024-09-20T22:57:36.765 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.5099999999999998e-06 2024-09-20T22:57:36.765 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.765 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.765 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:57:36.765 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:19.307385+0000", 2024-09-20T22:57:36.765 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.8771999999999998e-05 2024-09-20T22:57:36.765 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.765 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.765 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:57:36.765 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:19.307413+0000", 2024-09-20T22:57:36.765 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.8373000000000001e-05 2024-09-20T22:57:36.765 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.766 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.766 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:57:36.766 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:19.316602+0000", 2024-09-20T22:57:36.766 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0091889950000000001 2024-09-20T22:57:36.766 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.766 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.766 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:57:36.766 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:19.316659+0000", 2024-09-20T22:57:36.766 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.7528000000000002e-05 2024-09-20T22:57:36.766 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:57:36.766 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:57:36.766 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:57:36.766 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.766 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.767 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24652.0:7356 7.a e738/94 7:5aaef466:::smithi08477203-40:head v 738'878, mlcod=736'875)", 2024-09-20T22:57:36.767 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:53:38.313061+0000", 2024-09-20T22:57:36.767 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 238.422567993, 2024-09-20T22:57:36.767 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.027472555999999999, 2024-09-20T22:57:36.767 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:57:36.767 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:57:36.767 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:57:36.767 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:57:36.767 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.767 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:57:36.767 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:38.313061+0000", 2024-09-20T22:57:36.767 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:57:36.767 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.768 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.768 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:57:36.768 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:38.313061+0000", 2024-09-20T22:57:36.768 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:57:36.768 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.768 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.768 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:57:36.768 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:38.313061+0000", 2024-09-20T22:57:36.768 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.0399999999999996e-07 2024-09-20T22:57:36.768 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.768 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.768 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:57:36.768 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:38.316261+0000", 2024-09-20T22:57:36.768 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0031994129999999999 2024-09-20T22:57:36.769 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.769 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.769 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:57:36.769 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:38.316282+0000", 2024-09-20T22:57:36.769 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.1090999999999998e-05 2024-09-20T22:57:36.769 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.769 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.769 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:57:36.769 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:38.316295+0000", 2024-09-20T22:57:36.769 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3226e-05 2024-09-20T22:57:36.769 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.769 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.769 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:57:36.769 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:38.316385+0000", 2024-09-20T22:57:36.770 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.0048000000000002e-05 2024-09-20T22:57:36.770 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.770 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.770 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:57:36.770 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:38.316411+0000", 2024-09-20T22:57:36.770 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.6401e-05 2024-09-20T22:57:36.770 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.770 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.770 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:57:36.770 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:38.340477+0000", 2024-09-20T22:57:36.770 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.024065468999999999 2024-09-20T22:57:36.770 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.770 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.771 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:57:36.771 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:38.340533+0000", 2024-09-20T22:57:36.771 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.6403999999999997e-05 2024-09-20T22:57:36.771 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:57:36.771 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:57:36.771 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:57:36.771 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.771 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.771 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24652.0:7432 7.0 e744/632 7:064f66ab:::smithi08477203-45:head v 744'684, mlcod=743'681)", 2024-09-20T22:57:36.771 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:53:44.436593+0000", 2024-09-20T22:57:36.771 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 232.299036091, 2024-09-20T22:57:36.771 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.040892389000000001, 2024-09-20T22:57:36.771 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:57:36.772 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:57:36.772 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:57:36.772 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:57:36.772 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.772 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:57:36.772 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:44.436593+0000", 2024-09-20T22:57:36.772 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:57:36.772 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.772 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.772 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:57:36.772 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:44.436593+0000", 2024-09-20T22:57:36.772 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:57:36.772 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.772 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.772 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:57:36.773 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:44.436593+0000", 2024-09-20T22:57:36.773 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.6699999999999999e-07 2024-09-20T22:57:36.773 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.773 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.773 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:57:36.773 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:44.475632+0000", 2024-09-20T22:57:36.773 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.039039010999999998 2024-09-20T22:57:36.773 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.773 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.773 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:57:36.773 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:44.475645+0000", 2024-09-20T22:57:36.773 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3084999999999999e-05 2024-09-20T22:57:36.773 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.773 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.774 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:57:36.774 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:44.475651+0000", 2024-09-20T22:57:36.774 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.1190000000000002e-06 2024-09-20T22:57:36.774 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.774 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.774 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:57:36.774 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:44.475714+0000", 2024-09-20T22:57:36.774 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.3239999999999998e-05 2024-09-20T22:57:36.774 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.774 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.774 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:57:36.774 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:44.475741+0000", 2024-09-20T22:57:36.774 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.6911e-05 2024-09-20T22:57:36.774 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.775 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.775 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:57:36.775 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:44.477457+0000", 2024-09-20T22:57:36.775 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0017154959999999999 2024-09-20T22:57:36.775 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.775 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.775 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:57:36.775 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:53:44.477485+0000", 2024-09-20T22:57:36.775 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.8160000000000001e-05 2024-09-20T22:57:36.775 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:57:36.775 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:57:36.775 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:57:36.775 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.776 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.776 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:7625 7.e 7:7c62ced4:::smithi08477203-17:head [copy-from ver 1253] snapc 159=[159,158,157,156,155,153,151,14f,14e,14c,14a,149,148,145,142,13e,13a,139,132,130,12f,12e,12a,124,121,112,103] ondisk+write+known_if_redirected+supports_pool_eio e766)", 2024-09-20T22:57:36.776 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:54:06.336464+0000", 2024-09-20T22:57:36.776 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 210.399164353, 2024-09-20T22:57:36.776 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014856506, 2024-09-20T22:57:36.776 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:57:36.776 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:57:36.776 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:57:36.776 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:57:36.776 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:57:36.776 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:57:36.776 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 7625 2024-09-20T22:57:36.776 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.776 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:57:36.777 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.777 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:57:36.777 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:54:06.336464+0000", 2024-09-20T22:57:36.777 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:57:36.777 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.777 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.777 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:57:36.777 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:54:06.336464+0000", 2024-09-20T22:57:36.777 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:57:36.777 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.777 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.777 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:57:36.777 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:54:06.336466+0000", 2024-09-20T22:57:36.778 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.7770000000000001e-06 2024-09-20T22:57:36.778 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.778 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.778 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:57:36.778 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:54:06.336475+0000", 2024-09-20T22:57:36.778 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.9579999999999996e-06 2024-09-20T22:57:36.778 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.778 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.778 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:57:36.778 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:54:06.336491+0000", 2024-09-20T22:57:36.778 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.6079999999999999e-05 2024-09-20T22:57:36.778 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.778 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.779 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:57:36.779 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:54:06.336501+0000", 2024-09-20T22:57:36.779 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.8239999999999995e-06 2024-09-20T22:57:36.779 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.779 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.779 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:57:36.779 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:54:06.336571+0000", 2024-09-20T22:57:36.779 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.9746999999999995e-05 2024-09-20T22:57:36.779 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.779 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.779 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op must wait for map", 2024-09-20T22:57:36.779 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:54:06.336580+0000", 2024-09-20T22:57:36.779 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.2900000000000008e-06 2024-09-20T22:57:36.779 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.780 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.780 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:57:36.780 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:54:06.338593+0000", 2024-09-20T22:57:36.780 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0020125939999999999 2024-09-20T22:57:36.780 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.780 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.780 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:57:36.780 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:54:06.338748+0000", 2024-09-20T22:57:36.780 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00015523199999999999 2024-09-20T22:57:36.780 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.780 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.780 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 0,5", 2024-09-20T22:57:36.780 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:54:06.344247+0000", 2024-09-20T22:57:36.780 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0054991079999999999 2024-09-20T22:57:36.781 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.781 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.781 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:57:36.781 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:54:06.349881+0000", 2024-09-20T22:57:36.781 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0056344689999999996 2024-09-20T22:57:36.781 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.781 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.781 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:57:36.781 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:54:06.351077+0000", 2024-09-20T22:57:36.781 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001196016 2024-09-20T22:57:36.781 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.781 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.781 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:57:36.781 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:54:06.351225+0000", 2024-09-20T22:57:36.781 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014721100000000001 2024-09-20T22:57:36.782 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.782 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.782 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:57:36.782 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:54:06.351265+0000", 2024-09-20T22:57:36.782 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.985e-05 2024-09-20T22:57:36.782 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.782 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.782 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:57:36.782 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:54:06.351321+0000", 2024-09-20T22:57:36.782 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.6350000000000001e-05 2024-09-20T22:57:36.782 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:57:36.782 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:57:36.782 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:57:36.782 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.783 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.783 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:8051 7.8 7:1e9e2524:::smithi08477203-5:head [write 420472~427736,stat] snapc 171=[171,16f,16e,16d,16c,16b,167,166,165,160,15f,15e,15c,15a,158,157,156,155,153,14f,14e,14a,149,145,142,13e,13a,139,12f,12e,124] ondisk+write+known_if_redirected+supports_pool_eio e810)", 2024-09-20T22:57:36.783 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:54:50.344852+0000", 2024-09-20T22:57:36.783 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 166.39077675600001, 2024-09-20T22:57:36.783 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013666950000000001, 2024-09-20T22:57:36.783 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:57:36.783 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:57:36.783 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:57:36.783 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:57:36.783 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:57:36.783 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:57:36.783 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 8051 2024-09-20T22:57:36.783 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.783 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:57:36.784 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.784 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:57:36.784 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:54:50.344852+0000", 2024-09-20T22:57:36.784 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:57:36.784 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.784 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.784 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:57:36.784 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:54:50.344852+0000", 2024-09-20T22:57:36.784 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:57:36.784 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.784 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.784 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:57:36.784 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:54:50.344854+0000", 2024-09-20T22:57:36.784 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.21e-06 2024-09-20T22:57:36.784 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.785 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.785 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:57:36.785 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:54:50.345132+0000", 2024-09-20T22:57:36.785 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00027765999999999999 2024-09-20T22:57:36.785 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.785 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.785 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:57:36.785 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:54:50.345150+0000", 2024-09-20T22:57:36.785 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.7876e-05 2024-09-20T22:57:36.785 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.785 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.785 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:57:36.785 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:54:50.345160+0000", 2024-09-20T22:57:36.785 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0349e-05 2024-09-20T22:57:36.786 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.786 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.786 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:57:36.786 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:54:50.345261+0000", 2024-09-20T22:57:36.786 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000100621 2024-09-20T22:57:36.786 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.786 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.786 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op must wait for map", 2024-09-20T22:57:36.786 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:54:50.345268+0000", 2024-09-20T22:57:36.786 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.5320000000000001e-06 2024-09-20T22:57:36.786 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.786 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.786 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:57:36.786 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:54:50.350192+0000", 2024-09-20T22:57:36.786 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0049241659999999998 2024-09-20T22:57:36.787 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.787 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.787 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:57:36.787 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:54:50.350315+0000", 2024-09-20T22:57:36.787 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000122948 2024-09-20T22:57:36.787 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.787 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.787 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 4,7", 2024-09-20T22:57:36.787 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:54:50.350506+0000", 2024-09-20T22:57:36.787 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000190609 2024-09-20T22:57:36.787 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.787 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.787 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:57:36.787 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:54:50.354160+0000", 2024-09-20T22:57:36.787 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.003654058 2024-09-20T22:57:36.788 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.788 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.788 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:57:36.788 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:54:50.354235+0000", 2024-09-20T22:57:36.788 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.4894000000000005e-05 2024-09-20T22:57:36.788 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.788 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.788 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:57:36.788 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:54:50.358382+0000", 2024-09-20T22:57:36.788 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0041471839999999999 2024-09-20T22:57:36.788 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.788 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.788 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:57:36.788 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:54:50.358448+0000", 2024-09-20T22:57:36.788 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.5857000000000003e-05 2024-09-20T22:57:36.789 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.789 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.789 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:57:36.789 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:54:50.358519+0000", 2024-09-20T22:57:36.789 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.0986000000000001e-05 2024-09-20T22:57:36.789 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:57:36.789 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:57:36.789 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:57:36.789 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.789 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.789 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:8458 7.e 7:7c62ced4:::smithi08477203-17:head [copy-from ver 2639] snapc 18d=[18d,18c,18b,18a,189,188,187,186,185,184,183,182,181,180,17f,17e,17d,17c,17b,17a,179,177,176,175,174,173,172,16f,16e,16c,16b,166,160,15c,15a,158,157,155,14f,14e,14a,149,13a,139,12f,12e] ondisk+write+known_if_redirected+supports_pool_eio e851)", 2024-09-20T22:57:36.789 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:55:31.382098+0000", 2024-09-20T22:57:36.789 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 125.353530486, 2024-09-20T22:57:36.790 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.019106850000000002, 2024-09-20T22:57:36.790 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:57:36.790 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:57:36.790 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:57:36.790 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:57:36.790 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:57:36.790 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:57:36.790 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 8458 2024-09-20T22:57:36.790 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.790 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:57:36.790 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.790 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:57:36.790 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:55:31.382098+0000", 2024-09-20T22:57:36.790 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:57:36.791 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.791 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.791 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:57:36.791 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:55:31.382098+0000", 2024-09-20T22:57:36.791 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:57:36.791 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.791 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.791 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:57:36.791 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:55:31.382098+0000", 2024-09-20T22:57:36.791 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.5400000000000002e-07 2024-09-20T22:57:36.791 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.791 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.791 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:57:36.792 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:55:31.382100+0000", 2024-09-20T22:57:36.792 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.905e-06 2024-09-20T22:57:36.792 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.792 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.792 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:57:36.792 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:55:31.382103+0000", 2024-09-20T22:57:36.792 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.9450000000000002e-06 2024-09-20T22:57:36.792 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.792 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.792 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:57:36.792 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:55:31.382106+0000", 2024-09-20T22:57:36.792 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.7810000000000001e-06 2024-09-20T22:57:36.793 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.793 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.793 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:57:36.793 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:55:31.382166+0000", 2024-09-20T22:57:36.793 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.9876000000000001e-05 2024-09-20T22:57:36.793 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.793 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.793 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:57:36.793 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:55:31.382296+0000", 2024-09-20T22:57:36.793 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00012992 2024-09-20T22:57:36.793 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.793 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.793 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 0,5", 2024-09-20T22:57:36.793 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:55:31.389774+0000", 2024-09-20T22:57:36.793 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0074777969999999996 2024-09-20T22:57:36.794 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.794 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.794 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:57:36.794 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:55:31.396494+0000", 2024-09-20T22:57:36.794 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0067207869999999998 2024-09-20T22:57:36.794 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.794 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.794 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:57:36.794 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:55:31.397877+0000", 2024-09-20T22:57:36.794 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001382306 2024-09-20T22:57:36.794 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.794 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.794 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:57:36.794 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:55:31.401048+0000", 2024-09-20T22:57:36.795 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0031707739999999999 2024-09-20T22:57:36.795 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.795 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.795 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:57:36.795 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:55:31.401130+0000", 2024-09-20T22:57:36.795 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.2281999999999996e-05 2024-09-20T22:57:36.795 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.795 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.795 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:57:36.795 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:55:31.401205+0000", 2024-09-20T22:57:36.795 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.5222999999999995e-05 2024-09-20T22:57:36.795 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:57:36.795 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:57:36.796 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:57:36.796 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.796 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.796 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:8459 7.e 7:7c62ced4:::smithi08477203-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e851)", 2024-09-20T22:57:36.796 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:55:31.382114+0000", 2024-09-20T22:57:36.796 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 125.35351424300001, 2024-09-20T22:57:36.796 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.012195655999999999, 2024-09-20T22:57:36.796 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:57:36.796 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:57:36.796 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-20T22:57:36.796 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:57:36.796 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:57:36.796 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:57:36.796 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 8459 2024-09-20T22:57:36.797 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.797 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:57:36.797 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.797 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:57:36.797 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:55:31.382114+0000", 2024-09-20T22:57:36.797 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:57:36.797 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.797 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.797 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:57:36.797 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:55:31.382114+0000", 2024-09-20T22:57:36.797 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:57:36.797 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.797 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.798 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:57:36.798 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:55:31.382115+0000", 2024-09-20T22:57:36.798 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.1e-07 2024-09-20T22:57:36.798 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.798 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.798 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:57:36.798 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:55:31.382116+0000", 2024-09-20T22:57:36.798 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1790000000000001e-06 2024-09-20T22:57:36.798 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.798 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.798 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:57:36.798 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:55:31.382120+0000", 2024-09-20T22:57:36.798 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.2699999999999998e-06 2024-09-20T22:57:36.798 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.799 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.799 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:57:36.799 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:55:31.382123+0000", 2024-09-20T22:57:36.799 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.6800000000000002e-06 2024-09-20T22:57:36.799 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.799 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.799 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:57:36.799 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:55:31.382514+0000", 2024-09-20T22:57:36.799 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00039134700000000001 2024-09-20T22:57:36.799 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.799 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.799 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for blocked object", 2024-09-20T22:57:36.799 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:55:31.382576+0000", 2024-09-20T22:57:36.799 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.1954999999999999e-05 2024-09-20T22:57:36.800 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.800 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.800 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:57:36.800 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:55:31.394061+0000", 2024-09-20T22:57:36.800 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011484604000000001 2024-09-20T22:57:36.800 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.800 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.800 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:57:36.800 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:55:31.394187+0000", 2024-09-20T22:57:36.800 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000126513 2024-09-20T22:57:36.800 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.800 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.800 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:57:36.800 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:55:31.394310+0000", 2024-09-20T22:57:36.801 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000122798 2024-09-20T22:57:36.801 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:57:36.801 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:57:36.801 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:57:36.801 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.801 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.801 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(client.24652.0:8839 7.5 e885/456 7:a160b5d4:::smithi08477203-16:head v 885'490, mlcod=883'486)", 2024-09-20T22:57:36.801 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:56:05.513370+0000", 2024-09-20T22:57:36.801 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 91.222258595, 2024-09-20T22:57:36.801 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.014010404000000001, 2024-09-20T22:57:36.801 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:57:36.801 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:57:36.801 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:57:36.801 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:57:36.802 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.802 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:57:36.802 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:56:05.513370+0000", 2024-09-20T22:57:36.802 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:57:36.802 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.802 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.802 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:57:36.802 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:56:05.513370+0000", 2024-09-20T22:57:36.802 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:57:36.802 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.802 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.802 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:57:36.802 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:56:05.513371+0000", 2024-09-20T22:57:36.803 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.266e-06 2024-09-20T22:57:36.803 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.803 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.803 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:57:36.803 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:56:05.520144+0000", 2024-09-20T22:57:36.803 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0067725349999999997 2024-09-20T22:57:36.803 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.803 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.803 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:57:36.803 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:56:05.520164+0000", 2024-09-20T22:57:36.803 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.0523999999999999e-05 2024-09-20T22:57:36.803 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.803 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.804 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:57:36.804 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:56:05.520173+0000", 2024-09-20T22:57:36.804 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.0520000000000007e-06 2024-09-20T22:57:36.804 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.804 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.804 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:57:36.804 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:56:05.520281+0000", 2024-09-20T22:57:36.804 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00010733100000000001 2024-09-20T22:57:36.804 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.804 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.804 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:57:36.804 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:56:05.520317+0000", 2024-09-20T22:57:36.804 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.5710999999999997e-05 2024-09-20T22:57:36.804 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.805 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.805 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:57:36.805 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:56:05.527309+0000", 2024-09-20T22:57:36.805 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0069921870000000004 2024-09-20T22:57:36.805 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.805 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.805 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:57:36.805 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:56:05.527381+0000", 2024-09-20T22:57:36.805 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.1798000000000006e-05 2024-09-20T22:57:36.805 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:57:36.805 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:57:36.805 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:57:36.805 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.805 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.806 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_repop(osd.6.0:3767 7.c e894/890 7:318c7512:::smithi08477203-44:157 v 894'2891, mlcod=893'2887)", 2024-09-20T22:57:36.806 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:56:14.343156+0000", 2024-09-20T22:57:36.806 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 82.392472573999996, 2024-09-20T22:57:36.806 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010799886999999999, 2024-09-20T22:57:36.806 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:57:36.806 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:57:36.806 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:57:36.806 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:57:36.806 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.806 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:57:36.806 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:56:14.343156+0000", 2024-09-20T22:57:36.806 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:57:36.806 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.806 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.807 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:57:36.807 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:56:14.343156+0000", 2024-09-20T22:57:36.807 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:57:36.807 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.807 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.807 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:57:36.807 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:56:14.343157+0000", 2024-09-20T22:57:36.807 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 3.8099999999999998e-07 2024-09-20T22:57:36.807 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.807 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.807 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:57:36.807 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:56:14.343163+0000", 2024-09-20T22:57:36.807 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.28e-06 2024-09-20T22:57:36.807 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.807 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.808 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:57:36.808 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:56:14.343169+0000", 2024-09-20T22:57:36.808 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 6.5640000000000002e-06 2024-09-20T22:57:36.808 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.808 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.808 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:57:36.808 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:56:14.343176+0000", 2024-09-20T22:57:36.808 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.1239999999999999e-06 2024-09-20T22:57:36.808 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.808 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.808 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:57:36.808 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:56:14.343258+0000", 2024-09-20T22:57:36.808 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.1464999999999993e-05 2024-09-20T22:57:36.808 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.808 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.809 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:57:36.809 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:56:14.343283+0000", 2024-09-20T22:57:36.809 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.4859e-05 2024-09-20T22:57:36.809 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.809 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.810 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:57:36.810 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:56:14.353905+0000", 2024-09-20T22:57:36.810 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010622365 2024-09-20T22:57:36.810 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.810 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.810 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:57:36.810 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:56:14.353956+0000", 2024-09-20T22:57:36.810 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.0849000000000001e-05 2024-09-20T22:57:36.810 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:57:36.810 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:57:36.810 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:57:36.810 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.811 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.811 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:8990 7.8 7:12e9c8db:::smithi08477203-11:head [copy-from ver 1869] snapc 1a4=[1a4,1a3,1a2,1a1,1a0,19f,19e,19d,19c,19b,19a,198,197,196,193,190,18f,18e,18d,18c,189,185,183,182,180,17e,17d,175,174,173,16f,16e,16c,16b,160,158,14e,14a,13a,12f] ondisk+write+known_if_redirected+supports_pool_eio e903)", 2024-09-20T22:57:36.811 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:56:23.345568+0000", 2024-09-20T22:57:36.811 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 73.390060879999993, 2024-09-20T22:57:36.811 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.027407589, 2024-09-20T22:57:36.811 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:57:36.811 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:57:36.811 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:57:36.811 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:57:36.811 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:57:36.811 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:57:36.811 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 8990 2024-09-20T22:57:36.811 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.811 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:57:36.812 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.812 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:57:36.812 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:56:23.345568+0000", 2024-09-20T22:57:36.812 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:57:36.812 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.812 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.812 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:57:36.812 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:56:23.345568+0000", 2024-09-20T22:57:36.812 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:57:36.812 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.812 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.812 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:57:36.812 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:56:23.345570+0000", 2024-09-20T22:57:36.812 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.0999999999999998e-06 2024-09-20T22:57:36.813 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.813 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.813 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:57:36.813 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:56:23.345580+0000", 2024-09-20T22:57:36.813 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0217999999999999e-05 2024-09-20T22:57:36.813 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.813 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.813 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:57:36.813 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:56:23.345599+0000", 2024-09-20T22:57:36.813 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.9046999999999999e-05 2024-09-20T22:57:36.813 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.813 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.813 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:57:36.813 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:56:23.345611+0000", 2024-09-20T22:57:36.814 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1609e-05 2024-09-20T22:57:36.814 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.814 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.814 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:57:36.814 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:56:23.345702+0000", 2024-09-20T22:57:36.814 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.0791999999999994e-05 2024-09-20T22:57:36.814 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.814 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.814 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op must wait for map", 2024-09-20T22:57:36.814 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:56:23.345711+0000", 2024-09-20T22:57:36.814 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.7880000000000001e-06 2024-09-20T22:57:36.814 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.814 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.814 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:57:36.815 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:56:23.350765+0000", 2024-09-20T22:57:36.815 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.005054078 2024-09-20T22:57:36.815 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.815 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.815 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:57:36.815 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:56:23.350895+0000", 2024-09-20T22:57:36.815 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00012988400000000001 2024-09-20T22:57:36.815 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.815 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.815 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 3,4", 2024-09-20T22:57:36.815 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:56:23.359133+0000", 2024-09-20T22:57:36.815 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0082376050000000003 2024-09-20T22:57:36.815 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.815 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.816 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:57:36.816 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:56:23.371446+0000", 2024-09-20T22:57:36.816 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.012313154999999999 2024-09-20T22:57:36.816 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.816 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.816 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:57:36.816 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:56:23.372712+0000", 2024-09-20T22:57:36.816 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0012661860000000001 2024-09-20T22:57:36.816 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.816 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.816 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:57:36.816 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:56:23.372854+0000", 2024-09-20T22:57:36.816 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00014162800000000001 2024-09-20T22:57:36.816 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.816 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.817 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:57:36.817 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:56:23.372932+0000", 2024-09-20T22:57:36.817 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.7776999999999998e-05 2024-09-20T22:57:36.817 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.817 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.817 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:57:36.817 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:56:23.372975+0000", 2024-09-20T22:57:36.817 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.3721999999999999e-05 2024-09-20T22:57:36.817 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:57:36.817 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:57:36.817 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:57:36.817 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.817 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.817 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:8991 7.8 7:12e9c8db:::smithi08477203-11:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e903)", 2024-09-20T22:57:36.818 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:56:23.345641+0000", 2024-09-20T22:57:36.818 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 73.389987585, 2024-09-20T22:57:36.818 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.01924379, 2024-09-20T22:57:36.818 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:57:36.818 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:57:36.818 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-20T22:57:36.818 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:57:36.818 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:57:36.818 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:57:36.818 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 8991 2024-09-20T22:57:36.818 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.818 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:57:36.818 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.819 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:57:36.819 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:56:23.345641+0000", 2024-09-20T22:57:36.819 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:57:36.819 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.819 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.819 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:57:36.819 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:56:23.345641+0000", 2024-09-20T22:57:36.819 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:57:36.819 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.819 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.819 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:57:36.819 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:56:23.345642+0000", 2024-09-20T22:57:36.819 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.82e-07 2024-09-20T22:57:36.819 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.820 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.820 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:57:36.820 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:56:23.345645+0000", 2024-09-20T22:57:36.820 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.9380000000000001e-06 2024-09-20T22:57:36.820 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.820 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.820 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:57:36.820 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:56:23.345656+0000", 2024-09-20T22:57:36.820 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.1348e-05 2024-09-20T22:57:36.820 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.820 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.820 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:57:36.820 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:56:23.345661+0000", 2024-09-20T22:57:36.820 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.4729999999999999e-06 2024-09-20T22:57:36.820 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.821 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.821 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:57:36.821 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:56:23.345785+0000", 2024-09-20T22:57:36.821 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00012388699999999999 2024-09-20T22:57:36.821 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.821 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.821 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting_for_map not empty", 2024-09-20T22:57:36.821 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:56:23.345793+0000", 2024-09-20T22:57:36.821 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.0989999999999999e-06 2024-09-20T22:57:36.821 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.821 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.821 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:57:36.821 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:56:23.351058+0000", 2024-09-20T22:57:36.822 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00526495 2024-09-20T22:57:36.822 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.822 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.822 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for blocked object", 2024-09-20T22:57:36.822 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:56:23.351109+0000", 2024-09-20T22:57:36.822 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.1177000000000001e-05 2024-09-20T22:57:36.822 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.822 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.822 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:57:36.822 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:56:23.364695+0000", 2024-09-20T22:57:36.822 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.013585104000000001 2024-09-20T22:57:36.822 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.822 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.822 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:57:36.823 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:56:23.364805+0000", 2024-09-20T22:57:36.823 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000109948 2024-09-20T22:57:36.823 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.823 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.823 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:57:36.823 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:56:23.364885+0000", 2024-09-20T22:57:36.823 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.0383999999999995e-05 2024-09-20T22:57:36.823 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:57:36.823 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:57:36.823 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:57:36.823 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.823 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.823 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:9290 7.e 7:71b5c915:::smithi08477203-6:head [rollback 197] snapc 1b0=[1b0,1af,1ae,1ad,1ac,1ab,1aa,1a9,1a8,1a7,1a6,1a5,1a4,1a3,1a2,1a0,19d,19c,19a,197,196,193,190,18f,18e,18d,189,185,180,17e,17d,174,173,16e,16c,16b,160,158,14e,14a] ondisk+write+known_if_redirected+supports_pool_eio e927)", 2024-09-20T22:57:36.824 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:56:47.389213+0000", 2024-09-20T22:57:36.824 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 49.346415657999998, 2024-09-20T22:57:36.824 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010836807, 2024-09-20T22:57:36.824 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:57:36.824 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:57:36.824 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-20T22:57:36.824 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:57:36.824 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:57:36.824 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:57:36.824 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 9290 2024-09-20T22:57:36.824 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.824 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:57:36.824 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.825 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:57:36.825 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:56:47.389213+0000", 2024-09-20T22:57:36.825 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:57:36.825 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.825 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.825 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:57:36.825 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:56:47.389213+0000", 2024-09-20T22:57:36.825 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:57:36.825 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.825 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.825 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:57:36.825 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:56:47.389215+0000", 2024-09-20T22:57:36.825 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.0990000000000001e-06 2024-09-20T22:57:36.826 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.826 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.826 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:57:36.826 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:56:47.389224+0000", 2024-09-20T22:57:36.826 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 8.8429999999999999e-06 2024-09-20T22:57:36.826 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.826 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.826 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:57:36.826 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:56:47.389248+0000", 2024-09-20T22:57:36.826 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.4326000000000001e-05 2024-09-20T22:57:36.826 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.826 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.826 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:57:36.827 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:56:47.389259+0000", 2024-09-20T22:57:36.827 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0631e-05 2024-09-20T22:57:36.827 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.827 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.827 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:57:36.827 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:56:47.389334+0000", 2024-09-20T22:57:36.827 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.5532999999999997e-05 2024-09-20T22:57:36.827 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.827 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.827 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op must wait for map", 2024-09-20T22:57:36.827 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:56:47.389344+0000", 2024-09-20T22:57:36.827 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 9.115e-06 2024-09-20T22:57:36.827 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.828 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.828 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:57:36.828 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:56:47.396594+0000", 2024-09-20T22:57:36.828 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0072503439999999997 2024-09-20T22:57:36.828 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.828 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.828 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:57:36.828 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:56:47.396715+0000", 2024-09-20T22:57:36.828 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000120683 2024-09-20T22:57:36.828 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.828 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.828 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:57:36.828 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:56:47.400050+0000", 2024-09-20T22:57:36.828 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0033352329999999999 2024-09-20T22:57:36.828 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:57:36.828 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:57:36.828 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:57:36.829 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.829 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.829 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:9470 7.8 7:12e9c8db:::smithi08477203-11:head [write 514644~485365,stat] snapc 1b8=[1b8,1b7,1b6,1b5,1b4,1b3,1b2,1b1,1b0,1af,1ae,1ad,1ac,1ab,1aa,1a9,1a8,1a7,1a6,1a5,1a4,1a3,1a2,1a0,19d,19c,19a,197,196,193,190,18f,18e,18d,185,180,17e,173,16e,16c,16b,160,158,14e,14a] ondisk+write+known_if_redirected+supports_pool_eio e938)", 2024-09-20T22:57:36.829 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:56:58.368392+0000", 2024-09-20T22:57:36.829 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 38.367236699000003, 2024-09-20T22:57:36.829 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.010163419, 2024-09-20T22:57:36.829 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:57:36.829 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:57:36.829 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:57:36.829 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:57:36.829 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:57:36.829 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:57:36.829 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 9470 2024-09-20T22:57:36.829 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.830 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:57:36.830 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.830 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:57:36.830 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:56:58.368392+0000", 2024-09-20T22:57:36.830 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:57:36.830 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.830 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.830 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:57:36.830 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:56:58.368392+0000", 2024-09-20T22:57:36.830 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:57:36.830 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.830 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.830 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:57:36.830 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:56:58.368394+0000", 2024-09-20T22:57:36.831 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.8759999999999999e-06 2024-09-20T22:57:36.831 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.831 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.831 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:57:36.831 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:56:58.368713+0000", 2024-09-20T22:57:36.831 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00031958399999999998 2024-09-20T22:57:36.831 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.831 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.831 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:57:36.831 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:56:58.368727+0000", 2024-09-20T22:57:36.831 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3594000000000001e-05 2024-09-20T22:57:36.831 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.831 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.831 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:57:36.832 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:56:58.368737+0000", 2024-09-20T22:57:36.832 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0008e-05 2024-09-20T22:57:36.832 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.832 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.832 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:57:36.832 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:56:58.368903+0000", 2024-09-20T22:57:36.832 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00016577200000000001 2024-09-20T22:57:36.832 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.832 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.832 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:57:36.832 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:56:58.369027+0000", 2024-09-20T22:57:36.832 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00012430300000000001 2024-09-20T22:57:36.832 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.832 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.832 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 3,4", 2024-09-20T22:57:36.833 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:56:58.369339+0000", 2024-09-20T22:57:36.833 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00031184700000000002 2024-09-20T22:57:36.833 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.833 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.833 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:57:36.833 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:56:58.373323+0000", 2024-09-20T22:57:36.833 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0039844119999999997 2024-09-20T22:57:36.833 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.833 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.833 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:57:36.833 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:56:58.375487+0000", 2024-09-20T22:57:36.833 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0021630999999999998 2024-09-20T22:57:36.833 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.833 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.834 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:57:36.834 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:56:58.378406+0000", 2024-09-20T22:57:36.834 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0029199389999999999 2024-09-20T22:57:36.834 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.834 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.834 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:57:36.834 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:56:58.378485+0000", 2024-09-20T22:57:36.834 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.8943000000000006e-05 2024-09-20T22:57:36.834 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.834 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.834 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:57:36.834 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:56:58.378555+0000", 2024-09-20T22:57:36.834 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 7.0041000000000002e-05 2024-09-20T22:57:36.835 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:57:36.835 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:57:36.835 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:57:36.835 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.835 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.835 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:9776 7.5 7:a160b5d4:::smithi08477203-16:head [copy-from ver 3141] snapc 1c9=[1c9,1c8,1c6,1c5,1c4,1c3,1c2,1bf,1be,1bc,1bb,1ba,1b9,1b8,1b7,1b6,1b5,1b4,1b3,1b2,1b1,1b0,1ae,1ad,1ac,1aa,1a9,1a8,1a7,1a6,1a5,1a3,1a2,1a0,19d,19c,19a,197,196,190,18f,18e,185,180,17e,173,16e,16c,160,158,14e,14a] ondisk+write+known_if_redirected+supports_pool_eio e965)", 2024-09-20T22:57:36.835 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:57:25.552666+0000", 2024-09-20T22:57:36.835 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 11.182962899, 2024-09-20T22:57:36.835 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.018344829999999999, 2024-09-20T22:57:36.835 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:57:36.835 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:57:36.835 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:57:36.835 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:57:36.835 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:57:36.835 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:57:36.836 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 9776 2024-09-20T22:57:36.836 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.836 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:57:36.836 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.836 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:57:36.836 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:57:25.552666+0000", 2024-09-20T22:57:36.836 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:57:36.836 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.836 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.836 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:57:36.836 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:57:25.552666+0000", 2024-09-20T22:57:36.836 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:57:36.836 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.836 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.836 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:57:36.837 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:57:25.552668+0000", 2024-09-20T22:57:36.837 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.1720000000000001e-06 2024-09-20T22:57:36.837 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.837 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.837 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:57:36.837 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:57:25.552680+0000", 2024-09-20T22:57:36.837 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.2063e-05 2024-09-20T22:57:36.837 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.837 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.837 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:57:36.837 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:57:25.552694+0000", 2024-09-20T22:57:36.837 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.3816e-05 2024-09-20T22:57:36.837 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.837 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.838 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:57:36.838 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:57:25.552705+0000", 2024-09-20T22:57:36.838 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 1.0873000000000001e-05 2024-09-20T22:57:36.838 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.838 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.838 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:57:36.838 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:57:25.554060+0000", 2024-09-20T22:57:36.838 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.001354966 2024-09-20T22:57:36.838 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.838 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.838 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:57:36.838 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:57:25.554191+0000", 2024-09-20T22:57:36.838 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013121199999999999 2024-09-20T22:57:36.838 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.839 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.839 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for subops from 0,1", 2024-09-20T22:57:36.839 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:57:25.561227+0000", 2024-09-20T22:57:36.839 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.007036591 2024-09-20T22:57:36.839 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.839 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.839 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "op_commit", 2024-09-20T22:57:36.839 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:57:25.570463+0000", 2024-09-20T22:57:36.839 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0092350520000000005 2024-09-20T22:57:36.839 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.839 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.839 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:57:36.839 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:57:25.570658+0000", 2024-09-20T22:57:36.840 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00019504 2024-09-20T22:57:36.840 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.840 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.840 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:57:36.840 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:57:25.570914+0000", 2024-09-20T22:57:36.840 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00025609299999999999 2024-09-20T22:57:36.840 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.840 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.840 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "commit_sent", 2024-09-20T22:57:36.840 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:57:25.570965+0000", 2024-09-20T22:57:36.840 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.1341000000000002e-05 2024-09-20T22:57:36.840 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.840 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.840 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:57:36.841 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:57:25.571011+0000", 2024-09-20T22:57:36.841 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.5611000000000001e-05 2024-09-20T22:57:36.841 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:57:36.841 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:57:36.841 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:57:36.841 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.841 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.841 INFO:teuthology.orchestra.run.smithi084.stdout: "description": "osd_op(client.24652.0:9777 7.5 7:a160b5d4:::smithi08477203-16:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e965)", 2024-09-20T22:57:36.841 INFO:teuthology.orchestra.run.smithi084.stdout: "initiated_at": "2024-09-20T22:57:25.552721+0000", 2024-09-20T22:57:36.841 INFO:teuthology.orchestra.run.smithi084.stdout: "age": 11.182907729, 2024-09-20T22:57:36.841 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.011915176, 2024-09-20T22:57:36.841 INFO:teuthology.orchestra.run.smithi084.stdout: "continuous": false, 2024-09-20T22:57:36.841 INFO:teuthology.orchestra.run.smithi084.stdout: "type_data": { 2024-09-20T22:57:36.841 INFO:teuthology.orchestra.run.smithi084.stdout: "flag_point": "started", 2024-09-20T22:57:36.841 INFO:teuthology.orchestra.run.smithi084.stdout: "client_info": { 2024-09-20T22:57:36.842 INFO:teuthology.orchestra.run.smithi084.stdout: "client": "client.24652", 2024-09-20T22:57:36.842 INFO:teuthology.orchestra.run.smithi084.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:57:36.842 INFO:teuthology.orchestra.run.smithi084.stdout: "tid": 9777 2024-09-20T22:57:36.842 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.842 INFO:teuthology.orchestra.run.smithi084.stdout: "events": [ 2024-09-20T22:57:36.842 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.842 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "initiated", 2024-09-20T22:57:36.842 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:57:25.552721+0000", 2024-09-20T22:57:36.842 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:57:36.842 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.842 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.842 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "header_read", 2024-09-20T22:57:36.842 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:57:25.552721+0000", 2024-09-20T22:57:36.842 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0 2024-09-20T22:57:36.842 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.843 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.843 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "throttled", 2024-09-20T22:57:36.843 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:57:25.552721+0000", 2024-09-20T22:57:36.843 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.4000000000000002e-07 2024-09-20T22:57:36.843 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.843 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.843 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "all_read", 2024-09-20T22:57:36.843 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:57:25.552724+0000", 2024-09-20T22:57:36.843 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 2.605e-06 2024-09-20T22:57:36.843 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.843 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.843 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "dispatched", 2024-09-20T22:57:36.843 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:57:25.552730+0000", 2024-09-20T22:57:36.843 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.6370000000000004e-06 2024-09-20T22:57:36.844 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.844 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.844 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "queued_for_pg", 2024-09-20T22:57:36.844 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:57:25.552734+0000", 2024-09-20T22:57:36.844 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 4.8119999999999998e-06 2024-09-20T22:57:36.844 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.844 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.844 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:57:36.844 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:57:25.554378+0000", 2024-09-20T22:57:36.844 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.0016434209999999999 2024-09-20T22:57:36.844 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.844 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.844 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "waiting for blocked object", 2024-09-20T22:57:36.845 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:57:25.554430+0000", 2024-09-20T22:57:36.845 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 5.2599999999999998e-05 2024-09-20T22:57:36.845 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.845 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.845 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "reached_pg", 2024-09-20T22:57:36.845 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:57:25.564364+0000", 2024-09-20T22:57:36.845 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.009933875 2024-09-20T22:57:36.845 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.845 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.845 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "started", 2024-09-20T22:57:36.845 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:57:25.564499+0000", 2024-09-20T22:57:36.845 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.000134268 2024-09-20T22:57:36.845 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:36.845 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:36.845 INFO:teuthology.orchestra.run.smithi084.stdout: "event": "done", 2024-09-20T22:57:36.846 INFO:teuthology.orchestra.run.smithi084.stdout: "time": "2024-09-20T22:57:25.564636+0000", 2024-09-20T22:57:36.846 INFO:teuthology.orchestra.run.smithi084.stdout: "duration": 0.00013751799999999999 2024-09-20T22:57:36.846 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:57:36.846 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:57:36.846 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:57:36.846 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:57:36.846 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:57:36.846 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-20T22:57:36.846 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:57:36.868 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:36.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:36.869 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:57:36.969 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:57:36.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:36.991 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:36.991 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:57:37.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:36 smithi154 ceph-mon[33264]: osdmap e975: 8 total, 7 up, 7 in 2024-09-20T22:57:37.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:36 smithi154 ceph-mon[33264]: 5.10 scrub starts 2024-09-20T22:57:37.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:36 smithi154 ceph-mon[33264]: 5.10 scrub ok 2024-09-20T22:57:37.092 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:57:37.112 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:37.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:37.113 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:57:37.214 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:57:37.235 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:37.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:37.236 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:57:37.337 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:57:37.358 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:37.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:37.359 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:57:37.364 INFO:journalctl@ceph.osd.3.smithi084.stdout:Sep 20 22:57:37 smithi084 podman[212682]: 2024-09-20 22:57:37.08909329 +0000 UTC m=+1.297974182 container create a516fee9da47e4cc143d72d860ae50abded1c23486d8453c20b40a81c5a1294a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-3-activate, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-9e2ac70, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, ceph=True) 2024-09-20T22:57:37.460 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:57:37.483 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:37.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:37.484 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:57:37.585 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:57:37.606 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:37.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:37.607 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:57:37.619 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is unset 2024-09-20T22:57:37.622 INFO:tasks.rados.rados.0.smithi084.stdout:3953: done (3 left) 2024-09-20T22:57:37.623 INFO:tasks.rados.rados.0.smithi084.stdout:3954: done (2 left) 2024-09-20T22:57:37.623 INFO:tasks.rados.rados.0.smithi084.stdout:3956: done (1 left) 2024-09-20T22:57:37.623 INFO:tasks.rados.rados.0.smithi084.stdout:3957: done (0 left) 2024-09-20T22:57:37.623 INFO:tasks.rados.rados.0.smithi084.stdout:3958: read oid 45 snap -1 2024-09-20T22:57:37.623 INFO:tasks.rados.rados.0.smithi084.stdout:3958: expect deleted 2024-09-20T22:57:37.623 INFO:tasks.rados.rados.0.smithi084.stdout:3959: rollback oid 38 current snap is 459 2024-09-20T22:57:37.623 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 38 to 416 2024-09-20T22:57:37.623 INFO:tasks.rados.rados.0.smithi084.stdout:3960: snap_remove snap 451 2024-09-20T22:57:37.708 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:57:37.714 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:37 smithi084 ceph-mon[40024]: pgmap v1365: 168 pgs: 1 active+clean+remapped, 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+clean+snaptrim, 41 active+undersized, 1 active+remapped+backfill_toofull, 36 active+undersized+degraded, 87 active+clean; 827 MiB data, 2.3 GiB used, 623 GiB / 626 GiB avail; 300/1704 objects degraded (17.606%); 55/1704 objects misplaced (3.228%) 2024-09-20T22:57:37.714 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:37 smithi084 ceph-mon[40024]: 4.d scrub starts 2024-09-20T22:57:37.714 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:37 smithi084 ceph-mon[40024]: 4.d scrub ok 2024-09-20T22:57:37.714 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:37 smithi084 ceph-mon[40024]: osdmap e976: 8 total, 7 up, 7 in 2024-09-20T22:57:37.714 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:37 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/2991139035' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:57:37.714 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:37 smithi084 ceph-mon[40024]: 3.1a scrub starts 2024-09-20T22:57:37.714 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:37 smithi084 ceph-mon[40024]: 3.1a scrub ok 2024-09-20T22:57:37.714 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:37 smithi084 ceph-mon[29774]: pgmap v1365: 168 pgs: 1 active+clean+remapped, 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+clean+snaptrim, 41 active+undersized, 1 active+remapped+backfill_toofull, 36 active+undersized+degraded, 87 active+clean; 827 MiB data, 2.3 GiB used, 623 GiB / 626 GiB avail; 300/1704 objects degraded (17.606%); 55/1704 objects misplaced (3.228%) 2024-09-20T22:57:37.715 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:37 smithi084 ceph-mon[29774]: 4.d scrub starts 2024-09-20T22:57:37.715 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:37 smithi084 ceph-mon[29774]: 4.d scrub ok 2024-09-20T22:57:37.715 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:37 smithi084 ceph-mon[29774]: osdmap e976: 8 total, 7 up, 7 in 2024-09-20T22:57:37.715 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:37 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/2991139035' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:57:37.715 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:37 smithi084 ceph-mon[29774]: 3.1a scrub starts 2024-09-20T22:57:37.715 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:37 smithi084 ceph-mon[29774]: 3.1a scrub ok 2024-09-20T22:57:37.715 INFO:journalctl@ceph.osd.3.smithi084.stdout:Sep 20 22:57:37 smithi084 podman[212682]: 2024-09-20 22:57:37.443033957 +0000 UTC m=+1.651914846 container init a516fee9da47e4cc143d72d860ae50abded1c23486d8453c20b40a81c5a1294a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-3-activate, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, RELEASE=squid-9e2ac70, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-09-20T22:57:37.715 INFO:journalctl@ceph.osd.3.smithi084.stdout:Sep 20 22:57:37 smithi084 podman[212682]: 2024-09-20 22:57:37.446410922 +0000 UTC m=+1.655291815 container start a516fee9da47e4cc143d72d860ae50abded1c23486d8453c20b40a81c5a1294a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-3-activate, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, ceph=True, RELEASE=squid-9e2ac70, GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-09-20T22:57:37.715 INFO:journalctl@ceph.osd.3.smithi084.stdout:Sep 20 22:57:37 smithi084 podman[212682]: 2024-09-20 22:57:37.526239498 +0000 UTC m=+1.735120403 container attach a516fee9da47e4cc143d72d860ae50abded1c23486d8453c20b40a81c5a1294a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-3-activate, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, RELEASE=squid-9e2ac70, CEPH_POINT_RELEASE=) 2024-09-20T22:57:37.716 INFO:journalctl@ceph.osd.3.smithi084.stdout:Sep 20 22:57:37 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-3-activate[212868]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T22:57:37.716 INFO:journalctl@ceph.osd.3.smithi084.stdout:Sep 20 22:57:37 smithi084 bash[212682]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T22:57:37.716 INFO:journalctl@ceph.osd.3.smithi084.stdout:Sep 20 22:57:37 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-3-activate[212868]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T22:57:37.716 INFO:journalctl@ceph.osd.3.smithi084.stdout:Sep 20 22:57:37 smithi084 bash[212682]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T22:57:37.730 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:37.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:37.731 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:57:37.737 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-20T22:57:37.833 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:57:37.854 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:37.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:37.855 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:57:37.956 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:57:37.979 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:37.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:37.980 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:57:38.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:37 smithi154 ceph-mon[33264]: pgmap v1365: 168 pgs: 1 active+clean+remapped, 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+clean+snaptrim, 41 active+undersized, 1 active+remapped+backfill_toofull, 36 active+undersized+degraded, 87 active+clean; 827 MiB data, 2.3 GiB used, 623 GiB / 626 GiB avail; 300/1704 objects degraded (17.606%); 55/1704 objects misplaced (3.228%) 2024-09-20T22:57:38.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:37 smithi154 ceph-mon[33264]: 4.d scrub starts 2024-09-20T22:57:38.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:37 smithi154 ceph-mon[33264]: 4.d scrub ok 2024-09-20T22:57:38.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:37 smithi154 ceph-mon[33264]: osdmap e976: 8 total, 7 up, 7 in 2024-09-20T22:57:38.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:37 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/2991139035' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:57:38.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:37 smithi154 ceph-mon[33264]: 3.1a scrub starts 2024-09-20T22:57:38.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:37 smithi154 ceph-mon[33264]: 3.1a scrub ok 2024-09-20T22:57:38.081 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:57:38.104 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:38.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:38.105 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:57:38.158 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-20T22:57:38.206 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:57:38.227 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:38.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:38.228 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:57:38.329 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:57:38.350 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:38.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:38.351 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:57:38.357 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:57:38 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: ::ffff:172.21.15.154 - - [20/Sep/2024:22:57:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-20T22:57:38.358 INFO:journalctl@ceph.osd.3.smithi084.stdout:Sep 20 22:57:38 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-3-activate[212868]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-20T22:57:38.358 INFO:journalctl@ceph.osd.3.smithi084.stdout:Sep 20 22:57:38 smithi084 bash[212682]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-20T22:57:38.358 INFO:journalctl@ceph.osd.3.smithi084.stdout:Sep 20 22:57:38 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-3-activate[212868]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-3 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-09-20T22:57:38.358 INFO:journalctl@ceph.osd.3.smithi084.stdout:Sep 20 22:57:38 smithi084 bash[212682]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-3 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-09-20T22:57:38.358 INFO:journalctl@ceph.osd.3.smithi084.stdout:Sep 20 22:57:38 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-3-activate[212868]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-09-20T22:57:38.358 INFO:journalctl@ceph.osd.3.smithi084.stdout:Sep 20 22:57:38 smithi084 bash[212682]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-09-20T22:57:38.374 INFO:tasks.rados.rados.0.smithi084.stdout:3960: done (2 left) 2024-09-20T22:57:38.374 INFO:tasks.rados.rados.0.smithi084.stdout:3961: snap_remove snap 398 2024-09-20T22:57:38.415 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:57:38.451 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:57:38.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:38.473 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:38.473 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:57:38.574 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:57:38.595 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:38.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:38.596 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:57:38.613 INFO:journalctl@ceph.osd.3.smithi084.stdout:Sep 20 22:57:38 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-3-activate[212868]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-09-20T22:57:38.613 INFO:journalctl@ceph.osd.3.smithi084.stdout:Sep 20 22:57:38 smithi084 bash[212682]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-09-20T22:57:38.613 INFO:journalctl@ceph.osd.3.smithi084.stdout:Sep 20 22:57:38 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-3-activate[212868]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-3/block 2024-09-20T22:57:38.613 INFO:journalctl@ceph.osd.3.smithi084.stdout:Sep 20 22:57:38 smithi084 bash[212682]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-3/block 2024-09-20T22:57:38.613 INFO:journalctl@ceph.osd.3.smithi084.stdout:Sep 20 22:57:38 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-3-activate[212868]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-20T22:57:38.613 INFO:journalctl@ceph.osd.3.smithi084.stdout:Sep 20 22:57:38 smithi084 bash[212682]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-20T22:57:38.613 INFO:journalctl@ceph.osd.3.smithi084.stdout:Sep 20 22:57:38 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-3-activate[212868]: --> ceph-volume raw activate successful for osd ID: 3 2024-09-20T22:57:38.613 INFO:journalctl@ceph.osd.3.smithi084.stdout:Sep 20 22:57:38 smithi084 bash[212682]: --> ceph-volume raw activate successful for osd ID: 3 2024-09-20T22:57:38.697 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:57:38.719 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:38.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:38.720 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:57:38.817 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:38 smithi154 ceph-mon[33264]: 5.15 scrub starts 2024-09-20T22:57:38.817 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:38 smithi154 ceph-mon[33264]: 5.15 scrub ok 2024-09-20T22:57:38.817 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:38 smithi154 ceph-mon[33264]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T22:57:38.817 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:38 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/2991139035' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T22:57:38.817 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:38 smithi154 ceph-mon[33264]: osdmap e977: 8 total, 7 up, 7 in 2024-09-20T22:57:38.817 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:38 smithi154 ceph-mon[33264]: 4.e scrub starts 2024-09-20T22:57:38.817 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:38 smithi154 ceph-mon[33264]: 4.e scrub ok 2024-09-20T22:57:38.817 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:38 smithi154 ceph-mon[33264]: Health check update: Degraded data redundancy: 300/1704 objects degraded (17.606%), 37 pgs degraded (PG_DEGRADED) 2024-09-20T22:57:38.817 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:38 smithi154 ceph-mon[33264]: osdmap e978: 8 total, 7 up, 7 in 2024-09-20T22:57:38.817 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:38 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:57:38.817 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:38 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:57:38.821 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:57:38.842 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:38.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:38.843 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:57:38.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:38 smithi084 ceph-mon[29774]: 5.15 scrub starts 2024-09-20T22:57:38.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:38 smithi084 ceph-mon[29774]: 5.15 scrub ok 2024-09-20T22:57:38.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:38 smithi084 ceph-mon[29774]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T22:57:38.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:38 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/2991139035' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T22:57:38.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:38 smithi084 ceph-mon[29774]: osdmap e977: 8 total, 7 up, 7 in 2024-09-20T22:57:38.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:38 smithi084 ceph-mon[29774]: 4.e scrub starts 2024-09-20T22:57:38.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:38 smithi084 ceph-mon[29774]: 4.e scrub ok 2024-09-20T22:57:38.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:38 smithi084 ceph-mon[29774]: Health check update: Degraded data redundancy: 300/1704 objects degraded (17.606%), 37 pgs degraded (PG_DEGRADED) 2024-09-20T22:57:38.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:38 smithi084 ceph-mon[29774]: osdmap e978: 8 total, 7 up, 7 in 2024-09-20T22:57:38.866 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:38 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:57:38.866 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:38 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:57:38.866 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:38 smithi084 ceph-mon[40024]: 5.15 scrub starts 2024-09-20T22:57:38.866 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:38 smithi084 ceph-mon[40024]: 5.15 scrub ok 2024-09-20T22:57:38.866 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:38 smithi084 ceph-mon[40024]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T22:57:38.866 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:38 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/2991139035' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T22:57:38.866 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:38 smithi084 ceph-mon[40024]: osdmap e977: 8 total, 7 up, 7 in 2024-09-20T22:57:38.866 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:38 smithi084 ceph-mon[40024]: 4.e scrub starts 2024-09-20T22:57:38.866 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:38 smithi084 ceph-mon[40024]: 4.e scrub ok 2024-09-20T22:57:38.867 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:38 smithi084 ceph-mon[40024]: Health check update: Degraded data redundancy: 300/1704 objects degraded (17.606%), 37 pgs degraded (PG_DEGRADED) 2024-09-20T22:57:38.867 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:38 smithi084 ceph-mon[40024]: osdmap e978: 8 total, 7 up, 7 in 2024-09-20T22:57:38.867 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:38 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:57:38.867 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:38 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:57:38.916 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-20T22:57:38.916 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [], 2024-09-20T22:57:38.916 INFO:teuthology.orchestra.run.smithi154.stdout: "num_ops": 0 2024-09-20T22:57:38.916 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-20T22:57:38.944 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:57:38.985 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:38.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:38.986 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:57:39.087 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:57:39.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:39.111 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:39.111 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:57:39.212 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:57:39.233 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:39.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:39.235 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:57:39.282 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:57:39.336 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:57:39.358 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:39.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:39.359 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:57:39.364 INFO:journalctl@ceph.osd.3.smithi084.stdout:Sep 20 22:57:38 smithi084 podman[212682]: 2024-09-20 22:57:38.944886756 +0000 UTC m=+3.153767646 container died a516fee9da47e4cc143d72d860ae50abded1c23486d8453c20b40a81c5a1294a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-3-activate, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=squid-9e2ac70, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-09-20T22:57:39.382 INFO:tasks.rados.rados.0.smithi084.stdout:3961: done (2 left) 2024-09-20T22:57:39.382 INFO:tasks.rados.rados.0.smithi084.stdout:3962: read oid 48 snap -1 2024-09-20T22:57:39.382 INFO:tasks.rados.rados.0.smithi084.stdout:3962: expect (ObjNum 1141 snap 385 seq_num 1141) 2024-09-20T22:57:39.382 INFO:tasks.rados.rados.0.smithi084.stdout:3963: write oid 24 current snap is 459 2024-09-20T22:57:39.382 INFO:tasks.rados.rados.0.smithi084.stdout:3963: seq_num 1342 ranges {554994=662286,1997188=557806} 2024-09-20T22:57:39.389 INFO:tasks.rados.rados.0.smithi084.stdout:3963: writing smithi08477203-24 from 554994 to 1217280 tid 1 2024-09-20T22:57:39.397 INFO:tasks.rados.rados.0.smithi084.stdout:3963: writing smithi08477203-24 from 1997188 to 2554994 tid 2 2024-09-20T22:57:39.397 INFO:tasks.rados.rados.0.smithi084.stdout:3964: copy_from oid 49 from oid 5 current snap is 459 2024-09-20T22:57:39.397 INFO:tasks.rados.rados.0.smithi084.stdout:3959: finishing rollback tid 0 to smithi08477203-38 2024-09-20T22:57:39.398 INFO:tasks.rados.rados.0.smithi084.stdout:3959: finishing rollback tid 1 to smithi08477203-38 2024-09-20T22:57:39.398 INFO:tasks.rados.rados.0.smithi084.stdout:3959: finishing rollback tid 2 to smithi08477203-38 2024-09-20T22:57:39.398 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 38 v 2863 (ObjNum 1143 snap 389 seq_num 1143) dirty exists 2024-09-20T22:57:39.398 INFO:tasks.rados.rados.0.smithi084.stdout:3958: done (4 left) 2024-09-20T22:57:39.398 INFO:tasks.rados.rados.0.smithi084.stdout:3959: done (3 left) 2024-09-20T22:57:39.398 INFO:tasks.rados.rados.0.smithi084.stdout:3965: read oid 13 snap -1 2024-09-20T22:57:39.398 INFO:tasks.rados.rados.0.smithi084.stdout:3965: expect (ObjNum 1074 snap 358 seq_num 1074) 2024-09-20T22:57:39.398 INFO:tasks.rados.rados.0.smithi084.stdout:3966: read oid 32 snap -1 2024-09-20T22:57:39.398 INFO:tasks.rados.rados.0.smithi084.stdout:3966: expect (ObjNum 1289 snap 436 seq_num 1289) 2024-09-20T22:57:39.398 INFO:tasks.rados.rados.0.smithi084.stdout:3967: write oid 25 current snap is 459 2024-09-20T22:57:39.399 INFO:tasks.rados.rados.0.smithi084.stdout:3962: expect (ObjNum 1141 snap 385 seq_num 1141) 2024-09-20T22:57:39.460 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:57:39.481 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:39.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:39.482 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:57:39.542 INFO:tasks.rados.rados.0.smithi084.stdout:3963: finishing write tid 1 to smithi08477203-24 2024-09-20T22:57:39.542 INFO:tasks.rados.rados.0.smithi084.stdout:3963: finishing write tid 2 to smithi08477203-24 2024-09-20T22:57:39.542 INFO:tasks.rados.rados.0.smithi084.stdout:3963: finishing write tid 3 to smithi08477203-24 2024-09-20T22:57:39.542 INFO:tasks.rados.rados.0.smithi084.stdout:3963: finishing write tid 4 to smithi08477203-24 2024-09-20T22:57:39.542 INFO:tasks.rados.rados.0.smithi084.stdout:3963: oid 24 updating version 0 to 1964 2024-09-20T22:57:39.542 INFO:tasks.rados.rados.0.smithi084.stdout:3963: oid 24 updating version 1964 to 1965 2024-09-20T22:57:39.542 INFO:tasks.rados.rados.0.smithi084.stdout:3963: oid 24 updating version 1965 to 1966 2024-09-20T22:57:39.542 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 24 v 1966 (ObjNum 1342 snap 459 seq_num 1342) dirty exists 2024-09-20T22:57:39.542 INFO:tasks.rados.rados.0.smithi084.stdout:3963: left oid 24 (ObjNum 1342 snap 459 seq_num 1342) 2024-09-20T22:57:39.543 INFO:tasks.rados.rados.0.smithi084.stdout:3966: expect (ObjNum 1289 snap 436 seq_num 1289) 2024-09-20T22:57:39.583 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:57:39.604 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:39.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:39.605 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:57:39.673 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.4 dump_blocked_ops 2024-09-20T22:57:39.678 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:39 smithi084 ceph-mon[40024]: 1.1 scrub starts 2024-09-20T22:57:39.678 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:39 smithi084 ceph-mon[40024]: 1.1 scrub ok 2024-09-20T22:57:39.678 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:39 smithi084 ceph-mon[40024]: pgmap v1368: 168 pgs: 1 active+clean+remapped, 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+clean+snaptrim, 41 active+undersized, 1 active+remapped+backfill_toofull, 36 active+undersized+degraded, 87 active+clean; 827 MiB data, 2.3 GiB used, 623 GiB / 626 GiB avail; 300/1704 objects degraded (17.606%); 55/1704 objects misplaced (3.228%) 2024-09-20T22:57:39.678 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:39 smithi084 ceph-mon[40024]: 5.6 deep-scrub starts 2024-09-20T22:57:39.678 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:39 smithi084 ceph-mon[40024]: 5.6 deep-scrub ok 2024-09-20T22:57:39.678 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:39 smithi084 ceph-mon[40024]: purged_snaps scrub starts 2024-09-20T22:57:39.678 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:39 smithi084 ceph-mon[40024]: purged_snaps scrub ok 2024-09-20T22:57:39.678 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:39 smithi084 ceph-mon[40024]: 6.10 scrub starts 2024-09-20T22:57:39.678 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:39 smithi084 ceph-mon[40024]: 6.10 scrub ok 2024-09-20T22:57:39.678 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:39 smithi084 ceph-mon[40024]: osdmap e979: 8 total, 7 up, 7 in 2024-09-20T22:57:39.679 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:39 smithi084 ceph-mon[29774]: 1.1 scrub starts 2024-09-20T22:57:39.679 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:39 smithi084 ceph-mon[29774]: 1.1 scrub ok 2024-09-20T22:57:39.679 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:39 smithi084 ceph-mon[29774]: pgmap v1368: 168 pgs: 1 active+clean+remapped, 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+clean+snaptrim, 41 active+undersized, 1 active+remapped+backfill_toofull, 36 active+undersized+degraded, 87 active+clean; 827 MiB data, 2.3 GiB used, 623 GiB / 626 GiB avail; 300/1704 objects degraded (17.606%); 55/1704 objects misplaced (3.228%) 2024-09-20T22:57:39.679 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:39 smithi084 ceph-mon[29774]: 5.6 deep-scrub starts 2024-09-20T22:57:39.679 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:39 smithi084 ceph-mon[29774]: 5.6 deep-scrub ok 2024-09-20T22:57:39.679 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:39 smithi084 ceph-mon[29774]: purged_snaps scrub starts 2024-09-20T22:57:39.679 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:39 smithi084 ceph-mon[29774]: purged_snaps scrub ok 2024-09-20T22:57:39.679 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:39 smithi084 ceph-mon[29774]: 6.10 scrub starts 2024-09-20T22:57:39.679 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:39 smithi084 ceph-mon[29774]: 6.10 scrub ok 2024-09-20T22:57:39.679 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:39 smithi084 ceph-mon[29774]: osdmap e979: 8 total, 7 up, 7 in 2024-09-20T22:57:39.706 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:57:39.711 INFO:tasks.rados.rados.0.smithi084.stdout:3965: expect (ObjNum 1074 snap 358 seq_num 1074) 2024-09-20T22:57:39.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:39.737 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:39.737 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:57:39.778 INFO:tasks.rados.rados.0.smithi084.stdout:3964: finishing copy_from racing read to smithi08477203-49 2024-09-20T22:57:39.778 INFO:tasks.rados.rados.0.smithi084.stdout:3964: finishing copy_from to smithi08477203-49 2024-09-20T22:57:39.778 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 49 v 2662 (ObjNum 1241 snap 421 seq_num 1241) dirty exists 2024-09-20T22:57:39.778 INFO:tasks.rados.rados.0.smithi084.stdout:3967: seq_num 1343 ranges {571801=737535,1939317=489789} 2024-09-20T22:57:39.779 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:57:39.779 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:57:39.785 INFO:tasks.rados.rados.0.smithi084.stdout:3967: writing smithi08477203-25 from 571801 to 1309336 tid 1 2024-09-20T22:57:39.792 INFO:tasks.rados.rados.0.smithi084.stdout:3967: writing smithi08477203-25 from 1939317 to 2429106 tid 2 2024-09-20T22:57:39.792 INFO:tasks.rados.rados.0.smithi084.stdout:3962: done (5 left) 2024-09-20T22:57:39.792 INFO:tasks.rados.rados.0.smithi084.stdout:3963: done (4 left) 2024-09-20T22:57:39.792 INFO:tasks.rados.rados.0.smithi084.stdout:3964: done (3 left) 2024-09-20T22:57:39.792 INFO:tasks.rados.rados.0.smithi084.stdout:3965: done (2 left) 2024-09-20T22:57:39.792 INFO:tasks.rados.rados.0.smithi084.stdout:3966: done (1 left) 2024-09-20T22:57:39.792 INFO:tasks.rados.rados.0.smithi084.stdout:3968: snap_create 2024-09-20T22:57:39.792 INFO:tasks.rados.rados.0.smithi084.stdout:3967: finishing write tid 1 to smithi08477203-25 2024-09-20T22:57:39.795 INFO:tasks.rados.rados.0.smithi084.stdout:3967: finishing write tid 2 to smithi08477203-25 2024-09-20T22:57:39.795 INFO:tasks.rados.rados.0.smithi084.stdout:3967: finishing write tid 3 to smithi08477203-25 2024-09-20T22:57:39.795 INFO:tasks.rados.rados.0.smithi084.stdout:3967: finishing write tid 4 to smithi08477203-25 2024-09-20T22:57:39.795 INFO:tasks.rados.rados.0.smithi084.stdout:3967: oid 25 updating version 0 to 3151 2024-09-20T22:57:39.795 INFO:tasks.rados.rados.0.smithi084.stdout:3967: oid 25 updating version 3151 to 3153 2024-09-20T22:57:39.796 INFO:tasks.rados.rados.0.smithi084.stdout:3967: oid 25 version 3153 is already newer than 3152 2024-09-20T22:57:39.796 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 25 v 3153 (ObjNum 1343 snap 459 seq_num 1343) dirty exists 2024-09-20T22:57:39.796 INFO:tasks.rados.rados.0.smithi084.stdout:3967: left oid 25 (ObjNum 1343 snap 459 seq_num 1343) 2024-09-20T22:57:39.797 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:57:39.798 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:57:39.813 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:57:39.814 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:57:39.814 INFO:teuthology.orchestra.run.smithi084.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-20T22:57:39.830 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:57:39.830 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:57:39.838 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:57:39.847 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:57:39.847 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:57:39.859 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:39.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:39.860 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:57:39.864 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:57:39.864 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:57:39.882 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:57:39.882 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:57:39.924 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:57:39.961 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:57:39.983 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:39.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:39.984 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:57:40.004 INFO:journalctl@ceph.iscsi.iscsi.a.smithi154.stdout:Sep 20 22:57:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-iscsi-iscsi-a[59730]: debug there is no tcmu-runner data available 2024-09-20T22:57:40.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:39 smithi154 ceph-mon[33264]: 1.1 scrub starts 2024-09-20T22:57:40.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:39 smithi154 ceph-mon[33264]: 1.1 scrub ok 2024-09-20T22:57:40.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:39 smithi154 ceph-mon[33264]: pgmap v1368: 168 pgs: 1 active+clean+remapped, 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+clean+snaptrim, 41 active+undersized, 1 active+remapped+backfill_toofull, 36 active+undersized+degraded, 87 active+clean; 827 MiB data, 2.3 GiB used, 623 GiB / 626 GiB avail; 300/1704 objects degraded (17.606%); 55/1704 objects misplaced (3.228%) 2024-09-20T22:57:40.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:39 smithi154 ceph-mon[33264]: 5.6 deep-scrub starts 2024-09-20T22:57:40.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:39 smithi154 ceph-mon[33264]: 5.6 deep-scrub ok 2024-09-20T22:57:40.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:39 smithi154 ceph-mon[33264]: purged_snaps scrub starts 2024-09-20T22:57:40.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:39 smithi154 ceph-mon[33264]: purged_snaps scrub ok 2024-09-20T22:57:40.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:39 smithi154 ceph-mon[33264]: 6.10 scrub starts 2024-09-20T22:57:40.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:39 smithi154 ceph-mon[33264]: 6.10 scrub ok 2024-09-20T22:57:40.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:39 smithi154 ceph-mon[33264]: osdmap e979: 8 total, 7 up, 7 in 2024-09-20T22:57:40.085 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:57:40.106 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:40.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:40.107 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:57:40.208 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:57:40.258 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:40.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:40.259 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:57:40.360 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:57:40.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:40.382 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:40.382 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:57:40.435 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-20T22:57:40.435 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:57:40.483 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:57:40.505 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:40.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:40.506 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:57:40.607 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:57:40.614 INFO:journalctl@ceph.osd.3.smithi084.stdout:Sep 20 22:57:40 smithi084 podman[213181]: 2024-09-20 22:57:40.322030537 +0000 UTC m=+1.899781468 container remove a516fee9da47e4cc143d72d860ae50abded1c23486d8453c20b40a81c5a1294a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-3-activate, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-9e2ac70, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS) 2024-09-20T22:57:40.627 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 22:57:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=sqlstore.transactions t=2024-09-20T22:57:40.523045849Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-09-20T22:57:40.631 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:40.632 INFO:tasks.rados.rados.0.smithi084.stdout:3967: done (1 left) 2024-09-20T22:57:40.632 INFO:tasks.rados.rados.0.smithi084.stdout:3968: done (0 left) 2024-09-20T22:57:40.632 INFO:tasks.rados.rados.0.smithi084.stdout:3969: snap_remove snap 417 2024-09-20T22:57:40.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:40.632 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:57:40.733 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:57:40.754 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:40.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:40.755 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:57:40.856 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:57:40.876 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:40 smithi084 ceph-mon[40024]: 3.3 scrub starts 2024-09-20T22:57:40.876 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:40 smithi084 ceph-mon[40024]: 3.3 scrub ok 2024-09-20T22:57:40.876 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:40 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/3798999918' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:57:40.877 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:40 smithi084 ceph-mon[29774]: 3.3 scrub starts 2024-09-20T22:57:40.877 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:40 smithi084 ceph-mon[29774]: 3.3 scrub ok 2024-09-20T22:57:40.877 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:40 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/3798999918' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:57:40.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:40.878 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:40.878 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:57:40.925 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd set noscrub 2024-09-20T22:57:40.971 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:40 smithi154 ceph-mon[33264]: 3.3 scrub starts 2024-09-20T22:57:40.971 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:40 smithi154 ceph-mon[33264]: 3.3 scrub ok 2024-09-20T22:57:40.971 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:40 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/3798999918' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:57:40.978 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:57:40.999 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.4 dump_historic_ops 2024-09-20T22:57:41.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:41.000 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:41.001 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:57:41.102 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:57:41.123 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:41.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:41.124 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:57:41.224 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:57:41.241 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:57:41.244 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-20T22:57:41.244 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-20T22:57:41.247 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:41.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:41.248 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:57:41.349 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:57:41.369 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:41.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:41.370 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:57:41.471 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:57:41.492 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:41.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:41.493 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:57:41.516 INFO:journalctl@ceph.osd.3.smithi084.stdout:Sep 20 22:57:41 smithi084 podman[213401]: 2024-09-20 22:57:41.264408613 +0000 UTC m=+0.847522201 image pull cc9251b744a3faafca7c3095f79bfff81873afe823f31a03a11e015a34abcdbd quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5 2024-09-20T22:57:41.516 INFO:journalctl@ceph.osd.3.smithi084.stdout:Sep 20 22:57:41 smithi084 podman[213401]: 2024-09-20 22:57:41.387292358 +0000 UTC m=+0.970405923 container create 4b99a5b19760e8d69d99cf992ccfdfd60a94511163b1c5c5d47819c81aa481a2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-3, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, RELEASE=squid-9e2ac70, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-09-20T22:57:41.594 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:57:41.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:41.617 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:41.617 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:57:41.632 INFO:tasks.rados.rados.0.smithi084.stdout:3969: done (0 left) 2024-09-20T22:57:41.632 INFO:tasks.rados.rados.0.smithi084.stdout:3970: write oid 43 current snap is 460 2024-09-20T22:57:41.632 INFO:tasks.rados.rados.0.smithi084.stdout:3970: seq_num 1344 ranges {588608=412784,1597799=749800} 2024-09-20T22:57:41.637 INFO:tasks.rados.rados.0.smithi084.stdout:3970: writing smithi08477203-43 from 588608 to 1001392 tid 1 2024-09-20T22:57:41.647 INFO:tasks.rados.rados.0.smithi084.stdout:3970: writing smithi08477203-43 from 1597799 to 2347599 tid 2 2024-09-20T22:57:41.647 INFO:tasks.rados.rados.0.smithi084.stdout:3971: snap_remove snap 382 2024-09-20T22:57:41.718 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:57:41.739 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:41.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:41.740 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:57:41.741 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-20T22:57:41.741 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:57:41.764 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-20T22:57:41.764 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:57:41.826 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:41 smithi084 ceph-mon[40024]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:57:41.826 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:41 smithi084 ceph-mon[40024]: pgmap v1371: 168 pgs: 1 active+clean+remapped, 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+clean+snaptrim, 41 active+undersized, 1 active+remapped+backfill_toofull, 36 active+undersized+degraded, 87 active+clean; 834 MiB data, 2.3 GiB used, 624 GiB / 626 GiB avail; 5.2 MiB/s rd, 776 KiB/s wr, 6 op/s; 303/1719 objects degraded (17.627%); 55/1719 objects misplaced (3.200%) 2024-09-20T22:57:41.827 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:41 smithi084 ceph-mon[40024]: osdmap e980: 8 total, 7 up, 7 in 2024-09-20T22:57:41.827 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:41 smithi084 ceph-mon[40024]: 4.9 scrub starts 2024-09-20T22:57:41.827 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:41 smithi084 ceph-mon[40024]: 4.9 scrub ok 2024-09-20T22:57:41.827 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:41 smithi084 ceph-mon[29774]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:57:41.827 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:41 smithi084 ceph-mon[29774]: pgmap v1371: 168 pgs: 1 active+clean+remapped, 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+clean+snaptrim, 41 active+undersized, 1 active+remapped+backfill_toofull, 36 active+undersized+degraded, 87 active+clean; 834 MiB data, 2.3 GiB used, 624 GiB / 626 GiB avail; 5.2 MiB/s rd, 776 KiB/s wr, 6 op/s; 303/1719 objects degraded (17.627%); 55/1719 objects misplaced (3.200%) 2024-09-20T22:57:41.827 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:41 smithi084 ceph-mon[29774]: osdmap e980: 8 total, 7 up, 7 in 2024-09-20T22:57:41.827 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:41 smithi084 ceph-mon[29774]: 4.9 scrub starts 2024-09-20T22:57:41.827 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:41 smithi084 ceph-mon[29774]: 4.9 scrub ok 2024-09-20T22:57:41.828 INFO:journalctl@ceph.osd.3.smithi084.stdout:Sep 20 22:57:41 smithi084 podman[213401]: 2024-09-20 22:57:41.647772004 +0000 UTC m=+1.230885575 container init 4b99a5b19760e8d69d99cf992ccfdfd60a94511163b1c5c5d47819c81aa481a2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-3, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=squid-9e2ac70, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731) 2024-09-20T22:57:41.828 INFO:journalctl@ceph.osd.3.smithi084.stdout:Sep 20 22:57:41 smithi084 podman[213401]: 2024-09-20 22:57:41.652348982 +0000 UTC m=+1.235462543 container start 4b99a5b19760e8d69d99cf992ccfdfd60a94511163b1c5c5d47819c81aa481a2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-3, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=squid-9e2ac70, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0) 2024-09-20T22:57:41.828 INFO:journalctl@ceph.osd.3.smithi084.stdout:Sep 20 22:57:41 smithi084 ceph-osd[213566]: -- 172.21.15.84:0/2617655961 <== mon.2 v2:172.21.15.84:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56150b97da40 con 0x56150c744400 2024-09-20T22:57:41.828 INFO:journalctl@ceph.osd.3.smithi084.stdout:Sep 20 22:57:41 smithi084 bash[213401]: 4b99a5b19760e8d69d99cf992ccfdfd60a94511163b1c5c5d47819c81aa481a2 2024-09-20T22:57:41.828 INFO:journalctl@ceph.osd.3.smithi084.stdout:Sep 20 22:57:41 smithi084 systemd[1]: Started Ceph osd.3 for a9c0bc70-77a0-11ef-baf5-efdc52797490. 2024-09-20T22:57:41.841 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:57:41.862 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:41.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:41.863 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:57:41.963 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:57:41.985 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:41.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:41.986 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:57:42.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:41 smithi154 ceph-mon[33264]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:57:42.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:41 smithi154 ceph-mon[33264]: pgmap v1371: 168 pgs: 1 active+clean+remapped, 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+clean+snaptrim, 41 active+undersized, 1 active+remapped+backfill_toofull, 36 active+undersized+degraded, 87 active+clean; 834 MiB data, 2.3 GiB used, 624 GiB / 626 GiB avail; 5.2 MiB/s rd, 776 KiB/s wr, 6 op/s; 303/1719 objects degraded (17.627%); 55/1719 objects misplaced (3.200%) 2024-09-20T22:57:42.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:41 smithi154 ceph-mon[33264]: osdmap e980: 8 total, 7 up, 7 in 2024-09-20T22:57:42.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:41 smithi154 ceph-mon[33264]: 4.9 scrub starts 2024-09-20T22:57:42.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:41 smithi154 ceph-mon[33264]: 4.9 scrub ok 2024-09-20T22:57:42.087 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:57:42.109 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:42.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:42.110 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:57:42.210 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:57:42.231 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:42.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:42.232 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:57:42.293 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:57:42.333 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:57:42.353 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:42.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:42.354 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:57:42.382 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-20T22:57:42.455 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:57:42.486 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:42.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:42.487 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:57:42.588 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:57:42.608 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:42.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:42.609 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:57:42.641 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is set 2024-09-20T22:57:42.648 INFO:tasks.rados.rados.0.smithi084.stdout:3971: done (1 left) 2024-09-20T22:57:42.648 INFO:tasks.rados.rados.0.smithi084.stdout:3972: snap_remove snap 404 2024-09-20T22:57:42.675 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:57:42.697 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:42 smithi084 ceph-mon[29774]: osdmap e981: 8 total, 7 up, 7 in 2024-09-20T22:57:42.697 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:42 smithi084 ceph-mon[29774]: purged_snaps scrub starts 2024-09-20T22:57:42.697 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:42 smithi084 ceph-mon[29774]: purged_snaps scrub ok 2024-09-20T22:57:42.697 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:42 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/3620753788' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-20T22:57:42.697 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:42 smithi084 ceph-mon[40024]: osdmap e981: 8 total, 7 up, 7 in 2024-09-20T22:57:42.697 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:42 smithi084 ceph-mon[40024]: purged_snaps scrub starts 2024-09-20T22:57:42.698 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:42 smithi084 ceph-mon[40024]: purged_snaps scrub ok 2024-09-20T22:57:42.698 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:42 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/3620753788' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-20T22:57:42.710 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:57:42.731 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:42.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:42.732 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:57:42.792 INFO:teuthology.orchestra.run.smithi084.stderr:no valid command found; 10 closest matches: 2024-09-20T22:57:42.792 INFO:teuthology.orchestra.run.smithi084.stderr:0 2024-09-20T22:57:42.792 INFO:teuthology.orchestra.run.smithi084.stderr:1 2024-09-20T22:57:42.792 INFO:teuthology.orchestra.run.smithi084.stderr:2 2024-09-20T22:57:42.792 INFO:teuthology.orchestra.run.smithi084.stderr:abort 2024-09-20T22:57:42.792 INFO:teuthology.orchestra.run.smithi084.stderr:assert 2024-09-20T22:57:42.792 INFO:teuthology.orchestra.run.smithi084.stderr:config diff 2024-09-20T22:57:42.792 INFO:teuthology.orchestra.run.smithi084.stderr:config diff get 2024-09-20T22:57:42.792 INFO:teuthology.orchestra.run.smithi084.stderr:config get 2024-09-20T22:57:42.792 INFO:teuthology.orchestra.run.smithi084.stderr:config help [] 2024-09-20T22:57:42.792 INFO:teuthology.orchestra.run.smithi084.stderr:config set ... 2024-09-20T22:57:42.792 INFO:teuthology.orchestra.run.smithi084.stderr:admin_socket: invalid command 2024-09-20T22:57:42.833 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:57:42.854 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:42.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:42.855 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:57:42.917 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:42 smithi154 ceph-mon[33264]: osdmap e981: 8 total, 7 up, 7 in 2024-09-20T22:57:42.917 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:42 smithi154 ceph-mon[33264]: purged_snaps scrub starts 2024-09-20T22:57:42.917 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:42 smithi154 ceph-mon[33264]: purged_snaps scrub ok 2024-09-20T22:57:42.917 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:42 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/3620753788' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-20T22:57:42.955 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:57:42.977 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:42.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:42.978 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:57:43.078 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:57:43.100 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:43.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:43.101 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:57:43.109 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-20T22:57:43.109 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:57:43.201 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:57:43.224 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:43.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:43.225 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:57:43.245 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-20T22:57:43.267 INFO:journalctl@ceph.osd.3.smithi084.stdout:Sep 20 22:57:43 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-3[213561]: 2024-09-20T22:57:43.059+0000 7f2fcc410640 -1 Falling back to public interface 2024-09-20T22:57:43.325 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:57:43.346 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:43.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:43.347 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:57:43.448 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:57:43.469 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:43.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:43.470 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:57:43.571 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:57:43.592 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:43.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:43.593 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:57:43.650 INFO:tasks.rados.rados.0.smithi084.stdout:3972: done (1 left) 2024-09-20T22:57:43.650 INFO:tasks.rados.rados.0.smithi084.stdout:3973: rollback oid 38 current snap is 460 2024-09-20T22:57:43.650 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 38 to 437 2024-09-20T22:57:43.651 INFO:tasks.rados.rados.0.smithi084.stdout:3974: read oid 50 snap 396 2024-09-20T22:57:43.651 INFO:tasks.rados.rados.0.smithi084.stdout:3974: expect (ObjNum 1128 snap 381 seq_num 1128) 2024-09-20T22:57:43.651 INFO:tasks.rados.rados.0.smithi084.stdout:3975: read oid 15 snap 387 2024-09-20T22:57:43.651 INFO:tasks.rados.rados.0.smithi084.stdout:3975: expect (ObjNum 1097 snap 368 seq_num 1097) 2024-09-20T22:57:43.651 INFO:tasks.rados.rados.0.smithi084.stdout:3976: snap_create 2024-09-20T22:57:43.651 INFO:tasks.rados.rados.0.smithi084.stdout:3970: finishing write tid 1 to smithi08477203-43 2024-09-20T22:57:43.651 INFO:tasks.rados.rados.0.smithi084.stdout:3970: finishing write tid 2 to smithi08477203-43 2024-09-20T22:57:43.651 INFO:tasks.rados.rados.0.smithi084.stdout:3970: finishing write tid 3 to smithi08477203-43 2024-09-20T22:57:43.651 INFO:tasks.rados.rados.0.smithi084.stdout:3970: finishing write tid 4 to smithi08477203-43 2024-09-20T22:57:43.651 INFO:tasks.rados.rados.0.smithi084.stdout:3970: oid 43 updating version 0 to 1958 2024-09-20T22:57:43.651 INFO:tasks.rados.rados.0.smithi084.stdout:3970: oid 43 version 1958 is already newer than 1956 2024-09-20T22:57:43.651 INFO:tasks.rados.rados.0.smithi084.stdout:3970: oid 43 version 1958 is already newer than 1957 2024-09-20T22:57:43.651 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 43 v 1958 (ObjNum 1344 snap 460 seq_num 1344) dirty exists 2024-09-20T22:57:43.651 INFO:tasks.rados.rados.0.smithi084.stdout:3970: left oid 43 (ObjNum 1344 snap 460 seq_num 1344) 2024-09-20T22:57:43.660 INFO:tasks.rados.rados.0.smithi084.stdout:3974: expect (ObjNum 1128 snap 381 seq_num 1128) 2024-09-20T22:57:43.693 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:57:43.716 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:43.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:43.717 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:57:43.749 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.5 dump_blocked_ops 2024-09-20T22:57:43.817 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:57:43.836 INFO:tasks.rados.rados.0.smithi084.stdout:3975: expect (ObjNum 1097 snap 368 seq_num 1097) 2024-09-20T22:57:43.839 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:43.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:43.842 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:57:43.918 INFO:tasks.rados.rados.0.smithi084.stdout:3973: finishing rollback tid 0 to smithi08477203-38 2024-09-20T22:57:43.918 INFO:tasks.rados.rados.0.smithi084.stdout:3973: finishing rollback tid 1 to smithi08477203-38 2024-09-20T22:57:43.918 INFO:tasks.rados.rados.0.smithi084.stdout:3973: finishing rollback tid 2 to smithi08477203-38 2024-09-20T22:57:43.918 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 38 v 2866 (ObjNum 1276 snap 433 seq_num 1276) dirty exists 2024-09-20T22:57:43.942 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:57:43.963 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:43.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:43.964 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:57:43.995 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:57:44.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:43 smithi154 ceph-mon[33264]: 1.a scrub starts 2024-09-20T22:57:44.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:43 smithi154 ceph-mon[33264]: 1.a scrub ok 2024-09-20T22:57:44.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:43 smithi154 ceph-mon[33264]: pgmap v1374: 168 pgs: 1 active+clean+remapped, 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+clean+snaptrim, 41 active+undersized, 1 active+remapped+backfill_toofull, 36 active+undersized+degraded, 87 active+clean; 834 MiB data, 2.3 GiB used, 624 GiB / 626 GiB avail; 5.2 MiB/s rd, 776 KiB/s wr, 6 op/s; 303/1719 objects degraded (17.627%); 55/1719 objects misplaced (3.200%) 2024-09-20T22:57:44.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:43 smithi154 ceph-mon[33264]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:57:44.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:43 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/3620753788' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T22:57:44.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:43 smithi154 ceph-mon[33264]: osdmap e982: 8 total, 7 up, 7 in 2024-09-20T22:57:44.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:43 smithi154 ceph-mon[33264]: Health check update: Degraded data redundancy: 303/1719 objects degraded (17.627%), 37 pgs degraded (PG_DEGRADED) 2024-09-20T22:57:44.065 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:57:44.085 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:44.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:44.086 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:57:44.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:43 smithi084 ceph-mon[40024]: 1.a scrub starts 2024-09-20T22:57:44.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:43 smithi084 ceph-mon[40024]: 1.a scrub ok 2024-09-20T22:57:44.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:43 smithi084 ceph-mon[40024]: pgmap v1374: 168 pgs: 1 active+clean+remapped, 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+clean+snaptrim, 41 active+undersized, 1 active+remapped+backfill_toofull, 36 active+undersized+degraded, 87 active+clean; 834 MiB data, 2.3 GiB used, 624 GiB / 626 GiB avail; 5.2 MiB/s rd, 776 KiB/s wr, 6 op/s; 303/1719 objects degraded (17.627%); 55/1719 objects misplaced (3.200%) 2024-09-20T22:57:44.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:43 smithi084 ceph-mon[40024]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:57:44.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:43 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/3620753788' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T22:57:44.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:43 smithi084 ceph-mon[40024]: osdmap e982: 8 total, 7 up, 7 in 2024-09-20T22:57:44.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:43 smithi084 ceph-mon[40024]: Health check update: Degraded data redundancy: 303/1719 objects degraded (17.627%), 37 pgs degraded (PG_DEGRADED) 2024-09-20T22:57:44.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:43 smithi084 ceph-mon[29774]: 1.a scrub starts 2024-09-20T22:57:44.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:43 smithi084 ceph-mon[29774]: 1.a scrub ok 2024-09-20T22:57:44.116 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:43 smithi084 ceph-mon[29774]: pgmap v1374: 168 pgs: 1 active+clean+remapped, 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+clean+snaptrim, 41 active+undersized, 1 active+remapped+backfill_toofull, 36 active+undersized+degraded, 87 active+clean; 834 MiB data, 2.3 GiB used, 624 GiB / 626 GiB avail; 5.2 MiB/s rd, 776 KiB/s wr, 6 op/s; 303/1719 objects degraded (17.627%); 55/1719 objects misplaced (3.200%) 2024-09-20T22:57:44.116 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:43 smithi084 ceph-mon[29774]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:57:44.116 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:43 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/3620753788' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T22:57:44.116 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:43 smithi084 ceph-mon[29774]: osdmap e982: 8 total, 7 up, 7 in 2024-09-20T22:57:44.116 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:43 smithi084 ceph-mon[29774]: Health check update: Degraded data redundancy: 303/1719 objects degraded (17.627%), 37 pgs degraded (PG_DEGRADED) 2024-09-20T22:57:44.187 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:57:44.209 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:44.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:44.210 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:57:44.221 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-20T22:57:44.221 INFO:tasks.cephadm.ceph_manager.ceph:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-09-20T22:57:44.258 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:57:44.310 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:57:44.332 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:44.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:44.333 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:57:44.434 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:57:44.453 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-20T22:57:44.453 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:57:44.455 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:44.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:44.456 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:57:44.557 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:57:44.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:44.579 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:44.579 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:57:44.657 INFO:tasks.rados.rados.0.smithi084.stdout:3970: done (4 left) 2024-09-20T22:57:44.658 INFO:tasks.rados.rados.0.smithi084.stdout:3973: done (3 left) 2024-09-20T22:57:44.658 INFO:tasks.rados.rados.0.smithi084.stdout:3974: done (2 left) 2024-09-20T22:57:44.658 INFO:tasks.rados.rados.0.smithi084.stdout:3975: done (1 left) 2024-09-20T22:57:44.658 INFO:tasks.rados.rados.0.smithi084.stdout:3976: done (0 left) 2024-09-20T22:57:44.658 INFO:tasks.rados.rados.0.smithi084.stdout:3977: delete oid 16 current snap is 461 2024-09-20T22:57:44.662 INFO:tasks.rados.rados.0.smithi084.stdout:3977: done (0 left) 2024-09-20T22:57:44.662 INFO:tasks.rados.rados.0.smithi084.stdout:3978: write oid 9 current snap is 461 2024-09-20T22:57:44.662 INFO:tasks.rados.rados.0.smithi084.stdout:3978: seq_num 1346 ranges {622222=679635,1831116=753469} 2024-09-20T22:57:44.669 INFO:tasks.rados.rados.0.smithi084.stdout:3978: writing smithi08477203-9 from 622222 to 1301857 tid 1 2024-09-20T22:57:44.676 INFO:tasks.rados.rados.0.smithi084.stdout:3978: writing smithi08477203-9 from 1831116 to 2584585 tid 2 2024-09-20T22:57:44.676 INFO:tasks.rados.rados.0.smithi084.stdout:3979: snap_create 2024-09-20T22:57:44.680 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:57:44.680 INFO:tasks.rados.rados.0.smithi084.stdout:3978: finishing write tid 1 to smithi08477203-9 2024-09-20T22:57:44.681 INFO:tasks.rados.rados.0.smithi084.stdout:3978: finishing write tid 2 to smithi08477203-9 2024-09-20T22:57:44.681 INFO:tasks.rados.rados.0.smithi084.stdout:3978: finishing write tid 3 to smithi08477203-9 2024-09-20T22:57:44.681 INFO:tasks.rados.rados.0.smithi084.stdout:3978: finishing write tid 4 to smithi08477203-9 2024-09-20T22:57:44.682 INFO:tasks.rados.rados.0.smithi084.stdout:3978: oid 9 updating version 0 to 2694 2024-09-20T22:57:44.682 INFO:tasks.rados.rados.0.smithi084.stdout:3978: oid 9 version 2694 is already newer than 2692 2024-09-20T22:57:44.682 INFO:tasks.rados.rados.0.smithi084.stdout:3978: oid 9 version 2694 is already newer than 2693 2024-09-20T22:57:44.682 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 9 v 2694 (ObjNum 1346 snap 461 seq_num 1346) dirty exists 2024-09-20T22:57:44.682 INFO:tasks.rados.rados.0.smithi084.stdout:3978: left oid 9 (ObjNum 1346 snap 461 seq_num 1346) 2024-09-20T22:57:44.701 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:44.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:44.702 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:57:44.722 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:57:44.723 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:57:44.739 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:57:44.739 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:57:44.739 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:44 smithi084 ceph-mon[40024]: osdmap e983: 8 total, 7 up, 7 in 2024-09-20T22:57:44.740 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:44 smithi084 ceph-mon[40024]: purged_snaps scrub starts 2024-09-20T22:57:44.740 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:44 smithi084 ceph-mon[40024]: purged_snaps scrub ok 2024-09-20T22:57:44.740 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:44 smithi084 ceph-mon[29774]: osdmap e983: 8 total, 7 up, 7 in 2024-09-20T22:57:44.740 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:44 smithi084 ceph-mon[29774]: purged_snaps scrub starts 2024-09-20T22:57:44.740 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:44 smithi084 ceph-mon[29774]: purged_snaps scrub ok 2024-09-20T22:57:44.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:44 smithi154 ceph-mon[33264]: osdmap e983: 8 total, 7 up, 7 in 2024-09-20T22:57:44.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:44 smithi154 ceph-mon[33264]: purged_snaps scrub starts 2024-09-20T22:57:44.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:44 smithi154 ceph-mon[33264]: purged_snaps scrub ok 2024-09-20T22:57:44.755 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:57:44.755 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:57:44.755 INFO:teuthology.orchestra.run.smithi084.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-09-20T22:57:44.772 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:57:44.772 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:57:44.788 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:57:44.788 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:57:44.803 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:57:44.804 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:57:44.805 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:57:44.821 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:57:44.821 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:57:44.824 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:44.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:44.825 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:57:44.925 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:57:44.947 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:44.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:44.948 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:57:45.049 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:57:45.066 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.5 dump_historic_ops 2024-09-20T22:57:45.070 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:45.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:45.071 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:57:45.172 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:57:45.193 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:45.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:45.194 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:57:45.295 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:57:45.310 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:57:45.315 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:45.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:45.316 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:57:45.417 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:57:45.438 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:45.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:45.440 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:57:45.540 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:57:45.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:45.563 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:45.563 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:57:45.580 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-20T22:57:45.580 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-20T22:57:45.664 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:57:45.665 INFO:tasks.rados.rados.0.smithi084.stdout:3978: done (1 left) 2024-09-20T22:57:45.665 INFO:tasks.rados.rados.0.smithi084.stdout:3979: done (0 left) 2024-09-20T22:57:45.666 INFO:tasks.rados.rados.0.smithi084.stdout:3980: snap_remove snap 437 2024-09-20T22:57:45.685 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:45.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:45.686 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:57:45.701 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:45 smithi154 ceph-mon[33264]: pgmap v1377: 168 pgs: 1 active+clean+remapped, 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+clean+snaptrim, 41 active+undersized, 1 active+remapped+backfill_toofull, 36 active+undersized+degraded, 87 active+clean; 838 MiB data, 2.3 GiB used, 623 GiB / 626 GiB avail; 11 MiB/s rd, 1.5 MiB/s wr, 14 op/s; 306/1725 objects degraded (17.739%); 55/1725 objects misplaced (3.188%) 2024-09-20T22:57:45.701 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:45 smithi154 ceph-mon[33264]: osdmap e984: 8 total, 7 up, 7 in 2024-09-20T22:57:45.701 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:45 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/3517535894' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:57:45.702 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:45 smithi154 ceph-mon[33264]: purged_snaps scrub starts 2024-09-20T22:57:45.702 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:45 smithi154 ceph-mon[33264]: purged_snaps scrub ok 2024-09-20T22:57:45.737 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-20T22:57:45.787 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:57:45.800 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-20T22:57:45.801 INFO:teuthology.orchestra.run.smithi154.stdout: "size": 20, 2024-09-20T22:57:45.801 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 600, 2024-09-20T22:57:45.801 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [ 2024-09-20T22:57:45.801 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.801 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:4444 7.0 7:05bb442d:::smithi08477203-24:head [copy-from ver 421] snapc b8=[b8,b7,b6,b5,b4,b3,b2,b1,b0,ae,a9,a7,a6,a5,9e,9c,96,95,87,86] ondisk+write+known_if_redirected+supports_pool_eio e447)", 2024-09-20T22:57:45.801 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:48:47.227411+0000", 2024-09-20T22:57:45.801 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 538.57055094999998, 2024-09-20T22:57:45.801 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.033566952999999997, 2024-09-20T22:57:45.801 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:57:45.801 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:57:45.801 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:57:45.802 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:57:45.802 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:57:45.802 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:57:45.802 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 4444 2024-09-20T22:57:45.802 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.802 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:57:45.802 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.802 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:57:45.802 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.227411+0000", 2024-09-20T22:57:45.802 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:57:45.802 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.802 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.803 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:57:45.803 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.227411+0000", 2024-09-20T22:57:45.803 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:57:45.803 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.803 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.803 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:57:45.803 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.227413+0000", 2024-09-20T22:57:45.803 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.057e-06 2024-09-20T22:57:45.803 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.803 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.803 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:57:45.803 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.227422+0000", 2024-09-20T22:57:45.803 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.3219999999999999e-06 2024-09-20T22:57:45.803 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.804 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.804 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:57:45.804 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.227442+0000", 2024-09-20T22:57:45.804 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9449999999999998e-05 2024-09-20T22:57:45.804 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.804 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.804 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:57:45.804 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.227452+0000", 2024-09-20T22:57:45.804 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0699999999999999e-05 2024-09-20T22:57:45.804 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.804 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.804 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:57:45.804 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.227531+0000", 2024-09-20T22:57:45.805 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.8993999999999996e-05 2024-09-20T22:57:45.805 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.805 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.805 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op must wait for map", 2024-09-20T22:57:45.805 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.227542+0000", 2024-09-20T22:57:45.805 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0797e-05 2024-09-20T22:57:45.805 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.805 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.805 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:57:45.805 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.229501+0000", 2024-09-20T22:57:45.805 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0019594080000000002 2024-09-20T22:57:45.805 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.805 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.805 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:57:45.806 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.229616+0000", 2024-09-20T22:57:45.806 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000114577 2024-09-20T22:57:45.806 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.806 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.806 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 1,3", 2024-09-20T22:57:45.806 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.241752+0000", 2024-09-20T22:57:45.806 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.012135920999999999 2024-09-20T22:57:45.806 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.806 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.806 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:57:45.806 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.249601+0000", 2024-09-20T22:57:45.806 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0078493120000000006 2024-09-20T22:57:45.807 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.807 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.807 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:57:45.807 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.260731+0000", 2024-09-20T22:57:45.807 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.011129781 2024-09-20T22:57:45.807 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.807 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.807 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:57:45.807 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.260861+0000", 2024-09-20T22:57:45.807 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00013027899999999999 2024-09-20T22:57:45.807 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.807 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.807 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:57:45.807 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.260913+0000", 2024-09-20T22:57:45.808 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.2172999999999997e-05 2024-09-20T22:57:45.808 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.808 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.808 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:57:45.808 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.260978+0000", 2024-09-20T22:57:45.808 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.4182000000000003e-05 2024-09-20T22:57:45.808 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:57:45.808 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:57:45.808 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:57:45.808 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.808 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.808 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:4445 7.0 7:05bb442d:::smithi08477203-24:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e447)", 2024-09-20T22:57:45.809 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:48:47.227478+0000", 2024-09-20T22:57:45.809 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 538.57048401300005, 2024-09-20T22:57:45.809 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.019465758999999999, 2024-09-20T22:57:45.809 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:57:45.809 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:57:45.809 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:57:45.809 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:57:45.809 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:57:45.809 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:57:45.809 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 4445 2024-09-20T22:57:45.809 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.809 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:57:45.809 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.809 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:57:45.809 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.227478+0000", 2024-09-20T22:57:45.810 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:57:45.810 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.810 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.810 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:57:45.810 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.227478+0000", 2024-09-20T22:57:45.810 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:57:45.810 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.810 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.810 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:57:45.810 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.227478+0000", 2024-09-20T22:57:45.810 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.2200000000000004e-07 2024-09-20T22:57:45.810 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.811 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.811 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:57:45.811 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.227481+0000", 2024-09-20T22:57:45.811 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.0060000000000001e-06 2024-09-20T22:57:45.811 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.811 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.811 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:57:45.811 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.227500+0000", 2024-09-20T22:57:45.811 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9071000000000001e-05 2024-09-20T22:57:45.811 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.811 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.811 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:57:45.811 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.227507+0000", 2024-09-20T22:57:45.811 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.1779999999999997e-06 2024-09-20T22:57:45.811 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.812 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.812 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:57:45.812 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.227628+0000", 2024-09-20T22:57:45.812 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000120627 2024-09-20T22:57:45.812 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.812 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.812 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting_for_map not empty", 2024-09-20T22:57:45.812 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.227633+0000", 2024-09-20T22:57:45.812 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.4730000000000002e-06 2024-09-20T22:57:45.812 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.812 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.812 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:57:45.812 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.229766+0000", 2024-09-20T22:57:45.812 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0021332339999999999 2024-09-20T22:57:45.812 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.813 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.813 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for blocked object", 2024-09-20T22:57:45.813 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.229814+0000", 2024-09-20T22:57:45.813 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.7746e-05 2024-09-20T22:57:45.813 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.813 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.813 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:57:45.813 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.246707+0000", 2024-09-20T22:57:45.813 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.016893542000000001 2024-09-20T22:57:45.813 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.813 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.813 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:57:45.813 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.246863+0000", 2024-09-20T22:57:45.813 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000156105 2024-09-20T22:57:45.814 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.814 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.814 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:57:45.814 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:48:47.246943+0000", 2024-09-20T22:57:45.814 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.0154999999999994e-05 2024-09-20T22:57:45.814 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:57:45.814 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:57:45.814 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:57:45.814 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.814 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.814 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:6487 7.b 7:d3ca3619:::smithi08477203-39:head [copy-from ver 1231] snapc 121=[121,120,11f,11e,11d,11b,119,118,116,115,114,112,110,10e,10c,107,105,103,fd,fb,fa,f8,f5,ed,d7,c2,bf] ondisk+write+known_if_redirected+supports_pool_eio e653)", 2024-09-20T22:57:45.814 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:52:13.479898+0000", 2024-09-20T22:57:45.814 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 332.31806343599999, 2024-09-20T22:57:45.814 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.023339255, 2024-09-20T22:57:45.815 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:57:45.815 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:57:45.815 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:57:45.815 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:57:45.816 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:57:45.816 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:57:45.816 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 6487 2024-09-20T22:57:45.816 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.816 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:57:45.816 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.816 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:57:45.816 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:13.479898+0000", 2024-09-20T22:57:45.816 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:57:45.816 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.816 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.816 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:57:45.816 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:13.479898+0000", 2024-09-20T22:57:45.816 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:57:45.816 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.817 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.817 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:57:45.817 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:13.479900+0000", 2024-09-20T22:57:45.817 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.415e-06 2024-09-20T22:57:45.817 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.817 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.817 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:57:45.817 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:13.479907+0000", 2024-09-20T22:57:45.817 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.0740000000000004e-06 2024-09-20T22:57:45.817 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.817 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.817 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:57:45.817 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:13.479916+0000", 2024-09-20T22:57:45.818 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.6420000000000002e-06 2024-09-20T22:57:45.818 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.818 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.818 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:57:45.818 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:13.479922+0000", 2024-09-20T22:57:45.818 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.1070000000000001e-06 2024-09-20T22:57:45.818 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.818 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.818 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:57:45.818 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:13.479981+0000", 2024-09-20T22:57:45.818 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.8613000000000003e-05 2024-09-20T22:57:45.818 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.818 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.818 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:57:45.819 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:13.480095+0000", 2024-09-20T22:57:45.819 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00011437799999999999 2024-09-20T22:57:45.819 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.819 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.819 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 2,6", 2024-09-20T22:57:45.819 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:13.496711+0000", 2024-09-20T22:57:45.819 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.016615938 2024-09-20T22:57:45.819 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.819 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.819 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:57:45.819 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:13.502323+0000", 2024-09-20T22:57:45.819 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0056115799999999997 2024-09-20T22:57:45.819 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.819 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.819 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:57:45.820 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:13.502878+0000", 2024-09-20T22:57:45.820 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00055555000000000003 2024-09-20T22:57:45.820 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.820 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.820 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:57:45.820 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:13.503163+0000", 2024-09-20T22:57:45.820 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000284037 2024-09-20T22:57:45.820 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.820 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.820 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:57:45.820 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:13.503201+0000", 2024-09-20T22:57:45.821 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.8569000000000003e-05 2024-09-20T22:57:45.821 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.821 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.821 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:57:45.821 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:13.503237+0000", 2024-09-20T22:57:45.821 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.6352e-05 2024-09-20T22:57:45.821 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:57:45.821 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:57:45.821 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:57:45.821 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.822 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.822 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:6488 7.b 7:d3ca3619:::smithi08477203-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e653)", 2024-09-20T22:57:45.822 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:52:13.479939+0000", 2024-09-20T22:57:45.822 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 332.31802275899997, 2024-09-20T22:57:45.822 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.019688051000000002, 2024-09-20T22:57:45.822 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:57:45.822 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:57:45.822 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:57:45.822 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:57:45.822 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:57:45.822 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:57:45.822 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 6488 2024-09-20T22:57:45.822 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.823 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:57:45.823 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.823 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:57:45.823 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:13.479939+0000", 2024-09-20T22:57:45.823 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:57:45.823 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.823 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.823 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:57:45.823 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:13.479939+0000", 2024-09-20T22:57:45.823 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:57:45.823 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.823 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.823 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:57:45.823 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:13.479940+0000", 2024-09-20T22:57:45.823 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.7199999999999998e-07 2024-09-20T22:57:45.824 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.824 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.824 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:57:45.824 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:13.479942+0000", 2024-09-20T22:57:45.824 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.853e-06 2024-09-20T22:57:45.824 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.824 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.824 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:57:45.824 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:13.479951+0000", 2024-09-20T22:57:45.824 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.1100000000000003e-06 2024-09-20T22:57:45.824 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.824 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.824 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:57:45.824 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:13.479956+0000", 2024-09-20T22:57:45.825 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.4190000000000001e-06 2024-09-20T22:57:45.825 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.825 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.825 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:57:45.825 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:13.480258+0000", 2024-09-20T22:57:45.825 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00030201400000000003 2024-09-20T22:57:45.825 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.825 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.825 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for blocked object", 2024-09-20T22:57:45.825 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:13.480300+0000", 2024-09-20T22:57:45.825 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.1816000000000001e-05 2024-09-20T22:57:45.825 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.825 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.825 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:57:45.826 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:13.499455+0000", 2024-09-20T22:57:45.826 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.019155583 2024-09-20T22:57:45.826 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.826 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.826 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:57:45.826 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:13.499533+0000", 2024-09-20T22:57:45.826 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.7972000000000005e-05 2024-09-20T22:57:45.826 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.826 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.826 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:57:45.826 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:13.499627+0000", 2024-09-20T22:57:45.826 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.3611999999999998e-05 2024-09-20T22:57:45.826 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:57:45.826 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:57:45.827 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:57:45.827 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.827 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.827 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:6770 7.0 7:064f66ab:::smithi08477203-45:head [copy-from ver 1338] snapc 135=[135,134,133,132,131,130,12f,12e,12d,12c,12b,12a,128,127,126,124,122,121,11f,11b,119,112,10e,10c,107,103,fb,fa,f8,ed,d7,c2,bf] ondisk+write+known_if_redirected+supports_pool_eio e687)", 2024-09-20T22:57:45.827 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:52:47.286358+0000", 2024-09-20T22:57:45.827 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 298.51160410699998, 2024-09-20T22:57:45.827 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.024891383, 2024-09-20T22:57:45.827 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:57:45.827 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:57:45.827 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:57:45.827 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:57:45.827 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:57:45.827 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:57:45.827 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 6770 2024-09-20T22:57:45.827 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.828 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:57:45.828 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.828 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:57:45.828 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:47.286358+0000", 2024-09-20T22:57:45.828 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:57:45.828 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.828 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.828 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:57:45.828 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:47.286358+0000", 2024-09-20T22:57:45.828 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:57:45.828 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.828 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.828 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:57:45.829 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:47.286360+0000", 2024-09-20T22:57:45.829 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.1260000000000001e-06 2024-09-20T22:57:45.829 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.829 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.829 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:57:45.829 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:47.286369+0000", 2024-09-20T22:57:45.829 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.431e-06 2024-09-20T22:57:45.829 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.829 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.829 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:57:45.829 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:47.286391+0000", 2024-09-20T22:57:45.829 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.1824e-05 2024-09-20T22:57:45.829 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.829 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.830 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:57:45.830 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:47.286402+0000", 2024-09-20T22:57:45.830 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.0801e-05 2024-09-20T22:57:45.830 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.830 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.830 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:57:45.830 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:47.286608+0000", 2024-09-20T22:57:45.830 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00020635400000000001 2024-09-20T22:57:45.830 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.830 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.830 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op must wait for map", 2024-09-20T22:57:45.830 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:47.286616+0000", 2024-09-20T22:57:45.830 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.7389999999999999e-06 2024-09-20T22:57:45.831 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.831 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.831 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:57:45.831 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:47.289915+0000", 2024-09-20T22:57:45.831 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.003299022 2024-09-20T22:57:45.831 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.831 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.831 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:57:45.831 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:47.290021+0000", 2024-09-20T22:57:45.831 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000106267 2024-09-20T22:57:45.831 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.831 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.831 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 2,3", 2024-09-20T22:57:45.831 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:47.299972+0000", 2024-09-20T22:57:45.832 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0099504399999999996 2024-09-20T22:57:45.832 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.832 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.832 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:57:45.832 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:47.304349+0000", 2024-09-20T22:57:45.832 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0043777549999999997 2024-09-20T22:57:45.832 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.832 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.832 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:57:45.832 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:47.310757+0000", 2024-09-20T22:57:45.832 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0064077650000000002 2024-09-20T22:57:45.832 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.832 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.833 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:57:45.833 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:47.311157+0000", 2024-09-20T22:57:45.833 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00039990000000000002 2024-09-20T22:57:45.833 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.833 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.833 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:57:45.833 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:47.311206+0000", 2024-09-20T22:57:45.833 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.8906000000000002e-05 2024-09-20T22:57:45.833 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.833 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.833 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:57:45.833 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:47.311249+0000", 2024-09-20T22:57:45.833 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.3053000000000003e-05 2024-09-20T22:57:45.834 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:57:45.834 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:57:45.834 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:57:45.834 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.834 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.834 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:6771 7.0 7:064f66ab:::smithi08477203-45:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e687)", 2024-09-20T22:57:45.834 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:52:47.286426+0000", 2024-09-20T22:57:45.834 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 298.51153525299998, 2024-09-20T22:57:45.834 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.017336328000000002, 2024-09-20T22:57:45.834 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:57:45.834 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:57:45.834 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:57:45.835 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:57:45.835 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:57:45.835 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:57:45.835 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 6771 2024-09-20T22:57:45.835 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.835 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:57:45.835 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.835 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:57:45.835 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:47.286426+0000", 2024-09-20T22:57:45.835 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:57:45.835 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.835 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.835 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:57:45.835 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:47.286426+0000", 2024-09-20T22:57:45.836 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:57:45.836 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.836 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.836 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:57:45.836 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:47.286427+0000", 2024-09-20T22:57:45.836 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.2099999999999996e-07 2024-09-20T22:57:45.836 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.836 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.836 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:57:45.836 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:47.286432+0000", 2024-09-20T22:57:45.836 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.1030000000000001e-06 2024-09-20T22:57:45.836 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.836 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.837 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:57:45.837 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:47.286440+0000", 2024-09-20T22:57:45.837 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.8990000000000001e-06 2024-09-20T22:57:45.837 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.837 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.837 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:57:45.837 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:47.286446+0000", 2024-09-20T22:57:45.837 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.8089999999999998e-06 2024-09-20T22:57:45.837 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.837 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.837 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:57:45.837 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:47.286640+0000", 2024-09-20T22:57:45.837 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00019450899999999999 2024-09-20T22:57:45.838 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.838 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.838 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting_for_map not empty", 2024-09-20T22:57:45.838 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:47.286648+0000", 2024-09-20T22:57:45.838 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.8150000000000006e-06 2024-09-20T22:57:45.838 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.838 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.838 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:57:45.838 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:47.290185+0000", 2024-09-20T22:57:45.838 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0035371030000000002 2024-09-20T22:57:45.838 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.838 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.838 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for blocked object", 2024-09-20T22:57:45.839 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:47.290239+0000", 2024-09-20T22:57:45.839 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.3671000000000002e-05 2024-09-20T22:57:45.839 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.839 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.839 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:57:45.839 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:47.303495+0000", 2024-09-20T22:57:45.839 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.013256396 2024-09-20T22:57:45.839 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.839 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.839 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:57:45.839 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:47.303636+0000", 2024-09-20T22:57:45.840 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00014082600000000001 2024-09-20T22:57:45.840 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.840 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.840 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:57:45.840 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:47.303763+0000", 2024-09-20T22:57:45.840 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00012657599999999999 2024-09-20T22:57:45.840 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:57:45.840 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:57:45.840 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:57:45.840 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.840 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.840 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:6790 7.0 7:064f66ab:::smithi08477203-45:head [copy-from ver 1815] snapc 135=[135,134,133,132,131,130,12f,12e,12d,12c,12b,12a,128,127,126,124,122,121,11f,11b,119,112,10e,10c,107,103,fb,fa,f8,ed,d7,c2,bf] ondisk+write+known_if_redirected+supports_pool_eio e687)", 2024-09-20T22:57:45.840 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:52:47.698638+0000", 2024-09-20T22:57:45.840 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 298.099324061, 2024-09-20T22:57:45.841 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.029923735, 2024-09-20T22:57:45.841 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:57:45.841 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:57:45.841 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:57:45.841 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:57:45.841 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:57:45.841 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:57:45.841 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 6790 2024-09-20T22:57:45.841 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.841 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:57:45.841 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.841 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:57:45.841 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:47.698638+0000", 2024-09-20T22:57:45.842 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:57:45.842 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.842 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.842 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:57:45.842 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:47.698638+0000", 2024-09-20T22:57:45.842 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:57:45.842 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.842 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.842 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:57:45.842 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:47.698639+0000", 2024-09-20T22:57:45.842 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.5880000000000001e-06 2024-09-20T22:57:45.842 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.842 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.842 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:57:45.843 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:47.698647+0000", 2024-09-20T22:57:45.843 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.6040000000000004e-06 2024-09-20T22:57:45.843 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.843 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.843 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:57:45.843 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:47.698662+0000", 2024-09-20T22:57:45.843 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4997999999999999e-05 2024-09-20T22:57:45.843 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.843 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.843 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:57:45.843 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:47.698669+0000", 2024-09-20T22:57:45.843 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.0400000000000004e-06 2024-09-20T22:57:45.843 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.843 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.844 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:57:45.844 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:47.698744+0000", 2024-09-20T22:57:45.844 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.5018999999999994e-05 2024-09-20T22:57:45.844 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.844 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.844 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:57:45.844 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:47.698841+0000", 2024-09-20T22:57:45.844 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.7229e-05 2024-09-20T22:57:45.844 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.844 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.844 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 2,3", 2024-09-20T22:57:45.844 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:47.711667+0000", 2024-09-20T22:57:45.844 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.012825731999999999 2024-09-20T22:57:45.844 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.845 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.845 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:57:45.845 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:47.717229+0000", 2024-09-20T22:57:45.845 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0055624200000000002 2024-09-20T22:57:45.845 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.845 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.845 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:57:45.845 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:47.726665+0000", 2024-09-20T22:57:45.845 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0094358689999999995 2024-09-20T22:57:45.845 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.845 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.845 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:57:45.845 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:47.728457+0000", 2024-09-20T22:57:45.845 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0017914789999999999 2024-09-20T22:57:45.845 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.846 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.846 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:57:45.846 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:47.728511+0000", 2024-09-20T22:57:45.846 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.3953000000000003e-05 2024-09-20T22:57:45.846 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.846 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.846 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:57:45.846 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:47.728561+0000", 2024-09-20T22:57:45.846 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.0803999999999997e-05 2024-09-20T22:57:45.846 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:57:45.846 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:57:45.846 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:57:45.846 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.847 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.847 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:6791 7.0 7:064f66ab:::smithi08477203-45:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e687)", 2024-09-20T22:57:45.847 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:52:47.698687+0000", 2024-09-20T22:57:45.847 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 298.09927477100001, 2024-09-20T22:57:45.847 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.017803079999999999, 2024-09-20T22:57:45.847 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:57:45.847 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:57:45.847 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:57:45.847 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:57:45.847 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:57:45.847 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:57:45.847 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 6791 2024-09-20T22:57:45.847 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.847 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:57:45.847 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.848 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:57:45.848 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:47.698687+0000", 2024-09-20T22:57:45.848 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:57:45.848 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.848 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.848 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:57:45.848 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:47.698687+0000", 2024-09-20T22:57:45.848 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:57:45.848 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.848 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.848 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:57:45.848 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:47.698687+0000", 2024-09-20T22:57:45.848 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.2099999999999997e-07 2024-09-20T22:57:45.848 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.849 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.849 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:57:45.849 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:47.698690+0000", 2024-09-20T22:57:45.849 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.067e-06 2024-09-20T22:57:45.849 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.849 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.849 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:57:45.849 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:47.698702+0000", 2024-09-20T22:57:45.849 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1222e-05 2024-09-20T22:57:45.849 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.849 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.849 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:57:45.849 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:47.698708+0000", 2024-09-20T22:57:45.849 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.8479999999999997e-06 2024-09-20T22:57:45.850 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.850 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.850 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:57:45.850 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:47.699009+0000", 2024-09-20T22:57:45.850 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00030181000000000001 2024-09-20T22:57:45.850 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.850 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.850 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for blocked object", 2024-09-20T22:57:45.850 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:47.699075+0000", 2024-09-20T22:57:45.850 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.6021999999999999e-05 2024-09-20T22:57:45.850 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.850 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.850 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:57:45.851 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:47.716226+0000", 2024-09-20T22:57:45.851 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.017150657 2024-09-20T22:57:45.851 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.851 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.851 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:57:45.851 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:47.716352+0000", 2024-09-20T22:57:45.851 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00012609599999999999 2024-09-20T22:57:45.851 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.851 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.851 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:57:45.851 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:47.716490+0000", 2024-09-20T22:57:45.851 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000137837 2024-09-20T22:57:45.851 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:57:45.851 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:57:45.852 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:57:45.852 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.852 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.852 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:6816 7.b 7:dc95255e:::smithi08477203-1:head [copy-from ver 751] snapc 136=[136,135,134,133,132,131,130,12f,12e,12d,12c,12b,12a,128,127,126,124,122,121,11f,11b,119,112,10e,10c,107,103,fb,fa,f8,ed,d7,c2,bf] ondisk+write+known_if_redirected+supports_pool_eio e688)", 2024-09-20T22:57:45.852 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:52:48.282781+0000", 2024-09-20T22:57:45.852 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 297.51518046500001, 2024-09-20T22:57:45.852 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.036288688999999999, 2024-09-20T22:57:45.852 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:57:45.852 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:57:45.852 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:57:45.852 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:57:45.852 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:57:45.852 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:57:45.852 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 6816 2024-09-20T22:57:45.853 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.853 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:57:45.853 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.853 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:57:45.853 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.282781+0000", 2024-09-20T22:57:45.853 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:57:45.853 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.853 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.853 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:57:45.853 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.282781+0000", 2024-09-20T22:57:45.853 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:57:45.853 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.854 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.854 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:57:45.854 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.282783+0000", 2024-09-20T22:57:45.854 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.381e-06 2024-09-20T22:57:45.854 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.854 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.854 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:57:45.854 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.282789+0000", 2024-09-20T22:57:45.854 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.2210000000000002e-06 2024-09-20T22:57:45.854 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.854 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.854 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:57:45.854 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.282800+0000", 2024-09-20T22:57:45.855 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1581e-05 2024-09-20T22:57:45.855 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.855 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.855 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:57:45.855 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.282812+0000", 2024-09-20T22:57:45.855 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1797000000000001e-05 2024-09-20T22:57:45.855 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.855 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.855 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:57:45.855 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.282886+0000", 2024-09-20T22:57:45.855 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.4327999999999997e-05 2024-09-20T22:57:45.855 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.855 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.855 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op must wait for map", 2024-09-20T22:57:45.856 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.282896+0000", 2024-09-20T22:57:45.856 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.7410000000000004e-06 2024-09-20T22:57:45.856 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.856 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.856 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:57:45.856 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.285446+0000", 2024-09-20T22:57:45.856 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0025500319999999998 2024-09-20T22:57:45.856 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.856 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.856 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:57:45.856 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.285598+0000", 2024-09-20T22:57:45.856 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000151302 2024-09-20T22:57:45.856 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.856 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.857 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 2,6", 2024-09-20T22:57:45.857 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.300505+0000", 2024-09-20T22:57:45.857 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.014907204 2024-09-20T22:57:45.857 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.857 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.857 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:57:45.857 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.309549+0000", 2024-09-20T22:57:45.857 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0090438819999999996 2024-09-20T22:57:45.857 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.857 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.857 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:57:45.857 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.318842+0000", 2024-09-20T22:57:45.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:45.858 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:45.859 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:57:45.859 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0092930649999999997 2024-09-20T22:57:45.860 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.860 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.860 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:57:45.860 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.318947+0000", 2024-09-20T22:57:45.860 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00010507500000000001 2024-09-20T22:57:45.860 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.860 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.860 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:57:45.860 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.319009+0000", 2024-09-20T22:57:45.860 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.2607000000000006e-05 2024-09-20T22:57:45.860 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.860 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.860 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:57:45.861 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.319070+0000", 2024-09-20T22:57:45.861 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.0473000000000002e-05 2024-09-20T22:57:45.861 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:57:45.861 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:57:45.861 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:57:45.861 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.861 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.861 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:6817 7.b 7:dc95255e:::smithi08477203-1:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e688)", 2024-09-20T22:57:45.861 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:52:48.282837+0000", 2024-09-20T22:57:45.861 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 297.51512489800001, 2024-09-20T22:57:45.861 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.022790719000000001, 2024-09-20T22:57:45.861 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:57:45.861 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:57:45.861 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:57:45.862 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:57:45.862 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:57:45.862 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:57:45.862 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 6817 2024-09-20T22:57:45.862 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.862 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:57:45.862 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.862 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:57:45.862 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.282837+0000", 2024-09-20T22:57:45.862 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:57:45.862 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.862 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.862 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:57:45.863 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.282837+0000", 2024-09-20T22:57:45.863 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:57:45.863 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.863 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.863 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:57:45.863 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.282837+0000", 2024-09-20T22:57:45.863 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.0200000000000002e-07 2024-09-20T22:57:45.863 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.863 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.863 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:57:45.863 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.282840+0000", 2024-09-20T22:57:45.863 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.157e-06 2024-09-20T22:57:45.863 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.863 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.864 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:57:45.864 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.282847+0000", 2024-09-20T22:57:45.864 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.0159999999999997e-06 2024-09-20T22:57:45.864 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.864 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.864 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:57:45.864 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.282852+0000", 2024-09-20T22:57:45.864 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.6250000000000004e-06 2024-09-20T22:57:45.864 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.864 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.864 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:57:45.864 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.282955+0000", 2024-09-20T22:57:45.864 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000103056 2024-09-20T22:57:45.864 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.865 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.865 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting_for_map not empty", 2024-09-20T22:57:45.865 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.282963+0000", 2024-09-20T22:57:45.865 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.1650000000000006e-06 2024-09-20T22:57:45.865 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.865 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.865 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:57:45.865 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.285785+0000", 2024-09-20T22:57:45.865 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0028221259999999999 2024-09-20T22:57:45.865 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.865 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.865 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for blocked object", 2024-09-20T22:57:45.865 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.285873+0000", 2024-09-20T22:57:45.865 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.7231999999999994e-05 2024-09-20T22:57:45.865 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.866 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.866 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:57:45.866 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.305344+0000", 2024-09-20T22:57:45.866 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.019471366 2024-09-20T22:57:45.866 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.866 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.866 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:57:45.866 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.305484+0000", 2024-09-20T22:57:45.866 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00013981900000000001 2024-09-20T22:57:45.866 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.866 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.866 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:57:45.866 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.305627+0000", 2024-09-20T22:57:45.867 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00014355499999999999 2024-09-20T22:57:45.867 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:57:45.867 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:57:45.867 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:57:45.867 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.867 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.867 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:6842 7.0 7:0f524219:::smithi08477203-50:head [copy-from ver 1338] snapc 136=[136,135,134,132,131,130,12f,12e,12d,12c,12b,12a,128,127,126,124,122,121,11f,11b,119,112,10e,10c,107,103,fb,fa,f8,ed,d7,c2,bf] ondisk+write+known_if_redirected+supports_pool_eio e689)", 2024-09-20T22:57:45.867 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:52:49.294403+0000", 2024-09-20T22:57:45.867 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 296.50355832899999, 2024-09-20T22:57:45.867 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.020667376000000001, 2024-09-20T22:57:45.867 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:57:45.867 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:57:45.867 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:57:45.868 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:57:45.868 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:57:45.868 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:57:45.868 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 6842 2024-09-20T22:57:45.868 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.868 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:57:45.868 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.868 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:57:45.868 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:49.294403+0000", 2024-09-20T22:57:45.868 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:57:45.868 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.868 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.868 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:57:45.869 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:49.294403+0000", 2024-09-20T22:57:45.869 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:57:45.869 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.869 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.869 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:57:45.869 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:49.294406+0000", 2024-09-20T22:57:45.869 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.7199999999999998e-06 2024-09-20T22:57:45.869 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.869 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.869 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:57:45.869 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:49.294416+0000", 2024-09-20T22:57:45.869 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.7219999999999994e-06 2024-09-20T22:57:45.869 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.869 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.870 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:57:45.870 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:49.294437+0000", 2024-09-20T22:57:45.870 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.1514000000000001e-05 2024-09-20T22:57:45.870 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.870 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.870 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:57:45.870 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:49.294447+0000", 2024-09-20T22:57:45.870 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.6860000000000001e-06 2024-09-20T22:57:45.870 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.870 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.870 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:57:45.871 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:49.294546+0000", 2024-09-20T22:57:45.871 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.9258000000000004e-05 2024-09-20T22:57:45.871 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.871 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.871 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op must wait for map", 2024-09-20T22:57:45.871 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:49.294555+0000", 2024-09-20T22:57:45.871 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.7269999999999996e-06 2024-09-20T22:57:45.871 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.871 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.871 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:57:45.871 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:49.297887+0000", 2024-09-20T22:57:45.871 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0033321370000000002 2024-09-20T22:57:45.871 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.871 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.871 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:57:45.872 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:49.298017+0000", 2024-09-20T22:57:45.872 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00013003099999999999 2024-09-20T22:57:45.872 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.872 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.872 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 2,3", 2024-09-20T22:57:45.872 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:49.301096+0000", 2024-09-20T22:57:45.872 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0030791289999999999 2024-09-20T22:57:45.872 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.872 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.872 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:57:45.872 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:49.305218+0000", 2024-09-20T22:57:45.872 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0041216530000000003 2024-09-20T22:57:45.872 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.873 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.873 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:57:45.873 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:49.308776+0000", 2024-09-20T22:57:45.873 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0035582809999999999 2024-09-20T22:57:45.873 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.873 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.873 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:57:45.873 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:49.314968+0000", 2024-09-20T22:57:45.873 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.006191378 2024-09-20T22:57:45.873 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.873 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.873 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:57:45.873 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:49.315022+0000", 2024-09-20T22:57:45.873 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.4185999999999999e-05 2024-09-20T22:57:45.873 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.874 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.874 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:57:45.874 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:49.315071+0000", 2024-09-20T22:57:45.874 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.8953999999999999e-05 2024-09-20T22:57:45.874 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:57:45.874 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:57:45.874 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:57:45.874 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.874 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.874 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:7143 7.0 7:0f524219:::smithi08477203-50:head [copy-from ver 1826] snapc 145=[145,144,142,141,140,13f,13e,13d,13c,13b,13a,139,138,136,135,132,131,130,12f,12e,12d,12c,12a,128,127,124,122,121,11f,11b,112,103,fa,ed] ondisk+write+known_if_redirected+supports_pool_eio e719)", 2024-09-20T22:57:45.874 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:53:19.295040+0000", 2024-09-20T22:57:45.874 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 266.50292187399998, 2024-09-20T22:57:45.874 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.022201044999999999, 2024-09-20T22:57:45.875 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:57:45.875 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:57:45.875 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:57:45.875 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:57:45.875 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:57:45.875 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:57:45.875 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 7143 2024-09-20T22:57:45.875 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.875 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:57:45.875 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.875 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:57:45.875 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:19.295040+0000", 2024-09-20T22:57:45.875 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:57:45.875 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.875 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.876 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:57:45.876 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:19.295040+0000", 2024-09-20T22:57:45.876 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:57:45.876 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.876 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.876 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:57:45.876 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:19.295041+0000", 2024-09-20T22:57:45.876 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2950000000000001e-06 2024-09-20T22:57:45.876 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.876 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.876 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:57:45.876 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:19.295044+0000", 2024-09-20T22:57:45.876 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.433e-06 2024-09-20T22:57:45.876 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.877 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.877 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:57:45.877 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:19.295056+0000", 2024-09-20T22:57:45.877 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1317e-05 2024-09-20T22:57:45.877 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.877 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.877 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:57:45.877 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:19.295059+0000", 2024-09-20T22:57:45.877 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.6100000000000002e-06 2024-09-20T22:57:45.877 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.877 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.877 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:57:45.877 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:19.295124+0000", 2024-09-20T22:57:45.877 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.4338999999999995e-05 2024-09-20T22:57:45.878 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.878 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.878 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:57:45.878 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:19.295239+0000", 2024-09-20T22:57:45.878 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00011550899999999999 2024-09-20T22:57:45.878 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.878 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.878 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 2,3", 2024-09-20T22:57:45.878 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:19.300456+0000", 2024-09-20T22:57:45.878 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.005217078 2024-09-20T22:57:45.878 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.878 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.878 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:57:45.878 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:19.308370+0000", 2024-09-20T22:57:45.879 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0079131719999999996 2024-09-20T22:57:45.879 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.879 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.879 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:57:45.879 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:19.316928+0000", 2024-09-20T22:57:45.879 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0085582820000000004 2024-09-20T22:57:45.879 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.879 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.879 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:57:45.879 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:19.317133+0000", 2024-09-20T22:57:45.879 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000205064 2024-09-20T22:57:45.879 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.879 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.879 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:57:45.879 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:19.317188+0000", 2024-09-20T22:57:45.880 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.5492000000000003e-05 2024-09-20T22:57:45.880 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.880 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.880 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:57:45.880 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:19.317241+0000", 2024-09-20T22:57:45.880 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.2454000000000003e-05 2024-09-20T22:57:45.880 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:57:45.880 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:57:45.880 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:57:45.880 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.880 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.880 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:7260 7.0 7:064f66ab:::smithi08477203-45:head [write 440614~641741] snapc 149=[149,148,147,146,145,144,142,141,140,13e,13d,13c,13b,13a,139,138,136,135,132,130,12f,12e,12d,12c,12a,128,127,124,121,11f,112,103,ed] ondisk+write+known_if_redirected+supports_pool_eio e728)", 2024-09-20T22:57:45.880 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:53:28.321973+0000", 2024-09-20T22:57:45.880 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 257.47598891799998, 2024-09-20T22:57:45.881 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.020697989, 2024-09-20T22:57:45.881 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:57:45.881 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:57:45.881 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:57:45.881 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:57:45.881 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:57:45.881 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:57:45.881 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 7260 2024-09-20T22:57:45.881 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.881 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:57:45.881 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.881 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:57:45.881 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:28.321973+0000", 2024-09-20T22:57:45.881 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:57:45.882 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.882 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.882 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:57:45.882 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:28.321973+0000", 2024-09-20T22:57:45.882 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:57:45.882 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.882 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.882 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:57:45.883 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:28.321974+0000", 2024-09-20T22:57:45.883 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2699999999999999e-06 2024-09-20T22:57:45.883 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.883 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.883 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:57:45.883 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:28.322726+0000", 2024-09-20T22:57:45.883 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00075168700000000001 2024-09-20T22:57:45.883 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.883 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.883 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:57:45.883 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:28.322740+0000", 2024-09-20T22:57:45.883 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.453e-05 2024-09-20T22:57:45.883 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.884 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.884 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:57:45.884 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:28.322748+0000", 2024-09-20T22:57:45.884 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.6790000000000005e-06 2024-09-20T22:57:45.884 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.884 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.884 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:57:45.884 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:28.322810+0000", 2024-09-20T22:57:45.884 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.1878000000000006e-05 2024-09-20T22:57:45.884 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.884 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.884 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op must wait for map", 2024-09-20T22:57:45.884 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:28.322823+0000", 2024-09-20T22:57:45.884 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2841e-05 2024-09-20T22:57:45.884 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.885 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.885 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:57:45.885 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:28.327245+0000", 2024-09-20T22:57:45.885 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0044223400000000003 2024-09-20T22:57:45.885 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.885 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.885 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:57:45.885 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:28.327380+0000", 2024-09-20T22:57:45.885 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00013474599999999999 2024-09-20T22:57:45.885 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.885 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.885 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 2,3", 2024-09-20T22:57:45.885 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:28.327639+0000", 2024-09-20T22:57:45.885 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00025918600000000001 2024-09-20T22:57:45.886 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.886 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.886 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:57:45.886 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:28.329313+0000", 2024-09-20T22:57:45.886 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0016739750000000001 2024-09-20T22:57:45.886 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.886 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.886 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:57:45.886 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:28.340810+0000", 2024-09-20T22:57:45.886 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.011496719000000001 2024-09-20T22:57:45.886 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.886 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.886 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:57:45.886 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:28.342500+0000", 2024-09-20T22:57:45.887 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001689938 2024-09-20T22:57:45.887 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.887 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.887 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:57:45.887 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:28.342592+0000", 2024-09-20T22:57:45.887 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.2289999999999999e-05 2024-09-20T22:57:45.887 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.887 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.887 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:57:45.887 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:28.342671+0000", 2024-09-20T22:57:45.887 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.8910000000000002e-05 2024-09-20T22:57:45.887 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:57:45.887 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:57:45.887 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:57:45.888 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.888 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.888 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:7432 7.0 7:064f66ab:::smithi08477203-45:head [write 427175~721527,stat] snapc 14e=[14e,14c,14a,149,148,145,142,141,13e,13d,13c,13b,13a,139,132,130,12f,12e,12c,12a,128,127,124,121,11f,112,103] ondisk+write+known_if_redirected+supports_pool_eio e744)", 2024-09-20T22:57:45.888 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:53:44.426844+0000", 2024-09-20T22:57:45.888 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 241.37111763999999, 2024-09-20T22:57:45.888 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.051040752000000002, 2024-09-20T22:57:45.888 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:57:45.888 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:57:45.888 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:57:45.888 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:57:45.888 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:57:45.888 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:57:45.888 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 7432 2024-09-20T22:57:45.888 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.889 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:57:45.889 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.889 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:57:45.889 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:44.426844+0000", 2024-09-20T22:57:45.889 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:57:45.889 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.889 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.889 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:57:45.889 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:44.426844+0000", 2024-09-20T22:57:45.889 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:57:45.889 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.889 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.889 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:57:45.889 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:44.426846+0000", 2024-09-20T22:57:45.890 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.5379999999999999e-06 2024-09-20T22:57:45.890 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.890 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.890 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:57:45.890 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:44.427622+0000", 2024-09-20T22:57:45.890 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00077623299999999998 2024-09-20T22:57:45.890 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.890 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.890 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:57:45.890 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:44.427636+0000", 2024-09-20T22:57:45.890 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4606999999999999e-05 2024-09-20T22:57:45.890 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.891 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.891 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:57:45.891 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:44.427645+0000", 2024-09-20T22:57:45.891 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.6789999999999994e-06 2024-09-20T22:57:45.891 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.891 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.891 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:57:45.891 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:44.427707+0000", 2024-09-20T22:57:45.891 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.2383000000000002e-05 2024-09-20T22:57:45.891 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.891 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.891 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:57:45.892 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:44.427805+0000", 2024-09-20T22:57:45.892 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.7084999999999996e-05 2024-09-20T22:57:45.892 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.892 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.892 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 2,3", 2024-09-20T22:57:45.892 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:44.427962+0000", 2024-09-20T22:57:45.892 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00015762400000000001 2024-09-20T22:57:45.892 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.892 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.892 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:57:45.892 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:44.429626+0000", 2024-09-20T22:57:45.892 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00166421 2024-09-20T22:57:45.893 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.893 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.893 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:57:45.893 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:44.430495+0000", 2024-09-20T22:57:45.893 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00086868299999999998 2024-09-20T22:57:45.893 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.893 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.893 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:57:45.893 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:44.477712+0000", 2024-09-20T22:57:45.893 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.047216739000000001 2024-09-20T22:57:45.893 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.893 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.894 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:57:45.894 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:44.477805+0000", 2024-09-20T22:57:45.894 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.3424000000000005e-05 2024-09-20T22:57:45.894 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.894 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.894 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:57:45.894 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:44.477885+0000", 2024-09-20T22:57:45.894 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.9547000000000003e-05 2024-09-20T22:57:45.894 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:57:45.894 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:57:45.894 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:57:45.894 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.894 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.895 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:7433 7.0 7:064f66ab:::smithi08477203-45:head [write 1930749~735360] snapc 14e=[14e,14c,14a,149,148,145,142,141,13e,13d,13c,13b,13a,139,132,130,12f,12e,12c,12a,128,127,124,121,11f,112,103] ondisk+write+known_if_redirected+supports_pool_eio e744)", 2024-09-20T22:57:45.895 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:53:44.435104+0000", 2024-09-20T22:57:45.895 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 241.362857625, 2024-09-20T22:57:45.895 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.043021579999999997, 2024-09-20T22:57:45.895 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:57:45.895 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:57:45.895 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:57:45.895 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:57:45.895 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:57:45.895 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:57:45.895 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 7433 2024-09-20T22:57:45.895 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.895 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:57:45.896 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.896 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:57:45.896 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:44.435104+0000", 2024-09-20T22:57:45.896 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:57:45.896 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.896 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.896 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:57:45.896 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:44.435104+0000", 2024-09-20T22:57:45.896 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:57:45.896 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.896 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.896 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:57:45.896 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:44.435106+0000", 2024-09-20T22:57:45.897 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.5549999999999999e-06 2024-09-20T22:57:45.897 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.897 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.897 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:57:45.897 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:44.435886+0000", 2024-09-20T22:57:45.897 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00078014899999999995 2024-09-20T22:57:45.897 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.897 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.897 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:57:45.897 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:44.435900+0000", 2024-09-20T22:57:45.897 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4576000000000001e-05 2024-09-20T22:57:45.897 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.897 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.898 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:57:45.898 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:44.435908+0000", 2024-09-20T22:57:45.898 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.1380000000000007e-06 2024-09-20T22:57:45.898 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.898 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.898 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:57:45.898 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:44.435961+0000", 2024-09-20T22:57:45.898 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.2561e-05 2024-09-20T22:57:45.898 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.898 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.898 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:57:45.899 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:44.436043+0000", 2024-09-20T22:57:45.899 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.2213999999999996e-05 2024-09-20T22:57:45.899 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.899 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.899 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 2,3", 2024-09-20T22:57:45.899 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:44.436179+0000", 2024-09-20T22:57:45.899 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000135776 2024-09-20T22:57:45.899 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.899 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.899 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:57:45.899 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:44.438389+0000", 2024-09-20T22:57:45.899 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.002210203 2024-09-20T22:57:45.900 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.900 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.900 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:57:45.900 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:44.438925+0000", 2024-09-20T22:57:45.900 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00053610599999999995 2024-09-20T22:57:45.900 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.900 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.900 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:57:45.900 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:44.477956+0000", 2024-09-20T22:57:45.900 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.039030861 2024-09-20T22:57:45.900 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.900 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.901 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:57:45.901 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:44.478047+0000", 2024-09-20T22:57:45.901 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.0612999999999995e-05 2024-09-20T22:57:45.901 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.901 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.901 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:57:45.901 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:44.478126+0000", 2024-09-20T22:57:45.901 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.8827999999999998e-05 2024-09-20T22:57:45.901 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:57:45.901 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:57:45.901 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:57:45.902 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.902 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.902 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:7434 7.0 7:064f66ab:::smithi08477203-45:head [setxattr _header (58) in=7b,truncate 3227175] snapc 14e=[14e,14c,14a,149,148,145,142,141,13e,13d,13c,13b,13a,139,132,130,12f,12e,12c,12a,128,127,124,121,11f,112,103] ondisk+write+known_if_redirected+supports_pool_eio e744)", 2024-09-20T22:57:45.902 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:53:44.435927+0000", 2024-09-20T22:57:45.902 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 241.36203467600001, 2024-09-20T22:57:45.902 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.042348438000000002, 2024-09-20T22:57:45.902 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:57:45.902 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:57:45.902 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:57:45.902 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:57:45.902 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:57:45.903 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:57:45.903 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 7434 2024-09-20T22:57:45.903 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.903 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:57:45.903 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.903 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:57:45.903 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:44.435927+0000", 2024-09-20T22:57:45.903 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:57:45.903 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.903 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.903 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:57:45.903 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:44.435927+0000", 2024-09-20T22:57:45.904 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:57:45.904 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.904 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.904 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:57:45.904 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:44.435928+0000", 2024-09-20T22:57:45.904 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.5000000000000001e-07 2024-09-20T22:57:45.904 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.904 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.904 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:57:45.904 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:44.435933+0000", 2024-09-20T22:57:45.904 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.8790000000000002e-06 2024-09-20T22:57:45.904 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.905 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.905 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:57:45.905 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:44.435950+0000", 2024-09-20T22:57:45.905 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.7087000000000001e-05 2024-09-20T22:57:45.905 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.905 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.905 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:57:45.905 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:44.435957+0000", 2024-09-20T22:57:45.905 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.8399999999999997e-06 2024-09-20T22:57:45.905 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.905 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.905 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:57:45.905 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:44.437409+0000", 2024-09-20T22:57:45.906 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0014522350000000001 2024-09-20T22:57:45.906 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.906 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.906 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:57:45.906 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:44.437551+0000", 2024-09-20T22:57:45.906 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00014222700000000001 2024-09-20T22:57:45.906 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.906 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.906 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 2,3", 2024-09-20T22:57:45.906 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:44.437794+0000", 2024-09-20T22:57:45.906 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00024253499999999999 2024-09-20T22:57:45.906 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.906 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.907 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:57:45.907 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:44.438399+0000", 2024-09-20T22:57:45.907 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00060537799999999995 2024-09-20T22:57:45.907 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.907 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.907 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:57:45.907 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:44.439025+0000", 2024-09-20T22:57:45.907 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00062642000000000004 2024-09-20T22:57:45.907 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.907 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.907 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:57:45.907 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:44.478167+0000", 2024-09-20T22:57:45.908 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.039141148000000001 2024-09-20T22:57:45.908 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.908 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.908 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:57:45.908 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:44.478217+0000", 2024-09-20T22:57:45.908 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.0182000000000002e-05 2024-09-20T22:57:45.908 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.908 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.908 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:57:45.908 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:44.478275+0000", 2024-09-20T22:57:45.908 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.8656999999999998e-05 2024-09-20T22:57:45.908 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:57:45.908 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:57:45.908 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:57:45.909 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.909 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.909 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:7435 7.0 7:064f66ab:::smithi08477203-45:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e744)", 2024-09-20T22:57:45.909 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:53:44.435969+0000", 2024-09-20T22:57:45.909 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 241.36199295599999, 2024-09-20T22:57:45.909 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.042610406000000003, 2024-09-20T22:57:45.909 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:57:45.909 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:57:45.909 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:57:45.909 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:57:45.909 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:57:45.909 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:57:45.909 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 7435 2024-09-20T22:57:45.909 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.909 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:57:45.910 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.910 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:57:45.910 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:44.435969+0000", 2024-09-20T22:57:45.910 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:57:45.910 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.910 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.910 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:57:45.910 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:44.435969+0000", 2024-09-20T22:57:45.910 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:57:45.910 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.910 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.910 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:57:45.910 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:44.435969+0000", 2024-09-20T22:57:45.910 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.2799999999999996e-07 2024-09-20T22:57:45.911 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.911 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.911 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:57:45.911 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:44.435972+0000", 2024-09-20T22:57:45.911 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.0979999999999998e-06 2024-09-20T22:57:45.911 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.911 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.911 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:57:45.911 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:44.435981+0000", 2024-09-20T22:57:45.911 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.1300000000000001e-06 2024-09-20T22:57:45.911 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.911 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.911 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:57:45.912 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:44.435986+0000", 2024-09-20T22:57:45.912 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.9619999999999998e-06 2024-09-20T22:57:45.912 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.912 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.912 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:57:45.912 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:44.438239+0000", 2024-09-20T22:57:45.912 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0022527609999999998 2024-09-20T22:57:45.912 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.912 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.912 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for rw locks", 2024-09-20T22:57:45.912 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:44.438361+0000", 2024-09-20T22:57:45.912 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000121589 2024-09-20T22:57:45.912 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.912 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.913 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:57:45.913 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:44.478322+0000", 2024-09-20T22:57:45.913 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.03996077 2024-09-20T22:57:45.913 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.913 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.913 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:57:45.913 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:44.478438+0000", 2024-09-20T22:57:45.913 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00011638599999999999 2024-09-20T22:57:45.913 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.913 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.913 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:57:45.913 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:44.478579+0000", 2024-09-20T22:57:45.913 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000141082 2024-09-20T22:57:45.913 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:57:45.913 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:57:45.914 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:57:45.914 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.914 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.914 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:9020 7.a e905/94 7:5122292a:::smithi08477203-33:head v 905'1157, mlcod=901'1153)", 2024-09-20T22:57:45.914 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:56:25.655795+0000", 2024-09-20T22:57:45.914 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 80.142166908999997, 2024-09-20T22:57:45.914 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0091280860000000005, 2024-09-20T22:57:45.914 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:57:45.914 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:57:45.914 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:57:45.914 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:57:45.914 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.915 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:57:45.915 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:56:25.655795+0000", 2024-09-20T22:57:45.915 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:57:45.915 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.915 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.915 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:57:45.915 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:56:25.655795+0000", 2024-09-20T22:57:45.915 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:57:45.915 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.915 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.915 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:57:45.915 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:56:25.655796+0000", 2024-09-20T22:57:45.915 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.536e-06 2024-09-20T22:57:45.915 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.915 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.916 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:57:45.916 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:56:25.658447+0000", 2024-09-20T22:57:45.916 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0026505629999999999 2024-09-20T22:57:45.916 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.916 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.916 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:57:45.916 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:56:25.658469+0000", 2024-09-20T22:57:45.916 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.2478999999999999e-05 2024-09-20T22:57:45.916 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.916 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.916 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:57:45.916 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:56:25.658481+0000", 2024-09-20T22:57:45.916 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1409e-05 2024-09-20T22:57:45.916 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.917 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.917 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:57:45.917 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:56:25.658559+0000", 2024-09-20T22:57:45.917 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.8047999999999995e-05 2024-09-20T22:57:45.917 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.917 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.917 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:57:45.917 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:56:25.658587+0000", 2024-09-20T22:57:45.917 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.7761999999999999e-05 2024-09-20T22:57:45.917 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.917 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.917 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:57:45.917 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:56:25.664873+0000", 2024-09-20T22:57:45.917 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0062868189999999999 2024-09-20T22:57:45.917 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.918 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.918 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:57:45.918 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:56:25.664923+0000", 2024-09-20T22:57:45.918 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.9469999999999999e-05 2024-09-20T22:57:45.918 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:57:45.918 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:57:45.918 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:57:45.918 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.918 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.918 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(osd.6.48:139 7.0 7:05bb442d:::smithi08477203-24:head [assert-version v1956,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e962)", 2024-09-20T22:57:45.918 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:57:22.540475+0000", 2024-09-20T22:57:45.918 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 23.257486539999999, 2024-09-20T22:57:45.918 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0061172220000000003, 2024-09-20T22:57:45.918 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:57:45.919 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:57:45.919 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:57:45.919 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:57:45.919 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.919 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:57:45.919 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:57:22.540475+0000", 2024-09-20T22:57:45.919 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:57:45.919 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.919 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.919 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:57:45.919 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:57:22.540475+0000", 2024-09-20T22:57:45.919 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:57:45.919 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.919 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.920 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:57:45.920 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:57:22.540477+0000", 2024-09-20T22:57:45.920 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.7400000000000001e-06 2024-09-20T22:57:45.920 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.920 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.920 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:57:45.920 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:57:22.540484+0000", 2024-09-20T22:57:45.920 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.951e-06 2024-09-20T22:57:45.920 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.920 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.920 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:57:45.920 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:57:22.540498+0000", 2024-09-20T22:57:45.920 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4459e-05 2024-09-20T22:57:45.920 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.921 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.921 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:57:45.921 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:57:22.540505+0000", 2024-09-20T22:57:45.921 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.0010000000000004e-06 2024-09-20T22:57:45.921 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.921 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.921 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:57:45.921 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:57:22.540563+0000", 2024-09-20T22:57:45.921 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.8183999999999997e-05 2024-09-20T22:57:45.921 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.921 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.921 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op must wait for map", 2024-09-20T22:57:45.921 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:57:22.540569+0000", 2024-09-20T22:57:45.921 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.891e-06 2024-09-20T22:57:45.922 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.922 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.922 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:57:45.922 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:57:22.543323+0000", 2024-09-20T22:57:45.922 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0027538839999999998 2024-09-20T22:57:45.922 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.922 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.922 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:57:45.922 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:57:22.543435+0000", 2024-09-20T22:57:45.922 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00011181800000000001 2024-09-20T22:57:45.922 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.922 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.922 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:57:45.922 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:57:22.546592+0000", 2024-09-20T22:57:45.923 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0031572940000000002 2024-09-20T22:57:45.923 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:57:45.923 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:57:45.923 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:57:45.923 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.923 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.923 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:9799 7.b e965/779 7:df02c955:::smithi08477203-9:head v 965'1597, mlcod=963'1594)", 2024-09-20T22:57:45.923 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:57:25.599004+0000", 2024-09-20T22:57:45.923 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 20.198957747000001, 2024-09-20T22:57:45.923 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0086609430000000008, 2024-09-20T22:57:45.923 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:57:45.923 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:57:45.923 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:57:45.924 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:57:45.924 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.924 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:57:45.924 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:57:25.599004+0000", 2024-09-20T22:57:45.924 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:57:45.924 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.924 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.924 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:57:45.924 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:57:25.599004+0000", 2024-09-20T22:57:45.924 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:57:45.924 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.924 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.924 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:57:45.924 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:57:25.599004+0000", 2024-09-20T22:57:45.925 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.8599999999999998e-07 2024-09-20T22:57:45.925 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.925 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.925 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:57:45.925 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:57:25.603435+0000", 2024-09-20T22:57:45.925 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0044304849999999996 2024-09-20T22:57:45.925 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.925 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.925 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:57:45.925 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:57:25.603464+0000", 2024-09-20T22:57:45.925 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.8690000000000001e-05 2024-09-20T22:57:45.925 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.925 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.925 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:57:45.926 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:57:25.603478+0000", 2024-09-20T22:57:45.926 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4021e-05 2024-09-20T22:57:45.926 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.926 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.926 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:57:45.926 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:57:25.603564+0000", 2024-09-20T22:57:45.926 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.6667000000000002e-05 2024-09-20T22:57:45.926 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.926 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.926 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:57:45.926 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:57:25.603597+0000", 2024-09-20T22:57:45.926 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.2877999999999999e-05 2024-09-20T22:57:45.926 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.927 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.927 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:57:45.927 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:57:25.607615+0000", 2024-09-20T22:57:45.927 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0040181490000000004 2024-09-20T22:57:45.927 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:45.927 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:45.927 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:57:45.927 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:57:25.607665+0000", 2024-09-20T22:57:45.927 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.9466999999999999e-05 2024-09-20T22:57:45.927 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:57:45.927 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:57:45.927 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:57:45.927 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:57:45.927 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:57:45.927 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-20T22:57:45.928 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:45 smithi084 ceph-mon[29774]: pgmap v1377: 168 pgs: 1 active+clean+remapped, 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+clean+snaptrim, 41 active+undersized, 1 active+remapped+backfill_toofull, 36 active+undersized+degraded, 87 active+clean; 838 MiB data, 2.3 GiB used, 623 GiB / 626 GiB avail; 11 MiB/s rd, 1.5 MiB/s wr, 14 op/s; 306/1725 objects degraded (17.739%); 55/1725 objects misplaced (3.188%) 2024-09-20T22:57:45.928 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:45 smithi084 ceph-mon[29774]: osdmap e984: 8 total, 7 up, 7 in 2024-09-20T22:57:45.928 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:45 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/3517535894' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:57:45.928 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:45 smithi084 ceph-mon[29774]: purged_snaps scrub starts 2024-09-20T22:57:45.928 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:45 smithi084 ceph-mon[29774]: purged_snaps scrub ok 2024-09-20T22:57:45.929 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:45 smithi084 ceph-mon[40024]: pgmap v1377: 168 pgs: 1 active+clean+remapped, 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+clean+snaptrim, 41 active+undersized, 1 active+remapped+backfill_toofull, 36 active+undersized+degraded, 87 active+clean; 838 MiB data, 2.3 GiB used, 623 GiB / 626 GiB avail; 11 MiB/s rd, 1.5 MiB/s wr, 14 op/s; 306/1725 objects degraded (17.739%); 55/1725 objects misplaced (3.188%) 2024-09-20T22:57:45.929 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:45 smithi084 ceph-mon[40024]: osdmap e984: 8 total, 7 up, 7 in 2024-09-20T22:57:45.929 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:45 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/3517535894' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:57:45.929 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:45 smithi084 ceph-mon[40024]: purged_snaps scrub starts 2024-09-20T22:57:45.929 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:45 smithi084 ceph-mon[40024]: purged_snaps scrub ok 2024-09-20T22:57:45.959 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:57:45.981 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:45.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:45.982 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:57:45.983 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:57:46.083 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:57:46.105 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:46.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:46.106 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:57:46.207 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:57:46.228 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:46.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:46.229 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:57:46.330 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:57:46.352 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:46.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:46.353 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:57:46.454 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:57:46.478 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:46.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:46.479 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:57:46.580 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:57:46.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:46.604 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:46.604 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:57:46.666 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is set 2024-09-20T22:57:46.677 INFO:tasks.rados.rados.0.smithi084.stdout:3980: done (0 left) 2024-09-20T22:57:46.677 INFO:tasks.rados.rados.0.smithi084.stdout:3981: read oid 8 snap -1 2024-09-20T22:57:46.677 INFO:tasks.rados.rados.0.smithi084.stdout:3981: expect (ObjNum 1205 snap 412 seq_num 1205) 2024-09-20T22:57:46.678 INFO:tasks.rados.rados.0.smithi084.stdout:3982: copy_from oid 21 from oid 50 current snap is 462 2024-09-20T22:57:46.678 INFO:tasks.rados.rados.0.smithi084.stdout:3983: write (excl) oid 9 current snap is 462 2024-09-20T22:57:46.678 INFO:tasks.rados.rados.0.smithi084.stdout:3983: seq_num 1347 ranges {639029=754884,2173245=465784} 2024-09-20T22:57:46.682 INFO:tasks.rados.rados.0.smithi084.stdout:3983: writing smithi08477203-9 from 639029 to 1393913 tid 1 2024-09-20T22:57:46.690 INFO:tasks.rados.rados.0.smithi084.stdout:3983: writing smithi08477203-9 from 2173245 to 2639029 tid 2 2024-09-20T22:57:46.690 INFO:tasks.rados.rados.0.smithi084.stdout:3984: snap_create 2024-09-20T22:57:46.692 INFO:tasks.rados.rados.0.smithi084.stdout:3981: expect (ObjNum 1205 snap 412 seq_num 1205) 2024-09-20T22:57:46.705 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:57:46.709 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-20T22:57:46.727 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:46.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:46.729 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:57:46.772 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:46 smithi084 ceph-mon[40024]: osdmap e985: 8 total, 7 up, 7 in 2024-09-20T22:57:46.772 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:46 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/673509948' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:57:46.773 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:46 smithi084 ceph-mon[29774]: osdmap e985: 8 total, 7 up, 7 in 2024-09-20T22:57:46.773 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:46 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/673509948' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:57:46.816 INFO:tasks.rados.rados.0.smithi084.stdout:3982: finishing copy_from racing read to smithi08477203-21 2024-09-20T22:57:46.817 INFO:tasks.rados.rados.0.smithi084.stdout:3983: finishing write tid 1 to smithi08477203-9 2024-09-20T22:57:46.817 INFO:tasks.rados.rados.0.smithi084.stdout:3983: finishing write tid 2 to smithi08477203-9 2024-09-20T22:57:46.817 INFO:tasks.rados.rados.0.smithi084.stdout:3983: finishing write tid 3 to smithi08477203-9 2024-09-20T22:57:46.817 INFO:tasks.rados.rados.0.smithi084.stdout:3983: finishing write tid 4 to smithi08477203-9 2024-09-20T22:57:46.817 INFO:tasks.rados.rados.0.smithi084.stdout:3983: oid 9 updating version 0 to 2697 2024-09-20T22:57:46.817 INFO:tasks.rados.rados.0.smithi084.stdout:3983: oid 9 version 2697 is already newer than 2696 2024-09-20T22:57:46.817 INFO:tasks.rados.rados.0.smithi084.stdout:3983: oid 9 version 2697 is already newer than 2695 2024-09-20T22:57:46.817 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 9 v 2697 (ObjNum 1347 snap 462 seq_num 1347) dirty exists 2024-09-20T22:57:46.817 INFO:tasks.rados.rados.0.smithi084.stdout:3983: left oid 9 (ObjNum 1347 snap 462 seq_num 1347) 2024-09-20T22:57:46.817 INFO:tasks.rados.rados.0.smithi084.stdout:3982: finishing copy_from to smithi08477203-21 2024-09-20T22:57:46.817 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 21 v 1907 (ObjNum 1128 snap 381 seq_num 1128) dirty exists 2024-09-20T22:57:46.830 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:57:46.851 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:46.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:46.852 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:57:46.953 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:57:46.957 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:57:46.976 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:46.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:46.977 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:57:47.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:46 smithi154 ceph-mon[33264]: osdmap e985: 8 total, 7 up, 7 in 2024-09-20T22:57:47.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:46 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/673509948' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:57:47.078 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:57:47.099 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:47.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:47.100 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:57:47.114 INFO:journalctl@ceph.osd.3.smithi084.stdout:Sep 20 22:57:46 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-3[213561]: 2024-09-20T22:57:46.769+0000 7f2fcc410640 -1 osd.3 939 log_to_monitors true 2024-09-20T22:57:47.201 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:57:47.222 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:47.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:47.223 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:57:47.324 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:57:47.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:47.349 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:47.350 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:57:47.451 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:57:47.472 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:47.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:47.473 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:57:47.574 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:57:47.575 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-20T22:57:47.576 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [], 2024-09-20T22:57:47.576 INFO:teuthology.orchestra.run.smithi154.stdout: "num_ops": 0 2024-09-20T22:57:47.576 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-20T22:57:47.582 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-20T22:57:47.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:47.595 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:47.596 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:57:47.679 INFO:tasks.rados.rados.0.smithi084.stdout:3981: done (3 left) 2024-09-20T22:57:47.679 INFO:tasks.rados.rados.0.smithi084.stdout:3982: done (2 left) 2024-09-20T22:57:47.679 INFO:tasks.rados.rados.0.smithi084.stdout:3983: done (1 left) 2024-09-20T22:57:47.679 INFO:tasks.rados.rados.0.smithi084.stdout:3984: done (0 left) 2024-09-20T22:57:47.680 INFO:tasks.rados.rados.0.smithi084.stdout:3985: write (excl) oid 9 current snap is 463 2024-09-20T22:57:47.680 INFO:tasks.rados.rados.0.smithi084.stdout:3985: seq_num 1348 ranges {655836=430133,1831727=757138} 2024-09-20T22:57:47.685 INFO:tasks.rados.rados.0.smithi084.stdout:3985: writing smithi08477203-9 from 655836 to 1085969 tid 1 2024-09-20T22:57:47.693 INFO:tasks.rados.rados.0.smithi084.stdout:3985: writing smithi08477203-9 from 1831727 to 2588865 tid 2 2024-09-20T22:57:47.693 INFO:tasks.rados.rados.0.smithi084.stdout:3986: write (excl) oid 30 current snap is 463 2024-09-20T22:57:47.694 INFO:tasks.rados.rados.0.smithi084.stdout:3986: seq_num 1349 ranges {672643=505382,1890209=500796} 2024-09-20T22:57:47.696 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:57:47.701 INFO:tasks.rados.rados.0.smithi084.stdout:3986: writing smithi08477203-30 from 672643 to 1178025 tid 1 2024-09-20T22:57:47.708 INFO:tasks.rados.rados.0.smithi084.stdout:3986: writing smithi08477203-30 from 1890209 to 2391005 tid 2 2024-09-20T22:57:47.709 INFO:tasks.rados.rados.0.smithi084.stdout:3987: rollback oid 21 current snap is 463 2024-09-20T22:57:47.709 INFO:tasks.rados.rados.0.smithi084.stdout:rollback oid 21 to 397 2024-09-20T22:57:47.709 INFO:tasks.rados.rados.0.smithi084.stdout:3988: copy_from oid 1 from oid 11 current snap is 463 2024-09-20T22:57:47.710 INFO:tasks.rados.rados.0.smithi084.stdout:3989: snap_remove snap 408 2024-09-20T22:57:47.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:47.722 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:47.722 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:57:47.823 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:57:47.845 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:47.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:47.846 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:57:47.861 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:57:47.946 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:57:47.967 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:47.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:47.968 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:57:47.983 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:47 smithi084 ceph-mon[40024]: pgmap v1380: 168 pgs: 1 active+clean+remapped, 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+clean+snaptrim, 41 active+undersized, 1 active+remapped+backfill_toofull, 36 active+undersized+degraded, 87 active+clean; 838 MiB data, 2.3 GiB used, 623 GiB / 626 GiB avail; 11 MiB/s rd, 1.5 MiB/s wr, 14 op/s; 306/1725 objects degraded (17.739%); 55/1725 objects misplaced (3.188%) 2024-09-20T22:57:47.983 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:47 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/673509948' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T22:57:47.983 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:47 smithi084 ceph-mon[40024]: osdmap e986: 8 total, 7 up, 7 in 2024-09-20T22:57:47.983 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:47 smithi084 ceph-mon[40024]: from='osd.3 v2:172.21.15.84:6813/3784535689' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-20T22:57:47.983 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:47 smithi084 ceph-mon[40024]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-20T22:57:47.984 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:47 smithi084 ceph-mon[29774]: pgmap v1380: 168 pgs: 1 active+clean+remapped, 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+clean+snaptrim, 41 active+undersized, 1 active+remapped+backfill_toofull, 36 active+undersized+degraded, 87 active+clean; 838 MiB data, 2.3 GiB used, 623 GiB / 626 GiB avail; 11 MiB/s rd, 1.5 MiB/s wr, 14 op/s; 306/1725 objects degraded (17.739%); 55/1725 objects misplaced (3.188%) 2024-09-20T22:57:47.984 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:47 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/673509948' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T22:57:47.984 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:47 smithi084 ceph-mon[29774]: osdmap e986: 8 total, 7 up, 7 in 2024-09-20T22:57:47.984 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:47 smithi084 ceph-mon[29774]: from='osd.3 v2:172.21.15.84:6813/3784535689' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-20T22:57:47.984 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:47 smithi084 ceph-mon[29774]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-20T22:57:48.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:47 smithi154 ceph-mon[33264]: pgmap v1380: 168 pgs: 1 active+clean+remapped, 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+clean+snaptrim, 41 active+undersized, 1 active+remapped+backfill_toofull, 36 active+undersized+degraded, 87 active+clean; 838 MiB data, 2.3 GiB used, 623 GiB / 626 GiB avail; 11 MiB/s rd, 1.5 MiB/s wr, 14 op/s; 306/1725 objects degraded (17.739%); 55/1725 objects misplaced (3.188%) 2024-09-20T22:57:48.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:47 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/673509948' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T22:57:48.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:47 smithi154 ceph-mon[33264]: osdmap e986: 8 total, 7 up, 7 in 2024-09-20T22:57:48.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:47 smithi154 ceph-mon[33264]: from='osd.3 v2:172.21.15.84:6813/3784535689' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-20T22:57:48.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:47 smithi154 ceph-mon[33264]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-20T22:57:48.069 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:57:48.090 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:48.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:48.091 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:57:48.192 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:57:48.212 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:48.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:48.213 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:57:48.251 INFO:journalctl@ceph.osd.3.smithi084.stdout:Sep 20 22:57:48 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-3[213561]: 2024-09-20T22:57:48.190+0000 7f2fc01b3640 -1 osd.3 987 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-20T22:57:48.310 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.6 dump_blocked_ops 2024-09-20T22:57:48.314 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:57:48.372 INFO:tasks.rados.rados.0.smithi084.stdout:3989: done (4 left) 2024-09-20T22:57:48.373 INFO:tasks.rados.rados.0.smithi084.stdout:3990: delete oid 39 current snap is 463 2024-09-20T22:57:48.373 INFO:tasks.rados.rados.0.smithi084.stdout:3988: finishing copy_from racing read to smithi08477203-1 2024-09-20T22:57:48.373 INFO:tasks.rados.rados.0.smithi084.stdout:3985: finishing write tid 1 to smithi08477203-9 2024-09-20T22:57:48.373 INFO:tasks.rados.rados.0.smithi084.stdout:3986: finishing write tid 1 to smithi08477203-30 2024-09-20T22:57:48.373 INFO:tasks.rados.rados.0.smithi084.stdout:3986: finishing write tid 2 to smithi08477203-30 2024-09-20T22:57:48.373 INFO:tasks.rados.rados.0.smithi084.stdout:3986: finishing write tid 3 to smithi08477203-30 2024-09-20T22:57:48.373 INFO:tasks.rados.rados.0.smithi084.stdout:3986: finishing write tid 4 to smithi08477203-30 2024-09-20T22:57:48.373 INFO:tasks.rados.rados.0.smithi084.stdout:3986: oid 30 updating version 0 to 2652 2024-09-20T22:57:48.373 INFO:tasks.rados.rados.0.smithi084.stdout:3986: oid 30 updating version 2652 to 2653 2024-09-20T22:57:48.373 INFO:tasks.rados.rados.0.smithi084.stdout:3986: oid 30 updating version 2653 to 2654 2024-09-20T22:57:48.374 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 30 v 2654 (ObjNum 1349 snap 463 seq_num 1349) dirty exists 2024-09-20T22:57:48.374 INFO:tasks.rados.rados.0.smithi084.stdout:3986: left oid 30 (ObjNum 1349 snap 463 seq_num 1349) 2024-09-20T22:57:48.374 INFO:tasks.rados.rados.0.smithi084.stdout:3985: finishing write tid 2 to smithi08477203-9 2024-09-20T22:57:48.374 INFO:tasks.rados.rados.0.smithi084.stdout:3985: finishing write tid 3 to smithi08477203-9 2024-09-20T22:57:48.374 INFO:tasks.rados.rados.0.smithi084.stdout:3987: finishing rollback tid 0 to smithi08477203-21 2024-09-20T22:57:48.374 INFO:tasks.rados.rados.0.smithi084.stdout:3985: finishing write tid 4 to smithi08477203-9 2024-09-20T22:57:48.374 INFO:tasks.rados.rados.0.smithi084.stdout:3985: oid 9 updating version 0 to 2700 2024-09-20T22:57:48.374 INFO:tasks.rados.rados.0.smithi084.stdout:3985: oid 9 version 2700 is already newer than 2699 2024-09-20T22:57:48.374 INFO:tasks.rados.rados.0.smithi084.stdout:3985: oid 9 version 2700 is already newer than 2698 2024-09-20T22:57:48.374 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 9 v 2700 (ObjNum 1348 snap 463 seq_num 1348) dirty exists 2024-09-20T22:57:48.374 INFO:tasks.rados.rados.0.smithi084.stdout:3985: left oid 9 (ObjNum 1348 snap 463 seq_num 1348) 2024-09-20T22:57:48.375 INFO:tasks.rados.rados.0.smithi084.stdout:3987: finishing rollback tid 1 to smithi08477203-21 2024-09-20T22:57:48.375 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 21 v 2691 (ObjNum 33 snap 0 seq_num 2091720448) dirty dne 2024-09-20T22:57:48.375 INFO:tasks.rados.rados.0.smithi084.stdout:3988: finishing copy_from to smithi08477203-1 2024-09-20T22:57:48.375 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 1 v 2672 (ObjNum 1332 snap 456 seq_num 1332) dirty exists 2024-09-20T22:57:48.375 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:57:48.375 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:57:48.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:48.378 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:48.379 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:57:48.398 INFO:tasks.rados.rados.0.smithi084.stdout:3985: done (4 left) 2024-09-20T22:57:48.399 INFO:tasks.rados.rados.0.smithi084.stdout:3986: done (3 left) 2024-09-20T22:57:48.399 INFO:tasks.rados.rados.0.smithi084.stdout:3987: done (2 left) 2024-09-20T22:57:48.399 INFO:tasks.rados.rados.0.smithi084.stdout:3988: done (1 left) 2024-09-20T22:57:48.399 INFO:tasks.rados.rados.0.smithi084.stdout:3990: done (0 left) 2024-09-20T22:57:48.399 INFO:tasks.rados.rados.0.smithi084.stdout:3991: read oid 37 snap -1 2024-09-20T22:57:48.399 INFO:tasks.rados.rados.0.smithi084.stdout:3991: expect (ObjNum 1189 snap 408 seq_num 1189) 2024-09-20T22:57:48.399 INFO:tasks.rados.rados.0.smithi084.stdout:3992: write (excl) oid 30 current snap is 463 2024-09-20T22:57:48.399 INFO:tasks.rados.rados.0.smithi084.stdout:3992: seq_num 1351 ranges {706257=655880,1890820=504465} 2024-09-20T22:57:48.400 INFO:tasks.rados.rados.0.smithi084.stdout:3992: writing smithi08477203-30 from 706257 to 1362137 tid 1 2024-09-20T22:57:48.406 INFO:tasks.rados.rados.0.smithi084.stdout:3992: writing smithi08477203-30 from 1890820 to 2395285 tid 2 2024-09-20T22:57:48.406 INFO:tasks.rados.rados.0.smithi084.stdout:3993: delete oid 6 current snap is 463 2024-09-20T22:57:48.408 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:57:48.409 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:57:48.420 INFO:tasks.rados.rados.0.smithi084.stdout:3992: finishing write tid 1 to smithi08477203-30 2024-09-20T22:57:48.420 INFO:tasks.rados.rados.0.smithi084.stdout:3992: finishing write tid 2 to smithi08477203-30 2024-09-20T22:57:48.436 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:57:48.437 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:57:48.441 INFO:tasks.rados.rados.0.smithi084.stdout:3992: finishing write tid 3 to smithi08477203-30 2024-09-20T22:57:48.441 INFO:tasks.rados.rados.0.smithi084.stdout:3992: finishing write tid 4 to smithi08477203-30 2024-09-20T22:57:48.441 INFO:tasks.rados.rados.0.smithi084.stdout:3992: oid 30 updating version 0 to 2656 2024-09-20T22:57:48.441 INFO:tasks.rados.rados.0.smithi084.stdout:3992: oid 30 updating version 2656 to 2657 2024-09-20T22:57:48.441 INFO:tasks.rados.rados.0.smithi084.stdout:3992: oid 30 version 2657 is already newer than 2655 2024-09-20T22:57:48.441 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 30 v 2657 (ObjNum 1351 snap 463 seq_num 1351) dirty exists 2024-09-20T22:57:48.442 INFO:tasks.rados.rados.0.smithi084.stdout:3992: left oid 30 (ObjNum 1351 snap 463 seq_num 1351) 2024-09-20T22:57:48.442 INFO:tasks.rados.rados.0.smithi084.stdout:3992: done (2 left) 2024-09-20T22:57:48.442 INFO:tasks.rados.rados.0.smithi084.stdout:3993: done (1 left) 2024-09-20T22:57:48.442 INFO:tasks.rados.rados.0.smithi084.stdout:3994: snap_create 2024-09-20T22:57:48.455 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-20T22:57:48.455 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T22:57:48.471 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:57:48.472 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:57:48.479 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:57:48.487 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:57:48.488 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:57:48.501 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:48.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:48.502 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:57:48.505 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:57:48.505 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:57:48.505 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:57:48 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: ::ffff:172.21.15.154 - - [20/Sep/2024:22:57:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-20T22:57:48.522 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:57:48.522 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:57:48.584 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:57:48.603 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:57:48.624 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:48.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:48.625 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:57:48.726 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:57:48.747 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:48.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:48.748 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:57:48.849 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:57:48.861 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:48 smithi084 ceph-mon[40024]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-20T22:57:48.861 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:48 smithi084 ceph-mon[40024]: osdmap e987: 8 total, 7 up, 7 in 2024-09-20T22:57:48.861 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:48 smithi084 ceph-mon[40024]: from='osd.3 v2:172.21.15.84:6813/3784535689' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi084", "root=default"]}]: dispatch 2024-09-20T22:57:48.861 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:48 smithi084 ceph-mon[40024]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi084", "root=default"]}]: dispatch 2024-09-20T22:57:48.861 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:48 smithi084 ceph-mon[40024]: from='osd.3 ' entity='osd.3' 2024-09-20T22:57:48.861 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:48 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/1429480918' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:57:48.862 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:48 smithi084 ceph-mon[40024]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:57:48.862 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:48 smithi084 ceph-mon[40024]: Health check update: Degraded data redundancy: 306/1725 objects degraded (17.739%), 37 pgs degraded (PG_DEGRADED) 2024-09-20T22:57:48.862 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:48 smithi084 ceph-mon[40024]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-20T22:57:48.862 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:48 smithi084 ceph-mon[40024]: osd.3 v2:172.21.15.84:6813/3784535689 boot 2024-09-20T22:57:48.862 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:48 smithi084 ceph-mon[40024]: osdmap e988: 8 total, 8 up, 7 in 2024-09-20T22:57:48.862 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:48 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T22:57:48.863 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:48 smithi084 ceph-mon[29774]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-20T22:57:48.863 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:48 smithi084 ceph-mon[29774]: osdmap e987: 8 total, 7 up, 7 in 2024-09-20T22:57:48.863 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:48 smithi084 ceph-mon[29774]: from='osd.3 v2:172.21.15.84:6813/3784535689' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi084", "root=default"]}]: dispatch 2024-09-20T22:57:48.863 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:48 smithi084 ceph-mon[29774]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi084", "root=default"]}]: dispatch 2024-09-20T22:57:48.863 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:48 smithi084 ceph-mon[29774]: from='osd.3 ' entity='osd.3' 2024-09-20T22:57:48.863 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:48 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/1429480918' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:57:48.863 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:48 smithi084 ceph-mon[29774]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:57:48.863 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:48 smithi084 ceph-mon[29774]: Health check update: Degraded data redundancy: 306/1725 objects degraded (17.739%), 37 pgs degraded (PG_DEGRADED) 2024-09-20T22:57:48.864 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:48 smithi084 ceph-mon[29774]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-20T22:57:48.864 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:48 smithi084 ceph-mon[29774]: osd.3 v2:172.21.15.84:6813/3784535689 boot 2024-09-20T22:57:48.864 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:48 smithi084 ceph-mon[29774]: osdmap e988: 8 total, 8 up, 7 in 2024-09-20T22:57:48.864 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:48 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T22:57:48.870 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:48.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:48.871 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:57:48.962 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:48 smithi154 ceph-mon[33264]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-20T22:57:48.962 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:48 smithi154 ceph-mon[33264]: osdmap e987: 8 total, 7 up, 7 in 2024-09-20T22:57:48.962 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:48 smithi154 ceph-mon[33264]: from='osd.3 v2:172.21.15.84:6813/3784535689' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi084", "root=default"]}]: dispatch 2024-09-20T22:57:48.962 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:48 smithi154 ceph-mon[33264]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi084", "root=default"]}]: dispatch 2024-09-20T22:57:48.962 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:48 smithi154 ceph-mon[33264]: from='osd.3 ' entity='osd.3' 2024-09-20T22:57:48.962 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:48 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/1429480918' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:57:48.962 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:48 smithi154 ceph-mon[33264]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:57:48.962 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:48 smithi154 ceph-mon[33264]: Health check update: Degraded data redundancy: 306/1725 objects degraded (17.739%), 37 pgs degraded (PG_DEGRADED) 2024-09-20T22:57:48.963 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:48 smithi154 ceph-mon[33264]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-20T22:57:48.963 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:48 smithi154 ceph-mon[33264]: osd.3 v2:172.21.15.84:6813/3784535689 boot 2024-09-20T22:57:48.963 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:48 smithi154 ceph-mon[33264]: osdmap e988: 8 total, 8 up, 7 in 2024-09-20T22:57:48.963 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:48 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T22:57:48.972 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:57:48.993 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:48.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:48.994 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:57:49.060 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-20T22:57:49.061 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:57:49.095 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:57:49.116 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:49.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:49.117 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:57:49.218 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:57:49.222 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-20T22:57:49.240 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:49.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:49.241 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:57:49.341 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:57:49.360 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd unset noscrub 2024-09-20T22:57:49.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:49.363 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:49.364 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:57:49.381 INFO:tasks.rados.rados.0.smithi084.stdout:3994: done (1 left) 2024-09-20T22:57:49.381 INFO:tasks.rados.rados.0.smithi084.stdout:3995: copy_from oid 46 from oid 16 current snap is 464 2024-09-20T22:57:49.382 INFO:tasks.rados.rados.0.smithi084.stdout:3996: write (excl) oid 12 current snap is 464 2024-09-20T22:57:49.382 INFO:tasks.rados.rados.0.smithi084.stdout:3996: seq_num 1353 ranges {739871=406378,1607784=508134} 2024-09-20T22:57:49.385 INFO:tasks.rados.rados.0.smithi084.stdout:3996: writing smithi08477203-12 from 739871 to 1146249 tid 1 2024-09-20T22:57:49.391 INFO:tasks.rados.rados.0.smithi084.stdout:3996: writing smithi08477203-12 from 1607784 to 2115918 tid 2 2024-09-20T22:57:49.392 INFO:tasks.rados.rados.0.smithi084.stdout:3995: finishing copy_from to smithi08477203-46 2024-09-20T22:57:49.392 INFO:tasks.rados.rados.0.smithi084.stdout:3995: got expected ENOENT (src dne) 2024-09-20T22:57:49.392 INFO:tasks.rados.rados.0.smithi084.stdout:3995: finishing copy_from racing read to smithi08477203-46 2024-09-20T22:57:49.392 INFO:tasks.rados.rados.0.smithi084.stdout:3995: done (2 left) 2024-09-20T22:57:49.392 INFO:tasks.rados.rados.0.smithi084.stdout:3997: snap_create 2024-09-20T22:57:49.427 INFO:tasks.rados.rados.0.smithi084.stdout:3991: expect (ObjNum 1189 snap 408 seq_num 1189) 2024-09-20T22:57:49.465 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:57:49.506 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:49.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:49.507 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:57:49.578 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:57:49.608 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:57:49.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:49.634 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:49.634 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:57:49.697 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.6 dump_historic_ops 2024-09-20T22:57:49.736 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:57:49.761 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:49.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:49.762 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:57:49.862 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:57:49.884 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:49.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:49.885 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:57:49.951 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:57:49.986 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:57:50.004 INFO:journalctl@ceph.iscsi.iscsi.a.smithi154.stdout:Sep 20 22:57:49 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-iscsi-iscsi-a[59730]: debug there is no tcmu-runner data available 2024-09-20T22:57:50.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:49 smithi154 ceph-mon[33264]: pgmap v1383: 168 pgs: 1 active+clean+remapped, 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+clean+snaptrim, 41 active+undersized, 1 active+remapped+backfill_toofull, 36 active+undersized+degraded, 87 active+clean; 838 MiB data, 2.3 GiB used, 624 GiB / 626 GiB avail; 306/1725 objects degraded (17.739%); 55/1725 objects misplaced (3.188%) 2024-09-20T22:57:50.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:49 smithi154 ceph-mon[33264]: osdmap e989: 8 total, 8 up, 7 in 2024-09-20T22:57:50.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:49 smithi154 ceph-mon[33264]: 7.14 continuing backfill to osd.2 from (456'176,960'724] MIN to 960'724 2024-09-20T22:57:50.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:49 smithi154 ceph-mon[33264]: 7.14 starting backfill to osd.3 from (456'176,936'716] MIN to 960'724 2024-09-20T22:57:50.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:49 smithi154 ceph-mon[33264]: 7.14 continuing backfill to osd.5 from (456'176,960'724] MIN to 960'724 2024-09-20T22:57:50.008 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:50.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:50.009 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:57:50.016 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:49 smithi084 ceph-mon[29774]: pgmap v1383: 168 pgs: 1 active+clean+remapped, 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+clean+snaptrim, 41 active+undersized, 1 active+remapped+backfill_toofull, 36 active+undersized+degraded, 87 active+clean; 838 MiB data, 2.3 GiB used, 624 GiB / 626 GiB avail; 306/1725 objects degraded (17.739%); 55/1725 objects misplaced (3.188%) 2024-09-20T22:57:50.016 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:49 smithi084 ceph-mon[29774]: osdmap e989: 8 total, 8 up, 7 in 2024-09-20T22:57:50.016 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:49 smithi084 ceph-mon[29774]: 7.14 continuing backfill to osd.2 from (456'176,960'724] MIN to 960'724 2024-09-20T22:57:50.016 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:49 smithi084 ceph-mon[29774]: 7.14 starting backfill to osd.3 from (456'176,936'716] MIN to 960'724 2024-09-20T22:57:50.016 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:49 smithi084 ceph-mon[29774]: 7.14 continuing backfill to osd.5 from (456'176,960'724] MIN to 960'724 2024-09-20T22:57:50.016 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:49 smithi084 ceph-mon[40024]: pgmap v1383: 168 pgs: 1 active+clean+remapped, 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+clean+snaptrim, 41 active+undersized, 1 active+remapped+backfill_toofull, 36 active+undersized+degraded, 87 active+clean; 838 MiB data, 2.3 GiB used, 624 GiB / 626 GiB avail; 306/1725 objects degraded (17.739%); 55/1725 objects misplaced (3.188%) 2024-09-20T22:57:50.016 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:49 smithi084 ceph-mon[40024]: osdmap e989: 8 total, 8 up, 7 in 2024-09-20T22:57:50.016 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:49 smithi084 ceph-mon[40024]: 7.14 continuing backfill to osd.2 from (456'176,960'724] MIN to 960'724 2024-09-20T22:57:50.016 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:49 smithi084 ceph-mon[40024]: 7.14 starting backfill to osd.3 from (456'176,936'716] MIN to 960'724 2024-09-20T22:57:50.016 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:49 smithi084 ceph-mon[40024]: 7.14 continuing backfill to osd.5 from (456'176,960'724] MIN to 960'724 2024-09-20T22:57:50.110 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:57:50.115 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-20T22:57:50.115 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:57:50.133 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:50.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:50.134 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:57:50.234 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:57:50.255 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:50.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:50.256 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:57:50.357 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:57:50.378 INFO:tasks.rados.rados.0.smithi084.stdout:3991: done (2 left) 2024-09-20T22:57:50.378 INFO:tasks.rados.rados.0.smithi084.stdout:3997: done (1 left) 2024-09-20T22:57:50.378 INFO:tasks.rados.rados.0.smithi084.stdout:3998: copy_from oid 16 from oid 46 current snap is 465 2024-09-20T22:57:50.379 INFO:tasks.rados.rados.0.smithi084.stdout:3999: copy_from oid 9 from oid 44 current snap is 465 2024-09-20T22:57:50.379 INFO:tasks.rados.rados.0.smithi084.stdout: waiting on 3 2024-09-20T22:57:50.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:50.384 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:50.384 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:57:50.390 INFO:tasks.rados.rados.0.smithi084.stdout:3998: finishing copy_from racing read to smithi08477203-16 2024-09-20T22:57:50.390 INFO:tasks.rados.rados.0.smithi084.stdout:3996: finishing write tid 1 to smithi08477203-12 2024-09-20T22:57:50.395 INFO:tasks.rados.rados.0.smithi084.stdout:3998: finishing copy_from to smithi08477203-16 2024-09-20T22:57:50.395 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 16 v 2650 (ObjNum 1314 snap 453 seq_num 1314) dirty exists 2024-09-20T22:57:50.395 INFO:tasks.rados.rados.0.smithi084.stdout:3998: done (2 left) 2024-09-20T22:57:50.396 INFO:tasks.rados.rados.0.smithi084.stdout: waiting on 2 2024-09-20T22:57:50.396 INFO:tasks.rados.rados.0.smithi084.stdout:3996: finishing write tid 2 to smithi08477203-12 2024-09-20T22:57:50.397 INFO:tasks.rados.rados.0.smithi084.stdout:3999: finishing copy_from racing read to smithi08477203-9 2024-09-20T22:57:50.397 INFO:tasks.rados.rados.0.smithi084.stdout:3996: finishing write tid 3 to smithi08477203-12 2024-09-20T22:57:50.397 INFO:tasks.rados.rados.0.smithi084.stdout:3996: finishing write tid 4 to smithi08477203-12 2024-09-20T22:57:50.397 INFO:tasks.rados.rados.0.smithi084.stdout:3996: oid 12 updating version 0 to 2856 2024-09-20T22:57:50.397 INFO:tasks.rados.rados.0.smithi084.stdout:3996: oid 12 version 2856 is already newer than 2855 2024-09-20T22:57:50.397 INFO:tasks.rados.rados.0.smithi084.stdout:3996: oid 12 version 2856 is already newer than 2854 2024-09-20T22:57:50.397 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 12 v 2856 (ObjNum 1353 snap 464 seq_num 1353) dirty exists 2024-09-20T22:57:50.397 INFO:tasks.rados.rados.0.smithi084.stdout:3996: left oid 12 (ObjNum 1353 snap 464 seq_num 1353) 2024-09-20T22:57:50.397 INFO:tasks.rados.rados.0.smithi084.stdout:3996: done (1 left) 2024-09-20T22:57:50.398 INFO:tasks.rados.rados.0.smithi084.stdout: waiting on 1 2024-09-20T22:57:50.403 INFO:tasks.rados.rados.0.smithi084.stdout:3999: finishing copy_from to smithi08477203-9 2024-09-20T22:57:50.403 INFO:tasks.rados.rados.0.smithi084.stdout:update_object_version oid 9 v 2842 (ObjNum 1202 snap 411 seq_num 1202) dirty exists 2024-09-20T22:57:50.403 INFO:tasks.rados.rados.0.smithi084.stdout:3999: done (0 left) 2024-09-20T22:57:50.404 INFO:tasks.rados.rados.0.smithi084.stderr:0 errors. 2024-09-20T22:57:50.405 INFO:tasks.rados.rados.0.smithi084.stderr: 2024-09-20T22:57:50.408 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd dump --format=json 2024-09-20T22:57:50.454 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-20T22:57:50.455 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:57:50.485 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:57:50.515 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:50.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:50.516 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:57:50.616 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:57:50.694 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:50.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:50.695 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:57:50.796 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:57:50.816 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:50.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:50.817 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:57:50.838 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:50 smithi084 ceph-mon[29774]: purged_snaps scrub starts 2024-09-20T22:57:50.838 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:50 smithi084 ceph-mon[29774]: purged_snaps scrub ok 2024-09-20T22:57:50.838 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:50 smithi084 ceph-mon[29774]: osdmap e990: 8 total, 8 up, 7 in 2024-09-20T22:57:50.838 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:50 smithi084 ceph-mon[29774]: 7.17 continuing backfill to osd.1 from (436'456,930'2006] MIN to 930'2006 2024-09-20T22:57:50.838 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:50 smithi084 ceph-mon[29774]: 7.17 continuing backfill to osd.2 from (436'456,930'2006] MIN to 930'2006 2024-09-20T22:57:50.838 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:50 smithi084 ceph-mon[29774]: 7.17 continuing backfill to osd.5 from (436'456,930'2006] MIN to 930'2006 2024-09-20T22:57:50.839 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:50 smithi084 ceph-mon[40024]: purged_snaps scrub starts 2024-09-20T22:57:50.839 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:50 smithi084 ceph-mon[40024]: purged_snaps scrub ok 2024-09-20T22:57:50.839 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:50 smithi084 ceph-mon[40024]: osdmap e990: 8 total, 8 up, 7 in 2024-09-20T22:57:50.839 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:50 smithi084 ceph-mon[40024]: 7.17 continuing backfill to osd.1 from (436'456,930'2006] MIN to 930'2006 2024-09-20T22:57:50.839 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:50 smithi084 ceph-mon[40024]: 7.17 continuing backfill to osd.2 from (436'456,930'2006] MIN to 930'2006 2024-09-20T22:57:50.839 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:50 smithi084 ceph-mon[40024]: 7.17 continuing backfill to osd.5 from (436'456,930'2006] MIN to 930'2006 2024-09-20T22:57:50.877 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:57:50.918 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:57:50.939 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:50.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:50.940 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:57:51.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:50 smithi154 ceph-mon[33264]: purged_snaps scrub starts 2024-09-20T22:57:51.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:50 smithi154 ceph-mon[33264]: purged_snaps scrub ok 2024-09-20T22:57:51.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:50 smithi154 ceph-mon[33264]: osdmap e990: 8 total, 8 up, 7 in 2024-09-20T22:57:51.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:50 smithi154 ceph-mon[33264]: 7.17 continuing backfill to osd.1 from (436'456,930'2006] MIN to 930'2006 2024-09-20T22:57:51.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:50 smithi154 ceph-mon[33264]: 7.17 continuing backfill to osd.2 from (436'456,930'2006] MIN to 930'2006 2024-09-20T22:57:51.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:50 smithi154 ceph-mon[33264]: 7.17 continuing backfill to osd.5 from (436'456,930'2006] MIN to 930'2006 2024-09-20T22:57:51.041 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:57:51.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:51.065 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:51.065 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:57:51.118 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-20T22:57:51.166 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:57:51.186 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:51.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:51.188 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:57:51.288 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:57:51.311 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:51.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:51.312 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:57:51.327 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-20T22:57:51.366 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:57:51.412 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:57:51.433 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:51.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:51.434 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:57:51.477 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:57:51.535 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:57:51.556 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:51.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:51.557 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:57:51.658 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:57:51.679 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:51.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:51.680 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:57:51.722 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:51 smithi084 ceph-mon[29774]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:57:51.722 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:51 smithi084 ceph-mon[29774]: pgmap v1386: 168 pgs: 2 remapped+peering, 39 peering, 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+clean+snaptrim, 19 active+undersized, 19 active+undersized+degraded, 87 active+clean; 839 MiB data, 2.3 GiB used, 624 GiB / 626 GiB avail; 3.2 MiB/s rd, 2.7 MiB/s wr, 12 op/s; 133/1734 objects degraded (7.670%); 34/1734 objects misplaced (1.961%) 2024-09-20T22:57:51.722 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:51 smithi084 ceph-mon[29774]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-09-20T22:57:51.722 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:51 smithi084 ceph-mon[29774]: osdmap e991: 8 total, 8 up, 7 in 2024-09-20T22:57:51.723 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:51 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/1317207363' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:57:51.723 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:51 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:57:51.723 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:51 smithi084 ceph-mon[40024]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:57:51.723 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:51 smithi084 ceph-mon[40024]: pgmap v1386: 168 pgs: 2 remapped+peering, 39 peering, 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+clean+snaptrim, 19 active+undersized, 19 active+undersized+degraded, 87 active+clean; 839 MiB data, 2.3 GiB used, 624 GiB / 626 GiB avail; 3.2 MiB/s rd, 2.7 MiB/s wr, 12 op/s; 133/1734 objects degraded (7.670%); 34/1734 objects misplaced (1.961%) 2024-09-20T22:57:51.723 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:51 smithi084 ceph-mon[40024]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-09-20T22:57:51.723 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:51 smithi084 ceph-mon[40024]: osdmap e991: 8 total, 8 up, 7 in 2024-09-20T22:57:51.723 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:51 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/1317207363' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:57:51.723 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:51 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:57:51.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:51 smithi154 ceph-mon[33264]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:57:51.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:51 smithi154 ceph-mon[33264]: pgmap v1386: 168 pgs: 2 remapped+peering, 39 peering, 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+clean+snaptrim, 19 active+undersized, 19 active+undersized+degraded, 87 active+clean; 839 MiB data, 2.3 GiB used, 624 GiB / 626 GiB avail; 3.2 MiB/s rd, 2.7 MiB/s wr, 12 op/s; 133/1734 objects degraded (7.670%); 34/1734 objects misplaced (1.961%) 2024-09-20T22:57:51.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:51 smithi154 ceph-mon[33264]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-09-20T22:57:51.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:51 smithi154 ceph-mon[33264]: osdmap e991: 8 total, 8 up, 7 in 2024-09-20T22:57:51.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:51 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/1317207363' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:57:51.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:51 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:57:51.781 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:57:51.802 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:51.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:51.803 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:57:51.893 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-20T22:57:51.893 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:57:51.904 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:57:51.926 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:51.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:51.928 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:57:52.029 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:57:52.029 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:57:52.050 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:52.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:52.051 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:57:52.123 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-20T22:57:52.123 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":991,"fsid":"a9c0bc70-77a0-11ef-baf5-efdc52797490","created":"2024-09-20T22:36:30.240626+0000","modified":"2024-09-20T22:57:51.367673+0000","last_up_change":"2024-09-20T22:57:48.362816+0000","last_in_change":"2024-09-20T22:56:07.496853+0000","flags":"noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5806080,"flags_set":["nodeep-scrub","noscrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":19,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-20T22:39:21.984381+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":11,"pg_placement_num":11,"pg_placement_num_target":11,"pg_num_target":11,"pg_num_pending":11,"last_pg_merge_meta":{"source_pgid":"1.b","ready_epoch":299,"last_epoch_started":298,"last_epoch_clean":298,"source_version":"77'128","target_version":"77'128"},"last_change":"769","last_force_op_resend":"0","last_force_op_resend_prenautilus":"299","last_force_op_resend_preluminous":"297","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.9099999666213989,"score_stable":1.9099999666213989,"optimal_score":1,"raw_score_acting":1.9099999666213989,"raw_score_stable":1.9099999666213989,"primary_affinity_weighted":0.875,"average_primary_affinity":1,"average_primary_affinity_weighted":0.87999999523162842}},{"pool":2,"pool_name":"datapool","create_time":"2024-09-20T22:41:35.820081+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":69,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.3299999237060547,"score_stable":2.3299999237060547,"optimal_score":0.86000001430511475,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":0.875,"average_primary_affinity":1,"average_primary_affinity_weighted":0.87999999523162842}},{"pool":3,"pool_name":".rgw.root","create_time":"2024-09-20T22:41:36.546389+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":0.875,"average_primary_affinity":1,"average_primary_affinity_weighted":0.87999999523162842}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-09-20T22:41:38.454640+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.190000057220459,"score_stable":2.190000057220459,"optimal_score":1,"raw_score_acting":2.190000057220459,"raw_score_stable":2.190000057220459,"primary_affinity_weighted":0.875,"average_primary_affinity":1,"average_primary_affinity_weighted":0.87999999523162842}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-09-20T22:41:40.407129+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"72","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.309999942779541,"score_stable":1.309999942779541,"optimal_score":1,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":0.875,"average_primary_affinity":1,"average_primary_affinity_weighted":0.87999999523162842}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-09-20T22:41:42.476716+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"74","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":0.875,"average_primary_affinity":1,"average_primary_affinity_weighted":0.87999999523162842}},{"pool":7,"pool_name":"unique_pool_0","create_time":"2024-09-20T22:42:51.354358+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":26,"pg_placement_num":26,"pg_placement_num_target":26,"pg_num_target":26,"pg_num_pending":26,"last_pg_merge_meta":{"source_pgid":"7.10","ready_epoch":484,"last_epoch_started":483,"last_epoch_clean":483,"source_version":"480'172","target_version":"480'253"},"last_change":"990","last_force_op_resend":"0","last_force_op_resend_prenautilus":"484","last_force_op_resend_preluminous":"932","auid":0,"snap_mode":"selfmanaged","snap_seq":466,"snap_epoch":990,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.875,"average_primary_affinity":1,"average_primary_affinity_weighted":0.87999999523162842}}],"osds":[{"osd":0,"uuid":"ea36a489-6f91-4b2f-b784-b90506b9805d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":990,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6801","nonce":268084626}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6802","nonce":268084626}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6804","nonce":268084626}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6803","nonce":268084626}]},"public_addr":"172.21.15.84:6801/268084626","cluster_addr":"172.21.15.84:6802/268084626","heartbeat_back_addr":"172.21.15.84:6804/268084626","heartbeat_front_addr":"172.21.15.84:6803/268084626","state":["exists","up"]},{"osd":1,"uuid":"c8649d2f-27ef-47f8-bc94-f1cd9e62bdb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":988,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6805","nonce":418374136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6806","nonce":418374136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6808","nonce":418374136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6807","nonce":418374136}]},"public_addr":"172.21.15.84:6805/418374136","cluster_addr":"172.21.15.84:6806/418374136","heartbeat_back_addr":"172.21.15.84:6808/418374136","heartbeat_front_addr":"172.21.15.84:6807/418374136","state":["exists","up"]},{"osd":2,"uuid":"c728eb48-a0f0-4de4-a651-7f3bec867280","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":988,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6809","nonce":3977168611}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6810","nonce":3977168611}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6812","nonce":3977168611}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6811","nonce":3977168611}]},"public_addr":"172.21.15.84:6809/3977168611","cluster_addr":"172.21.15.84:6810/3977168611","heartbeat_back_addr":"172.21.15.84:6812/3977168611","heartbeat_front_addr":"172.21.15.84:6811/3977168611","state":["exists","up"]},{"osd":3,"uuid":"573f44eb-0853-4630-a7d8-9caaeb256f9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":25,"last_clean_end":939,"up_from":988,"up_thru":990,"down_at":940,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6813","nonce":3784535689}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6814","nonce":3784535689}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6816","nonce":3784535689}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6815","nonce":3784535689}]},"public_addr":"172.21.15.84:6813/3784535689","cluster_addr":"172.21.15.84:6814/3784535689","heartbeat_back_addr":"172.21.15.84:6816/3784535689","heartbeat_front_addr":"172.21.15.84:6815/3784535689","state":["exists","up"]},{"osd":4,"uuid":"a52c3b94-c5f3-490f-8ac8-09061b9b2abb","up":1,"in":1,"weight":1,"primary_affinity":0,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":830,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":1793136002}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6801","nonce":1793136002}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6803","nonce":1793136002}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":1793136002}]},"public_addr":"172.21.15.154:6800/1793136002","cluster_addr":"172.21.15.154:6801/1793136002","heartbeat_back_addr":"172.21.15.154:6803/1793136002","heartbeat_front_addr":"172.21.15.154:6802/1793136002","state":["exists","up"]},{"osd":5,"uuid":"4fa0dc8e-25cc-46cd-96ff-33f20b6faec3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":988,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":493899927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6805","nonce":493899927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6807","nonce":493899927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":493899927}]},"public_addr":"172.21.15.154:6804/493899927","cluster_addr":"172.21.15.154:6805/493899927","heartbeat_back_addr":"172.21.15.154:6807/493899927","heartbeat_front_addr":"172.21.15.154:6806/493899927","state":["exists","up"]},{"osd":6,"uuid":"56788d74-810e-4827-a615-958c12f8a368","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":989,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":2375091033}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6809","nonce":2375091033}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6811","nonce":2375091033}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":2375091033}]},"public_addr":"172.21.15.154:6808/2375091033","cluster_addr":"172.21.15.154:6809/2375091033","heartbeat_back_addr":"172.21.15.154:6811/2375091033","heartbeat_front_addr":"172.21.15.154:6810/2375091033","state":["exists","up"]},{"osd":7,"uuid":"83503727-afc1-4b95-8b96-ecb63f3338c2","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":836,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":3717704491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6813","nonce":3717704491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6815","nonce":3717704491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":3717704491}]},"public_addr":"172.21.15.154:6812/3717704491","cluster_addr":"172.21.15.154:6813/3717704491","heartbeat_back_addr":"172.21.15.154:6815/3717704491","heartbeat_front_addr":"172.21.15.154:6814/3717704491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T22:57:42.287953+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T22:57:38.488570+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T22:57:08.865769+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T22:57:48.144965+0000","dead_epoch":939},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T22:56:43.520988+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T22:57:21.222014+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T22:57:44.138630+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-09-20T22:57:45.361525+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.a","mappings":[{"from":6,"to":2}]},{"pgid":"4.4","mappings":[{"from":1,"to":5}]},{"pgid":"4.6","mappings":[{"from":1,"to":5}]},{"pgid":"4.9","mappings":[{"from":3,"to":2}]},{"pgid":"4.11","mappings":[{"from":3,"to":5}]},{"pgid":"4.1b","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.84:0/542020545":"2024-09-21T22:42:07.596045+0000","172.21.15.84:0/25873236":"2024-09-21T22:36:54.917812+0000","172.21.15.84:6800/820908470":"2024-09-21T22:36:54.917812+0000","172.21.15.84:0/567614908":"2024-09-21T22:36:54.917812+0000","172.21.15.84:0/2963622459":"2024-09-21T22:42:07.596045+0000","172.21.15.84:0/1176849276":"2024-09-21T22:37:17.690104+0000","172.21.15.84:6800/2932420050":"2024-09-21T22:37:17.690104+0000","172.21.15.84:0/3879263635":"2024-09-21T22:37:17.690104+0000","172.21.15.84:0/969472279":"2024-09-21T22:36:54.917812+0000","172.21.15.84:0/1361143214":"2024-09-21T22:37:17.690104+0000","172.21.15.84:0/3353126955":"2024-09-21T22:42:07.596045+0000","172.21.15.84:6800/4024278029":"2024-09-21T22:42:07.596045+0000","172.21.15.84:0/547902262":"2024-09-21T22:42:07.596045+0000","172.21.15.84:0/1242929317":"2024-09-21T22:42:07.596045+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":7,"snaps":[{"begin":363,"length":1},{"begin":371,"length":1},{"begin":381,"length":1},{"begin":384,"length":1},{"begin":386,"length":3},{"begin":390,"length":8},{"begin":400,"length":16},{"begin":417,"length":1},{"begin":419,"length":2},{"begin":422,"length":1},{"begin":426,"length":2},{"begin":431,"length":1},{"begin":437,"length":1},{"begin":439,"length":1},{"begin":442,"length":1},{"begin":445,"length":2},{"begin":448,"length":2},{"begin":453,"length":1},{"begin":455,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-20T22:57:52.151 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:57:52.173 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:52.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:52.174 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:57:52.275 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:57:52.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:52.298 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:52.298 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:57:52.376 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is unset 2024-09-20T22:57:52.398 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:57:52.420 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:52.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:52.421 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:57:52.522 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:57:52.545 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:52.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:52.546 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:57:52.555 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.7 dump_blocked_ops 2024-09-20T22:57:52.647 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:57:52.668 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:52.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:52.669 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:57:52.770 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:57:52.790 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:52.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:52.791 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:57:52.845 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:57:52.892 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:57:52.914 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:52.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:52.915 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:57:52.966 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:52 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/3674074910' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-20T22:57:52.966 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:52 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T22:57:52.966 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:52 smithi154 ceph-mon[33264]: osdmap e992: 8 total, 8 up, 7 in 2024-09-20T22:57:53.016 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:57:53.025 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:52 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/3674074910' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-20T22:57:53.025 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:52 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T22:57:53.025 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:52 smithi084 ceph-mon[40024]: osdmap e992: 8 total, 8 up, 7 in 2024-09-20T22:57:53.025 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:52 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/3674074910' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-20T22:57:53.025 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:52 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T22:57:53.025 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:52 smithi084 ceph-mon[29774]: osdmap e992: 8 total, 8 up, 7 in 2024-09-20T22:57:53.038 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:53.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:53.039 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:57:53.045 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:57:53.045 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:57:53.046 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:57:53.046 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:57:53.046 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:57:53.046 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-20T22:57:53.046 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:57:53.046 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:57:53.046 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:57:53.047 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:57:53.047 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T22:57:53.047 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:57:53.047 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:57:53.047 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:57:53.047 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:57:53.048 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:57:53.139 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:57:53.161 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:53.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:53.162 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:57:53.263 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:57:53.284 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:53.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:53.285 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:57:53.293 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-20T22:57:53.294 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [], 2024-09-20T22:57:53.294 INFO:teuthology.orchestra.run.smithi154.stdout: "complaint_time": 30, 2024-09-20T22:57:53.294 INFO:teuthology.orchestra.run.smithi154.stdout: "num_blocked_ops": 0 2024-09-20T22:57:53.294 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-20T22:57:53.385 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:57:53.408 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:53.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:53.409 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:57:53.509 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:57:53.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:53.531 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:53.531 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:57:53.632 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:57:53.653 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:53.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:53.655 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:57:53.658 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph pg dump --format=json 2024-09-20T22:57:53.756 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:57:53.777 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:53.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:53.778 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:57:53.879 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:57:53.891 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:53 smithi154 ceph-mon[33264]: pgmap v1389: 168 pgs: 2 remapped+peering, 39 peering, 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+clean+snaptrim, 19 active+undersized, 19 active+undersized+degraded, 87 active+clean; 839 MiB data, 2.3 GiB used, 624 GiB / 626 GiB avail; 3.2 MiB/s rd, 2.7 MiB/s wr, 12 op/s; 133/1734 objects degraded (7.670%); 34/1734 objects misplaced (1.961%) 2024-09-20T22:57:53.892 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:53 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/3484272084' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:57:53.892 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:53 smithi154 ceph-mon[33264]: 7.17 scrub starts 2024-09-20T22:57:53.892 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:53 smithi154 ceph-mon[33264]: 7.17 scrub ok 2024-09-20T22:57:53.892 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:53 smithi154 ceph-mon[33264]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:57:53.892 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:53 smithi154 ceph-mon[33264]: Health check update: Degraded data redundancy: 133/1734 objects degraded (7.670%), 20 pgs degraded (PG_DEGRADED) 2024-09-20T22:57:53.892 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:53 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:57:53.892 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:53 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:57:53.901 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:53.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:53.902 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:57:53.920 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.7 dump_historic_ops 2024-09-20T22:57:54.003 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:57:54.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:54.026 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:54.026 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:57:54.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:53 smithi084 ceph-mon[29774]: pgmap v1389: 168 pgs: 2 remapped+peering, 39 peering, 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+clean+snaptrim, 19 active+undersized, 19 active+undersized+degraded, 87 active+clean; 839 MiB data, 2.3 GiB used, 624 GiB / 626 GiB avail; 3.2 MiB/s rd, 2.7 MiB/s wr, 12 op/s; 133/1734 objects degraded (7.670%); 34/1734 objects misplaced (1.961%) 2024-09-20T22:57:54.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:53 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/3484272084' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:57:54.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:53 smithi084 ceph-mon[29774]: 7.17 scrub starts 2024-09-20T22:57:54.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:53 smithi084 ceph-mon[29774]: 7.17 scrub ok 2024-09-20T22:57:54.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:53 smithi084 ceph-mon[29774]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:57:54.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:53 smithi084 ceph-mon[29774]: Health check update: Degraded data redundancy: 133/1734 objects degraded (7.670%), 20 pgs degraded (PG_DEGRADED) 2024-09-20T22:57:54.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:53 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:57:54.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:53 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:57:54.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:53 smithi084 ceph-mon[40024]: pgmap v1389: 168 pgs: 2 remapped+peering, 39 peering, 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+clean+snaptrim, 19 active+undersized, 19 active+undersized+degraded, 87 active+clean; 839 MiB data, 2.3 GiB used, 624 GiB / 626 GiB avail; 3.2 MiB/s rd, 2.7 MiB/s wr, 12 op/s; 133/1734 objects degraded (7.670%); 34/1734 objects misplaced (1.961%) 2024-09-20T22:57:54.116 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:53 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/3484272084' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:57:54.116 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:53 smithi084 ceph-mon[40024]: 7.17 scrub starts 2024-09-20T22:57:54.116 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:53 smithi084 ceph-mon[40024]: 7.17 scrub ok 2024-09-20T22:57:54.116 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:53 smithi084 ceph-mon[40024]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:57:54.116 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:53 smithi084 ceph-mon[40024]: Health check update: Degraded data redundancy: 133/1734 objects degraded (7.670%), 20 pgs degraded (PG_DEGRADED) 2024-09-20T22:57:54.116 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:53 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:57:54.116 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:53 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:57:54.127 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:57:54.148 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:54.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:54.149 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:57:54.205 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:57:54.250 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:57:54.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:54.272 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:54.272 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:57:54.372 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:57:54.393 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:54.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:54.394 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:57:54.495 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:57:54.516 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:54.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:54.517 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:57:54.618 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:57:54.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:54.643 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:54.643 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:57:54.704 INFO:teuthology.orchestra.run.smithi154.stdout:{ 2024-09-20T22:57:54.704 INFO:teuthology.orchestra.run.smithi154.stdout: "size": 20, 2024-09-20T22:57:54.704 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 600, 2024-09-20T22:57:54.704 INFO:teuthology.orchestra.run.smithi154.stdout: "ops": [ 2024-09-20T22:57:54.704 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.704 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:6253 7.9 e629/602 7:91f2d47c:::smithi08477203-8:head v 629'948, mlcod=627'945)", 2024-09-20T22:57:54.705 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:51:49.626110+0000", 2024-09-20T22:57:54.705 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 365.07004549800001, 2024-09-20T22:57:54.705 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.017238303, 2024-09-20T22:57:54.705 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:57:54.705 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:57:54.705 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:57:54.705 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:57:54.705 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.705 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:57:54.705 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:49.626110+0000", 2024-09-20T22:57:54.705 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:57:54.705 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.706 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.706 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:57:54.706 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:49.626110+0000", 2024-09-20T22:57:54.706 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:57:54.706 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.706 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.706 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:57:54.706 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:49.626111+0000", 2024-09-20T22:57:54.706 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.33e-06 2024-09-20T22:57:54.706 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.706 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.706 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:57:54.706 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:49.633470+0000", 2024-09-20T22:57:54.707 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0073589700000000003 2024-09-20T22:57:54.707 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.707 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.707 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:57:54.707 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:49.633492+0000", 2024-09-20T22:57:54.707 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.1443000000000001e-05 2024-09-20T22:57:54.707 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.707 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.707 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:57:54.707 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:49.633503+0000", 2024-09-20T22:57:54.707 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1610000000000001e-05 2024-09-20T22:57:54.707 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.707 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.708 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:57:54.708 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:49.633575+0000", 2024-09-20T22:57:54.708 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.1433999999999994e-05 2024-09-20T22:57:54.708 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.708 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.708 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:57:54.708 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:49.633600+0000", 2024-09-20T22:57:54.708 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.4960000000000002e-05 2024-09-20T22:57:54.708 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.708 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.708 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:57:54.708 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:49.643306+0000", 2024-09-20T22:57:54.708 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0097065109999999993 2024-09-20T22:57:54.709 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.709 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.709 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:57:54.709 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:51:49.643348+0000", 2024-09-20T22:57:54.709 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.2045000000000001e-05 2024-09-20T22:57:54.709 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:57:54.709 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:57:54.709 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:57:54.709 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.709 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.709 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:6659 7.5 7:a160b5d4:::smithi08477203-16:head [copy-from ver 1807] snapc 12d=[12d,12c,12b,12a,128,127,126,124,122,121,120,11f,11e,11b,119,118,115,112,10e,10c,107,103,fd,fb,fa,f8,ed,d7,c2,bf] ondisk+write+known_if_redirected+supports_pool_eio e674)", 2024-09-20T22:57:54.709 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:52:34.370352+0000", 2024-09-20T22:57:54.709 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 320.32580400400002, 2024-09-20T22:57:54.709 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.018083765000000002, 2024-09-20T22:57:54.710 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:57:54.710 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:57:54.710 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:57:54.710 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:57:54.710 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:57:54.710 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:57:54.710 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 6659 2024-09-20T22:57:54.710 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.710 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:57:54.710 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.710 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:57:54.710 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:34.370352+0000", 2024-09-20T22:57:54.710 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:57:54.711 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.711 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.711 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:57:54.711 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:34.370352+0000", 2024-09-20T22:57:54.711 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:57:54.711 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.711 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.711 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:57:54.711 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:34.370353+0000", 2024-09-20T22:57:54.711 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.61e-06 2024-09-20T22:57:54.711 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.711 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.711 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:57:54.712 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:34.370360+0000", 2024-09-20T22:57:54.712 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.6760000000000001e-06 2024-09-20T22:57:54.712 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.712 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.712 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:57:54.712 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:34.370371+0000", 2024-09-20T22:57:54.712 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1155000000000001e-05 2024-09-20T22:57:54.712 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.712 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.712 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:57:54.712 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:34.370380+0000", 2024-09-20T22:57:54.712 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.9039999999999998e-06 2024-09-20T22:57:54.712 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.712 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.713 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:57:54.713 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:34.370461+0000", 2024-09-20T22:57:54.713 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.0667000000000005e-05 2024-09-20T22:57:54.713 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.713 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.713 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:57:54.713 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:34.370571+0000", 2024-09-20T22:57:54.713 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000110129 2024-09-20T22:57:54.713 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.713 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.713 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 0,2", 2024-09-20T22:57:54.713 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:34.378118+0000", 2024-09-20T22:57:54.713 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0075469489999999998 2024-09-20T22:57:54.713 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.714 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.714 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:57:54.714 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:34.382656+0000", 2024-09-20T22:57:54.714 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0045383239999999998 2024-09-20T22:57:54.714 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.714 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.714 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:57:54.714 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:34.385168+0000", 2024-09-20T22:57:54.714 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0025123770000000001 2024-09-20T22:57:54.714 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.714 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.714 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:57:54.714 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:34.388343+0000", 2024-09-20T22:57:54.715 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.003174713 2024-09-20T22:57:54.715 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.715 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.715 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:57:54.715 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:34.388382+0000", 2024-09-20T22:57:54.715 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.8550000000000002e-05 2024-09-20T22:57:54.715 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.715 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.715 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:57:54.715 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:34.388435+0000", 2024-09-20T22:57:54.715 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.3711000000000001e-05 2024-09-20T22:57:54.715 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:57:54.715 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:57:54.715 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:57:54.716 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.716 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.716 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:6829 7.5 7:a160b5d4:::smithi08477203-16:head [write 548159~419732,stat] snapc 136=[136,135,134,133,132,131,130,12f,12e,12d,12c,12b,12a,128,127,126,124,122,121,11f,11b,119,112,10e,10c,107,103,fb,fa,f8,ed,d7,c2,bf] ondisk+write+known_if_redirected+supports_pool_eio e688)", 2024-09-20T22:57:54.716 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:52:48.308503+0000", 2024-09-20T22:57:54.716 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 306.38765267899998, 2024-09-20T22:57:54.716 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.020227210999999998, 2024-09-20T22:57:54.716 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:57:54.716 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:57:54.716 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:57:54.716 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:57:54.716 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:57:54.716 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:57:54.716 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 6829 2024-09-20T22:57:54.717 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.717 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:57:54.717 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.717 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:57:54.717 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.308503+0000", 2024-09-20T22:57:54.717 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:57:54.717 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.717 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.717 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:57:54.717 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.308503+0000", 2024-09-20T22:57:54.717 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:57:54.717 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.717 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.717 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:57:54.718 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.308505+0000", 2024-09-20T22:57:54.718 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.765e-06 2024-09-20T22:57:54.718 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.718 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.718 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:57:54.718 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.308957+0000", 2024-09-20T22:57:54.718 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00045254399999999998 2024-09-20T22:57:54.718 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.718 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.718 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:57:54.718 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.308975+0000", 2024-09-20T22:57:54.718 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.7489000000000001e-05 2024-09-20T22:57:54.718 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.718 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.719 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:57:54.719 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.308983+0000", 2024-09-20T22:57:54.719 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.4500000000000004e-06 2024-09-20T22:57:54.719 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.719 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.719 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:57:54.719 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.309063+0000", 2024-09-20T22:57:54.719 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.0264000000000003e-05 2024-09-20T22:57:54.719 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.719 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.719 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:57:54.720 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.309200+0000", 2024-09-20T22:57:54.720 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00013697500000000001 2024-09-20T22:57:54.720 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.720 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.720 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 0,2", 2024-09-20T22:57:54.720 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.309374+0000", 2024-09-20T22:57:54.720 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000173597 2024-09-20T22:57:54.720 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.720 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.720 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:57:54.720 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.318972+0000", 2024-09-20T22:57:54.721 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0095980430000000005 2024-09-20T22:57:54.721 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.721 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.721 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:57:54.721 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.326825+0000", 2024-09-20T22:57:54.721 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0078531720000000003 2024-09-20T22:57:54.721 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.721 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.721 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:57:54.721 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.328638+0000", 2024-09-20T22:57:54.721 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0018131969999999999 2024-09-20T22:57:54.721 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.722 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.722 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:57:54.722 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.328685+0000", 2024-09-20T22:57:54.722 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.6536999999999998e-05 2024-09-20T22:57:54.722 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.722 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.722 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:57:54.722 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:52:48.328730+0000", 2024-09-20T22:57:54.722 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.5178e-05 2024-09-20T22:57:54.722 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:57:54.722 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:57:54.722 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:57:54.722 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.722 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.723 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:7173 7.8 e721/430 7:1e9e2524:::smithi08477203-5:head v 721'865, mlcod=721'861)", 2024-09-20T22:57:54.723 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:53:21.336783+0000", 2024-09-20T22:57:54.723 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 273.35937280899998, 2024-09-20T22:57:54.723 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.015521446, 2024-09-20T22:57:54.723 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:57:54.723 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:57:54.723 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:57:54.723 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:57:54.723 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.723 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:57:54.723 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:21.336783+0000", 2024-09-20T22:57:54.723 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:57:54.724 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.724 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.724 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:57:54.724 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:21.336783+0000", 2024-09-20T22:57:54.724 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:57:54.724 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.724 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.724 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:57:54.724 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:21.336784+0000", 2024-09-20T22:57:54.724 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.7899999999999997e-07 2024-09-20T22:57:54.724 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.724 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.725 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:57:54.725 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:21.344323+0000", 2024-09-20T22:57:54.725 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0075393090000000001 2024-09-20T22:57:54.725 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.725 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.725 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:57:54.725 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:21.344349+0000", 2024-09-20T22:57:54.725 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.5712e-05 2024-09-20T22:57:54.725 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.725 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.725 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:57:54.725 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:21.344362+0000", 2024-09-20T22:57:54.725 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3131e-05 2024-09-20T22:57:54.725 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.726 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.726 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:57:54.726 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:21.344434+0000", 2024-09-20T22:57:54.726 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.1837999999999999e-05 2024-09-20T22:57:54.726 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.726 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.726 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:57:54.726 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:21.344454+0000", 2024-09-20T22:57:54.726 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.0404999999999999e-05 2024-09-20T22:57:54.726 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.726 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.726 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:57:54.726 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:21.352250+0000", 2024-09-20T22:57:54.726 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0077961489999999996 2024-09-20T22:57:54.726 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.727 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.727 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:57:54.727 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:21.352304+0000", 2024-09-20T22:57:54.727 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.4122999999999997e-05 2024-09-20T22:57:54.727 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:57:54.727 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:57:54.727 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:57:54.727 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.727 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.727 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:7387 7.9 e743/602 7:907aec0e:::smithi08477203-3:head v 743'1242, mlcod=741'1234)", 2024-09-20T22:57:54.727 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:53:43.309636+0000", 2024-09-20T22:57:54.727 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 251.38651963699999, 2024-09-20T22:57:54.727 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.032548296999999997, 2024-09-20T22:57:54.727 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:57:54.728 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:57:54.728 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:57:54.728 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:57:54.728 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.728 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:57:54.728 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:43.309636+0000", 2024-09-20T22:57:54.728 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:57:54.728 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.728 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.728 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:57:54.728 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:43.309636+0000", 2024-09-20T22:57:54.728 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:57:54.728 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.728 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.729 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:57:54.729 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:43.309637+0000", 2024-09-20T22:57:54.729 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.1699999999999997e-07 2024-09-20T22:57:54.729 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.729 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.729 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:57:54.729 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:43.338457+0000", 2024-09-20T22:57:54.729 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.028819827999999999 2024-09-20T22:57:54.729 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.729 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.729 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:57:54.729 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:43.338482+0000", 2024-09-20T22:57:54.729 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.55e-05 2024-09-20T22:57:54.729 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.730 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.730 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:57:54.730 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:43.338493+0000", 2024-09-20T22:57:54.730 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1308000000000001e-05 2024-09-20T22:57:54.730 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.730 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.730 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:57:54.730 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:43.338561+0000", 2024-09-20T22:57:54.730 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.7535999999999998e-05 2024-09-20T22:57:54.730 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.730 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.730 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:57:54.730 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:43.338585+0000", 2024-09-20T22:57:54.731 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.3649e-05 2024-09-20T22:57:54.731 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.731 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.731 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:57:54.731 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:43.342155+0000", 2024-09-20T22:57:54.731 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0035700509999999999 2024-09-20T22:57:54.731 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.731 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.731 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:57:54.731 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:53:43.342184+0000", 2024-09-20T22:57:54.731 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.9607999999999998e-05 2024-09-20T22:57:54.731 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:57:54.731 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:57:54.731 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:57:54.731 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.732 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.732 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24415.0:744 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch (1) in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e775)", 2024-09-20T22:57:54.732 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:54:16.547604+0000", 2024-09-20T22:57:54.732 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 218.148551674, 2024-09-20T22:57:54.732 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.79000451299999996, 2024-09-20T22:57:54.732 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:57:54.732 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:57:54.732 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:57:54.732 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:57:54.732 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24415", 2024-09-20T22:57:54.732 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.154:0/2056244102", 2024-09-20T22:57:54.732 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 744 2024-09-20T22:57:54.732 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.733 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:57:54.733 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.733 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:57:54.733 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:54:16.547604+0000", 2024-09-20T22:57:54.733 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:57:54.733 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.733 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.733 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:57:54.733 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:54:16.547604+0000", 2024-09-20T22:57:54.733 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:57:54.733 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.733 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.733 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:57:54.734 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:54:16.547607+0000", 2024-09-20T22:57:54.734 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.7460000000000001e-06 2024-09-20T22:57:54.734 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.734 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.734 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:57:54.734 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:54:16.547620+0000", 2024-09-20T22:57:54.734 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3164e-05 2024-09-20T22:57:54.734 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.734 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.734 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:57:54.734 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:54:16.547647+0000", 2024-09-20T22:57:54.734 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.7679e-05 2024-09-20T22:57:54.734 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.734 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.735 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:57:54.735 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:54:16.547663+0000", 2024-09-20T22:57:54.735 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.523e-05 2024-09-20T22:57:54.735 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.735 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.735 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:57:54.735 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:54:16.547788+0000", 2024-09-20T22:57:54.735 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00012490999999999999 2024-09-20T22:57:54.735 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.735 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.735 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for peered", 2024-09-20T22:57:54.735 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:54:16.547801+0000", 2024-09-20T22:57:54.735 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3158e-05 2024-09-20T22:57:54.735 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.736 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.736 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:57:54.736 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:54:17.337429+0000", 2024-09-20T22:57:54.736 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.78962777299999998 2024-09-20T22:57:54.736 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.736 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.736 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:57:54.736 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:54:17.337522+0000", 2024-09-20T22:57:54.736 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.3556999999999999e-05 2024-09-20T22:57:54.736 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.736 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.736 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:57:54.736 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:54:17.337608+0000", 2024-09-20T22:57:54.736 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.6296000000000002e-05 2024-09-20T22:57:54.736 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:57:54.737 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:57:54.737 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:57:54.737 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.737 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.737 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:7834 7.c e786/94 7:3f0fe232:::smithi08477203-4:head v 786'2534, mlcod=785'2528)", 2024-09-20T22:57:54.737 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:54:26.332738+0000", 2024-09-20T22:57:54.737 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 208.36341768, 2024-09-20T22:57:54.737 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.010142463000000001, 2024-09-20T22:57:54.737 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:57:54.737 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:57:54.737 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:57:54.737 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:57:54.737 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.737 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:57:54.737 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:54:26.332738+0000", 2024-09-20T22:57:54.738 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:57:54.738 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.738 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.738 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:57:54.738 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:54:26.332738+0000", 2024-09-20T22:57:54.738 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:57:54.738 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.738 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.738 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:57:54.738 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:54:26.332739+0000", 2024-09-20T22:57:54.738 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2130000000000001e-06 2024-09-20T22:57:54.738 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.738 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.738 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:57:54.739 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:54:26.335206+0000", 2024-09-20T22:57:54.739 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0024668260000000001 2024-09-20T22:57:54.739 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.739 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.739 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:57:54.739 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:54:26.335237+0000", 2024-09-20T22:57:54.739 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.1106999999999999e-05 2024-09-20T22:57:54.739 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.739 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.739 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:57:54.739 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:54:26.335252+0000", 2024-09-20T22:57:54.739 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.5288e-05 2024-09-20T22:57:54.739 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.739 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.739 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:57:54.740 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:54:26.335348+0000", 2024-09-20T22:57:54.740 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.5351000000000002e-05 2024-09-20T22:57:54.740 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.740 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.740 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:57:54.740 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:54:26.335379+0000", 2024-09-20T22:57:54.740 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.1807000000000002e-05 2024-09-20T22:57:54.740 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.740 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.740 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:57:54.740 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:54:26.342798+0000", 2024-09-20T22:57:54.740 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0074187660000000002 2024-09-20T22:57:54.740 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.740 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.740 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:57:54.740 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:54:26.342880+0000", 2024-09-20T22:57:54.741 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.2105000000000001e-05 2024-09-20T22:57:54.741 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:57:54.741 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:57:54.741 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:57:54.741 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.741 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.741 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:8165 7.f e824/391 7:fa55e5ae:::smithi08477203-7:head v 824'927, mlcod=819'923)", 2024-09-20T22:57:54.741 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:55:04.351014+0000", 2024-09-20T22:57:54.741 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 170.34514151400001, 2024-09-20T22:57:54.741 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0096140429999999992, 2024-09-20T22:57:54.741 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:57:54.741 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:57:54.741 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:57:54.741 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:57:54.741 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.741 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:57:54.741 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:04.351014+0000", 2024-09-20T22:57:54.742 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:57:54.742 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.742 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.742 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:57:54.742 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:04.351014+0000", 2024-09-20T22:57:54.742 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:57:54.742 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.742 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.742 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:57:54.742 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:04.351014+0000", 2024-09-20T22:57:54.742 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.9999999999999999e-07 2024-09-20T22:57:54.742 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.742 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.742 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:57:54.742 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:04.354512+0000", 2024-09-20T22:57:54.742 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0034980990000000002 2024-09-20T22:57:54.743 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.743 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.743 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:57:54.743 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:04.354544+0000", 2024-09-20T22:57:54.743 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.1294999999999998e-05 2024-09-20T22:57:54.743 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.743 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.743 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:57:54.743 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:04.354557+0000", 2024-09-20T22:57:54.743 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.3825999999999999e-05 2024-09-20T22:57:54.743 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.743 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.743 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:57:54.743 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:04.354685+0000", 2024-09-20T22:57:54.743 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00012776000000000001 2024-09-20T22:57:54.744 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.744 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.744 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:57:54.744 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:04.354717+0000", 2024-09-20T22:57:54.744 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.2054000000000002e-05 2024-09-20T22:57:54.744 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.744 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.744 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:57:54.744 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:04.360567+0000", 2024-09-20T22:57:54.744 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0058492709999999996 2024-09-20T22:57:54.744 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.744 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.744 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:57:54.744 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:04.360628+0000", 2024-09-20T22:57:54.744 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.1538000000000006e-05 2024-09-20T22:57:54.744 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:57:54.745 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:57:54.745 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:57:54.745 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.745 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.745 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:8189 7.c e825/94 7:318c7512:::smithi08477203-44:head v 825'2641, mlcod=825'2638)", 2024-09-20T22:57:54.745 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:55:05.581130+0000", 2024-09-20T22:57:54.745 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 169.11502573999999, 2024-09-20T22:57:54.745 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0077411670000000002, 2024-09-20T22:57:54.745 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:57:54.745 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:57:54.745 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:57:54.745 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:57:54.745 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.745 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:57:54.745 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:05.581130+0000", 2024-09-20T22:57:54.745 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:57:54.745 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.746 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.746 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:57:54.746 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:05.581130+0000", 2024-09-20T22:57:54.746 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:57:54.746 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.746 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:57:54.747 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.747 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:57:54.748 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:05.581133+0000", 2024-09-20T22:57:54.748 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.2040000000000002e-06 2024-09-20T22:57:54.748 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.748 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.748 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:57:54.748 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:05.582999+0000", 2024-09-20T22:57:54.748 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0018663919999999999 2024-09-20T22:57:54.748 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.748 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.748 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:57:54.748 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:05.583026+0000", 2024-09-20T22:57:54.748 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.6503999999999999e-05 2024-09-20T22:57:54.748 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.749 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.749 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:57:54.749 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:05.583040+0000", 2024-09-20T22:57:54.749 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.4430000000000001e-05 2024-09-20T22:57:54.749 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.749 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.749 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:57:54.749 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:05.583132+0000", 2024-09-20T22:57:54.749 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.1939000000000002e-05 2024-09-20T22:57:54.749 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.750 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.750 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:57:54.750 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:05.583163+0000", 2024-09-20T22:57:54.750 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.0341999999999999e-05 2024-09-20T22:57:54.750 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.750 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.750 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:57:54.750 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:05.588810+0000", 2024-09-20T22:57:54.750 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0056476629999999998 2024-09-20T22:57:54.750 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.750 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.750 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:57:54.750 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:05.588871+0000", 2024-09-20T22:57:54.751 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.0693000000000003e-05 2024-09-20T22:57:54.751 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:57:54.751 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:57:54.751 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:57:54.751 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.751 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.751 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:8294 7.5 7:a160b5d4:::smithi08477203-16:head [read 0~3042133,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e831)", 2024-09-20T22:57:54.751 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:55:11.340475+0000", 2024-09-20T22:57:54.751 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 163.355680893, 2024-09-20T22:57:54.751 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0075077099999999999, 2024-09-20T22:57:54.751 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:57:54.751 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:57:54.751 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:57:54.751 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:57:54.751 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:57:54.752 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:57:54.752 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 8294 2024-09-20T22:57:54.752 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.752 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:57:54.752 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.752 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:57:54.752 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:11.340475+0000", 2024-09-20T22:57:54.752 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:57:54.752 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.752 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.752 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:57:54.752 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:11.340475+0000", 2024-09-20T22:57:54.752 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:57:54.752 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.752 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.753 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:57:54.753 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:11.340475+0000", 2024-09-20T22:57:54.753 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.1099999999999996e-07 2024-09-20T22:57:54.753 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.753 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.753 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:57:54.753 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:11.340479+0000", 2024-09-20T22:57:54.753 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.4170000000000001e-06 2024-09-20T22:57:54.753 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.753 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.753 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:57:54.753 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:11.340484+0000", 2024-09-20T22:57:54.753 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.7570000000000001e-06 2024-09-20T22:57:54.753 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.754 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.754 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:57:54.754 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:11.340489+0000", 2024-09-20T22:57:54.754 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.3819999999999997e-06 2024-09-20T22:57:54.754 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.754 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.754 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:57:54.754 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:11.340800+0000", 2024-09-20T22:57:54.754 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00031126299999999999 2024-09-20T22:57:54.754 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.754 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.754 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for rw locks", 2024-09-20T22:57:54.754 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:11.340908+0000", 2024-09-20T22:57:54.754 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000108088 2024-09-20T22:57:54.754 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.755 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.755 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:57:54.755 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:11.347505+0000", 2024-09-20T22:57:54.755 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0065973050000000004 2024-09-20T22:57:54.755 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.755 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.755 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:57:54.755 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:11.347565+0000", 2024-09-20T22:57:54.755 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 5.9567000000000001e-05 2024-09-20T22:57:54.755 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.755 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.755 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:57:54.755 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:11.347982+0000", 2024-09-20T22:57:54.756 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00041742 2024-09-20T22:57:54.756 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:57:54.756 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:57:54.756 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:57:54.756 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.756 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.756 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:8442 7.c e849/94 7:31d0211f:::smithi08477203-25:head v 849'2704, mlcod=845'2700)", 2024-09-20T22:57:54.756 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:55:29.375191+0000", 2024-09-20T22:57:54.756 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 145.32096468899999, 2024-09-20T22:57:54.756 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.011331713, 2024-09-20T22:57:54.756 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:57:54.756 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:57:54.756 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:57:54.757 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:57:54.757 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.757 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:57:54.757 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:29.375191+0000", 2024-09-20T22:57:54.757 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:57:54.757 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.757 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.757 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:57:54.757 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:29.375191+0000", 2024-09-20T22:57:54.757 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:57:54.757 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.757 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.757 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:57:54.757 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:29.375191+0000", 2024-09-20T22:57:54.757 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.3200000000000001e-07 2024-09-20T22:57:54.757 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.758 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.758 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:57:54.758 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:29.377130+0000", 2024-09-20T22:57:54.758 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0019390779999999999 2024-09-20T22:57:54.758 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.758 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.758 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:57:54.758 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:29.377142+0000", 2024-09-20T22:57:54.758 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1712000000000001e-05 2024-09-20T22:57:54.758 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.758 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.758 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:57:54.758 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:29.377174+0000", 2024-09-20T22:57:54.758 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.1727000000000003e-05 2024-09-20T22:57:54.759 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.759 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.759 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:57:54.759 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:29.377522+0000", 2024-09-20T22:57:54.759 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00034803200000000002 2024-09-20T22:57:54.759 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.759 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.759 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:57:54.759 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:29.377553+0000", 2024-09-20T22:57:54.759 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.1130999999999997e-05 2024-09-20T22:57:54.759 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.759 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.759 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:57:54.760 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:29.386478+0000", 2024-09-20T22:57:54.760 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0089250430000000006 2024-09-20T22:57:54.760 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.760 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.760 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:57:54.760 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:55:29.386523+0000", 2024-09-20T22:57:54.760 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.4758000000000001e-05 2024-09-20T22:57:54.760 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:57:54.760 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:57:54.760 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:57:54.760 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.760 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.760 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:8804 7.c e883/94 7:318c7512:::smithi08477203-44:head v 883'2842, mlcod=881'2835)", 2024-09-20T22:57:54.760 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:56:03.367132+0000", 2024-09-20T22:57:54.760 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 111.329023189, 2024-09-20T22:57:54.761 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0084714460000000005, 2024-09-20T22:57:54.761 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:57:54.761 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:57:54.761 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:57:54.761 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:57:54.761 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.761 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:57:54.761 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:56:03.367132+0000", 2024-09-20T22:57:54.761 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:57:54.761 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.761 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.761 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:57:54.762 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:56:03.367132+0000", 2024-09-20T22:57:54.762 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:57:54.762 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.762 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.762 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:57:54.762 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:56:03.367133+0000", 2024-09-20T22:57:54.762 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 9.3099999999999996e-07 2024-09-20T22:57:54.762 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.762 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.762 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:57:54.762 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:56:03.367141+0000", 2024-09-20T22:57:54.762 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.2540000000000003e-06 2024-09-20T22:57:54.763 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.763 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.763 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:57:54.763 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:56:03.367153+0000", 2024-09-20T22:57:54.763 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2534999999999999e-05 2024-09-20T22:57:54.763 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.763 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.763 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:57:54.763 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:56:03.367160+0000", 2024-09-20T22:57:54.763 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 7.2320000000000004e-06 2024-09-20T22:57:54.763 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.763 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.763 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:57:54.764 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:56:03.367454+0000", 2024-09-20T22:57:54.764 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00029333099999999997 2024-09-20T22:57:54.764 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.764 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.764 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:57:54.764 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:56:03.367480+0000", 2024-09-20T22:57:54.764 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.6781999999999999e-05 2024-09-20T22:57:54.764 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.764 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.764 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:57:54.764 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:56:03.375540+0000", 2024-09-20T22:57:54.764 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0080594840000000004 2024-09-20T22:57:54.764 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.764 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.765 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:57:54.765 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:56:03.375604+0000", 2024-09-20T22:57:54.765 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.3897000000000002e-05 2024-09-20T22:57:54.765 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:57:54.765 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:57:54.765 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:57:54.765 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.765 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.765 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_repop(client.24652.0:8832 7.c e884/94 7:31d0211f:::smithi08477203-25:head v 884'2847, mlcod=883'2844)", 2024-09-20T22:57:54.765 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:56:04.509874+0000", 2024-09-20T22:57:54.765 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 110.186281619, 2024-09-20T22:57:54.765 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00504032, 2024-09-20T22:57:54.765 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:57:54.765 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:57:54.766 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:57:54.766 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:57:54.766 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.766 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:57:54.766 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:56:04.509874+0000", 2024-09-20T22:57:54.766 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:57:54.766 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.766 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.766 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:57:54.766 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:56:04.509874+0000", 2024-09-20T22:57:54.766 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:57:54.766 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.766 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.766 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:57:54.767 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:56:04.509874+0000", 2024-09-20T22:57:54.767 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.7300000000000002e-07 2024-09-20T22:57:54.767 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.767 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.767 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:57:54.767 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:56:04.511431+0000", 2024-09-20T22:57:54.767 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.001556766 2024-09-20T22:57:54.767 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.767 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.767 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:57:54.767 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:56:04.511454+0000", 2024-09-20T22:57:54.767 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.3269e-05 2024-09-20T22:57:54.767 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.768 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.768 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:57:54.768 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:56:04.511467+0000", 2024-09-20T22:57:54.768 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.2877999999999999e-05 2024-09-20T22:57:54.768 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.768 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.768 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:57:54.768 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:56:04.511554+0000", 2024-09-20T22:57:54.768 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.7087000000000002e-05 2024-09-20T22:57:54.768 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.768 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.768 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:57:54.768 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:56:04.511576+0000", 2024-09-20T22:57:54.768 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 2.1999000000000001e-05 2024-09-20T22:57:54.769 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.769 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.769 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:57:54.769 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:56:04.514874+0000", 2024-09-20T22:57:54.769 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.003297538 2024-09-20T22:57:54.769 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.769 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.769 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:57:54.769 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:56:04.514914+0000", 2024-09-20T22:57:54.769 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.0509999999999997e-05 2024-09-20T22:57:54.769 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:57:54.769 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:57:54.770 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:57:54.770 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.770 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.770 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:8839 7.5 7:a160b5d4:::smithi08477203-16:head [copy-from ver 2842] snapc 19d=[19d,19c,19b,19a,199,198,197,196,195,193,191,190,18f,18e,18d,18c,18a,189,185,183,182,180,17e,17d,17b,17a,179,177,175,174,173,172,16f,16e,16c,16b,160,15c,158,14f,14e,14a,13a,12f] ondisk+write+known_if_redirected+supports_pool_eio e885)", 2024-09-20T22:57:54.770 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:56:05.501335+0000", 2024-09-20T22:57:54.770 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 109.194820943, 2024-09-20T22:57:54.770 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.026669249999999999, 2024-09-20T22:57:54.770 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:57:54.770 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:57:54.770 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-20T22:57:54.770 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:57:54.770 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:57:54.771 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:57:54.771 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 8839 2024-09-20T22:57:54.771 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.771 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:57:54.771 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.771 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:57:54.771 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:56:05.501335+0000", 2024-09-20T22:57:54.771 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:57:54.771 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.771 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.771 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:57:54.771 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:56:05.501335+0000", 2024-09-20T22:57:54.771 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:57:54.771 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.772 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.772 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:57:54.772 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:56:05.501336+0000", 2024-09-20T22:57:54.772 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.643e-06 2024-09-20T22:57:54.772 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.772 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.772 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:57:54.772 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:56:05.501343+0000", 2024-09-20T22:57:54.772 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.3470000000000004e-06 2024-09-20T22:57:54.772 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.772 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.772 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:57:54.772 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:56:05.501354+0000", 2024-09-20T22:57:54.772 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 1.1561e-05 2024-09-20T22:57:54.773 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.773 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.773 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:57:54.773 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:56:05.501363+0000", 2024-09-20T22:57:54.773 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.5040000000000002e-06 2024-09-20T22:57:54.773 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.773 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.773 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:57:54.773 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:56:05.501446+0000", 2024-09-20T22:57:54.773 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 8.3673999999999999e-05 2024-09-20T22:57:54.773 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.773 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.773 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:57:54.773 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:56:05.501608+0000", 2024-09-20T22:57:54.774 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.000161399 2024-09-20T22:57:54.774 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.774 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.774 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for subops from 0,2", 2024-09-20T22:57:54.774 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:56:05.512738+0000", 2024-09-20T22:57:54.774 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.011130509 2024-09-20T22:57:54.774 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.774 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.774 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "op_commit", 2024-09-20T22:57:54.774 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:56:05.516390+0000", 2024-09-20T22:57:54.774 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.0036521320000000002 2024-09-20T22:57:54.774 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.774 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.774 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:57:54.774 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:56:05.527650+0000", 2024-09-20T22:57:54.775 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.011259567 2024-09-20T22:57:54.775 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.775 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.775 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "sub_op_commit_rec", 2024-09-20T22:57:54.775 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:56:05.527894+0000", 2024-09-20T22:57:54.775 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00024384499999999999 2024-09-20T22:57:54.775 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.775 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.775 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "commit_sent", 2024-09-20T22:57:54.775 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:56:05.527942+0000", 2024-09-20T22:57:54.775 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 4.8569000000000002e-05 2024-09-20T22:57:54.775 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.775 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.775 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:57:54.775 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:56:05.528004+0000", 2024-09-20T22:57:54.776 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.1500000000000004e-05 2024-09-20T22:57:54.776 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:57:54.776 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:57:54.776 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:57:54.776 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.776 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.776 INFO:teuthology.orchestra.run.smithi154.stdout: "description": "osd_op(client.24652.0:8840 7.5 7:a160b5d4:::smithi08477203-16:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e885)", 2024-09-20T22:57:54.776 INFO:teuthology.orchestra.run.smithi154.stdout: "initiated_at": "2024-09-20T22:56:05.501431+0000", 2024-09-20T22:57:54.776 INFO:teuthology.orchestra.run.smithi154.stdout: "age": 109.194725005, 2024-09-20T22:57:54.776 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.014446565, 2024-09-20T22:57:54.776 INFO:teuthology.orchestra.run.smithi154.stdout: "continuous": false, 2024-09-20T22:57:54.776 INFO:teuthology.orchestra.run.smithi154.stdout: "type_data": { 2024-09-20T22:57:54.776 INFO:teuthology.orchestra.run.smithi154.stdout: "flag_point": "started", 2024-09-20T22:57:54.777 INFO:teuthology.orchestra.run.smithi154.stdout: "client_info": { 2024-09-20T22:57:54.777 INFO:teuthology.orchestra.run.smithi154.stdout: "client": "client.24652", 2024-09-20T22:57:54.777 INFO:teuthology.orchestra.run.smithi154.stdout: "client_addr": "172.21.15.84:0/3101266294", 2024-09-20T22:57:54.777 INFO:teuthology.orchestra.run.smithi154.stdout: "tid": 8840 2024-09-20T22:57:54.777 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.777 INFO:teuthology.orchestra.run.smithi154.stdout: "events": [ 2024-09-20T22:57:54.777 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.777 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "initiated", 2024-09-20T22:57:54.777 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:56:05.501431+0000", 2024-09-20T22:57:54.777 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:57:54.777 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.777 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.777 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "header_read", 2024-09-20T22:57:54.777 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:56:05.501431+0000", 2024-09-20T22:57:54.778 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0 2024-09-20T22:57:54.778 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.778 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.778 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "throttled", 2024-09-20T22:57:54.778 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:56:05.501431+0000", 2024-09-20T22:57:54.778 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.1600000000000001e-07 2024-09-20T22:57:54.778 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.778 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.778 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "all_read", 2024-09-20T22:57:54.778 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:56:05.501434+0000", 2024-09-20T22:57:54.778 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 3.0699999999999998e-06 2024-09-20T22:57:54.778 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.778 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.778 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "dispatched", 2024-09-20T22:57:54.779 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:56:05.501441+0000", 2024-09-20T22:57:54.779 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.7830000000000001e-06 2024-09-20T22:57:54.779 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.779 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.779 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "queued_for_pg", 2024-09-20T22:57:54.779 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:56:05.501447+0000", 2024-09-20T22:57:54.779 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.1630000000000001e-06 2024-09-20T22:57:54.779 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.779 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.779 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:57:54.779 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:56:05.501833+0000", 2024-09-20T22:57:54.779 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00038532200000000001 2024-09-20T22:57:54.779 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.779 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.779 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "waiting for blocked object", 2024-09-20T22:57:54.780 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:56:05.501902+0000", 2024-09-20T22:57:54.780 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 6.9587999999999998e-05 2024-09-20T22:57:54.780 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.780 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.780 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "reached_pg", 2024-09-20T22:57:54.780 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:56:05.515543+0000", 2024-09-20T22:57:54.780 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.013640664 2024-09-20T22:57:54.780 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.780 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.780 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "started", 2024-09-20T22:57:54.780 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:56:05.515736+0000", 2024-09-20T22:57:54.780 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00019273099999999999 2024-09-20T22:57:54.780 INFO:teuthology.orchestra.run.smithi154.stdout: }, 2024-09-20T22:57:54.780 INFO:teuthology.orchestra.run.smithi154.stdout: { 2024-09-20T22:57:54.780 INFO:teuthology.orchestra.run.smithi154.stdout: "event": "done", 2024-09-20T22:57:54.781 INFO:teuthology.orchestra.run.smithi154.stdout: "time": "2024-09-20T22:56:05.515877+0000", 2024-09-20T22:57:54.781 INFO:teuthology.orchestra.run.smithi154.stdout: "duration": 0.00014162800000000001 2024-09-20T22:57:54.781 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:57:54.781 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:57:54.781 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:57:54.781 INFO:teuthology.orchestra.run.smithi154.stdout: } 2024-09-20T22:57:54.781 INFO:teuthology.orchestra.run.smithi154.stdout: ] 2024-09-20T22:57:54.781 INFO:teuthology.orchestra.run.smithi154.stdout:} 2024-09-20T22:57:54.782 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:57:54.842 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:54.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:54.843 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:57:54.943 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:57:54.964 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:54.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:54.965 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:57:54.967 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:54 smithi084 ceph-mon[40024]: pgmap v1391: 168 pgs: 2 active+clean+snaptrim, 166 active+clean; 793 MiB data, 2.3 GiB used, 623 GiB / 626 GiB avail; 18 MiB/s, 0 objects/s recovering 2024-09-20T22:57:54.967 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:54 smithi084 ceph-mon[40024]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-09-20T22:57:54.967 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:54 smithi084 ceph-mon[40024]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 133/1734 objects degraded (7.670%), 20 pgs degraded) 2024-09-20T22:57:54.967 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:54 smithi084 ceph-mon[29774]: pgmap v1391: 168 pgs: 2 active+clean+snaptrim, 166 active+clean; 793 MiB data, 2.3 GiB used, 623 GiB / 626 GiB avail; 18 MiB/s, 0 objects/s recovering 2024-09-20T22:57:54.967 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:54 smithi084 ceph-mon[29774]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-09-20T22:57:54.967 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:54 smithi084 ceph-mon[29774]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 133/1734 objects degraded (7.670%), 20 pgs degraded) 2024-09-20T22:57:55.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:54 smithi154 ceph-mon[33264]: pgmap v1391: 168 pgs: 2 active+clean+snaptrim, 166 active+clean; 793 MiB data, 2.3 GiB used, 623 GiB / 626 GiB avail; 18 MiB/s, 0 objects/s recovering 2024-09-20T22:57:55.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:54 smithi154 ceph-mon[33264]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-09-20T22:57:55.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:54 smithi154 ceph-mon[33264]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 133/1734 objects degraded (7.670%), 20 pgs degraded) 2024-09-20T22:57:55.066 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:57:55.088 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:55.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:55.089 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:57:55.190 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:57:55.212 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:55.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:55.213 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:57:55.314 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:57:55.335 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:55.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:55.336 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:57:55.383 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-20T22:57:55.386 INFO:teuthology.orchestra.run.smithi084.stderr:dumped all 2024-09-20T22:57:55.436 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:57:55.458 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:55.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:55.459 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:57:55.560 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:57:55.581 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:55.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:55.582 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:57:55.584 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-20T22:57:55.683 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:57:55.705 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:55.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:55.706 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:57:55.806 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:57:55.828 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:55.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:55.829 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:57:55.873 INFO:tasks.thrashosds.thrasher:in_osds: [2, 3, 5, 6, 0, 4, 1] out_osds: [7] dead_osds: [] live_osds: [0, 1, 2, 4, 5, 6, 7, 3] 2024-09-20T22:57:55.873 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-20T22:57:55.874 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-09-20T22:57:55.874 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd dump -f json-pretty 2024-09-20T22:57:55.895 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:55 smithi084 ceph-mon[40024]: from='client.17598 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T22:57:55.896 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:55 smithi084 ceph-mon[29774]: from='client.17598 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T22:57:55.929 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:57:55.951 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:55.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:55.953 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:57:56.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:55 smithi154 ceph-mon[33264]: from='client.17598 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T22:57:56.053 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:57:56.075 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:56.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:56.076 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:57:56.126 INFO:teuthology.orchestra.run.smithi084.stdout:{"pg_ready":true,"pg_map":{"version":1391,"stamp":"2024-09-20T22:57:53.899418+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":831728298,"num_objects":553,"num_object_clones":303,"num_object_copies":1659,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":553,"num_whiteouts":10,"num_read":7048,"num_read_kb":5340600,"num_write":4255,"num_write_kb":1374866,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1433,"num_bytes_recovered":696313350,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":27072,"ondisk_log_size":27072,"up":504,"acting":504,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":508,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":2432212,"kb_used_data":1949488,"kb_used_omap":157,"kb_used_meta":482402,"kb_avail":653783852,"statfs":{"total":671965249536,"available":669474664448,"internally_reserved":0,"allocated":1996275712,"data_stored":2497030892,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":161525,"internal_metadata":493979915},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-47737293,"num_objects":-25,"num_object_clones":-24,"num_object_copies":-75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-133,"num_objects_misplaced":-34,"num_objects_unfound":0,"num_objects_dirty":-25,"num_whiteouts":-2,"num_read":-336,"num_read_kb":-374850,"num_write":-248,"num_write_kb":-85216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":86925213,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.532295"},"pg_stats":[{"pgid":"3.1f","version":"86'2","reported_seq":1257,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-20T22:57:52.374457+0000","last_change":"2024-09-20T22:56:47.723346+0000","last_active":"2024-09-20T22:57:52.374457+0000","last_peered":"2024-09-20T22:57:52.374457+0000","last_clean":"2024-09-20T22:57:52.374457+0000","last_became_active":"2024-09-20T22:47:49.222114+0000","last_became_peered":"2024-09-20T22:47:49.222114+0000","last_unstale":"2024-09-20T22:57:52.374457+0000","last_undegraded":"2024-09-20T22:57:52.374457+0000","last_fullsized":"2024-09-20T22:57:52.374457+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'2","last_scrub_stamp":"2024-09-20T22:56:47.723216+0000","last_deep_scrub":"86'2","last_deep_scrub_stamp":"2024-09-20T22:56:47.723216+0000","last_clean_scrub_stamp":"2024-09-20T22:56:47.723216+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:59.566322+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":436,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"625'15","reported_seq":820,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-20T22:57:52.567622+0000","last_change":"2024-09-20T22:56:14.882214+0000","last_active":"2024-09-20T22:57:52.567622+0000","last_peered":"2024-09-20T22:57:52.567622+0000","last_clean":"2024-09-20T22:57:52.567622+0000","last_became_active":"2024-09-20T22:55:17.341556+0000","last_became_peered":"2024-09-20T22:55:17.341556+0000","last_unstale":"2024-09-20T22:57:52.567622+0000","last_undegraded":"2024-09-20T22:57:52.567622+0000","last_fullsized":"2024-09-20T22:57:52.567622+0000","mapping_epoch":836,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":837,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'15","last_scrub_stamp":"2024-09-20T22:56:14.882123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:56:14.882123+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:55.294110+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":454,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-20T22:57:52.377471+0000","last_change":"2024-09-20T22:56:08.363427+0000","last_active":"2024-09-20T22:57:52.377471+0000","last_peered":"2024-09-20T22:57:52.377471+0000","last_clean":"2024-09-20T22:57:52.377471+0000","last_became_active":"2024-09-20T22:56:08.362405+0000","last_became_peered":"2024-09-20T22:56:08.362405+0000","last_unstale":"2024-09-20T22:57:52.377471+0000","last_undegraded":"2024-09-20T22:57:52.377471+0000","last_fullsized":"2024-09-20T22:57:52.377471+0000","mapping_epoch":887,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:55:34.919340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:55:34.919340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":1113,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-20T22:57:52.376038+0000","last_change":"2024-09-20T22:56:22.519967+0000","last_active":"2024-09-20T22:57:52.376038+0000","last_peered":"2024-09-20T22:57:52.376038+0000","last_clean":"2024-09-20T22:57:52.376038+0000","last_became_active":"2024-09-20T22:55:17.342068+0000","last_became_peered":"2024-09-20T22:55:17.342068+0000","last_unstale":"2024-09-20T22:57:52.376038+0000","last_undegraded":"2024-09-20T22:57:52.376038+0000","last_fullsized":"2024-09-20T22:57:52.376038+0000","mapping_epoch":836,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":837,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:56:22.519884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:56:22.519884+0000","last_clean_scrub_stamp":"2024-09-20T22:56:22.519884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:54.751145+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":1790,"reported_epoch":991,"state":"active+clean","last_fresh":"2024-09-20T22:57:51.375969+0000","last_change":"2024-09-20T22:57:49.427504+0000","last_active":"2024-09-20T22:57:51.375969+0000","last_peered":"2024-09-20T22:57:51.375969+0000","last_clean":"2024-09-20T22:57:51.375969+0000","last_became_active":"2024-09-20T22:57:49.427228+0000","last_became_peered":"2024-09-20T22:57:49.427228+0000","last_unstale":"2024-09-20T22:57:51.375969+0000","last_undegraded":"2024-09-20T22:57:51.375969+0000","last_fullsized":"2024-09-20T22:57:51.375969+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:54:15.238344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:54:15.238344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:55.630176+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":1789,"reported_epoch":991,"state":"active+clean","last_fresh":"2024-09-20T22:57:51.375257+0000","last_change":"2024-09-20T22:57:49.409395+0000","last_active":"2024-09-20T22:57:51.375257+0000","last_peered":"2024-09-20T22:57:51.375257+0000","last_clean":"2024-09-20T22:57:51.375257+0000","last_became_active":"2024-09-20T22:57:49.405928+0000","last_became_peered":"2024-09-20T22:57:49.405928+0000","last_unstale":"2024-09-20T22:57:51.375257+0000","last_undegraded":"2024-09-20T22:57:51.375257+0000","last_fullsized":"2024-09-20T22:57:51.375257+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:54:47.427927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:54:47.427927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:55.630282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"625'27","reported_seq":1867,"reported_epoch":991,"state":"active+clean","last_fresh":"2024-09-20T22:57:51.376376+0000","last_change":"2024-09-20T22:57:49.416557+0000","last_active":"2024-09-20T22:57:51.376376+0000","last_peered":"2024-09-20T22:57:51.376376+0000","last_clean":"2024-09-20T22:57:51.376376+0000","last_became_active":"2024-09-20T22:57:49.413932+0000","last_became_peered":"2024-09-20T22:57:49.413932+0000","last_unstale":"2024-09-20T22:57:51.376376+0000","last_undegraded":"2024-09-20T22:57:51.376376+0000","last_fullsized":"2024-09-20T22:57:51.376376+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'27","last_scrub_stamp":"2024-09-20T22:54:45.390169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:54:45.390169+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:55.630250+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":1110,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-20T22:57:52.567735+0000","last_change":"2024-09-20T22:57:13.194616+0000","last_active":"2024-09-20T22:57:52.567735+0000","last_peered":"2024-09-20T22:57:52.567735+0000","last_clean":"2024-09-20T22:57:52.567735+0000","last_became_active":"2024-09-20T22:55:17.341160+0000","last_became_peered":"2024-09-20T22:55:17.341160+0000","last_unstale":"2024-09-20T22:57:52.567735+0000","last_undegraded":"2024-09-20T22:57:52.567735+0000","last_fullsized":"2024-09-20T22:57:52.567735+0000","mapping_epoch":836,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":837,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:57:13.194499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:57:13.194499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:39.408547+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.19","version":"988'2004","reported_seq":6632,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-20T22:57:52.567929+0000","last_change":"2024-09-20T22:57:48.440765+0000","last_active":"2024-09-20T22:57:52.567929+0000","last_peered":"2024-09-20T22:57:52.567929+0000","last_clean":"2024-09-20T22:57:52.567929+0000","last_became_active":"2024-09-20T22:56:58.364978+0000","last_became_peered":"2024-09-20T22:56:58.364978+0000","last_unstale":"2024-09-20T22:57:52.567929+0000","last_undegraded":"2024-09-20T22:57:52.567929+0000","last_fullsized":"2024-09-20T22:57:52.567929+0000","mapping_epoch":937,"log_start":"423'280","ondisk_log_start":"423'280","created":932,"last_epoch_clean":938,"parent":"0.0","parent_split_bits":5,"last_scrub":"942'1889","last_scrub_stamp":"2024-09-20T22:57:04.163707+0000","last_deep_scrub":"590'828","last_deep_scrub_stamp":"2024-09-20T22:51:10.909364+0000","last_clean_scrub_stamp":"2024-09-20T22:57:04.163707+0000","objects_scrubbed":23,"log_size":1724,"log_dups_size":649,"ondisk_log_size":1724,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:27.711921+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.070243691999999996,"stat_sum":{"num_bytes":56543939,"num_objects":23,"num_object_clones":20,"num_object_copies":69,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":19,"num_read_kb":6766,"num_write":21,"num_write_kb":6300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,5],"acting":[6,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"16b","length":"1"},{"start":"173","length":"1"},{"start":"17d","length":"1"},{"start":"180","length":"1"},{"start":"182","length":"3"},{"start":"186","length":"8"},{"start":"190","length":"10"},{"start":"1a1","length":"1"},{"start":"1a3","length":"2"},{"start":"1a6","length":"1"},{"start":"1aa","length":"2"},{"start":"1af","length":"1"},{"start":"1b5","length":"1"},{"start":"1b7","length":"1"},{"start":"1ba","length":"1"},{"start":"1bd","length":"2"},{"start":"1c0","length":"2"},{"start":"1c5","length":"1"},{"start":"1c7","length":"1"}]},{"pgid":"3.1d","version":"0'0","reported_seq":1923,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-20T22:57:52.376736+0000","last_change":"2024-09-20T22:56:20.497450+0000","last_active":"2024-09-20T22:57:52.376736+0000","last_peered":"2024-09-20T22:57:52.376736+0000","last_clean":"2024-09-20T22:57:52.376736+0000","last_became_active":"2024-09-20T22:48:05.242059+0000","last_became_peered":"2024-09-20T22:48:05.242059+0000","last_unstale":"2024-09-20T22:57:52.376736+0000","last_undegraded":"2024-09-20T22:57:52.376736+0000","last_fullsized":"2024-09-20T22:57:52.376736+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:56:20.497377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:46:00.999939+0000","last_clean_scrub_stamp":"2024-09-20T22:56:20.497377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:54.751078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"625'15","reported_seq":679,"reported_epoch":991,"state":"active+clean","last_fresh":"2024-09-20T22:57:51.376759+0000","last_change":"2024-09-20T22:57:49.405919+0000","last_active":"2024-09-20T22:57:51.376759+0000","last_peered":"2024-09-20T22:57:51.376759+0000","last_clean":"2024-09-20T22:57:51.376759+0000","last_became_active":"2024-09-20T22:57:49.403576+0000","last_became_peered":"2024-09-20T22:57:49.403576+0000","last_unstale":"2024-09-20T22:57:51.376759+0000","last_undegraded":"2024-09-20T22:57:51.376759+0000","last_fullsized":"2024-09-20T22:57:51.376759+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'15","last_scrub_stamp":"2024-09-20T22:53:51.486202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:53:51.486202+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:55.630070+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":1921,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-20T22:57:52.376156+0000","last_change":"2024-09-20T22:56:42.536987+0000","last_active":"2024-09-20T22:57:52.376156+0000","last_peered":"2024-09-20T22:57:52.376156+0000","last_clean":"2024-09-20T22:57:52.376156+0000","last_became_active":"2024-09-20T22:56:08.356419+0000","last_became_peered":"2024-09-20T22:56:08.356419+0000","last_unstale":"2024-09-20T22:57:52.376156+0000","last_undegraded":"2024-09-20T22:57:52.376156+0000","last_fullsized":"2024-09-20T22:57:52.376156+0000","mapping_epoch":887,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:56:42.536863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:56:42.536863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:54.751215+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":1271,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-20T22:57:52.372304+0000","last_change":"2024-09-20T22:56:08.391304+0000","last_active":"2024-09-20T22:57:52.372304+0000","last_peered":"2024-09-20T22:57:52.372304+0000","last_clean":"2024-09-20T22:57:52.372304+0000","last_became_active":"2024-09-20T22:56:08.390640+0000","last_became_peered":"2024-09-20T22:56:08.390640+0000","last_unstale":"2024-09-20T22:57:52.372304+0000","last_undegraded":"2024-09-20T22:57:52.372304+0000","last_fullsized":"2024-09-20T22:57:52.372304+0000","mapping_epoch":887,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:55:55.454925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:49:08.664407+0000","last_clean_scrub_stamp":"2024-09-20T22:55:55.454925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:56.680587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.18","version":"988'1502","reported_seq":5917,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-20T22:57:52.371952+0000","last_change":"2024-09-20T22:57:48.386070+0000","last_active":"2024-09-20T22:57:52.371952+0000","last_peered":"2024-09-20T22:57:52.371952+0000","last_clean":"2024-09-20T22:57:52.371952+0000","last_became_active":"2024-09-20T22:56:58.365227+0000","last_became_peered":"2024-09-20T22:56:58.365227+0000","last_unstale":"2024-09-20T22:57:52.371952+0000","last_undegraded":"2024-09-20T22:57:52.371952+0000","last_fullsized":"2024-09-20T22:57:52.371952+0000","mapping_epoch":937,"log_start":"426'268","ondisk_log_start":"426'268","created":932,"last_epoch_clean":938,"parent":"0.0","parent_split_bits":5,"last_scrub":"942'1432","last_scrub_stamp":"2024-09-20T22:57:03.731020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:57:03.731020+0000","objects_scrubbed":17,"log_size":1234,"log_dups_size":445,"ondisk_log_size":1234,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:30.917366+0000","scrub_duration":8,"objects_trimmed":1,"snaptrim_duration":0.019121448999999999,"stat_sum":{"num_bytes":27703213,"num_objects":15,"num_object_clones":13,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":21,"num_read_kb":27807,"num_write":4,"num_write_kb":996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"16b","length":"1"},{"start":"173","length":"1"},{"start":"17d","length":"1"},{"start":"180","length":"1"},{"start":"182","length":"3"},{"start":"186","length":"8"},{"start":"190","length":"10"},{"start":"1a1","length":"1"},{"start":"1a3","length":"2"},{"start":"1a6","length":"1"},{"start":"1aa","length":"2"},{"start":"1af","length":"1"},{"start":"1b5","length":"1"},{"start":"1b7","length":"1"},{"start":"1ba","length":"1"},{"start":"1bd","length":"2"},{"start":"1c0","length":"2"},{"start":"1c5","length":"1"},{"start":"1c7","length":"1"}]},{"pgid":"3.1c","version":"0'0","reported_seq":1935,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-20T22:57:52.376803+0000","last_change":"2024-09-20T22:56:13.490259+0000","last_active":"2024-09-20T22:57:52.376803+0000","last_peered":"2024-09-20T22:57:52.376803+0000","last_clean":"2024-09-20T22:57:52.376803+0000","last_became_active":"2024-09-20T22:54:17.342619+0000","last_became_peered":"2024-09-20T22:54:17.342619+0000","last_unstale":"2024-09-20T22:57:52.376803+0000","last_undegraded":"2024-09-20T22:57:52.376803+0000","last_fullsized":"2024-09-20T22:57:52.376803+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:56:13.490178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:56:13.490178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:54.751007+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"625'9","reported_seq":657,"reported_epoch":991,"state":"active+clean","last_fresh":"2024-09-20T22:57:51.376060+0000","last_change":"2024-09-20T22:57:49.424289+0000","last_active":"2024-09-20T22:57:51.376060+0000","last_peered":"2024-09-20T22:57:51.376060+0000","last_clean":"2024-09-20T22:57:51.376060+0000","last_became_active":"2024-09-20T22:57:49.423498+0000","last_became_peered":"2024-09-20T22:57:49.423498+0000","last_unstale":"2024-09-20T22:57:51.376060+0000","last_undegraded":"2024-09-20T22:57:51.376060+0000","last_fullsized":"2024-09-20T22:57:51.376060+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'9","last_scrub_stamp":"2024-09-20T22:56:24.469428+0000","last_deep_scrub":"181'7","last_deep_scrub_stamp":"2024-09-20T22:48:41.049381+0000","last_clean_scrub_stamp":"2024-09-20T22:56:24.469428+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:55.630750+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":9,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":1712,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-20T22:57:52.377453+0000","last_change":"2024-09-20T22:56:08.357180+0000","last_active":"2024-09-20T22:57:52.377453+0000","last_peered":"2024-09-20T22:57:52.377453+0000","last_clean":"2024-09-20T22:57:52.377453+0000","last_became_active":"2024-09-20T22:56:08.356857+0000","last_became_peered":"2024-09-20T22:56:08.356857+0000","last_unstale":"2024-09-20T22:57:52.377453+0000","last_undegraded":"2024-09-20T22:57:52.377453+0000","last_fullsized":"2024-09-20T22:57:52.377453+0000","mapping_epoch":887,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:55:59.860836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:47:40.361958+0000","last_clean_scrub_stamp":"2024-09-20T22:55:59.860836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":1923,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-20T22:57:52.376235+0000","last_change":"2024-09-20T22:57:49.402679+0000","last_active":"2024-09-20T22:57:52.376235+0000","last_peered":"2024-09-20T22:57:52.376235+0000","last_clean":"2024-09-20T22:57:52.376235+0000","last_became_active":"2024-09-20T22:57:49.402447+0000","last_became_peered":"2024-09-20T22:57:49.402447+0000","last_unstale":"2024-09-20T22:57:52.376235+0000","last_undegraded":"2024-09-20T22:57:52.376235+0000","last_fullsized":"2024-09-20T22:57:52.376235+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:56:14.512566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:56:14.512566+0000","last_clean_scrub_stamp":"2024-09-20T22:56:14.512566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:54.750974+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":337,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-20T22:57:52.568672+0000","last_change":"2024-09-20T22:57:12.227470+0000","last_active":"2024-09-20T22:57:52.568672+0000","last_peered":"2024-09-20T22:57:52.568672+0000","last_clean":"2024-09-20T22:57:52.568672+0000","last_became_active":"2024-09-20T22:55:17.343025+0000","last_became_peered":"2024-09-20T22:55:17.343025+0000","last_unstale":"2024-09-20T22:57:52.568672+0000","last_undegraded":"2024-09-20T22:57:52.568672+0000","last_fullsized":"2024-09-20T22:57:52.568672+0000","mapping_epoch":836,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":837,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:57:12.227392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:57:12.227392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:25.441591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":1265,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-20T22:57:52.374313+0000","last_change":"2024-09-20T22:57:12.715005+0000","last_active":"2024-09-20T22:57:52.374313+0000","last_peered":"2024-09-20T22:57:52.374313+0000","last_clean":"2024-09-20T22:57:52.374313+0000","last_became_active":"2024-09-20T22:56:08.391965+0000","last_became_peered":"2024-09-20T22:56:08.391965+0000","last_unstale":"2024-09-20T22:57:52.374313+0000","last_undegraded":"2024-09-20T22:57:52.374313+0000","last_fullsized":"2024-09-20T22:57:52.374313+0000","mapping_epoch":887,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:57:12.714907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:49:53.933601+0000","last_clean_scrub_stamp":"2024-09-20T22:57:12.714907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:20.512112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"625'34","reported_seq":2025,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-20T22:57:52.373851+0000","last_change":"2024-09-20T22:57:49.430803+0000","last_active":"2024-09-20T22:57:52.373851+0000","last_peered":"2024-09-20T22:57:52.373851+0000","last_clean":"2024-09-20T22:57:52.373851+0000","last_became_active":"2024-09-20T22:57:49.430403+0000","last_became_peered":"2024-09-20T22:57:49.430403+0000","last_unstale":"2024-09-20T22:57:52.373851+0000","last_undegraded":"2024-09-20T22:57:52.373851+0000","last_fullsized":"2024-09-20T22:57:52.373851+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'34","last_scrub_stamp":"2024-09-20T22:55:04.301955+0000","last_deep_scrub":"113'22","last_deep_scrub_stamp":"2024-09-20T22:43:12.191878+0000","last_clean_scrub_stamp":"2024-09-20T22:55:04.301955+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:55.309964+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":456,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-20T22:57:52.378245+0000","last_change":"2024-09-20T22:55:57.217769+0000","last_active":"2024-09-20T22:57:52.378245+0000","last_peered":"2024-09-20T22:57:52.378245+0000","last_clean":"2024-09-20T22:57:52.378245+0000","last_became_active":"2024-09-20T22:54:17.349501+0000","last_became_peered":"2024-09-20T22:54:17.349501+0000","last_unstale":"2024-09-20T22:57:52.378245+0000","last_undegraded":"2024-09-20T22:57:52.378245+0000","last_fullsized":"2024-09-20T22:57:52.378245+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:55:57.217692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:55:57.217692+0000","last_clean_scrub_stamp":"2024-09-20T22:55:57.217692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":1769,"reported_epoch":991,"state":"active+clean","last_fresh":"2024-09-20T22:57:51.377497+0000","last_change":"2024-09-20T22:57:49.423684+0000","last_active":"2024-09-20T22:57:51.377497+0000","last_peered":"2024-09-20T22:57:51.377497+0000","last_clean":"2024-09-20T22:57:51.377497+0000","last_became_active":"2024-09-20T22:57:49.419671+0000","last_became_peered":"2024-09-20T22:57:49.419671+0000","last_unstale":"2024-09-20T22:57:51.377497+0000","last_undegraded":"2024-09-20T22:57:51.377497+0000","last_fullsized":"2024-09-20T22:57:51.377497+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:54:02.131659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:45:25.002199+0000","last_clean_scrub_stamp":"2024-09-20T22:54:02.131659+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:55.630033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":1120,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-20T22:57:52.377345+0000","last_change":"2024-09-20T22:57:37.206101+0000","last_active":"2024-09-20T22:57:52.377345+0000","last_peered":"2024-09-20T22:57:52.377345+0000","last_clean":"2024-09-20T22:57:52.377345+0000","last_became_active":"2024-09-20T22:55:17.346400+0000","last_became_peered":"2024-09-20T22:55:17.346400+0000","last_unstale":"2024-09-20T22:57:52.377345+0000","last_undegraded":"2024-09-20T22:57:52.377345+0000","last_fullsized":"2024-09-20T22:57:52.377345+0000","mapping_epoch":836,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":837,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:57:37.206000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:50:41.885844+0000","last_clean_scrub_stamp":"2024-09-20T22:57:37.206000+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:48.705725+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"625'18","reported_seq":1847,"reported_epoch":991,"state":"active+clean","last_fresh":"2024-09-20T22:57:51.376577+0000","last_change":"2024-09-20T22:57:49.406206+0000","last_active":"2024-09-20T22:57:51.376577+0000","last_peered":"2024-09-20T22:57:51.376577+0000","last_clean":"2024-09-20T22:57:51.376577+0000","last_became_active":"2024-09-20T22:57:49.403366+0000","last_became_peered":"2024-09-20T22:57:49.403366+0000","last_unstale":"2024-09-20T22:57:51.376577+0000","last_undegraded":"2024-09-20T22:57:51.376577+0000","last_fullsized":"2024-09-20T22:57:51.376577+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'18","last_scrub_stamp":"2024-09-20T22:54:18.284289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:54:18.284289+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:55.630213+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":750,"reported_epoch":991,"state":"active+clean","last_fresh":"2024-09-20T22:57:51.376446+0000","last_change":"2024-09-20T22:57:49.431948+0000","last_active":"2024-09-20T22:57:51.376446+0000","last_peered":"2024-09-20T22:57:51.376446+0000","last_clean":"2024-09-20T22:57:51.376446+0000","last_became_active":"2024-09-20T22:57:49.426209+0000","last_became_peered":"2024-09-20T22:57:49.426209+0000","last_unstale":"2024-09-20T22:57:51.376446+0000","last_undegraded":"2024-09-20T22:57:51.376446+0000","last_fullsized":"2024-09-20T22:57:51.376446+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:56:18.447131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:56:18.447131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:55.630887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"154'2","reported_seq":409,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-20T22:57:52.376538+0000","last_change":"2024-09-20T22:57:49.427430+0000","last_active":"2024-09-20T22:57:52.376538+0000","last_peered":"2024-09-20T22:57:52.376538+0000","last_clean":"2024-09-20T22:57:52.376538+0000","last_became_active":"2024-09-20T22:57:49.427136+0000","last_became_peered":"2024-09-20T22:57:49.427136+0000","last_unstale":"2024-09-20T22:57:52.376538+0000","last_undegraded":"2024-09-20T22:57:52.376538+0000","last_fullsized":"2024-09-20T22:57:52.376538+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"154'2","last_scrub_stamp":"2024-09-20T22:55:06.675346+0000","last_deep_scrub":"154'2","last_deep_scrub_stamp":"2024-09-20T22:43:54.134066+0000","last_clean_scrub_stamp":"2024-09-20T22:55:06.675346+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:54.750768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":403,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"894'2","reported_seq":470,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-20T22:57:52.377433+0000","last_change":"2024-09-20T22:57:49.405855+0000","last_active":"2024-09-20T22:57:52.377433+0000","last_peered":"2024-09-20T22:57:52.377433+0000","last_clean":"2024-09-20T22:57:52.377433+0000","last_became_active":"2024-09-20T22:57:49.405458+0000","last_became_peered":"2024-09-20T22:57:49.405458+0000","last_unstale":"2024-09-20T22:57:52.377433+0000","last_undegraded":"2024-09-20T22:57:52.377433+0000","last_fullsized":"2024-09-20T22:57:52.377433+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"894'2","last_scrub_stamp":"2024-09-20T22:56:15.522649+0000","last_deep_scrub":"894'2","last_deep_scrub_stamp":"2024-09-20T22:56:15.522649+0000","last_clean_scrub_stamp":"2024-09-20T22:56:15.522649+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"625'25","reported_seq":1308,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-20T22:57:52.374534+0000","last_change":"2024-09-20T22:57:49.414546+0000","last_active":"2024-09-20T22:57:52.374534+0000","last_peered":"2024-09-20T22:57:52.374534+0000","last_clean":"2024-09-20T22:57:52.374534+0000","last_became_active":"2024-09-20T22:57:49.411398+0000","last_became_peered":"2024-09-20T22:57:49.411398+0000","last_unstale":"2024-09-20T22:57:52.374534+0000","last_undegraded":"2024-09-20T22:57:52.374534+0000","last_fullsized":"2024-09-20T22:57:52.374534+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'25","last_scrub_stamp":"2024-09-20T22:55:26.646181+0000","last_deep_scrub":"151'15","last_deep_scrub_stamp":"2024-09-20T22:43:50.715761+0000","last_clean_scrub_stamp":"2024-09-20T22:55:26.646181+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:56.680290+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"888'12","reported_seq":2166,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-20T22:57:52.569100+0000","last_change":"2024-09-20T22:56:42.107724+0000","last_active":"2024-09-20T22:57:52.569100+0000","last_peered":"2024-09-20T22:57:52.569100+0000","last_clean":"2024-09-20T22:57:52.569100+0000","last_became_active":"2024-09-20T22:56:08.368232+0000","last_became_peered":"2024-09-20T22:56:08.368232+0000","last_unstale":"2024-09-20T22:57:52.569100+0000","last_undegraded":"2024-09-20T22:57:52.569100+0000","last_fullsized":"2024-09-20T22:57:52.569100+0000","mapping_epoch":887,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":0,"last_scrub":"888'12","last_scrub_stamp":"2024-09-20T22:56:42.107635+0000","last_deep_scrub":"661'11","last_deep_scrub_stamp":"2024-09-20T22:52:21.907943+0000","last_clean_scrub_stamp":"2024-09-20T22:56:42.107635+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:09.490301+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.c","version":"989'3221","reported_seq":7909,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-20T22:57:52.568521+0000","last_change":"2024-09-20T22:57:49.743510+0000","last_active":"2024-09-20T22:57:52.568521+0000","last_peered":"2024-09-20T22:57:52.568521+0000","last_clean":"2024-09-20T22:57:52.568521+0000","last_became_active":"2024-09-20T22:57:49.423914+0000","last_became_peered":"2024-09-20T22:57:49.423914+0000","last_unstale":"2024-09-20T22:57:52.568521+0000","last_undegraded":"2024-09-20T22:57:52.568521+0000","last_fullsized":"2024-09-20T22:57:52.568521+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"902'2941","last_scrub_stamp":"2024-09-20T22:56:22.993088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:56:22.993088+0000","objects_scrubbed":36,"log_size":3221,"log_dups_size":0,"ondisk_log_size":3221,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:56.294202+0000","scrub_duration":11,"objects_trimmed":5,"snaptrim_duration":0.079379093999999997,"stat_sum":{"num_bytes":108964875,"num_objects":47,"num_object_clones":40,"num_object_copies":141,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":2,"num_read":1155,"num_read_kb":1186566,"num_write":778,"num_write_kb":308677,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":189,"num_bytes_recovered":87252218,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"16b","length":"1"},{"start":"173","length":"1"},{"start":"17d","length":"1"},{"start":"180","length":"1"},{"start":"182","length":"3"},{"start":"186","length":"8"},{"start":"190","length":"10"},{"start":"1a1","length":"1"},{"start":"1a3","length":"2"},{"start":"1a6","length":"1"},{"start":"1aa","length":"2"},{"start":"1af","length":"1"},{"start":"1b5","length":"1"},{"start":"1b7","length":"1"},{"start":"1ba","length":"1"},{"start":"1bd","length":"2"},{"start":"1c0","length":"2"},{"start":"1c5","length":"1"},{"start":"1c7","length":"1"}]},{"pgid":"1.a","version":"77'128","reported_seq":2180,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-20T22:57:52.375937+0000","last_change":"2024-09-20T22:57:41.688804+0000","last_active":"2024-09-20T22:57:52.375937+0000","last_peered":"2024-09-20T22:57:52.375937+0000","last_clean":"2024-09-20T22:57:52.375937+0000","last_became_active":"2024-09-20T22:42:52.407791+0000","last_became_peered":"2024-09-20T22:42:52.407791+0000","last_unstale":"2024-09-20T22:57:52.375937+0000","last_undegraded":"2024-09-20T22:57:52.375937+0000","last_fullsized":"2024-09-20T22:57:52.375937+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:57:41.688681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:57:41.688681+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:00.430067+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"625'27","reported_seq":481,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-20T22:57:52.378163+0000","last_change":"2024-09-20T22:57:49.415583+0000","last_active":"2024-09-20T22:57:52.378163+0000","last_peered":"2024-09-20T22:57:52.378163+0000","last_clean":"2024-09-20T22:57:52.378163+0000","last_became_active":"2024-09-20T22:57:49.415260+0000","last_became_peered":"2024-09-20T22:57:49.415260+0000","last_unstale":"2024-09-20T22:57:52.378163+0000","last_undegraded":"2024-09-20T22:57:52.378163+0000","last_fullsized":"2024-09-20T22:57:52.378163+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'27","last_scrub_stamp":"2024-09-20T22:55:50.160922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:55:50.160922+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":1806,"reported_epoch":991,"state":"active+clean","last_fresh":"2024-09-20T22:57:51.375331+0000","last_change":"2024-09-20T22:57:49.414013+0000","last_active":"2024-09-20T22:57:51.375331+0000","last_peered":"2024-09-20T22:57:51.375331+0000","last_clean":"2024-09-20T22:57:51.375331+0000","last_became_active":"2024-09-20T22:57:49.407789+0000","last_became_peered":"2024-09-20T22:57:49.407789+0000","last_unstale":"2024-09-20T22:57:51.375331+0000","last_undegraded":"2024-09-20T22:57:51.375331+0000","last_fullsized":"2024-09-20T22:57:51.375331+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:55:09.369892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:49:12.070675+0000","last_clean_scrub_stamp":"2024-09-20T22:55:09.369892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:55.630423+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"838'13","reported_seq":979,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-20T22:57:52.376686+0000","last_change":"2024-09-20T22:56:46.487242+0000","last_active":"2024-09-20T22:57:52.376686+0000","last_peered":"2024-09-20T22:57:52.376686+0000","last_clean":"2024-09-20T22:57:52.376686+0000","last_became_active":"2024-09-20T22:55:17.343505+0000","last_became_peered":"2024-09-20T22:55:17.343505+0000","last_unstale":"2024-09-20T22:57:52.376686+0000","last_undegraded":"2024-09-20T22:57:52.376686+0000","last_fullsized":"2024-09-20T22:57:52.376686+0000","mapping_epoch":836,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":837,"parent":"0.0","parent_split_bits":0,"last_scrub":"838'13","last_scrub_stamp":"2024-09-20T22:56:46.487164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:56:46.487164+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:02.187115+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":1924,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-20T22:57:52.376522+0000","last_change":"2024-09-20T22:56:41.547004+0000","last_active":"2024-09-20T22:57:52.376522+0000","last_peered":"2024-09-20T22:57:52.376522+0000","last_clean":"2024-09-20T22:57:52.376522+0000","last_became_active":"2024-09-20T22:54:17.340327+0000","last_became_peered":"2024-09-20T22:54:17.340327+0000","last_unstale":"2024-09-20T22:57:52.376522+0000","last_undegraded":"2024-09-20T22:57:52.376522+0000","last_fullsized":"2024-09-20T22:57:52.376522+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:56:41.546923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:56:41.546923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:04.615941+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3","version":"931'475","reported_seq":4670,"reported_epoch":991,"state":"active+clean","last_fresh":"2024-09-20T22:57:51.377318+0000","last_change":"2024-09-20T22:57:49.471606+0000","last_active":"2024-09-20T22:57:51.377318+0000","last_peered":"2024-09-20T22:57:51.377318+0000","last_clean":"2024-09-20T22:57:51.377318+0000","last_became_active":"2024-09-20T22:57:49.410503+0000","last_became_peered":"2024-09-20T22:57:49.410503+0000","last_unstale":"2024-09-20T22:57:51.377318+0000","last_undegraded":"2024-09-20T22:57:51.377318+0000","last_fullsized":"2024-09-20T22:57:51.377318+0000","mapping_epoch":988,"log_start":"456'129","ondisk_log_start":"456'129","created":94,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"849'426","last_scrub_stamp":"2024-09-20T22:55:29.475387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:55:29.475387+0000","objects_scrubbed":9,"log_size":346,"log_dups_size":129,"ondisk_log_size":346,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:55.629698+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.059598702000000003,"stat_sum":{"num_bytes":9034874,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":69,"num_read_kb":85168,"num_write":54,"num_write_kb":17420,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":10098178,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"16b","length":"1"},{"start":"173","length":"1"},{"start":"17d","length":"1"},{"start":"180","length":"1"},{"start":"182","length":"3"},{"start":"186","length":"8"},{"start":"190","length":"10"},{"start":"1a1","length":"1"},{"start":"1a3","length":"2"},{"start":"1a6","length":"1"},{"start":"1aa","length":"2"},{"start":"1af","length":"1"},{"start":"1b5","length":"1"},{"start":"1b7","length":"1"},{"start":"1ba","length":"1"},{"start":"1bd","length":"2"},{"start":"1c0","length":"2"},{"start":"1c5","length":"1"},{"start":"1c7","length":"1"}]},{"pgid":"1.5","version":"77'128","reported_seq":1196,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-20T22:57:52.373542+0000","last_change":"2024-09-20T22:57:14.262350+0000","last_active":"2024-09-20T22:57:52.373542+0000","last_peered":"2024-09-20T22:57:52.373542+0000","last_clean":"2024-09-20T22:57:52.373542+0000","last_became_active":"2024-09-20T22:55:17.345460+0000","last_became_peered":"2024-09-20T22:55:17.345460+0000","last_unstale":"2024-09-20T22:57:52.373542+0000","last_undegraded":"2024-09-20T22:57:52.373542+0000","last_fullsized":"2024-09-20T22:57:52.373542+0000","mapping_epoch":836,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":837,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:57:14.262265+0000","last_deep_scrub":"77'128","last_deep_scrub_stamp":"2024-09-20T22:57:14.262265+0000","last_clean_scrub_stamp":"2024-09-20T22:57:14.262265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:39.904856+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"625'30","reported_seq":1907,"reported_epoch":991,"state":"active+clean","last_fresh":"2024-09-20T22:57:51.375228+0000","last_change":"2024-09-20T22:57:49.414226+0000","last_active":"2024-09-20T22:57:51.375228+0000","last_peered":"2024-09-20T22:57:51.375228+0000","last_clean":"2024-09-20T22:57:51.375228+0000","last_became_active":"2024-09-20T22:57:49.408104+0000","last_became_peered":"2024-09-20T22:57:49.408104+0000","last_unstale":"2024-09-20T22:57:51.375228+0000","last_undegraded":"2024-09-20T22:57:51.375228+0000","last_fullsized":"2024-09-20T22:57:51.375228+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'30","last_scrub_stamp":"2024-09-20T22:53:48.346341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:53:48.346341+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:55.630000+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":39,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":1827,"reported_epoch":991,"state":"active+clean","last_fresh":"2024-09-20T22:57:51.378834+0000","last_change":"2024-09-20T22:57:49.433634+0000","last_active":"2024-09-20T22:57:51.378834+0000","last_peered":"2024-09-20T22:57:51.378834+0000","last_clean":"2024-09-20T22:57:51.378834+0000","last_became_active":"2024-09-20T22:57:49.432799+0000","last_became_peered":"2024-09-20T22:57:49.432799+0000","last_unstale":"2024-09-20T22:57:51.378834+0000","last_undegraded":"2024-09-20T22:57:51.378834+0000","last_fullsized":"2024-09-20T22:57:51.378834+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:56:15.465230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:56:15.465230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:55.630784+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":743,"reported_epoch":991,"state":"active+clean","last_fresh":"2024-09-20T22:57:51.375985+0000","last_change":"2024-09-20T22:57:49.416335+0000","last_active":"2024-09-20T22:57:51.375985+0000","last_peered":"2024-09-20T22:57:51.375985+0000","last_clean":"2024-09-20T22:57:51.375985+0000","last_became_active":"2024-09-20T22:57:49.413482+0000","last_became_peered":"2024-09-20T22:57:49.413482+0000","last_unstale":"2024-09-20T22:57:51.375985+0000","last_undegraded":"2024-09-20T22:57:51.375985+0000","last_fullsized":"2024-09-20T22:57:51.375985+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:56:47.432483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:56:47.432483+0000","last_clean_scrub_stamp":"2024-09-20T22:56:47.432483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:59.904016+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":692,"reported_epoch":991,"state":"active+clean","last_fresh":"2024-09-20T22:57:51.376846+0000","last_change":"2024-09-20T22:57:49.404772+0000","last_active":"2024-09-20T22:57:51.376846+0000","last_peered":"2024-09-20T22:57:51.376846+0000","last_clean":"2024-09-20T22:57:51.376846+0000","last_became_active":"2024-09-20T22:57:49.402383+0000","last_became_peered":"2024-09-20T22:57:49.402383+0000","last_unstale":"2024-09-20T22:57:51.376846+0000","last_undegraded":"2024-09-20T22:57:51.376846+0000","last_fullsized":"2024-09-20T22:57:51.376846+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:55:48.456896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:53.036601+0000","last_clean_scrub_stamp":"2024-09-20T22:55:48.456896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:55.630574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2","version":"989'1026","reported_seq":4760,"reported_epoch":991,"state":"active+clean","last_fresh":"2024-09-20T22:57:51.377358+0000","last_change":"2024-09-20T22:57:49.607113+0000","last_active":"2024-09-20T22:57:51.377358+0000","last_peered":"2024-09-20T22:57:51.377358+0000","last_clean":"2024-09-20T22:57:51.377358+0000","last_became_active":"2024-09-20T22:57:49.400698+0000","last_became_peered":"2024-09-20T22:57:49.400698+0000","last_unstale":"2024-09-20T22:57:51.377358+0000","last_undegraded":"2024-09-20T22:57:51.377358+0000","last_fullsized":"2024-09-20T22:57:51.377358+0000","mapping_epoch":988,"log_start":"471'464","ondisk_log_start":"471'464","created":94,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"282'120","last_scrub_stamp":"2024-09-20T22:46:22.099642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:46:22.099642+0000","objects_scrubbed":1,"log_size":562,"log_dups_size":610,"ondisk_log_size":562,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:55.629735+0000","scrub_duration":1,"objects_trimmed":1,"snaptrim_duration":0.100157336,"stat_sum":{"num_bytes":28403903,"num_objects":10,"num_object_clones":8,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":270,"num_read_kb":283312,"num_write":170,"num_write_kb":67181,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":81163953,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"16b","length":"1"},{"start":"173","length":"1"},{"start":"17d","length":"1"},{"start":"180","length":"1"},{"start":"182","length":"3"},{"start":"186","length":"8"},{"start":"190","length":"10"},{"start":"1a1","length":"1"},{"start":"1a3","length":"2"},{"start":"1a6","length":"1"},{"start":"1aa","length":"2"},{"start":"1af","length":"1"},{"start":"1b5","length":"1"},{"start":"1b7","length":"1"},{"start":"1ba","length":"1"},{"start":"1bd","length":"2"},{"start":"1c0","length":"2"},{"start":"1c5","length":"1"},{"start":"1c7","length":"1"}]},{"pgid":"1.4","version":"855'129","reported_seq":2114,"reported_epoch":991,"state":"active+clean","last_fresh":"2024-09-20T22:57:51.376922+0000","last_change":"2024-09-20T22:57:49.426337+0000","last_active":"2024-09-20T22:57:51.376922+0000","last_peered":"2024-09-20T22:57:51.376922+0000","last_clean":"2024-09-20T22:57:51.376922+0000","last_became_active":"2024-09-20T22:57:49.423690+0000","last_became_peered":"2024-09-20T22:57:49.423690+0000","last_unstale":"2024-09-20T22:57:51.376922+0000","last_undegraded":"2024-09-20T22:57:51.376922+0000","last_fullsized":"2024-09-20T22:57:51.376922+0000","mapping_epoch":988,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":3,"last_scrub":"855'129","last_scrub_stamp":"2024-09-20T22:55:35.536654+0000","last_deep_scrub":"855'129","last_deep_scrub_stamp":"2024-09-20T22:55:35.536654+0000","last_clean_scrub_stamp":"2024-09-20T22:55:35.536654+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":59,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:55.630461+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"625'28","reported_seq":825,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-20T22:57:52.376324+0000","last_change":"2024-09-20T22:57:05.659881+0000","last_active":"2024-09-20T22:57:52.376324+0000","last_peered":"2024-09-20T22:57:52.376324+0000","last_clean":"2024-09-20T22:57:52.376324+0000","last_became_active":"2024-09-20T22:55:17.344611+0000","last_became_peered":"2024-09-20T22:55:17.344611+0000","last_unstale":"2024-09-20T22:57:52.376324+0000","last_undegraded":"2024-09-20T22:57:52.376324+0000","last_fullsized":"2024-09-20T22:57:52.376324+0000","mapping_epoch":836,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":837,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'28","last_scrub_stamp":"2024-09-20T22:57:05.659800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:57:05.659800+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:10.175018+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":40,"num_read_kb":33,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.6","version":"855'2","reported_seq":1288,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-20T22:57:52.374386+0000","last_change":"2024-09-20T22:55:35.573049+0000","last_active":"2024-09-20T22:57:52.374386+0000","last_peered":"2024-09-20T22:57:52.374386+0000","last_clean":"2024-09-20T22:57:52.374386+0000","last_became_active":"2024-09-20T22:54:17.340423+0000","last_became_peered":"2024-09-20T22:54:17.340423+0000","last_unstale":"2024-09-20T22:57:52.374386+0000","last_undegraded":"2024-09-20T22:57:52.374386+0000","last_fullsized":"2024-09-20T22:57:52.374386+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"855'2","last_scrub_stamp":"2024-09-20T22:55:35.572977+0000","last_deep_scrub":"855'2","last_deep_scrub_stamp":"2024-09-20T22:55:35.572977+0000","last_clean_scrub_stamp":"2024-09-20T22:55:35.572977+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:56.680503+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":230,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"989'18","reported_seq":1998,"reported_epoch":991,"state":"active+clean","last_fresh":"2024-09-20T22:57:51.375522+0000","last_change":"2024-09-20T22:57:49.413392+0000","last_active":"2024-09-20T22:57:51.375522+0000","last_peered":"2024-09-20T22:57:51.375522+0000","last_clean":"2024-09-20T22:57:51.375522+0000","last_became_active":"2024-09-20T22:57:49.401445+0000","last_became_peered":"2024-09-20T22:57:49.401445+0000","last_unstale":"2024-09-20T22:57:51.375522+0000","last_undegraded":"2024-09-20T22:57:51.375522+0000","last_fullsized":"2024-09-20T22:57:51.375522+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"778'16","last_scrub_stamp":"2024-09-20T22:55:10.378046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:55:10.378046+0000","objects_scrubbed":1,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:55.630353+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":225,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-20T22:57:52.568398+0000","last_change":"2024-09-20T22:57:10.158684+0000","last_active":"2024-09-20T22:57:52.568398+0000","last_peered":"2024-09-20T22:57:52.568398+0000","last_clean":"2024-09-20T22:57:52.568398+0000","last_became_active":"2024-09-20T22:56:08.367182+0000","last_became_peered":"2024-09-20T22:56:08.367182+0000","last_unstale":"2024-09-20T22:57:52.568398+0000","last_undegraded":"2024-09-20T22:57:52.568398+0000","last_fullsized":"2024-09-20T22:57:52.568398+0000","mapping_epoch":887,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:57:10.158581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:57:10.158581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:32.505201+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1","version":"989'621","reported_seq":4904,"reported_epoch":991,"state":"active+clean","last_fresh":"2024-09-20T22:57:51.376616+0000","last_change":"2024-09-20T22:57:49.713215+0000","last_active":"2024-09-20T22:57:51.376616+0000","last_peered":"2024-09-20T22:57:51.376616+0000","last_clean":"2024-09-20T22:57:51.376616+0000","last_became_active":"2024-09-20T22:57:49.404952+0000","last_became_peered":"2024-09-20T22:57:49.404952+0000","last_unstale":"2024-09-20T22:57:51.376616+0000","last_undegraded":"2024-09-20T22:57:51.376616+0000","last_fullsized":"2024-09-20T22:57:51.376616+0000","mapping_epoch":988,"log_start":"474'259","ondisk_log_start":"474'259","created":94,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"794'485","last_scrub_stamp":"2024-09-20T22:54:38.213249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:54:38.213249+0000","objects_scrubbed":8,"log_size":362,"log_dups_size":259,"ondisk_log_size":362,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:55.629774+0000","scrub_duration":3,"objects_trimmed":3,"snaptrim_duration":0.062553687999999996,"stat_sum":{"num_bytes":19838211,"num_objects":8,"num_object_clones":7,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":95,"num_read_kb":100306,"num_write":82,"num_write_kb":43338,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":19945014,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"16b","length":"1"},{"start":"173","length":"1"},{"start":"17d","length":"1"},{"start":"180","length":"1"},{"start":"182","length":"3"},{"start":"186","length":"8"},{"start":"190","length":"10"},{"start":"1a1","length":"1"},{"start":"1a3","length":"2"},{"start":"1a6","length":"1"},{"start":"1aa","length":"2"},{"start":"1af","length":"1"},{"start":"1b5","length":"1"},{"start":"1b7","length":"1"},{"start":"1ba","length":"1"},{"start":"1bd","length":"2"},{"start":"1c0","length":"2"},{"start":"1c5","length":"1"},{"start":"1c7","length":"1"}]},{"pgid":"1.7","version":"77'128","reported_seq":2239,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-20T22:57:52.568928+0000","last_change":"2024-09-20T22:57:49.411121+0000","last_active":"2024-09-20T22:57:52.568928+0000","last_peered":"2024-09-20T22:57:52.568928+0000","last_clean":"2024-09-20T22:57:52.568928+0000","last_became_active":"2024-09-20T22:57:49.410770+0000","last_became_peered":"2024-09-20T22:57:49.410770+0000","last_unstale":"2024-09-20T22:57:52.568928+0000","last_undegraded":"2024-09-20T22:57:52.568928+0000","last_fullsized":"2024-09-20T22:57:52.568928+0000","mapping_epoch":988,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:55:50.928296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:55:50.928296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:55.294002+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"904'25","reported_seq":481,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-20T22:57:52.377666+0000","last_change":"2024-09-20T22:57:33.088721+0000","last_active":"2024-09-20T22:57:52.377666+0000","last_peered":"2024-09-20T22:57:52.377666+0000","last_clean":"2024-09-20T22:57:52.377666+0000","last_became_active":"2024-09-20T22:54:17.349113+0000","last_became_peered":"2024-09-20T22:54:17.349113+0000","last_unstale":"2024-09-20T22:57:52.377666+0000","last_undegraded":"2024-09-20T22:57:52.377666+0000","last_fullsized":"2024-09-20T22:57:52.377666+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"904'25","last_scrub_stamp":"2024-09-20T22:57:33.088629+0000","last_deep_scrub":"904'25","last_deep_scrub_stamp":"2024-09-20T22:56:24.575150+0000","last_clean_scrub_stamp":"2024-09-20T22:57:33.088629+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:44.765537+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":28,"num_read_kb":23,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":1921,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-20T22:57:52.376829+0000","last_change":"2024-09-20T22:57:49.396609+0000","last_active":"2024-09-20T22:57:52.376829+0000","last_peered":"2024-09-20T22:57:52.376829+0000","last_clean":"2024-09-20T22:57:52.376829+0000","last_became_active":"2024-09-20T22:57:49.396185+0000","last_became_peered":"2024-09-20T22:57:49.396185+0000","last_unstale":"2024-09-20T22:57:52.376829+0000","last_undegraded":"2024-09-20T22:57:52.376829+0000","last_fullsized":"2024-09-20T22:57:52.376829+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:55:53.487366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:55:53.487366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:54.750942+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"733'12","reported_seq":1380,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-20T22:57:52.374261+0000","last_change":"2024-09-20T22:55:50.416762+0000","last_active":"2024-09-20T22:57:52.374261+0000","last_peered":"2024-09-20T22:57:52.374261+0000","last_clean":"2024-09-20T22:57:52.374261+0000","last_became_active":"2024-09-20T22:47:49.227264+0000","last_became_peered":"2024-09-20T22:47:49.227264+0000","last_unstale":"2024-09-20T22:57:52.374261+0000","last_undegraded":"2024-09-20T22:57:52.374261+0000","last_fullsized":"2024-09-20T22:57:52.374261+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"733'12","last_scrub_stamp":"2024-09-20T22:55:50.416689+0000","last_deep_scrub":"733'12","last_deep_scrub_stamp":"2024-09-20T22:53:33.928499+0000","last_clean_scrub_stamp":"2024-09-20T22:55:50.416689+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:56.680533+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":1263,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-20T22:57:52.372106+0000","last_change":"2024-09-20T22:57:49.416890+0000","last_active":"2024-09-20T22:57:52.372106+0000","last_peered":"2024-09-20T22:57:52.372106+0000","last_clean":"2024-09-20T22:57:52.372106+0000","last_became_active":"2024-09-20T22:57:49.416550+0000","last_became_peered":"2024-09-20T22:57:49.416550+0000","last_unstale":"2024-09-20T22:57:52.372106+0000","last_undegraded":"2024-09-20T22:57:52.372106+0000","last_fullsized":"2024-09-20T22:57:52.372106+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:55:57.400645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:55:57.400645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:56.680560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.0","version":"992'1036","reported_seq":5532,"reported_epoch":992,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T22:57:53.542845+0000","last_change":"2024-09-20T22:57:53.542845+0000","last_active":"2024-09-20T22:57:53.542845+0000","last_peered":"2024-09-20T22:57:53.542845+0000","last_clean":"2024-09-20T22:57:53.542845+0000","last_became_active":"2024-09-20T22:57:49.412781+0000","last_became_peered":"2024-09-20T22:57:49.412781+0000","last_unstale":"2024-09-20T22:57:53.542845+0000","last_undegraded":"2024-09-20T22:57:53.542845+0000","last_fullsized":"2024-09-20T22:57:53.542845+0000","mapping_epoch":988,"log_start":"480'253","ondisk_log_start":"480'253","created":94,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"823'833","last_scrub_stamp":"2024-09-20T22:55:07.264241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:55:07.264241+0000","objects_scrubbed":25,"log_size":783,"log_dups_size":416,"ondisk_log_size":783,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":24,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:55.714853+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":1.1668099919999999,"stat_sum":{"num_bytes":26841138,"num_objects":13,"num_object_clones":12,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":1,"num_read":173,"num_read_kb":171902,"num_write":148,"num_write_kb":53734,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":64008579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"16b","length":"1"},{"start":"173","length":"1"},{"start":"17d","length":"1"},{"start":"182","length":"3"},{"start":"186","length":"7"},{"start":"191","length":"2"},{"start":"194","length":"2"},{"start":"198","length":"2"},{"start":"19b","length":"1"},{"start":"19e","length":"2"},{"start":"1a1","length":"1"}]},{"pgid":"1.6","version":"77'128","reported_seq":2246,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-20T22:57:52.568773+0000","last_change":"2024-09-20T22:57:49.407065+0000","last_active":"2024-09-20T22:57:52.568773+0000","last_peered":"2024-09-20T22:57:52.568773+0000","last_clean":"2024-09-20T22:57:52.568773+0000","last_became_active":"2024-09-20T22:57:49.406637+0000","last_became_peered":"2024-09-20T22:57:49.406637+0000","last_unstale":"2024-09-20T22:57:52.568773+0000","last_undegraded":"2024-09-20T22:57:52.568773+0000","last_fullsized":"2024-09-20T22:57:52.568773+0000","mapping_epoch":988,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:56:12.929842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:56:12.929842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:55.294049+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"625'37","reported_seq":1318,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-20T22:57:52.373927+0000","last_change":"2024-09-20T22:56:48.676919+0000","last_active":"2024-09-20T22:57:52.373927+0000","last_peered":"2024-09-20T22:57:52.373927+0000","last_clean":"2024-09-20T22:57:52.373927+0000","last_became_active":"2024-09-20T22:56:08.367015+0000","last_became_peered":"2024-09-20T22:56:08.367015+0000","last_unstale":"2024-09-20T22:57:52.373927+0000","last_undegraded":"2024-09-20T22:57:52.373927+0000","last_fullsized":"2024-09-20T22:57:52.373927+0000","mapping_epoch":887,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'37","last_scrub_stamp":"2024-09-20T22:56:48.676788+0000","last_deep_scrub":"515'27","last_deep_scrub_stamp":"2024-09-20T22:49:55.964153+0000","last_clean_scrub_stamp":"2024-09-20T22:56:48.676788+0000","objects_scrubbed":8,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:06.918757+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":53,"num_read_kb":39,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":456,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-20T22:57:52.378218+0000","last_change":"2024-09-20T22:57:35.151377+0000","last_active":"2024-09-20T22:57:52.378218+0000","last_peered":"2024-09-20T22:57:52.378218+0000","last_clean":"2024-09-20T22:57:52.378218+0000","last_became_active":"2024-09-20T22:54:17.341248+0000","last_became_peered":"2024-09-20T22:54:17.341248+0000","last_unstale":"2024-09-20T22:57:52.378218+0000","last_undegraded":"2024-09-20T22:57:52.378218+0000","last_fullsized":"2024-09-20T22:57:52.378218+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:57:35.151279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:48:23.320031+0000","last_clean_scrub_stamp":"2024-09-20T22:57:35.151279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:39.976778+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":1929,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-20T22:57:52.568023+0000","last_change":"2024-09-20T22:55:53.886918+0000","last_active":"2024-09-20T22:57:52.568023+0000","last_peered":"2024-09-20T22:57:52.568023+0000","last_clean":"2024-09-20T22:57:52.568023+0000","last_became_active":"2024-09-20T22:47:49.214055+0000","last_became_peered":"2024-09-20T22:47:49.214055+0000","last_unstale":"2024-09-20T22:57:52.568023+0000","last_undegraded":"2024-09-20T22:57:52.568023+0000","last_fullsized":"2024-09-20T22:57:52.568023+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:55:53.886833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:55:53.886833+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:55.294033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":448,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-20T22:57:52.377541+0000","last_change":"2024-09-20T22:55:54.151421+0000","last_active":"2024-09-20T22:57:52.377541+0000","last_peered":"2024-09-20T22:57:52.377541+0000","last_clean":"2024-09-20T22:57:52.377541+0000","last_became_active":"2024-09-20T22:54:17.340458+0000","last_became_peered":"2024-09-20T22:54:17.340458+0000","last_unstale":"2024-09-20T22:57:52.377541+0000","last_undegraded":"2024-09-20T22:57:52.377541+0000","last_fullsized":"2024-09-20T22:57:52.377541+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:55:54.151316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:48:41.208853+0000","last_clean_scrub_stamp":"2024-09-20T22:55:54.151316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.7","version":"989'2066","reported_seq":6541,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-20T22:57:52.568999+0000","last_change":"2024-09-20T22:57:49.644293+0000","last_active":"2024-09-20T22:57:52.568999+0000","last_peered":"2024-09-20T22:57:52.568999+0000","last_clean":"2024-09-20T22:57:52.568999+0000","last_became_active":"2024-09-20T22:57:49.410914+0000","last_became_peered":"2024-09-20T22:57:49.410914+0000","last_unstale":"2024-09-20T22:57:52.568999+0000","last_undegraded":"2024-09-20T22:57:52.568999+0000","last_fullsized":"2024-09-20T22:57:52.568999+0000","mapping_epoch":988,"log_start":"436'456","ondisk_log_start":"436'456","created":94,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"879'1788","last_scrub_stamp":"2024-09-20T22:55:59.964830+0000","last_deep_scrub":"879'1788","last_deep_scrub_stamp":"2024-09-20T22:55:59.964830+0000","last_clean_scrub_stamp":"2024-09-20T22:55:59.964830+0000","objects_scrubbed":29,"log_size":1610,"log_dups_size":886,"ondisk_log_size":1610,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:56.294184+0000","scrub_duration":151,"objects_trimmed":1,"snaptrim_duration":0.069128024999999996,"stat_sum":{"num_bytes":49828672,"num_objects":19,"num_object_clones":17,"num_object_copies":57,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":342,"num_read_kb":378528,"num_write":260,"num_write_kb":90390,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":63,"num_bytes_recovered":41725053,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"16b","length":"1"},{"start":"173","length":"1"},{"start":"17d","length":"1"},{"start":"180","length":"1"},{"start":"182","length":"3"},{"start":"186","length":"8"},{"start":"190","length":"10"},{"start":"1a1","length":"1"},{"start":"1a3","length":"2"},{"start":"1a6","length":"1"},{"start":"1aa","length":"2"},{"start":"1af","length":"1"},{"start":"1b5","length":"1"},{"start":"1b7","length":"1"},{"start":"1ba","length":"1"},{"start":"1bd","length":"2"},{"start":"1c0","length":"2"},{"start":"1c5","length":"1"},{"start":"1c7","length":"1"}]},{"pgid":"1.1","version":"77'128","reported_seq":2229,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-20T22:57:52.376014+0000","last_change":"2024-09-20T22:57:37.743124+0000","last_active":"2024-09-20T22:57:52.376014+0000","last_peered":"2024-09-20T22:57:52.376014+0000","last_clean":"2024-09-20T22:57:52.376014+0000","last_became_active":"2024-09-20T22:47:52.208102+0000","last_became_peered":"2024-09-20T22:47:52.208102+0000","last_unstale":"2024-09-20T22:57:52.376014+0000","last_undegraded":"2024-09-20T22:57:52.376014+0000","last_fullsized":"2024-09-20T22:57:52.376014+0000","mapping_epoch":391,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:57:37.743043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:57:37.743043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:54.685166+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"989'55","reported_seq":407,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-20T22:57:52.376473+0000","last_change":"2024-09-20T22:57:49.427243+0000","last_active":"2024-09-20T22:57:52.376473+0000","last_peered":"2024-09-20T22:57:52.376473+0000","last_clean":"2024-09-20T22:57:52.376473+0000","last_became_active":"2024-09-20T22:57:49.426958+0000","last_became_peered":"2024-09-20T22:57:49.426958+0000","last_unstale":"2024-09-20T22:57:52.376473+0000","last_undegraded":"2024-09-20T22:57:52.376473+0000","last_fullsized":"2024-09-20T22:57:52.376473+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"833'53","last_scrub_stamp":"2024-09-20T22:56:17.472832+0000","last_deep_scrub":"86'38","last_deep_scrub_stamp":"2024-09-20T22:42:43.935677+0000","last_clean_scrub_stamp":"2024-09-20T22:56:17.472832+0000","objects_scrubbed":10,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:54.751182+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":1074,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":377,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-20T22:57:52.374129+0000","last_change":"2024-09-20T22:57:39.577432+0000","last_active":"2024-09-20T22:57:52.374129+0000","last_peered":"2024-09-20T22:57:52.374129+0000","last_clean":"2024-09-20T22:57:52.374129+0000","last_became_active":"2024-09-20T22:55:17.340038+0000","last_became_peered":"2024-09-20T22:55:17.340038+0000","last_unstale":"2024-09-20T22:57:52.374129+0000","last_undegraded":"2024-09-20T22:57:52.374129+0000","last_fullsized":"2024-09-20T22:57:52.374129+0000","mapping_epoch":836,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":837,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:57:39.577360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:57:39.577360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:04.367204+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"69'2","reported_seq":1953,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-20T22:57:52.376527+0000","last_change":"2024-09-20T22:57:03.613282+0000","last_active":"2024-09-20T22:57:52.376527+0000","last_peered":"2024-09-20T22:57:52.376527+0000","last_clean":"2024-09-20T22:57:52.376527+0000","last_became_active":"2024-09-20T22:54:17.341059+0000","last_became_peered":"2024-09-20T22:54:17.341059+0000","last_unstale":"2024-09-20T22:57:52.376527+0000","last_undegraded":"2024-09-20T22:57:52.376527+0000","last_fullsized":"2024-09-20T22:57:52.376527+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2024-09-20T22:57:03.613204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:36.130684+0000","last_clean_scrub_stamp":"2024-09-20T22:57:03.613204+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:15.544078+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00087280599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":1271,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-20T22:57:52.373867+0000","last_change":"2024-09-20T22:56:13.481362+0000","last_active":"2024-09-20T22:57:52.373867+0000","last_peered":"2024-09-20T22:57:52.373867+0000","last_clean":"2024-09-20T22:57:52.373867+0000","last_became_active":"2024-09-20T22:54:17.339860+0000","last_became_peered":"2024-09-20T22:54:17.339860+0000","last_unstale":"2024-09-20T22:57:52.373867+0000","last_undegraded":"2024-09-20T22:57:52.373867+0000","last_fullsized":"2024-09-20T22:57:52.373867+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:56:13.481292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:56:13.481292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:56.680614+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":1802,"reported_epoch":991,"state":"active+clean","last_fresh":"2024-09-20T22:57:51.378254+0000","last_change":"2024-09-20T22:57:49.407286+0000","last_active":"2024-09-20T22:57:51.378254+0000","last_peered":"2024-09-20T22:57:51.378254+0000","last_clean":"2024-09-20T22:57:51.378254+0000","last_became_active":"2024-09-20T22:57:49.398514+0000","last_became_peered":"2024-09-20T22:57:49.398514+0000","last_unstale":"2024-09-20T22:57:51.378254+0000","last_undegraded":"2024-09-20T22:57:51.378254+0000","last_fullsized":"2024-09-20T22:57:51.378254+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:55:51.494144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:55:51.494144+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:55.630609+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"988'758","reported_seq":3744,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-20T22:57:52.373239+0000","last_change":"2024-09-20T22:57:48.440745+0000","last_active":"2024-09-20T22:57:52.373239+0000","last_peered":"2024-09-20T22:57:52.373239+0000","last_clean":"2024-09-20T22:57:52.373239+0000","last_became_active":"2024-09-20T22:56:53.366703+0000","last_became_peered":"2024-09-20T22:56:53.366703+0000","last_unstale":"2024-09-20T22:57:52.373239+0000","last_undegraded":"2024-09-20T22:57:52.373239+0000","last_fullsized":"2024-09-20T22:57:52.373239+0000","mapping_epoch":779,"log_start":"456'176","ondisk_log_start":"456'176","created":94,"last_epoch_clean":933,"parent":"0.0","parent_split_bits":0,"last_scrub":"941'718","last_scrub_stamp":"2024-09-20T22:57:06.750095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:57:06.750095+0000","objects_scrubbed":5,"log_size":582,"log_dups_size":318,"ondisk_log_size":582,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:36.737266+0000","scrub_duration":3,"objects_trimmed":1,"snaptrim_duration":0.071871422000000004,"stat_sum":{"num_bytes":12201763,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":11,"num_read_kb":7000,"num_write":8,"num_write_kb":2179,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"16b","length":"1"},{"start":"173","length":"1"},{"start":"17d","length":"1"},{"start":"180","length":"1"},{"start":"182","length":"3"},{"start":"186","length":"8"},{"start":"190","length":"10"},{"start":"1a1","length":"1"},{"start":"1a3","length":"2"},{"start":"1a6","length":"1"},{"start":"1aa","length":"2"},{"start":"1af","length":"1"},{"start":"1b5","length":"1"},{"start":"1b7","length":"1"},{"start":"1ba","length":"1"},{"start":"1bd","length":"2"},{"start":"1c0","length":"2"},{"start":"1c5","length":"1"},{"start":"1c7","length":"1"}]},{"pgid":"1.2","version":"77'128","reported_seq":1255,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-20T22:57:52.372107+0000","last_change":"2024-09-20T22:56:12.520096+0000","last_active":"2024-09-20T22:57:52.372107+0000","last_peered":"2024-09-20T22:57:52.372107+0000","last_clean":"2024-09-20T22:57:52.372107+0000","last_became_active":"2024-09-20T22:56:11.355063+0000","last_became_peered":"2024-09-20T22:56:11.355063+0000","last_unstale":"2024-09-20T22:57:52.372107+0000","last_undegraded":"2024-09-20T22:57:52.372107+0000","last_fullsized":"2024-09-20T22:57:52.372107+0000","mapping_epoch":890,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":891,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:56:12.519992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:56:12.519992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:56.680642+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"625'22","reported_seq":471,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-20T22:57:52.378294+0000","last_change":"2024-09-20T22:57:06.349767+0000","last_active":"2024-09-20T22:57:52.378294+0000","last_peered":"2024-09-20T22:57:52.378294+0000","last_clean":"2024-09-20T22:57:52.378294+0000","last_became_active":"2024-09-20T22:54:17.349415+0000","last_became_peered":"2024-09-20T22:54:17.349415+0000","last_unstale":"2024-09-20T22:57:52.378294+0000","last_undegraded":"2024-09-20T22:57:52.378294+0000","last_fullsized":"2024-09-20T22:57:52.378294+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'22","last_scrub_stamp":"2024-09-20T22:57:06.349682+0000","last_deep_scrub":"443'18","last_deep_scrub_stamp":"2024-09-20T22:48:44.182517+0000","last_clean_scrub_stamp":"2024-09-20T22:57:06.349682+0000","objects_scrubbed":5,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:11.297930+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":24,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":1320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":456,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-20T22:57:52.377915+0000","last_change":"2024-09-20T22:56:00.249870+0000","last_active":"2024-09-20T22:57:52.377915+0000","last_peered":"2024-09-20T22:57:52.377915+0000","last_clean":"2024-09-20T22:57:52.377915+0000","last_became_active":"2024-09-20T22:54:17.343806+0000","last_became_peered":"2024-09-20T22:54:17.343806+0000","last_unstale":"2024-09-20T22:57:52.377915+0000","last_undegraded":"2024-09-20T22:57:52.377915+0000","last_fullsized":"2024-09-20T22:57:52.377915+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:56:00.249796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:56:00.249796+0000","last_clean_scrub_stamp":"2024-09-20T22:56:00.249796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"380'2","reported_seq":1974,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-20T22:57:52.372608+0000","last_change":"2024-09-20T22:57:49.417639+0000","last_active":"2024-09-20T22:57:52.372608+0000","last_peered":"2024-09-20T22:57:52.372608+0000","last_clean":"2024-09-20T22:57:52.372608+0000","last_became_active":"2024-09-20T22:57:49.416996+0000","last_became_peered":"2024-09-20T22:57:49.416996+0000","last_unstale":"2024-09-20T22:57:52.372608+0000","last_undegraded":"2024-09-20T22:57:52.372608+0000","last_fullsized":"2024-09-20T22:57:52.372608+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"380'2","last_scrub_stamp":"2024-09-20T22:55:57.061312+0000","last_deep_scrub":"380'2","last_deep_scrub_stamp":"2024-09-20T22:47:40.983039+0000","last_clean_scrub_stamp":"2024-09-20T22:55:57.061312+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:55.310043+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001301459,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":1925,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-20T22:57:52.374290+0000","last_change":"2024-09-20T22:57:38.381568+0000","last_active":"2024-09-20T22:57:52.374290+0000","last_peered":"2024-09-20T22:57:52.374290+0000","last_clean":"2024-09-20T22:57:52.374290+0000","last_became_active":"2024-09-20T22:56:08.375555+0000","last_became_peered":"2024-09-20T22:56:08.375555+0000","last_unstale":"2024-09-20T22:57:52.374290+0000","last_undegraded":"2024-09-20T22:57:52.374290+0000","last_fullsized":"2024-09-20T22:57:52.374290+0000","mapping_epoch":887,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:57:38.381457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:57:38.381457+0000","last_clean_scrub_stamp":"2024-09-20T22:57:38.381457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:05.474433+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":234,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-20T22:57:52.568688+0000","last_change":"2024-09-20T22:57:49.404111+0000","last_active":"2024-09-20T22:57:52.568688+0000","last_peered":"2024-09-20T22:57:52.568688+0000","last_clean":"2024-09-20T22:57:52.568688+0000","last_became_active":"2024-09-20T22:57:49.403788+0000","last_became_peered":"2024-09-20T22:57:49.403788+0000","last_unstale":"2024-09-20T22:57:52.568688+0000","last_undegraded":"2024-09-20T22:57:52.568688+0000","last_fullsized":"2024-09-20T22:57:52.568688+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:56:51.218428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:56:51.218428+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:57.918997+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.5","version":"990'592","reported_seq":5161,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-20T22:57:52.373423+0000","last_change":"2024-09-20T22:57:48.440435+0000","last_active":"2024-09-20T22:57:52.373423+0000","last_peered":"2024-09-20T22:57:52.373423+0000","last_clean":"2024-09-20T22:57:52.373423+0000","last_became_active":"2024-09-20T22:56:53.369151+0000","last_became_peered":"2024-09-20T22:56:53.369151+0000","last_unstale":"2024-09-20T22:57:52.373423+0000","last_undegraded":"2024-09-20T22:57:52.373423+0000","last_fullsized":"2024-09-20T22:57:52.373423+0000","mapping_epoch":890,"log_start":"450'190","ondisk_log_start":"450'190","created":94,"last_epoch_clean":933,"parent":"0.0","parent_split_bits":0,"last_scrub":"941'546","last_scrub_stamp":"2024-09-20T22:57:04.121297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:57:04.121297+0000","objects_scrubbed":7,"log_size":402,"log_dups_size":240,"ondisk_log_size":402,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:30.793975+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.059506083000000001,"stat_sum":{"num_bytes":15343641,"num_objects":8,"num_object_clones":7,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":4,"num_write_kb":4269,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"16b","length":"1"},{"start":"173","length":"1"},{"start":"17d","length":"1"},{"start":"180","length":"1"},{"start":"182","length":"3"},{"start":"186","length":"8"},{"start":"190","length":"10"},{"start":"1a1","length":"1"},{"start":"1a3","length":"2"},{"start":"1a6","length":"1"},{"start":"1aa","length":"2"},{"start":"1af","length":"1"},{"start":"1b5","length":"1"},{"start":"1b7","length":"1"},{"start":"1ba","length":"1"},{"start":"1bd","length":"2"},{"start":"1c0","length":"2"},{"start":"1c5","length":"1"},{"start":"1c7","length":"1"}]},{"pgid":"1.3","version":"77'128","reported_seq":1475,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-20T22:57:52.378424+0000","last_change":"2024-09-20T22:56:21.532248+0000","last_active":"2024-09-20T22:57:52.378424+0000","last_peered":"2024-09-20T22:57:52.378424+0000","last_clean":"2024-09-20T22:57:52.378424+0000","last_became_active":"2024-09-20T22:54:20.413191+0000","last_became_peered":"2024-09-20T22:54:20.413191+0000","last_unstale":"2024-09-20T22:57:52.378424+0000","last_undegraded":"2024-09-20T22:57:52.378424+0000","last_fullsized":"2024-09-20T22:57:52.378424+0000","mapping_epoch":779,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":780,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:56:21.532173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:56:21.532173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"828'30","reported_seq":1351,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-20T22:57:52.374535+0000","last_change":"2024-09-20T22:57:09.738754+0000","last_active":"2024-09-20T22:57:52.374535+0000","last_peered":"2024-09-20T22:57:52.374535+0000","last_clean":"2024-09-20T22:57:52.374535+0000","last_became_active":"2024-09-20T22:55:11.346018+0000","last_became_peered":"2024-09-20T22:55:11.346018+0000","last_unstale":"2024-09-20T22:57:52.374535+0000","last_undegraded":"2024-09-20T22:57:52.374535+0000","last_fullsized":"2024-09-20T22:57:52.374535+0000","mapping_epoch":830,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":831,"parent":"0.0","parent_split_bits":0,"last_scrub":"828'30","last_scrub_stamp":"2024-09-20T22:57:09.738669+0000","last_deep_scrub":"828'30","last_deep_scrub_stamp":"2024-09-20T22:55:08.705571+0000","last_clean_scrub_stamp":"2024-09-20T22:57:09.738669+0000","objects_scrubbed":6,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:37.471839+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":1268,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-20T22:57:52.372718+0000","last_change":"2024-09-20T22:57:49.411179+0000","last_active":"2024-09-20T22:57:52.372718+0000","last_peered":"2024-09-20T22:57:52.372718+0000","last_clean":"2024-09-20T22:57:52.372718+0000","last_became_active":"2024-09-20T22:57:49.410539+0000","last_became_peered":"2024-09-20T22:57:49.410539+0000","last_unstale":"2024-09-20T22:57:52.372718+0000","last_undegraded":"2024-09-20T22:57:52.372718+0000","last_fullsized":"2024-09-20T22:57:52.372718+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:55:25.640520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:44:19.834641+0000","last_clean_scrub_stamp":"2024-09-20T22:55:25.640520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:56.680347+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"75'5","reported_seq":2603,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-20T22:57:52.408160+0000","last_change":"2024-09-20T22:57:49.402647+0000","last_active":"2024-09-20T22:57:52.408160+0000","last_peered":"2024-09-20T22:57:52.408160+0000","last_clean":"2024-09-20T22:57:52.408160+0000","last_became_active":"2024-09-20T22:57:49.402122+0000","last_became_peered":"2024-09-20T22:57:49.402122+0000","last_unstale":"2024-09-20T22:57:52.408160+0000","last_undegraded":"2024-09-20T22:57:52.408160+0000","last_fullsized":"2024-09-20T22:57:52.408160+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-09-20T22:56:12.471215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:36.130684+0000","last_clean_scrub_stamp":"2024-09-20T22:56:12.471215+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00078631700000000003,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":850,"num_read_kb":845,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1556,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":1936,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-20T22:57:52.376954+0000","last_change":"2024-09-20T22:57:13.660607+0000","last_active":"2024-09-20T22:57:52.376954+0000","last_peered":"2024-09-20T22:57:52.376954+0000","last_clean":"2024-09-20T22:57:52.376954+0000","last_became_active":"2024-09-20T22:54:17.339805+0000","last_became_peered":"2024-09-20T22:54:17.339805+0000","last_unstale":"2024-09-20T22:57:52.376954+0000","last_undegraded":"2024-09-20T22:57:52.376954+0000","last_fullsized":"2024-09-20T22:57:52.376954+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:57:13.660520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:57:13.660520+0000","last_clean_scrub_stamp":"2024-09-20T22:57:13.660520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:37.166154+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":457,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-20T22:57:52.378128+0000","last_change":"2024-09-20T22:57:49.429847+0000","last_active":"2024-09-20T22:57:52.378128+0000","last_peered":"2024-09-20T22:57:52.378128+0000","last_clean":"2024-09-20T22:57:52.378128+0000","last_became_active":"2024-09-20T22:57:49.429412+0000","last_became_peered":"2024-09-20T22:57:49.429412+0000","last_unstale":"2024-09-20T22:57:52.378128+0000","last_undegraded":"2024-09-20T22:57:52.378128+0000","last_fullsized":"2024-09-20T22:57:52.378128+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:55:11.066868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:55:11.066868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.6","version":"989'890","reported_seq":5369,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-20T22:57:52.374277+0000","last_change":"2024-09-20T22:57:49.531739+0000","last_active":"2024-09-20T22:57:52.374277+0000","last_peered":"2024-09-20T22:57:52.374277+0000","last_clean":"2024-09-20T22:57:52.374277+0000","last_became_active":"2024-09-20T22:57:49.412758+0000","last_became_peered":"2024-09-20T22:57:49.412758+0000","last_unstale":"2024-09-20T22:57:52.374277+0000","last_undegraded":"2024-09-20T22:57:52.374277+0000","last_fullsized":"2024-09-20T22:57:52.374277+0000","mapping_epoch":988,"log_start":"443'181","ondisk_log_start":"443'181","created":94,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"902'811","last_scrub_stamp":"2024-09-20T22:56:22.658684+0000","last_deep_scrub":"902'811","last_deep_scrub_stamp":"2024-09-20T22:56:22.658684+0000","last_clean_scrub_stamp":"2024-09-20T22:56:22.658684+0000","objects_scrubbed":21,"log_size":709,"log_dups_size":314,"ondisk_log_size":709,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:55.657687+0000","scrub_duration":109,"objects_trimmed":2,"snaptrim_duration":0.046809684999999997,"stat_sum":{"num_bytes":24820850,"num_objects":10,"num_object_clones":9,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":121,"num_read_kb":117044,"num_write":113,"num_write_kb":46821,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":40131419,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"16b","length":"1"},{"start":"173","length":"1"},{"start":"17d","length":"1"},{"start":"180","length":"1"},{"start":"182","length":"3"},{"start":"186","length":"8"},{"start":"190","length":"10"},{"start":"1a1","length":"1"},{"start":"1a3","length":"2"},{"start":"1a6","length":"1"},{"start":"1aa","length":"2"},{"start":"1af","length":"1"},{"start":"1b5","length":"1"},{"start":"1b7","length":"1"},{"start":"1ba","length":"1"},{"start":"1bd","length":"2"},{"start":"1c0","length":"2"},{"start":"1c5","length":"1"},{"start":"1c7","length":"1"}]},{"pgid":"4.5","version":"888'33","reported_seq":2277,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-20T22:57:52.568837+0000","last_change":"2024-09-20T22:56:21.986738+0000","last_active":"2024-09-20T22:57:52.568837+0000","last_peered":"2024-09-20T22:57:52.568837+0000","last_clean":"2024-09-20T22:57:52.568837+0000","last_became_active":"2024-09-20T22:56:08.358532+0000","last_became_peered":"2024-09-20T22:56:08.358532+0000","last_unstale":"2024-09-20T22:57:52.568837+0000","last_undegraded":"2024-09-20T22:57:52.568837+0000","last_fullsized":"2024-09-20T22:57:52.568837+0000","mapping_epoch":887,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":0,"last_scrub":"888'33","last_scrub_stamp":"2024-09-20T22:56:21.986633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:56:21.986633+0000","objects_scrubbed":8,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:55.294094+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":770,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"67'1","reported_seq":1807,"reported_epoch":991,"state":"active+clean","last_fresh":"2024-09-20T22:57:51.377168+0000","last_change":"2024-09-20T22:57:49.401711+0000","last_active":"2024-09-20T22:57:51.377168+0000","last_peered":"2024-09-20T22:57:51.377168+0000","last_clean":"2024-09-20T22:57:51.377168+0000","last_became_active":"2024-09-20T22:57:49.397191+0000","last_became_peered":"2024-09-20T22:57:49.397191+0000","last_unstale":"2024-09-20T22:57:51.377168+0000","last_undegraded":"2024-09-20T22:57:51.377168+0000","last_fullsized":"2024-09-20T22:57:51.377168+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-09-20T22:55:33.544218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:55:33.544218+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:55.630539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"77'128","reported_seq":1254,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-20T22:57:52.371952+0000","last_change":"2024-09-20T22:56:43.780891+0000","last_active":"2024-09-20T22:57:52.371952+0000","last_peered":"2024-09-20T22:57:52.371952+0000","last_clean":"2024-09-20T22:57:52.371952+0000","last_became_active":"2024-09-20T22:56:41.368240+0000","last_became_peered":"2024-09-20T22:56:41.368240+0000","last_unstale":"2024-09-20T22:57:52.371952+0000","last_undegraded":"2024-09-20T22:57:52.371952+0000","last_fullsized":"2024-09-20T22:57:52.371952+0000","mapping_epoch":920,"log_start":"77'128","ondisk_log_start":"77'128","created":19,"last_epoch_clean":921,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:56:43.780781+0000","last_deep_scrub":"77'128","last_deep_scrub_stamp":"2024-09-20T22:47:22.417099+0000","last_clean_scrub_stamp":"2024-09-20T22:56:43.780781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:56.680693+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":220,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-20T22:57:52.373710+0000","last_change":"2024-09-20T22:56:08.366982+0000","last_active":"2024-09-20T22:57:52.373710+0000","last_peered":"2024-09-20T22:57:52.373710+0000","last_clean":"2024-09-20T22:57:52.373710+0000","last_became_active":"2024-09-20T22:56:08.366671+0000","last_became_peered":"2024-09-20T22:56:08.366671+0000","last_unstale":"2024-09-20T22:57:52.373710+0000","last_undegraded":"2024-09-20T22:57:52.373710+0000","last_fullsized":"2024-09-20T22:57:52.373710+0000","mapping_epoch":887,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:55:10.707805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:55:10.707805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:55.309979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":1921,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-20T22:57:52.376888+0000","last_change":"2024-09-20T22:57:49.410905+0000","last_active":"2024-09-20T22:57:52.376888+0000","last_peered":"2024-09-20T22:57:52.376888+0000","last_clean":"2024-09-20T22:57:52.376888+0000","last_became_active":"2024-09-20T22:57:49.410488+0000","last_became_peered":"2024-09-20T22:57:49.410488+0000","last_unstale":"2024-09-20T22:57:52.376888+0000","last_undegraded":"2024-09-20T22:57:52.376888+0000","last_fullsized":"2024-09-20T22:57:52.376888+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:55:50.483056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:55:50.483056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:54.750909+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"988'954","reported_seq":5584,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-20T22:57:52.568559+0000","last_change":"2024-09-20T22:57:48.439778+0000","last_active":"2024-09-20T22:57:52.568559+0000","last_peered":"2024-09-20T22:57:52.568559+0000","last_clean":"2024-09-20T22:57:52.568559+0000","last_became_active":"2024-09-20T22:54:20.414528+0000","last_became_peered":"2024-09-20T22:54:20.414528+0000","last_unstale":"2024-09-20T22:57:52.568559+0000","last_undegraded":"2024-09-20T22:57:52.568559+0000","last_fullsized":"2024-09-20T22:57:52.568559+0000","mapping_epoch":779,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":780,"parent":"0.0","parent_split_bits":0,"last_scrub":"970'914","last_scrub_stamp":"2024-09-20T22:57:31.284226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:57:31.284226+0000","objects_scrubbed":16,"log_size":954,"log_dups_size":0,"ondisk_log_size":954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:37.068423+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.064640501000000003,"stat_sum":{"num_bytes":34316407,"num_objects":14,"num_object_clones":12,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":291,"num_read_kb":291495,"num_write":147,"num_write_kb":69585,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":88,"num_bytes_recovered":51113242,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"16b","length":"1"},{"start":"173","length":"1"},{"start":"17d","length":"1"},{"start":"180","length":"1"},{"start":"182","length":"3"},{"start":"186","length":"8"},{"start":"190","length":"10"},{"start":"1a1","length":"1"},{"start":"1a3","length":"2"},{"start":"1a6","length":"1"},{"start":"1aa","length":"2"},{"start":"1af","length":"1"},{"start":"1b5","length":"1"},{"start":"1b7","length":"1"},{"start":"1ba","length":"1"},{"start":"1bd","length":"2"},{"start":"1c0","length":"2"},{"start":"1c5","length":"1"},{"start":"1c7","length":"1"}]},{"pgid":"4.e","version":"850'24","reported_seq":840,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-20T22:57:52.568791+0000","last_change":"2024-09-20T22:57:38.226530+0000","last_active":"2024-09-20T22:57:52.568791+0000","last_peered":"2024-09-20T22:57:52.568791+0000","last_clean":"2024-09-20T22:57:52.568791+0000","last_became_active":"2024-09-20T22:55:17.342390+0000","last_became_peered":"2024-09-20T22:55:17.342390+0000","last_unstale":"2024-09-20T22:57:52.568791+0000","last_undegraded":"2024-09-20T22:57:52.568791+0000","last_fullsized":"2024-09-20T22:57:52.568791+0000","mapping_epoch":836,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":837,"parent":"0.0","parent_split_bits":0,"last_scrub":"850'24","last_scrub_stamp":"2024-09-20T22:57:38.226418+0000","last_deep_scrub":"850'24","last_deep_scrub_stamp":"2024-09-20T22:55:30.779018+0000","last_clean_scrub_stamp":"2024-09-20T22:57:38.226418+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:49.332407+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":23,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":854,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-20T22:57:52.372550+0000","last_change":"2024-09-20T22:56:17.135450+0000","last_active":"2024-09-20T22:57:52.372550+0000","last_peered":"2024-09-20T22:57:52.372550+0000","last_clean":"2024-09-20T22:57:52.372550+0000","last_became_active":"2024-09-20T22:56:08.359647+0000","last_became_peered":"2024-09-20T22:56:08.359647+0000","last_unstale":"2024-09-20T22:57:52.372550+0000","last_undegraded":"2024-09-20T22:57:52.372550+0000","last_fullsized":"2024-09-20T22:57:52.372550+0000","mapping_epoch":887,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:56:17.135379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:56:17.135379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:55.310087+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":1916,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-20T22:57:52.376652+0000","last_change":"2024-09-20T22:57:11.628063+0000","last_active":"2024-09-20T22:57:52.376652+0000","last_peered":"2024-09-20T22:57:52.376652+0000","last_clean":"2024-09-20T22:57:52.376652+0000","last_became_active":"2024-09-20T22:48:05.233778+0000","last_became_peered":"2024-09-20T22:48:05.233778+0000","last_unstale":"2024-09-20T22:57:52.376652+0000","last_undegraded":"2024-09-20T22:57:52.376652+0000","last_fullsized":"2024-09-20T22:57:52.376652+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:57:11.627990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:54:47.166836+0000","last_clean_scrub_stamp":"2024-09-20T22:57:11.627990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:34.275838+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":1776,"reported_epoch":991,"state":"active+clean","last_fresh":"2024-09-20T22:57:51.376971+0000","last_change":"2024-09-20T22:57:49.427590+0000","last_active":"2024-09-20T22:57:51.376971+0000","last_peered":"2024-09-20T22:57:51.376971+0000","last_clean":"2024-09-20T22:57:51.376971+0000","last_became_active":"2024-09-20T22:57:49.425373+0000","last_became_peered":"2024-09-20T22:57:49.425373+0000","last_unstale":"2024-09-20T22:57:51.376971+0000","last_undegraded":"2024-09-20T22:57:51.376971+0000","last_fullsized":"2024-09-20T22:57:51.376971+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:56:46.453177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:48:40.127123+0000","last_clean_scrub_stamp":"2024-09-20T22:56:46.453177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:56.360115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.e","version":"988'1072","reported_seq":5556,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-20T22:57:52.374059+0000","last_change":"2024-09-20T22:57:48.441312+0000","last_active":"2024-09-20T22:57:52.374059+0000","last_peered":"2024-09-20T22:57:52.374059+0000","last_clean":"2024-09-20T22:57:52.374059+0000","last_became_active":"2024-09-20T22:47:52.209452+0000","last_became_peered":"2024-09-20T22:47:52.209452+0000","last_unstale":"2024-09-20T22:57:52.374059+0000","last_undegraded":"2024-09-20T22:57:52.374059+0000","last_fullsized":"2024-09-20T22:57:52.374059+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"944'1032","last_scrub_stamp":"2024-09-20T22:57:05.131799+0000","last_deep_scrub":"174'60","last_deep_scrub_stamp":"2024-09-20T22:44:18.301776+0000","last_clean_scrub_stamp":"2024-09-20T22:57:05.131799+0000","objects_scrubbed":14,"log_size":1072,"log_dups_size":0,"ondisk_log_size":1072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:28.855536+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.057156603,"stat_sum":{"num_bytes":32496050,"num_objects":14,"num_object_clones":12,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":2,"num_read":306,"num_read_kb":300627,"num_write":228,"num_write_kb":83627,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":23110741,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"16b","length":"1"},{"start":"173","length":"1"},{"start":"17d","length":"1"},{"start":"180","length":"1"},{"start":"182","length":"3"},{"start":"186","length":"8"},{"start":"190","length":"10"},{"start":"1a1","length":"1"},{"start":"1a3","length":"2"},{"start":"1a6","length":"1"},{"start":"1aa","length":"2"},{"start":"1af","length":"1"},{"start":"1b5","length":"1"},{"start":"1b7","length":"1"},{"start":"1ba","length":"1"},{"start":"1bd","length":"2"},{"start":"1c0","length":"2"},{"start":"1c5","length":"1"},{"start":"1c7","length":"1"}]},{"pgid":"1.8","version":"77'128","reported_seq":2181,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-20T22:57:52.375848+0000","last_change":"2024-09-20T22:57:49.409192+0000","last_active":"2024-09-20T22:57:52.375848+0000","last_peered":"2024-09-20T22:57:52.375848+0000","last_clean":"2024-09-20T22:57:52.375848+0000","last_became_active":"2024-09-20T22:57:49.408874+0000","last_became_peered":"2024-09-20T22:57:49.408874+0000","last_unstale":"2024-09-20T22:57:52.375848+0000","last_undegraded":"2024-09-20T22:57:52.375848+0000","last_fullsized":"2024-09-20T22:57:52.375848+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:54:46.127943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:54:46.127943+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:54.750732+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"625'31","reported_seq":867,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-20T22:57:52.373753+0000","last_change":"2024-09-20T22:57:36.368878+0000","last_active":"2024-09-20T22:57:52.373753+0000","last_peered":"2024-09-20T22:57:52.373753+0000","last_clean":"2024-09-20T22:57:52.373753+0000","last_became_active":"2024-09-20T22:55:17.342262+0000","last_became_peered":"2024-09-20T22:55:17.342262+0000","last_unstale":"2024-09-20T22:57:52.373753+0000","last_undegraded":"2024-09-20T22:57:52.373753+0000","last_fullsized":"2024-09-20T22:57:52.373753+0000","mapping_epoch":836,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":837,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'31","last_scrub_stamp":"2024-09-20T22:57:36.368756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:57:36.368756+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:53.304747+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":48,"num_read_kb":38,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":1480,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-20T22:57:52.372834+0000","last_change":"2024-09-20T22:56:51.152802+0000","last_active":"2024-09-20T22:57:52.372834+0000","last_peered":"2024-09-20T22:57:52.372834+0000","last_clean":"2024-09-20T22:57:52.372834+0000","last_became_active":"2024-09-20T22:54:17.332636+0000","last_became_peered":"2024-09-20T22:54:17.332636+0000","last_unstale":"2024-09-20T22:57:52.372834+0000","last_undegraded":"2024-09-20T22:57:52.372834+0000","last_fullsized":"2024-09-20T22:57:52.372834+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:56:51.152732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:35.492562+0000","last_clean_scrub_stamp":"2024-09-20T22:56:51.152732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:18.450632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":456,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-20T22:57:52.378154+0000","last_change":"2024-09-20T22:56:49.666908+0000","last_active":"2024-09-20T22:57:52.378154+0000","last_peered":"2024-09-20T22:57:52.378154+0000","last_clean":"2024-09-20T22:57:52.378154+0000","last_became_active":"2024-09-20T22:54:17.349144+0000","last_became_peered":"2024-09-20T22:54:17.349144+0000","last_unstale":"2024-09-20T22:57:52.378154+0000","last_undegraded":"2024-09-20T22:57:52.378154+0000","last_fullsized":"2024-09-20T22:57:52.378154+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:56:49.666824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:56:49.666824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:15.812144+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":1944,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-20T22:57:52.374606+0000","last_change":"2024-09-20T22:57:49.411075+0000","last_active":"2024-09-20T22:57:52.374606+0000","last_peered":"2024-09-20T22:57:52.374606+0000","last_clean":"2024-09-20T22:57:52.374606+0000","last_became_active":"2024-09-20T22:57:49.410743+0000","last_became_peered":"2024-09-20T22:57:49.410743+0000","last_unstale":"2024-09-20T22:57:52.374606+0000","last_undegraded":"2024-09-20T22:57:52.374606+0000","last_fullsized":"2024-09-20T22:57:52.374606+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:56:43.023211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:56:43.023211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:55.142756+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"989'1180","reported_seq":5809,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-20T22:57:52.374497+0000","last_change":"2024-09-20T22:57:49.680248+0000","last_active":"2024-09-20T22:57:52.374497+0000","last_peered":"2024-09-20T22:57:52.374497+0000","last_clean":"2024-09-20T22:57:52.374497+0000","last_became_active":"2024-09-20T22:57:49.415940+0000","last_became_peered":"2024-09-20T22:57:49.415940+0000","last_unstale":"2024-09-20T22:57:52.374497+0000","last_undegraded":"2024-09-20T22:57:52.374497+0000","last_fullsized":"2024-09-20T22:57:52.374497+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"868'991","last_scrub_stamp":"2024-09-20T22:55:49.469629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:55:49.469629+0000","objects_scrubbed":15,"log_size":1180,"log_dups_size":0,"ondisk_log_size":1180,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:56.680473+0000","scrub_duration":8,"objects_trimmed":1,"snaptrim_duration":0.058896384000000003,"stat_sum":{"num_bytes":41006889,"num_objects":19,"num_object_clones":16,"num_object_copies":57,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":483,"num_read_kb":511223,"num_write":268,"num_write_kb":92130,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":121,"num_bytes_recovered":52177428,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"16b","length":"1"},{"start":"173","length":"1"},{"start":"17d","length":"1"},{"start":"180","length":"1"},{"start":"182","length":"3"},{"start":"186","length":"8"},{"start":"190","length":"10"},{"start":"1a1","length":"1"},{"start":"1a3","length":"2"},{"start":"1a6","length":"1"},{"start":"1aa","length":"2"},{"start":"1af","length":"1"},{"start":"1b5","length":"1"},{"start":"1b7","length":"1"},{"start":"1ba","length":"1"},{"start":"1bd","length":"2"},{"start":"1c0","length":"2"},{"start":"1c5","length":"1"},{"start":"1c7","length":"1"}]},{"pgid":"1.9","version":"77'128","reported_seq":453,"reported_epoch":991,"state":"active+clean","last_fresh":"2024-09-20T22:57:51.376238+0000","last_change":"2024-09-20T22:57:49.418054+0000","last_active":"2024-09-20T22:57:51.376238+0000","last_peered":"2024-09-20T22:57:51.376238+0000","last_clean":"2024-09-20T22:57:51.376238+0000","last_became_active":"2024-09-20T22:57:49.415117+0000","last_became_peered":"2024-09-20T22:57:49.415117+0000","last_unstale":"2024-09-20T22:57:51.376238+0000","last_undegraded":"2024-09-20T22:57:51.376238+0000","last_fullsized":"2024-09-20T22:57:51.376238+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:56:11.417116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:56:11.417116+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:55.630853+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"877'25","reported_seq":690,"reported_epoch":991,"state":"active+clean","last_fresh":"2024-09-20T22:57:51.376776+0000","last_change":"2024-09-20T22:57:49.426619+0000","last_active":"2024-09-20T22:57:51.376776+0000","last_peered":"2024-09-20T22:57:51.376776+0000","last_clean":"2024-09-20T22:57:51.376776+0000","last_became_active":"2024-09-20T22:57:49.424440+0000","last_became_peered":"2024-09-20T22:57:49.424440+0000","last_unstale":"2024-09-20T22:57:51.376776+0000","last_undegraded":"2024-09-20T22:57:51.376776+0000","last_fullsized":"2024-09-20T22:57:51.376776+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"877'25","last_scrub_stamp":"2024-09-20T22:55:57.521557+0000","last_deep_scrub":"877'25","last_deep_scrub_stamp":"2024-09-20T22:55:57.521557+0000","last_clean_scrub_stamp":"2024-09-20T22:55:57.521557+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:55.630711+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":28,"num_read_kb":23,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":1800,"reported_epoch":991,"state":"active+clean","last_fresh":"2024-09-20T22:57:51.376421+0000","last_change":"2024-09-20T22:57:49.433012+0000","last_active":"2024-09-20T22:57:51.376421+0000","last_peered":"2024-09-20T22:57:51.376421+0000","last_clean":"2024-09-20T22:57:51.376421+0000","last_became_active":"2024-09-20T22:57:49.432735+0000","last_became_peered":"2024-09-20T22:57:49.432735+0000","last_unstale":"2024-09-20T22:57:51.376421+0000","last_undegraded":"2024-09-20T22:57:51.376421+0000","last_fullsized":"2024-09-20T22:57:51.376421+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:55:58.553666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:55:58.553666+0000","last_clean_scrub_stamp":"2024-09-20T22:55:58.553666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:55.630641+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"888'10","reported_seq":2134,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-20T22:57:52.373699+0000","last_change":"2024-09-20T22:56:08.398936+0000","last_active":"2024-09-20T22:57:52.373699+0000","last_peered":"2024-09-20T22:57:52.373699+0000","last_clean":"2024-09-20T22:57:52.373699+0000","last_became_active":"2024-09-20T22:56:08.374292+0000","last_became_peered":"2024-09-20T22:56:08.374292+0000","last_unstale":"2024-09-20T22:57:52.373699+0000","last_undegraded":"2024-09-20T22:57:52.373699+0000","last_fullsized":"2024-09-20T22:57:52.373699+0000","mapping_epoch":887,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'9","last_scrub_stamp":"2024-09-20T22:55:36.178528+0000","last_deep_scrub":"261'9","last_deep_scrub_stamp":"2024-09-20T22:46:50.974735+0000","last_clean_scrub_stamp":"2024-09-20T22:55:36.178528+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:55.310011+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":462,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-20T22:57:52.377872+0000","last_change":"2024-09-20T22:56:23.542870+0000","last_active":"2024-09-20T22:57:52.377872+0000","last_peered":"2024-09-20T22:57:52.377872+0000","last_clean":"2024-09-20T22:57:52.377872+0000","last_became_active":"2024-09-20T22:56:01.477943+0000","last_became_peered":"2024-09-20T22:56:01.477943+0000","last_unstale":"2024-09-20T22:57:52.377872+0000","last_undegraded":"2024-09-20T22:57:52.377872+0000","last_fullsized":"2024-09-20T22:57:52.377872+0000","mapping_epoch":880,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":881,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:56:23.542803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:56:23.542803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.8","version":"989'1442","reported_seq":5791,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-20T22:57:52.372108+0000","last_change":"2024-09-20T22:57:49.507525+0000","last_active":"2024-09-20T22:57:52.372108+0000","last_peered":"2024-09-20T22:57:52.372108+0000","last_clean":"2024-09-20T22:57:52.372108+0000","last_became_active":"2024-09-20T22:57:49.415389+0000","last_became_peered":"2024-09-20T22:57:49.415389+0000","last_unstale":"2024-09-20T22:57:52.372108+0000","last_undegraded":"2024-09-20T22:57:52.372108+0000","last_fullsized":"2024-09-20T22:57:52.372108+0000","mapping_epoch":988,"log_start":"426'268","ondisk_log_start":"426'268","created":94,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"851'1165","last_scrub_stamp":"2024-09-20T22:55:32.164332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:55:32.164332+0000","objects_scrubbed":19,"log_size":1174,"log_dups_size":445,"ondisk_log_size":1174,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:56.353739+0000","scrub_duration":7,"objects_trimmed":1,"snaptrim_duration":0.052166095000000003,"stat_sum":{"num_bytes":26007451,"num_objects":13,"num_object_clones":11,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":216,"num_read_kb":232964,"num_write":167,"num_write_kb":55868,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":37295599,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"16b","length":"1"},{"start":"173","length":"1"},{"start":"17d","length":"1"},{"start":"180","length":"1"},{"start":"182","length":"3"},{"start":"186","length":"8"},{"start":"190","length":"10"},{"start":"1a1","length":"1"},{"start":"1a3","length":"2"},{"start":"1a6","length":"1"},{"start":"1aa","length":"2"},{"start":"1af","length":"1"},{"start":"1b5","length":"1"},{"start":"1b7","length":"1"},{"start":"1ba","length":"1"},{"start":"1bd","length":"2"},{"start":"1c0","length":"2"},{"start":"1c5","length":"1"},{"start":"1c7","length":"1"}]},{"pgid":"4.b","version":"625'15","reported_seq":1320,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-20T22:57:52.374033+0000","last_change":"2024-09-20T22:55:32.601025+0000","last_active":"2024-09-20T22:57:52.374033+0000","last_peered":"2024-09-20T22:57:52.374033+0000","last_clean":"2024-09-20T22:57:52.374033+0000","last_became_active":"2024-09-20T22:54:17.340998+0000","last_became_peered":"2024-09-20T22:54:17.340998+0000","last_unstale":"2024-09-20T22:57:52.374033+0000","last_undegraded":"2024-09-20T22:57:52.374033+0000","last_fullsized":"2024-09-20T22:57:52.374033+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'15","last_scrub_stamp":"2024-09-20T22:55:32.600891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:55:32.600891+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:56.680440+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":1921,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-20T22:57:52.376877+0000","last_change":"2024-09-20T22:57:49.425922+0000","last_active":"2024-09-20T22:57:52.376877+0000","last_peered":"2024-09-20T22:57:52.376877+0000","last_clean":"2024-09-20T22:57:52.376877+0000","last_became_active":"2024-09-20T22:57:49.425707+0000","last_became_peered":"2024-09-20T22:57:49.425707+0000","last_unstale":"2024-09-20T22:57:52.376877+0000","last_undegraded":"2024-09-20T22:57:52.376877+0000","last_fullsized":"2024-09-20T22:57:52.376877+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:56:24.518228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:56:24.518228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:54.751111+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":1944,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-20T22:57:52.373028+0000","last_change":"2024-09-20T22:57:49.411494+0000","last_active":"2024-09-20T22:57:52.373028+0000","last_peered":"2024-09-20T22:57:52.373028+0000","last_clean":"2024-09-20T22:57:52.373028+0000","last_became_active":"2024-09-20T22:57:49.411210+0000","last_became_peered":"2024-09-20T22:57:49.411210+0000","last_unstale":"2024-09-20T22:57:52.373028+0000","last_undegraded":"2024-09-20T22:57:52.373028+0000","last_fullsized":"2024-09-20T22:57:52.373028+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:54:47.368529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:54:47.368529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:55.309947+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":1268,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-20T22:57:52.372836+0000","last_change":"2024-09-20T22:57:49.411393+0000","last_active":"2024-09-20T22:57:52.372836+0000","last_peered":"2024-09-20T22:57:52.372836+0000","last_clean":"2024-09-20T22:57:52.372836+0000","last_became_active":"2024-09-20T22:57:49.410896+0000","last_became_peered":"2024-09-20T22:57:49.410896+0000","last_unstale":"2024-09-20T22:57:52.372836+0000","last_undegraded":"2024-09-20T22:57:52.372836+0000","last_fullsized":"2024-09-20T22:57:52.372836+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:56:44.736139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:56:44.736139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:56.680718+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"988'1928","reported_seq":6481,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-20T22:57:52.372934+0000","last_change":"2024-09-20T22:57:48.442076+0000","last_active":"2024-09-20T22:57:52.372934+0000","last_peered":"2024-09-20T22:57:52.372934+0000","last_clean":"2024-09-20T22:57:52.372934+0000","last_became_active":"2024-09-20T22:56:53.367443+0000","last_became_peered":"2024-09-20T22:56:53.367443+0000","last_unstale":"2024-09-20T22:57:52.372934+0000","last_undegraded":"2024-09-20T22:57:52.372934+0000","last_fullsized":"2024-09-20T22:57:52.372934+0000","mapping_epoch":779,"log_start":"423'280","ondisk_log_start":"423'280","created":94,"last_epoch_clean":933,"parent":"0.0","parent_split_bits":0,"last_scrub":"944'1882","last_scrub_stamp":"2024-09-20T22:57:05.737732+0000","last_deep_scrub":"590'828","last_deep_scrub_stamp":"2024-09-20T22:51:10.909364+0000","last_clean_scrub_stamp":"2024-09-20T22:57:05.737732+0000","objects_scrubbed":14,"log_size":1648,"log_dups_size":649,"ondisk_log_size":1648,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:35.520582+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.064944348999999998,"stat_sum":{"num_bytes":31503774,"num_objects":11,"num_object_clones":9,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":1,"num_read":18,"num_read_kb":22008,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"16b","length":"1"},{"start":"173","length":"1"},{"start":"17d","length":"1"},{"start":"180","length":"1"},{"start":"182","length":"3"},{"start":"186","length":"8"},{"start":"190","length":"10"},{"start":"1a1","length":"1"},{"start":"1a3","length":"2"},{"start":"1a6","length":"1"},{"start":"1aa","length":"2"},{"start":"1af","length":"1"},{"start":"1b5","length":"1"},{"start":"1b7","length":"1"},{"start":"1ba","length":"1"},{"start":"1bd","length":"2"},{"start":"1c0","length":"2"},{"start":"1c5","length":"1"},{"start":"1c7","length":"1"}]},{"pgid":"4.a","version":"625'44","reported_seq":2071,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-20T22:57:52.568109+0000","last_change":"2024-09-20T22:57:06.158084+0000","last_active":"2024-09-20T22:57:52.568109+0000","last_peered":"2024-09-20T22:57:52.568109+0000","last_clean":"2024-09-20T22:57:52.568109+0000","last_became_active":"2024-09-20T22:56:08.348928+0000","last_became_peered":"2024-09-20T22:56:08.348928+0000","last_unstale":"2024-09-20T22:57:52.568109+0000","last_undegraded":"2024-09-20T22:57:52.568109+0000","last_fullsized":"2024-09-20T22:57:52.568109+0000","mapping_epoch":887,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'44","last_scrub_stamp":"2024-09-20T22:57:06.157986+0000","last_deep_scrub":"113'28","last_deep_scrub_stamp":"2024-09-20T22:43:12.725900+0000","last_clean_scrub_stamp":"2024-09-20T22:57:06.157986+0000","objects_scrubbed":9,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:18.682809+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":228,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-20T22:57:52.376632+0000","last_change":"2024-09-20T22:57:49.404176+0000","last_active":"2024-09-20T22:57:52.376632+0000","last_peered":"2024-09-20T22:57:52.376632+0000","last_clean":"2024-09-20T22:57:52.376632+0000","last_became_active":"2024-09-20T22:57:49.403858+0000","last_became_peered":"2024-09-20T22:57:49.403858+0000","last_unstale":"2024-09-20T22:57:52.376632+0000","last_undegraded":"2024-09-20T22:57:52.376632+0000","last_fullsized":"2024-09-20T22:57:52.376632+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:55:29.772589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:55:29.772589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:54.750801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":1931,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-20T22:57:52.373147+0000","last_change":"2024-09-20T22:55:56.107908+0000","last_active":"2024-09-20T22:57:52.373147+0000","last_peered":"2024-09-20T22:57:52.373147+0000","last_clean":"2024-09-20T22:57:52.373147+0000","last_became_active":"2024-09-20T22:47:49.223220+0000","last_became_peered":"2024-09-20T22:47:49.223220+0000","last_unstale":"2024-09-20T22:57:52.373147+0000","last_undegraded":"2024-09-20T22:57:52.373147+0000","last_fullsized":"2024-09-20T22:57:52.373147+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:55:56.107827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:54:21.496068+0000","last_clean_scrub_stamp":"2024-09-20T22:55:56.107827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:55.310058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":228,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-20T22:57:52.372294+0000","last_change":"2024-09-20T22:57:49.399320+0000","last_active":"2024-09-20T22:57:52.372294+0000","last_peered":"2024-09-20T22:57:52.372294+0000","last_clean":"2024-09-20T22:57:52.372294+0000","last_became_active":"2024-09-20T22:57:49.399016+0000","last_became_peered":"2024-09-20T22:57:49.399016+0000","last_unstale":"2024-09-20T22:57:52.372294+0000","last_undegraded":"2024-09-20T22:57:52.372294+0000","last_fullsized":"2024-09-20T22:57:52.372294+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:55:54.870627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:55:54.870627+0000","last_clean_scrub_stamp":"2024-09-20T22:55:54.870627+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:55.310073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.a","version":"989'1256","reported_seq":5513,"reported_epoch":991,"state":"active+clean","last_fresh":"2024-09-20T22:57:51.377255+0000","last_change":"2024-09-20T22:57:49.673268+0000","last_active":"2024-09-20T22:57:51.377255+0000","last_peered":"2024-09-20T22:57:51.377255+0000","last_clean":"2024-09-20T22:57:51.377255+0000","last_became_active":"2024-09-20T22:57:49.401412+0000","last_became_peered":"2024-09-20T22:57:49.401412+0000","last_unstale":"2024-09-20T22:57:51.377255+0000","last_undegraded":"2024-09-20T22:57:51.377255+0000","last_fullsized":"2024-09-20T22:57:51.377255+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"851'1076","last_scrub_stamp":"2024-09-20T22:55:32.567038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:55:32.567038+0000","objects_scrubbed":21,"log_size":1256,"log_dups_size":0,"ondisk_log_size":1256,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:55.630500+0000","scrub_duration":10,"objects_trimmed":5,"snaptrim_duration":0.120863633,"stat_sum":{"num_bytes":64995264,"num_objects":25,"num_object_clones":22,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":485,"num_read_kb":502840,"num_write":373,"num_write_kb":163696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":11877612,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"16b","length":"1"},{"start":"173","length":"1"},{"start":"17d","length":"1"},{"start":"180","length":"1"},{"start":"182","length":"3"},{"start":"186","length":"8"},{"start":"190","length":"10"},{"start":"1a1","length":"1"},{"start":"1a3","length":"2"},{"start":"1a6","length":"1"},{"start":"1aa","length":"2"},{"start":"1af","length":"1"},{"start":"1b5","length":"1"},{"start":"1b7","length":"1"},{"start":"1ba","length":"1"},{"start":"1bd","length":"2"},{"start":"1c0","length":"2"},{"start":"1c5","length":"1"},{"start":"1c7","length":"1"}]},{"pgid":"4.9","version":"686'23","reported_seq":1168,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-20T22:57:52.377374+0000","last_change":"2024-09-20T22:57:41.298315+0000","last_active":"2024-09-20T22:57:52.377374+0000","last_peered":"2024-09-20T22:57:52.377374+0000","last_clean":"2024-09-20T22:57:52.377374+0000","last_became_active":"2024-09-20T22:55:17.346757+0000","last_became_peered":"2024-09-20T22:55:17.346757+0000","last_unstale":"2024-09-20T22:57:52.377374+0000","last_undegraded":"2024-09-20T22:57:52.377374+0000","last_fullsized":"2024-09-20T22:57:52.377374+0000","mapping_epoch":836,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":837,"parent":"0.0","parent_split_bits":0,"last_scrub":"686'23","last_scrub_stamp":"2024-09-20T22:57:41.298234+0000","last_deep_scrub":"686'23","last_deep_scrub_stamp":"2024-09-20T22:52:46.768812+0000","last_clean_scrub_stamp":"2024-09-20T22:57:41.298234+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:09.189056+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":21,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":1733,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-20T22:57:52.377749+0000","last_change":"2024-09-20T22:57:11.250944+0000","last_active":"2024-09-20T22:57:52.377749+0000","last_peered":"2024-09-20T22:57:52.377749+0000","last_clean":"2024-09-20T22:57:52.377749+0000","last_became_active":"2024-09-20T22:56:08.358713+0000","last_became_peered":"2024-09-20T22:56:08.358713+0000","last_unstale":"2024-09-20T22:57:52.377749+0000","last_undegraded":"2024-09-20T22:57:52.377749+0000","last_fullsized":"2024-09-20T22:57:52.377749+0000","mapping_epoch":887,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:57:11.250858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:46:52.465940+0000","last_clean_scrub_stamp":"2024-09-20T22:57:11.250858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:35.003707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":1951,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-20T22:57:52.372442+0000","last_change":"2024-09-20T22:57:10.236664+0000","last_active":"2024-09-20T22:57:52.372442+0000","last_peered":"2024-09-20T22:57:52.372442+0000","last_clean":"2024-09-20T22:57:52.372442+0000","last_became_active":"2024-09-20T22:54:17.340269+0000","last_became_peered":"2024-09-20T22:54:17.340269+0000","last_unstale":"2024-09-20T22:57:52.372442+0000","last_undegraded":"2024-09-20T22:57:52.372442+0000","last_fullsized":"2024-09-20T22:57:52.372442+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:57:10.236592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:53:07.104891+0000","last_clean_scrub_stamp":"2024-09-20T22:57:10.236592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:35.494601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":1794,"reported_epoch":991,"state":"active+clean","last_fresh":"2024-09-20T22:57:51.377237+0000","last_change":"2024-09-20T22:57:49.422638+0000","last_active":"2024-09-20T22:57:51.377237+0000","last_peered":"2024-09-20T22:57:51.377237+0000","last_clean":"2024-09-20T22:57:51.377237+0000","last_became_active":"2024-09-20T22:57:49.419390+0000","last_became_peered":"2024-09-20T22:57:49.419390+0000","last_unstale":"2024-09-20T22:57:51.377237+0000","last_undegraded":"2024-09-20T22:57:51.377237+0000","last_fullsized":"2024-09-20T22:57:51.377237+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:53.242204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:55.170379+0000","last_clean_scrub_stamp":"2024-09-20T22:53:53.242204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:55.629963+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"992'1663","reported_seq":6166,"reported_epoch":992,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T22:57:52.568604+0000","last_change":"2024-09-20T22:57:52.566023+0000","last_active":"2024-09-20T22:57:52.568604+0000","last_peered":"2024-09-20T22:57:52.568604+0000","last_clean":"2024-09-20T22:57:52.568604+0000","last_became_active":"2024-09-20T22:54:20.415706+0000","last_became_peered":"2024-09-20T22:54:20.415706+0000","last_unstale":"2024-09-20T22:57:52.568604+0000","last_undegraded":"2024-09-20T22:57:52.568604+0000","last_fullsized":"2024-09-20T22:57:52.568604+0000","mapping_epoch":779,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":780,"parent":"0.0","parent_split_bits":0,"last_scrub":"794'1255","last_scrub_stamp":"2024-09-20T22:54:37.866509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:54:37.866509+0000","objects_scrubbed":15,"log_size":1663,"log_dups_size":0,"ondisk_log_size":1663,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":13,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.187232763,"stat_sum":{"num_bytes":71055695,"num_objects":34,"num_object_clones":29,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":2,"num_read":681,"num_read_kb":757216,"num_write":460,"num_write_kb":173264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":168,"num_bytes_recovered":68651320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"16b","length":"1"},{"start":"173","length":"1"},{"start":"17d","length":"1"},{"start":"180","length":"1"},{"start":"182","length":"3"},{"start":"186","length":"8"},{"start":"190","length":"a"},{"start":"19b","length":"1"},{"start":"19d","length":"3"},{"start":"1a1","length":"1"},{"start":"1a4","length":"1"},{"start":"1ab","length":"1"},{"start":"1af","length":"1"},{"start":"1c0","length":"1"}]},{"pgid":"4.8","version":"625'27","reported_seq":1973,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-20T22:57:52.375988+0000","last_change":"2024-09-20T22:57:10.649039+0000","last_active":"2024-09-20T22:57:52.375988+0000","last_peered":"2024-09-20T22:57:52.375988+0000","last_clean":"2024-09-20T22:57:52.375988+0000","last_became_active":"2024-09-20T22:56:08.360719+0000","last_became_peered":"2024-09-20T22:56:08.360719+0000","last_unstale":"2024-09-20T22:57:52.375988+0000","last_undegraded":"2024-09-20T22:57:52.375988+0000","last_fullsized":"2024-09-20T22:57:52.375988+0000","mapping_epoch":887,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'27","last_scrub_stamp":"2024-09-20T22:57:10.648909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:57:10.648909+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:13.332381+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'2","reported_seq":931,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-20T22:57:52.374263+0000","last_change":"2024-09-20T22:56:08.422471+0000","last_active":"2024-09-20T22:57:52.374263+0000","last_peered":"2024-09-20T22:57:52.374263+0000","last_clean":"2024-09-20T22:57:52.374263+0000","last_became_active":"2024-09-20T22:56:08.392282+0000","last_became_peered":"2024-09-20T22:56:08.392282+0000","last_unstale":"2024-09-20T22:57:52.374263+0000","last_undegraded":"2024-09-20T22:57:52.374263+0000","last_fullsized":"2024-09-20T22:57:52.374263+0000","mapping_epoch":887,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-09-20T22:55:33.770279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:55:33.770279+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:56.680408+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":276,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.9","version":"888'11","reported_seq":276,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-20T22:57:52.568115+0000","last_change":"2024-09-20T22:56:48.156966+0000","last_active":"2024-09-20T22:57:52.568115+0000","last_peered":"2024-09-20T22:57:52.568115+0000","last_clean":"2024-09-20T22:57:52.568115+0000","last_became_active":"2024-09-20T22:56:08.357520+0000","last_became_peered":"2024-09-20T22:56:08.357520+0000","last_unstale":"2024-09-20T22:57:52.568115+0000","last_undegraded":"2024-09-20T22:57:52.568115+0000","last_fullsized":"2024-09-20T22:57:52.568115+0000","mapping_epoch":887,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":0,"last_scrub":"888'11","last_scrub_stamp":"2024-09-20T22:56:48.156843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:56:48.156843+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:57.269177+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":1912,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-20T22:57:52.376141+0000","last_change":"2024-09-20T22:57:14.615994+0000","last_active":"2024-09-20T22:57:52.376141+0000","last_peered":"2024-09-20T22:57:52.376141+0000","last_clean":"2024-09-20T22:57:52.376141+0000","last_became_active":"2024-09-20T22:47:49.227569+0000","last_became_peered":"2024-09-20T22:47:49.227569+0000","last_unstale":"2024-09-20T22:57:52.376141+0000","last_undegraded":"2024-09-20T22:57:52.376141+0000","last_fullsized":"2024-09-20T22:57:52.376141+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:57:14.615909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:57:14.615909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:29.335354+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.14","version":"989'730","reported_seq":3753,"reported_epoch":991,"state":"active+clean","last_fresh":"2024-09-20T22:57:51.381825+0000","last_change":"2024-09-20T22:57:51.381495+0000","last_active":"2024-09-20T22:57:51.381825+0000","last_peered":"2024-09-20T22:57:51.381825+0000","last_clean":"2024-09-20T22:57:51.381825+0000","last_became_active":"2024-09-20T22:57:51.380711+0000","last_became_peered":"2024-09-20T22:57:51.380711+0000","last_unstale":"2024-09-20T22:57:51.381825+0000","last_undegraded":"2024-09-20T22:57:51.381825+0000","last_fullsized":"2024-09-20T22:57:51.381825+0000","mapping_epoch":990,"log_start":"456'176","ondisk_log_start":"456'176","created":932,"last_epoch_clean":991,"parent":"0.0","parent_split_bits":5,"last_scrub":"853'620","last_scrub_stamp":"2024-09-20T22:55:34.565081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:55:34.565081+0000","objects_scrubbed":8,"log_size":554,"log_dups_size":318,"ondisk_log_size":554,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:56.672822+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.037993079999999999,"stat_sum":{"num_bytes":16142342,"num_objects":7,"num_object_clones":6,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":112,"num_read_kb":107256,"num_write":99,"num_write_kb":34830,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":33042853,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"16b","length":"1"},{"start":"173","length":"1"},{"start":"17d","length":"1"},{"start":"180","length":"1"},{"start":"182","length":"3"},{"start":"186","length":"8"},{"start":"190","length":"10"},{"start":"1a1","length":"1"},{"start":"1a3","length":"2"},{"start":"1a6","length":"1"},{"start":"1aa","length":"2"},{"start":"1af","length":"1"},{"start":"1b5","length":"1"},{"start":"1b7","length":"1"},{"start":"1ba","length":"1"},{"start":"1bd","length":"2"},{"start":"1c0","length":"2"},{"start":"1c5","length":"1"},{"start":"1c7","length":"1"}]},{"pgid":"3.10","version":"0'0","reported_seq":1944,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-20T22:57:52.567800+0000","last_change":"2024-09-20T22:56:17.950955+0000","last_active":"2024-09-20T22:57:52.567800+0000","last_peered":"2024-09-20T22:57:52.567800+0000","last_clean":"2024-09-20T22:57:52.567800+0000","last_became_active":"2024-09-20T22:47:49.213292+0000","last_became_peered":"2024-09-20T22:47:49.213292+0000","last_unstale":"2024-09-20T22:57:52.567800+0000","last_undegraded":"2024-09-20T22:57:52.567800+0000","last_fullsized":"2024-09-20T22:57:52.567800+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:56:17.950879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:56:17.950879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:55.294078+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"625'12","reported_seq":1852,"reported_epoch":991,"state":"active+clean","last_fresh":"2024-09-20T22:57:51.378932+0000","last_change":"2024-09-20T22:57:49.427905+0000","last_active":"2024-09-20T22:57:51.378932+0000","last_peered":"2024-09-20T22:57:51.378932+0000","last_clean":"2024-09-20T22:57:51.378932+0000","last_became_active":"2024-09-20T22:57:49.426479+0000","last_became_peered":"2024-09-20T22:57:49.426479+0000","last_unstale":"2024-09-20T22:57:51.378932+0000","last_undegraded":"2024-09-20T22:57:51.378932+0000","last_fullsized":"2024-09-20T22:57:51.378932+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'12","last_scrub_stamp":"2024-09-20T22:53:26.319144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:53:26.319144+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:55.629812+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":1924,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-20T22:57:52.376811+0000","last_change":"2024-09-20T22:57:49.408789+0000","last_active":"2024-09-20T22:57:52.376811+0000","last_peered":"2024-09-20T22:57:52.376811+0000","last_clean":"2024-09-20T22:57:52.376811+0000","last_became_active":"2024-09-20T22:57:49.408529+0000","last_became_peered":"2024-09-20T22:57:49.408529+0000","last_unstale":"2024-09-20T22:57:52.376811+0000","last_undegraded":"2024-09-20T22:57:52.376811+0000","last_fullsized":"2024-09-20T22:57:52.376811+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:55:29.324286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:09.705632+0000","last_clean_scrub_stamp":"2024-09-20T22:55:29.324286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:54.750841+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"74'1","reported_seq":240,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-20T22:57:52.568687+0000","last_change":"2024-09-20T22:56:47.178707+0000","last_active":"2024-09-20T22:57:52.568687+0000","last_peered":"2024-09-20T22:57:52.568687+0000","last_clean":"2024-09-20T22:57:52.568687+0000","last_became_active":"2024-09-20T22:56:08.354987+0000","last_became_peered":"2024-09-20T22:56:08.354987+0000","last_unstale":"2024-09-20T22:57:52.568687+0000","last_undegraded":"2024-09-20T22:57:52.568687+0000","last_fullsized":"2024-09-20T22:57:52.568687+0000","mapping_epoch":887,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-09-20T22:56:47.178627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:56:47.178627+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:11.125701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.15","version":"930'540","reported_seq":5102,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-20T22:57:52.378008+0000","last_change":"2024-09-20T22:57:48.374841+0000","last_active":"2024-09-20T22:57:52.378008+0000","last_peered":"2024-09-20T22:57:52.378008+0000","last_clean":"2024-09-20T22:57:52.378008+0000","last_became_active":"2024-09-20T22:56:58.367157+0000","last_became_peered":"2024-09-20T22:56:58.367157+0000","last_unstale":"2024-09-20T22:57:52.378008+0000","last_undegraded":"2024-09-20T22:57:52.378008+0000","last_fullsized":"2024-09-20T22:57:52.378008+0000","mapping_epoch":937,"log_start":"450'190","ondisk_log_start":"450'190","created":932,"last_epoch_clean":938,"parent":"0.0","parent_split_bits":5,"last_scrub":"930'540","last_scrub_stamp":"2024-09-20T22:57:04.343926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:57:04.343926+0000","objects_scrubbed":0,"log_size":350,"log_dups_size":240,"ondisk_log_size":350,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:18.165862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011806360000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"16b","length":"1"},{"start":"173","length":"1"},{"start":"17d","length":"1"},{"start":"180","length":"1"},{"start":"182","length":"3"},{"start":"186","length":"8"},{"start":"190","length":"10"},{"start":"1a1","length":"1"},{"start":"1a3","length":"2"},{"start":"1a6","length":"1"},{"start":"1aa","length":"2"},{"start":"1af","length":"1"},{"start":"1b5","length":"1"},{"start":"1b7","length":"1"},{"start":"1ba","length":"1"},{"start":"1bd","length":"2"},{"start":"1c0","length":"2"},{"start":"1c5","length":"1"},{"start":"1c7","length":"1"}]},{"pgid":"4.16","version":"625'15","reported_seq":1297,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-20T22:57:52.374426+0000","last_change":"2024-09-20T22:57:49.413436+0000","last_active":"2024-09-20T22:57:52.374426+0000","last_peered":"2024-09-20T22:57:52.374426+0000","last_clean":"2024-09-20T22:57:52.374426+0000","last_became_active":"2024-09-20T22:57:49.412096+0000","last_became_peered":"2024-09-20T22:57:49.412096+0000","last_unstale":"2024-09-20T22:57:52.374426+0000","last_undegraded":"2024-09-20T22:57:52.374426+0000","last_fullsized":"2024-09-20T22:57:52.374426+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'15","last_scrub_stamp":"2024-09-20T22:55:27.637946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:55:27.637946+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:56.680379+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":944,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-20T22:57:52.568286+0000","last_change":"2024-09-20T22:56:11.967360+0000","last_active":"2024-09-20T22:57:52.568286+0000","last_peered":"2024-09-20T22:57:52.568286+0000","last_clean":"2024-09-20T22:57:52.568286+0000","last_became_active":"2024-09-20T22:56:08.357576+0000","last_became_peered":"2024-09-20T22:56:08.357576+0000","last_unstale":"2024-09-20T22:57:52.568286+0000","last_undegraded":"2024-09-20T22:57:52.568286+0000","last_fullsized":"2024-09-20T22:57:52.568286+0000","mapping_epoch":887,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:56:11.967266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:56:11.967266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:55.294140+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":1806,"reported_epoch":991,"state":"active+clean","last_fresh":"2024-09-20T22:57:51.377635+0000","last_change":"2024-09-20T22:57:49.433827+0000","last_active":"2024-09-20T22:57:51.377635+0000","last_peered":"2024-09-20T22:57:51.377635+0000","last_clean":"2024-09-20T22:57:51.377635+0000","last_became_active":"2024-09-20T22:57:49.433318+0000","last_became_peered":"2024-09-20T22:57:49.433318+0000","last_unstale":"2024-09-20T22:57:51.377635+0000","last_undegraded":"2024-09-20T22:57:51.377635+0000","last_fullsized":"2024-09-20T22:57:51.377635+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:55:52.481737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:55:52.481737+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:55.630674+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":1952,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-20T22:57:52.373564+0000","last_change":"2024-09-20T22:56:42.057480+0000","last_active":"2024-09-20T22:57:52.373564+0000","last_peered":"2024-09-20T22:57:52.373564+0000","last_clean":"2024-09-20T22:57:52.373564+0000","last_became_active":"2024-09-20T22:56:08.369854+0000","last_became_peered":"2024-09-20T22:56:08.369854+0000","last_unstale":"2024-09-20T22:57:52.373564+0000","last_undegraded":"2024-09-20T22:57:52.373564+0000","last_fullsized":"2024-09-20T22:57:52.373564+0000","mapping_epoch":887,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:56:42.057380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:53:58.169894+0000","last_clean_scrub_stamp":"2024-09-20T22:56:42.057380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:55.310103+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.16","version":"988'903","reported_seq":5424,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-20T22:57:52.373361+0000","last_change":"2024-09-20T22:57:48.439743+0000","last_active":"2024-09-20T22:57:52.373361+0000","last_peered":"2024-09-20T22:57:52.373361+0000","last_clean":"2024-09-20T22:57:52.373361+0000","last_became_active":"2024-09-20T22:56:58.372517+0000","last_became_peered":"2024-09-20T22:56:58.372517+0000","last_unstale":"2024-09-20T22:57:52.373361+0000","last_undegraded":"2024-09-20T22:57:52.373361+0000","last_fullsized":"2024-09-20T22:57:52.373361+0000","mapping_epoch":937,"log_start":"443'181","ondisk_log_start":"443'181","created":932,"last_epoch_clean":938,"parent":"0.0","parent_split_bits":5,"last_scrub":"944'873","last_scrub_stamp":"2024-09-20T22:57:07.764870+0000","last_deep_scrub":"902'811","last_deep_scrub_stamp":"2024-09-20T22:56:22.658684+0000","last_clean_scrub_stamp":"2024-09-20T22:57:07.764870+0000","objects_scrubbed":12,"log_size":722,"log_dups_size":314,"ondisk_log_size":722,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:18.543615+0000","scrub_duration":3,"objects_trimmed":1,"snaptrim_duration":0.070922279000000005,"stat_sum":{"num_bytes":21972561,"num_objects":9,"num_object_clones":8,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"16b","length":"1"},{"start":"173","length":"1"},{"start":"17d","length":"1"},{"start":"180","length":"1"},{"start":"182","length":"3"},{"start":"186","length":"8"},{"start":"190","length":"10"},{"start":"1a1","length":"1"},{"start":"1a3","length":"2"},{"start":"1a6","length":"1"},{"start":"1aa","length":"2"},{"start":"1af","length":"1"},{"start":"1b5","length":"1"},{"start":"1b7","length":"1"},{"start":"1ba","length":"1"},{"start":"1bd","length":"2"},{"start":"1c0","length":"2"},{"start":"1c5","length":"1"},{"start":"1c7","length":"1"}]},{"pgid":"4.15","version":"625'15","reported_seq":1957,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-20T22:57:52.376670+0000","last_change":"2024-09-20T22:57:49.404308+0000","last_active":"2024-09-20T22:57:52.376670+0000","last_peered":"2024-09-20T22:57:52.376670+0000","last_clean":"2024-09-20T22:57:52.376670+0000","last_became_active":"2024-09-20T22:57:49.403807+0000","last_became_peered":"2024-09-20T22:57:49.403807+0000","last_unstale":"2024-09-20T22:57:52.376670+0000","last_undegraded":"2024-09-20T22:57:52.376670+0000","last_fullsized":"2024-09-20T22:57:52.376670+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'15","last_scrub_stamp":"2024-09-20T22:56:15.488641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:56:15.488641+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:54.751044+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":1276,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-20T22:57:52.372198+0000","last_change":"2024-09-20T22:57:49.399263+0000","last_active":"2024-09-20T22:57:52.372198+0000","last_peered":"2024-09-20T22:57:52.372198+0000","last_clean":"2024-09-20T22:57:52.372198+0000","last_became_active":"2024-09-20T22:57:49.398995+0000","last_became_peered":"2024-09-20T22:57:49.398995+0000","last_unstale":"2024-09-20T22:57:52.372198+0000","last_undegraded":"2024-09-20T22:57:52.372198+0000","last_fullsized":"2024-09-20T22:57:52.372198+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:56:16.523567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:56:16.523567+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:56.680666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"989'11","reported_seq":1985,"reported_epoch":991,"state":"active+clean","last_fresh":"2024-09-20T22:57:51.376737+0000","last_change":"2024-09-20T22:57:49.432210+0000","last_active":"2024-09-20T22:57:51.376737+0000","last_peered":"2024-09-20T22:57:51.376737+0000","last_clean":"2024-09-20T22:57:51.376737+0000","last_became_active":"2024-09-20T22:57:49.430688+0000","last_became_peered":"2024-09-20T22:57:49.430688+0000","last_unstale":"2024-09-20T22:57:51.376737+0000","last_undegraded":"2024-09-20T22:57:51.376737+0000","last_fullsized":"2024-09-20T22:57:51.376737+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-09-20T22:53:59.165134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:53:59.165134+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:55.630110+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":869,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-20T22:57:52.374582+0000","last_change":"2024-09-20T22:55:29.161667+0000","last_active":"2024-09-20T22:57:52.374582+0000","last_peered":"2024-09-20T22:57:52.374582+0000","last_clean":"2024-09-20T22:57:52.374582+0000","last_became_active":"2024-09-20T22:55:17.346270+0000","last_became_peered":"2024-09-20T22:55:17.346270+0000","last_unstale":"2024-09-20T22:57:52.374582+0000","last_undegraded":"2024-09-20T22:57:52.374582+0000","last_fullsized":"2024-09-20T22:57:52.374582+0000","mapping_epoch":836,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":837,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:55:29.161595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:55:29.161595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:55.309994+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.17","version":"990'2020","reported_seq":6511,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-20T22:57:52.717318+0000","last_change":"2024-09-20T22:57:52.717318+0000","last_active":"2024-09-20T22:57:52.717318+0000","last_peered":"2024-09-20T22:57:52.717318+0000","last_clean":"2024-09-20T22:57:52.717318+0000","last_became_active":"2024-09-20T22:57:52.378717+0000","last_became_peered":"2024-09-20T22:57:52.378717+0000","last_unstale":"2024-09-20T22:57:52.717318+0000","last_undegraded":"2024-09-20T22:57:52.717318+0000","last_fullsized":"2024-09-20T22:57:52.717318+0000","mapping_epoch":991,"log_start":"436'456","ondisk_log_start":"436'456","created":932,"last_epoch_clean":992,"parent":"0.0","parent_split_bits":5,"last_scrub":"990'2020","last_scrub_stamp":"2024-09-20T22:57:52.717196+0000","last_deep_scrub":"879'1788","last_deep_scrub_stamp":"2024-09-20T22:55:59.964830+0000","last_clean_scrub_stamp":"2024-09-20T22:57:52.717196+0000","objects_scrubbed":10,"log_size":1564,"log_dups_size":886,"ondisk_log_size":1564,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:10.038236+0000","scrub_duration":3,"objects_trimmed":1,"snaptrim_duration":0.034776473000000002,"stat_sum":{"num_bytes":26840801,"num_objects":10,"num_object_clones":8,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"16b","length":"1"},{"start":"173","length":"1"},{"start":"17d","length":"1"},{"start":"180","length":"1"},{"start":"182","length":"3"},{"start":"186","length":"8"},{"start":"190","length":"10"},{"start":"1a1","length":"1"},{"start":"1a3","length":"2"},{"start":"1a6","length":"1"},{"start":"1aa","length":"2"},{"start":"1af","length":"1"},{"start":"1b5","length":"1"},{"start":"1b7","length":"1"},{"start":"1ba","length":"1"},{"start":"1bd","length":"2"},{"start":"1c0","length":"2"},{"start":"1c5","length":"1"},{"start":"1c7","length":"1"}]},{"pgid":"4.14","version":"625'20","reported_seq":1884,"reported_epoch":991,"state":"active+clean","last_fresh":"2024-09-20T22:57:51.376660+0000","last_change":"2024-09-20T22:57:49.431205+0000","last_active":"2024-09-20T22:57:51.376660+0000","last_peered":"2024-09-20T22:57:51.376660+0000","last_clean":"2024-09-20T22:57:51.376660+0000","last_became_active":"2024-09-20T22:57:49.427276+0000","last_became_peered":"2024-09-20T22:57:49.427276+0000","last_unstale":"2024-09-20T22:57:51.376660+0000","last_undegraded":"2024-09-20T22:57:51.376660+0000","last_fullsized":"2024-09-20T22:57:51.376660+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'20","last_scrub_stamp":"2024-09-20T22:55:23.474362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:55:23.474362+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:55.630391+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":934,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-20T22:57:52.373228+0000","last_change":"2024-09-20T22:56:08.372321+0000","last_active":"2024-09-20T22:57:52.373228+0000","last_peered":"2024-09-20T22:57:52.373228+0000","last_clean":"2024-09-20T22:57:52.373228+0000","last_became_active":"2024-09-20T22:56:08.371946+0000","last_became_peered":"2024-09-20T22:56:08.371946+0000","last_unstale":"2024-09-20T22:57:52.373228+0000","last_undegraded":"2024-09-20T22:57:52.373228+0000","last_fullsized":"2024-09-20T22:57:52.373228+0000","mapping_epoch":887,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:55:49.897082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:55:49.897082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:55.310026+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.15","version":"778'12","reported_seq":2159,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-20T22:57:52.376557+0000","last_change":"2024-09-20T22:57:36.726648+0000","last_active":"2024-09-20T22:57:52.376557+0000","last_peered":"2024-09-20T22:57:52.376557+0000","last_clean":"2024-09-20T22:57:52.376557+0000","last_became_active":"2024-09-20T22:54:17.341014+0000","last_became_peered":"2024-09-20T22:54:17.341014+0000","last_unstale":"2024-09-20T22:57:52.376557+0000","last_undegraded":"2024-09-20T22:57:52.376557+0000","last_fullsized":"2024-09-20T22:57:52.376557+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"778'12","last_scrub_stamp":"2024-09-20T22:57:36.726568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:57:36.726568+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:52.611026+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":1268,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-20T22:57:52.373593+0000","last_change":"2024-09-20T22:57:49.412393+0000","last_active":"2024-09-20T22:57:52.373593+0000","last_peered":"2024-09-20T22:57:52.373593+0000","last_clean":"2024-09-20T22:57:52.373593+0000","last_became_active":"2024-09-20T22:57:49.411010+0000","last_became_peered":"2024-09-20T22:57:49.411010+0000","last_unstale":"2024-09-20T22:57:52.373593+0000","last_undegraded":"2024-09-20T22:57:52.373593+0000","last_fullsized":"2024-09-20T22:57:52.373593+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:56:49.691267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:50:21.853070+0000","last_clean_scrub_stamp":"2024-09-20T22:56:49.691267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:02.273990+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.10","version":"989'1011","reported_seq":5439,"reported_epoch":991,"state":"active+clean","last_fresh":"2024-09-20T22:57:51.375932+0000","last_change":"2024-09-20T22:57:49.606967+0000","last_active":"2024-09-20T22:57:51.375932+0000","last_peered":"2024-09-20T22:57:51.375932+0000","last_clean":"2024-09-20T22:57:51.375932+0000","last_became_active":"2024-09-20T22:57:49.398604+0000","last_became_peered":"2024-09-20T22:57:49.398604+0000","last_unstale":"2024-09-20T22:57:51.375932+0000","last_undegraded":"2024-09-20T22:57:51.375932+0000","last_fullsized":"2024-09-20T22:57:51.375932+0000","mapping_epoch":988,"log_start":"480'253","ondisk_log_start":"480'253","created":932,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":5,"last_scrub":"823'833","last_scrub_stamp":"2024-09-20T22:55:07.264241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:55:07.264241+0000","objects_scrubbed":25,"log_size":758,"log_dups_size":383,"ondisk_log_size":758,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:55.629658+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.080082789000000001,"stat_sum":{"num_bytes":21422199,"num_objects":10,"num_object_clones":8,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":1,"num_read":166,"num_read_kb":171898,"num_write":132,"num_write_kb":45902,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":74379213,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"16b","length":"1"},{"start":"173","length":"1"},{"start":"17d","length":"1"},{"start":"180","length":"1"},{"start":"182","length":"3"},{"start":"186","length":"8"},{"start":"190","length":"10"},{"start":"1a1","length":"1"},{"start":"1a3","length":"2"},{"start":"1a6","length":"1"},{"start":"1aa","length":"2"},{"start":"1af","length":"1"},{"start":"1b5","length":"1"},{"start":"1b7","length":"1"},{"start":"1ba","length":"1"},{"start":"1bd","length":"2"},{"start":"1c0","length":"2"},{"start":"1c5","length":"1"},{"start":"1c7","length":"1"}]},{"pgid":"4.13","version":"625'19","reported_seq":835,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-20T22:57:52.568537+0000","last_change":"2024-09-20T22:55:55.832120+0000","last_active":"2024-09-20T22:57:52.568537+0000","last_peered":"2024-09-20T22:57:52.568537+0000","last_clean":"2024-09-20T22:57:52.568537+0000","last_became_active":"2024-09-20T22:55:17.342351+0000","last_became_peered":"2024-09-20T22:55:17.342351+0000","last_unstale":"2024-09-20T22:57:52.568537+0000","last_undegraded":"2024-09-20T22:57:52.568537+0000","last_fullsized":"2024-09-20T22:57:52.568537+0000","mapping_epoch":836,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":837,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'19","last_scrub_stamp":"2024-09-20T22:55:55.832023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:55:55.832023+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:55.294018+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":23,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":736,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-20T22:57:52.567383+0000","last_change":"2024-09-20T22:57:15.158099+0000","last_active":"2024-09-20T22:57:52.567383+0000","last_peered":"2024-09-20T22:57:52.567383+0000","last_clean":"2024-09-20T22:57:52.567383+0000","last_became_active":"2024-09-20T22:56:08.369206+0000","last_became_peered":"2024-09-20T22:56:08.369206+0000","last_unstale":"2024-09-20T22:57:52.567383+0000","last_undegraded":"2024-09-20T22:57:52.567383+0000","last_fullsized":"2024-09-20T22:57:52.567383+0000","mapping_epoch":887,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:57:15.158013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:52:47.717490+0000","last_clean_scrub_stamp":"2024-09-20T22:57:15.158013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:32.667409+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":457,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-20T22:57:52.377578+0000","last_change":"2024-09-20T22:57:49.398684+0000","last_active":"2024-09-20T22:57:52.377578+0000","last_peered":"2024-09-20T22:57:52.377578+0000","last_clean":"2024-09-20T22:57:52.377578+0000","last_became_active":"2024-09-20T22:57:49.398369+0000","last_became_peered":"2024-09-20T22:57:49.398369+0000","last_unstale":"2024-09-20T22:57:52.377578+0000","last_undegraded":"2024-09-20T22:57:52.377578+0000","last_fullsized":"2024-09-20T22:57:52.377578+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:55:03.853725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:55:03.853725+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":1740,"reported_epoch":991,"state":"active+clean","last_fresh":"2024-09-20T22:57:51.376065+0000","last_change":"2024-09-20T22:57:49.416021+0000","last_active":"2024-09-20T22:57:51.376065+0000","last_peered":"2024-09-20T22:57:51.376065+0000","last_clean":"2024-09-20T22:57:51.376065+0000","last_became_active":"2024-09-20T22:57:49.413148+0000","last_became_peered":"2024-09-20T22:57:49.413148+0000","last_unstale":"2024-09-20T22:57:51.376065+0000","last_undegraded":"2024-09-20T22:57:51.376065+0000","last_fullsized":"2024-09-20T22:57:51.376065+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:56:22.490523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:56:22.490523+0000","last_clean_scrub_stamp":"2024-09-20T22:56:22.490523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:55.630816+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"929'565","reported_seq":4991,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-20T22:57:52.567946+0000","last_change":"2024-09-20T22:57:50.386505+0000","last_active":"2024-09-20T22:57:52.567946+0000","last_peered":"2024-09-20T22:57:52.567946+0000","last_clean":"2024-09-20T22:57:52.567946+0000","last_became_active":"2024-09-20T22:57:50.385308+0000","last_became_peered":"2024-09-20T22:57:50.385308+0000","last_unstale":"2024-09-20T22:57:52.567946+0000","last_undegraded":"2024-09-20T22:57:52.567946+0000","last_fullsized":"2024-09-20T22:57:52.567946+0000","mapping_epoch":989,"log_start":"474'259","ondisk_log_start":"474'259","created":932,"last_epoch_clean":990,"parent":"0.0","parent_split_bits":5,"last_scrub":"929'565","last_scrub_stamp":"2024-09-20T22:57:33.220990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:57:33.220990+0000","objects_scrubbed":0,"log_size":306,"log_dups_size":338,"ondisk_log_size":306,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:02.936257+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00070306199999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"16b","length":"1"},{"start":"173","length":"1"},{"start":"17d","length":"1"},{"start":"180","length":"1"},{"start":"182","length":"3"},{"start":"186","length":"8"},{"start":"190","length":"10"},{"start":"1a1","length":"1"},{"start":"1a3","length":"2"},{"start":"1a6","length":"1"},{"start":"1aa","length":"2"},{"start":"1af","length":"1"},{"start":"1b5","length":"1"},{"start":"1b7","length":"1"},{"start":"1ba","length":"1"},{"start":"1bd","length":"2"},{"start":"1c0","length":"2"},{"start":"1c5","length":"1"},{"start":"1c7","length":"1"}]},{"pgid":"4.12","version":"625'19","reported_seq":458,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-20T22:57:52.377780+0000","last_change":"2024-09-20T22:55:33.932286+0000","last_active":"2024-09-20T22:57:52.377780+0000","last_peered":"2024-09-20T22:57:52.377780+0000","last_clean":"2024-09-20T22:57:52.377780+0000","last_became_active":"2024-09-20T22:54:17.340539+0000","last_became_peered":"2024-09-20T22:54:17.340539+0000","last_unstale":"2024-09-20T22:57:52.377780+0000","last_undegraded":"2024-09-20T22:57:52.377780+0000","last_fullsized":"2024-09-20T22:57:52.377780+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'19","last_scrub_stamp":"2024-09-20T22:55:33.932211+0000","last_deep_scrub":"404'13","last_deep_scrub_stamp":"2024-09-20T22:48:04.276282+0000","last_clean_scrub_stamp":"2024-09-20T22:55:33.932211+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":834,"reported_epoch":991,"state":"active+clean","last_fresh":"2024-09-20T22:57:51.376726+0000","last_change":"2024-09-20T22:57:49.406406+0000","last_active":"2024-09-20T22:57:51.376726+0000","last_peered":"2024-09-20T22:57:51.376726+0000","last_clean":"2024-09-20T22:57:51.376726+0000","last_became_active":"2024-09-20T22:57:49.403043+0000","last_became_peered":"2024-09-20T22:57:49.403043+0000","last_unstale":"2024-09-20T22:57:51.376726+0000","last_undegraded":"2024-09-20T22:57:51.376726+0000","last_fullsized":"2024-09-20T22:57:51.376726+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:54:46.541109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:31.138067+0000","last_clean_scrub_stamp":"2024-09-20T22:54:46.541109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:55.630318+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":1746,"reported_epoch":991,"state":"active+clean","last_fresh":"2024-09-20T22:57:51.376149+0000","last_change":"2024-09-20T22:57:49.433147+0000","last_active":"2024-09-20T22:57:51.376149+0000","last_peered":"2024-09-20T22:57:51.376149+0000","last_clean":"2024-09-20T22:57:51.376149+0000","last_became_active":"2024-09-20T22:57:49.432838+0000","last_became_peered":"2024-09-20T22:57:49.432838+0000","last_unstale":"2024-09-20T22:57:51.376149+0000","last_undegraded":"2024-09-20T22:57:51.376149+0000","last_fullsized":"2024-09-20T22:57:51.376149+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:34.345077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:53:34.345077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:55.629845+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":1265,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-20T22:57:52.372204+0000","last_change":"2024-09-20T22:57:38.552117+0000","last_active":"2024-09-20T22:57:52.372204+0000","last_peered":"2024-09-20T22:57:52.372204+0000","last_clean":"2024-09-20T22:57:52.372204+0000","last_became_active":"2024-09-20T22:54:17.344021+0000","last_became_peered":"2024-09-20T22:54:17.344021+0000","last_unstale":"2024-09-20T22:57:52.372204+0000","last_undegraded":"2024-09-20T22:57:52.372204+0000","last_fullsized":"2024-09-20T22:57:52.372204+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:57:38.552046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:55:09.664839+0000","last_clean_scrub_stamp":"2024-09-20T22:57:38.552046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:07.271978+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.12","version":"988'1150","reported_seq":5054,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-20T22:57:52.567698+0000","last_change":"2024-09-20T22:57:48.417172+0000","last_active":"2024-09-20T22:57:52.567698+0000","last_peered":"2024-09-20T22:57:52.567698+0000","last_clean":"2024-09-20T22:57:52.567698+0000","last_became_active":"2024-09-20T22:57:03.429039+0000","last_became_peered":"2024-09-20T22:57:03.429039+0000","last_unstale":"2024-09-20T22:57:52.567698+0000","last_undegraded":"2024-09-20T22:57:52.567698+0000","last_fullsized":"2024-09-20T22:57:52.567698+0000","mapping_epoch":942,"log_start":"471'464","ondisk_log_start":"471'464","created":932,"last_epoch_clean":943,"parent":"0.0","parent_split_bits":5,"last_scrub":"943'1027","last_scrub_stamp":"2024-09-20T22:57:05.204065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:57:05.204065+0000","objects_scrubbed":15,"log_size":686,"log_dups_size":687,"ondisk_log_size":686,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:17.391441+0000","scrub_duration":6,"objects_trimmed":2,"snaptrim_duration":0.046814117000000002,"stat_sum":{"num_bytes":45111058,"num_objects":18,"num_object_clones":15,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":1,"num_read":29,"num_read_kb":39201,"num_write":15,"num_write_kb":14614,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"16b","length":"1"},{"start":"173","length":"1"},{"start":"17d","length":"1"},{"start":"180","length":"1"},{"start":"182","length":"3"},{"start":"186","length":"8"},{"start":"190","length":"10"},{"start":"1a1","length":"1"},{"start":"1a3","length":"2"},{"start":"1a6","length":"1"},{"start":"1aa","length":"2"},{"start":"1af","length":"1"},{"start":"1b5","length":"1"},{"start":"1b7","length":"1"},{"start":"1ba","length":"1"},{"start":"1bd","length":"2"},{"start":"1c0","length":"2"},{"start":"1c5","length":"1"},{"start":"1c7","length":"1"}]},{"pgid":"4.11","version":"625'24","reported_seq":123,"reported_epoch":991,"state":"active+clean","last_fresh":"2024-09-20T22:57:51.376152+0000","last_change":"2024-09-20T22:57:49.414600+0000","last_active":"2024-09-20T22:57:51.376152+0000","last_peered":"2024-09-20T22:57:51.376152+0000","last_clean":"2024-09-20T22:57:51.376152+0000","last_became_active":"2024-09-20T22:57:49.408666+0000","last_became_peered":"2024-09-20T22:57:49.408666+0000","last_unstale":"2024-09-20T22:57:51.376152+0000","last_undegraded":"2024-09-20T22:57:51.376152+0000","last_fullsized":"2024-09-20T22:57:51.376152+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'24","last_scrub_stamp":"2024-09-20T22:53:33.089580+0000","last_deep_scrub":"470'16","last_deep_scrub_stamp":"2024-09-20T22:49:10.659868+0000","last_clean_scrub_stamp":"2024-09-20T22:53:33.089580+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:55.629882+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":1938,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-20T22:57:52.376876+0000","last_change":"2024-09-20T22:57:49.405502+0000","last_active":"2024-09-20T22:57:52.376876+0000","last_peered":"2024-09-20T22:57:52.376876+0000","last_clean":"2024-09-20T22:57:52.376876+0000","last_became_active":"2024-09-20T22:57:49.405256+0000","last_became_peered":"2024-09-20T22:57:49.405256+0000","last_unstale":"2024-09-20T22:57:52.376876+0000","last_undegraded":"2024-09-20T22:57:52.376876+0000","last_fullsized":"2024-09-20T22:57:52.376876+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:55:54.521725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:45:24.765414+0000","last_clean_scrub_stamp":"2024-09-20T22:55:54.521725+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:54.750873+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":936,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-20T22:57:52.567489+0000","last_change":"2024-09-20T22:57:36.239427+0000","last_active":"2024-09-20T22:57:52.567489+0000","last_peered":"2024-09-20T22:57:52.567489+0000","last_clean":"2024-09-20T22:57:52.567489+0000","last_became_active":"2024-09-20T22:56:08.355950+0000","last_became_peered":"2024-09-20T22:56:08.355950+0000","last_unstale":"2024-09-20T22:57:52.567489+0000","last_undegraded":"2024-09-20T22:57:52.567489+0000","last_fullsized":"2024-09-20T22:57:52.567489+0000","mapping_epoch":887,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:57:36.239352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:50.174861+0000","last_clean_scrub_stamp":"2024-09-20T22:57:36.239352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:38.372074+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":1738,"reported_epoch":991,"state":"active+clean","last_fresh":"2024-09-20T22:57:51.377153+0000","last_change":"2024-09-20T22:57:49.419702+0000","last_active":"2024-09-20T22:57:51.377153+0000","last_peered":"2024-09-20T22:57:51.377153+0000","last_clean":"2024-09-20T22:57:51.377153+0000","last_became_active":"2024-09-20T22:57:49.410970+0000","last_became_peered":"2024-09-20T22:57:49.410970+0000","last_unstale":"2024-09-20T22:57:51.377153+0000","last_undegraded":"2024-09-20T22:57:51.377153+0000","last_fullsized":"2024-09-20T22:57:51.377153+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:54:22.276321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:54:22.276321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:55.630143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"988'504","reported_seq":4889,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-20T22:57:52.372770+0000","last_change":"2024-09-20T22:57:48.420084+0000","last_active":"2024-09-20T22:57:52.372770+0000","last_peered":"2024-09-20T22:57:52.372770+0000","last_clean":"2024-09-20T22:57:52.372770+0000","last_became_active":"2024-09-20T22:57:03.429099+0000","last_became_peered":"2024-09-20T22:57:03.429099+0000","last_unstale":"2024-09-20T22:57:52.372770+0000","last_undegraded":"2024-09-20T22:57:52.372770+0000","last_fullsized":"2024-09-20T22:57:52.372770+0000","mapping_epoch":942,"log_start":"456'129","ondisk_log_start":"456'129","created":932,"last_epoch_clean":943,"parent":"0.0","parent_split_bits":5,"last_scrub":"944'488","last_scrub_stamp":"2024-09-20T22:57:04.743360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:57:04.743360+0000","objects_scrubbed":9,"log_size":375,"log_dups_size":149,"ondisk_log_size":375,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:31.671359+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.051105540999999997,"stat_sum":{"num_bytes":18736221,"num_objects":8,"num_object_clones":7,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":19,"num_read_kb":37757,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"16b","length":"1"},{"start":"173","length":"1"},{"start":"17d","length":"1"},{"start":"180","length":"1"},{"start":"182","length":"3"},{"start":"186","length":"8"},{"start":"190","length":"10"},{"start":"1a1","length":"1"},{"start":"1a3","length":"2"},{"start":"1a6","length":"1"},{"start":"1aa","length":"2"},{"start":"1af","length":"1"},{"start":"1b5","length":"1"},{"start":"1b7","length":"1"},{"start":"1ba","length":"1"},{"start":"1bd","length":"2"},{"start":"1c0","length":"2"},{"start":"1c5","length":"1"},{"start":"1c7","length":"1"}]},{"pgid":"4.10","version":"625'8","reported_seq":1810,"reported_epoch":991,"state":"active+clean","last_fresh":"2024-09-20T22:57:51.375798+0000","last_change":"2024-09-20T22:57:49.412934+0000","last_active":"2024-09-20T22:57:51.375798+0000","last_peered":"2024-09-20T22:57:51.375798+0000","last_clean":"2024-09-20T22:57:51.375798+0000","last_became_active":"2024-09-20T22:57:49.401068+0000","last_became_peered":"2024-09-20T22:57:49.401068+0000","last_unstale":"2024-09-20T22:57:51.375798+0000","last_undegraded":"2024-09-20T22:57:51.375798+0000","last_fullsized":"2024-09-20T22:57:51.375798+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'8","last_scrub_stamp":"2024-09-20T22:56:42.475598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:56:42.475598+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:55.630931+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":1262,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-20T22:57:52.374360+0000","last_change":"2024-09-20T22:57:49.415511+0000","last_active":"2024-09-20T22:57:52.374360+0000","last_peered":"2024-09-20T22:57:52.374360+0000","last_clean":"2024-09-20T22:57:52.374360+0000","last_became_active":"2024-09-20T22:57:49.412134+0000","last_became_peered":"2024-09-20T22:57:49.412134+0000","last_unstale":"2024-09-20T22:57:52.374360+0000","last_undegraded":"2024-09-20T22:57:52.374360+0000","last_fullsized":"2024-09-20T22:57:52.374360+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:55:11.688508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:55:11.688508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:56.680321+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":1954,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-20T22:57:52.568188+0000","last_change":"2024-09-20T22:56:39.112737+0000","last_active":"2024-09-20T22:57:52.568188+0000","last_peered":"2024-09-20T22:57:52.568188+0000","last_clean":"2024-09-20T22:57:52.568188+0000","last_became_active":"2024-09-20T22:56:08.357087+0000","last_became_peered":"2024-09-20T22:56:08.357087+0000","last_unstale":"2024-09-20T22:57:52.568188+0000","last_undegraded":"2024-09-20T22:57:52.568188+0000","last_fullsized":"2024-09-20T22:57:52.568188+0000","mapping_epoch":887,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:56:39.112639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:56:39.112639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:57.269193+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":931,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-20T22:57:52.373152+0000","last_change":"2024-09-20T22:57:49.398452+0000","last_active":"2024-09-20T22:57:52.373152+0000","last_peered":"2024-09-20T22:57:52.373152+0000","last_clean":"2024-09-20T22:57:52.373152+0000","last_became_active":"2024-09-20T22:57:49.398162+0000","last_became_peered":"2024-09-20T22:57:49.398162+0000","last_unstale":"2024-09-20T22:57:52.373152+0000","last_undegraded":"2024-09-20T22:57:52.373152+0000","last_fullsized":"2024-09-20T22:57:52.373152+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:56:40.102311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:55:08.681729+0000","last_clean_scrub_stamp":"2024-09-20T22:56:40.102311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:08.020100+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":456,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-20T22:57:52.378297+0000","last_change":"2024-09-20T22:57:09.252912+0000","last_active":"2024-09-20T22:57:52.378297+0000","last_peered":"2024-09-20T22:57:52.378297+0000","last_clean":"2024-09-20T22:57:52.378297+0000","last_became_active":"2024-09-20T22:54:17.348206+0000","last_became_peered":"2024-09-20T22:54:17.348206+0000","last_unstale":"2024-09-20T22:57:52.378297+0000","last_undegraded":"2024-09-20T22:57:52.378297+0000","last_fullsized":"2024-09-20T22:57:52.378297+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:57:09.252832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:57:09.252832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:21.209966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":1768,"reported_epoch":991,"state":"active+clean","last_fresh":"2024-09-20T22:57:51.375811+0000","last_change":"2024-09-20T22:57:49.413803+0000","last_active":"2024-09-20T22:57:51.375811+0000","last_peered":"2024-09-20T22:57:51.375811+0000","last_clean":"2024-09-20T22:57:51.375811+0000","last_became_active":"2024-09-20T22:57:49.407194+0000","last_became_peered":"2024-09-20T22:57:49.407194+0000","last_unstale":"2024-09-20T22:57:51.375811+0000","last_undegraded":"2024-09-20T22:57:51.375811+0000","last_fullsized":"2024-09-20T22:57:51.375811+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:31.274390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:53:31.274390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:55.629932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"625'19","reported_seq":1999,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-20T22:57:52.569065+0000","last_change":"2024-09-20T22:56:18.981584+0000","last_active":"2024-09-20T22:57:52.569065+0000","last_peered":"2024-09-20T22:57:52.569065+0000","last_clean":"2024-09-20T22:57:52.569065+0000","last_became_active":"2024-09-20T22:54:17.342236+0000","last_became_peered":"2024-09-20T22:54:17.342236+0000","last_unstale":"2024-09-20T22:57:52.569065+0000","last_undegraded":"2024-09-20T22:57:52.569065+0000","last_fullsized":"2024-09-20T22:57:52.569065+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'19","last_scrub_stamp":"2024-09-20T22:56:18.981492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:56:18.981492+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:55.294064+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":1260,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-20T22:57:52.374185+0000","last_change":"2024-09-20T22:57:08.720376+0000","last_active":"2024-09-20T22:57:52.374185+0000","last_peered":"2024-09-20T22:57:52.374185+0000","last_clean":"2024-09-20T22:57:52.374185+0000","last_became_active":"2024-09-20T22:56:08.391532+0000","last_became_peered":"2024-09-20T22:56:08.391532+0000","last_unstale":"2024-09-20T22:57:52.374185+0000","last_undegraded":"2024-09-20T22:57:52.374185+0000","last_fullsized":"2024-09-20T22:57:52.374185+0000","mapping_epoch":887,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:57:08.720315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:54:43.813540+0000","last_clean_scrub_stamp":"2024-09-20T22:57:08.720315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:09.299138+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":26,"stat_sum":{"num_bytes":831131791,"num_objects":354,"num_object_clones":303,"num_object_copies":1062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":354,"num_whiteouts":10,"num_read":5084,"num_read_kb":5338884,"num_write":3533,"num_write_kb":1374821,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":951,"num_bytes_recovered":695972422,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2086481920,"data_stored":2693638501,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":25797,"ondisk_log_size":25797,"up":78,"acting":78,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1097,"num_read_kb":854,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":437,"num_bytes_recovered":10204,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":462848,"data_stored":12360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":770,"ondisk_log_size":770,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":1034,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":4931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":850,"num_read_kb":845,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":1594,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":385,"ondisk_log_size":385,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":0,"seq":253403070663,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430413,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":422496,"kb_used_data":337224,"kb_used_omap":1,"kb_used_meta":85246,"kb_avail":93322656,"statfs":{"total":95995035648,"available":95562399744,"internally_reserved":0,"allocated":345317376,"data_stored":441108875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":87292358},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798692050,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":367320,"kb_used_data":295872,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93377832,"statfs":{"total":95995035648,"available":95618899968,"internally_reserved":0,"allocated":302972928,"data_stored":352786741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986391,"num_pgs":73,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":169912,"kb_used_data":124832,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":93575240,"statfs":{"total":95995035648,"available":95821045760,"internally_reserved":0,"allocated":127827968,"data_stored":165561056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":988,"seq":4243427688450,"num_pgs":80,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":392928,"kb_used_data":358152,"kb_used_omap":148,"kb_used_meta":34603,"kb_avail":93352224,"statfs":{"total":95995035648,"available":95592677376,"internally_reserved":0,"allocated":366747648,"data_stored":480202663,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":151956,"internal_metadata":35434092},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411554,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":448400,"kb_used_data":363128,"kb_used_omap":1,"kb_used_meta":85246,"kb_avail":93296752,"statfs":{"total":95995035648,"available":95535874048,"internally_reserved":0,"allocated":371843072,"data_stored":442887419,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":87292357},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575079,"num_pgs":80,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":271384,"kb_used_data":195776,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93473768,"statfs":{"total":95995035648,"available":95717138432,"internally_reserved":0,"allocated":200474624,"data_stored":252465356,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738604,"num_pgs":74,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":359772,"kb_used_data":274504,"kb_used_omap":1,"kb_used_meta":85246,"kb_avail":93385380,"statfs":{"total":95995035648,"available":95626629120,"internally_reserved":0,"allocated":281092096,"data_stored":362018782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":87292357},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":271130624,"data_stored":355911924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":190472192,"data_stored":246357923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":361631744,"data_stored":436518233,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":356196352,"data_stored":473511632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":117862400,"data_stored":159454854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":292409344,"data_stored":346088520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":335048704,"data_stored":434674605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":161730560,"data_stored":241120810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-20T22:57:56.133 INFO:tasks.cephadm.ceph_manager.ceph:pg 7.0 in trimming, state: active+clean+snaptrim 2024-09-20T22:57:56.134 INFO:tasks.cephadm.ceph_manager.ceph:pg 7.b in trimming, state: active+clean+snaptrim 2024-09-20T22:57:56.134 INFO:tasks.cephadm.ceph_manager.ceph:unique_pool_0 still trimming, waiting 2024-09-20T22:57:56.177 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:57:56.178 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:57:56.188 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:57:56.200 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:56.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:56.201 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:57:56.215 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-20T22:57:56.302 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:57:56.323 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:56.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:56.324 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:57:56.425 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:57:56.447 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:56.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:56.449 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:57:56.549 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:57:56.571 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:56.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:56.572 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:57:56.673 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:57:56.694 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:56.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:56.695 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:57:56.737 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:57:56.782 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-20T22:57:56.796 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:57:56.819 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:56.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:56.820 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:57:56.899 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:56 smithi084 ceph-mon[40024]: 5.a scrub starts 2024-09-20T22:57:56.899 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:56 smithi084 ceph-mon[40024]: 5.a scrub ok 2024-09-20T22:57:56.900 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:56 smithi084 ceph-mon[40024]: 5.12 scrub starts 2024-09-20T22:57:56.900 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:56 smithi084 ceph-mon[40024]: 5.12 scrub ok 2024-09-20T22:57:56.900 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:56 smithi084 ceph-mon[40024]: 7.10 scrub starts 2024-09-20T22:57:56.900 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:56 smithi084 ceph-mon[40024]: 7.10 scrub ok 2024-09-20T22:57:56.900 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:56 smithi084 ceph-mon[40024]: 7.6 scrub starts 2024-09-20T22:57:56.900 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:56 smithi084 ceph-mon[40024]: 7.6 scrub ok 2024-09-20T22:57:56.900 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:56 smithi084 ceph-mon[40024]: pgmap v1392: 168 pgs: 2 active+clean+snaptrim, 166 active+clean; 793 MiB data, 2.3 GiB used, 623 GiB / 626 GiB avail; 14 MiB/s, 0 objects/s recovering 2024-09-20T22:57:56.900 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:56 smithi084 ceph-mon[40024]: 7.7 scrub starts 2024-09-20T22:57:56.900 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:56 smithi084 ceph-mon[40024]: 7.7 scrub ok 2024-09-20T22:57:56.900 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-20T22:57:56.901 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-20T22:57:56.901 INFO:teuthology.orchestra.run.smithi084.stdout: "epoch": 992, 2024-09-20T22:57:56.901 INFO:teuthology.orchestra.run.smithi084.stdout: "fsid": "a9c0bc70-77a0-11ef-baf5-efdc52797490", 2024-09-20T22:57:56.901 INFO:teuthology.orchestra.run.smithi084.stdout: "created": "2024-09-20T22:36:30.240626+0000", 2024-09-20T22:57:56.901 INFO:teuthology.orchestra.run.smithi084.stdout: "modified": "2024-09-20T22:57:52.369055+0000", 2024-09-20T22:57:56.901 INFO:teuthology.orchestra.run.smithi084.stdout: "last_up_change": "2024-09-20T22:57:48.362816+0000", 2024-09-20T22:57:56.901 INFO:teuthology.orchestra.run.smithi084.stdout: "last_in_change": "2024-09-20T22:56:07.496853+0000", 2024-09-20T22:57:56.901 INFO:teuthology.orchestra.run.smithi084.stdout: "flags": "nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-20T22:57:56.901 INFO:teuthology.orchestra.run.smithi084.stdout: "flags_num": 5804032, 2024-09-20T22:57:56.901 INFO:teuthology.orchestra.run.smithi084.stdout: "flags_set": [ 2024-09-20T22:57:56.901 INFO:teuthology.orchestra.run.smithi084.stdout: "nodeep-scrub", 2024-09-20T22:57:56.901 INFO:teuthology.orchestra.run.smithi084.stdout: "pglog_hardlimit", 2024-09-20T22:57:56.901 INFO:teuthology.orchestra.run.smithi084.stdout: "purged_snapdirs", 2024-09-20T22:57:56.901 INFO:teuthology.orchestra.run.smithi084.stdout: "recovery_deletes", 2024-09-20T22:57:56.901 INFO:teuthology.orchestra.run.smithi084.stdout: "sortbitwise" 2024-09-20T22:57:56.902 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2024-09-20T22:57:56.902 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_version": 19, 2024-09-20T22:57:56.902 INFO:teuthology.orchestra.run.smithi084.stdout: "full_ratio": 0.94999998807907104, 2024-09-20T22:57:56.902 INFO:teuthology.orchestra.run.smithi084.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-20T22:57:56.902 INFO:teuthology.orchestra.run.smithi084.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-20T22:57:56.902 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_snapshot": "", 2024-09-20T22:57:56.902 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_max": 7, 2024-09-20T22:57:56.902 INFO:teuthology.orchestra.run.smithi084.stdout: "max_osd": 8, 2024-09-20T22:57:56.902 INFO:teuthology.orchestra.run.smithi084.stdout: "require_min_compat_client": "luminous", 2024-09-20T22:57:56.902 INFO:teuthology.orchestra.run.smithi084.stdout: "min_compat_client": "luminous", 2024-09-20T22:57:56.902 INFO:teuthology.orchestra.run.smithi084.stdout: "require_osd_release": "squid", 2024-09-20T22:57:56.902 INFO:teuthology.orchestra.run.smithi084.stdout: "allow_crimson": false, 2024-09-20T22:57:56.902 INFO:teuthology.orchestra.run.smithi084.stdout: "pools": [ 2024-09-20T22:57:56.902 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:56.903 INFO:teuthology.orchestra.run.smithi084.stdout: "pool": 1, 2024-09-20T22:57:56.903 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_name": ".mgr", 2024-09-20T22:57:56.903 INFO:teuthology.orchestra.run.smithi084.stdout: "create_time": "2024-09-20T22:39:21.984381+0000", 2024-09-20T22:57:56.903 INFO:teuthology.orchestra.run.smithi084.stdout: "flags": 1, 2024-09-20T22:57:56.903 INFO:teuthology.orchestra.run.smithi084.stdout: "flags_names": "hashpspool", 2024-09-20T22:57:56.903 INFO:teuthology.orchestra.run.smithi084.stdout: "type": 1, 2024-09-20T22:57:56.903 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 3, 2024-09-20T22:57:56.903 INFO:teuthology.orchestra.run.smithi084.stdout: "min_size": 2, 2024-09-20T22:57:56.903 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_rule": 0, 2024-09-20T22:57:56.903 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_count": 0, 2024-09-20T22:57:56.903 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_target": 0, 2024-09-20T22:57:56.903 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_barrier": 0, 2024-09-20T22:57:56.903 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-20T22:57:56.903 INFO:teuthology.orchestra.run.smithi084.stdout: "is_stretch_pool": false, 2024-09-20T22:57:56.904 INFO:teuthology.orchestra.run.smithi084.stdout: "object_hash": 2, 2024-09-20T22:57:56.904 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_autoscale_mode": "off", 2024-09-20T22:57:56.904 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num": 11, 2024-09-20T22:57:56.904 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num": 11, 2024-09-20T22:57:56.904 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num_target": 11, 2024-09-20T22:57:56.904 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_target": 11, 2024-09-20T22:57:56.904 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_pending": 11, 2024-09-20T22:57:56.904 INFO:teuthology.orchestra.run.smithi084.stdout: "last_pg_merge_meta": { 2024-09-20T22:57:56.904 INFO:teuthology.orchestra.run.smithi084.stdout: "source_pgid": "1.b", 2024-09-20T22:57:56.904 INFO:teuthology.orchestra.run.smithi084.stdout: "ready_epoch": 299, 2024-09-20T22:57:56.904 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_started": 298, 2024-09-20T22:57:56.904 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_clean": 298, 2024-09-20T22:57:56.904 INFO:teuthology.orchestra.run.smithi084.stdout: "source_version": "77'128", 2024-09-20T22:57:56.904 INFO:teuthology.orchestra.run.smithi084.stdout: "target_version": "77'128" 2024-09-20T22:57:56.904 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:56.905 INFO:teuthology.orchestra.run.smithi084.stdout: "last_change": "769", 2024-09-20T22:57:56.905 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend": "0", 2024-09-20T22:57:56.905 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_prenautilus": "299", 2024-09-20T22:57:56.905 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_preluminous": "297", 2024-09-20T22:57:56.905 INFO:teuthology.orchestra.run.smithi084.stdout: "auid": 0, 2024-09-20T22:57:56.905 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_mode": "selfmanaged", 2024-09-20T22:57:56.905 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_seq": 0, 2024-09-20T22:57:56.905 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_epoch": 0, 2024-09-20T22:57:56.905 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_snaps": [], 2024-09-20T22:57:56.905 INFO:teuthology.orchestra.run.smithi084.stdout: "removed_snaps": "[]", 2024-09-20T22:57:56.905 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_bytes": 0, 2024-09-20T22:57:56.905 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_objects": 0, 2024-09-20T22:57:56.905 INFO:teuthology.orchestra.run.smithi084.stdout: "tiers": [], 2024-09-20T22:57:56.905 INFO:teuthology.orchestra.run.smithi084.stdout: "tier_of": -1, 2024-09-20T22:57:56.906 INFO:teuthology.orchestra.run.smithi084.stdout: "read_tier": -1, 2024-09-20T22:57:56.906 INFO:teuthology.orchestra.run.smithi084.stdout: "write_tier": -1, 2024-09-20T22:57:56.906 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_mode": "none", 2024-09-20T22:57:56.906 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_bytes": 0, 2024-09-20T22:57:56.906 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_objects": 0, 2024-09-20T22:57:56.906 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-20T22:57:56.906 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-20T22:57:56.906 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-20T22:57:56.906 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_flush_age": 0, 2024-09-20T22:57:56.906 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_evict_age": 0, 2024-09-20T22:57:56.906 INFO:teuthology.orchestra.run.smithi084.stdout: "erasure_code_profile": "", 2024-09-20T22:57:56.906 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_params": { 2024-09-20T22:57:56.906 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "none" 2024-09-20T22:57:56.906 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:56.907 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_period": 0, 2024-09-20T22:57:56.907 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_count": 0, 2024-09-20T22:57:56.907 INFO:teuthology.orchestra.run.smithi084.stdout: "use_gmt_hitset": true, 2024-09-20T22:57:56.907 INFO:teuthology.orchestra.run.smithi084.stdout: "min_read_recency_for_promote": 0, 2024-09-20T22:57:56.907 INFO:teuthology.orchestra.run.smithi084.stdout: "min_write_recency_for_promote": 0, 2024-09-20T22:57:56.907 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_grade_decay_rate": 0, 2024-09-20T22:57:56.907 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_search_last_n": 0, 2024-09-20T22:57:56.907 INFO:teuthology.orchestra.run.smithi084.stdout: "grade_table": [], 2024-09-20T22:57:56.907 INFO:teuthology.orchestra.run.smithi084.stdout: "stripe_width": 0, 2024-09-20T22:57:56.907 INFO:teuthology.orchestra.run.smithi084.stdout: "expected_num_objects": 0, 2024-09-20T22:57:56.907 INFO:teuthology.orchestra.run.smithi084.stdout: "fast_read": false, 2024-09-20T22:57:56.907 INFO:teuthology.orchestra.run.smithi084.stdout: "options": { 2024-09-20T22:57:56.907 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_max": 32, 2024-09-20T22:57:56.908 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_min": 1 2024-09-20T22:57:56.908 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:56.908 INFO:teuthology.orchestra.run.smithi084.stdout: "application_metadata": { 2024-09-20T22:57:56.908 INFO:teuthology.orchestra.run.smithi084.stdout: "mgr": {} 2024-09-20T22:57:56.908 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:56.908 INFO:teuthology.orchestra.run.smithi084.stdout: "read_balance": { 2024-09-20T22:57:56.908 INFO:teuthology.orchestra.run.smithi084.stdout: "score_type": "Fair distribution", 2024-09-20T22:57:56.908 INFO:teuthology.orchestra.run.smithi084.stdout: "score_acting": 1.9099999666213989, 2024-09-20T22:57:56.908 INFO:teuthology.orchestra.run.smithi084.stdout: "score_stable": 1.9099999666213989, 2024-09-20T22:57:56.908 INFO:teuthology.orchestra.run.smithi084.stdout: "optimal_score": 1, 2024-09-20T22:57:56.908 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_acting": 1.9099999666213989, 2024-09-20T22:57:56.908 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_stable": 1.9099999666213989, 2024-09-20T22:57:56.908 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity_weighted": 0.875, 2024-09-20T22:57:56.908 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity": 1, 2024-09-20T22:57:56.909 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-09-20T22:57:56.909 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:57:56.909 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:56.909 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:56.909 INFO:teuthology.orchestra.run.smithi084.stdout: "pool": 2, 2024-09-20T22:57:56.909 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_name": "datapool", 2024-09-20T22:57:56.909 INFO:teuthology.orchestra.run.smithi084.stdout: "create_time": "2024-09-20T22:41:35.820081+0000", 2024-09-20T22:57:56.909 INFO:teuthology.orchestra.run.smithi084.stdout: "flags": 8193, 2024-09-20T22:57:56.909 INFO:teuthology.orchestra.run.smithi084.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-20T22:57:56.909 INFO:teuthology.orchestra.run.smithi084.stdout: "type": 1, 2024-09-20T22:57:56.909 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 3, 2024-09-20T22:57:56.909 INFO:teuthology.orchestra.run.smithi084.stdout: "min_size": 2, 2024-09-20T22:57:56.909 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_rule": 0, 2024-09-20T22:57:56.909 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_count": 0, 2024-09-20T22:57:56.910 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_target": 0, 2024-09-20T22:57:56.910 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_barrier": 0, 2024-09-20T22:57:56.910 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-20T22:57:56.910 INFO:teuthology.orchestra.run.smithi084.stdout: "is_stretch_pool": false, 2024-09-20T22:57:56.910 INFO:teuthology.orchestra.run.smithi084.stdout: "object_hash": 2, 2024-09-20T22:57:56.910 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_autoscale_mode": "off", 2024-09-20T22:57:56.910 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num": 3, 2024-09-20T22:57:56.910 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num": 3, 2024-09-20T22:57:56.910 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num_target": 3, 2024-09-20T22:57:56.910 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_target": 3, 2024-09-20T22:57:56.910 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_pending": 3, 2024-09-20T22:57:56.910 INFO:teuthology.orchestra.run.smithi084.stdout: "last_pg_merge_meta": { 2024-09-20T22:57:56.910 INFO:teuthology.orchestra.run.smithi084.stdout: "source_pgid": "0.0", 2024-09-20T22:57:56.910 INFO:teuthology.orchestra.run.smithi084.stdout: "ready_epoch": 0, 2024-09-20T22:57:56.910 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_started": 0, 2024-09-20T22:57:56.911 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_clean": 0, 2024-09-20T22:57:56.911 INFO:teuthology.orchestra.run.smithi084.stdout: "source_version": "0'0", 2024-09-20T22:57:56.911 INFO:teuthology.orchestra.run.smithi084.stdout: "target_version": "0'0" 2024-09-20T22:57:56.911 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:56.911 INFO:teuthology.orchestra.run.smithi084.stdout: "last_change": "69", 2024-09-20T22:57:56.911 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend": "0", 2024-09-20T22:57:56.911 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-20T22:57:56.911 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_preluminous": "0", 2024-09-20T22:57:56.911 INFO:teuthology.orchestra.run.smithi084.stdout: "auid": 0, 2024-09-20T22:57:56.911 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_mode": "selfmanaged", 2024-09-20T22:57:56.911 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_seq": 2, 2024-09-20T22:57:56.911 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_epoch": 69, 2024-09-20T22:57:56.911 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_snaps": [], 2024-09-20T22:57:56.911 INFO:teuthology.orchestra.run.smithi084.stdout: "removed_snaps": "[]", 2024-09-20T22:57:56.912 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_bytes": 0, 2024-09-20T22:57:56.912 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_objects": 0, 2024-09-20T22:57:56.912 INFO:teuthology.orchestra.run.smithi084.stdout: "tiers": [], 2024-09-20T22:57:56.912 INFO:teuthology.orchestra.run.smithi084.stdout: "tier_of": -1, 2024-09-20T22:57:56.912 INFO:teuthology.orchestra.run.smithi084.stdout: "read_tier": -1, 2024-09-20T22:57:56.912 INFO:teuthology.orchestra.run.smithi084.stdout: "write_tier": -1, 2024-09-20T22:57:56.912 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_mode": "none", 2024-09-20T22:57:56.912 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_bytes": 0, 2024-09-20T22:57:56.912 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_objects": 0, 2024-09-20T22:57:56.912 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-20T22:57:56.912 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-20T22:57:56.912 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-20T22:57:56.912 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_flush_age": 0, 2024-09-20T22:57:56.912 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_evict_age": 0, 2024-09-20T22:57:56.913 INFO:teuthology.orchestra.run.smithi084.stdout: "erasure_code_profile": "", 2024-09-20T22:57:56.913 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_params": { 2024-09-20T22:57:56.913 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "none" 2024-09-20T22:57:56.913 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:56.913 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_period": 0, 2024-09-20T22:57:56.913 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_count": 0, 2024-09-20T22:57:56.913 INFO:teuthology.orchestra.run.smithi084.stdout: "use_gmt_hitset": true, 2024-09-20T22:57:56.913 INFO:teuthology.orchestra.run.smithi084.stdout: "min_read_recency_for_promote": 0, 2024-09-20T22:57:56.913 INFO:teuthology.orchestra.run.smithi084.stdout: "min_write_recency_for_promote": 0, 2024-09-20T22:57:56.913 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_grade_decay_rate": 0, 2024-09-20T22:57:56.913 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_search_last_n": 0, 2024-09-20T22:57:56.913 INFO:teuthology.orchestra.run.smithi084.stdout: "grade_table": [], 2024-09-20T22:57:56.913 INFO:teuthology.orchestra.run.smithi084.stdout: "stripe_width": 0, 2024-09-20T22:57:56.913 INFO:teuthology.orchestra.run.smithi084.stdout: "expected_num_objects": 0, 2024-09-20T22:57:56.914 INFO:teuthology.orchestra.run.smithi084.stdout: "fast_read": false, 2024-09-20T22:57:56.914 INFO:teuthology.orchestra.run.smithi084.stdout: "options": {}, 2024-09-20T22:57:56.914 INFO:teuthology.orchestra.run.smithi084.stdout: "application_metadata": { 2024-09-20T22:57:56.914 INFO:teuthology.orchestra.run.smithi084.stdout: "rbd": {} 2024-09-20T22:57:56.914 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:56.914 INFO:teuthology.orchestra.run.smithi084.stdout: "read_balance": { 2024-09-20T22:57:56.914 INFO:teuthology.orchestra.run.smithi084.stdout: "score_type": "Fair distribution", 2024-09-20T22:57:56.914 INFO:teuthology.orchestra.run.smithi084.stdout: "score_acting": 2.3299999237060547, 2024-09-20T22:57:56.914 INFO:teuthology.orchestra.run.smithi084.stdout: "score_stable": 2.3299999237060547, 2024-09-20T22:57:56.914 INFO:teuthology.orchestra.run.smithi084.stdout: "optimal_score": 0.86000001430511475, 2024-09-20T22:57:56.914 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_acting": 2, 2024-09-20T22:57:56.914 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_stable": 2, 2024-09-20T22:57:56.914 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity_weighted": 0.875, 2024-09-20T22:57:56.914 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity": 1, 2024-09-20T22:57:56.914 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-09-20T22:57:56.915 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:57:56.915 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:56.915 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:56.915 INFO:teuthology.orchestra.run.smithi084.stdout: "pool": 3, 2024-09-20T22:57:56.915 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_name": ".rgw.root", 2024-09-20T22:57:56.915 INFO:teuthology.orchestra.run.smithi084.stdout: "create_time": "2024-09-20T22:41:36.546389+0000", 2024-09-20T22:57:56.915 INFO:teuthology.orchestra.run.smithi084.stdout: "flags": 1, 2024-09-20T22:57:56.915 INFO:teuthology.orchestra.run.smithi084.stdout: "flags_names": "hashpspool", 2024-09-20T22:57:56.915 INFO:teuthology.orchestra.run.smithi084.stdout: "type": 1, 2024-09-20T22:57:56.915 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 3, 2024-09-20T22:57:56.915 INFO:teuthology.orchestra.run.smithi084.stdout: "min_size": 2, 2024-09-20T22:57:56.915 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_rule": 0, 2024-09-20T22:57:56.915 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_count": 0, 2024-09-20T22:57:56.915 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_target": 0, 2024-09-20T22:57:56.915 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_barrier": 0, 2024-09-20T22:57:56.915 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-20T22:57:56.916 INFO:teuthology.orchestra.run.smithi084.stdout: "is_stretch_pool": false, 2024-09-20T22:57:56.916 INFO:teuthology.orchestra.run.smithi084.stdout: "object_hash": 2, 2024-09-20T22:57:56.916 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_autoscale_mode": "off", 2024-09-20T22:57:56.916 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num": 32, 2024-09-20T22:57:56.916 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num": 32, 2024-09-20T22:57:56.916 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num_target": 32, 2024-09-20T22:57:56.916 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_target": 32, 2024-09-20T22:57:56.916 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_pending": 32, 2024-09-20T22:57:56.916 INFO:teuthology.orchestra.run.smithi084.stdout: "last_pg_merge_meta": { 2024-09-20T22:57:56.916 INFO:teuthology.orchestra.run.smithi084.stdout: "source_pgid": "0.0", 2024-09-20T22:57:56.916 INFO:teuthology.orchestra.run.smithi084.stdout: "ready_epoch": 0, 2024-09-20T22:57:56.916 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:56 smithi084 ceph-mon[29774]: 5.a scrub starts 2024-09-20T22:57:56.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:56 smithi084 ceph-mon[29774]: 5.a scrub ok 2024-09-20T22:57:56.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:56 smithi084 ceph-mon[29774]: 5.12 scrub starts 2024-09-20T22:57:56.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:56 smithi084 ceph-mon[29774]: 5.12 scrub ok 2024-09-20T22:57:56.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:56 smithi084 ceph-mon[29774]: 7.10 scrub starts 2024-09-20T22:57:56.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:56 smithi084 ceph-mon[29774]: 7.10 scrub ok 2024-09-20T22:57:56.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:56 smithi084 ceph-mon[29774]: 7.6 scrub starts 2024-09-20T22:57:56.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:56 smithi084 ceph-mon[29774]: 7.6 scrub ok 2024-09-20T22:57:56.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:56 smithi084 ceph-mon[29774]: pgmap v1392: 168 pgs: 2 active+clean+snaptrim, 166 active+clean; 793 MiB data, 2.3 GiB used, 623 GiB / 626 GiB avail; 14 MiB/s, 0 objects/s recovering 2024-09-20T22:57:56.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:56 smithi084 ceph-mon[29774]: 7.7 scrub starts 2024-09-20T22:57:56.917 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:56 smithi084 ceph-mon[29774]: 7.7 scrub ok 2024-09-20T22:57:56.918 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_started": 0, 2024-09-20T22:57:56.918 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_clean": 0, 2024-09-20T22:57:56.918 INFO:teuthology.orchestra.run.smithi084.stdout: "source_version": "0'0", 2024-09-20T22:57:56.918 INFO:teuthology.orchestra.run.smithi084.stdout: "target_version": "0'0" 2024-09-20T22:57:56.918 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:56.918 INFO:teuthology.orchestra.run.smithi084.stdout: "last_change": "68", 2024-09-20T22:57:56.918 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend": "0", 2024-09-20T22:57:56.918 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-20T22:57:56.918 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_preluminous": "0", 2024-09-20T22:57:56.918 INFO:teuthology.orchestra.run.smithi084.stdout: "auid": 0, 2024-09-20T22:57:56.918 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_mode": "selfmanaged", 2024-09-20T22:57:56.919 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_seq": 0, 2024-09-20T22:57:56.919 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_epoch": 0, 2024-09-20T22:57:56.919 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_snaps": [], 2024-09-20T22:57:56.919 INFO:teuthology.orchestra.run.smithi084.stdout: "removed_snaps": "[]", 2024-09-20T22:57:56.919 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_bytes": 0, 2024-09-20T22:57:56.919 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_objects": 0, 2024-09-20T22:57:56.919 INFO:teuthology.orchestra.run.smithi084.stdout: "tiers": [], 2024-09-20T22:57:56.919 INFO:teuthology.orchestra.run.smithi084.stdout: "tier_of": -1, 2024-09-20T22:57:56.919 INFO:teuthology.orchestra.run.smithi084.stdout: "read_tier": -1, 2024-09-20T22:57:56.919 INFO:teuthology.orchestra.run.smithi084.stdout: "write_tier": -1, 2024-09-20T22:57:56.919 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_mode": "none", 2024-09-20T22:57:56.919 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_bytes": 0, 2024-09-20T22:57:56.919 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_objects": 0, 2024-09-20T22:57:56.919 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-20T22:57:56.919 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-20T22:57:56.920 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-20T22:57:56.920 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_flush_age": 0, 2024-09-20T22:57:56.920 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_evict_age": 0, 2024-09-20T22:57:56.920 INFO:teuthology.orchestra.run.smithi084.stdout: "erasure_code_profile": "", 2024-09-20T22:57:56.920 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_params": { 2024-09-20T22:57:56.920 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "none" 2024-09-20T22:57:56.920 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:56.920 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_period": 0, 2024-09-20T22:57:56.920 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_count": 0, 2024-09-20T22:57:56.920 INFO:teuthology.orchestra.run.smithi084.stdout: "use_gmt_hitset": true, 2024-09-20T22:57:56.920 INFO:teuthology.orchestra.run.smithi084.stdout: "min_read_recency_for_promote": 0, 2024-09-20T22:57:56.920 INFO:teuthology.orchestra.run.smithi084.stdout: "min_write_recency_for_promote": 0, 2024-09-20T22:57:56.920 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_grade_decay_rate": 0, 2024-09-20T22:57:56.920 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_search_last_n": 0, 2024-09-20T22:57:56.920 INFO:teuthology.orchestra.run.smithi084.stdout: "grade_table": [], 2024-09-20T22:57:56.921 INFO:teuthology.orchestra.run.smithi084.stdout: "stripe_width": 0, 2024-09-20T22:57:56.921 INFO:teuthology.orchestra.run.smithi084.stdout: "expected_num_objects": 0, 2024-09-20T22:57:56.921 INFO:teuthology.orchestra.run.smithi084.stdout: "fast_read": false, 2024-09-20T22:57:56.921 INFO:teuthology.orchestra.run.smithi084.stdout: "options": {}, 2024-09-20T22:57:56.921 INFO:teuthology.orchestra.run.smithi084.stdout: "application_metadata": { 2024-09-20T22:57:56.921 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw": {} 2024-09-20T22:57:56.921 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:56.921 INFO:teuthology.orchestra.run.smithi084.stdout: "read_balance": { 2024-09-20T22:57:56.921 INFO:teuthology.orchestra.run.smithi084.stdout: "score_type": "Fair distribution", 2024-09-20T22:57:56.921 INFO:teuthology.orchestra.run.smithi084.stdout: "score_acting": 1.75, 2024-09-20T22:57:56.921 INFO:teuthology.orchestra.run.smithi084.stdout: "score_stable": 1.75, 2024-09-20T22:57:56.921 INFO:teuthology.orchestra.run.smithi084.stdout: "optimal_score": 1, 2024-09-20T22:57:56.921 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_acting": 1.75, 2024-09-20T22:57:56.921 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_stable": 1.75, 2024-09-20T22:57:56.922 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity_weighted": 0.875, 2024-09-20T22:57:56.922 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity": 1, 2024-09-20T22:57:56.922 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-09-20T22:57:56.922 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:57:56.922 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:56.922 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:56.922 INFO:teuthology.orchestra.run.smithi084.stdout: "pool": 4, 2024-09-20T22:57:56.922 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_name": "default.rgw.log", 2024-09-20T22:57:56.922 INFO:teuthology.orchestra.run.smithi084.stdout: "create_time": "2024-09-20T22:41:38.454640+0000", 2024-09-20T22:57:56.922 INFO:teuthology.orchestra.run.smithi084.stdout: "flags": 1, 2024-09-20T22:57:56.922 INFO:teuthology.orchestra.run.smithi084.stdout: "flags_names": "hashpspool", 2024-09-20T22:57:56.922 INFO:teuthology.orchestra.run.smithi084.stdout: "type": 1, 2024-09-20T22:57:56.922 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 3, 2024-09-20T22:57:56.922 INFO:teuthology.orchestra.run.smithi084.stdout: "min_size": 2, 2024-09-20T22:57:56.923 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_rule": 0, 2024-09-20T22:57:56.923 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_count": 0, 2024-09-20T22:57:56.923 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_target": 0, 2024-09-20T22:57:56.923 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_barrier": 0, 2024-09-20T22:57:56.923 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-20T22:57:56.923 INFO:teuthology.orchestra.run.smithi084.stdout: "is_stretch_pool": false, 2024-09-20T22:57:56.923 INFO:teuthology.orchestra.run.smithi084.stdout: "object_hash": 2, 2024-09-20T22:57:56.923 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_autoscale_mode": "off", 2024-09-20T22:57:56.923 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num": 32, 2024-09-20T22:57:56.923 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num": 32, 2024-09-20T22:57:56.923 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num_target": 32, 2024-09-20T22:57:56.923 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_target": 32, 2024-09-20T22:57:56.923 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_pending": 32, 2024-09-20T22:57:56.923 INFO:teuthology.orchestra.run.smithi084.stdout: "last_pg_merge_meta": { 2024-09-20T22:57:56.923 INFO:teuthology.orchestra.run.smithi084.stdout: "source_pgid": "0.0", 2024-09-20T22:57:56.924 INFO:teuthology.orchestra.run.smithi084.stdout: "ready_epoch": 0, 2024-09-20T22:57:56.924 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_started": 0, 2024-09-20T22:57:56.924 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_clean": 0, 2024-09-20T22:57:56.924 INFO:teuthology.orchestra.run.smithi084.stdout: "source_version": "0'0", 2024-09-20T22:57:56.924 INFO:teuthology.orchestra.run.smithi084.stdout: "target_version": "0'0" 2024-09-20T22:57:56.924 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:56.924 INFO:teuthology.orchestra.run.smithi084.stdout: "last_change": "70", 2024-09-20T22:57:56.924 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend": "0", 2024-09-20T22:57:56.924 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-20T22:57:56.924 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_preluminous": "0", 2024-09-20T22:57:56.924 INFO:teuthology.orchestra.run.smithi084.stdout: "auid": 0, 2024-09-20T22:57:56.924 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_mode": "selfmanaged", 2024-09-20T22:57:56.924 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_seq": 0, 2024-09-20T22:57:56.924 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_epoch": 0, 2024-09-20T22:57:56.925 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_snaps": [], 2024-09-20T22:57:56.925 INFO:teuthology.orchestra.run.smithi084.stdout: "removed_snaps": "[]", 2024-09-20T22:57:56.925 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_bytes": 0, 2024-09-20T22:57:56.925 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_objects": 0, 2024-09-20T22:57:56.925 INFO:teuthology.orchestra.run.smithi084.stdout: "tiers": [], 2024-09-20T22:57:56.925 INFO:teuthology.orchestra.run.smithi084.stdout: "tier_of": -1, 2024-09-20T22:57:56.925 INFO:teuthology.orchestra.run.smithi084.stdout: "read_tier": -1, 2024-09-20T22:57:56.925 INFO:teuthology.orchestra.run.smithi084.stdout: "write_tier": -1, 2024-09-20T22:57:56.925 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_mode": "none", 2024-09-20T22:57:56.925 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_bytes": 0, 2024-09-20T22:57:56.925 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_objects": 0, 2024-09-20T22:57:56.925 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-20T22:57:56.925 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-20T22:57:56.925 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-20T22:57:56.925 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_flush_age": 0, 2024-09-20T22:57:56.926 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_evict_age": 0, 2024-09-20T22:57:56.926 INFO:teuthology.orchestra.run.smithi084.stdout: "erasure_code_profile": "", 2024-09-20T22:57:56.926 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_params": { 2024-09-20T22:57:56.926 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "none" 2024-09-20T22:57:56.926 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:56.926 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_period": 0, 2024-09-20T22:57:56.926 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_count": 0, 2024-09-20T22:57:56.926 INFO:teuthology.orchestra.run.smithi084.stdout: "use_gmt_hitset": true, 2024-09-20T22:57:56.926 INFO:teuthology.orchestra.run.smithi084.stdout: "min_read_recency_for_promote": 0, 2024-09-20T22:57:56.926 INFO:teuthology.orchestra.run.smithi084.stdout: "min_write_recency_for_promote": 0, 2024-09-20T22:57:56.926 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_grade_decay_rate": 0, 2024-09-20T22:57:56.926 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_search_last_n": 0, 2024-09-20T22:57:56.926 INFO:teuthology.orchestra.run.smithi084.stdout: "grade_table": [], 2024-09-20T22:57:56.926 INFO:teuthology.orchestra.run.smithi084.stdout: "stripe_width": 0, 2024-09-20T22:57:56.926 INFO:teuthology.orchestra.run.smithi084.stdout: "expected_num_objects": 0, 2024-09-20T22:57:56.927 INFO:teuthology.orchestra.run.smithi084.stdout: "fast_read": false, 2024-09-20T22:57:56.927 INFO:teuthology.orchestra.run.smithi084.stdout: "options": {}, 2024-09-20T22:57:56.927 INFO:teuthology.orchestra.run.smithi084.stdout: "application_metadata": { 2024-09-20T22:57:56.927 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw": {} 2024-09-20T22:57:56.927 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:56.927 INFO:teuthology.orchestra.run.smithi084.stdout: "read_balance": { 2024-09-20T22:57:56.927 INFO:teuthology.orchestra.run.smithi084.stdout: "score_type": "Fair distribution", 2024-09-20T22:57:56.927 INFO:teuthology.orchestra.run.smithi084.stdout: "score_acting": 2.190000057220459, 2024-09-20T22:57:56.927 INFO:teuthology.orchestra.run.smithi084.stdout: "score_stable": 2.190000057220459, 2024-09-20T22:57:56.927 INFO:teuthology.orchestra.run.smithi084.stdout: "optimal_score": 1, 2024-09-20T22:57:56.927 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_acting": 2.190000057220459, 2024-09-20T22:57:56.927 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_stable": 2.190000057220459, 2024-09-20T22:57:56.927 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity_weighted": 0.875, 2024-09-20T22:57:56.928 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity": 1, 2024-09-20T22:57:56.928 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-09-20T22:57:56.928 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:57:56.928 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:56.928 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:56.928 INFO:teuthology.orchestra.run.smithi084.stdout: "pool": 5, 2024-09-20T22:57:56.928 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_name": "default.rgw.control", 2024-09-20T22:57:56.928 INFO:teuthology.orchestra.run.smithi084.stdout: "create_time": "2024-09-20T22:41:40.407129+0000", 2024-09-20T22:57:56.928 INFO:teuthology.orchestra.run.smithi084.stdout: "flags": 1, 2024-09-20T22:57:56.928 INFO:teuthology.orchestra.run.smithi084.stdout: "flags_names": "hashpspool", 2024-09-20T22:57:56.928 INFO:teuthology.orchestra.run.smithi084.stdout: "type": 1, 2024-09-20T22:57:56.928 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 3, 2024-09-20T22:57:56.928 INFO:teuthology.orchestra.run.smithi084.stdout: "min_size": 2, 2024-09-20T22:57:56.928 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_rule": 0, 2024-09-20T22:57:56.928 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_count": 0, 2024-09-20T22:57:56.928 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_target": 0, 2024-09-20T22:57:56.929 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_barrier": 0, 2024-09-20T22:57:56.929 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-20T22:57:56.929 INFO:teuthology.orchestra.run.smithi084.stdout: "is_stretch_pool": false, 2024-09-20T22:57:56.929 INFO:teuthology.orchestra.run.smithi084.stdout: "object_hash": 2, 2024-09-20T22:57:56.929 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_autoscale_mode": "off", 2024-09-20T22:57:56.929 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num": 32, 2024-09-20T22:57:56.929 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num": 32, 2024-09-20T22:57:56.929 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num_target": 32, 2024-09-20T22:57:56.929 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_target": 32, 2024-09-20T22:57:56.929 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_pending": 32, 2024-09-20T22:57:56.929 INFO:teuthology.orchestra.run.smithi084.stdout: "last_pg_merge_meta": { 2024-09-20T22:57:56.929 INFO:teuthology.orchestra.run.smithi084.stdout: "source_pgid": "0.0", 2024-09-20T22:57:56.929 INFO:teuthology.orchestra.run.smithi084.stdout: "ready_epoch": 0, 2024-09-20T22:57:56.929 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_started": 0, 2024-09-20T22:57:56.930 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_clean": 0, 2024-09-20T22:57:56.930 INFO:teuthology.orchestra.run.smithi084.stdout: "source_version": "0'0", 2024-09-20T22:57:56.930 INFO:teuthology.orchestra.run.smithi084.stdout: "target_version": "0'0" 2024-09-20T22:57:56.930 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:56.930 INFO:teuthology.orchestra.run.smithi084.stdout: "last_change": "72", 2024-09-20T22:57:56.930 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend": "0", 2024-09-20T22:57:56.930 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-20T22:57:56.930 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_preluminous": "0", 2024-09-20T22:57:56.930 INFO:teuthology.orchestra.run.smithi084.stdout: "auid": 0, 2024-09-20T22:57:56.930 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_mode": "selfmanaged", 2024-09-20T22:57:56.930 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_seq": 0, 2024-09-20T22:57:56.930 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_epoch": 0, 2024-09-20T22:57:56.930 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_snaps": [], 2024-09-20T22:57:56.930 INFO:teuthology.orchestra.run.smithi084.stdout: "removed_snaps": "[]", 2024-09-20T22:57:56.930 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_bytes": 0, 2024-09-20T22:57:56.931 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_objects": 0, 2024-09-20T22:57:56.931 INFO:teuthology.orchestra.run.smithi084.stdout: "tiers": [], 2024-09-20T22:57:56.931 INFO:teuthology.orchestra.run.smithi084.stdout: "tier_of": -1, 2024-09-20T22:57:56.931 INFO:teuthology.orchestra.run.smithi084.stdout: "read_tier": -1, 2024-09-20T22:57:56.931 INFO:teuthology.orchestra.run.smithi084.stdout: "write_tier": -1, 2024-09-20T22:57:56.931 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_mode": "none", 2024-09-20T22:57:56.931 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_bytes": 0, 2024-09-20T22:57:56.931 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_objects": 0, 2024-09-20T22:57:56.931 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-20T22:57:56.931 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-20T22:57:56.931 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-20T22:57:56.931 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_flush_age": 0, 2024-09-20T22:57:56.931 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_evict_age": 0, 2024-09-20T22:57:56.931 INFO:teuthology.orchestra.run.smithi084.stdout: "erasure_code_profile": "", 2024-09-20T22:57:56.931 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_params": { 2024-09-20T22:57:56.932 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "none" 2024-09-20T22:57:56.932 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:56.932 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_period": 0, 2024-09-20T22:57:56.932 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_count": 0, 2024-09-20T22:57:56.932 INFO:teuthology.orchestra.run.smithi084.stdout: "use_gmt_hitset": true, 2024-09-20T22:57:56.932 INFO:teuthology.orchestra.run.smithi084.stdout: "min_read_recency_for_promote": 0, 2024-09-20T22:57:56.932 INFO:teuthology.orchestra.run.smithi084.stdout: "min_write_recency_for_promote": 0, 2024-09-20T22:57:56.932 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_grade_decay_rate": 0, 2024-09-20T22:57:56.932 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_search_last_n": 0, 2024-09-20T22:57:56.932 INFO:teuthology.orchestra.run.smithi084.stdout: "grade_table": [], 2024-09-20T22:57:56.932 INFO:teuthology.orchestra.run.smithi084.stdout: "stripe_width": 0, 2024-09-20T22:57:56.932 INFO:teuthology.orchestra.run.smithi084.stdout: "expected_num_objects": 0, 2024-09-20T22:57:56.932 INFO:teuthology.orchestra.run.smithi084.stdout: "fast_read": false, 2024-09-20T22:57:56.932 INFO:teuthology.orchestra.run.smithi084.stdout: "options": {}, 2024-09-20T22:57:56.932 INFO:teuthology.orchestra.run.smithi084.stdout: "application_metadata": { 2024-09-20T22:57:56.933 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw": {} 2024-09-20T22:57:56.933 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:56.933 INFO:teuthology.orchestra.run.smithi084.stdout: "read_balance": { 2024-09-20T22:57:56.933 INFO:teuthology.orchestra.run.smithi084.stdout: "score_type": "Fair distribution", 2024-09-20T22:57:56.933 INFO:teuthology.orchestra.run.smithi084.stdout: "score_acting": 1.309999942779541, 2024-09-20T22:57:56.933 INFO:teuthology.orchestra.run.smithi084.stdout: "score_stable": 1.309999942779541, 2024-09-20T22:57:56.933 INFO:teuthology.orchestra.run.smithi084.stdout: "optimal_score": 1, 2024-09-20T22:57:56.933 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_acting": 1.309999942779541, 2024-09-20T22:57:56.933 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_stable": 1.309999942779541, 2024-09-20T22:57:56.933 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity_weighted": 0.875, 2024-09-20T22:57:56.933 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity": 1, 2024-09-20T22:57:56.933 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-09-20T22:57:56.933 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:57:56.933 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:56.933 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:56.934 INFO:teuthology.orchestra.run.smithi084.stdout: "pool": 6, 2024-09-20T22:57:56.934 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_name": "default.rgw.meta", 2024-09-20T22:57:56.934 INFO:teuthology.orchestra.run.smithi084.stdout: "create_time": "2024-09-20T22:41:42.476716+0000", 2024-09-20T22:57:56.934 INFO:teuthology.orchestra.run.smithi084.stdout: "flags": 1, 2024-09-20T22:57:56.934 INFO:teuthology.orchestra.run.smithi084.stdout: "flags_names": "hashpspool", 2024-09-20T22:57:56.934 INFO:teuthology.orchestra.run.smithi084.stdout: "type": 1, 2024-09-20T22:57:56.934 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 3, 2024-09-20T22:57:56.934 INFO:teuthology.orchestra.run.smithi084.stdout: "min_size": 2, 2024-09-20T22:57:56.934 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_rule": 0, 2024-09-20T22:57:56.934 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_count": 0, 2024-09-20T22:57:56.934 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_target": 0, 2024-09-20T22:57:56.934 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_barrier": 0, 2024-09-20T22:57:56.934 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-20T22:57:56.934 INFO:teuthology.orchestra.run.smithi084.stdout: "is_stretch_pool": false, 2024-09-20T22:57:56.935 INFO:teuthology.orchestra.run.smithi084.stdout: "object_hash": 2, 2024-09-20T22:57:56.935 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_autoscale_mode": "off", 2024-09-20T22:57:56.935 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num": 32, 2024-09-20T22:57:56.935 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num": 32, 2024-09-20T22:57:56.935 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num_target": 32, 2024-09-20T22:57:56.935 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_target": 32, 2024-09-20T22:57:56.935 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_pending": 32, 2024-09-20T22:57:56.935 INFO:teuthology.orchestra.run.smithi084.stdout: "last_pg_merge_meta": { 2024-09-20T22:57:56.935 INFO:teuthology.orchestra.run.smithi084.stdout: "source_pgid": "0.0", 2024-09-20T22:57:56.935 INFO:teuthology.orchestra.run.smithi084.stdout: "ready_epoch": 0, 2024-09-20T22:57:56.935 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_started": 0, 2024-09-20T22:57:56.935 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_clean": 0, 2024-09-20T22:57:56.935 INFO:teuthology.orchestra.run.smithi084.stdout: "source_version": "0'0", 2024-09-20T22:57:56.935 INFO:teuthology.orchestra.run.smithi084.stdout: "target_version": "0'0" 2024-09-20T22:57:56.935 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:56.936 INFO:teuthology.orchestra.run.smithi084.stdout: "last_change": "74", 2024-09-20T22:57:56.936 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend": "0", 2024-09-20T22:57:56.936 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-20T22:57:56.936 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_preluminous": "0", 2024-09-20T22:57:56.936 INFO:teuthology.orchestra.run.smithi084.stdout: "auid": 0, 2024-09-20T22:57:56.936 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_mode": "selfmanaged", 2024-09-20T22:57:56.936 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_seq": 0, 2024-09-20T22:57:56.936 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_epoch": 0, 2024-09-20T22:57:56.936 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_snaps": [], 2024-09-20T22:57:56.936 INFO:teuthology.orchestra.run.smithi084.stdout: "removed_snaps": "[]", 2024-09-20T22:57:56.936 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_bytes": 0, 2024-09-20T22:57:56.936 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_objects": 0, 2024-09-20T22:57:56.936 INFO:teuthology.orchestra.run.smithi084.stdout: "tiers": [], 2024-09-20T22:57:56.936 INFO:teuthology.orchestra.run.smithi084.stdout: "tier_of": -1, 2024-09-20T22:57:56.937 INFO:teuthology.orchestra.run.smithi084.stdout: "read_tier": -1, 2024-09-20T22:57:56.937 INFO:teuthology.orchestra.run.smithi084.stdout: "write_tier": -1, 2024-09-20T22:57:56.937 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_mode": "none", 2024-09-20T22:57:56.937 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_bytes": 0, 2024-09-20T22:57:56.937 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_objects": 0, 2024-09-20T22:57:56.937 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-20T22:57:56.937 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-20T22:57:56.937 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-20T22:57:56.937 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_flush_age": 0, 2024-09-20T22:57:56.937 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_evict_age": 0, 2024-09-20T22:57:56.937 INFO:teuthology.orchestra.run.smithi084.stdout: "erasure_code_profile": "", 2024-09-20T22:57:56.937 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_params": { 2024-09-20T22:57:56.937 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "none" 2024-09-20T22:57:56.937 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:56.938 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_period": 0, 2024-09-20T22:57:56.938 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_count": 0, 2024-09-20T22:57:56.938 INFO:teuthology.orchestra.run.smithi084.stdout: "use_gmt_hitset": true, 2024-09-20T22:57:56.938 INFO:teuthology.orchestra.run.smithi084.stdout: "min_read_recency_for_promote": 0, 2024-09-20T22:57:56.938 INFO:teuthology.orchestra.run.smithi084.stdout: "min_write_recency_for_promote": 0, 2024-09-20T22:57:56.938 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_grade_decay_rate": 0, 2024-09-20T22:57:56.938 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_search_last_n": 0, 2024-09-20T22:57:56.938 INFO:teuthology.orchestra.run.smithi084.stdout: "grade_table": [], 2024-09-20T22:57:56.938 INFO:teuthology.orchestra.run.smithi084.stdout: "stripe_width": 0, 2024-09-20T22:57:56.938 INFO:teuthology.orchestra.run.smithi084.stdout: "expected_num_objects": 0, 2024-09-20T22:57:56.938 INFO:teuthology.orchestra.run.smithi084.stdout: "fast_read": false, 2024-09-20T22:57:56.938 INFO:teuthology.orchestra.run.smithi084.stdout: "options": { 2024-09-20T22:57:56.938 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_autoscale_bias": 4 2024-09-20T22:57:56.938 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:56.939 INFO:teuthology.orchestra.run.smithi084.stdout: "application_metadata": { 2024-09-20T22:57:56.939 INFO:teuthology.orchestra.run.smithi084.stdout: "rgw": {} 2024-09-20T22:57:56.939 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:56.939 INFO:teuthology.orchestra.run.smithi084.stdout: "read_balance": { 2024-09-20T22:57:56.939 INFO:teuthology.orchestra.run.smithi084.stdout: "score_type": "Fair distribution", 2024-09-20T22:57:56.939 INFO:teuthology.orchestra.run.smithi084.stdout: "score_acting": 1.75, 2024-09-20T22:57:56.939 INFO:teuthology.orchestra.run.smithi084.stdout: "score_stable": 1.75, 2024-09-20T22:57:56.939 INFO:teuthology.orchestra.run.smithi084.stdout: "optimal_score": 1, 2024-09-20T22:57:56.939 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_acting": 1.75, 2024-09-20T22:57:56.939 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_stable": 1.75, 2024-09-20T22:57:56.939 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity_weighted": 0.875, 2024-09-20T22:57:56.939 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity": 1, 2024-09-20T22:57:56.939 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-09-20T22:57:56.939 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:57:56.939 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:56.940 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:56.940 INFO:teuthology.orchestra.run.smithi084.stdout: "pool": 7, 2024-09-20T22:57:56.940 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_name": "unique_pool_0", 2024-09-20T22:57:56.940 INFO:teuthology.orchestra.run.smithi084.stdout: "create_time": "2024-09-20T22:42:51.354358+0000", 2024-09-20T22:57:56.940 INFO:teuthology.orchestra.run.smithi084.stdout: "flags": 8193, 2024-09-20T22:57:56.940 INFO:teuthology.orchestra.run.smithi084.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-20T22:57:56.940 INFO:teuthology.orchestra.run.smithi084.stdout: "type": 1, 2024-09-20T22:57:56.940 INFO:teuthology.orchestra.run.smithi084.stdout: "size": 3, 2024-09-20T22:57:56.940 INFO:teuthology.orchestra.run.smithi084.stdout: "min_size": 2, 2024-09-20T22:57:56.940 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_rule": 0, 2024-09-20T22:57:56.940 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_count": 0, 2024-09-20T22:57:56.940 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_target": 0, 2024-09-20T22:57:56.940 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_barrier": 0, 2024-09-20T22:57:56.940 INFO:teuthology.orchestra.run.smithi084.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-20T22:57:56.940 INFO:teuthology.orchestra.run.smithi084.stdout: "is_stretch_pool": false, 2024-09-20T22:57:56.941 INFO:teuthology.orchestra.run.smithi084.stdout: "object_hash": 2, 2024-09-20T22:57:56.941 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_autoscale_mode": "off", 2024-09-20T22:57:56.941 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num": 26, 2024-09-20T22:57:56.941 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num": 26, 2024-09-20T22:57:56.941 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_placement_num_target": 26, 2024-09-20T22:57:56.941 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_target": 26, 2024-09-20T22:57:56.941 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_num_pending": 26, 2024-09-20T22:57:56.941 INFO:teuthology.orchestra.run.smithi084.stdout: "last_pg_merge_meta": { 2024-09-20T22:57:56.942 INFO:teuthology.orchestra.run.smithi084.stdout: "source_pgid": "7.10", 2024-09-20T22:57:56.942 INFO:teuthology.orchestra.run.smithi084.stdout: "ready_epoch": 484, 2024-09-20T22:57:56.942 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_started": 483, 2024-09-20T22:57:56.942 INFO:teuthology.orchestra.run.smithi084.stdout: "last_epoch_clean": 483, 2024-09-20T22:57:56.942 INFO:teuthology.orchestra.run.smithi084.stdout: "source_version": "480'172", 2024-09-20T22:57:56.942 INFO:teuthology.orchestra.run.smithi084.stdout: "target_version": "480'253" 2024-09-20T22:57:56.942 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:56.942 INFO:teuthology.orchestra.run.smithi084.stdout: "last_change": "990", 2024-09-20T22:57:56.942 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend": "0", 2024-09-20T22:57:56.942 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_prenautilus": "484", 2024-09-20T22:57:56.942 INFO:teuthology.orchestra.run.smithi084.stdout: "last_force_op_resend_preluminous": "932", 2024-09-20T22:57:56.942 INFO:teuthology.orchestra.run.smithi084.stdout: "auid": 0, 2024-09-20T22:57:56.942 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_mode": "selfmanaged", 2024-09-20T22:57:56.943 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_seq": 466, 2024-09-20T22:57:56.943 INFO:teuthology.orchestra.run.smithi084.stdout: "snap_epoch": 990, 2024-09-20T22:57:56.943 INFO:teuthology.orchestra.run.smithi084.stdout: "pool_snaps": [], 2024-09-20T22:57:56.943 INFO:teuthology.orchestra.run.smithi084.stdout: "removed_snaps": "[]", 2024-09-20T22:57:56.943 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_bytes": 0, 2024-09-20T22:57:56.943 INFO:teuthology.orchestra.run.smithi084.stdout: "quota_max_objects": 0, 2024-09-20T22:57:56.943 INFO:teuthology.orchestra.run.smithi084.stdout: "tiers": [], 2024-09-20T22:57:56.943 INFO:teuthology.orchestra.run.smithi084.stdout: "tier_of": -1, 2024-09-20T22:57:56.943 INFO:teuthology.orchestra.run.smithi084.stdout: "read_tier": -1, 2024-09-20T22:57:56.943 INFO:teuthology.orchestra.run.smithi084.stdout: "write_tier": -1, 2024-09-20T22:57:56.943 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_mode": "none", 2024-09-20T22:57:56.943 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_bytes": 0, 2024-09-20T22:57:56.943 INFO:teuthology.orchestra.run.smithi084.stdout: "target_max_objects": 0, 2024-09-20T22:57:56.943 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-20T22:57:56.944 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-20T22:57:56.944 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-20T22:57:56.944 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_flush_age": 0, 2024-09-20T22:57:56.944 INFO:teuthology.orchestra.run.smithi084.stdout: "cache_min_evict_age": 0, 2024-09-20T22:57:56.944 INFO:teuthology.orchestra.run.smithi084.stdout: "erasure_code_profile": "", 2024-09-20T22:57:56.944 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_params": { 2024-09-20T22:57:56.944 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "none" 2024-09-20T22:57:56.944 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:56.944 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_period": 0, 2024-09-20T22:57:56.944 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_count": 0, 2024-09-20T22:57:56.944 INFO:teuthology.orchestra.run.smithi084.stdout: "use_gmt_hitset": true, 2024-09-20T22:57:56.944 INFO:teuthology.orchestra.run.smithi084.stdout: "min_read_recency_for_promote": 0, 2024-09-20T22:57:56.944 INFO:teuthology.orchestra.run.smithi084.stdout: "min_write_recency_for_promote": 0, 2024-09-20T22:57:56.944 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_grade_decay_rate": 0, 2024-09-20T22:57:56.944 INFO:teuthology.orchestra.run.smithi084.stdout: "hit_set_search_last_n": 0, 2024-09-20T22:57:56.945 INFO:teuthology.orchestra.run.smithi084.stdout: "grade_table": [], 2024-09-20T22:57:56.945 INFO:teuthology.orchestra.run.smithi084.stdout: "stripe_width": 0, 2024-09-20T22:57:56.945 INFO:teuthology.orchestra.run.smithi084.stdout: "expected_num_objects": 0, 2024-09-20T22:57:56.945 INFO:teuthology.orchestra.run.smithi084.stdout: "fast_read": false, 2024-09-20T22:57:56.945 INFO:teuthology.orchestra.run.smithi084.stdout: "options": {}, 2024-09-20T22:57:56.945 INFO:teuthology.orchestra.run.smithi084.stdout: "application_metadata": { 2024-09-20T22:57:56.945 INFO:teuthology.orchestra.run.smithi084.stdout: "rados": {} 2024-09-20T22:57:56.945 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:56.945 INFO:teuthology.orchestra.run.smithi084.stdout: "read_balance": { 2024-09-20T22:57:56.945 INFO:teuthology.orchestra.run.smithi084.stdout: "score_type": "Fair distribution", 2024-09-20T22:57:56.945 INFO:teuthology.orchestra.run.smithi084.stdout: "score_acting": 1.8799999952316284, 2024-09-20T22:57:56.945 INFO:teuthology.orchestra.run.smithi084.stdout: "score_stable": 1.8799999952316284, 2024-09-20T22:57:56.945 INFO:teuthology.orchestra.run.smithi084.stdout: "optimal_score": 1, 2024-09-20T22:57:56.945 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_acting": 1.8799999952316284, 2024-09-20T22:57:56.946 INFO:teuthology.orchestra.run.smithi084.stdout: "raw_score_stable": 1.8799999952316284, 2024-09-20T22:57:56.946 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity_weighted": 0.875, 2024-09-20T22:57:56.946 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity": 1, 2024-09-20T22:57:56.946 INFO:teuthology.orchestra.run.smithi084.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-09-20T22:57:56.946 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:57:56.946 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:57:56.946 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2024-09-20T22:57:56.946 INFO:teuthology.orchestra.run.smithi084.stdout: "osds": [ 2024-09-20T22:57:56.946 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:56.946 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 0, 2024-09-20T22:57:56.946 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "ea36a489-6f91-4b2f-b784-b90506b9805d", 2024-09-20T22:57:56.946 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-09-20T22:57:56.946 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-09-20T22:57:56.946 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-09-20T22:57:56.947 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-09-20T22:57:56.947 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-09-20T22:57:56.947 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-09-20T22:57:56.947 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 8, 2024-09-20T22:57:56.947 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 990, 2024-09-20T22:57:56.947 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-09-20T22:57:56.947 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-09-20T22:57:56.947 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-09-20T22:57:56.947 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:57:56.947 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:56.947 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:57:56.947 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6801", 2024-09-20T22:57:56.947 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 268084626 2024-09-20T22:57:56.947 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:57:56.947 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:57:56.948 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:56.948 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-09-20T22:57:56.948 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:57:56.948 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:56.948 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:57:56.948 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6802", 2024-09-20T22:57:56.948 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 268084626 2024-09-20T22:57:56.948 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:57:56.948 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:57:56.948 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:56.948 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-09-20T22:57:56.948 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:57:56.948 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:56.948 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:57:56.949 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6804", 2024-09-20T22:57:56.949 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 268084626 2024-09-20T22:57:56.949 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:57:56.949 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:57:56.949 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:56.949 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-09-20T22:57:56.949 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:57:56.949 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:56.949 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:57:56.949 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6803", 2024-09-20T22:57:56.949 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 268084626 2024-09-20T22:57:56.949 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:57:56.949 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:57:56.949 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:56.949 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.84:6801/268084626", 2024-09-20T22:57:56.950 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.84:6802/268084626", 2024-09-20T22:57:56.950 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.84:6804/268084626", 2024-09-20T22:57:56.950 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.84:6803/268084626", 2024-09-20T22:57:56.950 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-09-20T22:57:56.950 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-09-20T22:57:56.950 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-09-20T22:57:56.950 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:57:56.950 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:56.950 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:56.950 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 1, 2024-09-20T22:57:56.950 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "c8649d2f-27ef-47f8-bc94-f1cd9e62bdb8", 2024-09-20T22:57:56.950 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-09-20T22:57:56.950 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-09-20T22:57:56.950 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-09-20T22:57:56.951 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-09-20T22:57:56.951 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-09-20T22:57:56.951 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-09-20T22:57:56.951 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 13, 2024-09-20T22:57:56.951 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 988, 2024-09-20T22:57:56.951 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-09-20T22:57:56.951 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-09-20T22:57:56.951 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-09-20T22:57:56.951 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:57:56.951 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:56.951 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:57:56.951 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6805", 2024-09-20T22:57:56.951 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 418374136 2024-09-20T22:57:56.951 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:57:56.952 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:57:56.952 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:56.952 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-09-20T22:57:56.952 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:57:56.952 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:56.952 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:57:56.952 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6806", 2024-09-20T22:57:56.952 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 418374136 2024-09-20T22:57:56.952 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:57:56.952 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:57:56.952 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:56.952 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-09-20T22:57:56.952 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:57:56.952 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:56.952 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:57:56.953 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6808", 2024-09-20T22:57:56.953 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 418374136 2024-09-20T22:57:56.953 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:57:56.953 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:57:56.953 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:56.953 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-09-20T22:57:56.953 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:57:56.953 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:56.953 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:57:56.953 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6807", 2024-09-20T22:57:56.953 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 418374136 2024-09-20T22:57:56.953 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:57:56.953 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:57:56.954 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:56.954 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.84:6805/418374136", 2024-09-20T22:57:56.954 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.84:6806/418374136", 2024-09-20T22:57:56.954 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.84:6808/418374136", 2024-09-20T22:57:56.954 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.84:6807/418374136", 2024-09-20T22:57:56.954 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-09-20T22:57:56.954 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-09-20T22:57:56.954 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-09-20T22:57:56.954 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:57:56.954 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:56.954 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:56.954 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 2, 2024-09-20T22:57:56.954 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "c728eb48-a0f0-4de4-a651-7f3bec867280", 2024-09-20T22:57:56.954 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-09-20T22:57:56.954 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-09-20T22:57:56.955 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-09-20T22:57:56.955 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-09-20T22:57:56.955 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-09-20T22:57:56.955 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-09-20T22:57:56.955 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 18, 2024-09-20T22:57:56.955 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 988, 2024-09-20T22:57:56.955 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-09-20T22:57:56.955 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-09-20T22:57:56.955 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-09-20T22:57:56.955 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:57:56.955 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:56.955 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:57:56.955 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6809", 2024-09-20T22:57:56.955 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3977168611 2024-09-20T22:57:56.955 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:57:56.956 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:57:56.956 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:56.956 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-09-20T22:57:56.956 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:57:56.956 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:56.956 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:57:56.956 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6810", 2024-09-20T22:57:56.956 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3977168611 2024-09-20T22:57:56.956 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:57:56.956 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:57:56.956 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:56.956 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-09-20T22:57:56.956 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:57:56.956 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:56.957 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:57:56.957 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6812", 2024-09-20T22:57:56.957 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3977168611 2024-09-20T22:57:56.957 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:57:56.957 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:57:56.957 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:56.957 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-09-20T22:57:56.957 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:57:56.957 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:56.957 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:57:56.957 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6811", 2024-09-20T22:57:56.957 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3977168611 2024-09-20T22:57:56.957 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:57:56.957 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:57:56.957 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:56.958 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.84:6809/3977168611", 2024-09-20T22:57:56.958 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.84:6810/3977168611", 2024-09-20T22:57:56.958 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.84:6812/3977168611", 2024-09-20T22:57:56.958 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.84:6811/3977168611", 2024-09-20T22:57:56.958 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-09-20T22:57:56.958 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-09-20T22:57:56.958 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-09-20T22:57:56.958 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:57:56.958 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:56.958 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:56.958 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 3, 2024-09-20T22:57:56.958 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "573f44eb-0853-4630-a7d8-9caaeb256f9b", 2024-09-20T22:57:56.958 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-09-20T22:57:56.958 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-09-20T22:57:56.959 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-09-20T22:57:56.959 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-09-20T22:57:56.959 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 25, 2024-09-20T22:57:56.959 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 939, 2024-09-20T22:57:56.959 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 988, 2024-09-20T22:57:56.959 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 990, 2024-09-20T22:57:56.959 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 940, 2024-09-20T22:57:56.959 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-09-20T22:57:56.959 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-09-20T22:57:56.959 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:57:56.959 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:56.959 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:57:56.959 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6813", 2024-09-20T22:57:56.960 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3784535689 2024-09-20T22:57:56.960 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:57:56.960 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:57:56.960 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:56.960 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-09-20T22:57:56.960 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:57:56.960 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:56.960 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:57:56.960 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6814", 2024-09-20T22:57:56.960 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3784535689 2024-09-20T22:57:56.960 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:57:56.960 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:57:56.960 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:56.960 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-09-20T22:57:56.961 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:57:56.961 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:56.961 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:57:56.961 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6816", 2024-09-20T22:57:56.961 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3784535689 2024-09-20T22:57:56.961 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:57:56.961 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:57:56.961 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:56.961 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-09-20T22:57:56.961 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:57:56.961 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:56.961 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:57:56.961 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.84:6815", 2024-09-20T22:57:56.961 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3784535689 2024-09-20T22:57:56.962 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:57:56.962 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:57:56.962 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:56.962 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.84:6813/3784535689", 2024-09-20T22:57:56.962 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.84:6814/3784535689", 2024-09-20T22:57:56.962 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.84:6816/3784535689", 2024-09-20T22:57:56.962 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.84:6815/3784535689", 2024-09-20T22:57:56.962 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-09-20T22:57:56.962 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-09-20T22:57:56.962 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-09-20T22:57:56.962 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:57:56.962 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:56.962 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:56.962 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 4, 2024-09-20T22:57:56.962 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "a52c3b94-c5f3-490f-8ac8-09061b9b2abb", 2024-09-20T22:57:56.963 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-09-20T22:57:56.963 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-09-20T22:57:56.963 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-09-20T22:57:56.963 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 0, 2024-09-20T22:57:56.963 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-09-20T22:57:56.963 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-09-20T22:57:56.963 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 31, 2024-09-20T22:57:56.963 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 830, 2024-09-20T22:57:56.963 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-09-20T22:57:56.963 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-09-20T22:57:56.963 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-09-20T22:57:56.963 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:57:56.963 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:56.963 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:57:56.964 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.154:6800", 2024-09-20T22:57:56.964 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1793136002 2024-09-20T22:57:56.964 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:57:56.964 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:57:56.964 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:56.964 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-09-20T22:57:56.964 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:57:56.964 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:56.964 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:57:56.964 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.154:6801", 2024-09-20T22:57:56.964 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1793136002 2024-09-20T22:57:56.964 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:57:56.964 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:57:56.964 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:56.964 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-09-20T22:57:56.965 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:57:56.965 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:56.965 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:57:56.965 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.154:6803", 2024-09-20T22:57:56.965 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1793136002 2024-09-20T22:57:56.965 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:57:56.965 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:57:56.965 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:56.965 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-09-20T22:57:56.965 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:57:56.965 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:56.965 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:57:56.965 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.154:6802", 2024-09-20T22:57:56.965 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 1793136002 2024-09-20T22:57:56.965 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:57:56.966 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:57:56.966 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:56.966 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.154:6800/1793136002", 2024-09-20T22:57:56.966 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.154:6801/1793136002", 2024-09-20T22:57:56.966 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.154:6803/1793136002", 2024-09-20T22:57:56.966 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.154:6802/1793136002", 2024-09-20T22:57:56.966 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-09-20T22:57:56.966 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-09-20T22:57:56.966 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-09-20T22:57:56.966 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:57:56.966 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:56.966 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:56.966 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 5, 2024-09-20T22:57:56.966 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "4fa0dc8e-25cc-46cd-96ff-33f20b6faec3", 2024-09-20T22:57:56.967 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-09-20T22:57:56.967 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-09-20T22:57:56.967 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-09-20T22:57:56.967 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-09-20T22:57:56.967 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-09-20T22:57:56.967 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-09-20T22:57:56.967 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 40, 2024-09-20T22:57:56.967 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 991, 2024-09-20T22:57:56.967 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-09-20T22:57:56.967 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-09-20T22:57:56.967 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-09-20T22:57:56.967 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:57:56.967 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:56.967 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:57:56.968 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.154:6804", 2024-09-20T22:57:56.968 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 493899927 2024-09-20T22:57:56.968 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:57:56.968 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:57:56.968 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:56.968 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-09-20T22:57:56.968 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:57:56.968 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:56.968 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:57:56.969 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:57:56.969 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.154:6805", 2024-09-20T22:57:56.969 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 493899927 2024-09-20T22:57:56.969 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:57:56.970 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:57:56.970 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:56.970 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-09-20T22:57:56.970 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:57:56.970 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:56.970 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:57:56.970 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.154:6807", 2024-09-20T22:57:56.970 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 493899927 2024-09-20T22:57:56.970 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:57:56.970 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:57:56.970 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:56.970 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-09-20T22:57:56.970 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:57:56.970 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:56.970 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:57:56.971 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.154:6806", 2024-09-20T22:57:56.971 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 493899927 2024-09-20T22:57:56.971 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:57:56.971 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:57:56.971 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:56.971 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.154:6804/493899927", 2024-09-20T22:57:56.971 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.154:6805/493899927", 2024-09-20T22:57:56.971 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.154:6807/493899927", 2024-09-20T22:57:56.971 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.154:6806/493899927", 2024-09-20T22:57:56.971 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-09-20T22:57:56.971 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-09-20T22:57:56.971 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-09-20T22:57:56.971 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:57:56.971 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:56.972 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:56.972 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 6, 2024-09-20T22:57:56.972 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "56788d74-810e-4827-a615-958c12f8a368", 2024-09-20T22:57:56.972 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-09-20T22:57:56.972 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 1, 2024-09-20T22:57:56.972 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 1, 2024-09-20T22:57:56.972 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-09-20T22:57:56.972 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-09-20T22:57:56.972 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-09-20T22:57:56.972 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 48, 2024-09-20T22:57:56.972 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 989, 2024-09-20T22:57:56.972 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-09-20T22:57:56.972 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-09-20T22:57:56.972 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-09-20T22:57:56.972 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:57:56.973 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:56.973 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:57:56.973 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.154:6808", 2024-09-20T22:57:56.973 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2375091033 2024-09-20T22:57:56.973 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:57:56.973 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:57:56.973 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:56.973 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-09-20T22:57:56.973 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:57:56.973 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:56.973 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:57:56.973 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.154:6809", 2024-09-20T22:57:56.973 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2375091033 2024-09-20T22:57:56.973 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:57:56.973 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:57:56.974 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:56.974 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-09-20T22:57:56.974 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:57:56.974 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:56.974 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:57:56.974 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.154:6811", 2024-09-20T22:57:56.974 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2375091033 2024-09-20T22:57:56.974 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:57:56.974 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:57:56.974 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:56.974 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-09-20T22:57:56.974 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:57:56.974 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:56.975 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:57:56.975 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.154:6810", 2024-09-20T22:57:56.975 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 2375091033 2024-09-20T22:57:56.975 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:57:56.975 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:57:56.975 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:56.975 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.154:6808/2375091033", 2024-09-20T22:57:56.975 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.154:6809/2375091033", 2024-09-20T22:57:56.975 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.154:6811/2375091033", 2024-09-20T22:57:56.975 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.154:6810/2375091033", 2024-09-20T22:57:56.975 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-09-20T22:57:56.975 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-09-20T22:57:56.975 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-09-20T22:57:56.975 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:57:56.975 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:56.976 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:56.976 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 7, 2024-09-20T22:57:56.976 INFO:teuthology.orchestra.run.smithi084.stdout: "uuid": "83503727-afc1-4b95-8b96-ecb63f3338c2", 2024-09-20T22:57:56.976 INFO:teuthology.orchestra.run.smithi084.stdout: "up": 1, 2024-09-20T22:57:56.976 INFO:teuthology.orchestra.run.smithi084.stdout: "in": 0, 2024-09-20T22:57:56.976 INFO:teuthology.orchestra.run.smithi084.stdout: "weight": 0, 2024-09-20T22:57:56.976 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_affinity": 1, 2024-09-20T22:57:56.976 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_begin": 0, 2024-09-20T22:57:56.976 INFO:teuthology.orchestra.run.smithi084.stdout: "last_clean_end": 0, 2024-09-20T22:57:56.976 INFO:teuthology.orchestra.run.smithi084.stdout: "up_from": 59, 2024-09-20T22:57:56.976 INFO:teuthology.orchestra.run.smithi084.stdout: "up_thru": 836, 2024-09-20T22:57:56.976 INFO:teuthology.orchestra.run.smithi084.stdout: "down_at": 0, 2024-09-20T22:57:56.976 INFO:teuthology.orchestra.run.smithi084.stdout: "lost_at": 0, 2024-09-20T22:57:56.976 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addrs": { 2024-09-20T22:57:56.977 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:57:56.977 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:56.977 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:57:56.977 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.154:6812", 2024-09-20T22:57:56.977 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3717704491 2024-09-20T22:57:56.977 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:57:56.977 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:57:56.977 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:56.977 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addrs": { 2024-09-20T22:57:56.977 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:57:56.977 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:56.977 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:57:56.977 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.154:6813", 2024-09-20T22:57:56.977 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3717704491 2024-09-20T22:57:56.977 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:57:56.978 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:57:56.978 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:56.978 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addrs": { 2024-09-20T22:57:56.978 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:57:56.978 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:56.978 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:57:56.978 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.154:6815", 2024-09-20T22:57:56.978 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3717704491 2024-09-20T22:57:56.978 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:57:56.978 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:57:56.978 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:56.978 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addrs": { 2024-09-20T22:57:56.978 INFO:teuthology.orchestra.run.smithi084.stdout: "addrvec": [ 2024-09-20T22:57:56.978 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:56.978 INFO:teuthology.orchestra.run.smithi084.stdout: "type": "v2", 2024-09-20T22:57:56.979 INFO:teuthology.orchestra.run.smithi084.stdout: "addr": "172.21.15.154:6814", 2024-09-20T22:57:56.979 INFO:teuthology.orchestra.run.smithi084.stdout: "nonce": 3717704491 2024-09-20T22:57:56.979 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:57:56.979 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:57:56.979 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:56.979 INFO:teuthology.orchestra.run.smithi084.stdout: "public_addr": "172.21.15.154:6812/3717704491", 2024-09-20T22:57:56.979 INFO:teuthology.orchestra.run.smithi084.stdout: "cluster_addr": "172.21.15.154:6813/3717704491", 2024-09-20T22:57:56.979 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_back_addr": "172.21.15.154:6815/3717704491", 2024-09-20T22:57:56.979 INFO:teuthology.orchestra.run.smithi084.stdout: "heartbeat_front_addr": "172.21.15.154:6814/3717704491", 2024-09-20T22:57:56.979 INFO:teuthology.orchestra.run.smithi084.stdout: "state": [ 2024-09-20T22:57:56.979 INFO:teuthology.orchestra.run.smithi084.stdout: "exists", 2024-09-20T22:57:56.979 INFO:teuthology.orchestra.run.smithi084.stdout: "up" 2024-09-20T22:57:56.979 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:57:56.979 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:57:56.980 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2024-09-20T22:57:56.980 INFO:teuthology.orchestra.run.smithi084.stdout: "osd_xinfo": [ 2024-09-20T22:57:56.980 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:56.980 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 0, 2024-09-20T22:57:56.980 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-09-20T22:57:56.980 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-09-20T22:57:56.980 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-09-20T22:57:56.980 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540701547738038271, 2024-09-20T22:57:56.980 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-09-20T22:57:56.980 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-09-20T22:57:42.287953+0000", 2024-09-20T22:57:56.980 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-09-20T22:57:56.980 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:56.980 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:56.980 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 1, 2024-09-20T22:57:56.981 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-09-20T22:57:56.981 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-09-20T22:57:56.981 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-09-20T22:57:56.981 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540701547738038271, 2024-09-20T22:57:56.981 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-09-20T22:57:56.981 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-09-20T22:57:38.488570+0000", 2024-09-20T22:57:56.981 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-09-20T22:57:56.981 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:56.981 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:56.981 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 2, 2024-09-20T22:57:56.981 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-09-20T22:57:56.982 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-09-20T22:57:56.982 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-09-20T22:57:56.982 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540701547738038271, 2024-09-20T22:57:56.982 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-09-20T22:57:56.982 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-09-20T22:57:08.865769+0000", 2024-09-20T22:57:56.982 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-09-20T22:57:56.982 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:56.982 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:56.982 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 3, 2024-09-20T22:57:56.982 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-09-20T22:57:56.982 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-09-20T22:57:56.982 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-09-20T22:57:56.982 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540701547738038271, 2024-09-20T22:57:56.983 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-09-20T22:57:56.983 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-09-20T22:57:48.144965+0000", 2024-09-20T22:57:56.983 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 939 2024-09-20T22:57:56.983 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:56.983 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:56.983 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 4, 2024-09-20T22:57:56.983 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-09-20T22:57:56.983 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-09-20T22:57:56.983 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-09-20T22:57:56.983 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540701547738038271, 2024-09-20T22:57:56.983 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-09-20T22:57:56.983 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-09-20T22:56:43.520988+0000", 2024-09-20T22:57:56.983 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-09-20T22:57:56.984 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:56.984 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:56.984 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 5, 2024-09-20T22:57:56.984 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-09-20T22:57:56.984 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-09-20T22:57:56.984 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-09-20T22:57:56.984 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540701547738038271, 2024-09-20T22:57:56.984 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-09-20T22:57:56.984 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-09-20T22:57:21.222014+0000", 2024-09-20T22:57:56.984 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-09-20T22:57:56.984 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:56.984 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:56.984 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 6, 2024-09-20T22:57:56.985 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-09-20T22:57:56.985 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-09-20T22:57:56.985 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-09-20T22:57:56.985 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540701547738038271, 2024-09-20T22:57:56.985 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 0, 2024-09-20T22:57:56.985 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-09-20T22:57:44.138630+0000", 2024-09-20T22:57:56.985 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-09-20T22:57:56.985 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:56.985 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:56.985 INFO:teuthology.orchestra.run.smithi084.stdout: "osd": 7, 2024-09-20T22:57:56.985 INFO:teuthology.orchestra.run.smithi084.stdout: "down_stamp": "0.000000", 2024-09-20T22:57:56.985 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_probability": 0, 2024-09-20T22:57:56.986 INFO:teuthology.orchestra.run.smithi084.stdout: "laggy_interval": 0, 2024-09-20T22:57:56.986 INFO:teuthology.orchestra.run.smithi084.stdout: "features": 4540701547738038271, 2024-09-20T22:57:56.986 INFO:teuthology.orchestra.run.smithi084.stdout: "old_weight": 65536, 2024-09-20T22:57:56.986 INFO:teuthology.orchestra.run.smithi084.stdout: "last_purged_snaps_scrub": "2024-09-20T22:57:45.361525+0000", 2024-09-20T22:57:56.986 INFO:teuthology.orchestra.run.smithi084.stdout: "dead_epoch": 0 2024-09-20T22:57:56.986 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:57:56.986 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2024-09-20T22:57:56.986 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_upmap": [], 2024-09-20T22:57:56.986 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_upmap_items": [ 2024-09-20T22:57:56.986 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:56.986 INFO:teuthology.orchestra.run.smithi084.stdout: "pgid": "3.a", 2024-09-20T22:57:56.986 INFO:teuthology.orchestra.run.smithi084.stdout: "mappings": [ 2024-09-20T22:57:56.987 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:56.987 INFO:teuthology.orchestra.run.smithi084.stdout: "from": 6, 2024-09-20T22:57:56.987 INFO:teuthology.orchestra.run.smithi084.stdout: "to": 2 2024-09-20T22:57:56.987 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:57:56.987 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:57:56.987 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:56.987 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:56.987 INFO:teuthology.orchestra.run.smithi084.stdout: "pgid": "4.4", 2024-09-20T22:57:56.987 INFO:teuthology.orchestra.run.smithi084.stdout: "mappings": [ 2024-09-20T22:57:56.987 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:56.987 INFO:teuthology.orchestra.run.smithi084.stdout: "from": 1, 2024-09-20T22:57:56.987 INFO:teuthology.orchestra.run.smithi084.stdout: "to": 5 2024-09-20T22:57:56.987 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:57:56.987 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:57:56.987 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:56.988 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:56.988 INFO:teuthology.orchestra.run.smithi084.stdout: "pgid": "4.6", 2024-09-20T22:57:56.988 INFO:teuthology.orchestra.run.smithi084.stdout: "mappings": [ 2024-09-20T22:57:56.988 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:56.988 INFO:teuthology.orchestra.run.smithi084.stdout: "from": 1, 2024-09-20T22:57:56.988 INFO:teuthology.orchestra.run.smithi084.stdout: "to": 5 2024-09-20T22:57:56.988 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:57:56.988 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:57:56.988 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:56.988 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:56.988 INFO:teuthology.orchestra.run.smithi084.stdout: "pgid": "4.9", 2024-09-20T22:57:56.988 INFO:teuthology.orchestra.run.smithi084.stdout: "mappings": [ 2024-09-20T22:57:56.988 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:56.989 INFO:teuthology.orchestra.run.smithi084.stdout: "from": 3, 2024-09-20T22:57:56.989 INFO:teuthology.orchestra.run.smithi084.stdout: "to": 2 2024-09-20T22:57:56.989 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:57:56.989 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:57:56.989 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:56.989 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:56.989 INFO:teuthology.orchestra.run.smithi084.stdout: "pgid": "4.11", 2024-09-20T22:57:56.989 INFO:teuthology.orchestra.run.smithi084.stdout: "mappings": [ 2024-09-20T22:57:56.989 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:56.989 INFO:teuthology.orchestra.run.smithi084.stdout: "from": 3, 2024-09-20T22:57:56.989 INFO:teuthology.orchestra.run.smithi084.stdout: "to": 5 2024-09-20T22:57:56.989 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:57:56.989 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:57:56.989 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:56.990 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:56.990 INFO:teuthology.orchestra.run.smithi084.stdout: "pgid": "4.1b", 2024-09-20T22:57:56.990 INFO:teuthology.orchestra.run.smithi084.stdout: "mappings": [ 2024-09-20T22:57:56.990 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:56.990 INFO:teuthology.orchestra.run.smithi084.stdout: "from": 1, 2024-09-20T22:57:56.990 INFO:teuthology.orchestra.run.smithi084.stdout: "to": 2 2024-09-20T22:57:56.990 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:57:56.990 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:57:56.990 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:57:56.990 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2024-09-20T22:57:56.990 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_upmap_primaries": [], 2024-09-20T22:57:56.990 INFO:teuthology.orchestra.run.smithi084.stdout: "pg_temp": [], 2024-09-20T22:57:56.990 INFO:teuthology.orchestra.run.smithi084.stdout: "primary_temp": [], 2024-09-20T22:57:56.990 INFO:teuthology.orchestra.run.smithi084.stdout: "blocklist": { 2024-09-20T22:57:56.990 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/542020545": "2024-09-21T22:42:07.596045+0000", 2024-09-20T22:57:56.991 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/25873236": "2024-09-21T22:36:54.917812+0000", 2024-09-20T22:57:56.991 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:6800/820908470": "2024-09-21T22:36:54.917812+0000", 2024-09-20T22:57:56.991 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/567614908": "2024-09-21T22:36:54.917812+0000", 2024-09-20T22:57:56.991 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/2963622459": "2024-09-21T22:42:07.596045+0000", 2024-09-20T22:57:56.991 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/1176849276": "2024-09-21T22:37:17.690104+0000", 2024-09-20T22:57:56.991 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:6800/2932420050": "2024-09-21T22:37:17.690104+0000", 2024-09-20T22:57:56.991 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/3879263635": "2024-09-21T22:37:17.690104+0000", 2024-09-20T22:57:56.991 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/969472279": "2024-09-21T22:36:54.917812+0000", 2024-09-20T22:57:56.991 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/1361143214": "2024-09-21T22:37:17.690104+0000", 2024-09-20T22:57:56.991 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/3353126955": "2024-09-21T22:42:07.596045+0000", 2024-09-20T22:57:56.991 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:6800/4024278029": "2024-09-21T22:42:07.596045+0000", 2024-09-20T22:57:56.991 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/547902262": "2024-09-21T22:42:07.596045+0000", 2024-09-20T22:57:56.991 INFO:teuthology.orchestra.run.smithi084.stdout: "172.21.15.84:0/1242929317": "2024-09-21T22:42:07.596045+0000" 2024-09-20T22:57:56.991 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:56.991 INFO:teuthology.orchestra.run.smithi084.stdout: "range_blocklist": {}, 2024-09-20T22:57:56.992 INFO:teuthology.orchestra.run.smithi084.stdout: "erasure_code_profiles": { 2024-09-20T22:57:56.992 INFO:teuthology.orchestra.run.smithi084.stdout: "default": { 2024-09-20T22:57:56.992 INFO:teuthology.orchestra.run.smithi084.stdout: "crush-failure-domain": "osd", 2024-09-20T22:57:56.992 INFO:teuthology.orchestra.run.smithi084.stdout: "k": "2", 2024-09-20T22:57:56.992 INFO:teuthology.orchestra.run.smithi084.stdout: "m": "1", 2024-09-20T22:57:56.992 INFO:teuthology.orchestra.run.smithi084.stdout: "plugin": "jerasure", 2024-09-20T22:57:56.992 INFO:teuthology.orchestra.run.smithi084.stdout: "technique": "reed_sol_van" 2024-09-20T22:57:56.992 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:57:56.992 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:56.992 INFO:teuthology.orchestra.run.smithi084.stdout: "removed_snaps_queue": [ 2024-09-20T22:57:56.992 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:56.992 INFO:teuthology.orchestra.run.smithi084.stdout: "pool": 7, 2024-09-20T22:57:56.992 INFO:teuthology.orchestra.run.smithi084.stdout: "snaps": [ 2024-09-20T22:57:56.992 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:56.992 INFO:teuthology.orchestra.run.smithi084.stdout: "begin": 363, 2024-09-20T22:57:56.993 INFO:teuthology.orchestra.run.smithi084.stdout: "length": 1 2024-09-20T22:57:56.993 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:56.993 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:56.993 INFO:teuthology.orchestra.run.smithi084.stdout: "begin": 371, 2024-09-20T22:57:56.993 INFO:teuthology.orchestra.run.smithi084.stdout: "length": 1 2024-09-20T22:57:56.993 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:56.993 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:56.993 INFO:teuthology.orchestra.run.smithi084.stdout: "begin": 381, 2024-09-20T22:57:56.993 INFO:teuthology.orchestra.run.smithi084.stdout: "length": 1 2024-09-20T22:57:56.993 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:56.993 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:56.993 INFO:teuthology.orchestra.run.smithi084.stdout: "begin": 384, 2024-09-20T22:57:56.993 INFO:teuthology.orchestra.run.smithi084.stdout: "length": 1 2024-09-20T22:57:56.993 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:56.993 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:56.994 INFO:teuthology.orchestra.run.smithi084.stdout: "begin": 386, 2024-09-20T22:57:56.994 INFO:teuthology.orchestra.run.smithi084.stdout: "length": 3 2024-09-20T22:57:56.994 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:56.994 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:56.994 INFO:teuthology.orchestra.run.smithi084.stdout: "begin": 390, 2024-09-20T22:57:56.994 INFO:teuthology.orchestra.run.smithi084.stdout: "length": 8 2024-09-20T22:57:56.994 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:56.994 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:56.994 INFO:teuthology.orchestra.run.smithi084.stdout: "begin": 400, 2024-09-20T22:57:56.994 INFO:teuthology.orchestra.run.smithi084.stdout: "length": 16 2024-09-20T22:57:56.994 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:56.994 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:56.994 INFO:teuthology.orchestra.run.smithi084.stdout: "begin": 417, 2024-09-20T22:57:56.994 INFO:teuthology.orchestra.run.smithi084.stdout: "length": 1 2024-09-20T22:57:56.994 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:56.995 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:56.995 INFO:teuthology.orchestra.run.smithi084.stdout: "begin": 419, 2024-09-20T22:57:56.995 INFO:teuthology.orchestra.run.smithi084.stdout: "length": 2 2024-09-20T22:57:56.995 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:56.995 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:56.995 INFO:teuthology.orchestra.run.smithi084.stdout: "begin": 422, 2024-09-20T22:57:56.995 INFO:teuthology.orchestra.run.smithi084.stdout: "length": 1 2024-09-20T22:57:56.995 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:56.995 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:56.995 INFO:teuthology.orchestra.run.smithi084.stdout: "begin": 426, 2024-09-20T22:57:56.995 INFO:teuthology.orchestra.run.smithi084.stdout: "length": 2 2024-09-20T22:57:56.995 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:56.995 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:56.995 INFO:teuthology.orchestra.run.smithi084.stdout: "begin": 431, 2024-09-20T22:57:56.995 INFO:teuthology.orchestra.run.smithi084.stdout: "length": 1 2024-09-20T22:57:56.996 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:56.996 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:56.996 INFO:teuthology.orchestra.run.smithi084.stdout: "begin": 437, 2024-09-20T22:57:56.996 INFO:teuthology.orchestra.run.smithi084.stdout: "length": 1 2024-09-20T22:57:56.996 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:56.996 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:56.996 INFO:teuthology.orchestra.run.smithi084.stdout: "begin": 439, 2024-09-20T22:57:56.996 INFO:teuthology.orchestra.run.smithi084.stdout: "length": 1 2024-09-20T22:57:56.996 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:56.996 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:56.996 INFO:teuthology.orchestra.run.smithi084.stdout: "begin": 442, 2024-09-20T22:57:56.996 INFO:teuthology.orchestra.run.smithi084.stdout: "length": 1 2024-09-20T22:57:56.996 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:56.996 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:56.996 INFO:teuthology.orchestra.run.smithi084.stdout: "begin": 445, 2024-09-20T22:57:56.997 INFO:teuthology.orchestra.run.smithi084.stdout: "length": 2 2024-09-20T22:57:56.997 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:56.997 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:56.997 INFO:teuthology.orchestra.run.smithi084.stdout: "begin": 448, 2024-09-20T22:57:56.997 INFO:teuthology.orchestra.run.smithi084.stdout: "length": 2 2024-09-20T22:57:56.997 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:56.997 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:56.997 INFO:teuthology.orchestra.run.smithi084.stdout: "begin": 453, 2024-09-20T22:57:56.997 INFO:teuthology.orchestra.run.smithi084.stdout: "length": 1 2024-09-20T22:57:56.997 INFO:teuthology.orchestra.run.smithi084.stdout: }, 2024-09-20T22:57:56.997 INFO:teuthology.orchestra.run.smithi084.stdout: { 2024-09-20T22:57:56.997 INFO:teuthology.orchestra.run.smithi084.stdout: "begin": 455, 2024-09-20T22:57:56.997 INFO:teuthology.orchestra.run.smithi084.stdout: "length": 1 2024-09-20T22:57:56.997 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:57:56.997 INFO:teuthology.orchestra.run.smithi084.stdout: ] 2024-09-20T22:57:56.998 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:57:56.998 INFO:teuthology.orchestra.run.smithi084.stdout: ], 2024-09-20T22:57:56.998 INFO:teuthology.orchestra.run.smithi084.stdout: "new_removed_snaps": [], 2024-09-20T22:57:56.998 INFO:teuthology.orchestra.run.smithi084.stdout: "new_purged_snaps": [], 2024-09-20T22:57:56.998 INFO:teuthology.orchestra.run.smithi084.stdout: "crush_node_flags": {}, 2024-09-20T22:57:56.998 INFO:teuthology.orchestra.run.smithi084.stdout: "device_class_flags": {}, 2024-09-20T22:57:56.998 INFO:teuthology.orchestra.run.smithi084.stdout: "stretch_mode": { 2024-09-20T22:57:56.998 INFO:teuthology.orchestra.run.smithi084.stdout: "stretch_mode_enabled": false, 2024-09-20T22:57:56.998 INFO:teuthology.orchestra.run.smithi084.stdout: "stretch_bucket_count": 0, 2024-09-20T22:57:56.998 INFO:teuthology.orchestra.run.smithi084.stdout: "degraded_stretch_mode": 0, 2024-09-20T22:57:56.998 INFO:teuthology.orchestra.run.smithi084.stdout: "recovering_stretch_mode": 0, 2024-09-20T22:57:56.998 INFO:teuthology.orchestra.run.smithi084.stdout: "stretch_mode_bucket": 0 2024-09-20T22:57:56.998 INFO:teuthology.orchestra.run.smithi084.stdout: } 2024-09-20T22:57:56.998 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-20T22:57:57.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:56 smithi154 ceph-mon[33264]: 5.a scrub starts 2024-09-20T22:57:57.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:56 smithi154 ceph-mon[33264]: 5.a scrub ok 2024-09-20T22:57:57.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:56 smithi154 ceph-mon[33264]: 5.12 scrub starts 2024-09-20T22:57:57.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:56 smithi154 ceph-mon[33264]: 5.12 scrub ok 2024-09-20T22:57:57.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:56 smithi154 ceph-mon[33264]: 7.10 scrub starts 2024-09-20T22:57:57.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:56 smithi154 ceph-mon[33264]: 7.10 scrub ok 2024-09-20T22:57:57.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:56 smithi154 ceph-mon[33264]: 7.6 scrub starts 2024-09-20T22:57:57.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:56 smithi154 ceph-mon[33264]: 7.6 scrub ok 2024-09-20T22:57:57.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:56 smithi154 ceph-mon[33264]: pgmap v1392: 168 pgs: 2 active+clean+snaptrim, 166 active+clean; 793 MiB data, 2.3 GiB used, 623 GiB / 626 GiB avail; 14 MiB/s, 0 objects/s recovering 2024-09-20T22:57:57.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:56 smithi154 ceph-mon[33264]: 7.7 scrub starts 2024-09-20T22:57:57.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:56 smithi154 ceph-mon[33264]: 7.7 scrub ok 2024-09-20T22:57:57.039 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:57.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:57.040 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:57:57.141 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:57:57.161 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:57.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:57.162 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:57:57.263 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:57:57.284 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:57.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:57.285 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:57:57.386 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:57:57.408 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:57.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:57.409 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:57:57.510 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:57:57.531 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:57.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:57.532 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:57:57.633 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:57:57.657 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:57.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:57.658 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:57:57.758 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:57:57.807 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:57.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:57.808 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:57:57.844 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:57 smithi084 ceph-mon[40024]: 7.3 scrub starts 2024-09-20T22:57:57.845 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:57 smithi084 ceph-mon[40024]: 7.3 scrub ok 2024-09-20T22:57:57.845 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:57 smithi084 ceph-mon[40024]: 4.1e scrub starts 2024-09-20T22:57:57.845 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:57 smithi084 ceph-mon[40024]: 4.1e scrub ok 2024-09-20T22:57:57.845 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:57 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/559139086' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-20T22:57:57.845 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:57 smithi084 ceph-mon[29774]: 7.3 scrub starts 2024-09-20T22:57:57.845 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:57 smithi084 ceph-mon[29774]: 7.3 scrub ok 2024-09-20T22:57:57.845 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:57 smithi084 ceph-mon[29774]: 4.1e scrub starts 2024-09-20T22:57:57.845 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:57 smithi084 ceph-mon[29774]: 4.1e scrub ok 2024-09-20T22:57:57.845 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:57 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/559139086' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-20T22:57:57.909 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:57:57.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:57.932 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:57.932 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:57:58.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:57 smithi154 ceph-mon[33264]: 7.3 scrub starts 2024-09-20T22:57:58.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:57 smithi154 ceph-mon[33264]: 7.3 scrub ok 2024-09-20T22:57:58.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:57 smithi154 ceph-mon[33264]: 4.1e scrub starts 2024-09-20T22:57:58.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:57 smithi154 ceph-mon[33264]: 4.1e scrub ok 2024-09-20T22:57:58.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:57 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/559139086' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-20T22:57:58.033 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:57:58.055 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:58.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:58.056 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:57:58.146 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-20T22:57:58.146 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2024-09-20T22:57:58.146 INFO:teuthology.orchestra.run.smithi084.stdout: "num_ops": 0 2024-09-20T22:57:58.146 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-20T22:57:58.156 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:57:58.182 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:58.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:58.183 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:57:58.284 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:57:58.309 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:58.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:58.310 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:57:58.373 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is unset 2024-09-20T22:57:58.410 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:57:58.434 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:58.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:58.435 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:57:58.535 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:57:58.557 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:58.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:58.558 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:57:58.614 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:57:58 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: ::ffff:172.21.15.154 - - [20/Sep/2024:22:57:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-20T22:57:58.659 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:57:58.680 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:58.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:58.681 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:57:58.782 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:57:58.806 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:58.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:58.807 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:57:58.893 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:58 smithi084 ceph-mon[40024]: 7.8 scrub starts 2024-09-20T22:57:58.893 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:58 smithi084 ceph-mon[40024]: 7.8 scrub ok 2024-09-20T22:57:58.893 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:58 smithi084 ceph-mon[40024]: 6.4 scrub starts 2024-09-20T22:57:58.893 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:58 smithi084 ceph-mon[40024]: 6.4 scrub ok 2024-09-20T22:57:58.893 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:58 smithi084 ceph-mon[40024]: 7.2 scrub starts 2024-09-20T22:57:58.893 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:58 smithi084 ceph-mon[40024]: 7.2 scrub ok 2024-09-20T22:57:58.893 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:58 smithi084 ceph-mon[40024]: pgmap v1393: 168 pgs: 2 active+clean+snaptrim, 166 active+clean; 767 MiB data, 2.3 GiB used, 624 GiB / 626 GiB avail 2024-09-20T22:57:58.893 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:58 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/771826956' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:57:58.893 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:58 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:57:58.894 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:58 smithi084 ceph-mon[40024]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T22:57:58.894 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:58 smithi084 ceph-mon[40024]: Cluster is now healthy 2024-09-20T22:57:58.894 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:58 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T22:57:58.894 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:58 smithi084 ceph-mon[40024]: osdmap e993: 8 total, 8 up, 7 in 2024-09-20T22:57:58.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:58 smithi084 ceph-mon[29774]: 7.8 scrub starts 2024-09-20T22:57:58.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:58 smithi084 ceph-mon[29774]: 7.8 scrub ok 2024-09-20T22:57:58.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:58 smithi084 ceph-mon[29774]: 6.4 scrub starts 2024-09-20T22:57:58.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:58 smithi084 ceph-mon[29774]: 6.4 scrub ok 2024-09-20T22:57:58.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:58 smithi084 ceph-mon[29774]: 7.2 scrub starts 2024-09-20T22:57:58.894 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:58 smithi084 ceph-mon[29774]: 7.2 scrub ok 2024-09-20T22:57:58.895 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:58 smithi084 ceph-mon[29774]: pgmap v1393: 168 pgs: 2 active+clean+snaptrim, 166 active+clean; 767 MiB data, 2.3 GiB used, 624 GiB / 626 GiB avail 2024-09-20T22:57:58.895 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:58 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/771826956' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:57:58.895 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:58 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:57:58.895 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:58 smithi084 ceph-mon[29774]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T22:57:58.895 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:58 smithi084 ceph-mon[29774]: Cluster is now healthy 2024-09-20T22:57:58.895 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:58 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T22:57:58.895 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:58 smithi084 ceph-mon[29774]: osdmap e993: 8 total, 8 up, 7 in 2024-09-20T22:57:58.897 INFO:tasks.thrashosds.thrasher:j is {'epoch': 992, 'fsid': 'a9c0bc70-77a0-11ef-baf5-efdc52797490', 'created': '2024-09-20T22:36:30.240626+0000', 'modified': '2024-09-20T22:57:52.369055+0000', 'last_up_change': '2024-09-20T22:57:48.362816+0000', 'last_in_change': '2024-09-20T22:56:07.496853+0000', 'flags': 'nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5804032, 'flags_set': ['nodeep-scrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 19, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 7, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-20T22:39:21.984381+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 11, 'pg_placement_num': 11, 'pg_placement_num_target': 11, 'pg_num_target': 11, 'pg_num_pending': 11, 'last_pg_merge_meta': {'source_pgid': '1.b', 'ready_epoch': 299, 'last_epoch_started': 298, 'last_epoch_clean': 298, 'source_version': "77'128", 'target_version': "77'128"}, 'last_change': '769', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '299', 'last_force_op_resend_preluminous': '297', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.909999966621399, 'score_stable': 1.909999966621399, 'optimal_score': 1, 'raw_score_acting': 1.909999966621399, 'raw_score_stable': 1.909999966621399, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-09-20T22:41:35.820081+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '69', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 69, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.3299999237060547, 'score_stable': 2.3299999237060547, 'optimal_score': 0.8600000143051147, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-09-20T22:41:36.546389+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-09-20T22:41:38.454640+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '70', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.190000057220459, 'score_stable': 2.190000057220459, 'optimal_score': 1, 'raw_score_acting': 2.190000057220459, 'raw_score_stable': 2.190000057220459, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-09-20T22:41:40.407129+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '72', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.309999942779541, 'score_stable': 1.309999942779541, 'optimal_score': 1, 'raw_score_acting': 1.309999942779541, 'raw_score_stable': 1.309999942779541, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-09-20T22:41:42.476716+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '74', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 7, 'pool_name': 'unique_pool_0', 'create_time': '2024-09-20T22:42:51.354358+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 26, 'pg_placement_num': 26, 'pg_placement_num_target': 26, 'pg_num_target': 26, 'pg_num_pending': 26, 'last_pg_merge_meta': {'source_pgid': '7.10', 'ready_epoch': 484, 'last_epoch_started': 483, 'last_epoch_clean': 483, 'source_version': "480'172", 'target_version': "480'253"}, 'last_change': '990', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '484', 'last_force_op_resend_preluminous': '932', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 466, 'snap_epoch': 990, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}], 'osds': [{'osd': 0, 'uuid': 'ea36a489-6f91-4b2f-b784-b90506b9805d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 990, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6801', 'nonce': 268084626}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6802', 'nonce': 268084626}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6804', 'nonce': 268084626}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6803', 'nonce': 268084626}]}, 'public_addr': '172.21.15.84:6801/268084626', 'cluster_addr': '172.21.15.84:6802/268084626', 'heartbeat_back_addr': '172.21.15.84:6804/268084626', 'heartbeat_front_addr': '172.21.15.84:6803/268084626', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'c8649d2f-27ef-47f8-bc94-f1cd9e62bdb8', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 988, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6805', 'nonce': 418374136}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6806', 'nonce': 418374136}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6808', 'nonce': 418374136}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6807', 'nonce': 418374136}]}, 'public_addr': '172.21.15.84:6805/418374136', 'cluster_addr': '172.21.15.84:6806/418374136', 'heartbeat_back_addr': '172.21.15.84:6808/418374136', 'heartbeat_front_addr': '172.21.15.84:6807/418374136', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'c728eb48-a0f0-4de4-a651-7f3bec867280', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 988, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6809', 'nonce': 3977168611}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6810', 'nonce': 3977168611}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6812', 'nonce': 3977168611}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6811', 'nonce': 3977168611}]}, 'public_addr': '172.21.15.84:6809/3977168611', 'cluster_addr': '172.21.15.84:6810/3977168611', 'heartbeat_back_addr': '172.21.15.84:6812/3977168611', 'heartbeat_front_addr': '172.21.15.84:6811/3977168611', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '573f44eb-0853-4630-a7d8-9caaeb256f9b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 25, 'last_clean_end': 939, 'up_from': 988, 'up_thru': 990, 'down_at': 940, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6813', 'nonce': 3784535689}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6814', 'nonce': 3784535689}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6816', 'nonce': 3784535689}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.84:6815', 'nonce': 3784535689}]}, 'public_addr': '172.21.15.84:6813/3784535689', 'cluster_addr': '172.21.15.84:6814/3784535689', 'heartbeat_back_addr': '172.21.15.84:6816/3784535689', 'heartbeat_front_addr': '172.21.15.84:6815/3784535689', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'a52c3b94-c5f3-490f-8ac8-09061b9b2abb', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 830, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6800', 'nonce': 1793136002}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6801', 'nonce': 1793136002}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6803', 'nonce': 1793136002}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6802', 'nonce': 1793136002}]}, 'public_addr': '172.21.15.154:6800/1793136002', 'cluster_addr': '172.21.15.154:6801/1793136002', 'heartbeat_back_addr': '172.21.15.154:6803/1793136002', 'heartbeat_front_addr': '172.21.15.154:6802/1793136002', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '4fa0dc8e-25cc-46cd-96ff-33f20b6faec3', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 40, 'up_thru': 991, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6804', 'nonce': 493899927}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6805', 'nonce': 493899927}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6807', 'nonce': 493899927}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6806', 'nonce': 493899927}]}, 'public_addr': '172.21.15.154:6804/493899927', 'cluster_addr': '172.21.15.154:6805/493899927', 'heartbeat_back_addr': '172.21.15.154:6807/493899927', 'heartbeat_front_addr': '172.21.15.154:6806/493899927', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '56788d74-810e-4827-a615-958c12f8a368', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 48, 'up_thru': 989, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6808', 'nonce': 2375091033}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6809', 'nonce': 2375091033}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6811', 'nonce': 2375091033}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6810', 'nonce': 2375091033}]}, 'public_addr': '172.21.15.154:6808/2375091033', 'cluster_addr': '172.21.15.154:6809/2375091033', 'heartbeat_back_addr': '172.21.15.154:6811/2375091033', 'heartbeat_front_addr': '172.21.15.154:6810/2375091033', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '83503727-afc1-4b95-8b96-ecb63f3338c2', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 59, 'up_thru': 836, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6812', 'nonce': 3717704491}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6813', 'nonce': 3717704491}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6815', 'nonce': 3717704491}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.154:6814', 'nonce': 3717704491}]}, 'public_addr': '172.21.15.154:6812/3717704491', 'cluster_addr': '172.21.15.154:6813/3717704491', 'heartbeat_back_addr': '172.21.15.154:6815/3717704491', 'heartbeat_front_addr': '172.21.15.154:6814/3717704491', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-20T22:57:42.287953+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-20T22:57:38.488570+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-20T22:57:08.865769+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-20T22:57:48.144965+0000', 'dead_epoch': 939}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-20T22:56:43.520988+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-20T22:57:21.222014+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-20T22:57:44.138630+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-09-20T22:57:45.361525+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '3.a', 'mappings': [{'from': 6, 'to': 2}]}, {'pgid': '4.4', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.6', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.9', 'mappings': [{'from': 3, 'to': 2}]}, {'pgid': '4.11', 'mappings': [{'from': 3, 'to': 5}]}, {'pgid': '4.1b', 'mappings': [{'from': 1, 'to': 2}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.84:0/542020545': '2024-09-21T22:42:07.596045+0000', '172.21.15.84:0/25873236': '2024-09-21T22:36:54.917812+0000', '172.21.15.84:6800/820908470': '2024-09-21T22:36:54.917812+0000', '172.21.15.84:0/567614908': '2024-09-21T22:36:54.917812+0000', '172.21.15.84:0/2963622459': '2024-09-21T22:42:07.596045+0000', '172.21.15.84:0/1176849276': '2024-09-21T22:37:17.690104+0000', '172.21.15.84:6800/2932420050': '2024-09-21T22:37:17.690104+0000', '172.21.15.84:0/3879263635': '2024-09-21T22:37:17.690104+0000', '172.21.15.84:0/969472279': '2024-09-21T22:36:54.917812+0000', '172.21.15.84:0/1361143214': '2024-09-21T22:37:17.690104+0000', '172.21.15.84:0/3353126955': '2024-09-21T22:42:07.596045+0000', '172.21.15.84:6800/4024278029': '2024-09-21T22:42:07.596045+0000', '172.21.15.84:0/547902262': '2024-09-21T22:42:07.596045+0000', '172.21.15.84:0/1242929317': '2024-09-21T22:42:07.596045+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [{'pool': 7, 'snaps': [{'begin': 363, 'length': 1}, {'begin': 371, 'length': 1}, {'begin': 381, 'length': 1}, {'begin': 384, 'length': 1}, {'begin': 386, 'length': 3}, {'begin': 390, 'length': 8}, {'begin': 400, 'length': 16}, {'begin': 417, 'length': 1}, {'begin': 419, 'length': 2}, {'begin': 422, 'length': 1}, {'begin': 426, 'length': 2}, {'begin': 431, 'length': 1}, {'begin': 437, 'length': 1}, {'begin': 439, 'length': 1}, {'begin': 442, 'length': 1}, {'begin': 445, 'length': 2}, {'begin': 448, 'length': 2}, {'begin': 453, 'length': 1}, {'begin': 455, 'length': 1}]}], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-20T22:57:58.897 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph pg dump --format=json 2024-09-20T22:57:58.908 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:57:58.932 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:58.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:58.933 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:57:59.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:58 smithi154 ceph-mon[33264]: 7.8 scrub starts 2024-09-20T22:57:59.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:58 smithi154 ceph-mon[33264]: 7.8 scrub ok 2024-09-20T22:57:59.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:58 smithi154 ceph-mon[33264]: 6.4 scrub starts 2024-09-20T22:57:59.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:58 smithi154 ceph-mon[33264]: 6.4 scrub ok 2024-09-20T22:57:59.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:58 smithi154 ceph-mon[33264]: 7.2 scrub starts 2024-09-20T22:57:59.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:58 smithi154 ceph-mon[33264]: 7.2 scrub ok 2024-09-20T22:57:59.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:58 smithi154 ceph-mon[33264]: pgmap v1393: 168 pgs: 2 active+clean+snaptrim, 166 active+clean; 767 MiB data, 2.3 GiB used, 624 GiB / 626 GiB avail 2024-09-20T22:57:59.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:58 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/771826956' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:57:59.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:58 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:57:59.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:58 smithi154 ceph-mon[33264]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T22:57:59.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:58 smithi154 ceph-mon[33264]: Cluster is now healthy 2024-09-20T22:57:59.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:58 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T22:57:59.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:58 smithi154 ceph-mon[33264]: osdmap e993: 8 total, 8 up, 7 in 2024-09-20T22:57:59.034 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:57:59.059 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:59.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:59.060 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:57:59.160 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:57:59.181 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:59.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:59.182 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:57:59.283 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:57:59.305 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:59.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:59.306 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:57:59.406 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:57:59.433 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:59.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:59.434 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:57:59.534 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:57:59.555 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:59.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:59.556 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:57:59.657 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:57:59.680 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:59.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:59.681 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:57:59.686 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.3 dump_blocked_ops 2024-09-20T22:57:59.783 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:57:59.804 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:57:59.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:59.805 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:57:59.905 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:57:59.931 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:57:59.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:57:59.932 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:58:00.004 INFO:journalctl@ceph.iscsi.iscsi.a.smithi154.stdout:Sep 20 22:57:59 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-iscsi-iscsi-a[59730]: debug there is no tcmu-runner data available 2024-09-20T22:58:00.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:59 smithi154 ceph-mon[33264]: 1.7 scrub starts 2024-09-20T22:58:00.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:59 smithi154 ceph-mon[33264]: 1.7 scrub ok 2024-09-20T22:58:00.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:59 smithi154 ceph-mon[33264]: 5.1d scrub starts 2024-09-20T22:58:00.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:59 smithi154 ceph-mon[33264]: 5.1d scrub ok 2024-09-20T22:58:00.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:59 smithi154 ceph-mon[33264]: 7.1 scrub starts 2024-09-20T22:58:00.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:59 smithi154 ceph-mon[33264]: 7.1 scrub ok 2024-09-20T22:58:00.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:59 smithi154 ceph-mon[33264]: 1.8 scrub starts 2024-09-20T22:58:00.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:57:59 smithi154 ceph-mon[33264]: 1.8 scrub ok 2024-09-20T22:58:00.033 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:58:00.037 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:59 smithi084 ceph-mon[29774]: 1.7 scrub starts 2024-09-20T22:58:00.037 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:59 smithi084 ceph-mon[29774]: 1.7 scrub ok 2024-09-20T22:58:00.037 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:59 smithi084 ceph-mon[29774]: 5.1d scrub starts 2024-09-20T22:58:00.037 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:59 smithi084 ceph-mon[29774]: 5.1d scrub ok 2024-09-20T22:58:00.037 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:59 smithi084 ceph-mon[29774]: 7.1 scrub starts 2024-09-20T22:58:00.037 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:59 smithi084 ceph-mon[29774]: 7.1 scrub ok 2024-09-20T22:58:00.037 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:59 smithi084 ceph-mon[29774]: 1.8 scrub starts 2024-09-20T22:58:00.037 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:57:59 smithi084 ceph-mon[29774]: 1.8 scrub ok 2024-09-20T22:58:00.038 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:59 smithi084 ceph-mon[40024]: 1.7 scrub starts 2024-09-20T22:58:00.038 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:59 smithi084 ceph-mon[40024]: 1.7 scrub ok 2024-09-20T22:58:00.038 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:59 smithi084 ceph-mon[40024]: 5.1d scrub starts 2024-09-20T22:58:00.038 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:59 smithi084 ceph-mon[40024]: 5.1d scrub ok 2024-09-20T22:58:00.038 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:59 smithi084 ceph-mon[40024]: 7.1 scrub starts 2024-09-20T22:58:00.038 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:59 smithi084 ceph-mon[40024]: 7.1 scrub ok 2024-09-20T22:58:00.038 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:59 smithi084 ceph-mon[40024]: 1.8 scrub starts 2024-09-20T22:58:00.038 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:57:59 smithi084 ceph-mon[40024]: 1.8 scrub ok 2024-09-20T22:58:00.043 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:58:00.057 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:58:00.061 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:58:00.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:58:00.062 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:58:00.163 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:58:00.188 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:58:00.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:58:00.189 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:58:00.290 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:58:00.315 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:58:00.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:58:00.316 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:58:00.417 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:58:00.441 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:58:00.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:58:00.442 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:58:00.542 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:58:00.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:58:00.565 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:58:00.565 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:58:00.657 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:58:00.658 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:58:00.666 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:58:00.672 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:58:00.673 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:58:00.687 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:58:00.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:58:00.688 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:58:00.689 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:58:00.689 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:58:00.706 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-20T22:58:00.706 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T22:58:00.723 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:58:00.724 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:58:00.741 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:58:00.741 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:58:00.760 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:58:00.760 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:58:00.777 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:58:00.777 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:58:00.789 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:58:00.812 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:58:00.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:58:00.814 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:58:00.914 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:58:00.941 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:58:00.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:58:00.942 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:58:01.043 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:58:01.070 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:00 smithi084 ceph-mon[40024]: 4.13 scrub starts 2024-09-20T22:58:01.070 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:00 smithi084 ceph-mon[40024]: 4.13 scrub ok 2024-09-20T22:58:01.070 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:00 smithi084 ceph-mon[40024]: 7.14 scrub starts 2024-09-20T22:58:01.070 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:00 smithi084 ceph-mon[40024]: 7.14 scrub ok 2024-09-20T22:58:01.070 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:00 smithi084 ceph-mon[40024]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:58:01.070 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:00 smithi084 ceph-mon[40024]: pgmap v1395: 168 pgs: 2 active+clean+snaptrim, 166 active+clean; 766 MiB data, 2.3 GiB used, 624 GiB / 626 GiB avail 2024-09-20T22:58:01.070 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:00 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/1220750036' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:58:01.071 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:00 smithi084 ceph-mon[29774]: 4.13 scrub starts 2024-09-20T22:58:01.071 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:00 smithi084 ceph-mon[29774]: 4.13 scrub ok 2024-09-20T22:58:01.071 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:00 smithi084 ceph-mon[29774]: 7.14 scrub starts 2024-09-20T22:58:01.071 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:00 smithi084 ceph-mon[29774]: 7.14 scrub ok 2024-09-20T22:58:01.071 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:00 smithi084 ceph-mon[29774]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:58:01.071 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:00 smithi084 ceph-mon[29774]: pgmap v1395: 168 pgs: 2 active+clean+snaptrim, 166 active+clean; 766 MiB data, 2.3 GiB used, 624 GiB / 626 GiB avail 2024-09-20T22:58:01.071 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:00 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/1220750036' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:58:01.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:58:01.072 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:58:01.072 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:58:01.173 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:58:01.200 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:58:01.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:58:01.201 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:58:01.215 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:58:01.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:00 smithi154 ceph-mon[33264]: 4.13 scrub starts 2024-09-20T22:58:01.265 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:00 smithi154 ceph-mon[33264]: 4.13 scrub ok 2024-09-20T22:58:01.265 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:00 smithi154 ceph-mon[33264]: 7.14 scrub starts 2024-09-20T22:58:01.266 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:00 smithi154 ceph-mon[33264]: 7.14 scrub ok 2024-09-20T22:58:01.266 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:00 smithi154 ceph-mon[33264]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:58:01.266 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:00 smithi154 ceph-mon[33264]: pgmap v1395: 168 pgs: 2 active+clean+snaptrim, 166 active+clean; 766 MiB data, 2.3 GiB used, 624 GiB / 626 GiB avail 2024-09-20T22:58:01.266 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:00 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/1220750036' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:58:01.302 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:58:01.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:58:01.325 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:58:01.325 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:58:01.343 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-20T22:58:01.353 INFO:teuthology.orchestra.run.smithi084.stderr:dumped all 2024-09-20T22:58:01.426 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:58:01.447 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:58:01.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:58:01.448 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:58:01.549 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:58:01.572 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:58:01.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:58:01.573 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:58:01.674 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:58:01.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:58:01.699 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:58:01.699 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:58:01.800 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:58:01.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:58:01.825 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:58:01.826 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:58:01.926 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:58:01.930 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:01 smithi084 ceph-mon[40024]: 4.17 scrub starts 2024-09-20T22:58:01.930 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:01 smithi084 ceph-mon[40024]: 4.17 scrub ok 2024-09-20T22:58:01.930 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:01 smithi084 ceph-mon[40024]: from='client.17622 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T22:58:01.930 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:01 smithi084 ceph-mon[29774]: 4.17 scrub starts 2024-09-20T22:58:01.931 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:01 smithi084 ceph-mon[29774]: 4.17 scrub ok 2024-09-20T22:58:01.931 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:01 smithi084 ceph-mon[29774]: from='client.17622 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T22:58:01.953 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:58:01.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:58:01.955 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:58:02.056 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:58:02.079 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:58:02.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:58:02.080 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:58:02.084 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd set noscrub 2024-09-20T22:58:02.181 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:58:02.207 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:58:02.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:58:02.208 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:58:02.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:01 smithi154 ceph-mon[33264]: 4.17 scrub starts 2024-09-20T22:58:02.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:01 smithi154 ceph-mon[33264]: 4.17 scrub ok 2024-09-20T22:58:02.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:01 smithi154 ceph-mon[33264]: from='client.17622 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T22:58:02.309 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:58:02.331 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:58:02.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:58:02.332 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:58:02.433 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:58:02.456 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:58:02.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:58:02.456 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:58:02.545 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-20T22:58:02.545 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:58:02.557 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:58:02.584 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:58:02.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:58:02.585 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:58:02.686 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:58:02.711 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:58:02.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:58:02.712 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:58:02.812 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:58:02.838 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:58:02.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:58:02.839 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:58:02.939 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:58:02.962 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:58:02.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:58:02.962 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:58:03.063 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:58:03.084 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:58:03.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:58:03.085 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:58:03.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:02 smithi084 ceph-mon[29774]: 4.1c scrub starts 2024-09-20T22:58:03.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:02 smithi084 ceph-mon[29774]: 4.1c scrub ok 2024-09-20T22:58:03.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:02 smithi084 ceph-mon[29774]: 5.2 scrub starts 2024-09-20T22:58:03.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:02 smithi084 ceph-mon[29774]: 5.2 scrub ok 2024-09-20T22:58:03.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:02 smithi084 ceph-mon[29774]: 4.f scrub starts 2024-09-20T22:58:03.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:02 smithi084 ceph-mon[29774]: 4.f scrub ok 2024-09-20T22:58:03.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:02 smithi084 ceph-mon[29774]: 6.1c scrub starts 2024-09-20T22:58:03.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:02 smithi084 ceph-mon[29774]: 6.1c scrub ok 2024-09-20T22:58:03.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:02 smithi084 ceph-mon[29774]: pgmap v1396: 168 pgs: 2 active+clean+snaptrim, 166 active+clean; 766 MiB data, 2.3 GiB used, 624 GiB / 626 GiB avail 2024-09-20T22:58:03.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:02 smithi084 ceph-mon[40024]: 4.1c scrub starts 2024-09-20T22:58:03.116 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:02 smithi084 ceph-mon[40024]: 4.1c scrub ok 2024-09-20T22:58:03.116 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:02 smithi084 ceph-mon[40024]: 5.2 scrub starts 2024-09-20T22:58:03.116 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:02 smithi084 ceph-mon[40024]: 5.2 scrub ok 2024-09-20T22:58:03.116 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:02 smithi084 ceph-mon[40024]: 4.f scrub starts 2024-09-20T22:58:03.116 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:02 smithi084 ceph-mon[40024]: 4.f scrub ok 2024-09-20T22:58:03.116 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:02 smithi084 ceph-mon[40024]: 6.1c scrub starts 2024-09-20T22:58:03.116 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:02 smithi084 ceph-mon[40024]: 6.1c scrub ok 2024-09-20T22:58:03.116 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:02 smithi084 ceph-mon[40024]: pgmap v1396: 168 pgs: 2 active+clean+snaptrim, 166 active+clean; 766 MiB data, 2.3 GiB used, 624 GiB / 626 GiB avail 2024-09-20T22:58:03.186 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:58:03.212 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:58:03.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:58:03.213 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:58:03.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:02 smithi154 ceph-mon[33264]: 4.1c scrub starts 2024-09-20T22:58:03.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:02 smithi154 ceph-mon[33264]: 4.1c scrub ok 2024-09-20T22:58:03.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:02 smithi154 ceph-mon[33264]: 5.2 scrub starts 2024-09-20T22:58:03.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:02 smithi154 ceph-mon[33264]: 5.2 scrub ok 2024-09-20T22:58:03.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:02 smithi154 ceph-mon[33264]: 4.f scrub starts 2024-09-20T22:58:03.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:02 smithi154 ceph-mon[33264]: 4.f scrub ok 2024-09-20T22:58:03.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:02 smithi154 ceph-mon[33264]: 6.1c scrub starts 2024-09-20T22:58:03.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:02 smithi154 ceph-mon[33264]: 6.1c scrub ok 2024-09-20T22:58:03.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:02 smithi154 ceph-mon[33264]: pgmap v1396: 168 pgs: 2 active+clean+snaptrim, 166 active+clean; 766 MiB data, 2.3 GiB used, 624 GiB / 626 GiB avail 2024-09-20T22:58:03.301 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.3 dump_historic_ops 2024-09-20T22:58:03.314 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:58:03.365 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:58:03.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:58:03.366 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:58:03.467 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:58:03.488 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:58:03.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:58:03.489 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:58:03.590 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:58:03.611 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:58:03.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:58:03.612 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:58:03.705 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-20T22:58:03.713 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:58:03.738 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:58:03.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:58:03.739 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:58:03.777 INFO:teuthology.orchestra.run.smithi084.stdout:{"pg_ready":true,"pg_map":{"version":1395,"stamp":"2024-09-20T22:57:59.901199+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":803677379,"num_objects":543,"num_object_clones":296,"num_object_copies":1629,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":543,"num_whiteouts":10,"num_read":5869,"num_read_kb":4071691,"num_write":3359,"num_write_kb":1051284,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1161,"num_bytes_recovered":411519935,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":27086,"ondisk_log_size":27086,"up":504,"acting":504,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":504,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":2415604,"kb_used_data":1932924,"kb_used_omap":157,"kb_used_meta":482402,"kb_avail":653800460,"statfs":{"total":671965249536,"available":669491671040,"internally_reserved":0,"allocated":1979314176,"data_stored":2473605274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":161525,"internal_metadata":493979915},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-75788212,"num_objects":-35,"num_object_clones":-31,"num_object_copies":-105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-133,"num_objects_misplaced":-34,"num_objects_unfound":0,"num_objects_dirty":-35,"num_whiteouts":-2,"num_read":-1515,"num_read_kb":-1643759,"num_write":-1144,"num_write_kb":-408798,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-271,"num_bytes_recovered":-197868202,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002259"},"pg_stats":[{"pgid":"3.1f","version":"86'2","reported_seq":1259,"reported_epoch":993,"state":"active+clean","last_fresh":"2024-09-20T22:57:58.372182+0000","last_change":"2024-09-20T22:56:47.723346+0000","last_active":"2024-09-20T22:57:58.372182+0000","last_peered":"2024-09-20T22:57:58.372182+0000","last_clean":"2024-09-20T22:57:58.372182+0000","last_became_active":"2024-09-20T22:47:49.222114+0000","last_became_peered":"2024-09-20T22:47:49.222114+0000","last_unstale":"2024-09-20T22:57:58.372182+0000","last_undegraded":"2024-09-20T22:57:58.372182+0000","last_fullsized":"2024-09-20T22:57:58.372182+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'2","last_scrub_stamp":"2024-09-20T22:56:47.723216+0000","last_deep_scrub":"86'2","last_deep_scrub_stamp":"2024-09-20T22:56:47.723216+0000","last_clean_scrub_stamp":"2024-09-20T22:56:47.723216+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:59.566322+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":436,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"625'15","reported_seq":822,"reported_epoch":993,"state":"active+clean","last_fresh":"2024-09-20T22:57:58.419895+0000","last_change":"2024-09-20T22:56:14.882214+0000","last_active":"2024-09-20T22:57:58.419895+0000","last_peered":"2024-09-20T22:57:58.419895+0000","last_clean":"2024-09-20T22:57:58.419895+0000","last_became_active":"2024-09-20T22:55:17.341556+0000","last_became_peered":"2024-09-20T22:55:17.341556+0000","last_unstale":"2024-09-20T22:57:58.419895+0000","last_undegraded":"2024-09-20T22:57:58.419895+0000","last_fullsized":"2024-09-20T22:57:58.419895+0000","mapping_epoch":836,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":837,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'15","last_scrub_stamp":"2024-09-20T22:56:14.882123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:56:14.882123+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":456,"reported_epoch":993,"state":"active+clean","last_fresh":"2024-09-20T22:57:58.369764+0000","last_change":"2024-09-20T22:56:08.363427+0000","last_active":"2024-09-20T22:57:58.369764+0000","last_peered":"2024-09-20T22:57:58.369764+0000","last_clean":"2024-09-20T22:57:58.369764+0000","last_became_active":"2024-09-20T22:56:08.362405+0000","last_became_peered":"2024-09-20T22:56:08.362405+0000","last_unstale":"2024-09-20T22:57:58.369764+0000","last_undegraded":"2024-09-20T22:57:58.369764+0000","last_fullsized":"2024-09-20T22:57:58.369764+0000","mapping_epoch":887,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:55:34.919340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:55:34.919340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":1115,"reported_epoch":993,"state":"active+clean","last_fresh":"2024-09-20T22:57:58.420672+0000","last_change":"2024-09-20T22:56:22.519967+0000","last_active":"2024-09-20T22:57:58.420672+0000","last_peered":"2024-09-20T22:57:58.420672+0000","last_clean":"2024-09-20T22:57:58.420672+0000","last_became_active":"2024-09-20T22:55:17.342068+0000","last_became_peered":"2024-09-20T22:55:17.342068+0000","last_unstale":"2024-09-20T22:57:58.420672+0000","last_undegraded":"2024-09-20T22:57:58.420672+0000","last_fullsized":"2024-09-20T22:57:58.420672+0000","mapping_epoch":836,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":837,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:56:22.519884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:56:22.519884+0000","last_clean_scrub_stamp":"2024-09-20T22:56:22.519884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":1792,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-20T22:57:52.718363+0000","last_change":"2024-09-20T22:57:49.427504+0000","last_active":"2024-09-20T22:57:52.718363+0000","last_peered":"2024-09-20T22:57:52.718363+0000","last_clean":"2024-09-20T22:57:52.718363+0000","last_became_active":"2024-09-20T22:57:49.427228+0000","last_became_peered":"2024-09-20T22:57:49.427228+0000","last_unstale":"2024-09-20T22:57:52.718363+0000","last_undegraded":"2024-09-20T22:57:52.718363+0000","last_fullsized":"2024-09-20T22:57:52.718363+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:54:15.238344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:54:15.238344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:55.630176+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":1791,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-20T22:57:52.718894+0000","last_change":"2024-09-20T22:57:49.409395+0000","last_active":"2024-09-20T22:57:52.718894+0000","last_peered":"2024-09-20T22:57:52.718894+0000","last_clean":"2024-09-20T22:57:52.718894+0000","last_became_active":"2024-09-20T22:57:49.405928+0000","last_became_peered":"2024-09-20T22:57:49.405928+0000","last_unstale":"2024-09-20T22:57:52.718894+0000","last_undegraded":"2024-09-20T22:57:52.718894+0000","last_fullsized":"2024-09-20T22:57:52.718894+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:54:47.427927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:54:47.427927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:55.630282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"625'27","reported_seq":1869,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-20T22:57:52.718021+0000","last_change":"2024-09-20T22:57:49.416557+0000","last_active":"2024-09-20T22:57:52.718021+0000","last_peered":"2024-09-20T22:57:52.718021+0000","last_clean":"2024-09-20T22:57:52.718021+0000","last_became_active":"2024-09-20T22:57:49.413932+0000","last_became_peered":"2024-09-20T22:57:49.413932+0000","last_unstale":"2024-09-20T22:57:52.718021+0000","last_undegraded":"2024-09-20T22:57:52.718021+0000","last_fullsized":"2024-09-20T22:57:52.718021+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'27","last_scrub_stamp":"2024-09-20T22:54:45.390169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:54:45.390169+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:55.630250+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":1112,"reported_epoch":993,"state":"active+clean","last_fresh":"2024-09-20T22:57:58.419963+0000","last_change":"2024-09-20T22:57:13.194616+0000","last_active":"2024-09-20T22:57:58.419963+0000","last_peered":"2024-09-20T22:57:58.419963+0000","last_clean":"2024-09-20T22:57:58.419963+0000","last_became_active":"2024-09-20T22:55:17.341160+0000","last_became_peered":"2024-09-20T22:55:17.341160+0000","last_unstale":"2024-09-20T22:57:58.419963+0000","last_undegraded":"2024-09-20T22:57:58.419963+0000","last_fullsized":"2024-09-20T22:57:58.419963+0000","mapping_epoch":836,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":837,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:57:13.194499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:57:13.194499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:39.408547+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.19","version":"988'2004","reported_seq":6634,"reported_epoch":993,"state":"active+clean","last_fresh":"2024-09-20T22:57:58.419779+0000","last_change":"2024-09-20T22:57:48.440765+0000","last_active":"2024-09-20T22:57:58.419779+0000","last_peered":"2024-09-20T22:57:58.419779+0000","last_clean":"2024-09-20T22:57:58.419779+0000","last_became_active":"2024-09-20T22:56:58.364978+0000","last_became_peered":"2024-09-20T22:56:58.364978+0000","last_unstale":"2024-09-20T22:57:58.419779+0000","last_undegraded":"2024-09-20T22:57:58.419779+0000","last_fullsized":"2024-09-20T22:57:58.419779+0000","mapping_epoch":937,"log_start":"423'280","ondisk_log_start":"423'280","created":932,"last_epoch_clean":938,"parent":"0.0","parent_split_bits":5,"last_scrub":"942'1889","last_scrub_stamp":"2024-09-20T22:57:04.163707+0000","last_deep_scrub":"590'828","last_deep_scrub_stamp":"2024-09-20T22:51:10.909364+0000","last_clean_scrub_stamp":"2024-09-20T22:57:04.163707+0000","objects_scrubbed":23,"log_size":1724,"log_dups_size":649,"ondisk_log_size":1724,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:27.711921+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.070243691999999996,"stat_sum":{"num_bytes":56543939,"num_objects":23,"num_object_clones":20,"num_object_copies":69,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":19,"num_read_kb":6766,"num_write":21,"num_write_kb":6300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,5],"acting":[6,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"180","length":"1"},{"start":"18d","length":"1"},{"start":"190","length":"1"},{"start":"193","length":"1"},{"start":"196","length":"2"},{"start":"19a","length":"1"},{"start":"19c","length":"2"},{"start":"1a3","length":"2"},{"start":"1a6","length":"1"},{"start":"1aa","length":"2"},{"start":"1af","length":"1"},{"start":"1b5","length":"1"},{"start":"1b7","length":"1"},{"start":"1ba","length":"1"},{"start":"1bd","length":"2"},{"start":"1c0","length":"2"},{"start":"1c5","length":"1"},{"start":"1c7","length":"1"}]},{"pgid":"3.1d","version":"0'0","reported_seq":1925,"reported_epoch":993,"state":"active+clean","last_fresh":"2024-09-20T22:57:58.421116+0000","last_change":"2024-09-20T22:56:20.497450+0000","last_active":"2024-09-20T22:57:58.421116+0000","last_peered":"2024-09-20T22:57:58.421116+0000","last_clean":"2024-09-20T22:57:58.421116+0000","last_became_active":"2024-09-20T22:48:05.242059+0000","last_became_peered":"2024-09-20T22:48:05.242059+0000","last_unstale":"2024-09-20T22:57:58.421116+0000","last_undegraded":"2024-09-20T22:57:58.421116+0000","last_fullsized":"2024-09-20T22:57:58.421116+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:56:20.497377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:46:00.999939+0000","last_clean_scrub_stamp":"2024-09-20T22:56:20.497377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"625'15","reported_seq":681,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-20T22:57:52.718324+0000","last_change":"2024-09-20T22:57:49.405919+0000","last_active":"2024-09-20T22:57:52.718324+0000","last_peered":"2024-09-20T22:57:52.718324+0000","last_clean":"2024-09-20T22:57:52.718324+0000","last_became_active":"2024-09-20T22:57:49.403576+0000","last_became_peered":"2024-09-20T22:57:49.403576+0000","last_unstale":"2024-09-20T22:57:52.718324+0000","last_undegraded":"2024-09-20T22:57:52.718324+0000","last_fullsized":"2024-09-20T22:57:52.718324+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'15","last_scrub_stamp":"2024-09-20T22:53:51.486202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:53:51.486202+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:55.630070+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":1923,"reported_epoch":993,"state":"active+clean","last_fresh":"2024-09-20T22:57:58.420904+0000","last_change":"2024-09-20T22:56:42.536987+0000","last_active":"2024-09-20T22:57:58.420904+0000","last_peered":"2024-09-20T22:57:58.420904+0000","last_clean":"2024-09-20T22:57:58.420904+0000","last_became_active":"2024-09-20T22:56:08.356419+0000","last_became_peered":"2024-09-20T22:56:08.356419+0000","last_unstale":"2024-09-20T22:57:58.420904+0000","last_undegraded":"2024-09-20T22:57:58.420904+0000","last_fullsized":"2024-09-20T22:57:58.420904+0000","mapping_epoch":887,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:56:42.536863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:56:42.536863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":1273,"reported_epoch":993,"state":"active+clean","last_fresh":"2024-09-20T22:57:58.369385+0000","last_change":"2024-09-20T22:56:08.391304+0000","last_active":"2024-09-20T22:57:58.369385+0000","last_peered":"2024-09-20T22:57:58.369385+0000","last_clean":"2024-09-20T22:57:58.369385+0000","last_became_active":"2024-09-20T22:56:08.390640+0000","last_became_peered":"2024-09-20T22:56:08.390640+0000","last_unstale":"2024-09-20T22:57:58.369385+0000","last_undegraded":"2024-09-20T22:57:58.369385+0000","last_fullsized":"2024-09-20T22:57:58.369385+0000","mapping_epoch":887,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:55:55.454925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:49:08.664407+0000","last_clean_scrub_stamp":"2024-09-20T22:55:55.454925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.18","version":"988'1502","reported_seq":5919,"reported_epoch":993,"state":"active+clean","last_fresh":"2024-09-20T22:57:58.368997+0000","last_change":"2024-09-20T22:57:48.386070+0000","last_active":"2024-09-20T22:57:58.368997+0000","last_peered":"2024-09-20T22:57:58.368997+0000","last_clean":"2024-09-20T22:57:58.368997+0000","last_became_active":"2024-09-20T22:56:58.365227+0000","last_became_peered":"2024-09-20T22:56:58.365227+0000","last_unstale":"2024-09-20T22:57:58.368997+0000","last_undegraded":"2024-09-20T22:57:58.368997+0000","last_fullsized":"2024-09-20T22:57:58.368997+0000","mapping_epoch":937,"log_start":"426'268","ondisk_log_start":"426'268","created":932,"last_epoch_clean":938,"parent":"0.0","parent_split_bits":5,"last_scrub":"942'1432","last_scrub_stamp":"2024-09-20T22:57:03.731020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:57:03.731020+0000","objects_scrubbed":17,"log_size":1234,"log_dups_size":445,"ondisk_log_size":1234,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:30.917366+0000","scrub_duration":8,"objects_trimmed":1,"snaptrim_duration":0.019121448999999999,"stat_sum":{"num_bytes":27703213,"num_objects":15,"num_object_clones":13,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":21,"num_read_kb":27807,"num_write":4,"num_write_kb":996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"180","length":"1"},{"start":"18d","length":"1"},{"start":"190","length":"1"},{"start":"193","length":"1"},{"start":"196","length":"2"},{"start":"19a","length":"1"},{"start":"19c","length":"2"},{"start":"1a3","length":"2"},{"start":"1a6","length":"1"},{"start":"1aa","length":"2"},{"start":"1af","length":"1"},{"start":"1b5","length":"1"},{"start":"1b7","length":"1"},{"start":"1ba","length":"1"},{"start":"1bd","length":"2"},{"start":"1c0","length":"2"},{"start":"1c5","length":"1"},{"start":"1c7","length":"1"}]},{"pgid":"3.1c","version":"0'0","reported_seq":1937,"reported_epoch":993,"state":"active+clean","last_fresh":"2024-09-20T22:57:58.421218+0000","last_change":"2024-09-20T22:56:13.490259+0000","last_active":"2024-09-20T22:57:58.421218+0000","last_peered":"2024-09-20T22:57:58.421218+0000","last_clean":"2024-09-20T22:57:58.421218+0000","last_became_active":"2024-09-20T22:54:17.342619+0000","last_became_peered":"2024-09-20T22:54:17.342619+0000","last_unstale":"2024-09-20T22:57:58.421218+0000","last_undegraded":"2024-09-20T22:57:58.421218+0000","last_fullsized":"2024-09-20T22:57:58.421218+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:56:13.490178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:56:13.490178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"625'9","reported_seq":659,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-20T22:57:52.718443+0000","last_change":"2024-09-20T22:57:49.424289+0000","last_active":"2024-09-20T22:57:52.718443+0000","last_peered":"2024-09-20T22:57:52.718443+0000","last_clean":"2024-09-20T22:57:52.718443+0000","last_became_active":"2024-09-20T22:57:49.423498+0000","last_became_peered":"2024-09-20T22:57:49.423498+0000","last_unstale":"2024-09-20T22:57:52.718443+0000","last_undegraded":"2024-09-20T22:57:52.718443+0000","last_fullsized":"2024-09-20T22:57:52.718443+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'9","last_scrub_stamp":"2024-09-20T22:56:24.469428+0000","last_deep_scrub":"181'7","last_deep_scrub_stamp":"2024-09-20T22:48:41.049381+0000","last_clean_scrub_stamp":"2024-09-20T22:56:24.469428+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:55.630750+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":9,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":1714,"reported_epoch":993,"state":"active+clean","last_fresh":"2024-09-20T22:57:58.370232+0000","last_change":"2024-09-20T22:56:08.357180+0000","last_active":"2024-09-20T22:57:58.370232+0000","last_peered":"2024-09-20T22:57:58.370232+0000","last_clean":"2024-09-20T22:57:58.370232+0000","last_became_active":"2024-09-20T22:56:08.356857+0000","last_became_peered":"2024-09-20T22:56:08.356857+0000","last_unstale":"2024-09-20T22:57:58.370232+0000","last_undegraded":"2024-09-20T22:57:58.370232+0000","last_fullsized":"2024-09-20T22:57:58.370232+0000","mapping_epoch":887,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:55:59.860836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:47:40.361958+0000","last_clean_scrub_stamp":"2024-09-20T22:55:59.860836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":1925,"reported_epoch":993,"state":"active+clean","last_fresh":"2024-09-20T22:57:58.420449+0000","last_change":"2024-09-20T22:57:49.402679+0000","last_active":"2024-09-20T22:57:58.420449+0000","last_peered":"2024-09-20T22:57:58.420449+0000","last_clean":"2024-09-20T22:57:58.420449+0000","last_became_active":"2024-09-20T22:57:49.402447+0000","last_became_peered":"2024-09-20T22:57:49.402447+0000","last_unstale":"2024-09-20T22:57:58.420449+0000","last_undegraded":"2024-09-20T22:57:58.420449+0000","last_fullsized":"2024-09-20T22:57:58.420449+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:56:14.512566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:56:14.512566+0000","last_clean_scrub_stamp":"2024-09-20T22:56:14.512566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":339,"reported_epoch":993,"state":"active+clean","last_fresh":"2024-09-20T22:57:58.421695+0000","last_change":"2024-09-20T22:57:12.227470+0000","last_active":"2024-09-20T22:57:58.421695+0000","last_peered":"2024-09-20T22:57:58.421695+0000","last_clean":"2024-09-20T22:57:58.421695+0000","last_became_active":"2024-09-20T22:55:17.343025+0000","last_became_peered":"2024-09-20T22:55:17.343025+0000","last_unstale":"2024-09-20T22:57:58.421695+0000","last_undegraded":"2024-09-20T22:57:58.421695+0000","last_fullsized":"2024-09-20T22:57:58.421695+0000","mapping_epoch":836,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":837,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:57:12.227392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:57:12.227392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:25.441591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":1267,"reported_epoch":993,"state":"active+clean","last_fresh":"2024-09-20T22:57:58.370250+0000","last_change":"2024-09-20T22:57:12.715005+0000","last_active":"2024-09-20T22:57:58.370250+0000","last_peered":"2024-09-20T22:57:58.370250+0000","last_clean":"2024-09-20T22:57:58.370250+0000","last_became_active":"2024-09-20T22:56:08.391965+0000","last_became_peered":"2024-09-20T22:56:08.391965+0000","last_unstale":"2024-09-20T22:57:58.370250+0000","last_undegraded":"2024-09-20T22:57:58.370250+0000","last_fullsized":"2024-09-20T22:57:58.370250+0000","mapping_epoch":887,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:57:12.714907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:49:53.933601+0000","last_clean_scrub_stamp":"2024-09-20T22:57:12.714907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:20.512112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"625'34","reported_seq":2027,"reported_epoch":993,"state":"active+clean","last_fresh":"2024-09-20T22:57:58.421204+0000","last_change":"2024-09-20T22:57:49.430803+0000","last_active":"2024-09-20T22:57:58.421204+0000","last_peered":"2024-09-20T22:57:58.421204+0000","last_clean":"2024-09-20T22:57:58.421204+0000","last_became_active":"2024-09-20T22:57:49.430403+0000","last_became_peered":"2024-09-20T22:57:49.430403+0000","last_unstale":"2024-09-20T22:57:58.421204+0000","last_undegraded":"2024-09-20T22:57:58.421204+0000","last_fullsized":"2024-09-20T22:57:58.421204+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'34","last_scrub_stamp":"2024-09-20T22:55:04.301955+0000","last_deep_scrub":"113'22","last_deep_scrub_stamp":"2024-09-20T22:43:12.191878+0000","last_clean_scrub_stamp":"2024-09-20T22:55:04.301955+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":466,"reported_epoch":993,"state":"active+clean","last_fresh":"2024-09-20T22:57:58.390824+0000","last_change":"2024-09-20T22:57:58.390860+0000","last_active":"2024-09-20T22:57:58.390824+0000","last_peered":"2024-09-20T22:57:58.390824+0000","last_clean":"2024-09-20T22:57:58.390824+0000","last_became_active":"2024-09-20T22:54:17.349501+0000","last_became_peered":"2024-09-20T22:54:17.349501+0000","last_unstale":"2024-09-20T22:57:58.390824+0000","last_undegraded":"2024-09-20T22:57:58.390824+0000","last_fullsized":"2024-09-20T22:57:58.390824+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:57:58.390757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:55:57.217692+0000","last_clean_scrub_stamp":"2024-09-20T22:57:58.390757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:20.023219+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":1771,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-20T22:57:52.719022+0000","last_change":"2024-09-20T22:57:49.423684+0000","last_active":"2024-09-20T22:57:52.719022+0000","last_peered":"2024-09-20T22:57:52.719022+0000","last_clean":"2024-09-20T22:57:52.719022+0000","last_became_active":"2024-09-20T22:57:49.419671+0000","last_became_peered":"2024-09-20T22:57:49.419671+0000","last_unstale":"2024-09-20T22:57:52.719022+0000","last_undegraded":"2024-09-20T22:57:52.719022+0000","last_fullsized":"2024-09-20T22:57:52.719022+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:54:02.131659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:45:25.002199+0000","last_clean_scrub_stamp":"2024-09-20T22:54:02.131659+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:55.630033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":1122,"reported_epoch":993,"state":"active+clean","last_fresh":"2024-09-20T22:57:58.371596+0000","last_change":"2024-09-20T22:57:37.206101+0000","last_active":"2024-09-20T22:57:58.371596+0000","last_peered":"2024-09-20T22:57:58.371596+0000","last_clean":"2024-09-20T22:57:58.371596+0000","last_became_active":"2024-09-20T22:55:17.346400+0000","last_became_peered":"2024-09-20T22:55:17.346400+0000","last_unstale":"2024-09-20T22:57:58.371596+0000","last_undegraded":"2024-09-20T22:57:58.371596+0000","last_fullsized":"2024-09-20T22:57:58.371596+0000","mapping_epoch":836,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":837,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:57:37.206000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:50:41.885844+0000","last_clean_scrub_stamp":"2024-09-20T22:57:37.206000+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:48.705725+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"625'18","reported_seq":1849,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-20T22:57:52.718810+0000","last_change":"2024-09-20T22:57:49.406206+0000","last_active":"2024-09-20T22:57:52.718810+0000","last_peered":"2024-09-20T22:57:52.718810+0000","last_clean":"2024-09-20T22:57:52.718810+0000","last_became_active":"2024-09-20T22:57:49.403366+0000","last_became_peered":"2024-09-20T22:57:49.403366+0000","last_unstale":"2024-09-20T22:57:52.718810+0000","last_undegraded":"2024-09-20T22:57:52.718810+0000","last_fullsized":"2024-09-20T22:57:52.718810+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'18","last_scrub_stamp":"2024-09-20T22:54:18.284289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:54:18.284289+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:55.630213+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":752,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-20T22:57:52.718437+0000","last_change":"2024-09-20T22:57:49.431948+0000","last_active":"2024-09-20T22:57:52.718437+0000","last_peered":"2024-09-20T22:57:52.718437+0000","last_clean":"2024-09-20T22:57:52.718437+0000","last_became_active":"2024-09-20T22:57:49.426209+0000","last_became_peered":"2024-09-20T22:57:49.426209+0000","last_unstale":"2024-09-20T22:57:52.718437+0000","last_undegraded":"2024-09-20T22:57:52.718437+0000","last_fullsized":"2024-09-20T22:57:52.718437+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:56:18.447131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:56:18.447131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:55.630887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"154'2","reported_seq":411,"reported_epoch":993,"state":"active+clean","last_fresh":"2024-09-20T22:57:58.420866+0000","last_change":"2024-09-20T22:57:49.427430+0000","last_active":"2024-09-20T22:57:58.420866+0000","last_peered":"2024-09-20T22:57:58.420866+0000","last_clean":"2024-09-20T22:57:58.420866+0000","last_became_active":"2024-09-20T22:57:49.427136+0000","last_became_peered":"2024-09-20T22:57:49.427136+0000","last_unstale":"2024-09-20T22:57:58.420866+0000","last_undegraded":"2024-09-20T22:57:58.420866+0000","last_fullsized":"2024-09-20T22:57:58.420866+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"154'2","last_scrub_stamp":"2024-09-20T22:55:06.675346+0000","last_deep_scrub":"154'2","last_deep_scrub_stamp":"2024-09-20T22:43:54.134066+0000","last_clean_scrub_stamp":"2024-09-20T22:55:06.675346+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":403,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"894'2","reported_seq":472,"reported_epoch":993,"state":"active+clean","last_fresh":"2024-09-20T22:57:58.369697+0000","last_change":"2024-09-20T22:57:49.405855+0000","last_active":"2024-09-20T22:57:58.369697+0000","last_peered":"2024-09-20T22:57:58.369697+0000","last_clean":"2024-09-20T22:57:58.369697+0000","last_became_active":"2024-09-20T22:57:49.405458+0000","last_became_peered":"2024-09-20T22:57:49.405458+0000","last_unstale":"2024-09-20T22:57:58.369697+0000","last_undegraded":"2024-09-20T22:57:58.369697+0000","last_fullsized":"2024-09-20T22:57:58.369697+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"894'2","last_scrub_stamp":"2024-09-20T22:56:15.522649+0000","last_deep_scrub":"894'2","last_deep_scrub_stamp":"2024-09-20T22:56:15.522649+0000","last_clean_scrub_stamp":"2024-09-20T22:56:15.522649+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"625'25","reported_seq":1318,"reported_epoch":993,"state":"active+clean","last_fresh":"2024-09-20T22:57:58.371610+0000","last_change":"2024-09-20T22:57:56.738440+0000","last_active":"2024-09-20T22:57:58.371610+0000","last_peered":"2024-09-20T22:57:58.371610+0000","last_clean":"2024-09-20T22:57:58.371610+0000","last_became_active":"2024-09-20T22:57:49.411398+0000","last_became_peered":"2024-09-20T22:57:49.411398+0000","last_unstale":"2024-09-20T22:57:58.371610+0000","last_undegraded":"2024-09-20T22:57:58.371610+0000","last_fullsized":"2024-09-20T22:57:58.371610+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'25","last_scrub_stamp":"2024-09-20T22:57:56.738344+0000","last_deep_scrub":"151'15","last_deep_scrub_stamp":"2024-09-20T22:43:50.715761+0000","last_clean_scrub_stamp":"2024-09-20T22:57:56.738344+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:19.842693+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"888'12","reported_seq":2169,"reported_epoch":993,"state":"active+clean","last_fresh":"2024-09-20T22:57:58.422187+0000","last_change":"2024-09-20T22:56:42.107724+0000","last_active":"2024-09-20T22:57:58.422187+0000","last_peered":"2024-09-20T22:57:58.422187+0000","last_clean":"2024-09-20T22:57:58.422187+0000","last_became_active":"2024-09-20T22:56:08.368232+0000","last_became_peered":"2024-09-20T22:56:08.368232+0000","last_unstale":"2024-09-20T22:57:58.422187+0000","last_undegraded":"2024-09-20T22:57:58.422187+0000","last_fullsized":"2024-09-20T22:57:58.422187+0000","mapping_epoch":887,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":0,"last_scrub":"888'12","last_scrub_stamp":"2024-09-20T22:56:42.107635+0000","last_deep_scrub":"661'11","last_deep_scrub_stamp":"2024-09-20T22:52:21.907943+0000","last_clean_scrub_stamp":"2024-09-20T22:56:42.107635+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:09.490301+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.c","version":"989'3221","reported_seq":7911,"reported_epoch":993,"state":"active+clean","last_fresh":"2024-09-20T22:57:58.421842+0000","last_change":"2024-09-20T22:57:49.743510+0000","last_active":"2024-09-20T22:57:58.421842+0000","last_peered":"2024-09-20T22:57:58.421842+0000","last_clean":"2024-09-20T22:57:58.421842+0000","last_became_active":"2024-09-20T22:57:49.423914+0000","last_became_peered":"2024-09-20T22:57:49.423914+0000","last_unstale":"2024-09-20T22:57:58.421842+0000","last_undegraded":"2024-09-20T22:57:58.421842+0000","last_fullsized":"2024-09-20T22:57:58.421842+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"902'2941","last_scrub_stamp":"2024-09-20T22:56:22.993088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:56:22.993088+0000","objects_scrubbed":36,"log_size":3221,"log_dups_size":0,"ondisk_log_size":3221,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":5,"snaptrim_duration":0.079379093999999997,"stat_sum":{"num_bytes":108964875,"num_objects":47,"num_object_clones":40,"num_object_copies":141,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":2,"num_read":1155,"num_read_kb":1186566,"num_write":778,"num_write_kb":308677,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":189,"num_bytes_recovered":87252218,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"180","length":"1"},{"start":"18d","length":"1"},{"start":"190","length":"1"},{"start":"193","length":"1"},{"start":"196","length":"2"},{"start":"19a","length":"1"},{"start":"19c","length":"2"},{"start":"1a3","length":"2"},{"start":"1a6","length":"1"},{"start":"1aa","length":"2"},{"start":"1af","length":"1"},{"start":"1b5","length":"1"},{"start":"1b7","length":"1"},{"start":"1ba","length":"1"},{"start":"1bd","length":"2"},{"start":"1c0","length":"2"},{"start":"1c5","length":"1"},{"start":"1c7","length":"1"}]},{"pgid":"1.a","version":"77'128","reported_seq":2183,"reported_epoch":993,"state":"active+clean","last_fresh":"2024-09-20T22:57:58.420731+0000","last_change":"2024-09-20T22:57:41.688804+0000","last_active":"2024-09-20T22:57:58.420731+0000","last_peered":"2024-09-20T22:57:58.420731+0000","last_clean":"2024-09-20T22:57:58.420731+0000","last_became_active":"2024-09-20T22:42:52.407791+0000","last_became_peered":"2024-09-20T22:42:52.407791+0000","last_unstale":"2024-09-20T22:57:58.420731+0000","last_undegraded":"2024-09-20T22:57:58.420731+0000","last_fullsized":"2024-09-20T22:57:58.420731+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:57:41.688681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:57:41.688681+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:00.430067+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"625'27","reported_seq":483,"reported_epoch":993,"state":"active+clean","last_fresh":"2024-09-20T22:57:58.370998+0000","last_change":"2024-09-20T22:57:49.415583+0000","last_active":"2024-09-20T22:57:58.370998+0000","last_peered":"2024-09-20T22:57:58.370998+0000","last_clean":"2024-09-20T22:57:58.370998+0000","last_became_active":"2024-09-20T22:57:49.415260+0000","last_became_peered":"2024-09-20T22:57:49.415260+0000","last_unstale":"2024-09-20T22:57:58.370998+0000","last_undegraded":"2024-09-20T22:57:58.370998+0000","last_fullsized":"2024-09-20T22:57:58.370998+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'27","last_scrub_stamp":"2024-09-20T22:55:50.160922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:55:50.160922+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":1808,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-20T22:57:52.717583+0000","last_change":"2024-09-20T22:57:49.414013+0000","last_active":"2024-09-20T22:57:52.717583+0000","last_peered":"2024-09-20T22:57:52.717583+0000","last_clean":"2024-09-20T22:57:52.717583+0000","last_became_active":"2024-09-20T22:57:49.407789+0000","last_became_peered":"2024-09-20T22:57:49.407789+0000","last_unstale":"2024-09-20T22:57:52.717583+0000","last_undegraded":"2024-09-20T22:57:52.717583+0000","last_fullsized":"2024-09-20T22:57:52.717583+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:55:09.369892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:49:12.070675+0000","last_clean_scrub_stamp":"2024-09-20T22:55:09.369892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:55.630423+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"838'13","reported_seq":982,"reported_epoch":993,"state":"active+clean","last_fresh":"2024-09-20T22:57:58.421559+0000","last_change":"2024-09-20T22:56:46.487242+0000","last_active":"2024-09-20T22:57:58.421559+0000","last_peered":"2024-09-20T22:57:58.421559+0000","last_clean":"2024-09-20T22:57:58.421559+0000","last_became_active":"2024-09-20T22:55:17.343505+0000","last_became_peered":"2024-09-20T22:55:17.343505+0000","last_unstale":"2024-09-20T22:57:58.421559+0000","last_undegraded":"2024-09-20T22:57:58.421559+0000","last_fullsized":"2024-09-20T22:57:58.421559+0000","mapping_epoch":836,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":837,"parent":"0.0","parent_split_bits":0,"last_scrub":"838'13","last_scrub_stamp":"2024-09-20T22:56:46.487164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:56:46.487164+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:02.187115+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":1926,"reported_epoch":993,"state":"active+clean","last_fresh":"2024-09-20T22:57:58.420516+0000","last_change":"2024-09-20T22:56:41.547004+0000","last_active":"2024-09-20T22:57:58.420516+0000","last_peered":"2024-09-20T22:57:58.420516+0000","last_clean":"2024-09-20T22:57:58.420516+0000","last_became_active":"2024-09-20T22:54:17.340327+0000","last_became_peered":"2024-09-20T22:54:17.340327+0000","last_unstale":"2024-09-20T22:57:58.420516+0000","last_undegraded":"2024-09-20T22:57:58.420516+0000","last_fullsized":"2024-09-20T22:57:58.420516+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:56:41.546923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:56:41.546923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:04.615941+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3","version":"931'475","reported_seq":4680,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-20T22:57:56.654959+0000","last_change":"2024-09-20T22:57:56.654985+0000","last_active":"2024-09-20T22:57:56.654959+0000","last_peered":"2024-09-20T22:57:56.654959+0000","last_clean":"2024-09-20T22:57:56.654959+0000","last_became_active":"2024-09-20T22:57:49.410503+0000","last_became_peered":"2024-09-20T22:57:49.410503+0000","last_unstale":"2024-09-20T22:57:56.654959+0000","last_undegraded":"2024-09-20T22:57:56.654959+0000","last_fullsized":"2024-09-20T22:57:56.654959+0000","mapping_epoch":988,"log_start":"456'129","ondisk_log_start":"456'129","created":94,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"931'475","last_scrub_stamp":"2024-09-20T22:57:56.654880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:57:56.654880+0000","objects_scrubbed":0,"log_size":346,"log_dups_size":129,"ondisk_log_size":346,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:57.483771+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.059598702000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"16b","length":"1"},{"start":"173","length":"1"},{"start":"17d","length":"1"},{"start":"180","length":"1"},{"start":"182","length":"3"},{"start":"186","length":"8"},{"start":"190","length":"10"},{"start":"1a1","length":"1"},{"start":"1a3","length":"2"},{"start":"1a6","length":"1"},{"start":"1aa","length":"2"},{"start":"1af","length":"1"},{"start":"1b5","length":"1"},{"start":"1b7","length":"1"},{"start":"1ba","length":"1"},{"start":"1bd","length":"2"},{"start":"1c0","length":"2"},{"start":"1c5","length":"1"},{"start":"1c7","length":"1"}]},{"pgid":"1.5","version":"77'128","reported_seq":1198,"reported_epoch":993,"state":"active+clean","last_fresh":"2024-09-20T22:57:58.422625+0000","last_change":"2024-09-20T22:57:14.262350+0000","last_active":"2024-09-20T22:57:58.422625+0000","last_peered":"2024-09-20T22:57:58.422625+0000","last_clean":"2024-09-20T22:57:58.422625+0000","last_became_active":"2024-09-20T22:55:17.345460+0000","last_became_peered":"2024-09-20T22:55:17.345460+0000","last_unstale":"2024-09-20T22:57:58.422625+0000","last_undegraded":"2024-09-20T22:57:58.422625+0000","last_fullsized":"2024-09-20T22:57:58.422625+0000","mapping_epoch":836,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":837,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:57:14.262265+0000","last_deep_scrub":"77'128","last_deep_scrub_stamp":"2024-09-20T22:57:14.262265+0000","last_clean_scrub_stamp":"2024-09-20T22:57:14.262265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:39.904856+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"625'30","reported_seq":1909,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-20T22:57:52.717725+0000","last_change":"2024-09-20T22:57:49.414226+0000","last_active":"2024-09-20T22:57:52.717725+0000","last_peered":"2024-09-20T22:57:52.717725+0000","last_clean":"2024-09-20T22:57:52.717725+0000","last_became_active":"2024-09-20T22:57:49.408104+0000","last_became_peered":"2024-09-20T22:57:49.408104+0000","last_unstale":"2024-09-20T22:57:52.717725+0000","last_undegraded":"2024-09-20T22:57:52.717725+0000","last_fullsized":"2024-09-20T22:57:52.717725+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'30","last_scrub_stamp":"2024-09-20T22:53:48.346341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:53:48.346341+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:55.630000+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":39,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":1829,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-20T22:57:52.719269+0000","last_change":"2024-09-20T22:57:49.433634+0000","last_active":"2024-09-20T22:57:52.719269+0000","last_peered":"2024-09-20T22:57:52.719269+0000","last_clean":"2024-09-20T22:57:52.719269+0000","last_became_active":"2024-09-20T22:57:49.432799+0000","last_became_peered":"2024-09-20T22:57:49.432799+0000","last_unstale":"2024-09-20T22:57:52.719269+0000","last_undegraded":"2024-09-20T22:57:52.719269+0000","last_fullsized":"2024-09-20T22:57:52.719269+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:56:15.465230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:56:15.465230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:55.630784+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":745,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-20T22:57:52.717577+0000","last_change":"2024-09-20T22:57:49.416335+0000","last_active":"2024-09-20T22:57:52.717577+0000","last_peered":"2024-09-20T22:57:52.717577+0000","last_clean":"2024-09-20T22:57:52.717577+0000","last_became_active":"2024-09-20T22:57:49.413482+0000","last_became_peered":"2024-09-20T22:57:49.413482+0000","last_unstale":"2024-09-20T22:57:52.717577+0000","last_undegraded":"2024-09-20T22:57:52.717577+0000","last_fullsized":"2024-09-20T22:57:52.717577+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:56:47.432483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:56:47.432483+0000","last_clean_scrub_stamp":"2024-09-20T22:56:47.432483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:59.904016+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":694,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-20T22:57:52.718419+0000","last_change":"2024-09-20T22:57:49.404772+0000","last_active":"2024-09-20T22:57:52.718419+0000","last_peered":"2024-09-20T22:57:52.718419+0000","last_clean":"2024-09-20T22:57:52.718419+0000","last_became_active":"2024-09-20T22:57:49.402383+0000","last_became_peered":"2024-09-20T22:57:49.402383+0000","last_unstale":"2024-09-20T22:57:52.718419+0000","last_undegraded":"2024-09-20T22:57:52.718419+0000","last_fullsized":"2024-09-20T22:57:52.718419+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:55:48.456896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:53.036601+0000","last_clean_scrub_stamp":"2024-09-20T22:55:48.456896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:55.630574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2","version":"989'1026","reported_seq":4770,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-20T22:57:57.697845+0000","last_change":"2024-09-20T22:57:57.697875+0000","last_active":"2024-09-20T22:57:57.697845+0000","last_peered":"2024-09-20T22:57:57.697845+0000","last_clean":"2024-09-20T22:57:57.697845+0000","last_became_active":"2024-09-20T22:57:49.400698+0000","last_became_peered":"2024-09-20T22:57:49.400698+0000","last_unstale":"2024-09-20T22:57:57.697845+0000","last_undegraded":"2024-09-20T22:57:57.697845+0000","last_fullsized":"2024-09-20T22:57:57.697845+0000","mapping_epoch":988,"log_start":"471'464","ondisk_log_start":"471'464","created":94,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"989'1026","last_scrub_stamp":"2024-09-20T22:57:57.697789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:57:57.697789+0000","objects_scrubbed":5,"log_size":562,"log_dups_size":610,"ondisk_log_size":562,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:16.856606+0000","scrub_duration":2,"objects_trimmed":1,"snaptrim_duration":0.100157336,"stat_sum":{"num_bytes":15831841,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"16b","length":"1"},{"start":"173","length":"1"},{"start":"17d","length":"1"},{"start":"180","length":"1"},{"start":"182","length":"3"},{"start":"186","length":"8"},{"start":"190","length":"10"},{"start":"1a1","length":"1"},{"start":"1a3","length":"2"},{"start":"1a6","length":"1"},{"start":"1aa","length":"2"},{"start":"1af","length":"1"},{"start":"1b5","length":"1"},{"start":"1b7","length":"1"},{"start":"1ba","length":"1"},{"start":"1bd","length":"2"},{"start":"1c0","length":"2"},{"start":"1c5","length":"1"},{"start":"1c7","length":"1"}]},{"pgid":"1.4","version":"855'129","reported_seq":2116,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-20T22:57:52.718936+0000","last_change":"2024-09-20T22:57:49.426337+0000","last_active":"2024-09-20T22:57:52.718936+0000","last_peered":"2024-09-20T22:57:52.718936+0000","last_clean":"2024-09-20T22:57:52.718936+0000","last_became_active":"2024-09-20T22:57:49.423690+0000","last_became_peered":"2024-09-20T22:57:49.423690+0000","last_unstale":"2024-09-20T22:57:52.718936+0000","last_undegraded":"2024-09-20T22:57:52.718936+0000","last_fullsized":"2024-09-20T22:57:52.718936+0000","mapping_epoch":988,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":3,"last_scrub":"855'129","last_scrub_stamp":"2024-09-20T22:55:35.536654+0000","last_deep_scrub":"855'129","last_deep_scrub_stamp":"2024-09-20T22:55:35.536654+0000","last_clean_scrub_stamp":"2024-09-20T22:55:35.536654+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":59,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:55.630461+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"625'28","reported_seq":827,"reported_epoch":993,"state":"active+clean","last_fresh":"2024-09-20T22:57:58.420557+0000","last_change":"2024-09-20T22:57:05.659881+0000","last_active":"2024-09-20T22:57:58.420557+0000","last_peered":"2024-09-20T22:57:58.420557+0000","last_clean":"2024-09-20T22:57:58.420557+0000","last_became_active":"2024-09-20T22:55:17.344611+0000","last_became_peered":"2024-09-20T22:55:17.344611+0000","last_unstale":"2024-09-20T22:57:58.420557+0000","last_undegraded":"2024-09-20T22:57:58.420557+0000","last_fullsized":"2024-09-20T22:57:58.420557+0000","mapping_epoch":836,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":837,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'28","last_scrub_stamp":"2024-09-20T22:57:05.659800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:57:05.659800+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:10.175018+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":40,"num_read_kb":33,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.6","version":"855'2","reported_seq":1290,"reported_epoch":993,"state":"active+clean","last_fresh":"2024-09-20T22:57:58.371366+0000","last_change":"2024-09-20T22:55:35.573049+0000","last_active":"2024-09-20T22:57:58.371366+0000","last_peered":"2024-09-20T22:57:58.371366+0000","last_clean":"2024-09-20T22:57:58.371366+0000","last_became_active":"2024-09-20T22:54:17.340423+0000","last_became_peered":"2024-09-20T22:54:17.340423+0000","last_unstale":"2024-09-20T22:57:58.371366+0000","last_undegraded":"2024-09-20T22:57:58.371366+0000","last_fullsized":"2024-09-20T22:57:58.371366+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"855'2","last_scrub_stamp":"2024-09-20T22:55:35.572977+0000","last_deep_scrub":"855'2","last_deep_scrub_stamp":"2024-09-20T22:55:35.572977+0000","last_clean_scrub_stamp":"2024-09-20T22:55:35.572977+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":230,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"989'18","reported_seq":2001,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-20T22:57:52.717881+0000","last_change":"2024-09-20T22:57:49.413392+0000","last_active":"2024-09-20T22:57:52.717881+0000","last_peered":"2024-09-20T22:57:52.717881+0000","last_clean":"2024-09-20T22:57:52.717881+0000","last_became_active":"2024-09-20T22:57:49.401445+0000","last_became_peered":"2024-09-20T22:57:49.401445+0000","last_unstale":"2024-09-20T22:57:52.717881+0000","last_undegraded":"2024-09-20T22:57:52.717881+0000","last_fullsized":"2024-09-20T22:57:52.717881+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"778'16","last_scrub_stamp":"2024-09-20T22:55:10.378046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:55:10.378046+0000","objects_scrubbed":1,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:55.630353+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":227,"reported_epoch":993,"state":"active+clean","last_fresh":"2024-09-20T22:57:58.421307+0000","last_change":"2024-09-20T22:57:10.158684+0000","last_active":"2024-09-20T22:57:58.421307+0000","last_peered":"2024-09-20T22:57:58.421307+0000","last_clean":"2024-09-20T22:57:58.421307+0000","last_became_active":"2024-09-20T22:56:08.367182+0000","last_became_peered":"2024-09-20T22:56:08.367182+0000","last_unstale":"2024-09-20T22:57:58.421307+0000","last_undegraded":"2024-09-20T22:57:58.421307+0000","last_fullsized":"2024-09-20T22:57:58.421307+0000","mapping_epoch":887,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:57:10.158581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:57:10.158581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:32.505201+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1","version":"989'621","reported_seq":4906,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-20T22:57:52.718262+0000","last_change":"2024-09-20T22:57:49.713215+0000","last_active":"2024-09-20T22:57:52.718262+0000","last_peered":"2024-09-20T22:57:52.718262+0000","last_clean":"2024-09-20T22:57:52.718262+0000","last_became_active":"2024-09-20T22:57:49.404952+0000","last_became_peered":"2024-09-20T22:57:49.404952+0000","last_unstale":"2024-09-20T22:57:52.718262+0000","last_undegraded":"2024-09-20T22:57:52.718262+0000","last_fullsized":"2024-09-20T22:57:52.718262+0000","mapping_epoch":988,"log_start":"474'259","ondisk_log_start":"474'259","created":94,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"794'485","last_scrub_stamp":"2024-09-20T22:54:38.213249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:54:38.213249+0000","objects_scrubbed":8,"log_size":362,"log_dups_size":259,"ondisk_log_size":362,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:55.629774+0000","scrub_duration":3,"objects_trimmed":3,"snaptrim_duration":0.062553687999999996,"stat_sum":{"num_bytes":19838211,"num_objects":8,"num_object_clones":7,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":95,"num_read_kb":100306,"num_write":82,"num_write_kb":43338,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":19945014,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"16b","length":"1"},{"start":"173","length":"1"},{"start":"17d","length":"1"},{"start":"180","length":"1"},{"start":"182","length":"3"},{"start":"186","length":"8"},{"start":"190","length":"10"},{"start":"1a1","length":"1"},{"start":"1a3","length":"2"},{"start":"1a6","length":"1"},{"start":"1aa","length":"2"},{"start":"1af","length":"1"},{"start":"1b5","length":"1"},{"start":"1b7","length":"1"},{"start":"1ba","length":"1"},{"start":"1bd","length":"2"},{"start":"1c0","length":"2"},{"start":"1c5","length":"1"},{"start":"1c7","length":"1"}]},{"pgid":"1.7","version":"77'128","reported_seq":2250,"reported_epoch":993,"state":"active+clean","last_fresh":"2024-09-20T22:57:58.421781+0000","last_change":"2024-09-20T22:57:58.373829+0000","last_active":"2024-09-20T22:57:58.421781+0000","last_peered":"2024-09-20T22:57:58.421781+0000","last_clean":"2024-09-20T22:57:58.421781+0000","last_became_active":"2024-09-20T22:57:49.410770+0000","last_became_peered":"2024-09-20T22:57:49.410770+0000","last_unstale":"2024-09-20T22:57:58.421781+0000","last_undegraded":"2024-09-20T22:57:58.421781+0000","last_fullsized":"2024-09-20T22:57:58.421781+0000","mapping_epoch":988,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:57:58.373688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:57:58.373688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:23.927456+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"904'25","reported_seq":483,"reported_epoch":993,"state":"active+clean","last_fresh":"2024-09-20T22:57:58.370395+0000","last_change":"2024-09-20T22:57:33.088721+0000","last_active":"2024-09-20T22:57:58.370395+0000","last_peered":"2024-09-20T22:57:58.370395+0000","last_clean":"2024-09-20T22:57:58.370395+0000","last_became_active":"2024-09-20T22:54:17.349113+0000","last_became_peered":"2024-09-20T22:54:17.349113+0000","last_unstale":"2024-09-20T22:57:58.370395+0000","last_undegraded":"2024-09-20T22:57:58.370395+0000","last_fullsized":"2024-09-20T22:57:58.370395+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"904'25","last_scrub_stamp":"2024-09-20T22:57:33.088629+0000","last_deep_scrub":"904'25","last_deep_scrub_stamp":"2024-09-20T22:56:24.575150+0000","last_clean_scrub_stamp":"2024-09-20T22:57:33.088629+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:44.765537+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":28,"num_read_kb":23,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":1923,"reported_epoch":993,"state":"active+clean","last_fresh":"2024-09-20T22:57:58.421366+0000","last_change":"2024-09-20T22:57:49.396609+0000","last_active":"2024-09-20T22:57:58.421366+0000","last_peered":"2024-09-20T22:57:58.421366+0000","last_clean":"2024-09-20T22:57:58.421366+0000","last_became_active":"2024-09-20T22:57:49.396185+0000","last_became_peered":"2024-09-20T22:57:49.396185+0000","last_unstale":"2024-09-20T22:57:58.421366+0000","last_undegraded":"2024-09-20T22:57:58.421366+0000","last_fullsized":"2024-09-20T22:57:58.421366+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:55:53.487366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:55:53.487366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"733'12","reported_seq":1383,"reported_epoch":993,"state":"active+clean","last_fresh":"2024-09-20T22:57:58.370218+0000","last_change":"2024-09-20T22:55:50.416762+0000","last_active":"2024-09-20T22:57:58.370218+0000","last_peered":"2024-09-20T22:57:58.370218+0000","last_clean":"2024-09-20T22:57:58.370218+0000","last_became_active":"2024-09-20T22:47:49.227264+0000","last_became_peered":"2024-09-20T22:47:49.227264+0000","last_unstale":"2024-09-20T22:57:58.370218+0000","last_undegraded":"2024-09-20T22:57:58.370218+0000","last_fullsized":"2024-09-20T22:57:58.370218+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"733'12","last_scrub_stamp":"2024-09-20T22:55:50.416689+0000","last_deep_scrub":"733'12","last_deep_scrub_stamp":"2024-09-20T22:53:33.928499+0000","last_clean_scrub_stamp":"2024-09-20T22:55:50.416689+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":1265,"reported_epoch":993,"state":"active+clean","last_fresh":"2024-09-20T22:57:58.369226+0000","last_change":"2024-09-20T22:57:49.416890+0000","last_active":"2024-09-20T22:57:58.369226+0000","last_peered":"2024-09-20T22:57:58.369226+0000","last_clean":"2024-09-20T22:57:58.369226+0000","last_became_active":"2024-09-20T22:57:49.416550+0000","last_became_peered":"2024-09-20T22:57:49.416550+0000","last_unstale":"2024-09-20T22:57:58.369226+0000","last_undegraded":"2024-09-20T22:57:58.369226+0000","last_fullsized":"2024-09-20T22:57:58.369226+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:55:57.400645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:55:57.400645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.0","version":"992'1040","reported_seq":5602,"reported_epoch":993,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T22:57:58.420362+0000","last_change":"2024-09-20T22:57:57.553963+0000","last_active":"2024-09-20T22:57:58.420362+0000","last_peered":"2024-09-20T22:57:58.420362+0000","last_clean":"2024-09-20T22:57:58.420362+0000","last_became_active":"2024-09-20T22:57:49.412781+0000","last_became_peered":"2024-09-20T22:57:49.412781+0000","last_unstale":"2024-09-20T22:57:58.420362+0000","last_undegraded":"2024-09-20T22:57:58.420362+0000","last_fullsized":"2024-09-20T22:57:58.420362+0000","mapping_epoch":988,"log_start":"480'253","ondisk_log_start":"480'253","created":94,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"823'833","last_scrub_stamp":"2024-09-20T22:55:07.264241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:55:07.264241+0000","objects_scrubbed":25,"log_size":787,"log_dups_size":416,"ondisk_log_size":787,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":8,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:00.734123+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":5.1779121210000003,"stat_sum":{"num_bytes":26841138,"num_objects":13,"num_object_clones":12,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":1,"num_read":173,"num_read_kb":171902,"num_write":148,"num_write_kb":53734,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":64008579,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"180","length":"1"},{"start":"18d","length":"1"},{"start":"190","length":"1"},{"start":"193","length":"1"},{"start":"196","length":"2"},{"start":"19a","length":"1"},{"start":"19c","length":"2"},{"start":"1a3","length":"2"},{"start":"1a6","length":"1"},{"start":"1aa","length":"2"},{"start":"1af","length":"1"},{"start":"1b5","length":"1"}]},{"pgid":"1.6","version":"77'128","reported_seq":2249,"reported_epoch":993,"state":"active+clean","last_fresh":"2024-09-20T22:57:58.422114+0000","last_change":"2024-09-20T22:57:49.407065+0000","last_active":"2024-09-20T22:57:58.422114+0000","last_peered":"2024-09-20T22:57:58.422114+0000","last_clean":"2024-09-20T22:57:58.422114+0000","last_became_active":"2024-09-20T22:57:49.406637+0000","last_became_peered":"2024-09-20T22:57:49.406637+0000","last_unstale":"2024-09-20T22:57:58.422114+0000","last_undegraded":"2024-09-20T22:57:58.422114+0000","last_fullsized":"2024-09-20T22:57:58.422114+0000","mapping_epoch":988,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:56:12.929842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:56:12.929842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"625'37","reported_seq":1320,"reported_epoch":993,"state":"active+clean","last_fresh":"2024-09-20T22:57:58.369973+0000","last_change":"2024-09-20T22:56:48.676919+0000","last_active":"2024-09-20T22:57:58.369973+0000","last_peered":"2024-09-20T22:57:58.369973+0000","last_clean":"2024-09-20T22:57:58.369973+0000","last_became_active":"2024-09-20T22:56:08.367015+0000","last_became_peered":"2024-09-20T22:56:08.367015+0000","last_unstale":"2024-09-20T22:57:58.369973+0000","last_undegraded":"2024-09-20T22:57:58.369973+0000","last_fullsized":"2024-09-20T22:57:58.369973+0000","mapping_epoch":887,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'37","last_scrub_stamp":"2024-09-20T22:56:48.676788+0000","last_deep_scrub":"515'27","last_deep_scrub_stamp":"2024-09-20T22:49:55.964153+0000","last_clean_scrub_stamp":"2024-09-20T22:56:48.676788+0000","objects_scrubbed":8,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:06.918757+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":53,"num_read_kb":39,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":458,"reported_epoch":993,"state":"active+clean","last_fresh":"2024-09-20T22:57:58.371440+0000","last_change":"2024-09-20T22:57:35.151377+0000","last_active":"2024-09-20T22:57:58.371440+0000","last_peered":"2024-09-20T22:57:58.371440+0000","last_clean":"2024-09-20T22:57:58.371440+0000","last_became_active":"2024-09-20T22:54:17.341248+0000","last_became_peered":"2024-09-20T22:54:17.341248+0000","last_unstale":"2024-09-20T22:57:58.371440+0000","last_undegraded":"2024-09-20T22:57:58.371440+0000","last_fullsized":"2024-09-20T22:57:58.371440+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:57:35.151279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:48:23.320031+0000","last_clean_scrub_stamp":"2024-09-20T22:57:35.151279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:39.976778+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":1931,"reported_epoch":993,"state":"active+clean","last_fresh":"2024-09-20T22:57:58.420513+0000","last_change":"2024-09-20T22:55:53.886918+0000","last_active":"2024-09-20T22:57:58.420513+0000","last_peered":"2024-09-20T22:57:58.420513+0000","last_clean":"2024-09-20T22:57:58.420513+0000","last_became_active":"2024-09-20T22:47:49.214055+0000","last_became_peered":"2024-09-20T22:47:49.214055+0000","last_unstale":"2024-09-20T22:57:58.420513+0000","last_undegraded":"2024-09-20T22:57:58.420513+0000","last_fullsized":"2024-09-20T22:57:58.420513+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:55:53.886833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:55:53.886833+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":450,"reported_epoch":993,"state":"active+clean","last_fresh":"2024-09-20T22:57:58.369790+0000","last_change":"2024-09-20T22:55:54.151421+0000","last_active":"2024-09-20T22:57:58.369790+0000","last_peered":"2024-09-20T22:57:58.369790+0000","last_clean":"2024-09-20T22:57:58.369790+0000","last_became_active":"2024-09-20T22:54:17.340458+0000","last_became_peered":"2024-09-20T22:54:17.340458+0000","last_unstale":"2024-09-20T22:57:58.369790+0000","last_undegraded":"2024-09-20T22:57:58.369790+0000","last_fullsized":"2024-09-20T22:57:58.369790+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:55:54.151316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:48:41.208853+0000","last_clean_scrub_stamp":"2024-09-20T22:55:54.151316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.7","version":"989'2066","reported_seq":6551,"reported_epoch":993,"state":"active+clean","last_fresh":"2024-09-20T22:57:58.422116+0000","last_change":"2024-09-20T22:57:56.323805+0000","last_active":"2024-09-20T22:57:58.422116+0000","last_peered":"2024-09-20T22:57:58.422116+0000","last_clean":"2024-09-20T22:57:58.422116+0000","last_became_active":"2024-09-20T22:57:49.410914+0000","last_became_peered":"2024-09-20T22:57:49.410914+0000","last_unstale":"2024-09-20T22:57:58.422116+0000","last_undegraded":"2024-09-20T22:57:58.422116+0000","last_fullsized":"2024-09-20T22:57:58.422116+0000","mapping_epoch":988,"log_start":"436'456","ondisk_log_start":"436'456","created":94,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"989'2066","last_scrub_stamp":"2024-09-20T22:57:56.323701+0000","last_deep_scrub":"879'1788","last_deep_scrub_stamp":"2024-09-20T22:55:59.964830+0000","last_clean_scrub_stamp":"2024-09-20T22:57:56.323701+0000","objects_scrubbed":26,"log_size":1610,"log_dups_size":886,"ondisk_log_size":1610,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:59.666686+0000","scrub_duration":7,"objects_trimmed":1,"snaptrim_duration":0.069128024999999996,"stat_sum":{"num_bytes":62288349,"num_objects":26,"num_object_clones":23,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"180","length":"1"},{"start":"18d","length":"1"},{"start":"190","length":"1"},{"start":"193","length":"1"},{"start":"196","length":"2"},{"start":"19a","length":"1"},{"start":"19c","length":"2"},{"start":"1a3","length":"2"},{"start":"1a6","length":"1"},{"start":"1aa","length":"2"},{"start":"1af","length":"1"},{"start":"1b5","length":"1"},{"start":"1b7","length":"1"},{"start":"1ba","length":"1"},{"start":"1bd","length":"2"},{"start":"1c0","length":"2"},{"start":"1c5","length":"1"},{"start":"1c7","length":"1"}]},{"pgid":"1.1","version":"77'128","reported_seq":2231,"reported_epoch":993,"state":"active+clean","last_fresh":"2024-09-20T22:57:58.420219+0000","last_change":"2024-09-20T22:57:37.743124+0000","last_active":"2024-09-20T22:57:58.420219+0000","last_peered":"2024-09-20T22:57:58.420219+0000","last_clean":"2024-09-20T22:57:58.420219+0000","last_became_active":"2024-09-20T22:47:52.208102+0000","last_became_peered":"2024-09-20T22:47:52.208102+0000","last_unstale":"2024-09-20T22:57:58.420219+0000","last_undegraded":"2024-09-20T22:57:58.420219+0000","last_fullsized":"2024-09-20T22:57:58.420219+0000","mapping_epoch":391,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:57:37.743043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:57:37.743043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:54.685166+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"989'55","reported_seq":410,"reported_epoch":993,"state":"active+clean","last_fresh":"2024-09-20T22:57:58.420739+0000","last_change":"2024-09-20T22:57:49.427243+0000","last_active":"2024-09-20T22:57:58.420739+0000","last_peered":"2024-09-20T22:57:58.420739+0000","last_clean":"2024-09-20T22:57:58.420739+0000","last_became_active":"2024-09-20T22:57:49.426958+0000","last_became_peered":"2024-09-20T22:57:49.426958+0000","last_unstale":"2024-09-20T22:57:58.420739+0000","last_undegraded":"2024-09-20T22:57:58.420739+0000","last_fullsized":"2024-09-20T22:57:58.420739+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"833'53","last_scrub_stamp":"2024-09-20T22:56:17.472832+0000","last_deep_scrub":"86'38","last_deep_scrub_stamp":"2024-09-20T22:42:43.935677+0000","last_clean_scrub_stamp":"2024-09-20T22:56:17.472832+0000","objects_scrubbed":10,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":1074,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":379,"reported_epoch":993,"state":"active+clean","last_fresh":"2024-09-20T22:57:58.370172+0000","last_change":"2024-09-20T22:57:39.577432+0000","last_active":"2024-09-20T22:57:58.370172+0000","last_peered":"2024-09-20T22:57:58.370172+0000","last_clean":"2024-09-20T22:57:58.370172+0000","last_became_active":"2024-09-20T22:55:17.340038+0000","last_became_peered":"2024-09-20T22:55:17.340038+0000","last_unstale":"2024-09-20T22:57:58.370172+0000","last_undegraded":"2024-09-20T22:57:58.370172+0000","last_fullsized":"2024-09-20T22:57:58.370172+0000","mapping_epoch":836,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":837,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:57:39.577360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:57:39.577360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:04.367204+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"69'2","reported_seq":1955,"reported_epoch":993,"state":"active+clean","last_fresh":"2024-09-20T22:57:58.421332+0000","last_change":"2024-09-20T22:57:03.613282+0000","last_active":"2024-09-20T22:57:58.421332+0000","last_peered":"2024-09-20T22:57:58.421332+0000","last_clean":"2024-09-20T22:57:58.421332+0000","last_became_active":"2024-09-20T22:54:17.341059+0000","last_became_peered":"2024-09-20T22:54:17.341059+0000","last_unstale":"2024-09-20T22:57:58.421332+0000","last_undegraded":"2024-09-20T22:57:58.421332+0000","last_fullsized":"2024-09-20T22:57:58.421332+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2024-09-20T22:57:03.613204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:36.130684+0000","last_clean_scrub_stamp":"2024-09-20T22:57:03.613204+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:15.544078+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00087280599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":1273,"reported_epoch":993,"state":"active+clean","last_fresh":"2024-09-20T22:57:58.370068+0000","last_change":"2024-09-20T22:56:13.481362+0000","last_active":"2024-09-20T22:57:58.370068+0000","last_peered":"2024-09-20T22:57:58.370068+0000","last_clean":"2024-09-20T22:57:58.370068+0000","last_became_active":"2024-09-20T22:54:17.339860+0000","last_became_peered":"2024-09-20T22:54:17.339860+0000","last_unstale":"2024-09-20T22:57:58.370068+0000","last_undegraded":"2024-09-20T22:57:58.370068+0000","last_fullsized":"2024-09-20T22:57:58.370068+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:56:13.481292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:56:13.481292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":1804,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-20T22:57:52.719238+0000","last_change":"2024-09-20T22:57:49.407286+0000","last_active":"2024-09-20T22:57:52.719238+0000","last_peered":"2024-09-20T22:57:52.719238+0000","last_clean":"2024-09-20T22:57:52.719238+0000","last_became_active":"2024-09-20T22:57:49.398514+0000","last_became_peered":"2024-09-20T22:57:49.398514+0000","last_unstale":"2024-09-20T22:57:52.719238+0000","last_undegraded":"2024-09-20T22:57:52.719238+0000","last_fullsized":"2024-09-20T22:57:52.719238+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:55:51.494144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:55:51.494144+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:55.630609+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"988'758","reported_seq":3746,"reported_epoch":993,"state":"active+clean","last_fresh":"2024-09-20T22:57:58.369986+0000","last_change":"2024-09-20T22:57:48.440745+0000","last_active":"2024-09-20T22:57:58.369986+0000","last_peered":"2024-09-20T22:57:58.369986+0000","last_clean":"2024-09-20T22:57:58.369986+0000","last_became_active":"2024-09-20T22:56:53.366703+0000","last_became_peered":"2024-09-20T22:56:53.366703+0000","last_unstale":"2024-09-20T22:57:58.369986+0000","last_undegraded":"2024-09-20T22:57:58.369986+0000","last_fullsized":"2024-09-20T22:57:58.369986+0000","mapping_epoch":779,"log_start":"456'176","ondisk_log_start":"456'176","created":94,"last_epoch_clean":933,"parent":"0.0","parent_split_bits":0,"last_scrub":"941'718","last_scrub_stamp":"2024-09-20T22:57:06.750095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:57:06.750095+0000","objects_scrubbed":5,"log_size":582,"log_dups_size":318,"ondisk_log_size":582,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:36.737266+0000","scrub_duration":3,"objects_trimmed":1,"snaptrim_duration":0.071871422000000004,"stat_sum":{"num_bytes":12201763,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":11,"num_read_kb":7000,"num_write":8,"num_write_kb":2179,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"180","length":"1"},{"start":"18d","length":"1"},{"start":"190","length":"1"},{"start":"193","length":"1"},{"start":"196","length":"2"},{"start":"19a","length":"1"},{"start":"19c","length":"2"},{"start":"1a3","length":"2"},{"start":"1a6","length":"1"},{"start":"1aa","length":"2"},{"start":"1af","length":"1"},{"start":"1b5","length":"1"},{"start":"1b7","length":"1"},{"start":"1ba","length":"1"},{"start":"1bd","length":"2"},{"start":"1c0","length":"2"},{"start":"1c5","length":"1"},{"start":"1c7","length":"1"}]},{"pgid":"1.2","version":"77'128","reported_seq":1257,"reported_epoch":993,"state":"active+clean","last_fresh":"2024-09-20T22:57:58.369301+0000","last_change":"2024-09-20T22:56:12.520096+0000","last_active":"2024-09-20T22:57:58.369301+0000","last_peered":"2024-09-20T22:57:58.369301+0000","last_clean":"2024-09-20T22:57:58.369301+0000","last_became_active":"2024-09-20T22:56:11.355063+0000","last_became_peered":"2024-09-20T22:56:11.355063+0000","last_unstale":"2024-09-20T22:57:58.369301+0000","last_undegraded":"2024-09-20T22:57:58.369301+0000","last_fullsized":"2024-09-20T22:57:58.369301+0000","mapping_epoch":890,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":891,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:56:12.519992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:56:12.519992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"625'22","reported_seq":473,"reported_epoch":993,"state":"active+clean","last_fresh":"2024-09-20T22:57:58.371570+0000","last_change":"2024-09-20T22:57:06.349767+0000","last_active":"2024-09-20T22:57:58.371570+0000","last_peered":"2024-09-20T22:57:58.371570+0000","last_clean":"2024-09-20T22:57:58.371570+0000","last_became_active":"2024-09-20T22:54:17.349415+0000","last_became_peered":"2024-09-20T22:54:17.349415+0000","last_unstale":"2024-09-20T22:57:58.371570+0000","last_undegraded":"2024-09-20T22:57:58.371570+0000","last_fullsized":"2024-09-20T22:57:58.371570+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'22","last_scrub_stamp":"2024-09-20T22:57:06.349682+0000","last_deep_scrub":"443'18","last_deep_scrub_stamp":"2024-09-20T22:48:44.182517+0000","last_clean_scrub_stamp":"2024-09-20T22:57:06.349682+0000","objects_scrubbed":5,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:11.297930+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":24,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":1320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":458,"reported_epoch":993,"state":"active+clean","last_fresh":"2024-09-20T22:57:58.369948+0000","last_change":"2024-09-20T22:56:00.249870+0000","last_active":"2024-09-20T22:57:58.369948+0000","last_peered":"2024-09-20T22:57:58.369948+0000","last_clean":"2024-09-20T22:57:58.369948+0000","last_became_active":"2024-09-20T22:54:17.343806+0000","last_became_peered":"2024-09-20T22:54:17.343806+0000","last_unstale":"2024-09-20T22:57:58.369948+0000","last_undegraded":"2024-09-20T22:57:58.369948+0000","last_fullsized":"2024-09-20T22:57:58.369948+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:56:00.249796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:56:00.249796+0000","last_clean_scrub_stamp":"2024-09-20T22:56:00.249796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"380'2","reported_seq":1976,"reported_epoch":993,"state":"active+clean","last_fresh":"2024-09-20T22:57:58.421055+0000","last_change":"2024-09-20T22:57:49.417639+0000","last_active":"2024-09-20T22:57:58.421055+0000","last_peered":"2024-09-20T22:57:58.421055+0000","last_clean":"2024-09-20T22:57:58.421055+0000","last_became_active":"2024-09-20T22:57:49.416996+0000","last_became_peered":"2024-09-20T22:57:49.416996+0000","last_unstale":"2024-09-20T22:57:58.421055+0000","last_undegraded":"2024-09-20T22:57:58.421055+0000","last_fullsized":"2024-09-20T22:57:58.421055+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"380'2","last_scrub_stamp":"2024-09-20T22:55:57.061312+0000","last_deep_scrub":"380'2","last_deep_scrub_stamp":"2024-09-20T22:47:40.983039+0000","last_clean_scrub_stamp":"2024-09-20T22:55:57.061312+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001301459,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":1927,"reported_epoch":993,"state":"active+clean","last_fresh":"2024-09-20T22:57:58.423184+0000","last_change":"2024-09-20T22:57:38.381568+0000","last_active":"2024-09-20T22:57:58.423184+0000","last_peered":"2024-09-20T22:57:58.423184+0000","last_clean":"2024-09-20T22:57:58.423184+0000","last_became_active":"2024-09-20T22:56:08.375555+0000","last_became_peered":"2024-09-20T22:56:08.375555+0000","last_unstale":"2024-09-20T22:57:58.423184+0000","last_undegraded":"2024-09-20T22:57:58.423184+0000","last_fullsized":"2024-09-20T22:57:58.423184+0000","mapping_epoch":887,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:57:38.381457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:57:38.381457+0000","last_clean_scrub_stamp":"2024-09-20T22:57:38.381457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:05.474433+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":236,"reported_epoch":993,"state":"active+clean","last_fresh":"2024-09-20T22:57:58.421845+0000","last_change":"2024-09-20T22:57:49.404111+0000","last_active":"2024-09-20T22:57:58.421845+0000","last_peered":"2024-09-20T22:57:58.421845+0000","last_clean":"2024-09-20T22:57:58.421845+0000","last_became_active":"2024-09-20T22:57:49.403788+0000","last_became_peered":"2024-09-20T22:57:49.403788+0000","last_unstale":"2024-09-20T22:57:58.421845+0000","last_undegraded":"2024-09-20T22:57:58.421845+0000","last_fullsized":"2024-09-20T22:57:58.421845+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:56:51.218428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:56:51.218428+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.5","version":"990'592","reported_seq":5163,"reported_epoch":993,"state":"active+clean","last_fresh":"2024-09-20T22:57:58.422674+0000","last_change":"2024-09-20T22:57:48.440435+0000","last_active":"2024-09-20T22:57:58.422674+0000","last_peered":"2024-09-20T22:57:58.422674+0000","last_clean":"2024-09-20T22:57:58.422674+0000","last_became_active":"2024-09-20T22:56:53.369151+0000","last_became_peered":"2024-09-20T22:56:53.369151+0000","last_unstale":"2024-09-20T22:57:58.422674+0000","last_undegraded":"2024-09-20T22:57:58.422674+0000","last_fullsized":"2024-09-20T22:57:58.422674+0000","mapping_epoch":890,"log_start":"450'190","ondisk_log_start":"450'190","created":94,"last_epoch_clean":933,"parent":"0.0","parent_split_bits":0,"last_scrub":"941'546","last_scrub_stamp":"2024-09-20T22:57:04.121297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:57:04.121297+0000","objects_scrubbed":7,"log_size":402,"log_dups_size":240,"ondisk_log_size":402,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:30.793975+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.059506083000000001,"stat_sum":{"num_bytes":15343641,"num_objects":8,"num_object_clones":7,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":4,"num_write_kb":4269,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"180","length":"1"},{"start":"18d","length":"1"},{"start":"190","length":"1"},{"start":"193","length":"1"},{"start":"196","length":"2"},{"start":"19a","length":"1"},{"start":"19c","length":"2"},{"start":"1a3","length":"2"},{"start":"1a6","length":"1"},{"start":"1aa","length":"2"},{"start":"1af","length":"1"},{"start":"1b5","length":"1"},{"start":"1b7","length":"1"},{"start":"1ba","length":"1"},{"start":"1bd","length":"2"},{"start":"1c0","length":"2"},{"start":"1c5","length":"1"},{"start":"1c7","length":"1"}]},{"pgid":"1.3","version":"77'128","reported_seq":1477,"reported_epoch":993,"state":"active+clean","last_fresh":"2024-09-20T22:57:58.370568+0000","last_change":"2024-09-20T22:56:21.532248+0000","last_active":"2024-09-20T22:57:58.370568+0000","last_peered":"2024-09-20T22:57:58.370568+0000","last_clean":"2024-09-20T22:57:58.370568+0000","last_became_active":"2024-09-20T22:54:20.413191+0000","last_became_peered":"2024-09-20T22:54:20.413191+0000","last_unstale":"2024-09-20T22:57:58.370568+0000","last_undegraded":"2024-09-20T22:57:58.370568+0000","last_fullsized":"2024-09-20T22:57:58.370568+0000","mapping_epoch":779,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":780,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:56:21.532173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:56:21.532173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"828'30","reported_seq":1353,"reported_epoch":993,"state":"active+clean","last_fresh":"2024-09-20T22:57:58.371719+0000","last_change":"2024-09-20T22:57:09.738754+0000","last_active":"2024-09-20T22:57:58.371719+0000","last_peered":"2024-09-20T22:57:58.371719+0000","last_clean":"2024-09-20T22:57:58.371719+0000","last_became_active":"2024-09-20T22:55:11.346018+0000","last_became_peered":"2024-09-20T22:55:11.346018+0000","last_unstale":"2024-09-20T22:57:58.371719+0000","last_undegraded":"2024-09-20T22:57:58.371719+0000","last_fullsized":"2024-09-20T22:57:58.371719+0000","mapping_epoch":830,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":831,"parent":"0.0","parent_split_bits":0,"last_scrub":"828'30","last_scrub_stamp":"2024-09-20T22:57:09.738669+0000","last_deep_scrub":"828'30","last_deep_scrub_stamp":"2024-09-20T22:55:08.705571+0000","last_clean_scrub_stamp":"2024-09-20T22:57:09.738669+0000","objects_scrubbed":6,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:37.471839+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":1270,"reported_epoch":993,"state":"active+clean","last_fresh":"2024-09-20T22:57:58.369392+0000","last_change":"2024-09-20T22:57:49.411179+0000","last_active":"2024-09-20T22:57:58.369392+0000","last_peered":"2024-09-20T22:57:58.369392+0000","last_clean":"2024-09-20T22:57:58.369392+0000","last_became_active":"2024-09-20T22:57:49.410539+0000","last_became_peered":"2024-09-20T22:57:49.410539+0000","last_unstale":"2024-09-20T22:57:58.369392+0000","last_undegraded":"2024-09-20T22:57:58.369392+0000","last_fullsized":"2024-09-20T22:57:58.369392+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:55:25.640520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:44:19.834641+0000","last_clean_scrub_stamp":"2024-09-20T22:55:25.640520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"75'5","reported_seq":2610,"reported_epoch":993,"state":"active+clean","last_fresh":"2024-09-20T22:57:58.369816+0000","last_change":"2024-09-20T22:57:49.402647+0000","last_active":"2024-09-20T22:57:58.369816+0000","last_peered":"2024-09-20T22:57:58.369816+0000","last_clean":"2024-09-20T22:57:58.369816+0000","last_became_active":"2024-09-20T22:57:49.402122+0000","last_became_peered":"2024-09-20T22:57:49.402122+0000","last_unstale":"2024-09-20T22:57:58.369816+0000","last_undegraded":"2024-09-20T22:57:58.369816+0000","last_fullsized":"2024-09-20T22:57:58.369816+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-09-20T22:56:12.471215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:36.130684+0000","last_clean_scrub_stamp":"2024-09-20T22:56:12.471215+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00078631700000000003,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":855,"num_read_kb":850,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1556,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":1938,"reported_epoch":993,"state":"active+clean","last_fresh":"2024-09-20T22:57:58.422801+0000","last_change":"2024-09-20T22:57:13.660607+0000","last_active":"2024-09-20T22:57:58.422801+0000","last_peered":"2024-09-20T22:57:58.422801+0000","last_clean":"2024-09-20T22:57:58.422801+0000","last_became_active":"2024-09-20T22:54:17.339805+0000","last_became_peered":"2024-09-20T22:54:17.339805+0000","last_unstale":"2024-09-20T22:57:58.422801+0000","last_undegraded":"2024-09-20T22:57:58.422801+0000","last_fullsized":"2024-09-20T22:57:58.422801+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:57:13.660520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:57:13.660520+0000","last_clean_scrub_stamp":"2024-09-20T22:57:13.660520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:37.166154+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":467,"reported_epoch":993,"state":"active+clean","last_fresh":"2024-09-20T22:57:58.371240+0000","last_change":"2024-09-20T22:57:57.433001+0000","last_active":"2024-09-20T22:57:58.371240+0000","last_peered":"2024-09-20T22:57:58.371240+0000","last_clean":"2024-09-20T22:57:58.371240+0000","last_became_active":"2024-09-20T22:57:49.429412+0000","last_became_peered":"2024-09-20T22:57:49.429412+0000","last_unstale":"2024-09-20T22:57:58.371240+0000","last_undegraded":"2024-09-20T22:57:58.371240+0000","last_fullsized":"2024-09-20T22:57:58.371240+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:57:57.432865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:57:57.432865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:13.282483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.6","version":"989'890","reported_seq":5379,"reported_epoch":993,"state":"active+clean","last_fresh":"2024-09-20T22:57:58.371217+0000","last_change":"2024-09-20T22:57:55.700976+0000","last_active":"2024-09-20T22:57:58.371217+0000","last_peered":"2024-09-20T22:57:58.371217+0000","last_clean":"2024-09-20T22:57:58.371217+0000","last_became_active":"2024-09-20T22:57:49.412758+0000","last_became_peered":"2024-09-20T22:57:49.412758+0000","last_unstale":"2024-09-20T22:57:58.371217+0000","last_undegraded":"2024-09-20T22:57:58.371217+0000","last_fullsized":"2024-09-20T22:57:58.371217+0000","mapping_epoch":988,"log_start":"443'181","ondisk_log_start":"443'181","created":94,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"989'890","last_scrub_stamp":"2024-09-20T22:57:55.700872+0000","last_deep_scrub":"902'811","last_deep_scrub_stamp":"2024-09-20T22:56:22.658684+0000","last_clean_scrub_stamp":"2024-09-20T22:57:55.700872+0000","objects_scrubbed":9,"log_size":709,"log_dups_size":314,"ondisk_log_size":709,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:17.908725+0000","scrub_duration":3,"objects_trimmed":2,"snaptrim_duration":0.046809684999999997,"stat_sum":{"num_bytes":20144663,"num_objects":9,"num_object_clones":8,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"180","length":"1"},{"start":"18d","length":"1"},{"start":"190","length":"1"},{"start":"193","length":"1"},{"start":"196","length":"2"},{"start":"19a","length":"1"},{"start":"19c","length":"2"},{"start":"1a3","length":"2"},{"start":"1a6","length":"1"},{"start":"1aa","length":"2"},{"start":"1af","length":"1"},{"start":"1b5","length":"1"},{"start":"1b7","length":"1"},{"start":"1ba","length":"1"},{"start":"1bd","length":"2"},{"start":"1c0","length":"2"},{"start":"1c5","length":"1"},{"start":"1c7","length":"1"}]},{"pgid":"4.5","version":"888'33","reported_seq":2280,"reported_epoch":993,"state":"active+clean","last_fresh":"2024-09-20T22:57:58.421945+0000","last_change":"2024-09-20T22:56:21.986738+0000","last_active":"2024-09-20T22:57:58.421945+0000","last_peered":"2024-09-20T22:57:58.421945+0000","last_clean":"2024-09-20T22:57:58.421945+0000","last_became_active":"2024-09-20T22:56:08.358532+0000","last_became_peered":"2024-09-20T22:56:08.358532+0000","last_unstale":"2024-09-20T22:57:58.421945+0000","last_undegraded":"2024-09-20T22:57:58.421945+0000","last_fullsized":"2024-09-20T22:57:58.421945+0000","mapping_epoch":887,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":0,"last_scrub":"888'33","last_scrub_stamp":"2024-09-20T22:56:21.986633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:56:21.986633+0000","objects_scrubbed":8,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":770,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"67'1","reported_seq":1809,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-20T22:57:52.719037+0000","last_change":"2024-09-20T22:57:49.401711+0000","last_active":"2024-09-20T22:57:52.719037+0000","last_peered":"2024-09-20T22:57:52.719037+0000","last_clean":"2024-09-20T22:57:52.719037+0000","last_became_active":"2024-09-20T22:57:49.397191+0000","last_became_peered":"2024-09-20T22:57:49.397191+0000","last_unstale":"2024-09-20T22:57:52.719037+0000","last_undegraded":"2024-09-20T22:57:52.719037+0000","last_fullsized":"2024-09-20T22:57:52.719037+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-09-20T22:55:33.544218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:55:33.544218+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:55.630539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"77'128","reported_seq":1256,"reported_epoch":993,"state":"active+clean","last_fresh":"2024-09-20T22:57:58.369117+0000","last_change":"2024-09-20T22:56:43.780891+0000","last_active":"2024-09-20T22:57:58.369117+0000","last_peered":"2024-09-20T22:57:58.369117+0000","last_clean":"2024-09-20T22:57:58.369117+0000","last_became_active":"2024-09-20T22:56:41.368240+0000","last_became_peered":"2024-09-20T22:56:41.368240+0000","last_unstale":"2024-09-20T22:57:58.369117+0000","last_undegraded":"2024-09-20T22:57:58.369117+0000","last_fullsized":"2024-09-20T22:57:58.369117+0000","mapping_epoch":920,"log_start":"77'128","ondisk_log_start":"77'128","created":19,"last_epoch_clean":921,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:56:43.780781+0000","last_deep_scrub":"77'128","last_deep_scrub_stamp":"2024-09-20T22:47:22.417099+0000","last_clean_scrub_stamp":"2024-09-20T22:56:43.780781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":222,"reported_epoch":993,"state":"active+clean","last_fresh":"2024-09-20T22:57:58.420891+0000","last_change":"2024-09-20T22:56:08.366982+0000","last_active":"2024-09-20T22:57:58.420891+0000","last_peered":"2024-09-20T22:57:58.420891+0000","last_clean":"2024-09-20T22:57:58.420891+0000","last_became_active":"2024-09-20T22:56:08.366671+0000","last_became_peered":"2024-09-20T22:56:08.366671+0000","last_unstale":"2024-09-20T22:57:58.420891+0000","last_undegraded":"2024-09-20T22:57:58.420891+0000","last_fullsized":"2024-09-20T22:57:58.420891+0000","mapping_epoch":887,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:55:10.707805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:55:10.707805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":1923,"reported_epoch":993,"state":"active+clean","last_fresh":"2024-09-20T22:57:58.422276+0000","last_change":"2024-09-20T22:57:49.410905+0000","last_active":"2024-09-20T22:57:58.422276+0000","last_peered":"2024-09-20T22:57:58.422276+0000","last_clean":"2024-09-20T22:57:58.422276+0000","last_became_active":"2024-09-20T22:57:49.410488+0000","last_became_peered":"2024-09-20T22:57:49.410488+0000","last_unstale":"2024-09-20T22:57:58.422276+0000","last_undegraded":"2024-09-20T22:57:58.422276+0000","last_fullsized":"2024-09-20T22:57:58.422276+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:55:50.483056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:55:50.483056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"988'954","reported_seq":5586,"reported_epoch":993,"state":"active+clean","last_fresh":"2024-09-20T22:57:58.421535+0000","last_change":"2024-09-20T22:57:48.439778+0000","last_active":"2024-09-20T22:57:58.421535+0000","last_peered":"2024-09-20T22:57:58.421535+0000","last_clean":"2024-09-20T22:57:58.421535+0000","last_became_active":"2024-09-20T22:54:20.414528+0000","last_became_peered":"2024-09-20T22:54:20.414528+0000","last_unstale":"2024-09-20T22:57:58.421535+0000","last_undegraded":"2024-09-20T22:57:58.421535+0000","last_fullsized":"2024-09-20T22:57:58.421535+0000","mapping_epoch":779,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":780,"parent":"0.0","parent_split_bits":0,"last_scrub":"970'914","last_scrub_stamp":"2024-09-20T22:57:31.284226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:57:31.284226+0000","objects_scrubbed":16,"log_size":954,"log_dups_size":0,"ondisk_log_size":954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:37.068423+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.064640501000000003,"stat_sum":{"num_bytes":34316407,"num_objects":14,"num_object_clones":12,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":291,"num_read_kb":291495,"num_write":147,"num_write_kb":69585,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":88,"num_bytes_recovered":51113242,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"180","length":"1"},{"start":"18d","length":"1"},{"start":"190","length":"1"},{"start":"193","length":"1"},{"start":"196","length":"2"},{"start":"19a","length":"1"},{"start":"19c","length":"2"},{"start":"1a3","length":"2"},{"start":"1a6","length":"1"},{"start":"1aa","length":"2"},{"start":"1af","length":"1"},{"start":"1b5","length":"1"},{"start":"1b7","length":"1"},{"start":"1ba","length":"1"},{"start":"1bd","length":"2"},{"start":"1c0","length":"2"},{"start":"1c5","length":"1"},{"start":"1c7","length":"1"}]},{"pgid":"4.e","version":"850'24","reported_seq":842,"reported_epoch":993,"state":"active+clean","last_fresh":"2024-09-20T22:57:58.422166+0000","last_change":"2024-09-20T22:57:38.226530+0000","last_active":"2024-09-20T22:57:58.422166+0000","last_peered":"2024-09-20T22:57:58.422166+0000","last_clean":"2024-09-20T22:57:58.422166+0000","last_became_active":"2024-09-20T22:55:17.342390+0000","last_became_peered":"2024-09-20T22:55:17.342390+0000","last_unstale":"2024-09-20T22:57:58.422166+0000","last_undegraded":"2024-09-20T22:57:58.422166+0000","last_fullsized":"2024-09-20T22:57:58.422166+0000","mapping_epoch":836,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":837,"parent":"0.0","parent_split_bits":0,"last_scrub":"850'24","last_scrub_stamp":"2024-09-20T22:57:38.226418+0000","last_deep_scrub":"850'24","last_deep_scrub_stamp":"2024-09-20T22:55:30.779018+0000","last_clean_scrub_stamp":"2024-09-20T22:57:38.226418+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:49.332407+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":23,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":856,"reported_epoch":993,"state":"active+clean","last_fresh":"2024-09-20T22:57:58.419862+0000","last_change":"2024-09-20T22:56:17.135450+0000","last_active":"2024-09-20T22:57:58.419862+0000","last_peered":"2024-09-20T22:57:58.419862+0000","last_clean":"2024-09-20T22:57:58.419862+0000","last_became_active":"2024-09-20T22:56:08.359647+0000","last_became_peered":"2024-09-20T22:56:08.359647+0000","last_unstale":"2024-09-20T22:57:58.419862+0000","last_undegraded":"2024-09-20T22:57:58.419862+0000","last_fullsized":"2024-09-20T22:57:58.419862+0000","mapping_epoch":887,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:56:17.135379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:56:17.135379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":1918,"reported_epoch":993,"state":"active+clean","last_fresh":"2024-09-20T22:57:58.422188+0000","last_change":"2024-09-20T22:57:11.628063+0000","last_active":"2024-09-20T22:57:58.422188+0000","last_peered":"2024-09-20T22:57:58.422188+0000","last_clean":"2024-09-20T22:57:58.422188+0000","last_became_active":"2024-09-20T22:48:05.233778+0000","last_became_peered":"2024-09-20T22:48:05.233778+0000","last_unstale":"2024-09-20T22:57:58.422188+0000","last_undegraded":"2024-09-20T22:57:58.422188+0000","last_fullsized":"2024-09-20T22:57:58.422188+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:57:11.627990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:54:47.166836+0000","last_clean_scrub_stamp":"2024-09-20T22:57:11.627990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:34.275838+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":1778,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-20T22:57:52.719009+0000","last_change":"2024-09-20T22:57:49.427590+0000","last_active":"2024-09-20T22:57:52.719009+0000","last_peered":"2024-09-20T22:57:52.719009+0000","last_clean":"2024-09-20T22:57:52.719009+0000","last_became_active":"2024-09-20T22:57:49.425373+0000","last_became_peered":"2024-09-20T22:57:49.425373+0000","last_unstale":"2024-09-20T22:57:52.719009+0000","last_undegraded":"2024-09-20T22:57:52.719009+0000","last_fullsized":"2024-09-20T22:57:52.719009+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:56:46.453177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:48:40.127123+0000","last_clean_scrub_stamp":"2024-09-20T22:56:46.453177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:56.360115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.e","version":"988'1072","reported_seq":5558,"reported_epoch":993,"state":"active+clean","last_fresh":"2024-09-20T22:57:58.423195+0000","last_change":"2024-09-20T22:57:48.441312+0000","last_active":"2024-09-20T22:57:58.423195+0000","last_peered":"2024-09-20T22:57:58.423195+0000","last_clean":"2024-09-20T22:57:58.423195+0000","last_became_active":"2024-09-20T22:47:52.209452+0000","last_became_peered":"2024-09-20T22:47:52.209452+0000","last_unstale":"2024-09-20T22:57:58.423195+0000","last_undegraded":"2024-09-20T22:57:58.423195+0000","last_fullsized":"2024-09-20T22:57:58.423195+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"944'1032","last_scrub_stamp":"2024-09-20T22:57:05.131799+0000","last_deep_scrub":"174'60","last_deep_scrub_stamp":"2024-09-20T22:44:18.301776+0000","last_clean_scrub_stamp":"2024-09-20T22:57:05.131799+0000","objects_scrubbed":14,"log_size":1072,"log_dups_size":0,"ondisk_log_size":1072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:28.855536+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.057156603,"stat_sum":{"num_bytes":32496050,"num_objects":14,"num_object_clones":12,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":2,"num_read":306,"num_read_kb":300627,"num_write":228,"num_write_kb":83627,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":23110741,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"180","length":"1"},{"start":"18d","length":"1"},{"start":"190","length":"1"},{"start":"193","length":"1"},{"start":"196","length":"2"},{"start":"19a","length":"1"},{"start":"19c","length":"2"},{"start":"1a3","length":"2"},{"start":"1a6","length":"1"},{"start":"1aa","length":"2"},{"start":"1af","length":"1"},{"start":"1b5","length":"1"},{"start":"1b7","length":"1"},{"start":"1ba","length":"1"},{"start":"1bd","length":"2"},{"start":"1c0","length":"2"},{"start":"1c5","length":"1"},{"start":"1c7","length":"1"}]},{"pgid":"1.8","version":"77'128","reported_seq":2184,"reported_epoch":993,"state":"active+clean","last_fresh":"2024-09-20T22:57:58.420318+0000","last_change":"2024-09-20T22:57:49.409192+0000","last_active":"2024-09-20T22:57:58.420318+0000","last_peered":"2024-09-20T22:57:58.420318+0000","last_clean":"2024-09-20T22:57:58.420318+0000","last_became_active":"2024-09-20T22:57:49.408874+0000","last_became_peered":"2024-09-20T22:57:49.408874+0000","last_unstale":"2024-09-20T22:57:58.420318+0000","last_undegraded":"2024-09-20T22:57:58.420318+0000","last_fullsized":"2024-09-20T22:57:58.420318+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:54:46.127943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:54:46.127943+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"625'31","reported_seq":869,"reported_epoch":993,"state":"active+clean","last_fresh":"2024-09-20T22:57:58.423334+0000","last_change":"2024-09-20T22:57:36.368878+0000","last_active":"2024-09-20T22:57:58.423334+0000","last_peered":"2024-09-20T22:57:58.423334+0000","last_clean":"2024-09-20T22:57:58.423334+0000","last_became_active":"2024-09-20T22:55:17.342262+0000","last_became_peered":"2024-09-20T22:55:17.342262+0000","last_unstale":"2024-09-20T22:57:58.423334+0000","last_undegraded":"2024-09-20T22:57:58.423334+0000","last_fullsized":"2024-09-20T22:57:58.423334+0000","mapping_epoch":836,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":837,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'31","last_scrub_stamp":"2024-09-20T22:57:36.368756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:57:36.368756+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:53.304747+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":48,"num_read_kb":38,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":1482,"reported_epoch":993,"state":"active+clean","last_fresh":"2024-09-20T22:57:58.419593+0000","last_change":"2024-09-20T22:56:51.152802+0000","last_active":"2024-09-20T22:57:58.419593+0000","last_peered":"2024-09-20T22:57:58.419593+0000","last_clean":"2024-09-20T22:57:58.419593+0000","last_became_active":"2024-09-20T22:54:17.332636+0000","last_became_peered":"2024-09-20T22:54:17.332636+0000","last_unstale":"2024-09-20T22:57:58.419593+0000","last_undegraded":"2024-09-20T22:57:58.419593+0000","last_fullsized":"2024-09-20T22:57:58.419593+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:56:51.152732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:35.492562+0000","last_clean_scrub_stamp":"2024-09-20T22:56:51.152732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:18.450632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":458,"reported_epoch":993,"state":"active+clean","last_fresh":"2024-09-20T22:57:58.371337+0000","last_change":"2024-09-20T22:56:49.666908+0000","last_active":"2024-09-20T22:57:58.371337+0000","last_peered":"2024-09-20T22:57:58.371337+0000","last_clean":"2024-09-20T22:57:58.371337+0000","last_became_active":"2024-09-20T22:54:17.349144+0000","last_became_peered":"2024-09-20T22:54:17.349144+0000","last_unstale":"2024-09-20T22:57:58.371337+0000","last_undegraded":"2024-09-20T22:57:58.371337+0000","last_fullsized":"2024-09-20T22:57:58.371337+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:56:49.666824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:56:49.666824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:15.812144+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":1946,"reported_epoch":993,"state":"active+clean","last_fresh":"2024-09-20T22:57:58.423446+0000","last_change":"2024-09-20T22:57:49.411075+0000","last_active":"2024-09-20T22:57:58.423446+0000","last_peered":"2024-09-20T22:57:58.423446+0000","last_clean":"2024-09-20T22:57:58.423446+0000","last_became_active":"2024-09-20T22:57:49.410743+0000","last_became_peered":"2024-09-20T22:57:49.410743+0000","last_unstale":"2024-09-20T22:57:58.423446+0000","last_undegraded":"2024-09-20T22:57:58.423446+0000","last_fullsized":"2024-09-20T22:57:58.423446+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:56:43.023211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:56:43.023211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"989'1180","reported_seq":5811,"reported_epoch":993,"state":"active+clean","last_fresh":"2024-09-20T22:57:58.372269+0000","last_change":"2024-09-20T22:57:49.680248+0000","last_active":"2024-09-20T22:57:58.372269+0000","last_peered":"2024-09-20T22:57:58.372269+0000","last_clean":"2024-09-20T22:57:58.372269+0000","last_became_active":"2024-09-20T22:57:49.415940+0000","last_became_peered":"2024-09-20T22:57:49.415940+0000","last_unstale":"2024-09-20T22:57:58.372269+0000","last_undegraded":"2024-09-20T22:57:58.372269+0000","last_fullsized":"2024-09-20T22:57:58.372269+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"868'991","last_scrub_stamp":"2024-09-20T22:55:49.469629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:55:49.469629+0000","objects_scrubbed":15,"log_size":1180,"log_dups_size":0,"ondisk_log_size":1180,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":1,"snaptrim_duration":0.058896384000000003,"stat_sum":{"num_bytes":41006889,"num_objects":19,"num_object_clones":16,"num_object_copies":57,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":483,"num_read_kb":511223,"num_write":268,"num_write_kb":92130,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":121,"num_bytes_recovered":52177428,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"180","length":"1"},{"start":"18d","length":"1"},{"start":"190","length":"1"},{"start":"193","length":"1"},{"start":"196","length":"2"},{"start":"19a","length":"1"},{"start":"19c","length":"2"},{"start":"1a3","length":"2"},{"start":"1a6","length":"1"},{"start":"1aa","length":"2"},{"start":"1af","length":"1"},{"start":"1b5","length":"1"},{"start":"1b7","length":"1"},{"start":"1ba","length":"1"},{"start":"1bd","length":"2"},{"start":"1c0","length":"2"},{"start":"1c5","length":"1"},{"start":"1c7","length":"1"}]},{"pgid":"1.9","version":"77'128","reported_seq":455,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-20T22:57:52.717955+0000","last_change":"2024-09-20T22:57:49.418054+0000","last_active":"2024-09-20T22:57:52.717955+0000","last_peered":"2024-09-20T22:57:52.717955+0000","last_clean":"2024-09-20T22:57:52.717955+0000","last_became_active":"2024-09-20T22:57:49.415117+0000","last_became_peered":"2024-09-20T22:57:49.415117+0000","last_unstale":"2024-09-20T22:57:52.717955+0000","last_undegraded":"2024-09-20T22:57:52.717955+0000","last_fullsized":"2024-09-20T22:57:52.717955+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:56:11.417116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:56:11.417116+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:55.630853+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"877'25","reported_seq":692,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-20T22:57:52.718759+0000","last_change":"2024-09-20T22:57:49.426619+0000","last_active":"2024-09-20T22:57:52.718759+0000","last_peered":"2024-09-20T22:57:52.718759+0000","last_clean":"2024-09-20T22:57:52.718759+0000","last_became_active":"2024-09-20T22:57:49.424440+0000","last_became_peered":"2024-09-20T22:57:49.424440+0000","last_unstale":"2024-09-20T22:57:52.718759+0000","last_undegraded":"2024-09-20T22:57:52.718759+0000","last_fullsized":"2024-09-20T22:57:52.718759+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"877'25","last_scrub_stamp":"2024-09-20T22:55:57.521557+0000","last_deep_scrub":"877'25","last_deep_scrub_stamp":"2024-09-20T22:55:57.521557+0000","last_clean_scrub_stamp":"2024-09-20T22:55:57.521557+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:55.630711+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":28,"num_read_kb":23,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":1802,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-20T22:57:52.718542+0000","last_change":"2024-09-20T22:57:49.433012+0000","last_active":"2024-09-20T22:57:52.718542+0000","last_peered":"2024-09-20T22:57:52.718542+0000","last_clean":"2024-09-20T22:57:52.718542+0000","last_became_active":"2024-09-20T22:57:49.432735+0000","last_became_peered":"2024-09-20T22:57:49.432735+0000","last_unstale":"2024-09-20T22:57:52.718542+0000","last_undegraded":"2024-09-20T22:57:52.718542+0000","last_fullsized":"2024-09-20T22:57:52.718542+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:55:58.553666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:55:58.553666+0000","last_clean_scrub_stamp":"2024-09-20T22:55:58.553666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:55.630641+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"888'10","reported_seq":2137,"reported_epoch":993,"state":"active+clean","last_fresh":"2024-09-20T22:57:58.422840+0000","last_change":"2024-09-20T22:56:08.398936+0000","last_active":"2024-09-20T22:57:58.422840+0000","last_peered":"2024-09-20T22:57:58.422840+0000","last_clean":"2024-09-20T22:57:58.422840+0000","last_became_active":"2024-09-20T22:56:08.374292+0000","last_became_peered":"2024-09-20T22:56:08.374292+0000","last_unstale":"2024-09-20T22:57:58.422840+0000","last_undegraded":"2024-09-20T22:57:58.422840+0000","last_fullsized":"2024-09-20T22:57:58.422840+0000","mapping_epoch":887,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":0,"last_scrub":"261'9","last_scrub_stamp":"2024-09-20T22:55:36.178528+0000","last_deep_scrub":"261'9","last_deep_scrub_stamp":"2024-09-20T22:46:50.974735+0000","last_clean_scrub_stamp":"2024-09-20T22:55:36.178528+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":464,"reported_epoch":993,"state":"active+clean","last_fresh":"2024-09-20T22:57:58.370362+0000","last_change":"2024-09-20T22:56:23.542870+0000","last_active":"2024-09-20T22:57:58.370362+0000","last_peered":"2024-09-20T22:57:58.370362+0000","last_clean":"2024-09-20T22:57:58.370362+0000","last_became_active":"2024-09-20T22:56:01.477943+0000","last_became_peered":"2024-09-20T22:56:01.477943+0000","last_unstale":"2024-09-20T22:57:58.370362+0000","last_undegraded":"2024-09-20T22:57:58.370362+0000","last_fullsized":"2024-09-20T22:57:58.370362+0000","mapping_epoch":880,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":881,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:56:23.542803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:56:23.542803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.8","version":"989'1442","reported_seq":5801,"reported_epoch":993,"state":"active+clean","last_fresh":"2024-09-20T22:57:58.419457+0000","last_change":"2024-09-20T22:57:57.346215+0000","last_active":"2024-09-20T22:57:58.419457+0000","last_peered":"2024-09-20T22:57:58.419457+0000","last_clean":"2024-09-20T22:57:58.419457+0000","last_became_active":"2024-09-20T22:57:49.415389+0000","last_became_peered":"2024-09-20T22:57:49.415389+0000","last_unstale":"2024-09-20T22:57:58.419457+0000","last_undegraded":"2024-09-20T22:57:58.419457+0000","last_fullsized":"2024-09-20T22:57:58.419457+0000","mapping_epoch":988,"log_start":"426'268","ondisk_log_start":"426'268","created":94,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"989'1442","last_scrub_stamp":"2024-09-20T22:57:57.346049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:57:57.346049+0000","objects_scrubbed":9,"log_size":1174,"log_dups_size":445,"ondisk_log_size":1174,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:02.128887+0000","scrub_duration":5,"objects_trimmed":1,"snaptrim_duration":0.052166095000000003,"stat_sum":{"num_bytes":18691964,"num_objects":9,"num_object_clones":8,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"180","length":"1"},{"start":"18d","length":"1"},{"start":"190","length":"1"},{"start":"193","length":"1"},{"start":"196","length":"2"},{"start":"19a","length":"1"},{"start":"19c","length":"2"},{"start":"1a3","length":"2"},{"start":"1a6","length":"1"},{"start":"1aa","length":"2"},{"start":"1af","length":"1"},{"start":"1b5","length":"1"},{"start":"1b7","length":"1"},{"start":"1ba","length":"1"},{"start":"1bd","length":"2"},{"start":"1c0","length":"2"},{"start":"1c5","length":"1"},{"start":"1c7","length":"1"}]},{"pgid":"4.b","version":"625'15","reported_seq":1322,"reported_epoch":993,"state":"active+clean","last_fresh":"2024-09-20T22:57:58.370095+0000","last_change":"2024-09-20T22:55:32.601025+0000","last_active":"2024-09-20T22:57:58.370095+0000","last_peered":"2024-09-20T22:57:58.370095+0000","last_clean":"2024-09-20T22:57:58.370095+0000","last_became_active":"2024-09-20T22:54:17.340998+0000","last_became_peered":"2024-09-20T22:54:17.340998+0000","last_unstale":"2024-09-20T22:57:58.370095+0000","last_undegraded":"2024-09-20T22:57:58.370095+0000","last_fullsized":"2024-09-20T22:57:58.370095+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'15","last_scrub_stamp":"2024-09-20T22:55:32.600891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:55:32.600891+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":1923,"reported_epoch":993,"state":"active+clean","last_fresh":"2024-09-20T22:57:58.421387+0000","last_change":"2024-09-20T22:57:49.425922+0000","last_active":"2024-09-20T22:57:58.421387+0000","last_peered":"2024-09-20T22:57:58.421387+0000","last_clean":"2024-09-20T22:57:58.421387+0000","last_became_active":"2024-09-20T22:57:49.425707+0000","last_became_peered":"2024-09-20T22:57:49.425707+0000","last_unstale":"2024-09-20T22:57:58.421387+0000","last_undegraded":"2024-09-20T22:57:58.421387+0000","last_fullsized":"2024-09-20T22:57:58.421387+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:56:24.518228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:56:24.518228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":1954,"reported_epoch":993,"state":"active+clean","last_fresh":"2024-09-20T22:57:58.419793+0000","last_change":"2024-09-20T22:57:55.320989+0000","last_active":"2024-09-20T22:57:58.419793+0000","last_peered":"2024-09-20T22:57:58.419793+0000","last_clean":"2024-09-20T22:57:58.419793+0000","last_became_active":"2024-09-20T22:57:49.411210+0000","last_became_peered":"2024-09-20T22:57:49.411210+0000","last_unstale":"2024-09-20T22:57:58.419793+0000","last_undegraded":"2024-09-20T22:57:58.419793+0000","last_fullsized":"2024-09-20T22:57:58.419793+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:57:55.320885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:57:55.320885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:55.718488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":1270,"reported_epoch":993,"state":"active+clean","last_fresh":"2024-09-20T22:57:58.369479+0000","last_change":"2024-09-20T22:57:49.411393+0000","last_active":"2024-09-20T22:57:58.369479+0000","last_peered":"2024-09-20T22:57:58.369479+0000","last_clean":"2024-09-20T22:57:58.369479+0000","last_became_active":"2024-09-20T22:57:49.410896+0000","last_became_peered":"2024-09-20T22:57:49.410896+0000","last_unstale":"2024-09-20T22:57:58.369479+0000","last_undegraded":"2024-09-20T22:57:58.369479+0000","last_fullsized":"2024-09-20T22:57:58.369479+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:56:44.736139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:56:44.736139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"988'1928","reported_seq":6483,"reported_epoch":993,"state":"active+clean","last_fresh":"2024-09-20T22:57:58.369642+0000","last_change":"2024-09-20T22:57:48.442076+0000","last_active":"2024-09-20T22:57:58.369642+0000","last_peered":"2024-09-20T22:57:58.369642+0000","last_clean":"2024-09-20T22:57:58.369642+0000","last_became_active":"2024-09-20T22:56:53.367443+0000","last_became_peered":"2024-09-20T22:56:53.367443+0000","last_unstale":"2024-09-20T22:57:58.369642+0000","last_undegraded":"2024-09-20T22:57:58.369642+0000","last_fullsized":"2024-09-20T22:57:58.369642+0000","mapping_epoch":779,"log_start":"423'280","ondisk_log_start":"423'280","created":94,"last_epoch_clean":933,"parent":"0.0","parent_split_bits":0,"last_scrub":"944'1882","last_scrub_stamp":"2024-09-20T22:57:05.737732+0000","last_deep_scrub":"590'828","last_deep_scrub_stamp":"2024-09-20T22:51:10.909364+0000","last_clean_scrub_stamp":"2024-09-20T22:57:05.737732+0000","objects_scrubbed":14,"log_size":1648,"log_dups_size":649,"ondisk_log_size":1648,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:35.520582+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.064944348999999998,"stat_sum":{"num_bytes":31503774,"num_objects":11,"num_object_clones":9,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":1,"num_read":18,"num_read_kb":22008,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"180","length":"1"},{"start":"18d","length":"1"},{"start":"190","length":"1"},{"start":"193","length":"1"},{"start":"196","length":"2"},{"start":"19a","length":"1"},{"start":"19c","length":"2"},{"start":"1a3","length":"2"},{"start":"1a6","length":"1"},{"start":"1aa","length":"2"},{"start":"1af","length":"1"},{"start":"1b5","length":"1"},{"start":"1b7","length":"1"},{"start":"1ba","length":"1"},{"start":"1bd","length":"2"},{"start":"1c0","length":"2"},{"start":"1c5","length":"1"},{"start":"1c7","length":"1"}]},{"pgid":"4.a","version":"625'44","reported_seq":2073,"reported_epoch":993,"state":"active+clean","last_fresh":"2024-09-20T22:57:58.420675+0000","last_change":"2024-09-20T22:57:06.158084+0000","last_active":"2024-09-20T22:57:58.420675+0000","last_peered":"2024-09-20T22:57:58.420675+0000","last_clean":"2024-09-20T22:57:58.420675+0000","last_became_active":"2024-09-20T22:56:08.348928+0000","last_became_peered":"2024-09-20T22:56:08.348928+0000","last_unstale":"2024-09-20T22:57:58.420675+0000","last_undegraded":"2024-09-20T22:57:58.420675+0000","last_fullsized":"2024-09-20T22:57:58.420675+0000","mapping_epoch":887,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'44","last_scrub_stamp":"2024-09-20T22:57:06.157986+0000","last_deep_scrub":"113'28","last_deep_scrub_stamp":"2024-09-20T22:43:12.725900+0000","last_clean_scrub_stamp":"2024-09-20T22:57:06.157986+0000","objects_scrubbed":9,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:18.682809+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":230,"reported_epoch":993,"state":"active+clean","last_fresh":"2024-09-20T22:57:58.420909+0000","last_change":"2024-09-20T22:57:49.404176+0000","last_active":"2024-09-20T22:57:58.420909+0000","last_peered":"2024-09-20T22:57:58.420909+0000","last_clean":"2024-09-20T22:57:58.420909+0000","last_became_active":"2024-09-20T22:57:49.403858+0000","last_became_peered":"2024-09-20T22:57:49.403858+0000","last_unstale":"2024-09-20T22:57:58.420909+0000","last_undegraded":"2024-09-20T22:57:58.420909+0000","last_fullsized":"2024-09-20T22:57:58.420909+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:55:29.772589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:55:29.772589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":1933,"reported_epoch":993,"state":"active+clean","last_fresh":"2024-09-20T22:57:58.420347+0000","last_change":"2024-09-20T22:55:56.107908+0000","last_active":"2024-09-20T22:57:58.420347+0000","last_peered":"2024-09-20T22:57:58.420347+0000","last_clean":"2024-09-20T22:57:58.420347+0000","last_became_active":"2024-09-20T22:47:49.223220+0000","last_became_peered":"2024-09-20T22:47:49.223220+0000","last_unstale":"2024-09-20T22:57:58.420347+0000","last_undegraded":"2024-09-20T22:57:58.420347+0000","last_fullsized":"2024-09-20T22:57:58.420347+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:55:56.107827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:54:21.496068+0000","last_clean_scrub_stamp":"2024-09-20T22:55:56.107827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":230,"reported_epoch":993,"state":"active+clean","last_fresh":"2024-09-20T22:57:58.419756+0000","last_change":"2024-09-20T22:57:49.399320+0000","last_active":"2024-09-20T22:57:58.419756+0000","last_peered":"2024-09-20T22:57:58.419756+0000","last_clean":"2024-09-20T22:57:58.419756+0000","last_became_active":"2024-09-20T22:57:49.399016+0000","last_became_peered":"2024-09-20T22:57:49.399016+0000","last_unstale":"2024-09-20T22:57:58.419756+0000","last_undegraded":"2024-09-20T22:57:58.419756+0000","last_fullsized":"2024-09-20T22:57:58.419756+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:55:54.870627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:55:54.870627+0000","last_clean_scrub_stamp":"2024-09-20T22:55:54.870627+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.a","version":"989'1256","reported_seq":5515,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-20T22:57:52.718839+0000","last_change":"2024-09-20T22:57:49.673268+0000","last_active":"2024-09-20T22:57:52.718839+0000","last_peered":"2024-09-20T22:57:52.718839+0000","last_clean":"2024-09-20T22:57:52.718839+0000","last_became_active":"2024-09-20T22:57:49.401412+0000","last_became_peered":"2024-09-20T22:57:49.401412+0000","last_unstale":"2024-09-20T22:57:52.718839+0000","last_undegraded":"2024-09-20T22:57:52.718839+0000","last_fullsized":"2024-09-20T22:57:52.718839+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"851'1076","last_scrub_stamp":"2024-09-20T22:55:32.567038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:55:32.567038+0000","objects_scrubbed":21,"log_size":1256,"log_dups_size":0,"ondisk_log_size":1256,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:55.630500+0000","scrub_duration":10,"objects_trimmed":5,"snaptrim_duration":0.120863633,"stat_sum":{"num_bytes":64995264,"num_objects":25,"num_object_clones":22,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":485,"num_read_kb":502840,"num_write":373,"num_write_kb":163696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":11877612,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"16b","length":"1"},{"start":"173","length":"1"},{"start":"17d","length":"1"},{"start":"180","length":"1"},{"start":"182","length":"3"},{"start":"186","length":"8"},{"start":"190","length":"10"},{"start":"1a1","length":"1"},{"start":"1a3","length":"2"},{"start":"1a6","length":"1"},{"start":"1aa","length":"2"},{"start":"1af","length":"1"},{"start":"1b5","length":"1"},{"start":"1b7","length":"1"},{"start":"1ba","length":"1"},{"start":"1bd","length":"2"},{"start":"1c0","length":"2"},{"start":"1c5","length":"1"},{"start":"1c7","length":"1"}]},{"pgid":"4.9","version":"686'23","reported_seq":1170,"reported_epoch":993,"state":"active+clean","last_fresh":"2024-09-20T22:57:58.369667+0000","last_change":"2024-09-20T22:57:41.298315+0000","last_active":"2024-09-20T22:57:58.369667+0000","last_peered":"2024-09-20T22:57:58.369667+0000","last_clean":"2024-09-20T22:57:58.369667+0000","last_became_active":"2024-09-20T22:55:17.346757+0000","last_became_peered":"2024-09-20T22:55:17.346757+0000","last_unstale":"2024-09-20T22:57:58.369667+0000","last_undegraded":"2024-09-20T22:57:58.369667+0000","last_fullsized":"2024-09-20T22:57:58.369667+0000","mapping_epoch":836,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":837,"parent":"0.0","parent_split_bits":0,"last_scrub":"686'23","last_scrub_stamp":"2024-09-20T22:57:41.298234+0000","last_deep_scrub":"686'23","last_deep_scrub_stamp":"2024-09-20T22:52:46.768812+0000","last_clean_scrub_stamp":"2024-09-20T22:57:41.298234+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:09.189056+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":21,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":1735,"reported_epoch":993,"state":"active+clean","last_fresh":"2024-09-20T22:57:58.370187+0000","last_change":"2024-09-20T22:57:11.250944+0000","last_active":"2024-09-20T22:57:58.370187+0000","last_peered":"2024-09-20T22:57:58.370187+0000","last_clean":"2024-09-20T22:57:58.370187+0000","last_became_active":"2024-09-20T22:56:08.358713+0000","last_became_peered":"2024-09-20T22:56:08.358713+0000","last_unstale":"2024-09-20T22:57:58.370187+0000","last_undegraded":"2024-09-20T22:57:58.370187+0000","last_fullsized":"2024-09-20T22:57:58.370187+0000","mapping_epoch":887,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:57:11.250858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:46:52.465940+0000","last_clean_scrub_stamp":"2024-09-20T22:57:11.250858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:35.003707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":1953,"reported_epoch":993,"state":"active+clean","last_fresh":"2024-09-20T22:57:58.419921+0000","last_change":"2024-09-20T22:57:10.236664+0000","last_active":"2024-09-20T22:57:58.419921+0000","last_peered":"2024-09-20T22:57:58.419921+0000","last_clean":"2024-09-20T22:57:58.419921+0000","last_became_active":"2024-09-20T22:54:17.340269+0000","last_became_peered":"2024-09-20T22:54:17.340269+0000","last_unstale":"2024-09-20T22:57:58.419921+0000","last_undegraded":"2024-09-20T22:57:58.419921+0000","last_fullsized":"2024-09-20T22:57:58.419921+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:57:10.236592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:53:07.104891+0000","last_clean_scrub_stamp":"2024-09-20T22:57:10.236592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:35.494601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":1796,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-20T22:57:52.718643+0000","last_change":"2024-09-20T22:57:49.422638+0000","last_active":"2024-09-20T22:57:52.718643+0000","last_peered":"2024-09-20T22:57:52.718643+0000","last_clean":"2024-09-20T22:57:52.718643+0000","last_became_active":"2024-09-20T22:57:49.419390+0000","last_became_peered":"2024-09-20T22:57:49.419390+0000","last_unstale":"2024-09-20T22:57:52.718643+0000","last_undegraded":"2024-09-20T22:57:52.718643+0000","last_fullsized":"2024-09-20T22:57:52.718643+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:53.242204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:55.170379+0000","last_clean_scrub_stamp":"2024-09-20T22:53:53.242204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:55.629963+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"993'1673","reported_seq":6188,"reported_epoch":993,"state":"active+clean+snaptrim","last_fresh":"2024-09-20T22:57:58.578710+0000","last_change":"2024-09-20T22:57:58.576045+0000","last_active":"2024-09-20T22:57:58.578710+0000","last_peered":"2024-09-20T22:57:58.578710+0000","last_clean":"2024-09-20T22:57:58.578710+0000","last_became_active":"2024-09-20T22:54:20.415706+0000","last_became_peered":"2024-09-20T22:54:20.415706+0000","last_unstale":"2024-09-20T22:57:58.578710+0000","last_undegraded":"2024-09-20T22:57:58.578710+0000","last_fullsized":"2024-09-20T22:57:58.578710+0000","mapping_epoch":779,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":780,"parent":"0.0","parent_split_bits":0,"last_scrub":"794'1255","last_scrub_stamp":"2024-09-20T22:54:37.866509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:54:37.866509+0000","objects_scrubbed":15,"log_size":1673,"log_dups_size":0,"ondisk_log_size":1673,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":10,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:00.453836+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.204046914,"stat_sum":{"num_bytes":69638720,"num_objects":33,"num_object_clones":28,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":2,"num_read":681,"num_read_kb":757216,"num_write":460,"num_write_kb":173264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":168,"num_bytes_recovered":68651320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"180","length":"1"},{"start":"18d","length":"1"},{"start":"190","length":"1"},{"start":"193","length":"1"},{"start":"196","length":"2"},{"start":"19a","length":"1"},{"start":"19c","length":"2"},{"start":"1a3","length":"2"},{"start":"1ab","length":"1"},{"start":"1af","length":"1"},{"start":"1c0","length":"1"}]},{"pgid":"4.8","version":"625'27","reported_seq":1975,"reported_epoch":993,"state":"active+clean","last_fresh":"2024-09-20T22:57:58.420467+0000","last_change":"2024-09-20T22:57:10.649039+0000","last_active":"2024-09-20T22:57:58.420467+0000","last_peered":"2024-09-20T22:57:58.420467+0000","last_clean":"2024-09-20T22:57:58.420467+0000","last_became_active":"2024-09-20T22:56:08.360719+0000","last_became_peered":"2024-09-20T22:56:08.360719+0000","last_unstale":"2024-09-20T22:57:58.420467+0000","last_undegraded":"2024-09-20T22:57:58.420467+0000","last_fullsized":"2024-09-20T22:57:58.420467+0000","mapping_epoch":887,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'27","last_scrub_stamp":"2024-09-20T22:57:10.648909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:57:10.648909+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:13.332381+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'2","reported_seq":933,"reported_epoch":993,"state":"active+clean","last_fresh":"2024-09-20T22:57:58.372007+0000","last_change":"2024-09-20T22:56:08.422471+0000","last_active":"2024-09-20T22:57:58.372007+0000","last_peered":"2024-09-20T22:57:58.372007+0000","last_clean":"2024-09-20T22:57:58.372007+0000","last_became_active":"2024-09-20T22:56:08.392282+0000","last_became_peered":"2024-09-20T22:56:08.392282+0000","last_unstale":"2024-09-20T22:57:58.372007+0000","last_undegraded":"2024-09-20T22:57:58.372007+0000","last_fullsized":"2024-09-20T22:57:58.372007+0000","mapping_epoch":887,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-09-20T22:55:33.770279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:55:33.770279+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":276,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.9","version":"888'11","reported_seq":279,"reported_epoch":993,"state":"active+clean","last_fresh":"2024-09-20T22:57:58.419917+0000","last_change":"2024-09-20T22:56:48.156966+0000","last_active":"2024-09-20T22:57:58.419917+0000","last_peered":"2024-09-20T22:57:58.419917+0000","last_clean":"2024-09-20T22:57:58.419917+0000","last_became_active":"2024-09-20T22:56:08.357520+0000","last_became_peered":"2024-09-20T22:56:08.357520+0000","last_unstale":"2024-09-20T22:57:58.419917+0000","last_undegraded":"2024-09-20T22:57:58.419917+0000","last_fullsized":"2024-09-20T22:57:58.419917+0000","mapping_epoch":887,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":0,"last_scrub":"888'11","last_scrub_stamp":"2024-09-20T22:56:48.156843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:56:48.156843+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":1914,"reported_epoch":993,"state":"active+clean","last_fresh":"2024-09-20T22:57:58.421582+0000","last_change":"2024-09-20T22:57:14.615994+0000","last_active":"2024-09-20T22:57:58.421582+0000","last_peered":"2024-09-20T22:57:58.421582+0000","last_clean":"2024-09-20T22:57:58.421582+0000","last_became_active":"2024-09-20T22:47:49.227569+0000","last_became_peered":"2024-09-20T22:47:49.227569+0000","last_unstale":"2024-09-20T22:57:58.421582+0000","last_undegraded":"2024-09-20T22:57:58.421582+0000","last_fullsized":"2024-09-20T22:57:58.421582+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:57:14.615909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:57:14.615909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:29.335354+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.14","version":"989'730","reported_seq":3755,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-20T22:57:52.719036+0000","last_change":"2024-09-20T22:57:51.381495+0000","last_active":"2024-09-20T22:57:52.719036+0000","last_peered":"2024-09-20T22:57:52.719036+0000","last_clean":"2024-09-20T22:57:52.719036+0000","last_became_active":"2024-09-20T22:57:51.380711+0000","last_became_peered":"2024-09-20T22:57:51.380711+0000","last_unstale":"2024-09-20T22:57:52.719036+0000","last_undegraded":"2024-09-20T22:57:52.719036+0000","last_fullsized":"2024-09-20T22:57:52.719036+0000","mapping_epoch":990,"log_start":"456'176","ondisk_log_start":"456'176","created":932,"last_epoch_clean":991,"parent":"0.0","parent_split_bits":5,"last_scrub":"853'620","last_scrub_stamp":"2024-09-20T22:55:34.565081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:55:34.565081+0000","objects_scrubbed":8,"log_size":554,"log_dups_size":318,"ondisk_log_size":554,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:56.672822+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.037993079999999999,"stat_sum":{"num_bytes":16142342,"num_objects":7,"num_object_clones":6,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":112,"num_read_kb":107256,"num_write":99,"num_write_kb":34830,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":33042853,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"16b","length":"1"},{"start":"173","length":"1"},{"start":"17d","length":"1"},{"start":"180","length":"1"},{"start":"182","length":"3"},{"start":"186","length":"8"},{"start":"190","length":"10"},{"start":"1a1","length":"1"},{"start":"1a3","length":"2"},{"start":"1a6","length":"1"},{"start":"1aa","length":"2"},{"start":"1af","length":"1"},{"start":"1b5","length":"1"},{"start":"1b7","length":"1"},{"start":"1ba","length":"1"},{"start":"1bd","length":"2"},{"start":"1c0","length":"2"},{"start":"1c5","length":"1"},{"start":"1c7","length":"1"}]},{"pgid":"3.10","version":"0'0","reported_seq":1947,"reported_epoch":993,"state":"active+clean","last_fresh":"2024-09-20T22:57:58.420095+0000","last_change":"2024-09-20T22:56:17.950955+0000","last_active":"2024-09-20T22:57:58.420095+0000","last_peered":"2024-09-20T22:57:58.420095+0000","last_clean":"2024-09-20T22:57:58.420095+0000","last_became_active":"2024-09-20T22:47:49.213292+0000","last_became_peered":"2024-09-20T22:47:49.213292+0000","last_unstale":"2024-09-20T22:57:58.420095+0000","last_undegraded":"2024-09-20T22:57:58.420095+0000","last_fullsized":"2024-09-20T22:57:58.420095+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:56:17.950879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:56:17.950879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"625'12","reported_seq":1854,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-20T22:57:52.719279+0000","last_change":"2024-09-20T22:57:49.427905+0000","last_active":"2024-09-20T22:57:52.719279+0000","last_peered":"2024-09-20T22:57:52.719279+0000","last_clean":"2024-09-20T22:57:52.719279+0000","last_became_active":"2024-09-20T22:57:49.426479+0000","last_became_peered":"2024-09-20T22:57:49.426479+0000","last_unstale":"2024-09-20T22:57:52.719279+0000","last_undegraded":"2024-09-20T22:57:52.719279+0000","last_fullsized":"2024-09-20T22:57:52.719279+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'12","last_scrub_stamp":"2024-09-20T22:53:26.319144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:53:26.319144+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:55.629812+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":1926,"reported_epoch":993,"state":"active+clean","last_fresh":"2024-09-20T22:57:58.421599+0000","last_change":"2024-09-20T22:57:49.408789+0000","last_active":"2024-09-20T22:57:58.421599+0000","last_peered":"2024-09-20T22:57:58.421599+0000","last_clean":"2024-09-20T22:57:58.421599+0000","last_became_active":"2024-09-20T22:57:49.408529+0000","last_became_peered":"2024-09-20T22:57:49.408529+0000","last_unstale":"2024-09-20T22:57:58.421599+0000","last_undegraded":"2024-09-20T22:57:58.421599+0000","last_fullsized":"2024-09-20T22:57:58.421599+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:55:29.324286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:09.705632+0000","last_clean_scrub_stamp":"2024-09-20T22:55:29.324286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"74'1","reported_seq":242,"reported_epoch":993,"state":"active+clean","last_fresh":"2024-09-20T22:57:58.421714+0000","last_change":"2024-09-20T22:56:47.178707+0000","last_active":"2024-09-20T22:57:58.421714+0000","last_peered":"2024-09-20T22:57:58.421714+0000","last_clean":"2024-09-20T22:57:58.421714+0000","last_became_active":"2024-09-20T22:56:08.354987+0000","last_became_peered":"2024-09-20T22:56:08.354987+0000","last_unstale":"2024-09-20T22:57:58.421714+0000","last_undegraded":"2024-09-20T22:57:58.421714+0000","last_fullsized":"2024-09-20T22:57:58.421714+0000","mapping_epoch":887,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-09-20T22:56:47.178627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:56:47.178627+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:11.125701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.15","version":"930'540","reported_seq":5104,"reported_epoch":993,"state":"active+clean","last_fresh":"2024-09-20T22:57:58.369941+0000","last_change":"2024-09-20T22:57:48.374841+0000","last_active":"2024-09-20T22:57:58.369941+0000","last_peered":"2024-09-20T22:57:58.369941+0000","last_clean":"2024-09-20T22:57:58.369941+0000","last_became_active":"2024-09-20T22:56:58.367157+0000","last_became_peered":"2024-09-20T22:56:58.367157+0000","last_unstale":"2024-09-20T22:57:58.369941+0000","last_undegraded":"2024-09-20T22:57:58.369941+0000","last_fullsized":"2024-09-20T22:57:58.369941+0000","mapping_epoch":937,"log_start":"450'190","ondisk_log_start":"450'190","created":932,"last_epoch_clean":938,"parent":"0.0","parent_split_bits":5,"last_scrub":"930'540","last_scrub_stamp":"2024-09-20T22:57:04.343926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:57:04.343926+0000","objects_scrubbed":0,"log_size":350,"log_dups_size":240,"ondisk_log_size":350,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:18.165862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011806360000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"180","length":"1"},{"start":"18d","length":"1"},{"start":"190","length":"1"},{"start":"193","length":"1"},{"start":"196","length":"2"},{"start":"19a","length":"1"},{"start":"19c","length":"2"},{"start":"1a3","length":"2"},{"start":"1a6","length":"1"},{"start":"1aa","length":"2"},{"start":"1af","length":"1"},{"start":"1b5","length":"1"},{"start":"1b7","length":"1"},{"start":"1ba","length":"1"},{"start":"1bd","length":"2"},{"start":"1c0","length":"2"},{"start":"1c5","length":"1"},{"start":"1c7","length":"1"}]},{"pgid":"4.16","version":"625'15","reported_seq":1299,"reported_epoch":993,"state":"active+clean","last_fresh":"2024-09-20T22:57:58.371462+0000","last_change":"2024-09-20T22:57:49.413436+0000","last_active":"2024-09-20T22:57:58.371462+0000","last_peered":"2024-09-20T22:57:58.371462+0000","last_clean":"2024-09-20T22:57:58.371462+0000","last_became_active":"2024-09-20T22:57:49.412096+0000","last_became_peered":"2024-09-20T22:57:49.412096+0000","last_unstale":"2024-09-20T22:57:58.371462+0000","last_undegraded":"2024-09-20T22:57:58.371462+0000","last_fullsized":"2024-09-20T22:57:58.371462+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'15","last_scrub_stamp":"2024-09-20T22:55:27.637946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:55:27.637946+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":946,"reported_epoch":993,"state":"active+clean","last_fresh":"2024-09-20T22:57:58.421088+0000","last_change":"2024-09-20T22:56:11.967360+0000","last_active":"2024-09-20T22:57:58.421088+0000","last_peered":"2024-09-20T22:57:58.421088+0000","last_clean":"2024-09-20T22:57:58.421088+0000","last_became_active":"2024-09-20T22:56:08.357576+0000","last_became_peered":"2024-09-20T22:56:08.357576+0000","last_unstale":"2024-09-20T22:57:58.421088+0000","last_undegraded":"2024-09-20T22:57:58.421088+0000","last_fullsized":"2024-09-20T22:57:58.421088+0000","mapping_epoch":887,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:56:11.967266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:56:11.967266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":1808,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-20T22:57:52.719169+0000","last_change":"2024-09-20T22:57:49.433827+0000","last_active":"2024-09-20T22:57:52.719169+0000","last_peered":"2024-09-20T22:57:52.719169+0000","last_clean":"2024-09-20T22:57:52.719169+0000","last_became_active":"2024-09-20T22:57:49.433318+0000","last_became_peered":"2024-09-20T22:57:49.433318+0000","last_unstale":"2024-09-20T22:57:52.719169+0000","last_undegraded":"2024-09-20T22:57:52.719169+0000","last_fullsized":"2024-09-20T22:57:52.719169+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:55:52.481737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:55:52.481737+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:55.630674+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":1954,"reported_epoch":993,"state":"active+clean","last_fresh":"2024-09-20T22:57:58.420701+0000","last_change":"2024-09-20T22:56:42.057480+0000","last_active":"2024-09-20T22:57:58.420701+0000","last_peered":"2024-09-20T22:57:58.420701+0000","last_clean":"2024-09-20T22:57:58.420701+0000","last_became_active":"2024-09-20T22:56:08.369854+0000","last_became_peered":"2024-09-20T22:56:08.369854+0000","last_unstale":"2024-09-20T22:57:58.420701+0000","last_undegraded":"2024-09-20T22:57:58.420701+0000","last_fullsized":"2024-09-20T22:57:58.420701+0000","mapping_epoch":887,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:56:42.057380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:53:58.169894+0000","last_clean_scrub_stamp":"2024-09-20T22:56:42.057380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.16","version":"988'903","reported_seq":5426,"reported_epoch":993,"state":"active+clean","last_fresh":"2024-09-20T22:57:58.370592+0000","last_change":"2024-09-20T22:57:48.439743+0000","last_active":"2024-09-20T22:57:58.370592+0000","last_peered":"2024-09-20T22:57:58.370592+0000","last_clean":"2024-09-20T22:57:58.370592+0000","last_became_active":"2024-09-20T22:56:58.372517+0000","last_became_peered":"2024-09-20T22:56:58.372517+0000","last_unstale":"2024-09-20T22:57:58.370592+0000","last_undegraded":"2024-09-20T22:57:58.370592+0000","last_fullsized":"2024-09-20T22:57:58.370592+0000","mapping_epoch":937,"log_start":"443'181","ondisk_log_start":"443'181","created":932,"last_epoch_clean":938,"parent":"0.0","parent_split_bits":5,"last_scrub":"944'873","last_scrub_stamp":"2024-09-20T22:57:07.764870+0000","last_deep_scrub":"902'811","last_deep_scrub_stamp":"2024-09-20T22:56:22.658684+0000","last_clean_scrub_stamp":"2024-09-20T22:57:07.764870+0000","objects_scrubbed":12,"log_size":722,"log_dups_size":314,"ondisk_log_size":722,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:18.543615+0000","scrub_duration":3,"objects_trimmed":1,"snaptrim_duration":0.070922279000000005,"stat_sum":{"num_bytes":21972561,"num_objects":9,"num_object_clones":8,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"180","length":"1"},{"start":"18d","length":"1"},{"start":"190","length":"1"},{"start":"193","length":"1"},{"start":"196","length":"2"},{"start":"19a","length":"1"},{"start":"19c","length":"2"},{"start":"1a3","length":"2"},{"start":"1a6","length":"1"},{"start":"1aa","length":"2"},{"start":"1af","length":"1"},{"start":"1b5","length":"1"},{"start":"1b7","length":"1"},{"start":"1ba","length":"1"},{"start":"1bd","length":"2"},{"start":"1c0","length":"2"},{"start":"1c5","length":"1"},{"start":"1c7","length":"1"}]},{"pgid":"4.15","version":"625'15","reported_seq":1959,"reported_epoch":993,"state":"active+clean","last_fresh":"2024-09-20T22:57:58.421017+0000","last_change":"2024-09-20T22:57:49.404308+0000","last_active":"2024-09-20T22:57:58.421017+0000","last_peered":"2024-09-20T22:57:58.421017+0000","last_clean":"2024-09-20T22:57:58.421017+0000","last_became_active":"2024-09-20T22:57:49.403807+0000","last_became_peered":"2024-09-20T22:57:49.403807+0000","last_unstale":"2024-09-20T22:57:58.421017+0000","last_undegraded":"2024-09-20T22:57:58.421017+0000","last_fullsized":"2024-09-20T22:57:58.421017+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'15","last_scrub_stamp":"2024-09-20T22:56:15.488641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:56:15.488641+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":1278,"reported_epoch":993,"state":"active+clean","last_fresh":"2024-09-20T22:57:58.369553+0000","last_change":"2024-09-20T22:57:49.399263+0000","last_active":"2024-09-20T22:57:58.369553+0000","last_peered":"2024-09-20T22:57:58.369553+0000","last_clean":"2024-09-20T22:57:58.369553+0000","last_became_active":"2024-09-20T22:57:49.398995+0000","last_became_peered":"2024-09-20T22:57:49.398995+0000","last_unstale":"2024-09-20T22:57:58.369553+0000","last_undegraded":"2024-09-20T22:57:58.369553+0000","last_fullsized":"2024-09-20T22:57:58.369553+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:56:16.523567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:56:16.523567+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"989'11","reported_seq":1988,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-20T22:57:52.718810+0000","last_change":"2024-09-20T22:57:49.432210+0000","last_active":"2024-09-20T22:57:52.718810+0000","last_peered":"2024-09-20T22:57:52.718810+0000","last_clean":"2024-09-20T22:57:52.718810+0000","last_became_active":"2024-09-20T22:57:49.430688+0000","last_became_peered":"2024-09-20T22:57:49.430688+0000","last_unstale":"2024-09-20T22:57:52.718810+0000","last_undegraded":"2024-09-20T22:57:52.718810+0000","last_fullsized":"2024-09-20T22:57:52.718810+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-09-20T22:53:59.165134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:53:59.165134+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:55.630110+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":871,"reported_epoch":993,"state":"active+clean","last_fresh":"2024-09-20T22:57:58.423426+0000","last_change":"2024-09-20T22:55:29.161667+0000","last_active":"2024-09-20T22:57:58.423426+0000","last_peered":"2024-09-20T22:57:58.423426+0000","last_clean":"2024-09-20T22:57:58.423426+0000","last_became_active":"2024-09-20T22:55:17.346270+0000","last_became_peered":"2024-09-20T22:55:17.346270+0000","last_unstale":"2024-09-20T22:57:58.423426+0000","last_undegraded":"2024-09-20T22:57:58.423426+0000","last_fullsized":"2024-09-20T22:57:58.423426+0000","mapping_epoch":836,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":837,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:55:29.161595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:55:29.161595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.17","version":"990'2020","reported_seq":6513,"reported_epoch":993,"state":"active+clean","last_fresh":"2024-09-20T22:57:58.422533+0000","last_change":"2024-09-20T22:57:52.717318+0000","last_active":"2024-09-20T22:57:58.422533+0000","last_peered":"2024-09-20T22:57:58.422533+0000","last_clean":"2024-09-20T22:57:58.422533+0000","last_became_active":"2024-09-20T22:57:52.378717+0000","last_became_peered":"2024-09-20T22:57:52.378717+0000","last_unstale":"2024-09-20T22:57:58.422533+0000","last_undegraded":"2024-09-20T22:57:58.422533+0000","last_fullsized":"2024-09-20T22:57:58.422533+0000","mapping_epoch":991,"log_start":"436'456","ondisk_log_start":"436'456","created":932,"last_epoch_clean":992,"parent":"0.0","parent_split_bits":5,"last_scrub":"990'2020","last_scrub_stamp":"2024-09-20T22:57:52.717196+0000","last_deep_scrub":"879'1788","last_deep_scrub_stamp":"2024-09-20T22:55:59.964830+0000","last_clean_scrub_stamp":"2024-09-20T22:57:52.717196+0000","objects_scrubbed":10,"log_size":1564,"log_dups_size":886,"ondisk_log_size":1564,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:10.038236+0000","scrub_duration":3,"objects_trimmed":1,"snaptrim_duration":0.034776473000000002,"stat_sum":{"num_bytes":26840801,"num_objects":10,"num_object_clones":8,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"180","length":"1"},{"start":"18d","length":"1"},{"start":"190","length":"1"},{"start":"193","length":"1"},{"start":"196","length":"2"},{"start":"19a","length":"1"},{"start":"19c","length":"2"},{"start":"1a3","length":"2"},{"start":"1a6","length":"1"},{"start":"1aa","length":"2"},{"start":"1af","length":"1"},{"start":"1b5","length":"1"},{"start":"1b7","length":"1"},{"start":"1ba","length":"1"},{"start":"1bd","length":"2"},{"start":"1c0","length":"2"},{"start":"1c5","length":"1"},{"start":"1c7","length":"1"}]},{"pgid":"4.14","version":"625'20","reported_seq":1886,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-20T22:57:52.718643+0000","last_change":"2024-09-20T22:57:49.431205+0000","last_active":"2024-09-20T22:57:52.718643+0000","last_peered":"2024-09-20T22:57:52.718643+0000","last_clean":"2024-09-20T22:57:52.718643+0000","last_became_active":"2024-09-20T22:57:49.427276+0000","last_became_peered":"2024-09-20T22:57:49.427276+0000","last_unstale":"2024-09-20T22:57:52.718643+0000","last_undegraded":"2024-09-20T22:57:52.718643+0000","last_fullsized":"2024-09-20T22:57:52.718643+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'20","last_scrub_stamp":"2024-09-20T22:55:23.474362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:55:23.474362+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:55.630391+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":936,"reported_epoch":993,"state":"active+clean","last_fresh":"2024-09-20T22:57:58.420433+0000","last_change":"2024-09-20T22:56:08.372321+0000","last_active":"2024-09-20T22:57:58.420433+0000","last_peered":"2024-09-20T22:57:58.420433+0000","last_clean":"2024-09-20T22:57:58.420433+0000","last_became_active":"2024-09-20T22:56:08.371946+0000","last_became_peered":"2024-09-20T22:56:08.371946+0000","last_unstale":"2024-09-20T22:57:58.420433+0000","last_undegraded":"2024-09-20T22:57:58.420433+0000","last_fullsized":"2024-09-20T22:57:58.420433+0000","mapping_epoch":887,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:55:49.897082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:55:49.897082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.15","version":"778'12","reported_seq":2162,"reported_epoch":993,"state":"active+clean","last_fresh":"2024-09-20T22:57:58.420735+0000","last_change":"2024-09-20T22:57:36.726648+0000","last_active":"2024-09-20T22:57:58.420735+0000","last_peered":"2024-09-20T22:57:58.420735+0000","last_clean":"2024-09-20T22:57:58.420735+0000","last_became_active":"2024-09-20T22:54:17.341014+0000","last_became_peered":"2024-09-20T22:54:17.341014+0000","last_unstale":"2024-09-20T22:57:58.420735+0000","last_undegraded":"2024-09-20T22:57:58.420735+0000","last_fullsized":"2024-09-20T22:57:58.420735+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"778'12","last_scrub_stamp":"2024-09-20T22:57:36.726568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:57:36.726568+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:52.611026+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":1270,"reported_epoch":993,"state":"active+clean","last_fresh":"2024-09-20T22:57:58.370954+0000","last_change":"2024-09-20T22:57:49.412393+0000","last_active":"2024-09-20T22:57:58.370954+0000","last_peered":"2024-09-20T22:57:58.370954+0000","last_clean":"2024-09-20T22:57:58.370954+0000","last_became_active":"2024-09-20T22:57:49.411010+0000","last_became_peered":"2024-09-20T22:57:49.411010+0000","last_unstale":"2024-09-20T22:57:58.370954+0000","last_undegraded":"2024-09-20T22:57:58.370954+0000","last_fullsized":"2024-09-20T22:57:58.370954+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:56:49.691267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:50:21.853070+0000","last_clean_scrub_stamp":"2024-09-20T22:56:49.691267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:02.273990+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.10","version":"989'1011","reported_seq":5449,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-20T22:57:55.682099+0000","last_change":"2024-09-20T22:57:55.682134+0000","last_active":"2024-09-20T22:57:55.682099+0000","last_peered":"2024-09-20T22:57:55.682099+0000","last_clean":"2024-09-20T22:57:55.682099+0000","last_became_active":"2024-09-20T22:57:49.398604+0000","last_became_peered":"2024-09-20T22:57:49.398604+0000","last_unstale":"2024-09-20T22:57:55.682099+0000","last_undegraded":"2024-09-20T22:57:55.682099+0000","last_fullsized":"2024-09-20T22:57:55.682099+0000","mapping_epoch":988,"log_start":"480'253","ondisk_log_start":"480'253","created":932,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":5,"last_scrub":"989'1011","last_scrub_stamp":"2024-09-20T22:57:55.682037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:57:55.682037+0000","objects_scrubbed":8,"log_size":758,"log_dups_size":383,"ondisk_log_size":758,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:16.360864+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.080082789000000001,"stat_sum":{"num_bytes":15927188,"num_objects":8,"num_object_clones":7,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"16b","length":"1"},{"start":"173","length":"1"},{"start":"17d","length":"1"},{"start":"180","length":"1"},{"start":"182","length":"3"},{"start":"186","length":"8"},{"start":"190","length":"10"},{"start":"1a1","length":"1"},{"start":"1a3","length":"2"},{"start":"1a6","length":"1"},{"start":"1aa","length":"2"},{"start":"1af","length":"1"},{"start":"1b5","length":"1"},{"start":"1b7","length":"1"},{"start":"1ba","length":"1"},{"start":"1bd","length":"2"},{"start":"1c0","length":"2"},{"start":"1c5","length":"1"},{"start":"1c7","length":"1"}]},{"pgid":"4.13","version":"625'19","reported_seq":837,"reported_epoch":993,"state":"active+clean","last_fresh":"2024-09-20T22:57:58.421577+0000","last_change":"2024-09-20T22:55:55.832120+0000","last_active":"2024-09-20T22:57:58.421577+0000","last_peered":"2024-09-20T22:57:58.421577+0000","last_clean":"2024-09-20T22:57:58.421577+0000","last_became_active":"2024-09-20T22:55:17.342351+0000","last_became_peered":"2024-09-20T22:55:17.342351+0000","last_unstale":"2024-09-20T22:57:58.421577+0000","last_undegraded":"2024-09-20T22:57:58.421577+0000","last_fullsized":"2024-09-20T22:57:58.421577+0000","mapping_epoch":836,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":837,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'19","last_scrub_stamp":"2024-09-20T22:55:55.832023+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:55:55.832023+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":23,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":738,"reported_epoch":993,"state":"active+clean","last_fresh":"2024-09-20T22:57:58.422395+0000","last_change":"2024-09-20T22:57:15.158099+0000","last_active":"2024-09-20T22:57:58.422395+0000","last_peered":"2024-09-20T22:57:58.422395+0000","last_clean":"2024-09-20T22:57:58.422395+0000","last_became_active":"2024-09-20T22:56:08.369206+0000","last_became_peered":"2024-09-20T22:56:08.369206+0000","last_unstale":"2024-09-20T22:57:58.422395+0000","last_undegraded":"2024-09-20T22:57:58.422395+0000","last_fullsized":"2024-09-20T22:57:58.422395+0000","mapping_epoch":887,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:57:15.158013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:52:47.717490+0000","last_clean_scrub_stamp":"2024-09-20T22:57:15.158013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:32.667409+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":467,"reported_epoch":993,"state":"active+clean","last_fresh":"2024-09-20T22:57:58.370315+0000","last_change":"2024-09-20T22:57:55.455704+0000","last_active":"2024-09-20T22:57:58.370315+0000","last_peered":"2024-09-20T22:57:58.370315+0000","last_clean":"2024-09-20T22:57:58.370315+0000","last_became_active":"2024-09-20T22:57:49.398369+0000","last_became_peered":"2024-09-20T22:57:49.398369+0000","last_unstale":"2024-09-20T22:57:58.370315+0000","last_undegraded":"2024-09-20T22:57:58.370315+0000","last_fullsized":"2024-09-20T22:57:58.370315+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:57:55.455633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:57:55.455633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:04.115059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":1742,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-20T22:57:52.717678+0000","last_change":"2024-09-20T22:57:49.416021+0000","last_active":"2024-09-20T22:57:52.717678+0000","last_peered":"2024-09-20T22:57:52.717678+0000","last_clean":"2024-09-20T22:57:52.717678+0000","last_became_active":"2024-09-20T22:57:49.413148+0000","last_became_peered":"2024-09-20T22:57:49.413148+0000","last_unstale":"2024-09-20T22:57:52.717678+0000","last_undegraded":"2024-09-20T22:57:52.717678+0000","last_fullsized":"2024-09-20T22:57:52.717678+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:56:22.490523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:56:22.490523+0000","last_clean_scrub_stamp":"2024-09-20T22:56:22.490523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:55.630816+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"929'565","reported_seq":4993,"reported_epoch":993,"state":"active+clean","last_fresh":"2024-09-20T22:57:58.419614+0000","last_change":"2024-09-20T22:57:50.386505+0000","last_active":"2024-09-20T22:57:58.419614+0000","last_peered":"2024-09-20T22:57:58.419614+0000","last_clean":"2024-09-20T22:57:58.419614+0000","last_became_active":"2024-09-20T22:57:50.385308+0000","last_became_peered":"2024-09-20T22:57:50.385308+0000","last_unstale":"2024-09-20T22:57:58.419614+0000","last_undegraded":"2024-09-20T22:57:58.419614+0000","last_fullsized":"2024-09-20T22:57:58.419614+0000","mapping_epoch":989,"log_start":"474'259","ondisk_log_start":"474'259","created":932,"last_epoch_clean":990,"parent":"0.0","parent_split_bits":5,"last_scrub":"929'565","last_scrub_stamp":"2024-09-20T22:57:33.220990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:57:33.220990+0000","objects_scrubbed":0,"log_size":306,"log_dups_size":338,"ondisk_log_size":306,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:02.936257+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00070306199999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"180","length":"1"},{"start":"18d","length":"1"},{"start":"190","length":"1"},{"start":"193","length":"1"},{"start":"196","length":"2"},{"start":"19a","length":"1"},{"start":"19c","length":"2"},{"start":"1a3","length":"2"},{"start":"1a6","length":"1"},{"start":"1aa","length":"2"},{"start":"1af","length":"1"},{"start":"1b5","length":"1"},{"start":"1b7","length":"1"},{"start":"1ba","length":"1"},{"start":"1bd","length":"2"},{"start":"1c0","length":"2"},{"start":"1c5","length":"1"},{"start":"1c7","length":"1"}]},{"pgid":"4.12","version":"625'19","reported_seq":460,"reported_epoch":993,"state":"active+clean","last_fresh":"2024-09-20T22:57:58.371570+0000","last_change":"2024-09-20T22:55:33.932286+0000","last_active":"2024-09-20T22:57:58.371570+0000","last_peered":"2024-09-20T22:57:58.371570+0000","last_clean":"2024-09-20T22:57:58.371570+0000","last_became_active":"2024-09-20T22:54:17.340539+0000","last_became_peered":"2024-09-20T22:54:17.340539+0000","last_unstale":"2024-09-20T22:57:58.371570+0000","last_undegraded":"2024-09-20T22:57:58.371570+0000","last_fullsized":"2024-09-20T22:57:58.371570+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'19","last_scrub_stamp":"2024-09-20T22:55:33.932211+0000","last_deep_scrub":"404'13","last_deep_scrub_stamp":"2024-09-20T22:48:04.276282+0000","last_clean_scrub_stamp":"2024-09-20T22:55:33.932211+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":836,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-20T22:57:52.718861+0000","last_change":"2024-09-20T22:57:49.406406+0000","last_active":"2024-09-20T22:57:52.718861+0000","last_peered":"2024-09-20T22:57:52.718861+0000","last_clean":"2024-09-20T22:57:52.718861+0000","last_became_active":"2024-09-20T22:57:49.403043+0000","last_became_peered":"2024-09-20T22:57:49.403043+0000","last_unstale":"2024-09-20T22:57:52.718861+0000","last_undegraded":"2024-09-20T22:57:52.718861+0000","last_fullsized":"2024-09-20T22:57:52.718861+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:54:46.541109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:31.138067+0000","last_clean_scrub_stamp":"2024-09-20T22:54:46.541109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:55.630318+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":1748,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-20T22:57:52.718450+0000","last_change":"2024-09-20T22:57:49.433147+0000","last_active":"2024-09-20T22:57:52.718450+0000","last_peered":"2024-09-20T22:57:52.718450+0000","last_clean":"2024-09-20T22:57:52.718450+0000","last_became_active":"2024-09-20T22:57:49.432838+0000","last_became_peered":"2024-09-20T22:57:49.432838+0000","last_unstale":"2024-09-20T22:57:52.718450+0000","last_undegraded":"2024-09-20T22:57:52.718450+0000","last_fullsized":"2024-09-20T22:57:52.718450+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:34.345077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:53:34.345077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:55.629845+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":1267,"reported_epoch":993,"state":"active+clean","last_fresh":"2024-09-20T22:57:58.369478+0000","last_change":"2024-09-20T22:57:38.552117+0000","last_active":"2024-09-20T22:57:58.369478+0000","last_peered":"2024-09-20T22:57:58.369478+0000","last_clean":"2024-09-20T22:57:58.369478+0000","last_became_active":"2024-09-20T22:54:17.344021+0000","last_became_peered":"2024-09-20T22:54:17.344021+0000","last_unstale":"2024-09-20T22:57:58.369478+0000","last_undegraded":"2024-09-20T22:57:58.369478+0000","last_fullsized":"2024-09-20T22:57:58.369478+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:57:38.552046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:55:09.664839+0000","last_clean_scrub_stamp":"2024-09-20T22:57:38.552046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:07.271978+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.12","version":"988'1150","reported_seq":5056,"reported_epoch":993,"state":"active+clean","last_fresh":"2024-09-20T22:57:58.420239+0000","last_change":"2024-09-20T22:57:48.417172+0000","last_active":"2024-09-20T22:57:58.420239+0000","last_peered":"2024-09-20T22:57:58.420239+0000","last_clean":"2024-09-20T22:57:58.420239+0000","last_became_active":"2024-09-20T22:57:03.429039+0000","last_became_peered":"2024-09-20T22:57:03.429039+0000","last_unstale":"2024-09-20T22:57:58.420239+0000","last_undegraded":"2024-09-20T22:57:58.420239+0000","last_fullsized":"2024-09-20T22:57:58.420239+0000","mapping_epoch":942,"log_start":"471'464","ondisk_log_start":"471'464","created":932,"last_epoch_clean":943,"parent":"0.0","parent_split_bits":5,"last_scrub":"943'1027","last_scrub_stamp":"2024-09-20T22:57:05.204065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:57:05.204065+0000","objects_scrubbed":15,"log_size":686,"log_dups_size":687,"ondisk_log_size":686,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:17.391441+0000","scrub_duration":6,"objects_trimmed":2,"snaptrim_duration":0.046814117000000002,"stat_sum":{"num_bytes":45111058,"num_objects":18,"num_object_clones":15,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":1,"num_read":29,"num_read_kb":39201,"num_write":15,"num_write_kb":14614,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"180","length":"1"},{"start":"18d","length":"1"},{"start":"190","length":"1"},{"start":"193","length":"1"},{"start":"196","length":"2"},{"start":"19a","length":"1"},{"start":"19c","length":"2"},{"start":"1a3","length":"2"},{"start":"1a6","length":"1"},{"start":"1aa","length":"2"},{"start":"1af","length":"1"},{"start":"1b5","length":"1"},{"start":"1b7","length":"1"},{"start":"1ba","length":"1"},{"start":"1bd","length":"2"},{"start":"1c0","length":"2"},{"start":"1c5","length":"1"},{"start":"1c7","length":"1"}]},{"pgid":"4.11","version":"625'24","reported_seq":125,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-20T22:57:52.717849+0000","last_change":"2024-09-20T22:57:49.414600+0000","last_active":"2024-09-20T22:57:52.717849+0000","last_peered":"2024-09-20T22:57:52.717849+0000","last_clean":"2024-09-20T22:57:52.717849+0000","last_became_active":"2024-09-20T22:57:49.408666+0000","last_became_peered":"2024-09-20T22:57:49.408666+0000","last_unstale":"2024-09-20T22:57:52.717849+0000","last_undegraded":"2024-09-20T22:57:52.717849+0000","last_fullsized":"2024-09-20T22:57:52.717849+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'24","last_scrub_stamp":"2024-09-20T22:53:33.089580+0000","last_deep_scrub":"470'16","last_deep_scrub_stamp":"2024-09-20T22:49:10.659868+0000","last_clean_scrub_stamp":"2024-09-20T22:53:33.089580+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:55.629882+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":1940,"reported_epoch":993,"state":"active+clean","last_fresh":"2024-09-20T22:57:58.421792+0000","last_change":"2024-09-20T22:57:49.405502+0000","last_active":"2024-09-20T22:57:58.421792+0000","last_peered":"2024-09-20T22:57:58.421792+0000","last_clean":"2024-09-20T22:57:58.421792+0000","last_became_active":"2024-09-20T22:57:49.405256+0000","last_became_peered":"2024-09-20T22:57:49.405256+0000","last_unstale":"2024-09-20T22:57:58.421792+0000","last_undegraded":"2024-09-20T22:57:58.421792+0000","last_fullsized":"2024-09-20T22:57:58.421792+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:55:54.521725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:45:24.765414+0000","last_clean_scrub_stamp":"2024-09-20T22:55:54.521725+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":938,"reported_epoch":993,"state":"active+clean","last_fresh":"2024-09-20T22:57:58.419699+0000","last_change":"2024-09-20T22:57:36.239427+0000","last_active":"2024-09-20T22:57:58.419699+0000","last_peered":"2024-09-20T22:57:58.419699+0000","last_clean":"2024-09-20T22:57:58.419699+0000","last_became_active":"2024-09-20T22:56:08.355950+0000","last_became_peered":"2024-09-20T22:56:08.355950+0000","last_unstale":"2024-09-20T22:57:58.419699+0000","last_undegraded":"2024-09-20T22:57:58.419699+0000","last_fullsized":"2024-09-20T22:57:58.419699+0000","mapping_epoch":887,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:57:36.239352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:50.174861+0000","last_clean_scrub_stamp":"2024-09-20T22:57:36.239352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:38.372074+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":1740,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-20T22:57:52.718548+0000","last_change":"2024-09-20T22:57:49.419702+0000","last_active":"2024-09-20T22:57:52.718548+0000","last_peered":"2024-09-20T22:57:52.718548+0000","last_clean":"2024-09-20T22:57:52.718548+0000","last_became_active":"2024-09-20T22:57:49.410970+0000","last_became_peered":"2024-09-20T22:57:49.410970+0000","last_unstale":"2024-09-20T22:57:52.718548+0000","last_undegraded":"2024-09-20T22:57:52.718548+0000","last_fullsized":"2024-09-20T22:57:52.718548+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:54:22.276321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:54:22.276321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:55.630143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"988'504","reported_seq":4891,"reported_epoch":993,"state":"active+clean","last_fresh":"2024-09-20T22:57:58.369822+0000","last_change":"2024-09-20T22:57:48.420084+0000","last_active":"2024-09-20T22:57:58.369822+0000","last_peered":"2024-09-20T22:57:58.369822+0000","last_clean":"2024-09-20T22:57:58.369822+0000","last_became_active":"2024-09-20T22:57:03.429099+0000","last_became_peered":"2024-09-20T22:57:03.429099+0000","last_unstale":"2024-09-20T22:57:58.369822+0000","last_undegraded":"2024-09-20T22:57:58.369822+0000","last_fullsized":"2024-09-20T22:57:58.369822+0000","mapping_epoch":942,"log_start":"456'129","ondisk_log_start":"456'129","created":932,"last_epoch_clean":943,"parent":"0.0","parent_split_bits":5,"last_scrub":"944'488","last_scrub_stamp":"2024-09-20T22:57:04.743360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:57:04.743360+0000","objects_scrubbed":9,"log_size":375,"log_dups_size":149,"ondisk_log_size":375,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:31.671359+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.051105540999999997,"stat_sum":{"num_bytes":18736221,"num_objects":8,"num_object_clones":7,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":19,"num_read_kb":37757,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"180","length":"1"},{"start":"18d","length":"1"},{"start":"190","length":"1"},{"start":"193","length":"1"},{"start":"196","length":"2"},{"start":"19a","length":"1"},{"start":"19c","length":"2"},{"start":"1a3","length":"2"},{"start":"1a6","length":"1"},{"start":"1aa","length":"2"},{"start":"1af","length":"1"},{"start":"1b5","length":"1"},{"start":"1b7","length":"1"},{"start":"1ba","length":"1"},{"start":"1bd","length":"2"},{"start":"1c0","length":"2"},{"start":"1c5","length":"1"},{"start":"1c7","length":"1"}]},{"pgid":"4.10","version":"625'8","reported_seq":1812,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-20T22:57:52.718020+0000","last_change":"2024-09-20T22:57:49.412934+0000","last_active":"2024-09-20T22:57:52.718020+0000","last_peered":"2024-09-20T22:57:52.718020+0000","last_clean":"2024-09-20T22:57:52.718020+0000","last_became_active":"2024-09-20T22:57:49.401068+0000","last_became_peered":"2024-09-20T22:57:49.401068+0000","last_unstale":"2024-09-20T22:57:52.718020+0000","last_undegraded":"2024-09-20T22:57:52.718020+0000","last_fullsized":"2024-09-20T22:57:52.718020+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'8","last_scrub_stamp":"2024-09-20T22:56:42.475598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:56:42.475598+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:55.630931+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":1264,"reported_epoch":993,"state":"active+clean","last_fresh":"2024-09-20T22:57:58.372101+0000","last_change":"2024-09-20T22:57:49.415511+0000","last_active":"2024-09-20T22:57:58.372101+0000","last_peered":"2024-09-20T22:57:58.372101+0000","last_clean":"2024-09-20T22:57:58.372101+0000","last_became_active":"2024-09-20T22:57:49.412134+0000","last_became_peered":"2024-09-20T22:57:49.412134+0000","last_unstale":"2024-09-20T22:57:58.372101+0000","last_undegraded":"2024-09-20T22:57:58.372101+0000","last_fullsized":"2024-09-20T22:57:58.372101+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:55:11.688508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:55:11.688508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":1956,"reported_epoch":993,"state":"active+clean","last_fresh":"2024-09-20T22:57:58.420982+0000","last_change":"2024-09-20T22:56:39.112737+0000","last_active":"2024-09-20T22:57:58.420982+0000","last_peered":"2024-09-20T22:57:58.420982+0000","last_clean":"2024-09-20T22:57:58.420982+0000","last_became_active":"2024-09-20T22:56:08.357087+0000","last_became_peered":"2024-09-20T22:56:08.357087+0000","last_unstale":"2024-09-20T22:57:58.420982+0000","last_undegraded":"2024-09-20T22:57:58.420982+0000","last_fullsized":"2024-09-20T22:57:58.420982+0000","mapping_epoch":887,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:56:39.112639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:56:39.112639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":933,"reported_epoch":993,"state":"active+clean","last_fresh":"2024-09-20T22:57:58.419956+0000","last_change":"2024-09-20T22:57:49.398452+0000","last_active":"2024-09-20T22:57:58.419956+0000","last_peered":"2024-09-20T22:57:58.419956+0000","last_clean":"2024-09-20T22:57:58.419956+0000","last_became_active":"2024-09-20T22:57:49.398162+0000","last_became_peered":"2024-09-20T22:57:49.398162+0000","last_unstale":"2024-09-20T22:57:58.419956+0000","last_undegraded":"2024-09-20T22:57:58.419956+0000","last_fullsized":"2024-09-20T22:57:58.419956+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:56:40.102311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:55:08.681729+0000","last_clean_scrub_stamp":"2024-09-20T22:56:40.102311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:08.020100+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":458,"reported_epoch":993,"state":"active+clean","last_fresh":"2024-09-20T22:57:58.370995+0000","last_change":"2024-09-20T22:57:09.252912+0000","last_active":"2024-09-20T22:57:58.370995+0000","last_peered":"2024-09-20T22:57:58.370995+0000","last_clean":"2024-09-20T22:57:58.370995+0000","last_became_active":"2024-09-20T22:54:17.348206+0000","last_became_peered":"2024-09-20T22:54:17.348206+0000","last_unstale":"2024-09-20T22:57:58.370995+0000","last_undegraded":"2024-09-20T22:57:58.370995+0000","last_fullsized":"2024-09-20T22:57:58.370995+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:57:09.252832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:57:09.252832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:21.209966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":1770,"reported_epoch":992,"state":"active+clean","last_fresh":"2024-09-20T22:57:52.718044+0000","last_change":"2024-09-20T22:57:49.413803+0000","last_active":"2024-09-20T22:57:52.718044+0000","last_peered":"2024-09-20T22:57:52.718044+0000","last_clean":"2024-09-20T22:57:52.718044+0000","last_became_active":"2024-09-20T22:57:49.407194+0000","last_became_peered":"2024-09-20T22:57:49.407194+0000","last_unstale":"2024-09-20T22:57:52.718044+0000","last_undegraded":"2024-09-20T22:57:52.718044+0000","last_fullsized":"2024-09-20T22:57:52.718044+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:31.274390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:53:31.274390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:57:55.629932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"625'19","reported_seq":2001,"reported_epoch":993,"state":"active+clean","last_fresh":"2024-09-20T22:57:58.422301+0000","last_change":"2024-09-20T22:56:18.981584+0000","last_active":"2024-09-20T22:57:58.422301+0000","last_peered":"2024-09-20T22:57:58.422301+0000","last_clean":"2024-09-20T22:57:58.422301+0000","last_became_active":"2024-09-20T22:54:17.342236+0000","last_became_peered":"2024-09-20T22:54:17.342236+0000","last_unstale":"2024-09-20T22:57:58.422301+0000","last_undegraded":"2024-09-20T22:57:58.422301+0000","last_fullsized":"2024-09-20T22:57:58.422301+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'19","last_scrub_stamp":"2024-09-20T22:56:18.981492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:56:18.981492+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":1262,"reported_epoch":993,"state":"active+clean","last_fresh":"2024-09-20T22:57:58.371058+0000","last_change":"2024-09-20T22:57:08.720376+0000","last_active":"2024-09-20T22:57:58.371058+0000","last_peered":"2024-09-20T22:57:58.371058+0000","last_clean":"2024-09-20T22:57:58.371058+0000","last_became_active":"2024-09-20T22:56:08.391532+0000","last_became_peered":"2024-09-20T22:56:08.391532+0000","last_unstale":"2024-09-20T22:57:58.371058+0000","last_undegraded":"2024-09-20T22:57:58.371058+0000","last_fullsized":"2024-09-20T22:57:58.371058+0000","mapping_epoch":887,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:57:08.720315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:54:43.813540+0000","last_clean_scrub_stamp":"2024-09-20T22:57:08.720315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:09.299138+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":26,"stat_sum":{"num_bytes":803080872,"num_objects":344,"num_object_clones":296,"num_object_copies":1032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":344,"num_whiteouts":10,"num_read":3900,"num_read_kb":4069970,"num_write":2637,"num_write_kb":1051239,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":679,"num_bytes_recovered":411179007,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2069483520,"data_stored":2670190378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":25811,"ondisk_log_size":25811,"up":78,"acting":78,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1097,"num_read_kb":854,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":437,"num_bytes_recovered":10204,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":462848,"data_stored":12360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":770,"ondisk_log_size":770,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":1034,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":4931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":855,"num_read_kb":850,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":1594,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":385,"ondisk_log_size":385,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":0,"seq":253403070664,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430414,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":421236,"kb_used_data":335972,"kb_used_omap":1,"kb_used_meta":85246,"kb_avail":93323916,"statfs":{"total":95995035648,"available":95563689984,"internally_reserved":0,"allocated":344035328,"data_stored":439748734,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":87292358},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798692051,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":366060,"kb_used_data":294620,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93379092,"statfs":{"total":95995035648,"available":95620190208,"internally_reserved":0,"allocated":301690880,"data_stored":351426600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986392,"num_pgs":73,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":169904,"kb_used_data":124832,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":93575248,"statfs":{"total":95995035648,"available":95821053952,"internally_reserved":0,"allocated":127827968,"data_stored":165561056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":988,"seq":4243427688451,"num_pgs":79,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":380088,"kb_used_data":345320,"kb_used_omap":148,"kb_used_meta":34603,"kb_avail":93365064,"statfs":{"total":95995035648,"available":95605825536,"internally_reserved":0,"allocated":353607680,"data_stored":460842230,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":151956,"internal_metadata":35434092},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411555,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":448392,"kb_used_data":363128,"kb_used_omap":1,"kb_used_meta":85246,"kb_avail":93296760,"statfs":{"total":95995035648,"available":95535882240,"internally_reserved":0,"allocated":371843072,"data_stored":442887419,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":87292357},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575080,"num_pgs":79,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":270140,"kb_used_data":194536,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93475012,"statfs":{"total":95995035648,"available":95718412288,"internally_reserved":0,"allocated":199204864,"data_stored":251112834,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738605,"num_pgs":73,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":359784,"kb_used_data":274516,"kb_used_omap":1,"kb_used_meta":85246,"kb_avail":93385368,"statfs":{"total":95995035648,"available":95626616832,"internally_reserved":0,"allocated":281104384,"data_stored":362026401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":87292357},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":271130624,"data_stored":355911924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":189190144,"data_stored":244997782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":361631744,"data_stored":436518233,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":343044096,"data_stored":454143932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":117862400,"data_stored":159454854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":291127296,"data_stored":344728379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":333766656,"data_stored":433314464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":161730560,"data_stored":241120810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-20T22:58:03.784 INFO:tasks.thrashosds.thrasher:Setting 4.3 to [5, 2, 0] 2024-09-20T22:58:03.784 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '4.3', '5', '2', '0'] 2024-09-20T22:58:03.784 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd pg-upmap 4.3 5 2 0 2024-09-20T22:58:03.839 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:58:03.840 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:58:03.840 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:58:03.865 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:58:03.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:58:03.866 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:58:03.967 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:58:03.992 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:58:03.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:58:03.993 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:58:03.996 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:03 smithi084 ceph-mon[40024]: 5.4 scrub starts 2024-09-20T22:58:03.996 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:03 smithi084 ceph-mon[40024]: 5.4 scrub ok 2024-09-20T22:58:03.997 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:03 smithi084 ceph-mon[40024]: 3.17 deep-scrub starts 2024-09-20T22:58:03.997 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:03 smithi084 ceph-mon[40024]: 3.17 deep-scrub ok 2024-09-20T22:58:03.997 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:03 smithi084 ceph-mon[40024]: osdmap e994: 8 total, 8 up, 7 in 2024-09-20T22:58:03.997 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:03 smithi084 ceph-mon[29774]: 5.4 scrub starts 2024-09-20T22:58:03.997 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:03 smithi084 ceph-mon[29774]: 5.4 scrub ok 2024-09-20T22:58:03.997 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:03 smithi084 ceph-mon[29774]: 3.17 deep-scrub starts 2024-09-20T22:58:03.997 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:03 smithi084 ceph-mon[29774]: 3.17 deep-scrub ok 2024-09-20T22:58:03.997 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:03 smithi084 ceph-mon[29774]: osdmap e994: 8 total, 8 up, 7 in 2024-09-20T22:58:04.094 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:58:04.119 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:58:04.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:58:04.120 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:58:04.221 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:58:04.246 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:58:04.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:58:04.247 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:58:04.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:03 smithi154 ceph-mon[33264]: 5.4 scrub starts 2024-09-20T22:58:04.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:03 smithi154 ceph-mon[33264]: 5.4 scrub ok 2024-09-20T22:58:04.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:03 smithi154 ceph-mon[33264]: 3.17 deep-scrub starts 2024-09-20T22:58:04.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:03 smithi154 ceph-mon[33264]: 3.17 deep-scrub ok 2024-09-20T22:58:04.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:03 smithi154 ceph-mon[33264]: osdmap e994: 8 total, 8 up, 7 in 2024-09-20T22:58:04.348 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:58:04.368 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:58:04.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:58:04.369 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:58:04.470 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:58:04.491 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:58:04.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:58:04.492 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:58:04.518 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-20T22:58:04.518 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:58:04.592 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:58:04.618 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:58:04.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:58:04.619 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:58:04.720 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:58:04.742 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:58:04.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:58:04.743 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:58:04.844 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:58:04.865 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:58:04.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:58:04.866 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:58:04.967 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:58:04.993 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:58:04.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:58:04.994 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:58:05.046 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:04 smithi084 ceph-mon[29774]: 1.6 scrub starts 2024-09-20T22:58:05.046 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:04 smithi084 ceph-mon[29774]: 1.6 scrub ok 2024-09-20T22:58:05.046 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:04 smithi084 ceph-mon[29774]: 4.12 scrub starts 2024-09-20T22:58:05.046 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:04 smithi084 ceph-mon[29774]: 4.12 scrub ok 2024-09-20T22:58:05.046 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:04 smithi084 ceph-mon[29774]: 5.13 scrub starts 2024-09-20T22:58:05.046 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:04 smithi084 ceph-mon[29774]: 5.13 scrub ok 2024-09-20T22:58:05.047 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:04 smithi084 ceph-mon[29774]: pgmap v1398: 168 pgs: 1 active+clean+snaptrim, 167 active+clean; 776 MiB data, 2.3 GiB used, 624 GiB / 626 GiB avail 2024-09-20T22:58:05.047 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:04 smithi084 ceph-mon[40024]: 1.6 scrub starts 2024-09-20T22:58:05.047 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:04 smithi084 ceph-mon[40024]: 1.6 scrub ok 2024-09-20T22:58:05.047 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:04 smithi084 ceph-mon[40024]: 4.12 scrub starts 2024-09-20T22:58:05.047 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:04 smithi084 ceph-mon[40024]: 4.12 scrub ok 2024-09-20T22:58:05.047 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:04 smithi084 ceph-mon[40024]: 5.13 scrub starts 2024-09-20T22:58:05.047 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:04 smithi084 ceph-mon[40024]: 5.13 scrub ok 2024-09-20T22:58:05.047 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:04 smithi084 ceph-mon[40024]: pgmap v1398: 168 pgs: 1 active+clean+snaptrim, 167 active+clean; 776 MiB data, 2.3 GiB used, 624 GiB / 626 GiB avail 2024-09-20T22:58:05.052 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:58:05.094 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:58:05.115 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:58:05.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:58:05.116 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:58:05.217 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:58:05.242 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:58:05.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:58:05.243 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:58:05.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:04 smithi154 ceph-mon[33264]: 1.6 scrub starts 2024-09-20T22:58:05.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:04 smithi154 ceph-mon[33264]: 1.6 scrub ok 2024-09-20T22:58:05.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:04 smithi154 ceph-mon[33264]: 4.12 scrub starts 2024-09-20T22:58:05.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:04 smithi154 ceph-mon[33264]: 4.12 scrub ok 2024-09-20T22:58:05.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:04 smithi154 ceph-mon[33264]: 5.13 scrub starts 2024-09-20T22:58:05.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:04 smithi154 ceph-mon[33264]: 5.13 scrub ok 2024-09-20T22:58:05.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:04 smithi154 ceph-mon[33264]: pgmap v1398: 168 pgs: 1 active+clean+snaptrim, 167 active+clean; 776 MiB data, 2.3 GiB used, 624 GiB / 626 GiB avail 2024-09-20T22:58:05.344 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:58:05.369 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:58:05.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:58:05.370 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:58:05.470 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:58:05.494 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:58:05.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:58:05.496 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:58:05.567 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-20T22:58:05.596 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:58:05.631 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:58:05.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:58:05.632 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:58:05.733 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:58:05.758 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:58:05.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:58:05.759 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:58:05.779 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is set 2024-09-20T22:58:05.793 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:05 smithi084 ceph-mon[40024]: 4.1f scrub starts 2024-09-20T22:58:05.793 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:05 smithi084 ceph-mon[40024]: 4.1f scrub ok 2024-09-20T22:58:05.793 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:05 smithi084 ceph-mon[40024]: 3.1 scrub starts 2024-09-20T22:58:05.793 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:05 smithi084 ceph-mon[40024]: 3.1 scrub ok 2024-09-20T22:58:05.793 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:05 smithi084 ceph-mon[40024]: 4.11 scrub starts 2024-09-20T22:58:05.794 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:05 smithi084 ceph-mon[40024]: 4.11 scrub ok 2024-09-20T22:58:05.794 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:05 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/3224111069' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-20T22:58:05.794 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:05 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-20T22:58:05.794 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:05 smithi084 ceph-mon[29774]: 4.1f scrub starts 2024-09-20T22:58:05.794 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:05 smithi084 ceph-mon[29774]: 4.1f scrub ok 2024-09-20T22:58:05.794 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:05 smithi084 ceph-mon[29774]: 3.1 scrub starts 2024-09-20T22:58:05.794 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:05 smithi084 ceph-mon[29774]: 3.1 scrub ok 2024-09-20T22:58:05.794 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:05 smithi084 ceph-mon[29774]: 4.11 scrub starts 2024-09-20T22:58:05.794 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:05 smithi084 ceph-mon[29774]: 4.11 scrub ok 2024-09-20T22:58:05.794 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:05 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/3224111069' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-20T22:58:05.794 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:05 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-20T22:58:05.860 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:58:05.882 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:58:05.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:58:05.883 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:58:05.984 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:58:06.010 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:58:06.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:58:06.011 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:58:06.112 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:58:06.133 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:58:06.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:58:06.134 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:58:06.135 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph pg dump --format=json 2024-09-20T22:58:06.236 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:58:06.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:05 smithi154 ceph-mon[33264]: 4.1f scrub starts 2024-09-20T22:58:06.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:05 smithi154 ceph-mon[33264]: 4.1f scrub ok 2024-09-20T22:58:06.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:05 smithi154 ceph-mon[33264]: 3.1 scrub starts 2024-09-20T22:58:06.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:05 smithi154 ceph-mon[33264]: 3.1 scrub ok 2024-09-20T22:58:06.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:05 smithi154 ceph-mon[33264]: 4.11 scrub starts 2024-09-20T22:58:06.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:05 smithi154 ceph-mon[33264]: 4.11 scrub ok 2024-09-20T22:58:06.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:05 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/3224111069' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-20T22:58:06.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:05 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-20T22:58:06.262 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:58:06.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:58:06.263 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:58:06.364 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:58:06.385 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:58:06.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:58:06.386 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:58:06.486 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:58:06.507 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:58:06.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:58:06.508 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:58:06.609 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:58:06.631 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:58:06.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:58:06.632 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:58:06.733 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:58:06.758 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:58:06.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:58:06.759 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:58:06.860 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:58:06.864 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:06 smithi084 ceph-mon[29774]: 6.17 scrub starts 2024-09-20T22:58:06.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:06 smithi084 ceph-mon[29774]: 6.17 scrub ok 2024-09-20T22:58:06.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:06 smithi084 ceph-mon[29774]: 3.10 scrub starts 2024-09-20T22:58:06.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:06 smithi084 ceph-mon[29774]: 3.10 scrub ok 2024-09-20T22:58:06.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:06 smithi084 ceph-mon[29774]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:58:06.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:06 smithi084 ceph-mon[29774]: 7.0 scrub starts 2024-09-20T22:58:06.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:06 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T22:58:06.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:06 smithi084 ceph-mon[29774]: 7.0 scrub ok 2024-09-20T22:58:06.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:06 smithi084 ceph-mon[29774]: osdmap e995: 8 total, 8 up, 7 in 2024-09-20T22:58:06.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:06 smithi084 ceph-mon[29774]: pgmap v1400: 168 pgs: 1 active+clean+snaptrim, 167 active+clean; 776 MiB data, 2.3 GiB used, 624 GiB / 626 GiB avail 2024-09-20T22:58:06.866 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:06 smithi084 ceph-mon[40024]: 6.17 scrub starts 2024-09-20T22:58:06.866 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:06 smithi084 ceph-mon[40024]: 6.17 scrub ok 2024-09-20T22:58:06.866 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:06 smithi084 ceph-mon[40024]: 3.10 scrub starts 2024-09-20T22:58:06.866 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:06 smithi084 ceph-mon[40024]: 3.10 scrub ok 2024-09-20T22:58:06.866 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:06 smithi084 ceph-mon[40024]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:58:06.866 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:06 smithi084 ceph-mon[40024]: 7.0 scrub starts 2024-09-20T22:58:06.866 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:06 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T22:58:06.866 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:06 smithi084 ceph-mon[40024]: 7.0 scrub ok 2024-09-20T22:58:06.866 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:06 smithi084 ceph-mon[40024]: osdmap e995: 8 total, 8 up, 7 in 2024-09-20T22:58:06.867 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:06 smithi084 ceph-mon[40024]: pgmap v1400: 168 pgs: 1 active+clean+snaptrim, 167 active+clean; 776 MiB data, 2.3 GiB used, 624 GiB / 626 GiB avail 2024-09-20T22:58:06.881 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:58:06.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:58:06.882 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:58:06.983 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:58:07.004 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:58:07.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:58:07.005 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:58:07.044 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:58:07.107 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:58:07.128 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:58:07.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:58:07.128 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:58:07.229 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:58:07.252 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:58:07.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:58:07.253 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:58:07.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:06 smithi154 ceph-mon[33264]: 6.17 scrub starts 2024-09-20T22:58:07.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:06 smithi154 ceph-mon[33264]: 6.17 scrub ok 2024-09-20T22:58:07.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:06 smithi154 ceph-mon[33264]: 3.10 scrub starts 2024-09-20T22:58:07.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:06 smithi154 ceph-mon[33264]: 3.10 scrub ok 2024-09-20T22:58:07.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:06 smithi154 ceph-mon[33264]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:58:07.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:06 smithi154 ceph-mon[33264]: 7.0 scrub starts 2024-09-20T22:58:07.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:06 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-20T22:58:07.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:06 smithi154 ceph-mon[33264]: 7.0 scrub ok 2024-09-20T22:58:07.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:06 smithi154 ceph-mon[33264]: osdmap e995: 8 total, 8 up, 7 in 2024-09-20T22:58:07.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:06 smithi154 ceph-mon[33264]: pgmap v1400: 168 pgs: 1 active+clean+snaptrim, 167 active+clean; 776 MiB data, 2.3 GiB used, 624 GiB / 626 GiB avail 2024-09-20T22:58:07.353 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:58:07.375 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:58:07.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:58:07.376 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:58:07.476 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:58:07.482 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-20T22:58:07.482 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:58:07.498 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:58:07.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:58:07.499 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:58:07.600 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:58:07.621 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:58:07.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:58:07.622 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:58:07.723 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:58:07.744 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:58:07.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:58:07.745 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:58:07.784 INFO:teuthology.orchestra.run.smithi084.stderr:set 4.3 pg_upmap mapping to [5,2,0] 2024-09-20T22:58:07.846 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:58:07.868 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:58:07.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:58:07.869 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:58:07.970 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:58:07.991 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:58:07.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:58:07.992 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:58:08.093 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:58:08.114 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:58:08.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:07 smithi084 ceph-mon[40024]: 5.d deep-scrub starts 2024-09-20T22:58:08.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:07 smithi084 ceph-mon[40024]: 5.d deep-scrub ok 2024-09-20T22:58:08.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:07 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/2683938757' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "4.3", "id": [5, 2, 0]}]: dispatch 2024-09-20T22:58:08.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:07 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:58:08.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:07 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:58:08.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:07 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:58:08.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:07 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:58:08.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:07 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:58:08.116 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:07 smithi084 ceph-mon[29774]: 5.d deep-scrub starts 2024-09-20T22:58:08.116 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:07 smithi084 ceph-mon[29774]: 5.d deep-scrub ok 2024-09-20T22:58:08.116 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:07 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/2683938757' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "4.3", "id": [5, 2, 0]}]: dispatch 2024-09-20T22:58:08.116 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:07 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:58:08.116 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:07 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:58:08.116 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:07 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:58:08.116 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:07 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:58:08.116 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:07 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:58:08.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:58:08.117 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:58:08.217 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:58:08.239 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:58:08.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:58:08.240 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:58:08.252 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.0 dump_blocked_ops 2024-09-20T22:58:08.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:07 smithi154 ceph-mon[33264]: 5.d deep-scrub starts 2024-09-20T22:58:08.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:07 smithi154 ceph-mon[33264]: 5.d deep-scrub ok 2024-09-20T22:58:08.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:07 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/2683938757' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "4.3", "id": [5, 2, 0]}]: dispatch 2024-09-20T22:58:08.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:07 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:58:08.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:07 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:58:08.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:07 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:58:08.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:07 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:58:08.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:07 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:58:08.341 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:58:08.363 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:58:08.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:58:08.364 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:58:08.465 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:58:08.486 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:58:08.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:58:08.487 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:58:08.554 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:58:08 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: ::ffff:172.21.15.154 - - [20/Sep/2024:22:58:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-20T22:58:08.589 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:58:08.610 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:58:08.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:58:08.611 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:58:08.637 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-20T22:58:08.712 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:58:08.734 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:58:08.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:58:08.735 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:58:08.836 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:58:08.858 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:58:08.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:58:08.859 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:58:08.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:08 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/2683938757' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "4.3", "id": [5, 2, 0]}]': finished 2024-09-20T22:58:08.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:08 smithi084 ceph-mon[40024]: osdmap e996: 8 total, 8 up, 7 in 2024-09-20T22:58:08.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:08 smithi084 ceph-mon[40024]: pgmap v1402: 168 pgs: 1 active+clean+snaptrim, 167 active+clean; 776 MiB data, 2.3 GiB used, 624 GiB / 626 GiB avail 2024-09-20T22:58:08.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:08 smithi084 ceph-mon[40024]: osdmap e997: 8 total, 8 up, 7 in 2024-09-20T22:58:08.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:08 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:58:08.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:08 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:58:08.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:08 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/2683938757' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "4.3", "id": [5, 2, 0]}]': finished 2024-09-20T22:58:08.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:08 smithi084 ceph-mon[29774]: osdmap e996: 8 total, 8 up, 7 in 2024-09-20T22:58:08.866 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:08 smithi084 ceph-mon[29774]: pgmap v1402: 168 pgs: 1 active+clean+snaptrim, 167 active+clean; 776 MiB data, 2.3 GiB used, 624 GiB / 626 GiB avail 2024-09-20T22:58:08.866 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:08 smithi084 ceph-mon[29774]: osdmap e997: 8 total, 8 up, 7 in 2024-09-20T22:58:08.866 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:08 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:58:08.866 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:08 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:58:08.959 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:58:08.980 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:58:08.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:58:08.981 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:58:09.020 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:58:09.020 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:58:09.082 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:58:09.103 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:58:09.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:58:09.104 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:58:09.204 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:58:09.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:58:09.226 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:58:09.227 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:58:09.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:08 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/2683938757' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "4.3", "id": [5, 2, 0]}]': finished 2024-09-20T22:58:09.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:08 smithi154 ceph-mon[33264]: osdmap e996: 8 total, 8 up, 7 in 2024-09-20T22:58:09.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:08 smithi154 ceph-mon[33264]: pgmap v1402: 168 pgs: 1 active+clean+snaptrim, 167 active+clean; 776 MiB data, 2.3 GiB used, 624 GiB / 626 GiB avail 2024-09-20T22:58:09.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:08 smithi154 ceph-mon[33264]: osdmap e997: 8 total, 8 up, 7 in 2024-09-20T22:58:09.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:08 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:58:09.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:08 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:58:09.327 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:58:09.349 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:58:09.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:58:09.350 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:58:09.451 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:58:09.473 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:58:09.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:58:09.475 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:58:09.576 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:58:09.600 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:58:09.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:58:09.601 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:58:09.702 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:58:09.724 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:58:09.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:58:09.725 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:58:09.778 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:58:09.778 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:58:09.793 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:58:09.793 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:58:09.812 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:58:09.812 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:58:09.826 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:58:09.834 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-20T22:58:09.834 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T22:58:09.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:58:09.850 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:58:09.850 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:58:09.851 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:58:09.851 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:58:09.866 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:58:09.866 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:58:09.883 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:58:09.883 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:58:09.899 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:58:09.899 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:58:09.951 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:58:09.974 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:58:09.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:58:09.975 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:58:10.076 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:58:10.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:58:10.097 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:58:10.097 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:58:10.114 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:09 smithi084 ceph-mon[29774]: purged_snaps scrub starts 2024-09-20T22:58:10.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:09 smithi084 ceph-mon[29774]: purged_snaps scrub ok 2024-09-20T22:58:10.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:09 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/3947275815' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:58:10.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:09 smithi084 ceph-mon[40024]: purged_snaps scrub starts 2024-09-20T22:58:10.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:09 smithi084 ceph-mon[40024]: purged_snaps scrub ok 2024-09-20T22:58:10.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:09 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/3947275815' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:58:10.195 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-20T22:58:10.198 INFO:teuthology.orchestra.run.smithi084.stderr:dumped all 2024-09-20T22:58:10.198 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:58:10.221 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:58:10.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:58:10.222 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:58:10.226 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:58:10.254 INFO:journalctl@ceph.iscsi.iscsi.a.smithi154.stdout:Sep 20 22:58:09 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-iscsi-iscsi-a[59730]: debug there is no tcmu-runner data available 2024-09-20T22:58:10.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:09 smithi154 ceph-mon[33264]: purged_snaps scrub starts 2024-09-20T22:58:10.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:09 smithi154 ceph-mon[33264]: purged_snaps scrub ok 2024-09-20T22:58:10.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:09 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/3947275815' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:58:10.323 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:58:10.345 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:58:10.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:58:10.346 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:58:10.446 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:58:10.467 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:58:10.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:58:10.468 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:58:10.569 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:58:10.592 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:58:10.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:58:10.593 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:58:10.693 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:58:10.714 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:58:10.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:58:10.715 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:58:10.816 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:58:10.837 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:58:10.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:58:10.839 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:58:10.940 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:58:10.962 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:58:10.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:58:10.963 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:58:11.064 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:58:11.086 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:58:11.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:58:11.088 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:58:11.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:10 smithi084 ceph-mon[40024]: osdmap e998: 8 total, 8 up, 7 in 2024-09-20T22:58:11.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:10 smithi084 ceph-mon[40024]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:58:11.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:10 smithi084 ceph-mon[40024]: pgmap v1405: 168 pgs: 1 activating+degraded, 167 active+clean; 782 MiB data, 2.3 GiB used, 624 GiB / 626 GiB avail; 8/1650 objects degraded (0.485%) 2024-09-20T22:58:11.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:10 smithi084 ceph-mon[40024]: from='client.17646 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T22:58:11.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:10 smithi084 ceph-mon[29774]: osdmap e998: 8 total, 8 up, 7 in 2024-09-20T22:58:11.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:10 smithi084 ceph-mon[29774]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:58:11.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:10 smithi084 ceph-mon[29774]: pgmap v1405: 168 pgs: 1 activating+degraded, 167 active+clean; 782 MiB data, 2.3 GiB used, 624 GiB / 626 GiB avail; 8/1650 objects degraded (0.485%) 2024-09-20T22:58:11.116 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:10 smithi084 ceph-mon[29774]: from='client.17646 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T22:58:11.189 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:58:11.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:58:11.214 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:58:11.215 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:58:11.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:10 smithi154 ceph-mon[33264]: osdmap e998: 8 total, 8 up, 7 in 2024-09-20T22:58:11.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:10 smithi154 ceph-mon[33264]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:58:11.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:10 smithi154 ceph-mon[33264]: pgmap v1405: 168 pgs: 1 activating+degraded, 167 active+clean; 782 MiB data, 2.3 GiB used, 624 GiB / 626 GiB avail; 8/1650 objects degraded (0.485%) 2024-09-20T22:58:11.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:10 smithi154 ceph-mon[33264]: from='client.17646 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T22:58:11.315 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:58:11.336 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:58:11.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:58:11.337 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:58:11.438 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:58:11.460 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:58:11.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:58:11.461 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:58:11.562 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:58:11.585 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:58:11.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:58:11.585 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:58:11.602 INFO:teuthology.orchestra.run.smithi084.stdout:{"pg_ready":true,"pg_map":{"version":1405,"stamp":"2024-09-20T22:58:09.903851+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":819593922,"num_objects":550,"num_object_clones":301,"num_object_copies":1650,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":550,"num_whiteouts":10,"num_read":5498,"num_read_kb":3692236,"num_write":3030,"num_write_kb":919382,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1078,"num_bytes_recovered":294523159,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":27094,"ondisk_log_size":27094,"up":504,"acting":504,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":504,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":2415956,"kb_used_data":1933236,"kb_used_omap":157,"kb_used_meta":482402,"kb_avail":653800108,"statfs":{"total":671965249536,"available":669491310592,"internally_reserved":0,"allocated":1979633664,"data_stored":2473795444,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":161525,"internal_metadata":493979915},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":5495011,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":-169,"num_read_kb":-171898,"num_write":-148,"num_write_kb":-53734,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-42,"num_bytes_recovered":-64008909,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.126909"},"pg_stats":[{"pgid":"3.1f","version":"86'2","reported_seq":1267,"reported_epoch":997,"state":"active+clean","last_fresh":"2024-09-20T22:58:08.375606+0000","last_change":"2024-09-20T22:56:47.723346+0000","last_active":"2024-09-20T22:58:08.375606+0000","last_peered":"2024-09-20T22:58:08.375606+0000","last_clean":"2024-09-20T22:58:08.375606+0000","last_became_active":"2024-09-20T22:47:49.222114+0000","last_became_peered":"2024-09-20T22:47:49.222114+0000","last_unstale":"2024-09-20T22:58:08.375606+0000","last_undegraded":"2024-09-20T22:58:08.375606+0000","last_fullsized":"2024-09-20T22:58:08.375606+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'2","last_scrub_stamp":"2024-09-20T22:56:47.723216+0000","last_deep_scrub":"86'2","last_deep_scrub_stamp":"2024-09-20T22:56:47.723216+0000","last_clean_scrub_stamp":"2024-09-20T22:56:47.723216+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:12.539740+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":436,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"625'15","reported_seq":830,"reported_epoch":997,"state":"active+clean","last_fresh":"2024-09-20T22:58:08.372137+0000","last_change":"2024-09-20T22:56:14.882214+0000","last_active":"2024-09-20T22:58:08.372137+0000","last_peered":"2024-09-20T22:58:08.372137+0000","last_clean":"2024-09-20T22:58:08.372137+0000","last_became_active":"2024-09-20T22:55:17.341556+0000","last_became_peered":"2024-09-20T22:55:17.341556+0000","last_unstale":"2024-09-20T22:58:08.372137+0000","last_undegraded":"2024-09-20T22:58:08.372137+0000","last_fullsized":"2024-09-20T22:58:08.372137+0000","mapping_epoch":836,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":837,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'15","last_scrub_stamp":"2024-09-20T22:56:14.882123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:56:14.882123+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":464,"reported_epoch":997,"state":"active+clean","last_fresh":"2024-09-20T22:58:08.377951+0000","last_change":"2024-09-20T22:56:08.363427+0000","last_active":"2024-09-20T22:58:08.377951+0000","last_peered":"2024-09-20T22:58:08.377951+0000","last_clean":"2024-09-20T22:58:08.377951+0000","last_became_active":"2024-09-20T22:56:08.362405+0000","last_became_peered":"2024-09-20T22:56:08.362405+0000","last_unstale":"2024-09-20T22:58:08.377951+0000","last_undegraded":"2024-09-20T22:58:08.377951+0000","last_fullsized":"2024-09-20T22:58:08.377951+0000","mapping_epoch":887,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:55:34.919340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:55:34.919340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:11.406302+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":1123,"reported_epoch":997,"state":"active+clean","last_fresh":"2024-09-20T22:58:08.375898+0000","last_change":"2024-09-20T22:56:22.519967+0000","last_active":"2024-09-20T22:58:08.375898+0000","last_peered":"2024-09-20T22:58:08.375898+0000","last_clean":"2024-09-20T22:58:08.375898+0000","last_became_active":"2024-09-20T22:55:17.342068+0000","last_became_peered":"2024-09-20T22:55:17.342068+0000","last_unstale":"2024-09-20T22:58:08.375898+0000","last_undegraded":"2024-09-20T22:58:08.375898+0000","last_fullsized":"2024-09-20T22:58:08.375898+0000","mapping_epoch":836,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":837,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:56:22.519884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:56:22.519884+0000","last_clean_scrub_stamp":"2024-09-20T22:56:22.519884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":1798,"reported_epoch":995,"state":"active+clean","last_fresh":"2024-09-20T22:58:06.267311+0000","last_change":"2024-09-20T22:57:49.427504+0000","last_active":"2024-09-20T22:58:06.267311+0000","last_peered":"2024-09-20T22:58:06.267311+0000","last_clean":"2024-09-20T22:58:06.267311+0000","last_became_active":"2024-09-20T22:57:49.427228+0000","last_became_peered":"2024-09-20T22:57:49.427228+0000","last_unstale":"2024-09-20T22:58:06.267311+0000","last_undegraded":"2024-09-20T22:58:06.267311+0000","last_fullsized":"2024-09-20T22:58:06.267311+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:54:15.238344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:54:15.238344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":1797,"reported_epoch":995,"state":"active+clean","last_fresh":"2024-09-20T22:58:06.267732+0000","last_change":"2024-09-20T22:57:49.409395+0000","last_active":"2024-09-20T22:58:06.267732+0000","last_peered":"2024-09-20T22:58:06.267732+0000","last_clean":"2024-09-20T22:58:06.267732+0000","last_became_active":"2024-09-20T22:57:49.405928+0000","last_became_peered":"2024-09-20T22:57:49.405928+0000","last_unstale":"2024-09-20T22:58:06.267732+0000","last_undegraded":"2024-09-20T22:58:06.267732+0000","last_fullsized":"2024-09-20T22:58:06.267732+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:54:47.427927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:54:47.427927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"625'27","reported_seq":1875,"reported_epoch":995,"state":"active+clean","last_fresh":"2024-09-20T22:58:06.267128+0000","last_change":"2024-09-20T22:57:49.416557+0000","last_active":"2024-09-20T22:58:06.267128+0000","last_peered":"2024-09-20T22:58:06.267128+0000","last_clean":"2024-09-20T22:58:06.267128+0000","last_became_active":"2024-09-20T22:57:49.413932+0000","last_became_peered":"2024-09-20T22:57:49.413932+0000","last_unstale":"2024-09-20T22:58:06.267128+0000","last_undegraded":"2024-09-20T22:58:06.267128+0000","last_fullsized":"2024-09-20T22:58:06.267128+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'27","last_scrub_stamp":"2024-09-20T22:54:45.390169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:54:45.390169+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":1120,"reported_epoch":997,"state":"active+clean","last_fresh":"2024-09-20T22:58:08.372235+0000","last_change":"2024-09-20T22:57:13.194616+0000","last_active":"2024-09-20T22:58:08.372235+0000","last_peered":"2024-09-20T22:58:08.372235+0000","last_clean":"2024-09-20T22:58:08.372235+0000","last_became_active":"2024-09-20T22:55:17.341160+0000","last_became_peered":"2024-09-20T22:55:17.341160+0000","last_unstale":"2024-09-20T22:58:08.372235+0000","last_undegraded":"2024-09-20T22:58:08.372235+0000","last_fullsized":"2024-09-20T22:58:08.372235+0000","mapping_epoch":836,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":837,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:57:13.194499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:57:13.194499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:39.408547+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.19","version":"988'2004","reported_seq":6642,"reported_epoch":997,"state":"active+clean","last_fresh":"2024-09-20T22:58:08.372212+0000","last_change":"2024-09-20T22:57:48.440765+0000","last_active":"2024-09-20T22:58:08.372212+0000","last_peered":"2024-09-20T22:58:08.372212+0000","last_clean":"2024-09-20T22:58:08.372212+0000","last_became_active":"2024-09-20T22:56:58.364978+0000","last_became_peered":"2024-09-20T22:56:58.364978+0000","last_unstale":"2024-09-20T22:58:08.372212+0000","last_undegraded":"2024-09-20T22:58:08.372212+0000","last_fullsized":"2024-09-20T22:58:08.372212+0000","mapping_epoch":937,"log_start":"423'280","ondisk_log_start":"423'280","created":932,"last_epoch_clean":938,"parent":"0.0","parent_split_bits":5,"last_scrub":"942'1889","last_scrub_stamp":"2024-09-20T22:57:04.163707+0000","last_deep_scrub":"590'828","last_deep_scrub_stamp":"2024-09-20T22:51:10.909364+0000","last_clean_scrub_stamp":"2024-09-20T22:57:04.163707+0000","objects_scrubbed":23,"log_size":1724,"log_dups_size":649,"ondisk_log_size":1724,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:27.711921+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.070243691999999996,"stat_sum":{"num_bytes":56543939,"num_objects":23,"num_object_clones":20,"num_object_copies":69,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":19,"num_read_kb":6766,"num_write":21,"num_write_kb":6300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,5],"acting":[6,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"1b5","length":"1"},{"start":"1b7","length":"1"},{"start":"1ba","length":"1"},{"start":"1bd","length":"2"},{"start":"1c1","length":"1"},{"start":"1c5","length":"1"},{"start":"1c7","length":"1"}]},{"pgid":"3.1d","version":"0'0","reported_seq":1933,"reported_epoch":997,"state":"active+clean","last_fresh":"2024-09-20T22:58:08.376475+0000","last_change":"2024-09-20T22:56:20.497450+0000","last_active":"2024-09-20T22:58:08.376475+0000","last_peered":"2024-09-20T22:58:08.376475+0000","last_clean":"2024-09-20T22:58:08.376475+0000","last_became_active":"2024-09-20T22:48:05.242059+0000","last_became_peered":"2024-09-20T22:48:05.242059+0000","last_unstale":"2024-09-20T22:58:08.376475+0000","last_undegraded":"2024-09-20T22:58:08.376475+0000","last_fullsized":"2024-09-20T22:58:08.376475+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:56:20.497377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:46:00.999939+0000","last_clean_scrub_stamp":"2024-09-20T22:56:20.497377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"625'15","reported_seq":687,"reported_epoch":995,"state":"active+clean","last_fresh":"2024-09-20T22:58:06.267038+0000","last_change":"2024-09-20T22:57:49.405919+0000","last_active":"2024-09-20T22:58:06.267038+0000","last_peered":"2024-09-20T22:58:06.267038+0000","last_clean":"2024-09-20T22:58:06.267038+0000","last_became_active":"2024-09-20T22:57:49.403576+0000","last_became_peered":"2024-09-20T22:57:49.403576+0000","last_unstale":"2024-09-20T22:58:06.267038+0000","last_undegraded":"2024-09-20T22:58:06.267038+0000","last_fullsized":"2024-09-20T22:58:06.267038+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'15","last_scrub_stamp":"2024-09-20T22:53:51.486202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:53:51.486202+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":1931,"reported_epoch":997,"state":"active+clean","last_fresh":"2024-09-20T22:58:08.376039+0000","last_change":"2024-09-20T22:56:42.536987+0000","last_active":"2024-09-20T22:58:08.376039+0000","last_peered":"2024-09-20T22:58:08.376039+0000","last_clean":"2024-09-20T22:58:08.376039+0000","last_became_active":"2024-09-20T22:56:08.356419+0000","last_became_peered":"2024-09-20T22:56:08.356419+0000","last_unstale":"2024-09-20T22:58:08.376039+0000","last_undegraded":"2024-09-20T22:58:08.376039+0000","last_fullsized":"2024-09-20T22:58:08.376039+0000","mapping_epoch":887,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:56:42.536863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:56:42.536863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":1281,"reported_epoch":997,"state":"active+clean","last_fresh":"2024-09-20T22:58:08.374497+0000","last_change":"2024-09-20T22:56:08.391304+0000","last_active":"2024-09-20T22:58:08.374497+0000","last_peered":"2024-09-20T22:58:08.374497+0000","last_clean":"2024-09-20T22:58:08.374497+0000","last_became_active":"2024-09-20T22:56:08.390640+0000","last_became_peered":"2024-09-20T22:56:08.390640+0000","last_unstale":"2024-09-20T22:58:08.374497+0000","last_undegraded":"2024-09-20T22:58:08.374497+0000","last_fullsized":"2024-09-20T22:58:08.374497+0000","mapping_epoch":887,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:55:55.454925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:49:08.664407+0000","last_clean_scrub_stamp":"2024-09-20T22:55:55.454925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:12.539648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.18","version":"988'1502","reported_seq":5927,"reported_epoch":997,"state":"active+clean","last_fresh":"2024-09-20T22:58:08.373805+0000","last_change":"2024-09-20T22:57:48.386070+0000","last_active":"2024-09-20T22:58:08.373805+0000","last_peered":"2024-09-20T22:58:08.373805+0000","last_clean":"2024-09-20T22:58:08.373805+0000","last_became_active":"2024-09-20T22:56:58.365227+0000","last_became_peered":"2024-09-20T22:56:58.365227+0000","last_unstale":"2024-09-20T22:58:08.373805+0000","last_undegraded":"2024-09-20T22:58:08.373805+0000","last_fullsized":"2024-09-20T22:58:08.373805+0000","mapping_epoch":937,"log_start":"426'268","ondisk_log_start":"426'268","created":932,"last_epoch_clean":938,"parent":"0.0","parent_split_bits":5,"last_scrub":"942'1432","last_scrub_stamp":"2024-09-20T22:57:03.731020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:57:03.731020+0000","objects_scrubbed":17,"log_size":1234,"log_dups_size":445,"ondisk_log_size":1234,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:30.917366+0000","scrub_duration":8,"objects_trimmed":1,"snaptrim_duration":0.019121448999999999,"stat_sum":{"num_bytes":27703213,"num_objects":15,"num_object_clones":13,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":21,"num_read_kb":27807,"num_write":4,"num_write_kb":996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"1b5","length":"1"},{"start":"1b7","length":"1"},{"start":"1ba","length":"1"},{"start":"1bd","length":"2"},{"start":"1c1","length":"1"},{"start":"1c5","length":"1"},{"start":"1c7","length":"1"}]},{"pgid":"3.1c","version":"0'0","reported_seq":1945,"reported_epoch":997,"state":"active+clean","last_fresh":"2024-09-20T22:58:08.376652+0000","last_change":"2024-09-20T22:56:13.490259+0000","last_active":"2024-09-20T22:58:08.376652+0000","last_peered":"2024-09-20T22:58:08.376652+0000","last_clean":"2024-09-20T22:58:08.376652+0000","last_became_active":"2024-09-20T22:54:17.342619+0000","last_became_peered":"2024-09-20T22:54:17.342619+0000","last_unstale":"2024-09-20T22:58:08.376652+0000","last_undegraded":"2024-09-20T22:58:08.376652+0000","last_fullsized":"2024-09-20T22:58:08.376652+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:56:13.490178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:56:13.490178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"625'9","reported_seq":665,"reported_epoch":995,"state":"active+clean","last_fresh":"2024-09-20T22:58:06.267042+0000","last_change":"2024-09-20T22:57:49.424289+0000","last_active":"2024-09-20T22:58:06.267042+0000","last_peered":"2024-09-20T22:58:06.267042+0000","last_clean":"2024-09-20T22:58:06.267042+0000","last_became_active":"2024-09-20T22:57:49.423498+0000","last_became_peered":"2024-09-20T22:57:49.423498+0000","last_unstale":"2024-09-20T22:58:06.267042+0000","last_undegraded":"2024-09-20T22:58:06.267042+0000","last_fullsized":"2024-09-20T22:58:06.267042+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'9","last_scrub_stamp":"2024-09-20T22:56:24.469428+0000","last_deep_scrub":"181'7","last_deep_scrub_stamp":"2024-09-20T22:48:41.049381+0000","last_clean_scrub_stamp":"2024-09-20T22:56:24.469428+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":9,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":1722,"reported_epoch":997,"state":"active+clean","last_fresh":"2024-09-20T22:58:08.378738+0000","last_change":"2024-09-20T22:56:08.357180+0000","last_active":"2024-09-20T22:58:08.378738+0000","last_peered":"2024-09-20T22:58:08.378738+0000","last_clean":"2024-09-20T22:58:08.378738+0000","last_became_active":"2024-09-20T22:56:08.356857+0000","last_became_peered":"2024-09-20T22:56:08.356857+0000","last_unstale":"2024-09-20T22:58:08.378738+0000","last_undegraded":"2024-09-20T22:58:08.378738+0000","last_fullsized":"2024-09-20T22:58:08.378738+0000","mapping_epoch":887,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:55:59.860836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:47:40.361958+0000","last_clean_scrub_stamp":"2024-09-20T22:55:59.860836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:11.406350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":1933,"reported_epoch":997,"state":"active+clean","last_fresh":"2024-09-20T22:58:08.375902+0000","last_change":"2024-09-20T22:57:49.402679+0000","last_active":"2024-09-20T22:58:08.375902+0000","last_peered":"2024-09-20T22:58:08.375902+0000","last_clean":"2024-09-20T22:58:08.375902+0000","last_became_active":"2024-09-20T22:57:49.402447+0000","last_became_peered":"2024-09-20T22:57:49.402447+0000","last_unstale":"2024-09-20T22:58:08.375902+0000","last_undegraded":"2024-09-20T22:58:08.375902+0000","last_fullsized":"2024-09-20T22:58:08.375902+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:56:14.512566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:56:14.512566+0000","last_clean_scrub_stamp":"2024-09-20T22:56:14.512566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":347,"reported_epoch":997,"state":"active+clean","last_fresh":"2024-09-20T22:58:08.373065+0000","last_change":"2024-09-20T22:57:12.227470+0000","last_active":"2024-09-20T22:58:08.373065+0000","last_peered":"2024-09-20T22:58:08.373065+0000","last_clean":"2024-09-20T22:58:08.373065+0000","last_became_active":"2024-09-20T22:55:17.343025+0000","last_became_peered":"2024-09-20T22:55:17.343025+0000","last_unstale":"2024-09-20T22:58:08.373065+0000","last_undegraded":"2024-09-20T22:58:08.373065+0000","last_fullsized":"2024-09-20T22:58:08.373065+0000","mapping_epoch":836,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":837,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:57:12.227392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:57:12.227392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:25.441591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":1275,"reported_epoch":997,"state":"active+clean","last_fresh":"2024-09-20T22:58:08.375853+0000","last_change":"2024-09-20T22:57:12.715005+0000","last_active":"2024-09-20T22:58:08.375853+0000","last_peered":"2024-09-20T22:58:08.375853+0000","last_clean":"2024-09-20T22:58:08.375853+0000","last_became_active":"2024-09-20T22:56:08.391965+0000","last_became_peered":"2024-09-20T22:56:08.391965+0000","last_unstale":"2024-09-20T22:58:08.375853+0000","last_undegraded":"2024-09-20T22:58:08.375853+0000","last_fullsized":"2024-09-20T22:58:08.375853+0000","mapping_epoch":887,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:57:12.714907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:49:53.933601+0000","last_clean_scrub_stamp":"2024-09-20T22:57:12.714907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:20.512112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"625'34","reported_seq":2041,"reported_epoch":996,"state":"active+clean","last_fresh":"2024-09-20T22:58:07.790827+0000","last_change":"2024-09-20T22:58:01.296040+0000","last_active":"2024-09-20T22:58:07.790827+0000","last_peered":"2024-09-20T22:58:07.790827+0000","last_clean":"2024-09-20T22:58:07.790827+0000","last_became_active":"2024-09-20T22:57:49.430403+0000","last_became_peered":"2024-09-20T22:57:49.430403+0000","last_unstale":"2024-09-20T22:58:07.790827+0000","last_undegraded":"2024-09-20T22:58:07.790827+0000","last_fullsized":"2024-09-20T22:58:07.790827+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'34","last_scrub_stamp":"2024-09-20T22:58:01.295956+0000","last_deep_scrub":"113'22","last_deep_scrub_stamp":"2024-09-20T22:43:12.191878+0000","last_clean_scrub_stamp":"2024-09-20T22:58:01.295956+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:30.939636+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":474,"reported_epoch":997,"state":"active+clean","last_fresh":"2024-09-20T22:58:08.379001+0000","last_change":"2024-09-20T22:57:58.390860+0000","last_active":"2024-09-20T22:58:08.379001+0000","last_peered":"2024-09-20T22:58:08.379001+0000","last_clean":"2024-09-20T22:58:08.379001+0000","last_became_active":"2024-09-20T22:54:17.349501+0000","last_became_peered":"2024-09-20T22:54:17.349501+0000","last_unstale":"2024-09-20T22:58:08.379001+0000","last_undegraded":"2024-09-20T22:58:08.379001+0000","last_fullsized":"2024-09-20T22:58:08.379001+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:57:58.390757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:55:57.217692+0000","last_clean_scrub_stamp":"2024-09-20T22:57:58.390757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:20.023219+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":1777,"reported_epoch":995,"state":"active+clean","last_fresh":"2024-09-20T22:58:06.267868+0000","last_change":"2024-09-20T22:57:49.423684+0000","last_active":"2024-09-20T22:58:06.267868+0000","last_peered":"2024-09-20T22:58:06.267868+0000","last_clean":"2024-09-20T22:58:06.267868+0000","last_became_active":"2024-09-20T22:57:49.419671+0000","last_became_peered":"2024-09-20T22:57:49.419671+0000","last_unstale":"2024-09-20T22:58:06.267868+0000","last_undegraded":"2024-09-20T22:58:06.267868+0000","last_fullsized":"2024-09-20T22:58:06.267868+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:54:02.131659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:45:25.002199+0000","last_clean_scrub_stamp":"2024-09-20T22:54:02.131659+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":1130,"reported_epoch":997,"state":"active+clean","last_fresh":"2024-09-20T22:58:08.378579+0000","last_change":"2024-09-20T22:57:37.206101+0000","last_active":"2024-09-20T22:58:08.378579+0000","last_peered":"2024-09-20T22:58:08.378579+0000","last_clean":"2024-09-20T22:58:08.378579+0000","last_became_active":"2024-09-20T22:55:17.346400+0000","last_became_peered":"2024-09-20T22:55:17.346400+0000","last_unstale":"2024-09-20T22:58:08.378579+0000","last_undegraded":"2024-09-20T22:58:08.378579+0000","last_fullsized":"2024-09-20T22:58:08.378579+0000","mapping_epoch":836,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":837,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:57:37.206000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:50:41.885844+0000","last_clean_scrub_stamp":"2024-09-20T22:57:37.206000+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:48.705725+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"625'18","reported_seq":1855,"reported_epoch":995,"state":"active+clean","last_fresh":"2024-09-20T22:58:06.267644+0000","last_change":"2024-09-20T22:57:49.406206+0000","last_active":"2024-09-20T22:58:06.267644+0000","last_peered":"2024-09-20T22:58:06.267644+0000","last_clean":"2024-09-20T22:58:06.267644+0000","last_became_active":"2024-09-20T22:57:49.403366+0000","last_became_peered":"2024-09-20T22:57:49.403366+0000","last_unstale":"2024-09-20T22:58:06.267644+0000","last_undegraded":"2024-09-20T22:58:06.267644+0000","last_fullsized":"2024-09-20T22:58:06.267644+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'18","last_scrub_stamp":"2024-09-20T22:54:18.284289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:54:18.284289+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":758,"reported_epoch":995,"state":"active+clean","last_fresh":"2024-09-20T22:58:06.267565+0000","last_change":"2024-09-20T22:57:49.431948+0000","last_active":"2024-09-20T22:58:06.267565+0000","last_peered":"2024-09-20T22:58:06.267565+0000","last_clean":"2024-09-20T22:58:06.267565+0000","last_became_active":"2024-09-20T22:57:49.426209+0000","last_became_peered":"2024-09-20T22:57:49.426209+0000","last_unstale":"2024-09-20T22:58:06.267565+0000","last_undegraded":"2024-09-20T22:58:06.267565+0000","last_fullsized":"2024-09-20T22:58:06.267565+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:56:18.447131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:56:18.447131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"154'2","reported_seq":427,"reported_epoch":997,"state":"active+clean","last_fresh":"2024-09-20T22:58:08.376410+0000","last_change":"2024-09-20T22:58:01.796528+0000","last_active":"2024-09-20T22:58:08.376410+0000","last_peered":"2024-09-20T22:58:08.376410+0000","last_clean":"2024-09-20T22:58:08.376410+0000","last_became_active":"2024-09-20T22:57:49.427136+0000","last_became_peered":"2024-09-20T22:57:49.427136+0000","last_unstale":"2024-09-20T22:58:08.376410+0000","last_undegraded":"2024-09-20T22:58:08.376410+0000","last_fullsized":"2024-09-20T22:58:08.376410+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"154'2","last_scrub_stamp":"2024-09-20T22:58:01.796371+0000","last_deep_scrub":"154'2","last_deep_scrub_stamp":"2024-09-20T22:43:54.134066+0000","last_clean_scrub_stamp":"2024-09-20T22:58:01.796371+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:18.875708+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":403,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"894'2","reported_seq":480,"reported_epoch":997,"state":"active+clean","last_fresh":"2024-09-20T22:58:08.377929+0000","last_change":"2024-09-20T22:57:49.405855+0000","last_active":"2024-09-20T22:58:08.377929+0000","last_peered":"2024-09-20T22:58:08.377929+0000","last_clean":"2024-09-20T22:58:08.377929+0000","last_became_active":"2024-09-20T22:57:49.405458+0000","last_became_peered":"2024-09-20T22:57:49.405458+0000","last_unstale":"2024-09-20T22:58:08.377929+0000","last_undegraded":"2024-09-20T22:58:08.377929+0000","last_fullsized":"2024-09-20T22:58:08.377929+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"894'2","last_scrub_stamp":"2024-09-20T22:56:15.522649+0000","last_deep_scrub":"894'2","last_deep_scrub_stamp":"2024-09-20T22:56:15.522649+0000","last_clean_scrub_stamp":"2024-09-20T22:56:15.522649+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:11.406381+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"625'25","reported_seq":1326,"reported_epoch":997,"state":"active+clean","last_fresh":"2024-09-20T22:58:08.375984+0000","last_change":"2024-09-20T22:57:56.738440+0000","last_active":"2024-09-20T22:58:08.375984+0000","last_peered":"2024-09-20T22:58:08.375984+0000","last_clean":"2024-09-20T22:58:08.375984+0000","last_became_active":"2024-09-20T22:57:49.411398+0000","last_became_peered":"2024-09-20T22:57:49.411398+0000","last_unstale":"2024-09-20T22:58:08.375984+0000","last_undegraded":"2024-09-20T22:58:08.375984+0000","last_fullsized":"2024-09-20T22:58:08.375984+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'25","last_scrub_stamp":"2024-09-20T22:57:56.738344+0000","last_deep_scrub":"151'15","last_deep_scrub_stamp":"2024-09-20T22:43:50.715761+0000","last_clean_scrub_stamp":"2024-09-20T22:57:56.738344+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:19.842693+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"888'12","reported_seq":2179,"reported_epoch":997,"state":"active+clean","last_fresh":"2024-09-20T22:58:08.373472+0000","last_change":"2024-09-20T22:56:42.107724+0000","last_active":"2024-09-20T22:58:08.373472+0000","last_peered":"2024-09-20T22:58:08.373472+0000","last_clean":"2024-09-20T22:58:08.373472+0000","last_became_active":"2024-09-20T22:56:08.368232+0000","last_became_peered":"2024-09-20T22:56:08.368232+0000","last_unstale":"2024-09-20T22:58:08.373472+0000","last_undegraded":"2024-09-20T22:58:08.373472+0000","last_fullsized":"2024-09-20T22:58:08.373472+0000","mapping_epoch":887,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":0,"last_scrub":"888'12","last_scrub_stamp":"2024-09-20T22:56:42.107635+0000","last_deep_scrub":"661'11","last_deep_scrub_stamp":"2024-09-20T22:52:21.907943+0000","last_clean_scrub_stamp":"2024-09-20T22:56:42.107635+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:09.490301+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.c","version":"989'3221","reported_seq":7919,"reported_epoch":997,"state":"active+clean","last_fresh":"2024-09-20T22:58:08.372629+0000","last_change":"2024-09-20T22:57:49.743510+0000","last_active":"2024-09-20T22:58:08.372629+0000","last_peered":"2024-09-20T22:58:08.372629+0000","last_clean":"2024-09-20T22:58:08.372629+0000","last_became_active":"2024-09-20T22:57:49.423914+0000","last_became_peered":"2024-09-20T22:57:49.423914+0000","last_unstale":"2024-09-20T22:58:08.372629+0000","last_undegraded":"2024-09-20T22:58:08.372629+0000","last_fullsized":"2024-09-20T22:58:08.372629+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"902'2941","last_scrub_stamp":"2024-09-20T22:56:22.993088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:56:22.993088+0000","objects_scrubbed":36,"log_size":3221,"log_dups_size":0,"ondisk_log_size":3221,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":5,"snaptrim_duration":0.079379093999999997,"stat_sum":{"num_bytes":108964875,"num_objects":47,"num_object_clones":40,"num_object_copies":141,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":2,"num_read":1155,"num_read_kb":1186566,"num_write":778,"num_write_kb":308677,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":189,"num_bytes_recovered":87252218,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"1b5","length":"1"},{"start":"1b7","length":"1"},{"start":"1ba","length":"1"},{"start":"1bd","length":"2"},{"start":"1c1","length":"1"},{"start":"1c5","length":"1"},{"start":"1c7","length":"1"}]},{"pgid":"1.a","version":"77'128","reported_seq":2191,"reported_epoch":997,"state":"active+clean","last_fresh":"2024-09-20T22:58:08.375888+0000","last_change":"2024-09-20T22:57:41.688804+0000","last_active":"2024-09-20T22:58:08.375888+0000","last_peered":"2024-09-20T22:58:08.375888+0000","last_clean":"2024-09-20T22:58:08.375888+0000","last_became_active":"2024-09-20T22:42:52.407791+0000","last_became_peered":"2024-09-20T22:42:52.407791+0000","last_unstale":"2024-09-20T22:58:08.375888+0000","last_undegraded":"2024-09-20T22:58:08.375888+0000","last_fullsized":"2024-09-20T22:58:08.375888+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:57:41.688681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:57:41.688681+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:00.430067+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"625'27","reported_seq":499,"reported_epoch":997,"state":"active+clean","last_fresh":"2024-09-20T22:58:08.379540+0000","last_change":"2024-09-20T22:58:01.424871+0000","last_active":"2024-09-20T22:58:08.379540+0000","last_peered":"2024-09-20T22:58:08.379540+0000","last_clean":"2024-09-20T22:58:08.379540+0000","last_became_active":"2024-09-20T22:57:49.415260+0000","last_became_peered":"2024-09-20T22:57:49.415260+0000","last_unstale":"2024-09-20T22:58:08.379540+0000","last_undegraded":"2024-09-20T22:58:08.379540+0000","last_fullsized":"2024-09-20T22:58:08.379540+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'27","last_scrub_stamp":"2024-09-20T22:58:01.424772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:58:01.424772+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:30.323196+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":1814,"reported_epoch":995,"state":"active+clean","last_fresh":"2024-09-20T22:58:06.266745+0000","last_change":"2024-09-20T22:57:49.414013+0000","last_active":"2024-09-20T22:58:06.266745+0000","last_peered":"2024-09-20T22:58:06.266745+0000","last_clean":"2024-09-20T22:58:06.266745+0000","last_became_active":"2024-09-20T22:57:49.407789+0000","last_became_peered":"2024-09-20T22:57:49.407789+0000","last_unstale":"2024-09-20T22:58:06.266745+0000","last_undegraded":"2024-09-20T22:58:06.266745+0000","last_fullsized":"2024-09-20T22:58:06.266745+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:55:09.369892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:49:12.070675+0000","last_clean_scrub_stamp":"2024-09-20T22:55:09.369892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"838'13","reported_seq":992,"reported_epoch":997,"state":"active+clean","last_fresh":"2024-09-20T22:58:08.376673+0000","last_change":"2024-09-20T22:56:46.487242+0000","last_active":"2024-09-20T22:58:08.376673+0000","last_peered":"2024-09-20T22:58:08.376673+0000","last_clean":"2024-09-20T22:58:08.376673+0000","last_became_active":"2024-09-20T22:55:17.343505+0000","last_became_peered":"2024-09-20T22:55:17.343505+0000","last_unstale":"2024-09-20T22:58:08.376673+0000","last_undegraded":"2024-09-20T22:58:08.376673+0000","last_fullsized":"2024-09-20T22:58:08.376673+0000","mapping_epoch":836,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":837,"parent":"0.0","parent_split_bits":0,"last_scrub":"838'13","last_scrub_stamp":"2024-09-20T22:56:46.487164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:56:46.487164+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":1934,"reported_epoch":997,"state":"active+clean","last_fresh":"2024-09-20T22:58:08.376182+0000","last_change":"2024-09-20T22:56:41.547004+0000","last_active":"2024-09-20T22:58:08.376182+0000","last_peered":"2024-09-20T22:58:08.376182+0000","last_clean":"2024-09-20T22:58:08.376182+0000","last_became_active":"2024-09-20T22:54:17.340327+0000","last_became_peered":"2024-09-20T22:54:17.340327+0000","last_unstale":"2024-09-20T22:58:08.376182+0000","last_undegraded":"2024-09-20T22:58:08.376182+0000","last_fullsized":"2024-09-20T22:58:08.376182+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:56:41.546923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:56:41.546923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3","version":"931'475","reported_seq":4686,"reported_epoch":995,"state":"active+clean","last_fresh":"2024-09-20T22:58:06.267530+0000","last_change":"2024-09-20T22:57:56.654985+0000","last_active":"2024-09-20T22:58:06.267530+0000","last_peered":"2024-09-20T22:58:06.267530+0000","last_clean":"2024-09-20T22:58:06.267530+0000","last_became_active":"2024-09-20T22:57:49.410503+0000","last_became_peered":"2024-09-20T22:57:49.410503+0000","last_unstale":"2024-09-20T22:58:06.267530+0000","last_undegraded":"2024-09-20T22:58:06.267530+0000","last_fullsized":"2024-09-20T22:58:06.267530+0000","mapping_epoch":988,"log_start":"456'129","ondisk_log_start":"456'129","created":94,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"931'475","last_scrub_stamp":"2024-09-20T22:57:56.654880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:57:56.654880+0000","objects_scrubbed":0,"log_size":346,"log_dups_size":129,"ondisk_log_size":346,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:57.483771+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.059598702000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"1a6","length":"1"},{"start":"1aa","length":"1"},{"start":"1b5","length":"1"},{"start":"1b7","length":"1"},{"start":"1ba","length":"1"},{"start":"1bd","length":"2"},{"start":"1c0","length":"2"},{"start":"1c5","length":"1"},{"start":"1c7","length":"1"}]},{"pgid":"1.5","version":"77'128","reported_seq":1204,"reported_epoch":996,"state":"active+clean","last_fresh":"2024-09-20T22:58:07.790884+0000","last_change":"2024-09-20T22:57:14.262350+0000","last_active":"2024-09-20T22:58:07.790884+0000","last_peered":"2024-09-20T22:58:07.790884+0000","last_clean":"2024-09-20T22:58:07.790884+0000","last_became_active":"2024-09-20T22:55:17.345460+0000","last_became_peered":"2024-09-20T22:55:17.345460+0000","last_unstale":"2024-09-20T22:58:07.790884+0000","last_undegraded":"2024-09-20T22:58:07.790884+0000","last_fullsized":"2024-09-20T22:58:07.790884+0000","mapping_epoch":836,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":837,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:57:14.262265+0000","last_deep_scrub":"77'128","last_deep_scrub_stamp":"2024-09-20T22:57:14.262265+0000","last_clean_scrub_stamp":"2024-09-20T22:57:14.262265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:39.904856+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.0","version":"625'30","reported_seq":1915,"reported_epoch":995,"state":"active+clean","last_fresh":"2024-09-20T22:58:06.266728+0000","last_change":"2024-09-20T22:57:49.414226+0000","last_active":"2024-09-20T22:58:06.266728+0000","last_peered":"2024-09-20T22:58:06.266728+0000","last_clean":"2024-09-20T22:58:06.266728+0000","last_became_active":"2024-09-20T22:57:49.408104+0000","last_became_peered":"2024-09-20T22:57:49.408104+0000","last_unstale":"2024-09-20T22:58:06.266728+0000","last_undegraded":"2024-09-20T22:58:06.266728+0000","last_fullsized":"2024-09-20T22:58:06.266728+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'30","last_scrub_stamp":"2024-09-20T22:53:48.346341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:53:48.346341+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":39,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":1835,"reported_epoch":995,"state":"active+clean","last_fresh":"2024-09-20T22:58:06.268137+0000","last_change":"2024-09-20T22:57:49.433634+0000","last_active":"2024-09-20T22:58:06.268137+0000","last_peered":"2024-09-20T22:58:06.268137+0000","last_clean":"2024-09-20T22:58:06.268137+0000","last_became_active":"2024-09-20T22:57:49.432799+0000","last_became_peered":"2024-09-20T22:57:49.432799+0000","last_unstale":"2024-09-20T22:58:06.268137+0000","last_undegraded":"2024-09-20T22:58:06.268137+0000","last_fullsized":"2024-09-20T22:58:06.268137+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:56:15.465230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:56:15.465230+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":751,"reported_epoch":995,"state":"active+clean","last_fresh":"2024-09-20T22:58:06.266829+0000","last_change":"2024-09-20T22:57:49.416335+0000","last_active":"2024-09-20T22:58:06.266829+0000","last_peered":"2024-09-20T22:58:06.266829+0000","last_clean":"2024-09-20T22:58:06.266829+0000","last_became_active":"2024-09-20T22:57:49.413482+0000","last_became_peered":"2024-09-20T22:57:49.413482+0000","last_unstale":"2024-09-20T22:58:06.266829+0000","last_undegraded":"2024-09-20T22:58:06.266829+0000","last_fullsized":"2024-09-20T22:58:06.266829+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:56:47.432483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:56:47.432483+0000","last_clean_scrub_stamp":"2024-09-20T22:56:47.432483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":700,"reported_epoch":995,"state":"active+clean","last_fresh":"2024-09-20T22:58:06.267073+0000","last_change":"2024-09-20T22:57:49.404772+0000","last_active":"2024-09-20T22:58:06.267073+0000","last_peered":"2024-09-20T22:58:06.267073+0000","last_clean":"2024-09-20T22:58:06.267073+0000","last_became_active":"2024-09-20T22:57:49.402383+0000","last_became_peered":"2024-09-20T22:57:49.402383+0000","last_unstale":"2024-09-20T22:58:06.267073+0000","last_undegraded":"2024-09-20T22:58:06.267073+0000","last_fullsized":"2024-09-20T22:58:06.267073+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:55:48.456896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:53.036601+0000","last_clean_scrub_stamp":"2024-09-20T22:55:48.456896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2","version":"989'1026","reported_seq":4776,"reported_epoch":995,"state":"active+clean","last_fresh":"2024-09-20T22:58:06.267277+0000","last_change":"2024-09-20T22:57:57.697875+0000","last_active":"2024-09-20T22:58:06.267277+0000","last_peered":"2024-09-20T22:58:06.267277+0000","last_clean":"2024-09-20T22:58:06.267277+0000","last_became_active":"2024-09-20T22:57:49.400698+0000","last_became_peered":"2024-09-20T22:57:49.400698+0000","last_unstale":"2024-09-20T22:58:06.267277+0000","last_undegraded":"2024-09-20T22:58:06.267277+0000","last_fullsized":"2024-09-20T22:58:06.267277+0000","mapping_epoch":988,"log_start":"471'464","ondisk_log_start":"471'464","created":94,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"989'1026","last_scrub_stamp":"2024-09-20T22:57:57.697789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:57:57.697789+0000","objects_scrubbed":5,"log_size":562,"log_dups_size":610,"ondisk_log_size":562,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:16.856606+0000","scrub_duration":2,"objects_trimmed":1,"snaptrim_duration":0.100157336,"stat_sum":{"num_bytes":15831841,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"1a6","length":"1"},{"start":"1aa","length":"1"},{"start":"1b5","length":"1"},{"start":"1b7","length":"1"},{"start":"1ba","length":"1"},{"start":"1bd","length":"2"},{"start":"1c0","length":"2"},{"start":"1c5","length":"1"},{"start":"1c7","length":"1"}]},{"pgid":"1.4","version":"855'129","reported_seq":2122,"reported_epoch":995,"state":"active+clean","last_fresh":"2024-09-20T22:58:06.268072+0000","last_change":"2024-09-20T22:57:49.426337+0000","last_active":"2024-09-20T22:58:06.268072+0000","last_peered":"2024-09-20T22:58:06.268072+0000","last_clean":"2024-09-20T22:58:06.268072+0000","last_became_active":"2024-09-20T22:57:49.423690+0000","last_became_peered":"2024-09-20T22:57:49.423690+0000","last_unstale":"2024-09-20T22:58:06.268072+0000","last_undegraded":"2024-09-20T22:58:06.268072+0000","last_fullsized":"2024-09-20T22:58:06.268072+0000","mapping_epoch":988,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":3,"last_scrub":"855'129","last_scrub_stamp":"2024-09-20T22:55:35.536654+0000","last_deep_scrub":"855'129","last_deep_scrub_stamp":"2024-09-20T22:55:35.536654+0000","last_clean_scrub_stamp":"2024-09-20T22:55:35.536654+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":59,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"625'28","reported_seq":835,"reported_epoch":997,"state":"active+clean","last_fresh":"2024-09-20T22:58:08.375942+0000","last_change":"2024-09-20T22:57:05.659881+0000","last_active":"2024-09-20T22:58:08.375942+0000","last_peered":"2024-09-20T22:58:08.375942+0000","last_clean":"2024-09-20T22:58:08.375942+0000","last_became_active":"2024-09-20T22:55:17.344611+0000","last_became_peered":"2024-09-20T22:55:17.344611+0000","last_unstale":"2024-09-20T22:58:08.375942+0000","last_undegraded":"2024-09-20T22:58:08.375942+0000","last_fullsized":"2024-09-20T22:58:08.375942+0000","mapping_epoch":836,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":837,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'28","last_scrub_stamp":"2024-09-20T22:57:05.659800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:57:05.659800+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:10.175018+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":40,"num_read_kb":33,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.6","version":"855'2","reported_seq":1298,"reported_epoch":997,"state":"active+clean","last_fresh":"2024-09-20T22:58:08.375738+0000","last_change":"2024-09-20T22:55:35.573049+0000","last_active":"2024-09-20T22:58:08.375738+0000","last_peered":"2024-09-20T22:58:08.375738+0000","last_clean":"2024-09-20T22:58:08.375738+0000","last_became_active":"2024-09-20T22:54:17.340423+0000","last_became_peered":"2024-09-20T22:54:17.340423+0000","last_unstale":"2024-09-20T22:58:08.375738+0000","last_undegraded":"2024-09-20T22:58:08.375738+0000","last_fullsized":"2024-09-20T22:58:08.375738+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"855'2","last_scrub_stamp":"2024-09-20T22:55:35.572977+0000","last_deep_scrub":"855'2","last_deep_scrub_stamp":"2024-09-20T22:55:35.572977+0000","last_clean_scrub_stamp":"2024-09-20T22:55:35.572977+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:12.539600+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":230,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"989'18","reported_seq":2009,"reported_epoch":995,"state":"active+clean","last_fresh":"2024-09-20T22:58:06.266932+0000","last_change":"2024-09-20T22:57:49.413392+0000","last_active":"2024-09-20T22:58:06.266932+0000","last_peered":"2024-09-20T22:58:06.266932+0000","last_clean":"2024-09-20T22:58:06.266932+0000","last_became_active":"2024-09-20T22:57:49.401445+0000","last_became_peered":"2024-09-20T22:57:49.401445+0000","last_unstale":"2024-09-20T22:58:06.266932+0000","last_undegraded":"2024-09-20T22:58:06.266932+0000","last_fullsized":"2024-09-20T22:58:06.266932+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"778'16","last_scrub_stamp":"2024-09-20T22:55:10.378046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:55:10.378046+0000","objects_scrubbed":1,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":235,"reported_epoch":997,"state":"active+clean","last_fresh":"2024-09-20T22:58:08.372602+0000","last_change":"2024-09-20T22:57:10.158684+0000","last_active":"2024-09-20T22:58:08.372602+0000","last_peered":"2024-09-20T22:58:08.372602+0000","last_clean":"2024-09-20T22:58:08.372602+0000","last_became_active":"2024-09-20T22:56:08.367182+0000","last_became_peered":"2024-09-20T22:56:08.367182+0000","last_unstale":"2024-09-20T22:58:08.372602+0000","last_undegraded":"2024-09-20T22:58:08.372602+0000","last_fullsized":"2024-09-20T22:58:08.372602+0000","mapping_epoch":887,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:57:10.158581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:57:10.158581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:32.505201+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1","version":"989'621","reported_seq":4920,"reported_epoch":995,"state":"active+clean","last_fresh":"2024-09-20T22:58:06.267257+0000","last_change":"2024-09-20T22:57:58.740378+0000","last_active":"2024-09-20T22:58:06.267257+0000","last_peered":"2024-09-20T22:58:06.267257+0000","last_clean":"2024-09-20T22:58:06.267257+0000","last_became_active":"2024-09-20T22:57:49.404952+0000","last_became_peered":"2024-09-20T22:57:49.404952+0000","last_unstale":"2024-09-20T22:58:06.267257+0000","last_undegraded":"2024-09-20T22:58:06.267257+0000","last_fullsized":"2024-09-20T22:58:06.267257+0000","mapping_epoch":988,"log_start":"474'259","ondisk_log_start":"474'259","created":94,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"989'621","last_scrub_stamp":"2024-09-20T22:57:58.740287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:57:58.740287+0000","objects_scrubbed":14,"log_size":362,"log_dups_size":259,"ondisk_log_size":362,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:17.575278+0000","scrub_duration":5,"objects_trimmed":3,"snaptrim_duration":0.062553687999999996,"stat_sum":{"num_bytes":32739918,"num_objects":14,"num_object_clones":12,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"1a6","length":"1"},{"start":"1aa","length":"1"},{"start":"1b5","length":"1"},{"start":"1b7","length":"1"},{"start":"1ba","length":"1"},{"start":"1bd","length":"2"},{"start":"1c0","length":"2"},{"start":"1c5","length":"1"},{"start":"1c7","length":"1"}]},{"pgid":"1.7","version":"77'128","reported_seq":2258,"reported_epoch":997,"state":"active+clean","last_fresh":"2024-09-20T22:58:08.373237+0000","last_change":"2024-09-20T22:57:58.373829+0000","last_active":"2024-09-20T22:58:08.373237+0000","last_peered":"2024-09-20T22:58:08.373237+0000","last_clean":"2024-09-20T22:58:08.373237+0000","last_became_active":"2024-09-20T22:57:49.410770+0000","last_became_peered":"2024-09-20T22:57:49.410770+0000","last_unstale":"2024-09-20T22:58:08.373237+0000","last_undegraded":"2024-09-20T22:58:08.373237+0000","last_fullsized":"2024-09-20T22:58:08.373237+0000","mapping_epoch":988,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:57:58.373688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:57:58.373688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:23.927456+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"904'25","reported_seq":491,"reported_epoch":997,"state":"active+clean","last_fresh":"2024-09-20T22:58:08.378876+0000","last_change":"2024-09-20T22:57:33.088721+0000","last_active":"2024-09-20T22:58:08.378876+0000","last_peered":"2024-09-20T22:58:08.378876+0000","last_clean":"2024-09-20T22:58:08.378876+0000","last_became_active":"2024-09-20T22:54:17.349113+0000","last_became_peered":"2024-09-20T22:54:17.349113+0000","last_unstale":"2024-09-20T22:58:08.378876+0000","last_undegraded":"2024-09-20T22:58:08.378876+0000","last_fullsized":"2024-09-20T22:58:08.378876+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"904'25","last_scrub_stamp":"2024-09-20T22:57:33.088629+0000","last_deep_scrub":"904'25","last_deep_scrub_stamp":"2024-09-20T22:56:24.575150+0000","last_clean_scrub_stamp":"2024-09-20T22:57:33.088629+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:44.765537+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":28,"num_read_kb":23,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":1931,"reported_epoch":997,"state":"active+clean","last_fresh":"2024-09-20T22:58:08.376571+0000","last_change":"2024-09-20T22:57:49.396609+0000","last_active":"2024-09-20T22:58:08.376571+0000","last_peered":"2024-09-20T22:58:08.376571+0000","last_clean":"2024-09-20T22:58:08.376571+0000","last_became_active":"2024-09-20T22:57:49.396185+0000","last_became_peered":"2024-09-20T22:57:49.396185+0000","last_unstale":"2024-09-20T22:58:08.376571+0000","last_undegraded":"2024-09-20T22:58:08.376571+0000","last_fullsized":"2024-09-20T22:58:08.376571+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:55:53.487366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:55:53.487366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"733'12","reported_seq":1393,"reported_epoch":997,"state":"active+clean","last_fresh":"2024-09-20T22:58:08.375804+0000","last_change":"2024-09-20T22:55:50.416762+0000","last_active":"2024-09-20T22:58:08.375804+0000","last_peered":"2024-09-20T22:58:08.375804+0000","last_clean":"2024-09-20T22:58:08.375804+0000","last_became_active":"2024-09-20T22:47:49.227264+0000","last_became_peered":"2024-09-20T22:47:49.227264+0000","last_unstale":"2024-09-20T22:58:08.375804+0000","last_undegraded":"2024-09-20T22:58:08.375804+0000","last_fullsized":"2024-09-20T22:58:08.375804+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"733'12","last_scrub_stamp":"2024-09-20T22:55:50.416689+0000","last_deep_scrub":"733'12","last_deep_scrub_stamp":"2024-09-20T22:53:33.928499+0000","last_clean_scrub_stamp":"2024-09-20T22:55:50.416689+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:12.539616+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":1273,"reported_epoch":997,"state":"active+clean","last_fresh":"2024-09-20T22:58:08.374266+0000","last_change":"2024-09-20T22:57:49.416890+0000","last_active":"2024-09-20T22:58:08.374266+0000","last_peered":"2024-09-20T22:58:08.374266+0000","last_clean":"2024-09-20T22:58:08.374266+0000","last_became_active":"2024-09-20T22:57:49.416550+0000","last_became_peered":"2024-09-20T22:57:49.416550+0000","last_unstale":"2024-09-20T22:58:08.374266+0000","last_undegraded":"2024-09-20T22:58:08.374266+0000","last_fullsized":"2024-09-20T22:58:08.374266+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:55:57.400645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:55:57.400645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:12.539632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.0","version":"993'1044","reported_seq":5652,"reported_epoch":997,"state":"active+clean","last_fresh":"2024-09-20T22:58:08.375707+0000","last_change":"2024-09-20T22:58:05.781033+0000","last_active":"2024-09-20T22:58:08.375707+0000","last_peered":"2024-09-20T22:58:08.375707+0000","last_clean":"2024-09-20T22:58:08.375707+0000","last_became_active":"2024-09-20T22:57:49.412781+0000","last_became_peered":"2024-09-20T22:57:49.412781+0000","last_unstale":"2024-09-20T22:58:08.375707+0000","last_undegraded":"2024-09-20T22:58:08.375707+0000","last_fullsized":"2024-09-20T22:58:08.375707+0000","mapping_epoch":988,"log_start":"480'253","ondisk_log_start":"480'253","created":94,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"993'1044","last_scrub_stamp":"2024-09-20T22:58:05.780880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:58:05.780880+0000","objects_scrubbed":15,"log_size":791,"log_dups_size":416,"ondisk_log_size":791,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:24.521784+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.17545657200000001,"stat_sum":{"num_bytes":32336149,"num_objects":15,"num_object_clones":13,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"1b5","length":"1"},{"start":"1b7","length":"1"},{"start":"1ba","length":"1"},{"start":"1bd","length":"2"},{"start":"1c1","length":"1"},{"start":"1c5","length":"1"},{"start":"1c7","length":"1"}]},{"pgid":"1.6","version":"77'128","reported_seq":2265,"reported_epoch":997,"state":"active+clean","last_fresh":"2024-09-20T22:58:08.373159+0000","last_change":"2024-09-20T22:58:03.362917+0000","last_active":"2024-09-20T22:58:08.373159+0000","last_peered":"2024-09-20T22:58:08.373159+0000","last_clean":"2024-09-20T22:58:08.373159+0000","last_became_active":"2024-09-20T22:57:49.406637+0000","last_became_peered":"2024-09-20T22:57:49.406637+0000","last_unstale":"2024-09-20T22:58:08.373159+0000","last_undegraded":"2024-09-20T22:58:08.373159+0000","last_fullsized":"2024-09-20T22:58:08.373159+0000","mapping_epoch":988,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:58:03.362805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:58:03.362805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:32.522555+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"625'37","reported_seq":537,"reported_epoch":997,"state":"activating+degraded","last_fresh":"2024-09-20T22:58:08.377805+0000","last_change":"2024-09-20T22:58:08.377805+0000","last_active":"2024-09-20T22:58:07.793780+0000","last_peered":"2024-09-20T22:51:45.818287+0000","last_clean":"2024-09-20T22:51:45.818287+0000","last_became_active":"2024-09-20T22:47:49.222009+0000","last_became_peered":"2024-09-20T22:47:49.222009+0000","last_unstale":"2024-09-20T22:58:08.377805+0000","last_undegraded":"2024-09-20T22:58:08.376540+0000","last_fullsized":"2024-09-20T22:58:08.377805+0000","mapping_epoch":996,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'37","last_scrub_stamp":"2024-09-20T22:56:48.676788+0000","last_deep_scrub":"515'27","last_deep_scrub_stamp":"2024-09-20T22:49:55.964153+0000","last_clean_scrub_stamp":"2024-09-20T22:56:48.676788+0000","objects_scrubbed":8,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":38,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":["5","0","1"],"object_location_counts":[{"shards":"0,1,5","objects":8}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":466,"reported_epoch":997,"state":"active+clean","last_fresh":"2024-09-20T22:58:08.378586+0000","last_change":"2024-09-20T22:57:35.151377+0000","last_active":"2024-09-20T22:58:08.378586+0000","last_peered":"2024-09-20T22:58:08.378586+0000","last_clean":"2024-09-20T22:58:08.378586+0000","last_became_active":"2024-09-20T22:54:17.341248+0000","last_became_peered":"2024-09-20T22:54:17.341248+0000","last_unstale":"2024-09-20T22:58:08.378586+0000","last_undegraded":"2024-09-20T22:58:08.378586+0000","last_fullsized":"2024-09-20T22:58:08.378586+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:57:35.151279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:48:23.320031+0000","last_clean_scrub_stamp":"2024-09-20T22:57:35.151279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:39.976778+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":1947,"reported_epoch":997,"state":"active+clean","last_fresh":"2024-09-20T22:58:08.372515+0000","last_change":"2024-09-20T22:58:01.380830+0000","last_active":"2024-09-20T22:58:08.372515+0000","last_peered":"2024-09-20T22:58:08.372515+0000","last_clean":"2024-09-20T22:58:08.372515+0000","last_became_active":"2024-09-20T22:47:49.214055+0000","last_became_peered":"2024-09-20T22:47:49.214055+0000","last_unstale":"2024-09-20T22:58:08.372515+0000","last_undegraded":"2024-09-20T22:58:08.372515+0000","last_fullsized":"2024-09-20T22:58:08.372515+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:58:01.380733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:58:01.380733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:16.552034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":458,"reported_epoch":997,"state":"active+clean","last_fresh":"2024-09-20T22:58:08.378045+0000","last_change":"2024-09-20T22:55:54.151421+0000","last_active":"2024-09-20T22:58:08.378045+0000","last_peered":"2024-09-20T22:58:08.378045+0000","last_clean":"2024-09-20T22:58:08.378045+0000","last_became_active":"2024-09-20T22:54:17.340458+0000","last_became_peered":"2024-09-20T22:54:17.340458+0000","last_unstale":"2024-09-20T22:58:08.378045+0000","last_undegraded":"2024-09-20T22:58:08.378045+0000","last_fullsized":"2024-09-20T22:58:08.378045+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:55:54.151316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:48:41.208853+0000","last_clean_scrub_stamp":"2024-09-20T22:55:54.151316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:11.406319+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.7","version":"989'2066","reported_seq":6560,"reported_epoch":997,"state":"active+clean","last_fresh":"2024-09-20T22:58:08.373458+0000","last_change":"2024-09-20T22:57:56.323805+0000","last_active":"2024-09-20T22:58:08.373458+0000","last_peered":"2024-09-20T22:58:08.373458+0000","last_clean":"2024-09-20T22:58:08.373458+0000","last_became_active":"2024-09-20T22:57:49.410914+0000","last_became_peered":"2024-09-20T22:57:49.410914+0000","last_unstale":"2024-09-20T22:58:08.373458+0000","last_undegraded":"2024-09-20T22:58:08.373458+0000","last_fullsized":"2024-09-20T22:58:08.373458+0000","mapping_epoch":988,"log_start":"436'456","ondisk_log_start":"436'456","created":94,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"989'2066","last_scrub_stamp":"2024-09-20T22:57:56.323701+0000","last_deep_scrub":"879'1788","last_deep_scrub_stamp":"2024-09-20T22:55:59.964830+0000","last_clean_scrub_stamp":"2024-09-20T22:57:56.323701+0000","objects_scrubbed":26,"log_size":1610,"log_dups_size":886,"ondisk_log_size":1610,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:59.666686+0000","scrub_duration":7,"objects_trimmed":1,"snaptrim_duration":0.069128024999999996,"stat_sum":{"num_bytes":62288349,"num_objects":26,"num_object_clones":23,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"1b5","length":"1"},{"start":"1b7","length":"1"},{"start":"1ba","length":"1"},{"start":"1bd","length":"2"},{"start":"1c1","length":"1"},{"start":"1c5","length":"1"},{"start":"1c7","length":"1"}]},{"pgid":"1.1","version":"77'128","reported_seq":2239,"reported_epoch":997,"state":"active+clean","last_fresh":"2024-09-20T22:58:08.375794+0000","last_change":"2024-09-20T22:57:37.743124+0000","last_active":"2024-09-20T22:58:08.375794+0000","last_peered":"2024-09-20T22:58:08.375794+0000","last_clean":"2024-09-20T22:58:08.375794+0000","last_became_active":"2024-09-20T22:47:52.208102+0000","last_became_peered":"2024-09-20T22:47:52.208102+0000","last_unstale":"2024-09-20T22:58:08.375794+0000","last_undegraded":"2024-09-20T22:58:08.375794+0000","last_fullsized":"2024-09-20T22:58:08.375794+0000","mapping_epoch":391,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:57:37.743043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:57:37.743043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:54.685166+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"989'55","reported_seq":420,"reported_epoch":997,"state":"active+clean","last_fresh":"2024-09-20T22:58:08.376304+0000","last_change":"2024-09-20T22:57:49.427243+0000","last_active":"2024-09-20T22:58:08.376304+0000","last_peered":"2024-09-20T22:58:08.376304+0000","last_clean":"2024-09-20T22:58:08.376304+0000","last_became_active":"2024-09-20T22:57:49.426958+0000","last_became_peered":"2024-09-20T22:57:49.426958+0000","last_unstale":"2024-09-20T22:58:08.376304+0000","last_undegraded":"2024-09-20T22:58:08.376304+0000","last_fullsized":"2024-09-20T22:58:08.376304+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"833'53","last_scrub_stamp":"2024-09-20T22:56:17.472832+0000","last_deep_scrub":"86'38","last_deep_scrub_stamp":"2024-09-20T22:42:43.935677+0000","last_clean_scrub_stamp":"2024-09-20T22:56:17.472832+0000","objects_scrubbed":10,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":1074,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":387,"reported_epoch":997,"state":"active+clean","last_fresh":"2024-09-20T22:58:08.375644+0000","last_change":"2024-09-20T22:57:39.577432+0000","last_active":"2024-09-20T22:58:08.375644+0000","last_peered":"2024-09-20T22:58:08.375644+0000","last_clean":"2024-09-20T22:58:08.375644+0000","last_became_active":"2024-09-20T22:55:17.340038+0000","last_became_peered":"2024-09-20T22:55:17.340038+0000","last_unstale":"2024-09-20T22:58:08.375644+0000","last_undegraded":"2024-09-20T22:58:08.375644+0000","last_fullsized":"2024-09-20T22:58:08.375644+0000","mapping_epoch":836,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":837,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:57:39.577360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:57:39.577360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:04.367204+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"69'2","reported_seq":1963,"reported_epoch":997,"state":"active+clean","last_fresh":"2024-09-20T22:58:08.376264+0000","last_change":"2024-09-20T22:57:03.613282+0000","last_active":"2024-09-20T22:58:08.376264+0000","last_peered":"2024-09-20T22:58:08.376264+0000","last_clean":"2024-09-20T22:58:08.376264+0000","last_became_active":"2024-09-20T22:54:17.341059+0000","last_became_peered":"2024-09-20T22:54:17.341059+0000","last_unstale":"2024-09-20T22:58:08.376264+0000","last_undegraded":"2024-09-20T22:58:08.376264+0000","last_fullsized":"2024-09-20T22:58:08.376264+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2024-09-20T22:57:03.613204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:36.130684+0000","last_clean_scrub_stamp":"2024-09-20T22:57:03.613204+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:15.544078+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00087280599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":1281,"reported_epoch":997,"state":"active+clean","last_fresh":"2024-09-20T22:58:08.375567+0000","last_change":"2024-09-20T22:56:13.481362+0000","last_active":"2024-09-20T22:58:08.375567+0000","last_peered":"2024-09-20T22:58:08.375567+0000","last_clean":"2024-09-20T22:58:08.375567+0000","last_became_active":"2024-09-20T22:54:17.339860+0000","last_became_peered":"2024-09-20T22:54:17.339860+0000","last_unstale":"2024-09-20T22:58:08.375567+0000","last_undegraded":"2024-09-20T22:58:08.375567+0000","last_fullsized":"2024-09-20T22:58:08.375567+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:56:13.481292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:56:13.481292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:12.539662+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":1810,"reported_epoch":995,"state":"active+clean","last_fresh":"2024-09-20T22:58:06.268022+0000","last_change":"2024-09-20T22:57:49.407286+0000","last_active":"2024-09-20T22:58:06.268022+0000","last_peered":"2024-09-20T22:58:06.268022+0000","last_clean":"2024-09-20T22:58:06.268022+0000","last_became_active":"2024-09-20T22:57:49.398514+0000","last_became_peered":"2024-09-20T22:57:49.398514+0000","last_unstale":"2024-09-20T22:58:06.268022+0000","last_undegraded":"2024-09-20T22:58:06.268022+0000","last_fullsized":"2024-09-20T22:58:06.268022+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:55:51.494144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:55:51.494144+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"988'758","reported_seq":3754,"reported_epoch":997,"state":"active+clean","last_fresh":"2024-09-20T22:58:08.375228+0000","last_change":"2024-09-20T22:57:48.440745+0000","last_active":"2024-09-20T22:58:08.375228+0000","last_peered":"2024-09-20T22:58:08.375228+0000","last_clean":"2024-09-20T22:58:08.375228+0000","last_became_active":"2024-09-20T22:56:53.366703+0000","last_became_peered":"2024-09-20T22:56:53.366703+0000","last_unstale":"2024-09-20T22:58:08.375228+0000","last_undegraded":"2024-09-20T22:58:08.375228+0000","last_fullsized":"2024-09-20T22:58:08.375228+0000","mapping_epoch":779,"log_start":"456'176","ondisk_log_start":"456'176","created":94,"last_epoch_clean":933,"parent":"0.0","parent_split_bits":0,"last_scrub":"941'718","last_scrub_stamp":"2024-09-20T22:57:06.750095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:57:06.750095+0000","objects_scrubbed":5,"log_size":582,"log_dups_size":318,"ondisk_log_size":582,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:36.737266+0000","scrub_duration":3,"objects_trimmed":1,"snaptrim_duration":0.071871422000000004,"stat_sum":{"num_bytes":12201763,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":11,"num_read_kb":7000,"num_write":8,"num_write_kb":2179,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"1b5","length":"1"},{"start":"1b7","length":"1"},{"start":"1ba","length":"1"},{"start":"1bd","length":"2"},{"start":"1c1","length":"1"},{"start":"1c5","length":"1"},{"start":"1c7","length":"1"}]},{"pgid":"1.2","version":"77'128","reported_seq":1265,"reported_epoch":997,"state":"active+clean","last_fresh":"2024-09-20T22:58:08.374707+0000","last_change":"2024-09-20T22:56:12.520096+0000","last_active":"2024-09-20T22:58:08.374707+0000","last_peered":"2024-09-20T22:58:08.374707+0000","last_clean":"2024-09-20T22:58:08.374707+0000","last_became_active":"2024-09-20T22:56:11.355063+0000","last_became_peered":"2024-09-20T22:56:11.355063+0000","last_unstale":"2024-09-20T22:58:08.374707+0000","last_undegraded":"2024-09-20T22:58:08.374707+0000","last_fullsized":"2024-09-20T22:58:08.374707+0000","mapping_epoch":890,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":891,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:56:12.519992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:56:12.519992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:12.539679+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"625'22","reported_seq":481,"reported_epoch":997,"state":"active+clean","last_fresh":"2024-09-20T22:58:08.379575+0000","last_change":"2024-09-20T22:57:06.349767+0000","last_active":"2024-09-20T22:58:08.379575+0000","last_peered":"2024-09-20T22:58:08.379575+0000","last_clean":"2024-09-20T22:58:08.379575+0000","last_became_active":"2024-09-20T22:54:17.349415+0000","last_became_peered":"2024-09-20T22:54:17.349415+0000","last_unstale":"2024-09-20T22:58:08.379575+0000","last_undegraded":"2024-09-20T22:58:08.379575+0000","last_fullsized":"2024-09-20T22:58:08.379575+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'22","last_scrub_stamp":"2024-09-20T22:57:06.349682+0000","last_deep_scrub":"443'18","last_deep_scrub_stamp":"2024-09-20T22:48:44.182517+0000","last_clean_scrub_stamp":"2024-09-20T22:57:06.349682+0000","objects_scrubbed":5,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:11.297930+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":24,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":1320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":466,"reported_epoch":997,"state":"active+clean","last_fresh":"2024-09-20T22:58:08.378172+0000","last_change":"2024-09-20T22:56:00.249870+0000","last_active":"2024-09-20T22:58:08.378172+0000","last_peered":"2024-09-20T22:58:08.378172+0000","last_clean":"2024-09-20T22:58:08.378172+0000","last_became_active":"2024-09-20T22:54:17.343806+0000","last_became_peered":"2024-09-20T22:54:17.343806+0000","last_unstale":"2024-09-20T22:58:08.378172+0000","last_undegraded":"2024-09-20T22:58:08.378172+0000","last_fullsized":"2024-09-20T22:58:08.378172+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:56:00.249796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:56:00.249796+0000","last_clean_scrub_stamp":"2024-09-20T22:56:00.249796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:11.406364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"380'2","reported_seq":1982,"reported_epoch":996,"state":"active+clean","last_fresh":"2024-09-20T22:58:07.789546+0000","last_change":"2024-09-20T22:57:49.417639+0000","last_active":"2024-09-20T22:58:07.789546+0000","last_peered":"2024-09-20T22:58:07.789546+0000","last_clean":"2024-09-20T22:58:07.789546+0000","last_became_active":"2024-09-20T22:57:49.416996+0000","last_became_peered":"2024-09-20T22:57:49.416996+0000","last_unstale":"2024-09-20T22:58:07.789546+0000","last_undegraded":"2024-09-20T22:58:07.789546+0000","last_fullsized":"2024-09-20T22:58:07.789546+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"380'2","last_scrub_stamp":"2024-09-20T22:55:57.061312+0000","last_deep_scrub":"380'2","last_deep_scrub_stamp":"2024-09-20T22:47:40.983039+0000","last_clean_scrub_stamp":"2024-09-20T22:55:57.061312+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:12.237741+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001301459,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":1933,"reported_epoch":996,"state":"active+clean","last_fresh":"2024-09-20T22:58:07.790590+0000","last_change":"2024-09-20T22:57:38.381568+0000","last_active":"2024-09-20T22:58:07.790590+0000","last_peered":"2024-09-20T22:58:07.790590+0000","last_clean":"2024-09-20T22:58:07.790590+0000","last_became_active":"2024-09-20T22:56:08.375555+0000","last_became_peered":"2024-09-20T22:56:08.375555+0000","last_unstale":"2024-09-20T22:58:07.790590+0000","last_undegraded":"2024-09-20T22:58:07.790590+0000","last_fullsized":"2024-09-20T22:58:07.790590+0000","mapping_epoch":887,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:57:38.381457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:57:38.381457+0000","last_clean_scrub_stamp":"2024-09-20T22:57:38.381457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:05.474433+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":244,"reported_epoch":997,"state":"active+clean","last_fresh":"2024-09-20T22:58:08.373225+0000","last_change":"2024-09-20T22:57:49.404111+0000","last_active":"2024-09-20T22:58:08.373225+0000","last_peered":"2024-09-20T22:58:08.373225+0000","last_clean":"2024-09-20T22:58:08.373225+0000","last_became_active":"2024-09-20T22:57:49.403788+0000","last_became_peered":"2024-09-20T22:57:49.403788+0000","last_unstale":"2024-09-20T22:58:08.373225+0000","last_undegraded":"2024-09-20T22:58:08.373225+0000","last_fullsized":"2024-09-20T22:58:08.373225+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:56:51.218428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:56:51.218428+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.5","version":"990'592","reported_seq":5169,"reported_epoch":996,"state":"active+clean","last_fresh":"2024-09-20T22:58:07.790766+0000","last_change":"2024-09-20T22:57:48.440435+0000","last_active":"2024-09-20T22:58:07.790766+0000","last_peered":"2024-09-20T22:58:07.790766+0000","last_clean":"2024-09-20T22:58:07.790766+0000","last_became_active":"2024-09-20T22:56:53.369151+0000","last_became_peered":"2024-09-20T22:56:53.369151+0000","last_unstale":"2024-09-20T22:58:07.790766+0000","last_undegraded":"2024-09-20T22:58:07.790766+0000","last_fullsized":"2024-09-20T22:58:07.790766+0000","mapping_epoch":890,"log_start":"450'190","ondisk_log_start":"450'190","created":94,"last_epoch_clean":933,"parent":"0.0","parent_split_bits":0,"last_scrub":"941'546","last_scrub_stamp":"2024-09-20T22:57:04.121297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:57:04.121297+0000","objects_scrubbed":7,"log_size":402,"log_dups_size":240,"ondisk_log_size":402,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:30.793975+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.059506083000000001,"stat_sum":{"num_bytes":15343641,"num_objects":8,"num_object_clones":7,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":4,"num_write_kb":4269,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"1a6","length":"1"},{"start":"1aa","length":"1"},{"start":"1b5","length":"1"},{"start":"1b7","length":"1"},{"start":"1ba","length":"1"},{"start":"1bd","length":"2"},{"start":"1c0","length":"2"},{"start":"1c5","length":"1"},{"start":"1c7","length":"1"}]},{"pgid":"1.3","version":"77'128","reported_seq":1485,"reported_epoch":997,"state":"active+clean","last_fresh":"2024-09-20T22:58:08.378461+0000","last_change":"2024-09-20T22:56:21.532248+0000","last_active":"2024-09-20T22:58:08.378461+0000","last_peered":"2024-09-20T22:58:08.378461+0000","last_clean":"2024-09-20T22:58:08.378461+0000","last_became_active":"2024-09-20T22:54:20.413191+0000","last_became_peered":"2024-09-20T22:54:20.413191+0000","last_unstale":"2024-09-20T22:58:08.378461+0000","last_undegraded":"2024-09-20T22:58:08.378461+0000","last_fullsized":"2024-09-20T22:58:08.378461+0000","mapping_epoch":779,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":780,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:56:21.532173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:56:21.532173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:11.406398+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"828'30","reported_seq":1361,"reported_epoch":997,"state":"active+clean","last_fresh":"2024-09-20T22:58:08.375889+0000","last_change":"2024-09-20T22:57:09.738754+0000","last_active":"2024-09-20T22:58:08.375889+0000","last_peered":"2024-09-20T22:58:08.375889+0000","last_clean":"2024-09-20T22:58:08.375889+0000","last_became_active":"2024-09-20T22:55:11.346018+0000","last_became_peered":"2024-09-20T22:55:11.346018+0000","last_unstale":"2024-09-20T22:58:08.375889+0000","last_undegraded":"2024-09-20T22:58:08.375889+0000","last_fullsized":"2024-09-20T22:58:08.375889+0000","mapping_epoch":830,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":831,"parent":"0.0","parent_split_bits":0,"last_scrub":"828'30","last_scrub_stamp":"2024-09-20T22:57:09.738669+0000","last_deep_scrub":"828'30","last_deep_scrub_stamp":"2024-09-20T22:55:08.705571+0000","last_clean_scrub_stamp":"2024-09-20T22:57:09.738669+0000","objects_scrubbed":6,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:37.471839+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":1286,"reported_epoch":997,"state":"active+clean","last_fresh":"2024-09-20T22:58:08.374743+0000","last_change":"2024-09-20T22:58:04.615065+0000","last_active":"2024-09-20T22:58:08.374743+0000","last_peered":"2024-09-20T22:58:08.374743+0000","last_clean":"2024-09-20T22:58:08.374743+0000","last_became_active":"2024-09-20T22:57:49.410539+0000","last_became_peered":"2024-09-20T22:57:49.410539+0000","last_unstale":"2024-09-20T22:58:08.374743+0000","last_undegraded":"2024-09-20T22:58:08.374743+0000","last_fullsized":"2024-09-20T22:58:08.374743+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:58:04.614957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:44:19.834641+0000","last_clean_scrub_stamp":"2024-09-20T22:58:04.614957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:21.673315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"75'5","reported_seq":2628,"reported_epoch":997,"state":"active+clean","last_fresh":"2024-09-20T22:58:08.377998+0000","last_change":"2024-09-20T22:57:49.402647+0000","last_active":"2024-09-20T22:58:08.377998+0000","last_peered":"2024-09-20T22:58:08.377998+0000","last_clean":"2024-09-20T22:58:08.377998+0000","last_became_active":"2024-09-20T22:57:49.402122+0000","last_became_peered":"2024-09-20T22:57:49.402122+0000","last_unstale":"2024-09-20T22:58:08.377998+0000","last_undegraded":"2024-09-20T22:58:08.377998+0000","last_fullsized":"2024-09-20T22:58:08.377998+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-09-20T22:56:12.471215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:36.130684+0000","last_clean_scrub_stamp":"2024-09-20T22:56:12.471215+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:11.406335+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00078631700000000003,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":865,"num_read_kb":860,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1556,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":1946,"reported_epoch":997,"state":"active+clean","last_fresh":"2024-09-20T22:58:08.377058+0000","last_change":"2024-09-20T22:57:13.660607+0000","last_active":"2024-09-20T22:58:08.377058+0000","last_peered":"2024-09-20T22:58:08.377058+0000","last_clean":"2024-09-20T22:58:08.377058+0000","last_became_active":"2024-09-20T22:54:17.339805+0000","last_became_peered":"2024-09-20T22:54:17.339805+0000","last_unstale":"2024-09-20T22:58:08.377058+0000","last_undegraded":"2024-09-20T22:58:08.377058+0000","last_fullsized":"2024-09-20T22:58:08.377058+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:57:13.660520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:57:13.660520+0000","last_clean_scrub_stamp":"2024-09-20T22:57:13.660520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:37.166154+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":475,"reported_epoch":997,"state":"active+clean","last_fresh":"2024-09-20T22:58:08.378393+0000","last_change":"2024-09-20T22:57:57.433001+0000","last_active":"2024-09-20T22:58:08.378393+0000","last_peered":"2024-09-20T22:58:08.378393+0000","last_clean":"2024-09-20T22:58:08.378393+0000","last_became_active":"2024-09-20T22:57:49.429412+0000","last_became_peered":"2024-09-20T22:57:49.429412+0000","last_unstale":"2024-09-20T22:58:08.378393+0000","last_undegraded":"2024-09-20T22:58:08.378393+0000","last_fullsized":"2024-09-20T22:58:08.378393+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:57:57.432865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:57:57.432865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:13.282483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.6","version":"989'890","reported_seq":5387,"reported_epoch":997,"state":"active+clean","last_fresh":"2024-09-20T22:58:08.375605+0000","last_change":"2024-09-20T22:57:55.700976+0000","last_active":"2024-09-20T22:58:08.375605+0000","last_peered":"2024-09-20T22:58:08.375605+0000","last_clean":"2024-09-20T22:58:08.375605+0000","last_became_active":"2024-09-20T22:57:49.412758+0000","last_became_peered":"2024-09-20T22:57:49.412758+0000","last_unstale":"2024-09-20T22:58:08.375605+0000","last_undegraded":"2024-09-20T22:58:08.375605+0000","last_fullsized":"2024-09-20T22:58:08.375605+0000","mapping_epoch":988,"log_start":"443'181","ondisk_log_start":"443'181","created":94,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"989'890","last_scrub_stamp":"2024-09-20T22:57:55.700872+0000","last_deep_scrub":"902'811","last_deep_scrub_stamp":"2024-09-20T22:56:22.658684+0000","last_clean_scrub_stamp":"2024-09-20T22:57:55.700872+0000","objects_scrubbed":9,"log_size":709,"log_dups_size":314,"ondisk_log_size":709,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:17.908725+0000","scrub_duration":3,"objects_trimmed":2,"snaptrim_duration":0.046809684999999997,"stat_sum":{"num_bytes":20144663,"num_objects":9,"num_object_clones":8,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"1b5","length":"1"},{"start":"1b7","length":"1"},{"start":"1ba","length":"1"},{"start":"1bd","length":"2"},{"start":"1c1","length":"1"},{"start":"1c5","length":"1"},{"start":"1c7","length":"1"}]},{"pgid":"4.5","version":"888'33","reported_seq":2290,"reported_epoch":997,"state":"active+clean","last_fresh":"2024-09-20T22:58:08.373329+0000","last_change":"2024-09-20T22:56:21.986738+0000","last_active":"2024-09-20T22:58:08.373329+0000","last_peered":"2024-09-20T22:58:08.373329+0000","last_clean":"2024-09-20T22:58:08.373329+0000","last_became_active":"2024-09-20T22:56:08.358532+0000","last_became_peered":"2024-09-20T22:56:08.358532+0000","last_unstale":"2024-09-20T22:58:08.373329+0000","last_undegraded":"2024-09-20T22:58:08.373329+0000","last_fullsized":"2024-09-20T22:58:08.373329+0000","mapping_epoch":887,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":0,"last_scrub":"888'33","last_scrub_stamp":"2024-09-20T22:56:21.986633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:56:21.986633+0000","objects_scrubbed":8,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":770,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"67'1","reported_seq":1815,"reported_epoch":995,"state":"active+clean","last_fresh":"2024-09-20T22:58:06.267182+0000","last_change":"2024-09-20T22:57:49.401711+0000","last_active":"2024-09-20T22:58:06.267182+0000","last_peered":"2024-09-20T22:58:06.267182+0000","last_clean":"2024-09-20T22:58:06.267182+0000","last_became_active":"2024-09-20T22:57:49.397191+0000","last_became_peered":"2024-09-20T22:57:49.397191+0000","last_unstale":"2024-09-20T22:58:06.267182+0000","last_undegraded":"2024-09-20T22:58:06.267182+0000","last_fullsized":"2024-09-20T22:58:06.267182+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-09-20T22:55:33.544218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:55:33.544218+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"77'128","reported_seq":1265,"reported_epoch":997,"state":"active+clean","last_fresh":"2024-09-20T22:58:08.374190+0000","last_change":"2024-09-20T22:56:43.780891+0000","last_active":"2024-09-20T22:58:08.374190+0000","last_peered":"2024-09-20T22:58:08.374190+0000","last_clean":"2024-09-20T22:58:08.374190+0000","last_became_active":"2024-09-20T22:56:41.368240+0000","last_became_peered":"2024-09-20T22:56:41.368240+0000","last_unstale":"2024-09-20T22:58:08.374190+0000","last_undegraded":"2024-09-20T22:58:08.374190+0000","last_fullsized":"2024-09-20T22:58:08.374190+0000","mapping_epoch":920,"log_start":"77'128","ondisk_log_start":"77'128","created":19,"last_epoch_clean":921,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:56:43.780781+0000","last_deep_scrub":"77'128","last_deep_scrub_stamp":"2024-09-20T22:47:22.417099+0000","last_clean_scrub_stamp":"2024-09-20T22:56:43.780781+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:12.539710+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":236,"reported_epoch":996,"state":"active+clean","last_fresh":"2024-09-20T22:58:07.790511+0000","last_change":"2024-09-20T22:58:02.248882+0000","last_active":"2024-09-20T22:58:07.790511+0000","last_peered":"2024-09-20T22:58:07.790511+0000","last_clean":"2024-09-20T22:58:07.790511+0000","last_became_active":"2024-09-20T22:56:08.366671+0000","last_became_peered":"2024-09-20T22:56:08.366671+0000","last_unstale":"2024-09-20T22:58:07.790511+0000","last_undegraded":"2024-09-20T22:58:07.790511+0000","last_fullsized":"2024-09-20T22:58:07.790511+0000","mapping_epoch":887,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:58:02.248808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:58:02.248808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:20.341275+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":1931,"reported_epoch":997,"state":"active+clean","last_fresh":"2024-09-20T22:58:08.376883+0000","last_change":"2024-09-20T22:57:49.410905+0000","last_active":"2024-09-20T22:58:08.376883+0000","last_peered":"2024-09-20T22:58:08.376883+0000","last_clean":"2024-09-20T22:58:08.376883+0000","last_became_active":"2024-09-20T22:57:49.410488+0000","last_became_peered":"2024-09-20T22:57:49.410488+0000","last_unstale":"2024-09-20T22:58:08.376883+0000","last_undegraded":"2024-09-20T22:58:08.376883+0000","last_fullsized":"2024-09-20T22:58:08.376883+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:55:50.483056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:55:50.483056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"988'954","reported_seq":5594,"reported_epoch":997,"state":"active+clean","last_fresh":"2024-09-20T22:58:08.373092+0000","last_change":"2024-09-20T22:57:48.439778+0000","last_active":"2024-09-20T22:58:08.373092+0000","last_peered":"2024-09-20T22:58:08.373092+0000","last_clean":"2024-09-20T22:58:08.373092+0000","last_became_active":"2024-09-20T22:54:20.414528+0000","last_became_peered":"2024-09-20T22:54:20.414528+0000","last_unstale":"2024-09-20T22:58:08.373092+0000","last_undegraded":"2024-09-20T22:58:08.373092+0000","last_fullsized":"2024-09-20T22:58:08.373092+0000","mapping_epoch":779,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":780,"parent":"0.0","parent_split_bits":0,"last_scrub":"970'914","last_scrub_stamp":"2024-09-20T22:57:31.284226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:57:31.284226+0000","objects_scrubbed":16,"log_size":954,"log_dups_size":0,"ondisk_log_size":954,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:37.068423+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.064640501000000003,"stat_sum":{"num_bytes":34316407,"num_objects":14,"num_object_clones":12,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":291,"num_read_kb":291495,"num_write":147,"num_write_kb":69585,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":88,"num_bytes_recovered":51113242,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"1b5","length":"1"},{"start":"1b7","length":"1"},{"start":"1ba","length":"1"},{"start":"1bd","length":"2"},{"start":"1c1","length":"1"},{"start":"1c5","length":"1"},{"start":"1c7","length":"1"}]},{"pgid":"4.e","version":"850'24","reported_seq":850,"reported_epoch":997,"state":"active+clean","last_fresh":"2024-09-20T22:58:08.373186+0000","last_change":"2024-09-20T22:57:38.226530+0000","last_active":"2024-09-20T22:58:08.373186+0000","last_peered":"2024-09-20T22:58:08.373186+0000","last_clean":"2024-09-20T22:58:08.373186+0000","last_became_active":"2024-09-20T22:55:17.342390+0000","last_became_peered":"2024-09-20T22:55:17.342390+0000","last_unstale":"2024-09-20T22:58:08.373186+0000","last_undegraded":"2024-09-20T22:58:08.373186+0000","last_fullsized":"2024-09-20T22:58:08.373186+0000","mapping_epoch":836,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":837,"parent":"0.0","parent_split_bits":0,"last_scrub":"850'24","last_scrub_stamp":"2024-09-20T22:57:38.226418+0000","last_deep_scrub":"850'24","last_deep_scrub_stamp":"2024-09-20T22:55:30.779018+0000","last_clean_scrub_stamp":"2024-09-20T22:57:38.226418+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:49.332407+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":23,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":862,"reported_epoch":996,"state":"active+clean","last_fresh":"2024-09-20T22:58:07.789401+0000","last_change":"2024-09-20T22:56:17.135450+0000","last_active":"2024-09-20T22:58:07.789401+0000","last_peered":"2024-09-20T22:58:07.789401+0000","last_clean":"2024-09-20T22:58:07.789401+0000","last_became_active":"2024-09-20T22:56:08.359647+0000","last_became_peered":"2024-09-20T22:56:08.359647+0000","last_unstale":"2024-09-20T22:58:07.789401+0000","last_undegraded":"2024-09-20T22:58:07.789401+0000","last_fullsized":"2024-09-20T22:58:07.789401+0000","mapping_epoch":887,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:56:17.135379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:56:17.135379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:12.237787+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":1926,"reported_epoch":997,"state":"active+clean","last_fresh":"2024-09-20T22:58:08.376785+0000","last_change":"2024-09-20T22:57:11.628063+0000","last_active":"2024-09-20T22:58:08.376785+0000","last_peered":"2024-09-20T22:58:08.376785+0000","last_clean":"2024-09-20T22:58:08.376785+0000","last_became_active":"2024-09-20T22:48:05.233778+0000","last_became_peered":"2024-09-20T22:48:05.233778+0000","last_unstale":"2024-09-20T22:58:08.376785+0000","last_undegraded":"2024-09-20T22:58:08.376785+0000","last_fullsized":"2024-09-20T22:58:08.376785+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:57:11.627990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:54:47.166836+0000","last_clean_scrub_stamp":"2024-09-20T22:57:11.627990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:34.275838+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":1784,"reported_epoch":995,"state":"active+clean","last_fresh":"2024-09-20T22:58:06.267916+0000","last_change":"2024-09-20T22:57:49.427590+0000","last_active":"2024-09-20T22:58:06.267916+0000","last_peered":"2024-09-20T22:58:06.267916+0000","last_clean":"2024-09-20T22:58:06.267916+0000","last_became_active":"2024-09-20T22:57:49.425373+0000","last_became_peered":"2024-09-20T22:57:49.425373+0000","last_unstale":"2024-09-20T22:58:06.267916+0000","last_undegraded":"2024-09-20T22:58:06.267916+0000","last_fullsized":"2024-09-20T22:58:06.267916+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:56:46.453177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:48:40.127123+0000","last_clean_scrub_stamp":"2024-09-20T22:56:46.453177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.e","version":"988'1072","reported_seq":5564,"reported_epoch":996,"state":"active+clean","last_fresh":"2024-09-20T22:58:07.790350+0000","last_change":"2024-09-20T22:57:48.441312+0000","last_active":"2024-09-20T22:58:07.790350+0000","last_peered":"2024-09-20T22:58:07.790350+0000","last_clean":"2024-09-20T22:58:07.790350+0000","last_became_active":"2024-09-20T22:47:52.209452+0000","last_became_peered":"2024-09-20T22:47:52.209452+0000","last_unstale":"2024-09-20T22:58:07.790350+0000","last_undegraded":"2024-09-20T22:58:07.790350+0000","last_fullsized":"2024-09-20T22:58:07.790350+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"944'1032","last_scrub_stamp":"2024-09-20T22:57:05.131799+0000","last_deep_scrub":"174'60","last_deep_scrub_stamp":"2024-09-20T22:44:18.301776+0000","last_clean_scrub_stamp":"2024-09-20T22:57:05.131799+0000","objects_scrubbed":14,"log_size":1072,"log_dups_size":0,"ondisk_log_size":1072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:28.855536+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.057156603,"stat_sum":{"num_bytes":32496050,"num_objects":14,"num_object_clones":12,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":2,"num_read":306,"num_read_kb":300627,"num_write":228,"num_write_kb":83627,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":23110741,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"1a6","length":"1"},{"start":"1aa","length":"1"},{"start":"1b5","length":"1"},{"start":"1b7","length":"1"},{"start":"1ba","length":"1"},{"start":"1bd","length":"2"},{"start":"1c0","length":"2"},{"start":"1c5","length":"1"},{"start":"1c7","length":"1"}]},{"pgid":"1.8","version":"77'128","reported_seq":2200,"reported_epoch":997,"state":"active+clean","last_fresh":"2024-09-20T22:58:08.375679+0000","last_change":"2024-09-20T22:57:58.769515+0000","last_active":"2024-09-20T22:58:08.375679+0000","last_peered":"2024-09-20T22:58:08.375679+0000","last_clean":"2024-09-20T22:58:08.375679+0000","last_became_active":"2024-09-20T22:57:49.408874+0000","last_became_peered":"2024-09-20T22:57:49.408874+0000","last_unstale":"2024-09-20T22:58:08.375679+0000","last_undegraded":"2024-09-20T22:58:08.375679+0000","last_fullsized":"2024-09-20T22:58:08.375679+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:57:58.769384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:57:58.769384+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:12.464680+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"625'31","reported_seq":875,"reported_epoch":996,"state":"active+clean","last_fresh":"2024-09-20T22:58:07.791096+0000","last_change":"2024-09-20T22:57:36.368878+0000","last_active":"2024-09-20T22:58:07.791096+0000","last_peered":"2024-09-20T22:58:07.791096+0000","last_clean":"2024-09-20T22:58:07.791096+0000","last_became_active":"2024-09-20T22:55:17.342262+0000","last_became_peered":"2024-09-20T22:55:17.342262+0000","last_unstale":"2024-09-20T22:58:07.791096+0000","last_undegraded":"2024-09-20T22:58:07.791096+0000","last_fullsized":"2024-09-20T22:58:07.791096+0000","mapping_epoch":836,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":837,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'31","last_scrub_stamp":"2024-09-20T22:57:36.368756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:57:36.368756+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:53.304747+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":48,"num_read_kb":38,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":1488,"reported_epoch":996,"state":"active+clean","last_fresh":"2024-09-20T22:58:07.789754+0000","last_change":"2024-09-20T22:56:51.152802+0000","last_active":"2024-09-20T22:58:07.789754+0000","last_peered":"2024-09-20T22:58:07.789754+0000","last_clean":"2024-09-20T22:58:07.789754+0000","last_became_active":"2024-09-20T22:54:17.332636+0000","last_became_peered":"2024-09-20T22:54:17.332636+0000","last_unstale":"2024-09-20T22:58:07.789754+0000","last_undegraded":"2024-09-20T22:58:07.789754+0000","last_fullsized":"2024-09-20T22:58:07.789754+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:56:51.152732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:35.492562+0000","last_clean_scrub_stamp":"2024-09-20T22:56:51.152732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:18.450632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":466,"reported_epoch":997,"state":"active+clean","last_fresh":"2024-09-20T22:58:08.378495+0000","last_change":"2024-09-20T22:56:49.666908+0000","last_active":"2024-09-20T22:58:08.378495+0000","last_peered":"2024-09-20T22:58:08.378495+0000","last_clean":"2024-09-20T22:58:08.378495+0000","last_became_active":"2024-09-20T22:54:17.349144+0000","last_became_peered":"2024-09-20T22:54:17.349144+0000","last_unstale":"2024-09-20T22:58:08.378495+0000","last_undegraded":"2024-09-20T22:58:08.378495+0000","last_fullsized":"2024-09-20T22:58:08.378495+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:56:49.666824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:56:49.666824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:15.812144+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":1952,"reported_epoch":996,"state":"active+clean","last_fresh":"2024-09-20T22:58:07.791522+0000","last_change":"2024-09-20T22:57:49.411075+0000","last_active":"2024-09-20T22:58:07.791522+0000","last_peered":"2024-09-20T22:58:07.791522+0000","last_clean":"2024-09-20T22:58:07.791522+0000","last_became_active":"2024-09-20T22:57:49.410743+0000","last_became_peered":"2024-09-20T22:57:49.410743+0000","last_unstale":"2024-09-20T22:58:07.791522+0000","last_undegraded":"2024-09-20T22:58:07.791522+0000","last_fullsized":"2024-09-20T22:58:07.791522+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:56:43.023211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:56:43.023211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:12.237818+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"989'1180","reported_seq":5819,"reported_epoch":997,"state":"active+clean","last_fresh":"2024-09-20T22:58:08.375708+0000","last_change":"2024-09-20T22:57:49.680248+0000","last_active":"2024-09-20T22:58:08.375708+0000","last_peered":"2024-09-20T22:58:08.375708+0000","last_clean":"2024-09-20T22:58:08.375708+0000","last_became_active":"2024-09-20T22:57:49.415940+0000","last_became_peered":"2024-09-20T22:57:49.415940+0000","last_unstale":"2024-09-20T22:58:08.375708+0000","last_undegraded":"2024-09-20T22:58:08.375708+0000","last_fullsized":"2024-09-20T22:58:08.375708+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"868'991","last_scrub_stamp":"2024-09-20T22:55:49.469629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:55:49.469629+0000","objects_scrubbed":15,"log_size":1180,"log_dups_size":0,"ondisk_log_size":1180,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:12.539584+0000","scrub_duration":8,"objects_trimmed":1,"snaptrim_duration":0.058896384000000003,"stat_sum":{"num_bytes":41006889,"num_objects":19,"num_object_clones":16,"num_object_copies":57,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":483,"num_read_kb":511223,"num_write":268,"num_write_kb":92130,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":121,"num_bytes_recovered":52177428,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"1b5","length":"1"},{"start":"1b7","length":"1"},{"start":"1ba","length":"1"},{"start":"1bd","length":"2"},{"start":"1c1","length":"1"},{"start":"1c5","length":"1"},{"start":"1c7","length":"1"}]},{"pgid":"1.9","version":"77'128","reported_seq":461,"reported_epoch":995,"state":"active+clean","last_fresh":"2024-09-20T22:58:06.267060+0000","last_change":"2024-09-20T22:57:49.418054+0000","last_active":"2024-09-20T22:58:06.267060+0000","last_peered":"2024-09-20T22:58:06.267060+0000","last_clean":"2024-09-20T22:58:06.267060+0000","last_became_active":"2024-09-20T22:57:49.415117+0000","last_became_peered":"2024-09-20T22:57:49.415117+0000","last_unstale":"2024-09-20T22:58:06.267060+0000","last_undegraded":"2024-09-20T22:58:06.267060+0000","last_fullsized":"2024-09-20T22:58:06.267060+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:56:11.417116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:56:11.417116+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"877'25","reported_seq":698,"reported_epoch":995,"state":"active+clean","last_fresh":"2024-09-20T22:58:06.267786+0000","last_change":"2024-09-20T22:57:49.426619+0000","last_active":"2024-09-20T22:58:06.267786+0000","last_peered":"2024-09-20T22:58:06.267786+0000","last_clean":"2024-09-20T22:58:06.267786+0000","last_became_active":"2024-09-20T22:57:49.424440+0000","last_became_peered":"2024-09-20T22:57:49.424440+0000","last_unstale":"2024-09-20T22:58:06.267786+0000","last_undegraded":"2024-09-20T22:58:06.267786+0000","last_fullsized":"2024-09-20T22:58:06.267786+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"877'25","last_scrub_stamp":"2024-09-20T22:55:57.521557+0000","last_deep_scrub":"877'25","last_deep_scrub_stamp":"2024-09-20T22:55:57.521557+0000","last_clean_scrub_stamp":"2024-09-20T22:55:57.521557+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":28,"num_read_kb":23,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":1808,"reported_epoch":995,"state":"active+clean","last_fresh":"2024-09-20T22:58:06.267381+0000","last_change":"2024-09-20T22:57:49.433012+0000","last_active":"2024-09-20T22:58:06.267381+0000","last_peered":"2024-09-20T22:58:06.267381+0000","last_clean":"2024-09-20T22:58:06.267381+0000","last_became_active":"2024-09-20T22:57:49.432735+0000","last_became_peered":"2024-09-20T22:57:49.432735+0000","last_unstale":"2024-09-20T22:58:06.267381+0000","last_undegraded":"2024-09-20T22:58:06.267381+0000","last_fullsized":"2024-09-20T22:58:06.267381+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:55:58.553666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:55:58.553666+0000","last_clean_scrub_stamp":"2024-09-20T22:55:58.553666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"888'10","reported_seq":2153,"reported_epoch":996,"state":"active+clean","last_fresh":"2024-09-20T22:58:07.791047+0000","last_change":"2024-09-20T22:58:06.192595+0000","last_active":"2024-09-20T22:58:07.791047+0000","last_peered":"2024-09-20T22:58:07.791047+0000","last_clean":"2024-09-20T22:58:07.791047+0000","last_became_active":"2024-09-20T22:56:08.374292+0000","last_became_peered":"2024-09-20T22:56:08.374292+0000","last_unstale":"2024-09-20T22:58:07.791047+0000","last_undegraded":"2024-09-20T22:58:07.791047+0000","last_fullsized":"2024-09-20T22:58:07.791047+0000","mapping_epoch":887,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":0,"last_scrub":"888'10","last_scrub_stamp":"2024-09-20T22:58:06.192521+0000","last_deep_scrub":"888'10","last_deep_scrub_stamp":"2024-09-20T22:58:06.192521+0000","last_clean_scrub_stamp":"2024-09-20T22:58:06.192521+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:35.529081+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":472,"reported_epoch":997,"state":"active+clean","last_fresh":"2024-09-20T22:58:08.379311+0000","last_change":"2024-09-20T22:56:23.542870+0000","last_active":"2024-09-20T22:58:08.379311+0000","last_peered":"2024-09-20T22:58:08.379311+0000","last_clean":"2024-09-20T22:58:08.379311+0000","last_became_active":"2024-09-20T22:56:01.477943+0000","last_became_peered":"2024-09-20T22:56:01.477943+0000","last_unstale":"2024-09-20T22:58:08.379311+0000","last_undegraded":"2024-09-20T22:58:08.379311+0000","last_fullsized":"2024-09-20T22:58:08.379311+0000","mapping_epoch":880,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":881,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:56:23.542803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:56:23.542803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:11.406413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.8","version":"989'1442","reported_seq":5808,"reported_epoch":996,"state":"active+clean","last_fresh":"2024-09-20T22:58:07.788954+0000","last_change":"2024-09-20T22:57:57.346215+0000","last_active":"2024-09-20T22:58:07.788954+0000","last_peered":"2024-09-20T22:58:07.788954+0000","last_clean":"2024-09-20T22:58:07.788954+0000","last_became_active":"2024-09-20T22:57:49.415389+0000","last_became_peered":"2024-09-20T22:57:49.415389+0000","last_unstale":"2024-09-20T22:58:07.788954+0000","last_undegraded":"2024-09-20T22:58:07.788954+0000","last_fullsized":"2024-09-20T22:58:07.788954+0000","mapping_epoch":988,"log_start":"426'268","ondisk_log_start":"426'268","created":94,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"989'1442","last_scrub_stamp":"2024-09-20T22:57:57.346049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:57:57.346049+0000","objects_scrubbed":9,"log_size":1174,"log_dups_size":445,"ondisk_log_size":1174,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:02.128887+0000","scrub_duration":5,"objects_trimmed":1,"snaptrim_duration":0.052166095000000003,"stat_sum":{"num_bytes":18691964,"num_objects":9,"num_object_clones":8,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"1a6","length":"1"},{"start":"1aa","length":"1"},{"start":"1b5","length":"1"},{"start":"1b7","length":"1"},{"start":"1ba","length":"1"},{"start":"1bd","length":"2"},{"start":"1c0","length":"2"},{"start":"1c5","length":"1"},{"start":"1c7","length":"1"}]},{"pgid":"4.b","version":"625'15","reported_seq":1330,"reported_epoch":997,"state":"active+clean","last_fresh":"2024-09-20T22:58:08.375531+0000","last_change":"2024-09-20T22:55:32.601025+0000","last_active":"2024-09-20T22:58:08.375531+0000","last_peered":"2024-09-20T22:58:08.375531+0000","last_clean":"2024-09-20T22:58:08.375531+0000","last_became_active":"2024-09-20T22:54:17.340998+0000","last_became_peered":"2024-09-20T22:54:17.340998+0000","last_unstale":"2024-09-20T22:58:08.375531+0000","last_undegraded":"2024-09-20T22:58:08.375531+0000","last_fullsized":"2024-09-20T22:58:08.375531+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'15","last_scrub_stamp":"2024-09-20T22:55:32.600891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:55:32.600891+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:12.539556+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":1931,"reported_epoch":997,"state":"active+clean","last_fresh":"2024-09-20T22:58:08.376692+0000","last_change":"2024-09-20T22:57:49.425922+0000","last_active":"2024-09-20T22:58:08.376692+0000","last_peered":"2024-09-20T22:58:08.376692+0000","last_clean":"2024-09-20T22:58:08.376692+0000","last_became_active":"2024-09-20T22:57:49.425707+0000","last_became_peered":"2024-09-20T22:57:49.425707+0000","last_unstale":"2024-09-20T22:58:08.376692+0000","last_undegraded":"2024-09-20T22:58:08.376692+0000","last_fullsized":"2024-09-20T22:58:08.376692+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:56:24.518228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:56:24.518228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":1960,"reported_epoch":996,"state":"active+clean","last_fresh":"2024-09-20T22:58:07.789841+0000","last_change":"2024-09-20T22:57:55.320989+0000","last_active":"2024-09-20T22:58:07.789841+0000","last_peered":"2024-09-20T22:58:07.789841+0000","last_clean":"2024-09-20T22:58:07.789841+0000","last_became_active":"2024-09-20T22:57:49.411210+0000","last_became_peered":"2024-09-20T22:57:49.411210+0000","last_unstale":"2024-09-20T22:58:07.789841+0000","last_undegraded":"2024-09-20T22:58:07.789841+0000","last_fullsized":"2024-09-20T22:58:07.789841+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:57:55.320885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:57:55.320885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:55.718488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":1278,"reported_epoch":997,"state":"active+clean","last_fresh":"2024-09-20T22:58:08.374781+0000","last_change":"2024-09-20T22:57:49.411393+0000","last_active":"2024-09-20T22:58:08.374781+0000","last_peered":"2024-09-20T22:58:08.374781+0000","last_clean":"2024-09-20T22:58:08.374781+0000","last_became_active":"2024-09-20T22:57:49.410896+0000","last_became_peered":"2024-09-20T22:57:49.410896+0000","last_unstale":"2024-09-20T22:58:08.374781+0000","last_undegraded":"2024-09-20T22:58:08.374781+0000","last_fullsized":"2024-09-20T22:58:08.374781+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:56:44.736139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:56:44.736139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:12.539725+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"988'1928","reported_seq":6491,"reported_epoch":997,"state":"active+clean","last_fresh":"2024-09-20T22:58:08.374913+0000","last_change":"2024-09-20T22:57:48.442076+0000","last_active":"2024-09-20T22:58:08.374913+0000","last_peered":"2024-09-20T22:58:08.374913+0000","last_clean":"2024-09-20T22:58:08.374913+0000","last_became_active":"2024-09-20T22:56:53.367443+0000","last_became_peered":"2024-09-20T22:56:53.367443+0000","last_unstale":"2024-09-20T22:58:08.374913+0000","last_undegraded":"2024-09-20T22:58:08.374913+0000","last_fullsized":"2024-09-20T22:58:08.374913+0000","mapping_epoch":779,"log_start":"423'280","ondisk_log_start":"423'280","created":94,"last_epoch_clean":933,"parent":"0.0","parent_split_bits":0,"last_scrub":"944'1882","last_scrub_stamp":"2024-09-20T22:57:05.737732+0000","last_deep_scrub":"590'828","last_deep_scrub_stamp":"2024-09-20T22:51:10.909364+0000","last_clean_scrub_stamp":"2024-09-20T22:57:05.737732+0000","objects_scrubbed":14,"log_size":1648,"log_dups_size":649,"ondisk_log_size":1648,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:35.520582+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.064944348999999998,"stat_sum":{"num_bytes":31503774,"num_objects":11,"num_object_clones":9,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":1,"num_read":18,"num_read_kb":22008,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"1b5","length":"1"},{"start":"1b7","length":"1"},{"start":"1ba","length":"1"},{"start":"1bd","length":"2"},{"start":"1c1","length":"1"},{"start":"1c5","length":"1"},{"start":"1c7","length":"1"}]},{"pgid":"4.a","version":"625'44","reported_seq":2081,"reported_epoch":997,"state":"active+clean","last_fresh":"2024-09-20T22:58:08.372490+0000","last_change":"2024-09-20T22:57:06.158084+0000","last_active":"2024-09-20T22:58:08.372490+0000","last_peered":"2024-09-20T22:58:08.372490+0000","last_clean":"2024-09-20T22:58:08.372490+0000","last_became_active":"2024-09-20T22:56:08.348928+0000","last_became_peered":"2024-09-20T22:56:08.348928+0000","last_unstale":"2024-09-20T22:58:08.372490+0000","last_undegraded":"2024-09-20T22:58:08.372490+0000","last_fullsized":"2024-09-20T22:58:08.372490+0000","mapping_epoch":887,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'44","last_scrub_stamp":"2024-09-20T22:57:06.157986+0000","last_deep_scrub":"113'28","last_deep_scrub_stamp":"2024-09-20T22:43:12.725900+0000","last_clean_scrub_stamp":"2024-09-20T22:57:06.157986+0000","objects_scrubbed":9,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:18.682809+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":238,"reported_epoch":997,"state":"active+clean","last_fresh":"2024-09-20T22:58:08.376391+0000","last_change":"2024-09-20T22:57:49.404176+0000","last_active":"2024-09-20T22:58:08.376391+0000","last_peered":"2024-09-20T22:58:08.376391+0000","last_clean":"2024-09-20T22:58:08.376391+0000","last_became_active":"2024-09-20T22:57:49.403858+0000","last_became_peered":"2024-09-20T22:57:49.403858+0000","last_unstale":"2024-09-20T22:58:08.376391+0000","last_undegraded":"2024-09-20T22:58:08.376391+0000","last_fullsized":"2024-09-20T22:58:08.376391+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:55:29.772589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:55:29.772589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":1939,"reported_epoch":996,"state":"active+clean","last_fresh":"2024-09-20T22:58:07.789529+0000","last_change":"2024-09-20T22:55:56.107908+0000","last_active":"2024-09-20T22:58:07.789529+0000","last_peered":"2024-09-20T22:58:07.789529+0000","last_clean":"2024-09-20T22:58:07.789529+0000","last_became_active":"2024-09-20T22:47:49.223220+0000","last_became_peered":"2024-09-20T22:47:49.223220+0000","last_unstale":"2024-09-20T22:58:07.789529+0000","last_undegraded":"2024-09-20T22:58:07.789529+0000","last_fullsized":"2024-09-20T22:58:07.789529+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:55:56.107827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:54:21.496068+0000","last_clean_scrub_stamp":"2024-09-20T22:55:56.107827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:12.237757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":236,"reported_epoch":996,"state":"active+clean","last_fresh":"2024-09-20T22:58:07.789020+0000","last_change":"2024-09-20T22:57:49.399320+0000","last_active":"2024-09-20T22:58:07.789020+0000","last_peered":"2024-09-20T22:58:07.789020+0000","last_clean":"2024-09-20T22:58:07.789020+0000","last_became_active":"2024-09-20T22:57:49.399016+0000","last_became_peered":"2024-09-20T22:57:49.399016+0000","last_unstale":"2024-09-20T22:58:07.789020+0000","last_undegraded":"2024-09-20T22:58:07.789020+0000","last_fullsized":"2024-09-20T22:58:07.789020+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:55:54.870627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:55:54.870627+0000","last_clean_scrub_stamp":"2024-09-20T22:55:54.870627+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:12.237772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.a","version":"989'1256","reported_seq":5521,"reported_epoch":995,"state":"active+clean","last_fresh":"2024-09-20T22:58:06.267242+0000","last_change":"2024-09-20T22:57:49.673268+0000","last_active":"2024-09-20T22:58:06.267242+0000","last_peered":"2024-09-20T22:58:06.267242+0000","last_clean":"2024-09-20T22:58:06.267242+0000","last_became_active":"2024-09-20T22:57:49.401412+0000","last_became_peered":"2024-09-20T22:57:49.401412+0000","last_unstale":"2024-09-20T22:58:06.267242+0000","last_undegraded":"2024-09-20T22:58:06.267242+0000","last_fullsized":"2024-09-20T22:58:06.267242+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"851'1076","last_scrub_stamp":"2024-09-20T22:55:32.567038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:55:32.567038+0000","objects_scrubbed":21,"log_size":1256,"log_dups_size":0,"ondisk_log_size":1256,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":5,"snaptrim_duration":0.120863633,"stat_sum":{"num_bytes":64995264,"num_objects":25,"num_object_clones":22,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":485,"num_read_kb":502840,"num_write":373,"num_write_kb":163696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":11877612,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"1a6","length":"1"},{"start":"1aa","length":"1"},{"start":"1b5","length":"1"},{"start":"1b7","length":"1"},{"start":"1ba","length":"1"},{"start":"1bd","length":"2"},{"start":"1c0","length":"2"},{"start":"1c5","length":"1"},{"start":"1c7","length":"1"}]},{"pgid":"4.9","version":"686'23","reported_seq":1178,"reported_epoch":997,"state":"active+clean","last_fresh":"2024-09-20T22:58:08.377838+0000","last_change":"2024-09-20T22:57:41.298315+0000","last_active":"2024-09-20T22:58:08.377838+0000","last_peered":"2024-09-20T22:58:08.377838+0000","last_clean":"2024-09-20T22:58:08.377838+0000","last_became_active":"2024-09-20T22:55:17.346757+0000","last_became_peered":"2024-09-20T22:55:17.346757+0000","last_unstale":"2024-09-20T22:58:08.377838+0000","last_undegraded":"2024-09-20T22:58:08.377838+0000","last_fullsized":"2024-09-20T22:58:08.377838+0000","mapping_epoch":836,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":837,"parent":"0.0","parent_split_bits":0,"last_scrub":"686'23","last_scrub_stamp":"2024-09-20T22:57:41.298234+0000","last_deep_scrub":"686'23","last_deep_scrub_stamp":"2024-09-20T22:52:46.768812+0000","last_clean_scrub_stamp":"2024-09-20T22:57:41.298234+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:09.189056+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":21,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":1743,"reported_epoch":997,"state":"active+clean","last_fresh":"2024-09-20T22:58:08.379211+0000","last_change":"2024-09-20T22:57:11.250944+0000","last_active":"2024-09-20T22:58:08.379211+0000","last_peered":"2024-09-20T22:58:08.379211+0000","last_clean":"2024-09-20T22:58:08.379211+0000","last_became_active":"2024-09-20T22:56:08.358713+0000","last_became_peered":"2024-09-20T22:56:08.358713+0000","last_unstale":"2024-09-20T22:58:08.379211+0000","last_undegraded":"2024-09-20T22:58:08.379211+0000","last_fullsized":"2024-09-20T22:58:08.379211+0000","mapping_epoch":887,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:57:11.250858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:46:52.465940+0000","last_clean_scrub_stamp":"2024-09-20T22:57:11.250858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:35.003707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":1959,"reported_epoch":996,"state":"active+clean","last_fresh":"2024-09-20T22:58:07.789119+0000","last_change":"2024-09-20T22:57:10.236664+0000","last_active":"2024-09-20T22:58:07.789119+0000","last_peered":"2024-09-20T22:58:07.789119+0000","last_clean":"2024-09-20T22:58:07.789119+0000","last_became_active":"2024-09-20T22:54:17.340269+0000","last_became_peered":"2024-09-20T22:54:17.340269+0000","last_unstale":"2024-09-20T22:58:07.789119+0000","last_undegraded":"2024-09-20T22:58:07.789119+0000","last_fullsized":"2024-09-20T22:58:07.789119+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:57:10.236592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:53:07.104891+0000","last_clean_scrub_stamp":"2024-09-20T22:57:10.236592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:35.494601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":1802,"reported_epoch":995,"state":"active+clean","last_fresh":"2024-09-20T22:58:06.267511+0000","last_change":"2024-09-20T22:57:49.422638+0000","last_active":"2024-09-20T22:58:06.267511+0000","last_peered":"2024-09-20T22:58:06.267511+0000","last_clean":"2024-09-20T22:58:06.267511+0000","last_became_active":"2024-09-20T22:57:49.419390+0000","last_became_peered":"2024-09-20T22:57:49.419390+0000","last_unstale":"2024-09-20T22:58:06.267511+0000","last_undegraded":"2024-09-20T22:58:06.267511+0000","last_fullsized":"2024-09-20T22:58:06.267511+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:53.242204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:55.170379+0000","last_clean_scrub_stamp":"2024-09-20T22:53:53.242204+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"993'1677","reported_seq":6238,"reported_epoch":997,"state":"active+clean","last_fresh":"2024-09-20T22:58:08.378622+0000","last_change":"2024-09-20T22:58:04.586380+0000","last_active":"2024-09-20T22:58:08.378622+0000","last_peered":"2024-09-20T22:58:08.378622+0000","last_clean":"2024-09-20T22:58:08.378622+0000","last_became_active":"2024-09-20T22:54:20.415706+0000","last_became_peered":"2024-09-20T22:54:20.415706+0000","last_unstale":"2024-09-20T22:58:08.378622+0000","last_undegraded":"2024-09-20T22:58:08.378622+0000","last_fullsized":"2024-09-20T22:58:08.378622+0000","mapping_epoch":779,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":780,"parent":"0.0","parent_split_bits":0,"last_scrub":"794'1255","last_scrub_stamp":"2024-09-20T22:54:37.866509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:54:37.866509+0000","objects_scrubbed":15,"log_size":1677,"log_dups_size":0,"ondisk_log_size":1677,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":1.2126732330000001,"stat_sum":{"num_bytes":69638720,"num_objects":33,"num_object_clones":28,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":2,"num_read":681,"num_read_kb":757216,"num_write":460,"num_write_kb":173264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":168,"num_bytes_recovered":68651320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1b5","length":"1"},{"start":"1b7","length":"1"},{"start":"1ba","length":"1"},{"start":"1bd","length":"2"},{"start":"1c1","length":"1"},{"start":"1c5","length":"1"},{"start":"1c7","length":"1"}]},{"pgid":"4.8","version":"625'27","reported_seq":1983,"reported_epoch":997,"state":"active+clean","last_fresh":"2024-09-20T22:58:08.375857+0000","last_change":"2024-09-20T22:57:10.649039+0000","last_active":"2024-09-20T22:58:08.375857+0000","last_peered":"2024-09-20T22:58:08.375857+0000","last_clean":"2024-09-20T22:58:08.375857+0000","last_became_active":"2024-09-20T22:56:08.360719+0000","last_became_peered":"2024-09-20T22:56:08.360719+0000","last_unstale":"2024-09-20T22:58:08.375857+0000","last_undegraded":"2024-09-20T22:58:08.375857+0000","last_fullsized":"2024-09-20T22:58:08.375857+0000","mapping_epoch":887,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'27","last_scrub_stamp":"2024-09-20T22:57:10.648909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:57:10.648909+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:13.332381+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'2","reported_seq":941,"reported_epoch":997,"state":"active+clean","last_fresh":"2024-09-20T22:58:08.375474+0000","last_change":"2024-09-20T22:56:08.422471+0000","last_active":"2024-09-20T22:58:08.375474+0000","last_peered":"2024-09-20T22:58:08.375474+0000","last_clean":"2024-09-20T22:58:08.375474+0000","last_became_active":"2024-09-20T22:56:08.392282+0000","last_became_peered":"2024-09-20T22:56:08.392282+0000","last_unstale":"2024-09-20T22:58:08.375474+0000","last_undegraded":"2024-09-20T22:58:08.375474+0000","last_fullsized":"2024-09-20T22:58:08.375474+0000","mapping_epoch":887,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-09-20T22:55:33.770279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:55:33.770279+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:12.539539+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":276,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.9","version":"888'11","reported_seq":289,"reported_epoch":997,"state":"active+clean","last_fresh":"2024-09-20T22:58:08.372585+0000","last_change":"2024-09-20T22:56:48.156966+0000","last_active":"2024-09-20T22:58:08.372585+0000","last_peered":"2024-09-20T22:58:08.372585+0000","last_clean":"2024-09-20T22:58:08.372585+0000","last_became_active":"2024-09-20T22:56:08.357520+0000","last_became_peered":"2024-09-20T22:56:08.357520+0000","last_unstale":"2024-09-20T22:58:08.372585+0000","last_undegraded":"2024-09-20T22:58:08.372585+0000","last_fullsized":"2024-09-20T22:58:08.372585+0000","mapping_epoch":887,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":0,"last_scrub":"888'11","last_scrub_stamp":"2024-09-20T22:56:48.156843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:56:48.156843+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":1922,"reported_epoch":997,"state":"active+clean","last_fresh":"2024-09-20T22:58:08.376026+0000","last_change":"2024-09-20T22:57:14.615994+0000","last_active":"2024-09-20T22:58:08.376026+0000","last_peered":"2024-09-20T22:58:08.376026+0000","last_clean":"2024-09-20T22:58:08.376026+0000","last_became_active":"2024-09-20T22:47:49.227569+0000","last_became_peered":"2024-09-20T22:47:49.227569+0000","last_unstale":"2024-09-20T22:58:08.376026+0000","last_undegraded":"2024-09-20T22:58:08.376026+0000","last_fullsized":"2024-09-20T22:58:08.376026+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:57:14.615909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:57:14.615909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:29.335354+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.14","version":"989'730","reported_seq":3769,"reported_epoch":995,"state":"active+clean","last_fresh":"2024-09-20T22:58:06.268023+0000","last_change":"2024-09-20T22:57:59.723457+0000","last_active":"2024-09-20T22:58:06.268023+0000","last_peered":"2024-09-20T22:58:06.268023+0000","last_clean":"2024-09-20T22:58:06.268023+0000","last_became_active":"2024-09-20T22:57:51.380711+0000","last_became_peered":"2024-09-20T22:57:51.380711+0000","last_unstale":"2024-09-20T22:58:06.268023+0000","last_undegraded":"2024-09-20T22:58:06.268023+0000","last_fullsized":"2024-09-20T22:58:06.268023+0000","mapping_epoch":990,"log_start":"456'176","ondisk_log_start":"456'176","created":932,"last_epoch_clean":991,"parent":"0.0","parent_split_bits":5,"last_scrub":"989'730","last_scrub_stamp":"2024-09-20T22:57:59.723364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:57:59.723364+0000","objects_scrubbed":6,"log_size":554,"log_dups_size":318,"ondisk_log_size":554,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:21.548392+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.037993079999999999,"stat_sum":{"num_bytes":13662167,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"1a6","length":"1"},{"start":"1aa","length":"1"},{"start":"1b5","length":"1"},{"start":"1b7","length":"1"},{"start":"1ba","length":"1"},{"start":"1bd","length":"2"},{"start":"1c0","length":"2"},{"start":"1c5","length":"1"},{"start":"1c7","length":"1"}]},{"pgid":"3.10","version":"0'0","reported_seq":1963,"reported_epoch":997,"state":"active+clean","last_fresh":"2024-09-20T22:58:08.372437+0000","last_change":"2024-09-20T22:58:05.337254+0000","last_active":"2024-09-20T22:58:08.372437+0000","last_peered":"2024-09-20T22:58:08.372437+0000","last_clean":"2024-09-20T22:58:08.372437+0000","last_became_active":"2024-09-20T22:47:49.213292+0000","last_became_peered":"2024-09-20T22:47:49.213292+0000","last_unstale":"2024-09-20T22:58:08.372437+0000","last_undegraded":"2024-09-20T22:58:08.372437+0000","last_fullsized":"2024-09-20T22:58:08.372437+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:58:05.337127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:58:05.337127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:15.915595+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"625'12","reported_seq":1868,"reported_epoch":995,"state":"active+clean","last_fresh":"2024-09-20T22:58:06.268178+0000","last_change":"2024-09-20T22:58:00.707463+0000","last_active":"2024-09-20T22:58:06.268178+0000","last_peered":"2024-09-20T22:58:06.268178+0000","last_clean":"2024-09-20T22:58:06.268178+0000","last_became_active":"2024-09-20T22:57:49.426479+0000","last_became_peered":"2024-09-20T22:57:49.426479+0000","last_unstale":"2024-09-20T22:58:06.268178+0000","last_undegraded":"2024-09-20T22:58:06.268178+0000","last_fullsized":"2024-09-20T22:58:06.268178+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'12","last_scrub_stamp":"2024-09-20T22:58:00.707335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:58:00.707335+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:25.721332+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":1934,"reported_epoch":997,"state":"active+clean","last_fresh":"2024-09-20T22:58:08.376984+0000","last_change":"2024-09-20T22:57:49.408789+0000","last_active":"2024-09-20T22:58:08.376984+0000","last_peered":"2024-09-20T22:58:08.376984+0000","last_clean":"2024-09-20T22:58:08.376984+0000","last_became_active":"2024-09-20T22:57:49.408529+0000","last_became_peered":"2024-09-20T22:57:49.408529+0000","last_unstale":"2024-09-20T22:58:08.376984+0000","last_undegraded":"2024-09-20T22:58:08.376984+0000","last_fullsized":"2024-09-20T22:58:08.376984+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:55:29.324286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:09.705632+0000","last_clean_scrub_stamp":"2024-09-20T22:55:29.324286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"74'1","reported_seq":250,"reported_epoch":997,"state":"active+clean","last_fresh":"2024-09-20T22:58:08.373179+0000","last_change":"2024-09-20T22:56:47.178707+0000","last_active":"2024-09-20T22:58:08.373179+0000","last_peered":"2024-09-20T22:58:08.373179+0000","last_clean":"2024-09-20T22:58:08.373179+0000","last_became_active":"2024-09-20T22:56:08.354987+0000","last_became_peered":"2024-09-20T22:56:08.354987+0000","last_unstale":"2024-09-20T22:58:08.373179+0000","last_undegraded":"2024-09-20T22:58:08.373179+0000","last_fullsized":"2024-09-20T22:58:08.373179+0000","mapping_epoch":887,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-09-20T22:56:47.178627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:56:47.178627+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:11.125701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.15","version":"930'540","reported_seq":5112,"reported_epoch":997,"state":"active+clean","last_fresh":"2024-09-20T22:58:08.378122+0000","last_change":"2024-09-20T22:57:48.374841+0000","last_active":"2024-09-20T22:58:08.378122+0000","last_peered":"2024-09-20T22:58:08.378122+0000","last_clean":"2024-09-20T22:58:08.378122+0000","last_became_active":"2024-09-20T22:56:58.367157+0000","last_became_peered":"2024-09-20T22:56:58.367157+0000","last_unstale":"2024-09-20T22:58:08.378122+0000","last_undegraded":"2024-09-20T22:58:08.378122+0000","last_fullsized":"2024-09-20T22:58:08.378122+0000","mapping_epoch":937,"log_start":"450'190","ondisk_log_start":"450'190","created":932,"last_epoch_clean":938,"parent":"0.0","parent_split_bits":5,"last_scrub":"930'540","last_scrub_stamp":"2024-09-20T22:57:04.343926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:57:04.343926+0000","objects_scrubbed":0,"log_size":350,"log_dups_size":240,"ondisk_log_size":350,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:18.165862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011806360000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1b5","length":"1"},{"start":"1b7","length":"1"},{"start":"1ba","length":"1"},{"start":"1bd","length":"2"},{"start":"1c1","length":"1"},{"start":"1c5","length":"1"},{"start":"1c7","length":"1"}]},{"pgid":"4.16","version":"625'15","reported_seq":1307,"reported_epoch":997,"state":"active+clean","last_fresh":"2024-09-20T22:58:08.375781+0000","last_change":"2024-09-20T22:57:49.413436+0000","last_active":"2024-09-20T22:58:08.375781+0000","last_peered":"2024-09-20T22:58:08.375781+0000","last_clean":"2024-09-20T22:58:08.375781+0000","last_became_active":"2024-09-20T22:57:49.412096+0000","last_became_peered":"2024-09-20T22:57:49.412096+0000","last_unstale":"2024-09-20T22:58:08.375781+0000","last_undegraded":"2024-09-20T22:58:08.375781+0000","last_fullsized":"2024-09-20T22:58:08.375781+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'15","last_scrub_stamp":"2024-09-20T22:55:27.637946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:55:27.637946+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:12.539521+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":954,"reported_epoch":997,"state":"active+clean","last_fresh":"2024-09-20T22:58:08.372927+0000","last_change":"2024-09-20T22:56:11.967360+0000","last_active":"2024-09-20T22:58:08.372927+0000","last_peered":"2024-09-20T22:58:08.372927+0000","last_clean":"2024-09-20T22:58:08.372927+0000","last_became_active":"2024-09-20T22:56:08.357576+0000","last_became_peered":"2024-09-20T22:56:08.357576+0000","last_unstale":"2024-09-20T22:58:08.372927+0000","last_undegraded":"2024-09-20T22:58:08.372927+0000","last_fullsized":"2024-09-20T22:58:08.372927+0000","mapping_epoch":887,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:56:11.967266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:56:11.967266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":1814,"reported_epoch":995,"state":"active+clean","last_fresh":"2024-09-20T22:58:06.267976+0000","last_change":"2024-09-20T22:57:49.433827+0000","last_active":"2024-09-20T22:58:06.267976+0000","last_peered":"2024-09-20T22:58:06.267976+0000","last_clean":"2024-09-20T22:58:06.267976+0000","last_became_active":"2024-09-20T22:57:49.433318+0000","last_became_peered":"2024-09-20T22:57:49.433318+0000","last_unstale":"2024-09-20T22:58:06.267976+0000","last_undegraded":"2024-09-20T22:58:06.267976+0000","last_fullsized":"2024-09-20T22:58:06.267976+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:55:52.481737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:55:52.481737+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":1960,"reported_epoch":996,"state":"active+clean","last_fresh":"2024-09-20T22:58:07.790296+0000","last_change":"2024-09-20T22:56:42.057480+0000","last_active":"2024-09-20T22:58:07.790296+0000","last_peered":"2024-09-20T22:58:07.790296+0000","last_clean":"2024-09-20T22:58:07.790296+0000","last_became_active":"2024-09-20T22:56:08.369854+0000","last_became_peered":"2024-09-20T22:56:08.369854+0000","last_unstale":"2024-09-20T22:58:07.790296+0000","last_undegraded":"2024-09-20T22:58:07.790296+0000","last_fullsized":"2024-09-20T22:58:07.790296+0000","mapping_epoch":887,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:56:42.057380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:53:58.169894+0000","last_clean_scrub_stamp":"2024-09-20T22:56:42.057380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:12.237802+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.16","version":"988'903","reported_seq":5434,"reported_epoch":997,"state":"active+clean","last_fresh":"2024-09-20T22:58:08.375157+0000","last_change":"2024-09-20T22:57:48.439743+0000","last_active":"2024-09-20T22:58:08.375157+0000","last_peered":"2024-09-20T22:58:08.375157+0000","last_clean":"2024-09-20T22:58:08.375157+0000","last_became_active":"2024-09-20T22:56:58.372517+0000","last_became_peered":"2024-09-20T22:56:58.372517+0000","last_unstale":"2024-09-20T22:58:08.375157+0000","last_undegraded":"2024-09-20T22:58:08.375157+0000","last_fullsized":"2024-09-20T22:58:08.375157+0000","mapping_epoch":937,"log_start":"443'181","ondisk_log_start":"443'181","created":932,"last_epoch_clean":938,"parent":"0.0","parent_split_bits":5,"last_scrub":"944'873","last_scrub_stamp":"2024-09-20T22:57:07.764870+0000","last_deep_scrub":"902'811","last_deep_scrub_stamp":"2024-09-20T22:56:22.658684+0000","last_clean_scrub_stamp":"2024-09-20T22:57:07.764870+0000","objects_scrubbed":12,"log_size":722,"log_dups_size":314,"ondisk_log_size":722,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:18.543615+0000","scrub_duration":3,"objects_trimmed":1,"snaptrim_duration":0.070922279000000005,"stat_sum":{"num_bytes":21972561,"num_objects":9,"num_object_clones":8,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"1b5","length":"1"},{"start":"1b7","length":"1"},{"start":"1ba","length":"1"},{"start":"1bd","length":"2"},{"start":"1c1","length":"1"},{"start":"1c5","length":"1"},{"start":"1c7","length":"1"}]},{"pgid":"4.15","version":"625'15","reported_seq":1967,"reported_epoch":997,"state":"active+clean","last_fresh":"2024-09-20T22:58:08.376435+0000","last_change":"2024-09-20T22:57:49.404308+0000","last_active":"2024-09-20T22:58:08.376435+0000","last_peered":"2024-09-20T22:58:08.376435+0000","last_clean":"2024-09-20T22:58:08.376435+0000","last_became_active":"2024-09-20T22:57:49.403807+0000","last_became_peered":"2024-09-20T22:57:49.403807+0000","last_unstale":"2024-09-20T22:58:08.376435+0000","last_undegraded":"2024-09-20T22:58:08.376435+0000","last_fullsized":"2024-09-20T22:58:08.376435+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'15","last_scrub_stamp":"2024-09-20T22:56:15.488641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:56:15.488641+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":1286,"reported_epoch":997,"state":"active+clean","last_fresh":"2024-09-20T22:58:08.374896+0000","last_change":"2024-09-20T22:57:49.399263+0000","last_active":"2024-09-20T22:58:08.374896+0000","last_peered":"2024-09-20T22:58:08.374896+0000","last_clean":"2024-09-20T22:58:08.374896+0000","last_became_active":"2024-09-20T22:57:49.398995+0000","last_became_peered":"2024-09-20T22:57:49.398995+0000","last_unstale":"2024-09-20T22:58:08.374896+0000","last_undegraded":"2024-09-20T22:58:08.374896+0000","last_fullsized":"2024-09-20T22:58:08.374896+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:56:16.523567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:56:16.523567+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:12.539693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"989'11","reported_seq":1996,"reported_epoch":995,"state":"active+clean","last_fresh":"2024-09-20T22:58:06.267682+0000","last_change":"2024-09-20T22:57:49.432210+0000","last_active":"2024-09-20T22:58:06.267682+0000","last_peered":"2024-09-20T22:58:06.267682+0000","last_clean":"2024-09-20T22:58:06.267682+0000","last_became_active":"2024-09-20T22:57:49.430688+0000","last_became_peered":"2024-09-20T22:57:49.430688+0000","last_unstale":"2024-09-20T22:58:06.267682+0000","last_undegraded":"2024-09-20T22:58:06.267682+0000","last_fullsized":"2024-09-20T22:58:06.267682+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-09-20T22:53:59.165134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:53:59.165134+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":885,"reported_epoch":996,"state":"active+clean","last_fresh":"2024-09-20T22:58:07.791440+0000","last_change":"2024-09-20T22:58:05.222409+0000","last_active":"2024-09-20T22:58:07.791440+0000","last_peered":"2024-09-20T22:58:07.791440+0000","last_clean":"2024-09-20T22:58:07.791440+0000","last_became_active":"2024-09-20T22:55:17.346270+0000","last_became_peered":"2024-09-20T22:55:17.346270+0000","last_unstale":"2024-09-20T22:58:07.791440+0000","last_undegraded":"2024-09-20T22:58:07.791440+0000","last_fullsized":"2024-09-20T22:58:07.791440+0000","mapping_epoch":836,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":837,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:58:05.222339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:58:05.222339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:33.466206+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,5],"acting":[2,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.17","version":"990'2020","reported_seq":6521,"reported_epoch":997,"state":"active+clean","last_fresh":"2024-09-20T22:58:08.377070+0000","last_change":"2024-09-20T22:57:52.717318+0000","last_active":"2024-09-20T22:58:08.377070+0000","last_peered":"2024-09-20T22:58:08.377070+0000","last_clean":"2024-09-20T22:58:08.377070+0000","last_became_active":"2024-09-20T22:57:52.378717+0000","last_became_peered":"2024-09-20T22:57:52.378717+0000","last_unstale":"2024-09-20T22:58:08.377070+0000","last_undegraded":"2024-09-20T22:58:08.377070+0000","last_fullsized":"2024-09-20T22:58:08.377070+0000","mapping_epoch":991,"log_start":"436'456","ondisk_log_start":"436'456","created":932,"last_epoch_clean":992,"parent":"0.0","parent_split_bits":5,"last_scrub":"990'2020","last_scrub_stamp":"2024-09-20T22:57:52.717196+0000","last_deep_scrub":"879'1788","last_deep_scrub_stamp":"2024-09-20T22:55:59.964830+0000","last_clean_scrub_stamp":"2024-09-20T22:57:52.717196+0000","objects_scrubbed":10,"log_size":1564,"log_dups_size":886,"ondisk_log_size":1564,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:10.038236+0000","scrub_duration":3,"objects_trimmed":1,"snaptrim_duration":0.034776473000000002,"stat_sum":{"num_bytes":26840801,"num_objects":10,"num_object_clones":8,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"1b5","length":"1"},{"start":"1b7","length":"1"},{"start":"1ba","length":"1"},{"start":"1bd","length":"2"},{"start":"1c1","length":"1"},{"start":"1c5","length":"1"},{"start":"1c7","length":"1"}]},{"pgid":"4.14","version":"625'20","reported_seq":1892,"reported_epoch":995,"state":"active+clean","last_fresh":"2024-09-20T22:58:06.267651+0000","last_change":"2024-09-20T22:57:49.431205+0000","last_active":"2024-09-20T22:58:06.267651+0000","last_peered":"2024-09-20T22:58:06.267651+0000","last_clean":"2024-09-20T22:58:06.267651+0000","last_became_active":"2024-09-20T22:57:49.427276+0000","last_became_peered":"2024-09-20T22:57:49.427276+0000","last_unstale":"2024-09-20T22:58:06.267651+0000","last_undegraded":"2024-09-20T22:58:06.267651+0000","last_fullsized":"2024-09-20T22:58:06.267651+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'20","last_scrub_stamp":"2024-09-20T22:55:23.474362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:55:23.474362+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":942,"reported_epoch":996,"state":"active+clean","last_fresh":"2024-09-20T22:58:07.789613+0000","last_change":"2024-09-20T22:56:08.372321+0000","last_active":"2024-09-20T22:58:07.789613+0000","last_peered":"2024-09-20T22:58:07.789613+0000","last_clean":"2024-09-20T22:58:07.789613+0000","last_became_active":"2024-09-20T22:56:08.371946+0000","last_became_peered":"2024-09-20T22:56:08.371946+0000","last_unstale":"2024-09-20T22:58:07.789613+0000","last_undegraded":"2024-09-20T22:58:07.789613+0000","last_fullsized":"2024-09-20T22:58:07.789613+0000","mapping_epoch":887,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:55:49.897082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:55:49.897082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:12.237721+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.15","version":"778'12","reported_seq":2172,"reported_epoch":997,"state":"active+clean","last_fresh":"2024-09-20T22:58:08.376337+0000","last_change":"2024-09-20T22:57:36.726648+0000","last_active":"2024-09-20T22:58:08.376337+0000","last_peered":"2024-09-20T22:58:08.376337+0000","last_clean":"2024-09-20T22:58:08.376337+0000","last_became_active":"2024-09-20T22:54:17.341014+0000","last_became_peered":"2024-09-20T22:54:17.341014+0000","last_unstale":"2024-09-20T22:58:08.376337+0000","last_undegraded":"2024-09-20T22:58:08.376337+0000","last_fullsized":"2024-09-20T22:58:08.376337+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"778'12","last_scrub_stamp":"2024-09-20T22:57:36.726568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:57:36.726568+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:52.611026+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":1278,"reported_epoch":997,"state":"active+clean","last_fresh":"2024-09-20T22:58:08.375638+0000","last_change":"2024-09-20T22:57:49.412393+0000","last_active":"2024-09-20T22:58:08.375638+0000","last_peered":"2024-09-20T22:58:08.375638+0000","last_clean":"2024-09-20T22:58:08.375638+0000","last_became_active":"2024-09-20T22:57:49.411010+0000","last_became_peered":"2024-09-20T22:57:49.411010+0000","last_unstale":"2024-09-20T22:58:08.375638+0000","last_undegraded":"2024-09-20T22:58:08.375638+0000","last_fullsized":"2024-09-20T22:58:08.375638+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:56:49.691267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:50:21.853070+0000","last_clean_scrub_stamp":"2024-09-20T22:56:49.691267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:12.539755+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.10","version":"989'1011","reported_seq":5455,"reported_epoch":995,"state":"active+clean","last_fresh":"2024-09-20T22:58:06.267369+0000","last_change":"2024-09-20T22:57:55.682134+0000","last_active":"2024-09-20T22:58:06.267369+0000","last_peered":"2024-09-20T22:58:06.267369+0000","last_clean":"2024-09-20T22:58:06.267369+0000","last_became_active":"2024-09-20T22:57:49.398604+0000","last_became_peered":"2024-09-20T22:57:49.398604+0000","last_unstale":"2024-09-20T22:58:06.267369+0000","last_undegraded":"2024-09-20T22:58:06.267369+0000","last_fullsized":"2024-09-20T22:58:06.267369+0000","mapping_epoch":988,"log_start":"480'253","ondisk_log_start":"480'253","created":932,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":5,"last_scrub":"989'1011","last_scrub_stamp":"2024-09-20T22:57:55.682037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:57:55.682037+0000","objects_scrubbed":8,"log_size":758,"log_dups_size":383,"ondisk_log_size":758,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:16.360864+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.080082789000000001,"stat_sum":{"num_bytes":15927188,"num_objects":8,"num_object_clones":7,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"1a6","length":"1"},{"start":"1aa","length":"1"},{"start":"1b5","length":"1"},{"start":"1b7","length":"1"},{"start":"1ba","length":"1"},{"start":"1bd","length":"2"},{"start":"1c0","length":"2"},{"start":"1c5","length":"1"},{"start":"1c7","length":"1"}]},{"pgid":"4.13","version":"625'19","reported_seq":853,"reported_epoch":997,"state":"active+clean","last_fresh":"2024-09-20T22:58:08.372813+0000","last_change":"2024-09-20T22:57:59.380762+0000","last_active":"2024-09-20T22:58:08.372813+0000","last_peered":"2024-09-20T22:58:08.372813+0000","last_clean":"2024-09-20T22:58:08.372813+0000","last_became_active":"2024-09-20T22:55:17.342351+0000","last_became_peered":"2024-09-20T22:55:17.342351+0000","last_unstale":"2024-09-20T22:58:08.372813+0000","last_undegraded":"2024-09-20T22:58:08.372813+0000","last_fullsized":"2024-09-20T22:58:08.372813+0000","mapping_epoch":836,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":837,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'19","last_scrub_stamp":"2024-09-20T22:57:59.380659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:57:59.380659+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:17.347804+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":23,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":746,"reported_epoch":997,"state":"active+clean","last_fresh":"2024-09-20T22:58:08.372878+0000","last_change":"2024-09-20T22:57:15.158099+0000","last_active":"2024-09-20T22:58:08.372878+0000","last_peered":"2024-09-20T22:58:08.372878+0000","last_clean":"2024-09-20T22:58:08.372878+0000","last_became_active":"2024-09-20T22:56:08.369206+0000","last_became_peered":"2024-09-20T22:56:08.369206+0000","last_unstale":"2024-09-20T22:58:08.372878+0000","last_undegraded":"2024-09-20T22:58:08.372878+0000","last_fullsized":"2024-09-20T22:58:08.372878+0000","mapping_epoch":887,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:57:15.158013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:52:47.717490+0000","last_clean_scrub_stamp":"2024-09-20T22:57:15.158013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:32.667409+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":475,"reported_epoch":997,"state":"active+clean","last_fresh":"2024-09-20T22:58:08.378774+0000","last_change":"2024-09-20T22:57:55.455704+0000","last_active":"2024-09-20T22:58:08.378774+0000","last_peered":"2024-09-20T22:58:08.378774+0000","last_clean":"2024-09-20T22:58:08.378774+0000","last_became_active":"2024-09-20T22:57:49.398369+0000","last_became_peered":"2024-09-20T22:57:49.398369+0000","last_unstale":"2024-09-20T22:58:08.378774+0000","last_undegraded":"2024-09-20T22:58:08.378774+0000","last_fullsized":"2024-09-20T22:58:08.378774+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:57:55.455633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:57:55.455633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:04.115059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":1748,"reported_epoch":995,"state":"active+clean","last_fresh":"2024-09-20T22:58:06.266834+0000","last_change":"2024-09-20T22:57:49.416021+0000","last_active":"2024-09-20T22:58:06.266834+0000","last_peered":"2024-09-20T22:58:06.266834+0000","last_clean":"2024-09-20T22:58:06.266834+0000","last_became_active":"2024-09-20T22:57:49.413148+0000","last_became_peered":"2024-09-20T22:57:49.413148+0000","last_unstale":"2024-09-20T22:58:06.266834+0000","last_undegraded":"2024-09-20T22:58:06.266834+0000","last_fullsized":"2024-09-20T22:58:06.266834+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:56:22.490523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:56:22.490523+0000","last_clean_scrub_stamp":"2024-09-20T22:56:22.490523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"929'565","reported_seq":5001,"reported_epoch":997,"state":"active+clean","last_fresh":"2024-09-20T22:58:08.372228+0000","last_change":"2024-09-20T22:57:50.386505+0000","last_active":"2024-09-20T22:58:08.372228+0000","last_peered":"2024-09-20T22:58:08.372228+0000","last_clean":"2024-09-20T22:58:08.372228+0000","last_became_active":"2024-09-20T22:57:50.385308+0000","last_became_peered":"2024-09-20T22:57:50.385308+0000","last_unstale":"2024-09-20T22:58:08.372228+0000","last_undegraded":"2024-09-20T22:58:08.372228+0000","last_fullsized":"2024-09-20T22:58:08.372228+0000","mapping_epoch":989,"log_start":"474'259","ondisk_log_start":"474'259","created":932,"last_epoch_clean":990,"parent":"0.0","parent_split_bits":5,"last_scrub":"929'565","last_scrub_stamp":"2024-09-20T22:57:33.220990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:57:33.220990+0000","objects_scrubbed":0,"log_size":306,"log_dups_size":338,"ondisk_log_size":306,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:02.936257+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00070306199999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"1b5","length":"1"},{"start":"1b7","length":"1"},{"start":"1ba","length":"1"},{"start":"1bd","length":"2"},{"start":"1c1","length":"1"},{"start":"1c5","length":"1"},{"start":"1c7","length":"1"}]},{"pgid":"4.12","version":"625'19","reported_seq":476,"reported_epoch":997,"state":"active+clean","last_fresh":"2024-09-20T22:58:08.379040+0000","last_change":"2024-09-20T22:58:03.394296+0000","last_active":"2024-09-20T22:58:08.379040+0000","last_peered":"2024-09-20T22:58:08.379040+0000","last_clean":"2024-09-20T22:58:08.379040+0000","last_became_active":"2024-09-20T22:54:17.340539+0000","last_became_peered":"2024-09-20T22:54:17.340539+0000","last_unstale":"2024-09-20T22:58:08.379040+0000","last_undegraded":"2024-09-20T22:58:08.379040+0000","last_fullsized":"2024-09-20T22:58:08.379040+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'19","last_scrub_stamp":"2024-09-20T22:58:03.394221+0000","last_deep_scrub":"404'13","last_deep_scrub_stamp":"2024-09-20T22:48:04.276282+0000","last_clean_scrub_stamp":"2024-09-20T22:58:03.394221+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:25.311781+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":842,"reported_epoch":995,"state":"active+clean","last_fresh":"2024-09-20T22:58:06.267733+0000","last_change":"2024-09-20T22:57:49.406406+0000","last_active":"2024-09-20T22:58:06.267733+0000","last_peered":"2024-09-20T22:58:06.267733+0000","last_clean":"2024-09-20T22:58:06.267733+0000","last_became_active":"2024-09-20T22:57:49.403043+0000","last_became_peered":"2024-09-20T22:57:49.403043+0000","last_unstale":"2024-09-20T22:58:06.267733+0000","last_undegraded":"2024-09-20T22:58:06.267733+0000","last_fullsized":"2024-09-20T22:58:06.267733+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:54:46.541109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:31.138067+0000","last_clean_scrub_stamp":"2024-09-20T22:54:46.541109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":1762,"reported_epoch":995,"state":"active+clean","last_fresh":"2024-09-20T22:58:06.267329+0000","last_change":"2024-09-20T22:58:03.746286+0000","last_active":"2024-09-20T22:58:06.267329+0000","last_peered":"2024-09-20T22:58:06.267329+0000","last_clean":"2024-09-20T22:58:06.267329+0000","last_became_active":"2024-09-20T22:57:49.432838+0000","last_became_peered":"2024-09-20T22:57:49.432838+0000","last_unstale":"2024-09-20T22:58:06.267329+0000","last_undegraded":"2024-09-20T22:58:06.267329+0000","last_fullsized":"2024-09-20T22:58:06.267329+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:58:03.746169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:58:03.746169+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:27.258169+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":1275,"reported_epoch":997,"state":"active+clean","last_fresh":"2024-09-20T22:58:08.374517+0000","last_change":"2024-09-20T22:57:38.552117+0000","last_active":"2024-09-20T22:58:08.374517+0000","last_peered":"2024-09-20T22:58:08.374517+0000","last_clean":"2024-09-20T22:58:08.374517+0000","last_became_active":"2024-09-20T22:54:17.344021+0000","last_became_peered":"2024-09-20T22:54:17.344021+0000","last_unstale":"2024-09-20T22:58:08.374517+0000","last_undegraded":"2024-09-20T22:58:08.374517+0000","last_fullsized":"2024-09-20T22:58:08.374517+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:57:38.552046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:55:09.664839+0000","last_clean_scrub_stamp":"2024-09-20T22:57:38.552046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:07.271978+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.12","version":"988'1150","reported_seq":5064,"reported_epoch":997,"state":"active+clean","last_fresh":"2024-09-20T22:58:08.372296+0000","last_change":"2024-09-20T22:57:48.417172+0000","last_active":"2024-09-20T22:58:08.372296+0000","last_peered":"2024-09-20T22:58:08.372296+0000","last_clean":"2024-09-20T22:58:08.372296+0000","last_became_active":"2024-09-20T22:57:03.429039+0000","last_became_peered":"2024-09-20T22:57:03.429039+0000","last_unstale":"2024-09-20T22:58:08.372296+0000","last_undegraded":"2024-09-20T22:58:08.372296+0000","last_fullsized":"2024-09-20T22:58:08.372296+0000","mapping_epoch":942,"log_start":"471'464","ondisk_log_start":"471'464","created":932,"last_epoch_clean":943,"parent":"0.0","parent_split_bits":5,"last_scrub":"943'1027","last_scrub_stamp":"2024-09-20T22:57:05.204065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:57:05.204065+0000","objects_scrubbed":15,"log_size":686,"log_dups_size":687,"ondisk_log_size":686,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:17.391441+0000","scrub_duration":6,"objects_trimmed":2,"snaptrim_duration":0.046814117000000002,"stat_sum":{"num_bytes":45111058,"num_objects":18,"num_object_clones":15,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":1,"num_read":29,"num_read_kb":39201,"num_write":15,"num_write_kb":14614,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"1b5","length":"1"},{"start":"1b7","length":"1"},{"start":"1ba","length":"1"},{"start":"1bd","length":"2"},{"start":"1c1","length":"1"},{"start":"1c5","length":"1"},{"start":"1c7","length":"1"}]},{"pgid":"4.11","version":"625'24","reported_seq":139,"reported_epoch":995,"state":"active+clean","last_fresh":"2024-09-20T22:58:06.266961+0000","last_change":"2024-09-20T22:58:04.738164+0000","last_active":"2024-09-20T22:58:06.266961+0000","last_peered":"2024-09-20T22:58:06.266961+0000","last_clean":"2024-09-20T22:58:06.266961+0000","last_became_active":"2024-09-20T22:57:49.408666+0000","last_became_peered":"2024-09-20T22:57:49.408666+0000","last_unstale":"2024-09-20T22:58:06.266961+0000","last_undegraded":"2024-09-20T22:58:06.266961+0000","last_fullsized":"2024-09-20T22:58:06.266961+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'24","last_scrub_stamp":"2024-09-20T22:58:04.738032+0000","last_deep_scrub":"470'16","last_deep_scrub_stamp":"2024-09-20T22:49:10.659868+0000","last_clean_scrub_stamp":"2024-09-20T22:58:04.738032+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:09.058684+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":1948,"reported_epoch":997,"state":"active+clean","last_fresh":"2024-09-20T22:58:08.376903+0000","last_change":"2024-09-20T22:57:49.405502+0000","last_active":"2024-09-20T22:58:08.376903+0000","last_peered":"2024-09-20T22:58:08.376903+0000","last_clean":"2024-09-20T22:58:08.376903+0000","last_became_active":"2024-09-20T22:57:49.405256+0000","last_became_peered":"2024-09-20T22:57:49.405256+0000","last_unstale":"2024-09-20T22:58:08.376903+0000","last_undegraded":"2024-09-20T22:58:08.376903+0000","last_fullsized":"2024-09-20T22:58:08.376903+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:55:54.521725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:45:24.765414+0000","last_clean_scrub_stamp":"2024-09-20T22:55:54.521725+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":946,"reported_epoch":997,"state":"active+clean","last_fresh":"2024-09-20T22:58:08.372069+0000","last_change":"2024-09-20T22:57:36.239427+0000","last_active":"2024-09-20T22:58:08.372069+0000","last_peered":"2024-09-20T22:58:08.372069+0000","last_clean":"2024-09-20T22:58:08.372069+0000","last_became_active":"2024-09-20T22:56:08.355950+0000","last_became_peered":"2024-09-20T22:56:08.355950+0000","last_unstale":"2024-09-20T22:58:08.372069+0000","last_undegraded":"2024-09-20T22:58:08.372069+0000","last_fullsized":"2024-09-20T22:58:08.372069+0000","mapping_epoch":887,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:57:36.239352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:50.174861+0000","last_clean_scrub_stamp":"2024-09-20T22:57:36.239352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:38.372074+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":1746,"reported_epoch":995,"state":"active+clean","last_fresh":"2024-09-20T22:58:06.267418+0000","last_change":"2024-09-20T22:57:49.419702+0000","last_active":"2024-09-20T22:58:06.267418+0000","last_peered":"2024-09-20T22:58:06.267418+0000","last_clean":"2024-09-20T22:58:06.267418+0000","last_became_active":"2024-09-20T22:57:49.410970+0000","last_became_peered":"2024-09-20T22:57:49.410970+0000","last_unstale":"2024-09-20T22:58:06.267418+0000","last_undegraded":"2024-09-20T22:58:06.267418+0000","last_fullsized":"2024-09-20T22:58:06.267418+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:54:22.276321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:54:22.276321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"988'504","reported_seq":4899,"reported_epoch":997,"state":"active+clean","last_fresh":"2024-09-20T22:58:08.375196+0000","last_change":"2024-09-20T22:57:48.420084+0000","last_active":"2024-09-20T22:58:08.375196+0000","last_peered":"2024-09-20T22:58:08.375196+0000","last_clean":"2024-09-20T22:58:08.375196+0000","last_became_active":"2024-09-20T22:57:03.429099+0000","last_became_peered":"2024-09-20T22:57:03.429099+0000","last_unstale":"2024-09-20T22:58:08.375196+0000","last_undegraded":"2024-09-20T22:58:08.375196+0000","last_fullsized":"2024-09-20T22:58:08.375196+0000","mapping_epoch":942,"log_start":"456'129","ondisk_log_start":"456'129","created":932,"last_epoch_clean":943,"parent":"0.0","parent_split_bits":5,"last_scrub":"944'488","last_scrub_stamp":"2024-09-20T22:57:04.743360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:42:51.354519+0000","last_clean_scrub_stamp":"2024-09-20T22:57:04.743360+0000","objects_scrubbed":9,"log_size":375,"log_dups_size":149,"ondisk_log_size":375,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:31.671359+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.051105540999999997,"stat_sum":{"num_bytes":18736221,"num_objects":8,"num_object_clones":7,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":19,"num_read_kb":37757,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"1b5","length":"1"},{"start":"1b7","length":"1"},{"start":"1ba","length":"1"},{"start":"1bd","length":"2"},{"start":"1c1","length":"1"},{"start":"1c5","length":"1"},{"start":"1c7","length":"1"}]},{"pgid":"4.10","version":"625'8","reported_seq":1818,"reported_epoch":995,"state":"active+clean","last_fresh":"2024-09-20T22:58:06.267109+0000","last_change":"2024-09-20T22:57:49.412934+0000","last_active":"2024-09-20T22:58:06.267109+0000","last_peered":"2024-09-20T22:58:06.267109+0000","last_clean":"2024-09-20T22:58:06.267109+0000","last_became_active":"2024-09-20T22:57:49.401068+0000","last_became_peered":"2024-09-20T22:57:49.401068+0000","last_unstale":"2024-09-20T22:58:06.267109+0000","last_undegraded":"2024-09-20T22:58:06.267109+0000","last_fullsized":"2024-09-20T22:58:06.267109+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'8","last_scrub_stamp":"2024-09-20T22:56:42.475598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:56:42.475598+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":1280,"reported_epoch":997,"state":"active+clean","last_fresh":"2024-09-20T22:58:08.375498+0000","last_change":"2024-09-20T22:58:02.611377+0000","last_active":"2024-09-20T22:58:08.375498+0000","last_peered":"2024-09-20T22:58:08.375498+0000","last_clean":"2024-09-20T22:58:08.375498+0000","last_became_active":"2024-09-20T22:57:49.412134+0000","last_became_peered":"2024-09-20T22:57:49.412134+0000","last_unstale":"2024-09-20T22:58:08.375498+0000","last_undegraded":"2024-09-20T22:58:08.375498+0000","last_fullsized":"2024-09-20T22:58:08.375498+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:58:02.611308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:58:02.611308+0000","last_clean_scrub_stamp":"2024-09-20T22:58:02.611308+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:16.278853+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":1965,"reported_epoch":997,"state":"active+clean","last_fresh":"2024-09-20T22:58:08.372693+0000","last_change":"2024-09-20T22:56:39.112737+0000","last_active":"2024-09-20T22:58:08.372693+0000","last_peered":"2024-09-20T22:58:08.372693+0000","last_clean":"2024-09-20T22:58:08.372693+0000","last_became_active":"2024-09-20T22:56:08.357087+0000","last_became_peered":"2024-09-20T22:56:08.357087+0000","last_unstale":"2024-09-20T22:58:08.372693+0000","last_undegraded":"2024-09-20T22:58:08.372693+0000","last_fullsized":"2024-09-20T22:58:08.372693+0000","mapping_epoch":887,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:56:39.112639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:56:39.112639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":939,"reported_epoch":996,"state":"active+clean","last_fresh":"2024-09-20T22:58:07.789937+0000","last_change":"2024-09-20T22:57:49.398452+0000","last_active":"2024-09-20T22:58:07.789937+0000","last_peered":"2024-09-20T22:58:07.789937+0000","last_clean":"2024-09-20T22:58:07.789937+0000","last_became_active":"2024-09-20T22:57:49.398162+0000","last_became_peered":"2024-09-20T22:57:49.398162+0000","last_unstale":"2024-09-20T22:58:07.789937+0000","last_undegraded":"2024-09-20T22:58:07.789937+0000","last_fullsized":"2024-09-20T22:58:07.789937+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:56:40.102311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:55:08.681729+0000","last_clean_scrub_stamp":"2024-09-20T22:56:40.102311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:08.020100+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":466,"reported_epoch":997,"state":"active+clean","last_fresh":"2024-09-20T22:58:08.379030+0000","last_change":"2024-09-20T22:57:09.252912+0000","last_active":"2024-09-20T22:58:08.379030+0000","last_peered":"2024-09-20T22:58:08.379030+0000","last_clean":"2024-09-20T22:58:08.379030+0000","last_became_active":"2024-09-20T22:54:17.348206+0000","last_became_peered":"2024-09-20T22:54:17.348206+0000","last_unstale":"2024-09-20T22:58:08.379030+0000","last_undegraded":"2024-09-20T22:58:08.379030+0000","last_fullsized":"2024-09-20T22:58:08.379030+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:57:09.252832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:57:09.252832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:21.209966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":1776,"reported_epoch":995,"state":"active+clean","last_fresh":"2024-09-20T22:58:06.267212+0000","last_change":"2024-09-20T22:57:49.413803+0000","last_active":"2024-09-20T22:58:06.267212+0000","last_peered":"2024-09-20T22:58:06.267212+0000","last_clean":"2024-09-20T22:58:06.267212+0000","last_became_active":"2024-09-20T22:57:49.407194+0000","last_became_peered":"2024-09-20T22:57:49.407194+0000","last_unstale":"2024-09-20T22:58:06.267212+0000","last_undegraded":"2024-09-20T22:58:06.267212+0000","last_fullsized":"2024-09-20T22:58:06.267212+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:53:31.274390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:53:31.274390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"625'19","reported_seq":2017,"reported_epoch":997,"state":"active+clean","last_fresh":"2024-09-20T22:58:08.373509+0000","last_change":"2024-09-20T22:58:04.391050+0000","last_active":"2024-09-20T22:58:08.373509+0000","last_peered":"2024-09-20T22:58:08.373509+0000","last_clean":"2024-09-20T22:58:08.373509+0000","last_became_active":"2024-09-20T22:54:17.342236+0000","last_became_peered":"2024-09-20T22:54:17.342236+0000","last_unstale":"2024-09-20T22:58:08.373509+0000","last_undegraded":"2024-09-20T22:58:08.373509+0000","last_fullsized":"2024-09-20T22:58:08.373509+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'19","last_scrub_stamp":"2024-09-20T22:58:04.390954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:58:04.390954+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:15.161588+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":1270,"reported_epoch":997,"state":"active+clean","last_fresh":"2024-09-20T22:58:08.375455+0000","last_change":"2024-09-20T22:57:08.720376+0000","last_active":"2024-09-20T22:58:08.375455+0000","last_peered":"2024-09-20T22:58:08.375455+0000","last_clean":"2024-09-20T22:58:08.375455+0000","last_became_active":"2024-09-20T22:56:08.391532+0000","last_became_peered":"2024-09-20T22:56:08.391532+0000","last_unstale":"2024-09-20T22:58:08.375455+0000","last_undegraded":"2024-09-20T22:58:08.375455+0000","last_fullsized":"2024-09-20T22:58:08.375455+0000","mapping_epoch":887,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:57:08.720315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:54:43.813540+0000","last_clean_scrub_stamp":"2024-09-20T22:57:08.720315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:58:09.299138+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":26,"stat_sum":{"num_bytes":818997415,"num_objects":351,"num_object_clones":301,"num_object_copies":1053,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":351,"num_whiteouts":10,"num_read":3520,"num_read_kb":3690506,"num_write":2308,"num_write_kb":919337,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":604,"num_bytes_recovered":294182561,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2069483520,"data_stored":2670190378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":25819,"ondisk_log_size":25819,"up":78,"acting":78,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1096,"num_read_kb":853,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":429,"num_bytes_recovered":9874,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":462848,"data_stored":12360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":770,"ondisk_log_size":770,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":1034,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":4931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":865,"num_read_kb":860,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":1594,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":385,"ondisk_log_size":385,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":0,"seq":253403070666,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430416,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":421292,"kb_used_data":336020,"kb_used_omap":1,"kb_used_meta":85246,"kb_avail":93323860,"statfs":{"total":95995035648,"available":95563632640,"internally_reserved":0,"allocated":344084480,"data_stored":439778060,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":87292358},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798692053,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":366104,"kb_used_data":294656,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93379048,"statfs":{"total":95995035648,"available":95620145152,"internally_reserved":0,"allocated":301727744,"data_stored":351448761,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986394,"num_pgs":73,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":169956,"kb_used_data":124880,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":93575196,"statfs":{"total":95995035648,"available":95821000704,"internally_reserved":0,"allocated":127877120,"data_stored":165590382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":988,"seq":4243427688453,"num_pgs":79,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":380128,"kb_used_data":345356,"kb_used_omap":148,"kb_used_meta":34603,"kb_avail":93365024,"statfs":{"total":95995035648,"available":95605784576,"internally_reserved":0,"allocated":353644544,"data_stored":460864391,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":151956,"internal_metadata":35434092},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411557,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":448448,"kb_used_data":363176,"kb_used_omap":1,"kb_used_meta":85246,"kb_avail":93296704,"statfs":{"total":95995035648,"available":95535824896,"internally_reserved":0,"allocated":371892224,"data_stored":442916745,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":87292357},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575082,"num_pgs":79,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":270192,"kb_used_data":194584,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93474960,"statfs":{"total":95995035648,"available":95718359040,"internally_reserved":0,"allocated":199254016,"data_stored":251141769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738607,"num_pgs":73,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":359836,"kb_used_data":274564,"kb_used_omap":1,"kb_used_meta":85246,"kb_avail":93385316,"statfs":{"total":95995035648,"available":95626563584,"internally_reserved":0,"allocated":281153536,"data_stored":362055336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":87292357},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":271130624,"data_stored":355911924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":189190144,"data_stored":244997782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":361631744,"data_stored":436518233,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":343044096,"data_stored":454143932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":117862400,"data_stored":159454854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":291127296,"data_stored":344728379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":333766656,"data_stored":433314464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":161730560,"data_stored":241120810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-20T22:58:11.610 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_0 2024-09-20T22:58:11.611 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd pool rm unique_pool_0 unique_pool_0 --yes-i-really-really-mean-it 2024-09-20T22:58:11.686 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:58:11.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:58:11.708 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:58:11.708 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:58:11.809 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:58:11.809 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:58:11.821 INFO:teuthology.orchestra.run.smithi084.stdout:{ 2024-09-20T22:58:11.821 INFO:teuthology.orchestra.run.smithi084.stdout: "ops": [], 2024-09-20T22:58:11.821 INFO:teuthology.orchestra.run.smithi084.stdout: "complaint_time": 30, 2024-09-20T22:58:11.822 INFO:teuthology.orchestra.run.smithi084.stdout: "num_blocked_ops": 0 2024-09-20T22:58:11.822 INFO:teuthology.orchestra.run.smithi084.stdout:} 2024-09-20T22:58:11.833 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:11 smithi084 ceph-mon[40024]: Health check failed: Degraded data redundancy: 8/1650 objects degraded (0.485%), 1 pg degraded (PG_DEGRADED) 2024-09-20T22:58:11.833 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:11 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:58:11.833 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:11 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:58:11.834 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:11 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:58:11.834 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:11 smithi084 ceph-mon[29774]: Health check failed: Degraded data redundancy: 8/1650 objects degraded (0.485%), 1 pg degraded (PG_DEGRADED) 2024-09-20T22:58:11.834 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:11 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:58:11.834 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:11 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:58:11.835 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:11 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:58:11.835 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:58:11.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:58:11.836 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:58:11.937 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:58:11.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:58:11.959 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:58:11.959 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:58:12.060 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:58:12.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:58:12.083 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:58:12.083 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:58:12.184 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:58:12.205 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:58:12.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:58:12.206 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:58:12.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:11 smithi154 ceph-mon[33264]: Health check failed: Degraded data redundancy: 8/1650 objects degraded (0.485%), 1 pg degraded (PG_DEGRADED) 2024-09-20T22:58:12.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:11 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:58:12.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:11 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:58:12.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:11 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:58:12.307 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:58:12.330 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:58:12.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:58:12.331 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:58:12.431 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:58:12.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:58:12.459 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:58:12.459 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:58:12.559 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:58:12.579 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:58:12.584 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:58:12.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:58:12.585 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:58:12.686 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:58:12.707 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:58:12.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:58:12.708 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:58:12.808 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3 2024-09-20T22:58:12.825 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is set 2024-09-20T22:58:12.832 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:58:12.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:58:12.833 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.3' 2024-09-20T22:58:12.934 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:58:12.955 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:58:12.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:58:12.956 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:58:13.057 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:58:13.084 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:58:13.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:58:13.085 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:58:13.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:12 smithi084 ceph-mon[29774]: pgmap v1406: 168 pgs: 1 activating+degraded, 167 active+clean; 782 MiB data, 2.3 GiB used, 624 GiB / 626 GiB avail; 8/1650 objects degraded (0.485%) 2024-09-20T22:58:13.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:12 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/1700646359' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:58:13.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:12 smithi084 ceph-mon[40024]: pgmap v1406: 168 pgs: 1 activating+degraded, 167 active+clean; 782 MiB data, 2.3 GiB used, 624 GiB / 626 GiB avail; 8/1650 objects degraded (0.485%) 2024-09-20T22:58:13.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:12 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/1700646359' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:58:13.175 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-20T22:58:13.186 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:58:13.231 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.0 dump_historic_ops 2024-09-20T22:58:13.241 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:58:13.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:58:13.242 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:58:13.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:12 smithi154 ceph-mon[33264]: pgmap v1406: 168 pgs: 1 activating+degraded, 167 active+clean; 782 MiB data, 2.3 GiB used, 624 GiB / 626 GiB avail; 8/1650 objects degraded (0.485%) 2024-09-20T22:58:13.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:12 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/1700646359' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:58:13.342 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:58:13.366 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:58:13.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:58:13.367 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:58:13.467 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:58:13.489 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:58:13.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:58:13.490 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:58:13.590 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:58:13.617 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:58:13.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:58:13.618 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:58:13.718 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:58:13.744 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:58:13.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:58:13.745 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:58:13.846 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:58:13.867 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:58:13.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:58:13.868 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:58:13.969 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:58:13.995 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:58:13.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:58:13.996 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:58:14.022 INFO:tasks.thrashosds.thrasher:in_osds: [2, 3, 5, 6, 0, 4, 1] out_osds: [7] dead_osds: [] live_osds: [0, 1, 2, 4, 5, 6, 7, 3] 2024-09-20T22:58:14.023 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-20T22:58:14.023 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-20T22:58:14.023 INFO:tasks.thrashosds.thrasher:Setting osd 5 primary_affinity to 0.880205 2024-09-20T22:58:14.023 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd primary-affinity 5 0.88020494279675 2024-09-20T22:58:14.097 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:58:14.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:13 smithi084 ceph-mon[29774]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:58:14.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:13 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/1700646359' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T22:58:14.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:13 smithi084 ceph-mon[29774]: osdmap e999: 8 total, 8 up, 7 in 2024-09-20T22:58:14.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:13 smithi084 ceph-mon[29774]: osdmap e1000: 8 total, 8 up, 7 in 2024-09-20T22:58:14.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:13 smithi084 ceph-mon[40024]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:58:14.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:13 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/1700646359' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T22:58:14.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:13 smithi084 ceph-mon[40024]: osdmap e999: 8 total, 8 up, 7 in 2024-09-20T22:58:14.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:13 smithi084 ceph-mon[40024]: osdmap e1000: 8 total, 8 up, 7 in 2024-09-20T22:58:14.121 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:58:14.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:58:14.122 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:58:14.192 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:58:14.223 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:58:14.235 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:58:14.248 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:58:14.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:58:14.249 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:58:14.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:13 smithi154 ceph-mon[33264]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:58:14.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:13 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/1700646359' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-20T22:58:14.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:13 smithi154 ceph-mon[33264]: osdmap e999: 8 total, 8 up, 7 in 2024-09-20T22:58:14.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:13 smithi154 ceph-mon[33264]: osdmap e1000: 8 total, 8 up, 7 in 2024-09-20T22:58:14.350 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:58:14.371 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:58:14.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:58:14.372 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:58:14.473 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:58:14.495 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:58:14.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:58:14.495 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:58:14.596 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:58:14.622 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:58:14.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:58:14.623 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:58:14.724 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:58:14.744 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:58:14.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:58:14.745 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:58:14.839 INFO:teuthology.orchestra.run.smithi084.stderr:pool 'unique_pool_0' removed 2024-09-20T22:58:14.846 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:58:14.872 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:58:14.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:58:14.874 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:58:14.897 INFO:teuthology.orchestra.run.smithi084.stdout:osd.0: {} 2024-09-20T22:58:14.897 INFO:teuthology.orchestra.run.smithi084.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-20T22:58:14.914 INFO:teuthology.orchestra.run.smithi084.stdout:osd.1: {} 2024-09-20T22:58:14.914 INFO:teuthology.orchestra.run.smithi084.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-20T22:58:14.930 INFO:teuthology.orchestra.run.smithi084.stdout:osd.2: {} 2024-09-20T22:58:14.930 INFO:teuthology.orchestra.run.smithi084.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-20T22:58:14.946 INFO:teuthology.orchestra.run.smithi084.stdout:osd.3: {} 2024-09-20T22:58:14.946 INFO:teuthology.orchestra.run.smithi084.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-20T22:58:14.963 INFO:teuthology.orchestra.run.smithi084.stdout:osd.4: {} 2024-09-20T22:58:14.963 INFO:teuthology.orchestra.run.smithi084.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-20T22:58:14.975 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:58:14.980 INFO:teuthology.orchestra.run.smithi084.stdout:osd.5: {} 2024-09-20T22:58:14.980 INFO:teuthology.orchestra.run.smithi084.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-20T22:58:14.996 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:58:14.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:58:14.997 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:58:14.998 INFO:teuthology.orchestra.run.smithi084.stdout:osd.6: {} 2024-09-20T22:58:14.998 INFO:teuthology.orchestra.run.smithi084.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-20T22:58:15.015 INFO:teuthology.orchestra.run.smithi084.stdout:osd.7: {} 2024-09-20T22:58:15.015 INFO:teuthology.orchestra.run.smithi084.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-20T22:58:15.098 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:58:15.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:14 smithi084 ceph-mon[40024]: pgmap v1409: 168 pgs: 168 active+clean; 782 MiB data, 2.3 GiB used, 624 GiB / 626 GiB avail 2024-09-20T22:58:15.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:14 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/912883877' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-20T22:58:15.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:14 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-20T22:58:15.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:14 smithi084 ceph-mon[29774]: pgmap v1409: 168 pgs: 168 active+clean; 782 MiB data, 2.3 GiB used, 624 GiB / 626 GiB avail 2024-09-20T22:58:15.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:14 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/912883877' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-20T22:58:15.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:14 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-20T22:58:15.124 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:58:15.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:58:15.125 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:58:15.226 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7 2024-09-20T22:58:15.252 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:58:15.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:58:15.253 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.7' 2024-09-20T22:58:15.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:14 smithi154 ceph-mon[33264]: pgmap v1409: 168 pgs: 168 active+clean; 782 MiB data, 2.3 GiB used, 624 GiB / 626 GiB avail 2024-09-20T22:58:15.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:14 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/912883877' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-20T22:58:15.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:14 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-20T22:58:15.355 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:58:15.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:58:15.380 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:58:15.380 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:58:15.481 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5 2024-09-20T22:58:15.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:58:15.504 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:58:15.505 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.5' 2024-09-20T22:58:15.605 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2 2024-09-20T22:58:15.628 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:58:15.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:58:15.630 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.2' 2024-09-20T22:58:15.730 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4 2024-09-20T22:58:15.754 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:58:15.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:58:15.755 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.4' 2024-09-20T22:58:15.856 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0 2024-09-20T22:58:15.877 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:58:15.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:58:15.878 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.0' 2024-09-20T22:58:15.979 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:58:16.002 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:58:16.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:58:16.003 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:58:16.104 DEBUG:teuthology.orchestra.run.smithi154:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6 2024-09-20T22:58:16.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:15 smithi084 ceph-mon[29774]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 8/1650 objects degraded (0.485%), 1 pg degraded) 2024-09-20T22:58:16.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:15 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-09-20T22:58:16.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:15 smithi084 ceph-mon[29774]: osdmap e1001: 8 total, 8 up, 7 in 2024-09-20T22:58:16.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:15 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/2048427603' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:58:16.116 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:15 smithi084 ceph-mon[40024]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 8/1650 objects degraded (0.485%), 1 pg degraded) 2024-09-20T22:58:16.116 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:15 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-09-20T22:58:16.116 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:15 smithi084 ceph-mon[40024]: osdmap e1001: 8 total, 8 up, 7 in 2024-09-20T22:58:16.116 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:15 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/2048427603' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:58:16.127 INFO:teuthology.orchestra.run.smithi154.stderr:sudo: docker: command not found 2024-09-20T22:58:16.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:58:16.128 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi154 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.6' 2024-09-20T22:58:16.230 DEBUG:teuthology.orchestra.run.smithi084:> sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1 2024-09-20T22:58:16.234 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd unset noscrub 2024-09-20T22:58:16.252 INFO:teuthology.orchestra.run.smithi084.stderr:sudo: docker: command not found 2024-09-20T22:58:16.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T22:58:16.253 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi084 with status 1: 'sudo docker kill -s 1 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd.1' 2024-09-20T22:58:16.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:15 smithi154 ceph-mon[33264]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 8/1650 objects degraded (0.485%), 1 pg degraded) 2024-09-20T22:58:16.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:15 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-09-20T22:58:16.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:15 smithi154 ceph-mon[33264]: osdmap e1001: 8 total, 8 up, 7 in 2024-09-20T22:58:16.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:15 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/2048427603' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-20T22:58:16.271 DEBUG:teuthology.run_tasks:Unwinding manager thrashosds 2024-09-20T22:58:16.280 INFO:tasks.thrashosds:joining thrashosds 2024-09-20T22:58:16.454 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-20T22:58:16.454 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:58:16.659 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:58:17.029 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:16 smithi084 ceph-mon[40024]: pgmap v1411: 142 pgs: 142 active+clean; 583 KiB data, 2.3 GiB used, 624 GiB / 626 GiB avail 2024-09-20T22:58:17.030 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:16 smithi084 ceph-mon[29774]: pgmap v1411: 142 pgs: 142 active+clean; 583 KiB data, 2.3 GiB used, 624 GiB / 626 GiB avail 2024-09-20T22:58:17.063 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-20T22:58:17.068 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:58:17.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:16 smithi154 ceph-mon[33264]: pgmap v1411: 142 pgs: 142 active+clean; 583 KiB data, 2.3 GiB used, 624 GiB / 626 GiB avail 2024-09-20T22:58:17.846 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is unset 2024-09-20T22:58:18.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:17 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/2536115634' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:58:18.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:17 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:58:18.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:17 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/2536115634' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:58:18.116 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:17 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:58:18.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:17 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/2536115634' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:58:18.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:17 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:58:18.540 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:58:18 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: ::ffff:172.21.15.154 - - [20/Sep/2024:22:58:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-20T22:58:18.819 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:58:18.839 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:18 smithi084 ceph-mon[29774]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:58:18.839 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:18 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T22:58:18.839 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:18 smithi084 ceph-mon[29774]: osdmap e1002: 8 total, 8 up, 7 in 2024-09-20T22:58:18.839 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:18 smithi084 ceph-mon[29774]: pgmap v1413: 142 pgs: 142 active+clean; 583 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-20T22:58:19.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:18 smithi084 ceph-mon[40024]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:58:19.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:18 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T22:58:19.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:18 smithi084 ceph-mon[40024]: osdmap e1002: 8 total, 8 up, 7 in 2024-09-20T22:58:19.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:18 smithi084 ceph-mon[40024]: pgmap v1413: 142 pgs: 142 active+clean; 583 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-20T22:58:19.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:18 smithi154 ceph-mon[33264]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-20T22:58:19.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:18 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T22:58:19.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:18 smithi154 ceph-mon[33264]: osdmap e1002: 8 total, 8 up, 7 in 2024-09-20T22:58:19.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:18 smithi154 ceph-mon[33264]: pgmap v1413: 142 pgs: 142 active+clean; 583 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-20T22:58:19.296 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-20T22:58:19.296 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:58:19.851 INFO:teuthology.orchestra.run.smithi084.stderr:set osd.5 primary-affinity to 0.880205 (8576852) 2024-09-20T22:58:19.924 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:19 smithi084 ceph-mon[40024]: 3.13 scrub starts 2024-09-20T22:58:19.925 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:19 smithi084 ceph-mon[40024]: 3.13 scrub ok 2024-09-20T22:58:19.925 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:19 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/2494374229' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.88020494279675}]: dispatch 2024-09-20T22:58:19.925 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:19 smithi084 ceph-mon[40024]: 4.16 scrub starts 2024-09-20T22:58:19.925 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:19 smithi084 ceph-mon[40024]: 4.16 scrub ok 2024-09-20T22:58:19.925 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:19 smithi084 ceph-mon[29774]: 3.13 scrub starts 2024-09-20T22:58:19.925 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:19 smithi084 ceph-mon[29774]: 3.13 scrub ok 2024-09-20T22:58:19.925 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:19 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/2494374229' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.88020494279675}]: dispatch 2024-09-20T22:58:19.925 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:19 smithi084 ceph-mon[29774]: 4.16 scrub starts 2024-09-20T22:58:19.925 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:19 smithi084 ceph-mon[29774]: 4.16 scrub ok 2024-09-20T22:58:19.956 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.1 dump_blocked_ops 2024-09-20T22:58:20.254 INFO:journalctl@ceph.iscsi.iscsi.a.smithi154.stdout:Sep 20 22:58:19 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-iscsi-iscsi-a[59730]: debug there is no tcmu-runner data available 2024-09-20T22:58:20.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:19 smithi154 ceph-mon[33264]: 3.13 scrub starts 2024-09-20T22:58:20.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:19 smithi154 ceph-mon[33264]: 3.13 scrub ok 2024-09-20T22:58:20.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:19 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/2494374229' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.88020494279675}]: dispatch 2024-09-20T22:58:20.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:19 smithi154 ceph-mon[33264]: 4.16 scrub starts 2024-09-20T22:58:20.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:19 smithi154 ceph-mon[33264]: 4.16 scrub ok 2024-09-20T22:58:20.569 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd unset noscrub 2024-09-20T22:58:20.742 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:58:21.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:20 smithi084 ceph-mon[29774]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:58:21.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:20 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/2494374229' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.88020494279675}]': finished 2024-09-20T22:58:21.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:20 smithi084 ceph-mon[29774]: osdmap e1003: 8 total, 8 up, 7 in 2024-09-20T22:58:21.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:20 smithi084 ceph-mon[29774]: pgmap v1415: 142 pgs: 142 active+clean; 583 KiB data, 545 MiB used, 625 GiB / 626 GiB avail 2024-09-20T22:58:21.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:20 smithi084 ceph-mon[29774]: 3.f scrub starts 2024-09-20T22:58:21.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:20 smithi084 ceph-mon[29774]: 3.f scrub ok 2024-09-20T22:58:21.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:20 smithi084 ceph-mon[40024]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:58:21.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:20 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/2494374229' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.88020494279675}]': finished 2024-09-20T22:58:21.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:20 smithi084 ceph-mon[40024]: osdmap e1003: 8 total, 8 up, 7 in 2024-09-20T22:58:21.116 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:20 smithi084 ceph-mon[40024]: pgmap v1415: 142 pgs: 142 active+clean; 583 KiB data, 545 MiB used, 625 GiB / 626 GiB avail 2024-09-20T22:58:21.116 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:20 smithi084 ceph-mon[40024]: 3.f scrub starts 2024-09-20T22:58:21.116 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:20 smithi084 ceph-mon[40024]: 3.f scrub ok 2024-09-20T22:58:21.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:20 smithi154 ceph-mon[33264]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:58:21.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:20 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/2494374229' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 0.88020494279675}]': finished 2024-09-20T22:58:21.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:20 smithi154 ceph-mon[33264]: osdmap e1003: 8 total, 8 up, 7 in 2024-09-20T22:58:21.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:20 smithi154 ceph-mon[33264]: pgmap v1415: 142 pgs: 142 active+clean; 583 KiB data, 545 MiB used, 625 GiB / 626 GiB avail 2024-09-20T22:58:21.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:20 smithi154 ceph-mon[33264]: 3.f scrub starts 2024-09-20T22:58:21.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:20 smithi154 ceph-mon[33264]: 3.f scrub ok 2024-09-20T22:58:21.289 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:58:21.296 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-20T22:58:21.296 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:58:21.923 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.1 dump_historic_ops 2024-09-20T22:58:22.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:21 smithi084 ceph-mon[29774]: osdmap e1004: 8 total, 8 up, 7 in 2024-09-20T22:58:22.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:21 smithi084 ceph-mon[40024]: osdmap e1004: 8 total, 8 up, 7 in 2024-09-20T22:58:22.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:21 smithi154 ceph-mon[33264]: osdmap e1004: 8 total, 8 up, 7 in 2024-09-20T22:58:22.392 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:58:22.867 INFO:teuthology.orchestra.run.smithi084.stderr:noscrub is unset 2024-09-20T22:58:22.885 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-20T22:58:22.885 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:58:23.070 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:22 smithi084 ceph-mon[29774]: 1.8 scrub starts 2024-09-20T22:58:23.070 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:22 smithi084 ceph-mon[29774]: 1.8 scrub ok 2024-09-20T22:58:23.070 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:22 smithi084 ceph-mon[29774]: 5.19 scrub starts 2024-09-20T22:58:23.071 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:22 smithi084 ceph-mon[29774]: 5.19 scrub ok 2024-09-20T22:58:23.071 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:22 smithi084 ceph-mon[29774]: 3.d scrub starts 2024-09-20T22:58:23.071 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:22 smithi084 ceph-mon[29774]: 3.d scrub ok 2024-09-20T22:58:23.071 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:22 smithi084 ceph-mon[29774]: pgmap v1417: 142 pgs: 142 active+clean; 583 KiB data, 545 MiB used, 625 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:58:23.071 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:22 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/1393576593' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:58:23.071 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:22 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:58:23.071 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:22 smithi084 ceph-mon[40024]: 1.8 scrub starts 2024-09-20T22:58:23.071 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:22 smithi084 ceph-mon[40024]: 1.8 scrub ok 2024-09-20T22:58:23.071 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:22 smithi084 ceph-mon[40024]: 5.19 scrub starts 2024-09-20T22:58:23.071 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:22 smithi084 ceph-mon[40024]: 5.19 scrub ok 2024-09-20T22:58:23.071 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:22 smithi084 ceph-mon[40024]: 3.d scrub starts 2024-09-20T22:58:23.071 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:22 smithi084 ceph-mon[40024]: 3.d scrub ok 2024-09-20T22:58:23.072 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:22 smithi084 ceph-mon[40024]: pgmap v1417: 142 pgs: 142 active+clean; 583 KiB data, 545 MiB used, 625 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:58:23.072 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:22 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/1393576593' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:58:23.072 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:22 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:58:23.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:22 smithi154 ceph-mon[33264]: 1.8 scrub starts 2024-09-20T22:58:23.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:22 smithi154 ceph-mon[33264]: 1.8 scrub ok 2024-09-20T22:58:23.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:22 smithi154 ceph-mon[33264]: 5.19 scrub starts 2024-09-20T22:58:23.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:22 smithi154 ceph-mon[33264]: 5.19 scrub ok 2024-09-20T22:58:23.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:22 smithi154 ceph-mon[33264]: 3.d scrub starts 2024-09-20T22:58:23.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:22 smithi154 ceph-mon[33264]: 3.d scrub ok 2024-09-20T22:58:23.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:22 smithi154 ceph-mon[33264]: pgmap v1417: 142 pgs: 142 active+clean; 583 KiB data, 545 MiB used, 625 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:58:23.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:22 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/1393576593' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:58:23.256 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:22 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-20T22:58:23.854 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-20T22:58:24.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:23 smithi084 ceph-mon[40024]: 1.a scrub starts 2024-09-20T22:58:24.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:23 smithi084 ceph-mon[40024]: 1.a scrub ok 2024-09-20T22:58:24.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:23 smithi084 ceph-mon[40024]: 5.16 scrub starts 2024-09-20T22:58:24.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:23 smithi084 ceph-mon[40024]: 5.16 scrub ok 2024-09-20T22:58:24.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:23 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T22:58:24.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:23 smithi084 ceph-mon[40024]: osdmap e1005: 8 total, 8 up, 7 in 2024-09-20T22:58:24.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:23 smithi084 ceph-mon[40024]: osdmap e1006: 8 total, 8 up, 7 in 2024-09-20T22:58:24.116 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:23 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:58:24.116 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:23 smithi084 ceph-mon[29774]: 1.a scrub starts 2024-09-20T22:58:24.116 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:23 smithi084 ceph-mon[29774]: 1.a scrub ok 2024-09-20T22:58:24.116 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:23 smithi084 ceph-mon[29774]: 5.16 scrub starts 2024-09-20T22:58:24.116 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:23 smithi084 ceph-mon[29774]: 5.16 scrub ok 2024-09-20T22:58:24.116 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:23 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T22:58:24.116 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:23 smithi084 ceph-mon[29774]: osdmap e1005: 8 total, 8 up, 7 in 2024-09-20T22:58:24.116 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:23 smithi084 ceph-mon[29774]: osdmap e1006: 8 total, 8 up, 7 in 2024-09-20T22:58:24.116 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:23 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:58:24.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:23 smithi154 ceph-mon[33264]: 1.a scrub starts 2024-09-20T22:58:24.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:23 smithi154 ceph-mon[33264]: 1.a scrub ok 2024-09-20T22:58:24.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:23 smithi154 ceph-mon[33264]: 5.16 scrub starts 2024-09-20T22:58:24.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:23 smithi154 ceph-mon[33264]: 5.16 scrub ok 2024-09-20T22:58:24.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:23 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-20T22:58:24.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:23 smithi154 ceph-mon[33264]: osdmap e1005: 8 total, 8 up, 7 in 2024-09-20T22:58:24.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:23 smithi154 ceph-mon[33264]: osdmap e1006: 8 total, 8 up, 7 in 2024-09-20T22:58:24.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:23 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:58:24.323 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-20T22:58:24.433 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:58:25.019 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:58:25.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:24 smithi084 ceph-mon[40024]: 2.1 scrub starts 2024-09-20T22:58:25.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:24 smithi084 ceph-mon[40024]: 2.1 scrub ok 2024-09-20T22:58:25.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:24 smithi084 ceph-mon[40024]: 6.1 scrub starts 2024-09-20T22:58:25.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:24 smithi084 ceph-mon[40024]: 6.1 scrub ok 2024-09-20T22:58:25.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:24 smithi084 ceph-mon[40024]: pgmap v1420: 142 pgs: 142 active+clean; 583 KiB data, 549 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:58:25.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:24 smithi084 ceph-mon[29774]: 2.1 scrub starts 2024-09-20T22:58:25.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:24 smithi084 ceph-mon[29774]: 2.1 scrub ok 2024-09-20T22:58:25.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:24 smithi084 ceph-mon[29774]: 6.1 scrub starts 2024-09-20T22:58:25.116 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:24 smithi084 ceph-mon[29774]: 6.1 scrub ok 2024-09-20T22:58:25.116 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:24 smithi084 ceph-mon[29774]: pgmap v1420: 142 pgs: 142 active+clean; 583 KiB data, 549 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:58:25.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:24 smithi154 ceph-mon[33264]: 2.1 scrub starts 2024-09-20T22:58:25.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:24 smithi154 ceph-mon[33264]: 2.1 scrub ok 2024-09-20T22:58:25.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:24 smithi154 ceph-mon[33264]: 6.1 scrub starts 2024-09-20T22:58:25.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:24 smithi154 ceph-mon[33264]: 6.1 scrub ok 2024-09-20T22:58:25.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:24 smithi154 ceph-mon[33264]: pgmap v1420: 142 pgs: 142 active+clean; 583 KiB data, 549 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:58:25.468 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-20T22:58:25.468 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:58:25.657 INFO:tasks.thrashosds.thrasher:inning osd 2024-09-20T22:58:25.657 INFO:tasks.thrashosds.thrasher:Adding osd 7 2024-09-20T22:58:25.658 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd in 7 2024-09-20T22:58:25.883 INFO:teuthology.orchestra.run.smithi084.stderr:nodeep-scrub is unset 2024-09-20T22:58:26.138 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:25 smithi084 ceph-mon[40024]: 4.b scrub starts 2024-09-20T22:58:26.138 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:25 smithi084 ceph-mon[40024]: 4.b scrub ok 2024-09-20T22:58:26.139 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:25 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/3543901679' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:58:26.139 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:25 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:58:26.139 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:25 smithi084 ceph-mon[29774]: 4.b scrub starts 2024-09-20T22:58:26.139 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:25 smithi084 ceph-mon[29774]: 4.b scrub ok 2024-09-20T22:58:26.139 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:25 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/3543901679' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:58:26.139 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:25 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:58:26.185 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.2 dump_blocked_ops 2024-09-20T22:58:26.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:25 smithi154 ceph-mon[33264]: 4.b scrub starts 2024-09-20T22:58:26.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:25 smithi154 ceph-mon[33264]: 4.b scrub ok 2024-09-20T22:58:26.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:25 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/3543901679' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:58:26.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:25 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-20T22:58:26.439 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:58:26.902 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:58:27.089 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:26 smithi084 ceph-mon[40024]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T22:58:27.089 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:26 smithi084 ceph-mon[40024]: Cluster is now healthy 2024-09-20T22:58:27.089 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:26 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T22:58:27.089 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:26 smithi084 ceph-mon[40024]: osdmap e1007: 8 total, 8 up, 7 in 2024-09-20T22:58:27.089 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:26 smithi084 ceph-mon[40024]: 3.16 scrub starts 2024-09-20T22:58:27.089 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:26 smithi084 ceph-mon[40024]: 3.16 scrub ok 2024-09-20T22:58:27.089 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:26 smithi084 ceph-mon[40024]: pgmap v1422: 142 pgs: 142 active+clean; 583 KiB data, 549 MiB used, 625 GiB / 626 GiB avail; 1013 B/s rd, 0 op/s 2024-09-20T22:58:27.090 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:26 smithi084 ceph-mon[29774]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T22:58:27.090 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:26 smithi084 ceph-mon[29774]: Cluster is now healthy 2024-09-20T22:58:27.090 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:26 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T22:58:27.090 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:26 smithi084 ceph-mon[29774]: osdmap e1007: 8 total, 8 up, 7 in 2024-09-20T22:58:27.090 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:26 smithi084 ceph-mon[29774]: 3.16 scrub starts 2024-09-20T22:58:27.090 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:26 smithi084 ceph-mon[29774]: 3.16 scrub ok 2024-09-20T22:58:27.090 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:26 smithi084 ceph-mon[29774]: pgmap v1422: 142 pgs: 142 active+clean; 583 KiB data, 549 MiB used, 625 GiB / 626 GiB avail; 1013 B/s rd, 0 op/s 2024-09-20T22:58:27.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:26 smithi154 ceph-mon[33264]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-20T22:58:27.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:26 smithi154 ceph-mon[33264]: Cluster is now healthy 2024-09-20T22:58:27.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:26 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-20T22:58:27.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:26 smithi154 ceph-mon[33264]: osdmap e1007: 8 total, 8 up, 7 in 2024-09-20T22:58:27.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:26 smithi154 ceph-mon[33264]: 3.16 scrub starts 2024-09-20T22:58:27.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:26 smithi154 ceph-mon[33264]: 3.16 scrub ok 2024-09-20T22:58:27.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:26 smithi154 ceph-mon[33264]: pgmap v1422: 142 pgs: 142 active+clean; 583 KiB data, 549 MiB used, 625 GiB / 626 GiB avail; 1013 B/s rd, 0 op/s 2024-09-20T22:58:27.791 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:27 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mon-a[29770]: 2024-09-20T22:58:27.498+0000 7f26ad5e8640 -1 mon.a@0(leader).osd e1007 definitely_dead 0 2024-09-20T22:58:27.876 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-20T22:58:27.876 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:58:27.893 INFO:teuthology.orchestra.run.smithi084.stderr:marked in osd.7. 2024-09-20T22:58:28.092 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:27 smithi084 ceph-mon[40024]: 5.b scrub starts 2024-09-20T22:58:28.092 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:27 smithi084 ceph-mon[40024]: 5.b scrub ok 2024-09-20T22:58:28.092 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:27 smithi084 ceph-mon[40024]: 4.5 scrub starts 2024-09-20T22:58:28.092 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:27 smithi084 ceph-mon[40024]: 4.5 scrub ok 2024-09-20T22:58:28.092 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:27 smithi084 ceph-mon[40024]: 3.6 scrub starts 2024-09-20T22:58:28.093 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:27 smithi084 ceph-mon[40024]: 3.6 scrub ok 2024-09-20T22:58:28.093 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:27 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/4201276575' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["7"]}]: dispatch 2024-09-20T22:58:28.093 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:27 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["7"]}]: dispatch 2024-09-20T22:58:28.093 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:27 smithi084 ceph-mon[29774]: 5.b scrub starts 2024-09-20T22:58:28.093 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:27 smithi084 ceph-mon[29774]: 5.b scrub ok 2024-09-20T22:58:28.093 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:27 smithi084 ceph-mon[29774]: 4.5 scrub starts 2024-09-20T22:58:28.093 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:27 smithi084 ceph-mon[29774]: 4.5 scrub ok 2024-09-20T22:58:28.093 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:27 smithi084 ceph-mon[29774]: 3.6 scrub starts 2024-09-20T22:58:28.093 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:27 smithi084 ceph-mon[29774]: 3.6 scrub ok 2024-09-20T22:58:28.093 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:27 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/4201276575' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["7"]}]: dispatch 2024-09-20T22:58:28.093 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:27 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["7"]}]: dispatch 2024-09-20T22:58:28.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:27 smithi154 ceph-mon[33264]: 5.b scrub starts 2024-09-20T22:58:28.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:27 smithi154 ceph-mon[33264]: 5.b scrub ok 2024-09-20T22:58:28.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:27 smithi154 ceph-mon[33264]: 4.5 scrub starts 2024-09-20T22:58:28.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:27 smithi154 ceph-mon[33264]: 4.5 scrub ok 2024-09-20T22:58:28.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:27 smithi154 ceph-mon[33264]: 3.6 scrub starts 2024-09-20T22:58:28.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:27 smithi154 ceph-mon[33264]: 3.6 scrub ok 2024-09-20T22:58:28.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:27 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/4201276575' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["7"]}]: dispatch 2024-09-20T22:58:28.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:27 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["7"]}]: dispatch 2024-09-20T22:58:28.364 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:58:28 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: ::ffff:172.21.15.154 - - [20/Sep/2024:22:58:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-20T22:58:28.867 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.2 dump_historic_ops 2024-09-20T22:58:29.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:28 smithi084 ceph-mon[29774]: 5.3 deep-scrub starts 2024-09-20T22:58:29.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:28 smithi084 ceph-mon[29774]: 5.3 deep-scrub ok 2024-09-20T22:58:29.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:28 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["7"]}]': finished 2024-09-20T22:58:29.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:28 smithi084 ceph-mon[29774]: osdmap e1008: 8 total, 8 up, 8 in 2024-09-20T22:58:29.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:28 smithi084 ceph-mon[29774]: pgmap v1424: 142 pgs: 142 active+clean; 583 KiB data, 549 MiB used, 625 GiB / 626 GiB avail; 1014 B/s rd, 0 op/s 2024-09-20T22:58:29.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:28 smithi084 ceph-mon[29774]: osdmap e1009: 8 total, 8 up, 8 in 2024-09-20T22:58:29.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:28 smithi084 ceph-mon[40024]: 5.3 deep-scrub starts 2024-09-20T22:58:29.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:28 smithi084 ceph-mon[40024]: 5.3 deep-scrub ok 2024-09-20T22:58:29.116 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:28 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["7"]}]': finished 2024-09-20T22:58:29.116 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:28 smithi084 ceph-mon[40024]: osdmap e1008: 8 total, 8 up, 8 in 2024-09-20T22:58:29.116 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:28 smithi084 ceph-mon[40024]: pgmap v1424: 142 pgs: 142 active+clean; 583 KiB data, 549 MiB used, 625 GiB / 626 GiB avail; 1014 B/s rd, 0 op/s 2024-09-20T22:58:29.116 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:28 smithi084 ceph-mon[40024]: osdmap e1009: 8 total, 8 up, 8 in 2024-09-20T22:58:29.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:28 smithi154 ceph-mon[33264]: 5.3 deep-scrub starts 2024-09-20T22:58:29.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:28 smithi154 ceph-mon[33264]: 5.3 deep-scrub ok 2024-09-20T22:58:29.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:28 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["7"]}]': finished 2024-09-20T22:58:29.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:28 smithi154 ceph-mon[33264]: osdmap e1008: 8 total, 8 up, 8 in 2024-09-20T22:58:29.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:28 smithi154 ceph-mon[33264]: pgmap v1424: 142 pgs: 142 active+clean; 583 KiB data, 549 MiB used, 625 GiB / 626 GiB avail; 1014 B/s rd, 0 op/s 2024-09-20T22:58:29.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:28 smithi154 ceph-mon[33264]: osdmap e1009: 8 total, 8 up, 8 in 2024-09-20T22:58:29.308 INFO:tasks.thrashosds.thrasher:Added osd 7 2024-09-20T22:58:29.308 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=600 2024-09-20T22:58:29.353 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:58:29.864 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-20T22:58:29.864 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:58:30.254 INFO:journalctl@ceph.iscsi.iscsi.a.smithi154.stdout:Sep 20 22:58:29 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-iscsi-iscsi-a[59730]: debug there is no tcmu-runner data available 2024-09-20T22:58:30.527 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:30 smithi084 ceph-mon[40024]: 6.0 scrub starts 2024-09-20T22:58:30.528 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:30 smithi084 ceph-mon[40024]: 6.0 scrub ok 2024-09-20T22:58:30.528 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:30 smithi084 ceph-mon[40024]: 3.18 scrub starts 2024-09-20T22:58:30.528 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:30 smithi084 ceph-mon[40024]: 3.18 scrub ok 2024-09-20T22:58:30.528 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:30 smithi084 ceph-mon[40024]: 3.15 scrub starts 2024-09-20T22:58:30.528 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:30 smithi084 ceph-mon[40024]: 3.15 scrub ok 2024-09-20T22:58:30.528 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:30 smithi084 ceph-mon[40024]: 1.2 starting backfill to osd.7 from (0'0,0'0] MAX to 77'128 2024-09-20T22:58:30.528 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:30 smithi084 ceph-mon[40024]: 1.0 starting backfill to osd.7 from (0'0,0'0] MAX to 77'128 2024-09-20T22:58:30.528 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:30 smithi084 ceph-mon[40024]: osdmap e1010: 8 total, 8 up, 8 in 2024-09-20T22:58:30.528 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:30 smithi084 ceph-mon[40024]: 1.7 starting backfill to osd.7 from (0'0,0'0] MAX to 77'128 2024-09-20T22:58:30.528 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:30 smithi084 ceph-mon[40024]: 1.4 starting backfill to osd.7 from (0'0,0'0] MAX to 855'129 2024-09-20T22:58:30.528 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:30 smithi084 ceph-mon[40024]: purged_snaps scrub starts 2024-09-20T22:58:30.528 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:30 smithi084 ceph-mon[40024]: purged_snaps scrub ok 2024-09-20T22:58:30.529 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:30 smithi084 ceph-mon[40024]: 4.18 scrub starts 2024-09-20T22:58:30.529 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:30 smithi084 ceph-mon[40024]: 4.18 scrub ok 2024-09-20T22:58:30.529 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:30 smithi084 ceph-mon[40024]: purged_snaps scrub starts 2024-09-20T22:58:30.529 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:30 smithi084 ceph-mon[40024]: purged_snaps scrub ok 2024-09-20T22:58:30.529 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:30 smithi084 ceph-mon[29774]: 6.0 scrub starts 2024-09-20T22:58:30.529 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:30 smithi084 ceph-mon[29774]: 6.0 scrub ok 2024-09-20T22:58:30.529 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:30 smithi084 ceph-mon[29774]: 3.18 scrub starts 2024-09-20T22:58:30.529 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:30 smithi084 ceph-mon[29774]: 3.18 scrub ok 2024-09-20T22:58:30.529 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:30 smithi084 ceph-mon[29774]: 3.15 scrub starts 2024-09-20T22:58:30.529 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:30 smithi084 ceph-mon[29774]: 3.15 scrub ok 2024-09-20T22:58:30.529 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:30 smithi084 ceph-mon[29774]: 1.2 starting backfill to osd.7 from (0'0,0'0] MAX to 77'128 2024-09-20T22:58:30.530 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:30 smithi084 ceph-mon[29774]: 1.0 starting backfill to osd.7 from (0'0,0'0] MAX to 77'128 2024-09-20T22:58:30.530 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:30 smithi084 ceph-mon[29774]: osdmap e1010: 8 total, 8 up, 8 in 2024-09-20T22:58:30.530 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:30 smithi084 ceph-mon[29774]: 1.7 starting backfill to osd.7 from (0'0,0'0] MAX to 77'128 2024-09-20T22:58:30.530 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:30 smithi084 ceph-mon[29774]: 1.4 starting backfill to osd.7 from (0'0,0'0] MAX to 855'129 2024-09-20T22:58:30.530 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:30 smithi084 ceph-mon[29774]: purged_snaps scrub starts 2024-09-20T22:58:30.530 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:30 smithi084 ceph-mon[29774]: purged_snaps scrub ok 2024-09-20T22:58:30.530 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:30 smithi084 ceph-mon[29774]: 4.18 scrub starts 2024-09-20T22:58:30.530 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:30 smithi084 ceph-mon[29774]: 4.18 scrub ok 2024-09-20T22:58:30.530 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:30 smithi084 ceph-mon[29774]: purged_snaps scrub starts 2024-09-20T22:58:30.530 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:30 smithi084 ceph-mon[29774]: purged_snaps scrub ok 2024-09-20T22:58:30.553 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-20T22:58:30.616 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:58:30.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:30 smithi154 ceph-mon[33264]: 6.0 scrub starts 2024-09-20T22:58:30.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:30 smithi154 ceph-mon[33264]: 6.0 scrub ok 2024-09-20T22:58:30.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:30 smithi154 ceph-mon[33264]: 3.18 scrub starts 2024-09-20T22:58:30.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:30 smithi154 ceph-mon[33264]: 3.18 scrub ok 2024-09-20T22:58:30.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:30 smithi154 ceph-mon[33264]: 3.15 scrub starts 2024-09-20T22:58:30.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:30 smithi154 ceph-mon[33264]: 3.15 scrub ok 2024-09-20T22:58:30.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:30 smithi154 ceph-mon[33264]: 1.2 starting backfill to osd.7 from (0'0,0'0] MAX to 77'128 2024-09-20T22:58:30.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:30 smithi154 ceph-mon[33264]: 1.0 starting backfill to osd.7 from (0'0,0'0] MAX to 77'128 2024-09-20T22:58:30.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:30 smithi154 ceph-mon[33264]: osdmap e1010: 8 total, 8 up, 8 in 2024-09-20T22:58:30.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:30 smithi154 ceph-mon[33264]: 1.7 starting backfill to osd.7 from (0'0,0'0] MAX to 77'128 2024-09-20T22:58:30.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:30 smithi154 ceph-mon[33264]: 1.4 starting backfill to osd.7 from (0'0,0'0] MAX to 855'129 2024-09-20T22:58:30.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:30 smithi154 ceph-mon[33264]: purged_snaps scrub starts 2024-09-20T22:58:30.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:30 smithi154 ceph-mon[33264]: purged_snaps scrub ok 2024-09-20T22:58:30.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:30 smithi154 ceph-mon[33264]: 4.18 scrub starts 2024-09-20T22:58:30.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:30 smithi154 ceph-mon[33264]: 4.18 scrub ok 2024-09-20T22:58:30.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:30 smithi154 ceph-mon[33264]: purged_snaps scrub starts 2024-09-20T22:58:30.756 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:30 smithi154 ceph-mon[33264]: purged_snaps scrub ok 2024-09-20T22:58:30.809 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:58:31.079 INFO:teuthology.orchestra.run.smithi084.stdout:mon.a: {} 2024-09-20T22:58:31.080 INFO:teuthology.orchestra.run.smithi084.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-20T22:58:31.088 INFO:teuthology.orchestra.run.smithi084.stdout:mon.b: {} 2024-09-20T22:58:31.089 INFO:teuthology.orchestra.run.smithi084.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-20T22:58:31.097 INFO:teuthology.orchestra.run.smithi084.stdout:mon.c: {} 2024-09-20T22:58:31.097 INFO:teuthology.orchestra.run.smithi084.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-20T22:58:31.295 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-20T22:58:31.296 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:58:31.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:31 smithi154 ceph-mon[33264]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:58:31.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:31 smithi154 ceph-mon[33264]: 4.0 scrub starts 2024-09-20T22:58:31.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:31 smithi154 ceph-mon[33264]: 4.0 scrub ok 2024-09-20T22:58:31.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:31 smithi154 ceph-mon[33264]: pgmap v1427: 142 pgs: 3 remapped+peering, 139 active+clean; 583 KiB data, 662 MiB used, 715 GiB / 715 GiB avail; 7 B/s, 11 objects/s recovering 2024-09-20T22:58:31.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:31 smithi154 ceph-mon[33264]: 3.9 scrub starts 2024-09-20T22:58:31.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:31 smithi154 ceph-mon[33264]: 3.9 scrub ok 2024-09-20T22:58:31.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:31 smithi154 ceph-mon[33264]: osdmap e1011: 8 total, 8 up, 8 in 2024-09-20T22:58:31.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:31 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/4105530337' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-20T22:58:31.718 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:31 smithi084 ceph-mon[40024]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:58:31.718 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:31 smithi084 ceph-mon[40024]: 4.0 scrub starts 2024-09-20T22:58:31.719 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:31 smithi084 ceph-mon[40024]: 4.0 scrub ok 2024-09-20T22:58:31.719 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:31 smithi084 ceph-mon[40024]: pgmap v1427: 142 pgs: 3 remapped+peering, 139 active+clean; 583 KiB data, 662 MiB used, 715 GiB / 715 GiB avail; 7 B/s, 11 objects/s recovering 2024-09-20T22:58:31.719 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:31 smithi084 ceph-mon[40024]: 3.9 scrub starts 2024-09-20T22:58:31.719 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:31 smithi084 ceph-mon[40024]: 3.9 scrub ok 2024-09-20T22:58:31.719 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:31 smithi084 ceph-mon[40024]: osdmap e1011: 8 total, 8 up, 8 in 2024-09-20T22:58:31.719 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:31 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/4105530337' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-20T22:58:31.719 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:31 smithi084 ceph-mon[29774]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:58:31.719 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:31 smithi084 ceph-mon[29774]: 4.0 scrub starts 2024-09-20T22:58:31.719 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:31 smithi084 ceph-mon[29774]: 4.0 scrub ok 2024-09-20T22:58:31.719 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:31 smithi084 ceph-mon[29774]: pgmap v1427: 142 pgs: 3 remapped+peering, 139 active+clean; 583 KiB data, 662 MiB used, 715 GiB / 715 GiB avail; 7 B/s, 11 objects/s recovering 2024-09-20T22:58:31.719 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:31 smithi084 ceph-mon[29774]: 3.9 scrub starts 2024-09-20T22:58:31.720 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:31 smithi084 ceph-mon[29774]: 3.9 scrub ok 2024-09-20T22:58:31.720 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:31 smithi084 ceph-mon[29774]: osdmap e1011: 8 total, 8 up, 8 in 2024-09-20T22:58:31.720 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:31 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/4105530337' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-20T22:58:31.744 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd reweight 0 1 2024-09-20T22:58:31.990 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:58:32.131 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.4 dump_blocked_ops 2024-09-20T22:58:32.381 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:58:32.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:32 smithi154 ceph-mon[33264]: 3.0 scrub starts 2024-09-20T22:58:32.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:32 smithi154 ceph-mon[33264]: 3.0 scrub ok 2024-09-20T22:58:32.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:32 smithi154 ceph-mon[33264]: osdmap e1012: 8 total, 8 up, 8 in 2024-09-20T22:58:32.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:32 smithi084 ceph-mon[29774]: 3.0 scrub starts 2024-09-20T22:58:32.866 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:32 smithi084 ceph-mon[29774]: 3.0 scrub ok 2024-09-20T22:58:32.866 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:32 smithi084 ceph-mon[29774]: osdmap e1012: 8 total, 8 up, 8 in 2024-09-20T22:58:32.866 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:32 smithi084 ceph-mon[40024]: 3.0 scrub starts 2024-09-20T22:58:32.866 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:32 smithi084 ceph-mon[40024]: 3.0 scrub ok 2024-09-20T22:58:32.866 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:32 smithi084 ceph-mon[40024]: osdmap e1012: 8 total, 8 up, 8 in 2024-09-20T22:58:32.868 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-20T22:58:32.869 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:58:33.419 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.0 to 1 (10000) 2024-09-20T22:58:33.522 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.4 dump_historic_ops 2024-09-20T22:58:33.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:33 smithi154 ceph-mon[33264]: 6.18 scrub starts 2024-09-20T22:58:33.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:33 smithi154 ceph-mon[33264]: 6.18 scrub ok 2024-09-20T22:58:33.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:33 smithi154 ceph-mon[33264]: 6.b scrub starts 2024-09-20T22:58:33.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:33 smithi154 ceph-mon[33264]: 6.b scrub ok 2024-09-20T22:58:33.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:33 smithi154 ceph-mon[33264]: 6.7 scrub starts 2024-09-20T22:58:33.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:33 smithi154 ceph-mon[33264]: 6.7 scrub ok 2024-09-20T22:58:33.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:33 smithi154 ceph-mon[33264]: pgmap v1430: 142 pgs: 3 remapped+peering, 139 active+clean; 583 KiB data, 662 MiB used, 715 GiB / 715 GiB avail; 7 B/s, 11 objects/s recovering 2024-09-20T22:58:33.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:33 smithi154 ceph-mon[33264]: 6.8 scrub starts 2024-09-20T22:58:33.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:33 smithi154 ceph-mon[33264]: 6.8 scrub ok 2024-09-20T22:58:33.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:33 smithi154 ceph-mon[33264]: 3.19 scrub starts 2024-09-20T22:58:33.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:33 smithi154 ceph-mon[33264]: 3.19 scrub ok 2024-09-20T22:58:33.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:33 smithi154 ceph-mon[33264]: 5.1f scrub starts 2024-09-20T22:58:33.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:33 smithi154 ceph-mon[33264]: 5.1f scrub ok 2024-09-20T22:58:33.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:33 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/2783377994' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-09-20T22:58:33.770 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:58:33.776 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:33 smithi084 ceph-mon[40024]: 6.18 scrub starts 2024-09-20T22:58:33.776 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:33 smithi084 ceph-mon[40024]: 6.18 scrub ok 2024-09-20T22:58:33.776 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:33 smithi084 ceph-mon[40024]: 6.b scrub starts 2024-09-20T22:58:33.776 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:33 smithi084 ceph-mon[40024]: 6.b scrub ok 2024-09-20T22:58:33.776 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:33 smithi084 ceph-mon[40024]: 6.7 scrub starts 2024-09-20T22:58:33.776 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:33 smithi084 ceph-mon[40024]: 6.7 scrub ok 2024-09-20T22:58:33.776 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:33 smithi084 ceph-mon[40024]: pgmap v1430: 142 pgs: 3 remapped+peering, 139 active+clean; 583 KiB data, 662 MiB used, 715 GiB / 715 GiB avail; 7 B/s, 11 objects/s recovering 2024-09-20T22:58:33.776 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:33 smithi084 ceph-mon[40024]: 6.8 scrub starts 2024-09-20T22:58:33.776 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:33 smithi084 ceph-mon[40024]: 6.8 scrub ok 2024-09-20T22:58:33.776 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:33 smithi084 ceph-mon[40024]: 3.19 scrub starts 2024-09-20T22:58:33.776 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:33 smithi084 ceph-mon[40024]: 3.19 scrub ok 2024-09-20T22:58:33.776 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:33 smithi084 ceph-mon[40024]: 5.1f scrub starts 2024-09-20T22:58:33.776 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:33 smithi084 ceph-mon[40024]: 5.1f scrub ok 2024-09-20T22:58:33.777 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:33 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/2783377994' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-09-20T22:58:33.777 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:33 smithi084 ceph-mon[29774]: 6.18 scrub starts 2024-09-20T22:58:33.777 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:33 smithi084 ceph-mon[29774]: 6.18 scrub ok 2024-09-20T22:58:33.777 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:33 smithi084 ceph-mon[29774]: 6.b scrub starts 2024-09-20T22:58:33.777 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:33 smithi084 ceph-mon[29774]: 6.b scrub ok 2024-09-20T22:58:33.777 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:33 smithi084 ceph-mon[29774]: 6.7 scrub starts 2024-09-20T22:58:33.777 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:33 smithi084 ceph-mon[29774]: 6.7 scrub ok 2024-09-20T22:58:33.777 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:33 smithi084 ceph-mon[29774]: pgmap v1430: 142 pgs: 3 remapped+peering, 139 active+clean; 583 KiB data, 662 MiB used, 715 GiB / 715 GiB avail; 7 B/s, 11 objects/s recovering 2024-09-20T22:58:33.777 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:33 smithi084 ceph-mon[29774]: 6.8 scrub starts 2024-09-20T22:58:33.777 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:33 smithi084 ceph-mon[29774]: 6.8 scrub ok 2024-09-20T22:58:33.777 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:33 smithi084 ceph-mon[29774]: 3.19 scrub starts 2024-09-20T22:58:33.778 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:33 smithi084 ceph-mon[29774]: 3.19 scrub ok 2024-09-20T22:58:33.778 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:33 smithi084 ceph-mon[29774]: 5.1f scrub starts 2024-09-20T22:58:33.778 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:33 smithi084 ceph-mon[29774]: 5.1f scrub ok 2024-09-20T22:58:33.778 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:33 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/2783377994' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-09-20T22:58:34.185 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd primary-affinity 0 1 2024-09-20T22:58:34.220 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-20T22:58:34.220 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:58:34.437 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:58:34.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:34 smithi154 ceph-mon[33264]: 4.4 deep-scrub starts 2024-09-20T22:58:34.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:34 smithi154 ceph-mon[33264]: 4.4 deep-scrub ok 2024-09-20T22:58:34.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:34 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/2783377994' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-20T22:58:34.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:34 smithi154 ceph-mon[33264]: osdmap e1013: 8 total, 8 up, 8 in 2024-09-20T22:58:34.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:34 smithi154 ceph-mon[33264]: 5.11 scrub starts 2024-09-20T22:58:34.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:34 smithi154 ceph-mon[33264]: 5.11 scrub ok 2024-09-20T22:58:34.701 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:34 smithi084 ceph-mon[40024]: 4.4 deep-scrub starts 2024-09-20T22:58:34.701 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:34 smithi084 ceph-mon[40024]: 4.4 deep-scrub ok 2024-09-20T22:58:34.701 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:34 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/2783377994' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-20T22:58:34.701 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:34 smithi084 ceph-mon[40024]: osdmap e1013: 8 total, 8 up, 8 in 2024-09-20T22:58:34.701 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:34 smithi084 ceph-mon[40024]: 5.11 scrub starts 2024-09-20T22:58:34.701 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:34 smithi084 ceph-mon[40024]: 5.11 scrub ok 2024-09-20T22:58:34.701 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:34 smithi084 ceph-mon[29774]: 4.4 deep-scrub starts 2024-09-20T22:58:34.702 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:34 smithi084 ceph-mon[29774]: 4.4 deep-scrub ok 2024-09-20T22:58:34.702 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:34 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/2783377994' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-20T22:58:34.702 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:34 smithi084 ceph-mon[29774]: osdmap e1013: 8 total, 8 up, 8 in 2024-09-20T22:58:34.702 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:34 smithi084 ceph-mon[29774]: 5.11 scrub starts 2024-09-20T22:58:34.702 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:34 smithi084 ceph-mon[29774]: 5.11 scrub ok 2024-09-20T22:58:34.845 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-20T22:58:35.091 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:58:35.435 INFO:teuthology.orchestra.run.smithi084.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-09-20T22:58:35.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:35 smithi154 ceph-mon[33264]: pgmap v1432: 142 pgs: 2 active+remapped+backfill_toofull, 140 active+clean; 583 KiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 1/597 objects misplaced (0.168%); 48 B/s, 3 objects/s recovering 2024-09-20T22:58:35.504 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:35 smithi154 ceph-mon[33264]: 3.11 scrub starts 2024-09-20T22:58:35.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:35 smithi154 ceph-mon[33264]: 3.11 scrub ok 2024-09-20T22:58:35.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:35 smithi154 ceph-mon[33264]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-20T22:58:35.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:35 smithi154 ceph-mon[33264]: 5.5 scrub starts 2024-09-20T22:58:35.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:35 smithi154 ceph-mon[33264]: 5.5 scrub ok 2024-09-20T22:58:35.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:35 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/568249161' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-09-20T22:58:35.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:35 smithi154 ceph-mon[33264]: 1.3 scrub starts 2024-09-20T22:58:35.505 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:35 smithi154 ceph-mon[33264]: 1.3 scrub ok 2024-09-20T22:58:35.529 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-20T22:58:35.529 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:58:35.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:35 smithi084 ceph-mon[40024]: pgmap v1432: 142 pgs: 2 active+remapped+backfill_toofull, 140 active+clean; 583 KiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 1/597 objects misplaced (0.168%); 48 B/s, 3 objects/s recovering 2024-09-20T22:58:35.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:35 smithi084 ceph-mon[40024]: 3.11 scrub starts 2024-09-20T22:58:35.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:35 smithi084 ceph-mon[40024]: 3.11 scrub ok 2024-09-20T22:58:35.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:35 smithi084 ceph-mon[40024]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-20T22:58:35.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:35 smithi084 ceph-mon[40024]: 5.5 scrub starts 2024-09-20T22:58:35.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:35 smithi084 ceph-mon[40024]: 5.5 scrub ok 2024-09-20T22:58:35.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:35 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/568249161' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-09-20T22:58:35.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:35 smithi084 ceph-mon[40024]: 1.3 scrub starts 2024-09-20T22:58:35.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:35 smithi084 ceph-mon[40024]: 1.3 scrub ok 2024-09-20T22:58:35.866 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:35 smithi084 ceph-mon[29774]: pgmap v1432: 142 pgs: 2 active+remapped+backfill_toofull, 140 active+clean; 583 KiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 1/597 objects misplaced (0.168%); 48 B/s, 3 objects/s recovering 2024-09-20T22:58:35.866 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:35 smithi084 ceph-mon[29774]: 3.11 scrub starts 2024-09-20T22:58:35.866 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:35 smithi084 ceph-mon[29774]: 3.11 scrub ok 2024-09-20T22:58:35.866 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:35 smithi084 ceph-mon[29774]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-20T22:58:35.866 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:35 smithi084 ceph-mon[29774]: 5.5 scrub starts 2024-09-20T22:58:35.866 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:35 smithi084 ceph-mon[29774]: 5.5 scrub ok 2024-09-20T22:58:35.866 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:35 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/568249161' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-09-20T22:58:35.866 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:35 smithi084 ceph-mon[29774]: 1.3 scrub starts 2024-09-20T22:58:35.866 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:35 smithi084 ceph-mon[29774]: 1.3 scrub ok 2024-09-20T22:58:36.127 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.5 dump_blocked_ops 2024-09-20T22:58:36.275 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd reweight 1 1 2024-09-20T22:58:36.374 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:58:36.526 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:58:36.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:36 smithi084 ceph-mon[40024]: 5.14 scrub starts 2024-09-20T22:58:36.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:36 smithi084 ceph-mon[40024]: 5.14 scrub ok 2024-09-20T22:58:36.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:36 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/568249161' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-09-20T22:58:36.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:36 smithi084 ceph-mon[40024]: osdmap e1014: 8 total, 8 up, 8 in 2024-09-20T22:58:36.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:36 smithi084 ceph-mon[40024]: 3.12 scrub starts 2024-09-20T22:58:36.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:36 smithi084 ceph-mon[40024]: 3.12 scrub ok 2024-09-20T22:58:36.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:36 smithi084 ceph-mon[29774]: 5.14 scrub starts 2024-09-20T22:58:36.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:36 smithi084 ceph-mon[29774]: 5.14 scrub ok 2024-09-20T22:58:36.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:36 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/568249161' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-09-20T22:58:36.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:36 smithi084 ceph-mon[29774]: osdmap e1014: 8 total, 8 up, 8 in 2024-09-20T22:58:36.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:36 smithi084 ceph-mon[29774]: 3.12 scrub starts 2024-09-20T22:58:36.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:36 smithi084 ceph-mon[29774]: 3.12 scrub ok 2024-09-20T22:58:36.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:36 smithi154 ceph-mon[33264]: 5.14 scrub starts 2024-09-20T22:58:36.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:36 smithi154 ceph-mon[33264]: 5.14 scrub ok 2024-09-20T22:58:36.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:36 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/568249161' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-09-20T22:58:36.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:36 smithi154 ceph-mon[33264]: osdmap e1014: 8 total, 8 up, 8 in 2024-09-20T22:58:36.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:36 smithi154 ceph-mon[33264]: 3.12 scrub starts 2024-09-20T22:58:36.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:36 smithi154 ceph-mon[33264]: 3.12 scrub ok 2024-09-20T22:58:36.877 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-20T22:58:36.878 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:58:37.453 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.1 to 1 (10000) 2024-09-20T22:58:37.491 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:37 smithi154 ceph-mon[33264]: 6.1f scrub starts 2024-09-20T22:58:37.491 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:37 smithi154 ceph-mon[33264]: 6.1f scrub ok 2024-09-20T22:58:37.491 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:37 smithi154 ceph-mon[33264]: pgmap v1434: 142 pgs: 2 active+remapped+backfill_toofull, 140 active+clean; 583 KiB data, 646 MiB used, 715 GiB / 715 GiB avail; 929 B/s rd, 0 op/s; 1/597 objects misplaced (0.168%); 39 B/s, 3 objects/s recovering 2024-09-20T22:58:37.491 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:37 smithi154 ceph-mon[33264]: 3.5 scrub starts 2024-09-20T22:58:37.491 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:37 smithi154 ceph-mon[33264]: 3.5 scrub ok 2024-09-20T22:58:37.492 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:37 smithi154 ceph-mon[33264]: 6.f scrub starts 2024-09-20T22:58:37.492 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:37 smithi154 ceph-mon[33264]: 6.f scrub ok 2024-09-20T22:58:37.492 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:37 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/3830004810' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-09-20T22:58:37.492 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:37 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-09-20T22:58:37.519 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.5 dump_historic_ops 2024-09-20T22:58:37.765 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:58:37.801 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:37 smithi084 ceph-mon[29774]: 6.1f scrub starts 2024-09-20T22:58:37.801 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:37 smithi084 ceph-mon[29774]: 6.1f scrub ok 2024-09-20T22:58:37.801 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:37 smithi084 ceph-mon[29774]: pgmap v1434: 142 pgs: 2 active+remapped+backfill_toofull, 140 active+clean; 583 KiB data, 646 MiB used, 715 GiB / 715 GiB avail; 929 B/s rd, 0 op/s; 1/597 objects misplaced (0.168%); 39 B/s, 3 objects/s recovering 2024-09-20T22:58:37.801 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:37 smithi084 ceph-mon[29774]: 3.5 scrub starts 2024-09-20T22:58:37.801 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:37 smithi084 ceph-mon[29774]: 3.5 scrub ok 2024-09-20T22:58:37.801 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:37 smithi084 ceph-mon[29774]: 6.f scrub starts 2024-09-20T22:58:37.801 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:37 smithi084 ceph-mon[29774]: 6.f scrub ok 2024-09-20T22:58:37.801 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:37 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/3830004810' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-09-20T22:58:37.801 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:37 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-09-20T22:58:37.802 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:37 smithi084 ceph-mon[40024]: 6.1f scrub starts 2024-09-20T22:58:37.802 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:37 smithi084 ceph-mon[40024]: 6.1f scrub ok 2024-09-20T22:58:37.802 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:37 smithi084 ceph-mon[40024]: pgmap v1434: 142 pgs: 2 active+remapped+backfill_toofull, 140 active+clean; 583 KiB data, 646 MiB used, 715 GiB / 715 GiB avail; 929 B/s rd, 0 op/s; 1/597 objects misplaced (0.168%); 39 B/s, 3 objects/s recovering 2024-09-20T22:58:37.802 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:37 smithi084 ceph-mon[40024]: 3.5 scrub starts 2024-09-20T22:58:37.802 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:37 smithi084 ceph-mon[40024]: 3.5 scrub ok 2024-09-20T22:58:37.802 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:37 smithi084 ceph-mon[40024]: 6.f scrub starts 2024-09-20T22:58:37.802 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:37 smithi084 ceph-mon[40024]: 6.f scrub ok 2024-09-20T22:58:37.802 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:37 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/3830004810' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-09-20T22:58:37.802 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:37 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-09-20T22:58:38.218 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-20T22:58:38.218 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:58:38.248 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd primary-affinity 1 1 2024-09-20T22:58:38.496 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:58:38.615 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:58:38 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: ::ffff:172.21.15.154 - - [20/Sep/2024:22:58:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-20T22:58:38.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:38 smithi084 ceph-mon[40024]: 6.14 scrub starts 2024-09-20T22:58:38.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:38 smithi084 ceph-mon[40024]: 6.14 scrub ok 2024-09-20T22:58:38.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:38 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-20T22:58:38.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:38 smithi084 ceph-mon[40024]: osdmap e1015: 8 total, 8 up, 8 in 2024-09-20T22:58:38.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:38 smithi084 ceph-mon[40024]: osdmap e1016: 8 total, 8 up, 8 in 2024-09-20T22:58:38.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:38 smithi084 ceph-mon[29774]: 6.14 scrub starts 2024-09-20T22:58:38.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:38 smithi084 ceph-mon[29774]: 6.14 scrub ok 2024-09-20T22:58:38.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:38 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-20T22:58:38.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:38 smithi084 ceph-mon[29774]: osdmap e1015: 8 total, 8 up, 8 in 2024-09-20T22:58:38.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:38 smithi084 ceph-mon[29774]: osdmap e1016: 8 total, 8 up, 8 in 2024-09-20T22:58:38.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:38 smithi154 ceph-mon[33264]: 6.14 scrub starts 2024-09-20T22:58:38.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:38 smithi154 ceph-mon[33264]: 6.14 scrub ok 2024-09-20T22:58:38.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:38 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-20T22:58:38.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:38 smithi154 ceph-mon[33264]: osdmap e1015: 8 total, 8 up, 8 in 2024-09-20T22:58:38.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:38 smithi154 ceph-mon[33264]: osdmap e1016: 8 total, 8 up, 8 in 2024-09-20T22:58:38.997 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-20T22:58:39.244 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:58:39.544 INFO:teuthology.orchestra.run.smithi084.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-09-20T22:58:39.644 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:39 smithi154 ceph-mon[33264]: pgmap v1436: 142 pgs: 2 active+remapped+backfill_toofull, 140 active+clean; 583 KiB data, 646 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 1/597 objects misplaced (0.168%); 36 B/s, 2 objects/s recovering 2024-09-20T22:58:39.644 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:39 smithi154 ceph-mon[33264]: 6.19 scrub starts 2024-09-20T22:58:39.644 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:39 smithi154 ceph-mon[33264]: 6.19 scrub ok 2024-09-20T22:58:39.644 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:39 smithi154 ceph-mon[33264]: 1.9 scrub starts 2024-09-20T22:58:39.644 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:39 smithi154 ceph-mon[33264]: 1.9 scrub ok 2024-09-20T22:58:39.645 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:39 smithi154 ceph-mon[33264]: 3.a scrub starts 2024-09-20T22:58:39.645 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:39 smithi154 ceph-mon[33264]: 3.a scrub ok 2024-09-20T22:58:39.645 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:39 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:58:39.645 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:39 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:58:39.645 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:39 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/1121767573' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-09-20T22:58:39.743 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-20T22:58:39.744 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:58:39.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:39 smithi084 ceph-mon[40024]: pgmap v1436: 142 pgs: 2 active+remapped+backfill_toofull, 140 active+clean; 583 KiB data, 646 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 1/597 objects misplaced (0.168%); 36 B/s, 2 objects/s recovering 2024-09-20T22:58:39.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:39 smithi084 ceph-mon[40024]: 6.19 scrub starts 2024-09-20T22:58:39.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:39 smithi084 ceph-mon[40024]: 6.19 scrub ok 2024-09-20T22:58:39.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:39 smithi084 ceph-mon[40024]: 1.9 scrub starts 2024-09-20T22:58:39.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:39 smithi084 ceph-mon[40024]: 1.9 scrub ok 2024-09-20T22:58:39.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:39 smithi084 ceph-mon[40024]: 3.a scrub starts 2024-09-20T22:58:39.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:39 smithi084 ceph-mon[40024]: 3.a scrub ok 2024-09-20T22:58:39.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:39 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:58:39.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:39 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:58:39.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:39 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/1121767573' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-09-20T22:58:39.866 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:39 smithi084 ceph-mon[29774]: pgmap v1436: 142 pgs: 2 active+remapped+backfill_toofull, 140 active+clean; 583 KiB data, 646 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 1/597 objects misplaced (0.168%); 36 B/s, 2 objects/s recovering 2024-09-20T22:58:39.866 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:39 smithi084 ceph-mon[29774]: 6.19 scrub starts 2024-09-20T22:58:39.866 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:39 smithi084 ceph-mon[29774]: 6.19 scrub ok 2024-09-20T22:58:39.866 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:39 smithi084 ceph-mon[29774]: 1.9 scrub starts 2024-09-20T22:58:39.866 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:39 smithi084 ceph-mon[29774]: 1.9 scrub ok 2024-09-20T22:58:39.866 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:39 smithi084 ceph-mon[29774]: 3.a scrub starts 2024-09-20T22:58:39.866 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:39 smithi084 ceph-mon[29774]: 3.a scrub ok 2024-09-20T22:58:39.866 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:39 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:58:39.866 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:39 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:58:39.866 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:39 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/1121767573' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-09-20T22:58:40.004 INFO:journalctl@ceph.iscsi.iscsi.a.smithi154.stdout:Sep 20 22:58:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-iscsi-iscsi-a[59730]: debug there is no tcmu-runner data available 2024-09-20T22:58:40.249 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd reweight 2 1 2024-09-20T22:58:40.353 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.6 dump_blocked_ops 2024-09-20T22:58:40.496 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:58:40.600 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:58:40.717 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:40 smithi154 ceph-mon[33264]: 4.1a scrub starts 2024-09-20T22:58:40.717 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:40 smithi154 ceph-mon[33264]: 4.1a scrub ok 2024-09-20T22:58:40.717 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:40 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/1121767573' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-09-20T22:58:40.717 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:40 smithi154 ceph-mon[33264]: osdmap e1017: 8 total, 8 up, 8 in 2024-09-20T22:58:40.855 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:40 smithi084 ceph-mon[29774]: 4.1a scrub starts 2024-09-20T22:58:40.856 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:40 smithi084 ceph-mon[29774]: 4.1a scrub ok 2024-09-20T22:58:40.856 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:40 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/1121767573' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-09-20T22:58:40.856 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:40 smithi084 ceph-mon[29774]: osdmap e1017: 8 total, 8 up, 8 in 2024-09-20T22:58:40.856 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:40 smithi084 ceph-mon[40024]: 4.1a scrub starts 2024-09-20T22:58:40.856 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:40 smithi084 ceph-mon[40024]: 4.1a scrub ok 2024-09-20T22:58:40.856 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:40 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/1121767573' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-09-20T22:58:40.856 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:40 smithi084 ceph-mon[40024]: osdmap e1017: 8 total, 8 up, 8 in 2024-09-20T22:58:41.052 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-20T22:58:41.052 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:58:41.555 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.2 to 1 (10000) 2024-09-20T22:58:41.608 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.6 dump_historic_ops 2024-09-20T22:58:41.852 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:58:41.861 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:41 smithi084 ceph-mon[40024]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:58:41.861 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:41 smithi084 ceph-mon[40024]: pgmap v1439: 142 pgs: 2 active+remapped+backfill_toofull, 140 active+clean; 583 KiB data, 647 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 1/597 objects misplaced (0.168%) 2024-09-20T22:58:41.861 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:41 smithi084 ceph-mon[40024]: 5.8 scrub starts 2024-09-20T22:58:41.861 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:41 smithi084 ceph-mon[40024]: 5.8 scrub ok 2024-09-20T22:58:41.861 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:41 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/3804474087' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-09-20T22:58:41.861 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:41 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-09-20T22:58:41.862 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:41 smithi084 ceph-mon[29774]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:58:41.862 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:41 smithi084 ceph-mon[29774]: pgmap v1439: 142 pgs: 2 active+remapped+backfill_toofull, 140 active+clean; 583 KiB data, 647 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 1/597 objects misplaced (0.168%) 2024-09-20T22:58:41.862 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:41 smithi084 ceph-mon[29774]: 5.8 scrub starts 2024-09-20T22:58:41.862 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:41 smithi084 ceph-mon[29774]: 5.8 scrub ok 2024-09-20T22:58:41.862 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:41 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/3804474087' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-09-20T22:58:41.862 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:41 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-09-20T22:58:41.973 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:41 smithi154 ceph-mon[33264]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:58:41.974 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:41 smithi154 ceph-mon[33264]: pgmap v1439: 142 pgs: 2 active+remapped+backfill_toofull, 140 active+clean; 583 KiB data, 647 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 1/597 objects misplaced (0.168%) 2024-09-20T22:58:41.974 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:41 smithi154 ceph-mon[33264]: 5.8 scrub starts 2024-09-20T22:58:41.974 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:41 smithi154 ceph-mon[33264]: 5.8 scrub ok 2024-09-20T22:58:41.974 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:41 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/3804474087' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-09-20T22:58:41.974 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:41 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-09-20T22:58:42.343 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-20T22:58:42.343 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:58:42.354 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd primary-affinity 2 1 2024-09-20T22:58:42.605 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:58:42.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:42 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-20T22:58:42.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:42 smithi084 ceph-mon[40024]: osdmap e1018: 8 total, 8 up, 8 in 2024-09-20T22:58:42.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:42 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-20T22:58:42.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:42 smithi084 ceph-mon[29774]: osdmap e1018: 8 total, 8 up, 8 in 2024-09-20T22:58:42.905 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:42 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-20T22:58:42.905 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:42 smithi154 ceph-mon[33264]: osdmap e1018: 8 total, 8 up, 8 in 2024-09-20T22:58:42.932 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-20T22:58:43.177 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:58:43.561 INFO:teuthology.orchestra.run.smithi084.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-09-20T22:58:43.652 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-20T22:58:43.652 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:58:43.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:43 smithi084 ceph-mon[29774]: pgmap v1441: 142 pgs: 2 active+remapped+backfill_toofull, 140 active+clean; 583 KiB data, 647 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 1/597 objects misplaced (0.168%) 2024-09-20T22:58:43.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:43 smithi084 ceph-mon[29774]: 1.5 scrub starts 2024-09-20T22:58:43.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:43 smithi084 ceph-mon[29774]: 1.5 scrub ok 2024-09-20T22:58:43.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:43 smithi084 ceph-mon[29774]: 6.1e scrub starts 2024-09-20T22:58:43.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:43 smithi084 ceph-mon[29774]: 6.1e scrub ok 2024-09-20T22:58:43.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:43 smithi084 ceph-mon[29774]: 5.1a scrub starts 2024-09-20T22:58:43.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:43 smithi084 ceph-mon[29774]: 5.1a scrub ok 2024-09-20T22:58:43.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:43 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/1462868383' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-09-20T22:58:43.866 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:43 smithi084 ceph-mon[40024]: pgmap v1441: 142 pgs: 2 active+remapped+backfill_toofull, 140 active+clean; 583 KiB data, 647 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 1/597 objects misplaced (0.168%) 2024-09-20T22:58:43.866 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:43 smithi084 ceph-mon[40024]: 1.5 scrub starts 2024-09-20T22:58:43.866 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:43 smithi084 ceph-mon[40024]: 1.5 scrub ok 2024-09-20T22:58:43.866 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:43 smithi084 ceph-mon[40024]: 6.1e scrub starts 2024-09-20T22:58:43.866 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:43 smithi084 ceph-mon[40024]: 6.1e scrub ok 2024-09-20T22:58:43.866 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:43 smithi084 ceph-mon[40024]: 5.1a scrub starts 2024-09-20T22:58:43.866 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:43 smithi084 ceph-mon[40024]: 5.1a scrub ok 2024-09-20T22:58:43.866 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:43 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/1462868383' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-09-20T22:58:43.947 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:43 smithi154 ceph-mon[33264]: pgmap v1441: 142 pgs: 2 active+remapped+backfill_toofull, 140 active+clean; 583 KiB data, 647 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 1/597 objects misplaced (0.168%) 2024-09-20T22:58:43.947 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:43 smithi154 ceph-mon[33264]: 1.5 scrub starts 2024-09-20T22:58:43.947 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:43 smithi154 ceph-mon[33264]: 1.5 scrub ok 2024-09-20T22:58:43.947 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:43 smithi154 ceph-mon[33264]: 6.1e scrub starts 2024-09-20T22:58:43.947 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:43 smithi154 ceph-mon[33264]: 6.1e scrub ok 2024-09-20T22:58:43.947 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:43 smithi154 ceph-mon[33264]: 5.1a scrub starts 2024-09-20T22:58:43.947 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:43 smithi154 ceph-mon[33264]: 5.1a scrub ok 2024-09-20T22:58:43.947 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:43 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/1462868383' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-09-20T22:58:44.217 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.7 dump_blocked_ops 2024-09-20T22:58:44.319 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd reweight 4 1 2024-09-20T22:58:44.463 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:58:44.568 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:58:44.834 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:44 smithi084 ceph-mon[40024]: 6.13 scrub starts 2024-09-20T22:58:44.834 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:44 smithi084 ceph-mon[40024]: 6.13 scrub ok 2024-09-20T22:58:44.834 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:44 smithi084 ceph-mon[40024]: 4.a scrub starts 2024-09-20T22:58:44.834 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:44 smithi084 ceph-mon[40024]: 4.a scrub ok 2024-09-20T22:58:44.834 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:44 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/1462868383' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-09-20T22:58:44.834 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:44 smithi084 ceph-mon[40024]: osdmap e1019: 8 total, 8 up, 8 in 2024-09-20T22:58:44.834 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:44 smithi084 ceph-mon[40024]: 6.9 scrub starts 2024-09-20T22:58:44.834 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:44 smithi084 ceph-mon[40024]: 6.9 scrub ok 2024-09-20T22:58:44.834 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:44 smithi084 ceph-mon[29774]: 6.13 scrub starts 2024-09-20T22:58:44.834 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:44 smithi084 ceph-mon[29774]: 6.13 scrub ok 2024-09-20T22:58:44.835 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:44 smithi084 ceph-mon[29774]: 4.a scrub starts 2024-09-20T22:58:44.835 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:44 smithi084 ceph-mon[29774]: 4.a scrub ok 2024-09-20T22:58:44.835 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:44 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/1462868383' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-09-20T22:58:44.835 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:44 smithi084 ceph-mon[29774]: osdmap e1019: 8 total, 8 up, 8 in 2024-09-20T22:58:44.835 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:44 smithi084 ceph-mon[29774]: 6.9 scrub starts 2024-09-20T22:58:44.835 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:44 smithi084 ceph-mon[29774]: 6.9 scrub ok 2024-09-20T22:58:44.842 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:44 smithi154 ceph-mon[33264]: 6.13 scrub starts 2024-09-20T22:58:44.842 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:44 smithi154 ceph-mon[33264]: 6.13 scrub ok 2024-09-20T22:58:44.842 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:44 smithi154 ceph-mon[33264]: 4.a scrub starts 2024-09-20T22:58:44.842 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:44 smithi154 ceph-mon[33264]: 4.a scrub ok 2024-09-20T22:58:44.842 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:44 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/1462868383' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-09-20T22:58:44.842 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:44 smithi154 ceph-mon[33264]: osdmap e1019: 8 total, 8 up, 8 in 2024-09-20T22:58:44.842 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:44 smithi154 ceph-mon[33264]: 6.9 scrub starts 2024-09-20T22:58:44.843 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:44 smithi154 ceph-mon[33264]: 6.9 scrub ok 2024-09-20T22:58:44.942 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-20T22:58:44.942 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:58:45.557 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.7 dump_historic_ops 2024-09-20T22:58:45.576 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.4 to 1 (10000) 2024-09-20T22:58:45.805 INFO:teuthology.orchestra.run.smithi154.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.b/config 2024-09-20T22:58:45.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:45 smithi084 ceph-mon[29774]: pgmap v1443: 142 pgs: 2 active+remapped+backfill_toofull, 140 active+clean; 583 KiB data, 647 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 1 op/s; 1/597 objects misplaced (0.168%) 2024-09-20T22:58:45.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:45 smithi084 ceph-mon[29774]: 6.1b scrub starts 2024-09-20T22:58:45.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:45 smithi084 ceph-mon[29774]: 6.1b scrub ok 2024-09-20T22:58:45.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:45 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/117365379' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-09-20T22:58:45.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:45 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-09-20T22:58:45.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:45 smithi084 ceph-mon[40024]: pgmap v1443: 142 pgs: 2 active+remapped+backfill_toofull, 140 active+clean; 583 KiB data, 647 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 1 op/s; 1/597 objects misplaced (0.168%) 2024-09-20T22:58:45.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:45 smithi084 ceph-mon[40024]: 6.1b scrub starts 2024-09-20T22:58:45.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:45 smithi084 ceph-mon[40024]: 6.1b scrub ok 2024-09-20T22:58:45.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:45 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/117365379' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-09-20T22:58:45.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:45 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-09-20T22:58:45.926 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:45 smithi154 ceph-mon[33264]: pgmap v1443: 142 pgs: 2 active+remapped+backfill_toofull, 140 active+clean; 583 KiB data, 647 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 1 op/s; 1/597 objects misplaced (0.168%) 2024-09-20T22:58:45.926 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:45 smithi154 ceph-mon[33264]: 6.1b scrub starts 2024-09-20T22:58:45.926 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:45 smithi154 ceph-mon[33264]: 6.1b scrub ok 2024-09-20T22:58:45.926 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:45 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/117365379' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-09-20T22:58:45.926 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:45 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-09-20T22:58:46.266 INFO:teuthology.orchestra.run.smithi154.stdout:ERROR: (22) Invalid argument 2024-09-20T22:58:46.266 INFO:teuthology.orchestra.run.smithi154.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:58:46.409 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd primary-affinity 4 1 2024-09-20T22:58:46.658 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:58:46.779 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:46 smithi084 ceph-mon[40024]: 3.1c scrub starts 2024-09-20T22:58:46.779 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:46 smithi084 ceph-mon[40024]: 3.1c scrub ok 2024-09-20T22:58:46.779 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:46 smithi084 ceph-mon[40024]: 4.1d scrub starts 2024-09-20T22:58:46.779 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:46 smithi084 ceph-mon[40024]: 4.1d scrub ok 2024-09-20T22:58:46.780 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:46 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-20T22:58:46.780 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:46 smithi084 ceph-mon[40024]: osdmap e1020: 8 total, 8 up, 8 in 2024-09-20T22:58:46.780 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:46 smithi084 ceph-mon[40024]: 2.0 scrub starts 2024-09-20T22:58:46.780 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:46 smithi084 ceph-mon[40024]: 2.0 scrub ok 2024-09-20T22:58:46.780 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:46 smithi084 ceph-mon[29774]: 3.1c scrub starts 2024-09-20T22:58:46.780 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:46 smithi084 ceph-mon[29774]: 3.1c scrub ok 2024-09-20T22:58:46.780 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:46 smithi084 ceph-mon[29774]: 4.1d scrub starts 2024-09-20T22:58:46.780 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:46 smithi084 ceph-mon[29774]: 4.1d scrub ok 2024-09-20T22:58:46.780 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:46 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-20T22:58:46.780 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:46 smithi084 ceph-mon[29774]: osdmap e1020: 8 total, 8 up, 8 in 2024-09-20T22:58:46.780 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:46 smithi084 ceph-mon[29774]: 2.0 scrub starts 2024-09-20T22:58:46.780 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:46 smithi084 ceph-mon[29774]: 2.0 scrub ok 2024-09-20T22:58:46.847 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:46 smithi154 ceph-mon[33264]: 3.1c scrub starts 2024-09-20T22:58:46.847 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:46 smithi154 ceph-mon[33264]: 3.1c scrub ok 2024-09-20T22:58:46.847 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:46 smithi154 ceph-mon[33264]: 4.1d scrub starts 2024-09-20T22:58:46.847 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:46 smithi154 ceph-mon[33264]: 4.1d scrub ok 2024-09-20T22:58:46.847 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:46 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-20T22:58:46.847 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:46 smithi154 ceph-mon[33264]: osdmap e1020: 8 total, 8 up, 8 in 2024-09-20T22:58:46.848 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:46 smithi154 ceph-mon[33264]: 2.0 scrub starts 2024-09-20T22:58:46.848 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:46 smithi154 ceph-mon[33264]: 2.0 scrub ok 2024-09-20T22:58:46.849 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-20T22:58:47.205 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:58:47.590 INFO:teuthology.orchestra.run.smithi084.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-09-20T22:58:47.675 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-20T22:58:47.675 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:58:47.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:47 smithi084 ceph-mon[40024]: 4.19 scrub starts 2024-09-20T22:58:47.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:47 smithi084 ceph-mon[40024]: 4.19 scrub ok 2024-09-20T22:58:47.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:47 smithi084 ceph-mon[40024]: pgmap v1445: 142 pgs: 2 active+remapped+backfill_toofull, 140 active+clean; 583 KiB data, 647 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 1/597 objects misplaced (0.168%) 2024-09-20T22:58:47.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:47 smithi084 ceph-mon[40024]: 4.15 scrub starts 2024-09-20T22:58:47.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:47 smithi084 ceph-mon[40024]: 4.15 scrub ok 2024-09-20T22:58:47.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:47 smithi084 ceph-mon[40024]: 5.6 scrub starts 2024-09-20T22:58:47.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:47 smithi084 ceph-mon[40024]: 5.6 scrub ok 2024-09-20T22:58:47.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:47 smithi084 ceph-mon[40024]: purged_snaps scrub starts 2024-09-20T22:58:47.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:47 smithi084 ceph-mon[40024]: purged_snaps scrub ok 2024-09-20T22:58:47.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:47 smithi084 ceph-mon[40024]: osdmap e1021: 8 total, 8 up, 8 in 2024-09-20T22:58:47.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:47 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/534536193' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-09-20T22:58:47.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:47 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-09-20T22:58:47.866 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:47 smithi084 ceph-mon[29774]: 4.19 scrub starts 2024-09-20T22:58:47.866 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:47 smithi084 ceph-mon[29774]: 4.19 scrub ok 2024-09-20T22:58:47.866 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:47 smithi084 ceph-mon[29774]: pgmap v1445: 142 pgs: 2 active+remapped+backfill_toofull, 140 active+clean; 583 KiB data, 647 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 1/597 objects misplaced (0.168%) 2024-09-20T22:58:47.866 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:47 smithi084 ceph-mon[29774]: 4.15 scrub starts 2024-09-20T22:58:47.866 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:47 smithi084 ceph-mon[29774]: 4.15 scrub ok 2024-09-20T22:58:47.866 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:47 smithi084 ceph-mon[29774]: 5.6 scrub starts 2024-09-20T22:58:47.866 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:47 smithi084 ceph-mon[29774]: 5.6 scrub ok 2024-09-20T22:58:47.866 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:47 smithi084 ceph-mon[29774]: purged_snaps scrub starts 2024-09-20T22:58:47.866 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:47 smithi084 ceph-mon[29774]: purged_snaps scrub ok 2024-09-20T22:58:47.866 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:47 smithi084 ceph-mon[29774]: osdmap e1021: 8 total, 8 up, 8 in 2024-09-20T22:58:47.866 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:47 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/534536193' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-09-20T22:58:47.866 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:47 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-09-20T22:58:48.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:47 smithi154 ceph-mon[33264]: 4.19 scrub starts 2024-09-20T22:58:48.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:47 smithi154 ceph-mon[33264]: 4.19 scrub ok 2024-09-20T22:58:48.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:47 smithi154 ceph-mon[33264]: pgmap v1445: 142 pgs: 2 active+remapped+backfill_toofull, 140 active+clean; 583 KiB data, 647 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 1/597 objects misplaced (0.168%) 2024-09-20T22:58:48.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:47 smithi154 ceph-mon[33264]: 4.15 scrub starts 2024-09-20T22:58:48.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:47 smithi154 ceph-mon[33264]: 4.15 scrub ok 2024-09-20T22:58:48.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:47 smithi154 ceph-mon[33264]: 5.6 scrub starts 2024-09-20T22:58:48.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:47 smithi154 ceph-mon[33264]: 5.6 scrub ok 2024-09-20T22:58:48.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:47 smithi154 ceph-mon[33264]: purged_snaps scrub starts 2024-09-20T22:58:48.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:47 smithi154 ceph-mon[33264]: purged_snaps scrub ok 2024-09-20T22:58:48.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:47 smithi154 ceph-mon[33264]: osdmap e1021: 8 total, 8 up, 8 in 2024-09-20T22:58:48.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:47 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/534536193' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-09-20T22:58:48.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:47 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-09-20T22:58:48.323 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:58:48 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: ::ffff:172.21.15.154 - - [20/Sep/2024:22:58:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-20T22:58:48.351 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd reweight 5 1 2024-09-20T22:58:48.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:48 smithi084 ceph-mon[29774]: 3.1e scrub starts 2024-09-20T22:58:48.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:48 smithi084 ceph-mon[29774]: 3.1e scrub ok 2024-09-20T22:58:48.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:48 smithi084 ceph-mon[29774]: 6.e scrub starts 2024-09-20T22:58:48.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:48 smithi084 ceph-mon[29774]: 6.e scrub ok 2024-09-20T22:58:48.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:48 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-09-20T22:58:48.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:48 smithi084 ceph-mon[29774]: osdmap e1022: 8 total, 8 up, 8 in 2024-09-20T22:58:48.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:48 smithi084 ceph-mon[29774]: 3.1f scrub starts 2024-09-20T22:58:48.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:48 smithi084 ceph-mon[29774]: 3.1f scrub ok 2024-09-20T22:58:48.866 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:48 smithi084 ceph-mon[40024]: 3.1e scrub starts 2024-09-20T22:58:48.866 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:48 smithi084 ceph-mon[40024]: 3.1e scrub ok 2024-09-20T22:58:48.866 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:48 smithi084 ceph-mon[40024]: 6.e scrub starts 2024-09-20T22:58:48.866 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:48 smithi084 ceph-mon[40024]: 6.e scrub ok 2024-09-20T22:58:48.866 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:48 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-09-20T22:58:48.866 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:48 smithi084 ceph-mon[40024]: osdmap e1022: 8 total, 8 up, 8 in 2024-09-20T22:58:48.866 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:48 smithi084 ceph-mon[40024]: 3.1f scrub starts 2024-09-20T22:58:48.866 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:48 smithi084 ceph-mon[40024]: 3.1f scrub ok 2024-09-20T22:58:48.992 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.3 dump_blocked_ops 2024-09-20T22:58:49.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:48 smithi154 ceph-mon[33264]: 3.1e scrub starts 2024-09-20T22:58:49.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:48 smithi154 ceph-mon[33264]: 3.1e scrub ok 2024-09-20T22:58:49.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:48 smithi154 ceph-mon[33264]: 6.e scrub starts 2024-09-20T22:58:49.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:48 smithi154 ceph-mon[33264]: 6.e scrub ok 2024-09-20T22:58:49.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:48 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-09-20T22:58:49.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:48 smithi154 ceph-mon[33264]: osdmap e1022: 8 total, 8 up, 8 in 2024-09-20T22:58:49.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:48 smithi154 ceph-mon[33264]: 3.1f scrub starts 2024-09-20T22:58:49.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:48 smithi154 ceph-mon[33264]: 3.1f scrub ok 2024-09-20T22:58:49.041 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:58:49.555 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:58:49.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:49 smithi084 ceph-mon[40024]: pgmap v1448: 142 pgs: 2 active+remapped+backfill_toofull, 140 active+clean; 583 KiB data, 647 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 1/597 objects misplaced (0.168%) 2024-09-20T22:58:49.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:49 smithi084 ceph-mon[40024]: 4.7 scrub starts 2024-09-20T22:58:49.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:49 smithi084 ceph-mon[40024]: 4.7 scrub ok 2024-09-20T22:58:49.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:49 smithi084 ceph-mon[40024]: osdmap e1023: 8 total, 8 up, 8 in 2024-09-20T22:58:49.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:49 smithi084 ceph-mon[40024]: 6.15 scrub starts 2024-09-20T22:58:49.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:49 smithi084 ceph-mon[40024]: 6.15 scrub ok 2024-09-20T22:58:49.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:49 smithi084 ceph-mon[29774]: pgmap v1448: 142 pgs: 2 active+remapped+backfill_toofull, 140 active+clean; 583 KiB data, 647 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 1/597 objects misplaced (0.168%) 2024-09-20T22:58:49.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:49 smithi084 ceph-mon[29774]: 4.7 scrub starts 2024-09-20T22:58:49.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:49 smithi084 ceph-mon[29774]: 4.7 scrub ok 2024-09-20T22:58:49.866 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:49 smithi084 ceph-mon[29774]: osdmap e1023: 8 total, 8 up, 8 in 2024-09-20T22:58:49.866 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:49 smithi084 ceph-mon[29774]: 6.15 scrub starts 2024-09-20T22:58:49.866 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:49 smithi084 ceph-mon[29774]: 6.15 scrub ok 2024-09-20T22:58:49.866 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:49 smithi154 ceph-mon[33264]: pgmap v1448: 142 pgs: 2 active+remapped+backfill_toofull, 140 active+clean; 583 KiB data, 647 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 1/597 objects misplaced (0.168%) 2024-09-20T22:58:49.866 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:49 smithi154 ceph-mon[33264]: 4.7 scrub starts 2024-09-20T22:58:49.866 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:49 smithi154 ceph-mon[33264]: 4.7 scrub ok 2024-09-20T22:58:49.866 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:49 smithi154 ceph-mon[33264]: osdmap e1023: 8 total, 8 up, 8 in 2024-09-20T22:58:49.866 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:49 smithi154 ceph-mon[33264]: 6.15 scrub starts 2024-09-20T22:58:49.866 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:49 smithi154 ceph-mon[33264]: 6.15 scrub ok 2024-09-20T22:58:50.166 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-20T22:58:50.166 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:58:50.254 INFO:journalctl@ceph.iscsi.iscsi.a.smithi154.stdout:Sep 20 22:58:49 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-iscsi-iscsi-a[59730]: debug there is no tcmu-runner data available 2024-09-20T22:58:50.611 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.5 to 1 (10000) 2024-09-20T22:58:50.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:50 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/3994611725' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-20T22:58:50.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:50 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-20T22:58:50.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:50 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/3994611725' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-20T22:58:50.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:50 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-20T22:58:50.940 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph daemon osd.3 dump_historic_ops 2024-09-20T22:58:51.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:50 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/3994611725' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-20T22:58:51.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:50 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-20T22:58:51.569 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd primary-affinity 5 1 2024-09-20T22:58:51.635 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:58:51.863 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:51 smithi084 ceph-mon[29774]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:58:51.864 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:51 smithi084 ceph-mon[29774]: pgmap v1450: 142 pgs: 3 activating, 2 active+remapped+backfill_toofull, 137 active+clean; 583 KiB data, 647 MiB used, 715 GiB / 715 GiB avail; 2.3 KiB/s rd, 2 op/s; 1/597 objects misplaced (0.168%); 152 B/s, 5 objects/s recovering 2024-09-20T22:58:51.864 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:51 smithi084 ceph-mon[29774]: 3.1d scrub starts 2024-09-20T22:58:51.864 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:51 smithi084 ceph-mon[29774]: 3.1d scrub ok 2024-09-20T22:58:51.864 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:51 smithi084 ceph-mon[29774]: 4.c scrub starts 2024-09-20T22:58:51.864 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:51 smithi084 ceph-mon[29774]: 4.c scrub ok 2024-09-20T22:58:51.864 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:51 smithi084 ceph-mon[29774]: Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-09-20T22:58:51.864 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:51 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-20T22:58:51.864 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:51 smithi084 ceph-mon[29774]: osdmap e1024: 8 total, 8 up, 8 in 2024-09-20T22:58:51.864 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:51 smithi084 ceph-mon[40024]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:58:51.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:51 smithi084 ceph-mon[40024]: pgmap v1450: 142 pgs: 3 activating, 2 active+remapped+backfill_toofull, 137 active+clean; 583 KiB data, 647 MiB used, 715 GiB / 715 GiB avail; 2.3 KiB/s rd, 2 op/s; 1/597 objects misplaced (0.168%); 152 B/s, 5 objects/s recovering 2024-09-20T22:58:51.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:51 smithi084 ceph-mon[40024]: 3.1d scrub starts 2024-09-20T22:58:51.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:51 smithi084 ceph-mon[40024]: 3.1d scrub ok 2024-09-20T22:58:51.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:51 smithi084 ceph-mon[40024]: 4.c scrub starts 2024-09-20T22:58:51.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:51 smithi084 ceph-mon[40024]: 4.c scrub ok 2024-09-20T22:58:51.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:51 smithi084 ceph-mon[40024]: Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-09-20T22:58:51.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:51 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-20T22:58:51.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:51 smithi084 ceph-mon[40024]: osdmap e1024: 8 total, 8 up, 8 in 2024-09-20T22:58:52.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:51 smithi154 ceph-mon[33264]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:58:52.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:51 smithi154 ceph-mon[33264]: pgmap v1450: 142 pgs: 3 activating, 2 active+remapped+backfill_toofull, 137 active+clean; 583 KiB data, 647 MiB used, 715 GiB / 715 GiB avail; 2.3 KiB/s rd, 2 op/s; 1/597 objects misplaced (0.168%); 152 B/s, 5 objects/s recovering 2024-09-20T22:58:52.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:51 smithi154 ceph-mon[33264]: 3.1d scrub starts 2024-09-20T22:58:52.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:51 smithi154 ceph-mon[33264]: 3.1d scrub ok 2024-09-20T22:58:52.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:51 smithi154 ceph-mon[33264]: 4.c scrub starts 2024-09-20T22:58:52.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:51 smithi154 ceph-mon[33264]: 4.c scrub ok 2024-09-20T22:58:52.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:51 smithi154 ceph-mon[33264]: Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-09-20T22:58:52.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:51 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-20T22:58:52.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:51 smithi154 ceph-mon[33264]: osdmap e1024: 8 total, 8 up, 8 in 2024-09-20T22:58:52.103 INFO:teuthology.orchestra.run.smithi084.stdout:ERROR: (22) Invalid argument 2024-09-20T22:58:52.104 INFO:teuthology.orchestra.run.smithi084.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-20T22:58:52.354 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:58:52.842 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:52 smithi084 ceph-mon[29774]: 6.2 scrub starts 2024-09-20T22:58:52.842 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:52 smithi084 ceph-mon[29774]: 6.2 scrub ok 2024-09-20T22:58:52.842 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:52 smithi084 ceph-mon[40024]: 6.2 scrub starts 2024-09-20T22:58:52.842 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:52 smithi084 ceph-mon[40024]: 6.2 scrub ok 2024-09-20T22:58:53.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:52 smithi154 ceph-mon[33264]: 6.2 scrub starts 2024-09-20T22:58:53.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:52 smithi154 ceph-mon[33264]: 6.2 scrub ok 2024-09-20T22:58:53.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:53 smithi084 ceph-mon[29774]: pgmap v1452: 142 pgs: 3 activating, 2 active+remapped+backfill_toofull, 137 active+clean; 583 KiB data, 647 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 1 op/s; 1/597 objects misplaced (0.168%); 123 B/s, 4 objects/s recovering 2024-09-20T22:58:53.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:53 smithi084 ceph-mon[29774]: 6.16 scrub starts 2024-09-20T22:58:53.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:53 smithi084 ceph-mon[29774]: 6.16 scrub ok 2024-09-20T22:58:53.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:53 smithi084 ceph-mon[29774]: 4.1b scrub starts 2024-09-20T22:58:53.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:53 smithi084 ceph-mon[29774]: 4.1b scrub ok 2024-09-20T22:58:53.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:53 smithi084 ceph-mon[29774]: purged_snaps scrub starts 2024-09-20T22:58:53.866 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:53 smithi084 ceph-mon[29774]: purged_snaps scrub ok 2024-09-20T22:58:53.866 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:53 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/201310694' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-09-20T22:58:53.866 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:53 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:58:53.866 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:53 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:58:53.866 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:53 smithi084 ceph-mon[40024]: pgmap v1452: 142 pgs: 3 activating, 2 active+remapped+backfill_toofull, 137 active+clean; 583 KiB data, 647 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 1 op/s; 1/597 objects misplaced (0.168%); 123 B/s, 4 objects/s recovering 2024-09-20T22:58:53.866 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:53 smithi084 ceph-mon[40024]: 6.16 scrub starts 2024-09-20T22:58:53.866 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:53 smithi084 ceph-mon[40024]: 6.16 scrub ok 2024-09-20T22:58:53.866 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:53 smithi084 ceph-mon[40024]: 4.1b scrub starts 2024-09-20T22:58:53.866 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:53 smithi084 ceph-mon[40024]: 4.1b scrub ok 2024-09-20T22:58:53.867 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:53 smithi084 ceph-mon[40024]: purged_snaps scrub starts 2024-09-20T22:58:53.867 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:53 smithi084 ceph-mon[40024]: purged_snaps scrub ok 2024-09-20T22:58:53.867 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:53 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/201310694' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-09-20T22:58:53.867 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:53 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:58:53.867 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:53 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:58:54.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:53 smithi154 ceph-mon[33264]: pgmap v1452: 142 pgs: 3 activating, 2 active+remapped+backfill_toofull, 137 active+clean; 583 KiB data, 647 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 1 op/s; 1/597 objects misplaced (0.168%); 123 B/s, 4 objects/s recovering 2024-09-20T22:58:54.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:53 smithi154 ceph-mon[33264]: 6.16 scrub starts 2024-09-20T22:58:54.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:53 smithi154 ceph-mon[33264]: 6.16 scrub ok 2024-09-20T22:58:54.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:53 smithi154 ceph-mon[33264]: 4.1b scrub starts 2024-09-20T22:58:54.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:53 smithi154 ceph-mon[33264]: 4.1b scrub ok 2024-09-20T22:58:54.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:53 smithi154 ceph-mon[33264]: purged_snaps scrub starts 2024-09-20T22:58:54.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:53 smithi154 ceph-mon[33264]: purged_snaps scrub ok 2024-09-20T22:58:54.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:53 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/201310694' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-09-20T22:58:54.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:53 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:58:54.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:53 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:58:54.385 INFO:teuthology.orchestra.run.smithi084.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-09-20T22:58:54.766 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:54 smithi084 ceph-mon[29774]: 6.5 deep-scrub starts 2024-09-20T22:58:54.766 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:54 smithi084 ceph-mon[29774]: 6.5 deep-scrub ok 2024-09-20T22:58:54.766 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:54 smithi084 ceph-mon[29774]: 5.1c scrub starts 2024-09-20T22:58:54.766 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:54 smithi084 ceph-mon[29774]: 5.1c scrub ok 2024-09-20T22:58:54.766 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:54 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/201310694' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-09-20T22:58:54.766 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:54 smithi084 ceph-mon[29774]: osdmap e1025: 8 total, 8 up, 8 in 2024-09-20T22:58:54.767 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:54 smithi084 ceph-mon[40024]: 6.5 deep-scrub starts 2024-09-20T22:58:54.767 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:54 smithi084 ceph-mon[40024]: 6.5 deep-scrub ok 2024-09-20T22:58:54.767 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:54 smithi084 ceph-mon[40024]: 5.1c scrub starts 2024-09-20T22:58:54.767 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:54 smithi084 ceph-mon[40024]: 5.1c scrub ok 2024-09-20T22:58:54.767 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:54 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/201310694' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-09-20T22:58:54.767 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:54 smithi084 ceph-mon[40024]: osdmap e1025: 8 total, 8 up, 8 in 2024-09-20T22:58:55.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:54 smithi154 ceph-mon[33264]: 6.5 deep-scrub starts 2024-09-20T22:58:55.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:54 smithi154 ceph-mon[33264]: 6.5 deep-scrub ok 2024-09-20T22:58:55.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:54 smithi154 ceph-mon[33264]: 5.1c scrub starts 2024-09-20T22:58:55.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:54 smithi154 ceph-mon[33264]: 5.1c scrub ok 2024-09-20T22:58:55.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:54 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/201310694' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-09-20T22:58:55.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:54 smithi154 ceph-mon[33264]: osdmap e1025: 8 total, 8 up, 8 in 2024-09-20T22:58:55.144 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd reweight 6 1 2024-09-20T22:58:55.403 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:58:55.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:55 smithi084 ceph-mon[40024]: pgmap v1453: 142 pgs: 2 active+remapped+backfill_toofull, 140 active+clean; 583 KiB data, 648 MiB used, 715 GiB / 715 GiB avail; 2.4 KiB/s rd, 2 op/s; 1/597 objects misplaced (0.168%); 104 B/s, 3 objects/s recovering 2024-09-20T22:58:55.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:55 smithi084 ceph-mon[40024]: purged_snaps scrub starts 2024-09-20T22:58:55.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:55 smithi084 ceph-mon[40024]: purged_snaps scrub ok 2024-09-20T22:58:55.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:55 smithi084 ceph-mon[40024]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive) 2024-09-20T22:58:55.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:55 smithi084 ceph-mon[40024]: osdmap e1026: 8 total, 8 up, 8 in 2024-09-20T22:58:55.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:55 smithi084 ceph-mon[29774]: pgmap v1453: 142 pgs: 2 active+remapped+backfill_toofull, 140 active+clean; 583 KiB data, 648 MiB used, 715 GiB / 715 GiB avail; 2.4 KiB/s rd, 2 op/s; 1/597 objects misplaced (0.168%); 104 B/s, 3 objects/s recovering 2024-09-20T22:58:55.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:55 smithi084 ceph-mon[29774]: purged_snaps scrub starts 2024-09-20T22:58:55.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:55 smithi084 ceph-mon[29774]: purged_snaps scrub ok 2024-09-20T22:58:55.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:55 smithi084 ceph-mon[29774]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive) 2024-09-20T22:58:55.866 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:55 smithi084 ceph-mon[29774]: osdmap e1026: 8 total, 8 up, 8 in 2024-09-20T22:58:56.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:55 smithi154 ceph-mon[33264]: pgmap v1453: 142 pgs: 2 active+remapped+backfill_toofull, 140 active+clean; 583 KiB data, 648 MiB used, 715 GiB / 715 GiB avail; 2.4 KiB/s rd, 2 op/s; 1/597 objects misplaced (0.168%); 104 B/s, 3 objects/s recovering 2024-09-20T22:58:56.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:55 smithi154 ceph-mon[33264]: purged_snaps scrub starts 2024-09-20T22:58:56.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:55 smithi154 ceph-mon[33264]: purged_snaps scrub ok 2024-09-20T22:58:56.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:55 smithi154 ceph-mon[33264]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive) 2024-09-20T22:58:56.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:55 smithi154 ceph-mon[33264]: osdmap e1026: 8 total, 8 up, 8 in 2024-09-20T22:58:56.618 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.6 to 1 (10000) 2024-09-20T22:58:56.924 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:56 smithi084 ceph-mon[40024]: 5.0 scrub starts 2024-09-20T22:58:56.924 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:56 smithi084 ceph-mon[40024]: 5.0 scrub ok 2024-09-20T22:58:56.924 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:56 smithi084 ceph-mon[40024]: 3.c scrub starts 2024-09-20T22:58:56.924 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:56 smithi084 ceph-mon[40024]: 3.c scrub ok 2024-09-20T22:58:56.924 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:56 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/277559574' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-09-20T22:58:56.924 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:56 smithi084 ceph-mon[40024]: 6.1a deep-scrub starts 2024-09-20T22:58:56.924 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:56 smithi084 ceph-mon[40024]: 6.1a deep-scrub ok 2024-09-20T22:58:56.924 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:56 smithi084 ceph-mon[29774]: 5.0 scrub starts 2024-09-20T22:58:56.924 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:56 smithi084 ceph-mon[29774]: 5.0 scrub ok 2024-09-20T22:58:56.924 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:56 smithi084 ceph-mon[29774]: 3.c scrub starts 2024-09-20T22:58:56.925 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:56 smithi084 ceph-mon[29774]: 3.c scrub ok 2024-09-20T22:58:56.925 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:56 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/277559574' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-09-20T22:58:56.925 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:56 smithi084 ceph-mon[29774]: 6.1a deep-scrub starts 2024-09-20T22:58:56.925 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:56 smithi084 ceph-mon[29774]: 6.1a deep-scrub ok 2024-09-20T22:58:57.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:56 smithi154 ceph-mon[33264]: 5.0 scrub starts 2024-09-20T22:58:57.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:56 smithi154 ceph-mon[33264]: 5.0 scrub ok 2024-09-20T22:58:57.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:56 smithi154 ceph-mon[33264]: 3.c scrub starts 2024-09-20T22:58:57.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:56 smithi154 ceph-mon[33264]: 3.c scrub ok 2024-09-20T22:58:57.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:56 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/277559574' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-09-20T22:58:57.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:56 smithi154 ceph-mon[33264]: 6.1a deep-scrub starts 2024-09-20T22:58:57.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:56 smithi154 ceph-mon[33264]: 6.1a deep-scrub ok 2024-09-20T22:58:57.336 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd primary-affinity 6 1 2024-09-20T22:58:57.582 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:58:57.977 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:57 smithi084 ceph-mon[40024]: pgmap v1456: 142 pgs: 2 active+remapped+backfill_toofull, 140 active+clean; 583 KiB data, 648 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 1/597 objects misplaced (0.168%) 2024-09-20T22:58:57.977 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:57 smithi084 ceph-mon[40024]: 5.a scrub starts 2024-09-20T22:58:57.977 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:57 smithi084 ceph-mon[40024]: 5.a scrub ok 2024-09-20T22:58:57.977 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:57 smithi084 ceph-mon[40024]: purged_snaps scrub starts 2024-09-20T22:58:57.977 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:57 smithi084 ceph-mon[40024]: purged_snaps scrub ok 2024-09-20T22:58:57.977 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:57 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/277559574' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-20T22:58:57.977 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:57 smithi084 ceph-mon[40024]: osdmap e1027: 8 total, 8 up, 8 in 2024-09-20T22:58:57.978 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:57 smithi084 ceph-mon[40024]: 5.e scrub starts 2024-09-20T22:58:57.978 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:57 smithi084 ceph-mon[40024]: 5.e scrub ok 2024-09-20T22:58:57.978 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:57 smithi084 ceph-mon[40024]: 6.12 scrub starts 2024-09-20T22:58:57.978 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:57 smithi084 ceph-mon[40024]: 6.12 scrub ok 2024-09-20T22:58:57.978 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:57 smithi084 ceph-mon[29774]: pgmap v1456: 142 pgs: 2 active+remapped+backfill_toofull, 140 active+clean; 583 KiB data, 648 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 1/597 objects misplaced (0.168%) 2024-09-20T22:58:57.978 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:57 smithi084 ceph-mon[29774]: 5.a scrub starts 2024-09-20T22:58:57.978 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:57 smithi084 ceph-mon[29774]: 5.a scrub ok 2024-09-20T22:58:57.978 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:57 smithi084 ceph-mon[29774]: purged_snaps scrub starts 2024-09-20T22:58:57.978 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:57 smithi084 ceph-mon[29774]: purged_snaps scrub ok 2024-09-20T22:58:57.978 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:57 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/277559574' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-20T22:58:57.978 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:57 smithi084 ceph-mon[29774]: osdmap e1027: 8 total, 8 up, 8 in 2024-09-20T22:58:57.979 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:57 smithi084 ceph-mon[29774]: 5.e scrub starts 2024-09-20T22:58:57.979 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:57 smithi084 ceph-mon[29774]: 5.e scrub ok 2024-09-20T22:58:57.979 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:57 smithi084 ceph-mon[29774]: 6.12 scrub starts 2024-09-20T22:58:57.979 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:57 smithi084 ceph-mon[29774]: 6.12 scrub ok 2024-09-20T22:58:58.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:57 smithi154 ceph-mon[33264]: pgmap v1456: 142 pgs: 2 active+remapped+backfill_toofull, 140 active+clean; 583 KiB data, 648 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 1/597 objects misplaced (0.168%) 2024-09-20T22:58:58.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:57 smithi154 ceph-mon[33264]: 5.a scrub starts 2024-09-20T22:58:58.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:57 smithi154 ceph-mon[33264]: 5.a scrub ok 2024-09-20T22:58:58.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:57 smithi154 ceph-mon[33264]: purged_snaps scrub starts 2024-09-20T22:58:58.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:57 smithi154 ceph-mon[33264]: purged_snaps scrub ok 2024-09-20T22:58:58.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:57 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/277559574' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-20T22:58:58.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:57 smithi154 ceph-mon[33264]: osdmap e1027: 8 total, 8 up, 8 in 2024-09-20T22:58:58.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:57 smithi154 ceph-mon[33264]: 5.e scrub starts 2024-09-20T22:58:58.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:57 smithi154 ceph-mon[33264]: 5.e scrub ok 2024-09-20T22:58:58.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:57 smithi154 ceph-mon[33264]: 6.12 scrub starts 2024-09-20T22:58:58.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:57 smithi154 ceph-mon[33264]: 6.12 scrub ok 2024-09-20T22:58:58.385 INFO:teuthology.orchestra.run.smithi084.stderr:set osd.6 primary-affinity to 1 (8655362) 2024-09-20T22:58:58.614 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:58:58 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: ::ffff:172.21.15.154 - - [20/Sep/2024:22:58:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-20T22:58:58.947 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:58 smithi084 ceph-mon[40024]: 5.c scrub starts 2024-09-20T22:58:58.948 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:58 smithi084 ceph-mon[40024]: 5.c scrub ok 2024-09-20T22:58:58.948 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:58 smithi084 ceph-mon[40024]: 5.1e scrub starts 2024-09-20T22:58:58.948 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:58 smithi084 ceph-mon[40024]: 5.1e scrub ok 2024-09-20T22:58:58.948 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:58 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/3544218674' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-09-20T22:58:58.948 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:58 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/3544218674' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-09-20T22:58:58.948 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:58 smithi084 ceph-mon[40024]: osdmap e1028: 8 total, 8 up, 8 in 2024-09-20T22:58:58.948 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:58 smithi084 ceph-mon[29774]: 5.c scrub starts 2024-09-20T22:58:58.948 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:58 smithi084 ceph-mon[29774]: 5.c scrub ok 2024-09-20T22:58:58.948 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:58 smithi084 ceph-mon[29774]: 5.1e scrub starts 2024-09-20T22:58:58.948 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:58 smithi084 ceph-mon[29774]: 5.1e scrub ok 2024-09-20T22:58:58.948 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:58 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/3544218674' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-09-20T22:58:58.948 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:58 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/3544218674' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-09-20T22:58:58.949 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:58 smithi084 ceph-mon[29774]: osdmap e1028: 8 total, 8 up, 8 in 2024-09-20T22:58:58.977 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd reweight 7 1 2024-09-20T22:58:59.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:58 smithi154 ceph-mon[33264]: 5.c scrub starts 2024-09-20T22:58:59.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:58 smithi154 ceph-mon[33264]: 5.c scrub ok 2024-09-20T22:58:59.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:58 smithi154 ceph-mon[33264]: 5.1e scrub starts 2024-09-20T22:58:59.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:58 smithi154 ceph-mon[33264]: 5.1e scrub ok 2024-09-20T22:58:59.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:58 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/3544218674' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-09-20T22:58:59.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:58 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/3544218674' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-09-20T22:58:59.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:58 smithi154 ceph-mon[33264]: osdmap e1028: 8 total, 8 up, 8 in 2024-09-20T22:58:59.222 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:58:59.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:59 smithi084 ceph-mon[40024]: pgmap v1458: 142 pgs: 2 active+remapped+backfill_toofull, 140 active+clean; 583 KiB data, 648 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 1/597 objects misplaced (0.168%) 2024-09-20T22:58:59.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:59 smithi084 ceph-mon[40024]: 6.1d deep-scrub starts 2024-09-20T22:58:59.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:59 smithi084 ceph-mon[40024]: 6.1d deep-scrub ok 2024-09-20T22:58:59.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:59 smithi084 ceph-mon[40024]: 5.1 scrub starts 2024-09-20T22:58:59.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:58:59 smithi084 ceph-mon[40024]: 5.1 scrub ok 2024-09-20T22:58:59.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:59 smithi084 ceph-mon[29774]: pgmap v1458: 142 pgs: 2 active+remapped+backfill_toofull, 140 active+clean; 583 KiB data, 648 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 1/597 objects misplaced (0.168%) 2024-09-20T22:58:59.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:59 smithi084 ceph-mon[29774]: 6.1d deep-scrub starts 2024-09-20T22:58:59.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:59 smithi084 ceph-mon[29774]: 6.1d deep-scrub ok 2024-09-20T22:58:59.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:59 smithi084 ceph-mon[29774]: 5.1 scrub starts 2024-09-20T22:58:59.866 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:58:59 smithi084 ceph-mon[29774]: 5.1 scrub ok 2024-09-20T22:59:00.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:59 smithi154 ceph-mon[33264]: pgmap v1458: 142 pgs: 2 active+remapped+backfill_toofull, 140 active+clean; 583 KiB data, 648 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 1/597 objects misplaced (0.168%) 2024-09-20T22:59:00.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:59 smithi154 ceph-mon[33264]: 6.1d deep-scrub starts 2024-09-20T22:59:00.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:59 smithi154 ceph-mon[33264]: 6.1d deep-scrub ok 2024-09-20T22:59:00.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:59 smithi154 ceph-mon[33264]: 5.1 scrub starts 2024-09-20T22:59:00.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:58:59 smithi154 ceph-mon[33264]: 5.1 scrub ok 2024-09-20T22:59:00.005 INFO:journalctl@ceph.iscsi.iscsi.a.smithi154.stdout:Sep 20 22:58:59 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-iscsi-iscsi-a[59730]: debug there is no tcmu-runner data available 2024-09-20T22:59:00.642 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.7 to 1 (10000) 2024-09-20T22:59:00.971 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:00 smithi084 ceph-mon[40024]: 4.3 scrub starts 2024-09-20T22:59:00.972 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:00 smithi084 ceph-mon[40024]: 4.3 scrub ok 2024-09-20T22:59:00.972 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:00 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/1752194454' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-09-20T22:59:00.972 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:00 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-09-20T22:59:00.972 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:00 smithi084 ceph-mon[40024]: 3.14 scrub starts 2024-09-20T22:59:00.972 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:00 smithi084 ceph-mon[40024]: 3.14 scrub ok 2024-09-20T22:59:00.972 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:00 smithi084 ceph-mon[29774]: 4.3 scrub starts 2024-09-20T22:59:00.972 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:00 smithi084 ceph-mon[29774]: 4.3 scrub ok 2024-09-20T22:59:00.972 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:00 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/1752194454' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-09-20T22:59:00.972 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:00 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-09-20T22:59:00.972 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:00 smithi084 ceph-mon[29774]: 3.14 scrub starts 2024-09-20T22:59:00.972 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:00 smithi084 ceph-mon[29774]: 3.14 scrub ok 2024-09-20T22:59:01.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:00 smithi154 ceph-mon[33264]: 4.3 scrub starts 2024-09-20T22:59:01.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:00 smithi154 ceph-mon[33264]: 4.3 scrub ok 2024-09-20T22:59:01.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:00 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/1752194454' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-09-20T22:59:01.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:00 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-09-20T22:59:01.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:00 smithi154 ceph-mon[33264]: 3.14 scrub starts 2024-09-20T22:59:01.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:00 smithi154 ceph-mon[33264]: 3.14 scrub ok 2024-09-20T22:59:01.366 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd primary-affinity 7 1 2024-09-20T22:59:01.620 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:59:01.902 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:01 smithi084 ceph-mon[40024]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:59:01.902 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:01 smithi084 ceph-mon[40024]: pgmap v1460: 142 pgs: 2 active+remapped+backfill_toofull, 140 active+clean; 583 KiB data, 648 MiB used, 715 GiB / 715 GiB avail; 925 B/s rd, 0 op/s; 1/597 objects misplaced (0.168%); 112 B/s, 2 objects/s recovering 2024-09-20T22:59:01.902 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:01 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-20T22:59:01.902 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:01 smithi084 ceph-mon[40024]: osdmap e1029: 8 total, 8 up, 8 in 2024-09-20T22:59:01.903 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:01 smithi084 ceph-mon[40024]: 4.6 scrub starts 2024-09-20T22:59:01.903 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:01 smithi084 ceph-mon[40024]: 4.6 scrub ok 2024-09-20T22:59:01.903 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:01 smithi084 ceph-mon[29774]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:59:01.903 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:01 smithi084 ceph-mon[29774]: pgmap v1460: 142 pgs: 2 active+remapped+backfill_toofull, 140 active+clean; 583 KiB data, 648 MiB used, 715 GiB / 715 GiB avail; 925 B/s rd, 0 op/s; 1/597 objects misplaced (0.168%); 112 B/s, 2 objects/s recovering 2024-09-20T22:59:01.903 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:01 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-20T22:59:01.903 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:01 smithi084 ceph-mon[29774]: osdmap e1029: 8 total, 8 up, 8 in 2024-09-20T22:59:01.903 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:01 smithi084 ceph-mon[29774]: 4.6 scrub starts 2024-09-20T22:59:01.903 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:01 smithi084 ceph-mon[29774]: 4.6 scrub ok 2024-09-20T22:59:02.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:01 smithi154 ceph-mon[33264]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:59:02.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:01 smithi154 ceph-mon[33264]: pgmap v1460: 142 pgs: 2 active+remapped+backfill_toofull, 140 active+clean; 583 KiB data, 648 MiB used, 715 GiB / 715 GiB avail; 925 B/s rd, 0 op/s; 1/597 objects misplaced (0.168%); 112 B/s, 2 objects/s recovering 2024-09-20T22:59:02.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:01 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-20T22:59:02.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:01 smithi154 ceph-mon[33264]: osdmap e1029: 8 total, 8 up, 8 in 2024-09-20T22:59:02.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:01 smithi154 ceph-mon[33264]: 4.6 scrub starts 2024-09-20T22:59:02.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:01 smithi154 ceph-mon[33264]: 4.6 scrub ok 2024-09-20T22:59:02.658 INFO:teuthology.orchestra.run.smithi084.stderr:set osd.7 primary-affinity to 1 (8655362) 2024-09-20T22:59:02.962 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:02 smithi084 ceph-mon[40024]: 3.8 scrub starts 2024-09-20T22:59:02.963 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:02 smithi084 ceph-mon[40024]: 3.8 scrub ok 2024-09-20T22:59:02.963 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:02 smithi084 ceph-mon[40024]: 5.1b scrub starts 2024-09-20T22:59:02.963 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:02 smithi084 ceph-mon[40024]: 5.1b scrub ok 2024-09-20T22:59:02.963 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:02 smithi084 ceph-mon[40024]: osdmap e1030: 8 total, 8 up, 8 in 2024-09-20T22:59:02.963 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:02 smithi084 ceph-mon[40024]: 4.1 deep-scrub starts 2024-09-20T22:59:02.963 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:02 smithi084 ceph-mon[40024]: 4.1 deep-scrub ok 2024-09-20T22:59:02.963 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:02 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/1956531682' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-09-20T22:59:02.964 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:02 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-09-20T22:59:02.964 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:02 smithi084 ceph-mon[40024]: 1.2 scrub starts 2024-09-20T22:59:02.964 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:02 smithi084 ceph-mon[40024]: 1.2 scrub ok 2024-09-20T22:59:02.964 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:02 smithi084 ceph-mon[29774]: 3.8 scrub starts 2024-09-20T22:59:02.964 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:02 smithi084 ceph-mon[29774]: 3.8 scrub ok 2024-09-20T22:59:02.964 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:02 smithi084 ceph-mon[29774]: 5.1b scrub starts 2024-09-20T22:59:02.964 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:02 smithi084 ceph-mon[29774]: 5.1b scrub ok 2024-09-20T22:59:02.964 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:02 smithi084 ceph-mon[29774]: osdmap e1030: 8 total, 8 up, 8 in 2024-09-20T22:59:02.965 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:02 smithi084 ceph-mon[29774]: 4.1 deep-scrub starts 2024-09-20T22:59:02.965 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:02 smithi084 ceph-mon[29774]: 4.1 deep-scrub ok 2024-09-20T22:59:02.965 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:02 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/1956531682' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-09-20T22:59:02.965 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:02 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-09-20T22:59:02.965 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:02 smithi084 ceph-mon[29774]: 1.2 scrub starts 2024-09-20T22:59:02.965 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:02 smithi084 ceph-mon[29774]: 1.2 scrub ok 2024-09-20T22:59:03.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:02 smithi154 ceph-mon[33264]: 3.8 scrub starts 2024-09-20T22:59:03.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:02 smithi154 ceph-mon[33264]: 3.8 scrub ok 2024-09-20T22:59:03.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:02 smithi154 ceph-mon[33264]: 5.1b scrub starts 2024-09-20T22:59:03.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:02 smithi154 ceph-mon[33264]: 5.1b scrub ok 2024-09-20T22:59:03.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:02 smithi154 ceph-mon[33264]: osdmap e1030: 8 total, 8 up, 8 in 2024-09-20T22:59:03.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:02 smithi154 ceph-mon[33264]: 4.1 deep-scrub starts 2024-09-20T22:59:03.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:02 smithi154 ceph-mon[33264]: 4.1 deep-scrub ok 2024-09-20T22:59:03.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:02 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/1956531682' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-09-20T22:59:03.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:02 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-09-20T22:59:03.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:02 smithi154 ceph-mon[33264]: 1.2 scrub starts 2024-09-20T22:59:03.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:02 smithi154 ceph-mon[33264]: 1.2 scrub ok 2024-09-20T22:59:03.386 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd reweight 3 1 2024-09-20T22:59:03.641 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:59:03.919 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:03 smithi084 ceph-mon[29774]: pgmap v1463: 142 pgs: 2 active+remapped+backfill_toofull, 140 active+clean; 583 KiB data, 648 MiB used, 715 GiB / 715 GiB avail; 966 B/s rd, 0 op/s; 1/597 objects misplaced (0.168%); 117 B/s, 3 objects/s recovering 2024-09-20T22:59:03.920 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:03 smithi084 ceph-mon[29774]: 6.d deep-scrub starts 2024-09-20T22:59:03.920 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:03 smithi084 ceph-mon[29774]: 6.d deep-scrub ok 2024-09-20T22:59:03.920 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:03 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-09-20T22:59:03.920 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:03 smithi084 ceph-mon[29774]: osdmap e1031: 8 total, 8 up, 8 in 2024-09-20T22:59:03.920 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:03 smithi084 ceph-mon[29774]: 1.0 scrub starts 2024-09-20T22:59:03.920 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:03 smithi084 ceph-mon[29774]: 1.0 scrub ok 2024-09-20T22:59:03.920 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:03 smithi084 ceph-mon[29774]: osdmap e1032: 8 total, 8 up, 8 in 2024-09-20T22:59:03.920 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:03 smithi084 ceph-mon[40024]: pgmap v1463: 142 pgs: 2 active+remapped+backfill_toofull, 140 active+clean; 583 KiB data, 648 MiB used, 715 GiB / 715 GiB avail; 966 B/s rd, 0 op/s; 1/597 objects misplaced (0.168%); 117 B/s, 3 objects/s recovering 2024-09-20T22:59:03.920 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:03 smithi084 ceph-mon[40024]: 6.d deep-scrub starts 2024-09-20T22:59:03.920 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:03 smithi084 ceph-mon[40024]: 6.d deep-scrub ok 2024-09-20T22:59:03.920 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:03 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-09-20T22:59:03.921 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:03 smithi084 ceph-mon[40024]: osdmap e1031: 8 total, 8 up, 8 in 2024-09-20T22:59:03.921 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:03 smithi084 ceph-mon[40024]: 1.0 scrub starts 2024-09-20T22:59:03.921 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:03 smithi084 ceph-mon[40024]: 1.0 scrub ok 2024-09-20T22:59:03.921 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:03 smithi084 ceph-mon[40024]: osdmap e1032: 8 total, 8 up, 8 in 2024-09-20T22:59:04.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:03 smithi154 ceph-mon[33264]: pgmap v1463: 142 pgs: 2 active+remapped+backfill_toofull, 140 active+clean; 583 KiB data, 648 MiB used, 715 GiB / 715 GiB avail; 966 B/s rd, 0 op/s; 1/597 objects misplaced (0.168%); 117 B/s, 3 objects/s recovering 2024-09-20T22:59:04.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:03 smithi154 ceph-mon[33264]: 6.d deep-scrub starts 2024-09-20T22:59:04.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:03 smithi154 ceph-mon[33264]: 6.d deep-scrub ok 2024-09-20T22:59:04.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:03 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-09-20T22:59:04.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:03 smithi154 ceph-mon[33264]: osdmap e1031: 8 total, 8 up, 8 in 2024-09-20T22:59:04.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:03 smithi154 ceph-mon[33264]: 1.0 scrub starts 2024-09-20T22:59:04.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:03 smithi154 ceph-mon[33264]: 1.0 scrub ok 2024-09-20T22:59:04.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:03 smithi154 ceph-mon[33264]: osdmap e1032: 8 total, 8 up, 8 in 2024-09-20T22:59:04.668 INFO:teuthology.orchestra.run.smithi084.stderr:reweighted osd.3 to 1 (10000) 2024-09-20T22:59:05.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:04 smithi154 ceph-mon[33264]: 4.14 scrub starts 2024-09-20T22:59:05.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:04 smithi154 ceph-mon[33264]: 4.14 scrub ok 2024-09-20T22:59:05.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:04 smithi154 ceph-mon[33264]: 4.8 scrub starts 2024-09-20T22:59:05.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:04 smithi154 ceph-mon[33264]: 4.8 scrub ok 2024-09-20T22:59:05.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:04 smithi154 ceph-mon[33264]: 3.4 scrub starts 2024-09-20T22:59:05.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:04 smithi154 ceph-mon[33264]: 3.4 scrub ok 2024-09-20T22:59:05.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:04 smithi154 ceph-mon[33264]: 5.18 scrub starts 2024-09-20T22:59:05.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:04 smithi154 ceph-mon[33264]: 5.18 scrub ok 2024-09-20T22:59:05.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:04 smithi154 ceph-mon[33264]: 5.9 scrub starts 2024-09-20T22:59:05.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:04 smithi154 ceph-mon[33264]: 5.9 scrub ok 2024-09-20T22:59:05.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:04 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/4085282368' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-09-20T22:59:05.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:04 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-09-20T22:59:05.054 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:04 smithi084 ceph-mon[40024]: 4.14 scrub starts 2024-09-20T22:59:05.054 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:04 smithi084 ceph-mon[40024]: 4.14 scrub ok 2024-09-20T22:59:05.054 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:04 smithi084 ceph-mon[40024]: 4.8 scrub starts 2024-09-20T22:59:05.054 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:04 smithi084 ceph-mon[40024]: 4.8 scrub ok 2024-09-20T22:59:05.054 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:04 smithi084 ceph-mon[40024]: 3.4 scrub starts 2024-09-20T22:59:05.055 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:04 smithi084 ceph-mon[40024]: 3.4 scrub ok 2024-09-20T22:59:05.055 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:04 smithi084 ceph-mon[40024]: 5.18 scrub starts 2024-09-20T22:59:05.055 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:04 smithi084 ceph-mon[40024]: 5.18 scrub ok 2024-09-20T22:59:05.055 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:04 smithi084 ceph-mon[40024]: 5.9 scrub starts 2024-09-20T22:59:05.055 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:04 smithi084 ceph-mon[40024]: 5.9 scrub ok 2024-09-20T22:59:05.055 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:04 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/4085282368' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-09-20T22:59:05.055 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:04 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-09-20T22:59:05.055 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:04 smithi084 ceph-mon[29774]: 4.14 scrub starts 2024-09-20T22:59:05.055 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:04 smithi084 ceph-mon[29774]: 4.14 scrub ok 2024-09-20T22:59:05.055 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:04 smithi084 ceph-mon[29774]: 4.8 scrub starts 2024-09-20T22:59:05.055 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:04 smithi084 ceph-mon[29774]: 4.8 scrub ok 2024-09-20T22:59:05.055 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:04 smithi084 ceph-mon[29774]: 3.4 scrub starts 2024-09-20T22:59:05.056 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:04 smithi084 ceph-mon[29774]: 3.4 scrub ok 2024-09-20T22:59:05.056 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:04 smithi084 ceph-mon[29774]: 5.18 scrub starts 2024-09-20T22:59:05.056 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:04 smithi084 ceph-mon[29774]: 5.18 scrub ok 2024-09-20T22:59:05.056 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:04 smithi084 ceph-mon[29774]: 5.9 scrub starts 2024-09-20T22:59:05.056 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:04 smithi084 ceph-mon[29774]: 5.9 scrub ok 2024-09-20T22:59:05.056 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:04 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/4085282368' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-09-20T22:59:05.056 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:04 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-09-20T22:59:05.517 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd primary-affinity 3 1 2024-09-20T22:59:05.769 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:59:05.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:05 smithi084 ceph-mon[40024]: pgmap v1466: 142 pgs: 142 active+clean; 583 KiB data, 648 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 80 KiB/s, 0 objects/s recovering 2024-09-20T22:59:05.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:05 smithi084 ceph-mon[40024]: 2.2 scrub starts 2024-09-20T22:59:05.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:05 smithi084 ceph-mon[40024]: 2.2 scrub ok 2024-09-20T22:59:05.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:05 smithi084 ceph-mon[40024]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull) 2024-09-20T22:59:05.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:05 smithi084 ceph-mon[40024]: Cluster is now healthy 2024-09-20T22:59:05.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:05 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-20T22:59:05.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:05 smithi084 ceph-mon[40024]: osdmap e1033: 8 total, 8 up, 8 in 2024-09-20T22:59:05.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:05 smithi084 ceph-mon[40024]: 3.1b scrub starts 2024-09-20T22:59:05.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:05 smithi084 ceph-mon[40024]: 3.1b scrub ok 2024-09-20T22:59:05.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:05 smithi084 ceph-mon[40024]: 3.1a deep-scrub starts 2024-09-20T22:59:05.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:05 smithi084 ceph-mon[40024]: 3.1a deep-scrub ok 2024-09-20T22:59:05.866 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:05 smithi084 ceph-mon[29774]: pgmap v1466: 142 pgs: 142 active+clean; 583 KiB data, 648 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 80 KiB/s, 0 objects/s recovering 2024-09-20T22:59:05.866 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:05 smithi084 ceph-mon[29774]: 2.2 scrub starts 2024-09-20T22:59:05.866 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:05 smithi084 ceph-mon[29774]: 2.2 scrub ok 2024-09-20T22:59:05.866 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:05 smithi084 ceph-mon[29774]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull) 2024-09-20T22:59:05.866 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:05 smithi084 ceph-mon[29774]: Cluster is now healthy 2024-09-20T22:59:05.866 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:05 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-20T22:59:05.866 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:05 smithi084 ceph-mon[29774]: osdmap e1033: 8 total, 8 up, 8 in 2024-09-20T22:59:05.866 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:05 smithi084 ceph-mon[29774]: 3.1b scrub starts 2024-09-20T22:59:05.866 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:05 smithi084 ceph-mon[29774]: 3.1b scrub ok 2024-09-20T22:59:05.866 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:05 smithi084 ceph-mon[29774]: 3.1a deep-scrub starts 2024-09-20T22:59:05.866 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:05 smithi084 ceph-mon[29774]: 3.1a deep-scrub ok 2024-09-20T22:59:06.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:05 smithi154 ceph-mon[33264]: pgmap v1466: 142 pgs: 142 active+clean; 583 KiB data, 648 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 80 KiB/s, 0 objects/s recovering 2024-09-20T22:59:06.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:05 smithi154 ceph-mon[33264]: 2.2 scrub starts 2024-09-20T22:59:06.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:05 smithi154 ceph-mon[33264]: 2.2 scrub ok 2024-09-20T22:59:06.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:05 smithi154 ceph-mon[33264]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull) 2024-09-20T22:59:06.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:05 smithi154 ceph-mon[33264]: Cluster is now healthy 2024-09-20T22:59:06.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:05 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-20T22:59:06.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:05 smithi154 ceph-mon[33264]: osdmap e1033: 8 total, 8 up, 8 in 2024-09-20T22:59:06.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:05 smithi154 ceph-mon[33264]: 3.1b scrub starts 2024-09-20T22:59:06.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:05 smithi154 ceph-mon[33264]: 3.1b scrub ok 2024-09-20T22:59:06.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:05 smithi154 ceph-mon[33264]: 3.1a deep-scrub starts 2024-09-20T22:59:06.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:05 smithi154 ceph-mon[33264]: 3.1a deep-scrub ok 2024-09-20T22:59:06.682 INFO:teuthology.orchestra.run.smithi084.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-09-20T22:59:07.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:06 smithi154 ceph-mon[33264]: 3.2 scrub starts 2024-09-20T22:59:07.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:06 smithi154 ceph-mon[33264]: 3.2 scrub ok 2024-09-20T22:59:07.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:06 smithi154 ceph-mon[33264]: 4.d scrub starts 2024-09-20T22:59:07.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:06 smithi154 ceph-mon[33264]: 4.d scrub ok 2024-09-20T22:59:07.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:06 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/1596739034' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-09-20T22:59:07.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:06 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-09-20T22:59:07.054 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:06 smithi084 ceph-mon[40024]: 3.2 scrub starts 2024-09-20T22:59:07.054 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:06 smithi084 ceph-mon[40024]: 3.2 scrub ok 2024-09-20T22:59:07.054 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:06 smithi084 ceph-mon[40024]: 4.d scrub starts 2024-09-20T22:59:07.054 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:06 smithi084 ceph-mon[40024]: 4.d scrub ok 2024-09-20T22:59:07.055 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:06 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/1596739034' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-09-20T22:59:07.055 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:06 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-09-20T22:59:07.055 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:06 smithi084 ceph-mon[29774]: 3.2 scrub starts 2024-09-20T22:59:07.055 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:06 smithi084 ceph-mon[29774]: 3.2 scrub ok 2024-09-20T22:59:07.055 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:06 smithi084 ceph-mon[29774]: 4.d scrub starts 2024-09-20T22:59:07.055 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:06 smithi084 ceph-mon[29774]: 4.d scrub ok 2024-09-20T22:59:07.055 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:06 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/1596739034' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-09-20T22:59:07.055 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:06 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-09-20T22:59:07.504 INFO:tasks.thrashosds.thrasher:joining the do_sighup greenlet 2024-09-20T22:59:07.504 INFO:tasks.thrashosds.thrasher:joining the do_optrack_toggle greenlet 2024-09-20T22:59:07.504 INFO:tasks.thrashosds.thrasher:joining the do_dump_ops greenlet 2024-09-20T22:59:07.504 INFO:tasks.thrashosds.thrasher:joining the do_noscrub_toggle greenlet 2024-09-20T22:59:07.504 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-09-20T22:59:07.505 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd dump --format=json 2024-09-20T22:59:07.757 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:59:07.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:07 smithi084 ceph-mon[40024]: pgmap v1468: 142 pgs: 142 active+clean; 583 KiB data, 648 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 75 KiB/s, 0 objects/s recovering 2024-09-20T22:59:07.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:07 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-09-20T22:59:07.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:07 smithi084 ceph-mon[40024]: osdmap e1034: 8 total, 8 up, 8 in 2024-09-20T22:59:07.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:07 smithi084 ceph-mon[29774]: pgmap v1468: 142 pgs: 142 active+clean; 583 KiB data, 648 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 75 KiB/s, 0 objects/s recovering 2024-09-20T22:59:07.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:07 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-09-20T22:59:07.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:07 smithi084 ceph-mon[29774]: osdmap e1034: 8 total, 8 up, 8 in 2024-09-20T22:59:08.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:07 smithi154 ceph-mon[33264]: pgmap v1468: 142 pgs: 142 active+clean; 583 KiB data, 648 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 75 KiB/s, 0 objects/s recovering 2024-09-20T22:59:08.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:07 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-09-20T22:59:08.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:07 smithi154 ceph-mon[33264]: osdmap e1034: 8 total, 8 up, 8 in 2024-09-20T22:59:08.364 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-20T22:59:08.364 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":1034,"fsid":"a9c0bc70-77a0-11ef-baf5-efdc52797490","created":"2024-09-20T22:36:30.240626+0000","modified":"2024-09-20T22:59:06.671878+0000","last_up_change":"2024-09-20T22:57:48.362816+0000","last_in_change":"2024-09-20T22:58:27.877911+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":19,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-20T22:39:21.984381+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":11,"pg_placement_num":11,"pg_placement_num_target":11,"pg_num_target":11,"pg_num_pending":11,"last_pg_merge_meta":{"source_pgid":"1.b","ready_epoch":299,"last_epoch_started":298,"last_epoch_clean":298,"source_version":"77'128","target_version":"77'128"},"last_change":"769","last_force_op_resend":"0","last_force_op_resend_prenautilus":"299","last_force_op_resend_preluminous":"297","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.1800000667572021,"score_stable":2.1800000667572021,"optimal_score":1,"raw_score_acting":2.1800000667572021,"raw_score_stable":2.1800000667572021,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"datapool","create_time":"2024-09-20T22:41:35.820081+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":69,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":".rgw.root","create_time":"2024-09-20T22:41:36.546389+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-09-20T22:41:38.454640+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-09-20T22:41:40.407129+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"72","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-09-20T22:41:42.476716+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"74","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ea36a489-6f91-4b2f-b784-b90506b9805d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":1028,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6801","nonce":268084626}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6802","nonce":268084626}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6804","nonce":268084626}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6803","nonce":268084626}]},"public_addr":"172.21.15.84:6801/268084626","cluster_addr":"172.21.15.84:6802/268084626","heartbeat_back_addr":"172.21.15.84:6804/268084626","heartbeat_front_addr":"172.21.15.84:6803/268084626","state":["exists","up"]},{"osd":1,"uuid":"c8649d2f-27ef-47f8-bc94-f1cd9e62bdb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":1008,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6805","nonce":418374136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6806","nonce":418374136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6808","nonce":418374136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6807","nonce":418374136}]},"public_addr":"172.21.15.84:6805/418374136","cluster_addr":"172.21.15.84:6806/418374136","heartbeat_back_addr":"172.21.15.84:6808/418374136","heartbeat_front_addr":"172.21.15.84:6807/418374136","state":["exists","up"]},{"osd":2,"uuid":"c728eb48-a0f0-4de4-a651-7f3bec867280","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":1010,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6809","nonce":3977168611}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6810","nonce":3977168611}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6812","nonce":3977168611}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6811","nonce":3977168611}]},"public_addr":"172.21.15.84:6809/3977168611","cluster_addr":"172.21.15.84:6810/3977168611","heartbeat_back_addr":"172.21.15.84:6812/3977168611","heartbeat_front_addr":"172.21.15.84:6811/3977168611","state":["exists","up"]},{"osd":3,"uuid":"573f44eb-0853-4630-a7d8-9caaeb256f9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":25,"last_clean_end":939,"up_from":988,"up_thru":1029,"down_at":940,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6813","nonce":3784535689}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6814","nonce":3784535689}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6816","nonce":3784535689}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6815","nonce":3784535689}]},"public_addr":"172.21.15.84:6813/3784535689","cluster_addr":"172.21.15.84:6814/3784535689","heartbeat_back_addr":"172.21.15.84:6816/3784535689","heartbeat_front_addr":"172.21.15.84:6815/3784535689","state":["exists","up"]},{"osd":4,"uuid":"a52c3b94-c5f3-490f-8ac8-09061b9b2abb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":1022,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":1793136002}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6801","nonce":1793136002}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6803","nonce":1793136002}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":1793136002}]},"public_addr":"172.21.15.154:6800/1793136002","cluster_addr":"172.21.15.154:6801/1793136002","heartbeat_back_addr":"172.21.15.154:6803/1793136002","heartbeat_front_addr":"172.21.15.154:6802/1793136002","state":["exists","up"]},{"osd":5,"uuid":"4fa0dc8e-25cc-46cd-96ff-33f20b6faec3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":1025,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":493899927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6805","nonce":493899927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6807","nonce":493899927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":493899927}]},"public_addr":"172.21.15.154:6804/493899927","cluster_addr":"172.21.15.154:6805/493899927","heartbeat_back_addr":"172.21.15.154:6807/493899927","heartbeat_front_addr":"172.21.15.154:6806/493899927","state":["exists","up"]},{"osd":6,"uuid":"56788d74-810e-4827-a615-958c12f8a368","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":1011,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":2375091033}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6809","nonce":2375091033}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6811","nonce":2375091033}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":2375091033}]},"public_addr":"172.21.15.154:6808/2375091033","cluster_addr":"172.21.15.154:6809/2375091033","heartbeat_back_addr":"172.21.15.154:6811/2375091033","heartbeat_front_addr":"172.21.15.154:6810/2375091033","state":["exists","up"]},{"osd":7,"uuid":"83503727-afc1-4b95-8b96-ecb63f3338c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":1029,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":3717704491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6813","nonce":3717704491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6815","nonce":3717704491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":3717704491}]},"public_addr":"172.21.15.154:6812/3717704491","cluster_addr":"172.21.15.154:6813/3717704491","heartbeat_back_addr":"172.21.15.154:6815/3717704491","heartbeat_front_addr":"172.21.15.154:6814/3717704491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T22:57:42.287953+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T22:58:46.345225+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T22:58:29.991590+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T22:58:56.450092+0000","dead_epoch":939},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T22:58:08.973557+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T22:58:29.430576+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T22:58:54.343179+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T22:58:53.416463+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"4.3","osds":[5,2,0]}],"pg_upmap_items":[{"pgid":"3.a","mappings":[{"from":6,"to":2}]},{"pgid":"4.4","mappings":[{"from":1,"to":5}]},{"pgid":"4.6","mappings":[{"from":1,"to":5}]},{"pgid":"4.9","mappings":[{"from":3,"to":2}]},{"pgid":"4.11","mappings":[{"from":3,"to":5}]},{"pgid":"4.1b","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.84:0/542020545":"2024-09-21T22:42:07.596045+0000","172.21.15.84:0/25873236":"2024-09-21T22:36:54.917812+0000","172.21.15.84:6800/820908470":"2024-09-21T22:36:54.917812+0000","172.21.15.84:0/567614908":"2024-09-21T22:36:54.917812+0000","172.21.15.84:0/2963622459":"2024-09-21T22:42:07.596045+0000","172.21.15.84:0/1176849276":"2024-09-21T22:37:17.690104+0000","172.21.15.84:6800/2932420050":"2024-09-21T22:37:17.690104+0000","172.21.15.84:0/3879263635":"2024-09-21T22:37:17.690104+0000","172.21.15.84:0/969472279":"2024-09-21T22:36:54.917812+0000","172.21.15.84:0/1361143214":"2024-09-21T22:37:17.690104+0000","172.21.15.84:0/3353126955":"2024-09-21T22:42:07.596045+0000","172.21.15.84:6800/4024278029":"2024-09-21T22:42:07.596045+0000","172.21.15.84:0/547902262":"2024-09-21T22:42:07.596045+0000","172.21.15.84:0/1242929317":"2024-09-21T22:42:07.596045+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-20T22:59:08.615 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:59:08 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: ::ffff:172.21.15.154 - - [20/Sep/2024:22:59:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-20T22:59:09.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:08 smithi154 ceph-mon[33264]: 4.e scrub starts 2024-09-20T22:59:09.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:08 smithi154 ceph-mon[33264]: 4.e scrub ok 2024-09-20T22:59:09.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:08 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/1466884962' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-20T22:59:09.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:08 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:59:09.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:08 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:59:09.071 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:08 smithi084 ceph-mon[29774]: 4.e scrub starts 2024-09-20T22:59:09.071 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:08 smithi084 ceph-mon[29774]: 4.e scrub ok 2024-09-20T22:59:09.071 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:08 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/1466884962' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-20T22:59:09.071 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:08 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:59:09.071 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:08 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:59:09.071 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:08 smithi084 ceph-mon[40024]: 4.e scrub starts 2024-09-20T22:59:09.071 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:08 smithi084 ceph-mon[40024]: 4.e scrub ok 2024-09-20T22:59:09.072 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:08 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/1466884962' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-20T22:59:09.072 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:08 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:59:09.072 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:08 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:59:09.100 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-09-20T22:59:09.101 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd dump --format=json 2024-09-20T22:59:09.345 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:59:09.808 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:09 smithi084 ceph-mon[40024]: pgmap v1470: 142 pgs: 142 active+clean; 583 KiB data, 648 MiB used, 715 GiB / 715 GiB avail; 972 B/s rd, 0 op/s; 61 KiB/s, 0 objects/s recovering 2024-09-20T22:59:09.808 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:09 smithi084 ceph-mon[40024]: 6.a scrub starts 2024-09-20T22:59:09.808 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:09 smithi084 ceph-mon[40024]: 6.a scrub ok 2024-09-20T22:59:09.808 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:09 smithi084 ceph-mon[40024]: 1.4 scrub starts 2024-09-20T22:59:09.808 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:09 smithi084 ceph-mon[40024]: 1.4 scrub ok 2024-09-20T22:59:09.808 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:09 smithi084 ceph-mon[40024]: 4.2 scrub starts 2024-09-20T22:59:09.808 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:09 smithi084 ceph-mon[40024]: 4.2 scrub ok 2024-09-20T22:59:09.808 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:09 smithi084 ceph-mon[40024]: 6.10 scrub starts 2024-09-20T22:59:09.808 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:09 smithi084 ceph-mon[40024]: 6.10 scrub ok 2024-09-20T22:59:09.808 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:09 smithi084 ceph-mon[40024]: 3.3 scrub starts 2024-09-20T22:59:09.809 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:09 smithi084 ceph-mon[40024]: 3.3 scrub ok 2024-09-20T22:59:09.809 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:09 smithi084 ceph-mon[29774]: pgmap v1470: 142 pgs: 142 active+clean; 583 KiB data, 648 MiB used, 715 GiB / 715 GiB avail; 972 B/s rd, 0 op/s; 61 KiB/s, 0 objects/s recovering 2024-09-20T22:59:09.809 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:09 smithi084 ceph-mon[29774]: 6.a scrub starts 2024-09-20T22:59:09.809 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:09 smithi084 ceph-mon[29774]: 6.a scrub ok 2024-09-20T22:59:09.809 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:09 smithi084 ceph-mon[29774]: 1.4 scrub starts 2024-09-20T22:59:09.809 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:09 smithi084 ceph-mon[29774]: 1.4 scrub ok 2024-09-20T22:59:09.809 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:09 smithi084 ceph-mon[29774]: 4.2 scrub starts 2024-09-20T22:59:09.809 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:09 smithi084 ceph-mon[29774]: 4.2 scrub ok 2024-09-20T22:59:09.809 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:09 smithi084 ceph-mon[29774]: 6.10 scrub starts 2024-09-20T22:59:09.809 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:09 smithi084 ceph-mon[29774]: 6.10 scrub ok 2024-09-20T22:59:09.810 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:09 smithi084 ceph-mon[29774]: 3.3 scrub starts 2024-09-20T22:59:09.810 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:09 smithi084 ceph-mon[29774]: 3.3 scrub ok 2024-09-20T22:59:10.004 INFO:journalctl@ceph.iscsi.iscsi.a.smithi154.stdout:Sep 20 22:59:09 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-iscsi-iscsi-a[59730]: debug there is no tcmu-runner data available 2024-09-20T22:59:10.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:09 smithi154 ceph-mon[33264]: pgmap v1470: 142 pgs: 142 active+clean; 583 KiB data, 648 MiB used, 715 GiB / 715 GiB avail; 972 B/s rd, 0 op/s; 61 KiB/s, 0 objects/s recovering 2024-09-20T22:59:10.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:09 smithi154 ceph-mon[33264]: 6.a scrub starts 2024-09-20T22:59:10.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:09 smithi154 ceph-mon[33264]: 6.a scrub ok 2024-09-20T22:59:10.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:09 smithi154 ceph-mon[33264]: 1.4 scrub starts 2024-09-20T22:59:10.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:09 smithi154 ceph-mon[33264]: 1.4 scrub ok 2024-09-20T22:59:10.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:09 smithi154 ceph-mon[33264]: 4.2 scrub starts 2024-09-20T22:59:10.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:09 smithi154 ceph-mon[33264]: 4.2 scrub ok 2024-09-20T22:59:10.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:09 smithi154 ceph-mon[33264]: 6.10 scrub starts 2024-09-20T22:59:10.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:09 smithi154 ceph-mon[33264]: 6.10 scrub ok 2024-09-20T22:59:10.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:09 smithi154 ceph-mon[33264]: 3.3 scrub starts 2024-09-20T22:59:10.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:09 smithi154 ceph-mon[33264]: 3.3 scrub ok 2024-09-20T22:59:10.067 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-20T22:59:10.067 INFO:teuthology.orchestra.run.smithi084.stdout:{"epoch":1034,"fsid":"a9c0bc70-77a0-11ef-baf5-efdc52797490","created":"2024-09-20T22:36:30.240626+0000","modified":"2024-09-20T22:59:06.671878+0000","last_up_change":"2024-09-20T22:57:48.362816+0000","last_in_change":"2024-09-20T22:58:27.877911+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":19,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-20T22:39:21.984381+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":11,"pg_placement_num":11,"pg_placement_num_target":11,"pg_num_target":11,"pg_num_pending":11,"last_pg_merge_meta":{"source_pgid":"1.b","ready_epoch":299,"last_epoch_started":298,"last_epoch_clean":298,"source_version":"77'128","target_version":"77'128"},"last_change":"769","last_force_op_resend":"0","last_force_op_resend_prenautilus":"299","last_force_op_resend_preluminous":"297","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.1800000667572021,"score_stable":2.1800000667572021,"optimal_score":1,"raw_score_acting":2.1800000667572021,"raw_score_stable":2.1800000667572021,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"datapool","create_time":"2024-09-20T22:41:35.820081+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":69,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":".rgw.root","create_time":"2024-09-20T22:41:36.546389+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-09-20T22:41:38.454640+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-09-20T22:41:40.407129+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"72","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-09-20T22:41:42.476716+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"74","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"ea36a489-6f91-4b2f-b784-b90506b9805d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":1028,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6801","nonce":268084626}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6802","nonce":268084626}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6804","nonce":268084626}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6803","nonce":268084626}]},"public_addr":"172.21.15.84:6801/268084626","cluster_addr":"172.21.15.84:6802/268084626","heartbeat_back_addr":"172.21.15.84:6804/268084626","heartbeat_front_addr":"172.21.15.84:6803/268084626","state":["exists","up"]},{"osd":1,"uuid":"c8649d2f-27ef-47f8-bc94-f1cd9e62bdb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":1008,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6805","nonce":418374136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6806","nonce":418374136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6808","nonce":418374136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6807","nonce":418374136}]},"public_addr":"172.21.15.84:6805/418374136","cluster_addr":"172.21.15.84:6806/418374136","heartbeat_back_addr":"172.21.15.84:6808/418374136","heartbeat_front_addr":"172.21.15.84:6807/418374136","state":["exists","up"]},{"osd":2,"uuid":"c728eb48-a0f0-4de4-a651-7f3bec867280","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":1010,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6809","nonce":3977168611}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6810","nonce":3977168611}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6812","nonce":3977168611}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6811","nonce":3977168611}]},"public_addr":"172.21.15.84:6809/3977168611","cluster_addr":"172.21.15.84:6810/3977168611","heartbeat_back_addr":"172.21.15.84:6812/3977168611","heartbeat_front_addr":"172.21.15.84:6811/3977168611","state":["exists","up"]},{"osd":3,"uuid":"573f44eb-0853-4630-a7d8-9caaeb256f9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":25,"last_clean_end":939,"up_from":988,"up_thru":1029,"down_at":940,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6813","nonce":3784535689}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6814","nonce":3784535689}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6816","nonce":3784535689}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.84:6815","nonce":3784535689}]},"public_addr":"172.21.15.84:6813/3784535689","cluster_addr":"172.21.15.84:6814/3784535689","heartbeat_back_addr":"172.21.15.84:6816/3784535689","heartbeat_front_addr":"172.21.15.84:6815/3784535689","state":["exists","up"]},{"osd":4,"uuid":"a52c3b94-c5f3-490f-8ac8-09061b9b2abb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":1022,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6800","nonce":1793136002}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6801","nonce":1793136002}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6803","nonce":1793136002}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6802","nonce":1793136002}]},"public_addr":"172.21.15.154:6800/1793136002","cluster_addr":"172.21.15.154:6801/1793136002","heartbeat_back_addr":"172.21.15.154:6803/1793136002","heartbeat_front_addr":"172.21.15.154:6802/1793136002","state":["exists","up"]},{"osd":5,"uuid":"4fa0dc8e-25cc-46cd-96ff-33f20b6faec3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":1025,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6804","nonce":493899927}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6805","nonce":493899927}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6807","nonce":493899927}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6806","nonce":493899927}]},"public_addr":"172.21.15.154:6804/493899927","cluster_addr":"172.21.15.154:6805/493899927","heartbeat_back_addr":"172.21.15.154:6807/493899927","heartbeat_front_addr":"172.21.15.154:6806/493899927","state":["exists","up"]},{"osd":6,"uuid":"56788d74-810e-4827-a615-958c12f8a368","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":1011,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6808","nonce":2375091033}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6809","nonce":2375091033}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6811","nonce":2375091033}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6810","nonce":2375091033}]},"public_addr":"172.21.15.154:6808/2375091033","cluster_addr":"172.21.15.154:6809/2375091033","heartbeat_back_addr":"172.21.15.154:6811/2375091033","heartbeat_front_addr":"172.21.15.154:6810/2375091033","state":["exists","up"]},{"osd":7,"uuid":"83503727-afc1-4b95-8b96-ecb63f3338c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":1029,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6812","nonce":3717704491}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6813","nonce":3717704491}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6815","nonce":3717704491}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.154:6814","nonce":3717704491}]},"public_addr":"172.21.15.154:6812/3717704491","cluster_addr":"172.21.15.154:6813/3717704491","heartbeat_back_addr":"172.21.15.154:6815/3717704491","heartbeat_front_addr":"172.21.15.154:6814/3717704491","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T22:57:42.287953+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T22:58:46.345225+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T22:58:29.991590+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T22:58:56.450092+0000","dead_epoch":939},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T22:58:08.973557+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T22:58:29.430576+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T22:58:54.343179+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T22:58:53.416463+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"4.3","osds":[5,2,0]}],"pg_upmap_items":[{"pgid":"3.a","mappings":[{"from":6,"to":2}]},{"pgid":"4.4","mappings":[{"from":1,"to":5}]},{"pgid":"4.6","mappings":[{"from":1,"to":5}]},{"pgid":"4.9","mappings":[{"from":3,"to":2}]},{"pgid":"4.11","mappings":[{"from":3,"to":5}]},{"pgid":"4.1b","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.84:0/542020545":"2024-09-21T22:42:07.596045+0000","172.21.15.84:0/25873236":"2024-09-21T22:36:54.917812+0000","172.21.15.84:6800/820908470":"2024-09-21T22:36:54.917812+0000","172.21.15.84:0/567614908":"2024-09-21T22:36:54.917812+0000","172.21.15.84:0/2963622459":"2024-09-21T22:42:07.596045+0000","172.21.15.84:0/1176849276":"2024-09-21T22:37:17.690104+0000","172.21.15.84:6800/2932420050":"2024-09-21T22:37:17.690104+0000","172.21.15.84:0/3879263635":"2024-09-21T22:37:17.690104+0000","172.21.15.84:0/969472279":"2024-09-21T22:36:54.917812+0000","172.21.15.84:0/1361143214":"2024-09-21T22:37:17.690104+0000","172.21.15.84:0/3353126955":"2024-09-21T22:42:07.596045+0000","172.21.15.84:6800/4024278029":"2024-09-21T22:42:07.596045+0000","172.21.15.84:0/547902262":"2024-09-21T22:42:07.596045+0000","172.21.15.84:0/1242929317":"2024-09-21T22:42:07.596045+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-20T22:59:10.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:10 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/1165834166' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-20T22:59:10.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:10 smithi084 ceph-mon[29774]: 6.17 scrub starts 2024-09-20T22:59:10.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:10 smithi084 ceph-mon[29774]: 6.17 scrub ok 2024-09-20T22:59:10.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:10 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/1165834166' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-20T22:59:10.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:10 smithi084 ceph-mon[40024]: 6.17 scrub starts 2024-09-20T22:59:10.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:10 smithi084 ceph-mon[40024]: 6.17 scrub ok 2024-09-20T22:59:10.950 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph tell osd.0 flush_pg_stats 2024-09-20T22:59:10.951 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph tell osd.1 flush_pg_stats 2024-09-20T22:59:10.951 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph tell osd.2 flush_pg_stats 2024-09-20T22:59:10.952 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph tell osd.3 flush_pg_stats 2024-09-20T22:59:10.952 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph tell osd.4 flush_pg_stats 2024-09-20T22:59:10.952 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph tell osd.5 flush_pg_stats 2024-09-20T22:59:10.953 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph tell osd.6 flush_pg_stats 2024-09-20T22:59:10.953 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph tell osd.7 flush_pg_stats 2024-09-20T22:59:11.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:10 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/1165834166' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-20T22:59:11.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:10 smithi154 ceph-mon[33264]: 6.17 scrub starts 2024-09-20T22:59:11.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:10 smithi154 ceph-mon[33264]: 6.17 scrub ok 2024-09-20T22:59:11.398 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:59:11.464 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:59:11.464 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:59:11.484 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:59:11.492 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:59:11.492 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:59:11.499 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:59:11.504 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:59:11.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:11 smithi084 ceph-mon[29774]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:59:11.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:11 smithi084 ceph-mon[29774]: pgmap v1471: 142 pgs: 142 active+clean; 583 KiB data, 648 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-20T22:59:11.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:11 smithi084 ceph-mon[29774]: 6.6 scrub starts 2024-09-20T22:59:11.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:11 smithi084 ceph-mon[29774]: 6.6 scrub ok 2024-09-20T22:59:11.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:11 smithi084 ceph-mon[29774]: purged_snaps scrub starts 2024-09-20T22:59:11.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:11 smithi084 ceph-mon[29774]: purged_snaps scrub ok 2024-09-20T22:59:11.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:11 smithi084 ceph-mon[29774]: osdmap e1035: 8 total, 8 up, 8 in 2024-09-20T22:59:11.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:11 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:59:11.866 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:11 smithi084 ceph-mon[40024]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:59:11.866 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:11 smithi084 ceph-mon[40024]: pgmap v1471: 142 pgs: 142 active+clean; 583 KiB data, 648 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-20T22:59:11.866 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:11 smithi084 ceph-mon[40024]: 6.6 scrub starts 2024-09-20T22:59:11.866 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:11 smithi084 ceph-mon[40024]: 6.6 scrub ok 2024-09-20T22:59:11.866 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:11 smithi084 ceph-mon[40024]: purged_snaps scrub starts 2024-09-20T22:59:11.866 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:11 smithi084 ceph-mon[40024]: purged_snaps scrub ok 2024-09-20T22:59:11.866 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:11 smithi084 ceph-mon[40024]: osdmap e1035: 8 total, 8 up, 8 in 2024-09-20T22:59:11.866 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:11 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:59:12.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:11 smithi154 ceph-mon[33264]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:59:12.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:11 smithi154 ceph-mon[33264]: pgmap v1471: 142 pgs: 142 active+clean; 583 KiB data, 648 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-20T22:59:12.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:11 smithi154 ceph-mon[33264]: 6.6 scrub starts 2024-09-20T22:59:12.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:11 smithi154 ceph-mon[33264]: 6.6 scrub ok 2024-09-20T22:59:12.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:11 smithi154 ceph-mon[33264]: purged_snaps scrub starts 2024-09-20T22:59:12.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:11 smithi154 ceph-mon[33264]: purged_snaps scrub ok 2024-09-20T22:59:12.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:11 smithi154 ceph-mon[33264]: osdmap e1035: 8 total, 8 up, 8 in 2024-09-20T22:59:12.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:11 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:59:12.924 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:12 smithi084 ceph-mon[40024]: 3.b scrub starts 2024-09-20T22:59:12.925 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:12 smithi084 ceph-mon[40024]: 3.b scrub ok 2024-09-20T22:59:12.925 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:12 smithi084 ceph-mon[40024]: 5.d deep-scrub starts 2024-09-20T22:59:12.925 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:12 smithi084 ceph-mon[40024]: 5.d deep-scrub ok 2024-09-20T22:59:12.925 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:12 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:59:12.925 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:12 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:59:12.925 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:12 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:59:12.925 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:12 smithi084 ceph-mon[40024]: 5.f scrub starts 2024-09-20T22:59:12.925 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:12 smithi084 ceph-mon[40024]: 5.f scrub ok 2024-09-20T22:59:12.925 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:12 smithi084 ceph-mon[40024]: 6.3 scrub starts 2024-09-20T22:59:12.925 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:12 smithi084 ceph-mon[40024]: 6.3 scrub ok 2024-09-20T22:59:12.926 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:12 smithi084 ceph-mon[29774]: 3.b scrub starts 2024-09-20T22:59:12.926 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:12 smithi084 ceph-mon[29774]: 3.b scrub ok 2024-09-20T22:59:12.926 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:12 smithi084 ceph-mon[29774]: 5.d deep-scrub starts 2024-09-20T22:59:12.926 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:12 smithi084 ceph-mon[29774]: 5.d deep-scrub ok 2024-09-20T22:59:12.926 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:12 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:59:12.926 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:12 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:59:12.926 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:12 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:59:12.926 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:12 smithi084 ceph-mon[29774]: 5.f scrub starts 2024-09-20T22:59:12.926 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:12 smithi084 ceph-mon[29774]: 5.f scrub ok 2024-09-20T22:59:12.926 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:12 smithi084 ceph-mon[29774]: 6.3 scrub starts 2024-09-20T22:59:12.926 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:12 smithi084 ceph-mon[29774]: 6.3 scrub ok 2024-09-20T22:59:13.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:12 smithi154 ceph-mon[33264]: 3.b scrub starts 2024-09-20T22:59:13.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:12 smithi154 ceph-mon[33264]: 3.b scrub ok 2024-09-20T22:59:13.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:12 smithi154 ceph-mon[33264]: 5.d deep-scrub starts 2024-09-20T22:59:13.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:12 smithi154 ceph-mon[33264]: 5.d deep-scrub ok 2024-09-20T22:59:13.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:12 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:59:13.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:12 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:59:13.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:12 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:59:13.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:12 smithi154 ceph-mon[33264]: 5.f scrub starts 2024-09-20T22:59:13.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:12 smithi154 ceph-mon[33264]: 5.f scrub ok 2024-09-20T22:59:13.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:12 smithi154 ceph-mon[33264]: 6.3 scrub starts 2024-09-20T22:59:13.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:12 smithi154 ceph-mon[33264]: 6.3 scrub ok 2024-09-20T22:59:13.345 INFO:teuthology.orchestra.run.smithi084.stdout:206158430429 2024-09-20T22:59:13.346 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd last-stat-seq osd.6 2024-09-20T22:59:13.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:13 smithi084 ceph-mon[40024]: pgmap v1473: 142 pgs: 142 active+clean; 583 KiB data, 648 MiB used, 715 GiB / 715 GiB avail; 705 B/s rd, 0 op/s 2024-09-20T22:59:13.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:13 smithi084 ceph-mon[40024]: 3.7 scrub starts 2024-09-20T22:59:13.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:13 smithi084 ceph-mon[40024]: 3.7 scrub ok 2024-09-20T22:59:13.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:13 smithi084 ceph-mon[40024]: 4.9 scrub starts 2024-09-20T22:59:13.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:13 smithi084 ceph-mon[40024]: 4.9 scrub ok 2024-09-20T22:59:13.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:13 smithi084 ceph-mon[40024]: osdmap e1036: 8 total, 8 up, 8 in 2024-09-20T22:59:13.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:13 smithi084 ceph-mon[29774]: pgmap v1473: 142 pgs: 142 active+clean; 583 KiB data, 648 MiB used, 715 GiB / 715 GiB avail; 705 B/s rd, 0 op/s 2024-09-20T22:59:13.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:13 smithi084 ceph-mon[29774]: 3.7 scrub starts 2024-09-20T22:59:13.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:13 smithi084 ceph-mon[29774]: 3.7 scrub ok 2024-09-20T22:59:13.866 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:13 smithi084 ceph-mon[29774]: 4.9 scrub starts 2024-09-20T22:59:13.866 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:13 smithi084 ceph-mon[29774]: 4.9 scrub ok 2024-09-20T22:59:13.866 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:13 smithi084 ceph-mon[29774]: osdmap e1036: 8 total, 8 up, 8 in 2024-09-20T22:59:14.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:13 smithi154 ceph-mon[33264]: pgmap v1473: 142 pgs: 142 active+clean; 583 KiB data, 648 MiB used, 715 GiB / 715 GiB avail; 705 B/s rd, 0 op/s 2024-09-20T22:59:14.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:13 smithi154 ceph-mon[33264]: 3.7 scrub starts 2024-09-20T22:59:14.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:13 smithi154 ceph-mon[33264]: 3.7 scrub ok 2024-09-20T22:59:14.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:13 smithi154 ceph-mon[33264]: 4.9 scrub starts 2024-09-20T22:59:14.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:13 smithi154 ceph-mon[33264]: 4.9 scrub ok 2024-09-20T22:59:14.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:13 smithi154 ceph-mon[33264]: osdmap e1036: 8 total, 8 up, 8 in 2024-09-20T22:59:14.964 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:14 smithi084 ceph-mon[40024]: 5.12 scrub starts 2024-09-20T22:59:14.964 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:14 smithi084 ceph-mon[40024]: 5.12 scrub ok 2024-09-20T22:59:14.964 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:14 smithi084 ceph-mon[40024]: 5.7 scrub starts 2024-09-20T22:59:14.965 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:14 smithi084 ceph-mon[40024]: 5.7 scrub ok 2024-09-20T22:59:14.965 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:14 smithi084 ceph-mon[40024]: 4.13 scrub starts 2024-09-20T22:59:14.965 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:14 smithi084 ceph-mon[40024]: 4.13 scrub ok 2024-09-20T22:59:14.965 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:14 smithi084 ceph-mon[29774]: 5.12 scrub starts 2024-09-20T22:59:14.965 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:14 smithi084 ceph-mon[29774]: 5.12 scrub ok 2024-09-20T22:59:14.965 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:14 smithi084 ceph-mon[29774]: 5.7 scrub starts 2024-09-20T22:59:14.965 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:14 smithi084 ceph-mon[29774]: 5.7 scrub ok 2024-09-20T22:59:14.965 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:14 smithi084 ceph-mon[29774]: 4.13 scrub starts 2024-09-20T22:59:14.965 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:14 smithi084 ceph-mon[29774]: 4.13 scrub ok 2024-09-20T22:59:14.994 INFO:teuthology.orchestra.run.smithi084.stdout:253403070679 2024-09-20T22:59:14.994 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd last-stat-seq osd.7 2024-09-20T22:59:15.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:14 smithi154 ceph-mon[33264]: 5.12 scrub starts 2024-09-20T22:59:15.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:14 smithi154 ceph-mon[33264]: 5.12 scrub ok 2024-09-20T22:59:15.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:14 smithi154 ceph-mon[33264]: 5.7 scrub starts 2024-09-20T22:59:15.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:14 smithi154 ceph-mon[33264]: 5.7 scrub ok 2024-09-20T22:59:15.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:14 smithi154 ceph-mon[33264]: 4.13 scrub starts 2024-09-20T22:59:15.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:14 smithi154 ceph-mon[33264]: 4.13 scrub ok 2024-09-20T22:59:15.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:15 smithi084 ceph-mon[40024]: pgmap v1475: 142 pgs: 142 active+clean; 583 KiB data, 649 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-09-20T22:59:15.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:15 smithi084 ceph-mon[40024]: 5.17 scrub starts 2024-09-20T22:59:15.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:15 smithi084 ceph-mon[40024]: 5.17 scrub ok 2024-09-20T22:59:15.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:15 smithi084 ceph-mon[29774]: pgmap v1475: 142 pgs: 142 active+clean; 583 KiB data, 649 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-09-20T22:59:15.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:15 smithi084 ceph-mon[29774]: 5.17 scrub starts 2024-09-20T22:59:15.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:15 smithi084 ceph-mon[29774]: 5.17 scrub ok 2024-09-20T22:59:16.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:15 smithi154 ceph-mon[33264]: pgmap v1475: 142 pgs: 142 active+clean; 583 KiB data, 649 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-09-20T22:59:16.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:15 smithi154 ceph-mon[33264]: 5.17 scrub starts 2024-09-20T22:59:16.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:15 smithi154 ceph-mon[33264]: 5.17 scrub ok 2024-09-20T22:59:16.636 INFO:teuthology.orchestra.run.smithi084.stdout:77309411571 2024-09-20T22:59:16.636 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd last-stat-seq osd.2 2024-09-20T22:59:17.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:17 smithi084 ceph-mon[40024]: pgmap v1476: 142 pgs: 142 active+clean; 583 KiB data, 649 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:59:17.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:17 smithi084 ceph-mon[40024]: 6.4 scrub starts 2024-09-20T22:59:17.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:17 smithi084 ceph-mon[40024]: 6.4 scrub ok 2024-09-20T22:59:17.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:17 smithi084 ceph-mon[40024]: 5.15 scrub starts 2024-09-20T22:59:17.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:17 smithi084 ceph-mon[40024]: 5.15 scrub ok 2024-09-20T22:59:17.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:17 smithi084 ceph-mon[29774]: pgmap v1476: 142 pgs: 142 active+clean; 583 KiB data, 649 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:59:17.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:17 smithi084 ceph-mon[29774]: 6.4 scrub starts 2024-09-20T22:59:17.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:17 smithi084 ceph-mon[29774]: 6.4 scrub ok 2024-09-20T22:59:17.866 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:17 smithi084 ceph-mon[29774]: 5.15 scrub starts 2024-09-20T22:59:17.866 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:17 smithi084 ceph-mon[29774]: 5.15 scrub ok 2024-09-20T22:59:18.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:17 smithi154 ceph-mon[33264]: pgmap v1476: 142 pgs: 142 active+clean; 583 KiB data, 649 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:59:18.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:17 smithi154 ceph-mon[33264]: 6.4 scrub starts 2024-09-20T22:59:18.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:17 smithi154 ceph-mon[33264]: 6.4 scrub ok 2024-09-20T22:59:18.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:17 smithi154 ceph-mon[33264]: 5.15 scrub starts 2024-09-20T22:59:18.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:17 smithi154 ceph-mon[33264]: 5.15 scrub ok 2024-09-20T22:59:18.204 INFO:teuthology.orchestra.run.smithi084.stdout:34359738621 2024-09-20T22:59:18.205 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd last-stat-seq osd.0 2024-09-20T22:59:18.523 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:59:18 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: ::ffff:172.21.15.154 - - [20/Sep/2024:22:59:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-20T22:59:18.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:18 smithi084 ceph-mon[40024]: 6.11 scrub starts 2024-09-20T22:59:18.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:18 smithi084 ceph-mon[40024]: 6.11 scrub ok 2024-09-20T22:59:18.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:18 smithi084 ceph-mon[40024]: 4.1f deep-scrub starts 2024-09-20T22:59:18.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:18 smithi084 ceph-mon[40024]: 4.1f deep-scrub ok 2024-09-20T22:59:18.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:18 smithi084 ceph-mon[40024]: 3.17 scrub starts 2024-09-20T22:59:18.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:18 smithi084 ceph-mon[40024]: 3.17 scrub ok 2024-09-20T22:59:18.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:18 smithi084 ceph-mon[40024]: 1.1 scrub starts 2024-09-20T22:59:18.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:18 smithi084 ceph-mon[40024]: 1.1 scrub ok 2024-09-20T22:59:18.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:18 smithi084 ceph-mon[29774]: 6.11 scrub starts 2024-09-20T22:59:18.866 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:18 smithi084 ceph-mon[29774]: 6.11 scrub ok 2024-09-20T22:59:18.866 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:18 smithi084 ceph-mon[29774]: 4.1f deep-scrub starts 2024-09-20T22:59:18.866 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:18 smithi084 ceph-mon[29774]: 4.1f deep-scrub ok 2024-09-20T22:59:18.866 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:18 smithi084 ceph-mon[29774]: 3.17 scrub starts 2024-09-20T22:59:18.866 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:18 smithi084 ceph-mon[29774]: 3.17 scrub ok 2024-09-20T22:59:18.866 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:18 smithi084 ceph-mon[29774]: 1.1 scrub starts 2024-09-20T22:59:18.866 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:18 smithi084 ceph-mon[29774]: 1.1 scrub ok 2024-09-20T22:59:18.952 INFO:teuthology.orchestra.run.smithi084.stdout:4243427688467 2024-09-20T22:59:18.953 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd last-stat-seq osd.3 2024-09-20T22:59:19.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:18 smithi154 ceph-mon[33264]: 6.11 scrub starts 2024-09-20T22:59:19.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:18 smithi154 ceph-mon[33264]: 6.11 scrub ok 2024-09-20T22:59:19.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:18 smithi154 ceph-mon[33264]: 4.1f deep-scrub starts 2024-09-20T22:59:19.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:18 smithi154 ceph-mon[33264]: 4.1f deep-scrub ok 2024-09-20T22:59:19.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:18 smithi154 ceph-mon[33264]: 3.17 scrub starts 2024-09-20T22:59:19.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:18 smithi154 ceph-mon[33264]: 3.17 scrub ok 2024-09-20T22:59:19.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:18 smithi154 ceph-mon[33264]: 1.1 scrub starts 2024-09-20T22:59:19.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:18 smithi154 ceph-mon[33264]: 1.1 scrub ok 2024-09-20T22:59:19.599 INFO:teuthology.orchestra.run.smithi084.stdout:55834575096 2024-09-20T22:59:19.599 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd last-stat-seq osd.1 2024-09-20T22:59:19.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:19 smithi084 ceph-mon[29774]: pgmap v1477: 142 pgs: 142 active+clean; 583 KiB data, 649 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T22:59:19.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:19 smithi084 ceph-mon[29774]: 3.10 scrub starts 2024-09-20T22:59:19.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:19 smithi084 ceph-mon[29774]: 3.10 scrub ok 2024-09-20T22:59:19.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:19 smithi084 ceph-mon[40024]: pgmap v1477: 142 pgs: 142 active+clean; 583 KiB data, 649 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T22:59:19.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:19 smithi084 ceph-mon[40024]: 3.10 scrub starts 2024-09-20T22:59:19.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:19 smithi084 ceph-mon[40024]: 3.10 scrub ok 2024-09-20T22:59:20.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:19 smithi154 ceph-mon[33264]: pgmap v1477: 142 pgs: 142 active+clean; 583 KiB data, 649 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-20T22:59:20.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:19 smithi154 ceph-mon[33264]: 3.10 scrub starts 2024-09-20T22:59:20.005 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:19 smithi154 ceph-mon[33264]: 3.10 scrub ok 2024-09-20T22:59:20.005 INFO:journalctl@ceph.iscsi.iscsi.a.smithi154.stdout:Sep 20 22:59:19 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-iscsi-iscsi-a[59730]: debug there is no tcmu-runner data available 2024-09-20T22:59:20.095 INFO:teuthology.orchestra.run.smithi084.stdout:171798692067 2024-09-20T22:59:20.095 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd last-stat-seq osd.5 2024-09-20T22:59:20.198 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:59:20.198 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:59:20.199 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:59:20.199 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:59:20.199 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:59:20.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:20 smithi084 ceph-mon[40024]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:59:20.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:20 smithi084 ceph-mon[40024]: pgmap v1478: 142 pgs: 142 active+clean; 583 KiB data, 649 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T22:59:20.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:20 smithi084 ceph-mon[29774]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:59:20.866 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:20 smithi084 ceph-mon[29774]: pgmap v1478: 142 pgs: 142 active+clean; 583 KiB data, 649 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T22:59:21.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:20 smithi154 ceph-mon[33264]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:59:21.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:20 smithi154 ceph-mon[33264]: pgmap v1478: 142 pgs: 142 active+clean; 583 KiB data, 649 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T22:59:21.857 INFO:teuthology.orchestra.run.smithi084.stdout:253403070681 2024-09-20T22:59:22.020 INFO:teuthology.orchestra.run.smithi084.stdout:77309411572 2024-09-20T22:59:22.020 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:21 smithi084 ceph-mon[40024]: 5.1d deep-scrub starts 2024-09-20T22:59:22.020 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:21 smithi084 ceph-mon[40024]: 5.1d deep-scrub ok 2024-09-20T22:59:22.021 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:21 smithi084 ceph-mon[40024]: 4.11 scrub starts 2024-09-20T22:59:22.021 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:21 smithi084 ceph-mon[40024]: 4.11 scrub ok 2024-09-20T22:59:22.021 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:21 smithi084 ceph-mon[29774]: 5.1d deep-scrub starts 2024-09-20T22:59:22.021 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:21 smithi084 ceph-mon[29774]: 5.1d deep-scrub ok 2024-09-20T22:59:22.021 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:21 smithi084 ceph-mon[29774]: 4.11 scrub starts 2024-09-20T22:59:22.021 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:21 smithi084 ceph-mon[29774]: 4.11 scrub ok 2024-09-20T22:59:22.223 INFO:teuthology.orchestra.run.smithi084.stdout:206158430431 2024-09-20T22:59:22.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:21 smithi154 ceph-mon[33264]: 5.1d deep-scrub starts 2024-09-20T22:59:22.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:21 smithi154 ceph-mon[33264]: 5.1d deep-scrub ok 2024-09-20T22:59:22.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:21 smithi154 ceph-mon[33264]: 4.11 scrub starts 2024-09-20T22:59:22.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:21 smithi154 ceph-mon[33264]: 4.11 scrub ok 2024-09-20T22:59:22.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:22 smithi084 ceph-mon[40024]: 5.2 scrub starts 2024-09-20T22:59:22.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:22 smithi084 ceph-mon[40024]: 5.2 scrub ok 2024-09-20T22:59:22.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:22 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/3394902717' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-20T22:59:22.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:22 smithi084 ceph-mon[40024]: pgmap v1479: 142 pgs: 142 active+clean; 583 KiB data, 649 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T22:59:22.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:22 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/1767768516' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-20T22:59:22.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:22 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/1028211466' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-20T22:59:22.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:22 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/4264196025' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-20T22:59:22.866 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:22 smithi084 ceph-mon[29774]: 5.2 scrub starts 2024-09-20T22:59:22.866 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:22 smithi084 ceph-mon[29774]: 5.2 scrub ok 2024-09-20T22:59:22.866 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:22 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/3394902717' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-20T22:59:22.866 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:22 smithi084 ceph-mon[29774]: pgmap v1479: 142 pgs: 142 active+clean; 583 KiB data, 649 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T22:59:22.866 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:22 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/1767768516' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-20T22:59:22.866 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:22 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/1028211466' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-20T22:59:22.866 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:22 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/4264196025' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-20T22:59:22.904 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411571 got 77309411572 for osd.2 2024-09-20T22:59:22.904 DEBUG:teuthology.parallel:result is None 2024-09-20T22:59:23.002 INFO:teuthology.orchestra.run.smithi084.stdout:55834575097 2024-09-20T22:59:23.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:22 smithi154 ceph-mon[33264]: 5.2 scrub starts 2024-09-20T22:59:23.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:22 smithi154 ceph-mon[33264]: 5.2 scrub ok 2024-09-20T22:59:23.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:22 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/3394902717' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-20T22:59:23.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:22 smithi154 ceph-mon[33264]: pgmap v1479: 142 pgs: 142 active+clean; 583 KiB data, 649 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-20T22:59:23.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:22 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/1767768516' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-20T22:59:23.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:22 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/1028211466' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-20T22:59:23.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:22 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/4264196025' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-20T22:59:23.481 INFO:teuthology.orchestra.run.smithi084.stdout:4243427688468 2024-09-20T22:59:23.711 INFO:teuthology.orchestra.run.smithi084.stdout:133143986408 2024-09-20T22:59:23.711 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph osd last-stat-seq osd.4 2024-09-20T22:59:24.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:23 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/123126675' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-20T22:59:24.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:23 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:59:24.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:23 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/123126675' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-20T22:59:24.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:23 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:59:24.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:23 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/123126675' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-20T22:59:24.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:23 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T22:59:24.322 INFO:tasks.cephadm.ceph_manager.ceph:need seq 253403070679 got 253403070681 for osd.7 2024-09-20T22:59:24.322 DEBUG:teuthology.parallel:result is None 2024-09-20T22:59:24.914 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:24 smithi084 ceph-mon[40024]: pgmap v1480: 142 pgs: 142 active+clean; 583 KiB data, 649 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T22:59:24.914 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:24 smithi084 ceph-mon[29774]: pgmap v1480: 142 pgs: 142 active+clean; 583 KiB data, 649 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T22:59:24.946 INFO:tasks.cephadm.ceph_manager.ceph:need seq 4243427688467 got 4243427688468 for osd.3 2024-09-20T22:59:24.946 DEBUG:teuthology.parallel:result is None 2024-09-20T22:59:25.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:24 smithi154 ceph-mon[33264]: pgmap v1480: 142 pgs: 142 active+clean; 583 KiB data, 649 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-20T22:59:25.654 INFO:tasks.cephadm.ceph_manager.ceph:need seq 206158430429 got 206158430431 for osd.6 2024-09-20T22:59:25.654 DEBUG:teuthology.parallel:result is None 2024-09-20T22:59:25.717 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:59:25.717 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:59:26.054 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:25 smithi084 ceph-mon[29774]: 3.e scrub starts 2024-09-20T22:59:26.054 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:25 smithi084 ceph-mon[29774]: 3.e scrub ok 2024-09-20T22:59:26.054 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:25 smithi084 ceph-mon[29774]: 1.7 scrub starts 2024-09-20T22:59:26.054 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:25 smithi084 ceph-mon[29774]: 1.7 scrub ok 2024-09-20T22:59:26.055 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:25 smithi084 ceph-mon[29774]: 4.1e scrub starts 2024-09-20T22:59:26.055 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:25 smithi084 ceph-mon[29774]: 4.1e scrub ok 2024-09-20T22:59:26.055 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:25 smithi084 ceph-mon[40024]: 3.e scrub starts 2024-09-20T22:59:26.055 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:25 smithi084 ceph-mon[40024]: 3.e scrub ok 2024-09-20T22:59:26.055 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:25 smithi084 ceph-mon[40024]: 1.7 scrub starts 2024-09-20T22:59:26.055 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:25 smithi084 ceph-mon[40024]: 1.7 scrub ok 2024-09-20T22:59:26.055 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:25 smithi084 ceph-mon[40024]: 4.1e scrub starts 2024-09-20T22:59:26.055 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:25 smithi084 ceph-mon[40024]: 4.1e scrub ok 2024-09-20T22:59:26.087 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834575096 got 55834575097 for osd.1 2024-09-20T22:59:26.088 DEBUG:teuthology.parallel:result is None 2024-09-20T22:59:26.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:25 smithi154 ceph-mon[33264]: 3.e scrub starts 2024-09-20T22:59:26.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:25 smithi154 ceph-mon[33264]: 3.e scrub ok 2024-09-20T22:59:26.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:25 smithi154 ceph-mon[33264]: 1.7 scrub starts 2024-09-20T22:59:26.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:25 smithi154 ceph-mon[33264]: 1.7 scrub ok 2024-09-20T22:59:26.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:25 smithi154 ceph-mon[33264]: 4.1e scrub starts 2024-09-20T22:59:26.255 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:25 smithi154 ceph-mon[33264]: 4.1e scrub ok 2024-09-20T22:59:26.662 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:59:26.854 INFO:teuthology.orchestra.run.smithi084.stdout:34359738623 2024-09-20T22:59:26.900 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:26 smithi084 ceph-mon[40024]: 5.10 scrub starts 2024-09-20T22:59:26.900 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:26 smithi084 ceph-mon[40024]: 5.10 scrub ok 2024-09-20T22:59:26.900 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:26 smithi084 ceph-mon[40024]: pgmap v1481: 142 pgs: 142 active+clean; 583 KiB data, 649 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:59:26.900 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:26 smithi084 ceph-mon[29774]: 5.10 scrub starts 2024-09-20T22:59:26.900 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:26 smithi084 ceph-mon[29774]: 5.10 scrub ok 2024-09-20T22:59:26.900 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:26 smithi084 ceph-mon[29774]: pgmap v1481: 142 pgs: 142 active+clean; 583 KiB data, 649 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:59:27.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:26 smithi154 ceph-mon[33264]: 5.10 scrub starts 2024-09-20T22:59:27.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:26 smithi154 ceph-mon[33264]: 5.10 scrub ok 2024-09-20T22:59:27.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:26 smithi154 ceph-mon[33264]: pgmap v1481: 142 pgs: 142 active+clean; 583 KiB data, 649 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:59:27.449 INFO:teuthology.orchestra.run.smithi084.stdout:171798692069 2024-09-20T22:59:28.042 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:27 smithi084 ceph-mon[40024]: 4.10 scrub starts 2024-09-20T22:59:28.042 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:27 smithi084 ceph-mon[40024]: 4.10 scrub ok 2024-09-20T22:59:28.043 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:27 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/1065706158' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-20T22:59:28.043 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:27 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/1027244246' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-20T22:59:28.043 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:27 smithi084 ceph-mon[29774]: 4.10 scrub starts 2024-09-20T22:59:28.043 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:27 smithi084 ceph-mon[29774]: 4.10 scrub ok 2024-09-20T22:59:28.043 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:27 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/1065706158' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-20T22:59:28.043 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:27 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/1027244246' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-20T22:59:28.044 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738621 got 34359738623 for osd.0 2024-09-20T22:59:28.044 DEBUG:teuthology.parallel:result is None 2024-09-20T22:59:28.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:27 smithi154 ceph-mon[33264]: 4.10 scrub starts 2024-09-20T22:59:28.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:27 smithi154 ceph-mon[33264]: 4.10 scrub ok 2024-09-20T22:59:28.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:27 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/1065706158' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-20T22:59:28.254 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:27 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/1027244246' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-20T22:59:28.364 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:59:28 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: ::ffff:172.21.15.154 - - [20/Sep/2024:22:59:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-20T22:59:28.780 INFO:tasks.cephadm.ceph_manager.ceph:need seq 171798692067 got 171798692069 for osd.5 2024-09-20T22:59:28.780 DEBUG:teuthology.parallel:result is None 2024-09-20T22:59:29.144 INFO:teuthology.orchestra.run.smithi084.stdout:133143986410 2024-09-20T22:59:29.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:29 smithi154 ceph-mon[33264]: pgmap v1482: 142 pgs: 142 active+clean; 583 KiB data, 649 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:59:29.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:29 smithi154 ceph-mon[33264]: 6.c scrub starts 2024-09-20T22:59:29.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:29 smithi154 ceph-mon[33264]: 6.c scrub ok 2024-09-20T22:59:29.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:29 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/2139281031' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-20T22:59:29.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:29 smithi084 ceph-mon[40024]: pgmap v1482: 142 pgs: 142 active+clean; 583 KiB data, 649 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:59:29.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:29 smithi084 ceph-mon[40024]: 6.c scrub starts 2024-09-20T22:59:29.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:29 smithi084 ceph-mon[40024]: 6.c scrub ok 2024-09-20T22:59:29.865 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:29 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/2139281031' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-20T22:59:29.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:29 smithi084 ceph-mon[29774]: pgmap v1482: 142 pgs: 142 active+clean; 583 KiB data, 649 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:59:29.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:29 smithi084 ceph-mon[29774]: 6.c scrub starts 2024-09-20T22:59:29.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:29 smithi084 ceph-mon[29774]: 6.c scrub ok 2024-09-20T22:59:29.865 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:29 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/2139281031' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-20T22:59:29.900 INFO:tasks.cephadm.ceph_manager.ceph:need seq 133143986408 got 133143986410 for osd.4 2024-09-20T22:59:29.900 DEBUG:teuthology.parallel:result is None 2024-09-20T22:59:29.900 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-09-20T22:59:29.900 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph pg dump --format=json 2024-09-20T22:59:30.148 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:59:30.254 INFO:journalctl@ceph.iscsi.iscsi.a.smithi154.stdout:Sep 20 22:59:29 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-iscsi-iscsi-a[59730]: debug there is no tcmu-runner data available 2024-09-20T22:59:30.549 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:30 smithi084 ceph-mon[29774]: 6.1c scrub starts 2024-09-20T22:59:30.549 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:30 smithi084 ceph-mon[29774]: 6.1c scrub ok 2024-09-20T22:59:30.549 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:30 smithi084 ceph-mon[29774]: 3.1 scrub starts 2024-09-20T22:59:30.549 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:30 smithi084 ceph-mon[29774]: 3.1 scrub ok 2024-09-20T22:59:30.550 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:30 smithi084 ceph-mon[40024]: 6.1c scrub starts 2024-09-20T22:59:30.550 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:30 smithi084 ceph-mon[40024]: 6.1c scrub ok 2024-09-20T22:59:30.550 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:30 smithi084 ceph-mon[40024]: 3.1 scrub starts 2024-09-20T22:59:30.550 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:30 smithi084 ceph-mon[40024]: 3.1 scrub ok 2024-09-20T22:59:30.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:30 smithi154 ceph-mon[33264]: 6.1c scrub starts 2024-09-20T22:59:30.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:30 smithi154 ceph-mon[33264]: 6.1c scrub ok 2024-09-20T22:59:30.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:30 smithi154 ceph-mon[33264]: 3.1 scrub starts 2024-09-20T22:59:30.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:30 smithi154 ceph-mon[33264]: 3.1 scrub ok 2024-09-20T22:59:30.814 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-20T22:59:30.816 INFO:teuthology.orchestra.run.smithi084.stderr:dumped all 2024-09-20T22:59:31.503 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:31 smithi084 ceph-mon[40024]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:59:31.503 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:31 smithi084 ceph-mon[40024]: pgmap v1483: 142 pgs: 142 active+clean; 583 KiB data, 649 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:59:31.503 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:31 smithi084 ceph-mon[29774]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:59:31.504 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:31 smithi084 ceph-mon[29774]: pgmap v1483: 142 pgs: 142 active+clean; 583 KiB data, 649 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:59:31.532 INFO:teuthology.orchestra.run.smithi084.stdout:{"pg_ready":true,"pg_map":{"version":1483,"stamp":"2024-09-20T22:59:29.926555+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596507,"num_objects":199,"num_object_clones":0,"num_object_copies":597,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":199,"num_whiteouts":0,"num_read":1540,"num_read_kb":1292,"num_write":722,"num_write_kb":45,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":564,"num_bytes_recovered":669409,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1296,"ondisk_log_size":1296,"up":426,"acting":426,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":426,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":664192,"kb_used_data":83772,"kb_used_omap":159,"kb_used_meta":580128,"kb_avail":749297024,"statfs":{"total":767960285184,"available":767280152576,"internally_reserved":0,"allocated":85782528,"data_stored":53025633,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":163115,"internal_metadata":594051797},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":15,"num_read_kb":15,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003727"},"pg_stats":[{"pgid":"3.1f","version":"86'2","reported_seq":1354,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:13.389990+0000","last_change":"2024-09-20T22:58:47.675255+0000","last_active":"2024-09-20T22:59:13.389990+0000","last_peered":"2024-09-20T22:59:13.389990+0000","last_clean":"2024-09-20T22:59:13.389990+0000","last_became_active":"2024-09-20T22:47:49.222114+0000","last_became_peered":"2024-09-20T22:47:49.222114+0000","last_unstale":"2024-09-20T22:59:13.389990+0000","last_undegraded":"2024-09-20T22:59:13.389990+0000","last_fullsized":"2024-09-20T22:59:13.389990+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'2","last_scrub_stamp":"2024-09-20T22:58:47.675170+0000","last_deep_scrub":"86'2","last_deep_scrub_stamp":"2024-09-20T22:56:47.723216+0000","last_clean_scrub_stamp":"2024-09-20T22:58:47.675170+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T23:00:17.341427+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":436,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"625'15","reported_seq":983,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:13.390022+0000","last_change":"2024-09-20T22:58:48.609445+0000","last_active":"2024-09-20T22:59:13.390022+0000","last_peered":"2024-09-20T22:59:13.390022+0000","last_clean":"2024-09-20T22:59:13.390022+0000","last_became_active":"2024-09-20T22:58:48.609061+0000","last_became_peered":"2024-09-20T22:58:48.609061+0000","last_unstale":"2024-09-20T22:59:13.390022+0000","last_undegraded":"2024-09-20T22:59:13.390022+0000","last_fullsized":"2024-09-20T22:59:13.390022+0000","mapping_epoch":1022,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1023,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'15","last_scrub_stamp":"2024-09-20T22:58:29.448675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:58:29.448675+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:37.818163+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":557,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:14.015532+0000","last_change":"2024-09-20T22:58:28.399056+0000","last_active":"2024-09-20T22:59:14.015532+0000","last_peered":"2024-09-20T22:59:14.015532+0000","last_clean":"2024-09-20T22:59:14.015532+0000","last_became_active":"2024-09-20T22:58:28.398686+0000","last_became_peered":"2024-09-20T22:58:28.398686+0000","last_unstale":"2024-09-20T22:59:14.015532+0000","last_undegraded":"2024-09-20T22:59:14.015532+0000","last_fullsized":"2024-09-20T22:59:14.015532+0000","mapping_epoch":1008,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":1009,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:58:21.323903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:58:21.323903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:49.625461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":952,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:13.390376+0000","last_change":"2024-09-20T22:58:56.218532+0000","last_active":"2024-09-20T22:59:13.390376+0000","last_peered":"2024-09-20T22:59:13.390376+0000","last_clean":"2024-09-20T22:59:13.390376+0000","last_became_active":"2024-09-20T22:58:48.609762+0000","last_became_peered":"2024-09-20T22:58:48.609762+0000","last_unstale":"2024-09-20T22:59:13.390376+0000","last_undegraded":"2024-09-20T22:59:13.390376+0000","last_fullsized":"2024-09-20T22:59:13.390376+0000","mapping_epoch":1022,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":1023,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:58:56.218432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:58:56.218432+0000","last_clean_scrub_stamp":"2024-09-20T22:58:56.218432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T23:00:22.404129+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":1893,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:13.486173+0000","last_change":"2024-09-20T22:58:43.919784+0000","last_active":"2024-09-20T22:59:13.486173+0000","last_peered":"2024-09-20T22:59:13.486173+0000","last_clean":"2024-09-20T22:59:13.486173+0000","last_became_active":"2024-09-20T22:58:28.405275+0000","last_became_peered":"2024-09-20T22:58:28.405275+0000","last_unstale":"2024-09-20T22:59:13.486173+0000","last_undegraded":"2024-09-20T22:59:13.486173+0000","last_fullsized":"2024-09-20T22:59:13.486173+0000","mapping_epoch":1008,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":1009,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:58:43.919676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:58:43.919676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:45.701337+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":1886,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:13.486525+0000","last_change":"2024-09-20T22:58:46.836440+0000","last_active":"2024-09-20T22:59:13.486525+0000","last_peered":"2024-09-20T22:59:13.486525+0000","last_clean":"2024-09-20T22:59:13.486525+0000","last_became_active":"2024-09-20T22:57:49.405928+0000","last_became_peered":"2024-09-20T22:57:49.405928+0000","last_unstale":"2024-09-20T22:59:13.486525+0000","last_undegraded":"2024-09-20T22:59:13.486525+0000","last_fullsized":"2024-09-20T22:59:13.486525+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:58:46.836366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:58:46.836366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T23:00:14.051353+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"625'27","reported_seq":1964,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:13.486381+0000","last_change":"2024-09-20T22:58:45.874779+0000","last_active":"2024-09-20T22:59:13.486381+0000","last_peered":"2024-09-20T22:59:13.486381+0000","last_clean":"2024-09-20T22:59:13.486381+0000","last_became_active":"2024-09-20T22:57:49.413932+0000","last_became_peered":"2024-09-20T22:57:49.413932+0000","last_unstale":"2024-09-20T22:59:13.486381+0000","last_undegraded":"2024-09-20T22:59:13.486381+0000","last_fullsized":"2024-09-20T22:59:13.486381+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'27","last_scrub_stamp":"2024-09-20T22:58:45.874695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:58:45.874695+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T23:00:12.578444+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":944,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:13.390004+0000","last_change":"2024-09-20T22:59:04.188018+0000","last_active":"2024-09-20T22:59:13.390004+0000","last_peered":"2024-09-20T22:59:13.390004+0000","last_clean":"2024-09-20T22:59:13.390004+0000","last_became_active":"2024-09-20T22:58:48.609458+0000","last_became_peered":"2024-09-20T22:58:48.609458+0000","last_unstale":"2024-09-20T22:59:13.390004+0000","last_undegraded":"2024-09-20T22:59:13.390004+0000","last_fullsized":"2024-09-20T22:59:13.390004+0000","mapping_epoch":1022,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":1023,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:59:04.187927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:59:04.187927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T23:00:28.919832+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":2019,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:14.015365+0000","last_change":"2024-09-20T22:58:49.931571+0000","last_active":"2024-09-20T22:59:14.015365+0000","last_peered":"2024-09-20T22:59:14.015365+0000","last_clean":"2024-09-20T22:59:14.015365+0000","last_became_active":"2024-09-20T22:48:05.242059+0000","last_became_peered":"2024-09-20T22:48:05.242059+0000","last_unstale":"2024-09-20T22:59:14.015365+0000","last_undegraded":"2024-09-20T22:59:14.015365+0000","last_fullsized":"2024-09-20T22:59:14.015365+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:58:49.931482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:46:00.999939+0000","last_clean_scrub_stamp":"2024-09-20T22:58:49.931482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T23:00:06.415733+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"625'15","reported_seq":951,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:13.390392+0000","last_change":"2024-09-20T22:58:48.605304+0000","last_active":"2024-09-20T22:59:13.390392+0000","last_peered":"2024-09-20T22:59:13.390392+0000","last_clean":"2024-09-20T22:59:13.390392+0000","last_became_active":"2024-09-20T22:58:48.604796+0000","last_became_peered":"2024-09-20T22:58:48.604796+0000","last_unstale":"2024-09-20T22:59:13.390392+0000","last_undegraded":"2024-09-20T22:59:13.390392+0000","last_fullsized":"2024-09-20T22:59:13.390392+0000","mapping_epoch":1022,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1023,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'15","last_scrub_stamp":"2024-09-20T22:58:38.891685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:58:38.891685+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:53.871927+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":2023,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:14.014919+0000","last_change":"2024-09-20T22:59:01.041705+0000","last_active":"2024-09-20T22:59:14.014919+0000","last_peered":"2024-09-20T22:59:14.014919+0000","last_clean":"2024-09-20T22:59:14.014919+0000","last_became_active":"2024-09-20T22:58:28.404062+0000","last_became_peered":"2024-09-20T22:58:28.404062+0000","last_unstale":"2024-09-20T22:59:14.014919+0000","last_undegraded":"2024-09-20T22:59:14.014919+0000","last_fullsized":"2024-09-20T22:59:14.014919+0000","mapping_epoch":1008,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":1009,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:59:01.041618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:59:01.041618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T23:00:18.951667+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":1373,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:13.389421+0000","last_change":"2024-09-20T22:58:31.561377+0000","last_active":"2024-09-20T22:59:13.389421+0000","last_peered":"2024-09-20T22:59:13.389421+0000","last_clean":"2024-09-20T22:59:13.389421+0000","last_became_active":"2024-09-20T22:58:28.416222+0000","last_became_peered":"2024-09-20T22:58:28.416222+0000","last_unstale":"2024-09-20T22:59:13.389421+0000","last_undegraded":"2024-09-20T22:59:13.389421+0000","last_fullsized":"2024-09-20T22:59:13.389421+0000","mapping_epoch":1008,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":1009,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:58:31.561311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:49:08.664407+0000","last_clean_scrub_stamp":"2024-09-20T22:58:31.561311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:47.583014+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":2031,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:14.015183+0000","last_change":"2024-09-20T22:58:44.889584+0000","last_active":"2024-09-20T22:59:14.015183+0000","last_peered":"2024-09-20T22:59:14.015183+0000","last_clean":"2024-09-20T22:59:14.015183+0000","last_became_active":"2024-09-20T22:54:17.342619+0000","last_became_peered":"2024-09-20T22:54:17.342619+0000","last_unstale":"2024-09-20T22:59:14.015183+0000","last_undegraded":"2024-09-20T22:59:14.015183+0000","last_fullsized":"2024-09-20T22:59:14.015183+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:58:44.889510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:58:44.889510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T23:00:03.935621+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"625'9","reported_seq":985,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:13.390747+0000","last_change":"2024-09-20T22:58:53.199092+0000","last_active":"2024-09-20T22:59:13.390747+0000","last_peered":"2024-09-20T22:59:13.390747+0000","last_clean":"2024-09-20T22:59:13.390747+0000","last_became_active":"2024-09-20T22:58:48.609715+0000","last_became_peered":"2024-09-20T22:58:48.609715+0000","last_unstale":"2024-09-20T22:59:13.390747+0000","last_undegraded":"2024-09-20T22:59:13.390747+0000","last_fullsized":"2024-09-20T22:59:13.390747+0000","mapping_epoch":1022,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1023,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'9","last_scrub_stamp":"2024-09-20T22:58:53.198994+0000","last_deep_scrub":"181'7","last_deep_scrub_stamp":"2024-09-20T22:48:41.049381+0000","last_clean_scrub_stamp":"2024-09-20T22:58:53.198994+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T23:00:14.829955+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":9,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":73,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:14.159607+0000","last_change":"2024-09-20T22:58:43.194716+0000","last_active":"2024-09-20T22:59:14.159607+0000","last_peered":"2024-09-20T22:59:14.159607+0000","last_clean":"2024-09-20T22:59:14.159607+0000","last_became_active":"2024-09-20T22:58:28.411249+0000","last_became_peered":"2024-09-20T22:58:28.411249+0000","last_unstale":"2024-09-20T22:59:14.159607+0000","last_undegraded":"2024-09-20T22:59:14.159607+0000","last_fullsized":"2024-09-20T22:59:14.159607+0000","mapping_epoch":1008,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":1009,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:58:43.194615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:47:40.361958+0000","last_clean_scrub_stamp":"2024-09-20T22:58:43.194615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:45.713548+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":2019,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:14.014720+0000","last_change":"2024-09-20T22:58:37.932389+0000","last_active":"2024-09-20T22:59:14.014720+0000","last_peered":"2024-09-20T22:59:14.014720+0000","last_clean":"2024-09-20T22:59:14.014720+0000","last_became_active":"2024-09-20T22:57:49.402447+0000","last_became_peered":"2024-09-20T22:57:49.402447+0000","last_unstale":"2024-09-20T22:59:14.014720+0000","last_undegraded":"2024-09-20T22:59:14.014720+0000","last_fullsized":"2024-09-20T22:59:14.014720+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:58:37.932289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:56:14.512566+0000","last_clean_scrub_stamp":"2024-09-20T22:58:37.932289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:51.158159+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":99,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:13.391051+0000","last_change":"2024-09-20T22:58:48.612664+0000","last_active":"2024-09-20T22:59:13.391051+0000","last_peered":"2024-09-20T22:59:13.391051+0000","last_clean":"2024-09-20T22:59:13.391051+0000","last_became_active":"2024-09-20T22:58:48.612440+0000","last_became_peered":"2024-09-20T22:58:48.612440+0000","last_unstale":"2024-09-20T22:59:13.391051+0000","last_undegraded":"2024-09-20T22:59:13.391051+0000","last_fullsized":"2024-09-20T22:59:13.391051+0000","mapping_epoch":1022,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":1023,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:58:42.503926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:58:42.503926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T23:00:06.749766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":1367,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:13.390542+0000","last_change":"2024-09-20T22:59:04.817192+0000","last_active":"2024-09-20T22:59:13.390542+0000","last_peered":"2024-09-20T22:59:13.390542+0000","last_clean":"2024-09-20T22:59:13.390542+0000","last_became_active":"2024-09-20T22:58:28.408957+0000","last_became_peered":"2024-09-20T22:58:28.408957+0000","last_unstale":"2024-09-20T22:59:13.390542+0000","last_undegraded":"2024-09-20T22:59:13.390542+0000","last_fullsized":"2024-09-20T22:59:13.390542+0000","mapping_epoch":1008,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1009,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:59:04.817072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:49:53.933601+0000","last_clean_scrub_stamp":"2024-09-20T22:59:04.817072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T23:00:22.752292+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"625'34","reported_seq":2122,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:13.391536+0000","last_change":"2024-09-20T22:58:01.296040+0000","last_active":"2024-09-20T22:59:13.391536+0000","last_peered":"2024-09-20T22:59:13.391536+0000","last_clean":"2024-09-20T22:59:13.391536+0000","last_became_active":"2024-09-20T22:57:49.430403+0000","last_became_peered":"2024-09-20T22:57:49.430403+0000","last_unstale":"2024-09-20T22:59:13.391536+0000","last_undegraded":"2024-09-20T22:59:13.391536+0000","last_fullsized":"2024-09-20T22:59:13.391536+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'34","last_scrub_stamp":"2024-09-20T22:58:01.295956+0000","last_deep_scrub":"113'22","last_deep_scrub_stamp":"2024-09-20T22:43:12.191878+0000","last_clean_scrub_stamp":"2024-09-20T22:58:01.295956+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:30.939636+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":561,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:20.446597+0000","last_change":"2024-09-20T22:59:20.446630+0000","last_active":"2024-09-20T22:59:20.446597+0000","last_peered":"2024-09-20T22:59:20.446597+0000","last_clean":"2024-09-20T22:59:20.446597+0000","last_became_active":"2024-09-20T22:54:17.349501+0000","last_became_peered":"2024-09-20T22:54:17.349501+0000","last_unstale":"2024-09-20T22:59:20.446597+0000","last_undegraded":"2024-09-20T22:59:20.446597+0000","last_fullsized":"2024-09-20T22:59:20.446597+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:59:20.446504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:59:20.446504+0000","last_clean_scrub_stamp":"2024-09-20T22:59:20.446504+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T23:00:25.083082+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":1866,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:13.486537+0000","last_change":"2024-09-20T22:58:35.892172+0000","last_active":"2024-09-20T22:59:13.486537+0000","last_peered":"2024-09-20T22:59:13.486537+0000","last_clean":"2024-09-20T22:59:13.486537+0000","last_became_active":"2024-09-20T22:57:49.419671+0000","last_became_peered":"2024-09-20T22:57:49.419671+0000","last_unstale":"2024-09-20T22:59:13.486537+0000","last_undegraded":"2024-09-20T22:59:13.486537+0000","last_fullsized":"2024-09-20T22:59:13.486537+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:58:35.892096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:45:25.002199+0000","last_clean_scrub_stamp":"2024-09-20T22:58:35.892096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:52.759843+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":952,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:13.390663+0000","last_change":"2024-09-20T22:59:05.195013+0000","last_active":"2024-09-20T22:59:13.390663+0000","last_peered":"2024-09-20T22:59:13.390663+0000","last_clean":"2024-09-20T22:59:13.390663+0000","last_became_active":"2024-09-20T22:58:48.609194+0000","last_became_peered":"2024-09-20T22:58:48.609194+0000","last_unstale":"2024-09-20T22:59:13.390663+0000","last_undegraded":"2024-09-20T22:59:13.390663+0000","last_fullsized":"2024-09-20T22:59:13.390663+0000","mapping_epoch":1022,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1023,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:59:05.194924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:59:05.194924+0000","last_clean_scrub_stamp":"2024-09-20T22:59:05.194924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T23:00:18.806408+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"625'18","reported_seq":1944,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:13.486253+0000","last_change":"2024-09-20T22:58:44.904887+0000","last_active":"2024-09-20T22:59:13.486253+0000","last_peered":"2024-09-20T22:59:13.486253+0000","last_clean":"2024-09-20T22:59:13.486253+0000","last_became_active":"2024-09-20T22:57:49.403366+0000","last_became_peered":"2024-09-20T22:57:49.403366+0000","last_unstale":"2024-09-20T22:59:13.486253+0000","last_undegraded":"2024-09-20T22:59:13.486253+0000","last_fullsized":"2024-09-20T22:59:13.486253+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'18","last_scrub_stamp":"2024-09-20T22:58:44.904792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:58:44.904792+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:45.535866+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":940,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:13.390715+0000","last_change":"2024-09-20T22:58:54.239279+0000","last_active":"2024-09-20T22:59:13.390715+0000","last_peered":"2024-09-20T22:59:13.390715+0000","last_clean":"2024-09-20T22:59:13.390715+0000","last_became_active":"2024-09-20T22:58:48.608447+0000","last_became_peered":"2024-09-20T22:58:48.608447+0000","last_unstale":"2024-09-20T22:59:13.390715+0000","last_undegraded":"2024-09-20T22:59:13.390715+0000","last_fullsized":"2024-09-20T22:59:13.390715+0000","mapping_epoch":1022,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":1023,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:58:54.239199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:58:54.239199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T23:00:21.655788+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"154'2","reported_seq":77,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:28.619562+0000","last_change":"2024-09-20T22:59:28.619606+0000","last_active":"2024-09-20T22:59:28.619562+0000","last_peered":"2024-09-20T22:59:28.619562+0000","last_clean":"2024-09-20T22:59:28.619562+0000","last_became_active":"2024-09-20T22:58:28.419741+0000","last_became_peered":"2024-09-20T22:58:28.419741+0000","last_unstale":"2024-09-20T22:59:28.619562+0000","last_undegraded":"2024-09-20T22:59:28.619562+0000","last_fullsized":"2024-09-20T22:59:28.619562+0000","mapping_epoch":1008,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":1009,"parent":"0.0","parent_split_bits":0,"last_scrub":"154'2","last_scrub_stamp":"2024-09-20T22:59:28.619469+0000","last_deep_scrub":"154'2","last_deep_scrub_stamp":"2024-09-20T22:43:54.134066+0000","last_clean_scrub_stamp":"2024-09-20T22:59:28.619469+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T23:00:44.846126+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":403,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"894'2","reported_seq":567,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:14.015393+0000","last_change":"2024-09-20T22:58:32.360859+0000","last_active":"2024-09-20T22:59:14.015393+0000","last_peered":"2024-09-20T22:59:14.015393+0000","last_clean":"2024-09-20T22:59:14.015393+0000","last_became_active":"2024-09-20T22:57:49.405458+0000","last_became_peered":"2024-09-20T22:57:49.405458+0000","last_unstale":"2024-09-20T22:59:14.015393+0000","last_undegraded":"2024-09-20T22:59:14.015393+0000","last_fullsized":"2024-09-20T22:59:14.015393+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"894'2","last_scrub_stamp":"2024-09-20T22:58:32.360782+0000","last_deep_scrub":"894'2","last_deep_scrub_stamp":"2024-09-20T22:56:15.522649+0000","last_clean_scrub_stamp":"2024-09-20T22:58:32.360782+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:58.358240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"625'25","reported_seq":1430,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:24.781166+0000","last_change":"2024-09-20T22:59:24.781214+0000","last_active":"2024-09-20T22:59:24.781166+0000","last_peered":"2024-09-20T22:59:24.781166+0000","last_clean":"2024-09-20T22:59:24.781166+0000","last_became_active":"2024-09-20T22:58:28.410047+0000","last_became_peered":"2024-09-20T22:58:28.410047+0000","last_unstale":"2024-09-20T22:59:24.781166+0000","last_undegraded":"2024-09-20T22:59:24.781166+0000","last_fullsized":"2024-09-20T22:59:24.781166+0000","mapping_epoch":1008,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1009,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'25","last_scrub_stamp":"2024-09-20T22:59:24.781101+0000","last_deep_scrub":"151'15","last_deep_scrub_stamp":"2024-09-20T22:43:50.715761+0000","last_clean_scrub_stamp":"2024-09-20T22:59:24.781101+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T23:00:43.559126+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"1009'13","reported_seq":2292,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:23.399948+0000","last_change":"2024-09-20T22:58:32.407579+0000","last_active":"2024-09-20T22:59:23.399948+0000","last_peered":"2024-09-20T22:59:23.399948+0000","last_clean":"2024-09-20T22:59:23.399948+0000","last_became_active":"2024-09-20T22:58:28.399826+0000","last_became_peered":"2024-09-20T22:58:28.399826+0000","last_unstale":"2024-09-20T22:59:23.399948+0000","last_undegraded":"2024-09-20T22:59:23.399948+0000","last_fullsized":"2024-09-20T22:59:23.399948+0000","mapping_epoch":1008,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":1009,"parent":"0.0","parent_split_bits":0,"last_scrub":"1009'13","last_scrub_stamp":"2024-09-20T22:58:32.407440+0000","last_deep_scrub":"661'11","last_deep_scrub_stamp":"2024-09-20T22:52:21.907943+0000","last_clean_scrub_stamp":"2024-09-20T22:58:32.407440+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:51.848588+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.a","version":"77'128","reported_seq":2231,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:14.014558+0000","last_change":"2024-09-20T22:58:55.396912+0000","last_active":"2024-09-20T22:59:14.014558+0000","last_peered":"2024-09-20T22:59:14.014558+0000","last_clean":"2024-09-20T22:59:14.014558+0000","last_became_active":"2024-09-20T22:58:55.396426+0000","last_became_peered":"2024-09-20T22:58:55.396426+0000","last_unstale":"2024-09-20T22:59:14.014558+0000","last_undegraded":"2024-09-20T22:59:14.014558+0000","last_fullsized":"2024-09-20T22:59:14.014558+0000","mapping_epoch":1025,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":1026,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:58:22.154735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:58:22.154735+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:29.181918+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"625'27","reported_seq":578,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:14.016400+0000","last_change":"2024-09-20T22:58:01.424871+0000","last_active":"2024-09-20T22:59:14.016400+0000","last_peered":"2024-09-20T22:59:14.016400+0000","last_clean":"2024-09-20T22:59:14.016400+0000","last_became_active":"2024-09-20T22:57:49.415260+0000","last_became_peered":"2024-09-20T22:57:49.415260+0000","last_unstale":"2024-09-20T22:59:14.016400+0000","last_undegraded":"2024-09-20T22:59:14.016400+0000","last_fullsized":"2024-09-20T22:59:14.016400+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'27","last_scrub_stamp":"2024-09-20T22:58:01.424772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:58:01.424772+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:30.323196+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":1909,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:13.485910+0000","last_change":"2024-09-20T22:59:00.884919+0000","last_active":"2024-09-20T22:59:13.485910+0000","last_peered":"2024-09-20T22:59:13.485910+0000","last_clean":"2024-09-20T22:59:13.485910+0000","last_became_active":"2024-09-20T22:58:28.406214+0000","last_became_peered":"2024-09-20T22:58:28.406214+0000","last_unstale":"2024-09-20T22:59:13.485910+0000","last_undegraded":"2024-09-20T22:59:13.485910+0000","last_fullsized":"2024-09-20T22:59:13.485910+0000","mapping_epoch":1008,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1009,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:59:00.884824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:49:12.070675+0000","last_clean_scrub_stamp":"2024-09-20T22:59:00.884824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T23:00:03.144291+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"1023'15","reported_seq":1040,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:23.399949+0000","last_change":"2024-09-20T22:58:57.200171+0000","last_active":"2024-09-20T22:59:23.399949+0000","last_peered":"2024-09-20T22:59:23.399949+0000","last_clean":"2024-09-20T22:59:23.399949+0000","last_became_active":"2024-09-20T22:58:48.610492+0000","last_became_peered":"2024-09-20T22:58:48.610492+0000","last_unstale":"2024-09-20T22:59:23.399949+0000","last_undegraded":"2024-09-20T22:59:23.399949+0000","last_fullsized":"2024-09-20T22:59:23.399949+0000","mapping_epoch":1022,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":1023,"parent":"0.0","parent_split_bits":0,"last_scrub":"1023'15","last_scrub_stamp":"2024-09-20T22:58:57.200085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:58:57.200085+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:59.588926+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":2019,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:14.015078+0000","last_change":"2024-09-20T22:59:02.014253+0000","last_active":"2024-09-20T22:59:14.015078+0000","last_peered":"2024-09-20T22:59:14.015078+0000","last_clean":"2024-09-20T22:59:14.015078+0000","last_became_active":"2024-09-20T22:54:17.340327+0000","last_became_peered":"2024-09-20T22:54:17.340327+0000","last_unstale":"2024-09-20T22:59:14.015078+0000","last_undegraded":"2024-09-20T22:59:14.015078+0000","last_fullsized":"2024-09-20T22:59:14.015078+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:59:02.014154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:59:02.014154+0000","last_clean_scrub_stamp":"2024-09-20T22:59:02.014154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T23:00:22.922237+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.5","version":"77'128","reported_seq":1801,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:13.391234+0000","last_change":"2024-09-20T22:58:48.611905+0000","last_active":"2024-09-20T22:59:13.391234+0000","last_peered":"2024-09-20T22:59:13.391234+0000","last_clean":"2024-09-20T22:59:13.391234+0000","last_became_active":"2024-09-20T22:58:48.610563+0000","last_became_peered":"2024-09-20T22:58:48.610563+0000","last_unstale":"2024-09-20T22:59:13.391234+0000","last_undegraded":"2024-09-20T22:59:13.391234+0000","last_fullsized":"2024-09-20T22:59:13.391234+0000","mapping_epoch":1022,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":1023,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:58:42.277786+0000","last_deep_scrub":"77'128","last_deep_scrub_stamp":"2024-09-20T22:57:14.262265+0000","last_clean_scrub_stamp":"2024-09-20T22:58:42.277786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:59.697230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"625'30","reported_seq":2028,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:13.485914+0000","last_change":"2024-09-20T22:58:29.898911+0000","last_active":"2024-09-20T22:59:13.485914+0000","last_peered":"2024-09-20T22:59:13.485914+0000","last_clean":"2024-09-20T22:59:13.485914+0000","last_became_active":"2024-09-20T22:58:28.408416+0000","last_became_peered":"2024-09-20T22:58:28.408416+0000","last_unstale":"2024-09-20T22:59:13.485914+0000","last_undegraded":"2024-09-20T22:59:13.485914+0000","last_fullsized":"2024-09-20T22:59:13.485914+0000","mapping_epoch":1008,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1009,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'30","last_scrub_stamp":"2024-09-20T22:58:29.898807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:58:29.898807+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:59.376020+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":39,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":1930,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:13.486750+0000","last_change":"2024-09-20T22:59:12.098368+0000","last_active":"2024-09-20T22:59:13.486750+0000","last_peered":"2024-09-20T22:59:13.486750+0000","last_clean":"2024-09-20T22:59:13.486750+0000","last_became_active":"2024-09-20T22:58:28.409296+0000","last_became_peered":"2024-09-20T22:58:28.409296+0000","last_unstale":"2024-09-20T22:59:13.486750+0000","last_undegraded":"2024-09-20T22:59:13.486750+0000","last_fullsized":"2024-09-20T22:59:13.486750+0000","mapping_epoch":1008,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1009,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:59:12.098263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:59:12.098263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T23:00:22.984205+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":941,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:13.390441+0000","last_change":"2024-09-20T22:58:59.149598+0000","last_active":"2024-09-20T22:59:13.390441+0000","last_peered":"2024-09-20T22:59:13.390441+0000","last_clean":"2024-09-20T22:59:13.390441+0000","last_became_active":"2024-09-20T22:58:49.164911+0000","last_became_peered":"2024-09-20T22:58:49.164911+0000","last_unstale":"2024-09-20T22:59:13.390441+0000","last_undegraded":"2024-09-20T22:59:13.390441+0000","last_fullsized":"2024-09-20T22:59:13.390441+0000","mapping_epoch":1022,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":1023,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:58:59.149475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:56:47.432483+0000","last_clean_scrub_stamp":"2024-09-20T22:58:59.149475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T23:00:02.308884+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":933,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:13.390480+0000","last_change":"2024-09-20T22:58:52.223664+0000","last_active":"2024-09-20T22:59:13.390480+0000","last_peered":"2024-09-20T22:59:13.390480+0000","last_clean":"2024-09-20T22:59:13.390480+0000","last_became_active":"2024-09-20T22:58:48.608756+0000","last_became_peered":"2024-09-20T22:58:48.608756+0000","last_unstale":"2024-09-20T22:59:13.390480+0000","last_undegraded":"2024-09-20T22:59:13.390480+0000","last_fullsized":"2024-09-20T22:59:13.390480+0000","mapping_epoch":1022,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":1023,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:58:52.223572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:53.036601+0000","last_clean_scrub_stamp":"2024-09-20T22:58:52.223572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:58.874292+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"855'129","reported_seq":2232,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:13.486596+0000","last_change":"2024-09-20T22:59:08.033384+0000","last_active":"2024-09-20T22:59:13.486596+0000","last_peered":"2024-09-20T22:59:13.486596+0000","last_clean":"2024-09-20T22:59:13.486596+0000","last_became_active":"2024-09-20T22:59:01.645724+0000","last_became_peered":"2024-09-20T22:59:01.645724+0000","last_unstale":"2024-09-20T22:59:13.486596+0000","last_undegraded":"2024-09-20T22:59:13.486596+0000","last_fullsized":"2024-09-20T22:59:13.486596+0000","mapping_epoch":1029,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":1030,"parent":"0.0","parent_split_bits":3,"last_scrub":"855'129","last_scrub_stamp":"2024-09-20T22:59:08.033310+0000","last_deep_scrub":"855'129","last_deep_scrub_stamp":"2024-09-20T22:55:35.536654+0000","last_clean_scrub_stamp":"2024-09-20T22:59:08.033310+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":59,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T23:00:12.485840+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":655360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"1030'35","reported_seq":985,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:13.390353+0000","last_change":"2024-09-20T22:59:02.172329+0000","last_active":"2024-09-20T22:59:13.390353+0000","last_peered":"2024-09-20T22:59:13.390353+0000","last_clean":"2024-09-20T22:59:13.390353+0000","last_became_active":"2024-09-20T22:58:48.611983+0000","last_became_peered":"2024-09-20T22:58:48.611983+0000","last_unstale":"2024-09-20T22:59:13.390353+0000","last_undegraded":"2024-09-20T22:59:13.390353+0000","last_fullsized":"2024-09-20T22:59:13.390353+0000","mapping_epoch":1022,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1023,"parent":"0.0","parent_split_bits":0,"last_scrub":"1030'35","last_scrub_stamp":"2024-09-20T22:59:02.172205+0000","last_deep_scrub":"1030'35","last_deep_scrub_stamp":"2024-09-20T22:59:02.172205+0000","last_clean_scrub_stamp":"2024-09-20T22:59:02.172205+0000","objects_scrubbed":7,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T23:00:17.266739+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":41,"num_read_kb":34,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"855'2","reported_seq":1385,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:13.390833+0000","last_change":"2024-09-20T22:58:26.489611+0000","last_active":"2024-09-20T22:59:13.390833+0000","last_peered":"2024-09-20T22:59:13.390833+0000","last_clean":"2024-09-20T22:59:13.390833+0000","last_became_active":"2024-09-20T22:54:17.340423+0000","last_became_peered":"2024-09-20T22:54:17.340423+0000","last_unstale":"2024-09-20T22:59:13.390833+0000","last_undegraded":"2024-09-20T22:59:13.390833+0000","last_fullsized":"2024-09-20T22:59:13.390833+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"855'2","last_scrub_stamp":"2024-09-20T22:58:26.489536+0000","last_deep_scrub":"855'2","last_deep_scrub_stamp":"2024-09-20T22:55:35.572977+0000","last_clean_scrub_stamp":"2024-09-20T22:58:26.489536+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:52.640705+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":230,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"989'18","reported_seq":2114,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:18.399448+0000","last_change":"2024-09-20T22:58:54.817713+0000","last_active":"2024-09-20T22:59:18.399448+0000","last_peered":"2024-09-20T22:59:18.399448+0000","last_clean":"2024-09-20T22:59:18.399448+0000","last_became_active":"2024-09-20T22:57:49.401445+0000","last_became_peered":"2024-09-20T22:57:49.401445+0000","last_unstale":"2024-09-20T22:59:18.399448+0000","last_undegraded":"2024-09-20T22:59:18.399448+0000","last_fullsized":"2024-09-20T22:59:18.399448+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"989'18","last_scrub_stamp":"2024-09-20T22:58:54.817628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:58:54.817628+0000","objects_scrubbed":1,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T23:00:22.475889+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":73,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:14.159500+0000","last_change":"2024-09-20T22:59:12.347762+0000","last_active":"2024-09-20T22:59:14.159500+0000","last_peered":"2024-09-20T22:59:14.159500+0000","last_clean":"2024-09-20T22:59:14.159500+0000","last_became_active":"2024-09-20T22:58:28.406517+0000","last_became_peered":"2024-09-20T22:58:28.406517+0000","last_unstale":"2024-09-20T22:59:14.159500+0000","last_undegraded":"2024-09-20T22:59:14.159500+0000","last_fullsized":"2024-09-20T22:59:14.159500+0000","mapping_epoch":1008,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":1009,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:59:12.347689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:59:12.347689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T23:00:26.078906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.7","version":"77'128","reported_seq":2359,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:24.565956+0000","last_change":"2024-09-20T22:59:24.565981+0000","last_active":"2024-09-20T22:59:24.565956+0000","last_peered":"2024-09-20T22:59:24.565956+0000","last_clean":"2024-09-20T22:59:24.565956+0000","last_became_active":"2024-09-20T22:58:31.410353+0000","last_became_peered":"2024-09-20T22:58:31.410353+0000","last_unstale":"2024-09-20T22:59:24.565956+0000","last_undegraded":"2024-09-20T22:59:24.565956+0000","last_fullsized":"2024-09-20T22:59:24.565956+0000","mapping_epoch":1011,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":1012,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:59:24.565892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:59:24.565892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T23:00:45.563252+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"904'25","reported_seq":578,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:14.015635+0000","last_change":"2024-09-20T22:59:08.413880+0000","last_active":"2024-09-20T22:59:14.015635+0000","last_peered":"2024-09-20T22:59:14.015635+0000","last_clean":"2024-09-20T22:59:14.015635+0000","last_became_active":"2024-09-20T22:54:17.349113+0000","last_became_peered":"2024-09-20T22:54:17.349113+0000","last_unstale":"2024-09-20T22:59:14.015635+0000","last_undegraded":"2024-09-20T22:59:14.015635+0000","last_fullsized":"2024-09-20T22:59:14.015635+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"904'25","last_scrub_stamp":"2024-09-20T22:59:08.413795+0000","last_deep_scrub":"904'25","last_deep_scrub_stamp":"2024-09-20T22:56:24.575150+0000","last_clean_scrub_stamp":"2024-09-20T22:59:08.413795+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T23:00:29.997661+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":28,"num_read_kb":23,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":2017,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:14.015473+0000","last_change":"2024-09-20T22:58:35.955562+0000","last_active":"2024-09-20T22:59:14.015473+0000","last_peered":"2024-09-20T22:59:14.015473+0000","last_clean":"2024-09-20T22:59:14.015473+0000","last_became_active":"2024-09-20T22:57:49.396185+0000","last_became_peered":"2024-09-20T22:57:49.396185+0000","last_unstale":"2024-09-20T22:59:14.015473+0000","last_undegraded":"2024-09-20T22:59:14.015473+0000","last_fullsized":"2024-09-20T22:59:14.015473+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:58:35.955496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:58:35.955496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:49.044151+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"733'12","reported_seq":1496,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:23.399608+0000","last_change":"2024-09-20T22:58:27.469722+0000","last_active":"2024-09-20T22:59:23.399608+0000","last_peered":"2024-09-20T22:59:23.399608+0000","last_clean":"2024-09-20T22:59:23.399608+0000","last_became_active":"2024-09-20T22:47:49.227264+0000","last_became_peered":"2024-09-20T22:47:49.227264+0000","last_unstale":"2024-09-20T22:59:23.399608+0000","last_undegraded":"2024-09-20T22:59:23.399608+0000","last_fullsized":"2024-09-20T22:59:23.399608+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"733'12","last_scrub_stamp":"2024-09-20T22:58:27.469646+0000","last_deep_scrub":"733'12","last_deep_scrub_stamp":"2024-09-20T22:58:27.469646+0000","last_clean_scrub_stamp":"2024-09-20T22:58:27.469646+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:56.471372+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":1359,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:13.389276+0000","last_change":"2024-09-20T22:58:28.488885+0000","last_active":"2024-09-20T22:59:13.389276+0000","last_peered":"2024-09-20T22:59:13.389276+0000","last_clean":"2024-09-20T22:59:13.389276+0000","last_became_active":"2024-09-20T22:57:49.416550+0000","last_became_peered":"2024-09-20T22:57:49.416550+0000","last_unstale":"2024-09-20T22:59:13.389276+0000","last_undegraded":"2024-09-20T22:59:13.389276+0000","last_fullsized":"2024-09-20T22:59:13.389276+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:58:28.488787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:58:28.488787+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:35.965266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"77'128","reported_seq":2343,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:13.746857+0000","last_change":"2024-09-20T22:58:03.362917+0000","last_active":"2024-09-20T22:59:13.746857+0000","last_peered":"2024-09-20T22:59:13.746857+0000","last_clean":"2024-09-20T22:59:13.746857+0000","last_became_active":"2024-09-20T22:57:49.406637+0000","last_became_peered":"2024-09-20T22:57:49.406637+0000","last_unstale":"2024-09-20T22:59:13.746857+0000","last_undegraded":"2024-09-20T22:59:13.746857+0000","last_fullsized":"2024-09-20T22:59:13.746857+0000","mapping_epoch":988,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:58:03.362805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:58:03.362805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:32.522555+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"625'37","reported_seq":646,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:14.014923+0000","last_change":"2024-09-20T22:58:59.018734+0000","last_active":"2024-09-20T22:59:14.014923+0000","last_peered":"2024-09-20T22:59:14.014923+0000","last_clean":"2024-09-20T22:59:14.014923+0000","last_became_active":"2024-09-20T22:58:09.157511+0000","last_became_peered":"2024-09-20T22:58:09.157511+0000","last_unstale":"2024-09-20T22:59:14.014923+0000","last_undegraded":"2024-09-20T22:59:14.014923+0000","last_fullsized":"2024-09-20T22:59:14.014923+0000","mapping_epoch":996,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":997,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'37","last_scrub_stamp":"2024-09-20T22:58:59.018611+0000","last_deep_scrub":"515'27","last_deep_scrub_stamp":"2024-09-20T22:49:55.964153+0000","last_clean_scrub_stamp":"2024-09-20T22:58:59.018611+0000","objects_scrubbed":8,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T23:00:27.942616+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":38,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":553,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:14.016144+0000","last_change":"2024-09-20T22:59:03.402620+0000","last_active":"2024-09-20T22:59:14.016144+0000","last_peered":"2024-09-20T22:59:14.016144+0000","last_clean":"2024-09-20T22:59:14.016144+0000","last_became_active":"2024-09-20T22:54:17.341248+0000","last_became_peered":"2024-09-20T22:54:17.341248+0000","last_unstale":"2024-09-20T22:59:14.016144+0000","last_undegraded":"2024-09-20T22:59:14.016144+0000","last_fullsized":"2024-09-20T22:59:14.016144+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:59:03.402478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:48:23.320031+0000","last_clean_scrub_stamp":"2024-09-20T22:59:03.402478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T23:00:27.744280+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":2034,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:21.548467+0000","last_change":"2024-09-20T22:59:21.548491+0000","last_active":"2024-09-20T22:59:21.548467+0000","last_peered":"2024-09-20T22:59:21.548467+0000","last_clean":"2024-09-20T22:59:21.548467+0000","last_became_active":"2024-09-20T22:47:49.214055+0000","last_became_peered":"2024-09-20T22:47:49.214055+0000","last_unstale":"2024-09-20T22:59:21.548467+0000","last_undegraded":"2024-09-20T22:59:21.548467+0000","last_fullsized":"2024-09-20T22:59:21.548467+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:59:21.548401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:59:21.548401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T23:00:26.729576+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":545,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:14.015534+0000","last_change":"2024-09-20T22:58:23.325856+0000","last_active":"2024-09-20T22:59:14.015534+0000","last_peered":"2024-09-20T22:59:14.015534+0000","last_clean":"2024-09-20T22:59:14.015534+0000","last_became_active":"2024-09-20T22:54:17.340458+0000","last_became_peered":"2024-09-20T22:54:17.340458+0000","last_unstale":"2024-09-20T22:59:14.015534+0000","last_undegraded":"2024-09-20T22:59:14.015534+0000","last_fullsized":"2024-09-20T22:59:14.015534+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:58:23.325766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:48:41.208853+0000","last_clean_scrub_stamp":"2024-09-20T22:58:23.325766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:44.933821+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"77'128","reported_seq":2325,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:17.904859+0000","last_change":"2024-09-20T22:59:17.904859+0000","last_active":"2024-09-20T22:59:17.904859+0000","last_peered":"2024-09-20T22:59:17.904859+0000","last_clean":"2024-09-20T22:59:17.904859+0000","last_became_active":"2024-09-20T22:47:52.208102+0000","last_became_peered":"2024-09-20T22:47:52.208102+0000","last_unstale":"2024-09-20T22:59:17.904859+0000","last_undegraded":"2024-09-20T22:59:17.904859+0000","last_fullsized":"2024-09-20T22:59:17.904859+0000","mapping_epoch":391,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:59:17.904754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:59:17.904754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T23:00:39.070223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"1026'57","reported_seq":471,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:23.721188+0000","last_change":"2024-09-20T22:58:55.397509+0000","last_active":"2024-09-20T22:59:23.721188+0000","last_peered":"2024-09-20T22:59:23.721188+0000","last_clean":"2024-09-20T22:59:23.721188+0000","last_became_active":"2024-09-20T22:58:55.397171+0000","last_became_peered":"2024-09-20T22:58:55.397171+0000","last_unstale":"2024-09-20T22:59:23.721188+0000","last_undegraded":"2024-09-20T22:59:23.721188+0000","last_fullsized":"2024-09-20T22:59:23.721188+0000","mapping_epoch":1025,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1026,"parent":"0.0","parent_split_bits":0,"last_scrub":"1006'56","last_scrub_stamp":"2024-09-20T22:58:33.199281+0000","last_deep_scrub":"1006'56","last_deep_scrub_stamp":"2024-09-20T22:58:33.199281+0000","last_clean_scrub_stamp":"2024-09-20T22:58:33.199281+0000","objects_scrubbed":10,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T23:00:00.564014+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":1074,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":933,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:13.390524+0000","last_change":"2024-09-20T22:59:09.156959+0000","last_active":"2024-09-20T22:59:13.390524+0000","last_peered":"2024-09-20T22:59:13.390524+0000","last_clean":"2024-09-20T22:59:13.390524+0000","last_became_active":"2024-09-20T22:58:48.608668+0000","last_became_peered":"2024-09-20T22:58:48.608668+0000","last_unstale":"2024-09-20T22:59:13.390524+0000","last_undegraded":"2024-09-20T22:59:13.390524+0000","last_fullsized":"2024-09-20T22:59:13.390524+0000","mapping_epoch":1022,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1023,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:59:09.156871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:59:09.156871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T23:00:21.048647+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"69'2","reported_seq":2012,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:14.014955+0000","last_change":"2024-09-20T22:59:04.001271+0000","last_active":"2024-09-20T22:59:14.014955+0000","last_peered":"2024-09-20T22:59:14.014955+0000","last_clean":"2024-09-20T22:59:14.014955+0000","last_became_active":"2024-09-20T22:58:55.394665+0000","last_became_peered":"2024-09-20T22:58:55.394665+0000","last_unstale":"2024-09-20T22:59:14.014955+0000","last_undegraded":"2024-09-20T22:59:14.014955+0000","last_fullsized":"2024-09-20T22:59:14.014955+0000","mapping_epoch":1025,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":1026,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2024-09-20T22:59:04.001168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:36.130684+0000","last_clean_scrub_stamp":"2024-09-20T22:59:04.001168+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T23:00:29.544905+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00087280599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":1368,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:13.390619+0000","last_change":"2024-09-20T22:58:34.555808+0000","last_active":"2024-09-20T22:59:13.390619+0000","last_peered":"2024-09-20T22:59:13.390619+0000","last_clean":"2024-09-20T22:59:13.390619+0000","last_became_active":"2024-09-20T22:54:17.339860+0000","last_became_peered":"2024-09-20T22:54:17.339860+0000","last_unstale":"2024-09-20T22:59:13.390619+0000","last_undegraded":"2024-09-20T22:59:13.390619+0000","last_fullsized":"2024-09-20T22:59:13.390619+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:58:34.555736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:58:34.555736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:45.003937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":1905,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:13.486839+0000","last_change":"2024-09-20T22:59:10.033024+0000","last_active":"2024-09-20T22:59:13.486839+0000","last_peered":"2024-09-20T22:59:13.486839+0000","last_clean":"2024-09-20T22:59:13.486839+0000","last_became_active":"2024-09-20T22:58:28.410700+0000","last_became_peered":"2024-09-20T22:58:28.410700+0000","last_unstale":"2024-09-20T22:59:13.486839+0000","last_undegraded":"2024-09-20T22:59:13.486839+0000","last_fullsized":"2024-09-20T22:59:13.486839+0000","mapping_epoch":1008,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":1009,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:59:10.032928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:59:10.032928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T23:00:16.267119+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"77'128","reported_seq":1368,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:14.159413+0000","last_change":"2024-09-20T22:59:02.307466+0000","last_active":"2024-09-20T22:59:14.159413+0000","last_peered":"2024-09-20T22:59:14.159413+0000","last_clean":"2024-09-20T22:59:14.159413+0000","last_became_active":"2024-09-20T22:59:01.642508+0000","last_became_peered":"2024-09-20T22:59:01.642508+0000","last_unstale":"2024-09-20T22:59:14.159413+0000","last_undegraded":"2024-09-20T22:59:14.159413+0000","last_fullsized":"2024-09-20T22:59:14.159413+0000","mapping_epoch":1029,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":1030,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:59:02.307342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:59:02.307342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T23:00:26.538592+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"625'22","reported_seq":568,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:14.016428+0000","last_change":"2024-09-20T22:58:48.547764+0000","last_active":"2024-09-20T22:59:14.016428+0000","last_peered":"2024-09-20T22:59:14.016428+0000","last_clean":"2024-09-20T22:59:14.016428+0000","last_became_active":"2024-09-20T22:54:17.349415+0000","last_became_peered":"2024-09-20T22:54:17.349415+0000","last_unstale":"2024-09-20T22:59:14.016428+0000","last_undegraded":"2024-09-20T22:59:14.016428+0000","last_fullsized":"2024-09-20T22:59:14.016428+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'22","last_scrub_stamp":"2024-09-20T22:58:48.547682+0000","last_deep_scrub":"443'18","last_deep_scrub_stamp":"2024-09-20T22:48:44.182517+0000","last_clean_scrub_stamp":"2024-09-20T22:58:48.547682+0000","objects_scrubbed":5,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:51.430246+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":24,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":1320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":553,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:14.015547+0000","last_change":"2024-09-20T22:58:31.390541+0000","last_active":"2024-09-20T22:59:14.015547+0000","last_peered":"2024-09-20T22:59:14.015547+0000","last_clean":"2024-09-20T22:59:14.015547+0000","last_became_active":"2024-09-20T22:54:17.343806+0000","last_became_peered":"2024-09-20T22:54:17.343806+0000","last_unstale":"2024-09-20T22:59:14.015547+0000","last_undegraded":"2024-09-20T22:59:14.015547+0000","last_fullsized":"2024-09-20T22:59:14.015547+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:58:31.390467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:56:00.249796+0000","last_clean_scrub_stamp":"2024-09-20T22:58:31.390467+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:44.179485+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"380'2","reported_seq":2071,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:13.391330+0000","last_change":"2024-09-20T22:58:23.141502+0000","last_active":"2024-09-20T22:59:13.391330+0000","last_peered":"2024-09-20T22:59:13.391330+0000","last_clean":"2024-09-20T22:59:13.391330+0000","last_became_active":"2024-09-20T22:57:49.416996+0000","last_became_peered":"2024-09-20T22:57:49.416996+0000","last_unstale":"2024-09-20T22:59:13.391330+0000","last_undegraded":"2024-09-20T22:59:13.391330+0000","last_fullsized":"2024-09-20T22:59:13.391330+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"380'2","last_scrub_stamp":"2024-09-20T22:58:23.141431+0000","last_deep_scrub":"380'2","last_deep_scrub_stamp":"2024-09-20T22:47:40.983039+0000","last_clean_scrub_stamp":"2024-09-20T22:58:23.141431+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:35.912629+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001301459,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":2028,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:13.390176+0000","last_change":"2024-09-20T22:58:46.283696+0000","last_active":"2024-09-20T22:59:13.390176+0000","last_peered":"2024-09-20T22:59:13.390176+0000","last_clean":"2024-09-20T22:59:13.390176+0000","last_became_active":"2024-09-20T22:58:28.408036+0000","last_became_peered":"2024-09-20T22:58:28.408036+0000","last_unstale":"2024-09-20T22:59:13.390176+0000","last_undegraded":"2024-09-20T22:59:13.390176+0000","last_fullsized":"2024-09-20T22:59:13.390176+0000","mapping_epoch":1008,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":1009,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:58:46.283615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:57:38.381457+0000","last_clean_scrub_stamp":"2024-09-20T22:58:46.283615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T23:00:00.632985+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":73,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:14.159866+0000","last_change":"2024-09-20T22:58:54.109803+0000","last_active":"2024-09-20T22:59:14.159866+0000","last_peered":"2024-09-20T22:59:14.159866+0000","last_clean":"2024-09-20T22:59:14.159866+0000","last_became_active":"2024-09-20T22:58:28.409911+0000","last_became_peered":"2024-09-20T22:58:28.409911+0000","last_unstale":"2024-09-20T22:59:14.159866+0000","last_undegraded":"2024-09-20T22:59:14.159866+0000","last_fullsized":"2024-09-20T22:59:14.159866+0000","mapping_epoch":1008,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":1009,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:58:54.109704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:58:54.109704+0000","last_clean_scrub_stamp":"2024-09-20T22:58:54.109704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T23:00:12.653208+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"77'128","reported_seq":1572,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:14.015832+0000","last_change":"2024-09-20T22:58:35.306441+0000","last_active":"2024-09-20T22:59:14.015832+0000","last_peered":"2024-09-20T22:59:14.015832+0000","last_clean":"2024-09-20T22:59:14.015832+0000","last_became_active":"2024-09-20T22:54:20.413191+0000","last_became_peered":"2024-09-20T22:54:20.413191+0000","last_unstale":"2024-09-20T22:59:14.015832+0000","last_undegraded":"2024-09-20T22:59:14.015832+0000","last_fullsized":"2024-09-20T22:59:14.015832+0000","mapping_epoch":779,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":780,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:58:35.306366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:58:35.306366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:39.963250+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"828'30","reported_seq":1448,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:13.390973+0000","last_change":"2024-09-20T22:59:00.715919+0000","last_active":"2024-09-20T22:59:13.390973+0000","last_peered":"2024-09-20T22:59:13.390973+0000","last_clean":"2024-09-20T22:59:13.390973+0000","last_became_active":"2024-09-20T22:55:11.346018+0000","last_became_peered":"2024-09-20T22:55:11.346018+0000","last_unstale":"2024-09-20T22:59:13.390973+0000","last_undegraded":"2024-09-20T22:59:13.390973+0000","last_fullsized":"2024-09-20T22:59:13.390973+0000","mapping_epoch":830,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":831,"parent":"0.0","parent_split_bits":0,"last_scrub":"828'30","last_scrub_stamp":"2024-09-20T22:59:00.715824+0000","last_deep_scrub":"828'30","last_deep_scrub_stamp":"2024-09-20T22:55:08.705571+0000","last_clean_scrub_stamp":"2024-09-20T22:59:00.715824+0000","objects_scrubbed":6,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T23:00:03.113273+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":1365,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:13.389587+0000","last_change":"2024-09-20T22:58:04.615065+0000","last_active":"2024-09-20T22:59:13.389587+0000","last_peered":"2024-09-20T22:59:13.389587+0000","last_clean":"2024-09-20T22:59:13.389587+0000","last_became_active":"2024-09-20T22:57:49.410539+0000","last_became_peered":"2024-09-20T22:57:49.410539+0000","last_unstale":"2024-09-20T22:59:13.389587+0000","last_undegraded":"2024-09-20T22:59:13.389587+0000","last_fullsized":"2024-09-20T22:59:13.389587+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:58:04.614957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:44:19.834641+0000","last_clean_scrub_stamp":"2024-09-20T22:58:04.614957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:21.673315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"75'5","reported_seq":138,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:27.559273+0000","last_change":"2024-09-20T22:58:46.149273+0000","last_active":"2024-09-20T22:59:27.559273+0000","last_peered":"2024-09-20T22:59:27.559273+0000","last_clean":"2024-09-20T22:59:27.559273+0000","last_became_active":"2024-09-20T22:58:28.407748+0000","last_became_peered":"2024-09-20T22:58:28.407748+0000","last_unstale":"2024-09-20T22:59:27.559273+0000","last_undegraded":"2024-09-20T22:59:27.559273+0000","last_fullsized":"2024-09-20T22:59:27.559273+0000","mapping_epoch":1008,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":1009,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-09-20T22:58:46.149200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:36.130684+0000","last_clean_scrub_stamp":"2024-09-20T22:58:46.149200+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T23:00:06.719526+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00078631700000000003,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":422,"num_read_kb":417,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":2032,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:14.015765+0000","last_change":"2024-09-20T22:59:14.015586+0000","last_active":"2024-09-20T22:59:14.015765+0000","last_peered":"2024-09-20T22:59:14.015765+0000","last_clean":"2024-09-20T22:59:14.015765+0000","last_became_active":"2024-09-20T22:54:17.339805+0000","last_became_peered":"2024-09-20T22:54:17.339805+0000","last_unstale":"2024-09-20T22:59:14.015765+0000","last_undegraded":"2024-09-20T22:59:14.015765+0000","last_fullsized":"2024-09-20T22:59:14.015765+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:59:14.015458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:57:13.660520+0000","last_clean_scrub_stamp":"2024-09-20T22:59:14.015458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T23:00:18.904523+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":562,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:16.488882+0000","last_change":"2024-09-20T22:59:16.488912+0000","last_active":"2024-09-20T22:59:16.488882+0000","last_peered":"2024-09-20T22:59:16.488882+0000","last_clean":"2024-09-20T22:59:16.488882+0000","last_became_active":"2024-09-20T22:57:49.429412+0000","last_became_peered":"2024-09-20T22:57:49.429412+0000","last_unstale":"2024-09-20T22:59:16.488882+0000","last_undegraded":"2024-09-20T22:59:16.488882+0000","last_fullsized":"2024-09-20T22:59:16.488882+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:59:16.488818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:59:16.488818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T23:00:23.730775+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"1009'34","reported_seq":2417,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:23.400258+0000","last_change":"2024-09-20T22:58:28.445239+0000","last_active":"2024-09-20T22:59:23.400258+0000","last_peered":"2024-09-20T22:59:23.400258+0000","last_clean":"2024-09-20T22:59:23.400258+0000","last_became_active":"2024-09-20T22:58:28.398920+0000","last_became_peered":"2024-09-20T22:58:28.398920+0000","last_unstale":"2024-09-20T22:59:23.400258+0000","last_undegraded":"2024-09-20T22:59:23.400258+0000","last_fullsized":"2024-09-20T22:59:23.400258+0000","mapping_epoch":1008,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1009,"parent":"0.0","parent_split_bits":0,"last_scrub":"888'33","last_scrub_stamp":"2024-09-20T22:58:26.407098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:58:26.407098+0000","objects_scrubbed":8,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:46.923321+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":924,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"67'1","reported_seq":1904,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:13.486322+0000","last_change":"2024-09-20T22:59:04.932715+0000","last_active":"2024-09-20T22:59:13.486322+0000","last_peered":"2024-09-20T22:59:13.486322+0000","last_clean":"2024-09-20T22:59:13.486322+0000","last_became_active":"2024-09-20T22:57:49.397191+0000","last_became_peered":"2024-09-20T22:57:49.397191+0000","last_unstale":"2024-09-20T22:59:13.486322+0000","last_undegraded":"2024-09-20T22:59:13.486322+0000","last_fullsized":"2024-09-20T22:59:13.486322+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-09-20T22:59:04.932639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:59:04.932639+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T23:00:32.833984+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"77'128","reported_seq":1363,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:14.159296+0000","last_change":"2024-09-20T22:59:03.275326+0000","last_active":"2024-09-20T22:59:14.159296+0000","last_peered":"2024-09-20T22:59:14.159296+0000","last_clean":"2024-09-20T22:59:14.159296+0000","last_became_active":"2024-09-20T22:58:31.407541+0000","last_became_peered":"2024-09-20T22:58:31.407541+0000","last_unstale":"2024-09-20T22:59:14.159296+0000","last_undegraded":"2024-09-20T22:59:14.159296+0000","last_fullsized":"2024-09-20T22:59:14.159296+0000","mapping_epoch":1011,"log_start":"77'128","ondisk_log_start":"77'128","created":19,"last_epoch_clean":1012,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:59:03.275234+0000","last_deep_scrub":"77'128","last_deep_scrub_stamp":"2024-09-20T22:47:22.417099+0000","last_clean_scrub_stamp":"2024-09-20T22:59:03.275234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T23:00:28.479860+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":65,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:14.159924+0000","last_change":"2024-09-20T22:58:28.417931+0000","last_active":"2024-09-20T22:59:14.159924+0000","last_peered":"2024-09-20T22:59:14.159924+0000","last_clean":"2024-09-20T22:59:14.159924+0000","last_became_active":"2024-09-20T22:58:28.417612+0000","last_became_peered":"2024-09-20T22:58:28.417612+0000","last_unstale":"2024-09-20T22:59:14.159924+0000","last_undegraded":"2024-09-20T22:59:14.159924+0000","last_fullsized":"2024-09-20T22:59:14.159924+0000","mapping_epoch":1008,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":1009,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:58:02.248808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:58:02.248808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:20.300201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":2017,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:14.015589+0000","last_change":"2024-09-20T22:58:31.907856+0000","last_active":"2024-09-20T22:59:14.015589+0000","last_peered":"2024-09-20T22:59:14.015589+0000","last_clean":"2024-09-20T22:59:14.015589+0000","last_became_active":"2024-09-20T22:57:49.410488+0000","last_became_peered":"2024-09-20T22:57:49.410488+0000","last_unstale":"2024-09-20T22:59:14.015589+0000","last_undegraded":"2024-09-20T22:59:14.015589+0000","last_fullsized":"2024-09-20T22:59:14.015589+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:58:31.907778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:58:31.907778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:38.295707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"850'24","reported_seq":587,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:13.391166+0000","last_change":"2024-09-20T22:59:08.152719+0000","last_active":"2024-09-20T22:59:13.391166+0000","last_peered":"2024-09-20T22:59:13.391166+0000","last_clean":"2024-09-20T22:59:13.391166+0000","last_became_active":"2024-09-20T22:58:48.609688+0000","last_became_peered":"2024-09-20T22:58:48.609688+0000","last_unstale":"2024-09-20T22:59:13.391166+0000","last_undegraded":"2024-09-20T22:59:13.391166+0000","last_fullsized":"2024-09-20T22:59:13.391166+0000","mapping_epoch":1022,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1023,"parent":"0.0","parent_split_bits":0,"last_scrub":"850'24","last_scrub_stamp":"2024-09-20T22:59:08.152583+0000","last_deep_scrub":"850'24","last_deep_scrub_stamp":"2024-09-20T22:55:30.779018+0000","last_clean_scrub_stamp":"2024-09-20T22:59:08.152583+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T23:00:15.420325+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":23,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":933,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:13.390084+0000","last_change":"2024-09-20T22:58:49.166015+0000","last_active":"2024-09-20T22:59:13.390084+0000","last_peered":"2024-09-20T22:59:13.390084+0000","last_clean":"2024-09-20T22:59:13.390084+0000","last_became_active":"2024-09-20T22:58:49.164972+0000","last_became_peered":"2024-09-20T22:58:49.164972+0000","last_unstale":"2024-09-20T22:59:13.390084+0000","last_undegraded":"2024-09-20T22:59:13.390084+0000","last_fullsized":"2024-09-20T22:59:13.390084+0000","mapping_epoch":1022,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1023,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:58:30.124573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:58:30.124573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:36.161664+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":2011,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:14.015507+0000","last_change":"2024-09-20T22:59:11.957364+0000","last_active":"2024-09-20T22:59:14.015507+0000","last_peered":"2024-09-20T22:59:14.015507+0000","last_clean":"2024-09-20T22:59:14.015507+0000","last_became_active":"2024-09-20T22:48:05.233778+0000","last_became_peered":"2024-09-20T22:48:05.233778+0000","last_unstale":"2024-09-20T22:59:14.015507+0000","last_undegraded":"2024-09-20T22:59:14.015507+0000","last_fullsized":"2024-09-20T22:59:14.015507+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:59:11.957292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:54:47.166836+0000","last_clean_scrub_stamp":"2024-09-20T22:59:11.957292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T23:00:23.803711+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":1865,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:13.486674+0000","last_change":"2024-09-20T22:57:49.427590+0000","last_active":"2024-09-20T22:59:13.486674+0000","last_peered":"2024-09-20T22:59:13.486674+0000","last_clean":"2024-09-20T22:59:13.486674+0000","last_became_active":"2024-09-20T22:57:49.425373+0000","last_became_peered":"2024-09-20T22:57:49.425373+0000","last_unstale":"2024-09-20T22:59:13.486674+0000","last_undegraded":"2024-09-20T22:59:13.486674+0000","last_fullsized":"2024-09-20T22:59:13.486674+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:56:46.453177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:48:40.127123+0000","last_clean_scrub_stamp":"2024-09-20T22:56:46.453177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"77'128","reported_seq":2240,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:14.014441+0000","last_change":"2024-09-20T22:58:55.396348+0000","last_active":"2024-09-20T22:59:14.014441+0000","last_peered":"2024-09-20T22:59:14.014441+0000","last_clean":"2024-09-20T22:59:14.014441+0000","last_became_active":"2024-09-20T22:58:55.396082+0000","last_became_peered":"2024-09-20T22:58:55.396082+0000","last_unstale":"2024-09-20T22:59:14.014441+0000","last_undegraded":"2024-09-20T22:59:14.014441+0000","last_fullsized":"2024-09-20T22:59:14.014441+0000","mapping_epoch":1025,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":1026,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:58:21.144759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:58:21.144759+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:31.432198+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"625'31","reported_seq":1028,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:13.391179+0000","last_change":"2024-09-20T22:59:06.149910+0000","last_active":"2024-09-20T22:59:13.391179+0000","last_peered":"2024-09-20T22:59:13.391179+0000","last_clean":"2024-09-20T22:59:13.391179+0000","last_became_active":"2024-09-20T22:58:48.608367+0000","last_became_peered":"2024-09-20T22:58:48.608367+0000","last_unstale":"2024-09-20T22:59:13.391179+0000","last_undegraded":"2024-09-20T22:59:13.391179+0000","last_fullsized":"2024-09-20T22:59:13.391179+0000","mapping_epoch":1022,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1023,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'31","last_scrub_stamp":"2024-09-20T22:59:06.149783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:59:06.149783+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T23:00:29.230470+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":49,"num_read_kb":39,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":1577,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:13.391155+0000","last_change":"2024-09-20T22:58:38.164880+0000","last_active":"2024-09-20T22:59:13.391155+0000","last_peered":"2024-09-20T22:59:13.391155+0000","last_clean":"2024-09-20T22:59:13.391155+0000","last_became_active":"2024-09-20T22:54:17.332636+0000","last_became_peered":"2024-09-20T22:54:17.332636+0000","last_unstale":"2024-09-20T22:59:13.391155+0000","last_undegraded":"2024-09-20T22:59:13.391155+0000","last_fullsized":"2024-09-20T22:59:13.391155+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:58:38.164809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:35.492562+0000","last_clean_scrub_stamp":"2024-09-20T22:58:38.164809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:38.777158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":553,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:14.016057+0000","last_change":"2024-09-20T22:58:57.350393+0000","last_active":"2024-09-20T22:59:14.016057+0000","last_peered":"2024-09-20T22:59:14.016057+0000","last_clean":"2024-09-20T22:59:14.016057+0000","last_became_active":"2024-09-20T22:54:17.349144+0000","last_became_peered":"2024-09-20T22:54:17.349144+0000","last_unstale":"2024-09-20T22:59:14.016057+0000","last_undegraded":"2024-09-20T22:59:14.016057+0000","last_fullsized":"2024-09-20T22:59:14.016057+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:58:57.350319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:58:57.350319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T23:00:22.329053+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":2040,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:13.390390+0000","last_change":"2024-09-20T22:58:36.156131+0000","last_active":"2024-09-20T22:59:13.390390+0000","last_peered":"2024-09-20T22:59:13.390390+0000","last_clean":"2024-09-20T22:59:13.390390+0000","last_became_active":"2024-09-20T22:57:49.410743+0000","last_became_peered":"2024-09-20T22:57:49.410743+0000","last_unstale":"2024-09-20T22:59:13.390390+0000","last_undegraded":"2024-09-20T22:59:13.390390+0000","last_fullsized":"2024-09-20T22:59:13.390390+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:58:36.156047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:58:36.156047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:47.621359+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"77'128","reported_seq":2046,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:14.160023+0000","last_change":"2024-09-20T22:58:38.128312+0000","last_active":"2024-09-20T22:59:14.160023+0000","last_peered":"2024-09-20T22:59:14.160023+0000","last_clean":"2024-09-20T22:59:14.160023+0000","last_became_active":"2024-09-20T22:58:31.410332+0000","last_became_peered":"2024-09-20T22:58:31.410332+0000","last_unstale":"2024-09-20T22:59:14.160023+0000","last_undegraded":"2024-09-20T22:59:14.160023+0000","last_fullsized":"2024-09-20T22:59:14.160023+0000","mapping_epoch":1011,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":1012,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:58:38.128211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:58:38.128211+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T23:00:03.049766+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"877'25","reported_seq":626,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:13.390789+0000","last_change":"2024-09-20T22:58:50.255914+0000","last_active":"2024-09-20T22:59:13.390789+0000","last_peered":"2024-09-20T22:59:13.390789+0000","last_clean":"2024-09-20T22:59:13.390789+0000","last_became_active":"2024-09-20T22:58:48.609782+0000","last_became_peered":"2024-09-20T22:58:48.609782+0000","last_unstale":"2024-09-20T22:59:13.390789+0000","last_undegraded":"2024-09-20T22:59:13.390789+0000","last_fullsized":"2024-09-20T22:59:13.390789+0000","mapping_epoch":1022,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1023,"parent":"0.0","parent_split_bits":0,"last_scrub":"877'25","last_scrub_stamp":"2024-09-20T22:58:50.255758+0000","last_deep_scrub":"877'25","last_deep_scrub_stamp":"2024-09-20T22:55:57.521557+0000","last_clean_scrub_stamp":"2024-09-20T22:58:50.255758+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:59.724961+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":28,"num_read_kb":23,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":1897,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:13.486292+0000","last_change":"2024-09-20T22:59:11.061184+0000","last_active":"2024-09-20T22:59:13.486292+0000","last_peered":"2024-09-20T22:59:13.486292+0000","last_clean":"2024-09-20T22:59:13.486292+0000","last_became_active":"2024-09-20T22:57:49.432735+0000","last_became_peered":"2024-09-20T22:57:49.432735+0000","last_unstale":"2024-09-20T22:59:13.486292+0000","last_undegraded":"2024-09-20T22:59:13.486292+0000","last_fullsized":"2024-09-20T22:59:13.486292+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:59:11.061060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:55:58.553666+0000","last_clean_scrub_stamp":"2024-09-20T22:59:11.061060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T23:00:32.245416+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"1009'11","reported_seq":2267,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:23.399593+0000","last_change":"2024-09-20T22:59:11.140265+0000","last_active":"2024-09-20T22:59:23.399593+0000","last_peered":"2024-09-20T22:59:23.399593+0000","last_clean":"2024-09-20T22:59:23.399593+0000","last_became_active":"2024-09-20T22:58:28.415972+0000","last_became_peered":"2024-09-20T22:58:28.415972+0000","last_unstale":"2024-09-20T22:59:23.399593+0000","last_undegraded":"2024-09-20T22:59:23.399593+0000","last_fullsized":"2024-09-20T22:59:23.399593+0000","mapping_epoch":1008,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":1009,"parent":"0.0","parent_split_bits":0,"last_scrub":"1009'11","last_scrub_stamp":"2024-09-20T22:59:11.140112+0000","last_deep_scrub":"1009'11","last_deep_scrub_stamp":"2024-09-20T22:59:11.140112+0000","last_clean_scrub_stamp":"2024-09-20T22:59:11.140112+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T23:00:25.669832+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":262,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:13.390779+0000","last_change":"2024-09-20T22:58:48.609604+0000","last_active":"2024-09-20T22:59:13.390779+0000","last_peered":"2024-09-20T22:59:13.390779+0000","last_clean":"2024-09-20T22:59:13.390779+0000","last_became_active":"2024-09-20T22:58:48.609096+0000","last_became_peered":"2024-09-20T22:58:48.609096+0000","last_unstale":"2024-09-20T22:59:13.390779+0000","last_undegraded":"2024-09-20T22:59:13.390779+0000","last_fullsized":"2024-09-20T22:59:13.390779+0000","mapping_epoch":1022,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":1023,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:58:47.501202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:58:47.501202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:49.521274+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"625'15","reported_seq":1417,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:13.390233+0000","last_change":"2024-09-20T22:58:24.514509+0000","last_active":"2024-09-20T22:59:13.390233+0000","last_peered":"2024-09-20T22:59:13.390233+0000","last_clean":"2024-09-20T22:59:13.390233+0000","last_became_active":"2024-09-20T22:54:17.340998+0000","last_became_peered":"2024-09-20T22:54:17.340998+0000","last_unstale":"2024-09-20T22:59:13.390233+0000","last_undegraded":"2024-09-20T22:59:13.390233+0000","last_fullsized":"2024-09-20T22:59:13.390233+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'15","last_scrub_stamp":"2024-09-20T22:58:24.514380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:58:24.514380+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:45.247411+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":2016,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:14.015276+0000","last_change":"2024-09-20T22:58:55.026655+0000","last_active":"2024-09-20T22:59:14.015276+0000","last_peered":"2024-09-20T22:59:14.015276+0000","last_clean":"2024-09-20T22:59:14.015276+0000","last_became_active":"2024-09-20T22:57:49.425707+0000","last_became_peered":"2024-09-20T22:57:49.425707+0000","last_unstale":"2024-09-20T22:59:14.015276+0000","last_undegraded":"2024-09-20T22:59:14.015276+0000","last_fullsized":"2024-09-20T22:59:14.015276+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:58:55.026573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:58:55.026573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T23:00:20.516790+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":2049,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:13.391278+0000","last_change":"2024-09-20T22:58:56.209343+0000","last_active":"2024-09-20T22:59:13.391278+0000","last_peered":"2024-09-20T22:59:13.391278+0000","last_clean":"2024-09-20T22:59:13.391278+0000","last_became_active":"2024-09-20T22:57:49.411210+0000","last_became_peered":"2024-09-20T22:57:49.411210+0000","last_unstale":"2024-09-20T22:59:13.391278+0000","last_undegraded":"2024-09-20T22:59:13.391278+0000","last_fullsized":"2024-09-20T22:59:13.391278+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:58:56.209272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:58:56.209272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T23:00:11.157085+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":1371,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:13.389616+0000","last_change":"2024-09-20T22:58:43.672949+0000","last_active":"2024-09-20T22:59:13.389616+0000","last_peered":"2024-09-20T22:59:13.389616+0000","last_clean":"2024-09-20T22:59:13.389616+0000","last_became_active":"2024-09-20T22:58:28.404456+0000","last_became_peered":"2024-09-20T22:58:28.404456+0000","last_unstale":"2024-09-20T22:59:13.389616+0000","last_undegraded":"2024-09-20T22:59:13.389616+0000","last_fullsized":"2024-09-20T22:59:13.389616+0000","mapping_epoch":1008,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":1009,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:58:43.672863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:58:43.672863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:48.874288+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"625'44","reported_seq":2194,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:13.746586+0000","last_change":"2024-09-20T22:58:43.553049+0000","last_active":"2024-09-20T22:59:13.746586+0000","last_peered":"2024-09-20T22:59:13.746586+0000","last_clean":"2024-09-20T22:59:13.746586+0000","last_became_active":"2024-09-20T22:58:28.399556+0000","last_became_peered":"2024-09-20T22:58:28.399556+0000","last_unstale":"2024-09-20T22:59:13.746586+0000","last_undegraded":"2024-09-20T22:59:13.746586+0000","last_fullsized":"2024-09-20T22:59:13.746586+0000","mapping_epoch":1008,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1009,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'44","last_scrub_stamp":"2024-09-20T22:58:43.552928+0000","last_deep_scrub":"113'28","last_deep_scrub_stamp":"2024-09-20T22:43:12.725900+0000","last_clean_scrub_stamp":"2024-09-20T22:58:43.552928+0000","objects_scrubbed":9,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T23:00:08.433363+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":65,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:14.159848+0000","last_change":"2024-09-20T22:58:28.419770+0000","last_active":"2024-09-20T22:59:14.159848+0000","last_peered":"2024-09-20T22:59:14.159848+0000","last_clean":"2024-09-20T22:59:14.159848+0000","last_became_active":"2024-09-20T22:58:28.419355+0000","last_became_peered":"2024-09-20T22:58:28.419355+0000","last_unstale":"2024-09-20T22:59:14.159848+0000","last_undegraded":"2024-09-20T22:59:14.159848+0000","last_fullsized":"2024-09-20T22:59:14.159848+0000","mapping_epoch":1008,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1009,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:58:21.787147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:58:21.787147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:41.463239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":2027,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:13.389844+0000","last_change":"2024-09-20T22:58:26.172414+0000","last_active":"2024-09-20T22:59:13.389844+0000","last_peered":"2024-09-20T22:59:13.389844+0000","last_clean":"2024-09-20T22:59:13.389844+0000","last_became_active":"2024-09-20T22:47:49.223220+0000","last_became_peered":"2024-09-20T22:47:49.223220+0000","last_unstale":"2024-09-20T22:59:13.389844+0000","last_undegraded":"2024-09-20T22:59:13.389844+0000","last_fullsized":"2024-09-20T22:59:13.389844+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:58:26.172290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:54:21.496068+0000","last_clean_scrub_stamp":"2024-09-20T22:58:26.172290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:38.945178+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":73,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:14.159219+0000","last_change":"2024-09-20T22:58:32.148797+0000","last_active":"2024-09-20T22:59:14.159219+0000","last_peered":"2024-09-20T22:59:14.159219+0000","last_clean":"2024-09-20T22:59:14.159219+0000","last_became_active":"2024-09-20T22:58:28.407140+0000","last_became_peered":"2024-09-20T22:58:28.407140+0000","last_unstale":"2024-09-20T22:59:14.159219+0000","last_undegraded":"2024-09-20T22:59:14.159219+0000","last_fullsized":"2024-09-20T22:59:14.159219+0000","mapping_epoch":1008,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":1009,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:58:32.148715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:55:54.870627+0000","last_clean_scrub_stamp":"2024-09-20T22:58:32.148715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:54.796373+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"686'23","reported_seq":1013,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:13.390205+0000","last_change":"2024-09-20T22:59:13.118143+0000","last_active":"2024-09-20T22:59:13.390205+0000","last_peered":"2024-09-20T22:59:13.390205+0000","last_clean":"2024-09-20T22:59:13.390205+0000","last_became_active":"2024-09-20T22:58:48.608668+0000","last_became_peered":"2024-09-20T22:58:48.608668+0000","last_unstale":"2024-09-20T22:59:13.390205+0000","last_undegraded":"2024-09-20T22:59:13.390205+0000","last_fullsized":"2024-09-20T22:59:13.390205+0000","mapping_epoch":1022,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1023,"parent":"0.0","parent_split_bits":0,"last_scrub":"686'23","last_scrub_stamp":"2024-09-20T22:59:13.118048+0000","last_deep_scrub":"686'23","last_deep_scrub_stamp":"2024-09-20T22:52:46.768812+0000","last_clean_scrub_stamp":"2024-09-20T22:59:13.118048+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T23:00:25.994963+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":21,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":1100,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":73,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:24.562635+0000","last_change":"2024-09-20T22:59:24.562657+0000","last_active":"2024-09-20T22:59:24.562635+0000","last_peered":"2024-09-20T22:59:24.562635+0000","last_clean":"2024-09-20T22:59:24.562635+0000","last_became_active":"2024-09-20T22:58:28.411334+0000","last_became_peered":"2024-09-20T22:58:28.411334+0000","last_unstale":"2024-09-20T22:59:24.562635+0000","last_undegraded":"2024-09-20T22:59:24.562635+0000","last_fullsized":"2024-09-20T22:59:24.562635+0000","mapping_epoch":1008,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1009,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:59:24.562570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:46:52.465940+0000","last_clean_scrub_stamp":"2024-09-20T22:59:24.562570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T23:00:46.044367+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":2048,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:13.391015+0000","last_change":"2024-09-20T22:58:40.224975+0000","last_active":"2024-09-20T22:59:13.391015+0000","last_peered":"2024-09-20T22:59:13.391015+0000","last_clean":"2024-09-20T22:59:13.391015+0000","last_became_active":"2024-09-20T22:54:17.340269+0000","last_became_peered":"2024-09-20T22:54:17.340269+0000","last_unstale":"2024-09-20T22:59:13.391015+0000","last_undegraded":"2024-09-20T22:59:13.391015+0000","last_fullsized":"2024-09-20T22:59:13.391015+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:58:40.224818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:53:07.104891+0000","last_clean_scrub_stamp":"2024-09-20T22:58:40.224818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T23:00:06.212292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":1897,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:13.486410+0000","last_change":"2024-09-20T22:58:31.875658+0000","last_active":"2024-09-20T22:59:13.486410+0000","last_peered":"2024-09-20T22:59:13.486410+0000","last_clean":"2024-09-20T22:59:13.486410+0000","last_became_active":"2024-09-20T22:58:28.413507+0000","last_became_peered":"2024-09-20T22:58:28.413507+0000","last_unstale":"2024-09-20T22:59:13.486410+0000","last_undegraded":"2024-09-20T22:59:13.486410+0000","last_fullsized":"2024-09-20T22:59:13.486410+0000","mapping_epoch":1008,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":1009,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:58:31.875534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:55.170379+0000","last_clean_scrub_stamp":"2024-09-20T22:58:31.875534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:51.255952+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"625'27","reported_seq":2091,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:14.014497+0000","last_change":"2024-09-20T22:59:02.973258+0000","last_active":"2024-09-20T22:59:14.014497+0000","last_peered":"2024-09-20T22:59:14.014497+0000","last_clean":"2024-09-20T22:59:14.014497+0000","last_became_active":"2024-09-20T22:58:28.405911+0000","last_became_peered":"2024-09-20T22:58:28.405911+0000","last_unstale":"2024-09-20T22:59:14.014497+0000","last_undegraded":"2024-09-20T22:59:14.014497+0000","last_fullsized":"2024-09-20T22:59:14.014497+0000","mapping_epoch":1008,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1009,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'27","last_scrub_stamp":"2024-09-20T22:59:02.973154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:59:02.973154+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T23:00:16.168389+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'2","reported_seq":71,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:14.160221+0000","last_change":"2024-09-20T22:58:28.439602+0000","last_active":"2024-09-20T22:59:14.160221+0000","last_peered":"2024-09-20T22:59:14.160221+0000","last_clean":"2024-09-20T22:59:14.160221+0000","last_became_active":"2024-09-20T22:58:28.409991+0000","last_became_peered":"2024-09-20T22:58:28.409991+0000","last_unstale":"2024-09-20T22:59:14.160221+0000","last_undegraded":"2024-09-20T22:59:14.160221+0000","last_fullsized":"2024-09-20T22:59:14.160221+0000","mapping_epoch":1008,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1009,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-09-20T22:58:20.423976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:58:20.423976+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:43.884021+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"1009'12","reported_seq":90,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:23.399996+0000","last_change":"2024-09-20T22:59:04.289724+0000","last_active":"2024-09-20T22:59:23.399996+0000","last_peered":"2024-09-20T22:59:23.399996+0000","last_clean":"2024-09-20T22:59:23.399996+0000","last_became_active":"2024-09-20T22:58:28.410156+0000","last_became_peered":"2024-09-20T22:58:28.410156+0000","last_unstale":"2024-09-20T22:59:23.399996+0000","last_undegraded":"2024-09-20T22:59:23.399996+0000","last_fullsized":"2024-09-20T22:59:23.399996+0000","mapping_epoch":1008,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":1009,"parent":"0.0","parent_split_bits":0,"last_scrub":"1009'12","last_scrub_stamp":"2024-09-20T22:59:04.289629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:59:04.289629+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T23:00:19.811072+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":2008,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:14.014695+0000","last_change":"2024-09-20T22:59:08.015203+0000","last_active":"2024-09-20T22:59:14.014695+0000","last_peered":"2024-09-20T22:59:14.014695+0000","last_clean":"2024-09-20T22:59:14.014695+0000","last_became_active":"2024-09-20T22:47:49.227569+0000","last_became_peered":"2024-09-20T22:47:49.227569+0000","last_unstale":"2024-09-20T22:59:14.014695+0000","last_undegraded":"2024-09-20T22:59:14.014695+0000","last_fullsized":"2024-09-20T22:59:14.014695+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:59:08.015117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:59:08.015117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T23:00:22.604964+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":2049,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:18.560067+0000","last_change":"2024-09-20T22:59:18.560091+0000","last_active":"2024-09-20T22:59:18.560067+0000","last_peered":"2024-09-20T22:59:18.560067+0000","last_clean":"2024-09-20T22:59:18.560067+0000","last_became_active":"2024-09-20T22:47:49.213292+0000","last_became_peered":"2024-09-20T22:47:49.213292+0000","last_unstale":"2024-09-20T22:59:18.560067+0000","last_undegraded":"2024-09-20T22:59:18.560067+0000","last_fullsized":"2024-09-20T22:59:18.560067+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:59:18.560002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:59:18.560002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T23:00:29.853798+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"625'12","reported_seq":1963,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:13.486779+0000","last_change":"2024-09-20T22:58:28.449237+0000","last_active":"2024-09-20T22:59:13.486779+0000","last_peered":"2024-09-20T22:59:13.486779+0000","last_clean":"2024-09-20T22:59:13.486779+0000","last_became_active":"2024-09-20T22:58:28.408845+0000","last_became_peered":"2024-09-20T22:58:28.408845+0000","last_unstale":"2024-09-20T22:59:13.486779+0000","last_undegraded":"2024-09-20T22:59:13.486779+0000","last_fullsized":"2024-09-20T22:59:13.486779+0000","mapping_epoch":1008,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1009,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'12","last_scrub_stamp":"2024-09-20T22:58:00.707335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:58:00.707335+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:22.874486+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":2020,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:14.015439+0000","last_change":"2024-09-20T22:58:22.783603+0000","last_active":"2024-09-20T22:59:14.015439+0000","last_peered":"2024-09-20T22:59:14.015439+0000","last_clean":"2024-09-20T22:59:14.015439+0000","last_became_active":"2024-09-20T22:57:49.408529+0000","last_became_peered":"2024-09-20T22:57:49.408529+0000","last_unstale":"2024-09-20T22:59:14.015439+0000","last_undegraded":"2024-09-20T22:59:14.015439+0000","last_fullsized":"2024-09-20T22:59:14.015439+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:58:22.783512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:09.705632+0000","last_clean_scrub_stamp":"2024-09-20T22:58:22.783512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:45.356564+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"74'1","reported_seq":77,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:14.160035+0000","last_change":"2024-09-20T22:58:49.165424+0000","last_active":"2024-09-20T22:59:14.160035+0000","last_peered":"2024-09-20T22:59:14.160035+0000","last_clean":"2024-09-20T22:59:14.160035+0000","last_became_active":"2024-09-20T22:58:28.411228+0000","last_became_peered":"2024-09-20T22:58:28.411228+0000","last_unstale":"2024-09-20T22:59:14.160035+0000","last_undegraded":"2024-09-20T22:59:14.160035+0000","last_fullsized":"2024-09-20T22:59:14.160035+0000","mapping_epoch":1008,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":1009,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-09-20T22:58:49.165295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:58:49.165295+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T23:00:04.313986+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"625'15","reported_seq":1409,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:13.390868+0000","last_change":"2024-09-20T22:58:28.439708+0000","last_active":"2024-09-20T22:59:13.390868+0000","last_peered":"2024-09-20T22:59:13.390868+0000","last_clean":"2024-09-20T22:59:13.390868+0000","last_became_active":"2024-09-20T22:58:28.405850+0000","last_became_peered":"2024-09-20T22:58:28.405850+0000","last_unstale":"2024-09-20T22:59:13.390868+0000","last_undegraded":"2024-09-20T22:59:13.390868+0000","last_fullsized":"2024-09-20T22:59:13.390868+0000","mapping_epoch":1008,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1009,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'15","last_scrub_stamp":"2024-09-20T22:58:19.403482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:58:19.403482+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:33.775613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":73,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:14.160206+0000","last_change":"2024-09-20T22:58:34.150809+0000","last_active":"2024-09-20T22:59:14.160206+0000","last_peered":"2024-09-20T22:59:14.160206+0000","last_clean":"2024-09-20T22:59:14.160206+0000","last_became_active":"2024-09-20T22:58:28.410707+0000","last_became_peered":"2024-09-20T22:58:28.410707+0000","last_unstale":"2024-09-20T22:59:14.160206+0000","last_undegraded":"2024-09-20T22:59:14.160206+0000","last_fullsized":"2024-09-20T22:59:14.160206+0000","mapping_epoch":1008,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1009,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:58:34.150733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:58:34.150733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:35.124109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":1909,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:14.161396+0000","last_change":"2024-09-20T22:59:14.161420+0000","last_active":"2024-09-20T22:59:14.161396+0000","last_peered":"2024-09-20T22:59:14.161396+0000","last_clean":"2024-09-20T22:59:14.161396+0000","last_became_active":"2024-09-20T22:58:28.406337+0000","last_became_peered":"2024-09-20T22:58:28.406337+0000","last_unstale":"2024-09-20T22:59:14.161396+0000","last_undegraded":"2024-09-20T22:59:14.161396+0000","last_fullsized":"2024-09-20T22:59:14.161396+0000","mapping_epoch":1008,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":1009,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:59:14.161329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:59:14.161329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T23:00:16.069258+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":2055,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:13.391766+0000","last_change":"2024-09-20T22:58:37.129420+0000","last_active":"2024-09-20T22:59:13.391766+0000","last_peered":"2024-09-20T22:59:13.391766+0000","last_clean":"2024-09-20T22:59:13.391766+0000","last_became_active":"2024-09-20T22:58:28.408803+0000","last_became_peered":"2024-09-20T22:58:28.408803+0000","last_unstale":"2024-09-20T22:59:13.391766+0000","last_undegraded":"2024-09-20T22:59:13.391766+0000","last_fullsized":"2024-09-20T22:59:13.391766+0000","mapping_epoch":1008,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":1009,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:58:37.129346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:53:58.169894+0000","last_clean_scrub_stamp":"2024-09-20T22:58:37.129346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:55.758007+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"625'15","reported_seq":2069,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:14.015306+0000","last_change":"2024-09-20T22:58:45.916686+0000","last_active":"2024-09-20T22:59:14.015306+0000","last_peered":"2024-09-20T22:59:14.015306+0000","last_clean":"2024-09-20T22:59:14.015306+0000","last_became_active":"2024-09-20T22:58:28.406352+0000","last_became_peered":"2024-09-20T22:58:28.406352+0000","last_unstale":"2024-09-20T22:59:14.015306+0000","last_undegraded":"2024-09-20T22:59:14.015306+0000","last_fullsized":"2024-09-20T22:59:14.015306+0000","mapping_epoch":1008,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1009,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'15","last_scrub_stamp":"2024-09-20T22:58:45.916597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:58:45.916597+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:50.823985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":1379,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:13.389884+0000","last_change":"2024-09-20T22:58:35.606336+0000","last_active":"2024-09-20T22:59:13.389884+0000","last_peered":"2024-09-20T22:59:13.389884+0000","last_clean":"2024-09-20T22:59:13.389884+0000","last_became_active":"2024-09-20T22:58:28.405095+0000","last_became_peered":"2024-09-20T22:58:28.405095+0000","last_unstale":"2024-09-20T22:59:13.389884+0000","last_undegraded":"2024-09-20T22:59:13.389884+0000","last_fullsized":"2024-09-20T22:59:13.389884+0000","mapping_epoch":1008,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1009,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:58:35.606262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:58:35.606262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T23:00:01.674790+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"1009'12","reported_seq":2111,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:18.399554+0000","last_change":"2024-09-20T22:58:34.905724+0000","last_active":"2024-09-20T22:59:18.399554+0000","last_peered":"2024-09-20T22:59:18.399554+0000","last_clean":"2024-09-20T22:59:18.399554+0000","last_became_active":"2024-09-20T22:58:28.411994+0000","last_became_peered":"2024-09-20T22:58:28.411994+0000","last_unstale":"2024-09-20T22:59:18.399554+0000","last_undegraded":"2024-09-20T22:59:18.399554+0000","last_fullsized":"2024-09-20T22:59:18.399554+0000","mapping_epoch":1008,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":1009,"parent":"0.0","parent_split_bits":0,"last_scrub":"1009'12","last_scrub_stamp":"2024-09-20T22:58:34.905643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:58:34.905643+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:47.284009+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":932,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:13.391221+0000","last_change":"2024-09-20T22:59:10.134268+0000","last_active":"2024-09-20T22:59:13.391221+0000","last_peered":"2024-09-20T22:59:13.391221+0000","last_clean":"2024-09-20T22:59:13.391221+0000","last_became_active":"2024-09-20T22:58:48.610883+0000","last_became_peered":"2024-09-20T22:58:48.610883+0000","last_unstale":"2024-09-20T22:59:13.391221+0000","last_undegraded":"2024-09-20T22:59:13.391221+0000","last_fullsized":"2024-09-20T22:59:13.391221+0000","mapping_epoch":1022,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":1023,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:59:10.134177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:59:10.134177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T23:00:25.594502+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"625'20","reported_seq":1999,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:13.486280+0000","last_change":"2024-09-20T22:59:02.907018+0000","last_active":"2024-09-20T22:59:13.486280+0000","last_peered":"2024-09-20T22:59:13.486280+0000","last_clean":"2024-09-20T22:59:13.486280+0000","last_became_active":"2024-09-20T22:58:28.411505+0000","last_became_peered":"2024-09-20T22:58:28.411505+0000","last_unstale":"2024-09-20T22:59:13.486280+0000","last_undegraded":"2024-09-20T22:59:13.486280+0000","last_fullsized":"2024-09-20T22:59:13.486280+0000","mapping_epoch":1008,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1009,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'20","last_scrub_stamp":"2024-09-20T22:59:02.906940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:59:02.906940+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T23:00:03.205418+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":65,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:14.159700+0000","last_change":"2024-09-20T22:58:28.410981+0000","last_active":"2024-09-20T22:59:14.159700+0000","last_peered":"2024-09-20T22:59:14.159700+0000","last_clean":"2024-09-20T22:59:14.159700+0000","last_became_active":"2024-09-20T22:58:28.410106+0000","last_became_peered":"2024-09-20T22:58:28.410106+0000","last_unstale":"2024-09-20T22:59:14.159700+0000","last_undegraded":"2024-09-20T22:59:14.159700+0000","last_fullsized":"2024-09-20T22:59:14.159700+0000","mapping_epoch":1008,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1009,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:58:18.200504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:58:18.200504+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:39.182436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"778'12","reported_seq":2274,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:23.399992+0000","last_change":"2024-09-20T22:59:16.941628+0000","last_active":"2024-09-20T22:59:23.399992+0000","last_peered":"2024-09-20T22:59:23.399992+0000","last_clean":"2024-09-20T22:59:23.399992+0000","last_became_active":"2024-09-20T22:54:17.341014+0000","last_became_peered":"2024-09-20T22:54:17.341014+0000","last_unstale":"2024-09-20T22:59:23.399992+0000","last_undegraded":"2024-09-20T22:59:23.399992+0000","last_fullsized":"2024-09-20T22:59:23.399992+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"778'12","last_scrub_stamp":"2024-09-20T22:59:16.941530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:59:16.941530+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T23:00:43.942982+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":1370,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:13.390741+0000","last_change":"2024-09-20T22:58:52.745765+0000","last_active":"2024-09-20T22:59:13.390741+0000","last_peered":"2024-09-20T22:59:13.390741+0000","last_clean":"2024-09-20T22:59:13.390741+0000","last_became_active":"2024-09-20T22:58:28.405530+0000","last_became_peered":"2024-09-20T22:58:28.405530+0000","last_unstale":"2024-09-20T22:59:13.390741+0000","last_undegraded":"2024-09-20T22:59:13.390741+0000","last_fullsized":"2024-09-20T22:59:13.390741+0000","mapping_epoch":1008,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":1009,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:58:52.745690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:50:21.853070+0000","last_clean_scrub_stamp":"2024-09-20T22:58:52.745690+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T23:00:11.125777+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"625'19","reported_seq":1001,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:14.154930+0000","last_change":"2024-09-20T22:59:14.154999+0000","last_active":"2024-09-20T22:59:14.154930+0000","last_peered":"2024-09-20T22:59:14.154930+0000","last_clean":"2024-09-20T22:59:14.154930+0000","last_became_active":"2024-09-20T22:58:48.609950+0000","last_became_peered":"2024-09-20T22:58:48.609950+0000","last_unstale":"2024-09-20T22:59:14.154930+0000","last_undegraded":"2024-09-20T22:59:14.154930+0000","last_fullsized":"2024-09-20T22:59:14.154930+0000","mapping_epoch":1022,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1023,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'19","last_scrub_stamp":"2024-09-20T22:59:14.154830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:59:14.154830+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T23:00:41.980251+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":941,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:13.390658+0000","last_change":"2024-09-20T22:59:00.174402+0000","last_active":"2024-09-20T22:59:13.390658+0000","last_peered":"2024-09-20T22:59:13.390658+0000","last_clean":"2024-09-20T22:59:13.390658+0000","last_became_active":"2024-09-20T22:58:49.164889+0000","last_became_peered":"2024-09-20T22:58:49.164889+0000","last_unstale":"2024-09-20T22:59:13.390658+0000","last_undegraded":"2024-09-20T22:59:13.390658+0000","last_fullsized":"2024-09-20T22:59:13.390658+0000","mapping_epoch":1022,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1023,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:59:00.174315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:52:47.717490+0000","last_clean_scrub_stamp":"2024-09-20T22:59:00.174315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T23:00:03.795895+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":562,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:14.015544+0000","last_change":"2024-09-20T22:59:13.421244+0000","last_active":"2024-09-20T22:59:14.015544+0000","last_peered":"2024-09-20T22:59:14.015544+0000","last_clean":"2024-09-20T22:59:14.015544+0000","last_became_active":"2024-09-20T22:57:49.398369+0000","last_became_peered":"2024-09-20T22:57:49.398369+0000","last_unstale":"2024-09-20T22:59:14.015544+0000","last_undegraded":"2024-09-20T22:59:14.015544+0000","last_fullsized":"2024-09-20T22:59:14.015544+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:59:13.421167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:59:13.421167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T23:00:40.031830+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":1837,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:17.221717+0000","last_change":"2024-09-20T22:59:17.221735+0000","last_active":"2024-09-20T22:59:17.221717+0000","last_peered":"2024-09-20T22:59:17.221717+0000","last_clean":"2024-09-20T22:59:17.221717+0000","last_became_active":"2024-09-20T22:57:49.413148+0000","last_became_peered":"2024-09-20T22:57:49.413148+0000","last_unstale":"2024-09-20T22:59:17.221717+0000","last_undegraded":"2024-09-20T22:59:17.221717+0000","last_fullsized":"2024-09-20T22:59:17.221717+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:59:17.221665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:56:22.490523+0000","last_clean_scrub_stamp":"2024-09-20T22:59:17.221665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T23:00:18.885974+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"625'19","reported_seq":555,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:14.015738+0000","last_change":"2024-09-20T22:58:03.394296+0000","last_active":"2024-09-20T22:59:14.015738+0000","last_peered":"2024-09-20T22:59:14.015738+0000","last_clean":"2024-09-20T22:59:14.015738+0000","last_became_active":"2024-09-20T22:54:17.340539+0000","last_became_peered":"2024-09-20T22:54:17.340539+0000","last_unstale":"2024-09-20T22:59:14.015738+0000","last_undegraded":"2024-09-20T22:59:14.015738+0000","last_fullsized":"2024-09-20T22:59:14.015738+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'19","last_scrub_stamp":"2024-09-20T22:58:03.394221+0000","last_deep_scrub":"404'13","last_deep_scrub_stamp":"2024-09-20T22:48:04.276282+0000","last_clean_scrub_stamp":"2024-09-20T22:58:03.394221+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:25.311781+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":73,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:14.160071+0000","last_change":"2024-09-20T22:58:29.123277+0000","last_active":"2024-09-20T22:59:14.160071+0000","last_peered":"2024-09-20T22:59:14.160071+0000","last_clean":"2024-09-20T22:59:14.160071+0000","last_became_active":"2024-09-20T22:58:28.412950+0000","last_became_peered":"2024-09-20T22:58:28.412950+0000","last_unstale":"2024-09-20T22:59:14.160071+0000","last_undegraded":"2024-09-20T22:59:14.160071+0000","last_fullsized":"2024-09-20T22:59:14.160071+0000","mapping_epoch":1008,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1009,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:58:29.123179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:31.138067+0000","last_clean_scrub_stamp":"2024-09-20T22:58:29.123179+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:53.882930+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":1843,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:13.486223+0000","last_change":"2024-09-20T22:58:03.746286+0000","last_active":"2024-09-20T22:59:13.486223+0000","last_peered":"2024-09-20T22:59:13.486223+0000","last_clean":"2024-09-20T22:59:13.486223+0000","last_became_active":"2024-09-20T22:57:49.432838+0000","last_became_peered":"2024-09-20T22:57:49.432838+0000","last_unstale":"2024-09-20T22:59:13.486223+0000","last_undegraded":"2024-09-20T22:59:13.486223+0000","last_fullsized":"2024-09-20T22:59:13.486223+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:58:03.746169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:58:03.746169+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:27.258169+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":1362,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:13.389527+0000","last_change":"2024-09-20T22:59:08.735055+0000","last_active":"2024-09-20T22:59:13.389527+0000","last_peered":"2024-09-20T22:59:13.389527+0000","last_clean":"2024-09-20T22:59:13.389527+0000","last_became_active":"2024-09-20T22:54:17.344021+0000","last_became_peered":"2024-09-20T22:54:17.344021+0000","last_unstale":"2024-09-20T22:59:13.389527+0000","last_undegraded":"2024-09-20T22:59:13.389527+0000","last_fullsized":"2024-09-20T22:59:13.389527+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:59:08.734987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:55:09.664839+0000","last_clean_scrub_stamp":"2024-09-20T22:59:08.734987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T23:00:15.341798+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"625'24","reported_seq":1553,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:20.928284+0000","last_change":"2024-09-20T22:59:20.928329+0000","last_active":"2024-09-20T22:59:20.928284+0000","last_peered":"2024-09-20T22:59:20.928284+0000","last_clean":"2024-09-20T22:59:20.928284+0000","last_became_active":"2024-09-20T22:58:55.393537+0000","last_became_peered":"2024-09-20T22:58:55.393537+0000","last_unstale":"2024-09-20T22:59:20.928284+0000","last_undegraded":"2024-09-20T22:59:20.928284+0000","last_fullsized":"2024-09-20T22:59:20.928284+0000","mapping_epoch":1025,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1026,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'24","last_scrub_stamp":"2024-09-20T22:59:20.928206+0000","last_deep_scrub":"470'16","last_deep_scrub_stamp":"2024-09-20T22:49:10.659868+0000","last_clean_scrub_stamp":"2024-09-20T22:59:20.928206+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T23:00:22.381114+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":2040,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:14.015443+0000","last_change":"2024-09-20T22:58:28.406826+0000","last_active":"2024-09-20T22:59:14.015443+0000","last_peered":"2024-09-20T22:59:14.015443+0000","last_clean":"2024-09-20T22:59:14.015443+0000","last_became_active":"2024-09-20T22:58:28.405836+0000","last_became_peered":"2024-09-20T22:58:28.405836+0000","last_unstale":"2024-09-20T22:59:14.015443+0000","last_undegraded":"2024-09-20T22:59:14.015443+0000","last_fullsized":"2024-09-20T22:59:14.015443+0000","mapping_epoch":1008,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1009,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:58:25.905810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:45:24.765414+0000","last_clean_scrub_stamp":"2024-09-20T22:58:25.905810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:54.228873+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":73,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:25.572688+0000","last_change":"2024-09-20T22:59:25.572714+0000","last_active":"2024-09-20T22:59:25.572688+0000","last_peered":"2024-09-20T22:59:25.572688+0000","last_clean":"2024-09-20T22:59:25.572688+0000","last_became_active":"2024-09-20T22:58:28.408967+0000","last_became_peered":"2024-09-20T22:58:28.408967+0000","last_unstale":"2024-09-20T22:59:25.572688+0000","last_undegraded":"2024-09-20T22:59:25.572688+0000","last_fullsized":"2024-09-20T22:59:25.572688+0000","mapping_epoch":1008,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":1009,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:59:25.572617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:50.174861+0000","last_clean_scrub_stamp":"2024-09-20T22:59:25.572617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T23:00:46.141169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":1835,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:13.486314+0000","last_change":"2024-09-20T22:58:42.917038+0000","last_active":"2024-09-20T22:59:13.486314+0000","last_peered":"2024-09-20T22:59:13.486314+0000","last_clean":"2024-09-20T22:59:13.486314+0000","last_became_active":"2024-09-20T22:57:49.410970+0000","last_became_peered":"2024-09-20T22:57:49.410970+0000","last_unstale":"2024-09-20T22:59:13.486314+0000","last_undegraded":"2024-09-20T22:59:13.486314+0000","last_fullsized":"2024-09-20T22:59:13.486314+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:58:42.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:58:42.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T23:00:12.008846+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"625'8","reported_seq":1907,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:26.303005+0000","last_change":"2024-09-20T22:59:26.303034+0000","last_active":"2024-09-20T22:59:26.303005+0000","last_peered":"2024-09-20T22:59:26.303005+0000","last_clean":"2024-09-20T22:59:26.303005+0000","last_became_active":"2024-09-20T22:57:49.401068+0000","last_became_peered":"2024-09-20T22:57:49.401068+0000","last_unstale":"2024-09-20T22:59:26.303005+0000","last_undegraded":"2024-09-20T22:59:26.303005+0000","last_fullsized":"2024-09-20T22:59:26.303005+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'8","last_scrub_stamp":"2024-09-20T22:59:26.302943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:59:26.302943+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T23:00:32.370010+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":1366,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:17.850321+0000","last_change":"2024-09-20T22:59:17.850356+0000","last_active":"2024-09-20T22:59:17.850321+0000","last_peered":"2024-09-20T22:59:17.850321+0000","last_clean":"2024-09-20T22:59:17.850321+0000","last_became_active":"2024-09-20T22:57:49.412134+0000","last_became_peered":"2024-09-20T22:57:49.412134+0000","last_unstale":"2024-09-20T22:59:17.850321+0000","last_undegraded":"2024-09-20T22:59:17.850321+0000","last_fullsized":"2024-09-20T22:59:17.850321+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:59:17.850257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:58:02.611308+0000","last_clean_scrub_stamp":"2024-09-20T22:59:17.850257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T23:00:40.535978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":2057,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:13.746345+0000","last_change":"2024-09-20T22:58:33.443155+0000","last_active":"2024-09-20T22:59:13.746345+0000","last_peered":"2024-09-20T22:59:13.746345+0000","last_clean":"2024-09-20T22:59:13.746345+0000","last_became_active":"2024-09-20T22:58:28.398589+0000","last_became_peered":"2024-09-20T22:58:28.398589+0000","last_unstale":"2024-09-20T22:59:13.746345+0000","last_undegraded":"2024-09-20T22:59:13.746345+0000","last_fullsized":"2024-09-20T22:59:13.746345+0000","mapping_epoch":1008,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":1009,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:58:33.443029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:58:33.443029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T23:00:01.673636+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":73,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:14.159343+0000","last_change":"2024-09-20T22:58:57.216383+0000","last_active":"2024-09-20T22:59:14.159343+0000","last_peered":"2024-09-20T22:59:14.159343+0000","last_clean":"2024-09-20T22:59:14.159343+0000","last_became_active":"2024-09-20T22:58:28.412797+0000","last_became_peered":"2024-09-20T22:58:28.412797+0000","last_unstale":"2024-09-20T22:59:14.159343+0000","last_undegraded":"2024-09-20T22:59:14.159343+0000","last_fullsized":"2024-09-20T22:59:14.159343+0000","mapping_epoch":1008,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":1009,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:58:57.216303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:55:08.681729+0000","last_clean_scrub_stamp":"2024-09-20T22:58:57.216303+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T23:00:13.723641+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":553,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:14.016356+0000","last_change":"2024-09-20T22:58:58.331621+0000","last_active":"2024-09-20T22:59:14.016356+0000","last_peered":"2024-09-20T22:59:14.016356+0000","last_clean":"2024-09-20T22:59:14.016356+0000","last_became_active":"2024-09-20T22:54:17.348206+0000","last_became_peered":"2024-09-20T22:54:17.348206+0000","last_unstale":"2024-09-20T22:59:14.016356+0000","last_undegraded":"2024-09-20T22:59:14.016356+0000","last_fullsized":"2024-09-20T22:59:14.016356+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:58:58.331549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:58:58.331549+0000","last_clean_scrub_stamp":"2024-09-20T22:58:58.331549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:58.949836+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":1865,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:13.486146+0000","last_change":"2024-09-20T22:58:28.880848+0000","last_active":"2024-09-20T22:59:13.486146+0000","last_peered":"2024-09-20T22:59:13.486146+0000","last_clean":"2024-09-20T22:59:13.486146+0000","last_became_active":"2024-09-20T22:57:49.407194+0000","last_became_peered":"2024-09-20T22:57:49.407194+0000","last_unstale":"2024-09-20T22:59:13.486146+0000","last_undegraded":"2024-09-20T22:59:13.486146+0000","last_fullsized":"2024-09-20T22:59:13.486146+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:58:28.880776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:58:28.880776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:31.422169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"1036'24","reported_seq":2109,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:17.594603+0000","last_change":"2024-09-20T22:59:17.594640+0000","last_active":"2024-09-20T22:59:17.594603+0000","last_peered":"2024-09-20T22:59:17.594603+0000","last_clean":"2024-09-20T22:59:17.594603+0000","last_became_active":"2024-09-20T22:54:17.342236+0000","last_became_peered":"2024-09-20T22:54:17.342236+0000","last_unstale":"2024-09-20T22:59:17.594603+0000","last_undegraded":"2024-09-20T22:59:17.594603+0000","last_fullsized":"2024-09-20T22:59:17.594603+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"1036'24","last_scrub_stamp":"2024-09-20T22:59:17.594541+0000","last_deep_scrub":"1036'24","last_deep_scrub_stamp":"2024-09-20T22:59:17.594541+0000","last_clean_scrub_stamp":"2024-09-20T22:59:17.594541+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T23:00:36.558346+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":1362,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:13.390777+0000","last_change":"2024-09-20T22:58:57.634893+0000","last_active":"2024-09-20T22:59:13.390777+0000","last_peered":"2024-09-20T22:59:13.390777+0000","last_clean":"2024-09-20T22:59:13.390777+0000","last_became_active":"2024-09-20T22:58:28.405330+0000","last_became_peered":"2024-09-20T22:58:28.405330+0000","last_unstale":"2024-09-20T22:59:13.390777+0000","last_undegraded":"2024-09-20T22:59:13.390777+0000","last_fullsized":"2024-09-20T22:59:13.390777+0000","mapping_epoch":1008,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":1009,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:58:57.634790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:54:43.813540+0000","last_clean_scrub_stamp":"2024-09-20T22:58:57.634790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T23:00:01.656707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":429,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":105,"ondisk_log_size":105,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1101,"num_read_kb":858,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":514,"num_bytes_recovered":11678,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":785,"ondisk_log_size":785,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":1126,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":422,"num_read_kb":417,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":655360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":385,"ondisk_log_size":385,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070683,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":91664,"kb_used_data":10560,"kb_used_omap":1,"kb_used_meta":81086,"kb_avail":93653488,"statfs":{"total":95995035648,"available":95901171712,"internally_reserved":0,"allocated":10813440,"data_stored":6734015,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":83032522},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430433,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":99660,"kb_used_data":10236,"kb_used_omap":1,"kb_used_meta":89406,"kb_avail":93645492,"statfs":{"total":95995035648,"available":95892983808,"internally_reserved":0,"allocated":10481664,"data_stored":6406573,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":91552198},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798692070,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":86440,"kb_used_data":10840,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93658712,"statfs":{"total":95995035648,"available":95906521088,"internally_reserved":0,"allocated":11100160,"data_stored":6998191,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986411,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":59492,"kb_used_data":10260,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":93685660,"statfs":{"total":95995035648,"available":95934115840,"internally_reserved":0,"allocated":10506240,"data_stored":6406185,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":988,"seq":4243427688470,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":45604,"kb_used_data":10836,"kb_used_omap":148,"kb_used_meta":34603,"kb_avail":93699548,"statfs":{"total":95995035648,"available":95948337152,"internally_reserved":0,"allocated":11096064,"data_stored":6997489,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":151956,"internal_metadata":35434092},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411574,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":95780,"kb_used_data":10512,"kb_used_omap":1,"kb_used_meta":85246,"kb_avail":93649372,"statfs":{"total":95995035648,"available":95896956928,"internally_reserved":0,"allocated":10764288,"data_stored":6669499,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":87292357},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575099,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":85872,"kb_used_data":10272,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93659280,"statfs":{"total":95995035648,"available":95907102720,"internally_reserved":0,"allocated":10518528,"data_stored":6406884,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738624,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":99680,"kb_used_data":10256,"kb_used_omap":1,"kb_used_meta":89406,"kb_avail":93645472,"statfs":{"total":95995035648,"available":95892963328,"internally_reserved":0,"allocated":10502144,"data_stored":6406797,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":91552197},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-20T22:59:31.538 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph pg dump --format=json 2024-09-20T22:59:31.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:31 smithi154 ceph-mon[33264]: from='client.14556 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-20T22:59:31.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:31 smithi154 ceph-mon[33264]: pgmap v1483: 142 pgs: 142 active+clean; 583 KiB data, 649 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-20T22:59:31.784 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:59:32.430 INFO:teuthology.orchestra.run.smithi084.stdout: 2024-09-20T22:59:32.430 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:32 smithi084 ceph-mon[40024]: 1.a scrub starts 2024-09-20T22:59:32.430 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:32 smithi084 ceph-mon[40024]: 1.a scrub ok 2024-09-20T22:59:32.430 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:32 smithi084 ceph-mon[40024]: from='client.17916 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T22:59:32.431 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:32 smithi084 ceph-mon[40024]: 4.12 scrub starts 2024-09-20T22:59:32.431 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:32 smithi084 ceph-mon[40024]: 4.12 scrub ok 2024-09-20T22:59:32.431 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:32 smithi084 ceph-mon[40024]: 4.1c scrub starts 2024-09-20T22:59:32.431 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:32 smithi084 ceph-mon[40024]: 4.1c scrub ok 2024-09-20T22:59:32.431 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:32 smithi084 ceph-mon[29774]: 1.a scrub starts 2024-09-20T22:59:32.431 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:32 smithi084 ceph-mon[29774]: 1.a scrub ok 2024-09-20T22:59:32.431 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:32 smithi084 ceph-mon[29774]: from='client.17916 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T22:59:32.432 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:32 smithi084 ceph-mon[29774]: 4.12 scrub starts 2024-09-20T22:59:32.432 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:32 smithi084 ceph-mon[29774]: 4.12 scrub ok 2024-09-20T22:59:32.432 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:32 smithi084 ceph-mon[29774]: 4.1c scrub starts 2024-09-20T22:59:32.432 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:32 smithi084 ceph-mon[29774]: 4.1c scrub ok 2024-09-20T22:59:32.436 INFO:teuthology.orchestra.run.smithi084.stderr:dumped all 2024-09-20T22:59:32.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:32 smithi154 ceph-mon[33264]: 1.a scrub starts 2024-09-20T22:59:32.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:32 smithi154 ceph-mon[33264]: 1.a scrub ok 2024-09-20T22:59:32.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:32 smithi154 ceph-mon[33264]: from='client.17916 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T22:59:32.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:32 smithi154 ceph-mon[33264]: 4.12 scrub starts 2024-09-20T22:59:32.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:32 smithi154 ceph-mon[33264]: 4.12 scrub ok 2024-09-20T22:59:32.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:32 smithi154 ceph-mon[33264]: 4.1c scrub starts 2024-09-20T22:59:32.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:32 smithi154 ceph-mon[33264]: 4.1c scrub ok 2024-09-20T22:59:33.191 INFO:teuthology.orchestra.run.smithi084.stdout:{"pg_ready":true,"pg_map":{"version":1484,"stamp":"2024-09-20T22:59:31.927273+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596507,"num_objects":199,"num_object_clones":0,"num_object_copies":597,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":199,"num_whiteouts":0,"num_read":1540,"num_read_kb":1292,"num_write":722,"num_write_kb":45,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":564,"num_bytes_recovered":669409,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1296,"ondisk_log_size":1296,"up":426,"acting":426,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":426,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":664192,"kb_used_data":83772,"kb_used_omap":159,"kb_used_meta":580128,"kb_avail":749297024,"statfs":{"total":767960285184,"available":767280152576,"internally_reserved":0,"allocated":85782528,"data_stored":53025633,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":163115,"internal_metadata":594051797},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003862"},"pg_stats":[{"pgid":"3.1f","version":"86'2","reported_seq":1354,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:13.389990+0000","last_change":"2024-09-20T22:58:47.675255+0000","last_active":"2024-09-20T22:59:13.389990+0000","last_peered":"2024-09-20T22:59:13.389990+0000","last_clean":"2024-09-20T22:59:13.389990+0000","last_became_active":"2024-09-20T22:47:49.222114+0000","last_became_peered":"2024-09-20T22:47:49.222114+0000","last_unstale":"2024-09-20T22:59:13.389990+0000","last_undegraded":"2024-09-20T22:59:13.389990+0000","last_fullsized":"2024-09-20T22:59:13.389990+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'2","last_scrub_stamp":"2024-09-20T22:58:47.675170+0000","last_deep_scrub":"86'2","last_deep_scrub_stamp":"2024-09-20T22:56:47.723216+0000","last_clean_scrub_stamp":"2024-09-20T22:58:47.675170+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T23:00:17.341427+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":436,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"625'15","reported_seq":983,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:13.390022+0000","last_change":"2024-09-20T22:58:48.609445+0000","last_active":"2024-09-20T22:59:13.390022+0000","last_peered":"2024-09-20T22:59:13.390022+0000","last_clean":"2024-09-20T22:59:13.390022+0000","last_became_active":"2024-09-20T22:58:48.609061+0000","last_became_peered":"2024-09-20T22:58:48.609061+0000","last_unstale":"2024-09-20T22:59:13.390022+0000","last_undegraded":"2024-09-20T22:59:13.390022+0000","last_fullsized":"2024-09-20T22:59:13.390022+0000","mapping_epoch":1022,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1023,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'15","last_scrub_stamp":"2024-09-20T22:58:29.448675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:58:29.448675+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:37.818163+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":557,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:14.015532+0000","last_change":"2024-09-20T22:58:28.399056+0000","last_active":"2024-09-20T22:59:14.015532+0000","last_peered":"2024-09-20T22:59:14.015532+0000","last_clean":"2024-09-20T22:59:14.015532+0000","last_became_active":"2024-09-20T22:58:28.398686+0000","last_became_peered":"2024-09-20T22:58:28.398686+0000","last_unstale":"2024-09-20T22:59:14.015532+0000","last_undegraded":"2024-09-20T22:59:14.015532+0000","last_fullsized":"2024-09-20T22:59:14.015532+0000","mapping_epoch":1008,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":1009,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:58:21.323903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:58:21.323903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:49.625461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":952,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:13.390376+0000","last_change":"2024-09-20T22:58:56.218532+0000","last_active":"2024-09-20T22:59:13.390376+0000","last_peered":"2024-09-20T22:59:13.390376+0000","last_clean":"2024-09-20T22:59:13.390376+0000","last_became_active":"2024-09-20T22:58:48.609762+0000","last_became_peered":"2024-09-20T22:58:48.609762+0000","last_unstale":"2024-09-20T22:59:13.390376+0000","last_undegraded":"2024-09-20T22:59:13.390376+0000","last_fullsized":"2024-09-20T22:59:13.390376+0000","mapping_epoch":1022,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":1023,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:58:56.218432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:58:56.218432+0000","last_clean_scrub_stamp":"2024-09-20T22:58:56.218432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T23:00:22.404129+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":1893,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:13.486173+0000","last_change":"2024-09-20T22:58:43.919784+0000","last_active":"2024-09-20T22:59:13.486173+0000","last_peered":"2024-09-20T22:59:13.486173+0000","last_clean":"2024-09-20T22:59:13.486173+0000","last_became_active":"2024-09-20T22:58:28.405275+0000","last_became_peered":"2024-09-20T22:58:28.405275+0000","last_unstale":"2024-09-20T22:59:13.486173+0000","last_undegraded":"2024-09-20T22:59:13.486173+0000","last_fullsized":"2024-09-20T22:59:13.486173+0000","mapping_epoch":1008,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":1009,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:58:43.919676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:58:43.919676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:45.701337+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":1886,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:13.486525+0000","last_change":"2024-09-20T22:58:46.836440+0000","last_active":"2024-09-20T22:59:13.486525+0000","last_peered":"2024-09-20T22:59:13.486525+0000","last_clean":"2024-09-20T22:59:13.486525+0000","last_became_active":"2024-09-20T22:57:49.405928+0000","last_became_peered":"2024-09-20T22:57:49.405928+0000","last_unstale":"2024-09-20T22:59:13.486525+0000","last_undegraded":"2024-09-20T22:59:13.486525+0000","last_fullsized":"2024-09-20T22:59:13.486525+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:58:46.836366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:58:46.836366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T23:00:14.051353+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"625'27","reported_seq":1964,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:13.486381+0000","last_change":"2024-09-20T22:58:45.874779+0000","last_active":"2024-09-20T22:59:13.486381+0000","last_peered":"2024-09-20T22:59:13.486381+0000","last_clean":"2024-09-20T22:59:13.486381+0000","last_became_active":"2024-09-20T22:57:49.413932+0000","last_became_peered":"2024-09-20T22:57:49.413932+0000","last_unstale":"2024-09-20T22:59:13.486381+0000","last_undegraded":"2024-09-20T22:59:13.486381+0000","last_fullsized":"2024-09-20T22:59:13.486381+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'27","last_scrub_stamp":"2024-09-20T22:58:45.874695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:58:45.874695+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T23:00:12.578444+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":944,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:13.390004+0000","last_change":"2024-09-20T22:59:04.188018+0000","last_active":"2024-09-20T22:59:13.390004+0000","last_peered":"2024-09-20T22:59:13.390004+0000","last_clean":"2024-09-20T22:59:13.390004+0000","last_became_active":"2024-09-20T22:58:48.609458+0000","last_became_peered":"2024-09-20T22:58:48.609458+0000","last_unstale":"2024-09-20T22:59:13.390004+0000","last_undegraded":"2024-09-20T22:59:13.390004+0000","last_fullsized":"2024-09-20T22:59:13.390004+0000","mapping_epoch":1022,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":1023,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:59:04.187927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:59:04.187927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T23:00:28.919832+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":2019,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:14.015365+0000","last_change":"2024-09-20T22:58:49.931571+0000","last_active":"2024-09-20T22:59:14.015365+0000","last_peered":"2024-09-20T22:59:14.015365+0000","last_clean":"2024-09-20T22:59:14.015365+0000","last_became_active":"2024-09-20T22:48:05.242059+0000","last_became_peered":"2024-09-20T22:48:05.242059+0000","last_unstale":"2024-09-20T22:59:14.015365+0000","last_undegraded":"2024-09-20T22:59:14.015365+0000","last_fullsized":"2024-09-20T22:59:14.015365+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:58:49.931482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:46:00.999939+0000","last_clean_scrub_stamp":"2024-09-20T22:58:49.931482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T23:00:06.415733+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"625'15","reported_seq":951,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:13.390392+0000","last_change":"2024-09-20T22:58:48.605304+0000","last_active":"2024-09-20T22:59:13.390392+0000","last_peered":"2024-09-20T22:59:13.390392+0000","last_clean":"2024-09-20T22:59:13.390392+0000","last_became_active":"2024-09-20T22:58:48.604796+0000","last_became_peered":"2024-09-20T22:58:48.604796+0000","last_unstale":"2024-09-20T22:59:13.390392+0000","last_undegraded":"2024-09-20T22:59:13.390392+0000","last_fullsized":"2024-09-20T22:59:13.390392+0000","mapping_epoch":1022,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1023,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'15","last_scrub_stamp":"2024-09-20T22:58:38.891685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:58:38.891685+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:53.871927+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":2023,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:14.014919+0000","last_change":"2024-09-20T22:59:01.041705+0000","last_active":"2024-09-20T22:59:14.014919+0000","last_peered":"2024-09-20T22:59:14.014919+0000","last_clean":"2024-09-20T22:59:14.014919+0000","last_became_active":"2024-09-20T22:58:28.404062+0000","last_became_peered":"2024-09-20T22:58:28.404062+0000","last_unstale":"2024-09-20T22:59:14.014919+0000","last_undegraded":"2024-09-20T22:59:14.014919+0000","last_fullsized":"2024-09-20T22:59:14.014919+0000","mapping_epoch":1008,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":1009,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:59:01.041618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:59:01.041618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T23:00:18.951667+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":1373,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:13.389421+0000","last_change":"2024-09-20T22:58:31.561377+0000","last_active":"2024-09-20T22:59:13.389421+0000","last_peered":"2024-09-20T22:59:13.389421+0000","last_clean":"2024-09-20T22:59:13.389421+0000","last_became_active":"2024-09-20T22:58:28.416222+0000","last_became_peered":"2024-09-20T22:58:28.416222+0000","last_unstale":"2024-09-20T22:59:13.389421+0000","last_undegraded":"2024-09-20T22:59:13.389421+0000","last_fullsized":"2024-09-20T22:59:13.389421+0000","mapping_epoch":1008,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":1009,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:58:31.561311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:49:08.664407+0000","last_clean_scrub_stamp":"2024-09-20T22:58:31.561311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:47.583014+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":2031,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:14.015183+0000","last_change":"2024-09-20T22:58:44.889584+0000","last_active":"2024-09-20T22:59:14.015183+0000","last_peered":"2024-09-20T22:59:14.015183+0000","last_clean":"2024-09-20T22:59:14.015183+0000","last_became_active":"2024-09-20T22:54:17.342619+0000","last_became_peered":"2024-09-20T22:54:17.342619+0000","last_unstale":"2024-09-20T22:59:14.015183+0000","last_undegraded":"2024-09-20T22:59:14.015183+0000","last_fullsized":"2024-09-20T22:59:14.015183+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:58:44.889510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:58:44.889510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T23:00:03.935621+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"625'9","reported_seq":985,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:13.390747+0000","last_change":"2024-09-20T22:58:53.199092+0000","last_active":"2024-09-20T22:59:13.390747+0000","last_peered":"2024-09-20T22:59:13.390747+0000","last_clean":"2024-09-20T22:59:13.390747+0000","last_became_active":"2024-09-20T22:58:48.609715+0000","last_became_peered":"2024-09-20T22:58:48.609715+0000","last_unstale":"2024-09-20T22:59:13.390747+0000","last_undegraded":"2024-09-20T22:59:13.390747+0000","last_fullsized":"2024-09-20T22:59:13.390747+0000","mapping_epoch":1022,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1023,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'9","last_scrub_stamp":"2024-09-20T22:58:53.198994+0000","last_deep_scrub":"181'7","last_deep_scrub_stamp":"2024-09-20T22:48:41.049381+0000","last_clean_scrub_stamp":"2024-09-20T22:58:53.198994+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T23:00:14.829955+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":9,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":73,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:14.159607+0000","last_change":"2024-09-20T22:58:43.194716+0000","last_active":"2024-09-20T22:59:14.159607+0000","last_peered":"2024-09-20T22:59:14.159607+0000","last_clean":"2024-09-20T22:59:14.159607+0000","last_became_active":"2024-09-20T22:58:28.411249+0000","last_became_peered":"2024-09-20T22:58:28.411249+0000","last_unstale":"2024-09-20T22:59:14.159607+0000","last_undegraded":"2024-09-20T22:59:14.159607+0000","last_fullsized":"2024-09-20T22:59:14.159607+0000","mapping_epoch":1008,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":1009,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:58:43.194615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:47:40.361958+0000","last_clean_scrub_stamp":"2024-09-20T22:58:43.194615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:45.713548+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":2019,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:14.014720+0000","last_change":"2024-09-20T22:58:37.932389+0000","last_active":"2024-09-20T22:59:14.014720+0000","last_peered":"2024-09-20T22:59:14.014720+0000","last_clean":"2024-09-20T22:59:14.014720+0000","last_became_active":"2024-09-20T22:57:49.402447+0000","last_became_peered":"2024-09-20T22:57:49.402447+0000","last_unstale":"2024-09-20T22:59:14.014720+0000","last_undegraded":"2024-09-20T22:59:14.014720+0000","last_fullsized":"2024-09-20T22:59:14.014720+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:58:37.932289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:56:14.512566+0000","last_clean_scrub_stamp":"2024-09-20T22:58:37.932289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:51.158159+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":99,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:13.391051+0000","last_change":"2024-09-20T22:58:48.612664+0000","last_active":"2024-09-20T22:59:13.391051+0000","last_peered":"2024-09-20T22:59:13.391051+0000","last_clean":"2024-09-20T22:59:13.391051+0000","last_became_active":"2024-09-20T22:58:48.612440+0000","last_became_peered":"2024-09-20T22:58:48.612440+0000","last_unstale":"2024-09-20T22:59:13.391051+0000","last_undegraded":"2024-09-20T22:59:13.391051+0000","last_fullsized":"2024-09-20T22:59:13.391051+0000","mapping_epoch":1022,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":1023,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:58:42.503926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:58:42.503926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T23:00:06.749766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":1367,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:13.390542+0000","last_change":"2024-09-20T22:59:04.817192+0000","last_active":"2024-09-20T22:59:13.390542+0000","last_peered":"2024-09-20T22:59:13.390542+0000","last_clean":"2024-09-20T22:59:13.390542+0000","last_became_active":"2024-09-20T22:58:28.408957+0000","last_became_peered":"2024-09-20T22:58:28.408957+0000","last_unstale":"2024-09-20T22:59:13.390542+0000","last_undegraded":"2024-09-20T22:59:13.390542+0000","last_fullsized":"2024-09-20T22:59:13.390542+0000","mapping_epoch":1008,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1009,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:59:04.817072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:49:53.933601+0000","last_clean_scrub_stamp":"2024-09-20T22:59:04.817072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T23:00:22.752292+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"625'34","reported_seq":2122,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:13.391536+0000","last_change":"2024-09-20T22:58:01.296040+0000","last_active":"2024-09-20T22:59:13.391536+0000","last_peered":"2024-09-20T22:59:13.391536+0000","last_clean":"2024-09-20T22:59:13.391536+0000","last_became_active":"2024-09-20T22:57:49.430403+0000","last_became_peered":"2024-09-20T22:57:49.430403+0000","last_unstale":"2024-09-20T22:59:13.391536+0000","last_undegraded":"2024-09-20T22:59:13.391536+0000","last_fullsized":"2024-09-20T22:59:13.391536+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'34","last_scrub_stamp":"2024-09-20T22:58:01.295956+0000","last_deep_scrub":"113'22","last_deep_scrub_stamp":"2024-09-20T22:43:12.191878+0000","last_clean_scrub_stamp":"2024-09-20T22:58:01.295956+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:30.939636+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":561,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:20.446597+0000","last_change":"2024-09-20T22:59:20.446630+0000","last_active":"2024-09-20T22:59:20.446597+0000","last_peered":"2024-09-20T22:59:20.446597+0000","last_clean":"2024-09-20T22:59:20.446597+0000","last_became_active":"2024-09-20T22:54:17.349501+0000","last_became_peered":"2024-09-20T22:54:17.349501+0000","last_unstale":"2024-09-20T22:59:20.446597+0000","last_undegraded":"2024-09-20T22:59:20.446597+0000","last_fullsized":"2024-09-20T22:59:20.446597+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:59:20.446504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:59:20.446504+0000","last_clean_scrub_stamp":"2024-09-20T22:59:20.446504+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T23:00:25.083082+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":1866,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:13.486537+0000","last_change":"2024-09-20T22:58:35.892172+0000","last_active":"2024-09-20T22:59:13.486537+0000","last_peered":"2024-09-20T22:59:13.486537+0000","last_clean":"2024-09-20T22:59:13.486537+0000","last_became_active":"2024-09-20T22:57:49.419671+0000","last_became_peered":"2024-09-20T22:57:49.419671+0000","last_unstale":"2024-09-20T22:59:13.486537+0000","last_undegraded":"2024-09-20T22:59:13.486537+0000","last_fullsized":"2024-09-20T22:59:13.486537+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:58:35.892096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:45:25.002199+0000","last_clean_scrub_stamp":"2024-09-20T22:58:35.892096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:52.759843+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":952,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:13.390663+0000","last_change":"2024-09-20T22:59:05.195013+0000","last_active":"2024-09-20T22:59:13.390663+0000","last_peered":"2024-09-20T22:59:13.390663+0000","last_clean":"2024-09-20T22:59:13.390663+0000","last_became_active":"2024-09-20T22:58:48.609194+0000","last_became_peered":"2024-09-20T22:58:48.609194+0000","last_unstale":"2024-09-20T22:59:13.390663+0000","last_undegraded":"2024-09-20T22:59:13.390663+0000","last_fullsized":"2024-09-20T22:59:13.390663+0000","mapping_epoch":1022,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1023,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:59:05.194924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:59:05.194924+0000","last_clean_scrub_stamp":"2024-09-20T22:59:05.194924+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T23:00:18.806408+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"625'18","reported_seq":1944,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:13.486253+0000","last_change":"2024-09-20T22:58:44.904887+0000","last_active":"2024-09-20T22:59:13.486253+0000","last_peered":"2024-09-20T22:59:13.486253+0000","last_clean":"2024-09-20T22:59:13.486253+0000","last_became_active":"2024-09-20T22:57:49.403366+0000","last_became_peered":"2024-09-20T22:57:49.403366+0000","last_unstale":"2024-09-20T22:59:13.486253+0000","last_undegraded":"2024-09-20T22:59:13.486253+0000","last_fullsized":"2024-09-20T22:59:13.486253+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'18","last_scrub_stamp":"2024-09-20T22:58:44.904792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:58:44.904792+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:45.535866+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":940,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:13.390715+0000","last_change":"2024-09-20T22:58:54.239279+0000","last_active":"2024-09-20T22:59:13.390715+0000","last_peered":"2024-09-20T22:59:13.390715+0000","last_clean":"2024-09-20T22:59:13.390715+0000","last_became_active":"2024-09-20T22:58:48.608447+0000","last_became_peered":"2024-09-20T22:58:48.608447+0000","last_unstale":"2024-09-20T22:59:13.390715+0000","last_undegraded":"2024-09-20T22:59:13.390715+0000","last_fullsized":"2024-09-20T22:59:13.390715+0000","mapping_epoch":1022,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":1023,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:58:54.239199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:58:54.239199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T23:00:21.655788+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"154'2","reported_seq":77,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:28.619562+0000","last_change":"2024-09-20T22:59:28.619606+0000","last_active":"2024-09-20T22:59:28.619562+0000","last_peered":"2024-09-20T22:59:28.619562+0000","last_clean":"2024-09-20T22:59:28.619562+0000","last_became_active":"2024-09-20T22:58:28.419741+0000","last_became_peered":"2024-09-20T22:58:28.419741+0000","last_unstale":"2024-09-20T22:59:28.619562+0000","last_undegraded":"2024-09-20T22:59:28.619562+0000","last_fullsized":"2024-09-20T22:59:28.619562+0000","mapping_epoch":1008,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":1009,"parent":"0.0","parent_split_bits":0,"last_scrub":"154'2","last_scrub_stamp":"2024-09-20T22:59:28.619469+0000","last_deep_scrub":"154'2","last_deep_scrub_stamp":"2024-09-20T22:43:54.134066+0000","last_clean_scrub_stamp":"2024-09-20T22:59:28.619469+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T23:00:44.846126+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":403,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"894'2","reported_seq":567,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:14.015393+0000","last_change":"2024-09-20T22:58:32.360859+0000","last_active":"2024-09-20T22:59:14.015393+0000","last_peered":"2024-09-20T22:59:14.015393+0000","last_clean":"2024-09-20T22:59:14.015393+0000","last_became_active":"2024-09-20T22:57:49.405458+0000","last_became_peered":"2024-09-20T22:57:49.405458+0000","last_unstale":"2024-09-20T22:59:14.015393+0000","last_undegraded":"2024-09-20T22:59:14.015393+0000","last_fullsized":"2024-09-20T22:59:14.015393+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"894'2","last_scrub_stamp":"2024-09-20T22:58:32.360782+0000","last_deep_scrub":"894'2","last_deep_scrub_stamp":"2024-09-20T22:56:15.522649+0000","last_clean_scrub_stamp":"2024-09-20T22:58:32.360782+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:58.358240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"625'25","reported_seq":1430,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:24.781166+0000","last_change":"2024-09-20T22:59:24.781214+0000","last_active":"2024-09-20T22:59:24.781166+0000","last_peered":"2024-09-20T22:59:24.781166+0000","last_clean":"2024-09-20T22:59:24.781166+0000","last_became_active":"2024-09-20T22:58:28.410047+0000","last_became_peered":"2024-09-20T22:58:28.410047+0000","last_unstale":"2024-09-20T22:59:24.781166+0000","last_undegraded":"2024-09-20T22:59:24.781166+0000","last_fullsized":"2024-09-20T22:59:24.781166+0000","mapping_epoch":1008,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1009,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'25","last_scrub_stamp":"2024-09-20T22:59:24.781101+0000","last_deep_scrub":"151'15","last_deep_scrub_stamp":"2024-09-20T22:43:50.715761+0000","last_clean_scrub_stamp":"2024-09-20T22:59:24.781101+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T23:00:43.559126+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"1009'13","reported_seq":2292,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:23.399948+0000","last_change":"2024-09-20T22:58:32.407579+0000","last_active":"2024-09-20T22:59:23.399948+0000","last_peered":"2024-09-20T22:59:23.399948+0000","last_clean":"2024-09-20T22:59:23.399948+0000","last_became_active":"2024-09-20T22:58:28.399826+0000","last_became_peered":"2024-09-20T22:58:28.399826+0000","last_unstale":"2024-09-20T22:59:23.399948+0000","last_undegraded":"2024-09-20T22:59:23.399948+0000","last_fullsized":"2024-09-20T22:59:23.399948+0000","mapping_epoch":1008,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":1009,"parent":"0.0","parent_split_bits":0,"last_scrub":"1009'13","last_scrub_stamp":"2024-09-20T22:58:32.407440+0000","last_deep_scrub":"661'11","last_deep_scrub_stamp":"2024-09-20T22:52:21.907943+0000","last_clean_scrub_stamp":"2024-09-20T22:58:32.407440+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:51.848588+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.a","version":"77'128","reported_seq":2231,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:14.014558+0000","last_change":"2024-09-20T22:58:55.396912+0000","last_active":"2024-09-20T22:59:14.014558+0000","last_peered":"2024-09-20T22:59:14.014558+0000","last_clean":"2024-09-20T22:59:14.014558+0000","last_became_active":"2024-09-20T22:58:55.396426+0000","last_became_peered":"2024-09-20T22:58:55.396426+0000","last_unstale":"2024-09-20T22:59:14.014558+0000","last_undegraded":"2024-09-20T22:59:14.014558+0000","last_fullsized":"2024-09-20T22:59:14.014558+0000","mapping_epoch":1025,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":1026,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:58:22.154735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:58:22.154735+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:29.181918+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"625'27","reported_seq":578,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:14.016400+0000","last_change":"2024-09-20T22:58:01.424871+0000","last_active":"2024-09-20T22:59:14.016400+0000","last_peered":"2024-09-20T22:59:14.016400+0000","last_clean":"2024-09-20T22:59:14.016400+0000","last_became_active":"2024-09-20T22:57:49.415260+0000","last_became_peered":"2024-09-20T22:57:49.415260+0000","last_unstale":"2024-09-20T22:59:14.016400+0000","last_undegraded":"2024-09-20T22:59:14.016400+0000","last_fullsized":"2024-09-20T22:59:14.016400+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'27","last_scrub_stamp":"2024-09-20T22:58:01.424772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:58:01.424772+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:30.323196+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":1909,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:13.485910+0000","last_change":"2024-09-20T22:59:00.884919+0000","last_active":"2024-09-20T22:59:13.485910+0000","last_peered":"2024-09-20T22:59:13.485910+0000","last_clean":"2024-09-20T22:59:13.485910+0000","last_became_active":"2024-09-20T22:58:28.406214+0000","last_became_peered":"2024-09-20T22:58:28.406214+0000","last_unstale":"2024-09-20T22:59:13.485910+0000","last_undegraded":"2024-09-20T22:59:13.485910+0000","last_fullsized":"2024-09-20T22:59:13.485910+0000","mapping_epoch":1008,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1009,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:59:00.884824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:49:12.070675+0000","last_clean_scrub_stamp":"2024-09-20T22:59:00.884824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T23:00:03.144291+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"1023'15","reported_seq":1040,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:23.399949+0000","last_change":"2024-09-20T22:58:57.200171+0000","last_active":"2024-09-20T22:59:23.399949+0000","last_peered":"2024-09-20T22:59:23.399949+0000","last_clean":"2024-09-20T22:59:23.399949+0000","last_became_active":"2024-09-20T22:58:48.610492+0000","last_became_peered":"2024-09-20T22:58:48.610492+0000","last_unstale":"2024-09-20T22:59:23.399949+0000","last_undegraded":"2024-09-20T22:59:23.399949+0000","last_fullsized":"2024-09-20T22:59:23.399949+0000","mapping_epoch":1022,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":1023,"parent":"0.0","parent_split_bits":0,"last_scrub":"1023'15","last_scrub_stamp":"2024-09-20T22:58:57.200085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:58:57.200085+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:59.588926+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":2019,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:14.015078+0000","last_change":"2024-09-20T22:59:02.014253+0000","last_active":"2024-09-20T22:59:14.015078+0000","last_peered":"2024-09-20T22:59:14.015078+0000","last_clean":"2024-09-20T22:59:14.015078+0000","last_became_active":"2024-09-20T22:54:17.340327+0000","last_became_peered":"2024-09-20T22:54:17.340327+0000","last_unstale":"2024-09-20T22:59:14.015078+0000","last_undegraded":"2024-09-20T22:59:14.015078+0000","last_fullsized":"2024-09-20T22:59:14.015078+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:59:02.014154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:59:02.014154+0000","last_clean_scrub_stamp":"2024-09-20T22:59:02.014154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T23:00:22.922237+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.5","version":"77'128","reported_seq":1801,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:13.391234+0000","last_change":"2024-09-20T22:58:48.611905+0000","last_active":"2024-09-20T22:59:13.391234+0000","last_peered":"2024-09-20T22:59:13.391234+0000","last_clean":"2024-09-20T22:59:13.391234+0000","last_became_active":"2024-09-20T22:58:48.610563+0000","last_became_peered":"2024-09-20T22:58:48.610563+0000","last_unstale":"2024-09-20T22:59:13.391234+0000","last_undegraded":"2024-09-20T22:59:13.391234+0000","last_fullsized":"2024-09-20T22:59:13.391234+0000","mapping_epoch":1022,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":1023,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:58:42.277786+0000","last_deep_scrub":"77'128","last_deep_scrub_stamp":"2024-09-20T22:57:14.262265+0000","last_clean_scrub_stamp":"2024-09-20T22:58:42.277786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:59.697230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"625'30","reported_seq":2028,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:13.485914+0000","last_change":"2024-09-20T22:58:29.898911+0000","last_active":"2024-09-20T22:59:13.485914+0000","last_peered":"2024-09-20T22:59:13.485914+0000","last_clean":"2024-09-20T22:59:13.485914+0000","last_became_active":"2024-09-20T22:58:28.408416+0000","last_became_peered":"2024-09-20T22:58:28.408416+0000","last_unstale":"2024-09-20T22:59:13.485914+0000","last_undegraded":"2024-09-20T22:59:13.485914+0000","last_fullsized":"2024-09-20T22:59:13.485914+0000","mapping_epoch":1008,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1009,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'30","last_scrub_stamp":"2024-09-20T22:58:29.898807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:58:29.898807+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:59.376020+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":39,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":1930,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:13.486750+0000","last_change":"2024-09-20T22:59:12.098368+0000","last_active":"2024-09-20T22:59:13.486750+0000","last_peered":"2024-09-20T22:59:13.486750+0000","last_clean":"2024-09-20T22:59:13.486750+0000","last_became_active":"2024-09-20T22:58:28.409296+0000","last_became_peered":"2024-09-20T22:58:28.409296+0000","last_unstale":"2024-09-20T22:59:13.486750+0000","last_undegraded":"2024-09-20T22:59:13.486750+0000","last_fullsized":"2024-09-20T22:59:13.486750+0000","mapping_epoch":1008,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1009,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:59:12.098263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:59:12.098263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T23:00:22.984205+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":941,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:13.390441+0000","last_change":"2024-09-20T22:58:59.149598+0000","last_active":"2024-09-20T22:59:13.390441+0000","last_peered":"2024-09-20T22:59:13.390441+0000","last_clean":"2024-09-20T22:59:13.390441+0000","last_became_active":"2024-09-20T22:58:49.164911+0000","last_became_peered":"2024-09-20T22:58:49.164911+0000","last_unstale":"2024-09-20T22:59:13.390441+0000","last_undegraded":"2024-09-20T22:59:13.390441+0000","last_fullsized":"2024-09-20T22:59:13.390441+0000","mapping_epoch":1022,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":1023,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:58:59.149475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:56:47.432483+0000","last_clean_scrub_stamp":"2024-09-20T22:58:59.149475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T23:00:02.308884+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":933,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:13.390480+0000","last_change":"2024-09-20T22:58:52.223664+0000","last_active":"2024-09-20T22:59:13.390480+0000","last_peered":"2024-09-20T22:59:13.390480+0000","last_clean":"2024-09-20T22:59:13.390480+0000","last_became_active":"2024-09-20T22:58:48.608756+0000","last_became_peered":"2024-09-20T22:58:48.608756+0000","last_unstale":"2024-09-20T22:59:13.390480+0000","last_undegraded":"2024-09-20T22:59:13.390480+0000","last_fullsized":"2024-09-20T22:59:13.390480+0000","mapping_epoch":1022,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":1023,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:58:52.223572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:53.036601+0000","last_clean_scrub_stamp":"2024-09-20T22:58:52.223572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:58.874292+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"855'129","reported_seq":2232,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:13.486596+0000","last_change":"2024-09-20T22:59:08.033384+0000","last_active":"2024-09-20T22:59:13.486596+0000","last_peered":"2024-09-20T22:59:13.486596+0000","last_clean":"2024-09-20T22:59:13.486596+0000","last_became_active":"2024-09-20T22:59:01.645724+0000","last_became_peered":"2024-09-20T22:59:01.645724+0000","last_unstale":"2024-09-20T22:59:13.486596+0000","last_undegraded":"2024-09-20T22:59:13.486596+0000","last_fullsized":"2024-09-20T22:59:13.486596+0000","mapping_epoch":1029,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":1030,"parent":"0.0","parent_split_bits":3,"last_scrub":"855'129","last_scrub_stamp":"2024-09-20T22:59:08.033310+0000","last_deep_scrub":"855'129","last_deep_scrub_stamp":"2024-09-20T22:55:35.536654+0000","last_clean_scrub_stamp":"2024-09-20T22:59:08.033310+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":59,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T23:00:12.485840+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":655360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"1030'35","reported_seq":985,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:13.390353+0000","last_change":"2024-09-20T22:59:02.172329+0000","last_active":"2024-09-20T22:59:13.390353+0000","last_peered":"2024-09-20T22:59:13.390353+0000","last_clean":"2024-09-20T22:59:13.390353+0000","last_became_active":"2024-09-20T22:58:48.611983+0000","last_became_peered":"2024-09-20T22:58:48.611983+0000","last_unstale":"2024-09-20T22:59:13.390353+0000","last_undegraded":"2024-09-20T22:59:13.390353+0000","last_fullsized":"2024-09-20T22:59:13.390353+0000","mapping_epoch":1022,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1023,"parent":"0.0","parent_split_bits":0,"last_scrub":"1030'35","last_scrub_stamp":"2024-09-20T22:59:02.172205+0000","last_deep_scrub":"1030'35","last_deep_scrub_stamp":"2024-09-20T22:59:02.172205+0000","last_clean_scrub_stamp":"2024-09-20T22:59:02.172205+0000","objects_scrubbed":7,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T23:00:17.266739+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":41,"num_read_kb":34,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"855'2","reported_seq":1385,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:13.390833+0000","last_change":"2024-09-20T22:58:26.489611+0000","last_active":"2024-09-20T22:59:13.390833+0000","last_peered":"2024-09-20T22:59:13.390833+0000","last_clean":"2024-09-20T22:59:13.390833+0000","last_became_active":"2024-09-20T22:54:17.340423+0000","last_became_peered":"2024-09-20T22:54:17.340423+0000","last_unstale":"2024-09-20T22:59:13.390833+0000","last_undegraded":"2024-09-20T22:59:13.390833+0000","last_fullsized":"2024-09-20T22:59:13.390833+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"855'2","last_scrub_stamp":"2024-09-20T22:58:26.489536+0000","last_deep_scrub":"855'2","last_deep_scrub_stamp":"2024-09-20T22:55:35.572977+0000","last_clean_scrub_stamp":"2024-09-20T22:58:26.489536+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:52.640705+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":230,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"989'18","reported_seq":2114,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:18.399448+0000","last_change":"2024-09-20T22:58:54.817713+0000","last_active":"2024-09-20T22:59:18.399448+0000","last_peered":"2024-09-20T22:59:18.399448+0000","last_clean":"2024-09-20T22:59:18.399448+0000","last_became_active":"2024-09-20T22:57:49.401445+0000","last_became_peered":"2024-09-20T22:57:49.401445+0000","last_unstale":"2024-09-20T22:59:18.399448+0000","last_undegraded":"2024-09-20T22:59:18.399448+0000","last_fullsized":"2024-09-20T22:59:18.399448+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"989'18","last_scrub_stamp":"2024-09-20T22:58:54.817628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:58:54.817628+0000","objects_scrubbed":1,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T23:00:22.475889+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":73,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:14.159500+0000","last_change":"2024-09-20T22:59:12.347762+0000","last_active":"2024-09-20T22:59:14.159500+0000","last_peered":"2024-09-20T22:59:14.159500+0000","last_clean":"2024-09-20T22:59:14.159500+0000","last_became_active":"2024-09-20T22:58:28.406517+0000","last_became_peered":"2024-09-20T22:58:28.406517+0000","last_unstale":"2024-09-20T22:59:14.159500+0000","last_undegraded":"2024-09-20T22:59:14.159500+0000","last_fullsized":"2024-09-20T22:59:14.159500+0000","mapping_epoch":1008,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":1009,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:59:12.347689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:59:12.347689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T23:00:26.078906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.7","version":"77'128","reported_seq":2359,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:24.565956+0000","last_change":"2024-09-20T22:59:24.565981+0000","last_active":"2024-09-20T22:59:24.565956+0000","last_peered":"2024-09-20T22:59:24.565956+0000","last_clean":"2024-09-20T22:59:24.565956+0000","last_became_active":"2024-09-20T22:58:31.410353+0000","last_became_peered":"2024-09-20T22:58:31.410353+0000","last_unstale":"2024-09-20T22:59:24.565956+0000","last_undegraded":"2024-09-20T22:59:24.565956+0000","last_fullsized":"2024-09-20T22:59:24.565956+0000","mapping_epoch":1011,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":1012,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:59:24.565892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:59:24.565892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T23:00:45.563252+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"904'25","reported_seq":578,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:14.015635+0000","last_change":"2024-09-20T22:59:08.413880+0000","last_active":"2024-09-20T22:59:14.015635+0000","last_peered":"2024-09-20T22:59:14.015635+0000","last_clean":"2024-09-20T22:59:14.015635+0000","last_became_active":"2024-09-20T22:54:17.349113+0000","last_became_peered":"2024-09-20T22:54:17.349113+0000","last_unstale":"2024-09-20T22:59:14.015635+0000","last_undegraded":"2024-09-20T22:59:14.015635+0000","last_fullsized":"2024-09-20T22:59:14.015635+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"904'25","last_scrub_stamp":"2024-09-20T22:59:08.413795+0000","last_deep_scrub":"904'25","last_deep_scrub_stamp":"2024-09-20T22:56:24.575150+0000","last_clean_scrub_stamp":"2024-09-20T22:59:08.413795+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T23:00:29.997661+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":28,"num_read_kb":23,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":2017,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:14.015473+0000","last_change":"2024-09-20T22:58:35.955562+0000","last_active":"2024-09-20T22:59:14.015473+0000","last_peered":"2024-09-20T22:59:14.015473+0000","last_clean":"2024-09-20T22:59:14.015473+0000","last_became_active":"2024-09-20T22:57:49.396185+0000","last_became_peered":"2024-09-20T22:57:49.396185+0000","last_unstale":"2024-09-20T22:59:14.015473+0000","last_undegraded":"2024-09-20T22:59:14.015473+0000","last_fullsized":"2024-09-20T22:59:14.015473+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:58:35.955496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:58:35.955496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:49.044151+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"733'12","reported_seq":1496,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:23.399608+0000","last_change":"2024-09-20T22:58:27.469722+0000","last_active":"2024-09-20T22:59:23.399608+0000","last_peered":"2024-09-20T22:59:23.399608+0000","last_clean":"2024-09-20T22:59:23.399608+0000","last_became_active":"2024-09-20T22:47:49.227264+0000","last_became_peered":"2024-09-20T22:47:49.227264+0000","last_unstale":"2024-09-20T22:59:23.399608+0000","last_undegraded":"2024-09-20T22:59:23.399608+0000","last_fullsized":"2024-09-20T22:59:23.399608+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"733'12","last_scrub_stamp":"2024-09-20T22:58:27.469646+0000","last_deep_scrub":"733'12","last_deep_scrub_stamp":"2024-09-20T22:58:27.469646+0000","last_clean_scrub_stamp":"2024-09-20T22:58:27.469646+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:56.471372+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":1359,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:13.389276+0000","last_change":"2024-09-20T22:58:28.488885+0000","last_active":"2024-09-20T22:59:13.389276+0000","last_peered":"2024-09-20T22:59:13.389276+0000","last_clean":"2024-09-20T22:59:13.389276+0000","last_became_active":"2024-09-20T22:57:49.416550+0000","last_became_peered":"2024-09-20T22:57:49.416550+0000","last_unstale":"2024-09-20T22:59:13.389276+0000","last_undegraded":"2024-09-20T22:59:13.389276+0000","last_fullsized":"2024-09-20T22:59:13.389276+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:58:28.488787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:58:28.488787+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:35.965266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.6","version":"77'128","reported_seq":2343,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:13.746857+0000","last_change":"2024-09-20T22:58:03.362917+0000","last_active":"2024-09-20T22:59:13.746857+0000","last_peered":"2024-09-20T22:59:13.746857+0000","last_clean":"2024-09-20T22:59:13.746857+0000","last_became_active":"2024-09-20T22:57:49.406637+0000","last_became_peered":"2024-09-20T22:57:49.406637+0000","last_unstale":"2024-09-20T22:59:13.746857+0000","last_undegraded":"2024-09-20T22:59:13.746857+0000","last_fullsized":"2024-09-20T22:59:13.746857+0000","mapping_epoch":988,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:58:03.362805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:58:03.362805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:32.522555+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"625'37","reported_seq":646,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:14.014923+0000","last_change":"2024-09-20T22:58:59.018734+0000","last_active":"2024-09-20T22:59:14.014923+0000","last_peered":"2024-09-20T22:59:14.014923+0000","last_clean":"2024-09-20T22:59:14.014923+0000","last_became_active":"2024-09-20T22:58:09.157511+0000","last_became_peered":"2024-09-20T22:58:09.157511+0000","last_unstale":"2024-09-20T22:59:14.014923+0000","last_undegraded":"2024-09-20T22:59:14.014923+0000","last_fullsized":"2024-09-20T22:59:14.014923+0000","mapping_epoch":996,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":997,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'37","last_scrub_stamp":"2024-09-20T22:58:59.018611+0000","last_deep_scrub":"515'27","last_deep_scrub_stamp":"2024-09-20T22:49:55.964153+0000","last_clean_scrub_stamp":"2024-09-20T22:58:59.018611+0000","objects_scrubbed":8,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T23:00:27.942616+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":38,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":553,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:14.016144+0000","last_change":"2024-09-20T22:59:03.402620+0000","last_active":"2024-09-20T22:59:14.016144+0000","last_peered":"2024-09-20T22:59:14.016144+0000","last_clean":"2024-09-20T22:59:14.016144+0000","last_became_active":"2024-09-20T22:54:17.341248+0000","last_became_peered":"2024-09-20T22:54:17.341248+0000","last_unstale":"2024-09-20T22:59:14.016144+0000","last_undegraded":"2024-09-20T22:59:14.016144+0000","last_fullsized":"2024-09-20T22:59:14.016144+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:59:03.402478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:48:23.320031+0000","last_clean_scrub_stamp":"2024-09-20T22:59:03.402478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T23:00:27.744280+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":2034,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:21.548467+0000","last_change":"2024-09-20T22:59:21.548491+0000","last_active":"2024-09-20T22:59:21.548467+0000","last_peered":"2024-09-20T22:59:21.548467+0000","last_clean":"2024-09-20T22:59:21.548467+0000","last_became_active":"2024-09-20T22:47:49.214055+0000","last_became_peered":"2024-09-20T22:47:49.214055+0000","last_unstale":"2024-09-20T22:59:21.548467+0000","last_undegraded":"2024-09-20T22:59:21.548467+0000","last_fullsized":"2024-09-20T22:59:21.548467+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:59:21.548401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:59:21.548401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T23:00:26.729576+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":545,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:14.015534+0000","last_change":"2024-09-20T22:58:23.325856+0000","last_active":"2024-09-20T22:59:14.015534+0000","last_peered":"2024-09-20T22:59:14.015534+0000","last_clean":"2024-09-20T22:59:14.015534+0000","last_became_active":"2024-09-20T22:54:17.340458+0000","last_became_peered":"2024-09-20T22:54:17.340458+0000","last_unstale":"2024-09-20T22:59:14.015534+0000","last_undegraded":"2024-09-20T22:59:14.015534+0000","last_fullsized":"2024-09-20T22:59:14.015534+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:58:23.325766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:48:41.208853+0000","last_clean_scrub_stamp":"2024-09-20T22:58:23.325766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:44.933821+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"77'128","reported_seq":2325,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:17.904859+0000","last_change":"2024-09-20T22:59:17.904859+0000","last_active":"2024-09-20T22:59:17.904859+0000","last_peered":"2024-09-20T22:59:17.904859+0000","last_clean":"2024-09-20T22:59:17.904859+0000","last_became_active":"2024-09-20T22:47:52.208102+0000","last_became_peered":"2024-09-20T22:47:52.208102+0000","last_unstale":"2024-09-20T22:59:17.904859+0000","last_undegraded":"2024-09-20T22:59:17.904859+0000","last_fullsized":"2024-09-20T22:59:17.904859+0000","mapping_epoch":391,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:59:17.904754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:59:17.904754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T23:00:39.070223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"1026'57","reported_seq":471,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:23.721188+0000","last_change":"2024-09-20T22:58:55.397509+0000","last_active":"2024-09-20T22:59:23.721188+0000","last_peered":"2024-09-20T22:59:23.721188+0000","last_clean":"2024-09-20T22:59:23.721188+0000","last_became_active":"2024-09-20T22:58:55.397171+0000","last_became_peered":"2024-09-20T22:58:55.397171+0000","last_unstale":"2024-09-20T22:59:23.721188+0000","last_undegraded":"2024-09-20T22:59:23.721188+0000","last_fullsized":"2024-09-20T22:59:23.721188+0000","mapping_epoch":1025,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1026,"parent":"0.0","parent_split_bits":0,"last_scrub":"1006'56","last_scrub_stamp":"2024-09-20T22:58:33.199281+0000","last_deep_scrub":"1006'56","last_deep_scrub_stamp":"2024-09-20T22:58:33.199281+0000","last_clean_scrub_stamp":"2024-09-20T22:58:33.199281+0000","objects_scrubbed":10,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T23:00:00.564014+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":1074,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":933,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:13.390524+0000","last_change":"2024-09-20T22:59:09.156959+0000","last_active":"2024-09-20T22:59:13.390524+0000","last_peered":"2024-09-20T22:59:13.390524+0000","last_clean":"2024-09-20T22:59:13.390524+0000","last_became_active":"2024-09-20T22:58:48.608668+0000","last_became_peered":"2024-09-20T22:58:48.608668+0000","last_unstale":"2024-09-20T22:59:13.390524+0000","last_undegraded":"2024-09-20T22:59:13.390524+0000","last_fullsized":"2024-09-20T22:59:13.390524+0000","mapping_epoch":1022,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1023,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:59:09.156871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:59:09.156871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T23:00:21.048647+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"69'2","reported_seq":2012,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:14.014955+0000","last_change":"2024-09-20T22:59:04.001271+0000","last_active":"2024-09-20T22:59:14.014955+0000","last_peered":"2024-09-20T22:59:14.014955+0000","last_clean":"2024-09-20T22:59:14.014955+0000","last_became_active":"2024-09-20T22:58:55.394665+0000","last_became_peered":"2024-09-20T22:58:55.394665+0000","last_unstale":"2024-09-20T22:59:14.014955+0000","last_undegraded":"2024-09-20T22:59:14.014955+0000","last_fullsized":"2024-09-20T22:59:14.014955+0000","mapping_epoch":1025,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":1026,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2024-09-20T22:59:04.001168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:36.130684+0000","last_clean_scrub_stamp":"2024-09-20T22:59:04.001168+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T23:00:29.544905+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00087280599999999995,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":1368,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:13.390619+0000","last_change":"2024-09-20T22:58:34.555808+0000","last_active":"2024-09-20T22:59:13.390619+0000","last_peered":"2024-09-20T22:59:13.390619+0000","last_clean":"2024-09-20T22:59:13.390619+0000","last_became_active":"2024-09-20T22:54:17.339860+0000","last_became_peered":"2024-09-20T22:54:17.339860+0000","last_unstale":"2024-09-20T22:59:13.390619+0000","last_undegraded":"2024-09-20T22:59:13.390619+0000","last_fullsized":"2024-09-20T22:59:13.390619+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:58:34.555736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:58:34.555736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:45.003937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":1905,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:13.486839+0000","last_change":"2024-09-20T22:59:10.033024+0000","last_active":"2024-09-20T22:59:13.486839+0000","last_peered":"2024-09-20T22:59:13.486839+0000","last_clean":"2024-09-20T22:59:13.486839+0000","last_became_active":"2024-09-20T22:58:28.410700+0000","last_became_peered":"2024-09-20T22:58:28.410700+0000","last_unstale":"2024-09-20T22:59:13.486839+0000","last_undegraded":"2024-09-20T22:59:13.486839+0000","last_fullsized":"2024-09-20T22:59:13.486839+0000","mapping_epoch":1008,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":1009,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:59:10.032928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:59:10.032928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T23:00:16.267119+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"77'128","reported_seq":1368,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:14.159413+0000","last_change":"2024-09-20T22:59:02.307466+0000","last_active":"2024-09-20T22:59:14.159413+0000","last_peered":"2024-09-20T22:59:14.159413+0000","last_clean":"2024-09-20T22:59:14.159413+0000","last_became_active":"2024-09-20T22:59:01.642508+0000","last_became_peered":"2024-09-20T22:59:01.642508+0000","last_unstale":"2024-09-20T22:59:14.159413+0000","last_undegraded":"2024-09-20T22:59:14.159413+0000","last_fullsized":"2024-09-20T22:59:14.159413+0000","mapping_epoch":1029,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":1030,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:59:02.307342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:59:02.307342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T23:00:26.538592+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"625'22","reported_seq":568,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:14.016428+0000","last_change":"2024-09-20T22:58:48.547764+0000","last_active":"2024-09-20T22:59:14.016428+0000","last_peered":"2024-09-20T22:59:14.016428+0000","last_clean":"2024-09-20T22:59:14.016428+0000","last_became_active":"2024-09-20T22:54:17.349415+0000","last_became_peered":"2024-09-20T22:54:17.349415+0000","last_unstale":"2024-09-20T22:59:14.016428+0000","last_undegraded":"2024-09-20T22:59:14.016428+0000","last_fullsized":"2024-09-20T22:59:14.016428+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'22","last_scrub_stamp":"2024-09-20T22:58:48.547682+0000","last_deep_scrub":"443'18","last_deep_scrub_stamp":"2024-09-20T22:48:44.182517+0000","last_clean_scrub_stamp":"2024-09-20T22:58:48.547682+0000","objects_scrubbed":5,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:51.430246+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":24,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":1320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":553,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:14.015547+0000","last_change":"2024-09-20T22:58:31.390541+0000","last_active":"2024-09-20T22:59:14.015547+0000","last_peered":"2024-09-20T22:59:14.015547+0000","last_clean":"2024-09-20T22:59:14.015547+0000","last_became_active":"2024-09-20T22:54:17.343806+0000","last_became_peered":"2024-09-20T22:54:17.343806+0000","last_unstale":"2024-09-20T22:59:14.015547+0000","last_undegraded":"2024-09-20T22:59:14.015547+0000","last_fullsized":"2024-09-20T22:59:14.015547+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:58:31.390467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:56:00.249796+0000","last_clean_scrub_stamp":"2024-09-20T22:58:31.390467+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:44.179485+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"380'2","reported_seq":2071,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:13.391330+0000","last_change":"2024-09-20T22:58:23.141502+0000","last_active":"2024-09-20T22:59:13.391330+0000","last_peered":"2024-09-20T22:59:13.391330+0000","last_clean":"2024-09-20T22:59:13.391330+0000","last_became_active":"2024-09-20T22:57:49.416996+0000","last_became_peered":"2024-09-20T22:57:49.416996+0000","last_unstale":"2024-09-20T22:59:13.391330+0000","last_undegraded":"2024-09-20T22:59:13.391330+0000","last_fullsized":"2024-09-20T22:59:13.391330+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"380'2","last_scrub_stamp":"2024-09-20T22:58:23.141431+0000","last_deep_scrub":"380'2","last_deep_scrub_stamp":"2024-09-20T22:47:40.983039+0000","last_clean_scrub_stamp":"2024-09-20T22:58:23.141431+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:35.912629+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001301459,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":2028,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:13.390176+0000","last_change":"2024-09-20T22:58:46.283696+0000","last_active":"2024-09-20T22:59:13.390176+0000","last_peered":"2024-09-20T22:59:13.390176+0000","last_clean":"2024-09-20T22:59:13.390176+0000","last_became_active":"2024-09-20T22:58:28.408036+0000","last_became_peered":"2024-09-20T22:58:28.408036+0000","last_unstale":"2024-09-20T22:59:13.390176+0000","last_undegraded":"2024-09-20T22:59:13.390176+0000","last_fullsized":"2024-09-20T22:59:13.390176+0000","mapping_epoch":1008,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":1009,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:58:46.283615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:57:38.381457+0000","last_clean_scrub_stamp":"2024-09-20T22:58:46.283615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T23:00:00.632985+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":73,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:14.159866+0000","last_change":"2024-09-20T22:58:54.109803+0000","last_active":"2024-09-20T22:59:14.159866+0000","last_peered":"2024-09-20T22:59:14.159866+0000","last_clean":"2024-09-20T22:59:14.159866+0000","last_became_active":"2024-09-20T22:58:28.409911+0000","last_became_peered":"2024-09-20T22:58:28.409911+0000","last_unstale":"2024-09-20T22:59:14.159866+0000","last_undegraded":"2024-09-20T22:59:14.159866+0000","last_fullsized":"2024-09-20T22:59:14.159866+0000","mapping_epoch":1008,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":1009,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:58:54.109704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:58:54.109704+0000","last_clean_scrub_stamp":"2024-09-20T22:58:54.109704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T23:00:12.653208+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"77'128","reported_seq":1572,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:14.015832+0000","last_change":"2024-09-20T22:58:35.306441+0000","last_active":"2024-09-20T22:59:14.015832+0000","last_peered":"2024-09-20T22:59:14.015832+0000","last_clean":"2024-09-20T22:59:14.015832+0000","last_became_active":"2024-09-20T22:54:20.413191+0000","last_became_peered":"2024-09-20T22:54:20.413191+0000","last_unstale":"2024-09-20T22:59:14.015832+0000","last_undegraded":"2024-09-20T22:59:14.015832+0000","last_fullsized":"2024-09-20T22:59:14.015832+0000","mapping_epoch":779,"log_start":"77'128","ondisk_log_start":"77'128","created":89,"last_epoch_clean":780,"parent":"0.0","parent_split_bits":3,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:58:35.306366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:58:35.306366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:39.963250+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"828'30","reported_seq":1448,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:13.390973+0000","last_change":"2024-09-20T22:59:00.715919+0000","last_active":"2024-09-20T22:59:13.390973+0000","last_peered":"2024-09-20T22:59:13.390973+0000","last_clean":"2024-09-20T22:59:13.390973+0000","last_became_active":"2024-09-20T22:55:11.346018+0000","last_became_peered":"2024-09-20T22:55:11.346018+0000","last_unstale":"2024-09-20T22:59:13.390973+0000","last_undegraded":"2024-09-20T22:59:13.390973+0000","last_fullsized":"2024-09-20T22:59:13.390973+0000","mapping_epoch":830,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":831,"parent":"0.0","parent_split_bits":0,"last_scrub":"828'30","last_scrub_stamp":"2024-09-20T22:59:00.715824+0000","last_deep_scrub":"828'30","last_deep_scrub_stamp":"2024-09-20T22:55:08.705571+0000","last_clean_scrub_stamp":"2024-09-20T22:59:00.715824+0000","objects_scrubbed":6,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T23:00:03.113273+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":1365,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:13.389587+0000","last_change":"2024-09-20T22:58:04.615065+0000","last_active":"2024-09-20T22:59:13.389587+0000","last_peered":"2024-09-20T22:59:13.389587+0000","last_clean":"2024-09-20T22:59:13.389587+0000","last_became_active":"2024-09-20T22:57:49.410539+0000","last_became_peered":"2024-09-20T22:57:49.410539+0000","last_unstale":"2024-09-20T22:59:13.389587+0000","last_undegraded":"2024-09-20T22:59:13.389587+0000","last_fullsized":"2024-09-20T22:59:13.389587+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:58:04.614957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:44:19.834641+0000","last_clean_scrub_stamp":"2024-09-20T22:58:04.614957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:21.673315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"75'5","reported_seq":138,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:27.559273+0000","last_change":"2024-09-20T22:58:46.149273+0000","last_active":"2024-09-20T22:59:27.559273+0000","last_peered":"2024-09-20T22:59:27.559273+0000","last_clean":"2024-09-20T22:59:27.559273+0000","last_became_active":"2024-09-20T22:58:28.407748+0000","last_became_peered":"2024-09-20T22:58:28.407748+0000","last_unstale":"2024-09-20T22:59:27.559273+0000","last_undegraded":"2024-09-20T22:59:27.559273+0000","last_fullsized":"2024-09-20T22:59:27.559273+0000","mapping_epoch":1008,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":1009,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-09-20T22:58:46.149200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:36.130684+0000","last_clean_scrub_stamp":"2024-09-20T22:58:46.149200+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T23:00:06.719526+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00078631700000000003,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":422,"num_read_kb":417,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":2032,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:14.015765+0000","last_change":"2024-09-20T22:59:14.015586+0000","last_active":"2024-09-20T22:59:14.015765+0000","last_peered":"2024-09-20T22:59:14.015765+0000","last_clean":"2024-09-20T22:59:14.015765+0000","last_became_active":"2024-09-20T22:54:17.339805+0000","last_became_peered":"2024-09-20T22:54:17.339805+0000","last_unstale":"2024-09-20T22:59:14.015765+0000","last_undegraded":"2024-09-20T22:59:14.015765+0000","last_fullsized":"2024-09-20T22:59:14.015765+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:59:14.015458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:57:13.660520+0000","last_clean_scrub_stamp":"2024-09-20T22:59:14.015458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T23:00:18.904523+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":562,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:16.488882+0000","last_change":"2024-09-20T22:59:16.488912+0000","last_active":"2024-09-20T22:59:16.488882+0000","last_peered":"2024-09-20T22:59:16.488882+0000","last_clean":"2024-09-20T22:59:16.488882+0000","last_became_active":"2024-09-20T22:57:49.429412+0000","last_became_peered":"2024-09-20T22:57:49.429412+0000","last_unstale":"2024-09-20T22:59:16.488882+0000","last_undegraded":"2024-09-20T22:59:16.488882+0000","last_fullsized":"2024-09-20T22:59:16.488882+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:59:16.488818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:59:16.488818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T23:00:23.730775+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"1009'34","reported_seq":2417,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:23.400258+0000","last_change":"2024-09-20T22:58:28.445239+0000","last_active":"2024-09-20T22:59:23.400258+0000","last_peered":"2024-09-20T22:59:23.400258+0000","last_clean":"2024-09-20T22:59:23.400258+0000","last_became_active":"2024-09-20T22:58:28.398920+0000","last_became_peered":"2024-09-20T22:58:28.398920+0000","last_unstale":"2024-09-20T22:59:23.400258+0000","last_undegraded":"2024-09-20T22:59:23.400258+0000","last_fullsized":"2024-09-20T22:59:23.400258+0000","mapping_epoch":1008,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1009,"parent":"0.0","parent_split_bits":0,"last_scrub":"888'33","last_scrub_stamp":"2024-09-20T22:58:26.407098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:58:26.407098+0000","objects_scrubbed":8,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:46.923321+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":924,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"67'1","reported_seq":1904,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:13.486322+0000","last_change":"2024-09-20T22:59:04.932715+0000","last_active":"2024-09-20T22:59:13.486322+0000","last_peered":"2024-09-20T22:59:13.486322+0000","last_clean":"2024-09-20T22:59:13.486322+0000","last_became_active":"2024-09-20T22:57:49.397191+0000","last_became_peered":"2024-09-20T22:57:49.397191+0000","last_unstale":"2024-09-20T22:59:13.486322+0000","last_undegraded":"2024-09-20T22:59:13.486322+0000","last_fullsized":"2024-09-20T22:59:13.486322+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-09-20T22:59:04.932639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:59:04.932639+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T23:00:32.833984+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"77'128","reported_seq":1363,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:14.159296+0000","last_change":"2024-09-20T22:59:03.275326+0000","last_active":"2024-09-20T22:59:14.159296+0000","last_peered":"2024-09-20T22:59:14.159296+0000","last_clean":"2024-09-20T22:59:14.159296+0000","last_became_active":"2024-09-20T22:58:31.407541+0000","last_became_peered":"2024-09-20T22:58:31.407541+0000","last_unstale":"2024-09-20T22:59:14.159296+0000","last_undegraded":"2024-09-20T22:59:14.159296+0000","last_fullsized":"2024-09-20T22:59:14.159296+0000","mapping_epoch":1011,"log_start":"77'128","ondisk_log_start":"77'128","created":19,"last_epoch_clean":1012,"parent":"0.0","parent_split_bits":0,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:59:03.275234+0000","last_deep_scrub":"77'128","last_deep_scrub_stamp":"2024-09-20T22:47:22.417099+0000","last_clean_scrub_stamp":"2024-09-20T22:59:03.275234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T23:00:28.479860+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":65,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:14.159924+0000","last_change":"2024-09-20T22:58:28.417931+0000","last_active":"2024-09-20T22:59:14.159924+0000","last_peered":"2024-09-20T22:59:14.159924+0000","last_clean":"2024-09-20T22:59:14.159924+0000","last_became_active":"2024-09-20T22:58:28.417612+0000","last_became_peered":"2024-09-20T22:58:28.417612+0000","last_unstale":"2024-09-20T22:59:14.159924+0000","last_undegraded":"2024-09-20T22:59:14.159924+0000","last_fullsized":"2024-09-20T22:59:14.159924+0000","mapping_epoch":1008,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":1009,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:58:02.248808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:58:02.248808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:20.300201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":2017,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:14.015589+0000","last_change":"2024-09-20T22:58:31.907856+0000","last_active":"2024-09-20T22:59:14.015589+0000","last_peered":"2024-09-20T22:59:14.015589+0000","last_clean":"2024-09-20T22:59:14.015589+0000","last_became_active":"2024-09-20T22:57:49.410488+0000","last_became_peered":"2024-09-20T22:57:49.410488+0000","last_unstale":"2024-09-20T22:59:14.015589+0000","last_undegraded":"2024-09-20T22:59:14.015589+0000","last_fullsized":"2024-09-20T22:59:14.015589+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:58:31.907778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:58:31.907778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:38.295707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"850'24","reported_seq":587,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:13.391166+0000","last_change":"2024-09-20T22:59:08.152719+0000","last_active":"2024-09-20T22:59:13.391166+0000","last_peered":"2024-09-20T22:59:13.391166+0000","last_clean":"2024-09-20T22:59:13.391166+0000","last_became_active":"2024-09-20T22:58:48.609688+0000","last_became_peered":"2024-09-20T22:58:48.609688+0000","last_unstale":"2024-09-20T22:59:13.391166+0000","last_undegraded":"2024-09-20T22:59:13.391166+0000","last_fullsized":"2024-09-20T22:59:13.391166+0000","mapping_epoch":1022,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1023,"parent":"0.0","parent_split_bits":0,"last_scrub":"850'24","last_scrub_stamp":"2024-09-20T22:59:08.152583+0000","last_deep_scrub":"850'24","last_deep_scrub_stamp":"2024-09-20T22:55:30.779018+0000","last_clean_scrub_stamp":"2024-09-20T22:59:08.152583+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T23:00:15.420325+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":23,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":933,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:13.390084+0000","last_change":"2024-09-20T22:58:49.166015+0000","last_active":"2024-09-20T22:59:13.390084+0000","last_peered":"2024-09-20T22:59:13.390084+0000","last_clean":"2024-09-20T22:59:13.390084+0000","last_became_active":"2024-09-20T22:58:49.164972+0000","last_became_peered":"2024-09-20T22:58:49.164972+0000","last_unstale":"2024-09-20T22:59:13.390084+0000","last_undegraded":"2024-09-20T22:59:13.390084+0000","last_fullsized":"2024-09-20T22:59:13.390084+0000","mapping_epoch":1022,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1023,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:58:30.124573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:58:30.124573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:36.161664+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":2011,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:14.015507+0000","last_change":"2024-09-20T22:59:11.957364+0000","last_active":"2024-09-20T22:59:14.015507+0000","last_peered":"2024-09-20T22:59:14.015507+0000","last_clean":"2024-09-20T22:59:14.015507+0000","last_became_active":"2024-09-20T22:48:05.233778+0000","last_became_peered":"2024-09-20T22:48:05.233778+0000","last_unstale":"2024-09-20T22:59:14.015507+0000","last_undegraded":"2024-09-20T22:59:14.015507+0000","last_fullsized":"2024-09-20T22:59:14.015507+0000","mapping_epoch":404,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":405,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:59:11.957292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:54:47.166836+0000","last_clean_scrub_stamp":"2024-09-20T22:59:11.957292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T23:00:23.803711+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":1865,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:13.486674+0000","last_change":"2024-09-20T22:57:49.427590+0000","last_active":"2024-09-20T22:59:13.486674+0000","last_peered":"2024-09-20T22:59:13.486674+0000","last_clean":"2024-09-20T22:59:13.486674+0000","last_became_active":"2024-09-20T22:57:49.425373+0000","last_became_peered":"2024-09-20T22:57:49.425373+0000","last_unstale":"2024-09-20T22:59:13.486674+0000","last_undegraded":"2024-09-20T22:59:13.486674+0000","last_fullsized":"2024-09-20T22:59:13.486674+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:56:46.453177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:48:40.127123+0000","last_clean_scrub_stamp":"2024-09-20T22:56:46.453177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"77'128","reported_seq":2240,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:14.014441+0000","last_change":"2024-09-20T22:58:55.396348+0000","last_active":"2024-09-20T22:59:14.014441+0000","last_peered":"2024-09-20T22:59:14.014441+0000","last_clean":"2024-09-20T22:59:14.014441+0000","last_became_active":"2024-09-20T22:58:55.396082+0000","last_became_peered":"2024-09-20T22:58:55.396082+0000","last_unstale":"2024-09-20T22:59:14.014441+0000","last_undegraded":"2024-09-20T22:59:14.014441+0000","last_fullsized":"2024-09-20T22:59:14.014441+0000","mapping_epoch":1025,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":1026,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:58:21.144759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:58:21.144759+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:31.432198+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"625'31","reported_seq":1028,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:13.391179+0000","last_change":"2024-09-20T22:59:06.149910+0000","last_active":"2024-09-20T22:59:13.391179+0000","last_peered":"2024-09-20T22:59:13.391179+0000","last_clean":"2024-09-20T22:59:13.391179+0000","last_became_active":"2024-09-20T22:58:48.608367+0000","last_became_peered":"2024-09-20T22:58:48.608367+0000","last_unstale":"2024-09-20T22:59:13.391179+0000","last_undegraded":"2024-09-20T22:59:13.391179+0000","last_fullsized":"2024-09-20T22:59:13.391179+0000","mapping_epoch":1022,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1023,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'31","last_scrub_stamp":"2024-09-20T22:59:06.149783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:59:06.149783+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T23:00:29.230470+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":49,"num_read_kb":39,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":1577,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:13.391155+0000","last_change":"2024-09-20T22:58:38.164880+0000","last_active":"2024-09-20T22:59:13.391155+0000","last_peered":"2024-09-20T22:59:13.391155+0000","last_clean":"2024-09-20T22:59:13.391155+0000","last_became_active":"2024-09-20T22:54:17.332636+0000","last_became_peered":"2024-09-20T22:54:17.332636+0000","last_unstale":"2024-09-20T22:59:13.391155+0000","last_undegraded":"2024-09-20T22:59:13.391155+0000","last_fullsized":"2024-09-20T22:59:13.391155+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:58:38.164809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:35.492562+0000","last_clean_scrub_stamp":"2024-09-20T22:58:38.164809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:38.777158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":553,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:14.016057+0000","last_change":"2024-09-20T22:58:57.350393+0000","last_active":"2024-09-20T22:59:14.016057+0000","last_peered":"2024-09-20T22:59:14.016057+0000","last_clean":"2024-09-20T22:59:14.016057+0000","last_became_active":"2024-09-20T22:54:17.349144+0000","last_became_peered":"2024-09-20T22:54:17.349144+0000","last_unstale":"2024-09-20T22:59:14.016057+0000","last_undegraded":"2024-09-20T22:59:14.016057+0000","last_fullsized":"2024-09-20T22:59:14.016057+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:58:57.350319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:58:57.350319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T23:00:22.329053+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":2040,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:13.390390+0000","last_change":"2024-09-20T22:58:36.156131+0000","last_active":"2024-09-20T22:59:13.390390+0000","last_peered":"2024-09-20T22:59:13.390390+0000","last_clean":"2024-09-20T22:59:13.390390+0000","last_became_active":"2024-09-20T22:57:49.410743+0000","last_became_peered":"2024-09-20T22:57:49.410743+0000","last_unstale":"2024-09-20T22:59:13.390390+0000","last_undegraded":"2024-09-20T22:59:13.390390+0000","last_fullsized":"2024-09-20T22:59:13.390390+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:58:36.156047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:58:36.156047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:47.621359+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"77'128","reported_seq":2046,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:14.160023+0000","last_change":"2024-09-20T22:58:38.128312+0000","last_active":"2024-09-20T22:59:14.160023+0000","last_peered":"2024-09-20T22:59:14.160023+0000","last_clean":"2024-09-20T22:59:14.160023+0000","last_became_active":"2024-09-20T22:58:31.410332+0000","last_became_peered":"2024-09-20T22:58:31.410332+0000","last_unstale":"2024-09-20T22:59:14.160023+0000","last_undegraded":"2024-09-20T22:59:14.160023+0000","last_fullsized":"2024-09-20T22:59:14.160023+0000","mapping_epoch":1011,"log_start":"0'0","ondisk_log_start":"0'0","created":89,"last_epoch_clean":1012,"parent":"0.0","parent_split_bits":4,"last_scrub":"77'128","last_scrub_stamp":"2024-09-20T22:58:38.128211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:39:22.222297+0000","last_clean_scrub_stamp":"2024-09-20T22:58:38.128211+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T23:00:03.049766+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"877'25","reported_seq":626,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:13.390789+0000","last_change":"2024-09-20T22:58:50.255914+0000","last_active":"2024-09-20T22:59:13.390789+0000","last_peered":"2024-09-20T22:59:13.390789+0000","last_clean":"2024-09-20T22:59:13.390789+0000","last_became_active":"2024-09-20T22:58:48.609782+0000","last_became_peered":"2024-09-20T22:58:48.609782+0000","last_unstale":"2024-09-20T22:59:13.390789+0000","last_undegraded":"2024-09-20T22:59:13.390789+0000","last_fullsized":"2024-09-20T22:59:13.390789+0000","mapping_epoch":1022,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1023,"parent":"0.0","parent_split_bits":0,"last_scrub":"877'25","last_scrub_stamp":"2024-09-20T22:58:50.255758+0000","last_deep_scrub":"877'25","last_deep_scrub_stamp":"2024-09-20T22:55:57.521557+0000","last_clean_scrub_stamp":"2024-09-20T22:58:50.255758+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:59.724961+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":28,"num_read_kb":23,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":1897,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:13.486292+0000","last_change":"2024-09-20T22:59:11.061184+0000","last_active":"2024-09-20T22:59:13.486292+0000","last_peered":"2024-09-20T22:59:13.486292+0000","last_clean":"2024-09-20T22:59:13.486292+0000","last_became_active":"2024-09-20T22:57:49.432735+0000","last_became_peered":"2024-09-20T22:57:49.432735+0000","last_unstale":"2024-09-20T22:59:13.486292+0000","last_undegraded":"2024-09-20T22:59:13.486292+0000","last_fullsized":"2024-09-20T22:59:13.486292+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:59:11.061060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:55:58.553666+0000","last_clean_scrub_stamp":"2024-09-20T22:59:11.061060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T23:00:32.245416+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"1009'11","reported_seq":2267,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:23.399593+0000","last_change":"2024-09-20T22:59:11.140265+0000","last_active":"2024-09-20T22:59:23.399593+0000","last_peered":"2024-09-20T22:59:23.399593+0000","last_clean":"2024-09-20T22:59:23.399593+0000","last_became_active":"2024-09-20T22:58:28.415972+0000","last_became_peered":"2024-09-20T22:58:28.415972+0000","last_unstale":"2024-09-20T22:59:23.399593+0000","last_undegraded":"2024-09-20T22:59:23.399593+0000","last_fullsized":"2024-09-20T22:59:23.399593+0000","mapping_epoch":1008,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":1009,"parent":"0.0","parent_split_bits":0,"last_scrub":"1009'11","last_scrub_stamp":"2024-09-20T22:59:11.140112+0000","last_deep_scrub":"1009'11","last_deep_scrub_stamp":"2024-09-20T22:59:11.140112+0000","last_clean_scrub_stamp":"2024-09-20T22:59:11.140112+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T23:00:25.669832+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":262,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:13.390779+0000","last_change":"2024-09-20T22:58:48.609604+0000","last_active":"2024-09-20T22:59:13.390779+0000","last_peered":"2024-09-20T22:59:13.390779+0000","last_clean":"2024-09-20T22:59:13.390779+0000","last_became_active":"2024-09-20T22:58:48.609096+0000","last_became_peered":"2024-09-20T22:58:48.609096+0000","last_unstale":"2024-09-20T22:59:13.390779+0000","last_undegraded":"2024-09-20T22:59:13.390779+0000","last_fullsized":"2024-09-20T22:59:13.390779+0000","mapping_epoch":1022,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":1023,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:58:47.501202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:58:47.501202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:49.521274+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"625'15","reported_seq":1417,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:13.390233+0000","last_change":"2024-09-20T22:58:24.514509+0000","last_active":"2024-09-20T22:59:13.390233+0000","last_peered":"2024-09-20T22:59:13.390233+0000","last_clean":"2024-09-20T22:59:13.390233+0000","last_became_active":"2024-09-20T22:54:17.340998+0000","last_became_peered":"2024-09-20T22:54:17.340998+0000","last_unstale":"2024-09-20T22:59:13.390233+0000","last_undegraded":"2024-09-20T22:59:13.390233+0000","last_fullsized":"2024-09-20T22:59:13.390233+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'15","last_scrub_stamp":"2024-09-20T22:58:24.514380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:58:24.514380+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:45.247411+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":2016,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:14.015276+0000","last_change":"2024-09-20T22:58:55.026655+0000","last_active":"2024-09-20T22:59:14.015276+0000","last_peered":"2024-09-20T22:59:14.015276+0000","last_clean":"2024-09-20T22:59:14.015276+0000","last_became_active":"2024-09-20T22:57:49.425707+0000","last_became_peered":"2024-09-20T22:57:49.425707+0000","last_unstale":"2024-09-20T22:59:14.015276+0000","last_undegraded":"2024-09-20T22:59:14.015276+0000","last_fullsized":"2024-09-20T22:59:14.015276+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:58:55.026573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:58:55.026573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T23:00:20.516790+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":2049,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:13.391278+0000","last_change":"2024-09-20T22:58:56.209343+0000","last_active":"2024-09-20T22:59:13.391278+0000","last_peered":"2024-09-20T22:59:13.391278+0000","last_clean":"2024-09-20T22:59:13.391278+0000","last_became_active":"2024-09-20T22:57:49.411210+0000","last_became_peered":"2024-09-20T22:57:49.411210+0000","last_unstale":"2024-09-20T22:59:13.391278+0000","last_undegraded":"2024-09-20T22:59:13.391278+0000","last_fullsized":"2024-09-20T22:59:13.391278+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:58:56.209272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:58:56.209272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T23:00:11.157085+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":1371,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:13.389616+0000","last_change":"2024-09-20T22:58:43.672949+0000","last_active":"2024-09-20T22:59:13.389616+0000","last_peered":"2024-09-20T22:59:13.389616+0000","last_clean":"2024-09-20T22:59:13.389616+0000","last_became_active":"2024-09-20T22:58:28.404456+0000","last_became_peered":"2024-09-20T22:58:28.404456+0000","last_unstale":"2024-09-20T22:59:13.389616+0000","last_undegraded":"2024-09-20T22:59:13.389616+0000","last_fullsized":"2024-09-20T22:59:13.389616+0000","mapping_epoch":1008,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":1009,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:58:43.672863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:58:43.672863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:48.874288+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"625'44","reported_seq":2194,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:13.746586+0000","last_change":"2024-09-20T22:58:43.553049+0000","last_active":"2024-09-20T22:59:13.746586+0000","last_peered":"2024-09-20T22:59:13.746586+0000","last_clean":"2024-09-20T22:59:13.746586+0000","last_became_active":"2024-09-20T22:58:28.399556+0000","last_became_peered":"2024-09-20T22:58:28.399556+0000","last_unstale":"2024-09-20T22:59:13.746586+0000","last_undegraded":"2024-09-20T22:59:13.746586+0000","last_fullsized":"2024-09-20T22:59:13.746586+0000","mapping_epoch":1008,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1009,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'44","last_scrub_stamp":"2024-09-20T22:58:43.552928+0000","last_deep_scrub":"113'28","last_deep_scrub_stamp":"2024-09-20T22:43:12.725900+0000","last_clean_scrub_stamp":"2024-09-20T22:58:43.552928+0000","objects_scrubbed":9,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T23:00:08.433363+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":65,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:14.159848+0000","last_change":"2024-09-20T22:58:28.419770+0000","last_active":"2024-09-20T22:59:14.159848+0000","last_peered":"2024-09-20T22:59:14.159848+0000","last_clean":"2024-09-20T22:59:14.159848+0000","last_became_active":"2024-09-20T22:58:28.419355+0000","last_became_peered":"2024-09-20T22:58:28.419355+0000","last_unstale":"2024-09-20T22:59:14.159848+0000","last_undegraded":"2024-09-20T22:59:14.159848+0000","last_fullsized":"2024-09-20T22:59:14.159848+0000","mapping_epoch":1008,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1009,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:58:21.787147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:58:21.787147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:41.463239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":2027,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:13.389844+0000","last_change":"2024-09-20T22:58:26.172414+0000","last_active":"2024-09-20T22:59:13.389844+0000","last_peered":"2024-09-20T22:59:13.389844+0000","last_clean":"2024-09-20T22:59:13.389844+0000","last_became_active":"2024-09-20T22:47:49.223220+0000","last_became_peered":"2024-09-20T22:47:49.223220+0000","last_unstale":"2024-09-20T22:59:13.389844+0000","last_undegraded":"2024-09-20T22:59:13.389844+0000","last_fullsized":"2024-09-20T22:59:13.389844+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:58:26.172290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:54:21.496068+0000","last_clean_scrub_stamp":"2024-09-20T22:58:26.172290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:38.945178+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":73,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:14.159219+0000","last_change":"2024-09-20T22:58:32.148797+0000","last_active":"2024-09-20T22:59:14.159219+0000","last_peered":"2024-09-20T22:59:14.159219+0000","last_clean":"2024-09-20T22:59:14.159219+0000","last_became_active":"2024-09-20T22:58:28.407140+0000","last_became_peered":"2024-09-20T22:58:28.407140+0000","last_unstale":"2024-09-20T22:59:14.159219+0000","last_undegraded":"2024-09-20T22:59:14.159219+0000","last_fullsized":"2024-09-20T22:59:14.159219+0000","mapping_epoch":1008,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":1009,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:58:32.148715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:55:54.870627+0000","last_clean_scrub_stamp":"2024-09-20T22:58:32.148715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:54.796373+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"686'23","reported_seq":1013,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:13.390205+0000","last_change":"2024-09-20T22:59:13.118143+0000","last_active":"2024-09-20T22:59:13.390205+0000","last_peered":"2024-09-20T22:59:13.390205+0000","last_clean":"2024-09-20T22:59:13.390205+0000","last_became_active":"2024-09-20T22:58:48.608668+0000","last_became_peered":"2024-09-20T22:58:48.608668+0000","last_unstale":"2024-09-20T22:59:13.390205+0000","last_undegraded":"2024-09-20T22:59:13.390205+0000","last_fullsized":"2024-09-20T22:59:13.390205+0000","mapping_epoch":1022,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1023,"parent":"0.0","parent_split_bits":0,"last_scrub":"686'23","last_scrub_stamp":"2024-09-20T22:59:13.118048+0000","last_deep_scrub":"686'23","last_deep_scrub_stamp":"2024-09-20T22:52:46.768812+0000","last_clean_scrub_stamp":"2024-09-20T22:59:13.118048+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T23:00:25.994963+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":21,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":1100,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":73,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:24.562635+0000","last_change":"2024-09-20T22:59:24.562657+0000","last_active":"2024-09-20T22:59:24.562635+0000","last_peered":"2024-09-20T22:59:24.562635+0000","last_clean":"2024-09-20T22:59:24.562635+0000","last_became_active":"2024-09-20T22:58:28.411334+0000","last_became_peered":"2024-09-20T22:58:28.411334+0000","last_unstale":"2024-09-20T22:59:24.562635+0000","last_undegraded":"2024-09-20T22:59:24.562635+0000","last_fullsized":"2024-09-20T22:59:24.562635+0000","mapping_epoch":1008,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1009,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:59:24.562570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:46:52.465940+0000","last_clean_scrub_stamp":"2024-09-20T22:59:24.562570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T23:00:46.044367+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":2048,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:13.391015+0000","last_change":"2024-09-20T22:58:40.224975+0000","last_active":"2024-09-20T22:59:13.391015+0000","last_peered":"2024-09-20T22:59:13.391015+0000","last_clean":"2024-09-20T22:59:13.391015+0000","last_became_active":"2024-09-20T22:54:17.340269+0000","last_became_peered":"2024-09-20T22:54:17.340269+0000","last_unstale":"2024-09-20T22:59:13.391015+0000","last_undegraded":"2024-09-20T22:59:13.391015+0000","last_fullsized":"2024-09-20T22:59:13.391015+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:58:40.224818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:53:07.104891+0000","last_clean_scrub_stamp":"2024-09-20T22:58:40.224818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T23:00:06.212292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":1897,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:13.486410+0000","last_change":"2024-09-20T22:58:31.875658+0000","last_active":"2024-09-20T22:59:13.486410+0000","last_peered":"2024-09-20T22:59:13.486410+0000","last_clean":"2024-09-20T22:59:13.486410+0000","last_became_active":"2024-09-20T22:58:28.413507+0000","last_became_peered":"2024-09-20T22:58:28.413507+0000","last_unstale":"2024-09-20T22:59:13.486410+0000","last_undegraded":"2024-09-20T22:59:13.486410+0000","last_fullsized":"2024-09-20T22:59:13.486410+0000","mapping_epoch":1008,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":1009,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:58:31.875534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:55.170379+0000","last_clean_scrub_stamp":"2024-09-20T22:58:31.875534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:51.255952+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"625'27","reported_seq":2091,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:14.014497+0000","last_change":"2024-09-20T22:59:02.973258+0000","last_active":"2024-09-20T22:59:14.014497+0000","last_peered":"2024-09-20T22:59:14.014497+0000","last_clean":"2024-09-20T22:59:14.014497+0000","last_became_active":"2024-09-20T22:58:28.405911+0000","last_became_peered":"2024-09-20T22:58:28.405911+0000","last_unstale":"2024-09-20T22:59:14.014497+0000","last_undegraded":"2024-09-20T22:59:14.014497+0000","last_fullsized":"2024-09-20T22:59:14.014497+0000","mapping_epoch":1008,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1009,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'27","last_scrub_stamp":"2024-09-20T22:59:02.973154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:59:02.973154+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T23:00:16.168389+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'2","reported_seq":71,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:14.160221+0000","last_change":"2024-09-20T22:58:28.439602+0000","last_active":"2024-09-20T22:59:14.160221+0000","last_peered":"2024-09-20T22:59:14.160221+0000","last_clean":"2024-09-20T22:59:14.160221+0000","last_became_active":"2024-09-20T22:58:28.409991+0000","last_became_peered":"2024-09-20T22:58:28.409991+0000","last_unstale":"2024-09-20T22:59:14.160221+0000","last_undegraded":"2024-09-20T22:59:14.160221+0000","last_fullsized":"2024-09-20T22:59:14.160221+0000","mapping_epoch":1008,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1009,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-09-20T22:58:20.423976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:58:20.423976+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:43.884021+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"1009'12","reported_seq":90,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:23.399996+0000","last_change":"2024-09-20T22:59:04.289724+0000","last_active":"2024-09-20T22:59:23.399996+0000","last_peered":"2024-09-20T22:59:23.399996+0000","last_clean":"2024-09-20T22:59:23.399996+0000","last_became_active":"2024-09-20T22:58:28.410156+0000","last_became_peered":"2024-09-20T22:58:28.410156+0000","last_unstale":"2024-09-20T22:59:23.399996+0000","last_undegraded":"2024-09-20T22:59:23.399996+0000","last_fullsized":"2024-09-20T22:59:23.399996+0000","mapping_epoch":1008,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":1009,"parent":"0.0","parent_split_bits":0,"last_scrub":"1009'12","last_scrub_stamp":"2024-09-20T22:59:04.289629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:59:04.289629+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T23:00:19.811072+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":2008,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:14.014695+0000","last_change":"2024-09-20T22:59:08.015203+0000","last_active":"2024-09-20T22:59:14.014695+0000","last_peered":"2024-09-20T22:59:14.014695+0000","last_clean":"2024-09-20T22:59:14.014695+0000","last_became_active":"2024-09-20T22:47:49.227569+0000","last_became_peered":"2024-09-20T22:47:49.227569+0000","last_unstale":"2024-09-20T22:59:14.014695+0000","last_undegraded":"2024-09-20T22:59:14.014695+0000","last_fullsized":"2024-09-20T22:59:14.014695+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:59:08.015117+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:59:08.015117+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T23:00:22.604964+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":2049,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:18.560067+0000","last_change":"2024-09-20T22:59:18.560091+0000","last_active":"2024-09-20T22:59:18.560067+0000","last_peered":"2024-09-20T22:59:18.560067+0000","last_clean":"2024-09-20T22:59:18.560067+0000","last_became_active":"2024-09-20T22:47:49.213292+0000","last_became_peered":"2024-09-20T22:47:49.213292+0000","last_unstale":"2024-09-20T22:59:18.560067+0000","last_undegraded":"2024-09-20T22:59:18.560067+0000","last_fullsized":"2024-09-20T22:59:18.560067+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:59:18.560002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:59:18.560002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T23:00:29.853798+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"625'12","reported_seq":1963,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:13.486779+0000","last_change":"2024-09-20T22:58:28.449237+0000","last_active":"2024-09-20T22:59:13.486779+0000","last_peered":"2024-09-20T22:59:13.486779+0000","last_clean":"2024-09-20T22:59:13.486779+0000","last_became_active":"2024-09-20T22:58:28.408845+0000","last_became_peered":"2024-09-20T22:58:28.408845+0000","last_unstale":"2024-09-20T22:59:13.486779+0000","last_undegraded":"2024-09-20T22:59:13.486779+0000","last_fullsized":"2024-09-20T22:59:13.486779+0000","mapping_epoch":1008,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1009,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'12","last_scrub_stamp":"2024-09-20T22:58:00.707335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:58:00.707335+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:22.874486+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":2020,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:14.015439+0000","last_change":"2024-09-20T22:58:22.783603+0000","last_active":"2024-09-20T22:59:14.015439+0000","last_peered":"2024-09-20T22:59:14.015439+0000","last_clean":"2024-09-20T22:59:14.015439+0000","last_became_active":"2024-09-20T22:57:49.408529+0000","last_became_peered":"2024-09-20T22:57:49.408529+0000","last_unstale":"2024-09-20T22:59:14.015439+0000","last_undegraded":"2024-09-20T22:59:14.015439+0000","last_fullsized":"2024-09-20T22:59:14.015439+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:58:22.783512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:09.705632+0000","last_clean_scrub_stamp":"2024-09-20T22:58:22.783512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:45.356564+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"74'1","reported_seq":77,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:14.160035+0000","last_change":"2024-09-20T22:58:49.165424+0000","last_active":"2024-09-20T22:59:14.160035+0000","last_peered":"2024-09-20T22:59:14.160035+0000","last_clean":"2024-09-20T22:59:14.160035+0000","last_became_active":"2024-09-20T22:58:28.411228+0000","last_became_peered":"2024-09-20T22:58:28.411228+0000","last_unstale":"2024-09-20T22:59:14.160035+0000","last_undegraded":"2024-09-20T22:59:14.160035+0000","last_fullsized":"2024-09-20T22:59:14.160035+0000","mapping_epoch":1008,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":1009,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-09-20T22:58:49.165295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:58:49.165295+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T23:00:04.313986+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"625'15","reported_seq":1409,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:13.390868+0000","last_change":"2024-09-20T22:58:28.439708+0000","last_active":"2024-09-20T22:59:13.390868+0000","last_peered":"2024-09-20T22:59:13.390868+0000","last_clean":"2024-09-20T22:59:13.390868+0000","last_became_active":"2024-09-20T22:58:28.405850+0000","last_became_peered":"2024-09-20T22:58:28.405850+0000","last_unstale":"2024-09-20T22:59:13.390868+0000","last_undegraded":"2024-09-20T22:59:13.390868+0000","last_fullsized":"2024-09-20T22:59:13.390868+0000","mapping_epoch":1008,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1009,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'15","last_scrub_stamp":"2024-09-20T22:58:19.403482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:58:19.403482+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:33.775613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":73,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:14.160206+0000","last_change":"2024-09-20T22:58:34.150809+0000","last_active":"2024-09-20T22:59:14.160206+0000","last_peered":"2024-09-20T22:59:14.160206+0000","last_clean":"2024-09-20T22:59:14.160206+0000","last_became_active":"2024-09-20T22:58:28.410707+0000","last_became_peered":"2024-09-20T22:58:28.410707+0000","last_unstale":"2024-09-20T22:59:14.160206+0000","last_undegraded":"2024-09-20T22:59:14.160206+0000","last_fullsized":"2024-09-20T22:59:14.160206+0000","mapping_epoch":1008,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1009,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:58:34.150733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:58:34.150733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:35.124109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":1909,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:14.161396+0000","last_change":"2024-09-20T22:59:14.161420+0000","last_active":"2024-09-20T22:59:14.161396+0000","last_peered":"2024-09-20T22:59:14.161396+0000","last_clean":"2024-09-20T22:59:14.161396+0000","last_became_active":"2024-09-20T22:58:28.406337+0000","last_became_peered":"2024-09-20T22:58:28.406337+0000","last_unstale":"2024-09-20T22:59:14.161396+0000","last_undegraded":"2024-09-20T22:59:14.161396+0000","last_fullsized":"2024-09-20T22:59:14.161396+0000","mapping_epoch":1008,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":1009,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:59:14.161329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:59:14.161329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T23:00:16.069258+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":2055,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:13.391766+0000","last_change":"2024-09-20T22:58:37.129420+0000","last_active":"2024-09-20T22:59:13.391766+0000","last_peered":"2024-09-20T22:59:13.391766+0000","last_clean":"2024-09-20T22:59:13.391766+0000","last_became_active":"2024-09-20T22:58:28.408803+0000","last_became_peered":"2024-09-20T22:58:28.408803+0000","last_unstale":"2024-09-20T22:59:13.391766+0000","last_undegraded":"2024-09-20T22:59:13.391766+0000","last_fullsized":"2024-09-20T22:59:13.391766+0000","mapping_epoch":1008,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":1009,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:58:37.129346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:53:58.169894+0000","last_clean_scrub_stamp":"2024-09-20T22:58:37.129346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:55.758007+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"625'15","reported_seq":2069,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:14.015306+0000","last_change":"2024-09-20T22:58:45.916686+0000","last_active":"2024-09-20T22:59:14.015306+0000","last_peered":"2024-09-20T22:59:14.015306+0000","last_clean":"2024-09-20T22:59:14.015306+0000","last_became_active":"2024-09-20T22:58:28.406352+0000","last_became_peered":"2024-09-20T22:58:28.406352+0000","last_unstale":"2024-09-20T22:59:14.015306+0000","last_undegraded":"2024-09-20T22:59:14.015306+0000","last_fullsized":"2024-09-20T22:59:14.015306+0000","mapping_epoch":1008,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1009,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'15","last_scrub_stamp":"2024-09-20T22:58:45.916597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:58:45.916597+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:50.823985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":1379,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:13.389884+0000","last_change":"2024-09-20T22:58:35.606336+0000","last_active":"2024-09-20T22:59:13.389884+0000","last_peered":"2024-09-20T22:59:13.389884+0000","last_clean":"2024-09-20T22:59:13.389884+0000","last_became_active":"2024-09-20T22:58:28.405095+0000","last_became_peered":"2024-09-20T22:58:28.405095+0000","last_unstale":"2024-09-20T22:59:13.389884+0000","last_undegraded":"2024-09-20T22:59:13.389884+0000","last_fullsized":"2024-09-20T22:59:13.389884+0000","mapping_epoch":1008,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1009,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:58:35.606262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:58:35.606262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T23:00:01.674790+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"1009'12","reported_seq":2111,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:18.399554+0000","last_change":"2024-09-20T22:58:34.905724+0000","last_active":"2024-09-20T22:59:18.399554+0000","last_peered":"2024-09-20T22:59:18.399554+0000","last_clean":"2024-09-20T22:59:18.399554+0000","last_became_active":"2024-09-20T22:58:28.411994+0000","last_became_peered":"2024-09-20T22:58:28.411994+0000","last_unstale":"2024-09-20T22:59:18.399554+0000","last_undegraded":"2024-09-20T22:59:18.399554+0000","last_fullsized":"2024-09-20T22:59:18.399554+0000","mapping_epoch":1008,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":1009,"parent":"0.0","parent_split_bits":0,"last_scrub":"1009'12","last_scrub_stamp":"2024-09-20T22:58:34.905643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:58:34.905643+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:47.284009+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":932,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:13.391221+0000","last_change":"2024-09-20T22:59:10.134268+0000","last_active":"2024-09-20T22:59:13.391221+0000","last_peered":"2024-09-20T22:59:13.391221+0000","last_clean":"2024-09-20T22:59:13.391221+0000","last_became_active":"2024-09-20T22:58:48.610883+0000","last_became_peered":"2024-09-20T22:58:48.610883+0000","last_unstale":"2024-09-20T22:59:13.391221+0000","last_undegraded":"2024-09-20T22:59:13.391221+0000","last_fullsized":"2024-09-20T22:59:13.391221+0000","mapping_epoch":1022,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":1023,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:59:10.134177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:59:10.134177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T23:00:25.594502+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"625'20","reported_seq":1999,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:13.486280+0000","last_change":"2024-09-20T22:59:02.907018+0000","last_active":"2024-09-20T22:59:13.486280+0000","last_peered":"2024-09-20T22:59:13.486280+0000","last_clean":"2024-09-20T22:59:13.486280+0000","last_became_active":"2024-09-20T22:58:28.411505+0000","last_became_peered":"2024-09-20T22:58:28.411505+0000","last_unstale":"2024-09-20T22:59:13.486280+0000","last_undegraded":"2024-09-20T22:59:13.486280+0000","last_fullsized":"2024-09-20T22:59:13.486280+0000","mapping_epoch":1008,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1009,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'20","last_scrub_stamp":"2024-09-20T22:59:02.906940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:59:02.906940+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T23:00:03.205418+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":65,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:14.159700+0000","last_change":"2024-09-20T22:58:28.410981+0000","last_active":"2024-09-20T22:59:14.159700+0000","last_peered":"2024-09-20T22:59:14.159700+0000","last_clean":"2024-09-20T22:59:14.159700+0000","last_became_active":"2024-09-20T22:58:28.410106+0000","last_became_peered":"2024-09-20T22:58:28.410106+0000","last_unstale":"2024-09-20T22:59:14.159700+0000","last_undegraded":"2024-09-20T22:59:14.159700+0000","last_fullsized":"2024-09-20T22:59:14.159700+0000","mapping_epoch":1008,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1009,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:58:18.200504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:58:18.200504+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:39.182436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"778'12","reported_seq":2274,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:23.399992+0000","last_change":"2024-09-20T22:59:16.941628+0000","last_active":"2024-09-20T22:59:23.399992+0000","last_peered":"2024-09-20T22:59:23.399992+0000","last_clean":"2024-09-20T22:59:23.399992+0000","last_became_active":"2024-09-20T22:54:17.341014+0000","last_became_peered":"2024-09-20T22:54:17.341014+0000","last_unstale":"2024-09-20T22:59:23.399992+0000","last_undegraded":"2024-09-20T22:59:23.399992+0000","last_fullsized":"2024-09-20T22:59:23.399992+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"778'12","last_scrub_stamp":"2024-09-20T22:59:16.941530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:59:16.941530+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T23:00:43.942982+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":1370,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:13.390741+0000","last_change":"2024-09-20T22:58:52.745765+0000","last_active":"2024-09-20T22:59:13.390741+0000","last_peered":"2024-09-20T22:59:13.390741+0000","last_clean":"2024-09-20T22:59:13.390741+0000","last_became_active":"2024-09-20T22:58:28.405530+0000","last_became_peered":"2024-09-20T22:58:28.405530+0000","last_unstale":"2024-09-20T22:59:13.390741+0000","last_undegraded":"2024-09-20T22:59:13.390741+0000","last_fullsized":"2024-09-20T22:59:13.390741+0000","mapping_epoch":1008,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":1009,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:58:52.745690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:50:21.853070+0000","last_clean_scrub_stamp":"2024-09-20T22:58:52.745690+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T23:00:11.125777+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"625'19","reported_seq":1001,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:14.154930+0000","last_change":"2024-09-20T22:59:14.154999+0000","last_active":"2024-09-20T22:59:14.154930+0000","last_peered":"2024-09-20T22:59:14.154930+0000","last_clean":"2024-09-20T22:59:14.154930+0000","last_became_active":"2024-09-20T22:58:48.609950+0000","last_became_peered":"2024-09-20T22:58:48.609950+0000","last_unstale":"2024-09-20T22:59:14.154930+0000","last_undegraded":"2024-09-20T22:59:14.154930+0000","last_fullsized":"2024-09-20T22:59:14.154930+0000","mapping_epoch":1022,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1023,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'19","last_scrub_stamp":"2024-09-20T22:59:14.154830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:59:14.154830+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T23:00:41.980251+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":941,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:13.390658+0000","last_change":"2024-09-20T22:59:00.174402+0000","last_active":"2024-09-20T22:59:13.390658+0000","last_peered":"2024-09-20T22:59:13.390658+0000","last_clean":"2024-09-20T22:59:13.390658+0000","last_became_active":"2024-09-20T22:58:49.164889+0000","last_became_peered":"2024-09-20T22:58:49.164889+0000","last_unstale":"2024-09-20T22:59:13.390658+0000","last_undegraded":"2024-09-20T22:59:13.390658+0000","last_fullsized":"2024-09-20T22:59:13.390658+0000","mapping_epoch":1022,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1023,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:59:00.174315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:52:47.717490+0000","last_clean_scrub_stamp":"2024-09-20T22:59:00.174315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T23:00:03.795895+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":562,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:14.015544+0000","last_change":"2024-09-20T22:59:13.421244+0000","last_active":"2024-09-20T22:59:14.015544+0000","last_peered":"2024-09-20T22:59:14.015544+0000","last_clean":"2024-09-20T22:59:14.015544+0000","last_became_active":"2024-09-20T22:57:49.398369+0000","last_became_peered":"2024-09-20T22:57:49.398369+0000","last_unstale":"2024-09-20T22:59:14.015544+0000","last_undegraded":"2024-09-20T22:59:14.015544+0000","last_fullsized":"2024-09-20T22:59:14.015544+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:59:13.421167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:59:13.421167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T23:00:40.031830+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":1837,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:17.221717+0000","last_change":"2024-09-20T22:59:17.221735+0000","last_active":"2024-09-20T22:59:17.221717+0000","last_peered":"2024-09-20T22:59:17.221717+0000","last_clean":"2024-09-20T22:59:17.221717+0000","last_became_active":"2024-09-20T22:57:49.413148+0000","last_became_peered":"2024-09-20T22:57:49.413148+0000","last_unstale":"2024-09-20T22:59:17.221717+0000","last_undegraded":"2024-09-20T22:59:17.221717+0000","last_fullsized":"2024-09-20T22:59:17.221717+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:59:17.221665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:56:22.490523+0000","last_clean_scrub_stamp":"2024-09-20T22:59:17.221665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T23:00:18.885974+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"625'19","reported_seq":555,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:14.015738+0000","last_change":"2024-09-20T22:58:03.394296+0000","last_active":"2024-09-20T22:59:14.015738+0000","last_peered":"2024-09-20T22:59:14.015738+0000","last_clean":"2024-09-20T22:59:14.015738+0000","last_became_active":"2024-09-20T22:54:17.340539+0000","last_became_peered":"2024-09-20T22:54:17.340539+0000","last_unstale":"2024-09-20T22:59:14.015738+0000","last_undegraded":"2024-09-20T22:59:14.015738+0000","last_fullsized":"2024-09-20T22:59:14.015738+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'19","last_scrub_stamp":"2024-09-20T22:58:03.394221+0000","last_deep_scrub":"404'13","last_deep_scrub_stamp":"2024-09-20T22:48:04.276282+0000","last_clean_scrub_stamp":"2024-09-20T22:58:03.394221+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:25.311781+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":73,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:14.160071+0000","last_change":"2024-09-20T22:58:29.123277+0000","last_active":"2024-09-20T22:59:14.160071+0000","last_peered":"2024-09-20T22:59:14.160071+0000","last_clean":"2024-09-20T22:59:14.160071+0000","last_became_active":"2024-09-20T22:58:28.412950+0000","last_became_peered":"2024-09-20T22:58:28.412950+0000","last_unstale":"2024-09-20T22:59:14.160071+0000","last_undegraded":"2024-09-20T22:59:14.160071+0000","last_fullsized":"2024-09-20T22:59:14.160071+0000","mapping_epoch":1008,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1009,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:58:29.123179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:31.138067+0000","last_clean_scrub_stamp":"2024-09-20T22:58:29.123179+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:53.882930+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":1843,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:13.486223+0000","last_change":"2024-09-20T22:58:03.746286+0000","last_active":"2024-09-20T22:59:13.486223+0000","last_peered":"2024-09-20T22:59:13.486223+0000","last_clean":"2024-09-20T22:59:13.486223+0000","last_became_active":"2024-09-20T22:57:49.432838+0000","last_became_peered":"2024-09-20T22:57:49.432838+0000","last_unstale":"2024-09-20T22:59:13.486223+0000","last_undegraded":"2024-09-20T22:59:13.486223+0000","last_fullsized":"2024-09-20T22:59:13.486223+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:58:03.746169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:58:03.746169+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:27.258169+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":1362,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:13.389527+0000","last_change":"2024-09-20T22:59:08.735055+0000","last_active":"2024-09-20T22:59:13.389527+0000","last_peered":"2024-09-20T22:59:13.389527+0000","last_clean":"2024-09-20T22:59:13.389527+0000","last_became_active":"2024-09-20T22:54:17.344021+0000","last_became_peered":"2024-09-20T22:54:17.344021+0000","last_unstale":"2024-09-20T22:59:13.389527+0000","last_undegraded":"2024-09-20T22:59:13.389527+0000","last_fullsized":"2024-09-20T22:59:13.389527+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:59:08.734987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:55:09.664839+0000","last_clean_scrub_stamp":"2024-09-20T22:59:08.734987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T23:00:15.341798+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"625'24","reported_seq":1553,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:20.928284+0000","last_change":"2024-09-20T22:59:20.928329+0000","last_active":"2024-09-20T22:59:20.928284+0000","last_peered":"2024-09-20T22:59:20.928284+0000","last_clean":"2024-09-20T22:59:20.928284+0000","last_became_active":"2024-09-20T22:58:55.393537+0000","last_became_peered":"2024-09-20T22:58:55.393537+0000","last_unstale":"2024-09-20T22:59:20.928284+0000","last_undegraded":"2024-09-20T22:59:20.928284+0000","last_fullsized":"2024-09-20T22:59:20.928284+0000","mapping_epoch":1025,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":1026,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'24","last_scrub_stamp":"2024-09-20T22:59:20.928206+0000","last_deep_scrub":"470'16","last_deep_scrub_stamp":"2024-09-20T22:49:10.659868+0000","last_clean_scrub_stamp":"2024-09-20T22:59:20.928206+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T23:00:22.381114+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":2040,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:14.015443+0000","last_change":"2024-09-20T22:58:28.406826+0000","last_active":"2024-09-20T22:59:14.015443+0000","last_peered":"2024-09-20T22:59:14.015443+0000","last_clean":"2024-09-20T22:59:14.015443+0000","last_became_active":"2024-09-20T22:58:28.405836+0000","last_became_peered":"2024-09-20T22:58:28.405836+0000","last_unstale":"2024-09-20T22:59:14.015443+0000","last_undegraded":"2024-09-20T22:59:14.015443+0000","last_fullsized":"2024-09-20T22:59:14.015443+0000","mapping_epoch":1008,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":1009,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:58:25.905810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:45:24.765414+0000","last_clean_scrub_stamp":"2024-09-20T22:58:25.905810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:54.228873+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":73,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:25.572688+0000","last_change":"2024-09-20T22:59:25.572714+0000","last_active":"2024-09-20T22:59:25.572688+0000","last_peered":"2024-09-20T22:59:25.572688+0000","last_clean":"2024-09-20T22:59:25.572688+0000","last_became_active":"2024-09-20T22:58:28.408967+0000","last_became_peered":"2024-09-20T22:58:28.408967+0000","last_unstale":"2024-09-20T22:59:25.572688+0000","last_undegraded":"2024-09-20T22:59:25.572688+0000","last_fullsized":"2024-09-20T22:59:25.572688+0000","mapping_epoch":1008,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":1009,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:59:25.572617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:43:50.174861+0000","last_clean_scrub_stamp":"2024-09-20T22:59:25.572617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T23:00:46.141169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":1835,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:13.486314+0000","last_change":"2024-09-20T22:58:42.917038+0000","last_active":"2024-09-20T22:59:13.486314+0000","last_peered":"2024-09-20T22:59:13.486314+0000","last_clean":"2024-09-20T22:59:13.486314+0000","last_became_active":"2024-09-20T22:57:49.410970+0000","last_became_peered":"2024-09-20T22:57:49.410970+0000","last_unstale":"2024-09-20T22:59:13.486314+0000","last_undegraded":"2024-09-20T22:59:13.486314+0000","last_fullsized":"2024-09-20T22:59:13.486314+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:58:42.916963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:43.176430+0000","last_clean_scrub_stamp":"2024-09-20T22:58:42.916963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T23:00:12.008846+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"625'8","reported_seq":1907,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:26.303005+0000","last_change":"2024-09-20T22:59:26.303034+0000","last_active":"2024-09-20T22:59:26.303005+0000","last_peered":"2024-09-20T22:59:26.303005+0000","last_clean":"2024-09-20T22:59:26.303005+0000","last_became_active":"2024-09-20T22:57:49.401068+0000","last_became_peered":"2024-09-20T22:57:49.401068+0000","last_unstale":"2024-09-20T22:59:26.303005+0000","last_undegraded":"2024-09-20T22:59:26.303005+0000","last_fullsized":"2024-09-20T22:59:26.303005+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"625'8","last_scrub_stamp":"2024-09-20T22:59:26.302943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:39.147434+0000","last_clean_scrub_stamp":"2024-09-20T22:59:26.302943+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T23:00:32.370010+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":1366,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:17.850321+0000","last_change":"2024-09-20T22:59:17.850356+0000","last_active":"2024-09-20T22:59:17.850321+0000","last_peered":"2024-09-20T22:59:17.850321+0000","last_clean":"2024-09-20T22:59:17.850321+0000","last_became_active":"2024-09-20T22:57:49.412134+0000","last_became_peered":"2024-09-20T22:57:49.412134+0000","last_unstale":"2024-09-20T22:59:17.850321+0000","last_undegraded":"2024-09-20T22:59:17.850321+0000","last_fullsized":"2024-09-20T22:59:17.850321+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:59:17.850257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:58:02.611308+0000","last_clean_scrub_stamp":"2024-09-20T22:59:17.850257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T23:00:40.535978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":2057,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:13.746345+0000","last_change":"2024-09-20T22:58:33.443155+0000","last_active":"2024-09-20T22:59:13.746345+0000","last_peered":"2024-09-20T22:59:13.746345+0000","last_clean":"2024-09-20T22:59:13.746345+0000","last_became_active":"2024-09-20T22:58:28.398589+0000","last_became_peered":"2024-09-20T22:58:28.398589+0000","last_unstale":"2024-09-20T22:59:13.746345+0000","last_undegraded":"2024-09-20T22:59:13.746345+0000","last_fullsized":"2024-09-20T22:59:13.746345+0000","mapping_epoch":1008,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":1009,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:58:33.443029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:41.163059+0000","last_clean_scrub_stamp":"2024-09-20T22:58:33.443029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T23:00:01.673636+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":73,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:14.159343+0000","last_change":"2024-09-20T22:58:57.216383+0000","last_active":"2024-09-20T22:59:14.159343+0000","last_peered":"2024-09-20T22:59:14.159343+0000","last_clean":"2024-09-20T22:59:14.159343+0000","last_became_active":"2024-09-20T22:58:28.412797+0000","last_became_peered":"2024-09-20T22:58:28.412797+0000","last_unstale":"2024-09-20T22:59:14.159343+0000","last_undegraded":"2024-09-20T22:59:14.159343+0000","last_fullsized":"2024-09-20T22:59:14.159343+0000","mapping_epoch":1008,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":1009,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:58:57.216303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:55:08.681729+0000","last_clean_scrub_stamp":"2024-09-20T22:58:57.216303+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T23:00:13.723641+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":553,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:14.016356+0000","last_change":"2024-09-20T22:58:58.331621+0000","last_active":"2024-09-20T22:59:14.016356+0000","last_peered":"2024-09-20T22:59:14.016356+0000","last_clean":"2024-09-20T22:59:14.016356+0000","last_became_active":"2024-09-20T22:54:17.348206+0000","last_became_peered":"2024-09-20T22:54:17.348206+0000","last_unstale":"2024-09-20T22:59:14.016356+0000","last_undegraded":"2024-09-20T22:59:14.016356+0000","last_fullsized":"2024-09-20T22:59:14.016356+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:58:58.331549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:58:58.331549+0000","last_clean_scrub_stamp":"2024-09-20T22:58:58.331549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:58.949836+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":1865,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:13.486146+0000","last_change":"2024-09-20T22:58:28.880848+0000","last_active":"2024-09-20T22:59:13.486146+0000","last_peered":"2024-09-20T22:59:13.486146+0000","last_clean":"2024-09-20T22:59:13.486146+0000","last_became_active":"2024-09-20T22:57:49.407194+0000","last_became_peered":"2024-09-20T22:57:49.407194+0000","last_unstale":"2024-09-20T22:59:13.486146+0000","last_undegraded":"2024-09-20T22:59:13.486146+0000","last_fullsized":"2024-09-20T22:59:13.486146+0000","mapping_epoch":988,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:58:28.880776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:41:37.135621+0000","last_clean_scrub_stamp":"2024-09-20T22:58:28.880776+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T22:59:31.422169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"1036'24","reported_seq":2109,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:17.594603+0000","last_change":"2024-09-20T22:59:17.594640+0000","last_active":"2024-09-20T22:59:17.594603+0000","last_peered":"2024-09-20T22:59:17.594603+0000","last_clean":"2024-09-20T22:59:17.594603+0000","last_became_active":"2024-09-20T22:54:17.342236+0000","last_became_peered":"2024-09-20T22:54:17.342236+0000","last_unstale":"2024-09-20T22:59:17.594603+0000","last_undegraded":"2024-09-20T22:59:17.594603+0000","last_fullsized":"2024-09-20T22:59:17.594603+0000","mapping_epoch":776,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":777,"parent":"0.0","parent_split_bits":0,"last_scrub":"1036'24","last_scrub_stamp":"2024-09-20T22:59:17.594541+0000","last_deep_scrub":"1036'24","last_deep_scrub_stamp":"2024-09-20T22:59:17.594541+0000","last_clean_scrub_stamp":"2024-09-20T22:59:17.594541+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T23:00:36.558346+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":1362,"reported_epoch":1036,"state":"active+clean","last_fresh":"2024-09-20T22:59:13.390777+0000","last_change":"2024-09-20T22:58:57.634893+0000","last_active":"2024-09-20T22:59:13.390777+0000","last_peered":"2024-09-20T22:59:13.390777+0000","last_clean":"2024-09-20T22:59:13.390777+0000","last_became_active":"2024-09-20T22:58:28.405330+0000","last_became_peered":"2024-09-20T22:58:28.405330+0000","last_unstale":"2024-09-20T22:59:13.390777+0000","last_undegraded":"2024-09-20T22:59:13.390777+0000","last_fullsized":"2024-09-20T22:59:13.390777+0000","mapping_epoch":1008,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":1009,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T22:58:57.634790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T22:54:43.813540+0000","last_clean_scrub_stamp":"2024-09-20T22:58:57.634790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T23:00:01.656707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":429,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":105,"ondisk_log_size":105,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1101,"num_read_kb":858,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":514,"num_bytes_recovered":11678,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":785,"ondisk_log_size":785,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":1126,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":422,"num_read_kb":417,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":655360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":385,"ondisk_log_size":385,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070683,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":91664,"kb_used_data":10560,"kb_used_omap":1,"kb_used_meta":81086,"kb_avail":93653488,"statfs":{"total":95995035648,"available":95901171712,"internally_reserved":0,"allocated":10813440,"data_stored":6734015,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":83032522},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":48,"seq":206158430433,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":99660,"kb_used_data":10236,"kb_used_omap":1,"kb_used_meta":89406,"kb_avail":93645492,"statfs":{"total":95995035648,"available":95892983808,"internally_reserved":0,"allocated":10481664,"data_stored":6406573,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":91552198},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":40,"seq":171798692070,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":86440,"kb_used_data":10840,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93658712,"statfs":{"total":95995035648,"available":95906521088,"internally_reserved":0,"allocated":11100160,"data_stored":6998191,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986411,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":59492,"kb_used_data":10260,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":93685660,"statfs":{"total":95995035648,"available":95934115840,"internally_reserved":0,"allocated":10506240,"data_stored":6406185,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":988,"seq":4243427688470,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":45604,"kb_used_data":10836,"kb_used_omap":148,"kb_used_meta":34603,"kb_avail":93699548,"statfs":{"total":95995035648,"available":95948337152,"internally_reserved":0,"allocated":11096064,"data_stored":6997489,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":151956,"internal_metadata":35434092},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411574,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":95780,"kb_used_data":10512,"kb_used_omap":1,"kb_used_meta":85246,"kb_avail":93649372,"statfs":{"total":95995035648,"available":95896956928,"internally_reserved":0,"allocated":10764288,"data_stored":6669499,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":87292357},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575099,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":85872,"kb_used_data":10272,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":93659280,"statfs":{"total":95995035648,"available":95907102720,"internally_reserved":0,"allocated":10518528,"data_stored":6406884,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738624,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":99680,"kb_used_data":10256,"kb_used_omap":1,"kb_used_meta":89406,"kb_avail":93645472,"statfs":{"total":95995035648,"available":95892963328,"internally_reserved":0,"allocated":10502144,"data_stored":6406797,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":91552197},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-20T22:59:33.197 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-09-20T22:59:33.197 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 -- ceph config rm mgr mgr_debug_aggressive_pg_num_changes 2024-09-20T22:59:33.447 INFO:teuthology.orchestra.run.smithi084.stderr:Inferring config /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/mon.a/config 2024-09-20T22:59:33.568 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:33 smithi084 ceph-mon[40024]: 1.8 scrub starts 2024-09-20T22:59:33.568 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:33 smithi084 ceph-mon[40024]: 1.8 scrub ok 2024-09-20T22:59:33.569 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:33 smithi084 ceph-mon[40024]: pgmap v1484: 142 pgs: 142 active+clean; 583 KiB data, 649 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:59:33.569 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:33 smithi084 ceph-mon[40024]: from='client.17922 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T22:59:33.569 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:33 smithi084 ceph-mon[29774]: 1.8 scrub starts 2024-09-20T22:59:33.569 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:33 smithi084 ceph-mon[29774]: 1.8 scrub ok 2024-09-20T22:59:33.569 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:33 smithi084 ceph-mon[29774]: pgmap v1484: 142 pgs: 142 active+clean; 583 KiB data, 649 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:59:33.569 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:33 smithi084 ceph-mon[29774]: from='client.17922 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T22:59:33.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:33 smithi154 ceph-mon[33264]: 1.8 scrub starts 2024-09-20T22:59:33.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:33 smithi154 ceph-mon[33264]: 1.8 scrub ok 2024-09-20T22:59:33.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:33 smithi154 ceph-mon[33264]: pgmap v1484: 142 pgs: 142 active+clean; 583 KiB data, 649 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-20T22:59:33.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:33 smithi154 ceph-mon[33264]: from='client.17922 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T22:59:34.264 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:59:34 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: [20/Sep/2024:22:59:34] ENGINE Bus STOPPING 2024-09-20T22:59:34.264 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:59:34 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: [20/Sep/2024:22:59:34] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-20T22:59:34.264 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:59:34 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: [20/Sep/2024:22:59:34] ENGINE Bus STOPPED 2024-09-20T22:59:34.615 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:59:34 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: [20/Sep/2024:22:59:34] ENGINE Bus STARTING 2024-09-20T22:59:34.615 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:59:34 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: [20/Sep/2024:22:59:34] ENGINE Serving on http://:::9283 2024-09-20T22:59:34.615 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:59:34 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: [20/Sep/2024:22:59:34] ENGINE Bus STARTED 2024-09-20T22:59:34.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:34 smithi084 ceph-mon[40024]: 4.17 scrub starts 2024-09-20T22:59:34.615 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:34 smithi084 ceph-mon[40024]: 4.17 scrub ok 2024-09-20T22:59:34.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:34 smithi084 ceph-mon[40024]: 1.6 deep-scrub starts 2024-09-20T22:59:34.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:34 smithi084 ceph-mon[40024]: 1.6 deep-scrub ok 2024-09-20T22:59:34.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:34 smithi084 ceph-mon[40024]: from='client.? 172.21.15.84:0/1477524913' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-09-20T22:59:34.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:34 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-09-20T22:59:34.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:34 smithi084 ceph-mon[40024]: from='client.? ' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-09-20T22:59:34.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:34 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:59:34.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:34 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:59:34.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:34 smithi084 ceph-mon[40024]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:59:34.616 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:34 smithi084 ceph-mon[40024]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:59:34.616 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:34 smithi084 ceph-mon[29774]: 4.17 scrub starts 2024-09-20T22:59:34.617 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:34 smithi084 ceph-mon[29774]: 4.17 scrub ok 2024-09-20T22:59:34.617 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:34 smithi084 ceph-mon[29774]: 1.6 deep-scrub starts 2024-09-20T22:59:34.617 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:34 smithi084 ceph-mon[29774]: 1.6 deep-scrub ok 2024-09-20T22:59:34.617 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:34 smithi084 ceph-mon[29774]: from='client.? 172.21.15.84:0/1477524913' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-09-20T22:59:34.617 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:34 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-09-20T22:59:34.617 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:34 smithi084 ceph-mon[29774]: from='client.? ' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-09-20T22:59:34.617 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:34 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:59:34.617 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:34 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:59:34.617 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:34 smithi084 ceph-mon[29774]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:59:34.617 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:34 smithi084 ceph-mon[29774]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:59:34.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:34 smithi154 ceph-mon[33264]: 4.17 scrub starts 2024-09-20T22:59:34.754 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:34 smithi154 ceph-mon[33264]: 4.17 scrub ok 2024-09-20T22:59:34.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:34 smithi154 ceph-mon[33264]: 1.6 deep-scrub starts 2024-09-20T22:59:34.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:34 smithi154 ceph-mon[33264]: 1.6 deep-scrub ok 2024-09-20T22:59:34.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:34 smithi154 ceph-mon[33264]: from='client.? 172.21.15.84:0/1477524913' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-09-20T22:59:34.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:34 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-09-20T22:59:34.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:34 smithi154 ceph-mon[33264]: from='client.? ' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-09-20T22:59:34.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:34 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T22:59:34.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:34 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T22:59:34.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:34 smithi154 ceph-mon[33264]: from='mgr.24524 172.21.15.84:0/3231473716' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T22:59:34.755 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:34 smithi154 ceph-mon[33264]: from='mgr.24524 ' entity='mgr.y' 2024-09-20T22:59:34.864 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-09-20T22:59:34.873 INFO:tasks.cephadm:Teardown begin 2024-09-20T22:59:34.873 DEBUG:teuthology.orchestra.run.smithi084:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-20T22:59:34.899 DEBUG:teuthology.orchestra.run.smithi154:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-20T22:59:34.932 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-09-20T22:59:34.932 DEBUG:teuthology.orchestra.run.smithi084:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-09-20T22:59:34.954 DEBUG:teuthology.orchestra.run.smithi154:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-09-20T22:59:34.990 INFO:tasks.cephadm:Stopping all daemons... 2024-09-20T22:59:34.990 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-09-20T22:59:34.990 DEBUG:teuthology.orchestra.run.smithi084:> sudo systemctl stop ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490@mon.a 2024-09-20T22:59:35.115 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:35 smithi084 systemd[1]: Stopping Ceph mon.a for a9c0bc70-77a0-11ef-baf5-efdc52797490... 2024-09-20T22:59:35.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:35 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mon-a[29770]: 2024-09-20T22:59:35.279+0000 7f26b2df3640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-20T22:59:35.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:35 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mon-a[29770]: 2024-09-20T22:59:35.279+0000 7f26b2df3640 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-09-20T22:59:35.615 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:35 smithi084 podman[225580]: 2024-09-20 22:59:35.361559849 +0000 UTC m=+0.183987854 container died 318fc1d3b260ddf478f7c5eae8aeb5b15890ed113f21be937bb46cbddb726aed (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mon-a, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, RELEASE=squid-9e2ac70) 2024-09-20T22:59:36.016 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:35 smithi084 podman[225580]: 2024-09-20 22:59:35.771201286 +0000 UTC m=+0.593629276 container cleanup 318fc1d3b260ddf478f7c5eae8aeb5b15890ed113f21be937bb46cbddb726aed (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mon-a, ceph=True, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-9e2ac70, GIT_CLEAN=True, org.label-schema.build-date=20240731) 2024-09-20T22:59:36.016 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:35 smithi084 bash[225580]: ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mon-a 2024-09-20T22:59:36.250 DEBUG:teuthology.orchestra.run.smithi084:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490@mon.a.service' 2024-09-20T22:59:36.272 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:36 smithi084 podman[225594]: 2024-09-20 22:59:36.014443751 +0000 UTC m=+0.654368157 container remove 318fc1d3b260ddf478f7c5eae8aeb5b15890ed113f21be937bb46cbddb726aed (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mon-a, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=squid-9e2ac70, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True) 2024-09-20T22:59:36.272 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:36 smithi084 systemd[1]: ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490@mon.a.service: Deactivated successfully. 2024-09-20T22:59:36.272 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:36 smithi084 systemd[1]: Stopped Ceph mon.a for a9c0bc70-77a0-11ef-baf5-efdc52797490. 2024-09-20T22:59:36.272 INFO:journalctl@ceph.mon.a.smithi084.stdout:Sep 20 22:59:36 smithi084 systemd[1]: ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490@mon.a.service: Consumed 30.876s CPU time. 2024-09-20T22:59:36.288 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-20T22:59:36.288 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-09-20T22:59:36.288 INFO:tasks.cephadm.mon.b:Stopping mon.c... 2024-09-20T22:59:36.288 DEBUG:teuthology.orchestra.run.smithi084:> sudo systemctl stop ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490@mon.c 2024-09-20T22:59:36.594 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:36 smithi084 systemd[1]: Stopping Ceph mon.c for a9c0bc70-77a0-11ef-baf5-efdc52797490... 2024-09-20T22:59:36.859 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:59:36 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: [20/Sep/2024:22:59:36] ENGINE Bus STOPPING 2024-09-20T22:59:36.859 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:36 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mon-c[40020]: 2024-09-20T22:59:36.591+0000 7f23ff5a1640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-20T22:59:36.859 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:36 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mon-c[40020]: 2024-09-20T22:59:36.591+0000 7f23ff5a1640 -1 mon.c@2(peon) e3 *** Got Signal Terminated *** 2024-09-20T22:59:37.115 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:36 smithi084 podman[225702]: 2024-09-20 22:59:36.858159196 +0000 UTC m=+0.352196040 container died 3b6b6457d59519c55f4d6253c507f9feca4a312c74fe6f8d808bd2a8d0325339 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mon-c, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, RELEASE=squid-9e2ac70, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.37.2) 2024-09-20T22:59:37.115 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:59:36 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: [20/Sep/2024:22:59:36] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-20T22:59:37.115 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:59:36 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: [20/Sep/2024:22:59:36] ENGINE Bus STOPPED 2024-09-20T22:59:37.115 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:59:36 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: [20/Sep/2024:22:59:36] ENGINE Bus STARTING 2024-09-20T22:59:37.115 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:59:36 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: [20/Sep/2024:22:59:36] ENGINE Serving on http://:::9283 2024-09-20T22:59:37.115 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:59:36 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: [20/Sep/2024:22:59:36] ENGINE Bus STARTED 2024-09-20T22:59:37.579 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:37 smithi084 podman[225702]: 2024-09-20 22:59:37.400958093 +0000 UTC m=+0.894994927 container cleanup 3b6b6457d59519c55f4d6253c507f9feca4a312c74fe6f8d808bd2a8d0325339 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mon-c, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-9e2ac70, ceph=True) 2024-09-20T22:59:37.579 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:37 smithi084 bash[225702]: ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mon-c 2024-09-20T22:59:37.824 DEBUG:teuthology.orchestra.run.smithi084:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490@mon.c.service' 2024-09-20T22:59:37.846 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:37 smithi084 podman[225714]: 2024-09-20 22:59:37.577423082 +0000 UTC m=+0.715425326 container remove 3b6b6457d59519c55f4d6253c507f9feca4a312c74fe6f8d808bd2a8d0325339 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mon-c, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=squid-9e2ac70, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, io.buildah.version=1.37.2) 2024-09-20T22:59:37.846 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:37 smithi084 systemd[1]: ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490@mon.c.service: Deactivated successfully. 2024-09-20T22:59:37.846 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:37 smithi084 systemd[1]: Stopped Ceph mon.c for a9c0bc70-77a0-11ef-baf5-efdc52797490. 2024-09-20T22:59:37.846 INFO:journalctl@ceph.mon.c.smithi084.stdout:Sep 20 22:59:37 smithi084 systemd[1]: ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490@mon.c.service: Consumed 20.583s CPU time. 2024-09-20T22:59:37.858 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-20T22:59:37.858 INFO:tasks.cephadm.mon.b:Stopped mon.c 2024-09-20T22:59:37.858 INFO:tasks.cephadm.mon.b:Stopping mon.b... 2024-09-20T22:59:37.858 DEBUG:teuthology.orchestra.run.smithi154:> sudo systemctl stop ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490@mon.b 2024-09-20T22:59:38.167 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:37 smithi154 systemd[1]: Stopping Ceph mon.b for a9c0bc70-77a0-11ef-baf5-efdc52797490... 2024-09-20T22:59:38.421 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:38 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mon-b[33260]: 2024-09-20T22:59:38.165+0000 7f9d791cd640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-20T22:59:38.421 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:38 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mon-b[33260]: 2024-09-20T22:59:38.165+0000 7f9d791cd640 -1 mon.b@1(peon) e3 *** Got Signal Terminated *** 2024-09-20T22:59:38.615 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:59:38 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: ::ffff:172.21.15.154 - - [20/Sep/2024:22:59:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-20T22:59:38.696 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:38 smithi154 podman[152920]: 2024-09-20 22:59:38.420175827 +0000 UTC m=+0.366858165 container died b435b5fa6e615a27631771e5bb5513b86005c268f8c9c21ff0f58238ad65a7f4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mon-b, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-9e2ac70, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-09-20T22:59:39.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:38 smithi154 podman[152920]: 2024-09-20 22:59:38.852933021 +0000 UTC m=+0.799615345 container cleanup b435b5fa6e615a27631771e5bb5513b86005c268f8c9c21ff0f58238ad65a7f4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mon-b, io.buildah.version=1.37.2, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, RELEASE=squid-9e2ac70, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0) 2024-09-20T22:59:39.004 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:38 smithi154 bash[152920]: ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mon-b 2024-09-20T22:59:39.294 DEBUG:teuthology.orchestra.run.smithi154:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490@mon.b.service' 2024-09-20T22:59:39.315 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:39 smithi154 podman[152932]: 2024-09-20 22:59:39.063260443 +0000 UTC m=+0.642635124 container remove b435b5fa6e615a27631771e5bb5513b86005c268f8c9c21ff0f58238ad65a7f4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mon-b, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, RELEASE=squid-9e2ac70, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-20T22:59:39.316 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:39 smithi154 systemd[1]: ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490@mon.b.service: Deactivated successfully. 2024-09-20T22:59:39.316 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:39 smithi154 systemd[1]: Stopped Ceph mon.b for a9c0bc70-77a0-11ef-baf5-efdc52797490. 2024-09-20T22:59:39.316 INFO:journalctl@ceph.mon.b.smithi154.stdout:Sep 20 22:59:39 smithi154 systemd[1]: ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490@mon.b.service: Consumed 19.860s CPU time. 2024-09-20T22:59:39.336 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-20T22:59:39.336 INFO:tasks.cephadm.mon.b:Stopped mon.b 2024-09-20T22:59:39.336 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-09-20T22:59:39.336 DEBUG:teuthology.orchestra.run.smithi084:> sudo systemctl stop ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490@mgr.y 2024-09-20T22:59:39.614 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:59:39 smithi084 systemd[1]: Stopping Ceph mgr.y for a9c0bc70-77a0-11ef-baf5-efdc52797490... 2024-09-20T22:59:39.899 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:59:39 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: 2024-09-20T22:59:39.613+0000 7fd616f78640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.y -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-20T22:59:39.900 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:59:39 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y[30000]: 2024-09-20T22:59:39.613+0000 7fd616f78640 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-09-20T22:59:39.900 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:59:39 smithi084 podman[225833]: 2024-09-20 22:59:39.645237068 +0000 UTC m=+0.121578996 container died 5780dd378cf0d7032d08f9f77dd852f2e85a2c7fdf3f902478f46e1e457e829d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=squid-9e2ac70, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True) 2024-09-20T22:59:40.365 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:59:40 smithi084 podman[225833]: 2024-09-20 22:59:40.072013088 +0000 UTC m=+0.548355004 container cleanup 5780dd378cf0d7032d08f9f77dd852f2e85a2c7fdf3f902478f46e1e457e829d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, GIT_CLEAN=True, RELEASE=squid-9e2ac70, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True) 2024-09-20T22:59:40.365 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:59:40 smithi084 bash[225833]: ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y 2024-09-20T22:59:40.630 INFO:journalctl@ceph.mgr.y.smithi084.stdout:Sep 20 22:59:40 smithi084 podman[225846]: 2024-09-20 22:59:40.377582705 +0000 UTC m=+0.729347998 container remove 5780dd378cf0d7032d08f9f77dd852f2e85a2c7fdf3f902478f46e1e457e829d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-y, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-9e2ac70, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2) 2024-09-20T22:59:40.648 DEBUG:teuthology.orchestra.run.smithi084:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490@mgr.y.service' 2024-09-20T22:59:40.682 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-20T22:59:40.682 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-09-20T22:59:40.682 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-09-20T22:59:40.682 DEBUG:teuthology.orchestra.run.smithi154:> sudo systemctl stop ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490@mgr.x 2024-09-20T22:59:41.004 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Sep 20 22:59:40 smithi154 systemd[1]: Stopping Ceph mgr.x for a9c0bc70-77a0-11ef-baf5-efdc52797490... 2024-09-20T22:59:41.352 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Sep 20 22:59:41 smithi154 podman[153043]: 2024-09-20 22:59:41.022296478 +0000 UTC m=+0.141748046 container died d0c4a10d6a7d4a02405b22afc9bea57dc7977dc16bf4e7a62d18f41beb29469e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-x, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, RELEASE=squid-9e2ac70, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-09-20T22:59:41.677 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Sep 20 22:59:41 smithi154 podman[153043]: 2024-09-20 22:59:41.432871055 +0000 UTC m=+0.552322605 container cleanup d0c4a10d6a7d4a02405b22afc9bea57dc7977dc16bf4e7a62d18f41beb29469e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-x, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, RELEASE=squid-9e2ac70, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-09-20T22:59:41.677 INFO:journalctl@ceph.mgr.x.smithi154.stdout:Sep 20 22:59:41 smithi154 bash[153043]: ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-mgr-x 2024-09-20T22:59:41.850 DEBUG:teuthology.orchestra.run.smithi154:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490@mgr.x.service' 2024-09-20T22:59:41.885 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-20T22:59:41.885 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-09-20T22:59:41.885 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-09-20T22:59:41.885 DEBUG:teuthology.orchestra.run.smithi084:> sudo systemctl stop ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490@osd.0 2024-09-20T22:59:42.179 INFO:journalctl@ceph.osd.0.smithi084.stdout:Sep 20 22:59:41 smithi084 systemd[1]: Stopping Ceph osd.0 for a9c0bc70-77a0-11ef-baf5-efdc52797490... 2024-09-20T22:59:42.615 INFO:journalctl@ceph.osd.0.smithi084.stdout:Sep 20 22:59:42 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-0[43832]: 2024-09-20T22:59:42.176+0000 7f13e24a6640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-20T22:59:42.615 INFO:journalctl@ceph.osd.0.smithi084.stdout:Sep 20 22:59:42 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-0[43832]: 2024-09-20T22:59:42.176+0000 7f13e24a6640 -1 osd.0 1036 *** Got signal Terminated *** 2024-09-20T22:59:42.615 INFO:journalctl@ceph.osd.0.smithi084.stdout:Sep 20 22:59:42 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-0[43832]: 2024-09-20T22:59:42.176+0000 7f13e24a6640 -1 osd.0 1036 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-20T22:59:47.878 INFO:journalctl@ceph.osd.0.smithi084.stdout:Sep 20 22:59:47 smithi084 podman[225965]: 2024-09-20 22:59:47.613844771 +0000 UTC m=+5.542800244 container died 76518acfdea65e61656de2bdd9ab4b0d1bdc94502158c8a56d3d99831c9c03dc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, RELEASE=squid-9e2ac70, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-09-20T22:59:48.337 INFO:journalctl@ceph.osd.0.smithi084.stdout:Sep 20 22:59:48 smithi084 podman[225965]: 2024-09-20 22:59:48.051333876 +0000 UTC m=+5.980289403 container cleanup 76518acfdea65e61656de2bdd9ab4b0d1bdc94502158c8a56d3d99831c9c03dc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-9e2ac70, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, io.buildah.version=1.37.2) 2024-09-20T22:59:48.338 INFO:journalctl@ceph.osd.0.smithi084.stdout:Sep 20 22:59:48 smithi084 bash[225965]: ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-0 2024-09-20T22:59:48.614 INFO:journalctl@ceph.osd.0.smithi084.stdout:Sep 20 22:59:48 smithi084 podman[225981]: 2024-09-20 22:59:48.336327621 +0000 UTC m=+0.715673924 container remove 76518acfdea65e61656de2bdd9ab4b0d1bdc94502158c8a56d3d99831c9c03dc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-0, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=squid-9e2ac70, io.buildah.version=1.37.2, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-20T22:59:49.115 INFO:journalctl@ceph.osd.0.smithi084.stdout:Sep 20 22:59:48 smithi084 podman[226069]: 2024-09-20 22:59:48.721936531 +0000 UTC m=+0.015615100 image pull cc9251b744a3faafca7c3095f79bfff81873afe823f31a03a11e015a34abcdbd quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5 2024-09-20T22:59:49.115 INFO:journalctl@ceph.osd.0.smithi084.stdout:Sep 20 22:59:48 smithi084 podman[226069]: 2024-09-20 22:59:48.820607512 +0000 UTC m=+0.114286096 container create 5163f12d14994b85baa5af5e1a65b103af07e1815ac49437eab2416069f534ae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-0-deactivate, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, RELEASE=squid-9e2ac70, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-20T22:59:49.115 INFO:journalctl@ceph.osd.0.smithi084.stdout:Sep 20 22:59:49 smithi084 podman[226069]: 2024-09-20 22:59:49.048096929 +0000 UTC m=+0.341775491 container init 5163f12d14994b85baa5af5e1a65b103af07e1815ac49437eab2416069f534ae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-0-deactivate, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_CLEAN=True, ceph=True, RELEASE=squid-9e2ac70, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-09-20T22:59:49.115 INFO:journalctl@ceph.osd.0.smithi084.stdout:Sep 20 22:59:49 smithi084 podman[226069]: 2024-09-20 22:59:49.05136137 +0000 UTC m=+0.345039918 container start 5163f12d14994b85baa5af5e1a65b103af07e1815ac49437eab2416069f534ae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-0-deactivate, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=squid-9e2ac70, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-20T22:59:49.615 INFO:journalctl@ceph.osd.0.smithi084.stdout:Sep 20 22:59:49 smithi084 podman[226069]: 2024-09-20 22:59:49.13269243 +0000 UTC m=+0.426370984 container attach 5163f12d14994b85baa5af5e1a65b103af07e1815ac49437eab2416069f534ae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-0-deactivate, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, RELEASE=squid-9e2ac70, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0) 2024-09-20T22:59:49.615 INFO:journalctl@ceph.osd.0.smithi084.stdout:Sep 20 22:59:49 smithi084 podman[226069]: 2024-09-20 22:59:49.238084234 +0000 UTC m=+0.531762788 container died 5163f12d14994b85baa5af5e1a65b103af07e1815ac49437eab2416069f534ae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-0-deactivate, org.label-schema.schema-version=1.0, RELEASE=squid-9e2ac70, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=) 2024-09-20T22:59:49.805 DEBUG:teuthology.orchestra.run.smithi084:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490@osd.0.service' 2024-09-20T22:59:49.845 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-20T22:59:49.845 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-09-20T22:59:49.845 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-09-20T22:59:49.845 DEBUG:teuthology.orchestra.run.smithi084:> sudo systemctl stop ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490@osd.1 2024-09-20T22:59:50.115 INFO:journalctl@ceph.osd.1.smithi084.stdout:Sep 20 22:59:49 smithi084 systemd[1]: Stopping Ceph osd.1 for a9c0bc70-77a0-11ef-baf5-efdc52797490... 2024-09-20T22:59:50.615 INFO:journalctl@ceph.osd.1.smithi084.stdout:Sep 20 22:59:50 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-1[49440]: 2024-09-20T22:59:50.179+0000 7f194b234640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-20T22:59:50.615 INFO:journalctl@ceph.osd.1.smithi084.stdout:Sep 20 22:59:50 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-1[49440]: 2024-09-20T22:59:50.179+0000 7f194b234640 -1 osd.1 1036 *** Got signal Terminated *** 2024-09-20T22:59:50.615 INFO:journalctl@ceph.osd.1.smithi084.stdout:Sep 20 22:59:50 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-1[49440]: 2024-09-20T22:59:50.179+0000 7f194b234640 -1 osd.1 1036 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-20T22:59:55.940 INFO:journalctl@ceph.osd.1.smithi084.stdout:Sep 20 22:59:55 smithi084 podman[226171]: 2024-09-20 22:59:55.638448188 +0000 UTC m=+5.572478437 container died 4247e4d6b957e70ad3bef7dec6dcf62bf30f996dc8a6c46182a5be5fdd8a3c4c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-1, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-9e2ac70, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-09-20T22:59:56.299 INFO:journalctl@ceph.osd.1.smithi084.stdout:Sep 20 22:59:56 smithi084 podman[226171]: 2024-09-20 22:59:56.11282828 +0000 UTC m=+6.046858526 container cleanup 4247e4d6b957e70ad3bef7dec6dcf62bf30f996dc8a6c46182a5be5fdd8a3c4c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-1, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.37.2, RELEASE=squid-9e2ac70, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-20T22:59:56.299 INFO:journalctl@ceph.osd.1.smithi084.stdout:Sep 20 22:59:56 smithi084 bash[226171]: ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-1 2024-09-20T22:59:56.614 INFO:journalctl@ceph.osd.1.smithi084.stdout:Sep 20 22:59:56 smithi084 podman[226188]: 2024-09-20 22:59:56.297691032 +0000 UTC m=+0.651670612 container remove 4247e4d6b957e70ad3bef7dec6dcf62bf30f996dc8a6c46182a5be5fdd8a3c4c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-1, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=squid-9e2ac70, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731) 2024-09-20T22:59:57.027 INFO:journalctl@ceph.osd.1.smithi084.stdout:Sep 20 22:59:56 smithi084 podman[226276]: 2024-09-20 22:59:56.629942054 +0000 UTC m=+0.015841370 image pull cc9251b744a3faafca7c3095f79bfff81873afe823f31a03a11e015a34abcdbd quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5 2024-09-20T22:59:57.027 INFO:journalctl@ceph.osd.1.smithi084.stdout:Sep 20 22:59:56 smithi084 podman[226276]: 2024-09-20 22:59:56.731638571 +0000 UTC m=+0.117537913 container create 5c090f126f2ef56d4b7ab9d4aa763529602d7128b4072c9d6523f8ac74cd8a31 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-1-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=squid-9e2ac70, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-20T22:59:57.027 INFO:journalctl@ceph.osd.1.smithi084.stdout:Sep 20 22:59:56 smithi084 podman[226276]: 2024-09-20 22:59:56.958714897 +0000 UTC m=+0.344614208 container init 5c090f126f2ef56d4b7ab9d4aa763529602d7128b4072c9d6523f8ac74cd8a31 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-1-deactivate, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-9e2ac70, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-09-20T22:59:57.027 INFO:journalctl@ceph.osd.1.smithi084.stdout:Sep 20 22:59:56 smithi084 podman[226276]: 2024-09-20 22:59:56.961919187 +0000 UTC m=+0.347818531 container start 5c090f126f2ef56d4b7ab9d4aa763529602d7128b4072c9d6523f8ac74cd8a31 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-1-deactivate, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, RELEASE=squid-9e2ac70, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731) 2024-09-20T22:59:57.365 INFO:journalctl@ceph.osd.1.smithi084.stdout:Sep 20 22:59:57 smithi084 podman[226276]: 2024-09-20 22:59:57.025272792 +0000 UTC m=+0.411172102 container attach 5c090f126f2ef56d4b7ab9d4aa763529602d7128b4072c9d6523f8ac74cd8a31 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-1-deactivate, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=squid-9e2ac70, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731) 2024-09-20T22:59:57.365 INFO:journalctl@ceph.osd.1.smithi084.stdout:Sep 20 22:59:57 smithi084 podman[226276]: 2024-09-20 22:59:57.156044833 +0000 UTC m=+0.541944146 container died 5c090f126f2ef56d4b7ab9d4aa763529602d7128b4072c9d6523f8ac74cd8a31 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-1-deactivate, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, RELEASE=squid-9e2ac70, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-20T22:59:57.741 DEBUG:teuthology.orchestra.run.smithi084:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490@osd.1.service' 2024-09-20T22:59:57.779 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-20T22:59:57.780 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-09-20T22:59:57.780 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-09-20T22:59:57.780 DEBUG:teuthology.orchestra.run.smithi084:> sudo systemctl stop ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490@osd.2 2024-09-20T22:59:58.100 INFO:journalctl@ceph.osd.2.smithi084.stdout:Sep 20 22:59:57 smithi084 systemd[1]: Stopping Ceph osd.2 for a9c0bc70-77a0-11ef-baf5-efdc52797490... 2024-09-20T22:59:58.364 INFO:journalctl@ceph.osd.2.smithi084.stdout:Sep 20 22:59:58 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-2[55375]: 2024-09-20T22:59:58.097+0000 7fd493d81640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-20T22:59:58.365 INFO:journalctl@ceph.osd.2.smithi084.stdout:Sep 20 22:59:58 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-2[55375]: 2024-09-20T22:59:58.097+0000 7fd493d81640 -1 osd.2 1036 *** Got signal Terminated *** 2024-09-20T22:59:58.365 INFO:journalctl@ceph.osd.2.smithi084.stdout:Sep 20 22:59:58 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-2[55375]: 2024-09-20T22:59:58.097+0000 7fd493d81640 -1 osd.2 1036 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-20T23:00:03.751 INFO:journalctl@ceph.osd.2.smithi084.stdout:Sep 20 23:00:03 smithi084 podman[226378]: 2024-09-20 23:00:03.479448626 +0000 UTC m=+5.482013849 container died d4d600b9a8ac58b79883dfb135e386bb50b073d8a250278e9f7827d716cfad6c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-2, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=squid-9e2ac70, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-09-20T23:00:04.092 INFO:journalctl@ceph.osd.2.smithi084.stdout:Sep 20 23:00:03 smithi084 podman[226378]: 2024-09-20 23:00:03.905841636 +0000 UTC m=+5.908406848 container cleanup d4d600b9a8ac58b79883dfb135e386bb50b073d8a250278e9f7827d716cfad6c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-2, CEPH_POINT_RELEASE=, ceph=True, RELEASE=squid-9e2ac70, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux ) 2024-09-20T23:00:04.092 INFO:journalctl@ceph.osd.2.smithi084.stdout:Sep 20 23:00:03 smithi084 bash[226378]: ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-2 2024-09-20T23:00:04.364 INFO:journalctl@ceph.osd.2.smithi084.stdout:Sep 20 23:00:04 smithi084 podman[226393]: 2024-09-20 23:00:04.090401315 +0000 UTC m=+0.613225291 container remove d4d600b9a8ac58b79883dfb135e386bb50b073d8a250278e9f7827d716cfad6c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-2, RELEASE=squid-9e2ac70, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True) 2024-09-20T23:00:04.812 INFO:journalctl@ceph.osd.2.smithi084.stdout:Sep 20 23:00:04 smithi084 podman[226483]: 2024-09-20 23:00:04.426636846 +0000 UTC m=+0.015896598 image pull cc9251b744a3faafca7c3095f79bfff81873afe823f31a03a11e015a34abcdbd quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5 2024-09-20T23:00:04.812 INFO:journalctl@ceph.osd.2.smithi084.stdout:Sep 20 23:00:04 smithi084 podman[226483]: 2024-09-20 23:00:04.52435634 +0000 UTC m=+0.113616081 container create 4cebe33dc7e3717ef7f27a051032071c51a6abe09f9aa84c3c35ab77c560ccaf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-2-deactivate, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=squid-9e2ac70, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-20T23:00:04.812 INFO:journalctl@ceph.osd.2.smithi084.stdout:Sep 20 23:00:04 smithi084 podman[226483]: 2024-09-20 23:00:04.735091712 +0000 UTC m=+0.324351466 container init 4cebe33dc7e3717ef7f27a051032071c51a6abe09f9aa84c3c35ab77c560ccaf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-2-deactivate, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, RELEASE=squid-9e2ac70, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-09-20T23:00:04.812 INFO:journalctl@ceph.osd.2.smithi084.stdout:Sep 20 23:00:04 smithi084 podman[226483]: 2024-09-20 23:00:04.738510828 +0000 UTC m=+0.327770572 container start 4cebe33dc7e3717ef7f27a051032071c51a6abe09f9aa84c3c35ab77c560ccaf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-2-deactivate, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-9e2ac70, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.vendor=CentOS) 2024-09-20T23:00:04.812 INFO:journalctl@ceph.osd.2.smithi084.stdout:Sep 20 23:00:04 smithi084 podman[226483]: 2024-09-20 23:00:04.810252952 +0000 UTC m=+0.399512712 container attach 4cebe33dc7e3717ef7f27a051032071c51a6abe09f9aa84c3c35ab77c560ccaf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-2-deactivate, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, RELEASE=squid-9e2ac70, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2) 2024-09-20T23:00:05.115 INFO:journalctl@ceph.osd.2.smithi084.stdout:Sep 20 23:00:04 smithi084 podman[226483]: 2024-09-20 23:00:04.935540581 +0000 UTC m=+0.524800329 container died 4cebe33dc7e3717ef7f27a051032071c51a6abe09f9aa84c3c35ab77c560ccaf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-2-deactivate, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, RELEASE=squid-9e2ac70, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-09-20T23:00:05.643 DEBUG:teuthology.orchestra.run.smithi084:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490@osd.2.service' 2024-09-20T23:00:05.687 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-20T23:00:05.687 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-09-20T23:00:05.687 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-09-20T23:00:05.688 DEBUG:teuthology.orchestra.run.smithi084:> sudo systemctl stop ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490@osd.3 2024-09-20T23:00:06.010 INFO:journalctl@ceph.osd.3.smithi084.stdout:Sep 20 23:00:05 smithi084 systemd[1]: Stopping Ceph osd.3 for a9c0bc70-77a0-11ef-baf5-efdc52797490... 2024-09-20T23:00:06.365 INFO:journalctl@ceph.osd.3.smithi084.stdout:Sep 20 23:00:06 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-3[213561]: 2024-09-20T23:00:06.007+0000 7f2fc93a8640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-20T23:00:06.365 INFO:journalctl@ceph.osd.3.smithi084.stdout:Sep 20 23:00:06 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-3[213561]: 2024-09-20T23:00:06.007+0000 7f2fc93a8640 -1 osd.3 1036 *** Got signal Terminated *** 2024-09-20T23:00:06.365 INFO:journalctl@ceph.osd.3.smithi084.stdout:Sep 20 23:00:06 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-3[213561]: 2024-09-20T23:00:06.007+0000 7f2fc93a8640 -1 osd.3 1036 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-20T23:00:10.254 INFO:journalctl@ceph.osd.6.smithi154.stdout:Sep 20 23:00:09 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-6[49255]: 2024-09-20T23:00:09.778+0000 7f2ff16a5640 -1 osd.6 1036 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-09-20T22:59:45.949062+0000 front 2024-09-20T22:59:45.949180+0000 (oldest deadline 2024-09-20T23:00:09.448861+0000) 2024-09-20T23:00:11.156 INFO:journalctl@ceph.osd.6.smithi154.stdout:Sep 20 23:00:10 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-6[49255]: 2024-09-20T23:00:10.827+0000 7f2ff16a5640 -1 osd.6 1036 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-09-20T22:59:45.949062+0000 front 2024-09-20T22:59:45.949180+0000 (oldest deadline 2024-09-20T23:00:09.448861+0000) 2024-09-20T23:00:11.504 INFO:journalctl@ceph.osd.4.smithi154.stdout:Sep 20 23:00:11 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-4[38043]: 2024-09-20T23:00:11.171+0000 7f44dc0a3640 -1 osd.4 1036 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-09-20T22:59:44.634301+0000 front 2024-09-20T22:59:44.634400+0000 (oldest deadline 2024-09-20T23:00:10.534351+0000) 2024-09-20T23:00:11.767 INFO:journalctl@ceph.osd.3.smithi084.stdout:Sep 20 23:00:11 smithi084 podman[226585]: 2024-09-20 23:00:11.481290101 +0000 UTC m=+5.560120248 container died 4b99a5b19760e8d69d99cf992ccfdfd60a94511163b1c5c5d47819c81aa481a2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-3, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, ceph=True, RELEASE=squid-9e2ac70) 2024-09-20T23:00:12.004 INFO:journalctl@ceph.osd.5.smithi154.stdout:Sep 20 23:00:11 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-5[43596]: 2024-09-20T23:00:11.687+0000 7fb11f7ec640 -1 osd.5 1036 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-09-20T22:59:45.278275+0000 front 2024-09-20T22:59:45.278360+0000 (oldest deadline 2024-09-20T23:00:11.678666+0000) 2024-09-20T23:00:12.005 INFO:journalctl@ceph.osd.6.smithi154.stdout:Sep 20 23:00:11 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-6[49255]: 2024-09-20T23:00:11.821+0000 7f2ff16a5640 -1 osd.6 1036 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-09-20T22:59:45.949062+0000 front 2024-09-20T22:59:45.949180+0000 (oldest deadline 2024-09-20T23:00:09.448861+0000) 2024-09-20T23:00:12.005 INFO:journalctl@ceph.osd.7.smithi154.stdout:Sep 20 23:00:11 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-7[54754]: 2024-09-20T23:00:11.680+0000 7fee5b5be640 -1 osd.7 1036 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-09-20T22:59:47.071784+0000 front 2024-09-20T22:59:47.071677+0000 (oldest deadline 2024-09-20T23:00:11.171776+0000) 2024-09-20T23:00:12.114 INFO:journalctl@ceph.osd.3.smithi084.stdout:Sep 20 23:00:11 smithi084 podman[226585]: 2024-09-20 23:00:11.957419936 +0000 UTC m=+6.036250082 container cleanup 4b99a5b19760e8d69d99cf992ccfdfd60a94511163b1c5c5d47819c81aa481a2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-3, RELEASE=squid-9e2ac70, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2) 2024-09-20T23:00:12.115 INFO:journalctl@ceph.osd.3.smithi084.stdout:Sep 20 23:00:11 smithi084 bash[226585]: ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-3 2024-09-20T23:00:12.504 INFO:journalctl@ceph.osd.4.smithi154.stdout:Sep 20 23:00:12 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-4[38043]: 2024-09-20T23:00:12.190+0000 7f44dc0a3640 -1 osd.4 1036 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-09-20T22:59:44.634301+0000 front 2024-09-20T22:59:44.634400+0000 (oldest deadline 2024-09-20T23:00:10.534351+0000) 2024-09-20T23:00:12.615 INFO:journalctl@ceph.osd.3.smithi084.stdout:Sep 20 23:00:12 smithi084 podman[226600]: 2024-09-20 23:00:12.301076927 +0000 UTC m=+0.810830153 container remove 4b99a5b19760e8d69d99cf992ccfdfd60a94511163b1c5c5d47819c81aa481a2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-3, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, RELEASE=squid-9e2ac70, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux ) 2024-09-20T23:00:12.880 INFO:journalctl@ceph.osd.3.smithi084.stdout:Sep 20 23:00:12 smithi084 podman[226690]: 2024-09-20 23:00:12.518945546 +0000 UTC m=+0.015305187 image pull cc9251b744a3faafca7c3095f79bfff81873afe823f31a03a11e015a34abcdbd quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5 2024-09-20T23:00:12.880 INFO:journalctl@ceph.osd.3.smithi084.stdout:Sep 20 23:00:12 smithi084 podman[226690]: 2024-09-20 23:00:12.618128974 +0000 UTC m=+0.114488596 container create 5d59fc3394334388caa9a2c0bbe48acc75f0576563b345c7b83f949f67a69854 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-3-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=squid-9e2ac70, maintainer=Guillaume Abrioux ) 2024-09-20T23:00:12.880 INFO:journalctl@ceph.osd.3.smithi084.stdout:Sep 20 23:00:12 smithi084 podman[226690]: 2024-09-20 23:00:12.80458333 +0000 UTC m=+0.300942964 container init 5d59fc3394334388caa9a2c0bbe48acc75f0576563b345c7b83f949f67a69854 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-3-deactivate, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=squid-9e2ac70, org.label-schema.schema-version=1.0) 2024-09-20T23:00:12.881 INFO:journalctl@ceph.osd.3.smithi084.stdout:Sep 20 23:00:12 smithi084 podman[226690]: 2024-09-20 23:00:12.808014911 +0000 UTC m=+0.304374541 container start 5d59fc3394334388caa9a2c0bbe48acc75f0576563b345c7b83f949f67a69854 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-3-deactivate, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-9e2ac70, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-09-20T23:00:12.881 INFO:journalctl@ceph.osd.3.smithi084.stdout:Sep 20 23:00:12 smithi084 podman[226690]: 2024-09-20 23:00:12.878602802 +0000 UTC m=+0.374962449 container attach 5d59fc3394334388caa9a2c0bbe48acc75f0576563b345c7b83f949f67a69854 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-3-deactivate, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=squid-9e2ac70, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-20T23:00:13.004 INFO:journalctl@ceph.osd.5.smithi154.stdout:Sep 20 23:00:12 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-5[43596]: 2024-09-20T23:00:12.639+0000 7fb11f7ec640 -1 osd.5 1036 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-09-20T22:59:45.278275+0000 front 2024-09-20T22:59:45.278360+0000 (oldest deadline 2024-09-20T23:00:11.678666+0000) 2024-09-20T23:00:13.004 INFO:journalctl@ceph.osd.7.smithi154.stdout:Sep 20 23:00:12 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-7[54754]: 2024-09-20T23:00:12.657+0000 7fee5b5be640 -1 osd.7 1036 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-09-20T22:59:47.071784+0000 front 2024-09-20T22:59:47.071677+0000 (oldest deadline 2024-09-20T23:00:11.171776+0000) 2024-09-20T23:00:13.005 INFO:journalctl@ceph.osd.6.smithi154.stdout:Sep 20 23:00:12 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-6[49255]: 2024-09-20T23:00:12.835+0000 7f2ff16a5640 -1 osd.6 1036 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-09-20T22:59:45.949062+0000 front 2024-09-20T22:59:45.949180+0000 (oldest deadline 2024-09-20T23:00:09.448861+0000) 2024-09-20T23:00:13.270 INFO:journalctl@ceph.osd.3.smithi084.stdout:Sep 20 23:00:12 smithi084 podman[226690]: 2024-09-20 23:00:12.998435773 +0000 UTC m=+0.494795420 container died 5d59fc3394334388caa9a2c0bbe48acc75f0576563b345c7b83f949f67a69854 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-3-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=squid-9e2ac70, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731) 2024-09-20T23:00:13.504 INFO:journalctl@ceph.osd.4.smithi154.stdout:Sep 20 23:00:13 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-4[38043]: 2024-09-20T23:00:13.198+0000 7f44dc0a3640 -1 osd.4 1036 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-09-20T22:59:44.634301+0000 front 2024-09-20T22:59:44.634400+0000 (oldest deadline 2024-09-20T23:00:10.534351+0000) 2024-09-20T23:00:13.559 DEBUG:teuthology.orchestra.run.smithi084:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490@osd.3.service' 2024-09-20T23:00:13.597 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-20T23:00:13.598 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-09-20T23:00:13.598 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-09-20T23:00:13.598 DEBUG:teuthology.orchestra.run.smithi154:> sudo systemctl stop ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490@osd.4 2024-09-20T23:00:13.885 INFO:journalctl@ceph.osd.4.smithi154.stdout:Sep 20 23:00:13 smithi154 systemd[1]: Stopping Ceph osd.4 for a9c0bc70-77a0-11ef-baf5-efdc52797490... 2024-09-20T23:00:13.886 INFO:journalctl@ceph.osd.5.smithi154.stdout:Sep 20 23:00:13 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-5[43596]: 2024-09-20T23:00:13.673+0000 7fb11f7ec640 -1 osd.5 1036 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-09-20T22:59:45.278275+0000 front 2024-09-20T22:59:45.278360+0000 (oldest deadline 2024-09-20T23:00:11.678666+0000) 2024-09-20T23:00:13.886 INFO:journalctl@ceph.osd.7.smithi154.stdout:Sep 20 23:00:13 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-7[54754]: 2024-09-20T23:00:13.646+0000 7fee5b5be640 -1 osd.7 1036 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-09-20T22:59:47.071784+0000 front 2024-09-20T22:59:47.071677+0000 (oldest deadline 2024-09-20T23:00:11.171776+0000) 2024-09-20T23:00:14.188 INFO:journalctl@ceph.osd.6.smithi154.stdout:Sep 20 23:00:13 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-6[49255]: 2024-09-20T23:00:13.883+0000 7f2ff16a5640 -1 osd.6 1036 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-09-20T22:59:45.949062+0000 front 2024-09-20T22:59:45.949180+0000 (oldest deadline 2024-09-20T23:00:09.448861+0000) 2024-09-20T23:00:14.188 INFO:journalctl@ceph.osd.4.smithi154.stdout:Sep 20 23:00:13 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-4[38043]: 2024-09-20T23:00:13.922+0000 7f44dfa8d640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-20T23:00:14.188 INFO:journalctl@ceph.osd.4.smithi154.stdout:Sep 20 23:00:13 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-4[38043]: 2024-09-20T23:00:13.922+0000 7f44dfa8d640 -1 osd.4 1036 *** Got signal Terminated *** 2024-09-20T23:00:14.188 INFO:journalctl@ceph.osd.4.smithi154.stdout:Sep 20 23:00:13 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-4[38043]: 2024-09-20T23:00:13.922+0000 7f44dfa8d640 -1 osd.4 1036 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-20T23:00:14.504 INFO:journalctl@ceph.osd.4.smithi154.stdout:Sep 20 23:00:14 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-4[38043]: 2024-09-20T23:00:14.186+0000 7f44dc0a3640 -1 osd.4 1036 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-09-20T22:59:44.634301+0000 front 2024-09-20T22:59:44.634400+0000 (oldest deadline 2024-09-20T23:00:10.534351+0000) 2024-09-20T23:00:15.004 INFO:journalctl@ceph.osd.6.smithi154.stdout:Sep 20 23:00:14 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-6[49255]: 2024-09-20T23:00:14.881+0000 7f2ff16a5640 -1 osd.6 1036 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-09-20T22:59:45.949062+0000 front 2024-09-20T22:59:45.949180+0000 (oldest deadline 2024-09-20T23:00:09.448861+0000) 2024-09-20T23:00:15.005 INFO:journalctl@ceph.osd.5.smithi154.stdout:Sep 20 23:00:14 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-5[43596]: 2024-09-20T23:00:14.708+0000 7fb11f7ec640 -1 osd.5 1036 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-09-20T22:59:45.278275+0000 front 2024-09-20T22:59:45.278360+0000 (oldest deadline 2024-09-20T23:00:11.678666+0000) 2024-09-20T23:00:15.005 INFO:journalctl@ceph.osd.7.smithi154.stdout:Sep 20 23:00:14 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-7[54754]: 2024-09-20T23:00:14.655+0000 7fee5b5be640 -1 osd.7 1036 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-09-20T22:59:47.071784+0000 front 2024-09-20T22:59:47.071677+0000 (oldest deadline 2024-09-20T23:00:11.171776+0000) 2024-09-20T23:00:15.504 INFO:journalctl@ceph.osd.4.smithi154.stdout:Sep 20 23:00:15 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-4[38043]: 2024-09-20T23:00:15.140+0000 7f44dc0a3640 -1 osd.4 1036 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-09-20T22:59:44.634301+0000 front 2024-09-20T22:59:44.634400+0000 (oldest deadline 2024-09-20T23:00:10.534351+0000) 2024-09-20T23:00:16.004 INFO:journalctl@ceph.osd.6.smithi154.stdout:Sep 20 23:00:15 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-6[49255]: 2024-09-20T23:00:15.889+0000 7f2ff16a5640 -1 osd.6 1036 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-09-20T22:59:45.949062+0000 front 2024-09-20T22:59:45.949180+0000 (oldest deadline 2024-09-20T23:00:09.448861+0000) 2024-09-20T23:00:16.004 INFO:journalctl@ceph.osd.7.smithi154.stdout:Sep 20 23:00:15 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-7[54754]: 2024-09-20T23:00:15.649+0000 7fee5b5be640 -1 osd.7 1036 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-09-20T22:59:47.071784+0000 front 2024-09-20T22:59:47.071677+0000 (oldest deadline 2024-09-20T23:00:11.171776+0000) 2024-09-20T23:00:16.005 INFO:journalctl@ceph.osd.5.smithi154.stdout:Sep 20 23:00:15 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-5[43596]: 2024-09-20T23:00:15.742+0000 7fb11f7ec640 -1 osd.5 1036 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-09-20T22:59:45.278275+0000 front 2024-09-20T22:59:45.278360+0000 (oldest deadline 2024-09-20T23:00:11.678666+0000) 2024-09-20T23:00:16.005 INFO:journalctl@ceph.osd.5.smithi154.stdout:Sep 20 23:00:15 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-5[43596]: 2024-09-20T23:00:15.742+0000 7fb11f7ec640 -1 osd.5 1036 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-09-20T22:59:53.979697+0000 front 2024-09-20T22:59:53.979548+0000 (oldest deadline 2024-09-20T23:00:15.679666+0000) 2024-09-20T23:00:16.504 INFO:journalctl@ceph.osd.4.smithi154.stdout:Sep 20 23:00:16 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-4[38043]: 2024-09-20T23:00:16.179+0000 7f44dc0a3640 -1 osd.4 1036 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-09-20T22:59:44.634301+0000 front 2024-09-20T22:59:44.634400+0000 (oldest deadline 2024-09-20T23:00:10.534351+0000) 2024-09-20T23:00:16.903 INFO:journalctl@ceph.osd.7.smithi154.stdout:Sep 20 23:00:16 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-7[54754]: 2024-09-20T23:00:16.600+0000 7fee5b5be640 -1 osd.7 1036 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-09-20T22:59:47.071784+0000 front 2024-09-20T22:59:47.071677+0000 (oldest deadline 2024-09-20T23:00:11.171776+0000) 2024-09-20T23:00:16.903 INFO:journalctl@ceph.osd.7.smithi154.stdout:Sep 20 23:00:16 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-7[54754]: 2024-09-20T23:00:16.600+0000 7fee5b5be640 -1 osd.7 1036 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-09-20T22:59:51.172472+0000 front 2024-09-20T22:59:51.172371+0000 (oldest deadline 2024-09-20T23:00:16.472189+0000) 2024-09-20T23:00:16.903 INFO:journalctl@ceph.osd.5.smithi154.stdout:Sep 20 23:00:16 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-5[43596]: 2024-09-20T23:00:16.724+0000 7fb11f7ec640 -1 osd.5 1036 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-09-20T22:59:45.278275+0000 front 2024-09-20T22:59:45.278360+0000 (oldest deadline 2024-09-20T23:00:11.678666+0000) 2024-09-20T23:00:16.903 INFO:journalctl@ceph.osd.5.smithi154.stdout:Sep 20 23:00:16 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-5[43596]: 2024-09-20T23:00:16.724+0000 7fb11f7ec640 -1 osd.5 1036 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-09-20T22:59:53.979697+0000 front 2024-09-20T22:59:53.979548+0000 (oldest deadline 2024-09-20T23:00:15.679666+0000) 2024-09-20T23:00:17.178 INFO:journalctl@ceph.osd.6.smithi154.stdout:Sep 20 23:00:16 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-6[49255]: 2024-09-20T23:00:16.901+0000 7f2ff16a5640 -1 osd.6 1036 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-09-20T22:59:45.949062+0000 front 2024-09-20T22:59:45.949180+0000 (oldest deadline 2024-09-20T23:00:09.448861+0000) 2024-09-20T23:00:17.504 INFO:journalctl@ceph.osd.4.smithi154.stdout:Sep 20 23:00:17 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-4[38043]: 2024-09-20T23:00:17.176+0000 7f44dc0a3640 -1 osd.4 1036 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-09-20T22:59:44.634301+0000 front 2024-09-20T22:59:44.634400+0000 (oldest deadline 2024-09-20T23:00:10.534351+0000) 2024-09-20T23:00:17.504 INFO:journalctl@ceph.osd.4.smithi154.stdout:Sep 20 23:00:17 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-4[38043]: 2024-09-20T23:00:17.176+0000 7f44dc0a3640 -1 osd.4 1036 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-09-20T22:59:52.235331+0000 front 2024-09-20T22:59:52.235369+0000 (oldest deadline 2024-09-20T23:00:16.935335+0000) 2024-09-20T23:00:18.004 INFO:journalctl@ceph.osd.5.smithi154.stdout:Sep 20 23:00:17 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-5[43596]: 2024-09-20T23:00:17.763+0000 7fb11f7ec640 -1 osd.5 1036 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-09-20T22:59:45.278275+0000 front 2024-09-20T22:59:45.278360+0000 (oldest deadline 2024-09-20T23:00:11.678666+0000) 2024-09-20T23:00:18.004 INFO:journalctl@ceph.osd.5.smithi154.stdout:Sep 20 23:00:17 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-5[43596]: 2024-09-20T23:00:17.763+0000 7fb11f7ec640 -1 osd.5 1036 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-09-20T22:59:53.979697+0000 front 2024-09-20T22:59:53.979548+0000 (oldest deadline 2024-09-20T23:00:15.679666+0000) 2024-09-20T23:00:18.005 INFO:journalctl@ceph.osd.7.smithi154.stdout:Sep 20 23:00:17 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-7[54754]: 2024-09-20T23:00:17.650+0000 7fee5b5be640 -1 osd.7 1036 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-09-20T22:59:47.071784+0000 front 2024-09-20T22:59:47.071677+0000 (oldest deadline 2024-09-20T23:00:11.171776+0000) 2024-09-20T23:00:18.005 INFO:journalctl@ceph.osd.7.smithi154.stdout:Sep 20 23:00:17 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-7[54754]: 2024-09-20T23:00:17.650+0000 7fee5b5be640 -1 osd.7 1036 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-09-20T22:59:51.172472+0000 front 2024-09-20T22:59:51.172371+0000 (oldest deadline 2024-09-20T23:00:16.472189+0000) 2024-09-20T23:00:18.005 INFO:journalctl@ceph.osd.6.smithi154.stdout:Sep 20 23:00:17 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-6[49255]: 2024-09-20T23:00:17.897+0000 7f2ff16a5640 -1 osd.6 1036 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-09-20T22:59:45.949062+0000 front 2024-09-20T22:59:45.949180+0000 (oldest deadline 2024-09-20T23:00:09.448861+0000) 2024-09-20T23:00:18.005 INFO:journalctl@ceph.osd.6.smithi154.stdout:Sep 20 23:00:17 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-6[49255]: 2024-09-20T23:00:17.897+0000 7f2ff16a5640 -1 osd.6 1036 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-09-20T22:59:51.650246+0000 front 2024-09-20T22:59:51.650286+0000 (oldest deadline 2024-09-20T23:00:16.950204+0000) 2024-09-20T23:00:18.504 INFO:journalctl@ceph.osd.4.smithi154.stdout:Sep 20 23:00:18 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-4[38043]: 2024-09-20T23:00:18.224+0000 7f44dc0a3640 -1 osd.4 1036 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-09-20T22:59:44.634301+0000 front 2024-09-20T22:59:44.634400+0000 (oldest deadline 2024-09-20T23:00:10.534351+0000) 2024-09-20T23:00:18.504 INFO:journalctl@ceph.osd.4.smithi154.stdout:Sep 20 23:00:18 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-4[38043]: 2024-09-20T23:00:18.224+0000 7f44dc0a3640 -1 osd.4 1036 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-09-20T22:59:52.235331+0000 front 2024-09-20T22:59:52.235369+0000 (oldest deadline 2024-09-20T23:00:16.935335+0000) 2024-09-20T23:00:19.004 INFO:journalctl@ceph.osd.7.smithi154.stdout:Sep 20 23:00:18 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-7[54754]: 2024-09-20T23:00:18.699+0000 7fee5b5be640 -1 osd.7 1036 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-09-20T22:59:47.071784+0000 front 2024-09-20T22:59:47.071677+0000 (oldest deadline 2024-09-20T23:00:11.171776+0000) 2024-09-20T23:00:19.004 INFO:journalctl@ceph.osd.7.smithi154.stdout:Sep 20 23:00:18 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-7[54754]: 2024-09-20T23:00:18.699+0000 7fee5b5be640 -1 osd.7 1036 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-09-20T22:59:51.172472+0000 front 2024-09-20T22:59:51.172371+0000 (oldest deadline 2024-09-20T23:00:16.472189+0000) 2024-09-20T23:00:19.005 INFO:journalctl@ceph.osd.5.smithi154.stdout:Sep 20 23:00:18 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-5[43596]: 2024-09-20T23:00:18.714+0000 7fb11f7ec640 -1 osd.5 1036 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-09-20T22:59:45.278275+0000 front 2024-09-20T22:59:45.278360+0000 (oldest deadline 2024-09-20T23:00:11.678666+0000) 2024-09-20T23:00:19.005 INFO:journalctl@ceph.osd.5.smithi154.stdout:Sep 20 23:00:18 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-5[43596]: 2024-09-20T23:00:18.714+0000 7fb11f7ec640 -1 osd.5 1036 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-09-20T22:59:53.979697+0000 front 2024-09-20T22:59:53.979548+0000 (oldest deadline 2024-09-20T23:00:15.679666+0000) 2024-09-20T23:00:19.005 INFO:journalctl@ceph.osd.6.smithi154.stdout:Sep 20 23:00:18 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-6[49255]: 2024-09-20T23:00:18.938+0000 7f2ff16a5640 -1 osd.6 1036 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-09-20T22:59:45.949062+0000 front 2024-09-20T22:59:45.949180+0000 (oldest deadline 2024-09-20T23:00:09.448861+0000) 2024-09-20T23:00:19.005 INFO:journalctl@ceph.osd.6.smithi154.stdout:Sep 20 23:00:18 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-6[49255]: 2024-09-20T23:00:18.938+0000 7f2ff16a5640 -1 osd.6 1036 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-09-20T22:59:51.650246+0000 front 2024-09-20T22:59:51.650286+0000 (oldest deadline 2024-09-20T23:00:16.950204+0000) 2024-09-20T23:00:19.505 INFO:journalctl@ceph.osd.4.smithi154.stdout:Sep 20 23:00:19 smithi154 podman[153183]: 2024-09-20 23:00:19.24938354 +0000 UTC m=+5.453522635 container died 5523d2ea6f707d789c2e5d072b1ad110973345c44875156b12fc1fab57a5eb62 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-4, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, RELEASE=squid-9e2ac70, io.buildah.version=1.37.2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=) 2024-09-20T23:00:19.954 INFO:journalctl@ceph.osd.5.smithi154.stdout:Sep 20 23:00:19 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-5[43596]: 2024-09-20T23:00:19.710+0000 7fb11f7ec640 -1 osd.5 1036 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-09-20T22:59:45.278275+0000 front 2024-09-20T22:59:45.278360+0000 (oldest deadline 2024-09-20T23:00:11.678666+0000) 2024-09-20T23:00:19.954 INFO:journalctl@ceph.osd.5.smithi154.stdout:Sep 20 23:00:19 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-5[43596]: 2024-09-20T23:00:19.710+0000 7fb11f7ec640 -1 osd.5 1036 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-09-20T22:59:53.979697+0000 front 2024-09-20T22:59:53.979548+0000 (oldest deadline 2024-09-20T23:00:15.679666+0000) 2024-09-20T23:00:19.954 INFO:journalctl@ceph.osd.4.smithi154.stdout:Sep 20 23:00:19 smithi154 podman[153183]: 2024-09-20 23:00:19.760086945 +0000 UTC m=+5.964226082 container cleanup 5523d2ea6f707d789c2e5d072b1ad110973345c44875156b12fc1fab57a5eb62 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-4, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, RELEASE=squid-9e2ac70, CEPH_POINT_RELEASE=) 2024-09-20T23:00:19.955 INFO:journalctl@ceph.osd.4.smithi154.stdout:Sep 20 23:00:19 smithi154 bash[153183]: ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-4 2024-09-20T23:00:19.955 INFO:journalctl@ceph.osd.7.smithi154.stdout:Sep 20 23:00:19 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-7[54754]: 2024-09-20T23:00:19.732+0000 7fee5b5be640 -1 osd.7 1036 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-09-20T22:59:47.071784+0000 front 2024-09-20T22:59:47.071677+0000 (oldest deadline 2024-09-20T23:00:11.171776+0000) 2024-09-20T23:00:19.955 INFO:journalctl@ceph.osd.7.smithi154.stdout:Sep 20 23:00:19 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-7[54754]: 2024-09-20T23:00:19.732+0000 7fee5b5be640 -1 osd.7 1036 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-09-20T22:59:51.172472+0000 front 2024-09-20T22:59:51.172371+0000 (oldest deadline 2024-09-20T23:00:16.472189+0000) 2024-09-20T23:00:20.254 INFO:journalctl@ceph.osd.4.smithi154.stdout:Sep 20 23:00:19 smithi154 podman[153199]: 2024-09-20 23:00:19.953051504 +0000 UTC m=+0.700572684 container remove 5523d2ea6f707d789c2e5d072b1ad110973345c44875156b12fc1fab57a5eb62 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-4, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=squid-9e2ac70, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0) 2024-09-20T23:00:20.255 INFO:journalctl@ceph.osd.6.smithi154.stdout:Sep 20 23:00:19 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-6[49255]: 2024-09-20T23:00:19.974+0000 7f2ff16a5640 -1 osd.6 1036 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-09-20T22:59:45.949062+0000 front 2024-09-20T22:59:45.949180+0000 (oldest deadline 2024-09-20T23:00:09.448861+0000) 2024-09-20T23:00:20.255 INFO:journalctl@ceph.osd.6.smithi154.stdout:Sep 20 23:00:19 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-6[49255]: 2024-09-20T23:00:19.974+0000 7f2ff16a5640 -1 osd.6 1036 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-09-20T22:59:51.650246+0000 front 2024-09-20T22:59:51.650286+0000 (oldest deadline 2024-09-20T23:00:16.950204+0000) 2024-09-20T23:00:20.558 INFO:journalctl@ceph.osd.4.smithi154.stdout:Sep 20 23:00:20 smithi154 podman[153290]: 2024-09-20 23:00:20.179599639 +0000 UTC m=+0.015965323 image pull cc9251b744a3faafca7c3095f79bfff81873afe823f31a03a11e015a34abcdbd quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5 2024-09-20T23:00:20.558 INFO:journalctl@ceph.osd.4.smithi154.stdout:Sep 20 23:00:20 smithi154 podman[153290]: 2024-09-20 23:00:20.290684665 +0000 UTC m=+0.127050346 container create 7a7aeba6ac6cead23d855f1b7e0994b9a21c57d1d293e5d7ed60310077241dce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-4-deactivate, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.license=GPLv2, RELEASE=squid-9e2ac70, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-20T23:00:20.558 INFO:journalctl@ceph.osd.4.smithi154.stdout:Sep 20 23:00:20 smithi154 podman[153290]: 2024-09-20 23:00:20.498602052 +0000 UTC m=+0.334967738 container init 7a7aeba6ac6cead23d855f1b7e0994b9a21c57d1d293e5d7ed60310077241dce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-4-deactivate, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=squid-9e2ac70, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-09-20T23:00:20.558 INFO:journalctl@ceph.osd.4.smithi154.stdout:Sep 20 23:00:20 smithi154 podman[153290]: 2024-09-20 23:00:20.502048352 +0000 UTC m=+0.338414032 container start 7a7aeba6ac6cead23d855f1b7e0994b9a21c57d1d293e5d7ed60310077241dce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-4-deactivate, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=squid-9e2ac70, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-20T23:00:20.558 INFO:journalctl@ceph.osd.4.smithi154.stdout:Sep 20 23:00:20 smithi154 podman[153290]: 2024-09-20 23:00:20.556944723 +0000 UTC m=+0.393310414 container attach 7a7aeba6ac6cead23d855f1b7e0994b9a21c57d1d293e5d7ed60310077241dce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-4-deactivate, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-9e2ac70, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-09-20T23:00:20.979 INFO:journalctl@ceph.osd.4.smithi154.stdout:Sep 20 23:00:20 smithi154 podman[153290]: 2024-09-20 23:00:20.692326525 +0000 UTC m=+0.528692207 container died 7a7aeba6ac6cead23d855f1b7e0994b9a21c57d1d293e5d7ed60310077241dce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-4-deactivate, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, RELEASE=squid-9e2ac70, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-09-20T23:00:20.979 INFO:journalctl@ceph.osd.5.smithi154.stdout:Sep 20 23:00:20 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-5[43596]: 2024-09-20T23:00:20.689+0000 7fb11f7ec640 -1 osd.5 1036 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-09-20T22:59:45.278275+0000 front 2024-09-20T22:59:45.278360+0000 (oldest deadline 2024-09-20T23:00:11.678666+0000) 2024-09-20T23:00:20.979 INFO:journalctl@ceph.osd.5.smithi154.stdout:Sep 20 23:00:20 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-5[43596]: 2024-09-20T23:00:20.689+0000 7fb11f7ec640 -1 osd.5 1036 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-09-20T22:59:53.979697+0000 front 2024-09-20T22:59:53.979548+0000 (oldest deadline 2024-09-20T23:00:15.679666+0000) 2024-09-20T23:00:20.979 INFO:journalctl@ceph.osd.7.smithi154.stdout:Sep 20 23:00:20 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-7[54754]: 2024-09-20T23:00:20.716+0000 7fee5b5be640 -1 osd.7 1036 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-09-20T22:59:47.071784+0000 front 2024-09-20T22:59:47.071677+0000 (oldest deadline 2024-09-20T23:00:11.171776+0000) 2024-09-20T23:00:20.979 INFO:journalctl@ceph.osd.7.smithi154.stdout:Sep 20 23:00:20 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-7[54754]: 2024-09-20T23:00:20.716+0000 7fee5b5be640 -1 osd.7 1036 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-09-20T22:59:51.172472+0000 front 2024-09-20T22:59:51.172371+0000 (oldest deadline 2024-09-20T23:00:16.472189+0000) 2024-09-20T23:00:21.254 INFO:journalctl@ceph.osd.6.smithi154.stdout:Sep 20 23:00:21 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-6[49255]: 2024-09-20T23:00:21.024+0000 7f2ff16a5640 -1 osd.6 1036 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-09-20T22:59:45.949062+0000 front 2024-09-20T22:59:45.949180+0000 (oldest deadline 2024-09-20T23:00:09.448861+0000) 2024-09-20T23:00:21.254 INFO:journalctl@ceph.osd.6.smithi154.stdout:Sep 20 23:00:21 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-6[49255]: 2024-09-20T23:00:21.024+0000 7f2ff16a5640 -1 osd.6 1036 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-09-20T22:59:51.650246+0000 front 2024-09-20T22:59:51.650286+0000 (oldest deadline 2024-09-20T23:00:16.950204+0000) 2024-09-20T23:00:21.292 DEBUG:teuthology.orchestra.run.smithi154:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490@osd.4.service' 2024-09-20T23:00:21.333 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-20T23:00:21.334 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-09-20T23:00:21.334 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-09-20T23:00:21.334 DEBUG:teuthology.orchestra.run.smithi154:> sudo systemctl stop ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490@osd.5 2024-09-20T23:00:21.558 INFO:journalctl@ceph.osd.5.smithi154.stdout:Sep 20 23:00:21 smithi154 systemd[1]: Stopping Ceph osd.5 for a9c0bc70-77a0-11ef-baf5-efdc52797490... 2024-09-20T23:00:21.992 INFO:journalctl@ceph.osd.5.smithi154.stdout:Sep 20 23:00:21 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-5[43596]: 2024-09-20T23:00:21.655+0000 7fb11f7ec640 -1 osd.5 1036 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-09-20T22:59:45.278275+0000 front 2024-09-20T22:59:45.278360+0000 (oldest deadline 2024-09-20T23:00:11.678666+0000) 2024-09-20T23:00:21.992 INFO:journalctl@ceph.osd.5.smithi154.stdout:Sep 20 23:00:21 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-5[43596]: 2024-09-20T23:00:21.655+0000 7fb11f7ec640 -1 osd.5 1036 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-09-20T22:59:53.979697+0000 front 2024-09-20T22:59:53.979548+0000 (oldest deadline 2024-09-20T23:00:15.679666+0000) 2024-09-20T23:00:21.992 INFO:journalctl@ceph.osd.5.smithi154.stdout:Sep 20 23:00:21 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-5[43596]: 2024-09-20T23:00:21.696+0000 7fb1231d6640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-20T23:00:21.992 INFO:journalctl@ceph.osd.5.smithi154.stdout:Sep 20 23:00:21 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-5[43596]: 2024-09-20T23:00:21.696+0000 7fb1231d6640 -1 osd.5 1036 *** Got signal Terminated *** 2024-09-20T23:00:21.992 INFO:journalctl@ceph.osd.5.smithi154.stdout:Sep 20 23:00:21 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-5[43596]: 2024-09-20T23:00:21.696+0000 7fb1231d6640 -1 osd.5 1036 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-20T23:00:21.992 INFO:journalctl@ceph.osd.7.smithi154.stdout:Sep 20 23:00:21 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-7[54754]: 2024-09-20T23:00:21.679+0000 7fee5b5be640 -1 osd.7 1036 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-09-20T22:59:47.071784+0000 front 2024-09-20T22:59:47.071677+0000 (oldest deadline 2024-09-20T23:00:11.171776+0000) 2024-09-20T23:00:21.993 INFO:journalctl@ceph.osd.7.smithi154.stdout:Sep 20 23:00:21 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-7[54754]: 2024-09-20T23:00:21.679+0000 7fee5b5be640 -1 osd.7 1036 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-09-20T22:59:51.172472+0000 front 2024-09-20T22:59:51.172371+0000 (oldest deadline 2024-09-20T23:00:16.472189+0000) 2024-09-20T23:00:22.254 INFO:journalctl@ceph.osd.6.smithi154.stdout:Sep 20 23:00:21 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-6[49255]: 2024-09-20T23:00:21.989+0000 7f2ff16a5640 -1 osd.6 1036 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-09-20T22:59:45.949062+0000 front 2024-09-20T22:59:45.949180+0000 (oldest deadline 2024-09-20T23:00:09.448861+0000) 2024-09-20T23:00:22.254 INFO:journalctl@ceph.osd.6.smithi154.stdout:Sep 20 23:00:21 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-6[49255]: 2024-09-20T23:00:21.989+0000 7f2ff16a5640 -1 osd.6 1036 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-09-20T22:59:51.650246+0000 front 2024-09-20T22:59:51.650286+0000 (oldest deadline 2024-09-20T23:00:16.950204+0000) 2024-09-20T23:00:22.972 INFO:journalctl@ceph.osd.5.smithi154.stdout:Sep 20 23:00:22 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-5[43596]: 2024-09-20T23:00:22.607+0000 7fb11f7ec640 -1 osd.5 1036 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-09-20T22:59:45.278275+0000 front 2024-09-20T22:59:45.278360+0000 (oldest deadline 2024-09-20T23:00:11.678666+0000) 2024-09-20T23:00:22.972 INFO:journalctl@ceph.osd.5.smithi154.stdout:Sep 20 23:00:22 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-5[43596]: 2024-09-20T23:00:22.607+0000 7fb11f7ec640 -1 osd.5 1036 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-09-20T22:59:53.979697+0000 front 2024-09-20T22:59:53.979548+0000 (oldest deadline 2024-09-20T23:00:15.679666+0000) 2024-09-20T23:00:22.973 INFO:journalctl@ceph.osd.7.smithi154.stdout:Sep 20 23:00:22 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-7[54754]: 2024-09-20T23:00:22.645+0000 7fee5b5be640 -1 osd.7 1036 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-09-20T22:59:47.071784+0000 front 2024-09-20T22:59:47.071677+0000 (oldest deadline 2024-09-20T23:00:11.171776+0000) 2024-09-20T23:00:22.973 INFO:journalctl@ceph.osd.7.smithi154.stdout:Sep 20 23:00:22 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-7[54754]: 2024-09-20T23:00:22.645+0000 7fee5b5be640 -1 osd.7 1036 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-09-20T22:59:51.172472+0000 front 2024-09-20T22:59:51.172371+0000 (oldest deadline 2024-09-20T23:00:16.472189+0000) 2024-09-20T23:00:23.254 INFO:journalctl@ceph.osd.6.smithi154.stdout:Sep 20 23:00:22 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-6[49255]: 2024-09-20T23:00:22.970+0000 7f2ff16a5640 -1 osd.6 1036 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-09-20T22:59:45.949062+0000 front 2024-09-20T22:59:45.949180+0000 (oldest deadline 2024-09-20T23:00:09.448861+0000) 2024-09-20T23:00:23.254 INFO:journalctl@ceph.osd.6.smithi154.stdout:Sep 20 23:00:22 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-6[49255]: 2024-09-20T23:00:22.970+0000 7f2ff16a5640 -1 osd.6 1036 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-09-20T22:59:51.650246+0000 front 2024-09-20T22:59:51.650286+0000 (oldest deadline 2024-09-20T23:00:16.950204+0000) 2024-09-20T23:00:23.961 INFO:journalctl@ceph.osd.5.smithi154.stdout:Sep 20 23:00:23 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-5[43596]: 2024-09-20T23:00:23.591+0000 7fb11f7ec640 -1 osd.5 1036 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-09-20T22:59:45.278275+0000 front 2024-09-20T22:59:45.278360+0000 (oldest deadline 2024-09-20T23:00:11.678666+0000) 2024-09-20T23:00:23.961 INFO:journalctl@ceph.osd.5.smithi154.stdout:Sep 20 23:00:23 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-5[43596]: 2024-09-20T23:00:23.591+0000 7fb11f7ec640 -1 osd.5 1036 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-09-20T22:59:53.979697+0000 front 2024-09-20T22:59:53.979548+0000 (oldest deadline 2024-09-20T23:00:15.679666+0000) 2024-09-20T23:00:23.961 INFO:journalctl@ceph.osd.7.smithi154.stdout:Sep 20 23:00:23 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-7[54754]: 2024-09-20T23:00:23.690+0000 7fee5b5be640 -1 osd.7 1036 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-09-20T22:59:47.071784+0000 front 2024-09-20T22:59:47.071677+0000 (oldest deadline 2024-09-20T23:00:11.171776+0000) 2024-09-20T23:00:23.962 INFO:journalctl@ceph.osd.7.smithi154.stdout:Sep 20 23:00:23 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-7[54754]: 2024-09-20T23:00:23.690+0000 7fee5b5be640 -1 osd.7 1036 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-09-20T22:59:51.172472+0000 front 2024-09-20T22:59:51.172371+0000 (oldest deadline 2024-09-20T23:00:16.472189+0000) 2024-09-20T23:00:24.254 INFO:journalctl@ceph.osd.6.smithi154.stdout:Sep 20 23:00:23 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-6[49255]: 2024-09-20T23:00:23.959+0000 7f2ff16a5640 -1 osd.6 1036 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-09-20T22:59:45.949062+0000 front 2024-09-20T22:59:45.949180+0000 (oldest deadline 2024-09-20T23:00:09.448861+0000) 2024-09-20T23:00:24.255 INFO:journalctl@ceph.osd.6.smithi154.stdout:Sep 20 23:00:23 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-6[49255]: 2024-09-20T23:00:23.959+0000 7f2ff16a5640 -1 osd.6 1036 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-09-20T22:59:51.650246+0000 front 2024-09-20T22:59:51.650286+0000 (oldest deadline 2024-09-20T23:00:16.950204+0000) 2024-09-20T23:00:24.255 INFO:journalctl@ceph.prometheus.a.smithi154.stdout:Sep 20 23:00:24 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-prometheus-a[67141]: ts=2024-09-20T23:00:24.159Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph-exporter msg="Unable to refresh target groups" err="Get \"http://172.21.15.84:8765/sd/prometheus/sd-config?service=ceph-exporter\": dial tcp 172.21.15.84:8765: connect: connection refused" 2024-09-20T23:00:24.255 INFO:journalctl@ceph.prometheus.a.smithi154.stdout:Sep 20 23:00:24 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-prometheus-a[67141]: ts=2024-09-20T23:00:24.161Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=node msg="Unable to refresh target groups" err="Get \"http://172.21.15.84:8765/sd/prometheus/sd-config?service=node-exporter\": dial tcp 172.21.15.84:8765: connect: connection refused" 2024-09-20T23:00:24.255 INFO:journalctl@ceph.prometheus.a.smithi154.stdout:Sep 20 23:00:24 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-prometheus-a[67141]: ts=2024-09-20T23:00:24.161Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=nvmeof msg="Unable to refresh target groups" err="Get \"http://172.21.15.84:8765/sd/prometheus/sd-config?service=nvmeof\": dial tcp 172.21.15.84:8765: connect: connection refused" 2024-09-20T23:00:24.255 INFO:journalctl@ceph.prometheus.a.smithi154.stdout:Sep 20 23:00:24 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-prometheus-a[67141]: ts=2024-09-20T23:00:24.161Z caller=refresh.go:99 level=error component="discovery manager notify" discovery=http config=config-0 msg="Unable to refresh target groups" err="Get \"http://172.21.15.84:8765/sd/prometheus/sd-config?service=alertmanager\": dial tcp 172.21.15.84:8765: connect: connection refused" 2024-09-20T23:00:24.255 INFO:journalctl@ceph.prometheus.a.smithi154.stdout:Sep 20 23:00:24 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-prometheus-a[67141]: ts=2024-09-20T23:00:24.161Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph msg="Unable to refresh target groups" err="Get \"http://172.21.15.84:8765/sd/prometheus/sd-config?service=mgr-prometheus\": dial tcp 172.21.15.84:8765: connect: connection refused" 2024-09-20T23:00:24.972 INFO:journalctl@ceph.osd.7.smithi154.stdout:Sep 20 23:00:24 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-7[54754]: 2024-09-20T23:00:24.667+0000 7fee5b5be640 -1 osd.7 1036 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-09-20T22:59:47.071784+0000 front 2024-09-20T22:59:47.071677+0000 (oldest deadline 2024-09-20T23:00:11.171776+0000) 2024-09-20T23:00:24.972 INFO:journalctl@ceph.osd.7.smithi154.stdout:Sep 20 23:00:24 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-7[54754]: 2024-09-20T23:00:24.667+0000 7fee5b5be640 -1 osd.7 1036 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-09-20T22:59:51.172472+0000 front 2024-09-20T22:59:51.172371+0000 (oldest deadline 2024-09-20T23:00:16.472189+0000) 2024-09-20T23:00:24.972 INFO:journalctl@ceph.osd.7.smithi154.stdout:Sep 20 23:00:24 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-7[54754]: 2024-09-20T23:00:24.667+0000 7fee5b5be640 -1 osd.7 1036 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-09-20T23:00:00.573109+0000 front 2024-09-20T23:00:00.573085+0000 (oldest deadline 2024-09-20T23:00:24.072955+0000) 2024-09-20T23:00:24.972 INFO:journalctl@ceph.osd.5.smithi154.stdout:Sep 20 23:00:24 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-5[43596]: 2024-09-20T23:00:24.606+0000 7fb11f7ec640 -1 osd.5 1036 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-09-20T22:59:45.278275+0000 front 2024-09-20T22:59:45.278360+0000 (oldest deadline 2024-09-20T23:00:11.678666+0000) 2024-09-20T23:00:24.972 INFO:journalctl@ceph.osd.5.smithi154.stdout:Sep 20 23:00:24 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-5[43596]: 2024-09-20T23:00:24.606+0000 7fb11f7ec640 -1 osd.5 1036 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-09-20T22:59:53.979697+0000 front 2024-09-20T22:59:53.979548+0000 (oldest deadline 2024-09-20T23:00:15.679666+0000) 2024-09-20T23:00:24.972 INFO:journalctl@ceph.osd.5.smithi154.stdout:Sep 20 23:00:24 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-5[43596]: 2024-09-20T23:00:24.606+0000 7fb11f7ec640 -1 osd.5 1036 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-09-20T23:00:01.381589+0000 front 2024-09-20T23:00:01.381676+0000 (oldest deadline 2024-09-20T23:00:23.681563+0000) 2024-09-20T23:00:25.254 INFO:journalctl@ceph.osd.6.smithi154.stdout:Sep 20 23:00:24 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-6[49255]: 2024-09-20T23:00:24.970+0000 7f2ff16a5640 -1 osd.6 1036 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-09-20T22:59:45.949062+0000 front 2024-09-20T22:59:45.949180+0000 (oldest deadline 2024-09-20T23:00:09.448861+0000) 2024-09-20T23:00:25.254 INFO:journalctl@ceph.osd.6.smithi154.stdout:Sep 20 23:00:24 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-6[49255]: 2024-09-20T23:00:24.970+0000 7f2ff16a5640 -1 osd.6 1036 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-09-20T22:59:51.650246+0000 front 2024-09-20T22:59:51.650286+0000 (oldest deadline 2024-09-20T23:00:16.950204+0000) 2024-09-20T23:00:25.255 INFO:journalctl@ceph.osd.6.smithi154.stdout:Sep 20 23:00:24 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-6[49255]: 2024-09-20T23:00:24.970+0000 7f2ff16a5640 -1 osd.6 1036 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-09-20T22:59:59.250995+0000 front 2024-09-20T22:59:59.251043+0000 (oldest deadline 2024-09-20T23:00:24.550992+0000) 2024-09-20T23:00:25.988 INFO:journalctl@ceph.osd.5.smithi154.stdout:Sep 20 23:00:25 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-5[43596]: 2024-09-20T23:00:25.611+0000 7fb11f7ec640 -1 osd.5 1036 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-09-20T22:59:45.278275+0000 front 2024-09-20T22:59:45.278360+0000 (oldest deadline 2024-09-20T23:00:11.678666+0000) 2024-09-20T23:00:25.988 INFO:journalctl@ceph.osd.5.smithi154.stdout:Sep 20 23:00:25 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-5[43596]: 2024-09-20T23:00:25.611+0000 7fb11f7ec640 -1 osd.5 1036 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-09-20T22:59:53.979697+0000 front 2024-09-20T22:59:53.979548+0000 (oldest deadline 2024-09-20T23:00:15.679666+0000) 2024-09-20T23:00:25.988 INFO:journalctl@ceph.osd.5.smithi154.stdout:Sep 20 23:00:25 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-5[43596]: 2024-09-20T23:00:25.611+0000 7fb11f7ec640 -1 osd.5 1036 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-09-20T23:00:01.381589+0000 front 2024-09-20T23:00:01.381676+0000 (oldest deadline 2024-09-20T23:00:23.681563+0000) 2024-09-20T23:00:25.988 INFO:journalctl@ceph.osd.7.smithi154.stdout:Sep 20 23:00:25 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-7[54754]: 2024-09-20T23:00:25.625+0000 7fee5b5be640 -1 osd.7 1036 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-09-20T22:59:47.071784+0000 front 2024-09-20T22:59:47.071677+0000 (oldest deadline 2024-09-20T23:00:11.171776+0000) 2024-09-20T23:00:25.988 INFO:journalctl@ceph.osd.7.smithi154.stdout:Sep 20 23:00:25 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-7[54754]: 2024-09-20T23:00:25.625+0000 7fee5b5be640 -1 osd.7 1036 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-09-20T22:59:51.172472+0000 front 2024-09-20T22:59:51.172371+0000 (oldest deadline 2024-09-20T23:00:16.472189+0000) 2024-09-20T23:00:25.988 INFO:journalctl@ceph.osd.7.smithi154.stdout:Sep 20 23:00:25 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-7[54754]: 2024-09-20T23:00:25.625+0000 7fee5b5be640 -1 osd.7 1036 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-09-20T23:00:00.573109+0000 front 2024-09-20T23:00:00.573085+0000 (oldest deadline 2024-09-20T23:00:24.072955+0000) 2024-09-20T23:00:26.255 INFO:journalctl@ceph.osd.6.smithi154.stdout:Sep 20 23:00:25 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-6[49255]: 2024-09-20T23:00:25.986+0000 7f2ff16a5640 -1 osd.6 1036 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-09-20T22:59:45.949062+0000 front 2024-09-20T22:59:45.949180+0000 (oldest deadline 2024-09-20T23:00:09.448861+0000) 2024-09-20T23:00:26.255 INFO:journalctl@ceph.osd.6.smithi154.stdout:Sep 20 23:00:25 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-6[49255]: 2024-09-20T23:00:25.986+0000 7f2ff16a5640 -1 osd.6 1036 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-09-20T22:59:51.650246+0000 front 2024-09-20T22:59:51.650286+0000 (oldest deadline 2024-09-20T23:00:16.950204+0000) 2024-09-20T23:00:26.255 INFO:journalctl@ceph.osd.6.smithi154.stdout:Sep 20 23:00:25 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-6[49255]: 2024-09-20T23:00:25.986+0000 7f2ff16a5640 -1 osd.6 1036 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-09-20T22:59:59.250995+0000 front 2024-09-20T22:59:59.251043+0000 (oldest deadline 2024-09-20T23:00:24.550992+0000) 2024-09-20T23:00:26.972 INFO:journalctl@ceph.osd.5.smithi154.stdout:Sep 20 23:00:26 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-5[43596]: 2024-09-20T23:00:26.652+0000 7fb11f7ec640 -1 osd.5 1036 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-09-20T22:59:45.278275+0000 front 2024-09-20T22:59:45.278360+0000 (oldest deadline 2024-09-20T23:00:11.678666+0000) 2024-09-20T23:00:26.973 INFO:journalctl@ceph.osd.5.smithi154.stdout:Sep 20 23:00:26 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-5[43596]: 2024-09-20T23:00:26.652+0000 7fb11f7ec640 -1 osd.5 1036 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-09-20T22:59:53.979697+0000 front 2024-09-20T22:59:53.979548+0000 (oldest deadline 2024-09-20T23:00:15.679666+0000) 2024-09-20T23:00:26.973 INFO:journalctl@ceph.osd.5.smithi154.stdout:Sep 20 23:00:26 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-5[43596]: 2024-09-20T23:00:26.652+0000 7fb11f7ec640 -1 osd.5 1036 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-09-20T23:00:01.381589+0000 front 2024-09-20T23:00:01.381676+0000 (oldest deadline 2024-09-20T23:00:23.681563+0000) 2024-09-20T23:00:26.973 INFO:journalctl@ceph.osd.7.smithi154.stdout:Sep 20 23:00:26 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-7[54754]: 2024-09-20T23:00:26.640+0000 7fee5b5be640 -1 osd.7 1036 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-09-20T22:59:47.071784+0000 front 2024-09-20T22:59:47.071677+0000 (oldest deadline 2024-09-20T23:00:11.171776+0000) 2024-09-20T23:00:26.973 INFO:journalctl@ceph.osd.7.smithi154.stdout:Sep 20 23:00:26 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-7[54754]: 2024-09-20T23:00:26.640+0000 7fee5b5be640 -1 osd.7 1036 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-09-20T22:59:51.172472+0000 front 2024-09-20T22:59:51.172371+0000 (oldest deadline 2024-09-20T23:00:16.472189+0000) 2024-09-20T23:00:26.973 INFO:journalctl@ceph.osd.7.smithi154.stdout:Sep 20 23:00:26 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-7[54754]: 2024-09-20T23:00:26.640+0000 7fee5b5be640 -1 osd.7 1036 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-09-20T23:00:00.573109+0000 front 2024-09-20T23:00:00.573085+0000 (oldest deadline 2024-09-20T23:00:24.072955+0000) 2024-09-20T23:00:27.254 INFO:journalctl@ceph.osd.5.smithi154.stdout:Sep 20 23:00:27 smithi154 podman[153394]: 2024-09-20 23:00:27.128008409 +0000 UTC m=+5.551360479 container died 42c1f261969b6b5c63c7a10ea772bce9a0409965e9486202389caa6f409c32ef (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-5, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, RELEASE=squid-9e2ac70, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-09-20T23:00:27.255 INFO:journalctl@ceph.osd.6.smithi154.stdout:Sep 20 23:00:26 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-6[49255]: 2024-09-20T23:00:26.970+0000 7f2ff16a5640 -1 osd.6 1036 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-09-20T22:59:45.949062+0000 front 2024-09-20T22:59:45.949180+0000 (oldest deadline 2024-09-20T23:00:09.448861+0000) 2024-09-20T23:00:27.255 INFO:journalctl@ceph.osd.6.smithi154.stdout:Sep 20 23:00:26 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-6[49255]: 2024-09-20T23:00:26.970+0000 7f2ff16a5640 -1 osd.6 1036 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-09-20T22:59:51.650246+0000 front 2024-09-20T22:59:51.650286+0000 (oldest deadline 2024-09-20T23:00:16.950204+0000) 2024-09-20T23:00:27.255 INFO:journalctl@ceph.osd.6.smithi154.stdout:Sep 20 23:00:26 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-6[49255]: 2024-09-20T23:00:26.970+0000 7f2ff16a5640 -1 osd.6 1036 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-09-20T22:59:59.250995+0000 front 2024-09-20T22:59:59.251043+0000 (oldest deadline 2024-09-20T23:00:24.550992+0000) 2024-09-20T23:00:27.652 INFO:journalctl@ceph.osd.5.smithi154.stdout:Sep 20 23:00:27 smithi154 podman[153394]: 2024-09-20 23:00:27.471863948 +0000 UTC m=+5.895215995 container cleanup 42c1f261969b6b5c63c7a10ea772bce9a0409965e9486202389caa6f409c32ef (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-5, RELEASE=squid-9e2ac70, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2) 2024-09-20T23:00:27.653 INFO:journalctl@ceph.osd.5.smithi154.stdout:Sep 20 23:00:27 smithi154 bash[153394]: ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-5 2024-09-20T23:00:27.653 INFO:journalctl@ceph.osd.5.smithi154.stdout:Sep 20 23:00:27 smithi154 podman[153421]: 2024-09-20 23:00:27.623449714 +0000 UTC m=+0.488650891 container remove 42c1f261969b6b5c63c7a10ea772bce9a0409965e9486202389caa6f409c32ef (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-9e2ac70, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=) 2024-09-20T23:00:27.925 INFO:journalctl@ceph.osd.7.smithi154.stdout:Sep 20 23:00:27 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-7[54754]: 2024-09-20T23:00:27.650+0000 7fee5b5be640 -1 osd.7 1036 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-09-20T22:59:47.071784+0000 front 2024-09-20T22:59:47.071677+0000 (oldest deadline 2024-09-20T23:00:11.171776+0000) 2024-09-20T23:00:27.925 INFO:journalctl@ceph.osd.7.smithi154.stdout:Sep 20 23:00:27 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-7[54754]: 2024-09-20T23:00:27.650+0000 7fee5b5be640 -1 osd.7 1036 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-09-20T22:59:51.172472+0000 front 2024-09-20T22:59:51.172371+0000 (oldest deadline 2024-09-20T23:00:16.472189+0000) 2024-09-20T23:00:27.925 INFO:journalctl@ceph.osd.7.smithi154.stdout:Sep 20 23:00:27 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-7[54754]: 2024-09-20T23:00:27.650+0000 7fee5b5be640 -1 osd.7 1036 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-09-20T23:00:00.573109+0000 front 2024-09-20T23:00:00.573085+0000 (oldest deadline 2024-09-20T23:00:24.072955+0000) 2024-09-20T23:00:28.183 INFO:journalctl@ceph.osd.5.smithi154.stdout:Sep 20 23:00:28 smithi154 podman[153511]: 2024-09-20 23:00:27.943794736 +0000 UTC m=+0.016422258 image pull cc9251b744a3faafca7c3095f79bfff81873afe823f31a03a11e015a34abcdbd quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5 2024-09-20T23:00:28.183 INFO:journalctl@ceph.osd.5.smithi154.stdout:Sep 20 23:00:28 smithi154 podman[153511]: 2024-09-20 23:00:28.049101424 +0000 UTC m=+0.121728937 container create dfd78da4bda19fa7544ffbf109b8eb854526d2cbdfe0bcaf08c8a7bc84a77f74 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-5-deactivate, maintainer=Guillaume Abrioux , RELEASE=squid-9e2ac70, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-20T23:00:28.183 INFO:journalctl@ceph.osd.6.smithi154.stdout:Sep 20 23:00:27 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-6[49255]: 2024-09-20T23:00:27.923+0000 7f2ff16a5640 -1 osd.6 1036 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-09-20T22:59:45.949062+0000 front 2024-09-20T22:59:45.949180+0000 (oldest deadline 2024-09-20T23:00:09.448861+0000) 2024-09-20T23:00:28.183 INFO:journalctl@ceph.osd.6.smithi154.stdout:Sep 20 23:00:27 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-6[49255]: 2024-09-20T23:00:27.923+0000 7f2ff16a5640 -1 osd.6 1036 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-09-20T22:59:51.650246+0000 front 2024-09-20T22:59:51.650286+0000 (oldest deadline 2024-09-20T23:00:16.950204+0000) 2024-09-20T23:00:28.183 INFO:journalctl@ceph.osd.6.smithi154.stdout:Sep 20 23:00:27 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-6[49255]: 2024-09-20T23:00:27.923+0000 7f2ff16a5640 -1 osd.6 1036 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-09-20T22:59:59.250995+0000 front 2024-09-20T22:59:59.251043+0000 (oldest deadline 2024-09-20T23:00:24.550992+0000) 2024-09-20T23:00:28.632 INFO:journalctl@ceph.osd.5.smithi154.stdout:Sep 20 23:00:28 smithi154 podman[153511]: 2024-09-20 23:00:28.285081051 +0000 UTC m=+0.357708560 container init dfd78da4bda19fa7544ffbf109b8eb854526d2cbdfe0bcaf08c8a7bc84a77f74 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-5-deactivate, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240731, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-9e2ac70, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-20T23:00:28.632 INFO:journalctl@ceph.osd.5.smithi154.stdout:Sep 20 23:00:28 smithi154 podman[153511]: 2024-09-20 23:00:28.28830451 +0000 UTC m=+0.360932022 container start dfd78da4bda19fa7544ffbf109b8eb854526d2cbdfe0bcaf08c8a7bc84a77f74 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-5-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, RELEASE=squid-9e2ac70, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-20T23:00:28.632 INFO:journalctl@ceph.osd.5.smithi154.stdout:Sep 20 23:00:28 smithi154 podman[153511]: 2024-09-20 23:00:28.351604945 +0000 UTC m=+0.424232476 container attach dfd78da4bda19fa7544ffbf109b8eb854526d2cbdfe0bcaf08c8a7bc84a77f74 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-5-deactivate, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, RELEASE=squid-9e2ac70, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-09-20T23:00:28.632 INFO:journalctl@ceph.osd.5.smithi154.stdout:Sep 20 23:00:28 smithi154 podman[153511]: 2024-09-20 23:00:28.479344619 +0000 UTC m=+0.551972140 container died dfd78da4bda19fa7544ffbf109b8eb854526d2cbdfe0bcaf08c8a7bc84a77f74 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-5-deactivate, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-9e2ac70, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS) 2024-09-20T23:00:28.930 INFO:journalctl@ceph.osd.7.smithi154.stdout:Sep 20 23:00:28 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-7[54754]: 2024-09-20T23:00:28.629+0000 7fee5b5be640 -1 osd.7 1036 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-09-20T22:59:47.071784+0000 front 2024-09-20T22:59:47.071677+0000 (oldest deadline 2024-09-20T23:00:11.171776+0000) 2024-09-20T23:00:28.930 INFO:journalctl@ceph.osd.7.smithi154.stdout:Sep 20 23:00:28 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-7[54754]: 2024-09-20T23:00:28.629+0000 7fee5b5be640 -1 osd.7 1036 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-09-20T22:59:51.172472+0000 front 2024-09-20T22:59:51.172371+0000 (oldest deadline 2024-09-20T23:00:16.472189+0000) 2024-09-20T23:00:28.930 INFO:journalctl@ceph.osd.7.smithi154.stdout:Sep 20 23:00:28 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-7[54754]: 2024-09-20T23:00:28.629+0000 7fee5b5be640 -1 osd.7 1036 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-09-20T23:00:00.573109+0000 front 2024-09-20T23:00:00.573085+0000 (oldest deadline 2024-09-20T23:00:24.072955+0000) 2024-09-20T23:00:29.054 DEBUG:teuthology.orchestra.run.smithi154:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490@osd.5.service' 2024-09-20T23:00:29.093 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-20T23:00:29.093 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-09-20T23:00:29.093 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-09-20T23:00:29.094 DEBUG:teuthology.orchestra.run.smithi154:> sudo systemctl stop ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490@osd.6 2024-09-20T23:00:29.254 INFO:journalctl@ceph.osd.6.smithi154.stdout:Sep 20 23:00:28 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-6[49255]: 2024-09-20T23:00:28.928+0000 7f2ff16a5640 -1 osd.6 1036 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-09-20T22:59:45.949062+0000 front 2024-09-20T22:59:45.949180+0000 (oldest deadline 2024-09-20T23:00:09.448861+0000) 2024-09-20T23:00:29.254 INFO:journalctl@ceph.osd.6.smithi154.stdout:Sep 20 23:00:28 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-6[49255]: 2024-09-20T23:00:28.928+0000 7f2ff16a5640 -1 osd.6 1036 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-09-20T22:59:51.650246+0000 front 2024-09-20T22:59:51.650286+0000 (oldest deadline 2024-09-20T23:00:16.950204+0000) 2024-09-20T23:00:29.254 INFO:journalctl@ceph.osd.6.smithi154.stdout:Sep 20 23:00:28 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-6[49255]: 2024-09-20T23:00:28.928+0000 7f2ff16a5640 -1 osd.6 1036 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-09-20T22:59:59.250995+0000 front 2024-09-20T22:59:59.251043+0000 (oldest deadline 2024-09-20T23:00:24.550992+0000) 2024-09-20T23:00:29.254 INFO:journalctl@ceph.osd.6.smithi154.stdout:Sep 20 23:00:29 smithi154 systemd[1]: Stopping Ceph osd.6 for a9c0bc70-77a0-11ef-baf5-efdc52797490... 2024-09-20T23:00:29.754 INFO:journalctl@ceph.osd.7.smithi154.stdout:Sep 20 23:00:29 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-7[54754]: 2024-09-20T23:00:29.607+0000 7fee5b5be640 -1 osd.7 1036 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-09-20T22:59:47.071784+0000 front 2024-09-20T22:59:47.071677+0000 (oldest deadline 2024-09-20T23:00:11.171776+0000) 2024-09-20T23:00:29.754 INFO:journalctl@ceph.osd.7.smithi154.stdout:Sep 20 23:00:29 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-7[54754]: 2024-09-20T23:00:29.607+0000 7fee5b5be640 -1 osd.7 1036 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-09-20T22:59:51.172472+0000 front 2024-09-20T22:59:51.172371+0000 (oldest deadline 2024-09-20T23:00:16.472189+0000) 2024-09-20T23:00:29.754 INFO:journalctl@ceph.osd.7.smithi154.stdout:Sep 20 23:00:29 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-7[54754]: 2024-09-20T23:00:29.607+0000 7fee5b5be640 -1 osd.7 1036 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-09-20T23:00:00.573109+0000 front 2024-09-20T23:00:00.573085+0000 (oldest deadline 2024-09-20T23:00:24.072955+0000) 2024-09-20T23:00:29.755 INFO:journalctl@ceph.osd.6.smithi154.stdout:Sep 20 23:00:29 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-6[49255]: 2024-09-20T23:00:29.424+0000 7f2ff9097640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-20T23:00:29.755 INFO:journalctl@ceph.osd.6.smithi154.stdout:Sep 20 23:00:29 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-6[49255]: 2024-09-20T23:00:29.424+0000 7f2ff9097640 -1 osd.6 1036 *** Got signal Terminated *** 2024-09-20T23:00:29.755 INFO:journalctl@ceph.osd.6.smithi154.stdout:Sep 20 23:00:29 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-6[49255]: 2024-09-20T23:00:29.424+0000 7f2ff9097640 -1 osd.6 1036 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-20T23:00:30.254 INFO:journalctl@ceph.osd.6.smithi154.stdout:Sep 20 23:00:29 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-6[49255]: 2024-09-20T23:00:29.938+0000 7f2ff16a5640 -1 osd.6 1036 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-09-20T22:59:45.949062+0000 front 2024-09-20T22:59:45.949180+0000 (oldest deadline 2024-09-20T23:00:09.448861+0000) 2024-09-20T23:00:30.254 INFO:journalctl@ceph.osd.6.smithi154.stdout:Sep 20 23:00:29 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-6[49255]: 2024-09-20T23:00:29.938+0000 7f2ff16a5640 -1 osd.6 1036 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-09-20T22:59:51.650246+0000 front 2024-09-20T22:59:51.650286+0000 (oldest deadline 2024-09-20T23:00:16.950204+0000) 2024-09-20T23:00:30.254 INFO:journalctl@ceph.osd.6.smithi154.stdout:Sep 20 23:00:29 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-6[49255]: 2024-09-20T23:00:29.938+0000 7f2ff16a5640 -1 osd.6 1036 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-09-20T22:59:59.250995+0000 front 2024-09-20T22:59:59.251043+0000 (oldest deadline 2024-09-20T23:00:24.550992+0000) 2024-09-20T23:00:30.974 INFO:journalctl@ceph.osd.7.smithi154.stdout:Sep 20 23:00:30 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-7[54754]: 2024-09-20T23:00:30.625+0000 7fee5b5be640 -1 osd.7 1036 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-09-20T22:59:47.071784+0000 front 2024-09-20T22:59:47.071677+0000 (oldest deadline 2024-09-20T23:00:11.171776+0000) 2024-09-20T23:00:30.974 INFO:journalctl@ceph.osd.7.smithi154.stdout:Sep 20 23:00:30 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-7[54754]: 2024-09-20T23:00:30.625+0000 7fee5b5be640 -1 osd.7 1036 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-09-20T22:59:51.172472+0000 front 2024-09-20T22:59:51.172371+0000 (oldest deadline 2024-09-20T23:00:16.472189+0000) 2024-09-20T23:00:30.975 INFO:journalctl@ceph.osd.7.smithi154.stdout:Sep 20 23:00:30 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-7[54754]: 2024-09-20T23:00:30.625+0000 7fee5b5be640 -1 osd.7 1036 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-09-20T23:00:00.573109+0000 front 2024-09-20T23:00:00.573085+0000 (oldest deadline 2024-09-20T23:00:24.072955+0000) 2024-09-20T23:00:31.254 INFO:journalctl@ceph.osd.6.smithi154.stdout:Sep 20 23:00:30 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-6[49255]: 2024-09-20T23:00:30.973+0000 7f2ff16a5640 -1 osd.6 1036 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-09-20T22:59:45.949062+0000 front 2024-09-20T22:59:45.949180+0000 (oldest deadline 2024-09-20T23:00:09.448861+0000) 2024-09-20T23:00:31.254 INFO:journalctl@ceph.osd.6.smithi154.stdout:Sep 20 23:00:30 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-6[49255]: 2024-09-20T23:00:30.973+0000 7f2ff16a5640 -1 osd.6 1036 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-09-20T22:59:51.650246+0000 front 2024-09-20T22:59:51.650286+0000 (oldest deadline 2024-09-20T23:00:16.950204+0000) 2024-09-20T23:00:31.254 INFO:journalctl@ceph.osd.6.smithi154.stdout:Sep 20 23:00:30 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-6[49255]: 2024-09-20T23:00:30.973+0000 7f2ff16a5640 -1 osd.6 1036 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-09-20T22:59:59.250995+0000 front 2024-09-20T22:59:59.251043+0000 (oldest deadline 2024-09-20T23:00:24.550992+0000) 2024-09-20T23:00:31.927 INFO:journalctl@ceph.osd.7.smithi154.stdout:Sep 20 23:00:31 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-7[54754]: 2024-09-20T23:00:31.673+0000 7fee5b5be640 -1 osd.7 1036 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-09-20T22:59:47.071784+0000 front 2024-09-20T22:59:47.071677+0000 (oldest deadline 2024-09-20T23:00:11.171776+0000) 2024-09-20T23:00:31.927 INFO:journalctl@ceph.osd.7.smithi154.stdout:Sep 20 23:00:31 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-7[54754]: 2024-09-20T23:00:31.673+0000 7fee5b5be640 -1 osd.7 1036 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-09-20T22:59:51.172472+0000 front 2024-09-20T22:59:51.172371+0000 (oldest deadline 2024-09-20T23:00:16.472189+0000) 2024-09-20T23:00:31.927 INFO:journalctl@ceph.osd.7.smithi154.stdout:Sep 20 23:00:31 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-7[54754]: 2024-09-20T23:00:31.673+0000 7fee5b5be640 -1 osd.7 1036 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-09-20T23:00:00.573109+0000 front 2024-09-20T23:00:00.573085+0000 (oldest deadline 2024-09-20T23:00:24.072955+0000) 2024-09-20T23:00:31.927 INFO:journalctl@ceph.osd.7.smithi154.stdout:Sep 20 23:00:31 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-7[54754]: 2024-09-20T23:00:31.673+0000 7fee5b5be640 -1 osd.7 1036 heartbeat_check: no reply from 172.21.15.84:6815 osd.3 since back 2024-09-20T23:00:08.173714+0000 front 2024-09-20T23:00:08.173807+0000 (oldest deadline 2024-09-20T23:00:31.574003+0000) 2024-09-20T23:00:32.254 INFO:journalctl@ceph.osd.6.smithi154.stdout:Sep 20 23:00:31 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-6[49255]: 2024-09-20T23:00:31.925+0000 7f2ff16a5640 -1 osd.6 1036 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-09-20T22:59:45.949062+0000 front 2024-09-20T22:59:45.949180+0000 (oldest deadline 2024-09-20T23:00:09.448861+0000) 2024-09-20T23:00:32.254 INFO:journalctl@ceph.osd.6.smithi154.stdout:Sep 20 23:00:31 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-6[49255]: 2024-09-20T23:00:31.925+0000 7f2ff16a5640 -1 osd.6 1036 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-09-20T22:59:51.650246+0000 front 2024-09-20T22:59:51.650286+0000 (oldest deadline 2024-09-20T23:00:16.950204+0000) 2024-09-20T23:00:32.255 INFO:journalctl@ceph.osd.6.smithi154.stdout:Sep 20 23:00:31 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-6[49255]: 2024-09-20T23:00:31.925+0000 7f2ff16a5640 -1 osd.6 1036 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-09-20T22:59:59.250995+0000 front 2024-09-20T22:59:59.251043+0000 (oldest deadline 2024-09-20T23:00:24.550992+0000) 2024-09-20T23:00:32.926 INFO:journalctl@ceph.osd.7.smithi154.stdout:Sep 20 23:00:32 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-7[54754]: 2024-09-20T23:00:32.653+0000 7fee5b5be640 -1 osd.7 1036 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-09-20T22:59:47.071784+0000 front 2024-09-20T22:59:47.071677+0000 (oldest deadline 2024-09-20T23:00:11.171776+0000) 2024-09-20T23:00:32.926 INFO:journalctl@ceph.osd.7.smithi154.stdout:Sep 20 23:00:32 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-7[54754]: 2024-09-20T23:00:32.653+0000 7fee5b5be640 -1 osd.7 1036 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-09-20T22:59:51.172472+0000 front 2024-09-20T22:59:51.172371+0000 (oldest deadline 2024-09-20T23:00:16.472189+0000) 2024-09-20T23:00:32.926 INFO:journalctl@ceph.osd.7.smithi154.stdout:Sep 20 23:00:32 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-7[54754]: 2024-09-20T23:00:32.653+0000 7fee5b5be640 -1 osd.7 1036 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-09-20T23:00:00.573109+0000 front 2024-09-20T23:00:00.573085+0000 (oldest deadline 2024-09-20T23:00:24.072955+0000) 2024-09-20T23:00:32.926 INFO:journalctl@ceph.osd.7.smithi154.stdout:Sep 20 23:00:32 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-7[54754]: 2024-09-20T23:00:32.653+0000 7fee5b5be640 -1 osd.7 1036 heartbeat_check: no reply from 172.21.15.84:6815 osd.3 since back 2024-09-20T23:00:08.173714+0000 front 2024-09-20T23:00:08.173807+0000 (oldest deadline 2024-09-20T23:00:31.574003+0000) 2024-09-20T23:00:33.254 INFO:journalctl@ceph.osd.6.smithi154.stdout:Sep 20 23:00:32 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-6[49255]: 2024-09-20T23:00:32.924+0000 7f2ff16a5640 -1 osd.6 1036 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-09-20T22:59:45.949062+0000 front 2024-09-20T22:59:45.949180+0000 (oldest deadline 2024-09-20T23:00:09.448861+0000) 2024-09-20T23:00:33.254 INFO:journalctl@ceph.osd.6.smithi154.stdout:Sep 20 23:00:32 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-6[49255]: 2024-09-20T23:00:32.924+0000 7f2ff16a5640 -1 osd.6 1036 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-09-20T22:59:51.650246+0000 front 2024-09-20T22:59:51.650286+0000 (oldest deadline 2024-09-20T23:00:16.950204+0000) 2024-09-20T23:00:33.254 INFO:journalctl@ceph.osd.6.smithi154.stdout:Sep 20 23:00:32 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-6[49255]: 2024-09-20T23:00:32.924+0000 7f2ff16a5640 -1 osd.6 1036 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-09-20T22:59:59.250995+0000 front 2024-09-20T22:59:59.251043+0000 (oldest deadline 2024-09-20T23:00:24.550992+0000) 2024-09-20T23:00:33.926 INFO:journalctl@ceph.osd.7.smithi154.stdout:Sep 20 23:00:33 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-7[54754]: 2024-09-20T23:00:33.651+0000 7fee5b5be640 -1 osd.7 1036 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-09-20T22:59:47.071784+0000 front 2024-09-20T22:59:47.071677+0000 (oldest deadline 2024-09-20T23:00:11.171776+0000) 2024-09-20T23:00:33.926 INFO:journalctl@ceph.osd.7.smithi154.stdout:Sep 20 23:00:33 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-7[54754]: 2024-09-20T23:00:33.651+0000 7fee5b5be640 -1 osd.7 1036 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-09-20T22:59:51.172472+0000 front 2024-09-20T22:59:51.172371+0000 (oldest deadline 2024-09-20T23:00:16.472189+0000) 2024-09-20T23:00:33.926 INFO:journalctl@ceph.osd.7.smithi154.stdout:Sep 20 23:00:33 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-7[54754]: 2024-09-20T23:00:33.651+0000 7fee5b5be640 -1 osd.7 1036 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-09-20T23:00:00.573109+0000 front 2024-09-20T23:00:00.573085+0000 (oldest deadline 2024-09-20T23:00:24.072955+0000) 2024-09-20T23:00:33.926 INFO:journalctl@ceph.osd.7.smithi154.stdout:Sep 20 23:00:33 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-7[54754]: 2024-09-20T23:00:33.651+0000 7fee5b5be640 -1 osd.7 1036 heartbeat_check: no reply from 172.21.15.84:6815 osd.3 since back 2024-09-20T23:00:08.173714+0000 front 2024-09-20T23:00:08.173807+0000 (oldest deadline 2024-09-20T23:00:31.574003+0000) 2024-09-20T23:00:34.254 INFO:journalctl@ceph.osd.6.smithi154.stdout:Sep 20 23:00:33 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-6[49255]: 2024-09-20T23:00:33.925+0000 7f2ff16a5640 -1 osd.6 1036 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-09-20T22:59:45.949062+0000 front 2024-09-20T22:59:45.949180+0000 (oldest deadline 2024-09-20T23:00:09.448861+0000) 2024-09-20T23:00:34.254 INFO:journalctl@ceph.osd.6.smithi154.stdout:Sep 20 23:00:33 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-6[49255]: 2024-09-20T23:00:33.925+0000 7f2ff16a5640 -1 osd.6 1036 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-09-20T22:59:51.650246+0000 front 2024-09-20T22:59:51.650286+0000 (oldest deadline 2024-09-20T23:00:16.950204+0000) 2024-09-20T23:00:34.254 INFO:journalctl@ceph.osd.6.smithi154.stdout:Sep 20 23:00:33 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-6[49255]: 2024-09-20T23:00:33.925+0000 7f2ff16a5640 -1 osd.6 1036 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-09-20T22:59:59.250995+0000 front 2024-09-20T22:59:59.251043+0000 (oldest deadline 2024-09-20T23:00:24.550992+0000) 2024-09-20T23:00:35.004 INFO:journalctl@ceph.osd.6.smithi154.stdout:Sep 20 23:00:34 smithi154 podman[153615]: 2024-09-20 23:00:34.922707143 +0000 UTC m=+5.611398160 container died 5d7522faa6d2d46bc67708914e937344ac237767695d16aab604326326a8adcd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-6, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=squid-9e2ac70) 2024-09-20T23:00:35.004 INFO:journalctl@ceph.osd.7.smithi154.stdout:Sep 20 23:00:34 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-7[54754]: 2024-09-20T23:00:34.675+0000 7fee5b5be640 -1 osd.7 1036 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-09-20T22:59:47.071784+0000 front 2024-09-20T22:59:47.071677+0000 (oldest deadline 2024-09-20T23:00:11.171776+0000) 2024-09-20T23:00:35.005 INFO:journalctl@ceph.osd.7.smithi154.stdout:Sep 20 23:00:34 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-7[54754]: 2024-09-20T23:00:34.675+0000 7fee5b5be640 -1 osd.7 1036 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-09-20T22:59:51.172472+0000 front 2024-09-20T22:59:51.172371+0000 (oldest deadline 2024-09-20T23:00:16.472189+0000) 2024-09-20T23:00:35.005 INFO:journalctl@ceph.osd.7.smithi154.stdout:Sep 20 23:00:34 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-7[54754]: 2024-09-20T23:00:34.675+0000 7fee5b5be640 -1 osd.7 1036 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-09-20T23:00:00.573109+0000 front 2024-09-20T23:00:00.573085+0000 (oldest deadline 2024-09-20T23:00:24.072955+0000) 2024-09-20T23:00:35.005 INFO:journalctl@ceph.osd.7.smithi154.stdout:Sep 20 23:00:34 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-7[54754]: 2024-09-20T23:00:34.675+0000 7fee5b5be640 -1 osd.7 1036 heartbeat_check: no reply from 172.21.15.84:6815 osd.3 since back 2024-09-20T23:00:08.173714+0000 front 2024-09-20T23:00:08.173807+0000 (oldest deadline 2024-09-20T23:00:31.574003+0000) 2024-09-20T23:00:35.477 INFO:journalctl@ceph.osd.6.smithi154.stdout:Sep 20 23:00:35 smithi154 podman[153615]: 2024-09-20 23:00:35.307969795 +0000 UTC m=+5.996660813 container cleanup 5d7522faa6d2d46bc67708914e937344ac237767695d16aab604326326a8adcd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-6, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-9e2ac70, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-09-20T23:00:35.477 INFO:journalctl@ceph.osd.6.smithi154.stdout:Sep 20 23:00:35 smithi154 bash[153615]: ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-6 2024-09-20T23:00:35.754 INFO:journalctl@ceph.osd.6.smithi154.stdout:Sep 20 23:00:35 smithi154 podman[153633]: 2024-09-20 23:00:35.475779876 +0000 UTC m=+0.543021647 container remove 5d7522faa6d2d46bc67708914e937344ac237767695d16aab604326326a8adcd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-6, GIT_BRANCH=HEAD, RELEASE=squid-9e2ac70, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-09-20T23:00:35.754 INFO:journalctl@ceph.osd.7.smithi154.stdout:Sep 20 23:00:35 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-7[54754]: 2024-09-20T23:00:35.629+0000 7fee5b5be640 -1 osd.7 1036 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-09-20T22:59:47.071784+0000 front 2024-09-20T22:59:47.071677+0000 (oldest deadline 2024-09-20T23:00:11.171776+0000) 2024-09-20T23:00:35.754 INFO:journalctl@ceph.osd.7.smithi154.stdout:Sep 20 23:00:35 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-7[54754]: 2024-09-20T23:00:35.629+0000 7fee5b5be640 -1 osd.7 1036 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-09-20T22:59:51.172472+0000 front 2024-09-20T22:59:51.172371+0000 (oldest deadline 2024-09-20T23:00:16.472189+0000) 2024-09-20T23:00:35.755 INFO:journalctl@ceph.osd.7.smithi154.stdout:Sep 20 23:00:35 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-7[54754]: 2024-09-20T23:00:35.629+0000 7fee5b5be640 -1 osd.7 1036 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-09-20T23:00:00.573109+0000 front 2024-09-20T23:00:00.573085+0000 (oldest deadline 2024-09-20T23:00:24.072955+0000) 2024-09-20T23:00:35.755 INFO:journalctl@ceph.osd.7.smithi154.stdout:Sep 20 23:00:35 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-7[54754]: 2024-09-20T23:00:35.629+0000 7fee5b5be640 -1 osd.7 1036 heartbeat_check: no reply from 172.21.15.84:6815 osd.3 since back 2024-09-20T23:00:08.173714+0000 front 2024-09-20T23:00:08.173807+0000 (oldest deadline 2024-09-20T23:00:31.574003+0000) 2024-09-20T23:00:36.186 INFO:journalctl@ceph.osd.6.smithi154.stdout:Sep 20 23:00:35 smithi154 podman[153719]: 2024-09-20 23:00:35.787737371 +0000 UTC m=+0.016271862 image pull cc9251b744a3faafca7c3095f79bfff81873afe823f31a03a11e015a34abcdbd quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5 2024-09-20T23:00:36.186 INFO:journalctl@ceph.osd.6.smithi154.stdout:Sep 20 23:00:35 smithi154 podman[153719]: 2024-09-20 23:00:35.885021482 +0000 UTC m=+0.113556019 container create 35ab3d75cd34db7d49f8725a58afbaf7b3466fe226bb7f9c9620eba8217d8aef (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-6-deactivate, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=squid-9e2ac70, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=) 2024-09-20T23:00:36.186 INFO:journalctl@ceph.osd.6.smithi154.stdout:Sep 20 23:00:36 smithi154 podman[153719]: 2024-09-20 23:00:36.079493091 +0000 UTC m=+0.308027592 container init 35ab3d75cd34db7d49f8725a58afbaf7b3466fe226bb7f9c9620eba8217d8aef (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-6-deactivate, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, RELEASE=squid-9e2ac70, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-09-20T23:00:36.186 INFO:journalctl@ceph.osd.6.smithi154.stdout:Sep 20 23:00:36 smithi154 podman[153719]: 2024-09-20 23:00:36.083148083 +0000 UTC m=+0.311682576 container start 35ab3d75cd34db7d49f8725a58afbaf7b3466fe226bb7f9c9620eba8217d8aef (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-6-deactivate, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , RELEASE=squid-9e2ac70, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-20T23:00:36.187 INFO:journalctl@ceph.osd.6.smithi154.stdout:Sep 20 23:00:36 smithi154 podman[153719]: 2024-09-20 23:00:36.184689402 +0000 UTC m=+0.413223905 container attach 35ab3d75cd34db7d49f8725a58afbaf7b3466fe226bb7f9c9620eba8217d8aef (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-6-deactivate, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=squid-9e2ac70, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2) 2024-09-20T23:00:36.504 INFO:journalctl@ceph.osd.6.smithi154.stdout:Sep 20 23:00:36 smithi154 podman[153719]: 2024-09-20 23:00:36.27464082 +0000 UTC m=+0.503175327 container died 35ab3d75cd34db7d49f8725a58afbaf7b3466fe226bb7f9c9620eba8217d8aef (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-6-deactivate, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=squid-9e2ac70) 2024-09-20T23:00:36.801 INFO:journalctl@ceph.osd.7.smithi154.stdout:Sep 20 23:00:36 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-7[54754]: 2024-09-20T23:00:36.581+0000 7fee5b5be640 -1 osd.7 1036 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-09-20T22:59:47.071784+0000 front 2024-09-20T22:59:47.071677+0000 (oldest deadline 2024-09-20T23:00:11.171776+0000) 2024-09-20T23:00:36.801 INFO:journalctl@ceph.osd.7.smithi154.stdout:Sep 20 23:00:36 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-7[54754]: 2024-09-20T23:00:36.581+0000 7fee5b5be640 -1 osd.7 1036 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-09-20T22:59:51.172472+0000 front 2024-09-20T22:59:51.172371+0000 (oldest deadline 2024-09-20T23:00:16.472189+0000) 2024-09-20T23:00:36.801 INFO:journalctl@ceph.osd.7.smithi154.stdout:Sep 20 23:00:36 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-7[54754]: 2024-09-20T23:00:36.581+0000 7fee5b5be640 -1 osd.7 1036 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-09-20T23:00:00.573109+0000 front 2024-09-20T23:00:00.573085+0000 (oldest deadline 2024-09-20T23:00:24.072955+0000) 2024-09-20T23:00:36.801 INFO:journalctl@ceph.osd.7.smithi154.stdout:Sep 20 23:00:36 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-7[54754]: 2024-09-20T23:00:36.581+0000 7fee5b5be640 -1 osd.7 1036 heartbeat_check: no reply from 172.21.15.84:6815 osd.3 since back 2024-09-20T23:00:08.173714+0000 front 2024-09-20T23:00:08.173807+0000 (oldest deadline 2024-09-20T23:00:31.574003+0000) 2024-09-20T23:00:36.825 DEBUG:teuthology.orchestra.run.smithi154:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490@osd.6.service' 2024-09-20T23:00:36.862 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-20T23:00:36.862 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-09-20T23:00:36.863 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-09-20T23:00:36.863 DEBUG:teuthology.orchestra.run.smithi154:> sudo systemctl stop ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490@osd.7 2024-09-20T23:00:37.196 INFO:journalctl@ceph.osd.7.smithi154.stdout:Sep 20 23:00:36 smithi154 systemd[1]: Stopping Ceph osd.7 for a9c0bc70-77a0-11ef-baf5-efdc52797490... 2024-09-20T23:00:37.504 INFO:journalctl@ceph.osd.7.smithi154.stdout:Sep 20 23:00:37 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-7[54754]: 2024-09-20T23:00:37.194+0000 7fee62fb0640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-20T23:00:37.504 INFO:journalctl@ceph.osd.7.smithi154.stdout:Sep 20 23:00:37 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-7[54754]: 2024-09-20T23:00:37.194+0000 7fee62fb0640 -1 osd.7 1036 *** Got signal Terminated *** 2024-09-20T23:00:37.504 INFO:journalctl@ceph.osd.7.smithi154.stdout:Sep 20 23:00:37 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-7[54754]: 2024-09-20T23:00:37.194+0000 7fee62fb0640 -1 osd.7 1036 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-20T23:00:38.004 INFO:journalctl@ceph.osd.7.smithi154.stdout:Sep 20 23:00:37 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-7[54754]: 2024-09-20T23:00:37.578+0000 7fee5b5be640 -1 osd.7 1036 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-09-20T22:59:47.071784+0000 front 2024-09-20T22:59:47.071677+0000 (oldest deadline 2024-09-20T23:00:11.171776+0000) 2024-09-20T23:00:38.005 INFO:journalctl@ceph.osd.7.smithi154.stdout:Sep 20 23:00:37 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-7[54754]: 2024-09-20T23:00:37.578+0000 7fee5b5be640 -1 osd.7 1036 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-09-20T22:59:51.172472+0000 front 2024-09-20T22:59:51.172371+0000 (oldest deadline 2024-09-20T23:00:16.472189+0000) 2024-09-20T23:00:38.005 INFO:journalctl@ceph.osd.7.smithi154.stdout:Sep 20 23:00:37 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-7[54754]: 2024-09-20T23:00:37.578+0000 7fee5b5be640 -1 osd.7 1036 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-09-20T23:00:00.573109+0000 front 2024-09-20T23:00:00.573085+0000 (oldest deadline 2024-09-20T23:00:24.072955+0000) 2024-09-20T23:00:38.005 INFO:journalctl@ceph.osd.7.smithi154.stdout:Sep 20 23:00:37 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-7[54754]: 2024-09-20T23:00:37.578+0000 7fee5b5be640 -1 osd.7 1036 heartbeat_check: no reply from 172.21.15.84:6815 osd.3 since back 2024-09-20T23:00:08.173714+0000 front 2024-09-20T23:00:08.173807+0000 (oldest deadline 2024-09-20T23:00:31.574003+0000) 2024-09-20T23:00:39.004 INFO:journalctl@ceph.osd.7.smithi154.stdout:Sep 20 23:00:38 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-7[54754]: 2024-09-20T23:00:38.541+0000 7fee5b5be640 -1 osd.7 1036 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-09-20T22:59:47.071784+0000 front 2024-09-20T22:59:47.071677+0000 (oldest deadline 2024-09-20T23:00:11.171776+0000) 2024-09-20T23:00:39.005 INFO:journalctl@ceph.osd.7.smithi154.stdout:Sep 20 23:00:38 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-7[54754]: 2024-09-20T23:00:38.541+0000 7fee5b5be640 -1 osd.7 1036 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-09-20T22:59:51.172472+0000 front 2024-09-20T22:59:51.172371+0000 (oldest deadline 2024-09-20T23:00:16.472189+0000) 2024-09-20T23:00:39.005 INFO:journalctl@ceph.osd.7.smithi154.stdout:Sep 20 23:00:38 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-7[54754]: 2024-09-20T23:00:38.541+0000 7fee5b5be640 -1 osd.7 1036 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-09-20T23:00:00.573109+0000 front 2024-09-20T23:00:00.573085+0000 (oldest deadline 2024-09-20T23:00:24.072955+0000) 2024-09-20T23:00:39.005 INFO:journalctl@ceph.osd.7.smithi154.stdout:Sep 20 23:00:38 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-7[54754]: 2024-09-20T23:00:38.541+0000 7fee5b5be640 -1 osd.7 1036 heartbeat_check: no reply from 172.21.15.84:6815 osd.3 since back 2024-09-20T23:00:08.173714+0000 front 2024-09-20T23:00:08.173807+0000 (oldest deadline 2024-09-20T23:00:31.574003+0000) 2024-09-20T23:00:39.754 INFO:journalctl@ceph.osd.7.smithi154.stdout:Sep 20 23:00:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-7[54754]: 2024-09-20T23:00:39.499+0000 7fee5b5be640 -1 osd.7 1036 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-09-20T22:59:47.071784+0000 front 2024-09-20T22:59:47.071677+0000 (oldest deadline 2024-09-20T23:00:11.171776+0000) 2024-09-20T23:00:39.754 INFO:journalctl@ceph.osd.7.smithi154.stdout:Sep 20 23:00:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-7[54754]: 2024-09-20T23:00:39.499+0000 7fee5b5be640 -1 osd.7 1036 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-09-20T22:59:51.172472+0000 front 2024-09-20T22:59:51.172371+0000 (oldest deadline 2024-09-20T23:00:16.472189+0000) 2024-09-20T23:00:39.754 INFO:journalctl@ceph.osd.7.smithi154.stdout:Sep 20 23:00:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-7[54754]: 2024-09-20T23:00:39.499+0000 7fee5b5be640 -1 osd.7 1036 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-09-20T23:00:00.573109+0000 front 2024-09-20T23:00:00.573085+0000 (oldest deadline 2024-09-20T23:00:24.072955+0000) 2024-09-20T23:00:39.754 INFO:journalctl@ceph.osd.7.smithi154.stdout:Sep 20 23:00:39 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-7[54754]: 2024-09-20T23:00:39.499+0000 7fee5b5be640 -1 osd.7 1036 heartbeat_check: no reply from 172.21.15.84:6815 osd.3 since back 2024-09-20T23:00:08.173714+0000 front 2024-09-20T23:00:08.173807+0000 (oldest deadline 2024-09-20T23:00:31.574003+0000) 2024-09-20T23:00:41.004 INFO:journalctl@ceph.osd.7.smithi154.stdout:Sep 20 23:00:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-7[54754]: 2024-09-20T23:00:40.534+0000 7fee5b5be640 -1 osd.7 1036 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-09-20T22:59:47.071784+0000 front 2024-09-20T22:59:47.071677+0000 (oldest deadline 2024-09-20T23:00:11.171776+0000) 2024-09-20T23:00:41.004 INFO:journalctl@ceph.osd.7.smithi154.stdout:Sep 20 23:00:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-7[54754]: 2024-09-20T23:00:40.534+0000 7fee5b5be640 -1 osd.7 1036 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-09-20T22:59:51.172472+0000 front 2024-09-20T22:59:51.172371+0000 (oldest deadline 2024-09-20T23:00:16.472189+0000) 2024-09-20T23:00:41.004 INFO:journalctl@ceph.osd.7.smithi154.stdout:Sep 20 23:00:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-7[54754]: 2024-09-20T23:00:40.534+0000 7fee5b5be640 -1 osd.7 1036 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-09-20T23:00:00.573109+0000 front 2024-09-20T23:00:00.573085+0000 (oldest deadline 2024-09-20T23:00:24.072955+0000) 2024-09-20T23:00:41.005 INFO:journalctl@ceph.osd.7.smithi154.stdout:Sep 20 23:00:40 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-7[54754]: 2024-09-20T23:00:40.534+0000 7fee5b5be640 -1 osd.7 1036 heartbeat_check: no reply from 172.21.15.84:6815 osd.3 since back 2024-09-20T23:00:08.173714+0000 front 2024-09-20T23:00:08.173807+0000 (oldest deadline 2024-09-20T23:00:31.574003+0000) 2024-09-20T23:00:41.906 INFO:journalctl@ceph.osd.7.smithi154.stdout:Sep 20 23:00:41 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-7[54754]: 2024-09-20T23:00:41.574+0000 7fee5b5be640 -1 osd.7 1036 heartbeat_check: no reply from 172.21.15.84:6803 osd.0 since back 2024-09-20T22:59:47.071784+0000 front 2024-09-20T22:59:47.071677+0000 (oldest deadline 2024-09-20T23:00:11.171776+0000) 2024-09-20T23:00:41.906 INFO:journalctl@ceph.osd.7.smithi154.stdout:Sep 20 23:00:41 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-7[54754]: 2024-09-20T23:00:41.574+0000 7fee5b5be640 -1 osd.7 1036 heartbeat_check: no reply from 172.21.15.84:6807 osd.1 since back 2024-09-20T22:59:51.172472+0000 front 2024-09-20T22:59:51.172371+0000 (oldest deadline 2024-09-20T23:00:16.472189+0000) 2024-09-20T23:00:41.906 INFO:journalctl@ceph.osd.7.smithi154.stdout:Sep 20 23:00:41 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-7[54754]: 2024-09-20T23:00:41.574+0000 7fee5b5be640 -1 osd.7 1036 heartbeat_check: no reply from 172.21.15.84:6811 osd.2 since back 2024-09-20T23:00:00.573109+0000 front 2024-09-20T23:00:00.573085+0000 (oldest deadline 2024-09-20T23:00:24.072955+0000) 2024-09-20T23:00:41.906 INFO:journalctl@ceph.osd.7.smithi154.stdout:Sep 20 23:00:41 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-7[54754]: 2024-09-20T23:00:41.574+0000 7fee5b5be640 -1 osd.7 1036 heartbeat_check: no reply from 172.21.15.84:6815 osd.3 since back 2024-09-20T23:00:08.173714+0000 front 2024-09-20T23:00:08.173807+0000 (oldest deadline 2024-09-20T23:00:31.574003+0000) 2024-09-20T23:00:43.029 INFO:journalctl@ceph.osd.7.smithi154.stdout:Sep 20 23:00:42 smithi154 podman[153821]: 2024-09-20 23:00:42.754120525 +0000 UTC m=+5.667271527 container died 0134c2544b9343faf6f50a66779edb8078cc8fd74a4352c554309fc080ed9511 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-7, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-9e2ac70, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0) 2024-09-20T23:00:43.379 INFO:journalctl@ceph.osd.7.smithi154.stdout:Sep 20 23:00:43 smithi154 podman[153821]: 2024-09-20 23:00:43.118268635 +0000 UTC m=+6.031419637 container cleanup 0134c2544b9343faf6f50a66779edb8078cc8fd74a4352c554309fc080ed9511 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-7, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, RELEASE=squid-9e2ac70, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-20T23:00:43.379 INFO:journalctl@ceph.osd.7.smithi154.stdout:Sep 20 23:00:43 smithi154 bash[153821]: ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-7 2024-09-20T23:00:43.711 INFO:journalctl@ceph.osd.7.smithi154.stdout:Sep 20 23:00:43 smithi154 podman[153848]: 2024-09-20 23:00:43.377768642 +0000 UTC m=+0.620114765 container remove 0134c2544b9343faf6f50a66779edb8078cc8fd74a4352c554309fc080ed9511 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-7, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=squid-9e2ac70) 2024-09-20T23:00:44.004 INFO:journalctl@ceph.osd.7.smithi154.stdout:Sep 20 23:00:43 smithi154 podman[153934]: 2024-09-20 23:00:43.612754439 +0000 UTC m=+0.016236046 image pull cc9251b744a3faafca7c3095f79bfff81873afe823f31a03a11e015a34abcdbd quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5 2024-09-20T23:00:44.004 INFO:journalctl@ceph.osd.7.smithi154.stdout:Sep 20 23:00:43 smithi154 podman[153934]: 2024-09-20 23:00:43.720622615 +0000 UTC m=+0.124104264 container create 875f0c52119506f5061462e8c7790e38c7174a1a5b8caa2634608c97af54eaca (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-7-deactivate, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-9e2ac70) 2024-09-20T23:00:44.004 INFO:journalctl@ceph.osd.7.smithi154.stdout:Sep 20 23:00:43 smithi154 podman[153934]: 2024-09-20 23:00:43.939939203 +0000 UTC m=+0.343420817 container init 875f0c52119506f5061462e8c7790e38c7174a1a5b8caa2634608c97af54eaca (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-7-deactivate, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, RELEASE=squid-9e2ac70, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-09-20T23:00:44.004 INFO:journalctl@ceph.osd.7.smithi154.stdout:Sep 20 23:00:43 smithi154 podman[153934]: 2024-09-20 23:00:43.943167721 +0000 UTC m=+0.346649330 container start 875f0c52119506f5061462e8c7790e38c7174a1a5b8caa2634608c97af54eaca (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-7-deactivate, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=squid-9e2ac70, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True) 2024-09-20T23:00:44.410 INFO:journalctl@ceph.osd.7.smithi154.stdout:Sep 20 23:00:44 smithi154 podman[153934]: 2024-09-20 23:00:44.032850103 +0000 UTC m=+0.436331718 container attach 875f0c52119506f5061462e8c7790e38c7174a1a5b8caa2634608c97af54eaca (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-7-deactivate, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-9e2ac70, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-20T23:00:44.410 INFO:journalctl@ceph.osd.7.smithi154.stdout:Sep 20 23:00:44 smithi154 podman[153934]: 2024-09-20 23:00:44.135267657 +0000 UTC m=+0.538749264 container died 875f0c52119506f5061462e8c7790e38c7174a1a5b8caa2634608c97af54eaca (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-7-deactivate, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-9e2ac70, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-09-20T23:00:44.663 INFO:journalctl@ceph.osd.7.smithi154.stdout:Sep 20 23:00:44 smithi154 podman[153958]: 2024-09-20 23:00:44.650977687 +0000 UTC m=+0.504914927 container remove 875f0c52119506f5061462e8c7790e38c7174a1a5b8caa2634608c97af54eaca (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-osd-7-deactivate, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, RELEASE=squid-9e2ac70, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-09-20T23:00:44.668 DEBUG:teuthology.orchestra.run.smithi154:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490@osd.7.service' 2024-09-20T23:00:44.706 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-20T23:00:44.706 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-09-20T23:00:44.707 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopping rgw.foo.a... 2024-09-20T23:00:44.707 DEBUG:teuthology.orchestra.run.smithi084:> sudo systemctl stop ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490@rgw.foo.a 2024-09-20T23:00:45.006 INFO:journalctl@ceph.rgw.foo.a.smithi084.stdout:Sep 20 23:00:44 smithi084 systemd[1]: Stopping Ceph rgw.foo.a for a9c0bc70-77a0-11ef-baf5-efdc52797490... 2024-09-20T23:00:45.257 INFO:journalctl@ceph.rgw.foo.a.smithi084.stdout:Sep 20 23:00:45 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-rgw-foo-a[66081]: 2024-09-20T23:00:45.003+0000 7f95fae07640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/radosgw -n client.rgw.foo.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-20T23:00:45.258 INFO:journalctl@ceph.rgw.foo.a.smithi084.stdout:Sep 20 23:00:45 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-rgw-foo-a[66081]: 2024-09-20T23:00:45.003+0000 7f95fe673880 -1 shutting down 2024-09-20T23:00:55.346 INFO:journalctl@ceph.rgw.foo.a.smithi084.stdout:Sep 20 23:00:55 smithi084 bash[226805]: time="2024-09-20T23:00:55Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-rgw-foo-a in 10 seconds, resorting to SIGKILL" 2024-09-20T23:00:55.346 INFO:journalctl@ceph.rgw.foo.a.smithi084.stdout:Sep 20 23:00:55 smithi084 podman[226805]: 2024-09-20 23:00:55.032121286 +0000 UTC m=+10.128526923 container died 965332737d0498104993f06eef5bf117688ec60a7a771574f9a7de8a0aea92b7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-rgw-foo-a, maintainer=Guillaume Abrioux , RELEASE=squid-9e2ac70, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True) 2024-09-20T23:00:55.615 INFO:journalctl@ceph.rgw.foo.a.smithi084.stdout:Sep 20 23:00:55 smithi084 podman[226805]: 2024-09-20 23:00:55.518447236 +0000 UTC m=+10.614852916 container cleanup 965332737d0498104993f06eef5bf117688ec60a7a771574f9a7de8a0aea92b7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-rgw-foo-a, org.label-schema.build-date=20240731, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=squid-9e2ac70, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-20T23:00:55.615 INFO:journalctl@ceph.rgw.foo.a.smithi084.stdout:Sep 20 23:00:55 smithi084 bash[226805]: ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-rgw-foo-a 2024-09-20T23:00:55.851 DEBUG:teuthology.orchestra.run.smithi084:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490@rgw.foo.a.service' 2024-09-20T23:00:55.887 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-20T23:00:55.888 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopped rgw.foo.a 2024-09-20T23:00:55.888 INFO:tasks.cephadm.prometheus.a:Stopping prometheus.a... 2024-09-20T23:00:55.888 DEBUG:teuthology.orchestra.run.smithi154:> sudo systemctl stop ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490@prometheus.a 2024-09-20T23:00:56.187 INFO:journalctl@ceph.prometheus.a.smithi154.stdout:Sep 20 23:00:55 smithi154 systemd[1]: Stopping Ceph prometheus.a for a9c0bc70-77a0-11ef-baf5-efdc52797490... 2024-09-20T23:00:56.447 INFO:journalctl@ceph.prometheus.a.smithi154.stdout:Sep 20 23:00:56 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-prometheus-a[67141]: ts=2024-09-20T23:00:56.186Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-09-20T23:00:56.447 INFO:journalctl@ceph.prometheus.a.smithi154.stdout:Sep 20 23:00:56 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-prometheus-a[67141]: ts=2024-09-20T23:00:56.186Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-09-20T23:00:56.447 INFO:journalctl@ceph.prometheus.a.smithi154.stdout:Sep 20 23:00:56 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-prometheus-a[67141]: ts=2024-09-20T23:00:56.186Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-09-20T23:00:56.447 INFO:journalctl@ceph.prometheus.a.smithi154.stdout:Sep 20 23:00:56 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-prometheus-a[67141]: ts=2024-09-20T23:00:56.186Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-09-20T23:00:56.447 INFO:journalctl@ceph.prometheus.a.smithi154.stdout:Sep 20 23:00:56 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-prometheus-a[67141]: ts=2024-09-20T23:00:56.186Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-09-20T23:00:56.447 INFO:journalctl@ceph.prometheus.a.smithi154.stdout:Sep 20 23:00:56 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-prometheus-a[67141]: ts=2024-09-20T23:00:56.186Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-09-20T23:00:56.448 INFO:journalctl@ceph.prometheus.a.smithi154.stdout:Sep 20 23:00:56 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-prometheus-a[67141]: ts=2024-09-20T23:00:56.186Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-09-20T23:00:56.448 INFO:journalctl@ceph.prometheus.a.smithi154.stdout:Sep 20 23:00:56 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-prometheus-a[67141]: ts=2024-09-20T23:00:56.186Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-09-20T23:00:56.448 INFO:journalctl@ceph.prometheus.a.smithi154.stdout:Sep 20 23:00:56 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-prometheus-a[67141]: ts=2024-09-20T23:00:56.186Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-09-20T23:00:56.448 INFO:journalctl@ceph.prometheus.a.smithi154.stdout:Sep 20 23:00:56 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-prometheus-a[67141]: ts=2024-09-20T23:00:56.186Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-09-20T23:00:56.448 INFO:journalctl@ceph.prometheus.a.smithi154.stdout:Sep 20 23:00:56 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-prometheus-a[67141]: ts=2024-09-20T23:00:56.186Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-09-20T23:00:56.448 INFO:journalctl@ceph.prometheus.a.smithi154.stdout:Sep 20 23:00:56 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-prometheus-a[67141]: ts=2024-09-20T23:00:56.186Z caller=main.go:1147 level=info msg="See you next time!" 2024-09-20T23:00:56.448 INFO:journalctl@ceph.prometheus.a.smithi154.stdout:Sep 20 23:00:56 smithi154 podman[154038]: 2024-09-20 23:00:56.196500306 +0000 UTC m=+0.110846672 container died 5c4f4836aceb1176504ae6451e7c68b4f7ea65a6eb77cc448425fb930ada26fc (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-20T23:00:56.754 INFO:journalctl@ceph.prometheus.a.smithi154.stdout:Sep 20 23:00:56 smithi154 podman[154038]: 2024-09-20 23:00:56.636261214 +0000 UTC m=+0.550607575 container cleanup 5c4f4836aceb1176504ae6451e7c68b4f7ea65a6eb77cc448425fb930ada26fc (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-20T23:00:56.754 INFO:journalctl@ceph.prometheus.a.smithi154.stdout:Sep 20 23:00:56 smithi154 bash[154038]: ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-prometheus-a 2024-09-20T23:00:57.049 DEBUG:teuthology.orchestra.run.smithi154:> sudo pkill -f 'journalctl -f -n 0 -u ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490@prometheus.a.service' 2024-09-20T23:00:57.070 INFO:journalctl@ceph.prometheus.a.smithi154.stdout:Sep 20 23:00:56 smithi154 podman[154050]: 2024-09-20 23:00:56.804106597 +0000 UTC m=+0.604338881 container remove 5c4f4836aceb1176504ae6451e7c68b4f7ea65a6eb77cc448425fb930ada26fc (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-20T23:00:57.071 INFO:journalctl@ceph.prometheus.a.smithi154.stdout:Sep 20 23:00:57 smithi154 systemd[1]: ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490@prometheus.a.service: Deactivated successfully. 2024-09-20T23:00:57.071 INFO:journalctl@ceph.prometheus.a.smithi154.stdout:Sep 20 23:00:57 smithi154 systemd[1]: Stopped Ceph prometheus.a for a9c0bc70-77a0-11ef-baf5-efdc52797490. 2024-09-20T23:00:57.071 INFO:journalctl@ceph.prometheus.a.smithi154.stdout:Sep 20 23:00:57 smithi154 systemd[1]: ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490@prometheus.a.service: Consumed 3.699s CPU time. 2024-09-20T23:00:57.084 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-20T23:00:57.084 INFO:tasks.cephadm.prometheus.a:Stopped prometheus.a 2024-09-20T23:00:57.084 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 --force --keep-logs 2024-09-20T23:00:57.279 INFO:teuthology.orchestra.run.smithi084.stdout:Deleting cluster with fsid: a9c0bc70-77a0-11ef-baf5-efdc52797490 2024-09-20T23:00:59.115 INFO:journalctl@ceph.alertmanager.a.smithi084.stdout:Sep 20 23:00:58 smithi084 systemd[1]: Stopping Ceph alertmanager.a for a9c0bc70-77a0-11ef-baf5-efdc52797490... 2024-09-20T23:00:59.550 INFO:journalctl@ceph.alertmanager.a.smithi084.stdout:Sep 20 23:00:59 smithi084 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-alertmanager-a[80657]: ts=2024-09-20T23:00:59.123Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-09-20T23:00:59.550 INFO:journalctl@ceph.alertmanager.a.smithi084.stdout:Sep 20 23:00:59 smithi084 podman[227309]: 2024-09-20 23:00:59.164315678 +0000 UTC m=+0.140393467 container died 355d6bdc525d4efb9427e6bd14cfbfc93cf60304576dd4d8b6088a6fb50b0281 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-20T23:00:59.550 INFO:journalctl@ceph.alertmanager.a.smithi084.stdout:Sep 20 23:00:59 smithi084 podman[227309]: 2024-09-20 23:00:59.548581163 +0000 UTC m=+0.524658981 container cleanup 355d6bdc525d4efb9427e6bd14cfbfc93cf60304576dd4d8b6088a6fb50b0281 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-20T23:00:59.864 INFO:journalctl@ceph.alertmanager.a.smithi084.stdout:Sep 20 23:00:59 smithi084 bash[227309]: ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-alertmanager-a 2024-09-20T23:00:59.865 INFO:journalctl@ceph.alertmanager.a.smithi084.stdout:Sep 20 23:00:59 smithi084 podman[227322]: 2024-09-20 23:00:59.708167169 +0000 UTC m=+0.535367586 container remove 355d6bdc525d4efb9427e6bd14cfbfc93cf60304576dd4d8b6088a6fb50b0281 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-20T23:00:59.865 INFO:journalctl@ceph.alertmanager.a.smithi084.stdout:Sep 20 23:00:59 smithi084 podman[227322]: 2024-09-20 23:00:59.783703098 +0000 UTC m=+0.610903514 volume remove 0ed528b373d9ea1f04e053aaada923b3cb6fd1bd18b53b4878c12a3f3e749132 2024-09-20T23:01:00.364 INFO:journalctl@ceph.alertmanager.a.smithi084.stdout:Sep 20 23:01:00 smithi084 systemd[1]: ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490@alertmanager.a.service: Deactivated successfully. 2024-09-20T23:01:00.365 INFO:journalctl@ceph.alertmanager.a.smithi084.stdout:Sep 20 23:01:00 smithi084 systemd[1]: Stopped Ceph alertmanager.a for a9c0bc70-77a0-11ef-baf5-efdc52797490. 2024-09-20T23:01:00.365 INFO:journalctl@ceph.alertmanager.a.smithi084.stdout:Sep 20 23:01:00 smithi084 systemd[1]: ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490@alertmanager.a.service: Consumed 1.473s CPU time. 2024-09-20T23:01:00.365 INFO:journalctl@ceph.node-exporter.a.smithi084.stdout:Sep 20 23:01:00 smithi084 systemd[1]: Stopping Ceph node-exporter.a for a9c0bc70-77a0-11ef-baf5-efdc52797490... 2024-09-20T23:01:00.763 INFO:journalctl@ceph.node-exporter.a.smithi084.stdout:Sep 20 23:01:00 smithi084 podman[227435]: 2024-09-20 23:01:00.497219507 +0000 UTC m=+0.150515730 container died ea6c1ea853e358f65915c309c5219781960766415d008c985531d46bd82c7b3b (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-20T23:01:01.097 INFO:journalctl@ceph.node-exporter.a.smithi084.stdout:Sep 20 23:01:00 smithi084 podman[227435]: 2024-09-20 23:01:00.928147435 +0000 UTC m=+0.581443656 container cleanup ea6c1ea853e358f65915c309c5219781960766415d008c985531d46bd82c7b3b (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-20T23:01:01.097 INFO:journalctl@ceph.node-exporter.a.smithi084.stdout:Sep 20 23:01:00 smithi084 bash[227435]: ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-node-exporter-a 2024-09-20T23:01:01.349 INFO:journalctl@ceph.node-exporter.a.smithi084.stdout:Sep 20 23:01:01 smithi084 podman[227447]: 2024-09-20 23:01:01.09592859 +0000 UTC m=+0.598191413 container remove ea6c1ea853e358f65915c309c5219781960766415d008c985531d46bd82c7b3b (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-20T23:01:01.349 INFO:journalctl@ceph.node-exporter.a.smithi084.stdout:Sep 20 23:01:01 smithi084 systemd[1]: ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-09-20T23:01:01.349 INFO:journalctl@ceph.node-exporter.a.smithi084.stdout:Sep 20 23:01:01 smithi084 systemd[1]: ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490@node-exporter.a.service: Failed with result 'exit-code'. 2024-09-20T23:01:01.349 INFO:journalctl@ceph.node-exporter.a.smithi084.stdout:Sep 20 23:01:01 smithi084 systemd[1]: Stopped Ceph node-exporter.a for a9c0bc70-77a0-11ef-baf5-efdc52797490. 2024-09-20T23:01:01.349 INFO:journalctl@ceph.node-exporter.a.smithi084.stdout:Sep 20 23:01:01 smithi084 systemd[1]: ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490@node-exporter.a.service: Consumed 3.505s CPU time. 2024-09-20T23:01:01.947 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 --force --keep-logs 2024-09-20T23:01:02.148 INFO:teuthology.orchestra.run.smithi154.stdout:Deleting cluster with fsid: a9c0bc70-77a0-11ef-baf5-efdc52797490 2024-09-20T23:01:03.668 INFO:journalctl@ceph.iscsi.iscsi.a.smithi154.stdout:Sep 20 23:01:03 smithi154 systemd[1]: Stopping Ceph iscsi.iscsi.a for a9c0bc70-77a0-11ef-baf5-efdc52797490... 2024-09-20T23:01:04.004 INFO:journalctl@ceph.iscsi.iscsi.a.smithi154.stdout:Sep 20 23:01:03 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-iscsi-iscsi-a[59730]: debug Shutdown received 2024-09-20T23:01:13.968 INFO:journalctl@ceph.iscsi.iscsi.a.smithi154.stdout:Sep 20 23:01:13 smithi154 bash[154462]: time="2024-09-20T23:01:13Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-iscsi-iscsi-a in 10 seconds, resorting to SIGKILL" 2024-09-20T23:01:13.968 INFO:journalctl@ceph.iscsi.iscsi.a.smithi154.stdout:Sep 20 23:01:13 smithi154 podman[154462]: 2024-09-20 23:01:13.683371813 +0000 UTC m=+10.125924220 container died 5a673a67abedc3e3b721b39df9309f1c8484cb4208530eb77544cf1480f20c7b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-iscsi-iscsi-a, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=squid-9e2ac70, io.buildah.version=1.37.2) 2024-09-20T23:01:14.254 INFO:journalctl@ceph.iscsi.iscsi.a.smithi154.stdout:Sep 20 23:01:14 smithi154 podman[154462]: 2024-09-20 23:01:14.150451087 +0000 UTC m=+10.593003485 container cleanup 5a673a67abedc3e3b721b39df9309f1c8484cb4208530eb77544cf1480f20c7b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-iscsi-iscsi-a, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=squid-9e2ac70, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-09-20T23:01:14.254 INFO:journalctl@ceph.iscsi.iscsi.a.smithi154.stdout:Sep 20 23:01:14 smithi154 bash[154462]: ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-iscsi-iscsi-a 2024-09-20T23:01:14.724 INFO:journalctl@ceph.iscsi.iscsi.a.smithi154.stdout:Sep 20 23:01:14 smithi154 podman[154475]: 2024-09-20 23:01:14.385104859 +0000 UTC m=+0.700546072 container remove 5a673a67abedc3e3b721b39df9309f1c8484cb4208530eb77544cf1480f20c7b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-iscsi-iscsi-a, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, RELEASE=squid-9e2ac70, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-09-20T23:01:14.724 INFO:journalctl@ceph.iscsi.iscsi.a.smithi154.stdout:Sep 20 23:01:14 smithi154 systemd[1]: ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490@iscsi.iscsi.a.service: Main process exited, code=exited, status=137/n/a 2024-09-20T23:01:14.724 INFO:journalctl@ceph.iscsi.iscsi.a.smithi154.stdout:Sep 20 23:01:14 smithi154 systemd[1]: ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490@iscsi.iscsi.a.service: Failed with result 'exit-code'. 2024-09-20T23:01:14.724 INFO:journalctl@ceph.iscsi.iscsi.a.smithi154.stdout:Sep 20 23:01:14 smithi154 systemd[1]: Stopped Ceph iscsi.iscsi.a for a9c0bc70-77a0-11ef-baf5-efdc52797490. 2024-09-20T23:01:14.724 INFO:journalctl@ceph.iscsi.iscsi.a.smithi154.stdout:Sep 20 23:01:14 smithi154 systemd[1]: ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490@iscsi.iscsi.a.service: Consumed 2.579s CPU time. 2024-09-20T23:01:16.315 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 23:01:16 smithi154 systemd[1]: Stopping Ceph grafana.a for a9c0bc70-77a0-11ef-baf5-efdc52797490... 2024-09-20T23:01:16.316 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 23:01:16 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=server t=2024-09-20T23:01:16.251170933Z level=info msg="Shutdown started" reason="System signal: terminated" 2024-09-20T23:01:16.316 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 23:01:16 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=tracing t=2024-09-20T23:01:16.25163471Z level=info msg="Closing tracing" 2024-09-20T23:01:16.316 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 23:01:16 smithi154 ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a[62618]: logger=ticker t=2024-09-20T23:01:16.251675256Z level=info msg=stopped last_tick=2024-09-20T23:01:10Z 2024-09-20T23:01:16.316 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 23:01:16 smithi154 podman[154748]: 2024-09-20 23:01:16.261495427 +0000 UTC m=+0.101959527 container died 919357740c28646a2ab509029b568ec9ad0633a06f1a561fc4007b3f8dba9eed (image=quay.io/ceph/grafana:9.4.12, name=ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a, maintainer=Grafana Labs ) 2024-09-20T23:01:16.877 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 23:01:16 smithi154 podman[154748]: 2024-09-20 23:01:16.617181501 +0000 UTC m=+0.457645628 container cleanup 919357740c28646a2ab509029b568ec9ad0633a06f1a561fc4007b3f8dba9eed (image=quay.io/ceph/grafana:9.4.12, name=ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a, maintainer=Grafana Labs ) 2024-09-20T23:01:16.878 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 23:01:16 smithi154 bash[154748]: ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a 2024-09-20T23:01:17.238 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 23:01:16 smithi154 podman[154760]: 2024-09-20 23:01:16.876618539 +0000 UTC m=+0.616370874 container remove 919357740c28646a2ab509029b568ec9ad0633a06f1a561fc4007b3f8dba9eed (image=quay.io/ceph/grafana:9.4.12, name=ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-grafana-a, maintainer=Grafana Labs ) 2024-09-20T23:01:17.238 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 23:01:17 smithi154 systemd[1]: ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490@grafana.a.service: Deactivated successfully. 2024-09-20T23:01:17.238 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 23:01:17 smithi154 systemd[1]: Stopped Ceph grafana.a for a9c0bc70-77a0-11ef-baf5-efdc52797490. 2024-09-20T23:01:17.238 INFO:journalctl@ceph.grafana.a.smithi154.stdout:Sep 20 23:01:17 smithi154 systemd[1]: ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490@grafana.a.service: Consumed 12.454s CPU time. 2024-09-20T23:01:17.492 INFO:journalctl@ceph.node-exporter.b.smithi154.stdout:Sep 20 23:01:17 smithi154 systemd[1]: Stopping Ceph node-exporter.b for a9c0bc70-77a0-11ef-baf5-efdc52797490... 2024-09-20T23:01:17.754 INFO:journalctl@ceph.node-exporter.b.smithi154.stdout:Sep 20 23:01:17 smithi154 podman[154877]: 2024-09-20 23:01:17.491110739 +0000 UTC m=+0.114837889 container died 750facd7728a4051f1d7a46be367007cdedb623e0e04a2fd7a44da494645fe3e (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-20T23:01:18.100 INFO:journalctl@ceph.node-exporter.b.smithi154.stdout:Sep 20 23:01:17 smithi154 podman[154877]: 2024-09-20 23:01:17.779108846 +0000 UTC m=+0.402835989 container cleanup 750facd7728a4051f1d7a46be367007cdedb623e0e04a2fd7a44da494645fe3e (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-20T23:01:18.100 INFO:journalctl@ceph.node-exporter.b.smithi154.stdout:Sep 20 23:01:17 smithi154 bash[154877]: ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-node-exporter-b 2024-09-20T23:01:18.100 INFO:journalctl@ceph.node-exporter.b.smithi154.stdout:Sep 20 23:01:17 smithi154 podman[154889]: 2024-09-20 23:01:17.996789337 +0000 UTC m=+0.508280713 container remove 750facd7728a4051f1d7a46be367007cdedb623e0e04a2fd7a44da494645fe3e (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-20T23:01:18.100 INFO:journalctl@ceph.node-exporter.b.smithi154.stdout:Sep 20 23:01:17 smithi154 systemd[1]: ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-09-20T23:01:18.504 INFO:journalctl@ceph.node-exporter.b.smithi154.stdout:Sep 20 23:01:18 smithi154 systemd[1]: ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490@node-exporter.b.service: Failed with result 'exit-code'. 2024-09-20T23:01:18.504 INFO:journalctl@ceph.node-exporter.b.smithi154.stdout:Sep 20 23:01:18 smithi154 systemd[1]: Stopped Ceph node-exporter.b for a9c0bc70-77a0-11ef-baf5-efdc52797490. 2024-09-20T23:01:18.505 INFO:journalctl@ceph.node-exporter.b.smithi154.stdout:Sep 20 23:01:18 smithi154 systemd[1]: ceph-a9c0bc70-77a0-11ef-baf5-efdc52797490@node-exporter.b.service: Consumed 3.304s CPU time. 2024-09-20T23:01:18.769 DEBUG:teuthology.orchestra.run.smithi084:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-20T23:01:18.802 DEBUG:teuthology.orchestra.run.smithi154:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-20T23:01:18.833 INFO:tasks.cephadm:Archiving crash dumps... 2024-09-20T23:01:18.834 DEBUG:teuthology.misc:Transferring archived files from smithi084:/var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/crash to /home/teuthworker/archive/teuthology-2024-09-20_21:08:02-orch-squid-distro-default-smithi/7913018/remote/smithi084/crash 2024-09-20T23:01:18.835 DEBUG:teuthology.orchestra.run.smithi084:> sudo tar c -f - -C /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/crash -- . 2024-09-20T23:01:18.885 INFO:teuthology.orchestra.run.smithi084.stderr:tar: /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/crash: Cannot open: No such file or directory 2024-09-20T23:01:18.885 INFO:teuthology.orchestra.run.smithi084.stderr:tar: Error is not recoverable: exiting now 2024-09-20T23:01:18.887 DEBUG:teuthology.misc:Transferring archived files from smithi154:/var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/crash to /home/teuthworker/archive/teuthology-2024-09-20_21:08:02-orch-squid-distro-default-smithi/7913018/remote/smithi154/crash 2024-09-20T23:01:18.888 DEBUG:teuthology.orchestra.run.smithi154:> sudo tar c -f - -C /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/crash -- . 2024-09-20T23:01:18.942 INFO:teuthology.orchestra.run.smithi154.stderr:tar: /var/lib/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/crash: Cannot open: No such file or directory 2024-09-20T23:01:18.943 INFO:teuthology.orchestra.run.smithi154.stderr:tar: Error is not recoverable: exiting now 2024-09-20T23:01:18.944 INFO:tasks.cephadm:Checking cluster log for badness... 2024-09-20T23:01:18.944 DEBUG:teuthology.orchestra.run.smithi084:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/ceph.log | egrep CEPHADM_ | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v CEPHADM_STRAY_DAEMON | egrep -v CEPHADM_FAILED_DAEMON | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2024-09-20T23:01:18.975 INFO:tasks.cephadm:Compressing logs... 2024-09-20T23:01:18.975 DEBUG:teuthology.orchestra.run.smithi084:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-20T23:01:19.019 DEBUG:teuthology.orchestra.run.smithi154:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-20T23:01:19.042 INFO:teuthology.orchestra.run.smithi084.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/ceph-osd.0.log 2024-09-20T23:01:19.042 INFO:teuthology.orchestra.run.smithi084.stderr:: No such file or directory 2024-09-20T23:01:19.042 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/ceph-osd.1.log 2024-09-20T23:01:19.045 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/objectstore_tool.2.log 2024-09-20T23:01:19.045 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/ceph-osd.0.log: /var/log/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/ceph-volume.log 2024-09-20T23:01:19.045 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/objectstore_tool.2.log: gzip -5 --verbose -- /var/log/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/ceph.audit.log 2024-09-20T23:01:19.045 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/ceph-mon.c.log 2024-09-20T23:01:19.046 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/ceph.cephadm.log 2024-09-20T23:01:19.047 INFO:teuthology.orchestra.run.smithi154.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/tcmu-runner.log 2024-09-20T23:01:19.047 INFO:teuthology.orchestra.run.smithi154.stderr:: No such file or directory 2024-09-20T23:01:19.047 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/ceph-osd.7.log 2024-09-20T23:01:19.047 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/ceph-volume.log 2024-09-20T23:01:19.048 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/tcmu-runner.log: 63.9% -- replaced with /var/log/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/tcmu-runner.log.gz 2024-09-20T23:01:19.048 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/ceph-osd.4.log 2024-09-20T23:01:19.048 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/ceph.audit.log 2024-09-20T23:01:19.048 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/ceph.cephadm.log 2024-09-20T23:01:19.048 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/ceph-osd.6.log 2024-09-20T23:01:19.048 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/ceph-osd.5.log 2024-09-20T23:01:19.048 INFO:teuthology.orchestra.run.smithi154.stderr: 79.8%/var/log/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/ceph-osd.6.log: -- replaced with /var/log/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/ceph.cephadm.log.gz 2024-09-20T23:01:19.048 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/ceph.log 2024-09-20T23:01:19.048 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/ceph-mgr.x.log 2024-09-20T23:01:19.048 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/ceph-osd.5.log: /var/log/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/ceph.log: /var/log/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/ceph-mon.b.log 2024-09-20T23:01:19.048 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/ceph-mon.a.log 2024-09-20T23:01:19.049 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/ceph-osd.2.log 2024-09-20T23:01:19.049 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/ceph-mon.a.log: 89.2% -- replaced with /var/log/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/ceph.cephadm.log.gz 2024-09-20T23:01:19.049 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/ceph-mgr.y.log 2024-09-20T23:01:19.050 INFO:teuthology.orchestra.run.smithi154.stderr: 90.0% -- replaced with /var/log/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/ceph.audit.log.gz 2024-09-20T23:01:19.053 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/ceph-osd.2.log: 94.0% -- replaced with /var/log/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/ceph.audit.log.gz 2024-09-20T23:01:19.054 INFO:teuthology.orchestra.run.smithi154.stderr: 86.0% -- replaced with /var/log/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/ceph.log.gz 2024-09-20T23:01:19.055 INFO:teuthology.orchestra.run.smithi154.stderr: 91.1% -- replaced with /var/log/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/ceph-volume.log.gz 2024-09-20T23:01:19.056 INFO:teuthology.orchestra.run.smithi084.stderr: 91.2% -- replaced with /var/log/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/ceph-volume.log.gz 2024-09-20T23:01:19.056 INFO:teuthology.orchestra.run.smithi154.stderr: 91.9% -- replaced with /var/log/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/ceph-mgr.x.log.gz 2024-09-20T23:01:19.057 INFO:teuthology.orchestra.run.smithi154.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-09-20T23:01:19.058 INFO:teuthology.orchestra.run.smithi084.stderr: 94.7% -- replaced with /var/log/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/objectstore_tool.2.log.gz 2024-09-20T23:01:19.066 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/ceph.log 2024-09-20T23:01:19.066 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/ceph-client.rgw.foo.a.log 2024-09-20T23:01:19.067 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/ceph.log: gzip -5 --verbose -- /var/log/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/ceph-osd.3.log 2024-09-20T23:01:19.067 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/ceph-client.rgw.foo.a.log: 58.2% -- replaced with /var/log/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/ceph-client.rgw.foo.a.log.gz 2024-09-20T23:01:19.071 INFO:teuthology.orchestra.run.smithi084.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-09-20T23:01:19.075 INFO:teuthology.orchestra.run.smithi154.stderr:/var/log/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/ceph-mon.b.log: /var/log/ceph/cephadm.log: 92.6% -- replaced with /var/log/ceph/cephadm.log.gz 2024-09-20T23:01:19.083 INFO:teuthology.orchestra.run.smithi084.stderr:/var/log/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/ceph-osd.3.log: /var/log/ceph/cephadm.log: 92.0% -- replaced with /var/log/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/ceph.log.gz 2024-09-20T23:01:19.097 INFO:teuthology.orchestra.run.smithi084.stderr: 95.7% -- replaced with /var/log/ceph/cephadm.log.gz 2024-09-20T23:01:19.322 INFO:teuthology.orchestra.run.smithi084.stderr: 90.0% -- replaced with /var/log/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/ceph-mgr.y.log.gz 2024-09-20T23:01:19.591 INFO:teuthology.orchestra.run.smithi154.stderr: 91.0% -- replaced with /var/log/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/ceph-mon.b.log.gz 2024-09-20T23:01:19.936 INFO:teuthology.orchestra.run.smithi084.stderr: 90.8% -- replaced with /var/log/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/ceph-mon.c.log.gz 2024-09-20T23:01:20.959 INFO:teuthology.orchestra.run.smithi084.stderr: 90.4% -- replaced with /var/log/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/ceph-mon.a.log.gz 2024-09-20T23:01:22.483 INFO:teuthology.orchestra.run.smithi154.stderr: 94.0% -- replaced with /var/log/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/ceph-osd.4.log.gz 2024-09-20T23:01:23.842 INFO:teuthology.orchestra.run.smithi154.stderr: 93.8% -- replaced with /var/log/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/ceph-osd.7.log.gz 2024-09-20T23:01:24.052 INFO:teuthology.orchestra.run.smithi154.stderr: 93.9% -- replaced with /var/log/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/ceph-osd.5.log.gz 2024-09-20T23:01:24.235 INFO:teuthology.orchestra.run.smithi084.stderr: 93.8% -- replaced with /var/log/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/ceph-osd.2.log.gz 2024-09-20T23:01:24.299 INFO:teuthology.orchestra.run.smithi084.stderr: 94.0% -- replaced with /var/log/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/ceph-osd.1.log.gz 2024-09-20T23:01:24.726 INFO:teuthology.orchestra.run.smithi084.stderr: 94.0% -- replaced with /var/log/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/ceph-osd.0.log.gz 2024-09-20T23:01:24.831 INFO:teuthology.orchestra.run.smithi154.stderr: 93.8% -- replaced with /var/log/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/ceph-osd.6.log.gz 2024-09-20T23:01:24.832 INFO:teuthology.orchestra.run.smithi154.stderr: 2024-09-20T23:01:24.832 INFO:teuthology.orchestra.run.smithi154.stderr:real 0m5.800s 2024-09-20T23:01:24.832 INFO:teuthology.orchestra.run.smithi154.stderr:user 0m18.839s 2024-09-20T23:01:24.832 INFO:teuthology.orchestra.run.smithi154.stderr:sys 0m0.766s 2024-09-20T23:01:25.534 INFO:teuthology.orchestra.run.smithi084.stderr: 94.0% -- replaced with /var/log/ceph/a9c0bc70-77a0-11ef-baf5-efdc52797490/ceph-osd.3.log.gz 2024-09-20T23:01:25.536 INFO:teuthology.orchestra.run.smithi084.stderr: 2024-09-20T23:01:25.537 INFO:teuthology.orchestra.run.smithi084.stderr:real 0m6.505s 2024-09-20T23:01:25.537 INFO:teuthology.orchestra.run.smithi084.stderr:user 0m24.766s 2024-09-20T23:01:25.537 INFO:teuthology.orchestra.run.smithi084.stderr:sys 0m0.918s 2024-09-20T23:01:25.537 INFO:tasks.cephadm:Archiving logs... 2024-09-20T23:01:25.537 DEBUG:teuthology.misc:Transferring archived files from smithi084:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-20_21:08:02-orch-squid-distro-default-smithi/7913018/remote/smithi084/log 2024-09-20T23:01:25.538 DEBUG:teuthology.orchestra.run.smithi084:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-20T23:01:28.779 DEBUG:teuthology.misc:Transferring archived files from smithi154:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-20_21:08:02-orch-squid-distro-default-smithi/7913018/remote/smithi154/log 2024-09-20T23:01:28.780 DEBUG:teuthology.orchestra.run.smithi154:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-20T23:01:31.392 INFO:tasks.cephadm:Removing cluster... 2024-09-20T23:01:31.393 DEBUG:teuthology.orchestra.run.smithi084:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 --force 2024-09-20T23:01:31.594 INFO:teuthology.orchestra.run.smithi084.stdout:Deleting cluster with fsid: a9c0bc70-77a0-11ef-baf5-efdc52797490 2024-09-20T23:01:31.876 DEBUG:teuthology.orchestra.run.smithi154:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid a9c0bc70-77a0-11ef-baf5-efdc52797490 --force 2024-09-20T23:01:32.081 INFO:teuthology.orchestra.run.smithi154.stdout:Deleting cluster with fsid: a9c0bc70-77a0-11ef-baf5-efdc52797490 2024-09-20T23:01:32.355 INFO:tasks.cephadm:Removing cephadm ... 2024-09-20T23:01:32.355 DEBUG:teuthology.orchestra.run.smithi084:> rm -rf /home/ubuntu/cephtest/cephadm 2024-09-20T23:01:32.370 DEBUG:teuthology.orchestra.run.smithi154:> rm -rf /home/ubuntu/cephtest/cephadm 2024-09-20T23:01:32.386 INFO:tasks.cephadm:Teardown complete 2024-09-20T23:01:32.386 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-20T23:01:32.396 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-20T23:01:32.396 DEBUG:teuthology.orchestra.run.smithi084:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-20T23:01:32.413 DEBUG:teuthology.orchestra.run.smithi154:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-20T23:01:32.501 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-20T23:01:32.516 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-20T23:01:32.516 DEBUG:teuthology.orchestra.run.smithi084:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-20T23:01:32.518 DEBUG:teuthology.orchestra.run.smithi154:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-20T23:01:32.530 INFO:teuthology.orchestra.run.smithi084.stderr:bash: line 1: ntpq: command not found 2024-09-20T23:01:32.533 INFO:teuthology.orchestra.run.smithi154.stderr:bash: line 1: ntpq: command not found 2024-09-20T23:01:32.535 INFO:teuthology.orchestra.run.smithi084.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-20T23:01:32.535 INFO:teuthology.orchestra.run.smithi084.stdout:=============================================================================== 2024-09-20T23:01:32.535 INFO:teuthology.orchestra.run.smithi084.stdout:^* hv01.front.sepia.ceph.com 3 6 377 7 -253us[ -260us] +/- 50ms 2024-09-20T23:01:32.535 INFO:teuthology.orchestra.run.smithi084.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 8 -297us[ -305us] +/- 57ms 2024-09-20T23:01:32.535 INFO:teuthology.orchestra.run.smithi084.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 8 +942us[ +934us] +/- 87ms 2024-09-20T23:01:32.535 INFO:teuthology.orchestra.run.smithi084.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-09-20T23:01:32.537 INFO:teuthology.orchestra.run.smithi154.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-20T23:01:32.537 INFO:teuthology.orchestra.run.smithi154.stdout:=============================================================================== 2024-09-20T23:01:32.537 INFO:teuthology.orchestra.run.smithi154.stdout:^* hv01.front.sepia.ceph.com 3 6 377 8 -267us[ -269us] +/- 50ms 2024-09-20T23:01:32.537 INFO:teuthology.orchestra.run.smithi154.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 9 -317us[ -319us] +/- 57ms 2024-09-20T23:01:32.537 INFO:teuthology.orchestra.run.smithi154.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 11 +921us[ +919us] +/- 87ms 2024-09-20T23:01:32.537 INFO:teuthology.orchestra.run.smithi154.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-09-20T23:01:32.538 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-20T23:01:32.547 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-20T23:01:32.548 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-20T23:01:32.555 DEBUG:teuthology.orchestra.run.smithi084:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-20T23:01:32.634 DEBUG:teuthology.orchestra.run.smithi154:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-20T23:01:32.674 DEBUG:teuthology.orchestra.run.smithi084:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-20T23:01:32.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T23:01:32.701 DEBUG:teuthology.orchestra.run.smithi154:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-20T23:01:32.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T23:01:32.739 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-20T23:01:32.749 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-20T23:01:32.793 INFO:teuthology.task.internal:Duration was 1914.639151 seconds 2024-09-20T23:01:32.793 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-20T23:01:32.802 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-20T23:01:32.802 DEBUG:teuthology.orchestra.run.smithi084:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-20T23:01:32.804 DEBUG:teuthology.orchestra.run.smithi154:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-20T23:01:32.840 INFO:teuthology.orchestra.run.smithi084.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-20T23:01:32.843 INFO:teuthology.orchestra.run.smithi154.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-20T23:01:33.151 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-20T23:01:33.151 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi084.front.sepia.ceph.com 2024-09-20T23:01:33.151 DEBUG:teuthology.orchestra.run.smithi084:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-20T23:01:33.211 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi154.front.sepia.ceph.com 2024-09-20T23:01:33.211 DEBUG:teuthology.orchestra.run.smithi154:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-20T23:01:33.240 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-20T23:01:33.240 DEBUG:teuthology.orchestra.run.smithi084:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-20T23:01:33.254 DEBUG:teuthology.orchestra.run.smithi154:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-20T23:01:33.307 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-20T23:01:33.307 DEBUG:teuthology.orchestra.run.smithi084:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-20T23:01:33.321 DEBUG:teuthology.orchestra.run.smithi154:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-20T23:01:35.531 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-20T23:01:35.541 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-20T23:01:35.541 DEBUG:teuthology.orchestra.run.smithi084:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-20T23:01:35.566 DEBUG:teuthology.orchestra.run.smithi154:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-20T23:01:35.597 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-20T23:01:35.607 DEBUG:teuthology.orchestra.run.smithi084:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-20T23:01:35.609 DEBUG:teuthology.orchestra.run.smithi154:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-20T23:01:35.630 INFO:teuthology.orchestra.run.smithi084.stdout:kernel.core_pattern = core 2024-09-20T23:01:35.664 INFO:teuthology.orchestra.run.smithi154.stdout:kernel.core_pattern = core 2024-09-20T23:01:35.690 DEBUG:teuthology.orchestra.run.smithi084:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-20T23:01:35.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T23:01:35.718 DEBUG:teuthology.orchestra.run.smithi154:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-20T23:01:35.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T23:01:35.745 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-20T23:01:35.755 INFO:teuthology.task.internal:Transferring archived files... 2024-09-20T23:01:35.756 DEBUG:teuthology.misc:Transferring archived files from smithi084:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-20_21:08:02-orch-squid-distro-default-smithi/7913018/remote/smithi084 2024-09-20T23:01:35.756 DEBUG:teuthology.orchestra.run.smithi084:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-20T23:01:35.830 DEBUG:teuthology.misc:Transferring archived files from smithi154:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-20_21:08:02-orch-squid-distro-default-smithi/7913018/remote/smithi154 2024-09-20T23:01:35.830 DEBUG:teuthology.orchestra.run.smithi154:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-20T23:01:35.871 INFO:teuthology.task.internal:Removing archive directory... 2024-09-20T23:01:35.872 DEBUG:teuthology.orchestra.run.smithi084:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-20T23:01:35.874 DEBUG:teuthology.orchestra.run.smithi154:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-20T23:01:35.921 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-20T23:01:35.932 INFO:teuthology.task.internal:Not uploading archives. 2024-09-20T23:01:35.932 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-20T23:01:35.944 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-20T23:01:35.945 DEBUG:teuthology.orchestra.run.smithi084:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-20T23:01:35.947 DEBUG:teuthology.orchestra.run.smithi154:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-20T23:01:35.959 INFO:teuthology.orchestra.run.smithi084.stdout: 265324 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 20 23:01 /home/ubuntu/cephtest 2024-09-20T23:01:35.977 INFO:teuthology.orchestra.run.smithi154.stdout: 265607 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 20 23:01 /home/ubuntu/cephtest 2024-09-20T23:01:35.979 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-20T23:01:35.989 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-20T23:01:36.013 INFO:teuthology.run:Summary data: description: orch/cephadm/thrash/{0-distro/centos_9.stream 1-start 2-thrash 3-tasks/snaps-few-objects fixed-2 msgr/async-v2only root} duration: 1914.6391513347626 flavor: default owner: scheduled_teuthology@teuthology success: true 2024-09-20T23:01:36.013 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-20T23:01:36.093 INFO:teuthology.run:pass